2024-03-23T23:50:14.988 INFO:root:teuthology version: 0.0.1.dev262+ge691533 2024-03-23T23:50:14.989 DEBUG:teuthology.run:Teuthology command: teuthology --archive /home/teuthworker/archive/mchangir-2024-03-22_09:49:57-fs-wip-mchangir-testing-main-20240318.032620-testing-default-smithi/7616426 --description fs/workload/{0-centos_9.stream begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/crc wsync/no} objectstore-ec/bluestore-comp omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{balancer/random export-check n/5 replication/always} standby-replay tasks/{0-subvolume/{with-no-extra-options} 1-check-counter 2-scrub/yes 3-snaps/no 4-flush/yes 5-workunit/suites/blogbench}} --name mchangir-2024-03-22_09:49:57-fs-wip-mchangir-testing-main-20240318.032620-testing-default-smithi --verbose --owner mchangir -- /home/teuthworker/archive/mchangir-2024-03-22_09:49:57-fs-wip-mchangir-testing-main-20240318.032620-testing-default-smithi/7616426/orig.config.yaml 2024-03-23T23:50:15.043 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-03-23T23:50:15.135 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/mchangir-2024-03-22_09:49:57-fs-wip-mchangir-testing-main-20240318.032620-testing-default-smithi/7616426 branch: wip-mchangir-testing-main-20240318.032620 description: fs/workload/{0-centos_9.stream begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/crc wsync/no} objectstore-ec/bluestore-comp omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{balancer/random export-check n/5 replication/always} standby-replay tasks/{0-subvolume/{with-no-extra-options} 1-check-counter 2-scrub/yes 3-snaps/no 4-flush/yes 5-workunit/suites/blogbench}} email: mchangir@redhat.com first_in_suite: false job_id: '7616426' kernel: client: branch: testing flavor: default kdb: true sha1: 5ec3e6e88b2e2783e09498588bb1f433ae9c8986 ktype: upstream last_in_suite: false log-rotate: ceph-mds: 10G ceph-osd: 10G machine_type: smithi name: mchangir-2024-03-22_09:49:57-fs-wip-mchangir-testing-main-20240318.032620-testing-default-smithi no_nested_subset: false nuke-on-error: true openstack: - volumes: count: 4 size: 20 - machine: disk: 200 os_type: centos os_version: 9.stream overrides: admin_socket: branch: wip-mchangir-testing-main-20240318.032620 ceph: cephfs: max_mds: 5 session_timeout: 300 standby_replay: true conf: client: client mount timeout: 600 debug client: 20 debug ms: 1 rados mon op timeout: 900 rados osd op timeout: 900 global: ms die on skipped message: false mds: debug mds: 20 debug mds balancer: 20 debug ms: 1 mds bal fragment size max: 10000 mds bal merge size: 5 mds bal split bits: 3 mds bal split size: 100 mds debug frag: true mds debug scatterstat: true mds op complaint time: 180 mds verify scatter: true mds_bal_replicate_threshold: 1 mds_export_ephemeral_random_max: 0.1 osd op complaint time: 180 rados mon op timeout: 900 rados osd op timeout: 900 mgr: cephsqlite lock renewal timeout: 900000 debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 mon op complaint time: 120 osd: bluestore block size: 96636764160 bluestore compression mode: aggressive bluestore fsck on mount: true debug bluefs: 20 debug bluestore: 20 debug ms: 1 debug osd: 20 debug rocksdb: 10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd failsafe full ratio: 0.95 osd objectstore: bluestore osd op complaint time: 180 osd shutdown pgref assert: true osd_max_omap_entries_per_request: 10 flavor: default fs: xfs log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - FS_DEGRADED - FS_INLINE_DATA_DEPRECATED - FS_WITH_FAILED_MDS - MDS_ALL_DOWN - MDS_DAMAGE - MDS_DEGRADED - MDS_FAILED - MDS_INSUFFICIENT_STANDBY - MDS_UP_LESS_THAN_MAX - POOL_APP_NOT_ENABLED - overall HEALTH_ - Replacing daemon - deprecated feature inline_data - overall HEALTH_ - \(OSD_DOWN\) - \(OSD_ - but it is still running - is not responding - MON_DOWN - slow metadata IO - SLOW_OPS - slow request sha1: 3be9a5c9bc793e11e2800a8c0c696e8b46742033 subvols: create: 2 ceph-deploy: conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: {} ceph-fuse: client.0: mount_subvol_num: 0 check-counter: counters: mds: - mds.exported - mds.imported - min: 3 name: mds_cache.dir_update - min: 3 name: mds_cache.dir_update_receipt - mds.dir_split install: ceph: flavor: default sha1: 3be9a5c9bc793e11e2800a8c0c696e8b46742033 kclient: client.0: mount_subvol_num: 1 mntopts: - ms_mode=crc - nowsync syntax: v2 selinux: allowlist: - scontext=system_u:system_r:logrotate_t:s0 - scontext=system_u:system_r:getty_t:s0 thrashosds: bdev_inject_crash: 2 bdev_inject_crash_probability: 0.5 workunit: branch: wip-mchangir-testing-main-20240318.032620 sha1: 3be9a5c9bc793e11e2800a8c0c696e8b46742033 owner: mchangir priority: 100 repo: https://git.ceph.com/ceph-ci.git roles: - - mon.a - mgr.x - mds.a - mds.d - mds.g - mds.j - osd.0 - osd.3 - osd.6 - osd.9 - client.0 - - mon.b - mgr.y - mds.b - mds.e - mds.h - mds.k - osd.1 - osd.4 - osd.7 - osd.10 - - mon.c - mgr.z - mds.c - mds.f - mds.i - mds.l - osd.2 - osd.5 - osd.8 - osd.11 seed: 3550 sha1: 3be9a5c9bc793e11e2800a8c0c696e8b46742033 sleep_before_teardown: 0 subset: 613/1024 suite: fs suite_branch: wip-mchangir-testing-main-20240318.032620 suite_path: /home/teuthworker/src/git.ceph.com_ceph-c_3be9a5c9bc793e11e2800a8c0c696e8b46742033/qa suite_relpath: qa suite_repo: https://git.ceph.com/ceph-ci.git suite_sha1: 3be9a5c9bc793e11e2800a8c0c696e8b46742033 syslog: ignorelist: - WARNING*.*check_session_state - WARNING*.*__ceph_remove_cap targets: smithi118.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBBlaeceO3l80mbVjXsSD+t1Wov6FWYi/ovAtd/z08/ggsRtiB08THEOm0bcwWVZq1bpm398iczQlzsa2AwRQes4= smithi189.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBKQdMYxCUwDO5QhusrlDLYOxsl8ViXX3O+yQwiEJKQ2h8QS8MohdjeprFvUyCTskkodW4SXdzmdPU4SDalnIq0Y= smithi193.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBNDmlR/5bkAuee8YMDSu1SCf+5K9inCnGUjJWullR/Sz4q3z2jv3TqvdMHakwJWH0BJp4IADsmtLmbry2mVtNII= tasks: - install: extra_packages: deb: - python3-cephfs - cephfs-shell - cephfs-top - cephfs-mirror rpm: - python3-cephfs - cephfs-shell - cephfs-top - cephfs-mirror extra_system_packages: deb: - bison - flex - libelf-dev - libssl-dev - network-manager - iproute2 - util-linux - dump - indent - libaio-dev - libtool-bin - uuid-dev - xfslibs-dev - postgresql - postgresql-client - postgresql-common - postgresql-contrib rpm: - bison - flex - elfutils-libelf-devel - openssl-devel - NetworkManager - iproute - util-linux - libacl-devel - libaio-devel - libattr-devel - libtool - libuuid-devel - xfsdump - xfsprogs - xfsprogs-devel - libaio-devel - libtool - libuuid-devel - xfsprogs-devel - postgresql - postgresql-server - postgresql-contrib - cephadm: roleless: false - cephadm.shell: mon.a: - ceph orch status - ceph orch ps - ceph orch ls - ceph orch host ls - ceph orch device ls - cephadm.shell: mon.a: - ceph fs dump - ceph osd dump - fs.ready: timeout: 300 - kclient: null - exec: mon.a: - ceph fs set cephfs balance_automate false - ceph fs subvolumegroup pin cephfs qa random 0.10 - check-counter: {} - fwd_scrub: scrub_timeout: 900 sleep_between_iterations: 1 - background_exec: mon.a: - while sleep 13; do ceph tell mds.cephfs:0 flush journal; done - workunit: clients: all: - suites/blogbench.sh teuthology: fragments_dropped: [] meta: {} postmerge: - local function is_kupstream() return yaml.ktype == 'upstream' end - local function is_kdistro() return yaml.ktype == 'distro' end - local function is_fuse() return false end - local function syntax_version() return yaml.overrides.kclient.syntax end teuthology_branch: main teuthology_sha1: e691533f9cbb33d85b2187bba20d7102f098636d timestamp: 2024-03-22_09:49:57 tube: smithi user: mchangir verbose: false worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.2226885 2024-03-23T23:50:15.135 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/git.ceph.com_ceph-c_3be9a5c9bc793e11e2800a8c0c696e8b46742033/qa; will attempt to use it 2024-03-23T23:50:15.136 INFO:teuthology.run:Found tasks at /home/teuthworker/src/git.ceph.com_ceph-c_3be9a5c9bc793e11e2800a8c0c696e8b46742033/qa/tasks 2024-03-23T23:50:15.136 INFO:teuthology.run_tasks:Running task internal.check_packages... 2024-03-23T23:50:15.138 INFO:teuthology.task.internal:Checking packages... 2024-03-23T23:50:15.162 INFO:teuthology.task.internal:Checking packages for os_type 'centos', flavor 'default' and ceph hash '3be9a5c9bc793e11e2800a8c0c696e8b46742033' 2024-03-23T23:50:15.162 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-03-23T23:50:15.162 INFO:teuthology.packaging:ref: None 2024-03-23T23:50:15.162 INFO:teuthology.packaging:tag: None 2024-03-23T23:50:15.162 INFO:teuthology.packaging:branch: wip-mchangir-testing-main-20240318.032620 2024-03-23T23:50:15.162 INFO:teuthology.packaging:sha1: 3be9a5c9bc793e11e2800a8c0c696e8b46742033 2024-03-23T23:50:15.162 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&ref=wip-mchangir-testing-main-20240318.032620 2024-03-23T23:50:15.306 INFO:teuthology.task.internal:Found packages for ceph version 19.0.0-2145.g3be9a5c9 2024-03-23T23:50:15.308 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2024-03-23T23:50:15.314 INFO:teuthology.task.internal:no buildpackages task found 2024-03-23T23:50:15.314 INFO:teuthology.run_tasks:Running task internal.save_config... 2024-03-23T23:50:15.334 INFO:teuthology.task.internal:Saving configuration 2024-03-23T23:50:15.357 INFO:teuthology.run_tasks:Running task internal.check_lock... 2024-03-23T23:50:15.363 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-03-23T23:50:15.391 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi118.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/mchangir-2024-03-22_09:49:57-fs-wip-mchangir-testing-main-20240318.032620-testing-default-smithi/7616426', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-23 23:44:24.053971', 'locked_by': 'mchangir', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBBlaeceO3l80mbVjXsSD+t1Wov6FWYi/ovAtd/z08/ggsRtiB08THEOm0bcwWVZq1bpm398iczQlzsa2AwRQes4='} 2024-03-23T23:50:15.413 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi189.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/mchangir-2024-03-22_09:49:57-fs-wip-mchangir-testing-main-20240318.032620-testing-default-smithi/7616426', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-23 23:44:24.051807', 'locked_by': 'mchangir', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBKQdMYxCUwDO5QhusrlDLYOxsl8ViXX3O+yQwiEJKQ2h8QS8MohdjeprFvUyCTskkodW4SXdzmdPU4SDalnIq0Y='} 2024-03-23T23:50:15.438 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi193.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/mchangir-2024-03-22_09:49:57-fs-wip-mchangir-testing-main-20240318.032620-testing-default-smithi/7616426', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-23 23:44:24.053007', 'locked_by': 'mchangir', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBNDmlR/5bkAuee8YMDSu1SCf+5K9inCnGUjJWullR/Sz4q3z2jv3TqvdMHakwJWH0BJp4IADsmtLmbry2mVtNII='} 2024-03-23T23:50:15.438 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2024-03-23T23:50:15.445 INFO:teuthology.task.internal:roles: ubuntu@smithi118.front.sepia.ceph.com - ['mon.a', 'mgr.x', 'mds.a', 'mds.d', 'mds.g', 'mds.j', 'osd.0', 'osd.3', 'osd.6', 'osd.9', 'client.0'] 2024-03-23T23:50:15.445 INFO:teuthology.task.internal:roles: ubuntu@smithi189.front.sepia.ceph.com - ['mon.b', 'mgr.y', 'mds.b', 'mds.e', 'mds.h', 'mds.k', 'osd.1', 'osd.4', 'osd.7', 'osd.10'] 2024-03-23T23:50:15.445 INFO:teuthology.task.internal:roles: ubuntu@smithi193.front.sepia.ceph.com - ['mon.c', 'mgr.z', 'mds.c', 'mds.f', 'mds.i', 'mds.l', 'osd.2', 'osd.5', 'osd.8', 'osd.11'] 2024-03-23T23:50:15.445 INFO:teuthology.run_tasks:Running task console_log... 2024-03-23T23:50:15.547 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7f09540e0f70>, signals=[15]) 2024-03-23T23:50:15.547 INFO:teuthology.run_tasks:Running task internal.connect... 2024-03-23T23:50:15.553 INFO:teuthology.task.internal:Opening connections... 2024-03-23T23:50:15.553 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi118.front.sepia.ceph.com 2024-03-23T23:50:15.555 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi118.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-23T23:50:15.646 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi189.front.sepia.ceph.com 2024-03-23T23:50:15.647 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi189.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-23T23:50:15.739 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi193.front.sepia.ceph.com 2024-03-23T23:50:15.739 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi193.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-23T23:50:15.828 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2024-03-23T23:50:15.835 DEBUG:teuthology.orchestra.run.smithi118:> uname -m 2024-03-23T23:50:15.852 INFO:teuthology.orchestra.run.smithi118.stdout:x86_64 2024-03-23T23:50:15.852 DEBUG:teuthology.orchestra.run.smithi118:> cat /etc/os-release 2024-03-23T23:50:15.908 INFO:teuthology.orchestra.run.smithi118.stdout:NAME="CentOS Stream" 2024-03-23T23:50:15.909 INFO:teuthology.orchestra.run.smithi118.stdout:VERSION="9" 2024-03-23T23:50:15.909 INFO:teuthology.orchestra.run.smithi118.stdout:ID="centos" 2024-03-23T23:50:15.909 INFO:teuthology.orchestra.run.smithi118.stdout:ID_LIKE="rhel fedora" 2024-03-23T23:50:15.909 INFO:teuthology.orchestra.run.smithi118.stdout:VERSION_ID="9" 2024-03-23T23:50:15.909 INFO:teuthology.orchestra.run.smithi118.stdout:PLATFORM_ID="platform:el9" 2024-03-23T23:50:15.909 INFO:teuthology.orchestra.run.smithi118.stdout:PRETTY_NAME="CentOS Stream 9" 2024-03-23T23:50:15.909 INFO:teuthology.orchestra.run.smithi118.stdout:ANSI_COLOR="0;31" 2024-03-23T23:50:15.909 INFO:teuthology.orchestra.run.smithi118.stdout:LOGO="fedora-logo-icon" 2024-03-23T23:50:15.909 INFO:teuthology.orchestra.run.smithi118.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-03-23T23:50:15.910 INFO:teuthology.orchestra.run.smithi118.stdout:HOME_URL="https://centos.org/" 2024-03-23T23:50:15.910 INFO:teuthology.orchestra.run.smithi118.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-03-23T23:50:15.910 INFO:teuthology.orchestra.run.smithi118.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-03-23T23:50:15.910 INFO:teuthology.orchestra.run.smithi118.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-03-23T23:50:15.910 INFO:teuthology.lock.ops:Updating smithi118.front.sepia.ceph.com on lock server 2024-03-23T23:50:15.930 DEBUG:teuthology.orchestra.run.smithi189:> uname -m 2024-03-23T23:50:15.948 INFO:teuthology.orchestra.run.smithi189.stdout:x86_64 2024-03-23T23:50:15.948 DEBUG:teuthology.orchestra.run.smithi189:> cat /etc/os-release 2024-03-23T23:50:16.006 INFO:teuthology.orchestra.run.smithi189.stdout:NAME="CentOS Stream" 2024-03-23T23:50:16.006 INFO:teuthology.orchestra.run.smithi189.stdout:VERSION="9" 2024-03-23T23:50:16.006 INFO:teuthology.orchestra.run.smithi189.stdout:ID="centos" 2024-03-23T23:50:16.006 INFO:teuthology.orchestra.run.smithi189.stdout:ID_LIKE="rhel fedora" 2024-03-23T23:50:16.006 INFO:teuthology.orchestra.run.smithi189.stdout:VERSION_ID="9" 2024-03-23T23:50:16.006 INFO:teuthology.orchestra.run.smithi189.stdout:PLATFORM_ID="platform:el9" 2024-03-23T23:50:16.006 INFO:teuthology.orchestra.run.smithi189.stdout:PRETTY_NAME="CentOS Stream 9" 2024-03-23T23:50:16.006 INFO:teuthology.orchestra.run.smithi189.stdout:ANSI_COLOR="0;31" 2024-03-23T23:50:16.006 INFO:teuthology.orchestra.run.smithi189.stdout:LOGO="fedora-logo-icon" 2024-03-23T23:50:16.007 INFO:teuthology.orchestra.run.smithi189.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-03-23T23:50:16.007 INFO:teuthology.orchestra.run.smithi189.stdout:HOME_URL="https://centos.org/" 2024-03-23T23:50:16.007 INFO:teuthology.orchestra.run.smithi189.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-03-23T23:50:16.007 INFO:teuthology.orchestra.run.smithi189.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-03-23T23:50:16.007 INFO:teuthology.orchestra.run.smithi189.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-03-23T23:50:16.007 INFO:teuthology.lock.ops:Updating smithi189.front.sepia.ceph.com on lock server 2024-03-23T23:50:16.029 DEBUG:teuthology.orchestra.run.smithi193:> uname -m 2024-03-23T23:50:16.047 INFO:teuthology.orchestra.run.smithi193.stdout:x86_64 2024-03-23T23:50:16.048 DEBUG:teuthology.orchestra.run.smithi193:> cat /etc/os-release 2024-03-23T23:50:16.104 INFO:teuthology.orchestra.run.smithi193.stdout:NAME="CentOS Stream" 2024-03-23T23:50:16.105 INFO:teuthology.orchestra.run.smithi193.stdout:VERSION="9" 2024-03-23T23:50:16.105 INFO:teuthology.orchestra.run.smithi193.stdout:ID="centos" 2024-03-23T23:50:16.105 INFO:teuthology.orchestra.run.smithi193.stdout:ID_LIKE="rhel fedora" 2024-03-23T23:50:16.105 INFO:teuthology.orchestra.run.smithi193.stdout:VERSION_ID="9" 2024-03-23T23:50:16.105 INFO:teuthology.orchestra.run.smithi193.stdout:PLATFORM_ID="platform:el9" 2024-03-23T23:50:16.105 INFO:teuthology.orchestra.run.smithi193.stdout:PRETTY_NAME="CentOS Stream 9" 2024-03-23T23:50:16.105 INFO:teuthology.orchestra.run.smithi193.stdout:ANSI_COLOR="0;31" 2024-03-23T23:50:16.105 INFO:teuthology.orchestra.run.smithi193.stdout:LOGO="fedora-logo-icon" 2024-03-23T23:50:16.105 INFO:teuthology.orchestra.run.smithi193.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-03-23T23:50:16.105 INFO:teuthology.orchestra.run.smithi193.stdout:HOME_URL="https://centos.org/" 2024-03-23T23:50:16.105 INFO:teuthology.orchestra.run.smithi193.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-03-23T23:50:16.105 INFO:teuthology.orchestra.run.smithi193.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-03-23T23:50:16.105 INFO:teuthology.orchestra.run.smithi193.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-03-23T23:50:16.106 INFO:teuthology.lock.ops:Updating smithi193.front.sepia.ceph.com on lock server 2024-03-23T23:50:16.129 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2024-03-23T23:50:16.143 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2024-03-23T23:50:16.149 INFO:teuthology.task.internal:Checking for old test directory... 2024-03-23T23:50:16.150 DEBUG:teuthology.orchestra.run.smithi118:> test '!' -e /home/ubuntu/cephtest 2024-03-23T23:50:16.152 DEBUG:teuthology.orchestra.run.smithi189:> test '!' -e /home/ubuntu/cephtest 2024-03-23T23:50:16.154 DEBUG:teuthology.orchestra.run.smithi193:> test '!' -e /home/ubuntu/cephtest 2024-03-23T23:50:16.170 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2024-03-23T23:50:16.176 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2024-03-23T23:50:16.177 DEBUG:teuthology.orchestra.run.smithi118:> test -z $(ls -A /var/lib/ceph) 2024-03-23T23:50:16.209 DEBUG:teuthology.orchestra.run.smithi189:> test -z $(ls -A /var/lib/ceph) 2024-03-23T23:50:16.211 DEBUG:teuthology.orchestra.run.smithi193:> test -z $(ls -A /var/lib/ceph) 2024-03-23T23:50:16.316 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2024-03-23T23:50:16.391 INFO:teuthology.run_tasks:Running task kernel... 2024-03-23T23:50:16.407 INFO:teuthology.task.kernel:normalize config orig: {'client': {'branch': 'testing'}, 'flavor': 'default', 'kdb': True, 'sha1': '5ec3e6e88b2e2783e09498588bb1f433ae9c8986'} 2024-03-23T23:50:16.407 INFO:teuthology.task.kernel:normalize config final: {'client.0': {'branch': 'testing'}} 2024-03-23T23:50:16.408 INFO:teuthology.task.kernel:config {'client.0': {'branch': 'testing'}}, timeout 300 2024-03-23T23:50:16.408 DEBUG:teuthology.orchestra.run.smithi118:> test -f /run/.containerenv -o -f /.dockerenv 2024-03-23T23:50:16.424 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-23T23:50:16.425 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=kernel&flavor=default&distros=centos%2F9%2Fx86_64&ref=testing 2024-03-23T23:50:16.717 DEBUG:teuthology.task.kernel:sha1 for client.0 is 5ec3e6e88b2e2783e09498588bb1f433ae9c8986 2024-03-23T23:50:16.717 INFO:teuthology.task.kernel:Checking kernel version of client.0, want "5ec3e6e88b2e2783e09498588bb1f433ae9c8986"... 2024-03-23T23:50:16.718 DEBUG:teuthology.orchestra.run.smithi118:> uname -r 2024-03-23T23:50:16.734 INFO:teuthology.orchestra.run.smithi118.stdout:5.14.0-361.el9.x86_64 2024-03-23T23:50:16.734 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-361.el9.x86_64 vs 5ec3e6e88b2e2783e09498588bb1f433ae9c8986 2024-03-23T23:50:16.735 DEBUG:teuthology.task.kernel:failed to parse current kernel version 2024-03-23T23:50:16.735 DEBUG:teuthology.orchestra.run.smithi118:> sudo yum upgrade -y linux-firmware 2024-03-23T23:50:19.955 INFO:teuthology.orchestra.run.smithi118.stdout:CentOS-9 - AppStream 12 MB/s | 6.7 MB 00:00 2024-03-23T23:50:21.597 INFO:teuthology.orchestra.run.smithi118.stdout:CentOS-9 - BaseOS 3.7 MB/s | 2.0 MB 00:00 2024-03-23T23:50:22.404 INFO:teuthology.orchestra.run.smithi118.stdout:CentOS Stream 9 - BaseOS 18 MB/s | 8.0 MB 00:00 2024-03-23T23:50:24.355 INFO:teuthology.orchestra.run.smithi118.stdout:CentOS Stream 9 - AppStream 21 MB/s | 19 MB 00:00 2024-03-23T23:50:29.439 INFO:teuthology.orchestra.run.smithi118.stdout:CentOS Stream 9 - CRB 3.7 MB/s | 6.1 MB 00:01 2024-03-23T23:50:30.881 INFO:teuthology.orchestra.run.smithi118.stdout:CentOS Stream 9 - Extras packages 48 kB/s | 16 kB 00:00 2024-03-23T23:50:32.170 INFO:teuthology.orchestra.run.smithi118.stdout:Extra Packages for Enterprise Linux 18 MB/s | 21 MB 00:01 2024-03-23T23:50:37.007 INFO:teuthology.orchestra.run.smithi118.stdout:lab-extras 37 kB/s | 1.7 kB 00:00 2024-03-23T23:50:39.204 INFO:teuthology.orchestra.run.smithi118.stdout:Dependencies resolved. 2024-03-23T23:50:39.215 INFO:teuthology.orchestra.run.smithi118.stdout:================================================================================ 2024-03-23T23:50:39.215 INFO:teuthology.orchestra.run.smithi118.stdout: Package Arch Version Repository Size 2024-03-23T23:50:39.215 INFO:teuthology.orchestra.run.smithi118.stdout:================================================================================ 2024-03-23T23:50:39.215 INFO:teuthology.orchestra.run.smithi118.stdout:Upgrading: 2024-03-23T23:50:39.215 INFO:teuthology.orchestra.run.smithi118.stdout: linux-firmware noarch 20240219-143.el9 CentOS-BaseOS 375 M 2024-03-23T23:50:39.215 INFO:teuthology.orchestra.run.smithi118.stdout: 2024-03-23T23:50:39.215 INFO:teuthology.orchestra.run.smithi118.stdout:Transaction Summary 2024-03-23T23:50:39.216 INFO:teuthology.orchestra.run.smithi118.stdout:================================================================================ 2024-03-23T23:50:39.216 INFO:teuthology.orchestra.run.smithi118.stdout:Upgrade 1 Package 2024-03-23T23:50:39.216 INFO:teuthology.orchestra.run.smithi118.stdout: 2024-03-23T23:50:39.216 INFO:teuthology.orchestra.run.smithi118.stdout:Total download size: 375 M 2024-03-23T23:50:39.216 INFO:teuthology.orchestra.run.smithi118.stdout:Downloading Packages: 2024-03-23T23:50:47.691 INFO:teuthology.orchestra.run.smithi118.stdout:linux-firmware-20240219-143.el9.noarch.rpm 44 MB/s | 375 MB 00:08 2024-03-23T23:50:47.692 INFO:teuthology.orchestra.run.smithi118.stdout:-------------------------------------------------------------------------------- 2024-03-23T23:50:47.692 INFO:teuthology.orchestra.run.smithi118.stdout:Total 44 MB/s | 375 MB 00:08 2024-03-23T23:50:47.692 INFO:teuthology.orchestra.run.smithi118.stdout:Running transaction check 2024-03-23T23:50:47.777 INFO:teuthology.orchestra.run.smithi118.stdout:Transaction check succeeded. 2024-03-23T23:50:47.778 INFO:teuthology.orchestra.run.smithi118.stdout:Running transaction test 2024-03-23T23:50:50.239 INFO:teuthology.orchestra.run.smithi118.stdout:Transaction test succeeded. 2024-03-23T23:50:50.240 INFO:teuthology.orchestra.run.smithi118.stdout:Running transaction 2024-03-23T23:50:56.547 INFO:teuthology.orchestra.run.smithi118.stdout: Preparing : 1/1 2024-03-23T23:50:56.782 INFO:teuthology.orchestra.run.smithi118.stdout: Upgrading : linux-firmware-20240219-143.el9.noarch 1/2 2024-03-23T23:50:56.856 INFO:teuthology.orchestra.run.smithi118.stdout: Cleanup : linux-firmware-20230814-139.el9.noarch 2/2 2024-03-23T23:50:58.376 INFO:teuthology.orchestra.run.smithi118.stdout: Running scriptlet: linux-firmware-20230814-139.el9.noarch 2/2 2024-03-23T23:50:58.376 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : linux-firmware-20240219-143.el9.noarch 1/2 2024-03-23T23:50:58.766 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : linux-firmware-20230814-139.el9.noarch 2/2 2024-03-23T23:50:58.766 INFO:teuthology.orchestra.run.smithi118.stdout: 2024-03-23T23:50:58.766 INFO:teuthology.orchestra.run.smithi118.stdout:Upgraded: 2024-03-23T23:50:58.767 INFO:teuthology.orchestra.run.smithi118.stdout: linux-firmware-20240219-143.el9.noarch 2024-03-23T23:50:58.767 INFO:teuthology.orchestra.run.smithi118.stdout: 2024-03-23T23:50:58.767 INFO:teuthology.orchestra.run.smithi118.stdout:Complete! 2024-03-23T23:50:59.051 INFO:teuthology.task.kernel:Downloading kernel 5ec3e6e88b2e2783e09498588bb1f433ae9c8986 on client.0... 2024-03-23T23:50:59.052 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=kernel&flavor=default&distros=centos%2F9%2Fx86_64&sha1=5ec3e6e88b2e2783e09498588bb1f433ae9c8986 2024-03-23T23:50:59.209 INFO:teuthology.task.kernel:fetching, builder baseurl is https://4.chacra.ceph.com/r/kernel/testing/5ec3e6e88b2e2783e09498588bb1f433ae9c8986/centos/9/flavors/default/x86_64/ 2024-03-23T23:50:59.209 DEBUG:teuthology.orchestra.run.smithi118:> rm -f /tmp/kernel.x86_64.rpm && echo kernel-6.8.0_g5ec3e6e88b2e-1.x86_64.rpm | wget -nv -O /tmp/kernel.x86_64.rpm --base=https://4.chacra.ceph.com/r/kernel/testing/5ec3e6e88b2e2783e09498588bb1f433ae9c8986/centos/9/flavors/default/x86_64/ --input-file=- 2024-03-23T23:50:59.215 DEBUG:teuthology.task.kernel:Waiting for download/copy to ubuntu@smithi118.front.sepia.ceph.com to complete... 2024-03-23T23:51:10.622 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-23 23:51:10 URL:https://4.chacra.ceph.com/r/kernel/testing/5ec3e6e88b2e2783e09498588bb1f433ae9c8986/centos/9/flavors/default/x86_64/kernel-6.8.0_g5ec3e6e88b2e-1.x86_64.rpm [336309687/336309687] -> "/tmp/kernel.x86_64.rpm" [1] 2024-03-23T23:51:10.622 INFO:teuthology.orchestra.run.smithi118.stderr:FINISHED --2024-03-23 23:51:10-- 2024-03-23T23:51:10.622 INFO:teuthology.orchestra.run.smithi118.stderr:Total wall clock time: 11s 2024-03-23T23:51:10.622 INFO:teuthology.orchestra.run.smithi118.stderr:Downloaded: 1 files, 321M in 11s (28.9 MB/s) 2024-03-23T23:51:10.624 INFO:teuthology.task.kernel:Installing kernel 5ec3e6e88b2e2783e09498588bb1f433ae9c8986 on client.0... 2024-03-23T23:51:10.625 DEBUG:teuthology.orchestra.run.smithi118:> sudo rpm -ivh --oldpackage --replacefiles --replacepkgs /tmp/kernel.x86_64.rpm 2024-03-23T23:51:12.566 INFO:teuthology.orchestra.run.smithi118.stdout:Verifying... ######################################## 2024-03-23T23:51:12.693 INFO:teuthology.orchestra.run.smithi118.stdout:Preparing... ######################################## 2024-03-23T23:51:12.771 INFO:teuthology.orchestra.run.smithi118.stdout:Updating / installing... 2024-03-23T23:51:20.254 INFO:teuthology.orchestra.run.smithi118.stdout:kernel-6.8.0_g5ec3e6e88b2e-1 ######################################## 2024-03-23T23:51:38.177 INFO:teuthology.orchestra.run.smithi118.stderr:dracut: Disabling early microcode, because kernel does not support it. CONFIG_MICROCODE_[AMD|INTEL]!=y 2024-03-23T23:52:00.569 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi118.front.sepia.ceph.com, path=/tmp/kernel.x86_64.rpm, version=None) 2024-03-23T23:52:00.569 DEBUG:teuthology.orchestra.run.smithi118:> rpm -qlp /tmp/kernel.x86_64.rpm 2024-03-23T23:52:00.645 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e 2024-03-23T23:52:00.645 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/System.map 2024-03-23T23:52:00.645 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/config 2024-03-23T23:52:00.645 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel 2024-03-23T23:52:00.645 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/arch 2024-03-23T23:52:00.645 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/arch/x86 2024-03-23T23:52:00.646 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/arch/x86/crypto 2024-03-23T23:52:00.646 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/arch/x86/crypto/blowfish-x86_64.ko 2024-03-23T23:52:00.646 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/arch/x86/crypto/camellia-aesni-avx-x86_64.ko 2024-03-23T23:52:00.646 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/arch/x86/crypto/camellia-aesni-avx2.ko 2024-03-23T23:52:00.646 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/arch/x86/crypto/camellia-x86_64.ko 2024-03-23T23:52:00.646 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/arch/x86/crypto/cast5-avx-x86_64.ko 2024-03-23T23:52:00.646 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/arch/x86/crypto/cast6-avx-x86_64.ko 2024-03-23T23:52:00.646 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/arch/x86/crypto/crc32-pclmul.ko 2024-03-23T23:52:00.646 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/arch/x86/crypto/crc32c-intel.ko 2024-03-23T23:52:00.646 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/arch/x86/crypto/crct10dif-pclmul.ko 2024-03-23T23:52:00.646 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/arch/x86/crypto/des3_ede-x86_64.ko 2024-03-23T23:52:00.646 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/arch/x86/crypto/ghash-clmulni-intel.ko 2024-03-23T23:52:00.646 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/arch/x86/crypto/serpent-avx-x86_64.ko 2024-03-23T23:52:00.647 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/arch/x86/crypto/serpent-avx2.ko 2024-03-23T23:52:00.647 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/arch/x86/crypto/serpent-sse2-x86_64.ko 2024-03-23T23:52:00.647 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/arch/x86/crypto/sha1-ssse3.ko 2024-03-23T23:52:00.647 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/arch/x86/crypto/sha256-ssse3.ko 2024-03-23T23:52:00.647 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/arch/x86/crypto/sha512-ssse3.ko 2024-03-23T23:52:00.647 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/arch/x86/crypto/twofish-avx-x86_64.ko 2024-03-23T23:52:00.647 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/arch/x86/crypto/twofish-x86_64-3way.ko 2024-03-23T23:52:00.647 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/arch/x86/crypto/twofish-x86_64.ko 2024-03-23T23:52:00.647 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/arch/x86/kernel 2024-03-23T23:52:00.647 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/arch/x86/kernel/cpu 2024-03-23T23:52:00.647 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/arch/x86/kernel/cpu/mce 2024-03-23T23:52:00.647 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/arch/x86/kernel/cpu/mce/mce-inject.ko 2024-03-23T23:52:00.647 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/arch/x86/kvm 2024-03-23T23:52:00.647 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/arch/x86/kvm/kvm-amd.ko 2024-03-23T23:52:00.648 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/arch/x86/kvm/kvm-intel.ko 2024-03-23T23:52:00.648 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/arch/x86/kvm/kvm.ko 2024-03-23T23:52:00.648 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/crypto 2024-03-23T23:52:00.648 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/crypto/ansi_cprng.ko 2024-03-23T23:52:00.648 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/crypto/anubis.ko 2024-03-23T23:52:00.648 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/crypto/arc4.ko 2024-03-23T23:52:00.648 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/crypto/async_tx 2024-03-23T23:52:00.648 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/crypto/async_tx/async_memcpy.ko 2024-03-23T23:52:00.648 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/crypto/async_tx/async_pq.ko 2024-03-23T23:52:00.648 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/crypto/async_tx/async_raid6_recov.ko 2024-03-23T23:52:00.648 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/crypto/async_tx/async_tx.ko 2024-03-23T23:52:00.648 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/crypto/async_tx/async_xor.ko 2024-03-23T23:52:00.648 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/crypto/async_tx/raid6test.ko 2024-03-23T23:52:00.649 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/crypto/authenc.ko 2024-03-23T23:52:00.649 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/crypto/authencesn.ko 2024-03-23T23:52:00.649 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/crypto/blake2b_generic.ko 2024-03-23T23:52:00.649 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/crypto/blowfish_common.ko 2024-03-23T23:52:00.649 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/crypto/blowfish_generic.ko 2024-03-23T23:52:00.649 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/crypto/camellia_generic.ko 2024-03-23T23:52:00.649 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/crypto/cast5_generic.ko 2024-03-23T23:52:00.649 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/crypto/cast6_generic.ko 2024-03-23T23:52:00.649 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/crypto/cast_common.ko 2024-03-23T23:52:00.649 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/crypto/ccm.ko 2024-03-23T23:52:00.649 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/crypto/cmac.ko 2024-03-23T23:52:00.649 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/crypto/crc32_generic.ko 2024-03-23T23:52:00.649 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/crypto/crypto_engine.ko 2024-03-23T23:52:00.649 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/crypto/crypto_user.ko 2024-03-23T23:52:00.650 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/crypto/cts.ko 2024-03-23T23:52:00.650 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/crypto/des_generic.ko 2024-03-23T23:52:00.650 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/crypto/dh_generic.ko 2024-03-23T23:52:00.650 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/crypto/echainiv.ko 2024-03-23T23:52:00.650 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/crypto/essiv.ko 2024-03-23T23:52:00.650 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/crypto/fcrypt.ko 2024-03-23T23:52:00.650 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/crypto/khazad.ko 2024-03-23T23:52:00.650 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/crypto/lz4.ko 2024-03-23T23:52:00.650 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/crypto/lz4hc.ko 2024-03-23T23:52:00.650 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/crypto/md4.ko 2024-03-23T23:52:00.650 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/crypto/michael_mic.ko 2024-03-23T23:52:00.650 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/crypto/pcbc.ko 2024-03-23T23:52:00.650 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/crypto/pcrypt.ko 2024-03-23T23:52:00.650 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/crypto/rmd160.ko 2024-03-23T23:52:00.651 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/crypto/seed.ko 2024-03-23T23:52:00.651 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/crypto/serpent_generic.ko 2024-03-23T23:52:00.651 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/crypto/tcrypt.ko 2024-03-23T23:52:00.651 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/crypto/tea.ko 2024-03-23T23:52:00.651 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/crypto/twofish_common.ko 2024-03-23T23:52:00.651 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/crypto/twofish_generic.ko 2024-03-23T23:52:00.651 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/crypto/vmac.ko 2024-03-23T23:52:00.651 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/crypto/wp512.ko 2024-03-23T23:52:00.651 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/crypto/xcbc.ko 2024-03-23T23:52:00.651 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/crypto/xor.ko 2024-03-23T23:52:00.651 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/crypto/xxhash_generic.ko 2024-03-23T23:52:00.651 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers 2024-03-23T23:52:00.651 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/acpi 2024-03-23T23:52:00.651 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/acpi/acpi_ipmi.ko 2024-03-23T23:52:00.652 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/acpi/acpi_pad.ko 2024-03-23T23:52:00.652 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/acpi/custom_method.ko 2024-03-23T23:52:00.652 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/acpi/ec_sys.ko 2024-03-23T23:52:00.652 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/acpi/platform_profile.ko 2024-03-23T23:52:00.652 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/acpi/sbs.ko 2024-03-23T23:52:00.652 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/acpi/sbshc.ko 2024-03-23T23:52:00.652 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/acpi/video.ko 2024-03-23T23:52:00.652 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ata 2024-03-23T23:52:00.652 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ata/acard-ahci.ko 2024-03-23T23:52:00.652 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ata/ahci_platform.ko 2024-03-23T23:52:00.652 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ata/ata_generic.ko 2024-03-23T23:52:00.652 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ata/libahci_platform.ko 2024-03-23T23:52:00.652 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ata/pata_acpi.ko 2024-03-23T23:52:00.652 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ata/pata_ali.ko 2024-03-23T23:52:00.652 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ata/pata_amd.ko 2024-03-23T23:52:00.653 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ata/pata_artop.ko 2024-03-23T23:52:00.653 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ata/pata_atiixp.ko 2024-03-23T23:52:00.653 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ata/pata_atp867x.ko 2024-03-23T23:52:00.653 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ata/pata_cmd640.ko 2024-03-23T23:52:00.653 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ata/pata_cmd64x.ko 2024-03-23T23:52:00.653 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ata/pata_cypress.ko 2024-03-23T23:52:00.653 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ata/pata_efar.ko 2024-03-23T23:52:00.653 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ata/pata_hpt366.ko 2024-03-23T23:52:00.653 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ata/pata_hpt37x.ko 2024-03-23T23:52:00.653 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ata/pata_hpt3x2n.ko 2024-03-23T23:52:00.653 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ata/pata_hpt3x3.ko 2024-03-23T23:52:00.653 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ata/pata_it8213.ko 2024-03-23T23:52:00.653 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ata/pata_it821x.ko 2024-03-23T23:52:00.653 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ata/pata_jmicron.ko 2024-03-23T23:52:00.654 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ata/pata_marvell.ko 2024-03-23T23:52:00.654 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ata/pata_mpiix.ko 2024-03-23T23:52:00.654 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ata/pata_netcell.ko 2024-03-23T23:52:00.654 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ata/pata_ninja32.ko 2024-03-23T23:52:00.654 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ata/pata_ns87410.ko 2024-03-23T23:52:00.654 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ata/pata_ns87415.ko 2024-03-23T23:52:00.654 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ata/pata_oldpiix.ko 2024-03-23T23:52:00.654 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ata/pata_opti.ko 2024-03-23T23:52:00.654 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ata/pata_optidma.ko 2024-03-23T23:52:00.654 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ata/pata_pcmcia.ko 2024-03-23T23:52:00.654 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ata/pata_pdc2027x.ko 2024-03-23T23:52:00.654 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ata/pata_pdc202xx_old.ko 2024-03-23T23:52:00.654 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ata/pata_piccolo.ko 2024-03-23T23:52:00.655 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ata/pata_rdc.ko 2024-03-23T23:52:00.655 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ata/pata_sch.ko 2024-03-23T23:52:00.655 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ata/pata_serverworks.ko 2024-03-23T23:52:00.655 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ata/pata_sil680.ko 2024-03-23T23:52:00.655 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ata/pata_sis.ko 2024-03-23T23:52:00.655 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ata/pata_sl82c105.ko 2024-03-23T23:52:00.655 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ata/pata_triflex.ko 2024-03-23T23:52:00.655 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ata/pata_via.ko 2024-03-23T23:52:00.655 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ata/pdc_adma.ko 2024-03-23T23:52:00.655 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ata/sata_inic162x.ko 2024-03-23T23:52:00.655 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ata/sata_mv.ko 2024-03-23T23:52:00.655 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ata/sata_nv.ko 2024-03-23T23:52:00.655 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ata/sata_promise.ko 2024-03-23T23:52:00.655 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ata/sata_qstor.ko 2024-03-23T23:52:00.655 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ata/sata_sil.ko 2024-03-23T23:52:00.656 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ata/sata_sil24.ko 2024-03-23T23:52:00.656 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ata/sata_sis.ko 2024-03-23T23:52:00.656 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ata/sata_svw.ko 2024-03-23T23:52:00.656 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ata/sata_sx4.ko 2024-03-23T23:52:00.656 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ata/sata_uli.ko 2024-03-23T23:52:00.656 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ata/sata_via.ko 2024-03-23T23:52:00.656 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ata/sata_vsc.ko 2024-03-23T23:52:00.656 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/auxdisplay 2024-03-23T23:52:00.656 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/auxdisplay/ks0108.ko 2024-03-23T23:52:00.656 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/base 2024-03-23T23:52:00.656 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/base/regmap 2024-03-23T23:52:00.656 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/base/regmap/regmap-i2c.ko 2024-03-23T23:52:00.656 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/base/regmap/regmap-mmio.ko 2024-03-23T23:52:00.656 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/bcma 2024-03-23T23:52:00.657 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/bcma/bcma.ko 2024-03-23T23:52:00.657 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/block 2024-03-23T23:52:00.657 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/block/aoe 2024-03-23T23:52:00.657 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/block/aoe/aoe.ko 2024-03-23T23:52:00.657 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/block/brd.ko 2024-03-23T23:52:00.657 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/block/drbd 2024-03-23T23:52:00.657 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/block/drbd/drbd.ko 2024-03-23T23:52:00.657 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/block/floppy.ko 2024-03-23T23:52:00.657 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/block/loop.ko 2024-03-23T23:52:00.657 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/block/mtip32xx 2024-03-23T23:52:00.657 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/block/mtip32xx/mtip32xx.ko 2024-03-23T23:52:00.657 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/block/nbd.ko 2024-03-23T23:52:00.657 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/block/null_blk 2024-03-23T23:52:00.657 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/block/null_blk/null_blk.ko 2024-03-23T23:52:00.658 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/block/pktcdvd.ko 2024-03-23T23:52:00.658 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/block/rbd.ko 2024-03-23T23:52:00.658 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/block/virtio_blk.ko 2024-03-23T23:52:00.658 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/block/xen-blkback 2024-03-23T23:52:00.658 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/block/xen-blkback/xen-blkback.ko 2024-03-23T23:52:00.658 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/block/xen-blkfront.ko 2024-03-23T23:52:00.658 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/block/zram 2024-03-23T23:52:00.658 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/block/zram/zram.ko 2024-03-23T23:52:00.658 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/char 2024-03-23T23:52:00.658 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/char/hangcheck-timer.ko 2024-03-23T23:52:00.658 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/char/hw_random 2024-03-23T23:52:00.658 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/char/hw_random/amd-rng.ko 2024-03-23T23:52:00.658 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/char/hw_random/intel-rng.ko 2024-03-23T23:52:00.658 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/char/hw_random/timeriomem-rng.ko 2024-03-23T23:52:00.659 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/char/hw_random/via-rng.ko 2024-03-23T23:52:00.659 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/char/hw_random/virtio-rng.ko 2024-03-23T23:52:00.659 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/char/ipmi 2024-03-23T23:52:00.659 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/char/ipmi/ipmi_devintf.ko 2024-03-23T23:52:00.659 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/char/ipmi/ipmi_msghandler.ko 2024-03-23T23:52:00.659 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/char/ipmi/ipmi_poweroff.ko 2024-03-23T23:52:00.659 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/char/ipmi/ipmi_si.ko 2024-03-23T23:52:00.659 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/char/ipmi/ipmi_ssif.ko 2024-03-23T23:52:00.659 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/char/ipmi/ipmi_watchdog.ko 2024-03-23T23:52:00.659 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/char/lp.ko 2024-03-23T23:52:00.659 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/char/mwave 2024-03-23T23:52:00.659 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/char/mwave/mwave.ko 2024-03-23T23:52:00.659 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/char/ppdev.ko 2024-03-23T23:52:00.659 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/char/tlclk.ko 2024-03-23T23:52:00.660 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/char/tpm 2024-03-23T23:52:00.660 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/char/tpm/tpm.ko 2024-03-23T23:52:00.660 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/char/tpm/tpm_atmel.ko 2024-03-23T23:52:00.660 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/char/tpm/tpm_infineon.ko 2024-03-23T23:52:00.660 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/char/tpm/tpm_nsc.ko 2024-03-23T23:52:00.660 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/char/tpm/tpm_tis.ko 2024-03-23T23:52:00.660 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/char/tpm/tpm_tis_core.ko 2024-03-23T23:52:00.660 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/char/uv_mmtimer.ko 2024-03-23T23:52:00.660 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/char/virtio_console.ko 2024-03-23T23:52:00.660 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/cpufreq 2024-03-23T23:52:00.660 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/cpufreq/acpi-cpufreq.ko 2024-03-23T23:52:00.661 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/cpufreq/amd_freq_sensitivity.ko 2024-03-23T23:52:00.661 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/cpufreq/p4-clockmod.ko 2024-03-23T23:52:00.661 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/cpufreq/pcc-cpufreq.ko 2024-03-23T23:52:00.661 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/cpufreq/powernow-k8.ko 2024-03-23T23:52:00.661 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/cpufreq/speedstep-lib.ko 2024-03-23T23:52:00.661 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/crypto 2024-03-23T23:52:00.661 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/crypto/ccp 2024-03-23T23:52:00.661 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/crypto/ccp/ccp-crypto.ko 2024-03-23T23:52:00.661 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/crypto/ccp/ccp.ko 2024-03-23T23:52:00.661 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/crypto/intel 2024-03-23T23:52:00.661 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/crypto/intel/qat 2024-03-23T23:52:00.662 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/crypto/intel/qat/qat_common 2024-03-23T23:52:00.662 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/crypto/intel/qat/qat_common/intel_qat.ko 2024-03-23T23:52:00.662 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/crypto/intel/qat/qat_dh895xcc 2024-03-23T23:52:00.662 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/crypto/intel/qat/qat_dh895xcc/qat_dh895xcc.ko 2024-03-23T23:52:00.662 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/crypto/padlock-aes.ko 2024-03-23T23:52:00.662 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/crypto/padlock-sha.ko 2024-03-23T23:52:00.662 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/crypto/virtio 2024-03-23T23:52:00.662 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/crypto/virtio/virtio_crypto.ko 2024-03-23T23:52:00.662 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/dca 2024-03-23T23:52:00.662 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/dca/dca.ko 2024-03-23T23:52:00.662 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/devfreq 2024-03-23T23:52:00.662 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/devfreq/governor_simpleondemand.ko 2024-03-23T23:52:00.662 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/dma 2024-03-23T23:52:00.663 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/dma/dw 2024-03-23T23:52:00.663 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/dma/dw/dw_dmac.ko 2024-03-23T23:52:00.663 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/dma/ioat 2024-03-23T23:52:00.663 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/dma/ioat/ioatdma.ko 2024-03-23T23:52:00.663 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/edac 2024-03-23T23:52:00.663 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/edac/amd64_edac.ko 2024-03-23T23:52:00.663 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/edac/e752x_edac.ko 2024-03-23T23:52:00.663 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/edac/edac_mce_amd.ko 2024-03-23T23:52:00.663 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/edac/i3000_edac.ko 2024-03-23T23:52:00.663 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/edac/i3200_edac.ko 2024-03-23T23:52:00.664 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/edac/i5100_edac.ko 2024-03-23T23:52:00.664 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/edac/i5400_edac.ko 2024-03-23T23:52:00.664 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/edac/i7300_edac.ko 2024-03-23T23:52:00.664 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/edac/i7core_edac.ko 2024-03-23T23:52:00.664 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/edac/i82975x_edac.ko 2024-03-23T23:52:00.664 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/edac/ie31200_edac.ko 2024-03-23T23:52:00.664 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/edac/sb_edac.ko 2024-03-23T23:52:00.664 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/edac/x38_edac.ko 2024-03-23T23:52:00.664 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/firmware 2024-03-23T23:52:00.664 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/firmware/edd.ko 2024-03-23T23:52:00.664 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/firmware/iscsi_ibft.ko 2024-03-23T23:52:00.665 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/gpio 2024-03-23T23:52:00.665 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/gpio/gpio-ich.ko 2024-03-23T23:52:00.665 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/gpio/gpio-viperboard.ko 2024-03-23T23:52:00.665 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid 2024-03-23T23:52:00.665 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/hid-appleir.ko 2024-03-23T23:52:00.665 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/hid-aureal.ko 2024-03-23T23:52:00.665 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/hid-axff.ko 2024-03-23T23:52:00.665 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/hid-dr.ko 2024-03-23T23:52:00.665 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/hid-elecom.ko 2024-03-23T23:52:00.666 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/hid-elo.ko 2024-03-23T23:52:00.666 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/hid-emsff.ko 2024-03-23T23:52:00.666 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/hid-gaff.ko 2024-03-23T23:52:00.666 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/hid-gt683r.ko 2024-03-23T23:52:00.666 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/hid-gyration.ko 2024-03-23T23:52:00.666 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/hid-holtek-kbd.ko 2024-03-23T23:52:00.666 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/hid-holtek-mouse.ko 2024-03-23T23:52:00.666 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/hid-holtekff.ko 2024-03-23T23:52:00.666 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/hid-hyperv.ko 2024-03-23T23:52:00.666 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/hid-icade.ko 2024-03-23T23:52:00.666 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/hid-keytouch.ko 2024-03-23T23:52:00.666 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/hid-kye.ko 2024-03-23T23:52:00.666 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/hid-lcpower.ko 2024-03-23T23:52:00.667 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/hid-led.ko 2024-03-23T23:52:00.667 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/hid-lenovo.ko 2024-03-23T23:52:00.667 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/hid-logitech-dj.ko 2024-03-23T23:52:00.667 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/hid-logitech-hidpp.ko 2024-03-23T23:52:00.667 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/hid-multitouch.ko 2024-03-23T23:52:00.667 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/hid-ortek.ko 2024-03-23T23:52:00.667 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/hid-penmount.ko 2024-03-23T23:52:00.667 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/hid-petalynx.ko 2024-03-23T23:52:00.668 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/hid-picolcd.ko 2024-03-23T23:52:00.668 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/hid-pl.ko 2024-03-23T23:52:00.668 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/hid-plantronics.ko 2024-03-23T23:52:00.668 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/hid-primax.ko 2024-03-23T23:52:00.668 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/hid-rmi.ko 2024-03-23T23:52:00.668 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/hid-roccat-arvo.ko 2024-03-23T23:52:00.668 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/hid-roccat-common.ko 2024-03-23T23:52:00.668 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/hid-roccat-isku.ko 2024-03-23T23:52:00.668 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/hid-roccat-kone.ko 2024-03-23T23:52:00.668 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/hid-roccat-koneplus.ko 2024-03-23T23:52:00.668 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/hid-roccat-konepure.ko 2024-03-23T23:52:00.668 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/hid-roccat-kovaplus.ko 2024-03-23T23:52:00.669 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/hid-roccat-lua.ko 2024-03-23T23:52:00.669 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/hid-roccat-pyra.ko 2024-03-23T23:52:00.669 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/hid-roccat-ryos.ko 2024-03-23T23:52:00.669 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/hid-roccat-savu.ko 2024-03-23T23:52:00.669 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/hid-roccat.ko 2024-03-23T23:52:00.669 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/hid-saitek.ko 2024-03-23T23:52:00.669 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/hid-samsung.ko 2024-03-23T23:52:00.669 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/hid-sensor-hub.ko 2024-03-23T23:52:00.669 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/hid-sjoy.ko 2024-03-23T23:52:00.670 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/hid-sony.ko 2024-03-23T23:52:00.670 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/hid-speedlink.ko 2024-03-23T23:52:00.670 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/hid-steelseries.ko 2024-03-23T23:52:00.670 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/hid-sunplus.ko 2024-03-23T23:52:00.670 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/hid-thrustmaster.ko 2024-03-23T23:52:00.670 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/hid-tivo.ko 2024-03-23T23:52:00.670 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/hid-tmff.ko 2024-03-23T23:52:00.670 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/hid-topseed.ko 2024-03-23T23:52:00.670 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/hid-twinhan.ko 2024-03-23T23:52:00.670 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/hid-uclogic.ko 2024-03-23T23:52:00.670 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/hid-waltop.ko 2024-03-23T23:52:00.671 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/hid-wiimote.ko 2024-03-23T23:52:00.671 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/hid-xinmo.ko 2024-03-23T23:52:00.671 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/hid-zpff.ko 2024-03-23T23:52:00.671 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/hid-zydacron.ko 2024-03-23T23:52:00.671 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/uhid.ko 2024-03-23T23:52:00.671 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hid/wacom.ko 2024-03-23T23:52:00.671 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hv 2024-03-23T23:52:00.671 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hv/hv_balloon.ko 2024-03-23T23:52:00.671 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hv/hv_utils.ko 2024-03-23T23:52:00.671 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hv/hv_vmbus.ko 2024-03-23T23:52:00.671 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon 2024-03-23T23:52:00.672 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/abituguru.ko 2024-03-23T23:52:00.672 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/abituguru3.ko 2024-03-23T23:52:00.672 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/acpi_power_meter.ko 2024-03-23T23:52:00.672 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/ad7414.ko 2024-03-23T23:52:00.672 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/ad7418.ko 2024-03-23T23:52:00.672 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/adc128d818.ko 2024-03-23T23:52:00.672 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/adm1025.ko 2024-03-23T23:52:00.672 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/adm1026.ko 2024-03-23T23:52:00.672 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/adm1029.ko 2024-03-23T23:52:00.673 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/adm1031.ko 2024-03-23T23:52:00.673 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/adm9240.ko 2024-03-23T23:52:00.673 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/ads7828.ko 2024-03-23T23:52:00.673 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/adt7410.ko 2024-03-23T23:52:00.673 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/adt7411.ko 2024-03-23T23:52:00.673 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/adt7462.ko 2024-03-23T23:52:00.673 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/adt7470.ko 2024-03-23T23:52:00.673 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/adt7475.ko 2024-03-23T23:52:00.673 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/adt7x10.ko 2024-03-23T23:52:00.674 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/amc6821.ko 2024-03-23T23:52:00.674 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/applesmc.ko 2024-03-23T23:52:00.674 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/asb100.ko 2024-03-23T23:52:00.674 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/asc7621.ko 2024-03-23T23:52:00.674 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/asus_atk0110.ko 2024-03-23T23:52:00.674 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/atxp1.ko 2024-03-23T23:52:00.674 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/coretemp.ko 2024-03-23T23:52:00.674 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/dell-smm-hwmon.ko 2024-03-23T23:52:00.674 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/dme1737.ko 2024-03-23T23:52:00.674 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/ds1621.ko 2024-03-23T23:52:00.674 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/ds620.ko 2024-03-23T23:52:00.674 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/emc1403.ko 2024-03-23T23:52:00.674 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/emc6w201.ko 2024-03-23T23:52:00.674 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/f71805f.ko 2024-03-23T23:52:00.675 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/f71882fg.ko 2024-03-23T23:52:00.675 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/f75375s.ko 2024-03-23T23:52:00.675 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/fam15h_power.ko 2024-03-23T23:52:00.675 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/fschmd.ko 2024-03-23T23:52:00.675 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/g760a.ko 2024-03-23T23:52:00.675 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/g762.ko 2024-03-23T23:52:00.675 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/gl518sm.ko 2024-03-23T23:52:00.675 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/gl520sm.ko 2024-03-23T23:52:00.675 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/hwmon-vid.ko 2024-03-23T23:52:00.675 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/i5500_temp.ko 2024-03-23T23:52:00.675 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/i5k_amb.ko 2024-03-23T23:52:00.675 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/ibmaem.ko 2024-03-23T23:52:00.675 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/ibmpex.ko 2024-03-23T23:52:00.675 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/ina209.ko 2024-03-23T23:52:00.676 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/ina2xx.ko 2024-03-23T23:52:00.676 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/it87.ko 2024-03-23T23:52:00.676 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/k10temp.ko 2024-03-23T23:52:00.676 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/k8temp.ko 2024-03-23T23:52:00.676 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/lineage-pem.ko 2024-03-23T23:52:00.676 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/lm63.ko 2024-03-23T23:52:00.676 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/lm73.ko 2024-03-23T23:52:00.676 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/lm75.ko 2024-03-23T23:52:00.676 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/lm77.ko 2024-03-23T23:52:00.676 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/lm78.ko 2024-03-23T23:52:00.676 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/lm80.ko 2024-03-23T23:52:00.676 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/lm83.ko 2024-03-23T23:52:00.676 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/lm85.ko 2024-03-23T23:52:00.677 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/lm87.ko 2024-03-23T23:52:00.677 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/lm90.ko 2024-03-23T23:52:00.677 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/lm92.ko 2024-03-23T23:52:00.677 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/lm93.ko 2024-03-23T23:52:00.677 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/lm95234.ko 2024-03-23T23:52:00.677 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/lm95241.ko 2024-03-23T23:52:00.677 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/lm95245.ko 2024-03-23T23:52:00.677 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/ltc2945.ko 2024-03-23T23:52:00.677 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/ltc4151.ko 2024-03-23T23:52:00.677 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/ltc4215.ko 2024-03-23T23:52:00.677 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/ltc4222.ko 2024-03-23T23:52:00.677 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/ltc4245.ko 2024-03-23T23:52:00.677 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/ltc4260.ko 2024-03-23T23:52:00.677 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/ltc4261.ko 2024-03-23T23:52:00.678 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/max16065.ko 2024-03-23T23:52:00.678 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/max1619.ko 2024-03-23T23:52:00.678 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/max1668.ko 2024-03-23T23:52:00.678 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/max197.ko 2024-03-23T23:52:00.678 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/max6639.ko 2024-03-23T23:52:00.678 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/max6650.ko 2024-03-23T23:52:00.678 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/max6697.ko 2024-03-23T23:52:00.678 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/mcp3021.ko 2024-03-23T23:52:00.678 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/nct6683.ko 2024-03-23T23:52:00.678 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/nct6775-core.ko 2024-03-23T23:52:00.678 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/nct6775.ko 2024-03-23T23:52:00.678 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/nct7802.ko 2024-03-23T23:52:00.678 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/pc87360.ko 2024-03-23T23:52:00.678 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/pc87427.ko 2024-03-23T23:52:00.679 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/pcf8591.ko 2024-03-23T23:52:00.679 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/pmbus 2024-03-23T23:52:00.679 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/pmbus/adm1275.ko 2024-03-23T23:52:00.679 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/pmbus/lm25066.ko 2024-03-23T23:52:00.679 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/pmbus/ltc2978.ko 2024-03-23T23:52:00.679 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/pmbus/max16064.ko 2024-03-23T23:52:00.679 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/pmbus/max34440.ko 2024-03-23T23:52:00.679 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/pmbus/max8688.ko 2024-03-23T23:52:00.679 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/pmbus/pmbus.ko 2024-03-23T23:52:00.679 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/pmbus/pmbus_core.ko 2024-03-23T23:52:00.679 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/pmbus/tps40422.ko 2024-03-23T23:52:00.679 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/pmbus/ucd9000.ko 2024-03-23T23:52:00.679 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/pmbus/ucd9200.ko 2024-03-23T23:52:00.679 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/pmbus/zl6100.ko 2024-03-23T23:52:00.680 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/powr1220.ko 2024-03-23T23:52:00.680 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/sch5627.ko 2024-03-23T23:52:00.680 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/sch5636.ko 2024-03-23T23:52:00.680 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/sch56xx-common.ko 2024-03-23T23:52:00.680 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/sht15.ko 2024-03-23T23:52:00.680 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/sht21.ko 2024-03-23T23:52:00.680 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/shtc1.ko 2024-03-23T23:52:00.680 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/sis5595.ko 2024-03-23T23:52:00.680 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/smsc47b397.ko 2024-03-23T23:52:00.680 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/smsc47m1.ko 2024-03-23T23:52:00.680 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/smsc47m192.ko 2024-03-23T23:52:00.680 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/thmc50.ko 2024-03-23T23:52:00.680 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/tmp102.ko 2024-03-23T23:52:00.680 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/tmp103.ko 2024-03-23T23:52:00.681 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/tmp401.ko 2024-03-23T23:52:00.681 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/tmp421.ko 2024-03-23T23:52:00.681 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/via-cputemp.ko 2024-03-23T23:52:00.681 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/via686a.ko 2024-03-23T23:52:00.681 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/vt1211.ko 2024-03-23T23:52:00.681 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/vt8231.ko 2024-03-23T23:52:00.681 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/w83627ehf.ko 2024-03-23T23:52:00.681 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/w83627hf.ko 2024-03-23T23:52:00.681 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/w83781d.ko 2024-03-23T23:52:00.681 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/w83791d.ko 2024-03-23T23:52:00.681 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/w83792d.ko 2024-03-23T23:52:00.681 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/w83793.ko 2024-03-23T23:52:00.681 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/w83795.ko 2024-03-23T23:52:00.681 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/w83l785ts.ko 2024-03-23T23:52:00.682 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/hwmon/w83l786ng.ko 2024-03-23T23:52:00.682 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/i2c 2024-03-23T23:52:00.682 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/i2c/algos 2024-03-23T23:52:00.682 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/i2c/algos/i2c-algo-bit.ko 2024-03-23T23:52:00.682 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/i2c/algos/i2c-algo-pca.ko 2024-03-23T23:52:00.682 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/i2c/busses 2024-03-23T23:52:00.682 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/i2c/busses/i2c-amd756-s4882.ko 2024-03-23T23:52:00.682 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/i2c/busses/i2c-amd756.ko 2024-03-23T23:52:00.682 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/i2c/busses/i2c-amd8111.ko 2024-03-23T23:52:00.682 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/i2c/busses/i2c-ccgx-ucsi.ko 2024-03-23T23:52:00.682 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/i2c/busses/i2c-designware-core.ko 2024-03-23T23:52:00.682 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/i2c/busses/i2c-designware-pci.ko 2024-03-23T23:52:00.682 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/i2c/busses/i2c-designware-platform.ko 2024-03-23T23:52:00.682 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/i2c/busses/i2c-diolan-u2c.ko 2024-03-23T23:52:00.683 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/i2c/busses/i2c-i801.ko 2024-03-23T23:52:00.683 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/i2c/busses/i2c-isch.ko 2024-03-23T23:52:00.683 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/i2c/busses/i2c-ismt.ko 2024-03-23T23:52:00.683 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/i2c/busses/i2c-nforce2-s4985.ko 2024-03-23T23:52:00.683 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/i2c/busses/i2c-nforce2.ko 2024-03-23T23:52:00.683 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/i2c/busses/i2c-parport.ko 2024-03-23T23:52:00.683 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/i2c/busses/i2c-pca-platform.ko 2024-03-23T23:52:00.683 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/i2c/busses/i2c-piix4.ko 2024-03-23T23:52:00.683 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/i2c/busses/i2c-scmi.ko 2024-03-23T23:52:00.683 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/i2c/busses/i2c-simtec.ko 2024-03-23T23:52:00.683 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/i2c/busses/i2c-sis96x.ko 2024-03-23T23:52:00.683 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/i2c/busses/i2c-tiny-usb.ko 2024-03-23T23:52:00.683 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/i2c/busses/i2c-via.ko 2024-03-23T23:52:00.683 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/i2c/busses/i2c-viapro.ko 2024-03-23T23:52:00.684 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/i2c/busses/i2c-viperboard.ko 2024-03-23T23:52:00.684 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/i2c/i2c-dev.ko 2024-03-23T23:52:00.684 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/i2c/i2c-mux.ko 2024-03-23T23:52:00.684 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/i2c/i2c-slave-eeprom.ko 2024-03-23T23:52:00.684 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/i2c/i2c-smbus.ko 2024-03-23T23:52:00.684 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/i2c/i2c-stub.ko 2024-03-23T23:52:00.684 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/infiniband 2024-03-23T23:52:00.684 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/infiniband/core 2024-03-23T23:52:00.684 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/infiniband/core/ib_cm.ko 2024-03-23T23:52:00.684 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/infiniband/core/ib_core.ko 2024-03-23T23:52:00.684 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/infiniband/core/ib_umad.ko 2024-03-23T23:52:00.684 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/infiniband/core/ib_uverbs.ko 2024-03-23T23:52:00.684 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/infiniband/core/iw_cm.ko 2024-03-23T23:52:00.684 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/infiniband/core/rdma_cm.ko 2024-03-23T23:52:00.685 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/infiniband/core/rdma_ucm.ko 2024-03-23T23:52:00.685 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/infiniband/hw 2024-03-23T23:52:00.685 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/infiniband/hw/cxgb4 2024-03-23T23:52:00.685 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/infiniband/hw/cxgb4/iw_cxgb4.ko 2024-03-23T23:52:00.685 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/infiniband/hw/mlx4 2024-03-23T23:52:00.685 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/infiniband/hw/mlx4/mlx4_ib.ko 2024-03-23T23:52:00.685 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/infiniband/hw/mlx5 2024-03-23T23:52:00.685 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/infiniband/hw/mlx5/mlx5_ib.ko 2024-03-23T23:52:00.685 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/infiniband/hw/mthca 2024-03-23T23:52:00.685 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/infiniband/hw/mthca/ib_mthca.ko 2024-03-23T23:52:00.685 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/infiniband/ulp 2024-03-23T23:52:00.685 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/infiniband/ulp/ipoib 2024-03-23T23:52:00.685 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/infiniband/ulp/ipoib/ib_ipoib.ko 2024-03-23T23:52:00.685 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/infiniband/ulp/iser 2024-03-23T23:52:00.686 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/infiniband/ulp/iser/ib_iser.ko 2024-03-23T23:52:00.686 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/infiniband/ulp/isert 2024-03-23T23:52:00.686 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/infiniband/ulp/isert/ib_isert.ko 2024-03-23T23:52:00.686 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/infiniband/ulp/srp 2024-03-23T23:52:00.686 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/infiniband/ulp/srp/ib_srp.ko 2024-03-23T23:52:00.686 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/infiniband/ulp/srpt 2024-03-23T23:52:00.686 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/infiniband/ulp/srpt/ib_srpt.ko 2024-03-23T23:52:00.686 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/input 2024-03-23T23:52:00.686 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/input/gameport 2024-03-23T23:52:00.686 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/input/gameport/emu10k1-gp.ko 2024-03-23T23:52:00.686 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/input/gameport/fm801-gp.ko 2024-03-23T23:52:00.686 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/input/gameport/gameport.ko 2024-03-23T23:52:00.686 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/input/joydev.ko 2024-03-23T23:52:00.686 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/input/keyboard 2024-03-23T23:52:00.687 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/input/keyboard/gpio_keys.ko 2024-03-23T23:52:00.687 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/input/mouse 2024-03-23T23:52:00.687 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/input/mouse/appletouch.ko 2024-03-23T23:52:00.687 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/input/mouse/bcm5974.ko 2024-03-23T23:52:00.687 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/input/mouse/cyapatp.ko 2024-03-23T23:52:00.687 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/input/mouse/elan_i2c.ko 2024-03-23T23:52:00.687 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/input/mouse/sermouse.ko 2024-03-23T23:52:00.687 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/input/mouse/synaptics_i2c.ko 2024-03-23T23:52:00.687 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/input/mouse/synaptics_usb.ko 2024-03-23T23:52:00.687 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/input/mouse/vsxxxaa.ko 2024-03-23T23:52:00.687 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/input/rmi4 2024-03-23T23:52:00.687 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/input/rmi4/rmi_core.ko 2024-03-23T23:52:00.687 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/input/serio 2024-03-23T23:52:00.687 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/input/serio/altera_ps2.ko 2024-03-23T23:52:00.688 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/input/serio/arc_ps2.ko 2024-03-23T23:52:00.688 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/input/serio/hyperv-keyboard.ko 2024-03-23T23:52:00.688 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/input/serio/serio_raw.ko 2024-03-23T23:52:00.688 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/input/sparse-keymap.ko 2024-03-23T23:52:00.688 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/leds 2024-03-23T23:52:00.688 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/leds/leds-blinkm.ko 2024-03-23T23:52:00.688 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/leds/leds-lm3530.ko 2024-03-23T23:52:00.688 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/leds/leds-lp3944.ko 2024-03-23T23:52:00.688 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/leds/leds-ss4200.ko 2024-03-23T23:52:00.688 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/leds/trigger 2024-03-23T23:52:00.688 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/leds/trigger/ledtrig-audio.ko 2024-03-23T23:52:00.688 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/leds/trigger/ledtrig-backlight.ko 2024-03-23T23:52:00.688 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/leds/trigger/ledtrig-camera.ko 2024-03-23T23:52:00.688 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/leds/trigger/ledtrig-default-on.ko 2024-03-23T23:52:00.689 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/leds/trigger/ledtrig-gpio.ko 2024-03-23T23:52:00.689 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/leds/trigger/ledtrig-heartbeat.ko 2024-03-23T23:52:00.689 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/leds/trigger/ledtrig-oneshot.ko 2024-03-23T23:52:00.689 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/leds/trigger/ledtrig-timer.ko 2024-03-23T23:52:00.689 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/leds/trigger/ledtrig-transient.ko 2024-03-23T23:52:00.689 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/md 2024-03-23T23:52:00.689 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/md/bcache 2024-03-23T23:52:00.689 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/md/bcache/bcache.ko 2024-03-23T23:52:00.689 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/md/dm-bio-prison.ko 2024-03-23T23:52:00.689 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/md/dm-cache-smq.ko 2024-03-23T23:52:00.689 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/md/dm-cache.ko 2024-03-23T23:52:00.689 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/md/dm-crypt.ko 2024-03-23T23:52:00.689 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/md/dm-delay.ko 2024-03-23T23:52:00.689 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/md/dm-flakey.ko 2024-03-23T23:52:00.690 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/md/dm-log-userspace.ko 2024-03-23T23:52:00.690 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/md/dm-multipath.ko 2024-03-23T23:52:00.690 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/md/dm-queue-length.ko 2024-03-23T23:52:00.690 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/md/dm-raid.ko 2024-03-23T23:52:00.690 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/md/dm-round-robin.ko 2024-03-23T23:52:00.690 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/md/dm-service-time.ko 2024-03-23T23:52:00.690 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/md/dm-switch.ko 2024-03-23T23:52:00.690 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/md/dm-thin-pool.ko 2024-03-23T23:52:00.690 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/md/dm-verity.ko 2024-03-23T23:52:00.690 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/md/persistent-data 2024-03-23T23:52:00.690 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/md/persistent-data/dm-persistent-data.ko 2024-03-23T23:52:00.690 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/md/raid0.ko 2024-03-23T23:52:00.690 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/md/raid1.ko 2024-03-23T23:52:00.690 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/md/raid10.ko 2024-03-23T23:52:00.691 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/md/raid456.ko 2024-03-23T23:52:00.691 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/message 2024-03-23T23:52:00.691 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/message/fusion 2024-03-23T23:52:00.691 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/message/fusion/mptbase.ko 2024-03-23T23:52:00.691 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/message/fusion/mptctl.ko 2024-03-23T23:52:00.691 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/message/fusion/mptfc.ko 2024-03-23T23:52:00.691 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/message/fusion/mptlan.ko 2024-03-23T23:52:00.691 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/message/fusion/mptsas.ko 2024-03-23T23:52:00.691 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/message/fusion/mptscsih.ko 2024-03-23T23:52:00.691 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/message/fusion/mptspi.ko 2024-03-23T23:52:00.691 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/mfd 2024-03-23T23:52:00.691 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/mfd/lpc_ich.ko 2024-03-23T23:52:00.691 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/mfd/lpc_sch.ko 2024-03-23T23:52:00.691 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/mfd/mfd-core.ko 2024-03-23T23:52:00.691 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/mfd/sm501.ko 2024-03-23T23:52:00.692 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/mfd/viperboard.ko 2024-03-23T23:52:00.692 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/mfd/vx855.ko 2024-03-23T23:52:00.692 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/mfd/wl1273-core.ko 2024-03-23T23:52:00.692 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/misc 2024-03-23T23:52:00.692 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/misc/altera-stapl 2024-03-23T23:52:00.692 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/misc/altera-stapl/altera-stapl.ko 2024-03-23T23:52:00.692 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/misc/apds9802als.ko 2024-03-23T23:52:00.692 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/misc/apds990x.ko 2024-03-23T23:52:00.693 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/misc/bh1770glc.ko 2024-03-23T23:52:00.693 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/misc/cb710 2024-03-23T23:52:00.693 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/misc/cb710/cb710.ko 2024-03-23T23:52:00.693 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/misc/eeprom 2024-03-23T23:52:00.693 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/misc/eeprom/at24.ko 2024-03-23T23:52:00.693 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/misc/eeprom/eeprom_93cx6.ko 2024-03-23T23:52:00.693 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/misc/eeprom/max6875.ko 2024-03-23T23:52:00.693 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/misc/enclosure.ko 2024-03-23T23:52:00.693 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/misc/hpilo.ko 2024-03-23T23:52:00.693 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/misc/isl29003.ko 2024-03-23T23:52:00.694 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/misc/isl29020.ko 2024-03-23T23:52:00.694 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/misc/lis3lv02d 2024-03-23T23:52:00.694 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/misc/lis3lv02d/lis3lv02d.ko 2024-03-23T23:52:00.694 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/misc/lis3lv02d/lis3lv02d_i2c.ko 2024-03-23T23:52:00.694 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/misc/mei 2024-03-23T23:52:00.694 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/misc/mei/mei-me.ko 2024-03-23T23:52:00.694 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/misc/mei/mei-txe.ko 2024-03-23T23:52:00.694 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/misc/mei/mei.ko 2024-03-23T23:52:00.694 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/misc/sgi-gru 2024-03-23T23:52:00.694 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/misc/sgi-gru/gru.ko 2024-03-23T23:52:00.694 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/misc/sgi-xp 2024-03-23T23:52:00.694 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/misc/sgi-xp/xp.ko 2024-03-23T23:52:00.694 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/misc/sgi-xp/xpc.ko 2024-03-23T23:52:00.695 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/misc/sgi-xp/xpnet.ko 2024-03-23T23:52:00.695 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/misc/tifm_7xx1.ko 2024-03-23T23:52:00.695 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/misc/tifm_core.ko 2024-03-23T23:52:00.695 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/misc/tsl2550.ko 2024-03-23T23:52:00.695 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/misc/vmw_balloon.ko 2024-03-23T23:52:00.695 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/misc/vmw_vmci 2024-03-23T23:52:00.695 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/misc/vmw_vmci/vmw_vmci.ko 2024-03-23T23:52:00.695 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net 2024-03-23T23:52:00.695 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/bonding 2024-03-23T23:52:00.695 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/bonding/bonding.ko 2024-03-23T23:52:00.695 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/dsa 2024-03-23T23:52:00.695 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/dsa/b53 2024-03-23T23:52:00.695 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/dsa/b53/b53_common.ko 2024-03-23T23:52:00.695 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/dsa/bcm-sf2.ko 2024-03-23T23:52:00.696 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/dsa/mv88e6060.ko 2024-03-23T23:52:00.696 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/dsa/mv88e6xxx 2024-03-23T23:52:00.696 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/dsa/mv88e6xxx/mv88e6xxx.ko 2024-03-23T23:52:00.696 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/dummy.ko 2024-03-23T23:52:00.696 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/eql.ko 2024-03-23T23:52:00.696 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet 2024-03-23T23:52:00.696 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/3com 2024-03-23T23:52:00.696 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/3com/3c574_cs.ko 2024-03-23T23:52:00.696 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/3com/3c589_cs.ko 2024-03-23T23:52:00.696 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/3com/3c59x.ko 2024-03-23T23:52:00.696 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/3com/typhoon.ko 2024-03-23T23:52:00.696 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/8390 2024-03-23T23:52:00.696 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/8390/8390.ko 2024-03-23T23:52:00.696 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/8390/axnet_cs.ko 2024-03-23T23:52:00.697 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/8390/ne2k-pci.ko 2024-03-23T23:52:00.697 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/8390/pcnet_cs.ko 2024-03-23T23:52:00.697 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/adaptec 2024-03-23T23:52:00.697 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/adaptec/starfire.ko 2024-03-23T23:52:00.697 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/agere 2024-03-23T23:52:00.697 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/agere/et131x.ko 2024-03-23T23:52:00.697 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/alteon 2024-03-23T23:52:00.697 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/alteon/acenic.ko 2024-03-23T23:52:00.697 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/altera 2024-03-23T23:52:00.697 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/altera/altera_tse.ko 2024-03-23T23:52:00.697 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/amd 2024-03-23T23:52:00.697 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/amd/amd8111e.ko 2024-03-23T23:52:00.697 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/amd/nmclan_cs.ko 2024-03-23T23:52:00.697 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/amd/pcnet32.ko 2024-03-23T23:52:00.698 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/atheros 2024-03-23T23:52:00.698 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/atheros/alx 2024-03-23T23:52:00.698 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/atheros/alx/alx.ko 2024-03-23T23:52:00.698 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/atheros/atl1c 2024-03-23T23:52:00.698 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/atheros/atl1c/atl1c.ko 2024-03-23T23:52:00.698 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/atheros/atl1e 2024-03-23T23:52:00.698 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/atheros/atl1e/atl1e.ko 2024-03-23T23:52:00.698 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/atheros/atlx 2024-03-23T23:52:00.698 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/atheros/atlx/atl1.ko 2024-03-23T23:52:00.698 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/atheros/atlx/atl2.ko 2024-03-23T23:52:00.698 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/broadcom 2024-03-23T23:52:00.698 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/broadcom/b44.ko 2024-03-23T23:52:00.698 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/broadcom/bnx2.ko 2024-03-23T23:52:00.699 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/broadcom/bnx2x 2024-03-23T23:52:00.699 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/broadcom/bnx2x/bnx2x.ko 2024-03-23T23:52:00.699 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/broadcom/cnic.ko 2024-03-23T23:52:00.699 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/broadcom/genet 2024-03-23T23:52:00.699 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/broadcom/genet/genet.ko 2024-03-23T23:52:00.699 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/broadcom/tg3.ko 2024-03-23T23:52:00.699 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/brocade 2024-03-23T23:52:00.699 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/brocade/bna 2024-03-23T23:52:00.699 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/brocade/bna/bna.ko 2024-03-23T23:52:00.699 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/chelsio 2024-03-23T23:52:00.699 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/chelsio/cxgb 2024-03-23T23:52:00.699 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/chelsio/cxgb/cxgb.ko 2024-03-23T23:52:00.699 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/chelsio/cxgb3 2024-03-23T23:52:00.699 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/chelsio/cxgb3/cxgb3.ko 2024-03-23T23:52:00.700 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/chelsio/cxgb4 2024-03-23T23:52:00.700 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/chelsio/cxgb4/cxgb4.ko 2024-03-23T23:52:00.700 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/chelsio/cxgb4vf 2024-03-23T23:52:00.700 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/chelsio/cxgb4vf/cxgb4vf.ko 2024-03-23T23:52:00.700 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/chelsio/libcxgb 2024-03-23T23:52:00.700 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/chelsio/libcxgb/libcxgb.ko 2024-03-23T23:52:00.700 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/cisco 2024-03-23T23:52:00.700 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/cisco/enic 2024-03-23T23:52:00.700 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/cisco/enic/enic.ko 2024-03-23T23:52:00.700 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/dec 2024-03-23T23:52:00.700 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/dec/tulip 2024-03-23T23:52:00.700 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/dec/tulip/de2104x.ko 2024-03-23T23:52:00.700 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/dec/tulip/dmfe.ko 2024-03-23T23:52:00.700 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/dec/tulip/tulip.ko 2024-03-23T23:52:00.701 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/dec/tulip/uli526x.ko 2024-03-23T23:52:00.701 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/dec/tulip/winbond-840.ko 2024-03-23T23:52:00.701 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/dec/tulip/xircom_cb.ko 2024-03-23T23:52:00.701 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/dlink 2024-03-23T23:52:00.701 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/dlink/dl2k.ko 2024-03-23T23:52:00.701 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/dlink/sundance.ko 2024-03-23T23:52:00.701 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/dnet.ko 2024-03-23T23:52:00.701 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/emulex 2024-03-23T23:52:00.701 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/emulex/benet 2024-03-23T23:52:00.701 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/emulex/benet/be2net.ko 2024-03-23T23:52:00.701 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/ethoc.ko 2024-03-23T23:52:00.701 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/fealnx.ko 2024-03-23T23:52:00.701 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/intel 2024-03-23T23:52:00.702 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/intel/e100.ko 2024-03-23T23:52:00.702 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/intel/e1000 2024-03-23T23:52:00.702 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/intel/e1000/e1000.ko 2024-03-23T23:52:00.702 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/intel/e1000e 2024-03-23T23:52:00.702 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/intel/e1000e/e1000e.ko 2024-03-23T23:52:00.702 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/intel/fm10k 2024-03-23T23:52:00.702 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/intel/fm10k/fm10k.ko 2024-03-23T23:52:00.702 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/intel/i40e 2024-03-23T23:52:00.702 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/intel/i40e/i40e.ko 2024-03-23T23:52:00.702 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/intel/iavf 2024-03-23T23:52:00.702 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/intel/iavf/iavf.ko 2024-03-23T23:52:00.702 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/intel/igb 2024-03-23T23:52:00.702 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/intel/igb/igb.ko 2024-03-23T23:52:00.703 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/intel/igbvf 2024-03-23T23:52:00.703 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/intel/igbvf/igbvf.ko 2024-03-23T23:52:00.703 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/intel/ixgbe 2024-03-23T23:52:00.703 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/intel/ixgbe/ixgbe.ko 2024-03-23T23:52:00.703 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/intel/ixgbevf 2024-03-23T23:52:00.703 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/intel/ixgbevf/ixgbevf.ko 2024-03-23T23:52:00.703 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/jme.ko 2024-03-23T23:52:00.703 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/marvell 2024-03-23T23:52:00.703 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/marvell/mvmdio.ko 2024-03-23T23:52:00.703 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/marvell/skge.ko 2024-03-23T23:52:00.703 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/marvell/sky2.ko 2024-03-23T23:52:00.703 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/mellanox 2024-03-23T23:52:00.703 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/mellanox/mlx4 2024-03-23T23:52:00.704 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/mellanox/mlx4/mlx4_core.ko 2024-03-23T23:52:00.704 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/mellanox/mlx4/mlx4_en.ko 2024-03-23T23:52:00.704 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/mellanox/mlx5 2024-03-23T23:52:00.704 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/mellanox/mlx5/core 2024-03-23T23:52:00.704 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/mellanox/mlx5/core/mlx5_core.ko 2024-03-23T23:52:00.704 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/micrel 2024-03-23T23:52:00.704 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/micrel/ksz884x.ko 2024-03-23T23:52:00.704 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/myricom 2024-03-23T23:52:00.704 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/myricom/myri10ge 2024-03-23T23:52:00.704 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/myricom/myri10ge/myri10ge.ko 2024-03-23T23:52:00.704 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/natsemi 2024-03-23T23:52:00.704 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/natsemi/natsemi.ko 2024-03-23T23:52:00.704 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/natsemi/ns83820.ko 2024-03-23T23:52:00.705 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/neterion 2024-03-23T23:52:00.705 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/neterion/s2io.ko 2024-03-23T23:52:00.705 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/nvidia 2024-03-23T23:52:00.705 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/nvidia/forcedeth.ko 2024-03-23T23:52:00.705 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/packetengines 2024-03-23T23:52:00.705 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/packetengines/hamachi.ko 2024-03-23T23:52:00.705 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/packetengines/yellowfin.ko 2024-03-23T23:52:00.705 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/qlogic 2024-03-23T23:52:00.705 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/qlogic/netxen 2024-03-23T23:52:00.705 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/qlogic/netxen/netxen_nic.ko 2024-03-23T23:52:00.705 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/qlogic/qla3xxx.ko 2024-03-23T23:52:00.705 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/qlogic/qlcnic 2024-03-23T23:52:00.705 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/qlogic/qlcnic/qlcnic.ko 2024-03-23T23:52:00.705 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/rdc 2024-03-23T23:52:00.706 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/rdc/r6040.ko 2024-03-23T23:52:00.706 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/realtek 2024-03-23T23:52:00.706 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/realtek/8139cp.ko 2024-03-23T23:52:00.706 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/realtek/8139too.ko 2024-03-23T23:52:00.706 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/realtek/atp.ko 2024-03-23T23:52:00.706 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/realtek/r8169.ko 2024-03-23T23:52:00.706 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/rocker 2024-03-23T23:52:00.706 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/rocker/rocker.ko 2024-03-23T23:52:00.706 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/sfc 2024-03-23T23:52:00.706 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/sfc/sfc.ko 2024-03-23T23:52:00.706 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/silan 2024-03-23T23:52:00.706 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/silan/sc92031.ko 2024-03-23T23:52:00.706 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/sis 2024-03-23T23:52:00.707 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/sis/sis190.ko 2024-03-23T23:52:00.707 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/sis/sis900.ko 2024-03-23T23:52:00.707 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/smsc 2024-03-23T23:52:00.707 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/smsc/epic100.ko 2024-03-23T23:52:00.707 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/smsc/smc91c92_cs.ko 2024-03-23T23:52:00.707 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/smsc/smsc911x.ko 2024-03-23T23:52:00.707 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/smsc/smsc9420.ko 2024-03-23T23:52:00.707 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/stmicro 2024-03-23T23:52:00.707 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/stmicro/stmmac 2024-03-23T23:52:00.707 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/stmicro/stmmac/dwmac-intel.ko 2024-03-23T23:52:00.707 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/stmicro/stmmac/stmmac.ko 2024-03-23T23:52:00.707 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/sun 2024-03-23T23:52:00.707 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/sun/cassini.ko 2024-03-23T23:52:00.707 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/sun/niu.ko 2024-03-23T23:52:00.708 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/sun/sungem.ko 2024-03-23T23:52:00.708 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/sun/sunhme.ko 2024-03-23T23:52:00.708 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/tehuti 2024-03-23T23:52:00.708 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/tehuti/tehuti.ko 2024-03-23T23:52:00.708 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/ti 2024-03-23T23:52:00.708 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/ti/tlan.ko 2024-03-23T23:52:00.708 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/via 2024-03-23T23:52:00.708 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/via/via-rhine.ko 2024-03-23T23:52:00.708 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/via/via-velocity.ko 2024-03-23T23:52:00.708 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/wiznet 2024-03-23T23:52:00.708 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/wiznet/w5100.ko 2024-03-23T23:52:00.708 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/wiznet/w5300.ko 2024-03-23T23:52:00.708 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/xircom 2024-03-23T23:52:00.708 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ethernet/xircom/xirc2ps_cs.ko 2024-03-23T23:52:00.709 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/geneve.ko 2024-03-23T23:52:00.709 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/hyperv 2024-03-23T23:52:00.709 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/hyperv/hv_netvsc.ko 2024-03-23T23:52:00.709 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ieee802154 2024-03-23T23:52:00.709 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ieee802154/fakelb.ko 2024-03-23T23:52:00.709 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ifb.ko 2024-03-23T23:52:00.709 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ipvlan 2024-03-23T23:52:00.709 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ipvlan/ipvlan.ko 2024-03-23T23:52:00.709 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/macvlan.ko 2024-03-23T23:52:00.709 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/macvtap.ko 2024-03-23T23:52:00.709 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/mdio 2024-03-23T23:52:00.709 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/mdio.ko 2024-03-23T23:52:00.709 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/mdio/mdio-bcm-unimac.ko 2024-03-23T23:52:00.709 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/mdio/mdio-bitbang.ko 2024-03-23T23:52:00.709 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/mdio/mdio-regmap.ko 2024-03-23T23:52:00.710 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/mii.ko 2024-03-23T23:52:00.710 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/net_failover.ko 2024-03-23T23:52:00.710 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/netconsole.ko 2024-03-23T23:52:00.710 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/nlmon.ko 2024-03-23T23:52:00.710 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/pcs 2024-03-23T23:52:00.710 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/pcs/pcs-lynx.ko 2024-03-23T23:52:00.710 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/pcs/pcs_xpcs.ko 2024-03-23T23:52:00.710 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/phy 2024-03-23T23:52:00.710 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/phy/amd.ko 2024-03-23T23:52:00.710 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/phy/bcm-phy-lib.ko 2024-03-23T23:52:00.710 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/phy/bcm-phy-ptp.ko 2024-03-23T23:52:00.710 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/phy/bcm7xxx.ko 2024-03-23T23:52:00.710 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/phy/bcm87xx.ko 2024-03-23T23:52:00.711 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/phy/broadcom.ko 2024-03-23T23:52:00.711 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/phy/cicada.ko 2024-03-23T23:52:00.711 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/phy/davicom.ko 2024-03-23T23:52:00.711 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/phy/dp83640.ko 2024-03-23T23:52:00.711 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/phy/et1011c.ko 2024-03-23T23:52:00.711 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/phy/icplus.ko 2024-03-23T23:52:00.711 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/phy/lxt.ko 2024-03-23T23:52:00.711 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/phy/marvell.ko 2024-03-23T23:52:00.711 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/phy/micrel.ko 2024-03-23T23:52:00.711 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/phy/national.ko 2024-03-23T23:52:00.711 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/phy/phylink.ko 2024-03-23T23:52:00.711 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/phy/qsemi.ko 2024-03-23T23:52:00.711 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/phy/realtek.ko 2024-03-23T23:52:00.711 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/phy/smsc.ko 2024-03-23T23:52:00.712 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/phy/ste10Xp.ko 2024-03-23T23:52:00.712 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/phy/vitesse.ko 2024-03-23T23:52:00.712 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ppp 2024-03-23T23:52:00.712 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ppp/bsd_comp.ko 2024-03-23T23:52:00.712 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ppp/ppp_async.ko 2024-03-23T23:52:00.712 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ppp/ppp_deflate.ko 2024-03-23T23:52:00.712 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ppp/ppp_generic.ko 2024-03-23T23:52:00.712 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ppp/ppp_mppe.ko 2024-03-23T23:52:00.712 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ppp/ppp_synctty.ko 2024-03-23T23:52:00.712 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ppp/pppoe.ko 2024-03-23T23:52:00.712 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ppp/pppox.ko 2024-03-23T23:52:00.712 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/ppp/pptp.ko 2024-03-23T23:52:00.712 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/slip 2024-03-23T23:52:00.712 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/slip/slhc.ko 2024-03-23T23:52:00.713 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/slip/slip.ko 2024-03-23T23:52:00.713 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/sungem_phy.ko 2024-03-23T23:52:00.713 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/tap.ko 2024-03-23T23:52:00.713 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/team 2024-03-23T23:52:00.713 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/team/team.ko 2024-03-23T23:52:00.713 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/team/team_mode_activebackup.ko 2024-03-23T23:52:00.713 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/team/team_mode_broadcast.ko 2024-03-23T23:52:00.713 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/team/team_mode_loadbalance.ko 2024-03-23T23:52:00.713 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/team/team_mode_random.ko 2024-03-23T23:52:00.713 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/team/team_mode_roundrobin.ko 2024-03-23T23:52:00.713 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/tun.ko 2024-03-23T23:52:00.713 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/veth.ko 2024-03-23T23:52:00.713 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/virtio_net.ko 2024-03-23T23:52:00.713 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/vmxnet3 2024-03-23T23:52:00.714 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/vmxnet3/vmxnet3.ko 2024-03-23T23:52:00.714 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/vxlan 2024-03-23T23:52:00.714 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/vxlan/vxlan.ko 2024-03-23T23:52:00.714 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/xen-netback 2024-03-23T23:52:00.714 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/xen-netback/xen-netback.ko 2024-03-23T23:52:00.714 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/net/xen-netfront.ko 2024-03-23T23:52:00.714 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ntb 2024-03-23T23:52:00.714 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ntb/ntb.ko 2024-03-23T23:52:00.714 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/nvme 2024-03-23T23:52:00.714 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/nvme/host 2024-03-23T23:52:00.714 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/nvme/host/nvme-core.ko 2024-03-23T23:52:00.714 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/nvme/host/nvme.ko 2024-03-23T23:52:00.714 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/parport 2024-03-23T23:52:00.714 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/parport/parport.ko 2024-03-23T23:52:00.715 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/parport/parport_cs.ko 2024-03-23T23:52:00.715 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/parport/parport_pc.ko 2024-03-23T23:52:00.715 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/parport/parport_serial.ko 2024-03-23T23:52:00.715 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/pci 2024-03-23T23:52:00.715 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/pci/hotplug 2024-03-23T23:52:00.715 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/pci/hotplug/acpiphp_ibm.ko 2024-03-23T23:52:00.715 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/pci/pcie 2024-03-23T23:52:00.715 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/pci/pcie/aer_inject.ko 2024-03-23T23:52:00.715 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/pci/xen-pcifront.ko 2024-03-23T23:52:00.715 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/pcmcia 2024-03-23T23:52:00.715 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/pcmcia/i82092.ko 2024-03-23T23:52:00.715 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/pcmcia/pd6729.ko 2024-03-23T23:52:00.715 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/pcmcia/yenta_socket.ko 2024-03-23T23:52:00.715 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/pinctrl 2024-03-23T23:52:00.715 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/pinctrl/intel 2024-03-23T23:52:00.716 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/pinctrl/intel/pinctrl-cherryview.ko 2024-03-23T23:52:00.716 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/platform 2024-03-23T23:52:00.716 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/platform/x86 2024-03-23T23:52:00.716 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/platform/x86/acer-wmi.ko 2024-03-23T23:52:00.716 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/platform/x86/acerhdf.ko 2024-03-23T23:52:00.716 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/platform/x86/apple-gmux.ko 2024-03-23T23:52:00.716 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/platform/x86/asus-laptop.ko 2024-03-23T23:52:00.716 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/platform/x86/asus-nb-wmi.ko 2024-03-23T23:52:00.716 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/platform/x86/asus-wmi.ko 2024-03-23T23:52:00.716 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/platform/x86/classmate-laptop.ko 2024-03-23T23:52:00.716 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/platform/x86/eeepc-laptop.ko 2024-03-23T23:52:00.716 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/platform/x86/eeepc-wmi.ko 2024-03-23T23:52:00.716 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/platform/x86/fujitsu-laptop.ko 2024-03-23T23:52:00.716 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/platform/x86/fujitsu-tablet.ko 2024-03-23T23:52:00.717 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/platform/x86/hdaps.ko 2024-03-23T23:52:00.717 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/platform/x86/intel 2024-03-23T23:52:00.717 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/platform/x86/intel/intel-rst.ko 2024-03-23T23:52:00.717 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/platform/x86/intel_ips.ko 2024-03-23T23:52:00.717 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/platform/x86/msi-wmi.ko 2024-03-23T23:52:00.717 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/platform/x86/mxm-wmi.ko 2024-03-23T23:52:00.717 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/platform/x86/panasonic-laptop.ko 2024-03-23T23:52:00.717 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/platform/x86/samsung-laptop.ko 2024-03-23T23:52:00.717 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/platform/x86/samsung-q10.ko 2024-03-23T23:52:00.717 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/platform/x86/topstar-laptop.ko 2024-03-23T23:52:00.717 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/platform/x86/toshiba_bluetooth.ko 2024-03-23T23:52:00.717 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/platform/x86/toshiba_haps.ko 2024-03-23T23:52:00.717 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/platform/x86/wmi-bmof.ko 2024-03-23T23:52:00.717 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/platform/x86/wmi.ko 2024-03-23T23:52:00.717 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/powercap 2024-03-23T23:52:00.718 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/powercap/intel_rapl_common.ko 2024-03-23T23:52:00.718 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/powercap/intel_rapl_msr.ko 2024-03-23T23:52:00.718 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/pps 2024-03-23T23:52:00.718 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/pps/clients 2024-03-23T23:52:00.718 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/pps/clients/pps-gpio.ko 2024-03-23T23:52:00.718 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/pps/clients/pps-ldisc.ko 2024-03-23T23:52:00.718 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/pps/clients/pps_parport.ko 2024-03-23T23:52:00.718 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/pps/pps_core.ko 2024-03-23T23:52:00.718 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ptp 2024-03-23T23:52:00.718 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ptp/ptp.ko 2024-03-23T23:52:00.718 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ptp/ptp_kvm.ko 2024-03-23T23:52:00.718 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/rtc 2024-03-23T23:52:00.718 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/rtc/rtc-bq32k.ko 2024-03-23T23:52:00.718 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/rtc/rtc-ds1286.ko 2024-03-23T23:52:00.719 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/rtc/rtc-ds1307.ko 2024-03-23T23:52:00.719 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/rtc/rtc-ds1374.ko 2024-03-23T23:52:00.719 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/rtc/rtc-ds1511.ko 2024-03-23T23:52:00.719 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/rtc/rtc-ds1553.ko 2024-03-23T23:52:00.719 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/rtc/rtc-ds1672.ko 2024-03-23T23:52:00.719 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/rtc/rtc-ds1742.ko 2024-03-23T23:52:00.719 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/rtc/rtc-ds2404.ko 2024-03-23T23:52:00.719 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/rtc/rtc-ds3232.ko 2024-03-23T23:52:00.719 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/rtc/rtc-em3027.ko 2024-03-23T23:52:00.719 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/rtc/rtc-fm3130.ko 2024-03-23T23:52:00.719 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/rtc/rtc-isl12022.ko 2024-03-23T23:52:00.719 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/rtc/rtc-isl1208.ko 2024-03-23T23:52:00.719 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/rtc/rtc-m41t80.ko 2024-03-23T23:52:00.720 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/rtc/rtc-m48t35.ko 2024-03-23T23:52:00.720 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/rtc/rtc-m48t59.ko 2024-03-23T23:52:00.720 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/rtc/rtc-max6900.ko 2024-03-23T23:52:00.720 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/rtc/rtc-msm6242.ko 2024-03-23T23:52:00.720 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/rtc/rtc-pcf2127.ko 2024-03-23T23:52:00.720 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/rtc/rtc-pcf85063.ko 2024-03-23T23:52:00.720 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/rtc/rtc-pcf8523.ko 2024-03-23T23:52:00.720 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/rtc/rtc-pcf8563.ko 2024-03-23T23:52:00.720 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/rtc/rtc-pcf8583.ko 2024-03-23T23:52:00.720 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/rtc/rtc-rp5c01.ko 2024-03-23T23:52:00.720 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/rtc/rtc-rs5c372.ko 2024-03-23T23:52:00.720 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/rtc/rtc-rv3029c2.ko 2024-03-23T23:52:00.720 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/rtc/rtc-rx8025.ko 2024-03-23T23:52:00.720 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/rtc/rtc-rx8581.ko 2024-03-23T23:52:00.720 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/rtc/rtc-stk17ta8.ko 2024-03-23T23:52:00.721 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/rtc/rtc-x1205.ko 2024-03-23T23:52:00.721 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi 2024-03-23T23:52:00.721 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/3w-9xxx.ko 2024-03-23T23:52:00.721 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/3w-sas.ko 2024-03-23T23:52:00.721 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/3w-xxxx.ko 2024-03-23T23:52:00.721 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/BusLogic.ko 2024-03-23T23:52:00.721 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/a100u2w.ko 2024-03-23T23:52:00.721 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/aacraid 2024-03-23T23:52:00.721 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/aacraid/aacraid.ko 2024-03-23T23:52:00.721 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/advansys.ko 2024-03-23T23:52:00.721 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/aic7xxx 2024-03-23T23:52:00.721 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/aic7xxx/aic79xx.ko 2024-03-23T23:52:00.721 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/aic7xxx/aic7xxx.ko 2024-03-23T23:52:00.721 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/am53c974.ko 2024-03-23T23:52:00.722 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/arcmsr 2024-03-23T23:52:00.722 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/arcmsr/arcmsr.ko 2024-03-23T23:52:00.722 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/atp870u.ko 2024-03-23T23:52:00.722 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/be2iscsi 2024-03-23T23:52:00.722 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/be2iscsi/be2iscsi.ko 2024-03-23T23:52:00.722 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/bfa 2024-03-23T23:52:00.722 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/bfa/bfa.ko 2024-03-23T23:52:00.722 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/bnx2fc 2024-03-23T23:52:00.722 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/bnx2fc/bnx2fc.ko 2024-03-23T23:52:00.722 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/bnx2i 2024-03-23T23:52:00.722 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/bnx2i/bnx2i.ko 2024-03-23T23:52:00.722 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/ch.ko 2024-03-23T23:52:00.722 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/csiostor 2024-03-23T23:52:00.722 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/csiostor/csiostor.ko 2024-03-23T23:52:00.723 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/cxgbi 2024-03-23T23:52:00.723 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/cxgbi/cxgb3i 2024-03-23T23:52:00.723 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/cxgbi/cxgb3i/cxgb3i.ko 2024-03-23T23:52:00.723 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/cxgbi/cxgb4i 2024-03-23T23:52:00.723 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/cxgbi/cxgb4i/cxgb4i.ko 2024-03-23T23:52:00.723 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/cxgbi/libcxgbi.ko 2024-03-23T23:52:00.723 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/dc395x.ko 2024-03-23T23:52:00.723 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/device_handler 2024-03-23T23:52:00.723 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/device_handler/scsi_dh_alua.ko 2024-03-23T23:52:00.723 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/device_handler/scsi_dh_emc.ko 2024-03-23T23:52:00.723 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/device_handler/scsi_dh_hp_sw.ko 2024-03-23T23:52:00.723 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/device_handler/scsi_dh_rdac.ko 2024-03-23T23:52:00.723 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/dmx3191d.ko 2024-03-23T23:52:00.723 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/esas2r 2024-03-23T23:52:00.723 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/esas2r/esas2r.ko 2024-03-23T23:52:00.724 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/esp_scsi.ko 2024-03-23T23:52:00.724 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/fcoe 2024-03-23T23:52:00.724 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/fcoe/fcoe.ko 2024-03-23T23:52:00.724 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/fcoe/libfcoe.ko 2024-03-23T23:52:00.724 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/fnic 2024-03-23T23:52:00.724 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/fnic/fnic.ko 2024-03-23T23:52:00.724 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/hpsa.ko 2024-03-23T23:52:00.724 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/hptiop.ko 2024-03-23T23:52:00.724 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/hv_storvsc.ko 2024-03-23T23:52:00.724 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/initio.ko 2024-03-23T23:52:00.724 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/ipr.ko 2024-03-23T23:52:00.724 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/ips.ko 2024-03-23T23:52:00.724 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/isci 2024-03-23T23:52:00.724 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/isci/isci.ko 2024-03-23T23:52:00.725 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/iscsi_boot_sysfs.ko 2024-03-23T23:52:00.725 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/iscsi_tcp.ko 2024-03-23T23:52:00.725 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/libfc 2024-03-23T23:52:00.725 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/libfc/libfc.ko 2024-03-23T23:52:00.725 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/libiscsi.ko 2024-03-23T23:52:00.725 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/libiscsi_tcp.ko 2024-03-23T23:52:00.725 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/libsas 2024-03-23T23:52:00.725 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/libsas/libsas.ko 2024-03-23T23:52:00.725 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/lpfc 2024-03-23T23:52:00.725 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/lpfc/lpfc.ko 2024-03-23T23:52:00.725 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/megaraid 2024-03-23T23:52:00.725 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/megaraid.ko 2024-03-23T23:52:00.725 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/megaraid/megaraid_mbox.ko 2024-03-23T23:52:00.725 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/megaraid/megaraid_mm.ko 2024-03-23T23:52:00.725 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/megaraid/megaraid_sas.ko 2024-03-23T23:52:00.726 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/mpt3sas 2024-03-23T23:52:00.726 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/mpt3sas/mpt3sas.ko 2024-03-23T23:52:00.726 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/mvsas 2024-03-23T23:52:00.726 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/mvsas/mvsas.ko 2024-03-23T23:52:00.726 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/mvumi.ko 2024-03-23T23:52:00.726 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/pm8001 2024-03-23T23:52:00.726 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/pm8001/pm80xx.ko 2024-03-23T23:52:00.726 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/pmcraid.ko 2024-03-23T23:52:00.726 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/qla1280.ko 2024-03-23T23:52:00.726 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/qla2xxx 2024-03-23T23:52:00.726 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/qla2xxx/qla2xxx.ko 2024-03-23T23:52:00.726 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/qla2xxx/tcm_qla2xxx.ko 2024-03-23T23:52:00.726 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/qla4xxx 2024-03-23T23:52:00.726 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/qla4xxx/qla4xxx.ko 2024-03-23T23:52:00.727 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/raid_class.ko 2024-03-23T23:52:00.727 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/scsi_debug.ko 2024-03-23T23:52:00.727 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/scsi_transport_fc.ko 2024-03-23T23:52:00.727 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/scsi_transport_iscsi.ko 2024-03-23T23:52:00.727 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/scsi_transport_sas.ko 2024-03-23T23:52:00.727 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/scsi_transport_spi.ko 2024-03-23T23:52:00.727 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/scsi_transport_srp.ko 2024-03-23T23:52:00.727 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/ses.ko 2024-03-23T23:52:00.727 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/st.ko 2024-03-23T23:52:00.727 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/stex.ko 2024-03-23T23:52:00.727 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/sym53c8xx_2 2024-03-23T23:52:00.727 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/sym53c8xx_2/sym53c8xx.ko 2024-03-23T23:52:00.727 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/virtio_scsi.ko 2024-03-23T23:52:00.727 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/vmw_pvscsi.ko 2024-03-23T23:52:00.728 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/scsi/wd719x.ko 2024-03-23T23:52:00.728 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ssb 2024-03-23T23:52:00.728 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ssb/ssb.ko 2024-03-23T23:52:00.728 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/target 2024-03-23T23:52:00.728 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/target/iscsi 2024-03-23T23:52:00.728 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/target/iscsi/iscsi_target_mod.ko 2024-03-23T23:52:00.728 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/target/loopback 2024-03-23T23:52:00.728 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/target/loopback/tcm_loop.ko 2024-03-23T23:52:00.728 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/target/target_core_file.ko 2024-03-23T23:52:00.728 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/target/target_core_iblock.ko 2024-03-23T23:52:00.728 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/target/target_core_mod.ko 2024-03-23T23:52:00.728 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/target/target_core_pscsi.ko 2024-03-23T23:52:00.728 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/target/target_core_user.ko 2024-03-23T23:52:00.728 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/target/tcm_fc 2024-03-23T23:52:00.728 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/target/tcm_fc/tcm_fc.ko 2024-03-23T23:52:00.729 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/thermal 2024-03-23T23:52:00.729 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/thermal/intel 2024-03-23T23:52:00.729 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/thermal/intel/int340x_thermal 2024-03-23T23:52:00.729 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/thermal/intel/int340x_thermal/acpi_thermal_rel.ko 2024-03-23T23:52:00.729 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/thermal/intel/int340x_thermal/int3400_thermal.ko 2024-03-23T23:52:00.729 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/thermal/intel/int340x_thermal/int3401_thermal.ko 2024-03-23T23:52:00.729 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/thermal/intel/int340x_thermal/int3402_thermal.ko 2024-03-23T23:52:00.729 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/thermal/intel/int340x_thermal/int3403_thermal.ko 2024-03-23T23:52:00.729 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/thermal/intel/int340x_thermal/int340x_thermal_zone.ko 2024-03-23T23:52:00.729 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_device.ko 2024-03-23T23:52:00.729 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_device_pci.ko 2024-03-23T23:52:00.729 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_device_pci_legacy.ko 2024-03-23T23:52:00.729 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_mbox.ko 2024-03-23T23:52:00.729 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_power_floor.ko 2024-03-23T23:52:00.730 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_rapl.ko 2024-03-23T23:52:00.730 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_rfim.ko 2024-03-23T23:52:00.730 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_wt_hint.ko 2024-03-23T23:52:00.730 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/thermal/intel/int340x_thermal/processor_thermal_wt_req.ko 2024-03-23T23:52:00.730 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/thermal/intel/intel_soc_dts_iosf.ko 2024-03-23T23:52:00.730 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/thermal/intel/intel_soc_dts_thermal.ko 2024-03-23T23:52:00.730 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/thermal/intel/x86_pkg_temp_thermal.ko 2024-03-23T23:52:00.730 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/tty 2024-03-23T23:52:00.730 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/tty/ipwireless 2024-03-23T23:52:00.730 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/tty/ipwireless/ipwireless.ko 2024-03-23T23:52:00.730 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/tty/n_gsm.ko 2024-03-23T23:52:00.730 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/tty/n_hdlc.ko 2024-03-23T23:52:00.730 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/tty/nozomi.ko 2024-03-23T23:52:00.731 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/tty/serial 2024-03-23T23:52:00.731 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/tty/serial/8250 2024-03-23T23:52:00.731 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/tty/serial/8250/serial_cs.ko 2024-03-23T23:52:00.731 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/tty/serial/arc_uart.ko 2024-03-23T23:52:00.731 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/tty/serial/jsm 2024-03-23T23:52:00.731 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/tty/serial/jsm/jsm.ko 2024-03-23T23:52:00.731 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ufs 2024-03-23T23:52:00.731 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ufs/core 2024-03-23T23:52:00.731 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ufs/core/ufshcd-core.ko 2024-03-23T23:52:00.731 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ufs/host 2024-03-23T23:52:00.731 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/ufs/host/ufshcd-pci.ko 2024-03-23T23:52:00.731 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/uio 2024-03-23T23:52:00.731 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/uio/uio.ko 2024-03-23T23:52:00.731 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/uio/uio_aec.ko 2024-03-23T23:52:00.731 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/uio/uio_cif.ko 2024-03-23T23:52:00.732 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/uio/uio_pci_generic.ko 2024-03-23T23:52:00.732 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/uio/uio_sercos3.ko 2024-03-23T23:52:00.732 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb 2024-03-23T23:52:00.732 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/class 2024-03-23T23:52:00.732 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/class/cdc-acm.ko 2024-03-23T23:52:00.732 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/class/cdc-wdm.ko 2024-03-23T23:52:00.732 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/class/usblp.ko 2024-03-23T23:52:00.732 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/class/usbtmc.ko 2024-03-23T23:52:00.732 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/host 2024-03-23T23:52:00.732 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/host/sl811-hcd.ko 2024-03-23T23:52:00.732 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/image 2024-03-23T23:52:00.732 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/image/mdc800.ko 2024-03-23T23:52:00.732 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/image/microtek.ko 2024-03-23T23:52:00.732 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/misc 2024-03-23T23:52:00.733 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/misc/adutux.ko 2024-03-23T23:52:00.733 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/misc/appledisplay.ko 2024-03-23T23:52:00.733 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/misc/emi26.ko 2024-03-23T23:52:00.733 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/misc/emi62.ko 2024-03-23T23:52:00.733 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/misc/ezusb.ko 2024-03-23T23:52:00.733 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/misc/idmouse.ko 2024-03-23T23:52:00.733 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/misc/iowarrior.ko 2024-03-23T23:52:00.733 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/misc/isight_firmware.ko 2024-03-23T23:52:00.733 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/misc/ldusb.ko 2024-03-23T23:52:00.733 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/misc/legousbtower.ko 2024-03-23T23:52:00.733 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/misc/sisusbvga 2024-03-23T23:52:00.733 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/misc/sisusbvga/sisusbvga.ko 2024-03-23T23:52:00.733 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/misc/trancevibrator.ko 2024-03-23T23:52:00.733 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/misc/usb3503.ko 2024-03-23T23:52:00.734 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/misc/usblcd.ko 2024-03-23T23:52:00.734 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/misc/usbsevseg.ko 2024-03-23T23:52:00.734 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/misc/uss720.ko 2024-03-23T23:52:00.734 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/misc/yurex.ko 2024-03-23T23:52:00.734 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/phy 2024-03-23T23:52:00.734 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/phy/phy-generic.ko 2024-03-23T23:52:00.734 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/serial 2024-03-23T23:52:00.734 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/serial/aircable.ko 2024-03-23T23:52:00.734 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/serial/ark3116.ko 2024-03-23T23:52:00.734 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/serial/belkin_sa.ko 2024-03-23T23:52:00.734 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/serial/ch341.ko 2024-03-23T23:52:00.734 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/serial/cp210x.ko 2024-03-23T23:52:00.734 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/serial/cyberjack.ko 2024-03-23T23:52:00.735 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/serial/cypress_m8.ko 2024-03-23T23:52:00.735 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/serial/digi_acceleport.ko 2024-03-23T23:52:00.735 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/serial/empeg.ko 2024-03-23T23:52:00.735 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/serial/ftdi_sio.ko 2024-03-23T23:52:00.735 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/serial/garmin_gps.ko 2024-03-23T23:52:00.735 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/serial/io_edgeport.ko 2024-03-23T23:52:00.735 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/serial/io_ti.ko 2024-03-23T23:52:00.735 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/serial/ipaq.ko 2024-03-23T23:52:00.735 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/serial/ipw.ko 2024-03-23T23:52:00.735 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/serial/ir-usb.ko 2024-03-23T23:52:00.735 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/serial/iuu_phoenix.ko 2024-03-23T23:52:00.735 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/serial/keyspan.ko 2024-03-23T23:52:00.735 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/serial/keyspan_pda.ko 2024-03-23T23:52:00.735 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/serial/kl5kusb105.ko 2024-03-23T23:52:00.736 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/serial/kobil_sct.ko 2024-03-23T23:52:00.736 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/serial/mct_u232.ko 2024-03-23T23:52:00.736 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/serial/mos7720.ko 2024-03-23T23:52:00.736 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/serial/mos7840.ko 2024-03-23T23:52:00.736 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/serial/navman.ko 2024-03-23T23:52:00.736 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/serial/omninet.ko 2024-03-23T23:52:00.736 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/serial/opticon.ko 2024-03-23T23:52:00.736 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/serial/option.ko 2024-03-23T23:52:00.736 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/serial/oti6858.ko 2024-03-23T23:52:00.736 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/serial/pl2303.ko 2024-03-23T23:52:00.736 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/serial/qcaux.ko 2024-03-23T23:52:00.736 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/serial/qcserial.ko 2024-03-23T23:52:00.736 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/serial/quatech2.ko 2024-03-23T23:52:00.737 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/serial/safe_serial.ko 2024-03-23T23:52:00.737 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/serial/sierra.ko 2024-03-23T23:52:00.737 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/serial/spcp8x5.ko 2024-03-23T23:52:00.737 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/serial/ssu100.ko 2024-03-23T23:52:00.737 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/serial/symbolserial.ko 2024-03-23T23:52:00.737 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/serial/ti_usb_3410_5052.ko 2024-03-23T23:52:00.737 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/serial/usb-serial-simple.ko 2024-03-23T23:52:00.737 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/serial/usb_debug.ko 2024-03-23T23:52:00.737 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/serial/usb_wwan.ko 2024-03-23T23:52:00.737 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/serial/visor.ko 2024-03-23T23:52:00.737 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/serial/whiteheat.ko 2024-03-23T23:52:00.737 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/serial/xsens_mt.ko 2024-03-23T23:52:00.737 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/storage 2024-03-23T23:52:00.737 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/storage/uas.ko 2024-03-23T23:52:00.738 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/storage/ums-alauda.ko 2024-03-23T23:52:00.738 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/storage/ums-cypress.ko 2024-03-23T23:52:00.738 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/storage/ums-datafab.ko 2024-03-23T23:52:00.738 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/storage/ums-eneub6250.ko 2024-03-23T23:52:00.738 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/storage/ums-freecom.ko 2024-03-23T23:52:00.738 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/storage/ums-isd200.ko 2024-03-23T23:52:00.738 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/storage/ums-jumpshot.ko 2024-03-23T23:52:00.738 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/storage/ums-karma.ko 2024-03-23T23:52:00.738 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/storage/ums-onetouch.ko 2024-03-23T23:52:00.738 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/storage/ums-realtek.ko 2024-03-23T23:52:00.738 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/storage/ums-sddr09.ko 2024-03-23T23:52:00.738 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/storage/ums-sddr55.ko 2024-03-23T23:52:00.738 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/storage/ums-usbat.ko 2024-03-23T23:52:00.739 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/storage/usb-storage.ko 2024-03-23T23:52:00.739 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/usbip 2024-03-23T23:52:00.739 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/usbip/usbip-core.ko 2024-03-23T23:52:00.739 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/usbip/usbip-host.ko 2024-03-23T23:52:00.739 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/usb/usbip/vhci-hcd.ko 2024-03-23T23:52:00.739 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/vfio 2024-03-23T23:52:00.739 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/vfio/pci 2024-03-23T23:52:00.739 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/vfio/pci/vfio-pci-core.ko 2024-03-23T23:52:00.739 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/vfio/pci/vfio-pci.ko 2024-03-23T23:52:00.739 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/vfio/vfio.ko 2024-03-23T23:52:00.739 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/vfio/vfio_iommu_type1.ko 2024-03-23T23:52:00.739 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/vhost 2024-03-23T23:52:00.739 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/vhost/vhost.ko 2024-03-23T23:52:00.739 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/vhost/vhost_iotlb.ko 2024-03-23T23:52:00.740 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/vhost/vhost_net.ko 2024-03-23T23:52:00.740 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/vhost/vhost_scsi.ko 2024-03-23T23:52:00.740 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/video 2024-03-23T23:52:00.740 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/video/backlight 2024-03-23T23:52:00.740 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/video/backlight/apple_bl.ko 2024-03-23T23:52:00.740 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/video/backlight/lcd.ko 2024-03-23T23:52:00.740 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/video/backlight/platform_lcd.ko 2024-03-23T23:52:00.740 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/virtio 2024-03-23T23:52:00.740 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/virtio/virtio.ko 2024-03-23T23:52:00.740 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/virtio/virtio_balloon.ko 2024-03-23T23:52:00.740 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/virtio/virtio_mmio.ko 2024-03-23T23:52:00.740 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/virtio/virtio_pci.ko 2024-03-23T23:52:00.740 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/virtio/virtio_pci_legacy_dev.ko 2024-03-23T23:52:00.740 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/virtio/virtio_pci_modern_dev.ko 2024-03-23T23:52:00.741 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/virtio/virtio_ring.ko 2024-03-23T23:52:00.741 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/w1 2024-03-23T23:52:00.741 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/w1/masters 2024-03-23T23:52:00.741 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/w1/masters/ds2482.ko 2024-03-23T23:52:00.741 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/w1/masters/ds2490.ko 2024-03-23T23:52:00.741 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/w1/slaves 2024-03-23T23:52:00.741 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/w1/slaves/w1_ds2406.ko 2024-03-23T23:52:00.741 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/w1/slaves/w1_ds2408.ko 2024-03-23T23:52:00.741 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/w1/slaves/w1_ds2413.ko 2024-03-23T23:52:00.741 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/w1/slaves/w1_ds2423.ko 2024-03-23T23:52:00.741 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/w1/slaves/w1_ds2431.ko 2024-03-23T23:52:00.741 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/w1/slaves/w1_ds2433.ko 2024-03-23T23:52:00.741 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/w1/slaves/w1_ds2780.ko 2024-03-23T23:52:00.741 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/w1/slaves/w1_ds2781.ko 2024-03-23T23:52:00.742 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/w1/slaves/w1_ds28e04.ko 2024-03-23T23:52:00.742 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/w1/slaves/w1_smem.ko 2024-03-23T23:52:00.742 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/w1/slaves/w1_therm.ko 2024-03-23T23:52:00.742 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/w1/wire.ko 2024-03-23T23:52:00.742 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/watchdog 2024-03-23T23:52:00.742 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/watchdog/alim1535_wdt.ko 2024-03-23T23:52:00.742 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/watchdog/alim7101_wdt.ko 2024-03-23T23:52:00.742 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/watchdog/f71808e_wdt.ko 2024-03-23T23:52:00.742 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/watchdog/hpwdt.ko 2024-03-23T23:52:00.742 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/watchdog/i6300esb.ko 2024-03-23T23:52:00.742 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/watchdog/iTCO_vendor_support.ko 2024-03-23T23:52:00.742 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/watchdog/iTCO_wdt.ko 2024-03-23T23:52:00.742 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/watchdog/ib700wdt.ko 2024-03-23T23:52:00.742 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/watchdog/ibmasr.ko 2024-03-23T23:52:00.743 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/watchdog/ie6xx_wdt.ko 2024-03-23T23:52:00.743 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/watchdog/it8712f_wdt.ko 2024-03-23T23:52:00.743 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/watchdog/it87_wdt.ko 2024-03-23T23:52:00.743 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/watchdog/machzwd.ko 2024-03-23T23:52:00.743 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/watchdog/nv_tco.ko 2024-03-23T23:52:00.743 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/watchdog/pcwd_pci.ko 2024-03-23T23:52:00.743 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/watchdog/pcwd_usb.ko 2024-03-23T23:52:00.743 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/watchdog/sbc_fitpc2_wdt.ko 2024-03-23T23:52:00.743 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/watchdog/sch311x_wdt.ko 2024-03-23T23:52:00.743 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/watchdog/softdog.ko 2024-03-23T23:52:00.743 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/watchdog/sp5100_tco.ko 2024-03-23T23:52:00.743 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/watchdog/via_wdt.ko 2024-03-23T23:52:00.743 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/watchdog/w83627hf_wdt.ko 2024-03-23T23:52:00.743 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/watchdog/w83877f_wdt.ko 2024-03-23T23:52:00.744 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/watchdog/w83977f_wdt.ko 2024-03-23T23:52:00.744 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/watchdog/wdt_pci.ko 2024-03-23T23:52:00.744 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/watchdog/xen_wdt.ko 2024-03-23T23:52:00.744 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/xen 2024-03-23T23:52:00.744 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/xen/xen-acpi-processor.ko 2024-03-23T23:52:00.744 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/xen/xen-evtchn.ko 2024-03-23T23:52:00.744 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/xen/xen-gntalloc.ko 2024-03-23T23:52:00.744 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/xen/xen-gntdev.ko 2024-03-23T23:52:00.744 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/xen/xen-pciback 2024-03-23T23:52:00.744 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/xen/xen-pciback/xen-pciback.ko 2024-03-23T23:52:00.744 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/xen/xen-privcmd.ko 2024-03-23T23:52:00.744 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/xen/xenfs 2024-03-23T23:52:00.744 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/drivers/xen/xenfs/xenfs.ko 2024-03-23T23:52:00.744 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs 2024-03-23T23:52:00.745 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/9p 2024-03-23T23:52:00.745 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/9p/9p.ko 2024-03-23T23:52:00.745 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/affs 2024-03-23T23:52:00.745 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/affs/affs.ko 2024-03-23T23:52:00.745 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/befs 2024-03-23T23:52:00.745 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/befs/befs.ko 2024-03-23T23:52:00.745 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/binfmt_misc.ko 2024-03-23T23:52:00.745 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/btrfs 2024-03-23T23:52:00.745 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/btrfs/btrfs.ko 2024-03-23T23:52:00.745 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/cachefiles 2024-03-23T23:52:00.745 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/cachefiles/cachefiles.ko 2024-03-23T23:52:00.745 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/ceph 2024-03-23T23:52:00.745 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/ceph/ceph.ko 2024-03-23T23:52:00.745 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/coda 2024-03-23T23:52:00.746 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/coda/coda.ko 2024-03-23T23:52:00.746 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/cramfs 2024-03-23T23:52:00.746 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/cramfs/cramfs.ko 2024-03-23T23:52:00.746 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/dlm 2024-03-23T23:52:00.746 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/dlm/dlm.ko 2024-03-23T23:52:00.746 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/ecryptfs 2024-03-23T23:52:00.746 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/ecryptfs/ecryptfs.ko 2024-03-23T23:52:00.746 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/f2fs 2024-03-23T23:52:00.746 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/f2fs/f2fs.ko 2024-03-23T23:52:00.746 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/fat 2024-03-23T23:52:00.746 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/fat/fat.ko 2024-03-23T23:52:00.746 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/fat/msdos.ko 2024-03-23T23:52:00.746 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/fat/vfat.ko 2024-03-23T23:52:00.746 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/fuse 2024-03-23T23:52:00.747 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/fuse/cuse.ko 2024-03-23T23:52:00.747 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/fuse/fuse.ko 2024-03-23T23:52:00.747 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/gfs2 2024-03-23T23:52:00.747 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/gfs2/gfs2.ko 2024-03-23T23:52:00.747 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/hfs 2024-03-23T23:52:00.747 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/hfs/hfs.ko 2024-03-23T23:52:00.747 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/hfsplus 2024-03-23T23:52:00.747 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/hfsplus/hfsplus.ko 2024-03-23T23:52:00.747 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/isofs 2024-03-23T23:52:00.747 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/isofs/isofs.ko 2024-03-23T23:52:00.747 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/jfs 2024-03-23T23:52:00.747 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/jfs/jfs.ko 2024-03-23T23:52:00.747 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/lockd 2024-03-23T23:52:00.748 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/lockd/lockd.ko 2024-03-23T23:52:00.748 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/minix 2024-03-23T23:52:00.748 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/minix/minix.ko 2024-03-23T23:52:00.748 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/netfs 2024-03-23T23:52:00.748 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/netfs/netfs.ko 2024-03-23T23:52:00.748 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/nfs 2024-03-23T23:52:00.748 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/nfs/blocklayout 2024-03-23T23:52:00.748 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/nfs/blocklayout/blocklayoutdriver.ko 2024-03-23T23:52:00.748 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/nfs/filelayout 2024-03-23T23:52:00.748 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/nfs/filelayout/nfs_layout_nfsv41_files.ko 2024-03-23T23:52:00.748 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/nfs/flexfilelayout 2024-03-23T23:52:00.748 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/nfs/flexfilelayout/nfs_layout_flexfiles.ko 2024-03-23T23:52:00.749 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/nfs/nfs.ko 2024-03-23T23:52:00.749 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/nfs/nfsv3.ko 2024-03-23T23:52:00.749 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/nfs/nfsv4.ko 2024-03-23T23:52:00.749 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/nfs_common 2024-03-23T23:52:00.749 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/nfs_common/grace.ko 2024-03-23T23:52:00.749 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/nfs_common/nfs_acl.ko 2024-03-23T23:52:00.749 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/nfsd 2024-03-23T23:52:00.749 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/nfsd/nfsd.ko 2024-03-23T23:52:00.749 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/nilfs2 2024-03-23T23:52:00.749 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/nilfs2/nilfs2.ko 2024-03-23T23:52:00.749 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/nls 2024-03-23T23:52:00.749 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/nls/mac-celtic.ko 2024-03-23T23:52:00.749 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/nls/mac-centeuro.ko 2024-03-23T23:52:00.749 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/nls/mac-croatian.ko 2024-03-23T23:52:00.750 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/nls/mac-cyrillic.ko 2024-03-23T23:52:00.750 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/nls/mac-gaelic.ko 2024-03-23T23:52:00.750 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/nls/mac-greek.ko 2024-03-23T23:52:00.750 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/nls/mac-iceland.ko 2024-03-23T23:52:00.750 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/nls/mac-inuit.ko 2024-03-23T23:52:00.750 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/nls/mac-roman.ko 2024-03-23T23:52:00.750 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/nls/mac-romanian.ko 2024-03-23T23:52:00.750 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/nls/mac-turkish.ko 2024-03-23T23:52:00.750 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/nls/nls_cp1250.ko 2024-03-23T23:52:00.750 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/nls/nls_cp1251.ko 2024-03-23T23:52:00.750 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/nls/nls_cp1255.ko 2024-03-23T23:52:00.750 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/nls/nls_cp737.ko 2024-03-23T23:52:00.750 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/nls/nls_cp775.ko 2024-03-23T23:52:00.750 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/nls/nls_cp850.ko 2024-03-23T23:52:00.751 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/nls/nls_cp852.ko 2024-03-23T23:52:00.751 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/nls/nls_cp855.ko 2024-03-23T23:52:00.751 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/nls/nls_cp857.ko 2024-03-23T23:52:00.751 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/nls/nls_cp860.ko 2024-03-23T23:52:00.751 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/nls/nls_cp861.ko 2024-03-23T23:52:00.751 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/nls/nls_cp862.ko 2024-03-23T23:52:00.751 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/nls/nls_cp863.ko 2024-03-23T23:52:00.751 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/nls/nls_cp864.ko 2024-03-23T23:52:00.751 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/nls/nls_cp865.ko 2024-03-23T23:52:00.751 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/nls/nls_cp866.ko 2024-03-23T23:52:00.751 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/nls/nls_cp869.ko 2024-03-23T23:52:00.751 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/nls/nls_cp874.ko 2024-03-23T23:52:00.751 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/nls/nls_cp932.ko 2024-03-23T23:52:00.751 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/nls/nls_cp936.ko 2024-03-23T23:52:00.751 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/nls/nls_cp949.ko 2024-03-23T23:52:00.752 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/nls/nls_cp950.ko 2024-03-23T23:52:00.752 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/nls/nls_euc-jp.ko 2024-03-23T23:52:00.752 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/nls/nls_iso8859-1.ko 2024-03-23T23:52:00.752 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/nls/nls_iso8859-13.ko 2024-03-23T23:52:00.752 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/nls/nls_iso8859-14.ko 2024-03-23T23:52:00.752 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/nls/nls_iso8859-15.ko 2024-03-23T23:52:00.752 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/nls/nls_iso8859-2.ko 2024-03-23T23:52:00.752 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/nls/nls_iso8859-3.ko 2024-03-23T23:52:00.752 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/nls/nls_iso8859-4.ko 2024-03-23T23:52:00.752 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/nls/nls_iso8859-5.ko 2024-03-23T23:52:00.752 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/nls/nls_iso8859-6.ko 2024-03-23T23:52:00.752 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/nls/nls_iso8859-7.ko 2024-03-23T23:52:00.752 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/nls/nls_iso8859-9.ko 2024-03-23T23:52:00.752 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/nls/nls_koi8-r.ko 2024-03-23T23:52:00.752 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/nls/nls_koi8-ru.ko 2024-03-23T23:52:00.753 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/nls/nls_koi8-u.ko 2024-03-23T23:52:00.753 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/nls/nls_ucs2_utils.ko 2024-03-23T23:52:00.753 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/nls/nls_utf8.ko 2024-03-23T23:52:00.753 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/ocfs2 2024-03-23T23:52:00.753 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/ocfs2/cluster 2024-03-23T23:52:00.753 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/ocfs2/cluster/ocfs2_nodemanager.ko 2024-03-23T23:52:00.753 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/ocfs2/dlm 2024-03-23T23:52:00.753 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/ocfs2/dlm/ocfs2_dlm.ko 2024-03-23T23:52:00.753 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/ocfs2/dlmfs 2024-03-23T23:52:00.753 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/ocfs2/dlmfs/ocfs2_dlmfs.ko 2024-03-23T23:52:00.753 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/ocfs2/ocfs2.ko 2024-03-23T23:52:00.753 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/ocfs2/ocfs2_stack_o2cb.ko 2024-03-23T23:52:00.753 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/ocfs2/ocfs2_stack_user.ko 2024-03-23T23:52:00.754 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/ocfs2/ocfs2_stackglue.ko 2024-03-23T23:52:00.754 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/overlayfs 2024-03-23T23:52:00.754 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/overlayfs/overlay.ko 2024-03-23T23:52:00.754 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/pstore 2024-03-23T23:52:00.754 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/pstore/ramoops.ko 2024-03-23T23:52:00.754 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/reiserfs 2024-03-23T23:52:00.754 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/reiserfs/reiserfs.ko 2024-03-23T23:52:00.754 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/romfs 2024-03-23T23:52:00.754 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/romfs/romfs.ko 2024-03-23T23:52:00.754 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/smb 2024-03-23T23:52:00.754 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/smb/client 2024-03-23T23:52:00.754 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/smb/client/cifs.ko 2024-03-23T23:52:00.755 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/smb/common 2024-03-23T23:52:00.755 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/smb/common/cifs_arc4.ko 2024-03-23T23:52:00.755 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/smb/common/cifs_md4.ko 2024-03-23T23:52:00.755 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/squashfs 2024-03-23T23:52:00.755 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/squashfs/squashfs.ko 2024-03-23T23:52:00.755 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/sysv 2024-03-23T23:52:00.755 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/sysv/sysv.ko 2024-03-23T23:52:00.755 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/udf 2024-03-23T23:52:00.755 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/udf/udf.ko 2024-03-23T23:52:00.755 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/ufs 2024-03-23T23:52:00.755 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/ufs/ufs.ko 2024-03-23T23:52:00.755 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/xfs 2024-03-23T23:52:00.755 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/fs/xfs/xfs.ko 2024-03-23T23:52:00.755 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/kernel 2024-03-23T23:52:00.756 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/kernel/rcu 2024-03-23T23:52:00.756 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/kernel/rcu/rcutorture.ko 2024-03-23T23:52:00.756 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/kernel/torture.ko 2024-03-23T23:52:00.756 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/kernel/trace 2024-03-23T23:52:00.756 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/kernel/trace/ring_buffer_benchmark.ko 2024-03-23T23:52:00.756 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/lib 2024-03-23T23:52:00.756 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/lib/asn1_encoder.ko 2024-03-23T23:52:00.756 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/lib/crc-itu-t.ko 2024-03-23T23:52:00.756 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/lib/crc8.ko 2024-03-23T23:52:00.756 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/lib/crypto 2024-03-23T23:52:00.756 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/lib/crypto/libarc4.ko 2024-03-23T23:52:00.756 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/lib/crypto/libdes.ko 2024-03-23T23:52:00.756 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/lib/libcrc32c.ko 2024-03-23T23:52:00.756 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/lib/lru_cache.ko 2024-03-23T23:52:00.756 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/lib/lz4 2024-03-23T23:52:00.757 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/lib/lz4/lz4_compress.ko 2024-03-23T23:52:00.757 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/lib/lz4/lz4hc_compress.ko 2024-03-23T23:52:00.757 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/lib/math 2024-03-23T23:52:00.757 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/lib/math/cordic.ko 2024-03-23T23:52:00.757 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/lib/raid6 2024-03-23T23:52:00.757 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/lib/raid6/raid6_pq.ko 2024-03-23T23:52:00.757 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/lib/reed_solomon 2024-03-23T23:52:00.757 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/lib/reed_solomon/reed_solomon.ko 2024-03-23T23:52:00.757 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/lib/ts_bm.ko 2024-03-23T23:52:00.757 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/lib/ts_fsm.ko 2024-03-23T23:52:00.757 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/lib/ts_kmp.ko 2024-03-23T23:52:00.757 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/lib/zstd 2024-03-23T23:52:00.757 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/lib/zstd/zstd_compress.ko 2024-03-23T23:52:00.757 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/mm 2024-03-23T23:52:00.758 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/mm/hwpoison-inject.ko 2024-03-23T23:52:00.758 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net 2024-03-23T23:52:00.758 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/6lowpan 2024-03-23T23:52:00.758 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/6lowpan/6lowpan.ko 2024-03-23T23:52:00.758 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/6lowpan/nhc_dest.ko 2024-03-23T23:52:00.758 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/6lowpan/nhc_fragment.ko 2024-03-23T23:52:00.758 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/6lowpan/nhc_hop.ko 2024-03-23T23:52:00.758 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/6lowpan/nhc_ipv6.ko 2024-03-23T23:52:00.758 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/6lowpan/nhc_mobility.ko 2024-03-23T23:52:00.758 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/6lowpan/nhc_routing.ko 2024-03-23T23:52:00.758 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/6lowpan/nhc_udp.ko 2024-03-23T23:52:00.758 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/802 2024-03-23T23:52:00.758 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/802/garp.ko 2024-03-23T23:52:00.758 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/802/mrp.ko 2024-03-23T23:52:00.759 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/802/p8022.ko 2024-03-23T23:52:00.759 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/802/psnap.ko 2024-03-23T23:52:00.759 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/802/stp.ko 2024-03-23T23:52:00.759 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/8021q 2024-03-23T23:52:00.759 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/8021q/8021q.ko 2024-03-23T23:52:00.759 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/9p 2024-03-23T23:52:00.759 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/9p/9pnet.ko 2024-03-23T23:52:00.759 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/9p/9pnet_fd.ko 2024-03-23T23:52:00.759 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/9p/9pnet_rdma.ko 2024-03-23T23:52:00.759 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/9p/9pnet_virtio.ko 2024-03-23T23:52:00.759 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/batman-adv 2024-03-23T23:52:00.759 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/batman-adv/batman-adv.ko 2024-03-23T23:52:00.759 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/bridge 2024-03-23T23:52:00.759 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/bridge/br_netfilter.ko 2024-03-23T23:52:00.760 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/bridge/bridge.ko 2024-03-23T23:52:00.760 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/bridge/netfilter 2024-03-23T23:52:00.760 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/bridge/netfilter/ebt_802_3.ko 2024-03-23T23:52:00.760 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/bridge/netfilter/ebt_among.ko 2024-03-23T23:52:00.760 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/bridge/netfilter/ebt_arp.ko 2024-03-23T23:52:00.760 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/bridge/netfilter/ebt_arpreply.ko 2024-03-23T23:52:00.760 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/bridge/netfilter/ebt_dnat.ko 2024-03-23T23:52:00.760 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/bridge/netfilter/ebt_ip.ko 2024-03-23T23:52:00.760 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/bridge/netfilter/ebt_ip6.ko 2024-03-23T23:52:00.760 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/bridge/netfilter/ebt_limit.ko 2024-03-23T23:52:00.760 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/bridge/netfilter/ebt_log.ko 2024-03-23T23:52:00.760 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/bridge/netfilter/ebt_mark.ko 2024-03-23T23:52:00.760 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/bridge/netfilter/ebt_mark_m.ko 2024-03-23T23:52:00.760 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/bridge/netfilter/ebt_nflog.ko 2024-03-23T23:52:00.760 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/bridge/netfilter/ebt_pkttype.ko 2024-03-23T23:52:00.761 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/bridge/netfilter/ebt_redirect.ko 2024-03-23T23:52:00.761 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/bridge/netfilter/ebt_snat.ko 2024-03-23T23:52:00.761 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/bridge/netfilter/ebt_stp.ko 2024-03-23T23:52:00.761 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/bridge/netfilter/ebt_vlan.ko 2024-03-23T23:52:00.761 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/bridge/netfilter/ebtable_broute.ko 2024-03-23T23:52:00.761 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/bridge/netfilter/ebtable_filter.ko 2024-03-23T23:52:00.761 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/bridge/netfilter/ebtable_nat.ko 2024-03-23T23:52:00.761 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/bridge/netfilter/ebtables.ko 2024-03-23T23:52:00.761 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/bridge/netfilter/nf_conntrack_bridge.ko 2024-03-23T23:52:00.761 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/bridge/netfilter/nft_meta_bridge.ko 2024-03-23T23:52:00.761 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/bridge/netfilter/nft_reject_bridge.ko 2024-03-23T23:52:00.761 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ceph 2024-03-23T23:52:00.761 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ceph/libceph.ko 2024-03-23T23:52:00.761 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/core 2024-03-23T23:52:00.761 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/core/failover.ko 2024-03-23T23:52:00.762 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/core/pktgen.ko 2024-03-23T23:52:00.762 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/dccp 2024-03-23T23:52:00.762 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/dccp/dccp.ko 2024-03-23T23:52:00.762 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/dccp/dccp_diag.ko 2024-03-23T23:52:00.762 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/dccp/dccp_ipv4.ko 2024-03-23T23:52:00.762 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/dccp/dccp_ipv6.ko 2024-03-23T23:52:00.762 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/dns_resolver 2024-03-23T23:52:00.762 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/dns_resolver/dns_resolver.ko 2024-03-23T23:52:00.762 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/dsa 2024-03-23T23:52:00.762 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/dsa/dsa_core.ko 2024-03-23T23:52:00.762 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/dsa/tag_brcm.ko 2024-03-23T23:52:00.762 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/dsa/tag_dsa.ko 2024-03-23T23:52:00.762 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/dsa/tag_none.ko 2024-03-23T23:52:00.762 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/dsa/tag_trailer.ko 2024-03-23T23:52:00.763 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ieee802154 2024-03-23T23:52:00.763 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ieee802154/6lowpan 2024-03-23T23:52:00.763 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ieee802154/6lowpan/ieee802154_6lowpan.ko 2024-03-23T23:52:00.763 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ieee802154/ieee802154.ko 2024-03-23T23:52:00.763 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ieee802154/ieee802154_socket.ko 2024-03-23T23:52:00.763 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv4 2024-03-23T23:52:00.763 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv4/ah4.ko 2024-03-23T23:52:00.763 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv4/esp4.ko 2024-03-23T23:52:00.763 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv4/fou.ko 2024-03-23T23:52:00.763 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv4/gre.ko 2024-03-23T23:52:00.763 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv4/inet_diag.ko 2024-03-23T23:52:00.763 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv4/ip_gre.ko 2024-03-23T23:52:00.763 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv4/ip_tunnel.ko 2024-03-23T23:52:00.763 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv4/ip_vti.ko 2024-03-23T23:52:00.763 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv4/ipcomp.ko 2024-03-23T23:52:00.764 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv4/ipip.ko 2024-03-23T23:52:00.764 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv4/netfilter 2024-03-23T23:52:00.764 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv4/netfilter/arp_tables.ko 2024-03-23T23:52:00.764 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv4/netfilter/arpt_mangle.ko 2024-03-23T23:52:00.764 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv4/netfilter/arptable_filter.ko 2024-03-23T23:52:00.764 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv4/netfilter/ipt_ECN.ko 2024-03-23T23:52:00.764 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv4/netfilter/ipt_SYNPROXY.ko 2024-03-23T23:52:00.764 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv4/netfilter/ipt_ah.ko 2024-03-23T23:52:00.764 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv4/netfilter/ipt_rpfilter.ko 2024-03-23T23:52:00.764 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv4/netfilter/iptable_mangle.ko 2024-03-23T23:52:00.764 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv4/netfilter/iptable_nat.ko 2024-03-23T23:52:00.764 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv4/netfilter/iptable_raw.ko 2024-03-23T23:52:00.764 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv4/netfilter/iptable_security.ko 2024-03-23T23:52:00.764 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv4/netfilter/nf_defrag_ipv4.ko 2024-03-23T23:52:00.765 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv4/netfilter/nf_dup_ipv4.ko 2024-03-23T23:52:00.765 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv4/netfilter/nf_nat_h323.ko 2024-03-23T23:52:00.765 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv4/netfilter/nf_nat_pptp.ko 2024-03-23T23:52:00.765 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv4/netfilter/nf_nat_snmp_basic.ko 2024-03-23T23:52:00.765 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv4/netfilter/nf_socket_ipv4.ko 2024-03-23T23:52:00.765 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv4/netfilter/nf_tproxy_ipv4.ko 2024-03-23T23:52:00.765 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv4/netfilter/nft_dup_ipv4.ko 2024-03-23T23:52:00.765 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv4/netfilter/nft_fib_ipv4.ko 2024-03-23T23:52:00.765 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv4/netfilter/nft_reject_ipv4.ko 2024-03-23T23:52:00.765 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv4/tcp_bic.ko 2024-03-23T23:52:00.765 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv4/tcp_dctcp.ko 2024-03-23T23:52:00.765 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv4/tcp_diag.ko 2024-03-23T23:52:00.765 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv4/tcp_highspeed.ko 2024-03-23T23:52:00.765 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv4/tcp_htcp.ko 2024-03-23T23:52:00.766 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv4/tcp_hybla.ko 2024-03-23T23:52:00.766 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv4/tcp_illinois.ko 2024-03-23T23:52:00.766 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv4/tcp_lp.ko 2024-03-23T23:52:00.766 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv4/tcp_scalable.ko 2024-03-23T23:52:00.766 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv4/tcp_vegas.ko 2024-03-23T23:52:00.766 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv4/tcp_veno.ko 2024-03-23T23:52:00.766 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv4/tcp_westwood.ko 2024-03-23T23:52:00.766 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv4/tcp_yeah.ko 2024-03-23T23:52:00.766 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv4/tunnel4.ko 2024-03-23T23:52:00.766 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv4/udp_diag.ko 2024-03-23T23:52:00.766 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv4/udp_tunnel.ko 2024-03-23T23:52:00.766 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv4/xfrm4_tunnel.ko 2024-03-23T23:52:00.766 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv6 2024-03-23T23:52:00.766 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv6/ah6.ko 2024-03-23T23:52:00.767 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv6/esp6.ko 2024-03-23T23:52:00.767 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv6/fou6.ko 2024-03-23T23:52:00.767 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv6/ip6_tunnel.ko 2024-03-23T23:52:00.767 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv6/ip6_udp_tunnel.ko 2024-03-23T23:52:00.767 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv6/ip6_vti.ko 2024-03-23T23:52:00.767 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv6/ipcomp6.ko 2024-03-23T23:52:00.767 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv6/netfilter 2024-03-23T23:52:00.767 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv6/netfilter/ip6_tables.ko 2024-03-23T23:52:00.767 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv6/netfilter/ip6t_REJECT.ko 2024-03-23T23:52:00.767 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv6/netfilter/ip6t_SYNPROXY.ko 2024-03-23T23:52:00.767 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv6/netfilter/ip6t_ah.ko 2024-03-23T23:52:00.767 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv6/netfilter/ip6t_eui64.ko 2024-03-23T23:52:00.767 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv6/netfilter/ip6t_frag.ko 2024-03-23T23:52:00.767 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv6/netfilter/ip6t_hbh.ko 2024-03-23T23:52:00.768 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv6/netfilter/ip6t_ipv6header.ko 2024-03-23T23:52:00.768 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv6/netfilter/ip6t_mh.ko 2024-03-23T23:52:00.768 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv6/netfilter/ip6t_rpfilter.ko 2024-03-23T23:52:00.768 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv6/netfilter/ip6t_rt.ko 2024-03-23T23:52:00.768 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv6/netfilter/ip6table_filter.ko 2024-03-23T23:52:00.768 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv6/netfilter/ip6table_mangle.ko 2024-03-23T23:52:00.768 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv6/netfilter/ip6table_nat.ko 2024-03-23T23:52:00.768 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv6/netfilter/ip6table_raw.ko 2024-03-23T23:52:00.768 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv6/netfilter/ip6table_security.ko 2024-03-23T23:52:00.768 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv6/netfilter/nf_defrag_ipv6.ko 2024-03-23T23:52:00.768 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv6/netfilter/nf_dup_ipv6.ko 2024-03-23T23:52:00.768 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv6/netfilter/nf_reject_ipv6.ko 2024-03-23T23:52:00.768 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv6/netfilter/nf_socket_ipv6.ko 2024-03-23T23:52:00.768 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv6/netfilter/nf_tproxy_ipv6.ko 2024-03-23T23:52:00.768 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv6/netfilter/nft_dup_ipv6.ko 2024-03-23T23:52:00.769 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv6/netfilter/nft_fib_ipv6.ko 2024-03-23T23:52:00.769 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv6/netfilter/nft_reject_ipv6.ko 2024-03-23T23:52:00.769 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv6/sit.ko 2024-03-23T23:52:00.769 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv6/tunnel6.ko 2024-03-23T23:52:00.769 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/ipv6/xfrm6_tunnel.ko 2024-03-23T23:52:00.769 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/key 2024-03-23T23:52:00.769 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/key/af_key.ko 2024-03-23T23:52:00.769 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/l2tp 2024-03-23T23:52:00.769 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/l2tp/l2tp_core.ko 2024-03-23T23:52:00.769 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/l2tp/l2tp_debugfs.ko 2024-03-23T23:52:00.769 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/l2tp/l2tp_eth.ko 2024-03-23T23:52:00.769 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/l2tp/l2tp_ip.ko 2024-03-23T23:52:00.769 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/l2tp/l2tp_ip6.ko 2024-03-23T23:52:00.770 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/l2tp/l2tp_netlink.ko 2024-03-23T23:52:00.770 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/l2tp/l2tp_ppp.ko 2024-03-23T23:52:00.770 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/llc 2024-03-23T23:52:00.770 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/llc/llc.ko 2024-03-23T23:52:00.770 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/mac802154 2024-03-23T23:52:00.770 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/mac802154/mac802154.ko 2024-03-23T23:52:00.770 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/mpls 2024-03-23T23:52:00.770 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/mpls/mpls_gso.ko 2024-03-23T23:52:00.770 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter 2024-03-23T23:52:00.770 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/ipset 2024-03-23T23:52:00.770 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/ipset/ip_set.ko 2024-03-23T23:52:00.770 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/ipset/ip_set_bitmap_ip.ko 2024-03-23T23:52:00.770 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/ipset/ip_set_bitmap_ipmac.ko 2024-03-23T23:52:00.770 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/ipset/ip_set_bitmap_port.ko 2024-03-23T23:52:00.771 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/ipset/ip_set_hash_ip.ko 2024-03-23T23:52:00.771 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/ipset/ip_set_hash_ipmark.ko 2024-03-23T23:52:00.771 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/ipset/ip_set_hash_ipport.ko 2024-03-23T23:52:00.771 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/ipset/ip_set_hash_ipportip.ko 2024-03-23T23:52:00.771 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/ipset/ip_set_hash_ipportnet.ko 2024-03-23T23:52:00.771 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/ipset/ip_set_hash_mac.ko 2024-03-23T23:52:00.771 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/ipset/ip_set_hash_net.ko 2024-03-23T23:52:00.771 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/ipset/ip_set_hash_netiface.ko 2024-03-23T23:52:00.771 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/ipset/ip_set_hash_netnet.ko 2024-03-23T23:52:00.771 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/ipset/ip_set_hash_netport.ko 2024-03-23T23:52:00.771 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/ipset/ip_set_hash_netportnet.ko 2024-03-23T23:52:00.771 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/ipset/ip_set_list_set.ko 2024-03-23T23:52:00.771 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/ipvs 2024-03-23T23:52:00.772 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/ipvs/ip_vs.ko 2024-03-23T23:52:00.772 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/ipvs/ip_vs_dh.ko 2024-03-23T23:52:00.772 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/ipvs/ip_vs_fo.ko 2024-03-23T23:52:00.772 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/ipvs/ip_vs_ftp.ko 2024-03-23T23:52:00.772 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/ipvs/ip_vs_lblc.ko 2024-03-23T23:52:00.772 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/ipvs/ip_vs_lblcr.ko 2024-03-23T23:52:00.772 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/ipvs/ip_vs_lc.ko 2024-03-23T23:52:00.772 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/ipvs/ip_vs_nq.ko 2024-03-23T23:52:00.772 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/ipvs/ip_vs_pe_sip.ko 2024-03-23T23:52:00.772 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/ipvs/ip_vs_rr.ko 2024-03-23T23:52:00.772 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/ipvs/ip_vs_sed.ko 2024-03-23T23:52:00.772 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/ipvs/ip_vs_sh.ko 2024-03-23T23:52:00.772 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/ipvs/ip_vs_wlc.ko 2024-03-23T23:52:00.772 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/ipvs/ip_vs_wrr.ko 2024-03-23T23:52:00.773 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/nf_conncount.ko 2024-03-23T23:52:00.773 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/nf_conntrack.ko 2024-03-23T23:52:00.773 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/nf_conntrack_amanda.ko 2024-03-23T23:52:00.773 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/nf_conntrack_broadcast.ko 2024-03-23T23:52:00.773 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/nf_conntrack_ftp.ko 2024-03-23T23:52:00.773 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/nf_conntrack_h323.ko 2024-03-23T23:52:00.773 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/nf_conntrack_irc.ko 2024-03-23T23:52:00.773 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/nf_conntrack_netbios_ns.ko 2024-03-23T23:52:00.774 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/nf_conntrack_netlink.ko 2024-03-23T23:52:00.774 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/nf_conntrack_pptp.ko 2024-03-23T23:52:00.774 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/nf_conntrack_sane.ko 2024-03-23T23:52:00.774 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/nf_conntrack_sip.ko 2024-03-23T23:52:00.774 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/nf_conntrack_snmp.ko 2024-03-23T23:52:00.774 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/nf_conntrack_tftp.ko 2024-03-23T23:52:00.774 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/nf_dup_netdev.ko 2024-03-23T23:52:00.774 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/nf_flow_table.ko 2024-03-23T23:52:00.774 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/nf_flow_table_inet.ko 2024-03-23T23:52:00.774 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/nf_log_syslog.ko 2024-03-23T23:52:00.774 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/nf_nat.ko 2024-03-23T23:52:00.774 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/nf_nat_amanda.ko 2024-03-23T23:52:00.774 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/nf_nat_ftp.ko 2024-03-23T23:52:00.775 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/nf_nat_irc.ko 2024-03-23T23:52:00.775 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/nf_nat_sip.ko 2024-03-23T23:52:00.775 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/nf_nat_tftp.ko 2024-03-23T23:52:00.775 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/nf_synproxy_core.ko 2024-03-23T23:52:00.775 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/nf_tables.ko 2024-03-23T23:52:00.775 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/nfnetlink.ko 2024-03-23T23:52:00.775 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/nfnetlink_acct.ko 2024-03-23T23:52:00.775 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/nfnetlink_log.ko 2024-03-23T23:52:00.775 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/nfnetlink_osf.ko 2024-03-23T23:52:00.776 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/nfnetlink_queue.ko 2024-03-23T23:52:00.776 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/nft_chain_nat.ko 2024-03-23T23:52:00.776 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/nft_compat.ko 2024-03-23T23:52:00.776 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/nft_ct.ko 2024-03-23T23:52:00.776 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/nft_dup_netdev.ko 2024-03-23T23:52:00.776 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/nft_fib.ko 2024-03-23T23:52:00.776 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/nft_fwd_netdev.ko 2024-03-23T23:52:00.776 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/nft_hash.ko 2024-03-23T23:52:00.776 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/nft_limit.ko 2024-03-23T23:52:00.776 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/nft_log.ko 2024-03-23T23:52:00.776 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/nft_masq.ko 2024-03-23T23:52:00.776 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/nft_nat.ko 2024-03-23T23:52:00.777 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/nft_numgen.ko 2024-03-23T23:52:00.777 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/nft_queue.ko 2024-03-23T23:52:00.777 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/nft_quota.ko 2024-03-23T23:52:00.777 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/nft_redir.ko 2024-03-23T23:52:00.777 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/nft_reject.ko 2024-03-23T23:52:00.777 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/nft_reject_inet.ko 2024-03-23T23:52:00.777 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/nft_socket.ko 2024-03-23T23:52:00.777 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/nft_tunnel.ko 2024-03-23T23:52:00.777 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_AUDIT.ko 2024-03-23T23:52:00.777 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_CHECKSUM.ko 2024-03-23T23:52:00.778 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_CLASSIFY.ko 2024-03-23T23:52:00.778 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_CONNSECMARK.ko 2024-03-23T23:52:00.778 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_CT.ko 2024-03-23T23:52:00.778 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_DSCP.ko 2024-03-23T23:52:00.778 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_HL.ko 2024-03-23T23:52:00.778 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_HMARK.ko 2024-03-23T23:52:00.778 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_IDLETIMER.ko 2024-03-23T23:52:00.778 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_LED.ko 2024-03-23T23:52:00.778 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_LOG.ko 2024-03-23T23:52:00.778 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_MASQUERADE.ko 2024-03-23T23:52:00.778 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_NETMAP.ko 2024-03-23T23:52:00.778 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_NFLOG.ko 2024-03-23T23:52:00.779 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_NFQUEUE.ko 2024-03-23T23:52:00.779 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_RATEEST.ko 2024-03-23T23:52:00.779 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_REDIRECT.ko 2024-03-23T23:52:00.779 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_SECMARK.ko 2024-03-23T23:52:00.779 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_TCPMSS.ko 2024-03-23T23:52:00.779 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_TCPOPTSTRIP.ko 2024-03-23T23:52:00.779 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_TEE.ko 2024-03-23T23:52:00.779 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_TPROXY.ko 2024-03-23T23:52:00.779 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_TRACE.ko 2024-03-23T23:52:00.780 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_addrtype.ko 2024-03-23T23:52:00.780 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_bpf.ko 2024-03-23T23:52:00.780 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_cgroup.ko 2024-03-23T23:52:00.780 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_cluster.ko 2024-03-23T23:52:00.780 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_comment.ko 2024-03-23T23:52:00.780 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_connbytes.ko 2024-03-23T23:52:00.780 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_connlabel.ko 2024-03-23T23:52:00.780 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_connlimit.ko 2024-03-23T23:52:00.780 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_connmark.ko 2024-03-23T23:52:00.780 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_conntrack.ko 2024-03-23T23:52:00.780 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_cpu.ko 2024-03-23T23:52:00.781 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_dccp.ko 2024-03-23T23:52:00.781 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_devgroup.ko 2024-03-23T23:52:00.781 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_dscp.ko 2024-03-23T23:52:00.781 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_ecn.ko 2024-03-23T23:52:00.781 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_esp.ko 2024-03-23T23:52:00.781 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_hashlimit.ko 2024-03-23T23:52:00.781 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_helper.ko 2024-03-23T23:52:00.781 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_hl.ko 2024-03-23T23:52:00.781 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_ipcomp.ko 2024-03-23T23:52:00.781 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_iprange.ko 2024-03-23T23:52:00.781 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_ipvs.ko 2024-03-23T23:52:00.781 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_l2tp.ko 2024-03-23T23:52:00.781 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_length.ko 2024-03-23T23:52:00.782 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_limit.ko 2024-03-23T23:52:00.782 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_mac.ko 2024-03-23T23:52:00.782 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_mark.ko 2024-03-23T23:52:00.782 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_multiport.ko 2024-03-23T23:52:00.782 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_nat.ko 2024-03-23T23:52:00.782 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_nfacct.ko 2024-03-23T23:52:00.782 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_osf.ko 2024-03-23T23:52:00.782 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_owner.ko 2024-03-23T23:52:00.782 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_physdev.ko 2024-03-23T23:52:00.782 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_pkttype.ko 2024-03-23T23:52:00.782 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_policy.ko 2024-03-23T23:52:00.782 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_quota.ko 2024-03-23T23:52:00.783 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_rateest.ko 2024-03-23T23:52:00.783 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_realm.ko 2024-03-23T23:52:00.783 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_recent.ko 2024-03-23T23:52:00.783 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_sctp.ko 2024-03-23T23:52:00.783 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_set.ko 2024-03-23T23:52:00.783 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_socket.ko 2024-03-23T23:52:00.783 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_state.ko 2024-03-23T23:52:00.783 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_statistic.ko 2024-03-23T23:52:00.783 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_string.ko 2024-03-23T23:52:00.783 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_tcpmss.ko 2024-03-23T23:52:00.783 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_time.ko 2024-03-23T23:52:00.783 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netfilter/xt_u32.ko 2024-03-23T23:52:00.783 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netlink 2024-03-23T23:52:00.783 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/netlink/netlink_diag.ko 2024-03-23T23:52:00.784 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/nsh 2024-03-23T23:52:00.784 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/nsh/nsh.ko 2024-03-23T23:52:00.784 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/openvswitch 2024-03-23T23:52:00.784 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/openvswitch/openvswitch.ko 2024-03-23T23:52:00.784 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/openvswitch/vport-geneve.ko 2024-03-23T23:52:00.784 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/openvswitch/vport-gre.ko 2024-03-23T23:52:00.784 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/openvswitch/vport-vxlan.ko 2024-03-23T23:52:00.784 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/packet 2024-03-23T23:52:00.784 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/packet/af_packet_diag.ko 2024-03-23T23:52:00.784 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/rds 2024-03-23T23:52:00.784 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/rds/rds.ko 2024-03-23T23:52:00.784 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/rds/rds_rdma.ko 2024-03-23T23:52:00.784 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/rds/rds_tcp.ko 2024-03-23T23:52:00.785 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/sched 2024-03-23T23:52:00.785 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/sched/act_csum.ko 2024-03-23T23:52:00.785 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/sched/act_gact.ko 2024-03-23T23:52:00.785 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/sched/act_mirred.ko 2024-03-23T23:52:00.785 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/sched/act_nat.ko 2024-03-23T23:52:00.785 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/sched/act_pedit.ko 2024-03-23T23:52:00.785 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/sched/act_police.ko 2024-03-23T23:52:00.785 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/sched/act_simple.ko 2024-03-23T23:52:00.785 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/sched/act_skbedit.ko 2024-03-23T23:52:00.785 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/sched/act_vlan.ko 2024-03-23T23:52:00.785 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/sched/cls_basic.ko 2024-03-23T23:52:00.786 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/sched/cls_bpf.ko 2024-03-23T23:52:00.786 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/sched/cls_flow.ko 2024-03-23T23:52:00.786 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/sched/cls_fw.ko 2024-03-23T23:52:00.786 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/sched/cls_route.ko 2024-03-23T23:52:00.786 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/sched/cls_u32.ko 2024-03-23T23:52:00.786 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/sched/em_cmp.ko 2024-03-23T23:52:00.786 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/sched/em_ipset.ko 2024-03-23T23:52:00.786 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/sched/em_meta.ko 2024-03-23T23:52:00.786 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/sched/em_nbyte.ko 2024-03-23T23:52:00.787 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/sched/em_text.ko 2024-03-23T23:52:00.787 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/sched/em_u32.ko 2024-03-23T23:52:00.787 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/sched/sch_choke.ko 2024-03-23T23:52:00.787 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/sched/sch_codel.ko 2024-03-23T23:52:00.787 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/sched/sch_drr.ko 2024-03-23T23:52:00.787 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/sched/sch_fq.ko 2024-03-23T23:52:00.787 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/sched/sch_gred.ko 2024-03-23T23:52:00.787 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/sched/sch_hfsc.ko 2024-03-23T23:52:00.787 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/sched/sch_hhf.ko 2024-03-23T23:52:00.787 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/sched/sch_htb.ko 2024-03-23T23:52:00.788 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/sched/sch_ingress.ko 2024-03-23T23:52:00.788 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/sched/sch_mqprio.ko 2024-03-23T23:52:00.788 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/sched/sch_mqprio_lib.ko 2024-03-23T23:52:00.788 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/sched/sch_multiq.ko 2024-03-23T23:52:00.788 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/sched/sch_netem.ko 2024-03-23T23:52:00.788 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/sched/sch_pie.ko 2024-03-23T23:52:00.788 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/sched/sch_plug.ko 2024-03-23T23:52:00.788 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/sched/sch_prio.ko 2024-03-23T23:52:00.788 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/sched/sch_qfq.ko 2024-03-23T23:52:00.788 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/sched/sch_red.ko 2024-03-23T23:52:00.788 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/sched/sch_sfb.ko 2024-03-23T23:52:00.788 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/sched/sch_sfq.ko 2024-03-23T23:52:00.788 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/sched/sch_tbf.ko 2024-03-23T23:52:00.788 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/sched/sch_teql.ko 2024-03-23T23:52:00.789 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/sctp 2024-03-23T23:52:00.789 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/sctp/sctp.ko 2024-03-23T23:52:00.789 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/sctp/sctp_diag.ko 2024-03-23T23:52:00.789 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/sunrpc 2024-03-23T23:52:00.789 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/sunrpc/auth_gss 2024-03-23T23:52:00.789 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/sunrpc/auth_gss/auth_rpcgss.ko 2024-03-23T23:52:00.789 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/sunrpc/auth_gss/rpcsec_gss_krb5.ko 2024-03-23T23:52:00.789 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/sunrpc/sunrpc.ko 2024-03-23T23:52:00.789 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/sunrpc/xprtrdma 2024-03-23T23:52:00.789 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/sunrpc/xprtrdma/rpcrdma.ko 2024-03-23T23:52:00.789 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/tipc 2024-03-23T23:52:00.789 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/tipc/diag.ko 2024-03-23T23:52:00.789 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/tipc/tipc.ko 2024-03-23T23:52:00.790 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/unix 2024-03-23T23:52:00.790 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/unix/unix_diag.ko 2024-03-23T23:52:00.790 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/vmw_vsock 2024-03-23T23:52:00.790 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/vmw_vsock/vmw_vsock_virtio_transport_common.ko 2024-03-23T23:52:00.790 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/vmw_vsock/vmw_vsock_vmci_transport.ko 2024-03-23T23:52:00.790 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/vmw_vsock/vsock.ko 2024-03-23T23:52:00.790 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/vmw_vsock/vsock_diag.ko 2024-03-23T23:52:00.790 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/vmw_vsock/vsock_loopback.ko 2024-03-23T23:52:00.790 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/xfrm 2024-03-23T23:52:00.790 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/net/xfrm/xfrm_ipcomp.ko 2024-03-23T23:52:00.790 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/security 2024-03-23T23:52:00.790 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/security/keys 2024-03-23T23:52:00.790 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/security/keys/encrypted-keys 2024-03-23T23:52:00.791 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/security/keys/encrypted-keys/encrypted-keys.ko 2024-03-23T23:52:00.791 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/security/keys/trusted-keys 2024-03-23T23:52:00.791 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/security/keys/trusted-keys/trusted.ko 2024-03-23T23:52:00.791 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/virt 2024-03-23T23:52:00.791 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/virt/lib 2024-03-23T23:52:00.791 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/kernel/virt/lib/irqbypass.ko 2024-03-23T23:52:00.791 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/modules.alias 2024-03-23T23:52:00.791 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/modules.alias.bin 2024-03-23T23:52:00.791 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/modules.builtin 2024-03-23T23:52:00.791 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/modules.builtin.alias.bin 2024-03-23T23:52:00.791 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/modules.builtin.bin 2024-03-23T23:52:00.791 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/modules.builtin.modinfo 2024-03-23T23:52:00.791 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/modules.dep 2024-03-23T23:52:00.791 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/modules.dep.bin 2024-03-23T23:52:00.792 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/modules.devname 2024-03-23T23:52:00.792 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/modules.order 2024-03-23T23:52:00.792 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/modules.softdep 2024-03-23T23:52:00.792 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/modules.symbols 2024-03-23T23:52:00.792 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/modules.symbols.bin 2024-03-23T23:52:00.792 INFO:teuthology.orchestra.run.smithi118.stdout:/lib/modules/6.8.0-g5ec3e6e88b2e/vmlinuz 2024-03-23T23:52:00.796 DEBUG:teuthology.task.kernel:get_image_version: 6.8.0-g5ec3e6e88b2e 2024-03-23T23:52:00.796 DEBUG:teuthology.orchestra.run.smithi118:> rpm --scripts -qp /tmp/kernel.x86_64.rpm 2024-03-23T23:52:00.819 INFO:teuthology.orchestra.run.smithi118.stdout:postinstall scriptlet (using /bin/sh): 2024-03-23T23:52:00.819 INFO:teuthology.orchestra.run.smithi118.stdout:if [ -x /usr/bin/kernel-install ]; then 2024-03-23T23:52:00.819 INFO:teuthology.orchestra.run.smithi118.stdout: /usr/bin/kernel-install add 6.8.0-g5ec3e6e88b2e /lib/modules/6.8.0-g5ec3e6e88b2e/vmlinuz 2024-03-23T23:52:00.819 INFO:teuthology.orchestra.run.smithi118.stdout:fi 2024-03-23T23:52:00.819 INFO:teuthology.orchestra.run.smithi118.stdout:for file in vmlinuz System.map config; do 2024-03-23T23:52:00.820 INFO:teuthology.orchestra.run.smithi118.stdout: if ! cmp --silent "/lib/modules/6.8.0-g5ec3e6e88b2e/${file}" "/boot/${file}-6.8.0-g5ec3e6e88b2e"; then 2024-03-23T23:52:00.820 INFO:teuthology.orchestra.run.smithi118.stdout: cp "/lib/modules/6.8.0-g5ec3e6e88b2e/${file}" "/boot/${file}-6.8.0-g5ec3e6e88b2e" 2024-03-23T23:52:00.820 INFO:teuthology.orchestra.run.smithi118.stdout: fi 2024-03-23T23:52:00.820 INFO:teuthology.orchestra.run.smithi118.stdout:done 2024-03-23T23:52:00.820 INFO:teuthology.orchestra.run.smithi118.stdout:preuninstall scriptlet (using /bin/sh): 2024-03-23T23:52:00.820 INFO:teuthology.orchestra.run.smithi118.stdout:if [ -x /sbin/new-kernel-pkg ]; then 2024-03-23T23:52:00.820 INFO:teuthology.orchestra.run.smithi118.stdout:new-kernel-pkg --remove 6.8.0-g5ec3e6e88b2e --rminitrd --initrdfile=/boot/initramfs-6.8.0-g5ec3e6e88b2e.img 2024-03-23T23:52:00.820 INFO:teuthology.orchestra.run.smithi118.stdout:elif [ -x /usr/bin/kernel-install ]; then 2024-03-23T23:52:00.820 INFO:teuthology.orchestra.run.smithi118.stdout:kernel-install remove 6.8.0-g5ec3e6e88b2e 2024-03-23T23:52:00.820 INFO:teuthology.orchestra.run.smithi118.stdout:fi 2024-03-23T23:52:00.820 INFO:teuthology.orchestra.run.smithi118.stdout:postuninstall scriptlet (using /bin/sh): 2024-03-23T23:52:00.821 INFO:teuthology.orchestra.run.smithi118.stdout:if [ -x /sbin/update-bootloader ]; then 2024-03-23T23:52:00.821 INFO:teuthology.orchestra.run.smithi118.stdout:/sbin/update-bootloader --remove 6.8.0-g5ec3e6e88b2e 2024-03-23T23:52:00.821 INFO:teuthology.orchestra.run.smithi118.stdout:fi 2024-03-23T23:52:00.821 DEBUG:teuthology.orchestra.run.smithi118:> sudo rpm -qi grub2-tools 2024-03-23T23:52:00.893 INFO:teuthology.orchestra.run.smithi118.stdout:Name : grub2-tools 2024-03-23T23:52:00.893 INFO:teuthology.orchestra.run.smithi118.stdout:Epoch : 1 2024-03-23T23:52:00.893 INFO:teuthology.orchestra.run.smithi118.stdout:Version : 2.06 2024-03-23T23:52:00.894 INFO:teuthology.orchestra.run.smithi118.stdout:Release : 68.el9 2024-03-23T23:52:00.894 INFO:teuthology.orchestra.run.smithi118.stdout:Architecture: x86_64 2024-03-23T23:52:00.894 INFO:teuthology.orchestra.run.smithi118.stdout:Install Date: Fri 01 Sep 2023 09:14:30 PM UTC 2024-03-23T23:52:00.894 INFO:teuthology.orchestra.run.smithi118.stdout:Group : Unspecified 2024-03-23T23:52:00.894 INFO:teuthology.orchestra.run.smithi118.stdout:Size : 8264254 2024-03-23T23:52:00.894 INFO:teuthology.orchestra.run.smithi118.stdout:License : GPLv3+ 2024-03-23T23:52:00.894 INFO:teuthology.orchestra.run.smithi118.stdout:Signature : RSA/SHA256, Mon 07 Aug 2023 01:57:16 PM UTC, Key ID 05b555b38483c65d 2024-03-23T23:52:00.894 INFO:teuthology.orchestra.run.smithi118.stdout:Source RPM : grub2-2.06-68.el9.src.rpm 2024-03-23T23:52:00.894 INFO:teuthology.orchestra.run.smithi118.stdout:Build Date : Wed 02 Aug 2023 03:15:48 PM UTC 2024-03-23T23:52:00.894 INFO:teuthology.orchestra.run.smithi118.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2024-03-23T23:52:00.895 INFO:teuthology.orchestra.run.smithi118.stdout:Packager : builder@centos.org 2024-03-23T23:52:00.895 INFO:teuthology.orchestra.run.smithi118.stdout:Vendor : CentOS 2024-03-23T23:52:00.895 INFO:teuthology.orchestra.run.smithi118.stdout:URL : http://www.gnu.org/software/grub/ 2024-03-23T23:52:00.895 INFO:teuthology.orchestra.run.smithi118.stdout:Summary : Support tools for GRUB. 2024-03-23T23:52:00.895 INFO:teuthology.orchestra.run.smithi118.stdout:Description : 2024-03-23T23:52:00.895 INFO:teuthology.orchestra.run.smithi118.stdout: 2024-03-23T23:52:00.895 INFO:teuthology.orchestra.run.smithi118.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-03-23T23:52:00.895 INFO:teuthology.orchestra.run.smithi118.stdout:customizable bootloader with modular architecture. It supports a rich 2024-03-23T23:52:00.895 INFO:teuthology.orchestra.run.smithi118.stdout:variety of kernel formats, file systems, computer architectures and 2024-03-23T23:52:00.895 INFO:teuthology.orchestra.run.smithi118.stdout:hardware devices. 2024-03-23T23:52:00.895 INFO:teuthology.orchestra.run.smithi118.stdout: 2024-03-23T23:52:00.895 INFO:teuthology.orchestra.run.smithi118.stdout:This subpackage provides tools for support of all platforms. 2024-03-23T23:52:00.896 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-03-23T23:52:00.896 INFO:teuthology.task.kernel:Updating grub on smithi118 to boot 6.8.0-g5ec3e6e88b2e 2024-03-23T23:52:00.896 DEBUG:teuthology.orchestra.run.smithi118:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-03-23T23:52:01.498 INFO:teuthology.orchestra.run.smithi118.stderr:Generating grub configuration file ... 2024-03-23T23:52:02.883 INFO:teuthology.orchestra.run.smithi118.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2024-03-23T23:52:02.917 INFO:teuthology.orchestra.run.smithi118.stderr:done 2024-03-23T23:52:02.925 DEBUG:teuthology.orchestra.run.smithi118:> mktemp 2024-03-23T23:52:02.941 INFO:teuthology.orchestra.run.smithi118.stdout:/tmp/tmp.MhNyjzwmMH 2024-03-23T23:52:02.941 DEBUG:teuthology.orchestra.run.smithi118:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.MhNyjzwmMH 2024-03-23T23:52:03.009 DEBUG:teuthology.orchestra.run.smithi118:> sudo chmod 0666 /tmp/tmp.MhNyjzwmMH 2024-03-23T23:52:03.140 DEBUG:teuthology.orchestra.remote:smithi118:/tmp/tmp.MhNyjzwmMH is 6KB 2024-03-23T23:52:03.188 DEBUG:teuthology.orchestra.run.smithi118:> rm -fr /tmp/tmp.MhNyjzwmMH 2024-03-23T23:52:03.203 DEBUG:teuthology.orchestra.run.smithi118:> sudo /bin/ls /boot/loader/entries || true 2024-03-23T23:52:03.267 INFO:teuthology.orchestra.run.smithi118.stdout:15949e3c88704300882482eddfd7baae-0-rescue.conf 2024-03-23T23:52:03.268 INFO:teuthology.orchestra.run.smithi118.stdout:15949e3c88704300882482eddfd7baae-5.14.0-239.el9.x86_64.conf 2024-03-23T23:52:03.268 INFO:teuthology.orchestra.run.smithi118.stdout:15949e3c88704300882482eddfd7baae-5.14.0-361.el9.x86_64.conf 2024-03-23T23:52:03.268 INFO:teuthology.orchestra.run.smithi118.stdout:15949e3c88704300882482eddfd7baae-6.8.0-g5ec3e6e88b2e.conf 2024-03-23T23:52:03.269 DEBUG:teuthology.orchestra.run.smithi118:> sudo grub2-set-default 15949e3c88704300882482eddfd7baae-6.8.0-g5ec3e6e88b2e 2024-03-23T23:52:03.413 DEBUG:teuthology.orchestra.run.smithi118:> sudo shutdown -r now 2024-03-23T23:52:33.416 INFO:teuthology.task.kernel:Checking client client.0 for new kernel version... 2024-03-23T23:52:33.417 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi118.front.sepia.ceph.com' 2024-03-23T23:52:33.418 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi118.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-23T23:52:51.742 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.118 2024-03-23T23:53:00.743 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi118.front.sepia.ceph.com' 2024-03-23T23:53:00.744 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi118.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-23T23:53:03.810 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.118 2024-03-23T23:53:15.811 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi118.front.sepia.ceph.com' 2024-03-23T23:53:15.813 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi118.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-23T23:53:18.878 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.118 2024-03-23T23:53:33.879 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi118.front.sepia.ceph.com' 2024-03-23T23:53:33.880 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi118.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-23T23:54:05.935 DEBUG:teuthology.orchestra.run.smithi118:> true 2024-03-23T23:54:06.637 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi118.front.sepia.ceph.com' 2024-03-23T23:54:06.637 INFO:teuthology.task.kernel:Checking kernel version of client.0, want "6.8.0-g5ec3e6e88b2e"... 2024-03-23T23:54:06.637 DEBUG:teuthology.orchestra.run.smithi118:> uname -r 2024-03-23T23:54:06.674 INFO:teuthology.orchestra.run.smithi118.stdout:6.8.0-g5ec3e6e88b2e 2024-03-23T23:54:06.675 DEBUG:teuthology.task.kernel:current kernel version is 6.8.0-g5ec3e6e88b2e vs 6.8.0-g5ec3e6e88b2e 2024-03-23T23:54:06.675 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-03-23T23:54:06.675 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-03-23T23:54:07.676 DEBUG:teuthology.parallel:result is None 2024-03-23T23:54:07.676 INFO:teuthology.run_tasks:Running task internal.base... 2024-03-23T23:54:07.683 INFO:teuthology.task.internal:Creating test directory... 2024-03-23T23:54:07.683 DEBUG:teuthology.orchestra.run.smithi118:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-03-23T23:54:07.691 DEBUG:teuthology.orchestra.run.smithi189:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-03-23T23:54:07.696 DEBUG:teuthology.orchestra.run.smithi193:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-03-23T23:54:07.734 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2024-03-23T23:54:07.741 INFO:teuthology.run_tasks:Running task internal.archive... 2024-03-23T23:54:07.752 INFO:teuthology.task.internal:Creating archive directory... 2024-03-23T23:54:07.752 DEBUG:teuthology.orchestra.run.smithi118:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-03-23T23:54:07.784 DEBUG:teuthology.orchestra.run.smithi189:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-03-23T23:54:07.787 DEBUG:teuthology.orchestra.run.smithi193:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-03-23T23:54:07.899 INFO:teuthology.run_tasks:Running task internal.coredump... 2024-03-23T23:54:07.905 INFO:teuthology.task.internal:Enabling coredump saving... 2024-03-23T23:54:07.905 DEBUG:teuthology.orchestra.run.smithi189:> test -f /run/.containerenv -o -f /.dockerenv 2024-03-23T23:54:07.921 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-23T23:54:07.922 DEBUG:teuthology.orchestra.run.smithi193:> test -f /run/.containerenv -o -f /.dockerenv 2024-03-23T23:54:07.938 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-23T23:54:07.939 DEBUG:teuthology.orchestra.run.smithi118:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-03-23T23:54:07.944 DEBUG:teuthology.orchestra.run.smithi189:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-03-23T23:54:07.965 DEBUG:teuthology.orchestra.run.smithi193:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-03-23T23:54:08.001 INFO:teuthology.orchestra.run.smithi189.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-23T23:54:08.025 INFO:teuthology.orchestra.run.smithi189.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-23T23:54:08.033 INFO:teuthology.orchestra.run.smithi193.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-23T23:54:08.052 INFO:teuthology.orchestra.run.smithi193.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-23T23:54:08.071 INFO:teuthology.orchestra.run.smithi118.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-23T23:54:08.145 INFO:teuthology.orchestra.run.smithi118.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-23T23:54:08.150 INFO:teuthology.run_tasks:Running task internal.sudo... 2024-03-23T23:54:08.158 INFO:teuthology.task.internal:Configuring sudo... 2024-03-23T23:54:08.159 DEBUG:teuthology.orchestra.run.smithi118:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-03-23T23:54:08.163 DEBUG:teuthology.orchestra.run.smithi189:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-03-23T23:54:08.166 DEBUG:teuthology.orchestra.run.smithi193:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-03-23T23:54:08.241 INFO:teuthology.run_tasks:Running task internal.syslog... 2024-03-23T23:54:08.252 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2024-03-23T23:54:08.252 DEBUG:teuthology.orchestra.run.smithi118:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-03-23T23:54:08.257 DEBUG:teuthology.orchestra.run.smithi189:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-03-23T23:54:08.259 DEBUG:teuthology.orchestra.run.smithi193:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-03-23T23:54:08.295 DEBUG:teuthology.orchestra.run.smithi118:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-23T23:54:08.407 DEBUG:teuthology.orchestra.run.smithi118:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-23T23:54:08.486 DEBUG:teuthology.orchestra.run.smithi118:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-23T23:54:08.555 DEBUG:teuthology.orchestra.run.smithi118:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-23T23:54:08.638 DEBUG:teuthology.orchestra.run.smithi118:> set -ex 2024-03-23T23:54:08.639 DEBUG:teuthology.orchestra.run.smithi118:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-03-23T23:54:08.733 DEBUG:teuthology.orchestra.run.smithi189:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-23T23:54:08.771 DEBUG:teuthology.orchestra.run.smithi189:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-23T23:54:08.835 DEBUG:teuthology.orchestra.run.smithi189:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-23T23:54:08.919 DEBUG:teuthology.orchestra.run.smithi189:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-23T23:54:08.964 DEBUG:teuthology.orchestra.run.smithi189:> set -ex 2024-03-23T23:54:08.964 DEBUG:teuthology.orchestra.run.smithi189:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-03-23T23:54:09.032 DEBUG:teuthology.orchestra.run.smithi193:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-23T23:54:09.079 DEBUG:teuthology.orchestra.run.smithi193:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-23T23:54:09.135 DEBUG:teuthology.orchestra.run.smithi193:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-23T23:54:09.216 DEBUG:teuthology.orchestra.run.smithi193:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-23T23:54:09.262 DEBUG:teuthology.orchestra.run.smithi193:> set -ex 2024-03-23T23:54:09.262 DEBUG:teuthology.orchestra.run.smithi193:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-03-23T23:54:09.331 DEBUG:teuthology.orchestra.run.smithi118:> sudo service rsyslog restart 2024-03-23T23:54:09.338 DEBUG:teuthology.orchestra.run.smithi189:> sudo service rsyslog restart 2024-03-23T23:54:09.341 DEBUG:teuthology.orchestra.run.smithi193:> sudo service rsyslog restart 2024-03-23T23:54:09.377 INFO:teuthology.orchestra.run.smithi189.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-03-23T23:54:09.408 INFO:teuthology.orchestra.run.smithi193.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-03-23T23:54:09.446 INFO:teuthology.orchestra.run.smithi118.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-03-23T23:54:09.881 INFO:teuthology.run_tasks:Running task internal.timer... 2024-03-23T23:54:09.889 INFO:teuthology.task.internal:Starting timer... 2024-03-23T23:54:09.889 INFO:teuthology.run_tasks:Running task pcp... 2024-03-23T23:54:09.898 INFO:teuthology.run_tasks:Running task selinux... 2024-03-23T23:54:09.907 DEBUG:teuthology.task:Applying overrides for task selinux: {'allowlist': ['scontext=system_u:system_r:logrotate_t:s0', 'scontext=system_u:system_r:getty_t:s0']} 2024-03-23T23:54:09.907 DEBUG:teuthology.orchestra.run.smithi118:> sudo service auditd rotate 2024-03-23T23:54:10.020 INFO:teuthology.orchestra.run.smithi118.stdout:Rotating logs: 2024-03-23T23:54:10.024 DEBUG:teuthology.orchestra.run.smithi189:> sudo service auditd rotate 2024-03-23T23:54:10.070 INFO:teuthology.orchestra.run.smithi189.stdout:Rotating logs: 2024-03-23T23:54:10.072 DEBUG:teuthology.orchestra.run.smithi193:> sudo service auditd rotate 2024-03-23T23:54:10.138 INFO:teuthology.orchestra.run.smithi193.stdout:Rotating logs: 2024-03-23T23:54:10.139 DEBUG:teuthology.task.selinux:Getting current SELinux state 2024-03-23T23:54:10.139 DEBUG:teuthology.orchestra.run.smithi118:> /usr/sbin/getenforce 2024-03-23T23:54:10.179 INFO:teuthology.orchestra.run.smithi118.stdout:Permissive 2024-03-23T23:54:10.180 DEBUG:teuthology.orchestra.run.smithi189:> /usr/sbin/getenforce 2024-03-23T23:54:10.201 INFO:teuthology.orchestra.run.smithi189.stdout:Permissive 2024-03-23T23:54:10.202 DEBUG:teuthology.orchestra.run.smithi193:> /usr/sbin/getenforce 2024-03-23T23:54:10.223 INFO:teuthology.orchestra.run.smithi193.stdout:Permissive 2024-03-23T23:54:10.223 DEBUG:teuthology.task.selinux:Existing SELinux modes: {'ubuntu@smithi118.front.sepia.ceph.com': 'permissive', 'ubuntu@smithi189.front.sepia.ceph.com': 'permissive', 'ubuntu@smithi193.front.sepia.ceph.com': 'permissive'} 2024-03-23T23:54:10.223 DEBUG:teuthology.orchestra.run.smithi118:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-03-23T23:54:10.306 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-23T23:54:10.306 DEBUG:teuthology.orchestra.run.smithi189:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-03-23T23:54:10.336 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-23T23:54:10.337 DEBUG:teuthology.orchestra.run.smithi193:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-03-23T23:54:10.364 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-23T23:54:10.365 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2024-03-23T23:54:10.365 DEBUG:teuthology.orchestra.run.smithi118:> sudo /usr/sbin/setenforce permissive 2024-03-23T23:54:10.468 DEBUG:teuthology.orchestra.run.smithi189:> sudo /usr/sbin/setenforce permissive 2024-03-23T23:54:10.503 DEBUG:teuthology.orchestra.run.smithi193:> sudo /usr/sbin/setenforce permissive 2024-03-23T23:54:10.533 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2024-03-23T23:54:10.549 DEBUG:teuthology.repo_utils:Setting repo remote to https://git.ceph.com/ceph-cm-ansible.git 2024-03-23T23:54:10.555 INFO:teuthology.repo_utils:Fetching git.ceph.com_ceph-cm-ansible_main from origin 2024-03-23T23:54:10.699 DEBUG:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2024-03-23T23:54:10.715 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2024-03-23T23:54:10.716 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi118.front.sepia.ceph.com,smithi189.front.sepia.ceph.com,smithi193.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2024-03-24T00:00:39.003 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi118.front.sepia.ceph.com'), Remote(name='ubuntu@smithi189.front.sepia.ceph.com'), Remote(name='ubuntu@smithi193.front.sepia.ceph.com')] 2024-03-24T00:00:39.026 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi118.front.sepia.ceph.com' 2024-03-24T00:00:39.027 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi118.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-24T00:00:39.173 DEBUG:teuthology.orchestra.run.smithi118:> true 2024-03-24T00:00:39.330 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi118.front.sepia.ceph.com' 2024-03-24T00:00:39.330 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi189.front.sepia.ceph.com' 2024-03-24T00:00:39.331 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi189.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-24T00:00:39.421 DEBUG:teuthology.orchestra.run.smithi189:> true 2024-03-24T00:00:39.489 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi189.front.sepia.ceph.com' 2024-03-24T00:00:39.490 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi193.front.sepia.ceph.com' 2024-03-24T00:00:39.490 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi193.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-24T00:00:39.577 DEBUG:teuthology.orchestra.run.smithi193:> true 2024-03-24T00:00:39.645 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi193.front.sepia.ceph.com' 2024-03-24T00:00:39.646 INFO:teuthology.run_tasks:Running task clock... 2024-03-24T00:00:39.661 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2024-03-24T00:00:39.661 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-03-24T00:00:39.661 DEBUG:teuthology.orchestra.run.smithi118:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-24T00:00:39.669 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-03-24T00:00:39.669 DEBUG:teuthology.orchestra.run.smithi189:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-24T00:00:39.671 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-03-24T00:00:39.671 DEBUG:teuthology.orchestra.run.smithi193:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-24T00:00:39.702 INFO:teuthology.orchestra.run.smithi189.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-03-24T00:00:39.717 INFO:teuthology.orchestra.run.smithi193.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-03-24T00:00:39.719 INFO:teuthology.orchestra.run.smithi189.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-03-24T00:00:39.735 INFO:teuthology.orchestra.run.smithi193.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-03-24T00:00:39.750 INFO:teuthology.orchestra.run.smithi189.stderr:sudo: ntpd: command not found 2024-03-24T00:00:39.763 INFO:teuthology.orchestra.run.smithi189.stdout:506 Cannot talk to daemon 2024-03-24T00:00:39.764 INFO:teuthology.orchestra.run.smithi193.stderr:sudo: ntpd: command not found 2024-03-24T00:00:39.764 INFO:teuthology.orchestra.run.smithi118.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-03-24T00:00:39.777 INFO:teuthology.orchestra.run.smithi193.stdout:506 Cannot talk to daemon 2024-03-24T00:00:39.780 INFO:teuthology.orchestra.run.smithi189.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-03-24T00:00:39.793 INFO:teuthology.orchestra.run.smithi193.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-03-24T00:00:39.796 INFO:teuthology.orchestra.run.smithi189.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-03-24T00:00:39.809 INFO:teuthology.orchestra.run.smithi193.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-03-24T00:00:39.817 INFO:teuthology.orchestra.run.smithi118.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-03-24T00:00:39.840 INFO:teuthology.orchestra.run.smithi189.stderr:bash: line 1: ntpq: command not found 2024-03-24T00:00:39.843 INFO:teuthology.orchestra.run.smithi189.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-03-24T00:00:39.843 INFO:teuthology.orchestra.run.smithi189.stdout:=============================================================================== 2024-03-24T00:00:39.843 INFO:teuthology.orchestra.run.smithi189.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-24T00:00:39.843 INFO:teuthology.orchestra.run.smithi189.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-24T00:00:39.843 INFO:teuthology.orchestra.run.smithi189.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-24T00:00:39.844 INFO:teuthology.orchestra.run.smithi189.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-24T00:00:39.853 INFO:teuthology.orchestra.run.smithi193.stderr:bash: line 1: ntpq: command not found 2024-03-24T00:00:39.856 INFO:teuthology.orchestra.run.smithi193.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-03-24T00:00:39.857 INFO:teuthology.orchestra.run.smithi193.stdout:=============================================================================== 2024-03-24T00:00:39.857 INFO:teuthology.orchestra.run.smithi193.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-24T00:00:39.857 INFO:teuthology.orchestra.run.smithi193.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-24T00:00:39.857 INFO:teuthology.orchestra.run.smithi193.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-24T00:00:39.857 INFO:teuthology.orchestra.run.smithi193.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-24T00:00:39.914 INFO:teuthology.orchestra.run.smithi118.stderr:sudo: ntpd: command not found 2024-03-24T00:00:39.957 INFO:teuthology.orchestra.run.smithi118.stdout:506 Cannot talk to daemon 2024-03-24T00:00:40.009 INFO:teuthology.orchestra.run.smithi118.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-03-24T00:00:40.064 INFO:teuthology.orchestra.run.smithi118.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-03-24T00:00:40.188 INFO:teuthology.orchestra.run.smithi118.stderr:bash: line 1: ntpq: command not found 2024-03-24T00:00:40.197 INFO:teuthology.orchestra.run.smithi118.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-03-24T00:00:40.197 INFO:teuthology.orchestra.run.smithi118.stdout:=============================================================================== 2024-03-24T00:00:40.197 INFO:teuthology.orchestra.run.smithi118.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-24T00:00:40.198 INFO:teuthology.orchestra.run.smithi118.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-24T00:00:40.198 INFO:teuthology.orchestra.run.smithi118.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-24T00:00:40.198 INFO:teuthology.orchestra.run.smithi118.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-24T00:00:40.199 INFO:teuthology.run_tasks:Running task install... 2024-03-24T00:00:40.238 DEBUG:teuthology.task.install:project ceph 2024-03-24T00:00:40.238 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'flavor': 'default', 'sha1': '3be9a5c9bc793e11e2800a8c0c696e8b46742033'}} 2024-03-24T00:00:40.238 DEBUG:teuthology.task.install:config {'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'flavor': 'default', 'sha1': '3be9a5c9bc793e11e2800a8c0c696e8b46742033'} 2024-03-24T00:00:40.238 INFO:teuthology.task.install:Using flavor: default 2024-03-24T00:00:40.244 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'ceph-volume', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-fuse', 'ceph-volume', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd']} 2024-03-24T00:00:40.244 INFO:teuthology.task.install:extra packages: {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror']} 2024-03-24T00:00:40.245 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '3be9a5c9bc793e11e2800a8c0c696e8b46742033', 'tag': None, 'wait_for_package': False} 2024-03-24T00:00:40.245 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=3be9a5c9bc793e11e2800a8c0c696e8b46742033 2024-03-24T00:00:40.247 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '3be9a5c9bc793e11e2800a8c0c696e8b46742033', 'tag': None, 'wait_for_package': False} 2024-03-24T00:00:40.247 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=3be9a5c9bc793e11e2800a8c0c696e8b46742033 2024-03-24T00:00:40.248 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror'], 'rpm': ['python3-cephfs', 'cephfs-shell', 'cephfs-top', 'cephfs-mirror']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'network-manager', 'iproute2', 'util-linux', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev', 'postgresql', 'postgresql-client', 'postgresql-common', 'postgresql-contrib'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'NetworkManager', 'iproute', 'util-linux', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel', 'postgresql', 'postgresql-server', 'postgresql-contrib']}, 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '3be9a5c9bc793e11e2800a8c0c696e8b46742033', 'tag': None, 'wait_for_package': False} 2024-03-24T00:00:40.248 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=3be9a5c9bc793e11e2800a8c0c696e8b46742033 2024-03-24T00:00:40.390 INFO:teuthology.task.install.rpm:Pulling from https://4.chacra.ceph.com/r/ceph/wip-mchangir-testing-main-20240318.032620/3be9a5c9bc793e11e2800a8c0c696e8b46742033/centos/9/flavors/default/ 2024-03-24T00:00:40.391 INFO:teuthology.task.install.rpm:Package version is 19.0.0-2145.g3be9a5c9 2024-03-24T00:00:40.402 INFO:teuthology.task.install.rpm:Pulling from https://4.chacra.ceph.com/r/ceph/wip-mchangir-testing-main-20240318.032620/3be9a5c9bc793e11e2800a8c0c696e8b46742033/centos/9/flavors/default/ 2024-03-24T00:00:40.402 INFO:teuthology.task.install.rpm:Package version is 19.0.0-2145.g3be9a5c9 2024-03-24T00:00:40.459 INFO:teuthology.task.install.rpm:Pulling from https://4.chacra.ceph.com/r/ceph/wip-mchangir-testing-main-20240318.032620/3be9a5c9bc793e11e2800a8c0c696e8b46742033/centos/9/flavors/default/ 2024-03-24T00:00:40.459 INFO:teuthology.task.install.rpm:Package version is 19.0.0-2145.g3be9a5c9 2024-03-24T00:00:40.563 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://4.chacra.ceph.com/r/ceph/wip-mchangir-testing-main-20240318.032620/3be9a5c9bc793e11e2800a8c0c696e8b46742033/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://4.chacra.ceph.com/r/ceph/wip-mchangir-testing-main-20240318.032620/3be9a5c9bc793e11e2800a8c0c696e8b46742033/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://4.chacra.ceph.com/r/ceph/wip-mchangir-testing-main-20240318.032620/3be9a5c9bc793e11e2800a8c0c696e8b46742033/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-03-24T00:00:40.564 DEBUG:teuthology.orchestra.run.smithi118:> set -ex 2024-03-24T00:00:40.564 DEBUG:teuthology.orchestra.run.smithi118:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-03-24T00:00:40.569 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://4.chacra.ceph.com/r/ceph/wip-mchangir-testing-main-20240318.032620/3be9a5c9bc793e11e2800a8c0c696e8b46742033/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://4.chacra.ceph.com/r/ceph/wip-mchangir-testing-main-20240318.032620/3be9a5c9bc793e11e2800a8c0c696e8b46742033/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://4.chacra.ceph.com/r/ceph/wip-mchangir-testing-main-20240318.032620/3be9a5c9bc793e11e2800a8c0c696e8b46742033/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-03-24T00:00:40.569 DEBUG:teuthology.orchestra.run.smithi193:> set -ex 2024-03-24T00:00:40.569 DEBUG:teuthology.orchestra.run.smithi193:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-03-24T00:00:40.579 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://4.chacra.ceph.com/r/ceph/wip-mchangir-testing-main-20240318.032620/3be9a5c9bc793e11e2800a8c0c696e8b46742033/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://4.chacra.ceph.com/r/ceph/wip-mchangir-testing-main-20240318.032620/3be9a5c9bc793e11e2800a8c0c696e8b46742033/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://4.chacra.ceph.com/r/ceph/wip-mchangir-testing-main-20240318.032620/3be9a5c9bc793e11e2800a8c0c696e8b46742033/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-03-24T00:00:40.579 DEBUG:teuthology.orchestra.run.smithi189:> set -ex 2024-03-24T00:00:40.579 DEBUG:teuthology.orchestra.run.smithi189:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-03-24T00:00:40.598 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd, python3-cephfs, cephfs-shell, cephfs-top, cephfs-mirror, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib on remote rpm x86_64 2024-03-24T00:00:40.598 DEBUG:teuthology.orchestra.run.smithi193:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/3be9a5c9bc793e11e2800a8c0c696e8b46742033/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-03-24T00:00:40.611 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd, python3-cephfs, cephfs-shell, cephfs-top, cephfs-mirror, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib on remote rpm x86_64 2024-03-24T00:00:40.611 DEBUG:teuthology.orchestra.run.smithi189:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/3be9a5c9bc793e11e2800a8c0c696e8b46742033/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-03-24T00:00:40.666 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, ceph-volume, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd, python3-cephfs, cephfs-shell, cephfs-top, cephfs-mirror, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib, bison, flex, elfutils-libelf-devel, openssl-devel, NetworkManager, iproute, util-linux, libacl-devel, libaio-devel, libattr-devel, libtool, libuuid-devel, xfsdump, xfsprogs, xfsprogs-devel, libaio-devel, libtool, libuuid-devel, xfsprogs-devel, postgresql, postgresql-server, postgresql-contrib on remote rpm x86_64 2024-03-24T00:00:40.666 DEBUG:teuthology.orchestra.run.smithi118:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/3be9a5c9bc793e11e2800a8c0c696e8b46742033/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-03-24T00:00:40.668 DEBUG:teuthology.orchestra.run.smithi193:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-03-24T00:00:40.681 DEBUG:teuthology.orchestra.run.smithi189:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-03-24T00:00:40.750 DEBUG:teuthology.orchestra.run.smithi193:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-03-24T00:00:40.765 DEBUG:teuthology.orchestra.run.smithi189:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-03-24T00:00:40.804 DEBUG:teuthology.orchestra.run.smithi118:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-03-24T00:00:40.817 INFO:teuthology.orchestra.run.smithi193.stdout:check_obsoletes = 1 2024-03-24T00:00:40.819 DEBUG:teuthology.orchestra.run.smithi193:> sudo yum clean all 2024-03-24T00:00:40.832 INFO:teuthology.orchestra.run.smithi189.stdout:check_obsoletes = 1 2024-03-24T00:00:40.834 DEBUG:teuthology.orchestra.run.smithi189:> sudo yum clean all 2024-03-24T00:00:40.935 DEBUG:teuthology.orchestra.run.smithi118:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-03-24T00:00:41.022 INFO:teuthology.orchestra.run.smithi118.stdout:check_obsoletes = 1 2024-03-24T00:00:41.026 DEBUG:teuthology.orchestra.run.smithi118:> sudo yum clean all 2024-03-24T00:00:41.053 INFO:teuthology.orchestra.run.smithi193.stdout:74 files removed 2024-03-24T00:00:41.070 INFO:teuthology.orchestra.run.smithi189.stdout:74 files removed 2024-03-24T00:00:41.081 DEBUG:teuthology.orchestra.run.smithi193:> sudo yum -y install ceph-radosgw ceph-test ceph ceph-base cephadm ceph-immutable-object-cache ceph-mgr ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-rook ceph-mgr-cephadm ceph-fuse ceph-volume librados-devel libcephfs2 libcephfs-devel librados2 librbd1 python3-rados python3-rgw python3-cephfs python3-rbd rbd-fuse rbd-mirror rbd-nbd python3-cephfs cephfs-shell cephfs-top cephfs-mirror bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib 2024-03-24T00:00:41.097 DEBUG:teuthology.orchestra.run.smithi189:> sudo yum -y install ceph-radosgw ceph-test ceph ceph-base cephadm ceph-immutable-object-cache ceph-mgr ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-rook ceph-mgr-cephadm ceph-fuse ceph-volume librados-devel libcephfs2 libcephfs-devel librados2 librbd1 python3-rados python3-rgw python3-cephfs python3-rbd rbd-fuse rbd-mirror rbd-nbd python3-cephfs cephfs-shell cephfs-top cephfs-mirror bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib 2024-03-24T00:00:41.402 INFO:teuthology.orchestra.run.smithi118.stdout:74 files removed 2024-03-24T00:00:41.443 DEBUG:teuthology.orchestra.run.smithi118:> sudo yum -y install ceph-radosgw ceph-test ceph ceph-base cephadm ceph-immutable-object-cache ceph-mgr ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-rook ceph-mgr-cephadm ceph-fuse ceph-volume librados-devel libcephfs2 libcephfs-devel librados2 librbd1 python3-rados python3-rgw python3-cephfs python3-rbd rbd-fuse rbd-mirror rbd-nbd python3-cephfs cephfs-shell cephfs-top cephfs-mirror bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib bison flex elfutils-libelf-devel openssl-devel NetworkManager iproute util-linux libacl-devel libaio-devel libattr-devel libtool libuuid-devel xfsdump xfsprogs xfsprogs-devel libaio-devel libtool libuuid-devel xfsprogs-devel postgresql postgresql-server postgresql-contrib 2024-03-24T00:00:41.739 INFO:teuthology.orchestra.run.smithi189.stdout:ceph packages for x86_64 236 kB/s | 83 kB 00:00 2024-03-24T00:00:41.768 INFO:teuthology.orchestra.run.smithi193.stdout:ceph packages for x86_64 209 kB/s | 83 kB 00:00 2024-03-24T00:00:42.144 INFO:teuthology.orchestra.run.smithi193.stdout:ceph noarch packages 30 kB/s | 10 kB 00:00 2024-03-24T00:00:42.157 INFO:teuthology.orchestra.run.smithi189.stdout:ceph noarch packages 27 kB/s | 10 kB 00:00 2024-03-24T00:00:42.455 INFO:teuthology.orchestra.run.smithi118.stdout:ceph packages for x86_64 165 kB/s | 83 kB 00:00 2024-03-24T00:00:42.524 INFO:teuthology.orchestra.run.smithi189.stdout:ceph source packages 5.6 kB/s | 1.9 kB 00:00 2024-03-24T00:00:42.553 INFO:teuthology.orchestra.run.smithi193.stdout:ceph source packages 5.0 kB/s | 1.9 kB 00:00 2024-03-24T00:00:43.027 INFO:teuthology.orchestra.run.smithi189.stdout:CentOS-9 - AppStream 14 MB/s | 6.7 MB 00:00 2024-03-24T00:00:43.089 INFO:teuthology.orchestra.run.smithi118.stdout:ceph noarch packages 19 kB/s | 10 kB 00:00 2024-03-24T00:00:43.114 INFO:teuthology.orchestra.run.smithi193.stdout:CentOS-9 - AppStream 13 MB/s | 6.7 MB 00:00 2024-03-24T00:00:43.724 INFO:teuthology.orchestra.run.smithi118.stdout:ceph source packages 3.4 kB/s | 1.9 kB 00:00 2024-03-24T00:00:44.460 INFO:teuthology.orchestra.run.smithi189.stdout:CentOS-9 - BaseOS 6.1 MB/s | 2.0 MB 00:00 2024-03-24T00:00:44.597 INFO:teuthology.orchestra.run.smithi193.stdout:CentOS-9 - BaseOS 5.1 MB/s | 2.0 MB 00:00 2024-03-24T00:00:44.642 INFO:teuthology.orchestra.run.smithi118.stdout:CentOS-9 - AppStream 7.9 MB/s | 6.7 MB 00:00 2024-03-24T00:00:45.459 INFO:teuthology.orchestra.run.smithi189.stdout:CentOS Stream 9 - BaseOS 12 MB/s | 8.0 MB 00:00 2024-03-24T00:00:45.613 INFO:teuthology.orchestra.run.smithi193.stdout:CentOS Stream 9 - BaseOS 12 MB/s | 8.0 MB 00:00 2024-03-24T00:00:46.353 INFO:teuthology.orchestra.run.smithi118.stdout:CentOS-9 - BaseOS 4.9 MB/s | 2.0 MB 00:00 2024-03-24T00:00:47.373 INFO:teuthology.orchestra.run.smithi193.stdout:CentOS Stream 9 - AppStream 26 MB/s | 19 MB 00:00 2024-03-24T00:00:47.787 INFO:teuthology.orchestra.run.smithi118.stdout:CentOS Stream 9 - BaseOS 7.9 MB/s | 8.0 MB 00:01 2024-03-24T00:00:49.193 INFO:teuthology.orchestra.run.smithi189.stdout:CentOS Stream 9 - AppStream 7.0 MB/s | 19 MB 00:02 2024-03-24T00:00:50.472 INFO:teuthology.orchestra.run.smithi118.stdout:CentOS Stream 9 - AppStream 13 MB/s | 19 MB 00:01 2024-03-24T00:00:52.325 INFO:teuthology.orchestra.run.smithi193.stdout:CentOS Stream 9 - CRB 4.2 MB/s | 6.1 MB 00:01 2024-03-24T00:00:53.744 INFO:teuthology.orchestra.run.smithi189.stdout:CentOS Stream 9 - CRB 5.5 MB/s | 6.1 MB 00:01 2024-03-24T00:00:53.766 INFO:teuthology.orchestra.run.smithi193.stdout:CentOS Stream 9 - Extras packages 57 kB/s | 16 kB 00:00 2024-03-24T00:00:54.892 INFO:teuthology.orchestra.run.smithi118.stdout:CentOS Stream 9 - CRB 10 MB/s | 6.1 MB 00:00 2024-03-24T00:00:55.386 INFO:teuthology.orchestra.run.smithi189.stdout:CentOS Stream 9 - Extras packages 30 kB/s | 16 kB 00:00 2024-03-24T00:00:56.468 INFO:teuthology.orchestra.run.smithi118.stdout:CentOS Stream 9 - Extras packages 62 kB/s | 16 kB 00:00 2024-03-24T00:00:56.988 INFO:teuthology.orchestra.run.smithi189.stdout:Extra Packages for Enterprise Linux 14 MB/s | 21 MB 00:01 2024-03-24T00:00:58.117 INFO:teuthology.orchestra.run.smithi118.stdout:Extra Packages for Enterprise Linux 14 MB/s | 21 MB 00:01 2024-03-24T00:00:59.200 INFO:teuthology.orchestra.run.smithi193.stdout:Extra Packages for Enterprise Linux 3.9 MB/s | 21 MB 00:05 2024-03-24T00:01:01.846 INFO:teuthology.orchestra.run.smithi189.stdout:lab-extras 32 kB/s | 1.7 kB 00:00 2024-03-24T00:01:03.987 INFO:teuthology.orchestra.run.smithi118.stdout:lab-extras 33 kB/s | 1.7 kB 00:00 2024-03-24T00:01:04.219 INFO:teuthology.orchestra.run.smithi189.stdout:Package librados2-2:16.2.4-5.el9.x86_64 is already installed. 2024-03-24T00:01:04.220 INFO:teuthology.orchestra.run.smithi189.stdout:Package librbd1-2:16.2.4-5.el9.x86_64 is already installed. 2024-03-24T00:01:04.233 INFO:teuthology.orchestra.run.smithi193.stdout:lab-extras 14 kB/s | 1.7 kB 00:00 2024-03-24T00:01:04.234 INFO:teuthology.orchestra.run.smithi189.stdout:Package NetworkManager-1:1.44.0-2.el9.x86_64 is already installed. 2024-03-24T00:01:04.236 INFO:teuthology.orchestra.run.smithi189.stdout:Package iproute-6.2.0-5.el9.x86_64 is already installed. 2024-03-24T00:01:04.236 INFO:teuthology.orchestra.run.smithi189.stdout:Package util-linux-2.37.4-15.el9.x86_64 is already installed. 2024-03-24T00:01:04.238 INFO:teuthology.orchestra.run.smithi189.stdout:Package libacl-devel-2.3.1-3.el9.x86_64 is already installed. 2024-03-24T00:01:04.240 INFO:teuthology.orchestra.run.smithi189.stdout:Package libattr-devel-2.5.1-3.el9.x86_64 is already installed. 2024-03-24T00:01:04.243 INFO:teuthology.orchestra.run.smithi189.stdout:Package xfsdump-3.1.12-3.el9.x86_64 is already installed. 2024-03-24T00:01:04.244 INFO:teuthology.orchestra.run.smithi189.stdout:Package xfsprogs-5.19.0-4.el9.x86_64 is already installed. 2024-03-24T00:01:04.325 INFO:teuthology.orchestra.run.smithi189.stdout:Dependencies resolved. 2024-03-24T00:01:04.348 INFO:teuthology.orchestra.run.smithi189.stdout:============================================================================================ 2024-03-24T00:01:04.348 INFO:teuthology.orchestra.run.smithi189.stdout: Package Arch Version Repository Size 2024-03-24T00:01:04.348 INFO:teuthology.orchestra.run.smithi189.stdout:============================================================================================ 2024-03-24T00:01:04.349 INFO:teuthology.orchestra.run.smithi189.stdout:Installing: 2024-03-24T00:01:04.349 INFO:teuthology.orchestra.run.smithi189.stdout: bison x86_64 3.7.4-5.el9 CentOS-AppStream 920 k 2024-03-24T00:01:04.349 INFO:teuthology.orchestra.run.smithi189.stdout: ceph x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 6.6 k 2024-03-24T00:01:04.349 INFO:teuthology.orchestra.run.smithi189.stdout: ceph-base x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 5.4 M 2024-03-24T00:01:04.349 INFO:teuthology.orchestra.run.smithi189.stdout: ceph-fuse x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 890 k 2024-03-24T00:01:04.349 INFO:teuthology.orchestra.run.smithi189.stdout: ceph-immutable-object-cache x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 145 k 2024-03-24T00:01:04.349 INFO:teuthology.orchestra.run.smithi189.stdout: ceph-mgr x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 1.5 M 2024-03-24T00:01:04.349 INFO:teuthology.orchestra.run.smithi189.stdout: ceph-mgr-cephadm noarch 2:19.0.0-2145.g3be9a5c9.el9 ceph-noarch 139 k 2024-03-24T00:01:04.349 INFO:teuthology.orchestra.run.smithi189.stdout: ceph-mgr-dashboard noarch 2:19.0.0-2145.g3be9a5c9.el9 ceph-noarch 1.8 M 2024-03-24T00:01:04.349 INFO:teuthology.orchestra.run.smithi189.stdout: ceph-mgr-diskprediction-local noarch 2:19.0.0-2145.g3be9a5c9.el9 ceph-noarch 7.4 M 2024-03-24T00:01:04.349 INFO:teuthology.orchestra.run.smithi189.stdout: ceph-mgr-rook noarch 2:19.0.0-2145.g3be9a5c9.el9 ceph-noarch 49 k 2024-03-24T00:01:04.349 INFO:teuthology.orchestra.run.smithi189.stdout: ceph-radosgw x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 9.9 M 2024-03-24T00:01:04.350 INFO:teuthology.orchestra.run.smithi189.stdout: ceph-test x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 48 M 2024-03-24T00:01:04.350 INFO:teuthology.orchestra.run.smithi189.stdout: ceph-volume noarch 2:19.0.0-2145.g3be9a5c9.el9 ceph-noarch 264 k 2024-03-24T00:01:04.350 INFO:teuthology.orchestra.run.smithi189.stdout: cephadm noarch 2:19.0.0-2145.g3be9a5c9.el9 ceph-noarch 748 k 2024-03-24T00:01:04.350 INFO:teuthology.orchestra.run.smithi189.stdout: cephfs-mirror x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 220 k 2024-03-24T00:01:04.350 INFO:teuthology.orchestra.run.smithi189.stdout: cephfs-shell x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 27 k 2024-03-24T00:01:04.350 INFO:teuthology.orchestra.run.smithi189.stdout: cephfs-top noarch 2:19.0.0-2145.g3be9a5c9.el9 ceph-noarch 18 k 2024-03-24T00:01:04.350 INFO:teuthology.orchestra.run.smithi189.stdout: elfutils-libelf-devel x86_64 0.190-2.el9 appstream 21 k 2024-03-24T00:01:04.350 INFO:teuthology.orchestra.run.smithi189.stdout: flex x86_64 2.6.4-9.el9 CentOS-AppStream 307 k 2024-03-24T00:01:04.350 INFO:teuthology.orchestra.run.smithi189.stdout: libaio-devel x86_64 0.3.111-13.el9 CentOS-AppStream 10 k 2024-03-24T00:01:04.350 INFO:teuthology.orchestra.run.smithi189.stdout: libcephfs-devel x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 31 k 2024-03-24T00:01:04.350 INFO:teuthology.orchestra.run.smithi189.stdout: libcephfs2 x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 746 k 2024-03-24T00:01:04.350 INFO:teuthology.orchestra.run.smithi189.stdout: librados-devel x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 127 k 2024-03-24T00:01:04.350 INFO:teuthology.orchestra.run.smithi189.stdout: libtool x86_64 2.4.6-45.el9 CentOS-AppStream 578 k 2024-03-24T00:01:04.351 INFO:teuthology.orchestra.run.smithi189.stdout: libuuid-devel x86_64 2.37.4-18.el9 appstream 25 k 2024-03-24T00:01:04.351 INFO:teuthology.orchestra.run.smithi189.stdout: openssl-devel x86_64 1:3.0.7-27.el9 appstream 4.1 M 2024-03-24T00:01:04.351 INFO:teuthology.orchestra.run.smithi189.stdout: postgresql x86_64 13.11-1.el9 CentOS-AppStream 1.5 M 2024-03-24T00:01:04.351 INFO:teuthology.orchestra.run.smithi189.stdout: postgresql-contrib x86_64 13.11-1.el9 CentOS-AppStream 811 k 2024-03-24T00:01:04.351 INFO:teuthology.orchestra.run.smithi189.stdout: postgresql-server x86_64 13.11-1.el9 CentOS-AppStream 5.7 M 2024-03-24T00:01:04.351 INFO:teuthology.orchestra.run.smithi189.stdout: python3-cephfs x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 162 k 2024-03-24T00:01:04.351 INFO:teuthology.orchestra.run.smithi189.stdout: python3-rados x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 321 k 2024-03-24T00:01:04.351 INFO:teuthology.orchestra.run.smithi189.stdout: python3-rbd x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 298 k 2024-03-24T00:01:04.351 INFO:teuthology.orchestra.run.smithi189.stdout: python3-rgw x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 99 k 2024-03-24T00:01:04.351 INFO:teuthology.orchestra.run.smithi189.stdout: rbd-fuse x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 85 k 2024-03-24T00:01:04.351 INFO:teuthology.orchestra.run.smithi189.stdout: rbd-mirror x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 3.1 M 2024-03-24T00:01:04.351 INFO:teuthology.orchestra.run.smithi189.stdout: rbd-nbd x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 170 k 2024-03-24T00:01:04.352 INFO:teuthology.orchestra.run.smithi189.stdout: xfsprogs-devel x86_64 6.3.0-1.el9 appstream 94 k 2024-03-24T00:01:04.352 INFO:teuthology.orchestra.run.smithi189.stdout:Upgrading: 2024-03-24T00:01:04.352 INFO:teuthology.orchestra.run.smithi189.stdout: NetworkManager x86_64 1:1.47.2-1.el9 baseos 2.3 M 2024-03-24T00:01:04.352 INFO:teuthology.orchestra.run.smithi189.stdout: NetworkManager-libnm x86_64 1:1.47.2-1.el9 baseos 1.8 M 2024-03-24T00:01:04.352 INFO:teuthology.orchestra.run.smithi189.stdout: NetworkManager-team x86_64 1:1.47.2-1.el9 baseos 42 k 2024-03-24T00:01:04.352 INFO:teuthology.orchestra.run.smithi189.stdout: NetworkManager-tui x86_64 1:1.47.2-1.el9 baseos 248 k 2024-03-24T00:01:04.352 INFO:teuthology.orchestra.run.smithi189.stdout: acl x86_64 2.3.1-4.el9 baseos 71 k 2024-03-24T00:01:04.352 INFO:teuthology.orchestra.run.smithi189.stdout: elfutils-debuginfod-client x86_64 0.190-2.el9 baseos 36 k 2024-03-24T00:01:04.352 INFO:teuthology.orchestra.run.smithi189.stdout: elfutils-libelf x86_64 0.190-2.el9 baseos 192 k 2024-03-24T00:01:04.352 INFO:teuthology.orchestra.run.smithi189.stdout: elfutils-libs x86_64 0.190-2.el9 baseos 256 k 2024-03-24T00:01:04.352 INFO:teuthology.orchestra.run.smithi189.stdout: libacl x86_64 2.3.1-4.el9 baseos 23 k 2024-03-24T00:01:04.352 INFO:teuthology.orchestra.run.smithi189.stdout: libacl-devel x86_64 2.3.1-4.el9 appstream 74 k 2024-03-24T00:01:04.352 INFO:teuthology.orchestra.run.smithi189.stdout: libblkid x86_64 2.37.4-18.el9 baseos 108 k 2024-03-24T00:01:04.352 INFO:teuthology.orchestra.run.smithi189.stdout: libfdisk x86_64 2.37.4-18.el9 baseos 155 k 2024-03-24T00:01:04.353 INFO:teuthology.orchestra.run.smithi189.stdout: libmount x86_64 2.37.4-18.el9 baseos 136 k 2024-03-24T00:01:04.353 INFO:teuthology.orchestra.run.smithi189.stdout: librados2 x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 3.4 M 2024-03-24T00:01:04.353 INFO:teuthology.orchestra.run.smithi189.stdout: librbd1 x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 3.1 M 2024-03-24T00:01:04.353 INFO:teuthology.orchestra.run.smithi189.stdout: libsmartcols x86_64 2.37.4-18.el9 baseos 64 k 2024-03-24T00:01:04.353 INFO:teuthology.orchestra.run.smithi189.stdout: libuuid x86_64 2.37.4-18.el9 baseos 29 k 2024-03-24T00:01:04.353 INFO:teuthology.orchestra.run.smithi189.stdout: openssl x86_64 1:3.0.7-27.el9 baseos 1.2 M 2024-03-24T00:01:04.353 INFO:teuthology.orchestra.run.smithi189.stdout: openssl-libs x86_64 1:3.0.7-27.el9 baseos 2.2 M 2024-03-24T00:01:04.353 INFO:teuthology.orchestra.run.smithi189.stdout: util-linux x86_64 2.37.4-18.el9 baseos 2.3 M 2024-03-24T00:01:04.353 INFO:teuthology.orchestra.run.smithi189.stdout: util-linux-core x86_64 2.37.4-18.el9 baseos 465 k 2024-03-24T00:01:04.353 INFO:teuthology.orchestra.run.smithi189.stdout: util-linux-user x86_64 2.37.4-18.el9 baseos 31 k 2024-03-24T00:01:04.353 INFO:teuthology.orchestra.run.smithi189.stdout: xfsdump x86_64 3.1.12-4.el9 CentOS-BaseOS 321 k 2024-03-24T00:01:04.353 INFO:teuthology.orchestra.run.smithi189.stdout: xfsprogs x86_64 6.3.0-1.el9 baseos 1.1 M 2024-03-24T00:01:04.353 INFO:teuthology.orchestra.run.smithi189.stdout:Installing dependencies: 2024-03-24T00:01:04.354 INFO:teuthology.orchestra.run.smithi189.stdout: abseil-cpp x86_64 20211102.0-3.el9 epel 548 k 2024-03-24T00:01:04.354 INFO:teuthology.orchestra.run.smithi189.stdout: automake noarch 1.16.2-8.el9 CentOS-AppStream 662 k 2024-03-24T00:01:04.354 INFO:teuthology.orchestra.run.smithi189.stdout: boost-program-options x86_64 1.75.0-8.el9 CentOS-AppStream 106 k 2024-03-24T00:01:04.354 INFO:teuthology.orchestra.run.smithi189.stdout: ceph-common x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 20 M 2024-03-24T00:01:04.354 INFO:teuthology.orchestra.run.smithi189.stdout: ceph-grafana-dashboards noarch 2:19.0.0-2145.g3be9a5c9.el9 ceph-noarch 34 k 2024-03-24T00:01:04.354 INFO:teuthology.orchestra.run.smithi189.stdout: ceph-mds x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 2.3 M 2024-03-24T00:01:04.354 INFO:teuthology.orchestra.run.smithi189.stdout: ceph-mgr-modules-core noarch 2:19.0.0-2145.g3be9a5c9.el9 ceph-noarch 248 k 2024-03-24T00:01:04.354 INFO:teuthology.orchestra.run.smithi189.stdout: ceph-mon x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 4.7 M 2024-03-24T00:01:04.354 INFO:teuthology.orchestra.run.smithi189.stdout: ceph-osd x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 17 M 2024-03-24T00:01:04.354 INFO:teuthology.orchestra.run.smithi189.stdout: ceph-prometheus-alerts noarch 2:19.0.0-2145.g3be9a5c9.el9 ceph-noarch 16 k 2024-03-24T00:01:04.354 INFO:teuthology.orchestra.run.smithi189.stdout: ceph-selinux x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 25 k 2024-03-24T00:01:04.355 INFO:teuthology.orchestra.run.smithi189.stdout: flexiblas x86_64 3.0.4-8.el9 CentOS-AppStream 32 k 2024-03-24T00:01:04.355 INFO:teuthology.orchestra.run.smithi189.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 CentOS-AppStream 3.0 M 2024-03-24T00:01:04.355 INFO:teuthology.orchestra.run.smithi189.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 CentOS-AppStream 17 k 2024-03-24T00:01:04.355 INFO:teuthology.orchestra.run.smithi189.stdout: fuse x86_64 2.9.9-15.el9 CentOS-BaseOS 79 k 2024-03-24T00:01:04.355 INFO:teuthology.orchestra.run.smithi189.stdout: gperftools-libs x86_64 2.9.1-2.el9 epel 309 k 2024-03-24T00:01:04.355 INFO:teuthology.orchestra.run.smithi189.stdout: grpc-data noarch 1.46.7-10.el9 epel 19 k 2024-03-24T00:01:04.355 INFO:teuthology.orchestra.run.smithi189.stdout: jq x86_64 1.6-16.el9 baseos 187 k 2024-03-24T00:01:04.355 INFO:teuthology.orchestra.run.smithi189.stdout: libarrow x86_64 9.0.0-10.el9 epel 4.4 M 2024-03-24T00:01:04.355 INFO:teuthology.orchestra.run.smithi189.stdout: libarrow-doc noarch 9.0.0-10.el9 epel 26 k 2024-03-24T00:01:04.355 INFO:teuthology.orchestra.run.smithi189.stdout: libcephsqlite x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 163 k 2024-03-24T00:01:04.355 INFO:teuthology.orchestra.run.smithi189.stdout: libgfortran x86_64 11.4.1-3.el9 baseos 803 k 2024-03-24T00:01:04.355 INFO:teuthology.orchestra.run.smithi189.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-03-24T00:01:04.356 INFO:teuthology.orchestra.run.smithi189.stdout: libpmemobj x86_64 1.12.1-1.el9 CentOS-AppStream 159 k 2024-03-24T00:01:04.356 INFO:teuthology.orchestra.run.smithi189.stdout: libquadmath x86_64 11.4.1-3.el9 baseos 195 k 2024-03-24T00:01:04.356 INFO:teuthology.orchestra.run.smithi189.stdout: librabbitmq x86_64 0.11.0-7.el9 CentOS-AppStream 44 k 2024-03-24T00:01:04.356 INFO:teuthology.orchestra.run.smithi189.stdout: libradosstriper1 x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 501 k 2024-03-24T00:01:04.356 INFO:teuthology.orchestra.run.smithi189.stdout: librdkafka x86_64 1.6.1-102.el9 CentOS-AppStream 661 k 2024-03-24T00:01:04.356 INFO:teuthology.orchestra.run.smithi189.stdout: librgw2 x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 5.1 M 2024-03-24T00:01:04.356 INFO:teuthology.orchestra.run.smithi189.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-03-24T00:01:04.356 INFO:teuthology.orchestra.run.smithi189.stdout: libzstd-devel x86_64 1.5.1-2.el9 CentOS-AppStream 46 k 2024-03-24T00:01:04.356 INFO:teuthology.orchestra.run.smithi189.stdout: lttng-ust x86_64 2.12.0-6.el9 CentOS-AppStream 282 k 2024-03-24T00:01:04.356 INFO:teuthology.orchestra.run.smithi189.stdout: lua x86_64 5.4.4-4.el9 CentOS-AppStream 187 k 2024-03-24T00:01:04.356 INFO:teuthology.orchestra.run.smithi189.stdout: lua-devel x86_64 5.4.4-4.el9 crb 22 k 2024-03-24T00:01:04.356 INFO:teuthology.orchestra.run.smithi189.stdout: luarocks noarch 3.9.2-1.el9 epel 151 k 2024-03-24T00:01:04.357 INFO:teuthology.orchestra.run.smithi189.stdout: oniguruma x86_64 6.9.6-1.el9.5 CentOS-AppStream 217 k 2024-03-24T00:01:04.357 INFO:teuthology.orchestra.run.smithi189.stdout: openblas x86_64 0.3.21-2.el9 CentOS-AppStream 34 k 2024-03-24T00:01:04.357 INFO:teuthology.orchestra.run.smithi189.stdout: openblas-openmp x86_64 0.3.21-2.el9 CentOS-AppStream 4.7 M 2024-03-24T00:01:04.357 INFO:teuthology.orchestra.run.smithi189.stdout: parquet-libs x86_64 9.0.0-10.el9 epel 838 k 2024-03-24T00:01:04.357 INFO:teuthology.orchestra.run.smithi189.stdout: postgresql-private-libs x86_64 13.11-1.el9 CentOS-AppStream 134 k 2024-03-24T00:01:04.357 INFO:teuthology.orchestra.run.smithi189.stdout: protobuf x86_64 3.14.0-13.el9 CentOS-AppStream 1.0 M 2024-03-24T00:01:04.357 INFO:teuthology.orchestra.run.smithi189.stdout: protobuf-compiler x86_64 3.14.0-13.el9 crb 863 k 2024-03-24T00:01:04.357 INFO:teuthology.orchestra.run.smithi189.stdout: python3-asyncssh noarch 2.13.2-5.el9 epel 548 k 2024-03-24T00:01:04.357 INFO:teuthology.orchestra.run.smithi189.stdout: python3-attrs noarch 20.3.0-7.el9 CentOS-AppStream 78 k 2024-03-24T00:01:04.357 INFO:teuthology.orchestra.run.smithi189.stdout: python3-babel noarch 2.9.1-2.el9 CentOS-AppStream 5.8 M 2024-03-24T00:01:04.357 INFO:teuthology.orchestra.run.smithi189.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-03-24T00:01:04.357 INFO:teuthology.orchestra.run.smithi189.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-03-24T00:01:04.357 INFO:teuthology.orchestra.run.smithi189.stdout: python3-ceph-argparse x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 45 k 2024-03-24T00:01:04.357 INFO:teuthology.orchestra.run.smithi189.stdout: python3-ceph-common x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 131 k 2024-03-24T00:01:04.358 INFO:teuthology.orchestra.run.smithi189.stdout: python3-certifi noarch 2021.10.8-2.el9 epel 15 k 2024-03-24T00:01:04.358 INFO:teuthology.orchestra.run.smithi189.stdout: python3-cffi x86_64 1.14.5-5.el9 CentOS-BaseOS 241 k 2024-03-24T00:01:04.358 INFO:teuthology.orchestra.run.smithi189.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-03-24T00:01:04.358 INFO:teuthology.orchestra.run.smithi189.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-03-24T00:01:04.358 INFO:teuthology.orchestra.run.smithi189.stdout: python3-cmd2 noarch 2.4.2-2.el9 epel 303 k 2024-03-24T00:01:04.358 INFO:teuthology.orchestra.run.smithi189.stdout: python3-colorama noarch 0.4.4-12.el9 epel 35 k 2024-03-24T00:01:04.358 INFO:teuthology.orchestra.run.smithi189.stdout: python3-cryptography x86_64 36.0.1-4.el9 CentOS-BaseOS 1.1 M 2024-03-24T00:01:04.358 INFO:teuthology.orchestra.run.smithi189.stdout: python3-devel x86_64 3.9.17-2.el9 appstream 245 k 2024-03-24T00:01:04.358 INFO:teuthology.orchestra.run.smithi189.stdout: python3-google-auth noarch 1:2.28.2-1.el9 epel 203 k 2024-03-24T00:01:04.358 INFO:teuthology.orchestra.run.smithi189.stdout: python3-grpcio x86_64 1.46.7-10.el9 epel 2.0 M 2024-03-24T00:01:04.358 INFO:teuthology.orchestra.run.smithi189.stdout: python3-grpcio-tools x86_64 1.46.7-10.el9 epel 144 k 2024-03-24T00:01:04.358 INFO:teuthology.orchestra.run.smithi189.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-03-24T00:01:04.358 INFO:teuthology.orchestra.run.smithi189.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-03-24T00:01:04.359 INFO:teuthology.orchestra.run.smithi189.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2024-03-24T00:01:04.359 INFO:teuthology.orchestra.run.smithi189.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-03-24T00:01:04.359 INFO:teuthology.orchestra.run.smithi189.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-03-24T00:01:04.359 INFO:teuthology.orchestra.run.smithi189.stdout: python3-jinja2 noarch 2.11.3-5.el9 appstream 248 k 2024-03-24T00:01:04.359 INFO:teuthology.orchestra.run.smithi189.stdout: python3-jsonpatch noarch 1.21-16.el9 CentOS-AppStream 24 k 2024-03-24T00:01:04.359 INFO:teuthology.orchestra.run.smithi189.stdout: python3-jsonpointer noarch 2.0-4.el9 CentOS-AppStream 18 k 2024-03-24T00:01:04.359 INFO:teuthology.orchestra.run.smithi189.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-03-24T00:01:04.359 INFO:teuthology.orchestra.run.smithi189.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-03-24T00:01:04.359 INFO:teuthology.orchestra.run.smithi189.stdout: python3-mako noarch 1.1.4-6.el9 CentOS-AppStream 144 k 2024-03-24T00:01:04.359 INFO:teuthology.orchestra.run.smithi189.stdout: python3-markupsafe x86_64 1.1.1-12.el9 CentOS-AppStream 32 k 2024-03-24T00:01:04.359 INFO:teuthology.orchestra.run.smithi189.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-03-24T00:01:04.359 INFO:teuthology.orchestra.run.smithi189.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-03-24T00:01:04.359 INFO:teuthology.orchestra.run.smithi189.stdout: python3-numpy x86_64 1:1.20.1-5.el9 CentOS-AppStream 4.8 M 2024-03-24T00:01:04.359 INFO:teuthology.orchestra.run.smithi189.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 CentOS-AppStream 217 k 2024-03-24T00:01:04.360 INFO:teuthology.orchestra.run.smithi189.stdout: python3-oauthlib noarch 3.1.1-5.el9 CentOS-AppStream 174 k 2024-03-24T00:01:04.360 INFO:teuthology.orchestra.run.smithi189.stdout: python3-packaging noarch 20.9-5.el9 CentOS-AppStream 69 k 2024-03-24T00:01:04.360 INFO:teuthology.orchestra.run.smithi189.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-03-24T00:01:04.360 INFO:teuthology.orchestra.run.smithi189.stdout: python3-ply noarch 3.11-14.el9 CentOS-BaseOS 103 k 2024-03-24T00:01:04.360 INFO:teuthology.orchestra.run.smithi189.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-03-24T00:01:04.360 INFO:teuthology.orchestra.run.smithi189.stdout: python3-prettytable noarch 0.7.2-27.el9 CentOS-AppStream 41 k 2024-03-24T00:01:04.360 INFO:teuthology.orchestra.run.smithi189.stdout: python3-protobuf noarch 3.14.0-13.el9 CentOS-AppStream 238 k 2024-03-24T00:01:04.360 INFO:teuthology.orchestra.run.smithi189.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-03-24T00:01:04.360 INFO:teuthology.orchestra.run.smithi189.stdout: python3-pyasn1 noarch 0.4.8-6.el9 CentOS-AppStream 132 k 2024-03-24T00:01:04.360 INFO:teuthology.orchestra.run.smithi189.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 CentOS-AppStream 211 k 2024-03-24T00:01:04.360 INFO:teuthology.orchestra.run.smithi189.stdout: python3-pycparser noarch 2.20-6.el9 CentOS-BaseOS 124 k 2024-03-24T00:01:04.360 INFO:teuthology.orchestra.run.smithi189.stdout: python3-pyperclip noarch 1.8.2-2.el9 epel 28 k 2024-03-24T00:01:04.360 INFO:teuthology.orchestra.run.smithi189.stdout: python3-pytz noarch 2021.1-5.el9 CentOS-AppStream 47 k 2024-03-24T00:01:04.360 INFO:teuthology.orchestra.run.smithi189.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-03-24T00:01:04.361 INFO:teuthology.orchestra.run.smithi189.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-03-24T00:01:04.361 INFO:teuthology.orchestra.run.smithi189.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 CentOS-AppStream 43 k 2024-03-24T00:01:04.361 INFO:teuthology.orchestra.run.smithi189.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-03-24T00:01:04.361 INFO:teuthology.orchestra.run.smithi189.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-03-24T00:01:04.361 INFO:teuthology.orchestra.run.smithi189.stdout: python3-scipy x86_64 1.6.2-8.el9 CentOS-AppStream 16 M 2024-03-24T00:01:04.361 INFO:teuthology.orchestra.run.smithi189.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-03-24T00:01:04.361 INFO:teuthology.orchestra.run.smithi189.stdout: python3-toml noarch 0.10.2-6.el9 CentOS-AppStream 39 k 2024-03-24T00:01:04.361 INFO:teuthology.orchestra.run.smithi189.stdout: python3-typing-extensions noarch 4.4.0-2.el9 epel 51 k 2024-03-24T00:01:04.361 INFO:teuthology.orchestra.run.smithi189.stdout: python3-urllib3 noarch 1.26.5-5.el9 CentOS-BaseOS 187 k 2024-03-24T00:01:04.361 INFO:teuthology.orchestra.run.smithi189.stdout: python3-wcwidth noarch 0.2.5-8.el9 CentOS-AppStream 41 k 2024-03-24T00:01:04.361 INFO:teuthology.orchestra.run.smithi189.stdout: python3-webob noarch 1.8.7-6.el9 epel 230 k 2024-03-24T00:01:04.361 INFO:teuthology.orchestra.run.smithi189.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-03-24T00:01:04.362 INFO:teuthology.orchestra.run.smithi189.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-03-24T00:01:04.362 INFO:teuthology.orchestra.run.smithi189.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-03-24T00:01:04.362 INFO:teuthology.orchestra.run.smithi189.stdout: qatlib x86_64 23.11.0-1.el9 appstream 216 k 2024-03-24T00:01:04.362 INFO:teuthology.orchestra.run.smithi189.stdout: qatzip-libs x86_64 1.1.2-2.el9 appstream 44 k 2024-03-24T00:01:04.362 INFO:teuthology.orchestra.run.smithi189.stdout: re2 x86_64 1:20211101-3.el9 epel 192 k 2024-03-24T00:01:04.362 INFO:teuthology.orchestra.run.smithi189.stdout: socat x86_64 1.7.4.1-5.el9 CentOS-AppStream 300 k 2024-03-24T00:01:04.362 INFO:teuthology.orchestra.run.smithi189.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-03-24T00:01:04.362 INFO:teuthology.orchestra.run.smithi189.stdout: uuid x86_64 1.6.2-55.el9 CentOS-AppStream 56 k 2024-03-24T00:01:04.362 INFO:teuthology.orchestra.run.smithi189.stdout: xmlstarlet x86_64 1.6.1-20.el9 CentOS-AppStream 63 k 2024-03-24T00:01:04.362 INFO:teuthology.orchestra.run.smithi189.stdout: zlib-devel x86_64 1.2.11-40.el9 CentOS-AppStream 44 k 2024-03-24T00:01:04.362 INFO:teuthology.orchestra.run.smithi189.stdout:Installing weak dependencies: 2024-03-24T00:01:04.362 INFO:teuthology.orchestra.run.smithi189.stdout: qatlib-service x86_64 23.11.0-1.el9 appstream 35 k 2024-03-24T00:01:04.362 INFO:teuthology.orchestra.run.smithi189.stdout: 2024-03-24T00:01:04.362 INFO:teuthology.orchestra.run.smithi189.stdout:Transaction Summary 2024-03-24T00:01:04.363 INFO:teuthology.orchestra.run.smithi189.stdout:============================================================================================ 2024-03-24T00:01:04.363 INFO:teuthology.orchestra.run.smithi189.stdout:Install 150 Packages 2024-03-24T00:01:04.363 INFO:teuthology.orchestra.run.smithi189.stdout:Upgrade 24 Packages 2024-03-24T00:01:04.363 INFO:teuthology.orchestra.run.smithi189.stdout: 2024-03-24T00:01:04.363 INFO:teuthology.orchestra.run.smithi189.stdout:Total download size: 224 M 2024-03-24T00:01:04.363 INFO:teuthology.orchestra.run.smithi189.stdout:Downloading Packages: 2024-03-24T00:01:05.189 INFO:teuthology.orchestra.run.smithi189.stdout:(1/174): ceph-19.0.0-2145.g3be9a5c9.el9.x86_64. 39 kB/s | 6.6 kB 00:00 2024-03-24T00:01:05.398 INFO:teuthology.orchestra.run.smithi189.stdout:(2/174): ceph-fuse-19.0.0-2145.g3be9a5c9.el9.x8 4.2 MB/s | 890 kB 00:00 2024-03-24T00:01:05.473 INFO:teuthology.orchestra.run.smithi189.stdout:(3/174): ceph-immutable-object-cache-19.0.0-214 1.9 MB/s | 145 kB 00:00 2024-03-24T00:01:05.557 INFO:teuthology.orchestra.run.smithi189.stdout:(4/174): ceph-base-19.0.0-2145.g3be9a5c9.el9.x8 10 MB/s | 5.4 MB 00:00 2024-03-24T00:01:05.690 INFO:teuthology.orchestra.run.smithi189.stdout:(5/174): ceph-mgr-19.0.0-2145.g3be9a5c9.el9.x86 11 MB/s | 1.5 MB 00:00 2024-03-24T00:01:05.766 INFO:teuthology.orchestra.run.smithi189.stdout:(6/174): ceph-mds-19.0.0-2145.g3be9a5c9.el9.x86 7.9 MB/s | 2.3 MB 00:00 2024-03-24T00:01:05.983 INFO:teuthology.orchestra.run.smithi189.stdout:(7/174): ceph-mon-19.0.0-2145.g3be9a5c9.el9.x86 16 MB/s | 4.7 MB 00:00 2024-03-24T00:01:06.555 INFO:teuthology.orchestra.run.smithi193.stdout:Package librados2-2:16.2.4-5.el9.x86_64 is already installed. 2024-03-24T00:01:06.556 INFO:teuthology.orchestra.run.smithi193.stdout:Package librbd1-2:16.2.4-5.el9.x86_64 is already installed. 2024-03-24T00:01:06.562 INFO:teuthology.orchestra.run.smithi118.stdout:Package librados2-2:16.2.4-5.el9.x86_64 is already installed. 2024-03-24T00:01:06.563 INFO:teuthology.orchestra.run.smithi118.stdout:Package librbd1-2:16.2.4-5.el9.x86_64 is already installed. 2024-03-24T00:01:06.570 INFO:teuthology.orchestra.run.smithi193.stdout:Package NetworkManager-1:1.44.0-2.el9.x86_64 is already installed. 2024-03-24T00:01:06.571 INFO:teuthology.orchestra.run.smithi193.stdout:Package iproute-6.2.0-5.el9.x86_64 is already installed. 2024-03-24T00:01:06.572 INFO:teuthology.orchestra.run.smithi193.stdout:Package util-linux-2.37.4-15.el9.x86_64 is already installed. 2024-03-24T00:01:06.573 INFO:teuthology.orchestra.run.smithi193.stdout:Package libacl-devel-2.3.1-3.el9.x86_64 is already installed. 2024-03-24T00:01:06.575 INFO:teuthology.orchestra.run.smithi193.stdout:Package libattr-devel-2.5.1-3.el9.x86_64 is already installed. 2024-03-24T00:01:06.577 INFO:teuthology.orchestra.run.smithi193.stdout:Package xfsdump-3.1.12-3.el9.x86_64 is already installed. 2024-03-24T00:01:06.578 INFO:teuthology.orchestra.run.smithi193.stdout:Package xfsprogs-5.19.0-4.el9.x86_64 is already installed. 2024-03-24T00:01:06.587 INFO:teuthology.orchestra.run.smithi118.stdout:Package NetworkManager-1:1.44.0-2.el9.x86_64 is already installed. 2024-03-24T00:01:06.588 INFO:teuthology.orchestra.run.smithi118.stdout:Package iproute-6.2.0-5.el9.x86_64 is already installed. 2024-03-24T00:01:06.590 INFO:teuthology.orchestra.run.smithi118.stdout:Package util-linux-2.37.4-15.el9.x86_64 is already installed. 2024-03-24T00:01:06.591 INFO:teuthology.orchestra.run.smithi118.stdout:Package libacl-devel-2.3.1-3.el9.x86_64 is already installed. 2024-03-24T00:01:06.594 INFO:teuthology.orchestra.run.smithi118.stdout:Package libattr-devel-2.5.1-3.el9.x86_64 is already installed. 2024-03-24T00:01:06.597 INFO:teuthology.orchestra.run.smithi118.stdout:Package xfsdump-3.1.12-3.el9.x86_64 is already installed. 2024-03-24T00:01:06.599 INFO:teuthology.orchestra.run.smithi118.stdout:Package xfsprogs-5.19.0-4.el9.x86_64 is already installed. 2024-03-24T00:01:06.658 INFO:teuthology.orchestra.run.smithi189.stdout:(8/174): ceph-radosgw-19.0.0-2145.g3be9a5c9.el9 15 MB/s | 9.9 MB 00:00 2024-03-24T00:01:06.658 INFO:teuthology.orchestra.run.smithi193.stdout:Dependencies resolved. 2024-03-24T00:01:06.680 INFO:teuthology.orchestra.run.smithi193.stdout:============================================================================================ 2024-03-24T00:01:06.681 INFO:teuthology.orchestra.run.smithi193.stdout: Package Arch Version Repository Size 2024-03-24T00:01:06.681 INFO:teuthology.orchestra.run.smithi193.stdout:============================================================================================ 2024-03-24T00:01:06.681 INFO:teuthology.orchestra.run.smithi193.stdout:Installing: 2024-03-24T00:01:06.681 INFO:teuthology.orchestra.run.smithi193.stdout: bison x86_64 3.7.4-5.el9 CentOS-AppStream 920 k 2024-03-24T00:01:06.681 INFO:teuthology.orchestra.run.smithi193.stdout: ceph x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 6.6 k 2024-03-24T00:01:06.682 INFO:teuthology.orchestra.run.smithi193.stdout: ceph-base x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 5.4 M 2024-03-24T00:01:06.682 INFO:teuthology.orchestra.run.smithi193.stdout: ceph-fuse x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 890 k 2024-03-24T00:01:06.682 INFO:teuthology.orchestra.run.smithi193.stdout: ceph-immutable-object-cache x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 145 k 2024-03-24T00:01:06.682 INFO:teuthology.orchestra.run.smithi193.stdout: ceph-mgr x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 1.5 M 2024-03-24T00:01:06.682 INFO:teuthology.orchestra.run.smithi193.stdout: ceph-mgr-cephadm noarch 2:19.0.0-2145.g3be9a5c9.el9 ceph-noarch 139 k 2024-03-24T00:01:06.683 INFO:teuthology.orchestra.run.smithi193.stdout: ceph-mgr-dashboard noarch 2:19.0.0-2145.g3be9a5c9.el9 ceph-noarch 1.8 M 2024-03-24T00:01:06.683 INFO:teuthology.orchestra.run.smithi193.stdout: ceph-mgr-diskprediction-local noarch 2:19.0.0-2145.g3be9a5c9.el9 ceph-noarch 7.4 M 2024-03-24T00:01:06.683 INFO:teuthology.orchestra.run.smithi193.stdout: ceph-mgr-rook noarch 2:19.0.0-2145.g3be9a5c9.el9 ceph-noarch 49 k 2024-03-24T00:01:06.683 INFO:teuthology.orchestra.run.smithi193.stdout: ceph-radosgw x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 9.9 M 2024-03-24T00:01:06.683 INFO:teuthology.orchestra.run.smithi193.stdout: ceph-test x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 48 M 2024-03-24T00:01:06.684 INFO:teuthology.orchestra.run.smithi193.stdout: ceph-volume noarch 2:19.0.0-2145.g3be9a5c9.el9 ceph-noarch 264 k 2024-03-24T00:01:06.684 INFO:teuthology.orchestra.run.smithi193.stdout: cephadm noarch 2:19.0.0-2145.g3be9a5c9.el9 ceph-noarch 748 k 2024-03-24T00:01:06.684 INFO:teuthology.orchestra.run.smithi193.stdout: cephfs-mirror x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 220 k 2024-03-24T00:01:06.684 INFO:teuthology.orchestra.run.smithi193.stdout: cephfs-shell x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 27 k 2024-03-24T00:01:06.684 INFO:teuthology.orchestra.run.smithi193.stdout: cephfs-top noarch 2:19.0.0-2145.g3be9a5c9.el9 ceph-noarch 18 k 2024-03-24T00:01:06.684 INFO:teuthology.orchestra.run.smithi193.stdout: elfutils-libelf-devel x86_64 0.190-2.el9 appstream 21 k 2024-03-24T00:01:06.685 INFO:teuthology.orchestra.run.smithi193.stdout: flex x86_64 2.6.4-9.el9 CentOS-AppStream 307 k 2024-03-24T00:01:06.685 INFO:teuthology.orchestra.run.smithi193.stdout: libaio-devel x86_64 0.3.111-13.el9 CentOS-AppStream 10 k 2024-03-24T00:01:06.685 INFO:teuthology.orchestra.run.smithi193.stdout: libcephfs-devel x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 31 k 2024-03-24T00:01:06.685 INFO:teuthology.orchestra.run.smithi193.stdout: libcephfs2 x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 746 k 2024-03-24T00:01:06.685 INFO:teuthology.orchestra.run.smithi193.stdout: librados-devel x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 127 k 2024-03-24T00:01:06.685 INFO:teuthology.orchestra.run.smithi193.stdout: libtool x86_64 2.4.6-45.el9 CentOS-AppStream 578 k 2024-03-24T00:01:06.686 INFO:teuthology.orchestra.run.smithi193.stdout: libuuid-devel x86_64 2.37.4-18.el9 appstream 25 k 2024-03-24T00:01:06.686 INFO:teuthology.orchestra.run.smithi193.stdout: openssl-devel x86_64 1:3.0.7-27.el9 appstream 4.1 M 2024-03-24T00:01:06.686 INFO:teuthology.orchestra.run.smithi193.stdout: postgresql x86_64 13.11-1.el9 CentOS-AppStream 1.5 M 2024-03-24T00:01:06.686 INFO:teuthology.orchestra.run.smithi193.stdout: postgresql-contrib x86_64 13.11-1.el9 CentOS-AppStream 811 k 2024-03-24T00:01:06.686 INFO:teuthology.orchestra.run.smithi193.stdout: postgresql-server x86_64 13.11-1.el9 CentOS-AppStream 5.7 M 2024-03-24T00:01:06.686 INFO:teuthology.orchestra.run.smithi193.stdout: python3-cephfs x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 162 k 2024-03-24T00:01:06.687 INFO:teuthology.orchestra.run.smithi193.stdout: python3-rados x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 321 k 2024-03-24T00:01:06.687 INFO:teuthology.orchestra.run.smithi193.stdout: python3-rbd x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 298 k 2024-03-24T00:01:06.687 INFO:teuthology.orchestra.run.smithi193.stdout: python3-rgw x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 99 k 2024-03-24T00:01:06.687 INFO:teuthology.orchestra.run.smithi193.stdout: rbd-fuse x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 85 k 2024-03-24T00:01:06.687 INFO:teuthology.orchestra.run.smithi193.stdout: rbd-mirror x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 3.1 M 2024-03-24T00:01:06.688 INFO:teuthology.orchestra.run.smithi193.stdout: rbd-nbd x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 170 k 2024-03-24T00:01:06.688 INFO:teuthology.orchestra.run.smithi193.stdout: xfsprogs-devel x86_64 6.3.0-1.el9 appstream 94 k 2024-03-24T00:01:06.688 INFO:teuthology.orchestra.run.smithi193.stdout:Upgrading: 2024-03-24T00:01:06.688 INFO:teuthology.orchestra.run.smithi193.stdout: NetworkManager x86_64 1:1.47.2-1.el9 baseos 2.3 M 2024-03-24T00:01:06.688 INFO:teuthology.orchestra.run.smithi193.stdout: NetworkManager-libnm x86_64 1:1.47.2-1.el9 baseos 1.8 M 2024-03-24T00:01:06.689 INFO:teuthology.orchestra.run.smithi193.stdout: NetworkManager-team x86_64 1:1.47.2-1.el9 baseos 42 k 2024-03-24T00:01:06.689 INFO:teuthology.orchestra.run.smithi193.stdout: NetworkManager-tui x86_64 1:1.47.2-1.el9 baseos 248 k 2024-03-24T00:01:06.689 INFO:teuthology.orchestra.run.smithi193.stdout: acl x86_64 2.3.1-4.el9 baseos 71 k 2024-03-24T00:01:06.689 INFO:teuthology.orchestra.run.smithi193.stdout: elfutils-debuginfod-client x86_64 0.190-2.el9 baseos 36 k 2024-03-24T00:01:06.689 INFO:teuthology.orchestra.run.smithi193.stdout: elfutils-libelf x86_64 0.190-2.el9 baseos 192 k 2024-03-24T00:01:06.690 INFO:teuthology.orchestra.run.smithi193.stdout: elfutils-libs x86_64 0.190-2.el9 baseos 256 k 2024-03-24T00:01:06.690 INFO:teuthology.orchestra.run.smithi193.stdout: libacl x86_64 2.3.1-4.el9 baseos 23 k 2024-03-24T00:01:06.690 INFO:teuthology.orchestra.run.smithi193.stdout: libacl-devel x86_64 2.3.1-4.el9 appstream 74 k 2024-03-24T00:01:06.690 INFO:teuthology.orchestra.run.smithi193.stdout: libblkid x86_64 2.37.4-18.el9 baseos 108 k 2024-03-24T00:01:06.690 INFO:teuthology.orchestra.run.smithi193.stdout: libfdisk x86_64 2.37.4-18.el9 baseos 155 k 2024-03-24T00:01:06.690 INFO:teuthology.orchestra.run.smithi193.stdout: libmount x86_64 2.37.4-18.el9 baseos 136 k 2024-03-24T00:01:06.691 INFO:teuthology.orchestra.run.smithi193.stdout: librados2 x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 3.4 M 2024-03-24T00:01:06.691 INFO:teuthology.orchestra.run.smithi193.stdout: librbd1 x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 3.1 M 2024-03-24T00:01:06.691 INFO:teuthology.orchestra.run.smithi193.stdout: libsmartcols x86_64 2.37.4-18.el9 baseos 64 k 2024-03-24T00:01:06.691 INFO:teuthology.orchestra.run.smithi193.stdout: libuuid x86_64 2.37.4-18.el9 baseos 29 k 2024-03-24T00:01:06.691 INFO:teuthology.orchestra.run.smithi193.stdout: openssl x86_64 1:3.0.7-27.el9 baseos 1.2 M 2024-03-24T00:01:06.692 INFO:teuthology.orchestra.run.smithi193.stdout: openssl-libs x86_64 1:3.0.7-27.el9 baseos 2.2 M 2024-03-24T00:01:06.692 INFO:teuthology.orchestra.run.smithi193.stdout: util-linux x86_64 2.37.4-18.el9 baseos 2.3 M 2024-03-24T00:01:06.692 INFO:teuthology.orchestra.run.smithi193.stdout: util-linux-core x86_64 2.37.4-18.el9 baseos 465 k 2024-03-24T00:01:06.692 INFO:teuthology.orchestra.run.smithi193.stdout: util-linux-user x86_64 2.37.4-18.el9 baseos 31 k 2024-03-24T00:01:06.692 INFO:teuthology.orchestra.run.smithi193.stdout: xfsdump x86_64 3.1.12-4.el9 CentOS-BaseOS 321 k 2024-03-24T00:01:06.692 INFO:teuthology.orchestra.run.smithi193.stdout: xfsprogs x86_64 6.3.0-1.el9 baseos 1.1 M 2024-03-24T00:01:06.693 INFO:teuthology.orchestra.run.smithi193.stdout:Installing dependencies: 2024-03-24T00:01:06.693 INFO:teuthology.orchestra.run.smithi193.stdout: abseil-cpp x86_64 20211102.0-3.el9 epel 548 k 2024-03-24T00:01:06.693 INFO:teuthology.orchestra.run.smithi193.stdout: automake noarch 1.16.2-8.el9 CentOS-AppStream 662 k 2024-03-24T00:01:06.693 INFO:teuthology.orchestra.run.smithi193.stdout: boost-program-options x86_64 1.75.0-8.el9 CentOS-AppStream 106 k 2024-03-24T00:01:06.693 INFO:teuthology.orchestra.run.smithi193.stdout: ceph-common x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 20 M 2024-03-24T00:01:06.693 INFO:teuthology.orchestra.run.smithi193.stdout: ceph-grafana-dashboards noarch 2:19.0.0-2145.g3be9a5c9.el9 ceph-noarch 34 k 2024-03-24T00:01:06.694 INFO:teuthology.orchestra.run.smithi193.stdout: ceph-mds x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 2.3 M 2024-03-24T00:01:06.694 INFO:teuthology.orchestra.run.smithi193.stdout: ceph-mgr-modules-core noarch 2:19.0.0-2145.g3be9a5c9.el9 ceph-noarch 248 k 2024-03-24T00:01:06.694 INFO:teuthology.orchestra.run.smithi193.stdout: ceph-mon x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 4.7 M 2024-03-24T00:01:06.694 INFO:teuthology.orchestra.run.smithi193.stdout: ceph-osd x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 17 M 2024-03-24T00:01:06.694 INFO:teuthology.orchestra.run.smithi193.stdout: ceph-prometheus-alerts noarch 2:19.0.0-2145.g3be9a5c9.el9 ceph-noarch 16 k 2024-03-24T00:01:06.694 INFO:teuthology.orchestra.run.smithi193.stdout: ceph-selinux x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 25 k 2024-03-24T00:01:06.695 INFO:teuthology.orchestra.run.smithi193.stdout: flexiblas x86_64 3.0.4-8.el9 CentOS-AppStream 32 k 2024-03-24T00:01:06.695 INFO:teuthology.orchestra.run.smithi193.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 CentOS-AppStream 3.0 M 2024-03-24T00:01:06.695 INFO:teuthology.orchestra.run.smithi193.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 CentOS-AppStream 17 k 2024-03-24T00:01:06.695 INFO:teuthology.orchestra.run.smithi193.stdout: fuse x86_64 2.9.9-15.el9 CentOS-BaseOS 79 k 2024-03-24T00:01:06.695 INFO:teuthology.orchestra.run.smithi193.stdout: gperftools-libs x86_64 2.9.1-2.el9 epel 309 k 2024-03-24T00:01:06.696 INFO:teuthology.orchestra.run.smithi193.stdout: grpc-data noarch 1.46.7-10.el9 epel 19 k 2024-03-24T00:01:06.696 INFO:teuthology.orchestra.run.smithi193.stdout: jq x86_64 1.6-16.el9 baseos 187 k 2024-03-24T00:01:06.696 INFO:teuthology.orchestra.run.smithi193.stdout: libarrow x86_64 9.0.0-10.el9 epel 4.4 M 2024-03-24T00:01:06.696 INFO:teuthology.orchestra.run.smithi193.stdout: libarrow-doc noarch 9.0.0-10.el9 epel 26 k 2024-03-24T00:01:06.696 INFO:teuthology.orchestra.run.smithi193.stdout: libcephsqlite x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 163 k 2024-03-24T00:01:06.696 INFO:teuthology.orchestra.run.smithi193.stdout: libgfortran x86_64 11.4.1-3.el9 baseos 803 k 2024-03-24T00:01:06.697 INFO:teuthology.orchestra.run.smithi193.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-03-24T00:01:06.697 INFO:teuthology.orchestra.run.smithi193.stdout: libpmemobj x86_64 1.12.1-1.el9 CentOS-AppStream 159 k 2024-03-24T00:01:06.697 INFO:teuthology.orchestra.run.smithi193.stdout: libquadmath x86_64 11.4.1-3.el9 baseos 195 k 2024-03-24T00:01:06.697 INFO:teuthology.orchestra.run.smithi193.stdout: librabbitmq x86_64 0.11.0-7.el9 CentOS-AppStream 44 k 2024-03-24T00:01:06.697 INFO:teuthology.orchestra.run.smithi193.stdout: libradosstriper1 x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 501 k 2024-03-24T00:01:06.698 INFO:teuthology.orchestra.run.smithi193.stdout: librdkafka x86_64 1.6.1-102.el9 CentOS-AppStream 661 k 2024-03-24T00:01:06.698 INFO:teuthology.orchestra.run.smithi193.stdout: librgw2 x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 5.1 M 2024-03-24T00:01:06.698 INFO:teuthology.orchestra.run.smithi193.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-03-24T00:01:06.698 INFO:teuthology.orchestra.run.smithi193.stdout: libzstd-devel x86_64 1.5.1-2.el9 CentOS-AppStream 46 k 2024-03-24T00:01:06.698 INFO:teuthology.orchestra.run.smithi193.stdout: lttng-ust x86_64 2.12.0-6.el9 CentOS-AppStream 282 k 2024-03-24T00:01:06.698 INFO:teuthology.orchestra.run.smithi193.stdout: lua x86_64 5.4.4-4.el9 CentOS-AppStream 187 k 2024-03-24T00:01:06.699 INFO:teuthology.orchestra.run.smithi193.stdout: lua-devel x86_64 5.4.4-4.el9 crb 22 k 2024-03-24T00:01:06.699 INFO:teuthology.orchestra.run.smithi193.stdout: luarocks noarch 3.9.2-1.el9 epel 151 k 2024-03-24T00:01:06.699 INFO:teuthology.orchestra.run.smithi193.stdout: oniguruma x86_64 6.9.6-1.el9.5 CentOS-AppStream 217 k 2024-03-24T00:01:06.699 INFO:teuthology.orchestra.run.smithi193.stdout: openblas x86_64 0.3.21-2.el9 CentOS-AppStream 34 k 2024-03-24T00:01:06.699 INFO:teuthology.orchestra.run.smithi193.stdout: openblas-openmp x86_64 0.3.21-2.el9 CentOS-AppStream 4.7 M 2024-03-24T00:01:06.699 INFO:teuthology.orchestra.run.smithi193.stdout: parquet-libs x86_64 9.0.0-10.el9 epel 838 k 2024-03-24T00:01:06.700 INFO:teuthology.orchestra.run.smithi193.stdout: postgresql-private-libs x86_64 13.11-1.el9 CentOS-AppStream 134 k 2024-03-24T00:01:06.700 INFO:teuthology.orchestra.run.smithi193.stdout: protobuf x86_64 3.14.0-13.el9 CentOS-AppStream 1.0 M 2024-03-24T00:01:06.700 INFO:teuthology.orchestra.run.smithi193.stdout: protobuf-compiler x86_64 3.14.0-13.el9 crb 863 k 2024-03-24T00:01:06.700 INFO:teuthology.orchestra.run.smithi193.stdout: python3-asyncssh noarch 2.13.2-5.el9 epel 548 k 2024-03-24T00:01:06.700 INFO:teuthology.orchestra.run.smithi193.stdout: python3-attrs noarch 20.3.0-7.el9 CentOS-AppStream 78 k 2024-03-24T00:01:06.701 INFO:teuthology.orchestra.run.smithi193.stdout: python3-babel noarch 2.9.1-2.el9 CentOS-AppStream 5.8 M 2024-03-24T00:01:06.701 INFO:teuthology.orchestra.run.smithi193.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-03-24T00:01:06.701 INFO:teuthology.orchestra.run.smithi193.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-03-24T00:01:06.701 INFO:teuthology.orchestra.run.smithi193.stdout: python3-ceph-argparse x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 45 k 2024-03-24T00:01:06.701 INFO:teuthology.orchestra.run.smithi193.stdout: python3-ceph-common x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 131 k 2024-03-24T00:01:06.701 INFO:teuthology.orchestra.run.smithi193.stdout: python3-certifi noarch 2021.10.8-2.el9 epel 15 k 2024-03-24T00:01:06.702 INFO:teuthology.orchestra.run.smithi193.stdout: python3-cffi x86_64 1.14.5-5.el9 CentOS-BaseOS 241 k 2024-03-24T00:01:06.702 INFO:teuthology.orchestra.run.smithi193.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-03-24T00:01:06.702 INFO:teuthology.orchestra.run.smithi193.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-03-24T00:01:06.702 INFO:teuthology.orchestra.run.smithi193.stdout: python3-cmd2 noarch 2.4.2-2.el9 epel 303 k 2024-03-24T00:01:06.702 INFO:teuthology.orchestra.run.smithi193.stdout: python3-colorama noarch 0.4.4-12.el9 epel 35 k 2024-03-24T00:01:06.702 INFO:teuthology.orchestra.run.smithi193.stdout: python3-cryptography x86_64 36.0.1-4.el9 CentOS-BaseOS 1.1 M 2024-03-24T00:01:06.703 INFO:teuthology.orchestra.run.smithi193.stdout: python3-devel x86_64 3.9.17-2.el9 appstream 245 k 2024-03-24T00:01:06.703 INFO:teuthology.orchestra.run.smithi193.stdout: python3-google-auth noarch 1:2.28.2-1.el9 epel 203 k 2024-03-24T00:01:06.703 INFO:teuthology.orchestra.run.smithi193.stdout: python3-grpcio x86_64 1.46.7-10.el9 epel 2.0 M 2024-03-24T00:01:06.703 INFO:teuthology.orchestra.run.smithi193.stdout: python3-grpcio-tools x86_64 1.46.7-10.el9 epel 144 k 2024-03-24T00:01:06.703 INFO:teuthology.orchestra.run.smithi193.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-03-24T00:01:06.704 INFO:teuthology.orchestra.run.smithi193.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-03-24T00:01:06.704 INFO:teuthology.orchestra.run.smithi193.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2024-03-24T00:01:06.704 INFO:teuthology.orchestra.run.smithi193.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-03-24T00:01:06.704 INFO:teuthology.orchestra.run.smithi193.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-03-24T00:01:06.704 INFO:teuthology.orchestra.run.smithi193.stdout: python3-jinja2 noarch 2.11.3-5.el9 appstream 248 k 2024-03-24T00:01:06.704 INFO:teuthology.orchestra.run.smithi193.stdout: python3-jsonpatch noarch 1.21-16.el9 CentOS-AppStream 24 k 2024-03-24T00:01:06.705 INFO:teuthology.orchestra.run.smithi193.stdout: python3-jsonpointer noarch 2.0-4.el9 CentOS-AppStream 18 k 2024-03-24T00:01:06.705 INFO:teuthology.orchestra.run.smithi193.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-03-24T00:01:06.705 INFO:teuthology.orchestra.run.smithi193.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-03-24T00:01:06.705 INFO:teuthology.orchestra.run.smithi193.stdout: python3-mako noarch 1.1.4-6.el9 CentOS-AppStream 144 k 2024-03-24T00:01:06.705 INFO:teuthology.orchestra.run.smithi193.stdout: python3-markupsafe x86_64 1.1.1-12.el9 CentOS-AppStream 32 k 2024-03-24T00:01:06.706 INFO:teuthology.orchestra.run.smithi193.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-03-24T00:01:06.706 INFO:teuthology.orchestra.run.smithi193.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-03-24T00:01:06.706 INFO:teuthology.orchestra.run.smithi193.stdout: python3-numpy x86_64 1:1.20.1-5.el9 CentOS-AppStream 4.8 M 2024-03-24T00:01:06.706 INFO:teuthology.orchestra.run.smithi193.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 CentOS-AppStream 217 k 2024-03-24T00:01:06.706 INFO:teuthology.orchestra.run.smithi193.stdout: python3-oauthlib noarch 3.1.1-5.el9 CentOS-AppStream 174 k 2024-03-24T00:01:06.706 INFO:teuthology.orchestra.run.smithi193.stdout: python3-packaging noarch 20.9-5.el9 CentOS-AppStream 69 k 2024-03-24T00:01:06.706 INFO:teuthology.orchestra.run.smithi193.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-03-24T00:01:06.706 INFO:teuthology.orchestra.run.smithi193.stdout: python3-ply noarch 3.11-14.el9 CentOS-BaseOS 103 k 2024-03-24T00:01:06.707 INFO:teuthology.orchestra.run.smithi193.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-03-24T00:01:06.707 INFO:teuthology.orchestra.run.smithi193.stdout: python3-prettytable noarch 0.7.2-27.el9 CentOS-AppStream 41 k 2024-03-24T00:01:06.707 INFO:teuthology.orchestra.run.smithi193.stdout: python3-protobuf noarch 3.14.0-13.el9 CentOS-AppStream 238 k 2024-03-24T00:01:06.707 INFO:teuthology.orchestra.run.smithi193.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-03-24T00:01:06.707 INFO:teuthology.orchestra.run.smithi193.stdout: python3-pyasn1 noarch 0.4.8-6.el9 CentOS-AppStream 132 k 2024-03-24T00:01:06.707 INFO:teuthology.orchestra.run.smithi193.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 CentOS-AppStream 211 k 2024-03-24T00:01:06.707 INFO:teuthology.orchestra.run.smithi193.stdout: python3-pycparser noarch 2.20-6.el9 CentOS-BaseOS 124 k 2024-03-24T00:01:06.707 INFO:teuthology.orchestra.run.smithi193.stdout: python3-pyperclip noarch 1.8.2-2.el9 epel 28 k 2024-03-24T00:01:06.707 INFO:teuthology.orchestra.run.smithi193.stdout: python3-pytz noarch 2021.1-5.el9 CentOS-AppStream 47 k 2024-03-24T00:01:06.708 INFO:teuthology.orchestra.run.smithi193.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-03-24T00:01:06.708 INFO:teuthology.orchestra.run.smithi193.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-03-24T00:01:06.708 INFO:teuthology.orchestra.run.smithi193.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 CentOS-AppStream 43 k 2024-03-24T00:01:06.708 INFO:teuthology.orchestra.run.smithi193.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-03-24T00:01:06.708 INFO:teuthology.orchestra.run.smithi193.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-03-24T00:01:06.708 INFO:teuthology.orchestra.run.smithi193.stdout: python3-scipy x86_64 1.6.2-8.el9 CentOS-AppStream 16 M 2024-03-24T00:01:06.708 INFO:teuthology.orchestra.run.smithi193.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-03-24T00:01:06.708 INFO:teuthology.orchestra.run.smithi193.stdout: python3-toml noarch 0.10.2-6.el9 CentOS-AppStream 39 k 2024-03-24T00:01:06.708 INFO:teuthology.orchestra.run.smithi193.stdout: python3-typing-extensions noarch 4.4.0-2.el9 epel 51 k 2024-03-24T00:01:06.708 INFO:teuthology.orchestra.run.smithi193.stdout: python3-urllib3 noarch 1.26.5-5.el9 CentOS-BaseOS 187 k 2024-03-24T00:01:06.709 INFO:teuthology.orchestra.run.smithi193.stdout: python3-wcwidth noarch 0.2.5-8.el9 CentOS-AppStream 41 k 2024-03-24T00:01:06.709 INFO:teuthology.orchestra.run.smithi193.stdout: python3-webob noarch 1.8.7-6.el9 epel 230 k 2024-03-24T00:01:06.709 INFO:teuthology.orchestra.run.smithi193.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-03-24T00:01:06.709 INFO:teuthology.orchestra.run.smithi193.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-03-24T00:01:06.709 INFO:teuthology.orchestra.run.smithi193.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-03-24T00:01:06.709 INFO:teuthology.orchestra.run.smithi193.stdout: qatlib x86_64 23.11.0-1.el9 appstream 216 k 2024-03-24T00:01:06.709 INFO:teuthology.orchestra.run.smithi193.stdout: qatzip-libs x86_64 1.1.2-2.el9 appstream 44 k 2024-03-24T00:01:06.709 INFO:teuthology.orchestra.run.smithi193.stdout: re2 x86_64 1:20211101-3.el9 epel 192 k 2024-03-24T00:01:06.709 INFO:teuthology.orchestra.run.smithi193.stdout: socat x86_64 1.7.4.1-5.el9 CentOS-AppStream 300 k 2024-03-24T00:01:06.710 INFO:teuthology.orchestra.run.smithi193.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-03-24T00:01:06.710 INFO:teuthology.orchestra.run.smithi193.stdout: uuid x86_64 1.6.2-55.el9 CentOS-AppStream 56 k 2024-03-24T00:01:06.710 INFO:teuthology.orchestra.run.smithi193.stdout: xmlstarlet x86_64 1.6.1-20.el9 CentOS-AppStream 63 k 2024-03-24T00:01:06.710 INFO:teuthology.orchestra.run.smithi193.stdout: zlib-devel x86_64 1.2.11-40.el9 CentOS-AppStream 44 k 2024-03-24T00:01:06.710 INFO:teuthology.orchestra.run.smithi193.stdout:Installing weak dependencies: 2024-03-24T00:01:06.710 INFO:teuthology.orchestra.run.smithi193.stdout: qatlib-service x86_64 23.11.0-1.el9 appstream 35 k 2024-03-24T00:01:06.710 INFO:teuthology.orchestra.run.smithi193.stdout: 2024-03-24T00:01:06.710 INFO:teuthology.orchestra.run.smithi193.stdout:Transaction Summary 2024-03-24T00:01:06.710 INFO:teuthology.orchestra.run.smithi193.stdout:============================================================================================ 2024-03-24T00:01:06.711 INFO:teuthology.orchestra.run.smithi193.stdout:Install 150 Packages 2024-03-24T00:01:06.711 INFO:teuthology.orchestra.run.smithi193.stdout:Upgrade 24 Packages 2024-03-24T00:01:06.711 INFO:teuthology.orchestra.run.smithi193.stdout: 2024-03-24T00:01:06.712 INFO:teuthology.orchestra.run.smithi118.stdout:Dependencies resolved. 2024-03-24T00:01:06.712 INFO:teuthology.orchestra.run.smithi193.stdout:Total download size: 224 M 2024-03-24T00:01:06.712 INFO:teuthology.orchestra.run.smithi193.stdout:Downloading Packages: 2024-03-24T00:01:06.713 INFO:teuthology.orchestra.run.smithi118.stdout:============================================================================================ 2024-03-24T00:01:06.713 INFO:teuthology.orchestra.run.smithi118.stdout: Package Arch Version Repository Size 2024-03-24T00:01:06.713 INFO:teuthology.orchestra.run.smithi118.stdout:============================================================================================ 2024-03-24T00:01:06.713 INFO:teuthology.orchestra.run.smithi118.stdout:Installing: 2024-03-24T00:01:06.713 INFO:teuthology.orchestra.run.smithi118.stdout: bison x86_64 3.7.4-5.el9 CentOS-AppStream 920 k 2024-03-24T00:01:06.713 INFO:teuthology.orchestra.run.smithi118.stdout: ceph x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 6.6 k 2024-03-24T00:01:06.713 INFO:teuthology.orchestra.run.smithi118.stdout: ceph-base x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 5.4 M 2024-03-24T00:01:06.714 INFO:teuthology.orchestra.run.smithi118.stdout: ceph-fuse x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 890 k 2024-03-24T00:01:06.714 INFO:teuthology.orchestra.run.smithi118.stdout: ceph-immutable-object-cache x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 145 k 2024-03-24T00:01:06.714 INFO:teuthology.orchestra.run.smithi118.stdout: ceph-mgr x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 1.5 M 2024-03-24T00:01:06.714 INFO:teuthology.orchestra.run.smithi118.stdout: ceph-mgr-cephadm noarch 2:19.0.0-2145.g3be9a5c9.el9 ceph-noarch 139 k 2024-03-24T00:01:06.714 INFO:teuthology.orchestra.run.smithi118.stdout: ceph-mgr-dashboard noarch 2:19.0.0-2145.g3be9a5c9.el9 ceph-noarch 1.8 M 2024-03-24T00:01:06.714 INFO:teuthology.orchestra.run.smithi118.stdout: ceph-mgr-diskprediction-local noarch 2:19.0.0-2145.g3be9a5c9.el9 ceph-noarch 7.4 M 2024-03-24T00:01:06.714 INFO:teuthology.orchestra.run.smithi118.stdout: ceph-mgr-rook noarch 2:19.0.0-2145.g3be9a5c9.el9 ceph-noarch 49 k 2024-03-24T00:01:06.714 INFO:teuthology.orchestra.run.smithi118.stdout: ceph-radosgw x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 9.9 M 2024-03-24T00:01:06.715 INFO:teuthology.orchestra.run.smithi118.stdout: ceph-test x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 48 M 2024-03-24T00:01:06.715 INFO:teuthology.orchestra.run.smithi118.stdout: ceph-volume noarch 2:19.0.0-2145.g3be9a5c9.el9 ceph-noarch 264 k 2024-03-24T00:01:06.715 INFO:teuthology.orchestra.run.smithi118.stdout: cephadm noarch 2:19.0.0-2145.g3be9a5c9.el9 ceph-noarch 748 k 2024-03-24T00:01:06.715 INFO:teuthology.orchestra.run.smithi118.stdout: cephfs-mirror x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 220 k 2024-03-24T00:01:06.715 INFO:teuthology.orchestra.run.smithi118.stdout: cephfs-shell x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 27 k 2024-03-24T00:01:06.715 INFO:teuthology.orchestra.run.smithi118.stdout: cephfs-top noarch 2:19.0.0-2145.g3be9a5c9.el9 ceph-noarch 18 k 2024-03-24T00:01:06.715 INFO:teuthology.orchestra.run.smithi118.stdout: elfutils-libelf-devel x86_64 0.190-2.el9 appstream 21 k 2024-03-24T00:01:06.715 INFO:teuthology.orchestra.run.smithi118.stdout: flex x86_64 2.6.4-9.el9 CentOS-AppStream 307 k 2024-03-24T00:01:06.715 INFO:teuthology.orchestra.run.smithi118.stdout: libaio-devel x86_64 0.3.111-13.el9 CentOS-AppStream 10 k 2024-03-24T00:01:06.716 INFO:teuthology.orchestra.run.smithi118.stdout: libcephfs-devel x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 31 k 2024-03-24T00:01:06.716 INFO:teuthology.orchestra.run.smithi118.stdout: libcephfs2 x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 746 k 2024-03-24T00:01:06.716 INFO:teuthology.orchestra.run.smithi118.stdout: librados-devel x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 127 k 2024-03-24T00:01:06.716 INFO:teuthology.orchestra.run.smithi118.stdout: libtool x86_64 2.4.6-45.el9 CentOS-AppStream 578 k 2024-03-24T00:01:06.716 INFO:teuthology.orchestra.run.smithi118.stdout: libuuid-devel x86_64 2.37.4-18.el9 appstream 25 k 2024-03-24T00:01:06.716 INFO:teuthology.orchestra.run.smithi118.stdout: openssl-devel x86_64 1:3.0.7-27.el9 appstream 4.1 M 2024-03-24T00:01:06.716 INFO:teuthology.orchestra.run.smithi118.stdout: postgresql x86_64 13.11-1.el9 CentOS-AppStream 1.5 M 2024-03-24T00:01:06.716 INFO:teuthology.orchestra.run.smithi118.stdout: postgresql-contrib x86_64 13.11-1.el9 CentOS-AppStream 811 k 2024-03-24T00:01:06.716 INFO:teuthology.orchestra.run.smithi118.stdout: postgresql-server x86_64 13.11-1.el9 CentOS-AppStream 5.7 M 2024-03-24T00:01:06.716 INFO:teuthology.orchestra.run.smithi118.stdout: python3-cephfs x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 162 k 2024-03-24T00:01:06.717 INFO:teuthology.orchestra.run.smithi118.stdout: python3-rados x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 321 k 2024-03-24T00:01:06.717 INFO:teuthology.orchestra.run.smithi118.stdout: python3-rbd x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 298 k 2024-03-24T00:01:06.717 INFO:teuthology.orchestra.run.smithi118.stdout: python3-rgw x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 99 k 2024-03-24T00:01:06.717 INFO:teuthology.orchestra.run.smithi118.stdout: rbd-fuse x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 85 k 2024-03-24T00:01:06.717 INFO:teuthology.orchestra.run.smithi118.stdout: rbd-mirror x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 3.1 M 2024-03-24T00:01:06.717 INFO:teuthology.orchestra.run.smithi118.stdout: rbd-nbd x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 170 k 2024-03-24T00:01:06.717 INFO:teuthology.orchestra.run.smithi118.stdout: xfsprogs-devel x86_64 6.3.0-1.el9 appstream 94 k 2024-03-24T00:01:06.717 INFO:teuthology.orchestra.run.smithi118.stdout:Upgrading: 2024-03-24T00:01:06.718 INFO:teuthology.orchestra.run.smithi118.stdout: NetworkManager x86_64 1:1.47.2-1.el9 baseos 2.3 M 2024-03-24T00:01:06.718 INFO:teuthology.orchestra.run.smithi118.stdout: NetworkManager-libnm x86_64 1:1.47.2-1.el9 baseos 1.8 M 2024-03-24T00:01:06.718 INFO:teuthology.orchestra.run.smithi118.stdout: NetworkManager-team x86_64 1:1.47.2-1.el9 baseos 42 k 2024-03-24T00:01:06.718 INFO:teuthology.orchestra.run.smithi118.stdout: NetworkManager-tui x86_64 1:1.47.2-1.el9 baseos 248 k 2024-03-24T00:01:06.718 INFO:teuthology.orchestra.run.smithi118.stdout: acl x86_64 2.3.1-4.el9 baseos 71 k 2024-03-24T00:01:06.718 INFO:teuthology.orchestra.run.smithi118.stdout: elfutils-debuginfod-client x86_64 0.190-2.el9 baseos 36 k 2024-03-24T00:01:06.719 INFO:teuthology.orchestra.run.smithi118.stdout: elfutils-libelf x86_64 0.190-2.el9 baseos 192 k 2024-03-24T00:01:06.719 INFO:teuthology.orchestra.run.smithi118.stdout: elfutils-libs x86_64 0.190-2.el9 baseos 256 k 2024-03-24T00:01:06.719 INFO:teuthology.orchestra.run.smithi118.stdout: libacl x86_64 2.3.1-4.el9 baseos 23 k 2024-03-24T00:01:06.719 INFO:teuthology.orchestra.run.smithi118.stdout: libacl-devel x86_64 2.3.1-4.el9 appstream 74 k 2024-03-24T00:01:06.719 INFO:teuthology.orchestra.run.smithi118.stdout: libblkid x86_64 2.37.4-18.el9 baseos 108 k 2024-03-24T00:01:06.719 INFO:teuthology.orchestra.run.smithi118.stdout: libfdisk x86_64 2.37.4-18.el9 baseos 155 k 2024-03-24T00:01:06.720 INFO:teuthology.orchestra.run.smithi118.stdout: libmount x86_64 2.37.4-18.el9 baseos 136 k 2024-03-24T00:01:06.720 INFO:teuthology.orchestra.run.smithi118.stdout: librados2 x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 3.4 M 2024-03-24T00:01:06.720 INFO:teuthology.orchestra.run.smithi118.stdout: librbd1 x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 3.1 M 2024-03-24T00:01:06.720 INFO:teuthology.orchestra.run.smithi118.stdout: libsmartcols x86_64 2.37.4-18.el9 baseos 64 k 2024-03-24T00:01:06.720 INFO:teuthology.orchestra.run.smithi118.stdout: libuuid x86_64 2.37.4-18.el9 baseos 29 k 2024-03-24T00:01:06.721 INFO:teuthology.orchestra.run.smithi118.stdout: openssl x86_64 1:3.0.7-27.el9 baseos 1.2 M 2024-03-24T00:01:06.721 INFO:teuthology.orchestra.run.smithi118.stdout: openssl-libs x86_64 1:3.0.7-27.el9 baseos 2.2 M 2024-03-24T00:01:06.721 INFO:teuthology.orchestra.run.smithi118.stdout: util-linux x86_64 2.37.4-18.el9 baseos 2.3 M 2024-03-24T00:01:06.721 INFO:teuthology.orchestra.run.smithi118.stdout: util-linux-core x86_64 2.37.4-18.el9 baseos 465 k 2024-03-24T00:01:06.721 INFO:teuthology.orchestra.run.smithi118.stdout: util-linux-user x86_64 2.37.4-18.el9 baseos 31 k 2024-03-24T00:01:06.721 INFO:teuthology.orchestra.run.smithi118.stdout: xfsdump x86_64 3.1.12-4.el9 CentOS-BaseOS 321 k 2024-03-24T00:01:06.722 INFO:teuthology.orchestra.run.smithi118.stdout: xfsprogs x86_64 6.3.0-1.el9 baseos 1.1 M 2024-03-24T00:01:06.722 INFO:teuthology.orchestra.run.smithi118.stdout:Installing dependencies: 2024-03-24T00:01:06.722 INFO:teuthology.orchestra.run.smithi118.stdout: abseil-cpp x86_64 20211102.0-3.el9 epel 548 k 2024-03-24T00:01:06.722 INFO:teuthology.orchestra.run.smithi118.stdout: automake noarch 1.16.2-8.el9 CentOS-AppStream 662 k 2024-03-24T00:01:06.722 INFO:teuthology.orchestra.run.smithi118.stdout: boost-program-options x86_64 1.75.0-8.el9 CentOS-AppStream 106 k 2024-03-24T00:01:06.723 INFO:teuthology.orchestra.run.smithi118.stdout: ceph-common x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 20 M 2024-03-24T00:01:06.723 INFO:teuthology.orchestra.run.smithi118.stdout: ceph-grafana-dashboards noarch 2:19.0.0-2145.g3be9a5c9.el9 ceph-noarch 34 k 2024-03-24T00:01:06.723 INFO:teuthology.orchestra.run.smithi118.stdout: ceph-mds x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 2.3 M 2024-03-24T00:01:06.723 INFO:teuthology.orchestra.run.smithi118.stdout: ceph-mgr-modules-core noarch 2:19.0.0-2145.g3be9a5c9.el9 ceph-noarch 248 k 2024-03-24T00:01:06.723 INFO:teuthology.orchestra.run.smithi118.stdout: ceph-mon x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 4.7 M 2024-03-24T00:01:06.723 INFO:teuthology.orchestra.run.smithi118.stdout: ceph-osd x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 17 M 2024-03-24T00:01:06.724 INFO:teuthology.orchestra.run.smithi118.stdout: ceph-prometheus-alerts noarch 2:19.0.0-2145.g3be9a5c9.el9 ceph-noarch 16 k 2024-03-24T00:01:06.724 INFO:teuthology.orchestra.run.smithi118.stdout: ceph-selinux x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 25 k 2024-03-24T00:01:06.724 INFO:teuthology.orchestra.run.smithi118.stdout: flexiblas x86_64 3.0.4-8.el9 CentOS-AppStream 32 k 2024-03-24T00:01:06.724 INFO:teuthology.orchestra.run.smithi118.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 CentOS-AppStream 3.0 M 2024-03-24T00:01:06.724 INFO:teuthology.orchestra.run.smithi118.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 CentOS-AppStream 17 k 2024-03-24T00:01:06.724 INFO:teuthology.orchestra.run.smithi118.stdout: fuse x86_64 2.9.9-15.el9 CentOS-BaseOS 79 k 2024-03-24T00:01:06.724 INFO:teuthology.orchestra.run.smithi118.stdout: gperftools-libs x86_64 2.9.1-2.el9 epel 309 k 2024-03-24T00:01:06.725 INFO:teuthology.orchestra.run.smithi118.stdout: grpc-data noarch 1.46.7-10.el9 epel 19 k 2024-03-24T00:01:06.725 INFO:teuthology.orchestra.run.smithi118.stdout: jq x86_64 1.6-16.el9 baseos 187 k 2024-03-24T00:01:06.725 INFO:teuthology.orchestra.run.smithi118.stdout: libarrow x86_64 9.0.0-10.el9 epel 4.4 M 2024-03-24T00:01:06.725 INFO:teuthology.orchestra.run.smithi118.stdout: libarrow-doc noarch 9.0.0-10.el9 epel 26 k 2024-03-24T00:01:06.725 INFO:teuthology.orchestra.run.smithi118.stdout: libcephsqlite x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 163 k 2024-03-24T00:01:06.725 INFO:teuthology.orchestra.run.smithi118.stdout: libgfortran x86_64 11.4.1-3.el9 baseos 803 k 2024-03-24T00:01:06.726 INFO:teuthology.orchestra.run.smithi118.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-03-24T00:01:06.726 INFO:teuthology.orchestra.run.smithi118.stdout: libpmemobj x86_64 1.12.1-1.el9 CentOS-AppStream 159 k 2024-03-24T00:01:06.726 INFO:teuthology.orchestra.run.smithi118.stdout: libquadmath x86_64 11.4.1-3.el9 baseos 195 k 2024-03-24T00:01:06.726 INFO:teuthology.orchestra.run.smithi118.stdout: librabbitmq x86_64 0.11.0-7.el9 CentOS-AppStream 44 k 2024-03-24T00:01:06.726 INFO:teuthology.orchestra.run.smithi118.stdout: libradosstriper1 x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 501 k 2024-03-24T00:01:06.726 INFO:teuthology.orchestra.run.smithi118.stdout: librdkafka x86_64 1.6.1-102.el9 CentOS-AppStream 661 k 2024-03-24T00:01:06.727 INFO:teuthology.orchestra.run.smithi118.stdout: librgw2 x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 5.1 M 2024-03-24T00:01:06.727 INFO:teuthology.orchestra.run.smithi118.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-03-24T00:01:06.727 INFO:teuthology.orchestra.run.smithi118.stdout: libzstd-devel x86_64 1.5.1-2.el9 CentOS-AppStream 46 k 2024-03-24T00:01:06.727 INFO:teuthology.orchestra.run.smithi118.stdout: lttng-ust x86_64 2.12.0-6.el9 CentOS-AppStream 282 k 2024-03-24T00:01:06.727 INFO:teuthology.orchestra.run.smithi118.stdout: lua x86_64 5.4.4-4.el9 CentOS-AppStream 187 k 2024-03-24T00:01:06.727 INFO:teuthology.orchestra.run.smithi118.stdout: lua-devel x86_64 5.4.4-4.el9 crb 22 k 2024-03-24T00:01:06.728 INFO:teuthology.orchestra.run.smithi118.stdout: luarocks noarch 3.9.2-1.el9 epel 151 k 2024-03-24T00:01:06.728 INFO:teuthology.orchestra.run.smithi118.stdout: oniguruma x86_64 6.9.6-1.el9.5 CentOS-AppStream 217 k 2024-03-24T00:01:06.728 INFO:teuthology.orchestra.run.smithi118.stdout: openblas x86_64 0.3.21-2.el9 CentOS-AppStream 34 k 2024-03-24T00:01:06.728 INFO:teuthology.orchestra.run.smithi118.stdout: openblas-openmp x86_64 0.3.21-2.el9 CentOS-AppStream 4.7 M 2024-03-24T00:01:06.728 INFO:teuthology.orchestra.run.smithi118.stdout: parquet-libs x86_64 9.0.0-10.el9 epel 838 k 2024-03-24T00:01:06.729 INFO:teuthology.orchestra.run.smithi118.stdout: postgresql-private-libs x86_64 13.11-1.el9 CentOS-AppStream 134 k 2024-03-24T00:01:06.729 INFO:teuthology.orchestra.run.smithi118.stdout: protobuf x86_64 3.14.0-13.el9 CentOS-AppStream 1.0 M 2024-03-24T00:01:06.729 INFO:teuthology.orchestra.run.smithi118.stdout: protobuf-compiler x86_64 3.14.0-13.el9 crb 863 k 2024-03-24T00:01:06.729 INFO:teuthology.orchestra.run.smithi118.stdout: python3-asyncssh noarch 2.13.2-5.el9 epel 548 k 2024-03-24T00:01:06.729 INFO:teuthology.orchestra.run.smithi118.stdout: python3-attrs noarch 20.3.0-7.el9 CentOS-AppStream 78 k 2024-03-24T00:01:06.729 INFO:teuthology.orchestra.run.smithi118.stdout: python3-babel noarch 2.9.1-2.el9 CentOS-AppStream 5.8 M 2024-03-24T00:01:06.730 INFO:teuthology.orchestra.run.smithi118.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-03-24T00:01:06.730 INFO:teuthology.orchestra.run.smithi118.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-03-24T00:01:06.730 INFO:teuthology.orchestra.run.smithi118.stdout: python3-ceph-argparse x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 45 k 2024-03-24T00:01:06.730 INFO:teuthology.orchestra.run.smithi118.stdout: python3-ceph-common x86_64 2:19.0.0-2145.g3be9a5c9.el9 ceph 131 k 2024-03-24T00:01:06.730 INFO:teuthology.orchestra.run.smithi118.stdout: python3-certifi noarch 2021.10.8-2.el9 epel 15 k 2024-03-24T00:01:06.731 INFO:teuthology.orchestra.run.smithi118.stdout: python3-cffi x86_64 1.14.5-5.el9 CentOS-BaseOS 241 k 2024-03-24T00:01:06.731 INFO:teuthology.orchestra.run.smithi118.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-03-24T00:01:06.731 INFO:teuthology.orchestra.run.smithi118.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-03-24T00:01:06.731 INFO:teuthology.orchestra.run.smithi118.stdout: python3-cmd2 noarch 2.4.2-2.el9 epel 303 k 2024-03-24T00:01:06.731 INFO:teuthology.orchestra.run.smithi118.stdout: python3-colorama noarch 0.4.4-12.el9 epel 35 k 2024-03-24T00:01:06.731 INFO:teuthology.orchestra.run.smithi118.stdout: python3-cryptography x86_64 36.0.1-4.el9 CentOS-BaseOS 1.1 M 2024-03-24T00:01:06.732 INFO:teuthology.orchestra.run.smithi118.stdout: python3-devel x86_64 3.9.17-2.el9 appstream 245 k 2024-03-24T00:01:06.732 INFO:teuthology.orchestra.run.smithi118.stdout: python3-google-auth noarch 1:2.28.2-1.el9 epel 203 k 2024-03-24T00:01:06.732 INFO:teuthology.orchestra.run.smithi118.stdout: python3-grpcio x86_64 1.46.7-10.el9 epel 2.0 M 2024-03-24T00:01:06.732 INFO:teuthology.orchestra.run.smithi118.stdout: python3-grpcio-tools x86_64 1.46.7-10.el9 epel 144 k 2024-03-24T00:01:06.732 INFO:teuthology.orchestra.run.smithi118.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-03-24T00:01:06.732 INFO:teuthology.orchestra.run.smithi118.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-03-24T00:01:06.733 INFO:teuthology.orchestra.run.smithi118.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2024-03-24T00:01:06.733 INFO:teuthology.orchestra.run.smithi118.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-03-24T00:01:06.733 INFO:teuthology.orchestra.run.smithi118.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-03-24T00:01:06.733 INFO:teuthology.orchestra.run.smithi118.stdout: python3-jinja2 noarch 2.11.3-5.el9 appstream 248 k 2024-03-24T00:01:06.733 INFO:teuthology.orchestra.run.smithi118.stdout: python3-jsonpatch noarch 1.21-16.el9 CentOS-AppStream 24 k 2024-03-24T00:01:06.733 INFO:teuthology.orchestra.run.smithi118.stdout: python3-jsonpointer noarch 2.0-4.el9 CentOS-AppStream 18 k 2024-03-24T00:01:06.734 INFO:teuthology.orchestra.run.smithi118.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-03-24T00:01:06.734 INFO:teuthology.orchestra.run.smithi118.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-03-24T00:01:06.734 INFO:teuthology.orchestra.run.smithi118.stdout: python3-mako noarch 1.1.4-6.el9 CentOS-AppStream 144 k 2024-03-24T00:01:06.734 INFO:teuthology.orchestra.run.smithi118.stdout: python3-markupsafe x86_64 1.1.1-12.el9 CentOS-AppStream 32 k 2024-03-24T00:01:06.734 INFO:teuthology.orchestra.run.smithi118.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-03-24T00:01:06.734 INFO:teuthology.orchestra.run.smithi118.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-03-24T00:01:06.735 INFO:teuthology.orchestra.run.smithi118.stdout: python3-numpy x86_64 1:1.20.1-5.el9 CentOS-AppStream 4.8 M 2024-03-24T00:01:06.735 INFO:teuthology.orchestra.run.smithi118.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 CentOS-AppStream 217 k 2024-03-24T00:01:06.735 INFO:teuthology.orchestra.run.smithi118.stdout: python3-oauthlib noarch 3.1.1-5.el9 CentOS-AppStream 174 k 2024-03-24T00:01:06.735 INFO:teuthology.orchestra.run.smithi118.stdout: python3-packaging noarch 20.9-5.el9 CentOS-AppStream 69 k 2024-03-24T00:01:06.735 INFO:teuthology.orchestra.run.smithi118.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-03-24T00:01:06.735 INFO:teuthology.orchestra.run.smithi118.stdout: python3-ply noarch 3.11-14.el9 CentOS-BaseOS 103 k 2024-03-24T00:01:06.736 INFO:teuthology.orchestra.run.smithi118.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-03-24T00:01:06.736 INFO:teuthology.orchestra.run.smithi118.stdout: python3-prettytable noarch 0.7.2-27.el9 CentOS-AppStream 41 k 2024-03-24T00:01:06.736 INFO:teuthology.orchestra.run.smithi118.stdout: python3-protobuf noarch 3.14.0-13.el9 CentOS-AppStream 238 k 2024-03-24T00:01:06.736 INFO:teuthology.orchestra.run.smithi118.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-03-24T00:01:06.736 INFO:teuthology.orchestra.run.smithi118.stdout: python3-pyasn1 noarch 0.4.8-6.el9 CentOS-AppStream 132 k 2024-03-24T00:01:06.736 INFO:teuthology.orchestra.run.smithi118.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 CentOS-AppStream 211 k 2024-03-24T00:01:06.737 INFO:teuthology.orchestra.run.smithi118.stdout: python3-pycparser noarch 2.20-6.el9 CentOS-BaseOS 124 k 2024-03-24T00:01:06.737 INFO:teuthology.orchestra.run.smithi118.stdout: python3-pyperclip noarch 1.8.2-2.el9 epel 28 k 2024-03-24T00:01:06.737 INFO:teuthology.orchestra.run.smithi118.stdout: python3-pytz noarch 2021.1-5.el9 CentOS-AppStream 47 k 2024-03-24T00:01:06.737 INFO:teuthology.orchestra.run.smithi118.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-03-24T00:01:06.737 INFO:teuthology.orchestra.run.smithi118.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-03-24T00:01:06.737 INFO:teuthology.orchestra.run.smithi118.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 CentOS-AppStream 43 k 2024-03-24T00:01:06.737 INFO:teuthology.orchestra.run.smithi118.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-03-24T00:01:06.738 INFO:teuthology.orchestra.run.smithi118.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-03-24T00:01:06.738 INFO:teuthology.orchestra.run.smithi118.stdout: python3-scipy x86_64 1.6.2-8.el9 CentOS-AppStream 16 M 2024-03-24T00:01:06.738 INFO:teuthology.orchestra.run.smithi118.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-03-24T00:01:06.738 INFO:teuthology.orchestra.run.smithi118.stdout: python3-toml noarch 0.10.2-6.el9 CentOS-AppStream 39 k 2024-03-24T00:01:06.738 INFO:teuthology.orchestra.run.smithi118.stdout: python3-typing-extensions noarch 4.4.0-2.el9 epel 51 k 2024-03-24T00:01:06.739 INFO:teuthology.orchestra.run.smithi118.stdout: python3-urllib3 noarch 1.26.5-5.el9 CentOS-BaseOS 187 k 2024-03-24T00:01:06.739 INFO:teuthology.orchestra.run.smithi118.stdout: python3-wcwidth noarch 0.2.5-8.el9 CentOS-AppStream 41 k 2024-03-24T00:01:06.739 INFO:teuthology.orchestra.run.smithi118.stdout: python3-webob noarch 1.8.7-6.el9 epel 230 k 2024-03-24T00:01:06.739 INFO:teuthology.orchestra.run.smithi118.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-03-24T00:01:06.739 INFO:teuthology.orchestra.run.smithi118.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-03-24T00:01:06.739 INFO:teuthology.orchestra.run.smithi118.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-03-24T00:01:06.739 INFO:teuthology.orchestra.run.smithi118.stdout: qatlib x86_64 23.11.0-1.el9 appstream 216 k 2024-03-24T00:01:06.739 INFO:teuthology.orchestra.run.smithi118.stdout: qatzip-libs x86_64 1.1.2-2.el9 appstream 44 k 2024-03-24T00:01:06.740 INFO:teuthology.orchestra.run.smithi118.stdout: re2 x86_64 1:20211101-3.el9 epel 192 k 2024-03-24T00:01:06.740 INFO:teuthology.orchestra.run.smithi118.stdout: socat x86_64 1.7.4.1-5.el9 CentOS-AppStream 300 k 2024-03-24T00:01:06.740 INFO:teuthology.orchestra.run.smithi118.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-03-24T00:01:06.740 INFO:teuthology.orchestra.run.smithi118.stdout: uuid x86_64 1.6.2-55.el9 CentOS-AppStream 56 k 2024-03-24T00:01:06.740 INFO:teuthology.orchestra.run.smithi118.stdout: xmlstarlet x86_64 1.6.1-20.el9 CentOS-AppStream 63 k 2024-03-24T00:01:06.740 INFO:teuthology.orchestra.run.smithi118.stdout: zlib-devel x86_64 1.2.11-40.el9 CentOS-AppStream 44 k 2024-03-24T00:01:06.740 INFO:teuthology.orchestra.run.smithi118.stdout:Installing weak dependencies: 2024-03-24T00:01:06.740 INFO:teuthology.orchestra.run.smithi118.stdout: qatlib-service x86_64 23.11.0-1.el9 appstream 35 k 2024-03-24T00:01:06.740 INFO:teuthology.orchestra.run.smithi118.stdout: 2024-03-24T00:01:06.741 INFO:teuthology.orchestra.run.smithi118.stdout:Transaction Summary 2024-03-24T00:01:06.741 INFO:teuthology.orchestra.run.smithi118.stdout:============================================================================================ 2024-03-24T00:01:06.741 INFO:teuthology.orchestra.run.smithi118.stdout:Install 150 Packages 2024-03-24T00:01:06.741 INFO:teuthology.orchestra.run.smithi118.stdout:Upgrade 24 Packages 2024-03-24T00:01:06.741 INFO:teuthology.orchestra.run.smithi118.stdout: 2024-03-24T00:01:06.742 INFO:teuthology.orchestra.run.smithi189.stdout:(9/174): ceph-selinux-19.0.0-2145.g3be9a5c9.el9 431 kB/s | 25 kB 00:00 2024-03-24T00:01:06.742 INFO:teuthology.orchestra.run.smithi118.stdout:Total download size: 224 M 2024-03-24T00:01:06.742 INFO:teuthology.orchestra.run.smithi118.stdout:Downloading Packages: 2024-03-24T00:01:07.564 INFO:teuthology.orchestra.run.smithi118.stdout:(1/174): ceph-19.0.0-2145.g3be9a5c9.el9.x86_64. 14 kB/s | 6.6 kB 00:00 2024-03-24T00:01:07.885 INFO:teuthology.orchestra.run.smithi193.stdout:(1/174): ceph-19.0.0-2145.g3be9a5c9.el9.x86_64. 21 kB/s | 6.6 kB 00:00 2024-03-24T00:01:07.933 INFO:teuthology.orchestra.run.smithi189.stdout:(10/174): ceph-osd-19.0.0-2145.g3be9a5c9.el9.x8 7.8 MB/s | 17 MB 00:02 2024-03-24T00:01:08.034 INFO:teuthology.orchestra.run.smithi189.stdout:(11/174): cephfs-mirror-19.0.0-2145.g3be9a5c9.e 2.2 MB/s | 220 kB 00:00 2024-03-24T00:01:08.123 INFO:teuthology.orchestra.run.smithi118.stdout:(2/174): ceph-fuse-19.0.0-2145.g3be9a5c9.el9.x8 1.6 MB/s | 890 kB 00:00 2024-03-24T00:01:08.125 INFO:teuthology.orchestra.run.smithi189.stdout:(12/174): cephfs-shell-19.0.0-2145.g3be9a5c9.el 295 kB/s | 27 kB 00:00 2024-03-24T00:01:08.217 INFO:teuthology.orchestra.run.smithi189.stdout:(13/174): libcephfs-devel-19.0.0-2145.g3be9a5c9 341 kB/s | 31 kB 00:00 2024-03-24T00:01:08.255 INFO:teuthology.orchestra.run.smithi118.stdout:(3/174): ceph-immutable-object-cache-19.0.0-214 1.1 MB/s | 145 kB 00:00 2024-03-24T00:01:08.409 INFO:teuthology.orchestra.run.smithi189.stdout:(14/174): libcephfs2-19.0.0-2145.g3be9a5c9.el9. 3.8 MB/s | 746 kB 00:00 2024-03-24T00:01:08.485 INFO:teuthology.orchestra.run.smithi193.stdout:(2/174): ceph-fuse-19.0.0-2145.g3be9a5c9.el9.x8 1.4 MB/s | 890 kB 00:00 2024-03-24T00:01:08.501 INFO:teuthology.orchestra.run.smithi189.stdout:(15/174): libcephsqlite-19.0.0-2145.g3be9a5c9.e 1.7 MB/s | 163 kB 00:00 2024-03-24T00:01:08.593 INFO:teuthology.orchestra.run.smithi189.stdout:(16/174): librados-devel-19.0.0-2145.g3be9a5c9. 1.4 MB/s | 127 kB 00:00 2024-03-24T00:01:08.635 INFO:teuthology.orchestra.run.smithi193.stdout:(3/174): ceph-immutable-object-cache-19.0.0-214 968 kB/s | 145 kB 00:00 2024-03-24T00:01:09.069 INFO:teuthology.orchestra.run.smithi189.stdout:(17/174): libradosstriper1-19.0.0-2145.g3be9a5c 1.0 MB/s | 501 kB 00:00 2024-03-24T00:01:09.210 INFO:teuthology.orchestra.run.smithi118.stdout:(4/174): ceph-mds-19.0.0-2145.g3be9a5c9.el9.x86 2.4 MB/s | 2.3 MB 00:00 2024-03-24T00:01:09.511 INFO:teuthology.orchestra.run.smithi193.stdout:(4/174): ceph-base-19.0.0-2145.g3be9a5c9.el9.x8 2.8 MB/s | 5.4 MB 00:01 2024-03-24T00:01:09.790 INFO:teuthology.orchestra.run.smithi118.stdout:(5/174): ceph-mgr-19.0.0-2145.g3be9a5c9.el9.x86 2.6 MB/s | 1.5 MB 00:00 2024-03-24T00:01:09.844 INFO:teuthology.orchestra.run.smithi193.stdout:(5/174): ceph-mgr-19.0.0-2145.g3be9a5c9.el9.x86 4.5 MB/s | 1.5 MB 00:00 2024-03-24T00:01:09.952 INFO:teuthology.orchestra.run.smithi193.stdout:(6/174): ceph-mds-19.0.0-2145.g3be9a5c9.el9.x86 1.8 MB/s | 2.3 MB 00:01 2024-03-24T00:01:10.435 INFO:teuthology.orchestra.run.smithi189.stdout:(18/174): librgw2-19.0.0-2145.g3be9a5c9.el9.x86 3.8 MB/s | 5.1 MB 00:01 2024-03-24T00:01:10.544 INFO:teuthology.orchestra.run.smithi193.stdout:(7/174): ceph-mon-19.0.0-2145.g3be9a5c9.el9.x86 6.7 MB/s | 4.7 MB 00:00 2024-03-24T00:01:10.652 INFO:teuthology.orchestra.run.smithi189.stdout:(19/174): ceph-common-19.0.0-2145.g3be9a5c9.el9 3.6 MB/s | 20 MB 00:05 2024-03-24T00:01:10.669 INFO:teuthology.orchestra.run.smithi189.stdout:(20/174): python3-ceph-argparse-19.0.0-2145.g3b 193 kB/s | 45 kB 00:00 2024-03-24T00:01:10.761 INFO:teuthology.orchestra.run.smithi189.stdout:(21/174): python3-ceph-common-19.0.0-2145.g3be9 1.2 MB/s | 131 kB 00:00 2024-03-24T00:01:10.786 INFO:teuthology.orchestra.run.smithi189.stdout:(22/174): python3-cephfs-19.0.0-2145.g3be9a5c9. 1.4 MB/s | 162 kB 00:00 2024-03-24T00:01:10.962 INFO:teuthology.orchestra.run.smithi189.stdout:(23/174): python3-rados-19.0.0-2145.g3be9a5c9.e 1.6 MB/s | 321 kB 00:00 2024-03-24T00:01:11.206 INFO:teuthology.orchestra.run.smithi118.stdout:(6/174): ceph-mon-19.0.0-2145.g3be9a5c9.el9.x86 3.3 MB/s | 4.7 MB 00:01 2024-03-24T00:01:11.428 INFO:teuthology.orchestra.run.smithi193.stdout:(8/174): ceph-radosgw-19.0.0-2145.g3be9a5c9.el9 11 MB/s | 9.9 MB 00:00 2024-03-24T00:01:11.454 INFO:teuthology.orchestra.run.smithi189.stdout:(24/174): ceph-test-19.0.0-2145.g3be9a5c9.el9.x 10 MB/s | 48 MB 00:04 2024-03-24T00:01:11.470 INFO:teuthology.orchestra.run.smithi193.stdout:(9/174): ceph-selinux-19.0.0-2145.g3be9a5c9.el9 607 kB/s | 25 kB 00:00 2024-03-24T00:01:11.479 INFO:teuthology.orchestra.run.smithi189.stdout:(25/174): python3-rgw-19.0.0-2145.g3be9a5c9.el9 192 kB/s | 99 kB 00:00 2024-03-24T00:01:11.504 INFO:teuthology.orchestra.run.smithi189.stdout:(26/174): python3-rbd-19.0.0-2145.g3be9a5c9.el9 416 kB/s | 298 kB 00:00 2024-03-24T00:01:11.529 INFO:teuthology.orchestra.run.smithi189.stdout:(27/174): rbd-fuse-19.0.0-2145.g3be9a5c9.el9.x8 1.1 MB/s | 85 kB 00:00 2024-03-24T00:01:11.605 INFO:teuthology.orchestra.run.smithi189.stdout:(28/174): ceph-grafana-dashboards-19.0.0-2145.g 452 kB/s | 34 kB 00:00 2024-03-24T00:01:11.672 INFO:teuthology.orchestra.run.smithi189.stdout:(29/174): rbd-nbd-19.0.0-2145.g3be9a5c9.el9.x86 1.0 MB/s | 170 kB 00:00 2024-03-24T00:01:11.697 INFO:teuthology.orchestra.run.smithi189.stdout:(30/174): ceph-mgr-cephadm-19.0.0-2145.g3be9a5c 1.5 MB/s | 139 kB 00:00 2024-03-24T00:01:12.289 INFO:teuthology.orchestra.run.smithi189.stdout:(31/174): ceph-mgr-dashboard-19.0.0-2145.g3be9a 2.9 MB/s | 1.8 MB 00:00 2024-03-24T00:01:12.398 INFO:teuthology.orchestra.run.smithi189.stdout:(32/174): ceph-mgr-modules-core-19.0.0-2145.g3b 2.2 MB/s | 248 kB 00:00 2024-03-24T00:01:12.456 INFO:teuthology.orchestra.run.smithi189.stdout:(33/174): rbd-mirror-19.0.0-2145.g3be9a5c9.el9. 3.2 MB/s | 3.1 MB 00:00 2024-03-24T00:01:12.498 INFO:teuthology.orchestra.run.smithi189.stdout:(34/174): ceph-mgr-rook-19.0.0-2145.g3be9a5c9.e 492 kB/s | 49 kB 00:00 2024-03-24T00:01:12.540 INFO:teuthology.orchestra.run.smithi189.stdout:(35/174): ceph-prometheus-alerts-19.0.0-2145.g3 197 kB/s | 16 kB 00:00 2024-03-24T00:01:12.607 INFO:teuthology.orchestra.run.smithi189.stdout:(36/174): ceph-volume-19.0.0-2145.g3be9a5c9.el9 2.4 MB/s | 264 kB 00:00 2024-03-24T00:01:12.740 INFO:teuthology.orchestra.run.smithi189.stdout:(37/174): cephfs-top-19.0.0-2145.g3be9a5c9.el9. 137 kB/s | 18 kB 00:00 2024-03-24T00:01:12.832 INFO:teuthology.orchestra.run.smithi189.stdout:(38/174): cephadm-19.0.0-2145.g3be9a5c9.el9.noa 2.5 MB/s | 748 kB 00:00 2024-03-24T00:01:12.908 INFO:teuthology.orchestra.run.smithi189.stdout:(39/174): automake-1.16.2-8.el9.noarch.rpm 3.9 MB/s | 662 kB 00:00 2024-03-24T00:01:12.950 INFO:teuthology.orchestra.run.smithi189.stdout:(40/174): bison-3.7.4-5.el9.x86_64.rpm 7.7 MB/s | 920 kB 00:00 2024-03-24T00:01:12.975 INFO:teuthology.orchestra.run.smithi189.stdout:(41/174): boost-program-options-1.75.0-8.el9.x8 1.6 MB/s | 106 kB 00:00 2024-03-24T00:01:13.000 INFO:teuthology.orchestra.run.smithi189.stdout:(42/174): flex-2.6.4-9.el9.x86_64.rpm 6.0 MB/s | 307 kB 00:00 2024-03-24T00:01:13.015 INFO:teuthology.orchestra.run.smithi118.stdout:(7/174): ceph-base-19.0.0-2145.g3be9a5c9.el9.x8 924 kB/s | 5.4 MB 00:05 2024-03-24T00:01:13.025 INFO:teuthology.orchestra.run.smithi189.stdout:(43/174): flexiblas-3.0.4-8.el9.x86_64.rpm 640 kB/s | 32 kB 00:00 2024-03-24T00:01:13.159 INFO:teuthology.orchestra.run.smithi189.stdout:(44/174): flexiblas-openblas-openmp-3.0.4-8.el9 128 kB/s | 17 kB 00:00 2024-03-24T00:01:13.218 INFO:teuthology.orchestra.run.smithi189.stdout:(45/174): flexiblas-netlib-3.0.4-8.el9.x86_64.r 14 MB/s | 3.0 MB 00:00 2024-03-24T00:01:13.252 INFO:teuthology.orchestra.run.smithi189.stdout:(46/174): libaio-devel-0.3.111-13.el9.x86_64.rp 111 kB/s | 10 kB 00:00 2024-03-24T00:01:13.277 INFO:teuthology.orchestra.run.smithi189.stdout:(47/174): libpmemobj-1.12.1-1.el9.x86_64.rpm 2.6 MB/s | 159 kB 00:00 2024-03-24T00:01:13.302 INFO:teuthology.orchestra.run.smithi189.stdout:(48/174): librabbitmq-0.11.0-7.el9.x86_64.rpm 885 kB/s | 44 kB 00:00 2024-03-24T00:01:13.328 INFO:teuthology.orchestra.run.smithi189.stdout:(49/174): librdkafka-1.6.1-102.el9.x86_64.rpm 13 MB/s | 661 kB 00:00 2024-03-24T00:01:13.353 INFO:teuthology.orchestra.run.smithi189.stdout:(50/174): libtool-2.4.6-45.el9.x86_64.rpm 12 MB/s | 578 kB 00:00 2024-03-24T00:01:13.370 INFO:teuthology.orchestra.run.smithi189.stdout:(51/174): libzstd-devel-1.5.1-2.el9.x86_64.rpm 1.1 MB/s | 46 kB 00:00 2024-03-24T00:01:13.397 INFO:teuthology.orchestra.run.smithi189.stdout:(52/174): lttng-ust-2.12.0-6.el9.x86_64.rpm 6.6 MB/s | 282 kB 00:00 2024-03-24T00:01:13.420 INFO:teuthology.orchestra.run.smithi189.stdout:(53/174): lua-5.4.4-4.el9.x86_64.rpm 3.7 MB/s | 187 kB 00:00 2024-03-24T00:01:13.454 INFO:teuthology.orchestra.run.smithi189.stdout:(54/174): oniguruma-6.9.6-1.el9.5.x86_64.rpm 3.7 MB/s | 217 kB 00:00 2024-03-24T00:01:13.470 INFO:teuthology.orchestra.run.smithi189.stdout:(55/174): openblas-0.3.21-2.el9.x86_64.rpm 695 kB/s | 34 kB 00:00 2024-03-24T00:01:13.596 INFO:teuthology.orchestra.run.smithi189.stdout:(56/174): openblas-openmp-0.3.21-2.el9.x86_64.r 33 MB/s | 4.7 MB 00:00 2024-03-24T00:01:13.638 INFO:teuthology.orchestra.run.smithi189.stdout:(57/174): postgresql-13.11-1.el9.x86_64.rpm 9.1 MB/s | 1.5 MB 00:00 2024-03-24T00:01:13.672 INFO:teuthology.orchestra.run.smithi189.stdout:(58/174): postgresql-contrib-13.11-1.el9.x86_64 11 MB/s | 811 kB 00:00 2024-03-24T00:01:13.697 INFO:teuthology.orchestra.run.smithi189.stdout:(59/174): postgresql-private-libs-13.11-1.el9.x 2.3 MB/s | 134 kB 00:00 2024-03-24T00:01:13.789 INFO:teuthology.orchestra.run.smithi189.stdout:(60/174): ceph-mgr-diskprediction-local-19.0.0- 3.5 MB/s | 7.4 MB 00:02 2024-03-24T00:01:13.865 INFO:teuthology.orchestra.run.smithi189.stdout:(61/174): postgresql-server-13.11-1.el9.x86_64. 30 MB/s | 5.7 MB 00:00 2024-03-24T00:01:13.898 INFO:teuthology.orchestra.run.smithi189.stdout:(62/174): protobuf-3.14.0-13.el9.x86_64.rpm 5.1 MB/s | 1.0 MB 00:00 2024-03-24T00:01:13.932 INFO:teuthology.orchestra.run.smithi189.stdout:(63/174): python3-attrs-20.3.0-7.el9.noarch.rpm 556 kB/s | 78 kB 00:00 2024-03-24T00:01:14.040 INFO:teuthology.orchestra.run.smithi189.stdout:(64/174): python3-babel-2.9.1-2.el9.noarch.rpm 33 MB/s | 5.8 MB 00:00 2024-03-24T00:01:14.066 INFO:teuthology.orchestra.run.smithi189.stdout:(65/174): python3-jsonpointer-2.0-4.el9.noarch. 138 kB/s | 18 kB 00:00 2024-03-24T00:01:14.091 INFO:teuthology.orchestra.run.smithi189.stdout:(66/174): python3-jsonpatch-1.21-16.el9.noarch. 156 kB/s | 24 kB 00:00 2024-03-24T00:01:14.116 INFO:teuthology.orchestra.run.smithi189.stdout:(67/174): python3-mako-1.1.4-6.el9.noarch.rpm 1.9 MB/s | 144 kB 00:00 2024-03-24T00:01:14.141 INFO:teuthology.orchestra.run.smithi189.stdout:(68/174): python3-markupsafe-1.1.1-12.el9.x86_6 419 kB/s | 32 kB 00:00 2024-03-24T00:01:14.175 INFO:teuthology.orchestra.run.smithi189.stdout:(69/174): python3-numpy-f2py-1.20.1-5.el9.x86_6 3.6 MB/s | 217 kB 00:00 2024-03-24T00:01:14.267 INFO:teuthology.orchestra.run.smithi189.stdout:(70/174): python3-numpy-1.20.1-5.el9.x86_64.rpm 32 MB/s | 4.8 MB 00:00 2024-03-24T00:01:14.292 INFO:teuthology.orchestra.run.smithi189.stdout:(71/174): python3-oauthlib-3.1.1-5.el9.noarch.r 1.1 MB/s | 174 kB 00:00 2024-03-24T00:01:14.317 INFO:teuthology.orchestra.run.smithi189.stdout:(72/174): python3-packaging-20.9-5.el9.noarch.r 490 kB/s | 69 kB 00:00 2024-03-24T00:01:14.334 INFO:teuthology.orchestra.run.smithi189.stdout:(73/174): python3-prettytable-0.7.2-27.el9.noar 618 kB/s | 41 kB 00:00 2024-03-24T00:01:14.360 INFO:teuthology.orchestra.run.smithi189.stdout:(74/174): python3-protobuf-3.14.0-13.el9.noarch 5.6 MB/s | 238 kB 00:00 2024-03-24T00:01:14.385 INFO:teuthology.orchestra.run.smithi189.stdout:(75/174): python3-pyasn1-0.4.8-6.el9.noarch.rpm 1.9 MB/s | 132 kB 00:00 2024-03-24T00:01:14.410 INFO:teuthology.orchestra.run.smithi189.stdout:(76/174): python3-pyasn1-modules-0.4.8-6.el9.no 2.7 MB/s | 211 kB 00:00 2024-03-24T00:01:14.427 INFO:teuthology.orchestra.run.smithi189.stdout:(77/174): python3-requests-oauthlib-1.3.0-12.el 1.0 MB/s | 43 kB 00:00 2024-03-24T00:01:14.444 INFO:teuthology.orchestra.run.smithi189.stdout:(78/174): python3-pytz-2021.1-5.el9.noarch.rpm 811 kB/s | 47 kB 00:00 2024-03-24T00:01:14.536 INFO:teuthology.orchestra.run.smithi189.stdout:(79/174): python3-toml-0.10.2-6.el9.noarch.rpm 357 kB/s | 39 kB 00:00 2024-03-24T00:01:14.586 INFO:teuthology.orchestra.run.smithi189.stdout:(80/174): python3-wcwidth-0.2.5-8.el9.noarch.rp 291 kB/s | 41 kB 00:00 2024-03-24T00:01:14.753 INFO:teuthology.orchestra.run.smithi189.stdout:(81/174): python3-scipy-1.6.2-8.el9.x86_64.rpm 46 MB/s | 16 MB 00:00 2024-03-24T00:01:14.778 INFO:teuthology.orchestra.run.smithi189.stdout:(82/174): socat-1.7.4.1-5.el9.x86_64.rpm 1.2 MB/s | 300 kB 00:00 2024-03-24T00:01:14.804 INFO:teuthology.orchestra.run.smithi189.stdout:(83/174): uuid-1.6.2-55.el9.x86_64.rpm 262 kB/s | 56 kB 00:00 2024-03-24T00:01:14.829 INFO:teuthology.orchestra.run.smithi189.stdout:(84/174): xmlstarlet-1.6.1-20.el9.x86_64.rpm 839 kB/s | 63 kB 00:00 2024-03-24T00:01:14.854 INFO:teuthology.orchestra.run.smithi189.stdout:(85/174): fuse-2.9.9-15.el9.x86_64.rpm 1.5 MB/s | 79 kB 00:00 2024-03-24T00:01:14.879 INFO:teuthology.orchestra.run.smithi189.stdout:(86/174): zlib-devel-1.2.11-40.el9.x86_64.rpm 586 kB/s | 44 kB 00:00 2024-03-24T00:01:14.904 INFO:teuthology.orchestra.run.smithi189.stdout:(87/174): python3-cffi-1.14.5-5.el9.x86_64.rpm 3.1 MB/s | 241 kB 00:00 2024-03-24T00:01:14.947 INFO:teuthology.orchestra.run.smithi189.stdout:(88/174): python3-cryptography-36.0.1-4.el9.x86 12 MB/s | 1.1 MB 00:00 2024-03-24T00:01:14.972 INFO:teuthology.orchestra.run.smithi189.stdout:(89/174): python3-ply-3.11-14.el9.noarch.rpm 1.5 MB/s | 103 kB 00:00 2024-03-24T00:01:14.997 INFO:teuthology.orchestra.run.smithi189.stdout:(90/174): python3-pycparser-2.20-6.el9.noarch.r 1.3 MB/s | 124 kB 00:00 2024-03-24T00:01:15.022 INFO:teuthology.orchestra.run.smithi189.stdout:(91/174): python3-urllib3-1.26.5-5.el9.noarch.r 2.4 MB/s | 187 kB 00:00 2024-03-24T00:01:15.240 INFO:teuthology.orchestra.run.smithi189.stdout:(92/174): jq-1.6-16.el9.x86_64.rpm 775 kB/s | 187 kB 00:00 2024-03-24T00:01:15.265 INFO:teuthology.orchestra.run.smithi189.stdout:(93/174): libquadmath-11.4.1-3.el9.x86_64.rpm 807 kB/s | 195 kB 00:00 2024-03-24T00:01:15.281 INFO:teuthology.orchestra.run.smithi118.stdout:(8/174): ceph-osd-19.0.0-2145.g3be9a5c9.el9.x86 4.1 MB/s | 17 MB 00:04 2024-03-24T00:01:15.298 INFO:teuthology.orchestra.run.smithi189.stdout:(94/174): python3-requests-2.25.1-8.el9.noarch. 2.1 MB/s | 125 kB 00:00 2024-03-24T00:01:15.340 INFO:teuthology.orchestra.run.smithi189.stdout:(95/174): elfutils-libelf-devel-0.190-2.el9.x86 286 kB/s | 21 kB 00:00 2024-03-24T00:01:15.365 INFO:teuthology.orchestra.run.smithi189.stdout:(96/174): libuuid-devel-2.37.4-18.el9.x86_64.rp 380 kB/s | 25 kB 00:00 2024-03-24T00:01:15.373 INFO:teuthology.orchestra.run.smithi118.stdout:(9/174): ceph-selinux-19.0.0-2145.g3be9a5c9.el9 277 kB/s | 25 kB 00:00 2024-03-24T00:01:15.399 INFO:teuthology.orchestra.run.smithi189.stdout:(97/174): libgfortran-11.4.1-3.el9.x86_64.rpm 2.0 MB/s | 803 kB 00:00 2024-03-24T00:01:15.466 INFO:teuthology.orchestra.run.smithi189.stdout:(98/174): python3-devel-3.9.17-2.el9.x86_64.rpm 2.4 MB/s | 245 kB 00:00 2024-03-24T00:01:15.491 INFO:teuthology.orchestra.run.smithi189.stdout:(99/174): python3-jinja2-2.11.3-5.el9.noarch.rp 2.6 MB/s | 248 kB 00:00 2024-03-24T00:01:15.520 INFO:teuthology.orchestra.run.smithi193.stdout:(10/174): ceph-test-19.0.0-2145.g3be9a5c9.el9.x 12 MB/s | 48 MB 00:04 2024-03-24T00:01:15.550 INFO:teuthology.orchestra.run.smithi189.stdout:(100/174): qatlib-service-23.11.0-1.el9.x86_64. 598 kB/s | 35 kB 00:00 2024-03-24T00:01:15.575 INFO:teuthology.orchestra.run.smithi189.stdout:(101/174): qatlib-23.11.0-1.el9.x86_64.rpm 1.9 MB/s | 216 kB 00:00 2024-03-24T00:01:15.578 INFO:teuthology.orchestra.run.smithi193.stdout:(11/174): cephfs-mirror-19.0.0-2145.g3be9a5c9.e 3.7 MB/s | 220 kB 00:00 2024-03-24T00:01:15.642 INFO:teuthology.orchestra.run.smithi189.stdout:(102/174): qatzip-libs-1.1.2-2.el9.x86_64.rpm 485 kB/s | 44 kB 00:00 2024-03-24T00:01:15.701 INFO:teuthology.orchestra.run.smithi189.stdout:(103/174): openssl-devel-3.0.7-27.el9.x86_64.rp 11 MB/s | 4.1 MB 00:00 2024-03-24T00:01:15.726 INFO:teuthology.orchestra.run.smithi189.stdout:(104/174): xfsprogs-devel-6.3.0-1.el9.x86_64.rp 623 kB/s | 94 kB 00:00 2024-03-24T00:01:15.762 INFO:teuthology.orchestra.run.smithi193.stdout:(12/174): ceph-osd-19.0.0-2145.g3be9a5c9.el9.x8 2.9 MB/s | 17 MB 00:05 2024-03-24T00:01:15.784 INFO:teuthology.orchestra.run.smithi189.stdout:(105/174): lua-devel-5.4.4-4.el9.x86_64.rpm 157 kB/s | 22 kB 00:00 2024-03-24T00:01:15.787 INFO:teuthology.orchestra.run.smithi193.stdout:(13/174): cephfs-shell-19.0.0-2145.g3be9a5c9.el 130 kB/s | 27 kB 00:00 2024-03-24T00:01:15.812 INFO:teuthology.orchestra.run.smithi193.stdout:(14/174): libcephfs-devel-19.0.0-2145.g3be9a5c9 629 kB/s | 31 kB 00:00 2024-03-24T00:01:15.880 INFO:teuthology.orchestra.run.smithi193.stdout:(15/174): libcephfs2-19.0.0-2145.g3be9a5c9.el9. 7.9 MB/s | 746 kB 00:00 2024-03-24T00:01:15.905 INFO:teuthology.orchestra.run.smithi193.stdout:(16/174): libcephsqlite-19.0.0-2145.g3be9a5c9.e 1.7 MB/s | 163 kB 00:00 2024-03-24T00:01:15.938 INFO:teuthology.orchestra.run.smithi193.stdout:(17/174): librados-devel-19.0.0-2145.g3be9a5c9. 2.1 MB/s | 127 kB 00:00 2024-03-24T00:01:15.976 INFO:teuthology.orchestra.run.smithi189.stdout:(106/174): protobuf-compiler-3.14.0-13.el9.x86_ 3.1 MB/s | 863 kB 00:00 2024-03-24T00:01:16.039 INFO:teuthology.orchestra.run.smithi193.stdout:(18/174): libradosstriper1-19.0.0-2145.g3be9a5c 3.7 MB/s | 501 kB 00:00 2024-03-24T00:01:16.114 INFO:teuthology.orchestra.run.smithi193.stdout:(19/174): python3-ceph-argparse-19.0.0-2145.g3b 602 kB/s | 45 kB 00:00 2024-03-24T00:01:16.189 INFO:teuthology.orchestra.run.smithi193.stdout:(20/174): python3-ceph-common-19.0.0-2145.g3be9 1.7 MB/s | 131 kB 00:00 2024-03-24T00:01:16.243 INFO:teuthology.orchestra.run.smithi189.stdout:(107/174): grpc-data-1.46.7-10.el9.noarch.rpm 73 kB/s | 19 kB 00:00 2024-03-24T00:01:16.264 INFO:teuthology.orchestra.run.smithi193.stdout:(21/174): python3-cephfs-19.0.0-2145.g3be9a5c9. 2.1 MB/s | 162 kB 00:00 2024-03-24T00:01:16.302 INFO:teuthology.orchestra.run.smithi189.stdout:(108/174): gperftools-libs-2.9.1-2.el9.x86_64.r 597 kB/s | 309 kB 00:00 2024-03-24T00:01:16.327 INFO:teuthology.orchestra.run.smithi189.stdout:(109/174): abseil-cpp-20211102.0-3.el9.x86_64.r 912 kB/s | 548 kB 00:00 2024-03-24T00:01:16.348 INFO:teuthology.orchestra.run.smithi193.stdout:(22/174): librgw2-19.0.0-2145.g3be9a5c9.el9.x86 13 MB/s | 5.1 MB 00:00 2024-03-24T00:01:16.373 INFO:teuthology.orchestra.run.smithi193.stdout:(23/174): python3-rados-19.0.0-2145.g3be9a5c9.e 2.9 MB/s | 321 kB 00:00 2024-03-24T00:01:16.407 INFO:teuthology.orchestra.run.smithi193.stdout:(24/174): python3-rbd-19.0.0-2145.g3be9a5c9.el9 5.0 MB/s | 298 kB 00:00 2024-03-24T00:01:16.430 INFO:teuthology.orchestra.run.smithi193.stdout:(25/174): python3-rgw-19.0.0-2145.g3be9a5c9.el9 1.7 MB/s | 99 kB 00:00 2024-03-24T00:01:16.455 INFO:teuthology.orchestra.run.smithi189.stdout:(110/174): libarrow-doc-9.0.0-10.el9.noarch.rpm 170 kB/s | 26 kB 00:00 2024-03-24T00:01:16.455 INFO:teuthology.orchestra.run.smithi193.stdout:(26/174): rbd-fuse-19.0.0-2145.g3be9a5c9.el9.x8 1.8 MB/s | 85 kB 00:00 2024-03-24T00:01:16.480 INFO:teuthology.orchestra.run.smithi189.stdout:(111/174): liboath-2.6.7-2.el9.x86_64.rpm 318 kB/s | 49 kB 00:00 2024-03-24T00:01:16.524 INFO:teuthology.orchestra.run.smithi118.stdout:(10/174): ceph-common-19.0.0-2145.g3be9a5c9.el9 2.2 MB/s | 20 MB 00:09 2024-03-24T00:01:16.572 INFO:teuthology.orchestra.run.smithi189.stdout:(112/174): libunwind-1.6.2-1.el9.x86_64.rpm 577 kB/s | 67 kB 00:00 2024-03-24T00:01:16.591 INFO:teuthology.orchestra.run.smithi193.stdout:(27/174): rbd-nbd-19.0.0-2145.g3be9a5c9.el9.x86 1.2 MB/s | 170 kB 00:00 2024-03-24T00:01:16.597 INFO:teuthology.orchestra.run.smithi189.stdout:(113/174): luarocks-3.9.2-1.el9.noarch.rpm 1.3 MB/s | 151 kB 00:00 2024-03-24T00:01:16.599 INFO:teuthology.orchestra.run.smithi118.stdout:(11/174): cephfs-mirror-19.0.0-2145.g3be9a5c9.e 2.9 MB/s | 220 kB 00:00 2024-03-24T00:01:16.656 INFO:teuthology.orchestra.run.smithi193.stdout:(28/174): ceph-grafana-dashboards-19.0.0-2145.g 520 kB/s | 34 kB 00:00 2024-03-24T00:01:16.674 INFO:teuthology.orchestra.run.smithi118.stdout:(12/174): cephfs-shell-19.0.0-2145.g3be9a5c9.el 365 kB/s | 27 kB 00:00 2024-03-24T00:01:16.706 INFO:teuthology.orchestra.run.smithi193.stdout:(29/174): ceph-mgr-cephadm-19.0.0-2145.g3be9a5c 2.7 MB/s | 139 kB 00:00 2024-03-24T00:01:16.748 INFO:teuthology.orchestra.run.smithi189.stdout:(114/174): python3-asyncssh-2.13.2-5.el9.noarch 3.6 MB/s | 548 kB 00:00 2024-03-24T00:01:16.750 INFO:teuthology.orchestra.run.smithi118.stdout:(13/174): libcephfs-devel-19.0.0-2145.g3be9a5c9 421 kB/s | 31 kB 00:00 2024-03-24T00:01:16.865 INFO:teuthology.orchestra.run.smithi193.stdout:(30/174): ceph-mgr-dashboard-19.0.0-2145.g3be9a 11 MB/s | 1.8 MB 00:00 2024-03-24T00:01:16.924 INFO:teuthology.orchestra.run.smithi189.stdout:(115/174): parquet-libs-9.0.0-10.el9.x86_64.rpm 2.3 MB/s | 838 kB 00:00 2024-03-24T00:01:16.949 INFO:teuthology.orchestra.run.smithi189.stdout:(116/174): python3-bcrypt-3.2.2-1.el9.x86_64.rp 217 kB/s | 43 kB 00:00 2024-03-24T00:01:17.025 INFO:teuthology.orchestra.run.smithi118.stdout:(14/174): libcephfs2-19.0.0-2145.g3be9a5c9.el9. 2.7 MB/s | 746 kB 00:00 2024-03-24T00:01:17.049 INFO:teuthology.orchestra.run.smithi189.stdout:(117/174): libarrow-9.0.0-10.el9.x86_64.rpm 5.5 MB/s | 4.4 MB 00:00 2024-03-24T00:01:17.074 INFO:teuthology.orchestra.run.smithi189.stdout:(118/174): python3-cachetools-4.2.4-1.el9.noarc 214 kB/s | 32 kB 00:00 2024-03-24T00:01:17.090 INFO:teuthology.orchestra.run.smithi193.stdout:(31/174): rbd-mirror-19.0.0-2145.g3be9a5c9.el9. 4.7 MB/s | 3.1 MB 00:00 2024-03-24T00:01:17.100 INFO:teuthology.orchestra.run.smithi189.stdout:(119/174): python3-certifi-2021.10.8-2.el9.noar 99 kB/s | 15 kB 00:00 2024-03-24T00:01:17.109 INFO:teuthology.orchestra.run.smithi118.stdout:(15/174): libcephsqlite-19.0.0-2145.g3be9a5c9.e 1.9 MB/s | 163 kB 00:00 2024-03-24T00:01:17.158 INFO:teuthology.orchestra.run.smithi189.stdout:(120/174): python3-cheroot-8.6.0-4.el9.noarch.r 1.5 MB/s | 172 kB 00:00 2024-03-24T00:01:17.200 INFO:teuthology.orchestra.run.smithi189.stdout:(121/174): python3-cherrypy-18.6.1-2.el9.noarch 2.8 MB/s | 358 kB 00:00 2024-03-24T00:01:17.209 INFO:teuthology.orchestra.run.smithi118.stdout:(16/174): librados-devel-19.0.0-2145.g3be9a5c9. 1.3 MB/s | 127 kB 00:00 2024-03-24T00:01:17.216 INFO:teuthology.orchestra.run.smithi193.stdout:(32/174): ceph-mgr-modules-core-19.0.0-2145.g3b 1.9 MB/s | 248 kB 00:00 2024-03-24T00:01:17.225 INFO:teuthology.orchestra.run.smithi189.stdout:(122/174): python3-cmd2-2.4.2-2.el9.noarch.rpm 2.4 MB/s | 303 kB 00:00 2024-03-24T00:01:17.251 INFO:teuthology.orchestra.run.smithi189.stdout:(123/174): python3-colorama-0.4.4-12.el9.noarch 383 kB/s | 35 kB 00:00 2024-03-24T00:01:17.291 INFO:teuthology.orchestra.run.smithi193.stdout:(33/174): ceph-mgr-rook-19.0.0-2145.g3be9a5c9.e 660 kB/s | 49 kB 00:00 2024-03-24T00:01:17.326 INFO:teuthology.orchestra.run.smithi189.stdout:(124/174): python3-google-auth-2.28.2-1.el9.noa 1.6 MB/s | 203 kB 00:00 2024-03-24T00:01:17.351 INFO:teuthology.orchestra.run.smithi189.stdout:(125/174): python3-grpcio-tools-1.46.7-10.el9.x 1.4 MB/s | 144 kB 00:00 2024-03-24T00:01:17.384 INFO:teuthology.orchestra.run.smithi118.stdout:(17/174): libradosstriper1-19.0.0-2145.g3be9a5c 2.8 MB/s | 501 kB 00:00 2024-03-24T00:01:17.408 INFO:teuthology.orchestra.run.smithi193.stdout:(34/174): ceph-mgr-diskprediction-local-19.0.0- 14 MB/s | 7.4 MB 00:00 2024-03-24T00:01:17.426 INFO:teuthology.orchestra.run.smithi189.stdout:(126/174): python3-jaraco-8.2.1-3.el9.noarch.rp 107 kB/s | 11 kB 00:00 2024-03-24T00:01:17.433 INFO:teuthology.orchestra.run.smithi193.stdout:(35/174): ceph-prometheus-alerts-19.0.0-2145.g3 116 kB/s | 16 kB 00:00 2024-03-24T00:01:17.443 INFO:teuthology.orchestra.run.smithi189.stdout:(127/174): python3-jaraco-classes-3.2.1-5.el9.n 192 kB/s | 18 kB 00:00 2024-03-24T00:01:17.458 INFO:teuthology.orchestra.run.smithi193.stdout:(36/174): ceph-volume-19.0.0-2145.g3be9a5c9.el9 5.1 MB/s | 264 kB 00:00 2024-03-24T00:01:17.485 INFO:teuthology.orchestra.run.smithi189.stdout:(128/174): python3-grpcio-1.46.7-10.el9.x86_64. 7.9 MB/s | 2.0 MB 00:00 2024-03-24T00:01:17.500 INFO:teuthology.orchestra.run.smithi193.stdout:(37/174): cephfs-top-19.0.0-2145.g3be9a5c9.el9. 438 kB/s | 18 kB 00:00 2024-03-24T00:01:17.527 INFO:teuthology.orchestra.run.smithi189.stdout:(129/174): python3-jaraco-collections-3.0.0-8.e 231 kB/s | 23 kB 00:00 2024-03-24T00:01:17.552 INFO:teuthology.orchestra.run.smithi189.stdout:(130/174): python3-jaraco-functools-3.5.0-2.el9 179 kB/s | 19 kB 00:00 2024-03-24T00:01:17.592 INFO:teuthology.orchestra.run.smithi193.stdout:(38/174): cephadm-19.0.0-2145.g3be9a5c9.el9.noa 4.6 MB/s | 748 kB 00:00 2024-03-24T00:01:17.611 INFO:teuthology.orchestra.run.smithi189.stdout:(131/174): python3-jaraco-text-3.2.0-6.el9.noar 157 kB/s | 20 kB 00:00 2024-03-24T00:01:17.686 INFO:teuthology.orchestra.run.smithi189.stdout:(132/174): python3-logutils-0.3.5-21.el9.noarch 346 kB/s | 46 kB 00:00 2024-03-24T00:01:17.812 INFO:teuthology.orchestra.run.smithi189.stdout:(133/174): python3-natsort-7.1.1-5.el9.noarch.r 462 kB/s | 58 kB 00:00 2024-03-24T00:01:17.845 INFO:teuthology.orchestra.run.smithi189.stdout:(134/174): python3-kubernetes-26.1.0-2.el9.noar 3.2 MB/s | 1.0 MB 00:00 2024-03-24T00:01:17.909 INFO:teuthology.orchestra.run.smithi193.stdout:(39/174): bison-3.7.4-5.el9.x86_64.rpm 2.9 MB/s | 920 kB 00:00 2024-03-24T00:01:17.929 INFO:teuthology.orchestra.run.smithi189.stdout:(135/174): python3-pecan-1.4.2-3.el9.noarch.rpm 2.3 MB/s | 272 kB 00:00 2024-03-24T00:01:17.935 INFO:teuthology.orchestra.run.smithi193.stdout:(40/174): automake-1.16.2-8.el9.noarch.rpm 1.5 MB/s | 662 kB 00:00 2024-03-24T00:01:17.960 INFO:teuthology.orchestra.run.smithi193.stdout:(41/174): boost-program-options-1.75.0-8.el9.x8 2.1 MB/s | 106 kB 00:00 2024-03-24T00:01:17.962 INFO:teuthology.orchestra.run.smithi189.stdout:(136/174): python3-portend-3.1.0-2.el9.noarch.r 140 kB/s | 16 kB 00:00 2024-03-24T00:01:17.987 INFO:teuthology.orchestra.run.smithi189.stdout:(137/174): python3-more-itertools-8.12.0-2.el9. 209 kB/s | 79 kB 00:00 2024-03-24T00:01:17.994 INFO:teuthology.orchestra.run.smithi193.stdout:(42/174): flex-2.6.4-9.el9.x86_64.rpm 5.1 MB/s | 307 kB 00:00 2024-03-24T00:01:18.019 INFO:teuthology.orchestra.run.smithi193.stdout:(43/174): flexiblas-3.0.4-8.el9.x86_64.rpm 546 kB/s | 32 kB 00:00 2024-03-24T00:01:18.021 INFO:teuthology.orchestra.run.smithi189.stdout:(138/174): python3-pyOpenSSL-21.0.0-1.el9.noarc 976 kB/s | 90 kB 00:00 2024-03-24T00:01:18.080 INFO:teuthology.orchestra.run.smithi189.stdout:(139/174): python3-pyperclip-1.8.2-2.el9.noarch 241 kB/s | 28 kB 00:00 2024-03-24T00:01:18.085 INFO:teuthology.orchestra.run.smithi118.stdout:(18/174): librgw2-19.0.0-2145.g3be9a5c9.el9.x86 7.4 MB/s | 5.1 MB 00:00 2024-03-24T00:01:18.113 INFO:teuthology.orchestra.run.smithi189.stdout:(140/174): python3-repoze-lru-0.7-16.el9.noarch 245 kB/s | 31 kB 00:00 2024-03-24T00:01:18.127 INFO:teuthology.orchestra.run.smithi193.stdout:(44/174): flexiblas-netlib-3.0.4-8.el9.x86_64.r 22 MB/s | 3.0 MB 00:00 2024-03-24T00:01:18.147 INFO:teuthology.orchestra.run.smithi189.stdout:(141/174): python3-routes-2.5.1-5.el9.noarch.rp 1.5 MB/s | 188 kB 00:00 2024-03-24T00:01:18.151 INFO:teuthology.orchestra.run.smithi118.stdout:(19/174): python3-ceph-argparse-19.0.0-2145.g3b 687 kB/s | 45 kB 00:00 2024-03-24T00:01:18.153 INFO:teuthology.orchestra.run.smithi193.stdout:(45/174): flexiblas-openblas-openmp-3.0.4-8.el9 129 kB/s | 17 kB 00:00 2024-03-24T00:01:18.202 INFO:teuthology.orchestra.run.smithi118.stdout:(20/174): python3-ceph-common-19.0.0-2145.g3be9 2.6 MB/s | 131 kB 00:00 2024-03-24T00:01:18.203 INFO:teuthology.orchestra.run.smithi193.stdout:(46/174): libaio-devel-0.3.111-13.el9.x86_64.rp 202 kB/s | 10 kB 00:00 2024-03-24T00:01:18.214 INFO:teuthology.orchestra.run.smithi189.stdout:(142/174): python3-rsa-4.9-2.el9.noarch.rpm 441 kB/s | 59 kB 00:00 2024-03-24T00:01:18.237 INFO:teuthology.orchestra.run.smithi193.stdout:(47/174): libpmemobj-1.12.1-1.el9.x86_64.rpm 1.9 MB/s | 159 kB 00:00 2024-03-24T00:01:18.252 INFO:teuthology.orchestra.run.smithi118.stdout:(21/174): python3-cephfs-19.0.0-2145.g3be9a5c9. 3.2 MB/s | 162 kB 00:00 2024-03-24T00:01:18.256 INFO:teuthology.orchestra.run.smithi189.stdout:(143/174): python3-tempora-5.0.0-2.el9.noarch.r 251 kB/s | 36 kB 00:00 2024-03-24T00:01:18.262 INFO:teuthology.orchestra.run.smithi193.stdout:(48/174): librabbitmq-0.11.0-7.el9.x86_64.rpm 755 kB/s | 44 kB 00:00 2024-03-24T00:01:18.281 INFO:teuthology.orchestra.run.smithi189.stdout:(144/174): python3-typing-extensions-4.4.0-2.el 380 kB/s | 51 kB 00:00 2024-03-24T00:01:18.296 INFO:teuthology.orchestra.run.smithi193.stdout:(49/174): librdkafka-1.6.1-102.el9.x86_64.rpm 11 MB/s | 661 kB 00:00 2024-03-24T00:01:18.321 INFO:teuthology.orchestra.run.smithi193.stdout:(50/174): libtool-2.4.6-45.el9.x86_64.rpm 9.8 MB/s | 578 kB 00:00 2024-03-24T00:01:18.323 INFO:teuthology.orchestra.run.smithi189.stdout:(145/174): python3-webob-1.8.7-6.el9.noarch.rpm 2.1 MB/s | 230 kB 00:00 2024-03-24T00:01:18.346 INFO:teuthology.orchestra.run.smithi193.stdout:(51/174): libzstd-devel-1.5.1-2.el9.x86_64.rpm 940 kB/s | 46 kB 00:00 2024-03-24T00:01:18.352 INFO:teuthology.orchestra.run.smithi118.stdout:(22/174): python3-rados-19.0.0-2145.g3be9a5c9.e 3.2 MB/s | 321 kB 00:00 2024-03-24T00:01:18.357 INFO:teuthology.orchestra.run.smithi189.stdout:(146/174): python3-websocket-client-1.2.3-2.el9 890 kB/s | 90 kB 00:00 2024-03-24T00:01:18.380 INFO:teuthology.orchestra.run.smithi193.stdout:(52/174): lttng-ust-2.12.0-6.el9.x86_64.rpm 4.7 MB/s | 282 kB 00:00 2024-03-24T00:01:18.382 INFO:teuthology.orchestra.run.smithi189.stdout:(147/174): python3-werkzeug-2.0.3-3.el9.1.noarc 4.1 MB/s | 427 kB 00:00 2024-03-24T00:01:18.405 INFO:teuthology.orchestra.run.smithi193.stdout:(53/174): lua-5.4.4-4.el9.x86_64.rpm 3.1 MB/s | 187 kB 00:00 2024-03-24T00:01:18.416 INFO:teuthology.orchestra.run.smithi189.stdout:(148/174): python3-zc-lockfile-2.0-10.el9.noarc 216 kB/s | 20 kB 00:00 2024-03-24T00:01:18.431 INFO:teuthology.orchestra.run.smithi193.stdout:(54/174): oniguruma-6.9.6-1.el9.5.x86_64.rpm 4.2 MB/s | 217 kB 00:00 2024-03-24T00:01:18.453 INFO:teuthology.orchestra.run.smithi118.stdout:(23/174): python3-rbd-19.0.0-2145.g3be9a5c9.el9 2.9 MB/s | 298 kB 00:00 2024-03-24T00:01:18.456 INFO:teuthology.orchestra.run.smithi193.stdout:(55/174): openblas-0.3.21-2.el9.x86_64.rpm 687 kB/s | 34 kB 00:00 2024-03-24T00:01:18.483 INFO:teuthology.orchestra.run.smithi189.stdout:(149/174): re2-20211101-3.el9.x86_64.rpm 1.5 MB/s | 192 kB 00:00 2024-03-24T00:01:18.503 INFO:teuthology.orchestra.run.smithi118.stdout:(24/174): python3-rgw-19.0.0-2145.g3be9a5c9.el9 2.0 MB/s | 99 kB 00:00 2024-03-24T00:01:18.553 INFO:teuthology.orchestra.run.smithi118.stdout:(25/174): rbd-fuse-19.0.0-2145.g3be9a5c9.el9.x8 1.7 MB/s | 85 kB 00:00 2024-03-24T00:01:18.566 INFO:teuthology.orchestra.run.smithi189.stdout:(150/174): thrift-0.15.0-2.el9.x86_64.rpm 8.6 MB/s | 1.6 MB 00:00 2024-03-24T00:01:18.606 INFO:teuthology.orchestra.run.smithi193.stdout:(56/174): openblas-openmp-0.3.21-2.el9.x86_64.r 27 MB/s | 4.7 MB 00:00 2024-03-24T00:01:18.608 INFO:teuthology.orchestra.run.smithi189.stdout:(151/174): xfsdump-3.1.12-4.el9.x86_64.rpm 7.6 MB/s | 321 kB 00:00 2024-03-24T00:01:18.648 INFO:teuthology.orchestra.run.smithi193.stdout:(57/174): postgresql-13.11-1.el9.x86_64.rpm 8.0 MB/s | 1.5 MB 00:00 2024-03-24T00:01:18.690 INFO:teuthology.orchestra.run.smithi193.stdout:(58/174): postgresql-contrib-13.11-1.el9.x86_64 9.8 MB/s | 811 kB 00:00 2024-03-24T00:01:18.715 INFO:teuthology.orchestra.run.smithi193.stdout:(59/174): postgresql-private-libs-13.11-1.el9.x 2.0 MB/s | 134 kB 00:00 2024-03-24T00:01:18.737 INFO:teuthology.orchestra.run.smithi118.stdout:(26/174): ceph-radosgw-19.0.0-2145.g3be9a5c9.el 1.7 MB/s | 9.9 MB 00:05 2024-03-24T00:01:18.858 INFO:teuthology.orchestra.run.smithi193.stdout:(60/174): postgresql-server-13.11-1.el9.x86_64. 34 MB/s | 5.7 MB 00:00 2024-03-24T00:01:18.891 INFO:teuthology.orchestra.run.smithi193.stdout:(61/174): protobuf-3.14.0-13.el9.x86_64.rpm 5.8 MB/s | 1.0 MB 00:00 2024-03-24T00:01:18.916 INFO:teuthology.orchestra.run.smithi193.stdout:(62/174): python3-attrs-20.3.0-7.el9.noarch.rpm 1.3 MB/s | 78 kB 00:00 2024-03-24T00:01:18.924 INFO:teuthology.orchestra.run.smithi118.stdout:(27/174): rbd-nbd-19.0.0-2145.g3be9a5c9.el9.x86 919 kB/s | 170 kB 00:00 2024-03-24T00:01:18.950 INFO:teuthology.orchestra.run.smithi189.stdout:(152/174): librbd1-19.0.0-2145.g3be9a5c9.el9.x8 6.7 MB/s | 3.1 MB 00:00 2024-03-24T00:01:19.009 INFO:teuthology.orchestra.run.smithi189.stdout:(153/174): librados2-19.0.0-2145.g3be9a5c9.el9. 5.7 MB/s | 3.4 MB 00:00 2024-03-24T00:01:19.021 INFO:teuthology.orchestra.run.smithi118.stdout:(28/174): ceph-grafana-dashboards-19.0.0-2145.g 354 kB/s | 34 kB 00:00 2024-03-24T00:01:19.067 INFO:teuthology.orchestra.run.smithi193.stdout:(63/174): python3-babel-2.9.1-2.el9.noarch.rpm 33 MB/s | 5.8 MB 00:00 2024-03-24T00:01:19.083 INFO:teuthology.orchestra.run.smithi193.stdout:(64/174): python3-jsonpatch-1.21-16.el9.noarch. 148 kB/s | 24 kB 00:00 2024-03-24T00:01:19.100 INFO:teuthology.orchestra.run.smithi193.stdout:(65/174): python3-jsonpointer-2.0-4.el9.noarch. 544 kB/s | 18 kB 00:00 2024-03-24T00:01:19.126 INFO:teuthology.orchestra.run.smithi193.stdout:(66/174): python3-mako-1.1.4-6.el9.noarch.rpm 3.4 MB/s | 144 kB 00:00 2024-03-24T00:01:19.129 INFO:teuthology.orchestra.run.smithi118.stdout:(29/174): ceph-mgr-cephadm-19.0.0-2145.g3be9a5c 1.3 MB/s | 139 kB 00:00 2024-03-24T00:01:19.134 INFO:teuthology.orchestra.run.smithi189.stdout:(154/174): NetworkManager-team-1.47.2-1.el9.x86 337 kB/s | 42 kB 00:00 2024-03-24T00:01:19.143 INFO:teuthology.orchestra.run.smithi193.stdout:(67/174): python3-markupsafe-1.1.1-12.el9.x86_6 755 kB/s | 32 kB 00:00 2024-03-24T00:01:19.226 INFO:teuthology.orchestra.run.smithi193.stdout:(68/174): python3-numpy-f2py-1.20.1-5.el9.x86_6 2.6 MB/s | 217 kB 00:00 2024-03-24T00:01:19.230 INFO:teuthology.orchestra.run.smithi118.stdout:(30/174): rbd-mirror-19.0.0-2145.g3be9a5c9.el9. 4.6 MB/s | 3.1 MB 00:00 2024-03-24T00:01:19.276 INFO:teuthology.orchestra.run.smithi189.stdout:(155/174): NetworkManager-1.47.2-1.el9.x86_64.r 3.5 MB/s | 2.3 MB 00:00 2024-03-24T00:01:19.293 INFO:teuthology.orchestra.run.smithi193.stdout:(69/174): python3-numpy-1.20.1-5.el9.x86_64.rpm 29 MB/s | 4.8 MB 00:00 2024-03-24T00:01:19.301 INFO:teuthology.orchestra.run.smithi189.stdout:(156/174): NetworkManager-tui-1.47.2-1.el9.x86_ 1.5 MB/s | 248 kB 00:00 2024-03-24T00:01:19.318 INFO:teuthology.orchestra.run.smithi193.stdout:(70/174): python3-oauthlib-3.1.1-5.el9.noarch.r 1.9 MB/s | 174 kB 00:00 2024-03-24T00:01:19.335 INFO:teuthology.orchestra.run.smithi189.stdout:(157/174): acl-2.3.1-4.el9.x86_64.rpm 1.2 MB/s | 71 kB 00:00 2024-03-24T00:01:19.344 INFO:teuthology.orchestra.run.smithi193.stdout:(71/174): python3-packaging-20.9-5.el9.noarch.r 1.3 MB/s | 69 kB 00:00 2024-03-24T00:01:19.368 INFO:teuthology.orchestra.run.smithi189.stdout:(158/174): elfutils-debuginfod-client-0.190-2.e 545 kB/s | 36 kB 00:00 2024-03-24T00:01:19.369 INFO:teuthology.orchestra.run.smithi193.stdout:(72/174): python3-prettytable-0.7.2-27.el9.noar 820 kB/s | 41 kB 00:00 2024-03-24T00:01:19.394 INFO:teuthology.orchestra.run.smithi189.stdout:(159/174): elfutils-libelf-0.190-2.el9.x86_64.r 3.2 MB/s | 192 kB 00:00 2024-03-24T00:01:19.452 INFO:teuthology.orchestra.run.smithi189.stdout:(160/174): libacl-2.3.1-4.el9.x86_64.rpm 394 kB/s | 23 kB 00:00 2024-03-24T00:01:19.478 INFO:teuthology.orchestra.run.smithi189.stdout:(161/174): elfutils-libs-0.190-2.el9.x86_64.rpm 2.3 MB/s | 256 kB 00:00 2024-03-24T00:01:19.511 INFO:teuthology.orchestra.run.smithi189.stdout:(162/174): libblkid-2.37.4-18.el9.x86_64.rpm 1.8 MB/s | 108 kB 00:00 2024-03-24T00:01:19.561 INFO:teuthology.orchestra.run.smithi189.stdout:(163/174): NetworkManager-libnm-1.47.2-1.el9.x8 3.0 MB/s | 1.8 MB 00:00 2024-03-24T00:01:19.563 INFO:teuthology.orchestra.run.smithi193.stdout:(73/174): ceph-common-19.0.0-2145.g3be9a5c9.el9 1.7 MB/s | 20 MB 00:11 2024-03-24T00:01:19.587 INFO:teuthology.orchestra.run.smithi189.stdout:(164/174): libfdisk-2.37.4-18.el9.x86_64.rpm 1.4 MB/s | 155 kB 00:00 2024-03-24T00:01:19.588 INFO:teuthology.orchestra.run.smithi193.stdout:(74/174): python3-protobuf-3.14.0-13.el9.noarch 976 kB/s | 238 kB 00:00 2024-03-24T00:01:19.612 INFO:teuthology.orchestra.run.smithi189.stdout:(165/174): libmount-2.37.4-18.el9.x86_64.rpm 1.3 MB/s | 136 kB 00:00 2024-03-24T00:01:19.613 INFO:teuthology.orchestra.run.smithi193.stdout:(75/174): python3-pyasn1-0.4.8-6.el9.noarch.rpm 543 kB/s | 132 kB 00:00 2024-03-24T00:01:19.629 INFO:teuthology.orchestra.run.smithi189.stdout:(166/174): libsmartcols-2.37.4-18.el9.x86_64.rp 952 kB/s | 64 kB 00:00 2024-03-24T00:01:19.639 INFO:teuthology.orchestra.run.smithi193.stdout:(76/174): python3-pytz-2021.1-5.el9.noarch.rpm 951 kB/s | 47 kB 00:00 2024-03-24T00:01:19.646 INFO:teuthology.orchestra.run.smithi189.stdout:(167/174): libuuid-2.37.4-18.el9.x86_64.rpm 488 kB/s | 29 kB 00:00 2024-03-24T00:01:19.664 INFO:teuthology.orchestra.run.smithi193.stdout:(77/174): python3-pyasn1-modules-0.4.8-6.el9.no 2.7 MB/s | 211 kB 00:00 2024-03-24T00:01:19.689 INFO:teuthology.orchestra.run.smithi193.stdout:(78/174): python3-requests-oauthlib-1.3.0-12.el 568 kB/s | 43 kB 00:00 2024-03-24T00:01:19.830 INFO:teuthology.orchestra.run.smithi118.stdout:(31/174): ceph-mgr-dashboard-19.0.0-2145.g3be9a 2.6 MB/s | 1.8 MB 00:00 2024-03-24T00:01:19.854 INFO:teuthology.orchestra.run.smithi189.stdout:(168/174): openssl-3.0.7-27.el9.x86_64.rpm 5.0 MB/s | 1.2 MB 00:00 2024-03-24T00:01:19.898 INFO:teuthology.orchestra.run.smithi193.stdout:(79/174): python3-toml-0.10.2-6.el9.noarch.rpm 188 kB/s | 39 kB 00:00 2024-03-24T00:01:19.923 INFO:teuthology.orchestra.run.smithi193.stdout:(80/174): python3-wcwidth-0.2.5-8.el9.noarch.rp 178 kB/s | 41 kB 00:00 2024-03-24T00:01:19.963 INFO:teuthology.orchestra.run.smithi189.stdout:(169/174): util-linux-core-2.37.4-18.el9.x86_64 4.2 MB/s | 465 kB 00:00 2024-03-24T00:01:19.997 INFO:teuthology.orchestra.run.smithi118.stdout:(32/174): ceph-mgr-modules-core-19.0.0-2145.g3b 1.5 MB/s | 248 kB 00:00 2024-03-24T00:01:20.030 INFO:teuthology.orchestra.run.smithi189.stdout:(170/174): openssl-libs-3.0.7-27.el9.x86_64.rpm 5.4 MB/s | 2.2 MB 00:00 2024-03-24T00:01:20.055 INFO:teuthology.orchestra.run.smithi189.stdout:(171/174): util-linux-user-2.37.4-18.el9.x86_64 338 kB/s | 31 kB 00:00 2024-03-24T00:01:20.082 INFO:teuthology.orchestra.run.smithi193.stdout:(81/174): python3-scipy-1.6.2-8.el9.x86_64.rpm 35 MB/s | 16 MB 00:00 2024-03-24T00:01:20.107 INFO:teuthology.orchestra.run.smithi193.stdout:(82/174): socat-1.7.4.1-5.el9.x86_64.rpm 1.4 MB/s | 300 kB 00:00 2024-03-24T00:01:20.114 INFO:teuthology.orchestra.run.smithi189.stdout:(172/174): libacl-devel-2.3.1-4.el9.x86_64.rpm 1.2 MB/s | 74 kB 00:00 2024-03-24T00:01:20.132 INFO:teuthology.orchestra.run.smithi193.stdout:(83/174): uuid-1.6.2-55.el9.x86_64.rpm 273 kB/s | 56 kB 00:00 2024-03-24T00:01:20.149 INFO:teuthology.orchestra.run.smithi118.stdout:(33/174): ceph-mgr-diskprediction-local-19.0.0- 8.1 MB/s | 7.4 MB 00:00 2024-03-24T00:01:20.157 INFO:teuthology.orchestra.run.smithi193.stdout:(84/174): xmlstarlet-1.6.1-20.el9.x86_64.rpm 1.2 MB/s | 63 kB 00:00 2024-03-24T00:01:20.182 INFO:teuthology.orchestra.run.smithi193.stdout:(85/174): zlib-devel-1.2.11-40.el9.x86_64.rpm 588 kB/s | 44 kB 00:00 2024-03-24T00:01:20.197 INFO:teuthology.orchestra.run.smithi189.stdout:(173/174): util-linux-2.37.4-18.el9.x86_64.rpm 4.1 MB/s | 2.3 MB 00:00 2024-03-24T00:01:20.198 INFO:teuthology.orchestra.run.smithi118.stdout:(34/174): ceph-mgr-rook-19.0.0-2145.g3be9a5c9.e 247 kB/s | 49 kB 00:00 2024-03-24T00:01:20.208 INFO:teuthology.orchestra.run.smithi193.stdout:(86/174): fuse-2.9.9-15.el9.x86_64.rpm 1.0 MB/s | 79 kB 00:00 2024-03-24T00:01:20.231 INFO:teuthology.orchestra.run.smithi189.stdout:(174/174): xfsprogs-6.3.0-1.el9.x86_64.rpm 5.3 MB/s | 1.1 MB 00:00 2024-03-24T00:01:20.233 INFO:teuthology.orchestra.run.smithi193.stdout:(87/174): python3-cffi-1.14.5-5.el9.x86_64.rpm 4.7 MB/s | 241 kB 00:00 2024-03-24T00:01:20.236 INFO:teuthology.orchestra.run.smithi189.stdout:-------------------------------------------------------------------------------- 2024-03-24T00:01:20.236 INFO:teuthology.orchestra.run.smithi189.stdout:Total 14 MB/s | 224 MB 00:15 2024-03-24T00:01:20.248 INFO:teuthology.orchestra.run.smithi118.stdout:(35/174): ceph-prometheus-alerts-19.0.0-2145.g3 166 kB/s | 16 kB 00:00 2024-03-24T00:01:20.275 INFO:teuthology.orchestra.run.smithi193.stdout:(88/174): python3-cryptography-36.0.1-4.el9.x86 12 MB/s | 1.1 MB 00:00 2024-03-24T00:01:20.300 INFO:teuthology.orchestra.run.smithi193.stdout:(89/174): python3-ply-3.11-14.el9.noarch.rpm 1.1 MB/s | 103 kB 00:00 2024-03-24T00:01:20.325 INFO:teuthology.orchestra.run.smithi193.stdout:(90/174): python3-pycparser-2.20-6.el9.noarch.r 1.3 MB/s | 124 kB 00:00 2024-03-24T00:01:20.351 INFO:teuthology.orchestra.run.smithi193.stdout:(91/174): python3-urllib3-1.26.5-5.el9.noarch.r 3.7 MB/s | 187 kB 00:00 2024-03-24T00:01:20.373 INFO:teuthology.orchestra.run.smithi118.stdout:(36/174): ceph-volume-19.0.0-2145.g3be9a5c9.el9 1.5 MB/s | 264 kB 00:00 2024-03-24T00:01:20.407 INFO:teuthology.orchestra.run.smithi118.stdout:(37/174): cephadm-19.0.0-2145.g3be9a5c9.el9.noa 4.6 MB/s | 748 kB 00:00 2024-03-24T00:01:20.457 INFO:teuthology.orchestra.run.smithi118.stdout:(38/174): cephfs-top-19.0.0-2145.g3be9a5c9.el9. 220 kB/s | 18 kB 00:00 2024-03-24T00:01:20.468 INFO:teuthology.orchestra.run.smithi193.stdout:(92/174): jq-1.6-16.el9.x86_64.rpm 1.1 MB/s | 187 kB 00:00 2024-03-24T00:01:20.493 INFO:teuthology.orchestra.run.smithi193.stdout:(93/174): libquadmath-11.4.1-3.el9.x86_64.rpm 1.3 MB/s | 195 kB 00:00 2024-03-24T00:01:20.527 INFO:teuthology.orchestra.run.smithi193.stdout:(94/174): python3-requests-2.25.1-8.el9.noarch. 2.1 MB/s | 125 kB 00:00 2024-03-24T00:01:20.560 INFO:teuthology.orchestra.run.smithi193.stdout:(95/174): libgfortran-11.4.1-3.el9.x86_64.rpm 3.3 MB/s | 803 kB 00:00 2024-03-24T00:01:20.586 INFO:teuthology.orchestra.run.smithi193.stdout:(96/174): elfutils-libelf-devel-0.190-2.el9.x86 233 kB/s | 21 kB 00:00 2024-03-24T00:01:20.619 INFO:teuthology.orchestra.run.smithi193.stdout:(97/174): libuuid-devel-2.37.4-18.el9.x86_64.rp 276 kB/s | 25 kB 00:00 2024-03-24T00:01:20.683 INFO:teuthology.orchestra.run.smithi118.stdout:(39/174): automake-1.16.2-8.el9.noarch.rpm 2.4 MB/s | 662 kB 00:00 2024-03-24T00:01:20.694 INFO:teuthology.orchestra.run.smithi193.stdout:(98/174): python3-devel-3.9.17-2.el9.x86_64.rpm 2.2 MB/s | 245 kB 00:00 2024-03-24T00:01:20.710 INFO:teuthology.orchestra.run.smithi189.stdout:Running transaction check 2024-03-24T00:01:20.719 INFO:teuthology.orchestra.run.smithi193.stdout:(99/174): python3-jinja2-2.11.3-5.el9.noarch.rp 2.4 MB/s | 248 kB 00:00 2024-03-24T00:01:20.725 INFO:teuthology.orchestra.run.smithi118.stdout:(40/174): bison-3.7.4-5.el9.x86_64.rpm 4.5 MB/s | 920 kB 00:00 2024-03-24T00:01:20.745 INFO:teuthology.orchestra.run.smithi193.stdout:(100/174): qatlib-23.11.0-1.el9.x86_64.rpm 4.2 MB/s | 216 kB 00:00 2024-03-24T00:01:20.750 INFO:teuthology.orchestra.run.smithi118.stdout:(41/174): boost-program-options-1.75.0-8.el9.x8 1.7 MB/s | 106 kB 00:00 2024-03-24T00:01:20.762 INFO:teuthology.orchestra.run.smithi193.stdout:(101/174): qatlib-service-23.11.0-1.el9.x86_64. 830 kB/s | 35 kB 00:00 2024-03-24T00:01:20.779 INFO:teuthology.orchestra.run.smithi193.stdout:(102/174): qatzip-libs-1.1.2-2.el9.x86_64.rpm 1.3 MB/s | 44 kB 00:00 2024-03-24T00:01:20.784 INFO:teuthology.orchestra.run.smithi118.stdout:(42/174): flex-2.6.4-9.el9.x86_64.rpm 5.3 MB/s | 307 kB 00:00 2024-03-24T00:01:20.804 INFO:teuthology.orchestra.run.smithi193.stdout:(103/174): xfsprogs-devel-6.3.0-1.el9.x86_64.rp 2.2 MB/s | 94 kB 00:00 2024-03-24T00:01:20.867 INFO:teuthology.orchestra.run.smithi118.stdout:(43/174): flexiblas-3.0.4-8.el9.x86_64.rpm 278 kB/s | 32 kB 00:00 2024-03-24T00:01:20.913 INFO:teuthology.orchestra.run.smithi193.stdout:(104/174): openssl-devel-3.0.7-27.el9.x86_64.rp 12 MB/s | 4.1 MB 00:00 2024-03-24T00:01:20.946 INFO:teuthology.orchestra.run.smithi193.stdout:(105/174): lua-devel-5.4.4-4.el9.x86_64.rpm 133 kB/s | 22 kB 00:00 2024-03-24T00:01:20.959 INFO:teuthology.orchestra.run.smithi118.stdout:(44/174): flexiblas-netlib-3.0.4-8.el9.x86_64.r 17 MB/s | 3.0 MB 00:00 2024-03-24T00:01:20.984 INFO:teuthology.orchestra.run.smithi189.stdout:Transaction check succeeded. 2024-03-24T00:01:20.984 INFO:teuthology.orchestra.run.smithi189.stdout:Running transaction test 2024-03-24T00:01:21.001 INFO:teuthology.orchestra.run.smithi118.stdout:(45/174): flexiblas-openblas-openmp-3.0.4-8.el9 145 kB/s | 17 kB 00:00 2024-03-24T00:01:21.060 INFO:teuthology.orchestra.run.smithi118.stdout:(46/174): libaio-devel-0.3.111-13.el9.x86_64.rp 176 kB/s | 10 kB 00:00 2024-03-24T00:01:21.085 INFO:teuthology.orchestra.run.smithi118.stdout:(47/174): libpmemobj-1.12.1-1.el9.x86_64.rpm 1.9 MB/s | 159 kB 00:00 2024-03-24T00:01:21.127 INFO:teuthology.orchestra.run.smithi118.stdout:(48/174): librabbitmq-0.11.0-7.el9.x86_64.rpm 672 kB/s | 44 kB 00:00 2024-03-24T00:01:21.138 INFO:teuthology.orchestra.run.smithi193.stdout:(106/174): protobuf-compiler-3.14.0-13.el9.x86_ 2.5 MB/s | 863 kB 00:00 2024-03-24T00:01:21.169 INFO:teuthology.orchestra.run.smithi118.stdout:(49/174): librdkafka-1.6.1-102.el9.x86_64.rpm 7.8 MB/s | 661 kB 00:00 2024-03-24T00:01:21.203 INFO:teuthology.orchestra.run.smithi118.stdout:(50/174): libtool-2.4.6-45.el9.x86_64.rpm 8.2 MB/s | 578 kB 00:00 2024-03-24T00:01:21.245 INFO:teuthology.orchestra.run.smithi118.stdout:(51/174): libzstd-devel-1.5.1-2.el9.x86_64.rpm 665 kB/s | 46 kB 00:00 2024-03-24T00:01:21.278 INFO:teuthology.orchestra.run.smithi118.stdout:(52/174): lttng-ust-2.12.0-6.el9.x86_64.rpm 3.7 MB/s | 282 kB 00:00 2024-03-24T00:01:21.345 INFO:teuthology.orchestra.run.smithi118.stdout:(53/174): lua-5.4.4-4.el9.x86_64.rpm 1.9 MB/s | 187 kB 00:00 2024-03-24T00:01:21.372 INFO:teuthology.orchestra.run.smithi193.stdout:(107/174): grpc-data-1.46.7-10.el9.noarch.rpm 83 kB/s | 19 kB 00:00 2024-03-24T00:01:21.379 INFO:teuthology.orchestra.run.smithi118.stdout:(54/174): oniguruma-6.9.6-1.el9.5.x86_64.rpm 2.2 MB/s | 217 kB 00:00 2024-03-24T00:01:21.397 INFO:teuthology.orchestra.run.smithi193.stdout:(108/174): gperftools-libs-2.9.1-2.el9.x86_64.r 686 kB/s | 309 kB 00:00 2024-03-24T00:01:21.423 INFO:teuthology.orchestra.run.smithi118.stdout:(55/174): openblas-0.3.21-2.el9.x86_64.rpm 469 kB/s | 34 kB 00:00 2024-03-24T00:01:21.448 INFO:teuthology.orchestra.run.smithi193.stdout:(109/174): abseil-cpp-20211102.0-3.el9.x86_64.r 1.0 MB/s | 548 kB 00:00 2024-03-24T00:01:21.490 INFO:teuthology.orchestra.run.smithi193.stdout:(110/174): libarrow-doc-9.0.0-10.el9.noarch.rpm 283 kB/s | 26 kB 00:00 2024-03-24T00:01:21.540 INFO:teuthology.orchestra.run.smithi193.stdout:(111/174): liboath-2.6.7-2.el9.x86_64.rpm 529 kB/s | 49 kB 00:00 2024-03-24T00:01:21.582 INFO:teuthology.orchestra.run.smithi193.stdout:(112/174): libunwind-1.6.2-1.el9.x86_64.rpm 732 kB/s | 67 kB 00:00 2024-03-24T00:01:21.588 INFO:teuthology.orchestra.run.smithi118.stdout:(56/174): openblas-openmp-0.3.21-2.el9.x86_64.r 23 MB/s | 4.7 MB 00:00 2024-03-24T00:01:21.647 INFO:teuthology.orchestra.run.smithi118.stdout:(57/174): postgresql-13.11-1.el9.x86_64.rpm 7.2 MB/s | 1.5 MB 00:00 2024-03-24T00:01:21.697 INFO:teuthology.orchestra.run.smithi118.stdout:(58/174): postgresql-contrib-13.11-1.el9.x86_64 8.3 MB/s | 811 kB 00:00 2024-03-24T00:01:21.714 INFO:teuthology.orchestra.run.smithi118.stdout:(59/174): postgresql-private-libs-13.11-1.el9.x 2.2 MB/s | 134 kB 00:00 2024-03-24T00:01:21.807 INFO:teuthology.orchestra.run.smithi193.stdout:(113/174): luarocks-3.9.2-1.el9.noarch.rpm 566 kB/s | 151 kB 00:00 2024-03-24T00:01:21.898 INFO:teuthology.orchestra.run.smithi118.stdout:(60/174): postgresql-server-13.11-1.el9.x86_64. 29 MB/s | 5.7 MB 00:00 2024-03-24T00:01:21.916 INFO:teuthology.orchestra.run.smithi193.stdout:(114/174): parquet-libs-9.0.0-10.el9.x86_64.rpm 2.5 MB/s | 838 kB 00:00 2024-03-24T00:01:21.940 INFO:teuthology.orchestra.run.smithi118.stdout:(61/174): protobuf-3.14.0-13.el9.x86_64.rpm 4.6 MB/s | 1.0 MB 00:00 2024-03-24T00:01:21.941 INFO:teuthology.orchestra.run.smithi193.stdout:(115/174): python3-asyncssh-2.13.2-5.el9.noarch 4.0 MB/s | 548 kB 00:00 2024-03-24T00:01:22.025 INFO:teuthology.orchestra.run.smithi193.stdout:(116/174): python3-bcrypt-3.2.2-1.el9.x86_64.rp 400 kB/s | 43 kB 00:00 2024-03-24T00:01:22.049 INFO:teuthology.orchestra.run.smithi118.stdout:(62/174): python3-attrs-20.3.0-7.el9.noarch.rpm 548 kB/s | 78 kB 00:00 2024-03-24T00:01:22.050 INFO:teuthology.orchestra.run.smithi193.stdout:(117/174): python3-cachetools-4.2.4-1.el9.noarc 297 kB/s | 32 kB 00:00 2024-03-24T00:01:22.117 INFO:teuthology.orchestra.run.smithi193.stdout:(118/174): libarrow-9.0.0-10.el9.x86_64.rpm 5.9 MB/s | 4.4 MB 00:00 2024-03-24T00:01:22.142 INFO:teuthology.orchestra.run.smithi193.stdout:(119/174): python3-certifi-2021.10.8-2.el9.noar 128 kB/s | 15 kB 00:00 2024-03-24T00:01:22.168 INFO:teuthology.orchestra.run.smithi193.stdout:(120/174): python3-cheroot-8.6.0-4.el9.noarch.r 1.4 MB/s | 172 kB 00:00 2024-03-24T00:01:22.207 INFO:teuthology.orchestra.run.smithi118.stdout:(63/174): python3-babel-2.9.1-2.el9.noarch.rpm 22 MB/s | 5.8 MB 00:00 2024-03-24T00:01:22.218 INFO:teuthology.orchestra.run.smithi193.stdout:(121/174): python3-cherrypy-18.6.1-2.el9.noarch 3.5 MB/s | 358 kB 00:00 2024-03-24T00:01:22.232 INFO:teuthology.orchestra.run.smithi118.stdout:(64/174): python3-jsonpatch-1.21-16.el9.noarch. 145 kB/s | 24 kB 00:00 2024-03-24T00:01:22.252 INFO:teuthology.orchestra.run.smithi193.stdout:(122/174): python3-cmd2-2.4.2-2.el9.noarch.rpm 2.7 MB/s | 303 kB 00:00 2024-03-24T00:01:22.277 INFO:teuthology.orchestra.run.smithi193.stdout:(123/174): python3-colorama-0.4.4-12.el9.noarch 324 kB/s | 35 kB 00:00 2024-03-24T00:01:22.297 INFO:teuthology.orchestra.run.smithi118.stdout:(65/174): python3-jsonpointer-2.0-4.el9.noarch. 288 kB/s | 18 kB 00:00 2024-03-24T00:01:22.318 INFO:teuthology.orchestra.run.smithi193.stdout:(124/174): python3-google-auth-2.28.2-1.el9.noa 2.0 MB/s | 203 kB 00:00 2024-03-24T00:01:22.324 INFO:teuthology.orchestra.run.smithi118.stdout:(66/174): python3-mako-1.1.4-6.el9.noarch.rpm 1.6 MB/s | 144 kB 00:00 2024-03-24T00:01:22.375 INFO:teuthology.orchestra.run.smithi118.stdout:(67/174): python3-markupsafe-1.1.1-12.el9.x86_6 647 kB/s | 32 kB 00:00 2024-03-24T00:01:22.377 INFO:teuthology.orchestra.run.smithi193.stdout:(125/174): python3-grpcio-tools-1.46.7-10.el9.x 1.4 MB/s | 144 kB 00:00 2024-03-24T00:01:22.410 INFO:teuthology.orchestra.run.smithi193.stdout:(126/174): python3-jaraco-8.2.1-3.el9.noarch.rp 116 kB/s | 11 kB 00:00 2024-03-24T00:01:22.469 INFO:teuthology.orchestra.run.smithi193.stdout:(127/174): python3-jaraco-classes-3.2.1-5.el9.n 192 kB/s | 18 kB 00:00 2024-03-24T00:01:22.503 INFO:teuthology.orchestra.run.smithi193.stdout:(128/174): python3-jaraco-collections-3.0.0-8.e 251 kB/s | 23 kB 00:00 2024-03-24T00:01:22.508 INFO:teuthology.orchestra.run.smithi118.stdout:(68/174): python3-numpy-1.20.1-5.el9.x86_64.rpm 26 MB/s | 4.8 MB 00:00 2024-03-24T00:01:22.533 INFO:teuthology.orchestra.run.smithi118.stdout:(69/174): python3-numpy-f2py-1.20.1-5.el9.x86_6 1.5 MB/s | 217 kB 00:00 2024-03-24T00:01:22.561 INFO:teuthology.orchestra.run.smithi193.stdout:(129/174): python3-jaraco-functools-3.5.0-2.el9 211 kB/s | 19 kB 00:00 2024-03-24T00:01:22.576 INFO:teuthology.orchestra.run.smithi118.stdout:(70/174): python3-oauthlib-3.1.1-5.el9.noarch.r 4.2 MB/s | 174 kB 00:00 2024-03-24T00:01:22.595 INFO:teuthology.orchestra.run.smithi193.stdout:(130/174): python3-jaraco-text-3.2.0-6.el9.noar 214 kB/s | 20 kB 00:00 2024-03-24T00:01:22.601 INFO:teuthology.orchestra.run.smithi118.stdout:(71/174): python3-packaging-20.9-5.el9.noarch.r 1.0 MB/s | 69 kB 00:00 2024-03-24T00:01:22.635 INFO:teuthology.orchestra.run.smithi118.stdout:(72/174): python3-prettytable-0.7.2-27.el9.noar 1.2 MB/s | 41 kB 00:00 2024-03-24T00:01:22.668 INFO:teuthology.orchestra.run.smithi118.stdout:(73/174): python3-protobuf-3.14.0-13.el9.noarch 3.5 MB/s | 238 kB 00:00 2024-03-24T00:01:22.693 INFO:teuthology.orchestra.run.smithi118.stdout:(74/174): python3-pyasn1-0.4.8-6.el9.noarch.rpm 2.4 MB/s | 132 kB 00:00 2024-03-24T00:01:22.695 INFO:teuthology.orchestra.run.smithi193.stdout:(131/174): python3-kubernetes-26.1.0-2.el9.noar 7.6 MB/s | 1.0 MB 00:00 2024-03-24T00:01:22.710 INFO:teuthology.orchestra.run.smithi189.stdout:Transaction test succeeded. 2024-03-24T00:01:22.710 INFO:teuthology.orchestra.run.smithi189.stdout:Running transaction 2024-03-24T00:01:22.721 INFO:teuthology.orchestra.run.smithi193.stdout:(132/174): python3-logutils-0.3.5-21.el9.noarch 369 kB/s | 46 kB 00:00 2024-03-24T00:01:22.727 INFO:teuthology.orchestra.run.smithi118.stdout:(75/174): python3-pyasn1-modules-0.4.8-6.el9.no 3.6 MB/s | 211 kB 00:00 2024-03-24T00:01:22.752 INFO:teuthology.orchestra.run.smithi118.stdout:(76/174): python3-pytz-2021.1-5.el9.noarch.rpm 845 kB/s | 47 kB 00:00 2024-03-24T00:01:22.788 INFO:teuthology.orchestra.run.smithi193.stdout:(133/174): python3-more-itertools-8.12.0-2.el9. 858 kB/s | 79 kB 00:00 2024-03-24T00:01:22.811 INFO:teuthology.orchestra.run.smithi118.stdout:(77/174): python3-requests-oauthlib-1.3.0-12.el 520 kB/s | 43 kB 00:00 2024-03-24T00:01:22.830 INFO:teuthology.orchestra.run.smithi193.stdout:(134/174): python3-natsort-7.1.1-5.el9.noarch.r 530 kB/s | 58 kB 00:00 2024-03-24T00:01:22.897 INFO:teuthology.orchestra.run.smithi193.stdout:(135/174): python3-pecan-1.4.2-3.el9.noarch.rpm 2.4 MB/s | 272 kB 00:00 2024-03-24T00:01:22.930 INFO:teuthology.orchestra.run.smithi193.stdout:(136/174): python3-portend-3.1.0-2.el9.noarch.r 164 kB/s | 16 kB 00:00 2024-03-24T00:01:22.989 INFO:teuthology.orchestra.run.smithi193.stdout:(137/174): python3-pyOpenSSL-21.0.0-1.el9.noarc 977 kB/s | 90 kB 00:00 2024-03-24T00:01:23.014 INFO:teuthology.orchestra.run.smithi193.stdout:(138/174): python3-pyperclip-1.8.2-2.el9.noarch 337 kB/s | 28 kB 00:00 2024-03-24T00:01:23.073 INFO:teuthology.orchestra.run.smithi193.stdout:(139/174): python3-repoze-lru-0.7-16.el9.noarch 368 kB/s | 31 kB 00:00 2024-03-24T00:01:23.115 INFO:teuthology.orchestra.run.smithi193.stdout:(140/174): python3-grpcio-1.46.7-10.el9.x86_64. 2.4 MB/s | 2.0 MB 00:00 2024-03-24T00:01:23.140 INFO:teuthology.orchestra.run.smithi193.stdout:(141/174): python3-routes-2.5.1-5.el9.noarch.rp 1.5 MB/s | 188 kB 00:00 2024-03-24T00:01:23.145 INFO:teuthology.orchestra.run.smithi118.stdout:(78/174): python3-scipy-1.6.2-8.el9.x86_64.rpm 40 MB/s | 16 MB 00:00 2024-03-24T00:01:23.166 INFO:teuthology.orchestra.run.smithi193.stdout:(142/174): python3-rsa-4.9-2.el9.noarch.rpm 641 kB/s | 59 kB 00:00 2024-03-24T00:01:23.170 INFO:teuthology.orchestra.run.smithi118.stdout:(79/174): python3-toml-0.10.2-6.el9.noarch.rpm 112 kB/s | 39 kB 00:00 2024-03-24T00:01:23.203 INFO:teuthology.orchestra.run.smithi118.stdout:(80/174): python3-wcwidth-0.2.5-8.el9.noarch.rp 1.2 MB/s | 41 kB 00:00 2024-03-24T00:01:23.207 INFO:teuthology.orchestra.run.smithi193.stdout:(143/174): python3-tempora-5.0.0-2.el9.noarch.r 387 kB/s | 36 kB 00:00 2024-03-24T00:01:23.224 INFO:teuthology.orchestra.run.smithi193.stdout:(144/174): python3-typing-extensions-4.4.0-2.el 608 kB/s | 51 kB 00:00 2024-03-24T00:01:23.237 INFO:teuthology.orchestra.run.smithi118.stdout:(81/174): socat-1.7.4.1-5.el9.x86_64.rpm 4.5 MB/s | 300 kB 00:00 2024-03-24T00:01:23.266 INFO:teuthology.orchestra.run.smithi193.stdout:(145/174): python3-webob-1.8.7-6.el9.noarch.rpm 2.2 MB/s | 230 kB 00:00 2024-03-24T00:01:23.279 INFO:teuthology.orchestra.run.smithi118.stdout:(82/174): uuid-1.6.2-55.el9.x86_64.rpm 785 kB/s | 56 kB 00:00 2024-03-24T00:01:23.296 INFO:teuthology.orchestra.run.smithi118.stdout:(83/174): xmlstarlet-1.6.1-20.el9.x86_64.rpm 1.1 MB/s | 63 kB 00:00 2024-03-24T00:01:23.300 INFO:teuthology.orchestra.run.smithi193.stdout:(146/174): python3-websocket-client-1.2.3-2.el9 974 kB/s | 90 kB 00:00 2024-03-24T00:01:23.313 INFO:teuthology.orchestra.run.smithi118.stdout:(84/174): zlib-devel-1.2.11-40.el9.x86_64.rpm 1.3 MB/s | 44 kB 00:00 2024-03-24T00:01:23.325 INFO:teuthology.orchestra.run.smithi193.stdout:(147/174): python3-werkzeug-2.0.3-3.el9.1.noarc 4.2 MB/s | 427 kB 00:00 2024-03-24T00:01:23.330 INFO:teuthology.orchestra.run.smithi118.stdout:(85/174): fuse-2.9.9-15.el9.x86_64.rpm 2.4 MB/s | 79 kB 00:00 2024-03-24T00:01:23.358 INFO:teuthology.orchestra.run.smithi193.stdout:(148/174): python3-zc-lockfile-2.0-10.el9.noarc 217 kB/s | 20 kB 00:00 2024-03-24T00:01:23.364 INFO:teuthology.orchestra.run.smithi118.stdout:(86/174): python3-cffi-1.14.5-5.el9.x86_64.rpm 4.8 MB/s | 241 kB 00:00 2024-03-24T00:01:23.401 INFO:teuthology.orchestra.run.smithi193.stdout:(149/174): re2-20211101-3.el9.x86_64.rpm 1.9 MB/s | 192 kB 00:00 2024-03-24T00:01:23.422 INFO:teuthology.orchestra.run.smithi118.stdout:(87/174): python3-cryptography-36.0.1-4.el9.x86 13 MB/s | 1.1 MB 00:00 2024-03-24T00:01:23.447 INFO:teuthology.orchestra.run.smithi118.stdout:(88/174): python3-ply-3.11-14.el9.noarch.rpm 1.4 MB/s | 103 kB 00:00 2024-03-24T00:01:23.473 INFO:teuthology.orchestra.run.smithi118.stdout:(89/174): python3-pycparser-2.20-6.el9.noarch.r 2.5 MB/s | 124 kB 00:00 2024-03-24T00:01:23.507 INFO:teuthology.orchestra.run.smithi118.stdout:(90/174): python3-urllib3-1.26.5-5.el9.noarch.r 3.2 MB/s | 187 kB 00:00 2024-03-24T00:01:23.526 INFO:teuthology.orchestra.run.smithi193.stdout:(150/174): thrift-0.15.0-2.el9.x86_64.rpm 7.9 MB/s | 1.6 MB 00:00 2024-03-24T00:01:23.651 INFO:teuthology.orchestra.run.smithi193.stdout:(151/174): xfsdump-3.1.12-4.el9.x86_64.rpm 2.5 MB/s | 321 kB 00:00 2024-03-24T00:01:23.699 INFO:teuthology.orchestra.run.smithi118.stdout:(91/174): jq-1.6-16.el9.x86_64.rpm 839 kB/s | 187 kB 00:00 2024-03-24T00:01:23.818 INFO:teuthology.orchestra.run.smithi193.stdout:(152/174): librados2-19.0.0-2145.g3be9a5c9.el9. 7.4 MB/s | 3.4 MB 00:00 2024-03-24T00:01:24.077 INFO:teuthology.orchestra.run.smithi193.stdout:(153/174): NetworkManager-1.47.2-1.el9.x86_64.r 5.4 MB/s | 2.3 MB 00:00 2024-03-24T00:01:24.291 INFO:teuthology.orchestra.run.smithi118.stdout:(92/174): ceph-test-19.0.0-2145.g3be9a5c9.el9.x 5.3 MB/s | 48 MB 00:08 2024-03-24T00:01:24.324 INFO:teuthology.orchestra.run.smithi118.stdout:(93/174): libquadmath-11.4.1-3.el9.x86_64.rpm 313 kB/s | 195 kB 00:00 2024-03-24T00:01:24.335 INFO:teuthology.orchestra.run.smithi193.stdout:(154/174): NetworkManager-libnm-1.47.2-1.el9.x8 3.5 MB/s | 1.8 MB 00:00 2024-03-24T00:01:24.361 INFO:teuthology.orchestra.run.smithi193.stdout:(155/174): NetworkManager-team-1.47.2-1.el9.x86 148 kB/s | 42 kB 00:00 2024-03-24T00:01:24.375 INFO:teuthology.orchestra.run.smithi118.stdout:(94/174): libgfortran-11.4.1-3.el9.x86_64.rpm 927 kB/s | 803 kB 00:00 2024-03-24T00:01:24.411 INFO:teuthology.orchestra.run.smithi193.stdout:(156/174): acl-2.3.1-4.el9.x86_64.rpm 1.4 MB/s | 71 kB 00:00 2024-03-24T00:01:24.433 INFO:teuthology.orchestra.run.smithi118.stdout:(95/174): elfutils-libelf-devel-0.190-2.el9.x86 199 kB/s | 21 kB 00:00 2024-03-24T00:01:24.459 INFO:teuthology.orchestra.run.smithi118.stdout:(96/174): libuuid-devel-2.37.4-18.el9.x86_64.rp 307 kB/s | 25 kB 00:00 2024-03-24T00:01:24.461 INFO:teuthology.orchestra.run.smithi193.stdout:(157/174): NetworkManager-tui-1.47.2-1.el9.x86_ 1.9 MB/s | 248 kB 00:00 2024-03-24T00:01:24.484 INFO:teuthology.orchestra.run.smithi118.stdout:(97/174): python3-requests-2.25.1-8.el9.noarch. 652 kB/s | 125 kB 00:00 2024-03-24T00:01:24.486 INFO:teuthology.orchestra.run.smithi193.stdout:(158/174): elfutils-debuginfod-client-0.190-2.e 484 kB/s | 36 kB 00:00 2024-03-24T00:01:24.537 INFO:teuthology.orchestra.run.smithi193.stdout:(159/174): librbd1-19.0.0-2145.g3be9a5c9.el9.x8 2.8 MB/s | 3.1 MB 00:01 2024-03-24T00:01:24.551 INFO:teuthology.orchestra.run.smithi118.stdout:(98/174): python3-devel-3.9.17-2.el9.x86_64.rpm 2.6 MB/s | 245 kB 00:00 2024-03-24T00:01:24.562 INFO:teuthology.orchestra.run.smithi193.stdout:(160/174): elfutils-libelf-0.190-2.el9.x86_64.r 1.9 MB/s | 192 kB 00:00 2024-03-24T00:01:24.596 INFO:teuthology.orchestra.run.smithi193.stdout:(161/174): elfutils-libs-0.190-2.el9.x86_64.rpm 2.3 MB/s | 256 kB 00:00 2024-03-24T00:01:24.601 INFO:teuthology.orchestra.run.smithi118.stdout:(99/174): qatlib-23.11.0-1.el9.x86_64.rpm 4.3 MB/s | 216 kB 00:00 2024-03-24T00:01:24.623 INFO:teuthology.orchestra.run.smithi193.stdout:(162/174): libblkid-2.37.4-18.el9.x86_64.rpm 1.7 MB/s | 108 kB 00:00 2024-03-24T00:01:24.635 INFO:teuthology.orchestra.run.smithi118.stdout:(100/174): python3-jinja2-2.11.3-5.el9.noarch.r 1.6 MB/s | 248 kB 00:00 2024-03-24T00:01:24.648 INFO:teuthology.orchestra.run.smithi193.stdout:(163/174): libacl-2.3.1-4.el9.x86_64.rpm 207 kB/s | 23 kB 00:00 2024-03-24T00:01:24.660 INFO:teuthology.orchestra.run.smithi118.stdout:(101/174): qatlib-service-23.11.0-1.el9.x86_64. 603 kB/s | 35 kB 00:00 2024-03-24T00:01:24.674 INFO:teuthology.orchestra.run.smithi193.stdout:(164/174): libfdisk-2.37.4-18.el9.x86_64.rpm 1.9 MB/s | 155 kB 00:00 2024-03-24T00:01:24.685 INFO:teuthology.orchestra.run.smithi118.stdout:(102/174): qatzip-libs-1.1.2-2.el9.x86_64.rpm 900 kB/s | 44 kB 00:00 2024-03-24T00:01:24.699 INFO:teuthology.orchestra.run.smithi193.stdout:(165/174): libmount-2.37.4-18.el9.x86_64.rpm 1.8 MB/s | 136 kB 00:00 2024-03-24T00:01:24.710 INFO:teuthology.orchestra.run.smithi118.stdout:(103/174): xfsprogs-devel-6.3.0-1.el9.x86_64.rp 1.9 MB/s | 94 kB 00:00 2024-03-24T00:01:24.716 INFO:teuthology.orchestra.run.smithi193.stdout:(166/174): libuuid-2.37.4-18.el9.x86_64.rpm 685 kB/s | 29 kB 00:00 2024-03-24T00:01:24.741 INFO:teuthology.orchestra.run.smithi193.stdout:(167/174): libsmartcols-2.37.4-18.el9.x86_64.rp 690 kB/s | 64 kB 00:00 2024-03-24T00:01:24.827 INFO:teuthology.orchestra.run.smithi118.stdout:(104/174): openssl-devel-3.0.7-27.el9.x86_64.rp 11 MB/s | 4.1 MB 00:00 2024-03-24T00:01:24.925 INFO:teuthology.orchestra.run.smithi193.stdout:(168/174): openssl-3.0.7-27.el9.x86_64.rpm 5.4 MB/s | 1.2 MB 00:00 2024-03-24T00:01:25.059 INFO:teuthology.orchestra.run.smithi193.stdout:(169/174): util-linux-core-2.37.4-18.el9.x86_64 3.4 MB/s | 465 kB 00:00 2024-03-24T00:01:25.086 INFO:teuthology.orchestra.run.smithi118.stdout:(105/174): lua-devel-5.4.4-4.el9.x86_64.rpm 56 kB/s | 22 kB 00:00 2024-03-24T00:01:25.101 INFO:teuthology.orchestra.run.smithi193.stdout:(170/174): openssl-libs-3.0.7-27.el9.x86_64.rpm 5.6 MB/s | 2.2 MB 00:00 2024-03-24T00:01:25.126 INFO:teuthology.orchestra.run.smithi193.stdout:(171/174): util-linux-user-2.37.4-18.el9.x86_64 464 kB/s | 31 kB 00:00 2024-03-24T00:01:25.218 INFO:teuthology.orchestra.run.smithi193.stdout:(172/174): libacl-devel-2.3.1-4.el9.x86_64.rpm 810 kB/s | 74 kB 00:00 2024-03-24T00:01:25.261 INFO:teuthology.orchestra.run.smithi118.stdout:(106/174): protobuf-compiler-3.14.0-13.el9.x86_ 1.5 MB/s | 863 kB 00:00 2024-03-24T00:01:25.268 INFO:teuthology.orchestra.run.smithi193.stdout:(173/174): util-linux-2.37.4-18.el9.x86_64.rpm 4.3 MB/s | 2.3 MB 00:00 2024-03-24T00:01:25.327 INFO:teuthology.orchestra.run.smithi193.stdout:(174/174): xfsprogs-6.3.0-1.el9.x86_64.rpm 4.8 MB/s | 1.1 MB 00:00 2024-03-24T00:01:25.331 INFO:teuthology.orchestra.run.smithi193.stdout:-------------------------------------------------------------------------------- 2024-03-24T00:01:25.331 INFO:teuthology.orchestra.run.smithi193.stdout:Total 12 MB/s | 224 MB 00:18 2024-03-24T00:01:25.386 INFO:teuthology.orchestra.run.smithi118.stdout:(107/174): abseil-cpp-20211102.0-3.el9.x86_64.r 1.1 MB/s | 548 kB 00:00 2024-03-24T00:01:25.470 INFO:teuthology.orchestra.run.smithi118.stdout:(108/174): gperftools-libs-2.9.1-2.el9.x86_64.r 808 kB/s | 309 kB 00:00 2024-03-24T00:01:25.494 INFO:teuthology.orchestra.run.smithi118.stdout:(109/174): grpc-data-1.46.7-10.el9.noarch.rpm 85 kB/s | 19 kB 00:00 2024-03-24T00:01:25.578 INFO:teuthology.orchestra.run.smithi118.stdout:(110/174): libarrow-doc-9.0.0-10.el9.noarch.rpm 241 kB/s | 26 kB 00:00 2024-03-24T00:01:25.603 INFO:teuthology.orchestra.run.smithi118.stdout:(111/174): liboath-2.6.7-2.el9.x86_64.rpm 450 kB/s | 49 kB 00:00 2024-03-24T00:01:25.808 INFO:teuthology.orchestra.run.smithi193.stdout:Running transaction check 2024-03-24T00:01:25.979 INFO:teuthology.orchestra.run.smithi118.stdout:(112/174): libarrow-9.0.0-10.el9.x86_64.rpm 7.5 MB/s | 4.4 MB 00:00 2024-03-24T00:01:26.078 INFO:teuthology.orchestra.run.smithi193.stdout:Transaction check succeeded. 2024-03-24T00:01:26.078 INFO:teuthology.orchestra.run.smithi193.stdout:Running transaction test 2024-03-24T00:01:26.370 INFO:teuthology.orchestra.run.smithi118.stdout:(113/174): libunwind-1.6.2-1.el9.x86_64.rpm 85 kB/s | 67 kB 00:00 2024-03-24T00:01:26.530 INFO:teuthology.orchestra.run.smithi118.stdout:(114/174): luarocks-3.9.2-1.el9.noarch.rpm 163 kB/s | 151 kB 00:00 2024-03-24T00:01:26.622 INFO:teuthology.orchestra.run.smithi118.stdout:(115/174): python3-asyncssh-2.13.2-5.el9.noarch 2.1 MB/s | 548 kB 00:00 2024-03-24T00:01:26.647 INFO:teuthology.orchestra.run.smithi118.stdout:(116/174): python3-bcrypt-3.2.2-1.el9.x86_64.rp 375 kB/s | 43 kB 00:00 2024-03-24T00:01:26.689 INFO:teuthology.orchestra.run.smithi118.stdout:(117/174): parquet-libs-9.0.0-10.el9.x86_64.rpm 1.2 MB/s | 838 kB 00:00 2024-03-24T00:01:26.714 INFO:teuthology.orchestra.run.smithi118.stdout:(118/174): python3-certifi-2021.10.8-2.el9.noar 226 kB/s | 15 kB 00:00 2024-03-24T00:01:26.731 INFO:teuthology.orchestra.run.smithi118.stdout:(119/174): python3-cachetools-4.2.4-1.el9.noarc 297 kB/s | 32 kB 00:00 2024-03-24T00:01:26.764 INFO:teuthology.orchestra.run.smithi118.stdout:(120/174): python3-cheroot-8.6.0-4.el9.noarch.r 2.2 MB/s | 172 kB 00:00 2024-03-24T00:01:26.807 INFO:teuthology.orchestra.run.smithi118.stdout:(121/174): python3-colorama-0.4.4-12.el9.noarch 864 kB/s | 35 kB 00:00 2024-03-24T00:01:26.867 INFO:teuthology.orchestra.run.smithi118.stdout:(122/174): python3-cmd2-2.4.2-2.el9.noarch.rpm 2.2 MB/s | 303 kB 00:00 2024-03-24T00:01:26.902 INFO:teuthology.orchestra.run.smithi118.stdout:(123/174): python3-cherrypy-18.6.1-2.el9.noarch 1.9 MB/s | 358 kB 00:00 2024-03-24T00:01:26.933 INFO:teuthology.orchestra.run.smithi118.stdout:(124/174): python3-google-auth-2.28.2-1.el9.noa 1.6 MB/s | 203 kB 00:00 2024-03-24T00:01:26.966 INFO:teuthology.orchestra.run.smithi118.stdout:(125/174): python3-grpcio-tools-1.46.7-10.el9.x 2.1 MB/s | 144 kB 00:00 2024-03-24T00:01:26.992 INFO:teuthology.orchestra.run.smithi118.stdout:(126/174): python3-jaraco-8.2.1-3.el9.noarch.rp 185 kB/s | 11 kB 00:00 2024-03-24T00:01:27.017 INFO:teuthology.orchestra.run.smithi118.stdout:(127/174): python3-jaraco-classes-3.2.1-5.el9.n 357 kB/s | 18 kB 00:00 2024-03-24T00:01:27.042 INFO:teuthology.orchestra.run.smithi118.stdout:(128/174): python3-jaraco-collections-3.0.0-8.e 468 kB/s | 23 kB 00:00 2024-03-24T00:01:27.067 INFO:teuthology.orchestra.run.smithi118.stdout:(129/174): python3-jaraco-functools-3.5.0-2.el9 395 kB/s | 19 kB 00:00 2024-03-24T00:01:27.251 INFO:teuthology.orchestra.run.smithi118.stdout:(130/174): python3-jaraco-text-3.2.0-6.el9.noar 94 kB/s | 20 kB 00:00 2024-03-24T00:01:27.302 INFO:teuthology.orchestra.run.smithi118.stdout:(131/174): python3-logutils-0.3.5-21.el9.noarch 945 kB/s | 46 kB 00:00 2024-03-24T00:01:27.377 INFO:teuthology.orchestra.run.smithi118.stdout:(132/174): python3-kubernetes-26.1.0-2.el9.noar 3.3 MB/s | 1.0 MB 00:00 2024-03-24T00:01:27.402 INFO:teuthology.orchestra.run.smithi118.stdout:(133/174): python3-more-itertools-8.12.0-2.el9. 792 kB/s | 79 kB 00:00 2024-03-24T00:01:27.428 INFO:teuthology.orchestra.run.smithi118.stdout:(134/174): python3-natsort-7.1.1-5.el9.noarch.r 1.1 MB/s | 58 kB 00:00 2024-03-24T00:01:27.495 INFO:teuthology.orchestra.run.smithi118.stdout:(135/174): python3-grpcio-1.46.7-10.el9.x86_64. 3.3 MB/s | 2.0 MB 00:00 2024-03-24T00:01:27.520 INFO:teuthology.orchestra.run.smithi118.stdout:(136/174): python3-portend-3.1.0-2.el9.noarch.r 180 kB/s | 16 kB 00:00 2024-03-24T00:01:27.554 INFO:teuthology.orchestra.run.smithi118.stdout:(137/174): python3-pecan-1.4.2-3.el9.noarch.rpm 1.8 MB/s | 272 kB 00:00 2024-03-24T00:01:27.629 INFO:teuthology.orchestra.run.smithi118.stdout:(138/174): python3-pyperclip-1.8.2-2.el9.noarch 262 kB/s | 28 kB 00:00 2024-03-24T00:01:27.655 INFO:teuthology.orchestra.run.smithi118.stdout:(139/174): python3-pyOpenSSL-21.0.0-1.el9.noarc 568 kB/s | 90 kB 00:00 2024-03-24T00:01:27.679 INFO:teuthology.orchestra.run.smithi118.stdout:(140/174): python3-repoze-lru-0.7-16.el9.noarch 247 kB/s | 31 kB 00:00 2024-03-24T00:01:27.713 INFO:teuthology.orchestra.run.smithi118.stdout:(141/174): python3-routes-2.5.1-5.el9.noarch.rp 2.2 MB/s | 188 kB 00:00 2024-03-24T00:01:27.739 INFO:teuthology.orchestra.run.smithi118.stdout:(142/174): python3-tempora-5.0.0-2.el9.noarch.r 625 kB/s | 36 kB 00:00 2024-03-24T00:01:27.764 INFO:teuthology.orchestra.run.smithi118.stdout:(143/174): python3-rsa-4.9-2.el9.noarch.rpm 714 kB/s | 59 kB 00:00 2024-03-24T00:01:27.777 INFO:teuthology.orchestra.run.smithi193.stdout:Transaction test succeeded. 2024-03-24T00:01:27.777 INFO:teuthology.orchestra.run.smithi193.stdout:Running transaction 2024-03-24T00:01:27.789 INFO:teuthology.orchestra.run.smithi118.stdout:(144/174): python3-typing-extensions-4.4.0-2.el 683 kB/s | 51 kB 00:00 2024-03-24T00:01:27.840 INFO:teuthology.orchestra.run.smithi118.stdout:(145/174): python3-websocket-client-1.2.3-2.el9 1.2 MB/s | 90 kB 00:00 2024-03-24T00:01:27.873 INFO:teuthology.orchestra.run.smithi118.stdout:(146/174): python3-webob-1.8.7-6.el9.noarch.rpm 1.7 MB/s | 230 kB 00:00 2024-03-24T00:01:27.898 INFO:teuthology.orchestra.run.smithi118.stdout:(147/174): python3-zc-lockfile-2.0-10.el9.noarc 346 kB/s | 20 kB 00:00 2024-03-24T00:01:27.940 INFO:teuthology.orchestra.run.smithi118.stdout:(148/174): python3-werkzeug-2.0.3-3.el9.1.noarc 2.8 MB/s | 427 kB 00:00 2024-03-24T00:01:27.974 INFO:teuthology.orchestra.run.smithi118.stdout:(149/174): re2-20211101-3.el9.x86_64.rpm 1.9 MB/s | 192 kB 00:00 2024-03-24T00:01:28.158 INFO:teuthology.orchestra.run.smithi118.stdout:(150/174): thrift-0.15.0-2.el9.x86_64.rpm 6.1 MB/s | 1.6 MB 00:00 2024-03-24T00:01:28.333 INFO:teuthology.orchestra.run.smithi118.stdout:(151/174): xfsdump-3.1.12-4.el9.x86_64.rpm 1.8 MB/s | 321 kB 00:00 2024-03-24T00:01:28.600 INFO:teuthology.orchestra.run.smithi118.stdout:(152/174): librbd1-19.0.0-2145.g3be9a5c9.el9.x8 5.0 MB/s | 3.1 MB 00:00 2024-03-24T00:01:28.700 INFO:teuthology.orchestra.run.smithi118.stdout:(153/174): librados2-19.0.0-2145.g3be9a5c9.el9. 4.5 MB/s | 3.4 MB 00:00 2024-03-24T00:01:28.817 INFO:teuthology.orchestra.run.smithi118.stdout:(154/174): NetworkManager-team-1.47.2-1.el9.x86 364 kB/s | 42 kB 00:00 2024-03-24T00:01:28.942 INFO:teuthology.orchestra.run.smithi118.stdout:(155/174): NetworkManager-1.47.2-1.el9.x86_64.r 3.8 MB/s | 2.3 MB 00:00 2024-03-24T00:01:29.001 INFO:teuthology.orchestra.run.smithi118.stdout:(156/174): acl-2.3.1-4.el9.x86_64.rpm 1.2 MB/s | 71 kB 00:00 2024-03-24T00:01:29.034 INFO:teuthology.orchestra.run.smithi118.stdout:(157/174): NetworkManager-tui-1.47.2-1.el9.x86_ 1.1 MB/s | 248 kB 00:00 2024-03-24T00:01:29.060 INFO:teuthology.orchestra.run.smithi118.stdout:(158/174): elfutils-debuginfod-client-0.190-2.e 630 kB/s | 36 kB 00:00 2024-03-24T00:01:29.135 INFO:teuthology.orchestra.run.smithi118.stdout:(159/174): NetworkManager-libnm-1.47.2-1.el9.x8 3.4 MB/s | 1.8 MB 00:00 2024-03-24T00:01:29.161 INFO:teuthology.orchestra.run.smithi118.stdout:(160/174): elfutils-libelf-0.190-2.el9.x86_64.r 1.5 MB/s | 192 kB 00:00 2024-03-24T00:01:29.199 INFO:teuthology.orchestra.run.smithi118.stdout:(161/174): elfutils-libs-0.190-2.el9.x86_64.rpm 1.8 MB/s | 256 kB 00:00 2024-03-24T00:01:29.225 INFO:teuthology.orchestra.run.smithi118.stdout:(162/174): libacl-2.3.1-4.el9.x86_64.rpm 259 kB/s | 23 kB 00:00 2024-03-24T00:01:29.275 INFO:teuthology.orchestra.run.smithi118.stdout:(163/174): libblkid-2.37.4-18.el9.x86_64.rpm 957 kB/s | 108 kB 00:00 2024-03-24T00:01:29.308 INFO:teuthology.orchestra.run.smithi118.stdout:(164/174): libmount-2.37.4-18.el9.x86_64.rpm 1.6 MB/s | 136 kB 00:00 2024-03-24T00:01:29.334 INFO:teuthology.orchestra.run.smithi118.stdout:(165/174): libfdisk-2.37.4-18.el9.x86_64.rpm 1.4 MB/s | 155 kB 00:00 2024-03-24T00:01:29.359 INFO:teuthology.orchestra.run.smithi118.stdout:(166/174): libsmartcols-2.37.4-18.el9.x86_64.rp 769 kB/s | 64 kB 00:00 2024-03-24T00:01:29.384 INFO:teuthology.orchestra.run.smithi118.stdout:(167/174): libuuid-2.37.4-18.el9.x86_64.rpm 383 kB/s | 29 kB 00:00 2024-03-24T00:01:29.533 INFO:teuthology.orchestra.run.smithi189.stdout: Preparing : 1/1 2024-03-24T00:01:29.593 INFO:teuthology.orchestra.run.smithi118.stdout:(168/174): openssl-3.0.7-27.el9.x86_64.rpm 4.7 MB/s | 1.2 MB 00:00 2024-03-24T00:01:29.660 INFO:teuthology.orchestra.run.smithi189.stdout: Upgrading : openssl-libs-1:3.0.7-27.el9.x86_64 1/198 2024-03-24T00:01:29.694 INFO:teuthology.orchestra.run.smithi189.stdout: Upgrading : libuuid-2.37.4-18.el9.x86_64 2/198 2024-03-24T00:01:29.725 INFO:teuthology.orchestra.run.smithi189.stdout: Upgrading : libblkid-2.37.4-18.el9.x86_64 3/198 2024-03-24T00:01:29.734 INFO:teuthology.orchestra.run.smithi189.stdout: Running scriptlet: libblkid-2.37.4-18.el9.x86_64 3/198 2024-03-24T00:01:29.736 INFO:teuthology.orchestra.run.smithi118.stdout:(169/174): util-linux-core-2.37.4-18.el9.x86_64 3.2 MB/s | 465 kB 00:00 2024-03-24T00:01:29.774 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 4/198 2024-03-24T00:01:29.810 INFO:teuthology.orchestra.run.smithi118.stdout:(170/174): openssl-libs-3.0.7-27.el9.x86_64.rpm 4.8 MB/s | 2.2 MB 00:00 2024-03-24T00:01:29.828 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 5/198 2024-03-24T00:01:29.835 INFO:teuthology.orchestra.run.smithi118.stdout:(171/174): util-linux-user-2.37.4-18.el9.x86_64 319 kB/s | 31 kB 00:00 2024-03-24T00:01:29.861 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : thrift-0.15.0-2.el9.x86_64 6/198 2024-03-24T00:01:29.887 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 7/198 2024-03-24T00:01:29.902 INFO:teuthology.orchestra.run.smithi118.stdout:(172/174): util-linux-2.37.4-18.el9.x86_64.rpm 4.4 MB/s | 2.3 MB 00:00 2024-03-24T00:01:29.929 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 8/198 2024-03-24T00:01:30.011 INFO:teuthology.orchestra.run.smithi118.stdout:(173/174): libacl-devel-2.3.1-4.el9.x86_64.rpm 426 kB/s | 74 kB 00:00 2024-03-24T00:01:30.053 INFO:teuthology.orchestra.run.smithi118.stdout:(174/174): xfsprogs-6.3.0-1.el9.x86_64.rpm 4.4 MB/s | 1.1 MB 00:00 2024-03-24T00:01:30.068 INFO:teuthology.orchestra.run.smithi118.stdout:-------------------------------------------------------------------------------- 2024-03-24T00:01:30.068 INFO:teuthology.orchestra.run.smithi118.stdout:Total 9.6 MB/s | 224 MB 00:23 2024-03-24T00:01:30.250 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 9/198 2024-03-24T00:01:30.299 INFO:teuthology.orchestra.run.smithi189.stdout: Upgrading : librados2-2:19.0.0-2145.g3be9a5c9.el9.x86_64 10/198 2024-03-24T00:01:30.403 INFO:teuthology.orchestra.run.smithi189.stdout: Running scriptlet: librados2-2:19.0.0-2145.g3be9a5c9.el9.x86_64 10/198 2024-03-24T00:01:30.433 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : libcephfs2-2:19.0.0-2145.g3be9a5c9.el9.x86_64 11/198 2024-03-24T00:01:30.507 INFO:teuthology.orchestra.run.smithi189.stdout: Running scriptlet: libcephfs2-2:19.0.0-2145.g3be9a5c9.el9.x86_64 11/198 2024-03-24T00:01:30.537 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : python3-rados-2:19.0.0-2145.g3be9a5c9.el9.x86_64 12/198 2024-03-24T00:01:30.574 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : postgresql-private-libs-13.11-1.el9.x86_64 13/198 2024-03-24T00:01:30.745 INFO:teuthology.orchestra.run.smithi189.stdout: Upgrading : elfutils-libelf-0.190-2.el9.x86_64 14/198 2024-03-24T00:01:30.793 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : postgresql-13.11-1.el9.x86_64 15/198 2024-03-24T00:01:30.821 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : libcephsqlite-2:19.0.0-2145.g3be9a5c9.el9.x86_64 16/198 2024-03-24T00:01:30.907 INFO:teuthology.orchestra.run.smithi189.stdout: Running scriptlet: libcephsqlite-2:19.0.0-2145.g3be9a5c9.el9.x86_64 16/198 2024-03-24T00:01:30.930 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : libradosstriper1-2:19.0.0-2145.g3be9a5c9.el9.x86 17/198 2024-03-24T00:01:30.977 INFO:teuthology.orchestra.run.smithi189.stdout: Running scriptlet: libradosstriper1-2:19.0.0-2145.g3be9a5c9.el9.x86 17/198 2024-03-24T00:01:31.025 INFO:teuthology.orchestra.run.smithi189.stdout: Upgrading : libmount-2.37.4-18.el9.x86_64 18/198 2024-03-24T00:01:31.048 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : qatlib-service-23.11.0-1.el9.x86_64 19/198 2024-03-24T00:01:31.064 INFO:teuthology.orchestra.run.smithi189.stdout: Running scriptlet: qatlib-service-23.11.0-1.el9.x86_64 19/198 2024-03-24T00:01:31.206 INFO:teuthology.orchestra.run.smithi189.stdout: Running scriptlet: qatlib-23.11.0-1.el9.x86_64 20/198 2024-03-24T00:01:31.237 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : qatlib-23.11.0-1.el9.x86_64 20/198 2024-03-24T00:01:31.269 INFO:teuthology.orchestra.run.smithi189.stdout: Upgrading : libsmartcols-2.37.4-18.el9.x86_64 21/198 2024-03-24T00:01:31.376 INFO:teuthology.orchestra.run.smithi189.stdout: Upgrading : libacl-2.3.1-4.el9.x86_64 22/198 2024-03-24T00:01:31.472 INFO:teuthology.orchestra.run.smithi189.stdout: Upgrading : NetworkManager-libnm-1:1.47.2-1.el9.x86_64 23/198 2024-03-24T00:01:31.517 INFO:teuthology.orchestra.run.smithi118.stdout:Running transaction check 2024-03-24T00:01:31.532 INFO:teuthology.orchestra.run.smithi189.stdout: Running scriptlet: NetworkManager-libnm-1:1.47.2-1.el9.x86_64 23/198 2024-03-24T00:01:31.595 INFO:teuthology.orchestra.run.smithi189.stdout: Running scriptlet: NetworkManager-1:1.47.2-1.el9.x86_64 24/198 2024-03-24T00:01:31.666 INFO:teuthology.orchestra.run.smithi189.stdout: Upgrading : NetworkManager-1:1.47.2-1.el9.x86_64 24/198 2024-03-24T00:01:31.881 INFO:teuthology.orchestra.run.smithi189.stdout: Running scriptlet: NetworkManager-1:1.47.2-1.el9.x86_64 24/198 2024-03-24T00:01:31.901 INFO:teuthology.orchestra.run.smithi118.stdout:Transaction check succeeded. 2024-03-24T00:01:31.901 INFO:teuthology.orchestra.run.smithi118.stdout:Running transaction test 2024-03-24T00:01:32.010 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : re2-1:20211101-3.el9.x86_64 25/198 2024-03-24T00:01:32.098 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : libarrow-9.0.0-10.el9.x86_64 26/198 2024-03-24T00:01:32.137 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 27/198 2024-03-24T00:01:32.181 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : liboath-2.6.7-2.el9.x86_64 28/198 2024-03-24T00:01:32.216 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : libquadmath-11.4.1-3.el9.x86_64 29/198 2024-03-24T00:01:32.282 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : libgfortran-11.4.1-3.el9.x86_64 30/198 2024-03-24T00:01:32.336 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : python3-urllib3-1.26.5-5.el9.noarch 31/198 2024-03-24T00:01:32.374 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 32/198 2024-03-24T00:01:32.439 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 33/198 2024-03-24T00:01:32.474 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 34/198 2024-03-24T00:01:32.517 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 35/198 2024-03-24T00:01:32.549 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : protobuf-3.14.0-13.el9.x86_64 36/198 2024-03-24T00:01:32.586 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : lua-5.4.4-4.el9.x86_64 37/198 2024-03-24T00:01:32.655 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 38/198 2024-03-24T00:01:32.682 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : python3-ceph-common-2:19.0.0-2145.g3be9a5c9.el9. 39/198 2024-03-24T00:01:32.718 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : python3-ceph-argparse-2:19.0.0-2145.g3be9a5c9.el 40/198 2024-03-24T00:01:32.746 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : python3-cephfs-2:19.0.0-2145.g3be9a5c9.el9.x86_6 41/198 2024-03-24T00:01:32.815 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : lua-devel-5.4.4-4.el9.x86_64 42/198 2024-03-24T00:01:32.861 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : luarocks-3.9.2-1.el9.noarch 43/198 2024-03-24T00:01:32.934 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : protobuf-compiler-3.14.0-13.el9.x86_64 44/198 2024-03-24T00:01:33.067 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 45/198 2024-03-24T00:01:33.113 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 46/198 2024-03-24T00:01:33.630 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : python3-rsa-4.9-2.el9.noarch 47/198 2024-03-24T00:01:33.708 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 48/198 2024-03-24T00:01:33.778 INFO:teuthology.orchestra.run.smithi118.stdout:Transaction test succeeded. 2024-03-24T00:01:33.779 INFO:teuthology.orchestra.run.smithi118.stdout:Running transaction 2024-03-24T00:01:33.784 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : python3-jinja2-2.11.3-5.el9.noarch 49/198 2024-03-24T00:01:33.820 INFO:teuthology.orchestra.run.smithi189.stdout: Upgrading : util-linux-core-2.37.4-18.el9.x86_64 50/198 2024-03-24T00:01:33.828 INFO:teuthology.orchestra.run.smithi189.stdout: Running scriptlet: util-linux-core-2.37.4-18.el9.x86_64 50/198 2024-03-24T00:01:33.860 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : qatzip-libs-1.1.2-2.el9.x86_64 51/198 2024-03-24T00:01:33.909 INFO:teuthology.orchestra.run.smithi189.stdout: Upgrading : elfutils-libs-0.190-2.el9.x86_64 52/198 2024-03-24T00:01:33.946 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : librados-devel-2:19.0.0-2145.g3be9a5c9.el9.x86_6 53/198 2024-03-24T00:01:33.976 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 54/198 2024-03-24T00:01:34.390 INFO:teuthology.orchestra.run.smithi189.stdout: Upgrading : libfdisk-2.37.4-18.el9.x86_64 55/198 2024-03-24T00:01:34.592 INFO:teuthology.orchestra.run.smithi189.stdout: Upgrading : util-linux-2.37.4-18.el9.x86_64 56/198 2024-03-24T00:01:34.654 INFO:teuthology.orchestra.run.smithi189.stdout: Upgrading : xfsprogs-6.3.0-1.el9.x86_64 57/198 2024-03-24T00:01:34.698 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : libuuid-devel-2.37.4-18.el9.x86_64 58/198 2024-03-24T00:01:34.722 INFO:teuthology.orchestra.run.smithi193.stdout: Preparing : 1/1 2024-03-24T00:01:34.839 INFO:teuthology.orchestra.run.smithi193.stdout: Upgrading : openssl-libs-1:3.0.7-27.el9.x86_64 1/198 2024-03-24T00:01:34.847 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : socat-1.7.4.1-5.el9.x86_64 59/198 2024-03-24T00:01:34.900 INFO:teuthology.orchestra.run.smithi189.stdout: Upgrading : openssl-1:3.0.7-27.el9.x86_64 60/198 2024-03-24T00:01:34.955 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 61/198 2024-03-24T00:01:35.019 INFO:teuthology.orchestra.run.smithi193.stdout: Upgrading : libuuid-2.37.4-18.el9.x86_64 2/198 2024-03-24T00:01:35.028 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 62/198 2024-03-24T00:01:35.059 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : python3-webob-1.8.7-6.el9.noarch 63/198 2024-03-24T00:01:35.096 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : python3-typing-extensions-4.4.0-2.el9.noarch 64/198 2024-03-24T00:01:35.142 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 65/198 2024-03-24T00:01:35.179 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 66/198 2024-03-24T00:01:35.179 INFO:teuthology.orchestra.run.smithi193.stdout: Upgrading : libblkid-2.37.4-18.el9.x86_64 3/198 2024-03-24T00:01:35.191 INFO:teuthology.orchestra.run.smithi193.stdout: Running scriptlet: libblkid-2.37.4-18.el9.x86_64 3/198 2024-03-24T00:01:35.228 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : python3-pyperclip-1.8.2-2.el9.noarch 67/198 2024-03-24T00:01:35.238 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 4/198 2024-03-24T00:01:35.270 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 68/198 2024-03-24T00:01:35.302 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 5/198 2024-03-24T00:01:35.342 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : thrift-0.15.0-2.el9.x86_64 6/198 2024-03-24T00:01:35.367 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 7/198 2024-03-24T00:01:35.403 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 69/198 2024-03-24T00:01:35.411 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 8/198 2024-03-24T00:01:35.451 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 70/198 2024-03-24T00:01:35.482 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : python3-colorama-0.4.4-12.el9.noarch 71/198 2024-03-24T00:01:35.522 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : python3-certifi-2021.10.8-2.el9.noarch 72/198 2024-03-24T00:01:35.616 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 73/198 2024-03-24T00:01:35.656 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : python3-google-auth-1:2.28.2-1.el9.noarch 74/198 2024-03-24T00:01:35.696 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 75/198 2024-03-24T00:01:35.730 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : gperftools-libs-2.9.1-2.el9.x86_64 76/198 2024-03-24T00:01:35.737 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 9/198 2024-03-24T00:01:35.754 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : libarrow-doc-9.0.0-10.el9.noarch 77/198 2024-03-24T00:01:35.818 INFO:teuthology.orchestra.run.smithi193.stdout: Upgrading : librados2-2:19.0.0-2145.g3be9a5c9.el9.x86_64 10/198 2024-03-24T00:01:35.826 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : grpc-data-1.46.7-10.el9.noarch 78/198 2024-03-24T00:01:35.939 INFO:teuthology.orchestra.run.smithi193.stdout: Running scriptlet: librados2-2:19.0.0-2145.g3be9a5c9.el9.x86_64 10/198 2024-03-24T00:01:35.947 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : abseil-cpp-20211102.0-3.el9.x86_64 79/198 2024-03-24T00:01:35.977 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : libcephfs2-2:19.0.0-2145.g3be9a5c9.el9.x86_64 11/198 2024-03-24T00:01:36.045 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : python3-grpcio-1.46.7-10.el9.x86_64 80/198 2024-03-24T00:01:36.069 INFO:teuthology.orchestra.run.smithi193.stdout: Running scriptlet: libcephfs2-2:19.0.0-2145.g3be9a5c9.el9.x86_64 11/198 2024-03-24T00:01:36.087 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : python3-devel-3.9.17-2.el9.x86_64 81/198 2024-03-24T00:01:36.106 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : python3-rados-2:19.0.0-2145.g3be9a5c9.el9.x86_64 12/198 2024-03-24T00:01:36.141 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : postgresql-private-libs-13.11-1.el9.x86_64 13/198 2024-03-24T00:01:36.156 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : python3-ply-3.11-14.el9.noarch 82/198 2024-03-24T00:01:36.224 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 83/198 2024-03-24T00:01:36.314 INFO:teuthology.orchestra.run.smithi193.stdout: Upgrading : elfutils-libelf-0.190-2.el9.x86_64 14/198 2024-03-24T00:01:36.371 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : postgresql-13.11-1.el9.x86_64 15/198 2024-03-24T00:01:36.399 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : libcephsqlite-2:19.0.0-2145.g3be9a5c9.el9.x86_64 16/198 2024-03-24T00:01:36.417 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 84/198 2024-03-24T00:01:36.458 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 85/198 2024-03-24T00:01:36.502 INFO:teuthology.orchestra.run.smithi193.stdout: Running scriptlet: libcephsqlite-2:19.0.0-2145.g3be9a5c9.el9.x86_64 16/198 2024-03-24T00:01:36.526 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : libradosstriper1-2:19.0.0-2145.g3be9a5c9.el9.x86 17/198 2024-03-24T00:01:36.586 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 86/198 2024-03-24T00:01:36.589 INFO:teuthology.orchestra.run.smithi193.stdout: Running scriptlet: libradosstriper1-2:19.0.0-2145.g3be9a5c9.el9.x86 17/198 2024-03-24T00:01:36.629 INFO:teuthology.orchestra.run.smithi193.stdout: Upgrading : libmount-2.37.4-18.el9.x86_64 18/198 2024-03-24T00:01:36.631 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : python3-asyncssh-2.13.2-5.el9.noarch 87/198 2024-03-24T00:01:36.652 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : qatlib-service-23.11.0-1.el9.x86_64 19/198 2024-03-24T00:01:36.663 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 88/198 2024-03-24T00:01:36.668 INFO:teuthology.orchestra.run.smithi193.stdout: Running scriptlet: qatlib-service-23.11.0-1.el9.x86_64 19/198 2024-03-24T00:01:36.694 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : fuse-2.9.9-15.el9.x86_64 89/198 2024-03-24T00:01:36.722 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : zlib-devel-1.2.11-40.el9.x86_64 90/198 2024-03-24T00:01:36.756 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 91/198 2024-03-24T00:01:36.795 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : uuid-1.6.2-55.el9.x86_64 92/198 2024-03-24T00:01:36.827 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : python3-wcwidth-0.2.5-8.el9.noarch 93/198 2024-03-24T00:01:36.852 INFO:teuthology.orchestra.run.smithi193.stdout: Running scriptlet: qatlib-23.11.0-1.el9.x86_64 20/198 2024-03-24T00:01:36.857 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 94/198 2024-03-24T00:01:36.883 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : qatlib-23.11.0-1.el9.x86_64 20/198 2024-03-24T00:01:36.924 INFO:teuthology.orchestra.run.smithi193.stdout: Upgrading : libsmartcols-2.37.4-18.el9.x86_64 21/198 2024-03-24T00:01:36.934 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 95/198 2024-03-24T00:01:36.976 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 96/198 2024-03-24T00:01:37.009 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 97/198 2024-03-24T00:01:37.029 INFO:teuthology.orchestra.run.smithi193.stdout: Upgrading : libacl-2.3.1-4.el9.x86_64 22/198 2024-03-24T00:01:37.050 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 98/198 2024-03-24T00:01:37.085 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 99/198 2024-03-24T00:01:37.135 INFO:teuthology.orchestra.run.smithi193.stdout: Upgrading : NetworkManager-libnm-1:1.47.2-1.el9.x86_64 23/198 2024-03-24T00:01:37.175 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 100/198 2024-03-24T00:01:37.195 INFO:teuthology.orchestra.run.smithi193.stdout: Running scriptlet: NetworkManager-libnm-1:1.47.2-1.el9.x86_64 23/198 2024-03-24T00:01:37.264 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 101/198 2024-03-24T00:01:37.273 INFO:teuthology.orchestra.run.smithi193.stdout: Running scriptlet: NetworkManager-1:1.47.2-1.el9.x86_64 24/198 2024-03-24T00:01:37.307 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : python3-protobuf-3.14.0-13.el9.noarch 102/198 2024-03-24T00:01:37.338 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : python3-grpcio-tools-1.46.7-10.el9.x86_64 103/198 2024-03-24T00:01:37.346 INFO:teuthology.orchestra.run.smithi193.stdout: Upgrading : NetworkManager-1:1.47.2-1.el9.x86_64 24/198 2024-03-24T00:01:37.385 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 104/198 2024-03-24T00:01:37.483 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : python3-packaging-20.9-5.el9.noarch 105/198 2024-03-24T00:01:37.532 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 106/198 2024-03-24T00:01:37.548 INFO:teuthology.orchestra.run.smithi193.stdout: Running scriptlet: NetworkManager-1:1.47.2-1.el9.x86_64 24/198 2024-03-24T00:01:37.660 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : re2-1:20211101-3.el9.x86_64 25/198 2024-03-24T00:01:37.755 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : libarrow-9.0.0-10.el9.x86_64 26/198 2024-03-24T00:01:37.793 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 27/198 2024-03-24T00:01:37.820 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : liboath-2.6.7-2.el9.x86_64 28/198 2024-03-24T00:01:37.856 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : libquadmath-11.4.1-3.el9.x86_64 29/198 2024-03-24T00:01:37.929 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : libgfortran-11.4.1-3.el9.x86_64 30/198 2024-03-24T00:01:37.992 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : python3-urllib3-1.26.5-5.el9.noarch 31/198 2024-03-24T00:01:38.030 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 32/198 2024-03-24T00:01:38.096 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 33/198 2024-03-24T00:01:38.129 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 34/198 2024-03-24T00:01:38.173 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 35/198 2024-03-24T00:01:38.206 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : protobuf-3.14.0-13.el9.x86_64 36/198 2024-03-24T00:01:38.234 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : lua-5.4.4-4.el9.x86_64 37/198 2024-03-24T00:01:38.250 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 107/198 2024-03-24T00:01:38.295 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 108/198 2024-03-24T00:01:38.305 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 38/198 2024-03-24T00:01:38.329 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 109/198 2024-03-24T00:01:38.339 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : python3-ceph-common-2:19.0.0-2145.g3be9a5c9.el9. 39/198 2024-03-24T00:01:38.377 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 110/198 2024-03-24T00:01:38.384 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : python3-ceph-argparse-2:19.0.0-2145.g3be9a5c9.el 40/198 2024-03-24T00:01:38.420 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : python3-cephfs-2:19.0.0-2145.g3be9a5c9.el9.x86_6 41/198 2024-03-24T00:01:38.466 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : python3-attrs-20.3.0-7.el9.noarch 111/198 2024-03-24T00:01:38.497 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : python3-cmd2-2.4.2-2.el9.noarch 112/198 2024-03-24T00:01:38.498 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : lua-devel-5.4.4-4.el9.x86_64 42/198 2024-03-24T00:01:38.564 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : luarocks-3.9.2-1.el9.noarch 43/198 2024-03-24T00:01:38.642 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : protobuf-compiler-3.14.0-13.el9.x86_64 44/198 2024-03-24T00:01:38.677 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : openblas-0.3.21-2.el9.x86_64 113/198 2024-03-24T00:01:38.702 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : openblas-openmp-0.3.21-2.el9.x86_64 114/198 2024-03-24T00:01:38.779 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 45/198 2024-03-24T00:01:38.780 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 115/198 2024-03-24T00:01:38.830 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 46/198 2024-03-24T00:01:39.368 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : python3-rsa-4.9-2.el9.noarch 47/198 2024-03-24T00:01:39.422 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 116/198 2024-03-24T00:01:39.451 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 48/198 2024-03-24T00:01:39.526 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : python3-jinja2-2.11.3-5.el9.noarch 49/198 2024-03-24T00:01:39.531 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 117/198 2024-03-24T00:01:39.595 INFO:teuthology.orchestra.run.smithi193.stdout: Upgrading : util-linux-core-2.37.4-18.el9.x86_64 50/198 2024-03-24T00:01:39.604 INFO:teuthology.orchestra.run.smithi193.stdout: Running scriptlet: util-linux-core-2.37.4-18.el9.x86_64 50/198 2024-03-24T00:01:39.635 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : qatzip-libs-1.1.2-2.el9.x86_64 51/198 2024-03-24T00:01:39.675 INFO:teuthology.orchestra.run.smithi193.stdout: Upgrading : elfutils-libs-0.190-2.el9.x86_64 52/198 2024-03-24T00:01:39.714 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : librados-devel-2:19.0.0-2145.g3be9a5c9.el9.x86_6 53/198 2024-03-24T00:01:39.744 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 54/198 2024-03-24T00:01:39.979 INFO:teuthology.orchestra.run.smithi193.stdout: Upgrading : libfdisk-2.37.4-18.el9.x86_64 55/198 2024-03-24T00:01:40.167 INFO:teuthology.orchestra.run.smithi193.stdout: Upgrading : util-linux-2.37.4-18.el9.x86_64 56/198 2024-03-24T00:01:40.236 INFO:teuthology.orchestra.run.smithi193.stdout: Upgrading : xfsprogs-6.3.0-1.el9.x86_64 57/198 2024-03-24T00:01:40.280 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : libuuid-devel-2.37.4-18.el9.x86_64 58/198 2024-03-24T00:01:40.405 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : socat-1.7.4.1-5.el9.x86_64 59/198 2024-03-24T00:01:40.466 INFO:teuthology.orchestra.run.smithi193.stdout: Upgrading : openssl-1:3.0.7-27.el9.x86_64 60/198 2024-03-24T00:01:40.529 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 61/198 2024-03-24T00:01:40.619 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 62/198 2024-03-24T00:01:40.650 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : python3-webob-1.8.7-6.el9.noarch 63/198 2024-03-24T00:01:40.687 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : python3-typing-extensions-4.4.0-2.el9.noarch 64/198 2024-03-24T00:01:40.734 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 65/198 2024-03-24T00:01:40.737 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 118/198 2024-03-24T00:01:40.770 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 66/198 2024-03-24T00:01:40.813 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : python3-pyperclip-1.8.2-2.el9.noarch 67/198 2024-03-24T00:01:40.819 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 119/198 2024-03-24T00:01:40.850 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : oniguruma-6.9.6-1.el9.5.x86_64 120/198 2024-03-24T00:01:40.855 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 68/198 2024-03-24T00:01:40.881 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : jq-1.6-16.el9.x86_64 121/198 2024-03-24T00:01:40.908 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : libzstd-devel-1.5.1-2.el9.x86_64 122/198 2024-03-24T00:01:40.988 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 69/198 2024-03-24T00:01:41.036 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 70/198 2024-03-24T00:01:41.066 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : python3-colorama-0.4.4-12.el9.noarch 71/198 2024-03-24T00:01:41.107 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : python3-certifi-2021.10.8-2.el9.noarch 72/198 2024-03-24T00:01:41.202 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 73/198 2024-03-24T00:01:41.221 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 123/198 2024-03-24T00:01:41.242 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : python3-google-auth-1:2.28.2-1.el9.noarch 74/198 2024-03-24T00:01:41.255 INFO:teuthology.orchestra.run.smithi189.stdout: Upgrading : librbd1-2:19.0.0-2145.g3be9a5c9.el9.x86_64 124/198 2024-03-24T00:01:41.281 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 75/198 2024-03-24T00:01:41.307 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : gperftools-libs-2.9.1-2.el9.x86_64 76/198 2024-03-24T00:01:41.329 INFO:teuthology.orchestra.run.smithi189.stdout: Running scriptlet: librbd1-2:19.0.0-2145.g3be9a5c9.el9.x86_64 124/198 2024-03-24T00:01:41.332 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : libarrow-doc-9.0.0-10.el9.noarch 77/198 2024-03-24T00:01:41.360 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : python3-rbd-2:19.0.0-2145.g3be9a5c9.el9.x86_64 125/198 2024-03-24T00:01:41.396 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 126/198 2024-03-24T00:01:41.404 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : grpc-data-1.46.7-10.el9.noarch 78/198 2024-03-24T00:01:41.460 INFO:teuthology.orchestra.run.smithi118.stdout: Preparing : 1/1 2024-03-24T00:01:41.524 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : abseil-cpp-20211102.0-3.el9.x86_64 79/198 2024-03-24T00:01:41.632 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : python3-grpcio-1.46.7-10.el9.x86_64 80/198 2024-03-24T00:01:41.636 INFO:teuthology.orchestra.run.smithi118.stdout: Upgrading : openssl-libs-1:3.0.7-27.el9.x86_64 1/198 2024-03-24T00:01:41.682 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : python3-devel-3.9.17-2.el9.x86_64 81/198 2024-03-24T00:01:41.718 INFO:teuthology.orchestra.run.smithi118.stdout: Upgrading : libuuid-2.37.4-18.el9.x86_64 2/198 2024-03-24T00:01:41.731 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : python3-ply-3.11-14.el9.noarch 82/198 2024-03-24T00:01:41.787 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 83/198 2024-03-24T00:01:41.807 INFO:teuthology.orchestra.run.smithi118.stdout: Upgrading : libblkid-2.37.4-18.el9.x86_64 3/198 2024-03-24T00:01:41.851 INFO:teuthology.orchestra.run.smithi118.stdout: Running scriptlet: libblkid-2.37.4-18.el9.x86_64 3/198 2024-03-24T00:01:41.856 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : parquet-libs-9.0.0-10.el9.x86_64 127/198 2024-03-24T00:01:41.891 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : librgw2-2:19.0.0-2145.g3be9a5c9.el9.x86_64 128/198 2024-03-24T00:01:41.937 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 4/198 2024-03-24T00:01:41.963 INFO:teuthology.orchestra.run.smithi189.stdout: Running scriptlet: librgw2-2:19.0.0-2145.g3be9a5c9.el9.x86_64 128/198 2024-03-24T00:01:41.969 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 84/198 2024-03-24T00:01:41.993 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : python3-rgw-2:19.0.0-2145.g3be9a5c9.el9.x86_64 129/198 2024-03-24T00:01:42.021 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 85/198 2024-03-24T00:01:42.126 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 5/198 2024-03-24T00:01:42.148 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 86/198 2024-03-24T00:01:42.185 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : python3-asyncssh-2.13.2-5.el9.noarch 87/198 2024-03-24T00:01:42.189 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : thrift-0.15.0-2.el9.x86_64 6/198 2024-03-24T00:01:42.217 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 88/198 2024-03-24T00:01:42.231 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 7/198 2024-03-24T00:01:42.248 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : fuse-2.9.9-15.el9.x86_64 89/198 2024-03-24T00:01:42.276 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : zlib-devel-1.2.11-40.el9.x86_64 90/198 2024-03-24T00:01:42.311 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 91/198 2024-03-24T00:01:42.336 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 8/198 2024-03-24T00:01:42.358 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : uuid-1.6.2-55.el9.x86_64 92/198 2024-03-24T00:01:42.398 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : python3-wcwidth-0.2.5-8.el9.noarch 93/198 2024-03-24T00:01:42.428 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 94/198 2024-03-24T00:01:42.504 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 95/198 2024-03-24T00:01:42.539 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 96/198 2024-03-24T00:01:42.580 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 97/198 2024-03-24T00:01:42.622 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 98/198 2024-03-24T00:01:42.665 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 99/198 2024-03-24T00:01:42.762 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 100/198 2024-03-24T00:01:42.784 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 9/198 2024-03-24T00:01:42.844 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 101/198 2024-03-24T00:01:42.871 INFO:teuthology.orchestra.run.smithi118.stdout: Upgrading : librados2-2:19.0.0-2145.g3be9a5c9.el9.x86_64 10/198 2024-03-24T00:01:42.896 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : python3-protobuf-3.14.0-13.el9.noarch 102/198 2024-03-24T00:01:42.928 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : python3-grpcio-tools-1.46.7-10.el9.x86_64 103/198 2024-03-24T00:01:42.974 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 104/198 2024-03-24T00:01:43.072 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : python3-packaging-20.9-5.el9.noarch 105/198 2024-03-24T00:01:43.085 INFO:teuthology.orchestra.run.smithi118.stdout: Running scriptlet: librados2-2:19.0.0-2145.g3be9a5c9.el9.x86_64 10/198 2024-03-24T00:01:43.114 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : libcephfs2-2:19.0.0-2145.g3be9a5c9.el9.x86_64 11/198 2024-03-24T00:01:43.121 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 106/198 2024-03-24T00:01:43.271 INFO:teuthology.orchestra.run.smithi118.stdout: Running scriptlet: libcephfs2-2:19.0.0-2145.g3be9a5c9.el9.x86_64 11/198 2024-03-24T00:01:43.313 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : python3-rados-2:19.0.0-2145.g3be9a5c9.el9.x86_64 12/198 2024-03-24T00:01:43.372 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : postgresql-private-libs-13.11-1.el9.x86_64 13/198 2024-03-24T00:01:43.663 INFO:teuthology.orchestra.run.smithi118.stdout: Upgrading : elfutils-libelf-0.190-2.el9.x86_64 14/198 2024-03-24T00:01:43.760 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : postgresql-13.11-1.el9.x86_64 15/198 2024-03-24T00:01:43.794 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : libcephsqlite-2:19.0.0-2145.g3be9a5c9.el9.x86_64 16/198 2024-03-24T00:01:43.847 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 107/198 2024-03-24T00:01:43.902 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 108/198 2024-03-24T00:01:43.935 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 109/198 2024-03-24T00:01:43.984 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 110/198 2024-03-24T00:01:43.991 INFO:teuthology.orchestra.run.smithi118.stdout: Running scriptlet: libcephsqlite-2:19.0.0-2145.g3be9a5c9.el9.x86_64 16/198 2024-03-24T00:01:44.021 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : libradosstriper1-2:19.0.0-2145.g3be9a5c9.el9.x86 17/198 2024-03-24T00:01:44.071 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : python3-attrs-20.3.0-7.el9.noarch 111/198 2024-03-24T00:01:44.119 INFO:teuthology.orchestra.run.smithi118.stdout: Running scriptlet: libradosstriper1-2:19.0.0-2145.g3be9a5c9.el9.x86 17/198 2024-03-24T00:01:44.120 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : python3-cmd2-2.4.2-2.el9.noarch 112/198 2024-03-24T00:01:44.172 INFO:teuthology.orchestra.run.smithi118.stdout: Upgrading : libmount-2.37.4-18.el9.x86_64 18/198 2024-03-24T00:01:44.203 INFO:teuthology.orchestra.run.smithi189.stdout: Running scriptlet: ceph-common-2:19.0.0-2145.g3be9a5c9.el9.x86_64 130/198 2024-03-24T00:01:44.214 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : qatlib-service-23.11.0-1.el9.x86_64 19/198 2024-03-24T00:01:44.239 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : ceph-common-2:19.0.0-2145.g3be9a5c9.el9.x86_64 130/198 2024-03-24T00:01:44.285 INFO:teuthology.orchestra.run.smithi118.stdout: Running scriptlet: qatlib-service-23.11.0-1.el9.x86_64 19/198 2024-03-24T00:01:44.297 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : openblas-0.3.21-2.el9.x86_64 113/198 2024-03-24T00:01:44.326 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : openblas-openmp-0.3.21-2.el9.x86_64 114/198 2024-03-24T00:01:44.404 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 115/198 2024-03-24T00:01:44.598 INFO:teuthology.orchestra.run.smithi118.stdout: Running scriptlet: qatlib-23.11.0-1.el9.x86_64 20/198 2024-03-24T00:01:44.656 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : qatlib-23.11.0-1.el9.x86_64 20/198 2024-03-24T00:01:44.698 INFO:teuthology.orchestra.run.smithi118.stdout: Upgrading : libsmartcols-2.37.4-18.el9.x86_64 21/198 2024-03-24T00:01:44.776 INFO:teuthology.orchestra.run.smithi189.stdout: Running scriptlet: ceph-common-2:19.0.0-2145.g3be9a5c9.el9.x86_64 130/198 2024-03-24T00:01:44.809 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : ceph-base-2:19.0.0-2145.g3be9a5c9.el9.x86_64 131/198 2024-03-24T00:01:44.882 INFO:teuthology.orchestra.run.smithi189.stdout: Running scriptlet: ceph-base-2:19.0.0-2145.g3be9a5c9.el9.x86_64 131/198 2024-03-24T00:01:44.882 INFO:teuthology.orchestra.run.smithi189.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-03-24T00:01:44.882 INFO:teuthology.orchestra.run.smithi189.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-03-24T00:01:44.882 INFO:teuthology.orchestra.run.smithi189.stdout: 2024-03-24T00:01:44.910 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : ceph-selinux-2:19.0.0-2145.g3be9a5c9.el9.x86_64 132/198 2024-03-24T00:01:44.911 INFO:teuthology.orchestra.run.smithi118.stdout: Upgrading : libacl-2.3.1-4.el9.x86_64 22/198 2024-03-24T00:01:45.048 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 116/198 2024-03-24T00:01:45.150 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 117/198 2024-03-24T00:01:45.252 INFO:teuthology.orchestra.run.smithi118.stdout: Upgrading : NetworkManager-libnm-1:1.47.2-1.el9.x86_64 23/198 2024-03-24T00:01:45.343 INFO:teuthology.orchestra.run.smithi118.stdout: Running scriptlet: NetworkManager-libnm-1:1.47.2-1.el9.x86_64 23/198 2024-03-24T00:01:45.483 INFO:teuthology.orchestra.run.smithi118.stdout: Running scriptlet: NetworkManager-1:1.47.2-1.el9.x86_64 24/198 2024-03-24T00:01:45.534 INFO:teuthology.orchestra.run.smithi118.stdout: Upgrading : NetworkManager-1:1.47.2-1.el9.x86_64 24/198 2024-03-24T00:01:45.904 INFO:teuthology.orchestra.run.smithi118.stdout: Running scriptlet: NetworkManager-1:1.47.2-1.el9.x86_64 24/198 2024-03-24T00:01:46.071 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : re2-1:20211101-3.el9.x86_64 25/198 2024-03-24T00:01:46.205 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : libarrow-9.0.0-10.el9.x86_64 26/198 2024-03-24T00:01:46.248 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 27/198 2024-03-24T00:01:46.284 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : liboath-2.6.7-2.el9.x86_64 28/198 2024-03-24T00:01:46.340 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : libquadmath-11.4.1-3.el9.x86_64 29/198 2024-03-24T00:01:46.361 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 118/198 2024-03-24T00:01:46.452 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : libgfortran-11.4.1-3.el9.x86_64 30/198 2024-03-24T00:01:46.453 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 119/198 2024-03-24T00:01:46.484 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : oniguruma-6.9.6-1.el9.5.x86_64 120/198 2024-03-24T00:01:46.513 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : python3-urllib3-1.26.5-5.el9.noarch 31/198 2024-03-24T00:01:46.515 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : jq-1.6-16.el9.x86_64 121/198 2024-03-24T00:01:46.543 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : libzstd-devel-1.5.1-2.el9.x86_64 122/198 2024-03-24T00:01:46.564 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 32/198 2024-03-24T00:01:46.655 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 33/198 2024-03-24T00:01:46.728 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 34/198 2024-03-24T00:01:46.798 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 35/198 2024-03-24T00:01:46.841 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : protobuf-3.14.0-13.el9.x86_64 36/198 2024-03-24T00:01:46.854 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 123/198 2024-03-24T00:01:46.873 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : lua-5.4.4-4.el9.x86_64 37/198 2024-03-24T00:01:46.890 INFO:teuthology.orchestra.run.smithi193.stdout: Upgrading : librbd1-2:19.0.0-2145.g3be9a5c9.el9.x86_64 124/198 2024-03-24T00:01:46.981 INFO:teuthology.orchestra.run.smithi193.stdout: Running scriptlet: librbd1-2:19.0.0-2145.g3be9a5c9.el9.x86_64 124/198 2024-03-24T00:01:46.986 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 38/198 2024-03-24T00:01:47.019 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : python3-ceph-common-2:19.0.0-2145.g3be9a5c9.el9. 39/198 2024-03-24T00:01:47.020 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : python3-rbd-2:19.0.0-2145.g3be9a5c9.el9.x86_64 125/198 2024-03-24T00:01:47.064 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 126/198 2024-03-24T00:01:47.068 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : python3-ceph-argparse-2:19.0.0-2145.g3be9a5c9.el 40/198 2024-03-24T00:01:47.101 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : python3-cephfs-2:19.0.0-2145.g3be9a5c9.el9.x86_6 41/198 2024-03-24T00:01:47.213 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : lua-devel-5.4.4-4.el9.x86_64 42/198 2024-03-24T00:01:47.278 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : luarocks-3.9.2-1.el9.noarch 43/198 2024-03-24T00:01:47.368 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : protobuf-compiler-3.14.0-13.el9.x86_64 44/198 2024-03-24T00:01:47.524 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : parquet-libs-9.0.0-10.el9.x86_64 127/198 2024-03-24T00:01:47.539 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 45/198 2024-03-24T00:01:47.551 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : librgw2-2:19.0.0-2145.g3be9a5c9.el9.x86_64 128/198 2024-03-24T00:01:47.606 INFO:teuthology.orchestra.run.smithi193.stdout: Running scriptlet: librgw2-2:19.0.0-2145.g3be9a5c9.el9.x86_64 128/198 2024-03-24T00:01:47.624 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 46/198 2024-03-24T00:01:47.628 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : python3-rgw-2:19.0.0-2145.g3be9a5c9.el9.x86_64 129/198 2024-03-24T00:01:48.331 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : python3-rsa-4.9-2.el9.noarch 47/198 2024-03-24T00:01:48.453 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 48/198 2024-03-24T00:01:48.553 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : python3-jinja2-2.11.3-5.el9.noarch 49/198 2024-03-24T00:01:48.627 INFO:teuthology.orchestra.run.smithi118.stdout: Upgrading : util-linux-core-2.37.4-18.el9.x86_64 50/198 2024-03-24T00:01:48.650 INFO:teuthology.orchestra.run.smithi118.stdout: Running scriptlet: util-linux-core-2.37.4-18.el9.x86_64 50/198 2024-03-24T00:01:48.696 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : qatzip-libs-1.1.2-2.el9.x86_64 51/198 2024-03-24T00:01:48.753 INFO:teuthology.orchestra.run.smithi118.stdout: Upgrading : elfutils-libs-0.190-2.el9.x86_64 52/198 2024-03-24T00:01:48.803 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : librados-devel-2:19.0.0-2145.g3be9a5c9.el9.x86_6 53/198 2024-03-24T00:01:48.846 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 54/198 2024-03-24T00:01:49.184 INFO:teuthology.orchestra.run.smithi118.stdout: Upgrading : libfdisk-2.37.4-18.el9.x86_64 55/198 2024-03-24T00:01:49.431 INFO:teuthology.orchestra.run.smithi118.stdout: Upgrading : util-linux-2.37.4-18.el9.x86_64 56/198 2024-03-24T00:01:49.494 INFO:teuthology.orchestra.run.smithi118.stdout: Upgrading : xfsprogs-6.3.0-1.el9.x86_64 57/198 2024-03-24T00:01:49.548 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : libuuid-devel-2.37.4-18.el9.x86_64 58/198 2024-03-24T00:01:49.787 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : socat-1.7.4.1-5.el9.x86_64 59/198 2024-03-24T00:01:49.797 INFO:teuthology.orchestra.run.smithi193.stdout: Running scriptlet: ceph-common-2:19.0.0-2145.g3be9a5c9.el9.x86_64 130/198 2024-03-24T00:01:49.833 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : ceph-common-2:19.0.0-2145.g3be9a5c9.el9.x86_64 130/198 2024-03-24T00:01:49.842 INFO:teuthology.orchestra.run.smithi118.stdout: Upgrading : openssl-1:3.0.7-27.el9.x86_64 60/198 2024-03-24T00:01:49.919 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 61/198 2024-03-24T00:01:50.011 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 62/198 2024-03-24T00:01:50.075 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : python3-webob-1.8.7-6.el9.noarch 63/198 2024-03-24T00:01:50.117 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : python3-typing-extensions-4.4.0-2.el9.noarch 64/198 2024-03-24T00:01:50.185 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 65/198 2024-03-24T00:01:50.222 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 66/198 2024-03-24T00:01:50.285 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : python3-pyperclip-1.8.2-2.el9.noarch 67/198 2024-03-24T00:01:50.344 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 68/198 2024-03-24T00:01:50.370 INFO:teuthology.orchestra.run.smithi193.stdout: Running scriptlet: ceph-common-2:19.0.0-2145.g3be9a5c9.el9.x86_64 130/198 2024-03-24T00:01:50.412 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : ceph-base-2:19.0.0-2145.g3be9a5c9.el9.x86_64 131/198 2024-03-24T00:01:50.493 INFO:teuthology.orchestra.run.smithi193.stdout: Running scriptlet: ceph-base-2:19.0.0-2145.g3be9a5c9.el9.x86_64 131/198 2024-03-24T00:01:50.493 INFO:teuthology.orchestra.run.smithi193.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-03-24T00:01:50.493 INFO:teuthology.orchestra.run.smithi193.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-03-24T00:01:50.493 INFO:teuthology.orchestra.run.smithi193.stdout: 2024-03-24T00:01:50.530 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : ceph-selinux-2:19.0.0-2145.g3be9a5c9.el9.x86_64 132/198 2024-03-24T00:01:50.532 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 69/198 2024-03-24T00:01:50.604 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 70/198 2024-03-24T00:01:50.642 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : python3-colorama-0.4.4-12.el9.noarch 71/198 2024-03-24T00:01:50.700 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : python3-certifi-2021.10.8-2.el9.noarch 72/198 2024-03-24T00:01:50.839 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 73/198 2024-03-24T00:01:50.899 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : python3-google-auth-1:2.28.2-1.el9.noarch 74/198 2024-03-24T00:01:50.946 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 75/198 2024-03-24T00:01:50.982 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : gperftools-libs-2.9.1-2.el9.x86_64 76/198 2024-03-24T00:01:51.040 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : libarrow-doc-9.0.0-10.el9.noarch 77/198 2024-03-24T00:01:51.169 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : grpc-data-1.46.7-10.el9.noarch 78/198 2024-03-24T00:01:51.348 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : abseil-cpp-20211102.0-3.el9.x86_64 79/198 2024-03-24T00:01:51.502 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : python3-grpcio-1.46.7-10.el9.x86_64 80/198 2024-03-24T00:01:51.573 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : python3-devel-3.9.17-2.el9.x86_64 81/198 2024-03-24T00:01:51.639 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : python3-ply-3.11-14.el9.noarch 82/198 2024-03-24T00:01:51.729 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 83/198 2024-03-24T00:01:51.974 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 84/198 2024-03-24T00:01:52.053 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 85/198 2024-03-24T00:01:52.226 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 86/198 2024-03-24T00:01:52.274 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : python3-asyncssh-2.13.2-5.el9.noarch 87/198 2024-03-24T00:01:52.314 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 88/198 2024-03-24T00:01:52.345 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : fuse-2.9.9-15.el9.x86_64 89/198 2024-03-24T00:01:52.385 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : zlib-devel-1.2.11-40.el9.x86_64 90/198 2024-03-24T00:01:52.430 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 91/198 2024-03-24T00:01:52.521 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : uuid-1.6.2-55.el9.x86_64 92/198 2024-03-24T00:01:52.577 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : python3-wcwidth-0.2.5-8.el9.noarch 93/198 2024-03-24T00:01:52.625 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 94/198 2024-03-24T00:01:52.751 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 95/198 2024-03-24T00:01:52.804 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 96/198 2024-03-24T00:01:52.840 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 97/198 2024-03-24T00:01:52.905 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 98/198 2024-03-24T00:01:52.979 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 99/198 2024-03-24T00:01:53.137 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 100/198 2024-03-24T00:01:53.269 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 101/198 2024-03-24T00:01:53.330 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : python3-protobuf-3.14.0-13.el9.noarch 102/198 2024-03-24T00:01:53.366 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : python3-grpcio-tools-1.46.7-10.el9.x86_64 103/198 2024-03-24T00:01:53.422 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 104/198 2024-03-24T00:01:53.570 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : python3-packaging-20.9-5.el9.noarch 105/198 2024-03-24T00:01:53.656 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 106/198 2024-03-24T00:01:54.602 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 107/198 2024-03-24T00:01:54.656 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 108/198 2024-03-24T00:01:54.698 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 109/198 2024-03-24T00:01:54.755 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 110/198 2024-03-24T00:01:54.867 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : python3-attrs-20.3.0-7.el9.noarch 111/198 2024-03-24T00:01:54.933 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : python3-cmd2-2.4.2-2.el9.noarch 112/198 2024-03-24T00:01:55.167 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : openblas-0.3.21-2.el9.x86_64 113/198 2024-03-24T00:01:55.221 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : openblas-openmp-0.3.21-2.el9.x86_64 114/198 2024-03-24T00:01:55.303 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 115/198 2024-03-24T00:01:56.116 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 116/198 2024-03-24T00:01:56.240 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 117/198 2024-03-24T00:01:57.002 INFO:teuthology.orchestra.run.smithi189.stdout: Running scriptlet: ceph-selinux-2:19.0.0-2145.g3be9a5c9.el9.x86_64 132/198 2024-03-24T00:01:57.002 INFO:teuthology.orchestra.run.smithi189.stdout:skipping the directory /sys 2024-03-24T00:01:57.002 INFO:teuthology.orchestra.run.smithi189.stdout:skipping the directory /proc 2024-03-24T00:01:57.002 INFO:teuthology.orchestra.run.smithi189.stdout:skipping the directory /mnt 2024-03-24T00:01:57.002 INFO:teuthology.orchestra.run.smithi189.stdout:skipping the directory /var/tmp 2024-03-24T00:01:57.002 INFO:teuthology.orchestra.run.smithi189.stdout:skipping the directory /home 2024-03-24T00:01:57.002 INFO:teuthology.orchestra.run.smithi189.stdout:skipping the directory /root 2024-03-24T00:01:57.002 INFO:teuthology.orchestra.run.smithi189.stdout:skipping the directory /tmp 2024-03-24T00:01:57.002 INFO:teuthology.orchestra.run.smithi189.stdout: 2024-03-24T00:01:57.162 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : ceph-volume-2:19.0.0-2145.g3be9a5c9.el9.noarch 133/198 2024-03-24T00:01:57.178 INFO:teuthology.orchestra.run.smithi189.stdout: Running scriptlet: ceph-volume-2:19.0.0-2145.g3be9a5c9.el9.noarch 133/198 2024-03-24T00:01:57.178 INFO:teuthology.orchestra.run.smithi189.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-24T00:01:57.178 INFO:teuthology.orchestra.run.smithi189.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2024-03-24T00:01:57.178 INFO:teuthology.orchestra.run.smithi189.stdout: 2024-03-24T00:01:57.732 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 118/198 2024-03-24T00:01:57.826 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 119/198 2024-03-24T00:01:57.873 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : oniguruma-6.9.6-1.el9.5.x86_64 120/198 2024-03-24T00:01:57.904 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : jq-1.6-16.el9.x86_64 121/198 2024-03-24T00:01:57.933 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : libzstd-devel-1.5.1-2.el9.x86_64 122/198 2024-03-24T00:01:58.295 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 123/198 2024-03-24T00:01:58.326 INFO:teuthology.orchestra.run.smithi118.stdout: Upgrading : librbd1-2:19.0.0-2145.g3be9a5c9.el9.x86_64 124/198 2024-03-24T00:01:58.445 INFO:teuthology.orchestra.run.smithi118.stdout: Running scriptlet: librbd1-2:19.0.0-2145.g3be9a5c9.el9.x86_64 124/198 2024-03-24T00:01:58.474 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : python3-rbd-2:19.0.0-2145.g3be9a5c9.el9.x86_64 125/198 2024-03-24T00:01:58.528 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : boost-program-options-1.75.0-8.el9.x86_64 126/198 2024-03-24T00:01:58.664 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : ceph-osd-2:19.0.0-2145.g3be9a5c9.el9.x86_64 134/198 2024-03-24T00:01:58.694 INFO:teuthology.orchestra.run.smithi189.stdout: Running scriptlet: ceph-osd-2:19.0.0-2145.g3be9a5c9.el9.x86_64 134/198 2024-03-24T00:01:58.694 INFO:teuthology.orchestra.run.smithi189.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-24T00:01:58.694 INFO:teuthology.orchestra.run.smithi189.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-03-24T00:01:58.694 INFO:teuthology.orchestra.run.smithi189.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-03-24T00:01:58.694 INFO:teuthology.orchestra.run.smithi189.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-03-24T00:01:58.694 INFO:teuthology.orchestra.run.smithi189.stdout: 2024-03-24T00:01:58.916 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : ceph-mds-2:19.0.0-2145.g3be9a5c9.el9.x86_64 135/198 2024-03-24T00:01:58.945 INFO:teuthology.orchestra.run.smithi189.stdout: Running scriptlet: ceph-mds-2:19.0.0-2145.g3be9a5c9.el9.x86_64 135/198 2024-03-24T00:01:58.945 INFO:teuthology.orchestra.run.smithi189.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-24T00:01:58.945 INFO:teuthology.orchestra.run.smithi189.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-03-24T00:01:58.945 INFO:teuthology.orchestra.run.smithi189.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-03-24T00:01:58.945 INFO:teuthology.orchestra.run.smithi189.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-03-24T00:01:58.946 INFO:teuthology.orchestra.run.smithi189.stdout: 2024-03-24T00:01:59.040 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : parquet-libs-9.0.0-10.el9.x86_64 127/198 2024-03-24T00:01:59.215 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : librgw2-2:19.0.0-2145.g3be9a5c9.el9.x86_64 128/198 2024-03-24T00:01:59.316 INFO:teuthology.orchestra.run.smithi118.stdout: Running scriptlet: librgw2-2:19.0.0-2145.g3be9a5c9.el9.x86_64 128/198 2024-03-24T00:01:59.349 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : python3-rgw-2:19.0.0-2145.g3be9a5c9.el9.x86_64 129/198 2024-03-24T00:01:59.393 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : ceph-mon-2:19.0.0-2145.g3be9a5c9.el9.x86_64 136/198 2024-03-24T00:01:59.416 INFO:teuthology.orchestra.run.smithi189.stdout: Running scriptlet: ceph-mon-2:19.0.0-2145.g3be9a5c9.el9.x86_64 136/198 2024-03-24T00:01:59.416 INFO:teuthology.orchestra.run.smithi189.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-24T00:01:59.416 INFO:teuthology.orchestra.run.smithi189.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-03-24T00:01:59.416 INFO:teuthology.orchestra.run.smithi189.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-03-24T00:01:59.416 INFO:teuthology.orchestra.run.smithi189.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-03-24T00:01:59.416 INFO:teuthology.orchestra.run.smithi189.stdout: 2024-03-24T00:01:59.511 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : automake-1.16.2-8.el9.noarch 137/198 2024-03-24T00:01:59.930 INFO:teuthology.orchestra.run.smithi189.stdout: Running scriptlet: cephadm-2:19.0.0-2145.g3be9a5c9.el9.noarch 138/198 2024-03-24T00:01:59.959 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : cephadm-2:19.0.0-2145.g3be9a5c9.el9.noarch 138/198 2024-03-24T00:01:59.992 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : ceph-prometheus-alerts-2:19.0.0-2145.g3be9a5c9.e 139/198 2024-03-24T00:02:00.051 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : ceph-grafana-dashboards-2:19.0.0-2145.g3be9a5c9. 140/198 2024-03-24T00:02:00.074 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : ceph-mgr-cephadm-2:19.0.0-2145.g3be9a5c9.el9.noa 141/198 2024-03-24T00:02:00.307 INFO:teuthology.orchestra.run.smithi189.stdout: Running scriptlet: ceph-mgr-cephadm-2:19.0.0-2145.g3be9a5c9.el9.noa 141/198 2024-03-24T00:02:00.341 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : ceph-mgr-dashboard-2:19.0.0-2145.g3be9a5c9.el9.n 142/198 2024-03-24T00:02:01.309 INFO:teuthology.orchestra.run.smithi189.stdout: Running scriptlet: ceph-mgr-dashboard-2:19.0.0-2145.g3be9a5c9.el9.n 142/198 2024-03-24T00:02:01.334 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : ceph-mgr-diskprediction-local-2:19.0.0-2145.g3be 143/198 2024-03-24T00:02:01.432 INFO:teuthology.orchestra.run.smithi189.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:19.0.0-2145.g3be 143/198 2024-03-24T00:02:01.596 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : ceph-mgr-modules-core-2:19.0.0-2145.g3be9a5c9.el 144/198 2024-03-24T00:02:01.628 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : ceph-mgr-2:19.0.0-2145.g3be9a5c9.el9.x86_64 145/198 2024-03-24T00:02:01.650 INFO:teuthology.orchestra.run.smithi189.stdout: Running scriptlet: ceph-mgr-2:19.0.0-2145.g3be9a5c9.el9.x86_64 145/198 2024-03-24T00:02:01.650 INFO:teuthology.orchestra.run.smithi189.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-24T00:02:01.650 INFO:teuthology.orchestra.run.smithi189.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-03-24T00:02:01.650 INFO:teuthology.orchestra.run.smithi189.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-03-24T00:02:01.650 INFO:teuthology.orchestra.run.smithi189.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-03-24T00:02:01.650 INFO:teuthology.orchestra.run.smithi189.stdout: 2024-03-24T00:02:01.695 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : ceph-mgr-rook-2:19.0.0-2145.g3be9a5c9.el9.noarch 146/198 2024-03-24T00:02:01.706 INFO:teuthology.orchestra.run.smithi189.stdout: Running scriptlet: ceph-mgr-rook-2:19.0.0-2145.g3be9a5c9.el9.noarch 146/198 2024-03-24T00:02:01.751 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : ceph-2:19.0.0-2145.g3be9a5c9.el9.x86_64 147/198 2024-03-24T00:02:02.008 INFO:teuthology.orchestra.run.smithi118.stdout: Running scriptlet: ceph-common-2:19.0.0-2145.g3be9a5c9.el9.x86_64 130/198 2024-03-24T00:02:02.045 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : ceph-common-2:19.0.0-2145.g3be9a5c9.el9.x86_64 130/198 2024-03-24T00:02:02.084 INFO:teuthology.orchestra.run.smithi193.stdout: Running scriptlet: ceph-selinux-2:19.0.0-2145.g3be9a5c9.el9.x86_64 132/198 2024-03-24T00:02:02.084 INFO:teuthology.orchestra.run.smithi193.stdout:skipping the directory /sys 2024-03-24T00:02:02.084 INFO:teuthology.orchestra.run.smithi193.stdout:skipping the directory /proc 2024-03-24T00:02:02.084 INFO:teuthology.orchestra.run.smithi193.stdout:skipping the directory /mnt 2024-03-24T00:02:02.084 INFO:teuthology.orchestra.run.smithi193.stdout:skipping the directory /var/tmp 2024-03-24T00:02:02.084 INFO:teuthology.orchestra.run.smithi193.stdout:skipping the directory /home 2024-03-24T00:02:02.084 INFO:teuthology.orchestra.run.smithi193.stdout:skipping the directory /root 2024-03-24T00:02:02.084 INFO:teuthology.orchestra.run.smithi193.stdout:skipping the directory /tmp 2024-03-24T00:02:02.084 INFO:teuthology.orchestra.run.smithi193.stdout: 2024-03-24T00:02:02.237 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : ceph-volume-2:19.0.0-2145.g3be9a5c9.el9.noarch 133/198 2024-03-24T00:02:02.252 INFO:teuthology.orchestra.run.smithi193.stdout: Running scriptlet: ceph-volume-2:19.0.0-2145.g3be9a5c9.el9.noarch 133/198 2024-03-24T00:02:02.252 INFO:teuthology.orchestra.run.smithi193.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-24T00:02:02.252 INFO:teuthology.orchestra.run.smithi193.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2024-03-24T00:02:02.252 INFO:teuthology.orchestra.run.smithi193.stdout: 2024-03-24T00:02:02.623 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : libtool-2.4.6-45.el9.x86_64 148/198 2024-03-24T00:02:02.655 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : ceph-radosgw-2:19.0.0-2145.g3be9a5c9.el9.x86_64 149/198 2024-03-24T00:02:02.658 INFO:teuthology.orchestra.run.smithi118.stdout: Running scriptlet: ceph-common-2:19.0.0-2145.g3be9a5c9.el9.x86_64 130/198 2024-03-24T00:02:02.677 INFO:teuthology.orchestra.run.smithi189.stdout: Running scriptlet: ceph-radosgw-2:19.0.0-2145.g3be9a5c9.el9.x86_64 149/198 2024-03-24T00:02:02.677 INFO:teuthology.orchestra.run.smithi189.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-24T00:02:02.678 INFO:teuthology.orchestra.run.smithi189.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-03-24T00:02:02.678 INFO:teuthology.orchestra.run.smithi189.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-03-24T00:02:02.678 INFO:teuthology.orchestra.run.smithi189.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-03-24T00:02:02.678 INFO:teuthology.orchestra.run.smithi189.stdout: 2024-03-24T00:02:02.699 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : ceph-base-2:19.0.0-2145.g3be9a5c9.el9.x86_64 131/198 2024-03-24T00:02:02.722 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : ceph-immutable-object-cache-2:19.0.0-2145.g3be9a 150/198 2024-03-24T00:02:02.744 INFO:teuthology.orchestra.run.smithi189.stdout: Running scriptlet: ceph-immutable-object-cache-2:19.0.0-2145.g3be9a 150/198 2024-03-24T00:02:02.745 INFO:teuthology.orchestra.run.smithi189.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-24T00:02:02.745 INFO:teuthology.orchestra.run.smithi189.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-03-24T00:02:02.745 INFO:teuthology.orchestra.run.smithi189.stdout: 2024-03-24T00:02:02.790 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : cephfs-mirror-2:19.0.0-2145.g3be9a5c9.el9.x86_64 151/198 2024-03-24T00:02:02.812 INFO:teuthology.orchestra.run.smithi189.stdout: Running scriptlet: cephfs-mirror-2:19.0.0-2145.g3be9a5c9.el9.x86_64 151/198 2024-03-24T00:02:02.812 INFO:teuthology.orchestra.run.smithi189.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-24T00:02:02.812 INFO:teuthology.orchestra.run.smithi189.stdout:Invalid unit name "cephfs-mirror@*.service" escaped as "cephfs-mirror@\x2a.service". 2024-03-24T00:02:02.812 INFO:teuthology.orchestra.run.smithi189.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/cephfs-mirror.target → /usr/lib/systemd/system/cephfs-mirror.target. 2024-03-24T00:02:02.812 INFO:teuthology.orchestra.run.smithi189.stdout:Created symlink /etc/systemd/system/ceph.target.wants/cephfs-mirror.target → /usr/lib/systemd/system/cephfs-mirror.target. 2024-03-24T00:02:02.812 INFO:teuthology.orchestra.run.smithi189.stdout: 2024-03-24T00:02:02.866 INFO:teuthology.orchestra.run.smithi118.stdout: Running scriptlet: ceph-base-2:19.0.0-2145.g3be9a5c9.el9.x86_64 131/198 2024-03-24T00:02:02.867 INFO:teuthology.orchestra.run.smithi118.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-03-24T00:02:02.867 INFO:teuthology.orchestra.run.smithi118.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-03-24T00:02:02.867 INFO:teuthology.orchestra.run.smithi118.stdout: 2024-03-24T00:02:02.916 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : ceph-selinux-2:19.0.0-2145.g3be9a5c9.el9.x86_64 132/198 2024-03-24T00:02:03.124 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : rbd-mirror-2:19.0.0-2145.g3be9a5c9.el9.x86_64 152/198 2024-03-24T00:02:03.146 INFO:teuthology.orchestra.run.smithi189.stdout: Running scriptlet: rbd-mirror-2:19.0.0-2145.g3be9a5c9.el9.x86_64 152/198 2024-03-24T00:02:03.146 INFO:teuthology.orchestra.run.smithi189.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-24T00:02:03.146 INFO:teuthology.orchestra.run.smithi189.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-03-24T00:02:03.146 INFO:teuthology.orchestra.run.smithi189.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-03-24T00:02:03.146 INFO:teuthology.orchestra.run.smithi189.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-03-24T00:02:03.147 INFO:teuthology.orchestra.run.smithi189.stdout: 2024-03-24T00:02:03.740 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : ceph-osd-2:19.0.0-2145.g3be9a5c9.el9.x86_64 134/198 2024-03-24T00:02:03.768 INFO:teuthology.orchestra.run.smithi193.stdout: Running scriptlet: ceph-osd-2:19.0.0-2145.g3be9a5c9.el9.x86_64 134/198 2024-03-24T00:02:03.769 INFO:teuthology.orchestra.run.smithi193.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-24T00:02:03.769 INFO:teuthology.orchestra.run.smithi193.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-03-24T00:02:03.769 INFO:teuthology.orchestra.run.smithi193.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-03-24T00:02:03.769 INFO:teuthology.orchestra.run.smithi193.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-03-24T00:02:03.769 INFO:teuthology.orchestra.run.smithi193.stdout: 2024-03-24T00:02:04.005 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : ceph-mds-2:19.0.0-2145.g3be9a5c9.el9.x86_64 135/198 2024-03-24T00:02:04.025 INFO:teuthology.orchestra.run.smithi193.stdout: Running scriptlet: ceph-mds-2:19.0.0-2145.g3be9a5c9.el9.x86_64 135/198 2024-03-24T00:02:04.025 INFO:teuthology.orchestra.run.smithi193.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-24T00:02:04.026 INFO:teuthology.orchestra.run.smithi193.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-03-24T00:02:04.026 INFO:teuthology.orchestra.run.smithi193.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-03-24T00:02:04.026 INFO:teuthology.orchestra.run.smithi193.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-03-24T00:02:04.026 INFO:teuthology.orchestra.run.smithi193.stdout: 2024-03-24T00:02:04.470 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : ceph-mon-2:19.0.0-2145.g3be9a5c9.el9.x86_64 136/198 2024-03-24T00:02:04.492 INFO:teuthology.orchestra.run.smithi193.stdout: Running scriptlet: ceph-mon-2:19.0.0-2145.g3be9a5c9.el9.x86_64 136/198 2024-03-24T00:02:04.493 INFO:teuthology.orchestra.run.smithi193.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-24T00:02:04.493 INFO:teuthology.orchestra.run.smithi193.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-03-24T00:02:04.493 INFO:teuthology.orchestra.run.smithi193.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-03-24T00:02:04.493 INFO:teuthology.orchestra.run.smithi193.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-03-24T00:02:04.493 INFO:teuthology.orchestra.run.smithi193.stdout: 2024-03-24T00:02:04.580 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : automake-1.16.2-8.el9.noarch 137/198 2024-03-24T00:02:05.047 INFO:teuthology.orchestra.run.smithi193.stdout: Running scriptlet: cephadm-2:19.0.0-2145.g3be9a5c9.el9.noarch 138/198 2024-03-24T00:02:05.078 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : cephadm-2:19.0.0-2145.g3be9a5c9.el9.noarch 138/198 2024-03-24T00:02:05.110 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : ceph-prometheus-alerts-2:19.0.0-2145.g3be9a5c9.e 139/198 2024-03-24T00:02:05.171 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : ceph-grafana-dashboards-2:19.0.0-2145.g3be9a5c9. 140/198 2024-03-24T00:02:05.193 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : ceph-mgr-cephadm-2:19.0.0-2145.g3be9a5c9.el9.noa 141/198 2024-03-24T00:02:05.426 INFO:teuthology.orchestra.run.smithi193.stdout: Running scriptlet: ceph-mgr-cephadm-2:19.0.0-2145.g3be9a5c9.el9.noa 141/198 2024-03-24T00:02:05.469 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : ceph-mgr-dashboard-2:19.0.0-2145.g3be9a5c9.el9.n 142/198 2024-03-24T00:02:06.430 INFO:teuthology.orchestra.run.smithi193.stdout: Running scriptlet: ceph-mgr-dashboard-2:19.0.0-2145.g3be9a5c9.el9.n 142/198 2024-03-24T00:02:06.461 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : ceph-mgr-diskprediction-local-2:19.0.0-2145.g3be 143/198 2024-03-24T00:02:06.560 INFO:teuthology.orchestra.run.smithi193.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:19.0.0-2145.g3be 143/198 2024-03-24T00:02:06.732 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : ceph-mgr-modules-core-2:19.0.0-2145.g3be9a5c9.el 144/198 2024-03-24T00:02:06.764 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : ceph-mgr-2:19.0.0-2145.g3be9a5c9.el9.x86_64 145/198 2024-03-24T00:02:06.787 INFO:teuthology.orchestra.run.smithi193.stdout: Running scriptlet: ceph-mgr-2:19.0.0-2145.g3be9a5c9.el9.x86_64 145/198 2024-03-24T00:02:06.787 INFO:teuthology.orchestra.run.smithi193.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-24T00:02:06.787 INFO:teuthology.orchestra.run.smithi193.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-03-24T00:02:06.787 INFO:teuthology.orchestra.run.smithi193.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-03-24T00:02:06.788 INFO:teuthology.orchestra.run.smithi193.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-03-24T00:02:06.788 INFO:teuthology.orchestra.run.smithi193.stdout: 2024-03-24T00:02:06.831 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : ceph-mgr-rook-2:19.0.0-2145.g3be9a5c9.el9.noarch 146/198 2024-03-24T00:02:06.842 INFO:teuthology.orchestra.run.smithi193.stdout: Running scriptlet: ceph-mgr-rook-2:19.0.0-2145.g3be9a5c9.el9.noarch 146/198 2024-03-24T00:02:06.887 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : ceph-2:19.0.0-2145.g3be9a5c9.el9.x86_64 147/198 2024-03-24T00:02:07.578 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : ceph-test-2:19.0.0-2145.g3be9a5c9.el9.x86_64 153/198 2024-03-24T00:02:07.619 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : rbd-fuse-2:19.0.0-2145.g3be9a5c9.el9.x86_64 154/198 2024-03-24T00:02:07.650 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : rbd-nbd-2:19.0.0-2145.g3be9a5c9.el9.x86_64 155/198 2024-03-24T00:02:07.676 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : elfutils-libelf-devel-0.190-2.el9.x86_64 156/198 2024-03-24T00:02:07.759 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : libtool-2.4.6-45.el9.x86_64 148/198 2024-03-24T00:02:07.792 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : ceph-radosgw-2:19.0.0-2145.g3be9a5c9.el9.x86_64 149/198 2024-03-24T00:02:07.815 INFO:teuthology.orchestra.run.smithi193.stdout: Running scriptlet: ceph-radosgw-2:19.0.0-2145.g3be9a5c9.el9.x86_64 149/198 2024-03-24T00:02:07.815 INFO:teuthology.orchestra.run.smithi193.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-24T00:02:07.815 INFO:teuthology.orchestra.run.smithi193.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-03-24T00:02:07.815 INFO:teuthology.orchestra.run.smithi193.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-03-24T00:02:07.815 INFO:teuthology.orchestra.run.smithi193.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-03-24T00:02:07.815 INFO:teuthology.orchestra.run.smithi193.stdout: 2024-03-24T00:02:07.858 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : cephfs-shell-2:19.0.0-2145.g3be9a5c9.el9.x86_64 157/198 2024-03-24T00:02:07.859 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : ceph-immutable-object-cache-2:19.0.0-2145.g3be9a 150/198 2024-03-24T00:02:07.881 INFO:teuthology.orchestra.run.smithi193.stdout: Running scriptlet: ceph-immutable-object-cache-2:19.0.0-2145.g3be9a 150/198 2024-03-24T00:02:07.882 INFO:teuthology.orchestra.run.smithi193.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-24T00:02:07.882 INFO:teuthology.orchestra.run.smithi193.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-03-24T00:02:07.882 INFO:teuthology.orchestra.run.smithi193.stdout: 2024-03-24T00:02:07.927 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : cephfs-mirror-2:19.0.0-2145.g3be9a5c9.el9.x86_64 151/198 2024-03-24T00:02:07.950 INFO:teuthology.orchestra.run.smithi193.stdout: Running scriptlet: cephfs-mirror-2:19.0.0-2145.g3be9a5c9.el9.x86_64 151/198 2024-03-24T00:02:07.950 INFO:teuthology.orchestra.run.smithi193.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-24T00:02:07.950 INFO:teuthology.orchestra.run.smithi193.stdout:Invalid unit name "cephfs-mirror@*.service" escaped as "cephfs-mirror@\x2a.service". 2024-03-24T00:02:07.950 INFO:teuthology.orchestra.run.smithi193.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/cephfs-mirror.target → /usr/lib/systemd/system/cephfs-mirror.target. 2024-03-24T00:02:07.950 INFO:teuthology.orchestra.run.smithi193.stdout:Created symlink /etc/systemd/system/ceph.target.wants/cephfs-mirror.target → /usr/lib/systemd/system/cephfs-mirror.target. 2024-03-24T00:02:07.950 INFO:teuthology.orchestra.run.smithi193.stdout: 2024-03-24T00:02:07.969 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : postgresql-contrib-13.11-1.el9.x86_64 158/198 2024-03-24T00:02:08.010 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : ceph-fuse-2:19.0.0-2145.g3be9a5c9.el9.x86_64 159/198 2024-03-24T00:02:08.048 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : xfsprogs-devel-6.3.0-1.el9.x86_64 160/198 2024-03-24T00:02:08.110 INFO:teuthology.orchestra.run.smithi189.stdout: Upgrading : xfsdump-3.1.12-4.el9.x86_64 161/198 2024-03-24T00:02:08.152 INFO:teuthology.orchestra.run.smithi189.stdout: Upgrading : util-linux-user-2.37.4-18.el9.x86_64 162/198 2024-03-24T00:02:08.202 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : libcephfs-devel-2:19.0.0-2145.g3be9a5c9.el9.x86_ 163/198 2024-03-24T00:02:08.234 INFO:teuthology.orchestra.run.smithi189.stdout: Upgrading : elfutils-debuginfod-client-0.190-2.el9.x86_64 164/198 2024-03-24T00:02:08.252 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : rbd-mirror-2:19.0.0-2145.g3be9a5c9.el9.x86_64 152/198 2024-03-24T00:02:08.264 INFO:teuthology.orchestra.run.smithi189.stdout: Upgrading : NetworkManager-team-1:1.47.2-1.el9.x86_64 165/198 2024-03-24T00:02:08.275 INFO:teuthology.orchestra.run.smithi193.stdout: Running scriptlet: rbd-mirror-2:19.0.0-2145.g3be9a5c9.el9.x86_64 152/198 2024-03-24T00:02:08.275 INFO:teuthology.orchestra.run.smithi193.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-24T00:02:08.275 INFO:teuthology.orchestra.run.smithi193.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-03-24T00:02:08.275 INFO:teuthology.orchestra.run.smithi193.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-03-24T00:02:08.275 INFO:teuthology.orchestra.run.smithi193.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-03-24T00:02:08.275 INFO:teuthology.orchestra.run.smithi193.stdout: 2024-03-24T00:02:08.309 INFO:teuthology.orchestra.run.smithi189.stdout: Upgrading : NetworkManager-tui-1:1.47.2-1.el9.x86_64 166/198 2024-03-24T00:02:08.371 INFO:teuthology.orchestra.run.smithi189.stdout: Upgrading : acl-2.3.1-4.el9.x86_64 167/198 2024-03-24T00:02:08.404 INFO:teuthology.orchestra.run.smithi189.stdout: Upgrading : libacl-devel-2.3.1-4.el9.x86_64 168/198 2024-03-24T00:02:09.015 INFO:teuthology.orchestra.run.smithi189.stdout: Running scriptlet: postgresql-server-13.11-1.el9.x86_64 169/198 2024-03-24T00:02:09.052 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : postgresql-server-13.11-1.el9.x86_64 169/198 2024-03-24T00:02:09.071 INFO:teuthology.orchestra.run.smithi189.stdout: Running scriptlet: postgresql-server-13.11-1.el9.x86_64 169/198 2024-03-24T00:02:09.891 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : cephfs-top-2:19.0.0-2145.g3be9a5c9.el9.noarch 170/198 2024-03-24T00:02:09.972 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : openssl-devel-1:3.0.7-27.el9.x86_64 171/198 2024-03-24T00:02:10.012 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : libaio-devel-0.3.111-13.el9.x86_64 172/198 2024-03-24T00:02:10.091 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : flex-2.6.4-9.el9.x86_64 173/198 2024-03-24T00:02:10.133 INFO:teuthology.orchestra.run.smithi189.stdout: Installing : bison-3.7.4-5.el9.x86_64 174/198 2024-03-24T00:02:10.158 INFO:teuthology.orchestra.run.smithi189.stdout: Cleanup : NetworkManager-tui-1:1.44.0-2.el9.x86_64 175/198 2024-03-24T00:02:10.183 INFO:teuthology.orchestra.run.smithi189.stdout: Cleanup : elfutils-debuginfod-client-0.189-3.el9.x86_64 176/198 2024-03-24T00:02:10.210 INFO:teuthology.orchestra.run.smithi189.stdout: Cleanup : elfutils-libs-0.189-3.el9.x86_64 177/198 2024-03-24T00:02:10.210 INFO:teuthology.orchestra.run.smithi189.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 178/198 2024-03-24T00:02:10.298 INFO:teuthology.orchestra.run.smithi189.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 178/198 2024-03-24T00:02:10.327 INFO:teuthology.orchestra.run.smithi189.stdout: Cleanup : openssl-1:3.0.7-24.el9.x86_64 179/198 2024-03-24T00:02:10.327 INFO:teuthology.orchestra.run.smithi189.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 180/198 2024-03-24T00:02:10.412 INFO:teuthology.orchestra.run.smithi189.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 180/198 2024-03-24T00:02:10.438 INFO:teuthology.orchestra.run.smithi189.stdout: Cleanup : xfsdump-3.1.12-3.el9.x86_64 181/198 2024-03-24T00:02:10.472 INFO:teuthology.orchestra.run.smithi189.stdout: Cleanup : xfsprogs-5.19.0-4.el9.x86_64 182/198 2024-03-24T00:02:10.497 INFO:teuthology.orchestra.run.smithi189.stdout: Cleanup : acl-2.3.1-3.el9.x86_64 183/198 2024-03-24T00:02:10.522 INFO:teuthology.orchestra.run.smithi189.stdout: Cleanup : libacl-devel-2.3.1-3.el9.x86_64 184/198 2024-03-24T00:02:10.555 INFO:teuthology.orchestra.run.smithi189.stdout: Cleanup : util-linux-user-2.37.4-15.el9.x86_64 185/198 2024-03-24T00:02:10.602 INFO:teuthology.orchestra.run.smithi189.stdout: Cleanup : util-linux-2.37.4-15.el9.x86_64 186/198 2024-03-24T00:02:10.643 INFO:teuthology.orchestra.run.smithi189.stdout: Cleanup : util-linux-core-2.37.4-15.el9.x86_64 187/198 2024-03-24T00:02:10.668 INFO:teuthology.orchestra.run.smithi189.stdout: Cleanup : libfdisk-2.37.4-15.el9.x86_64 188/198 2024-03-24T00:02:10.693 INFO:teuthology.orchestra.run.smithi189.stdout: Cleanup : libmount-2.37.4-15.el9.x86_64 189/198 2024-03-24T00:02:10.718 INFO:teuthology.orchestra.run.smithi189.stdout: Cleanup : libblkid-2.37.4-15.el9.x86_64 190/198 2024-03-24T00:02:10.744 INFO:teuthology.orchestra.run.smithi189.stdout: Cleanup : NetworkManager-team-1:1.44.0-2.el9.x86_64 191/198 2024-03-24T00:02:10.751 INFO:teuthology.orchestra.run.smithi189.stdout: Running scriptlet: NetworkManager-1:1.44.0-2.el9.x86_64 192/198 2024-03-24T00:02:10.751 INFO:teuthology.orchestra.run.smithi189.stdout: Cleanup : NetworkManager-1:1.44.0-2.el9.x86_64 192/198 2024-03-24T00:02:10.930 INFO:teuthology.orchestra.run.smithi189.stdout: Running scriptlet: NetworkManager-1:1.44.0-2.el9.x86_64 192/198 2024-03-24T00:02:10.930 INFO:teuthology.orchestra.run.smithi189.stdout: Cleanup : NetworkManager-libnm-1:1.44.0-2.el9.x86_64 193/198 2024-03-24T00:02:10.996 INFO:teuthology.orchestra.run.smithi189.stdout: Running scriptlet: NetworkManager-libnm-1:1.44.0-2.el9.x86_64 193/198 2024-03-24T00:02:11.021 INFO:teuthology.orchestra.run.smithi189.stdout: Cleanup : libuuid-2.37.4-15.el9.x86_64 194/198 2024-03-24T00:02:11.054 INFO:teuthology.orchestra.run.smithi189.stdout: Cleanup : libsmartcols-2.37.4-15.el9.x86_64 195/198 2024-03-24T00:02:11.081 INFO:teuthology.orchestra.run.smithi189.stdout: Cleanup : libacl-2.3.1-3.el9.x86_64 196/198 2024-03-24T00:02:11.105 INFO:teuthology.orchestra.run.smithi189.stdout: Cleanup : openssl-libs-1:3.0.7-24.el9.x86_64 197/198 2024-03-24T00:02:11.170 INFO:teuthology.orchestra.run.smithi189.stdout: Cleanup : elfutils-libelf-0.189-3.el9.x86_64 198/198 2024-03-24T00:02:12.715 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : ceph-test-2:19.0.0-2145.g3be9a5c9.el9.x86_64 153/198 2024-03-24T00:02:12.756 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : rbd-fuse-2:19.0.0-2145.g3be9a5c9.el9.x86_64 154/198 2024-03-24T00:02:12.787 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : rbd-nbd-2:19.0.0-2145.g3be9a5c9.el9.x86_64 155/198 2024-03-24T00:02:12.814 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : elfutils-libelf-devel-0.190-2.el9.x86_64 156/198 2024-03-24T00:02:13.019 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : cephfs-shell-2:19.0.0-2145.g3be9a5c9.el9.x86_64 157/198 2024-03-24T00:02:13.131 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : postgresql-contrib-13.11-1.el9.x86_64 158/198 2024-03-24T00:02:13.171 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : ceph-fuse-2:19.0.0-2145.g3be9a5c9.el9.x86_64 159/198 2024-03-24T00:02:13.210 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : xfsprogs-devel-6.3.0-1.el9.x86_64 160/198 2024-03-24T00:02:13.263 INFO:teuthology.orchestra.run.smithi193.stdout: Upgrading : xfsdump-3.1.12-4.el9.x86_64 161/198 2024-03-24T00:02:13.315 INFO:teuthology.orchestra.run.smithi193.stdout: Upgrading : util-linux-user-2.37.4-18.el9.x86_64 162/198 2024-03-24T00:02:13.348 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : libcephfs-devel-2:19.0.0-2145.g3be9a5c9.el9.x86_ 163/198 2024-03-24T00:02:13.396 INFO:teuthology.orchestra.run.smithi193.stdout: Upgrading : elfutils-debuginfod-client-0.190-2.el9.x86_64 164/198 2024-03-24T00:02:13.426 INFO:teuthology.orchestra.run.smithi193.stdout: Upgrading : NetworkManager-team-1:1.47.2-1.el9.x86_64 165/198 2024-03-24T00:02:13.485 INFO:teuthology.orchestra.run.smithi193.stdout: Upgrading : NetworkManager-tui-1:1.47.2-1.el9.x86_64 166/198 2024-03-24T00:02:13.541 INFO:teuthology.orchestra.run.smithi193.stdout: Upgrading : acl-2.3.1-4.el9.x86_64 167/198 2024-03-24T00:02:13.583 INFO:teuthology.orchestra.run.smithi193.stdout: Upgrading : libacl-devel-2.3.1-4.el9.x86_64 168/198 2024-03-24T00:02:14.127 INFO:teuthology.orchestra.run.smithi193.stdout: Running scriptlet: postgresql-server-13.11-1.el9.x86_64 169/198 2024-03-24T00:02:14.165 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : postgresql-server-13.11-1.el9.x86_64 169/198 2024-03-24T00:02:14.185 INFO:teuthology.orchestra.run.smithi193.stdout: Running scriptlet: postgresql-server-13.11-1.el9.x86_64 169/198 2024-03-24T00:02:15.003 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : cephfs-top-2:19.0.0-2145.g3be9a5c9.el9.noarch 170/198 2024-03-24T00:02:15.086 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : openssl-devel-1:3.0.7-27.el9.x86_64 171/198 2024-03-24T00:02:15.127 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : libaio-devel-0.3.111-13.el9.x86_64 172/198 2024-03-24T00:02:15.208 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : flex-2.6.4-9.el9.x86_64 173/198 2024-03-24T00:02:15.265 INFO:teuthology.orchestra.run.smithi193.stdout: Installing : bison-3.7.4-5.el9.x86_64 174/198 2024-03-24T00:02:15.290 INFO:teuthology.orchestra.run.smithi193.stdout: Cleanup : NetworkManager-tui-1:1.44.0-2.el9.x86_64 175/198 2024-03-24T00:02:15.316 INFO:teuthology.orchestra.run.smithi193.stdout: Cleanup : elfutils-debuginfod-client-0.189-3.el9.x86_64 176/198 2024-03-24T00:02:15.342 INFO:teuthology.orchestra.run.smithi193.stdout: Cleanup : elfutils-libs-0.189-3.el9.x86_64 177/198 2024-03-24T00:02:15.342 INFO:teuthology.orchestra.run.smithi193.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 178/198 2024-03-24T00:02:15.430 INFO:teuthology.orchestra.run.smithi193.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 178/198 2024-03-24T00:02:15.460 INFO:teuthology.orchestra.run.smithi193.stdout: Cleanup : openssl-1:3.0.7-24.el9.x86_64 179/198 2024-03-24T00:02:15.460 INFO:teuthology.orchestra.run.smithi193.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 180/198 2024-03-24T00:02:15.544 INFO:teuthology.orchestra.run.smithi193.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 180/198 2024-03-24T00:02:15.571 INFO:teuthology.orchestra.run.smithi193.stdout: Cleanup : xfsdump-3.1.12-3.el9.x86_64 181/198 2024-03-24T00:02:15.613 INFO:teuthology.orchestra.run.smithi193.stdout: Cleanup : xfsprogs-5.19.0-4.el9.x86_64 182/198 2024-03-24T00:02:15.638 INFO:teuthology.orchestra.run.smithi193.stdout: Cleanup : acl-2.3.1-3.el9.x86_64 183/198 2024-03-24T00:02:15.663 INFO:teuthology.orchestra.run.smithi193.stdout: Cleanup : libacl-devel-2.3.1-3.el9.x86_64 184/198 2024-03-24T00:02:15.696 INFO:teuthology.orchestra.run.smithi193.stdout: Cleanup : util-linux-user-2.37.4-15.el9.x86_64 185/198 2024-03-24T00:02:15.736 INFO:teuthology.orchestra.run.smithi193.stdout: Cleanup : util-linux-2.37.4-15.el9.x86_64 186/198 2024-03-24T00:02:15.768 INFO:teuthology.orchestra.run.smithi193.stdout: Cleanup : util-linux-core-2.37.4-15.el9.x86_64 187/198 2024-03-24T00:02:15.801 INFO:teuthology.orchestra.run.smithi193.stdout: Cleanup : libfdisk-2.37.4-15.el9.x86_64 188/198 2024-03-24T00:02:15.827 INFO:teuthology.orchestra.run.smithi193.stdout: Cleanup : libmount-2.37.4-15.el9.x86_64 189/198 2024-03-24T00:02:15.852 INFO:teuthology.orchestra.run.smithi193.stdout: Cleanup : libblkid-2.37.4-15.el9.x86_64 190/198 2024-03-24T00:02:15.886 INFO:teuthology.orchestra.run.smithi193.stdout: Cleanup : NetworkManager-team-1:1.44.0-2.el9.x86_64 191/198 2024-03-24T00:02:15.893 INFO:teuthology.orchestra.run.smithi193.stdout: Running scriptlet: NetworkManager-1:1.44.0-2.el9.x86_64 192/198 2024-03-24T00:02:15.893 INFO:teuthology.orchestra.run.smithi193.stdout: Cleanup : NetworkManager-1:1.44.0-2.el9.x86_64 192/198 2024-03-24T00:02:15.998 INFO:teuthology.orchestra.run.smithi118.stdout: Running scriptlet: ceph-selinux-2:19.0.0-2145.g3be9a5c9.el9.x86_64 132/198 2024-03-24T00:02:15.999 INFO:teuthology.orchestra.run.smithi118.stdout:skipping the directory /sys 2024-03-24T00:02:15.999 INFO:teuthology.orchestra.run.smithi118.stdout:skipping the directory /proc 2024-03-24T00:02:15.999 INFO:teuthology.orchestra.run.smithi118.stdout:skipping the directory /mnt 2024-03-24T00:02:15.999 INFO:teuthology.orchestra.run.smithi118.stdout:skipping the directory /var/tmp 2024-03-24T00:02:15.999 INFO:teuthology.orchestra.run.smithi118.stdout:skipping the directory /home 2024-03-24T00:02:15.999 INFO:teuthology.orchestra.run.smithi118.stdout:skipping the directory /root 2024-03-24T00:02:15.999 INFO:teuthology.orchestra.run.smithi118.stdout:skipping the directory /tmp 2024-03-24T00:02:15.999 INFO:teuthology.orchestra.run.smithi118.stdout: 2024-03-24T00:02:16.051 INFO:teuthology.orchestra.run.smithi189.stdout: Running scriptlet: elfutils-libelf-0.189-3.el9.x86_64 198/198 2024-03-24T00:02:16.051 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : ceph-2:19.0.0-2145.g3be9a5c9.el9.x86_64 1/198 2024-03-24T00:02:16.052 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : ceph-base-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2/198 2024-03-24T00:02:16.052 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : ceph-common-2:19.0.0-2145.g3be9a5c9.el9.x86_64 3/198 2024-03-24T00:02:16.052 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : ceph-fuse-2:19.0.0-2145.g3be9a5c9.el9.x86_64 4/198 2024-03-24T00:02:16.052 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : ceph-immutable-object-cache-2:19.0.0-2145.g3be9a 5/198 2024-03-24T00:02:16.052 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : ceph-mds-2:19.0.0-2145.g3be9a5c9.el9.x86_64 6/198 2024-03-24T00:02:16.052 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : ceph-mgr-2:19.0.0-2145.g3be9a5c9.el9.x86_64 7/198 2024-03-24T00:02:16.052 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : ceph-mon-2:19.0.0-2145.g3be9a5c9.el9.x86_64 8/198 2024-03-24T00:02:16.052 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : ceph-osd-2:19.0.0-2145.g3be9a5c9.el9.x86_64 9/198 2024-03-24T00:02:16.052 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : ceph-radosgw-2:19.0.0-2145.g3be9a5c9.el9.x86_64 10/198 2024-03-24T00:02:16.052 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : ceph-selinux-2:19.0.0-2145.g3be9a5c9.el9.x86_64 11/198 2024-03-24T00:02:16.052 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : ceph-test-2:19.0.0-2145.g3be9a5c9.el9.x86_64 12/198 2024-03-24T00:02:16.052 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : cephfs-mirror-2:19.0.0-2145.g3be9a5c9.el9.x86_64 13/198 2024-03-24T00:02:16.052 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : cephfs-shell-2:19.0.0-2145.g3be9a5c9.el9.x86_64 14/198 2024-03-24T00:02:16.052 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : libcephfs-devel-2:19.0.0-2145.g3be9a5c9.el9.x86_ 15/198 2024-03-24T00:02:16.053 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : libcephfs2-2:19.0.0-2145.g3be9a5c9.el9.x86_64 16/198 2024-03-24T00:02:16.053 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : libcephsqlite-2:19.0.0-2145.g3be9a5c9.el9.x86_64 17/198 2024-03-24T00:02:16.053 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : librados-devel-2:19.0.0-2145.g3be9a5c9.el9.x86_6 18/198 2024-03-24T00:02:16.053 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : libradosstriper1-2:19.0.0-2145.g3be9a5c9.el9.x86 19/198 2024-03-24T00:02:16.053 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : librgw2-2:19.0.0-2145.g3be9a5c9.el9.x86_64 20/198 2024-03-24T00:02:16.053 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : python3-ceph-argparse-2:19.0.0-2145.g3be9a5c9.el 21/198 2024-03-24T00:02:16.053 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : python3-ceph-common-2:19.0.0-2145.g3be9a5c9.el9. 22/198 2024-03-24T00:02:16.053 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : python3-cephfs-2:19.0.0-2145.g3be9a5c9.el9.x86_6 23/198 2024-03-24T00:02:16.053 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : python3-rados-2:19.0.0-2145.g3be9a5c9.el9.x86_64 24/198 2024-03-24T00:02:16.053 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : python3-rbd-2:19.0.0-2145.g3be9a5c9.el9.x86_64 25/198 2024-03-24T00:02:16.053 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : python3-rgw-2:19.0.0-2145.g3be9a5c9.el9.x86_64 26/198 2024-03-24T00:02:16.053 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : rbd-fuse-2:19.0.0-2145.g3be9a5c9.el9.x86_64 27/198 2024-03-24T00:02:16.053 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : rbd-mirror-2:19.0.0-2145.g3be9a5c9.el9.x86_64 28/198 2024-03-24T00:02:16.053 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : rbd-nbd-2:19.0.0-2145.g3be9a5c9.el9.x86_64 29/198 2024-03-24T00:02:16.069 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : ceph-grafana-dashboards-2:19.0.0-2145.g3be9a5c9. 30/198 2024-03-24T00:02:16.069 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : ceph-mgr-cephadm-2:19.0.0-2145.g3be9a5c9.el9.noa 31/198 2024-03-24T00:02:16.069 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : ceph-mgr-dashboard-2:19.0.0-2145.g3be9a5c9.el9.n 32/198 2024-03-24T00:02:16.069 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : ceph-mgr-diskprediction-local-2:19.0.0-2145.g3be 33/198 2024-03-24T00:02:16.069 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : ceph-mgr-modules-core-2:19.0.0-2145.g3be9a5c9.el 34/198 2024-03-24T00:02:16.069 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : ceph-mgr-rook-2:19.0.0-2145.g3be9a5c9.el9.noarch 35/198 2024-03-24T00:02:16.069 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : ceph-prometheus-alerts-2:19.0.0-2145.g3be9a5c9.e 36/198 2024-03-24T00:02:16.069 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : ceph-volume-2:19.0.0-2145.g3be9a5c9.el9.noarch 37/198 2024-03-24T00:02:16.069 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : cephadm-2:19.0.0-2145.g3be9a5c9.el9.noarch 38/198 2024-03-24T00:02:16.069 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : cephfs-top-2:19.0.0-2145.g3be9a5c9.el9.noarch 39/198 2024-03-24T00:02:16.069 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : automake-1.16.2-8.el9.noarch 40/198 2024-03-24T00:02:16.069 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : bison-3.7.4-5.el9.x86_64 41/198 2024-03-24T00:02:16.070 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 42/198 2024-03-24T00:02:16.070 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : flex-2.6.4-9.el9.x86_64 43/198 2024-03-24T00:02:16.070 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 44/198 2024-03-24T00:02:16.070 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 45/198 2024-03-24T00:02:16.070 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 46/198 2024-03-24T00:02:16.070 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : libaio-devel-0.3.111-13.el9.x86_64 47/198 2024-03-24T00:02:16.070 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 48/198 2024-03-24T00:02:16.070 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 49/198 2024-03-24T00:02:16.070 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 50/198 2024-03-24T00:02:16.070 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : libtool-2.4.6-45.el9.x86_64 51/198 2024-03-24T00:02:16.070 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : libzstd-devel-1.5.1-2.el9.x86_64 52/198 2024-03-24T00:02:16.070 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 53/198 2024-03-24T00:02:16.070 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : lua-5.4.4-4.el9.x86_64 54/198 2024-03-24T00:02:16.070 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : oniguruma-6.9.6-1.el9.5.x86_64 55/198 2024-03-24T00:02:16.071 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : openblas-0.3.21-2.el9.x86_64 56/198 2024-03-24T00:02:16.071 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : openblas-openmp-0.3.21-2.el9.x86_64 57/198 2024-03-24T00:02:16.071 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : postgresql-13.11-1.el9.x86_64 58/198 2024-03-24T00:02:16.071 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : postgresql-contrib-13.11-1.el9.x86_64 59/198 2024-03-24T00:02:16.071 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : postgresql-private-libs-13.11-1.el9.x86_64 60/198 2024-03-24T00:02:16.071 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : postgresql-server-13.11-1.el9.x86_64 61/198 2024-03-24T00:02:16.071 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : protobuf-3.14.0-13.el9.x86_64 62/198 2024-03-24T00:02:16.071 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : python3-attrs-20.3.0-7.el9.noarch 63/198 2024-03-24T00:02:16.071 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 64/198 2024-03-24T00:02:16.071 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 65/198 2024-03-24T00:02:16.071 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 66/198 2024-03-24T00:02:16.071 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 67/198 2024-03-24T00:02:16.072 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 68/198 2024-03-24T00:02:16.072 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 69/198 2024-03-24T00:02:16.072 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 70/198 2024-03-24T00:02:16.072 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 71/198 2024-03-24T00:02:16.072 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 72/198 2024-03-24T00:02:16.072 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 73/198 2024-03-24T00:02:16.072 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : python3-protobuf-3.14.0-13.el9.noarch 74/198 2024-03-24T00:02:16.072 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 75/198 2024-03-24T00:02:16.072 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 76/198 2024-03-24T00:02:16.072 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 77/198 2024-03-24T00:02:16.072 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 78/198 2024-03-24T00:02:16.072 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 79/198 2024-03-24T00:02:16.072 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 80/198 2024-03-24T00:02:16.073 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : python3-wcwidth-0.2.5-8.el9.noarch 81/198 2024-03-24T00:02:16.073 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : socat-1.7.4.1-5.el9.x86_64 82/198 2024-03-24T00:02:16.073 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : uuid-1.6.2-55.el9.x86_64 83/198 2024-03-24T00:02:16.073 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 84/198 2024-03-24T00:02:16.073 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : zlib-devel-1.2.11-40.el9.x86_64 85/198 2024-03-24T00:02:16.073 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : fuse-2.9.9-15.el9.x86_64 86/198 2024-03-24T00:02:16.073 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 87/198 2024-03-24T00:02:16.073 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 88/198 2024-03-24T00:02:16.073 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : python3-ply-3.11-14.el9.noarch 89/198 2024-03-24T00:02:16.073 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 90/198 2024-03-24T00:02:16.073 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : python3-urllib3-1.26.5-5.el9.noarch 91/198 2024-03-24T00:02:16.073 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : jq-1.6-16.el9.x86_64 92/198 2024-03-24T00:02:16.073 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : libgfortran-11.4.1-3.el9.x86_64 93/198 2024-03-24T00:02:16.074 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : libquadmath-11.4.1-3.el9.x86_64 94/198 2024-03-24T00:02:16.074 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 95/198 2024-03-24T00:02:16.074 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : elfutils-libelf-devel-0.190-2.el9.x86_64 96/198 2024-03-24T00:02:16.074 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : libuuid-devel-2.37.4-18.el9.x86_64 97/198 2024-03-24T00:02:16.074 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : openssl-devel-1:3.0.7-27.el9.x86_64 98/198 2024-03-24T00:02:16.074 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : python3-devel-3.9.17-2.el9.x86_64 99/198 2024-03-24T00:02:16.074 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : python3-jinja2-2.11.3-5.el9.noarch 100/198 2024-03-24T00:02:16.074 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : qatlib-23.11.0-1.el9.x86_64 101/198 2024-03-24T00:02:16.074 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : qatlib-service-23.11.0-1.el9.x86_64 102/198 2024-03-24T00:02:16.074 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : qatzip-libs-1.1.2-2.el9.x86_64 103/198 2024-03-24T00:02:16.074 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : xfsprogs-devel-6.3.0-1.el9.x86_64 104/198 2024-03-24T00:02:16.074 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : lua-devel-5.4.4-4.el9.x86_64 105/198 2024-03-24T00:02:16.074 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : protobuf-compiler-3.14.0-13.el9.x86_64 106/198 2024-03-24T00:02:16.074 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : abseil-cpp-20211102.0-3.el9.x86_64 107/198 2024-03-24T00:02:16.075 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : gperftools-libs-2.9.1-2.el9.x86_64 108/198 2024-03-24T00:02:16.075 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : grpc-data-1.46.7-10.el9.noarch 109/198 2024-03-24T00:02:16.075 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : libarrow-9.0.0-10.el9.x86_64 110/198 2024-03-24T00:02:16.075 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : libarrow-doc-9.0.0-10.el9.noarch 111/198 2024-03-24T00:02:16.075 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 112/198 2024-03-24T00:02:16.075 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 113/198 2024-03-24T00:02:16.075 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : luarocks-3.9.2-1.el9.noarch 114/198 2024-03-24T00:02:16.075 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : parquet-libs-9.0.0-10.el9.x86_64 115/198 2024-03-24T00:02:16.075 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : python3-asyncssh-2.13.2-5.el9.noarch 116/198 2024-03-24T00:02:16.075 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 117/198 2024-03-24T00:02:16.075 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 118/198 2024-03-24T00:02:16.075 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : python3-certifi-2021.10.8-2.el9.noarch 119/198 2024-03-24T00:02:16.075 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 120/198 2024-03-24T00:02:16.075 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 121/198 2024-03-24T00:02:16.075 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : python3-cmd2-2.4.2-2.el9.noarch 122/198 2024-03-24T00:02:16.075 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : python3-colorama-0.4.4-12.el9.noarch 123/198 2024-03-24T00:02:16.076 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : python3-google-auth-1:2.28.2-1.el9.noarch 124/198 2024-03-24T00:02:16.076 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : python3-grpcio-1.46.7-10.el9.x86_64 125/198 2024-03-24T00:02:16.076 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : python3-grpcio-tools-1.46.7-10.el9.x86_64 126/198 2024-03-24T00:02:16.076 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 127/198 2024-03-24T00:02:16.076 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 128/198 2024-03-24T00:02:16.076 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 129/198 2024-03-24T00:02:16.076 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 130/198 2024-03-24T00:02:16.076 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 131/198 2024-03-24T00:02:16.076 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 132/198 2024-03-24T00:02:16.076 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 133/198 2024-03-24T00:02:16.076 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 134/198 2024-03-24T00:02:16.076 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 135/198 2024-03-24T00:02:16.076 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 136/198 2024-03-24T00:02:16.076 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 137/198 2024-03-24T00:02:16.077 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 138/198 2024-03-24T00:02:16.077 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : python3-pyperclip-1.8.2-2.el9.noarch 139/198 2024-03-24T00:02:16.077 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 140/198 2024-03-24T00:02:16.077 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 141/198 2024-03-24T00:02:16.077 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 142/198 2024-03-24T00:02:16.077 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 143/198 2024-03-24T00:02:16.077 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : python3-typing-extensions-4.4.0-2.el9.noarch 144/198 2024-03-24T00:02:16.077 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : python3-webob-1.8.7-6.el9.noarch 145/198 2024-03-24T00:02:16.077 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 146/198 2024-03-24T00:02:16.077 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 147/198 2024-03-24T00:02:16.077 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 148/198 2024-03-24T00:02:16.077 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : re2-1:20211101-3.el9.x86_64 149/198 2024-03-24T00:02:16.077 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 150/198 2024-03-24T00:02:16.077 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : librados2-2:19.0.0-2145.g3be9a5c9.el9.x86_64 151/198 2024-03-24T00:02:16.077 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 152/198 2024-03-24T00:02:16.078 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : librbd1-2:19.0.0-2145.g3be9a5c9.el9.x86_64 153/198 2024-03-24T00:02:16.078 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 154/198 2024-03-24T00:02:16.078 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : xfsdump-3.1.12-4.el9.x86_64 155/198 2024-03-24T00:02:16.078 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : xfsdump-3.1.12-3.el9.x86_64 156/198 2024-03-24T00:02:16.078 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : NetworkManager-1:1.47.2-1.el9.x86_64 157/198 2024-03-24T00:02:16.078 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : NetworkManager-1:1.44.0-2.el9.x86_64 158/198 2024-03-24T00:02:16.078 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : NetworkManager-libnm-1:1.47.2-1.el9.x86_64 159/198 2024-03-24T00:02:16.078 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : NetworkManager-libnm-1:1.44.0-2.el9.x86_64 160/198 2024-03-24T00:02:16.078 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : NetworkManager-team-1:1.47.2-1.el9.x86_64 161/198 2024-03-24T00:02:16.078 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : NetworkManager-team-1:1.44.0-2.el9.x86_64 162/198 2024-03-24T00:02:16.078 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : NetworkManager-tui-1:1.47.2-1.el9.x86_64 163/198 2024-03-24T00:02:16.078 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : NetworkManager-tui-1:1.44.0-2.el9.x86_64 164/198 2024-03-24T00:02:16.078 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : acl-2.3.1-4.el9.x86_64 165/198 2024-03-24T00:02:16.079 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : acl-2.3.1-3.el9.x86_64 166/198 2024-03-24T00:02:16.079 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : elfutils-debuginfod-client-0.190-2.el9.x86_64 167/198 2024-03-24T00:02:16.079 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : elfutils-debuginfod-client-0.189-3.el9.x86_64 168/198 2024-03-24T00:02:16.079 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : elfutils-libelf-0.190-2.el9.x86_64 169/198 2024-03-24T00:02:16.079 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : elfutils-libelf-0.189-3.el9.x86_64 170/198 2024-03-24T00:02:16.079 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : elfutils-libs-0.190-2.el9.x86_64 171/198 2024-03-24T00:02:16.079 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : elfutils-libs-0.189-3.el9.x86_64 172/198 2024-03-24T00:02:16.079 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : libacl-2.3.1-4.el9.x86_64 173/198 2024-03-24T00:02:16.079 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : libacl-2.3.1-3.el9.x86_64 174/198 2024-03-24T00:02:16.079 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : libblkid-2.37.4-18.el9.x86_64 175/198 2024-03-24T00:02:16.079 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : libblkid-2.37.4-15.el9.x86_64 176/198 2024-03-24T00:02:16.079 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : libfdisk-2.37.4-18.el9.x86_64 177/198 2024-03-24T00:02:16.079 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : libfdisk-2.37.4-15.el9.x86_64 178/198 2024-03-24T00:02:16.079 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : libmount-2.37.4-18.el9.x86_64 179/198 2024-03-24T00:02:16.079 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : libmount-2.37.4-15.el9.x86_64 180/198 2024-03-24T00:02:16.080 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : libsmartcols-2.37.4-18.el9.x86_64 181/198 2024-03-24T00:02:16.080 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : libsmartcols-2.37.4-15.el9.x86_64 182/198 2024-03-24T00:02:16.080 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : libuuid-2.37.4-18.el9.x86_64 183/198 2024-03-24T00:02:16.080 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : libuuid-2.37.4-15.el9.x86_64 184/198 2024-03-24T00:02:16.080 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : openssl-1:3.0.7-27.el9.x86_64 185/198 2024-03-24T00:02:16.080 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : openssl-1:3.0.7-24.el9.x86_64 186/198 2024-03-24T00:02:16.080 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : openssl-libs-1:3.0.7-27.el9.x86_64 187/198 2024-03-24T00:02:16.080 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : openssl-libs-1:3.0.7-24.el9.x86_64 188/198 2024-03-24T00:02:16.080 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : util-linux-2.37.4-18.el9.x86_64 189/198 2024-03-24T00:02:16.080 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : util-linux-2.37.4-15.el9.x86_64 190/198 2024-03-24T00:02:16.080 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : util-linux-core-2.37.4-18.el9.x86_64 191/198 2024-03-24T00:02:16.080 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : util-linux-core-2.37.4-15.el9.x86_64 192/198 2024-03-24T00:02:16.080 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : util-linux-user-2.37.4-18.el9.x86_64 193/198 2024-03-24T00:02:16.080 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : util-linux-user-2.37.4-15.el9.x86_64 194/198 2024-03-24T00:02:16.080 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : xfsprogs-6.3.0-1.el9.x86_64 195/198 2024-03-24T00:02:16.081 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : xfsprogs-5.19.0-4.el9.x86_64 196/198 2024-03-24T00:02:16.081 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : libacl-devel-2.3.1-4.el9.x86_64 197/198 2024-03-24T00:02:16.081 INFO:teuthology.orchestra.run.smithi193.stdout: Running scriptlet: NetworkManager-1:1.44.0-2.el9.x86_64 192/198 2024-03-24T00:02:16.081 INFO:teuthology.orchestra.run.smithi193.stdout: Cleanup : NetworkManager-libnm-1:1.44.0-2.el9.x86_64 193/198 2024-03-24T00:02:16.155 INFO:teuthology.orchestra.run.smithi193.stdout: Running scriptlet: NetworkManager-libnm-1:1.44.0-2.el9.x86_64 193/198 2024-03-24T00:02:16.180 INFO:teuthology.orchestra.run.smithi193.stdout: Cleanup : libuuid-2.37.4-15.el9.x86_64 194/198 2024-03-24T00:02:16.213 INFO:teuthology.orchestra.run.smithi193.stdout: Cleanup : libsmartcols-2.37.4-15.el9.x86_64 195/198 2024-03-24T00:02:16.226 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : ceph-volume-2:19.0.0-2145.g3be9a5c9.el9.noarch 133/198 2024-03-24T00:02:16.240 INFO:teuthology.orchestra.run.smithi193.stdout: Cleanup : libacl-2.3.1-3.el9.x86_64 196/198 2024-03-24T00:02:16.271 INFO:teuthology.orchestra.run.smithi118.stdout: Running scriptlet: ceph-volume-2:19.0.0-2145.g3be9a5c9.el9.noarch 133/198 2024-03-24T00:02:16.271 INFO:teuthology.orchestra.run.smithi118.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-24T00:02:16.271 INFO:teuthology.orchestra.run.smithi118.stdout:Invalid unit name "ceph-volume@*.service" escaped as "ceph-volume@\x2a.service". 2024-03-24T00:02:16.271 INFO:teuthology.orchestra.run.smithi118.stdout: 2024-03-24T00:02:16.273 INFO:teuthology.orchestra.run.smithi193.stdout: Cleanup : openssl-libs-1:3.0.7-24.el9.x86_64 197/198 2024-03-24T00:02:16.338 INFO:teuthology.orchestra.run.smithi193.stdout: Cleanup : elfutils-libelf-0.189-3.el9.x86_64 198/198 2024-03-24T00:02:17.911 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : ceph-osd-2:19.0.0-2145.g3be9a5c9.el9.x86_64 134/198 2024-03-24T00:02:18.002 INFO:teuthology.orchestra.run.smithi118.stdout: Running scriptlet: ceph-osd-2:19.0.0-2145.g3be9a5c9.el9.x86_64 134/198 2024-03-24T00:02:18.002 INFO:teuthology.orchestra.run.smithi118.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-24T00:02:18.002 INFO:teuthology.orchestra.run.smithi118.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-03-24T00:02:18.002 INFO:teuthology.orchestra.run.smithi118.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-03-24T00:02:18.003 INFO:teuthology.orchestra.run.smithi118.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-03-24T00:02:18.003 INFO:teuthology.orchestra.run.smithi118.stdout: 2024-03-24T00:02:18.253 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : ceph-mds-2:19.0.0-2145.g3be9a5c9.el9.x86_64 135/198 2024-03-24T00:02:18.329 INFO:teuthology.orchestra.run.smithi118.stdout: Running scriptlet: ceph-mds-2:19.0.0-2145.g3be9a5c9.el9.x86_64 135/198 2024-03-24T00:02:18.330 INFO:teuthology.orchestra.run.smithi118.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-24T00:02:18.330 INFO:teuthology.orchestra.run.smithi118.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-03-24T00:02:18.330 INFO:teuthology.orchestra.run.smithi118.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-03-24T00:02:18.330 INFO:teuthology.orchestra.run.smithi118.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-03-24T00:02:18.330 INFO:teuthology.orchestra.run.smithi118.stdout: 2024-03-24T00:02:18.812 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : ceph-mon-2:19.0.0-2145.g3be9a5c9.el9.x86_64 136/198 2024-03-24T00:02:18.877 INFO:teuthology.orchestra.run.smithi118.stdout: Running scriptlet: ceph-mon-2:19.0.0-2145.g3be9a5c9.el9.x86_64 136/198 2024-03-24T00:02:18.877 INFO:teuthology.orchestra.run.smithi118.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-24T00:02:18.877 INFO:teuthology.orchestra.run.smithi118.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-03-24T00:02:18.877 INFO:teuthology.orchestra.run.smithi118.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-03-24T00:02:18.878 INFO:teuthology.orchestra.run.smithi118.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-03-24T00:02:18.878 INFO:teuthology.orchestra.run.smithi118.stdout: 2024-03-24T00:02:18.997 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : automake-1.16.2-8.el9.noarch 137/198 2024-03-24T00:02:19.256 INFO:teuthology.orchestra.run.smithi189.stdout: Verifying : libacl-devel-2.3.1-3.el9.x86_64 198/198 2024-03-24T00:02:19.257 INFO:teuthology.orchestra.run.smithi189.stdout: 2024-03-24T00:02:19.257 INFO:teuthology.orchestra.run.smithi189.stdout:Upgraded: 2024-03-24T00:02:19.257 INFO:teuthology.orchestra.run.smithi189.stdout: NetworkManager-1:1.47.2-1.el9.x86_64 2024-03-24T00:02:19.257 INFO:teuthology.orchestra.run.smithi189.stdout: NetworkManager-libnm-1:1.47.2-1.el9.x86_64 2024-03-24T00:02:19.257 INFO:teuthology.orchestra.run.smithi189.stdout: NetworkManager-team-1:1.47.2-1.el9.x86_64 2024-03-24T00:02:19.257 INFO:teuthology.orchestra.run.smithi189.stdout: NetworkManager-tui-1:1.47.2-1.el9.x86_64 2024-03-24T00:02:19.257 INFO:teuthology.orchestra.run.smithi189.stdout: acl-2.3.1-4.el9.x86_64 2024-03-24T00:02:19.257 INFO:teuthology.orchestra.run.smithi189.stdout: elfutils-debuginfod-client-0.190-2.el9.x86_64 2024-03-24T00:02:19.257 INFO:teuthology.orchestra.run.smithi189.stdout: elfutils-libelf-0.190-2.el9.x86_64 2024-03-24T00:02:19.257 INFO:teuthology.orchestra.run.smithi189.stdout: elfutils-libs-0.190-2.el9.x86_64 2024-03-24T00:02:19.257 INFO:teuthology.orchestra.run.smithi189.stdout: libacl-2.3.1-4.el9.x86_64 2024-03-24T00:02:19.257 INFO:teuthology.orchestra.run.smithi189.stdout: libacl-devel-2.3.1-4.el9.x86_64 2024-03-24T00:02:19.258 INFO:teuthology.orchestra.run.smithi189.stdout: libblkid-2.37.4-18.el9.x86_64 2024-03-24T00:02:19.258 INFO:teuthology.orchestra.run.smithi189.stdout: libfdisk-2.37.4-18.el9.x86_64 2024-03-24T00:02:19.258 INFO:teuthology.orchestra.run.smithi189.stdout: libmount-2.37.4-18.el9.x86_64 2024-03-24T00:02:19.258 INFO:teuthology.orchestra.run.smithi189.stdout: librados2-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-24T00:02:19.258 INFO:teuthology.orchestra.run.smithi189.stdout: librbd1-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-24T00:02:19.258 INFO:teuthology.orchestra.run.smithi189.stdout: libsmartcols-2.37.4-18.el9.x86_64 2024-03-24T00:02:19.258 INFO:teuthology.orchestra.run.smithi189.stdout: libuuid-2.37.4-18.el9.x86_64 2024-03-24T00:02:19.258 INFO:teuthology.orchestra.run.smithi189.stdout: openssl-1:3.0.7-27.el9.x86_64 2024-03-24T00:02:19.258 INFO:teuthology.orchestra.run.smithi189.stdout: openssl-libs-1:3.0.7-27.el9.x86_64 2024-03-24T00:02:19.258 INFO:teuthology.orchestra.run.smithi189.stdout: util-linux-2.37.4-18.el9.x86_64 2024-03-24T00:02:19.258 INFO:teuthology.orchestra.run.smithi189.stdout: util-linux-core-2.37.4-18.el9.x86_64 2024-03-24T00:02:19.258 INFO:teuthology.orchestra.run.smithi189.stdout: util-linux-user-2.37.4-18.el9.x86_64 2024-03-24T00:02:19.258 INFO:teuthology.orchestra.run.smithi189.stdout: xfsdump-3.1.12-4.el9.x86_64 2024-03-24T00:02:19.259 INFO:teuthology.orchestra.run.smithi189.stdout: xfsprogs-6.3.0-1.el9.x86_64 2024-03-24T00:02:19.259 INFO:teuthology.orchestra.run.smithi189.stdout:Installed: 2024-03-24T00:02:19.259 INFO:teuthology.orchestra.run.smithi189.stdout: abseil-cpp-20211102.0-3.el9.x86_64 2024-03-24T00:02:19.259 INFO:teuthology.orchestra.run.smithi189.stdout: automake-1.16.2-8.el9.noarch 2024-03-24T00:02:19.259 INFO:teuthology.orchestra.run.smithi189.stdout: bison-3.7.4-5.el9.x86_64 2024-03-24T00:02:19.259 INFO:teuthology.orchestra.run.smithi189.stdout: boost-program-options-1.75.0-8.el9.x86_64 2024-03-24T00:02:19.259 INFO:teuthology.orchestra.run.smithi189.stdout: ceph-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-24T00:02:19.259 INFO:teuthology.orchestra.run.smithi189.stdout: ceph-base-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-24T00:02:19.259 INFO:teuthology.orchestra.run.smithi189.stdout: ceph-common-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-24T00:02:19.259 INFO:teuthology.orchestra.run.smithi189.stdout: ceph-fuse-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-24T00:02:19.259 INFO:teuthology.orchestra.run.smithi189.stdout: ceph-grafana-dashboards-2:19.0.0-2145.g3be9a5c9.el9.noarch 2024-03-24T00:02:19.259 INFO:teuthology.orchestra.run.smithi189.stdout: ceph-immutable-object-cache-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-24T00:02:19.260 INFO:teuthology.orchestra.run.smithi189.stdout: ceph-mds-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-24T00:02:19.260 INFO:teuthology.orchestra.run.smithi189.stdout: ceph-mgr-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-24T00:02:19.260 INFO:teuthology.orchestra.run.smithi189.stdout: ceph-mgr-cephadm-2:19.0.0-2145.g3be9a5c9.el9.noarch 2024-03-24T00:02:19.260 INFO:teuthology.orchestra.run.smithi189.stdout: ceph-mgr-dashboard-2:19.0.0-2145.g3be9a5c9.el9.noarch 2024-03-24T00:02:19.260 INFO:teuthology.orchestra.run.smithi189.stdout: ceph-mgr-diskprediction-local-2:19.0.0-2145.g3be9a5c9.el9.noarch 2024-03-24T00:02:19.260 INFO:teuthology.orchestra.run.smithi189.stdout: ceph-mgr-modules-core-2:19.0.0-2145.g3be9a5c9.el9.noarch 2024-03-24T00:02:19.260 INFO:teuthology.orchestra.run.smithi189.stdout: ceph-mgr-rook-2:19.0.0-2145.g3be9a5c9.el9.noarch 2024-03-24T00:02:19.260 INFO:teuthology.orchestra.run.smithi189.stdout: ceph-mon-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-24T00:02:19.260 INFO:teuthology.orchestra.run.smithi189.stdout: ceph-osd-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-24T00:02:19.260 INFO:teuthology.orchestra.run.smithi189.stdout: ceph-prometheus-alerts-2:19.0.0-2145.g3be9a5c9.el9.noarch 2024-03-24T00:02:19.260 INFO:teuthology.orchestra.run.smithi189.stdout: ceph-radosgw-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-24T00:02:19.260 INFO:teuthology.orchestra.run.smithi189.stdout: ceph-selinux-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-24T00:02:19.260 INFO:teuthology.orchestra.run.smithi189.stdout: ceph-test-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-24T00:02:19.260 INFO:teuthology.orchestra.run.smithi189.stdout: ceph-volume-2:19.0.0-2145.g3be9a5c9.el9.noarch 2024-03-24T00:02:19.261 INFO:teuthology.orchestra.run.smithi189.stdout: cephadm-2:19.0.0-2145.g3be9a5c9.el9.noarch 2024-03-24T00:02:19.261 INFO:teuthology.orchestra.run.smithi189.stdout: cephfs-mirror-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-24T00:02:19.261 INFO:teuthology.orchestra.run.smithi189.stdout: cephfs-shell-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-24T00:02:19.261 INFO:teuthology.orchestra.run.smithi189.stdout: cephfs-top-2:19.0.0-2145.g3be9a5c9.el9.noarch 2024-03-24T00:02:19.261 INFO:teuthology.orchestra.run.smithi189.stdout: elfutils-libelf-devel-0.190-2.el9.x86_64 2024-03-24T00:02:19.261 INFO:teuthology.orchestra.run.smithi189.stdout: flex-2.6.4-9.el9.x86_64 2024-03-24T00:02:19.261 INFO:teuthology.orchestra.run.smithi189.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-03-24T00:02:19.261 INFO:teuthology.orchestra.run.smithi189.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-03-24T00:02:19.261 INFO:teuthology.orchestra.run.smithi189.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-03-24T00:02:19.261 INFO:teuthology.orchestra.run.smithi189.stdout: fuse-2.9.9-15.el9.x86_64 2024-03-24T00:02:19.261 INFO:teuthology.orchestra.run.smithi189.stdout: gperftools-libs-2.9.1-2.el9.x86_64 2024-03-24T00:02:19.261 INFO:teuthology.orchestra.run.smithi189.stdout: grpc-data-1.46.7-10.el9.noarch 2024-03-24T00:02:19.261 INFO:teuthology.orchestra.run.smithi189.stdout: jq-1.6-16.el9.x86_64 2024-03-24T00:02:19.262 INFO:teuthology.orchestra.run.smithi189.stdout: libaio-devel-0.3.111-13.el9.x86_64 2024-03-24T00:02:19.262 INFO:teuthology.orchestra.run.smithi189.stdout: libarrow-9.0.0-10.el9.x86_64 2024-03-24T00:02:19.262 INFO:teuthology.orchestra.run.smithi189.stdout: libarrow-doc-9.0.0-10.el9.noarch 2024-03-24T00:02:19.262 INFO:teuthology.orchestra.run.smithi189.stdout: libcephfs-devel-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-24T00:02:19.262 INFO:teuthology.orchestra.run.smithi189.stdout: libcephfs2-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-24T00:02:19.262 INFO:teuthology.orchestra.run.smithi189.stdout: libcephsqlite-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-24T00:02:19.262 INFO:teuthology.orchestra.run.smithi189.stdout: libgfortran-11.4.1-3.el9.x86_64 2024-03-24T00:02:19.262 INFO:teuthology.orchestra.run.smithi189.stdout: liboath-2.6.7-2.el9.x86_64 2024-03-24T00:02:19.262 INFO:teuthology.orchestra.run.smithi189.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-03-24T00:02:19.262 INFO:teuthology.orchestra.run.smithi189.stdout: libquadmath-11.4.1-3.el9.x86_64 2024-03-24T00:02:19.262 INFO:teuthology.orchestra.run.smithi189.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-03-24T00:02:19.262 INFO:teuthology.orchestra.run.smithi189.stdout: librados-devel-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-24T00:02:19.263 INFO:teuthology.orchestra.run.smithi189.stdout: libradosstriper1-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-24T00:02:19.263 INFO:teuthology.orchestra.run.smithi189.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-03-24T00:02:19.263 INFO:teuthology.orchestra.run.smithi189.stdout: librgw2-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-24T00:02:19.263 INFO:teuthology.orchestra.run.smithi189.stdout: libtool-2.4.6-45.el9.x86_64 2024-03-24T00:02:19.263 INFO:teuthology.orchestra.run.smithi189.stdout: libunwind-1.6.2-1.el9.x86_64 2024-03-24T00:02:19.263 INFO:teuthology.orchestra.run.smithi189.stdout: libuuid-devel-2.37.4-18.el9.x86_64 2024-03-24T00:02:19.263 INFO:teuthology.orchestra.run.smithi189.stdout: libzstd-devel-1.5.1-2.el9.x86_64 2024-03-24T00:02:19.263 INFO:teuthology.orchestra.run.smithi189.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-03-24T00:02:19.263 INFO:teuthology.orchestra.run.smithi189.stdout: lua-5.4.4-4.el9.x86_64 2024-03-24T00:02:19.263 INFO:teuthology.orchestra.run.smithi189.stdout: lua-devel-5.4.4-4.el9.x86_64 2024-03-24T00:02:19.263 INFO:teuthology.orchestra.run.smithi189.stdout: luarocks-3.9.2-1.el9.noarch 2024-03-24T00:02:19.263 INFO:teuthology.orchestra.run.smithi189.stdout: oniguruma-6.9.6-1.el9.5.x86_64 2024-03-24T00:02:19.264 INFO:teuthology.orchestra.run.smithi189.stdout: openblas-0.3.21-2.el9.x86_64 2024-03-24T00:02:19.264 INFO:teuthology.orchestra.run.smithi189.stdout: openblas-openmp-0.3.21-2.el9.x86_64 2024-03-24T00:02:19.264 INFO:teuthology.orchestra.run.smithi189.stdout: openssl-devel-1:3.0.7-27.el9.x86_64 2024-03-24T00:02:19.264 INFO:teuthology.orchestra.run.smithi189.stdout: parquet-libs-9.0.0-10.el9.x86_64 2024-03-24T00:02:19.264 INFO:teuthology.orchestra.run.smithi189.stdout: postgresql-13.11-1.el9.x86_64 2024-03-24T00:02:19.264 INFO:teuthology.orchestra.run.smithi189.stdout: postgresql-contrib-13.11-1.el9.x86_64 2024-03-24T00:02:19.264 INFO:teuthology.orchestra.run.smithi189.stdout: postgresql-private-libs-13.11-1.el9.x86_64 2024-03-24T00:02:19.264 INFO:teuthology.orchestra.run.smithi189.stdout: postgresql-server-13.11-1.el9.x86_64 2024-03-24T00:02:19.264 INFO:teuthology.orchestra.run.smithi189.stdout: protobuf-3.14.0-13.el9.x86_64 2024-03-24T00:02:19.264 INFO:teuthology.orchestra.run.smithi189.stdout: protobuf-compiler-3.14.0-13.el9.x86_64 2024-03-24T00:02:19.264 INFO:teuthology.orchestra.run.smithi189.stdout: python3-asyncssh-2.13.2-5.el9.noarch 2024-03-24T00:02:19.264 INFO:teuthology.orchestra.run.smithi189.stdout: python3-attrs-20.3.0-7.el9.noarch 2024-03-24T00:02:19.264 INFO:teuthology.orchestra.run.smithi189.stdout: python3-babel-2.9.1-2.el9.noarch 2024-03-24T00:02:19.264 INFO:teuthology.orchestra.run.smithi189.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-03-24T00:02:19.265 INFO:teuthology.orchestra.run.smithi189.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-03-24T00:02:19.265 INFO:teuthology.orchestra.run.smithi189.stdout: python3-ceph-argparse-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-24T00:02:19.265 INFO:teuthology.orchestra.run.smithi189.stdout: python3-ceph-common-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-24T00:02:19.265 INFO:teuthology.orchestra.run.smithi189.stdout: python3-cephfs-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-24T00:02:19.265 INFO:teuthology.orchestra.run.smithi189.stdout: python3-certifi-2021.10.8-2.el9.noarch 2024-03-24T00:02:19.265 INFO:teuthology.orchestra.run.smithi189.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-03-24T00:02:19.265 INFO:teuthology.orchestra.run.smithi189.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-03-24T00:02:19.265 INFO:teuthology.orchestra.run.smithi189.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-03-24T00:02:19.265 INFO:teuthology.orchestra.run.smithi189.stdout: python3-cmd2-2.4.2-2.el9.noarch 2024-03-24T00:02:19.265 INFO:teuthology.orchestra.run.smithi189.stdout: python3-colorama-0.4.4-12.el9.noarch 2024-03-24T00:02:19.265 INFO:teuthology.orchestra.run.smithi189.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-03-24T00:02:19.266 INFO:teuthology.orchestra.run.smithi189.stdout: python3-devel-3.9.17-2.el9.x86_64 2024-03-24T00:02:19.266 INFO:teuthology.orchestra.run.smithi189.stdout: python3-google-auth-1:2.28.2-1.el9.noarch 2024-03-24T00:02:19.266 INFO:teuthology.orchestra.run.smithi189.stdout: python3-grpcio-1.46.7-10.el9.x86_64 2024-03-24T00:02:19.266 INFO:teuthology.orchestra.run.smithi189.stdout: python3-grpcio-tools-1.46.7-10.el9.x86_64 2024-03-24T00:02:19.266 INFO:teuthology.orchestra.run.smithi189.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-03-24T00:02:19.266 INFO:teuthology.orchestra.run.smithi189.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-03-24T00:02:19.266 INFO:teuthology.orchestra.run.smithi189.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-03-24T00:02:19.266 INFO:teuthology.orchestra.run.smithi189.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-03-24T00:02:19.266 INFO:teuthology.orchestra.run.smithi189.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-03-24T00:02:19.266 INFO:teuthology.orchestra.run.smithi189.stdout: python3-jinja2-2.11.3-5.el9.noarch 2024-03-24T00:02:19.266 INFO:teuthology.orchestra.run.smithi189.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-03-24T00:02:19.267 INFO:teuthology.orchestra.run.smithi189.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-03-24T00:02:19.267 INFO:teuthology.orchestra.run.smithi189.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-03-24T00:02:19.267 INFO:teuthology.orchestra.run.smithi189.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-03-24T00:02:19.267 INFO:teuthology.orchestra.run.smithi189.stdout: python3-mako-1.1.4-6.el9.noarch 2024-03-24T00:02:19.267 INFO:teuthology.orchestra.run.smithi189.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-03-24T00:02:19.267 INFO:teuthology.orchestra.run.smithi189.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-03-24T00:02:19.267 INFO:teuthology.orchestra.run.smithi189.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-03-24T00:02:19.267 INFO:teuthology.orchestra.run.smithi189.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-03-24T00:02:19.267 INFO:teuthology.orchestra.run.smithi189.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-03-24T00:02:19.267 INFO:teuthology.orchestra.run.smithi189.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-03-24T00:02:19.267 INFO:teuthology.orchestra.run.smithi189.stdout: python3-packaging-20.9-5.el9.noarch 2024-03-24T00:02:19.267 INFO:teuthology.orchestra.run.smithi189.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-03-24T00:02:19.267 INFO:teuthology.orchestra.run.smithi189.stdout: python3-ply-3.11-14.el9.noarch 2024-03-24T00:02:19.267 INFO:teuthology.orchestra.run.smithi189.stdout: python3-portend-3.1.0-2.el9.noarch 2024-03-24T00:02:19.268 INFO:teuthology.orchestra.run.smithi189.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-03-24T00:02:19.268 INFO:teuthology.orchestra.run.smithi189.stdout: python3-protobuf-3.14.0-13.el9.noarch 2024-03-24T00:02:19.268 INFO:teuthology.orchestra.run.smithi189.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-03-24T00:02:19.268 INFO:teuthology.orchestra.run.smithi189.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-03-24T00:02:19.268 INFO:teuthology.orchestra.run.smithi189.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-03-24T00:02:19.268 INFO:teuthology.orchestra.run.smithi189.stdout: python3-pycparser-2.20-6.el9.noarch 2024-03-24T00:02:19.268 INFO:teuthology.orchestra.run.smithi189.stdout: python3-pyperclip-1.8.2-2.el9.noarch 2024-03-24T00:02:19.268 INFO:teuthology.orchestra.run.smithi189.stdout: python3-pytz-2021.1-5.el9.noarch 2024-03-24T00:02:19.268 INFO:teuthology.orchestra.run.smithi189.stdout: python3-rados-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-24T00:02:19.268 INFO:teuthology.orchestra.run.smithi189.stdout: python3-rbd-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-24T00:02:19.268 INFO:teuthology.orchestra.run.smithi189.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-03-24T00:02:19.268 INFO:teuthology.orchestra.run.smithi189.stdout: python3-requests-2.25.1-8.el9.noarch 2024-03-24T00:02:19.268 INFO:teuthology.orchestra.run.smithi189.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-03-24T00:02:19.269 INFO:teuthology.orchestra.run.smithi189.stdout: python3-rgw-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-24T00:02:19.269 INFO:teuthology.orchestra.run.smithi189.stdout: python3-routes-2.5.1-5.el9.noarch 2024-03-24T00:02:19.269 INFO:teuthology.orchestra.run.smithi189.stdout: python3-rsa-4.9-2.el9.noarch 2024-03-24T00:02:19.269 INFO:teuthology.orchestra.run.smithi189.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-03-24T00:02:19.269 INFO:teuthology.orchestra.run.smithi189.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-03-24T00:02:19.269 INFO:teuthology.orchestra.run.smithi189.stdout: python3-toml-0.10.2-6.el9.noarch 2024-03-24T00:02:19.269 INFO:teuthology.orchestra.run.smithi189.stdout: python3-typing-extensions-4.4.0-2.el9.noarch 2024-03-24T00:02:19.269 INFO:teuthology.orchestra.run.smithi189.stdout: python3-urllib3-1.26.5-5.el9.noarch 2024-03-24T00:02:19.269 INFO:teuthology.orchestra.run.smithi189.stdout: python3-wcwidth-0.2.5-8.el9.noarch 2024-03-24T00:02:19.269 INFO:teuthology.orchestra.run.smithi189.stdout: python3-webob-1.8.7-6.el9.noarch 2024-03-24T00:02:19.269 INFO:teuthology.orchestra.run.smithi189.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-03-24T00:02:19.269 INFO:teuthology.orchestra.run.smithi189.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-03-24T00:02:19.269 INFO:teuthology.orchestra.run.smithi189.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-03-24T00:02:19.270 INFO:teuthology.orchestra.run.smithi189.stdout: qatlib-23.11.0-1.el9.x86_64 2024-03-24T00:02:19.270 INFO:teuthology.orchestra.run.smithi189.stdout: qatlib-service-23.11.0-1.el9.x86_64 2024-03-24T00:02:19.270 INFO:teuthology.orchestra.run.smithi189.stdout: qatzip-libs-1.1.2-2.el9.x86_64 2024-03-24T00:02:19.270 INFO:teuthology.orchestra.run.smithi189.stdout: rbd-fuse-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-24T00:02:19.270 INFO:teuthology.orchestra.run.smithi189.stdout: rbd-mirror-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-24T00:02:19.270 INFO:teuthology.orchestra.run.smithi189.stdout: rbd-nbd-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-24T00:02:19.270 INFO:teuthology.orchestra.run.smithi189.stdout: re2-1:20211101-3.el9.x86_64 2024-03-24T00:02:19.270 INFO:teuthology.orchestra.run.smithi189.stdout: socat-1.7.4.1-5.el9.x86_64 2024-03-24T00:02:19.270 INFO:teuthology.orchestra.run.smithi189.stdout: thrift-0.15.0-2.el9.x86_64 2024-03-24T00:02:19.270 INFO:teuthology.orchestra.run.smithi189.stdout: uuid-1.6.2-55.el9.x86_64 2024-03-24T00:02:19.270 INFO:teuthology.orchestra.run.smithi189.stdout: xfsprogs-devel-6.3.0-1.el9.x86_64 2024-03-24T00:02:19.270 INFO:teuthology.orchestra.run.smithi189.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-03-24T00:02:19.271 INFO:teuthology.orchestra.run.smithi189.stdout: zlib-devel-1.2.11-40.el9.x86_64 2024-03-24T00:02:19.271 INFO:teuthology.orchestra.run.smithi189.stdout: 2024-03-24T00:02:19.271 INFO:teuthology.orchestra.run.smithi189.stdout:Complete! 2024-03-24T00:02:19.703 INFO:teuthology.orchestra.run.smithi118.stdout: Running scriptlet: cephadm-2:19.0.0-2145.g3be9a5c9.el9.noarch 138/198 2024-03-24T00:02:19.715 DEBUG:teuthology.parallel:result is None 2024-03-24T00:02:19.749 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : cephadm-2:19.0.0-2145.g3be9a5c9.el9.noarch 138/198 2024-03-24T00:02:19.800 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : ceph-prometheus-alerts-2:19.0.0-2145.g3be9a5c9.e 139/198 2024-03-24T00:02:19.900 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : ceph-grafana-dashboards-2:19.0.0-2145.g3be9a5c9. 140/198 2024-03-24T00:02:19.944 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : ceph-mgr-cephadm-2:19.0.0-2145.g3be9a5c9.el9.noa 141/198 2024-03-24T00:02:20.273 INFO:teuthology.orchestra.run.smithi118.stdout: Running scriptlet: ceph-mgr-cephadm-2:19.0.0-2145.g3be9a5c9.el9.noa 141/198 2024-03-24T00:02:20.304 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : ceph-mgr-dashboard-2:19.0.0-2145.g3be9a5c9.el9.n 142/198 2024-03-24T00:02:21.414 INFO:teuthology.orchestra.run.smithi118.stdout: Running scriptlet: ceph-mgr-dashboard-2:19.0.0-2145.g3be9a5c9.el9.n 142/198 2024-03-24T00:02:21.469 INFO:teuthology.orchestra.run.smithi193.stdout: Running scriptlet: elfutils-libelf-0.189-3.el9.x86_64 198/198 2024-03-24T00:02:21.469 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : ceph-2:19.0.0-2145.g3be9a5c9.el9.x86_64 1/198 2024-03-24T00:02:21.469 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : ceph-base-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2/198 2024-03-24T00:02:21.469 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : ceph-common-2:19.0.0-2145.g3be9a5c9.el9.x86_64 3/198 2024-03-24T00:02:21.469 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : ceph-fuse-2:19.0.0-2145.g3be9a5c9.el9.x86_64 4/198 2024-03-24T00:02:21.469 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : ceph-immutable-object-cache-2:19.0.0-2145.g3be9a 5/198 2024-03-24T00:02:21.469 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : ceph-mds-2:19.0.0-2145.g3be9a5c9.el9.x86_64 6/198 2024-03-24T00:02:21.470 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : ceph-mgr-2:19.0.0-2145.g3be9a5c9.el9.x86_64 7/198 2024-03-24T00:02:21.470 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : ceph-mon-2:19.0.0-2145.g3be9a5c9.el9.x86_64 8/198 2024-03-24T00:02:21.470 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : ceph-osd-2:19.0.0-2145.g3be9a5c9.el9.x86_64 9/198 2024-03-24T00:02:21.470 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : ceph-radosgw-2:19.0.0-2145.g3be9a5c9.el9.x86_64 10/198 2024-03-24T00:02:21.470 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : ceph-selinux-2:19.0.0-2145.g3be9a5c9.el9.x86_64 11/198 2024-03-24T00:02:21.470 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : ceph-test-2:19.0.0-2145.g3be9a5c9.el9.x86_64 12/198 2024-03-24T00:02:21.470 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : cephfs-mirror-2:19.0.0-2145.g3be9a5c9.el9.x86_64 13/198 2024-03-24T00:02:21.470 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : cephfs-shell-2:19.0.0-2145.g3be9a5c9.el9.x86_64 14/198 2024-03-24T00:02:21.470 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : libcephfs-devel-2:19.0.0-2145.g3be9a5c9.el9.x86_ 15/198 2024-03-24T00:02:21.470 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : libcephfs2-2:19.0.0-2145.g3be9a5c9.el9.x86_64 16/198 2024-03-24T00:02:21.470 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : libcephsqlite-2:19.0.0-2145.g3be9a5c9.el9.x86_64 17/198 2024-03-24T00:02:21.470 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : librados-devel-2:19.0.0-2145.g3be9a5c9.el9.x86_6 18/198 2024-03-24T00:02:21.470 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : libradosstriper1-2:19.0.0-2145.g3be9a5c9.el9.x86 19/198 2024-03-24T00:02:21.471 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : librgw2-2:19.0.0-2145.g3be9a5c9.el9.x86_64 20/198 2024-03-24T00:02:21.471 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : python3-ceph-argparse-2:19.0.0-2145.g3be9a5c9.el 21/198 2024-03-24T00:02:21.471 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : python3-ceph-common-2:19.0.0-2145.g3be9a5c9.el9. 22/198 2024-03-24T00:02:21.471 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : python3-cephfs-2:19.0.0-2145.g3be9a5c9.el9.x86_6 23/198 2024-03-24T00:02:21.471 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : python3-rados-2:19.0.0-2145.g3be9a5c9.el9.x86_64 24/198 2024-03-24T00:02:21.471 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : python3-rbd-2:19.0.0-2145.g3be9a5c9.el9.x86_64 25/198 2024-03-24T00:02:21.471 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : python3-rgw-2:19.0.0-2145.g3be9a5c9.el9.x86_64 26/198 2024-03-24T00:02:21.471 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : rbd-fuse-2:19.0.0-2145.g3be9a5c9.el9.x86_64 27/198 2024-03-24T00:02:21.471 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : rbd-mirror-2:19.0.0-2145.g3be9a5c9.el9.x86_64 28/198 2024-03-24T00:02:21.471 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : rbd-nbd-2:19.0.0-2145.g3be9a5c9.el9.x86_64 29/198 2024-03-24T00:02:21.471 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : ceph-grafana-dashboards-2:19.0.0-2145.g3be9a5c9. 30/198 2024-03-24T00:02:21.471 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : ceph-mgr-cephadm-2:19.0.0-2145.g3be9a5c9.el9.noa 31/198 2024-03-24T00:02:21.471 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : ceph-mgr-dashboard-2:19.0.0-2145.g3be9a5c9.el9.n 32/198 2024-03-24T00:02:21.471 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : ceph-mgr-diskprediction-local-2:19.0.0-2145.g3be 33/198 2024-03-24T00:02:21.471 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : ceph-mgr-modules-core-2:19.0.0-2145.g3be9a5c9.el 34/198 2024-03-24T00:02:21.472 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : ceph-mgr-rook-2:19.0.0-2145.g3be9a5c9.el9.noarch 35/198 2024-03-24T00:02:21.472 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : ceph-prometheus-alerts-2:19.0.0-2145.g3be9a5c9.e 36/198 2024-03-24T00:02:21.472 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : ceph-volume-2:19.0.0-2145.g3be9a5c9.el9.noarch 37/198 2024-03-24T00:02:21.472 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : cephadm-2:19.0.0-2145.g3be9a5c9.el9.noarch 38/198 2024-03-24T00:02:21.472 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : cephfs-top-2:19.0.0-2145.g3be9a5c9.el9.noarch 39/198 2024-03-24T00:02:21.472 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : automake-1.16.2-8.el9.noarch 40/198 2024-03-24T00:02:21.472 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : bison-3.7.4-5.el9.x86_64 41/198 2024-03-24T00:02:21.472 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 42/198 2024-03-24T00:02:21.472 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : flex-2.6.4-9.el9.x86_64 43/198 2024-03-24T00:02:21.472 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 44/198 2024-03-24T00:02:21.472 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 45/198 2024-03-24T00:02:21.472 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 46/198 2024-03-24T00:02:21.472 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : libaio-devel-0.3.111-13.el9.x86_64 47/198 2024-03-24T00:02:21.473 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 48/198 2024-03-24T00:02:21.473 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 49/198 2024-03-24T00:02:21.473 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 50/198 2024-03-24T00:02:21.473 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : libtool-2.4.6-45.el9.x86_64 51/198 2024-03-24T00:02:21.473 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : libzstd-devel-1.5.1-2.el9.x86_64 52/198 2024-03-24T00:02:21.473 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 53/198 2024-03-24T00:02:21.473 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : lua-5.4.4-4.el9.x86_64 54/198 2024-03-24T00:02:21.473 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : oniguruma-6.9.6-1.el9.5.x86_64 55/198 2024-03-24T00:02:21.473 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : openblas-0.3.21-2.el9.x86_64 56/198 2024-03-24T00:02:21.473 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : openblas-openmp-0.3.21-2.el9.x86_64 57/198 2024-03-24T00:02:21.473 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : postgresql-13.11-1.el9.x86_64 58/198 2024-03-24T00:02:21.473 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : postgresql-contrib-13.11-1.el9.x86_64 59/198 2024-03-24T00:02:21.473 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : postgresql-private-libs-13.11-1.el9.x86_64 60/198 2024-03-24T00:02:21.473 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : postgresql-server-13.11-1.el9.x86_64 61/198 2024-03-24T00:02:21.474 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : protobuf-3.14.0-13.el9.x86_64 62/198 2024-03-24T00:02:21.474 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : python3-attrs-20.3.0-7.el9.noarch 63/198 2024-03-24T00:02:21.474 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 64/198 2024-03-24T00:02:21.474 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 65/198 2024-03-24T00:02:21.474 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 66/198 2024-03-24T00:02:21.474 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 67/198 2024-03-24T00:02:21.474 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 68/198 2024-03-24T00:02:21.474 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 69/198 2024-03-24T00:02:21.474 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 70/198 2024-03-24T00:02:21.474 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 71/198 2024-03-24T00:02:21.474 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 72/198 2024-03-24T00:02:21.474 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 73/198 2024-03-24T00:02:21.474 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : python3-protobuf-3.14.0-13.el9.noarch 74/198 2024-03-24T00:02:21.474 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 75/198 2024-03-24T00:02:21.475 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 76/198 2024-03-24T00:02:21.475 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 77/198 2024-03-24T00:02:21.475 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 78/198 2024-03-24T00:02:21.475 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 79/198 2024-03-24T00:02:21.475 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 80/198 2024-03-24T00:02:21.475 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : python3-wcwidth-0.2.5-8.el9.noarch 81/198 2024-03-24T00:02:21.475 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : socat-1.7.4.1-5.el9.x86_64 82/198 2024-03-24T00:02:21.475 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : uuid-1.6.2-55.el9.x86_64 83/198 2024-03-24T00:02:21.475 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 84/198 2024-03-24T00:02:21.475 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : zlib-devel-1.2.11-40.el9.x86_64 85/198 2024-03-24T00:02:21.475 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : fuse-2.9.9-15.el9.x86_64 86/198 2024-03-24T00:02:21.475 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 87/198 2024-03-24T00:02:21.475 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 88/198 2024-03-24T00:02:21.475 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : python3-ply-3.11-14.el9.noarch 89/198 2024-03-24T00:02:21.475 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 90/198 2024-03-24T00:02:21.476 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : python3-urllib3-1.26.5-5.el9.noarch 91/198 2024-03-24T00:02:21.476 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : jq-1.6-16.el9.x86_64 92/198 2024-03-24T00:02:21.476 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : libgfortran-11.4.1-3.el9.x86_64 93/198 2024-03-24T00:02:21.476 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : libquadmath-11.4.1-3.el9.x86_64 94/198 2024-03-24T00:02:21.476 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 95/198 2024-03-24T00:02:21.476 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : elfutils-libelf-devel-0.190-2.el9.x86_64 96/198 2024-03-24T00:02:21.476 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : libuuid-devel-2.37.4-18.el9.x86_64 97/198 2024-03-24T00:02:21.476 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : openssl-devel-1:3.0.7-27.el9.x86_64 98/198 2024-03-24T00:02:21.476 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : python3-devel-3.9.17-2.el9.x86_64 99/198 2024-03-24T00:02:21.476 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : python3-jinja2-2.11.3-5.el9.noarch 100/198 2024-03-24T00:02:21.476 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : qatlib-23.11.0-1.el9.x86_64 101/198 2024-03-24T00:02:21.476 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : qatlib-service-23.11.0-1.el9.x86_64 102/198 2024-03-24T00:02:21.476 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : qatzip-libs-1.1.2-2.el9.x86_64 103/198 2024-03-24T00:02:21.476 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : xfsprogs-devel-6.3.0-1.el9.x86_64 104/198 2024-03-24T00:02:21.477 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : lua-devel-5.4.4-4.el9.x86_64 105/198 2024-03-24T00:02:21.477 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : protobuf-compiler-3.14.0-13.el9.x86_64 106/198 2024-03-24T00:02:21.477 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : abseil-cpp-20211102.0-3.el9.x86_64 107/198 2024-03-24T00:02:21.477 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : gperftools-libs-2.9.1-2.el9.x86_64 108/198 2024-03-24T00:02:21.477 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : grpc-data-1.46.7-10.el9.noarch 109/198 2024-03-24T00:02:21.477 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : libarrow-9.0.0-10.el9.x86_64 110/198 2024-03-24T00:02:21.477 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : libarrow-doc-9.0.0-10.el9.noarch 111/198 2024-03-24T00:02:21.477 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 112/198 2024-03-24T00:02:21.477 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 113/198 2024-03-24T00:02:21.477 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : luarocks-3.9.2-1.el9.noarch 114/198 2024-03-24T00:02:21.477 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : parquet-libs-9.0.0-10.el9.x86_64 115/198 2024-03-24T00:02:21.477 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : python3-asyncssh-2.13.2-5.el9.noarch 116/198 2024-03-24T00:02:21.477 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 117/198 2024-03-24T00:02:21.477 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 118/198 2024-03-24T00:02:21.478 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : python3-certifi-2021.10.8-2.el9.noarch 119/198 2024-03-24T00:02:21.478 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 120/198 2024-03-24T00:02:21.478 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 121/198 2024-03-24T00:02:21.478 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : python3-cmd2-2.4.2-2.el9.noarch 122/198 2024-03-24T00:02:21.478 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : python3-colorama-0.4.4-12.el9.noarch 123/198 2024-03-24T00:02:21.478 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : python3-google-auth-1:2.28.2-1.el9.noarch 124/198 2024-03-24T00:02:21.478 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : python3-grpcio-1.46.7-10.el9.x86_64 125/198 2024-03-24T00:02:21.478 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : python3-grpcio-tools-1.46.7-10.el9.x86_64 126/198 2024-03-24T00:02:21.478 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 127/198 2024-03-24T00:02:21.478 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 128/198 2024-03-24T00:02:21.478 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 129/198 2024-03-24T00:02:21.478 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 130/198 2024-03-24T00:02:21.478 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 131/198 2024-03-24T00:02:21.478 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 132/198 2024-03-24T00:02:21.479 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 133/198 2024-03-24T00:02:21.479 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 134/198 2024-03-24T00:02:21.479 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 135/198 2024-03-24T00:02:21.479 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 136/198 2024-03-24T00:02:21.479 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 137/198 2024-03-24T00:02:21.479 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 138/198 2024-03-24T00:02:21.479 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : python3-pyperclip-1.8.2-2.el9.noarch 139/198 2024-03-24T00:02:21.479 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 140/198 2024-03-24T00:02:21.479 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 141/198 2024-03-24T00:02:21.479 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 142/198 2024-03-24T00:02:21.479 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 143/198 2024-03-24T00:02:21.479 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : python3-typing-extensions-4.4.0-2.el9.noarch 144/198 2024-03-24T00:02:21.479 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : python3-webob-1.8.7-6.el9.noarch 145/198 2024-03-24T00:02:21.480 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 146/198 2024-03-24T00:02:21.480 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 147/198 2024-03-24T00:02:21.480 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 148/198 2024-03-24T00:02:21.480 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : re2-1:20211101-3.el9.x86_64 149/198 2024-03-24T00:02:21.480 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 150/198 2024-03-24T00:02:21.480 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : librados2-2:19.0.0-2145.g3be9a5c9.el9.x86_64 151/198 2024-03-24T00:02:21.480 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 152/198 2024-03-24T00:02:21.480 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : librbd1-2:19.0.0-2145.g3be9a5c9.el9.x86_64 153/198 2024-03-24T00:02:21.480 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 154/198 2024-03-24T00:02:21.480 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : xfsdump-3.1.12-4.el9.x86_64 155/198 2024-03-24T00:02:21.480 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : xfsdump-3.1.12-3.el9.x86_64 156/198 2024-03-24T00:02:21.480 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : NetworkManager-1:1.47.2-1.el9.x86_64 157/198 2024-03-24T00:02:21.480 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : NetworkManager-1:1.44.0-2.el9.x86_64 158/198 2024-03-24T00:02:21.481 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : NetworkManager-libnm-1:1.47.2-1.el9.x86_64 159/198 2024-03-24T00:02:21.481 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : NetworkManager-libnm-1:1.44.0-2.el9.x86_64 160/198 2024-03-24T00:02:21.481 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : NetworkManager-team-1:1.47.2-1.el9.x86_64 161/198 2024-03-24T00:02:21.481 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : NetworkManager-team-1:1.44.0-2.el9.x86_64 162/198 2024-03-24T00:02:21.481 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : NetworkManager-tui-1:1.47.2-1.el9.x86_64 163/198 2024-03-24T00:02:21.481 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : NetworkManager-tui-1:1.44.0-2.el9.x86_64 164/198 2024-03-24T00:02:21.481 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : acl-2.3.1-4.el9.x86_64 165/198 2024-03-24T00:02:21.481 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : acl-2.3.1-3.el9.x86_64 166/198 2024-03-24T00:02:21.481 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : elfutils-debuginfod-client-0.190-2.el9.x86_64 167/198 2024-03-24T00:02:21.481 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : elfutils-debuginfod-client-0.189-3.el9.x86_64 168/198 2024-03-24T00:02:21.481 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : elfutils-libelf-0.190-2.el9.x86_64 169/198 2024-03-24T00:02:21.481 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : elfutils-libelf-0.189-3.el9.x86_64 170/198 2024-03-24T00:02:21.481 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : elfutils-libs-0.190-2.el9.x86_64 171/198 2024-03-24T00:02:21.481 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : elfutils-libs-0.189-3.el9.x86_64 172/198 2024-03-24T00:02:21.481 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : libacl-2.3.1-4.el9.x86_64 173/198 2024-03-24T00:02:21.482 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : libacl-2.3.1-3.el9.x86_64 174/198 2024-03-24T00:02:21.482 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : libblkid-2.37.4-18.el9.x86_64 175/198 2024-03-24T00:02:21.482 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : libblkid-2.37.4-15.el9.x86_64 176/198 2024-03-24T00:02:21.482 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : libfdisk-2.37.4-18.el9.x86_64 177/198 2024-03-24T00:02:21.482 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : libfdisk-2.37.4-15.el9.x86_64 178/198 2024-03-24T00:02:21.482 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : libmount-2.37.4-18.el9.x86_64 179/198 2024-03-24T00:02:21.482 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : libmount-2.37.4-15.el9.x86_64 180/198 2024-03-24T00:02:21.482 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : libsmartcols-2.37.4-18.el9.x86_64 181/198 2024-03-24T00:02:21.482 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : libsmartcols-2.37.4-15.el9.x86_64 182/198 2024-03-24T00:02:21.482 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : libuuid-2.37.4-18.el9.x86_64 183/198 2024-03-24T00:02:21.482 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : libuuid-2.37.4-15.el9.x86_64 184/198 2024-03-24T00:02:21.482 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : openssl-1:3.0.7-27.el9.x86_64 185/198 2024-03-24T00:02:21.482 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : openssl-1:3.0.7-24.el9.x86_64 186/198 2024-03-24T00:02:21.482 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : openssl-libs-1:3.0.7-27.el9.x86_64 187/198 2024-03-24T00:02:21.483 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : openssl-libs-1:3.0.7-24.el9.x86_64 188/198 2024-03-24T00:02:21.483 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : util-linux-2.37.4-18.el9.x86_64 189/198 2024-03-24T00:02:21.483 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : util-linux-2.37.4-15.el9.x86_64 190/198 2024-03-24T00:02:21.483 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : util-linux-core-2.37.4-18.el9.x86_64 191/198 2024-03-24T00:02:21.483 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : util-linux-core-2.37.4-15.el9.x86_64 192/198 2024-03-24T00:02:21.483 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : util-linux-user-2.37.4-18.el9.x86_64 193/198 2024-03-24T00:02:21.483 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : util-linux-user-2.37.4-15.el9.x86_64 194/198 2024-03-24T00:02:21.483 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : xfsprogs-6.3.0-1.el9.x86_64 195/198 2024-03-24T00:02:21.483 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : xfsprogs-5.19.0-4.el9.x86_64 196/198 2024-03-24T00:02:21.483 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : libacl-devel-2.3.1-4.el9.x86_64 197/198 2024-03-24T00:02:21.484 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : ceph-mgr-diskprediction-local-2:19.0.0-2145.g3be 143/198 2024-03-24T00:02:21.630 INFO:teuthology.orchestra.run.smithi118.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:19.0.0-2145.g3be 143/198 2024-03-24T00:02:21.803 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : ceph-mgr-modules-core-2:19.0.0-2145.g3be9a5c9.el 144/198 2024-03-24T00:02:21.833 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : ceph-mgr-2:19.0.0-2145.g3be9a5c9.el9.x86_64 145/198 2024-03-24T00:02:21.904 INFO:teuthology.orchestra.run.smithi118.stdout: Running scriptlet: ceph-mgr-2:19.0.0-2145.g3be9a5c9.el9.x86_64 145/198 2024-03-24T00:02:21.904 INFO:teuthology.orchestra.run.smithi118.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-24T00:02:21.904 INFO:teuthology.orchestra.run.smithi118.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-03-24T00:02:21.904 INFO:teuthology.orchestra.run.smithi118.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-03-24T00:02:21.904 INFO:teuthology.orchestra.run.smithi118.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-03-24T00:02:21.904 INFO:teuthology.orchestra.run.smithi118.stdout: 2024-03-24T00:02:21.976 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : ceph-mgr-rook-2:19.0.0-2145.g3be9a5c9.el9.noarch 146/198 2024-03-24T00:02:22.014 INFO:teuthology.orchestra.run.smithi118.stdout: Running scriptlet: ceph-mgr-rook-2:19.0.0-2145.g3be9a5c9.el9.noarch 146/198 2024-03-24T00:02:22.080 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : ceph-2:19.0.0-2145.g3be9a5c9.el9.x86_64 147/198 2024-03-24T00:02:23.075 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : libtool-2.4.6-45.el9.x86_64 148/198 2024-03-24T00:02:23.103 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : ceph-radosgw-2:19.0.0-2145.g3be9a5c9.el9.x86_64 149/198 2024-03-24T00:02:23.178 INFO:teuthology.orchestra.run.smithi118.stdout: Running scriptlet: ceph-radosgw-2:19.0.0-2145.g3be9a5c9.el9.x86_64 149/198 2024-03-24T00:02:23.178 INFO:teuthology.orchestra.run.smithi118.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-24T00:02:23.178 INFO:teuthology.orchestra.run.smithi118.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-03-24T00:02:23.178 INFO:teuthology.orchestra.run.smithi118.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-03-24T00:02:23.178 INFO:teuthology.orchestra.run.smithi118.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-03-24T00:02:23.178 INFO:teuthology.orchestra.run.smithi118.stdout: 2024-03-24T00:02:23.246 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : ceph-immutable-object-cache-2:19.0.0-2145.g3be9a 150/198 2024-03-24T00:02:23.326 INFO:teuthology.orchestra.run.smithi118.stdout: Running scriptlet: ceph-immutable-object-cache-2:19.0.0-2145.g3be9a 150/198 2024-03-24T00:02:23.326 INFO:teuthology.orchestra.run.smithi118.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-24T00:02:23.327 INFO:teuthology.orchestra.run.smithi118.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-03-24T00:02:23.327 INFO:teuthology.orchestra.run.smithi118.stdout: 2024-03-24T00:02:23.438 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : cephfs-mirror-2:19.0.0-2145.g3be9a5c9.el9.x86_64 151/198 2024-03-24T00:02:23.513 INFO:teuthology.orchestra.run.smithi118.stdout: Running scriptlet: cephfs-mirror-2:19.0.0-2145.g3be9a5c9.el9.x86_64 151/198 2024-03-24T00:02:23.513 INFO:teuthology.orchestra.run.smithi118.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-24T00:02:23.514 INFO:teuthology.orchestra.run.smithi118.stdout:Invalid unit name "cephfs-mirror@*.service" escaped as "cephfs-mirror@\x2a.service". 2024-03-24T00:02:23.514 INFO:teuthology.orchestra.run.smithi118.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/cephfs-mirror.target → /usr/lib/systemd/system/cephfs-mirror.target. 2024-03-24T00:02:23.514 INFO:teuthology.orchestra.run.smithi118.stdout:Created symlink /etc/systemd/system/ceph.target.wants/cephfs-mirror.target → /usr/lib/systemd/system/cephfs-mirror.target. 2024-03-24T00:02:23.514 INFO:teuthology.orchestra.run.smithi118.stdout: 2024-03-24T00:02:23.905 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : rbd-mirror-2:19.0.0-2145.g3be9a5c9.el9.x86_64 152/198 2024-03-24T00:02:23.977 INFO:teuthology.orchestra.run.smithi118.stdout: Running scriptlet: rbd-mirror-2:19.0.0-2145.g3be9a5c9.el9.x86_64 152/198 2024-03-24T00:02:23.977 INFO:teuthology.orchestra.run.smithi118.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-24T00:02:23.978 INFO:teuthology.orchestra.run.smithi118.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-03-24T00:02:23.978 INFO:teuthology.orchestra.run.smithi118.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-03-24T00:02:23.978 INFO:teuthology.orchestra.run.smithi118.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-03-24T00:02:23.978 INFO:teuthology.orchestra.run.smithi118.stdout: 2024-03-24T00:02:24.421 INFO:teuthology.orchestra.run.smithi193.stdout: Verifying : libacl-devel-2.3.1-3.el9.x86_64 198/198 2024-03-24T00:02:24.421 INFO:teuthology.orchestra.run.smithi193.stdout: 2024-03-24T00:02:24.421 INFO:teuthology.orchestra.run.smithi193.stdout:Upgraded: 2024-03-24T00:02:24.421 INFO:teuthology.orchestra.run.smithi193.stdout: NetworkManager-1:1.47.2-1.el9.x86_64 2024-03-24T00:02:24.421 INFO:teuthology.orchestra.run.smithi193.stdout: NetworkManager-libnm-1:1.47.2-1.el9.x86_64 2024-03-24T00:02:24.421 INFO:teuthology.orchestra.run.smithi193.stdout: NetworkManager-team-1:1.47.2-1.el9.x86_64 2024-03-24T00:02:24.421 INFO:teuthology.orchestra.run.smithi193.stdout: NetworkManager-tui-1:1.47.2-1.el9.x86_64 2024-03-24T00:02:24.421 INFO:teuthology.orchestra.run.smithi193.stdout: acl-2.3.1-4.el9.x86_64 2024-03-24T00:02:24.421 INFO:teuthology.orchestra.run.smithi193.stdout: elfutils-debuginfod-client-0.190-2.el9.x86_64 2024-03-24T00:02:24.421 INFO:teuthology.orchestra.run.smithi193.stdout: elfutils-libelf-0.190-2.el9.x86_64 2024-03-24T00:02:24.422 INFO:teuthology.orchestra.run.smithi193.stdout: elfutils-libs-0.190-2.el9.x86_64 2024-03-24T00:02:24.422 INFO:teuthology.orchestra.run.smithi193.stdout: libacl-2.3.1-4.el9.x86_64 2024-03-24T00:02:24.422 INFO:teuthology.orchestra.run.smithi193.stdout: libacl-devel-2.3.1-4.el9.x86_64 2024-03-24T00:02:24.422 INFO:teuthology.orchestra.run.smithi193.stdout: libblkid-2.37.4-18.el9.x86_64 2024-03-24T00:02:24.422 INFO:teuthology.orchestra.run.smithi193.stdout: libfdisk-2.37.4-18.el9.x86_64 2024-03-24T00:02:24.422 INFO:teuthology.orchestra.run.smithi193.stdout: libmount-2.37.4-18.el9.x86_64 2024-03-24T00:02:24.422 INFO:teuthology.orchestra.run.smithi193.stdout: librados2-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-24T00:02:24.422 INFO:teuthology.orchestra.run.smithi193.stdout: librbd1-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-24T00:02:24.422 INFO:teuthology.orchestra.run.smithi193.stdout: libsmartcols-2.37.4-18.el9.x86_64 2024-03-24T00:02:24.422 INFO:teuthology.orchestra.run.smithi193.stdout: libuuid-2.37.4-18.el9.x86_64 2024-03-24T00:02:24.422 INFO:teuthology.orchestra.run.smithi193.stdout: openssl-1:3.0.7-27.el9.x86_64 2024-03-24T00:02:24.422 INFO:teuthology.orchestra.run.smithi193.stdout: openssl-libs-1:3.0.7-27.el9.x86_64 2024-03-24T00:02:24.423 INFO:teuthology.orchestra.run.smithi193.stdout: util-linux-2.37.4-18.el9.x86_64 2024-03-24T00:02:24.423 INFO:teuthology.orchestra.run.smithi193.stdout: util-linux-core-2.37.4-18.el9.x86_64 2024-03-24T00:02:24.423 INFO:teuthology.orchestra.run.smithi193.stdout: util-linux-user-2.37.4-18.el9.x86_64 2024-03-24T00:02:24.423 INFO:teuthology.orchestra.run.smithi193.stdout: xfsdump-3.1.12-4.el9.x86_64 2024-03-24T00:02:24.423 INFO:teuthology.orchestra.run.smithi193.stdout: xfsprogs-6.3.0-1.el9.x86_64 2024-03-24T00:02:24.423 INFO:teuthology.orchestra.run.smithi193.stdout:Installed: 2024-03-24T00:02:24.423 INFO:teuthology.orchestra.run.smithi193.stdout: abseil-cpp-20211102.0-3.el9.x86_64 2024-03-24T00:02:24.423 INFO:teuthology.orchestra.run.smithi193.stdout: automake-1.16.2-8.el9.noarch 2024-03-24T00:02:24.423 INFO:teuthology.orchestra.run.smithi193.stdout: bison-3.7.4-5.el9.x86_64 2024-03-24T00:02:24.423 INFO:teuthology.orchestra.run.smithi193.stdout: boost-program-options-1.75.0-8.el9.x86_64 2024-03-24T00:02:24.423 INFO:teuthology.orchestra.run.smithi193.stdout: ceph-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-24T00:02:24.423 INFO:teuthology.orchestra.run.smithi193.stdout: ceph-base-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-24T00:02:24.423 INFO:teuthology.orchestra.run.smithi193.stdout: ceph-common-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-24T00:02:24.423 INFO:teuthology.orchestra.run.smithi193.stdout: ceph-fuse-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-24T00:02:24.423 INFO:teuthology.orchestra.run.smithi193.stdout: ceph-grafana-dashboards-2:19.0.0-2145.g3be9a5c9.el9.noarch 2024-03-24T00:02:24.424 INFO:teuthology.orchestra.run.smithi193.stdout: ceph-immutable-object-cache-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-24T00:02:24.424 INFO:teuthology.orchestra.run.smithi193.stdout: ceph-mds-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-24T00:02:24.424 INFO:teuthology.orchestra.run.smithi193.stdout: ceph-mgr-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-24T00:02:24.424 INFO:teuthology.orchestra.run.smithi193.stdout: ceph-mgr-cephadm-2:19.0.0-2145.g3be9a5c9.el9.noarch 2024-03-24T00:02:24.424 INFO:teuthology.orchestra.run.smithi193.stdout: ceph-mgr-dashboard-2:19.0.0-2145.g3be9a5c9.el9.noarch 2024-03-24T00:02:24.424 INFO:teuthology.orchestra.run.smithi193.stdout: ceph-mgr-diskprediction-local-2:19.0.0-2145.g3be9a5c9.el9.noarch 2024-03-24T00:02:24.424 INFO:teuthology.orchestra.run.smithi193.stdout: ceph-mgr-modules-core-2:19.0.0-2145.g3be9a5c9.el9.noarch 2024-03-24T00:02:24.424 INFO:teuthology.orchestra.run.smithi193.stdout: ceph-mgr-rook-2:19.0.0-2145.g3be9a5c9.el9.noarch 2024-03-24T00:02:24.424 INFO:teuthology.orchestra.run.smithi193.stdout: ceph-mon-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-24T00:02:24.424 INFO:teuthology.orchestra.run.smithi193.stdout: ceph-osd-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-24T00:02:24.424 INFO:teuthology.orchestra.run.smithi193.stdout: ceph-prometheus-alerts-2:19.0.0-2145.g3be9a5c9.el9.noarch 2024-03-24T00:02:24.425 INFO:teuthology.orchestra.run.smithi193.stdout: ceph-radosgw-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-24T00:02:24.425 INFO:teuthology.orchestra.run.smithi193.stdout: ceph-selinux-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-24T00:02:24.425 INFO:teuthology.orchestra.run.smithi193.stdout: ceph-test-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-24T00:02:24.425 INFO:teuthology.orchestra.run.smithi193.stdout: ceph-volume-2:19.0.0-2145.g3be9a5c9.el9.noarch 2024-03-24T00:02:24.425 INFO:teuthology.orchestra.run.smithi193.stdout: cephadm-2:19.0.0-2145.g3be9a5c9.el9.noarch 2024-03-24T00:02:24.425 INFO:teuthology.orchestra.run.smithi193.stdout: cephfs-mirror-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-24T00:02:24.425 INFO:teuthology.orchestra.run.smithi193.stdout: cephfs-shell-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-24T00:02:24.425 INFO:teuthology.orchestra.run.smithi193.stdout: cephfs-top-2:19.0.0-2145.g3be9a5c9.el9.noarch 2024-03-24T00:02:24.425 INFO:teuthology.orchestra.run.smithi193.stdout: elfutils-libelf-devel-0.190-2.el9.x86_64 2024-03-24T00:02:24.425 INFO:teuthology.orchestra.run.smithi193.stdout: flex-2.6.4-9.el9.x86_64 2024-03-24T00:02:24.425 INFO:teuthology.orchestra.run.smithi193.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-03-24T00:02:24.425 INFO:teuthology.orchestra.run.smithi193.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-03-24T00:02:24.425 INFO:teuthology.orchestra.run.smithi193.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-03-24T00:02:24.425 INFO:teuthology.orchestra.run.smithi193.stdout: fuse-2.9.9-15.el9.x86_64 2024-03-24T00:02:24.425 INFO:teuthology.orchestra.run.smithi193.stdout: gperftools-libs-2.9.1-2.el9.x86_64 2024-03-24T00:02:24.425 INFO:teuthology.orchestra.run.smithi193.stdout: grpc-data-1.46.7-10.el9.noarch 2024-03-24T00:02:24.426 INFO:teuthology.orchestra.run.smithi193.stdout: jq-1.6-16.el9.x86_64 2024-03-24T00:02:24.426 INFO:teuthology.orchestra.run.smithi193.stdout: libaio-devel-0.3.111-13.el9.x86_64 2024-03-24T00:02:24.426 INFO:teuthology.orchestra.run.smithi193.stdout: libarrow-9.0.0-10.el9.x86_64 2024-03-24T00:02:24.426 INFO:teuthology.orchestra.run.smithi193.stdout: libarrow-doc-9.0.0-10.el9.noarch 2024-03-24T00:02:24.426 INFO:teuthology.orchestra.run.smithi193.stdout: libcephfs-devel-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-24T00:02:24.426 INFO:teuthology.orchestra.run.smithi193.stdout: libcephfs2-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-24T00:02:24.426 INFO:teuthology.orchestra.run.smithi193.stdout: libcephsqlite-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-24T00:02:24.426 INFO:teuthology.orchestra.run.smithi193.stdout: libgfortran-11.4.1-3.el9.x86_64 2024-03-24T00:02:24.426 INFO:teuthology.orchestra.run.smithi193.stdout: liboath-2.6.7-2.el9.x86_64 2024-03-24T00:02:24.426 INFO:teuthology.orchestra.run.smithi193.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-03-24T00:02:24.427 INFO:teuthology.orchestra.run.smithi193.stdout: libquadmath-11.4.1-3.el9.x86_64 2024-03-24T00:02:24.427 INFO:teuthology.orchestra.run.smithi193.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-03-24T00:02:24.427 INFO:teuthology.orchestra.run.smithi193.stdout: librados-devel-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-24T00:02:24.427 INFO:teuthology.orchestra.run.smithi193.stdout: libradosstriper1-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-24T00:02:24.427 INFO:teuthology.orchestra.run.smithi193.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-03-24T00:02:24.427 INFO:teuthology.orchestra.run.smithi193.stdout: librgw2-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-24T00:02:24.427 INFO:teuthology.orchestra.run.smithi193.stdout: libtool-2.4.6-45.el9.x86_64 2024-03-24T00:02:24.427 INFO:teuthology.orchestra.run.smithi193.stdout: libunwind-1.6.2-1.el9.x86_64 2024-03-24T00:02:24.427 INFO:teuthology.orchestra.run.smithi193.stdout: libuuid-devel-2.37.4-18.el9.x86_64 2024-03-24T00:02:24.427 INFO:teuthology.orchestra.run.smithi193.stdout: libzstd-devel-1.5.1-2.el9.x86_64 2024-03-24T00:02:24.427 INFO:teuthology.orchestra.run.smithi193.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-03-24T00:02:24.427 INFO:teuthology.orchestra.run.smithi193.stdout: lua-5.4.4-4.el9.x86_64 2024-03-24T00:02:24.427 INFO:teuthology.orchestra.run.smithi193.stdout: lua-devel-5.4.4-4.el9.x86_64 2024-03-24T00:02:24.427 INFO:teuthology.orchestra.run.smithi193.stdout: luarocks-3.9.2-1.el9.noarch 2024-03-24T00:02:24.427 INFO:teuthology.orchestra.run.smithi193.stdout: oniguruma-6.9.6-1.el9.5.x86_64 2024-03-24T00:02:24.428 INFO:teuthology.orchestra.run.smithi193.stdout: openblas-0.3.21-2.el9.x86_64 2024-03-24T00:02:24.428 INFO:teuthology.orchestra.run.smithi193.stdout: openblas-openmp-0.3.21-2.el9.x86_64 2024-03-24T00:02:24.428 INFO:teuthology.orchestra.run.smithi193.stdout: openssl-devel-1:3.0.7-27.el9.x86_64 2024-03-24T00:02:24.428 INFO:teuthology.orchestra.run.smithi193.stdout: parquet-libs-9.0.0-10.el9.x86_64 2024-03-24T00:02:24.428 INFO:teuthology.orchestra.run.smithi193.stdout: postgresql-13.11-1.el9.x86_64 2024-03-24T00:02:24.428 INFO:teuthology.orchestra.run.smithi193.stdout: postgresql-contrib-13.11-1.el9.x86_64 2024-03-24T00:02:24.428 INFO:teuthology.orchestra.run.smithi193.stdout: postgresql-private-libs-13.11-1.el9.x86_64 2024-03-24T00:02:24.428 INFO:teuthology.orchestra.run.smithi193.stdout: postgresql-server-13.11-1.el9.x86_64 2024-03-24T00:02:24.428 INFO:teuthology.orchestra.run.smithi193.stdout: protobuf-3.14.0-13.el9.x86_64 2024-03-24T00:02:24.428 INFO:teuthology.orchestra.run.smithi193.stdout: protobuf-compiler-3.14.0-13.el9.x86_64 2024-03-24T00:02:24.428 INFO:teuthology.orchestra.run.smithi193.stdout: python3-asyncssh-2.13.2-5.el9.noarch 2024-03-24T00:02:24.428 INFO:teuthology.orchestra.run.smithi193.stdout: python3-attrs-20.3.0-7.el9.noarch 2024-03-24T00:02:24.429 INFO:teuthology.orchestra.run.smithi193.stdout: python3-babel-2.9.1-2.el9.noarch 2024-03-24T00:02:24.429 INFO:teuthology.orchestra.run.smithi193.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-03-24T00:02:24.429 INFO:teuthology.orchestra.run.smithi193.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-03-24T00:02:24.429 INFO:teuthology.orchestra.run.smithi193.stdout: python3-ceph-argparse-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-24T00:02:24.429 INFO:teuthology.orchestra.run.smithi193.stdout: python3-ceph-common-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-24T00:02:24.429 INFO:teuthology.orchestra.run.smithi193.stdout: python3-cephfs-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-24T00:02:24.429 INFO:teuthology.orchestra.run.smithi193.stdout: python3-certifi-2021.10.8-2.el9.noarch 2024-03-24T00:02:24.429 INFO:teuthology.orchestra.run.smithi193.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-03-24T00:02:24.429 INFO:teuthology.orchestra.run.smithi193.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-03-24T00:02:24.429 INFO:teuthology.orchestra.run.smithi193.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-03-24T00:02:24.429 INFO:teuthology.orchestra.run.smithi193.stdout: python3-cmd2-2.4.2-2.el9.noarch 2024-03-24T00:02:24.429 INFO:teuthology.orchestra.run.smithi193.stdout: python3-colorama-0.4.4-12.el9.noarch 2024-03-24T00:02:24.429 INFO:teuthology.orchestra.run.smithi193.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-03-24T00:02:24.429 INFO:teuthology.orchestra.run.smithi193.stdout: python3-devel-3.9.17-2.el9.x86_64 2024-03-24T00:02:24.429 INFO:teuthology.orchestra.run.smithi193.stdout: python3-google-auth-1:2.28.2-1.el9.noarch 2024-03-24T00:02:24.430 INFO:teuthology.orchestra.run.smithi193.stdout: python3-grpcio-1.46.7-10.el9.x86_64 2024-03-24T00:02:24.430 INFO:teuthology.orchestra.run.smithi193.stdout: python3-grpcio-tools-1.46.7-10.el9.x86_64 2024-03-24T00:02:24.430 INFO:teuthology.orchestra.run.smithi193.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-03-24T00:02:24.430 INFO:teuthology.orchestra.run.smithi193.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-03-24T00:02:24.430 INFO:teuthology.orchestra.run.smithi193.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-03-24T00:02:24.430 INFO:teuthology.orchestra.run.smithi193.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-03-24T00:02:24.430 INFO:teuthology.orchestra.run.smithi193.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-03-24T00:02:24.430 INFO:teuthology.orchestra.run.smithi193.stdout: python3-jinja2-2.11.3-5.el9.noarch 2024-03-24T00:02:24.430 INFO:teuthology.orchestra.run.smithi193.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-03-24T00:02:24.430 INFO:teuthology.orchestra.run.smithi193.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-03-24T00:02:24.430 INFO:teuthology.orchestra.run.smithi193.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-03-24T00:02:24.431 INFO:teuthology.orchestra.run.smithi193.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-03-24T00:02:24.431 INFO:teuthology.orchestra.run.smithi193.stdout: python3-mako-1.1.4-6.el9.noarch 2024-03-24T00:02:24.431 INFO:teuthology.orchestra.run.smithi193.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-03-24T00:02:24.431 INFO:teuthology.orchestra.run.smithi193.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-03-24T00:02:24.431 INFO:teuthology.orchestra.run.smithi193.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-03-24T00:02:24.431 INFO:teuthology.orchestra.run.smithi193.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-03-24T00:02:24.431 INFO:teuthology.orchestra.run.smithi193.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-03-24T00:02:24.431 INFO:teuthology.orchestra.run.smithi193.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-03-24T00:02:24.431 INFO:teuthology.orchestra.run.smithi193.stdout: python3-packaging-20.9-5.el9.noarch 2024-03-24T00:02:24.431 INFO:teuthology.orchestra.run.smithi193.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-03-24T00:02:24.431 INFO:teuthology.orchestra.run.smithi193.stdout: python3-ply-3.11-14.el9.noarch 2024-03-24T00:02:24.431 INFO:teuthology.orchestra.run.smithi193.stdout: python3-portend-3.1.0-2.el9.noarch 2024-03-24T00:02:24.431 INFO:teuthology.orchestra.run.smithi193.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-03-24T00:02:24.431 INFO:teuthology.orchestra.run.smithi193.stdout: python3-protobuf-3.14.0-13.el9.noarch 2024-03-24T00:02:24.432 INFO:teuthology.orchestra.run.smithi193.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-03-24T00:02:24.432 INFO:teuthology.orchestra.run.smithi193.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-03-24T00:02:24.432 INFO:teuthology.orchestra.run.smithi193.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-03-24T00:02:24.432 INFO:teuthology.orchestra.run.smithi193.stdout: python3-pycparser-2.20-6.el9.noarch 2024-03-24T00:02:24.432 INFO:teuthology.orchestra.run.smithi193.stdout: python3-pyperclip-1.8.2-2.el9.noarch 2024-03-24T00:02:24.432 INFO:teuthology.orchestra.run.smithi193.stdout: python3-pytz-2021.1-5.el9.noarch 2024-03-24T00:02:24.432 INFO:teuthology.orchestra.run.smithi193.stdout: python3-rados-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-24T00:02:24.432 INFO:teuthology.orchestra.run.smithi193.stdout: python3-rbd-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-24T00:02:24.432 INFO:teuthology.orchestra.run.smithi193.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-03-24T00:02:24.432 INFO:teuthology.orchestra.run.smithi193.stdout: python3-requests-2.25.1-8.el9.noarch 2024-03-24T00:02:24.432 INFO:teuthology.orchestra.run.smithi193.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-03-24T00:02:24.432 INFO:teuthology.orchestra.run.smithi193.stdout: python3-rgw-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-24T00:02:24.433 INFO:teuthology.orchestra.run.smithi193.stdout: python3-routes-2.5.1-5.el9.noarch 2024-03-24T00:02:24.433 INFO:teuthology.orchestra.run.smithi193.stdout: python3-rsa-4.9-2.el9.noarch 2024-03-24T00:02:24.433 INFO:teuthology.orchestra.run.smithi193.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-03-24T00:02:24.433 INFO:teuthology.orchestra.run.smithi193.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-03-24T00:02:24.433 INFO:teuthology.orchestra.run.smithi193.stdout: python3-toml-0.10.2-6.el9.noarch 2024-03-24T00:02:24.433 INFO:teuthology.orchestra.run.smithi193.stdout: python3-typing-extensions-4.4.0-2.el9.noarch 2024-03-24T00:02:24.433 INFO:teuthology.orchestra.run.smithi193.stdout: python3-urllib3-1.26.5-5.el9.noarch 2024-03-24T00:02:24.433 INFO:teuthology.orchestra.run.smithi193.stdout: python3-wcwidth-0.2.5-8.el9.noarch 2024-03-24T00:02:24.433 INFO:teuthology.orchestra.run.smithi193.stdout: python3-webob-1.8.7-6.el9.noarch 2024-03-24T00:02:24.433 INFO:teuthology.orchestra.run.smithi193.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-03-24T00:02:24.433 INFO:teuthology.orchestra.run.smithi193.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-03-24T00:02:24.433 INFO:teuthology.orchestra.run.smithi193.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-03-24T00:02:24.434 INFO:teuthology.orchestra.run.smithi193.stdout: qatlib-23.11.0-1.el9.x86_64 2024-03-24T00:02:24.434 INFO:teuthology.orchestra.run.smithi193.stdout: qatlib-service-23.11.0-1.el9.x86_64 2024-03-24T00:02:24.434 INFO:teuthology.orchestra.run.smithi193.stdout: qatzip-libs-1.1.2-2.el9.x86_64 2024-03-24T00:02:24.434 INFO:teuthology.orchestra.run.smithi193.stdout: rbd-fuse-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-24T00:02:24.434 INFO:teuthology.orchestra.run.smithi193.stdout: rbd-mirror-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-24T00:02:24.434 INFO:teuthology.orchestra.run.smithi193.stdout: rbd-nbd-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-24T00:02:24.434 INFO:teuthology.orchestra.run.smithi193.stdout: re2-1:20211101-3.el9.x86_64 2024-03-24T00:02:24.434 INFO:teuthology.orchestra.run.smithi193.stdout: socat-1.7.4.1-5.el9.x86_64 2024-03-24T00:02:24.434 INFO:teuthology.orchestra.run.smithi193.stdout: thrift-0.15.0-2.el9.x86_64 2024-03-24T00:02:24.434 INFO:teuthology.orchestra.run.smithi193.stdout: uuid-1.6.2-55.el9.x86_64 2024-03-24T00:02:24.434 INFO:teuthology.orchestra.run.smithi193.stdout: xfsprogs-devel-6.3.0-1.el9.x86_64 2024-03-24T00:02:24.434 INFO:teuthology.orchestra.run.smithi193.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-03-24T00:02:24.435 INFO:teuthology.orchestra.run.smithi193.stdout: zlib-devel-1.2.11-40.el9.x86_64 2024-03-24T00:02:24.435 INFO:teuthology.orchestra.run.smithi193.stdout: 2024-03-24T00:02:24.435 INFO:teuthology.orchestra.run.smithi193.stdout:Complete! 2024-03-24T00:02:24.907 DEBUG:teuthology.parallel:result is None 2024-03-24T00:02:28.778 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : ceph-test-2:19.0.0-2145.g3be9a5c9.el9.x86_64 153/198 2024-03-24T00:02:28.821 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : rbd-fuse-2:19.0.0-2145.g3be9a5c9.el9.x86_64 154/198 2024-03-24T00:02:28.850 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : rbd-nbd-2:19.0.0-2145.g3be9a5c9.el9.x86_64 155/198 2024-03-24T00:02:28.888 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : elfutils-libelf-devel-0.190-2.el9.x86_64 156/198 2024-03-24T00:02:29.134 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : cephfs-shell-2:19.0.0-2145.g3be9a5c9.el9.x86_64 157/198 2024-03-24T00:02:29.292 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : postgresql-contrib-13.11-1.el9.x86_64 158/198 2024-03-24T00:02:29.344 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : ceph-fuse-2:19.0.0-2145.g3be9a5c9.el9.x86_64 159/198 2024-03-24T00:02:29.384 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : xfsprogs-devel-6.3.0-1.el9.x86_64 160/198 2024-03-24T00:02:29.443 INFO:teuthology.orchestra.run.smithi118.stdout: Upgrading : xfsdump-3.1.12-4.el9.x86_64 161/198 2024-03-24T00:02:29.479 INFO:teuthology.orchestra.run.smithi118.stdout: Upgrading : util-linux-user-2.37.4-18.el9.x86_64 162/198 2024-03-24T00:02:29.514 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : libcephfs-devel-2:19.0.0-2145.g3be9a5c9.el9.x86_ 163/198 2024-03-24T00:02:29.552 INFO:teuthology.orchestra.run.smithi118.stdout: Upgrading : elfutils-debuginfod-client-0.190-2.el9.x86_64 164/198 2024-03-24T00:02:29.605 INFO:teuthology.orchestra.run.smithi118.stdout: Upgrading : NetworkManager-team-1:1.47.2-1.el9.x86_64 165/198 2024-03-24T00:02:29.686 INFO:teuthology.orchestra.run.smithi118.stdout: Upgrading : NetworkManager-tui-1:1.47.2-1.el9.x86_64 166/198 2024-03-24T00:02:29.802 INFO:teuthology.orchestra.run.smithi118.stdout: Upgrading : acl-2.3.1-4.el9.x86_64 167/198 2024-03-24T00:02:29.845 INFO:teuthology.orchestra.run.smithi118.stdout: Upgrading : libacl-devel-2.3.1-4.el9.x86_64 168/198 2024-03-24T00:02:30.786 INFO:teuthology.orchestra.run.smithi118.stdout: Running scriptlet: postgresql-server-13.11-1.el9.x86_64 169/198 2024-03-24T00:02:30.835 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : postgresql-server-13.11-1.el9.x86_64 169/198 2024-03-24T00:02:30.891 INFO:teuthology.orchestra.run.smithi118.stdout: Running scriptlet: postgresql-server-13.11-1.el9.x86_64 169/198 2024-03-24T00:02:32.205 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : cephfs-top-2:19.0.0-2145.g3be9a5c9.el9.noarch 170/198 2024-03-24T00:02:32.308 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : openssl-devel-1:3.0.7-27.el9.x86_64 171/198 2024-03-24T00:02:32.349 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : libaio-devel-0.3.111-13.el9.x86_64 172/198 2024-03-24T00:02:32.478 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : flex-2.6.4-9.el9.x86_64 173/198 2024-03-24T00:02:32.519 INFO:teuthology.orchestra.run.smithi118.stdout: Installing : bison-3.7.4-5.el9.x86_64 174/198 2024-03-24T00:02:32.552 INFO:teuthology.orchestra.run.smithi118.stdout: Cleanup : NetworkManager-tui-1:1.44.0-2.el9.x86_64 175/198 2024-03-24T00:02:32.611 INFO:teuthology.orchestra.run.smithi118.stdout: Cleanup : elfutils-debuginfod-client-0.189-3.el9.x86_64 176/198 2024-03-24T00:02:32.653 INFO:teuthology.orchestra.run.smithi118.stdout: Cleanup : elfutils-libs-0.189-3.el9.x86_64 177/198 2024-03-24T00:02:32.653 INFO:teuthology.orchestra.run.smithi118.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 178/198 2024-03-24T00:02:32.799 INFO:teuthology.orchestra.run.smithi118.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 178/198 2024-03-24T00:02:32.840 INFO:teuthology.orchestra.run.smithi118.stdout: Cleanup : openssl-1:3.0.7-24.el9.x86_64 179/198 2024-03-24T00:02:32.841 INFO:teuthology.orchestra.run.smithi118.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 180/198 2024-03-24T00:02:32.971 INFO:teuthology.orchestra.run.smithi118.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 180/198 2024-03-24T00:02:33.015 INFO:teuthology.orchestra.run.smithi118.stdout: Cleanup : xfsdump-3.1.12-3.el9.x86_64 181/198 2024-03-24T00:02:33.086 INFO:teuthology.orchestra.run.smithi118.stdout: Cleanup : xfsprogs-5.19.0-4.el9.x86_64 182/198 2024-03-24T00:02:33.150 INFO:teuthology.orchestra.run.smithi118.stdout: Cleanup : acl-2.3.1-3.el9.x86_64 183/198 2024-03-24T00:02:33.193 INFO:teuthology.orchestra.run.smithi118.stdout: Cleanup : libacl-devel-2.3.1-3.el9.x86_64 184/198 2024-03-24T00:02:33.258 INFO:teuthology.orchestra.run.smithi118.stdout: Cleanup : util-linux-user-2.37.4-15.el9.x86_64 185/198 2024-03-24T00:02:33.332 INFO:teuthology.orchestra.run.smithi118.stdout: Cleanup : util-linux-2.37.4-15.el9.x86_64 186/198 2024-03-24T00:02:33.382 INFO:teuthology.orchestra.run.smithi118.stdout: Cleanup : util-linux-core-2.37.4-15.el9.x86_64 187/198 2024-03-24T00:02:33.415 INFO:teuthology.orchestra.run.smithi118.stdout: Cleanup : libfdisk-2.37.4-15.el9.x86_64 188/198 2024-03-24T00:02:33.449 INFO:teuthology.orchestra.run.smithi118.stdout: Cleanup : libmount-2.37.4-15.el9.x86_64 189/198 2024-03-24T00:02:33.482 INFO:teuthology.orchestra.run.smithi118.stdout: Cleanup : libblkid-2.37.4-15.el9.x86_64 190/198 2024-03-24T00:02:33.515 INFO:teuthology.orchestra.run.smithi118.stdout: Cleanup : NetworkManager-team-1:1.44.0-2.el9.x86_64 191/198 2024-03-24T00:02:33.551 INFO:teuthology.orchestra.run.smithi118.stdout: Running scriptlet: NetworkManager-1:1.44.0-2.el9.x86_64 192/198 2024-03-24T00:02:33.551 INFO:teuthology.orchestra.run.smithi118.stdout: Cleanup : NetworkManager-1:1.44.0-2.el9.x86_64 192/198 2024-03-24T00:02:33.899 INFO:teuthology.orchestra.run.smithi118.stdout: Running scriptlet: NetworkManager-1:1.44.0-2.el9.x86_64 192/198 2024-03-24T00:02:33.900 INFO:teuthology.orchestra.run.smithi118.stdout: Cleanup : NetworkManager-libnm-1:1.44.0-2.el9.x86_64 193/198 2024-03-24T00:02:34.105 INFO:teuthology.orchestra.run.smithi118.stdout: Running scriptlet: NetworkManager-libnm-1:1.44.0-2.el9.x86_64 193/198 2024-03-24T00:02:34.136 INFO:teuthology.orchestra.run.smithi118.stdout: Cleanup : libuuid-2.37.4-15.el9.x86_64 194/198 2024-03-24T00:02:34.197 INFO:teuthology.orchestra.run.smithi118.stdout: Cleanup : libsmartcols-2.37.4-15.el9.x86_64 195/198 2024-03-24T00:02:34.229 INFO:teuthology.orchestra.run.smithi118.stdout: Cleanup : libacl-2.3.1-3.el9.x86_64 196/198 2024-03-24T00:02:34.270 INFO:teuthology.orchestra.run.smithi118.stdout: Cleanup : openssl-libs-1:3.0.7-24.el9.x86_64 197/198 2024-03-24T00:02:34.389 INFO:teuthology.orchestra.run.smithi118.stdout: Cleanup : elfutils-libelf-0.189-3.el9.x86_64 198/198 2024-03-24T00:02:34.641 INFO:teuthology.orchestra.run.smithi118.stdout: Running scriptlet: elfutils-libelf-0.189-3.el9.x86_64 198/198 2024-03-24T00:02:34.641 INFO:teuthology.orchestra.run.smithi118.stdout:Couldn't write '0' to 'kernel/yama/ptrace_scope', ignoring: No such file or directory 2024-03-24T00:02:34.641 INFO:teuthology.orchestra.run.smithi118.stdout: 2024-03-24T00:02:40.007 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : ceph-2:19.0.0-2145.g3be9a5c9.el9.x86_64 1/198 2024-03-24T00:02:40.008 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : ceph-base-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2/198 2024-03-24T00:02:40.008 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : ceph-common-2:19.0.0-2145.g3be9a5c9.el9.x86_64 3/198 2024-03-24T00:02:40.008 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : ceph-fuse-2:19.0.0-2145.g3be9a5c9.el9.x86_64 4/198 2024-03-24T00:02:40.008 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : ceph-immutable-object-cache-2:19.0.0-2145.g3be9a 5/198 2024-03-24T00:02:40.008 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : ceph-mds-2:19.0.0-2145.g3be9a5c9.el9.x86_64 6/198 2024-03-24T00:02:40.008 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : ceph-mgr-2:19.0.0-2145.g3be9a5c9.el9.x86_64 7/198 2024-03-24T00:02:40.008 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : ceph-mon-2:19.0.0-2145.g3be9a5c9.el9.x86_64 8/198 2024-03-24T00:02:40.008 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : ceph-osd-2:19.0.0-2145.g3be9a5c9.el9.x86_64 9/198 2024-03-24T00:02:40.008 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : ceph-radosgw-2:19.0.0-2145.g3be9a5c9.el9.x86_64 10/198 2024-03-24T00:02:40.008 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : ceph-selinux-2:19.0.0-2145.g3be9a5c9.el9.x86_64 11/198 2024-03-24T00:02:40.008 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : ceph-test-2:19.0.0-2145.g3be9a5c9.el9.x86_64 12/198 2024-03-24T00:02:40.008 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : cephfs-mirror-2:19.0.0-2145.g3be9a5c9.el9.x86_64 13/198 2024-03-24T00:02:40.008 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : cephfs-shell-2:19.0.0-2145.g3be9a5c9.el9.x86_64 14/198 2024-03-24T00:02:40.009 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : libcephfs-devel-2:19.0.0-2145.g3be9a5c9.el9.x86_ 15/198 2024-03-24T00:02:40.009 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : libcephfs2-2:19.0.0-2145.g3be9a5c9.el9.x86_64 16/198 2024-03-24T00:02:40.009 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : libcephsqlite-2:19.0.0-2145.g3be9a5c9.el9.x86_64 17/198 2024-03-24T00:02:40.009 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : librados-devel-2:19.0.0-2145.g3be9a5c9.el9.x86_6 18/198 2024-03-24T00:02:40.009 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : libradosstriper1-2:19.0.0-2145.g3be9a5c9.el9.x86 19/198 2024-03-24T00:02:40.009 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : librgw2-2:19.0.0-2145.g3be9a5c9.el9.x86_64 20/198 2024-03-24T00:02:40.009 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : python3-ceph-argparse-2:19.0.0-2145.g3be9a5c9.el 21/198 2024-03-24T00:02:40.009 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : python3-ceph-common-2:19.0.0-2145.g3be9a5c9.el9. 22/198 2024-03-24T00:02:40.009 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : python3-cephfs-2:19.0.0-2145.g3be9a5c9.el9.x86_6 23/198 2024-03-24T00:02:40.009 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : python3-rados-2:19.0.0-2145.g3be9a5c9.el9.x86_64 24/198 2024-03-24T00:02:40.009 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : python3-rbd-2:19.0.0-2145.g3be9a5c9.el9.x86_64 25/198 2024-03-24T00:02:40.009 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : python3-rgw-2:19.0.0-2145.g3be9a5c9.el9.x86_64 26/198 2024-03-24T00:02:40.009 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : rbd-fuse-2:19.0.0-2145.g3be9a5c9.el9.x86_64 27/198 2024-03-24T00:02:40.009 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : rbd-mirror-2:19.0.0-2145.g3be9a5c9.el9.x86_64 28/198 2024-03-24T00:02:40.010 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : rbd-nbd-2:19.0.0-2145.g3be9a5c9.el9.x86_64 29/198 2024-03-24T00:02:40.010 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : ceph-grafana-dashboards-2:19.0.0-2145.g3be9a5c9. 30/198 2024-03-24T00:02:40.010 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : ceph-mgr-cephadm-2:19.0.0-2145.g3be9a5c9.el9.noa 31/198 2024-03-24T00:02:40.010 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : ceph-mgr-dashboard-2:19.0.0-2145.g3be9a5c9.el9.n 32/198 2024-03-24T00:02:40.010 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : ceph-mgr-diskprediction-local-2:19.0.0-2145.g3be 33/198 2024-03-24T00:02:40.010 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : ceph-mgr-modules-core-2:19.0.0-2145.g3be9a5c9.el 34/198 2024-03-24T00:02:40.010 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : ceph-mgr-rook-2:19.0.0-2145.g3be9a5c9.el9.noarch 35/198 2024-03-24T00:02:40.010 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : ceph-prometheus-alerts-2:19.0.0-2145.g3be9a5c9.e 36/198 2024-03-24T00:02:40.010 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : ceph-volume-2:19.0.0-2145.g3be9a5c9.el9.noarch 37/198 2024-03-24T00:02:40.010 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : cephadm-2:19.0.0-2145.g3be9a5c9.el9.noarch 38/198 2024-03-24T00:02:40.010 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : cephfs-top-2:19.0.0-2145.g3be9a5c9.el9.noarch 39/198 2024-03-24T00:02:40.010 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : automake-1.16.2-8.el9.noarch 40/198 2024-03-24T00:02:40.010 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : bison-3.7.4-5.el9.x86_64 41/198 2024-03-24T00:02:40.010 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : boost-program-options-1.75.0-8.el9.x86_64 42/198 2024-03-24T00:02:40.011 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : flex-2.6.4-9.el9.x86_64 43/198 2024-03-24T00:02:40.011 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 44/198 2024-03-24T00:02:40.011 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 45/198 2024-03-24T00:02:40.011 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 46/198 2024-03-24T00:02:40.011 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : libaio-devel-0.3.111-13.el9.x86_64 47/198 2024-03-24T00:02:40.011 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 48/198 2024-03-24T00:02:40.011 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 49/198 2024-03-24T00:02:40.011 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 50/198 2024-03-24T00:02:40.011 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : libtool-2.4.6-45.el9.x86_64 51/198 2024-03-24T00:02:40.011 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : libzstd-devel-1.5.1-2.el9.x86_64 52/198 2024-03-24T00:02:40.011 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 53/198 2024-03-24T00:02:40.011 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : lua-5.4.4-4.el9.x86_64 54/198 2024-03-24T00:02:40.011 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : oniguruma-6.9.6-1.el9.5.x86_64 55/198 2024-03-24T00:02:40.011 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : openblas-0.3.21-2.el9.x86_64 56/198 2024-03-24T00:02:40.012 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : openblas-openmp-0.3.21-2.el9.x86_64 57/198 2024-03-24T00:02:40.012 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : postgresql-13.11-1.el9.x86_64 58/198 2024-03-24T00:02:40.012 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : postgresql-contrib-13.11-1.el9.x86_64 59/198 2024-03-24T00:02:40.012 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : postgresql-private-libs-13.11-1.el9.x86_64 60/198 2024-03-24T00:02:40.012 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : postgresql-server-13.11-1.el9.x86_64 61/198 2024-03-24T00:02:40.012 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : protobuf-3.14.0-13.el9.x86_64 62/198 2024-03-24T00:02:40.012 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : python3-attrs-20.3.0-7.el9.noarch 63/198 2024-03-24T00:02:40.012 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 64/198 2024-03-24T00:02:40.012 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 65/198 2024-03-24T00:02:40.012 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 66/198 2024-03-24T00:02:40.012 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 67/198 2024-03-24T00:02:40.012 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 68/198 2024-03-24T00:02:40.012 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 69/198 2024-03-24T00:02:40.012 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 70/198 2024-03-24T00:02:40.013 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 71/198 2024-03-24T00:02:40.013 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : python3-packaging-20.9-5.el9.noarch 72/198 2024-03-24T00:02:40.013 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 73/198 2024-03-24T00:02:40.013 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : python3-protobuf-3.14.0-13.el9.noarch 74/198 2024-03-24T00:02:40.013 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 75/198 2024-03-24T00:02:40.013 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 76/198 2024-03-24T00:02:40.013 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 77/198 2024-03-24T00:02:40.013 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 78/198 2024-03-24T00:02:40.013 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 79/198 2024-03-24T00:02:40.013 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 80/198 2024-03-24T00:02:40.013 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : python3-wcwidth-0.2.5-8.el9.noarch 81/198 2024-03-24T00:02:40.013 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : socat-1.7.4.1-5.el9.x86_64 82/198 2024-03-24T00:02:40.013 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : uuid-1.6.2-55.el9.x86_64 83/198 2024-03-24T00:02:40.014 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 84/198 2024-03-24T00:02:40.014 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : zlib-devel-1.2.11-40.el9.x86_64 85/198 2024-03-24T00:02:40.014 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : fuse-2.9.9-15.el9.x86_64 86/198 2024-03-24T00:02:40.014 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 87/198 2024-03-24T00:02:40.014 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 88/198 2024-03-24T00:02:40.014 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : python3-ply-3.11-14.el9.noarch 89/198 2024-03-24T00:02:40.014 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 90/198 2024-03-24T00:02:40.014 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : python3-urllib3-1.26.5-5.el9.noarch 91/198 2024-03-24T00:02:40.014 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : jq-1.6-16.el9.x86_64 92/198 2024-03-24T00:02:40.014 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : libgfortran-11.4.1-3.el9.x86_64 93/198 2024-03-24T00:02:40.014 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : libquadmath-11.4.1-3.el9.x86_64 94/198 2024-03-24T00:02:40.014 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 95/198 2024-03-24T00:02:40.014 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : elfutils-libelf-devel-0.190-2.el9.x86_64 96/198 2024-03-24T00:02:40.015 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : libuuid-devel-2.37.4-18.el9.x86_64 97/198 2024-03-24T00:02:40.015 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : openssl-devel-1:3.0.7-27.el9.x86_64 98/198 2024-03-24T00:02:40.015 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : python3-devel-3.9.17-2.el9.x86_64 99/198 2024-03-24T00:02:40.015 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : python3-jinja2-2.11.3-5.el9.noarch 100/198 2024-03-24T00:02:40.015 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : qatlib-23.11.0-1.el9.x86_64 101/198 2024-03-24T00:02:40.015 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : qatlib-service-23.11.0-1.el9.x86_64 102/198 2024-03-24T00:02:40.015 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : qatzip-libs-1.1.2-2.el9.x86_64 103/198 2024-03-24T00:02:40.015 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : xfsprogs-devel-6.3.0-1.el9.x86_64 104/198 2024-03-24T00:02:40.015 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : lua-devel-5.4.4-4.el9.x86_64 105/198 2024-03-24T00:02:40.015 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : protobuf-compiler-3.14.0-13.el9.x86_64 106/198 2024-03-24T00:02:40.015 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : abseil-cpp-20211102.0-3.el9.x86_64 107/198 2024-03-24T00:02:40.015 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : gperftools-libs-2.9.1-2.el9.x86_64 108/198 2024-03-24T00:02:40.015 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : grpc-data-1.46.7-10.el9.noarch 109/198 2024-03-24T00:02:40.016 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : libarrow-9.0.0-10.el9.x86_64 110/198 2024-03-24T00:02:40.016 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : libarrow-doc-9.0.0-10.el9.noarch 111/198 2024-03-24T00:02:40.016 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 112/198 2024-03-24T00:02:40.016 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 113/198 2024-03-24T00:02:40.016 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : luarocks-3.9.2-1.el9.noarch 114/198 2024-03-24T00:02:40.016 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : parquet-libs-9.0.0-10.el9.x86_64 115/198 2024-03-24T00:02:40.016 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : python3-asyncssh-2.13.2-5.el9.noarch 116/198 2024-03-24T00:02:40.016 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 117/198 2024-03-24T00:02:40.016 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 118/198 2024-03-24T00:02:40.016 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : python3-certifi-2021.10.8-2.el9.noarch 119/198 2024-03-24T00:02:40.016 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 120/198 2024-03-24T00:02:40.016 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 121/198 2024-03-24T00:02:40.016 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : python3-cmd2-2.4.2-2.el9.noarch 122/198 2024-03-24T00:02:40.016 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : python3-colorama-0.4.4-12.el9.noarch 123/198 2024-03-24T00:02:40.017 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : python3-google-auth-1:2.28.2-1.el9.noarch 124/198 2024-03-24T00:02:40.017 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : python3-grpcio-1.46.7-10.el9.x86_64 125/198 2024-03-24T00:02:40.017 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : python3-grpcio-tools-1.46.7-10.el9.x86_64 126/198 2024-03-24T00:02:40.017 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 127/198 2024-03-24T00:02:40.017 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 128/198 2024-03-24T00:02:40.017 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 129/198 2024-03-24T00:02:40.017 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 130/198 2024-03-24T00:02:40.017 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 131/198 2024-03-24T00:02:40.017 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 132/198 2024-03-24T00:02:40.017 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 133/198 2024-03-24T00:02:40.017 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 134/198 2024-03-24T00:02:40.017 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 135/198 2024-03-24T00:02:40.017 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 136/198 2024-03-24T00:02:40.017 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 137/198 2024-03-24T00:02:40.018 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 138/198 2024-03-24T00:02:40.018 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : python3-pyperclip-1.8.2-2.el9.noarch 139/198 2024-03-24T00:02:40.018 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 140/198 2024-03-24T00:02:40.018 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 141/198 2024-03-24T00:02:40.018 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 142/198 2024-03-24T00:02:40.018 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 143/198 2024-03-24T00:02:40.018 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : python3-typing-extensions-4.4.0-2.el9.noarch 144/198 2024-03-24T00:02:40.018 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : python3-webob-1.8.7-6.el9.noarch 145/198 2024-03-24T00:02:40.018 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 146/198 2024-03-24T00:02:40.018 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 147/198 2024-03-24T00:02:40.018 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 148/198 2024-03-24T00:02:40.018 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : re2-1:20211101-3.el9.x86_64 149/198 2024-03-24T00:02:40.019 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 150/198 2024-03-24T00:02:40.019 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : librados2-2:19.0.0-2145.g3be9a5c9.el9.x86_64 151/198 2024-03-24T00:02:40.019 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 152/198 2024-03-24T00:02:40.019 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : librbd1-2:19.0.0-2145.g3be9a5c9.el9.x86_64 153/198 2024-03-24T00:02:40.019 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 154/198 2024-03-24T00:02:40.019 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : xfsdump-3.1.12-4.el9.x86_64 155/198 2024-03-24T00:02:40.019 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : xfsdump-3.1.12-3.el9.x86_64 156/198 2024-03-24T00:02:40.019 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : NetworkManager-1:1.47.2-1.el9.x86_64 157/198 2024-03-24T00:02:40.019 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : NetworkManager-1:1.44.0-2.el9.x86_64 158/198 2024-03-24T00:02:40.019 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : NetworkManager-libnm-1:1.47.2-1.el9.x86_64 159/198 2024-03-24T00:02:40.019 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : NetworkManager-libnm-1:1.44.0-2.el9.x86_64 160/198 2024-03-24T00:02:40.019 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : NetworkManager-team-1:1.47.2-1.el9.x86_64 161/198 2024-03-24T00:02:40.019 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : NetworkManager-team-1:1.44.0-2.el9.x86_64 162/198 2024-03-24T00:02:40.019 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : NetworkManager-tui-1:1.47.2-1.el9.x86_64 163/198 2024-03-24T00:02:40.020 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : NetworkManager-tui-1:1.44.0-2.el9.x86_64 164/198 2024-03-24T00:02:40.020 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : acl-2.3.1-4.el9.x86_64 165/198 2024-03-24T00:02:40.020 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : acl-2.3.1-3.el9.x86_64 166/198 2024-03-24T00:02:40.020 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : elfutils-debuginfod-client-0.190-2.el9.x86_64 167/198 2024-03-24T00:02:40.020 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : elfutils-debuginfod-client-0.189-3.el9.x86_64 168/198 2024-03-24T00:02:40.020 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : elfutils-libelf-0.190-2.el9.x86_64 169/198 2024-03-24T00:02:40.020 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : elfutils-libelf-0.189-3.el9.x86_64 170/198 2024-03-24T00:02:40.020 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : elfutils-libs-0.190-2.el9.x86_64 171/198 2024-03-24T00:02:40.020 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : elfutils-libs-0.189-3.el9.x86_64 172/198 2024-03-24T00:02:40.020 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : libacl-2.3.1-4.el9.x86_64 173/198 2024-03-24T00:02:40.020 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : libacl-2.3.1-3.el9.x86_64 174/198 2024-03-24T00:02:40.020 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : libblkid-2.37.4-18.el9.x86_64 175/198 2024-03-24T00:02:40.020 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : libblkid-2.37.4-15.el9.x86_64 176/198 2024-03-24T00:02:40.021 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : libfdisk-2.37.4-18.el9.x86_64 177/198 2024-03-24T00:02:40.021 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : libfdisk-2.37.4-15.el9.x86_64 178/198 2024-03-24T00:02:40.021 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : libmount-2.37.4-18.el9.x86_64 179/198 2024-03-24T00:02:40.021 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : libmount-2.37.4-15.el9.x86_64 180/198 2024-03-24T00:02:40.021 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : libsmartcols-2.37.4-18.el9.x86_64 181/198 2024-03-24T00:02:40.021 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : libsmartcols-2.37.4-15.el9.x86_64 182/198 2024-03-24T00:02:40.021 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : libuuid-2.37.4-18.el9.x86_64 183/198 2024-03-24T00:02:40.021 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : libuuid-2.37.4-15.el9.x86_64 184/198 2024-03-24T00:02:40.021 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : openssl-1:3.0.7-27.el9.x86_64 185/198 2024-03-24T00:02:40.021 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : openssl-1:3.0.7-24.el9.x86_64 186/198 2024-03-24T00:02:40.021 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : openssl-libs-1:3.0.7-27.el9.x86_64 187/198 2024-03-24T00:02:40.021 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : openssl-libs-1:3.0.7-24.el9.x86_64 188/198 2024-03-24T00:02:40.021 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : util-linux-2.37.4-18.el9.x86_64 189/198 2024-03-24T00:02:40.022 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : util-linux-2.37.4-15.el9.x86_64 190/198 2024-03-24T00:02:40.022 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : util-linux-core-2.37.4-18.el9.x86_64 191/198 2024-03-24T00:02:40.022 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : util-linux-core-2.37.4-15.el9.x86_64 192/198 2024-03-24T00:02:40.022 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : util-linux-user-2.37.4-18.el9.x86_64 193/198 2024-03-24T00:02:40.022 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : util-linux-user-2.37.4-15.el9.x86_64 194/198 2024-03-24T00:02:40.022 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : xfsprogs-6.3.0-1.el9.x86_64 195/198 2024-03-24T00:02:40.022 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : xfsprogs-5.19.0-4.el9.x86_64 196/198 2024-03-24T00:02:40.022 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : libacl-devel-2.3.1-4.el9.x86_64 197/198 2024-03-24T00:02:42.529 INFO:teuthology.orchestra.run.smithi118.stdout: Verifying : libacl-devel-2.3.1-3.el9.x86_64 198/198 2024-03-24T00:02:42.530 INFO:teuthology.orchestra.run.smithi118.stdout: 2024-03-24T00:02:42.530 INFO:teuthology.orchestra.run.smithi118.stdout:Upgraded: 2024-03-24T00:02:42.530 INFO:teuthology.orchestra.run.smithi118.stdout: NetworkManager-1:1.47.2-1.el9.x86_64 2024-03-24T00:02:42.530 INFO:teuthology.orchestra.run.smithi118.stdout: NetworkManager-libnm-1:1.47.2-1.el9.x86_64 2024-03-24T00:02:42.530 INFO:teuthology.orchestra.run.smithi118.stdout: NetworkManager-team-1:1.47.2-1.el9.x86_64 2024-03-24T00:02:42.530 INFO:teuthology.orchestra.run.smithi118.stdout: NetworkManager-tui-1:1.47.2-1.el9.x86_64 2024-03-24T00:02:42.530 INFO:teuthology.orchestra.run.smithi118.stdout: acl-2.3.1-4.el9.x86_64 2024-03-24T00:02:42.530 INFO:teuthology.orchestra.run.smithi118.stdout: elfutils-debuginfod-client-0.190-2.el9.x86_64 2024-03-24T00:02:42.530 INFO:teuthology.orchestra.run.smithi118.stdout: elfutils-libelf-0.190-2.el9.x86_64 2024-03-24T00:02:42.530 INFO:teuthology.orchestra.run.smithi118.stdout: elfutils-libs-0.190-2.el9.x86_64 2024-03-24T00:02:42.531 INFO:teuthology.orchestra.run.smithi118.stdout: libacl-2.3.1-4.el9.x86_64 2024-03-24T00:02:42.531 INFO:teuthology.orchestra.run.smithi118.stdout: libacl-devel-2.3.1-4.el9.x86_64 2024-03-24T00:02:42.531 INFO:teuthology.orchestra.run.smithi118.stdout: libblkid-2.37.4-18.el9.x86_64 2024-03-24T00:02:42.531 INFO:teuthology.orchestra.run.smithi118.stdout: libfdisk-2.37.4-18.el9.x86_64 2024-03-24T00:02:42.531 INFO:teuthology.orchestra.run.smithi118.stdout: libmount-2.37.4-18.el9.x86_64 2024-03-24T00:02:42.531 INFO:teuthology.orchestra.run.smithi118.stdout: librados2-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-24T00:02:42.531 INFO:teuthology.orchestra.run.smithi118.stdout: librbd1-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-24T00:02:42.531 INFO:teuthology.orchestra.run.smithi118.stdout: libsmartcols-2.37.4-18.el9.x86_64 2024-03-24T00:02:42.531 INFO:teuthology.orchestra.run.smithi118.stdout: libuuid-2.37.4-18.el9.x86_64 2024-03-24T00:02:42.531 INFO:teuthology.orchestra.run.smithi118.stdout: openssl-1:3.0.7-27.el9.x86_64 2024-03-24T00:02:42.531 INFO:teuthology.orchestra.run.smithi118.stdout: openssl-libs-1:3.0.7-27.el9.x86_64 2024-03-24T00:02:42.531 INFO:teuthology.orchestra.run.smithi118.stdout: util-linux-2.37.4-18.el9.x86_64 2024-03-24T00:02:42.532 INFO:teuthology.orchestra.run.smithi118.stdout: util-linux-core-2.37.4-18.el9.x86_64 2024-03-24T00:02:42.532 INFO:teuthology.orchestra.run.smithi118.stdout: util-linux-user-2.37.4-18.el9.x86_64 2024-03-24T00:02:42.532 INFO:teuthology.orchestra.run.smithi118.stdout: xfsdump-3.1.12-4.el9.x86_64 2024-03-24T00:02:42.532 INFO:teuthology.orchestra.run.smithi118.stdout: xfsprogs-6.3.0-1.el9.x86_64 2024-03-24T00:02:42.532 INFO:teuthology.orchestra.run.smithi118.stdout:Installed: 2024-03-24T00:02:42.532 INFO:teuthology.orchestra.run.smithi118.stdout: abseil-cpp-20211102.0-3.el9.x86_64 2024-03-24T00:02:42.532 INFO:teuthology.orchestra.run.smithi118.stdout: automake-1.16.2-8.el9.noarch 2024-03-24T00:02:42.532 INFO:teuthology.orchestra.run.smithi118.stdout: bison-3.7.4-5.el9.x86_64 2024-03-24T00:02:42.532 INFO:teuthology.orchestra.run.smithi118.stdout: boost-program-options-1.75.0-8.el9.x86_64 2024-03-24T00:02:42.532 INFO:teuthology.orchestra.run.smithi118.stdout: ceph-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-24T00:02:42.532 INFO:teuthology.orchestra.run.smithi118.stdout: ceph-base-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-24T00:02:42.532 INFO:teuthology.orchestra.run.smithi118.stdout: ceph-common-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-24T00:02:42.533 INFO:teuthology.orchestra.run.smithi118.stdout: ceph-fuse-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-24T00:02:42.533 INFO:teuthology.orchestra.run.smithi118.stdout: ceph-grafana-dashboards-2:19.0.0-2145.g3be9a5c9.el9.noarch 2024-03-24T00:02:42.533 INFO:teuthology.orchestra.run.smithi118.stdout: ceph-immutable-object-cache-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-24T00:02:42.533 INFO:teuthology.orchestra.run.smithi118.stdout: ceph-mds-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-24T00:02:42.533 INFO:teuthology.orchestra.run.smithi118.stdout: ceph-mgr-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-24T00:02:42.533 INFO:teuthology.orchestra.run.smithi118.stdout: ceph-mgr-cephadm-2:19.0.0-2145.g3be9a5c9.el9.noarch 2024-03-24T00:02:42.533 INFO:teuthology.orchestra.run.smithi118.stdout: ceph-mgr-dashboard-2:19.0.0-2145.g3be9a5c9.el9.noarch 2024-03-24T00:02:42.533 INFO:teuthology.orchestra.run.smithi118.stdout: ceph-mgr-diskprediction-local-2:19.0.0-2145.g3be9a5c9.el9.noarch 2024-03-24T00:02:42.533 INFO:teuthology.orchestra.run.smithi118.stdout: ceph-mgr-modules-core-2:19.0.0-2145.g3be9a5c9.el9.noarch 2024-03-24T00:02:42.533 INFO:teuthology.orchestra.run.smithi118.stdout: ceph-mgr-rook-2:19.0.0-2145.g3be9a5c9.el9.noarch 2024-03-24T00:02:42.533 INFO:teuthology.orchestra.run.smithi118.stdout: ceph-mon-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-24T00:02:42.533 INFO:teuthology.orchestra.run.smithi118.stdout: ceph-osd-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-24T00:02:42.534 INFO:teuthology.orchestra.run.smithi118.stdout: ceph-prometheus-alerts-2:19.0.0-2145.g3be9a5c9.el9.noarch 2024-03-24T00:02:42.534 INFO:teuthology.orchestra.run.smithi118.stdout: ceph-radosgw-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-24T00:02:42.534 INFO:teuthology.orchestra.run.smithi118.stdout: ceph-selinux-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-24T00:02:42.534 INFO:teuthology.orchestra.run.smithi118.stdout: ceph-test-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-24T00:02:42.534 INFO:teuthology.orchestra.run.smithi118.stdout: ceph-volume-2:19.0.0-2145.g3be9a5c9.el9.noarch 2024-03-24T00:02:42.534 INFO:teuthology.orchestra.run.smithi118.stdout: cephadm-2:19.0.0-2145.g3be9a5c9.el9.noarch 2024-03-24T00:02:42.534 INFO:teuthology.orchestra.run.smithi118.stdout: cephfs-mirror-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-24T00:02:42.534 INFO:teuthology.orchestra.run.smithi118.stdout: cephfs-shell-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-24T00:02:42.534 INFO:teuthology.orchestra.run.smithi118.stdout: cephfs-top-2:19.0.0-2145.g3be9a5c9.el9.noarch 2024-03-24T00:02:42.534 INFO:teuthology.orchestra.run.smithi118.stdout: elfutils-libelf-devel-0.190-2.el9.x86_64 2024-03-24T00:02:42.534 INFO:teuthology.orchestra.run.smithi118.stdout: flex-2.6.4-9.el9.x86_64 2024-03-24T00:02:42.535 INFO:teuthology.orchestra.run.smithi118.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-03-24T00:02:42.535 INFO:teuthology.orchestra.run.smithi118.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-03-24T00:02:42.535 INFO:teuthology.orchestra.run.smithi118.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-03-24T00:02:42.535 INFO:teuthology.orchestra.run.smithi118.stdout: fuse-2.9.9-15.el9.x86_64 2024-03-24T00:02:42.535 INFO:teuthology.orchestra.run.smithi118.stdout: gperftools-libs-2.9.1-2.el9.x86_64 2024-03-24T00:02:42.535 INFO:teuthology.orchestra.run.smithi118.stdout: grpc-data-1.46.7-10.el9.noarch 2024-03-24T00:02:42.535 INFO:teuthology.orchestra.run.smithi118.stdout: jq-1.6-16.el9.x86_64 2024-03-24T00:02:42.535 INFO:teuthology.orchestra.run.smithi118.stdout: libaio-devel-0.3.111-13.el9.x86_64 2024-03-24T00:02:42.535 INFO:teuthology.orchestra.run.smithi118.stdout: libarrow-9.0.0-10.el9.x86_64 2024-03-24T00:02:42.535 INFO:teuthology.orchestra.run.smithi118.stdout: libarrow-doc-9.0.0-10.el9.noarch 2024-03-24T00:02:42.535 INFO:teuthology.orchestra.run.smithi118.stdout: libcephfs-devel-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-24T00:02:42.535 INFO:teuthology.orchestra.run.smithi118.stdout: libcephfs2-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-24T00:02:42.536 INFO:teuthology.orchestra.run.smithi118.stdout: libcephsqlite-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-24T00:02:42.536 INFO:teuthology.orchestra.run.smithi118.stdout: libgfortran-11.4.1-3.el9.x86_64 2024-03-24T00:02:42.536 INFO:teuthology.orchestra.run.smithi118.stdout: liboath-2.6.7-2.el9.x86_64 2024-03-24T00:02:42.536 INFO:teuthology.orchestra.run.smithi118.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-03-24T00:02:42.536 INFO:teuthology.orchestra.run.smithi118.stdout: libquadmath-11.4.1-3.el9.x86_64 2024-03-24T00:02:42.536 INFO:teuthology.orchestra.run.smithi118.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-03-24T00:02:42.536 INFO:teuthology.orchestra.run.smithi118.stdout: librados-devel-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-24T00:02:42.536 INFO:teuthology.orchestra.run.smithi118.stdout: libradosstriper1-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-24T00:02:42.536 INFO:teuthology.orchestra.run.smithi118.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-03-24T00:02:42.536 INFO:teuthology.orchestra.run.smithi118.stdout: librgw2-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-24T00:02:42.536 INFO:teuthology.orchestra.run.smithi118.stdout: libtool-2.4.6-45.el9.x86_64 2024-03-24T00:02:42.536 INFO:teuthology.orchestra.run.smithi118.stdout: libunwind-1.6.2-1.el9.x86_64 2024-03-24T00:02:42.536 INFO:teuthology.orchestra.run.smithi118.stdout: libuuid-devel-2.37.4-18.el9.x86_64 2024-03-24T00:02:42.537 INFO:teuthology.orchestra.run.smithi118.stdout: libzstd-devel-1.5.1-2.el9.x86_64 2024-03-24T00:02:42.537 INFO:teuthology.orchestra.run.smithi118.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-03-24T00:02:42.537 INFO:teuthology.orchestra.run.smithi118.stdout: lua-5.4.4-4.el9.x86_64 2024-03-24T00:02:42.537 INFO:teuthology.orchestra.run.smithi118.stdout: lua-devel-5.4.4-4.el9.x86_64 2024-03-24T00:02:42.537 INFO:teuthology.orchestra.run.smithi118.stdout: luarocks-3.9.2-1.el9.noarch 2024-03-24T00:02:42.537 INFO:teuthology.orchestra.run.smithi118.stdout: oniguruma-6.9.6-1.el9.5.x86_64 2024-03-24T00:02:42.537 INFO:teuthology.orchestra.run.smithi118.stdout: openblas-0.3.21-2.el9.x86_64 2024-03-24T00:02:42.537 INFO:teuthology.orchestra.run.smithi118.stdout: openblas-openmp-0.3.21-2.el9.x86_64 2024-03-24T00:02:42.537 INFO:teuthology.orchestra.run.smithi118.stdout: openssl-devel-1:3.0.7-27.el9.x86_64 2024-03-24T00:02:42.537 INFO:teuthology.orchestra.run.smithi118.stdout: parquet-libs-9.0.0-10.el9.x86_64 2024-03-24T00:02:42.537 INFO:teuthology.orchestra.run.smithi118.stdout: postgresql-13.11-1.el9.x86_64 2024-03-24T00:02:42.537 INFO:teuthology.orchestra.run.smithi118.stdout: postgresql-contrib-13.11-1.el9.x86_64 2024-03-24T00:02:42.538 INFO:teuthology.orchestra.run.smithi118.stdout: postgresql-private-libs-13.11-1.el9.x86_64 2024-03-24T00:02:42.538 INFO:teuthology.orchestra.run.smithi118.stdout: postgresql-server-13.11-1.el9.x86_64 2024-03-24T00:02:42.538 INFO:teuthology.orchestra.run.smithi118.stdout: protobuf-3.14.0-13.el9.x86_64 2024-03-24T00:02:42.538 INFO:teuthology.orchestra.run.smithi118.stdout: protobuf-compiler-3.14.0-13.el9.x86_64 2024-03-24T00:02:42.538 INFO:teuthology.orchestra.run.smithi118.stdout: python3-asyncssh-2.13.2-5.el9.noarch 2024-03-24T00:02:42.538 INFO:teuthology.orchestra.run.smithi118.stdout: python3-attrs-20.3.0-7.el9.noarch 2024-03-24T00:02:42.538 INFO:teuthology.orchestra.run.smithi118.stdout: python3-babel-2.9.1-2.el9.noarch 2024-03-24T00:02:42.538 INFO:teuthology.orchestra.run.smithi118.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-03-24T00:02:42.538 INFO:teuthology.orchestra.run.smithi118.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-03-24T00:02:42.538 INFO:teuthology.orchestra.run.smithi118.stdout: python3-ceph-argparse-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-24T00:02:42.538 INFO:teuthology.orchestra.run.smithi118.stdout: python3-ceph-common-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-24T00:02:42.538 INFO:teuthology.orchestra.run.smithi118.stdout: python3-cephfs-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-24T00:02:42.539 INFO:teuthology.orchestra.run.smithi118.stdout: python3-certifi-2021.10.8-2.el9.noarch 2024-03-24T00:02:42.539 INFO:teuthology.orchestra.run.smithi118.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-03-24T00:02:42.539 INFO:teuthology.orchestra.run.smithi118.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-03-24T00:02:42.539 INFO:teuthology.orchestra.run.smithi118.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-03-24T00:02:42.539 INFO:teuthology.orchestra.run.smithi118.stdout: python3-cmd2-2.4.2-2.el9.noarch 2024-03-24T00:02:42.539 INFO:teuthology.orchestra.run.smithi118.stdout: python3-colorama-0.4.4-12.el9.noarch 2024-03-24T00:02:42.539 INFO:teuthology.orchestra.run.smithi118.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-03-24T00:02:42.539 INFO:teuthology.orchestra.run.smithi118.stdout: python3-devel-3.9.17-2.el9.x86_64 2024-03-24T00:02:42.539 INFO:teuthology.orchestra.run.smithi118.stdout: python3-google-auth-1:2.28.2-1.el9.noarch 2024-03-24T00:02:42.539 INFO:teuthology.orchestra.run.smithi118.stdout: python3-grpcio-1.46.7-10.el9.x86_64 2024-03-24T00:02:42.539 INFO:teuthology.orchestra.run.smithi118.stdout: python3-grpcio-tools-1.46.7-10.el9.x86_64 2024-03-24T00:02:42.539 INFO:teuthology.orchestra.run.smithi118.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-03-24T00:02:42.540 INFO:teuthology.orchestra.run.smithi118.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-03-24T00:02:42.540 INFO:teuthology.orchestra.run.smithi118.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-03-24T00:02:42.540 INFO:teuthology.orchestra.run.smithi118.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-03-24T00:02:42.540 INFO:teuthology.orchestra.run.smithi118.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-03-24T00:02:42.540 INFO:teuthology.orchestra.run.smithi118.stdout: python3-jinja2-2.11.3-5.el9.noarch 2024-03-24T00:02:42.540 INFO:teuthology.orchestra.run.smithi118.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-03-24T00:02:42.540 INFO:teuthology.orchestra.run.smithi118.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-03-24T00:02:42.540 INFO:teuthology.orchestra.run.smithi118.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-03-24T00:02:42.540 INFO:teuthology.orchestra.run.smithi118.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-03-24T00:02:42.540 INFO:teuthology.orchestra.run.smithi118.stdout: python3-mako-1.1.4-6.el9.noarch 2024-03-24T00:02:42.540 INFO:teuthology.orchestra.run.smithi118.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-03-24T00:02:42.540 INFO:teuthology.orchestra.run.smithi118.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-03-24T00:02:42.540 INFO:teuthology.orchestra.run.smithi118.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-03-24T00:02:42.541 INFO:teuthology.orchestra.run.smithi118.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-03-24T00:02:42.541 INFO:teuthology.orchestra.run.smithi118.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-03-24T00:02:42.541 INFO:teuthology.orchestra.run.smithi118.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-03-24T00:02:42.541 INFO:teuthology.orchestra.run.smithi118.stdout: python3-packaging-20.9-5.el9.noarch 2024-03-24T00:02:42.541 INFO:teuthology.orchestra.run.smithi118.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-03-24T00:02:42.541 INFO:teuthology.orchestra.run.smithi118.stdout: python3-ply-3.11-14.el9.noarch 2024-03-24T00:02:42.541 INFO:teuthology.orchestra.run.smithi118.stdout: python3-portend-3.1.0-2.el9.noarch 2024-03-24T00:02:42.541 INFO:teuthology.orchestra.run.smithi118.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-03-24T00:02:42.541 INFO:teuthology.orchestra.run.smithi118.stdout: python3-protobuf-3.14.0-13.el9.noarch 2024-03-24T00:02:42.541 INFO:teuthology.orchestra.run.smithi118.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-03-24T00:02:42.541 INFO:teuthology.orchestra.run.smithi118.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-03-24T00:02:42.541 INFO:teuthology.orchestra.run.smithi118.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-03-24T00:02:42.542 INFO:teuthology.orchestra.run.smithi118.stdout: python3-pycparser-2.20-6.el9.noarch 2024-03-24T00:02:42.542 INFO:teuthology.orchestra.run.smithi118.stdout: python3-pyperclip-1.8.2-2.el9.noarch 2024-03-24T00:02:42.542 INFO:teuthology.orchestra.run.smithi118.stdout: python3-pytz-2021.1-5.el9.noarch 2024-03-24T00:02:42.542 INFO:teuthology.orchestra.run.smithi118.stdout: python3-rados-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-24T00:02:42.542 INFO:teuthology.orchestra.run.smithi118.stdout: python3-rbd-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-24T00:02:42.542 INFO:teuthology.orchestra.run.smithi118.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-03-24T00:02:42.542 INFO:teuthology.orchestra.run.smithi118.stdout: python3-requests-2.25.1-8.el9.noarch 2024-03-24T00:02:42.542 INFO:teuthology.orchestra.run.smithi118.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-03-24T00:02:42.542 INFO:teuthology.orchestra.run.smithi118.stdout: python3-rgw-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-24T00:02:42.542 INFO:teuthology.orchestra.run.smithi118.stdout: python3-routes-2.5.1-5.el9.noarch 2024-03-24T00:02:42.542 INFO:teuthology.orchestra.run.smithi118.stdout: python3-rsa-4.9-2.el9.noarch 2024-03-24T00:02:42.543 INFO:teuthology.orchestra.run.smithi118.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-03-24T00:02:42.543 INFO:teuthology.orchestra.run.smithi118.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-03-24T00:02:42.543 INFO:teuthology.orchestra.run.smithi118.stdout: python3-toml-0.10.2-6.el9.noarch 2024-03-24T00:02:42.543 INFO:teuthology.orchestra.run.smithi118.stdout: python3-typing-extensions-4.4.0-2.el9.noarch 2024-03-24T00:02:42.543 INFO:teuthology.orchestra.run.smithi118.stdout: python3-urllib3-1.26.5-5.el9.noarch 2024-03-24T00:02:42.543 INFO:teuthology.orchestra.run.smithi118.stdout: python3-wcwidth-0.2.5-8.el9.noarch 2024-03-24T00:02:42.543 INFO:teuthology.orchestra.run.smithi118.stdout: python3-webob-1.8.7-6.el9.noarch 2024-03-24T00:02:42.543 INFO:teuthology.orchestra.run.smithi118.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-03-24T00:02:42.543 INFO:teuthology.orchestra.run.smithi118.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-03-24T00:02:42.543 INFO:teuthology.orchestra.run.smithi118.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-03-24T00:02:42.543 INFO:teuthology.orchestra.run.smithi118.stdout: qatlib-23.11.0-1.el9.x86_64 2024-03-24T00:02:42.543 INFO:teuthology.orchestra.run.smithi118.stdout: qatlib-service-23.11.0-1.el9.x86_64 2024-03-24T00:02:42.543 INFO:teuthology.orchestra.run.smithi118.stdout: qatzip-libs-1.1.2-2.el9.x86_64 2024-03-24T00:02:42.544 INFO:teuthology.orchestra.run.smithi118.stdout: rbd-fuse-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-24T00:02:42.544 INFO:teuthology.orchestra.run.smithi118.stdout: rbd-mirror-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-24T00:02:42.544 INFO:teuthology.orchestra.run.smithi118.stdout: rbd-nbd-2:19.0.0-2145.g3be9a5c9.el9.x86_64 2024-03-24T00:02:42.544 INFO:teuthology.orchestra.run.smithi118.stdout: re2-1:20211101-3.el9.x86_64 2024-03-24T00:02:42.544 INFO:teuthology.orchestra.run.smithi118.stdout: socat-1.7.4.1-5.el9.x86_64 2024-03-24T00:02:42.544 INFO:teuthology.orchestra.run.smithi118.stdout: thrift-0.15.0-2.el9.x86_64 2024-03-24T00:02:42.544 INFO:teuthology.orchestra.run.smithi118.stdout: uuid-1.6.2-55.el9.x86_64 2024-03-24T00:02:42.544 INFO:teuthology.orchestra.run.smithi118.stdout: xfsprogs-devel-6.3.0-1.el9.x86_64 2024-03-24T00:02:42.544 INFO:teuthology.orchestra.run.smithi118.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-03-24T00:02:42.545 INFO:teuthology.orchestra.run.smithi118.stdout: zlib-devel-1.2.11-40.el9.x86_64 2024-03-24T00:02:42.545 INFO:teuthology.orchestra.run.smithi118.stdout: 2024-03-24T00:02:42.545 INFO:teuthology.orchestra.run.smithi118.stdout:Complete! 2024-03-24T00:02:43.192 DEBUG:teuthology.parallel:result is None 2024-03-24T00:02:43.193 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=3be9a5c9bc793e11e2800a8c0c696e8b46742033 2024-03-24T00:02:43.471 DEBUG:teuthology.orchestra.run.smithi118:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-03-24T00:02:43.541 INFO:teuthology.orchestra.run.smithi118.stdout:19.0.0-2145.g3be9a5c9.el9 2024-03-24T00:02:43.542 INFO:teuthology.packaging:The installed version of ceph is 19.0.0-2145.g3be9a5c9.el9 2024-03-24T00:02:43.542 INFO:teuthology.task.install:The correct ceph version 19.0.0-2145.g3be9a5c9 is installed. 2024-03-24T00:02:43.544 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=3be9a5c9bc793e11e2800a8c0c696e8b46742033 2024-03-24T00:02:43.684 DEBUG:teuthology.orchestra.run.smithi189:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-03-24T00:02:43.709 INFO:teuthology.orchestra.run.smithi189.stdout:19.0.0-2145.g3be9a5c9.el9 2024-03-24T00:02:43.710 INFO:teuthology.packaging:The installed version of ceph is 19.0.0-2145.g3be9a5c9.el9 2024-03-24T00:02:43.710 INFO:teuthology.task.install:The correct ceph version 19.0.0-2145.g3be9a5c9 is installed. 2024-03-24T00:02:43.711 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=3be9a5c9bc793e11e2800a8c0c696e8b46742033 2024-03-24T00:02:43.859 DEBUG:teuthology.orchestra.run.smithi193:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-03-24T00:02:43.885 INFO:teuthology.orchestra.run.smithi193.stdout:19.0.0-2145.g3be9a5c9.el9 2024-03-24T00:02:43.885 INFO:teuthology.packaging:The installed version of ceph is 19.0.0-2145.g3be9a5c9.el9 2024-03-24T00:02:43.885 INFO:teuthology.task.install:The correct ceph version 19.0.0-2145.g3be9a5c9 is installed. 2024-03-24T00:02:43.887 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2024-03-24T00:02:43.887 DEBUG:teuthology.orchestra.run.smithi118:> set -ex 2024-03-24T00:02:43.888 DEBUG:teuthology.orchestra.run.smithi118:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-03-24T00:02:43.974 DEBUG:teuthology.orchestra.run.smithi189:> set -ex 2024-03-24T00:02:43.974 DEBUG:teuthology.orchestra.run.smithi189:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-03-24T00:02:44.009 DEBUG:teuthology.orchestra.run.smithi193:> set -ex 2024-03-24T00:02:44.010 DEBUG:teuthology.orchestra.run.smithi193:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-03-24T00:02:44.045 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2024-03-24T00:02:44.045 DEBUG:teuthology.orchestra.run.smithi118:> set -ex 2024-03-24T00:02:44.045 DEBUG:teuthology.orchestra.run.smithi118:> sudo dd of=/usr/bin/daemon-helper 2024-03-24T00:02:44.119 DEBUG:teuthology.orchestra.run.smithi118:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-03-24T00:02:44.233 DEBUG:teuthology.orchestra.run.smithi189:> set -ex 2024-03-24T00:02:44.233 DEBUG:teuthology.orchestra.run.smithi189:> sudo dd of=/usr/bin/daemon-helper 2024-03-24T00:02:44.265 DEBUG:teuthology.orchestra.run.smithi189:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-03-24T00:02:44.335 DEBUG:teuthology.orchestra.run.smithi193:> set -ex 2024-03-24T00:02:44.335 DEBUG:teuthology.orchestra.run.smithi193:> sudo dd of=/usr/bin/daemon-helper 2024-03-24T00:02:44.368 DEBUG:teuthology.orchestra.run.smithi193:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-03-24T00:02:44.436 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2024-03-24T00:02:44.436 DEBUG:teuthology.orchestra.run.smithi118:> set -ex 2024-03-24T00:02:44.437 DEBUG:teuthology.orchestra.run.smithi118:> sudo dd of=/usr/bin/adjust-ulimits 2024-03-24T00:02:44.508 DEBUG:teuthology.orchestra.run.smithi118:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-03-24T00:02:44.617 DEBUG:teuthology.orchestra.run.smithi189:> set -ex 2024-03-24T00:02:44.617 DEBUG:teuthology.orchestra.run.smithi189:> sudo dd of=/usr/bin/adjust-ulimits 2024-03-24T00:02:44.644 DEBUG:teuthology.orchestra.run.smithi189:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-03-24T00:02:44.710 DEBUG:teuthology.orchestra.run.smithi193:> set -ex 2024-03-24T00:02:44.710 DEBUG:teuthology.orchestra.run.smithi193:> sudo dd of=/usr/bin/adjust-ulimits 2024-03-24T00:02:44.738 DEBUG:teuthology.orchestra.run.smithi193:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-03-24T00:02:44.805 INFO:teuthology.task.install.util:Shipping 'stdin-killer'... 2024-03-24T00:02:44.806 DEBUG:teuthology.orchestra.run.smithi118:> set -ex 2024-03-24T00:02:44.806 DEBUG:teuthology.orchestra.run.smithi118:> sudo dd of=/usr/bin/stdin-killer 2024-03-24T00:02:44.886 DEBUG:teuthology.orchestra.run.smithi118:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-03-24T00:02:44.993 DEBUG:teuthology.orchestra.run.smithi189:> set -ex 2024-03-24T00:02:44.993 DEBUG:teuthology.orchestra.run.smithi189:> sudo dd of=/usr/bin/stdin-killer 2024-03-24T00:02:45.020 DEBUG:teuthology.orchestra.run.smithi189:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-03-24T00:02:45.086 DEBUG:teuthology.orchestra.run.smithi193:> set -ex 2024-03-24T00:02:45.086 DEBUG:teuthology.orchestra.run.smithi193:> sudo dd of=/usr/bin/stdin-killer 2024-03-24T00:02:45.112 DEBUG:teuthology.orchestra.run.smithi193:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-03-24T00:02:45.179 INFO:teuthology.run_tasks:Running task cephadm... 2024-03-24T00:02:45.285 INFO:tasks.cephadm:Config: {'roleless': False, 'cephfs': {'max_mds': 5, 'session_timeout': 300, 'standby_replay': True}, 'conf': {'client': {'client mount timeout': 600, 'debug client': 20, 'debug ms': 1, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'global': {'ms die on skipped message': False}, 'mds': {'debug mds': 20, 'debug mds balancer': 20, 'debug ms': 1, 'mds bal fragment size max': 10000, 'mds bal merge size': 5, 'mds bal split bits': 3, 'mds bal split size': 100, 'mds debug frag': True, 'mds debug scatterstat': True, 'mds op complaint time': 180, 'mds verify scatter': True, 'mds_bal_replicate_threshold': 1, 'mds_export_ephemeral_random_max': 0.1, 'osd op complaint time': 180, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'mgr': {'cephsqlite lock renewal timeout': 900000, 'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon op complaint time': 120}, 'osd': {'bluestore block size': 96636764160, 'bluestore compression mode': 'aggressive', 'bluestore fsck on mount': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore', 'osd op complaint time': 180, 'osd shutdown pgref assert': True, 'osd_max_omap_entries_per_request': 10}}, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', 'FS_DEGRADED', 'FS_INLINE_DATA_DEPRECATED', 'FS_WITH_FAILED_MDS', 'MDS_ALL_DOWN', 'MDS_DAMAGE', 'MDS_DEGRADED', 'MDS_FAILED', 'MDS_INSUFFICIENT_STANDBY', 'MDS_UP_LESS_THAN_MAX', 'POOL_APP_NOT_ENABLED', 'overall HEALTH_', 'Replacing daemon', 'deprecated feature inline_data', 'overall HEALTH_', '\\(OSD_DOWN\\)', '\\(OSD_', 'but it is still running', 'is not responding', 'MON_DOWN', 'slow metadata IO', 'SLOW_OPS', 'slow request'], 'sha1': '3be9a5c9bc793e11e2800a8c0c696e8b46742033', 'subvols': {'create': 2}} 2024-03-24T00:02:45.285 INFO:tasks.cephadm:Cluster image is quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 2024-03-24T00:02:45.285 INFO:tasks.cephadm:Cluster fsid is d79a5420-e971-11ee-95cd-87774f69a715 2024-03-24T00:02:45.285 INFO:tasks.cephadm:Choosing monitor IPs and ports... 2024-03-24T00:02:45.285 INFO:tasks.cephadm:Monitor IPs: {'mon.a': '172.21.15.118', 'mon.b': '172.21.15.189', 'mon.c': '172.21.15.193'} 2024-03-24T00:02:45.286 INFO:tasks.cephadm:First mon is mon.a on smithi118 2024-03-24T00:02:45.286 INFO:tasks.cephadm:First mgr is x 2024-03-24T00:02:45.286 INFO:tasks.cephadm:Normalizing hostnames... 2024-03-24T00:02:45.286 DEBUG:teuthology.orchestra.run.smithi118:> sudo hostname $(hostname -s) 2024-03-24T00:02:45.354 DEBUG:teuthology.orchestra.run.smithi189:> sudo hostname $(hostname -s) 2024-03-24T00:02:45.381 DEBUG:teuthology.orchestra.run.smithi193:> sudo hostname $(hostname -s) 2024-03-24T00:02:45.408 INFO:tasks.cephadm:Downloading "compiled" cephadm from cachra 2024-03-24T00:02:45.408 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&sha1=3be9a5c9bc793e11e2800a8c0c696e8b46742033 2024-03-24T00:02:45.678 INFO:tasks.cephadm:builder_project result: [{'status': 'ready', 'sha1': '3be9a5c9bc793e11e2800a8c0c696e8b46742033', 'extra': {'build_url': 'https://jenkins.ceph.com/job/ceph-dev-new-build/ARCH=x86_64,AVAILABLE_ARCH=x86_64,AVAILABLE_DIST=centos9,DIST=centos9,MACHINE_SIZE=gigantic/77877/', 'root_build_cause': 'SCMTRIGGER', 'version': '19.0.0-2145-g3be9a5c9', 'node_name': '172.21.5.36+adami06', 'job_name': 'ceph-dev-new-build/ARCH=x86_64,AVAILABLE_ARCH=x86_64,AVAILABLE_DIST=centos9,DIST=centos9,MACHINE_SIZE=gigantic', 'package_manager_version': '19.0.0-2145.g3be9a5c9'}, 'url': 'https://4.chacra.ceph.com/r/ceph/wip-mchangir-testing-main-20240318.032620/3be9a5c9bc793e11e2800a8c0c696e8b46742033/centos/9/flavors/default/', 'distro_codename': None, 'modified': '2024-03-22 05:51:42.267384', 'distro_version': '9', 'project': 'ceph', 'flavor': 'default', 'ref': 'wip-mchangir-testing-main-20240318.032620', 'chacra_url': 'https://4.chacra.ceph.com/repos/ceph/wip-mchangir-testing-main-20240318.032620/3be9a5c9bc793e11e2800a8c0c696e8b46742033/centos/9/flavors/default/', 'archs': ['source', 'x86_64'], 'distro': 'centos'}] 2024-03-24T00:02:45.821 INFO:tasks.util.chacra:got chacra host 4.chacra.ceph.com, ref wip-mchangir-testing-main-20240318.032620, sha1 3be9a5c9bc793e11e2800a8c0c696e8b46742033 from https://shaman.ceph.com/api/search/?project=ceph&distros=centos%2F9%2Fx86_64&flavor=default&sha1=3be9a5c9bc793e11e2800a8c0c696e8b46742033 2024-03-24T00:02:45.823 INFO:tasks.cephadm:Discovered cachra url: https://4.chacra.ceph.com/binaries/ceph/wip-mchangir-testing-main-20240318.032620/3be9a5c9bc793e11e2800a8c0c696e8b46742033/centos/9/x86_64/flavors/default/cephadm 2024-03-24T00:02:45.823 DEBUG:teuthology.orchestra.run.smithi118:> curl --silent -L https://4.chacra.ceph.com/binaries/ceph/wip-mchangir-testing-main-20240318.032620/3be9a5c9bc793e11e2800a8c0c696e8b46742033/centos/9/x86_64/flavors/default/cephadm > /home/ubuntu/cephtest/cephadm && ls -l /home/ubuntu/cephtest/cephadm 2024-03-24T00:02:46.366 INFO:teuthology.orchestra.run.smithi118.stdout:-rw-r--r--. 1 ubuntu ubuntu 767371 Mar 24 00:02 /home/ubuntu/cephtest/cephadm 2024-03-24T00:02:46.367 DEBUG:teuthology.orchestra.run.smithi189:> curl --silent -L https://4.chacra.ceph.com/binaries/ceph/wip-mchangir-testing-main-20240318.032620/3be9a5c9bc793e11e2800a8c0c696e8b46742033/centos/9/x86_64/flavors/default/cephadm > /home/ubuntu/cephtest/cephadm && ls -l /home/ubuntu/cephtest/cephadm 2024-03-24T00:02:46.844 INFO:teuthology.orchestra.run.smithi189.stdout:-rw-r--r--. 1 ubuntu ubuntu 767371 Mar 24 00:02 /home/ubuntu/cephtest/cephadm 2024-03-24T00:02:46.845 DEBUG:teuthology.orchestra.run.smithi193:> curl --silent -L https://4.chacra.ceph.com/binaries/ceph/wip-mchangir-testing-main-20240318.032620/3be9a5c9bc793e11e2800a8c0c696e8b46742033/centos/9/x86_64/flavors/default/cephadm > /home/ubuntu/cephtest/cephadm && ls -l /home/ubuntu/cephtest/cephadm 2024-03-24T00:02:47.328 INFO:teuthology.orchestra.run.smithi193.stdout:-rw-r--r--. 1 ubuntu ubuntu 767371 Mar 24 00:02 /home/ubuntu/cephtest/cephadm 2024-03-24T00:02:47.328 DEBUG:teuthology.orchestra.run.smithi118:> test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm 2024-03-24T00:02:47.385 DEBUG:teuthology.orchestra.run.smithi189:> test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm 2024-03-24T00:02:47.404 DEBUG:teuthology.orchestra.run.smithi193:> test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm 2024-03-24T00:02:47.438 INFO:tasks.cephadm:Pulling image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 on all hosts... 2024-03-24T00:02:47.438 DEBUG:teuthology.orchestra.run.smithi118:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 pull 2024-03-24T00:02:47.445 DEBUG:teuthology.orchestra.run.smithi189:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 pull 2024-03-24T00:02:47.448 DEBUG:teuthology.orchestra.run.smithi193:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 pull 2024-03-24T00:02:48.238 INFO:teuthology.orchestra.run.smithi189.stderr:Pulling container image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033... 2024-03-24T00:02:48.311 INFO:teuthology.orchestra.run.smithi193.stderr:Pulling container image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033... 2024-03-24T00:02:49.310 INFO:teuthology.orchestra.run.smithi118.stderr:Pulling container image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033... 2024-03-24T00:03:22.437 INFO:teuthology.orchestra.run.smithi189.stdout:{ 2024-03-24T00:03:22.437 INFO:teuthology.orchestra.run.smithi189.stdout: "ceph_version": "ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev)", 2024-03-24T00:03:22.438 INFO:teuthology.orchestra.run.smithi189.stdout: "image_id": "c664afe56bb01db4f9afa9532c20598e201e664f2e9ba2be50db0d52674a26d2", 2024-03-24T00:03:22.438 INFO:teuthology.orchestra.run.smithi189.stdout: "repo_digests": [ 2024-03-24T00:03:22.438 INFO:teuthology.orchestra.run.smithi189.stdout: "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc" 2024-03-24T00:03:22.438 INFO:teuthology.orchestra.run.smithi189.stdout: ] 2024-03-24T00:03:22.438 INFO:teuthology.orchestra.run.smithi189.stdout:} 2024-03-24T00:03:27.329 INFO:teuthology.orchestra.run.smithi193.stdout:{ 2024-03-24T00:03:27.329 INFO:teuthology.orchestra.run.smithi193.stdout: "ceph_version": "ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev)", 2024-03-24T00:03:27.329 INFO:teuthology.orchestra.run.smithi193.stdout: "image_id": "c664afe56bb01db4f9afa9532c20598e201e664f2e9ba2be50db0d52674a26d2", 2024-03-24T00:03:27.329 INFO:teuthology.orchestra.run.smithi193.stdout: "repo_digests": [ 2024-03-24T00:03:27.329 INFO:teuthology.orchestra.run.smithi193.stdout: "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc" 2024-03-24T00:03:27.329 INFO:teuthology.orchestra.run.smithi193.stdout: ] 2024-03-24T00:03:27.329 INFO:teuthology.orchestra.run.smithi193.stdout:} 2024-03-24T00:03:36.136 INFO:teuthology.orchestra.run.smithi118.stdout:{ 2024-03-24T00:03:36.136 INFO:teuthology.orchestra.run.smithi118.stdout: "ceph_version": "ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev)", 2024-03-24T00:03:36.137 INFO:teuthology.orchestra.run.smithi118.stdout: "image_id": "c664afe56bb01db4f9afa9532c20598e201e664f2e9ba2be50db0d52674a26d2", 2024-03-24T00:03:36.137 INFO:teuthology.orchestra.run.smithi118.stdout: "repo_digests": [ 2024-03-24T00:03:36.137 INFO:teuthology.orchestra.run.smithi118.stdout: "quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc" 2024-03-24T00:03:36.137 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-03-24T00:03:36.137 INFO:teuthology.orchestra.run.smithi118.stdout:} 2024-03-24T00:03:36.174 DEBUG:teuthology.orchestra.run.smithi118:> sudo mkdir -p /etc/ceph 2024-03-24T00:03:36.271 DEBUG:teuthology.orchestra.run.smithi189:> sudo mkdir -p /etc/ceph 2024-03-24T00:03:36.306 DEBUG:teuthology.orchestra.run.smithi193:> sudo mkdir -p /etc/ceph 2024-03-24T00:03:36.340 DEBUG:teuthology.orchestra.run.smithi118:> sudo chmod 777 /etc/ceph 2024-03-24T00:03:36.434 DEBUG:teuthology.orchestra.run.smithi189:> sudo chmod 777 /etc/ceph 2024-03-24T00:03:36.466 DEBUG:teuthology.orchestra.run.smithi193:> sudo chmod 777 /etc/ceph 2024-03-24T00:03:36.497 INFO:tasks.cephadm:Writing seed config... 2024-03-24T00:03:36.498 INFO:tasks.cephadm: override: [client] client mount timeout = 600 2024-03-24T00:03:36.499 INFO:tasks.cephadm: override: [client] debug client = 20 2024-03-24T00:03:36.499 INFO:tasks.cephadm: override: [client] debug ms = 1 2024-03-24T00:03:36.499 INFO:tasks.cephadm: override: [client] rados mon op timeout = 900 2024-03-24T00:03:36.499 INFO:tasks.cephadm: override: [client] rados osd op timeout = 900 2024-03-24T00:03:36.499 INFO:tasks.cephadm: override: [global] ms die on skipped message = False 2024-03-24T00:03:36.499 INFO:tasks.cephadm: override: [mds] debug mds = 20 2024-03-24T00:03:36.499 INFO:tasks.cephadm: override: [mds] debug mds balancer = 20 2024-03-24T00:03:36.499 INFO:tasks.cephadm: override: [mds] debug ms = 1 2024-03-24T00:03:36.499 INFO:tasks.cephadm: override: [mds] mds bal fragment size max = 10000 2024-03-24T00:03:36.499 INFO:tasks.cephadm: override: [mds] mds bal merge size = 5 2024-03-24T00:03:36.499 INFO:tasks.cephadm: override: [mds] mds bal split bits = 3 2024-03-24T00:03:36.499 INFO:tasks.cephadm: override: [mds] mds bal split size = 100 2024-03-24T00:03:36.499 INFO:tasks.cephadm: override: [mds] mds debug frag = True 2024-03-24T00:03:36.499 INFO:tasks.cephadm: override: [mds] mds debug scatterstat = True 2024-03-24T00:03:36.499 INFO:tasks.cephadm: override: [mds] mds op complaint time = 180 2024-03-24T00:03:36.500 INFO:tasks.cephadm: override: [mds] mds verify scatter = True 2024-03-24T00:03:36.500 INFO:tasks.cephadm: override: [mds] mds_bal_replicate_threshold = 1 2024-03-24T00:03:36.500 INFO:tasks.cephadm: override: [mds] mds_export_ephemeral_random_max = 0.1 2024-03-24T00:03:36.500 INFO:tasks.cephadm: override: [mds] osd op complaint time = 180 2024-03-24T00:03:36.500 INFO:tasks.cephadm: override: [mds] rados mon op timeout = 900 2024-03-24T00:03:36.500 INFO:tasks.cephadm: override: [mds] rados osd op timeout = 900 2024-03-24T00:03:36.500 INFO:tasks.cephadm: override: [mgr] cephsqlite lock renewal timeout = 900000 2024-03-24T00:03:36.500 INFO:tasks.cephadm: override: [mgr] debug mgr = 20 2024-03-24T00:03:36.500 INFO:tasks.cephadm: override: [mgr] debug ms = 1 2024-03-24T00:03:36.500 INFO:tasks.cephadm: override: [mon] debug mon = 20 2024-03-24T00:03:36.500 INFO:tasks.cephadm: override: [mon] debug ms = 1 2024-03-24T00:03:36.500 INFO:tasks.cephadm: override: [mon] debug paxos = 20 2024-03-24T00:03:36.500 INFO:tasks.cephadm: override: [mon] mon op complaint time = 120 2024-03-24T00:03:36.500 INFO:tasks.cephadm: override: [osd] bluestore block size = 96636764160 2024-03-24T00:03:36.500 INFO:tasks.cephadm: override: [osd] bluestore compression mode = aggressive 2024-03-24T00:03:36.500 INFO:tasks.cephadm: override: [osd] bluestore fsck on mount = True 2024-03-24T00:03:36.501 INFO:tasks.cephadm: override: [osd] debug bluefs = 20 2024-03-24T00:03:36.501 INFO:tasks.cephadm: override: [osd] debug bluestore = 20 2024-03-24T00:03:36.501 INFO:tasks.cephadm: override: [osd] debug ms = 1 2024-03-24T00:03:36.501 INFO:tasks.cephadm: override: [osd] debug osd = 20 2024-03-24T00:03:36.501 INFO:tasks.cephadm: override: [osd] debug rocksdb = 10 2024-03-24T00:03:36.501 INFO:tasks.cephadm: override: [osd] mon osd backfillfull_ratio = 0.85 2024-03-24T00:03:36.501 INFO:tasks.cephadm: override: [osd] mon osd full ratio = 0.9 2024-03-24T00:03:36.501 INFO:tasks.cephadm: override: [osd] mon osd nearfull ratio = 0.8 2024-03-24T00:03:36.501 INFO:tasks.cephadm: override: [osd] osd failsafe full ratio = 0.95 2024-03-24T00:03:36.501 INFO:tasks.cephadm: override: [osd] osd objectstore = bluestore 2024-03-24T00:03:36.501 INFO:tasks.cephadm: override: [osd] osd op complaint time = 180 2024-03-24T00:03:36.501 INFO:tasks.cephadm: override: [osd] osd shutdown pgref assert = True 2024-03-24T00:03:36.501 INFO:tasks.cephadm: override: [osd] osd_max_omap_entries_per_request = 10 2024-03-24T00:03:36.505 DEBUG:teuthology.orchestra.run.smithi118:> set -ex 2024-03-24T00:03:36.505 DEBUG:teuthology.orchestra.run.smithi118:> dd of=/home/ubuntu/cephtest/seed.ceph.conf 2024-03-24T00:03:36.564 DEBUG:tasks.cephadm:Final config: [global] # make logging friendly to teuthology log_to_file = true log_to_stderr = false log to journald = false mon cluster log to file = true mon cluster log file level = debug mon clock drift allowed = 1.000 # replicate across OSDs, not hosts osd crush chooseleaf type = 0 #osd pool default size = 2 osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd # enable some debugging auth debug = true ms die on old message = true ms die on bug = true debug asserts on shutdown = true # adjust warnings mon max pg per osd = 10000# >= luminous mon pg warn max object skew = 0 mon osd allow primary affinity = true mon osd allow pg remap = true mon warn on legacy crush tunables = false mon warn on crush straw calc version zero = false mon warn on no sortbitwise = false mon warn on osd down out interval zero = false mon warn on too few osds = false mon_warn_on_pool_pg_num_not_power_of_two = false # disable pg_autoscaler by default for new pools osd_pool_default_pg_autoscale_mode = off # tests delete pools mon allow pool delete = true fsid = d79a5420-e971-11ee-95cd-87774f69a715 ms die on skipped message = False [osd] osd scrub load threshold = 5.0 osd scrub max interval = 600 osd mclock profile = high_recovery_ops osd recover clone overlap = true osd recovery max chunk = 1048576 osd deep scrub update digest min age = 30 osd map max advance = 10 osd memory target autotune = true # debugging osd debug shutdown = true osd debug op order = true osd debug verify stray on activate = true osd debug pg log writeout = true osd debug verify cached snaps = true osd debug verify missing on start = true osd debug misdirected ops = true osd op queue = debug_random osd op queue cut off = debug_random osd shutdown pgref assert = True bdev debug aio = true osd sloppy crc = true bluestore block size = 96636764160 bluestore compression mode = aggressive bluestore fsck on mount = True debug bluefs = 20 debug bluestore = 20 debug ms = 1 debug osd = 20 debug rocksdb = 10 mon osd backfillfull_ratio = 0.85 mon osd full ratio = 0.9 mon osd nearfull ratio = 0.8 osd failsafe full ratio = 0.95 osd objectstore = bluestore osd op complaint time = 180 osd_max_omap_entries_per_request = 10 [mgr] mon reweight min pgs per osd = 4 mon reweight min bytes per osd = 10 mgr/telemetry/nag = false cephsqlite lock renewal timeout = 900000 debug mgr = 20 debug ms = 1 [mon] mon data avail warn = 5 mon mgr mkfs grace = 240 mon reweight min pgs per osd = 4 mon osd reporter subtree level = osd mon osd prime pg temp = true mon reweight min bytes per osd = 10 # rotate auth tickets quickly to exercise renewal paths auth mon ticket ttl = 660# 11m auth service ticket ttl = 240# 4m # don't complain about global id reclaim mon_warn_on_insecure_global_id_reclaim = false mon_warn_on_insecure_global_id_reclaim_allowed = false debug mon = 20 debug ms = 1 debug paxos = 20 mon op complaint time = 120 [client.rgw] rgw cache enabled = true rgw enable ops log = true rgw enable usage log = true [client] client mount timeout = 600 debug client = 20 debug ms = 1 rados mon op timeout = 900 rados osd op timeout = 900 [mds] debug mds = 20 debug mds balancer = 20 debug ms = 1 mds bal fragment size max = 10000 mds bal merge size = 5 mds bal split bits = 3 mds bal split size = 100 mds debug frag = True mds debug scatterstat = True mds op complaint time = 180 mds verify scatter = True mds_bal_replicate_threshold = 1 mds_export_ephemeral_random_max = 0.1 osd op complaint time = 180 rados mon op timeout = 900 rados osd op timeout = 900 2024-03-24T00:03:36.564 DEBUG:teuthology.orchestra.run.smithi118:mon.a> sudo journalctl -f -n 0 -u ceph-d79a5420-e971-11ee-95cd-87774f69a715@mon.a.service 2024-03-24T00:03:36.612 DEBUG:teuthology.orchestra.run.smithi118:mgr.x> sudo journalctl -f -n 0 -u ceph-d79a5420-e971-11ee-95cd-87774f69a715@mgr.x.service 2024-03-24T00:03:36.657 INFO:tasks.cephadm:Bootstrapping... 2024-03-24T00:03:36.658 DEBUG:teuthology.orchestra.run.smithi118:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 -v bootstrap --fsid d79a5420-e971-11ee-95cd-87774f69a715 --config /home/ubuntu/cephtest/seed.ceph.conf --output-config /etc/ceph/ceph.conf --output-keyring /etc/ceph/ceph.client.admin.keyring --output-pub-ssh-key /home/ubuntu/cephtest/ceph.pub --mon-id a --mgr-id x --orphan-initial-daemons --skip-monitoring-stack --mon-ip 172.21.15.118 --skip-admin-label && sudo chmod +r /etc/ceph/ceph.client.admin.keyring 2024-03-24T00:03:36.938 INFO:teuthology.orchestra.run.smithi118.stdout:-------------------------------------------------------------------------------- 2024-03-24T00:03:36.938 INFO:teuthology.orchestra.run.smithi118.stdout:cephadm ['--image', 'quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033', '-v', 'bootstrap', '--fsid', 'd79a5420-e971-11ee-95cd-87774f69a715', '--config', '/home/ubuntu/cephtest/seed.ceph.conf', '--output-config', '/etc/ceph/ceph.conf', '--output-keyring', '/etc/ceph/ceph.client.admin.keyring', '--output-pub-ssh-key', '/home/ubuntu/cephtest/ceph.pub', '--mon-id', 'a', '--mgr-id', 'x', '--orphan-initial-daemons', '--skip-monitoring-stack', '--mon-ip', '172.21.15.118', '--skip-admin-label'] 2024-03-24T00:03:36.992 INFO:teuthology.orchestra.run.smithi118.stdout:/bin/podman: stdout 4.9.3-dev 2024-03-24T00:03:36.993 INFO:teuthology.orchestra.run.smithi118.stderr:Specifying an fsid for your cluster offers no advantages and may increase the likelihood of fsid conflicts. 2024-03-24T00:03:36.993 INFO:teuthology.orchestra.run.smithi118.stdout:Verifying podman|docker is present... 2024-03-24T00:03:37.041 INFO:teuthology.orchestra.run.smithi118.stdout:/bin/podman: stdout 4.9.3-dev 2024-03-24T00:03:37.041 INFO:teuthology.orchestra.run.smithi118.stdout:Verifying lvm2 is present... 2024-03-24T00:03:37.041 INFO:teuthology.orchestra.run.smithi118.stdout:Verifying time synchronization is in place... 2024-03-24T00:03:37.056 INFO:teuthology.orchestra.run.smithi118.stdout:Non-zero exit code 1 from systemctl is-enabled chrony.service 2024-03-24T00:03:37.056 INFO:teuthology.orchestra.run.smithi118.stdout:systemctl: stderr Failed to get unit file state for chrony.service: No such file or directory 2024-03-24T00:03:37.071 INFO:teuthology.orchestra.run.smithi118.stdout:Non-zero exit code 3 from systemctl is-active chrony.service 2024-03-24T00:03:37.071 INFO:teuthology.orchestra.run.smithi118.stdout:systemctl: stdout inactive 2024-03-24T00:03:37.090 INFO:teuthology.orchestra.run.smithi118.stdout:systemctl: stdout enabled 2024-03-24T00:03:37.108 INFO:teuthology.orchestra.run.smithi118.stdout:systemctl: stdout active 2024-03-24T00:03:37.109 INFO:teuthology.orchestra.run.smithi118.stdout:Unit chronyd.service is enabled and running 2024-03-24T00:03:37.109 INFO:teuthology.orchestra.run.smithi118.stdout:Repeating the final host check... 2024-03-24T00:03:37.179 INFO:teuthology.orchestra.run.smithi118.stdout:/bin/podman: stdout 4.9.3-dev 2024-03-24T00:03:37.180 INFO:teuthology.orchestra.run.smithi118.stdout:podman (/bin/podman) version 4.9.3 is present 2024-03-24T00:03:37.180 INFO:teuthology.orchestra.run.smithi118.stdout:systemctl is present 2024-03-24T00:03:37.180 INFO:teuthology.orchestra.run.smithi118.stdout:lvcreate is present 2024-03-24T00:03:37.196 INFO:teuthology.orchestra.run.smithi118.stdout:Non-zero exit code 1 from systemctl is-enabled chrony.service 2024-03-24T00:03:37.197 INFO:teuthology.orchestra.run.smithi118.stdout:systemctl: stderr Failed to get unit file state for chrony.service: No such file or directory 2024-03-24T00:03:37.213 INFO:teuthology.orchestra.run.smithi118.stdout:Non-zero exit code 3 from systemctl is-active chrony.service 2024-03-24T00:03:37.213 INFO:teuthology.orchestra.run.smithi118.stdout:systemctl: stdout inactive 2024-03-24T00:03:37.231 INFO:teuthology.orchestra.run.smithi118.stdout:systemctl: stdout enabled 2024-03-24T00:03:37.248 INFO:teuthology.orchestra.run.smithi118.stdout:systemctl: stdout active 2024-03-24T00:03:37.248 INFO:teuthology.orchestra.run.smithi118.stdout:Unit chronyd.service is enabled and running 2024-03-24T00:03:37.249 INFO:teuthology.orchestra.run.smithi118.stdout:Host looks OK 2024-03-24T00:03:37.249 INFO:teuthology.orchestra.run.smithi118.stdout:Cluster fsid: d79a5420-e971-11ee-95cd-87774f69a715 2024-03-24T00:03:37.249 INFO:teuthology.orchestra.run.smithi118.stdout:Acquiring lock 139772062010816 on /run/cephadm/d79a5420-e971-11ee-95cd-87774f69a715.lock 2024-03-24T00:03:37.249 INFO:teuthology.orchestra.run.smithi118.stdout:Lock 139772062010816 acquired on /run/cephadm/d79a5420-e971-11ee-95cd-87774f69a715.lock 2024-03-24T00:03:37.249 INFO:teuthology.orchestra.run.smithi118.stdout:Verifying IP 172.21.15.118 port 3300 ... 2024-03-24T00:03:37.250 INFO:teuthology.orchestra.run.smithi118.stdout:Verifying IP 172.21.15.118 port 6789 ... 2024-03-24T00:03:37.250 INFO:teuthology.orchestra.run.smithi118.stdout:Base mon IP(s) is [172.21.15.118:3300, 172.21.15.118:6789], mon addrv is [v2:172.21.15.118:3300,v1:172.21.15.118:6789] 2024-03-24T00:03:37.258 INFO:teuthology.orchestra.run.smithi118.stdout:/sbin/ip: stdout default via 172.21.15.254 dev enp3s0f1 proto dhcp src 172.21.15.118 metric 100 2024-03-24T00:03:37.258 INFO:teuthology.orchestra.run.smithi118.stdout:/sbin/ip: stdout 172.21.0.0/20 dev enp3s0f1 proto kernel scope link src 172.21.15.118 metric 100 2024-03-24T00:03:37.266 INFO:teuthology.orchestra.run.smithi118.stdout:/sbin/ip: stdout ::1 dev lo proto kernel metric 256 pref medium 2024-03-24T00:03:37.266 INFO:teuthology.orchestra.run.smithi118.stdout:/sbin/ip: stdout fe80::/64 dev enp3s0f1 proto kernel metric 256 pref medium 2024-03-24T00:03:37.266 INFO:teuthology.orchestra.run.smithi118.stdout:/sbin/ip: stdout default via fe80::327c:5e00:6487:71e0 dev enp3s0f1 proto ra metric 1024 expires 1794sec hoplimit 64 pref medium 2024-03-24T00:03:37.274 INFO:teuthology.orchestra.run.smithi118.stdout:/sbin/ip: stdout 1: lo: mtu 65536 state UNKNOWN qlen 1000 2024-03-24T00:03:37.274 INFO:teuthology.orchestra.run.smithi118.stdout:/sbin/ip: stdout inet6 ::1/128 scope host 2024-03-24T00:03:37.274 INFO:teuthology.orchestra.run.smithi118.stdout:/sbin/ip: stdout valid_lft forever preferred_lft forever 2024-03-24T00:03:37.274 INFO:teuthology.orchestra.run.smithi118.stdout:/sbin/ip: stdout 5: enp3s0f1: mtu 1500 state UP qlen 1000 2024-03-24T00:03:37.274 INFO:teuthology.orchestra.run.smithi118.stdout:/sbin/ip: stdout inet6 fe80::ec4:7aff:fe8f:d725/64 scope link 2024-03-24T00:03:37.275 INFO:teuthology.orchestra.run.smithi118.stdout:/sbin/ip: stdout valid_lft forever preferred_lft forever 2024-03-24T00:03:37.275 INFO:teuthology.orchestra.run.smithi118.stdout:Mon IP `172.21.15.118` is in CIDR network `172.21.0.0/20` 2024-03-24T00:03:37.275 INFO:teuthology.orchestra.run.smithi118.stdout:Mon IP `172.21.15.118` is in CIDR network `172.21.0.0/20` 2024-03-24T00:03:37.276 INFO:teuthology.orchestra.run.smithi118.stdout:Inferred mon public CIDR from local network configuration ['172.21.0.0/20', '172.21.0.0/20'] 2024-03-24T00:03:37.276 INFO:teuthology.orchestra.run.smithi118.stdout:Internal network (--cluster-network) has not been provided, OSD replication will default to the public_network 2024-03-24T00:03:37.278 INFO:teuthology.orchestra.run.smithi118.stdout:Pulling container image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033... 2024-03-24T00:03:46.676 INFO:teuthology.orchestra.run.smithi118.stdout:/bin/podman: stdout c664afe56bb01db4f9afa9532c20598e201e664f2e9ba2be50db0d52674a26d2 2024-03-24T00:03:46.676 INFO:teuthology.orchestra.run.smithi118.stdout:/bin/podman: stderr Trying to pull quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033... 2024-03-24T00:03:46.676 INFO:teuthology.orchestra.run.smithi118.stdout:/bin/podman: stderr Getting image source signatures 2024-03-24T00:03:46.677 INFO:teuthology.orchestra.run.smithi118.stdout:/bin/podman: stderr Copying blob sha256:4dd83c6ee8921fe17396d77c31e205985ae94c87d3f752465b5f04aaf9ae340a 2024-03-24T00:03:46.677 INFO:teuthology.orchestra.run.smithi118.stdout:/bin/podman: stderr Copying blob sha256:9516588f908cdb14b5e4062a6063db2d9291b355c3f32e2a99864608a4a7b649 2024-03-24T00:03:46.677 INFO:teuthology.orchestra.run.smithi118.stdout:/bin/podman: stderr Copying config sha256:c664afe56bb01db4f9afa9532c20598e201e664f2e9ba2be50db0d52674a26d2 2024-03-24T00:03:46.677 INFO:teuthology.orchestra.run.smithi118.stdout:/bin/podman: stderr Writing manifest to image destination 2024-03-24T00:03:49.096 INFO:teuthology.orchestra.run.smithi118.stdout:ceph: stdout ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-24T00:03:49.096 INFO:teuthology.orchestra.run.smithi118.stdout:Ceph version: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-24T00:03:49.096 INFO:teuthology.orchestra.run.smithi118.stdout:Extracting ceph user uid/gid from container image... 2024-03-24T00:03:50.344 INFO:teuthology.orchestra.run.smithi118.stdout:stat: stdout 167 167 2024-03-24T00:03:50.345 INFO:teuthology.orchestra.run.smithi118.stdout:Creating initial keys... 2024-03-24T00:03:51.677 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph-authtool: stdout AQDmbf9l7efvMRAAE9ALevAWugR1xXOPWTVO1g== 2024-03-24T00:03:52.825 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph-authtool: stdout AQDobf9l7Zc2CRAAS3haFuxGes1PS18Cl21erQ== 2024-03-24T00:03:53.908 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph-authtool: stdout AQDpbf9lCBiRDxAAQuRCXSyM6xoCdn0HyemJtg== 2024-03-24T00:03:53.911 INFO:teuthology.orchestra.run.smithi118.stdout:Creating initial monmap... 2024-03-24T00:03:55.140 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/monmaptool: stdout /usr/bin/monmaptool: monmap file /tmp/monmap 2024-03-24T00:03:55.141 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/monmaptool: stdout setting min_mon_release = quincy 2024-03-24T00:03:55.141 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/monmaptool: stdout /usr/bin/monmaptool: set fsid to d79a5420-e971-11ee-95cd-87774f69a715 2024-03-24T00:03:55.141 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/monmaptool: stdout /usr/bin/monmaptool: writing epoch 0 to /tmp/monmap (1 monitors) 2024-03-24T00:03:55.141 INFO:teuthology.orchestra.run.smithi118.stdout:monmaptool for a [v2:172.21.15.118:3300,v1:172.21.15.118:6789] on /usr/bin/monmaptool: monmap file /tmp/monmap 2024-03-24T00:03:55.141 INFO:teuthology.orchestra.run.smithi118.stdout:setting min_mon_release = quincy 2024-03-24T00:03:55.141 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/monmaptool: set fsid to d79a5420-e971-11ee-95cd-87774f69a715 2024-03-24T00:03:55.141 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/monmaptool: writing epoch 0 to /tmp/monmap (1 monitors) 2024-03-24T00:03:55.141 INFO:teuthology.orchestra.run.smithi118.stdout: 2024-03-24T00:03:55.141 INFO:teuthology.orchestra.run.smithi118.stdout:Creating mon... 2024-03-24T00:03:56.282 INFO:teuthology.orchestra.run.smithi118.stdout:create mon.a on 2024-03-24T00:03:56.646 INFO:teuthology.orchestra.run.smithi118.stdout:systemctl: stderr Removed "/etc/systemd/system/multi-user.target.wants/ceph.target". 2024-03-24T00:03:56.898 INFO:teuthology.orchestra.run.smithi118.stdout:systemctl: stderr Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /etc/systemd/system/ceph.target. 2024-03-24T00:03:57.188 INFO:teuthology.orchestra.run.smithi118.stdout:systemctl: stderr Created symlink /etc/systemd/system/multi-user.target.wants/ceph-d79a5420-e971-11ee-95cd-87774f69a715.target → /etc/systemd/system/ceph-d79a5420-e971-11ee-95cd-87774f69a715.target. 2024-03-24T00:03:57.188 INFO:teuthology.orchestra.run.smithi118.stdout:systemctl: stderr Created symlink /etc/systemd/system/ceph.target.wants/ceph-d79a5420-e971-11ee-95cd-87774f69a715.target → /etc/systemd/system/ceph-d79a5420-e971-11ee-95cd-87774f69a715.target. 2024-03-24T00:03:57.607 INFO:teuthology.orchestra.run.smithi118.stdout:Non-zero exit code 1 from systemctl reset-failed ceph-d79a5420-e971-11ee-95cd-87774f69a715@mon.a 2024-03-24T00:03:57.607 INFO:teuthology.orchestra.run.smithi118.stdout:systemctl: stderr Failed to reset failed state of unit ceph-d79a5420-e971-11ee-95cd-87774f69a715@mon.a.service: Unit ceph-d79a5420-e971-11ee-95cd-87774f69a715@mon.a.service not loaded. 2024-03-24T00:03:57.895 INFO:teuthology.orchestra.run.smithi118.stdout:systemctl: stderr Created symlink /etc/systemd/system/ceph-d79a5420-e971-11ee-95cd-87774f69a715.target.wants/ceph-d79a5420-e971-11ee-95cd-87774f69a715@mon.a.service → /etc/systemd/system/ceph-d79a5420-e971-11ee-95cd-87774f69a715@.service. 2024-03-24T00:03:58.566 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:03:58 smithi118 podman[31148]: 2024-03-24 00:03:58.328868338 +0000 UTC m=+0.180945388 container create 50ce263b4def37e26d2d8314244644a5602d5025e991840dc673ee920ede8788 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033, name=ceph-d79a5420-e971-11ee-95cd-87774f69a715-mon-a, org.label-schema.vendor=CentOS, GIT_CLEAN=True, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240319, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-24T00:03:58.567 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:03:58 smithi118 podman[31148]: 2024-03-24 00:03:58.565598281 +0000 UTC m=+0.417675401 container init 50ce263b4def37e26d2d8314244644a5602d5025e991840dc673ee920ede8788 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033, name=ceph-d79a5420-e971-11ee-95cd-87774f69a715-mon-a, GIT_BRANCH=HEAD, org.label-schema.build-date=20240319, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, ceph=True, GIT_CLEAN=True, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0) 2024-03-24T00:03:58.677 INFO:teuthology.orchestra.run.smithi118.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-03-24T00:03:58.677 INFO:teuthology.orchestra.run.smithi118.stdout:systemctl: stdout disabled 2024-03-24T00:03:58.694 INFO:teuthology.orchestra.run.smithi118.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-03-24T00:03:58.694 INFO:teuthology.orchestra.run.smithi118.stdout:systemctl: stdout inactive 2024-03-24T00:03:58.694 INFO:teuthology.orchestra.run.smithi118.stdout:firewalld.service is not enabled 2024-03-24T00:03:58.694 INFO:teuthology.orchestra.run.smithi118.stdout:Not possible to enable service . firewalld.service is not available 2024-03-24T00:03:58.695 INFO:teuthology.orchestra.run.smithi118.stdout:Waiting for mon to start... 2024-03-24T00:03:58.695 INFO:teuthology.orchestra.run.smithi118.stdout:Waiting for mon... 2024-03-24T00:03:58.824 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:03:58 smithi118 podman[31148]: 2024-03-24 00:03:58.575551176 +0000 UTC m=+0.427628225 container start 50ce263b4def37e26d2d8314244644a5602d5025e991840dc673ee920ede8788 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033, name=ceph-d79a5420-e971-11ee-95cd-87774f69a715-mon-a, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.build-date=20240319, org.label-schema.license=GPLv2, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, io.buildah.version=1.33.5, ceph=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_CLEAN=True, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux ) 2024-03-24T00:03:58.824 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:03:58 smithi118 bash[31148]: 50ce263b4def37e26d2d8314244644a5602d5025e991840dc673ee920ede8788 2024-03-24T00:03:58.824 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:03:58 smithi118 ceph-mon[31168]: mkfs d79a5420-e971-11ee-95cd-87774f69a715 2024-03-24T00:03:58.824 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:03:58 smithi118 systemd[1]: Started Ceph mon.a for d79a5420-e971-11ee-95cd-87774f69a715. 2024-03-24T00:03:58.824 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:03:58 smithi118 ceph-mon[31168]: mon.a is new leader, mons a in quorum (ranks 0) 2024-03-24T00:03:59.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:03:59 smithi118 ceph-mon[31168]: mon.a is new leader, mons a in quorum (ranks 0) 2024-03-24T00:03:59.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:03:59 smithi118 ceph-mon[31168]: monmap e1: 1 mons at {a=[v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0]} removed_ranks: {} disallowed_leaders: {} 2024-03-24T00:03:59.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:03:59 smithi118 ceph-mon[31168]: fsmap 2024-03-24T00:03:59.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:03:59 smithi118 ceph-mon[31168]: osdmap e1: 0 total, 0 up, 0 in 2024-03-24T00:03:59.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:03:59 smithi118 ceph-mon[31168]: mgrmap e1: no daemons active 2024-03-24T00:03:59.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:03:59 smithi118 ceph-mon[31168]: from='client.? 172.21.15.118:0/2445141426' entity='client.admin' cmd=[{"prefix": "status"}]: dispatch 2024-03-24T00:04:00.039 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stdout cluster: 2024-03-24T00:04:00.039 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stdout id: d79a5420-e971-11ee-95cd-87774f69a715 2024-03-24T00:04:00.040 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stdout health: HEALTH_OK 2024-03-24T00:04:00.040 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stdout 2024-03-24T00:04:00.040 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stdout services: 2024-03-24T00:04:00.041 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stdout mon: 1 daemons, quorum a (age 0.662262s) 2024-03-24T00:04:00.041 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stdout mgr: no daemons active 2024-03-24T00:04:00.041 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stdout osd: 0 osds: 0 up, 0 in 2024-03-24T00:04:00.041 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stdout 2024-03-24T00:04:00.042 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stdout data: 2024-03-24T00:04:00.042 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stdout pools: 0 pools, 0 pgs 2024-03-24T00:04:00.042 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stdout objects: 0 objects, 0 B 2024-03-24T00:04:00.042 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stdout usage: 0 B used, 0 B / 0 B avail 2024-03-24T00:04:00.042 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stdout pgs: 2024-03-24T00:04:00.042 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stdout 2024-03-24T00:04:00.043 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:03:59.219+0000 7f5798a00700 1 Processor -- start 2024-03-24T00:04:00.043 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:03:59.219+0000 7f5798a00700 1 -- start start 2024-03-24T00:04:00.043 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:03:59.220+0000 7f5798a00700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f5794106210 0x7f5794106630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:04:00.043 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:03:59.220+0000 7f5798a00700 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5794075860 con 0x7f5794106210 2024-03-24T00:04:00.044 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:03:59.221+0000 7f5792800700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f5794106210 0x7f5794106630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:04:00.044 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:03:59.221+0000 7f5792800700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f5794106210 0x7f5794106630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.118:3300/0 says I am v2:172.21.15.118:39766/0 (socket says 172.21.15.118:39766) 2024-03-24T00:04:00.044 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:03:59.221+0000 7f5792800700 1 -- 172.21.15.118:0/640476464 learned_addr learned my addr 172.21.15.118:0/640476464 (peer_addr_for_me v2:172.21.15.118:0/0) 2024-03-24T00:04:00.044 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:03:59.221+0000 7f5792800700 1 -- 172.21.15.118:0/640476464 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5794106b70 con 0x7f5794106210 2024-03-24T00:04:00.044 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:03:59.222+0000 7f5792800700 1 --2- 172.21.15.118:0/640476464 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f5794106210 0x7f5794106630 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7f5784009e80 tx=0x7f578400b330 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d22ba7b117727ae server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:04:00.045 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:03:59.222+0000 7f5791400700 1 -- 172.21.15.118:0/640476464 <== mon.0 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f5784017070 con 0x7f5794106210 2024-03-24T00:04:00.045 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:03:59.222+0000 7f5791400700 1 -- 172.21.15.118:0/640476464 <== mon.0 v2:172.21.15.118:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5784003a20 con 0x7f5794106210 2024-03-24T00:04:00.045 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:03:59.222+0000 7f5791400700 1 -- 172.21.15.118:0/640476464 <== mon.0 v2:172.21.15.118:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f5784003d20 con 0x7f5794106210 2024-03-24T00:04:00.046 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:03:59.223+0000 7f5798a00700 1 -- 172.21.15.118:0/640476464 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f5794106210 msgr2=0x7f5794106630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:04:00.046 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:03:59.223+0000 7f5798a00700 1 --2- 172.21.15.118:0/640476464 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f5794106210 0x7f5794106630 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7f5784009e80 tx=0x7f578400b330 comp rx=0 tx=0).stop 2024-03-24T00:04:00.046 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:03:59.223+0000 7f5798a00700 1 -- 172.21.15.118:0/640476464 shutdown_connections 2024-03-24T00:04:00.047 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:03:59.223+0000 7f5798a00700 1 --2- 172.21.15.118:0/640476464 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f5794106210 0x7f5794106630 unknown :-1 s=CLOSED pgs=1 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:04:00.047 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:03:59.223+0000 7f5798a00700 1 -- 172.21.15.118:0/640476464 >> 172.21.15.118:0/640476464 conn(0x7f5794101e30 msgr2=0x7f5794104250 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:04:00.047 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:03:59.223+0000 7f5798a00700 1 -- 172.21.15.118:0/640476464 shutdown_connections 2024-03-24T00:04:00.047 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:03:59.223+0000 7f5798a00700 1 -- 172.21.15.118:0/640476464 wait complete. 2024-03-24T00:04:00.047 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:03:59.224+0000 7f5798a00700 1 Processor -- start 2024-03-24T00:04:00.047 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:03:59.224+0000 7f5798a00700 1 -- start start 2024-03-24T00:04:00.048 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:03:59.224+0000 7f5798a00700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f5794106210 0x7f579419d270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:04:00.048 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:03:59.224+0000 7f5798a00700 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f57941070a0 con 0x7f5794106210 2024-03-24T00:04:00.048 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:03:59.225+0000 7f5792800700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f5794106210 0x7f579419d270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:04:00.048 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:03:59.225+0000 7f5792800700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f5794106210 0x7f579419d270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.118:3300/0 says I am v2:172.21.15.118:39780/0 (socket says 172.21.15.118:39780) 2024-03-24T00:04:00.048 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:03:59.225+0000 7f5792800700 1 -- 172.21.15.118:0/2445141426 learned_addr learned my addr 172.21.15.118:0/2445141426 (peer_addr_for_me v2:172.21.15.118:0/0) 2024-03-24T00:04:00.049 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:03:59.225+0000 7f5792800700 1 -- 172.21.15.118:0/2445141426 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f579419d7b0 con 0x7f5794106210 2024-03-24T00:04:00.049 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:03:59.225+0000 7f5792800700 1 --2- 172.21.15.118:0/2445141426 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f5794106210 0x7f579419d270 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7f5794074ca0 tx=0x7f5784004ad0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:04:00.049 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:03:59.225+0000 7f578b400700 1 -- 172.21.15.118:0/2445141426 <== mon.0 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f5784022020 con 0x7f5794106210 2024-03-24T00:04:00.049 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:03:59.226+0000 7f578b400700 1 -- 172.21.15.118:0/2445141426 <== mon.0 v2:172.21.15.118:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5784018040 con 0x7f5794106210 2024-03-24T00:04:00.049 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:03:59.226+0000 7f578b400700 1 -- 172.21.15.118:0/2445141426 <== mon.0 v2:172.21.15.118:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f5784017070 con 0x7f5794106210 2024-03-24T00:04:00.049 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:03:59.226+0000 7f5798a00700 1 -- 172.21.15.118:0/2445141426 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f579419da40 con 0x7f5794106210 2024-03-24T00:04:00.050 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:03:59.226+0000 7f5798a00700 1 -- 172.21.15.118:0/2445141426 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f57941a06e0 con 0x7f5794106210 2024-03-24T00:04:00.050 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:03:59.226+0000 7f578b400700 1 -- 172.21.15.118:0/2445141426 <== mon.0 v2:172.21.15.118:3300/0 4 ==== mgrmap(e 1) v1 ==== 807+0+0 (secure 0 0 0) 0x7f5784025020 con 0x7f5794106210 2024-03-24T00:04:00.050 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:03:59.226+0000 7f578b400700 1 -- 172.21.15.118:0/2445141426 <== mon.0 v2:172.21.15.118:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (secure 0 0 0) 0x7f578400f050 con 0x7f5794106210 2024-03-24T00:04:00.051 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:03:59.227+0000 7f5798a00700 1 -- 172.21.15.118:0/2445141426 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f57940637c0 con 0x7f5794106210 2024-03-24T00:04:00.051 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:03:59.230+0000 7f578b400700 1 -- 172.21.15.118:0/2445141426 <== mon.0 v2:172.21.15.118:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (secure 0 0 0) 0x7f578402c030 con 0x7f5794106210 2024-03-24T00:04:00.051 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:03:59.307+0000 7f5798a00700 1 -- 172.21.15.118:0/2445141426 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_command({"prefix": "status"} v 0) v1 -- 0x7f57941a0c00 con 0x7f5794106210 2024-03-24T00:04:00.051 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:03:59.308+0000 7f578b400700 1 -- 172.21.15.118:0/2445141426 <== mon.0 v2:172.21.15.118:3300/0 7 ==== mon_command_ack([{"prefix": "status"}]=0 v0) v1 ==== 54+0+317 (secure 0 0 0) 0x7f578401e480 con 0x7f5794106210 2024-03-24T00:04:00.052 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:03:59.309+0000 7f5798a00700 1 -- 172.21.15.118:0/2445141426 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f5794106210 msgr2=0x7f579419d270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:04:00.052 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:03:59.309+0000 7f5798a00700 1 --2- 172.21.15.118:0/2445141426 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f5794106210 0x7f579419d270 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7f5794074ca0 tx=0x7f5784004ad0 comp rx=0 tx=0).stop 2024-03-24T00:04:00.052 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:03:59.309+0000 7f5798a00700 1 -- 172.21.15.118:0/2445141426 shutdown_connections 2024-03-24T00:04:00.052 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:03:59.309+0000 7f5798a00700 1 --2- 172.21.15.118:0/2445141426 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f5794106210 0x7f579419d270 unknown :-1 s=CLOSED pgs=2 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:04:00.052 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:03:59.309+0000 7f5798a00700 1 -- 172.21.15.118:0/2445141426 >> 172.21.15.118:0/2445141426 conn(0x7f5794101e30 msgr2=0x7f5794193890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:04:00.052 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:03:59.309+0000 7f5798a00700 1 -- 172.21.15.118:0/2445141426 shutdown_connections 2024-03-24T00:04:00.052 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:03:59.309+0000 7f5798a00700 1 -- 172.21.15.118:0/2445141426 wait complete. 2024-03-24T00:04:00.053 INFO:teuthology.orchestra.run.smithi118.stdout:mon is available 2024-03-24T00:04:00.053 INFO:teuthology.orchestra.run.smithi118.stdout:Assimilating anything we can from ceph.conf... 2024-03-24T00:04:00.945 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:04:00 smithi118 ceph-mon[31168]: from='client.? 172.21.15.118:0/3631741566' entity='client.admin' cmd=[{"prefix": "config assimilate-conf"}]: dispatch 2024-03-24T00:04:00.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:04:00 smithi118 ceph-mon[31168]: from='client.? 172.21.15.118:0/3631741566' entity='client.admin' cmd='[{"prefix": "config assimilate-conf"}]': finished 2024-03-24T00:04:01.396 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stdout 2024-03-24T00:04:01.401 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stdout [global] 2024-03-24T00:04:01.401 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stdout fsid = d79a5420-e971-11ee-95cd-87774f69a715 2024-03-24T00:04:01.401 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stdout mon_host = [v2:172.21.15.118:3300,v1:172.21.15.118:6789] 2024-03-24T00:04:01.401 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stdout mon_osd_allow_pg_remap = true 2024-03-24T00:04:01.401 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stdout mon_osd_allow_primary_affinity = true 2024-03-24T00:04:01.401 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stdout mon_warn_on_no_sortbitwise = false 2024-03-24T00:04:01.401 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stdout osd_crush_chooseleaf_type = 0 2024-03-24T00:04:01.401 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stdout 2024-03-24T00:04:01.401 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stdout [mgr] 2024-03-24T00:04:01.401 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stdout mgr/telemetry/nag = false 2024-03-24T00:04:01.401 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stdout 2024-03-24T00:04:01.402 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stdout [osd] 2024-03-24T00:04:01.402 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stdout mon_osd_backfillfull_ratio = 0.85 2024-03-24T00:04:01.402 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stdout mon_osd_full_ratio = 0.9 2024-03-24T00:04:01.402 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stdout mon_osd_nearfull_ratio = 0.8 2024-03-24T00:04:01.402 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stdout osd_map_max_advance = 10 2024-03-24T00:04:01.402 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stdout osd_sloppy_crc = true 2024-03-24T00:04:01.402 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:04:00.601+0000 7f352d200700 1 Processor -- start 2024-03-24T00:04:01.402 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:04:00.601+0000 7f352d200700 1 -- start start 2024-03-24T00:04:01.402 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:04:00.602+0000 7f352d200700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f35280822f0 0x7f3528110d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:04:01.402 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:04:00.602+0000 7f352d200700 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f35281112d0 con 0x7f35280822f0 2024-03-24T00:04:01.403 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:04:00.602+0000 7f3525e00700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f35280822f0 0x7f3528110d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:04:01.403 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:04:00.602+0000 7f3525e00700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f35280822f0 0x7f3528110d00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.118:3300/0 says I am v2:172.21.15.118:39794/0 (socket says 172.21.15.118:39794) 2024-03-24T00:04:01.403 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:04:00.602+0000 7f3525e00700 1 -- 172.21.15.118:0/3724497155 learned_addr learned my addr 172.21.15.118:0/3724497155 (peer_addr_for_me v2:172.21.15.118:0/0) 2024-03-24T00:04:01.403 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:04:00.603+0000 7f3525e00700 1 -- 172.21.15.118:0/3724497155 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3528111450 con 0x7f35280822f0 2024-03-24T00:04:01.403 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:04:00.603+0000 7f3525e00700 1 --2- 172.21.15.118:0/3724497155 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f35280822f0 0x7f3528110d00 secure :-1 s=READY pgs=3 cs=0 l=1 rev1=1 crypto rx=0x7f35180098a0 tx=0x7f3518009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bc0766c8a3bf5f2a server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:04:01.403 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:04:00.603+0000 7f3524a00700 1 -- 172.21.15.118:0/3724497155 <== mon.0 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f3518004160 con 0x7f35280822f0 2024-03-24T00:04:01.403 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:04:00.603+0000 7f3524a00700 1 -- 172.21.15.118:0/3724497155 <== mon.0 v2:172.21.15.118:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3518004300 con 0x7f35280822f0 2024-03-24T00:04:01.403 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:04:00.603+0000 7f3524a00700 1 -- 172.21.15.118:0/3724497155 <== mon.0 v2:172.21.15.118:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f35180036a0 con 0x7f35280822f0 2024-03-24T00:04:01.403 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:04:00.604+0000 7f352d200700 1 -- 172.21.15.118:0/3724497155 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f35280822f0 msgr2=0x7f3528110d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:04:01.403 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:04:00.604+0000 7f352d200700 1 --2- 172.21.15.118:0/3724497155 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f35280822f0 0x7f3528110d00 secure :-1 s=READY pgs=3 cs=0 l=1 rev1=1 crypto rx=0x7f35180098a0 tx=0x7f3518009bb0 comp rx=0 tx=0).stop 2024-03-24T00:04:01.403 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:04:00.604+0000 7f352d200700 1 -- 172.21.15.118:0/3724497155 shutdown_connections 2024-03-24T00:04:01.403 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:04:00.604+0000 7f352d200700 1 --2- 172.21.15.118:0/3724497155 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f35280822f0 0x7f3528110d00 unknown :-1 s=CLOSED pgs=3 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:04:01.403 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:04:00.604+0000 7f352d200700 1 -- 172.21.15.118:0/3724497155 >> 172.21.15.118:0/3724497155 conn(0x7f352807d220 msgr2=0x7f352807f640 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:04:01.404 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:04:00.604+0000 7f352d200700 1 -- 172.21.15.118:0/3724497155 shutdown_connections 2024-03-24T00:04:01.404 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:04:00.604+0000 7f352d200700 1 -- 172.21.15.118:0/3724497155 wait complete. 2024-03-24T00:04:01.404 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:04:00.605+0000 7f352d200700 1 Processor -- start 2024-03-24T00:04:01.406 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:04:00.606+0000 7f352d200700 1 -- start start 2024-03-24T00:04:01.407 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:04:00.606+0000 7f352d200700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f35280822f0 0x7f35281a1870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:04:01.407 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:04:00.606+0000 7f352d200700 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3528111980 con 0x7f35280822f0 2024-03-24T00:04:01.407 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:04:00.606+0000 7f3525e00700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f35280822f0 0x7f35281a1870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:04:01.407 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:04:00.606+0000 7f3525e00700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f35280822f0 0x7f35281a1870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.118:3300/0 says I am v2:172.21.15.118:39806/0 (socket says 172.21.15.118:39806) 2024-03-24T00:04:01.407 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:04:00.606+0000 7f3525e00700 1 -- 172.21.15.118:0/3631741566 learned_addr learned my addr 172.21.15.118:0/3631741566 (peer_addr_for_me v2:172.21.15.118:0/0) 2024-03-24T00:04:01.407 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:04:00.607+0000 7f3525e00700 1 -- 172.21.15.118:0/3631741566 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f35281a1db0 con 0x7f35280822f0 2024-03-24T00:04:01.407 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:04:00.607+0000 7f3525e00700 1 --2- 172.21.15.118:0/3631741566 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f35280822f0 0x7f35281a1870 secure :-1 s=READY pgs=4 cs=0 l=1 rev1=1 crypto rx=0x7f3518000c00 tx=0x7f3518004750 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:04:01.407 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:04:00.607+0000 7f351ea00700 1 -- 172.21.15.118:0/3631741566 <== mon.0 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f3518003db0 con 0x7f35280822f0 2024-03-24T00:04:01.407 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:04:00.607+0000 7f351ea00700 1 -- 172.21.15.118:0/3631741566 <== mon.0 v2:172.21.15.118:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3518011630 con 0x7f35280822f0 2024-03-24T00:04:01.407 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:04:00.607+0000 7f352d200700 1 -- 172.21.15.118:0/3631741566 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f35281a2040 con 0x7f35280822f0 2024-03-24T00:04:01.407 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:04:00.607+0000 7f351ea00700 1 -- 172.21.15.118:0/3631741566 <== mon.0 v2:172.21.15.118:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f3518011950 con 0x7f35280822f0 2024-03-24T00:04:01.407 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:04:00.607+0000 7f352d200700 1 -- 172.21.15.118:0/3631741566 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f35281a4d30 con 0x7f35280822f0 2024-03-24T00:04:01.408 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:04:00.607+0000 7f351ea00700 1 -- 172.21.15.118:0/3631741566 <== mon.0 v2:172.21.15.118:3300/0 4 ==== mgrmap(e 1) v1 ==== 807+0+0 (secure 0 0 0) 0x7f351801a460 con 0x7f35280822f0 2024-03-24T00:04:01.408 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:04:00.608+0000 7f351ea00700 1 -- 172.21.15.118:0/3631741566 <== mon.0 v2:172.21.15.118:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (secure 0 0 0) 0x7f351801b780 con 0x7f35280822f0 2024-03-24T00:04:01.408 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:04:00.608+0000 7f352d200700 1 -- 172.21.15.118:0/3631741566 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f35280637c0 con 0x7f35280822f0 2024-03-24T00:04:01.408 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:04:00.611+0000 7f351ea00700 1 -- 172.21.15.118:0/3631741566 <== mon.0 v2:172.21.15.118:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (secure 0 0 0) 0x7f351801b980 con 0x7f35280822f0 2024-03-24T00:04:01.408 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:04:00.686+0000 7f352d200700 1 -- 172.21.15.118:0/3631741566 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_command({"prefix": "config assimilate-conf"} v 0) v1 -- 0x7f35281a2520 con 0x7f35280822f0 2024-03-24T00:04:01.408 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:04:00.691+0000 7f351ea00700 1 -- 172.21.15.118:0/3631741566 <== mon.0 v2:172.21.15.118:3300/0 7 ==== mon_command_ack([{"prefix": "config assimilate-conf"}]=0 v2) v1 ==== 70+0+431 (secure 0 0 0) 0x7f35281a2520 con 0x7f35280822f0 2024-03-24T00:04:01.408 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:04:00.692+0000 7f351ea00700 1 -- 172.21.15.118:0/3631741566 <== mon.0 v2:172.21.15.118:3300/0 8 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f351801ada0 con 0x7f35280822f0 2024-03-24T00:04:01.408 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:04:00.695+0000 7f352d200700 1 -- 172.21.15.118:0/3631741566 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f35280822f0 msgr2=0x7f35281a1870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:04:01.408 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:04:00.695+0000 7f352d200700 1 --2- 172.21.15.118:0/3631741566 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f35280822f0 0x7f35281a1870 secure :-1 s=READY pgs=4 cs=0 l=1 rev1=1 crypto rx=0x7f3518000c00 tx=0x7f3518004750 comp rx=0 tx=0).stop 2024-03-24T00:04:01.408 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:04:00.695+0000 7f352d200700 1 -- 172.21.15.118:0/3631741566 shutdown_connections 2024-03-24T00:04:01.408 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:04:00.695+0000 7f352d200700 1 --2- 172.21.15.118:0/3631741566 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f35280822f0 0x7f35281a1870 unknown :-1 s=CLOSED pgs=4 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:04:01.408 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:04:00.695+0000 7f352d200700 1 -- 172.21.15.118:0/3631741566 >> 172.21.15.118:0/3631741566 conn(0x7f352807d220 msgr2=0x7f352807f4e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:04:01.408 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:04:00.695+0000 7f352d200700 1 -- 172.21.15.118:0/3631741566 shutdown_connections 2024-03-24T00:04:01.409 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:04:00.695+0000 7f352d200700 1 -- 172.21.15.118:0/3631741566 wait complete. 2024-03-24T00:04:01.409 INFO:teuthology.orchestra.run.smithi118.stdout:Generating new minimal ceph.conf... 2024-03-24T00:04:02.388 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:04:02 smithi118 ceph-mon[31168]: from='client.? 172.21.15.118:0/3469480472' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T00:04:02.676 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:04:02.001+0000 7ff74be00700 1 Processor -- start 2024-03-24T00:04:02.677 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:04:02.001+0000 7ff74be00700 1 -- start start 2024-03-24T00:04:02.677 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:04:02.002+0000 7ff74be00700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7ff7440822f0 0x7ff744110d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:04:02.677 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:04:02.002+0000 7ff74be00700 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff7441112d0 con 0x7ff7440822f0 2024-03-24T00:04:02.677 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:04:02.002+0000 7ff749400700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7ff7440822f0 0x7ff744110d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:04:02.677 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:04:02.002+0000 7ff749400700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7ff7440822f0 0x7ff744110d00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.118:3300/0 says I am v2:172.21.15.118:39818/0 (socket says 172.21.15.118:39818) 2024-03-24T00:04:02.678 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:04:02.002+0000 7ff749400700 1 -- 172.21.15.118:0/3557636111 learned_addr learned my addr 172.21.15.118:0/3557636111 (peer_addr_for_me v2:172.21.15.118:0/0) 2024-03-24T00:04:02.678 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:04:02.003+0000 7ff749400700 1 -- 172.21.15.118:0/3557636111 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff744111450 con 0x7ff7440822f0 2024-03-24T00:04:02.678 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:04:02.003+0000 7ff749400700 1 --2- 172.21.15.118:0/3557636111 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7ff7440822f0 0x7ff744110d00 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7ff7380098a0 tx=0x7ff738009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=67abf2b774e9c221 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:04:02.678 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:04:02.003+0000 7ff743e00700 1 -- 172.21.15.118:0/3557636111 <== mon.0 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7ff738004160 con 0x7ff7440822f0 2024-03-24T00:04:02.678 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:04:02.003+0000 7ff743e00700 1 -- 172.21.15.118:0/3557636111 <== mon.0 v2:172.21.15.118:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7ff738004300 con 0x7ff7440822f0 2024-03-24T00:04:02.678 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:04:02.003+0000 7ff743e00700 1 -- 172.21.15.118:0/3557636111 <== mon.0 v2:172.21.15.118:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7ff738011b50 con 0x7ff7440822f0 2024-03-24T00:04:02.678 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:04:02.004+0000 7ff74be00700 1 -- 172.21.15.118:0/3557636111 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7ff7440822f0 msgr2=0x7ff744110d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:04:02.679 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:04:02.004+0000 7ff74be00700 1 --2- 172.21.15.118:0/3557636111 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7ff7440822f0 0x7ff744110d00 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7ff7380098a0 tx=0x7ff738009bb0 comp rx=0 tx=0).stop 2024-03-24T00:04:02.679 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:04:02.004+0000 7ff74be00700 1 -- 172.21.15.118:0/3557636111 shutdown_connections 2024-03-24T00:04:02.679 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:04:02.004+0000 7ff74be00700 1 --2- 172.21.15.118:0/3557636111 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7ff7440822f0 0x7ff744110d00 unknown :-1 s=CLOSED pgs=5 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:04:02.679 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:04:02.004+0000 7ff74be00700 1 -- 172.21.15.118:0/3557636111 >> 172.21.15.118:0/3557636111 conn(0x7ff74407d220 msgr2=0x7ff74407f640 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:04:02.679 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:04:02.004+0000 7ff74be00700 1 -- 172.21.15.118:0/3557636111 shutdown_connections 2024-03-24T00:04:02.679 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:04:02.004+0000 7ff74be00700 1 -- 172.21.15.118:0/3557636111 wait complete. 2024-03-24T00:04:02.679 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:04:02.005+0000 7ff74be00700 1 Processor -- start 2024-03-24T00:04:02.679 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:04:02.005+0000 7ff74be00700 1 -- start start 2024-03-24T00:04:02.679 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:04:02.006+0000 7ff74be00700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7ff7440822f0 0x7ff7441a2110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:04:02.679 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:04:02.006+0000 7ff74be00700 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff7441124e0 con 0x7ff7440822f0 2024-03-24T00:04:02.679 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:04:02.006+0000 7ff749400700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7ff7440822f0 0x7ff7441a2110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:04:02.679 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:04:02.006+0000 7ff749400700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7ff7440822f0 0x7ff7441a2110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.118:3300/0 says I am v2:172.21.15.118:39834/0 (socket says 172.21.15.118:39834) 2024-03-24T00:04:02.679 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:04:02.006+0000 7ff749400700 1 -- 172.21.15.118:0/3469480472 learned_addr learned my addr 172.21.15.118:0/3469480472 (peer_addr_for_me v2:172.21.15.118:0/0) 2024-03-24T00:04:02.679 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:04:02.006+0000 7ff749400700 1 -- 172.21.15.118:0/3469480472 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff7441a2650 con 0x7ff7440822f0 2024-03-24T00:04:02.680 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:04:02.007+0000 7ff749400700 1 --2- 172.21.15.118:0/3469480472 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7ff7440822f0 0x7ff7441a2110 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7ff738011850 tx=0x7ff738011c10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:04:02.680 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:04:02.007+0000 7ff742000700 1 -- 172.21.15.118:0/3469480472 <== mon.0 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7ff738003af0 con 0x7ff7440822f0 2024-03-24T00:04:02.680 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:04:02.007+0000 7ff742000700 1 -- 172.21.15.118:0/3469480472 <== mon.0 v2:172.21.15.118:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7ff738020070 con 0x7ff7440822f0 2024-03-24T00:04:02.680 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:04:02.007+0000 7ff742000700 1 -- 172.21.15.118:0/3469480472 <== mon.0 v2:172.21.15.118:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7ff7380195a0 con 0x7ff7440822f0 2024-03-24T00:04:02.680 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:04:02.007+0000 7ff74be00700 1 -- 172.21.15.118:0/3469480472 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff7441a28e0 con 0x7ff7440822f0 2024-03-24T00:04:02.680 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:04:02.007+0000 7ff74be00700 1 -- 172.21.15.118:0/3469480472 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff7441a55d0 con 0x7ff7440822f0 2024-03-24T00:04:02.680 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:04:02.007+0000 7ff742000700 1 -- 172.21.15.118:0/3469480472 <== mon.0 v2:172.21.15.118:3300/0 4 ==== mgrmap(e 1) v1 ==== 807+0+0 (secure 0 0 0) 0x7ff738019be0 con 0x7ff7440822f0 2024-03-24T00:04:02.680 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:04:02.007+0000 7ff742000700 1 -- 172.21.15.118:0/3469480472 <== mon.0 v2:172.21.15.118:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (secure 0 0 0) 0x7ff738011e00 con 0x7ff7440822f0 2024-03-24T00:04:02.680 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:04:02.008+0000 7ff74be00700 1 -- 172.21.15.118:0/3469480472 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff7440637c0 con 0x7ff7440822f0 2024-03-24T00:04:02.680 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:04:02.011+0000 7ff742000700 1 -- 172.21.15.118:0/3469480472 <== mon.0 v2:172.21.15.118:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (secure 0 0 0) 0x7ff738003c90 con 0x7ff7440822f0 2024-03-24T00:04:02.680 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:04:02.087+0000 7ff74be00700 1 -- 172.21.15.118:0/3469480472 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_command({"prefix": "config generate-minimal-conf"} v 0) v1 -- 0x7ff7441a57a0 con 0x7ff7440822f0 2024-03-24T00:04:02.680 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:04:02.088+0000 7ff742000700 1 -- 172.21.15.118:0/3469480472 <== mon.0 v2:172.21.15.118:3300/0 7 ==== mon_command_ack([{"prefix": "config generate-minimal-conf"}]=0 v2) v1 ==== 76+0+177 (secure 0 0 0) 0x7ff738025460 con 0x7ff7440822f0 2024-03-24T00:04:02.680 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:04:02.090+0000 7ff74be00700 1 -- 172.21.15.118:0/3469480472 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7ff7440822f0 msgr2=0x7ff7441a2110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:04:02.681 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:04:02.090+0000 7ff74be00700 1 --2- 172.21.15.118:0/3469480472 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7ff7440822f0 0x7ff7441a2110 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7ff738011850 tx=0x7ff738011c10 comp rx=0 tx=0).stop 2024-03-24T00:04:02.681 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:04:02.090+0000 7ff74be00700 1 -- 172.21.15.118:0/3469480472 shutdown_connections 2024-03-24T00:04:02.681 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:04:02.090+0000 7ff74be00700 1 --2- 172.21.15.118:0/3469480472 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7ff7440822f0 0x7ff7441a2110 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:04:02.681 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:04:02.090+0000 7ff74be00700 1 -- 172.21.15.118:0/3469480472 >> 172.21.15.118:0/3469480472 conn(0x7ff74407d220 msgr2=0x7ff74407de00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:04:02.681 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:04:02.090+0000 7ff74be00700 1 -- 172.21.15.118:0/3469480472 shutdown_connections 2024-03-24T00:04:02.681 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:04:02.090+0000 7ff74be00700 1 -- 172.21.15.118:0/3469480472 wait complete. 2024-03-24T00:04:02.681 INFO:teuthology.orchestra.run.smithi118.stdout:Restarting the monitor... 2024-03-24T00:04:02.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:04:02 smithi118 systemd[1]: Stopping Ceph mon.a for d79a5420-e971-11ee-95cd-87774f69a715... 2024-03-24T00:04:03.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:04:03 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mon-a[31164]: 2024-03-24T00:04:03.135+0000 7f06e3800700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.a -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-03-24T00:04:03.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:04:03 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mon-a[31164]: 2024-03-24T00:04:03.135+0000 7f06e3800700 -1 mon.a@0(leader) e1 *** Got Signal Terminated *** 2024-03-24T00:04:03.447 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:04:03 smithi118 podman[31409]: 2024-03-24 00:04:03.291318155 +0000 UTC m=+0.280035834 container died 50ce263b4def37e26d2d8314244644a5602d5025e991840dc673ee920ede8788 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033, name=ceph-d79a5420-e971-11ee-95cd-87774f69a715-mon-a, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240319, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.license=GPLv2, ceph=True, org.label-schema.vendor=CentOS) 2024-03-24T00:04:03.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:04:03 smithi118 podman[31409]: 2024-03-24 00:04:03.696435566 +0000 UTC m=+0.685153255 container cleanup 50ce263b4def37e26d2d8314244644a5602d5025e991840dc673ee920ede8788 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033, name=ceph-d79a5420-e971-11ee-95cd-87774f69a715-mon-a, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, org.label-schema.vendor=CentOS, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240319, GIT_CLEAN=True, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0) 2024-03-24T00:04:03.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:04:03 smithi118 bash[31409]: ceph-d79a5420-e971-11ee-95cd-87774f69a715-mon-a 2024-03-24T00:04:04.202 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:04:03 smithi118 podman[31422]: 2024-03-24 00:04:03.947279401 +0000 UTC m=+0.642883900 container remove 50ce263b4def37e26d2d8314244644a5602d5025e991840dc673ee920ede8788 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033, name=ceph-d79a5420-e971-11ee-95cd-87774f69a715-mon-a, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.vendor=CentOS, GIT_CLEAN=True, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240319) 2024-03-24T00:04:04.588 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:04:04 smithi118 systemd[1]: ceph-d79a5420-e971-11ee-95cd-87774f69a715@mon.a.service: Deactivated successfully. 2024-03-24T00:04:04.588 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:04:04 smithi118 systemd[1]: Stopped Ceph mon.a for d79a5420-e971-11ee-95cd-87774f69a715. 2024-03-24T00:04:04.588 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:04:04 smithi118 systemd[1]: ceph-d79a5420-e971-11ee-95cd-87774f69a715@mon.a.service: Consumed 1.723s CPU time. 2024-03-24T00:04:04.588 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:04:04 smithi118 systemd[1]: Starting Ceph mon.a for d79a5420-e971-11ee-95cd-87774f69a715... 2024-03-24T00:04:04.875 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:04:04 smithi118 podman[31516]: 2024-03-24 00:04:04.494656143 +0000 UTC m=+0.033607008 image pull c664afe56bb01db4f9afa9532c20598e201e664f2e9ba2be50db0d52674a26d2 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 2024-03-24T00:04:04.875 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:04:04 smithi118 podman[31516]: 2024-03-24 00:04:04.589707238 +0000 UTC m=+0.128658119 container create 9a3b56cb7cc8e0313c3963fe3db6b16bb1ecf7d71ab784c3a43ea706ab1ec49b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033, name=ceph-d79a5420-e971-11ee-95cd-87774f69a715-mon-a, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240319, GIT_BRANCH=HEAD, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, ceph=True, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.schema-version=1.0) 2024-03-24T00:04:04.875 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:04:04 smithi118 podman[31516]: 2024-03-24 00:04:04.818303829 +0000 UTC m=+0.357254737 container init 9a3b56cb7cc8e0313c3963fe3db6b16bb1ecf7d71ab784c3a43ea706ab1ec49b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033, name=ceph-d79a5420-e971-11ee-95cd-87774f69a715-mon-a, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, GIT_CLEAN=True, org.label-schema.build-date=20240319, org.label-schema.vendor=CentOS, ceph=True, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2) 2024-03-24T00:04:04.875 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:04:04 smithi118 podman[31516]: 2024-03-24 00:04:04.828214332 +0000 UTC m=+0.367165240 container start 9a3b56cb7cc8e0313c3963fe3db6b16bb1ecf7d71ab784c3a43ea706ab1ec49b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033, name=ceph-d79a5420-e971-11ee-95cd-87774f69a715-mon-a, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240319, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-24T00:04:04.875 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:04:04 smithi118 ceph-mon[31535]: set uid:gid to 167:167 (ceph:ceph) 2024-03-24T00:04:04.875 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:04:04 smithi118 ceph-mon[31535]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev), process ceph-mon, pid 2 2024-03-24T00:04:04.919 INFO:teuthology.orchestra.run.smithi118.stdout:Setting public_network to 172.21.0.0/20 in mon config section 2024-03-24T00:04:05.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:04:04 smithi118 ceph-mon[31535]: pidfile_write: ignore empty --pid-file 2024-03-24T00:04:05.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:04:04 smithi118 ceph-mon[31535]: load: jerasure load: lrc 2024-03-24T00:04:05.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:04:04 smithi118 ceph-mon[31535]: rocksdb: RocksDB version: 7.9.2 2024-03-24T00:04:05.197 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:04:04 smithi118 ceph-mon[31535]: rocksdb: Git sha 0 2024-03-24T00:04:05.197 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:04:04 smithi118 ceph-mon[31535]: rocksdb: Compile date 2024-03-22 04:58:55 2024-03-24T00:04:05.197 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:04:04 smithi118 ceph-mon[31535]: rocksdb: DB SUMMARY 2024-03-24T00:04:05.197 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:04:04 smithi118 ceph-mon[31535]: rocksdb: DB Session ID: 89JZQ24IUDZRLV2AQE2K 2024-03-24T00:04:05.197 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:04:04 smithi118 ceph-mon[31535]: rocksdb: CURRENT file: CURRENT 2024-03-24T00:04:05.197 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:04:04 smithi118 ceph-mon[31535]: rocksdb: IDENTITY file: IDENTITY 2024-03-24T00:04:05.197 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:04:04 smithi118 ceph-mon[31535]: rocksdb: MANIFEST file: MANIFEST-000010 size: 179 Bytes 2024-03-24T00:04:05.197 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:04:04 smithi118 ceph-mon[31535]: rocksdb: SST files in /var/lib/ceph/mon/ceph-a/store.db dir, Total Num: 1, files: 000008.sst 2024-03-24T00:04:05.197 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:04:04 smithi118 ceph-mon[31535]: rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-a/store.db: 000009.log size: 100776 ; 2024-03-24T00:04:05.197 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:04:04 smithi118 ceph-mon[31535]: rocksdb: Options.error_if_exists: 0 2024-03-24T00:04:05.197 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:04:04 smithi118 ceph-mon[31535]: rocksdb: Options.create_if_missing: 0 2024-03-24T00:04:05.197 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:04:04 smithi118 ceph-mon[31535]: rocksdb: Options.paranoid_checks: 1 2024-03-24T00:04:05.197 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:04:04 smithi118 ceph-mon[31535]: rocksdb: Options.flush_verify_memtable_count: 1 2024-03-24T00:04:05.197 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:04:04 smithi118 ceph-mon[31535]: rocksdb: Options.track_and_verify_wals_in_manifest: 0 2024-03-24T00:04:05.198 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:04:04 smithi118 ceph-mon[31535]: rocksdb: Options.verify_sst_unique_id_in_manifest: 1 2024-03-24T00:04:05.198 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:04:04 smithi118 ceph-mon[31535]: rocksdb: Options.env: 0x5578dee66b20 2024-03-24T00:04:05.198 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:04:04 smithi118 ceph-mon[31535]: rocksdb: Options.fs: PosixFileSystem 2024-03-24T00:04:05.198 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:04:04 smithi118 ceph-mon[31535]: rocksdb: Options.info_log: 0x5578e132b960 2024-03-24T00:04:05.198 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:04:04 smithi118 ceph-mon[31535]: rocksdb: Options.max_file_opening_threads: 16 2024-03-24T00:04:05.198 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:04:04 smithi118 ceph-mon[31535]: rocksdb: Options.statistics: (nil) 2024-03-24T00:04:05.198 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:04:04 smithi118 ceph-mon[31535]: rocksdb: Options.use_fsync: 0 2024-03-24T00:04:05.198 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:04:04 smithi118 ceph-mon[31535]: rocksdb: Options.max_log_file_size: 0 2024-03-24T00:04:05.198 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:04:04 smithi118 ceph-mon[31535]: rocksdb: Options.max_manifest_file_size: 1073741824 2024-03-24T00:04:05.198 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:04:04 smithi118 ceph-mon[31535]: rocksdb: Options.log_file_time_to_roll: 0 2024-03-24T00:04:05.198 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:04:04 smithi118 ceph-mon[31535]: rocksdb: Options.keep_log_file_num: 1000 2024-03-24T00:04:05.198 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:04:04 smithi118 ceph-mon[31535]: rocksdb: Options.recycle_log_file_num: 0 2024-03-24T00:04:05.198 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:04:04 smithi118 ceph-mon[31535]: rocksdb: Options.allow_fallocate: 1 2024-03-24T00:04:05.199 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:04:04 smithi118 ceph-mon[31535]: rocksdb: Options.allow_mmap_reads: 0 2024-03-24T00:04:05.199 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:04:04 smithi118 ceph-mon[31535]: rocksdb: Options.allow_mmap_writes: 0 2024-03-24T00:04:05.199 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:04:04 smithi118 ceph-mon[31535]: rocksdb: Options.use_direct_reads: 0 2024-03-24T00:04:05.199 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:04:04 smithi118 ceph-mon[31535]: rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2024-03-24T00:04:05.199 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:04:04 smithi118 ceph-mon[31535]: rocksdb: Options.create_missing_column_families: 0 2024-03-24T00:04:05.199 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:04:04 smithi118 ceph-mon[31535]: rocksdb: Options.db_log_dir: 2024-03-24T00:04:05.199 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:04:04 smithi118 ceph-mon[31535]: rocksdb: Options.wal_dir: 2024-03-24T00:04:05.199 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:04:04 smithi118 ceph-mon[31535]: rocksdb: Options.table_cache_numshardbits: 6 2024-03-24T00:04:05.199 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:04:04 smithi118 ceph-mon[31535]: rocksdb: Options.WAL_ttl_seconds: 0 2024-03-24T00:04:05.199 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:04:04 smithi118 ceph-mon[31535]: rocksdb: Options.WAL_size_limit_MB: 0 2024-03-24T00:04:05.200 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:04:04 smithi118 ceph-mon[31535]: rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2024-03-24T00:04:05.201 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:04:04 smithi118 ceph-mon[31535]: rocksdb: Options.manifest_preallocation_size: 4194304 2024-03-24T00:04:05.201 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:04:04 smithi118 ceph-mon[31535]: rocksdb: Options.is_fd_close_on_exec: 1 2024-03-24T00:04:05.201 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:04:04 smithi118 ceph-mon[31535]: rocksdb: Options.advise_random_on_open: 1 2024-03-24T00:04:05.201 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:04:04 smithi118 ceph-mon[31535]: rocksdb: Options.db_write_buffer_size: 0 2024-03-24T00:04:05.201 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:04:04 smithi118 ceph-mon[31535]: rocksdb: Options.write_buffer_manager: 0x5578e12fe1e0 2024-03-24T00:04:05.201 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:04:04 smithi118 ceph-mon[31535]: rocksdb: Options.access_hint_on_compaction_start: 1 2024-03-24T00:04:05.201 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:04:04 smithi118 ceph-mon[31535]: rocksdb: Options.random_access_max_buffer_size: 1048576 2024-03-24T00:04:05.201 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:04:04 smithi118 ceph-mon[31535]: rocksdb: Options.use_adaptive_mutex: 0 2024-03-24T00:04:05.201 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:04:04 smithi118 ceph-mon[31535]: rocksdb: Options.rate_limiter: (nil) 2024-03-24T00:04:05.201 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:04:04 smithi118 ceph-mon[31535]: rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2024-03-24T00:04:05.201 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:04:04 smithi118 ceph-mon[31535]: rocksdb: Options.wal_recovery_mode: 2 2024-03-24T00:04:05.201 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:04:04 smithi118 ceph-mon[31535]: rocksdb: Options.enable_thread_tracking: 0 2024-03-24T00:04:05.202 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:04:04 smithi118 ceph-mon[31535]: rocksdb: Options.enable_pipelined_write: 0 2024-03-24T00:04:05.202 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:04:04 smithi118 ceph-mon[31535]: rocksdb: Options.unordered_write: 0 2024-03-24T00:04:05.202 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:04:04 smithi118 ceph-mon[31535]: rocksdb: Options.allow_concurrent_memtable_write: 1 2024-03-24T00:04:05.202 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:04:04 smithi118 ceph-mon[31535]: rocksdb: Options.enable_write_thread_adaptive_yield: 1 2024-03-24T00:04:05.202 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:04:04 smithi118 ceph-mon[31535]: rocksdb: Options.write_thread_max_yield_usec: 100 2024-03-24T00:04:05.202 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:04:04 smithi118 ceph-mon[31535]: rocksdb: Options.write_thread_slow_yield_usec: 3 2024-03-24T00:04:05.202 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:04:04 smithi118 ceph-mon[31535]: rocksdb: Options.row_cache: None 2024-03-24T00:04:05.202 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:04:04 smithi118 ceph-mon[31535]: rocksdb: Options.wal_filter: None 2024-03-24T00:04:05.202 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:04:04 smithi118 ceph-mon[31535]: rocksdb: Options.avoid_flush_during_recovery: 0 2024-03-24T00:04:05.202 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:04:04 smithi118 ceph-mon[31535]: rocksdb: Options.allow_ingest_behind: 0 2024-03-24T00:04:05.202 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:04:04 smithi118 ceph-mon[31535]: rocksdb: Options.two_write_queues: 0 2024-03-24T00:04:05.203 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:04:04 smithi118 ceph-mon[31535]: rocksdb: Options.manual_wal_flush: 0 2024-03-24T00:04:05.203 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:04:04 smithi118 ceph-mon[31535]: rocksdb: Options.wal_compression: 0 2024-03-24T00:04:05.203 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:04:04 smithi118 ceph-mon[31535]: rocksdb: Options.atomic_flush: 0 2024-03-24T00:04:05.203 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:04:04 smithi118 ceph-mon[31535]: rocksdb: Options.avoid_unnecessary_blocking_io: 0 2024-03-24T00:04:05.203 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:04:04 smithi118 ceph-mon[31535]: rocksdb: Options.persist_stats_to_disk: 0 2024-03-24T00:04:05.203 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:04:04 smithi118 ceph-mon[31535]: rocksdb: Options.write_dbid_to_manifest: 0 2024-03-24T00:04:05.203 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:04:04 smithi118 ceph-mon[31535]: rocksdb: Options.log_readahead_size: 0 2024-03-24T00:04:05.203 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:04:04 smithi118 ceph-mon[31535]: rocksdb: Options.file_checksum_gen_factory: Unknown 2024-03-24T00:04:05.203 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:04:04 smithi118 ceph-mon[31535]: rocksdb: Options.best_efforts_recovery: 0 2024-03-24T00:04:05.203 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:04:04 smithi118 ceph-mon[31535]: rocksdb: Options.max_bgerror_resume_count: 2147483647 2024-03-24T00:04:05.203 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:04:04 smithi118 ceph-mon[31535]: rocksdb: Options.bgerror_resume_retry_interval: 1000000 2024-03-24T00:04:05.203 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:04:04 smithi118 ceph-mon[31535]: rocksdb: Options.allow_data_in_errors: 0 2024-03-24T00:04:05.203 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:04:04 smithi118 ceph-mon[31535]: rocksdb: Options.db_host_id: __hostname__ 2024-03-24T00:04:05.204 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:04:04 smithi118 ceph-mon[31535]: rocksdb: Options.enforce_single_del_contracts: true 2024-03-24T00:04:05.204 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:04:04 smithi118 ceph-mon[31535]: rocksdb: Options.max_background_jobs: 2 2024-03-24T00:04:05.204 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:04:04 smithi118 ceph-mon[31535]: rocksdb: Options.max_background_compactions: -1 2024-03-24T00:04:05.204 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:04:04 smithi118 ceph-mon[31535]: rocksdb: Options.max_subcompactions: 1 2024-03-24T00:04:05.204 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:04:04 smithi118 ceph-mon[31535]: rocksdb: Options.avoid_flush_during_shutdown: 0 2024-03-24T00:04:05.204 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:04:04 smithi118 ceph-mon[31535]: rocksdb: Options.writable_file_max_buffer_size: 1048576 2024-03-24T00:04:05.204 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:04:04 smithi118 ceph-mon[31535]: rocksdb: Options.delayed_write_rate : 16777216 2024-03-24T00:04:05.204 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:04:04 smithi118 ceph-mon[31535]: rocksdb: Options.max_total_wal_size: 0 2024-03-24T00:04:05.204 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:04:04 smithi118 ceph-mon[31535]: rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2024-03-24T00:04:05.204 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:04:04 smithi118 ceph-mon[31535]: rocksdb: Options.stats_dump_period_sec: 600 2024-03-24T00:04:05.204 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:04:04 smithi118 ceph-mon[31535]: rocksdb: Options.stats_persist_period_sec: 600 2024-03-24T00:04:05.204 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:04:04 smithi118 ceph-mon[31535]: rocksdb: Options.stats_history_buffer_size: 1048576 2024-03-24T00:04:05.204 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:04:04 smithi118 ceph-mon[31535]: rocksdb: Options.max_open_files: -1 2024-03-24T00:04:05.204 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:04:04 smithi118 ceph-mon[31535]: rocksdb: Options.bytes_per_sync: 0 2024-03-24T00:04:05.205 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:04:04 smithi118 ceph-mon[31535]: rocksdb: Options.wal_bytes_per_sync: 0 2024-03-24T00:04:05.205 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:04:04 smithi118 ceph-mon[31535]: rocksdb: Options.strict_bytes_per_sync: 0 2024-03-24T00:04:05.205 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:04:04 smithi118 ceph-mon[31535]: rocksdb: Options.compaction_readahead_size: 0 2024-03-24T00:04:05.205 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:04:04 smithi118 ceph-mon[31535]: rocksdb: Options.max_background_flushes: -1 2024-03-24T00:04:05.205 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:04:04 smithi118 ceph-mon[31535]: rocksdb: Compression algorithms supported: 2024-03-24T00:04:05.205 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:04:04 smithi118 ceph-mon[31535]: rocksdb: kZSTDNotFinalCompression supported: 0 2024-03-24T00:04:05.205 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:04:04 smithi118 ceph-mon[31535]: rocksdb: kZSTD supported: 0 2024-03-24T00:04:05.205 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:04:04 smithi118 ceph-mon[31535]: rocksdb: kXpressCompression supported: 0 2024-03-24T00:04:05.205 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:04:04 smithi118 ceph-mon[31535]: rocksdb: kLZ4HCCompression supported: 1 2024-03-24T00:04:05.205 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:04:04 smithi118 ceph-mon[31535]: rocksdb: kZlibCompression supported: 1 2024-03-24T00:04:05.205 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:04:04 smithi118 ceph-mon[31535]: rocksdb: kSnappyCompression supported: 1 2024-03-24T00:04:05.205 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:04:04 smithi118 ceph-mon[31535]: rocksdb: kLZ4Compression supported: 1 2024-03-24T00:04:05.205 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:04:04 smithi118 ceph-mon[31535]: rocksdb: kBZip2Compression supported: 0 2024-03-24T00:04:05.205 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:04:04 smithi118 ceph-mon[31535]: rocksdb: Fast CRC32 supported: Supported on x86 2024-03-24T00:04:05.206 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:04:04 smithi118 ceph-mon[31535]: rocksdb: DMutex implementation: pthread_mutex_t 2024-03-24T00:04:05.206 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:04:04 smithi118 ceph-mon[31535]: rocksdb: [db/version_set.cc:5527] Recovering from manifest file: /var/lib/ceph/mon/ceph-a/store.db/MANIFEST-000010 2024-03-24T00:04:05.206 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:04:04 smithi118 ceph-mon[31535]: rocksdb: [db/column_family.cc:630] --------------- Options for column family [default]: 2024-03-24T00:04:05.206 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:04:04 smithi118 ceph-mon[31535]: rocksdb: Options.comparator: leveldb.BytewiseComparator 2024-03-24T00:04:05.206 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:04:04 smithi118 ceph-mon[31535]: rocksdb: Options.merge_operator: 2024-03-24T00:04:05.206 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:04:04 smithi118 ceph-mon[31535]: rocksdb: Options.compaction_filter: None 2024-03-24T00:04:05.206 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:04:04 smithi118 ceph-mon[31535]: rocksdb: Options.compaction_filter_factory: None 2024-03-24T00:04:05.206 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:04:04 smithi118 ceph-mon[31535]: rocksdb: Options.sst_partitioner_factory: None 2024-03-24T00:04:05.206 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:04:04 smithi118 ceph-mon[31535]: rocksdb: Options.memtable_factory: SkipListFactory 2024-03-24T00:04:05.206 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:04:04 smithi118 ceph-mon[31535]: rocksdb: Options.table_factory: BlockBasedTable 2024-03-24T00:04:05.206 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:04:04 smithi118 ceph-mon[31535]: rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x5578e132ba60) 2024-03-24T00:04:05.206 INFO:journalctl@ceph.mon.a.smithi118.stdout: cache_index_and_filter_blocks: 1 2024-03-24T00:04:05.206 INFO:journalctl@ceph.mon.a.smithi118.stdout: cache_index_and_filter_blocks_with_high_priority: 0 2024-03-24T00:04:05.207 INFO:journalctl@ceph.mon.a.smithi118.stdout: pin_l0_filter_and_index_blocks_in_cache: 0 2024-03-24T00:04:05.207 INFO:journalctl@ceph.mon.a.smithi118.stdout: pin_top_level_index_and_filter: 1 2024-03-24T00:04:05.207 INFO:journalctl@ceph.mon.a.smithi118.stdout: index_type: 0 2024-03-24T00:04:05.207 INFO:journalctl@ceph.mon.a.smithi118.stdout: data_block_index_type: 0 2024-03-24T00:04:05.207 INFO:journalctl@ceph.mon.a.smithi118.stdout: index_shortening: 1 2024-03-24T00:04:05.207 INFO:journalctl@ceph.mon.a.smithi118.stdout: data_block_hash_table_util_ratio: 0.750000 2024-03-24T00:04:05.207 INFO:journalctl@ceph.mon.a.smithi118.stdout: checksum: 4 2024-03-24T00:04:05.207 INFO:journalctl@ceph.mon.a.smithi118.stdout: no_block_cache: 0 2024-03-24T00:04:05.207 INFO:journalctl@ceph.mon.a.smithi118.stdout: block_cache: 0x5578e1331090 2024-03-24T00:04:05.207 INFO:journalctl@ceph.mon.a.smithi118.stdout: block_cache_name: BinnedLRUCache 2024-03-24T00:04:05.207 INFO:journalctl@ceph.mon.a.smithi118.stdout: block_cache_options: 2024-03-24T00:04:05.207 INFO:journalctl@ceph.mon.a.smithi118.stdout: capacity : 536870912 2024-03-24T00:04:05.207 INFO:journalctl@ceph.mon.a.smithi118.stdout: num_shard_bits : 4 2024-03-24T00:04:05.207 INFO:journalctl@ceph.mon.a.smithi118.stdout: strict_capacity_limit : 0 2024-03-24T00:04:05.208 INFO:journalctl@ceph.mon.a.smithi118.stdout: high_pri_pool_ratio: 0.000 2024-03-24T00:04:05.208 INFO:journalctl@ceph.mon.a.smithi118.stdout: block_cache_compressed: (nil) 2024-03-24T00:04:05.208 INFO:journalctl@ceph.mon.a.smithi118.stdout: persistent_cache: (nil) 2024-03-24T00:04:05.208 INFO:journalctl@ceph.mon.a.smithi118.stdout: block_size: 4096 2024-03-24T00:04:05.208 INFO:journalctl@ceph.mon.a.smithi118.stdout: block_size_deviation: 10 2024-03-24T00:04:05.208 INFO:journalctl@ceph.mon.a.smithi118.stdout: block_restart_interval: 16 2024-03-24T00:04:05.208 INFO:journalctl@ceph.mon.a.smithi118.stdout: index_block_restart_interval: 1 2024-03-24T00:04:05.208 INFO:journalctl@ceph.mon.a.smithi118.stdout: metadata_block_size: 4096 2024-03-24T00:04:05.208 INFO:journalctl@ceph.mon.a.smithi118.stdout: partition_filters: 0 2024-03-24T00:04:05.208 INFO:journalctl@ceph.mon.a.smithi118.stdout: use_delta_encoding: 1 2024-03-24T00:04:05.208 INFO:journalctl@ceph.mon.a.smithi118.stdout: filter_policy: bloomfilter 2024-03-24T00:04:05.208 INFO:journalctl@ceph.mon.a.smithi118.stdout: whole_key_filtering: 1 2024-03-24T00:04:05.208 INFO:journalctl@ceph.mon.a.smithi118.stdout: verify_compression: 0 2024-03-24T00:04:05.208 INFO:journalctl@ceph.mon.a.smithi118.stdout: read_amp_bytes_per_bit: 0 2024-03-24T00:04:05.208 INFO:journalctl@ceph.mon.a.smithi118.stdout: format_version: 5 2024-03-24T00:04:05.209 INFO:journalctl@ceph.mon.a.smithi118.stdout: enable_index_compression: 1 2024-03-24T00:04:05.209 INFO:journalctl@ceph.mon.a.smithi118.stdout: block_align: 0 2024-03-24T00:04:05.209 INFO:journalctl@ceph.mon.a.smithi118.stdout: max_auto_readahead_size: 262144 2024-03-24T00:04:05.209 INFO:journalctl@ceph.mon.a.smithi118.stdout: prepopulate_block_cache: 0 2024-03-24T00:04:05.209 INFO:journalctl@ceph.mon.a.smithi118.stdout: initial_auto_readahead_size: 8192 2024-03-24T00:04:05.209 INFO:journalctl@ceph.mon.a.smithi118.stdout: num_file_reads_for_auto_readahead: 2 2024-03-24T00:04:05.209 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:04:04 smithi118 ceph-mon[31535]: rocksdb: Options.write_buffer_size: 33554432 2024-03-24T00:04:05.209 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:04:04 smithi118 ceph-mon[31535]: rocksdb: Options.max_write_buffer_number: 2 2024-03-24T00:04:05.209 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:04:04 smithi118 ceph-mon[31535]: rocksdb: Options.compression: NoCompression 2024-03-24T00:04:05.209 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:04:04 smithi118 ceph-mon[31535]: rocksdb: Options.bottommost_compression: Disabled 2024-03-24T00:04:05.209 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:04:04 smithi118 ceph-mon[31535]: rocksdb: Options.prefix_extractor: nullptr 2024-03-24T00:04:05.209 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:04:04 smithi118 ceph-mon[31535]: rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2024-03-24T00:04:05.209 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:04:04 smithi118 ceph-mon[31535]: rocksdb: Options.num_levels: 7 2024-03-24T00:04:05.209 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:04:04 smithi118 ceph-mon[31535]: rocksdb: Options.min_write_buffer_number_to_merge: 1 2024-03-24T00:04:05.210 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:04:04 smithi118 ceph-mon[31535]: rocksdb: Options.max_write_buffer_number_to_maintain: 0 2024-03-24T00:04:05.210 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:04:04 smithi118 ceph-mon[31535]: rocksdb: Options.max_write_buffer_size_to_maintain: 0 2024-03-24T00:04:05.210 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:04:04 smithi118 ceph-mon[31535]: rocksdb: Options.bottommost_compression_opts.window_bits: -14 2024-03-24T00:04:05.210 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:04:04 smithi118 ceph-mon[31535]: rocksdb: Options.bottommost_compression_opts.level: 32767 2024-03-24T00:04:05.210 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:04:04 smithi118 ceph-mon[31535]: rocksdb: Options.bottommost_compression_opts.strategy: 0 2024-03-24T00:04:05.210 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:04:04 smithi118 ceph-mon[31535]: rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2024-03-24T00:04:05.210 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:04:04 smithi118 ceph-mon[31535]: rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2024-03-24T00:04:05.210 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:04:04 smithi118 ceph-mon[31535]: rocksdb: Options.bottommost_compression_opts.parallel_threads: 1 2024-03-24T00:04:05.210 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:04:04 smithi118 ceph-mon[31535]: rocksdb: Options.bottommost_compression_opts.enabled: false 2024-03-24T00:04:05.210 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:04:04 smithi118 ceph-mon[31535]: rocksdb: Options.bottommost_compression_opts.max_dict_buffer_bytes: 0 2024-03-24T00:04:05.210 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:04:04 smithi118 ceph-mon[31535]: rocksdb: Options.bottommost_compression_opts.use_zstd_dict_trainer: true 2024-03-24T00:04:05.210 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:04:04 smithi118 ceph-mon[31535]: rocksdb: Options.compression_opts.window_bits: -14 2024-03-24T00:04:05.210 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:04:04 smithi118 ceph-mon[31535]: rocksdb: Options.compression_opts.level: 32767 2024-03-24T00:04:05.210 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:04:04 smithi118 ceph-mon[31535]: rocksdb: Options.compression_opts.strategy: 0 2024-03-24T00:04:05.211 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:04:04 smithi118 ceph-mon[31535]: rocksdb: Options.compression_opts.max_dict_bytes: 0 2024-03-24T00:04:05.211 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:04:04 smithi118 ceph-mon[31535]: rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2024-03-24T00:04:05.211 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:04:04 smithi118 ceph-mon[31535]: rocksdb: Options.compression_opts.use_zstd_dict_trainer: true 2024-03-24T00:04:05.211 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:04:04 smithi118 ceph-mon[31535]: rocksdb: Options.compression_opts.parallel_threads: 1 2024-03-24T00:04:05.211 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:04:04 smithi118 ceph-mon[31535]: rocksdb: Options.compression_opts.enabled: false 2024-03-24T00:04:05.211 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:04:04 smithi118 ceph-mon[31535]: rocksdb: Options.compression_opts.max_dict_buffer_bytes: 0 2024-03-24T00:04:05.211 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:04:04 smithi118 ceph-mon[31535]: rocksdb: Options.level0_file_num_compaction_trigger: 4 2024-03-24T00:04:05.211 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:04:04 smithi118 ceph-mon[31535]: rocksdb: Options.level0_slowdown_writes_trigger: 20 2024-03-24T00:04:05.211 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:04:04 smithi118 ceph-mon[31535]: rocksdb: Options.level0_stop_writes_trigger: 36 2024-03-24T00:04:05.211 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:04:04 smithi118 ceph-mon[31535]: rocksdb: Options.target_file_size_base: 67108864 2024-03-24T00:04:05.211 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:04:04 smithi118 ceph-mon[31535]: rocksdb: Options.target_file_size_multiplier: 1 2024-03-24T00:04:05.211 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:04:04 smithi118 ceph-mon[31535]: rocksdb: Options.max_bytes_for_level_base: 268435456 2024-03-24T00:04:05.211 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:04:04 smithi118 ceph-mon[31535]: rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2024-03-24T00:04:05.211 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:04:04 smithi118 ceph-mon[31535]: rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2024-03-24T00:04:05.211 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:04:04 smithi118 ceph-mon[31535]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2024-03-24T00:04:05.212 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:04:04 smithi118 ceph-mon[31535]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2024-03-24T00:04:05.212 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:04:04 smithi118 ceph-mon[31535]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2024-03-24T00:04:05.212 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:04:04 smithi118 ceph-mon[31535]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2024-03-24T00:04:05.212 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:04:04 smithi118 ceph-mon[31535]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2024-03-24T00:04:05.212 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:04:04 smithi118 ceph-mon[31535]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2024-03-24T00:04:05.212 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:04:04 smithi118 ceph-mon[31535]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2024-03-24T00:04:05.212 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:04:04 smithi118 ceph-mon[31535]: rocksdb: Options.max_sequential_skip_in_iterations: 8 2024-03-24T00:04:05.212 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:04:04 smithi118 ceph-mon[31535]: rocksdb: Options.max_compaction_bytes: 1677721600 2024-03-24T00:04:05.212 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:04:04 smithi118 ceph-mon[31535]: rocksdb: Options.ignore_max_compaction_bytes_for_input: true 2024-03-24T00:04:05.212 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:04:04 smithi118 ceph-mon[31535]: rocksdb: Options.arena_block_size: 1048576 2024-03-24T00:04:05.212 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:04:04 smithi118 ceph-mon[31535]: rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2024-03-24T00:04:05.212 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:04:04 smithi118 ceph-mon[31535]: rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2024-03-24T00:04:05.212 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:04:04 smithi118 ceph-mon[31535]: rocksdb: Options.disable_auto_compactions: 0 2024-03-24T00:04:05.212 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:04:04 smithi118 ceph-mon[31535]: rocksdb: Options.compaction_style: kCompactionStyleLevel 2024-03-24T00:04:05.212 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:04:04 smithi118 ceph-mon[31535]: rocksdb: Options.compaction_pri: kMinOverlappingRatio 2024-03-24T00:04:05.213 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:04:04 smithi118 ceph-mon[31535]: rocksdb: Options.compaction_options_universal.size_ratio: 1 2024-03-24T00:04:05.213 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:04:04 smithi118 ceph-mon[31535]: rocksdb: Options.compaction_options_universal.min_merge_width: 2 2024-03-24T00:04:05.213 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:04:04 smithi118 ceph-mon[31535]: rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2024-03-24T00:04:05.213 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:04:04 smithi118 ceph-mon[31535]: rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2024-03-24T00:04:05.213 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:04:04 smithi118 ceph-mon[31535]: rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2024-03-24T00:04:05.213 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:04:04 smithi118 ceph-mon[31535]: rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2024-03-24T00:04:05.213 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:04:04 smithi118 ceph-mon[31535]: rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2024-03-24T00:04:05.213 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:04:04 smithi118 ceph-mon[31535]: rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2024-03-24T00:04:05.213 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:04:04 smithi118 ceph-mon[31535]: rocksdb: Options.table_properties_collectors: 2024-03-24T00:04:05.213 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:04:04 smithi118 ceph-mon[31535]: rocksdb: Options.inplace_update_support: 0 2024-03-24T00:04:05.213 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:04:04 smithi118 ceph-mon[31535]: rocksdb: Options.inplace_update_num_locks: 10000 2024-03-24T00:04:05.213 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:04:04 smithi118 ceph-mon[31535]: rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2024-03-24T00:04:05.213 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:04:04 smithi118 ceph-mon[31535]: rocksdb: Options.memtable_whole_key_filtering: 0 2024-03-24T00:04:05.213 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:04:04 smithi118 ceph-mon[31535]: rocksdb: Options.memtable_huge_page_size: 0 2024-03-24T00:04:05.213 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:04:04 smithi118 ceph-mon[31535]: rocksdb: Options.bloom_locality: 0 2024-03-24T00:04:05.214 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:04:04 smithi118 ceph-mon[31535]: rocksdb: Options.max_successive_merges: 0 2024-03-24T00:04:05.214 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:04:04 smithi118 ceph-mon[31535]: rocksdb: Options.optimize_filters_for_hits: 0 2024-03-24T00:04:05.214 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:04:04 smithi118 ceph-mon[31535]: rocksdb: Options.paranoid_file_checks: 0 2024-03-24T00:04:05.214 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:04:04 smithi118 ceph-mon[31535]: rocksdb: Options.force_consistency_checks: 1 2024-03-24T00:04:05.214 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:04:04 smithi118 ceph-mon[31535]: rocksdb: Options.report_bg_io_stats: 0 2024-03-24T00:04:05.214 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:04:04 smithi118 ceph-mon[31535]: rocksdb: Options.ttl: 2592000 2024-03-24T00:04:05.214 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:04:04 smithi118 ceph-mon[31535]: rocksdb: Options.periodic_compaction_seconds: 0 2024-03-24T00:04:05.214 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:04:04 smithi118 ceph-mon[31535]: rocksdb: Options.preclude_last_level_data_seconds: 0 2024-03-24T00:04:05.214 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:04:04 smithi118 ceph-mon[31535]: rocksdb: Options.preserve_internal_time_seconds: 0 2024-03-24T00:04:05.214 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:04:04 smithi118 ceph-mon[31535]: rocksdb: Options.enable_blob_files: false 2024-03-24T00:04:05.214 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:04:04 smithi118 ceph-mon[31535]: rocksdb: Options.min_blob_size: 0 2024-03-24T00:04:05.214 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:04:04 smithi118 ceph-mon[31535]: rocksdb: Options.blob_file_size: 268435456 2024-03-24T00:04:05.214 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:04:04 smithi118 ceph-mon[31535]: rocksdb: Options.blob_compression_type: NoCompression 2024-03-24T00:04:05.214 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:04:04 smithi118 ceph-mon[31535]: rocksdb: Options.enable_blob_garbage_collection: false 2024-03-24T00:04:05.214 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:04:04 smithi118 ceph-mon[31535]: rocksdb: Options.blob_garbage_collection_age_cutoff: 0.250000 2024-03-24T00:04:05.215 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:04:04 smithi118 ceph-mon[31535]: rocksdb: Options.blob_garbage_collection_force_threshold: 1.000000 2024-03-24T00:04:05.215 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:04:04 smithi118 ceph-mon[31535]: rocksdb: Options.blob_compaction_readahead_size: 0 2024-03-24T00:04:05.215 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:04:04 smithi118 ceph-mon[31535]: rocksdb: Options.blob_file_starting_level: 0 2024-03-24T00:04:05.215 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:04:04 smithi118 ceph-mon[31535]: rocksdb: Options.experimental_mempurge_threshold: 0.000000 2024-03-24T00:04:05.215 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:04:04 smithi118 ceph-mon[31535]: rocksdb: [db/version_set.cc:5566] Recovered from manifest file:/var/lib/ceph/mon/ceph-a/store.db/MANIFEST-000010 succeeded,manifest_file_number is 10, next_file_number is 12, last_sequence is 5, log_number is 5,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 5 2024-03-24T00:04:05.215 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:04:04 smithi118 ceph-mon[31535]: rocksdb: [db/version_set.cc:5581] Column family [default] (ID 0), log number is 5 2024-03-24T00:04:05.215 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:04:04 smithi118 ceph-mon[31535]: rocksdb: [db/db_impl/db_impl_open.cc:539] DB ID: 3832990f-5b1c-4479-99de-9c7c7aafa68c 2024-03-24T00:04:05.215 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:04:04 smithi118 ceph-mon[31535]: rocksdb: EVENT_LOG_v1 {"time_micros": 1711238644885040, "job": 1, "event": "recovery_started", "wal_files": [9]} 2024-03-24T00:04:05.215 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:04:04 smithi118 ceph-mon[31535]: rocksdb: [db/db_impl/db_impl_open.cc:1043] Recovering log #9 mode 2 2024-03-24T00:04:05.215 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:04:04 smithi118 ceph-mon[31535]: rocksdb: EVENT_LOG_v1 {"time_micros": 1711238644887136, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 13, "file_size": 95895, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 8, "largest_seqno": 318, "table_properties": {"data_size": 93885, "index_size": 292, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 1, "index_value_is_delta_encoded": 1, "filter_size": 709, "raw_key_size": 13934, "raw_average_key_size": 50, "raw_value_size": 86313, "raw_average_value_size": 311, "num_data_blocks": 11, "num_entries": 277, "num_filter_entries": 277, "num_deletions": 3, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "bloomfilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; max_dict_buffer_bytes=0; use_zstd_dict_trainer=1; ", "creation_time": 1711238644, "oldest_key_time": 0, "file_creation_time": 0, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "3832990f-5b1c-4479-99de-9c7c7aafa68c", "db_session_id": "89JZQ24IUDZRLV2AQE2K", "orig_file_number": 13, "seqno_to_time_mapping": "N/A"}} 2024-03-24T00:04:05.215 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:04:04 smithi118 ceph-mon[31535]: rocksdb: EVENT_LOG_v1 {"time_micros": 1711238644887305, "job": 1, "event": "recovery_finished"} 2024-03-24T00:04:05.215 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:04:04 smithi118 ceph-mon[31535]: rocksdb: [db/version_set.cc:5047] Creating manifest 15 2024-03-24T00:04:05.215 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:04:04 smithi118 ceph-mon[31535]: rocksdb: [file/delete_scheduler.cc:74] Deleted file /var/lib/ceph/mon/ceph-a/store.db/000009.log immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024-03-24T00:04:05.215 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:04:04 smithi118 ceph-mon[31535]: rocksdb: [db/db_impl/db_impl_open.cc:1987] SstFileManager instance 0x5578e1424000 2024-03-24T00:04:05.215 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:04:04 smithi118 ceph-mon[31535]: rocksdb: DB pointer 0x5578e140e000 2024-03-24T00:04:05.216 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:04:04 smithi118 ceph-mon[31535]: rocksdb: [db/db_impl/db_impl.cc:1109] ------- DUMPING STATS ------- 2024-03-24T00:04:05.216 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:04:04 smithi118 ceph-mon[31535]: rocksdb: [db/db_impl/db_impl.cc:1111] 2024-03-24T00:04:05.216 INFO:journalctl@ceph.mon.a.smithi118.stdout: ** DB Stats ** 2024-03-24T00:04:05.216 INFO:journalctl@ceph.mon.a.smithi118.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-03-24T00:04:05.216 INFO:journalctl@ceph.mon.a.smithi118.stdout: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2024-03-24T00:04:05.216 INFO:journalctl@ceph.mon.a.smithi118.stdout: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-03-24T00:04:05.216 INFO:journalctl@ceph.mon.a.smithi118.stdout: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2024-03-24T00:04:05.216 INFO:journalctl@ceph.mon.a.smithi118.stdout: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2024-03-24T00:04:05.216 INFO:journalctl@ceph.mon.a.smithi118.stdout: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-03-24T00:04:05.216 INFO:journalctl@ceph.mon.a.smithi118.stdout: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2024-03-24T00:04:05.216 INFO:journalctl@ceph.mon.a.smithi118.stdout: 2024-03-24T00:04:05.217 INFO:journalctl@ceph.mon.a.smithi118.stdout: ** Compaction Stats [default] ** 2024-03-24T00:04:05.217 INFO:journalctl@ceph.mon.a.smithi118.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-03-24T00:04:05.217 INFO:journalctl@ceph.mon.a.smithi118.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ 2024-03-24T00:04:05.217 INFO:journalctl@ceph.mon.a.smithi118.stdout: L0 2/0 95.48 KB 0.5 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 64.3 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-03-24T00:04:05.217 INFO:journalctl@ceph.mon.a.smithi118.stdout: Sum 2/0 95.48 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 64.3 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-03-24T00:04:05.217 INFO:journalctl@ceph.mon.a.smithi118.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 64.3 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-03-24T00:04:05.217 INFO:journalctl@ceph.mon.a.smithi118.stdout: 2024-03-24T00:04:05.217 INFO:journalctl@ceph.mon.a.smithi118.stdout: ** Compaction Stats [default] ** 2024-03-24T00:04:05.217 INFO:journalctl@ceph.mon.a.smithi118.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-03-24T00:04:05.217 INFO:journalctl@ceph.mon.a.smithi118.stdout: --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-03-24T00:04:05.217 INFO:journalctl@ceph.mon.a.smithi118.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 64.3 0.00 0.00 1 0.001 0 0 0.0 0.0 2024-03-24T00:04:05.217 INFO:journalctl@ceph.mon.a.smithi118.stdout: 2024-03-24T00:04:05.217 INFO:journalctl@ceph.mon.a.smithi118.stdout: Blob file count: 0, total size: 0.0 GB, garbage size: 0.0 GB, space amp: 0.0 2024-03-24T00:04:05.217 INFO:journalctl@ceph.mon.a.smithi118.stdout: 2024-03-24T00:04:05.218 INFO:journalctl@ceph.mon.a.smithi118.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-03-24T00:04:05.218 INFO:journalctl@ceph.mon.a.smithi118.stdout: Flush(GB): cumulative 0.000, interval 0.000 2024-03-24T00:04:05.218 INFO:journalctl@ceph.mon.a.smithi118.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2024-03-24T00:04:05.218 INFO:journalctl@ceph.mon.a.smithi118.stdout: AddFile(Total Files): cumulative 0, interval 0 2024-03-24T00:04:05.218 INFO:journalctl@ceph.mon.a.smithi118.stdout: AddFile(L0 Files): cumulative 0, interval 0 2024-03-24T00:04:05.218 INFO:journalctl@ceph.mon.a.smithi118.stdout: AddFile(Keys): cumulative 0, interval 0 2024-03-24T00:04:05.218 INFO:journalctl@ceph.mon.a.smithi118.stdout: Cumulative compaction: 0.00 GB write, 13.94 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-03-24T00:04:05.218 INFO:journalctl@ceph.mon.a.smithi118.stdout: Interval compaction: 0.00 GB write, 13.94 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-03-24T00:04:05.218 INFO:journalctl@ceph.mon.a.smithi118.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-03-24T00:04:05.218 INFO:journalctl@ceph.mon.a.smithi118.stdout: Block cache BinnedLRUCache@0x5578e1331090#2 capacity: 512.00 MB usage: 1.39 KB table_size: 0 occupancy: 18446744073709551615 collections: 1 last_copies: 0 last_secs: 1.2e-05 secs_since: 0 2024-03-24T00:04:05.218 INFO:journalctl@ceph.mon.a.smithi118.stdout: Block cache entry stats(count,size,portion): FilterBlock(2,0.89 KB,0.000169873%) IndexBlock(2,0.50 KB,9.53674e-05%) Misc(1,0.00 KB,0%) 2024-03-24T00:04:05.218 INFO:journalctl@ceph.mon.a.smithi118.stdout: 2024-03-24T00:04:05.218 INFO:journalctl@ceph.mon.a.smithi118.stdout: ** File Read Latency Histogram By Level [default] ** 2024-03-24T00:04:05.218 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:04:04 smithi118 ceph-mon[31535]: starting mon.a rank 0 at public addrs [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] at bind addrs [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] mon_data /var/lib/ceph/mon/ceph-a fsid d79a5420-e971-11ee-95cd-87774f69a715 2024-03-24T00:04:05.219 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:04:04 smithi118 ceph-mon[31535]: mon.a@-1(???) e1 preinit fsid d79a5420-e971-11ee-95cd-87774f69a715 2024-03-24T00:04:05.219 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:04:04 smithi118 ceph-mon[31535]: mon.a@-1(???).mds e1 new map 2024-03-24T00:04:05.219 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:04:04 smithi118 ceph-mon[31535]: mon.a@-1(???).mds e1 print_map 2024-03-24T00:04:05.219 INFO:journalctl@ceph.mon.a.smithi118.stdout: e1 2024-03-24T00:04:05.219 INFO:journalctl@ceph.mon.a.smithi118.stdout: enable_multiple, ever_enabled_multiple: 1,1 2024-03-24T00:04:05.219 INFO:journalctl@ceph.mon.a.smithi118.stdout: default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2,11=minor log segments} 2024-03-24T00:04:05.219 INFO:journalctl@ceph.mon.a.smithi118.stdout: legacy client fscid: -1 2024-03-24T00:04:05.219 INFO:journalctl@ceph.mon.a.smithi118.stdout: 2024-03-24T00:04:05.219 INFO:journalctl@ceph.mon.a.smithi118.stdout: No filesystems configured 2024-03-24T00:04:05.219 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:04:04 smithi118 ceph-mon[31535]: mon.a@-1(???).osd e1 crush map has features 3314932999778484224, adjusting msgr requires 2024-03-24T00:04:05.219 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:04:04 smithi118 ceph-mon[31535]: mon.a@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-03-24T00:04:05.219 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:04:04 smithi118 ceph-mon[31535]: mon.a@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-03-24T00:04:05.219 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:04:04 smithi118 ceph-mon[31535]: mon.a@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-03-24T00:04:05.219 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:04:04 smithi118 ceph-mon[31535]: mon.a@-1(???).paxosservice(auth 1..2) refresh upgraded, format 0 -> 3 2024-03-24T00:04:05.220 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:04:04 smithi118 bash[31516]: 9a3b56cb7cc8e0313c3963fe3db6b16bb1ecf7d71ab784c3a43ea706ab1ec49b 2024-03-24T00:04:05.220 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:04:04 smithi118 systemd[1]: Started Ceph mon.a for d79a5420-e971-11ee-95cd-87774f69a715. 2024-03-24T00:04:05.220 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:04:04 smithi118 ceph-mon[31535]: mon.a is new leader, mons a in quorum (ranks 0) 2024-03-24T00:04:05.220 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:04:04 smithi118 ceph-mon[31535]: monmap e1: 1 mons at {a=[v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0]} removed_ranks: {} disallowed_leaders: {} 2024-03-24T00:04:05.220 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:04:04 smithi118 ceph-mon[31535]: fsmap 2024-03-24T00:04:05.220 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:04:04 smithi118 ceph-mon[31535]: osdmap e1: 0 total, 0 up, 0 in 2024-03-24T00:04:05.220 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:04:04 smithi118 ceph-mon[31535]: mgrmap e1: no daemons active 2024-03-24T00:04:06.286 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:04:05.523+0000 7f12ed400700 1 Processor -- start 2024-03-24T00:04:06.286 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:04:05.523+0000 7f12ed400700 1 -- start start 2024-03-24T00:04:06.286 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:04:05.524+0000 7f12ed400700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f12e8107b30 0x7f12e8107f50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:04:06.286 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:04:05.524+0000 7f12ed400700 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f12e8108520 con 0x7f12e8107b30 2024-03-24T00:04:06.286 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:04:05.524+0000 7f12e6800700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f12e8107b30 0x7f12e8107f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:04:06.286 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:04:05.524+0000 7f12e6800700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f12e8107b30 0x7f12e8107f50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.118:3300/0 says I am v2:172.21.15.118:47690/0 (socket says 172.21.15.118:47690) 2024-03-24T00:04:06.286 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:04:05.525+0000 7f12e6800700 1 -- 172.21.15.118:0/3564772434 learned_addr learned my addr 172.21.15.118:0/3564772434 (peer_addr_for_me v2:172.21.15.118:0/0) 2024-03-24T00:04:06.286 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:04:05.525+0000 7f12e6800700 1 -- 172.21.15.118:0/3564772434 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f12e8108d30 con 0x7f12e8107b30 2024-03-24T00:04:06.287 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:04:05.526+0000 7f12e6800700 1 --2- 172.21.15.118:0/3564772434 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f12e8107b30 0x7f12e8107f50 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7f12d80098a0 tx=0x7f12d8009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b5760a8e5b88c004 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:04:06.287 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:04:05.526+0000 7f12e5e00700 1 -- 172.21.15.118:0/3564772434 <== mon.0 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f12d8004160 con 0x7f12e8107b30 2024-03-24T00:04:06.287 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:04:05.526+0000 7f12e5e00700 1 -- 172.21.15.118:0/3564772434 <== mon.0 v2:172.21.15.118:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f12d8004300 con 0x7f12e8107b30 2024-03-24T00:04:06.287 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:04:05.526+0000 7f12e5e00700 1 -- 172.21.15.118:0/3564772434 <== mon.0 v2:172.21.15.118:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f12d8011b50 con 0x7f12e8107b30 2024-03-24T00:04:06.287 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:04:05.527+0000 7f12ed400700 1 -- 172.21.15.118:0/3564772434 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f12e8107b30 msgr2=0x7f12e8107f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:04:06.287 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:04:05.527+0000 7f12ed400700 1 --2- 172.21.15.118:0/3564772434 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f12e8107b30 0x7f12e8107f50 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7f12d80098a0 tx=0x7f12d8009bb0 comp rx=0 tx=0).stop 2024-03-24T00:04:06.287 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:04:05.527+0000 7f12ed400700 1 -- 172.21.15.118:0/3564772434 shutdown_connections 2024-03-24T00:04:06.287 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:04:05.527+0000 7f12ed400700 1 --2- 172.21.15.118:0/3564772434 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f12e8107b30 0x7f12e8107f50 unknown :-1 s=CLOSED pgs=1 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:04:06.287 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:04:05.527+0000 7f12ed400700 1 -- 172.21.15.118:0/3564772434 >> 172.21.15.118:0/3564772434 conn(0x7f12e8103150 msgr2=0x7f12e8105570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:04:06.287 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:04:05.527+0000 7f12ed400700 1 -- 172.21.15.118:0/3564772434 shutdown_connections 2024-03-24T00:04:06.287 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:04:05.528+0000 7f12ed400700 1 -- 172.21.15.118:0/3564772434 wait complete. 2024-03-24T00:04:06.287 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:04:05.529+0000 7f12ed400700 1 Processor -- start 2024-03-24T00:04:06.287 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:04:05.529+0000 7f12ed400700 1 -- start start 2024-03-24T00:04:06.287 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:04:05.529+0000 7f12ed400700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f12e8107b30 0x7f12e807db90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:04:06.288 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:04:05.529+0000 7f12ed400700 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f12e810c0e0 con 0x7f12e8107b30 2024-03-24T00:04:06.288 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:04:05.529+0000 7f12e6800700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f12e8107b30 0x7f12e807db90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:04:06.288 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:04:05.530+0000 7f12e6800700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f12e8107b30 0x7f12e807db90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.118:3300/0 says I am v2:172.21.15.118:47706/0 (socket says 172.21.15.118:47706) 2024-03-24T00:04:06.288 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:04:05.530+0000 7f12e6800700 1 -- 172.21.15.118:0/2647598832 learned_addr learned my addr 172.21.15.118:0/2647598832 (peer_addr_for_me v2:172.21.15.118:0/0) 2024-03-24T00:04:06.288 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:04:05.530+0000 7f12e6800700 1 -- 172.21.15.118:0/2647598832 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f12e807e0d0 con 0x7f12e8107b30 2024-03-24T00:04:06.288 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:04:05.530+0000 7f12e6800700 1 --2- 172.21.15.118:0/2647598832 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f12e8107b30 0x7f12e807db90 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7f12d8009fb0 tx=0x7f12d8011bf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:04:06.288 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:04:05.530+0000 7f12df400700 1 -- 172.21.15.118:0/2647598832 <== mon.0 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f12d80038b0 con 0x7f12e8107b30 2024-03-24T00:04:06.288 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:04:05.531+0000 7f12df400700 1 -- 172.21.15.118:0/2647598832 <== mon.0 v2:172.21.15.118:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f12d8011e50 con 0x7f12e8107b30 2024-03-24T00:04:06.288 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:04:05.531+0000 7f12ed400700 1 -- 172.21.15.118:0/2647598832 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f12e807c2c0 con 0x7f12e8107b30 2024-03-24T00:04:06.288 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:04:05.531+0000 7f12df400700 1 -- 172.21.15.118:0/2647598832 <== mon.0 v2:172.21.15.118:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f12d8019550 con 0x7f12e8107b30 2024-03-24T00:04:06.288 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:04:05.531+0000 7f12ed400700 1 -- 172.21.15.118:0/2647598832 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f12e807c800 con 0x7f12e8107b30 2024-03-24T00:04:06.288 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:04:05.531+0000 7f12df400700 1 -- 172.21.15.118:0/2647598832 <== mon.0 v2:172.21.15.118:3300/0 4 ==== mgrmap(e 1) v1 ==== 807+0+0 (secure 0 0 0) 0x7f12d8019b90 con 0x7f12e8107b30 2024-03-24T00:04:06.288 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:04:05.531+0000 7f12df400700 1 -- 172.21.15.118:0/2647598832 <== mon.0 v2:172.21.15.118:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (secure 0 0 0) 0x7f12d8024730 con 0x7f12e8107b30 2024-03-24T00:04:06.288 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:04:05.532+0000 7f12ed400700 1 -- 172.21.15.118:0/2647598832 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f12e80637c0 con 0x7f12e8107b30 2024-03-24T00:04:06.288 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:04:05.535+0000 7f12df400700 1 -- 172.21.15.118:0/2647598832 <== mon.0 v2:172.21.15.118:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (secure 0 0 0) 0x7f12d8003a50 con 0x7f12e8107b30 2024-03-24T00:04:06.289 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:04:05.611+0000 7f12ed400700 1 -- 172.21.15.118:0/2647598832 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_command([{prefix=config set, name=public_network}] v 0) v1 -- 0x7f12e807cd30 con 0x7f12e8107b30 2024-03-24T00:04:06.289 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:04:05.614+0000 7f12df400700 1 -- 172.21.15.118:0/2647598832 <== mon.0 v2:172.21.15.118:3300/0 7 ==== mon_command_ack([{prefix=config set, name=public_network}]=0 v3)=0 v3) v1 ==== 124+0+0 (secure 0 0 0) 0x7f12d8024930 con 0x7f12e8107b30 2024-03-24T00:04:06.289 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:04:05.615+0000 7f12ed400700 1 -- 172.21.15.118:0/2647598832 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f12e8107b30 msgr2=0x7f12e807db90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:04:06.289 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:04:05.615+0000 7f12ed400700 1 --2- 172.21.15.118:0/2647598832 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f12e8107b30 0x7f12e807db90 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7f12d8009fb0 tx=0x7f12d8011bf0 comp rx=0 tx=0).stop 2024-03-24T00:04:06.289 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:04:05.615+0000 7f12ed400700 1 -- 172.21.15.118:0/2647598832 shutdown_connections 2024-03-24T00:04:06.289 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:04:05.615+0000 7f12ed400700 1 --2- 172.21.15.118:0/2647598832 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f12e8107b30 0x7f12e807db90 unknown :-1 s=CLOSED pgs=2 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:04:06.289 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:04:05.615+0000 7f12ed400700 1 -- 172.21.15.118:0/2647598832 >> 172.21.15.118:0/2647598832 conn(0x7f12e8103150 msgr2=0x7f12e8103c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:04:06.289 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:04:05.615+0000 7f12ed400700 1 -- 172.21.15.118:0/2647598832 shutdown_connections 2024-03-24T00:04:06.289 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:04:05.615+0000 7f12ed400700 1 -- 172.21.15.118:0/2647598832 wait complete. 2024-03-24T00:04:06.290 INFO:teuthology.orchestra.run.smithi118.stdout:Wrote config to /etc/ceph/ceph.conf 2024-03-24T00:04:06.323 INFO:teuthology.orchestra.run.smithi118.stdout:Wrote keyring to /etc/ceph/ceph.client.admin.keyring 2024-03-24T00:04:06.323 INFO:teuthology.orchestra.run.smithi118.stdout:Creating mgr... 2024-03-24T00:04:06.325 INFO:teuthology.orchestra.run.smithi118.stdout:Verifying port 0.0.0.0:9283 ... 2024-03-24T00:04:06.326 INFO:teuthology.orchestra.run.smithi118.stdout:Verifying port 0.0.0.0:8765 ... 2024-03-24T00:04:06.755 INFO:teuthology.orchestra.run.smithi118.stdout:Non-zero exit code 1 from systemctl reset-failed ceph-d79a5420-e971-11ee-95cd-87774f69a715@mgr.x 2024-03-24T00:04:06.755 INFO:teuthology.orchestra.run.smithi118.stdout:systemctl: stderr Failed to reset failed state of unit ceph-d79a5420-e971-11ee-95cd-87774f69a715@mgr.x.service: Unit ceph-d79a5420-e971-11ee-95cd-87774f69a715@mgr.x.service not loaded. 2024-03-24T00:04:06.945 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:04:06 smithi118 ceph-mon[31535]: from='client.? 172.21.15.118:0/2647598832' entity='client.admin' 2024-03-24T00:04:07.023 INFO:teuthology.orchestra.run.smithi118.stdout:systemctl: stderr Created symlink /etc/systemd/system/ceph-d79a5420-e971-11ee-95cd-87774f69a715.target.wants/ceph-d79a5420-e971-11ee-95cd-87774f69a715@mgr.x.service → /etc/systemd/system/ceph-d79a5420-e971-11ee-95cd-87774f69a715@.service. 2024-03-24T00:04:07.413 INFO:journalctl@ceph.mgr.x.smithi118.stdout:Mar 24 00:04:07 smithi118 systemd[1]: Starting Ceph mgr.x for d79a5420-e971-11ee-95cd-87774f69a715... 2024-03-24T00:04:07.696 INFO:journalctl@ceph.mgr.x.smithi118.stdout:Mar 24 00:04:07 smithi118 podman[31747]: 2024-03-24 00:04:07.320617778 +0000 UTC m=+0.037312223 image pull c664afe56bb01db4f9afa9532c20598e201e664f2e9ba2be50db0d52674a26d2 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 2024-03-24T00:04:07.697 INFO:journalctl@ceph.mgr.x.smithi118.stdout:Mar 24 00:04:07 smithi118 podman[31747]: 2024-03-24 00:04:07.418075529 +0000 UTC m=+0.134769994 container create 16a004216cba58ac648fa39a41748df811122aec56543cff33c3b23f58bcd392 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033, name=ceph-d79a5420-e971-11ee-95cd-87774f69a715-mgr-x, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.build-date=20240319, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, ceph=True) 2024-03-24T00:04:07.697 INFO:journalctl@ceph.mgr.x.smithi118.stdout:Mar 24 00:04:07 smithi118 podman[31747]: 2024-03-24 00:04:07.696711274 +0000 UTC m=+0.413405746 container init 16a004216cba58ac648fa39a41748df811122aec56543cff33c3b23f58bcd392 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033, name=ceph-d79a5420-e971-11ee-95cd-87774f69a715-mgr-x, org.label-schema.build-date=20240319, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, ceph=True, GIT_CLEAN=True, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-24T00:04:07.818 INFO:teuthology.orchestra.run.smithi118.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-03-24T00:04:07.819 INFO:teuthology.orchestra.run.smithi118.stdout:systemctl: stdout disabled 2024-03-24T00:04:07.834 INFO:teuthology.orchestra.run.smithi118.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-03-24T00:04:07.834 INFO:teuthology.orchestra.run.smithi118.stdout:systemctl: stdout inactive 2024-03-24T00:04:07.834 INFO:teuthology.orchestra.run.smithi118.stdout:firewalld.service is not enabled 2024-03-24T00:04:07.834 INFO:teuthology.orchestra.run.smithi118.stdout:Not possible to enable service . firewalld.service is not available 2024-03-24T00:04:07.854 INFO:teuthology.orchestra.run.smithi118.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-03-24T00:04:07.854 INFO:teuthology.orchestra.run.smithi118.stdout:systemctl: stdout disabled 2024-03-24T00:04:07.873 INFO:teuthology.orchestra.run.smithi118.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-03-24T00:04:07.874 INFO:teuthology.orchestra.run.smithi118.stdout:systemctl: stdout inactive 2024-03-24T00:04:07.874 INFO:teuthology.orchestra.run.smithi118.stdout:firewalld.service is not enabled 2024-03-24T00:04:07.874 INFO:teuthology.orchestra.run.smithi118.stdout:Not possible to open ports <[9283, 8765]>. firewalld.service is not available 2024-03-24T00:04:07.874 INFO:teuthology.orchestra.run.smithi118.stdout:Waiting for mgr to start... 2024-03-24T00:04:07.874 INFO:teuthology.orchestra.run.smithi118.stdout:Waiting for mgr... 2024-03-24T00:04:08.022 INFO:journalctl@ceph.mgr.x.smithi118.stdout:Mar 24 00:04:07 smithi118 podman[31747]: 2024-03-24 00:04:07.708209326 +0000 UTC m=+0.424903763 container start 16a004216cba58ac648fa39a41748df811122aec56543cff33c3b23f58bcd392 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033, name=ceph-d79a5420-e971-11ee-95cd-87774f69a715-mgr-x, org.label-schema.build-date=20240319, io.buildah.version=1.33.5, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, ceph=True, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD) 2024-03-24T00:04:08.022 INFO:journalctl@ceph.mgr.x.smithi118.stdout:Mar 24 00:04:07 smithi118 bash[31747]: 16a004216cba58ac648fa39a41748df811122aec56543cff33c3b23f58bcd392 2024-03-24T00:04:08.022 INFO:journalctl@ceph.mgr.x.smithi118.stdout:Mar 24 00:04:07 smithi118 systemd[1]: Started Ceph mgr.x for d79a5420-e971-11ee-95cd-87774f69a715. 2024-03-24T00:04:08.548 INFO:journalctl@ceph.mgr.x.smithi118.stdout:Mar 24 00:04:08 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mgr-x[31762]: 2024-03-24T00:04:08.496+0000 7f8b59fb6200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-24T00:04:08.945 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:04:08 smithi118 ceph-mon[31535]: from='client.? 172.21.15.118:0/3424332897' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-03-24T00:04:08.946 INFO:journalctl@ceph.mgr.x.smithi118.stdout:Mar 24 00:04:08 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mgr-x[31762]: 2024-03-24T00:04:08.628+0000 7f8b59fb6200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-24T00:04:09.282 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stdout 2024-03-24T00:04:09.283 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stdout { 2024-03-24T00:04:09.283 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stdout "fsid": "d79a5420-e971-11ee-95cd-87774f69a715", 2024-03-24T00:04:09.284 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stdout "health": { 2024-03-24T00:04:09.284 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2024-03-24T00:04:09.284 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stdout "checks": {}, 2024-03-24T00:04:09.285 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stdout "mutes": [] 2024-03-24T00:04:09.285 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stdout }, 2024-03-24T00:04:09.285 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2024-03-24T00:04:09.285 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stdout "quorum": [ 2024-03-24T00:04:09.286 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stdout 0 2024-03-24T00:04:09.286 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stdout ], 2024-03-24T00:04:09.286 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2024-03-24T00:04:09.286 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stdout "a" 2024-03-24T00:04:09.286 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stdout ], 2024-03-24T00:04:09.286 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stdout "quorum_age": 3, 2024-03-24T00:04:09.287 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stdout "monmap": { 2024-03-24T00:04:09.287 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-24T00:04:09.288 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "squid", 2024-03-24T00:04:09.288 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stdout "num_mons": 1 2024-03-24T00:04:09.288 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stdout }, 2024-03-24T00:04:09.288 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stdout "osdmap": { 2024-03-24T00:04:09.288 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-24T00:04:09.288 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2024-03-24T00:04:09.288 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2024-03-24T00:04:09.288 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2024-03-24T00:04:09.290 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2024-03-24T00:04:09.290 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2024-03-24T00:04:09.290 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2024-03-24T00:04:09.290 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stdout }, 2024-03-24T00:04:09.290 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stdout "pgmap": { 2024-03-24T00:04:09.290 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2024-03-24T00:04:09.290 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2024-03-24T00:04:09.291 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2024-03-24T00:04:09.291 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2024-03-24T00:04:09.291 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2024-03-24T00:04:09.291 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2024-03-24T00:04:09.291 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2024-03-24T00:04:09.291 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2024-03-24T00:04:09.291 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stdout }, 2024-03-24T00:04:09.292 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stdout "fsmap": { 2024-03-24T00:04:09.293 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-24T00:04:09.293 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stdout "by_rank": [], 2024-03-24T00:04:09.293 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stdout "up:standby": 0 2024-03-24T00:04:09.293 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stdout }, 2024-03-24T00:04:09.293 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stdout "mgrmap": { 2024-03-24T00:04:09.293 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stdout "available": false, 2024-03-24T00:04:09.293 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2024-03-24T00:04:09.293 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stdout "modules": [ 2024-03-24T00:04:09.293 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stdout "iostat", 2024-03-24T00:04:09.293 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stdout "nfs", 2024-03-24T00:04:09.293 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stdout "restful" 2024-03-24T00:04:09.293 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stdout ], 2024-03-24T00:04:09.293 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stdout "services": {} 2024-03-24T00:04:09.295 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stdout }, 2024-03-24T00:04:09.295 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stdout "servicemap": { 2024-03-24T00:04:09.295 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-24T00:04:09.295 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stdout "modified": "2024-03-24T00:03:58.649100+0000", 2024-03-24T00:04:09.295 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stdout "services": {} 2024-03-24T00:04:09.295 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stdout }, 2024-03-24T00:04:09.295 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stdout "progress_events": {} 2024-03-24T00:04:09.296 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stdout } 2024-03-24T00:04:09.296 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:04:08.459+0000 7f6329400700 1 Processor -- start 2024-03-24T00:04:09.296 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:04:08.459+0000 7f6329400700 1 -- start start 2024-03-24T00:04:09.296 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:04:08.460+0000 7f6329400700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f6324108600 0x7f632410a9f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:04:09.296 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:04:08.460+0000 7f6329400700 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f632407be70 con 0x7f6324108600 2024-03-24T00:04:09.296 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:04:08.460+0000 7f6322800700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f6324108600 0x7f632410a9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:04:09.296 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:04:08.460+0000 7f6322800700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f6324108600 0x7f632410a9f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.118:3300/0 says I am v2:172.21.15.118:47736/0 (socket says 172.21.15.118:47736) 2024-03-24T00:04:09.298 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:04:08.461+0000 7f6322800700 1 -- 172.21.15.118:0/502559232 learned_addr learned my addr 172.21.15.118:0/502559232 (peer_addr_for_me v2:172.21.15.118:0/0) 2024-03-24T00:04:09.298 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:04:08.461+0000 7f6322800700 1 -- 172.21.15.118:0/502559232 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f632410af30 con 0x7f6324108600 2024-03-24T00:04:09.298 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:04:08.461+0000 7f6322800700 1 --2- 172.21.15.118:0/502559232 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f6324108600 0x7f632410a9f0 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7f63140098a0 tx=0x7f6314009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=445f000ae07caf43 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:04:09.298 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:04:08.462+0000 7f6321400700 1 -- 172.21.15.118:0/502559232 <== mon.0 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f6314004160 con 0x7f6324108600 2024-03-24T00:04:09.298 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:04:08.462+0000 7f6321400700 1 -- 172.21.15.118:0/502559232 <== mon.0 v2:172.21.15.118:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f6314004300 con 0x7f6324108600 2024-03-24T00:04:09.298 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:04:08.462+0000 7f6321400700 1 -- 172.21.15.118:0/502559232 <== mon.0 v2:172.21.15.118:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f6314011b50 con 0x7f6324108600 2024-03-24T00:04:09.298 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:04:08.462+0000 7f6329400700 1 -- 172.21.15.118:0/502559232 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f6324108600 msgr2=0x7f632410a9f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:04:09.298 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:04:08.462+0000 7f6329400700 1 --2- 172.21.15.118:0/502559232 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f6324108600 0x7f632410a9f0 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7f63140098a0 tx=0x7f6314009bb0 comp rx=0 tx=0).stop 2024-03-24T00:04:09.298 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:04:08.463+0000 7f6329400700 1 -- 172.21.15.118:0/502559232 shutdown_connections 2024-03-24T00:04:09.298 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:04:08.463+0000 7f6329400700 1 --2- 172.21.15.118:0/502559232 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f6324108600 0x7f632410a9f0 unknown :-1 s=CLOSED pgs=5 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:04:09.299 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:04:08.463+0000 7f6329400700 1 -- 172.21.15.118:0/502559232 >> 172.21.15.118:0/502559232 conn(0x7f6324101fd0 msgr2=0x7f6324104410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:04:09.299 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:04:08.463+0000 7f6329400700 1 -- 172.21.15.118:0/502559232 shutdown_connections 2024-03-24T00:04:09.299 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:04:08.463+0000 7f6329400700 1 -- 172.21.15.118:0/502559232 wait complete. 2024-03-24T00:04:09.299 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:04:08.464+0000 7f6329400700 1 Processor -- start 2024-03-24T00:04:09.299 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:04:08.464+0000 7f6329400700 1 -- start start 2024-03-24T00:04:09.301 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:04:08.465+0000 7f6329400700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f6324108600 0x7f6324107e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:04:09.301 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:04:08.465+0000 7f6329400700 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f632410bf00 con 0x7f6324108600 2024-03-24T00:04:09.301 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:04:08.465+0000 7f6322800700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f6324108600 0x7f6324107e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:04:09.301 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:04:08.465+0000 7f6322800700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f6324108600 0x7f6324107e80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.118:3300/0 says I am v2:172.21.15.118:47742/0 (socket says 172.21.15.118:47742) 2024-03-24T00:04:09.301 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:04:08.465+0000 7f6322800700 1 -- 172.21.15.118:0/3424332897 learned_addr learned my addr 172.21.15.118:0/3424332897 (peer_addr_for_me v2:172.21.15.118:0/0) 2024-03-24T00:04:09.301 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:04:08.466+0000 7f6322800700 1 -- 172.21.15.118:0/3424332897 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f63241083c0 con 0x7f6324108600 2024-03-24T00:04:09.301 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:04:08.466+0000 7f6322800700 1 --2- 172.21.15.118:0/3424332897 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f6324108600 0x7f6324107e80 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f6314011b60 tx=0x7f6314011c40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:04:09.301 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:04:08.466+0000 7f631b400700 1 -- 172.21.15.118:0/3424332897 <== mon.0 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f6314003b20 con 0x7f6324108600 2024-03-24T00:04:09.301 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:04:08.466+0000 7f631b400700 1 -- 172.21.15.118:0/3424332897 <== mon.0 v2:172.21.15.118:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f6314003cc0 con 0x7f6324108600 2024-03-24T00:04:09.302 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:04:08.466+0000 7f631b400700 1 -- 172.21.15.118:0/3424332897 <== mon.0 v2:172.21.15.118:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f6314024840 con 0x7f6324108600 2024-03-24T00:04:09.302 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:04:08.466+0000 7f6329400700 1 -- 172.21.15.118:0/3424332897 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f63241065b0 con 0x7f6324108600 2024-03-24T00:04:09.302 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:04:08.466+0000 7f6329400700 1 -- 172.21.15.118:0/3424332897 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6324106af0 con 0x7f6324108600 2024-03-24T00:04:09.302 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:04:08.467+0000 7f631b400700 1 -- 172.21.15.118:0/3424332897 <== mon.0 v2:172.21.15.118:3300/0 4 ==== mgrmap(e 1) v1 ==== 807+0+0 (secure 0 0 0) 0x7f6314022020 con 0x7f6324108600 2024-03-24T00:04:09.302 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:04:08.467+0000 7f631b400700 1 -- 172.21.15.118:0/3424332897 <== mon.0 v2:172.21.15.118:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (secure 0 0 0) 0x7f6314011e00 con 0x7f6324108600 2024-03-24T00:04:09.302 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:04:08.467+0000 7f6329400700 1 -- 172.21.15.118:0/3424332897 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f63240637c0 con 0x7f6324108600 2024-03-24T00:04:09.302 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:04:08.470+0000 7f631b400700 1 -- 172.21.15.118:0/3424332897 <== mon.0 v2:172.21.15.118:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (secure 0 0 0) 0x7f6314043dd0 con 0x7f6324108600 2024-03-24T00:04:09.302 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:04:08.548+0000 7f6329400700 1 -- 172.21.15.118:0/3424332897 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) v1 -- 0x7f6324106fd0 con 0x7f6324108600 2024-03-24T00:04:09.303 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:04:08.549+0000 7f631b400700 1 -- 172.21.15.118:0/3424332897 <== mon.0 v2:172.21.15.118:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) v1 ==== 79+0+1239 (secure 0 0 0) 0x7f6314032020 con 0x7f6324108600 2024-03-24T00:04:09.303 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:04:08.551+0000 7f6329400700 1 -- 172.21.15.118:0/3424332897 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f6324108600 msgr2=0x7f6324107e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:04:09.304 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:04:08.551+0000 7f6329400700 1 --2- 172.21.15.118:0/3424332897 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f6324108600 0x7f6324107e80 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f6314011b60 tx=0x7f6314011c40 comp rx=0 tx=0).stop 2024-03-24T00:04:09.304 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:04:08.551+0000 7f6329400700 1 -- 172.21.15.118:0/3424332897 shutdown_connections 2024-03-24T00:04:09.304 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:04:08.551+0000 7f6329400700 1 --2- 172.21.15.118:0/3424332897 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f6324108600 0x7f6324107e80 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:04:09.304 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:04:08.551+0000 7f6329400700 1 -- 172.21.15.118:0/3424332897 >> 172.21.15.118:0/3424332897 conn(0x7f6324101fd0 msgr2=0x7f6324102a90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:04:09.304 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:04:08.552+0000 7f6329400700 1 -- 172.21.15.118:0/3424332897 shutdown_connections 2024-03-24T00:04:09.304 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:04:08.552+0000 7f6329400700 1 -- 172.21.15.118:0/3424332897 wait complete. 2024-03-24T00:04:09.304 INFO:teuthology.orchestra.run.smithi118.stdout:mgr not available, waiting (1/15)... 2024-03-24T00:04:09.560 INFO:journalctl@ceph.mgr.x.smithi118.stdout:Mar 24 00:04:09 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mgr-x[31762]: 2024-03-24T00:04:09.441+0000 7f8b59fb6200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-24T00:04:09.858 INFO:journalctl@ceph.mgr.x.smithi118.stdout:Mar 24 00:04:09 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mgr-x[31762]: 2024-03-24T00:04:09.560+0000 7f8b59fb6200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-24T00:04:09.858 INFO:journalctl@ceph.mgr.x.smithi118.stdout:Mar 24 00:04:09 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mgr-x[31762]: 2024-03-24T00:04:09.736+0000 7f8b59fb6200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-24T00:04:09.858 INFO:journalctl@ceph.mgr.x.smithi118.stdout:Mar 24 00:04:09 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mgr-x[31762]: 2024-03-24T00:04:09.858+0000 7f8b59fb6200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-24T00:04:10.195 INFO:journalctl@ceph.mgr.x.smithi118.stdout:Mar 24 00:04:09 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mgr-x[31762]: 2024-03-24T00:04:09.966+0000 7f8b59fb6200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-24T00:04:10.666 INFO:journalctl@ceph.mgr.x.smithi118.stdout:Mar 24 00:04:10 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mgr-x[31762]: 2024-03-24T00:04:10.296+0000 7f8b59fb6200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-03-24T00:04:10.666 INFO:journalctl@ceph.mgr.x.smithi118.stdout:Mar 24 00:04:10 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mgr-x[31762]: 2024-03-24T00:04:10.433+0000 7f8b59fb6200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-24T00:04:10.666 INFO:journalctl@ceph.mgr.x.smithi118.stdout:Mar 24 00:04:10 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mgr-x[31762]: 2024-03-24T00:04:10.543+0000 7f8b59fb6200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-24T00:04:10.945 INFO:journalctl@ceph.mgr.x.smithi118.stdout:Mar 24 00:04:10 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mgr-x[31762]: 2024-03-24T00:04:10.666+0000 7f8b59fb6200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-24T00:04:10.957 INFO:journalctl@ceph.mgr.x.smithi118.stdout:Mar 24 00:04:10 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mgr-x[31762]: 2024-03-24T00:04:10.880+0000 7f8b59fb6200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-24T00:04:12.318 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:04:11 smithi118 ceph-mon[31535]: from='client.? 172.21.15.118:0/4182973945' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-03-24T00:04:12.318 INFO:journalctl@ceph.mgr.x.smithi118.stdout:Mar 24 00:04:12 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mgr-x[31762]: 2024-03-24T00:04:12.205+0000 7f8b59fb6200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-03-24T00:04:12.617 INFO:journalctl@ceph.mgr.x.smithi118.stdout:Mar 24 00:04:12 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mgr-x[31762]: 2024-03-24T00:04:12.376+0000 7f8b59fb6200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-24T00:04:12.625 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stdout 2024-03-24T00:04:12.625 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stdout { 2024-03-24T00:04:12.628 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stdout "fsid": "d79a5420-e971-11ee-95cd-87774f69a715", 2024-03-24T00:04:12.628 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stdout "health": { 2024-03-24T00:04:12.628 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2024-03-24T00:04:12.628 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stdout "checks": {}, 2024-03-24T00:04:12.628 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stdout "mutes": [] 2024-03-24T00:04:12.628 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stdout }, 2024-03-24T00:04:12.628 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2024-03-24T00:04:12.628 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stdout "quorum": [ 2024-03-24T00:04:12.628 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stdout 0 2024-03-24T00:04:12.628 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stdout ], 2024-03-24T00:04:12.628 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2024-03-24T00:04:12.628 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stdout "a" 2024-03-24T00:04:12.628 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stdout ], 2024-03-24T00:04:12.628 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stdout "quorum_age": 7, 2024-03-24T00:04:12.629 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stdout "monmap": { 2024-03-24T00:04:12.629 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-24T00:04:12.629 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "squid", 2024-03-24T00:04:12.629 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stdout "num_mons": 1 2024-03-24T00:04:12.629 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stdout }, 2024-03-24T00:04:12.629 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stdout "osdmap": { 2024-03-24T00:04:12.629 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-24T00:04:12.629 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2024-03-24T00:04:12.629 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2024-03-24T00:04:12.638 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2024-03-24T00:04:12.638 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2024-03-24T00:04:12.638 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2024-03-24T00:04:12.638 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2024-03-24T00:04:12.638 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stdout }, 2024-03-24T00:04:12.638 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stdout "pgmap": { 2024-03-24T00:04:12.638 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2024-03-24T00:04:12.638 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2024-03-24T00:04:12.638 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2024-03-24T00:04:12.638 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2024-03-24T00:04:12.639 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2024-03-24T00:04:12.639 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2024-03-24T00:04:12.639 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2024-03-24T00:04:12.639 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2024-03-24T00:04:12.639 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stdout }, 2024-03-24T00:04:12.639 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stdout "fsmap": { 2024-03-24T00:04:12.639 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-24T00:04:12.639 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stdout "by_rank": [], 2024-03-24T00:04:12.639 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stdout "up:standby": 0 2024-03-24T00:04:12.639 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stdout }, 2024-03-24T00:04:12.639 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stdout "mgrmap": { 2024-03-24T00:04:12.639 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stdout "available": false, 2024-03-24T00:04:12.639 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2024-03-24T00:04:12.639 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stdout "modules": [ 2024-03-24T00:04:12.640 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stdout "iostat", 2024-03-24T00:04:12.640 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stdout "nfs", 2024-03-24T00:04:12.640 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stdout "restful" 2024-03-24T00:04:12.640 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stdout ], 2024-03-24T00:04:12.640 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stdout "services": {} 2024-03-24T00:04:12.640 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stdout }, 2024-03-24T00:04:12.640 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stdout "servicemap": { 2024-03-24T00:04:12.640 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-24T00:04:12.640 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stdout "modified": "2024-03-24T00:03:58.649100+0000", 2024-03-24T00:04:12.640 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stdout "services": {} 2024-03-24T00:04:12.640 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stdout }, 2024-03-24T00:04:12.641 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stdout "progress_events": {} 2024-03-24T00:04:12.641 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stdout } 2024-03-24T00:04:12.641 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:04:11.855+0000 7f1426e00700 1 Processor -- start 2024-03-24T00:04:12.641 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:04:11.855+0000 7f1426e00700 1 -- start start 2024-03-24T00:04:12.641 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:04:11.855+0000 7f1426e00700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f1420106350 0x7f1420106770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:04:12.641 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:04:11.855+0000 7f1426e00700 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1420075900 con 0x7f1420106350 2024-03-24T00:04:12.641 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:04:11.856+0000 7f141fe00700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f1420106350 0x7f1420106770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:04:12.641 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:04:11.856+0000 7f141fe00700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f1420106350 0x7f1420106770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.118:3300/0 says I am v2:172.21.15.118:47750/0 (socket says 172.21.15.118:47750) 2024-03-24T00:04:12.641 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:04:11.856+0000 7f141fe00700 1 -- 172.21.15.118:0/1190192057 learned_addr learned my addr 172.21.15.118:0/1190192057 (peer_addr_for_me v2:172.21.15.118:0/0) 2024-03-24T00:04:12.641 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:04:11.856+0000 7f141fe00700 1 -- 172.21.15.118:0/1190192057 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1420106cb0 con 0x7f1420106350 2024-03-24T00:04:12.641 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:04:11.857+0000 7f141fe00700 1 --2- 172.21.15.118:0/1190192057 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f1420106350 0x7f1420106770 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f14100098a0 tx=0x7f1410009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4d8b94075719f6cf server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:04:12.641 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:04:11.857+0000 7f141ea00700 1 -- 172.21.15.118:0/1190192057 <== mon.0 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f1410004160 con 0x7f1420106350 2024-03-24T00:04:12.641 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:04:11.857+0000 7f141ea00700 1 -- 172.21.15.118:0/1190192057 <== mon.0 v2:172.21.15.118:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f1410004300 con 0x7f1420106350 2024-03-24T00:04:12.641 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:04:11.857+0000 7f141ea00700 1 -- 172.21.15.118:0/1190192057 <== mon.0 v2:172.21.15.118:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f1410011b50 con 0x7f1420106350 2024-03-24T00:04:12.642 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:04:11.858+0000 7f1426e00700 1 -- 172.21.15.118:0/1190192057 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f1420106350 msgr2=0x7f1420106770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:04:12.642 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:04:11.858+0000 7f1426e00700 1 --2- 172.21.15.118:0/1190192057 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f1420106350 0x7f1420106770 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f14100098a0 tx=0x7f1410009bb0 comp rx=0 tx=0).stop 2024-03-24T00:04:12.642 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:04:11.858+0000 7f1426e00700 1 -- 172.21.15.118:0/1190192057 shutdown_connections 2024-03-24T00:04:12.642 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:04:11.858+0000 7f1426e00700 1 --2- 172.21.15.118:0/1190192057 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f1420106350 0x7f1420106770 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:04:12.642 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:04:11.858+0000 7f1426e00700 1 -- 172.21.15.118:0/1190192057 >> 172.21.15.118:0/1190192057 conn(0x7f1420101fb0 msgr2=0x7f14201043d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:04:12.642 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:04:11.858+0000 7f1426e00700 1 -- 172.21.15.118:0/1190192057 shutdown_connections 2024-03-24T00:04:12.642 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:04:11.858+0000 7f1426e00700 1 -- 172.21.15.118:0/1190192057 wait complete. 2024-03-24T00:04:12.642 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:04:11.859+0000 7f1426e00700 1 Processor -- start 2024-03-24T00:04:12.642 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:04:11.859+0000 7f1426e00700 1 -- start start 2024-03-24T00:04:12.642 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:04:11.859+0000 7f1426e00700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f1420106350 0x7f142019dd20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:04:12.642 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:04:11.859+0000 7f1426e00700 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1420107d40 con 0x7f1420106350 2024-03-24T00:04:12.642 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:04:11.860+0000 7f141fe00700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f1420106350 0x7f142019dd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:04:12.642 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:04:11.860+0000 7f141fe00700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f1420106350 0x7f142019dd20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.118:3300/0 says I am v2:172.21.15.118:47762/0 (socket says 172.21.15.118:47762) 2024-03-24T00:04:12.642 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:04:11.860+0000 7f141fe00700 1 -- 172.21.15.118:0/4182973945 learned_addr learned my addr 172.21.15.118:0/4182973945 (peer_addr_for_me v2:172.21.15.118:0/0) 2024-03-24T00:04:12.643 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:04:11.860+0000 7f141fe00700 1 -- 172.21.15.118:0/4182973945 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f142019e260 con 0x7f1420106350 2024-03-24T00:04:12.643 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:04:11.860+0000 7f141fe00700 1 --2- 172.21.15.118:0/4182973945 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f1420106350 0x7f142019dd20 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f1410011b60 tx=0x7f1410011c40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:04:12.643 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:04:11.861+0000 7f141cc00700 1 -- 172.21.15.118:0/4182973945 <== mon.0 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f1410003970 con 0x7f1420106350 2024-03-24T00:04:12.643 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:04:11.861+0000 7f1426e00700 1 -- 172.21.15.118:0/4182973945 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f142019e4f0 con 0x7f1420106350 2024-03-24T00:04:12.643 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:04:11.861+0000 7f141cc00700 1 -- 172.21.15.118:0/4182973945 <== mon.0 v2:172.21.15.118:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f1410003b10 con 0x7f1420106350 2024-03-24T00:04:12.643 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:04:11.861+0000 7f141cc00700 1 -- 172.21.15.118:0/4182973945 <== mon.0 v2:172.21.15.118:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f1410024610 con 0x7f1420106350 2024-03-24T00:04:12.643 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:04:11.861+0000 7f1426e00700 1 -- 172.21.15.118:0/4182973945 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f14201a11d0 con 0x7f1420106350 2024-03-24T00:04:12.643 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:04:11.861+0000 7f141cc00700 1 -- 172.21.15.118:0/4182973945 <== mon.0 v2:172.21.15.118:3300/0 4 ==== mgrmap(e 1) v1 ==== 807+0+0 (secure 0 0 0) 0x7f1410024ca0 con 0x7f1420106350 2024-03-24T00:04:12.643 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:04:11.861+0000 7f141cc00700 1 -- 172.21.15.118:0/4182973945 <== mon.0 v2:172.21.15.118:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (secure 0 0 0) 0x7f1410011e00 con 0x7f1420106350 2024-03-24T00:04:12.643 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:04:11.862+0000 7f1426e00700 1 -- 172.21.15.118:0/4182973945 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f14200637c0 con 0x7f1420106350 2024-03-24T00:04:12.643 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:04:11.865+0000 7f141cc00700 1 -- 172.21.15.118:0/4182973945 <== mon.0 v2:172.21.15.118:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (secure 0 0 0) 0x7f1410043dd0 con 0x7f1420106350 2024-03-24T00:04:12.643 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:04:11.944+0000 7f1426e00700 1 -- 172.21.15.118:0/4182973945 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) v1 -- 0x7f142002dc80 con 0x7f1420106350 2024-03-24T00:04:12.643 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:04:11.945+0000 7f141cc00700 1 -- 172.21.15.118:0/4182973945 <== mon.0 v2:172.21.15.118:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) v1 ==== 79+0+1239 (secure 0 0 0) 0x7f1410032030 con 0x7f1420106350 2024-03-24T00:04:12.643 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:04:11.946+0000 7f1426e00700 1 -- 172.21.15.118:0/4182973945 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f1420106350 msgr2=0x7f142019dd20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:04:12.644 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:04:11.946+0000 7f1426e00700 1 --2- 172.21.15.118:0/4182973945 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f1420106350 0x7f142019dd20 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f1410011b60 tx=0x7f1410011c40 comp rx=0 tx=0).stop 2024-03-24T00:04:12.644 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:04:11.946+0000 7f1426e00700 1 -- 172.21.15.118:0/4182973945 shutdown_connections 2024-03-24T00:04:12.644 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:04:11.946+0000 7f1426e00700 1 --2- 172.21.15.118:0/4182973945 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f1420106350 0x7f142019dd20 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:04:12.644 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:04:11.946+0000 7f1426e00700 1 -- 172.21.15.118:0/4182973945 >> 172.21.15.118:0/4182973945 conn(0x7f1420101fb0 msgr2=0x7f1420194940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:04:12.644 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:04:11.947+0000 7f1426e00700 1 -- 172.21.15.118:0/4182973945 shutdown_connections 2024-03-24T00:04:12.644 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:04:11.947+0000 7f1426e00700 1 -- 172.21.15.118:0/4182973945 wait complete. 2024-03-24T00:04:12.644 INFO:teuthology.orchestra.run.smithi118.stdout:mgr not available, waiting (2/15)... 2024-03-24T00:04:12.911 INFO:journalctl@ceph.mgr.x.smithi118.stdout:Mar 24 00:04:12 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mgr-x[31762]: 2024-03-24T00:04:12.793+0000 7f8b59fb6200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-24T00:04:13.161 INFO:journalctl@ceph.mgr.x.smithi118.stdout:Mar 24 00:04:12 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mgr-x[31762]: 2024-03-24T00:04:12.910+0000 7f8b59fb6200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-24T00:04:13.445 INFO:journalctl@ceph.mgr.x.smithi118.stdout:Mar 24 00:04:13 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mgr-x[31762]: 2024-03-24T00:04:13.161+0000 7f8b59fb6200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-24T00:04:13.446 INFO:journalctl@ceph.mgr.x.smithi118.stdout:Mar 24 00:04:13 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mgr-x[31762]: 2024-03-24T00:04:13.403+0000 7f8b59fb6200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-24T00:04:14.085 INFO:journalctl@ceph.mgr.x.smithi118.stdout:Mar 24 00:04:13 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mgr-x[31762]: 2024-03-24T00:04:13.774+0000 7f8b59fb6200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-24T00:04:14.085 INFO:journalctl@ceph.mgr.x.smithi118.stdout:Mar 24 00:04:14 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mgr-x[31762]: 2024-03-24T00:04:14.085+0000 7f8b59fb6200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-24T00:04:14.695 INFO:journalctl@ceph.mgr.x.smithi118.stdout:Mar 24 00:04:14 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mgr-x[31762]: 2024-03-24T00:04:14.383+0000 7f8b59fb6200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-24T00:04:15.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:04:15 smithi118 ceph-mon[31535]: from='client.? 172.21.15.118:0/3090331090' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-03-24T00:04:15.696 INFO:journalctl@ceph.mgr.x.smithi118.stdout:Mar 24 00:04:15 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mgr-x[31762]: 2024-03-24T00:04:15.461+0000 7f8b59fb6200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-24T00:04:16.061 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stdout 2024-03-24T00:04:16.061 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stdout { 2024-03-24T00:04:16.061 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stdout "fsid": "d79a5420-e971-11ee-95cd-87774f69a715", 2024-03-24T00:04:16.061 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stdout "health": { 2024-03-24T00:04:16.061 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2024-03-24T00:04:16.061 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stdout "checks": {}, 2024-03-24T00:04:16.061 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stdout "mutes": [] 2024-03-24T00:04:16.061 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stdout }, 2024-03-24T00:04:16.061 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2024-03-24T00:04:16.061 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stdout "quorum": [ 2024-03-24T00:04:16.062 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stdout 0 2024-03-24T00:04:16.062 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stdout ], 2024-03-24T00:04:16.062 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2024-03-24T00:04:16.062 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stdout "a" 2024-03-24T00:04:16.062 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stdout ], 2024-03-24T00:04:16.062 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stdout "quorum_age": 10, 2024-03-24T00:04:16.062 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stdout "monmap": { 2024-03-24T00:04:16.062 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-24T00:04:16.062 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "squid", 2024-03-24T00:04:16.062 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stdout "num_mons": 1 2024-03-24T00:04:16.062 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stdout }, 2024-03-24T00:04:16.062 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stdout "osdmap": { 2024-03-24T00:04:16.062 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-24T00:04:16.062 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2024-03-24T00:04:16.062 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2024-03-24T00:04:16.063 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2024-03-24T00:04:16.063 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2024-03-24T00:04:16.070 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2024-03-24T00:04:16.070 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2024-03-24T00:04:16.070 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stdout }, 2024-03-24T00:04:16.070 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stdout "pgmap": { 2024-03-24T00:04:16.071 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2024-03-24T00:04:16.071 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2024-03-24T00:04:16.071 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2024-03-24T00:04:16.071 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2024-03-24T00:04:16.071 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2024-03-24T00:04:16.071 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2024-03-24T00:04:16.071 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2024-03-24T00:04:16.071 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2024-03-24T00:04:16.071 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stdout }, 2024-03-24T00:04:16.071 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stdout "fsmap": { 2024-03-24T00:04:16.071 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-24T00:04:16.072 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stdout "by_rank": [], 2024-03-24T00:04:16.072 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stdout "up:standby": 0 2024-03-24T00:04:16.072 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stdout }, 2024-03-24T00:04:16.072 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stdout "mgrmap": { 2024-03-24T00:04:16.072 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stdout "available": false, 2024-03-24T00:04:16.072 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2024-03-24T00:04:16.072 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stdout "modules": [ 2024-03-24T00:04:16.072 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stdout "iostat", 2024-03-24T00:04:16.072 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stdout "nfs", 2024-03-24T00:04:16.072 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stdout "restful" 2024-03-24T00:04:16.072 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stdout ], 2024-03-24T00:04:16.072 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stdout "services": {} 2024-03-24T00:04:16.073 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stdout }, 2024-03-24T00:04:16.073 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stdout "servicemap": { 2024-03-24T00:04:16.073 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-24T00:04:16.073 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stdout "modified": "2024-03-24T00:03:58.649100+0000", 2024-03-24T00:04:16.073 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stdout "services": {} 2024-03-24T00:04:16.073 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stdout }, 2024-03-24T00:04:16.073 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stdout "progress_events": {} 2024-03-24T00:04:16.073 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stdout } 2024-03-24T00:04:16.073 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:04:15.231+0000 7f8af9e00700 1 Processor -- start 2024-03-24T00:04:16.073 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:04:15.231+0000 7f8af9e00700 1 -- start start 2024-03-24T00:04:16.073 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:04:15.231+0000 7f8af9e00700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f8af41085e0 0x7f8af410a9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:04:16.073 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:04:15.231+0000 7f8af9e00700 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8af4075880 con 0x7f8af41085e0 2024-03-24T00:04:16.073 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:04:15.232+0000 7f8af2a00700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f8af41085e0 0x7f8af410a9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:04:16.073 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:04:15.232+0000 7f8af2a00700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f8af41085e0 0x7f8af410a9d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.118:3300/0 says I am v2:172.21.15.118:46204/0 (socket says 172.21.15.118:46204) 2024-03-24T00:04:16.074 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:04:15.232+0000 7f8af2a00700 1 -- 172.21.15.118:0/2324365985 learned_addr learned my addr 172.21.15.118:0/2324365985 (peer_addr_for_me v2:172.21.15.118:0/0) 2024-03-24T00:04:16.074 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:04:15.232+0000 7f8af2a00700 1 -- 172.21.15.118:0/2324365985 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8af410af10 con 0x7f8af41085e0 2024-03-24T00:04:16.074 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:04:15.232+0000 7f8af2a00700 1 --2- 172.21.15.118:0/2324365985 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f8af41085e0 0x7f8af410a9d0 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f8adc0098a0 tx=0x7f8adc009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=418b4407b612ca7f server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:04:16.074 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:04:15.233+0000 7f8af2000700 1 -- 172.21.15.118:0/2324365985 <== mon.0 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f8adc004160 con 0x7f8af41085e0 2024-03-24T00:04:16.074 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:04:15.233+0000 7f8af2000700 1 -- 172.21.15.118:0/2324365985 <== mon.0 v2:172.21.15.118:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f8adc004300 con 0x7f8af41085e0 2024-03-24T00:04:16.074 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:04:15.233+0000 7f8af2000700 1 -- 172.21.15.118:0/2324365985 <== mon.0 v2:172.21.15.118:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f8adc011b50 con 0x7f8af41085e0 2024-03-24T00:04:16.074 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:04:15.233+0000 7f8af9e00700 1 -- 172.21.15.118:0/2324365985 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f8af41085e0 msgr2=0x7f8af410a9d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:04:16.074 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:04:15.233+0000 7f8af9e00700 1 --2- 172.21.15.118:0/2324365985 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f8af41085e0 0x7f8af410a9d0 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f8adc0098a0 tx=0x7f8adc009bb0 comp rx=0 tx=0).stop 2024-03-24T00:04:16.074 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:04:15.233+0000 7f8af9e00700 1 -- 172.21.15.118:0/2324365985 shutdown_connections 2024-03-24T00:04:16.074 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:04:15.233+0000 7f8af9e00700 1 --2- 172.21.15.118:0/2324365985 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f8af41085e0 0x7f8af410a9d0 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:04:16.074 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:04:15.233+0000 7f8af9e00700 1 -- 172.21.15.118:0/2324365985 >> 172.21.15.118:0/2324365985 conn(0x7f8af4101fb0 msgr2=0x7f8af41043d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:04:16.074 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:04:15.234+0000 7f8af9e00700 1 -- 172.21.15.118:0/2324365985 shutdown_connections 2024-03-24T00:04:16.074 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:04:15.234+0000 7f8af9e00700 1 -- 172.21.15.118:0/2324365985 wait complete. 2024-03-24T00:04:16.075 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:04:15.235+0000 7f8af9e00700 1 Processor -- start 2024-03-24T00:04:16.075 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:04:15.235+0000 7f8af9e00700 1 -- start start 2024-03-24T00:04:16.075 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:04:15.235+0000 7f8af9e00700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f8af41085e0 0x7f8af419dd30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:04:16.075 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:04:15.235+0000 7f8af9e00700 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8af410bed0 con 0x7f8af41085e0 2024-03-24T00:04:16.075 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:04:15.235+0000 7f8af2a00700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f8af41085e0 0x7f8af419dd30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:04:16.075 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:04:15.235+0000 7f8af2a00700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f8af41085e0 0x7f8af419dd30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.118:3300/0 says I am v2:172.21.15.118:46212/0 (socket says 172.21.15.118:46212) 2024-03-24T00:04:16.075 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:04:15.235+0000 7f8af2a00700 1 -- 172.21.15.118:0/3090331090 learned_addr learned my addr 172.21.15.118:0/3090331090 (peer_addr_for_me v2:172.21.15.118:0/0) 2024-03-24T00:04:16.075 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:04:15.236+0000 7f8af2a00700 1 -- 172.21.15.118:0/3090331090 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8af419e270 con 0x7f8af41085e0 2024-03-24T00:04:16.075 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:04:15.236+0000 7f8af2a00700 1 --2- 172.21.15.118:0/3090331090 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f8af41085e0 0x7f8af419dd30 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f8adc011b60 tx=0x7f8adc011c40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:04:16.075 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:04:15.236+0000 7f8af8a00700 1 -- 172.21.15.118:0/3090331090 <== mon.0 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f8adc003970 con 0x7f8af41085e0 2024-03-24T00:04:16.075 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:04:15.236+0000 7f8af8a00700 1 -- 172.21.15.118:0/3090331090 <== mon.0 v2:172.21.15.118:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f8adc003b10 con 0x7f8af41085e0 2024-03-24T00:04:16.076 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:04:15.236+0000 7f8af9e00700 1 -- 172.21.15.118:0/3090331090 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8af419e500 con 0x7f8af41085e0 2024-03-24T00:04:16.076 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:04:15.236+0000 7f8af8a00700 1 -- 172.21.15.118:0/3090331090 <== mon.0 v2:172.21.15.118:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f8adc024610 con 0x7f8af41085e0 2024-03-24T00:04:16.076 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:04:15.237+0000 7f8af9e00700 1 -- 172.21.15.118:0/3090331090 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8af41a11f0 con 0x7f8af41085e0 2024-03-24T00:04:16.076 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:04:15.237+0000 7f8af8a00700 1 -- 172.21.15.118:0/3090331090 <== mon.0 v2:172.21.15.118:3300/0 4 ==== mgrmap(e 1) v1 ==== 807+0+0 (secure 0 0 0) 0x7f8adc024c50 con 0x7f8af41085e0 2024-03-24T00:04:16.076 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:04:15.237+0000 7f8af8a00700 1 -- 172.21.15.118:0/3090331090 <== mon.0 v2:172.21.15.118:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (secure 0 0 0) 0x7f8adc011e00 con 0x7f8af41085e0 2024-03-24T00:04:16.076 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:04:15.238+0000 7f8af9e00700 1 -- 172.21.15.118:0/3090331090 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8af40637c0 con 0x7f8af41085e0 2024-03-24T00:04:16.076 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:04:15.240+0000 7f8af8a00700 1 -- 172.21.15.118:0/3090331090 <== mon.0 v2:172.21.15.118:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+73141 (secure 0 0 0) 0x7f8adc043dd0 con 0x7f8af41085e0 2024-03-24T00:04:16.076 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:04:15.319+0000 7f8af9e00700 1 -- 172.21.15.118:0/3090331090 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) v1 -- 0x7f8af41a1790 con 0x7f8af41085e0 2024-03-24T00:04:16.076 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:04:15.320+0000 7f8af8a00700 1 -- 172.21.15.118:0/3090331090 <== mon.0 v2:172.21.15.118:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) v1 ==== 79+0+1240 (secure 0 0 0) 0x7f8adc032030 con 0x7f8af41085e0 2024-03-24T00:04:16.076 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:04:15.321+0000 7f8af9e00700 1 -- 172.21.15.118:0/3090331090 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f8af41085e0 msgr2=0x7f8af419dd30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:04:16.076 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:04:15.321+0000 7f8af9e00700 1 --2- 172.21.15.118:0/3090331090 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f8af41085e0 0x7f8af419dd30 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f8adc011b60 tx=0x7f8adc011c40 comp rx=0 tx=0).stop 2024-03-24T00:04:16.076 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:04:15.321+0000 7f8af9e00700 1 -- 172.21.15.118:0/3090331090 shutdown_connections 2024-03-24T00:04:16.076 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:04:15.321+0000 7f8af9e00700 1 --2- 172.21.15.118:0/3090331090 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f8af41085e0 0x7f8af419dd30 secure :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f8adc011b60 tx=0x7f8adc011c40 comp rx=0 tx=0).stop 2024-03-24T00:04:16.076 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:04:15.321+0000 7f8af9e00700 1 -- 172.21.15.118:0/3090331090 >> 172.21.15.118:0/3090331090 conn(0x7f8af4101fb0 msgr2=0x7f8af4102aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:04:16.077 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:04:15.321+0000 7f8af9e00700 1 -- 172.21.15.118:0/3090331090 shutdown_connections 2024-03-24T00:04:16.077 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:04:15.321+0000 7f8af9e00700 1 -- 172.21.15.118:0/3090331090 wait complete. 2024-03-24T00:04:16.077 INFO:teuthology.orchestra.run.smithi118.stdout:mgr not available, waiting (3/15)... 2024-03-24T00:04:16.445 INFO:journalctl@ceph.mgr.x.smithi118.stdout:Mar 24 00:04:16 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mgr-x[31762]: 2024-03-24T00:04:16.163+0000 7f8b59fb6200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-24T00:04:16.446 INFO:journalctl@ceph.mgr.x.smithi118.stdout:Mar 24 00:04:16 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mgr-x[31762]: 2024-03-24T00:04:16.290+0000 7f8b59fb6200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-24T00:04:16.946 INFO:journalctl@ceph.mgr.x.smithi118.stdout:Mar 24 00:04:16 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mgr-x[31762]: 2024-03-24T00:04:16.568+0000 7f8b59fb6200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-24T00:04:16.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:04:16 smithi118 ceph-mon[31535]: Activating manager daemon x 2024-03-24T00:04:16.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:04:16 smithi118 ceph-mon[31535]: mgrmap e2: x(active, starting, since 0.012676s) 2024-03-24T00:04:16.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:04:16 smithi118 ceph-mon[31535]: from='mgr.14100 172.21.15.118:0/1300317114' entity='mgr.x' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-03-24T00:04:16.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:04:16 smithi118 ceph-mon[31535]: from='mgr.14100 172.21.15.118:0/1300317114' entity='mgr.x' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-03-24T00:04:16.947 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:04:16 smithi118 ceph-mon[31535]: from='mgr.14100 172.21.15.118:0/1300317114' entity='mgr.x' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-03-24T00:04:16.947 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:04:16 smithi118 ceph-mon[31535]: from='mgr.14100 172.21.15.118:0/1300317114' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-24T00:04:16.947 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:04:16 smithi118 ceph-mon[31535]: from='mgr.14100 172.21.15.118:0/1300317114' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-03-24T00:04:16.947 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:04:16 smithi118 ceph-mon[31535]: Manager daemon x is now available 2024-03-24T00:04:16.947 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:04:16 smithi118 ceph-mon[31535]: from='mgr.14100 172.21.15.118:0/1300317114' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2024-03-24T00:04:16.947 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:04:16 smithi118 ceph-mon[31535]: from='mgr.14100 172.21.15.118:0/1300317114' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2024-03-24T00:04:16.947 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:04:16 smithi118 ceph-mon[31535]: from='mgr.14100 172.21.15.118:0/1300317114' entity='mgr.x' 2024-03-24T00:04:16.947 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:04:16 smithi118 ceph-mon[31535]: from='mgr.14100 172.21.15.118:0/1300317114' entity='mgr.x' 2024-03-24T00:04:17.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:04:17 smithi118 ceph-mon[31535]: from='mgr.14100 172.21.15.118:0/1300317114' entity='mgr.x' 2024-03-24T00:04:17.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:04:17 smithi118 ceph-mon[31535]: mgrmap e3: x(active, since 1.02225s) 2024-03-24T00:04:19.945 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:04:19 smithi118 ceph-mon[31535]: mgrmap e4: x(active, since 2s) 2024-03-24T00:04:19.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:04:19 smithi118 ceph-mon[31535]: from='client.? 172.21.15.118:0/2927107556' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-03-24T00:04:20.056 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stdout 2024-03-24T00:04:20.056 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stdout { 2024-03-24T00:04:20.056 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stdout "fsid": "d79a5420-e971-11ee-95cd-87774f69a715", 2024-03-24T00:04:20.056 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stdout "health": { 2024-03-24T00:04:20.056 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2024-03-24T00:04:20.056 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stdout "checks": {}, 2024-03-24T00:04:20.056 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stdout "mutes": [] 2024-03-24T00:04:20.057 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stdout }, 2024-03-24T00:04:20.057 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2024-03-24T00:04:20.057 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stdout "quorum": [ 2024-03-24T00:04:20.057 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stdout 0 2024-03-24T00:04:20.057 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stdout ], 2024-03-24T00:04:20.057 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2024-03-24T00:04:20.057 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stdout "a" 2024-03-24T00:04:20.057 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stdout ], 2024-03-24T00:04:20.057 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stdout "quorum_age": 14, 2024-03-24T00:04:20.057 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stdout "monmap": { 2024-03-24T00:04:20.057 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-24T00:04:20.057 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "squid", 2024-03-24T00:04:20.058 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stdout "num_mons": 1 2024-03-24T00:04:20.058 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stdout }, 2024-03-24T00:04:20.058 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stdout "osdmap": { 2024-03-24T00:04:20.058 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-24T00:04:20.058 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2024-03-24T00:04:20.058 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2024-03-24T00:04:20.058 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2024-03-24T00:04:20.058 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2024-03-24T00:04:20.058 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2024-03-24T00:04:20.058 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2024-03-24T00:04:20.058 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stdout }, 2024-03-24T00:04:20.058 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stdout "pgmap": { 2024-03-24T00:04:20.059 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2024-03-24T00:04:20.059 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2024-03-24T00:04:20.059 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2024-03-24T00:04:20.059 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2024-03-24T00:04:20.059 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2024-03-24T00:04:20.059 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2024-03-24T00:04:20.059 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2024-03-24T00:04:20.059 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2024-03-24T00:04:20.059 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stdout }, 2024-03-24T00:04:20.059 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stdout "fsmap": { 2024-03-24T00:04:20.059 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-24T00:04:20.059 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stdout "by_rank": [], 2024-03-24T00:04:20.060 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stdout "up:standby": 0 2024-03-24T00:04:20.060 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stdout }, 2024-03-24T00:04:20.060 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stdout "mgrmap": { 2024-03-24T00:04:20.060 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stdout "available": true, 2024-03-24T00:04:20.060 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2024-03-24T00:04:20.060 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stdout "modules": [ 2024-03-24T00:04:20.060 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stdout "iostat", 2024-03-24T00:04:20.060 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stdout "nfs", 2024-03-24T00:04:20.060 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stdout "restful" 2024-03-24T00:04:20.060 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stdout ], 2024-03-24T00:04:20.060 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stdout "services": {} 2024-03-24T00:04:20.060 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stdout }, 2024-03-24T00:04:20.061 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stdout "servicemap": { 2024-03-24T00:04:20.061 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-24T00:04:20.061 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stdout "modified": "2024-03-24T00:03:58.649100+0000", 2024-03-24T00:04:20.061 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stdout "services": {} 2024-03-24T00:04:20.061 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stdout }, 2024-03-24T00:04:20.061 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stdout "progress_events": {} 2024-03-24T00:04:20.061 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stdout } 2024-03-24T00:04:20.061 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:04:18.655+0000 7f2430a00700 1 Processor -- start 2024-03-24T00:04:20.061 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:04:18.656+0000 7f2430a00700 1 -- start start 2024-03-24T00:04:20.061 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:04:18.656+0000 7f2430a00700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f242c109bf0 0x7f242c10a010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:04:20.061 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:04:18.656+0000 7f2430a00700 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f242c10a5e0 con 0x7f242c109bf0 2024-03-24T00:04:20.061 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:04:18.656+0000 7f242b400700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f242c109bf0 0x7f242c10a010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:04:20.062 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:04:18.656+0000 7f242b400700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f242c109bf0 0x7f242c10a010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.118:3300/0 says I am v2:172.21.15.118:46292/0 (socket says 172.21.15.118:46292) 2024-03-24T00:04:20.062 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:04:18.656+0000 7f242b400700 1 -- 172.21.15.118:0/2871227025 learned_addr learned my addr 172.21.15.118:0/2871227025 (peer_addr_for_me v2:172.21.15.118:0/0) 2024-03-24T00:04:20.062 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:04:18.657+0000 7f242b400700 1 -- 172.21.15.118:0/2871227025 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f242c10ad50 con 0x7f242c109bf0 2024-03-24T00:04:20.062 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:04:18.657+0000 7f242b400700 1 --2- 172.21.15.118:0/2871227025 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f242c109bf0 0x7f242c10a010 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f241c0098a0 tx=0x7f241c009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1f881efbc536167 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:04:20.062 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:04:18.657+0000 7f242aa00700 1 -- 172.21.15.118:0/2871227025 <== mon.0 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f241c004160 con 0x7f242c109bf0 2024-03-24T00:04:20.062 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:04:18.657+0000 7f242aa00700 1 -- 172.21.15.118:0/2871227025 <== mon.0 v2:172.21.15.118:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f241c004300 con 0x7f242c109bf0 2024-03-24T00:04:20.062 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:04:18.657+0000 7f242aa00700 1 -- 172.21.15.118:0/2871227025 <== mon.0 v2:172.21.15.118:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f241c011b50 con 0x7f242c109bf0 2024-03-24T00:04:20.062 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:04:18.658+0000 7f2430a00700 1 -- 172.21.15.118:0/2871227025 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f242c109bf0 msgr2=0x7f242c10a010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:04:20.062 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:04:18.658+0000 7f2430a00700 1 --2- 172.21.15.118:0/2871227025 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f242c109bf0 0x7f242c10a010 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f241c0098a0 tx=0x7f241c009bb0 comp rx=0 tx=0).stop 2024-03-24T00:04:20.062 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:04:18.658+0000 7f2430a00700 1 -- 172.21.15.118:0/2871227025 shutdown_connections 2024-03-24T00:04:20.062 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:04:18.658+0000 7f2430a00700 1 --2- 172.21.15.118:0/2871227025 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f242c109bf0 0x7f242c10a010 unknown :-1 s=CLOSED pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:04:20.063 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:04:18.658+0000 7f2430a00700 1 -- 172.21.15.118:0/2871227025 >> 172.21.15.118:0/2871227025 conn(0x7f242c1053d0 msgr2=0x7f242c1077c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:04:20.063 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:04:18.658+0000 7f2430a00700 1 -- 172.21.15.118:0/2871227025 shutdown_connections 2024-03-24T00:04:20.063 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:04:18.658+0000 7f2430a00700 1 -- 172.21.15.118:0/2871227025 wait complete. 2024-03-24T00:04:20.063 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:04:18.659+0000 7f2430a00700 1 Processor -- start 2024-03-24T00:04:20.063 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:04:18.660+0000 7f2430a00700 1 -- start start 2024-03-24T00:04:20.063 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:04:18.660+0000 7f2430a00700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f242c109bf0 0x7f242c19db70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:04:20.063 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:04:18.660+0000 7f2430a00700 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f242c10e100 con 0x7f242c109bf0 2024-03-24T00:04:20.063 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:04:18.660+0000 7f242b400700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f242c109bf0 0x7f242c19db70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:04:20.063 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:04:18.660+0000 7f242b400700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f242c109bf0 0x7f242c19db70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.118:3300/0 says I am v2:172.21.15.118:46294/0 (socket says 172.21.15.118:46294) 2024-03-24T00:04:20.063 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:04:18.660+0000 7f242b400700 1 -- 172.21.15.118:0/2927107556 learned_addr learned my addr 172.21.15.118:0/2927107556 (peer_addr_for_me v2:172.21.15.118:0/0) 2024-03-24T00:04:20.063 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:04:18.660+0000 7f242b400700 1 -- 172.21.15.118:0/2927107556 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f242c19e0b0 con 0x7f242c109bf0 2024-03-24T00:04:20.063 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:04:18.661+0000 7f242b400700 1 --2- 172.21.15.118:0/2927107556 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f242c109bf0 0x7f242c19db70 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f241c011850 tx=0x7f241c011c10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:04:20.064 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:04:18.661+0000 7f2428c00700 1 -- 172.21.15.118:0/2927107556 <== mon.0 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f241c003af0 con 0x7f242c109bf0 2024-03-24T00:04:20.064 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:04:18.661+0000 7f2428c00700 1 -- 172.21.15.118:0/2927107556 <== mon.0 v2:172.21.15.118:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f241c020070 con 0x7f242c109bf0 2024-03-24T00:04:20.064 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:04:18.661+0000 7f2430a00700 1 -- 172.21.15.118:0/2927107556 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f242c19e340 con 0x7f242c109bf0 2024-03-24T00:04:20.064 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:04:18.661+0000 7f2428c00700 1 -- 172.21.15.118:0/2927107556 <== mon.0 v2:172.21.15.118:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f241c0195a0 con 0x7f242c109bf0 2024-03-24T00:04:20.064 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:04:18.661+0000 7f2430a00700 1 -- 172.21.15.118:0/2927107556 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f242c1a1030 con 0x7f242c109bf0 2024-03-24T00:04:20.064 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:04:18.662+0000 7f2428c00700 1 -- 172.21.15.118:0/2927107556 <== mon.0 v2:172.21.15.118:3300/0 4 ==== mgrmap(e 4) v1 ==== 46095+0+0 (secure 0 0 0) 0x7f241c019850 con 0x7f242c109bf0 2024-03-24T00:04:20.064 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:04:18.662+0000 7f2430a00700 1 -- 172.21.15.118:0/2927107556 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f242c0637c0 con 0x7f242c109bf0 2024-03-24T00:04:20.064 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:04:18.663+0000 7f2428c00700 1 --2- 172.21.15.118:0/2927107556 >> [v2:172.21.15.118:6800/1463147955,v1:172.21.15.118:6801/1463147955] conn(0x7f2410038f20 0x7f241003b3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:04:20.064 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:04:18.663+0000 7f2428c00700 1 -- 172.21.15.118:0/2927107556 <== mon.0 v2:172.21.15.118:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (secure 0 0 0) 0x7f241c011e00 con 0x7f242c109bf0 2024-03-24T00:04:20.064 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:04:18.663+0000 7f2423e00700 1 --2- 172.21.15.118:0/2927107556 >> [v2:172.21.15.118:6800/1463147955,v1:172.21.15.118:6801/1463147955] conn(0x7f2410038f20 0x7f241003b3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:04:20.064 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:04:18.664+0000 7f2423e00700 1 --2- 172.21.15.118:0/2927107556 >> [v2:172.21.15.118:6800/1463147955,v1:172.21.15.118:6801/1463147955] conn(0x7f2410038f20 0x7f241003b3e0 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f2414006fd0 tx=0x7f2414006e40 comp rx=0 tx=0).ready entity=mgr.14100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:04:20.065 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:04:18.669+0000 7f2428c00700 1 -- 172.21.15.118:0/2927107556 <== mon.0 v2:172.21.15.118:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f241c003c90 con 0x7f242c109bf0 2024-03-24T00:04:20.065 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:04:18.978+0000 7f2430a00700 1 -- 172.21.15.118:0/2927107556 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) v1 -- 0x7f242c1a1200 con 0x7f242c109bf0 2024-03-24T00:04:20.065 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:04:18.979+0000 7f2428c00700 1 -- 172.21.15.118:0/2927107556 <== mon.0 v2:172.21.15.118:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) v1 ==== 79+0+1239 (secure 0 0 0) 0x7f242c1a1200 con 0x7f242c109bf0 2024-03-24T00:04:20.065 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:04:18.982+0000 7f2430a00700 1 -- 172.21.15.118:0/2927107556 >> [v2:172.21.15.118:6800/1463147955,v1:172.21.15.118:6801/1463147955] conn(0x7f2410038f20 msgr2=0x7f241003b3e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:04:20.065 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:04:18.982+0000 7f2430a00700 1 --2- 172.21.15.118:0/2927107556 >> [v2:172.21.15.118:6800/1463147955,v1:172.21.15.118:6801/1463147955] conn(0x7f2410038f20 0x7f241003b3e0 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f2414006fd0 tx=0x7f2414006e40 comp rx=0 tx=0).stop 2024-03-24T00:04:20.065 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:04:18.982+0000 7f2430a00700 1 -- 172.21.15.118:0/2927107556 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f242c109bf0 msgr2=0x7f242c19db70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:04:20.065 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:04:18.982+0000 7f2430a00700 1 --2- 172.21.15.118:0/2927107556 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f242c109bf0 0x7f242c19db70 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f241c011850 tx=0x7f241c011c10 comp rx=0 tx=0).stop 2024-03-24T00:04:20.065 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:04:18.982+0000 7f2430a00700 1 -- 172.21.15.118:0/2927107556 shutdown_connections 2024-03-24T00:04:20.065 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:04:18.982+0000 7f2430a00700 1 --2- 172.21.15.118:0/2927107556 >> [v2:172.21.15.118:6800/1463147955,v1:172.21.15.118:6801/1463147955] conn(0x7f2410038f20 0x7f241003b3e0 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:04:20.065 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:04:18.982+0000 7f2430a00700 1 --2- 172.21.15.118:0/2927107556 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f242c109bf0 0x7f242c19db70 unknown :-1 s=CLOSED pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:04:20.065 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:04:18.982+0000 7f2430a00700 1 -- 172.21.15.118:0/2927107556 >> 172.21.15.118:0/2927107556 conn(0x7f242c1053d0 msgr2=0x7f242c106f00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:04:20.065 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:04:18.982+0000 7f2430a00700 1 -- 172.21.15.118:0/2927107556 shutdown_connections 2024-03-24T00:04:20.066 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:04:18.982+0000 7f2430a00700 1 -- 172.21.15.118:0/2927107556 wait complete. 2024-03-24T00:04:20.066 INFO:teuthology.orchestra.run.smithi118.stdout:mgr is available 2024-03-24T00:04:21.268 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:04:20 smithi118 ceph-mon[31535]: from='client.? 172.21.15.118:0/151556022' entity='client.admin' cmd=[{"prefix": "config assimilate-conf"}]: dispatch 2024-03-24T00:04:21.571 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stdout 2024-03-24T00:04:21.571 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stdout [global] 2024-03-24T00:04:21.571 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stdout fsid = d79a5420-e971-11ee-95cd-87774f69a715 2024-03-24T00:04:21.571 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stdout mon_osd_allow_pg_remap = true 2024-03-24T00:04:21.572 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stdout mon_osd_allow_primary_affinity = true 2024-03-24T00:04:21.572 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stdout mon_warn_on_no_sortbitwise = false 2024-03-24T00:04:21.572 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stdout osd_crush_chooseleaf_type = 0 2024-03-24T00:04:21.572 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stdout 2024-03-24T00:04:21.572 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stdout [mgr] 2024-03-24T00:04:21.572 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stdout mgr/telemetry/nag = false 2024-03-24T00:04:21.572 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stdout 2024-03-24T00:04:21.572 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stdout [osd] 2024-03-24T00:04:21.572 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stdout mon_osd_backfillfull_ratio = 0.85 2024-03-24T00:04:21.572 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stdout mon_osd_full_ratio = 0.9 2024-03-24T00:04:21.572 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stdout mon_osd_nearfull_ratio = 0.8 2024-03-24T00:04:21.572 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stdout osd_map_max_advance = 10 2024-03-24T00:04:21.572 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stdout osd_sloppy_crc = true 2024-03-24T00:04:21.572 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:04:20.681+0000 7f54ec800700 1 Processor -- start 2024-03-24T00:04:21.573 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:04:20.681+0000 7f54ec800700 1 -- start start 2024-03-24T00:04:21.573 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:04:20.682+0000 7f54ec800700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f54e407cdb0 0x7f54e407b210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:04:21.573 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:04:20.682+0000 7f54ec800700 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f54e407b7e0 con 0x7f54e407cdb0 2024-03-24T00:04:21.573 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:04:20.682+0000 7f54e9e00700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f54e407cdb0 0x7f54e407b210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:04:21.573 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:04:20.682+0000 7f54e9e00700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f54e407cdb0 0x7f54e407b210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.118:3300/0 says I am v2:172.21.15.118:46304/0 (socket says 172.21.15.118:46304) 2024-03-24T00:04:21.573 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:04:20.682+0000 7f54e9e00700 1 -- 172.21.15.118:0/473005477 learned_addr learned my addr 172.21.15.118:0/473005477 (peer_addr_for_me v2:172.21.15.118:0/0) 2024-03-24T00:04:21.573 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:04:20.683+0000 7f54e9e00700 1 -- 172.21.15.118:0/473005477 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f54e407b960 con 0x7f54e407cdb0 2024-03-24T00:04:21.573 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:04:20.683+0000 7f54e9e00700 1 --2- 172.21.15.118:0/473005477 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f54e407cdb0 0x7f54e407b210 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f54d80098a0 tx=0x7f54d8009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b93ac8d69fc96ee server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:04:21.573 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:04:20.683+0000 7f54e8a00700 1 -- 172.21.15.118:0/473005477 <== mon.0 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f54d8004160 con 0x7f54e407cdb0 2024-03-24T00:04:21.573 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:04:20.684+0000 7f54e8a00700 1 -- 172.21.15.118:0/473005477 <== mon.0 v2:172.21.15.118:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f54d8004300 con 0x7f54e407cdb0 2024-03-24T00:04:21.573 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:04:20.684+0000 7f54e8a00700 1 -- 172.21.15.118:0/473005477 <== mon.0 v2:172.21.15.118:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f54d8011b50 con 0x7f54e407cdb0 2024-03-24T00:04:21.573 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:04:20.684+0000 7f54ec800700 1 -- 172.21.15.118:0/473005477 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f54e407cdb0 msgr2=0x7f54e407b210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:04:21.573 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:04:20.684+0000 7f54ec800700 1 --2- 172.21.15.118:0/473005477 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f54e407cdb0 0x7f54e407b210 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f54d80098a0 tx=0x7f54d8009bb0 comp rx=0 tx=0).stop 2024-03-24T00:04:21.573 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:04:20.684+0000 7f54ec800700 1 -- 172.21.15.118:0/473005477 shutdown_connections 2024-03-24T00:04:21.574 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:04:20.684+0000 7f54ec800700 1 --2- 172.21.15.118:0/473005477 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f54e407cdb0 0x7f54e407b210 unknown :-1 s=CLOSED pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:04:21.574 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:04:20.684+0000 7f54ec800700 1 -- 172.21.15.118:0/473005477 >> 172.21.15.118:0/473005477 conn(0x7f54e4103080 msgr2=0x7f54e41054c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:04:21.574 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:04:20.684+0000 7f54ec800700 1 -- 172.21.15.118:0/473005477 shutdown_connections 2024-03-24T00:04:21.574 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:04:20.684+0000 7f54ec800700 1 -- 172.21.15.118:0/473005477 wait complete. 2024-03-24T00:04:21.574 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:04:20.685+0000 7f54ec800700 1 Processor -- start 2024-03-24T00:04:21.574 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:04:20.685+0000 7f54ec800700 1 -- start start 2024-03-24T00:04:21.577 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:04:20.686+0000 7f54ec800700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f54e407cdb0 0x7f54e419dce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:04:21.577 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:04:20.686+0000 7f54ec800700 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f54e410e0f0 con 0x7f54e407cdb0 2024-03-24T00:04:21.577 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:04:20.686+0000 7f54e9e00700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f54e407cdb0 0x7f54e419dce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:04:21.578 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:04:20.686+0000 7f54e9e00700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f54e407cdb0 0x7f54e419dce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.118:3300/0 says I am v2:172.21.15.118:46318/0 (socket says 172.21.15.118:46318) 2024-03-24T00:04:21.578 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:04:20.686+0000 7f54e9e00700 1 -- 172.21.15.118:0/151556022 learned_addr learned my addr 172.21.15.118:0/151556022 (peer_addr_for_me v2:172.21.15.118:0/0) 2024-03-24T00:04:21.578 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:04:20.686+0000 7f54e9e00700 1 -- 172.21.15.118:0/151556022 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f54e419e220 con 0x7f54e407cdb0 2024-03-24T00:04:21.578 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:04:20.687+0000 7f54e9e00700 1 --2- 172.21.15.118:0/151556022 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f54e407cdb0 0x7f54e419dce0 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7f54d8011670 tx=0x7f54d8011b60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:04:21.578 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:04:20.687+0000 7f54dea00700 1 -- 172.21.15.118:0/151556022 <== mon.0 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f54d80039c0 con 0x7f54e407cdb0 2024-03-24T00:04:21.578 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:04:20.687+0000 7f54dea00700 1 -- 172.21.15.118:0/151556022 <== mon.0 v2:172.21.15.118:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f54d8011e50 con 0x7f54e407cdb0 2024-03-24T00:04:21.578 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:04:20.687+0000 7f54ec800700 1 -- 172.21.15.118:0/151556022 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f54e419e4b0 con 0x7f54e407cdb0 2024-03-24T00:04:21.578 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:04:20.687+0000 7f54dea00700 1 -- 172.21.15.118:0/151556022 <== mon.0 v2:172.21.15.118:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f54d801a5c0 con 0x7f54e407cdb0 2024-03-24T00:04:21.578 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:04:20.687+0000 7f54ec800700 1 -- 172.21.15.118:0/151556022 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f54e41a1190 con 0x7f54e407cdb0 2024-03-24T00:04:21.578 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:04:20.688+0000 7f54dea00700 1 -- 172.21.15.118:0/151556022 <== mon.0 v2:172.21.15.118:3300/0 4 ==== mgrmap(e 4) v1 ==== 46095+0+0 (secure 0 0 0) 0x7f54d8024430 con 0x7f54e407cdb0 2024-03-24T00:04:21.578 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:04:20.688+0000 7f54ec800700 1 -- 172.21.15.118:0/151556022 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f54e40637c0 con 0x7f54e407cdb0 2024-03-24T00:04:21.578 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:04:20.689+0000 7f54dea00700 1 --2- 172.21.15.118:0/151556022 >> [v2:172.21.15.118:6800/1463147955,v1:172.21.15.118:6801/1463147955] conn(0x7f54cc038ed0 0x7f54cc03b390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:04:21.579 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:04:20.689+0000 7f54dea00700 1 -- 172.21.15.118:0/151556022 <== mon.0 v2:172.21.15.118:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (secure 0 0 0) 0x7f54d804d240 con 0x7f54e407cdb0 2024-03-24T00:04:21.579 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:04:20.689+0000 7f54e9400700 1 --2- 172.21.15.118:0/151556022 >> [v2:172.21.15.118:6800/1463147955,v1:172.21.15.118:6801/1463147955] conn(0x7f54cc038ed0 0x7f54cc03b390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:04:21.579 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:04:20.690+0000 7f54e9400700 1 --2- 172.21.15.118:0/151556022 >> [v2:172.21.15.118:6800/1463147955,v1:172.21.15.118:6801/1463147955] conn(0x7f54cc038ed0 0x7f54cc03b390 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f54d4006fd0 tx=0x7f54d4006e40 comp rx=0 tx=0).ready entity=mgr.14100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:04:21.579 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:04:20.694+0000 7f54dea00700 1 -- 172.21.15.118:0/151556022 <== mon.0 v2:172.21.15.118:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f54d804cca0 con 0x7f54e407cdb0 2024-03-24T00:04:21.579 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:04:20.896+0000 7f54ec800700 1 -- 172.21.15.118:0/151556022 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_command({"prefix": "config assimilate-conf"} v 0) v1 -- 0x7f54e41a1360 con 0x7f54e407cdb0 2024-03-24T00:04:21.579 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:04:20.898+0000 7f54dea00700 1 -- 172.21.15.118:0/151556022 <== mon.0 v2:172.21.15.118:3300/0 7 ==== mon_command_ack([{"prefix": "config assimilate-conf"}]=0 v3) v1 ==== 70+0+373 (secure 0 0 0) 0x7f54e41a1360 con 0x7f54e407cdb0 2024-03-24T00:04:21.579 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:04:20.901+0000 7f54ec800700 1 -- 172.21.15.118:0/151556022 >> [v2:172.21.15.118:6800/1463147955,v1:172.21.15.118:6801/1463147955] conn(0x7f54cc038ed0 msgr2=0x7f54cc03b390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:04:21.579 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:04:20.901+0000 7f54ec800700 1 --2- 172.21.15.118:0/151556022 >> [v2:172.21.15.118:6800/1463147955,v1:172.21.15.118:6801/1463147955] conn(0x7f54cc038ed0 0x7f54cc03b390 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f54d4006fd0 tx=0x7f54d4006e40 comp rx=0 tx=0).stop 2024-03-24T00:04:21.579 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:04:20.901+0000 7f54ec800700 1 -- 172.21.15.118:0/151556022 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f54e407cdb0 msgr2=0x7f54e419dce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:04:21.579 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:04:20.901+0000 7f54ec800700 1 --2- 172.21.15.118:0/151556022 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f54e407cdb0 0x7f54e419dce0 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7f54d8011670 tx=0x7f54d8011b60 comp rx=0 tx=0).stop 2024-03-24T00:04:21.579 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:04:20.901+0000 7f54ec800700 1 -- 172.21.15.118:0/151556022 shutdown_connections 2024-03-24T00:04:21.579 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:04:20.901+0000 7f54ec800700 1 --2- 172.21.15.118:0/151556022 >> [v2:172.21.15.118:6800/1463147955,v1:172.21.15.118:6801/1463147955] conn(0x7f54cc038ed0 0x7f54cc03b390 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:04:21.579 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:04:20.901+0000 7f54ec800700 1 --2- 172.21.15.118:0/151556022 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f54e407cdb0 0x7f54e419dce0 unknown :-1 s=CLOSED pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:04:21.579 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:04:20.901+0000 7f54ec800700 1 -- 172.21.15.118:0/151556022 >> 172.21.15.118:0/151556022 conn(0x7f54e4103080 msgr2=0x7f54e4103c30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:04:21.579 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:04:20.902+0000 7f54ec800700 1 -- 172.21.15.118:0/151556022 shutdown_connections 2024-03-24T00:04:21.579 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:04:20.902+0000 7f54ec800700 1 -- 172.21.15.118:0/151556022 wait complete. 2024-03-24T00:04:21.580 INFO:teuthology.orchestra.run.smithi118.stdout:Enabling cephadm module... 2024-03-24T00:04:22.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:04:22 smithi118 ceph-mon[31535]: from='client.? 172.21.15.118:0/3859313457' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "cephadm"}]: dispatch 2024-03-24T00:04:22.696 INFO:journalctl@ceph.mgr.x.smithi118.stdout:Mar 24 00:04:22 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mgr-x[31762]: ignoring --setuser ceph since I am not root 2024-03-24T00:04:22.696 INFO:journalctl@ceph.mgr.x.smithi118.stdout:Mar 24 00:04:22 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mgr-x[31762]: ignoring --setgroup ceph since I am not root 2024-03-24T00:04:23.199 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:04:22.137+0000 7f93ada00700 1 Processor -- start 2024-03-24T00:04:23.199 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:04:22.137+0000 7f93ada00700 1 -- start start 2024-03-24T00:04:23.200 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:04:22.137+0000 7f93ada00700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f93a8109db0 0x7f93a810a1d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:04:23.200 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:04:22.137+0000 7f93ada00700 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f93a810a7a0 con 0x7f93a8109db0 2024-03-24T00:04:23.200 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:04:22.138+0000 7f93a6a00700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f93a8109db0 0x7f93a810a1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:04:23.200 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:04:22.138+0000 7f93a6a00700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f93a8109db0 0x7f93a810a1d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.118:3300/0 says I am v2:172.21.15.118:46330/0 (socket says 172.21.15.118:46330) 2024-03-24T00:04:23.200 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:04:22.138+0000 7f93a6a00700 1 -- 172.21.15.118:0/948802885 learned_addr learned my addr 172.21.15.118:0/948802885 (peer_addr_for_me v2:172.21.15.118:0/0) 2024-03-24T00:04:23.200 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:04:22.138+0000 7f93a6a00700 1 -- 172.21.15.118:0/948802885 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f93a810af10 con 0x7f93a8109db0 2024-03-24T00:04:23.200 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:04:22.139+0000 7f93a6a00700 1 --2- 172.21.15.118:0/948802885 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f93a8109db0 0x7f93a810a1d0 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f93980098a0 tx=0x7f9398009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=eb6652dde7ff51c5 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:04:23.200 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:04:22.139+0000 7f93a5600700 1 -- 172.21.15.118:0/948802885 <== mon.0 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f9398004160 con 0x7f93a8109db0 2024-03-24T00:04:23.200 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:04:22.139+0000 7f93a5600700 1 -- 172.21.15.118:0/948802885 <== mon.0 v2:172.21.15.118:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f9398004300 con 0x7f93a8109db0 2024-03-24T00:04:23.200 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:04:22.139+0000 7f93a5600700 1 -- 172.21.15.118:0/948802885 <== mon.0 v2:172.21.15.118:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f9398011b50 con 0x7f93a8109db0 2024-03-24T00:04:23.200 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:04:22.139+0000 7f93ada00700 1 -- 172.21.15.118:0/948802885 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f93a8109db0 msgr2=0x7f93a810a1d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:04:23.200 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:04:22.139+0000 7f93ada00700 1 --2- 172.21.15.118:0/948802885 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f93a8109db0 0x7f93a810a1d0 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f93980098a0 tx=0x7f9398009bb0 comp rx=0 tx=0).stop 2024-03-24T00:04:23.200 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:04:22.140+0000 7f93ada00700 1 -- 172.21.15.118:0/948802885 shutdown_connections 2024-03-24T00:04:23.201 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:04:22.140+0000 7f93ada00700 1 --2- 172.21.15.118:0/948802885 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f93a8109db0 0x7f93a810a1d0 unknown :-1 s=CLOSED pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:04:23.201 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:04:22.140+0000 7f93ada00700 1 -- 172.21.15.118:0/948802885 >> 172.21.15.118:0/948802885 conn(0x7f93a807d220 msgr2=0x7f93a807d630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:04:23.201 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:04:22.140+0000 7f93ada00700 1 -- 172.21.15.118:0/948802885 shutdown_connections 2024-03-24T00:04:23.201 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:04:22.140+0000 7f93ada00700 1 -- 172.21.15.118:0/948802885 wait complete. 2024-03-24T00:04:23.201 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:04:22.141+0000 7f93ada00700 1 Processor -- start 2024-03-24T00:04:23.201 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:04:22.141+0000 7f93ada00700 1 -- start start 2024-03-24T00:04:23.201 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:04:22.141+0000 7f93ada00700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f93a8109db0 0x7f93a819dd30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:04:23.201 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:04:22.141+0000 7f93ada00700 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f93a810e2c0 con 0x7f93a8109db0 2024-03-24T00:04:23.201 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:04:22.141+0000 7f93a6a00700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f93a8109db0 0x7f93a819dd30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:04:23.201 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:04:22.142+0000 7f93a6a00700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f93a8109db0 0x7f93a819dd30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.118:3300/0 says I am v2:172.21.15.118:46342/0 (socket says 172.21.15.118:46342) 2024-03-24T00:04:23.201 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:04:22.142+0000 7f93a6a00700 1 -- 172.21.15.118:0/3859313457 learned_addr learned my addr 172.21.15.118:0/3859313457 (peer_addr_for_me v2:172.21.15.118:0/0) 2024-03-24T00:04:23.201 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:04:22.142+0000 7f93a6a00700 1 -- 172.21.15.118:0/3859313457 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f93a819e270 con 0x7f93a8109db0 2024-03-24T00:04:23.201 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:04:22.142+0000 7f93a6a00700 1 --2- 172.21.15.118:0/3859313457 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f93a8109db0 0x7f93a819dd30 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f9398011850 tx=0x7f9398011c10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:04:23.201 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:04:22.142+0000 7f939f400700 1 -- 172.21.15.118:0/3859313457 <== mon.0 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f9398003af0 con 0x7f93a8109db0 2024-03-24T00:04:23.202 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:04:22.142+0000 7f939f400700 1 -- 172.21.15.118:0/3859313457 <== mon.0 v2:172.21.15.118:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f9398020070 con 0x7f93a8109db0 2024-03-24T00:04:23.202 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:04:22.142+0000 7f939f400700 1 -- 172.21.15.118:0/3859313457 <== mon.0 v2:172.21.15.118:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f93980195a0 con 0x7f93a8109db0 2024-03-24T00:04:23.202 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:04:22.142+0000 7f93ada00700 1 -- 172.21.15.118:0/3859313457 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f93a819e500 con 0x7f93a8109db0 2024-03-24T00:04:23.202 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:04:22.143+0000 7f93ada00700 1 -- 172.21.15.118:0/3859313457 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f93a81a11f0 con 0x7f93a8109db0 2024-03-24T00:04:23.202 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:04:22.144+0000 7f93ada00700 1 -- 172.21.15.118:0/3859313457 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f93a80637c0 con 0x7f93a8109db0 2024-03-24T00:04:23.202 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:04:22.144+0000 7f939f400700 1 -- 172.21.15.118:0/3859313457 <== mon.0 v2:172.21.15.118:3300/0 4 ==== mgrmap(e 4) v1 ==== 46095+0+0 (secure 0 0 0) 0x7f9398019850 con 0x7f93a8109db0 2024-03-24T00:04:23.202 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:04:22.145+0000 7f939f400700 1 --2- 172.21.15.118:0/3859313457 >> [v2:172.21.15.118:6800/1463147955,v1:172.21.15.118:6801/1463147955] conn(0x7f938c038f20 0x7f938c03b3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:04:23.202 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:04:22.145+0000 7f939f400700 1 -- 172.21.15.118:0/3859313457 <== mon.0 v2:172.21.15.118:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (secure 0 0 0) 0x7f9398011e00 con 0x7f93a8109db0 2024-03-24T00:04:23.202 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:04:22.145+0000 7f93a6000700 1 --2- 172.21.15.118:0/3859313457 >> [v2:172.21.15.118:6800/1463147955,v1:172.21.15.118:6801/1463147955] conn(0x7f938c038f20 0x7f938c03b3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:04:23.202 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:04:22.146+0000 7f93a6000700 1 --2- 172.21.15.118:0/3859313457 >> [v2:172.21.15.118:6800/1463147955,v1:172.21.15.118:6801/1463147955] conn(0x7f938c038f20 0x7f938c03b3e0 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f9394006fd0 tx=0x7f9394006e40 comp rx=0 tx=0).ready entity=mgr.14100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:04:23.202 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:04:22.150+0000 7f939f400700 1 -- 172.21.15.118:0/3859313457 <== mon.0 v2:172.21.15.118:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f9398003c90 con 0x7f93a8109db0 2024-03-24T00:04:23.202 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:04:22.427+0000 7f93ada00700 1 -- 172.21.15.118:0/3859313457 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_command({"prefix": "mgr module enable", "module": "cephadm"} v 0) v1 -- 0x7f93a81a13c0 con 0x7f93a8109db0 2024-03-24T00:04:23.202 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:04:22.480+0000 7f939f400700 1 -- 172.21.15.118:0/3859313457 <== mon.0 v2:172.21.15.118:3300/0 7 ==== mon_command_ack([{"prefix": "mgr module enable", "module": "cephadm"}]=0 v5) v1 ==== 86+0+0 (secure 0 0 0) 0x7f93a81a13c0 con 0x7f93a8109db0 2024-03-24T00:04:23.202 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:04:22.483+0000 7f93ada00700 1 -- 172.21.15.118:0/3859313457 >> [v2:172.21.15.118:6800/1463147955,v1:172.21.15.118:6801/1463147955] conn(0x7f938c038f20 msgr2=0x7f938c03b3e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:04:23.203 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:04:22.483+0000 7f93ada00700 1 --2- 172.21.15.118:0/3859313457 >> [v2:172.21.15.118:6800/1463147955,v1:172.21.15.118:6801/1463147955] conn(0x7f938c038f20 0x7f938c03b3e0 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f9394006fd0 tx=0x7f9394006e40 comp rx=0 tx=0).stop 2024-03-24T00:04:23.203 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:04:22.483+0000 7f93ada00700 1 -- 172.21.15.118:0/3859313457 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f93a8109db0 msgr2=0x7f93a819dd30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:04:23.203 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:04:22.483+0000 7f93ada00700 1 --2- 172.21.15.118:0/3859313457 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f93a8109db0 0x7f93a819dd30 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f9398011850 tx=0x7f9398011c10 comp rx=0 tx=0).stop 2024-03-24T00:04:23.203 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:04:22.483+0000 7f93ada00700 1 -- 172.21.15.118:0/3859313457 shutdown_connections 2024-03-24T00:04:23.203 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:04:22.483+0000 7f93ada00700 1 --2- 172.21.15.118:0/3859313457 >> [v2:172.21.15.118:6800/1463147955,v1:172.21.15.118:6801/1463147955] conn(0x7f938c038f20 0x7f938c03b3e0 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:04:23.203 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:04:22.483+0000 7f93ada00700 1 --2- 172.21.15.118:0/3859313457 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f93a8109db0 0x7f93a819dd30 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:04:23.203 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:04:22.483+0000 7f93ada00700 1 -- 172.21.15.118:0/3859313457 >> 172.21.15.118:0/3859313457 conn(0x7f93a807d220 msgr2=0x7f93a81074e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:04:23.203 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:04:22.484+0000 7f93ada00700 1 -- 172.21.15.118:0/3859313457 shutdown_connections 2024-03-24T00:04:23.203 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:04:22.484+0000 7f93ada00700 1 -- 172.21.15.118:0/3859313457 wait complete. 2024-03-24T00:04:23.445 INFO:journalctl@ceph.mgr.x.smithi118.stdout:Mar 24 00:04:23 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mgr-x[31762]: 2024-03-24T00:04:23.286+0000 7f667cda2200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-24T00:04:23.446 INFO:journalctl@ceph.mgr.x.smithi118.stdout:Mar 24 00:04:23 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mgr-x[31762]: 2024-03-24T00:04:23.406+0000 7f667cda2200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-24T00:04:23.780 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:04:23 smithi118 ceph-mon[31535]: from='client.? 172.21.15.118:0/3859313457' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "cephadm"}]': finished 2024-03-24T00:04:23.780 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:04:23 smithi118 ceph-mon[31535]: mgrmap e5: x(active, since 5s) 2024-03-24T00:04:24.349 INFO:journalctl@ceph.mgr.x.smithi118.stdout:Mar 24 00:04:24 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mgr-x[31762]: 2024-03-24T00:04:24.228+0000 7f667cda2200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-24T00:04:24.652 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:04:24 smithi118 ceph-mon[31535]: from='client.? 172.21.15.118:0/3140391019' entity='client.admin' cmd=[{"prefix": "mgr stat"}]: dispatch 2024-03-24T00:04:24.653 INFO:journalctl@ceph.mgr.x.smithi118.stdout:Mar 24 00:04:24 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mgr-x[31762]: 2024-03-24T00:04:24.349+0000 7f667cda2200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-24T00:04:24.653 INFO:journalctl@ceph.mgr.x.smithi118.stdout:Mar 24 00:04:24 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mgr-x[31762]: 2024-03-24T00:04:24.529+0000 7f667cda2200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-24T00:04:24.653 INFO:journalctl@ceph.mgr.x.smithi118.stdout:Mar 24 00:04:24 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mgr-x[31762]: 2024-03-24T00:04:24.652+0000 7f667cda2200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-24T00:04:24.807 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stdout { 2024-03-24T00:04:24.807 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stdout "epoch": 5, 2024-03-24T00:04:24.807 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stdout "available": true, 2024-03-24T00:04:24.807 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stdout "active_name": "x", 2024-03-24T00:04:24.807 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stdout "num_standby": 0 2024-03-24T00:04:24.807 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stdout } 2024-03-24T00:04:24.807 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:04:23.842+0000 7f15cde00700 1 Processor -- start 2024-03-24T00:04:24.807 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:04:23.842+0000 7f15cde00700 1 -- start start 2024-03-24T00:04:24.807 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:04:23.843+0000 7f15cde00700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f15c807cd40 0x7f15c807b1f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:04:24.808 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:04:23.843+0000 7f15cde00700 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f15c807b730 con 0x7f15c807cd40 2024-03-24T00:04:24.808 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:04:23.843+0000 7f15cca00700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f15c807cd40 0x7f15c807b1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:04:24.808 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:04:23.843+0000 7f15cca00700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f15c807cd40 0x7f15c807b1f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.118:3300/0 says I am v2:172.21.15.118:51692/0 (socket says 172.21.15.118:51692) 2024-03-24T00:04:24.808 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:04:23.843+0000 7f15cca00700 1 -- 172.21.15.118:0/2987823392 learned_addr learned my addr 172.21.15.118:0/2987823392 (peer_addr_for_me v2:172.21.15.118:0/0) 2024-03-24T00:04:24.808 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:04:23.844+0000 7f15cca00700 1 -- 172.21.15.118:0/2987823392 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f15c807b8b0 con 0x7f15c807cd40 2024-03-24T00:04:24.808 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:04:23.844+0000 7f15cca00700 1 --2- 172.21.15.118:0/2987823392 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f15c807cd40 0x7f15c807b1f0 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f15b80098a0 tx=0x7f15b8009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ef5ac24021a6ae27 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:04:24.808 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:04:23.844+0000 7f15c7e00700 1 -- 172.21.15.118:0/2987823392 <== mon.0 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f15b8004160 con 0x7f15c807cd40 2024-03-24T00:04:24.808 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:04:23.844+0000 7f15c7e00700 1 -- 172.21.15.118:0/2987823392 <== mon.0 v2:172.21.15.118:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f15b8004300 con 0x7f15c807cd40 2024-03-24T00:04:24.808 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:04:23.844+0000 7f15c7e00700 1 -- 172.21.15.118:0/2987823392 <== mon.0 v2:172.21.15.118:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f15b8011b50 con 0x7f15c807cd40 2024-03-24T00:04:24.808 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:04:23.845+0000 7f15cde00700 1 -- 172.21.15.118:0/2987823392 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f15c807cd40 msgr2=0x7f15c807b1f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:04:24.808 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:04:23.845+0000 7f15cde00700 1 --2- 172.21.15.118:0/2987823392 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f15c807cd40 0x7f15c807b1f0 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f15b80098a0 tx=0x7f15b8009bb0 comp rx=0 tx=0).stop 2024-03-24T00:04:24.808 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:04:23.845+0000 7f15cde00700 1 -- 172.21.15.118:0/2987823392 shutdown_connections 2024-03-24T00:04:24.809 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:04:23.845+0000 7f15cde00700 1 --2- 172.21.15.118:0/2987823392 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f15c807cd40 0x7f15c807b1f0 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:04:24.809 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:04:23.845+0000 7f15cde00700 1 -- 172.21.15.118:0/2987823392 >> 172.21.15.118:0/2987823392 conn(0x7f15c8102f00 msgr2=0x7f15c8105340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:04:24.809 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:04:23.845+0000 7f15cde00700 1 -- 172.21.15.118:0/2987823392 shutdown_connections 2024-03-24T00:04:24.809 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:04:23.845+0000 7f15cde00700 1 -- 172.21.15.118:0/2987823392 wait complete. 2024-03-24T00:04:24.809 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:04:23.846+0000 7f15cde00700 1 Processor -- start 2024-03-24T00:04:24.809 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:04:23.846+0000 7f15cde00700 1 -- start start 2024-03-24T00:04:24.809 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:04:23.847+0000 7f15cde00700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f15c807cd40 0x7f15c819db30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:04:24.809 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:04:23.847+0000 7f15cde00700 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f15c810df30 con 0x7f15c807cd40 2024-03-24T00:04:24.809 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:04:23.847+0000 7f15cca00700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f15c807cd40 0x7f15c819db30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:04:24.809 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:04:23.847+0000 7f15cca00700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f15c807cd40 0x7f15c819db30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.118:3300/0 says I am v2:172.21.15.118:51696/0 (socket says 172.21.15.118:51696) 2024-03-24T00:04:24.809 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:04:23.847+0000 7f15cca00700 1 -- 172.21.15.118:0/3140391019 learned_addr learned my addr 172.21.15.118:0/3140391019 (peer_addr_for_me v2:172.21.15.118:0/0) 2024-03-24T00:04:24.809 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:04:23.847+0000 7f15cca00700 1 -- 172.21.15.118:0/3140391019 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f15c819e070 con 0x7f15c807cd40 2024-03-24T00:04:24.809 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:04:23.848+0000 7f15cca00700 1 --2- 172.21.15.118:0/3140391019 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f15c807cd40 0x7f15c819db30 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f15b8011b60 tx=0x7f15b8011c40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:04:24.810 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:04:23.848+0000 7f15c6000700 1 -- 172.21.15.118:0/3140391019 <== mon.0 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f15b8003970 con 0x7f15c807cd40 2024-03-24T00:04:24.810 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:04:23.848+0000 7f15c6000700 1 -- 172.21.15.118:0/3140391019 <== mon.0 v2:172.21.15.118:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f15b8003b10 con 0x7f15c807cd40 2024-03-24T00:04:24.810 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:04:23.848+0000 7f15cde00700 1 -- 172.21.15.118:0/3140391019 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f15c819e300 con 0x7f15c807cd40 2024-03-24T00:04:24.810 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:04:23.848+0000 7f15c6000700 1 -- 172.21.15.118:0/3140391019 <== mon.0 v2:172.21.15.118:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f15b8024610 con 0x7f15c807cd40 2024-03-24T00:04:24.810 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:04:23.848+0000 7f15cde00700 1 -- 172.21.15.118:0/3140391019 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f15c81a0ff0 con 0x7f15c807cd40 2024-03-24T00:04:24.810 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:04:23.849+0000 7f15c6000700 1 -- 172.21.15.118:0/3140391019 <== mon.0 v2:172.21.15.118:3300/0 4 ==== mgrmap(e 5) v1 ==== 46106+0+0 (secure 0 0 0) 0x7f15b8024910 con 0x7f15c807cd40 2024-03-24T00:04:24.810 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:04:23.849+0000 7f15cde00700 1 -- 172.21.15.118:0/3140391019 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f15c80637c0 con 0x7f15c807cd40 2024-03-24T00:04:24.810 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:04:23.850+0000 7f15c6000700 1 --2- 172.21.15.118:0/3140391019 >> [v2:172.21.15.118:6800/1463147955,v1:172.21.15.118:6801/1463147955] conn(0x7f15ac038f20 0x7f15ac03b3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:04:24.810 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:04:23.850+0000 7f15c6000700 1 -- 172.21.15.118:0/3140391019 <== mon.0 v2:172.21.15.118:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (secure 0 0 0) 0x7f15b8011e00 con 0x7f15c807cd40 2024-03-24T00:04:24.810 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:04:23.850+0000 7f15bfe00700 1 -- 172.21.15.118:0/3140391019 >> [v2:172.21.15.118:6800/1463147955,v1:172.21.15.118:6801/1463147955] conn(0x7f15ac038f20 msgr2=0x7f15ac03b3e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.118:6800/1463147955 2024-03-24T00:04:24.810 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:04:23.850+0000 7f15bfe00700 1 --2- 172.21.15.118:0/3140391019 >> [v2:172.21.15.118:6800/1463147955,v1:172.21.15.118:6801/1463147955] conn(0x7f15ac038f20 0x7f15ac03b3e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-24T00:04:24.810 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:04:23.859+0000 7f15c6000700 1 -- 172.21.15.118:0/3140391019 <== mon.0 v2:172.21.15.118:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f15b800ba00 con 0x7f15c807cd40 2024-03-24T00:04:24.810 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:04:24.051+0000 7f15bfe00700 1 -- 172.21.15.118:0/3140391019 >> [v2:172.21.15.118:6800/1463147955,v1:172.21.15.118:6801/1463147955] conn(0x7f15ac038f20 msgr2=0x7f15ac03b3e0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.118:6800/1463147955 2024-03-24T00:04:24.811 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:04:24.051+0000 7f15bfe00700 1 --2- 172.21.15.118:0/3140391019 >> [v2:172.21.15.118:6800/1463147955,v1:172.21.15.118:6801/1463147955] conn(0x7f15ac038f20 0x7f15ac03b3e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-24T00:04:24.811 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:04:24.136+0000 7f15cde00700 1 -- 172.21.15.118:0/3140391019 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_command({"prefix": "mgr stat"} v 0) v1 -- 0x7f15c81a11c0 con 0x7f15c807cd40 2024-03-24T00:04:24.811 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:04:24.136+0000 7f15c6000700 1 -- 172.21.15.118:0/3140391019 <== mon.0 v2:172.21.15.118:3300/0 7 ==== mon_command_ack([{"prefix": "mgr stat"}]=0 v5) v1 ==== 56+0+88 (secure 0 0 0) 0x7f15c81a11c0 con 0x7f15c807cd40 2024-03-24T00:04:24.811 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:04:24.140+0000 7f15cde00700 1 -- 172.21.15.118:0/3140391019 >> [v2:172.21.15.118:6800/1463147955,v1:172.21.15.118:6801/1463147955] conn(0x7f15ac038f20 msgr2=0x7f15ac03b3e0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-24T00:04:24.811 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:04:24.140+0000 7f15cde00700 1 --2- 172.21.15.118:0/3140391019 >> [v2:172.21.15.118:6800/1463147955,v1:172.21.15.118:6801/1463147955] conn(0x7f15ac038f20 0x7f15ac03b3e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:04:24.811 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:04:24.140+0000 7f15cde00700 1 -- 172.21.15.118:0/3140391019 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f15c807cd40 msgr2=0x7f15c819db30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:04:24.811 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:04:24.140+0000 7f15cde00700 1 --2- 172.21.15.118:0/3140391019 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f15c807cd40 0x7f15c819db30 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f15b8011b60 tx=0x7f15b8011c40 comp rx=0 tx=0).stop 2024-03-24T00:04:24.811 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:04:24.140+0000 7f15cde00700 1 -- 172.21.15.118:0/3140391019 shutdown_connections 2024-03-24T00:04:24.811 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:04:24.140+0000 7f15cde00700 1 --2- 172.21.15.118:0/3140391019 >> [v2:172.21.15.118:6800/1463147955,v1:172.21.15.118:6801/1463147955] conn(0x7f15ac038f20 0x7f15ac03b3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:04:24.811 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:04:24.140+0000 7f15cde00700 1 --2- 172.21.15.118:0/3140391019 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f15c807cd40 0x7f15c819db30 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:04:24.811 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:04:24.140+0000 7f15cde00700 1 -- 172.21.15.118:0/3140391019 >> 172.21.15.118:0/3140391019 conn(0x7f15c8102f00 msgr2=0x7f15c8103a90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:04:24.811 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:04:24.140+0000 7f15cde00700 1 -- 172.21.15.118:0/3140391019 shutdown_connections 2024-03-24T00:04:24.811 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:04:24.141+0000 7f15cde00700 1 -- 172.21.15.118:0/3140391019 wait complete. 2024-03-24T00:04:24.812 INFO:teuthology.orchestra.run.smithi118.stdout:Waiting for the mgr to restart... 2024-03-24T00:04:24.812 INFO:teuthology.orchestra.run.smithi118.stdout:Waiting for mgr epoch 5... 2024-03-24T00:04:24.945 INFO:journalctl@ceph.mgr.x.smithi118.stdout:Mar 24 00:04:24 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mgr-x[31762]: 2024-03-24T00:04:24.761+0000 7f667cda2200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-24T00:04:25.212 INFO:journalctl@ceph.mgr.x.smithi118.stdout:Mar 24 00:04:25 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mgr-x[31762]: 2024-03-24T00:04:25.114+0000 7f667cda2200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-03-24T00:04:25.498 INFO:journalctl@ceph.mgr.x.smithi118.stdout:Mar 24 00:04:25 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mgr-x[31762]: 2024-03-24T00:04:25.256+0000 7f667cda2200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-24T00:04:25.498 INFO:journalctl@ceph.mgr.x.smithi118.stdout:Mar 24 00:04:25 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mgr-x[31762]: 2024-03-24T00:04:25.375+0000 7f667cda2200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-24T00:04:25.498 INFO:journalctl@ceph.mgr.x.smithi118.stdout:Mar 24 00:04:25 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mgr-x[31762]: 2024-03-24T00:04:25.497+0000 7f667cda2200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-24T00:04:25.945 INFO:journalctl@ceph.mgr.x.smithi118.stdout:Mar 24 00:04:25 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mgr-x[31762]: 2024-03-24T00:04:25.713+0000 7f667cda2200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-24T00:04:27.174 INFO:journalctl@ceph.mgr.x.smithi118.stdout:Mar 24 00:04:27 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mgr-x[31762]: 2024-03-24T00:04:26.998+0000 7f667cda2200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-03-24T00:04:27.174 INFO:journalctl@ceph.mgr.x.smithi118.stdout:Mar 24 00:04:27 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mgr-x[31762]: 2024-03-24T00:04:27.174+0000 7f667cda2200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-24T00:04:27.945 INFO:journalctl@ceph.mgr.x.smithi118.stdout:Mar 24 00:04:27 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mgr-x[31762]: 2024-03-24T00:04:27.599+0000 7f667cda2200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-24T00:04:27.946 INFO:journalctl@ceph.mgr.x.smithi118.stdout:Mar 24 00:04:27 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mgr-x[31762]: 2024-03-24T00:04:27.719+0000 7f667cda2200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-24T00:04:28.377 INFO:journalctl@ceph.mgr.x.smithi118.stdout:Mar 24 00:04:27 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mgr-x[31762]: 2024-03-24T00:04:27.976+0000 7f667cda2200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-24T00:04:28.377 INFO:journalctl@ceph.mgr.x.smithi118.stdout:Mar 24 00:04:28 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mgr-x[31762]: 2024-03-24T00:04:28.216+0000 7f667cda2200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-24T00:04:28.695 INFO:journalctl@ceph.mgr.x.smithi118.stdout:Mar 24 00:04:28 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mgr-x[31762]: 2024-03-24T00:04:28.585+0000 7f667cda2200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-24T00:04:29.195 INFO:journalctl@ceph.mgr.x.smithi118.stdout:Mar 24 00:04:28 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mgr-x[31762]: 2024-03-24T00:04:28.898+0000 7f667cda2200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-24T00:04:29.195 INFO:journalctl@ceph.mgr.x.smithi118.stdout:Mar 24 00:04:29 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mgr-x[31762]: 2024-03-24T00:04:29.194+0000 7f667cda2200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-24T00:04:30.445 INFO:journalctl@ceph.mgr.x.smithi118.stdout:Mar 24 00:04:30 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mgr-x[31762]: 2024-03-24T00:04:30.193+0000 7f667cda2200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-24T00:04:31.195 INFO:journalctl@ceph.mgr.x.smithi118.stdout:Mar 24 00:04:30 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mgr-x[31762]: 2024-03-24T00:04:30.877+0000 7f667cda2200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-24T00:04:31.196 INFO:journalctl@ceph.mgr.x.smithi118.stdout:Mar 24 00:04:31 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mgr-x[31762]: 2024-03-24T00:04:31.003+0000 7f667cda2200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-24T00:04:31.696 INFO:journalctl@ceph.mgr.x.smithi118.stdout:Mar 24 00:04:31 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mgr-x[31762]: 2024-03-24T00:04:31.280+0000 7f667cda2200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-24T00:04:31.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:04:31 smithi118 ceph-mon[31535]: Active manager daemon x restarted 2024-03-24T00:04:31.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:04:31 smithi118 ceph-mon[31535]: Activating manager daemon x 2024-03-24T00:04:31.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:04:31 smithi118 ceph-mon[31535]: osdmap e2: 0 total, 0 up, 0 in 2024-03-24T00:04:31.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:04:31 smithi118 ceph-mon[31535]: mgrmap e6: x(active, starting, since 0.00501033s) 2024-03-24T00:04:31.697 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:04:31 smithi118 ceph-mon[31535]: from='mgr.14120 172.21.15.118:0/3476110238' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-24T00:04:31.697 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:04:31 smithi118 ceph-mon[31535]: from='mgr.14120 172.21.15.118:0/3476110238' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-03-24T00:04:31.697 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:04:31 smithi118 ceph-mon[31535]: from='mgr.14120 172.21.15.118:0/3476110238' entity='mgr.x' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-03-24T00:04:31.697 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:04:31 smithi118 ceph-mon[31535]: from='mgr.14120 172.21.15.118:0/3476110238' entity='mgr.x' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-03-24T00:04:31.697 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:04:31 smithi118 ceph-mon[31535]: from='mgr.14120 172.21.15.118:0/3476110238' entity='mgr.x' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-03-24T00:04:31.697 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:04:31 smithi118 ceph-mon[31535]: Manager daemon x is now available 2024-03-24T00:04:31.697 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:04:31 smithi118 ceph-mon[31535]: from='mgr.14120 172.21.15.118:0/3476110238' entity='mgr.x' 2024-03-24T00:04:32.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:04:32 smithi118 ceph-mon[31535]: Found migration_current of "None". Setting to last migration. 2024-03-24T00:04:32.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:04:32 smithi118 ceph-mon[31535]: from='mgr.14120 172.21.15.118:0/3476110238' entity='mgr.x' 2024-03-24T00:04:32.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:04:32 smithi118 ceph-mon[31535]: from='mgr.14120 172.21.15.118:0/3476110238' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T00:04:32.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:04:32 smithi118 ceph-mon[31535]: from='mgr.14120 172.21.15.118:0/3476110238' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T00:04:32.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:04:32 smithi118 ceph-mon[31535]: from='mgr.14120 172.21.15.118:0/3476110238' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T00:04:32.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:04:32 smithi118 ceph-mon[31535]: from='mgr.14120 172.21.15.118:0/3476110238' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2024-03-24T00:04:32.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:04:32 smithi118 ceph-mon[31535]: from='mgr.14120 172.21.15.118:0/3476110238' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2024-03-24T00:04:32.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:04:32 smithi118 ceph-mon[31535]: from='mgr.14120 172.21.15.118:0/3476110238' entity='mgr.x' 2024-03-24T00:04:32.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:04:32 smithi118 ceph-mon[31535]: from='mgr.14120 172.21.15.118:0/3476110238' entity='mgr.x' 2024-03-24T00:04:32.697 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:04:32 smithi118 ceph-mon[31535]: mgrmap e7: x(active, since 1.01565s) 2024-03-24T00:04:33.021 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stdout { 2024-03-24T00:04:33.029 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stdout "mgrmap_epoch": 7, 2024-03-24T00:04:33.029 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stdout "initialized": true 2024-03-24T00:04:33.029 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stdout } 2024-03-24T00:04:33.029 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:04:25.362+0000 7fef3ca00700 1 Processor -- start 2024-03-24T00:04:33.029 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:04:25.362+0000 7fef3ca00700 1 -- start start 2024-03-24T00:04:33.029 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:04:25.363+0000 7fef3ca00700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fef3807cd70 0x7fef3807b1d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:04:33.029 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:04:25.363+0000 7fef3ca00700 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fef3807b7a0 con 0x7fef3807cd70 2024-03-24T00:04:33.029 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:04:25.363+0000 7fef35e00700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fef3807cd70 0x7fef3807b1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:04:33.029 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:04:25.363+0000 7fef35e00700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fef3807cd70 0x7fef3807b1d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.118:3300/0 says I am v2:172.21.15.118:51698/0 (socket says 172.21.15.118:51698) 2024-03-24T00:04:33.030 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:04:25.363+0000 7fef35e00700 1 -- 172.21.15.118:0/925239929 learned_addr learned my addr 172.21.15.118:0/925239929 (peer_addr_for_me v2:172.21.15.118:0/0) 2024-03-24T00:04:33.030 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:04:25.364+0000 7fef35e00700 1 -- 172.21.15.118:0/925239929 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fef3807b920 con 0x7fef3807cd70 2024-03-24T00:04:33.030 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:04:25.364+0000 7fef35e00700 1 --2- 172.21.15.118:0/925239929 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fef3807cd70 0x7fef3807b1d0 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7fef280098a0 tx=0x7fef28009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3b6bf4f96b940f1 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:04:33.030 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:04:25.364+0000 7fef34a00700 1 -- 172.21.15.118:0/925239929 <== mon.0 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fef28004160 con 0x7fef3807cd70 2024-03-24T00:04:33.030 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:04:25.364+0000 7fef34a00700 1 -- 172.21.15.118:0/925239929 <== mon.0 v2:172.21.15.118:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fef28004300 con 0x7fef3807cd70 2024-03-24T00:04:33.030 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:04:25.364+0000 7fef34a00700 1 -- 172.21.15.118:0/925239929 <== mon.0 v2:172.21.15.118:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fef28011b50 con 0x7fef3807cd70 2024-03-24T00:04:33.030 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:04:25.365+0000 7fef3ca00700 1 -- 172.21.15.118:0/925239929 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fef3807cd70 msgr2=0x7fef3807b1d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:04:33.030 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:04:25.365+0000 7fef3ca00700 1 --2- 172.21.15.118:0/925239929 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fef3807cd70 0x7fef3807b1d0 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7fef280098a0 tx=0x7fef28009bb0 comp rx=0 tx=0).stop 2024-03-24T00:04:33.030 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:04:25.365+0000 7fef3ca00700 1 -- 172.21.15.118:0/925239929 shutdown_connections 2024-03-24T00:04:33.030 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:04:25.365+0000 7fef3ca00700 1 --2- 172.21.15.118:0/925239929 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fef3807cd70 0x7fef3807b1d0 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:04:33.030 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:04:25.365+0000 7fef3ca00700 1 -- 172.21.15.118:0/925239929 >> 172.21.15.118:0/925239929 conn(0x7fef38103080 msgr2=0x7fef381054c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:04:33.030 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:04:25.365+0000 7fef3ca00700 1 -- 172.21.15.118:0/925239929 shutdown_connections 2024-03-24T00:04:33.031 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:04:25.365+0000 7fef3ca00700 1 -- 172.21.15.118:0/925239929 wait complete. 2024-03-24T00:04:33.031 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:04:25.367+0000 7fef3ca00700 1 Processor -- start 2024-03-24T00:04:33.031 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:04:25.367+0000 7fef3ca00700 1 -- start start 2024-03-24T00:04:33.031 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:04:25.367+0000 7fef3ca00700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fef3807cd70 0x7fef3819dca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:04:33.031 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:04:25.368+0000 7fef3ca00700 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fef3810e0d0 con 0x7fef3807cd70 2024-03-24T00:04:33.031 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:04:25.368+0000 7fef35e00700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fef3807cd70 0x7fef3819dca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:04:33.031 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:04:25.368+0000 7fef35e00700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fef3807cd70 0x7fef3819dca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.118:3300/0 says I am v2:172.21.15.118:51706/0 (socket says 172.21.15.118:51706) 2024-03-24T00:04:33.031 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:04:25.368+0000 7fef35e00700 1 -- 172.21.15.118:0/2322031465 learned_addr learned my addr 172.21.15.118:0/2322031465 (peer_addr_for_me v2:172.21.15.118:0/0) 2024-03-24T00:04:33.031 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:04:25.368+0000 7fef35e00700 1 -- 172.21.15.118:0/2322031465 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fef3819e1e0 con 0x7fef3807cd70 2024-03-24T00:04:33.031 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:04:25.368+0000 7fef35e00700 1 --2- 172.21.15.118:0/2322031465 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fef3807cd70 0x7fef3819dca0 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7fef28011b60 tx=0x7fef28011c40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:04:33.031 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:04:25.369+0000 7fef2ea00700 1 -- 172.21.15.118:0/2322031465 <== mon.0 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fef28003b20 con 0x7fef3807cd70 2024-03-24T00:04:33.031 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:04:25.369+0000 7fef2ea00700 1 -- 172.21.15.118:0/2322031465 <== mon.0 v2:172.21.15.118:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fef28003cc0 con 0x7fef3807cd70 2024-03-24T00:04:33.031 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:04:25.369+0000 7fef2ea00700 1 -- 172.21.15.118:0/2322031465 <== mon.0 v2:172.21.15.118:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fef28024840 con 0x7fef3807cd70 2024-03-24T00:04:33.031 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:04:25.369+0000 7fef3ca00700 1 -- 172.21.15.118:0/2322031465 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fef3819e470 con 0x7fef3807cd70 2024-03-24T00:04:33.032 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:04:25.369+0000 7fef3ca00700 1 -- 172.21.15.118:0/2322031465 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fef381a1160 con 0x7fef3807cd70 2024-03-24T00:04:33.032 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:04:25.370+0000 7fef2ea00700 1 -- 172.21.15.118:0/2322031465 <== mon.0 v2:172.21.15.118:3300/0 4 ==== mgrmap(e 5) v1 ==== 46106+0+0 (secure 0 0 0) 0x7fef28022020 con 0x7fef3807cd70 2024-03-24T00:04:33.032 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:04:25.371+0000 7fef2ea00700 1 --2- 172.21.15.118:0/2322031465 >> [v2:172.21.15.118:6800/1463147955,v1:172.21.15.118:6801/1463147955] conn(0x7fef1c038f70 0x7fef1c03b430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:04:33.032 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:04:25.371+0000 7fef2ea00700 1 -- 172.21.15.118:0/2322031465 --> [v2:172.21.15.118:6800/1463147955,v1:172.21.15.118:6801/1463147955] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x7fef1c03bb40 con 0x7fef1c038f70 2024-03-24T00:04:33.032 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:04:25.371+0000 7fef2ea00700 1 -- 172.21.15.118:0/2322031465 <== mon.0 v2:172.21.15.118:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 725+0+0 (secure 0 0 0) 0x7fef28011e00 con 0x7fef3807cd70 2024-03-24T00:04:33.032 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:04:25.371+0000 7fef35400700 1 -- 172.21.15.118:0/2322031465 >> [v2:172.21.15.118:6800/1463147955,v1:172.21.15.118:6801/1463147955] conn(0x7fef1c038f70 msgr2=0x7fef1c03b430 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.118:6800/1463147955 2024-03-24T00:04:33.032 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:04:25.371+0000 7fef35400700 1 --2- 172.21.15.118:0/2322031465 >> [v2:172.21.15.118:6800/1463147955,v1:172.21.15.118:6801/1463147955] conn(0x7fef1c038f70 0x7fef1c03b430 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-24T00:04:33.032 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:04:25.572+0000 7fef35400700 1 -- 172.21.15.118:0/2322031465 >> [v2:172.21.15.118:6800/1463147955,v1:172.21.15.118:6801/1463147955] conn(0x7fef1c038f70 msgr2=0x7fef1c03b430 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.118:6800/1463147955 2024-03-24T00:04:33.032 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:04:25.572+0000 7fef35400700 1 --2- 172.21.15.118:0/2322031465 >> [v2:172.21.15.118:6800/1463147955,v1:172.21.15.118:6801/1463147955] conn(0x7fef1c038f70 0x7fef1c03b430 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-24T00:04:33.032 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:04:25.973+0000 7fef35400700 1 -- 172.21.15.118:0/2322031465 >> [v2:172.21.15.118:6800/1463147955,v1:172.21.15.118:6801/1463147955] conn(0x7fef1c038f70 msgr2=0x7fef1c03b430 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.118:6800/1463147955 2024-03-24T00:04:33.032 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:04:25.973+0000 7fef35400700 1 --2- 172.21.15.118:0/2322031465 >> [v2:172.21.15.118:6800/1463147955,v1:172.21.15.118:6801/1463147955] conn(0x7fef1c038f70 0x7fef1c03b430 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-03-24T00:04:33.032 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:04:26.774+0000 7fef35400700 1 -- 172.21.15.118:0/2322031465 >> [v2:172.21.15.118:6800/1463147955,v1:172.21.15.118:6801/1463147955] conn(0x7fef1c038f70 msgr2=0x7fef1c03b430 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.118:6800/1463147955 2024-03-24T00:04:33.032 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:04:26.774+0000 7fef35400700 1 --2- 172.21.15.118:0/2322031465 >> [v2:172.21.15.118:6800/1463147955,v1:172.21.15.118:6801/1463147955] conn(0x7fef1c038f70 0x7fef1c03b430 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-03-24T00:04:33.032 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:04:28.376+0000 7fef35400700 1 -- 172.21.15.118:0/2322031465 >> [v2:172.21.15.118:6800/1463147955,v1:172.21.15.118:6801/1463147955] conn(0x7fef1c038f70 msgr2=0x7fef1c03b430 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.118:6800/1463147955 2024-03-24T00:04:33.033 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:04:28.376+0000 7fef35400700 1 --2- 172.21.15.118:0/2322031465 >> [v2:172.21.15.118:6800/1463147955,v1:172.21.15.118:6801/1463147955] conn(0x7fef1c038f70 0x7fef1c03b430 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 3.200000 2024-03-24T00:04:33.033 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:04:31.287+0000 7fef2ea00700 1 -- 172.21.15.118:0/2322031465 <== mon.0 v2:172.21.15.118:3300/0 6 ==== mgrmap(e 6) v1 ==== 45873+0+0 (secure 0 0 0) 0x7fef28024a40 con 0x7fef3807cd70 2024-03-24T00:04:33.033 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:04:31.288+0000 7fef2ea00700 1 -- 172.21.15.118:0/2322031465 >> [v2:172.21.15.118:6800/1463147955,v1:172.21.15.118:6801/1463147955] conn(0x7fef1c038f70 msgr2=0x7fef1c03b430 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-24T00:04:33.033 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:04:31.288+0000 7fef2ea00700 1 --2- 172.21.15.118:0/2322031465 >> [v2:172.21.15.118:6800/1463147955,v1:172.21.15.118:6801/1463147955] conn(0x7fef1c038f70 0x7fef1c03b430 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:04:33.033 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:04:32.293+0000 7fef2ea00700 1 -- 172.21.15.118:0/2322031465 <== mon.0 v2:172.21.15.118:3300/0 7 ==== mgrmap(e 7) v1 ==== 46000+0+0 (secure 0 0 0) 0x7fef2804eb40 con 0x7fef3807cd70 2024-03-24T00:04:33.033 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:04:32.293+0000 7fef2ea00700 1 --2- 172.21.15.118:0/2322031465 >> [v2:172.21.15.118:6800/389249272,v1:172.21.15.118:6801/389249272] conn(0x7fef1c03c860 0x7fef1c03ec50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:04:33.033 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:04:32.293+0000 7fef2ea00700 1 -- 172.21.15.118:0/2322031465 --> [v2:172.21.15.118:6800/389249272,v1:172.21.15.118:6801/389249272] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x7fef1c03bb40 con 0x7fef1c03c860 2024-03-24T00:04:33.033 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:04:32.296+0000 7fef35400700 1 --2- 172.21.15.118:0/2322031465 >> [v2:172.21.15.118:6800/389249272,v1:172.21.15.118:6801/389249272] conn(0x7fef1c03c860 0x7fef1c03ec50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:04:33.033 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:04:32.301+0000 7fef35400700 1 --2- 172.21.15.118:0/2322031465 >> [v2:172.21.15.118:6800/389249272,v1:172.21.15.118:6801/389249272] conn(0x7fef1c03c860 0x7fef1c03ec50 secure :-1 s=READY pgs=4 cs=0 l=1 rev1=1 crypto rx=0x7fef24003a10 tx=0x7fef240092b0 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:04:33.033 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:04:32.303+0000 7fef2ea00700 1 -- 172.21.15.118:0/2322031465 <== mgr.14120 v2:172.21.15.118:6800/389249272 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+7774 (secure 0 0 0) 0x7fef1c03bb40 con 0x7fef1c03c860 2024-03-24T00:04:33.033 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:04:32.315+0000 7fef3ca00700 1 -- 172.21.15.118:0/2322031465 --> [v2:172.21.15.118:6800/389249272,v1:172.21.15.118:6801/389249272] -- command(tid 1: {"prefix": "mgr_status"}) v1 -- 0x7fef380637c0 con 0x7fef1c03c860 2024-03-24T00:04:33.033 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:04:32.316+0000 7fef2ea00700 1 -- 172.21.15.118:0/2322031465 <== mgr.14120 v2:172.21.15.118:6800/389249272 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+51 (secure 0 0 0) 0x7fef380637c0 con 0x7fef1c03c860 2024-03-24T00:04:33.033 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:04:32.317+0000 7fef3ca00700 1 -- 172.21.15.118:0/2322031465 >> [v2:172.21.15.118:6800/389249272,v1:172.21.15.118:6801/389249272] conn(0x7fef1c03c860 msgr2=0x7fef1c03ec50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:04:33.034 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:04:32.317+0000 7fef3ca00700 1 --2- 172.21.15.118:0/2322031465 >> [v2:172.21.15.118:6800/389249272,v1:172.21.15.118:6801/389249272] conn(0x7fef1c03c860 0x7fef1c03ec50 secure :-1 s=READY pgs=4 cs=0 l=1 rev1=1 crypto rx=0x7fef24003a10 tx=0x7fef240092b0 comp rx=0 tx=0).stop 2024-03-24T00:04:33.034 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:04:32.317+0000 7fef3ca00700 1 -- 172.21.15.118:0/2322031465 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fef3807cd70 msgr2=0x7fef3819dca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:04:33.034 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:04:32.317+0000 7fef3ca00700 1 --2- 172.21.15.118:0/2322031465 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fef3807cd70 0x7fef3819dca0 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7fef28011b60 tx=0x7fef28011c40 comp rx=0 tx=0).stop 2024-03-24T00:04:33.034 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:04:32.317+0000 7fef3ca00700 1 -- 172.21.15.118:0/2322031465 shutdown_connections 2024-03-24T00:04:33.034 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:04:32.317+0000 7fef3ca00700 1 --2- 172.21.15.118:0/2322031465 >> [v2:172.21.15.118:6800/389249272,v1:172.21.15.118:6801/389249272] conn(0x7fef1c03c860 0x7fef1c03ec50 unknown :-1 s=CLOSED pgs=4 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:04:33.034 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:04:32.317+0000 7fef3ca00700 1 --2- 172.21.15.118:0/2322031465 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fef3807cd70 0x7fef3819dca0 unknown :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:04:33.034 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:04:32.317+0000 7fef3ca00700 1 --2- 172.21.15.118:0/2322031465 >> [v2:172.21.15.118:6800/1463147955,v1:172.21.15.118:6801/1463147955] conn(0x7fef1c038f70 0x7fef1c03b430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:04:33.034 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:04:32.317+0000 7fef3ca00700 1 -- 172.21.15.118:0/2322031465 >> 172.21.15.118:0/2322031465 conn(0x7fef38103080 msgr2=0x7fef38103c00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:04:33.034 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:04:32.317+0000 7fef3ca00700 1 -- 172.21.15.118:0/2322031465 shutdown_connections 2024-03-24T00:04:33.034 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:04:32.317+0000 7fef3ca00700 1 -- 172.21.15.118:0/2322031465 wait complete. 2024-03-24T00:04:33.034 INFO:teuthology.orchestra.run.smithi118.stdout:mgr epoch 5 is available 2024-03-24T00:04:33.034 INFO:teuthology.orchestra.run.smithi118.stdout:Setting orchestrator backend to cephadm... 2024-03-24T00:04:33.639 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:04:33 smithi118 ceph-mon[31535]: from='client.14124 -' entity='client.admin' cmd=[{"prefix": "get_command_descriptions"}]: dispatch 2024-03-24T00:04:33.639 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:04:33 smithi118 ceph-mon[31535]: from='client.14124 -' entity='client.admin' cmd=[{"prefix": "mgr_status"}]: dispatch 2024-03-24T00:04:33.639 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:04:33 smithi118 ceph-mon[31535]: from='mgr.14120 172.21.15.118:0/3476110238' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T00:04:34.599 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:04:34 smithi118 ceph-mon[31535]: [24/Mar/2024:00:04:33] ENGINE Bus STARTING 2024-03-24T00:04:34.599 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:04:34 smithi118 ceph-mon[31535]: [24/Mar/2024:00:04:33] ENGINE Serving on http://172.21.15.118:8765 2024-03-24T00:04:34.600 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:04:34 smithi118 ceph-mon[31535]: [24/Mar/2024:00:04:33] ENGINE Serving on https://172.21.15.118:7150 2024-03-24T00:04:34.600 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:04:34 smithi118 ceph-mon[31535]: [24/Mar/2024:00:04:33] ENGINE Bus STARTED 2024-03-24T00:04:34.600 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:04:34 smithi118 ceph-mon[31535]: mgrmap e8: x(active, since 2s) 2024-03-24T00:04:34.600 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:04:34 smithi118 ceph-mon[31535]: from='mgr.14120 172.21.15.118:0/3476110238' entity='mgr.x' 2024-03-24T00:04:34.600 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:04:34 smithi118 ceph-mon[31535]: from='mgr.14120 172.21.15.118:0/3476110238' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T00:04:34.613 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:04:33.638+0000 7fef2ba00700 1 Processor -- start 2024-03-24T00:04:34.614 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:04:33.638+0000 7fef2ba00700 1 -- start start 2024-03-24T00:04:34.614 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:04:33.639+0000 7fef2ba00700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fef24109db0 0x7fef2410a1d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:04:34.614 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:04:33.639+0000 7fef2ba00700 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fef2410a7a0 con 0x7fef24109db0 2024-03-24T00:04:34.614 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:04:33.639+0000 7fef29000700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fef24109db0 0x7fef2410a1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:04:34.614 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:04:33.640+0000 7fef29000700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fef24109db0 0x7fef2410a1d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.118:3300/0 says I am v2:172.21.15.118:57322/0 (socket says 172.21.15.118:57322) 2024-03-24T00:04:34.621 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:04:33.640+0000 7fef29000700 1 -- 172.21.15.118:0/1129344884 learned_addr learned my addr 172.21.15.118:0/1129344884 (peer_addr_for_me v2:172.21.15.118:0/0) 2024-03-24T00:04:34.621 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:04:33.640+0000 7fef29000700 1 -- 172.21.15.118:0/1129344884 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fef2410af10 con 0x7fef24109db0 2024-03-24T00:04:34.621 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:04:33.640+0000 7fef29000700 1 --2- 172.21.15.118:0/1129344884 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fef24109db0 0x7fef2410a1d0 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7fef180098a0 tx=0x7fef18009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=6edd6a32595a4410 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:04:34.621 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:04:33.641+0000 7fef23400700 1 -- 172.21.15.118:0/1129344884 <== mon.0 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fef18004160 con 0x7fef24109db0 2024-03-24T00:04:34.621 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:04:33.641+0000 7fef23400700 1 -- 172.21.15.118:0/1129344884 <== mon.0 v2:172.21.15.118:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fef18004300 con 0x7fef24109db0 2024-03-24T00:04:34.621 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:04:33.641+0000 7fef23400700 1 -- 172.21.15.118:0/1129344884 <== mon.0 v2:172.21.15.118:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fef18011b50 con 0x7fef24109db0 2024-03-24T00:04:34.622 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:04:33.641+0000 7fef2ba00700 1 -- 172.21.15.118:0/1129344884 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fef24109db0 msgr2=0x7fef2410a1d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:04:34.622 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:04:33.641+0000 7fef2ba00700 1 --2- 172.21.15.118:0/1129344884 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fef24109db0 0x7fef2410a1d0 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7fef180098a0 tx=0x7fef18009bb0 comp rx=0 tx=0).stop 2024-03-24T00:04:34.622 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:04:33.642+0000 7fef2ba00700 1 -- 172.21.15.118:0/1129344884 shutdown_connections 2024-03-24T00:04:34.622 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:04:33.642+0000 7fef2ba00700 1 --2- 172.21.15.118:0/1129344884 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fef24109db0 0x7fef2410a1d0 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:04:34.622 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:04:33.642+0000 7fef2ba00700 1 -- 172.21.15.118:0/1129344884 >> 172.21.15.118:0/1129344884 conn(0x7fef2407d130 msgr2=0x7fef2407d560 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:04:34.622 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:04:33.642+0000 7fef2ba00700 1 -- 172.21.15.118:0/1129344884 shutdown_connections 2024-03-24T00:04:34.622 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:04:33.642+0000 7fef2ba00700 1 -- 172.21.15.118:0/1129344884 wait complete. 2024-03-24T00:04:34.622 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:04:33.643+0000 7fef2ba00700 1 Processor -- start 2024-03-24T00:04:34.622 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:04:33.643+0000 7fef2ba00700 1 -- start start 2024-03-24T00:04:34.622 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:04:33.643+0000 7fef2ba00700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fef24109db0 0x7fef2419dd50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:04:34.622 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:04:33.643+0000 7fef2ba00700 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fef2410e2c0 con 0x7fef24109db0 2024-03-24T00:04:34.623 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:04:33.643+0000 7fef29000700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fef24109db0 0x7fef2419dd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:04:34.623 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:04:33.643+0000 7fef29000700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fef24109db0 0x7fef2419dd50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.118:3300/0 says I am v2:172.21.15.118:57332/0 (socket says 172.21.15.118:57332) 2024-03-24T00:04:34.623 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:04:33.643+0000 7fef29000700 1 -- 172.21.15.118:0/2456043818 learned_addr learned my addr 172.21.15.118:0/2456043818 (peer_addr_for_me v2:172.21.15.118:0/0) 2024-03-24T00:04:34.623 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:04:33.644+0000 7fef29000700 1 -- 172.21.15.118:0/2456043818 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fef2419e290 con 0x7fef24109db0 2024-03-24T00:04:34.623 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:04:33.644+0000 7fef29000700 1 --2- 172.21.15.118:0/2456043818 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fef24109db0 0x7fef2419dd50 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7fef18009fb0 tx=0x7fef18011bf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:04:34.623 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:04:33.644+0000 7fef21600700 1 -- 172.21.15.118:0/2456043818 <== mon.0 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fef180038b0 con 0x7fef24109db0 2024-03-24T00:04:34.623 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:04:33.644+0000 7fef21600700 1 -- 172.21.15.118:0/2456043818 <== mon.0 v2:172.21.15.118:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fef18011e50 con 0x7fef24109db0 2024-03-24T00:04:34.623 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:04:33.644+0000 7fef2ba00700 1 -- 172.21.15.118:0/2456043818 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fef2419e520 con 0x7fef24109db0 2024-03-24T00:04:34.623 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:04:33.644+0000 7fef21600700 1 -- 172.21.15.118:0/2456043818 <== mon.0 v2:172.21.15.118:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fef18019550 con 0x7fef24109db0 2024-03-24T00:04:34.623 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:04:33.644+0000 7fef2ba00700 1 -- 172.21.15.118:0/2456043818 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fef241a1210 con 0x7fef24109db0 2024-03-24T00:04:34.623 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:04:33.645+0000 7fef21600700 1 -- 172.21.15.118:0/2456043818 <== mon.0 v2:172.21.15.118:3300/0 4 ==== mgrmap(e 8) v1 ==== 46106+0+0 (secure 0 0 0) 0x7fef18019850 con 0x7fef24109db0 2024-03-24T00:04:34.623 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:04:33.646+0000 7fef2ba00700 1 -- 172.21.15.118:0/2456043818 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fef240637c0 con 0x7fef24109db0 2024-03-24T00:04:34.623 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:04:33.646+0000 7fef21600700 1 --2- 172.21.15.118:0/2456043818 >> [v2:172.21.15.118:6800/389249272,v1:172.21.15.118:6801/389249272] conn(0x7fef0c038ed0 0x7fef0c03b390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:04:34.624 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:04:33.646+0000 7fef21600700 1 -- 172.21.15.118:0/2456043818 <== mon.0 v2:172.21.15.118:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 940+0+0 (secure 0 0 0) 0x7fef1804d190 con 0x7fef24109db0 2024-03-24T00:04:34.624 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:04:33.647+0000 7fef23e00700 1 --2- 172.21.15.118:0/2456043818 >> [v2:172.21.15.118:6800/389249272,v1:172.21.15.118:6801/389249272] conn(0x7fef0c038ed0 0x7fef0c03b390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:04:34.624 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:04:33.647+0000 7fef23e00700 1 --2- 172.21.15.118:0/2456043818 >> [v2:172.21.15.118:6800/389249272,v1:172.21.15.118:6801/389249272] conn(0x7fef0c038ed0 0x7fef0c03b390 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7fef10006fd0 tx=0x7fef10006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:04:34.624 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:04:33.652+0000 7fef21600700 1 -- 172.21.15.118:0/2456043818 <== mon.0 v2:172.21.15.118:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7fef1804cae0 con 0x7fef24109db0 2024-03-24T00:04:34.624 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:04:33.871+0000 7fef2ba00700 1 -- 172.21.15.118:0/2456043818 --> [v2:172.21.15.118:6800/389249272,v1:172.21.15.118:6801/389249272] -- mgr_command(tid 0: {"prefix": "orch set backend", "module_name": "cephadm", "target": ["mon-mgr", ""]}) v1 -- 0x7fef24107ca0 con 0x7fef0c038ed0 2024-03-24T00:04:34.624 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:04:33.885+0000 7fef21600700 1 -- 172.21.15.118:0/2456043818 <== mgr.14120 v2:172.21.15.118:6800/389249272 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7fef24107ca0 con 0x7fef0c038ed0 2024-03-24T00:04:34.624 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:04:33.888+0000 7fef2ba00700 1 -- 172.21.15.118:0/2456043818 >> [v2:172.21.15.118:6800/389249272,v1:172.21.15.118:6801/389249272] conn(0x7fef0c038ed0 msgr2=0x7fef0c03b390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:04:34.624 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:04:33.888+0000 7fef2ba00700 1 --2- 172.21.15.118:0/2456043818 >> [v2:172.21.15.118:6800/389249272,v1:172.21.15.118:6801/389249272] conn(0x7fef0c038ed0 0x7fef0c03b390 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7fef10006fd0 tx=0x7fef10006e40 comp rx=0 tx=0).stop 2024-03-24T00:04:34.624 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:04:33.888+0000 7fef2ba00700 1 -- 172.21.15.118:0/2456043818 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fef24109db0 msgr2=0x7fef2419dd50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:04:34.624 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:04:33.888+0000 7fef2ba00700 1 --2- 172.21.15.118:0/2456043818 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fef24109db0 0x7fef2419dd50 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7fef18009fb0 tx=0x7fef18011bf0 comp rx=0 tx=0).stop 2024-03-24T00:04:34.624 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:04:33.888+0000 7fef2ba00700 1 -- 172.21.15.118:0/2456043818 shutdown_connections 2024-03-24T00:04:34.624 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:04:33.888+0000 7fef2ba00700 1 --2- 172.21.15.118:0/2456043818 >> [v2:172.21.15.118:6800/389249272,v1:172.21.15.118:6801/389249272] conn(0x7fef0c038ed0 0x7fef0c03b390 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:04:34.624 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:04:33.888+0000 7fef2ba00700 1 --2- 172.21.15.118:0/2456043818 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fef24109db0 0x7fef2419dd50 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:04:34.624 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:04:33.888+0000 7fef2ba00700 1 -- 172.21.15.118:0/2456043818 >> 172.21.15.118:0/2456043818 conn(0x7fef2407d130 msgr2=0x7fef24107590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:04:34.625 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:04:33.889+0000 7fef2ba00700 1 -- 172.21.15.118:0/2456043818 shutdown_connections 2024-03-24T00:04:34.625 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:04:33.889+0000 7fef2ba00700 1 -- 172.21.15.118:0/2456043818 wait complete. 2024-03-24T00:04:35.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:04:35 smithi118 ceph-mon[31535]: from='client.14132 -' entity='client.admin' cmd=[{"prefix": "orch set backend", "module_name": "cephadm", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T00:04:36.184 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stdout value unchanged 2024-03-24T00:04:36.185 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:04:35.268+0000 7fb0a2800700 1 Processor -- start 2024-03-24T00:04:36.185 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:04:35.269+0000 7fb0a2800700 1 -- start start 2024-03-24T00:04:36.192 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:04:35.269+0000 7fb0a2800700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb09c081980 0x7fb09c081da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:04:36.192 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:04:35.269+0000 7fb0a2800700 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb09c082370 con 0x7fb09c081980 2024-03-24T00:04:36.193 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:04:35.269+0000 7fb0a1400700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb09c081980 0x7fb09c081da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:04:36.193 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:04:35.269+0000 7fb0a1400700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb09c081980 0x7fb09c081da0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.118:3300/0 says I am v2:172.21.15.118:57346/0 (socket says 172.21.15.118:57346) 2024-03-24T00:04:36.193 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:04:35.269+0000 7fb0a1400700 1 -- 172.21.15.118:0/2153625452 learned_addr learned my addr 172.21.15.118:0/2153625452 (peer_addr_for_me v2:172.21.15.118:0/0) 2024-03-24T00:04:36.193 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:04:35.270+0000 7fb0a1400700 1 -- 172.21.15.118:0/2153625452 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb09c10f150 con 0x7fb09c081980 2024-03-24T00:04:36.193 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:04:35.270+0000 7fb0a1400700 1 --2- 172.21.15.118:0/2153625452 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb09c081980 0x7fb09c081da0 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7fb0900098a0 tx=0x7fb090009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b9541ce39d1bd133 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:04:36.193 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:04:35.270+0000 7fb09be00700 1 -- 172.21.15.118:0/2153625452 <== mon.0 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fb090004160 con 0x7fb09c081980 2024-03-24T00:04:36.193 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:04:35.270+0000 7fb09be00700 1 -- 172.21.15.118:0/2153625452 <== mon.0 v2:172.21.15.118:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fb090004300 con 0x7fb09c081980 2024-03-24T00:04:36.193 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:04:35.270+0000 7fb09be00700 1 -- 172.21.15.118:0/2153625452 <== mon.0 v2:172.21.15.118:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fb090011b50 con 0x7fb09c081980 2024-03-24T00:04:36.194 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:04:35.271+0000 7fb0a2800700 1 -- 172.21.15.118:0/2153625452 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb09c081980 msgr2=0x7fb09c081da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:04:36.194 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:04:35.271+0000 7fb0a2800700 1 --2- 172.21.15.118:0/2153625452 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb09c081980 0x7fb09c081da0 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7fb0900098a0 tx=0x7fb090009bb0 comp rx=0 tx=0).stop 2024-03-24T00:04:36.194 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:04:35.271+0000 7fb0a2800700 1 -- 172.21.15.118:0/2153625452 shutdown_connections 2024-03-24T00:04:36.194 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:04:35.271+0000 7fb0a2800700 1 --2- 172.21.15.118:0/2153625452 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb09c081980 0x7fb09c081da0 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:04:36.194 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:04:35.271+0000 7fb0a2800700 1 -- 172.21.15.118:0/2153625452 >> 172.21.15.118:0/2153625452 conn(0x7fb09c07cf60 msgr2=0x7fb09c07f380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:04:36.194 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:04:35.271+0000 7fb0a2800700 1 -- 172.21.15.118:0/2153625452 shutdown_connections 2024-03-24T00:04:36.194 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:04:35.271+0000 7fb0a2800700 1 -- 172.21.15.118:0/2153625452 wait complete. 2024-03-24T00:04:36.194 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:04:35.272+0000 7fb0a2800700 1 Processor -- start 2024-03-24T00:04:36.194 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:04:35.272+0000 7fb0a2800700 1 -- start start 2024-03-24T00:04:36.194 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:04:35.273+0000 7fb0a2800700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb09c081980 0x7fb09c1a1f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:04:36.194 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:04:35.273+0000 7fb0a2800700 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb09c112500 con 0x7fb09c081980 2024-03-24T00:04:36.195 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:04:35.273+0000 7fb0a1400700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb09c081980 0x7fb09c1a1f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:04:36.195 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:04:35.273+0000 7fb0a1400700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb09c081980 0x7fb09c1a1f40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.118:3300/0 says I am v2:172.21.15.118:57354/0 (socket says 172.21.15.118:57354) 2024-03-24T00:04:36.195 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:04:35.273+0000 7fb0a1400700 1 -- 172.21.15.118:0/551144834 learned_addr learned my addr 172.21.15.118:0/551144834 (peer_addr_for_me v2:172.21.15.118:0/0) 2024-03-24T00:04:36.195 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:04:35.273+0000 7fb0a1400700 1 -- 172.21.15.118:0/551144834 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb09c1a2480 con 0x7fb09c081980 2024-03-24T00:04:36.195 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:04:35.274+0000 7fb0a1400700 1 --2- 172.21.15.118:0/551144834 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb09c081980 0x7fb09c1a1f40 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7fb090011670 tx=0x7fb090011b60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:04:36.195 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:04:35.274+0000 7fb09a000700 1 -- 172.21.15.118:0/551144834 <== mon.0 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fb0900039c0 con 0x7fb09c081980 2024-03-24T00:04:36.195 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:04:35.274+0000 7fb09a000700 1 -- 172.21.15.118:0/551144834 <== mon.0 v2:172.21.15.118:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fb090011e50 con 0x7fb09c081980 2024-03-24T00:04:36.195 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:04:35.274+0000 7fb09a000700 1 -- 172.21.15.118:0/551144834 <== mon.0 v2:172.21.15.118:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fb09001a5c0 con 0x7fb09c081980 2024-03-24T00:04:36.195 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:04:35.274+0000 7fb0a2800700 1 -- 172.21.15.118:0/551144834 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb09c1a2710 con 0x7fb09c081980 2024-03-24T00:04:36.195 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:04:35.274+0000 7fb0a2800700 1 -- 172.21.15.118:0/551144834 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb09c1a5400 con 0x7fb09c081980 2024-03-24T00:04:36.195 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:04:35.275+0000 7fb0a2800700 1 -- 172.21.15.118:0/551144834 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb09c0637c0 con 0x7fb09c081980 2024-03-24T00:04:36.195 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:04:35.275+0000 7fb09a000700 1 -- 172.21.15.118:0/551144834 <== mon.0 v2:172.21.15.118:3300/0 4 ==== mgrmap(e 8) v1 ==== 46106+0+0 (secure 0 0 0) 0x7fb090024430 con 0x7fb09c081980 2024-03-24T00:04:36.196 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:04:35.276+0000 7fb09a000700 1 --2- 172.21.15.118:0/551144834 >> [v2:172.21.15.118:6800/389249272,v1:172.21.15.118:6801/389249272] conn(0x7fb084038ed0 0x7fb08403b390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:04:36.196 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:04:35.276+0000 7fb09a000700 1 -- 172.21.15.118:0/551144834 <== mon.0 v2:172.21.15.118:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 940+0+0 (secure 0 0 0) 0x7fb09004d330 con 0x7fb09c081980 2024-03-24T00:04:36.196 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:04:35.276+0000 7fb0a0a00700 1 --2- 172.21.15.118:0/551144834 >> [v2:172.21.15.118:6800/389249272,v1:172.21.15.118:6801/389249272] conn(0x7fb084038ed0 0x7fb08403b390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:04:36.196 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:04:35.277+0000 7fb0a0a00700 1 --2- 172.21.15.118:0/551144834 >> [v2:172.21.15.118:6800/389249272,v1:172.21.15.118:6801/389249272] conn(0x7fb084038ed0 0x7fb08403b390 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7fb088006fd0 tx=0x7fb088006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:04:36.196 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:04:35.282+0000 7fb09a000700 1 -- 172.21.15.118:0/551144834 <== mon.0 v2:172.21.15.118:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7fb09004cc80 con 0x7fb09c081980 2024-03-24T00:04:36.196 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:04:35.501+0000 7fb0a2800700 1 -- 172.21.15.118:0/551144834 --> [v2:172.21.15.118:6800/389249272,v1:172.21.15.118:6801/389249272] -- mgr_command(tid 0: {"prefix": "cephadm set-user", "user": "root", "target": ["mon-mgr", ""]}) v1 -- 0x7fb09c07e1d0 con 0x7fb084038ed0 2024-03-24T00:04:36.196 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:04:35.503+0000 7fb09a000700 1 -- 172.21.15.118:0/551144834 <== mgr.14120 v2:172.21.15.118:6800/389249272 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+16 (secure 0 0 0) 0x7fb09c07e1d0 con 0x7fb084038ed0 2024-03-24T00:04:36.196 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:04:35.505+0000 7fb0a2800700 1 -- 172.21.15.118:0/551144834 >> [v2:172.21.15.118:6800/389249272,v1:172.21.15.118:6801/389249272] conn(0x7fb084038ed0 msgr2=0x7fb08403b390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:04:36.196 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:04:35.505+0000 7fb0a2800700 1 --2- 172.21.15.118:0/551144834 >> [v2:172.21.15.118:6800/389249272,v1:172.21.15.118:6801/389249272] conn(0x7fb084038ed0 0x7fb08403b390 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7fb088006fd0 tx=0x7fb088006e40 comp rx=0 tx=0).stop 2024-03-24T00:04:36.196 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:04:35.505+0000 7fb0a2800700 1 -- 172.21.15.118:0/551144834 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb09c081980 msgr2=0x7fb09c1a1f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:04:36.196 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:04:35.505+0000 7fb0a2800700 1 --2- 172.21.15.118:0/551144834 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb09c081980 0x7fb09c1a1f40 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7fb090011670 tx=0x7fb090011b60 comp rx=0 tx=0).stop 2024-03-24T00:04:36.196 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:04:35.505+0000 7fb0a2800700 1 -- 172.21.15.118:0/551144834 shutdown_connections 2024-03-24T00:04:36.196 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:04:35.505+0000 7fb0a2800700 1 --2- 172.21.15.118:0/551144834 >> [v2:172.21.15.118:6800/389249272,v1:172.21.15.118:6801/389249272] conn(0x7fb084038ed0 0x7fb08403b390 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:04:36.197 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:04:35.505+0000 7fb0a2800700 1 --2- 172.21.15.118:0/551144834 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb09c081980 0x7fb09c1a1f40 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:04:36.197 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:04:35.505+0000 7fb0a2800700 1 -- 172.21.15.118:0/551144834 >> 172.21.15.118:0/551144834 conn(0x7fb09c07cf60 msgr2=0x7fb09c07dac0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:04:36.197 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:04:35.505+0000 7fb0a2800700 1 -- 172.21.15.118:0/551144834 shutdown_connections 2024-03-24T00:04:36.197 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:04:35.506+0000 7fb0a2800700 1 -- 172.21.15.118:0/551144834 wait complete. 2024-03-24T00:04:36.197 INFO:teuthology.orchestra.run.smithi118.stdout:Generating ssh key... 2024-03-24T00:04:36.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:04:36 smithi118 ceph-mon[31535]: from='client.14134 -' entity='client.admin' cmd=[{"prefix": "cephadm set-user", "user": "root", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T00:04:37.307 INFO:journalctl@ceph.mgr.x.smithi118.stdout:Mar 24 00:04:37 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mgr-x[31762]: Generating public/private rsa key pair. 2024-03-24T00:04:37.307 INFO:journalctl@ceph.mgr.x.smithi118.stdout:Mar 24 00:04:37 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mgr-x[31762]: Your identification has been saved in /tmp/tmplqhsgv5n/key. 2024-03-24T00:04:37.307 INFO:journalctl@ceph.mgr.x.smithi118.stdout:Mar 24 00:04:37 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mgr-x[31762]: Your public key has been saved in /tmp/tmplqhsgv5n/key.pub. 2024-03-24T00:04:37.308 INFO:journalctl@ceph.mgr.x.smithi118.stdout:Mar 24 00:04:37 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mgr-x[31762]: The key fingerprint is: 2024-03-24T00:04:37.308 INFO:journalctl@ceph.mgr.x.smithi118.stdout:Mar 24 00:04:37 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mgr-x[31762]: SHA256:Cs69aOb3lm3v9hPiBO4jUyD1MNDn2tL9nxvQoE2JjAc ceph-d79a5420-e971-11ee-95cd-87774f69a715 2024-03-24T00:04:37.308 INFO:journalctl@ceph.mgr.x.smithi118.stdout:Mar 24 00:04:37 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mgr-x[31762]: The key's randomart image is: 2024-03-24T00:04:37.308 INFO:journalctl@ceph.mgr.x.smithi118.stdout:Mar 24 00:04:37 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mgr-x[31762]: +---[RSA 3072]----+ 2024-03-24T00:04:37.308 INFO:journalctl@ceph.mgr.x.smithi118.stdout:Mar 24 00:04:37 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mgr-x[31762]: | .o E | 2024-03-24T00:04:37.308 INFO:journalctl@ceph.mgr.x.smithi118.stdout:Mar 24 00:04:37 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mgr-x[31762]: | = = . . | 2024-03-24T00:04:37.308 INFO:journalctl@ceph.mgr.x.smithi118.stdout:Mar 24 00:04:37 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mgr-x[31762]: | . B + + | 2024-03-24T00:04:37.308 INFO:journalctl@ceph.mgr.x.smithi118.stdout:Mar 24 00:04:37 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mgr-x[31762]: | . . = + o | 2024-03-24T00:04:37.308 INFO:journalctl@ceph.mgr.x.smithi118.stdout:Mar 24 00:04:37 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mgr-x[31762]: | . .S* + o . | 2024-03-24T00:04:37.308 INFO:journalctl@ceph.mgr.x.smithi118.stdout:Mar 24 00:04:37 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mgr-x[31762]: | o o .o = + o | 2024-03-24T00:04:37.308 INFO:journalctl@ceph.mgr.x.smithi118.stdout:Mar 24 00:04:37 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mgr-x[31762]: | o o * o o o | 2024-03-24T00:04:37.308 INFO:journalctl@ceph.mgr.x.smithi118.stdout:Mar 24 00:04:37 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mgr-x[31762]: | o...= = o o o| 2024-03-24T00:04:37.308 INFO:journalctl@ceph.mgr.x.smithi118.stdout:Mar 24 00:04:37 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mgr-x[31762]: | +o..o.+ =o..=o| 2024-03-24T00:04:37.309 INFO:journalctl@ceph.mgr.x.smithi118.stdout:Mar 24 00:04:37 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mgr-x[31762]: +----[SHA256]-----+ 2024-03-24T00:04:37.998 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:04:36.819+0000 7f80fbc00700 1 Processor -- start 2024-03-24T00:04:37.999 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:04:36.819+0000 7f80fbc00700 1 -- start start 2024-03-24T00:04:37.999 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:04:36.819+0000 7f80fbc00700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f80f407e880 0x7f80f407eca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:04:37.999 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:04:36.819+0000 7f80fbc00700 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f80f407f270 con 0x7f80f407e880 2024-03-24T00:04:37.999 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:04:36.819+0000 7f80f9200700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f80f407e880 0x7f80f407eca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:04:37.999 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:04:36.820+0000 7f80f9200700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f80f407e880 0x7f80f407eca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.118:3300/0 says I am v2:172.21.15.118:57364/0 (socket says 172.21.15.118:57364) 2024-03-24T00:04:37.999 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:04:36.820+0000 7f80f9200700 1 -- 172.21.15.118:0/1736194340 learned_addr learned my addr 172.21.15.118:0/1736194340 (peer_addr_for_me v2:172.21.15.118:0/0) 2024-03-24T00:04:37.999 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:04:36.820+0000 7f80f9200700 1 -- 172.21.15.118:0/1736194340 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f80f407fa30 con 0x7f80f407e880 2024-03-24T00:04:37.999 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:04:36.820+0000 7f80f9200700 1 --2- 172.21.15.118:0/1736194340 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f80f407e880 0x7f80f407eca0 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f80e40098a0 tx=0x7f80e4009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=34e6ee77f6c91040 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:04:37.999 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:04:36.821+0000 7f80f3e00700 1 -- 172.21.15.118:0/1736194340 <== mon.0 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f80e4004160 con 0x7f80f407e880 2024-03-24T00:04:37.999 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:04:36.821+0000 7f80f3e00700 1 -- 172.21.15.118:0/1736194340 <== mon.0 v2:172.21.15.118:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f80e4004300 con 0x7f80f407e880 2024-03-24T00:04:37.999 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:04:36.821+0000 7f80f3e00700 1 -- 172.21.15.118:0/1736194340 <== mon.0 v2:172.21.15.118:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f80e4011b50 con 0x7f80f407e880 2024-03-24T00:04:37.999 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:04:36.821+0000 7f80fbc00700 1 -- 172.21.15.118:0/1736194340 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f80f407e880 msgr2=0x7f80f407eca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:04:38.000 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:04:36.821+0000 7f80fbc00700 1 --2- 172.21.15.118:0/1736194340 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f80f407e880 0x7f80f407eca0 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f80e40098a0 tx=0x7f80e4009bb0 comp rx=0 tx=0).stop 2024-03-24T00:04:38.000 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:04:36.821+0000 7f80fbc00700 1 -- 172.21.15.118:0/1736194340 shutdown_connections 2024-03-24T00:04:38.000 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:04:36.821+0000 7f80fbc00700 1 --2- 172.21.15.118:0/1736194340 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f80f407e880 0x7f80f407eca0 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:04:38.000 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:04:36.821+0000 7f80fbc00700 1 -- 172.21.15.118:0/1736194340 >> 172.21.15.118:0/1736194340 conn(0x7f80f407d220 msgr2=0x7f80f407d630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:04:38.000 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:04:36.822+0000 7f80fbc00700 1 -- 172.21.15.118:0/1736194340 shutdown_connections 2024-03-24T00:04:38.000 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:04:36.822+0000 7f80fbc00700 1 -- 172.21.15.118:0/1736194340 wait complete. 2024-03-24T00:04:38.000 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:04:36.823+0000 7f80fbc00700 1 Processor -- start 2024-03-24T00:04:38.000 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:04:36.823+0000 7f80fbc00700 1 -- start start 2024-03-24T00:04:38.000 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:04:36.823+0000 7f80fbc00700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f80f407e880 0x7f80f41a20d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:04:38.000 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:04:36.823+0000 7f80fbc00700 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f80f4080df0 con 0x7f80f407e880 2024-03-24T00:04:38.000 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:04:36.823+0000 7f80f9200700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f80f407e880 0x7f80f41a20d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:04:38.000 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:04:36.823+0000 7f80f9200700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f80f407e880 0x7f80f41a20d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.118:3300/0 says I am v2:172.21.15.118:57376/0 (socket says 172.21.15.118:57376) 2024-03-24T00:04:38.000 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:04:36.823+0000 7f80f9200700 1 -- 172.21.15.118:0/863454007 learned_addr learned my addr 172.21.15.118:0/863454007 (peer_addr_for_me v2:172.21.15.118:0/0) 2024-03-24T00:04:38.001 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:04:36.824+0000 7f80f9200700 1 -- 172.21.15.118:0/863454007 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f80f41a2610 con 0x7f80f407e880 2024-03-24T00:04:38.001 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:04:36.824+0000 7f80f9200700 1 --2- 172.21.15.118:0/863454007 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f80f407e880 0x7f80f41a20d0 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f80e4009fb0 tx=0x7f80e4011bf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:04:38.001 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:04:36.824+0000 7f80f2000700 1 -- 172.21.15.118:0/863454007 <== mon.0 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f80e40038b0 con 0x7f80f407e880 2024-03-24T00:04:38.001 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:04:36.824+0000 7f80f2000700 1 -- 172.21.15.118:0/863454007 <== mon.0 v2:172.21.15.118:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f80e4011e50 con 0x7f80f407e880 2024-03-24T00:04:38.001 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:04:36.824+0000 7f80fbc00700 1 -- 172.21.15.118:0/863454007 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f80f41a28a0 con 0x7f80f407e880 2024-03-24T00:04:38.001 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:04:36.824+0000 7f80f2000700 1 -- 172.21.15.118:0/863454007 <== mon.0 v2:172.21.15.118:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f80e4019420 con 0x7f80f407e880 2024-03-24T00:04:38.001 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:04:36.824+0000 7f80fbc00700 1 -- 172.21.15.118:0/863454007 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f80f41a5590 con 0x7f80f407e880 2024-03-24T00:04:38.001 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:04:36.825+0000 7f80fbc00700 1 -- 172.21.15.118:0/863454007 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f80f40637c0 con 0x7f80f407e880 2024-03-24T00:04:38.001 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:04:36.826+0000 7f80f2000700 1 -- 172.21.15.118:0/863454007 <== mon.0 v2:172.21.15.118:3300/0 4 ==== mgrmap(e 8) v1 ==== 46106+0+0 (secure 0 0 0) 0x7f80e4003a50 con 0x7f80f407e880 2024-03-24T00:04:38.001 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:04:36.826+0000 7f80f2000700 1 --2- 172.21.15.118:0/863454007 >> [v2:172.21.15.118:6800/389249272,v1:172.21.15.118:6801/389249272] conn(0x7f80d4038f20 0x7f80d403b3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:04:38.001 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:04:36.826+0000 7f80f2000700 1 -- 172.21.15.118:0/863454007 <== mon.0 v2:172.21.15.118:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 940+0+0 (secure 0 0 0) 0x7f80e404d0d0 con 0x7f80f407e880 2024-03-24T00:04:38.001 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:04:36.826+0000 7f80ebe00700 1 --2- 172.21.15.118:0/863454007 >> [v2:172.21.15.118:6800/389249272,v1:172.21.15.118:6801/389249272] conn(0x7f80d4038f20 0x7f80d403b3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:04:38.001 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:04:36.827+0000 7f80ebe00700 1 --2- 172.21.15.118:0/863454007 >> [v2:172.21.15.118:6800/389249272,v1:172.21.15.118:6801/389249272] conn(0x7f80d4038f20 0x7f80d403b3e0 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f80dc006fd0 tx=0x7f80dc006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:04:38.002 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:04:36.831+0000 7f80f2000700 1 -- 172.21.15.118:0/863454007 <== mon.0 v2:172.21.15.118:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f80e40196d0 con 0x7f80f407e880 2024-03-24T00:04:38.002 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:04:37.051+0000 7f80fbc00700 1 -- 172.21.15.118:0/863454007 --> [v2:172.21.15.118:6800/389249272,v1:172.21.15.118:6801/389249272] -- mgr_command(tid 0: {"prefix": "cephadm generate-key", "target": ["mon-mgr", ""]}) v1 -- 0x7f80f402dd60 con 0x7f80d4038f20 2024-03-24T00:04:38.002 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:04:37.306+0000 7f80f2000700 1 -- 172.21.15.118:0/863454007 <== mgr.14120 v2:172.21.15.118:6800/389249272 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7f80f402dd60 con 0x7f80d4038f20 2024-03-24T00:04:38.002 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:04:37.309+0000 7f80fbc00700 1 -- 172.21.15.118:0/863454007 >> [v2:172.21.15.118:6800/389249272,v1:172.21.15.118:6801/389249272] conn(0x7f80d4038f20 msgr2=0x7f80d403b3e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:04:38.002 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:04:37.309+0000 7f80fbc00700 1 --2- 172.21.15.118:0/863454007 >> [v2:172.21.15.118:6800/389249272,v1:172.21.15.118:6801/389249272] conn(0x7f80d4038f20 0x7f80d403b3e0 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f80dc006fd0 tx=0x7f80dc006e40 comp rx=0 tx=0).stop 2024-03-24T00:04:38.002 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:04:37.309+0000 7f80fbc00700 1 -- 172.21.15.118:0/863454007 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f80f407e880 msgr2=0x7f80f41a20d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:04:38.002 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:04:37.309+0000 7f80fbc00700 1 --2- 172.21.15.118:0/863454007 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f80f407e880 0x7f80f41a20d0 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f80e4009fb0 tx=0x7f80e4011bf0 comp rx=0 tx=0).stop 2024-03-24T00:04:38.002 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:04:37.309+0000 7f80fbc00700 1 -- 172.21.15.118:0/863454007 shutdown_connections 2024-03-24T00:04:38.002 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:04:37.309+0000 7f80fbc00700 1 --2- 172.21.15.118:0/863454007 >> [v2:172.21.15.118:6800/389249272,v1:172.21.15.118:6801/389249272] conn(0x7f80d4038f20 0x7f80d403b3e0 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:04:38.002 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:04:37.309+0000 7f80fbc00700 1 --2- 172.21.15.118:0/863454007 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f80f407e880 0x7f80f41a20d0 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:04:38.002 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:04:37.309+0000 7f80fbc00700 1 -- 172.21.15.118:0/863454007 >> 172.21.15.118:0/863454007 conn(0x7f80f407d220 msgr2=0x7f80f4198d90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:04:38.002 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:04:37.310+0000 7f80fbc00700 1 -- 172.21.15.118:0/863454007 shutdown_connections 2024-03-24T00:04:38.003 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:04:37.310+0000 7f80fbc00700 1 -- 172.21.15.118:0/863454007 wait complete. 2024-03-24T00:04:38.569 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:04:38 smithi118 ceph-mon[31535]: from='client.14136 -' entity='client.admin' cmd=[{"prefix": "cephadm generate-key", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T00:04:38.570 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:04:38 smithi118 ceph-mon[31535]: Generating ssh key... 2024-03-24T00:04:38.570 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:04:38 smithi118 ceph-mon[31535]: from='mgr.14120 172.21.15.118:0/3476110238' entity='mgr.x' 2024-03-24T00:04:38.570 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:04:38 smithi118 ceph-mon[31535]: from='mgr.14120 172.21.15.118:0/3476110238' entity='mgr.x' 2024-03-24T00:04:39.469 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stdout ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC0pkhYZJq78D9oX9I+RR9b6h4To50g4oxVXM9eLBnZXRbywjN22+kPS1kpmUgkHrAoHTlvADEKSFilDnEvvDC4h6ZKmju/6ACXSZm+tHgr3yYiwVxURvZvLNE/+a1hQ72eJdjwQzEbBkNwIfaKG/PPovP57yD8ZuEpbLTA6F84fOeFrfdjiz/ck6W7PYMrIHVyea28UYDadX8lCE+sPNjcgkTshWHVDFt6CgZcylGYch8Bx8AOui0VAQKYSjpcQHQa+gJfzPgqlRnoe1jWD2PyayCzR0y0UzUGgpgbIEbHBN3Pus+NX41Y8HZPv2qBu/4wD/5ZkHmN2uu8qeg8zhOnudz6mrUXSuPL8QiafgIdVfSdiznUZ61nqyNuukzCIRgtDJVz4eXwXeCZ084FHIpJwy18wdg7j0qUSIKRTEvMFjeQ6Y/jnpWnuH5BqHKJf0DF18w7BPVk1w5wvavscB2XFZ3LYmYieH3aTpdq65Rxxqyec0+BvDgG+WS62/fmhLk= ceph-d79a5420-e971-11ee-95cd-87774f69a715 2024-03-24T00:04:39.469 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:04:38.568+0000 7fb3c2800700 1 Processor -- start 2024-03-24T00:04:39.469 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:04:38.569+0000 7fb3c2800700 1 -- start start 2024-03-24T00:04:39.469 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:04:38.569+0000 7fb3c2800700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb3bc081980 0x7fb3bc081da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:04:39.469 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:04:38.569+0000 7fb3c2800700 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb3bc082370 con 0x7fb3bc081980 2024-03-24T00:04:39.469 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:04:38.570+0000 7fb3c1400700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb3bc081980 0x7fb3bc081da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:04:39.469 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:04:38.570+0000 7fb3c1400700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb3bc081980 0x7fb3bc081da0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.118:3300/0 says I am v2:172.21.15.118:57392/0 (socket says 172.21.15.118:57392) 2024-03-24T00:04:39.469 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:04:38.570+0000 7fb3c1400700 1 -- 172.21.15.118:0/347462655 learned_addr learned my addr 172.21.15.118:0/347462655 (peer_addr_for_me v2:172.21.15.118:0/0) 2024-03-24T00:04:39.469 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:04:38.570+0000 7fb3c1400700 1 -- 172.21.15.118:0/347462655 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb3bc10f150 con 0x7fb3bc081980 2024-03-24T00:04:39.469 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:04:38.571+0000 7fb3c1400700 1 --2- 172.21.15.118:0/347462655 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb3bc081980 0x7fb3bc081da0 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7fb3b00098a0 tx=0x7fb3b0009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f0da6857628ab8fc server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:04:39.470 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:04:38.571+0000 7fb3bbe00700 1 -- 172.21.15.118:0/347462655 <== mon.0 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fb3b0004160 con 0x7fb3bc081980 2024-03-24T00:04:39.470 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:04:38.571+0000 7fb3bbe00700 1 -- 172.21.15.118:0/347462655 <== mon.0 v2:172.21.15.118:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fb3b0004300 con 0x7fb3bc081980 2024-03-24T00:04:39.470 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:04:38.571+0000 7fb3bbe00700 1 -- 172.21.15.118:0/347462655 <== mon.0 v2:172.21.15.118:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fb3b0011b50 con 0x7fb3bc081980 2024-03-24T00:04:39.470 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:04:38.572+0000 7fb3c2800700 1 -- 172.21.15.118:0/347462655 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb3bc081980 msgr2=0x7fb3bc081da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:04:39.470 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:04:38.572+0000 7fb3c2800700 1 --2- 172.21.15.118:0/347462655 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb3bc081980 0x7fb3bc081da0 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7fb3b00098a0 tx=0x7fb3b0009bb0 comp rx=0 tx=0).stop 2024-03-24T00:04:39.470 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:04:38.572+0000 7fb3c2800700 1 -- 172.21.15.118:0/347462655 shutdown_connections 2024-03-24T00:04:39.470 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:04:38.572+0000 7fb3c2800700 1 --2- 172.21.15.118:0/347462655 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb3bc081980 0x7fb3bc081da0 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:04:39.470 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:04:38.572+0000 7fb3c2800700 1 -- 172.21.15.118:0/347462655 >> 172.21.15.118:0/347462655 conn(0x7fb3bc07cf60 msgr2=0x7fb3bc07f380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:04:39.470 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:04:38.572+0000 7fb3c2800700 1 -- 172.21.15.118:0/347462655 shutdown_connections 2024-03-24T00:04:39.470 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:04:38.572+0000 7fb3c2800700 1 -- 172.21.15.118:0/347462655 wait complete. 2024-03-24T00:04:39.470 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:04:38.573+0000 7fb3c2800700 1 Processor -- start 2024-03-24T00:04:39.470 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:04:38.573+0000 7fb3c2800700 1 -- start start 2024-03-24T00:04:39.470 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:04:38.573+0000 7fb3c2800700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb3bc081980 0x7fb3bc1a1f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:04:39.471 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:04:38.573+0000 7fb3c2800700 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb3bc112500 con 0x7fb3bc081980 2024-03-24T00:04:39.471 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:04:38.574+0000 7fb3c1400700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb3bc081980 0x7fb3bc1a1f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:04:39.471 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:04:38.574+0000 7fb3c1400700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb3bc081980 0x7fb3bc1a1f40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.118:3300/0 says I am v2:172.21.15.118:57400/0 (socket says 172.21.15.118:57400) 2024-03-24T00:04:39.471 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:04:38.574+0000 7fb3c1400700 1 -- 172.21.15.118:0/3348243729 learned_addr learned my addr 172.21.15.118:0/3348243729 (peer_addr_for_me v2:172.21.15.118:0/0) 2024-03-24T00:04:39.471 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:04:38.574+0000 7fb3c1400700 1 -- 172.21.15.118:0/3348243729 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb3bc1a2480 con 0x7fb3bc081980 2024-03-24T00:04:39.471 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:04:38.574+0000 7fb3c1400700 1 --2- 172.21.15.118:0/3348243729 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb3bc081980 0x7fb3bc1a1f40 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7fb3b0000c00 tx=0x7fb3b0011c10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:04:39.471 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:04:38.574+0000 7fb3ba000700 1 -- 172.21.15.118:0/3348243729 <== mon.0 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fb3b0003b20 con 0x7fb3bc081980 2024-03-24T00:04:39.471 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:04:38.574+0000 7fb3ba000700 1 -- 172.21.15.118:0/3348243729 <== mon.0 v2:172.21.15.118:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fb3b0003cc0 con 0x7fb3bc081980 2024-03-24T00:04:39.471 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:04:38.574+0000 7fb3ba000700 1 -- 172.21.15.118:0/3348243729 <== mon.0 v2:172.21.15.118:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fb3b0024910 con 0x7fb3bc081980 2024-03-24T00:04:39.471 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:04:38.574+0000 7fb3c2800700 1 -- 172.21.15.118:0/3348243729 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb3bc1a2710 con 0x7fb3bc081980 2024-03-24T00:04:39.471 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:04:38.575+0000 7fb3c2800700 1 -- 172.21.15.118:0/3348243729 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb3bc1a5400 con 0x7fb3bc081980 2024-03-24T00:04:39.471 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:04:38.576+0000 7fb3ba000700 1 -- 172.21.15.118:0/3348243729 <== mon.0 v2:172.21.15.118:3300/0 4 ==== mgrmap(e 8) v1 ==== 46106+0+0 (secure 0 0 0) 0x7fb3b0022020 con 0x7fb3bc081980 2024-03-24T00:04:39.471 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:04:38.576+0000 7fb3c2800700 1 -- 172.21.15.118:0/3348243729 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb3bc0637c0 con 0x7fb3bc081980 2024-03-24T00:04:39.471 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:04:38.576+0000 7fb3ba000700 1 --2- 172.21.15.118:0/3348243729 >> [v2:172.21.15.118:6800/389249272,v1:172.21.15.118:6801/389249272] conn(0x7fb3a4038f70 0x7fb3a403b430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:04:39.472 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:04:38.576+0000 7fb3ba000700 1 -- 172.21.15.118:0/3348243729 <== mon.0 v2:172.21.15.118:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 940+0+0 (secure 0 0 0) 0x7fb3b0011e00 con 0x7fb3bc081980 2024-03-24T00:04:39.472 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:04:38.576+0000 7fb3c0a00700 1 --2- 172.21.15.118:0/3348243729 >> [v2:172.21.15.118:6800/389249272,v1:172.21.15.118:6801/389249272] conn(0x7fb3a4038f70 0x7fb3a403b430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:04:39.472 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:04:38.577+0000 7fb3c0a00700 1 --2- 172.21.15.118:0/3348243729 >> [v2:172.21.15.118:6800/389249272,v1:172.21.15.118:6801/389249272] conn(0x7fb3a4038f70 0x7fb3a403b430 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7fb3a8006fd0 tx=0x7fb3a8006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:04:39.472 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:04:38.582+0000 7fb3ba000700 1 -- 172.21.15.118:0/3348243729 <== mon.0 v2:172.21.15.118:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7fb3b0016260 con 0x7fb3bc081980 2024-03-24T00:04:39.472 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:04:38.804+0000 7fb3c2800700 1 -- 172.21.15.118:0/3348243729 --> [v2:172.21.15.118:6800/389249272,v1:172.21.15.118:6801/389249272] -- mgr_command(tid 0: {"prefix": "cephadm get-pub-key", "target": ["mon-mgr", ""]}) v1 -- 0x7fb3bc07e1d0 con 0x7fb3a4038f70 2024-03-24T00:04:39.472 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:04:38.805+0000 7fb3ba000700 1 -- 172.21.15.118:0/3348243729 <== mgr.14120 v2:172.21.15.118:6800/389249272 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+595 (secure 0 0 0) 0x7fb3bc07e1d0 con 0x7fb3a4038f70 2024-03-24T00:04:39.472 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:04:38.808+0000 7fb3c2800700 1 -- 172.21.15.118:0/3348243729 >> [v2:172.21.15.118:6800/389249272,v1:172.21.15.118:6801/389249272] conn(0x7fb3a4038f70 msgr2=0x7fb3a403b430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:04:39.472 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:04:38.808+0000 7fb3c2800700 1 --2- 172.21.15.118:0/3348243729 >> [v2:172.21.15.118:6800/389249272,v1:172.21.15.118:6801/389249272] conn(0x7fb3a4038f70 0x7fb3a403b430 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7fb3a8006fd0 tx=0x7fb3a8006e40 comp rx=0 tx=0).stop 2024-03-24T00:04:39.472 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:04:38.808+0000 7fb3c2800700 1 -- 172.21.15.118:0/3348243729 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb3bc081980 msgr2=0x7fb3bc1a1f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:04:39.472 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:04:38.808+0000 7fb3c2800700 1 --2- 172.21.15.118:0/3348243729 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb3bc081980 0x7fb3bc1a1f40 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7fb3b0000c00 tx=0x7fb3b0011c10 comp rx=0 tx=0).stop 2024-03-24T00:04:39.472 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:04:38.808+0000 7fb3c2800700 1 -- 172.21.15.118:0/3348243729 shutdown_connections 2024-03-24T00:04:39.472 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:04:38.808+0000 7fb3c2800700 1 --2- 172.21.15.118:0/3348243729 >> [v2:172.21.15.118:6800/389249272,v1:172.21.15.118:6801/389249272] conn(0x7fb3a4038f70 0x7fb3a403b430 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:04:39.472 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:04:38.808+0000 7fb3c2800700 1 --2- 172.21.15.118:0/3348243729 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb3bc081980 0x7fb3bc1a1f40 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:04:39.473 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:04:38.808+0000 7fb3c2800700 1 -- 172.21.15.118:0/3348243729 >> 172.21.15.118:0/3348243729 conn(0x7fb3bc07cf60 msgr2=0x7fb3bc07dac0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:04:39.473 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:04:38.808+0000 7fb3c2800700 1 -- 172.21.15.118:0/3348243729 shutdown_connections 2024-03-24T00:04:39.473 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:04:38.808+0000 7fb3c2800700 1 -- 172.21.15.118:0/3348243729 wait complete. 2024-03-24T00:04:39.473 INFO:teuthology.orchestra.run.smithi118.stdout:Wrote public SSH key to /home/ubuntu/cephtest/ceph.pub 2024-03-24T00:04:39.473 INFO:teuthology.orchestra.run.smithi118.stdout:Adding key to root@localhost authorized_keys... 2024-03-24T00:04:39.473 INFO:teuthology.orchestra.run.smithi118.stdout:Adding host smithi118... 2024-03-24T00:04:39.758 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:04:39 smithi118 ceph-mon[31535]: from='client.14138 -' entity='client.admin' cmd=[{"prefix": "cephadm get-pub-key", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T00:04:40.945 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:04:40 smithi118 ceph-mon[31535]: from='client.14140 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi118", "addr": "172.21.15.118", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T00:04:42.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:04:42 smithi118 ceph-mon[31535]: Deploying cephadm binary to smithi118 2024-03-24T00:04:47.083 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stdout Added host 'smithi118' with addr '172.21.15.118' 2024-03-24T00:04:47.083 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:04:40.036+0000 7f768ce00700 1 Processor -- start 2024-03-24T00:04:47.083 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:04:40.036+0000 7f768ce00700 1 -- start start 2024-03-24T00:04:47.083 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:04:40.036+0000 7f768ce00700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f7688108560 0x7f768810a950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:04:47.084 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:04:40.036+0000 7f768ce00700 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7688075900 con 0x7f7688108560 2024-03-24T00:04:47.084 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:04:40.037+0000 7f7685e00700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f7688108560 0x7f768810a950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:04:47.084 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:04:40.037+0000 7f7685e00700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f7688108560 0x7f768810a950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.118:3300/0 says I am v2:172.21.15.118:57404/0 (socket says 172.21.15.118:57404) 2024-03-24T00:04:47.084 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:04:40.037+0000 7f7685e00700 1 -- 172.21.15.118:0/3833073759 learned_addr learned my addr 172.21.15.118:0/3833073759 (peer_addr_for_me v2:172.21.15.118:0/0) 2024-03-24T00:04:47.084 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:04:40.038+0000 7f7685e00700 1 -- 172.21.15.118:0/3833073759 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f768810ae90 con 0x7f7688108560 2024-03-24T00:04:47.084 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:04:40.038+0000 7f7685e00700 1 --2- 172.21.15.118:0/3833073759 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f7688108560 0x7f768810a950 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f76780098a0 tx=0x7f7678009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5683ba54ea6ebae9 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:04:47.084 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:04:40.038+0000 7f7685400700 1 -- 172.21.15.118:0/3833073759 <== mon.0 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f7678004160 con 0x7f7688108560 2024-03-24T00:04:47.084 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:04:40.039+0000 7f7685400700 1 -- 172.21.15.118:0/3833073759 <== mon.0 v2:172.21.15.118:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f7678004300 con 0x7f7688108560 2024-03-24T00:04:47.084 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:04:40.039+0000 7f7685400700 1 -- 172.21.15.118:0/3833073759 <== mon.0 v2:172.21.15.118:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f7678011b50 con 0x7f7688108560 2024-03-24T00:04:47.084 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:04:40.039+0000 7f768ce00700 1 -- 172.21.15.118:0/3833073759 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f7688108560 msgr2=0x7f768810a950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:04:47.085 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:04:40.039+0000 7f768ce00700 1 --2- 172.21.15.118:0/3833073759 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f7688108560 0x7f768810a950 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f76780098a0 tx=0x7f7678009bb0 comp rx=0 tx=0).stop 2024-03-24T00:04:47.085 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:04:40.039+0000 7f768ce00700 1 -- 172.21.15.118:0/3833073759 shutdown_connections 2024-03-24T00:04:47.085 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:04:40.039+0000 7f768ce00700 1 --2- 172.21.15.118:0/3833073759 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f7688108560 0x7f768810a950 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:04:47.085 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:04:40.039+0000 7f768ce00700 1 -- 172.21.15.118:0/3833073759 >> 172.21.15.118:0/3833073759 conn(0x7f7688101fb0 msgr2=0x7f76881043d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:04:47.085 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:04:40.040+0000 7f768ce00700 1 -- 172.21.15.118:0/3833073759 shutdown_connections 2024-03-24T00:04:47.085 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:04:40.040+0000 7f768ce00700 1 -- 172.21.15.118:0/3833073759 wait complete. 2024-03-24T00:04:47.085 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:04:40.041+0000 7f768ce00700 1 Processor -- start 2024-03-24T00:04:47.085 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:04:40.041+0000 7f768ce00700 1 -- start start 2024-03-24T00:04:47.085 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:04:40.041+0000 7f768ce00700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f7688108560 0x7f768819baf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:04:47.085 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:04:40.041+0000 7f768ce00700 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f768810bec0 con 0x7f7688108560 2024-03-24T00:04:47.085 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:04:40.041+0000 7f7685e00700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f7688108560 0x7f768819baf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:04:47.085 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:04:40.041+0000 7f7685e00700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f7688108560 0x7f768819baf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.118:3300/0 says I am v2:172.21.15.118:57418/0 (socket says 172.21.15.118:57418) 2024-03-24T00:04:47.085 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:04:40.041+0000 7f7685e00700 1 -- 172.21.15.118:0/713914431 learned_addr learned my addr 172.21.15.118:0/713914431 (peer_addr_for_me v2:172.21.15.118:0/0) 2024-03-24T00:04:47.085 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:04:40.042+0000 7f7685e00700 1 -- 172.21.15.118:0/713914431 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f768819c030 con 0x7f7688108560 2024-03-24T00:04:47.086 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:04:40.042+0000 7f7685e00700 1 --2- 172.21.15.118:0/713914431 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f7688108560 0x7f768819baf0 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f7678009fb0 tx=0x7f7678011bf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:04:47.086 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:04:40.042+0000 7f767ea00700 1 -- 172.21.15.118:0/713914431 <== mon.0 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f76780038b0 con 0x7f7688108560 2024-03-24T00:04:47.086 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:04:40.042+0000 7f767ea00700 1 -- 172.21.15.118:0/713914431 <== mon.0 v2:172.21.15.118:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f7678011e50 con 0x7f7688108560 2024-03-24T00:04:47.086 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:04:40.042+0000 7f768ce00700 1 -- 172.21.15.118:0/713914431 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f768819c2c0 con 0x7f7688108560 2024-03-24T00:04:47.086 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:04:40.042+0000 7f767ea00700 1 -- 172.21.15.118:0/713914431 <== mon.0 v2:172.21.15.118:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f7678019420 con 0x7f7688108560 2024-03-24T00:04:47.086 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:04:40.043+0000 7f768ce00700 1 -- 172.21.15.118:0/713914431 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f768819efb0 con 0x7f7688108560 2024-03-24T00:04:47.086 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:04:40.044+0000 7f767ea00700 1 -- 172.21.15.118:0/713914431 <== mon.0 v2:172.21.15.118:3300/0 4 ==== mgrmap(e 8) v1 ==== 46106+0+0 (secure 0 0 0) 0x7f7678003a50 con 0x7f7688108560 2024-03-24T00:04:47.086 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:04:40.044+0000 7f768ce00700 1 -- 172.21.15.118:0/713914431 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f76880637c0 con 0x7f7688108560 2024-03-24T00:04:47.086 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:04:40.044+0000 7f767ea00700 1 --2- 172.21.15.118:0/713914431 >> [v2:172.21.15.118:6800/389249272,v1:172.21.15.118:6801/389249272] conn(0x7f766c038f20 0x7f766c03b3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:04:47.086 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:04:40.044+0000 7f767ea00700 1 -- 172.21.15.118:0/713914431 <== mon.0 v2:172.21.15.118:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 940+0+0 (secure 0 0 0) 0x7f767804d120 con 0x7f7688108560 2024-03-24T00:04:47.086 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:04:40.044+0000 7f767fe00700 1 --2- 172.21.15.118:0/713914431 >> [v2:172.21.15.118:6800/389249272,v1:172.21.15.118:6801/389249272] conn(0x7f766c038f20 0x7f766c03b3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:04:47.087 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:04:40.045+0000 7f767fe00700 1 --2- 172.21.15.118:0/713914431 >> [v2:172.21.15.118:6800/389249272,v1:172.21.15.118:6801/389249272] conn(0x7f766c038f20 0x7f766c03b3e0 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f7670006fd0 tx=0x7f7670006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:04:47.087 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:04:40.050+0000 7f767ea00700 1 -- 172.21.15.118:0/713914431 <== mon.0 v2:172.21.15.118:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f767804ca70 con 0x7f7688108560 2024-03-24T00:04:47.087 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:04:40.277+0000 7f768ce00700 1 -- 172.21.15.118:0/713914431 --> [v2:172.21.15.118:6800/389249272,v1:172.21.15.118:6801/389249272] -- mgr_command(tid 0: {"prefix": "orch host add", "hostname": "smithi118", "addr": "172.21.15.118", "target": ["mon-mgr", ""]}) v1 -- 0x7f76881031f0 con 0x7f766c038f20 2024-03-24T00:04:47.087 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:04:46.455+0000 7f767ea00700 1 -- 172.21.15.118:0/713914431 <== mgr.14120 v2:172.21.15.118:6800/389249272 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+49 (secure 0 0 0) 0x7f76881031f0 con 0x7f766c038f20 2024-03-24T00:04:47.087 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:04:46.459+0000 7f768ce00700 1 -- 172.21.15.118:0/713914431 >> [v2:172.21.15.118:6800/389249272,v1:172.21.15.118:6801/389249272] conn(0x7f766c038f20 msgr2=0x7f766c03b3e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:04:47.088 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:04:46.459+0000 7f768ce00700 1 --2- 172.21.15.118:0/713914431 >> [v2:172.21.15.118:6800/389249272,v1:172.21.15.118:6801/389249272] conn(0x7f766c038f20 0x7f766c03b3e0 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f7670006fd0 tx=0x7f7670006e40 comp rx=0 tx=0).stop 2024-03-24T00:04:47.088 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:04:46.459+0000 7f768ce00700 1 -- 172.21.15.118:0/713914431 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f7688108560 msgr2=0x7f768819baf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:04:47.088 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:04:46.459+0000 7f768ce00700 1 --2- 172.21.15.118:0/713914431 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f7688108560 0x7f768819baf0 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f7678009fb0 tx=0x7f7678011bf0 comp rx=0 tx=0).stop 2024-03-24T00:04:47.088 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:04:46.460+0000 7f768ce00700 1 -- 172.21.15.118:0/713914431 shutdown_connections 2024-03-24T00:04:47.088 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:04:46.460+0000 7f768ce00700 1 --2- 172.21.15.118:0/713914431 >> [v2:172.21.15.118:6800/389249272,v1:172.21.15.118:6801/389249272] conn(0x7f766c038f20 0x7f766c03b3e0 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:04:47.088 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:04:46.460+0000 7f768ce00700 1 --2- 172.21.15.118:0/713914431 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f7688108560 0x7f768819baf0 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:04:47.088 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:04:46.460+0000 7f768ce00700 1 -- 172.21.15.118:0/713914431 >> 172.21.15.118:0/713914431 conn(0x7f7688101fb0 msgr2=0x7f7688102ae0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:04:47.088 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:04:46.460+0000 7f768ce00700 1 -- 172.21.15.118:0/713914431 shutdown_connections 2024-03-24T00:04:47.089 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:04:46.460+0000 7f768ce00700 1 -- 172.21.15.118:0/713914431 wait complete. 2024-03-24T00:04:47.089 INFO:teuthology.orchestra.run.smithi118.stdout:Deploying unmanaged mon service... 2024-03-24T00:04:47.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:04:47 smithi118 ceph-mon[31535]: from='mgr.14120 172.21.15.118:0/3476110238' entity='mgr.x' 2024-03-24T00:04:47.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:04:47 smithi118 ceph-mon[31535]: Added host smithi118 2024-03-24T00:04:47.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:04:47 smithi118 ceph-mon[31535]: from='mgr.14120 172.21.15.118:0/3476110238' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T00:04:49.109 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stdout Scheduled mon update... 2024-03-24T00:04:49.109 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:04:47.703+0000 7ff9f2800700 1 Processor -- start 2024-03-24T00:04:49.109 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:04:47.703+0000 7ff9f2800700 1 -- start start 2024-03-24T00:04:49.110 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:04:47.703+0000 7ff9f2800700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7ff9ec107af0 0x7ff9ec107f10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:04:49.110 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:04:47.703+0000 7ff9f2800700 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff9ec1084e0 con 0x7ff9ec107af0 2024-03-24T00:04:49.110 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:04:47.704+0000 7ff9eb400700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7ff9ec107af0 0x7ff9ec107f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:04:49.110 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:04:47.704+0000 7ff9eb400700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7ff9ec107af0 0x7ff9ec107f10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.118:3300/0 says I am v2:172.21.15.118:51448/0 (socket says 172.21.15.118:51448) 2024-03-24T00:04:49.111 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:04:47.704+0000 7ff9eb400700 1 -- 172.21.15.118:0/3033099531 learned_addr learned my addr 172.21.15.118:0/3033099531 (peer_addr_for_me v2:172.21.15.118:0/0) 2024-03-24T00:04:49.111 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:04:47.705+0000 7ff9eb400700 1 -- 172.21.15.118:0/3033099531 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff9ec108cf0 con 0x7ff9ec107af0 2024-03-24T00:04:49.111 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:04:47.705+0000 7ff9eb400700 1 --2- 172.21.15.118:0/3033099531 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7ff9ec107af0 0x7ff9ec107f10 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7ff9dc0098a0 tx=0x7ff9dc009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=378c6dd66acd3fc8 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:04:49.111 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:04:47.705+0000 7ff9ea000700 1 -- 172.21.15.118:0/3033099531 <== mon.0 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7ff9dc004160 con 0x7ff9ec107af0 2024-03-24T00:04:49.111 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:04:47.705+0000 7ff9ea000700 1 -- 172.21.15.118:0/3033099531 <== mon.0 v2:172.21.15.118:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7ff9dc004300 con 0x7ff9ec107af0 2024-03-24T00:04:49.112 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:04:47.705+0000 7ff9ea000700 1 -- 172.21.15.118:0/3033099531 <== mon.0 v2:172.21.15.118:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7ff9dc011b50 con 0x7ff9ec107af0 2024-03-24T00:04:49.112 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:04:47.706+0000 7ff9f2800700 1 -- 172.21.15.118:0/3033099531 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7ff9ec107af0 msgr2=0x7ff9ec107f10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:04:49.112 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:04:47.706+0000 7ff9f2800700 1 --2- 172.21.15.118:0/3033099531 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7ff9ec107af0 0x7ff9ec107f10 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7ff9dc0098a0 tx=0x7ff9dc009bb0 comp rx=0 tx=0).stop 2024-03-24T00:04:49.113 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:04:47.706+0000 7ff9f2800700 1 -- 172.21.15.118:0/3033099531 shutdown_connections 2024-03-24T00:04:49.113 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:04:47.706+0000 7ff9f2800700 1 --2- 172.21.15.118:0/3033099531 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7ff9ec107af0 0x7ff9ec107f10 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:04:49.113 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:04:47.706+0000 7ff9f2800700 1 -- 172.21.15.118:0/3033099531 >> 172.21.15.118:0/3033099531 conn(0x7ff9ec103070 msgr2=0x7ff9ec1054d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:04:49.113 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:04:47.707+0000 7ff9f2800700 1 -- 172.21.15.118:0/3033099531 shutdown_connections 2024-03-24T00:04:49.113 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:04:47.707+0000 7ff9f2800700 1 -- 172.21.15.118:0/3033099531 wait complete. 2024-03-24T00:04:49.113 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:04:47.708+0000 7ff9f2800700 1 Processor -- start 2024-03-24T00:04:49.113 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:04:47.708+0000 7ff9f2800700 1 -- start start 2024-03-24T00:04:49.113 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:04:47.708+0000 7ff9f2800700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7ff9ec107af0 0x7ff9ec19bac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:04:49.113 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:04:47.708+0000 7ff9f2800700 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff9ec10c0a0 con 0x7ff9ec107af0 2024-03-24T00:04:49.113 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:04:47.708+0000 7ff9eb400700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7ff9ec107af0 0x7ff9ec19bac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:04:49.113 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:04:47.708+0000 7ff9eb400700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7ff9ec107af0 0x7ff9ec19bac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.118:3300/0 says I am v2:172.21.15.118:51452/0 (socket says 172.21.15.118:51452) 2024-03-24T00:04:49.116 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:04:47.708+0000 7ff9eb400700 1 -- 172.21.15.118:0/175281733 learned_addr learned my addr 172.21.15.118:0/175281733 (peer_addr_for_me v2:172.21.15.118:0/0) 2024-03-24T00:04:49.116 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:04:47.709+0000 7ff9eb400700 1 -- 172.21.15.118:0/175281733 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff9ec19c000 con 0x7ff9ec107af0 2024-03-24T00:04:49.117 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:04:47.709+0000 7ff9eb400700 1 --2- 172.21.15.118:0/175281733 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7ff9ec107af0 0x7ff9ec19bac0 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7ff9dc011670 tx=0x7ff9dc011b60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:04:49.117 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:04:47.709+0000 7ff9e3e00700 1 -- 172.21.15.118:0/175281733 <== mon.0 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7ff9dc0039c0 con 0x7ff9ec107af0 2024-03-24T00:04:49.117 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:04:47.709+0000 7ff9e3e00700 1 -- 172.21.15.118:0/175281733 <== mon.0 v2:172.21.15.118:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7ff9dc011e50 con 0x7ff9ec107af0 2024-03-24T00:04:49.117 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:04:47.709+0000 7ff9f2800700 1 -- 172.21.15.118:0/175281733 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff9ec19c290 con 0x7ff9ec107af0 2024-03-24T00:04:49.117 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:04:47.709+0000 7ff9e3e00700 1 -- 172.21.15.118:0/175281733 <== mon.0 v2:172.21.15.118:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7ff9dc01a5c0 con 0x7ff9ec107af0 2024-03-24T00:04:49.117 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:04:47.710+0000 7ff9f2800700 1 -- 172.21.15.118:0/175281733 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff9ec19ef80 con 0x7ff9ec107af0 2024-03-24T00:04:49.117 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:04:47.711+0000 7ff9f2800700 1 -- 172.21.15.118:0/175281733 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff9ec0637c0 con 0x7ff9ec107af0 2024-03-24T00:04:49.117 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:04:47.711+0000 7ff9e3e00700 1 -- 172.21.15.118:0/175281733 <== mon.0 v2:172.21.15.118:3300/0 4 ==== mgrmap(e 8) v1 ==== 46106+0+0 (secure 0 0 0) 0x7ff9dc024430 con 0x7ff9ec107af0 2024-03-24T00:04:49.117 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:04:47.711+0000 7ff9e3e00700 1 --2- 172.21.15.118:0/175281733 >> [v2:172.21.15.118:6800/389249272,v1:172.21.15.118:6801/389249272] conn(0x7ff9d0038ed0 0x7ff9d003b390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:04:49.117 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:04:47.711+0000 7ff9e3e00700 1 -- 172.21.15.118:0/175281733 <== mon.0 v2:172.21.15.118:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 940+0+0 (secure 0 0 0) 0x7ff9dc04d350 con 0x7ff9ec107af0 2024-03-24T00:04:49.117 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:04:47.712+0000 7ff9eaa00700 1 --2- 172.21.15.118:0/175281733 >> [v2:172.21.15.118:6800/389249272,v1:172.21.15.118:6801/389249272] conn(0x7ff9d0038ed0 0x7ff9d003b390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:04:49.117 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:04:47.712+0000 7ff9eaa00700 1 --2- 172.21.15.118:0/175281733 >> [v2:172.21.15.118:6800/389249272,v1:172.21.15.118:6801/389249272] conn(0x7ff9d0038ed0 0x7ff9d003b390 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7ff9d8006fd0 tx=0x7ff9d8006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:04:49.118 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:04:47.717+0000 7ff9e3e00700 1 -- 172.21.15.118:0/175281733 <== mon.0 v2:172.21.15.118:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7ff9dc02d910 con 0x7ff9ec107af0 2024-03-24T00:04:49.118 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:04:47.961+0000 7ff9f2800700 1 -- 172.21.15.118:0/175281733 --> [v2:172.21.15.118:6800/389249272,v1:172.21.15.118:6801/389249272] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "mon", "unmanaged": true, "target": ["mon-mgr", ""]}) v1 -- 0x7ff9ec1042e0 con 0x7ff9d0038ed0 2024-03-24T00:04:49.118 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:04:47.973+0000 7ff9e3e00700 1 -- 172.21.15.118:0/175281733 <== mgr.14120 v2:172.21.15.118:6800/389249272 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+24 (secure 0 0 0) 0x7ff9ec1042e0 con 0x7ff9d0038ed0 2024-03-24T00:04:49.118 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:04:47.976+0000 7ff9f2800700 1 -- 172.21.15.118:0/175281733 >> [v2:172.21.15.118:6800/389249272,v1:172.21.15.118:6801/389249272] conn(0x7ff9d0038ed0 msgr2=0x7ff9d003b390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:04:49.118 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:04:47.976+0000 7ff9f2800700 1 --2- 172.21.15.118:0/175281733 >> [v2:172.21.15.118:6800/389249272,v1:172.21.15.118:6801/389249272] conn(0x7ff9d0038ed0 0x7ff9d003b390 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7ff9d8006fd0 tx=0x7ff9d8006e40 comp rx=0 tx=0).stop 2024-03-24T00:04:49.118 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:04:47.976+0000 7ff9f2800700 1 -- 172.21.15.118:0/175281733 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7ff9ec107af0 msgr2=0x7ff9ec19bac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:04:49.118 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:04:47.976+0000 7ff9f2800700 1 --2- 172.21.15.118:0/175281733 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7ff9ec107af0 0x7ff9ec19bac0 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7ff9dc011670 tx=0x7ff9dc011b60 comp rx=0 tx=0).stop 2024-03-24T00:04:49.118 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:04:47.976+0000 7ff9f2800700 1 -- 172.21.15.118:0/175281733 shutdown_connections 2024-03-24T00:04:49.118 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:04:47.976+0000 7ff9f2800700 1 --2- 172.21.15.118:0/175281733 >> [v2:172.21.15.118:6800/389249272,v1:172.21.15.118:6801/389249272] conn(0x7ff9d0038ed0 0x7ff9d003b390 unknown :-1 s=CLOSED pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:04:49.118 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:04:47.976+0000 7ff9f2800700 1 --2- 172.21.15.118:0/175281733 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7ff9ec107af0 0x7ff9ec19bac0 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:04:49.118 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:04:47.976+0000 7ff9f2800700 1 -- 172.21.15.118:0/175281733 >> 172.21.15.118:0/175281733 conn(0x7ff9ec103070 msgr2=0x7ff9ec103bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:04:49.118 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:04:47.977+0000 7ff9f2800700 1 -- 172.21.15.118:0/175281733 shutdown_connections 2024-03-24T00:04:49.118 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:04:47.977+0000 7ff9f2800700 1 -- 172.21.15.118:0/175281733 wait complete. 2024-03-24T00:04:49.119 INFO:teuthology.orchestra.run.smithi118.stdout:Deploying unmanaged mgr service... 2024-03-24T00:04:49.238 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:04:48 smithi118 ceph-mon[31535]: from='client.14142 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "unmanaged": true, "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T00:04:49.238 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:04:48 smithi118 ceph-mon[31535]: Saving service mon spec with placement count:5 2024-03-24T00:04:49.238 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:04:48 smithi118 ceph-mon[31535]: from='mgr.14120 172.21.15.118:0/3476110238' entity='mgr.x' 2024-03-24T00:04:51.444 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stdout Scheduled mgr update... 2024-03-24T00:04:51.444 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:04:50.227+0000 7f9f90600700 1 Processor -- start 2024-03-24T00:04:51.444 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:04:50.227+0000 7f9f90600700 1 -- start start 2024-03-24T00:04:51.444 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:04:50.228+0000 7f9f90600700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f9f881069f0 0x7f9f88106e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:04:51.444 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:04:50.228+0000 7f9f90600700 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9f881073e0 con 0x7f9f881069f0 2024-03-24T00:04:51.444 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:04:50.228+0000 7f9f8dc00700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f9f881069f0 0x7f9f88106e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:04:51.444 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:04:50.228+0000 7f9f8dc00700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f9f881069f0 0x7f9f88106e10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.118:3300/0 says I am v2:172.21.15.118:51466/0 (socket says 172.21.15.118:51466) 2024-03-24T00:04:51.450 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:04:50.228+0000 7f9f8dc00700 1 -- 172.21.15.118:0/2986369374 learned_addr learned my addr 172.21.15.118:0/2986369374 (peer_addr_for_me v2:172.21.15.118:0/0) 2024-03-24T00:04:51.450 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:04:50.229+0000 7f9f8dc00700 1 -- 172.21.15.118:0/2986369374 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9f88107bf0 con 0x7f9f881069f0 2024-03-24T00:04:51.450 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:04:50.229+0000 7f9f8dc00700 1 --2- 172.21.15.118:0/2986369374 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f9f881069f0 0x7f9f88106e10 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f9f7c0098a0 tx=0x7f9f7c009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=58399485f7a6e1d4 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:04:51.450 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:04:50.229+0000 7f9f87e00700 1 -- 172.21.15.118:0/2986369374 <== mon.0 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f9f7c004160 con 0x7f9f881069f0 2024-03-24T00:04:51.450 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:04:50.229+0000 7f9f87e00700 1 -- 172.21.15.118:0/2986369374 <== mon.0 v2:172.21.15.118:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f9f7c004300 con 0x7f9f881069f0 2024-03-24T00:04:51.450 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:04:50.229+0000 7f9f87e00700 1 -- 172.21.15.118:0/2986369374 <== mon.0 v2:172.21.15.118:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f9f7c011b50 con 0x7f9f881069f0 2024-03-24T00:04:51.450 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:04:50.230+0000 7f9f90600700 1 -- 172.21.15.118:0/2986369374 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f9f881069f0 msgr2=0x7f9f88106e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:04:51.450 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:04:50.230+0000 7f9f90600700 1 --2- 172.21.15.118:0/2986369374 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f9f881069f0 0x7f9f88106e10 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f9f7c0098a0 tx=0x7f9f7c009bb0 comp rx=0 tx=0).stop 2024-03-24T00:04:51.450 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:04:50.230+0000 7f9f90600700 1 -- 172.21.15.118:0/2986369374 shutdown_connections 2024-03-24T00:04:51.450 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:04:50.230+0000 7f9f90600700 1 --2- 172.21.15.118:0/2986369374 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f9f881069f0 0x7f9f88106e10 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:04:51.451 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:04:50.230+0000 7f9f90600700 1 -- 172.21.15.118:0/2986369374 >> 172.21.15.118:0/2986369374 conn(0x7f9f88101fb0 msgr2=0x7f9f881043d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:04:51.451 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:04:50.231+0000 7f9f90600700 1 -- 172.21.15.118:0/2986369374 shutdown_connections 2024-03-24T00:04:51.451 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:04:50.231+0000 7f9f90600700 1 -- 172.21.15.118:0/2986369374 wait complete. 2024-03-24T00:04:51.451 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:04:50.232+0000 7f9f90600700 1 Processor -- start 2024-03-24T00:04:51.451 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:04:50.232+0000 7f9f90600700 1 -- start start 2024-03-24T00:04:51.451 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:04:50.232+0000 7f9f90600700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f9f881069f0 0x7f9f8819dd60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:04:51.451 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:04:50.232+0000 7f9f90600700 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9f8810afa0 con 0x7f9f881069f0 2024-03-24T00:04:51.451 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:04:50.232+0000 7f9f8dc00700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f9f881069f0 0x7f9f8819dd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:04:51.451 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:04:50.232+0000 7f9f8dc00700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f9f881069f0 0x7f9f8819dd60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.118:3300/0 says I am v2:172.21.15.118:51478/0 (socket says 172.21.15.118:51478) 2024-03-24T00:04:51.451 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:04:50.232+0000 7f9f8dc00700 1 -- 172.21.15.118:0/3127112388 learned_addr learned my addr 172.21.15.118:0/3127112388 (peer_addr_for_me v2:172.21.15.118:0/0) 2024-03-24T00:04:51.451 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:04:50.233+0000 7f9f8dc00700 1 -- 172.21.15.118:0/3127112388 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9f8819e2a0 con 0x7f9f881069f0 2024-03-24T00:04:51.451 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:04:50.233+0000 7f9f8dc00700 1 --2- 172.21.15.118:0/3127112388 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f9f881069f0 0x7f9f8819dd60 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f9f7c011850 tx=0x7f9f7c011c10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:04:51.451 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:04:50.233+0000 7f9f86000700 1 -- 172.21.15.118:0/3127112388 <== mon.0 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f9f7c003af0 con 0x7f9f881069f0 2024-03-24T00:04:51.451 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:04:50.233+0000 7f9f86000700 1 -- 172.21.15.118:0/3127112388 <== mon.0 v2:172.21.15.118:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f9f7c020070 con 0x7f9f881069f0 2024-03-24T00:04:51.452 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:04:50.233+0000 7f9f90600700 1 -- 172.21.15.118:0/3127112388 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9f8819e530 con 0x7f9f881069f0 2024-03-24T00:04:51.452 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:04:50.233+0000 7f9f86000700 1 -- 172.21.15.118:0/3127112388 <== mon.0 v2:172.21.15.118:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f9f7c0195a0 con 0x7f9f881069f0 2024-03-24T00:04:51.452 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:04:50.233+0000 7f9f90600700 1 -- 172.21.15.118:0/3127112388 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9f881a1220 con 0x7f9f881069f0 2024-03-24T00:04:51.452 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:04:50.235+0000 7f9f86000700 1 -- 172.21.15.118:0/3127112388 <== mon.0 v2:172.21.15.118:3300/0 4 ==== mgrmap(e 8) v1 ==== 46106+0+0 (secure 0 0 0) 0x7f9f7c019870 con 0x7f9f881069f0 2024-03-24T00:04:51.452 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:04:50.235+0000 7f9f90600700 1 -- 172.21.15.118:0/3127112388 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9f880637c0 con 0x7f9f881069f0 2024-03-24T00:04:51.452 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:04:50.235+0000 7f9f86000700 1 --2- 172.21.15.118:0/3127112388 >> [v2:172.21.15.118:6800/389249272,v1:172.21.15.118:6801/389249272] conn(0x7f9f70038f70 0x7f9f7003b430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:04:51.452 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:04:50.235+0000 7f9f86000700 1 -- 172.21.15.118:0/3127112388 <== mon.0 v2:172.21.15.118:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 940+0+0 (secure 0 0 0) 0x7f9f7c011e00 con 0x7f9f881069f0 2024-03-24T00:04:51.452 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:04:50.235+0000 7f9f8d200700 1 --2- 172.21.15.118:0/3127112388 >> [v2:172.21.15.118:6800/389249272,v1:172.21.15.118:6801/389249272] conn(0x7f9f70038f70 0x7f9f7003b430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:04:51.452 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:04:50.236+0000 7f9f8d200700 1 --2- 172.21.15.118:0/3127112388 >> [v2:172.21.15.118:6800/389249272,v1:172.21.15.118:6801/389249272] conn(0x7f9f70038f70 0x7f9f7003b430 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f9f74006fd0 tx=0x7f9f74006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:04:51.452 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:04:50.241+0000 7f9f86000700 1 -- 172.21.15.118:0/3127112388 <== mon.0 v2:172.21.15.118:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f9f7c003c90 con 0x7f9f881069f0 2024-03-24T00:04:51.452 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:04:50.484+0000 7f9f90600700 1 -- 172.21.15.118:0/3127112388 --> [v2:172.21.15.118:6800/389249272,v1:172.21.15.118:6801/389249272] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "mgr", "unmanaged": true, "target": ["mon-mgr", ""]}) v1 -- 0x7f9f880008d0 con 0x7f9f70038f70 2024-03-24T00:04:51.452 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:04:50.494+0000 7f9f86000700 1 -- 172.21.15.118:0/3127112388 <== mgr.14120 v2:172.21.15.118:6800/389249272 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+24 (secure 0 0 0) 0x7f9f880008d0 con 0x7f9f70038f70 2024-03-24T00:04:51.453 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:04:50.496+0000 7f9f90600700 1 -- 172.21.15.118:0/3127112388 >> [v2:172.21.15.118:6800/389249272,v1:172.21.15.118:6801/389249272] conn(0x7f9f70038f70 msgr2=0x7f9f7003b430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:04:51.453 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:04:50.496+0000 7f9f90600700 1 --2- 172.21.15.118:0/3127112388 >> [v2:172.21.15.118:6800/389249272,v1:172.21.15.118:6801/389249272] conn(0x7f9f70038f70 0x7f9f7003b430 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f9f74006fd0 tx=0x7f9f74006e40 comp rx=0 tx=0).stop 2024-03-24T00:04:51.453 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:04:50.496+0000 7f9f90600700 1 -- 172.21.15.118:0/3127112388 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f9f881069f0 msgr2=0x7f9f8819dd60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:04:51.453 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:04:50.496+0000 7f9f90600700 1 --2- 172.21.15.118:0/3127112388 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f9f881069f0 0x7f9f8819dd60 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f9f7c011850 tx=0x7f9f7c011c10 comp rx=0 tx=0).stop 2024-03-24T00:04:51.453 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:04:50.497+0000 7f9f90600700 1 -- 172.21.15.118:0/3127112388 shutdown_connections 2024-03-24T00:04:51.453 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:04:50.497+0000 7f9f90600700 1 --2- 172.21.15.118:0/3127112388 >> [v2:172.21.15.118:6800/389249272,v1:172.21.15.118:6801/389249272] conn(0x7f9f70038f70 0x7f9f7003b430 unknown :-1 s=CLOSED pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:04:51.453 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:04:50.497+0000 7f9f90600700 1 --2- 172.21.15.118:0/3127112388 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f9f881069f0 0x7f9f8819dd60 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:04:51.453 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:04:50.497+0000 7f9f90600700 1 -- 172.21.15.118:0/3127112388 >> 172.21.15.118:0/3127112388 conn(0x7f9f88101fb0 msgr2=0x7f9f88102f30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:04:51.453 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:04:50.497+0000 7f9f90600700 1 -- 172.21.15.118:0/3127112388 shutdown_connections 2024-03-24T00:04:51.453 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:04:50.497+0000 7f9f90600700 1 -- 172.21.15.118:0/3127112388 wait complete. 2024-03-24T00:04:51.695 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:04:51 smithi118 ceph-mon[31535]: from='client.14144 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "unmanaged": true, "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T00:04:51.695 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:04:51 smithi118 ceph-mon[31535]: Saving service mgr spec with placement count:2 2024-03-24T00:04:51.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:04:51 smithi118 ceph-mon[31535]: from='mgr.14120 172.21.15.118:0/3476110238' entity='mgr.x' 2024-03-24T00:04:51.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:04:51 smithi118 ceph-mon[31535]: from='mgr.14120 172.21.15.118:0/3476110238' entity='mgr.x' 2024-03-24T00:04:52.623 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:04:52 smithi118 ceph-mon[31535]: pgmap v3: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T00:04:52.624 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:04:52 smithi118 ceph-mon[31535]: from='mgr.14120 172.21.15.118:0/3476110238' entity='mgr.x' 2024-03-24T00:04:52.624 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:04:52 smithi118 ceph-mon[31535]: from='client.? 172.21.15.118:0/1784424737' entity='client.admin' 2024-03-24T00:04:52.920 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:04:52.054+0000 7ff979600700 1 Processor -- start 2024-03-24T00:04:52.921 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:04:52.054+0000 7ff979600700 1 -- start start 2024-03-24T00:04:52.921 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:04:52.054+0000 7ff979600700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7ff97407cd90 0x7ff97407b1f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:04:52.921 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:04:52.054+0000 7ff979600700 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff97407b7c0 con 0x7ff97407cd90 2024-03-24T00:04:52.921 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:04:52.055+0000 7ff972a00700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7ff97407cd90 0x7ff97407b1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:04:52.922 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:04:52.055+0000 7ff972a00700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7ff97407cd90 0x7ff97407b1f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.118:3300/0 says I am v2:172.21.15.118:51482/0 (socket says 172.21.15.118:51482) 2024-03-24T00:04:52.922 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:04:52.055+0000 7ff972a00700 1 -- 172.21.15.118:0/267153262 learned_addr learned my addr 172.21.15.118:0/267153262 (peer_addr_for_me v2:172.21.15.118:0/0) 2024-03-24T00:04:52.922 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:04:52.055+0000 7ff972a00700 1 -- 172.21.15.118:0/267153262 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff97407b940 con 0x7ff97407cd90 2024-03-24T00:04:52.923 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:04:52.056+0000 7ff972a00700 1 --2- 172.21.15.118:0/267153262 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7ff97407cd90 0x7ff97407b1f0 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7ff9640098a0 tx=0x7ff964009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=92b3502129d86e8f server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:04:52.923 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:04:52.056+0000 7ff971600700 1 -- 172.21.15.118:0/267153262 <== mon.0 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7ff964004160 con 0x7ff97407cd90 2024-03-24T00:04:52.923 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:04:52.056+0000 7ff971600700 1 -- 172.21.15.118:0/267153262 <== mon.0 v2:172.21.15.118:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7ff964004300 con 0x7ff97407cd90 2024-03-24T00:04:52.923 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:04:52.056+0000 7ff971600700 1 -- 172.21.15.118:0/267153262 <== mon.0 v2:172.21.15.118:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7ff964011b20 con 0x7ff97407cd90 2024-03-24T00:04:52.923 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:04:52.057+0000 7ff979600700 1 -- 172.21.15.118:0/267153262 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7ff97407cd90 msgr2=0x7ff97407b1f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:04:52.924 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:04:52.057+0000 7ff979600700 1 --2- 172.21.15.118:0/267153262 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7ff97407cd90 0x7ff97407b1f0 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7ff9640098a0 tx=0x7ff964009bb0 comp rx=0 tx=0).stop 2024-03-24T00:04:52.924 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:04:52.057+0000 7ff979600700 1 -- 172.21.15.118:0/267153262 shutdown_connections 2024-03-24T00:04:52.926 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:04:52.057+0000 7ff979600700 1 --2- 172.21.15.118:0/267153262 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7ff97407cd90 0x7ff97407b1f0 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:04:52.926 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:04:52.057+0000 7ff979600700 1 -- 172.21.15.118:0/267153262 >> 172.21.15.118:0/267153262 conn(0x7ff974103070 msgr2=0x7ff9741054d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:04:52.926 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:04:52.057+0000 7ff979600700 1 -- 172.21.15.118:0/267153262 shutdown_connections 2024-03-24T00:04:52.926 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:04:52.057+0000 7ff979600700 1 -- 172.21.15.118:0/267153262 wait complete. 2024-03-24T00:04:52.926 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:04:52.058+0000 7ff979600700 1 Processor -- start 2024-03-24T00:04:52.926 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:04:52.058+0000 7ff979600700 1 -- start start 2024-03-24T00:04:52.926 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:04:52.059+0000 7ff979600700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7ff97407cd90 0x7ff97419dc10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:04:52.926 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:04:52.059+0000 7ff979600700 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff97410e170 con 0x7ff97407cd90 2024-03-24T00:04:52.926 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:04:52.059+0000 7ff972a00700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7ff97407cd90 0x7ff97419dc10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:04:52.926 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:04:52.059+0000 7ff972a00700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7ff97407cd90 0x7ff97419dc10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.118:3300/0 says I am v2:172.21.15.118:51498/0 (socket says 172.21.15.118:51498) 2024-03-24T00:04:52.926 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:04:52.059+0000 7ff972a00700 1 -- 172.21.15.118:0/1784424737 learned_addr learned my addr 172.21.15.118:0/1784424737 (peer_addr_for_me v2:172.21.15.118:0/0) 2024-03-24T00:04:52.926 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:04:52.060+0000 7ff972a00700 1 -- 172.21.15.118:0/1784424737 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff97419e150 con 0x7ff97407cd90 2024-03-24T00:04:52.927 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:04:52.060+0000 7ff972a00700 1 --2- 172.21.15.118:0/1784424737 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7ff97407cd90 0x7ff97419dc10 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7ff9640117e0 tx=0x7ff964011ba0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:04:52.927 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:04:52.060+0000 7ff96b400700 1 -- 172.21.15.118:0/1784424737 <== mon.0 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7ff964003ba0 con 0x7ff97407cd90 2024-03-24T00:04:52.927 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:04:52.060+0000 7ff96b400700 1 -- 172.21.15.118:0/1784424737 <== mon.0 v2:172.21.15.118:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7ff96401a910 con 0x7ff97407cd90 2024-03-24T00:04:52.927 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:04:52.060+0000 7ff979600700 1 -- 172.21.15.118:0/1784424737 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff97419e3e0 con 0x7ff97407cd90 2024-03-24T00:04:52.927 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:04:52.060+0000 7ff96b400700 1 -- 172.21.15.118:0/1784424737 <== mon.0 v2:172.21.15.118:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7ff9640248f0 con 0x7ff97407cd90 2024-03-24T00:04:52.929 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:04:52.060+0000 7ff979600700 1 -- 172.21.15.118:0/1784424737 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff97419e840 con 0x7ff97407cd90 2024-03-24T00:04:52.929 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:04:52.062+0000 7ff96b400700 1 -- 172.21.15.118:0/1784424737 <== mon.0 v2:172.21.15.118:3300/0 4 ==== mgrmap(e 8) v1 ==== 46106+0+0 (secure 0 0 0) 0x7ff964022020 con 0x7ff97407cd90 2024-03-24T00:04:52.929 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:04:52.062+0000 7ff979600700 1 -- 172.21.15.118:0/1784424737 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff9740637c0 con 0x7ff97407cd90 2024-03-24T00:04:52.929 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:04:52.062+0000 7ff96b400700 1 --2- 172.21.15.118:0/1784424737 >> [v2:172.21.15.118:6800/389249272,v1:172.21.15.118:6801/389249272] conn(0x7ff958038f70 0x7ff95803b430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:04:52.929 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:04:52.062+0000 7ff96b400700 1 -- 172.21.15.118:0/1784424737 <== mon.0 v2:172.21.15.118:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 940+0+0 (secure 0 0 0) 0x7ff96404d280 con 0x7ff97407cd90 2024-03-24T00:04:52.929 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:04:52.062+0000 7ff972000700 1 --2- 172.21.15.118:0/1784424737 >> [v2:172.21.15.118:6800/389249272,v1:172.21.15.118:6801/389249272] conn(0x7ff958038f70 0x7ff95803b430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:04:52.929 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:04:52.063+0000 7ff972000700 1 --2- 172.21.15.118:0/1784424737 >> [v2:172.21.15.118:6800/389249272,v1:172.21.15.118:6801/389249272] conn(0x7ff958038f70 0x7ff95803b430 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7ff960006fd0 tx=0x7ff960006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:04:52.929 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:04:52.068+0000 7ff96b400700 1 -- 172.21.15.118:0/1784424737 <== mon.0 v2:172.21.15.118:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7ff96401ab70 con 0x7ff97407cd90 2024-03-24T00:04:52.930 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:04:52.294+0000 7ff979600700 1 -- 172.21.15.118:0/1784424737 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_command([{prefix=config set, name=mgr/cephadm/container_init}] v 0) v1 -- 0x7ff9741a10d0 con 0x7ff97407cd90 2024-03-24T00:04:52.930 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:04:52.297+0000 7ff96b400700 1 -- 172.21.15.118:0/1784424737 <== mon.0 v2:172.21.15.118:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/cephadm/container_init}]=0 v7)=0 v7) v1 ==== 142+0+0 (secure 0 0 0) 0x7ff96402d3f0 con 0x7ff97407cd90 2024-03-24T00:04:52.930 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:04:52.300+0000 7ff979600700 1 -- 172.21.15.118:0/1784424737 >> [v2:172.21.15.118:6800/389249272,v1:172.21.15.118:6801/389249272] conn(0x7ff958038f70 msgr2=0x7ff95803b430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:04:52.930 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:04:52.300+0000 7ff979600700 1 --2- 172.21.15.118:0/1784424737 >> [v2:172.21.15.118:6800/389249272,v1:172.21.15.118:6801/389249272] conn(0x7ff958038f70 0x7ff95803b430 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7ff960006fd0 tx=0x7ff960006e40 comp rx=0 tx=0).stop 2024-03-24T00:04:52.930 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:04:52.300+0000 7ff979600700 1 -- 172.21.15.118:0/1784424737 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7ff97407cd90 msgr2=0x7ff97419dc10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:04:52.930 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:04:52.300+0000 7ff979600700 1 --2- 172.21.15.118:0/1784424737 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7ff97407cd90 0x7ff97419dc10 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7ff9640117e0 tx=0x7ff964011ba0 comp rx=0 tx=0).stop 2024-03-24T00:04:52.930 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:04:52.300+0000 7ff979600700 1 -- 172.21.15.118:0/1784424737 shutdown_connections 2024-03-24T00:04:52.930 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:04:52.300+0000 7ff979600700 1 --2- 172.21.15.118:0/1784424737 >> [v2:172.21.15.118:6800/389249272,v1:172.21.15.118:6801/389249272] conn(0x7ff958038f70 0x7ff95803b430 unknown :-1 s=CLOSED pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:04:52.930 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:04:52.300+0000 7ff979600700 1 --2- 172.21.15.118:0/1784424737 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7ff97407cd90 0x7ff97419dc10 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:04:52.930 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:04:52.300+0000 7ff979600700 1 -- 172.21.15.118:0/1784424737 >> 172.21.15.118:0/1784424737 conn(0x7ff974103070 msgr2=0x7ff974103b10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:04:52.930 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:04:52.300+0000 7ff979600700 1 -- 172.21.15.118:0/1784424737 shutdown_connections 2024-03-24T00:04:52.930 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:04:52.300+0000 7ff979600700 1 -- 172.21.15.118:0/1784424737 wait complete. 2024-03-24T00:04:54.810 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:04:54 smithi118 ceph-mon[31535]: pgmap v4: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T00:04:54.810 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:04:54 smithi118 ceph-mon[31535]: from='mgr.14120 172.21.15.118:0/3476110238' entity='mgr.x' 2024-03-24T00:04:54.810 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:04:54 smithi118 ceph-mon[31535]: from='client.? 172.21.15.118:0/2393769964' entity='client.admin' 2024-03-24T00:04:55.156 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:04:54.243+0000 7fc81ac00700 1 Processor -- start 2024-03-24T00:04:55.156 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:04:54.243+0000 7fc81ac00700 1 -- start start 2024-03-24T00:04:55.156 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:04:54.243+0000 7fc81ac00700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fc814107b40 0x7fc814107f60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:04:55.156 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:04:54.243+0000 7fc81ac00700 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc814108530 con 0x7fc814107b40 2024-03-24T00:04:55.156 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:04:54.244+0000 7fc813e00700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fc814107b40 0x7fc814107f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:04:55.156 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:04:54.244+0000 7fc813e00700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fc814107b40 0x7fc814107f60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.118:3300/0 says I am v2:172.21.15.118:49758/0 (socket says 172.21.15.118:49758) 2024-03-24T00:04:55.156 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:04:54.244+0000 7fc813e00700 1 -- 172.21.15.118:0/3828743863 learned_addr learned my addr 172.21.15.118:0/3828743863 (peer_addr_for_me v2:172.21.15.118:0/0) 2024-03-24T00:04:55.156 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:04:54.244+0000 7fc813e00700 1 -- 172.21.15.118:0/3828743863 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc814108d40 con 0x7fc814107b40 2024-03-24T00:04:55.156 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:04:54.244+0000 7fc813e00700 1 --2- 172.21.15.118:0/3828743863 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fc814107b40 0x7fc814107f60 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7fc8040098a0 tx=0x7fc804009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=47e2fe856fdd336d server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:04:55.156 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:04:54.245+0000 7fc812a00700 1 -- 172.21.15.118:0/3828743863 <== mon.0 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fc804004160 con 0x7fc814107b40 2024-03-24T00:04:55.156 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:04:54.245+0000 7fc812a00700 1 -- 172.21.15.118:0/3828743863 <== mon.0 v2:172.21.15.118:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fc804004300 con 0x7fc814107b40 2024-03-24T00:04:55.157 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:04:54.245+0000 7fc812a00700 1 -- 172.21.15.118:0/3828743863 <== mon.0 v2:172.21.15.118:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fc804011b20 con 0x7fc814107b40 2024-03-24T00:04:55.157 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:04:54.245+0000 7fc81ac00700 1 -- 172.21.15.118:0/3828743863 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fc814107b40 msgr2=0x7fc814107f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:04:55.157 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:04:54.245+0000 7fc81ac00700 1 --2- 172.21.15.118:0/3828743863 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fc814107b40 0x7fc814107f60 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7fc8040098a0 tx=0x7fc804009bb0 comp rx=0 tx=0).stop 2024-03-24T00:04:55.157 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:04:54.246+0000 7fc81ac00700 1 -- 172.21.15.118:0/3828743863 shutdown_connections 2024-03-24T00:04:55.157 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:04:54.246+0000 7fc81ac00700 1 --2- 172.21.15.118:0/3828743863 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fc814107b40 0x7fc814107f60 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:04:55.157 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:04:54.246+0000 7fc81ac00700 1 -- 172.21.15.118:0/3828743863 >> 172.21.15.118:0/3828743863 conn(0x7fc8141030c0 msgr2=0x7fc814105520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:04:55.157 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:04:54.246+0000 7fc81ac00700 1 -- 172.21.15.118:0/3828743863 shutdown_connections 2024-03-24T00:04:55.157 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:04:54.246+0000 7fc81ac00700 1 -- 172.21.15.118:0/3828743863 wait complete. 2024-03-24T00:04:55.157 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:04:54.247+0000 7fc81ac00700 1 Processor -- start 2024-03-24T00:04:55.157 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:04:54.247+0000 7fc81ac00700 1 -- start start 2024-03-24T00:04:55.157 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:04:54.247+0000 7fc81ac00700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fc814107b40 0x7fc81419ddb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:04:55.157 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:04:54.247+0000 7fc81ac00700 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc81410c0f0 con 0x7fc814107b40 2024-03-24T00:04:55.157 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:04:54.248+0000 7fc813e00700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fc814107b40 0x7fc81419ddb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:04:55.158 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:04:54.248+0000 7fc813e00700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fc814107b40 0x7fc81419ddb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.118:3300/0 says I am v2:172.21.15.118:49764/0 (socket says 172.21.15.118:49764) 2024-03-24T00:04:55.158 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:04:54.248+0000 7fc813e00700 1 -- 172.21.15.118:0/2393769964 learned_addr learned my addr 172.21.15.118:0/2393769964 (peer_addr_for_me v2:172.21.15.118:0/0) 2024-03-24T00:04:55.158 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:04:54.248+0000 7fc813e00700 1 -- 172.21.15.118:0/2393769964 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc81419e2f0 con 0x7fc814107b40 2024-03-24T00:04:55.158 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:04:54.249+0000 7fc813e00700 1 --2- 172.21.15.118:0/2393769964 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fc814107b40 0x7fc81419ddb0 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7fc804000c00 tx=0x7fc804011c10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:04:55.158 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:04:54.249+0000 7fc810c00700 1 -- 172.21.15.118:0/2393769964 <== mon.0 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fc804003b60 con 0x7fc814107b40 2024-03-24T00:04:55.158 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:04:54.249+0000 7fc810c00700 1 -- 172.21.15.118:0/2393769964 <== mon.0 v2:172.21.15.118:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fc80401a910 con 0x7fc814107b40 2024-03-24T00:04:55.158 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:04:54.249+0000 7fc81ac00700 1 -- 172.21.15.118:0/2393769964 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc81419e580 con 0x7fc814107b40 2024-03-24T00:04:55.158 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:04:54.249+0000 7fc810c00700 1 -- 172.21.15.118:0/2393769964 <== mon.0 v2:172.21.15.118:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fc804024930 con 0x7fc814107b40 2024-03-24T00:04:55.158 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:04:54.249+0000 7fc81ac00700 1 -- 172.21.15.118:0/2393769964 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc8141a1230 con 0x7fc814107b40 2024-03-24T00:04:55.158 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:04:54.250+0000 7fc81ac00700 1 -- 172.21.15.118:0/2393769964 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc8140637c0 con 0x7fc814107b40 2024-03-24T00:04:55.158 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:04:54.251+0000 7fc810c00700 1 -- 172.21.15.118:0/2393769964 <== mon.0 v2:172.21.15.118:3300/0 4 ==== mgrmap(e 8) v1 ==== 46106+0+0 (secure 0 0 0) 0x7fc804022020 con 0x7fc814107b40 2024-03-24T00:04:55.158 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:04:54.251+0000 7fc810c00700 1 --2- 172.21.15.118:0/2393769964 >> [v2:172.21.15.118:6800/389249272,v1:172.21.15.118:6801/389249272] conn(0x7fc7f8038f70 0x7fc7f803b430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:04:55.158 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:04:54.251+0000 7fc810c00700 1 -- 172.21.15.118:0/2393769964 <== mon.0 v2:172.21.15.118:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 940+0+0 (secure 0 0 0) 0x7fc80404d080 con 0x7fc814107b40 2024-03-24T00:04:55.159 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:04:54.252+0000 7fc813400700 1 --2- 172.21.15.118:0/2393769964 >> [v2:172.21.15.118:6800/389249272,v1:172.21.15.118:6801/389249272] conn(0x7fc7f8038f70 0x7fc7f803b430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:04:55.159 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:04:54.253+0000 7fc813400700 1 --2- 172.21.15.118:0/2393769964 >> [v2:172.21.15.118:6800/389249272,v1:172.21.15.118:6801/389249272] conn(0x7fc7f8038f70 0x7fc7f803b430 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7fc800006fd0 tx=0x7fc800006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:04:55.159 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:04:54.257+0000 7fc810c00700 1 -- 172.21.15.118:0/2393769964 <== mon.0 v2:172.21.15.118:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7fc80404c9d0 con 0x7fc814107b40 2024-03-24T00:04:55.159 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:04:54.481+0000 7fc81ac00700 1 -- 172.21.15.118:0/2393769964 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_command([{prefix=config set, name=mgr/dashboard/ssl_server_port}] v 0) v1 -- 0x7fc8141a1400 con 0x7fc814107b40 2024-03-24T00:04:55.159 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:04:54.487+0000 7fc810c00700 1 -- 172.21.15.118:0/2393769964 <== mon.0 v2:172.21.15.118:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/dashboard/ssl_server_port}]=0 v8)=0 v8) v1 ==== 130+0+0 (secure 0 0 0) 0x7fc80402d3f0 con 0x7fc814107b40 2024-03-24T00:04:55.159 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:04:54.490+0000 7fc81ac00700 1 -- 172.21.15.118:0/2393769964 >> [v2:172.21.15.118:6800/389249272,v1:172.21.15.118:6801/389249272] conn(0x7fc7f8038f70 msgr2=0x7fc7f803b430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:04:55.159 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:04:54.490+0000 7fc81ac00700 1 --2- 172.21.15.118:0/2393769964 >> [v2:172.21.15.118:6800/389249272,v1:172.21.15.118:6801/389249272] conn(0x7fc7f8038f70 0x7fc7f803b430 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7fc800006fd0 tx=0x7fc800006e40 comp rx=0 tx=0).stop 2024-03-24T00:04:55.159 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:04:54.490+0000 7fc81ac00700 1 -- 172.21.15.118:0/2393769964 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fc814107b40 msgr2=0x7fc81419ddb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:04:55.159 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:04:54.490+0000 7fc81ac00700 1 --2- 172.21.15.118:0/2393769964 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fc814107b40 0x7fc81419ddb0 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7fc804000c00 tx=0x7fc804011c10 comp rx=0 tx=0).stop 2024-03-24T00:04:55.159 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:04:54.491+0000 7fc81ac00700 1 -- 172.21.15.118:0/2393769964 shutdown_connections 2024-03-24T00:04:55.159 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:04:54.491+0000 7fc81ac00700 1 --2- 172.21.15.118:0/2393769964 >> [v2:172.21.15.118:6800/389249272,v1:172.21.15.118:6801/389249272] conn(0x7fc7f8038f70 0x7fc7f803b430 unknown :-1 s=CLOSED pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:04:55.159 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:04:54.491+0000 7fc81ac00700 1 --2- 172.21.15.118:0/2393769964 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fc814107b40 0x7fc81419ddb0 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:04:55.159 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:04:54.491+0000 7fc81ac00700 1 -- 172.21.15.118:0/2393769964 >> 172.21.15.118:0/2393769964 conn(0x7fc8141030c0 msgr2=0x7fc814103ba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:04:55.159 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:04:54.491+0000 7fc81ac00700 1 -- 172.21.15.118:0/2393769964 shutdown_connections 2024-03-24T00:04:55.159 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:04:54.491+0000 7fc81ac00700 1 -- 172.21.15.118:0/2393769964 wait complete. 2024-03-24T00:04:55.160 INFO:teuthology.orchestra.run.smithi118.stdout:Enabling the dashboard module... 2024-03-24T00:04:56.813 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:04:56 smithi118 ceph-mon[31535]: pgmap v5: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T00:04:56.813 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:04:56 smithi118 ceph-mon[31535]: from='client.? 172.21.15.118:0/1735641409' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "dashboard"}]: dispatch 2024-03-24T00:04:56.813 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:04:56 smithi118 ceph-mon[31535]: from='mgr.14120 172.21.15.118:0/3476110238' entity='mgr.x' 2024-03-24T00:04:56.813 INFO:journalctl@ceph.mgr.x.smithi118.stdout:Mar 24 00:04:56 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mgr-x[31762]: ignoring --setuser ceph since I am not root 2024-03-24T00:04:56.813 INFO:journalctl@ceph.mgr.x.smithi118.stdout:Mar 24 00:04:56 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mgr-x[31762]: ignoring --setgroup ceph since I am not root 2024-03-24T00:04:57.190 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:04:55.828+0000 7f0b29000700 1 Processor -- start 2024-03-24T00:04:57.195 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:04:55.828+0000 7f0b29000700 1 -- start start 2024-03-24T00:04:57.196 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:04:55.828+0000 7f0b29000700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f0b241063b0 0x7f0b241067d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:04:57.196 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:04:55.828+0000 7f0b29000700 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0b240758a0 con 0x7f0b241063b0 2024-03-24T00:04:57.196 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:04:55.829+0000 7f0b21e00700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f0b241063b0 0x7f0b241067d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:04:57.196 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:04:55.829+0000 7f0b21e00700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f0b241063b0 0x7f0b241067d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.118:3300/0 says I am v2:172.21.15.118:49780/0 (socket says 172.21.15.118:49780) 2024-03-24T00:04:57.196 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:04:55.829+0000 7f0b21e00700 1 -- 172.21.15.118:0/4047050393 learned_addr learned my addr 172.21.15.118:0/4047050393 (peer_addr_for_me v2:172.21.15.118:0/0) 2024-03-24T00:04:57.196 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:04:55.829+0000 7f0b21e00700 1 -- 172.21.15.118:0/4047050393 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0b24106d10 con 0x7f0b241063b0 2024-03-24T00:04:57.196 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:04:55.830+0000 7f0b21e00700 1 --2- 172.21.15.118:0/4047050393 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f0b241063b0 0x7f0b241067d0 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f0b140098a0 tx=0x7f0b14009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e9010d7f6439e67f server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:04:57.196 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:04:55.830+0000 7f0b20a00700 1 -- 172.21.15.118:0/4047050393 <== mon.0 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f0b14004160 con 0x7f0b241063b0 2024-03-24T00:04:57.196 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:04:55.830+0000 7f0b20a00700 1 -- 172.21.15.118:0/4047050393 <== mon.0 v2:172.21.15.118:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f0b14004300 con 0x7f0b241063b0 2024-03-24T00:04:57.196 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:04:55.830+0000 7f0b20a00700 1 -- 172.21.15.118:0/4047050393 <== mon.0 v2:172.21.15.118:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f0b14011b20 con 0x7f0b241063b0 2024-03-24T00:04:57.196 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:04:55.831+0000 7f0b29000700 1 -- 172.21.15.118:0/4047050393 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f0b241063b0 msgr2=0x7f0b241067d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:04:57.196 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:04:55.831+0000 7f0b29000700 1 --2- 172.21.15.118:0/4047050393 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f0b241063b0 0x7f0b241067d0 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f0b140098a0 tx=0x7f0b14009bb0 comp rx=0 tx=0).stop 2024-03-24T00:04:57.196 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:04:55.831+0000 7f0b29000700 1 -- 172.21.15.118:0/4047050393 shutdown_connections 2024-03-24T00:04:57.196 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:04:55.831+0000 7f0b29000700 1 --2- 172.21.15.118:0/4047050393 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f0b241063b0 0x7f0b241067d0 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:04:57.197 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:04:55.831+0000 7f0b29000700 1 -- 172.21.15.118:0/4047050393 >> 172.21.15.118:0/4047050393 conn(0x7f0b24101fb0 msgr2=0x7f0b241043d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:04:57.197 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:04:55.831+0000 7f0b29000700 1 -- 172.21.15.118:0/4047050393 shutdown_connections 2024-03-24T00:04:57.197 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:04:55.831+0000 7f0b29000700 1 -- 172.21.15.118:0/4047050393 wait complete. 2024-03-24T00:04:57.197 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:04:55.833+0000 7f0b29000700 1 Processor -- start 2024-03-24T00:04:57.197 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:04:55.833+0000 7f0b29000700 1 -- start start 2024-03-24T00:04:57.197 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:04:55.833+0000 7f0b29000700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f0b241063b0 0x7f0b2419dc50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:04:57.197 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:04:55.833+0000 7f0b29000700 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0b24107de0 con 0x7f0b241063b0 2024-03-24T00:04:57.197 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:04:55.833+0000 7f0b21e00700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f0b241063b0 0x7f0b2419dc50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:04:57.197 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:04:55.833+0000 7f0b21e00700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f0b241063b0 0x7f0b2419dc50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.118:3300/0 says I am v2:172.21.15.118:49782/0 (socket says 172.21.15.118:49782) 2024-03-24T00:04:57.197 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:04:55.833+0000 7f0b21e00700 1 -- 172.21.15.118:0/1735641409 learned_addr learned my addr 172.21.15.118:0/1735641409 (peer_addr_for_me v2:172.21.15.118:0/0) 2024-03-24T00:04:57.197 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:04:55.834+0000 7f0b21e00700 1 -- 172.21.15.118:0/1735641409 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0b2419e190 con 0x7f0b241063b0 2024-03-24T00:04:57.197 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:04:55.834+0000 7f0b21e00700 1 --2- 172.21.15.118:0/1735641409 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f0b241063b0 0x7f0b2419dc50 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f0b140116a0 tx=0x7f0b14011ba0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:04:57.197 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:04:55.834+0000 7f0b1aa00700 1 -- 172.21.15.118:0/1735641409 <== mon.0 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f0b14003a40 con 0x7f0b241063b0 2024-03-24T00:04:57.197 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:04:55.834+0000 7f0b1aa00700 1 -- 172.21.15.118:0/1735641409 <== mon.0 v2:172.21.15.118:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f0b14003be0 con 0x7f0b241063b0 2024-03-24T00:04:57.198 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:04:55.834+0000 7f0b29000700 1 -- 172.21.15.118:0/1735641409 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0b2419e420 con 0x7f0b241063b0 2024-03-24T00:04:57.198 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:04:55.834+0000 7f0b1aa00700 1 -- 172.21.15.118:0/1735641409 <== mon.0 v2:172.21.15.118:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f0b1401a750 con 0x7f0b241063b0 2024-03-24T00:04:57.198 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:04:55.835+0000 7f0b29000700 1 -- 172.21.15.118:0/1735641409 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0b2419e880 con 0x7f0b241063b0 2024-03-24T00:04:57.198 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:04:55.836+0000 7f0b1aa00700 1 -- 172.21.15.118:0/1735641409 <== mon.0 v2:172.21.15.118:3300/0 4 ==== mgrmap(e 8) v1 ==== 46106+0+0 (secure 0 0 0) 0x7f0b1401aa70 con 0x7f0b241063b0 2024-03-24T00:04:57.198 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:04:55.836+0000 7f0b29000700 1 -- 172.21.15.118:0/1735641409 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0b240637c0 con 0x7f0b241063b0 2024-03-24T00:04:57.198 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:04:55.836+0000 7f0b1aa00700 1 --2- 172.21.15.118:0/1735641409 >> [v2:172.21.15.118:6800/389249272,v1:172.21.15.118:6801/389249272] conn(0x7f0b08038f20 0x7f0b0803b3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:04:57.198 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:04:55.836+0000 7f0b1aa00700 1 -- 172.21.15.118:0/1735641409 <== mon.0 v2:172.21.15.118:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 940+0+0 (secure 0 0 0) 0x7f0b1404d720 con 0x7f0b241063b0 2024-03-24T00:04:57.198 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:04:55.837+0000 7f0b21400700 1 --2- 172.21.15.118:0/1735641409 >> [v2:172.21.15.118:6800/389249272,v1:172.21.15.118:6801/389249272] conn(0x7f0b08038f20 0x7f0b0803b3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:04:57.198 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:04:55.837+0000 7f0b21400700 1 --2- 172.21.15.118:0/1735641409 >> [v2:172.21.15.118:6800/389249272,v1:172.21.15.118:6801/389249272] conn(0x7f0b08038f20 0x7f0b0803b3e0 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f0b10006fd0 tx=0x7f0b10006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:04:57.198 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:04:55.842+0000 7f0b1aa00700 1 -- 172.21.15.118:0/1735641409 <== mon.0 v2:172.21.15.118:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f0b14011e20 con 0x7f0b241063b0 2024-03-24T00:04:57.198 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:04:56.140+0000 7f0b29000700 1 -- 172.21.15.118:0/1735641409 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_command({"prefix": "mgr module enable", "module": "dashboard"} v 0) v1 -- 0x7f0b24062da0 con 0x7f0b241063b0 2024-03-24T00:04:57.198 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:04:56.505+0000 7f0b1aa00700 1 -- 172.21.15.118:0/1735641409 <== mon.0 v2:172.21.15.118:3300/0 7 ==== mon_command_ack([{"prefix": "mgr module enable", "module": "dashboard"}]=0 v9) v1 ==== 88+0+0 (secure 0 0 0) 0x7f0b1402d910 con 0x7f0b241063b0 2024-03-24T00:04:57.199 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:04:56.508+0000 7f0b1aa00700 1 -- 172.21.15.118:0/1735641409 <== mon.0 v2:172.21.15.118:3300/0 8 ==== mgrmap(e 9) v1 ==== 46119+0+0 (secure 0 0 0) 0x7f0b1402d3f0 con 0x7f0b241063b0 2024-03-24T00:04:57.199 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:04:56.509+0000 7f0b29000700 1 -- 172.21.15.118:0/1735641409 >> [v2:172.21.15.118:6800/389249272,v1:172.21.15.118:6801/389249272] conn(0x7f0b08038f20 msgr2=0x7f0b0803b3e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:04:57.199 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:04:56.509+0000 7f0b29000700 1 --2- 172.21.15.118:0/1735641409 >> [v2:172.21.15.118:6800/389249272,v1:172.21.15.118:6801/389249272] conn(0x7f0b08038f20 0x7f0b0803b3e0 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f0b10006fd0 tx=0x7f0b10006e40 comp rx=0 tx=0).stop 2024-03-24T00:04:57.199 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:04:56.509+0000 7f0b29000700 1 -- 172.21.15.118:0/1735641409 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f0b241063b0 msgr2=0x7f0b2419dc50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:04:57.199 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:04:56.509+0000 7f0b29000700 1 --2- 172.21.15.118:0/1735641409 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f0b241063b0 0x7f0b2419dc50 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f0b140116a0 tx=0x7f0b14011ba0 comp rx=0 tx=0).stop 2024-03-24T00:04:57.199 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:04:56.510+0000 7f0b29000700 1 -- 172.21.15.118:0/1735641409 shutdown_connections 2024-03-24T00:04:57.199 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:04:56.510+0000 7f0b29000700 1 --2- 172.21.15.118:0/1735641409 >> [v2:172.21.15.118:6800/389249272,v1:172.21.15.118:6801/389249272] conn(0x7f0b08038f20 0x7f0b0803b3e0 unknown :-1 s=CLOSED pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:04:57.199 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:04:56.510+0000 7f0b29000700 1 --2- 172.21.15.118:0/1735641409 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f0b241063b0 0x7f0b2419dc50 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:04:57.199 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:04:56.510+0000 7f0b29000700 1 -- 172.21.15.118:0/1735641409 >> 172.21.15.118:0/1735641409 conn(0x7f0b24101fb0 msgr2=0x7f0b24194910 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:04:57.199 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:04:56.510+0000 7f0b29000700 1 -- 172.21.15.118:0/1735641409 shutdown_connections 2024-03-24T00:04:57.199 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:04:56.510+0000 7f0b29000700 1 -- 172.21.15.118:0/1735641409 wait complete. 2024-03-24T00:04:57.446 INFO:journalctl@ceph.mgr.x.smithi118.stdout:Mar 24 00:04:57 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mgr-x[31762]: 2024-03-24T00:04:57.383+0000 7f6357ded200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-24T00:04:57.805 INFO:journalctl@ceph.mgr.x.smithi118.stdout:Mar 24 00:04:57 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mgr-x[31762]: 2024-03-24T00:04:57.511+0000 7f6357ded200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-24T00:04:57.805 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:04:57 smithi118 ceph-mon[31535]: from='client.? 172.21.15.118:0/1735641409' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "dashboard"}]': finished 2024-03-24T00:04:57.806 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:04:57 smithi118 ceph-mon[31535]: mgrmap e9: x(active, since 25s) 2024-03-24T00:04:58.606 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:04:58 smithi118 ceph-mon[31535]: from='client.? 172.21.15.118:0/3199022714' entity='client.admin' cmd=[{"prefix": "mgr stat"}]: dispatch 2024-03-24T00:04:58.607 INFO:journalctl@ceph.mgr.x.smithi118.stdout:Mar 24 00:04:58 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mgr-x[31762]: 2024-03-24T00:04:58.340+0000 7f6357ded200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-24T00:04:58.607 INFO:journalctl@ceph.mgr.x.smithi118.stdout:Mar 24 00:04:58 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mgr-x[31762]: 2024-03-24T00:04:58.463+0000 7f6357ded200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-24T00:04:58.881 INFO:journalctl@ceph.mgr.x.smithi118.stdout:Mar 24 00:04:58 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mgr-x[31762]: 2024-03-24T00:04:58.640+0000 7f6357ded200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-24T00:04:58.881 INFO:journalctl@ceph.mgr.x.smithi118.stdout:Mar 24 00:04:58 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mgr-x[31762]: 2024-03-24T00:04:58.763+0000 7f6357ded200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-24T00:04:58.881 INFO:journalctl@ceph.mgr.x.smithi118.stdout:Mar 24 00:04:58 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mgr-x[31762]: 2024-03-24T00:04:58.881+0000 7f6357ded200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-24T00:04:59.476 INFO:journalctl@ceph.mgr.x.smithi118.stdout:Mar 24 00:04:59 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mgr-x[31762]: 2024-03-24T00:04:59.219+0000 7f6357ded200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-03-24T00:04:59.477 INFO:journalctl@ceph.mgr.x.smithi118.stdout:Mar 24 00:04:59 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mgr-x[31762]: 2024-03-24T00:04:59.360+0000 7f6357ded200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-24T00:04:59.477 INFO:journalctl@ceph.mgr.x.smithi118.stdout:Mar 24 00:04:59 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mgr-x[31762]: 2024-03-24T00:04:59.476+0000 7f6357ded200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-24T00:04:59.781 INFO:journalctl@ceph.mgr.x.smithi118.stdout:Mar 24 00:04:59 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mgr-x[31762]: 2024-03-24T00:04:59.599+0000 7f6357ded200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-24T00:04:59.800 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stdout { 2024-03-24T00:04:59.800 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stdout "epoch": 9, 2024-03-24T00:04:59.800 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stdout "available": true, 2024-03-24T00:04:59.800 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stdout "active_name": "x", 2024-03-24T00:04:59.800 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stdout "num_standby": 0 2024-03-24T00:04:59.800 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stdout } 2024-03-24T00:04:59.801 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:04:57.848+0000 7fb353000700 1 Processor -- start 2024-03-24T00:04:59.801 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:04:57.848+0000 7fb353000700 1 -- start start 2024-03-24T00:04:59.801 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:04:57.848+0000 7fb353000700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb34c1085c0 0x7fb34c10a9b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:04:59.801 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:04:57.848+0000 7fb353000700 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb34c0758a0 con 0x7fb34c1085c0 2024-03-24T00:04:59.801 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:04:57.849+0000 7fb34be00700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb34c1085c0 0x7fb34c10a9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:04:59.802 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:04:57.849+0000 7fb34be00700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb34c1085c0 0x7fb34c10a9b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.118:3300/0 says I am v2:172.21.15.118:49810/0 (socket says 172.21.15.118:49810) 2024-03-24T00:04:59.802 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:04:57.849+0000 7fb34be00700 1 -- 172.21.15.118:0/1717599528 learned_addr learned my addr 172.21.15.118:0/1717599528 (peer_addr_for_me v2:172.21.15.118:0/0) 2024-03-24T00:04:59.802 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:04:57.850+0000 7fb34be00700 1 -- 172.21.15.118:0/1717599528 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb34c10aef0 con 0x7fb34c1085c0 2024-03-24T00:04:59.802 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:04:57.850+0000 7fb34be00700 1 --2- 172.21.15.118:0/1717599528 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb34c1085c0 0x7fb34c10a9b0 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7fb33c0098a0 tx=0x7fb33c009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ff9b2f594c1a54b2 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:04:59.802 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:04:57.850+0000 7fb34aa00700 1 -- 172.21.15.118:0/1717599528 <== mon.0 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fb33c004160 con 0x7fb34c1085c0 2024-03-24T00:04:59.802 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:04:57.850+0000 7fb34aa00700 1 -- 172.21.15.118:0/1717599528 <== mon.0 v2:172.21.15.118:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fb33c004300 con 0x7fb34c1085c0 2024-03-24T00:04:59.803 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:04:57.850+0000 7fb34aa00700 1 -- 172.21.15.118:0/1717599528 <== mon.0 v2:172.21.15.118:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fb33c011b20 con 0x7fb34c1085c0 2024-03-24T00:04:59.803 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:04:57.851+0000 7fb353000700 1 -- 172.21.15.118:0/1717599528 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb34c1085c0 msgr2=0x7fb34c10a9b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:04:59.803 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:04:57.851+0000 7fb353000700 1 --2- 172.21.15.118:0/1717599528 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb34c1085c0 0x7fb34c10a9b0 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7fb33c0098a0 tx=0x7fb33c009bb0 comp rx=0 tx=0).stop 2024-03-24T00:04:59.803 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:04:57.851+0000 7fb353000700 1 -- 172.21.15.118:0/1717599528 shutdown_connections 2024-03-24T00:04:59.803 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:04:57.851+0000 7fb353000700 1 --2- 172.21.15.118:0/1717599528 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb34c1085c0 0x7fb34c10a9b0 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:04:59.804 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:04:57.851+0000 7fb353000700 1 -- 172.21.15.118:0/1717599528 >> 172.21.15.118:0/1717599528 conn(0x7fb34c101fb0 msgr2=0x7fb34c1043d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:04:59.804 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:04:57.851+0000 7fb353000700 1 -- 172.21.15.118:0/1717599528 shutdown_connections 2024-03-24T00:04:59.804 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:04:57.851+0000 7fb353000700 1 -- 172.21.15.118:0/1717599528 wait complete. 2024-03-24T00:04:59.804 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:04:57.852+0000 7fb353000700 1 Processor -- start 2024-03-24T00:04:59.804 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:04:57.852+0000 7fb353000700 1 -- start start 2024-03-24T00:04:59.804 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:04:57.853+0000 7fb353000700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb34c1085c0 0x7fb34c107e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:04:59.804 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:04:57.853+0000 7fb353000700 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb34c10bf00 con 0x7fb34c1085c0 2024-03-24T00:04:59.804 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:04:57.853+0000 7fb34be00700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb34c1085c0 0x7fb34c107e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:04:59.804 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:04:57.853+0000 7fb34be00700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb34c1085c0 0x7fb34c107e40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.118:3300/0 says I am v2:172.21.15.118:49812/0 (socket says 172.21.15.118:49812) 2024-03-24T00:04:59.804 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:04:57.853+0000 7fb34be00700 1 -- 172.21.15.118:0/3199022714 learned_addr learned my addr 172.21.15.118:0/3199022714 (peer_addr_for_me v2:172.21.15.118:0/0) 2024-03-24T00:04:59.805 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:04:57.854+0000 7fb34be00700 1 -- 172.21.15.118:0/3199022714 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb34c108380 con 0x7fb34c1085c0 2024-03-24T00:04:59.805 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:04:57.854+0000 7fb34be00700 1 --2- 172.21.15.118:0/3199022714 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb34c1085c0 0x7fb34c107e40 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7fb33c011b60 tx=0x7fb33c011c40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:04:59.805 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:04:57.854+0000 7fb348c00700 1 -- 172.21.15.118:0/3199022714 <== mon.0 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fb33c003b60 con 0x7fb34c1085c0 2024-03-24T00:04:59.805 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:04:57.854+0000 7fb348c00700 1 -- 172.21.15.118:0/3199022714 <== mon.0 v2:172.21.15.118:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fb33c01a910 con 0x7fb34c1085c0 2024-03-24T00:04:59.805 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:04:57.854+0000 7fb348c00700 1 -- 172.21.15.118:0/3199022714 <== mon.0 v2:172.21.15.118:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fb33c024930 con 0x7fb34c1085c0 2024-03-24T00:04:59.805 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:04:57.854+0000 7fb353000700 1 -- 172.21.15.118:0/3199022714 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb34c106570 con 0x7fb34c1085c0 2024-03-24T00:04:59.805 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:04:57.854+0000 7fb353000700 1 -- 172.21.15.118:0/3199022714 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb34c106a30 con 0x7fb34c1085c0 2024-03-24T00:04:59.805 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:04:57.856+0000 7fb353000700 1 -- 172.21.15.118:0/3199022714 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb34c0637c0 con 0x7fb34c1085c0 2024-03-24T00:04:59.805 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:04:57.856+0000 7fb348c00700 1 -- 172.21.15.118:0/3199022714 <== mon.0 v2:172.21.15.118:3300/0 4 ==== mgrmap(e 9) v1 ==== 46119+0+0 (secure 0 0 0) 0x7fb33c022020 con 0x7fb34c1085c0 2024-03-24T00:04:59.805 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:04:57.857+0000 7fb348c00700 1 --2- 172.21.15.118:0/3199022714 >> [v2:172.21.15.118:6800/389249272,v1:172.21.15.118:6801/389249272] conn(0x7fb330038fd0 0x7fb33003b490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:04:59.805 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:04:57.857+0000 7fb348c00700 1 -- 172.21.15.118:0/3199022714 <== mon.0 v2:172.21.15.118:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 940+0+0 (secure 0 0 0) 0x7fb33c04d150 con 0x7fb34c1085c0 2024-03-24T00:04:59.806 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:04:57.857+0000 7fb34b400700 1 -- 172.21.15.118:0/3199022714 >> [v2:172.21.15.118:6800/389249272,v1:172.21.15.118:6801/389249272] conn(0x7fb330038fd0 msgr2=0x7fb33003b490 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.118:6800/389249272 2024-03-24T00:04:59.806 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:04:57.857+0000 7fb34b400700 1 --2- 172.21.15.118:0/3199022714 >> [v2:172.21.15.118:6800/389249272,v1:172.21.15.118:6801/389249272] conn(0x7fb330038fd0 0x7fb33003b490 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-24T00:04:59.806 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:04:57.862+0000 7fb348c00700 1 -- 172.21.15.118:0/3199022714 <== mon.0 v2:172.21.15.118:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7fb33c00b3e0 con 0x7fb34c1085c0 2024-03-24T00:04:59.806 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:04:58.058+0000 7fb34b400700 1 -- 172.21.15.118:0/3199022714 >> [v2:172.21.15.118:6800/389249272,v1:172.21.15.118:6801/389249272] conn(0x7fb330038fd0 msgr2=0x7fb33003b490 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.118:6800/389249272 2024-03-24T00:04:59.806 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:04:58.058+0000 7fb34b400700 1 --2- 172.21.15.118:0/3199022714 >> [v2:172.21.15.118:6800/389249272,v1:172.21.15.118:6801/389249272] conn(0x7fb330038fd0 0x7fb33003b490 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-24T00:04:59.806 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:04:58.156+0000 7fb353000700 1 -- 172.21.15.118:0/3199022714 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_command({"prefix": "mgr stat"} v 0) v1 -- 0x7fb34c106d70 con 0x7fb34c1085c0 2024-03-24T00:04:59.807 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:04:58.157+0000 7fb348c00700 1 -- 172.21.15.118:0/3199022714 <== mon.0 v2:172.21.15.118:3300/0 7 ==== mon_command_ack([{"prefix": "mgr stat"}]=0 v9) v1 ==== 56+0+88 (secure 0 0 0) 0x7fb34c106d70 con 0x7fb34c1085c0 2024-03-24T00:04:59.807 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:04:58.159+0000 7fb353000700 1 -- 172.21.15.118:0/3199022714 >> [v2:172.21.15.118:6800/389249272,v1:172.21.15.118:6801/389249272] conn(0x7fb330038fd0 msgr2=0x7fb33003b490 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-24T00:04:59.807 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:04:58.159+0000 7fb353000700 1 --2- 172.21.15.118:0/3199022714 >> [v2:172.21.15.118:6800/389249272,v1:172.21.15.118:6801/389249272] conn(0x7fb330038fd0 0x7fb33003b490 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:04:59.807 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:04:58.159+0000 7fb353000700 1 -- 172.21.15.118:0/3199022714 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb34c1085c0 msgr2=0x7fb34c107e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:04:59.807 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:04:58.159+0000 7fb353000700 1 --2- 172.21.15.118:0/3199022714 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb34c1085c0 0x7fb34c107e40 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7fb33c011b60 tx=0x7fb33c011c40 comp rx=0 tx=0).stop 2024-03-24T00:04:59.808 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:04:58.160+0000 7fb353000700 1 -- 172.21.15.118:0/3199022714 shutdown_connections 2024-03-24T00:04:59.808 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:04:58.160+0000 7fb353000700 1 --2- 172.21.15.118:0/3199022714 >> [v2:172.21.15.118:6800/389249272,v1:172.21.15.118:6801/389249272] conn(0x7fb330038fd0 0x7fb33003b490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:04:59.808 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:04:58.160+0000 7fb353000700 1 --2- 172.21.15.118:0/3199022714 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb34c1085c0 0x7fb34c107e40 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:04:59.808 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:04:58.160+0000 7fb353000700 1 -- 172.21.15.118:0/3199022714 >> 172.21.15.118:0/3199022714 conn(0x7fb34c101fb0 msgr2=0x7fb34c102a50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:04:59.808 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:04:58.160+0000 7fb353000700 1 -- 172.21.15.118:0/3199022714 shutdown_connections 2024-03-24T00:04:59.808 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:04:58.160+0000 7fb353000700 1 -- 172.21.15.118:0/3199022714 wait complete. 2024-03-24T00:04:59.809 INFO:teuthology.orchestra.run.smithi118.stdout:Waiting for the mgr to restart... 2024-03-24T00:04:59.809 INFO:teuthology.orchestra.run.smithi118.stdout:Waiting for mgr epoch 9... 2024-03-24T00:05:00.064 INFO:journalctl@ceph.mgr.x.smithi118.stdout:Mar 24 00:04:59 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mgr-x[31762]: 2024-03-24T00:04:59.823+0000 7f6357ded200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-24T00:05:01.323 INFO:journalctl@ceph.mgr.x.smithi118.stdout:Mar 24 00:05:01 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mgr-x[31762]: 2024-03-24T00:05:01.155+0000 7f6357ded200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-03-24T00:05:01.323 INFO:journalctl@ceph.mgr.x.smithi118.stdout:Mar 24 00:05:01 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mgr-x[31762]: 2024-03-24T00:05:01.323+0000 7f6357ded200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-24T00:05:02.107 INFO:journalctl@ceph.mgr.x.smithi118.stdout:Mar 24 00:05:01 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mgr-x[31762]: 2024-03-24T00:05:01.736+0000 7f6357ded200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-24T00:05:02.107 INFO:journalctl@ceph.mgr.x.smithi118.stdout:Mar 24 00:05:01 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mgr-x[31762]: 2024-03-24T00:05:01.853+0000 7f6357ded200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-24T00:05:02.108 INFO:journalctl@ceph.mgr.x.smithi118.stdout:Mar 24 00:05:02 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mgr-x[31762]: 2024-03-24T00:05:02.107+0000 7f6357ded200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-24T00:05:02.445 INFO:journalctl@ceph.mgr.x.smithi118.stdout:Mar 24 00:05:02 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mgr-x[31762]: 2024-03-24T00:05:02.351+0000 7f6357ded200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-24T00:05:03.035 INFO:journalctl@ceph.mgr.x.smithi118.stdout:Mar 24 00:05:02 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mgr-x[31762]: 2024-03-24T00:05:02.720+0000 7f6357ded200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-24T00:05:03.036 INFO:journalctl@ceph.mgr.x.smithi118.stdout:Mar 24 00:05:03 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mgr-x[31762]: 2024-03-24T00:05:03.035+0000 7f6357ded200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-24T00:05:03.695 INFO:journalctl@ceph.mgr.x.smithi118.stdout:Mar 24 00:05:03 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mgr-x[31762]: 2024-03-24T00:05:03.336+0000 7f6357ded200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-24T00:05:04.695 INFO:journalctl@ceph.mgr.x.smithi118.stdout:Mar 24 00:05:04 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mgr-x[31762]: 2024-03-24T00:05:04.351+0000 7f6357ded200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-24T00:05:05.175 INFO:journalctl@ceph.mgr.x.smithi118.stdout:Mar 24 00:05:05 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mgr-x[31762]: 2024-03-24T00:05:05.046+0000 7f6357ded200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-24T00:05:05.175 INFO:journalctl@ceph.mgr.x.smithi118.stdout:Mar 24 00:05:05 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mgr-x[31762]: 2024-03-24T00:05:05.174+0000 7f6357ded200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-24T00:05:05.946 INFO:journalctl@ceph.mgr.x.smithi118.stdout:Mar 24 00:05:05 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mgr-x[31762]: 2024-03-24T00:05:05.454+0000 7f6357ded200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-24T00:05:05.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:05:05 smithi118 ceph-mon[31535]: Active manager daemon x restarted 2024-03-24T00:05:05.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:05:05 smithi118 ceph-mon[31535]: Activating manager daemon x 2024-03-24T00:05:05.947 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:05:05 smithi118 ceph-mon[31535]: osdmap e3: 0 total, 0 up, 0 in 2024-03-24T00:05:05.947 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:05:05 smithi118 ceph-mon[31535]: mgrmap e10: x(active, starting, since 0.00480657s) 2024-03-24T00:05:05.947 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:05:05 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-24T00:05:05.947 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:05:05 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "x", "id": "x"}]: dispatch 2024-03-24T00:05:05.947 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:05:05 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-03-24T00:05:05.947 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:05:05 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-03-24T00:05:05.947 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:05:05 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-03-24T00:05:05.947 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:05:05 smithi118 ceph-mon[31535]: Manager daemon x is now available 2024-03-24T00:05:06.790 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:05:06 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T00:05:06.790 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:05:06 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T00:05:06.791 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:05:06 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/mirror_snapshot_schedule"}]: dispatch 2024-03-24T00:05:06.791 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:05:06 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/x/trash_purge_schedule"}]: dispatch 2024-03-24T00:05:06.791 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:05:06 smithi118 ceph-mon[31535]: mgrmap e11: x(active, since 1.01838s) 2024-03-24T00:05:07.378 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stdout { 2024-03-24T00:05:07.386 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stdout "mgrmap_epoch": 11, 2024-03-24T00:05:07.386 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stdout "initialized": true 2024-03-24T00:05:07.386 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stdout } 2024-03-24T00:05:07.386 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:05:00.370+0000 7f4438c00700 1 Processor -- start 2024-03-24T00:05:07.387 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:05:00.371+0000 7f4438c00700 1 -- start start 2024-03-24T00:05:07.387 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:05:00.371+0000 7f4438c00700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f442c0083f0 0x7f442c006850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:05:07.387 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:05:00.371+0000 7f4438c00700 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f442c006e20 con 0x7f442c0083f0 2024-03-24T00:05:07.387 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:05:00.371+0000 7f4433400700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f442c0083f0 0x7f442c006850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:05:07.387 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:05:00.371+0000 7f4433400700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f442c0083f0 0x7f442c006850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.118:3300/0 says I am v2:172.21.15.118:49822/0 (socket says 172.21.15.118:49822) 2024-03-24T00:05:07.387 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:05:00.371+0000 7f4433400700 1 -- 172.21.15.118:0/2201512176 learned_addr learned my addr 172.21.15.118:0/2201512176 (peer_addr_for_me v2:172.21.15.118:0/0) 2024-03-24T00:05:07.387 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:05:00.372+0000 7f4433400700 1 -- 172.21.15.118:0/2201512176 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f442c006fa0 con 0x7f442c0083f0 2024-03-24T00:05:07.387 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:05:00.372+0000 7f4433400700 1 --2- 172.21.15.118:0/2201512176 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f442c0083f0 0x7f442c006850 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f44280098a0 tx=0x7f4428009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f8792f22d62a5a20 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:05:07.387 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:05:00.372+0000 7f4432000700 1 -- 172.21.15.118:0/2201512176 <== mon.0 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f4428004160 con 0x7f442c0083f0 2024-03-24T00:05:07.387 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:05:00.372+0000 7f4432000700 1 -- 172.21.15.118:0/2201512176 <== mon.0 v2:172.21.15.118:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f4428004300 con 0x7f442c0083f0 2024-03-24T00:05:07.387 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:05:00.372+0000 7f4432000700 1 -- 172.21.15.118:0/2201512176 <== mon.0 v2:172.21.15.118:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f4428011b20 con 0x7f442c0083f0 2024-03-24T00:05:07.387 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:05:00.373+0000 7f4438c00700 1 -- 172.21.15.118:0/2201512176 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f442c0083f0 msgr2=0x7f442c006850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:05:07.387 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:05:00.373+0000 7f4438c00700 1 --2- 172.21.15.118:0/2201512176 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f442c0083f0 0x7f442c006850 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7f44280098a0 tx=0x7f4428009bb0 comp rx=0 tx=0).stop 2024-03-24T00:05:07.388 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:05:00.373+0000 7f4438c00700 1 -- 172.21.15.118:0/2201512176 shutdown_connections 2024-03-24T00:05:07.388 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:05:00.373+0000 7f4438c00700 1 --2- 172.21.15.118:0/2201512176 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f442c0083f0 0x7f442c006850 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:05:07.388 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:05:00.373+0000 7f4438c00700 1 -- 172.21.15.118:0/2201512176 >> 172.21.15.118:0/2201512176 conn(0x7f442c090ef0 msgr2=0x7f442c093350 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:05:07.388 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:05:00.373+0000 7f4438c00700 1 -- 172.21.15.118:0/2201512176 shutdown_connections 2024-03-24T00:05:07.388 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:05:00.373+0000 7f4438c00700 1 -- 172.21.15.118:0/2201512176 wait complete. 2024-03-24T00:05:07.388 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:05:00.374+0000 7f4438c00700 1 Processor -- start 2024-03-24T00:05:07.388 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:05:00.374+0000 7f4438c00700 1 -- start start 2024-03-24T00:05:07.388 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:05:00.375+0000 7f4438c00700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f442c0083f0 0x7f442c12ba50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:05:07.388 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:05:00.375+0000 7f4438c00700 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f442c09bfb0 con 0x7f442c0083f0 2024-03-24T00:05:07.388 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:05:00.375+0000 7f4433400700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f442c0083f0 0x7f442c12ba50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:05:07.388 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:05:00.375+0000 7f4433400700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f442c0083f0 0x7f442c12ba50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.118:3300/0 says I am v2:172.21.15.118:49824/0 (socket says 172.21.15.118:49824) 2024-03-24T00:05:07.388 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:05:00.375+0000 7f4433400700 1 -- 172.21.15.118:0/193442553 learned_addr learned my addr 172.21.15.118:0/193442553 (peer_addr_for_me v2:172.21.15.118:0/0) 2024-03-24T00:05:07.388 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:05:00.375+0000 7f4433400700 1 -- 172.21.15.118:0/193442553 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f442c12bf90 con 0x7f442c0083f0 2024-03-24T00:05:07.388 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:05:00.376+0000 7f4433400700 1 --2- 172.21.15.118:0/193442553 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f442c0083f0 0x7f442c12ba50 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f44280117e0 tx=0x7f4428011b80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:05:07.389 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:05:00.376+0000 7f4427e00700 1 -- 172.21.15.118:0/193442553 <== mon.0 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f44280039e0 con 0x7f442c0083f0 2024-03-24T00:05:07.389 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:05:00.376+0000 7f4427e00700 1 -- 172.21.15.118:0/193442553 <== mon.0 v2:172.21.15.118:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f4428003b80 con 0x7f442c0083f0 2024-03-24T00:05:07.389 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:05:00.376+0000 7f4427e00700 1 -- 172.21.15.118:0/193442553 <== mon.0 v2:172.21.15.118:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f4428023770 con 0x7f442c0083f0 2024-03-24T00:05:07.389 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:05:00.376+0000 7f4438c00700 1 -- 172.21.15.118:0/193442553 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f442c12c220 con 0x7f442c0083f0 2024-03-24T00:05:07.389 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:05:00.376+0000 7f4438c00700 1 -- 172.21.15.118:0/193442553 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f442c12c680 con 0x7f442c0083f0 2024-03-24T00:05:07.389 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:05:00.377+0000 7f4427e00700 1 -- 172.21.15.118:0/193442553 <== mon.0 v2:172.21.15.118:3300/0 4 ==== mgrmap(e 9) v1 ==== 46119+0+0 (secure 0 0 0) 0x7f4428022030 con 0x7f442c0083f0 2024-03-24T00:05:07.389 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:05:00.378+0000 7f4427e00700 1 --2- 172.21.15.118:0/193442553 >> [v2:172.21.15.118:6800/389249272,v1:172.21.15.118:6801/389249272] conn(0x7f441c038fc0 0x7f441c03b480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:05:07.389 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:05:00.378+0000 7f4427e00700 1 -- 172.21.15.118:0/193442553 --> [v2:172.21.15.118:6800/389249272,v1:172.21.15.118:6801/389249272] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x7f441c03bb90 con 0x7f441c038fc0 2024-03-24T00:05:07.389 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:05:00.378+0000 7f4427e00700 1 -- 172.21.15.118:0/193442553 <== mon.0 v2:172.21.15.118:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 940+0+0 (secure 0 0 0) 0x7f442804cfa0 con 0x7f442c0083f0 2024-03-24T00:05:07.389 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:05:00.378+0000 7f4432a00700 1 -- 172.21.15.118:0/193442553 >> [v2:172.21.15.118:6800/389249272,v1:172.21.15.118:6801/389249272] conn(0x7f441c038fc0 msgr2=0x7f441c03b480 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.118:6800/389249272 2024-03-24T00:05:07.389 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:05:00.378+0000 7f4432a00700 1 --2- 172.21.15.118:0/193442553 >> [v2:172.21.15.118:6800/389249272,v1:172.21.15.118:6801/389249272] conn(0x7f441c038fc0 0x7f441c03b480 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-24T00:05:07.389 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:05:00.579+0000 7f4432a00700 1 -- 172.21.15.118:0/193442553 >> [v2:172.21.15.118:6800/389249272,v1:172.21.15.118:6801/389249272] conn(0x7f441c038fc0 msgr2=0x7f441c03b480 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.118:6800/389249272 2024-03-24T00:05:07.389 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:05:00.579+0000 7f4432a00700 1 --2- 172.21.15.118:0/193442553 >> [v2:172.21.15.118:6800/389249272,v1:172.21.15.118:6801/389249272] conn(0x7f441c038fc0 0x7f441c03b480 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-24T00:05:07.389 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:05:00.980+0000 7f4432a00700 1 -- 172.21.15.118:0/193442553 >> [v2:172.21.15.118:6800/389249272,v1:172.21.15.118:6801/389249272] conn(0x7f441c038fc0 msgr2=0x7f441c03b480 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.118:6800/389249272 2024-03-24T00:05:07.390 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:05:00.980+0000 7f4432a00700 1 --2- 172.21.15.118:0/193442553 >> [v2:172.21.15.118:6800/389249272,v1:172.21.15.118:6801/389249272] conn(0x7f441c038fc0 0x7f441c03b480 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-03-24T00:05:07.390 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:05:01.781+0000 7f4432a00700 1 -- 172.21.15.118:0/193442553 >> [v2:172.21.15.118:6800/389249272,v1:172.21.15.118:6801/389249272] conn(0x7f441c038fc0 msgr2=0x7f441c03b480 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.118:6800/389249272 2024-03-24T00:05:07.390 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:05:01.781+0000 7f4432a00700 1 --2- 172.21.15.118:0/193442553 >> [v2:172.21.15.118:6800/389249272,v1:172.21.15.118:6801/389249272] conn(0x7f441c038fc0 0x7f441c03b480 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-03-24T00:05:07.390 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:05:03.383+0000 7f4432a00700 1 -- 172.21.15.118:0/193442553 >> [v2:172.21.15.118:6800/389249272,v1:172.21.15.118:6801/389249272] conn(0x7f441c038fc0 msgr2=0x7f441c03b480 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.118:6800/389249272 2024-03-24T00:05:07.390 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:05:03.383+0000 7f4432a00700 1 --2- 172.21.15.118:0/193442553 >> [v2:172.21.15.118:6800/389249272,v1:172.21.15.118:6801/389249272] conn(0x7f441c038fc0 0x7f441c03b480 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 3.200000 2024-03-24T00:05:07.390 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:05:05.463+0000 7f4427e00700 1 -- 172.21.15.118:0/193442553 <== mon.0 v2:172.21.15.118:3300/0 6 ==== mgrmap(e 10) v1 ==== 45886+0+0 (secure 0 0 0) 0x7f4428023ac0 con 0x7f442c0083f0 2024-03-24T00:05:07.390 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:05:05.464+0000 7f4427e00700 1 -- 172.21.15.118:0/193442553 >> [v2:172.21.15.118:6800/389249272,v1:172.21.15.118:6801/389249272] conn(0x7f441c038fc0 msgr2=0x7f441c03b480 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-24T00:05:07.390 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:05:05.464+0000 7f4427e00700 1 --2- 172.21.15.118:0/193442553 >> [v2:172.21.15.118:6800/389249272,v1:172.21.15.118:6801/389249272] conn(0x7f441c038fc0 0x7f441c03b480 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:05:07.390 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:05:06.470+0000 7f4427e00700 1 -- 172.21.15.118:0/193442553 <== mon.0 v2:172.21.15.118:3300/0 7 ==== mgrmap(e 11) v1 ==== 46013+0+0 (secure 0 0 0) 0x7f442800ed00 con 0x7f442c0083f0 2024-03-24T00:05:07.390 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:05:06.470+0000 7f4427e00700 1 --2- 172.21.15.118:0/193442553 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7f441c03ca40 0x7f441c03ee30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:05:07.390 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:05:06.470+0000 7f4427e00700 1 -- 172.21.15.118:0/193442553 --> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x7f4428025e50 con 0x7f441c03ca40 2024-03-24T00:05:07.390 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:05:06.472+0000 7f4432a00700 1 --2- 172.21.15.118:0/193442553 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7f441c03ca40 0x7f441c03ee30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:05:07.390 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:05:06.475+0000 7f4432a00700 1 --2- 172.21.15.118:0/193442553 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7f441c03ca40 0x7f441c03ee30 secure :-1 s=READY pgs=3 cs=0 l=1 rev1=1 crypto rx=0x7f4420003a10 tx=0x7f4420009270 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:05:07.391 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:05:06.477+0000 7f4427e00700 1 -- 172.21.15.118:0/193442553 <== mgr.14152 v2:172.21.15.118:6800/2112912943 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+7774 (secure 0 0 0) 0x7f4428025e50 con 0x7f441c03ca40 2024-03-24T00:05:07.391 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:05:06.484+0000 7f4438c00700 1 -- 172.21.15.118:0/193442553 --> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] -- command(tid 1: {"prefix": "mgr_status"}) v1 -- 0x7f442c004940 con 0x7f441c03ca40 2024-03-24T00:05:07.391 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:05:06.485+0000 7f4427e00700 1 -- 172.21.15.118:0/193442553 <== mgr.14152 v2:172.21.15.118:6800/2112912943 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+52 (secure 0 0 0) 0x7f442c004940 con 0x7f441c03ca40 2024-03-24T00:05:07.391 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:05:06.485+0000 7f4438c00700 1 -- 172.21.15.118:0/193442553 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7f441c03ca40 msgr2=0x7f441c03ee30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:05:07.391 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:05:06.485+0000 7f4438c00700 1 --2- 172.21.15.118:0/193442553 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7f441c03ca40 0x7f441c03ee30 secure :-1 s=READY pgs=3 cs=0 l=1 rev1=1 crypto rx=0x7f4420003a10 tx=0x7f4420009270 comp rx=0 tx=0).stop 2024-03-24T00:05:07.391 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:05:06.485+0000 7f4438c00700 1 -- 172.21.15.118:0/193442553 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f442c0083f0 msgr2=0x7f442c12ba50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:05:07.391 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:05:06.485+0000 7f4438c00700 1 --2- 172.21.15.118:0/193442553 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f442c0083f0 0x7f442c12ba50 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f44280117e0 tx=0x7f4428011b80 comp rx=0 tx=0).stop 2024-03-24T00:05:07.391 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:05:06.485+0000 7f4438c00700 1 -- 172.21.15.118:0/193442553 shutdown_connections 2024-03-24T00:05:07.391 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:05:06.485+0000 7f4438c00700 1 --2- 172.21.15.118:0/193442553 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7f441c03ca40 0x7f441c03ee30 unknown :-1 s=CLOSED pgs=3 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:05:07.391 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:05:06.486+0000 7f4438c00700 1 --2- 172.21.15.118:0/193442553 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f442c0083f0 0x7f442c12ba50 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:05:07.391 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:05:06.486+0000 7f4438c00700 1 --2- 172.21.15.118:0/193442553 >> [v2:172.21.15.118:6800/389249272,v1:172.21.15.118:6801/389249272] conn(0x7f441c038fc0 0x7f441c03b480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:05:07.391 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:05:06.486+0000 7f4438c00700 1 -- 172.21.15.118:0/193442553 >> 172.21.15.118:0/193442553 conn(0x7f442c090ef0 msgr2=0x7f442c091910 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:05:07.391 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:05:06.486+0000 7f4438c00700 1 -- 172.21.15.118:0/193442553 shutdown_connections 2024-03-24T00:05:07.392 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:05:06.486+0000 7f4438c00700 1 -- 172.21.15.118:0/193442553 wait complete. 2024-03-24T00:05:07.392 INFO:teuthology.orchestra.run.smithi118.stdout:mgr epoch 9 is available 2024-03-24T00:05:07.392 INFO:teuthology.orchestra.run.smithi118.stdout:Generating a dashboard self-signed certificate... 2024-03-24T00:05:07.695 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:05:07 smithi118 ceph-mon[31535]: [24/Mar/2024:00:05:05] ENGINE Bus STARTING 2024-03-24T00:05:07.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:05:07 smithi118 ceph-mon[31535]: [24/Mar/2024:00:05:06] ENGINE Serving on http://172.21.15.118:8765 2024-03-24T00:05:07.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:05:07 smithi118 ceph-mon[31535]: [24/Mar/2024:00:05:06] ENGINE Serving on https://172.21.15.118:7150 2024-03-24T00:05:07.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:05:07 smithi118 ceph-mon[31535]: [24/Mar/2024:00:05:06] ENGINE Bus STARTED 2024-03-24T00:05:07.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:05:07 smithi118 ceph-mon[31535]: from='client.14156 -' entity='client.admin' cmd=[{"prefix": "get_command_descriptions"}]: dispatch 2024-03-24T00:05:07.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:05:07 smithi118 ceph-mon[31535]: from='client.14156 -' entity='client.admin' cmd=[{"prefix": "mgr_status"}]: dispatch 2024-03-24T00:05:08.790 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:05:08 smithi118 ceph-mon[31535]: mgrmap e12: x(active, since 2s) 2024-03-24T00:05:08.790 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:05:08 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T00:05:09.508 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stdout Self-signed certificate created 2024-03-24T00:05:09.509 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:05:08.372+0000 7fe284a00700 1 Processor -- start 2024-03-24T00:05:09.509 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:05:08.373+0000 7fe284a00700 1 -- start start 2024-03-24T00:05:09.509 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:05:08.373+0000 7fe284a00700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fe280107b10 0x7fe280107f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:05:09.509 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:05:08.373+0000 7fe284a00700 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe280108500 con 0x7fe280107b10 2024-03-24T00:05:09.509 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:05:08.374+0000 7fe27de00700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fe280107b10 0x7fe280107f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:05:09.509 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:05:08.374+0000 7fe27de00700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fe280107b10 0x7fe280107f30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.118:3300/0 says I am v2:172.21.15.118:60650/0 (socket says 172.21.15.118:60650) 2024-03-24T00:05:09.509 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:05:08.374+0000 7fe27de00700 1 -- 172.21.15.118:0/4272829986 learned_addr learned my addr 172.21.15.118:0/4272829986 (peer_addr_for_me v2:172.21.15.118:0/0) 2024-03-24T00:05:09.509 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:05:08.374+0000 7fe27de00700 1 -- 172.21.15.118:0/4272829986 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe280108d10 con 0x7fe280107b10 2024-03-24T00:05:09.509 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:05:08.375+0000 7fe27de00700 1 --2- 172.21.15.118:0/4272829986 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fe280107b10 0x7fe280107f30 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7fe2700098a0 tx=0x7fe270009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fb69200277c85c3e server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:05:09.509 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:05:08.375+0000 7fe27ca00700 1 -- 172.21.15.118:0/4272829986 <== mon.0 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fe270004160 con 0x7fe280107b10 2024-03-24T00:05:09.509 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:05:08.375+0000 7fe27ca00700 1 -- 172.21.15.118:0/4272829986 <== mon.0 v2:172.21.15.118:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fe270004300 con 0x7fe280107b10 2024-03-24T00:05:09.509 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:05:08.375+0000 7fe27ca00700 1 -- 172.21.15.118:0/4272829986 <== mon.0 v2:172.21.15.118:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fe270011b20 con 0x7fe280107b10 2024-03-24T00:05:09.510 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:05:08.376+0000 7fe284a00700 1 -- 172.21.15.118:0/4272829986 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fe280107b10 msgr2=0x7fe280107f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:05:09.510 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:05:08.376+0000 7fe284a00700 1 --2- 172.21.15.118:0/4272829986 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fe280107b10 0x7fe280107f30 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7fe2700098a0 tx=0x7fe270009bb0 comp rx=0 tx=0).stop 2024-03-24T00:05:09.510 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:05:08.376+0000 7fe284a00700 1 -- 172.21.15.118:0/4272829986 shutdown_connections 2024-03-24T00:05:09.510 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:05:08.376+0000 7fe284a00700 1 --2- 172.21.15.118:0/4272829986 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fe280107b10 0x7fe280107f30 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:05:09.510 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:05:08.376+0000 7fe284a00700 1 -- 172.21.15.118:0/4272829986 >> 172.21.15.118:0/4272829986 conn(0x7fe280103090 msgr2=0x7fe2801054f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:05:09.510 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:05:08.376+0000 7fe284a00700 1 -- 172.21.15.118:0/4272829986 shutdown_connections 2024-03-24T00:05:09.510 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:05:08.376+0000 7fe284a00700 1 -- 172.21.15.118:0/4272829986 wait complete. 2024-03-24T00:05:09.510 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:05:08.378+0000 7fe284a00700 1 Processor -- start 2024-03-24T00:05:09.510 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:05:08.378+0000 7fe284a00700 1 -- start start 2024-03-24T00:05:09.510 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:05:08.379+0000 7fe284a00700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fe28007ca40 0x7fe28007b1b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:05:09.510 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:05:08.379+0000 7fe284a00700 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe28010c0c0 con 0x7fe28007ca40 2024-03-24T00:05:09.510 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:05:08.379+0000 7fe27de00700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fe28007ca40 0x7fe28007b1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:05:09.510 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:05:08.379+0000 7fe27de00700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fe28007ca40 0x7fe28007b1b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.118:3300/0 says I am v2:172.21.15.118:60658/0 (socket says 172.21.15.118:60658) 2024-03-24T00:05:09.511 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:05:08.379+0000 7fe27de00700 1 -- 172.21.15.118:0/2124534249 learned_addr learned my addr 172.21.15.118:0/2124534249 (peer_addr_for_me v2:172.21.15.118:0/0) 2024-03-24T00:05:09.511 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:05:08.380+0000 7fe27de00700 1 -- 172.21.15.118:0/2124534249 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe28007ce60 con 0x7fe28007ca40 2024-03-24T00:05:09.511 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:05:08.380+0000 7fe27de00700 1 --2- 172.21.15.118:0/2124534249 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fe28007ca40 0x7fe28007b1b0 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7fe270000c00 tx=0x7fe270011ba0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:05:09.511 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:05:08.380+0000 7fe276a00700 1 -- 172.21.15.118:0/2124534249 <== mon.0 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fe2700036a0 con 0x7fe28007ca40 2024-03-24T00:05:09.511 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:05:08.380+0000 7fe276a00700 1 -- 172.21.15.118:0/2124534249 <== mon.0 v2:172.21.15.118:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fe27001a460 con 0x7fe28007ca40 2024-03-24T00:05:09.511 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:05:08.380+0000 7fe276a00700 1 -- 172.21.15.118:0/2124534249 <== mon.0 v2:172.21.15.118:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fe27001b440 con 0x7fe28007ca40 2024-03-24T00:05:09.511 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:05:08.380+0000 7fe284a00700 1 -- 172.21.15.118:0/2124534249 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe28007b6f0 con 0x7fe28007ca40 2024-03-24T00:05:09.511 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:05:08.380+0000 7fe284a00700 1 -- 172.21.15.118:0/2124534249 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe28007bb50 con 0x7fe28007ca40 2024-03-24T00:05:09.511 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:05:08.382+0000 7fe284a00700 1 -- 172.21.15.118:0/2124534249 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe2800637c0 con 0x7fe28007ca40 2024-03-24T00:05:09.512 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:05:08.382+0000 7fe276a00700 1 -- 172.21.15.118:0/2124534249 <== mon.0 v2:172.21.15.118:3300/0 4 ==== mgrmap(e 12) v1 ==== 46119+0+0 (secure 0 0 0) 0x7fe27001b740 con 0x7fe28007ca40 2024-03-24T00:05:09.512 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:05:08.382+0000 7fe276a00700 1 --2- 172.21.15.118:0/2124534249 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7fe264038f70 0x7fe26403b430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:05:09.512 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:05:08.382+0000 7fe276a00700 1 -- 172.21.15.118:0/2124534249 <== mon.0 v2:172.21.15.118:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1155+0+0 (secure 0 0 0) 0x7fe27004d720 con 0x7fe28007ca40 2024-03-24T00:05:09.512 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:05:08.383+0000 7fe27d400700 1 --2- 172.21.15.118:0/2124534249 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7fe264038f70 0x7fe26403b430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:05:09.512 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:05:08.383+0000 7fe27d400700 1 --2- 172.21.15.118:0/2124534249 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7fe264038f70 0x7fe26403b430 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7fe26c006fd0 tx=0x7fe26c006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:05:09.512 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:05:08.392+0000 7fe276a00700 1 -- 172.21.15.118:0/2124534249 <== mon.0 v2:172.21.15.118:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7fe270022e40 con 0x7fe28007ca40 2024-03-24T00:05:09.512 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:05:08.630+0000 7fe284a00700 1 -- 172.21.15.118:0/2124534249 --> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] -- mgr_command(tid 0: {"prefix": "dashboard create-self-signed-cert", "target": ["mon-mgr", ""]}) v1 -- 0x7fe2801043a0 con 0x7fe264038f70 2024-03-24T00:05:09.512 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:05:08.790+0000 7fe276a00700 1 -- 172.21.15.118:0/2124534249 <== mgr.14152 v2:172.21.15.118:6800/2112912943 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+32 (secure 0 0 0) 0x7fe2801043a0 con 0x7fe264038f70 2024-03-24T00:05:09.512 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:05:08.793+0000 7fe284a00700 1 -- 172.21.15.118:0/2124534249 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7fe264038f70 msgr2=0x7fe26403b430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:05:09.512 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:05:08.793+0000 7fe284a00700 1 --2- 172.21.15.118:0/2124534249 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7fe264038f70 0x7fe26403b430 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7fe26c006fd0 tx=0x7fe26c006e40 comp rx=0 tx=0).stop 2024-03-24T00:05:09.513 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:05:08.793+0000 7fe284a00700 1 -- 172.21.15.118:0/2124534249 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fe28007ca40 msgr2=0x7fe28007b1b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:05:09.513 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:05:08.793+0000 7fe284a00700 1 --2- 172.21.15.118:0/2124534249 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fe28007ca40 0x7fe28007b1b0 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7fe270000c00 tx=0x7fe270011ba0 comp rx=0 tx=0).stop 2024-03-24T00:05:09.513 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:05:08.793+0000 7fe284a00700 1 -- 172.21.15.118:0/2124534249 shutdown_connections 2024-03-24T00:05:09.513 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:05:08.793+0000 7fe284a00700 1 --2- 172.21.15.118:0/2124534249 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7fe264038f70 0x7fe26403b430 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:05:09.513 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:05:08.793+0000 7fe284a00700 1 --2- 172.21.15.118:0/2124534249 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fe28007ca40 0x7fe28007b1b0 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:05:09.513 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:05:08.793+0000 7fe284a00700 1 -- 172.21.15.118:0/2124534249 >> 172.21.15.118:0/2124534249 conn(0x7fe280103090 msgr2=0x7fe280103c90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:05:09.513 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:05:08.794+0000 7fe284a00700 1 -- 172.21.15.118:0/2124534249 shutdown_connections 2024-03-24T00:05:09.513 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:05:08.794+0000 7fe284a00700 1 -- 172.21.15.118:0/2124534249 wait complete. 2024-03-24T00:05:09.513 INFO:teuthology.orchestra.run.smithi118.stdout:Creating initial admin user... 2024-03-24T00:05:09.777 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:05:09 smithi118 ceph-mon[31535]: from='client.14164 -' entity='client.admin' cmd=[{"prefix": "dashboard create-self-signed-cert", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T00:05:09.777 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:05:09 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T00:05:09.778 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:05:09 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T00:05:10.600 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:05:10 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T00:05:10.601 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:05:10 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T00:05:11.690 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:05:11 smithi118 ceph-mon[31535]: from='client.14166 -' entity='client.admin' cmd=[{"prefix": "dashboard ac-user-create", "username": "admin", "rolename": "administrator", "force_password": true, "pwd_update_required": true, "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T00:05:12.225 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stdout {"username": "admin", "password": "$2b$12$OVkquWWFmITcBUyh7D1C/uQynu4uS4/P9KQPo07/uf5IAGb6WUlmC", "roles": ["administrator"], "name": null, "email": null, "lastUpdate": 1711238710, "enabled": true, "pwdExpirationDate": null, "pwdUpdateRequired": true} 2024-03-24T00:05:12.225 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:05:10.081+0000 7f7fd6000700 1 Processor -- start 2024-03-24T00:05:12.225 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:05:10.081+0000 7f7fd6000700 1 -- start start 2024-03-24T00:05:12.225 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:05:10.081+0000 7f7fd6000700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f7fd007cd90 0x7f7fd007b1f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:05:12.226 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:05:10.081+0000 7f7fd6000700 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7fd007b7c0 con 0x7f7fd007cd90 2024-03-24T00:05:12.227 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:05:10.082+0000 7f7fcf400700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f7fd007cd90 0x7f7fd007b1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:05:12.227 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:05:10.082+0000 7f7fcf400700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f7fd007cd90 0x7f7fd007b1f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.118:3300/0 says I am v2:172.21.15.118:60668/0 (socket says 172.21.15.118:60668) 2024-03-24T00:05:12.227 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:05:10.082+0000 7f7fcf400700 1 -- 172.21.15.118:0/3979644395 learned_addr learned my addr 172.21.15.118:0/3979644395 (peer_addr_for_me v2:172.21.15.118:0/0) 2024-03-24T00:05:12.227 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:05:10.082+0000 7f7fcf400700 1 -- 172.21.15.118:0/3979644395 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7fd007b940 con 0x7f7fd007cd90 2024-03-24T00:05:12.227 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:05:10.083+0000 7f7fcf400700 1 --2- 172.21.15.118:0/3979644395 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f7fd007cd90 0x7f7fd007b1f0 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f7fc00098a0 tx=0x7f7fc0009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f16894144c051530 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:05:12.227 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:05:10.083+0000 7f7fce000700 1 -- 172.21.15.118:0/3979644395 <== mon.0 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f7fc0004160 con 0x7f7fd007cd90 2024-03-24T00:05:12.227 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:05:10.083+0000 7f7fce000700 1 -- 172.21.15.118:0/3979644395 <== mon.0 v2:172.21.15.118:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f7fc0004300 con 0x7f7fd007cd90 2024-03-24T00:05:12.227 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:05:10.083+0000 7f7fce000700 1 -- 172.21.15.118:0/3979644395 <== mon.0 v2:172.21.15.118:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f7fc0011b20 con 0x7f7fd007cd90 2024-03-24T00:05:12.227 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:05:10.084+0000 7f7fd6000700 1 -- 172.21.15.118:0/3979644395 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f7fd007cd90 msgr2=0x7f7fd007b1f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:05:12.227 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:05:10.084+0000 7f7fd6000700 1 --2- 172.21.15.118:0/3979644395 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f7fd007cd90 0x7f7fd007b1f0 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f7fc00098a0 tx=0x7f7fc0009bb0 comp rx=0 tx=0).stop 2024-03-24T00:05:12.227 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:05:10.084+0000 7f7fd6000700 1 -- 172.21.15.118:0/3979644395 shutdown_connections 2024-03-24T00:05:12.227 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:05:10.084+0000 7f7fd6000700 1 --2- 172.21.15.118:0/3979644395 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f7fd007cd90 0x7f7fd007b1f0 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:05:12.231 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:05:10.084+0000 7f7fd6000700 1 -- 172.21.15.118:0/3979644395 >> 172.21.15.118:0/3979644395 conn(0x7f7fd0103080 msgr2=0x7f7fd01054c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:05:12.231 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:05:10.084+0000 7f7fd6000700 1 -- 172.21.15.118:0/3979644395 shutdown_connections 2024-03-24T00:05:12.231 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:05:10.084+0000 7f7fd6000700 1 -- 172.21.15.118:0/3979644395 wait complete. 2024-03-24T00:05:12.231 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:05:10.086+0000 7f7fd6000700 1 Processor -- start 2024-03-24T00:05:12.232 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:05:10.086+0000 7f7fd6000700 1 -- start start 2024-03-24T00:05:12.232 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:05:10.086+0000 7f7fd6000700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f7fd007cd90 0x7f7fd019dc40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:05:12.232 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:05:10.086+0000 7f7fd6000700 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7fd010e160 con 0x7f7fd007cd90 2024-03-24T00:05:12.232 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:05:10.086+0000 7f7fcf400700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f7fd007cd90 0x7f7fd019dc40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:05:12.232 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:05:10.086+0000 7f7fcf400700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f7fd007cd90 0x7f7fd019dc40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.118:3300/0 says I am v2:172.21.15.118:60674/0 (socket says 172.21.15.118:60674) 2024-03-24T00:05:12.232 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:05:10.086+0000 7f7fcf400700 1 -- 172.21.15.118:0/511893998 learned_addr learned my addr 172.21.15.118:0/511893998 (peer_addr_for_me v2:172.21.15.118:0/0) 2024-03-24T00:05:12.232 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:05:10.087+0000 7f7fcf400700 1 -- 172.21.15.118:0/511893998 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7fd019e180 con 0x7f7fd007cd90 2024-03-24T00:05:12.232 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:05:10.087+0000 7f7fcf400700 1 --2- 172.21.15.118:0/511893998 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f7fd007cd90 0x7f7fd019dc40 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f7fc00117e0 tx=0x7f7fc0011ba0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:05:12.232 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:05:10.087+0000 7f7fc7e00700 1 -- 172.21.15.118:0/511893998 <== mon.0 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f7fc0003ba0 con 0x7f7fd007cd90 2024-03-24T00:05:12.232 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:05:10.087+0000 7f7fc7e00700 1 -- 172.21.15.118:0/511893998 <== mon.0 v2:172.21.15.118:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f7fc001a910 con 0x7f7fd007cd90 2024-03-24T00:05:12.232 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:05:10.087+0000 7f7fc7e00700 1 -- 172.21.15.118:0/511893998 <== mon.0 v2:172.21.15.118:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f7fc00248f0 con 0x7f7fd007cd90 2024-03-24T00:05:12.232 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:05:10.088+0000 7f7fd6000700 1 -- 172.21.15.118:0/511893998 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7fd019e410 con 0x7f7fd007cd90 2024-03-24T00:05:12.232 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:05:10.088+0000 7f7fd6000700 1 -- 172.21.15.118:0/511893998 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7fd019e870 con 0x7f7fd007cd90 2024-03-24T00:05:12.232 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:05:10.089+0000 7f7fd6000700 1 -- 172.21.15.118:0/511893998 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7fd00637c0 con 0x7f7fd007cd90 2024-03-24T00:05:12.232 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:05:10.089+0000 7f7fc7e00700 1 -- 172.21.15.118:0/511893998 <== mon.0 v2:172.21.15.118:3300/0 4 ==== mgrmap(e 12) v1 ==== 46119+0+0 (secure 0 0 0) 0x7f7fc0022020 con 0x7f7fd007cd90 2024-03-24T00:05:12.233 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:05:10.090+0000 7f7fc7e00700 1 --2- 172.21.15.118:0/511893998 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7f7fb4038fd0 0x7f7fb403b490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:05:12.233 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:05:10.090+0000 7f7fc7e00700 1 -- 172.21.15.118:0/511893998 <== mon.0 v2:172.21.15.118:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1155+0+0 (secure 0 0 0) 0x7f7fc004d4b0 con 0x7f7fd007cd90 2024-03-24T00:05:12.233 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:05:10.090+0000 7f7fcea00700 1 --2- 172.21.15.118:0/511893998 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7f7fb4038fd0 0x7f7fb403b490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:05:12.233 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:05:10.091+0000 7f7fcea00700 1 --2- 172.21.15.118:0/511893998 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7f7fb4038fd0 0x7f7fb403b490 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f7fbc006fd0 tx=0x7f7fbc006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:05:12.233 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:05:10.096+0000 7f7fc7e00700 1 -- 172.21.15.118:0/511893998 <== mon.0 v2:172.21.15.118:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f7fc000b3e0 con 0x7f7fd007cd90 2024-03-24T00:05:12.233 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:05:10.347+0000 7f7fd6000700 1 -- 172.21.15.118:0/511893998 --> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] -- mgr_command(tid 0: {"prefix": "dashboard ac-user-create", "username": "admin", "rolename": "administrator", "force_password": true, "pwd_update_required": true, "target": ["mon-mgr", ""]}) v1 -- 0x7f7fd01a1220 con 0x7f7fb4038fd0 2024-03-24T00:05:12.233 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:05:10.581+0000 7f7fc7e00700 1 -- 172.21.15.118:0/511893998 <== mgr.14152 v2:172.21.15.118:6800/2112912943 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+252 (secure 0 0 0) 0x7f7fd01a1220 con 0x7f7fb4038fd0 2024-03-24T00:05:12.233 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:05:10.584+0000 7f7fd6000700 1 -- 172.21.15.118:0/511893998 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7f7fb4038fd0 msgr2=0x7f7fb403b490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:05:12.233 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:05:10.584+0000 7f7fd6000700 1 --2- 172.21.15.118:0/511893998 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7f7fb4038fd0 0x7f7fb403b490 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f7fbc006fd0 tx=0x7f7fbc006e40 comp rx=0 tx=0).stop 2024-03-24T00:05:12.233 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:05:10.584+0000 7f7fd6000700 1 -- 172.21.15.118:0/511893998 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f7fd007cd90 msgr2=0x7f7fd019dc40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:05:12.233 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:05:10.584+0000 7f7fd6000700 1 --2- 172.21.15.118:0/511893998 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f7fd007cd90 0x7f7fd019dc40 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f7fc00117e0 tx=0x7f7fc0011ba0 comp rx=0 tx=0).stop 2024-03-24T00:05:12.233 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:05:10.585+0000 7f7fd6000700 1 -- 172.21.15.118:0/511893998 shutdown_connections 2024-03-24T00:05:12.233 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:05:10.585+0000 7f7fd6000700 1 --2- 172.21.15.118:0/511893998 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7f7fb4038fd0 0x7f7fb403b490 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:05:12.233 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:05:10.585+0000 7f7fd6000700 1 --2- 172.21.15.118:0/511893998 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f7fd007cd90 0x7f7fd019dc40 secure :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f7fc00117e0 tx=0x7f7fc0011ba0 comp rx=0 tx=0).stop 2024-03-24T00:05:12.234 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:05:10.585+0000 7f7fd6000700 1 -- 172.21.15.118:0/511893998 >> 172.21.15.118:0/511893998 conn(0x7f7fd0103080 msgr2=0x7f7fd0103b70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:05:12.234 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:05:10.586+0000 7f7fd6000700 1 -- 172.21.15.118:0/511893998 shutdown_connections 2024-03-24T00:05:12.234 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:05:10.586+0000 7f7fd6000700 1 -- 172.21.15.118:0/511893998 wait complete. 2024-03-24T00:05:12.234 INFO:teuthology.orchestra.run.smithi118.stdout:Fetching dashboard port number... 2024-03-24T00:05:12.842 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:05:12 smithi118 ceph-mon[31535]: mgrmap e13: x(active, since 6s) 2024-03-24T00:05:13.714 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:05:13 smithi118 ceph-mon[31535]: from='client.? 172.21.15.118:0/955555418' entity='client.admin' cmd=[{"prefix": "config get", "who": "mgr", "key": "mgr/dashboard/ssl_server_port"}]: dispatch 2024-03-24T00:05:13.732 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stdout 8443 2024-03-24T00:05:13.734 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:05:12.842+0000 7f857d600700 1 Processor -- start 2024-03-24T00:05:13.734 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:05:12.842+0000 7f857d600700 1 -- start start 2024-03-24T00:05:13.734 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:05:12.843+0000 7f857d600700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f857810e150 0x7f857810e570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:05:13.734 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:05:12.843+0000 7f857d600700 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f857810eb40 con 0x7f857810e150 2024-03-24T00:05:13.734 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:05:12.843+0000 7f8576a00700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f857810e150 0x7f857810e570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:05:13.734 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:05:12.843+0000 7f8576a00700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f857810e150 0x7f857810e570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.118:3300/0 says I am v2:172.21.15.118:60680/0 (socket says 172.21.15.118:60680) 2024-03-24T00:05:13.734 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:05:12.843+0000 7f8576a00700 1 -- 172.21.15.118:0/807580268 learned_addr learned my addr 172.21.15.118:0/807580268 (peer_addr_for_me v2:172.21.15.118:0/0) 2024-03-24T00:05:13.734 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:05:12.844+0000 7f8576a00700 1 -- 172.21.15.118:0/807580268 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f857810f350 con 0x7f857810e150 2024-03-24T00:05:13.734 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:05:12.844+0000 7f8576a00700 1 --2- 172.21.15.118:0/807580268 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f857810e150 0x7f857810e570 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f85680098a0 tx=0x7f8568009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=74b623d870ffe60b server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:05:13.734 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:05:12.844+0000 7f8576000700 1 -- 172.21.15.118:0/807580268 <== mon.0 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f8568004160 con 0x7f857810e150 2024-03-24T00:05:13.735 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:05:12.844+0000 7f8576000700 1 -- 172.21.15.118:0/807580268 <== mon.0 v2:172.21.15.118:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f8568004300 con 0x7f857810e150 2024-03-24T00:05:13.735 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:05:12.845+0000 7f8576000700 1 -- 172.21.15.118:0/807580268 <== mon.0 v2:172.21.15.118:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f8568011b20 con 0x7f857810e150 2024-03-24T00:05:13.735 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:05:12.845+0000 7f857d600700 1 -- 172.21.15.118:0/807580268 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f857810e150 msgr2=0x7f857810e570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:05:13.735 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:05:12.845+0000 7f857d600700 1 --2- 172.21.15.118:0/807580268 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f857810e150 0x7f857810e570 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f85680098a0 tx=0x7f8568009bb0 comp rx=0 tx=0).stop 2024-03-24T00:05:13.735 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:05:12.845+0000 7f857d600700 1 -- 172.21.15.118:0/807580268 shutdown_connections 2024-03-24T00:05:13.739 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:05:12.845+0000 7f857d600700 1 --2- 172.21.15.118:0/807580268 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f857810e150 0x7f857810e570 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:05:13.739 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:05:12.845+0000 7f857d600700 1 -- 172.21.15.118:0/807580268 >> 172.21.15.118:0/807580268 conn(0x7f857807d120 msgr2=0x7f857807f560 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:05:13.739 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:05:12.845+0000 7f857d600700 1 -- 172.21.15.118:0/807580268 shutdown_connections 2024-03-24T00:05:13.739 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:05:12.846+0000 7f857d600700 1 -- 172.21.15.118:0/807580268 wait complete. 2024-03-24T00:05:13.740 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:05:12.846+0000 7f857d600700 1 Processor -- start 2024-03-24T00:05:13.740 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:05:12.847+0000 7f857d600700 1 -- start start 2024-03-24T00:05:13.740 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:05:12.847+0000 7f857d600700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f857810e150 0x7f85781a2140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:05:13.740 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:05:12.847+0000 7f857d600700 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8578112700 con 0x7f857810e150 2024-03-24T00:05:13.740 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:05:12.847+0000 7f8576a00700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f857810e150 0x7f85781a2140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:05:13.740 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:05:12.847+0000 7f8576a00700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f857810e150 0x7f85781a2140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.118:3300/0 says I am v2:172.21.15.118:60688/0 (socket says 172.21.15.118:60688) 2024-03-24T00:05:13.740 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:05:12.847+0000 7f8576a00700 1 -- 172.21.15.118:0/955555418 learned_addr learned my addr 172.21.15.118:0/955555418 (peer_addr_for_me v2:172.21.15.118:0/0) 2024-03-24T00:05:13.740 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:05:12.848+0000 7f8576a00700 1 -- 172.21.15.118:0/955555418 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f85781a2680 con 0x7f857810e150 2024-03-24T00:05:13.740 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:05:12.848+0000 7f8576a00700 1 --2- 172.21.15.118:0/955555418 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f857810e150 0x7f85781a2140 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f8568000c00 tx=0x7f8568011c10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:05:13.741 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:05:12.848+0000 7f856f400700 1 -- 172.21.15.118:0/955555418 <== mon.0 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f8568003b60 con 0x7f857810e150 2024-03-24T00:05:13.741 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:05:12.848+0000 7f856f400700 1 -- 172.21.15.118:0/955555418 <== mon.0 v2:172.21.15.118:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f856801a910 con 0x7f857810e150 2024-03-24T00:05:13.741 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:05:12.848+0000 7f857d600700 1 -- 172.21.15.118:0/955555418 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f85781a2910 con 0x7f857810e150 2024-03-24T00:05:13.741 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:05:12.848+0000 7f856f400700 1 -- 172.21.15.118:0/955555418 <== mon.0 v2:172.21.15.118:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f8568024930 con 0x7f857810e150 2024-03-24T00:05:13.741 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:05:12.848+0000 7f857d600700 1 -- 172.21.15.118:0/955555418 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f85781a2d70 con 0x7f857810e150 2024-03-24T00:05:13.741 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:05:12.849+0000 7f856f400700 1 -- 172.21.15.118:0/955555418 <== mon.0 v2:172.21.15.118:3300/0 4 ==== mgrmap(e 13) v1 ==== 46163+0+0 (secure 0 0 0) 0x7f8568022020 con 0x7f857810e150 2024-03-24T00:05:13.741 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:05:12.849+0000 7f857d600700 1 -- 172.21.15.118:0/955555418 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f85780637c0 con 0x7f857810e150 2024-03-24T00:05:13.741 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:05:12.850+0000 7f856f400700 1 --2- 172.21.15.118:0/955555418 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7f855c039010 0x7f855c03b4d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:05:13.741 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:05:12.850+0000 7f856f400700 1 -- 172.21.15.118:0/955555418 <== mon.0 v2:172.21.15.118:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1155+0+0 (secure 0 0 0) 0x7f856804d340 con 0x7f857810e150 2024-03-24T00:05:13.742 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:05:12.850+0000 7f856fe00700 1 --2- 172.21.15.118:0/955555418 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7f855c039010 0x7f855c03b4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:05:13.742 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:05:12.851+0000 7f856fe00700 1 --2- 172.21.15.118:0/955555418 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7f855c039010 0x7f855c03b4d0 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f8560006fd0 tx=0x7f8560006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:05:13.742 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:05:12.855+0000 7f856f400700 1 -- 172.21.15.118:0/955555418 <== mon.0 v2:172.21.15.118:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f856801ab70 con 0x7f857810e150 2024-03-24T00:05:13.742 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:05:13.077+0000 7f857d600700 1 -- 172.21.15.118:0/955555418 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_command({"prefix": "config get", "who": "mgr", "key": "mgr/dashboard/ssl_server_port"} v 0) v1 -- 0x7f85781a5740 con 0x7f857810e150 2024-03-24T00:05:13.742 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:05:13.078+0000 7f856f400700 1 -- 172.21.15.118:0/955555418 <== mon.0 v2:172.21.15.118:3300/0 7 ==== mon_command_ack([{"prefix": "config get", "who": "mgr", "key": "mgr/dashboard/ssl_server_port"}]=0 v8) v1 ==== 112+0+5 (secure 0 0 0) 0x7f856802d3f0 con 0x7f857810e150 2024-03-24T00:05:13.742 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:05:13.081+0000 7f857d600700 1 -- 172.21.15.118:0/955555418 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7f855c039010 msgr2=0x7f855c03b4d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:05:13.742 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:05:13.081+0000 7f857d600700 1 --2- 172.21.15.118:0/955555418 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7f855c039010 0x7f855c03b4d0 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f8560006fd0 tx=0x7f8560006e40 comp rx=0 tx=0).stop 2024-03-24T00:05:13.742 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:05:13.081+0000 7f857d600700 1 -- 172.21.15.118:0/955555418 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f857810e150 msgr2=0x7f85781a2140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:05:13.742 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:05:13.081+0000 7f857d600700 1 --2- 172.21.15.118:0/955555418 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f857810e150 0x7f85781a2140 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f8568000c00 tx=0x7f8568011c10 comp rx=0 tx=0).stop 2024-03-24T00:05:13.743 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:05:13.081+0000 7f857d600700 1 -- 172.21.15.118:0/955555418 shutdown_connections 2024-03-24T00:05:13.743 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:05:13.081+0000 7f857d600700 1 --2- 172.21.15.118:0/955555418 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7f855c039010 0x7f855c03b4d0 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:05:13.743 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:05:13.081+0000 7f857d600700 1 --2- 172.21.15.118:0/955555418 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f857810e150 0x7f85781a2140 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:05:13.743 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:05:13.081+0000 7f857d600700 1 -- 172.21.15.118:0/955555418 >> 172.21.15.118:0/955555418 conn(0x7f857807d120 msgr2=0x7f857807dc80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:05:13.743 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:05:13.081+0000 7f857d600700 1 -- 172.21.15.118:0/955555418 shutdown_connections 2024-03-24T00:05:13.743 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:05:13.081+0000 7f857d600700 1 -- 172.21.15.118:0/955555418 wait complete. 2024-03-24T00:05:13.773 INFO:teuthology.orchestra.run.smithi118.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-03-24T00:05:13.773 INFO:teuthology.orchestra.run.smithi118.stdout:systemctl: stdout disabled 2024-03-24T00:05:13.796 INFO:teuthology.orchestra.run.smithi118.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-03-24T00:05:13.796 INFO:teuthology.orchestra.run.smithi118.stdout:systemctl: stdout inactive 2024-03-24T00:05:13.796 INFO:teuthology.orchestra.run.smithi118.stdout:firewalld.service is not enabled 2024-03-24T00:05:13.796 INFO:teuthology.orchestra.run.smithi118.stdout:Not possible to open ports <[8443]>. firewalld.service is not available 2024-03-24T00:05:13.803 INFO:teuthology.orchestra.run.smithi118.stdout:Ceph Dashboard is now available at: 2024-03-24T00:05:13.803 INFO:teuthology.orchestra.run.smithi118.stdout: 2024-03-24T00:05:13.803 INFO:teuthology.orchestra.run.smithi118.stdout: URL: https://smithi118.front.sepia.ceph.com:8443/ 2024-03-24T00:05:13.803 INFO:teuthology.orchestra.run.smithi118.stdout: User: admin 2024-03-24T00:05:13.804 INFO:teuthology.orchestra.run.smithi118.stdout: Password: l1x8r3lq6p 2024-03-24T00:05:13.804 INFO:teuthology.orchestra.run.smithi118.stdout: 2024-03-24T00:05:13.804 INFO:teuthology.orchestra.run.smithi118.stdout:Saving cluster configuration to /var/lib/ceph/d79a5420-e971-11ee-95cd-87774f69a715/config directory 2024-03-24T00:05:13.806 INFO:teuthology.orchestra.run.smithi118.stdout:Enabling autotune for osd_memory_target 2024-03-24T00:05:15.354 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:05:14.460+0000 7fe48a200700 1 Processor -- start 2024-03-24T00:05:15.354 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:05:14.461+0000 7fe48a200700 1 -- start start 2024-03-24T00:05:15.354 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:05:14.461+0000 7fe48a200700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fe484106290 0x7fe4841066b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:05:15.354 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:05:14.461+0000 7fe48a200700 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe48407be00 con 0x7fe484106290 2024-03-24T00:05:15.354 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:05:14.462+0000 7fe488e00700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fe484106290 0x7fe4841066b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:05:15.355 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:05:14.462+0000 7fe488e00700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fe484106290 0x7fe4841066b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.118:3300/0 says I am v2:172.21.15.118:57256/0 (socket says 172.21.15.118:57256) 2024-03-24T00:05:15.355 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:05:14.462+0000 7fe488e00700 1 -- 172.21.15.118:0/552843496 learned_addr learned my addr 172.21.15.118:0/552843496 (peer_addr_for_me v2:172.21.15.118:0/0) 2024-03-24T00:05:15.355 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:05:14.462+0000 7fe488e00700 1 -- 172.21.15.118:0/552843496 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe484106bf0 con 0x7fe484106290 2024-03-24T00:05:15.355 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:05:14.462+0000 7fe488e00700 1 --2- 172.21.15.118:0/552843496 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fe484106290 0x7fe4841066b0 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7fe46c0098a0 tx=0x7fe46c009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f2586579497ecfb server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:05:15.355 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:05:14.462+0000 7fe483e00700 1 -- 172.21.15.118:0/552843496 <== mon.0 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fe46c004160 con 0x7fe484106290 2024-03-24T00:05:15.355 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:05:14.463+0000 7fe483e00700 1 -- 172.21.15.118:0/552843496 <== mon.0 v2:172.21.15.118:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fe46c004300 con 0x7fe484106290 2024-03-24T00:05:15.355 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:05:14.463+0000 7fe483e00700 1 -- 172.21.15.118:0/552843496 <== mon.0 v2:172.21.15.118:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fe46c011b20 con 0x7fe484106290 2024-03-24T00:05:15.355 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:05:14.463+0000 7fe48a200700 1 -- 172.21.15.118:0/552843496 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fe484106290 msgr2=0x7fe4841066b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:05:15.355 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:05:14.463+0000 7fe48a200700 1 --2- 172.21.15.118:0/552843496 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fe484106290 0x7fe4841066b0 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7fe46c0098a0 tx=0x7fe46c009bb0 comp rx=0 tx=0).stop 2024-03-24T00:05:15.355 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:05:14.463+0000 7fe48a200700 1 -- 172.21.15.118:0/552843496 shutdown_connections 2024-03-24T00:05:15.355 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:05:14.463+0000 7fe48a200700 1 --2- 172.21.15.118:0/552843496 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fe484106290 0x7fe4841066b0 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:05:15.355 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:05:14.463+0000 7fe48a200700 1 -- 172.21.15.118:0/552843496 >> 172.21.15.118:0/552843496 conn(0x7fe484101e90 msgr2=0x7fe4841042b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:05:15.356 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:05:14.463+0000 7fe48a200700 1 -- 172.21.15.118:0/552843496 shutdown_connections 2024-03-24T00:05:15.356 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:05:14.463+0000 7fe48a200700 1 -- 172.21.15.118:0/552843496 wait complete. 2024-03-24T00:05:15.356 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:05:14.464+0000 7fe48a200700 1 Processor -- start 2024-03-24T00:05:15.356 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:05:14.465+0000 7fe48a200700 1 -- start start 2024-03-24T00:05:15.356 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:05:14.465+0000 7fe48a200700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fe484106290 0x7fe48419daf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:05:15.356 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:05:14.465+0000 7fe48a200700 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe484107b40 con 0x7fe484106290 2024-03-24T00:05:15.356 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:05:14.465+0000 7fe488e00700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fe484106290 0x7fe48419daf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:05:15.356 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:05:14.465+0000 7fe488e00700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fe484106290 0x7fe48419daf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.118:3300/0 says I am v2:172.21.15.118:57266/0 (socket says 172.21.15.118:57266) 2024-03-24T00:05:15.356 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:05:14.465+0000 7fe488e00700 1 -- 172.21.15.118:0/253533346 learned_addr learned my addr 172.21.15.118:0/253533346 (peer_addr_for_me v2:172.21.15.118:0/0) 2024-03-24T00:05:15.356 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:05:14.466+0000 7fe488e00700 1 -- 172.21.15.118:0/253533346 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe48419e030 con 0x7fe484106290 2024-03-24T00:05:15.356 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:05:14.466+0000 7fe488e00700 1 --2- 172.21.15.118:0/253533346 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fe484106290 0x7fe48419daf0 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7fe46c011b60 tx=0x7fe46c011c40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:05:15.356 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:05:14.466+0000 7fe482000700 1 -- 172.21.15.118:0/253533346 <== mon.0 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fe46c003b60 con 0x7fe484106290 2024-03-24T00:05:15.357 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:05:14.466+0000 7fe482000700 1 -- 172.21.15.118:0/253533346 <== mon.0 v2:172.21.15.118:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fe46c01a910 con 0x7fe484106290 2024-03-24T00:05:15.357 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:05:14.466+0000 7fe482000700 1 -- 172.21.15.118:0/253533346 <== mon.0 v2:172.21.15.118:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fe46c024930 con 0x7fe484106290 2024-03-24T00:05:15.357 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:05:14.466+0000 7fe48a200700 1 -- 172.21.15.118:0/253533346 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe48419e2c0 con 0x7fe484106290 2024-03-24T00:05:15.357 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:05:14.466+0000 7fe48a200700 1 -- 172.21.15.118:0/253533346 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe48419e720 con 0x7fe484106290 2024-03-24T00:05:15.357 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:05:14.467+0000 7fe482000700 1 -- 172.21.15.118:0/253533346 <== mon.0 v2:172.21.15.118:3300/0 4 ==== mgrmap(e 13) v1 ==== 46163+0+0 (secure 0 0 0) 0x7fe46c022020 con 0x7fe484106290 2024-03-24T00:05:15.357 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:05:14.467+0000 7fe48a200700 1 -- 172.21.15.118:0/253533346 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe4840637c0 con 0x7fe484106290 2024-03-24T00:05:15.357 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:05:14.468+0000 7fe482000700 1 --2- 172.21.15.118:0/253533346 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7fe464039070 0x7fe46403b530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:05:15.357 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:05:14.468+0000 7fe482000700 1 -- 172.21.15.118:0/253533346 <== mon.0 v2:172.21.15.118:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1155+0+0 (secure 0 0 0) 0x7fe46c04d2c0 con 0x7fe484106290 2024-03-24T00:05:15.357 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:05:14.468+0000 7fe47be00700 1 --2- 172.21.15.118:0/253533346 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7fe464039070 0x7fe46403b530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:05:15.357 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:05:14.469+0000 7fe47be00700 1 --2- 172.21.15.118:0/253533346 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7fe464039070 0x7fe46403b530 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7fe474006fd0 tx=0x7fe474006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:05:15.357 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:05:14.473+0000 7fe482000700 1 -- 172.21.15.118:0/253533346 <== mon.0 v2:172.21.15.118:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7fe46c00b3e0 con 0x7fe484106290 2024-03-24T00:05:15.357 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:05:14.696+0000 7fe48a200700 1 -- 172.21.15.118:0/253533346 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_command([{prefix=config set, name=osd_memory_target_autotune}] v 0) v1 -- 0x7fe484062da0 con 0x7fe484106290 2024-03-24T00:05:15.357 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:05:14.697+0000 7fe482000700 1 -- 172.21.15.118:0/253533346 <== mon.0 v2:172.21.15.118:3300/0 7 ==== mon_command_ack([{prefix=config set, name=osd_memory_target_autotune}]=0 v8)=0 v8) v1 ==== 127+0+0 (secure 0 0 0) 0x7fe46c02dde0 con 0x7fe484106290 2024-03-24T00:05:15.357 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:05:14.700+0000 7fe48a200700 1 -- 172.21.15.118:0/253533346 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7fe464039070 msgr2=0x7fe46403b530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:05:15.358 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:05:14.700+0000 7fe48a200700 1 --2- 172.21.15.118:0/253533346 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7fe464039070 0x7fe46403b530 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7fe474006fd0 tx=0x7fe474006e40 comp rx=0 tx=0).stop 2024-03-24T00:05:15.358 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:05:14.700+0000 7fe48a200700 1 -- 172.21.15.118:0/253533346 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fe484106290 msgr2=0x7fe48419daf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:05:15.358 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:05:14.700+0000 7fe48a200700 1 --2- 172.21.15.118:0/253533346 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fe484106290 0x7fe48419daf0 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7fe46c011b60 tx=0x7fe46c011c40 comp rx=0 tx=0).stop 2024-03-24T00:05:15.358 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:05:14.700+0000 7fe48a200700 1 -- 172.21.15.118:0/253533346 shutdown_connections 2024-03-24T00:05:15.358 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:05:14.700+0000 7fe48a200700 1 --2- 172.21.15.118:0/253533346 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7fe464039070 0x7fe46403b530 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:05:15.358 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:05:14.700+0000 7fe48a200700 1 --2- 172.21.15.118:0/253533346 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fe484106290 0x7fe48419daf0 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:05:15.358 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:05:14.700+0000 7fe48a200700 1 -- 172.21.15.118:0/253533346 >> 172.21.15.118:0/253533346 conn(0x7fe484101e90 msgr2=0x7fe484194750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:05:15.358 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:05:14.700+0000 7fe48a200700 1 -- 172.21.15.118:0/253533346 shutdown_connections 2024-03-24T00:05:15.358 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:05:14.700+0000 7fe48a200700 1 -- 172.21.15.118:0/253533346 wait complete. 2024-03-24T00:05:16.993 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:05:15.957+0000 7fca67400700 1 Processor -- start 2024-03-24T00:05:16.994 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:05:15.957+0000 7fca67400700 1 -- start start 2024-03-24T00:05:16.994 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:05:15.958+0000 7fca67400700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fca601069f0 0x7fca60106e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:05:16.994 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:05:15.958+0000 7fca67400700 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fca601073e0 con 0x7fca601069f0 2024-03-24T00:05:16.994 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:05:15.958+0000 7fca64a00700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fca601069f0 0x7fca60106e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:05:16.995 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:05:15.958+0000 7fca64a00700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fca601069f0 0x7fca60106e10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.118:3300/0 says I am v2:172.21.15.118:57276/0 (socket says 172.21.15.118:57276) 2024-03-24T00:05:16.995 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:05:15.958+0000 7fca64a00700 1 -- 172.21.15.118:0/518353143 learned_addr learned my addr 172.21.15.118:0/518353143 (peer_addr_for_me v2:172.21.15.118:0/0) 2024-03-24T00:05:16.996 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:05:15.959+0000 7fca64a00700 1 -- 172.21.15.118:0/518353143 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fca60107bf0 con 0x7fca601069f0 2024-03-24T00:05:16.996 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:05:15.959+0000 7fca64a00700 1 --2- 172.21.15.118:0/518353143 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fca601069f0 0x7fca60106e10 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7fca540098a0 tx=0x7fca54009bb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=11567830427fe2f5 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:05:16.996 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:05:15.959+0000 7fca5f400700 1 -- 172.21.15.118:0/518353143 <== mon.0 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fca54004160 con 0x7fca601069f0 2024-03-24T00:05:16.996 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:05:15.959+0000 7fca5f400700 1 -- 172.21.15.118:0/518353143 <== mon.0 v2:172.21.15.118:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fca54004300 con 0x7fca601069f0 2024-03-24T00:05:16.996 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:05:15.959+0000 7fca5f400700 1 -- 172.21.15.118:0/518353143 <== mon.0 v2:172.21.15.118:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fca54011b20 con 0x7fca601069f0 2024-03-24T00:05:16.996 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:05:15.960+0000 7fca67400700 1 -- 172.21.15.118:0/518353143 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fca601069f0 msgr2=0x7fca60106e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:05:16.996 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:05:15.960+0000 7fca67400700 1 --2- 172.21.15.118:0/518353143 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fca601069f0 0x7fca60106e10 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7fca540098a0 tx=0x7fca54009bb0 comp rx=0 tx=0).stop 2024-03-24T00:05:16.996 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:05:15.960+0000 7fca67400700 1 -- 172.21.15.118:0/518353143 shutdown_connections 2024-03-24T00:05:16.996 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:05:15.960+0000 7fca67400700 1 --2- 172.21.15.118:0/518353143 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fca601069f0 0x7fca60106e10 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:05:16.998 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:05:15.960+0000 7fca67400700 1 -- 172.21.15.118:0/518353143 >> 172.21.15.118:0/518353143 conn(0x7fca60101fb0 msgr2=0x7fca601043d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:05:16.998 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:05:15.960+0000 7fca67400700 1 -- 172.21.15.118:0/518353143 shutdown_connections 2024-03-24T00:05:16.998 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:05:15.960+0000 7fca67400700 1 -- 172.21.15.118:0/518353143 wait complete. 2024-03-24T00:05:16.998 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:05:15.962+0000 7fca67400700 1 Processor -- start 2024-03-24T00:05:16.999 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:05:15.962+0000 7fca67400700 1 -- start start 2024-03-24T00:05:16.999 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:05:15.962+0000 7fca67400700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fca601069f0 0x7fca6019dd40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:05:16.999 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:05:15.962+0000 7fca67400700 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fca6010afa0 con 0x7fca601069f0 2024-03-24T00:05:16.999 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:05:15.963+0000 7fca64a00700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fca601069f0 0x7fca6019dd40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:05:16.999 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:05:15.963+0000 7fca64a00700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fca601069f0 0x7fca6019dd40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.118:3300/0 says I am v2:172.21.15.118:57286/0 (socket says 172.21.15.118:57286) 2024-03-24T00:05:16.999 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:05:15.963+0000 7fca64a00700 1 -- 172.21.15.118:0/1871999755 learned_addr learned my addr 172.21.15.118:0/1871999755 (peer_addr_for_me v2:172.21.15.118:0/0) 2024-03-24T00:05:16.999 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:05:15.963+0000 7fca64a00700 1 -- 172.21.15.118:0/1871999755 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fca6019e280 con 0x7fca601069f0 2024-03-24T00:05:16.999 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:05:15.963+0000 7fca64a00700 1 --2- 172.21.15.118:0/1871999755 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fca601069f0 0x7fca6019dd40 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7fca54000c00 tx=0x7fca54011c10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:05:16.999 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:05:15.964+0000 7fca5d600700 1 -- 172.21.15.118:0/1871999755 <== mon.0 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fca54003b60 con 0x7fca601069f0 2024-03-24T00:05:16.999 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:05:15.964+0000 7fca5d600700 1 -- 172.21.15.118:0/1871999755 <== mon.0 v2:172.21.15.118:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fca5401a910 con 0x7fca601069f0 2024-03-24T00:05:16.999 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:05:15.964+0000 7fca67400700 1 -- 172.21.15.118:0/1871999755 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fca6019e510 con 0x7fca601069f0 2024-03-24T00:05:16.999 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:05:15.964+0000 7fca5d600700 1 -- 172.21.15.118:0/1871999755 <== mon.0 v2:172.21.15.118:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fca54024930 con 0x7fca601069f0 2024-03-24T00:05:17.002 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:05:15.964+0000 7fca67400700 1 -- 172.21.15.118:0/1871999755 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fca6019e970 con 0x7fca601069f0 2024-03-24T00:05:17.002 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:05:15.965+0000 7fca5d600700 1 -- 172.21.15.118:0/1871999755 <== mon.0 v2:172.21.15.118:3300/0 4 ==== mgrmap(e 13) v1 ==== 46163+0+0 (secure 0 0 0) 0x7fca54022020 con 0x7fca601069f0 2024-03-24T00:05:17.002 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:05:15.965+0000 7fca67400700 1 -- 172.21.15.118:0/1871999755 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fca600637c0 con 0x7fca601069f0 2024-03-24T00:05:17.002 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:05:15.966+0000 7fca5d600700 1 --2- 172.21.15.118:0/1871999755 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7fca48039010 0x7fca4803b4d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:05:17.002 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:05:15.966+0000 7fca5d600700 1 -- 172.21.15.118:0/1871999755 <== mon.0 v2:172.21.15.118:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1155+0+0 (secure 0 0 0) 0x7fca5404d280 con 0x7fca601069f0 2024-03-24T00:05:17.002 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:05:15.966+0000 7fca5fe00700 1 --2- 172.21.15.118:0/1871999755 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7fca48039010 0x7fca4803b4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:05:17.002 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:05:15.967+0000 7fca5fe00700 1 --2- 172.21.15.118:0/1871999755 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7fca48039010 0x7fca4803b4d0 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7fca4c006fd0 tx=0x7fca4c006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:05:17.002 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:05:15.972+0000 7fca5d600700 1 -- 172.21.15.118:0/1871999755 <== mon.0 v2:172.21.15.118:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7fca5401ab70 con 0x7fca601069f0 2024-03-24T00:05:17.002 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:05:16.309+0000 7fca67400700 1 -- 172.21.15.118:0/1871999755 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_command([{prefix=config-key set, key=mgr/dashboard/cluster/status}] v 0) v1 -- 0x7fca60062da0 con 0x7fca601069f0 2024-03-24T00:05:17.002 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:05:16.314+0000 7fca5d600700 1 -- 172.21.15.118:0/1871999755 <== mon.0 v2:172.21.15.118:3300/0 7 ==== mon_command_ack([{prefix=config-key set, key=mgr/dashboard/cluster/status}]=0 set mgr/dashboard/cluster/status v28)=0 set mgr/dashboard/cluster/status v28) v1 ==== 153+0+0 (secure 0 0 0) 0x7fca5402d3f0 con 0x7fca601069f0 2024-03-24T00:05:17.002 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:05:16.317+0000 7fca67400700 1 -- 172.21.15.118:0/1871999755 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7fca48039010 msgr2=0x7fca4803b4d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:05:17.003 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:05:16.317+0000 7fca67400700 1 --2- 172.21.15.118:0/1871999755 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7fca48039010 0x7fca4803b4d0 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7fca4c006fd0 tx=0x7fca4c006e40 comp rx=0 tx=0).stop 2024-03-24T00:05:17.003 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:05:16.317+0000 7fca67400700 1 -- 172.21.15.118:0/1871999755 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fca601069f0 msgr2=0x7fca6019dd40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:05:17.003 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:05:16.317+0000 7fca67400700 1 --2- 172.21.15.118:0/1871999755 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fca601069f0 0x7fca6019dd40 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7fca54000c00 tx=0x7fca54011c10 comp rx=0 tx=0).stop 2024-03-24T00:05:17.003 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:05:16.317+0000 7fca67400700 1 -- 172.21.15.118:0/1871999755 shutdown_connections 2024-03-24T00:05:17.003 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:05:16.317+0000 7fca67400700 1 --2- 172.21.15.118:0/1871999755 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7fca48039010 0x7fca4803b4d0 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:05:17.003 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:05:16.317+0000 7fca67400700 1 --2- 172.21.15.118:0/1871999755 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fca601069f0 0x7fca6019dd40 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:05:17.003 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:05:16.317+0000 7fca67400700 1 -- 172.21.15.118:0/1871999755 >> 172.21.15.118:0/1871999755 conn(0x7fca60101fb0 msgr2=0x7fca60102e80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:05:17.003 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:05:16.317+0000 7fca67400700 1 -- 172.21.15.118:0/1871999755 shutdown_connections 2024-03-24T00:05:17.003 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr 2024-03-24T00:05:16.318+0000 7fca67400700 1 -- 172.21.15.118:0/1871999755 wait complete. 2024-03-24T00:05:17.003 INFO:teuthology.orchestra.run.smithi118.stdout:/usr/bin/ceph: stderr set mgr/dashboard/cluster/status 2024-03-24T00:05:17.003 INFO:teuthology.orchestra.run.smithi118.stdout:You can access the Ceph CLI as following in case of multi-cluster or non-default config: 2024-03-24T00:05:17.003 INFO:teuthology.orchestra.run.smithi118.stdout: 2024-03-24T00:05:17.003 INFO:teuthology.orchestra.run.smithi118.stdout: sudo /home/ubuntu/cephtest/cephadm shell --fsid d79a5420-e971-11ee-95cd-87774f69a715 -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring 2024-03-24T00:05:17.003 INFO:teuthology.orchestra.run.smithi118.stdout: 2024-03-24T00:05:17.004 INFO:teuthology.orchestra.run.smithi118.stdout:Or, if you are only running a single cluster on this host: 2024-03-24T00:05:17.004 INFO:teuthology.orchestra.run.smithi118.stdout: 2024-03-24T00:05:17.004 INFO:teuthology.orchestra.run.smithi118.stdout: sudo /home/ubuntu/cephtest/cephadm shell 2024-03-24T00:05:17.004 INFO:teuthology.orchestra.run.smithi118.stdout: 2024-03-24T00:05:17.004 INFO:teuthology.orchestra.run.smithi118.stdout:Please consider enabling telemetry to help improve Ceph: 2024-03-24T00:05:17.004 INFO:teuthology.orchestra.run.smithi118.stdout: 2024-03-24T00:05:17.005 INFO:teuthology.orchestra.run.smithi118.stdout: ceph telemetry on 2024-03-24T00:05:17.005 INFO:teuthology.orchestra.run.smithi118.stdout: 2024-03-24T00:05:17.005 INFO:teuthology.orchestra.run.smithi118.stdout:For more information see: 2024-03-24T00:05:17.005 INFO:teuthology.orchestra.run.smithi118.stdout: 2024-03-24T00:05:17.005 INFO:teuthology.orchestra.run.smithi118.stdout: https://docs.ceph.com/en/latest/mgr/telemetry/ 2024-03-24T00:05:17.005 INFO:teuthology.orchestra.run.smithi118.stdout: 2024-03-24T00:05:17.005 INFO:teuthology.orchestra.run.smithi118.stdout:Bootstrap complete. 2024-03-24T00:05:17.119 INFO:tasks.cephadm:Fetching config... 2024-03-24T00:05:17.120 DEBUG:teuthology.orchestra.run.smithi118:> set -ex 2024-03-24T00:05:17.120 DEBUG:teuthology.orchestra.run.smithi118:> dd if=/etc/ceph/ceph.conf of=/dev/stdout 2024-03-24T00:05:17.167 INFO:tasks.cephadm:Fetching client.admin keyring... 2024-03-24T00:05:17.167 DEBUG:teuthology.orchestra.run.smithi118:> set -ex 2024-03-24T00:05:17.167 DEBUG:teuthology.orchestra.run.smithi118:> dd if=/etc/ceph/ceph.client.admin.keyring of=/dev/stdout 2024-03-24T00:05:17.247 INFO:tasks.cephadm:Fetching mon keyring... 2024-03-24T00:05:17.247 DEBUG:teuthology.orchestra.run.smithi118:> set -ex 2024-03-24T00:05:17.247 DEBUG:teuthology.orchestra.run.smithi118:> sudo dd if=/var/lib/ceph/d79a5420-e971-11ee-95cd-87774f69a715/mon.a/keyring of=/dev/stdout 2024-03-24T00:05:17.378 INFO:tasks.cephadm:Fetching pub ssh key... 2024-03-24T00:05:17.378 DEBUG:teuthology.orchestra.run.smithi118:> set -ex 2024-03-24T00:05:17.378 DEBUG:teuthology.orchestra.run.smithi118:> dd if=/home/ubuntu/cephtest/ceph.pub of=/dev/stdout 2024-03-24T00:05:17.425 INFO:tasks.cephadm:Installing pub ssh key for root users... 2024-03-24T00:05:17.425 DEBUG:teuthology.orchestra.run.smithi118:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC0pkhYZJq78D9oX9I+RR9b6h4To50g4oxVXM9eLBnZXRbywjN22+kPS1kpmUgkHrAoHTlvADEKSFilDnEvvDC4h6ZKmju/6ACXSZm+tHgr3yYiwVxURvZvLNE/+a1hQ72eJdjwQzEbBkNwIfaKG/PPovP57yD8ZuEpbLTA6F84fOeFrfdjiz/ck6W7PYMrIHVyea28UYDadX8lCE+sPNjcgkTshWHVDFt6CgZcylGYch8Bx8AOui0VAQKYSjpcQHQa+gJfzPgqlRnoe1jWD2PyayCzR0y0UzUGgpgbIEbHBN3Pus+NX41Y8HZPv2qBu/4wD/5ZkHmN2uu8qeg8zhOnudz6mrUXSuPL8QiafgIdVfSdiznUZ61nqyNuukzCIRgtDJVz4eXwXeCZ084FHIpJwy18wdg7j0qUSIKRTEvMFjeQ6Y/jnpWnuH5BqHKJf0DF18w7BPVk1w5wvavscB2XFZ3LYmYieH3aTpdq65Rxxqyec0+BvDgG+WS62/fmhLk= ceph-d79a5420-e971-11ee-95cd-87774f69a715' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2024-03-24T00:05:17.564 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:05:17 smithi118 ceph-mon[31535]: from='client.? 172.21.15.118:0/1871999755' entity='client.admin' 2024-03-24T00:05:17.581 INFO:teuthology.orchestra.run.smithi118.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC0pkhYZJq78D9oX9I+RR9b6h4To50g4oxVXM9eLBnZXRbywjN22+kPS1kpmUgkHrAoHTlvADEKSFilDnEvvDC4h6ZKmju/6ACXSZm+tHgr3yYiwVxURvZvLNE/+a1hQ72eJdjwQzEbBkNwIfaKG/PPovP57yD8ZuEpbLTA6F84fOeFrfdjiz/ck6W7PYMrIHVyea28UYDadX8lCE+sPNjcgkTshWHVDFt6CgZcylGYch8Bx8AOui0VAQKYSjpcQHQa+gJfzPgqlRnoe1jWD2PyayCzR0y0UzUGgpgbIEbHBN3Pus+NX41Y8HZPv2qBu/4wD/5ZkHmN2uu8qeg8zhOnudz6mrUXSuPL8QiafgIdVfSdiznUZ61nqyNuukzCIRgtDJVz4eXwXeCZ084FHIpJwy18wdg7j0qUSIKRTEvMFjeQ6Y/jnpWnuH5BqHKJf0DF18w7BPVk1w5wvavscB2XFZ3LYmYieH3aTpdq65Rxxqyec0+BvDgG+WS62/fmhLk= ceph-d79a5420-e971-11ee-95cd-87774f69a715 2024-03-24T00:05:17.648 DEBUG:teuthology.orchestra.run.smithi189:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC0pkhYZJq78D9oX9I+RR9b6h4To50g4oxVXM9eLBnZXRbywjN22+kPS1kpmUgkHrAoHTlvADEKSFilDnEvvDC4h6ZKmju/6ACXSZm+tHgr3yYiwVxURvZvLNE/+a1hQ72eJdjwQzEbBkNwIfaKG/PPovP57yD8ZuEpbLTA6F84fOeFrfdjiz/ck6W7PYMrIHVyea28UYDadX8lCE+sPNjcgkTshWHVDFt6CgZcylGYch8Bx8AOui0VAQKYSjpcQHQa+gJfzPgqlRnoe1jWD2PyayCzR0y0UzUGgpgbIEbHBN3Pus+NX41Y8HZPv2qBu/4wD/5ZkHmN2uu8qeg8zhOnudz6mrUXSuPL8QiafgIdVfSdiznUZ61nqyNuukzCIRgtDJVz4eXwXeCZ084FHIpJwy18wdg7j0qUSIKRTEvMFjeQ6Y/jnpWnuH5BqHKJf0DF18w7BPVk1w5wvavscB2XFZ3LYmYieH3aTpdq65Rxxqyec0+BvDgG+WS62/fmhLk= ceph-d79a5420-e971-11ee-95cd-87774f69a715' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2024-03-24T00:05:17.696 INFO:teuthology.orchestra.run.smithi189.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC0pkhYZJq78D9oX9I+RR9b6h4To50g4oxVXM9eLBnZXRbywjN22+kPS1kpmUgkHrAoHTlvADEKSFilDnEvvDC4h6ZKmju/6ACXSZm+tHgr3yYiwVxURvZvLNE/+a1hQ72eJdjwQzEbBkNwIfaKG/PPovP57yD8ZuEpbLTA6F84fOeFrfdjiz/ck6W7PYMrIHVyea28UYDadX8lCE+sPNjcgkTshWHVDFt6CgZcylGYch8Bx8AOui0VAQKYSjpcQHQa+gJfzPgqlRnoe1jWD2PyayCzR0y0UzUGgpgbIEbHBN3Pus+NX41Y8HZPv2qBu/4wD/5ZkHmN2uu8qeg8zhOnudz6mrUXSuPL8QiafgIdVfSdiznUZ61nqyNuukzCIRgtDJVz4eXwXeCZ084FHIpJwy18wdg7j0qUSIKRTEvMFjeQ6Y/jnpWnuH5BqHKJf0DF18w7BPVk1w5wvavscB2XFZ3LYmYieH3aTpdq65Rxxqyec0+BvDgG+WS62/fmhLk= ceph-d79a5420-e971-11ee-95cd-87774f69a715 2024-03-24T00:05:17.712 DEBUG:teuthology.orchestra.run.smithi193:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC0pkhYZJq78D9oX9I+RR9b6h4To50g4oxVXM9eLBnZXRbywjN22+kPS1kpmUgkHrAoHTlvADEKSFilDnEvvDC4h6ZKmju/6ACXSZm+tHgr3yYiwVxURvZvLNE/+a1hQ72eJdjwQzEbBkNwIfaKG/PPovP57yD8ZuEpbLTA6F84fOeFrfdjiz/ck6W7PYMrIHVyea28UYDadX8lCE+sPNjcgkTshWHVDFt6CgZcylGYch8Bx8AOui0VAQKYSjpcQHQa+gJfzPgqlRnoe1jWD2PyayCzR0y0UzUGgpgbIEbHBN3Pus+NX41Y8HZPv2qBu/4wD/5ZkHmN2uu8qeg8zhOnudz6mrUXSuPL8QiafgIdVfSdiznUZ61nqyNuukzCIRgtDJVz4eXwXeCZ084FHIpJwy18wdg7j0qUSIKRTEvMFjeQ6Y/jnpWnuH5BqHKJf0DF18w7BPVk1w5wvavscB2XFZ3LYmYieH3aTpdq65Rxxqyec0+BvDgG+WS62/fmhLk= ceph-d79a5420-e971-11ee-95cd-87774f69a715' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2024-03-24T00:05:17.759 INFO:teuthology.orchestra.run.smithi193.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC0pkhYZJq78D9oX9I+RR9b6h4To50g4oxVXM9eLBnZXRbywjN22+kPS1kpmUgkHrAoHTlvADEKSFilDnEvvDC4h6ZKmju/6ACXSZm+tHgr3yYiwVxURvZvLNE/+a1hQ72eJdjwQzEbBkNwIfaKG/PPovP57yD8ZuEpbLTA6F84fOeFrfdjiz/ck6W7PYMrIHVyea28UYDadX8lCE+sPNjcgkTshWHVDFt6CgZcylGYch8Bx8AOui0VAQKYSjpcQHQa+gJfzPgqlRnoe1jWD2PyayCzR0y0UzUGgpgbIEbHBN3Pus+NX41Y8HZPv2qBu/4wD/5ZkHmN2uu8qeg8zhOnudz6mrUXSuPL8QiafgIdVfSdiznUZ61nqyNuukzCIRgtDJVz4eXwXeCZ084FHIpJwy18wdg7j0qUSIKRTEvMFjeQ6Y/jnpWnuH5BqHKJf0DF18w7BPVk1w5wvavscB2XFZ3LYmYieH3aTpdq65Rxxqyec0+BvDgG+WS62/fmhLk= ceph-d79a5420-e971-11ee-95cd-87774f69a715 2024-03-24T00:05:17.774 DEBUG:teuthology.orchestra.run.smithi118:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d79a5420-e971-11ee-95cd-87774f69a715 -- ceph config set mgr mgr/cephadm/allow_ptrace true 2024-03-24T00:05:18.186 INFO:teuthology.orchestra.run.smithi118.stderr:Inferring config /var/lib/ceph/d79a5420-e971-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T00:05:20.916 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:05:20.914+0000 7f5237000700 1 -- 172.21.15.118:0/316199301 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f5230076160 msgr2=0x7f52300745c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:05:20.916 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:05:20.914+0000 7f5237000700 1 --2- 172.21.15.118:0/316199301 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f5230076160 0x7f52300745c0 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f5220009910 tx=0x7f5220009c20 comp rx=0 tx=0).stop 2024-03-24T00:05:20.916 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:05:20.914+0000 7f5237000700 1 -- 172.21.15.118:0/316199301 shutdown_connections 2024-03-24T00:05:20.916 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:05:20.914+0000 7f5237000700 1 --2- 172.21.15.118:0/316199301 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f5230076160 0x7f52300745c0 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:05:20.916 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:05:20.914+0000 7f5237000700 1 -- 172.21.15.118:0/316199301 >> 172.21.15.118:0/316199301 conn(0x7f52300fd180 msgr2=0x7f52300ff5a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:05:20.916 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:05:20.914+0000 7f5237000700 1 -- 172.21.15.118:0/316199301 shutdown_connections 2024-03-24T00:05:20.916 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:05:20.914+0000 7f5237000700 1 -- 172.21.15.118:0/316199301 wait complete. 2024-03-24T00:05:20.917 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:05:20.916+0000 7f5237000700 1 Processor -- start 2024-03-24T00:05:20.918 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:05:20.916+0000 7f5237000700 1 -- start start 2024-03-24T00:05:20.918 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:05:20.916+0000 7f5237000700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f5230076160 0x7f5230199480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:05:20.918 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:05:20.916+0000 7f5237000700 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5230109e40 con 0x7f5230076160 2024-03-24T00:05:20.918 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:05:20.917+0000 7f522fe00700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f5230076160 0x7f5230199480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:05:20.918 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:05:20.917+0000 7f522fe00700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f5230076160 0x7f5230199480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.118:3300/0 says I am v2:172.21.15.118:57304/0 (socket says 172.21.15.118:57304) 2024-03-24T00:05:20.918 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:05:20.917+0000 7f522fe00700 1 -- 172.21.15.118:0/3450335421 learned_addr learned my addr 172.21.15.118:0/3450335421 (peer_addr_for_me v2:172.21.15.118:0/0) 2024-03-24T00:05:20.918 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:05:20.917+0000 7f522fe00700 1 -- 172.21.15.118:0/3450335421 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f52301999c0 con 0x7f5230076160 2024-03-24T00:05:20.918 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:05:20.917+0000 7f522fe00700 1 --2- 172.21.15.118:0/3450335421 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f5230076160 0x7f5230199480 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f5220004fa0 tx=0x7f5220005080 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:05:20.919 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:05:20.918+0000 7f522cc00700 1 -- 172.21.15.118:0/3450335421 <== mon.0 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f5220027070 con 0x7f5230076160 2024-03-24T00:05:20.919 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:05:20.918+0000 7f522cc00700 1 -- 172.21.15.118:0/3450335421 <== mon.0 v2:172.21.15.118:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f522000be40 con 0x7f5230076160 2024-03-24T00:05:20.919 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:05:20.918+0000 7f522cc00700 1 -- 172.21.15.118:0/3450335421 <== mon.0 v2:172.21.15.118:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f522000f460 con 0x7f5230076160 2024-03-24T00:05:20.919 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:05:20.918+0000 7f5237000700 1 -- 172.21.15.118:0/3450335421 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5230199c50 con 0x7f5230076160 2024-03-24T00:05:20.919 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:05:20.918+0000 7f5237000700 1 -- 172.21.15.118:0/3450335421 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f523019c940 con 0x7f5230076160 2024-03-24T00:05:20.919 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:05:20.919+0000 7f522cc00700 1 -- 172.21.15.118:0/3450335421 <== mon.0 v2:172.21.15.118:3300/0 4 ==== mgrmap(e 13) v1 ==== 46163+0+0 (secure 0 0 0) 0x7f522002a070 con 0x7f5230076160 2024-03-24T00:05:20.920 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:05:20.919+0000 7f5237000700 1 -- 172.21.15.118:0/3450335421 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f523010a5c0 con 0x7f5230076160 2024-03-24T00:05:20.920 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:05:20.920+0000 7f522cc00700 1 --2- 172.21.15.118:0/3450335421 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7f5214039020 0x7f521403b4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:05:20.920 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:05:20.920+0000 7f522cc00700 1 -- 172.21.15.118:0/3450335421 <== mon.0 v2:172.21.15.118:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1155+0+0 (secure 0 0 0) 0x7f522001e030 con 0x7f5230076160 2024-03-24T00:05:20.920 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:05:20.920+0000 7f522f400700 1 --2- 172.21.15.118:0/3450335421 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7f5214039020 0x7f521403b4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:05:20.921 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:05:20.921+0000 7f522f400700 1 --2- 172.21.15.118:0/3450335421 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7f5214039020 0x7f521403b4e0 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f521c006fd0 tx=0x7f521c006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:05:20.929 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:05:20.926+0000 7f522cc00700 1 -- 172.21.15.118:0/3450335421 <== mon.0 v2:172.21.15.118:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f5220013070 con 0x7f5230076160 2024-03-24T00:05:21.152 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:05:21.151+0000 7f5237000700 1 -- 172.21.15.118:0/3450335421 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_command([{prefix=config set, name=mgr/cephadm/allow_ptrace}] v 0) v1 -- 0x7f523002db40 con 0x7f5230076160 2024-03-24T00:05:21.156 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:05:21.156+0000 7f522cc00700 1 -- 172.21.15.118:0/3450335421 <== mon.0 v2:172.21.15.118:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/cephadm/allow_ptrace}]=0 v9)=0 v9) v1 ==== 125+0+0 (secure 0 0 0) 0x7f522002bde0 con 0x7f5230076160 2024-03-24T00:05:21.160 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:05:21.159+0000 7f5237000700 1 -- 172.21.15.118:0/3450335421 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7f5214039020 msgr2=0x7f521403b4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:05:21.160 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:05:21.159+0000 7f5237000700 1 --2- 172.21.15.118:0/3450335421 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7f5214039020 0x7f521403b4e0 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f521c006fd0 tx=0x7f521c006e40 comp rx=0 tx=0).stop 2024-03-24T00:05:21.160 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:05:21.159+0000 7f5237000700 1 -- 172.21.15.118:0/3450335421 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f5230076160 msgr2=0x7f5230199480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:05:21.160 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:05:21.159+0000 7f5237000700 1 --2- 172.21.15.118:0/3450335421 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f5230076160 0x7f5230199480 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f5220004fa0 tx=0x7f5220005080 comp rx=0 tx=0).stop 2024-03-24T00:05:21.160 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:05:21.159+0000 7f5237000700 1 -- 172.21.15.118:0/3450335421 shutdown_connections 2024-03-24T00:05:21.160 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:05:21.159+0000 7f5237000700 1 --2- 172.21.15.118:0/3450335421 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7f5214039020 0x7f521403b4e0 unknown :-1 s=CLOSED pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:05:21.160 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:05:21.159+0000 7f5237000700 1 --2- 172.21.15.118:0/3450335421 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f5230076160 0x7f5230199480 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:05:21.161 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:05:21.159+0000 7f5237000700 1 -- 172.21.15.118:0/3450335421 >> 172.21.15.118:0/3450335421 conn(0x7f52300fd180 msgr2=0x7f52300fdba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:05:21.161 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:05:21.159+0000 7f5237000700 1 -- 172.21.15.118:0/3450335421 shutdown_connections 2024-03-24T00:05:21.161 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:05:21.159+0000 7f5237000700 1 -- 172.21.15.118:0/3450335421 wait complete. 2024-03-24T00:05:21.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:05:21 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T00:05:21.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:05:21 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T00:05:21.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:05:21 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd/host:smithi118", "name": "osd_memory_target"}]: dispatch 2024-03-24T00:05:21.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:05:21 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T00:05:21.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:05:21 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T00:05:21.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:05:21 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T00:05:21.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:05:21 smithi118 ceph-mon[31535]: from='client.? 172.21.15.118:0/3450335421' entity='client.admin' 2024-03-24T00:05:21.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:05:21 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T00:05:21.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:05:21 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T00:05:21.953 INFO:tasks.cephadm:Distributing conf and client.admin keyring to all hosts + 0755 2024-03-24T00:05:21.953 DEBUG:teuthology.orchestra.run.smithi118:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d79a5420-e971-11ee-95cd-87774f69a715 -- ceph orch client-keyring set client.admin '*' --mode 0755 2024-03-24T00:05:22.366 INFO:teuthology.orchestra.run.smithi118.stderr:Inferring config /var/lib/ceph/d79a5420-e971-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T00:05:24.634 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:05:24.633+0000 7f2fa1400700 1 -- 172.21.15.118:0/3252624022 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f2f9c103c00 msgr2=0x7f2f9c104020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:05:24.634 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:05:24.633+0000 7f2fa1400700 1 --2- 172.21.15.118:0/3252624022 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f2f9c103c00 0x7f2f9c104020 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f2f8c009910 tx=0x7f2f8c009c20 comp rx=0 tx=0).stop 2024-03-24T00:05:24.634 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:05:24.633+0000 7f2fa1400700 1 -- 172.21.15.118:0/3252624022 shutdown_connections 2024-03-24T00:05:24.634 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:05:24.633+0000 7f2fa1400700 1 --2- 172.21.15.118:0/3252624022 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f2f9c103c00 0x7f2f9c104020 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:05:24.634 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:05:24.633+0000 7f2fa1400700 1 -- 172.21.15.118:0/3252624022 >> 172.21.15.118:0/3252624022 conn(0x7f2f9c0ff1a0 msgr2=0x7f2f9c1015e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:05:24.634 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:05:24.633+0000 7f2fa1400700 1 -- 172.21.15.118:0/3252624022 shutdown_connections 2024-03-24T00:05:24.634 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:05:24.633+0000 7f2fa1400700 1 -- 172.21.15.118:0/3252624022 wait complete. 2024-03-24T00:05:24.635 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:05:24.635+0000 7f2fa1400700 1 Processor -- start 2024-03-24T00:05:24.635 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:05:24.635+0000 7f2fa1400700 1 -- start start 2024-03-24T00:05:24.636 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:05:24.636+0000 7f2fa1400700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f2f9c103c00 0x7f2f9c1993a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:05:24.636 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:05:24.636+0000 7f2fa1400700 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2f9c109c90 con 0x7f2f9c103c00 2024-03-24T00:05:24.637 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:05:24.636+0000 7f2f9be00700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f2f9c103c00 0x7f2f9c1993a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:05:24.637 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:05:24.636+0000 7f2f9be00700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f2f9c103c00 0x7f2f9c1993a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.118:3300/0 says I am v2:172.21.15.118:56056/0 (socket says 172.21.15.118:56056) 2024-03-24T00:05:24.637 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:05:24.636+0000 7f2f9be00700 1 -- 172.21.15.118:0/1693366056 learned_addr learned my addr 172.21.15.118:0/1693366056 (peer_addr_for_me v2:172.21.15.118:0/0) 2024-03-24T00:05:24.637 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:05:24.636+0000 7f2f9be00700 1 -- 172.21.15.118:0/1693366056 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2f9c1998e0 con 0x7f2f9c103c00 2024-03-24T00:05:24.637 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:05:24.637+0000 7f2f9be00700 1 --2- 172.21.15.118:0/1693366056 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f2f9c103c00 0x7f2f9c1993a0 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f2f8c005850 tx=0x7f2f8c005030 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:05:24.638 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:05:24.637+0000 7f2f98c00700 1 -- 172.21.15.118:0/1693366056 <== mon.0 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f2f8c026070 con 0x7f2f9c103c00 2024-03-24T00:05:24.638 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:05:24.637+0000 7f2f98c00700 1 -- 172.21.15.118:0/1693366056 <== mon.0 v2:172.21.15.118:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f2f8c0044a0 con 0x7f2f9c103c00 2024-03-24T00:05:24.638 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:05:24.637+0000 7f2fa1400700 1 -- 172.21.15.118:0/1693366056 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2f9c199b70 con 0x7f2f9c103c00 2024-03-24T00:05:24.638 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:05:24.637+0000 7f2f98c00700 1 -- 172.21.15.118:0/1693366056 <== mon.0 v2:172.21.15.118:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f2f8c003ab0 con 0x7f2f9c103c00 2024-03-24T00:05:24.638 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:05:24.637+0000 7f2fa1400700 1 -- 172.21.15.118:0/1693366056 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2f9c19c7c0 con 0x7f2f9c103c00 2024-03-24T00:05:24.639 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:05:24.638+0000 7f2f98c00700 1 -- 172.21.15.118:0/1693366056 <== mon.0 v2:172.21.15.118:3300/0 4 ==== mgrmap(e 13) v1 ==== 46163+0+0 (secure 0 0 0) 0x7f2f8c029070 con 0x7f2f9c103c00 2024-03-24T00:05:24.639 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:05:24.638+0000 7f2fa1400700 1 -- 172.21.15.118:0/1693366056 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2f9c10a410 con 0x7f2f9c103c00 2024-03-24T00:05:24.639 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:05:24.639+0000 7f2f98c00700 1 --2- 172.21.15.118:0/1693366056 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7f2f80039020 0x7f2f8003b4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:05:24.639 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:05:24.639+0000 7f2f98c00700 1 -- 172.21.15.118:0/1693366056 <== mon.0 v2:172.21.15.118:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1155+0+0 (secure 0 0 0) 0x7f2f8c01e070 con 0x7f2f9c103c00 2024-03-24T00:05:24.640 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:05:24.639+0000 7f2f9b400700 1 --2- 172.21.15.118:0/1693366056 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7f2f80039020 0x7f2f8003b4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:05:24.640 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:05:24.640+0000 7f2f9b400700 1 --2- 172.21.15.118:0/1693366056 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7f2f80039020 0x7f2f8003b4e0 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f2f88006fd0 tx=0x7f2f88006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:05:24.645 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:05:24.644+0000 7f2f98c00700 1 -- 172.21.15.118:0/1693366056 <== mon.0 v2:172.21.15.118:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f2f8c013070 con 0x7f2f9c103c00 2024-03-24T00:05:24.870 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:05:24.869+0000 7f2fa1400700 1 -- 172.21.15.118:0/1693366056 --> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] -- mgr_command(tid 0: {"prefix": "orch client-keyring set", "entity": "client.admin", "placement": "*", "mode": "0755", "target": ["mon-mgr", ""]}) v1 -- 0x7f2f9c02db40 con 0x7f2f80039020 2024-03-24T00:05:24.877 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:05:24.877+0000 7f2f98c00700 1 -- 172.21.15.118:0/1693366056 <== mgr.14152 v2:172.21.15.118:6800/2112912943 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7f2f9c02db40 con 0x7f2f80039020 2024-03-24T00:05:24.880 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:05:24.879+0000 7f2fa1400700 1 -- 172.21.15.118:0/1693366056 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7f2f80039020 msgr2=0x7f2f8003b4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:05:24.880 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:05:24.879+0000 7f2fa1400700 1 --2- 172.21.15.118:0/1693366056 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7f2f80039020 0x7f2f8003b4e0 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f2f88006fd0 tx=0x7f2f88006e40 comp rx=0 tx=0).stop 2024-03-24T00:05:24.880 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:05:24.879+0000 7f2fa1400700 1 -- 172.21.15.118:0/1693366056 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f2f9c103c00 msgr2=0x7f2f9c1993a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:05:24.880 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:05:24.879+0000 7f2fa1400700 1 --2- 172.21.15.118:0/1693366056 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f2f9c103c00 0x7f2f9c1993a0 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f2f8c005850 tx=0x7f2f8c005030 comp rx=0 tx=0).stop 2024-03-24T00:05:24.881 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:05:24.880+0000 7f2fa1400700 1 -- 172.21.15.118:0/1693366056 shutdown_connections 2024-03-24T00:05:24.881 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:05:24.880+0000 7f2fa1400700 1 --2- 172.21.15.118:0/1693366056 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7f2f80039020 0x7f2f8003b4e0 unknown :-1 s=CLOSED pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:05:24.881 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:05:24.880+0000 7f2fa1400700 1 --2- 172.21.15.118:0/1693366056 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f2f9c103c00 0x7f2f9c1993a0 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:05:24.881 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:05:24.880+0000 7f2fa1400700 1 -- 172.21.15.118:0/1693366056 >> 172.21.15.118:0/1693366056 conn(0x7f2f9c0ff1a0 msgr2=0x7f2f9c0ffc90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:05:24.881 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:05:24.880+0000 7f2fa1400700 1 -- 172.21.15.118:0/1693366056 shutdown_connections 2024-03-24T00:05:24.881 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:05:24.880+0000 7f2fa1400700 1 -- 172.21.15.118:0/1693366056 wait complete. 2024-03-24T00:05:25.777 INFO:tasks.cephadm:Writing (initial) conf and keyring to smithi189 2024-03-24T00:05:25.777 DEBUG:teuthology.orchestra.run.smithi189:> set -ex 2024-03-24T00:05:25.777 DEBUG:teuthology.orchestra.run.smithi189:> dd of=/etc/ceph/ceph.conf 2024-03-24T00:05:25.795 DEBUG:teuthology.orchestra.run.smithi189:> set -ex 2024-03-24T00:05:25.795 DEBUG:teuthology.orchestra.run.smithi189:> dd of=/etc/ceph/ceph.client.admin.keyring 2024-03-24T00:05:25.853 INFO:tasks.cephadm:Adding host smithi189 to orchestrator... 2024-03-24T00:05:25.853 DEBUG:teuthology.orchestra.run.smithi118:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d79a5420-e971-11ee-95cd-87774f69a715 -- ceph orch host add smithi189 2024-03-24T00:05:26.065 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:05:25 smithi118 ceph-mon[31535]: from='client.14176 -' entity='client.admin' cmd=[{"prefix": "orch client-keyring set", "entity": "client.admin", "placement": "*", "mode": "0755", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T00:05:26.065 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:05:25 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T00:05:26.065 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:05:25 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T00:05:26.065 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:05:25 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T00:05:26.065 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:05:25 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T00:05:26.065 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:05:25 smithi118 ceph-mon[31535]: Updating smithi118:/etc/ceph/ceph.conf 2024-03-24T00:05:26.065 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:05:25 smithi118 ceph-mon[31535]: Updating smithi118:/var/lib/ceph/d79a5420-e971-11ee-95cd-87774f69a715/config/ceph.conf 2024-03-24T00:05:26.066 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:05:25 smithi118 ceph-mon[31535]: pgmap v3: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T00:05:26.278 INFO:teuthology.orchestra.run.smithi118.stderr:Inferring config /var/lib/ceph/d79a5420-e971-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T00:05:27.195 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:05:26 smithi118 ceph-mon[31535]: Updating smithi118:/etc/ceph/ceph.client.admin.keyring 2024-03-24T00:05:28.321 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:05:28.319+0000 7f2066200700 1 -- 172.21.15.118:0/2459012762 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f20601083f0 msgr2=0x7f20601087d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:05:28.321 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:05:28.319+0000 7f2066200700 1 --2- 172.21.15.118:0/2459012762 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f20601083f0 0x7f20601087d0 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f2050009910 tx=0x7f2050009c20 comp rx=0 tx=0).stop 2024-03-24T00:05:28.321 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:05:28.319+0000 7f2066200700 1 -- 172.21.15.118:0/2459012762 shutdown_connections 2024-03-24T00:05:28.321 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:05:28.319+0000 7f2066200700 1 --2- 172.21.15.118:0/2459012762 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f20601083f0 0x7f20601087d0 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:05:28.321 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:05:28.319+0000 7f2066200700 1 -- 172.21.15.118:0/2459012762 >> 172.21.15.118:0/2459012762 conn(0x7f20600779c0 msgr2=0x7f2060079de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:05:28.321 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:05:28.319+0000 7f2066200700 1 -- 172.21.15.118:0/2459012762 shutdown_connections 2024-03-24T00:05:28.321 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:05:28.319+0000 7f2066200700 1 -- 172.21.15.118:0/2459012762 wait complete. 2024-03-24T00:05:28.322 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:05:28.321+0000 7f2066200700 1 Processor -- start 2024-03-24T00:05:28.322 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:05:28.321+0000 7f2066200700 1 -- start start 2024-03-24T00:05:28.322 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:05:28.321+0000 7f2066200700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f20601083f0 0x7f20601a3930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:05:28.322 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:05:28.321+0000 7f2066200700 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f20601127b0 con 0x7f20601083f0 2024-03-24T00:05:28.322 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:05:28.321+0000 7f205f400700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f20601083f0 0x7f20601a3930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:05:28.322 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:05:28.321+0000 7f205f400700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f20601083f0 0x7f20601a3930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.118:3300/0 says I am v2:172.21.15.118:56070/0 (socket says 172.21.15.118:56070) 2024-03-24T00:05:28.322 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:05:28.321+0000 7f205f400700 1 -- 172.21.15.118:0/4249072835 learned_addr learned my addr 172.21.15.118:0/4249072835 (peer_addr_for_me v2:172.21.15.118:0/0) 2024-03-24T00:05:28.323 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:05:28.322+0000 7f205f400700 1 -- 172.21.15.118:0/4249072835 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f20601a3e70 con 0x7f20601083f0 2024-03-24T00:05:28.323 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:05:28.322+0000 7f205f400700 1 --2- 172.21.15.118:0/4249072835 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f20601083f0 0x7f20601a3930 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f2050005850 tx=0x7f2050005030 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:05:28.323 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:05:28.322+0000 7f2057e00700 1 -- 172.21.15.118:0/4249072835 <== mon.0 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f2050026070 con 0x7f20601083f0 2024-03-24T00:05:28.324 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:05:28.322+0000 7f2057e00700 1 -- 172.21.15.118:0/4249072835 <== mon.0 v2:172.21.15.118:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f205000bbd0 con 0x7f20601083f0 2024-03-24T00:05:28.324 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:05:28.322+0000 7f2066200700 1 -- 172.21.15.118:0/4249072835 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f20601a4160 con 0x7f20601083f0 2024-03-24T00:05:28.324 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:05:28.322+0000 7f2057e00700 1 -- 172.21.15.118:0/4249072835 <== mon.0 v2:172.21.15.118:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f2050003970 con 0x7f20601083f0 2024-03-24T00:05:28.324 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:05:28.323+0000 7f2066200700 1 -- 172.21.15.118:0/4249072835 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f206019daf0 con 0x7f20601083f0 2024-03-24T00:05:28.324 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:05:28.324+0000 7f2057e00700 1 -- 172.21.15.118:0/4249072835 <== mon.0 v2:172.21.15.118:3300/0 4 ==== mgrmap(e 13) v1 ==== 46163+0+0 (secure 0 0 0) 0x7f2050029070 con 0x7f20601083f0 2024-03-24T00:05:28.325 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:05:28.324+0000 7f2066200700 1 -- 172.21.15.118:0/4249072835 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f20601098c0 con 0x7f20601083f0 2024-03-24T00:05:28.325 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:05:28.324+0000 7f2057e00700 1 --2- 172.21.15.118:0/4249072835 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7f2044039020 0x7f204403b4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:05:28.325 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:05:28.324+0000 7f2057e00700 1 -- 172.21.15.118:0/4249072835 <== mon.0 v2:172.21.15.118:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1155+0+0 (secure 0 0 0) 0x7f205001e070 con 0x7f20601083f0 2024-03-24T00:05:28.325 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:05:28.324+0000 7f205ea00700 1 --2- 172.21.15.118:0/4249072835 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7f2044039020 0x7f204403b4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:05:28.325 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:05:28.325+0000 7f205ea00700 1 --2- 172.21.15.118:0/4249072835 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7f2044039020 0x7f204403b4e0 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f204c006fd0 tx=0x7f204c006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:05:28.330 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:05:28.330+0000 7f2057e00700 1 -- 172.21.15.118:0/4249072835 <== mon.0 v2:172.21.15.118:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f2050013070 con 0x7f20601083f0 2024-03-24T00:05:28.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:05:28 smithi118 ceph-mon[31535]: Updating smithi118:/var/lib/ceph/d79a5420-e971-11ee-95cd-87774f69a715/config/ceph.client.admin.keyring 2024-03-24T00:05:28.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:05:28 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T00:05:28.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:05:28 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T00:05:28.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:05:28 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T00:05:28.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:05:28 smithi118 ceph-mon[31535]: pgmap v4: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T00:05:28.556 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:05:28.555+0000 7f2066200700 1 -- 172.21.15.118:0/4249072835 --> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] -- mgr_command(tid 0: {"prefix": "orch host add", "hostname": "smithi189", "target": ["mon-mgr", ""]}) v1 -- 0x7f206019efe0 con 0x7f2044039020 2024-03-24T00:05:29.695 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:05:29 smithi118 ceph-mon[31535]: from='client.14178 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi189", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T00:05:30.695 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:05:30 smithi118 ceph-mon[31535]: Deploying cephadm binary to smithi189 2024-03-24T00:05:30.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:05:30 smithi118 ceph-mon[31535]: pgmap v5: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T00:05:32.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:05:32 smithi118 ceph-mon[31535]: pgmap v6: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T00:05:33.206 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:05:33.206+0000 7f2057e00700 1 -- 172.21.15.118:0/4249072835 <== mgr.14152 v2:172.21.15.118:6800/2112912943 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+49 (secure 0 0 0) 0x7f206019efe0 con 0x7f2044039020 2024-03-24T00:05:33.207 INFO:teuthology.orchestra.run.smithi118.stdout:Added host 'smithi189' with addr '172.21.15.189' 2024-03-24T00:05:33.210 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:05:33.210+0000 7f2066200700 1 -- 172.21.15.118:0/4249072835 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7f2044039020 msgr2=0x7f204403b4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:05:33.210 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:05:33.210+0000 7f2066200700 1 --2- 172.21.15.118:0/4249072835 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7f2044039020 0x7f204403b4e0 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f204c006fd0 tx=0x7f204c006e40 comp rx=0 tx=0).stop 2024-03-24T00:05:33.210 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:05:33.210+0000 7f2066200700 1 -- 172.21.15.118:0/4249072835 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f20601083f0 msgr2=0x7f20601a3930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:05:33.210 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:05:33.210+0000 7f2066200700 1 --2- 172.21.15.118:0/4249072835 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f20601083f0 0x7f20601a3930 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f2050005850 tx=0x7f2050005030 comp rx=0 tx=0).stop 2024-03-24T00:05:33.210 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:05:33.210+0000 7f2066200700 1 -- 172.21.15.118:0/4249072835 shutdown_connections 2024-03-24T00:05:33.210 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:05:33.210+0000 7f2066200700 1 --2- 172.21.15.118:0/4249072835 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7f2044039020 0x7f204403b4e0 unknown :-1 s=CLOSED pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:05:33.210 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:05:33.210+0000 7f2066200700 1 --2- 172.21.15.118:0/4249072835 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f20601083f0 0x7f20601a3930 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:05:33.211 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:05:33.210+0000 7f2066200700 1 -- 172.21.15.118:0/4249072835 >> 172.21.15.118:0/4249072835 conn(0x7f20600779c0 msgr2=0x7f2060109db0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:05:33.211 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:05:33.210+0000 7f2066200700 1 -- 172.21.15.118:0/4249072835 shutdown_connections 2024-03-24T00:05:33.211 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:05:33.210+0000 7f2066200700 1 -- 172.21.15.118:0/4249072835 wait complete. 2024-03-24T00:05:33.817 DEBUG:teuthology.orchestra.run.smithi118:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d79a5420-e971-11ee-95cd-87774f69a715 -- ceph orch host ls --format=json 2024-03-24T00:05:34.240 INFO:teuthology.orchestra.run.smithi118.stderr:Inferring config /var/lib/ceph/d79a5420-e971-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T00:05:34.505 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:05:34 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T00:05:34.505 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:05:34 smithi118 ceph-mon[31535]: Added host smithi189 2024-03-24T00:05:34.505 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:05:34 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T00:05:34.505 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:05:34 smithi118 ceph-mon[31535]: pgmap v7: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T00:05:34.506 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:05:34 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T00:05:35.386 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:05:35 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T00:05:36.114 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:05:36.113+0000 7f7d59200700 1 -- 172.21.15.118:0/627281228 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f7d541083f0 msgr2=0x7f7d541087d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:05:36.114 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:05:36.113+0000 7f7d59200700 1 --2- 172.21.15.118:0/627281228 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f7d541083f0 0x7f7d541087d0 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f7d44009910 tx=0x7f7d44009c20 comp rx=0 tx=0).stop 2024-03-24T00:05:36.114 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:05:36.114+0000 7f7d59200700 1 -- 172.21.15.118:0/627281228 shutdown_connections 2024-03-24T00:05:36.115 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:05:36.114+0000 7f7d59200700 1 --2- 172.21.15.118:0/627281228 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f7d541083f0 0x7f7d541087d0 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:05:36.115 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:05:36.114+0000 7f7d59200700 1 -- 172.21.15.118:0/627281228 >> 172.21.15.118:0/627281228 conn(0x7f7d540779c0 msgr2=0x7f7d54079de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:05:36.115 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:05:36.114+0000 7f7d59200700 1 -- 172.21.15.118:0/627281228 shutdown_connections 2024-03-24T00:05:36.115 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:05:36.114+0000 7f7d59200700 1 -- 172.21.15.118:0/627281228 wait complete. 2024-03-24T00:05:36.116 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:05:36.115+0000 7f7d59200700 1 Processor -- start 2024-03-24T00:05:36.116 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:05:36.116+0000 7f7d59200700 1 -- start start 2024-03-24T00:05:36.116 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:05:36.116+0000 7f7d59200700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f7d541083f0 0x7f7d541a3920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:05:36.116 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:05:36.116+0000 7f7d59200700 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7d541127b0 con 0x7f7d541083f0 2024-03-24T00:05:36.116 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:05:36.117+0000 7f7d51e00700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f7d541083f0 0x7f7d541a3920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:05:36.116 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:05:36.117+0000 7f7d51e00700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f7d541083f0 0x7f7d541a3920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.118:3300/0 says I am v2:172.21.15.118:41434/0 (socket says 172.21.15.118:41434) 2024-03-24T00:05:36.117 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:05:36.117+0000 7f7d51e00700 1 -- 172.21.15.118:0/2315368673 learned_addr learned my addr 172.21.15.118:0/2315368673 (peer_addr_for_me v2:172.21.15.118:0/0) 2024-03-24T00:05:36.117 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:05:36.117+0000 7f7d51e00700 1 -- 172.21.15.118:0/2315368673 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7d541a3e60 con 0x7f7d541083f0 2024-03-24T00:05:36.118 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:05:36.117+0000 7f7d51e00700 1 --2- 172.21.15.118:0/2315368673 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f7d541083f0 0x7f7d541a3920 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f7d44006010 tx=0x7f7d44005030 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:05:36.118 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:05:36.118+0000 7f7d4aa00700 1 -- 172.21.15.118:0/2315368673 <== mon.0 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f7d44026070 con 0x7f7d541083f0 2024-03-24T00:05:36.118 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:05:36.118+0000 7f7d59200700 1 -- 172.21.15.118:0/2315368673 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7d541a4150 con 0x7f7d541083f0 2024-03-24T00:05:36.118 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:05:36.118+0000 7f7d4aa00700 1 -- 172.21.15.118:0/2315368673 <== mon.0 v2:172.21.15.118:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f7d4400bbd0 con 0x7f7d541083f0 2024-03-24T00:05:36.118 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:05:36.118+0000 7f7d4aa00700 1 -- 172.21.15.118:0/2315368673 <== mon.0 v2:172.21.15.118:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f7d44003910 con 0x7f7d541083f0 2024-03-24T00:05:36.118 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:05:36.118+0000 7f7d59200700 1 -- 172.21.15.118:0/2315368673 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7d5419dae0 con 0x7f7d541083f0 2024-03-24T00:05:36.119 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:05:36.119+0000 7f7d59200700 1 -- 172.21.15.118:0/2315368673 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7d541098c0 con 0x7f7d541083f0 2024-03-24T00:05:36.119 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:05:36.119+0000 7f7d4aa00700 1 -- 172.21.15.118:0/2315368673 <== mon.0 v2:172.21.15.118:3300/0 4 ==== mgrmap(e 13) v1 ==== 46163+0+0 (secure 0 0 0) 0x7f7d44029070 con 0x7f7d541083f0 2024-03-24T00:05:36.120 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:05:36.120+0000 7f7d4aa00700 1 --2- 172.21.15.118:0/2315368673 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7f7d38038fc0 0x7f7d3803b480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:05:36.120 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:05:36.120+0000 7f7d4aa00700 1 -- 172.21.15.118:0/2315368673 <== mon.0 v2:172.21.15.118:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1155+0+0 (secure 0 0 0) 0x7f7d4401e070 con 0x7f7d541083f0 2024-03-24T00:05:36.120 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:05:36.120+0000 7f7d4be00700 1 --2- 172.21.15.118:0/2315368673 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7f7d38038fc0 0x7f7d3803b480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:05:36.121 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:05:36.121+0000 7f7d4be00700 1 --2- 172.21.15.118:0/2315368673 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7f7d38038fc0 0x7f7d3803b480 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7f7d3c006fd0 tx=0x7f7d3c006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:05:36.125 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:05:36.125+0000 7f7d4aa00700 1 -- 172.21.15.118:0/2315368673 <== mon.0 v2:172.21.15.118:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f7d44013070 con 0x7f7d541083f0 2024-03-24T00:05:36.354 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:05:36.354+0000 7f7d59200700 1 -- 172.21.15.118:0/2315368673 --> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] -- mgr_command(tid 0: {"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f7d5419efd0 con 0x7f7d38038fc0 2024-03-24T00:05:36.356 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:05:36.356+0000 7f7d4aa00700 1 -- 172.21.15.118:0/2315368673 <== mgr.14152 v2:172.21.15.118:6800/2112912943 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+161 (secure 0 0 0) 0x7f7d5419efd0 con 0x7f7d38038fc0 2024-03-24T00:05:36.356 INFO:teuthology.orchestra.run.smithi118.stdout: 2024-03-24T00:05:36.356 INFO:teuthology.orchestra.run.smithi118.stdout:[{"addr": "172.21.15.118", "hostname": "smithi118", "labels": [], "status": ""}, {"addr": "172.21.15.189", "hostname": "smithi189", "labels": [], "status": ""}] 2024-03-24T00:05:36.360 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:05:36.359+0000 7f7d59200700 1 -- 172.21.15.118:0/2315368673 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7f7d38038fc0 msgr2=0x7f7d3803b480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:05:36.360 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:05:36.359+0000 7f7d59200700 1 --2- 172.21.15.118:0/2315368673 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7f7d38038fc0 0x7f7d3803b480 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7f7d3c006fd0 tx=0x7f7d3c006e40 comp rx=0 tx=0).stop 2024-03-24T00:05:36.360 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:05:36.359+0000 7f7d59200700 1 -- 172.21.15.118:0/2315368673 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f7d541083f0 msgr2=0x7f7d541a3920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:05:36.360 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:05:36.359+0000 7f7d59200700 1 --2- 172.21.15.118:0/2315368673 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f7d541083f0 0x7f7d541a3920 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f7d44006010 tx=0x7f7d44005030 comp rx=0 tx=0).stop 2024-03-24T00:05:36.360 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:05:36.360+0000 7f7d59200700 1 -- 172.21.15.118:0/2315368673 shutdown_connections 2024-03-24T00:05:36.360 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:05:36.360+0000 7f7d59200700 1 --2- 172.21.15.118:0/2315368673 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7f7d38038fc0 0x7f7d3803b480 unknown :-1 s=CLOSED pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:05:36.360 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:05:36.360+0000 7f7d59200700 1 --2- 172.21.15.118:0/2315368673 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f7d541083f0 0x7f7d541a3920 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:05:36.360 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:05:36.360+0000 7f7d59200700 1 -- 172.21.15.118:0/2315368673 >> 172.21.15.118:0/2315368673 conn(0x7f7d540779c0 msgr2=0x7f7d54109db0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:05:36.360 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:05:36.360+0000 7f7d59200700 1 -- 172.21.15.118:0/2315368673 shutdown_connections 2024-03-24T00:05:36.360 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:05:36.360+0000 7f7d59200700 1 -- 172.21.15.118:0/2315368673 wait complete. 2024-03-24T00:05:36.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:05:36 smithi118 ceph-mon[31535]: pgmap v8: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T00:05:36.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:05:36 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T00:05:37.312 INFO:tasks.cephadm:Writing (initial) conf and keyring to smithi193 2024-03-24T00:05:37.312 DEBUG:teuthology.orchestra.run.smithi193:> set -ex 2024-03-24T00:05:37.312 DEBUG:teuthology.orchestra.run.smithi193:> dd of=/etc/ceph/ceph.conf 2024-03-24T00:05:37.332 DEBUG:teuthology.orchestra.run.smithi193:> set -ex 2024-03-24T00:05:37.332 DEBUG:teuthology.orchestra.run.smithi193:> dd of=/etc/ceph/ceph.client.admin.keyring 2024-03-24T00:05:37.389 INFO:tasks.cephadm:Adding host smithi193 to orchestrator... 2024-03-24T00:05:37.389 DEBUG:teuthology.orchestra.run.smithi118:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d79a5420-e971-11ee-95cd-87774f69a715 -- ceph orch host add smithi193 2024-03-24T00:05:37.478 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:05:37 smithi118 ceph-mon[31535]: from='client.14180 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-24T00:05:37.826 INFO:teuthology.orchestra.run.smithi118.stderr:Inferring config /var/lib/ceph/d79a5420-e971-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T00:05:38.379 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:05:38 smithi118 ceph-mon[31535]: pgmap v9: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T00:05:39.794 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:05:39.794+0000 7f41c5400700 1 -- 172.21.15.118:0/4293061409 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f41c0102e10 msgr2=0x7f41c01031f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:05:39.794 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:05:39.794+0000 7f41c5400700 1 --2- 172.21.15.118:0/4293061409 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f41c0102e10 0x7f41c01031f0 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f41b0009910 tx=0x7f41b0009c20 comp rx=0 tx=0).stop 2024-03-24T00:05:39.794 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:05:39.794+0000 7f41c5400700 1 -- 172.21.15.118:0/4293061409 shutdown_connections 2024-03-24T00:05:39.794 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:05:39.794+0000 7f41c5400700 1 --2- 172.21.15.118:0/4293061409 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f41c0102e10 0x7f41c01031f0 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:05:39.794 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:05:39.794+0000 7f41c5400700 1 -- 172.21.15.118:0/4293061409 >> 172.21.15.118:0/4293061409 conn(0x7f41c0077830 msgr2=0x7f41c00febb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:05:39.795 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:05:39.794+0000 7f41c5400700 1 -- 172.21.15.118:0/4293061409 shutdown_connections 2024-03-24T00:05:39.795 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:05:39.794+0000 7f41c5400700 1 -- 172.21.15.118:0/4293061409 wait complete. 2024-03-24T00:05:39.797 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:05:39.795+0000 7f41c5400700 1 Processor -- start 2024-03-24T00:05:39.797 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:05:39.795+0000 7f41c5400700 1 -- start start 2024-03-24T00:05:39.797 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:05:39.795+0000 7f41c5400700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f41c0102e10 0x7f41c019f660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:05:39.797 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:05:39.795+0000 7f41c5400700 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f41c010c1f0 con 0x7f41c0102e10 2024-03-24T00:05:39.797 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:05:39.796+0000 7f41be800700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f41c0102e10 0x7f41c019f660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:05:39.797 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:05:39.796+0000 7f41be800700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f41c0102e10 0x7f41c019f660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.118:3300/0 says I am v2:172.21.15.118:41458/0 (socket says 172.21.15.118:41458) 2024-03-24T00:05:39.797 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:05:39.796+0000 7f41be800700 1 -- 172.21.15.118:0/395897314 learned_addr learned my addr 172.21.15.118:0/395897314 (peer_addr_for_me v2:172.21.15.118:0/0) 2024-03-24T00:05:39.797 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:05:39.796+0000 7f41be800700 1 -- 172.21.15.118:0/395897314 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f41c019fba0 con 0x7f41c0102e10 2024-03-24T00:05:39.797 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:05:39.796+0000 7f41be800700 1 --2- 172.21.15.118:0/395897314 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f41c0102e10 0x7f41c019f660 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f41b0006010 tx=0x7f41b0005030 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:05:39.797 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:05:39.797+0000 7f41b7400700 1 -- 172.21.15.118:0/395897314 <== mon.0 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f41b0026070 con 0x7f41c0102e10 2024-03-24T00:05:39.797 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:05:39.797+0000 7f41b7400700 1 -- 172.21.15.118:0/395897314 <== mon.0 v2:172.21.15.118:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f41b000bc30 con 0x7f41c0102e10 2024-03-24T00:05:39.797 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:05:39.797+0000 7f41c5400700 1 -- 172.21.15.118:0/395897314 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f41c019fe30 con 0x7f41c0102e10 2024-03-24T00:05:39.797 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:05:39.797+0000 7f41b7400700 1 -- 172.21.15.118:0/395897314 <== mon.0 v2:172.21.15.118:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f41b00039a0 con 0x7f41c0102e10 2024-03-24T00:05:39.798 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:05:39.797+0000 7f41c5400700 1 -- 172.21.15.118:0/395897314 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f41c0199730 con 0x7f41c0102e10 2024-03-24T00:05:39.798 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:05:39.798+0000 7f41b7400700 1 -- 172.21.15.118:0/395897314 <== mon.0 v2:172.21.15.118:3300/0 4 ==== mgrmap(e 13) v1 ==== 46163+0+0 (secure 0 0 0) 0x7f41b0029070 con 0x7f41c0102e10 2024-03-24T00:05:39.799 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:05:39.798+0000 7f41c5400700 1 -- 172.21.15.118:0/395897314 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f41c0069d60 con 0x7f41c0102e10 2024-03-24T00:05:39.799 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:05:39.798+0000 7f41b7400700 1 --2- 172.21.15.118:0/395897314 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7f41a4039020 0x7f41a403b4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:05:39.799 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:05:39.798+0000 7f41b7400700 1 -- 172.21.15.118:0/395897314 <== mon.0 v2:172.21.15.118:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1155+0+0 (secure 0 0 0) 0x7f41b001e070 con 0x7f41c0102e10 2024-03-24T00:05:39.799 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:05:39.799+0000 7f41bde00700 1 --2- 172.21.15.118:0/395897314 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7f41a4039020 0x7f41a403b4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:05:39.799 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:05:39.799+0000 7f41bde00700 1 --2- 172.21.15.118:0/395897314 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7f41a4039020 0x7f41a403b4e0 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f41ac006fd0 tx=0x7f41ac006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:05:39.803 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:05:39.804+0000 7f41b7400700 1 -- 172.21.15.118:0/395897314 <== mon.0 v2:172.21.15.118:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f41b0013070 con 0x7f41c0102e10 2024-03-24T00:05:40.059 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:05:40.059+0000 7f41c5400700 1 -- 172.21.15.118:0/395897314 --> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] -- mgr_command(tid 0: {"prefix": "orch host add", "hostname": "smithi193", "target": ["mon-mgr", ""]}) v1 -- 0x7f41c019ac60 con 0x7f41a4039020 2024-03-24T00:05:40.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:05:40 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T00:05:40.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:05:40 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T00:05:40.447 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:05:40 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T00:05:40.447 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:05:40 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T00:05:40.447 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:05:40 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd/host:smithi189", "name": "osd_memory_target"}]: dispatch 2024-03-24T00:05:40.447 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:05:40 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T00:05:40.447 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:05:40 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T00:05:40.448 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:05:40 smithi118 ceph-mon[31535]: Updating smithi189:/etc/ceph/ceph.conf 2024-03-24T00:05:40.448 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:05:40 smithi118 ceph-mon[31535]: Updating smithi189:/var/lib/ceph/d79a5420-e971-11ee-95cd-87774f69a715/config/ceph.conf 2024-03-24T00:05:40.448 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:05:40 smithi118 ceph-mon[31535]: pgmap v10: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T00:05:40.448 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:05:40 smithi118 ceph-mon[31535]: Updating smithi189:/etc/ceph/ceph.client.admin.keyring 2024-03-24T00:05:41.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:05:41 smithi118 ceph-mon[31535]: Updating smithi189:/var/lib/ceph/d79a5420-e971-11ee-95cd-87774f69a715/config/ceph.client.admin.keyring 2024-03-24T00:05:41.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:05:41 smithi118 ceph-mon[31535]: from='client.14182 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi193", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T00:05:41.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:05:41 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T00:05:41.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:05:41 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T00:05:41.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:05:41 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T00:05:42.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:05:42 smithi118 ceph-mon[31535]: Deploying cephadm binary to smithi193 2024-03-24T00:05:42.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:05:42 smithi118 ceph-mon[31535]: pgmap v11: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T00:05:44.705 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:05:44.705+0000 7f41b7400700 1 -- 172.21.15.118:0/395897314 <== mgr.14152 v2:172.21.15.118:6800/2112912943 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+49 (secure 0 0 0) 0x7f41c019ac60 con 0x7f41a4039020 2024-03-24T00:05:44.705 INFO:teuthology.orchestra.run.smithi118.stdout:Added host 'smithi193' with addr '172.21.15.193' 2024-03-24T00:05:44.708 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:05:44.708+0000 7f41c5400700 1 -- 172.21.15.118:0/395897314 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7f41a4039020 msgr2=0x7f41a403b4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:05:44.708 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:05:44.708+0000 7f41c5400700 1 --2- 172.21.15.118:0/395897314 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7f41a4039020 0x7f41a403b4e0 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f41ac006fd0 tx=0x7f41ac006e40 comp rx=0 tx=0).stop 2024-03-24T00:05:44.708 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:05:44.708+0000 7f41c5400700 1 -- 172.21.15.118:0/395897314 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f41c0102e10 msgr2=0x7f41c019f660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:05:44.708 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:05:44.708+0000 7f41c5400700 1 --2- 172.21.15.118:0/395897314 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f41c0102e10 0x7f41c019f660 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f41b0006010 tx=0x7f41b0005030 comp rx=0 tx=0).stop 2024-03-24T00:05:44.709 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:05:44.708+0000 7f41c5400700 1 -- 172.21.15.118:0/395897314 shutdown_connections 2024-03-24T00:05:44.709 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:05:44.708+0000 7f41c5400700 1 --2- 172.21.15.118:0/395897314 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7f41a4039020 0x7f41a403b4e0 unknown :-1 s=CLOSED pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:05:44.709 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:05:44.708+0000 7f41c5400700 1 --2- 172.21.15.118:0/395897314 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f41c0102e10 0x7f41c019f660 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:05:44.709 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:05:44.708+0000 7f41c5400700 1 -- 172.21.15.118:0/395897314 >> 172.21.15.118:0/395897314 conn(0x7f41c0077830 msgr2=0x7f41c0107fd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:05:44.709 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:05:44.709+0000 7f41c5400700 1 -- 172.21.15.118:0/395897314 shutdown_connections 2024-03-24T00:05:44.709 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:05:44.709+0000 7f41c5400700 1 -- 172.21.15.118:0/395897314 wait complete. 2024-03-24T00:05:44.945 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:05:44 smithi118 ceph-mon[31535]: pgmap v12: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T00:05:45.429 DEBUG:teuthology.orchestra.run.smithi118:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d79a5420-e971-11ee-95cd-87774f69a715 -- ceph orch host ls --format=json 2024-03-24T00:05:45.869 INFO:teuthology.orchestra.run.smithi118.stderr:Inferring config /var/lib/ceph/d79a5420-e971-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T00:05:46.014 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:05:45 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T00:05:46.014 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:05:45 smithi118 ceph-mon[31535]: Added host smithi193 2024-03-24T00:05:46.014 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:05:45 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T00:05:46.014 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:05:45 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T00:05:46.014 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:05:45 smithi118 ceph-mon[31535]: pgmap v13: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T00:05:47.093 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:05:46 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T00:05:47.890 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:05:47.890+0000 7f7aaba00700 1 -- 172.21.15.118:0/4128303742 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f7aa4100940 msgr2=0x7f7aa4100d20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:05:47.891 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:05:47.890+0000 7f7aaba00700 1 --2- 172.21.15.118:0/4128303742 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f7aa4100940 0x7f7aa4100d20 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f7a98009910 tx=0x7f7a98009c20 comp rx=0 tx=0).stop 2024-03-24T00:05:47.891 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:05:47.890+0000 7f7aaba00700 1 -- 172.21.15.118:0/4128303742 shutdown_connections 2024-03-24T00:05:47.891 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:05:47.890+0000 7f7aaba00700 1 --2- 172.21.15.118:0/4128303742 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f7aa4100940 0x7f7aa4100d20 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:05:47.891 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:05:47.890+0000 7f7aaba00700 1 -- 172.21.15.118:0/4128303742 >> 172.21.15.118:0/4128303742 conn(0x7f7aa4077830 msgr2=0x7f7aa40febb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:05:47.891 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:05:47.891+0000 7f7aaba00700 1 -- 172.21.15.118:0/4128303742 shutdown_connections 2024-03-24T00:05:47.891 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:05:47.891+0000 7f7aaba00700 1 -- 172.21.15.118:0/4128303742 wait complete. 2024-03-24T00:05:47.892 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:05:47.892+0000 7f7aaba00700 1 Processor -- start 2024-03-24T00:05:47.892 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:05:47.892+0000 7f7aaba00700 1 -- start start 2024-03-24T00:05:47.892 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:05:47.892+0000 7f7aaba00700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f7aa4100940 0x7f7aa419f600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:05:47.892 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:05:47.892+0000 7f7aaba00700 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7aa410e3e0 con 0x7f7aa4100940 2024-03-24T00:05:47.892 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:05:47.892+0000 7f7aa9000700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f7aa4100940 0x7f7aa419f600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:05:47.892 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:05:47.893+0000 7f7aa9000700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f7aa4100940 0x7f7aa419f600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.118:3300/0 says I am v2:172.21.15.118:53688/0 (socket says 172.21.15.118:53688) 2024-03-24T00:05:47.892 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:05:47.893+0000 7f7aa9000700 1 -- 172.21.15.118:0/2202475254 learned_addr learned my addr 172.21.15.118:0/2202475254 (peer_addr_for_me v2:172.21.15.118:0/0) 2024-03-24T00:05:47.893 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:05:47.893+0000 7f7aa9000700 1 -- 172.21.15.118:0/2202475254 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7aa419fb40 con 0x7f7aa4100940 2024-03-24T00:05:47.893 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:05:47.893+0000 7f7aa9000700 1 --2- 172.21.15.118:0/2202475254 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f7aa4100940 0x7f7aa419f600 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f7a98006010 tx=0x7f7a98005030 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:05:47.893 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:05:47.893+0000 7f7aa1600700 1 -- 172.21.15.118:0/2202475254 <== mon.0 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f7a98026070 con 0x7f7aa4100940 2024-03-24T00:05:47.893 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:05:47.893+0000 7f7aa1600700 1 -- 172.21.15.118:0/2202475254 <== mon.0 v2:172.21.15.118:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f7a9800bc30 con 0x7f7aa4100940 2024-03-24T00:05:47.893 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:05:47.893+0000 7f7aaba00700 1 -- 172.21.15.118:0/2202475254 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7aa419fdd0 con 0x7f7aa4100940 2024-03-24T00:05:47.894 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:05:47.893+0000 7f7aa1600700 1 -- 172.21.15.118:0/2202475254 <== mon.0 v2:172.21.15.118:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f7a980039d0 con 0x7f7aa4100940 2024-03-24T00:05:47.894 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:05:47.894+0000 7f7aaba00700 1 -- 172.21.15.118:0/2202475254 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7aa41996d0 con 0x7f7aa4100940 2024-03-24T00:05:47.894 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:05:47.895+0000 7f7aa1600700 1 -- 172.21.15.118:0/2202475254 <== mon.0 v2:172.21.15.118:3300/0 4 ==== mgrmap(e 13) v1 ==== 46163+0+0 (secure 0 0 0) 0x7f7a98029070 con 0x7f7aa4100940 2024-03-24T00:05:47.894 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:05:47.895+0000 7f7aaba00700 1 -- 172.21.15.118:0/2202475254 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7aa4069dc0 con 0x7f7aa4100940 2024-03-24T00:05:47.895 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:05:47.895+0000 7f7aa1600700 1 --2- 172.21.15.118:0/2202475254 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7f7a8c039070 0x7f7a8c03b530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:05:47.895 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:05:47.895+0000 7f7aa1600700 1 -- 172.21.15.118:0/2202475254 <== mon.0 v2:172.21.15.118:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1155+0+0 (secure 0 0 0) 0x7f7a9801e070 con 0x7f7aa4100940 2024-03-24T00:05:47.895 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:05:47.895+0000 7f7aa3e00700 1 --2- 172.21.15.118:0/2202475254 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7f7a8c039070 0x7f7a8c03b530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:05:47.895 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:05:47.896+0000 7f7aa3e00700 1 --2- 172.21.15.118:0/2202475254 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7f7a8c039070 0x7f7a8c03b530 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7f7a90006fd0 tx=0x7f7a90006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:05:47.900 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:05:47.901+0000 7f7aa1600700 1 -- 172.21.15.118:0/2202475254 <== mon.0 v2:172.21.15.118:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f7a98013070 con 0x7f7aa4100940 2024-03-24T00:05:48.128 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:05:48.129+0000 7f7aaba00700 1 -- 172.21.15.118:0/2202475254 --> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] -- mgr_command(tid 0: {"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f7aa419ac00 con 0x7f7a8c039070 2024-03-24T00:05:48.130 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:05:48.131+0000 7f7aa1600700 1 -- 172.21.15.118:0/2202475254 <== mgr.14152 v2:172.21.15.118:6800/2112912943 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+241 (secure 0 0 0) 0x7f7aa419ac00 con 0x7f7a8c039070 2024-03-24T00:05:48.130 INFO:teuthology.orchestra.run.smithi118.stdout: 2024-03-24T00:05:48.130 INFO:teuthology.orchestra.run.smithi118.stdout:[{"addr": "172.21.15.118", "hostname": "smithi118", "labels": [], "status": ""}, {"addr": "172.21.15.189", "hostname": "smithi189", "labels": [], "status": ""}, {"addr": "172.21.15.193", "hostname": "smithi193", "labels": [], "status": ""}] 2024-03-24T00:05:48.133 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:05:48.133+0000 7f7aaba00700 1 -- 172.21.15.118:0/2202475254 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7f7a8c039070 msgr2=0x7f7a8c03b530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:05:48.133 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:05:48.134+0000 7f7aaba00700 1 --2- 172.21.15.118:0/2202475254 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7f7a8c039070 0x7f7a8c03b530 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7f7a90006fd0 tx=0x7f7a90006e40 comp rx=0 tx=0).stop 2024-03-24T00:05:48.133 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:05:48.134+0000 7f7aaba00700 1 -- 172.21.15.118:0/2202475254 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f7aa4100940 msgr2=0x7f7aa419f600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:05:48.133 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:05:48.134+0000 7f7aaba00700 1 --2- 172.21.15.118:0/2202475254 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f7aa4100940 0x7f7aa419f600 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f7a98006010 tx=0x7f7a98005030 comp rx=0 tx=0).stop 2024-03-24T00:05:48.134 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:05:48.134+0000 7f7aaba00700 1 -- 172.21.15.118:0/2202475254 shutdown_connections 2024-03-24T00:05:48.134 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:05:48.134+0000 7f7aaba00700 1 --2- 172.21.15.118:0/2202475254 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7f7a8c039070 0x7f7a8c03b530 unknown :-1 s=CLOSED pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:05:48.134 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:05:48.134+0000 7f7aaba00700 1 --2- 172.21.15.118:0/2202475254 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f7aa4100940 0x7f7aa419f600 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:05:48.134 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:05:48.134+0000 7f7aaba00700 1 -- 172.21.15.118:0/2202475254 >> 172.21.15.118:0/2202475254 conn(0x7f7aa4077830 msgr2=0x7f7aa40fe780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:05:48.135 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:05:48.134+0000 7f7aaba00700 1 -- 172.21.15.118:0/2202475254 shutdown_connections 2024-03-24T00:05:48.135 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:05:48.134+0000 7f7aaba00700 1 -- 172.21.15.118:0/2202475254 wait complete. 2024-03-24T00:05:48.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:05:48 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T00:05:48.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:05:48 smithi118 ceph-mon[31535]: pgmap v14: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T00:05:48.931 INFO:tasks.cephadm:Setting crush tunables to default 2024-03-24T00:05:48.931 DEBUG:teuthology.orchestra.run.smithi118:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d79a5420-e971-11ee-95cd-87774f69a715 -- ceph osd crush tunables default 2024-03-24T00:05:49.370 INFO:teuthology.orchestra.run.smithi118.stderr:Inferring config /var/lib/ceph/d79a5420-e971-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T00:05:49.512 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:05:49 smithi118 ceph-mon[31535]: from='client.14184 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-24T00:05:50.555 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:05:50 smithi118 ceph-mon[31535]: pgmap v15: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T00:05:51.333 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:05:51.333+0000 7fb0b1600700 1 -- 172.21.15.118:0/2828948644 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb0ac074560 msgr2=0x7fb0ac074940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:05:51.334 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:05:51.333+0000 7fb0b1600700 1 --2- 172.21.15.118:0/2828948644 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb0ac074560 0x7fb0ac074940 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7fb09c009910 tx=0x7fb09c009c20 comp rx=0 tx=0).stop 2024-03-24T00:05:51.334 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:05:51.333+0000 7fb0b1600700 1 -- 172.21.15.118:0/2828948644 shutdown_connections 2024-03-24T00:05:51.334 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:05:51.333+0000 7fb0b1600700 1 --2- 172.21.15.118:0/2828948644 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb0ac074560 0x7fb0ac074940 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:05:51.334 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:05:51.333+0000 7fb0b1600700 1 -- 172.21.15.118:0/2828948644 >> 172.21.15.118:0/2828948644 conn(0x7fb0ac0fd490 msgr2=0x7fb0ac0ff8b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:05:51.334 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:05:51.334+0000 7fb0b1600700 1 -- 172.21.15.118:0/2828948644 shutdown_connections 2024-03-24T00:05:51.334 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:05:51.334+0000 7fb0b1600700 1 -- 172.21.15.118:0/2828948644 wait complete. 2024-03-24T00:05:51.335 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:05:51.335+0000 7fb0b1600700 1 Processor -- start 2024-03-24T00:05:51.335 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:05:51.335+0000 7fb0b1600700 1 -- start start 2024-03-24T00:05:51.335 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:05:51.335+0000 7fb0b1600700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb0ac074560 0x7fb0ac19f400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:05:51.336 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:05:51.335+0000 7fb0b1600700 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb0ac10e220 con 0x7fb0ac074560 2024-03-24T00:05:51.336 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:05:51.336+0000 7fb0abe00700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb0ac074560 0x7fb0ac19f400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:05:51.336 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:05:51.336+0000 7fb0abe00700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb0ac074560 0x7fb0ac19f400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.118:3300/0 says I am v2:172.21.15.118:53698/0 (socket says 172.21.15.118:53698) 2024-03-24T00:05:51.336 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:05:51.336+0000 7fb0abe00700 1 -- 172.21.15.118:0/2863852192 learned_addr learned my addr 172.21.15.118:0/2863852192 (peer_addr_for_me v2:172.21.15.118:0/0) 2024-03-24T00:05:51.336 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:05:51.336+0000 7fb0abe00700 1 -- 172.21.15.118:0/2863852192 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb0ac19f940 con 0x7fb0ac074560 2024-03-24T00:05:51.336 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:05:51.336+0000 7fb0abe00700 1 --2- 172.21.15.118:0/2863852192 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb0ac074560 0x7fb0ac19f400 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7fb09c006010 tx=0x7fb09c005030 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:05:51.337 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:05:51.336+0000 7fb0a8c00700 1 -- 172.21.15.118:0/2863852192 <== mon.0 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fb09c026070 con 0x7fb0ac074560 2024-03-24T00:05:51.337 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:05:51.337+0000 7fb0a8c00700 1 -- 172.21.15.118:0/2863852192 <== mon.0 v2:172.21.15.118:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fb09c00bc30 con 0x7fb0ac074560 2024-03-24T00:05:51.337 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:05:51.337+0000 7fb0b1600700 1 -- 172.21.15.118:0/2863852192 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb0ac19fbd0 con 0x7fb0ac074560 2024-03-24T00:05:51.337 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:05:51.337+0000 7fb0a8c00700 1 -- 172.21.15.118:0/2863852192 <== mon.0 v2:172.21.15.118:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7fb09c0039a0 con 0x7fb0ac074560 2024-03-24T00:05:51.337 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:05:51.337+0000 7fb0b1600700 1 -- 172.21.15.118:0/2863852192 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb0ac1995c0 con 0x7fb0ac074560 2024-03-24T00:05:51.337 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:05:51.338+0000 7fb0b1600700 1 -- 172.21.15.118:0/2863852192 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb0ac075c10 con 0x7fb0ac074560 2024-03-24T00:05:51.338 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:05:51.338+0000 7fb0a8c00700 1 -- 172.21.15.118:0/2863852192 <== mon.0 v2:172.21.15.118:3300/0 4 ==== mgrmap(e 13) v1 ==== 46163+0+0 (secure 0 0 0) 0x7fb09c029070 con 0x7fb0ac074560 2024-03-24T00:05:51.338 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:05:51.339+0000 7fb0a8c00700 1 --2- 172.21.15.118:0/2863852192 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7fb090038fc0 0x7fb09003b480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:05:51.338 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:05:51.339+0000 7fb0a8c00700 1 -- 172.21.15.118:0/2863852192 <== mon.0 v2:172.21.15.118:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1155+0+0 (secure 0 0 0) 0x7fb09c01e070 con 0x7fb0ac074560 2024-03-24T00:05:51.338 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:05:51.339+0000 7fb0ab400700 1 --2- 172.21.15.118:0/2863852192 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7fb090038fc0 0x7fb09003b480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:05:51.339 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:05:51.340+0000 7fb0ab400700 1 --2- 172.21.15.118:0/2863852192 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7fb090038fc0 0x7fb09003b480 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7fb098006fd0 tx=0x7fb098006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:05:51.344 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:05:51.344+0000 7fb0a8c00700 1 -- 172.21.15.118:0/2863852192 <== mon.0 v2:172.21.15.118:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7fb09c013070 con 0x7fb0ac074560 2024-03-24T00:05:51.578 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:05:51.579+0000 7fb0b1600700 1 -- 172.21.15.118:0/2863852192 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_command({"prefix": "osd crush tunables", "profile": "default"} v 0) v1 -- 0x7fb0ac0637c0 con 0x7fb0ac074560 2024-03-24T00:05:51.617 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:05:51.617+0000 7fb0a8c00700 1 -- 172.21.15.118:0/2863852192 <== mon.0 v2:172.21.15.118:3300/0 7 ==== mon_command_ack([{"prefix": "osd crush tunables", "profile": "default"}]=0 adjusted tunables profile to default v4) v1 ==== 124+0+0 (secure 0 0 0) 0x7fb09c030090 con 0x7fb0ac074560 2024-03-24T00:05:51.621 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:05:51.621+0000 7fb0b1600700 1 -- 172.21.15.118:0/2863852192 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7fb090038fc0 msgr2=0x7fb09003b480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:05:51.621 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:05:51.621+0000 7fb0b1600700 1 --2- 172.21.15.118:0/2863852192 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7fb090038fc0 0x7fb09003b480 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7fb098006fd0 tx=0x7fb098006e40 comp rx=0 tx=0).stop 2024-03-24T00:05:51.621 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:05:51.621+0000 7fb0b1600700 1 -- 172.21.15.118:0/2863852192 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb0ac074560 msgr2=0x7fb0ac19f400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:05:51.621 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:05:51.621+0000 7fb0b1600700 1 --2- 172.21.15.118:0/2863852192 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb0ac074560 0x7fb0ac19f400 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7fb09c006010 tx=0x7fb09c005030 comp rx=0 tx=0).stop 2024-03-24T00:05:51.622 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:05:51.622+0000 7fb0b1600700 1 -- 172.21.15.118:0/2863852192 shutdown_connections 2024-03-24T00:05:51.622 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:05:51.622+0000 7fb0b1600700 1 --2- 172.21.15.118:0/2863852192 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7fb090038fc0 0x7fb09003b480 unknown :-1 s=CLOSED pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:05:51.622 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:05:51.622+0000 7fb0b1600700 1 --2- 172.21.15.118:0/2863852192 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb0ac074560 0x7fb0ac19f400 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:05:51.622 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:05:51.622+0000 7fb0b1600700 1 -- 172.21.15.118:0/2863852192 >> 172.21.15.118:0/2863852192 conn(0x7fb0ac0fd490 msgr2=0x7fb0ac105c80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:05:51.622 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:05:51.622+0000 7fb0b1600700 1 -- 172.21.15.118:0/2863852192 shutdown_connections 2024-03-24T00:05:51.622 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:05:51.622+0000 7fb0b1600700 1 -- 172.21.15.118:0/2863852192 wait complete. 2024-03-24T00:05:51.623 INFO:teuthology.orchestra.run.smithi118.stderr:adjusted tunables profile to default 2024-03-24T00:05:51.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:05:51 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T00:05:51.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:05:51 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T00:05:51.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:05:51 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T00:05:51.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:05:51 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T00:05:51.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:05:51 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd/host:smithi193", "name": "osd_memory_target"}]: dispatch 2024-03-24T00:05:51.947 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:05:51 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T00:05:51.947 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:05:51 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T00:05:51.947 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:05:51 smithi118 ceph-mon[31535]: Updating smithi193:/etc/ceph/ceph.conf 2024-03-24T00:05:51.947 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:05:51 smithi118 ceph-mon[31535]: from='client.? 172.21.15.118:0/2863852192' entity='client.admin' cmd=[{"prefix": "osd crush tunables", "profile": "default"}]: dispatch 2024-03-24T00:05:52.341 INFO:tasks.cephadm:Adding mon.a on smithi118 2024-03-24T00:05:52.341 INFO:tasks.cephadm:Adding mon.b on smithi189 2024-03-24T00:05:52.341 INFO:tasks.cephadm:Adding mon.c on smithi193 2024-03-24T00:05:52.342 DEBUG:teuthology.orchestra.run.smithi193:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d79a5420-e971-11ee-95cd-87774f69a715 -- ceph orch apply mon '3;smithi118:172.21.15.118=a;smithi189:172.21.15.189=b;smithi193:172.21.15.193=c' 2024-03-24T00:05:52.566 INFO:teuthology.orchestra.run.smithi193.stderr:Inferring config /var/lib/ceph/d79a5420-e971-11ee-95cd-87774f69a715/config/ceph.conf 2024-03-24T00:05:52.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:05:52 smithi118 ceph-mon[31535]: Updating smithi193:/var/lib/ceph/d79a5420-e971-11ee-95cd-87774f69a715/config/ceph.conf 2024-03-24T00:05:52.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:05:52 smithi118 ceph-mon[31535]: Updating smithi193:/etc/ceph/ceph.client.admin.keyring 2024-03-24T00:05:52.947 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:05:52 smithi118 ceph-mon[31535]: Updating smithi193:/var/lib/ceph/d79a5420-e971-11ee-95cd-87774f69a715/config/ceph.client.admin.keyring 2024-03-24T00:05:52.947 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:05:52 smithi118 ceph-mon[31535]: pgmap v16: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T00:05:52.947 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:05:52 smithi118 ceph-mon[31535]: from='client.? 172.21.15.118:0/2863852192' entity='client.admin' cmd='[{"prefix": "osd crush tunables", "profile": "default"}]': finished 2024-03-24T00:05:52.947 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:05:52 smithi118 ceph-mon[31535]: osdmap e4: 0 total, 0 up, 0 in 2024-03-24T00:05:52.947 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:05:52 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T00:05:52.947 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:05:52 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T00:05:52.947 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:05:52 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T00:05:53.998 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-24T00:05:53.998+0000 7f1ca58bd700 1 -- 172.21.15.193:0/1407274302 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f1ca0106a70 msgr2=0x7f1ca0106e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:05:53.998 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-24T00:05:53.998+0000 7f1ca58bd700 1 --2- 172.21.15.193:0/1407274302 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f1ca0106a70 0x7f1ca0106e50 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f1c90009910 tx=0x7f1c90009c20 comp rx=0 tx=0).stop 2024-03-24T00:05:53.999 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-24T00:05:53.998+0000 7f1ca58bd700 1 -- 172.21.15.193:0/1407274302 shutdown_connections 2024-03-24T00:05:53.999 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-24T00:05:53.998+0000 7f1ca58bd700 1 --2- 172.21.15.193:0/1407274302 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f1ca0106a70 0x7f1ca0106e50 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:05:53.999 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-24T00:05:53.998+0000 7f1ca58bd700 1 -- 172.21.15.193:0/1407274302 >> 172.21.15.193:0/1407274302 conn(0x7f1ca00fc5a0 msgr2=0x7f1ca00fe9c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:05:53.999 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-24T00:05:53.998+0000 7f1ca58bd700 1 -- 172.21.15.193:0/1407274302 shutdown_connections 2024-03-24T00:05:53.999 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-24T00:05:53.998+0000 7f1ca58bd700 1 -- 172.21.15.193:0/1407274302 wait complete. 2024-03-24T00:05:53.999 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-24T00:05:53.998+0000 7f1ca58bd700 1 Processor -- start 2024-03-24T00:05:53.999 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-24T00:05:53.999+0000 7f1ca58bd700 1 -- start start 2024-03-24T00:05:54.001 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-24T00:05:53.999+0000 7f1ca58bd700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f1ca0106a70 0x7f1ca0196620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:05:54.001 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-24T00:05:53.999+0000 7f1ca58bd700 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1ca010b0b0 con 0x7f1ca0106a70 2024-03-24T00:05:54.001 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-24T00:05:53.999+0000 7f1c9effd700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f1ca0106a70 0x7f1ca0196620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:05:54.001 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-24T00:05:53.999+0000 7f1c9effd700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f1ca0106a70 0x7f1ca0196620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.118:3300/0 says I am v2:172.21.15.193:41638/0 (socket says 172.21.15.193:41638) 2024-03-24T00:05:54.001 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-24T00:05:53.999+0000 7f1c9effd700 1 -- 172.21.15.193:0/1816488901 learned_addr learned my addr 172.21.15.193:0/1816488901 (peer_addr_for_me v2:172.21.15.193:0/0) 2024-03-24T00:05:54.001 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-24T00:05:54.000+0000 7f1c9effd700 1 -- 172.21.15.193:0/1816488901 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1ca0196b60 con 0x7f1ca0106a70 2024-03-24T00:05:54.001 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-24T00:05:54.000+0000 7f1c9effd700 1 --2- 172.21.15.193:0/1816488901 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f1ca0106a70 0x7f1ca0196620 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f1c90005850 tx=0x7f1c90005030 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:05:54.001 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-24T00:05:54.001+0000 7f1ca48bb700 1 -- 172.21.15.193:0/1816488901 <== mon.0 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f1c90026070 con 0x7f1ca0106a70 2024-03-24T00:05:54.001 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-24T00:05:54.001+0000 7f1ca58bd700 1 -- 172.21.15.193:0/1816488901 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1ca0196e50 con 0x7f1ca0106a70 2024-03-24T00:05:54.001 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-24T00:05:54.001+0000 7f1ca48bb700 1 -- 172.21.15.193:0/1816488901 <== mon.0 v2:172.21.15.118:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f1c9000bbd0 con 0x7f1ca0106a70 2024-03-24T00:05:54.001 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-24T00:05:54.001+0000 7f1ca58bd700 1 -- 172.21.15.193:0/1816488901 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1ca019aef0 con 0x7f1ca0106a70 2024-03-24T00:05:54.001 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-24T00:05:54.001+0000 7f1ca48bb700 1 -- 172.21.15.193:0/1816488901 <== mon.0 v2:172.21.15.118:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f1c900039b0 con 0x7f1ca0106a70 2024-03-24T00:05:54.002 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-24T00:05:54.002+0000 7f1ca58bd700 1 -- 172.21.15.193:0/1816488901 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1ca01021b0 con 0x7f1ca0106a70 2024-03-24T00:05:54.002 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-24T00:05:54.002+0000 7f1ca48bb700 1 -- 172.21.15.193:0/1816488901 <== mon.0 v2:172.21.15.118:3300/0 4 ==== mgrmap(e 13) v1 ==== 46163+0+0 (secure 0 0 0) 0x7f1c90029070 con 0x7f1ca0106a70 2024-03-24T00:05:54.002 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-24T00:05:54.002+0000 7f1ca48bb700 1 --2- 172.21.15.193:0/1816488901 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7f1c88039020 0x7f1c8803b4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:05:54.002 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-24T00:05:54.002+0000 7f1ca48bb700 1 -- 172.21.15.193:0/1816488901 <== mon.0 v2:172.21.15.118:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1155+0+0 (secure 0 0 0) 0x7f1c9001e070 con 0x7f1ca0106a70 2024-03-24T00:05:54.003 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-24T00:05:54.003+0000 7f1c9e7fc700 1 --2- 172.21.15.193:0/1816488901 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7f1c88039020 0x7f1c8803b4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:05:54.003 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-24T00:05:54.004+0000 7f1c9e7fc700 1 --2- 172.21.15.193:0/1816488901 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7f1c88039020 0x7f1c8803b4e0 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f1c94006fd0 tx=0x7f1c94006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:05:54.016 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-24T00:05:54.016+0000 7f1ca48bb700 1 -- 172.21.15.193:0/1816488901 <== mon.0 v2:172.21.15.118:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f1c90013070 con 0x7f1ca0106a70 2024-03-24T00:05:54.219 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-24T00:05:54.219+0000 7f1ca58bd700 1 -- 172.21.15.193:0/1816488901 --> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "mon", "placement": "3;smithi118:172.21.15.118=a;smithi189:172.21.15.189=b;smithi193:172.21.15.193=c", "target": ["mon-mgr", ""]}) v1 -- 0x7f1ca0197980 con 0x7f1c88039020 2024-03-24T00:05:54.234 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-24T00:05:54.235+0000 7f1ca48bb700 1 -- 172.21.15.193:0/1816488901 <== mgr.14152 v2:172.21.15.118:6800/2112912943 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+24 (secure 0 0 0) 0x7f1ca0197980 con 0x7f1c88039020 2024-03-24T00:05:54.235 INFO:teuthology.orchestra.run.smithi193.stdout:Scheduled mon update... 2024-03-24T00:05:54.237 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-24T00:05:54.237+0000 7f1ca58bd700 1 -- 172.21.15.193:0/1816488901 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7f1c88039020 msgr2=0x7f1c8803b4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:05:54.237 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-24T00:05:54.237+0000 7f1ca58bd700 1 --2- 172.21.15.193:0/1816488901 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7f1c88039020 0x7f1c8803b4e0 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f1c94006fd0 tx=0x7f1c94006e40 comp rx=0 tx=0).stop 2024-03-24T00:05:54.237 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-24T00:05:54.237+0000 7f1ca58bd700 1 -- 172.21.15.193:0/1816488901 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f1ca0106a70 msgr2=0x7f1ca0196620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:05:54.237 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-24T00:05:54.237+0000 7f1ca58bd700 1 --2- 172.21.15.193:0/1816488901 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f1ca0106a70 0x7f1ca0196620 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f1c90005850 tx=0x7f1c90005030 comp rx=0 tx=0).stop 2024-03-24T00:05:54.237 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-24T00:05:54.237+0000 7f1ca58bd700 1 -- 172.21.15.193:0/1816488901 shutdown_connections 2024-03-24T00:05:54.237 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-24T00:05:54.237+0000 7f1ca58bd700 1 --2- 172.21.15.193:0/1816488901 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7f1c88039020 0x7f1c8803b4e0 unknown :-1 s=CLOSED pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:05:54.238 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-24T00:05:54.237+0000 7f1ca58bd700 1 --2- 172.21.15.193:0/1816488901 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f1ca0106a70 0x7f1ca0196620 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:05:54.238 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-24T00:05:54.237+0000 7f1ca58bd700 1 -- 172.21.15.193:0/1816488901 >> 172.21.15.193:0/1816488901 conn(0x7f1ca00fc5a0 msgr2=0x7f1ca0108620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:05:54.238 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-24T00:05:54.237+0000 7f1ca58bd700 1 -- 172.21.15.193:0/1816488901 shutdown_connections 2024-03-24T00:05:54.238 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-24T00:05:54.237+0000 7f1ca58bd700 1 -- 172.21.15.193:0/1816488901 wait complete. 2024-03-24T00:05:54.910 DEBUG:teuthology.orchestra.run.smithi189:mon.b> sudo journalctl -f -n 0 -u ceph-d79a5420-e971-11ee-95cd-87774f69a715@mon.b.service 2024-03-24T00:05:54.912 DEBUG:teuthology.orchestra.run.smithi193:mon.c> sudo journalctl -f -n 0 -u ceph-d79a5420-e971-11ee-95cd-87774f69a715@mon.c.service 2024-03-24T00:05:54.915 INFO:tasks.cephadm:Waiting for 3 mons in monmap... 2024-03-24T00:05:54.915 DEBUG:teuthology.orchestra.run.smithi193:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d79a5420-e971-11ee-95cd-87774f69a715 -- ceph mon dump -f json 2024-03-24T00:05:54.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:05:54 smithi118 ceph-mon[31535]: pgmap v18: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T00:05:54.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:05:54 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T00:05:54.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:05:54 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T00:05:54.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:05:54 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T00:05:54.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:05:54 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T00:05:54.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:05:54 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T00:05:54.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:05:54 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-24T00:05:54.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:05:54 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T00:05:55.809 INFO:teuthology.orchestra.run.smithi193.stderr:Inferring config /var/lib/ceph/d79a5420-e971-11ee-95cd-87774f69a715/config/ceph.conf 2024-03-24T00:05:55.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:05:55 smithi118 ceph-mon[31535]: from='client.14188 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "placement": "3;smithi118:172.21.15.118=a;smithi189:172.21.15.189=b;smithi193:172.21.15.193=c", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T00:05:55.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:05:55 smithi118 ceph-mon[31535]: Saving service mon spec with placement smithi118:172.21.15.118=a;smithi189:172.21.15.189=b;smithi193:172.21.15.193=c;count:3 2024-03-24T00:05:55.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:05:55 smithi118 ceph-mon[31535]: Deploying daemon mon.c on smithi193 2024-03-24T00:05:56.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:05:56 smithi118 ceph-mon[31535]: pgmap v19: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T00:05:58.177 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-24T00:05:58.176+0000 7f263a955700 1 -- 172.21.15.193:0/890533441 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f2634106a40 msgr2=0x7f2634106e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:05:58.178 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-24T00:05:58.176+0000 7f263a955700 1 --2- 172.21.15.193:0/890533441 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f2634106a40 0x7f2634106e20 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f2624009910 tx=0x7f2624009c20 comp rx=0 tx=0).stop 2024-03-24T00:05:58.178 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-24T00:05:58.176+0000 7f263a955700 1 -- 172.21.15.193:0/890533441 shutdown_connections 2024-03-24T00:05:58.178 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-24T00:05:58.176+0000 7f263a955700 1 --2- 172.21.15.193:0/890533441 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f2634106a40 0x7f2634106e20 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:05:58.178 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-24T00:05:58.176+0000 7f263a955700 1 -- 172.21.15.193:0/890533441 >> 172.21.15.193:0/890533441 conn(0x7f2634077830 msgr2=0x7f26340febb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:05:58.178 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-24T00:05:58.176+0000 7f263a955700 1 -- 172.21.15.193:0/890533441 shutdown_connections 2024-03-24T00:05:58.178 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-24T00:05:58.177+0000 7f263a955700 1 -- 172.21.15.193:0/890533441 wait complete. 2024-03-24T00:05:58.178 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-24T00:05:58.177+0000 7f263a955700 1 Processor -- start 2024-03-24T00:05:58.178 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-24T00:05:58.177+0000 7f263a955700 1 -- start start 2024-03-24T00:05:58.178 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-24T00:05:58.178+0000 7f263a955700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f2634106a40 0x7f2634196540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:05:58.178 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-24T00:05:58.178+0000 7f263a955700 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f263410b080 con 0x7f2634106a40 2024-03-24T00:05:58.179 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-24T00:05:58.178+0000 7f2633fff700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f2634106a40 0x7f2634196540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:05:58.180 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-24T00:05:58.178+0000 7f2633fff700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f2634106a40 0x7f2634196540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.118:3300/0 says I am v2:172.21.15.193:41656/0 (socket says 172.21.15.193:41656) 2024-03-24T00:05:58.180 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-24T00:05:58.178+0000 7f2633fff700 1 -- 172.21.15.193:0/1148064473 learned_addr learned my addr 172.21.15.193:0/1148064473 (peer_addr_for_me v2:172.21.15.193:0/0) 2024-03-24T00:05:58.180 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-24T00:05:58.179+0000 7f2633fff700 1 -- 172.21.15.193:0/1148064473 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2634196a80 con 0x7f2634106a40 2024-03-24T00:05:58.180 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-24T00:05:58.179+0000 7f2633fff700 1 --2- 172.21.15.193:0/1148064473 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f2634106a40 0x7f2634196540 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f2624005850 tx=0x7f2624005030 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:05:58.180 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-24T00:05:58.180+0000 7f26317fa700 1 -- 172.21.15.193:0/1148064473 <== mon.0 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f2624028070 con 0x7f2634106a40 2024-03-24T00:05:58.180 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-24T00:05:58.180+0000 7f263a955700 1 -- 172.21.15.193:0/1148064473 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2634196d10 con 0x7f2634106a40 2024-03-24T00:05:58.180 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-24T00:05:58.180+0000 7f263a955700 1 -- 172.21.15.193:0/1148064473 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f263419ae10 con 0x7f2634106a40 2024-03-24T00:05:58.180 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-24T00:05:58.180+0000 7f26317fa700 1 -- 172.21.15.193:0/1148064473 <== mon.0 v2:172.21.15.118:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f262400bbd0 con 0x7f2634106a40 2024-03-24T00:05:58.180 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-24T00:05:58.180+0000 7f26317fa700 1 -- 172.21.15.193:0/1148064473 <== mon.0 v2:172.21.15.118:3300/0 3 ==== mon_map magic: 0 v1 ==== 205+0+0 (secure 0 0 0) 0x7f26240039b0 con 0x7f2634106a40 2024-03-24T00:05:58.180 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-24T00:05:58.181+0000 7f263a955700 1 -- 172.21.15.193:0/1148064473 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2620005330 con 0x7f2634106a40 2024-03-24T00:05:58.181 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-24T00:05:58.181+0000 7f26317fa700 1 -- 172.21.15.193:0/1148064473 <== mon.0 v2:172.21.15.118:3300/0 4 ==== mgrmap(e 13) v1 ==== 46163+0+0 (secure 0 0 0) 0x7f262402b070 con 0x7f2634106a40 2024-03-24T00:05:58.181 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-24T00:05:58.182+0000 7f26317fa700 1 --2- 172.21.15.193:0/1148064473 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7f261c038fc0 0x7f261c03b480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:05:58.181 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-24T00:05:58.182+0000 7f26317fa700 1 -- 172.21.15.193:0/1148064473 <== mon.0 v2:172.21.15.118:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1155+0+0 (secure 0 0 0) 0x7f2624020070 con 0x7f2634106a40 2024-03-24T00:05:58.182 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-24T00:05:58.182+0000 7f26337fe700 1 --2- 172.21.15.193:0/1148064473 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7f261c038fc0 0x7f261c03b480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:05:58.183 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-24T00:05:58.183+0000 7f26337fe700 1 --2- 172.21.15.193:0/1148064473 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7f261c038fc0 0x7f261c03b480 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f2628006fd0 tx=0x7f2628006e40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:05:58.196 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-24T00:05:58.197+0000 7f26317fa700 1 -- 172.21.15.193:0/1148064473 <== mon.0 v2:172.21.15.118:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f2624013070 con 0x7f2634106a40 2024-03-24T00:05:58.496 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-24T00:05:58.496+0000 7f263a955700 1 -- 172.21.15.193:0/1148064473 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f26200058f0 con 0x7f2634106a40 2024-03-24T00:05:58.497 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-24T00:05:58.497+0000 7f26317fa700 1 -- 172.21.15.193:0/1148064473 <== mon.0 v2:172.21.15.118:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+749 (secure 0 0 0) 0x7f262402d440 con 0x7f2634106a40 2024-03-24T00:05:58.497 INFO:teuthology.orchestra.run.smithi193.stdout: 2024-03-24T00:05:58.498 INFO:teuthology.orchestra.run.smithi193.stdout:{"epoch":1,"fsid":"d79a5420-e971-11ee-95cd-87774f69a715","modified":"2024-03-24T00:03:54.425344Z","created":"2024-03-24T00:03:54.425344Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:3300","nonce":0},{"type":"v1","addr":"172.21.15.118:6789","nonce":0}]},"addr":"172.21.15.118:6789/0","public_addr":"172.21.15.118:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-03-24T00:05:58.499 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-24T00:05:58.499+0000 7f263a955700 1 -- 172.21.15.193:0/1148064473 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7f261c038fc0 msgr2=0x7f261c03b480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:05:58.500 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-24T00:05:58.499+0000 7f263a955700 1 --2- 172.21.15.193:0/1148064473 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7f261c038fc0 0x7f261c03b480 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f2628006fd0 tx=0x7f2628006e40 comp rx=0 tx=0).stop 2024-03-24T00:05:58.500 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-24T00:05:58.499+0000 7f263a955700 1 -- 172.21.15.193:0/1148064473 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f2634106a40 msgr2=0x7f2634196540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:05:58.500 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-24T00:05:58.499+0000 7f263a955700 1 --2- 172.21.15.193:0/1148064473 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f2634106a40 0x7f2634196540 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f2624005850 tx=0x7f2624005030 comp rx=0 tx=0).stop 2024-03-24T00:05:58.500 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-24T00:05:58.500+0000 7f263a955700 1 -- 172.21.15.193:0/1148064473 shutdown_connections 2024-03-24T00:05:58.500 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-24T00:05:58.500+0000 7f263a955700 1 --2- 172.21.15.193:0/1148064473 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7f261c038fc0 0x7f261c03b480 unknown :-1 s=CLOSED pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:05:58.500 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-24T00:05:58.500+0000 7f263a955700 1 --2- 172.21.15.193:0/1148064473 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f2634106a40 0x7f2634196540 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:05:58.500 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-24T00:05:58.500+0000 7f263a955700 1 -- 172.21.15.193:0/1148064473 >> 172.21.15.193:0/1148064473 conn(0x7f2634077830 msgr2=0x7f2634102540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:05:58.500 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-24T00:05:58.500+0000 7f263a955700 1 -- 172.21.15.193:0/1148064473 shutdown_connections 2024-03-24T00:05:58.500 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-24T00:05:58.500+0000 7f263a955700 1 -- 172.21.15.193:0/1148064473 wait complete. 2024-03-24T00:05:58.501 INFO:teuthology.orchestra.run.smithi193.stderr:dumped monmap epoch 1 2024-03-24T00:05:58.711 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:05:58 smithi193 systemd[1]: Starting Ceph mon.c for d79a5420-e971-11ee-95cd-87774f69a715... 2024-03-24T00:05:58.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:05:58 smithi118 ceph-mon[31535]: pgmap v20: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T00:05:58.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:05:58 smithi118 ceph-mon[31535]: from='client.? 172.21.15.193:0/1148064473' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-24T00:05:59.711 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:05:59 smithi193 podman[36858]: 2024-03-24 00:05:59.211451283 +0000 UTC m=+0.089065701 container create 0284545a88c562a09f5395625948e2d9c1fc6f6487ee318a6e852bc19f0211ae (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-d79a5420-e971-11ee-95cd-87774f69a715-mon-c, org.label-schema.build-date=20240319, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, GIT_CLEAN=True, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, ceph=True) 2024-03-24T00:05:59.711 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:05:59 smithi193 podman[36858]: 2024-03-24 00:05:59.13981102 +0000 UTC m=+0.017425425 image pull c664afe56bb01db4f9afa9532c20598e201e664f2e9ba2be50db0d52674a26d2 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc 2024-03-24T00:05:59.712 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:05:59 smithi193 podman[36858]: 2024-03-24 00:05:59.380479586 +0000 UTC m=+0.258094000 container init 0284545a88c562a09f5395625948e2d9c1fc6f6487ee318a6e852bc19f0211ae (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-d79a5420-e971-11ee-95cd-87774f69a715-mon-c, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240319, ceph=True, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c) 2024-03-24T00:05:59.712 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:05:59 smithi193 podman[36858]: 2024-03-24 00:05:59.383298432 +0000 UTC m=+0.260912822 container start 0284545a88c562a09f5395625948e2d9c1fc6f6487ee318a6e852bc19f0211ae (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-d79a5420-e971-11ee-95cd-87774f69a715-mon-c, CEPH_POINT_RELEASE=, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, org.label-schema.license=GPLv2, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240319, GIT_CLEAN=True, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0) 2024-03-24T00:05:59.712 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:05:59 smithi193 ceph-mon[36878]: set uid:gid to 167:167 (ceph:ceph) 2024-03-24T00:05:59.712 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:05:59 smithi193 ceph-mon[36878]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev), process ceph-mon, pid 2 2024-03-24T00:05:59.712 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:05:59 smithi193 ceph-mon[36878]: pidfile_write: ignore empty --pid-file 2024-03-24T00:05:59.712 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:05:59 smithi193 ceph-mon[36878]: load: jerasure load: lrc 2024-03-24T00:05:59.712 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:05:59 smithi193 ceph-mon[36878]: rocksdb: RocksDB version: 7.9.2 2024-03-24T00:05:59.712 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:05:59 smithi193 ceph-mon[36878]: rocksdb: Git sha 0 2024-03-24T00:05:59.712 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:05:59 smithi193 ceph-mon[36878]: rocksdb: Compile date 2024-03-22 04:58:55 2024-03-24T00:05:59.712 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:05:59 smithi193 ceph-mon[36878]: rocksdb: DB SUMMARY 2024-03-24T00:05:59.713 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:05:59 smithi193 ceph-mon[36878]: rocksdb: DB Session ID: LP9C8V7XRW2SBXA7PZDM 2024-03-24T00:05:59.713 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:05:59 smithi193 ceph-mon[36878]: rocksdb: CURRENT file: CURRENT 2024-03-24T00:05:59.713 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:05:59 smithi193 ceph-mon[36878]: rocksdb: IDENTITY file: IDENTITY 2024-03-24T00:05:59.713 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:05:59 smithi193 ceph-mon[36878]: rocksdb: MANIFEST file: MANIFEST-000005 size: 59 Bytes 2024-03-24T00:05:59.713 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:05:59 smithi193 ceph-mon[36878]: rocksdb: SST files in /var/lib/ceph/mon/ceph-c/store.db dir, Total Num: 0, files: 2024-03-24T00:05:59.713 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:05:59 smithi193 ceph-mon[36878]: rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-c/store.db: 000004.log size: 511 ; 2024-03-24T00:05:59.713 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:05:59 smithi193 ceph-mon[36878]: rocksdb: Options.error_if_exists: 0 2024-03-24T00:05:59.713 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:05:59 smithi193 ceph-mon[36878]: rocksdb: Options.create_if_missing: 0 2024-03-24T00:05:59.713 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:05:59 smithi193 ceph-mon[36878]: rocksdb: Options.paranoid_checks: 1 2024-03-24T00:05:59.713 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:05:59 smithi193 ceph-mon[36878]: rocksdb: Options.flush_verify_memtable_count: 1 2024-03-24T00:05:59.713 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:05:59 smithi193 ceph-mon[36878]: rocksdb: Options.track_and_verify_wals_in_manifest: 0 2024-03-24T00:05:59.714 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:05:59 smithi193 ceph-mon[36878]: rocksdb: Options.verify_sst_unique_id_in_manifest: 1 2024-03-24T00:05:59.714 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:05:59 smithi193 ceph-mon[36878]: rocksdb: Options.env: 0x5607ac351b20 2024-03-24T00:05:59.714 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:05:59 smithi193 ceph-mon[36878]: rocksdb: Options.fs: PosixFileSystem 2024-03-24T00:05:59.714 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:05:59 smithi193 ceph-mon[36878]: rocksdb: Options.info_log: 0x5607aeb19920 2024-03-24T00:05:59.714 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:05:59 smithi193 ceph-mon[36878]: rocksdb: Options.max_file_opening_threads: 16 2024-03-24T00:05:59.714 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:05:59 smithi193 ceph-mon[36878]: rocksdb: Options.statistics: (nil) 2024-03-24T00:05:59.714 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:05:59 smithi193 ceph-mon[36878]: rocksdb: Options.use_fsync: 0 2024-03-24T00:05:59.714 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:05:59 smithi193 ceph-mon[36878]: rocksdb: Options.max_log_file_size: 0 2024-03-24T00:05:59.714 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:05:59 smithi193 ceph-mon[36878]: rocksdb: Options.max_manifest_file_size: 1073741824 2024-03-24T00:05:59.714 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:05:59 smithi193 ceph-mon[36878]: rocksdb: Options.log_file_time_to_roll: 0 2024-03-24T00:05:59.714 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:05:59 smithi193 ceph-mon[36878]: rocksdb: Options.keep_log_file_num: 1000 2024-03-24T00:05:59.715 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:05:59 smithi193 ceph-mon[36878]: rocksdb: Options.recycle_log_file_num: 0 2024-03-24T00:05:59.715 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:05:59 smithi193 ceph-mon[36878]: rocksdb: Options.allow_fallocate: 1 2024-03-24T00:05:59.715 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:05:59 smithi193 ceph-mon[36878]: rocksdb: Options.allow_mmap_reads: 0 2024-03-24T00:05:59.715 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:05:59 smithi193 ceph-mon[36878]: rocksdb: Options.allow_mmap_writes: 0 2024-03-24T00:05:59.715 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:05:59 smithi193 ceph-mon[36878]: rocksdb: Options.use_direct_reads: 0 2024-03-24T00:05:59.715 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:05:59 smithi193 ceph-mon[36878]: rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2024-03-24T00:05:59.715 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:05:59 smithi193 ceph-mon[36878]: rocksdb: Options.create_missing_column_families: 0 2024-03-24T00:05:59.715 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:05:59 smithi193 ceph-mon[36878]: rocksdb: Options.db_log_dir: 2024-03-24T00:05:59.715 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:05:59 smithi193 ceph-mon[36878]: rocksdb: Options.wal_dir: 2024-03-24T00:05:59.715 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:05:59 smithi193 ceph-mon[36878]: rocksdb: Options.table_cache_numshardbits: 6 2024-03-24T00:05:59.715 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:05:59 smithi193 ceph-mon[36878]: rocksdb: Options.WAL_ttl_seconds: 0 2024-03-24T00:05:59.715 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:05:59 smithi193 ceph-mon[36878]: rocksdb: Options.WAL_size_limit_MB: 0 2024-03-24T00:05:59.715 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:05:59 smithi193 ceph-mon[36878]: rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2024-03-24T00:05:59.715 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:05:59 smithi193 ceph-mon[36878]: rocksdb: Options.manifest_preallocation_size: 4194304 2024-03-24T00:05:59.716 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:05:59 smithi193 ceph-mon[36878]: rocksdb: Options.is_fd_close_on_exec: 1 2024-03-24T00:05:59.716 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:05:59 smithi193 ceph-mon[36878]: rocksdb: Options.advise_random_on_open: 1 2024-03-24T00:05:59.716 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:05:59 smithi193 ceph-mon[36878]: rocksdb: Options.db_write_buffer_size: 0 2024-03-24T00:05:59.716 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:05:59 smithi193 ceph-mon[36878]: rocksdb: Options.write_buffer_manager: 0x5607aeaec1e0 2024-03-24T00:05:59.716 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:05:59 smithi193 ceph-mon[36878]: rocksdb: Options.access_hint_on_compaction_start: 1 2024-03-24T00:05:59.716 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:05:59 smithi193 ceph-mon[36878]: rocksdb: Options.random_access_max_buffer_size: 1048576 2024-03-24T00:05:59.716 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:05:59 smithi193 ceph-mon[36878]: rocksdb: Options.use_adaptive_mutex: 0 2024-03-24T00:05:59.718 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:05:59 smithi193 ceph-mon[36878]: rocksdb: Options.rate_limiter: (nil) 2024-03-24T00:05:59.718 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:05:59 smithi193 ceph-mon[36878]: rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2024-03-24T00:05:59.718 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:05:59 smithi193 ceph-mon[36878]: rocksdb: Options.wal_recovery_mode: 2 2024-03-24T00:05:59.718 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:05:59 smithi193 ceph-mon[36878]: rocksdb: Options.enable_thread_tracking: 0 2024-03-24T00:05:59.718 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:05:59 smithi193 ceph-mon[36878]: rocksdb: Options.enable_pipelined_write: 0 2024-03-24T00:05:59.718 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:05:59 smithi193 ceph-mon[36878]: rocksdb: Options.unordered_write: 0 2024-03-24T00:05:59.718 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:05:59 smithi193 ceph-mon[36878]: rocksdb: Options.allow_concurrent_memtable_write: 1 2024-03-24T00:05:59.719 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:05:59 smithi193 ceph-mon[36878]: rocksdb: Options.enable_write_thread_adaptive_yield: 1 2024-03-24T00:05:59.719 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:05:59 smithi193 ceph-mon[36878]: rocksdb: Options.write_thread_max_yield_usec: 100 2024-03-24T00:05:59.719 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:05:59 smithi193 ceph-mon[36878]: rocksdb: Options.write_thread_slow_yield_usec: 3 2024-03-24T00:05:59.719 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:05:59 smithi193 ceph-mon[36878]: rocksdb: Options.row_cache: None 2024-03-24T00:05:59.719 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:05:59 smithi193 ceph-mon[36878]: rocksdb: Options.wal_filter: None 2024-03-24T00:05:59.719 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:05:59 smithi193 ceph-mon[36878]: rocksdb: Options.avoid_flush_during_recovery: 0 2024-03-24T00:05:59.719 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:05:59 smithi193 ceph-mon[36878]: rocksdb: Options.allow_ingest_behind: 0 2024-03-24T00:05:59.719 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:05:59 smithi193 ceph-mon[36878]: rocksdb: Options.two_write_queues: 0 2024-03-24T00:05:59.719 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:05:59 smithi193 ceph-mon[36878]: rocksdb: Options.manual_wal_flush: 0 2024-03-24T00:05:59.719 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:05:59 smithi193 ceph-mon[36878]: rocksdb: Options.wal_compression: 0 2024-03-24T00:05:59.719 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:05:59 smithi193 ceph-mon[36878]: rocksdb: Options.atomic_flush: 0 2024-03-24T00:05:59.719 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:05:59 smithi193 ceph-mon[36878]: rocksdb: Options.avoid_unnecessary_blocking_io: 0 2024-03-24T00:05:59.720 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:05:59 smithi193 ceph-mon[36878]: rocksdb: Options.persist_stats_to_disk: 0 2024-03-24T00:05:59.720 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:05:59 smithi193 ceph-mon[36878]: rocksdb: Options.write_dbid_to_manifest: 0 2024-03-24T00:05:59.720 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:05:59 smithi193 ceph-mon[36878]: rocksdb: Options.log_readahead_size: 0 2024-03-24T00:05:59.720 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:05:59 smithi193 ceph-mon[36878]: rocksdb: Options.file_checksum_gen_factory: Unknown 2024-03-24T00:05:59.720 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:05:59 smithi193 ceph-mon[36878]: rocksdb: Options.best_efforts_recovery: 0 2024-03-24T00:05:59.720 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:05:59 smithi193 ceph-mon[36878]: rocksdb: Options.max_bgerror_resume_count: 2147483647 2024-03-24T00:05:59.720 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:05:59 smithi193 ceph-mon[36878]: rocksdb: Options.bgerror_resume_retry_interval: 1000000 2024-03-24T00:05:59.720 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:05:59 smithi193 ceph-mon[36878]: rocksdb: Options.allow_data_in_errors: 0 2024-03-24T00:05:59.720 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:05:59 smithi193 ceph-mon[36878]: rocksdb: Options.db_host_id: __hostname__ 2024-03-24T00:05:59.720 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:05:59 smithi193 ceph-mon[36878]: rocksdb: Options.enforce_single_del_contracts: true 2024-03-24T00:05:59.720 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:05:59 smithi193 ceph-mon[36878]: rocksdb: Options.max_background_jobs: 2 2024-03-24T00:05:59.720 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:05:59 smithi193 ceph-mon[36878]: rocksdb: Options.max_background_compactions: -1 2024-03-24T00:05:59.721 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:05:59 smithi193 ceph-mon[36878]: rocksdb: Options.max_subcompactions: 1 2024-03-24T00:05:59.721 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:05:59 smithi193 ceph-mon[36878]: rocksdb: Options.avoid_flush_during_shutdown: 0 2024-03-24T00:05:59.721 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:05:59 smithi193 ceph-mon[36878]: rocksdb: Options.writable_file_max_buffer_size: 1048576 2024-03-24T00:05:59.721 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:05:59 smithi193 ceph-mon[36878]: rocksdb: Options.delayed_write_rate : 16777216 2024-03-24T00:05:59.721 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:05:59 smithi193 ceph-mon[36878]: rocksdb: Options.max_total_wal_size: 0 2024-03-24T00:05:59.721 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:05:59 smithi193 ceph-mon[36878]: rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2024-03-24T00:05:59.721 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:05:59 smithi193 ceph-mon[36878]: rocksdb: Options.stats_dump_period_sec: 600 2024-03-24T00:05:59.721 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:05:59 smithi193 ceph-mon[36878]: rocksdb: Options.stats_persist_period_sec: 600 2024-03-24T00:05:59.721 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:05:59 smithi193 ceph-mon[36878]: rocksdb: Options.stats_history_buffer_size: 1048576 2024-03-24T00:05:59.721 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:05:59 smithi193 ceph-mon[36878]: rocksdb: Options.max_open_files: -1 2024-03-24T00:05:59.721 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:05:59 smithi193 ceph-mon[36878]: rocksdb: Options.bytes_per_sync: 0 2024-03-24T00:05:59.721 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:05:59 smithi193 ceph-mon[36878]: rocksdb: Options.wal_bytes_per_sync: 0 2024-03-24T00:05:59.721 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:05:59 smithi193 ceph-mon[36878]: rocksdb: Options.strict_bytes_per_sync: 0 2024-03-24T00:05:59.721 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:05:59 smithi193 ceph-mon[36878]: rocksdb: Options.compaction_readahead_size: 0 2024-03-24T00:05:59.722 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:05:59 smithi193 ceph-mon[36878]: rocksdb: Options.max_background_flushes: -1 2024-03-24T00:05:59.722 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:05:59 smithi193 ceph-mon[36878]: rocksdb: Compression algorithms supported: 2024-03-24T00:05:59.722 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:05:59 smithi193 ceph-mon[36878]: rocksdb: kZSTDNotFinalCompression supported: 0 2024-03-24T00:05:59.722 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:05:59 smithi193 ceph-mon[36878]: rocksdb: kZSTD supported: 0 2024-03-24T00:05:59.722 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:05:59 smithi193 ceph-mon[36878]: rocksdb: kXpressCompression supported: 0 2024-03-24T00:05:59.722 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:05:59 smithi193 ceph-mon[36878]: rocksdb: kLZ4HCCompression supported: 1 2024-03-24T00:05:59.722 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:05:59 smithi193 ceph-mon[36878]: rocksdb: kZlibCompression supported: 1 2024-03-24T00:05:59.722 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:05:59 smithi193 ceph-mon[36878]: rocksdb: kSnappyCompression supported: 1 2024-03-24T00:05:59.722 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:05:59 smithi193 ceph-mon[36878]: rocksdb: kLZ4Compression supported: 1 2024-03-24T00:05:59.722 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:05:59 smithi193 ceph-mon[36878]: rocksdb: kBZip2Compression supported: 0 2024-03-24T00:05:59.722 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:05:59 smithi193 ceph-mon[36878]: rocksdb: Fast CRC32 supported: Supported on x86 2024-03-24T00:05:59.722 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:05:59 smithi193 ceph-mon[36878]: rocksdb: DMutex implementation: pthread_mutex_t 2024-03-24T00:05:59.722 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:05:59 smithi193 ceph-mon[36878]: rocksdb: [db/version_set.cc:5527] Recovering from manifest file: /var/lib/ceph/mon/ceph-c/store.db/MANIFEST-000005 2024-03-24T00:05:59.723 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:05:59 smithi193 ceph-mon[36878]: rocksdb: [db/column_family.cc:630] --------------- Options for column family [default]: 2024-03-24T00:05:59.723 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:05:59 smithi193 ceph-mon[36878]: rocksdb: Options.comparator: leveldb.BytewiseComparator 2024-03-24T00:05:59.723 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:05:59 smithi193 ceph-mon[36878]: rocksdb: Options.merge_operator: 2024-03-24T00:05:59.723 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:05:59 smithi193 ceph-mon[36878]: rocksdb: Options.compaction_filter: None 2024-03-24T00:05:59.723 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:05:59 smithi193 ceph-mon[36878]: rocksdb: Options.compaction_filter_factory: None 2024-03-24T00:05:59.723 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:05:59 smithi193 ceph-mon[36878]: rocksdb: Options.sst_partitioner_factory: None 2024-03-24T00:05:59.723 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:05:59 smithi193 ceph-mon[36878]: rocksdb: Options.memtable_factory: SkipListFactory 2024-03-24T00:05:59.723 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:05:59 smithi193 ceph-mon[36878]: rocksdb: Options.table_factory: BlockBasedTable 2024-03-24T00:05:59.723 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:05:59 smithi193 ceph-mon[36878]: rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x5607aeb19a20) 2024-03-24T00:05:59.723 INFO:journalctl@ceph.mon.c.smithi193.stdout: cache_index_and_filter_blocks: 1 2024-03-24T00:05:59.723 INFO:journalctl@ceph.mon.c.smithi193.stdout: cache_index_and_filter_blocks_with_high_priority: 0 2024-03-24T00:05:59.723 INFO:journalctl@ceph.mon.c.smithi193.stdout: pin_l0_filter_and_index_blocks_in_cache: 0 2024-03-24T00:05:59.724 INFO:journalctl@ceph.mon.c.smithi193.stdout: pin_top_level_index_and_filter: 1 2024-03-24T00:05:59.724 INFO:journalctl@ceph.mon.c.smithi193.stdout: index_type: 0 2024-03-24T00:05:59.724 INFO:journalctl@ceph.mon.c.smithi193.stdout: data_block_index_type: 0 2024-03-24T00:05:59.724 INFO:journalctl@ceph.mon.c.smithi193.stdout: index_shortening: 1 2024-03-24T00:05:59.724 INFO:journalctl@ceph.mon.c.smithi193.stdout: data_block_hash_table_util_ratio: 0.750000 2024-03-24T00:05:59.724 INFO:journalctl@ceph.mon.c.smithi193.stdout: checksum: 4 2024-03-24T00:05:59.724 INFO:journalctl@ceph.mon.c.smithi193.stdout: no_block_cache: 0 2024-03-24T00:05:59.724 INFO:journalctl@ceph.mon.c.smithi193.stdout: block_cache: 0x5607aeb1f090 2024-03-24T00:05:59.724 INFO:journalctl@ceph.mon.c.smithi193.stdout: block_cache_name: BinnedLRUCache 2024-03-24T00:05:59.724 INFO:journalctl@ceph.mon.c.smithi193.stdout: block_cache_options: 2024-03-24T00:05:59.724 INFO:journalctl@ceph.mon.c.smithi193.stdout: capacity : 536870912 2024-03-24T00:05:59.724 INFO:journalctl@ceph.mon.c.smithi193.stdout: num_shard_bits : 4 2024-03-24T00:05:59.725 INFO:journalctl@ceph.mon.c.smithi193.stdout: strict_capacity_limit : 0 2024-03-24T00:05:59.725 INFO:journalctl@ceph.mon.c.smithi193.stdout: high_pri_pool_ratio: 0.000 2024-03-24T00:05:59.725 INFO:journalctl@ceph.mon.c.smithi193.stdout: block_cache_compressed: (nil) 2024-03-24T00:05:59.725 INFO:journalctl@ceph.mon.c.smithi193.stdout: persistent_cache: (nil) 2024-03-24T00:05:59.725 INFO:journalctl@ceph.mon.c.smithi193.stdout: block_size: 4096 2024-03-24T00:05:59.725 INFO:journalctl@ceph.mon.c.smithi193.stdout: block_size_deviation: 10 2024-03-24T00:05:59.725 INFO:journalctl@ceph.mon.c.smithi193.stdout: block_restart_interval: 16 2024-03-24T00:05:59.725 INFO:journalctl@ceph.mon.c.smithi193.stdout: index_block_restart_interval: 1 2024-03-24T00:05:59.725 INFO:journalctl@ceph.mon.c.smithi193.stdout: metadata_block_size: 4096 2024-03-24T00:05:59.725 INFO:journalctl@ceph.mon.c.smithi193.stdout: partition_filters: 0 2024-03-24T00:05:59.725 INFO:journalctl@ceph.mon.c.smithi193.stdout: use_delta_encoding: 1 2024-03-24T00:05:59.725 INFO:journalctl@ceph.mon.c.smithi193.stdout: filter_policy: bloomfilter 2024-03-24T00:05:59.725 INFO:journalctl@ceph.mon.c.smithi193.stdout: whole_key_filtering: 1 2024-03-24T00:05:59.725 INFO:journalctl@ceph.mon.c.smithi193.stdout: verify_compression: 0 2024-03-24T00:05:59.726 INFO:journalctl@ceph.mon.c.smithi193.stdout: read_amp_bytes_per_bit: 0 2024-03-24T00:05:59.726 INFO:journalctl@ceph.mon.c.smithi193.stdout: format_version: 5 2024-03-24T00:05:59.726 INFO:journalctl@ceph.mon.c.smithi193.stdout: enable_index_compression: 1 2024-03-24T00:05:59.726 INFO:journalctl@ceph.mon.c.smithi193.stdout: block_align: 0 2024-03-24T00:05:59.726 INFO:journalctl@ceph.mon.c.smithi193.stdout: max_auto_readahead_size: 262144 2024-03-24T00:05:59.726 INFO:journalctl@ceph.mon.c.smithi193.stdout: prepopulate_block_cache: 0 2024-03-24T00:05:59.726 INFO:journalctl@ceph.mon.c.smithi193.stdout: initial_auto_readahead_size: 8192 2024-03-24T00:05:59.726 INFO:journalctl@ceph.mon.c.smithi193.stdout: num_file_reads_for_auto_readahead: 2 2024-03-24T00:05:59.727 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:05:59 smithi193 ceph-mon[36878]: rocksdb: Options.write_buffer_size: 33554432 2024-03-24T00:05:59.727 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:05:59 smithi193 ceph-mon[36878]: rocksdb: Options.max_write_buffer_number: 2 2024-03-24T00:05:59.727 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:05:59 smithi193 ceph-mon[36878]: rocksdb: Options.compression: NoCompression 2024-03-24T00:05:59.727 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:05:59 smithi193 ceph-mon[36878]: rocksdb: Options.bottommost_compression: Disabled 2024-03-24T00:05:59.727 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:05:59 smithi193 ceph-mon[36878]: rocksdb: Options.prefix_extractor: nullptr 2024-03-24T00:05:59.727 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:05:59 smithi193 ceph-mon[36878]: rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2024-03-24T00:05:59.727 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:05:59 smithi193 ceph-mon[36878]: rocksdb: Options.num_levels: 7 2024-03-24T00:05:59.727 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:05:59 smithi193 ceph-mon[36878]: rocksdb: Options.min_write_buffer_number_to_merge: 1 2024-03-24T00:05:59.727 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:05:59 smithi193 ceph-mon[36878]: rocksdb: Options.max_write_buffer_number_to_maintain: 0 2024-03-24T00:05:59.727 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:05:59 smithi193 ceph-mon[36878]: rocksdb: Options.max_write_buffer_size_to_maintain: 0 2024-03-24T00:05:59.727 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:05:59 smithi193 ceph-mon[36878]: rocksdb: Options.bottommost_compression_opts.window_bits: -14 2024-03-24T00:05:59.727 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:05:59 smithi193 ceph-mon[36878]: rocksdb: Options.bottommost_compression_opts.level: 32767 2024-03-24T00:05:59.727 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:05:59 smithi193 ceph-mon[36878]: rocksdb: Options.bottommost_compression_opts.strategy: 0 2024-03-24T00:05:59.727 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:05:59 smithi193 ceph-mon[36878]: rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2024-03-24T00:05:59.727 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:05:59 smithi193 ceph-mon[36878]: rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2024-03-24T00:05:59.728 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:05:59 smithi193 ceph-mon[36878]: rocksdb: Options.bottommost_compression_opts.parallel_threads: 1 2024-03-24T00:05:59.728 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:05:59 smithi193 ceph-mon[36878]: rocksdb: Options.bottommost_compression_opts.enabled: false 2024-03-24T00:05:59.728 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:05:59 smithi193 ceph-mon[36878]: rocksdb: Options.bottommost_compression_opts.max_dict_buffer_bytes: 0 2024-03-24T00:05:59.728 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:05:59 smithi193 ceph-mon[36878]: rocksdb: Options.bottommost_compression_opts.use_zstd_dict_trainer: true 2024-03-24T00:05:59.728 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:05:59 smithi193 ceph-mon[36878]: rocksdb: Options.compression_opts.window_bits: -14 2024-03-24T00:05:59.728 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:05:59 smithi193 ceph-mon[36878]: rocksdb: Options.compression_opts.level: 32767 2024-03-24T00:05:59.728 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:05:59 smithi193 ceph-mon[36878]: rocksdb: Options.compression_opts.strategy: 0 2024-03-24T00:05:59.728 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:05:59 smithi193 ceph-mon[36878]: rocksdb: Options.compression_opts.max_dict_bytes: 0 2024-03-24T00:05:59.728 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:05:59 smithi193 ceph-mon[36878]: rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2024-03-24T00:05:59.728 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:05:59 smithi193 ceph-mon[36878]: rocksdb: Options.compression_opts.use_zstd_dict_trainer: true 2024-03-24T00:05:59.729 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:05:59 smithi193 ceph-mon[36878]: rocksdb: Options.compression_opts.parallel_threads: 1 2024-03-24T00:05:59.729 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:05:59 smithi193 ceph-mon[36878]: rocksdb: Options.compression_opts.enabled: false 2024-03-24T00:05:59.729 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:05:59 smithi193 ceph-mon[36878]: rocksdb: Options.compression_opts.max_dict_buffer_bytes: 0 2024-03-24T00:05:59.729 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:05:59 smithi193 ceph-mon[36878]: rocksdb: Options.level0_file_num_compaction_trigger: 4 2024-03-24T00:05:59.729 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:05:59 smithi193 ceph-mon[36878]: rocksdb: Options.level0_slowdown_writes_trigger: 20 2024-03-24T00:05:59.729 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:05:59 smithi193 ceph-mon[36878]: rocksdb: Options.level0_stop_writes_trigger: 36 2024-03-24T00:05:59.729 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:05:59 smithi193 ceph-mon[36878]: rocksdb: Options.target_file_size_base: 67108864 2024-03-24T00:05:59.729 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:05:59 smithi193 ceph-mon[36878]: rocksdb: Options.target_file_size_multiplier: 1 2024-03-24T00:05:59.729 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:05:59 smithi193 ceph-mon[36878]: rocksdb: Options.max_bytes_for_level_base: 268435456 2024-03-24T00:05:59.730 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:05:59 smithi193 ceph-mon[36878]: rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2024-03-24T00:05:59.730 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:05:59 smithi193 ceph-mon[36878]: rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2024-03-24T00:05:59.730 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:05:59 smithi193 ceph-mon[36878]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2024-03-24T00:05:59.730 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:05:59 smithi193 ceph-mon[36878]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2024-03-24T00:05:59.730 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:05:59 smithi193 ceph-mon[36878]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2024-03-24T00:05:59.730 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:05:59 smithi193 ceph-mon[36878]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2024-03-24T00:05:59.730 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:05:59 smithi193 ceph-mon[36878]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2024-03-24T00:05:59.730 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:05:59 smithi193 ceph-mon[36878]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2024-03-24T00:05:59.730 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:05:59 smithi193 ceph-mon[36878]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2024-03-24T00:05:59.730 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:05:59 smithi193 ceph-mon[36878]: rocksdb: Options.max_sequential_skip_in_iterations: 8 2024-03-24T00:05:59.730 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:05:59 smithi193 ceph-mon[36878]: rocksdb: Options.max_compaction_bytes: 1677721600 2024-03-24T00:05:59.730 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:05:59 smithi193 ceph-mon[36878]: rocksdb: Options.ignore_max_compaction_bytes_for_input: true 2024-03-24T00:05:59.730 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:05:59 smithi193 ceph-mon[36878]: rocksdb: Options.arena_block_size: 1048576 2024-03-24T00:05:59.731 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:05:59 smithi193 ceph-mon[36878]: rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2024-03-24T00:05:59.731 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:05:59 smithi193 ceph-mon[36878]: rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2024-03-24T00:05:59.731 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:05:59 smithi193 ceph-mon[36878]: rocksdb: Options.disable_auto_compactions: 0 2024-03-24T00:05:59.731 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:05:59 smithi193 ceph-mon[36878]: rocksdb: Options.compaction_style: kCompactionStyleLevel 2024-03-24T00:05:59.731 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:05:59 smithi193 ceph-mon[36878]: rocksdb: Options.compaction_pri: kMinOverlappingRatio 2024-03-24T00:05:59.731 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:05:59 smithi193 ceph-mon[36878]: rocksdb: Options.compaction_options_universal.size_ratio: 1 2024-03-24T00:05:59.731 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:05:59 smithi193 ceph-mon[36878]: rocksdb: Options.compaction_options_universal.min_merge_width: 2 2024-03-24T00:05:59.731 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:05:59 smithi193 ceph-mon[36878]: rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2024-03-24T00:05:59.731 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:05:59 smithi193 ceph-mon[36878]: rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2024-03-24T00:05:59.731 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:05:59 smithi193 ceph-mon[36878]: rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2024-03-24T00:05:59.731 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:05:59 smithi193 ceph-mon[36878]: rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2024-03-24T00:05:59.731 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:05:59 smithi193 ceph-mon[36878]: rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2024-03-24T00:05:59.732 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:05:59 smithi193 ceph-mon[36878]: rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2024-03-24T00:05:59.732 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:05:59 smithi193 ceph-mon[36878]: rocksdb: Options.table_properties_collectors: 2024-03-24T00:05:59.732 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:05:59 smithi193 ceph-mon[36878]: rocksdb: Options.inplace_update_support: 0 2024-03-24T00:05:59.732 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:05:59 smithi193 ceph-mon[36878]: rocksdb: Options.inplace_update_num_locks: 10000 2024-03-24T00:05:59.732 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:05:59 smithi193 ceph-mon[36878]: rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2024-03-24T00:05:59.732 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:05:59 smithi193 ceph-mon[36878]: rocksdb: Options.memtable_whole_key_filtering: 0 2024-03-24T00:05:59.732 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:05:59 smithi193 ceph-mon[36878]: rocksdb: Options.memtable_huge_page_size: 0 2024-03-24T00:05:59.733 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:05:59 smithi193 ceph-mon[36878]: rocksdb: Options.bloom_locality: 0 2024-03-24T00:05:59.733 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:05:59 smithi193 ceph-mon[36878]: rocksdb: Options.max_successive_merges: 0 2024-03-24T00:05:59.733 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:05:59 smithi193 ceph-mon[36878]: rocksdb: Options.optimize_filters_for_hits: 0 2024-03-24T00:05:59.733 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:05:59 smithi193 ceph-mon[36878]: rocksdb: Options.paranoid_file_checks: 0 2024-03-24T00:05:59.733 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:05:59 smithi193 ceph-mon[36878]: rocksdb: Options.force_consistency_checks: 1 2024-03-24T00:05:59.733 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:05:59 smithi193 ceph-mon[36878]: rocksdb: Options.report_bg_io_stats: 0 2024-03-24T00:05:59.733 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:05:59 smithi193 ceph-mon[36878]: rocksdb: Options.ttl: 2592000 2024-03-24T00:05:59.733 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:05:59 smithi193 ceph-mon[36878]: rocksdb: Options.periodic_compaction_seconds: 0 2024-03-24T00:05:59.733 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:05:59 smithi193 ceph-mon[36878]: rocksdb: Options.preclude_last_level_data_seconds: 0 2024-03-24T00:05:59.733 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:05:59 smithi193 ceph-mon[36878]: rocksdb: Options.preserve_internal_time_seconds: 0 2024-03-24T00:05:59.733 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:05:59 smithi193 ceph-mon[36878]: rocksdb: Options.enable_blob_files: false 2024-03-24T00:05:59.733 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:05:59 smithi193 ceph-mon[36878]: rocksdb: Options.min_blob_size: 0 2024-03-24T00:05:59.733 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:05:59 smithi193 ceph-mon[36878]: rocksdb: Options.blob_file_size: 268435456 2024-03-24T00:05:59.733 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:05:59 smithi193 ceph-mon[36878]: rocksdb: Options.blob_compression_type: NoCompression 2024-03-24T00:05:59.734 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:05:59 smithi193 ceph-mon[36878]: rocksdb: Options.enable_blob_garbage_collection: false 2024-03-24T00:05:59.734 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:05:59 smithi193 ceph-mon[36878]: rocksdb: Options.blob_garbage_collection_age_cutoff: 0.250000 2024-03-24T00:05:59.734 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:05:59 smithi193 ceph-mon[36878]: rocksdb: Options.blob_garbage_collection_force_threshold: 1.000000 2024-03-24T00:05:59.734 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:05:59 smithi193 ceph-mon[36878]: rocksdb: Options.blob_compaction_readahead_size: 0 2024-03-24T00:05:59.734 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:05:59 smithi193 ceph-mon[36878]: rocksdb: Options.blob_file_starting_level: 0 2024-03-24T00:05:59.734 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:05:59 smithi193 ceph-mon[36878]: rocksdb: Options.experimental_mempurge_threshold: 0.000000 2024-03-24T00:05:59.734 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:05:59 smithi193 ceph-mon[36878]: rocksdb: [db/version_set.cc:5566] Recovered from manifest file:/var/lib/ceph/mon/ceph-c/store.db/MANIFEST-000005 succeeded,manifest_file_number is 5, next_file_number is 7, last_sequence is 0, log_number is 0,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 0 2024-03-24T00:05:59.734 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:05:59 smithi193 ceph-mon[36878]: rocksdb: [db/version_set.cc:5581] Column family [default] (ID 0), log number is 0 2024-03-24T00:05:59.734 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:05:59 smithi193 ceph-mon[36878]: rocksdb: [db/db_impl/db_impl_open.cc:539] DB ID: e161913c-fd5f-48ff-ac7d-017b9829f17a 2024-03-24T00:05:59.734 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:05:59 smithi193 ceph-mon[36878]: rocksdb: EVENT_LOG_v1 {"time_micros": 1711238759412584, "job": 1, "event": "recovery_started", "wal_files": [4]} 2024-03-24T00:05:59.734 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:05:59 smithi193 ceph-mon[36878]: rocksdb: [db/db_impl/db_impl_open.cc:1043] Recovering log #4 mode 2 2024-03-24T00:05:59.734 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:05:59 smithi193 ceph-mon[36878]: rocksdb: EVENT_LOG_v1 {"time_micros": 1711238759412996, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 8, "file_size": 1622, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 1, "largest_seqno": 5, "table_properties": {"data_size": 523, "index_size": 31, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 1, "index_value_is_delta_encoded": 1, "filter_size": 69, "raw_key_size": 115, "raw_average_key_size": 23, "raw_value_size": 401, "raw_average_value_size": 80, "num_data_blocks": 1, "num_entries": 5, "num_filter_entries": 5, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "bloomfilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; max_dict_buffer_bytes=0; use_zstd_dict_trainer=1; ", "creation_time": 1711238759, "oldest_key_time": 0, "file_creation_time": 0, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "e161913c-fd5f-48ff-ac7d-017b9829f17a", "db_session_id": "LP9C8V7XRW2SBXA7PZDM", "orig_file_number": 8, "seqno_to_time_mapping": "N/A"}} 2024-03-24T00:05:59.734 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:05:59 smithi193 ceph-mon[36878]: rocksdb: EVENT_LOG_v1 {"time_micros": 1711238759413084, "job": 1, "event": "recovery_finished"} 2024-03-24T00:05:59.735 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:05:59 smithi193 ceph-mon[36878]: rocksdb: [db/version_set.cc:5047] Creating manifest 10 2024-03-24T00:05:59.735 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:05:59 smithi193 ceph-mon[36878]: rocksdb: [file/delete_scheduler.cc:74] Deleted file /var/lib/ceph/mon/ceph-c/store.db/000004.log immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024-03-24T00:05:59.735 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:05:59 smithi193 ceph-mon[36878]: rocksdb: [db/db_impl/db_impl_open.cc:1987] SstFileManager instance 0x5607aec12000 2024-03-24T00:05:59.735 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:05:59 smithi193 ceph-mon[36878]: rocksdb: DB pointer 0x5607aebfc000 2024-03-24T00:05:59.735 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:05:59 smithi193 ceph-mon[36878]: rocksdb: [db/db_impl/db_impl.cc:1109] ------- DUMPING STATS ------- 2024-03-24T00:05:59.735 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:05:59 smithi193 ceph-mon[36878]: rocksdb: [db/db_impl/db_impl.cc:1111] 2024-03-24T00:05:59.735 INFO:journalctl@ceph.mon.c.smithi193.stdout: ** DB Stats ** 2024-03-24T00:05:59.735 INFO:journalctl@ceph.mon.c.smithi193.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-03-24T00:05:59.735 INFO:journalctl@ceph.mon.c.smithi193.stdout: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2024-03-24T00:05:59.735 INFO:journalctl@ceph.mon.c.smithi193.stdout: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-03-24T00:05:59.735 INFO:journalctl@ceph.mon.c.smithi193.stdout: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2024-03-24T00:05:59.735 INFO:journalctl@ceph.mon.c.smithi193.stdout: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2024-03-24T00:05:59.735 INFO:journalctl@ceph.mon.c.smithi193.stdout: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-03-24T00:05:59.736 INFO:journalctl@ceph.mon.c.smithi193.stdout: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2024-03-24T00:05:59.736 INFO:journalctl@ceph.mon.c.smithi193.stdout: 2024-03-24T00:05:59.736 INFO:journalctl@ceph.mon.c.smithi193.stdout: ** Compaction Stats [default] ** 2024-03-24T00:05:59.736 INFO:journalctl@ceph.mon.c.smithi193.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-03-24T00:05:59.736 INFO:journalctl@ceph.mon.c.smithi193.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ 2024-03-24T00:05:59.736 INFO:journalctl@ceph.mon.c.smithi193.stdout: L0 1/0 1.58 KB 0.2 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 3.9 0.00 0.00 1 0.000 0 0 0.0 0.0 2024-03-24T00:05:59.736 INFO:journalctl@ceph.mon.c.smithi193.stdout: Sum 1/0 1.58 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 3.9 0.00 0.00 1 0.000 0 0 0.0 0.0 2024-03-24T00:05:59.736 INFO:journalctl@ceph.mon.c.smithi193.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 3.9 0.00 0.00 1 0.000 0 0 0.0 0.0 2024-03-24T00:05:59.736 INFO:journalctl@ceph.mon.c.smithi193.stdout: 2024-03-24T00:05:59.736 INFO:journalctl@ceph.mon.c.smithi193.stdout: ** Compaction Stats [default] ** 2024-03-24T00:05:59.736 INFO:journalctl@ceph.mon.c.smithi193.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-03-24T00:05:59.736 INFO:journalctl@ceph.mon.c.smithi193.stdout: --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-03-24T00:05:59.736 INFO:journalctl@ceph.mon.c.smithi193.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 3.9 0.00 0.00 1 0.000 0 0 0.0 0.0 2024-03-24T00:05:59.736 INFO:journalctl@ceph.mon.c.smithi193.stdout: 2024-03-24T00:05:59.737 INFO:journalctl@ceph.mon.c.smithi193.stdout: Blob file count: 0, total size: 0.0 GB, garbage size: 0.0 GB, space amp: 0.0 2024-03-24T00:05:59.737 INFO:journalctl@ceph.mon.c.smithi193.stdout: 2024-03-24T00:05:59.737 INFO:journalctl@ceph.mon.c.smithi193.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-03-24T00:05:59.737 INFO:journalctl@ceph.mon.c.smithi193.stdout: Flush(GB): cumulative 0.000, interval 0.000 2024-03-24T00:05:59.737 INFO:journalctl@ceph.mon.c.smithi193.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2024-03-24T00:05:59.737 INFO:journalctl@ceph.mon.c.smithi193.stdout: AddFile(Total Files): cumulative 0, interval 0 2024-03-24T00:05:59.737 INFO:journalctl@ceph.mon.c.smithi193.stdout: AddFile(L0 Files): cumulative 0, interval 0 2024-03-24T00:05:59.737 INFO:journalctl@ceph.mon.c.smithi193.stdout: AddFile(Keys): cumulative 0, interval 0 2024-03-24T00:05:59.737 INFO:journalctl@ceph.mon.c.smithi193.stdout: Cumulative compaction: 0.00 GB write, 0.65 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-03-24T00:05:59.737 INFO:journalctl@ceph.mon.c.smithi193.stdout: Interval compaction: 0.00 GB write, 0.65 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-03-24T00:05:59.737 INFO:journalctl@ceph.mon.c.smithi193.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-03-24T00:05:59.737 INFO:journalctl@ceph.mon.c.smithi193.stdout: Block cache BinnedLRUCache@0x5607aeb1f090#2 capacity: 512.00 MB usage: 0.22 KB table_size: 0 occupancy: 18446744073709551615 collections: 1 last_copies: 0 last_secs: 1e-05 secs_since: 0 2024-03-24T00:05:59.737 INFO:journalctl@ceph.mon.c.smithi193.stdout: Block cache entry stats(count,size,portion): FilterBlock(1,0.11 KB,2.08616e-05%) IndexBlock(1,0.11 KB,2.08616e-05%) Misc(1,0.00 KB,0%) 2024-03-24T00:05:59.738 INFO:journalctl@ceph.mon.c.smithi193.stdout: 2024-03-24T00:05:59.738 INFO:journalctl@ceph.mon.c.smithi193.stdout: ** File Read Latency Histogram By Level [default] ** 2024-03-24T00:05:59.738 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:05:59 smithi193 ceph-mon[36878]: mon.c does not exist in monmap, will attempt to join an existing cluster 2024-03-24T00:05:59.738 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:05:59 smithi193 ceph-mon[36878]: using public_addr v2:172.21.15.193:0/0 -> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] 2024-03-24T00:05:59.738 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:05:59 smithi193 ceph-mon[36878]: starting mon.c rank -1 at public addrs [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] at bind addrs [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] mon_data /var/lib/ceph/mon/ceph-c fsid d79a5420-e971-11ee-95cd-87774f69a715 2024-03-24T00:05:59.738 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:05:59 smithi193 ceph-mon[36878]: mon.c@-1(???) e0 preinit fsid d79a5420-e971-11ee-95cd-87774f69a715 2024-03-24T00:05:59.738 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:05:59 smithi193 bash[36858]: 0284545a88c562a09f5395625948e2d9c1fc6f6487ee318a6e852bc19f0211ae 2024-03-24T00:05:59.738 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:05:59 smithi193 ceph-mon[36878]: mon.c@-1(synchronizing).mds e1 new map 2024-03-24T00:05:59.738 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:05:59 smithi193 ceph-mon[36878]: mon.c@-1(synchronizing).mds e1 print_map 2024-03-24T00:05:59.738 INFO:journalctl@ceph.mon.c.smithi193.stdout: e1 2024-03-24T00:05:59.738 INFO:journalctl@ceph.mon.c.smithi193.stdout: enable_multiple, ever_enabled_multiple: 1,1 2024-03-24T00:05:59.738 INFO:journalctl@ceph.mon.c.smithi193.stdout: default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2,11=minor log segments} 2024-03-24T00:05:59.739 INFO:journalctl@ceph.mon.c.smithi193.stdout: legacy client fscid: -1 2024-03-24T00:05:59.739 INFO:journalctl@ceph.mon.c.smithi193.stdout: 2024-03-24T00:05:59.739 INFO:journalctl@ceph.mon.c.smithi193.stdout: No filesystems configured 2024-03-24T00:05:59.739 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:05:59 smithi193 ceph-mon[36878]: mon.c@-1(synchronizing).osd e0 _set_cache_ratios kv ratio 0.25 inc ratio 0.375 full ratio 0.375 2024-03-24T00:05:59.739 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:05:59 smithi193 ceph-mon[36878]: mon.c@-1(synchronizing).osd e0 register_cache_with_pcm pcm target: 2147483648 pcm max: 1020054732 pcm min: 134217728 inc_osd_cache size: 1 2024-03-24T00:05:59.739 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:05:59 smithi193 ceph-mon[36878]: mon.c@-1(synchronizing).osd e1 e1: 0 total, 0 up, 0 in 2024-03-24T00:05:59.740 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:05:59 smithi193 ceph-mon[36878]: mon.c@-1(synchronizing).osd e2 e2: 0 total, 0 up, 0 in 2024-03-24T00:05:59.740 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:05:59 smithi193 ceph-mon[36878]: mon.c@-1(synchronizing).osd e3 e3: 0 total, 0 up, 0 in 2024-03-24T00:05:59.740 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:05:59 smithi193 ceph-mon[36878]: mon.c@-1(synchronizing).osd e4 e4: 0 total, 0 up, 0 in 2024-03-24T00:05:59.740 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:05:59 smithi193 ceph-mon[36878]: mon.c@-1(synchronizing).osd e4 crush map has features 3314932999778484224, adjusting msgr requires 2024-03-24T00:05:59.740 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:05:59 smithi193 ceph-mon[36878]: mon.c@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2024-03-24T00:05:59.740 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:05:59 smithi193 ceph-mon[36878]: mon.c@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2024-03-24T00:05:59.740 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:05:59 smithi193 ceph-mon[36878]: mon.c@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2024-03-24T00:05:59.740 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:05:59 smithi193 ceph-mon[36878]: pgmap v6: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T00:05:59.741 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:05:59 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T00:05:59.741 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:05:59 smithi193 ceph-mon[36878]: Added host smithi189 2024-03-24T00:05:59.741 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:05:59 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T00:05:59.741 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:05:59 smithi193 ceph-mon[36878]: pgmap v7: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T00:05:59.741 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:05:59 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T00:05:59.741 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:05:59 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T00:05:59.741 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:05:59 smithi193 ceph-mon[36878]: pgmap v8: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T00:05:59.742 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:05:59 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T00:05:59.742 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:05:59 smithi193 ceph-mon[36878]: from='client.14180 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-24T00:05:59.742 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:05:59 smithi193 ceph-mon[36878]: pgmap v9: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T00:05:59.742 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:05:59 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T00:05:59.742 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:05:59 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T00:05:59.742 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:05:59 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T00:05:59.742 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:05:59 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T00:05:59.742 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:05:59 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd/host:smithi189", "name": "osd_memory_target"}]: dispatch 2024-03-24T00:05:59.743 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:05:59 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T00:05:59.743 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:05:59 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T00:05:59.743 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:05:59 smithi193 ceph-mon[36878]: Updating smithi189:/etc/ceph/ceph.conf 2024-03-24T00:05:59.743 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:05:59 smithi193 ceph-mon[36878]: Updating smithi189:/var/lib/ceph/d79a5420-e971-11ee-95cd-87774f69a715/config/ceph.conf 2024-03-24T00:05:59.743 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:05:59 smithi193 ceph-mon[36878]: pgmap v10: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T00:05:59.743 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:05:59 smithi193 ceph-mon[36878]: Updating smithi189:/etc/ceph/ceph.client.admin.keyring 2024-03-24T00:05:59.744 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:05:59 smithi193 ceph-mon[36878]: Updating smithi189:/var/lib/ceph/d79a5420-e971-11ee-95cd-87774f69a715/config/ceph.client.admin.keyring 2024-03-24T00:05:59.744 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:05:59 smithi193 ceph-mon[36878]: from='client.14182 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi193", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T00:05:59.744 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:05:59 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T00:05:59.744 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:05:59 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T00:05:59.744 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:05:59 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T00:05:59.744 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:05:59 smithi193 ceph-mon[36878]: Deploying cephadm binary to smithi193 2024-03-24T00:05:59.745 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:05:59 smithi193 ceph-mon[36878]: pgmap v11: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T00:05:59.745 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:05:59 smithi193 ceph-mon[36878]: pgmap v12: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T00:05:59.745 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:05:59 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T00:05:59.745 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:05:59 smithi193 ceph-mon[36878]: Added host smithi193 2024-03-24T00:05:59.745 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:05:59 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T00:05:59.745 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:05:59 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T00:05:59.746 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:05:59 smithi193 ceph-mon[36878]: pgmap v13: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T00:05:59.746 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:05:59 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T00:05:59.747 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:05:59 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T00:05:59.747 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:05:59 smithi193 ceph-mon[36878]: pgmap v14: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T00:05:59.747 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:05:59 smithi193 ceph-mon[36878]: from='client.14184 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-24T00:05:59.747 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:05:59 smithi193 ceph-mon[36878]: pgmap v15: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T00:05:59.748 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:05:59 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T00:05:59.748 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:05:59 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T00:05:59.748 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:05:59 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T00:05:59.748 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:05:59 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T00:05:59.749 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:05:59 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd/host:smithi193", "name": "osd_memory_target"}]: dispatch 2024-03-24T00:05:59.749 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:05:59 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T00:05:59.750 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:05:59 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T00:05:59.750 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:05:59 smithi193 ceph-mon[36878]: Updating smithi193:/etc/ceph/ceph.conf 2024-03-24T00:05:59.750 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:05:59 smithi193 ceph-mon[36878]: from='client.? 172.21.15.118:0/2863852192' entity='client.admin' cmd=[{"prefix": "osd crush tunables", "profile": "default"}]: dispatch 2024-03-24T00:05:59.751 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:05:59 smithi193 ceph-mon[36878]: Updating smithi193:/var/lib/ceph/d79a5420-e971-11ee-95cd-87774f69a715/config/ceph.conf 2024-03-24T00:05:59.751 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:05:59 smithi193 ceph-mon[36878]: Updating smithi193:/etc/ceph/ceph.client.admin.keyring 2024-03-24T00:05:59.752 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:05:59 smithi193 ceph-mon[36878]: Updating smithi193:/var/lib/ceph/d79a5420-e971-11ee-95cd-87774f69a715/config/ceph.client.admin.keyring 2024-03-24T00:05:59.752 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:05:59 smithi193 ceph-mon[36878]: pgmap v16: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T00:05:59.752 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:05:59 smithi193 ceph-mon[36878]: from='client.? 172.21.15.118:0/2863852192' entity='client.admin' cmd='[{"prefix": "osd crush tunables", "profile": "default"}]': finished 2024-03-24T00:05:59.752 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:05:59 smithi193 ceph-mon[36878]: osdmap e4: 0 total, 0 up, 0 in 2024-03-24T00:05:59.752 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:05:59 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T00:05:59.752 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:05:59 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T00:05:59.752 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:05:59 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T00:05:59.752 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:05:59 smithi193 ceph-mon[36878]: pgmap v18: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T00:05:59.753 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:05:59 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T00:05:59.753 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:05:59 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T00:05:59.753 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:05:59 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T00:05:59.753 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:05:59 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T00:05:59.753 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:05:59 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T00:05:59.753 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:05:59 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-24T00:05:59.753 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:05:59 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T00:05:59.753 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:05:59 smithi193 ceph-mon[36878]: from='client.14188 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "placement": "3;smithi118:172.21.15.118=a;smithi189:172.21.15.189=b;smithi193:172.21.15.193=c", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T00:05:59.753 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:05:59 smithi193 ceph-mon[36878]: Saving service mon spec with placement smithi118:172.21.15.118=a;smithi189:172.21.15.189=b;smithi193:172.21.15.193=c;count:3 2024-03-24T00:05:59.753 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:05:59 smithi193 ceph-mon[36878]: Deploying daemon mon.c on smithi193 2024-03-24T00:05:59.753 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:05:59 smithi193 ceph-mon[36878]: pgmap v19: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T00:05:59.753 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:05:59 smithi193 ceph-mon[36878]: pgmap v20: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T00:05:59.754 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:05:59 smithi193 ceph-mon[36878]: from='client.? 172.21.15.193:0/1148064473' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-24T00:05:59.754 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:05:59 smithi193 ceph-mon[36878]: mon.c@-1(synchronizing).paxosservice(auth 1..3) refresh upgraded, format 0 -> 3 2024-03-24T00:05:59.754 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:05:59 smithi193 systemd[1]: Started Ceph mon.c for d79a5420-e971-11ee-95cd-87774f69a715. 2024-03-24T00:06:00.020 INFO:tasks.cephadm:Waiting for 3 mons in monmap... 2024-03-24T00:06:00.021 DEBUG:teuthology.orchestra.run.smithi193:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d79a5420-e971-11ee-95cd-87774f69a715 -- ceph mon dump -f json 2024-03-24T00:06:00.252 INFO:teuthology.orchestra.run.smithi193.stderr:Inferring config /var/lib/ceph/d79a5420-e971-11ee-95cd-87774f69a715/mon.c/config 2024-03-24T00:06:04.484 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-24T00:06:04.482+0000 7f09f8941700 1 -- 172.21.15.193:0/1834763226 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f09dc0035c0 msgr2=0x7f09dc005a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:06:04.484 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-24T00:06:04.482+0000 7f09f8941700 1 --2- 172.21.15.193:0/1834763226 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f09dc0035c0 0x7f09dc005a50 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f09f40743e0 tx=0x7f09e400feb0 comp rx=0 tx=0).stop 2024-03-24T00:06:04.484 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-24T00:06:04.482+0000 7f09f8941700 1 -- 172.21.15.193:0/1834763226 shutdown_connections 2024-03-24T00:06:04.484 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-24T00:06:04.482+0000 7f09f8941700 1 --2- 172.21.15.193:0/1834763226 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f09dc0035c0 0x7f09dc005a50 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:06:04.485 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-24T00:06:04.482+0000 7f09f8941700 1 -- 172.21.15.193:0/1834763226 >> 172.21.15.193:0/1834763226 conn(0x7f09f40fd730 msgr2=0x7f09f40ffb50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:06:04.485 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-24T00:06:04.482+0000 7f09f8941700 1 -- 172.21.15.193:0/1834763226 shutdown_connections 2024-03-24T00:06:04.485 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-24T00:06:04.482+0000 7f09f8941700 1 -- 172.21.15.193:0/1834763226 wait complete. 2024-03-24T00:06:04.485 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-24T00:06:04.483+0000 7f09f8941700 1 Processor -- start 2024-03-24T00:06:04.485 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-24T00:06:04.483+0000 7f09f8941700 1 -- start start 2024-03-24T00:06:04.485 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-24T00:06:04.483+0000 7f09f8941700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f09dc0035c0 0x7f09f419f6b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:06:04.485 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-24T00:06:04.483+0000 7f09f8941700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f09f419fbf0 0x7f09f4199830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:06:04.485 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-24T00:06:04.483+0000 7f09f8941700 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f09f410bf10 con 0x7f09f419fbf0 2024-03-24T00:06:04.485 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-24T00:06:04.483+0000 7f09f8941700 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f09f410bd90 con 0x7f09dc0035c0 2024-03-24T00:06:04.485 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-24T00:06:04.483+0000 7f09f259c700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f09dc0035c0 0x7f09f419f6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:06:04.485 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-24T00:06:04.483+0000 7f09f259c700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f09dc0035c0 0x7f09f419f6b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.193:56554/0 (socket says 172.21.15.193:56554) 2024-03-24T00:06:04.485 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-24T00:06:04.483+0000 7f09f259c700 1 -- 172.21.15.193:0/2575367044 learned_addr learned my addr 172.21.15.193:0/2575367044 (peer_addr_for_me v2:172.21.15.193:0/0) 2024-03-24T00:06:04.486 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-24T00:06:04.483+0000 7f09f259c700 1 -- 172.21.15.193:0/2575367044 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f09dc0035c0 msgr2=0x7f09f419f6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 12 2024-03-24T00:06:04.486 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-24T00:06:04.483+0000 7f09f259c700 1 -- 172.21.15.193:0/2575367044 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f09dc0035c0 msgr2=0x7f09f419f6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-03-24T00:06:04.486 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-24T00:06:04.483+0000 7f09f259c700 1 --2- 172.21.15.193:0/2575367044 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f09dc0035c0 0x7f09f419f6b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-03-24T00:06:04.486 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-24T00:06:04.483+0000 7f09f259c700 1 --2- 172.21.15.193:0/2575367044 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f09dc0035c0 0x7f09f419f6b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-24T00:06:04.486 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-24T00:06:04.484+0000 7f09f1d9b700 1 --2- 172.21.15.193:0/2575367044 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f09f419fbf0 0x7f09f4199830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:06:04.486 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-24T00:06:04.484+0000 7f09f1d9b700 1 -- 172.21.15.193:0/2575367044 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f09dc0035c0 msgr2=0x7f09f419f6b0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-24T00:06:04.486 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-24T00:06:04.484+0000 7f09f1d9b700 1 --2- 172.21.15.193:0/2575367044 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f09dc0035c0 0x7f09f419f6b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:06:04.486 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-24T00:06:04.484+0000 7f09f1d9b700 1 -- 172.21.15.193:0/2575367044 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f09f41a0230 con 0x7f09f419fbf0 2024-03-24T00:06:04.487 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-24T00:06:04.485+0000 7f09f1d9b700 1 --2- 172.21.15.193:0/2575367044 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f09f419fbf0 0x7f09f4199830 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f09e800b4d0 tx=0x7f09e800b890 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:06:04.487 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-24T00:06:04.487+0000 7f09e37fe700 1 -- 172.21.15.193:0/2575367044 <== mon.0 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 309+0+0 (secure 0 0 0) 0x7f09e8011630 con 0x7f09f419fbf0 2024-03-24T00:06:04.487 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-24T00:06:04.487+0000 7f09e37fe700 1 -- 172.21.15.193:0/2575367044 <== mon.0 v2:172.21.15.118:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f09e8011cb0 con 0x7f09f419fbf0 2024-03-24T00:06:04.487 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-24T00:06:04.487+0000 7f09e37fe700 1 -- 172.21.15.193:0/2575367044 <== mon.0 v2:172.21.15.118:3300/0 3 ==== mon_map magic: 0 v1 ==== 309+0+0 (secure 0 0 0) 0x7f09e8010e60 con 0x7f09f419fbf0 2024-03-24T00:06:04.487 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-24T00:06:04.487+0000 7f09f8941700 1 -- 172.21.15.193:0/2575367044 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f09f4199dd0 con 0x7f09f419fbf0 2024-03-24T00:06:04.487 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-24T00:06:04.487+0000 7f09f8941700 1 -- 172.21.15.193:0/2575367044 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f09f419a330 con 0x7f09f419fbf0 2024-03-24T00:06:04.488 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-24T00:06:04.488+0000 7f09f8941700 1 -- 172.21.15.193:0/2575367044 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f09f4075bf0 con 0x7f09f419fbf0 2024-03-24T00:06:04.488 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-24T00:06:04.488+0000 7f09e37fe700 1 -- 172.21.15.193:0/2575367044 <== mon.0 v2:172.21.15.118:3300/0 4 ==== mgrmap(e 13) v1 ==== 46163+0+0 (secure 0 0 0) 0x7f09e8018020 con 0x7f09f419fbf0 2024-03-24T00:06:04.489 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-24T00:06:04.489+0000 7f09e37fe700 1 --2- 172.21.15.193:0/2575367044 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7f09d4038d60 0x7f09d403b220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:06:04.489 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-24T00:06:04.489+0000 7f09e37fe700 1 -- 172.21.15.193:0/2575367044 <== mon.0 v2:172.21.15.118:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1155+0+0 (secure 0 0 0) 0x7f09e804bd70 con 0x7f09f419fbf0 2024-03-24T00:06:04.489 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-24T00:06:04.489+0000 7f09f259c700 1 --2- 172.21.15.193:0/2575367044 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7f09d4038d60 0x7f09d403b220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:06:04.490 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-24T00:06:04.490+0000 7f09f259c700 1 --2- 172.21.15.193:0/2575367044 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7f09d4038d60 0x7f09d403b220 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f09e400fa60 tx=0x7f09e4005960 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:06:04.505 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-24T00:06:04.505+0000 7f09e37fe700 1 -- 172.21.15.193:0/2575367044 <== mon.0 v2:172.21.15.118:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f09e801f020 con 0x7f09f419fbf0 2024-03-24T00:06:04.779 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-24T00:06:04.779+0000 7f09f8941700 1 -- 172.21.15.193:0/2575367044 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f09f419a6a0 con 0x7f09f419fbf0 2024-03-24T00:06:04.780 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-24T00:06:04.780+0000 7f09e37fe700 1 -- 172.21.15.193:0/2575367044 <== mon.0 v2:172.21.15.118:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 2 v2) v1 ==== 95+0+1019 (secure 0 0 0) 0x7f09e801aa40 con 0x7f09f419fbf0 2024-03-24T00:06:04.780 INFO:teuthology.orchestra.run.smithi193.stdout: 2024-03-24T00:06:04.780 INFO:teuthology.orchestra.run.smithi193.stdout:{"epoch":2,"fsid":"d79a5420-e971-11ee-95cd-87774f69a715","modified":"2024-03-24T00:05:59.445188Z","created":"2024-03-24T00:03:54.425344Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:3300","nonce":0},{"type":"v1","addr":"172.21.15.118:6789","nonce":0}]},"addr":"172.21.15.118:6789/0","public_addr":"172.21.15.118:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:3300","nonce":0},{"type":"v1","addr":"172.21.15.193:6789","nonce":0}]},"addr":"172.21.15.193:6789/0","public_addr":"172.21.15.193:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0,1]} 2024-03-24T00:06:04.783 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-24T00:06:04.782+0000 7f09f8941700 1 -- 172.21.15.193:0/2575367044 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7f09d4038d60 msgr2=0x7f09d403b220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:06:04.783 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-24T00:06:04.782+0000 7f09f8941700 1 --2- 172.21.15.193:0/2575367044 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7f09d4038d60 0x7f09d403b220 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f09e400fa60 tx=0x7f09e4005960 comp rx=0 tx=0).stop 2024-03-24T00:06:04.783 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-24T00:06:04.782+0000 7f09f8941700 1 -- 172.21.15.193:0/2575367044 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f09f419fbf0 msgr2=0x7f09f4199830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:06:04.783 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-24T00:06:04.782+0000 7f09f8941700 1 --2- 172.21.15.193:0/2575367044 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f09f419fbf0 0x7f09f4199830 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f09e800b4d0 tx=0x7f09e800b890 comp rx=0 tx=0).stop 2024-03-24T00:06:04.783 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-24T00:06:04.782+0000 7f09f8941700 1 -- 172.21.15.193:0/2575367044 shutdown_connections 2024-03-24T00:06:04.783 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-24T00:06:04.782+0000 7f09f8941700 1 --2- 172.21.15.193:0/2575367044 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7f09d4038d60 0x7f09d403b220 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:06:04.783 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-24T00:06:04.782+0000 7f09f8941700 1 --2- 172.21.15.193:0/2575367044 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f09dc0035c0 0x7f09f419f6b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:06:04.783 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-24T00:06:04.782+0000 7f09f8941700 1 --2- 172.21.15.193:0/2575367044 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f09f419fbf0 0x7f09f4199830 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:06:04.783 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-24T00:06:04.782+0000 7f09f8941700 1 -- 172.21.15.193:0/2575367044 >> 172.21.15.193:0/2575367044 conn(0x7f09f40fd730 msgr2=0x7f09f40ff5d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:06:04.783 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-24T00:06:04.782+0000 7f09f8941700 1 -- 172.21.15.193:0/2575367044 shutdown_connections 2024-03-24T00:06:04.783 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-24T00:06:04.782+0000 7f09f8941700 1 -- 172.21.15.193:0/2575367044 wait complete. 2024-03-24T00:06:04.784 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:06:04 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-24T00:06:04.784 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:06:04 smithi193 ceph-mon[36878]: mon.a calling monitor election 2024-03-24T00:06:04.784 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:06:04 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-24T00:06:04.784 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:06:04 smithi193 ceph-mon[36878]: pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T00:06:04.784 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:06:04 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-24T00:06:04.784 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:06:04 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-24T00:06:04.784 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:06:04 smithi193 ceph-mon[36878]: mon.c calling monitor election 2024-03-24T00:06:04.784 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:06:04 smithi193 ceph-mon[36878]: pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T00:06:04.784 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:06:04 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-24T00:06:04.784 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:06:04 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-24T00:06:04.784 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:06:04 smithi193 ceph-mon[36878]: pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T00:06:04.784 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:06:04 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-24T00:06:04.784 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:06:04 smithi193 ceph-mon[36878]: mon.a is new leader, mons a,c in quorum (ranks 0,1) 2024-03-24T00:06:04.785 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:06:04 smithi193 ceph-mon[36878]: monmap e2: 2 mons at {a=[v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0],c=[v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0]} removed_ranks: {} disallowed_leaders: {} 2024-03-24T00:06:04.785 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:06:04 smithi193 ceph-mon[36878]: fsmap 2024-03-24T00:06:04.785 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:06:04 smithi193 ceph-mon[36878]: osdmap e4: 0 total, 0 up, 0 in 2024-03-24T00:06:04.785 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:06:04 smithi193 ceph-mon[36878]: mgrmap e13: x(active, since 59s) 2024-03-24T00:06:04.785 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:06:04 smithi193 ceph-mon[36878]: overall HEALTH_OK 2024-03-24T00:06:04.785 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:06:04 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T00:06:04.785 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:06:04 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T00:06:04.785 INFO:teuthology.orchestra.run.smithi193.stderr:dumped monmap epoch 2 2024-03-24T00:06:04.947 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:06:04 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-24T00:06:04.947 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:06:04 smithi118 ceph-mon[31535]: mon.a calling monitor election 2024-03-24T00:06:04.947 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:06:04 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-24T00:06:04.947 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:06:04 smithi118 ceph-mon[31535]: pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T00:06:04.947 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:06:04 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-24T00:06:04.947 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:06:04 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-24T00:06:04.947 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:06:04 smithi118 ceph-mon[31535]: mon.c calling monitor election 2024-03-24T00:06:04.947 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:06:04 smithi118 ceph-mon[31535]: pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T00:06:04.948 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:06:04 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-24T00:06:04.948 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:06:04 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-24T00:06:04.948 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:06:04 smithi118 ceph-mon[31535]: pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T00:06:04.948 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:06:04 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-24T00:06:04.948 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:06:04 smithi118 ceph-mon[31535]: mon.a is new leader, mons a,c in quorum (ranks 0,1) 2024-03-24T00:06:04.948 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:06:04 smithi118 ceph-mon[31535]: monmap e2: 2 mons at {a=[v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0],c=[v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0]} removed_ranks: {} disallowed_leaders: {} 2024-03-24T00:06:04.948 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:06:04 smithi118 ceph-mon[31535]: fsmap 2024-03-24T00:06:04.948 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:06:04 smithi118 ceph-mon[31535]: osdmap e4: 0 total, 0 up, 0 in 2024-03-24T00:06:04.948 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:06:04 smithi118 ceph-mon[31535]: mgrmap e13: x(active, since 59s) 2024-03-24T00:06:04.949 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:06:04 smithi118 ceph-mon[31535]: overall HEALTH_OK 2024-03-24T00:06:04.949 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:06:04 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T00:06:04.949 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:06:04 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T00:06:05.710 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:06:05 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T00:06:05.711 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:06:05 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-24T00:06:05.711 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:06:05 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T00:06:05.711 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:06:05 smithi193 ceph-mon[36878]: Deploying daemon mon.b on smithi189 2024-03-24T00:06:05.711 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:06:05 smithi193 ceph-mon[36878]: from='client.? 172.21.15.193:0/2575367044' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-24T00:06:05.711 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:06:05 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-24T00:06:05.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:06:05 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T00:06:05.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:06:05 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-24T00:06:05.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:06:05 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T00:06:05.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:06:05 smithi118 ceph-mon[31535]: Deploying daemon mon.b on smithi189 2024-03-24T00:06:05.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:06:05 smithi118 ceph-mon[31535]: from='client.? 172.21.15.193:0/2575367044' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-24T00:06:05.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:06:05 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-24T00:06:06.409 INFO:tasks.cephadm:Waiting for 3 mons in monmap... 2024-03-24T00:06:06.410 DEBUG:teuthology.orchestra.run.smithi193:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d79a5420-e971-11ee-95cd-87774f69a715 -- ceph mon dump -f json 2024-03-24T00:06:06.652 INFO:teuthology.orchestra.run.smithi193.stderr:Inferring config /var/lib/ceph/d79a5420-e971-11ee-95cd-87774f69a715/mon.c/config 2024-03-24T00:06:06.710 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:06:06 smithi193 ceph-mon[36878]: pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T00:06:06.862 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:06:06 smithi118 ceph-mon[31535]: pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T00:06:08.085 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-24T00:06:08.084+0000 7f1b8f658700 1 -- 172.21.15.193:0/1138905131 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f1b88100100 msgr2=0x7f1b881004e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:06:08.085 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-24T00:06:08.084+0000 7f1b8f658700 1 --2- 172.21.15.193:0/1138905131 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f1b88100100 0x7f1b881004e0 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f1b840074e0 tx=0x7f1b8400af90 comp rx=0 tx=0).stop 2024-03-24T00:06:08.085 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-24T00:06:08.084+0000 7f1b8f658700 1 -- 172.21.15.193:0/1138905131 shutdown_connections 2024-03-24T00:06:08.085 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-24T00:06:08.084+0000 7f1b8f658700 1 --2- 172.21.15.193:0/1138905131 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f1b88100100 0x7f1b881004e0 unknown :-1 s=CLOSED pgs=102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:06:08.085 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-24T00:06:08.084+0000 7f1b8f658700 1 -- 172.21.15.193:0/1138905131 >> 172.21.15.193:0/1138905131 conn(0x7f1b880fb990 msgr2=0x7f1b880fddb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:06:08.086 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-24T00:06:08.084+0000 7f1b8f658700 1 -- 172.21.15.193:0/1138905131 shutdown_connections 2024-03-24T00:06:08.086 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-24T00:06:08.084+0000 7f1b8f658700 1 -- 172.21.15.193:0/1138905131 wait complete. 2024-03-24T00:06:08.086 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-24T00:06:08.084+0000 7f1b8f658700 1 Processor -- start 2024-03-24T00:06:08.086 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-24T00:06:08.085+0000 7f1b8f658700 1 -- start start 2024-03-24T00:06:08.086 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-24T00:06:08.085+0000 7f1b8f658700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f1b88100100 0x7f1b88101ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:06:08.086 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-24T00:06:08.085+0000 7f1b8f658700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f1b881020e0 0x7f1b88102560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:06:08.086 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-24T00:06:08.085+0000 7f1b8f658700 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1b8810a080 con 0x7f1b881020e0 2024-03-24T00:06:08.086 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-24T00:06:08.085+0000 7f1b8f658700 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1b88109f00 con 0x7f1b88100100 2024-03-24T00:06:08.086 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-24T00:06:08.085+0000 7f1b8d3f4700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f1b88100100 0x7f1b88101ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:06:08.086 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-24T00:06:08.085+0000 7f1b8d3f4700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f1b88100100 0x7f1b88101ba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.193:56556/0 (socket says 172.21.15.193:56556) 2024-03-24T00:06:08.086 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-24T00:06:08.085+0000 7f1b8d3f4700 1 -- 172.21.15.193:0/1890552299 learned_addr learned my addr 172.21.15.193:0/1890552299 (peer_addr_for_me v2:172.21.15.193:0/0) 2024-03-24T00:06:08.086 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-24T00:06:08.085+0000 7f1b8d3f4700 1 -- 172.21.15.193:0/1890552299 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f1b88100100 msgr2=0x7f1b88101ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 12 2024-03-24T00:06:08.086 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-24T00:06:08.085+0000 7f1b8d3f4700 1 -- 172.21.15.193:0/1890552299 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f1b88100100 msgr2=0x7f1b88101ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-03-24T00:06:08.087 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-24T00:06:08.085+0000 7f1b8d3f4700 1 --2- 172.21.15.193:0/1890552299 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f1b88100100 0x7f1b88101ba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-03-24T00:06:08.087 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-24T00:06:08.085+0000 7f1b8d3f4700 1 --2- 172.21.15.193:0/1890552299 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f1b88100100 0x7f1b88101ba0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-24T00:06:08.087 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-24T00:06:08.085+0000 7f1b8cbf3700 1 --2- 172.21.15.193:0/1890552299 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f1b881020e0 0x7f1b88102560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:06:08.088 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-24T00:06:08.086+0000 7f1b8cbf3700 1 -- 172.21.15.193:0/1890552299 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f1b88100100 msgr2=0x7f1b88101ba0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-24T00:06:08.088 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-24T00:06:08.086+0000 7f1b8cbf3700 1 --2- 172.21.15.193:0/1890552299 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f1b88100100 0x7f1b88101ba0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:06:08.088 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-24T00:06:08.086+0000 7f1b8cbf3700 1 -- 172.21.15.193:0/1890552299 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1b88103ac0 con 0x7f1b881020e0 2024-03-24T00:06:08.088 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-24T00:06:08.086+0000 7f1b8cbf3700 1 --2- 172.21.15.193:0/1890552299 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f1b881020e0 0x7f1b88102560 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f1b7800b4d0 tx=0x7f1b7800b890 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:06:08.089 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-24T00:06:08.086+0000 7f1b7e7fc700 1 -- 172.21.15.193:0/1890552299 <== mon.0 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 309+0+0 (secure 0 0 0) 0x7f1b78011630 con 0x7f1b881020e0 2024-03-24T00:06:08.089 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-24T00:06:08.087+0000 7f1b8f658700 1 -- 172.21.15.193:0/1890552299 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1b881a3ba0 con 0x7f1b881020e0 2024-03-24T00:06:08.089 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-24T00:06:08.087+0000 7f1b8f658700 1 -- 172.21.15.193:0/1890552299 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1b881a40e0 con 0x7f1b881020e0 2024-03-24T00:06:08.089 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-24T00:06:08.087+0000 7f1b7e7fc700 1 -- 172.21.15.193:0/1890552299 <== mon.0 v2:172.21.15.118:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f1b78011cb0 con 0x7f1b881020e0 2024-03-24T00:06:08.089 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-24T00:06:08.087+0000 7f1b7e7fc700 1 -- 172.21.15.193:0/1890552299 <== mon.0 v2:172.21.15.118:3300/0 3 ==== mon_map magic: 0 v1 ==== 309+0+0 (secure 0 0 0) 0x7f1b7801ae20 con 0x7f1b881020e0 2024-03-24T00:06:08.089 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-24T00:06:08.087+0000 7f1b8f658700 1 -- 172.21.15.193:0/1890552299 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1b6c005330 con 0x7f1b881020e0 2024-03-24T00:06:08.089 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-24T00:06:08.088+0000 7f1b7e7fc700 1 -- 172.21.15.193:0/1890552299 <== mon.0 v2:172.21.15.118:3300/0 4 ==== mgrmap(e 13) v1 ==== 46163+0+0 (secure 0 0 0) 0x7f1b78018020 con 0x7f1b881020e0 2024-03-24T00:06:08.089 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-24T00:06:08.088+0000 7f1b7e7fc700 1 --2- 172.21.15.193:0/1890552299 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7f1b74038d60 0x7f1b7403b220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:06:08.089 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-24T00:06:08.088+0000 7f1b7e7fc700 1 -- 172.21.15.193:0/1890552299 <== mon.0 v2:172.21.15.118:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1155+0+0 (secure 0 0 0) 0x7f1b7804bd70 con 0x7f1b881020e0 2024-03-24T00:06:08.089 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-24T00:06:08.089+0000 7f1b8d3f4700 1 --2- 172.21.15.193:0/1890552299 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7f1b74038d60 0x7f1b7403b220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:06:08.090 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-24T00:06:08.090+0000 7f1b8d3f4700 1 --2- 172.21.15.193:0/1890552299 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7f1b74038d60 0x7f1b7403b220 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f1b84008b30 tx=0x7f1b84005040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:06:08.103 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-24T00:06:08.104+0000 7f1b7e7fc700 1 -- 172.21.15.193:0/1890552299 <== mon.0 v2:172.21.15.118:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f1b7801f020 con 0x7f1b881020e0 2024-03-24T00:06:08.377 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-24T00:06:08.376+0000 7f1b8f658700 1 -- 172.21.15.193:0/1890552299 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f1b6c0058f0 con 0x7f1b881020e0 2024-03-24T00:06:08.377 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-24T00:06:08.377+0000 7f1b7e7fc700 1 -- 172.21.15.193:0/1890552299 <== mon.0 v2:172.21.15.118:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 2 v2) v1 ==== 95+0+1019 (secure 0 0 0) 0x7f1b78010380 con 0x7f1b881020e0 2024-03-24T00:06:08.377 INFO:teuthology.orchestra.run.smithi193.stdout: 2024-03-24T00:06:08.378 INFO:teuthology.orchestra.run.smithi193.stdout:{"epoch":2,"fsid":"d79a5420-e971-11ee-95cd-87774f69a715","modified":"2024-03-24T00:05:59.445188Z","created":"2024-03-24T00:03:54.425344Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:3300","nonce":0},{"type":"v1","addr":"172.21.15.118:6789","nonce":0}]},"addr":"172.21.15.118:6789/0","public_addr":"172.21.15.118:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:3300","nonce":0},{"type":"v1","addr":"172.21.15.193:6789","nonce":0}]},"addr":"172.21.15.193:6789/0","public_addr":"172.21.15.193:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0,1]} 2024-03-24T00:06:08.380 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-24T00:06:08.379+0000 7f1b8f658700 1 -- 172.21.15.193:0/1890552299 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7f1b74038d60 msgr2=0x7f1b7403b220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:06:08.380 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-24T00:06:08.379+0000 7f1b8f658700 1 --2- 172.21.15.193:0/1890552299 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7f1b74038d60 0x7f1b7403b220 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 crypto rx=0x7f1b84008b30 tx=0x7f1b84005040 comp rx=0 tx=0).stop 2024-03-24T00:06:08.380 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-24T00:06:08.379+0000 7f1b8f658700 1 -- 172.21.15.193:0/1890552299 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f1b881020e0 msgr2=0x7f1b88102560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:06:08.380 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-24T00:06:08.379+0000 7f1b8f658700 1 --2- 172.21.15.193:0/1890552299 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f1b881020e0 0x7f1b88102560 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f1b7800b4d0 tx=0x7f1b7800b890 comp rx=0 tx=0).stop 2024-03-24T00:06:08.380 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-24T00:06:08.379+0000 7f1b8f658700 1 -- 172.21.15.193:0/1890552299 shutdown_connections 2024-03-24T00:06:08.380 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-24T00:06:08.379+0000 7f1b8f658700 1 --2- 172.21.15.193:0/1890552299 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7f1b74038d60 0x7f1b7403b220 unknown :-1 s=CLOSED pgs=30 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:06:08.380 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-24T00:06:08.379+0000 7f1b8f658700 1 --2- 172.21.15.193:0/1890552299 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f1b88100100 0x7f1b88101ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:06:08.380 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-24T00:06:08.379+0000 7f1b8f658700 1 --2- 172.21.15.193:0/1890552299 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f1b881020e0 0x7f1b88102560 unknown :-1 s=CLOSED pgs=103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:06:08.380 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-24T00:06:08.379+0000 7f1b8f658700 1 -- 172.21.15.193:0/1890552299 >> 172.21.15.193:0/1890552299 conn(0x7f1b880fb990 msgr2=0x7f1b88107c70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:06:08.380 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-24T00:06:08.380+0000 7f1b8f658700 1 -- 172.21.15.193:0/1890552299 shutdown_connections 2024-03-24T00:06:08.380 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-24T00:06:08.380+0000 7f1b8f658700 1 -- 172.21.15.193:0/1890552299 wait complete. 2024-03-24T00:06:08.381 INFO:teuthology.orchestra.run.smithi193.stderr:dumped monmap epoch 2 2024-03-24T00:06:08.408 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:06:08 smithi189 podman[36168]: 2024-03-24 00:06:08.385577846 +0000 UTC m=+0.362098163 container start 332ea823179e88f4b6faf6793d78c9576c10138bb235aa5cd94334b2af6c7389 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-d79a5420-e971-11ee-95cd-87774f69a715-mon-b, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.build-date=20240319, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, io.buildah.version=1.33.5) 2024-03-24T00:06:08.704 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:06:08 smithi189 ceph-mon[36187]: set uid:gid to 167:167 (ceph:ceph) 2024-03-24T00:06:08.704 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:06:08 smithi189 ceph-mon[36187]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev), process ceph-mon, pid 2 2024-03-24T00:06:08.704 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:06:08 smithi189 ceph-mon[36187]: pidfile_write: ignore empty --pid-file 2024-03-24T00:06:08.704 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:06:08 smithi189 ceph-mon[36187]: load: jerasure load: lrc 2024-03-24T00:06:08.705 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:06:08 smithi189 ceph-mon[36187]: rocksdb: RocksDB version: 7.9.2 2024-03-24T00:06:08.705 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:06:08 smithi189 ceph-mon[36187]: rocksdb: Git sha 0 2024-03-24T00:06:08.705 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:06:08 smithi189 ceph-mon[36187]: rocksdb: Compile date 2024-03-22 04:58:55 2024-03-24T00:06:08.705 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:06:08 smithi189 ceph-mon[36187]: rocksdb: DB SUMMARY 2024-03-24T00:06:08.705 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:06:08 smithi189 ceph-mon[36187]: rocksdb: DB Session ID: KTQQ8DS913X6DB7P3LB9 2024-03-24T00:06:08.705 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:06:08 smithi189 ceph-mon[36187]: rocksdb: CURRENT file: CURRENT 2024-03-24T00:06:08.705 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:06:08 smithi189 ceph-mon[36187]: rocksdb: IDENTITY file: IDENTITY 2024-03-24T00:06:08.705 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:06:08 smithi189 ceph-mon[36187]: rocksdb: MANIFEST file: MANIFEST-000005 size: 59 Bytes 2024-03-24T00:06:08.705 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:06:08 smithi189 ceph-mon[36187]: rocksdb: SST files in /var/lib/ceph/mon/ceph-b/store.db dir, Total Num: 0, files: 2024-03-24T00:06:08.705 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:06:08 smithi189 ceph-mon[36187]: rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-b/store.db: 000004.log size: 636 ; 2024-03-24T00:06:08.705 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:06:08 smithi189 ceph-mon[36187]: rocksdb: Options.error_if_exists: 0 2024-03-24T00:06:08.705 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:06:08 smithi189 ceph-mon[36187]: rocksdb: Options.create_if_missing: 0 2024-03-24T00:06:08.705 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:06:08 smithi189 ceph-mon[36187]: rocksdb: Options.paranoid_checks: 1 2024-03-24T00:06:08.705 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:06:08 smithi189 ceph-mon[36187]: rocksdb: Options.flush_verify_memtable_count: 1 2024-03-24T00:06:08.705 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:06:08 smithi189 ceph-mon[36187]: rocksdb: Options.track_and_verify_wals_in_manifest: 0 2024-03-24T00:06:08.706 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:06:08 smithi189 ceph-mon[36187]: rocksdb: Options.verify_sst_unique_id_in_manifest: 1 2024-03-24T00:06:08.706 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:06:08 smithi189 ceph-mon[36187]: rocksdb: Options.env: 0x556804490b20 2024-03-24T00:06:08.706 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:06:08 smithi189 ceph-mon[36187]: rocksdb: Options.fs: PosixFileSystem 2024-03-24T00:06:08.706 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:06:08 smithi189 ceph-mon[36187]: rocksdb: Options.info_log: 0x5568057f7920 2024-03-24T00:06:08.706 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:06:08 smithi189 ceph-mon[36187]: rocksdb: Options.max_file_opening_threads: 16 2024-03-24T00:06:08.706 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:06:08 smithi189 ceph-mon[36187]: rocksdb: Options.statistics: (nil) 2024-03-24T00:06:08.706 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:06:08 smithi189 ceph-mon[36187]: rocksdb: Options.use_fsync: 0 2024-03-24T00:06:08.706 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:06:08 smithi189 ceph-mon[36187]: rocksdb: Options.max_log_file_size: 0 2024-03-24T00:06:08.706 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:06:08 smithi189 ceph-mon[36187]: rocksdb: Options.max_manifest_file_size: 1073741824 2024-03-24T00:06:08.706 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:06:08 smithi189 ceph-mon[36187]: rocksdb: Options.log_file_time_to_roll: 0 2024-03-24T00:06:08.706 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:06:08 smithi189 ceph-mon[36187]: rocksdb: Options.keep_log_file_num: 1000 2024-03-24T00:06:08.706 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:06:08 smithi189 ceph-mon[36187]: rocksdb: Options.recycle_log_file_num: 0 2024-03-24T00:06:08.706 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:06:08 smithi189 ceph-mon[36187]: rocksdb: Options.allow_fallocate: 1 2024-03-24T00:06:08.707 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:06:08 smithi189 ceph-mon[36187]: rocksdb: Options.allow_mmap_reads: 0 2024-03-24T00:06:08.707 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:06:08 smithi189 ceph-mon[36187]: rocksdb: Options.allow_mmap_writes: 0 2024-03-24T00:06:08.707 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:06:08 smithi189 ceph-mon[36187]: rocksdb: Options.use_direct_reads: 0 2024-03-24T00:06:08.707 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:06:08 smithi189 ceph-mon[36187]: rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2024-03-24T00:06:08.707 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:06:08 smithi189 ceph-mon[36187]: rocksdb: Options.create_missing_column_families: 0 2024-03-24T00:06:08.707 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:06:08 smithi189 ceph-mon[36187]: rocksdb: Options.db_log_dir: 2024-03-24T00:06:08.707 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:06:08 smithi189 ceph-mon[36187]: rocksdb: Options.wal_dir: 2024-03-24T00:06:08.707 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:06:08 smithi189 ceph-mon[36187]: rocksdb: Options.table_cache_numshardbits: 6 2024-03-24T00:06:08.709 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:06:08 smithi189 ceph-mon[36187]: rocksdb: Options.WAL_ttl_seconds: 0 2024-03-24T00:06:08.709 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:06:08 smithi189 ceph-mon[36187]: rocksdb: Options.WAL_size_limit_MB: 0 2024-03-24T00:06:08.709 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:06:08 smithi189 ceph-mon[36187]: rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2024-03-24T00:06:08.709 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:06:08 smithi189 ceph-mon[36187]: rocksdb: Options.manifest_preallocation_size: 4194304 2024-03-24T00:06:08.709 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:06:08 smithi189 ceph-mon[36187]: rocksdb: Options.is_fd_close_on_exec: 1 2024-03-24T00:06:08.709 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:06:08 smithi189 ceph-mon[36187]: rocksdb: Options.advise_random_on_open: 1 2024-03-24T00:06:08.709 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:06:08 smithi189 ceph-mon[36187]: rocksdb: Options.db_write_buffer_size: 0 2024-03-24T00:06:08.709 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:06:08 smithi189 ceph-mon[36187]: rocksdb: Options.write_buffer_manager: 0x5568057ca1e0 2024-03-24T00:06:08.709 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:06:08 smithi189 ceph-mon[36187]: rocksdb: Options.access_hint_on_compaction_start: 1 2024-03-24T00:06:08.709 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:06:08 smithi189 ceph-mon[36187]: rocksdb: Options.random_access_max_buffer_size: 1048576 2024-03-24T00:06:08.709 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:06:08 smithi189 ceph-mon[36187]: rocksdb: Options.use_adaptive_mutex: 0 2024-03-24T00:06:08.710 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:06:08 smithi189 ceph-mon[36187]: rocksdb: Options.rate_limiter: (nil) 2024-03-24T00:06:08.710 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:06:08 smithi189 ceph-mon[36187]: rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2024-03-24T00:06:08.710 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:06:08 smithi189 ceph-mon[36187]: rocksdb: Options.wal_recovery_mode: 2 2024-03-24T00:06:08.710 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:06:08 smithi189 ceph-mon[36187]: rocksdb: Options.enable_thread_tracking: 0 2024-03-24T00:06:08.710 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:06:08 smithi189 ceph-mon[36187]: rocksdb: Options.enable_pipelined_write: 0 2024-03-24T00:06:08.710 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:06:08 smithi189 ceph-mon[36187]: rocksdb: Options.unordered_write: 0 2024-03-24T00:06:08.710 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:06:08 smithi189 ceph-mon[36187]: rocksdb: Options.allow_concurrent_memtable_write: 1 2024-03-24T00:06:08.710 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:06:08 smithi189 ceph-mon[36187]: rocksdb: Options.enable_write_thread_adaptive_yield: 1 2024-03-24T00:06:08.710 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:06:08 smithi189 ceph-mon[36187]: rocksdb: Options.write_thread_max_yield_usec: 100 2024-03-24T00:06:08.710 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:06:08 smithi189 ceph-mon[36187]: rocksdb: Options.write_thread_slow_yield_usec: 3 2024-03-24T00:06:08.710 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:06:08 smithi189 ceph-mon[36187]: rocksdb: Options.row_cache: None 2024-03-24T00:06:08.710 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:06:08 smithi189 ceph-mon[36187]: rocksdb: Options.wal_filter: None 2024-03-24T00:06:08.710 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:06:08 smithi189 ceph-mon[36187]: rocksdb: Options.avoid_flush_during_recovery: 0 2024-03-24T00:06:08.710 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:06:08 smithi189 ceph-mon[36187]: rocksdb: Options.allow_ingest_behind: 0 2024-03-24T00:06:08.711 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:06:08 smithi189 ceph-mon[36187]: rocksdb: Options.two_write_queues: 0 2024-03-24T00:06:08.711 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:06:08 smithi189 ceph-mon[36187]: rocksdb: Options.manual_wal_flush: 0 2024-03-24T00:06:08.711 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:06:08 smithi189 ceph-mon[36187]: rocksdb: Options.wal_compression: 0 2024-03-24T00:06:08.711 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:06:08 smithi189 ceph-mon[36187]: rocksdb: Options.atomic_flush: 0 2024-03-24T00:06:08.711 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:06:08 smithi189 ceph-mon[36187]: rocksdb: Options.avoid_unnecessary_blocking_io: 0 2024-03-24T00:06:08.711 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:06:08 smithi189 ceph-mon[36187]: rocksdb: Options.persist_stats_to_disk: 0 2024-03-24T00:06:08.711 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:06:08 smithi189 ceph-mon[36187]: rocksdb: Options.write_dbid_to_manifest: 0 2024-03-24T00:06:08.711 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:06:08 smithi189 ceph-mon[36187]: rocksdb: Options.log_readahead_size: 0 2024-03-24T00:06:08.711 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:06:08 smithi189 ceph-mon[36187]: rocksdb: Options.file_checksum_gen_factory: Unknown 2024-03-24T00:06:08.711 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:06:08 smithi189 ceph-mon[36187]: rocksdb: Options.best_efforts_recovery: 0 2024-03-24T00:06:08.711 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:06:08 smithi189 ceph-mon[36187]: rocksdb: Options.max_bgerror_resume_count: 2147483647 2024-03-24T00:06:08.711 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:06:08 smithi189 ceph-mon[36187]: rocksdb: Options.bgerror_resume_retry_interval: 1000000 2024-03-24T00:06:08.711 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:06:08 smithi189 ceph-mon[36187]: rocksdb: Options.allow_data_in_errors: 0 2024-03-24T00:06:08.711 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:06:08 smithi189 ceph-mon[36187]: rocksdb: Options.db_host_id: __hostname__ 2024-03-24T00:06:08.712 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:06:08 smithi189 ceph-mon[36187]: rocksdb: Options.enforce_single_del_contracts: true 2024-03-24T00:06:08.712 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:06:08 smithi189 ceph-mon[36187]: rocksdb: Options.max_background_jobs: 2 2024-03-24T00:06:08.712 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:06:08 smithi189 ceph-mon[36187]: rocksdb: Options.max_background_compactions: -1 2024-03-24T00:06:08.712 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:06:08 smithi189 ceph-mon[36187]: rocksdb: Options.max_subcompactions: 1 2024-03-24T00:06:08.712 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:06:08 smithi189 ceph-mon[36187]: rocksdb: Options.avoid_flush_during_shutdown: 0 2024-03-24T00:06:08.712 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:06:08 smithi189 ceph-mon[36187]: rocksdb: Options.writable_file_max_buffer_size: 1048576 2024-03-24T00:06:08.712 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:06:08 smithi189 ceph-mon[36187]: rocksdb: Options.delayed_write_rate : 16777216 2024-03-24T00:06:08.712 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:06:08 smithi189 ceph-mon[36187]: rocksdb: Options.max_total_wal_size: 0 2024-03-24T00:06:08.712 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:06:08 smithi189 ceph-mon[36187]: rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2024-03-24T00:06:08.712 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:06:08 smithi189 ceph-mon[36187]: rocksdb: Options.stats_dump_period_sec: 600 2024-03-24T00:06:08.712 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:06:08 smithi189 ceph-mon[36187]: rocksdb: Options.stats_persist_period_sec: 600 2024-03-24T00:06:08.712 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:06:08 smithi189 ceph-mon[36187]: rocksdb: Options.stats_history_buffer_size: 1048576 2024-03-24T00:06:08.712 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:06:08 smithi189 ceph-mon[36187]: rocksdb: Options.max_open_files: -1 2024-03-24T00:06:08.712 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:06:08 smithi189 ceph-mon[36187]: rocksdb: Options.bytes_per_sync: 0 2024-03-24T00:06:08.713 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:06:08 smithi189 ceph-mon[36187]: rocksdb: Options.wal_bytes_per_sync: 0 2024-03-24T00:06:08.713 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:06:08 smithi189 ceph-mon[36187]: rocksdb: Options.strict_bytes_per_sync: 0 2024-03-24T00:06:08.713 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:06:08 smithi189 ceph-mon[36187]: rocksdb: Options.compaction_readahead_size: 0 2024-03-24T00:06:08.713 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:06:08 smithi189 ceph-mon[36187]: rocksdb: Options.max_background_flushes: -1 2024-03-24T00:06:08.713 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:06:08 smithi189 ceph-mon[36187]: rocksdb: Compression algorithms supported: 2024-03-24T00:06:08.713 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:06:08 smithi189 ceph-mon[36187]: rocksdb: kZSTDNotFinalCompression supported: 0 2024-03-24T00:06:08.713 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:06:08 smithi189 ceph-mon[36187]: rocksdb: kZSTD supported: 0 2024-03-24T00:06:08.713 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:06:08 smithi189 ceph-mon[36187]: rocksdb: kXpressCompression supported: 0 2024-03-24T00:06:08.713 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:06:08 smithi189 ceph-mon[36187]: rocksdb: kLZ4HCCompression supported: 1 2024-03-24T00:06:08.713 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:06:08 smithi189 ceph-mon[36187]: rocksdb: kZlibCompression supported: 1 2024-03-24T00:06:08.713 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:06:08 smithi189 ceph-mon[36187]: rocksdb: kSnappyCompression supported: 1 2024-03-24T00:06:08.713 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:06:08 smithi189 ceph-mon[36187]: rocksdb: kLZ4Compression supported: 1 2024-03-24T00:06:08.713 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:06:08 smithi189 ceph-mon[36187]: rocksdb: kBZip2Compression supported: 0 2024-03-24T00:06:08.713 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:06:08 smithi189 ceph-mon[36187]: rocksdb: Fast CRC32 supported: Supported on x86 2024-03-24T00:06:08.714 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:06:08 smithi189 ceph-mon[36187]: rocksdb: DMutex implementation: pthread_mutex_t 2024-03-24T00:06:08.714 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:06:08 smithi189 ceph-mon[36187]: rocksdb: [db/version_set.cc:5527] Recovering from manifest file: /var/lib/ceph/mon/ceph-b/store.db/MANIFEST-000005 2024-03-24T00:06:08.714 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:06:08 smithi189 ceph-mon[36187]: rocksdb: [db/column_family.cc:630] --------------- Options for column family [default]: 2024-03-24T00:06:08.714 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:06:08 smithi189 ceph-mon[36187]: rocksdb: Options.comparator: leveldb.BytewiseComparator 2024-03-24T00:06:08.714 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:06:08 smithi189 ceph-mon[36187]: rocksdb: Options.merge_operator: 2024-03-24T00:06:08.714 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:06:08 smithi189 ceph-mon[36187]: rocksdb: Options.compaction_filter: None 2024-03-24T00:06:08.714 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:06:08 smithi189 ceph-mon[36187]: rocksdb: Options.compaction_filter_factory: None 2024-03-24T00:06:08.714 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:06:08 smithi189 ceph-mon[36187]: rocksdb: Options.sst_partitioner_factory: None 2024-03-24T00:06:08.714 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:06:08 smithi189 ceph-mon[36187]: rocksdb: Options.memtable_factory: SkipListFactory 2024-03-24T00:06:08.714 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:06:08 smithi189 ceph-mon[36187]: rocksdb: Options.table_factory: BlockBasedTable 2024-03-24T00:06:08.714 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:06:08 smithi189 ceph-mon[36187]: rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x5568057f7a20) 2024-03-24T00:06:08.714 INFO:journalctl@ceph.mon.b.smithi189.stdout: cache_index_and_filter_blocks: 1 2024-03-24T00:06:08.714 INFO:journalctl@ceph.mon.b.smithi189.stdout: cache_index_and_filter_blocks_with_high_priority: 0 2024-03-24T00:06:08.715 INFO:journalctl@ceph.mon.b.smithi189.stdout: pin_l0_filter_and_index_blocks_in_cache: 0 2024-03-24T00:06:08.715 INFO:journalctl@ceph.mon.b.smithi189.stdout: pin_top_level_index_and_filter: 1 2024-03-24T00:06:08.715 INFO:journalctl@ceph.mon.b.smithi189.stdout: index_type: 0 2024-03-24T00:06:08.715 INFO:journalctl@ceph.mon.b.smithi189.stdout: data_block_index_type: 0 2024-03-24T00:06:08.715 INFO:journalctl@ceph.mon.b.smithi189.stdout: index_shortening: 1 2024-03-24T00:06:08.715 INFO:journalctl@ceph.mon.b.smithi189.stdout: data_block_hash_table_util_ratio: 0.750000 2024-03-24T00:06:08.715 INFO:journalctl@ceph.mon.b.smithi189.stdout: checksum: 4 2024-03-24T00:06:08.715 INFO:journalctl@ceph.mon.b.smithi189.stdout: no_block_cache: 0 2024-03-24T00:06:08.715 INFO:journalctl@ceph.mon.b.smithi189.stdout: block_cache: 0x5568057fd090 2024-03-24T00:06:08.715 INFO:journalctl@ceph.mon.b.smithi189.stdout: block_cache_name: BinnedLRUCache 2024-03-24T00:06:08.715 INFO:journalctl@ceph.mon.b.smithi189.stdout: block_cache_options: 2024-03-24T00:06:08.715 INFO:journalctl@ceph.mon.b.smithi189.stdout: capacity : 536870912 2024-03-24T00:06:08.715 INFO:journalctl@ceph.mon.b.smithi189.stdout: num_shard_bits : 4 2024-03-24T00:06:08.715 INFO:journalctl@ceph.mon.b.smithi189.stdout: strict_capacity_limit : 0 2024-03-24T00:06:08.716 INFO:journalctl@ceph.mon.b.smithi189.stdout: high_pri_pool_ratio: 0.000 2024-03-24T00:06:08.716 INFO:journalctl@ceph.mon.b.smithi189.stdout: block_cache_compressed: (nil) 2024-03-24T00:06:08.716 INFO:journalctl@ceph.mon.b.smithi189.stdout: persistent_cache: (nil) 2024-03-24T00:06:08.716 INFO:journalctl@ceph.mon.b.smithi189.stdout: block_size: 4096 2024-03-24T00:06:08.716 INFO:journalctl@ceph.mon.b.smithi189.stdout: block_size_deviation: 10 2024-03-24T00:06:08.716 INFO:journalctl@ceph.mon.b.smithi189.stdout: block_restart_interval: 16 2024-03-24T00:06:08.716 INFO:journalctl@ceph.mon.b.smithi189.stdout: index_block_restart_interval: 1 2024-03-24T00:06:08.716 INFO:journalctl@ceph.mon.b.smithi189.stdout: metadata_block_size: 4096 2024-03-24T00:06:08.716 INFO:journalctl@ceph.mon.b.smithi189.stdout: partition_filters: 0 2024-03-24T00:06:08.716 INFO:journalctl@ceph.mon.b.smithi189.stdout: use_delta_encoding: 1 2024-03-24T00:06:08.716 INFO:journalctl@ceph.mon.b.smithi189.stdout: filter_policy: bloomfilter 2024-03-24T00:06:08.716 INFO:journalctl@ceph.mon.b.smithi189.stdout: whole_key_filtering: 1 2024-03-24T00:06:08.716 INFO:journalctl@ceph.mon.b.smithi189.stdout: verify_compression: 0 2024-03-24T00:06:08.716 INFO:journalctl@ceph.mon.b.smithi189.stdout: read_amp_bytes_per_bit: 0 2024-03-24T00:06:08.716 INFO:journalctl@ceph.mon.b.smithi189.stdout: format_version: 5 2024-03-24T00:06:08.717 INFO:journalctl@ceph.mon.b.smithi189.stdout: enable_index_compression: 1 2024-03-24T00:06:08.717 INFO:journalctl@ceph.mon.b.smithi189.stdout: block_align: 0 2024-03-24T00:06:08.717 INFO:journalctl@ceph.mon.b.smithi189.stdout: max_auto_readahead_size: 262144 2024-03-24T00:06:08.717 INFO:journalctl@ceph.mon.b.smithi189.stdout: prepopulate_block_cache: 0 2024-03-24T00:06:08.717 INFO:journalctl@ceph.mon.b.smithi189.stdout: initial_auto_readahead_size: 8192 2024-03-24T00:06:08.717 INFO:journalctl@ceph.mon.b.smithi189.stdout: num_file_reads_for_auto_readahead: 2 2024-03-24T00:06:08.717 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:06:08 smithi189 ceph-mon[36187]: rocksdb: Options.write_buffer_size: 33554432 2024-03-24T00:06:08.717 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:06:08 smithi189 ceph-mon[36187]: rocksdb: Options.max_write_buffer_number: 2 2024-03-24T00:06:08.717 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:06:08 smithi189 ceph-mon[36187]: rocksdb: Options.compression: NoCompression 2024-03-24T00:06:08.717 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:06:08 smithi189 ceph-mon[36187]: rocksdb: Options.bottommost_compression: Disabled 2024-03-24T00:06:08.717 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:06:08 smithi189 ceph-mon[36187]: rocksdb: Options.prefix_extractor: nullptr 2024-03-24T00:06:08.717 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:06:08 smithi189 ceph-mon[36187]: rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2024-03-24T00:06:08.717 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:06:08 smithi189 ceph-mon[36187]: rocksdb: Options.num_levels: 7 2024-03-24T00:06:08.717 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:06:08 smithi189 ceph-mon[36187]: rocksdb: Options.min_write_buffer_number_to_merge: 1 2024-03-24T00:06:08.718 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:06:08 smithi189 ceph-mon[36187]: rocksdb: Options.max_write_buffer_number_to_maintain: 0 2024-03-24T00:06:08.718 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:06:08 smithi189 ceph-mon[36187]: rocksdb: Options.max_write_buffer_size_to_maintain: 0 2024-03-24T00:06:08.718 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:06:08 smithi189 ceph-mon[36187]: rocksdb: Options.bottommost_compression_opts.window_bits: -14 2024-03-24T00:06:08.718 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:06:08 smithi189 ceph-mon[36187]: rocksdb: Options.bottommost_compression_opts.level: 32767 2024-03-24T00:06:08.718 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:06:08 smithi189 ceph-mon[36187]: rocksdb: Options.bottommost_compression_opts.strategy: 0 2024-03-24T00:06:08.718 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:06:08 smithi189 ceph-mon[36187]: rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2024-03-24T00:06:08.718 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:06:08 smithi189 ceph-mon[36187]: rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2024-03-24T00:06:08.718 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:06:08 smithi189 ceph-mon[36187]: rocksdb: Options.bottommost_compression_opts.parallel_threads: 1 2024-03-24T00:06:08.718 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:06:08 smithi189 ceph-mon[36187]: rocksdb: Options.bottommost_compression_opts.enabled: false 2024-03-24T00:06:08.718 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:06:08 smithi189 ceph-mon[36187]: rocksdb: Options.bottommost_compression_opts.max_dict_buffer_bytes: 0 2024-03-24T00:06:08.718 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:06:08 smithi189 ceph-mon[36187]: rocksdb: Options.bottommost_compression_opts.use_zstd_dict_trainer: true 2024-03-24T00:06:08.718 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:06:08 smithi189 ceph-mon[36187]: rocksdb: Options.compression_opts.window_bits: -14 2024-03-24T00:06:08.718 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:06:08 smithi189 ceph-mon[36187]: rocksdb: Options.compression_opts.level: 32767 2024-03-24T00:06:08.718 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:06:08 smithi189 ceph-mon[36187]: rocksdb: Options.compression_opts.strategy: 0 2024-03-24T00:06:08.719 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:06:08 smithi189 ceph-mon[36187]: rocksdb: Options.compression_opts.max_dict_bytes: 0 2024-03-24T00:06:08.719 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:06:08 smithi189 ceph-mon[36187]: rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2024-03-24T00:06:08.719 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:06:08 smithi189 ceph-mon[36187]: rocksdb: Options.compression_opts.use_zstd_dict_trainer: true 2024-03-24T00:06:08.719 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:06:08 smithi189 ceph-mon[36187]: rocksdb: Options.compression_opts.parallel_threads: 1 2024-03-24T00:06:08.719 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:06:08 smithi189 ceph-mon[36187]: rocksdb: Options.compression_opts.enabled: false 2024-03-24T00:06:08.719 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:06:08 smithi189 ceph-mon[36187]: rocksdb: Options.compression_opts.max_dict_buffer_bytes: 0 2024-03-24T00:06:08.719 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:06:08 smithi189 ceph-mon[36187]: rocksdb: Options.level0_file_num_compaction_trigger: 4 2024-03-24T00:06:08.719 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:06:08 smithi189 ceph-mon[36187]: rocksdb: Options.level0_slowdown_writes_trigger: 20 2024-03-24T00:06:08.719 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:06:08 smithi189 ceph-mon[36187]: rocksdb: Options.level0_stop_writes_trigger: 36 2024-03-24T00:06:08.719 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:06:08 smithi189 ceph-mon[36187]: rocksdb: Options.target_file_size_base: 67108864 2024-03-24T00:06:08.719 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:06:08 smithi189 ceph-mon[36187]: rocksdb: Options.target_file_size_multiplier: 1 2024-03-24T00:06:08.719 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:06:08 smithi189 ceph-mon[36187]: rocksdb: Options.max_bytes_for_level_base: 268435456 2024-03-24T00:06:08.719 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:06:08 smithi189 ceph-mon[36187]: rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2024-03-24T00:06:08.719 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:06:08 smithi189 ceph-mon[36187]: rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2024-03-24T00:06:08.720 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:06:08 smithi189 ceph-mon[36187]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2024-03-24T00:06:08.720 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:06:08 smithi189 ceph-mon[36187]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2024-03-24T00:06:08.720 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:06:08 smithi189 ceph-mon[36187]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2024-03-24T00:06:08.720 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:06:08 smithi189 ceph-mon[36187]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2024-03-24T00:06:08.720 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:06:08 smithi189 ceph-mon[36187]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2024-03-24T00:06:08.720 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:06:08 smithi189 ceph-mon[36187]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2024-03-24T00:06:08.720 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:06:08 smithi189 ceph-mon[36187]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2024-03-24T00:06:08.720 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:06:08 smithi189 ceph-mon[36187]: rocksdb: Options.max_sequential_skip_in_iterations: 8 2024-03-24T00:06:08.720 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:06:08 smithi189 ceph-mon[36187]: rocksdb: Options.max_compaction_bytes: 1677721600 2024-03-24T00:06:08.720 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:06:08 smithi189 ceph-mon[36187]: rocksdb: Options.ignore_max_compaction_bytes_for_input: true 2024-03-24T00:06:08.720 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:06:08 smithi189 ceph-mon[36187]: rocksdb: Options.arena_block_size: 1048576 2024-03-24T00:06:08.720 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:06:08 smithi189 ceph-mon[36187]: rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2024-03-24T00:06:08.720 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:06:08 smithi189 ceph-mon[36187]: rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2024-03-24T00:06:08.720 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:06:08 smithi189 ceph-mon[36187]: rocksdb: Options.disable_auto_compactions: 0 2024-03-24T00:06:08.720 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:06:08 smithi189 ceph-mon[36187]: rocksdb: Options.compaction_style: kCompactionStyleLevel 2024-03-24T00:06:08.721 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:06:08 smithi189 ceph-mon[36187]: rocksdb: Options.compaction_pri: kMinOverlappingRatio 2024-03-24T00:06:08.721 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:06:08 smithi189 ceph-mon[36187]: rocksdb: Options.compaction_options_universal.size_ratio: 1 2024-03-24T00:06:08.721 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:06:08 smithi189 ceph-mon[36187]: rocksdb: Options.compaction_options_universal.min_merge_width: 2 2024-03-24T00:06:08.721 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:06:08 smithi189 ceph-mon[36187]: rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2024-03-24T00:06:08.721 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:06:08 smithi189 ceph-mon[36187]: rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2024-03-24T00:06:08.721 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:06:08 smithi189 ceph-mon[36187]: rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2024-03-24T00:06:08.721 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:06:08 smithi189 ceph-mon[36187]: rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2024-03-24T00:06:08.721 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:06:08 smithi189 ceph-mon[36187]: rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2024-03-24T00:06:08.721 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:06:08 smithi189 ceph-mon[36187]: rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2024-03-24T00:06:08.721 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:06:08 smithi189 ceph-mon[36187]: rocksdb: Options.table_properties_collectors: 2024-03-24T00:06:08.721 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:06:08 smithi189 ceph-mon[36187]: rocksdb: Options.inplace_update_support: 0 2024-03-24T00:06:08.721 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:06:08 smithi189 ceph-mon[36187]: rocksdb: Options.inplace_update_num_locks: 10000 2024-03-24T00:06:08.721 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:06:08 smithi189 ceph-mon[36187]: rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2024-03-24T00:06:08.721 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:06:08 smithi189 ceph-mon[36187]: rocksdb: Options.memtable_whole_key_filtering: 0 2024-03-24T00:06:08.721 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:06:08 smithi189 ceph-mon[36187]: rocksdb: Options.memtable_huge_page_size: 0 2024-03-24T00:06:08.721 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:06:08 smithi189 ceph-mon[36187]: rocksdb: Options.bloom_locality: 0 2024-03-24T00:06:08.722 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:06:08 smithi189 ceph-mon[36187]: rocksdb: Options.max_successive_merges: 0 2024-03-24T00:06:08.722 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:06:08 smithi189 ceph-mon[36187]: rocksdb: Options.optimize_filters_for_hits: 0 2024-03-24T00:06:08.722 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:06:08 smithi189 ceph-mon[36187]: rocksdb: Options.paranoid_file_checks: 0 2024-03-24T00:06:08.722 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:06:08 smithi189 ceph-mon[36187]: rocksdb: Options.force_consistency_checks: 1 2024-03-24T00:06:08.722 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:06:08 smithi189 ceph-mon[36187]: rocksdb: Options.report_bg_io_stats: 0 2024-03-24T00:06:08.722 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:06:08 smithi189 ceph-mon[36187]: rocksdb: Options.ttl: 2592000 2024-03-24T00:06:08.722 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:06:08 smithi189 ceph-mon[36187]: rocksdb: Options.periodic_compaction_seconds: 0 2024-03-24T00:06:08.722 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:06:08 smithi189 ceph-mon[36187]: rocksdb: Options.preclude_last_level_data_seconds: 0 2024-03-24T00:06:08.722 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:06:08 smithi189 ceph-mon[36187]: rocksdb: Options.preserve_internal_time_seconds: 0 2024-03-24T00:06:08.722 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:06:08 smithi189 ceph-mon[36187]: rocksdb: Options.enable_blob_files: false 2024-03-24T00:06:08.722 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:06:08 smithi189 ceph-mon[36187]: rocksdb: Options.min_blob_size: 0 2024-03-24T00:06:08.722 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:06:08 smithi189 ceph-mon[36187]: rocksdb: Options.blob_file_size: 268435456 2024-03-24T00:06:08.722 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:06:08 smithi189 ceph-mon[36187]: rocksdb: Options.blob_compression_type: NoCompression 2024-03-24T00:06:08.723 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:06:08 smithi189 ceph-mon[36187]: rocksdb: Options.enable_blob_garbage_collection: false 2024-03-24T00:06:08.723 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:06:08 smithi189 ceph-mon[36187]: rocksdb: Options.blob_garbage_collection_age_cutoff: 0.250000 2024-03-24T00:06:08.723 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:06:08 smithi189 ceph-mon[36187]: rocksdb: Options.blob_garbage_collection_force_threshold: 1.000000 2024-03-24T00:06:08.723 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:06:08 smithi189 ceph-mon[36187]: rocksdb: Options.blob_compaction_readahead_size: 0 2024-03-24T00:06:08.723 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:06:08 smithi189 ceph-mon[36187]: rocksdb: Options.blob_file_starting_level: 0 2024-03-24T00:06:08.723 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:06:08 smithi189 ceph-mon[36187]: rocksdb: Options.experimental_mempurge_threshold: 0.000000 2024-03-24T00:06:08.723 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:06:08 smithi189 ceph-mon[36187]: rocksdb: [db/version_set.cc:5566] Recovered from manifest file:/var/lib/ceph/mon/ceph-b/store.db/MANIFEST-000005 succeeded,manifest_file_number is 5, next_file_number is 7, last_sequence is 0, log_number is 0,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 0 2024-03-24T00:06:08.723 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:06:08 smithi189 ceph-mon[36187]: rocksdb: [db/version_set.cc:5581] Column family [default] (ID 0), log number is 0 2024-03-24T00:06:08.723 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:06:08 smithi189 ceph-mon[36187]: rocksdb: [db/db_impl/db_impl_open.cc:539] DB ID: c60a8cb8-01e2-47cb-96b3-221a869e25d4 2024-03-24T00:06:08.723 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:06:08 smithi189 ceph-mon[36187]: rocksdb: EVENT_LOG_v1 {"time_micros": 1711238768415118, "job": 1, "event": "recovery_started", "wal_files": [4]} 2024-03-24T00:06:08.723 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:06:08 smithi189 ceph-mon[36187]: rocksdb: [db/db_impl/db_impl_open.cc:1043] Recovering log #4 mode 2 2024-03-24T00:06:08.723 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:06:08 smithi189 ceph-mon[36187]: rocksdb: EVENT_LOG_v1 {"time_micros": 1711238768415531, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 8, "file_size": 1747, "file_checksum": "", "file_checksum_func_name": "Unknown", "smallest_seqno": 1, "largest_seqno": 5, "table_properties": {"data_size": 648, "index_size": 31, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 1, "index_value_is_delta_encoded": 1, "filter_size": 69, "raw_key_size": 115, "raw_average_key_size": 23, "raw_value_size": 526, "raw_average_value_size": 105, "num_data_blocks": 1, "num_entries": 5, "num_filter_entries": 5, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "bloomfilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; max_dict_buffer_bytes=0; use_zstd_dict_trainer=1; ", "creation_time": 1711238768, "oldest_key_time": 0, "file_creation_time": 0, "slow_compression_estimated_data_size": 0, "fast_compression_estimated_data_size": 0, "db_id": "c60a8cb8-01e2-47cb-96b3-221a869e25d4", "db_session_id": "KTQQ8DS913X6DB7P3LB9", "orig_file_number": 8, "seqno_to_time_mapping": "N/A"}} 2024-03-24T00:06:08.723 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:06:08 smithi189 ceph-mon[36187]: rocksdb: EVENT_LOG_v1 {"time_micros": 1711238768415609, "job": 1, "event": "recovery_finished"} 2024-03-24T00:06:08.723 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:06:08 smithi189 ceph-mon[36187]: rocksdb: [db/version_set.cc:5047] Creating manifest 10 2024-03-24T00:06:08.724 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:06:08 smithi189 ceph-mon[36187]: rocksdb: [file/delete_scheduler.cc:74] Deleted file /var/lib/ceph/mon/ceph-b/store.db/000004.log immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024-03-24T00:06:08.724 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:06:08 smithi189 ceph-mon[36187]: rocksdb: [db/db_impl/db_impl_open.cc:1987] SstFileManager instance 0x5568058f0000 2024-03-24T00:06:08.724 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:06:08 smithi189 ceph-mon[36187]: rocksdb: DB pointer 0x5568058da000 2024-03-24T00:06:08.724 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:06:08 smithi189 ceph-mon[36187]: rocksdb: [db/db_impl/db_impl.cc:1109] ------- DUMPING STATS ------- 2024-03-24T00:06:08.724 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:06:08 smithi189 ceph-mon[36187]: rocksdb: [db/db_impl/db_impl.cc:1111] 2024-03-24T00:06:08.724 INFO:journalctl@ceph.mon.b.smithi189.stdout: ** DB Stats ** 2024-03-24T00:06:08.724 INFO:journalctl@ceph.mon.b.smithi189.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-03-24T00:06:08.724 INFO:journalctl@ceph.mon.b.smithi189.stdout: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2024-03-24T00:06:08.724 INFO:journalctl@ceph.mon.b.smithi189.stdout: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-03-24T00:06:08.724 INFO:journalctl@ceph.mon.b.smithi189.stdout: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2024-03-24T00:06:08.724 INFO:journalctl@ceph.mon.b.smithi189.stdout: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2024-03-24T00:06:08.724 INFO:journalctl@ceph.mon.b.smithi189.stdout: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-03-24T00:06:08.724 INFO:journalctl@ceph.mon.b.smithi189.stdout: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2024-03-24T00:06:08.724 INFO:journalctl@ceph.mon.b.smithi189.stdout: 2024-03-24T00:06:08.724 INFO:journalctl@ceph.mon.b.smithi189.stdout: ** Compaction Stats [default] ** 2024-03-24T00:06:08.725 INFO:journalctl@ceph.mon.b.smithi189.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-03-24T00:06:08.725 INFO:journalctl@ceph.mon.b.smithi189.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ 2024-03-24T00:06:08.725 INFO:journalctl@ceph.mon.b.smithi189.stdout: L0 1/0 1.71 KB 0.2 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 4.2 0.00 0.00 1 0.000 0 0 0.0 0.0 2024-03-24T00:06:08.725 INFO:journalctl@ceph.mon.b.smithi189.stdout: Sum 1/0 1.71 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 4.2 0.00 0.00 1 0.000 0 0 0.0 0.0 2024-03-24T00:06:08.725 INFO:journalctl@ceph.mon.b.smithi189.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 4.2 0.00 0.00 1 0.000 0 0 0.0 0.0 2024-03-24T00:06:08.725 INFO:journalctl@ceph.mon.b.smithi189.stdout: 2024-03-24T00:06:08.725 INFO:journalctl@ceph.mon.b.smithi189.stdout: ** Compaction Stats [default] ** 2024-03-24T00:06:08.725 INFO:journalctl@ceph.mon.b.smithi189.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop Rblob(GB) Wblob(GB) 2024-03-24T00:06:08.725 INFO:journalctl@ceph.mon.b.smithi189.stdout: --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-03-24T00:06:08.725 INFO:journalctl@ceph.mon.b.smithi189.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 4.2 0.00 0.00 1 0.000 0 0 0.0 0.0 2024-03-24T00:06:08.725 INFO:journalctl@ceph.mon.b.smithi189.stdout: 2024-03-24T00:06:08.725 INFO:journalctl@ceph.mon.b.smithi189.stdout: Blob file count: 0, total size: 0.0 GB, garbage size: 0.0 GB, space amp: 0.0 2024-03-24T00:06:08.725 INFO:journalctl@ceph.mon.b.smithi189.stdout: 2024-03-24T00:06:08.725 INFO:journalctl@ceph.mon.b.smithi189.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-03-24T00:06:08.726 INFO:journalctl@ceph.mon.b.smithi189.stdout: Flush(GB): cumulative 0.000, interval 0.000 2024-03-24T00:06:08.726 INFO:journalctl@ceph.mon.b.smithi189.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2024-03-24T00:06:08.726 INFO:journalctl@ceph.mon.b.smithi189.stdout: AddFile(Total Files): cumulative 0, interval 0 2024-03-24T00:06:08.726 INFO:journalctl@ceph.mon.b.smithi189.stdout: AddFile(L0 Files): cumulative 0, interval 0 2024-03-24T00:06:08.726 INFO:journalctl@ceph.mon.b.smithi189.stdout: AddFile(Keys): cumulative 0, interval 0 2024-03-24T00:06:08.726 INFO:journalctl@ceph.mon.b.smithi189.stdout: Cumulative compaction: 0.00 GB write, 0.70 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-03-24T00:06:08.726 INFO:journalctl@ceph.mon.b.smithi189.stdout: Interval compaction: 0.00 GB write, 0.70 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-03-24T00:06:08.726 INFO:journalctl@ceph.mon.b.smithi189.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-03-24T00:06:08.726 INFO:journalctl@ceph.mon.b.smithi189.stdout: Block cache BinnedLRUCache@0x5568057fd090#2 capacity: 512.00 MB usage: 0.22 KB table_size: 0 occupancy: 18446744073709551615 collections: 1 last_copies: 0 last_secs: 1e-05 secs_since: 0 2024-03-24T00:06:08.726 INFO:journalctl@ceph.mon.b.smithi189.stdout: Block cache entry stats(count,size,portion): FilterBlock(1,0.11 KB,2.08616e-05%) IndexBlock(1,0.11 KB,2.08616e-05%) Misc(1,0.00 KB,0%) 2024-03-24T00:06:08.726 INFO:journalctl@ceph.mon.b.smithi189.stdout: 2024-03-24T00:06:08.726 INFO:journalctl@ceph.mon.b.smithi189.stdout: ** File Read Latency Histogram By Level [default] ** 2024-03-24T00:06:08.726 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:06:08 smithi189 ceph-mon[36187]: mon.b does not exist in monmap, will attempt to join an existing cluster 2024-03-24T00:06:08.726 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:06:08 smithi189 ceph-mon[36187]: using public_addr v2:172.21.15.189:0/0 -> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] 2024-03-24T00:06:08.727 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:06:08 smithi189 ceph-mon[36187]: starting mon.b rank -1 at public addrs [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] at bind addrs [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] mon_data /var/lib/ceph/mon/ceph-b fsid d79a5420-e971-11ee-95cd-87774f69a715 2024-03-24T00:06:08.727 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:06:08 smithi189 ceph-mon[36187]: mon.b@-1(???) e0 preinit fsid d79a5420-e971-11ee-95cd-87774f69a715 2024-03-24T00:06:08.727 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:06:08 smithi189 ceph-mon[36187]: mon.b@-1(synchronizing).mds e1 new map 2024-03-24T00:06:08.727 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:06:08 smithi189 ceph-mon[36187]: mon.b@-1(synchronizing).mds e1 print_map 2024-03-24T00:06:08.727 INFO:journalctl@ceph.mon.b.smithi189.stdout: e1 2024-03-24T00:06:08.727 INFO:journalctl@ceph.mon.b.smithi189.stdout: enable_multiple, ever_enabled_multiple: 1,1 2024-03-24T00:06:08.727 INFO:journalctl@ceph.mon.b.smithi189.stdout: default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2,11=minor log segments} 2024-03-24T00:06:08.727 INFO:journalctl@ceph.mon.b.smithi189.stdout: legacy client fscid: -1 2024-03-24T00:06:08.727 INFO:journalctl@ceph.mon.b.smithi189.stdout: 2024-03-24T00:06:08.727 INFO:journalctl@ceph.mon.b.smithi189.stdout: No filesystems configured 2024-03-24T00:06:08.727 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:06:08 smithi189 ceph-mon[36187]: mon.b@-1(synchronizing).osd e0 _set_cache_ratios kv ratio 0.25 inc ratio 0.375 full ratio 0.375 2024-03-24T00:06:08.727 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:06:08 smithi189 ceph-mon[36187]: mon.b@-1(synchronizing).osd e0 register_cache_with_pcm pcm target: 2147483648 pcm max: 1020054732 pcm min: 134217728 inc_osd_cache size: 1 2024-03-24T00:06:08.727 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:06:08 smithi189 ceph-mon[36187]: mon.b@-1(synchronizing).osd e1 e1: 0 total, 0 up, 0 in 2024-03-24T00:06:08.727 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:06:08 smithi189 ceph-mon[36187]: mon.b@-1(synchronizing).osd e2 e2: 0 total, 0 up, 0 in 2024-03-24T00:06:08.727 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:06:08 smithi189 ceph-mon[36187]: mon.b@-1(synchronizing).osd e3 e3: 0 total, 0 up, 0 in 2024-03-24T00:06:08.728 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:06:08 smithi189 ceph-mon[36187]: mon.b@-1(synchronizing).osd e4 e4: 0 total, 0 up, 0 in 2024-03-24T00:06:08.728 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:06:08 smithi189 ceph-mon[36187]: mon.b@-1(synchronizing).osd e4 crush map has features 3314932999778484224, adjusting msgr requires 2024-03-24T00:06:08.728 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:06:08 smithi189 ceph-mon[36187]: mon.b@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2024-03-24T00:06:08.728 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:06:08 smithi189 ceph-mon[36187]: mon.b@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2024-03-24T00:06:08.728 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:06:08 smithi189 ceph-mon[36187]: mon.b@-1(synchronizing).osd e4 crush map has features 288514050185494528, adjusting msgr requires 2024-03-24T00:06:08.728 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:06:08 smithi189 ceph-mon[36187]: pgmap v6: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T00:06:08.728 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:06:08 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T00:06:08.728 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:06:08 smithi189 ceph-mon[36187]: Added host smithi189 2024-03-24T00:06:08.728 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:06:08 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T00:06:08.728 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:06:08 smithi189 ceph-mon[36187]: pgmap v7: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T00:06:08.728 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:06:08 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T00:06:08.728 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:06:08 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T00:06:08.728 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:06:08 smithi189 ceph-mon[36187]: pgmap v8: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T00:06:08.728 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:06:08 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T00:06:08.729 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:06:08 smithi189 ceph-mon[36187]: from='client.14180 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-24T00:06:08.729 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:06:08 smithi189 ceph-mon[36187]: pgmap v9: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T00:06:08.729 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:06:08 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T00:06:08.729 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:06:08 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T00:06:08.729 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:06:08 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T00:06:08.729 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:06:08 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T00:06:08.729 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:06:08 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd/host:smithi189", "name": "osd_memory_target"}]: dispatch 2024-03-24T00:06:08.729 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:06:08 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T00:06:08.729 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:06:08 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T00:06:08.729 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:06:08 smithi189 ceph-mon[36187]: Updating smithi189:/etc/ceph/ceph.conf 2024-03-24T00:06:08.729 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:06:08 smithi189 ceph-mon[36187]: Updating smithi189:/var/lib/ceph/d79a5420-e971-11ee-95cd-87774f69a715/config/ceph.conf 2024-03-24T00:06:08.729 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:06:08 smithi189 ceph-mon[36187]: pgmap v10: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T00:06:08.729 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:06:08 smithi189 ceph-mon[36187]: Updating smithi189:/etc/ceph/ceph.client.admin.keyring 2024-03-24T00:06:08.729 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:06:08 smithi189 ceph-mon[36187]: Updating smithi189:/var/lib/ceph/d79a5420-e971-11ee-95cd-87774f69a715/config/ceph.client.admin.keyring 2024-03-24T00:06:08.730 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:06:08 smithi189 ceph-mon[36187]: from='client.14182 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi193", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T00:06:08.730 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:06:08 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T00:06:08.730 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:06:08 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T00:06:08.730 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:06:08 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T00:06:08.730 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:06:08 smithi189 ceph-mon[36187]: Deploying cephadm binary to smithi193 2024-03-24T00:06:08.730 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:06:08 smithi189 ceph-mon[36187]: pgmap v11: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T00:06:08.730 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:06:08 smithi189 ceph-mon[36187]: pgmap v12: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T00:06:08.730 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:06:08 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T00:06:08.730 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:06:08 smithi189 ceph-mon[36187]: Added host smithi193 2024-03-24T00:06:08.730 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:06:08 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T00:06:08.730 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:06:08 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T00:06:08.730 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:06:08 smithi189 ceph-mon[36187]: pgmap v13: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T00:06:08.730 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:06:08 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T00:06:08.731 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:06:08 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T00:06:08.731 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:06:08 smithi189 ceph-mon[36187]: pgmap v14: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T00:06:08.731 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:06:08 smithi189 ceph-mon[36187]: from='client.14184 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-24T00:06:08.731 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:06:08 smithi189 ceph-mon[36187]: pgmap v15: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T00:06:08.731 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:06:08 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T00:06:08.731 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:06:08 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T00:06:08.731 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:06:08 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T00:06:08.731 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:06:08 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T00:06:08.731 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:06:08 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd/host:smithi193", "name": "osd_memory_target"}]: dispatch 2024-03-24T00:06:08.731 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:06:08 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T00:06:08.731 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:06:08 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T00:06:08.731 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:06:08 smithi189 ceph-mon[36187]: Updating smithi193:/etc/ceph/ceph.conf 2024-03-24T00:06:08.731 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:06:08 smithi189 ceph-mon[36187]: from='client.? 172.21.15.118:0/2863852192' entity='client.admin' cmd=[{"prefix": "osd crush tunables", "profile": "default"}]: dispatch 2024-03-24T00:06:08.731 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:06:08 smithi189 ceph-mon[36187]: Updating smithi193:/var/lib/ceph/d79a5420-e971-11ee-95cd-87774f69a715/config/ceph.conf 2024-03-24T00:06:08.732 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:06:08 smithi189 ceph-mon[36187]: Updating smithi193:/etc/ceph/ceph.client.admin.keyring 2024-03-24T00:06:08.732 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:06:08 smithi189 ceph-mon[36187]: Updating smithi193:/var/lib/ceph/d79a5420-e971-11ee-95cd-87774f69a715/config/ceph.client.admin.keyring 2024-03-24T00:06:08.732 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:06:08 smithi189 ceph-mon[36187]: pgmap v16: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T00:06:08.732 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:06:08 smithi189 ceph-mon[36187]: from='client.? 172.21.15.118:0/2863852192' entity='client.admin' cmd='[{"prefix": "osd crush tunables", "profile": "default"}]': finished 2024-03-24T00:06:08.732 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:06:08 smithi189 ceph-mon[36187]: osdmap e4: 0 total, 0 up, 0 in 2024-03-24T00:06:08.732 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:06:08 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T00:06:08.732 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:06:08 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T00:06:08.732 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:06:08 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T00:06:08.732 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:06:08 smithi189 ceph-mon[36187]: pgmap v18: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T00:06:08.732 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:06:08 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T00:06:08.732 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:06:08 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T00:06:08.732 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:06:08 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T00:06:08.732 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:06:08 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T00:06:08.732 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:06:08 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T00:06:08.732 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:06:08 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-24T00:06:08.733 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:06:08 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T00:06:08.733 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:06:08 smithi189 ceph-mon[36187]: from='client.14188 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "placement": "3;smithi118:172.21.15.118=a;smithi189:172.21.15.189=b;smithi193:172.21.15.193=c", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T00:06:08.733 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:06:08 smithi189 ceph-mon[36187]: Saving service mon spec with placement smithi118:172.21.15.118=a;smithi189:172.21.15.189=b;smithi193:172.21.15.193=c;count:3 2024-03-24T00:06:08.733 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:06:08 smithi189 ceph-mon[36187]: Deploying daemon mon.c on smithi193 2024-03-24T00:06:08.733 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:06:08 smithi189 ceph-mon[36187]: pgmap v19: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T00:06:08.733 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:06:08 smithi189 ceph-mon[36187]: pgmap v20: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T00:06:08.733 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:06:08 smithi189 ceph-mon[36187]: from='client.? 172.21.15.193:0/1148064473' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-24T00:06:08.733 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:06:08 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-24T00:06:08.733 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:06:08 smithi189 ceph-mon[36187]: mon.a calling monitor election 2024-03-24T00:06:08.733 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:06:08 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-24T00:06:08.733 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:06:08 smithi189 ceph-mon[36187]: pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T00:06:08.733 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:06:08 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-24T00:06:08.733 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:06:08 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-24T00:06:08.733 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:06:08 smithi189 ceph-mon[36187]: mon.c calling monitor election 2024-03-24T00:06:08.734 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:06:08 smithi189 ceph-mon[36187]: pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T00:06:08.734 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:06:08 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-24T00:06:08.734 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:06:08 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-24T00:06:08.734 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:06:08 smithi189 ceph-mon[36187]: pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T00:06:08.734 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:06:08 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-24T00:06:08.734 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:06:08 smithi189 ceph-mon[36187]: mon.a is new leader, mons a,c in quorum (ranks 0,1) 2024-03-24T00:06:08.734 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:06:08 smithi189 ceph-mon[36187]: monmap e2: 2 mons at {a=[v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0],c=[v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0]} removed_ranks: {} disallowed_leaders: {} 2024-03-24T00:06:08.734 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:06:08 smithi189 ceph-mon[36187]: fsmap 2024-03-24T00:06:08.734 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:06:08 smithi189 ceph-mon[36187]: osdmap e4: 0 total, 0 up, 0 in 2024-03-24T00:06:08.734 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:06:08 smithi189 ceph-mon[36187]: mgrmap e13: x(active, since 59s) 2024-03-24T00:06:08.734 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:06:08 smithi189 ceph-mon[36187]: overall HEALTH_OK 2024-03-24T00:06:08.734 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:06:08 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T00:06:08.734 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:06:08 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T00:06:08.734 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:06:08 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T00:06:08.735 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:06:08 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-24T00:06:08.735 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:06:08 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T00:06:08.735 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:06:08 smithi189 ceph-mon[36187]: Deploying daemon mon.b on smithi189 2024-03-24T00:06:08.735 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:06:08 smithi189 ceph-mon[36187]: from='client.? 172.21.15.193:0/2575367044' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-24T00:06:08.735 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:06:08 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-24T00:06:08.735 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:06:08 smithi189 ceph-mon[36187]: pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T00:06:08.735 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:06:08 smithi189 ceph-mon[36187]: mon.b@-1(synchronizing).paxosservice(auth 1..3) refresh upgraded, format 0 -> 3 2024-03-24T00:06:08.735 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:06:08 smithi189 bash[36168]: 332ea823179e88f4b6faf6793d78c9576c10138bb235aa5cd94334b2af6c7389 2024-03-24T00:06:08.735 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:06:08 smithi189 systemd[1]: Started Ceph mon.b for d79a5420-e971-11ee-95cd-87774f69a715. 2024-03-24T00:06:09.871 INFO:tasks.cephadm:Waiting for 3 mons in monmap... 2024-03-24T00:06:09.872 DEBUG:teuthology.orchestra.run.smithi193:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d79a5420-e971-11ee-95cd-87774f69a715 -- ceph mon dump -f json 2024-03-24T00:06:10.113 INFO:teuthology.orchestra.run.smithi193.stderr:Inferring config /var/lib/ceph/d79a5420-e971-11ee-95cd-87774f69a715/mon.c/config 2024-03-24T00:06:13.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:06:13 smithi118 ceph-mon[31535]: mon.c calling monitor election 2024-03-24T00:06:13.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:06:13 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-24T00:06:13.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:06:13 smithi118 ceph-mon[31535]: mon.a calling monitor election 2024-03-24T00:06:13.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:06:13 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-24T00:06:13.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:06:13 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-24T00:06:13.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:06:13 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-24T00:06:13.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:06:13 smithi118 ceph-mon[31535]: pgmap v26: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T00:06:13.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:06:13 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-24T00:06:13.947 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:06:13 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-24T00:06:13.947 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:06:13 smithi118 ceph-mon[31535]: pgmap v27: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T00:06:13.947 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:06:13 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-24T00:06:13.947 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:06:13 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-24T00:06:13.947 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:06:13 smithi118 ceph-mon[31535]: mon.a is new leader, mons a,c in quorum (ranks 0,1) 2024-03-24T00:06:13.947 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:06:13 smithi118 ceph-mon[31535]: monmap e3: 3 mons at {a=[v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0],b=[v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0],c=[v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0]} removed_ranks: {} disallowed_leaders: {} 2024-03-24T00:06:13.947 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:06:13 smithi118 ceph-mon[31535]: fsmap 2024-03-24T00:06:13.947 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:06:13 smithi118 ceph-mon[31535]: osdmap e4: 0 total, 0 up, 0 in 2024-03-24T00:06:13.947 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:06:13 smithi118 ceph-mon[31535]: mgrmap e13: x(active, since 67s) 2024-03-24T00:06:13.947 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:06:13 smithi118 ceph-mon[31535]: Health check failed: 1/3 mons down, quorum a,c (MON_DOWN) 2024-03-24T00:06:13.947 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:06:13 smithi118 ceph-mon[31535]: Health detail: HEALTH_WARN 1/3 mons down, quorum a,c 2024-03-24T00:06:13.947 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:06:13 smithi118 ceph-mon[31535]: [WRN] MON_DOWN: 1/3 mons down, quorum a,c 2024-03-24T00:06:13.948 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:06:13 smithi118 ceph-mon[31535]: mon.b (rank 2) addr [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] is down (out of quorum) 2024-03-24T00:06:13.948 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:06:13 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T00:06:13.948 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:06:13 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T00:06:13.948 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:06:13 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T00:06:13.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:06:13 smithi193 ceph-mon[36878]: mon.c calling monitor election 2024-03-24T00:06:13.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:06:13 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-24T00:06:13.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:06:13 smithi193 ceph-mon[36878]: mon.a calling monitor election 2024-03-24T00:06:13.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:06:13 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-24T00:06:13.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:06:13 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-24T00:06:13.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:06:13 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-24T00:06:13.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:06:13 smithi193 ceph-mon[36878]: pgmap v26: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T00:06:13.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:06:13 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-24T00:06:13.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:06:13 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-24T00:06:13.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:06:13 smithi193 ceph-mon[36878]: pgmap v27: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T00:06:13.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:06:13 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-24T00:06:13.962 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:06:13 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-24T00:06:13.962 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:06:13 smithi193 ceph-mon[36878]: mon.a is new leader, mons a,c in quorum (ranks 0,1) 2024-03-24T00:06:13.962 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:06:13 smithi193 ceph-mon[36878]: monmap e3: 3 mons at {a=[v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0],b=[v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0],c=[v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0]} removed_ranks: {} disallowed_leaders: {} 2024-03-24T00:06:13.962 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:06:13 smithi193 ceph-mon[36878]: fsmap 2024-03-24T00:06:13.962 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:06:13 smithi193 ceph-mon[36878]: osdmap e4: 0 total, 0 up, 0 in 2024-03-24T00:06:13.962 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:06:13 smithi193 ceph-mon[36878]: mgrmap e13: x(active, since 67s) 2024-03-24T00:06:13.962 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:06:13 smithi193 ceph-mon[36878]: Health check failed: 1/3 mons down, quorum a,c (MON_DOWN) 2024-03-24T00:06:13.962 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:06:13 smithi193 ceph-mon[36878]: Health detail: HEALTH_WARN 1/3 mons down, quorum a,c 2024-03-24T00:06:13.962 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:06:13 smithi193 ceph-mon[36878]: [WRN] MON_DOWN: 1/3 mons down, quorum a,c 2024-03-24T00:06:13.962 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:06:13 smithi193 ceph-mon[36878]: mon.b (rank 2) addr [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] is down (out of quorum) 2024-03-24T00:06:13.962 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:06:13 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T00:06:13.962 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:06:13 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T00:06:13.962 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:06:13 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T00:06:14.444 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-24T00:06:14.444+0000 7fdd35ffb700 1 -- 172.21.15.193:0/2464567325 <== mon.0 v2:172.21.15.118:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdd2c0116d0 con 0x7fdd200035c0 2024-03-24T00:06:14.444 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-24T00:06:14.444+0000 7fdd3d960700 1 -- 172.21.15.193:0/2464567325 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fdd200035c0 msgr2=0x7fdd20005a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:06:14.445 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-24T00:06:14.444+0000 7fdd3d960700 1 --2- 172.21.15.193:0/2464567325 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fdd200035c0 0x7fdd20005a50 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7fdd38101bb0 tx=0x7fdd2c00feb0 comp rx=0 tx=0).stop 2024-03-24T00:06:14.445 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-24T00:06:14.445+0000 7fdd3d960700 1 -- 172.21.15.193:0/2464567325 shutdown_connections 2024-03-24T00:06:14.445 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-24T00:06:14.445+0000 7fdd3d960700 1 --2- 172.21.15.193:0/2464567325 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fdd200035c0 0x7fdd20005a50 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:06:14.445 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-24T00:06:14.445+0000 7fdd3d960700 1 -- 172.21.15.193:0/2464567325 >> 172.21.15.193:0/2464567325 conn(0x7fdd380fd750 msgr2=0x7fdd380ffb70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:06:14.445 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-24T00:06:14.445+0000 7fdd3d960700 1 -- 172.21.15.193:0/2464567325 shutdown_connections 2024-03-24T00:06:14.445 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-24T00:06:14.445+0000 7fdd3d960700 1 -- 172.21.15.193:0/2464567325 wait complete. 2024-03-24T00:06:14.447 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-24T00:06:14.445+0000 7fdd3d960700 1 Processor -- start 2024-03-24T00:06:14.447 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-24T00:06:14.445+0000 7fdd3d960700 1 -- start start 2024-03-24T00:06:14.447 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-24T00:06:14.446+0000 7fdd3d960700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fdd200035c0 0x7fdd38074490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:06:14.447 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-24T00:06:14.446+0000 7fdd3d960700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fdd380749d0 0x7fdd38074e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:06:14.447 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-24T00:06:14.446+0000 7fdd3d960700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fdd381a3b00 0x7fdd381a5ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:06:14.447 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-24T00:06:14.446+0000 7fdd3d960700 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdd38109cc0 con 0x7fdd380749d0 2024-03-24T00:06:14.447 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-24T00:06:14.446+0000 7fdd3d960700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdd38109b40 con 0x7fdd381a3b00 2024-03-24T00:06:14.447 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-24T00:06:14.446+0000 7fdd3d960700 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdd38109e40 con 0x7fdd200035c0 2024-03-24T00:06:14.447 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-24T00:06:14.446+0000 7fdd36ffd700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fdd200035c0 0x7fdd38074490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:06:14.447 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-24T00:06:14.446+0000 7fdd36ffd700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fdd200035c0 0x7fdd38074490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.193:53690/0 (socket says 172.21.15.193:53690) 2024-03-24T00:06:14.448 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-24T00:06:14.446+0000 7fdd36ffd700 1 -- 172.21.15.193:0/1357798125 learned_addr learned my addr 172.21.15.193:0/1357798125 (peer_addr_for_me v2:172.21.15.193:0/0) 2024-03-24T00:06:14.448 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-24T00:06:14.446+0000 7fdd367fc700 1 --2- 172.21.15.193:0/1357798125 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fdd380749d0 0x7fdd38074e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:06:14.448 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-24T00:06:14.446+0000 7fdd377fe700 1 --2- 172.21.15.193:0/1357798125 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fdd381a3b00 0x7fdd381a5ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:06:14.448 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-24T00:06:14.446+0000 7fdd36ffd700 1 -- 172.21.15.193:0/1357798125 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fdd381a3b00 msgr2=0x7fdd381a5ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:06:14.448 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-24T00:06:14.446+0000 7fdd36ffd700 1 --2- 172.21.15.193:0/1357798125 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fdd381a3b00 0x7fdd381a5ef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:06:14.448 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-24T00:06:14.446+0000 7fdd36ffd700 1 -- 172.21.15.193:0/1357798125 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fdd380749d0 msgr2=0x7fdd38074e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:06:14.448 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-24T00:06:14.446+0000 7fdd36ffd700 1 --2- 172.21.15.193:0/1357798125 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fdd380749d0 0x7fdd38074e50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:06:14.448 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-24T00:06:14.446+0000 7fdd36ffd700 1 -- 172.21.15.193:0/1357798125 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdd381a65c0 con 0x7fdd200035c0 2024-03-24T00:06:14.448 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-24T00:06:14.446+0000 7fdd36ffd700 1 --2- 172.21.15.193:0/1357798125 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fdd200035c0 0x7fdd38074490 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7fdd2c001eb0 tx=0x7fdd2c005810 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:06:14.448 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-24T00:06:14.446+0000 7fdd3c95e700 1 -- 172.21.15.193:0/1357798125 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdd2c028020 con 0x7fdd200035c0 2024-03-24T00:06:14.448 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-24T00:06:14.447+0000 7fdd3d960700 1 -- 172.21.15.193:0/1357798125 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdd381a6850 con 0x7fdd200035c0 2024-03-24T00:06:14.448 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-24T00:06:14.447+0000 7fdd3c95e700 1 -- 172.21.15.193:0/1357798125 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fdd2c005300 con 0x7fdd200035c0 2024-03-24T00:06:14.448 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-24T00:06:14.447+0000 7fdd3c95e700 1 -- 172.21.15.193:0/1357798125 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdd2c024a30 con 0x7fdd200035c0 2024-03-24T00:06:14.448 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-24T00:06:14.447+0000 7fdd3d960700 1 -- 172.21.15.193:0/1357798125 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdd381a6de0 con 0x7fdd200035c0 2024-03-24T00:06:14.449 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-24T00:06:14.447+0000 7fdd3c95e700 1 -- 172.21.15.193:0/1357798125 <== mon.1 v2:172.21.15.193:3300/0 4 ==== mgrmap(e 13) v1 ==== 46163+0+0 (secure 0 0 0) 0x7fdd2c02b020 con 0x7fdd200035c0 2024-03-24T00:06:14.449 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-24T00:06:14.447+0000 7fdd3c95e700 1 --2- 172.21.15.193:0/1357798125 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7fdd24039450 0x7fdd2403b910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:06:14.449 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-24T00:06:14.447+0000 7fdd3c95e700 1 -- 172.21.15.193:0/1357798125 <== mon.1 v2:172.21.15.193:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1155+0+0 (secure 0 0 0) 0x7fdd2c04f260 con 0x7fdd200035c0 2024-03-24T00:06:14.450 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-24T00:06:14.448+0000 7fdd3d960700 1 -- 172.21.15.193:0/1357798125 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdd380637c0 con 0x7fdd200035c0 2024-03-24T00:06:14.450 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-24T00:06:14.448+0000 7fdd367fc700 1 --2- 172.21.15.193:0/1357798125 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7fdd24039450 0x7fdd2403b910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:06:14.450 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-24T00:06:14.449+0000 7fdd367fc700 1 --2- 172.21.15.193:0/1357798125 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7fdd24039450 0x7fdd2403b910 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7fdd280078e0 tx=0x7fdd28008040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:06:14.453 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-24T00:06:14.453+0000 7fdd3c95e700 1 -- 172.21.15.193:0/1357798125 <== mon.1 v2:172.21.15.193:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7fdd2c032070 con 0x7fdd200035c0 2024-03-24T00:06:14.710 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:06:14 smithi193 ceph-mon[36878]: pgmap v28: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T00:06:14.710 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:06:14 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T00:06:14.711 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:06:14 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T00:06:14.711 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:06:14 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-24T00:06:14.737 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-24T00:06:14.737+0000 7fdd3d960700 1 -- 172.21.15.193:0/1357798125 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fdd381a7040 con 0x7fdd200035c0 2024-03-24T00:06:14.738 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-24T00:06:14.738+0000 7fdd3c95e700 1 -- 172.21.15.193:0/1357798125 <== mon.1 v2:172.21.15.193:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 3 v3) v1 ==== 95+0+1287 (secure 0 0 0) 0x7fdd2c010c90 con 0x7fdd200035c0 2024-03-24T00:06:14.738 INFO:teuthology.orchestra.run.smithi193.stdout: 2024-03-24T00:06:14.738 INFO:teuthology.orchestra.run.smithi193.stdout:{"epoch":3,"fsid":"d79a5420-e971-11ee-95cd-87774f69a715","modified":"2024-03-24T00:06:08.437943Z","created":"2024-03-24T00:03:54.425344Z","min_mon_release":19,"min_mon_release_name":"squid","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy","reef","squid"],"optional":[]},"mons":[{"rank":0,"name":"a","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:3300","nonce":0},{"type":"v1","addr":"172.21.15.118:6789","nonce":0}]},"addr":"172.21.15.118:6789/0","public_addr":"172.21.15.118:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"c","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:3300","nonce":0},{"type":"v1","addr":"172.21.15.193:6789","nonce":0}]},"addr":"172.21.15.193:6789/0","public_addr":"172.21.15.193:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":2,"name":"b","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:3300","nonce":0},{"type":"v1","addr":"172.21.15.189:6789","nonce":0}]},"addr":"172.21.15.189:6789/0","public_addr":"172.21.15.189:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0,1]} 2024-03-24T00:06:14.740 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-24T00:06:14.740+0000 7fdd3d960700 1 -- 172.21.15.193:0/1357798125 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7fdd24039450 msgr2=0x7fdd2403b910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:06:14.740 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-24T00:06:14.740+0000 7fdd3d960700 1 --2- 172.21.15.193:0/1357798125 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7fdd24039450 0x7fdd2403b910 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7fdd280078e0 tx=0x7fdd28008040 comp rx=0 tx=0).stop 2024-03-24T00:06:14.741 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-24T00:06:14.740+0000 7fdd3d960700 1 -- 172.21.15.193:0/1357798125 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fdd200035c0 msgr2=0x7fdd38074490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:06:14.741 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-24T00:06:14.740+0000 7fdd3d960700 1 --2- 172.21.15.193:0/1357798125 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fdd200035c0 0x7fdd38074490 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7fdd2c001eb0 tx=0x7fdd2c005810 comp rx=0 tx=0).stop 2024-03-24T00:06:14.741 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-24T00:06:14.740+0000 7fdd3d960700 1 -- 172.21.15.193:0/1357798125 shutdown_connections 2024-03-24T00:06:14.741 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-24T00:06:14.740+0000 7fdd3d960700 1 --2- 172.21.15.193:0/1357798125 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fdd381a3b00 0x7fdd381a5ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:06:14.741 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-24T00:06:14.740+0000 7fdd3d960700 1 --2- 172.21.15.193:0/1357798125 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7fdd24039450 0x7fdd2403b910 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:06:14.741 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-24T00:06:14.740+0000 7fdd3d960700 1 --2- 172.21.15.193:0/1357798125 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fdd200035c0 0x7fdd38074490 unknown :-1 s=CLOSED pgs=5 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:06:14.741 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-24T00:06:14.740+0000 7fdd3d960700 1 --2- 172.21.15.193:0/1357798125 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fdd380749d0 0x7fdd38074e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:06:14.741 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-24T00:06:14.740+0000 7fdd3d960700 1 -- 172.21.15.193:0/1357798125 >> 172.21.15.193:0/1357798125 conn(0x7fdd380fd750 msgr2=0x7fdd381058c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:06:14.741 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-24T00:06:14.741+0000 7fdd3d960700 1 -- 172.21.15.193:0/1357798125 shutdown_connections 2024-03-24T00:06:14.741 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-24T00:06:14.741+0000 7fdd3d960700 1 -- 172.21.15.193:0/1357798125 wait complete. 2024-03-24T00:06:14.742 INFO:teuthology.orchestra.run.smithi193.stderr:dumped monmap epoch 3 2024-03-24T00:06:14.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:06:14 smithi118 ceph-mon[31535]: pgmap v28: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T00:06:14.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:06:14 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T00:06:14.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:06:14 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T00:06:14.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:06:14 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-24T00:06:15.625 INFO:tasks.cephadm:Generating final ceph.conf file... 2024-03-24T00:06:15.626 DEBUG:teuthology.orchestra.run.smithi118:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d79a5420-e971-11ee-95cd-87774f69a715 -- ceph config generate-minimal-conf 2024-03-24T00:06:15.704 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:06:15 smithi189 ceph-mon[36187]: mon.c calling monitor election 2024-03-24T00:06:15.704 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:06:15 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-24T00:06:15.704 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:06:15 smithi189 ceph-mon[36187]: mon.a calling monitor election 2024-03-24T00:06:15.704 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:06:15 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-24T00:06:15.704 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:06:15 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-24T00:06:15.705 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:06:15 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-24T00:06:15.705 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:06:15 smithi189 ceph-mon[36187]: pgmap v26: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T00:06:15.705 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:06:15 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-24T00:06:15.705 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:06:15 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-24T00:06:15.705 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:06:15 smithi189 ceph-mon[36187]: pgmap v27: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T00:06:15.705 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:06:15 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-24T00:06:15.705 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:06:15 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-24T00:06:15.705 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:06:15 smithi189 ceph-mon[36187]: mon.a is new leader, mons a,c in quorum (ranks 0,1) 2024-03-24T00:06:15.705 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:06:15 smithi189 ceph-mon[36187]: monmap e3: 3 mons at {a=[v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0],b=[v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0],c=[v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0]} removed_ranks: {} disallowed_leaders: {} 2024-03-24T00:06:15.705 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:06:15 smithi189 ceph-mon[36187]: fsmap 2024-03-24T00:06:15.705 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:06:15 smithi189 ceph-mon[36187]: osdmap e4: 0 total, 0 up, 0 in 2024-03-24T00:06:15.706 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:06:15 smithi189 ceph-mon[36187]: mgrmap e13: x(active, since 67s) 2024-03-24T00:06:15.706 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:06:15 smithi189 ceph-mon[36187]: Health check failed: 1/3 mons down, quorum a,c (MON_DOWN) 2024-03-24T00:06:15.706 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:06:15 smithi189 ceph-mon[36187]: Health detail: HEALTH_WARN 1/3 mons down, quorum a,c 2024-03-24T00:06:15.706 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:06:15 smithi189 ceph-mon[36187]: [WRN] MON_DOWN: 1/3 mons down, quorum a,c 2024-03-24T00:06:15.706 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:06:15 smithi189 ceph-mon[36187]: mon.b (rank 2) addr [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] is down (out of quorum) 2024-03-24T00:06:15.706 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:06:15 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T00:06:15.706 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:06:15 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T00:06:15.707 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:06:15 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T00:06:15.707 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:06:15 smithi189 ceph-mon[36187]: pgmap v28: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T00:06:15.707 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:06:15 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T00:06:15.707 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:06:15 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T00:06:15.707 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:06:15 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-24T00:06:16.047 INFO:teuthology.orchestra.run.smithi118.stderr:Inferring config /var/lib/ceph/d79a5420-e971-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T00:06:16.346 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:06:16 smithi118 ceph-mon[31535]: mon.b calling monitor election 2024-03-24T00:06:16.346 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:06:16 smithi118 ceph-mon[31535]: Updating smithi118:/etc/ceph/ceph.conf 2024-03-24T00:06:16.347 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:06:16 smithi118 ceph-mon[31535]: Updating smithi189:/etc/ceph/ceph.conf 2024-03-24T00:06:16.347 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:06:16 smithi118 ceph-mon[31535]: Updating smithi193:/etc/ceph/ceph.conf 2024-03-24T00:06:16.347 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:06:16 smithi118 ceph-mon[31535]: Updating smithi193:/var/lib/ceph/d79a5420-e971-11ee-95cd-87774f69a715/config/ceph.conf 2024-03-24T00:06:16.347 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:06:16 smithi118 ceph-mon[31535]: Updating smithi189:/var/lib/ceph/d79a5420-e971-11ee-95cd-87774f69a715/config/ceph.conf 2024-03-24T00:06:16.347 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:06:16 smithi118 ceph-mon[31535]: mon.b calling monitor election 2024-03-24T00:06:16.347 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:06:16 smithi118 ceph-mon[31535]: mon.c calling monitor election 2024-03-24T00:06:16.347 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:06:16 smithi118 ceph-mon[31535]: mon.a calling monitor election 2024-03-24T00:06:16.347 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:06:16 smithi118 ceph-mon[31535]: mon.a is new leader, mons a,c,b in quorum (ranks 0,1,2) 2024-03-24T00:06:16.347 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:06:16 smithi118 ceph-mon[31535]: monmap e3: 3 mons at {a=[v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0],b=[v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0],c=[v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0]} removed_ranks: {} disallowed_leaders: {} 2024-03-24T00:06:16.347 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:06:16 smithi118 ceph-mon[31535]: fsmap 2024-03-24T00:06:16.347 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:06:16 smithi118 ceph-mon[31535]: osdmap e4: 0 total, 0 up, 0 in 2024-03-24T00:06:16.347 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:06:16 smithi118 ceph-mon[31535]: mgrmap e13: x(active, since 70s) 2024-03-24T00:06:16.347 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:06:16 smithi118 ceph-mon[31535]: Health check cleared: MON_DOWN (was: 1/3 mons down, quorum a,c) 2024-03-24T00:06:16.347 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:06:16 smithi118 ceph-mon[31535]: Cluster is now healthy 2024-03-24T00:06:16.348 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:06:16 smithi118 ceph-mon[31535]: pgmap v29: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T00:06:16.348 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:06:16 smithi118 ceph-mon[31535]: overall HEALTH_OK 2024-03-24T00:06:16.348 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:06:16 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T00:06:16.348 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:06:16 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T00:06:16.348 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:06:16 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T00:06:16.348 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:06:16 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T00:06:16.348 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:06:16 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T00:06:16.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:06:16 smithi189 ceph-mon[36187]: mon.b calling monitor election 2024-03-24T00:06:16.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:06:16 smithi189 ceph-mon[36187]: Updating smithi118:/etc/ceph/ceph.conf 2024-03-24T00:06:16.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:06:16 smithi189 ceph-mon[36187]: Updating smithi189:/etc/ceph/ceph.conf 2024-03-24T00:06:16.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:06:16 smithi189 ceph-mon[36187]: Updating smithi193:/etc/ceph/ceph.conf 2024-03-24T00:06:16.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:06:16 smithi189 ceph-mon[36187]: Updating smithi193:/var/lib/ceph/d79a5420-e971-11ee-95cd-87774f69a715/config/ceph.conf 2024-03-24T00:06:16.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:06:16 smithi189 ceph-mon[36187]: Updating smithi189:/var/lib/ceph/d79a5420-e971-11ee-95cd-87774f69a715/config/ceph.conf 2024-03-24T00:06:16.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:06:16 smithi189 ceph-mon[36187]: mon.b calling monitor election 2024-03-24T00:06:16.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:06:16 smithi189 ceph-mon[36187]: mon.c calling monitor election 2024-03-24T00:06:16.455 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:06:16 smithi189 ceph-mon[36187]: mon.a calling monitor election 2024-03-24T00:06:16.455 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:06:16 smithi189 ceph-mon[36187]: mon.a is new leader, mons a,c,b in quorum (ranks 0,1,2) 2024-03-24T00:06:16.455 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:06:16 smithi189 ceph-mon[36187]: monmap e3: 3 mons at {a=[v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0],b=[v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0],c=[v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0]} removed_ranks: {} disallowed_leaders: {} 2024-03-24T00:06:16.455 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:06:16 smithi189 ceph-mon[36187]: fsmap 2024-03-24T00:06:16.455 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:06:16 smithi189 ceph-mon[36187]: osdmap e4: 0 total, 0 up, 0 in 2024-03-24T00:06:16.455 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:06:16 smithi189 ceph-mon[36187]: mgrmap e13: x(active, since 70s) 2024-03-24T00:06:16.455 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:06:16 smithi189 ceph-mon[36187]: Health check cleared: MON_DOWN (was: 1/3 mons down, quorum a,c) 2024-03-24T00:06:16.455 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:06:16 smithi189 ceph-mon[36187]: Cluster is now healthy 2024-03-24T00:06:16.455 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:06:16 smithi189 ceph-mon[36187]: pgmap v29: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T00:06:16.455 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:06:16 smithi189 ceph-mon[36187]: overall HEALTH_OK 2024-03-24T00:06:16.455 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:06:16 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T00:06:16.455 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:06:16 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T00:06:16.455 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:06:16 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T00:06:16.455 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:06:16 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T00:06:16.456 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:06:16 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T00:06:16.460 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:06:16 smithi193 ceph-mon[36878]: mon.b calling monitor election 2024-03-24T00:06:16.460 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:06:16 smithi193 ceph-mon[36878]: Updating smithi118:/etc/ceph/ceph.conf 2024-03-24T00:06:16.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:06:16 smithi193 ceph-mon[36878]: Updating smithi189:/etc/ceph/ceph.conf 2024-03-24T00:06:16.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:06:16 smithi193 ceph-mon[36878]: Updating smithi193:/etc/ceph/ceph.conf 2024-03-24T00:06:16.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:06:16 smithi193 ceph-mon[36878]: Updating smithi193:/var/lib/ceph/d79a5420-e971-11ee-95cd-87774f69a715/config/ceph.conf 2024-03-24T00:06:16.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:06:16 smithi193 ceph-mon[36878]: Updating smithi189:/var/lib/ceph/d79a5420-e971-11ee-95cd-87774f69a715/config/ceph.conf 2024-03-24T00:06:16.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:06:16 smithi193 ceph-mon[36878]: mon.b calling monitor election 2024-03-24T00:06:16.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:06:16 smithi193 ceph-mon[36878]: mon.c calling monitor election 2024-03-24T00:06:16.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:06:16 smithi193 ceph-mon[36878]: mon.a calling monitor election 2024-03-24T00:06:16.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:06:16 smithi193 ceph-mon[36878]: mon.a is new leader, mons a,c,b in quorum (ranks 0,1,2) 2024-03-24T00:06:16.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:06:16 smithi193 ceph-mon[36878]: monmap e3: 3 mons at {a=[v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0],b=[v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0],c=[v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0]} removed_ranks: {} disallowed_leaders: {} 2024-03-24T00:06:16.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:06:16 smithi193 ceph-mon[36878]: fsmap 2024-03-24T00:06:16.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:06:16 smithi193 ceph-mon[36878]: osdmap e4: 0 total, 0 up, 0 in 2024-03-24T00:06:16.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:06:16 smithi193 ceph-mon[36878]: mgrmap e13: x(active, since 70s) 2024-03-24T00:06:16.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:06:16 smithi193 ceph-mon[36878]: Health check cleared: MON_DOWN (was: 1/3 mons down, quorum a,c) 2024-03-24T00:06:16.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:06:16 smithi193 ceph-mon[36878]: Cluster is now healthy 2024-03-24T00:06:16.462 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:06:16 smithi193 ceph-mon[36878]: pgmap v29: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T00:06:16.462 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:06:16 smithi193 ceph-mon[36878]: overall HEALTH_OK 2024-03-24T00:06:16.462 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:06:16 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T00:06:16.462 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:06:16 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T00:06:16.462 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:06:16 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T00:06:16.462 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:06:16 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T00:06:16.462 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:06:16 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T00:06:17.539 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:06:17 smithi118 ceph-mon[31535]: Updating smithi118:/var/lib/ceph/d79a5420-e971-11ee-95cd-87774f69a715/config/ceph.conf 2024-03-24T00:06:17.539 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:06:17 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T00:06:17.539 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:06:17 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T00:06:17.539 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:06:17 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T00:06:17.539 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:06:17 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-24T00:06:17.540 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:06:17 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T00:06:17.540 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:06:17 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T00:06:17.540 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:06:17 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T00:06:17.540 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:06:17 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T00:06:17.540 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:06:17 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-24T00:06:17.540 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:06:17 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-24T00:06:17.540 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:06:17 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T00:06:17.704 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:06:17 smithi189 ceph-mon[36187]: Updating smithi118:/var/lib/ceph/d79a5420-e971-11ee-95cd-87774f69a715/config/ceph.conf 2024-03-24T00:06:17.704 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:06:17 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T00:06:17.704 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:06:17 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T00:06:17.704 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:06:17 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T00:06:17.704 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:06:17 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-24T00:06:17.705 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:06:17 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T00:06:17.705 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:06:17 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T00:06:17.705 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:06:17 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T00:06:17.705 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:06:17 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T00:06:17.705 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:06:17 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-24T00:06:17.705 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:06:17 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-24T00:06:17.705 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:06:17 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T00:06:17.710 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:06:17 smithi193 ceph-mon[36878]: Updating smithi118:/var/lib/ceph/d79a5420-e971-11ee-95cd-87774f69a715/config/ceph.conf 2024-03-24T00:06:17.710 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:06:17 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T00:06:17.711 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:06:17 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T00:06:17.711 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:06:17 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T00:06:17.711 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:06:17 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-24T00:06:17.711 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:06:17 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T00:06:17.711 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:06:17 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T00:06:17.711 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:06:17 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T00:06:17.711 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:06:17 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T00:06:17.711 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:06:17 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-24T00:06:17.711 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:06:17 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-24T00:06:17.711 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:06:17 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T00:06:18.445 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:06:18 smithi118 ceph-mon[31535]: Reconfiguring mon.a (unknown last config time)... 2024-03-24T00:06:18.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:06:18 smithi118 ceph-mon[31535]: Reconfiguring daemon mon.a on smithi118 2024-03-24T00:06:18.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:06:18 smithi118 ceph-mon[31535]: pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T00:06:18.583 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:06:18.208+0000 7f56fc000700 1 -- 172.21.15.118:0/1379123683 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f56f4100940 msgr2=0x7f56f4100d20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:06:18.583 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:06:18.208+0000 7f56fc000700 1 --2- 172.21.15.118:0/1379123683 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f56f4100940 0x7f56f4100d20 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f56e8009910 tx=0x7f56e8009c20 comp rx=0 tx=0).stop 2024-03-24T00:06:18.583 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:06:18.208+0000 7f56fc000700 1 -- 172.21.15.118:0/1379123683 shutdown_connections 2024-03-24T00:06:18.583 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:06:18.208+0000 7f56fc000700 1 --2- 172.21.15.118:0/1379123683 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f56f4100940 0x7f56f4100d20 unknown :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:06:18.583 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:06:18.208+0000 7f56fc000700 1 -- 172.21.15.118:0/1379123683 >> 172.21.15.118:0/1379123683 conn(0x7f56f4077830 msgr2=0x7f56f40febb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:06:18.583 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:06:18.208+0000 7f56fc000700 1 -- 172.21.15.118:0/1379123683 shutdown_connections 2024-03-24T00:06:18.583 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:06:18.208+0000 7f56fc000700 1 -- 172.21.15.118:0/1379123683 wait complete. 2024-03-24T00:06:18.584 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:06:18.209+0000 7f56fc000700 1 Processor -- start 2024-03-24T00:06:18.584 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:06:18.209+0000 7f56fc000700 1 -- start start 2024-03-24T00:06:18.584 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:06:18.210+0000 7f56fc000700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f56f4100940 0x7f56f419f7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:06:18.584 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:06:18.210+0000 7f56fc000700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f56f419fd10 0x7f56f4199990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:06:18.584 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:06:18.210+0000 7f56fc000700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f56f4199ed0 0x7f56f419a380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:06:18.584 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:06:18.210+0000 7f56fc000700 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f56f410e390 con 0x7f56f4199ed0 2024-03-24T00:06:18.584 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:06:18.210+0000 7f56fc000700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f56f410e210 con 0x7f56f4100940 2024-03-24T00:06:18.584 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:06:18.210+0000 7f56fc000700 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f56f410e510 con 0x7f56f419fd10 2024-03-24T00:06:18.584 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:06:18.210+0000 7f56fa000700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f56f4199ed0 0x7f56f419a380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:06:18.584 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:06:18.210+0000 7f56f9600700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f56f4100940 0x7f56f419f7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:06:18.584 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:06:18.210+0000 7f56f8c00700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f56f419fd10 0x7f56f4199990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:06:18.584 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:06:18.210+0000 7f56fa000700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f56f4199ed0 0x7f56f419a380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.118:3300/0 says I am v2:172.21.15.118:45314/0 (socket says 172.21.15.118:45314) 2024-03-24T00:06:18.584 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:06:18.210+0000 7f56fa000700 1 -- 172.21.15.118:0/561655802 learned_addr learned my addr 172.21.15.118:0/561655802 (peer_addr_for_me v2:172.21.15.118:0/0) 2024-03-24T00:06:18.585 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:06:18.211+0000 7f56f9600700 1 -- 172.21.15.118:0/561655802 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f56f419fd10 msgr2=0x7f56f4199990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:06:18.585 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:06:18.211+0000 7f56f9600700 1 --2- 172.21.15.118:0/561655802 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f56f419fd10 0x7f56f4199990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:06:18.585 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:06:18.211+0000 7f56f9600700 1 -- 172.21.15.118:0/561655802 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f56f4199ed0 msgr2=0x7f56f419a380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:06:18.585 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:06:18.211+0000 7f56f9600700 1 --2- 172.21.15.118:0/561655802 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f56f4199ed0 0x7f56f419a380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:06:18.585 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:06:18.211+0000 7f56f9600700 1 -- 172.21.15.118:0/561655802 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f56f41a66a0 con 0x7f56f4100940 2024-03-24T00:06:18.585 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:06:18.211+0000 7f56f9600700 1 --2- 172.21.15.118:0/561655802 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f56f4100940 0x7f56f419f7d0 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7f56e8014000 tx=0x7f56e800bdc0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:06:18.585 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:06:18.211+0000 7f56fa000700 1 --2- 172.21.15.118:0/561655802 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f56f4199ed0 0x7f56f419a380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T00:06:18.585 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:06:18.211+0000 7f56f2000700 1 -- 172.21.15.118:0/561655802 <== mon.2 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f56e8026070 con 0x7f56f4100940 2024-03-24T00:06:18.585 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:06:18.211+0000 7f56fc000700 1 -- 172.21.15.118:0/561655802 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f56f41a6870 con 0x7f56f4100940 2024-03-24T00:06:18.585 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:06:18.211+0000 7f56fc000700 1 -- 172.21.15.118:0/561655802 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f56f41a6da0 con 0x7f56f4100940 2024-03-24T00:06:18.585 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:06:18.212+0000 7f56f2000700 1 -- 172.21.15.118:0/561655802 <== mon.2 v2:172.21.15.189:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f56e8005da0 con 0x7f56f4100940 2024-03-24T00:06:18.585 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:06:18.212+0000 7f56f2000700 1 -- 172.21.15.118:0/561655802 <== mon.2 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f56e80038e0 con 0x7f56f4100940 2024-03-24T00:06:18.585 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:06:18.213+0000 7f56fc000700 1 -- 172.21.15.118:0/561655802 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f56d4005330 con 0x7f56f4100940 2024-03-24T00:06:18.585 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:06:18.213+0000 7f56f2000700 1 -- 172.21.15.118:0/561655802 <== mon.2 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 13) v1 ==== 46163+0+0 (secure 0 0 0) 0x7f56e8029070 con 0x7f56f4100940 2024-03-24T00:06:18.585 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:06:18.214+0000 7f56f2000700 1 --2- 172.21.15.118:0/561655802 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7f56dc039450 0x7f56dc03b910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:06:18.586 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:06:18.214+0000 7f56f2000700 1 -- 172.21.15.118:0/561655802 <== mon.2 v2:172.21.15.189:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1155+0+0 (secure 0 0 0) 0x7f56e804c9a0 con 0x7f56f4100940 2024-03-24T00:06:18.586 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:06:18.214+0000 7f56f8c00700 1 --2- 172.21.15.118:0/561655802 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7f56dc039450 0x7f56dc03b910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:06:18.586 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:06:18.215+0000 7f56f8c00700 1 --2- 172.21.15.118:0/561655802 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7f56dc039450 0x7f56dc03b910 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f56e0009ad0 tx=0x7f56e0009210 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:06:18.586 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:06:18.219+0000 7f56f2000700 1 -- 172.21.15.118:0/561655802 <== mon.2 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f56e8013070 con 0x7f56f4100940 2024-03-24T00:06:18.586 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:06:18.447+0000 7f56fc000700 1 -- 172.21.15.118:0/561655802 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "config generate-minimal-conf"} v 0) v1 -- 0x7f56d4005620 con 0x7f56f4100940 2024-03-24T00:06:18.586 INFO:teuthology.orchestra.run.smithi118.stdout:# minimal ceph.conf for d79a5420-e971-11ee-95cd-87774f69a715 2024-03-24T00:06:18.586 INFO:teuthology.orchestra.run.smithi118.stdout:[global] 2024-03-24T00:06:18.586 INFO:teuthology.orchestra.run.smithi118.stdout: fsid = d79a5420-e971-11ee-95cd-87774f69a715 2024-03-24T00:06:18.586 INFO:teuthology.orchestra.run.smithi118.stdout: mon_host = [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] 2024-03-24T00:06:18.587 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:06:18.448+0000 7f56f2000700 1 -- 172.21.15.118:0/561655802 <== mon.2 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{"prefix": "config generate-minimal-conf"}]=0 v9) v1 ==== 76+0+277 (secure 0 0 0) 0x7f56e8030090 con 0x7f56f4100940 2024-03-24T00:06:18.587 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:06:18.452+0000 7f56fc000700 1 -- 172.21.15.118:0/561655802 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7f56dc039450 msgr2=0x7f56dc03b910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:06:18.587 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:06:18.452+0000 7f56fc000700 1 --2- 172.21.15.118:0/561655802 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7f56dc039450 0x7f56dc03b910 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f56e0009ad0 tx=0x7f56e0009210 comp rx=0 tx=0).stop 2024-03-24T00:06:18.587 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:06:18.452+0000 7f56fc000700 1 -- 172.21.15.118:0/561655802 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f56f4100940 msgr2=0x7f56f419f7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:06:18.587 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:06:18.452+0000 7f56fc000700 1 --2- 172.21.15.118:0/561655802 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f56f4100940 0x7f56f419f7d0 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7f56e8014000 tx=0x7f56e800bdc0 comp rx=0 tx=0).stop 2024-03-24T00:06:18.587 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:06:18.452+0000 7f56fc000700 1 -- 172.21.15.118:0/561655802 shutdown_connections 2024-03-24T00:06:18.587 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:06:18.452+0000 7f56fc000700 1 --2- 172.21.15.118:0/561655802 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f56f4100940 0x7f56f419f7d0 unknown :-1 s=CLOSED pgs=5 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:06:18.587 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:06:18.452+0000 7f56fc000700 1 --2- 172.21.15.118:0/561655802 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7f56dc039450 0x7f56dc03b910 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:06:18.587 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:06:18.452+0000 7f56fc000700 1 --2- 172.21.15.118:0/561655802 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f56f4199ed0 0x7f56f419a380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:06:18.587 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:06:18.452+0000 7f56fc000700 1 --2- 172.21.15.118:0/561655802 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f56f419fd10 0x7f56f4199990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:06:18.587 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:06:18.452+0000 7f56fc000700 1 -- 172.21.15.118:0/561655802 >> 172.21.15.118:0/561655802 conn(0x7f56f4077830 msgr2=0x7f56f4108f70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:06:18.587 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:06:18.453+0000 7f56fc000700 1 -- 172.21.15.118:0/561655802 shutdown_connections 2024-03-24T00:06:18.588 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:06:18.453+0000 7f56fc000700 1 -- 172.21.15.118:0/561655802 wait complete. 2024-03-24T00:06:18.703 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:06:18 smithi189 ceph-mon[36187]: Reconfiguring mon.a (unknown last config time)... 2024-03-24T00:06:18.704 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:06:18 smithi189 ceph-mon[36187]: Reconfiguring daemon mon.a on smithi118 2024-03-24T00:06:18.704 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:06:18 smithi189 ceph-mon[36187]: pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T00:06:18.710 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:06:18 smithi193 ceph-mon[36878]: Reconfiguring mon.a (unknown last config time)... 2024-03-24T00:06:18.711 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:06:18 smithi193 ceph-mon[36878]: Reconfiguring daemon mon.a on smithi118 2024-03-24T00:06:18.711 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:06:18 smithi193 ceph-mon[36878]: pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T00:06:19.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:06:19 smithi118 ceph-mon[31535]: from='client.? 172.21.15.118:0/561655802' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T00:06:19.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:06:19 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T00:06:19.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:06:19 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T00:06:19.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:06:19 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-24T00:06:19.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:06:19 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-24T00:06:19.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:06:19 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T00:06:19.704 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:06:19 smithi189 ceph-mon[36187]: from='client.? 172.21.15.118:0/561655802' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T00:06:19.704 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:06:19 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T00:06:19.704 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:06:19 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T00:06:19.704 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:06:19 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-24T00:06:19.705 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:06:19 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-24T00:06:19.705 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:06:19 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T00:06:19.711 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:06:19 smithi193 ceph-mon[36878]: from='client.? 172.21.15.118:0/561655802' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T00:06:19.711 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:06:19 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T00:06:19.711 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:06:19 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T00:06:19.711 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:06:19 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-24T00:06:19.711 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:06:19 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-24T00:06:19.711 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:06:19 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T00:06:19.755 INFO:tasks.cephadm:Distributing (final) config and client.admin keyring... 2024-03-24T00:06:19.755 DEBUG:teuthology.orchestra.run.smithi118:> set -ex 2024-03-24T00:06:19.755 DEBUG:teuthology.orchestra.run.smithi118:> sudo dd of=/etc/ceph/ceph.conf 2024-03-24T00:06:19.839 DEBUG:teuthology.orchestra.run.smithi118:> set -ex 2024-03-24T00:06:19.839 DEBUG:teuthology.orchestra.run.smithi118:> sudo dd of=/etc/ceph/ceph.client.admin.keyring 2024-03-24T00:06:19.974 DEBUG:teuthology.orchestra.run.smithi189:> set -ex 2024-03-24T00:06:19.974 DEBUG:teuthology.orchestra.run.smithi189:> sudo dd of=/etc/ceph/ceph.conf 2024-03-24T00:06:20.004 DEBUG:teuthology.orchestra.run.smithi189:> set -ex 2024-03-24T00:06:20.004 DEBUG:teuthology.orchestra.run.smithi189:> sudo dd of=/etc/ceph/ceph.client.admin.keyring 2024-03-24T00:06:20.071 DEBUG:teuthology.orchestra.run.smithi193:> set -ex 2024-03-24T00:06:20.072 DEBUG:teuthology.orchestra.run.smithi193:> sudo dd of=/etc/ceph/ceph.conf 2024-03-24T00:06:20.101 DEBUG:teuthology.orchestra.run.smithi193:> set -ex 2024-03-24T00:06:20.101 DEBUG:teuthology.orchestra.run.smithi193:> sudo dd of=/etc/ceph/ceph.client.admin.keyring 2024-03-24T00:06:20.169 INFO:tasks.cephadm:Adding mgr.x on smithi118 2024-03-24T00:06:20.169 INFO:tasks.cephadm:Adding mgr.y on smithi189 2024-03-24T00:06:20.169 INFO:tasks.cephadm:Adding mgr.z on smithi193 2024-03-24T00:06:20.169 DEBUG:teuthology.orchestra.run.smithi193:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d79a5420-e971-11ee-95cd-87774f69a715 -- ceph orch apply mgr '3;smithi118=x;smithi189=y;smithi193=z' 2024-03-24T00:06:20.437 INFO:teuthology.orchestra.run.smithi193.stderr:Inferring config /var/lib/ceph/d79a5420-e971-11ee-95cd-87774f69a715/mon.c/config 2024-03-24T00:06:20.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:06:20 smithi118 ceph-mon[31535]: Reconfiguring mon.b (monmap changed)... 2024-03-24T00:06:20.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:06:20 smithi118 ceph-mon[31535]: Reconfiguring daemon mon.b on smithi189 2024-03-24T00:06:20.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:06:20 smithi118 ceph-mon[31535]: pgmap v31: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T00:06:20.703 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:06:20 smithi189 ceph-mon[36187]: Reconfiguring mon.b (monmap changed)... 2024-03-24T00:06:20.703 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:06:20 smithi189 ceph-mon[36187]: Reconfiguring daemon mon.b on smithi189 2024-03-24T00:06:20.703 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:06:20 smithi189 ceph-mon[36187]: pgmap v31: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T00:06:20.710 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:06:20 smithi193 ceph-mon[36878]: Reconfiguring mon.b (monmap changed)... 2024-03-24T00:06:20.710 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:06:20 smithi193 ceph-mon[36878]: Reconfiguring daemon mon.b on smithi189 2024-03-24T00:06:20.710 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:06:20 smithi193 ceph-mon[36878]: pgmap v31: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T00:06:21.998 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:06:21 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T00:06:21.998 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:06:21 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T00:06:21.998 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:06:21 smithi193 ceph-mon[36878]: Reconfiguring mon.c (monmap changed)... 2024-03-24T00:06:21.998 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:06:21 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-24T00:06:21.998 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:06:21 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-24T00:06:21.999 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:06:21 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T00:06:21.999 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:06:21 smithi193 ceph-mon[36878]: Reconfiguring daemon mon.c on smithi193 2024-03-24T00:06:21.999 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:06:21 smithi193 ceph-mon[36878]: pgmap v32: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T00:06:22.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:06:21 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T00:06:22.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:06:21 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T00:06:22.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:06:21 smithi118 ceph-mon[31535]: Reconfiguring mon.c (monmap changed)... 2024-03-24T00:06:22.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:06:21 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-24T00:06:22.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:06:21 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-24T00:06:22.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:06:21 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T00:06:22.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:06:21 smithi118 ceph-mon[31535]: Reconfiguring daemon mon.c on smithi193 2024-03-24T00:06:22.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:06:21 smithi118 ceph-mon[31535]: pgmap v32: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T00:06:22.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:06:21 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T00:06:22.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:06:21 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T00:06:22.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:06:21 smithi189 ceph-mon[36187]: Reconfiguring mon.c (monmap changed)... 2024-03-24T00:06:22.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:06:21 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-24T00:06:22.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:06:21 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-24T00:06:22.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:06:21 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T00:06:22.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:06:21 smithi189 ceph-mon[36187]: Reconfiguring daemon mon.c on smithi193 2024-03-24T00:06:22.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:06:21 smithi189 ceph-mon[36187]: pgmap v32: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T00:06:22.688 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-24T00:06:22.688+0000 7f91de9c8700 1 -- 172.21.15.193:0/524150107 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f91d8100bb0 msgr2=0x7f91d8100fd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:06:22.688 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-24T00:06:22.688+0000 7f91de9c8700 1 --2- 172.21.15.193:0/524150107 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f91d8100bb0 0x7f91d8100fd0 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f91c8009910 tx=0x7f91c8009c20 comp rx=0 tx=0).stop 2024-03-24T00:06:22.688 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-24T00:06:22.688+0000 7f91de9c8700 1 -- 172.21.15.193:0/524150107 shutdown_connections 2024-03-24T00:06:22.688 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-24T00:06:22.688+0000 7f91de9c8700 1 --2- 172.21.15.193:0/524150107 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f91d8100bb0 0x7f91d8100fd0 unknown :-1 s=CLOSED pgs=110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:06:22.688 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-24T00:06:22.688+0000 7f91de9c8700 1 -- 172.21.15.193:0/524150107 >> 172.21.15.193:0/524150107 conn(0x7f91d80fc170 msgr2=0x7f91d80fe590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:06:22.688 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-24T00:06:22.688+0000 7f91de9c8700 1 -- 172.21.15.193:0/524150107 shutdown_connections 2024-03-24T00:06:22.688 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-24T00:06:22.688+0000 7f91de9c8700 1 -- 172.21.15.193:0/524150107 wait complete. 2024-03-24T00:06:22.691 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-24T00:06:22.688+0000 7f91de9c8700 1 Processor -- start 2024-03-24T00:06:22.691 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-24T00:06:22.689+0000 7f91de9c8700 1 -- start start 2024-03-24T00:06:22.691 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-24T00:06:22.689+0000 7f91de9c8700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f91d8100bb0 0x7f91d806a6a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:06:22.691 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-24T00:06:22.689+0000 7f91de9c8700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f91d819a200 0x7f91d819c5f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:06:22.691 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-24T00:06:22.689+0000 7f91de9c8700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f91d819cbd0 0x7f91d819f060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:06:22.691 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-24T00:06:22.689+0000 7f91de9c8700 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f91d8106bf0 con 0x7f91d819a200 2024-03-24T00:06:22.691 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-24T00:06:22.689+0000 7f91de9c8700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f91d8106a70 con 0x7f91d819cbd0 2024-03-24T00:06:22.691 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-24T00:06:22.689+0000 7f91de9c8700 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f91d8106d70 con 0x7f91d8100bb0 2024-03-24T00:06:22.691 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-24T00:06:22.689+0000 7f91d7fff700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f91d8100bb0 0x7f91d806a6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:06:22.692 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-24T00:06:22.689+0000 7f91d7fff700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f91d8100bb0 0x7f91d806a6a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.193:37834/0 (socket says 172.21.15.193:37834) 2024-03-24T00:06:22.692 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-24T00:06:22.689+0000 7f91d7fff700 1 -- 172.21.15.193:0/1503817375 learned_addr learned my addr 172.21.15.193:0/1503817375 (peer_addr_for_me v2:172.21.15.193:0/0) 2024-03-24T00:06:22.692 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-24T00:06:22.689+0000 7f91dcf65700 1 --2- 172.21.15.193:0/1503817375 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f91d819cbd0 0x7f91d819f060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:06:22.692 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-24T00:06:22.689+0000 7f91d77fe700 1 --2- 172.21.15.193:0/1503817375 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f91d819a200 0x7f91d819c5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:06:22.692 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-24T00:06:22.689+0000 7f91d7fff700 1 -- 172.21.15.193:0/1503817375 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f91d819cbd0 msgr2=0x7f91d819f060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:06:22.692 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-24T00:06:22.689+0000 7f91d7fff700 1 --2- 172.21.15.193:0/1503817375 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f91d819cbd0 0x7f91d819f060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:06:22.692 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-24T00:06:22.689+0000 7f91d7fff700 1 -- 172.21.15.193:0/1503817375 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f91d819a200 msgr2=0x7f91d819c5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:06:22.692 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-24T00:06:22.689+0000 7f91d7fff700 1 --2- 172.21.15.193:0/1503817375 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f91d819a200 0x7f91d819c5f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:06:22.692 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-24T00:06:22.689+0000 7f91d7fff700 1 -- 172.21.15.193:0/1503817375 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f91d819f930 con 0x7f91d8100bb0 2024-03-24T00:06:22.692 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-24T00:06:22.689+0000 7f91d77fe700 1 --2- 172.21.15.193:0/1503817375 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f91d819a200 0x7f91d819c5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-24T00:06:22.692 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-24T00:06:22.689+0000 7f91dcf65700 1 --2- 172.21.15.193:0/1503817375 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f91d819cbd0 0x7f91d819f060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T00:06:22.692 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-24T00:06:22.690+0000 7f91d7fff700 1 --2- 172.21.15.193:0/1503817375 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f91d8100bb0 0x7f91d806a6a0 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f91c8000c00 tx=0x7f91c80050b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:06:22.692 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-24T00:06:22.690+0000 7f91d57fa700 1 -- 172.21.15.193:0/1503817375 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f91c8027070 con 0x7f91d8100bb0 2024-03-24T00:06:22.693 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-24T00:06:22.690+0000 7f91d57fa700 1 -- 172.21.15.193:0/1503817375 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f91c8022470 con 0x7f91d8100bb0 2024-03-24T00:06:22.693 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-24T00:06:22.690+0000 7f91de9c8700 1 -- 172.21.15.193:0/1503817375 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f91d819fbc0 con 0x7f91d8100bb0 2024-03-24T00:06:22.693 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-24T00:06:22.690+0000 7f91d57fa700 1 -- 172.21.15.193:0/1503817375 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f91c800f6c0 con 0x7f91d8100bb0 2024-03-24T00:06:22.693 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-24T00:06:22.690+0000 7f91de9c8700 1 -- 172.21.15.193:0/1503817375 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f91d81a00f0 con 0x7f91d8100bb0 2024-03-24T00:06:22.693 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-24T00:06:22.691+0000 7f91d57fa700 1 -- 172.21.15.193:0/1503817375 <== mon.1 v2:172.21.15.193:3300/0 4 ==== mgrmap(e 13) v1 ==== 46163+0+0 (secure 0 0 0) 0x7f91c802a070 con 0x7f91d8100bb0 2024-03-24T00:06:22.693 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-24T00:06:22.691+0000 7f91de9c8700 1 -- 172.21.15.193:0/1503817375 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f91d80637c0 con 0x7f91d8100bb0 2024-03-24T00:06:22.693 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-24T00:06:22.691+0000 7f91d57fa700 1 --2- 172.21.15.193:0/1503817375 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7f91c0039450 0x7f91c003b910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:06:22.693 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-24T00:06:22.691+0000 7f91d57fa700 1 -- 172.21.15.193:0/1503817375 <== mon.1 v2:172.21.15.193:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1155+0+0 (secure 0 0 0) 0x7f91c804d3a0 con 0x7f91d8100bb0 2024-03-24T00:06:22.693 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-24T00:06:22.691+0000 7f91d77fe700 1 --2- 172.21.15.193:0/1503817375 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7f91c0039450 0x7f91c003b910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:06:22.694 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-24T00:06:22.692+0000 7f91d77fe700 1 --2- 172.21.15.193:0/1503817375 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7f91c0039450 0x7f91c003b910 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f91cc0079b0 tx=0x7f91cc008040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:06:22.696 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-24T00:06:22.696+0000 7f91d57fa700 1 -- 172.21.15.193:0/1503817375 <== mon.1 v2:172.21.15.193:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f91c8013070 con 0x7f91d8100bb0 2024-03-24T00:06:22.904 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-24T00:06:22.904+0000 7f91de9c8700 1 -- 172.21.15.193:0/1503817375 --> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "mgr", "placement": "3;smithi118=x;smithi189=y;smithi193=z", "target": ["mon-mgr", ""]}) v1 -- 0x7f91d8062660 con 0x7f91c0039450 2024-03-24T00:06:22.918 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-24T00:06:22.918+0000 7f91d57fa700 1 -- 172.21.15.193:0/1503817375 <== mgr.14152 v2:172.21.15.118:6800/2112912943 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+24 (secure 0 0 0) 0x7f91d8062660 con 0x7f91c0039450 2024-03-24T00:06:22.918 INFO:teuthology.orchestra.run.smithi193.stdout:Scheduled mgr update... 2024-03-24T00:06:22.920 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-24T00:06:22.920+0000 7f91de9c8700 1 -- 172.21.15.193:0/1503817375 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7f91c0039450 msgr2=0x7f91c003b910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:06:22.921 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-24T00:06:22.920+0000 7f91de9c8700 1 --2- 172.21.15.193:0/1503817375 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7f91c0039450 0x7f91c003b910 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f91cc0079b0 tx=0x7f91cc008040 comp rx=0 tx=0).stop 2024-03-24T00:06:22.921 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-24T00:06:22.920+0000 7f91de9c8700 1 -- 172.21.15.193:0/1503817375 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f91d8100bb0 msgr2=0x7f91d806a6a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:06:22.921 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-24T00:06:22.920+0000 7f91de9c8700 1 --2- 172.21.15.193:0/1503817375 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f91d8100bb0 0x7f91d806a6a0 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f91c8000c00 tx=0x7f91c80050b0 comp rx=0 tx=0).stop 2024-03-24T00:06:22.921 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-24T00:06:22.920+0000 7f91de9c8700 1 -- 172.21.15.193:0/1503817375 shutdown_connections 2024-03-24T00:06:22.921 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-24T00:06:22.920+0000 7f91de9c8700 1 --2- 172.21.15.193:0/1503817375 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f91d819cbd0 0x7f91d819f060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:06:22.921 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-24T00:06:22.921+0000 7f91de9c8700 1 --2- 172.21.15.193:0/1503817375 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7f91c0039450 0x7f91c003b910 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:06:22.921 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-24T00:06:22.921+0000 7f91de9c8700 1 --2- 172.21.15.193:0/1503817375 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f91d8100bb0 0x7f91d806a6a0 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:06:22.921 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-24T00:06:22.921+0000 7f91de9c8700 1 --2- 172.21.15.193:0/1503817375 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f91d819a200 0x7f91d819c5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:06:22.921 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-24T00:06:22.921+0000 7f91de9c8700 1 -- 172.21.15.193:0/1503817375 >> 172.21.15.193:0/1503817375 conn(0x7f91d80fc170 msgr2=0x7f91d80fcde0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:06:22.921 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-24T00:06:22.921+0000 7f91de9c8700 1 -- 172.21.15.193:0/1503817375 shutdown_connections 2024-03-24T00:06:22.921 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-24T00:06:22.921+0000 7f91de9c8700 1 -- 172.21.15.193:0/1503817375 wait complete. 2024-03-24T00:06:23.486 DEBUG:teuthology.orchestra.run.smithi189:mgr.y> sudo journalctl -f -n 0 -u ceph-d79a5420-e971-11ee-95cd-87774f69a715@mgr.y.service 2024-03-24T00:06:23.489 DEBUG:teuthology.orchestra.run.smithi193:mgr.z> sudo journalctl -f -n 0 -u ceph-d79a5420-e971-11ee-95cd-87774f69a715@mgr.z.service 2024-03-24T00:06:23.491 INFO:tasks.cephadm:Deploying OSDs... 2024-03-24T00:06:23.491 DEBUG:teuthology.orchestra.run.smithi118:> set -ex 2024-03-24T00:06:23.491 DEBUG:teuthology.orchestra.run.smithi118:> dd if=/scratch_devs of=/dev/stdout 2024-03-24T00:06:23.549 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-03-24T00:06:23.549 DEBUG:teuthology.orchestra.run.smithi118:> stat /dev/vg_nvme/lv_1 2024-03-24T00:06:23.600 INFO:teuthology.orchestra.run.smithi118.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-03-24T00:06:23.600 INFO:teuthology.orchestra.run.smithi118.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-24T00:06:23.600 INFO:teuthology.orchestra.run.smithi118.stdout:Device: 5h/5d Inode: 631 Links: 1 2024-03-24T00:06:23.600 INFO:teuthology.orchestra.run.smithi118.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-24T00:06:23.600 INFO:teuthology.orchestra.run.smithi118.stdout:Context: system_u:object_r:device_t:s0 2024-03-24T00:06:23.600 INFO:teuthology.orchestra.run.smithi118.stdout:Access: 2024-03-24 00:05:22.186853507 +0000 2024-03-24T00:06:23.600 INFO:teuthology.orchestra.run.smithi118.stdout:Modify: 2024-03-23 23:59:37.766006131 +0000 2024-03-24T00:06:23.600 INFO:teuthology.orchestra.run.smithi118.stdout:Change: 2024-03-23 23:59:37.766006131 +0000 2024-03-24T00:06:23.600 INFO:teuthology.orchestra.run.smithi118.stdout: Birth: 2024-03-23 23:59:37.766006131 +0000 2024-03-24T00:06:23.601 DEBUG:teuthology.orchestra.run.smithi118:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-03-24T00:06:23.685 INFO:teuthology.orchestra.run.smithi118.stderr:1+0 records in 2024-03-24T00:06:23.685 INFO:teuthology.orchestra.run.smithi118.stderr:1+0 records out 2024-03-24T00:06:23.685 INFO:teuthology.orchestra.run.smithi118.stderr:512 bytes copied, 0.000387418 s, 1.3 MB/s 2024-03-24T00:06:23.690 DEBUG:teuthology.orchestra.run.smithi118:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-03-24T00:06:23.739 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:06:23 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T00:06:23.739 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:06:23 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T00:06:23.739 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:06:23 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T00:06:23.739 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:06:23 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T00:06:23.739 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:06:23 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T00:06:23.739 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:06:23 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T00:06:23.739 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:06:23 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T00:06:23.739 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:06:23 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T00:06:23.739 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:06:23 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T00:06:23.740 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:06:23 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T00:06:23.740 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:06:23 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T00:06:23.740 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:06:23 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.z", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-24T00:06:23.740 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:06:23 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.z", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-03-24T00:06:23.740 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:06:23 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-03-24T00:06:23.740 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:06:23 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T00:06:23.743 DEBUG:teuthology.orchestra.run.smithi118:> stat /dev/vg_nvme/lv_2 2024-03-24T00:06:23.789 INFO:teuthology.orchestra.run.smithi118.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-03-24T00:06:23.789 INFO:teuthology.orchestra.run.smithi118.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-24T00:06:23.789 INFO:teuthology.orchestra.run.smithi118.stdout:Device: 5h/5d Inode: 653 Links: 1 2024-03-24T00:06:23.789 INFO:teuthology.orchestra.run.smithi118.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-24T00:06:23.789 INFO:teuthology.orchestra.run.smithi118.stdout:Context: system_u:object_r:device_t:s0 2024-03-24T00:06:23.789 INFO:teuthology.orchestra.run.smithi118.stdout:Access: 2024-03-24 00:05:22.244852167 +0000 2024-03-24T00:06:23.789 INFO:teuthology.orchestra.run.smithi118.stdout:Modify: 2024-03-23 23:59:38.330992962 +0000 2024-03-24T00:06:23.789 INFO:teuthology.orchestra.run.smithi118.stdout:Change: 2024-03-23 23:59:38.330992962 +0000 2024-03-24T00:06:23.789 INFO:teuthology.orchestra.run.smithi118.stdout: Birth: 2024-03-23 23:59:38.330992962 +0000 2024-03-24T00:06:23.790 DEBUG:teuthology.orchestra.run.smithi118:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-03-24T00:06:23.913 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:06:23 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T00:06:23.913 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:06:23 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T00:06:23.913 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:06:23 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T00:06:23.914 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:06:23 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T00:06:23.914 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:06:23 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T00:06:23.914 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:06:23 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T00:06:23.914 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:06:23 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T00:06:23.914 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:06:23 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T00:06:23.914 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:06:23 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T00:06:23.914 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:06:23 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T00:06:23.914 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:06:23 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T00:06:23.914 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:06:23 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.z", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-24T00:06:23.914 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:06:23 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.z", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-03-24T00:06:23.914 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:06:23 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-03-24T00:06:23.915 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:06:23 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T00:06:23.923 INFO:teuthology.orchestra.run.smithi118.stderr:1+0 records in 2024-03-24T00:06:23.924 INFO:teuthology.orchestra.run.smithi118.stderr:1+0 records out 2024-03-24T00:06:23.924 INFO:teuthology.orchestra.run.smithi118.stderr:512 bytes copied, 0.000268527 s, 1.9 MB/s 2024-03-24T00:06:23.928 DEBUG:teuthology.orchestra.run.smithi118:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-03-24T00:06:23.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:06:23 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T00:06:23.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:06:23 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T00:06:23.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:06:23 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T00:06:23.955 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:06:23 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T00:06:23.955 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:06:23 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T00:06:23.955 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:06:23 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T00:06:23.955 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:06:23 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T00:06:23.955 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:06:23 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T00:06:23.955 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:06:23 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T00:06:23.956 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:06:23 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T00:06:23.956 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:06:23 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T00:06:23.956 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:06:23 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.z", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-24T00:06:23.956 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:06:23 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.z", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-03-24T00:06:23.956 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:06:23 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-03-24T00:06:23.956 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:06:23 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T00:06:23.984 DEBUG:teuthology.orchestra.run.smithi118:> stat /dev/vg_nvme/lv_3 2024-03-24T00:06:24.032 INFO:teuthology.orchestra.run.smithi118.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-03-24T00:06:24.032 INFO:teuthology.orchestra.run.smithi118.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-24T00:06:24.032 INFO:teuthology.orchestra.run.smithi118.stdout:Device: 5h/5d Inode: 669 Links: 1 2024-03-24T00:06:24.032 INFO:teuthology.orchestra.run.smithi118.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-24T00:06:24.032 INFO:teuthology.orchestra.run.smithi118.stdout:Context: system_u:object_r:device_t:s0 2024-03-24T00:06:24.032 INFO:teuthology.orchestra.run.smithi118.stdout:Access: 2024-03-24 00:05:22.285851220 +0000 2024-03-24T00:06:24.032 INFO:teuthology.orchestra.run.smithi118.stdout:Modify: 2024-03-23 23:59:38.909979467 +0000 2024-03-24T00:06:24.032 INFO:teuthology.orchestra.run.smithi118.stdout:Change: 2024-03-23 23:59:38.909979467 +0000 2024-03-24T00:06:24.032 INFO:teuthology.orchestra.run.smithi118.stdout: Birth: 2024-03-23 23:59:38.909979467 +0000 2024-03-24T00:06:24.033 DEBUG:teuthology.orchestra.run.smithi118:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-03-24T00:06:24.119 INFO:teuthology.orchestra.run.smithi118.stderr:1+0 records in 2024-03-24T00:06:24.119 INFO:teuthology.orchestra.run.smithi118.stderr:1+0 records out 2024-03-24T00:06:24.119 INFO:teuthology.orchestra.run.smithi118.stderr:512 bytes copied, 0.000251004 s, 2.0 MB/s 2024-03-24T00:06:24.123 DEBUG:teuthology.orchestra.run.smithi118:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-03-24T00:06:24.180 DEBUG:teuthology.orchestra.run.smithi118:> stat /dev/vg_nvme/lv_4 2024-03-24T00:06:24.230 INFO:teuthology.orchestra.run.smithi118.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-03-24T00:06:24.230 INFO:teuthology.orchestra.run.smithi118.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-24T00:06:24.230 INFO:teuthology.orchestra.run.smithi118.stdout:Device: 5h/5d Inode: 688 Links: 1 2024-03-24T00:06:24.230 INFO:teuthology.orchestra.run.smithi118.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-24T00:06:24.230 INFO:teuthology.orchestra.run.smithi118.stdout:Context: system_u:object_r:device_t:s0 2024-03-24T00:06:24.230 INFO:teuthology.orchestra.run.smithi118.stdout:Access: 2024-03-24 00:05:22.330850181 +0000 2024-03-24T00:06:24.230 INFO:teuthology.orchestra.run.smithi118.stdout:Modify: 2024-03-23 23:59:39.472966345 +0000 2024-03-24T00:06:24.230 INFO:teuthology.orchestra.run.smithi118.stdout:Change: 2024-03-23 23:59:39.472966345 +0000 2024-03-24T00:06:24.231 INFO:teuthology.orchestra.run.smithi118.stdout: Birth: 2024-03-23 23:59:39.472966345 +0000 2024-03-24T00:06:24.231 DEBUG:teuthology.orchestra.run.smithi118:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-03-24T00:06:24.315 INFO:teuthology.orchestra.run.smithi118.stderr:1+0 records in 2024-03-24T00:06:24.315 INFO:teuthology.orchestra.run.smithi118.stderr:1+0 records out 2024-03-24T00:06:24.315 INFO:teuthology.orchestra.run.smithi118.stderr:512 bytes copied, 0.000241495 s, 2.1 MB/s 2024-03-24T00:06:24.318 DEBUG:teuthology.orchestra.run.smithi118:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-03-24T00:06:24.373 DEBUG:teuthology.orchestra.run.smithi189:> set -ex 2024-03-24T00:06:24.373 DEBUG:teuthology.orchestra.run.smithi189:> dd if=/scratch_devs of=/dev/stdout 2024-03-24T00:06:24.392 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-03-24T00:06:24.392 DEBUG:teuthology.orchestra.run.smithi189:> stat /dev/vg_nvme/lv_1 2024-03-24T00:06:24.449 INFO:teuthology.orchestra.run.smithi189.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-03-24T00:06:24.450 INFO:teuthology.orchestra.run.smithi189.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-24T00:06:24.450 INFO:teuthology.orchestra.run.smithi189.stdout:Device: 5h/5d Inode: 661 Links: 1 2024-03-24T00:06:24.450 INFO:teuthology.orchestra.run.smithi189.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-24T00:06:24.450 INFO:teuthology.orchestra.run.smithi189.stdout:Context: system_u:object_r:device_t:s0 2024-03-24T00:06:24.450 INFO:teuthology.orchestra.run.smithi189.stdout:Access: 2024-03-24 00:06:07.843475215 +0000 2024-03-24T00:06:24.450 INFO:teuthology.orchestra.run.smithi189.stdout:Modify: 2024-03-23 23:58:30.687114669 +0000 2024-03-24T00:06:24.450 INFO:teuthology.orchestra.run.smithi189.stdout:Change: 2024-03-23 23:58:30.687114669 +0000 2024-03-24T00:06:24.450 INFO:teuthology.orchestra.run.smithi189.stdout: Birth: 2024-03-23 23:58:30.687114669 +0000 2024-03-24T00:06:24.450 DEBUG:teuthology.orchestra.run.smithi189:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-03-24T00:06:24.518 INFO:teuthology.orchestra.run.smithi189.stderr:1+0 records in 2024-03-24T00:06:24.518 INFO:teuthology.orchestra.run.smithi189.stderr:1+0 records out 2024-03-24T00:06:24.519 INFO:teuthology.orchestra.run.smithi189.stderr:512 bytes copied, 0.000228572 s, 2.2 MB/s 2024-03-24T00:06:24.520 DEBUG:teuthology.orchestra.run.smithi189:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-03-24T00:06:24.577 DEBUG:teuthology.orchestra.run.smithi189:> stat /dev/vg_nvme/lv_2 2024-03-24T00:06:24.634 INFO:teuthology.orchestra.run.smithi189.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-03-24T00:06:24.634 INFO:teuthology.orchestra.run.smithi189.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-24T00:06:24.634 INFO:teuthology.orchestra.run.smithi189.stdout:Device: 5h/5d Inode: 680 Links: 1 2024-03-24T00:06:24.634 INFO:teuthology.orchestra.run.smithi189.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-24T00:06:24.634 INFO:teuthology.orchestra.run.smithi189.stdout:Context: system_u:object_r:device_t:s0 2024-03-24T00:06:24.634 INFO:teuthology.orchestra.run.smithi189.stdout:Access: 2024-03-24 00:06:07.843475215 +0000 2024-03-24T00:06:24.635 INFO:teuthology.orchestra.run.smithi189.stdout:Modify: 2024-03-23 23:58:31.039107649 +0000 2024-03-24T00:06:24.635 INFO:teuthology.orchestra.run.smithi189.stdout:Change: 2024-03-23 23:58:31.039107649 +0000 2024-03-24T00:06:24.635 INFO:teuthology.orchestra.run.smithi189.stdout: Birth: 2024-03-23 23:58:31.039107649 +0000 2024-03-24T00:06:24.635 DEBUG:teuthology.orchestra.run.smithi189:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-03-24T00:06:24.687 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:06:24 smithi193 ceph-mon[36878]: from='client.24106 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "placement": "3;smithi118=x;smithi189=y;smithi193=z", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T00:06:24.687 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:06:24 smithi193 ceph-mon[36878]: Saving service mgr spec with placement smithi118=x;smithi189=y;smithi193=z;count:3 2024-03-24T00:06:24.687 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:06:24 smithi193 ceph-mon[36878]: Deploying daemon mgr.z on smithi193 2024-03-24T00:06:24.687 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:06:24 smithi193 ceph-mon[36878]: pgmap v33: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T00:06:24.695 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:06:24 smithi118 ceph-mon[31535]: from='client.24106 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "placement": "3;smithi118=x;smithi189=y;smithi193=z", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T00:06:24.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:06:24 smithi118 ceph-mon[31535]: Saving service mgr spec with placement smithi118=x;smithi189=y;smithi193=z;count:3 2024-03-24T00:06:24.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:06:24 smithi118 ceph-mon[31535]: Deploying daemon mgr.z on smithi193 2024-03-24T00:06:24.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:06:24 smithi118 ceph-mon[31535]: pgmap v33: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T00:06:24.701 INFO:teuthology.orchestra.run.smithi189.stderr:1+0 records in 2024-03-24T00:06:24.701 INFO:teuthology.orchestra.run.smithi189.stderr:1+0 records out 2024-03-24T00:06:24.701 INFO:teuthology.orchestra.run.smithi189.stderr:512 bytes copied, 0.000235596 s, 2.2 MB/s 2024-03-24T00:06:24.702 DEBUG:teuthology.orchestra.run.smithi189:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-03-24T00:06:24.759 DEBUG:teuthology.orchestra.run.smithi189:> stat /dev/vg_nvme/lv_3 2024-03-24T00:06:24.814 INFO:teuthology.orchestra.run.smithi189.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-03-24T00:06:24.814 INFO:teuthology.orchestra.run.smithi189.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-24T00:06:24.814 INFO:teuthology.orchestra.run.smithi189.stdout:Device: 5h/5d Inode: 699 Links: 1 2024-03-24T00:06:24.814 INFO:teuthology.orchestra.run.smithi189.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-24T00:06:24.814 INFO:teuthology.orchestra.run.smithi189.stdout:Context: system_u:object_r:device_t:s0 2024-03-24T00:06:24.814 INFO:teuthology.orchestra.run.smithi189.stdout:Access: 2024-03-24 00:06:07.843475215 +0000 2024-03-24T00:06:24.814 INFO:teuthology.orchestra.run.smithi189.stdout:Modify: 2024-03-23 23:58:31.368101088 +0000 2024-03-24T00:06:24.814 INFO:teuthology.orchestra.run.smithi189.stdout:Change: 2024-03-23 23:58:31.368101088 +0000 2024-03-24T00:06:24.814 INFO:teuthology.orchestra.run.smithi189.stdout: Birth: 2024-03-23 23:58:31.368101088 +0000 2024-03-24T00:06:24.815 DEBUG:teuthology.orchestra.run.smithi189:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-03-24T00:06:24.878 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:06:24 smithi189 ceph-mon[36187]: from='client.24106 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "placement": "3;smithi118=x;smithi189=y;smithi193=z", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T00:06:24.878 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:06:24 smithi189 ceph-mon[36187]: Saving service mgr spec with placement smithi118=x;smithi189=y;smithi193=z;count:3 2024-03-24T00:06:24.878 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:06:24 smithi189 ceph-mon[36187]: Deploying daemon mgr.z on smithi193 2024-03-24T00:06:24.878 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:06:24 smithi189 ceph-mon[36187]: pgmap v33: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T00:06:24.880 INFO:teuthology.orchestra.run.smithi189.stderr:1+0 records in 2024-03-24T00:06:24.880 INFO:teuthology.orchestra.run.smithi189.stderr:1+0 records out 2024-03-24T00:06:24.880 INFO:teuthology.orchestra.run.smithi189.stderr:512 bytes copied, 0.000212227 s, 2.4 MB/s 2024-03-24T00:06:24.881 DEBUG:teuthology.orchestra.run.smithi189:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-03-24T00:06:24.938 DEBUG:teuthology.orchestra.run.smithi189:> stat /dev/vg_nvme/lv_4 2024-03-24T00:06:24.961 INFO:journalctl@ceph.mgr.z.smithi193.stdout:Mar 24 00:06:24 smithi193 systemd[1]: Starting Ceph mgr.z for d79a5420-e971-11ee-95cd-87774f69a715... 2024-03-24T00:06:24.993 INFO:teuthology.orchestra.run.smithi189.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-03-24T00:06:24.993 INFO:teuthology.orchestra.run.smithi189.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-24T00:06:24.993 INFO:teuthology.orchestra.run.smithi189.stdout:Device: 5h/5d Inode: 720 Links: 1 2024-03-24T00:06:24.993 INFO:teuthology.orchestra.run.smithi189.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-24T00:06:24.993 INFO:teuthology.orchestra.run.smithi189.stdout:Context: system_u:object_r:device_t:s0 2024-03-24T00:06:24.993 INFO:teuthology.orchestra.run.smithi189.stdout:Access: 2024-03-24 00:06:07.843475215 +0000 2024-03-24T00:06:24.993 INFO:teuthology.orchestra.run.smithi189.stdout:Modify: 2024-03-23 23:58:31.700094467 +0000 2024-03-24T00:06:24.994 INFO:teuthology.orchestra.run.smithi189.stdout:Change: 2024-03-23 23:58:31.700094467 +0000 2024-03-24T00:06:24.994 INFO:teuthology.orchestra.run.smithi189.stdout: Birth: 2024-03-23 23:58:31.700094467 +0000 2024-03-24T00:06:24.994 DEBUG:teuthology.orchestra.run.smithi189:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-03-24T00:06:25.058 INFO:teuthology.orchestra.run.smithi189.stderr:1+0 records in 2024-03-24T00:06:25.058 INFO:teuthology.orchestra.run.smithi189.stderr:1+0 records out 2024-03-24T00:06:25.058 INFO:teuthology.orchestra.run.smithi189.stderr:512 bytes copied, 0.000243599 s, 2.1 MB/s 2024-03-24T00:06:25.059 DEBUG:teuthology.orchestra.run.smithi189:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-03-24T00:06:25.116 DEBUG:teuthology.orchestra.run.smithi193:> set -ex 2024-03-24T00:06:25.116 DEBUG:teuthology.orchestra.run.smithi193:> dd if=/scratch_devs of=/dev/stdout 2024-03-24T00:06:25.132 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-03-24T00:06:25.133 DEBUG:teuthology.orchestra.run.smithi193:> stat /dev/vg_nvme/lv_1 2024-03-24T00:06:25.187 INFO:teuthology.orchestra.run.smithi193.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-03-24T00:06:25.187 INFO:teuthology.orchestra.run.smithi193.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-24T00:06:25.187 INFO:teuthology.orchestra.run.smithi193.stdout:Device: 5h/5d Inode: 660 Links: 1 2024-03-24T00:06:25.187 INFO:teuthology.orchestra.run.smithi193.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-24T00:06:25.187 INFO:teuthology.orchestra.run.smithi193.stdout:Context: system_u:object_r:device_t:s0 2024-03-24T00:06:25.187 INFO:teuthology.orchestra.run.smithi193.stdout:Access: 2024-03-24 00:06:24.793023749 +0000 2024-03-24T00:06:25.187 INFO:teuthology.orchestra.run.smithi193.stdout:Modify: 2024-03-24 00:00:12.923209209 +0000 2024-03-24T00:06:25.187 INFO:teuthology.orchestra.run.smithi193.stdout:Change: 2024-03-24 00:00:12.923209209 +0000 2024-03-24T00:06:25.187 INFO:teuthology.orchestra.run.smithi193.stdout: Birth: 2024-03-24 00:00:12.923209209 +0000 2024-03-24T00:06:25.188 DEBUG:teuthology.orchestra.run.smithi193:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-03-24T00:06:25.253 INFO:teuthology.orchestra.run.smithi193.stderr:1+0 records in 2024-03-24T00:06:25.253 INFO:teuthology.orchestra.run.smithi193.stderr:1+0 records out 2024-03-24T00:06:25.253 INFO:teuthology.orchestra.run.smithi193.stderr:512 bytes copied, 0.000265539 s, 1.9 MB/s 2024-03-24T00:06:25.254 DEBUG:teuthology.orchestra.run.smithi193:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-03-24T00:06:25.311 DEBUG:teuthology.orchestra.run.smithi193:> stat /dev/vg_nvme/lv_2 2024-03-24T00:06:25.365 INFO:teuthology.orchestra.run.smithi193.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-03-24T00:06:25.365 INFO:teuthology.orchestra.run.smithi193.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-24T00:06:25.365 INFO:teuthology.orchestra.run.smithi193.stdout:Device: 5h/5d Inode: 680 Links: 1 2024-03-24T00:06:25.365 INFO:teuthology.orchestra.run.smithi193.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-24T00:06:25.365 INFO:teuthology.orchestra.run.smithi193.stdout:Context: system_u:object_r:device_t:s0 2024-03-24T00:06:25.366 INFO:teuthology.orchestra.run.smithi193.stdout:Access: 2024-03-24 00:06:24.793023749 +0000 2024-03-24T00:06:25.366 INFO:teuthology.orchestra.run.smithi193.stdout:Modify: 2024-03-24 00:00:13.278203517 +0000 2024-03-24T00:06:25.366 INFO:teuthology.orchestra.run.smithi193.stdout:Change: 2024-03-24 00:00:13.278203517 +0000 2024-03-24T00:06:25.366 INFO:teuthology.orchestra.run.smithi193.stdout: Birth: 2024-03-24 00:00:13.278203517 +0000 2024-03-24T00:06:25.366 DEBUG:teuthology.orchestra.run.smithi193:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-03-24T00:06:25.383 INFO:journalctl@ceph.mgr.z.smithi193.stdout:Mar 24 00:06:25 smithi193 podman[38690]: 2024-03-24 00:06:25.056585792 +0000 UTC m=+0.100648392 container create df9c7647eda091a2686a260871afc3ace7bcfccd6aa82915bada036e7b01a781 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-d79a5420-e971-11ee-95cd-87774f69a715-mgr-z, CEPH_POINT_RELEASE=, ceph=True, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, org.label-schema.build-date=20240319, maintainer=Guillaume Abrioux , GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_CLEAN=True, org.label-schema.license=GPLv2, io.buildah.version=1.33.5) 2024-03-24T00:06:25.383 INFO:journalctl@ceph.mgr.z.smithi193.stdout:Mar 24 00:06:25 smithi193 podman[38690]: 2024-03-24 00:06:24.973504588 +0000 UTC m=+0.017567203 image pull c664afe56bb01db4f9afa9532c20598e201e664f2e9ba2be50db0d52674a26d2 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc 2024-03-24T00:06:25.384 INFO:journalctl@ceph.mgr.z.smithi193.stdout:Mar 24 00:06:25 smithi193 podman[38690]: 2024-03-24 00:06:25.27162845 +0000 UTC m=+0.315691060 container init df9c7647eda091a2686a260871afc3ace7bcfccd6aa82915bada036e7b01a781 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-d79a5420-e971-11ee-95cd-87774f69a715-mgr-z, GIT_CLEAN=True, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, ceph=True, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240319) 2024-03-24T00:06:25.384 INFO:journalctl@ceph.mgr.z.smithi193.stdout:Mar 24 00:06:25 smithi193 podman[38690]: 2024-03-24 00:06:25.274261969 +0000 UTC m=+0.318324577 container start df9c7647eda091a2686a260871afc3ace7bcfccd6aa82915bada036e7b01a781 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-d79a5420-e971-11ee-95cd-87774f69a715-mgr-z, org.label-schema.schema-version=1.0, GIT_CLEAN=True, ceph=True, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.build-date=20240319, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux ) 2024-03-24T00:06:25.384 INFO:journalctl@ceph.mgr.z.smithi193.stdout:Mar 24 00:06:25 smithi193 ceph-mgr[38759]: -- 172.21.15.193:0/3285759416 <== mon.1 v2:172.21.15.193:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x556a8e0e65a0 con 0x556a8e0bac00 2024-03-24T00:06:25.411 INFO:teuthology.orchestra.run.smithi193.stderr:1+0 records in 2024-03-24T00:06:25.411 INFO:teuthology.orchestra.run.smithi193.stderr:1+0 records out 2024-03-24T00:06:25.411 INFO:teuthology.orchestra.run.smithi193.stderr:512 bytes copied, 0.00018468 s, 2.8 MB/s 2024-03-24T00:06:25.412 DEBUG:teuthology.orchestra.run.smithi193:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-03-24T00:06:25.468 DEBUG:teuthology.orchestra.run.smithi193:> stat /dev/vg_nvme/lv_3 2024-03-24T00:06:25.524 INFO:teuthology.orchestra.run.smithi193.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-03-24T00:06:25.524 INFO:teuthology.orchestra.run.smithi193.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-24T00:06:25.524 INFO:teuthology.orchestra.run.smithi193.stdout:Device: 5h/5d Inode: 700 Links: 1 2024-03-24T00:06:25.524 INFO:teuthology.orchestra.run.smithi193.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-24T00:06:25.524 INFO:teuthology.orchestra.run.smithi193.stdout:Context: system_u:object_r:device_t:s0 2024-03-24T00:06:25.524 INFO:teuthology.orchestra.run.smithi193.stdout:Access: 2024-03-24 00:06:24.794023733 +0000 2024-03-24T00:06:25.524 INFO:teuthology.orchestra.run.smithi193.stdout:Modify: 2024-03-24 00:00:13.601198338 +0000 2024-03-24T00:06:25.525 INFO:teuthology.orchestra.run.smithi193.stdout:Change: 2024-03-24 00:00:13.601198338 +0000 2024-03-24T00:06:25.525 INFO:teuthology.orchestra.run.smithi193.stdout: Birth: 2024-03-24 00:00:13.601198338 +0000 2024-03-24T00:06:25.525 DEBUG:teuthology.orchestra.run.smithi193:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-03-24T00:06:25.590 INFO:teuthology.orchestra.run.smithi193.stderr:1+0 records in 2024-03-24T00:06:25.591 INFO:teuthology.orchestra.run.smithi193.stderr:1+0 records out 2024-03-24T00:06:25.591 INFO:teuthology.orchestra.run.smithi193.stderr:512 bytes copied, 0.000222569 s, 2.3 MB/s 2024-03-24T00:06:25.592 DEBUG:teuthology.orchestra.run.smithi193:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-03-24T00:06:25.647 DEBUG:teuthology.orchestra.run.smithi193:> stat /dev/vg_nvme/lv_4 2024-03-24T00:06:25.702 INFO:teuthology.orchestra.run.smithi193.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-03-24T00:06:25.703 INFO:teuthology.orchestra.run.smithi193.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-24T00:06:25.703 INFO:teuthology.orchestra.run.smithi193.stdout:Device: 5h/5d Inode: 719 Links: 1 2024-03-24T00:06:25.703 INFO:teuthology.orchestra.run.smithi193.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-24T00:06:25.703 INFO:teuthology.orchestra.run.smithi193.stdout:Context: system_u:object_r:device_t:s0 2024-03-24T00:06:25.703 INFO:teuthology.orchestra.run.smithi193.stdout:Access: 2024-03-24 00:06:24.794023733 +0000 2024-03-24T00:06:25.703 INFO:teuthology.orchestra.run.smithi193.stdout:Modify: 2024-03-24 00:00:13.931193047 +0000 2024-03-24T00:06:25.703 INFO:teuthology.orchestra.run.smithi193.stdout:Change: 2024-03-24 00:00:13.931193047 +0000 2024-03-24T00:06:25.703 INFO:teuthology.orchestra.run.smithi193.stdout: Birth: 2024-03-24 00:00:13.931193047 +0000 2024-03-24T00:06:25.703 DEBUG:teuthology.orchestra.run.smithi193:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-03-24T00:06:25.710 INFO:journalctl@ceph.mgr.z.smithi193.stdout:Mar 24 00:06:25 smithi193 bash[38690]: df9c7647eda091a2686a260871afc3ace7bcfccd6aa82915bada036e7b01a781 2024-03-24T00:06:25.710 INFO:journalctl@ceph.mgr.z.smithi193.stdout:Mar 24 00:06:25 smithi193 systemd[1]: Started Ceph mgr.z for d79a5420-e971-11ee-95cd-87774f69a715. 2024-03-24T00:06:25.735 INFO:teuthology.orchestra.run.smithi193.stderr:1+0 records in 2024-03-24T00:06:25.735 INFO:teuthology.orchestra.run.smithi193.stderr:1+0 records out 2024-03-24T00:06:25.735 INFO:teuthology.orchestra.run.smithi193.stderr:512 bytes copied, 0.000226188 s, 2.3 MB/s 2024-03-24T00:06:25.736 DEBUG:teuthology.orchestra.run.smithi193:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-03-24T00:06:25.793 INFO:tasks.cephadm:Deploying osd.0 on smithi118 with /dev/vg_nvme/lv_4... 2024-03-24T00:06:25.793 DEBUG:teuthology.orchestra.run.smithi118:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d79a5420-e971-11ee-95cd-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_4 2024-03-24T00:06:26.210 INFO:journalctl@ceph.mgr.z.smithi193.stdout:Mar 24 00:06:25 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mgr-z[38721]: 2024-03-24T00:06:25.836+0000 7f45128bd200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-24T00:06:26.211 INFO:journalctl@ceph.mgr.z.smithi193.stdout:Mar 24 00:06:25 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mgr-z[38721]: 2024-03-24T00:06:25.928+0000 7f45128bd200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-24T00:06:26.230 INFO:teuthology.orchestra.run.smithi118.stderr:Inferring config /var/lib/ceph/d79a5420-e971-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T00:06:26.704 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:06:26 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T00:06:26.704 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:06:26 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T00:06:26.704 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:06:26 smithi189 ceph-mon[36187]: pgmap v34: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T00:06:26.704 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:06:26 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T00:06:26.704 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:06:26 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-24T00:06:26.704 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:06:26 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-03-24T00:06:26.704 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:06:26 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-03-24T00:06:26.705 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:06:26 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T00:06:26.705 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:06:26 smithi189 ceph-mon[36187]: Deploying daemon mgr.y on smithi189 2024-03-24T00:06:26.710 INFO:journalctl@ceph.mgr.z.smithi193.stdout:Mar 24 00:06:26 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mgr-z[38721]: 2024-03-24T00:06:26.567+0000 7f45128bd200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-24T00:06:26.710 INFO:journalctl@ceph.mgr.z.smithi193.stdout:Mar 24 00:06:26 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mgr-z[38721]: 2024-03-24T00:06:26.657+0000 7f45128bd200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-24T00:06:26.711 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:06:26 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T00:06:26.711 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:06:26 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T00:06:26.711 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:06:26 smithi193 ceph-mon[36878]: pgmap v34: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T00:06:26.711 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:06:26 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T00:06:26.711 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:06:26 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-24T00:06:26.711 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:06:26 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-03-24T00:06:26.711 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:06:26 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-03-24T00:06:26.711 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:06:26 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T00:06:26.711 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:06:26 smithi193 ceph-mon[36878]: Deploying daemon mgr.y on smithi189 2024-03-24T00:06:26.734 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:06:26 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T00:06:26.734 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:06:26 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T00:06:26.734 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:06:26 smithi118 ceph-mon[31535]: pgmap v34: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T00:06:26.734 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:06:26 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T00:06:26.734 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:06:26 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-24T00:06:26.734 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:06:26 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.y", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-03-24T00:06:26.734 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:06:26 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-03-24T00:06:26.734 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:06:26 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T00:06:26.734 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:06:26 smithi118 ceph-mon[31535]: Deploying daemon mgr.y on smithi189 2024-03-24T00:06:27.213 INFO:journalctl@ceph.mgr.z.smithi193.stdout:Mar 24 00:06:26 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mgr-z[38721]: 2024-03-24T00:06:26.797+0000 7f45128bd200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-24T00:06:27.213 INFO:journalctl@ceph.mgr.z.smithi193.stdout:Mar 24 00:06:26 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mgr-z[38721]: 2024-03-24T00:06:26.893+0000 7f45128bd200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-24T00:06:27.213 INFO:journalctl@ceph.mgr.z.smithi193.stdout:Mar 24 00:06:26 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mgr-z[38721]: 2024-03-24T00:06:26.973+0000 7f45128bd200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-24T00:06:27.516 INFO:journalctl@ceph.mgr.z.smithi193.stdout:Mar 24 00:06:27 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mgr-z[38721]: 2024-03-24T00:06:27.232+0000 7f45128bd200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-03-24T00:06:27.516 INFO:journalctl@ceph.mgr.z.smithi193.stdout:Mar 24 00:06:27 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mgr-z[38721]: 2024-03-24T00:06:27.339+0000 7f45128bd200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-24T00:06:27.516 INFO:journalctl@ceph.mgr.z.smithi193.stdout:Mar 24 00:06:27 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mgr-z[38721]: 2024-03-24T00:06:27.423+0000 7f45128bd200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-24T00:06:27.516 INFO:journalctl@ceph.mgr.z.smithi193.stdout:Mar 24 00:06:27 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mgr-z[38721]: 2024-03-24T00:06:27.516+0000 7f45128bd200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-24T00:06:27.683 INFO:journalctl@ceph.mgr.y.smithi189.stdout:Mar 24 00:06:27 smithi189 systemd[1]: Starting Ceph mgr.y for d79a5420-e971-11ee-95cd-87774f69a715... 2024-03-24T00:06:27.954 INFO:journalctl@ceph.mgr.y.smithi189.stdout:Mar 24 00:06:27 smithi189 podman[37590]: 2024-03-24 00:06:27.683859271 +0000 UTC m=+0.088251221 container create 93099988c06e582719019354c097b6472802e97fa46a70b36b7d77cbad9480b3 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-d79a5420-e971-11ee-95cd-87774f69a715-mgr-y, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, ceph=True, org.label-schema.build-date=20240319, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, GIT_CLEAN=True) 2024-03-24T00:06:27.954 INFO:journalctl@ceph.mgr.y.smithi189.stdout:Mar 24 00:06:27 smithi189 podman[37590]: 2024-03-24 00:06:27.612947788 +0000 UTC m=+0.017339762 image pull c664afe56bb01db4f9afa9532c20598e201e664f2e9ba2be50db0d52674a26d2 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc 2024-03-24T00:06:27.954 INFO:journalctl@ceph.mgr.y.smithi189.stdout:Mar 24 00:06:27 smithi189 podman[37590]: 2024-03-24 00:06:27.894472912 +0000 UTC m=+0.298864871 container init 93099988c06e582719019354c097b6472802e97fa46a70b36b7d77cbad9480b3 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-d79a5420-e971-11ee-95cd-87774f69a715-mgr-y, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, org.label-schema.vendor=CentOS, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.build-date=20240319, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, ceph=True) 2024-03-24T00:06:27.954 INFO:journalctl@ceph.mgr.y.smithi189.stdout:Mar 24 00:06:27 smithi189 podman[37590]: 2024-03-24 00:06:27.896922566 +0000 UTC m=+0.301314522 container start 93099988c06e582719019354c097b6472802e97fa46a70b36b7d77cbad9480b3 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-d79a5420-e971-11ee-95cd-87774f69a715-mgr-y, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, GIT_CLEAN=True, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, org.label-schema.build-date=20240319, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True) 2024-03-24T00:06:27.954 INFO:journalctl@ceph.mgr.y.smithi189.stdout:Mar 24 00:06:27 smithi189 ceph-mgr[37609]: -- 172.21.15.189:0/3514846588 <== mon.2 v2:172.21.15.189:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x55ee400585a0 con 0x55ee4002cc00 2024-03-24T00:06:27.960 INFO:journalctl@ceph.mgr.z.smithi193.stdout:Mar 24 00:06:27 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mgr-z[38721]: 2024-03-24T00:06:27.693+0000 7f45128bd200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-24T00:06:28.453 INFO:journalctl@ceph.mgr.y.smithi189.stdout:Mar 24 00:06:27 smithi189 bash[37590]: 93099988c06e582719019354c097b6472802e97fa46a70b36b7d77cbad9480b3 2024-03-24T00:06:28.454 INFO:journalctl@ceph.mgr.y.smithi189.stdout:Mar 24 00:06:27 smithi189 systemd[1]: Started Ceph mgr.y for d79a5420-e971-11ee-95cd-87774f69a715. 2024-03-24T00:06:28.807 INFO:journalctl@ceph.mgr.z.smithi193.stdout:Mar 24 00:06:28 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mgr-z[38721]: 2024-03-24T00:06:28.672+0000 7f45128bd200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-03-24T00:06:28.807 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:06:28 smithi193 ceph-mon[36878]: pgmap v35: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T00:06:28.807 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:06:28 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T00:06:28.807 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:06:28 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T00:06:28.807 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:06:28 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T00:06:28.807 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:06:28 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T00:06:28.808 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:06:28 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T00:06:28.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:06:28 smithi118 ceph-mon[31535]: pgmap v35: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T00:06:28.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:06:28 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T00:06:28.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:06:28 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T00:06:28.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:06:28 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T00:06:28.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:06:28 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T00:06:28.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:06:28 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T00:06:28.953 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:06:28 smithi189 ceph-mon[36187]: pgmap v35: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T00:06:28.953 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:06:28 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T00:06:28.953 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:06:28 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T00:06:28.953 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:06:28 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T00:06:28.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:06:28 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T00:06:28.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:06:28 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T00:06:28.954 INFO:journalctl@ceph.mgr.y.smithi189.stdout:Mar 24 00:06:28 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mgr-y[37605]: 2024-03-24T00:06:28.458+0000 7fc354ac5200 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-03-24T00:06:28.954 INFO:journalctl@ceph.mgr.y.smithi189.stdout:Mar 24 00:06:28 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mgr-y[37605]: 2024-03-24T00:06:28.556+0000 7fc354ac5200 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-03-24T00:06:28.966 INFO:teuthology.orchestra.run.smithi118.stderr:--> Zapping: /dev/vg_nvme/lv_4 2024-03-24T00:06:28.966 INFO:teuthology.orchestra.run.smithi118.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_4 bs=1M count=10 conv=fsync 2024-03-24T00:06:28.966 INFO:teuthology.orchestra.run.smithi118.stderr: stderr: 10+0 records in 2024-03-24T00:06:28.966 INFO:teuthology.orchestra.run.smithi118.stderr:10+0 records out 2024-03-24T00:06:28.966 INFO:teuthology.orchestra.run.smithi118.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0379618 s, 276 MB/s 2024-03-24T00:06:28.966 INFO:teuthology.orchestra.run.smithi118.stderr:--> Zapping successful for: 2024-03-24T00:06:29.148 INFO:journalctl@ceph.mgr.z.smithi193.stdout:Mar 24 00:06:28 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mgr-z[38721]: 2024-03-24T00:06:28.807+0000 7f45128bd200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-24T00:06:29.449 INFO:journalctl@ceph.mgr.z.smithi193.stdout:Mar 24 00:06:29 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mgr-z[38721]: 2024-03-24T00:06:29.159+0000 7f45128bd200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-24T00:06:29.449 INFO:journalctl@ceph.mgr.z.smithi193.stdout:Mar 24 00:06:29 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mgr-z[38721]: 2024-03-24T00:06:29.252+0000 7f45128bd200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-24T00:06:29.449 INFO:journalctl@ceph.mgr.z.smithi193.stdout:Mar 24 00:06:29 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mgr-z[38721]: 2024-03-24T00:06:29.448+0000 7f45128bd200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-24T00:06:29.453 INFO:journalctl@ceph.mgr.y.smithi189.stdout:Mar 24 00:06:29 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mgr-y[37605]: 2024-03-24T00:06:29.235+0000 7fc354ac5200 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-03-24T00:06:29.453 INFO:journalctl@ceph.mgr.y.smithi189.stdout:Mar 24 00:06:29 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mgr-y[37605]: 2024-03-24T00:06:29.335+0000 7fc354ac5200 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-03-24T00:06:29.710 INFO:journalctl@ceph.mgr.z.smithi193.stdout:Mar 24 00:06:29 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mgr-z[38721]: 2024-03-24T00:06:29.656+0000 7f45128bd200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-24T00:06:29.813 DEBUG:teuthology.orchestra.run.smithi118:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d79a5420-e971-11ee-95cd-87774f69a715 -- ceph orch daemon add osd smithi118:vg_nvme/lv_4 2024-03-24T00:06:29.933 INFO:journalctl@ceph.mgr.y.smithi189.stdout:Mar 24 00:06:29 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mgr-y[37605]: 2024-03-24T00:06:29.479+0000 7fc354ac5200 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-03-24T00:06:29.933 INFO:journalctl@ceph.mgr.y.smithi189.stdout:Mar 24 00:06:29 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mgr-y[37605]: 2024-03-24T00:06:29.579+0000 7fc354ac5200 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-03-24T00:06:29.934 INFO:journalctl@ceph.mgr.y.smithi189.stdout:Mar 24 00:06:29 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mgr-y[37605]: 2024-03-24T00:06:29.668+0000 7fc354ac5200 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-03-24T00:06:29.934 INFO:journalctl@ceph.mgr.y.smithi189.stdout:Mar 24 00:06:29 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mgr-y[37605]: 2024-03-24T00:06:29.934+0000 7fc354ac5200 -1 mgr[py] Module rgw has missing NOTIFY_TYPES member 2024-03-24T00:06:30.195 INFO:journalctl@ceph.mgr.z.smithi193.stdout:Mar 24 00:06:29 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mgr-z[38721]: 2024-03-24T00:06:29.939+0000 7f45128bd200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-24T00:06:30.195 INFO:journalctl@ceph.mgr.z.smithi193.stdout:Mar 24 00:06:30 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mgr-z[38721]: 2024-03-24T00:06:30.195+0000 7f45128bd200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-24T00:06:30.203 INFO:journalctl@ceph.mgr.y.smithi189.stdout:Mar 24 00:06:30 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mgr-y[37605]: 2024-03-24T00:06:30.043+0000 7fc354ac5200 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-03-24T00:06:30.203 INFO:journalctl@ceph.mgr.y.smithi189.stdout:Mar 24 00:06:30 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mgr-y[37605]: 2024-03-24T00:06:30.128+0000 7fc354ac5200 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-03-24T00:06:30.223 INFO:teuthology.orchestra.run.smithi118.stderr:Inferring config /var/lib/ceph/d79a5420-e971-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T00:06:30.460 INFO:journalctl@ceph.mgr.z.smithi193.stdout:Mar 24 00:06:30 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mgr-z[38721]: 2024-03-24T00:06:30.422+0000 7f45128bd200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-24T00:06:30.626 INFO:journalctl@ceph.mgr.y.smithi189.stdout:Mar 24 00:06:30 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mgr-y[37605]: 2024-03-24T00:06:30.223+0000 7fc354ac5200 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-03-24T00:06:30.626 INFO:journalctl@ceph.mgr.y.smithi189.stdout:Mar 24 00:06:30 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mgr-y[37605]: 2024-03-24T00:06:30.398+0000 7fc354ac5200 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-03-24T00:06:30.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:06:30 smithi118 ceph-mon[31535]: pgmap v36: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T00:06:30.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:06:30 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T00:06:30.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:06:30 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T00:06:30.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:06:30 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T00:06:30.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:06:30 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T00:06:30.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:06:30 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T00:06:30.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:06:30 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T00:06:30.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:06:30 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T00:06:30.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:06:30 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-24T00:06:30.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:06:30 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-03-24T00:06:30.947 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:06:30 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T00:06:30.947 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:06:30 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T00:06:30.953 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:06:30 smithi189 ceph-mon[36187]: pgmap v36: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T00:06:30.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:06:30 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T00:06:30.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:06:30 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T00:06:30.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:06:30 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T00:06:30.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:06:30 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T00:06:30.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:06:30 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T00:06:30.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:06:30 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T00:06:30.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:06:30 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T00:06:30.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:06:30 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-24T00:06:30.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:06:30 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-03-24T00:06:30.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:06:30 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T00:06:30.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:06:30 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T00:06:30.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:06:30 smithi193 ceph-mon[36878]: pgmap v36: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T00:06:30.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:06:30 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T00:06:30.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:06:30 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T00:06:30.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:06:30 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T00:06:30.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:06:30 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T00:06:30.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:06:30 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T00:06:30.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:06:30 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T00:06:30.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:06:30 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T00:06:30.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:06:30 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.x", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-24T00:06:30.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:06:30 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "mgr services"}]: dispatch 2024-03-24T00:06:30.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:06:30 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T00:06:30.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:06:30 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T00:06:31.460 INFO:journalctl@ceph.mgr.z.smithi193.stdout:Mar 24 00:06:31 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mgr-z[38721]: 2024-03-24T00:06:31.198+0000 7f45128bd200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-24T00:06:31.631 INFO:journalctl@ceph.mgr.y.smithi189.stdout:Mar 24 00:06:31 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mgr-y[37605]: 2024-03-24T00:06:31.365+0000 7fc354ac5200 -1 mgr[py] Module rook has missing NOTIFY_TYPES member 2024-03-24T00:06:31.631 INFO:journalctl@ceph.mgr.y.smithi189.stdout:Mar 24 00:06:31 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mgr-y[37605]: 2024-03-24T00:06:31.495+0000 7fc354ac5200 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-03-24T00:06:31.670 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:06:31 smithi118 ceph-mon[31535]: Reconfiguring mgr.x (unknown last config time)... 2024-03-24T00:06:31.670 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:06:31 smithi118 ceph-mon[31535]: Reconfiguring daemon mgr.x on smithi118 2024-03-24T00:06:31.917 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:06:31 smithi189 ceph-mon[36187]: Reconfiguring mgr.x (unknown last config time)... 2024-03-24T00:06:31.917 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:06:31 smithi189 ceph-mon[36187]: Reconfiguring daemon mgr.x on smithi118 2024-03-24T00:06:31.917 INFO:journalctl@ceph.mgr.y.smithi189.stdout:Mar 24 00:06:31 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mgr-y[37605]: 2024-03-24T00:06:31.829+0000 7fc354ac5200 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-03-24T00:06:31.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:06:31 smithi193 ceph-mon[36878]: Reconfiguring mgr.x (unknown last config time)... 2024-03-24T00:06:31.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:06:31 smithi193 ceph-mon[36878]: Reconfiguring daemon mgr.x on smithi118 2024-03-24T00:06:31.961 INFO:journalctl@ceph.mgr.z.smithi193.stdout:Mar 24 00:06:31 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mgr-z[38721]: 2024-03-24T00:06:31.758+0000 7f45128bd200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-24T00:06:31.961 INFO:journalctl@ceph.mgr.z.smithi193.stdout:Mar 24 00:06:31 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mgr-z[38721]: 2024-03-24T00:06:31.851+0000 7f45128bd200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-24T00:06:32.203 INFO:journalctl@ceph.mgr.y.smithi189.stdout:Mar 24 00:06:31 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mgr-y[37605]: 2024-03-24T00:06:31.917+0000 7fc354ac5200 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-03-24T00:06:32.203 INFO:journalctl@ceph.mgr.y.smithi189.stdout:Mar 24 00:06:32 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mgr-y[37605]: 2024-03-24T00:06:32.111+0000 7fc354ac5200 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-03-24T00:06:32.460 INFO:journalctl@ceph.mgr.z.smithi193.stdout:Mar 24 00:06:32 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mgr-z[38721]: 2024-03-24T00:06:32.059+0000 7f45128bd200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-24T00:06:32.575 INFO:journalctl@ceph.mgr.y.smithi189.stdout:Mar 24 00:06:32 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mgr-y[37605]: 2024-03-24T00:06:32.293+0000 7fc354ac5200 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-03-24T00:06:32.658 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:06:32.657+0000 7f19e1c00700 1 -- 172.21.15.118:0/769237357 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f19dc0761c0 msgr2=0x7f19dc074620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:06:32.658 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:06:32.657+0000 7f19e1c00700 1 --2- 172.21.15.118:0/769237357 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f19dc0761c0 0x7f19dc074620 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f19cc0098f0 tx=0x7f19cc009c00 comp rx=0 tx=0).stop 2024-03-24T00:06:32.658 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:06:32.657+0000 7f19e1c00700 1 -- 172.21.15.118:0/769237357 shutdown_connections 2024-03-24T00:06:32.658 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:06:32.657+0000 7f19e1c00700 1 --2- 172.21.15.118:0/769237357 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f19dc0761c0 0x7f19dc074620 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:06:32.658 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:06:32.657+0000 7f19e1c00700 1 --2- 172.21.15.118:0/769237357 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f19dc10a0a0 0x7f19dc10c530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:06:32.658 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:06:32.657+0000 7f19e1c00700 1 --2- 172.21.15.118:0/769237357 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f19dc074bf0 0x7f19dc075070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:06:32.659 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:06:32.657+0000 7f19e1c00700 1 -- 172.21.15.118:0/769237357 >> 172.21.15.118:0/769237357 conn(0x7f19dc0fd730 msgr2=0x7f19dc0ffbb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:06:32.659 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:06:32.657+0000 7f19e1c00700 1 -- 172.21.15.118:0/769237357 shutdown_connections 2024-03-24T00:06:32.659 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:06:32.657+0000 7f19e1c00700 1 -- 172.21.15.118:0/769237357 wait complete. 2024-03-24T00:06:32.659 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:06:32.658+0000 7f19e1c00700 1 Processor -- start 2024-03-24T00:06:32.659 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:06:32.659+0000 7f19e1c00700 1 -- start start 2024-03-24T00:06:32.660 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:06:32.659+0000 7f19e1c00700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f19dc074bf0 0x7f19dc19a700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:06:32.661 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:06:32.659+0000 7f19e1c00700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f19dc0761c0 0x7f19dc19ac40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:06:32.661 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:06:32.659+0000 7f19e1c00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f19dc10a0a0 0x7f19dc1a1c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:06:32.661 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:06:32.659+0000 7f19e1c00700 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f19dc110000 con 0x7f19dc0761c0 2024-03-24T00:06:32.661 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:06:32.659+0000 7f19e1c00700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f19dc10fe80 con 0x7f19dc10a0a0 2024-03-24T00:06:32.661 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:06:32.659+0000 7f19e1c00700 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f19dc110180 con 0x7f19dc074bf0 2024-03-24T00:06:32.661 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:06:32.659+0000 7f19daa00700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f19dc074bf0 0x7f19dc19a700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:06:32.661 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:06:32.659+0000 7f19da000700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f19dc0761c0 0x7f19dc19ac40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:06:32.661 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:06:32.659+0000 7f19daa00700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f19dc074bf0 0x7f19dc19a700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.118:51864/0 (socket says 172.21.15.118:51864) 2024-03-24T00:06:32.661 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:06:32.659+0000 7f19daa00700 1 -- 172.21.15.118:0/2582164806 learned_addr learned my addr 172.21.15.118:0/2582164806 (peer_addr_for_me v2:172.21.15.118:0/0) 2024-03-24T00:06:32.661 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:06:32.659+0000 7f19db400700 1 --2- 172.21.15.118:0/2582164806 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f19dc10a0a0 0x7f19dc1a1c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:06:32.661 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:06:32.660+0000 7f19daa00700 1 -- 172.21.15.118:0/2582164806 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f19dc10a0a0 msgr2=0x7f19dc1a1c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:06:32.661 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:06:32.660+0000 7f19daa00700 1 --2- 172.21.15.118:0/2582164806 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f19dc10a0a0 0x7f19dc1a1c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:06:32.662 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:06:32.660+0000 7f19daa00700 1 -- 172.21.15.118:0/2582164806 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f19dc0761c0 msgr2=0x7f19dc19ac40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:06:32.662 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:06:32.660+0000 7f19daa00700 1 --2- 172.21.15.118:0/2582164806 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f19dc0761c0 0x7f19dc19ac40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:06:32.662 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:06:32.660+0000 7f19daa00700 1 -- 172.21.15.118:0/2582164806 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f19dc1a2310 con 0x7f19dc074bf0 2024-03-24T00:06:32.662 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:06:32.660+0000 7f19db400700 1 --2- 172.21.15.118:0/2582164806 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f19dc10a0a0 0x7f19dc1a1c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T00:06:32.662 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:06:32.660+0000 7f19daa00700 1 --2- 172.21.15.118:0/2582164806 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f19dc074bf0 0x7f19dc19a700 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f19cc00fd50 tx=0x7f19cc005040 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:06:32.662 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:06:32.660+0000 7f19d3400700 1 -- 172.21.15.118:0/2582164806 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f19cc027070 con 0x7f19dc074bf0 2024-03-24T00:06:32.662 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:06:32.660+0000 7f19d3400700 1 -- 172.21.15.118:0/2582164806 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f19cc00b560 con 0x7f19dc074bf0 2024-03-24T00:06:32.662 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:06:32.660+0000 7f19d3400700 1 -- 172.21.15.118:0/2582164806 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f19cc004bb0 con 0x7f19dc074bf0 2024-03-24T00:06:32.662 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:06:32.660+0000 7f19e1c00700 1 -- 172.21.15.118:0/2582164806 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f19dc1a25a0 con 0x7f19dc074bf0 2024-03-24T00:06:32.662 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:06:32.661+0000 7f19e1c00700 1 -- 172.21.15.118:0/2582164806 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f19dc1a2a50 con 0x7f19dc074bf0 2024-03-24T00:06:32.663 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:06:32.662+0000 7f19e1c00700 1 -- 172.21.15.118:0/2582164806 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f19dc068e50 con 0x7f19dc074bf0 2024-03-24T00:06:32.663 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:06:32.663+0000 7f19d3400700 1 -- 172.21.15.118:0/2582164806 <== mon.1 v2:172.21.15.193:3300/0 4 ==== mgrmap(e 14) v1 ==== 91699+0+0 (secure 0 0 0) 0x7f19cc02a070 con 0x7f19dc074bf0 2024-03-24T00:06:32.663 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:06:32.663+0000 7f19d3400700 1 --2- 172.21.15.118:0/2582164806 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7f19c006dd00 0x7f19c00701c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:06:32.663 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:06:32.664+0000 7f19d3400700 1 -- 172.21.15.118:0/2582164806 <== mon.1 v2:172.21.15.193:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1155+0+0 (secure 0 0 0) 0x7f19cc08d2c0 con 0x7f19dc074bf0 2024-03-24T00:06:32.664 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:06:32.664+0000 7f19da000700 1 --2- 172.21.15.118:0/2582164806 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7f19c006dd00 0x7f19c00701c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:06:32.664 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:06:32.664+0000 7f19da000700 1 --2- 172.21.15.118:0/2582164806 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7f19c006dd00 0x7f19c00701c0 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f19dc075ef0 tx=0x7f19c8009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:06:32.668 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:06:32.669+0000 7f19d3400700 1 -- 172.21.15.118:0/2582164806 <== mon.1 v2:172.21.15.193:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f19cc013030 con 0x7f19dc074bf0 2024-03-24T00:06:32.838 INFO:journalctl@ceph.mgr.y.smithi189.stdout:Mar 24 00:06:32 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mgr-y[37605]: 2024-03-24T00:06:32.574+0000 7fc354ac5200 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-03-24T00:06:32.838 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:06:32 smithi189 ceph-mon[36187]: pgmap v37: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T00:06:32.838 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:06:32 smithi189 ceph-mon[36187]: from='mgr.? 172.21.15.193:0/2490271273' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/crt"}]: dispatch 2024-03-24T00:06:32.838 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:06:32 smithi189 ceph-mon[36187]: Standby manager daemon z started 2024-03-24T00:06:32.838 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:06:32 smithi189 ceph-mon[36187]: from='mgr.? 172.21.15.193:0/2490271273' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-24T00:06:32.838 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:06:32 smithi189 ceph-mon[36187]: from='mgr.? 172.21.15.193:0/2490271273' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/key"}]: dispatch 2024-03-24T00:06:32.838 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:06:32 smithi189 ceph-mon[36187]: from='mgr.? 172.21.15.193:0/2490271273' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-24T00:06:32.902 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:06:32.902+0000 7f19e1c00700 1 -- 172.21.15.118:0/2582164806 --> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi118:vg_nvme/lv_4", "target": ["mon-mgr", ""]}) v1 -- 0x7f19dc0625d0 con 0x7f19c006dd00 2024-03-24T00:06:32.902 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:06:32 smithi118 ceph-mon[31535]: pgmap v37: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T00:06:32.902 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:06:32 smithi118 ceph-mon[31535]: from='mgr.? 172.21.15.193:0/2490271273' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/crt"}]: dispatch 2024-03-24T00:06:32.902 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:06:32 smithi118 ceph-mon[31535]: Standby manager daemon z started 2024-03-24T00:06:32.902 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:06:32 smithi118 ceph-mon[31535]: from='mgr.? 172.21.15.193:0/2490271273' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-24T00:06:32.902 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:06:32 smithi118 ceph-mon[31535]: from='mgr.? 172.21.15.193:0/2490271273' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/key"}]: dispatch 2024-03-24T00:06:32.903 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:06:32 smithi118 ceph-mon[31535]: from='mgr.? 172.21.15.193:0/2490271273' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-24T00:06:32.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:06:32 smithi193 ceph-mon[36878]: pgmap v37: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T00:06:32.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:06:32 smithi193 ceph-mon[36878]: from='mgr.? 172.21.15.193:0/2490271273' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/crt"}]: dispatch 2024-03-24T00:06:32.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:06:32 smithi193 ceph-mon[36878]: Standby manager daemon z started 2024-03-24T00:06:32.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:06:32 smithi193 ceph-mon[36878]: from='mgr.? 172.21.15.193:0/2490271273' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-24T00:06:32.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:06:32 smithi193 ceph-mon[36878]: from='mgr.? 172.21.15.193:0/2490271273' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/z/key"}]: dispatch 2024-03-24T00:06:32.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:06:32 smithi193 ceph-mon[36878]: from='mgr.? 172.21.15.193:0/2490271273' entity='mgr.z' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-24T00:06:33.203 INFO:journalctl@ceph.mgr.y.smithi189.stdout:Mar 24 00:06:32 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mgr-y[37605]: 2024-03-24T00:06:32.837+0000 7fc354ac5200 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-03-24T00:06:33.203 INFO:journalctl@ceph.mgr.y.smithi189.stdout:Mar 24 00:06:33 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mgr-y[37605]: 2024-03-24T00:06:33.070+0000 7fc354ac5200 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-03-24T00:06:33.941 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:06:33 smithi118 ceph-mon[31535]: mgrmap e14: x(active, since 87s), standbys: z 2024-03-24T00:06:33.941 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:06:33 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "z", "id": "z"}]: dispatch 2024-03-24T00:06:33.941 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:06:33 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-24T00:06:33.941 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:06:33 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-24T00:06:33.941 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:06:33 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T00:06:33.941 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:06:33 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T00:06:33.941 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:06:33 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T00:06:33.942 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:06:33 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T00:06:33.942 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:06:33 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T00:06:33.942 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:06:33 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T00:06:33.942 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:06:33 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T00:06:33.953 INFO:journalctl@ceph.mgr.y.smithi189.stdout:Mar 24 00:06:33 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mgr-y[37605]: 2024-03-24T00:06:33.868+0000 7fc354ac5200 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-03-24T00:06:33.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:06:33 smithi189 ceph-mon[36187]: mgrmap e14: x(active, since 87s), standbys: z 2024-03-24T00:06:33.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:06:33 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "z", "id": "z"}]: dispatch 2024-03-24T00:06:33.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:06:33 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-24T00:06:33.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:06:33 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-24T00:06:33.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:06:33 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T00:06:33.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:06:33 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T00:06:33.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:06:33 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T00:06:33.955 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:06:33 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T00:06:33.955 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:06:33 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T00:06:33.955 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:06:33 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T00:06:33.955 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:06:33 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T00:06:33.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:06:33 smithi193 ceph-mon[36878]: mgrmap e14: x(active, since 87s), standbys: z 2024-03-24T00:06:33.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:06:33 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "z", "id": "z"}]: dispatch 2024-03-24T00:06:33.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:06:33 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-24T00:06:33.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:06:33 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-24T00:06:33.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:06:33 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T00:06:33.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:06:33 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T00:06:33.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:06:33 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T00:06:33.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:06:33 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T00:06:33.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:06:33 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T00:06:33.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:06:33 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T00:06:33.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:06:33 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T00:06:34.703 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:06:34 smithi189 ceph-mon[36187]: from='client.24124 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi118:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T00:06:34.703 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:06:34 smithi189 ceph-mon[36187]: pgmap v38: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T00:06:34.704 INFO:journalctl@ceph.mgr.y.smithi189.stdout:Mar 24 00:06:34 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mgr-y[37605]: 2024-03-24T00:06:34.436+0000 7fc354ac5200 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-03-24T00:06:34.704 INFO:journalctl@ceph.mgr.y.smithi189.stdout:Mar 24 00:06:34 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mgr-y[37605]: 2024-03-24T00:06:34.534+0000 7fc354ac5200 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-03-24T00:06:34.842 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:06:34 smithi118 ceph-mon[31535]: from='client.24124 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi118:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T00:06:34.843 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:06:34 smithi118 ceph-mon[31535]: pgmap v38: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T00:06:34.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:06:34 smithi193 ceph-mon[36878]: from='client.24124 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi118:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T00:06:34.962 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:06:34 smithi193 ceph-mon[36878]: pgmap v38: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T00:06:35.204 INFO:journalctl@ceph.mgr.y.smithi189.stdout:Mar 24 00:06:34 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mgr-y[37605]: 2024-03-24T00:06:34.749+0000 7fc354ac5200 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-03-24T00:06:35.668 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:06:35.668+0000 7f19d3400700 1 -- 172.21.15.118:0/2582164806 <== mon.1 v2:172.21.15.193:3300/0 7 ==== mgrmap(e 15) v1 ==== 137235+0+0 (secure 0 0 0) 0x7f19cc058150 con 0x7f19dc074bf0 2024-03-24T00:06:35.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:06:35 smithi118 ceph-mon[31535]: Standby manager daemon y started 2024-03-24T00:06:35.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:06:35 smithi118 ceph-mon[31535]: from='mgr.? 172.21.15.189:0/3322180047' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/crt"}]: dispatch 2024-03-24T00:06:35.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:06:35 smithi118 ceph-mon[31535]: from='mgr.? 172.21.15.189:0/3322180047' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-24T00:06:35.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:06:35 smithi118 ceph-mon[31535]: from='mgr.? 172.21.15.189:0/3322180047' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/key"}]: dispatch 2024-03-24T00:06:35.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:06:35 smithi118 ceph-mon[31535]: from='mgr.? 172.21.15.189:0/3322180047' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-24T00:06:35.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:06:35 smithi189 ceph-mon[36187]: Standby manager daemon y started 2024-03-24T00:06:35.955 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:06:35 smithi189 ceph-mon[36187]: from='mgr.? 172.21.15.189:0/3322180047' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/crt"}]: dispatch 2024-03-24T00:06:35.955 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:06:35 smithi189 ceph-mon[36187]: from='mgr.? 172.21.15.189:0/3322180047' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-24T00:06:35.955 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:06:35 smithi189 ceph-mon[36187]: from='mgr.? 172.21.15.189:0/3322180047' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/key"}]: dispatch 2024-03-24T00:06:35.955 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:06:35 smithi189 ceph-mon[36187]: from='mgr.? 172.21.15.189:0/3322180047' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-24T00:06:35.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:06:35 smithi193 ceph-mon[36878]: Standby manager daemon y started 2024-03-24T00:06:35.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:06:35 smithi193 ceph-mon[36878]: from='mgr.? 172.21.15.189:0/3322180047' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/crt"}]: dispatch 2024-03-24T00:06:35.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:06:35 smithi193 ceph-mon[36878]: from='mgr.? 172.21.15.189:0/3322180047' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-24T00:06:35.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:06:35 smithi193 ceph-mon[36878]: from='mgr.? 172.21.15.189:0/3322180047' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/y/key"}]: dispatch 2024-03-24T00:06:35.962 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:06:35 smithi193 ceph-mon[36878]: from='mgr.? 172.21.15.189:0/3322180047' entity='mgr.y' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-24T00:06:36.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:06:36 smithi118 ceph-mon[31535]: pgmap v39: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T00:06:36.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:06:36 smithi118 ceph-mon[31535]: mgrmap e15: x(active, since 90s), standbys: z, y 2024-03-24T00:06:36.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:06:36 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-03-24T00:06:36.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:06:36 smithi118 ceph-mon[31535]: from='client.? 172.21.15.118:0/2661353918' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "c9590426-3901-4f68-a5ef-d57aa8fbac92"}]: dispatch 2024-03-24T00:06:36.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:06:36 smithi118 ceph-mon[31535]: from='client.? 172.21.15.118:0/2661353918' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "c9590426-3901-4f68-a5ef-d57aa8fbac92"}]': finished 2024-03-24T00:06:36.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:06:36 smithi118 ceph-mon[31535]: osdmap e5: 1 total, 0 up, 1 in 2024-03-24T00:06:36.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:06:36 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-24T00:06:36.953 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:06:36 smithi189 ceph-mon[36187]: pgmap v39: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T00:06:36.953 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:06:36 smithi189 ceph-mon[36187]: mgrmap e15: x(active, since 90s), standbys: z, y 2024-03-24T00:06:36.953 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:06:36 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-03-24T00:06:36.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:06:36 smithi189 ceph-mon[36187]: from='client.? 172.21.15.118:0/2661353918' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "c9590426-3901-4f68-a5ef-d57aa8fbac92"}]: dispatch 2024-03-24T00:06:36.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:06:36 smithi189 ceph-mon[36187]: from='client.? 172.21.15.118:0/2661353918' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "c9590426-3901-4f68-a5ef-d57aa8fbac92"}]': finished 2024-03-24T00:06:36.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:06:36 smithi189 ceph-mon[36187]: osdmap e5: 1 total, 0 up, 1 in 2024-03-24T00:06:36.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:06:36 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-24T00:06:36.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:06:36 smithi193 ceph-mon[36878]: pgmap v39: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T00:06:36.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:06:36 smithi193 ceph-mon[36878]: mgrmap e15: x(active, since 90s), standbys: z, y 2024-03-24T00:06:36.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:06:36 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "mgr metadata", "who": "y", "id": "y"}]: dispatch 2024-03-24T00:06:36.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:06:36 smithi193 ceph-mon[36878]: from='client.? 172.21.15.118:0/2661353918' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "c9590426-3901-4f68-a5ef-d57aa8fbac92"}]: dispatch 2024-03-24T00:06:36.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:06:36 smithi193 ceph-mon[36878]: from='client.? 172.21.15.118:0/2661353918' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "c9590426-3901-4f68-a5ef-d57aa8fbac92"}]': finished 2024-03-24T00:06:36.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:06:36 smithi193 ceph-mon[36878]: osdmap e5: 1 total, 0 up, 1 in 2024-03-24T00:06:36.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:06:36 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-24T00:06:37.945 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:06:37 smithi118 ceph-mon[31535]: from='client.? 172.21.15.118:0/3209610900' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-24T00:06:37.953 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:06:37 smithi189 ceph-mon[36187]: from='client.? 172.21.15.118:0/3209610900' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-24T00:06:37.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:06:37 smithi193 ceph-mon[36878]: from='client.? 172.21.15.118:0/3209610900' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-24T00:06:38.945 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:06:38 smithi118 ceph-mon[31535]: pgmap v41: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T00:06:38.953 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:06:38 smithi189 ceph-mon[36187]: pgmap v41: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T00:06:38.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:06:38 smithi193 ceph-mon[36878]: pgmap v41: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T00:06:40.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:06:40 smithi118 ceph-mon[31535]: pgmap v42: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T00:06:40.953 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:06:40 smithi189 ceph-mon[36187]: pgmap v42: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T00:06:40.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:06:40 smithi193 ceph-mon[36878]: pgmap v42: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T00:06:42.783 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:06:42 smithi118 ceph-mon[31535]: pgmap v43: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T00:06:42.953 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:06:42 smithi189 ceph-mon[36187]: pgmap v43: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T00:06:42.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:06:42 smithi193 ceph-mon[36878]: pgmap v43: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T00:06:44.910 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:06:44 smithi118 ceph-mon[31535]: pgmap v44: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T00:06:44.953 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:06:44 smithi189 ceph-mon[36187]: pgmap v44: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T00:06:44.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:06:44 smithi193 ceph-mon[36878]: pgmap v44: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T00:06:45.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:06:45 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-03-24T00:06:45.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:06:45 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T00:06:45.953 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:06:45 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-03-24T00:06:45.953 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:06:45 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T00:06:45.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:06:45 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-03-24T00:06:45.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:06:45 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T00:06:46.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:06:46 smithi118 ceph-mon[31535]: Deploying daemon osd.0 on smithi118 2024-03-24T00:06:46.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:06:46 smithi118 ceph-mon[31535]: pgmap v45: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T00:06:46.953 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:06:46 smithi189 ceph-mon[36187]: Deploying daemon osd.0 on smithi118 2024-03-24T00:06:46.953 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:06:46 smithi189 ceph-mon[36187]: pgmap v45: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T00:06:46.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:06:46 smithi193 ceph-mon[36878]: Deploying daemon osd.0 on smithi118 2024-03-24T00:06:46.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:06:46 smithi193 ceph-mon[36878]: pgmap v45: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T00:06:47.944 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:06:47 smithi118 ceph-mon[31535]: pgmap v46: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T00:06:47.953 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:06:47 smithi189 ceph-mon[36187]: pgmap v46: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T00:06:47.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:06:47 smithi193 ceph-mon[36878]: pgmap v46: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T00:06:50.850 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:06:50 smithi118 ceph-mon[31535]: pgmap v47: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T00:06:50.953 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:06:50 smithi189 ceph-mon[36187]: pgmap v47: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T00:06:50.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:06:50 smithi193 ceph-mon[36878]: pgmap v47: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T00:06:52.695 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:06:52 smithi118 ceph-mon[31535]: pgmap v48: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T00:06:52.855 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:06:52 smithi193 ceph-mon[36878]: pgmap v48: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T00:06:52.953 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:06:52 smithi189 ceph-mon[36187]: pgmap v48: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T00:06:54.384 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:06:54 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T00:06:54.384 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:06:54 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T00:06:54.384 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:06:54 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T00:06:54.384 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:06:54 smithi118 ceph-mon[31535]: pgmap v49: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T00:06:54.453 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:06:54 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T00:06:54.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:06:54 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T00:06:54.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:06:54 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T00:06:54.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:06:54 smithi189 ceph-mon[36187]: pgmap v49: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T00:06:54.460 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:06:54 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T00:06:54.460 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:06:54 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T00:06:54.460 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:06:54 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T00:06:54.460 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:06:54 smithi193 ceph-mon[36878]: pgmap v49: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T00:06:56.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:06:56 smithi118 ceph-mon[31535]: pgmap v50: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T00:06:56.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:06:56 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T00:06:56.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:06:56 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T00:06:56.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:06:56 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T00:06:56.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:06:56 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T00:06:56.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:06:56 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T00:06:56.953 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:06:56 smithi189 ceph-mon[36187]: pgmap v50: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T00:06:56.953 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:06:56 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T00:06:56.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:06:56 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T00:06:56.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:06:56 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T00:06:56.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:06:56 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T00:06:56.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:06:56 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T00:06:56.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:06:56 smithi193 ceph-mon[36878]: pgmap v50: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T00:06:56.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:06:56 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T00:06:56.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:06:56 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T00:06:56.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:06:56 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T00:06:56.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:06:56 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T00:06:56.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:06:56 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T00:06:57.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:06:57 smithi118 ceph-mon[31535]: from='osd.0 [v2:172.21.15.118:6802/557690061,v1:172.21.15.118:6803/557690061]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-03-24T00:06:57.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:06:57 smithi118 ceph-mon[31535]: from='osd.0 ' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-03-24T00:06:57.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:06:57 smithi189 ceph-mon[36187]: from='osd.0 [v2:172.21.15.118:6802/557690061,v1:172.21.15.118:6803/557690061]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-03-24T00:06:57.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:06:57 smithi189 ceph-mon[36187]: from='osd.0 ' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-03-24T00:06:57.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:06:57 smithi193 ceph-mon[36878]: from='osd.0 [v2:172.21.15.118:6802/557690061,v1:172.21.15.118:6803/557690061]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-03-24T00:06:57.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:06:57 smithi193 ceph-mon[36878]: from='osd.0 ' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-03-24T00:06:58.447 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:06:58.447+0000 7f19d3400700 1 -- 172.21.15.118:0/2582164806 <== mgr.14152 v2:172.21.15.118:6800/2112912943 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7f19dc0625d0 con 0x7f19c006dd00 2024-03-24T00:06:58.448 INFO:teuthology.orchestra.run.smithi118.stdout:Created osd(s) 0 on host 'smithi118' 2024-03-24T00:06:58.451 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:06:58.451+0000 7f19e1c00700 1 -- 172.21.15.118:0/2582164806 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7f19c006dd00 msgr2=0x7f19c00701c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:06:58.451 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:06:58.451+0000 7f19e1c00700 1 --2- 172.21.15.118:0/2582164806 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7f19c006dd00 0x7f19c00701c0 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f19dc075ef0 tx=0x7f19c8009260 comp rx=0 tx=0).stop 2024-03-24T00:06:58.451 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:06:58.451+0000 7f19e1c00700 1 -- 172.21.15.118:0/2582164806 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f19dc074bf0 msgr2=0x7f19dc19a700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:06:58.451 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:06:58.451+0000 7f19e1c00700 1 --2- 172.21.15.118:0/2582164806 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f19dc074bf0 0x7f19dc19a700 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f19cc00fd50 tx=0x7f19cc005040 comp rx=0 tx=0).stop 2024-03-24T00:06:58.451 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:06:58.451+0000 7f19e1c00700 1 -- 172.21.15.118:0/2582164806 shutdown_connections 2024-03-24T00:06:58.451 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:06:58.451+0000 7f19e1c00700 1 --2- 172.21.15.118:0/2582164806 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f19dc10a0a0 0x7f19dc1a1c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:06:58.451 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:06:58.451+0000 7f19e1c00700 1 --2- 172.21.15.118:0/2582164806 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7f19c006dd00 0x7f19c00701c0 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:06:58.451 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:06:58.451+0000 7f19e1c00700 1 --2- 172.21.15.118:0/2582164806 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f19dc074bf0 0x7f19dc19a700 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:06:58.451 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:06:58.451+0000 7f19e1c00700 1 --2- 172.21.15.118:0/2582164806 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f19dc0761c0 0x7f19dc19ac40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:06:58.451 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:06:58.451+0000 7f19e1c00700 1 -- 172.21.15.118:0/2582164806 >> 172.21.15.118:0/2582164806 conn(0x7f19dc0fd730 msgr2=0x7f19dc105fa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:06:58.452 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:06:58.451+0000 7f19e1c00700 1 -- 172.21.15.118:0/2582164806 shutdown_connections 2024-03-24T00:06:58.452 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:06:58.451+0000 7f19e1c00700 1 -- 172.21.15.118:0/2582164806 wait complete. 2024-03-24T00:06:58.832 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:06:58 smithi118 ceph-mon[31535]: pgmap v51: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T00:06:58.832 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:06:58 smithi118 ceph-mon[31535]: from='osd.0 ' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-03-24T00:06:58.832 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:06:58 smithi118 ceph-mon[31535]: from='osd.0 [v2:172.21.15.118:6802/557690061,v1:172.21.15.118:6803/557690061]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi118", "root=default"]}]: dispatch 2024-03-24T00:06:58.832 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:06:58 smithi118 ceph-mon[31535]: osdmap e6: 1 total, 0 up, 1 in 2024-03-24T00:06:58.832 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:06:58 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-24T00:06:58.833 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:06:58 smithi118 ceph-mon[31535]: from='osd.0 ' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi118", "root=default"]}]: dispatch 2024-03-24T00:06:58.833 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:06:58 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T00:06:58.833 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:06:58 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T00:06:58.833 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:06:58 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T00:06:58.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:06:58 smithi189 ceph-mon[36187]: pgmap v51: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T00:06:58.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:06:58 smithi189 ceph-mon[36187]: from='osd.0 ' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-03-24T00:06:58.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:06:58 smithi189 ceph-mon[36187]: from='osd.0 [v2:172.21.15.118:6802/557690061,v1:172.21.15.118:6803/557690061]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi118", "root=default"]}]: dispatch 2024-03-24T00:06:58.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:06:58 smithi189 ceph-mon[36187]: osdmap e6: 1 total, 0 up, 1 in 2024-03-24T00:06:58.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:06:58 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-24T00:06:58.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:06:58 smithi189 ceph-mon[36187]: from='osd.0 ' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi118", "root=default"]}]: dispatch 2024-03-24T00:06:58.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:06:58 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T00:06:58.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:06:58 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T00:06:58.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:06:58 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T00:06:58.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:06:58 smithi193 ceph-mon[36878]: pgmap v51: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T00:06:58.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:06:58 smithi193 ceph-mon[36878]: from='osd.0 ' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-03-24T00:06:58.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:06:58 smithi193 ceph-mon[36878]: from='osd.0 [v2:172.21.15.118:6802/557690061,v1:172.21.15.118:6803/557690061]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi118", "root=default"]}]: dispatch 2024-03-24T00:06:58.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:06:58 smithi193 ceph-mon[36878]: osdmap e6: 1 total, 0 up, 1 in 2024-03-24T00:06:58.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:06:58 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-24T00:06:58.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:06:58 smithi193 ceph-mon[36878]: from='osd.0 ' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi118", "root=default"]}]: dispatch 2024-03-24T00:06:58.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:06:58 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T00:06:58.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:06:58 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T00:06:58.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:06:58 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T00:06:59.153 DEBUG:teuthology.orchestra.run.smithi118:osd.0> sudo journalctl -f -n 0 -u ceph-d79a5420-e971-11ee-95cd-87774f69a715@osd.0.service 2024-03-24T00:06:59.158 INFO:tasks.cephadm:Deploying osd.1 on smithi189 with /dev/vg_nvme/lv_4... 2024-03-24T00:06:59.158 DEBUG:teuthology.orchestra.run.smithi189:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d79a5420-e971-11ee-95cd-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_4 2024-03-24T00:06:59.403 INFO:teuthology.orchestra.run.smithi189.stderr:Inferring config /var/lib/ceph/d79a5420-e971-11ee-95cd-87774f69a715/mon.b/config 2024-03-24T00:06:59.657 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:06:59 smithi118 ceph-mon[31535]: from='osd.0 ' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi118", "root=default"]}]': finished 2024-03-24T00:06:59.658 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:06:59 smithi118 ceph-mon[31535]: osdmap e7: 1 total, 0 up, 1 in 2024-03-24T00:06:59.658 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:06:59 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-24T00:06:59.658 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:06:59 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-24T00:06:59.781 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:06:59 smithi189 ceph-mon[36187]: from='osd.0 ' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi118", "root=default"]}]': finished 2024-03-24T00:06:59.782 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:06:59 smithi189 ceph-mon[36187]: osdmap e7: 1 total, 0 up, 1 in 2024-03-24T00:06:59.782 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:06:59 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-24T00:06:59.782 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:06:59 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-24T00:06:59.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:06:59 smithi193 ceph-mon[36878]: from='osd.0 ' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi118", "root=default"]}]': finished 2024-03-24T00:06:59.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:06:59 smithi193 ceph-mon[36878]: osdmap e7: 1 total, 0 up, 1 in 2024-03-24T00:06:59.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:06:59 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-24T00:06:59.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:06:59 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-24T00:07:00.813 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:07:00 smithi189 ceph-mon[36187]: purged_snaps scrub starts 2024-03-24T00:07:00.813 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:07:00 smithi189 ceph-mon[36187]: purged_snaps scrub ok 2024-03-24T00:07:00.813 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:07:00 smithi189 ceph-mon[36187]: pgmap v54: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T00:07:00.813 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:07:00 smithi189 ceph-mon[36187]: osd.0 [v2:172.21.15.118:6802/557690061,v1:172.21.15.118:6803/557690061] boot 2024-03-24T00:07:00.813 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:07:00 smithi189 ceph-mon[36187]: osdmap e8: 1 total, 1 up, 1 in 2024-03-24T00:07:00.813 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:07:00 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-24T00:07:00.895 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:07:00 smithi118 ceph-mon[31535]: purged_snaps scrub starts 2024-03-24T00:07:00.895 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:07:00 smithi118 ceph-mon[31535]: purged_snaps scrub ok 2024-03-24T00:07:00.895 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:07:00 smithi118 ceph-mon[31535]: pgmap v54: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T00:07:00.895 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:07:00 smithi118 ceph-mon[31535]: osd.0 [v2:172.21.15.118:6802/557690061,v1:172.21.15.118:6803/557690061] boot 2024-03-24T00:07:00.896 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:07:00 smithi118 ceph-mon[31535]: osdmap e8: 1 total, 1 up, 1 in 2024-03-24T00:07:00.896 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:07:00 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-24T00:07:00.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:07:00 smithi193 ceph-mon[36878]: purged_snaps scrub starts 2024-03-24T00:07:00.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:07:00 smithi193 ceph-mon[36878]: purged_snaps scrub ok 2024-03-24T00:07:00.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:07:00 smithi193 ceph-mon[36878]: pgmap v54: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T00:07:00.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:07:00 smithi193 ceph-mon[36878]: osd.0 [v2:172.21.15.118:6802/557690061,v1:172.21.15.118:6803/557690061] boot 2024-03-24T00:07:00.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:07:00 smithi193 ceph-mon[36878]: osdmap e8: 1 total, 1 up, 1 in 2024-03-24T00:07:00.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:07:00 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-24T00:07:01.378 INFO:teuthology.orchestra.run.smithi189.stderr:--> Zapping: /dev/vg_nvme/lv_4 2024-03-24T00:07:01.378 INFO:teuthology.orchestra.run.smithi189.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_4 bs=1M count=10 conv=fsync 2024-03-24T00:07:01.378 INFO:teuthology.orchestra.run.smithi189.stderr: stderr: 10+0 records in 2024-03-24T00:07:01.378 INFO:teuthology.orchestra.run.smithi189.stderr:10+0 records out 2024-03-24T00:07:01.378 INFO:teuthology.orchestra.run.smithi189.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0127314 s, 824 MB/s 2024-03-24T00:07:01.378 INFO:teuthology.orchestra.run.smithi189.stderr:--> Zapping successful for: 2024-03-24T00:07:01.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:07:01 smithi118 ceph-mon[31535]: osdmap e9: 1 total, 1 up, 1 in 2024-03-24T00:07:01.949 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:07:01 smithi189 ceph-mon[36187]: osdmap e9: 1 total, 1 up, 1 in 2024-03-24T00:07:01.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:07:01 smithi193 ceph-mon[36878]: osdmap e9: 1 total, 1 up, 1 in 2024-03-24T00:07:01.976 DEBUG:teuthology.orchestra.run.smithi189:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d79a5420-e971-11ee-95cd-87774f69a715 -- ceph orch daemon add osd smithi189:vg_nvme/lv_4 2024-03-24T00:07:02.208 INFO:teuthology.orchestra.run.smithi189.stderr:Inferring config /var/lib/ceph/d79a5420-e971-11ee-95cd-87774f69a715/mon.b/config 2024-03-24T00:07:02.710 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:07:02 smithi193 ceph-mon[36878]: pgmap v57: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-24T00:07:02.945 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:07:02 smithi118 ceph-mon[31535]: pgmap v57: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-24T00:07:02.953 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:07:02 smithi189 ceph-mon[36187]: pgmap v57: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-24T00:07:03.730 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-24T00:07:03.727+0000 7f817cb6b700 1 -- 172.21.15.189:0/1018878372 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8178103d80 msgr2=0x7f817810a630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:07:03.730 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-24T00:07:03.727+0000 7f817cb6b700 1 --2- 172.21.15.189:0/1018878372 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8178103d80 0x7f817810a630 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f817000b0d0 tx=0x7f817000b3e0 comp rx=0 tx=0).stop 2024-03-24T00:07:03.730 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-24T00:07:03.728+0000 7f817cb6b700 1 -- 172.21.15.189:0/1018878372 shutdown_connections 2024-03-24T00:07:03.730 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-24T00:07:03.728+0000 7f817cb6b700 1 --2- 172.21.15.189:0/1018878372 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8178103d80 0x7f817810a630 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:07:03.730 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-24T00:07:03.728+0000 7f817cb6b700 1 --2- 172.21.15.189:0/1018878372 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f81781021e0 0x7f8178102600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:07:03.730 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-24T00:07:03.728+0000 7f817cb6b700 1 --2- 172.21.15.189:0/1018878372 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f81781033e0 0x7f8178103840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:07:03.730 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-24T00:07:03.728+0000 7f817cb6b700 1 -- 172.21.15.189:0/1018878372 >> 172.21.15.189:0/1018878372 conn(0x7f81780fd7a0 msgr2=0x7f81780ffbc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:07:03.730 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-24T00:07:03.728+0000 7f817cb6b700 1 -- 172.21.15.189:0/1018878372 shutdown_connections 2024-03-24T00:07:03.731 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-24T00:07:03.728+0000 7f817cb6b700 1 -- 172.21.15.189:0/1018878372 wait complete. 2024-03-24T00:07:03.731 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-24T00:07:03.728+0000 7f817cb6b700 1 Processor -- start 2024-03-24T00:07:03.731 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-24T00:07:03.728+0000 7f817cb6b700 1 -- start start 2024-03-24T00:07:03.731 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-24T00:07:03.729+0000 7f817cb6b700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f81781021e0 0x7f81781983d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:07:03.731 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-24T00:07:03.729+0000 7f817cb6b700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f81781033e0 0x7f8178198910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:07:03.731 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-24T00:07:03.729+0000 7f817cb6b700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8178103d80 0x7f817819f990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:07:03.731 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-24T00:07:03.729+0000 7f817cb6b700 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f817810de50 con 0x7f81781033e0 2024-03-24T00:07:03.731 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-24T00:07:03.729+0000 7f817cb6b700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f817810dcd0 con 0x7f8178103d80 2024-03-24T00:07:03.731 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-24T00:07:03.729+0000 7f817cb6b700 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f817810dfd0 con 0x7f81781021e0 2024-03-24T00:07:03.731 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-24T00:07:03.729+0000 7f8176d9d700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8178103d80 0x7f817819f990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:07:03.732 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-24T00:07:03.729+0000 7f8176d9d700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8178103d80 0x7f817819f990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:43976/0 (socket says 172.21.15.189:43976) 2024-03-24T00:07:03.732 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-24T00:07:03.729+0000 7f8176d9d700 1 -- 172.21.15.189:0/88906418 learned_addr learned my addr 172.21.15.189:0/88906418 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-03-24T00:07:03.732 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-24T00:07:03.729+0000 7f817659c700 1 --2- 172.21.15.189:0/88906418 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f81781021e0 0x7f81781983d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:07:03.732 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-24T00:07:03.729+0000 7f8176d9d700 1 -- 172.21.15.189:0/88906418 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f81781021e0 msgr2=0x7f81781983d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:07:03.732 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-24T00:07:03.729+0000 7f8176d9d700 1 --2- 172.21.15.189:0/88906418 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f81781021e0 0x7f81781983d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:07:03.732 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-24T00:07:03.729+0000 7f8176d9d700 1 -- 172.21.15.189:0/88906418 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f81781033e0 msgr2=0x7f8178198910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:07:03.732 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-24T00:07:03.729+0000 7f8176d9d700 1 --2- 172.21.15.189:0/88906418 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f81781033e0 0x7f8178198910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:07:03.732 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-24T00:07:03.729+0000 7f8176d9d700 1 -- 172.21.15.189:0/88906418 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f81781a0030 con 0x7f8178103d80 2024-03-24T00:07:03.732 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-24T00:07:03.729+0000 7f8176d9d700 1 --2- 172.21.15.189:0/88906418 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8178103d80 0x7f817819f990 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f8170006070 tx=0x7f8170007d10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:07:03.732 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-24T00:07:03.729+0000 7f81677fe700 1 -- 172.21.15.189:0/88906418 <== mon.2 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8170021020 con 0x7f8178103d80 2024-03-24T00:07:03.733 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-24T00:07:03.729+0000 7f81677fe700 1 -- 172.21.15.189:0/88906418 <== mon.2 v2:172.21.15.189:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f817001d950 con 0x7f8178103d80 2024-03-24T00:07:03.733 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-24T00:07:03.729+0000 7f81677fe700 1 -- 172.21.15.189:0/88906418 <== mon.2 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f81700053f0 con 0x7f8178103d80 2024-03-24T00:07:03.733 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-24T00:07:03.729+0000 7f817cb6b700 1 -- 172.21.15.189:0/88906418 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f81781a02c0 con 0x7f8178103d80 2024-03-24T00:07:03.733 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-24T00:07:03.729+0000 7f817cb6b700 1 -- 172.21.15.189:0/88906418 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f81781a0830 con 0x7f8178103d80 2024-03-24T00:07:03.734 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-24T00:07:03.730+0000 7f817cb6b700 1 -- 172.21.15.189:0/88906418 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8178068e50 con 0x7f8178103d80 2024-03-24T00:07:03.734 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-24T00:07:03.731+0000 7f81677fe700 1 -- 172.21.15.189:0/88906418 <== mon.2 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 15) v1 ==== 137235+0+0 (secure 0 0 0) 0x7f8170024020 con 0x7f8178103d80 2024-03-24T00:07:03.734 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-24T00:07:03.732+0000 7f81677fe700 1 --2- 172.21.15.189:0/88906418 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7f81600a2800 0x7f81600a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:07:03.734 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-24T00:07:03.732+0000 7f81677fe700 1 -- 172.21.15.189:0/88906418 <== mon.2 v2:172.21.15.189:3300/0 5 ==== osd_map(9..9 src has 1..9) v4 ==== 1767+0+0 (secure 0 0 0) 0x7f81700ce790 con 0x7f8178103d80 2024-03-24T00:07:03.734 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-24T00:07:03.733+0000 7f817659c700 1 --2- 172.21.15.189:0/88906418 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7f81600a2800 0x7f81600a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:07:03.734 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-24T00:07:03.733+0000 7f817659c700 1 --2- 172.21.15.189:0/88906418 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7f81600a2800 0x7f81600a4cc0 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f8178103240 tx=0x7f816800b410 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:07:03.736 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-24T00:07:03.736+0000 7f81677fe700 1 -- 172.21.15.189:0/88906418 <== mon.2 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f817002b020 con 0x7f8178103d80 2024-03-24T00:07:03.942 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-24T00:07:03.941+0000 7f817cb6b700 1 -- 172.21.15.189:0/88906418 --> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi189:vg_nvme/lv_4", "target": ["mon-mgr", ""]}) v1 -- 0x7f81780625d0 con 0x7f81600a2800 2024-03-24T00:07:04.871 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:07:04 smithi189 ceph-mon[36187]: pgmap v58: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-24T00:07:04.871 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:07:04 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T00:07:04.871 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:07:04 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T00:07:04.871 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:07:04 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-24T00:07:04.871 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:07:04 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T00:07:04.871 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:07:04 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T00:07:04.871 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:07:04 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T00:07:04.871 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:07:04 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T00:07:04.871 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:07:04 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-24T00:07:04.871 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:07:04 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-24T00:07:04.871 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:07:04 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T00:07:04.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:07:04 smithi118 ceph-mon[31535]: pgmap v58: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-24T00:07:04.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:07:04 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T00:07:04.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:07:04 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T00:07:04.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:07:04 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-24T00:07:04.947 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:07:04 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T00:07:04.947 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:07:04 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T00:07:04.947 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:07:04 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T00:07:04.947 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:07:04 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T00:07:04.947 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:07:04 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-24T00:07:04.947 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:07:04 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-24T00:07:04.947 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:07:04 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T00:07:04.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:07:04 smithi193 ceph-mon[36878]: pgmap v58: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-24T00:07:04.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:07:04 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T00:07:04.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:07:04 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T00:07:04.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:07:04 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-24T00:07:04.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:07:04 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T00:07:04.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:07:04 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T00:07:04.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:07:04 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T00:07:04.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:07:04 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T00:07:04.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:07:04 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-24T00:07:04.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:07:04 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-24T00:07:04.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:07:04 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T00:07:05.730 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:07:05 smithi189 ceph-mon[36187]: Detected new or changed devices on smithi118 2024-03-24T00:07:05.730 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:07:05 smithi189 ceph-mon[36187]: Adjusting osd_memory_target on smithi118 to 17091M 2024-03-24T00:07:05.730 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:07:05 smithi189 ceph-mon[36187]: from='client.24149 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi189:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T00:07:05.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:07:05 smithi118 ceph-mon[31535]: Detected new or changed devices on smithi118 2024-03-24T00:07:05.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:07:05 smithi118 ceph-mon[31535]: Adjusting osd_memory_target on smithi118 to 17091M 2024-03-24T00:07:05.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:07:05 smithi118 ceph-mon[31535]: from='client.24149 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi189:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T00:07:05.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:07:05 smithi193 ceph-mon[36878]: Detected new or changed devices on smithi118 2024-03-24T00:07:05.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:07:05 smithi193 ceph-mon[36878]: Adjusting osd_memory_target on smithi118 to 17091M 2024-03-24T00:07:05.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:07:05 smithi193 ceph-mon[36878]: from='client.24149 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi189:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T00:07:06.902 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:07:06 smithi189 ceph-mon[36187]: pgmap v59: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-24T00:07:06.903 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:07:06 smithi189 ceph-mon[36187]: from='client.? 172.21.15.189:0/2901947225' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "fc7f827e-ecf6-4661-aad0-7a22951892e0"}]: dispatch 2024-03-24T00:07:06.903 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:07:06 smithi189 ceph-mon[36187]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "fc7f827e-ecf6-4661-aad0-7a22951892e0"}]: dispatch 2024-03-24T00:07:06.903 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:07:06 smithi189 ceph-mon[36187]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "fc7f827e-ecf6-4661-aad0-7a22951892e0"}]': finished 2024-03-24T00:07:06.903 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:07:06 smithi189 ceph-mon[36187]: osdmap e10: 2 total, 1 up, 2 in 2024-03-24T00:07:06.903 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:07:06 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-24T00:07:06.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:07:06 smithi118 ceph-mon[31535]: pgmap v59: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-24T00:07:06.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:07:06 smithi118 ceph-mon[31535]: from='client.? 172.21.15.189:0/2901947225' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "fc7f827e-ecf6-4661-aad0-7a22951892e0"}]: dispatch 2024-03-24T00:07:06.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:07:06 smithi118 ceph-mon[31535]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "fc7f827e-ecf6-4661-aad0-7a22951892e0"}]: dispatch 2024-03-24T00:07:06.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:07:06 smithi118 ceph-mon[31535]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "fc7f827e-ecf6-4661-aad0-7a22951892e0"}]': finished 2024-03-24T00:07:06.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:07:06 smithi118 ceph-mon[31535]: osdmap e10: 2 total, 1 up, 2 in 2024-03-24T00:07:06.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:07:06 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-24T00:07:06.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:07:06 smithi193 ceph-mon[36878]: pgmap v59: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-24T00:07:06.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:07:06 smithi193 ceph-mon[36878]: from='client.? 172.21.15.189:0/2901947225' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "fc7f827e-ecf6-4661-aad0-7a22951892e0"}]: dispatch 2024-03-24T00:07:06.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:07:06 smithi193 ceph-mon[36878]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "fc7f827e-ecf6-4661-aad0-7a22951892e0"}]: dispatch 2024-03-24T00:07:06.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:07:06 smithi193 ceph-mon[36878]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "fc7f827e-ecf6-4661-aad0-7a22951892e0"}]': finished 2024-03-24T00:07:06.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:07:06 smithi193 ceph-mon[36878]: osdmap e10: 2 total, 1 up, 2 in 2024-03-24T00:07:06.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:07:06 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-24T00:07:07.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:07:07 smithi118 ceph-mon[31535]: from='client.? 172.21.15.189:0/3295053247' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-24T00:07:07.953 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:07:07 smithi189 ceph-mon[36187]: from='client.? 172.21.15.189:0/3295053247' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-24T00:07:07.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:07:07 smithi193 ceph-mon[36878]: from='client.? 172.21.15.189:0/3295053247' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-24T00:07:08.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:07:08 smithi118 ceph-mon[31535]: pgmap v61: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-24T00:07:08.953 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:07:08 smithi189 ceph-mon[36187]: pgmap v61: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-24T00:07:08.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:07:08 smithi193 ceph-mon[36878]: pgmap v61: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-24T00:07:10.749 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:07:10 smithi189 ceph-mon[36187]: pgmap v62: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-24T00:07:10.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:07:10 smithi118 ceph-mon[31535]: pgmap v62: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-24T00:07:10.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:07:10 smithi193 ceph-mon[36878]: pgmap v62: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-24T00:07:12.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:07:12 smithi118 ceph-mon[31535]: pgmap v63: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-24T00:07:12.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:07:12 smithi189 ceph-mon[36187]: pgmap v63: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-24T00:07:12.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:07:12 smithi193 ceph-mon[36878]: pgmap v63: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-24T00:07:13.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:07:13 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-03-24T00:07:13.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:07:13 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T00:07:13.953 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:07:13 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-03-24T00:07:13.953 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:07:13 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T00:07:13.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:07:13 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-03-24T00:07:13.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:07:13 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T00:07:14.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:07:14 smithi118 ceph-mon[31535]: pgmap v64: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-24T00:07:14.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:07:14 smithi118 ceph-mon[31535]: Deploying daemon osd.1 on smithi189 2024-03-24T00:07:14.953 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:07:14 smithi189 ceph-mon[36187]: pgmap v64: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-24T00:07:14.953 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:07:14 smithi189 ceph-mon[36187]: Deploying daemon osd.1 on smithi189 2024-03-24T00:07:14.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:07:14 smithi193 ceph-mon[36878]: pgmap v64: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-24T00:07:14.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:07:14 smithi193 ceph-mon[36878]: Deploying daemon osd.1 on smithi189 2024-03-24T00:07:16.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:07:16 smithi118 ceph-mon[31535]: pgmap v65: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-24T00:07:16.953 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:07:16 smithi189 ceph-mon[36187]: pgmap v65: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-24T00:07:16.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:07:16 smithi193 ceph-mon[36878]: pgmap v65: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-24T00:07:18.930 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:07:18 smithi189 ceph-mon[36187]: pgmap v66: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-24T00:07:18.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:07:18 smithi118 ceph-mon[31535]: pgmap v66: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-24T00:07:18.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:07:18 smithi193 ceph-mon[36878]: pgmap v66: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-24T00:07:20.393 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:07:19 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T00:07:20.393 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:07:19 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T00:07:20.394 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:07:19 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T00:07:20.394 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:07:19 smithi189 ceph-mon[36187]: pgmap v67: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-24T00:07:20.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:07:19 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T00:07:20.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:07:19 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T00:07:20.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:07:19 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T00:07:20.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:07:19 smithi118 ceph-mon[31535]: pgmap v67: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-24T00:07:20.460 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:07:19 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T00:07:20.460 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:07:19 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T00:07:20.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:07:19 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T00:07:20.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:07:19 smithi193 ceph-mon[36878]: pgmap v67: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-24T00:07:22.279 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-24T00:07:22.279+0000 7f81677fe700 1 -- 172.21.15.189:0/88906418 <== mgr.14152 v2:172.21.15.118:6800/2112912943 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7f81780625d0 con 0x7f81600a2800 2024-03-24T00:07:22.280 INFO:teuthology.orchestra.run.smithi189.stdout:Created osd(s) 1 on host 'smithi189' 2024-03-24T00:07:22.281 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-24T00:07:22.282+0000 7f817cb6b700 1 -- 172.21.15.189:0/88906418 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7f81600a2800 msgr2=0x7f81600a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:07:22.281 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-24T00:07:22.282+0000 7f817cb6b700 1 --2- 172.21.15.189:0/88906418 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7f81600a2800 0x7f81600a4cc0 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f8178103240 tx=0x7f816800b410 comp rx=0 tx=0).stop 2024-03-24T00:07:22.281 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-24T00:07:22.282+0000 7f817cb6b700 1 -- 172.21.15.189:0/88906418 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8178103d80 msgr2=0x7f817819f990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:07:22.281 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-24T00:07:22.282+0000 7f817cb6b700 1 --2- 172.21.15.189:0/88906418 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8178103d80 0x7f817819f990 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f8170006070 tx=0x7f8170007d10 comp rx=0 tx=0).stop 2024-03-24T00:07:22.281 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-24T00:07:22.282+0000 7f817cb6b700 1 -- 172.21.15.189:0/88906418 shutdown_connections 2024-03-24T00:07:22.281 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-24T00:07:22.282+0000 7f817cb6b700 1 --2- 172.21.15.189:0/88906418 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8178103d80 0x7f817819f990 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:07:22.281 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-24T00:07:22.282+0000 7f817cb6b700 1 --2- 172.21.15.189:0/88906418 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7f81600a2800 0x7f81600a4cc0 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:07:22.282 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-24T00:07:22.282+0000 7f817cb6b700 1 --2- 172.21.15.189:0/88906418 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f81781021e0 0x7f81781983d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:07:22.282 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-24T00:07:22.282+0000 7f817cb6b700 1 --2- 172.21.15.189:0/88906418 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f81781033e0 0x7f8178198910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:07:22.282 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-24T00:07:22.282+0000 7f817cb6b700 1 -- 172.21.15.189:0/88906418 >> 172.21.15.189:0/88906418 conn(0x7f81780fd7a0 msgr2=0x7f8178106680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:07:22.282 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-24T00:07:22.282+0000 7f817cb6b700 1 -- 172.21.15.189:0/88906418 shutdown_connections 2024-03-24T00:07:22.282 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-24T00:07:22.282+0000 7f817cb6b700 1 -- 172.21.15.189:0/88906418 wait complete. 2024-03-24T00:07:22.842 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:07:22 smithi189 ceph-mon[36187]: pgmap v68: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-24T00:07:22.842 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:07:22 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T00:07:22.843 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:07:22 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T00:07:22.943 DEBUG:teuthology.orchestra.run.smithi189:osd.1> sudo journalctl -f -n 0 -u ceph-d79a5420-e971-11ee-95cd-87774f69a715@osd.1.service 2024-03-24T00:07:22.945 INFO:tasks.cephadm:Deploying osd.2 on smithi193 with /dev/vg_nvme/lv_4... 2024-03-24T00:07:22.946 DEBUG:teuthology.orchestra.run.smithi193:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d79a5420-e971-11ee-95cd-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_4 2024-03-24T00:07:22.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:07:22 smithi118 ceph-mon[31535]: pgmap v68: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-24T00:07:22.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:07:22 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T00:07:22.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:07:22 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T00:07:22.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:07:22 smithi193 ceph-mon[36878]: pgmap v68: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-24T00:07:22.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:07:22 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T00:07:22.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:07:22 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T00:07:23.191 INFO:teuthology.orchestra.run.smithi193.stderr:Inferring config /var/lib/ceph/d79a5420-e971-11ee-95cd-87774f69a715/mon.c/config 2024-03-24T00:07:23.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:07:23 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T00:07:23.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:07:23 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T00:07:23.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:07:23 smithi118 ceph-mon[31535]: from='osd.1 [v2:172.21.15.189:6800/4203029485,v1:172.21.15.189:6801/4203029485]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-03-24T00:07:23.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:07:23 smithi118 ceph-mon[31535]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-03-24T00:07:23.953 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:07:23 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T00:07:23.953 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:07:23 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T00:07:23.953 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:07:23 smithi189 ceph-mon[36187]: from='osd.1 [v2:172.21.15.189:6800/4203029485,v1:172.21.15.189:6801/4203029485]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-03-24T00:07:23.953 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:07:23 smithi189 ceph-mon[36187]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-03-24T00:07:23.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:07:23 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T00:07:23.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:07:23 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T00:07:23.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:07:23 smithi193 ceph-mon[36878]: from='osd.1 [v2:172.21.15.189:6800/4203029485,v1:172.21.15.189:6801/4203029485]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-03-24T00:07:23.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:07:23 smithi193 ceph-mon[36878]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-03-24T00:07:24.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:07:24 smithi118 ceph-mon[31535]: pgmap v69: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-24T00:07:24.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:07:24 smithi118 ceph-mon[31535]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-03-24T00:07:24.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:07:24 smithi118 ceph-mon[31535]: from='osd.1 [v2:172.21.15.189:6800/4203029485,v1:172.21.15.189:6801/4203029485]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi189", "root=default"]}]: dispatch 2024-03-24T00:07:24.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:07:24 smithi118 ceph-mon[31535]: osdmap e11: 2 total, 1 up, 2 in 2024-03-24T00:07:24.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:07:24 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-24T00:07:24.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:07:24 smithi118 ceph-mon[31535]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi189", "root=default"]}]: dispatch 2024-03-24T00:07:24.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:07:24 smithi189 ceph-mon[36187]: pgmap v69: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-24T00:07:24.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:07:24 smithi189 ceph-mon[36187]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-03-24T00:07:24.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:07:24 smithi189 ceph-mon[36187]: from='osd.1 [v2:172.21.15.189:6800/4203029485,v1:172.21.15.189:6801/4203029485]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi189", "root=default"]}]: dispatch 2024-03-24T00:07:24.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:07:24 smithi189 ceph-mon[36187]: osdmap e11: 2 total, 1 up, 2 in 2024-03-24T00:07:24.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:07:24 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-24T00:07:24.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:07:24 smithi189 ceph-mon[36187]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi189", "root=default"]}]: dispatch 2024-03-24T00:07:24.954 INFO:journalctl@ceph.osd.1.smithi189.stdout:Mar 24 00:07:24 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-osd-1[40812]: 2024-03-24T00:07:24.617+0000 7f747e2de700 -1 osd.1 0 waiting for initial osdmap 2024-03-24T00:07:24.954 INFO:journalctl@ceph.osd.1.smithi189.stdout:Mar 24 00:07:24 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-osd-1[40812]: 2024-03-24T00:07:24.620+0000 7f747a0d1700 -1 osd.1 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-24T00:07:24.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:07:24 smithi193 ceph-mon[36878]: pgmap v69: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-24T00:07:24.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:07:24 smithi193 ceph-mon[36878]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-03-24T00:07:24.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:07:24 smithi193 ceph-mon[36878]: from='osd.1 [v2:172.21.15.189:6800/4203029485,v1:172.21.15.189:6801/4203029485]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi189", "root=default"]}]: dispatch 2024-03-24T00:07:24.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:07:24 smithi193 ceph-mon[36878]: osdmap e11: 2 total, 1 up, 2 in 2024-03-24T00:07:24.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:07:24 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-24T00:07:24.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:07:24 smithi193 ceph-mon[36878]: from='osd.1 ' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi189", "root=default"]}]: dispatch 2024-03-24T00:07:25.165 INFO:teuthology.orchestra.run.smithi193.stderr:--> Zapping: /dev/vg_nvme/lv_4 2024-03-24T00:07:25.165 INFO:teuthology.orchestra.run.smithi193.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_4 bs=1M count=10 conv=fsync 2024-03-24T00:07:25.165 INFO:teuthology.orchestra.run.smithi193.stderr: stderr: 10+0 records in 2024-03-24T00:07:25.165 INFO:teuthology.orchestra.run.smithi193.stderr:10+0 records out 2024-03-24T00:07:25.165 INFO:teuthology.orchestra.run.smithi193.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0112818 s, 929 MB/s 2024-03-24T00:07:25.166 INFO:teuthology.orchestra.run.smithi193.stderr:--> Zapping successful for: 2024-03-24T00:07:25.679 DEBUG:teuthology.orchestra.run.smithi193:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d79a5420-e971-11ee-95cd-87774f69a715 -- ceph orch daemon add osd smithi193:vg_nvme/lv_4 2024-03-24T00:07:25.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:07:25 smithi118 ceph-mon[31535]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi189", "root=default"]}]': finished 2024-03-24T00:07:25.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:07:25 smithi118 ceph-mon[31535]: osdmap e12: 2 total, 1 up, 2 in 2024-03-24T00:07:25.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:07:25 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-24T00:07:25.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:07:25 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-24T00:07:25.953 INFO:teuthology.orchestra.run.smithi193.stderr:Inferring config /var/lib/ceph/d79a5420-e971-11ee-95cd-87774f69a715/mon.c/config 2024-03-24T00:07:25.953 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:07:25 smithi189 ceph-mon[36187]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi189", "root=default"]}]': finished 2024-03-24T00:07:25.953 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:07:25 smithi189 ceph-mon[36187]: osdmap e12: 2 total, 1 up, 2 in 2024-03-24T00:07:25.953 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:07:25 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-24T00:07:25.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:07:25 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-24T00:07:25.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:07:25 smithi193 ceph-mon[36878]: from='osd.1 ' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi189", "root=default"]}]': finished 2024-03-24T00:07:25.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:07:25 smithi193 ceph-mon[36878]: osdmap e12: 2 total, 1 up, 2 in 2024-03-24T00:07:25.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:07:25 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-24T00:07:25.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:07:25 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-24T00:07:26.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:07:26 smithi118 ceph-mon[31535]: purged_snaps scrub starts 2024-03-24T00:07:26.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:07:26 smithi118 ceph-mon[31535]: purged_snaps scrub ok 2024-03-24T00:07:26.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:07:26 smithi118 ceph-mon[31535]: pgmap v72: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-24T00:07:26.947 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:07:26 smithi118 ceph-mon[31535]: osd.1 [v2:172.21.15.189:6800/4203029485,v1:172.21.15.189:6801/4203029485] boot 2024-03-24T00:07:26.947 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:07:26 smithi118 ceph-mon[31535]: osdmap e13: 2 total, 2 up, 2 in 2024-03-24T00:07:26.947 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:07:26 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-24T00:07:26.947 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:07:26 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T00:07:26.947 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:07:26 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T00:07:26.947 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:07:26 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-24T00:07:26.947 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:07:26 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T00:07:26.947 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:07:26 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T00:07:26.947 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:07:26 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T00:07:26.947 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:07:26 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T00:07:26.947 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:07:26 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T00:07:26.947 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:07:26 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T00:07:26.947 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:07:26 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T00:07:26.948 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:07:26 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T00:07:26.953 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:07:26 smithi189 ceph-mon[36187]: purged_snaps scrub starts 2024-03-24T00:07:26.953 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:07:26 smithi189 ceph-mon[36187]: purged_snaps scrub ok 2024-03-24T00:07:26.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:07:26 smithi189 ceph-mon[36187]: pgmap v72: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-24T00:07:26.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:07:26 smithi189 ceph-mon[36187]: osd.1 [v2:172.21.15.189:6800/4203029485,v1:172.21.15.189:6801/4203029485] boot 2024-03-24T00:07:26.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:07:26 smithi189 ceph-mon[36187]: osdmap e13: 2 total, 2 up, 2 in 2024-03-24T00:07:26.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:07:26 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-24T00:07:26.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:07:26 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T00:07:26.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:07:26 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T00:07:26.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:07:26 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-24T00:07:26.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:07:26 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T00:07:26.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:07:26 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T00:07:26.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:07:26 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T00:07:26.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:07:26 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T00:07:26.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:07:26 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T00:07:26.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:07:26 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T00:07:26.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:07:26 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T00:07:26.955 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:07:26 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T00:07:26.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:07:26 smithi193 ceph-mon[36878]: purged_snaps scrub starts 2024-03-24T00:07:26.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:07:26 smithi193 ceph-mon[36878]: purged_snaps scrub ok 2024-03-24T00:07:26.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:07:26 smithi193 ceph-mon[36878]: pgmap v72: 0 pgs: ; 0 B data, 26 MiB used, 89 GiB / 89 GiB avail 2024-03-24T00:07:26.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:07:26 smithi193 ceph-mon[36878]: osd.1 [v2:172.21.15.189:6800/4203029485,v1:172.21.15.189:6801/4203029485] boot 2024-03-24T00:07:26.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:07:26 smithi193 ceph-mon[36878]: osdmap e13: 2 total, 2 up, 2 in 2024-03-24T00:07:26.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:07:26 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-24T00:07:26.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:07:26 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T00:07:26.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:07:26 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T00:07:26.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:07:26 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-24T00:07:26.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:07:26 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T00:07:26.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:07:26 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T00:07:26.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:07:26 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T00:07:26.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:07:26 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T00:07:26.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:07:26 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T00:07:26.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:07:26 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T00:07:26.962 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:07:26 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T00:07:26.962 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:07:26 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T00:07:27.392 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-24T00:07:27.389+0000 7f4e14111700 1 -- 172.21.15.193:0/3714351595 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f4e0c103d50 msgr2=0x7f4e0c10a600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:07:27.392 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-24T00:07:27.389+0000 7f4e14111700 1 --2- 172.21.15.193:0/3714351595 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f4e0c103d50 0x7f4e0c10a600 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f4e0400b0a0 tx=0x7f4e0400b3b0 comp rx=0 tx=0).stop 2024-03-24T00:07:27.392 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-24T00:07:27.389+0000 7f4e14111700 1 -- 172.21.15.193:0/3714351595 shutdown_connections 2024-03-24T00:07:27.392 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-24T00:07:27.389+0000 7f4e14111700 1 --2- 172.21.15.193:0/3714351595 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4e0c1021b0 0x7f4e0c1025d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:07:27.392 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-24T00:07:27.389+0000 7f4e14111700 1 --2- 172.21.15.193:0/3714351595 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f4e0c103d50 0x7f4e0c10a600 unknown :-1 s=CLOSED pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:07:27.392 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-24T00:07:27.389+0000 7f4e14111700 1 --2- 172.21.15.193:0/3714351595 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f4e0c1033b0 0x7f4e0c103810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:07:27.392 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-24T00:07:27.389+0000 7f4e14111700 1 -- 172.21.15.193:0/3714351595 >> 172.21.15.193:0/3714351595 conn(0x7f4e0c0fd770 msgr2=0x7f4e0c0ffb90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:07:27.392 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-24T00:07:27.390+0000 7f4e14111700 1 -- 172.21.15.193:0/3714351595 shutdown_connections 2024-03-24T00:07:27.392 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-24T00:07:27.390+0000 7f4e14111700 1 -- 172.21.15.193:0/3714351595 wait complete. 2024-03-24T00:07:27.392 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-24T00:07:27.390+0000 7f4e14111700 1 Processor -- start 2024-03-24T00:07:27.392 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-24T00:07:27.390+0000 7f4e14111700 1 -- start start 2024-03-24T00:07:27.393 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-24T00:07:27.390+0000 7f4e14111700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f4e0c1021b0 0x7f4e0c1984d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:07:27.393 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-24T00:07:27.390+0000 7f4e14111700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f4e0c1033b0 0x7f4e0c198a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:07:27.393 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-24T00:07:27.390+0000 7f4e14111700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4e0c103d50 0x7f4e0c19fa90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:07:27.393 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-24T00:07:27.390+0000 7f4e14111700 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4e0c10de90 con 0x7f4e0c1033b0 2024-03-24T00:07:27.393 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-24T00:07:27.390+0000 7f4e14111700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4e0c10dd10 con 0x7f4e0c103d50 2024-03-24T00:07:27.393 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-24T00:07:27.390+0000 7f4e14111700 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4e0c10e010 con 0x7f4e0c1021b0 2024-03-24T00:07:27.393 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-24T00:07:27.390+0000 7f4e11ead700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f4e0c1021b0 0x7f4e0c1984d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:07:27.393 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-24T00:07:27.390+0000 7f4e11ead700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f4e0c1021b0 0x7f4e0c1984d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.193:33484/0 (socket says 172.21.15.193:33484) 2024-03-24T00:07:27.393 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-24T00:07:27.391+0000 7f4e11ead700 1 -- 172.21.15.193:0/3475373085 learned_addr learned my addr 172.21.15.193:0/3475373085 (peer_addr_for_me v2:172.21.15.193:0/0) 2024-03-24T00:07:27.393 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-24T00:07:27.391+0000 7f4e11ead700 1 -- 172.21.15.193:0/3475373085 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4e0c103d50 msgr2=0x7f4e0c19fa90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:07:27.393 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-24T00:07:27.391+0000 7f4e11ead700 1 --2- 172.21.15.193:0/3475373085 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4e0c103d50 0x7f4e0c19fa90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:07:27.393 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-24T00:07:27.391+0000 7f4e11ead700 1 -- 172.21.15.193:0/3475373085 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f4e0c1033b0 msgr2=0x7f4e0c198a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:07:27.393 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-24T00:07:27.391+0000 7f4e11ead700 1 --2- 172.21.15.193:0/3475373085 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f4e0c1033b0 0x7f4e0c198a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:07:27.394 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-24T00:07:27.391+0000 7f4e11ead700 1 -- 172.21.15.193:0/3475373085 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4e0c1a0130 con 0x7f4e0c1021b0 2024-03-24T00:07:27.394 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-24T00:07:27.391+0000 7f4e11ead700 1 --2- 172.21.15.193:0/3475373085 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f4e0c1021b0 0x7f4e0c1984d0 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f4e0800b530 tx=0x7f4e0800b8f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:07:27.394 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-24T00:07:27.391+0000 7f4e02ffd700 1 -- 172.21.15.193:0/3475373085 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4e08015020 con 0x7f4e0c1021b0 2024-03-24T00:07:27.394 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-24T00:07:27.391+0000 7f4e02ffd700 1 -- 172.21.15.193:0/3475373085 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f4e08011d10 con 0x7f4e0c1021b0 2024-03-24T00:07:27.394 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-24T00:07:27.391+0000 7f4e14111700 1 -- 172.21.15.193:0/3475373085 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4e0c1a0420 con 0x7f4e0c1021b0 2024-03-24T00:07:27.394 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-24T00:07:27.391+0000 7f4e02ffd700 1 -- 172.21.15.193:0/3475373085 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4e0800f2e0 con 0x7f4e0c1021b0 2024-03-24T00:07:27.394 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-24T00:07:27.391+0000 7f4e14111700 1 -- 172.21.15.193:0/3475373085 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4e0c1a09e0 con 0x7f4e0c1021b0 2024-03-24T00:07:27.394 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-24T00:07:27.392+0000 7f4e14111700 1 -- 172.21.15.193:0/3475373085 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4e0c068e50 con 0x7f4e0c1021b0 2024-03-24T00:07:27.395 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-24T00:07:27.393+0000 7f4e02ffd700 1 -- 172.21.15.193:0/3475373085 <== mon.1 v2:172.21.15.193:3300/0 4 ==== mgrmap(e 15) v1 ==== 137235+0+0 (secure 0 0 0) 0x7f4e08018020 con 0x7f4e0c1021b0 2024-03-24T00:07:27.395 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-24T00:07:27.394+0000 7f4e02ffd700 1 --2- 172.21.15.193:0/3475373085 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7f4df80a2880 0x7f4df80a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:07:27.395 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-24T00:07:27.394+0000 7f4e02ffd700 1 -- 172.21.15.193:0/3475373085 <== mon.1 v2:172.21.15.193:3300/0 5 ==== osd_map(14..14 src has 1..14) v4 ==== 2309+0+0 (secure 0 0 0) 0x7f4e080ccf30 con 0x7f4e0c1021b0 2024-03-24T00:07:27.395 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-24T00:07:27.395+0000 7f4e116ac700 1 --2- 172.21.15.193:0/3475373085 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7f4df80a2880 0x7f4df80a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:07:27.395 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-24T00:07:27.395+0000 7f4e116ac700 1 --2- 172.21.15.193:0/3475373085 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7f4df80a2880 0x7f4df80a4d40 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f4dfc0097b0 tx=0x7f4dfc006d20 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:07:27.398 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-24T00:07:27.398+0000 7f4e02ffd700 1 -- 172.21.15.193:0/3475373085 <== mon.1 v2:172.21.15.193:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f4e0801f020 con 0x7f4e0c1021b0 2024-03-24T00:07:27.598 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-24T00:07:27.598+0000 7f4e14111700 1 -- 172.21.15.193:0/3475373085 --> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi193:vg_nvme/lv_4", "target": ["mon-mgr", ""]}) v1 -- 0x7f4e0c0625d0 con 0x7f4df80a2880 2024-03-24T00:07:27.835 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:07:27 smithi189 ceph-mon[36187]: Detected new or changed devices on smithi189 2024-03-24T00:07:27.836 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:07:27 smithi189 ceph-mon[36187]: Adjusting osd_memory_target on smithi189 to 17080M 2024-03-24T00:07:27.836 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:07:27 smithi189 ceph-mon[36187]: osdmap e14: 2 total, 2 up, 2 in 2024-03-24T00:07:27.836 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:07:27 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-24T00:07:27.836 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:07:27 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-24T00:07:27.836 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:07:27 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T00:07:27.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:07:27 smithi118 ceph-mon[31535]: Detected new or changed devices on smithi189 2024-03-24T00:07:27.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:07:27 smithi118 ceph-mon[31535]: Adjusting osd_memory_target on smithi189 to 17080M 2024-03-24T00:07:27.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:07:27 smithi118 ceph-mon[31535]: osdmap e14: 2 total, 2 up, 2 in 2024-03-24T00:07:27.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:07:27 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-24T00:07:27.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:07:27 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-24T00:07:27.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:07:27 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T00:07:27.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:07:27 smithi193 ceph-mon[36878]: Detected new or changed devices on smithi189 2024-03-24T00:07:27.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:07:27 smithi193 ceph-mon[36878]: Adjusting osd_memory_target on smithi189 to 17080M 2024-03-24T00:07:27.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:07:27 smithi193 ceph-mon[36878]: osdmap e14: 2 total, 2 up, 2 in 2024-03-24T00:07:27.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:07:27 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-24T00:07:27.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:07:27 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-24T00:07:27.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:07:27 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T00:07:28.842 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:07:28 smithi193 ceph-mon[36878]: pgmap v75: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-24T00:07:28.843 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:07:28 smithi193 ceph-mon[36878]: from='client.24169 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi193:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T00:07:28.945 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:07:28 smithi118 ceph-mon[31535]: pgmap v75: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-24T00:07:28.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:07:28 smithi118 ceph-mon[31535]: from='client.24169 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi193:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T00:07:28.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:07:28 smithi189 ceph-mon[36187]: pgmap v75: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-24T00:07:28.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:07:28 smithi189 ceph-mon[36187]: from='client.24169 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi193:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T00:07:30.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:07:30 smithi118 ceph-mon[31535]: pgmap v76: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-24T00:07:30.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:07:30 smithi118 ceph-mon[31535]: from='client.? 172.21.15.193:0/3745602826' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "e73ccfe9-0e5b-4734-a154-a2f0d829fcfb"}]: dispatch 2024-03-24T00:07:30.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:07:30 smithi118 ceph-mon[31535]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "e73ccfe9-0e5b-4734-a154-a2f0d829fcfb"}]: dispatch 2024-03-24T00:07:30.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:07:30 smithi118 ceph-mon[31535]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "e73ccfe9-0e5b-4734-a154-a2f0d829fcfb"}]': finished 2024-03-24T00:07:30.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:07:30 smithi118 ceph-mon[31535]: osdmap e15: 3 total, 2 up, 3 in 2024-03-24T00:07:30.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:07:30 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-24T00:07:30.953 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:07:30 smithi189 ceph-mon[36187]: pgmap v76: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-24T00:07:30.953 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:07:30 smithi189 ceph-mon[36187]: from='client.? 172.21.15.193:0/3745602826' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "e73ccfe9-0e5b-4734-a154-a2f0d829fcfb"}]: dispatch 2024-03-24T00:07:30.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:07:30 smithi189 ceph-mon[36187]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "e73ccfe9-0e5b-4734-a154-a2f0d829fcfb"}]: dispatch 2024-03-24T00:07:30.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:07:30 smithi189 ceph-mon[36187]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "e73ccfe9-0e5b-4734-a154-a2f0d829fcfb"}]': finished 2024-03-24T00:07:30.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:07:30 smithi189 ceph-mon[36187]: osdmap e15: 3 total, 2 up, 3 in 2024-03-24T00:07:30.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:07:30 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-24T00:07:30.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:07:30 smithi193 ceph-mon[36878]: pgmap v76: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-24T00:07:30.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:07:30 smithi193 ceph-mon[36878]: from='client.? 172.21.15.193:0/3745602826' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "e73ccfe9-0e5b-4734-a154-a2f0d829fcfb"}]: dispatch 2024-03-24T00:07:30.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:07:30 smithi193 ceph-mon[36878]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "e73ccfe9-0e5b-4734-a154-a2f0d829fcfb"}]: dispatch 2024-03-24T00:07:30.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:07:30 smithi193 ceph-mon[36878]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "e73ccfe9-0e5b-4734-a154-a2f0d829fcfb"}]': finished 2024-03-24T00:07:30.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:07:30 smithi193 ceph-mon[36878]: osdmap e15: 3 total, 2 up, 3 in 2024-03-24T00:07:30.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:07:30 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-24T00:07:31.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:07:31 smithi118 ceph-mon[31535]: from='client.? 172.21.15.193:0/3989622340' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-24T00:07:31.953 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:07:31 smithi189 ceph-mon[36187]: from='client.? 172.21.15.193:0/3989622340' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-24T00:07:31.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:07:31 smithi193 ceph-mon[36878]: from='client.? 172.21.15.193:0/3989622340' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-24T00:07:32.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:07:32 smithi118 ceph-mon[31535]: pgmap v78: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-24T00:07:32.953 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:07:32 smithi189 ceph-mon[36187]: pgmap v78: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-24T00:07:32.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:07:32 smithi193 ceph-mon[36878]: pgmap v78: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-24T00:07:34.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:07:34 smithi118 ceph-mon[31535]: pgmap v79: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-24T00:07:34.953 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:07:34 smithi189 ceph-mon[36187]: pgmap v79: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-24T00:07:34.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:07:34 smithi193 ceph-mon[36878]: pgmap v79: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-24T00:07:36.905 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:07:36 smithi193 ceph-mon[36878]: pgmap v80: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-24T00:07:36.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:07:36 smithi118 ceph-mon[31535]: pgmap v80: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-24T00:07:36.953 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:07:36 smithi189 ceph-mon[36187]: pgmap v80: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-24T00:07:37.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:07:37 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-03-24T00:07:37.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:07:37 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T00:07:37.946 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:07:37 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-03-24T00:07:37.946 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:07:37 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T00:07:37.953 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:07:37 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-03-24T00:07:37.953 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:07:37 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T00:07:38.745 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:07:38 smithi193 ceph-mon[36878]: Deploying daemon osd.2 on smithi193 2024-03-24T00:07:38.745 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:07:38 smithi193 ceph-mon[36878]: pgmap v81: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-24T00:07:38.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:07:38 smithi118 ceph-mon[31535]: Deploying daemon osd.2 on smithi193 2024-03-24T00:07:38.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:07:38 smithi118 ceph-mon[31535]: pgmap v81: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-24T00:07:38.953 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:07:38 smithi189 ceph-mon[36187]: Deploying daemon osd.2 on smithi193 2024-03-24T00:07:38.953 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:07:38 smithi189 ceph-mon[36187]: pgmap v81: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-24T00:07:40.002 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:07:39 smithi193 ceph-mon[36878]: pgmap v82: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-24T00:07:40.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:07:39 smithi118 ceph-mon[31535]: pgmap v82: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-24T00:07:40.203 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:07:39 smithi189 ceph-mon[36187]: pgmap v82: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-24T00:07:42.710 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:07:42 smithi193 ceph-mon[36878]: pgmap v83: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-24T00:07:42.711 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:07:42 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T00:07:42.711 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:07:42 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T00:07:42.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:07:42 smithi118 ceph-mon[31535]: pgmap v83: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-24T00:07:42.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:07:42 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T00:07:42.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:07:42 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T00:07:42.953 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:07:42 smithi189 ceph-mon[36187]: pgmap v83: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-24T00:07:42.953 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:07:42 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T00:07:42.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:07:42 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T00:07:43.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:07:43 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T00:07:43.953 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:07:43 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T00:07:43.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:07:43 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T00:07:44.834 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:07:44 smithi193 ceph-mon[36878]: pgmap v84: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-24T00:07:44.945 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:07:44 smithi118 ceph-mon[31535]: pgmap v84: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-24T00:07:44.953 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:07:44 smithi189 ceph-mon[36187]: pgmap v84: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-24T00:07:45.896 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:07:45 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T00:07:45.896 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:07:45 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T00:07:45.896 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:07:45 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T00:07:45.896 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:07:45 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T00:07:45.896 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:07:45 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T00:07:45.896 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:07:45 smithi193 ceph-mon[36878]: pgmap v85: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-24T00:07:45.982 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-24T00:07:45.982+0000 7f4e02ffd700 1 -- 172.21.15.193:0/3475373085 <== mgr.14152 v2:172.21.15.118:6800/2112912943 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7f4e0c0625d0 con 0x7f4df80a2880 2024-03-24T00:07:45.983 INFO:teuthology.orchestra.run.smithi193.stdout:Created osd(s) 2 on host 'smithi193' 2024-03-24T00:07:45.984 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-24T00:07:45.984+0000 7f4e14111700 1 -- 172.21.15.193:0/3475373085 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7f4df80a2880 msgr2=0x7f4df80a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:07:45.985 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-24T00:07:45.984+0000 7f4e14111700 1 --2- 172.21.15.193:0/3475373085 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7f4df80a2880 0x7f4df80a4d40 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f4dfc0097b0 tx=0x7f4dfc006d20 comp rx=0 tx=0).stop 2024-03-24T00:07:45.985 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-24T00:07:45.984+0000 7f4e14111700 1 -- 172.21.15.193:0/3475373085 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f4e0c1021b0 msgr2=0x7f4e0c1984d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:07:45.985 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-24T00:07:45.984+0000 7f4e14111700 1 --2- 172.21.15.193:0/3475373085 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f4e0c1021b0 0x7f4e0c1984d0 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f4e0800b530 tx=0x7f4e0800b8f0 comp rx=0 tx=0).stop 2024-03-24T00:07:45.985 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-24T00:07:45.984+0000 7f4e14111700 1 -- 172.21.15.193:0/3475373085 shutdown_connections 2024-03-24T00:07:45.985 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-24T00:07:45.984+0000 7f4e14111700 1 --2- 172.21.15.193:0/3475373085 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4e0c103d50 0x7f4e0c19fa90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:07:45.985 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-24T00:07:45.984+0000 7f4e14111700 1 --2- 172.21.15.193:0/3475373085 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7f4df80a2880 0x7f4df80a4d40 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:07:45.985 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-24T00:07:45.984+0000 7f4e14111700 1 --2- 172.21.15.193:0/3475373085 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f4e0c1021b0 0x7f4e0c1984d0 unknown :-1 s=CLOSED pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:07:45.985 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-24T00:07:45.984+0000 7f4e14111700 1 --2- 172.21.15.193:0/3475373085 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f4e0c1033b0 0x7f4e0c198a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:07:45.985 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-24T00:07:45.984+0000 7f4e14111700 1 -- 172.21.15.193:0/3475373085 >> 172.21.15.193:0/3475373085 conn(0x7f4e0c0fd770 msgr2=0x7f4e0c1066b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:07:45.985 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-24T00:07:45.984+0000 7f4e14111700 1 -- 172.21.15.193:0/3475373085 shutdown_connections 2024-03-24T00:07:45.985 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-24T00:07:45.985+0000 7f4e14111700 1 -- 172.21.15.193:0/3475373085 wait complete. 2024-03-24T00:07:46.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:07:45 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T00:07:46.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:07:45 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T00:07:46.197 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:07:45 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T00:07:46.197 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:07:45 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T00:07:46.197 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:07:45 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T00:07:46.197 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:07:45 smithi118 ceph-mon[31535]: pgmap v85: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-24T00:07:46.203 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:07:45 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T00:07:46.203 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:07:45 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T00:07:46.203 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:07:45 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T00:07:46.203 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:07:45 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T00:07:46.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:07:45 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T00:07:46.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:07:45 smithi189 ceph-mon[36187]: pgmap v85: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-24T00:07:46.577 DEBUG:teuthology.orchestra.run.smithi193:osd.2> sudo journalctl -f -n 0 -u ceph-d79a5420-e971-11ee-95cd-87774f69a715@osd.2.service 2024-03-24T00:07:46.580 INFO:tasks.cephadm:Deploying osd.3 on smithi118 with /dev/vg_nvme/lv_3... 2024-03-24T00:07:46.580 DEBUG:teuthology.orchestra.run.smithi118:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d79a5420-e971-11ee-95cd-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_3 2024-03-24T00:07:47.020 INFO:teuthology.orchestra.run.smithi118.stderr:Inferring config /var/lib/ceph/d79a5420-e971-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T00:07:47.409 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:07:46 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T00:07:47.409 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:07:46 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T00:07:47.409 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:07:46 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T00:07:47.409 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:07:46 smithi193 ceph-mon[36878]: from='osd.2 [v2:172.21.15.193:6800/3699076971,v1:172.21.15.193:6801/3699076971]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-03-24T00:07:47.409 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:07:46 smithi193 ceph-mon[36878]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-03-24T00:07:47.410 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:07:46 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T00:07:47.410 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:07:46 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T00:07:47.410 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:07:46 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T00:07:47.410 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:07:46 smithi189 ceph-mon[36187]: from='osd.2 [v2:172.21.15.193:6800/3699076971,v1:172.21.15.193:6801/3699076971]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-03-24T00:07:47.410 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:07:46 smithi189 ceph-mon[36187]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-03-24T00:07:47.410 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:07:46 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T00:07:47.411 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:07:46 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T00:07:47.411 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:07:46 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T00:07:47.411 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:07:46 smithi118 ceph-mon[31535]: from='osd.2 [v2:172.21.15.193:6800/3699076971,v1:172.21.15.193:6801/3699076971]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-03-24T00:07:47.411 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:07:46 smithi118 ceph-mon[31535]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-03-24T00:07:48.043 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:07:47 smithi193 ceph-mon[36878]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-03-24T00:07:48.043 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:07:47 smithi193 ceph-mon[36878]: osdmap e16: 3 total, 2 up, 3 in 2024-03-24T00:07:48.043 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:07:47 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-24T00:07:48.043 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:07:47 smithi193 ceph-mon[36878]: from='osd.2 [v2:172.21.15.193:6800/3699076971,v1:172.21.15.193:6801/3699076971]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi193", "root=default"]}]: dispatch 2024-03-24T00:07:48.043 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:07:47 smithi193 ceph-mon[36878]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi193", "root=default"]}]: dispatch 2024-03-24T00:07:48.044 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:07:47 smithi193 ceph-mon[36878]: pgmap v87: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-24T00:07:48.044 INFO:journalctl@ceph.osd.2.smithi193.stdout:Mar 24 00:07:47 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-osd-2[42239]: 2024-03-24T00:07:47.953+0000 7fdcca4ff700 -1 osd.2 0 waiting for initial osdmap 2024-03-24T00:07:48.044 INFO:journalctl@ceph.osd.2.smithi193.stdout:Mar 24 00:07:47 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-osd-2[42239]: 2024-03-24T00:07:47.960+0000 7fdcc62f2700 -1 osd.2 17 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-24T00:07:48.227 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:07:47 smithi118 ceph-mon[31535]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-03-24T00:07:48.227 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:07:47 smithi118 ceph-mon[31535]: osdmap e16: 3 total, 2 up, 3 in 2024-03-24T00:07:48.227 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:07:47 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-24T00:07:48.227 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:07:47 smithi118 ceph-mon[31535]: from='osd.2 [v2:172.21.15.193:6800/3699076971,v1:172.21.15.193:6801/3699076971]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi193", "root=default"]}]: dispatch 2024-03-24T00:07:48.227 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:07:47 smithi118 ceph-mon[31535]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi193", "root=default"]}]: dispatch 2024-03-24T00:07:48.227 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:07:47 smithi118 ceph-mon[31535]: pgmap v87: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-24T00:07:48.453 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:07:47 smithi189 ceph-mon[36187]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-03-24T00:07:48.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:07:47 smithi189 ceph-mon[36187]: osdmap e16: 3 total, 2 up, 3 in 2024-03-24T00:07:48.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:07:47 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-24T00:07:48.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:07:47 smithi189 ceph-mon[36187]: from='osd.2 [v2:172.21.15.193:6800/3699076971,v1:172.21.15.193:6801/3699076971]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi193", "root=default"]}]: dispatch 2024-03-24T00:07:48.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:07:47 smithi189 ceph-mon[36187]: from='osd.2 ' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi193", "root=default"]}]: dispatch 2024-03-24T00:07:48.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:07:47 smithi189 ceph-mon[36187]: pgmap v87: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-24T00:07:49.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:07:48 smithi118 ceph-mon[31535]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi193", "root=default"]}]': finished 2024-03-24T00:07:49.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:07:48 smithi118 ceph-mon[31535]: osdmap e17: 3 total, 2 up, 3 in 2024-03-24T00:07:49.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:07:48 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-24T00:07:49.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:07:48 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-24T00:07:49.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:07:48 smithi193 ceph-mon[36878]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi193", "root=default"]}]': finished 2024-03-24T00:07:49.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:07:48 smithi193 ceph-mon[36878]: osdmap e17: 3 total, 2 up, 3 in 2024-03-24T00:07:49.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:07:48 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-24T00:07:49.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:07:48 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-24T00:07:49.453 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:07:48 smithi189 ceph-mon[36187]: from='osd.2 ' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi193", "root=default"]}]': finished 2024-03-24T00:07:49.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:07:48 smithi189 ceph-mon[36187]: osdmap e17: 3 total, 2 up, 3 in 2024-03-24T00:07:49.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:07:48 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-24T00:07:49.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:07:48 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-24T00:07:49.789 INFO:teuthology.orchestra.run.smithi118.stderr:--> Zapping: /dev/vg_nvme/lv_3 2024-03-24T00:07:49.790 INFO:teuthology.orchestra.run.smithi118.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_3 bs=1M count=10 conv=fsync 2024-03-24T00:07:49.790 INFO:teuthology.orchestra.run.smithi118.stderr: stderr: 10+0 records in 2024-03-24T00:07:49.790 INFO:teuthology.orchestra.run.smithi118.stderr:10+0 records out 2024-03-24T00:07:49.790 INFO:teuthology.orchestra.run.smithi118.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0338812 s, 309 MB/s 2024-03-24T00:07:49.790 INFO:teuthology.orchestra.run.smithi118.stderr:--> Zapping successful for: 2024-03-24T00:07:50.300 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:07:50 smithi118 ceph-mon[31535]: purged_snaps scrub starts 2024-03-24T00:07:50.301 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:07:50 smithi118 ceph-mon[31535]: purged_snaps scrub ok 2024-03-24T00:07:50.301 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:07:50 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-24T00:07:50.301 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:07:50 smithi118 ceph-mon[31535]: osd.2 [v2:172.21.15.193:6800/3699076971,v1:172.21.15.193:6801/3699076971] boot 2024-03-24T00:07:50.301 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:07:50 smithi118 ceph-mon[31535]: osdmap e18: 3 total, 3 up, 3 in 2024-03-24T00:07:50.301 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:07:50 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-24T00:07:50.301 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:07:50 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T00:07:50.301 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:07:50 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T00:07:50.301 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:07:50 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-24T00:07:50.301 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:07:50 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T00:07:50.301 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:07:50 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T00:07:50.301 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:07:50 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T00:07:50.302 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:07:50 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T00:07:50.453 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:07:50 smithi189 ceph-mon[36187]: purged_snaps scrub starts 2024-03-24T00:07:50.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:07:50 smithi189 ceph-mon[36187]: purged_snaps scrub ok 2024-03-24T00:07:50.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:07:50 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-24T00:07:50.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:07:50 smithi189 ceph-mon[36187]: osd.2 [v2:172.21.15.193:6800/3699076971,v1:172.21.15.193:6801/3699076971] boot 2024-03-24T00:07:50.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:07:50 smithi189 ceph-mon[36187]: osdmap e18: 3 total, 3 up, 3 in 2024-03-24T00:07:50.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:07:50 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-24T00:07:50.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:07:50 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T00:07:50.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:07:50 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T00:07:50.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:07:50 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-24T00:07:50.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:07:50 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T00:07:50.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:07:50 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T00:07:50.455 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:07:50 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T00:07:50.455 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:07:50 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T00:07:50.460 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:07:50 smithi193 ceph-mon[36878]: purged_snaps scrub starts 2024-03-24T00:07:50.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:07:50 smithi193 ceph-mon[36878]: purged_snaps scrub ok 2024-03-24T00:07:50.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:07:50 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-24T00:07:50.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:07:50 smithi193 ceph-mon[36878]: osd.2 [v2:172.21.15.193:6800/3699076971,v1:172.21.15.193:6801/3699076971] boot 2024-03-24T00:07:50.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:07:50 smithi193 ceph-mon[36878]: osdmap e18: 3 total, 3 up, 3 in 2024-03-24T00:07:50.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:07:50 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-24T00:07:50.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:07:50 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T00:07:50.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:07:50 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T00:07:50.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:07:50 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-24T00:07:50.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:07:50 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T00:07:50.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:07:50 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T00:07:50.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:07:50 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T00:07:50.462 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:07:50 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T00:07:50.622 DEBUG:teuthology.orchestra.run.smithi118:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d79a5420-e971-11ee-95cd-87774f69a715 -- ceph orch daemon add osd smithi118:vg_nvme/lv_3 2024-03-24T00:07:51.028 INFO:teuthology.orchestra.run.smithi118.stderr:Inferring config /var/lib/ceph/d79a5420-e971-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T00:07:51.425 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:07:51 smithi118 ceph-mon[31535]: pgmap v90: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-24T00:07:51.425 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:07:51 smithi118 ceph-mon[31535]: Detected new or changed devices on smithi193 2024-03-24T00:07:51.425 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:07:51 smithi118 ceph-mon[31535]: Adjusting osd_memory_target on smithi193 to 17080M 2024-03-24T00:07:51.426 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:07:51 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]: dispatch 2024-03-24T00:07:51.426 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:07:51 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd='[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]': finished 2024-03-24T00:07:51.426 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:07:51 smithi118 ceph-mon[31535]: osdmap e19: 3 total, 3 up, 3 in 2024-03-24T00:07:51.426 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:07:51 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]: dispatch 2024-03-24T00:07:51.453 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:07:51 smithi189 ceph-mon[36187]: pgmap v90: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-24T00:07:51.453 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:07:51 smithi189 ceph-mon[36187]: Detected new or changed devices on smithi193 2024-03-24T00:07:51.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:07:51 smithi189 ceph-mon[36187]: Adjusting osd_memory_target on smithi193 to 17080M 2024-03-24T00:07:51.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:07:51 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]: dispatch 2024-03-24T00:07:51.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:07:51 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd='[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]': finished 2024-03-24T00:07:51.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:07:51 smithi189 ceph-mon[36187]: osdmap e19: 3 total, 3 up, 3 in 2024-03-24T00:07:51.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:07:51 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]: dispatch 2024-03-24T00:07:51.460 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:07:51 smithi193 ceph-mon[36878]: pgmap v90: 0 pgs: ; 0 B data, 53 MiB used, 179 GiB / 179 GiB avail 2024-03-24T00:07:51.460 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:07:51 smithi193 ceph-mon[36878]: Detected new or changed devices on smithi193 2024-03-24T00:07:51.460 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:07:51 smithi193 ceph-mon[36878]: Adjusting osd_memory_target on smithi193 to 17080M 2024-03-24T00:07:51.460 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:07:51 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]: dispatch 2024-03-24T00:07:51.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:07:51 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd='[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32, "yes_i_really_mean_it": true}]': finished 2024-03-24T00:07:51.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:07:51 smithi193 ceph-mon[36878]: osdmap e19: 3 total, 3 up, 3 in 2024-03-24T00:07:51.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:07:51 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]: dispatch 2024-03-24T00:07:52.400 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:07:52 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]': finished 2024-03-24T00:07:52.400 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:07:52 smithi118 ceph-mon[31535]: osdmap e20: 3 total, 3 up, 3 in 2024-03-24T00:07:52.400 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:07:52 smithi118 ceph-mon[31535]: pgmap v93: 1 pgs: 1 unknown; 0 B data, 80 MiB used, 268 GiB / 268 GiB avail 2024-03-24T00:07:52.704 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:07:52 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]': finished 2024-03-24T00:07:52.704 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:07:52 smithi189 ceph-mon[36187]: osdmap e20: 3 total, 3 up, 3 in 2024-03-24T00:07:52.704 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:07:52 smithi189 ceph-mon[36187]: pgmap v93: 1 pgs: 1 unknown; 0 B data, 80 MiB used, 268 GiB / 268 GiB avail 2024-03-24T00:07:52.710 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:07:52 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]': finished 2024-03-24T00:07:52.711 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:07:52 smithi193 ceph-mon[36878]: osdmap e20: 3 total, 3 up, 3 in 2024-03-24T00:07:52.711 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:07:52 smithi193 ceph-mon[36878]: pgmap v93: 1 pgs: 1 unknown; 0 B data, 80 MiB used, 268 GiB / 268 GiB avail 2024-03-24T00:07:52.996 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 24 00:07:52 smithi118 sudo[51012]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-03-24T00:07:52.996 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 24 00:07:52 smithi118 sudo[51012]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-24T00:07:52.996 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 24 00:07:52 smithi118 sudo[51012]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-24T00:07:52.997 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 24 00:07:52 smithi118 sudo[51012]: pam_unix(sudo:session): session closed for user root 2024-03-24T00:07:53.155 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:07:53.154+0000 7f44dda00700 1 -- 172.21.15.118:0/3142554416 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f44d81021d0 msgr2=0x7f44d81025f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:07:53.155 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:07:53.154+0000 7f44dda00700 1 --2- 172.21.15.118:0/3142554416 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f44d81021d0 0x7f44d81025f0 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f44c80099c0 tx=0x7f44c8009cd0 comp rx=0 tx=0).stop 2024-03-24T00:07:53.155 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:07:53.154+0000 7f44dda00700 1 -- 172.21.15.118:0/3142554416 shutdown_connections 2024-03-24T00:07:53.155 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:07:53.154+0000 7f44dda00700 1 --2- 172.21.15.118:0/3142554416 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f44d8103d70 0x7f44d810a620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:07:53.155 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:07:53.154+0000 7f44dda00700 1 --2- 172.21.15.118:0/3142554416 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f44d81021d0 0x7f44d81025f0 unknown :-1 s=CLOSED pgs=115 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:07:53.155 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:07:53.154+0000 7f44dda00700 1 --2- 172.21.15.118:0/3142554416 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f44d81033d0 0x7f44d8103830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:07:53.156 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:07:53.154+0000 7f44dda00700 1 -- 172.21.15.118:0/3142554416 >> 172.21.15.118:0/3142554416 conn(0x7f44d80fd730 msgr2=0x7f44d80ffbb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:07:53.156 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:07:53.154+0000 7f44dda00700 1 -- 172.21.15.118:0/3142554416 shutdown_connections 2024-03-24T00:07:53.156 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:07:53.154+0000 7f44dda00700 1 -- 172.21.15.118:0/3142554416 wait complete. 2024-03-24T00:07:53.156 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:07:53.155+0000 7f44dda00700 1 Processor -- start 2024-03-24T00:07:53.156 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:07:53.156+0000 7f44dda00700 1 -- start start 2024-03-24T00:07:53.157 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:07:53.156+0000 7f44dda00700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f44d81021d0 0x7f44d819a5e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:07:53.157 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:07:53.156+0000 7f44dda00700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f44d81033d0 0x7f44d819ab20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:07:53.157 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:07:53.156+0000 7f44dda00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f44d8103d70 0x7f44d81a1ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:07:53.157 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:07:53.156+0000 7f44dda00700 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f44d810ddc0 con 0x7f44d81033d0 2024-03-24T00:07:53.157 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:07:53.156+0000 7f44dda00700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f44d810dc40 con 0x7f44d8103d70 2024-03-24T00:07:53.157 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:07:53.156+0000 7f44dda00700 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f44d810df40 con 0x7f44d81021d0 2024-03-24T00:07:53.158 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:07:53.156+0000 7f44d6a00700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f44d81021d0 0x7f44d819a5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:07:53.158 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:07:53.156+0000 7f44cfe00700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f44d81033d0 0x7f44d819ab20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:07:53.158 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:07:53.156+0000 7f44d6a00700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f44d81021d0 0x7f44d819a5e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.118:52166/0 (socket says 172.21.15.118:52166) 2024-03-24T00:07:53.158 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:07:53.156+0000 7f44cfe00700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f44d81033d0 0x7f44d819ab20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.118:3300/0 says I am v2:172.21.15.118:45730/0 (socket says 172.21.15.118:45730) 2024-03-24T00:07:53.158 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:07:53.156+0000 7f44d6a00700 1 -- 172.21.15.118:0/2039767198 learned_addr learned my addr 172.21.15.118:0/2039767198 (peer_addr_for_me v2:172.21.15.118:0/0) 2024-03-24T00:07:53.158 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:07:53.156+0000 7f44d7400700 1 --2- 172.21.15.118:0/2039767198 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f44d8103d70 0x7f44d81a1ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:07:53.158 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:07:53.157+0000 7f44cfe00700 1 -- 172.21.15.118:0/2039767198 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f44d81021d0 msgr2=0x7f44d819a5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:07:53.158 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:07:53.157+0000 7f44cfe00700 1 --2- 172.21.15.118:0/2039767198 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f44d81021d0 0x7f44d819a5e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:07:53.158 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:07:53.157+0000 7f44cfe00700 1 -- 172.21.15.118:0/2039767198 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f44d8103d70 msgr2=0x7f44d81a1ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:07:53.158 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:07:53.157+0000 7f44cfe00700 1 --2- 172.21.15.118:0/2039767198 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f44d8103d70 0x7f44d81a1ba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:07:53.158 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:07:53.157+0000 7f44cfe00700 1 -- 172.21.15.118:0/2039767198 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f44d81a2240 con 0x7f44d81033d0 2024-03-24T00:07:53.158 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:07:53.157+0000 7f44cfe00700 1 --2- 172.21.15.118:0/2039767198 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f44d81033d0 0x7f44d819ab20 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f44c000ea20 tx=0x7f44c000ed30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:07:53.158 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:07:53.157+0000 7f44cf400700 1 -- 172.21.15.118:0/2039767198 <== mon.0 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f44c0004220 con 0x7f44d81033d0 2024-03-24T00:07:53.158 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:07:53.157+0000 7f44cf400700 1 -- 172.21.15.118:0/2039767198 <== mon.0 v2:172.21.15.118:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f44c000ce50 con 0x7f44d81033d0 2024-03-24T00:07:53.159 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:07:53.157+0000 7f44dda00700 1 -- 172.21.15.118:0/2039767198 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f44d81a2530 con 0x7f44d81033d0 2024-03-24T00:07:53.159 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:07:53.157+0000 7f44cf400700 1 -- 172.21.15.118:0/2039767198 <== mon.0 v2:172.21.15.118:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f44c0003dc0 con 0x7f44d81033d0 2024-03-24T00:07:53.159 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:07:53.157+0000 7f44dda00700 1 -- 172.21.15.118:0/2039767198 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f44d81a2ac0 con 0x7f44d81033d0 2024-03-24T00:07:53.159 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:07:53.158+0000 7f44dda00700 1 -- 172.21.15.118:0/2039767198 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f44d8068e50 con 0x7f44d81033d0 2024-03-24T00:07:53.161 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:07:53.162+0000 7f44cf400700 1 -- 172.21.15.118:0/2039767198 <== mon.0 v2:172.21.15.118:3300/0 4 ==== mgrmap(e 15) v1 ==== 137235+0+0 (secure 0 0 0) 0x7f44c000c970 con 0x7f44d81033d0 2024-03-24T00:07:53.162 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:07:53.163+0000 7f44cf400700 1 --2- 172.21.15.118:0/2039767198 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7f44bc0a2880 0x7f44bc0a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:07:53.163 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:07:53.163+0000 7f44cf400700 1 -- 172.21.15.118:0/2039767198 <== mon.0 v2:172.21.15.118:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 3237+0+0 (secure 0 0 0) 0x7f44c00cd820 con 0x7f44d81033d0 2024-03-24T00:07:53.163 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:07:53.164+0000 7f44d6a00700 1 --2- 172.21.15.118:0/2039767198 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7f44bc0a2880 0x7f44bc0a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:07:53.163 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:07:53.164+0000 7f44d6a00700 1 --2- 172.21.15.118:0/2039767198 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7f44bc0a2880 0x7f44bc0a4d40 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f44c8000c00 tx=0x7f44c8005fb0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:07:53.164 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:07:53.165+0000 7f44cf400700 1 -- 172.21.15.118:0/2039767198 <== mon.0 v2:172.21.15.118:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f44c009bd60 con 0x7f44d81033d0 2024-03-24T00:07:53.390 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:07:53.391+0000 7f44dda00700 1 -- 172.21.15.118:0/2039767198 --> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi118:vg_nvme/lv_3", "target": ["mon-mgr", ""]}) v1 -- 0x7f44d8075ae0 con 0x7f44bc0a2880 2024-03-24T00:07:53.391 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:07:53 smithi118 ceph-mon[31535]: osdmap e21: 3 total, 3 up, 3 in 2024-03-24T00:07:53.696 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 24 00:07:53 smithi118 sudo[51044]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-03-24T00:07:53.696 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 24 00:07:53 smithi118 sudo[51044]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-24T00:07:53.696 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 24 00:07:53 smithi118 sudo[51044]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-24T00:07:53.696 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 24 00:07:53 smithi118 sudo[51044]: pam_unix(sudo:session): session closed for user root 2024-03-24T00:07:53.703 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:07:53 smithi189 ceph-mon[36187]: osdmap e21: 3 total, 3 up, 3 in 2024-03-24T00:07:53.710 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:07:53 smithi193 ceph-mon[36878]: osdmap e21: 3 total, 3 up, 3 in 2024-03-24T00:07:53.961 INFO:journalctl@ceph.osd.1.smithi189.stdout:Mar 24 00:07:53 smithi189 sudo[44971]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-03-24T00:07:53.961 INFO:journalctl@ceph.osd.1.smithi189.stdout:Mar 24 00:07:53 smithi189 sudo[44971]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-24T00:07:53.961 INFO:journalctl@ceph.osd.1.smithi189.stdout:Mar 24 00:07:53 smithi189 sudo[44971]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-24T00:07:53.961 INFO:journalctl@ceph.osd.1.smithi189.stdout:Mar 24 00:07:53 smithi189 sudo[44971]: pam_unix(sudo:session): session closed for user root 2024-03-24T00:07:54.229 INFO:journalctl@ceph.osd.1.smithi189.stdout:Mar 24 00:07:53 smithi189 sudo[44974]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-03-24T00:07:54.229 INFO:journalctl@ceph.osd.1.smithi189.stdout:Mar 24 00:07:53 smithi189 sudo[44974]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-24T00:07:54.229 INFO:journalctl@ceph.osd.1.smithi189.stdout:Mar 24 00:07:53 smithi189 sudo[44974]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-24T00:07:54.229 INFO:journalctl@ceph.osd.1.smithi189.stdout:Mar 24 00:07:53 smithi189 sudo[44974]: pam_unix(sudo:session): session closed for user root 2024-03-24T00:07:54.488 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:07:54.488+0000 7f44cf400700 1 -- 172.21.15.118:0/2039767198 <== mon.0 v2:172.21.15.118:3300/0 7 ==== mgrmap(e 16) v1 ==== 137291+0+0 (secure 0 0 0) 0x7f44c0097cd0 con 0x7f44d81033d0 2024-03-24T00:07:54.488 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:07:54 smithi118 ceph-mon[31535]: from='client.14247 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi118:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T00:07:54.488 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:07:54 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-24T00:07:54.489 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:07:54 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-24T00:07:54.489 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:07:54 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T00:07:54.489 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:07:54 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T00:07:54.489 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:07:54 smithi118 ceph-mon[31535]: pgmap v95: 1 pgs: 1 creating+activating; 0 B data, 80 MiB used, 268 GiB / 268 GiB avail 2024-03-24T00:07:54.489 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:07:54 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T00:07:54.491 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:07:54 smithi193 ceph-mon[36878]: from='client.14247 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi118:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T00:07:54.491 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:07:54 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-24T00:07:54.491 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:07:54 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-24T00:07:54.491 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:07:54 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T00:07:54.491 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:07:54 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T00:07:54.491 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:07:54 smithi193 ceph-mon[36878]: pgmap v95: 1 pgs: 1 creating+activating; 0 B data, 80 MiB used, 268 GiB / 268 GiB avail 2024-03-24T00:07:54.492 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:07:54 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T00:07:54.492 INFO:journalctl@ceph.osd.2.smithi193.stdout:Mar 24 00:07:54 smithi193 sudo[46333]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-03-24T00:07:54.492 INFO:journalctl@ceph.osd.2.smithi193.stdout:Mar 24 00:07:54 smithi193 sudo[46333]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-24T00:07:54.492 INFO:journalctl@ceph.osd.2.smithi193.stdout:Mar 24 00:07:54 smithi193 sudo[46333]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-24T00:07:54.492 INFO:journalctl@ceph.osd.2.smithi193.stdout:Mar 24 00:07:54 smithi193 sudo[46333]: pam_unix(sudo:session): session closed for user root 2024-03-24T00:07:54.703 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:07:54 smithi189 ceph-mon[36187]: from='client.14247 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi118:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T00:07:54.704 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:07:54 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-24T00:07:54.704 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:07:54 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-24T00:07:54.704 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:07:54 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T00:07:54.704 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:07:54 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T00:07:54.704 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:07:54 smithi189 ceph-mon[36187]: pgmap v95: 1 pgs: 1 creating+activating; 0 B data, 80 MiB used, 268 GiB / 268 GiB avail 2024-03-24T00:07:54.704 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:07:54 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T00:07:54.961 INFO:journalctl@ceph.osd.2.smithi193.stdout:Mar 24 00:07:54 smithi193 sudo[46336]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-03-24T00:07:54.961 INFO:journalctl@ceph.osd.2.smithi193.stdout:Mar 24 00:07:54 smithi193 sudo[46336]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-24T00:07:54.961 INFO:journalctl@ceph.osd.2.smithi193.stdout:Mar 24 00:07:54 smithi193 sudo[46336]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-24T00:07:54.961 INFO:journalctl@ceph.osd.2.smithi193.stdout:Mar 24 00:07:54 smithi193 sudo[46336]: pam_unix(sudo:session): session closed for user root 2024-03-24T00:07:55.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:07:55 smithi118 sudo[51185]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-03-24T00:07:55.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:07:55 smithi118 sudo[51185]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-24T00:07:55.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:07:55 smithi118 sudo[51185]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-24T00:07:55.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:07:55 smithi118 sudo[51185]: pam_unix(sudo:session): session closed for user root 2024-03-24T00:07:55.725 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:07:55 smithi118 sudo[51209]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-03-24T00:07:55.726 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:07:55 smithi118 sudo[51209]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-24T00:07:55.726 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:07:55 smithi118 sudo[51209]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-24T00:07:55.726 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:07:55 smithi118 sudo[51209]: pam_unix(sudo:session): session closed for user root 2024-03-24T00:07:55.726 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:07:55 smithi118 ceph-mon[31535]: mgrmap e16: x(active, since 2m), standbys: z, y 2024-03-24T00:07:55.726 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:07:55 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T00:07:55.726 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:07:55 smithi118 ceph-mon[31535]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-03-24T00:07:55.921 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:07:55 smithi193 ceph-mon[36878]: mgrmap e16: x(active, since 2m), standbys: z, y 2024-03-24T00:07:55.922 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:07:55 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T00:07:55.922 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:07:55 smithi193 ceph-mon[36878]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-03-24T00:07:55.922 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:07:55 smithi193 sudo[46339]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-03-24T00:07:55.922 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:07:55 smithi193 sudo[46339]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-24T00:07:55.922 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:07:55 smithi193 sudo[46339]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-24T00:07:55.922 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:07:55 smithi193 sudo[46339]: pam_unix(sudo:session): session closed for user root 2024-03-24T00:07:55.953 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:07:55 smithi189 ceph-mon[36187]: mgrmap e16: x(active, since 2m), standbys: z, y 2024-03-24T00:07:55.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:07:55 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T00:07:55.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:07:55 smithi189 ceph-mon[36187]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-03-24T00:07:56.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:07:56 smithi193 sudo[46342]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-03-24T00:07:56.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:07:56 smithi193 sudo[46342]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-24T00:07:56.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:07:56 smithi193 sudo[46342]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-24T00:07:56.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:07:56 smithi193 sudo[46342]: pam_unix(sudo:session): session closed for user root 2024-03-24T00:07:56.653 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:07:56 smithi189 sudo[44977]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-03-24T00:07:56.654 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:07:56 smithi189 sudo[44977]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-24T00:07:56.654 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:07:56 smithi189 sudo[44977]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-24T00:07:56.654 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:07:56 smithi189 sudo[44977]: pam_unix(sudo:session): session closed for user root 2024-03-24T00:07:56.654 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:07:56 smithi189 sudo[44980]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-03-24T00:07:56.654 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:07:56 smithi189 sudo[44980]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-24T00:07:56.654 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:07:56 smithi189 sudo[44980]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-24T00:07:56.654 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:07:56 smithi189 sudo[44980]: pam_unix(sudo:session): session closed for user root 2024-03-24T00:07:56.915 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:07:56 smithi189 ceph-mon[36187]: pgmap v96: 1 pgs: 1 creating+activating; 0 B data, 80 MiB used, 268 GiB / 268 GiB avail 2024-03-24T00:07:56.915 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:07:56 smithi189 ceph-mon[36187]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-03-24T00:07:56.915 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:07:56 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-24T00:07:56.915 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:07:56 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-24T00:07:56.915 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:07:56 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-24T00:07:56.915 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:07:56 smithi189 ceph-mon[36187]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-03-24T00:07:56.915 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:07:56 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-24T00:07:56.915 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:07:56 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-24T00:07:56.915 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:07:56 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-24T00:07:56.916 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:07:56 smithi189 ceph-mon[36187]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-03-24T00:07:56.916 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:07:56 smithi189 ceph-mon[36187]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-03-24T00:07:56.916 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:07:56 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-24T00:07:56.916 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:07:56 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-24T00:07:56.916 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:07:56 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-24T00:07:56.916 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:07:56 smithi189 ceph-mon[36187]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-03-24T00:07:56.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:07:56 smithi118 ceph-mon[31535]: pgmap v96: 1 pgs: 1 creating+activating; 0 B data, 80 MiB used, 268 GiB / 268 GiB avail 2024-03-24T00:07:56.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:07:56 smithi118 ceph-mon[31535]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-03-24T00:07:56.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:07:56 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-24T00:07:56.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:07:56 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-24T00:07:56.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:07:56 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-24T00:07:56.947 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:07:56 smithi118 ceph-mon[31535]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-03-24T00:07:56.947 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:07:56 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-24T00:07:56.947 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:07:56 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-24T00:07:56.947 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:07:56 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-24T00:07:56.947 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:07:56 smithi118 ceph-mon[31535]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-03-24T00:07:56.947 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:07:56 smithi118 ceph-mon[31535]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-03-24T00:07:56.947 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:07:56 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-24T00:07:56.947 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:07:56 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-24T00:07:56.947 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:07:56 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-24T00:07:56.948 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:07:56 smithi118 ceph-mon[31535]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-03-24T00:07:56.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:07:56 smithi193 ceph-mon[36878]: pgmap v96: 1 pgs: 1 creating+activating; 0 B data, 80 MiB used, 268 GiB / 268 GiB avail 2024-03-24T00:07:56.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:07:56 smithi193 ceph-mon[36878]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-03-24T00:07:56.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:07:56 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-24T00:07:56.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:07:56 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-24T00:07:56.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:07:56 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-24T00:07:56.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:07:56 smithi193 ceph-mon[36878]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-03-24T00:07:56.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:07:56 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-24T00:07:56.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:07:56 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-24T00:07:56.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:07:56 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-24T00:07:56.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:07:56 smithi193 ceph-mon[36878]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-03-24T00:07:56.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:07:56 smithi193 ceph-mon[36878]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-03-24T00:07:56.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:07:56 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "a"}]: dispatch 2024-03-24T00:07:56.962 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:07:56 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "b"}]: dispatch 2024-03-24T00:07:56.962 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:07:56 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "mon metadata", "id": "c"}]: dispatch 2024-03-24T00:07:56.962 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:07:56 smithi193 ceph-mon[36878]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-03-24T00:07:57.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:07:57 smithi118 ceph-mon[31535]: from='client.? 172.21.15.118:0/1064477153' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "63fed43b-43ce-4031-8957-7fe94725c1ed"}]: dispatch 2024-03-24T00:07:57.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:07:57 smithi118 ceph-mon[31535]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "63fed43b-43ce-4031-8957-7fe94725c1ed"}]: dispatch 2024-03-24T00:07:57.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:07:57 smithi118 ceph-mon[31535]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "63fed43b-43ce-4031-8957-7fe94725c1ed"}]': finished 2024-03-24T00:07:57.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:07:57 smithi118 ceph-mon[31535]: osdmap e22: 4 total, 3 up, 4 in 2024-03-24T00:07:57.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:07:57 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-24T00:07:57.953 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:07:57 smithi189 ceph-mon[36187]: from='client.? 172.21.15.118:0/1064477153' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "63fed43b-43ce-4031-8957-7fe94725c1ed"}]: dispatch 2024-03-24T00:07:57.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:07:57 smithi189 ceph-mon[36187]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "63fed43b-43ce-4031-8957-7fe94725c1ed"}]: dispatch 2024-03-24T00:07:57.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:07:57 smithi189 ceph-mon[36187]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "63fed43b-43ce-4031-8957-7fe94725c1ed"}]': finished 2024-03-24T00:07:57.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:07:57 smithi189 ceph-mon[36187]: osdmap e22: 4 total, 3 up, 4 in 2024-03-24T00:07:57.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:07:57 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-24T00:07:57.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:07:57 smithi193 ceph-mon[36878]: from='client.? 172.21.15.118:0/1064477153' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "63fed43b-43ce-4031-8957-7fe94725c1ed"}]: dispatch 2024-03-24T00:07:57.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:07:57 smithi193 ceph-mon[36878]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "63fed43b-43ce-4031-8957-7fe94725c1ed"}]: dispatch 2024-03-24T00:07:57.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:07:57 smithi193 ceph-mon[36878]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "63fed43b-43ce-4031-8957-7fe94725c1ed"}]': finished 2024-03-24T00:07:57.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:07:57 smithi193 ceph-mon[36878]: osdmap e22: 4 total, 3 up, 4 in 2024-03-24T00:07:57.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:07:57 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-24T00:07:58.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:07:58 smithi118 ceph-mon[31535]: pgmap v98: 1 pgs: 1 active+clean; 577 KiB data, 80 MiB used, 268 GiB / 268 GiB avail 2024-03-24T00:07:58.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:07:58 smithi118 ceph-mon[31535]: from='client.? 172.21.15.118:0/1860570796' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-24T00:07:58.953 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:07:58 smithi189 ceph-mon[36187]: pgmap v98: 1 pgs: 1 active+clean; 577 KiB data, 80 MiB used, 268 GiB / 268 GiB avail 2024-03-24T00:07:58.953 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:07:58 smithi189 ceph-mon[36187]: from='client.? 172.21.15.118:0/1860570796' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-24T00:07:58.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:07:58 smithi193 ceph-mon[36878]: pgmap v98: 1 pgs: 1 active+clean; 577 KiB data, 80 MiB used, 268 GiB / 268 GiB avail 2024-03-24T00:07:58.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:07:58 smithi193 ceph-mon[36878]: from='client.? 172.21.15.118:0/1860570796' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-24T00:08:00.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:08:00 smithi118 ceph-mon[31535]: pgmap v99: 1 pgs: 1 active+clean; 577 KiB data, 80 MiB used, 268 GiB / 268 GiB avail 2024-03-24T00:08:00.953 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:08:00 smithi189 ceph-mon[36187]: pgmap v99: 1 pgs: 1 active+clean; 577 KiB data, 80 MiB used, 268 GiB / 268 GiB avail 2024-03-24T00:08:00.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:08:00 smithi193 ceph-mon[36878]: pgmap v99: 1 pgs: 1 active+clean; 577 KiB data, 80 MiB used, 268 GiB / 268 GiB avail 2024-03-24T00:08:02.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:08:02 smithi118 ceph-mon[31535]: pgmap v100: 1 pgs: 1 active+clean; 577 KiB data, 80 MiB used, 268 GiB / 268 GiB avail 2024-03-24T00:08:02.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:08:02 smithi189 ceph-mon[36187]: pgmap v100: 1 pgs: 1 active+clean; 577 KiB data, 80 MiB used, 268 GiB / 268 GiB avail 2024-03-24T00:08:02.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:08:02 smithi193 ceph-mon[36878]: pgmap v100: 1 pgs: 1 active+clean; 577 KiB data, 80 MiB used, 268 GiB / 268 GiB avail 2024-03-24T00:08:04.835 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:08:04 smithi118 ceph-mon[31535]: pgmap v101: 1 pgs: 1 active+clean; 577 KiB data, 80 MiB used, 268 GiB / 268 GiB avail; 1.4 KiB/s rd, 41 KiB/s wr, 3 op/s 2024-03-24T00:08:04.953 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:08:04 smithi189 ceph-mon[36187]: pgmap v101: 1 pgs: 1 active+clean; 577 KiB data, 80 MiB used, 268 GiB / 268 GiB avail; 1.4 KiB/s rd, 41 KiB/s wr, 3 op/s 2024-03-24T00:08:04.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:08:04 smithi193 ceph-mon[36878]: pgmap v101: 1 pgs: 1 active+clean; 577 KiB data, 80 MiB used, 268 GiB / 268 GiB avail; 1.4 KiB/s rd, 41 KiB/s wr, 3 op/s 2024-03-24T00:08:06.953 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:08:06 smithi189 ceph-mon[36187]: pgmap v102: 1 pgs: 1 active+clean; 577 KiB data, 80 MiB used, 268 GiB / 268 GiB avail; 1.1 KiB/s rd, 31 KiB/s wr, 2 op/s 2024-03-24T00:08:06.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:08:06 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-03-24T00:08:06.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:08:06 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T00:08:06.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:08:06 smithi193 ceph-mon[36878]: pgmap v102: 1 pgs: 1 active+clean; 577 KiB data, 80 MiB used, 268 GiB / 268 GiB avail; 1.1 KiB/s rd, 31 KiB/s wr, 2 op/s 2024-03-24T00:08:06.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:08:06 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-03-24T00:08:06.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:08:06 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T00:08:07.195 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:08:06 smithi118 ceph-mon[31535]: pgmap v102: 1 pgs: 1 active+clean; 577 KiB data, 80 MiB used, 268 GiB / 268 GiB avail; 1.1 KiB/s rd, 31 KiB/s wr, 2 op/s 2024-03-24T00:08:07.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:08:06 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-03-24T00:08:07.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:08:06 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T00:08:07.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:08:07 smithi118 ceph-mon[31535]: Deploying daemon osd.3 on smithi118 2024-03-24T00:08:07.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:08:07 smithi193 ceph-mon[36878]: Deploying daemon osd.3 on smithi118 2024-03-24T00:08:08.203 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:08:07 smithi189 ceph-mon[36187]: Deploying daemon osd.3 on smithi118 2024-03-24T00:08:08.908 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:08:08 smithi118 ceph-mon[31535]: pgmap v103: 1 pgs: 1 active+clean; 577 KiB data, 80 MiB used, 268 GiB / 268 GiB avail; 891 B/s rd, 25 KiB/s wr, 2 op/s 2024-03-24T00:08:08.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:08:08 smithi193 ceph-mon[36878]: pgmap v103: 1 pgs: 1 active+clean; 577 KiB data, 80 MiB used, 268 GiB / 268 GiB avail; 891 B/s rd, 25 KiB/s wr, 2 op/s 2024-03-24T00:08:09.203 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:08:08 smithi189 ceph-mon[36187]: pgmap v103: 1 pgs: 1 active+clean; 577 KiB data, 80 MiB used, 268 GiB / 268 GiB avail; 891 B/s rd, 25 KiB/s wr, 2 op/s 2024-03-24T00:08:09.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:08:09 smithi118 ceph-mon[31535]: pgmap v104: 1 pgs: 1 active+clean; 577 KiB data, 80 MiB used, 268 GiB / 268 GiB avail; 767 B/s rd, 22 KiB/s wr, 1 op/s 2024-03-24T00:08:10.203 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:08:09 smithi189 ceph-mon[36187]: pgmap v104: 1 pgs: 1 active+clean; 577 KiB data, 80 MiB used, 268 GiB / 268 GiB avail; 767 B/s rd, 22 KiB/s wr, 1 op/s 2024-03-24T00:08:10.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:08:09 smithi193 ceph-mon[36878]: pgmap v104: 1 pgs: 1 active+clean; 577 KiB data, 80 MiB used, 268 GiB / 268 GiB avail; 767 B/s rd, 22 KiB/s wr, 1 op/s 2024-03-24T00:08:12.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:08:12 smithi118 ceph-mon[31535]: pgmap v105: 1 pgs: 1 active+clean; 577 KiB data, 80 MiB used, 268 GiB / 268 GiB avail; 767 B/s rd, 22 KiB/s wr, 1 op/s 2024-03-24T00:08:12.953 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:08:12 smithi189 ceph-mon[36187]: pgmap v105: 1 pgs: 1 active+clean; 577 KiB data, 80 MiB used, 268 GiB / 268 GiB avail; 767 B/s rd, 22 KiB/s wr, 1 op/s 2024-03-24T00:08:12.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:08:12 smithi193 ceph-mon[36878]: pgmap v105: 1 pgs: 1 active+clean; 577 KiB data, 80 MiB used, 268 GiB / 268 GiB avail; 767 B/s rd, 22 KiB/s wr, 1 op/s 2024-03-24T00:08:14.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:08:14 smithi118 ceph-mon[31535]: pgmap v106: 1 pgs: 1 active+clean; 577 KiB data, 80 MiB used, 268 GiB / 268 GiB avail; 767 B/s rd, 22 KiB/s wr, 1 op/s 2024-03-24T00:08:14.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:08:14 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T00:08:14.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:08:14 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T00:08:14.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:08:14 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T00:08:14.953 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:08:14 smithi189 ceph-mon[36187]: pgmap v106: 1 pgs: 1 active+clean; 577 KiB data, 80 MiB used, 268 GiB / 268 GiB avail; 767 B/s rd, 22 KiB/s wr, 1 op/s 2024-03-24T00:08:14.953 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:08:14 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T00:08:14.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:08:14 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T00:08:14.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:08:14 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T00:08:14.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:08:14 smithi193 ceph-mon[36878]: pgmap v106: 1 pgs: 1 active+clean; 577 KiB data, 80 MiB used, 268 GiB / 268 GiB avail; 767 B/s rd, 22 KiB/s wr, 1 op/s 2024-03-24T00:08:14.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:08:14 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T00:08:14.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:08:14 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T00:08:14.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:08:14 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T00:08:16.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:08:16 smithi118 ceph-mon[31535]: pgmap v107: 1 pgs: 1 active+clean; 577 KiB data, 80 MiB used, 268 GiB / 268 GiB avail 2024-03-24T00:08:16.953 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:08:16 smithi189 ceph-mon[36187]: pgmap v107: 1 pgs: 1 active+clean; 577 KiB data, 80 MiB used, 268 GiB / 268 GiB avail 2024-03-24T00:08:16.962 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:08:16 smithi193 ceph-mon[36878]: pgmap v107: 1 pgs: 1 active+clean; 577 KiB data, 80 MiB used, 268 GiB / 268 GiB avail 2024-03-24T00:08:18.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:08:18 smithi118 ceph-mon[31535]: pgmap v108: 1 pgs: 1 active+clean; 577 KiB data, 80 MiB used, 268 GiB / 268 GiB avail 2024-03-24T00:08:18.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:08:18 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T00:08:18.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:08:18 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T00:08:18.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:08:18 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T00:08:18.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:08:18 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T00:08:18.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:08:18 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T00:08:18.697 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:08:18 smithi118 ceph-mon[31535]: from='osd.3 [v2:172.21.15.118:6810/4059617010,v1:172.21.15.118:6811/4059617010]' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-03-24T00:08:18.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:08:18 smithi189 ceph-mon[36187]: pgmap v108: 1 pgs: 1 active+clean; 577 KiB data, 80 MiB used, 268 GiB / 268 GiB avail 2024-03-24T00:08:18.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:08:18 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T00:08:18.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:08:18 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T00:08:18.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:08:18 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T00:08:18.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:08:18 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T00:08:18.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:08:18 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T00:08:18.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:08:18 smithi189 ceph-mon[36187]: from='osd.3 [v2:172.21.15.118:6810/4059617010,v1:172.21.15.118:6811/4059617010]' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-03-24T00:08:18.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:08:18 smithi193 ceph-mon[36878]: pgmap v108: 1 pgs: 1 active+clean; 577 KiB data, 80 MiB used, 268 GiB / 268 GiB avail 2024-03-24T00:08:18.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:08:18 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T00:08:18.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:08:18 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T00:08:18.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:08:18 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T00:08:18.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:08:18 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T00:08:18.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:08:18 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T00:08:18.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:08:18 smithi193 ceph-mon[36878]: from='osd.3 [v2:172.21.15.118:6810/4059617010,v1:172.21.15.118:6811/4059617010]' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-03-24T00:08:19.479 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:08:19.478+0000 7f44cf400700 1 -- 172.21.15.118:0/2039767198 <== mgr.14152 v2:172.21.15.118:6800/2112912943 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7f44d8075ae0 con 0x7f44bc0a2880 2024-03-24T00:08:19.479 INFO:teuthology.orchestra.run.smithi118.stdout:Created osd(s) 3 on host 'smithi118' 2024-03-24T00:08:19.482 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:08:19.481+0000 7f44dda00700 1 -- 172.21.15.118:0/2039767198 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7f44bc0a2880 msgr2=0x7f44bc0a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:08:19.482 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:08:19.481+0000 7f44dda00700 1 --2- 172.21.15.118:0/2039767198 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7f44bc0a2880 0x7f44bc0a4d40 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f44c8000c00 tx=0x7f44c8005fb0 comp rx=0 tx=0).stop 2024-03-24T00:08:19.483 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:08:19.481+0000 7f44dda00700 1 -- 172.21.15.118:0/2039767198 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f44d81033d0 msgr2=0x7f44d819ab20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:08:19.483 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:08:19.481+0000 7f44dda00700 1 --2- 172.21.15.118:0/2039767198 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f44d81033d0 0x7f44d819ab20 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f44c000ea20 tx=0x7f44c000ed30 comp rx=0 tx=0).stop 2024-03-24T00:08:19.483 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:08:19.481+0000 7f44dda00700 1 -- 172.21.15.118:0/2039767198 shutdown_connections 2024-03-24T00:08:19.483 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:08:19.481+0000 7f44dda00700 1 --2- 172.21.15.118:0/2039767198 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f44d8103d70 0x7f44d81a1ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:08:19.483 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:08:19.481+0000 7f44dda00700 1 --2- 172.21.15.118:0/2039767198 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7f44bc0a2880 0x7f44bc0a4d40 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:08:19.483 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:08:19.481+0000 7f44dda00700 1 --2- 172.21.15.118:0/2039767198 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f44d81021d0 0x7f44d819a5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:08:19.483 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:08:19.481+0000 7f44dda00700 1 --2- 172.21.15.118:0/2039767198 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f44d81033d0 0x7f44d819ab20 unknown :-1 s=CLOSED pgs=116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:08:19.483 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:08:19.481+0000 7f44dda00700 1 -- 172.21.15.118:0/2039767198 >> 172.21.15.118:0/2039767198 conn(0x7f44d80fd730 msgr2=0x7f44d80743c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:08:19.483 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:08:19.482+0000 7f44dda00700 1 -- 172.21.15.118:0/2039767198 shutdown_connections 2024-03-24T00:08:19.483 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:08:19.482+0000 7f44dda00700 1 -- 172.21.15.118:0/2039767198 wait complete. 2024-03-24T00:08:20.203 DEBUG:teuthology.orchestra.run.smithi118:osd.3> sudo journalctl -f -n 0 -u ceph-d79a5420-e971-11ee-95cd-87774f69a715@osd.3.service 2024-03-24T00:08:20.208 INFO:tasks.cephadm:Deploying osd.4 on smithi189 with /dev/vg_nvme/lv_3... 2024-03-24T00:08:20.209 DEBUG:teuthology.orchestra.run.smithi189:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d79a5420-e971-11ee-95cd-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_3 2024-03-24T00:08:20.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:08:20 smithi118 ceph-mon[31535]: from='osd.3 [v2:172.21.15.118:6810/4059617010,v1:172.21.15.118:6811/4059617010]' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-03-24T00:08:20.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:08:20 smithi118 ceph-mon[31535]: osdmap e23: 4 total, 3 up, 4 in 2024-03-24T00:08:20.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:08:20 smithi118 ceph-mon[31535]: from='osd.3 [v2:172.21.15.118:6810/4059617010,v1:172.21.15.118:6811/4059617010]' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi118", "root=default"]}]: dispatch 2024-03-24T00:08:20.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:08:20 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-24T00:08:20.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:08:20 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T00:08:20.447 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:08:20 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T00:08:20.447 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:08:20 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T00:08:20.447 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:08:20 smithi118 ceph-mon[31535]: pgmap v110: 1 pgs: 1 active+clean; 577 KiB data, 80 MiB used, 268 GiB / 268 GiB avail 2024-03-24T00:08:20.457 INFO:teuthology.orchestra.run.smithi189.stderr:Inferring config /var/lib/ceph/d79a5420-e971-11ee-95cd-87774f69a715/mon.b/config 2024-03-24T00:08:20.578 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:08:20 smithi189 ceph-mon[36187]: from='osd.3 [v2:172.21.15.118:6810/4059617010,v1:172.21.15.118:6811/4059617010]' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-03-24T00:08:20.579 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:08:20 smithi189 ceph-mon[36187]: osdmap e23: 4 total, 3 up, 4 in 2024-03-24T00:08:20.579 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:08:20 smithi189 ceph-mon[36187]: from='osd.3 [v2:172.21.15.118:6810/4059617010,v1:172.21.15.118:6811/4059617010]' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi118", "root=default"]}]: dispatch 2024-03-24T00:08:20.579 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:08:20 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-24T00:08:20.579 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:08:20 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T00:08:20.579 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:08:20 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T00:08:20.579 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:08:20 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T00:08:20.579 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:08:20 smithi189 ceph-mon[36187]: pgmap v110: 1 pgs: 1 active+clean; 577 KiB data, 80 MiB used, 268 GiB / 268 GiB avail 2024-03-24T00:08:20.711 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:08:20 smithi193 ceph-mon[36878]: from='osd.3 [v2:172.21.15.118:6810/4059617010,v1:172.21.15.118:6811/4059617010]' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-03-24T00:08:20.711 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:08:20 smithi193 ceph-mon[36878]: osdmap e23: 4 total, 3 up, 4 in 2024-03-24T00:08:20.711 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:08:20 smithi193 ceph-mon[36878]: from='osd.3 [v2:172.21.15.118:6810/4059617010,v1:172.21.15.118:6811/4059617010]' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi118", "root=default"]}]: dispatch 2024-03-24T00:08:20.711 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:08:20 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-24T00:08:20.711 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:08:20 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T00:08:20.711 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:08:20 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T00:08:20.711 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:08:20 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T00:08:20.711 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:08:20 smithi193 ceph-mon[36878]: pgmap v110: 1 pgs: 1 active+clean; 577 KiB data, 80 MiB used, 268 GiB / 268 GiB avail 2024-03-24T00:08:21.196 INFO:journalctl@ceph.osd.3.smithi118.stdout:Mar 24 00:08:20 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-osd-3[53301]: 2024-03-24T00:08:20.857+0000 7fca0de00700 -1 osd.3 0 waiting for initial osdmap 2024-03-24T00:08:21.196 INFO:journalctl@ceph.osd.3.smithi118.stdout:Mar 24 00:08:20 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-osd-3[53301]: 2024-03-24T00:08:20.880+0000 7fca03c00700 -1 osd.3 25 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-24T00:08:21.677 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:08:21 smithi189 ceph-mon[36187]: from='osd.3 [v2:172.21.15.118:6810/4059617010,v1:172.21.15.118:6811/4059617010]' entity='osd.3' cmd='[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi118", "root=default"]}]': finished 2024-03-24T00:08:21.677 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:08:21 smithi189 ceph-mon[36187]: osdmap e24: 4 total, 3 up, 4 in 2024-03-24T00:08:21.677 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:08:21 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-24T00:08:21.677 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:08:21 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-24T00:08:21.677 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:08:21 smithi189 ceph-mon[36187]: osdmap e25: 4 total, 3 up, 4 in 2024-03-24T00:08:21.677 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:08:21 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-24T00:08:21.677 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:08:21 smithi189 ceph-mon[36187]: from='osd.3 [v2:172.21.15.118:6810/4059617010,v1:172.21.15.118:6811/4059617010]' entity='osd.3' 2024-03-24T00:08:21.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:08:21 smithi118 ceph-mon[31535]: from='osd.3 [v2:172.21.15.118:6810/4059617010,v1:172.21.15.118:6811/4059617010]' entity='osd.3' cmd='[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi118", "root=default"]}]': finished 2024-03-24T00:08:21.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:08:21 smithi118 ceph-mon[31535]: osdmap e24: 4 total, 3 up, 4 in 2024-03-24T00:08:21.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:08:21 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-24T00:08:21.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:08:21 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-24T00:08:21.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:08:21 smithi118 ceph-mon[31535]: osdmap e25: 4 total, 3 up, 4 in 2024-03-24T00:08:21.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:08:21 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-24T00:08:21.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:08:21 smithi118 ceph-mon[31535]: from='osd.3 [v2:172.21.15.118:6810/4059617010,v1:172.21.15.118:6811/4059617010]' entity='osd.3' 2024-03-24T00:08:21.710 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:08:21 smithi193 ceph-mon[36878]: from='osd.3 [v2:172.21.15.118:6810/4059617010,v1:172.21.15.118:6811/4059617010]' entity='osd.3' cmd='[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi118", "root=default"]}]': finished 2024-03-24T00:08:21.710 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:08:21 smithi193 ceph-mon[36878]: osdmap e24: 4 total, 3 up, 4 in 2024-03-24T00:08:21.711 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:08:21 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-24T00:08:21.711 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:08:21 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-24T00:08:21.711 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:08:21 smithi193 ceph-mon[36878]: osdmap e25: 4 total, 3 up, 4 in 2024-03-24T00:08:21.711 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:08:21 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-24T00:08:21.711 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:08:21 smithi193 ceph-mon[36878]: from='osd.3 [v2:172.21.15.118:6810/4059617010,v1:172.21.15.118:6811/4059617010]' entity='osd.3' 2024-03-24T00:08:22.580 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:08:22 smithi189 ceph-mon[36187]: purged_snaps scrub starts 2024-03-24T00:08:22.580 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:08:22 smithi189 ceph-mon[36187]: purged_snaps scrub ok 2024-03-24T00:08:22.580 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:08:22 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-24T00:08:22.580 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:08:22 smithi189 ceph-mon[36187]: pgmap v113: 1 pgs: 1 active+clean; 577 KiB data, 80 MiB used, 268 GiB / 268 GiB avail 2024-03-24T00:08:22.580 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:08:22 smithi189 ceph-mon[36187]: osd.3 [v2:172.21.15.118:6810/4059617010,v1:172.21.15.118:6811/4059617010] boot 2024-03-24T00:08:22.580 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:08:22 smithi189 ceph-mon[36187]: osdmap e26: 4 total, 4 up, 4 in 2024-03-24T00:08:22.580 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:08:22 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-24T00:08:22.595 INFO:teuthology.orchestra.run.smithi189.stderr:--> Zapping: /dev/vg_nvme/lv_3 2024-03-24T00:08:22.595 INFO:teuthology.orchestra.run.smithi189.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_3 bs=1M count=10 conv=fsync 2024-03-24T00:08:22.595 INFO:teuthology.orchestra.run.smithi189.stderr: stderr: 10+0 records in 2024-03-24T00:08:22.595 INFO:teuthology.orchestra.run.smithi189.stderr:10+0 records out 2024-03-24T00:08:22.595 INFO:teuthology.orchestra.run.smithi189.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0113662 s, 923 MB/s 2024-03-24T00:08:22.595 INFO:teuthology.orchestra.run.smithi189.stderr:--> Zapping successful for: 2024-03-24T00:08:22.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:08:22 smithi118 ceph-mon[31535]: purged_snaps scrub starts 2024-03-24T00:08:22.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:08:22 smithi118 ceph-mon[31535]: purged_snaps scrub ok 2024-03-24T00:08:22.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:08:22 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-24T00:08:22.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:08:22 smithi118 ceph-mon[31535]: pgmap v113: 1 pgs: 1 active+clean; 577 KiB data, 80 MiB used, 268 GiB / 268 GiB avail 2024-03-24T00:08:22.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:08:22 smithi118 ceph-mon[31535]: osd.3 [v2:172.21.15.118:6810/4059617010,v1:172.21.15.118:6811/4059617010] boot 2024-03-24T00:08:22.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:08:22 smithi118 ceph-mon[31535]: osdmap e26: 4 total, 4 up, 4 in 2024-03-24T00:08:22.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:08:22 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-24T00:08:22.710 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:08:22 smithi193 ceph-mon[36878]: purged_snaps scrub starts 2024-03-24T00:08:22.710 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:08:22 smithi193 ceph-mon[36878]: purged_snaps scrub ok 2024-03-24T00:08:22.710 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:08:22 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-24T00:08:22.710 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:08:22 smithi193 ceph-mon[36878]: pgmap v113: 1 pgs: 1 active+clean; 577 KiB data, 80 MiB used, 268 GiB / 268 GiB avail 2024-03-24T00:08:22.711 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:08:22 smithi193 ceph-mon[36878]: osd.3 [v2:172.21.15.118:6810/4059617010,v1:172.21.15.118:6811/4059617010] boot 2024-03-24T00:08:22.711 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:08:22 smithi193 ceph-mon[36878]: osdmap e26: 4 total, 4 up, 4 in 2024-03-24T00:08:22.711 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:08:22 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-24T00:08:23.260 DEBUG:teuthology.orchestra.run.smithi189:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d79a5420-e971-11ee-95cd-87774f69a715 -- ceph orch daemon add osd smithi189:vg_nvme/lv_3 2024-03-24T00:08:23.506 INFO:teuthology.orchestra.run.smithi189.stderr:Inferring config /var/lib/ceph/d79a5420-e971-11ee-95cd-87774f69a715/mon.b/config 2024-03-24T00:08:23.871 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:08:23 smithi189 ceph-mon[36187]: osdmap e27: 4 total, 4 up, 4 in 2024-03-24T00:08:23.945 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:08:23 smithi118 ceph-mon[31535]: osdmap e27: 4 total, 4 up, 4 in 2024-03-24T00:08:23.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:08:23 smithi193 ceph-mon[36878]: osdmap e27: 4 total, 4 up, 4 in 2024-03-24T00:08:24.953 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:08:24 smithi189 ceph-mon[36187]: pgmap v116: 1 pgs: 1 remapped+peering; 577 KiB data, 111 MiB used, 358 GiB / 358 GiB avail 2024-03-24T00:08:24.953 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:08:24 smithi189 ceph-mon[36187]: osdmap e28: 4 total, 4 up, 4 in 2024-03-24T00:08:24.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:08:24 smithi193 ceph-mon[36878]: pgmap v116: 1 pgs: 1 remapped+peering; 577 KiB data, 111 MiB used, 358 GiB / 358 GiB avail 2024-03-24T00:08:24.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:08:24 smithi193 ceph-mon[36878]: osdmap e28: 4 total, 4 up, 4 in 2024-03-24T00:08:25.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:08:24 smithi118 ceph-mon[31535]: pgmap v116: 1 pgs: 1 remapped+peering; 577 KiB data, 111 MiB used, 358 GiB / 358 GiB avail 2024-03-24T00:08:25.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:08:24 smithi118 ceph-mon[31535]: osdmap e28: 4 total, 4 up, 4 in 2024-03-24T00:08:25.245 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-24T00:08:25.243+0000 7f4ce691c700 1 -- 172.21.15.189:0/1892601280 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4ce01033e0 msgr2=0x7f4ce0103840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:08:25.246 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-24T00:08:25.243+0000 7f4ce691c700 1 --2- 172.21.15.189:0/1892601280 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4ce01033e0 0x7f4ce0103840 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f4cd00099c0 tx=0x7f4cd0009cd0 comp rx=0 tx=0).stop 2024-03-24T00:08:25.246 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-24T00:08:25.243+0000 7f4ce691c700 1 -- 172.21.15.189:0/1892601280 shutdown_connections 2024-03-24T00:08:25.246 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-24T00:08:25.243+0000 7f4ce691c700 1 --2- 172.21.15.189:0/1892601280 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f4ce0103d80 0x7f4ce010a630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:08:25.246 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-24T00:08:25.243+0000 7f4ce691c700 1 --2- 172.21.15.189:0/1892601280 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f4ce01021e0 0x7f4ce0102600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:08:25.246 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-24T00:08:25.243+0000 7f4ce691c700 1 --2- 172.21.15.189:0/1892601280 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4ce01033e0 0x7f4ce0103840 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:08:25.246 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-24T00:08:25.243+0000 7f4ce691c700 1 -- 172.21.15.189:0/1892601280 >> 172.21.15.189:0/1892601280 conn(0x7f4ce00fd7a0 msgr2=0x7f4ce00ffbc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:08:25.246 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-24T00:08:25.243+0000 7f4ce691c700 1 -- 172.21.15.189:0/1892601280 shutdown_connections 2024-03-24T00:08:25.246 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-24T00:08:25.243+0000 7f4ce691c700 1 -- 172.21.15.189:0/1892601280 wait complete. 2024-03-24T00:08:25.246 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-24T00:08:25.244+0000 7f4ce691c700 1 Processor -- start 2024-03-24T00:08:25.246 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-24T00:08:25.244+0000 7f4ce691c700 1 -- start start 2024-03-24T00:08:25.247 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-24T00:08:25.244+0000 7f4ce691c700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f4ce01021e0 0x7f4ce0198380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:08:25.247 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-24T00:08:25.244+0000 7f4ce691c700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4ce01033e0 0x7f4ce01988c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:08:25.247 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-24T00:08:25.244+0000 7f4ce691c700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f4ce0103d80 0x7f4ce019f940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:08:25.247 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-24T00:08:25.244+0000 7f4ce691c700 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4ce010de30 con 0x7f4ce01021e0 2024-03-24T00:08:25.247 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-24T00:08:25.244+0000 7f4ce691c700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4ce010dcb0 con 0x7f4ce01033e0 2024-03-24T00:08:25.247 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-24T00:08:25.244+0000 7f4ce691c700 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4ce010dfb0 con 0x7f4ce0103d80 2024-03-24T00:08:25.247 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-24T00:08:25.244+0000 7f4cdf7fe700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4ce01033e0 0x7f4ce01988c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:08:25.247 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-24T00:08:25.244+0000 7f4cdf7fe700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4ce01033e0 0x7f4ce01988c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:49872/0 (socket says 172.21.15.189:49872) 2024-03-24T00:08:25.247 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-24T00:08:25.245+0000 7f4cdf7fe700 1 -- 172.21.15.189:0/3494068559 learned_addr learned my addr 172.21.15.189:0/3494068559 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-03-24T00:08:25.247 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-24T00:08:25.245+0000 7f4ce4eb9700 1 --2- 172.21.15.189:0/3494068559 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f4ce0103d80 0x7f4ce019f940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:08:25.247 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-24T00:08:25.245+0000 7f4cdf7fe700 1 -- 172.21.15.189:0/3494068559 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f4ce0103d80 msgr2=0x7f4ce019f940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:08:25.247 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-24T00:08:25.245+0000 7f4cdf7fe700 1 --2- 172.21.15.189:0/3494068559 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f4ce0103d80 0x7f4ce019f940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:08:25.247 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-24T00:08:25.245+0000 7f4cdf7fe700 1 -- 172.21.15.189:0/3494068559 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f4ce01021e0 msgr2=0x7f4ce0198380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:08:25.247 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-24T00:08:25.245+0000 7f4cdf7fe700 1 --2- 172.21.15.189:0/3494068559 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f4ce01021e0 0x7f4ce0198380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:08:25.248 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-24T00:08:25.245+0000 7f4cdf7fe700 1 -- 172.21.15.189:0/3494068559 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4ce019ffe0 con 0x7f4ce01033e0 2024-03-24T00:08:25.248 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-24T00:08:25.245+0000 7f4cdf7fe700 1 --2- 172.21.15.189:0/3494068559 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4ce01033e0 0x7f4ce01988c0 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f4cd0000c00 tx=0x7f4cd00057c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:08:25.248 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-24T00:08:25.245+0000 7f4cdd7fa700 1 -- 172.21.15.189:0/3494068559 <== mon.2 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4cd0027070 con 0x7f4ce01033e0 2024-03-24T00:08:25.248 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-24T00:08:25.245+0000 7f4ce691c700 1 -- 172.21.15.189:0/3494068559 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4ce01a0270 con 0x7f4ce01033e0 2024-03-24T00:08:25.248 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-24T00:08:25.245+0000 7f4cdd7fa700 1 -- 172.21.15.189:0/3494068559 <== mon.2 v2:172.21.15.189:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f4cd0004150 con 0x7f4ce01033e0 2024-03-24T00:08:25.248 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-24T00:08:25.245+0000 7f4cdd7fa700 1 -- 172.21.15.189:0/3494068559 <== mon.2 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4cd000f460 con 0x7f4ce01033e0 2024-03-24T00:08:25.248 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-24T00:08:25.245+0000 7f4ce691c700 1 -- 172.21.15.189:0/3494068559 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4ce01a0740 con 0x7f4ce01033e0 2024-03-24T00:08:25.249 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-24T00:08:25.246+0000 7f4ce691c700 1 -- 172.21.15.189:0/3494068559 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4ce0068e50 con 0x7f4ce01033e0 2024-03-24T00:08:25.249 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-24T00:08:25.247+0000 7f4cdd7fa700 1 -- 172.21.15.189:0/3494068559 <== mon.2 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 16) v1 ==== 137291+0+0 (secure 0 0 0) 0x7f4cd002a070 con 0x7f4ce01033e0 2024-03-24T00:08:25.249 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-24T00:08:25.248+0000 7f4cdd7fa700 1 --2- 172.21.15.189:0/3494068559 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7f4ccc0a2890 0x7f4ccc0a4d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:08:25.249 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-24T00:08:25.248+0000 7f4cdd7fa700 1 -- 172.21.15.189:0/3494068559 <== mon.2 v2:172.21.15.189:3300/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 3669+0+0 (secure 0 0 0) 0x7f4cd00cebf0 con 0x7f4ce01033e0 2024-03-24T00:08:25.249 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-24T00:08:25.248+0000 7f4cdffff700 1 --2- 172.21.15.189:0/3494068559 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7f4ccc0a2890 0x7f4ccc0a4d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:08:25.249 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-24T00:08:25.249+0000 7f4cdffff700 1 --2- 172.21.15.189:0/3494068559 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7f4ccc0a2890 0x7f4ccc0a4d50 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f4ce0103240 tx=0x7f4cc8005d20 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:08:25.252 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-24T00:08:25.252+0000 7f4cdd7fa700 1 -- 172.21.15.189:0/3494068559 <== mon.2 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f4cd0013070 con 0x7f4ce01033e0 2024-03-24T00:08:25.455 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-24T00:08:25.455+0000 7f4ce691c700 1 -- 172.21.15.189:0/3494068559 --> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi189:vg_nvme/lv_3", "target": ["mon-mgr", ""]}) v1 -- 0x7f4ce00625d0 con 0x7f4ccc0a2890 2024-03-24T00:08:25.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:08:25 smithi193 ceph-mon[36878]: osdmap e29: 4 total, 4 up, 4 in 2024-03-24T00:08:25.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:08:25 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T00:08:25.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:08:25 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T00:08:25.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:08:25 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-24T00:08:25.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:08:25 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-24T00:08:25.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:08:25 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T00:08:25.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:08:25 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T00:08:25.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:08:25 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T00:08:25.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:08:25 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T00:08:25.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:08:25 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-24T00:08:25.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:08:25 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-24T00:08:25.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:08:25 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T00:08:26.087 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:08:25 smithi189 ceph-mon[36187]: osdmap e29: 4 total, 4 up, 4 in 2024-03-24T00:08:26.087 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:08:25 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T00:08:26.087 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:08:25 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T00:08:26.087 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:08:25 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-24T00:08:26.087 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:08:25 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-24T00:08:26.087 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:08:25 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T00:08:26.087 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:08:25 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T00:08:26.087 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:08:25 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T00:08:26.087 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:08:25 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T00:08:26.087 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:08:25 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-24T00:08:26.087 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:08:25 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-24T00:08:26.087 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:08:25 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T00:08:26.099 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:08:25 smithi118 ceph-mon[31535]: osdmap e29: 4 total, 4 up, 4 in 2024-03-24T00:08:26.099 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:08:25 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T00:08:26.099 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:08:25 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T00:08:26.099 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:08:25 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-24T00:08:26.099 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:08:25 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-24T00:08:26.099 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:08:25 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T00:08:26.099 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:08:25 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T00:08:26.099 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:08:25 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T00:08:26.099 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:08:25 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T00:08:26.099 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:08:25 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-24T00:08:26.099 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:08:25 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-24T00:08:26.100 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:08:25 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T00:08:26.962 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:08:26 smithi193 ceph-mon[36878]: Detected new or changed devices on smithi118 2024-03-24T00:08:26.962 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:08:26 smithi193 ceph-mon[36878]: Adjusting osd_memory_target on smithi118 to 8545M 2024-03-24T00:08:26.962 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:08:26 smithi193 ceph-mon[36878]: from='client.24224 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi189:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T00:08:26.962 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:08:26 smithi193 ceph-mon[36878]: pgmap v119: 1 pgs: 1 remapped+peering; 577 KiB data, 111 MiB used, 358 GiB / 358 GiB avail 2024-03-24T00:08:26.962 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:08:26 smithi193 ceph-mon[36878]: osdmap e30: 4 total, 4 up, 4 in 2024-03-24T00:08:27.085 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:08:26 smithi189 ceph-mon[36187]: Detected new or changed devices on smithi118 2024-03-24T00:08:27.085 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:08:26 smithi189 ceph-mon[36187]: Adjusting osd_memory_target on smithi118 to 8545M 2024-03-24T00:08:27.085 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:08:26 smithi189 ceph-mon[36187]: from='client.24224 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi189:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T00:08:27.085 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:08:26 smithi189 ceph-mon[36187]: pgmap v119: 1 pgs: 1 remapped+peering; 577 KiB data, 111 MiB used, 358 GiB / 358 GiB avail 2024-03-24T00:08:27.085 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:08:26 smithi189 ceph-mon[36187]: osdmap e30: 4 total, 4 up, 4 in 2024-03-24T00:08:27.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:08:26 smithi118 ceph-mon[31535]: Detected new or changed devices on smithi118 2024-03-24T00:08:27.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:08:26 smithi118 ceph-mon[31535]: Adjusting osd_memory_target on smithi118 to 8545M 2024-03-24T00:08:27.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:08:26 smithi118 ceph-mon[31535]: from='client.24224 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi189:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T00:08:27.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:08:26 smithi118 ceph-mon[31535]: pgmap v119: 1 pgs: 1 remapped+peering; 577 KiB data, 111 MiB used, 358 GiB / 358 GiB avail 2024-03-24T00:08:27.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:08:26 smithi118 ceph-mon[31535]: osdmap e30: 4 total, 4 up, 4 in 2024-03-24T00:08:28.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:08:27 smithi118 ceph-mon[31535]: pgmap v121: 1 pgs: 1 active+clean; 577 KiB data, 111 MiB used, 358 GiB / 358 GiB avail; 119 KiB/s, 0 objects/s recovering 2024-03-24T00:08:28.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:08:27 smithi189 ceph-mon[36187]: pgmap v121: 1 pgs: 1 active+clean; 577 KiB data, 111 MiB used, 358 GiB / 358 GiB avail; 119 KiB/s, 0 objects/s recovering 2024-03-24T00:08:28.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:08:27 smithi193 ceph-mon[36878]: pgmap v121: 1 pgs: 1 active+clean; 577 KiB data, 111 MiB used, 358 GiB / 358 GiB avail; 119 KiB/s, 0 objects/s recovering 2024-03-24T00:08:28.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:08:28 smithi189 ceph-mon[36187]: from='client.? 172.21.15.189:0/3778526276' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "57377f66-c3b4-43c1-8e8e-bc873672906d"}]: dispatch 2024-03-24T00:08:28.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:08:28 smithi189 ceph-mon[36187]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "57377f66-c3b4-43c1-8e8e-bc873672906d"}]: dispatch 2024-03-24T00:08:28.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:08:28 smithi189 ceph-mon[36187]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "57377f66-c3b4-43c1-8e8e-bc873672906d"}]': finished 2024-03-24T00:08:28.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:08:28 smithi189 ceph-mon[36187]: osdmap e31: 5 total, 4 up, 5 in 2024-03-24T00:08:28.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:08:28 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-24T00:08:29.199 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:08:28 smithi118 ceph-mon[31535]: from='client.? 172.21.15.189:0/3778526276' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "57377f66-c3b4-43c1-8e8e-bc873672906d"}]: dispatch 2024-03-24T00:08:29.199 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:08:28 smithi118 ceph-mon[31535]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "57377f66-c3b4-43c1-8e8e-bc873672906d"}]: dispatch 2024-03-24T00:08:29.199 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:08:28 smithi118 ceph-mon[31535]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "57377f66-c3b4-43c1-8e8e-bc873672906d"}]': finished 2024-03-24T00:08:29.199 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:08:28 smithi118 ceph-mon[31535]: osdmap e31: 5 total, 4 up, 5 in 2024-03-24T00:08:29.199 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:08:28 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-24T00:08:29.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:08:28 smithi193 ceph-mon[36878]: from='client.? 172.21.15.189:0/3778526276' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "57377f66-c3b4-43c1-8e8e-bc873672906d"}]: dispatch 2024-03-24T00:08:29.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:08:28 smithi193 ceph-mon[36878]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "57377f66-c3b4-43c1-8e8e-bc873672906d"}]: dispatch 2024-03-24T00:08:29.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:08:28 smithi193 ceph-mon[36878]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "57377f66-c3b4-43c1-8e8e-bc873672906d"}]': finished 2024-03-24T00:08:29.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:08:28 smithi193 ceph-mon[36878]: osdmap e31: 5 total, 4 up, 5 in 2024-03-24T00:08:29.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:08:28 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-24T00:08:30.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:08:29 smithi118 ceph-mon[31535]: from='client.? 172.21.15.189:0/1547280966' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-24T00:08:30.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:08:29 smithi118 ceph-mon[31535]: pgmap v123: 1 pgs: 1 active+clean; 577 KiB data, 111 MiB used, 358 GiB / 358 GiB avail; 99 KiB/s, 0 objects/s recovering 2024-03-24T00:08:30.203 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:08:29 smithi189 ceph-mon[36187]: from='client.? 172.21.15.189:0/1547280966' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-24T00:08:30.203 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:08:29 smithi189 ceph-mon[36187]: pgmap v123: 1 pgs: 1 active+clean; 577 KiB data, 111 MiB used, 358 GiB / 358 GiB avail; 99 KiB/s, 0 objects/s recovering 2024-03-24T00:08:30.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:08:29 smithi193 ceph-mon[36878]: from='client.? 172.21.15.189:0/1547280966' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-24T00:08:30.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:08:29 smithi193 ceph-mon[36878]: pgmap v123: 1 pgs: 1 active+clean; 577 KiB data, 111 MiB used, 358 GiB / 358 GiB avail; 99 KiB/s, 0 objects/s recovering 2024-03-24T00:08:32.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:08:32 smithi118 ceph-mon[31535]: pgmap v124: 1 pgs: 1 active+clean; 577 KiB data, 111 MiB used, 358 GiB / 358 GiB avail; 84 KiB/s, 0 objects/s recovering 2024-03-24T00:08:32.953 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:08:32 smithi189 ceph-mon[36187]: pgmap v124: 1 pgs: 1 active+clean; 577 KiB data, 111 MiB used, 358 GiB / 358 GiB avail; 84 KiB/s, 0 objects/s recovering 2024-03-24T00:08:32.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:08:32 smithi193 ceph-mon[36878]: pgmap v124: 1 pgs: 1 active+clean; 577 KiB data, 111 MiB used, 358 GiB / 358 GiB avail; 84 KiB/s, 0 objects/s recovering 2024-03-24T00:08:34.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:08:34 smithi118 ceph-mon[31535]: pgmap v125: 1 pgs: 1 active+clean; 577 KiB data, 111 MiB used, 358 GiB / 358 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-03-24T00:08:34.953 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:08:34 smithi189 ceph-mon[36187]: pgmap v125: 1 pgs: 1 active+clean; 577 KiB data, 111 MiB used, 358 GiB / 358 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-03-24T00:08:34.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:08:34 smithi193 ceph-mon[36878]: pgmap v125: 1 pgs: 1 active+clean; 577 KiB data, 111 MiB used, 358 GiB / 358 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-03-24T00:08:35.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:08:35 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-03-24T00:08:35.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:08:35 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T00:08:35.953 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:08:35 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-03-24T00:08:35.953 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:08:35 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T00:08:35.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:08:35 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-03-24T00:08:35.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:08:35 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T00:08:36.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:08:36 smithi118 ceph-mon[31535]: Deploying daemon osd.4 on smithi189 2024-03-24T00:08:36.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:08:36 smithi118 ceph-mon[31535]: pgmap v126: 1 pgs: 1 active+clean; 577 KiB data, 111 MiB used, 358 GiB / 358 GiB avail; 59 KiB/s, 0 objects/s recovering 2024-03-24T00:08:36.953 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:08:36 smithi189 ceph-mon[36187]: Deploying daemon osd.4 on smithi189 2024-03-24T00:08:36.953 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:08:36 smithi189 ceph-mon[36187]: pgmap v126: 1 pgs: 1 active+clean; 577 KiB data, 111 MiB used, 358 GiB / 358 GiB avail; 59 KiB/s, 0 objects/s recovering 2024-03-24T00:08:36.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:08:36 smithi193 ceph-mon[36878]: Deploying daemon osd.4 on smithi189 2024-03-24T00:08:36.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:08:36 smithi193 ceph-mon[36878]: pgmap v126: 1 pgs: 1 active+clean; 577 KiB data, 111 MiB used, 358 GiB / 358 GiB avail; 59 KiB/s, 0 objects/s recovering 2024-03-24T00:08:38.704 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:08:38 smithi189 ceph-mon[36187]: pgmap v127: 1 pgs: 1 active+clean; 577 KiB data, 111 MiB used, 358 GiB / 358 GiB avail 2024-03-24T00:08:38.945 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:08:38 smithi118 ceph-mon[31535]: pgmap v127: 1 pgs: 1 active+clean; 577 KiB data, 111 MiB used, 358 GiB / 358 GiB avail 2024-03-24T00:08:38.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:08:38 smithi193 ceph-mon[36878]: pgmap v127: 1 pgs: 1 active+clean; 577 KiB data, 111 MiB used, 358 GiB / 358 GiB avail 2024-03-24T00:08:40.695 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:08:40 smithi189 ceph-mon[36187]: pgmap v128: 1 pgs: 1 active+clean; 577 KiB data, 111 MiB used, 358 GiB / 358 GiB avail 2024-03-24T00:08:40.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:08:40 smithi118 ceph-mon[31535]: pgmap v128: 1 pgs: 1 active+clean; 577 KiB data, 111 MiB used, 358 GiB / 358 GiB avail 2024-03-24T00:08:40.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:08:40 smithi193 ceph-mon[36878]: pgmap v128: 1 pgs: 1 active+clean; 577 KiB data, 111 MiB used, 358 GiB / 358 GiB avail 2024-03-24T00:08:42.203 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:08:41 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T00:08:42.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:08:41 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T00:08:42.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:08:41 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T00:08:42.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:08:41 smithi189 ceph-mon[36187]: pgmap v129: 1 pgs: 1 active+clean; 577 KiB data, 111 MiB used, 358 GiB / 358 GiB avail 2024-03-24T00:08:42.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:08:41 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T00:08:42.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:08:41 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T00:08:42.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:08:41 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T00:08:42.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:08:41 smithi118 ceph-mon[31535]: pgmap v129: 1 pgs: 1 active+clean; 577 KiB data, 111 MiB used, 358 GiB / 358 GiB avail 2024-03-24T00:08:42.460 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:08:41 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T00:08:42.460 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:08:41 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T00:08:42.460 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:08:41 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T00:08:42.460 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:08:41 smithi193 ceph-mon[36878]: pgmap v129: 1 pgs: 1 active+clean; 577 KiB data, 111 MiB used, 358 GiB / 358 GiB avail 2024-03-24T00:08:44.768 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:08:44 smithi189 ceph-mon[36187]: pgmap v130: 1 pgs: 1 active+clean; 577 KiB data, 111 MiB used, 358 GiB / 358 GiB avail 2024-03-24T00:08:44.769 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:08:44 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T00:08:44.769 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:08:44 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T00:08:44.769 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:08:44 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T00:08:44.769 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:08:44 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T00:08:44.769 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:08:44 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T00:08:44.841 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-24T00:08:44.840+0000 7f4cdd7fa700 1 -- 172.21.15.189:0/3494068559 <== mgr.14152 v2:172.21.15.118:6800/2112912943 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7f4ce00625d0 con 0x7f4ccc0a2890 2024-03-24T00:08:44.841 INFO:teuthology.orchestra.run.smithi189.stdout:Created osd(s) 4 on host 'smithi189' 2024-03-24T00:08:44.843 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-24T00:08:44.843+0000 7f4ce691c700 1 -- 172.21.15.189:0/3494068559 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7f4ccc0a2890 msgr2=0x7f4ccc0a4d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:08:44.844 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-24T00:08:44.843+0000 7f4ce691c700 1 --2- 172.21.15.189:0/3494068559 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7f4ccc0a2890 0x7f4ccc0a4d50 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f4ce0103240 tx=0x7f4cc8005d20 comp rx=0 tx=0).stop 2024-03-24T00:08:44.844 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-24T00:08:44.843+0000 7f4ce691c700 1 -- 172.21.15.189:0/3494068559 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4ce01033e0 msgr2=0x7f4ce01988c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:08:44.844 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-24T00:08:44.843+0000 7f4ce691c700 1 --2- 172.21.15.189:0/3494068559 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4ce01033e0 0x7f4ce01988c0 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f4cd0000c00 tx=0x7f4cd00057c0 comp rx=0 tx=0).stop 2024-03-24T00:08:44.844 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-24T00:08:44.843+0000 7f4ce691c700 1 -- 172.21.15.189:0/3494068559 shutdown_connections 2024-03-24T00:08:44.844 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-24T00:08:44.843+0000 7f4ce691c700 1 --2- 172.21.15.189:0/3494068559 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7f4ccc0a2890 0x7f4ccc0a4d50 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:08:44.844 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-24T00:08:44.843+0000 7f4ce691c700 1 --2- 172.21.15.189:0/3494068559 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f4ce0103d80 0x7f4ce019f940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:08:44.844 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-24T00:08:44.843+0000 7f4ce691c700 1 --2- 172.21.15.189:0/3494068559 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f4ce01021e0 0x7f4ce0198380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:08:44.844 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-24T00:08:44.843+0000 7f4ce691c700 1 --2- 172.21.15.189:0/3494068559 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4ce01033e0 0x7f4ce01988c0 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:08:44.844 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-24T00:08:44.843+0000 7f4ce691c700 1 -- 172.21.15.189:0/3494068559 >> 172.21.15.189:0/3494068559 conn(0x7f4ce00fd7a0 msgr2=0x7f4ce0106680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:08:44.844 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-24T00:08:44.843+0000 7f4ce691c700 1 -- 172.21.15.189:0/3494068559 shutdown_connections 2024-03-24T00:08:44.844 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-24T00:08:44.843+0000 7f4ce691c700 1 -- 172.21.15.189:0/3494068559 wait complete. 2024-03-24T00:08:44.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:08:44 smithi118 ceph-mon[31535]: pgmap v130: 1 pgs: 1 active+clean; 577 KiB data, 111 MiB used, 358 GiB / 358 GiB avail 2024-03-24T00:08:44.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:08:44 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T00:08:44.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:08:44 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T00:08:44.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:08:44 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T00:08:44.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:08:44 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T00:08:44.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:08:44 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T00:08:44.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:08:44 smithi193 ceph-mon[36878]: pgmap v130: 1 pgs: 1 active+clean; 577 KiB data, 111 MiB used, 358 GiB / 358 GiB avail 2024-03-24T00:08:44.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:08:44 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T00:08:44.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:08:44 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T00:08:44.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:08:44 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T00:08:44.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:08:44 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T00:08:44.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:08:44 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T00:08:45.530 DEBUG:teuthology.orchestra.run.smithi189:osd.4> sudo journalctl -f -n 0 -u ceph-d79a5420-e971-11ee-95cd-87774f69a715@osd.4.service 2024-03-24T00:08:45.533 INFO:tasks.cephadm:Deploying osd.5 on smithi193 with /dev/vg_nvme/lv_3... 2024-03-24T00:08:45.533 DEBUG:teuthology.orchestra.run.smithi193:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d79a5420-e971-11ee-95cd-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_3 2024-03-24T00:08:45.779 INFO:teuthology.orchestra.run.smithi193.stderr:Inferring config /var/lib/ceph/d79a5420-e971-11ee-95cd-87774f69a715/mon.c/config 2024-03-24T00:08:45.900 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:08:45 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T00:08:45.901 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:08:45 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T00:08:45.901 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:08:45 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T00:08:45.901 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:08:45 smithi193 ceph-mon[36878]: from='osd.4 [v2:172.21.15.189:6808/4075247068,v1:172.21.15.189:6809/4075247068]' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-03-24T00:08:45.901 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:08:45 smithi193 ceph-mon[36878]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-03-24T00:08:45.901 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:08:45 smithi193 ceph-mon[36878]: pgmap v131: 1 pgs: 1 active+clean; 577 KiB data, 111 MiB used, 358 GiB / 358 GiB avail 2024-03-24T00:08:46.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:08:45 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T00:08:46.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:08:45 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T00:08:46.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:08:45 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T00:08:46.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:08:45 smithi118 ceph-mon[31535]: from='osd.4 [v2:172.21.15.189:6808/4075247068,v1:172.21.15.189:6809/4075247068]' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-03-24T00:08:46.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:08:45 smithi118 ceph-mon[31535]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-03-24T00:08:46.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:08:45 smithi118 ceph-mon[31535]: pgmap v131: 1 pgs: 1 active+clean; 577 KiB data, 111 MiB used, 358 GiB / 358 GiB avail 2024-03-24T00:08:46.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:08:45 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T00:08:46.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:08:45 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T00:08:46.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:08:45 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T00:08:46.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:08:45 smithi189 ceph-mon[36187]: from='osd.4 [v2:172.21.15.189:6808/4075247068,v1:172.21.15.189:6809/4075247068]' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-03-24T00:08:46.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:08:45 smithi189 ceph-mon[36187]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-03-24T00:08:46.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:08:45 smithi189 ceph-mon[36187]: pgmap v131: 1 pgs: 1 active+clean; 577 KiB data, 111 MiB used, 358 GiB / 358 GiB avail 2024-03-24T00:08:47.035 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:08:46 smithi189 ceph-mon[36187]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-03-24T00:08:47.035 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:08:46 smithi189 ceph-mon[36187]: from='osd.4 [v2:172.21.15.189:6808/4075247068,v1:172.21.15.189:6809/4075247068]' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi189", "root=default"]}]: dispatch 2024-03-24T00:08:47.035 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:08:46 smithi189 ceph-mon[36187]: osdmap e32: 5 total, 4 up, 5 in 2024-03-24T00:08:47.035 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:08:46 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-24T00:08:47.036 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:08:46 smithi189 ceph-mon[36187]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi189", "root=default"]}]: dispatch 2024-03-24T00:08:47.090 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:08:46 smithi193 ceph-mon[36878]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-03-24T00:08:47.090 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:08:46 smithi193 ceph-mon[36878]: from='osd.4 [v2:172.21.15.189:6808/4075247068,v1:172.21.15.189:6809/4075247068]' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi189", "root=default"]}]: dispatch 2024-03-24T00:08:47.090 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:08:46 smithi193 ceph-mon[36878]: osdmap e32: 5 total, 4 up, 5 in 2024-03-24T00:08:47.091 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:08:46 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-24T00:08:47.091 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:08:46 smithi193 ceph-mon[36878]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi189", "root=default"]}]: dispatch 2024-03-24T00:08:47.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:08:46 smithi118 ceph-mon[31535]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-03-24T00:08:47.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:08:46 smithi118 ceph-mon[31535]: from='osd.4 [v2:172.21.15.189:6808/4075247068,v1:172.21.15.189:6809/4075247068]' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi189", "root=default"]}]: dispatch 2024-03-24T00:08:47.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:08:46 smithi118 ceph-mon[31535]: osdmap e32: 5 total, 4 up, 5 in 2024-03-24T00:08:47.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:08:46 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-24T00:08:47.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:08:46 smithi118 ceph-mon[31535]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi189", "root=default"]}]: dispatch 2024-03-24T00:08:47.454 INFO:journalctl@ceph.osd.4.smithi189.stdout:Mar 24 00:08:47 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-osd-4[47788]: 2024-03-24T00:08:47.262+0000 7f1b4957a700 -1 osd.4 0 waiting for initial osdmap 2024-03-24T00:08:47.454 INFO:journalctl@ceph.osd.4.smithi189.stdout:Mar 24 00:08:47 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-osd-4[47788]: 2024-03-24T00:08:47.270+0000 7f1b41365700 -1 osd.4 33 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-24T00:08:47.747 INFO:teuthology.orchestra.run.smithi193.stderr:--> Zapping: /dev/vg_nvme/lv_3 2024-03-24T00:08:47.747 INFO:teuthology.orchestra.run.smithi193.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_3 bs=1M count=10 conv=fsync 2024-03-24T00:08:47.747 INFO:teuthology.orchestra.run.smithi193.stderr: stderr: 10+0 records in 2024-03-24T00:08:47.748 INFO:teuthology.orchestra.run.smithi193.stderr:10+0 records out 2024-03-24T00:08:47.748 INFO:teuthology.orchestra.run.smithi193.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0113585 s, 923 MB/s 2024-03-24T00:08:47.748 INFO:teuthology.orchestra.run.smithi193.stderr:--> Zapping successful for: 2024-03-24T00:08:48.062 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:08:47 smithi193 ceph-mon[36878]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi189", "root=default"]}]': finished 2024-03-24T00:08:48.062 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:08:47 smithi193 ceph-mon[36878]: osdmap e33: 5 total, 4 up, 5 in 2024-03-24T00:08:48.062 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:08:47 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-24T00:08:48.062 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:08:47 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-24T00:08:48.062 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:08:47 smithi193 ceph-mon[36878]: from='osd.4 ' entity='osd.4' 2024-03-24T00:08:48.062 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:08:47 smithi193 ceph-mon[36878]: pgmap v134: 1 pgs: 1 active+clean; 577 KiB data, 111 MiB used, 358 GiB / 358 GiB avail 2024-03-24T00:08:48.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:08:47 smithi118 ceph-mon[31535]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi189", "root=default"]}]': finished 2024-03-24T00:08:48.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:08:47 smithi118 ceph-mon[31535]: osdmap e33: 5 total, 4 up, 5 in 2024-03-24T00:08:48.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:08:47 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-24T00:08:48.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:08:47 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-24T00:08:48.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:08:47 smithi118 ceph-mon[31535]: from='osd.4 ' entity='osd.4' 2024-03-24T00:08:48.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:08:47 smithi118 ceph-mon[31535]: pgmap v134: 1 pgs: 1 active+clean; 577 KiB data, 111 MiB used, 358 GiB / 358 GiB avail 2024-03-24T00:08:48.203 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:08:47 smithi189 ceph-mon[36187]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi189", "root=default"]}]': finished 2024-03-24T00:08:48.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:08:47 smithi189 ceph-mon[36187]: osdmap e33: 5 total, 4 up, 5 in 2024-03-24T00:08:48.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:08:47 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-24T00:08:48.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:08:47 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-24T00:08:48.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:08:47 smithi189 ceph-mon[36187]: from='osd.4 ' entity='osd.4' 2024-03-24T00:08:48.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:08:47 smithi189 ceph-mon[36187]: pgmap v134: 1 pgs: 1 active+clean; 577 KiB data, 111 MiB used, 358 GiB / 358 GiB avail 2024-03-24T00:08:48.415 DEBUG:teuthology.orchestra.run.smithi193:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d79a5420-e971-11ee-95cd-87774f69a715 -- ceph orch daemon add osd smithi193:vg_nvme/lv_3 2024-03-24T00:08:48.655 INFO:teuthology.orchestra.run.smithi193.stderr:Inferring config /var/lib/ceph/d79a5420-e971-11ee-95cd-87774f69a715/mon.c/config 2024-03-24T00:08:48.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:08:48 smithi193 ceph-mon[36878]: purged_snaps scrub starts 2024-03-24T00:08:48.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:08:48 smithi193 ceph-mon[36878]: purged_snaps scrub ok 2024-03-24T00:08:48.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:08:48 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-24T00:08:48.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:08:48 smithi193 ceph-mon[36878]: osd.4 [v2:172.21.15.189:6808/4075247068,v1:172.21.15.189:6809/4075247068] boot 2024-03-24T00:08:48.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:08:48 smithi193 ceph-mon[36878]: osdmap e34: 5 total, 5 up, 5 in 2024-03-24T00:08:48.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:08:48 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-24T00:08:48.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:08:48 smithi193 ceph-mon[36878]: Detected new or changed devices on smithi189 2024-03-24T00:08:48.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:08:48 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T00:08:48.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:08:48 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T00:08:48.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:08:48 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-24T00:08:48.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:08:48 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-24T00:08:48.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:08:48 smithi193 ceph-mon[36878]: Adjusting osd_memory_target on smithi189 to 8540M 2024-03-24T00:08:48.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:08:48 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T00:08:48.962 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:08:48 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T00:08:48.962 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:08:48 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T00:08:48.962 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:08:48 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T00:08:49.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:08:48 smithi118 ceph-mon[31535]: purged_snaps scrub starts 2024-03-24T00:08:49.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:08:48 smithi118 ceph-mon[31535]: purged_snaps scrub ok 2024-03-24T00:08:49.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:08:48 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-24T00:08:49.197 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:08:48 smithi118 ceph-mon[31535]: osd.4 [v2:172.21.15.189:6808/4075247068,v1:172.21.15.189:6809/4075247068] boot 2024-03-24T00:08:49.197 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:08:48 smithi118 ceph-mon[31535]: osdmap e34: 5 total, 5 up, 5 in 2024-03-24T00:08:49.197 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:08:48 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-24T00:08:49.197 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:08:48 smithi118 ceph-mon[31535]: Detected new or changed devices on smithi189 2024-03-24T00:08:49.197 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:08:48 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T00:08:49.197 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:08:48 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T00:08:49.197 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:08:48 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-24T00:08:49.197 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:08:48 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-24T00:08:49.197 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:08:48 smithi118 ceph-mon[31535]: Adjusting osd_memory_target on smithi189 to 8540M 2024-03-24T00:08:49.197 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:08:48 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T00:08:49.197 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:08:48 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T00:08:49.197 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:08:48 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T00:08:49.197 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:08:48 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T00:08:49.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:08:48 smithi189 ceph-mon[36187]: purged_snaps scrub starts 2024-03-24T00:08:49.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:08:48 smithi189 ceph-mon[36187]: purged_snaps scrub ok 2024-03-24T00:08:49.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:08:48 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-24T00:08:49.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:08:48 smithi189 ceph-mon[36187]: osd.4 [v2:172.21.15.189:6808/4075247068,v1:172.21.15.189:6809/4075247068] boot 2024-03-24T00:08:49.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:08:48 smithi189 ceph-mon[36187]: osdmap e34: 5 total, 5 up, 5 in 2024-03-24T00:08:49.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:08:48 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-24T00:08:49.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:08:48 smithi189 ceph-mon[36187]: Detected new or changed devices on smithi189 2024-03-24T00:08:49.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:08:48 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T00:08:49.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:08:48 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T00:08:49.205 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:08:48 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-24T00:08:49.205 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:08:48 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-24T00:08:49.205 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:08:48 smithi189 ceph-mon[36187]: Adjusting osd_memory_target on smithi189 to 8540M 2024-03-24T00:08:49.205 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:08:48 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T00:08:49.205 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:08:48 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T00:08:49.205 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:08:48 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T00:08:49.205 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:08:48 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T00:08:50.106 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:08:49 smithi193 ceph-mon[36878]: pgmap v136: 1 pgs: 1 active+clean; 577 KiB data, 142 MiB used, 447 GiB / 447 GiB avail 2024-03-24T00:08:50.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:08:49 smithi118 ceph-mon[31535]: pgmap v136: 1 pgs: 1 active+clean; 577 KiB data, 142 MiB used, 447 GiB / 447 GiB avail 2024-03-24T00:08:50.203 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:08:49 smithi189 ceph-mon[36187]: pgmap v136: 1 pgs: 1 active+clean; 577 KiB data, 142 MiB used, 447 GiB / 447 GiB avail 2024-03-24T00:08:50.313 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-24T00:08:50.311+0000 7f1a4c01a700 1 -- 172.21.15.193:0/2450201137 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f1a44074bf0 msgr2=0x7f1a44075070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:08:50.313 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-24T00:08:50.311+0000 7f1a4c01a700 1 --2- 172.21.15.193:0/2450201137 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f1a44074bf0 0x7f1a44075070 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f1a34009990 tx=0x7f1a34009ca0 comp rx=0 tx=0).stop 2024-03-24T00:08:50.313 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-24T00:08:50.311+0000 7f1a4c01a700 1 -- 172.21.15.193:0/2450201137 shutdown_connections 2024-03-24T00:08:50.313 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-24T00:08:50.311+0000 7f1a4c01a700 1 --2- 172.21.15.193:0/2450201137 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1a4410a090 0x7f1a4410c520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:08:50.313 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-24T00:08:50.311+0000 7f1a4c01a700 1 --2- 172.21.15.193:0/2450201137 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f1a440761c0 0x7f1a44074620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:08:50.313 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-24T00:08:50.311+0000 7f1a4c01a700 1 --2- 172.21.15.193:0/2450201137 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f1a44074bf0 0x7f1a44075070 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:08:50.313 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-24T00:08:50.311+0000 7f1a4c01a700 1 -- 172.21.15.193:0/2450201137 >> 172.21.15.193:0/2450201137 conn(0x7f1a440fd7a0 msgr2=0x7f1a440ffbc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:08:50.313 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-24T00:08:50.311+0000 7f1a4c01a700 1 -- 172.21.15.193:0/2450201137 shutdown_connections 2024-03-24T00:08:50.313 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-24T00:08:50.311+0000 7f1a4c01a700 1 -- 172.21.15.193:0/2450201137 wait complete. 2024-03-24T00:08:50.313 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-24T00:08:50.311+0000 7f1a4c01a700 1 Processor -- start 2024-03-24T00:08:50.314 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-24T00:08:50.311+0000 7f1a4c01a700 1 -- start start 2024-03-24T00:08:50.314 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-24T00:08:50.311+0000 7f1a4c01a700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f1a44074bf0 0x7f1a4419a790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:08:50.314 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-24T00:08:50.311+0000 7f1a4c01a700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1a440761c0 0x7f1a4419acd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:08:50.314 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-24T00:08:50.311+0000 7f1a4c01a700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f1a4410a090 0x7f1a441a1d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:08:50.314 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-24T00:08:50.311+0000 7f1a4c01a700 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1a441100a0 con 0x7f1a4410a090 2024-03-24T00:08:50.314 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-24T00:08:50.311+0000 7f1a4c01a700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1a4410ff20 con 0x7f1a440761c0 2024-03-24T00:08:50.314 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-24T00:08:50.311+0000 7f1a4c01a700 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1a44110220 con 0x7f1a44074bf0 2024-03-24T00:08:50.314 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-24T00:08:50.312+0000 7f1a49db6700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f1a44074bf0 0x7f1a4419a790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:08:50.314 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-24T00:08:50.312+0000 7f1a49db6700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f1a44074bf0 0x7f1a4419a790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.193:54238/0 (socket says 172.21.15.193:54238) 2024-03-24T00:08:50.314 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-24T00:08:50.312+0000 7f1a49db6700 1 -- 172.21.15.193:0/3217484308 learned_addr learned my addr 172.21.15.193:0/3217484308 (peer_addr_for_me v2:172.21.15.193:0/0) 2024-03-24T00:08:50.314 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-24T00:08:50.312+0000 7f1a495b5700 1 --2- 172.21.15.193:0/3217484308 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1a440761c0 0x7f1a4419acd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:08:50.314 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-24T00:08:50.312+0000 7f1a49db6700 1 -- 172.21.15.193:0/3217484308 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1a440761c0 msgr2=0x7f1a4419acd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:08:50.314 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-24T00:08:50.312+0000 7f1a49db6700 1 --2- 172.21.15.193:0/3217484308 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1a440761c0 0x7f1a4419acd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:08:50.315 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-24T00:08:50.312+0000 7f1a49db6700 1 -- 172.21.15.193:0/3217484308 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f1a4410a090 msgr2=0x7f1a441a1d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:08:50.315 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-24T00:08:50.312+0000 7f1a49db6700 1 --2- 172.21.15.193:0/3217484308 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f1a4410a090 0x7f1a441a1d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:08:50.315 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-24T00:08:50.312+0000 7f1a49db6700 1 -- 172.21.15.193:0/3217484308 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1a441a23f0 con 0x7f1a44074bf0 2024-03-24T00:08:50.315 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-24T00:08:50.312+0000 7f1a49db6700 1 --2- 172.21.15.193:0/3217484308 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f1a44074bf0 0x7f1a4419a790 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f1a4000b530 tx=0x7f1a4000b840 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:08:50.315 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-24T00:08:50.312+0000 7f1a3affd700 1 -- 172.21.15.193:0/3217484308 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1a40015020 con 0x7f1a44074bf0 2024-03-24T00:08:50.315 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-24T00:08:50.312+0000 7f1a4c01a700 1 -- 172.21.15.193:0/3217484308 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1a441a26e0 con 0x7f1a44074bf0 2024-03-24T00:08:50.315 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-24T00:08:50.312+0000 7f1a3affd700 1 -- 172.21.15.193:0/3217484308 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f1a40011d10 con 0x7f1a44074bf0 2024-03-24T00:08:50.315 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-24T00:08:50.312+0000 7f1a3affd700 1 -- 172.21.15.193:0/3217484308 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1a4000f2e0 con 0x7f1a44074bf0 2024-03-24T00:08:50.315 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-24T00:08:50.312+0000 7f1a4c01a700 1 -- 172.21.15.193:0/3217484308 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1a441a2cf0 con 0x7f1a44074bf0 2024-03-24T00:08:50.316 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-24T00:08:50.313+0000 7f1a4c01a700 1 -- 172.21.15.193:0/3217484308 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1a44068e50 con 0x7f1a44074bf0 2024-03-24T00:08:50.316 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-24T00:08:50.314+0000 7f1a3affd700 1 -- 172.21.15.193:0/3217484308 <== mon.1 v2:172.21.15.193:3300/0 4 ==== mgrmap(e 16) v1 ==== 137291+0+0 (secure 0 0 0) 0x7f1a40018020 con 0x7f1a44074bf0 2024-03-24T00:08:50.316 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-24T00:08:50.315+0000 7f1a3affd700 1 --2- 172.21.15.193:0/3217484308 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7f1a300a2890 0x7f1a300a4d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:08:50.316 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-24T00:08:50.315+0000 7f1a3affd700 1 -- 172.21.15.193:0/3217484308 <== mon.1 v2:172.21.15.193:3300/0 5 ==== osd_map(35..35 src has 1..35) v4 ==== 4101+0+0 (secure 0 0 0) 0x7f1a400cd550 con 0x7f1a44074bf0 2024-03-24T00:08:50.316 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-24T00:08:50.315+0000 7f1a495b5700 1 --2- 172.21.15.193:0/3217484308 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7f1a300a2890 0x7f1a300a4d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:08:50.317 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-24T00:08:50.316+0000 7f1a495b5700 1 --2- 172.21.15.193:0/3217484308 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7f1a300a2890 0x7f1a300a4d50 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f1a34003890 tx=0x7f1a34005fd0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:08:50.318 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-24T00:08:50.319+0000 7f1a3affd700 1 -- 172.21.15.193:0/3217484308 <== mon.1 v2:172.21.15.193:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f1a4001f020 con 0x7f1a44074bf0 2024-03-24T00:08:50.521 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-24T00:08:50.521+0000 7f1a4c01a700 1 -- 172.21.15.193:0/3217484308 --> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi193:vg_nvme/lv_3", "target": ["mon-mgr", ""]}) v1 -- 0x7f1a440625d0 con 0x7f1a300a2890 2024-03-24T00:08:50.858 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:08:50 smithi193 ceph-mon[36878]: osdmap e35: 5 total, 5 up, 5 in 2024-03-24T00:08:50.858 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:08:50 smithi193 ceph-mon[36878]: from='client.24244 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi193:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T00:08:51.150 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:08:50 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-24T00:08:51.150 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:08:50 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-24T00:08:51.150 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:08:50 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T00:08:51.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:08:50 smithi118 ceph-mon[31535]: osdmap e35: 5 total, 5 up, 5 in 2024-03-24T00:08:51.197 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:08:50 smithi118 ceph-mon[31535]: from='client.24244 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi193:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T00:08:51.197 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:08:50 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-24T00:08:51.197 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:08:50 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-24T00:08:51.197 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:08:50 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T00:08:51.203 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:08:50 smithi189 ceph-mon[36187]: osdmap e35: 5 total, 5 up, 5 in 2024-03-24T00:08:51.203 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:08:50 smithi189 ceph-mon[36187]: from='client.24244 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi193:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T00:08:51.203 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:08:50 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-24T00:08:51.203 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:08:50 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-24T00:08:51.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:08:50 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T00:08:52.137 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:08:51 smithi193 ceph-mon[36878]: pgmap v138: 1 pgs: 1 active+clean; 577 KiB data, 142 MiB used, 447 GiB / 447 GiB avail 2024-03-24T00:08:52.195 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:08:51 smithi118 ceph-mon[31535]: pgmap v138: 1 pgs: 1 active+clean; 577 KiB data, 142 MiB used, 447 GiB / 447 GiB avail 2024-03-24T00:08:52.203 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:08:51 smithi189 ceph-mon[36187]: pgmap v138: 1 pgs: 1 active+clean; 577 KiB data, 142 MiB used, 447 GiB / 447 GiB avail 2024-03-24T00:08:53.645 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:08:53 smithi193 ceph-mon[36878]: from='client.? 172.21.15.193:0/4032145453' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "91970976-2193-4224-81c0-dc96cb022c85"}]: dispatch 2024-03-24T00:08:53.645 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:08:53 smithi193 ceph-mon[36878]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "91970976-2193-4224-81c0-dc96cb022c85"}]: dispatch 2024-03-24T00:08:53.645 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:08:53 smithi193 ceph-mon[36878]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "91970976-2193-4224-81c0-dc96cb022c85"}]': finished 2024-03-24T00:08:53.645 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:08:53 smithi193 ceph-mon[36878]: osdmap e36: 6 total, 5 up, 6 in 2024-03-24T00:08:53.646 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:08:53 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-24T00:08:53.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:08:53 smithi118 ceph-mon[31535]: from='client.? 172.21.15.193:0/4032145453' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "91970976-2193-4224-81c0-dc96cb022c85"}]: dispatch 2024-03-24T00:08:53.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:08:53 smithi118 ceph-mon[31535]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "91970976-2193-4224-81c0-dc96cb022c85"}]: dispatch 2024-03-24T00:08:53.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:08:53 smithi118 ceph-mon[31535]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "91970976-2193-4224-81c0-dc96cb022c85"}]': finished 2024-03-24T00:08:53.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:08:53 smithi118 ceph-mon[31535]: osdmap e36: 6 total, 5 up, 6 in 2024-03-24T00:08:53.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:08:53 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-24T00:08:53.703 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:08:53 smithi189 ceph-mon[36187]: from='client.? 172.21.15.193:0/4032145453' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "91970976-2193-4224-81c0-dc96cb022c85"}]: dispatch 2024-03-24T00:08:53.704 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:08:53 smithi189 ceph-mon[36187]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "91970976-2193-4224-81c0-dc96cb022c85"}]: dispatch 2024-03-24T00:08:53.704 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:08:53 smithi189 ceph-mon[36187]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "91970976-2193-4224-81c0-dc96cb022c85"}]': finished 2024-03-24T00:08:53.704 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:08:53 smithi189 ceph-mon[36187]: osdmap e36: 6 total, 5 up, 6 in 2024-03-24T00:08:53.704 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:08:53 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-24T00:08:54.695 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:08:54 smithi118 ceph-mon[31535]: pgmap v140: 1 pgs: 1 active+clean; 577 KiB data, 142 MiB used, 447 GiB / 447 GiB avail 2024-03-24T00:08:54.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:08:54 smithi118 ceph-mon[31535]: from='client.? 172.21.15.193:0/2647209672' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-24T00:08:54.703 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:08:54 smithi189 ceph-mon[36187]: pgmap v140: 1 pgs: 1 active+clean; 577 KiB data, 142 MiB used, 447 GiB / 447 GiB avail 2024-03-24T00:08:54.703 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:08:54 smithi189 ceph-mon[36187]: from='client.? 172.21.15.193:0/2647209672' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-24T00:08:54.710 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:08:54 smithi193 ceph-mon[36878]: pgmap v140: 1 pgs: 1 active+clean; 577 KiB data, 142 MiB used, 447 GiB / 447 GiB avail 2024-03-24T00:08:54.710 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:08:54 smithi193 ceph-mon[36878]: from='client.? 172.21.15.193:0/2647209672' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-24T00:08:56.929 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:08:56 smithi193 ceph-mon[36878]: pgmap v141: 1 pgs: 1 active+clean; 577 KiB data, 142 MiB used, 447 GiB / 447 GiB avail 2024-03-24T00:08:56.945 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:08:56 smithi118 ceph-mon[31535]: pgmap v141: 1 pgs: 1 active+clean; 577 KiB data, 142 MiB used, 447 GiB / 447 GiB avail 2024-03-24T00:08:56.953 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:08:56 smithi189 ceph-mon[36187]: pgmap v141: 1 pgs: 1 active+clean; 577 KiB data, 142 MiB used, 447 GiB / 447 GiB avail 2024-03-24T00:08:58.945 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:08:58 smithi118 ceph-mon[31535]: pgmap v142: 1 pgs: 1 active+clean; 577 KiB data, 142 MiB used, 447 GiB / 447 GiB avail 2024-03-24T00:08:58.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:08:58 smithi189 ceph-mon[36187]: pgmap v142: 1 pgs: 1 active+clean; 577 KiB data, 142 MiB used, 447 GiB / 447 GiB avail 2024-03-24T00:08:58.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:08:58 smithi193 ceph-mon[36878]: pgmap v142: 1 pgs: 1 active+clean; 577 KiB data, 142 MiB used, 447 GiB / 447 GiB avail 2024-03-24T00:09:00.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:09:00 smithi118 ceph-mon[31535]: pgmap v143: 1 pgs: 1 active+clean; 577 KiB data, 142 MiB used, 447 GiB / 447 GiB avail 2024-03-24T00:09:00.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:09:00 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-03-24T00:09:00.947 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:09:00 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T00:09:00.953 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:09:00 smithi189 ceph-mon[36187]: pgmap v143: 1 pgs: 1 active+clean; 577 KiB data, 142 MiB used, 447 GiB / 447 GiB avail 2024-03-24T00:09:00.953 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:09:00 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-03-24T00:09:00.953 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:09:00 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T00:09:00.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:09:00 smithi193 ceph-mon[36878]: pgmap v143: 1 pgs: 1 active+clean; 577 KiB data, 142 MiB used, 447 GiB / 447 GiB avail 2024-03-24T00:09:00.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:09:00 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-03-24T00:09:00.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:09:00 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T00:09:01.914 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:09:01 smithi193 ceph-mon[36878]: Deploying daemon osd.5 on smithi193 2024-03-24T00:09:01.945 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:09:01 smithi118 ceph-mon[31535]: Deploying daemon osd.5 on smithi193 2024-03-24T00:09:01.953 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:09:01 smithi189 ceph-mon[36187]: Deploying daemon osd.5 on smithi193 2024-03-24T00:09:02.867 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:09:02 smithi193 ceph-mon[36878]: pgmap v144: 1 pgs: 1 active+clean; 577 KiB data, 142 MiB used, 447 GiB / 447 GiB avail 2024-03-24T00:09:02.945 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:09:02 smithi118 ceph-mon[31535]: pgmap v144: 1 pgs: 1 active+clean; 577 KiB data, 142 MiB used, 447 GiB / 447 GiB avail 2024-03-24T00:09:02.953 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:09:02 smithi189 ceph-mon[36187]: pgmap v144: 1 pgs: 1 active+clean; 577 KiB data, 142 MiB used, 447 GiB / 447 GiB avail 2024-03-24T00:09:04.710 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:09:04 smithi193 ceph-mon[36878]: pgmap v145: 1 pgs: 1 active+clean; 577 KiB data, 142 MiB used, 447 GiB / 447 GiB avail 2024-03-24T00:09:04.945 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:09:04 smithi118 ceph-mon[31535]: pgmap v145: 1 pgs: 1 active+clean; 577 KiB data, 142 MiB used, 447 GiB / 447 GiB avail 2024-03-24T00:09:04.953 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:09:04 smithi189 ceph-mon[36187]: pgmap v145: 1 pgs: 1 active+clean; 577 KiB data, 142 MiB used, 447 GiB / 447 GiB avail 2024-03-24T00:09:06.843 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:09:06 smithi193 ceph-mon[36878]: pgmap v146: 1 pgs: 1 active+clean; 577 KiB data, 142 MiB used, 447 GiB / 447 GiB avail 2024-03-24T00:09:06.843 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:09:06 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T00:09:06.843 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:09:06 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T00:09:06.844 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:09:06 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T00:09:06.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:09:06 smithi118 ceph-mon[31535]: pgmap v146: 1 pgs: 1 active+clean; 577 KiB data, 142 MiB used, 447 GiB / 447 GiB avail 2024-03-24T00:09:06.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:09:06 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T00:09:06.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:09:06 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T00:09:06.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:09:06 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T00:09:06.953 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:09:06 smithi189 ceph-mon[36187]: pgmap v146: 1 pgs: 1 active+clean; 577 KiB data, 142 MiB used, 447 GiB / 447 GiB avail 2024-03-24T00:09:06.953 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:09:06 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T00:09:06.953 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:09:06 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T00:09:06.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:09:06 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T00:09:08.711 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:09:08 smithi193 ceph-mon[36878]: pgmap v147: 1 pgs: 1 active+clean; 577 KiB data, 142 MiB used, 447 GiB / 447 GiB avail 2024-03-24T00:09:08.945 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:09:08 smithi118 ceph-mon[31535]: pgmap v147: 1 pgs: 1 active+clean; 577 KiB data, 142 MiB used, 447 GiB / 447 GiB avail 2024-03-24T00:09:08.953 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:09:08 smithi189 ceph-mon[36187]: pgmap v147: 1 pgs: 1 active+clean; 577 KiB data, 142 MiB used, 447 GiB / 447 GiB avail 2024-03-24T00:09:09.361 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-24T00:09:09.361+0000 7f1a3affd700 1 -- 172.21.15.193:0/3217484308 <== mgr.14152 v2:172.21.15.118:6800/2112912943 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7f1a440625d0 con 0x7f1a300a2890 2024-03-24T00:09:09.362 INFO:teuthology.orchestra.run.smithi193.stdout:Created osd(s) 5 on host 'smithi193' 2024-03-24T00:09:09.364 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-24T00:09:09.364+0000 7f1a4c01a700 1 -- 172.21.15.193:0/3217484308 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7f1a300a2890 msgr2=0x7f1a300a4d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:09:09.364 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-24T00:09:09.364+0000 7f1a4c01a700 1 --2- 172.21.15.193:0/3217484308 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7f1a300a2890 0x7f1a300a4d50 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f1a34003890 tx=0x7f1a34005fd0 comp rx=0 tx=0).stop 2024-03-24T00:09:09.364 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-24T00:09:09.364+0000 7f1a4c01a700 1 -- 172.21.15.193:0/3217484308 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f1a44074bf0 msgr2=0x7f1a4419a790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:09:09.364 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-24T00:09:09.364+0000 7f1a4c01a700 1 --2- 172.21.15.193:0/3217484308 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f1a44074bf0 0x7f1a4419a790 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f1a4000b530 tx=0x7f1a4000b840 comp rx=0 tx=0).stop 2024-03-24T00:09:09.364 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-24T00:09:09.364+0000 7f1a4c01a700 1 -- 172.21.15.193:0/3217484308 shutdown_connections 2024-03-24T00:09:09.365 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-24T00:09:09.364+0000 7f1a4c01a700 1 --2- 172.21.15.193:0/3217484308 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7f1a300a2890 0x7f1a300a4d50 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:09:09.365 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-24T00:09:09.364+0000 7f1a4c01a700 1 --2- 172.21.15.193:0/3217484308 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f1a4410a090 0x7f1a441a1d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:09:09.365 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-24T00:09:09.364+0000 7f1a4c01a700 1 --2- 172.21.15.193:0/3217484308 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f1a44074bf0 0x7f1a4419a790 unknown :-1 s=CLOSED pgs=24 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:09:09.365 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-24T00:09:09.364+0000 7f1a4c01a700 1 --2- 172.21.15.193:0/3217484308 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1a440761c0 0x7f1a4419acd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:09:09.365 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-24T00:09:09.364+0000 7f1a4c01a700 1 -- 172.21.15.193:0/3217484308 >> 172.21.15.193:0/3217484308 conn(0x7f1a440fd7a0 msgr2=0x7f1a44105f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:09:09.365 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-24T00:09:09.364+0000 7f1a4c01a700 1 -- 172.21.15.193:0/3217484308 shutdown_connections 2024-03-24T00:09:09.365 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-24T00:09:09.364+0000 7f1a4c01a700 1 -- 172.21.15.193:0/3217484308 wait complete. 2024-03-24T00:09:10.343 DEBUG:teuthology.orchestra.run.smithi193:osd.5> sudo journalctl -f -n 0 -u ceph-d79a5420-e971-11ee-95cd-87774f69a715@osd.5.service 2024-03-24T00:09:10.346 INFO:tasks.cephadm:Deploying osd.6 on smithi118 with /dev/vg_nvme/lv_2... 2024-03-24T00:09:10.346 DEBUG:teuthology.orchestra.run.smithi118:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d79a5420-e971-11ee-95cd-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_2 2024-03-24T00:09:10.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:09:10 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T00:09:10.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:09:10 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T00:09:10.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:09:10 smithi118 ceph-mon[31535]: pgmap v148: 1 pgs: 1 active+clean; 577 KiB data, 142 MiB used, 447 GiB / 447 GiB avail 2024-03-24T00:09:10.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:09:10 smithi118 ceph-mon[31535]: from='osd.5 [v2:172.21.15.193:6808/2920363402,v1:172.21.15.193:6809/2920363402]' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-03-24T00:09:10.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:09:10 smithi118 ceph-mon[31535]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-03-24T00:09:10.703 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:09:10 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T00:09:10.703 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:09:10 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T00:09:10.703 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:09:10 smithi189 ceph-mon[36187]: pgmap v148: 1 pgs: 1 active+clean; 577 KiB data, 142 MiB used, 447 GiB / 447 GiB avail 2024-03-24T00:09:10.704 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:09:10 smithi189 ceph-mon[36187]: from='osd.5 [v2:172.21.15.193:6808/2920363402,v1:172.21.15.193:6809/2920363402]' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-03-24T00:09:10.704 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:09:10 smithi189 ceph-mon[36187]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-03-24T00:09:10.710 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:09:10 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T00:09:10.710 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:09:10 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T00:09:10.710 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:09:10 smithi193 ceph-mon[36878]: pgmap v148: 1 pgs: 1 active+clean; 577 KiB data, 142 MiB used, 447 GiB / 447 GiB avail 2024-03-24T00:09:10.710 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:09:10 smithi193 ceph-mon[36878]: from='osd.5 [v2:172.21.15.193:6808/2920363402,v1:172.21.15.193:6809/2920363402]' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-03-24T00:09:10.711 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:09:10 smithi193 ceph-mon[36878]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-03-24T00:09:10.795 INFO:teuthology.orchestra.run.smithi118.stderr:Inferring config /var/lib/ceph/d79a5420-e971-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T00:09:11.460 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:09:11 smithi193 ceph-mon[36878]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-03-24T00:09:11.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:09:11 smithi193 ceph-mon[36878]: osdmap e37: 6 total, 5 up, 6 in 2024-03-24T00:09:11.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:09:11 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-24T00:09:11.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:09:11 smithi193 ceph-mon[36878]: from='osd.5 [v2:172.21.15.193:6808/2920363402,v1:172.21.15.193:6809/2920363402]' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi193", "root=default"]}]: dispatch 2024-03-24T00:09:11.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:09:11 smithi193 ceph-mon[36878]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi193", "root=default"]}]: dispatch 2024-03-24T00:09:11.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:09:11 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T00:09:11.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:09:11 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T00:09:11.461 INFO:journalctl@ceph.osd.5.smithi193.stdout:Mar 24 00:09:11 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-osd-5[49201]: 2024-03-24T00:09:11.333+0000 7f480baf9700 -1 osd.5 0 waiting for initial osdmap 2024-03-24T00:09:11.461 INFO:journalctl@ceph.osd.5.smithi193.stdout:Mar 24 00:09:11 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-osd-5[49201]: 2024-03-24T00:09:11.343+0000 7f48078ec700 -1 osd.5 38 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-24T00:09:11.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:09:11 smithi118 ceph-mon[31535]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-03-24T00:09:11.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:09:11 smithi118 ceph-mon[31535]: osdmap e37: 6 total, 5 up, 6 in 2024-03-24T00:09:11.697 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:09:11 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-24T00:09:11.697 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:09:11 smithi118 ceph-mon[31535]: from='osd.5 [v2:172.21.15.193:6808/2920363402,v1:172.21.15.193:6809/2920363402]' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi193", "root=default"]}]: dispatch 2024-03-24T00:09:11.697 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:09:11 smithi118 ceph-mon[31535]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi193", "root=default"]}]: dispatch 2024-03-24T00:09:11.697 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:09:11 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T00:09:11.697 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:09:11 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T00:09:11.703 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:09:11 smithi189 ceph-mon[36187]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-03-24T00:09:11.704 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:09:11 smithi189 ceph-mon[36187]: osdmap e37: 6 total, 5 up, 6 in 2024-03-24T00:09:11.704 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:09:11 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-24T00:09:11.704 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:09:11 smithi189 ceph-mon[36187]: from='osd.5 [v2:172.21.15.193:6808/2920363402,v1:172.21.15.193:6809/2920363402]' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi193", "root=default"]}]: dispatch 2024-03-24T00:09:11.704 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:09:11 smithi189 ceph-mon[36187]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi193", "root=default"]}]: dispatch 2024-03-24T00:09:11.704 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:09:11 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T00:09:11.704 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:09:11 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T00:09:12.390 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:09:12 smithi193 ceph-mon[36878]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi193", "root=default"]}]': finished 2024-03-24T00:09:12.390 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:09:12 smithi193 ceph-mon[36878]: osdmap e38: 6 total, 5 up, 6 in 2024-03-24T00:09:12.390 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:09:12 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-24T00:09:12.390 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:09:12 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-24T00:09:12.390 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:09:12 smithi193 ceph-mon[36878]: pgmap v151: 1 pgs: 1 active+clean; 577 KiB data, 142 MiB used, 447 GiB / 447 GiB avail 2024-03-24T00:09:12.531 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:09:12 smithi118 ceph-mon[31535]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi193", "root=default"]}]': finished 2024-03-24T00:09:12.532 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:09:12 smithi118 ceph-mon[31535]: osdmap e38: 6 total, 5 up, 6 in 2024-03-24T00:09:12.532 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:09:12 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-24T00:09:12.532 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:09:12 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-24T00:09:12.532 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:09:12 smithi118 ceph-mon[31535]: pgmap v151: 1 pgs: 1 active+clean; 577 KiB data, 142 MiB used, 447 GiB / 447 GiB avail 2024-03-24T00:09:12.704 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:09:12 smithi189 ceph-mon[36187]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi193", "root=default"]}]': finished 2024-03-24T00:09:12.704 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:09:12 smithi189 ceph-mon[36187]: osdmap e38: 6 total, 5 up, 6 in 2024-03-24T00:09:12.704 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:09:12 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-24T00:09:12.704 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:09:12 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-24T00:09:12.704 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:09:12 smithi189 ceph-mon[36187]: pgmap v151: 1 pgs: 1 active+clean; 577 KiB data, 142 MiB used, 447 GiB / 447 GiB avail 2024-03-24T00:09:13.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:09:13 smithi193 ceph-mon[36878]: purged_snaps scrub starts 2024-03-24T00:09:13.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:09:13 smithi193 ceph-mon[36878]: purged_snaps scrub ok 2024-03-24T00:09:13.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:09:13 smithi193 ceph-mon[36878]: osd.5 [v2:172.21.15.193:6808/2920363402,v1:172.21.15.193:6809/2920363402] boot 2024-03-24T00:09:13.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:09:13 smithi193 ceph-mon[36878]: osdmap e39: 6 total, 6 up, 6 in 2024-03-24T00:09:13.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:09:13 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-24T00:09:13.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:09:13 smithi193 ceph-mon[36878]: osdmap e40: 6 total, 6 up, 6 in 2024-03-24T00:09:13.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:09:13 smithi118 ceph-mon[31535]: purged_snaps scrub starts 2024-03-24T00:09:13.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:09:13 smithi118 ceph-mon[31535]: purged_snaps scrub ok 2024-03-24T00:09:13.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:09:13 smithi118 ceph-mon[31535]: osd.5 [v2:172.21.15.193:6808/2920363402,v1:172.21.15.193:6809/2920363402] boot 2024-03-24T00:09:13.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:09:13 smithi118 ceph-mon[31535]: osdmap e39: 6 total, 6 up, 6 in 2024-03-24T00:09:13.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:09:13 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-24T00:09:13.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:09:13 smithi118 ceph-mon[31535]: osdmap e40: 6 total, 6 up, 6 in 2024-03-24T00:09:13.703 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:09:13 smithi189 ceph-mon[36187]: purged_snaps scrub starts 2024-03-24T00:09:13.703 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:09:13 smithi189 ceph-mon[36187]: purged_snaps scrub ok 2024-03-24T00:09:13.703 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:09:13 smithi189 ceph-mon[36187]: osd.5 [v2:172.21.15.193:6808/2920363402,v1:172.21.15.193:6809/2920363402] boot 2024-03-24T00:09:13.703 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:09:13 smithi189 ceph-mon[36187]: osdmap e39: 6 total, 6 up, 6 in 2024-03-24T00:09:13.704 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:09:13 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-24T00:09:13.704 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:09:13 smithi189 ceph-mon[36187]: osdmap e40: 6 total, 6 up, 6 in 2024-03-24T00:09:13.801 INFO:teuthology.orchestra.run.smithi118.stderr:--> Zapping: /dev/vg_nvme/lv_2 2024-03-24T00:09:13.801 INFO:teuthology.orchestra.run.smithi118.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_2 bs=1M count=10 conv=fsync 2024-03-24T00:09:13.802 INFO:teuthology.orchestra.run.smithi118.stderr: stderr: 10+0 records in 2024-03-24T00:09:13.802 INFO:teuthology.orchestra.run.smithi118.stderr:10+0 records out 2024-03-24T00:09:13.802 INFO:teuthology.orchestra.run.smithi118.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0368382 s, 285 MB/s 2024-03-24T00:09:13.802 INFO:teuthology.orchestra.run.smithi118.stderr:--> Zapping successful for: 2024-03-24T00:09:14.548 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:09:14 smithi118 ceph-mon[31535]: pgmap v154: 1 pgs: 1 peering; 577 KiB data, 168 MiB used, 536 GiB / 536 GiB avail 2024-03-24T00:09:14.549 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:09:14 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T00:09:14.549 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:09:14 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T00:09:14.549 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:09:14 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-24T00:09:14.549 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:09:14 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-24T00:09:14.549 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:09:14 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T00:09:14.549 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:09:14 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T00:09:14.550 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:09:14 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T00:09:14.550 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:09:14 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T00:09:14.550 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:09:14 smithi118 ceph-mon[31535]: osdmap e41: 6 total, 6 up, 6 in 2024-03-24T00:09:14.615 DEBUG:teuthology.orchestra.run.smithi118:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d79a5420-e971-11ee-95cd-87774f69a715 -- ceph orch daemon add osd smithi118:vg_nvme/lv_2 2024-03-24T00:09:14.704 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:09:14 smithi189 ceph-mon[36187]: pgmap v154: 1 pgs: 1 peering; 577 KiB data, 168 MiB used, 536 GiB / 536 GiB avail 2024-03-24T00:09:14.704 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:09:14 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T00:09:14.704 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:09:14 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T00:09:14.704 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:09:14 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-24T00:09:14.704 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:09:14 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-24T00:09:14.704 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:09:14 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T00:09:14.704 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:09:14 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T00:09:14.704 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:09:14 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T00:09:14.704 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:09:14 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T00:09:14.704 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:09:14 smithi189 ceph-mon[36187]: osdmap e41: 6 total, 6 up, 6 in 2024-03-24T00:09:14.710 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:09:14 smithi193 ceph-mon[36878]: pgmap v154: 1 pgs: 1 peering; 577 KiB data, 168 MiB used, 536 GiB / 536 GiB avail 2024-03-24T00:09:14.710 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:09:14 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T00:09:14.711 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:09:14 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T00:09:14.711 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:09:14 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-24T00:09:14.711 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:09:14 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-24T00:09:14.711 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:09:14 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T00:09:14.711 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:09:14 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T00:09:14.711 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:09:14 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T00:09:14.711 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:09:14 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T00:09:14.712 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:09:14 smithi193 ceph-mon[36878]: osdmap e41: 6 total, 6 up, 6 in 2024-03-24T00:09:15.057 INFO:teuthology.orchestra.run.smithi118.stderr:Inferring config /var/lib/ceph/d79a5420-e971-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T00:09:15.542 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:09:15 smithi118 ceph-mon[31535]: Detected new or changed devices on smithi193 2024-03-24T00:09:15.542 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:09:15 smithi118 ceph-mon[31535]: Adjusting osd_memory_target on smithi193 to 8540M 2024-03-24T00:09:15.543 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:09:15 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T00:09:15.543 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:09:15 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T00:09:15.543 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:09:15 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T00:09:15.543 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:09:15 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T00:09:15.543 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:09:15 smithi118 ceph-mon[31535]: osdmap e42: 6 total, 6 up, 6 in 2024-03-24T00:09:15.704 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:09:15 smithi189 ceph-mon[36187]: Detected new or changed devices on smithi193 2024-03-24T00:09:15.704 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:09:15 smithi189 ceph-mon[36187]: Adjusting osd_memory_target on smithi193 to 8540M 2024-03-24T00:09:15.705 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:09:15 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T00:09:15.705 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:09:15 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T00:09:15.705 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:09:15 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T00:09:15.705 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:09:15 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T00:09:15.706 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:09:15 smithi189 ceph-mon[36187]: osdmap e42: 6 total, 6 up, 6 in 2024-03-24T00:09:15.711 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:09:15 smithi193 ceph-mon[36878]: Detected new or changed devices on smithi193 2024-03-24T00:09:15.711 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:09:15 smithi193 ceph-mon[36878]: Adjusting osd_memory_target on smithi193 to 8540M 2024-03-24T00:09:15.711 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:09:15 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T00:09:15.711 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:09:15 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T00:09:15.711 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:09:15 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T00:09:15.711 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:09:15 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T00:09:15.711 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:09:15 smithi193 ceph-mon[36878]: osdmap e42: 6 total, 6 up, 6 in 2024-03-24T00:09:16.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:09:16 smithi118 ceph-mon[31535]: pgmap v157: 1 pgs: 1 peering; 577 KiB data, 169 MiB used, 536 GiB / 536 GiB avail 2024-03-24T00:09:16.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:09:16 smithi118 ceph-mon[31535]: osdmap e43: 6 total, 6 up, 6 in 2024-03-24T00:09:16.703 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:09:16 smithi189 ceph-mon[36187]: pgmap v157: 1 pgs: 1 peering; 577 KiB data, 169 MiB used, 536 GiB / 536 GiB avail 2024-03-24T00:09:16.703 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:09:16 smithi189 ceph-mon[36187]: osdmap e43: 6 total, 6 up, 6 in 2024-03-24T00:09:16.710 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:09:16 smithi193 ceph-mon[36878]: pgmap v157: 1 pgs: 1 peering; 577 KiB data, 169 MiB used, 536 GiB / 536 GiB avail 2024-03-24T00:09:16.710 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:09:16 smithi193 ceph-mon[36878]: osdmap e43: 6 total, 6 up, 6 in 2024-03-24T00:09:17.302 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:09:17.302+0000 7f2a71800700 1 -- 172.21.15.118:0/3798206994 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f2a6c105610 msgr2=0x7f2a6c105a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:09:17.302 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:09:17.302+0000 7f2a71800700 1 --2- 172.21.15.118:0/3798206994 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f2a6c105610 0x7f2a6c105a70 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f2a580099c0 tx=0x7f2a58009cd0 comp rx=0 tx=0).stop 2024-03-24T00:09:17.302 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:09:17.302+0000 7f2a71800700 1 -- 172.21.15.118:0/3798206994 shutdown_connections 2024-03-24T00:09:17.302 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:09:17.302+0000 7f2a71800700 1 --2- 172.21.15.118:0/3798206994 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2a6c104410 0x7f2a6c104830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:09:17.302 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:09:17.302+0000 7f2a71800700 1 --2- 172.21.15.118:0/3798206994 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f2a6c105fb0 0x7f2a6c10c860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:09:17.303 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:09:17.302+0000 7f2a71800700 1 --2- 172.21.15.118:0/3798206994 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f2a6c105610 0x7f2a6c105a70 unknown :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:09:17.303 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:09:17.302+0000 7f2a71800700 1 -- 172.21.15.118:0/3798206994 >> 172.21.15.118:0/3798206994 conn(0x7f2a6c0ff9b0 msgr2=0x7f2a6c101df0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:09:17.303 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:09:17.302+0000 7f2a71800700 1 -- 172.21.15.118:0/3798206994 shutdown_connections 2024-03-24T00:09:17.303 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:09:17.302+0000 7f2a71800700 1 -- 172.21.15.118:0/3798206994 wait complete. 2024-03-24T00:09:17.304 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:09:17.303+0000 7f2a71800700 1 Processor -- start 2024-03-24T00:09:17.304 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:09:17.303+0000 7f2a71800700 1 -- start start 2024-03-24T00:09:17.304 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:09:17.304+0000 7f2a71800700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f2a6c104410 0x7f2a6c19a5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:09:17.304 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:09:17.304+0000 7f2a71800700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2a6c105610 0x7f2a6c19aaf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:09:17.304 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:09:17.304+0000 7f2a71800700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f2a6c105fb0 0x7f2a6c1a1b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:09:17.305 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:09:17.304+0000 7f2a71800700 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2a6c110050 con 0x7f2a6c104410 2024-03-24T00:09:17.305 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:09:17.304+0000 7f2a71800700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2a6c10fed0 con 0x7f2a6c105610 2024-03-24T00:09:17.305 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:09:17.304+0000 7f2a71800700 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2a6c1101d0 con 0x7f2a6c105fb0 2024-03-24T00:09:17.305 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:09:17.304+0000 7f2a6aa00700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f2a6c104410 0x7f2a6c19a5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:09:17.305 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:09:17.304+0000 7f2a6aa00700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f2a6c104410 0x7f2a6c19a5b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.118:3300/0 says I am v2:172.21.15.118:45714/0 (socket says 172.21.15.118:45714) 2024-03-24T00:09:17.305 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:09:17.304+0000 7f2a6aa00700 1 -- 172.21.15.118:0/2115888208 learned_addr learned my addr 172.21.15.118:0/2115888208 (peer_addr_for_me v2:172.21.15.118:0/0) 2024-03-24T00:09:17.305 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:09:17.304+0000 7f2a6a000700 1 --2- 172.21.15.118:0/2115888208 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2a6c105610 0x7f2a6c19aaf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:09:17.306 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:09:17.304+0000 7f2a6b400700 1 --2- 172.21.15.118:0/2115888208 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f2a6c105fb0 0x7f2a6c1a1b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:09:17.306 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:09:17.304+0000 7f2a6aa00700 1 -- 172.21.15.118:0/2115888208 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f2a6c105fb0 msgr2=0x7f2a6c1a1b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:09:17.306 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:09:17.304+0000 7f2a6aa00700 1 --2- 172.21.15.118:0/2115888208 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f2a6c105fb0 0x7f2a6c1a1b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:09:17.306 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:09:17.304+0000 7f2a6aa00700 1 -- 172.21.15.118:0/2115888208 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2a6c105610 msgr2=0x7f2a6c19aaf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:09:17.306 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:09:17.305+0000 7f2a6aa00700 1 --2- 172.21.15.118:0/2115888208 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2a6c105610 0x7f2a6c19aaf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:09:17.306 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:09:17.305+0000 7f2a6aa00700 1 -- 172.21.15.118:0/2115888208 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2a6c1a2210 con 0x7f2a6c104410 2024-03-24T00:09:17.306 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:09:17.305+0000 7f2a6a000700 1 --2- 172.21.15.118:0/2115888208 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2a6c105610 0x7f2a6c19aaf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T00:09:17.307 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:09:17.305+0000 7f2a6aa00700 1 --2- 172.21.15.118:0/2115888208 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f2a6c104410 0x7f2a6c19a5b0 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f2a5c00e9a0 tx=0x7f2a5c00ecb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:09:17.308 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:09:17.305+0000 7f2a63400700 1 -- 172.21.15.118:0/2115888208 <== mon.0 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2a5c004220 con 0x7f2a6c104410 2024-03-24T00:09:17.308 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:09:17.305+0000 7f2a63400700 1 -- 172.21.15.118:0/2115888208 <== mon.0 v2:172.21.15.118:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f2a5c0043c0 con 0x7f2a6c104410 2024-03-24T00:09:17.308 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:09:17.305+0000 7f2a71800700 1 -- 172.21.15.118:0/2115888208 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2a6c1a2500 con 0x7f2a6c104410 2024-03-24T00:09:17.308 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:09:17.305+0000 7f2a63400700 1 -- 172.21.15.118:0/2115888208 <== mon.0 v2:172.21.15.118:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2a5c003df0 con 0x7f2a6c104410 2024-03-24T00:09:17.308 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:09:17.305+0000 7f2a71800700 1 -- 172.21.15.118:0/2115888208 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2a6c1a2a40 con 0x7f2a6c104410 2024-03-24T00:09:17.308 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:09:17.306+0000 7f2a71800700 1 -- 172.21.15.118:0/2115888208 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2a6c068e50 con 0x7f2a6c104410 2024-03-24T00:09:17.308 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:09:17.309+0000 7f2a63400700 1 -- 172.21.15.118:0/2115888208 <== mon.0 v2:172.21.15.118:3300/0 4 ==== mgrmap(e 16) v1 ==== 137291+0+0 (secure 0 0 0) 0x7f2a5c00c810 con 0x7f2a6c104410 2024-03-24T00:09:17.309 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:09:17.310+0000 7f2a63400700 1 --2- 172.21.15.118:0/2115888208 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7f2a500a2890 0x7f2a500a4d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:09:17.310 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:09:17.310+0000 7f2a63400700 1 -- 172.21.15.118:0/2115888208 <== mon.0 v2:172.21.15.118:3300/0 5 ==== osd_map(43..43 src has 1..43) v4 ==== 4533+0+0 (secure 0 0 0) 0x7f2a5c0cdef0 con 0x7f2a6c104410 2024-03-24T00:09:17.310 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:09:17.310+0000 7f2a6a000700 1 --2- 172.21.15.118:0/2115888208 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7f2a500a2890 0x7f2a500a4d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:09:17.310 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:09:17.311+0000 7f2a6a000700 1 --2- 172.21.15.118:0/2115888208 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7f2a500a2890 0x7f2a500a4d50 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f2a58000c00 tx=0x7f2a58005fb0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:09:17.312 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:09:17.313+0000 7f2a63400700 1 -- 172.21.15.118:0/2115888208 <== mon.0 v2:172.21.15.118:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f2a5c09be40 con 0x7f2a6c104410 2024-03-24T00:09:17.549 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:09:17.549+0000 7f2a71800700 1 -- 172.21.15.118:0/2115888208 --> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi118:vg_nvme/lv_2", "target": ["mon-mgr", ""]}) v1 -- 0x7f2a6c0625d0 con 0x7f2a500a2890 2024-03-24T00:09:17.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:09:17 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-24T00:09:17.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:09:17 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-24T00:09:17.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:09:17 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T00:09:17.953 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:09:17 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-24T00:09:17.953 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:09:17 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-24T00:09:17.953 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:09:17 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T00:09:17.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:09:17 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-24T00:09:17.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:09:17 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-24T00:09:17.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:09:17 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T00:09:18.695 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:09:18 smithi118 ceph-mon[31535]: pgmap v159: 1 pgs: 1 active+clean; 577 KiB data, 169 MiB used, 536 GiB / 536 GiB avail; 137 KiB/s, 0 objects/s recovering 2024-03-24T00:09:18.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:09:18 smithi118 ceph-mon[31535]: from='client.14277 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi118:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T00:09:18.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:09:18 smithi189 ceph-mon[36187]: pgmap v159: 1 pgs: 1 active+clean; 577 KiB data, 169 MiB used, 536 GiB / 536 GiB avail; 137 KiB/s, 0 objects/s recovering 2024-03-24T00:09:18.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:09:18 smithi189 ceph-mon[36187]: from='client.14277 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi118:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T00:09:18.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:09:18 smithi193 ceph-mon[36878]: pgmap v159: 1 pgs: 1 active+clean; 577 KiB data, 169 MiB used, 536 GiB / 536 GiB avail; 137 KiB/s, 0 objects/s recovering 2024-03-24T00:09:18.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:09:18 smithi193 ceph-mon[36878]: from='client.14277 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi118:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T00:09:20.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:09:20 smithi118 ceph-mon[31535]: pgmap v160: 1 pgs: 1 active+clean; 577 KiB data, 169 MiB used, 536 GiB / 536 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-03-24T00:09:20.953 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:09:20 smithi189 ceph-mon[36187]: pgmap v160: 1 pgs: 1 active+clean; 577 KiB data, 169 MiB used, 536 GiB / 536 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-03-24T00:09:20.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:09:20 smithi193 ceph-mon[36878]: pgmap v160: 1 pgs: 1 active+clean; 577 KiB data, 169 MiB used, 536 GiB / 536 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-03-24T00:09:21.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:09:21 smithi118 ceph-mon[31535]: from='client.? 172.21.15.118:0/2449273897' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "0b77b196-8674-41c6-ad3d-726b5d01c8fd"}]: dispatch 2024-03-24T00:09:21.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:09:21 smithi118 ceph-mon[31535]: from='client.? 172.21.15.118:0/2449273897' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "0b77b196-8674-41c6-ad3d-726b5d01c8fd"}]': finished 2024-03-24T00:09:21.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:09:21 smithi118 ceph-mon[31535]: osdmap e44: 7 total, 6 up, 7 in 2024-03-24T00:09:21.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:09:21 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-24T00:09:21.953 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:09:21 smithi189 ceph-mon[36187]: from='client.? 172.21.15.118:0/2449273897' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "0b77b196-8674-41c6-ad3d-726b5d01c8fd"}]: dispatch 2024-03-24T00:09:21.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:09:21 smithi189 ceph-mon[36187]: from='client.? 172.21.15.118:0/2449273897' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "0b77b196-8674-41c6-ad3d-726b5d01c8fd"}]': finished 2024-03-24T00:09:21.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:09:21 smithi189 ceph-mon[36187]: osdmap e44: 7 total, 6 up, 7 in 2024-03-24T00:09:21.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:09:21 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-24T00:09:21.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:09:21 smithi193 ceph-mon[36878]: from='client.? 172.21.15.118:0/2449273897' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "0b77b196-8674-41c6-ad3d-726b5d01c8fd"}]: dispatch 2024-03-24T00:09:21.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:09:21 smithi193 ceph-mon[36878]: from='client.? 172.21.15.118:0/2449273897' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "0b77b196-8674-41c6-ad3d-726b5d01c8fd"}]': finished 2024-03-24T00:09:21.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:09:21 smithi193 ceph-mon[36878]: osdmap e44: 7 total, 6 up, 7 in 2024-03-24T00:09:21.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:09:21 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-24T00:09:22.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:09:22 smithi118 ceph-mon[31535]: pgmap v162: 1 pgs: 1 active+clean; 577 KiB data, 169 MiB used, 536 GiB / 536 GiB avail; 93 KiB/s, 0 objects/s recovering 2024-03-24T00:09:22.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:09:22 smithi118 ceph-mon[31535]: from='client.? 172.21.15.118:0/2547151373' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-24T00:09:22.953 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:09:22 smithi189 ceph-mon[36187]: pgmap v162: 1 pgs: 1 active+clean; 577 KiB data, 169 MiB used, 536 GiB / 536 GiB avail; 93 KiB/s, 0 objects/s recovering 2024-03-24T00:09:22.953 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:09:22 smithi189 ceph-mon[36187]: from='client.? 172.21.15.118:0/2547151373' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-24T00:09:22.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:09:22 smithi193 ceph-mon[36878]: pgmap v162: 1 pgs: 1 active+clean; 577 KiB data, 169 MiB used, 536 GiB / 536 GiB avail; 93 KiB/s, 0 objects/s recovering 2024-03-24T00:09:22.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:09:22 smithi193 ceph-mon[36878]: from='client.? 172.21.15.118:0/2547151373' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-24T00:09:24.945 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:09:24 smithi118 ceph-mon[31535]: pgmap v163: 1 pgs: 1 active+clean; 577 KiB data, 169 MiB used, 536 GiB / 536 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-03-24T00:09:24.953 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:09:24 smithi189 ceph-mon[36187]: pgmap v163: 1 pgs: 1 active+clean; 577 KiB data, 169 MiB used, 536 GiB / 536 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-03-24T00:09:24.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:09:24 smithi193 ceph-mon[36878]: pgmap v163: 1 pgs: 1 active+clean; 577 KiB data, 169 MiB used, 536 GiB / 536 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-03-24T00:09:26.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:09:26 smithi118 ceph-mon[31535]: pgmap v164: 1 pgs: 1 active+clean; 577 KiB data, 169 MiB used, 536 GiB / 536 GiB avail; 63 KiB/s, 0 objects/s recovering 2024-03-24T00:09:26.953 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:09:26 smithi189 ceph-mon[36187]: pgmap v164: 1 pgs: 1 active+clean; 577 KiB data, 169 MiB used, 536 GiB / 536 GiB avail; 63 KiB/s, 0 objects/s recovering 2024-03-24T00:09:26.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:09:26 smithi193 ceph-mon[36878]: pgmap v164: 1 pgs: 1 active+clean; 577 KiB data, 169 MiB used, 536 GiB / 536 GiB avail; 63 KiB/s, 0 objects/s recovering 2024-03-24T00:09:28.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:09:28 smithi118 ceph-mon[31535]: pgmap v165: 1 pgs: 1 active+clean; 577 KiB data, 169 MiB used, 536 GiB / 536 GiB avail 2024-03-24T00:09:28.953 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:09:28 smithi189 ceph-mon[36187]: pgmap v165: 1 pgs: 1 active+clean; 577 KiB data, 169 MiB used, 536 GiB / 536 GiB avail 2024-03-24T00:09:28.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:09:28 smithi193 ceph-mon[36878]: pgmap v165: 1 pgs: 1 active+clean; 577 KiB data, 169 MiB used, 536 GiB / 536 GiB avail 2024-03-24T00:09:30.862 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:09:30 smithi118 ceph-mon[31535]: pgmap v166: 1 pgs: 1 active+clean; 577 KiB data, 169 MiB used, 536 GiB / 536 GiB avail 2024-03-24T00:09:30.953 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:09:30 smithi189 ceph-mon[36187]: pgmap v166: 1 pgs: 1 active+clean; 577 KiB data, 169 MiB used, 536 GiB / 536 GiB avail 2024-03-24T00:09:30.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:09:30 smithi193 ceph-mon[36878]: pgmap v166: 1 pgs: 1 active+clean; 577 KiB data, 169 MiB used, 536 GiB / 536 GiB avail 2024-03-24T00:09:31.904 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:09:31 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2024-03-24T00:09:31.904 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:09:31 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T00:09:31.953 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:09:31 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2024-03-24T00:09:31.953 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:09:31 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T00:09:31.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:09:31 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.6"}]: dispatch 2024-03-24T00:09:31.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:09:31 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T00:09:32.913 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:09:32 smithi118 ceph-mon[31535]: Deploying daemon osd.6 on smithi118 2024-03-24T00:09:32.913 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:09:32 smithi118 ceph-mon[31535]: pgmap v167: 1 pgs: 1 active+clean; 577 KiB data, 169 MiB used, 536 GiB / 536 GiB avail 2024-03-24T00:09:32.953 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:09:32 smithi189 ceph-mon[36187]: Deploying daemon osd.6 on smithi118 2024-03-24T00:09:32.953 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:09:32 smithi189 ceph-mon[36187]: pgmap v167: 1 pgs: 1 active+clean; 577 KiB data, 169 MiB used, 536 GiB / 536 GiB avail 2024-03-24T00:09:32.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:09:32 smithi193 ceph-mon[36878]: Deploying daemon osd.6 on smithi118 2024-03-24T00:09:32.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:09:32 smithi193 ceph-mon[36878]: pgmap v167: 1 pgs: 1 active+clean; 577 KiB data, 169 MiB used, 536 GiB / 536 GiB avail 2024-03-24T00:09:34.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:09:34 smithi118 ceph-mon[31535]: pgmap v168: 1 pgs: 1 active+clean; 577 KiB data, 169 MiB used, 536 GiB / 536 GiB avail 2024-03-24T00:09:34.953 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:09:34 smithi189 ceph-mon[36187]: pgmap v168: 1 pgs: 1 active+clean; 577 KiB data, 169 MiB used, 536 GiB / 536 GiB avail 2024-03-24T00:09:34.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:09:34 smithi193 ceph-mon[36878]: pgmap v168: 1 pgs: 1 active+clean; 577 KiB data, 169 MiB used, 536 GiB / 536 GiB avail 2024-03-24T00:09:36.910 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:09:36 smithi118 ceph-mon[31535]: pgmap v169: 1 pgs: 1 active+clean; 577 KiB data, 169 MiB used, 536 GiB / 536 GiB avail 2024-03-24T00:09:36.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:09:36 smithi189 ceph-mon[36187]: pgmap v169: 1 pgs: 1 active+clean; 577 KiB data, 169 MiB used, 536 GiB / 536 GiB avail 2024-03-24T00:09:36.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:09:36 smithi193 ceph-mon[36878]: pgmap v169: 1 pgs: 1 active+clean; 577 KiB data, 169 MiB used, 536 GiB / 536 GiB avail 2024-03-24T00:09:38.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:09:38 smithi118 ceph-mon[31535]: pgmap v170: 1 pgs: 1 active+clean; 577 KiB data, 169 MiB used, 536 GiB / 536 GiB avail 2024-03-24T00:09:38.953 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:09:38 smithi189 ceph-mon[36187]: pgmap v170: 1 pgs: 1 active+clean; 577 KiB data, 169 MiB used, 536 GiB / 536 GiB avail 2024-03-24T00:09:38.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:09:38 smithi193 ceph-mon[36878]: pgmap v170: 1 pgs: 1 active+clean; 577 KiB data, 169 MiB used, 536 GiB / 536 GiB avail 2024-03-24T00:09:40.810 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:09:40 smithi118 ceph-mon[31535]: pgmap v171: 1 pgs: 1 active+clean; 577 KiB data, 169 MiB used, 536 GiB / 536 GiB avail 2024-03-24T00:09:40.811 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:09:40 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T00:09:40.811 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:09:40 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T00:09:40.811 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:09:40 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T00:09:40.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:09:40 smithi189 ceph-mon[36187]: pgmap v171: 1 pgs: 1 active+clean; 577 KiB data, 169 MiB used, 536 GiB / 536 GiB avail 2024-03-24T00:09:40.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:09:40 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T00:09:40.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:09:40 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T00:09:40.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:09:40 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T00:09:40.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:09:40 smithi193 ceph-mon[36878]: pgmap v171: 1 pgs: 1 active+clean; 577 KiB data, 169 MiB used, 536 GiB / 536 GiB avail 2024-03-24T00:09:40.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:09:40 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T00:09:40.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:09:40 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T00:09:40.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:09:40 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T00:09:42.729 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:09:42 smithi118 ceph-mon[31535]: pgmap v172: 1 pgs: 1 active+clean; 577 KiB data, 169 MiB used, 536 GiB / 536 GiB avail 2024-03-24T00:09:42.953 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:09:42 smithi189 ceph-mon[36187]: pgmap v172: 1 pgs: 1 active+clean; 577 KiB data, 169 MiB used, 536 GiB / 536 GiB avail 2024-03-24T00:09:42.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:09:42 smithi193 ceph-mon[36878]: pgmap v172: 1 pgs: 1 active+clean; 577 KiB data, 169 MiB used, 536 GiB / 536 GiB avail 2024-03-24T00:09:44.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:09:44 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T00:09:44.447 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:09:44 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T00:09:44.447 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:09:44 smithi118 ceph-mon[31535]: from='osd.6 [v2:172.21.15.118:6818/431354576,v1:172.21.15.118:6819/431354576]' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-03-24T00:09:44.447 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:09:44 smithi118 ceph-mon[31535]: from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-03-24T00:09:44.447 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:09:44 smithi118 ceph-mon[31535]: pgmap v173: 1 pgs: 1 active+clean; 577 KiB data, 169 MiB used, 536 GiB / 536 GiB avail 2024-03-24T00:09:44.648 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:09:44.647+0000 7f2a63400700 1 -- 172.21.15.118:0/2115888208 <== mgr.14152 v2:172.21.15.118:6800/2112912943 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7f2a6c0625d0 con 0x7f2a500a2890 2024-03-24T00:09:44.649 INFO:teuthology.orchestra.run.smithi118.stdout:Created osd(s) 6 on host 'smithi118' 2024-03-24T00:09:44.651 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:09:44.651+0000 7f2a71800700 1 -- 172.21.15.118:0/2115888208 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7f2a500a2890 msgr2=0x7f2a500a4d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:09:44.652 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:09:44.651+0000 7f2a71800700 1 --2- 172.21.15.118:0/2115888208 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7f2a500a2890 0x7f2a500a4d50 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f2a58000c00 tx=0x7f2a58005fb0 comp rx=0 tx=0).stop 2024-03-24T00:09:44.652 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:09:44.651+0000 7f2a71800700 1 -- 172.21.15.118:0/2115888208 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f2a6c104410 msgr2=0x7f2a6c19a5b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:09:44.652 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:09:44.651+0000 7f2a71800700 1 --2- 172.21.15.118:0/2115888208 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f2a6c104410 0x7f2a6c19a5b0 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f2a5c00e9a0 tx=0x7f2a5c00ecb0 comp rx=0 tx=0).stop 2024-03-24T00:09:44.652 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:09:44.651+0000 7f2a71800700 1 -- 172.21.15.118:0/2115888208 shutdown_connections 2024-03-24T00:09:44.652 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:09:44.651+0000 7f2a71800700 1 --2- 172.21.15.118:0/2115888208 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7f2a500a2890 0x7f2a500a4d50 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:09:44.652 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:09:44.651+0000 7f2a71800700 1 --2- 172.21.15.118:0/2115888208 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f2a6c105fb0 0x7f2a6c1a1b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:09:44.652 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:09:44.651+0000 7f2a71800700 1 --2- 172.21.15.118:0/2115888208 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f2a6c104410 0x7f2a6c19a5b0 unknown :-1 s=CLOSED pgs=121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:09:44.652 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:09:44.651+0000 7f2a71800700 1 --2- 172.21.15.118:0/2115888208 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2a6c105610 0x7f2a6c19aaf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:09:44.652 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:09:44.651+0000 7f2a71800700 1 -- 172.21.15.118:0/2115888208 >> 172.21.15.118:0/2115888208 conn(0x7f2a6c0ff9b0 msgr2=0x7f2a6c1088b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:09:44.652 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:09:44.651+0000 7f2a71800700 1 -- 172.21.15.118:0/2115888208 shutdown_connections 2024-03-24T00:09:44.652 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:09:44.651+0000 7f2a71800700 1 -- 172.21.15.118:0/2115888208 wait complete. 2024-03-24T00:09:44.704 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:09:44 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T00:09:44.704 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:09:44 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T00:09:44.704 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:09:44 smithi189 ceph-mon[36187]: from='osd.6 [v2:172.21.15.118:6818/431354576,v1:172.21.15.118:6819/431354576]' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-03-24T00:09:44.704 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:09:44 smithi189 ceph-mon[36187]: from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-03-24T00:09:44.704 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:09:44 smithi189 ceph-mon[36187]: pgmap v173: 1 pgs: 1 active+clean; 577 KiB data, 169 MiB used, 536 GiB / 536 GiB avail 2024-03-24T00:09:44.710 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:09:44 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T00:09:44.710 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:09:44 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T00:09:44.711 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:09:44 smithi193 ceph-mon[36878]: from='osd.6 [v2:172.21.15.118:6818/431354576,v1:172.21.15.118:6819/431354576]' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-03-24T00:09:44.711 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:09:44 smithi193 ceph-mon[36878]: from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]: dispatch 2024-03-24T00:09:44.711 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:09:44 smithi193 ceph-mon[36878]: pgmap v173: 1 pgs: 1 active+clean; 577 KiB data, 169 MiB used, 536 GiB / 536 GiB avail 2024-03-24T00:09:45.444 DEBUG:teuthology.orchestra.run.smithi118:osd.6> sudo journalctl -f -n 0 -u ceph-d79a5420-e971-11ee-95cd-87774f69a715@osd.6.service 2024-03-24T00:09:45.451 INFO:tasks.cephadm:Deploying osd.7 on smithi189 with /dev/vg_nvme/lv_2... 2024-03-24T00:09:45.451 DEBUG:teuthology.orchestra.run.smithi189:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d79a5420-e971-11ee-95cd-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_2 2024-03-24T00:09:45.695 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:09:45 smithi118 ceph-mon[31535]: from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2024-03-24T00:09:45.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:09:45 smithi118 ceph-mon[31535]: osdmap e45: 7 total, 6 up, 7 in 2024-03-24T00:09:45.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:09:45 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-24T00:09:45.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:09:45 smithi118 ceph-mon[31535]: from='osd.6 [v2:172.21.15.118:6818/431354576,v1:172.21.15.118:6819/431354576]' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi118", "root=default"]}]: dispatch 2024-03-24T00:09:45.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:09:45 smithi118 ceph-mon[31535]: from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi118", "root=default"]}]: dispatch 2024-03-24T00:09:45.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:09:45 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T00:09:45.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:09:45 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T00:09:45.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:09:45 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T00:09:45.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:09:45 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T00:09:45.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:09:45 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T00:09:45.697 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:09:45 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T00:09:45.700 INFO:teuthology.orchestra.run.smithi189.stderr:Inferring config /var/lib/ceph/d79a5420-e971-11ee-95cd-87774f69a715/mon.b/config 2024-03-24T00:09:45.703 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:09:45 smithi189 ceph-mon[36187]: from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2024-03-24T00:09:45.703 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:09:45 smithi189 ceph-mon[36187]: osdmap e45: 7 total, 6 up, 7 in 2024-03-24T00:09:45.703 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:09:45 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-24T00:09:45.703 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:09:45 smithi189 ceph-mon[36187]: from='osd.6 [v2:172.21.15.118:6818/431354576,v1:172.21.15.118:6819/431354576]' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi118", "root=default"]}]: dispatch 2024-03-24T00:09:45.704 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:09:45 smithi189 ceph-mon[36187]: from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi118", "root=default"]}]: dispatch 2024-03-24T00:09:45.704 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:09:45 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T00:09:45.704 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:09:45 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T00:09:45.704 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:09:45 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T00:09:45.704 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:09:45 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T00:09:45.704 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:09:45 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T00:09:45.704 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:09:45 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T00:09:45.710 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:09:45 smithi193 ceph-mon[36878]: from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["6"]}]': finished 2024-03-24T00:09:45.710 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:09:45 smithi193 ceph-mon[36878]: osdmap e45: 7 total, 6 up, 7 in 2024-03-24T00:09:45.710 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:09:45 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-24T00:09:45.710 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:09:45 smithi193 ceph-mon[36878]: from='osd.6 [v2:172.21.15.118:6818/431354576,v1:172.21.15.118:6819/431354576]' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi118", "root=default"]}]: dispatch 2024-03-24T00:09:45.711 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:09:45 smithi193 ceph-mon[36878]: from='osd.6 ' entity='osd.6' cmd=[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi118", "root=default"]}]: dispatch 2024-03-24T00:09:45.711 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:09:45 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T00:09:45.711 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:09:45 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T00:09:45.711 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:09:45 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T00:09:45.711 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:09:45 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T00:09:45.711 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:09:45 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T00:09:45.711 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:09:45 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T00:09:46.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:09:46 smithi189 ceph-mon[36187]: from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi118", "root=default"]}]': finished 2024-03-24T00:09:46.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:09:46 smithi189 ceph-mon[36187]: osdmap e46: 7 total, 6 up, 7 in 2024-03-24T00:09:46.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:09:46 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-24T00:09:46.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:09:46 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-24T00:09:46.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:09:46 smithi189 ceph-mon[36187]: pgmap v176: 1 pgs: 1 active+clean; 577 KiB data, 169 MiB used, 536 GiB / 536 GiB avail 2024-03-24T00:09:46.638 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:09:46 smithi118 ceph-mon[31535]: from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi118", "root=default"]}]': finished 2024-03-24T00:09:46.639 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:09:46 smithi118 ceph-mon[31535]: osdmap e46: 7 total, 6 up, 7 in 2024-03-24T00:09:46.639 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:09:46 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-24T00:09:46.639 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:09:46 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-24T00:09:46.639 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:09:46 smithi118 ceph-mon[31535]: pgmap v176: 1 pgs: 1 active+clean; 577 KiB data, 169 MiB used, 536 GiB / 536 GiB avail 2024-03-24T00:09:46.710 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:09:46 smithi193 ceph-mon[36878]: from='osd.6 ' entity='osd.6' cmd='[{"prefix": "osd crush create-or-move", "id": 6, "weight":0.0873, "args": ["host=smithi118", "root=default"]}]': finished 2024-03-24T00:09:46.711 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:09:46 smithi193 ceph-mon[36878]: osdmap e46: 7 total, 6 up, 7 in 2024-03-24T00:09:46.711 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:09:46 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-24T00:09:46.711 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:09:46 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-24T00:09:46.711 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:09:46 smithi193 ceph-mon[36878]: pgmap v176: 1 pgs: 1 active+clean; 577 KiB data, 169 MiB used, 536 GiB / 536 GiB avail 2024-03-24T00:09:47.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:09:47 smithi189 ceph-mon[36187]: purged_snaps scrub starts 2024-03-24T00:09:47.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:09:47 smithi189 ceph-mon[36187]: purged_snaps scrub ok 2024-03-24T00:09:47.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:09:47 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-24T00:09:47.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:09:47 smithi189 ceph-mon[36187]: osd.6 [v2:172.21.15.118:6818/431354576,v1:172.21.15.118:6819/431354576] boot 2024-03-24T00:09:47.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:09:47 smithi189 ceph-mon[36187]: osdmap e47: 7 total, 7 up, 7 in 2024-03-24T00:09:47.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:09:47 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-24T00:09:47.514 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:09:47 smithi118 ceph-mon[31535]: purged_snaps scrub starts 2024-03-24T00:09:47.514 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:09:47 smithi118 ceph-mon[31535]: purged_snaps scrub ok 2024-03-24T00:09:47.514 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:09:47 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-24T00:09:47.515 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:09:47 smithi118 ceph-mon[31535]: osd.6 [v2:172.21.15.118:6818/431354576,v1:172.21.15.118:6819/431354576] boot 2024-03-24T00:09:47.515 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:09:47 smithi118 ceph-mon[31535]: osdmap e47: 7 total, 7 up, 7 in 2024-03-24T00:09:47.515 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:09:47 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-24T00:09:47.710 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:09:47 smithi193 ceph-mon[36878]: purged_snaps scrub starts 2024-03-24T00:09:47.711 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:09:47 smithi193 ceph-mon[36878]: purged_snaps scrub ok 2024-03-24T00:09:47.711 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:09:47 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-24T00:09:47.711 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:09:47 smithi193 ceph-mon[36878]: osd.6 [v2:172.21.15.118:6818/431354576,v1:172.21.15.118:6819/431354576] boot 2024-03-24T00:09:47.711 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:09:47 smithi193 ceph-mon[36878]: osdmap e47: 7 total, 7 up, 7 in 2024-03-24T00:09:47.711 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:09:47 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 6}]: dispatch 2024-03-24T00:09:47.769 INFO:teuthology.orchestra.run.smithi189.stderr:--> Zapping: /dev/vg_nvme/lv_2 2024-03-24T00:09:47.769 INFO:teuthology.orchestra.run.smithi189.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_2 bs=1M count=10 conv=fsync 2024-03-24T00:09:47.769 INFO:teuthology.orchestra.run.smithi189.stderr: stderr: 10+0 records in 2024-03-24T00:09:47.769 INFO:teuthology.orchestra.run.smithi189.stderr:10+0 records out 2024-03-24T00:09:47.769 INFO:teuthology.orchestra.run.smithi189.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0139295 s, 753 MB/s 2024-03-24T00:09:47.769 INFO:teuthology.orchestra.run.smithi189.stderr:--> Zapping successful for: 2024-03-24T00:09:48.453 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:09:48 smithi189 ceph-mon[36187]: osdmap e48: 7 total, 7 up, 7 in 2024-03-24T00:09:48.453 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:09:48 smithi189 ceph-mon[36187]: pgmap v179: 1 pgs: 1 active+clean; 577 KiB data, 196 MiB used, 626 GiB / 626 GiB avail 2024-03-24T00:09:48.512 DEBUG:teuthology.orchestra.run.smithi189:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d79a5420-e971-11ee-95cd-87774f69a715 -- ceph orch daemon add osd smithi189:vg_nvme/lv_2 2024-03-24T00:09:48.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:09:48 smithi118 ceph-mon[31535]: osdmap e48: 7 total, 7 up, 7 in 2024-03-24T00:09:48.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:09:48 smithi118 ceph-mon[31535]: pgmap v179: 1 pgs: 1 active+clean; 577 KiB data, 196 MiB used, 626 GiB / 626 GiB avail 2024-03-24T00:09:48.710 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:09:48 smithi193 ceph-mon[36878]: osdmap e48: 7 total, 7 up, 7 in 2024-03-24T00:09:48.710 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:09:48 smithi193 ceph-mon[36878]: pgmap v179: 1 pgs: 1 active+clean; 577 KiB data, 196 MiB used, 626 GiB / 626 GiB avail 2024-03-24T00:09:48.761 INFO:teuthology.orchestra.run.smithi189.stderr:Inferring config /var/lib/ceph/d79a5420-e971-11ee-95cd-87774f69a715/mon.b/config 2024-03-24T00:09:50.458 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-24T00:09:50.456+0000 7f5b90b0a700 1 -- 172.21.15.189:0/3554402236 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5b8c102330 msgr2=0x7f5b8c102790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:09:50.458 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-24T00:09:50.456+0000 7f5b90b0a700 1 --2- 172.21.15.189:0/3554402236 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5b8c102330 0x7f5b8c102790 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7f5b800099c0 tx=0x7f5b80009cd0 comp rx=0 tx=0).stop 2024-03-24T00:09:50.458 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-24T00:09:50.457+0000 7f5b90b0a700 1 -- 172.21.15.189:0/3554402236 shutdown_connections 2024-03-24T00:09:50.458 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-24T00:09:50.457+0000 7f5b90b0a700 1 --2- 172.21.15.189:0/3554402236 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f5b8c102cd0 0x7f5b8c109580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:09:50.459 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-24T00:09:50.457+0000 7f5b90b0a700 1 --2- 172.21.15.189:0/3554402236 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f5b8c101130 0x7f5b8c101550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:09:50.459 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-24T00:09:50.457+0000 7f5b90b0a700 1 --2- 172.21.15.189:0/3554402236 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5b8c102330 0x7f5b8c102790 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:09:50.459 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-24T00:09:50.457+0000 7f5b90b0a700 1 -- 172.21.15.189:0/3554402236 >> 172.21.15.189:0/3554402236 conn(0x7f5b8c0fc6f0 msgr2=0x7f5b8c0feb10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:09:50.459 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-24T00:09:50.457+0000 7f5b90b0a700 1 -- 172.21.15.189:0/3554402236 shutdown_connections 2024-03-24T00:09:50.459 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-24T00:09:50.457+0000 7f5b90b0a700 1 -- 172.21.15.189:0/3554402236 wait complete. 2024-03-24T00:09:50.459 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-24T00:09:50.457+0000 7f5b90b0a700 1 Processor -- start 2024-03-24T00:09:50.459 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-24T00:09:50.457+0000 7f5b90b0a700 1 -- start start 2024-03-24T00:09:50.459 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-24T00:09:50.457+0000 7f5b90b0a700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f5b8c101130 0x7f5b8c06b450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:09:50.459 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-24T00:09:50.458+0000 7f5b90b0a700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f5b8c102330 0x7f5b8c19d240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:09:50.459 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-24T00:09:50.458+0000 7f5b90b0a700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5b8c102cd0 0x7f5b8c19f8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:09:50.459 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-24T00:09:50.458+0000 7f5b90b0a700 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5b8c10cda0 con 0x7f5b8c101130 2024-03-24T00:09:50.459 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-24T00:09:50.458+0000 7f5b90b0a700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5b8c10cc20 con 0x7f5b8c102cd0 2024-03-24T00:09:50.459 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-24T00:09:50.458+0000 7f5b90b0a700 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5b8c10cf20 con 0x7f5b8c102330 2024-03-24T00:09:50.459 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-24T00:09:50.458+0000 7f5b8ad9d700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5b8c102cd0 0x7f5b8c19f8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:09:50.460 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-24T00:09:50.458+0000 7f5b8ad9d700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5b8c102cd0 0x7f5b8c19f8d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:39236/0 (socket says 172.21.15.189:39236) 2024-03-24T00:09:50.460 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-24T00:09:50.458+0000 7f5b8ad9d700 1 -- 172.21.15.189:0/927429449 learned_addr learned my addr 172.21.15.189:0/927429449 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-03-24T00:09:50.460 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-24T00:09:50.458+0000 7f5b89d9b700 1 --2- 172.21.15.189:0/927429449 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f5b8c102330 0x7f5b8c19d240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:09:50.460 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-24T00:09:50.458+0000 7f5b8a59c700 1 --2- 172.21.15.189:0/927429449 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f5b8c101130 0x7f5b8c06b450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:09:50.460 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-24T00:09:50.458+0000 7f5b8ad9d700 1 -- 172.21.15.189:0/927429449 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f5b8c102330 msgr2=0x7f5b8c19d240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:09:50.460 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-24T00:09:50.458+0000 7f5b8ad9d700 1 --2- 172.21.15.189:0/927429449 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f5b8c102330 0x7f5b8c19d240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:09:50.460 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-24T00:09:50.458+0000 7f5b8ad9d700 1 -- 172.21.15.189:0/927429449 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f5b8c101130 msgr2=0x7f5b8c06b450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:09:50.460 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-24T00:09:50.458+0000 7f5b8ad9d700 1 --2- 172.21.15.189:0/927429449 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f5b8c101130 0x7f5b8c06b450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:09:50.460 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-24T00:09:50.458+0000 7f5b8ad9d700 1 -- 172.21.15.189:0/927429449 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5b8c1a01a0 con 0x7f5b8c102cd0 2024-03-24T00:09:50.460 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-24T00:09:50.458+0000 7f5b8ad9d700 1 --2- 172.21.15.189:0/927429449 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5b8c102cd0 0x7f5b8c19f8d0 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f5b8400b970 tx=0x7f5b8400bc80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:09:50.460 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-24T00:09:50.458+0000 7f5b7b7fe700 1 -- 172.21.15.189:0/927429449 <== mon.2 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5b84019070 con 0x7f5b8c102cd0 2024-03-24T00:09:50.460 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-24T00:09:50.458+0000 7f5b90b0a700 1 -- 172.21.15.189:0/927429449 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5b8c1a0490 con 0x7f5b8c102cd0 2024-03-24T00:09:50.460 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-24T00:09:50.458+0000 7f5b7b7fe700 1 -- 172.21.15.189:0/927429449 <== mon.2 v2:172.21.15.189:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f5b8400ce10 con 0x7f5b8c102cd0 2024-03-24T00:09:50.460 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-24T00:09:50.458+0000 7f5b7b7fe700 1 -- 172.21.15.189:0/927429449 <== mon.2 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5b84007370 con 0x7f5b8c102cd0 2024-03-24T00:09:50.461 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-24T00:09:50.458+0000 7f5b90b0a700 1 -- 172.21.15.189:0/927429449 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5b8c1a0aa0 con 0x7f5b8c102cd0 2024-03-24T00:09:50.461 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-24T00:09:50.460+0000 7f5b90b0a700 1 -- 172.21.15.189:0/927429449 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5b8c068e50 con 0x7f5b8c102cd0 2024-03-24T00:09:50.461 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-24T00:09:50.461+0000 7f5b7b7fe700 1 -- 172.21.15.189:0/927429449 <== mon.2 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 16) v1 ==== 137291+0+0 (secure 0 0 0) 0x7f5b8401c070 con 0x7f5b8c102cd0 2024-03-24T00:09:50.462 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-24T00:09:50.461+0000 7f5b7b7fe700 1 --2- 172.21.15.189:0/927429449 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7f5b740a2890 0x7f5b740a4d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:09:50.462 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-24T00:09:50.462+0000 7f5b7b7fe700 1 -- 172.21.15.189:0/927429449 <== mon.2 v2:172.21.15.189:3300/0 5 ==== osd_map(48..48 src has 1..48) v4 ==== 4965+0+0 (secure 0 0 0) 0x7f5b840cd640 con 0x7f5b8c102cd0 2024-03-24T00:09:50.462 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-24T00:09:50.462+0000 7f5b8a59c700 1 --2- 172.21.15.189:0/927429449 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7f5b740a2890 0x7f5b740a4d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:09:50.462 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-24T00:09:50.463+0000 7f5b8a59c700 1 --2- 172.21.15.189:0/927429449 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7f5b740a2890 0x7f5b740a4d50 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f5b7c006fd0 tx=0x7f5b7c008040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:09:50.464 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-24T00:09:50.465+0000 7f5b7b7fe700 1 -- 172.21.15.189:0/927429449 <== mon.2 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f5b8409b350 con 0x7f5b8c102cd0 2024-03-24T00:09:50.670 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-24T00:09:50.671+0000 7f5b90b0a700 1 -- 172.21.15.189:0/927429449 --> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi189:vg_nvme/lv_2", "target": ["mon-mgr", ""]}) v1 -- 0x7f5b8c0625d0 con 0x7f5b740a2890 2024-03-24T00:09:50.945 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:09:50 smithi118 ceph-mon[31535]: pgmap v180: 1 pgs: 1 active+clean; 577 KiB data, 196 MiB used, 626 GiB / 626 GiB avail 2024-03-24T00:09:50.953 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:09:50 smithi189 ceph-mon[36187]: pgmap v180: 1 pgs: 1 active+clean; 577 KiB data, 196 MiB used, 626 GiB / 626 GiB avail 2024-03-24T00:09:50.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:09:50 smithi193 ceph-mon[36878]: pgmap v180: 1 pgs: 1 active+clean; 577 KiB data, 196 MiB used, 626 GiB / 626 GiB avail 2024-03-24T00:09:51.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:09:51 smithi118 ceph-mon[31535]: from='client.24293 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi189:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T00:09:51.947 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:09:51 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-24T00:09:51.947 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:09:51 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-24T00:09:51.947 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:09:51 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T00:09:51.947 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:09:51 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T00:09:51.947 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:09:51 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T00:09:51.948 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:09:51 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-24T00:09:51.948 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:09:51 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-24T00:09:51.948 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:09:51 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-24T00:09:51.948 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:09:51 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T00:09:51.948 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:09:51 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T00:09:51.948 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:09:51 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T00:09:51.949 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:09:51 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T00:09:51.953 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:09:51 smithi189 ceph-mon[36187]: from='client.24293 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi189:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T00:09:51.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:09:51 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-24T00:09:51.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:09:51 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-24T00:09:51.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:09:51 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T00:09:51.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:09:51 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T00:09:51.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:09:51 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T00:09:51.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:09:51 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-24T00:09:51.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:09:51 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-24T00:09:51.955 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:09:51 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-24T00:09:51.955 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:09:51 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T00:09:51.955 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:09:51 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T00:09:51.955 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:09:51 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T00:09:51.955 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:09:51 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T00:09:51.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:09:51 smithi193 ceph-mon[36878]: from='client.24293 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi189:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T00:09:51.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:09:51 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-24T00:09:51.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:09:51 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-24T00:09:51.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:09:51 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T00:09:51.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:09:51 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T00:09:51.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:09:51 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T00:09:51.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:09:51 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-24T00:09:51.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:09:51 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-24T00:09:51.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:09:51 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-24T00:09:51.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:09:51 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T00:09:51.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:09:51 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T00:09:51.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:09:51 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T00:09:51.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:09:51 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T00:09:52.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:09:52 smithi118 ceph-mon[31535]: Detected new or changed devices on smithi118 2024-03-24T00:09:52.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:09:52 smithi118 ceph-mon[31535]: Adjusting osd_memory_target on smithi118 to 5697M 2024-03-24T00:09:52.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:09:52 smithi118 ceph-mon[31535]: pgmap v181: 1 pgs: 1 active+clean; 577 KiB data, 196 MiB used, 626 GiB / 626 GiB avail 2024-03-24T00:09:52.953 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:09:52 smithi189 ceph-mon[36187]: Detected new or changed devices on smithi118 2024-03-24T00:09:52.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:09:52 smithi189 ceph-mon[36187]: Adjusting osd_memory_target on smithi118 to 5697M 2024-03-24T00:09:52.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:09:52 smithi189 ceph-mon[36187]: pgmap v181: 1 pgs: 1 active+clean; 577 KiB data, 196 MiB used, 626 GiB / 626 GiB avail 2024-03-24T00:09:52.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:09:52 smithi193 ceph-mon[36878]: Detected new or changed devices on smithi118 2024-03-24T00:09:52.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:09:52 smithi193 ceph-mon[36878]: Adjusting osd_memory_target on smithi118 to 5697M 2024-03-24T00:09:52.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:09:52 smithi193 ceph-mon[36878]: pgmap v181: 1 pgs: 1 active+clean; 577 KiB data, 196 MiB used, 626 GiB / 626 GiB avail 2024-03-24T00:09:53.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:09:53 smithi118 ceph-mon[31535]: from='client.? 172.21.15.189:0/3358290706' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "1fbbd38f-4e8e-4d4c-8e98-66f6862bc4bd"}]: dispatch 2024-03-24T00:09:53.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:09:53 smithi118 ceph-mon[31535]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "1fbbd38f-4e8e-4d4c-8e98-66f6862bc4bd"}]: dispatch 2024-03-24T00:09:53.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:09:53 smithi118 ceph-mon[31535]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "1fbbd38f-4e8e-4d4c-8e98-66f6862bc4bd"}]': finished 2024-03-24T00:09:53.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:09:53 smithi118 ceph-mon[31535]: osdmap e49: 8 total, 7 up, 8 in 2024-03-24T00:09:53.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:09:53 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-24T00:09:53.953 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:09:53 smithi189 ceph-mon[36187]: from='client.? 172.21.15.189:0/3358290706' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "1fbbd38f-4e8e-4d4c-8e98-66f6862bc4bd"}]: dispatch 2024-03-24T00:09:53.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:09:53 smithi189 ceph-mon[36187]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "1fbbd38f-4e8e-4d4c-8e98-66f6862bc4bd"}]: dispatch 2024-03-24T00:09:53.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:09:53 smithi189 ceph-mon[36187]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "1fbbd38f-4e8e-4d4c-8e98-66f6862bc4bd"}]': finished 2024-03-24T00:09:53.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:09:53 smithi189 ceph-mon[36187]: osdmap e49: 8 total, 7 up, 8 in 2024-03-24T00:09:53.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:09:53 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-24T00:09:53.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:09:53 smithi193 ceph-mon[36878]: from='client.? 172.21.15.189:0/3358290706' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "1fbbd38f-4e8e-4d4c-8e98-66f6862bc4bd"}]: dispatch 2024-03-24T00:09:53.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:09:53 smithi193 ceph-mon[36878]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "1fbbd38f-4e8e-4d4c-8e98-66f6862bc4bd"}]: dispatch 2024-03-24T00:09:53.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:09:53 smithi193 ceph-mon[36878]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "1fbbd38f-4e8e-4d4c-8e98-66f6862bc4bd"}]': finished 2024-03-24T00:09:53.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:09:53 smithi193 ceph-mon[36878]: osdmap e49: 8 total, 7 up, 8 in 2024-03-24T00:09:53.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:09:53 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-24T00:09:54.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:09:54 smithi118 ceph-mon[31535]: pgmap v183: 1 pgs: 1 active+clean; 577 KiB data, 196 MiB used, 626 GiB / 626 GiB avail 2024-03-24T00:09:54.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:09:54 smithi118 ceph-mon[31535]: from='client.? 172.21.15.189:0/1593928383' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-24T00:09:54.953 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:09:54 smithi189 ceph-mon[36187]: pgmap v183: 1 pgs: 1 active+clean; 577 KiB data, 196 MiB used, 626 GiB / 626 GiB avail 2024-03-24T00:09:54.953 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:09:54 smithi189 ceph-mon[36187]: from='client.? 172.21.15.189:0/1593928383' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-24T00:09:54.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:09:54 smithi193 ceph-mon[36878]: pgmap v183: 1 pgs: 1 active+clean; 577 KiB data, 196 MiB used, 626 GiB / 626 GiB avail 2024-03-24T00:09:54.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:09:54 smithi193 ceph-mon[36878]: from='client.? 172.21.15.189:0/1593928383' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-24T00:09:56.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:09:56 smithi118 ceph-mon[31535]: pgmap v184: 1 pgs: 1 active+clean; 577 KiB data, 196 MiB used, 626 GiB / 626 GiB avail 2024-03-24T00:09:56.953 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:09:56 smithi189 ceph-mon[36187]: pgmap v184: 1 pgs: 1 active+clean; 577 KiB data, 196 MiB used, 626 GiB / 626 GiB avail 2024-03-24T00:09:56.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:09:56 smithi193 ceph-mon[36878]: pgmap v184: 1 pgs: 1 active+clean; 577 KiB data, 196 MiB used, 626 GiB / 626 GiB avail 2024-03-24T00:09:58.690 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:09:58 smithi189 ceph-mon[36187]: pgmap v185: 1 pgs: 1 active+clean; 577 KiB data, 196 MiB used, 626 GiB / 626 GiB avail 2024-03-24T00:09:58.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:09:58 smithi118 ceph-mon[31535]: pgmap v185: 1 pgs: 1 active+clean; 577 KiB data, 196 MiB used, 626 GiB / 626 GiB avail 2024-03-24T00:09:58.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:09:58 smithi193 ceph-mon[36878]: pgmap v185: 1 pgs: 1 active+clean; 577 KiB data, 196 MiB used, 626 GiB / 626 GiB avail 2024-03-24T00:10:00.704 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:10:00 smithi189 ceph-mon[36187]: pgmap v186: 1 pgs: 1 active+clean; 577 KiB data, 196 MiB used, 626 GiB / 626 GiB avail 2024-03-24T00:10:00.704 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:10:00 smithi189 ceph-mon[36187]: overall HEALTH_OK 2024-03-24T00:10:00.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:10:00 smithi118 ceph-mon[31535]: pgmap v186: 1 pgs: 1 active+clean; 577 KiB data, 196 MiB used, 626 GiB / 626 GiB avail 2024-03-24T00:10:00.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:10:00 smithi118 ceph-mon[31535]: overall HEALTH_OK 2024-03-24T00:10:00.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:10:00 smithi193 ceph-mon[36878]: pgmap v186: 1 pgs: 1 active+clean; 577 KiB data, 196 MiB used, 626 GiB / 626 GiB avail 2024-03-24T00:10:00.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:10:00 smithi193 ceph-mon[36878]: overall HEALTH_OK 2024-03-24T00:10:01.704 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:10:01 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2024-03-24T00:10:01.704 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:10:01 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T00:10:01.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:10:01 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2024-03-24T00:10:01.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:10:01 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T00:10:01.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:10:01 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.7"}]: dispatch 2024-03-24T00:10:01.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:10:01 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T00:10:02.704 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:10:02 smithi189 ceph-mon[36187]: Deploying daemon osd.7 on smithi189 2024-03-24T00:10:02.704 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:10:02 smithi189 ceph-mon[36187]: pgmap v187: 1 pgs: 1 active+clean; 577 KiB data, 196 MiB used, 626 GiB / 626 GiB avail 2024-03-24T00:10:02.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:10:02 smithi118 ceph-mon[31535]: Deploying daemon osd.7 on smithi189 2024-03-24T00:10:02.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:10:02 smithi118 ceph-mon[31535]: pgmap v187: 1 pgs: 1 active+clean; 577 KiB data, 196 MiB used, 626 GiB / 626 GiB avail 2024-03-24T00:10:02.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:10:02 smithi193 ceph-mon[36878]: Deploying daemon osd.7 on smithi189 2024-03-24T00:10:02.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:10:02 smithi193 ceph-mon[36878]: pgmap v187: 1 pgs: 1 active+clean; 577 KiB data, 196 MiB used, 626 GiB / 626 GiB avail 2024-03-24T00:10:04.805 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:10:04 smithi189 ceph-mon[36187]: pgmap v188: 1 pgs: 1 active+clean; 577 KiB data, 196 MiB used, 626 GiB / 626 GiB avail 2024-03-24T00:10:04.945 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:10:04 smithi118 ceph-mon[31535]: pgmap v188: 1 pgs: 1 active+clean; 577 KiB data, 196 MiB used, 626 GiB / 626 GiB avail 2024-03-24T00:10:04.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:10:04 smithi193 ceph-mon[36878]: pgmap v188: 1 pgs: 1 active+clean; 577 KiB data, 196 MiB used, 626 GiB / 626 GiB avail 2024-03-24T00:10:06.901 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:10:06 smithi118 ceph-mon[31535]: pgmap v189: 1 pgs: 1 active+clean; 577 KiB data, 196 MiB used, 626 GiB / 626 GiB avail 2024-03-24T00:10:06.901 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:10:06 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T00:10:06.901 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:10:06 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T00:10:06.901 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:10:06 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T00:10:06.953 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:10:06 smithi189 ceph-mon[36187]: pgmap v189: 1 pgs: 1 active+clean; 577 KiB data, 196 MiB used, 626 GiB / 626 GiB avail 2024-03-24T00:10:06.953 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:10:06 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T00:10:06.953 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:10:06 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T00:10:06.953 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:10:06 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T00:10:06.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:10:06 smithi193 ceph-mon[36878]: pgmap v189: 1 pgs: 1 active+clean; 577 KiB data, 196 MiB used, 626 GiB / 626 GiB avail 2024-03-24T00:10:06.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:10:06 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T00:10:06.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:10:06 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T00:10:06.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:10:06 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T00:10:08.909 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:10:08 smithi189 ceph-mon[36187]: pgmap v190: 1 pgs: 1 active+clean; 577 KiB data, 196 MiB used, 626 GiB / 626 GiB avail 2024-03-24T00:10:08.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:10:08 smithi118 ceph-mon[31535]: pgmap v190: 1 pgs: 1 active+clean; 577 KiB data, 196 MiB used, 626 GiB / 626 GiB avail 2024-03-24T00:10:08.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:10:08 smithi193 ceph-mon[36878]: pgmap v190: 1 pgs: 1 active+clean; 577 KiB data, 196 MiB used, 626 GiB / 626 GiB avail 2024-03-24T00:10:09.955 INFO:teuthology.orchestra.run.smithi189.stdout:Created osd(s) 7 on host 'smithi189' 2024-03-24T00:10:09.955 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-24T00:10:09.954+0000 7f5b7b7fe700 1 -- 172.21.15.189:0/927429449 <== mgr.14152 v2:172.21.15.118:6800/2112912943 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7f5b8c0625d0 con 0x7f5b740a2890 2024-03-24T00:10:09.957 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-24T00:10:09.957+0000 7f5b90b0a700 1 -- 172.21.15.189:0/927429449 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7f5b740a2890 msgr2=0x7f5b740a4d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:10:09.957 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-24T00:10:09.957+0000 7f5b90b0a700 1 --2- 172.21.15.189:0/927429449 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7f5b740a2890 0x7f5b740a4d50 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f5b7c006fd0 tx=0x7f5b7c008040 comp rx=0 tx=0).stop 2024-03-24T00:10:09.957 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-24T00:10:09.957+0000 7f5b90b0a700 1 -- 172.21.15.189:0/927429449 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5b8c102cd0 msgr2=0x7f5b8c19f8d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:10:09.957 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-24T00:10:09.957+0000 7f5b90b0a700 1 --2- 172.21.15.189:0/927429449 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5b8c102cd0 0x7f5b8c19f8d0 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f5b8400b970 tx=0x7f5b8400bc80 comp rx=0 tx=0).stop 2024-03-24T00:10:09.958 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-24T00:10:09.957+0000 7f5b90b0a700 1 -- 172.21.15.189:0/927429449 shutdown_connections 2024-03-24T00:10:09.958 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-24T00:10:09.957+0000 7f5b90b0a700 1 --2- 172.21.15.189:0/927429449 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5b8c102cd0 0x7f5b8c19f8d0 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:10:09.958 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-24T00:10:09.957+0000 7f5b90b0a700 1 --2- 172.21.15.189:0/927429449 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7f5b740a2890 0x7f5b740a4d50 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:10:09.958 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-24T00:10:09.957+0000 7f5b90b0a700 1 --2- 172.21.15.189:0/927429449 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f5b8c101130 0x7f5b8c06b450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:10:09.958 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-24T00:10:09.957+0000 7f5b90b0a700 1 --2- 172.21.15.189:0/927429449 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f5b8c102330 0x7f5b8c19d240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:10:09.958 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-24T00:10:09.957+0000 7f5b90b0a700 1 -- 172.21.15.189:0/927429449 >> 172.21.15.189:0/927429449 conn(0x7f5b8c0fc6f0 msgr2=0x7f5b8c105700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:10:09.958 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-24T00:10:09.957+0000 7f5b90b0a700 1 -- 172.21.15.189:0/927429449 shutdown_connections 2024-03-24T00:10:09.958 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-24T00:10:09.957+0000 7f5b90b0a700 1 -- 172.21.15.189:0/927429449 wait complete. 2024-03-24T00:10:10.570 DEBUG:teuthology.orchestra.run.smithi189:osd.7> sudo journalctl -f -n 0 -u ceph-d79a5420-e971-11ee-95cd-87774f69a715@osd.7.service 2024-03-24T00:10:10.573 INFO:tasks.cephadm:Deploying osd.8 on smithi193 with /dev/vg_nvme/lv_2... 2024-03-24T00:10:10.573 DEBUG:teuthology.orchestra.run.smithi193:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d79a5420-e971-11ee-95cd-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_2 2024-03-24T00:10:10.824 INFO:teuthology.orchestra.run.smithi193.stderr:Inferring config /var/lib/ceph/d79a5420-e971-11ee-95cd-87774f69a715/mon.c/config 2024-03-24T00:10:10.945 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:10:10 smithi193 ceph-mon[36878]: pgmap v191: 1 pgs: 1 active+clean; 577 KiB data, 196 MiB used, 626 GiB / 626 GiB avail 2024-03-24T00:10:10.946 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:10:10 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T00:10:10.946 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:10:10 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T00:10:10.946 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:10:10 smithi193 ceph-mon[36878]: from='osd.7 [v2:172.21.15.189:6816/3667611284,v1:172.21.15.189:6817/3667611284]' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-03-24T00:10:10.946 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:10:10 smithi193 ceph-mon[36878]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-03-24T00:10:10.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:10:10 smithi118 ceph-mon[31535]: pgmap v191: 1 pgs: 1 active+clean; 577 KiB data, 196 MiB used, 626 GiB / 626 GiB avail 2024-03-24T00:10:10.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:10:10 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T00:10:10.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:10:10 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T00:10:10.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:10:10 smithi118 ceph-mon[31535]: from='osd.7 [v2:172.21.15.189:6816/3667611284,v1:172.21.15.189:6817/3667611284]' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-03-24T00:10:10.947 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:10:10 smithi118 ceph-mon[31535]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-03-24T00:10:10.953 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:10:10 smithi189 ceph-mon[36187]: pgmap v191: 1 pgs: 1 active+clean; 577 KiB data, 196 MiB used, 626 GiB / 626 GiB avail 2024-03-24T00:10:10.953 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:10:10 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T00:10:10.953 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:10:10 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T00:10:10.953 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:10:10 smithi189 ceph-mon[36187]: from='osd.7 [v2:172.21.15.189:6816/3667611284,v1:172.21.15.189:6817/3667611284]' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-03-24T00:10:10.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:10:10 smithi189 ceph-mon[36187]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]: dispatch 2024-03-24T00:10:11.953 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:10:11 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T00:10:11.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:10:11 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T00:10:11.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:10:11 smithi189 ceph-mon[36187]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2024-03-24T00:10:11.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:10:11 smithi189 ceph-mon[36187]: osdmap e50: 8 total, 7 up, 8 in 2024-03-24T00:10:11.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:10:11 smithi189 ceph-mon[36187]: from='osd.7 [v2:172.21.15.189:6816/3667611284,v1:172.21.15.189:6817/3667611284]' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi189", "root=default"]}]: dispatch 2024-03-24T00:10:11.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:10:11 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-24T00:10:11.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:10:11 smithi189 ceph-mon[36187]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi189", "root=default"]}]: dispatch 2024-03-24T00:10:11.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:10:11 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T00:10:11.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:10:11 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T00:10:11.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:10:11 smithi193 ceph-mon[36878]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2024-03-24T00:10:11.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:10:11 smithi193 ceph-mon[36878]: osdmap e50: 8 total, 7 up, 8 in 2024-03-24T00:10:11.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:10:11 smithi193 ceph-mon[36878]: from='osd.7 [v2:172.21.15.189:6816/3667611284,v1:172.21.15.189:6817/3667611284]' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi189", "root=default"]}]: dispatch 2024-03-24T00:10:11.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:10:11 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-24T00:10:11.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:10:11 smithi193 ceph-mon[36878]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi189", "root=default"]}]: dispatch 2024-03-24T00:10:12.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:10:11 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T00:10:12.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:10:11 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T00:10:12.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:10:11 smithi118 ceph-mon[31535]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["7"]}]': finished 2024-03-24T00:10:12.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:10:11 smithi118 ceph-mon[31535]: osdmap e50: 8 total, 7 up, 8 in 2024-03-24T00:10:12.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:10:11 smithi118 ceph-mon[31535]: from='osd.7 [v2:172.21.15.189:6816/3667611284,v1:172.21.15.189:6817/3667611284]' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi189", "root=default"]}]: dispatch 2024-03-24T00:10:12.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:10:11 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-24T00:10:12.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:10:11 smithi118 ceph-mon[31535]: from='osd.7 ' entity='osd.7' cmd=[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi189", "root=default"]}]: dispatch 2024-03-24T00:10:12.365 INFO:journalctl@ceph.osd.7.smithi189.stdout:Mar 24 00:10:11 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-osd-7[54385]: 2024-03-24T00:10:11.964+0000 7fb77852e700 -1 osd.7 0 waiting for initial osdmap 2024-03-24T00:10:12.365 INFO:journalctl@ceph.osd.7.smithi189.stdout:Mar 24 00:10:11 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-osd-7[54385]: 2024-03-24T00:10:11.972+0000 7fb774321700 -1 osd.7 51 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-24T00:10:12.824 INFO:teuthology.orchestra.run.smithi193.stderr:--> Zapping: /dev/vg_nvme/lv_2 2024-03-24T00:10:12.824 INFO:teuthology.orchestra.run.smithi193.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_2 bs=1M count=10 conv=fsync 2024-03-24T00:10:12.824 INFO:teuthology.orchestra.run.smithi193.stderr: stderr: 10+0 records in 2024-03-24T00:10:12.824 INFO:teuthology.orchestra.run.smithi193.stderr:10+0 records out 2024-03-24T00:10:12.824 INFO:teuthology.orchestra.run.smithi193.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0112436 s, 933 MB/s 2024-03-24T00:10:12.824 INFO:teuthology.orchestra.run.smithi193.stderr:--> Zapping successful for: 2024-03-24T00:10:12.953 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:10:12 smithi189 ceph-mon[36187]: pgmap v193: 1 pgs: 1 active+clean; 577 KiB data, 196 MiB used, 626 GiB / 626 GiB avail 2024-03-24T00:10:12.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:10:12 smithi189 ceph-mon[36187]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi189", "root=default"]}]': finished 2024-03-24T00:10:12.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:10:12 smithi189 ceph-mon[36187]: osdmap e51: 8 total, 7 up, 8 in 2024-03-24T00:10:12.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:10:12 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-24T00:10:12.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:10:12 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-24T00:10:13.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:10:12 smithi118 ceph-mon[31535]: pgmap v193: 1 pgs: 1 active+clean; 577 KiB data, 196 MiB used, 626 GiB / 626 GiB avail 2024-03-24T00:10:13.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:10:12 smithi118 ceph-mon[31535]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi189", "root=default"]}]': finished 2024-03-24T00:10:13.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:10:12 smithi118 ceph-mon[31535]: osdmap e51: 8 total, 7 up, 8 in 2024-03-24T00:10:13.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:10:12 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-24T00:10:13.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:10:12 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-24T00:10:13.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:10:12 smithi193 ceph-mon[36878]: pgmap v193: 1 pgs: 1 active+clean; 577 KiB data, 196 MiB used, 626 GiB / 626 GiB avail 2024-03-24T00:10:13.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:10:12 smithi193 ceph-mon[36878]: from='osd.7 ' entity='osd.7' cmd='[{"prefix": "osd crush create-or-move", "id": 7, "weight":0.0873, "args": ["host=smithi189", "root=default"]}]': finished 2024-03-24T00:10:13.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:10:12 smithi193 ceph-mon[36878]: osdmap e51: 8 total, 7 up, 8 in 2024-03-24T00:10:13.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:10:12 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-24T00:10:13.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:10:12 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-24T00:10:13.499 DEBUG:teuthology.orchestra.run.smithi193:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d79a5420-e971-11ee-95cd-87774f69a715 -- ceph orch daemon add osd smithi193:vg_nvme/lv_2 2024-03-24T00:10:13.746 INFO:teuthology.orchestra.run.smithi193.stderr:Inferring config /var/lib/ceph/d79a5420-e971-11ee-95cd-87774f69a715/mon.c/config 2024-03-24T00:10:14.091 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:10:13 smithi193 ceph-mon[36878]: purged_snaps scrub starts 2024-03-24T00:10:14.091 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:10:13 smithi193 ceph-mon[36878]: purged_snaps scrub ok 2024-03-24T00:10:14.091 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:10:13 smithi193 ceph-mon[36878]: osd.7 [v2:172.21.15.189:6816/3667611284,v1:172.21.15.189:6817/3667611284] boot 2024-03-24T00:10:14.091 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:10:13 smithi193 ceph-mon[36878]: osdmap e52: 8 total, 8 up, 8 in 2024-03-24T00:10:14.091 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:10:13 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-24T00:10:14.091 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:10:13 smithi193 ceph-mon[36878]: pgmap v196: 1 pgs: 1 unknown; 577 KiB data, 223 MiB used, 715 GiB / 715 GiB avail 2024-03-24T00:10:14.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:10:13 smithi189 ceph-mon[36187]: purged_snaps scrub starts 2024-03-24T00:10:14.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:10:13 smithi189 ceph-mon[36187]: purged_snaps scrub ok 2024-03-24T00:10:14.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:10:13 smithi189 ceph-mon[36187]: osd.7 [v2:172.21.15.189:6816/3667611284,v1:172.21.15.189:6817/3667611284] boot 2024-03-24T00:10:14.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:10:13 smithi189 ceph-mon[36187]: osdmap e52: 8 total, 8 up, 8 in 2024-03-24T00:10:14.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:10:13 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-24T00:10:14.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:10:13 smithi189 ceph-mon[36187]: pgmap v196: 1 pgs: 1 unknown; 577 KiB data, 223 MiB used, 715 GiB / 715 GiB avail 2024-03-24T00:10:14.445 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:10:13 smithi118 ceph-mon[31535]: purged_snaps scrub starts 2024-03-24T00:10:14.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:10:13 smithi118 ceph-mon[31535]: purged_snaps scrub ok 2024-03-24T00:10:14.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:10:13 smithi118 ceph-mon[31535]: osd.7 [v2:172.21.15.189:6816/3667611284,v1:172.21.15.189:6817/3667611284] boot 2024-03-24T00:10:14.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:10:13 smithi118 ceph-mon[31535]: osdmap e52: 8 total, 8 up, 8 in 2024-03-24T00:10:14.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:10:13 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 7}]: dispatch 2024-03-24T00:10:14.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:10:13 smithi118 ceph-mon[31535]: pgmap v196: 1 pgs: 1 unknown; 577 KiB data, 223 MiB used, 715 GiB / 715 GiB avail 2024-03-24T00:10:15.071 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:10:14 smithi193 ceph-mon[36878]: Health check failed: Reduced data availability: 1 pg inactive (PG_AVAILABILITY) 2024-03-24T00:10:15.071 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:10:14 smithi193 ceph-mon[36878]: osdmap e53: 8 total, 8 up, 8 in 2024-03-24T00:10:15.071 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:10:14 smithi193 ceph-mon[36878]: Detected new or changed devices on smithi189 2024-03-24T00:10:15.071 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:10:14 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T00:10:15.071 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:10:14 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T00:10:15.071 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:10:14 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-24T00:10:15.071 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:10:14 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-24T00:10:15.071 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:10:14 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-24T00:10:15.071 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:10:14 smithi193 ceph-mon[36878]: Adjusting osd_memory_target on smithi189 to 5693M 2024-03-24T00:10:15.072 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:10:14 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T00:10:15.072 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:10:14 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T00:10:15.072 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:10:14 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T00:10:15.072 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:10:14 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T00:10:15.072 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:10:14 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T00:10:15.072 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:10:14 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T00:10:15.072 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:10:14 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T00:10:15.072 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:10:14 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T00:10:15.302 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-24T00:10:15.300+0000 7faab79e6700 1 -- 172.21.15.193:0/3237314116 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7faab0103d60 msgr2=0x7faab010a610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:10:15.302 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-24T00:10:15.300+0000 7faab79e6700 1 --2- 172.21.15.193:0/3237314116 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7faab0103d60 0x7faab010a610 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7faaa800b0a0 tx=0x7faaa800b3b0 comp rx=0 tx=0).stop 2024-03-24T00:10:15.302 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-24T00:10:15.300+0000 7faab79e6700 1 -- 172.21.15.193:0/3237314116 shutdown_connections 2024-03-24T00:10:15.302 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-24T00:10:15.300+0000 7faab79e6700 1 --2- 172.21.15.193:0/3237314116 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7faab0103d60 0x7faab010a610 unknown :-1 s=CLOSED pgs=35 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:10:15.303 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-24T00:10:15.300+0000 7faab79e6700 1 --2- 172.21.15.193:0/3237314116 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7faab01021c0 0x7faab01025e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:10:15.303 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-24T00:10:15.300+0000 7faab79e6700 1 --2- 172.21.15.193:0/3237314116 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7faab01033c0 0x7faab0103820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:10:15.303 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-24T00:10:15.300+0000 7faab79e6700 1 -- 172.21.15.193:0/3237314116 >> 172.21.15.193:0/3237314116 conn(0x7faab00fd7a0 msgr2=0x7faab00ffbc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:10:15.303 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-24T00:10:15.300+0000 7faab79e6700 1 -- 172.21.15.193:0/3237314116 shutdown_connections 2024-03-24T00:10:15.303 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-24T00:10:15.300+0000 7faab79e6700 1 -- 172.21.15.193:0/3237314116 wait complete. 2024-03-24T00:10:15.303 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-24T00:10:15.300+0000 7faab79e6700 1 Processor -- start 2024-03-24T00:10:15.303 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-24T00:10:15.300+0000 7faab79e6700 1 -- start start 2024-03-24T00:10:15.303 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-24T00:10:15.300+0000 7faab79e6700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7faab01021c0 0x7faab0075f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:10:15.303 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-24T00:10:15.300+0000 7faab79e6700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7faab01033c0 0x7faab0074580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:10:15.304 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-24T00:10:15.300+0000 7faab79e6700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7faab0103d60 0x7faab0074ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:10:15.304 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-24T00:10:15.300+0000 7faab79e6700 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faab010de30 con 0x7faab01033c0 2024-03-24T00:10:15.304 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-24T00:10:15.300+0000 7faab79e6700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faab010dcb0 con 0x7faab0103d60 2024-03-24T00:10:15.304 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-24T00:10:15.300+0000 7faab79e6700 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faab010dfb0 con 0x7faab01021c0 2024-03-24T00:10:15.304 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-24T00:10:15.300+0000 7faab5782700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7faab01021c0 0x7faab0075f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:10:15.304 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-24T00:10:15.301+0000 7faab5782700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7faab01021c0 0x7faab0075f30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.193:40794/0 (socket says 172.21.15.193:40794) 2024-03-24T00:10:15.304 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-24T00:10:15.301+0000 7faab5782700 1 -- 172.21.15.193:0/762465426 learned_addr learned my addr 172.21.15.193:0/762465426 (peer_addr_for_me v2:172.21.15.193:0/0) 2024-03-24T00:10:15.304 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-24T00:10:15.301+0000 7faab5f83700 1 --2- 172.21.15.193:0/762465426 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7faab0103d60 0x7faab0074ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:10:15.304 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-24T00:10:15.301+0000 7faab5782700 1 -- 172.21.15.193:0/762465426 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7faab0103d60 msgr2=0x7faab0074ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:10:15.305 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-24T00:10:15.301+0000 7faab5782700 1 --2- 172.21.15.193:0/762465426 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7faab0103d60 0x7faab0074ac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:10:15.305 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-24T00:10:15.301+0000 7faab5782700 1 -- 172.21.15.193:0/762465426 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7faab01033c0 msgr2=0x7faab0074580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:10:15.305 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-24T00:10:15.301+0000 7faab5782700 1 --2- 172.21.15.193:0/762465426 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7faab01033c0 0x7faab0074580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:10:15.305 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-24T00:10:15.301+0000 7faab5782700 1 -- 172.21.15.193:0/762465426 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faab0075370 con 0x7faab01021c0 2024-03-24T00:10:15.305 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-24T00:10:15.301+0000 7faab5f83700 1 --2- 172.21.15.193:0/762465426 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7faab0103d60 0x7faab0074ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T00:10:15.305 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-24T00:10:15.301+0000 7faab5782700 1 --2- 172.21.15.193:0/762465426 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7faab01021c0 0x7faab0075f30 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7faaac00b500 tx=0x7faaac00b8c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:10:15.305 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-24T00:10:15.301+0000 7faaa67fc700 1 -- 172.21.15.193:0/762465426 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faaac015020 con 0x7faab01021c0 2024-03-24T00:10:15.305 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-24T00:10:15.301+0000 7faab79e6700 1 -- 172.21.15.193:0/762465426 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7faab01a24e0 con 0x7faab01021c0 2024-03-24T00:10:15.305 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-24T00:10:15.301+0000 7faaa67fc700 1 -- 172.21.15.193:0/762465426 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7faaac011d10 con 0x7faab01021c0 2024-03-24T00:10:15.306 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-24T00:10:15.301+0000 7faaa67fc700 1 -- 172.21.15.193:0/762465426 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7faaac00f2e0 con 0x7faab01021c0 2024-03-24T00:10:15.306 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-24T00:10:15.301+0000 7faab79e6700 1 -- 172.21.15.193:0/762465426 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7faab01a2a70 con 0x7faab01021c0 2024-03-24T00:10:15.306 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-24T00:10:15.302+0000 7faab79e6700 1 -- 172.21.15.193:0/762465426 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7faab0068e50 con 0x7faab01021c0 2024-03-24T00:10:15.307 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-24T00:10:15.304+0000 7faaa67fc700 1 -- 172.21.15.193:0/762465426 <== mon.1 v2:172.21.15.193:3300/0 4 ==== mgrmap(e 16) v1 ==== 137291+0+0 (secure 0 0 0) 0x7faaac018020 con 0x7faab01021c0 2024-03-24T00:10:15.307 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-24T00:10:15.304+0000 7faaa67fc700 1 --2- 172.21.15.193:0/762465426 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7faa9c0a2890 0x7faa9c0a4d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:10:15.307 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-24T00:10:15.305+0000 7faaa67fc700 1 -- 172.21.15.193:0/762465426 <== mon.1 v2:172.21.15.193:3300/0 5 ==== osd_map(54..54 src has 1..54) v4 ==== 5426+0+0 (secure 0 0 0) 0x7faaac0cdbc0 con 0x7faab01021c0 2024-03-24T00:10:15.307 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-24T00:10:15.305+0000 7faab4f81700 1 --2- 172.21.15.193:0/762465426 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7faa9c0a2890 0x7faa9c0a4d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:10:15.307 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-24T00:10:15.306+0000 7faab4f81700 1 --2- 172.21.15.193:0/762465426 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7faa9c0a2890 0x7faa9c0a4d50 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7faaa0005fd0 tx=0x7faaa0005d00 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:10:15.308 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-24T00:10:15.308+0000 7faaa67fc700 1 -- 172.21.15.193:0/762465426 <== mon.1 v2:172.21.15.193:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7faaac01f020 con 0x7faab01021c0 2024-03-24T00:10:15.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:10:14 smithi118 ceph-mon[31535]: Health check failed: Reduced data availability: 1 pg inactive (PG_AVAILABILITY) 2024-03-24T00:10:15.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:10:14 smithi118 ceph-mon[31535]: osdmap e53: 8 total, 8 up, 8 in 2024-03-24T00:10:15.447 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:10:14 smithi118 ceph-mon[31535]: Detected new or changed devices on smithi189 2024-03-24T00:10:15.447 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:10:14 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T00:10:15.447 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:10:14 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T00:10:15.447 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:10:14 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-24T00:10:15.447 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:10:14 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-24T00:10:15.447 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:10:14 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-24T00:10:15.447 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:10:14 smithi118 ceph-mon[31535]: Adjusting osd_memory_target on smithi189 to 5693M 2024-03-24T00:10:15.447 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:10:14 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T00:10:15.447 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:10:14 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T00:10:15.447 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:10:14 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T00:10:15.447 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:10:14 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T00:10:15.447 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:10:14 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T00:10:15.447 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:10:14 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T00:10:15.447 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:10:14 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T00:10:15.448 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:10:14 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T00:10:15.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:10:14 smithi189 ceph-mon[36187]: Health check failed: Reduced data availability: 1 pg inactive (PG_AVAILABILITY) 2024-03-24T00:10:15.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:10:14 smithi189 ceph-mon[36187]: osdmap e53: 8 total, 8 up, 8 in 2024-03-24T00:10:15.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:10:14 smithi189 ceph-mon[36187]: Detected new or changed devices on smithi189 2024-03-24T00:10:15.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:10:14 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T00:10:15.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:10:14 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T00:10:15.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:10:14 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-24T00:10:15.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:10:14 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-24T00:10:15.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:10:14 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-24T00:10:15.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:10:14 smithi189 ceph-mon[36187]: Adjusting osd_memory_target on smithi189 to 5693M 2024-03-24T00:10:15.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:10:14 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T00:10:15.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:10:14 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T00:10:15.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:10:14 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T00:10:15.455 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:10:14 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T00:10:15.455 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:10:14 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T00:10:15.455 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:10:14 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T00:10:15.455 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:10:14 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T00:10:15.455 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:10:14 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T00:10:15.512 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-24T00:10:15.512+0000 7faab79e6700 1 -- 172.21.15.193:0/762465426 --> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi193:vg_nvme/lv_2", "target": ["mon-mgr", ""]}) v1 -- 0x7faab00625d0 con 0x7faa9c0a2890 2024-03-24T00:10:16.380 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:10:15 smithi193 ceph-mon[36878]: osdmap e54: 8 total, 8 up, 8 in 2024-03-24T00:10:16.381 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:10:15 smithi193 ceph-mon[36878]: from='client.24307 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi193:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T00:10:16.381 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:10:15 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-24T00:10:16.381 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:10:15 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-24T00:10:16.381 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:10:15 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T00:10:16.381 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:10:15 smithi193 ceph-mon[36878]: pgmap v199: 1 pgs: 1 unknown; 577 KiB data, 223 MiB used, 715 GiB / 715 GiB avail 2024-03-24T00:10:16.381 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:10:15 smithi193 ceph-mon[36878]: osdmap e55: 8 total, 8 up, 8 in 2024-03-24T00:10:16.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:10:15 smithi118 ceph-mon[31535]: osdmap e54: 8 total, 8 up, 8 in 2024-03-24T00:10:16.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:10:15 smithi118 ceph-mon[31535]: from='client.24307 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi193:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T00:10:16.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:10:15 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-24T00:10:16.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:10:15 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-24T00:10:16.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:10:15 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T00:10:16.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:10:15 smithi118 ceph-mon[31535]: pgmap v199: 1 pgs: 1 unknown; 577 KiB data, 223 MiB used, 715 GiB / 715 GiB avail 2024-03-24T00:10:16.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:10:15 smithi118 ceph-mon[31535]: osdmap e55: 8 total, 8 up, 8 in 2024-03-24T00:10:16.453 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:10:15 smithi189 ceph-mon[36187]: osdmap e54: 8 total, 8 up, 8 in 2024-03-24T00:10:16.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:10:15 smithi189 ceph-mon[36187]: from='client.24307 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi193:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T00:10:16.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:10:15 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-24T00:10:16.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:10:15 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-24T00:10:16.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:10:15 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T00:10:16.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:10:15 smithi189 ceph-mon[36187]: pgmap v199: 1 pgs: 1 unknown; 577 KiB data, 223 MiB used, 715 GiB / 715 GiB avail 2024-03-24T00:10:16.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:10:15 smithi189 ceph-mon[36187]: osdmap e55: 8 total, 8 up, 8 in 2024-03-24T00:10:18.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:10:17 smithi118 ceph-mon[31535]: osdmap e56: 8 total, 8 up, 8 in 2024-03-24T00:10:18.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:10:17 smithi118 ceph-mon[31535]: pgmap v202: 1 pgs: 1 unknown; 577 KiB data, 223 MiB used, 715 GiB / 715 GiB avail 2024-03-24T00:10:18.203 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:10:17 smithi189 ceph-mon[36187]: osdmap e56: 8 total, 8 up, 8 in 2024-03-24T00:10:18.203 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:10:17 smithi189 ceph-mon[36187]: pgmap v202: 1 pgs: 1 unknown; 577 KiB data, 223 MiB used, 715 GiB / 715 GiB avail 2024-03-24T00:10:18.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:10:17 smithi193 ceph-mon[36878]: osdmap e56: 8 total, 8 up, 8 in 2024-03-24T00:10:18.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:10:17 smithi193 ceph-mon[36878]: pgmap v202: 1 pgs: 1 unknown; 577 KiB data, 223 MiB used, 715 GiB / 715 GiB avail 2024-03-24T00:10:19.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:10:18 smithi118 ceph-mon[31535]: from='client.? 172.21.15.193:0/3305246179' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "6462c2b5-e4b3-4b0b-bcfa-4de2c89e93e0"}]: dispatch 2024-03-24T00:10:19.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:10:18 smithi118 ceph-mon[31535]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "6462c2b5-e4b3-4b0b-bcfa-4de2c89e93e0"}]: dispatch 2024-03-24T00:10:19.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:10:18 smithi118 ceph-mon[31535]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "6462c2b5-e4b3-4b0b-bcfa-4de2c89e93e0"}]': finished 2024-03-24T00:10:19.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:10:18 smithi118 ceph-mon[31535]: osdmap e57: 9 total, 8 up, 9 in 2024-03-24T00:10:19.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:10:18 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-03-24T00:10:19.203 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:10:18 smithi189 ceph-mon[36187]: from='client.? 172.21.15.193:0/3305246179' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "6462c2b5-e4b3-4b0b-bcfa-4de2c89e93e0"}]: dispatch 2024-03-24T00:10:19.203 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:10:18 smithi189 ceph-mon[36187]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "6462c2b5-e4b3-4b0b-bcfa-4de2c89e93e0"}]: dispatch 2024-03-24T00:10:19.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:10:18 smithi189 ceph-mon[36187]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "6462c2b5-e4b3-4b0b-bcfa-4de2c89e93e0"}]': finished 2024-03-24T00:10:19.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:10:18 smithi189 ceph-mon[36187]: osdmap e57: 9 total, 8 up, 9 in 2024-03-24T00:10:19.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:10:18 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-03-24T00:10:19.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:10:18 smithi193 ceph-mon[36878]: from='client.? 172.21.15.193:0/3305246179' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "6462c2b5-e4b3-4b0b-bcfa-4de2c89e93e0"}]: dispatch 2024-03-24T00:10:19.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:10:18 smithi193 ceph-mon[36878]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "6462c2b5-e4b3-4b0b-bcfa-4de2c89e93e0"}]: dispatch 2024-03-24T00:10:19.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:10:18 smithi193 ceph-mon[36878]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "6462c2b5-e4b3-4b0b-bcfa-4de2c89e93e0"}]': finished 2024-03-24T00:10:19.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:10:18 smithi193 ceph-mon[36878]: osdmap e57: 9 total, 8 up, 9 in 2024-03-24T00:10:19.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:10:18 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-03-24T00:10:20.112 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:10:19 smithi193 ceph-mon[36878]: from='client.? 172.21.15.193:0/2013022869' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-24T00:10:20.112 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:10:19 smithi193 ceph-mon[36878]: pgmap v204: 1 pgs: 1 active+clean; 577 KiB data, 224 MiB used, 715 GiB / 715 GiB avail 2024-03-24T00:10:20.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:10:19 smithi118 ceph-mon[31535]: from='client.? 172.21.15.193:0/2013022869' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-24T00:10:20.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:10:19 smithi118 ceph-mon[31535]: pgmap v204: 1 pgs: 1 active+clean; 577 KiB data, 224 MiB used, 715 GiB / 715 GiB avail 2024-03-24T00:10:20.203 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:10:19 smithi189 ceph-mon[36187]: from='client.? 172.21.15.193:0/2013022869' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-24T00:10:20.203 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:10:19 smithi189 ceph-mon[36187]: pgmap v204: 1 pgs: 1 active+clean; 577 KiB data, 224 MiB used, 715 GiB / 715 GiB avail 2024-03-24T00:10:21.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:10:20 smithi118 ceph-mon[31535]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg inactive) 2024-03-24T00:10:21.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:10:20 smithi118 ceph-mon[31535]: Cluster is now healthy 2024-03-24T00:10:21.203 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:10:20 smithi189 ceph-mon[36187]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg inactive) 2024-03-24T00:10:21.203 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:10:20 smithi189 ceph-mon[36187]: Cluster is now healthy 2024-03-24T00:10:21.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:10:20 smithi193 ceph-mon[36878]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg inactive) 2024-03-24T00:10:21.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:10:20 smithi193 ceph-mon[36878]: Cluster is now healthy 2024-03-24T00:10:22.119 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:10:21 smithi193 ceph-mon[36878]: pgmap v205: 1 pgs: 1 active+clean; 577 KiB data, 224 MiB used, 715 GiB / 715 GiB avail 2024-03-24T00:10:22.195 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:10:21 smithi118 ceph-mon[31535]: pgmap v205: 1 pgs: 1 active+clean; 577 KiB data, 224 MiB used, 715 GiB / 715 GiB avail 2024-03-24T00:10:22.203 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:10:21 smithi189 ceph-mon[36187]: pgmap v205: 1 pgs: 1 active+clean; 577 KiB data, 224 MiB used, 715 GiB / 715 GiB avail 2024-03-24T00:10:24.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:10:24 smithi118 ceph-mon[31535]: pgmap v206: 1 pgs: 1 active+clean; 577 KiB data, 224 MiB used, 715 GiB / 715 GiB avail 2024-03-24T00:10:24.953 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:10:24 smithi189 ceph-mon[36187]: pgmap v206: 1 pgs: 1 active+clean; 577 KiB data, 224 MiB used, 715 GiB / 715 GiB avail 2024-03-24T00:10:24.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:10:24 smithi193 ceph-mon[36878]: pgmap v206: 1 pgs: 1 active+clean; 577 KiB data, 224 MiB used, 715 GiB / 715 GiB avail 2024-03-24T00:10:26.705 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:10:26 smithi193 ceph-mon[36878]: pgmap v207: 1 pgs: 1 active+clean; 577 KiB data, 224 MiB used, 715 GiB / 715 GiB avail 2024-03-24T00:10:26.706 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:10:26 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.8"}]: dispatch 2024-03-24T00:10:26.706 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:10:26 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T00:10:26.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:10:26 smithi118 ceph-mon[31535]: pgmap v207: 1 pgs: 1 active+clean; 577 KiB data, 224 MiB used, 715 GiB / 715 GiB avail 2024-03-24T00:10:26.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:10:26 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.8"}]: dispatch 2024-03-24T00:10:26.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:10:26 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T00:10:26.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:10:26 smithi189 ceph-mon[36187]: pgmap v207: 1 pgs: 1 active+clean; 577 KiB data, 224 MiB used, 715 GiB / 715 GiB avail 2024-03-24T00:10:26.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:10:26 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.8"}]: dispatch 2024-03-24T00:10:26.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:10:26 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T00:10:27.939 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:10:27 smithi193 ceph-mon[36878]: Deploying daemon osd.8 on smithi193 2024-03-24T00:10:27.945 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:10:27 smithi118 ceph-mon[31535]: Deploying daemon osd.8 on smithi193 2024-03-24T00:10:27.953 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:10:27 smithi189 ceph-mon[36187]: Deploying daemon osd.8 on smithi193 2024-03-24T00:10:28.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:10:28 smithi118 ceph-mon[31535]: pgmap v208: 1 pgs: 1 active+clean; 577 KiB data, 224 MiB used, 715 GiB / 715 GiB avail 2024-03-24T00:10:28.953 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:10:28 smithi189 ceph-mon[36187]: pgmap v208: 1 pgs: 1 active+clean; 577 KiB data, 224 MiB used, 715 GiB / 715 GiB avail 2024-03-24T00:10:28.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:10:28 smithi193 ceph-mon[36878]: pgmap v208: 1 pgs: 1 active+clean; 577 KiB data, 224 MiB used, 715 GiB / 715 GiB avail 2024-03-24T00:10:30.767 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:10:30 smithi193 ceph-mon[36878]: pgmap v209: 1 pgs: 1 active+clean; 577 KiB data, 224 MiB used, 715 GiB / 715 GiB avail 2024-03-24T00:10:30.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:10:30 smithi118 ceph-mon[31535]: pgmap v209: 1 pgs: 1 active+clean; 577 KiB data, 224 MiB used, 715 GiB / 715 GiB avail 2024-03-24T00:10:30.953 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:10:30 smithi189 ceph-mon[36187]: pgmap v209: 1 pgs: 1 active+clean; 577 KiB data, 224 MiB used, 715 GiB / 715 GiB avail 2024-03-24T00:10:32.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:10:32 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T00:10:32.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:10:32 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T00:10:32.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:10:32 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T00:10:32.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:10:32 smithi118 ceph-mon[31535]: pgmap v210: 1 pgs: 1 active+clean; 577 KiB data, 224 MiB used, 715 GiB / 715 GiB avail 2024-03-24T00:10:32.703 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:10:32 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T00:10:32.703 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:10:32 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T00:10:32.703 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:10:32 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T00:10:32.703 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:10:32 smithi189 ceph-mon[36187]: pgmap v210: 1 pgs: 1 active+clean; 577 KiB data, 224 MiB used, 715 GiB / 715 GiB avail 2024-03-24T00:10:32.710 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:10:32 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T00:10:32.710 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:10:32 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T00:10:32.710 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:10:32 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T00:10:32.711 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:10:32 smithi193 ceph-mon[36878]: pgmap v210: 1 pgs: 1 active+clean; 577 KiB data, 224 MiB used, 715 GiB / 715 GiB avail 2024-03-24T00:10:34.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:10:34 smithi118 ceph-mon[31535]: pgmap v211: 1 pgs: 1 active+clean; 577 KiB data, 224 MiB used, 715 GiB / 715 GiB avail 2024-03-24T00:10:34.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:10:34 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T00:10:34.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:10:34 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T00:10:34.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:10:34 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T00:10:34.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:10:34 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T00:10:34.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:10:34 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T00:10:34.953 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:10:34 smithi189 ceph-mon[36187]: pgmap v211: 1 pgs: 1 active+clean; 577 KiB data, 224 MiB used, 715 GiB / 715 GiB avail 2024-03-24T00:10:34.953 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:10:34 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T00:10:34.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:10:34 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T00:10:34.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:10:34 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T00:10:34.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:10:34 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T00:10:34.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:10:34 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T00:10:34.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:10:34 smithi193 ceph-mon[36878]: pgmap v211: 1 pgs: 1 active+clean; 577 KiB data, 224 MiB used, 715 GiB / 715 GiB avail 2024-03-24T00:10:34.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:10:34 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T00:10:34.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:10:34 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T00:10:34.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:10:34 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T00:10:34.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:10:34 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T00:10:34.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:10:34 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T00:10:35.351 INFO:teuthology.orchestra.run.smithi193.stdout:Created osd(s) 8 on host 'smithi193' 2024-03-24T00:10:35.351 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-24T00:10:35.351+0000 7faaa67fc700 1 -- 172.21.15.193:0/762465426 <== mgr.14152 v2:172.21.15.118:6800/2112912943 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7faab00625d0 con 0x7faa9c0a2890 2024-03-24T00:10:35.354 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-24T00:10:35.353+0000 7faab79e6700 1 -- 172.21.15.193:0/762465426 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7faa9c0a2890 msgr2=0x7faa9c0a4d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:10:35.354 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-24T00:10:35.353+0000 7faab79e6700 1 --2- 172.21.15.193:0/762465426 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7faa9c0a2890 0x7faa9c0a4d50 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7faaa0005fd0 tx=0x7faaa0005d00 comp rx=0 tx=0).stop 2024-03-24T00:10:35.354 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-24T00:10:35.353+0000 7faab79e6700 1 -- 172.21.15.193:0/762465426 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7faab01021c0 msgr2=0x7faab0075f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:10:35.354 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-24T00:10:35.353+0000 7faab79e6700 1 --2- 172.21.15.193:0/762465426 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7faab01021c0 0x7faab0075f30 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7faaac00b500 tx=0x7faaac00b8c0 comp rx=0 tx=0).stop 2024-03-24T00:10:35.354 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-24T00:10:35.353+0000 7faab79e6700 1 -- 172.21.15.193:0/762465426 shutdown_connections 2024-03-24T00:10:35.354 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-24T00:10:35.353+0000 7faab79e6700 1 --2- 172.21.15.193:0/762465426 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7faab0103d60 0x7faab0074ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:10:35.354 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-24T00:10:35.353+0000 7faab79e6700 1 --2- 172.21.15.193:0/762465426 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7faa9c0a2890 0x7faa9c0a4d50 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:10:35.354 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-24T00:10:35.353+0000 7faab79e6700 1 --2- 172.21.15.193:0/762465426 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7faab01021c0 0x7faab0075f30 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:10:35.354 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-24T00:10:35.353+0000 7faab79e6700 1 --2- 172.21.15.193:0/762465426 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7faab01033c0 0x7faab0074580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:10:35.354 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-24T00:10:35.353+0000 7faab79e6700 1 -- 172.21.15.193:0/762465426 >> 172.21.15.193:0/762465426 conn(0x7faab00fd7a0 msgr2=0x7faab0106660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:10:35.354 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-24T00:10:35.354+0000 7faab79e6700 1 -- 172.21.15.193:0/762465426 shutdown_connections 2024-03-24T00:10:35.354 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-24T00:10:35.354+0000 7faab79e6700 1 -- 172.21.15.193:0/762465426 wait complete. 2024-03-24T00:10:35.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:10:35 smithi118 ceph-mon[31535]: from='osd.8 [v2:172.21.15.193:6816/3853800910,v1:172.21.15.193:6817/3853800910]' entity='osd.8' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]: dispatch 2024-03-24T00:10:35.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:10:35 smithi118 ceph-mon[31535]: from='osd.8 ' entity='osd.8' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]: dispatch 2024-03-24T00:10:35.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:10:35 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T00:10:35.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:10:35 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T00:10:35.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:10:35 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T00:10:35.953 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:10:35 smithi189 ceph-mon[36187]: from='osd.8 [v2:172.21.15.193:6816/3853800910,v1:172.21.15.193:6817/3853800910]' entity='osd.8' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]: dispatch 2024-03-24T00:10:35.953 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:10:35 smithi189 ceph-mon[36187]: from='osd.8 ' entity='osd.8' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]: dispatch 2024-03-24T00:10:35.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:10:35 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T00:10:35.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:10:35 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T00:10:35.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:10:35 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T00:10:35.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:10:35 smithi193 ceph-mon[36878]: from='osd.8 [v2:172.21.15.193:6816/3853800910,v1:172.21.15.193:6817/3853800910]' entity='osd.8' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]: dispatch 2024-03-24T00:10:35.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:10:35 smithi193 ceph-mon[36878]: from='osd.8 ' entity='osd.8' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]: dispatch 2024-03-24T00:10:35.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:10:35 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T00:10:35.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:10:35 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T00:10:35.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:10:35 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T00:10:36.003 DEBUG:teuthology.orchestra.run.smithi193:osd.8> sudo journalctl -f -n 0 -u ceph-d79a5420-e971-11ee-95cd-87774f69a715@osd.8.service 2024-03-24T00:10:36.006 INFO:tasks.cephadm:Deploying osd.9 on smithi118 with /dev/vg_nvme/lv_1... 2024-03-24T00:10:36.006 DEBUG:teuthology.orchestra.run.smithi118:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d79a5420-e971-11ee-95cd-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_1 2024-03-24T00:10:36.451 INFO:teuthology.orchestra.run.smithi118.stderr:Inferring config /var/lib/ceph/d79a5420-e971-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T00:10:36.863 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:10:36 smithi118 ceph-mon[31535]: pgmap v212: 1 pgs: 1 active+clean; 577 KiB data, 224 MiB used, 715 GiB / 715 GiB avail 2024-03-24T00:10:36.863 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:10:36 smithi118 ceph-mon[31535]: from='osd.8 ' entity='osd.8' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]': finished 2024-03-24T00:10:36.863 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:10:36 smithi118 ceph-mon[31535]: from='osd.8 [v2:172.21.15.193:6816/3853800910,v1:172.21.15.193:6817/3853800910]' entity='osd.8' cmd=[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi193", "root=default"]}]: dispatch 2024-03-24T00:10:36.863 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:10:36 smithi118 ceph-mon[31535]: osdmap e58: 9 total, 8 up, 9 in 2024-03-24T00:10:36.863 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:10:36 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-03-24T00:10:36.863 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:10:36 smithi118 ceph-mon[31535]: from='osd.8 ' entity='osd.8' cmd=[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi193", "root=default"]}]: dispatch 2024-03-24T00:10:36.902 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:10:36 smithi193 ceph-mon[36878]: pgmap v212: 1 pgs: 1 active+clean; 577 KiB data, 224 MiB used, 715 GiB / 715 GiB avail 2024-03-24T00:10:36.902 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:10:36 smithi193 ceph-mon[36878]: from='osd.8 ' entity='osd.8' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]': finished 2024-03-24T00:10:36.902 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:10:36 smithi193 ceph-mon[36878]: from='osd.8 [v2:172.21.15.193:6816/3853800910,v1:172.21.15.193:6817/3853800910]' entity='osd.8' cmd=[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi193", "root=default"]}]: dispatch 2024-03-24T00:10:36.902 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:10:36 smithi193 ceph-mon[36878]: osdmap e58: 9 total, 8 up, 9 in 2024-03-24T00:10:36.902 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:10:36 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-03-24T00:10:36.902 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:10:36 smithi193 ceph-mon[36878]: from='osd.8 ' entity='osd.8' cmd=[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi193", "root=default"]}]: dispatch 2024-03-24T00:10:36.953 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:10:36 smithi189 ceph-mon[36187]: pgmap v212: 1 pgs: 1 active+clean; 577 KiB data, 224 MiB used, 715 GiB / 715 GiB avail 2024-03-24T00:10:36.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:10:36 smithi189 ceph-mon[36187]: from='osd.8 ' entity='osd.8' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["8"]}]': finished 2024-03-24T00:10:36.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:10:36 smithi189 ceph-mon[36187]: from='osd.8 [v2:172.21.15.193:6816/3853800910,v1:172.21.15.193:6817/3853800910]' entity='osd.8' cmd=[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi193", "root=default"]}]: dispatch 2024-03-24T00:10:36.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:10:36 smithi189 ceph-mon[36187]: osdmap e58: 9 total, 8 up, 9 in 2024-03-24T00:10:36.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:10:36 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-03-24T00:10:36.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:10:36 smithi189 ceph-mon[36187]: from='osd.8 ' entity='osd.8' cmd=[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi193", "root=default"]}]: dispatch 2024-03-24T00:10:37.210 INFO:journalctl@ceph.osd.8.smithi193.stdout:Mar 24 00:10:37 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-osd-8[55794]: 2024-03-24T00:10:37.081+0000 7f6e923a7700 -1 osd.8 0 waiting for initial osdmap 2024-03-24T00:10:37.211 INFO:journalctl@ceph.osd.8.smithi193.stdout:Mar 24 00:10:37 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-osd-8[55794]: 2024-03-24T00:10:37.092+0000 7f6e8a192700 -1 osd.8 59 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-24T00:10:37.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:10:37 smithi118 ceph-mon[31535]: from='osd.8 ' entity='osd.8' cmd='[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi193", "root=default"]}]': finished 2024-03-24T00:10:37.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:10:37 smithi118 ceph-mon[31535]: osdmap e59: 9 total, 8 up, 9 in 2024-03-24T00:10:37.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:10:37 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-03-24T00:10:37.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:10:37 smithi118 ceph-mon[31535]: from='osd.8 ' entity='osd.8' 2024-03-24T00:10:37.953 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:10:37 smithi189 ceph-mon[36187]: from='osd.8 ' entity='osd.8' cmd='[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi193", "root=default"]}]': finished 2024-03-24T00:10:37.953 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:10:37 smithi189 ceph-mon[36187]: osdmap e59: 9 total, 8 up, 9 in 2024-03-24T00:10:37.953 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:10:37 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-03-24T00:10:37.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:10:37 smithi189 ceph-mon[36187]: from='osd.8 ' entity='osd.8' 2024-03-24T00:10:37.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:10:37 smithi193 ceph-mon[36878]: from='osd.8 ' entity='osd.8' cmd='[{"prefix": "osd crush create-or-move", "id": 8, "weight":0.0873, "args": ["host=smithi193", "root=default"]}]': finished 2024-03-24T00:10:37.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:10:37 smithi193 ceph-mon[36878]: osdmap e59: 9 total, 8 up, 9 in 2024-03-24T00:10:37.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:10:37 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-03-24T00:10:37.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:10:37 smithi193 ceph-mon[36878]: from='osd.8 ' entity='osd.8' 2024-03-24T00:10:38.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:10:38 smithi118 ceph-mon[31535]: purged_snaps scrub starts 2024-03-24T00:10:38.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:10:38 smithi118 ceph-mon[31535]: purged_snaps scrub ok 2024-03-24T00:10:38.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:10:38 smithi118 ceph-mon[31535]: pgmap v215: 1 pgs: 1 active+clean; 577 KiB data, 224 MiB used, 715 GiB / 715 GiB avail 2024-03-24T00:10:38.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:10:38 smithi118 ceph-mon[31535]: osd.8 [v2:172.21.15.193:6816/3853800910,v1:172.21.15.193:6817/3853800910] boot 2024-03-24T00:10:38.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:10:38 smithi118 ceph-mon[31535]: osdmap e60: 9 total, 9 up, 9 in 2024-03-24T00:10:38.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:10:38 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-03-24T00:10:38.953 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:10:38 smithi189 ceph-mon[36187]: purged_snaps scrub starts 2024-03-24T00:10:38.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:10:38 smithi189 ceph-mon[36187]: purged_snaps scrub ok 2024-03-24T00:10:38.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:10:38 smithi189 ceph-mon[36187]: pgmap v215: 1 pgs: 1 active+clean; 577 KiB data, 224 MiB used, 715 GiB / 715 GiB avail 2024-03-24T00:10:38.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:10:38 smithi189 ceph-mon[36187]: osd.8 [v2:172.21.15.193:6816/3853800910,v1:172.21.15.193:6817/3853800910] boot 2024-03-24T00:10:38.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:10:38 smithi189 ceph-mon[36187]: osdmap e60: 9 total, 9 up, 9 in 2024-03-24T00:10:38.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:10:38 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-03-24T00:10:38.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:10:38 smithi193 ceph-mon[36878]: purged_snaps scrub starts 2024-03-24T00:10:38.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:10:38 smithi193 ceph-mon[36878]: purged_snaps scrub ok 2024-03-24T00:10:38.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:10:38 smithi193 ceph-mon[36878]: pgmap v215: 1 pgs: 1 active+clean; 577 KiB data, 224 MiB used, 715 GiB / 715 GiB avail 2024-03-24T00:10:38.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:10:38 smithi193 ceph-mon[36878]: osd.8 [v2:172.21.15.193:6816/3853800910,v1:172.21.15.193:6817/3853800910] boot 2024-03-24T00:10:38.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:10:38 smithi193 ceph-mon[36878]: osdmap e60: 9 total, 9 up, 9 in 2024-03-24T00:10:38.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:10:38 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 8}]: dispatch 2024-03-24T00:10:39.399 INFO:teuthology.orchestra.run.smithi118.stderr:--> Zapping: /dev/vg_nvme/lv_1 2024-03-24T00:10:39.399 INFO:teuthology.orchestra.run.smithi118.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_1 bs=1M count=10 conv=fsync 2024-03-24T00:10:39.399 INFO:teuthology.orchestra.run.smithi118.stderr: stderr: 10+0 records in 2024-03-24T00:10:39.399 INFO:teuthology.orchestra.run.smithi118.stderr:10+0 records out 2024-03-24T00:10:39.399 INFO:teuthology.orchestra.run.smithi118.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0321313 s, 326 MB/s 2024-03-24T00:10:39.399 INFO:teuthology.orchestra.run.smithi118.stderr:--> Zapping successful for: 2024-03-24T00:10:39.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:10:39 smithi118 ceph-mon[31535]: osdmap e61: 9 total, 9 up, 9 in 2024-03-24T00:10:39.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:10:39 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T00:10:39.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:10:39 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T00:10:39.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:10:39 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-24T00:10:39.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:10:39 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-24T00:10:39.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:10:39 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-03-24T00:10:39.947 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:10:39 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T00:10:39.947 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:10:39 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T00:10:39.947 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:10:39 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T00:10:39.947 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:10:39 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T00:10:39.953 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:10:39 smithi189 ceph-mon[36187]: osdmap e61: 9 total, 9 up, 9 in 2024-03-24T00:10:39.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:10:39 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T00:10:39.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:10:39 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T00:10:39.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:10:39 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-24T00:10:39.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:10:39 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-24T00:10:39.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:10:39 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-03-24T00:10:39.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:10:39 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T00:10:39.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:10:39 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T00:10:39.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:10:39 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T00:10:39.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:10:39 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T00:10:39.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:10:39 smithi193 ceph-mon[36878]: osdmap e61: 9 total, 9 up, 9 in 2024-03-24T00:10:39.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:10:39 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T00:10:39.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:10:39 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T00:10:39.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:10:39 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-24T00:10:39.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:10:39 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-24T00:10:39.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:10:39 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-03-24T00:10:39.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:10:39 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T00:10:39.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:10:39 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T00:10:39.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:10:39 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T00:10:39.962 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:10:39 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T00:10:40.356 DEBUG:teuthology.orchestra.run.smithi118:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d79a5420-e971-11ee-95cd-87774f69a715 -- ceph orch daemon add osd smithi118:vg_nvme/lv_1 2024-03-24T00:10:40.793 INFO:teuthology.orchestra.run.smithi118.stderr:Inferring config /var/lib/ceph/d79a5420-e971-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T00:10:40.942 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:10:40 smithi118 ceph-mon[31535]: Detected new or changed devices on smithi193 2024-03-24T00:10:40.943 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:10:40 smithi118 ceph-mon[31535]: Adjusting osd_memory_target on smithi193 to 5693M 2024-03-24T00:10:40.943 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:10:40 smithi118 ceph-mon[31535]: pgmap v218: 1 pgs: 1 remapped+peering; 577 KiB data, 255 MiB used, 804 GiB / 805 GiB avail 2024-03-24T00:10:40.943 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:10:40 smithi118 ceph-mon[31535]: osdmap e62: 9 total, 9 up, 9 in 2024-03-24T00:10:40.953 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:10:40 smithi189 ceph-mon[36187]: Detected new or changed devices on smithi193 2024-03-24T00:10:40.953 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:10:40 smithi189 ceph-mon[36187]: Adjusting osd_memory_target on smithi193 to 5693M 2024-03-24T00:10:40.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:10:40 smithi189 ceph-mon[36187]: pgmap v218: 1 pgs: 1 remapped+peering; 577 KiB data, 255 MiB used, 804 GiB / 805 GiB avail 2024-03-24T00:10:40.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:10:40 smithi189 ceph-mon[36187]: osdmap e62: 9 total, 9 up, 9 in 2024-03-24T00:10:40.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:10:40 smithi193 ceph-mon[36878]: Detected new or changed devices on smithi193 2024-03-24T00:10:40.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:10:40 smithi193 ceph-mon[36878]: Adjusting osd_memory_target on smithi193 to 5693M 2024-03-24T00:10:40.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:10:40 smithi193 ceph-mon[36878]: pgmap v218: 1 pgs: 1 remapped+peering; 577 KiB data, 255 MiB used, 804 GiB / 805 GiB avail 2024-03-24T00:10:40.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:10:40 smithi193 ceph-mon[36878]: osdmap e62: 9 total, 9 up, 9 in 2024-03-24T00:10:41.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:10:41 smithi118 ceph-mon[31535]: osdmap e63: 9 total, 9 up, 9 in 2024-03-24T00:10:41.953 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:10:41 smithi189 ceph-mon[36187]: osdmap e63: 9 total, 9 up, 9 in 2024-03-24T00:10:41.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:10:41 smithi193 ceph-mon[36878]: osdmap e63: 9 total, 9 up, 9 in 2024-03-24T00:10:42.813 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:10:42 smithi118 ceph-mon[31535]: pgmap v221: 1 pgs: 1 remapped+peering; 577 KiB data, 255 MiB used, 804 GiB / 805 GiB avail 2024-03-24T00:10:42.813 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:10:42 smithi118 ceph-mon[31535]: osdmap e64: 9 total, 9 up, 9 in 2024-03-24T00:10:42.953 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:10:42 smithi189 ceph-mon[36187]: pgmap v221: 1 pgs: 1 remapped+peering; 577 KiB data, 255 MiB used, 804 GiB / 805 GiB avail 2024-03-24T00:10:42.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:10:42 smithi189 ceph-mon[36187]: osdmap e64: 9 total, 9 up, 9 in 2024-03-24T00:10:42.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:10:42 smithi193 ceph-mon[36878]: pgmap v221: 1 pgs: 1 remapped+peering; 577 KiB data, 255 MiB used, 804 GiB / 805 GiB avail 2024-03-24T00:10:42.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:10:42 smithi193 ceph-mon[36878]: osdmap e64: 9 total, 9 up, 9 in 2024-03-24T00:10:42.980 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:10:42.980+0000 7f1827600700 1 -- 172.21.15.118:0/2294940075 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f1820108810 msgr2=0x7f1820108c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:10:42.980 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:10:42.980+0000 7f1827600700 1 --2- 172.21.15.118:0/2294940075 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f1820108810 0x7f1820108c30 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7f18100099c0 tx=0x7f1810009cd0 comp rx=0 tx=0).stop 2024-03-24T00:10:42.980 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:10:42.980+0000 7f1827600700 1 -- 172.21.15.118:0/2294940075 shutdown_connections 2024-03-24T00:10:42.981 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:10:42.980+0000 7f1827600700 1 --2- 172.21.15.118:0/2294940075 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f182010a3b0 0x7f1820110c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:10:42.981 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:10:42.980+0000 7f1827600700 1 --2- 172.21.15.118:0/2294940075 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f1820108810 0x7f1820108c30 unknown :-1 s=CLOSED pgs=125 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:10:42.981 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:10:42.980+0000 7f1827600700 1 --2- 172.21.15.118:0/2294940075 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f1820109a10 0x7f1820109e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:10:42.981 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:10:42.980+0000 7f1827600700 1 -- 172.21.15.118:0/2294940075 >> 172.21.15.118:0/2294940075 conn(0x7f18200777e0 msgr2=0x7f1820079c20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:10:42.981 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:10:42.981+0000 7f1827600700 1 -- 172.21.15.118:0/2294940075 shutdown_connections 2024-03-24T00:10:42.981 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:10:42.981+0000 7f1827600700 1 -- 172.21.15.118:0/2294940075 wait complete. 2024-03-24T00:10:42.982 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:10:42.982+0000 7f1827600700 1 Processor -- start 2024-03-24T00:10:42.982 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:10:42.982+0000 7f1827600700 1 -- start start 2024-03-24T00:10:42.982 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:10:42.982+0000 7f1827600700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f1820108810 0x7f182019ea20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:10:42.982 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:10:42.982+0000 7f1827600700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f1820109a10 0x7f182019ef60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:10:42.982 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:10:42.982+0000 7f1827600700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f182010a3b0 0x7f18201a5fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:10:42.983 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:10:42.982+0000 7f1827600700 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1820114440 con 0x7f1820108810 2024-03-24T00:10:42.983 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:10:42.982+0000 7f1827600700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f18201142c0 con 0x7f182010a3b0 2024-03-24T00:10:42.983 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:10:42.982+0000 7f1827600700 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f18201145c0 con 0x7f1820109a10 2024-03-24T00:10:42.983 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:10:42.983+0000 7f1824c00700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f1820108810 0x7f182019ea20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:10:42.983 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:10:42.983+0000 7f1824c00700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f1820108810 0x7f182019ea20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.118:3300/0 says I am v2:172.21.15.118:36644/0 (socket says 172.21.15.118:36644) 2024-03-24T00:10:42.983 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:10:42.983+0000 7f1824c00700 1 -- 172.21.15.118:0/1120670263 learned_addr learned my addr 172.21.15.118:0/1120670263 (peer_addr_for_me v2:172.21.15.118:0/0) 2024-03-24T00:10:42.983 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:10:42.983+0000 7f1817e00700 1 --2- 172.21.15.118:0/1120670263 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f1820109a10 0x7f182019ef60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:10:42.983 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:10:42.983+0000 7f1825600700 1 --2- 172.21.15.118:0/1120670263 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f182010a3b0 0x7f18201a5fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:10:42.984 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:10:42.983+0000 7f1824c00700 1 -- 172.21.15.118:0/1120670263 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f1820109a10 msgr2=0x7f182019ef60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:10:42.984 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:10:42.983+0000 7f1824c00700 1 --2- 172.21.15.118:0/1120670263 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f1820109a10 0x7f182019ef60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:10:42.984 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:10:42.983+0000 7f1824c00700 1 -- 172.21.15.118:0/1120670263 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f182010a3b0 msgr2=0x7f18201a5fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:10:42.984 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:10:42.983+0000 7f1824c00700 1 --2- 172.21.15.118:0/1120670263 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f182010a3b0 0x7f18201a5fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:10:42.984 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:10:42.983+0000 7f1824c00700 1 -- 172.21.15.118:0/1120670263 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f18201a6680 con 0x7f1820108810 2024-03-24T00:10:42.984 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:10:42.983+0000 7f1817e00700 1 --2- 172.21.15.118:0/1120670263 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f1820109a10 0x7f182019ef60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T00:10:42.985 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:10:42.983+0000 7f1824c00700 1 --2- 172.21.15.118:0/1120670263 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f1820108810 0x7f182019ea20 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f181000bee0 tx=0x7f181000bf10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:10:42.985 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:10:42.984+0000 7f181e000700 1 -- 172.21.15.118:0/1120670263 <== mon.0 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1810009e60 con 0x7f1820108810 2024-03-24T00:10:42.985 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:10:42.984+0000 7f181e000700 1 -- 172.21.15.118:0/1120670263 <== mon.0 v2:172.21.15.118:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f1810003e30 con 0x7f1820108810 2024-03-24T00:10:42.985 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:10:42.984+0000 7f1827600700 1 -- 172.21.15.118:0/1120670263 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f18201a6910 con 0x7f1820108810 2024-03-24T00:10:42.985 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:10:42.984+0000 7f181e000700 1 -- 172.21.15.118:0/1120670263 <== mon.0 v2:172.21.15.118:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f18100173f0 con 0x7f1820108810 2024-03-24T00:10:42.985 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:10:42.984+0000 7f1827600700 1 -- 172.21.15.118:0/1120670263 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f18201a6d60 con 0x7f1820108810 2024-03-24T00:10:42.985 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:10:42.985+0000 7f1827600700 1 -- 172.21.15.118:0/1120670263 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1820068e50 con 0x7f1820108810 2024-03-24T00:10:42.986 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:10:42.987+0000 7f181e000700 1 -- 172.21.15.118:0/1120670263 <== mon.0 v2:172.21.15.118:3300/0 4 ==== mgrmap(e 16) v1 ==== 137291+0+0 (secure 0 0 0) 0x7f181000f660 con 0x7f1820108810 2024-03-24T00:10:42.989 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:10:42.989+0000 7f181e000700 1 --2- 172.21.15.118:0/1120670263 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7f18040a2890 0x7f18040a4d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:10:42.989 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:10:42.989+0000 7f181e000700 1 -- 172.21.15.118:0/1120670263 <== mon.0 v2:172.21.15.118:3300/0 5 ==== osd_map(64..64 src has 1..64) v4 ==== 5829+0+0 (secure 0 0 0) 0x7f18100ceed0 con 0x7f1820108810 2024-03-24T00:10:42.989 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:10:42.990+0000 7f1817e00700 1 --2- 172.21.15.118:0/1120670263 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7f18040a2890 0x7f18040a4d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:10:42.990 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:10:42.990+0000 7f1817e00700 1 --2- 172.21.15.118:0/1120670263 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7f18040a2890 0x7f18040a4d50 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f1808011490 tx=0x7f1808009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:10:42.994 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:10:42.994+0000 7f181e000700 1 -- 172.21.15.118:0/1120670263 <== mon.0 v2:172.21.15.118:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f1810013070 con 0x7f1820108810 2024-03-24T00:10:43.231 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:10:43.231+0000 7f1827600700 1 -- 172.21.15.118:0/1120670263 --> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi118:vg_nvme/lv_1", "target": ["mon-mgr", ""]}) v1 -- 0x7f18200625d0 con 0x7f18040a2890 2024-03-24T00:10:43.953 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:10:43 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-24T00:10:43.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:10:43 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-24T00:10:43.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:10:43 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T00:10:43.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:10:43 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-24T00:10:43.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:10:43 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-24T00:10:43.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:10:43 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T00:10:44.155 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:10:43 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-24T00:10:44.156 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:10:43 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-24T00:10:44.156 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:10:43 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T00:10:44.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:10:44 smithi118 ceph-mon[31535]: from='client.14304 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi118:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T00:10:44.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:10:44 smithi118 ceph-mon[31535]: pgmap v223: 1 pgs: 1 active+clean; 577 KiB data, 255 MiB used, 804 GiB / 805 GiB avail; 118 KiB/s, 0 objects/s recovering 2024-03-24T00:10:44.953 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:10:44 smithi189 ceph-mon[36187]: from='client.14304 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi118:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T00:10:44.953 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:10:44 smithi189 ceph-mon[36187]: pgmap v223: 1 pgs: 1 active+clean; 577 KiB data, 255 MiB used, 804 GiB / 805 GiB avail; 118 KiB/s, 0 objects/s recovering 2024-03-24T00:10:44.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:10:44 smithi193 ceph-mon[36878]: from='client.14304 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi118:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T00:10:44.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:10:44 smithi193 ceph-mon[36878]: pgmap v223: 1 pgs: 1 active+clean; 577 KiB data, 255 MiB used, 804 GiB / 805 GiB avail; 118 KiB/s, 0 objects/s recovering 2024-03-24T00:10:46.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:10:46 smithi193 ceph-mon[36878]: pgmap v224: 1 pgs: 1 active+clean; 577 KiB data, 255 MiB used, 804 GiB / 805 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-03-24T00:10:47.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:10:46 smithi118 ceph-mon[31535]: pgmap v224: 1 pgs: 1 active+clean; 577 KiB data, 255 MiB used, 804 GiB / 805 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-03-24T00:10:47.203 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:10:46 smithi189 ceph-mon[36187]: pgmap v224: 1 pgs: 1 active+clean; 577 KiB data, 255 MiB used, 804 GiB / 805 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-03-24T00:10:47.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:10:47 smithi193 ceph-mon[36878]: from='client.? 172.21.15.118:0/3797989517' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "54be277f-3596-422f-9de5-9005a8571c6f"}]: dispatch 2024-03-24T00:10:47.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:10:47 smithi193 ceph-mon[36878]: from='client.? 172.21.15.118:0/3797989517' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "54be277f-3596-422f-9de5-9005a8571c6f"}]': finished 2024-03-24T00:10:47.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:10:47 smithi193 ceph-mon[36878]: osdmap e65: 10 total, 9 up, 10 in 2024-03-24T00:10:47.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:10:47 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-03-24T00:10:48.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:10:47 smithi118 ceph-mon[31535]: from='client.? 172.21.15.118:0/3797989517' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "54be277f-3596-422f-9de5-9005a8571c6f"}]: dispatch 2024-03-24T00:10:48.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:10:47 smithi118 ceph-mon[31535]: from='client.? 172.21.15.118:0/3797989517' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "54be277f-3596-422f-9de5-9005a8571c6f"}]': finished 2024-03-24T00:10:48.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:10:47 smithi118 ceph-mon[31535]: osdmap e65: 10 total, 9 up, 10 in 2024-03-24T00:10:48.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:10:47 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-03-24T00:10:48.203 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:10:47 smithi189 ceph-mon[36187]: from='client.? 172.21.15.118:0/3797989517' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "54be277f-3596-422f-9de5-9005a8571c6f"}]: dispatch 2024-03-24T00:10:48.203 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:10:47 smithi189 ceph-mon[36187]: from='client.? 172.21.15.118:0/3797989517' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "54be277f-3596-422f-9de5-9005a8571c6f"}]': finished 2024-03-24T00:10:48.203 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:10:47 smithi189 ceph-mon[36187]: osdmap e65: 10 total, 9 up, 10 in 2024-03-24T00:10:48.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:10:47 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-03-24T00:10:48.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:10:48 smithi193 ceph-mon[36878]: pgmap v226: 1 pgs: 1 active+clean; 577 KiB data, 255 MiB used, 804 GiB / 805 GiB avail; 84 KiB/s, 0 objects/s recovering 2024-03-24T00:10:48.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:10:48 smithi193 ceph-mon[36878]: from='client.? 172.21.15.118:0/369922382' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-24T00:10:49.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:10:48 smithi118 ceph-mon[31535]: pgmap v226: 1 pgs: 1 active+clean; 577 KiB data, 255 MiB used, 804 GiB / 805 GiB avail; 84 KiB/s, 0 objects/s recovering 2024-03-24T00:10:49.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:10:48 smithi118 ceph-mon[31535]: from='client.? 172.21.15.118:0/369922382' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-24T00:10:49.203 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:10:48 smithi189 ceph-mon[36187]: pgmap v226: 1 pgs: 1 active+clean; 577 KiB data, 255 MiB used, 804 GiB / 805 GiB avail; 84 KiB/s, 0 objects/s recovering 2024-03-24T00:10:49.203 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:10:48 smithi189 ceph-mon[36187]: from='client.? 172.21.15.118:0/369922382' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-24T00:10:51.031 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:10:50 smithi193 ceph-mon[36878]: pgmap v227: 1 pgs: 1 active+clean; 577 KiB data, 255 MiB used, 804 GiB / 805 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-03-24T00:10:51.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:10:50 smithi118 ceph-mon[31535]: pgmap v227: 1 pgs: 1 active+clean; 577 KiB data, 255 MiB used, 804 GiB / 805 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-03-24T00:10:51.203 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:10:50 smithi189 ceph-mon[36187]: pgmap v227: 1 pgs: 1 active+clean; 577 KiB data, 255 MiB used, 804 GiB / 805 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-03-24T00:10:53.195 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:10:52 smithi118 ceph-mon[31535]: pgmap v228: 1 pgs: 1 active+clean; 577 KiB data, 255 MiB used, 804 GiB / 805 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-03-24T00:10:53.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:10:52 smithi189 ceph-mon[36187]: pgmap v228: 1 pgs: 1 active+clean; 577 KiB data, 255 MiB used, 804 GiB / 805 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-03-24T00:10:53.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:10:52 smithi193 ceph-mon[36878]: pgmap v228: 1 pgs: 1 active+clean; 577 KiB data, 255 MiB used, 804 GiB / 805 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-03-24T00:10:54.836 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:10:54 smithi118 ceph-mon[31535]: pgmap v229: 1 pgs: 1 active+clean; 577 KiB data, 255 MiB used, 804 GiB / 805 GiB avail 2024-03-24T00:10:55.203 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:10:54 smithi189 ceph-mon[36187]: pgmap v229: 1 pgs: 1 active+clean; 577 KiB data, 255 MiB used, 804 GiB / 805 GiB avail 2024-03-24T00:10:55.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:10:54 smithi193 ceph-mon[36878]: pgmap v229: 1 pgs: 1 active+clean; 577 KiB data, 255 MiB used, 804 GiB / 805 GiB avail 2024-03-24T00:10:57.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:10:56 smithi118 ceph-mon[31535]: pgmap v230: 1 pgs: 1 active+clean; 577 KiB data, 256 MiB used, 804 GiB / 805 GiB avail 2024-03-24T00:10:57.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:10:56 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.9"}]: dispatch 2024-03-24T00:10:57.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:10:56 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T00:10:57.203 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:10:56 smithi189 ceph-mon[36187]: pgmap v230: 1 pgs: 1 active+clean; 577 KiB data, 256 MiB used, 804 GiB / 805 GiB avail 2024-03-24T00:10:57.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:10:56 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.9"}]: dispatch 2024-03-24T00:10:57.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:10:56 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T00:10:57.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:10:56 smithi193 ceph-mon[36878]: pgmap v230: 1 pgs: 1 active+clean; 577 KiB data, 256 MiB used, 804 GiB / 805 GiB avail 2024-03-24T00:10:57.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:10:56 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.9"}]: dispatch 2024-03-24T00:10:57.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:10:56 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T00:10:58.016 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:10:57 smithi118 ceph-mon[31535]: Deploying daemon osd.9 on smithi118 2024-03-24T00:10:58.203 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:10:57 smithi189 ceph-mon[36187]: Deploying daemon osd.9 on smithi118 2024-03-24T00:10:58.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:10:57 smithi193 ceph-mon[36878]: Deploying daemon osd.9 on smithi118 2024-03-24T00:10:59.027 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:10:58 smithi118 ceph-mon[31535]: pgmap v231: 1 pgs: 1 active+clean; 577 KiB data, 256 MiB used, 804 GiB / 805 GiB avail 2024-03-24T00:10:59.203 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:10:58 smithi189 ceph-mon[36187]: pgmap v231: 1 pgs: 1 active+clean; 577 KiB data, 256 MiB used, 804 GiB / 805 GiB avail 2024-03-24T00:10:59.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:10:58 smithi193 ceph-mon[36878]: pgmap v231: 1 pgs: 1 active+clean; 577 KiB data, 256 MiB used, 804 GiB / 805 GiB avail 2024-03-24T00:10:59.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:10:59 smithi118 ceph-mon[31535]: pgmap v232: 1 pgs: 1 active+clean; 577 KiB data, 256 MiB used, 804 GiB / 805 GiB avail 2024-03-24T00:11:00.203 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:10:59 smithi189 ceph-mon[36187]: pgmap v232: 1 pgs: 1 active+clean; 577 KiB data, 256 MiB used, 804 GiB / 805 GiB avail 2024-03-24T00:11:00.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:10:59 smithi193 ceph-mon[36878]: pgmap v232: 1 pgs: 1 active+clean; 577 KiB data, 256 MiB used, 804 GiB / 805 GiB avail 2024-03-24T00:11:02.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:11:02 smithi118 ceph-mon[31535]: pgmap v233: 1 pgs: 1 active+clean; 577 KiB data, 256 MiB used, 804 GiB / 805 GiB avail 2024-03-24T00:11:02.953 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:11:02 smithi189 ceph-mon[36187]: pgmap v233: 1 pgs: 1 active+clean; 577 KiB data, 256 MiB used, 804 GiB / 805 GiB avail 2024-03-24T00:11:02.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:11:02 smithi193 ceph-mon[36878]: pgmap v233: 1 pgs: 1 active+clean; 577 KiB data, 256 MiB used, 804 GiB / 805 GiB avail 2024-03-24T00:11:04.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:11:04 smithi118 ceph-mon[31535]: pgmap v234: 1 pgs: 1 active+clean; 577 KiB data, 256 MiB used, 804 GiB / 805 GiB avail 2024-03-24T00:11:04.953 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:11:04 smithi189 ceph-mon[36187]: pgmap v234: 1 pgs: 1 active+clean; 577 KiB data, 256 MiB used, 804 GiB / 805 GiB avail 2024-03-24T00:11:04.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:11:04 smithi193 ceph-mon[36878]: pgmap v234: 1 pgs: 1 active+clean; 577 KiB data, 256 MiB used, 804 GiB / 805 GiB avail 2024-03-24T00:11:06.109 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:11:05 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T00:11:06.110 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:11:05 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T00:11:06.110 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:11:05 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T00:11:06.110 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:11:05 smithi118 ceph-mon[31535]: pgmap v235: 1 pgs: 1 active+clean; 577 KiB data, 256 MiB used, 804 GiB / 805 GiB avail 2024-03-24T00:11:06.203 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:11:05 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T00:11:06.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:11:05 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T00:11:06.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:11:05 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T00:11:06.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:11:05 smithi189 ceph-mon[36187]: pgmap v235: 1 pgs: 1 active+clean; 577 KiB data, 256 MiB used, 804 GiB / 805 GiB avail 2024-03-24T00:11:06.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:11:05 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T00:11:06.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:11:05 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T00:11:06.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:11:05 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T00:11:06.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:11:05 smithi193 ceph-mon[36878]: pgmap v235: 1 pgs: 1 active+clean; 577 KiB data, 256 MiB used, 804 GiB / 805 GiB avail 2024-03-24T00:11:08.945 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:11:08 smithi118 ceph-mon[31535]: pgmap v236: 1 pgs: 1 active+clean; 577 KiB data, 256 MiB used, 804 GiB / 805 GiB avail 2024-03-24T00:11:08.953 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:11:08 smithi189 ceph-mon[36187]: pgmap v236: 1 pgs: 1 active+clean; 577 KiB data, 256 MiB used, 804 GiB / 805 GiB avail 2024-03-24T00:11:08.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:11:08 smithi193 ceph-mon[36878]: pgmap v236: 1 pgs: 1 active+clean; 577 KiB data, 256 MiB used, 804 GiB / 805 GiB avail 2024-03-24T00:11:09.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:11:09 smithi118 ceph-mon[31535]: from='osd.9 [v2:172.21.15.118:6826/3248293936,v1:172.21.15.118:6827/3248293936]' entity='osd.9' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]: dispatch 2024-03-24T00:11:09.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:11:09 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T00:11:09.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:11:09 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T00:11:09.953 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:11:09 smithi189 ceph-mon[36187]: from='osd.9 [v2:172.21.15.118:6826/3248293936,v1:172.21.15.118:6827/3248293936]' entity='osd.9' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]: dispatch 2024-03-24T00:11:09.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:11:09 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T00:11:09.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:11:09 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T00:11:09.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:11:09 smithi193 ceph-mon[36878]: from='osd.9 [v2:172.21.15.118:6826/3248293936,v1:172.21.15.118:6827/3248293936]' entity='osd.9' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]: dispatch 2024-03-24T00:11:09.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:11:09 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T00:11:09.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:11:09 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T00:11:09.980 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:11:09.979+0000 7f181e000700 1 -- 172.21.15.118:0/1120670263 <== mgr.14152 v2:172.21.15.118:6800/2112912943 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7f18200625d0 con 0x7f18040a2890 2024-03-24T00:11:09.980 INFO:teuthology.orchestra.run.smithi118.stdout:Created osd(s) 9 on host 'smithi118' 2024-03-24T00:11:09.983 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:11:09.982+0000 7f1827600700 1 -- 172.21.15.118:0/1120670263 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7f18040a2890 msgr2=0x7f18040a4d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:11:09.983 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:11:09.982+0000 7f1827600700 1 --2- 172.21.15.118:0/1120670263 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7f18040a2890 0x7f18040a4d50 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f1808011490 tx=0x7f1808009260 comp rx=0 tx=0).stop 2024-03-24T00:11:09.984 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:11:09.982+0000 7f1827600700 1 -- 172.21.15.118:0/1120670263 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f1820108810 msgr2=0x7f182019ea20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:11:09.984 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:11:09.982+0000 7f1827600700 1 --2- 172.21.15.118:0/1120670263 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f1820108810 0x7f182019ea20 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f181000bee0 tx=0x7f181000bf10 comp rx=0 tx=0).stop 2024-03-24T00:11:09.984 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:11:09.983+0000 7f1827600700 1 -- 172.21.15.118:0/1120670263 shutdown_connections 2024-03-24T00:11:09.984 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:11:09.983+0000 7f1827600700 1 --2- 172.21.15.118:0/1120670263 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f182010a3b0 0x7f18201a5fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:11:09.984 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:11:09.983+0000 7f1827600700 1 --2- 172.21.15.118:0/1120670263 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7f18040a2890 0x7f18040a4d50 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:11:09.984 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:11:09.983+0000 7f1827600700 1 --2- 172.21.15.118:0/1120670263 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f1820108810 0x7f182019ea20 unknown :-1 s=CLOSED pgs=126 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:11:09.984 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:11:09.983+0000 7f1827600700 1 --2- 172.21.15.118:0/1120670263 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f1820109a10 0x7f182019ef60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:11:09.984 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:11:09.983+0000 7f1827600700 1 -- 172.21.15.118:0/1120670263 >> 172.21.15.118:0/1120670263 conn(0x7f18200777e0 msgr2=0x7f182010cd10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:11:09.984 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:11:09.983+0000 7f1827600700 1 -- 172.21.15.118:0/1120670263 shutdown_connections 2024-03-24T00:11:09.984 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:11:09.983+0000 7f1827600700 1 -- 172.21.15.118:0/1120670263 wait complete. 2024-03-24T00:11:10.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:11:10 smithi118 ceph-mon[31535]: pgmap v237: 1 pgs: 1 active+clean; 577 KiB data, 256 MiB used, 804 GiB / 805 GiB avail 2024-03-24T00:11:10.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:11:10 smithi118 ceph-mon[31535]: from='osd.9 [v2:172.21.15.118:6826/3248293936,v1:172.21.15.118:6827/3248293936]' entity='osd.9' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]': finished 2024-03-24T00:11:10.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:11:10 smithi118 ceph-mon[31535]: osdmap e66: 10 total, 9 up, 10 in 2024-03-24T00:11:10.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:11:10 smithi118 ceph-mon[31535]: from='osd.9 [v2:172.21.15.118:6826/3248293936,v1:172.21.15.118:6827/3248293936]' entity='osd.9' cmd=[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi118", "root=default"]}]: dispatch 2024-03-24T00:11:10.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:11:10 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-03-24T00:11:10.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:11:10 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T00:11:10.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:11:10 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T00:11:10.891 DEBUG:teuthology.orchestra.run.smithi118:osd.9> sudo journalctl -f -n 0 -u ceph-d79a5420-e971-11ee-95cd-87774f69a715@osd.9.service 2024-03-24T00:11:10.897 INFO:tasks.cephadm:Deploying osd.10 on smithi189 with /dev/vg_nvme/lv_1... 2024-03-24T00:11:10.897 DEBUG:teuthology.orchestra.run.smithi189:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d79a5420-e971-11ee-95cd-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_1 2024-03-24T00:11:10.926 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:11:10 smithi189 ceph-mon[36187]: pgmap v237: 1 pgs: 1 active+clean; 577 KiB data, 256 MiB used, 804 GiB / 805 GiB avail 2024-03-24T00:11:10.926 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:11:10 smithi189 ceph-mon[36187]: from='osd.9 [v2:172.21.15.118:6826/3248293936,v1:172.21.15.118:6827/3248293936]' entity='osd.9' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]': finished 2024-03-24T00:11:10.926 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:11:10 smithi189 ceph-mon[36187]: osdmap e66: 10 total, 9 up, 10 in 2024-03-24T00:11:10.927 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:11:10 smithi189 ceph-mon[36187]: from='osd.9 [v2:172.21.15.118:6826/3248293936,v1:172.21.15.118:6827/3248293936]' entity='osd.9' cmd=[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi118", "root=default"]}]: dispatch 2024-03-24T00:11:10.927 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:11:10 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-03-24T00:11:10.927 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:11:10 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T00:11:10.927 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:11:10 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T00:11:10.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:11:10 smithi193 ceph-mon[36878]: pgmap v237: 1 pgs: 1 active+clean; 577 KiB data, 256 MiB used, 804 GiB / 805 GiB avail 2024-03-24T00:11:10.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:11:10 smithi193 ceph-mon[36878]: from='osd.9 [v2:172.21.15.118:6826/3248293936,v1:172.21.15.118:6827/3248293936]' entity='osd.9' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["9"]}]': finished 2024-03-24T00:11:10.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:11:10 smithi193 ceph-mon[36878]: osdmap e66: 10 total, 9 up, 10 in 2024-03-24T00:11:10.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:11:10 smithi193 ceph-mon[36878]: from='osd.9 [v2:172.21.15.118:6826/3248293936,v1:172.21.15.118:6827/3248293936]' entity='osd.9' cmd=[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi118", "root=default"]}]: dispatch 2024-03-24T00:11:10.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:11:10 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-03-24T00:11:10.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:11:10 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T00:11:10.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:11:10 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T00:11:11.144 INFO:teuthology.orchestra.run.smithi189.stderr:Inferring config /var/lib/ceph/d79a5420-e971-11ee-95cd-87774f69a715/mon.b/config 2024-03-24T00:11:11.602 INFO:journalctl@ceph.osd.9.smithi118.stdout:Mar 24 00:11:11 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-osd-9[67027]: 2024-03-24T00:11:11.249+0000 7f69ff800700 -1 osd.9 0 waiting for initial osdmap 2024-03-24T00:11:11.602 INFO:journalctl@ceph.osd.9.smithi118.stdout:Mar 24 00:11:11 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-osd-9[67027]: 2024-03-24T00:11:11.277+0000 7f69f5600700 -1 osd.9 67 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-24T00:11:11.875 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:11:11 smithi118 ceph-mon[31535]: from='osd.9 [v2:172.21.15.118:6826/3248293936,v1:172.21.15.118:6827/3248293936]' entity='osd.9' cmd='[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi118", "root=default"]}]': finished 2024-03-24T00:11:11.875 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:11:11 smithi118 ceph-mon[31535]: osdmap e67: 10 total, 9 up, 10 in 2024-03-24T00:11:11.876 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:11:11 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-03-24T00:11:11.876 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:11:11 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-03-24T00:11:11.876 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:11:11 smithi118 ceph-mon[31535]: from='osd.9 [v2:172.21.15.118:6826/3248293936,v1:172.21.15.118:6827/3248293936]' entity='osd.9' 2024-03-24T00:11:11.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:11:11 smithi189 ceph-mon[36187]: from='osd.9 [v2:172.21.15.118:6826/3248293936,v1:172.21.15.118:6827/3248293936]' entity='osd.9' cmd='[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi118", "root=default"]}]': finished 2024-03-24T00:11:11.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:11:11 smithi189 ceph-mon[36187]: osdmap e67: 10 total, 9 up, 10 in 2024-03-24T00:11:11.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:11:11 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-03-24T00:11:11.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:11:11 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-03-24T00:11:11.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:11:11 smithi189 ceph-mon[36187]: from='osd.9 [v2:172.21.15.118:6826/3248293936,v1:172.21.15.118:6827/3248293936]' entity='osd.9' 2024-03-24T00:11:11.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:11:11 smithi193 ceph-mon[36878]: from='osd.9 [v2:172.21.15.118:6826/3248293936,v1:172.21.15.118:6827/3248293936]' entity='osd.9' cmd='[{"prefix": "osd crush create-or-move", "id": 9, "weight":0.0873, "args": ["host=smithi118", "root=default"]}]': finished 2024-03-24T00:11:11.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:11:11 smithi193 ceph-mon[36878]: osdmap e67: 10 total, 9 up, 10 in 2024-03-24T00:11:11.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:11:11 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-03-24T00:11:11.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:11:11 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-03-24T00:11:11.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:11:11 smithi193 ceph-mon[36878]: from='osd.9 [v2:172.21.15.118:6826/3248293936,v1:172.21.15.118:6827/3248293936]' entity='osd.9' 2024-03-24T00:11:12.853 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:11:12 smithi189 ceph-mon[36187]: purged_snaps scrub starts 2024-03-24T00:11:12.853 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:11:12 smithi189 ceph-mon[36187]: purged_snaps scrub ok 2024-03-24T00:11:12.853 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:11:12 smithi189 ceph-mon[36187]: pgmap v240: 1 pgs: 1 active+clean; 577 KiB data, 256 MiB used, 804 GiB / 805 GiB avail 2024-03-24T00:11:12.853 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:11:12 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-03-24T00:11:12.853 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:11:12 smithi189 ceph-mon[36187]: osd.9 [v2:172.21.15.118:6826/3248293936,v1:172.21.15.118:6827/3248293936] boot 2024-03-24T00:11:12.853 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:11:12 smithi189 ceph-mon[36187]: osdmap e68: 10 total, 10 up, 10 in 2024-03-24T00:11:12.853 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:11:12 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-03-24T00:11:12.921 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:11:12 smithi118 ceph-mon[31535]: purged_snaps scrub starts 2024-03-24T00:11:12.921 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:11:12 smithi118 ceph-mon[31535]: purged_snaps scrub ok 2024-03-24T00:11:12.921 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:11:12 smithi118 ceph-mon[31535]: pgmap v240: 1 pgs: 1 active+clean; 577 KiB data, 256 MiB used, 804 GiB / 805 GiB avail 2024-03-24T00:11:12.921 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:11:12 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-03-24T00:11:12.921 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:11:12 smithi118 ceph-mon[31535]: osd.9 [v2:172.21.15.118:6826/3248293936,v1:172.21.15.118:6827/3248293936] boot 2024-03-24T00:11:12.921 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:11:12 smithi118 ceph-mon[31535]: osdmap e68: 10 total, 10 up, 10 in 2024-03-24T00:11:12.921 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:11:12 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-03-24T00:11:12.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:11:12 smithi193 ceph-mon[36878]: purged_snaps scrub starts 2024-03-24T00:11:12.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:11:12 smithi193 ceph-mon[36878]: purged_snaps scrub ok 2024-03-24T00:11:12.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:11:12 smithi193 ceph-mon[36878]: pgmap v240: 1 pgs: 1 active+clean; 577 KiB data, 256 MiB used, 804 GiB / 805 GiB avail 2024-03-24T00:11:12.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:11:12 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-03-24T00:11:12.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:11:12 smithi193 ceph-mon[36878]: osd.9 [v2:172.21.15.118:6826/3248293936,v1:172.21.15.118:6827/3248293936] boot 2024-03-24T00:11:12.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:11:12 smithi193 ceph-mon[36878]: osdmap e68: 10 total, 10 up, 10 in 2024-03-24T00:11:12.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:11:12 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 9}]: dispatch 2024-03-24T00:11:13.294 INFO:teuthology.orchestra.run.smithi189.stderr:--> Zapping: /dev/vg_nvme/lv_1 2024-03-24T00:11:13.294 INFO:teuthology.orchestra.run.smithi189.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_1 bs=1M count=10 conv=fsync 2024-03-24T00:11:13.294 INFO:teuthology.orchestra.run.smithi189.stderr: stderr: 10+0 records in 2024-03-24T00:11:13.294 INFO:teuthology.orchestra.run.smithi189.stderr:10+0 records out 2024-03-24T00:11:13.294 INFO:teuthology.orchestra.run.smithi189.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.011404 s, 919 MB/s 2024-03-24T00:11:13.294 INFO:teuthology.orchestra.run.smithi189.stderr:--> Zapping successful for: 2024-03-24T00:11:14.029 DEBUG:teuthology.orchestra.run.smithi189:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d79a5420-e971-11ee-95cd-87774f69a715 -- ceph orch daemon add osd smithi189:vg_nvme/lv_1 2024-03-24T00:11:14.310 INFO:teuthology.orchestra.run.smithi189.stderr:Inferring config /var/lib/ceph/d79a5420-e971-11ee-95cd-87774f69a715/mon.b/config 2024-03-24T00:11:14.703 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:11:14 smithi189 ceph-mon[36187]: pgmap v242: 1 pgs: 1 active+clean; 577 KiB data, 287 MiB used, 894 GiB / 894 GiB avail 2024-03-24T00:11:14.704 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:11:14 smithi189 ceph-mon[36187]: osdmap e69: 10 total, 10 up, 10 in 2024-03-24T00:11:14.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:11:14 smithi118 ceph-mon[31535]: pgmap v242: 1 pgs: 1 active+clean; 577 KiB data, 287 MiB used, 894 GiB / 894 GiB avail 2024-03-24T00:11:14.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:11:14 smithi118 ceph-mon[31535]: osdmap e69: 10 total, 10 up, 10 in 2024-03-24T00:11:14.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:11:14 smithi193 ceph-mon[36878]: pgmap v242: 1 pgs: 1 active+clean; 577 KiB data, 287 MiB used, 894 GiB / 894 GiB avail 2024-03-24T00:11:14.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:11:14 smithi193 ceph-mon[36878]: osdmap e69: 10 total, 10 up, 10 in 2024-03-24T00:11:16.061 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-24T00:11:16.061+0000 7fe08b704700 1 -- 172.21.15.189:0/2478218850 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe0841021c0 msgr2=0x7fe0841025e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:11:16.061 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-24T00:11:16.061+0000 7fe08b704700 1 --2- 172.21.15.189:0/2478218850 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe0841021c0 0x7fe0841025e0 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7fe0800099c0 tx=0x7fe080009cd0 comp rx=0 tx=0).stop 2024-03-24T00:11:16.062 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-24T00:11:16.061+0000 7fe08b704700 1 -- 172.21.15.189:0/2478218850 shutdown_connections 2024-03-24T00:11:16.062 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-24T00:11:16.061+0000 7fe08b704700 1 --2- 172.21.15.189:0/2478218850 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe0841021c0 0x7fe0841025e0 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:11:16.062 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-24T00:11:16.061+0000 7fe08b704700 1 --2- 172.21.15.189:0/2478218850 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fe084103d60 0x7fe08410a610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:11:16.062 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-24T00:11:16.061+0000 7fe08b704700 1 --2- 172.21.15.189:0/2478218850 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fe0841033c0 0x7fe084103820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:11:16.062 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-24T00:11:16.061+0000 7fe08b704700 1 -- 172.21.15.189:0/2478218850 >> 172.21.15.189:0/2478218850 conn(0x7fe0840fd7a0 msgr2=0x7fe0840ffbc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:11:16.062 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-24T00:11:16.061+0000 7fe08b704700 1 -- 172.21.15.189:0/2478218850 shutdown_connections 2024-03-24T00:11:16.062 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-24T00:11:16.061+0000 7fe08b704700 1 -- 172.21.15.189:0/2478218850 wait complete. 2024-03-24T00:11:16.064 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-24T00:11:16.062+0000 7fe08b704700 1 Processor -- start 2024-03-24T00:11:16.064 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-24T00:11:16.062+0000 7fe08b704700 1 -- start start 2024-03-24T00:11:16.064 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-24T00:11:16.062+0000 7fe08b704700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe0841021c0 0x7fe084198400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:11:16.064 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-24T00:11:16.062+0000 7fe08b704700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fe0841033c0 0x7fe084198940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:11:16.064 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-24T00:11:16.062+0000 7fe08b704700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fe084103d60 0x7fe08419f9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:11:16.064 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-24T00:11:16.062+0000 7fe08b704700 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe08410de10 con 0x7fe084103d60 2024-03-24T00:11:16.064 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-24T00:11:16.062+0000 7fe08b704700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe08410dc90 con 0x7fe0841021c0 2024-03-24T00:11:16.064 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-24T00:11:16.062+0000 7fe08b704700 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe08410df90 con 0x7fe0841033c0 2024-03-24T00:11:16.064 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-24T00:11:16.062+0000 7fe0894a0700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe0841021c0 0x7fe084198400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:11:16.065 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-24T00:11:16.062+0000 7fe0894a0700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe0841021c0 0x7fe084198400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.189:38200/0 (socket says 172.21.15.189:38200) 2024-03-24T00:11:16.065 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-24T00:11:16.062+0000 7fe0894a0700 1 -- 172.21.15.189:0/670865682 learned_addr learned my addr 172.21.15.189:0/670865682 (peer_addr_for_me v2:172.21.15.189:0/0) 2024-03-24T00:11:16.065 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-24T00:11:16.062+0000 7fe088c9f700 1 --2- 172.21.15.189:0/670865682 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fe0841033c0 0x7fe084198940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:11:16.065 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-24T00:11:16.062+0000 7fe0894a0700 1 -- 172.21.15.189:0/670865682 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fe0841033c0 msgr2=0x7fe084198940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:11:16.065 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-24T00:11:16.062+0000 7fe0894a0700 1 --2- 172.21.15.189:0/670865682 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fe0841033c0 0x7fe084198940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:11:16.065 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-24T00:11:16.062+0000 7fe0894a0700 1 -- 172.21.15.189:0/670865682 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fe084103d60 msgr2=0x7fe08419f9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:11:16.065 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-24T00:11:16.062+0000 7fe0894a0700 1 --2- 172.21.15.189:0/670865682 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fe084103d60 0x7fe08419f9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:11:16.065 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-24T00:11:16.062+0000 7fe0894a0700 1 -- 172.21.15.189:0/670865682 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe0841a0060 con 0x7fe0841021c0 2024-03-24T00:11:16.065 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-24T00:11:16.062+0000 7fe088c9f700 1 --2- 172.21.15.189:0/670865682 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fe0841033c0 0x7fe084198940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-24T00:11:16.065 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-24T00:11:16.062+0000 7fe0894a0700 1 --2- 172.21.15.189:0/670865682 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe0841021c0 0x7fe084198400 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7fe08000c010 tx=0x7fe080009f10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:11:16.065 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-24T00:11:16.063+0000 7fe07a7fc700 1 -- 172.21.15.189:0/670865682 <== mon.2 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe080027070 con 0x7fe0841021c0 2024-03-24T00:11:16.065 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-24T00:11:16.063+0000 7fe08b704700 1 -- 172.21.15.189:0/670865682 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe0841a02f0 con 0x7fe0841021c0 2024-03-24T00:11:16.065 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-24T00:11:16.063+0000 7fe07a7fc700 1 -- 172.21.15.189:0/670865682 <== mon.2 v2:172.21.15.189:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fe080004150 con 0x7fe0841021c0 2024-03-24T00:11:16.065 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-24T00:11:16.063+0000 7fe07a7fc700 1 -- 172.21.15.189:0/670865682 <== mon.2 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe08000f460 con 0x7fe0841021c0 2024-03-24T00:11:16.065 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-24T00:11:16.063+0000 7fe08b704700 1 -- 172.21.15.189:0/670865682 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe0841a0850 con 0x7fe0841021c0 2024-03-24T00:11:16.066 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-24T00:11:16.064+0000 7fe08b704700 1 -- 172.21.15.189:0/670865682 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe084068e50 con 0x7fe0841021c0 2024-03-24T00:11:16.066 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-24T00:11:16.065+0000 7fe07a7fc700 1 -- 172.21.15.189:0/670865682 <== mon.2 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 16) v1 ==== 137291+0+0 (secure 0 0 0) 0x7fe08002a070 con 0x7fe0841021c0 2024-03-24T00:11:16.066 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-24T00:11:16.065+0000 7fe07a7fc700 1 --2- 172.21.15.189:0/670865682 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7fe0700a2890 0x7fe0700a4d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:11:16.066 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-24T00:11:16.066+0000 7fe07a7fc700 1 -- 172.21.15.189:0/670865682 <== mon.2 v2:172.21.15.189:3300/0 5 ==== osd_map(69..69 src has 1..69) v4 ==== 6261+0+0 (secure 0 0 0) 0x7fe0800cf220 con 0x7fe0841021c0 2024-03-24T00:11:16.066 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-24T00:11:16.066+0000 7fe088c9f700 1 --2- 172.21.15.189:0/670865682 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7fe0700a2890 0x7fe0700a4d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:11:16.067 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-24T00:11:16.067+0000 7fe088c9f700 1 --2- 172.21.15.189:0/670865682 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7fe0700a2890 0x7fe0700a4d50 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7fe074005fd0 tx=0x7fe074005ee0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:11:16.070 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-24T00:11:16.070+0000 7fe07a7fc700 1 -- 172.21.15.189:0/670865682 <== mon.2 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7fe08005e740 con 0x7fe0841021c0 2024-03-24T00:11:16.277 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-24T00:11:16.277+0000 7fe08b704700 1 -- 172.21.15.189:0/670865682 --> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi189:vg_nvme/lv_1", "target": ["mon-mgr", ""]}) v1 -- 0x7fe0840625d0 con 0x7fe0700a2890 2024-03-24T00:11:16.903 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:11:16 smithi189 ceph-mon[36187]: pgmap v244: 1 pgs: 1 active+clean; 577 KiB data, 287 MiB used, 894 GiB / 894 GiB avail 2024-03-24T00:11:16.903 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:11:16 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T00:11:16.903 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:11:16 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T00:11:16.903 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:11:16 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-24T00:11:16.903 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:11:16 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-24T00:11:16.903 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:11:16 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-24T00:11:16.904 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:11:16 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-03-24T00:11:16.904 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:11:16 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T00:11:16.904 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:11:16 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T00:11:16.904 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:11:16 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T00:11:16.904 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:11:16 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T00:11:16.904 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:11:16 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-24T00:11:16.904 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:11:16 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-24T00:11:16.904 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:11:16 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T00:11:16.904 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:11:16 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T00:11:16.904 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:11:16 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T00:11:16.904 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:11:16 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T00:11:16.904 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:11:16 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T00:11:16.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:11:16 smithi118 ceph-mon[31535]: pgmap v244: 1 pgs: 1 active+clean; 577 KiB data, 287 MiB used, 894 GiB / 894 GiB avail 2024-03-24T00:11:16.947 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:11:16 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T00:11:16.947 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:11:16 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T00:11:16.947 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:11:16 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-24T00:11:16.947 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:11:16 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-24T00:11:16.947 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:11:16 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-24T00:11:16.947 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:11:16 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-03-24T00:11:16.947 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:11:16 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T00:11:16.947 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:11:16 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T00:11:16.947 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:11:16 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T00:11:16.947 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:11:16 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T00:11:16.947 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:11:16 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-24T00:11:16.947 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:11:16 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-24T00:11:16.947 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:11:16 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T00:11:16.948 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:11:16 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T00:11:16.948 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:11:16 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T00:11:16.948 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:11:16 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T00:11:16.948 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:11:16 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T00:11:16.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:11:16 smithi193 ceph-mon[36878]: pgmap v244: 1 pgs: 1 active+clean; 577 KiB data, 287 MiB used, 894 GiB / 894 GiB avail 2024-03-24T00:11:16.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:11:16 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T00:11:16.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:11:16 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T00:11:16.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:11:16 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-24T00:11:16.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:11:16 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-24T00:11:16.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:11:16 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-24T00:11:16.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:11:16 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-03-24T00:11:16.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:11:16 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T00:11:16.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:11:16 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T00:11:16.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:11:16 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T00:11:16.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:11:16 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T00:11:16.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:11:16 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-24T00:11:16.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:11:16 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-24T00:11:16.962 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:11:16 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T00:11:16.962 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:11:16 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T00:11:16.962 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:11:16 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T00:11:16.962 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:11:16 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T00:11:16.962 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:11:16 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T00:11:17.793 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:11:17 smithi189 ceph-mon[36187]: Detected new or changed devices on smithi118 2024-03-24T00:11:17.794 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:11:17 smithi189 ceph-mon[36187]: Adjusting osd_memory_target on smithi118 to 4272M 2024-03-24T00:11:17.794 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:11:17 smithi189 ceph-mon[36187]: from='client.24356 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi189:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T00:11:17.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:11:17 smithi118 ceph-mon[31535]: Detected new or changed devices on smithi118 2024-03-24T00:11:17.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:11:17 smithi118 ceph-mon[31535]: Adjusting osd_memory_target on smithi118 to 4272M 2024-03-24T00:11:17.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:11:17 smithi118 ceph-mon[31535]: from='client.24356 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi189:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T00:11:17.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:11:17 smithi193 ceph-mon[36878]: Detected new or changed devices on smithi118 2024-03-24T00:11:17.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:11:17 smithi193 ceph-mon[36878]: Adjusting osd_memory_target on smithi118 to 4272M 2024-03-24T00:11:17.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:11:17 smithi193 ceph-mon[36878]: from='client.24356 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi189:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T00:11:18.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:11:18 smithi118 ceph-mon[31535]: pgmap v245: 1 pgs: 1 active+clean; 577 KiB data, 287 MiB used, 894 GiB / 894 GiB avail 2024-03-24T00:11:18.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:11:18 smithi189 ceph-mon[36187]: pgmap v245: 1 pgs: 1 active+clean; 577 KiB data, 287 MiB used, 894 GiB / 894 GiB avail 2024-03-24T00:11:18.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:11:18 smithi193 ceph-mon[36878]: pgmap v245: 1 pgs: 1 active+clean; 577 KiB data, 287 MiB used, 894 GiB / 894 GiB avail 2024-03-24T00:11:19.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:11:19 smithi118 ceph-mon[31535]: from='client.? 172.21.15.189:0/2087505681' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "7ba69d69-ac41-44ce-bc39-c049a8ca5900"}]: dispatch 2024-03-24T00:11:19.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:11:19 smithi118 ceph-mon[31535]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "7ba69d69-ac41-44ce-bc39-c049a8ca5900"}]: dispatch 2024-03-24T00:11:19.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:11:19 smithi118 ceph-mon[31535]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "7ba69d69-ac41-44ce-bc39-c049a8ca5900"}]': finished 2024-03-24T00:11:19.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:11:19 smithi118 ceph-mon[31535]: osdmap e70: 11 total, 10 up, 11 in 2024-03-24T00:11:19.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:11:19 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-03-24T00:11:19.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:11:19 smithi118 ceph-mon[31535]: from='client.? 172.21.15.189:0/698390277' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-24T00:11:19.953 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:11:19 smithi189 ceph-mon[36187]: from='client.? 172.21.15.189:0/2087505681' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "7ba69d69-ac41-44ce-bc39-c049a8ca5900"}]: dispatch 2024-03-24T00:11:19.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:11:19 smithi189 ceph-mon[36187]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "7ba69d69-ac41-44ce-bc39-c049a8ca5900"}]: dispatch 2024-03-24T00:11:19.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:11:19 smithi189 ceph-mon[36187]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "7ba69d69-ac41-44ce-bc39-c049a8ca5900"}]': finished 2024-03-24T00:11:19.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:11:19 smithi189 ceph-mon[36187]: osdmap e70: 11 total, 10 up, 11 in 2024-03-24T00:11:19.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:11:19 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-03-24T00:11:19.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:11:19 smithi189 ceph-mon[36187]: from='client.? 172.21.15.189:0/698390277' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-24T00:11:19.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:11:19 smithi193 ceph-mon[36878]: from='client.? 172.21.15.189:0/2087505681' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "7ba69d69-ac41-44ce-bc39-c049a8ca5900"}]: dispatch 2024-03-24T00:11:19.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:11:19 smithi193 ceph-mon[36878]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "7ba69d69-ac41-44ce-bc39-c049a8ca5900"}]: dispatch 2024-03-24T00:11:19.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:11:19 smithi193 ceph-mon[36878]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "7ba69d69-ac41-44ce-bc39-c049a8ca5900"}]': finished 2024-03-24T00:11:19.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:11:19 smithi193 ceph-mon[36878]: osdmap e70: 11 total, 10 up, 11 in 2024-03-24T00:11:19.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:11:19 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-03-24T00:11:19.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:11:19 smithi193 ceph-mon[36878]: from='client.? 172.21.15.189:0/698390277' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-24T00:11:20.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:11:20 smithi118 ceph-mon[31535]: pgmap v247: 1 pgs: 1 active+clean; 577 KiB data, 287 MiB used, 894 GiB / 894 GiB avail 2024-03-24T00:11:20.953 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:11:20 smithi189 ceph-mon[36187]: pgmap v247: 1 pgs: 1 active+clean; 577 KiB data, 287 MiB used, 894 GiB / 894 GiB avail 2024-03-24T00:11:20.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:11:20 smithi193 ceph-mon[36878]: pgmap v247: 1 pgs: 1 active+clean; 577 KiB data, 287 MiB used, 894 GiB / 894 GiB avail 2024-03-24T00:11:22.944 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:11:22 smithi189 ceph-mon[36187]: pgmap v248: 1 pgs: 1 active+clean; 577 KiB data, 287 MiB used, 894 GiB / 894 GiB avail 2024-03-24T00:11:22.945 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:11:22 smithi118 ceph-mon[31535]: pgmap v248: 1 pgs: 1 active+clean; 577 KiB data, 287 MiB used, 894 GiB / 894 GiB avail 2024-03-24T00:11:22.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:11:22 smithi193 ceph-mon[36878]: pgmap v248: 1 pgs: 1 active+clean; 577 KiB data, 287 MiB used, 894 GiB / 894 GiB avail 2024-03-24T00:11:24.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:11:24 smithi118 ceph-mon[31535]: pgmap v249: 1 pgs: 1 active+clean; 577 KiB data, 287 MiB used, 894 GiB / 894 GiB avail 2024-03-24T00:11:24.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:11:24 smithi189 ceph-mon[36187]: pgmap v249: 1 pgs: 1 active+clean; 577 KiB data, 287 MiB used, 894 GiB / 894 GiB avail 2024-03-24T00:11:24.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:11:24 smithi193 ceph-mon[36878]: pgmap v249: 1 pgs: 1 active+clean; 577 KiB data, 287 MiB used, 894 GiB / 894 GiB avail 2024-03-24T00:11:26.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:11:26 smithi118 ceph-mon[31535]: pgmap v250: 1 pgs: 1 active+clean; 577 KiB data, 287 MiB used, 894 GiB / 894 GiB avail 2024-03-24T00:11:26.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:11:26 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.10"}]: dispatch 2024-03-24T00:11:26.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:11:26 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T00:11:26.953 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:11:26 smithi189 ceph-mon[36187]: pgmap v250: 1 pgs: 1 active+clean; 577 KiB data, 287 MiB used, 894 GiB / 894 GiB avail 2024-03-24T00:11:26.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:11:26 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.10"}]: dispatch 2024-03-24T00:11:26.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:11:26 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T00:11:26.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:11:26 smithi193 ceph-mon[36878]: pgmap v250: 1 pgs: 1 active+clean; 577 KiB data, 287 MiB used, 894 GiB / 894 GiB avail 2024-03-24T00:11:26.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:11:26 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.10"}]: dispatch 2024-03-24T00:11:26.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:11:26 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T00:11:27.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:11:27 smithi118 ceph-mon[31535]: Deploying daemon osd.10 on smithi189 2024-03-24T00:11:27.953 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:11:27 smithi189 ceph-mon[36187]: Deploying daemon osd.10 on smithi189 2024-03-24T00:11:27.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:11:27 smithi193 ceph-mon[36878]: Deploying daemon osd.10 on smithi189 2024-03-24T00:11:28.945 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:11:28 smithi118 ceph-mon[31535]: pgmap v251: 1 pgs: 1 active+clean; 577 KiB data, 287 MiB used, 894 GiB / 894 GiB avail 2024-03-24T00:11:28.953 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:11:28 smithi189 ceph-mon[36187]: pgmap v251: 1 pgs: 1 active+clean; 577 KiB data, 287 MiB used, 894 GiB / 894 GiB avail 2024-03-24T00:11:28.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:11:28 smithi193 ceph-mon[36878]: pgmap v251: 1 pgs: 1 active+clean; 577 KiB data, 287 MiB used, 894 GiB / 894 GiB avail 2024-03-24T00:11:30.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:11:30 smithi189 ceph-mon[36187]: pgmap v252: 1 pgs: 1 active+clean; 577 KiB data, 287 MiB used, 894 GiB / 894 GiB avail 2024-03-24T00:11:30.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:11:30 smithi193 ceph-mon[36878]: pgmap v252: 1 pgs: 1 active+clean; 577 KiB data, 287 MiB used, 894 GiB / 894 GiB avail 2024-03-24T00:11:31.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:11:30 smithi118 ceph-mon[31535]: pgmap v252: 1 pgs: 1 active+clean; 577 KiB data, 287 MiB used, 894 GiB / 894 GiB avail 2024-03-24T00:11:32.863 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:11:32 smithi189 ceph-mon[36187]: pgmap v253: 1 pgs: 1 active+clean; 577 KiB data, 287 MiB used, 894 GiB / 894 GiB avail 2024-03-24T00:11:32.864 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:11:32 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T00:11:32.864 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:11:32 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T00:11:32.864 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:11:32 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T00:11:32.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:11:32 smithi193 ceph-mon[36878]: pgmap v253: 1 pgs: 1 active+clean; 577 KiB data, 287 MiB used, 894 GiB / 894 GiB avail 2024-03-24T00:11:32.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:11:32 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T00:11:32.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:11:32 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T00:11:32.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:11:32 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T00:11:33.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:11:32 smithi118 ceph-mon[31535]: pgmap v253: 1 pgs: 1 active+clean; 577 KiB data, 287 MiB used, 894 GiB / 894 GiB avail 2024-03-24T00:11:33.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:11:32 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T00:11:33.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:11:32 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T00:11:33.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:11:32 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T00:11:34.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:11:34 smithi189 ceph-mon[36187]: pgmap v254: 1 pgs: 1 active+clean; 577 KiB data, 287 MiB used, 894 GiB / 894 GiB avail 2024-03-24T00:11:34.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:11:34 smithi193 ceph-mon[36878]: pgmap v254: 1 pgs: 1 active+clean; 577 KiB data, 287 MiB used, 894 GiB / 894 GiB avail 2024-03-24T00:11:35.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:11:34 smithi118 ceph-mon[31535]: pgmap v254: 1 pgs: 1 active+clean; 577 KiB data, 287 MiB used, 894 GiB / 894 GiB avail 2024-03-24T00:11:36.005 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-24T00:11:36.005+0000 7fe07a7fc700 1 -- 172.21.15.189:0/670865682 <== mgr.14152 v2:172.21.15.118:6800/2112912943 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+38 (secure 0 0 0) 0x7fe0840625d0 con 0x7fe0700a2890 2024-03-24T00:11:36.005 INFO:teuthology.orchestra.run.smithi189.stdout:Created osd(s) 10 on host 'smithi189' 2024-03-24T00:11:36.007 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-24T00:11:36.008+0000 7fe08b704700 1 -- 172.21.15.189:0/670865682 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7fe0700a2890 msgr2=0x7fe0700a4d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:11:36.007 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-24T00:11:36.008+0000 7fe08b704700 1 --2- 172.21.15.189:0/670865682 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7fe0700a2890 0x7fe0700a4d50 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7fe074005fd0 tx=0x7fe074005ee0 comp rx=0 tx=0).stop 2024-03-24T00:11:36.008 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-24T00:11:36.008+0000 7fe08b704700 1 -- 172.21.15.189:0/670865682 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe0841021c0 msgr2=0x7fe084198400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:11:36.008 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-24T00:11:36.008+0000 7fe08b704700 1 --2- 172.21.15.189:0/670865682 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe0841021c0 0x7fe084198400 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7fe08000c010 tx=0x7fe080009f10 comp rx=0 tx=0).stop 2024-03-24T00:11:36.008 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-24T00:11:36.008+0000 7fe08b704700 1 -- 172.21.15.189:0/670865682 shutdown_connections 2024-03-24T00:11:36.008 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-24T00:11:36.008+0000 7fe08b704700 1 --2- 172.21.15.189:0/670865682 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe0841021c0 0x7fe084198400 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:11:36.008 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-24T00:11:36.008+0000 7fe08b704700 1 --2- 172.21.15.189:0/670865682 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7fe0700a2890 0x7fe0700a4d50 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:11:36.008 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-24T00:11:36.008+0000 7fe08b704700 1 --2- 172.21.15.189:0/670865682 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fe084103d60 0x7fe08419f9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:11:36.008 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-24T00:11:36.008+0000 7fe08b704700 1 --2- 172.21.15.189:0/670865682 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fe0841033c0 0x7fe084198940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:11:36.008 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-24T00:11:36.008+0000 7fe08b704700 1 -- 172.21.15.189:0/670865682 >> 172.21.15.189:0/670865682 conn(0x7fe0840fd7a0 msgr2=0x7fe0841066e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:11:36.008 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-24T00:11:36.008+0000 7fe08b704700 1 -- 172.21.15.189:0/670865682 shutdown_connections 2024-03-24T00:11:36.008 INFO:teuthology.orchestra.run.smithi189.stderr:2024-03-24T00:11:36.008+0000 7fe08b704700 1 -- 172.21.15.189:0/670865682 wait complete. 2024-03-24T00:11:36.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:11:36 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T00:11:36.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:11:36 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T00:11:36.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:11:36 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T00:11:36.447 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:11:36 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T00:11:36.447 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:11:36 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T00:11:36.447 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:11:36 smithi118 ceph-mon[31535]: pgmap v255: 1 pgs: 1 active+clean; 577 KiB data, 287 MiB used, 894 GiB / 894 GiB avail 2024-03-24T00:11:36.447 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:11:36 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T00:11:36.447 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:11:36 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T00:11:36.447 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:11:36 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T00:11:36.447 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:11:36 smithi118 ceph-mon[31535]: from='osd.10 [v2:172.21.15.189:6824/2112195760,v1:172.21.15.189:6825/2112195760]' entity='osd.10' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]: dispatch 2024-03-24T00:11:36.447 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:11:36 smithi118 ceph-mon[31535]: from='osd.10 ' entity='osd.10' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]: dispatch 2024-03-24T00:11:36.453 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:11:36 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T00:11:36.453 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:11:36 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T00:11:36.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:11:36 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T00:11:36.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:11:36 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T00:11:36.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:11:36 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T00:11:36.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:11:36 smithi189 ceph-mon[36187]: pgmap v255: 1 pgs: 1 active+clean; 577 KiB data, 287 MiB used, 894 GiB / 894 GiB avail 2024-03-24T00:11:36.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:11:36 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T00:11:36.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:11:36 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T00:11:36.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:11:36 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T00:11:36.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:11:36 smithi189 ceph-mon[36187]: from='osd.10 [v2:172.21.15.189:6824/2112195760,v1:172.21.15.189:6825/2112195760]' entity='osd.10' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]: dispatch 2024-03-24T00:11:36.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:11:36 smithi189 ceph-mon[36187]: from='osd.10 ' entity='osd.10' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]: dispatch 2024-03-24T00:11:36.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:11:36 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T00:11:36.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:11:36 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T00:11:36.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:11:36 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T00:11:36.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:11:36 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T00:11:36.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:11:36 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T00:11:36.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:11:36 smithi193 ceph-mon[36878]: pgmap v255: 1 pgs: 1 active+clean; 577 KiB data, 287 MiB used, 894 GiB / 894 GiB avail 2024-03-24T00:11:36.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:11:36 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T00:11:36.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:11:36 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T00:11:36.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:11:36 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T00:11:36.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:11:36 smithi193 ceph-mon[36878]: from='osd.10 [v2:172.21.15.189:6824/2112195760,v1:172.21.15.189:6825/2112195760]' entity='osd.10' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]: dispatch 2024-03-24T00:11:36.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:11:36 smithi193 ceph-mon[36878]: from='osd.10 ' entity='osd.10' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]: dispatch 2024-03-24T00:11:36.644 DEBUG:teuthology.orchestra.run.smithi189:osd.10> sudo journalctl -f -n 0 -u ceph-d79a5420-e971-11ee-95cd-87774f69a715@osd.10.service 2024-03-24T00:11:36.646 INFO:tasks.cephadm:Deploying osd.11 on smithi193 with /dev/vg_nvme/lv_1... 2024-03-24T00:11:36.646 DEBUG:teuthology.orchestra.run.smithi193:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d79a5420-e971-11ee-95cd-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_1 2024-03-24T00:11:36.892 INFO:teuthology.orchestra.run.smithi193.stderr:Inferring config /var/lib/ceph/d79a5420-e971-11ee-95cd-87774f69a715/mon.c/config 2024-03-24T00:11:38.133 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:11:37 smithi189 ceph-mon[36187]: from='osd.10 ' entity='osd.10' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]': finished 2024-03-24T00:11:38.134 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:11:37 smithi189 ceph-mon[36187]: from='osd.10 [v2:172.21.15.189:6824/2112195760,v1:172.21.15.189:6825/2112195760]' entity='osd.10' cmd=[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi189", "root=default"]}]: dispatch 2024-03-24T00:11:38.134 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:11:37 smithi189 ceph-mon[36187]: osdmap e71: 11 total, 10 up, 11 in 2024-03-24T00:11:38.134 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:11:37 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-03-24T00:11:38.134 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:11:37 smithi189 ceph-mon[36187]: from='osd.10 ' entity='osd.10' cmd=[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi189", "root=default"]}]: dispatch 2024-03-24T00:11:38.134 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:11:37 smithi189 ceph-mon[36187]: pgmap v257: 1 pgs: 1 active+clean; 577 KiB data, 287 MiB used, 894 GiB / 894 GiB avail 2024-03-24T00:11:38.134 INFO:journalctl@ceph.osd.10.smithi189.stdout:Mar 24 00:11:37 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-osd-10[61294]: 2024-03-24T00:11:37.981+0000 7f8050d29700 -1 osd.10 0 waiting for initial osdmap 2024-03-24T00:11:38.134 INFO:journalctl@ceph.osd.10.smithi189.stdout:Mar 24 00:11:37 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-osd-10[61294]: 2024-03-24T00:11:37.992+0000 7f804cb1c700 -1 osd.10 72 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-24T00:11:38.292 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:11:37 smithi193 ceph-mon[36878]: from='osd.10 ' entity='osd.10' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]': finished 2024-03-24T00:11:38.292 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:11:37 smithi193 ceph-mon[36878]: from='osd.10 [v2:172.21.15.189:6824/2112195760,v1:172.21.15.189:6825/2112195760]' entity='osd.10' cmd=[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi189", "root=default"]}]: dispatch 2024-03-24T00:11:38.292 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:11:37 smithi193 ceph-mon[36878]: osdmap e71: 11 total, 10 up, 11 in 2024-03-24T00:11:38.293 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:11:37 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-03-24T00:11:38.293 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:11:37 smithi193 ceph-mon[36878]: from='osd.10 ' entity='osd.10' cmd=[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi189", "root=default"]}]: dispatch 2024-03-24T00:11:38.293 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:11:37 smithi193 ceph-mon[36878]: pgmap v257: 1 pgs: 1 active+clean; 577 KiB data, 287 MiB used, 894 GiB / 894 GiB avail 2024-03-24T00:11:38.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:11:37 smithi118 ceph-mon[31535]: from='osd.10 ' entity='osd.10' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["10"]}]': finished 2024-03-24T00:11:38.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:11:37 smithi118 ceph-mon[31535]: from='osd.10 [v2:172.21.15.189:6824/2112195760,v1:172.21.15.189:6825/2112195760]' entity='osd.10' cmd=[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi189", "root=default"]}]: dispatch 2024-03-24T00:11:38.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:11:37 smithi118 ceph-mon[31535]: osdmap e71: 11 total, 10 up, 11 in 2024-03-24T00:11:38.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:11:37 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-03-24T00:11:38.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:11:37 smithi118 ceph-mon[31535]: from='osd.10 ' entity='osd.10' cmd=[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi189", "root=default"]}]: dispatch 2024-03-24T00:11:38.447 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:11:37 smithi118 ceph-mon[31535]: pgmap v257: 1 pgs: 1 active+clean; 577 KiB data, 287 MiB used, 894 GiB / 894 GiB avail 2024-03-24T00:11:38.978 INFO:teuthology.orchestra.run.smithi193.stderr:--> Zapping: /dev/vg_nvme/lv_1 2024-03-24T00:11:38.978 INFO:teuthology.orchestra.run.smithi193.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_1 bs=1M count=10 conv=fsync 2024-03-24T00:11:38.978 INFO:teuthology.orchestra.run.smithi193.stderr: stderr: 10+0 records in 2024-03-24T00:11:38.978 INFO:teuthology.orchestra.run.smithi193.stderr:10+0 records out 2024-03-24T00:11:38.978 INFO:teuthology.orchestra.run.smithi193.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0112502 s, 932 MB/s 2024-03-24T00:11:38.978 INFO:teuthology.orchestra.run.smithi193.stderr:--> Zapping successful for: 2024-03-24T00:11:39.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:11:38 smithi118 ceph-mon[31535]: from='osd.10 ' entity='osd.10' cmd='[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi189", "root=default"]}]': finished 2024-03-24T00:11:39.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:11:38 smithi118 ceph-mon[31535]: osdmap e72: 11 total, 10 up, 11 in 2024-03-24T00:11:39.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:11:38 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-03-24T00:11:39.203 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:11:38 smithi189 ceph-mon[36187]: from='osd.10 ' entity='osd.10' cmd='[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi189", "root=default"]}]': finished 2024-03-24T00:11:39.203 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:11:38 smithi189 ceph-mon[36187]: osdmap e72: 11 total, 10 up, 11 in 2024-03-24T00:11:39.203 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:11:38 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-03-24T00:11:39.282 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:11:38 smithi193 ceph-mon[36878]: from='osd.10 ' entity='osd.10' cmd='[{"prefix": "osd crush create-or-move", "id": 10, "weight":0.0873, "args": ["host=smithi189", "root=default"]}]': finished 2024-03-24T00:11:39.282 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:11:38 smithi193 ceph-mon[36878]: osdmap e72: 11 total, 10 up, 11 in 2024-03-24T00:11:39.283 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:11:38 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-03-24T00:11:39.593 DEBUG:teuthology.orchestra.run.smithi193:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d79a5420-e971-11ee-95cd-87774f69a715 -- ceph orch daemon add osd smithi193:vg_nvme/lv_1 2024-03-24T00:11:39.836 INFO:teuthology.orchestra.run.smithi193.stderr:Inferring config /var/lib/ceph/d79a5420-e971-11ee-95cd-87774f69a715/mon.c/config 2024-03-24T00:11:40.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:11:39 smithi193 ceph-mon[36878]: purged_snaps scrub starts 2024-03-24T00:11:40.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:11:39 smithi193 ceph-mon[36878]: purged_snaps scrub ok 2024-03-24T00:11:40.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:11:39 smithi193 ceph-mon[36878]: osd.10 [v2:172.21.15.189:6824/2112195760,v1:172.21.15.189:6825/2112195760] boot 2024-03-24T00:11:40.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:11:39 smithi193 ceph-mon[36878]: osdmap e73: 11 total, 11 up, 11 in 2024-03-24T00:11:40.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:11:39 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-03-24T00:11:40.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:11:39 smithi193 ceph-mon[36878]: Detected new or changed devices on smithi189 2024-03-24T00:11:40.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:11:39 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T00:11:40.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:11:39 smithi193 ceph-mon[36878]: pgmap v260: 1 pgs: 1 active+clean; 577 KiB data, 314 MiB used, 983 GiB / 983 GiB avail 2024-03-24T00:11:40.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:11:39 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T00:11:40.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:11:39 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-24T00:11:40.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:11:39 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-24T00:11:40.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:11:39 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-24T00:11:40.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:11:39 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-03-24T00:11:40.212 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:11:39 smithi193 ceph-mon[36878]: Adjusting osd_memory_target on smithi189 to 4270M 2024-03-24T00:11:40.212 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:11:39 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T00:11:40.212 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:11:39 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T00:11:40.212 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:11:39 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T00:11:40.212 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:11:39 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T00:11:40.212 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:11:39 smithi193 ceph-mon[36878]: osdmap e74: 11 total, 11 up, 11 in 2024-03-24T00:11:40.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:11:39 smithi118 ceph-mon[31535]: purged_snaps scrub starts 2024-03-24T00:11:40.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:11:39 smithi118 ceph-mon[31535]: purged_snaps scrub ok 2024-03-24T00:11:40.447 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:11:39 smithi118 ceph-mon[31535]: osd.10 [v2:172.21.15.189:6824/2112195760,v1:172.21.15.189:6825/2112195760] boot 2024-03-24T00:11:40.447 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:11:39 smithi118 ceph-mon[31535]: osdmap e73: 11 total, 11 up, 11 in 2024-03-24T00:11:40.447 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:11:39 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-03-24T00:11:40.447 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:11:39 smithi118 ceph-mon[31535]: Detected new or changed devices on smithi189 2024-03-24T00:11:40.447 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:11:39 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T00:11:40.447 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:11:39 smithi118 ceph-mon[31535]: pgmap v260: 1 pgs: 1 active+clean; 577 KiB data, 314 MiB used, 983 GiB / 983 GiB avail 2024-03-24T00:11:40.447 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:11:39 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T00:11:40.447 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:11:39 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-24T00:11:40.447 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:11:39 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-24T00:11:40.447 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:11:39 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-24T00:11:40.447 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:11:39 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-03-24T00:11:40.447 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:11:39 smithi118 ceph-mon[31535]: Adjusting osd_memory_target on smithi189 to 4270M 2024-03-24T00:11:40.447 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:11:39 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T00:11:40.447 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:11:39 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T00:11:40.447 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:11:39 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T00:11:40.447 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:11:39 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T00:11:40.448 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:11:39 smithi118 ceph-mon[31535]: osdmap e74: 11 total, 11 up, 11 in 2024-03-24T00:11:40.453 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:11:39 smithi189 ceph-mon[36187]: purged_snaps scrub starts 2024-03-24T00:11:40.453 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:11:39 smithi189 ceph-mon[36187]: purged_snaps scrub ok 2024-03-24T00:11:40.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:11:39 smithi189 ceph-mon[36187]: osd.10 [v2:172.21.15.189:6824/2112195760,v1:172.21.15.189:6825/2112195760] boot 2024-03-24T00:11:40.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:11:39 smithi189 ceph-mon[36187]: osdmap e73: 11 total, 11 up, 11 in 2024-03-24T00:11:40.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:11:39 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 10}]: dispatch 2024-03-24T00:11:40.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:11:39 smithi189 ceph-mon[36187]: Detected new or changed devices on smithi189 2024-03-24T00:11:40.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:11:39 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T00:11:40.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:11:39 smithi189 ceph-mon[36187]: pgmap v260: 1 pgs: 1 active+clean; 577 KiB data, 314 MiB used, 983 GiB / 983 GiB avail 2024-03-24T00:11:40.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:11:39 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T00:11:40.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:11:39 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-24T00:11:40.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:11:39 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-24T00:11:40.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:11:39 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-24T00:11:40.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:11:39 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-03-24T00:11:40.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:11:39 smithi189 ceph-mon[36187]: Adjusting osd_memory_target on smithi189 to 4270M 2024-03-24T00:11:40.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:11:39 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T00:11:40.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:11:39 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T00:11:40.455 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:11:39 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T00:11:40.455 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:11:39 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T00:11:40.455 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:11:39 smithi189 ceph-mon[36187]: osdmap e74: 11 total, 11 up, 11 in 2024-03-24T00:11:41.466 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-24T00:11:41.459+0000 7fe19c8d3700 1 -- 172.21.15.193:0/2658397645 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fe198101130 msgr2=0x7fe198101550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:11:41.466 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-24T00:11:41.459+0000 7fe19c8d3700 1 --2- 172.21.15.193:0/2658397645 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fe198101130 0x7fe198101550 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7fe188009990 tx=0x7fe188009ca0 comp rx=0 tx=0).stop 2024-03-24T00:11:41.467 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-24T00:11:41.460+0000 7fe19c8d3700 1 -- 172.21.15.193:0/2658397645 shutdown_connections 2024-03-24T00:11:41.467 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-24T00:11:41.460+0000 7fe19c8d3700 1 --2- 172.21.15.193:0/2658397645 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fe198102cd0 0x7fe198109580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:11:41.467 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-24T00:11:41.460+0000 7fe19c8d3700 1 --2- 172.21.15.193:0/2658397645 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fe198101130 0x7fe198101550 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:11:41.467 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-24T00:11:41.460+0000 7fe19c8d3700 1 --2- 172.21.15.193:0/2658397645 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe198102330 0x7fe198102790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:11:41.467 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-24T00:11:41.460+0000 7fe19c8d3700 1 -- 172.21.15.193:0/2658397645 >> 172.21.15.193:0/2658397645 conn(0x7fe1980fc6f0 msgr2=0x7fe1980feb10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:11:41.467 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-24T00:11:41.460+0000 7fe19c8d3700 1 -- 172.21.15.193:0/2658397645 shutdown_connections 2024-03-24T00:11:41.467 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-24T00:11:41.460+0000 7fe19c8d3700 1 -- 172.21.15.193:0/2658397645 wait complete. 2024-03-24T00:11:41.467 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-24T00:11:41.460+0000 7fe19c8d3700 1 Processor -- start 2024-03-24T00:11:41.467 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-24T00:11:41.460+0000 7fe19c8d3700 1 -- start start 2024-03-24T00:11:41.467 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-24T00:11:41.460+0000 7fe19c8d3700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fe198101130 0x7fe19806b3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:11:41.467 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-24T00:11:41.460+0000 7fe19c8d3700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe198102330 0x7fe19819d2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:11:41.467 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-24T00:11:41.460+0000 7fe19c8d3700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fe198102cd0 0x7fe19819f8e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:11:41.467 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-24T00:11:41.461+0000 7fe19c8d3700 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe19810cd80 con 0x7fe198102cd0 2024-03-24T00:11:41.467 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-24T00:11:41.461+0000 7fe19c8d3700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe19810cc00 con 0x7fe198102330 2024-03-24T00:11:41.468 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-24T00:11:41.461+0000 7fe19c8d3700 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe19810cf00 con 0x7fe198101130 2024-03-24T00:11:41.468 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-24T00:11:41.461+0000 7fe19659c700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fe198101130 0x7fe19806b3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:11:41.468 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-24T00:11:41.461+0000 7fe19659c700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fe198101130 0x7fe19806b3c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.193:55794/0 (socket says 172.21.15.193:55794) 2024-03-24T00:11:41.468 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-24T00:11:41.461+0000 7fe19659c700 1 -- 172.21.15.193:0/1205880611 learned_addr learned my addr 172.21.15.193:0/1205880611 (peer_addr_for_me v2:172.21.15.193:0/0) 2024-03-24T00:11:41.468 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-24T00:11:41.461+0000 7fe19659c700 1 -- 172.21.15.193:0/1205880611 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe198102330 msgr2=0x7fe19819d2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:11:41.468 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-24T00:11:41.461+0000 7fe19659c700 1 --2- 172.21.15.193:0/1205880611 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe198102330 0x7fe19819d2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:11:41.468 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-24T00:11:41.461+0000 7fe19659c700 1 -- 172.21.15.193:0/1205880611 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fe198102cd0 msgr2=0x7fe19819f8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:11:41.468 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-24T00:11:41.461+0000 7fe19659c700 1 --2- 172.21.15.193:0/1205880611 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fe198102cd0 0x7fe19819f8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:11:41.468 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-24T00:11:41.461+0000 7fe19659c700 1 -- 172.21.15.193:0/1205880611 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe1981a01b0 con 0x7fe198101130 2024-03-24T00:11:41.468 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-24T00:11:41.461+0000 7fe19659c700 1 --2- 172.21.15.193:0/1205880611 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fe198101130 0x7fe19806b3c0 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7fe188000c00 tx=0x7fe188009f90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:11:41.468 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-24T00:11:41.461+0000 7fe1877fe700 1 -- 172.21.15.193:0/1205880611 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe188027070 con 0x7fe198101130 2024-03-24T00:11:41.468 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-24T00:11:41.461+0000 7fe19c8d3700 1 -- 172.21.15.193:0/1205880611 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe1981a0440 con 0x7fe198101130 2024-03-24T00:11:41.468 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-24T00:11:41.461+0000 7fe1877fe700 1 -- 172.21.15.193:0/1205880611 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fe188022470 con 0x7fe198101130 2024-03-24T00:11:41.468 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-24T00:11:41.461+0000 7fe1877fe700 1 -- 172.21.15.193:0/1205880611 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe18800f6c0 con 0x7fe198101130 2024-03-24T00:11:41.468 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-24T00:11:41.461+0000 7fe19c8d3700 1 -- 172.21.15.193:0/1205880611 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe1981a0970 con 0x7fe198101130 2024-03-24T00:11:41.469 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-24T00:11:41.462+0000 7fe19c8d3700 1 -- 172.21.15.193:0/1205880611 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe198068e50 con 0x7fe198101130 2024-03-24T00:11:41.469 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-24T00:11:41.463+0000 7fe1877fe700 1 -- 172.21.15.193:0/1205880611 <== mon.1 v2:172.21.15.193:3300/0 4 ==== mgrmap(e 16) v1 ==== 137291+0+0 (secure 0 0 0) 0x7fe18802a070 con 0x7fe198101130 2024-03-24T00:11:41.469 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-24T00:11:41.464+0000 7fe1877fe700 1 --2- 172.21.15.193:0/1205880611 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7fe1800a2890 0x7fe1800a4d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:11:41.469 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-24T00:11:41.464+0000 7fe1877fe700 1 -- 172.21.15.193:0/1205880611 <== mon.1 v2:172.21.15.193:3300/0 5 ==== osd_map(75..75 src has 1..75) v4 ==== 6723+0+0 (secure 0 0 0) 0x7fe1880cf830 con 0x7fe198101130 2024-03-24T00:11:41.469 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-24T00:11:41.464+0000 7fe195d9b700 1 --2- 172.21.15.193:0/1205880611 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7fe1800a2890 0x7fe1800a4d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:11:41.469 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-24T00:11:41.465+0000 7fe195d9b700 1 --2- 172.21.15.193:0/1205880611 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7fe1800a2890 0x7fe1800a4d50 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7fe18c0097b0 tx=0x7fe18c006d20 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:11:41.469 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-24T00:11:41.467+0000 7fe1877fe700 1 -- 172.21.15.193:0/1205880611 <== mon.1 v2:172.21.15.193:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7fe188013070 con 0x7fe198101130 2024-03-24T00:11:41.670 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-24T00:11:41.670+0000 7fe19c8d3700 1 -- 172.21.15.193:0/1205880611 --> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi193:vg_nvme/lv_1", "target": ["mon-mgr", ""]}) v1 -- 0x7fe1980625d0 con 0x7fe1800a2890 2024-03-24T00:11:42.301 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:11:41 smithi193 ceph-mon[36878]: osdmap e75: 11 total, 11 up, 11 in 2024-03-24T00:11:42.301 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:11:41 smithi193 ceph-mon[36878]: pgmap v263: 1 pgs: 1 active+clean; 577 KiB data, 314 MiB used, 983 GiB / 983 GiB avail 2024-03-24T00:11:42.301 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:11:41 smithi193 ceph-mon[36878]: from='client.24370 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi193:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T00:11:42.301 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:11:41 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-24T00:11:42.301 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:11:41 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-24T00:11:42.301 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:11:41 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T00:11:42.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:11:41 smithi118 ceph-mon[31535]: osdmap e75: 11 total, 11 up, 11 in 2024-03-24T00:11:42.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:11:41 smithi118 ceph-mon[31535]: pgmap v263: 1 pgs: 1 active+clean; 577 KiB data, 314 MiB used, 983 GiB / 983 GiB avail 2024-03-24T00:11:42.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:11:41 smithi118 ceph-mon[31535]: from='client.24370 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi193:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T00:11:42.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:11:41 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-24T00:11:42.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:11:41 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-24T00:11:42.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:11:41 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T00:11:42.453 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:11:41 smithi189 ceph-mon[36187]: osdmap e75: 11 total, 11 up, 11 in 2024-03-24T00:11:42.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:11:41 smithi189 ceph-mon[36187]: pgmap v263: 1 pgs: 1 active+clean; 577 KiB data, 314 MiB used, 983 GiB / 983 GiB avail 2024-03-24T00:11:42.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:11:41 smithi189 ceph-mon[36187]: from='client.24370 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi193:vg_nvme/lv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T00:11:42.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:11:41 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-24T00:11:42.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:11:41 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-24T00:11:42.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:11:41 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T00:11:43.241 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:11:43 smithi193 ceph-mon[36878]: osdmap e76: 11 total, 11 up, 11 in 2024-03-24T00:11:43.445 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:11:43 smithi118 ceph-mon[31535]: osdmap e76: 11 total, 11 up, 11 in 2024-03-24T00:11:43.453 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:11:43 smithi189 ceph-mon[36187]: osdmap e76: 11 total, 11 up, 11 in 2024-03-24T00:11:44.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:11:44 smithi193 ceph-mon[36878]: osdmap e77: 11 total, 11 up, 11 in 2024-03-24T00:11:44.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:11:44 smithi193 ceph-mon[36878]: pgmap v266: 1 pgs: 1 peering; 577 KiB data, 315 MiB used, 983 GiB / 983 GiB avail; 144 KiB/s, 0 objects/s recovering 2024-03-24T00:11:44.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:11:44 smithi118 ceph-mon[31535]: osdmap e77: 11 total, 11 up, 11 in 2024-03-24T00:11:44.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:11:44 smithi118 ceph-mon[31535]: pgmap v266: 1 pgs: 1 peering; 577 KiB data, 315 MiB used, 983 GiB / 983 GiB avail; 144 KiB/s, 0 objects/s recovering 2024-03-24T00:11:44.453 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:11:44 smithi189 ceph-mon[36187]: osdmap e77: 11 total, 11 up, 11 in 2024-03-24T00:11:44.453 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:11:44 smithi189 ceph-mon[36187]: pgmap v266: 1 pgs: 1 peering; 577 KiB data, 315 MiB used, 983 GiB / 983 GiB avail; 144 KiB/s, 0 objects/s recovering 2024-03-24T00:11:45.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:11:45 smithi118 ceph-mon[31535]: from='client.? 172.21.15.193:0/3123722591' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "186f0b18-6b22-4eae-83b7-b21bbcce0817"}]: dispatch 2024-03-24T00:11:45.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:11:45 smithi118 ceph-mon[31535]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "186f0b18-6b22-4eae-83b7-b21bbcce0817"}]: dispatch 2024-03-24T00:11:45.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:11:45 smithi118 ceph-mon[31535]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "186f0b18-6b22-4eae-83b7-b21bbcce0817"}]': finished 2024-03-24T00:11:45.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:11:45 smithi118 ceph-mon[31535]: osdmap e78: 12 total, 11 up, 12 in 2024-03-24T00:11:45.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:11:45 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-03-24T00:11:45.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:11:45 smithi118 ceph-mon[31535]: from='client.? 172.21.15.193:0/1298207734' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-24T00:11:45.453 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:11:45 smithi189 ceph-mon[36187]: from='client.? 172.21.15.193:0/3123722591' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "186f0b18-6b22-4eae-83b7-b21bbcce0817"}]: dispatch 2024-03-24T00:11:45.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:11:45 smithi189 ceph-mon[36187]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "186f0b18-6b22-4eae-83b7-b21bbcce0817"}]: dispatch 2024-03-24T00:11:45.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:11:45 smithi189 ceph-mon[36187]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "186f0b18-6b22-4eae-83b7-b21bbcce0817"}]': finished 2024-03-24T00:11:45.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:11:45 smithi189 ceph-mon[36187]: osdmap e78: 12 total, 11 up, 12 in 2024-03-24T00:11:45.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:11:45 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-03-24T00:11:45.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:11:45 smithi189 ceph-mon[36187]: from='client.? 172.21.15.193:0/1298207734' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-24T00:11:45.460 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:11:45 smithi193 ceph-mon[36878]: from='client.? 172.21.15.193:0/3123722591' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "186f0b18-6b22-4eae-83b7-b21bbcce0817"}]: dispatch 2024-03-24T00:11:45.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:11:45 smithi193 ceph-mon[36878]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "186f0b18-6b22-4eae-83b7-b21bbcce0817"}]: dispatch 2024-03-24T00:11:45.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:11:45 smithi193 ceph-mon[36878]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "186f0b18-6b22-4eae-83b7-b21bbcce0817"}]': finished 2024-03-24T00:11:45.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:11:45 smithi193 ceph-mon[36878]: osdmap e78: 12 total, 11 up, 12 in 2024-03-24T00:11:45.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:11:45 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-03-24T00:11:45.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:11:45 smithi193 ceph-mon[36878]: from='client.? 172.21.15.193:0/1298207734' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-24T00:11:46.445 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:11:46 smithi118 ceph-mon[31535]: pgmap v268: 1 pgs: 1 peering; 577 KiB data, 315 MiB used, 983 GiB / 983 GiB avail; 126 KiB/s, 0 objects/s recovering 2024-03-24T00:11:46.453 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:11:46 smithi189 ceph-mon[36187]: pgmap v268: 1 pgs: 1 peering; 577 KiB data, 315 MiB used, 983 GiB / 983 GiB avail; 126 KiB/s, 0 objects/s recovering 2024-03-24T00:11:46.460 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:11:46 smithi193 ceph-mon[36878]: pgmap v268: 1 pgs: 1 peering; 577 KiB data, 315 MiB used, 983 GiB / 983 GiB avail; 126 KiB/s, 0 objects/s recovering 2024-03-24T00:11:48.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:11:48 smithi118 ceph-mon[31535]: pgmap v269: 1 pgs: 1 peering; 577 KiB data, 315 MiB used, 983 GiB / 983 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-03-24T00:11:48.953 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:11:48 smithi189 ceph-mon[36187]: pgmap v269: 1 pgs: 1 peering; 577 KiB data, 315 MiB used, 983 GiB / 983 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-03-24T00:11:48.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:11:48 smithi193 ceph-mon[36878]: pgmap v269: 1 pgs: 1 peering; 577 KiB data, 315 MiB used, 983 GiB / 983 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-03-24T00:11:50.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:11:50 smithi118 ceph-mon[31535]: pgmap v270: 1 pgs: 1 active+clean; 577 KiB data, 315 MiB used, 983 GiB / 983 GiB avail; 76 KiB/s, 0 objects/s recovering 2024-03-24T00:11:50.953 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:11:50 smithi189 ceph-mon[36187]: pgmap v270: 1 pgs: 1 active+clean; 577 KiB data, 315 MiB used, 983 GiB / 983 GiB avail; 76 KiB/s, 0 objects/s recovering 2024-03-24T00:11:50.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:11:50 smithi193 ceph-mon[36878]: pgmap v270: 1 pgs: 1 active+clean; 577 KiB data, 315 MiB used, 983 GiB / 983 GiB avail; 76 KiB/s, 0 objects/s recovering 2024-03-24T00:11:52.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:11:52 smithi118 ceph-mon[31535]: pgmap v271: 1 pgs: 1 active+clean; 577 KiB data, 315 MiB used, 983 GiB / 983 GiB avail 2024-03-24T00:11:52.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:11:52 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.11"}]: dispatch 2024-03-24T00:11:52.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:11:52 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T00:11:52.953 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:11:52 smithi189 ceph-mon[36187]: pgmap v271: 1 pgs: 1 active+clean; 577 KiB data, 315 MiB used, 983 GiB / 983 GiB avail 2024-03-24T00:11:52.953 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:11:52 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.11"}]: dispatch 2024-03-24T00:11:52.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:11:52 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T00:11:52.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:11:52 smithi193 ceph-mon[36878]: pgmap v271: 1 pgs: 1 active+clean; 577 KiB data, 315 MiB used, 983 GiB / 983 GiB avail 2024-03-24T00:11:52.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:11:52 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "osd.11"}]: dispatch 2024-03-24T00:11:52.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:11:52 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T00:11:53.710 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:11:53 smithi193 ceph-mon[36878]: Deploying daemon osd.11 on smithi193 2024-03-24T00:11:53.945 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:11:53 smithi118 ceph-mon[31535]: Deploying daemon osd.11 on smithi193 2024-03-24T00:11:53.953 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:11:53 smithi189 ceph-mon[36187]: Deploying daemon osd.11 on smithi193 2024-03-24T00:11:54.711 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:11:54 smithi193 ceph-mon[36878]: pgmap v272: 1 pgs: 1 active+clean; 577 KiB data, 315 MiB used, 983 GiB / 983 GiB avail 2024-03-24T00:11:54.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:11:54 smithi118 ceph-mon[31535]: pgmap v272: 1 pgs: 1 active+clean; 577 KiB data, 315 MiB used, 983 GiB / 983 GiB avail 2024-03-24T00:11:54.953 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:11:54 smithi189 ceph-mon[36187]: pgmap v272: 1 pgs: 1 active+clean; 577 KiB data, 315 MiB used, 983 GiB / 983 GiB avail 2024-03-24T00:11:56.831 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:11:56 smithi193 ceph-mon[36878]: pgmap v273: 1 pgs: 1 active+clean; 577 KiB data, 315 MiB used, 983 GiB / 983 GiB avail 2024-03-24T00:11:56.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:11:56 smithi118 ceph-mon[31535]: pgmap v273: 1 pgs: 1 active+clean; 577 KiB data, 315 MiB used, 983 GiB / 983 GiB avail 2024-03-24T00:11:56.953 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:11:56 smithi189 ceph-mon[36187]: pgmap v273: 1 pgs: 1 active+clean; 577 KiB data, 315 MiB used, 983 GiB / 983 GiB avail 2024-03-24T00:11:58.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:11:58 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T00:11:58.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:11:58 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T00:11:58.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:11:58 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T00:11:58.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:11:58 smithi118 ceph-mon[31535]: pgmap v274: 1 pgs: 1 active+clean; 577 KiB data, 315 MiB used, 983 GiB / 983 GiB avail 2024-03-24T00:11:58.710 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:11:58 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T00:11:58.710 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:11:58 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T00:11:58.711 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:11:58 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T00:11:58.711 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:11:58 smithi193 ceph-mon[36878]: pgmap v274: 1 pgs: 1 active+clean; 577 KiB data, 315 MiB used, 983 GiB / 983 GiB avail 2024-03-24T00:11:58.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:11:58 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T00:11:58.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:11:58 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T00:11:58.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:11:58 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T00:11:58.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:11:58 smithi189 ceph-mon[36187]: pgmap v274: 1 pgs: 1 active+clean; 577 KiB data, 315 MiB used, 983 GiB / 983 GiB avail 2024-03-24T00:12:00.751 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-24T00:12:00.751+0000 7fe1877fe700 1 -- 172.21.15.193:0/1205880611 <== mgr.14152 v2:172.21.15.118:6800/2112912943 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+38 (secure 0 0 0) 0x7fe1980625d0 con 0x7fe1800a2890 2024-03-24T00:12:00.752 INFO:teuthology.orchestra.run.smithi193.stdout:Created osd(s) 11 on host 'smithi193' 2024-03-24T00:12:00.754 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-24T00:12:00.754+0000 7fe19c8d3700 1 -- 172.21.15.193:0/1205880611 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7fe1800a2890 msgr2=0x7fe1800a4d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:12:00.754 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-24T00:12:00.754+0000 7fe19c8d3700 1 --2- 172.21.15.193:0/1205880611 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7fe1800a2890 0x7fe1800a4d50 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7fe18c0097b0 tx=0x7fe18c006d20 comp rx=0 tx=0).stop 2024-03-24T00:12:00.754 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-24T00:12:00.754+0000 7fe19c8d3700 1 -- 172.21.15.193:0/1205880611 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fe198101130 msgr2=0x7fe19806b3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:12:00.754 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-24T00:12:00.754+0000 7fe19c8d3700 1 --2- 172.21.15.193:0/1205880611 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fe198101130 0x7fe19806b3c0 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7fe188000c00 tx=0x7fe188009f90 comp rx=0 tx=0).stop 2024-03-24T00:12:00.754 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-24T00:12:00.754+0000 7fe19c8d3700 1 -- 172.21.15.193:0/1205880611 shutdown_connections 2024-03-24T00:12:00.754 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-24T00:12:00.754+0000 7fe19c8d3700 1 --2- 172.21.15.193:0/1205880611 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7fe1800a2890 0x7fe1800a4d50 unknown :-1 s=CLOSED pgs=111 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:12:00.754 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-24T00:12:00.754+0000 7fe19c8d3700 1 --2- 172.21.15.193:0/1205880611 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fe198102cd0 0x7fe19819f8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:12:00.754 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-24T00:12:00.754+0000 7fe19c8d3700 1 --2- 172.21.15.193:0/1205880611 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fe198101130 0x7fe19806b3c0 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:12:00.755 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-24T00:12:00.754+0000 7fe19c8d3700 1 --2- 172.21.15.193:0/1205880611 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe198102330 0x7fe19819d2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:12:00.755 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-24T00:12:00.754+0000 7fe19c8d3700 1 -- 172.21.15.193:0/1205880611 >> 172.21.15.193:0/1205880611 conn(0x7fe1980fc6f0 msgr2=0x7fe198105750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:12:00.755 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-24T00:12:00.754+0000 7fe19c8d3700 1 -- 172.21.15.193:0/1205880611 shutdown_connections 2024-03-24T00:12:00.755 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-24T00:12:00.754+0000 7fe19c8d3700 1 -- 172.21.15.193:0/1205880611 wait complete. 2024-03-24T00:12:00.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:12:00 smithi118 ceph-mon[31535]: pgmap v275: 1 pgs: 1 active+clean; 577 KiB data, 315 MiB used, 983 GiB / 983 GiB avail 2024-03-24T00:12:00.953 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:12:00 smithi189 ceph-mon[36187]: pgmap v275: 1 pgs: 1 active+clean; 577 KiB data, 315 MiB used, 983 GiB / 983 GiB avail 2024-03-24T00:12:00.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:12:00 smithi193 ceph-mon[36878]: pgmap v275: 1 pgs: 1 active+clean; 577 KiB data, 315 MiB used, 983 GiB / 983 GiB avail 2024-03-24T00:12:01.734 DEBUG:teuthology.orchestra.run.smithi193:osd.11> sudo journalctl -f -n 0 -u ceph-d79a5420-e971-11ee-95cd-87774f69a715@osd.11.service 2024-03-24T00:12:01.737 INFO:tasks.cephadm:Waiting for 12 OSDs to come up... 2024-03-24T00:12:01.737 DEBUG:teuthology.orchestra.run.smithi118:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d79a5420-e971-11ee-95cd-87774f69a715 -- ceph osd stat -f json 2024-03-24T00:12:01.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:12:01 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T00:12:01.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:12:01 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T00:12:01.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:12:01 smithi193 ceph-mon[36878]: from='osd.11 [v2:172.21.15.193:6824/3880515510,v1:172.21.15.193:6825/3880515510]' entity='osd.11' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]: dispatch 2024-03-24T00:12:01.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:12:01 smithi193 ceph-mon[36878]: from='osd.11 ' entity='osd.11' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]: dispatch 2024-03-24T00:12:02.183 INFO:teuthology.orchestra.run.smithi118.stderr:Inferring config /var/lib/ceph/d79a5420-e971-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T00:12:02.195 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:12:01 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T00:12:02.195 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:12:01 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T00:12:02.195 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:12:01 smithi118 ceph-mon[31535]: from='osd.11 [v2:172.21.15.193:6824/3880515510,v1:172.21.15.193:6825/3880515510]' entity='osd.11' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]: dispatch 2024-03-24T00:12:02.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:12:01 smithi118 ceph-mon[31535]: from='osd.11 ' entity='osd.11' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]: dispatch 2024-03-24T00:12:02.203 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:12:01 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T00:12:02.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:12:01 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T00:12:02.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:12:01 smithi189 ceph-mon[36187]: from='osd.11 [v2:172.21.15.193:6824/3880515510,v1:172.21.15.193:6825/3880515510]' entity='osd.11' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]: dispatch 2024-03-24T00:12:02.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:12:01 smithi189 ceph-mon[36187]: from='osd.11 ' entity='osd.11' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]: dispatch 2024-03-24T00:12:02.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:12:02 smithi118 ceph-mon[31535]: pgmap v276: 1 pgs: 1 active+clean; 577 KiB data, 315 MiB used, 983 GiB / 983 GiB avail 2024-03-24T00:12:02.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:12:02 smithi118 ceph-mon[31535]: from='osd.11 ' entity='osd.11' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]': finished 2024-03-24T00:12:02.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:12:02 smithi118 ceph-mon[31535]: from='osd.11 [v2:172.21.15.193:6824/3880515510,v1:172.21.15.193:6825/3880515510]' entity='osd.11' cmd=[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi193", "root=default"]}]: dispatch 2024-03-24T00:12:02.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:12:02 smithi118 ceph-mon[31535]: osdmap e79: 12 total, 11 up, 12 in 2024-03-24T00:12:02.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:12:02 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-03-24T00:12:02.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:12:02 smithi118 ceph-mon[31535]: from='osd.11 ' entity='osd.11' cmd=[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi193", "root=default"]}]: dispatch 2024-03-24T00:12:02.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:12:02 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T00:12:02.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:12:02 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T00:12:02.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:12:02 smithi193 ceph-mon[36878]: pgmap v276: 1 pgs: 1 active+clean; 577 KiB data, 315 MiB used, 983 GiB / 983 GiB avail 2024-03-24T00:12:02.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:12:02 smithi193 ceph-mon[36878]: from='osd.11 ' entity='osd.11' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]': finished 2024-03-24T00:12:02.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:12:02 smithi193 ceph-mon[36878]: from='osd.11 [v2:172.21.15.193:6824/3880515510,v1:172.21.15.193:6825/3880515510]' entity='osd.11' cmd=[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi193", "root=default"]}]: dispatch 2024-03-24T00:12:02.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:12:02 smithi193 ceph-mon[36878]: osdmap e79: 12 total, 11 up, 12 in 2024-03-24T00:12:02.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:12:02 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-03-24T00:12:02.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:12:02 smithi193 ceph-mon[36878]: from='osd.11 ' entity='osd.11' cmd=[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi193", "root=default"]}]: dispatch 2024-03-24T00:12:02.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:12:02 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T00:12:02.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:12:02 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T00:12:02.961 INFO:journalctl@ceph.osd.11.smithi193.stdout:Mar 24 00:12:02 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-osd-11[62452]: 2024-03-24T00:12:02.816+0000 7f0edff2b700 -1 osd.11 0 waiting for initial osdmap 2024-03-24T00:12:02.961 INFO:journalctl@ceph.osd.11.smithi193.stdout:Mar 24 00:12:02 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-osd-11[62452]: 2024-03-24T00:12:02.828+0000 7f0edbd1e700 -1 osd.11 80 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-24T00:12:03.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:12:02 smithi189 ceph-mon[36187]: pgmap v276: 1 pgs: 1 active+clean; 577 KiB data, 315 MiB used, 983 GiB / 983 GiB avail 2024-03-24T00:12:03.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:12:02 smithi189 ceph-mon[36187]: from='osd.11 ' entity='osd.11' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["11"]}]': finished 2024-03-24T00:12:03.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:12:02 smithi189 ceph-mon[36187]: from='osd.11 [v2:172.21.15.193:6824/3880515510,v1:172.21.15.193:6825/3880515510]' entity='osd.11' cmd=[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi193", "root=default"]}]: dispatch 2024-03-24T00:12:03.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:12:02 smithi189 ceph-mon[36187]: osdmap e79: 12 total, 11 up, 12 in 2024-03-24T00:12:03.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:12:02 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-03-24T00:12:03.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:12:02 smithi189 ceph-mon[36187]: from='osd.11 ' entity='osd.11' cmd=[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi193", "root=default"]}]: dispatch 2024-03-24T00:12:03.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:12:02 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T00:12:03.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:12:02 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T00:12:03.877 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:12:03 smithi193 ceph-mon[36878]: from='osd.11 ' entity='osd.11' cmd='[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi193", "root=default"]}]': finished 2024-03-24T00:12:03.878 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:12:03 smithi193 ceph-mon[36878]: osdmap e80: 12 total, 11 up, 12 in 2024-03-24T00:12:03.878 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:12:03 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-03-24T00:12:03.878 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:12:03 smithi193 ceph-mon[36878]: pgmap v279: 1 pgs: 1 active+clean; 577 KiB data, 315 MiB used, 983 GiB / 983 GiB avail 2024-03-24T00:12:04.101 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:12:03 smithi118 ceph-mon[31535]: from='osd.11 ' entity='osd.11' cmd='[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi193", "root=default"]}]': finished 2024-03-24T00:12:04.101 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:12:03 smithi118 ceph-mon[31535]: osdmap e80: 12 total, 11 up, 12 in 2024-03-24T00:12:04.101 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:12:03 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-03-24T00:12:04.101 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:12:03 smithi118 ceph-mon[31535]: pgmap v279: 1 pgs: 1 active+clean; 577 KiB data, 315 MiB used, 983 GiB / 983 GiB avail 2024-03-24T00:12:04.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:12:03 smithi189 ceph-mon[36187]: from='osd.11 ' entity='osd.11' cmd='[{"prefix": "osd crush create-or-move", "id": 11, "weight":0.0873, "args": ["host=smithi193", "root=default"]}]': finished 2024-03-24T00:12:04.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:12:03 smithi189 ceph-mon[36187]: osdmap e80: 12 total, 11 up, 12 in 2024-03-24T00:12:04.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:12:03 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-03-24T00:12:04.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:12:03 smithi189 ceph-mon[36187]: pgmap v279: 1 pgs: 1 active+clean; 577 KiB data, 315 MiB used, 983 GiB / 983 GiB avail 2024-03-24T00:12:04.412 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:04.412+0000 7f1e21600700 1 -- 172.21.15.118:0/2794407643 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f1e1c104410 msgr2=0x7f1e1c104830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:12:04.412 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:04.412+0000 7f1e21600700 1 --2- 172.21.15.118:0/2794407643 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f1e1c104410 0x7f1e1c104830 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f1e0c0099c0 tx=0x7f1e0c009cd0 comp rx=0 tx=0).stop 2024-03-24T00:12:04.413 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:04.412+0000 7f1e21600700 1 -- 172.21.15.118:0/2794407643 shutdown_connections 2024-03-24T00:12:04.413 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:04.412+0000 7f1e21600700 1 --2- 172.21.15.118:0/2794407643 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1e1c105fb0 0x7f1e1c10c860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:12:04.413 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:04.412+0000 7f1e21600700 1 --2- 172.21.15.118:0/2794407643 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f1e1c104410 0x7f1e1c104830 unknown :-1 s=CLOSED pgs=130 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:12:04.413 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:04.412+0000 7f1e21600700 1 --2- 172.21.15.118:0/2794407643 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f1e1c105610 0x7f1e1c105a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:12:04.413 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:04.412+0000 7f1e21600700 1 -- 172.21.15.118:0/2794407643 >> 172.21.15.118:0/2794407643 conn(0x7f1e1c0ff9b0 msgr2=0x7f1e1c101df0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:12:04.413 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:04.412+0000 7f1e21600700 1 -- 172.21.15.118:0/2794407643 shutdown_connections 2024-03-24T00:12:04.413 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:04.412+0000 7f1e21600700 1 -- 172.21.15.118:0/2794407643 wait complete. 2024-03-24T00:12:04.415 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:04.413+0000 7f1e21600700 1 Processor -- start 2024-03-24T00:12:04.416 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:04.413+0000 7f1e21600700 1 -- start start 2024-03-24T00:12:04.416 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:04.414+0000 7f1e21600700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f1e1c104410 0x7f1e1c19a5f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:12:04.416 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:04.414+0000 7f1e21600700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1e1c105610 0x7f1e1c19ab30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:12:04.416 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:04.414+0000 7f1e21600700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f1e1c105fb0 0x7f1e1c1a1bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:12:04.416 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:04.414+0000 7f1e21600700 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1e1c10ffd0 con 0x7f1e1c104410 2024-03-24T00:12:04.416 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:04.414+0000 7f1e21600700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1e1c10fe50 con 0x7f1e1c105610 2024-03-24T00:12:04.416 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:04.414+0000 7f1e21600700 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1e1c110150 con 0x7f1e1c105fb0 2024-03-24T00:12:04.416 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:04.414+0000 7f1e1aa00700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f1e1c104410 0x7f1e1c19a5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:12:04.416 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:04.414+0000 7f1e1aa00700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f1e1c104410 0x7f1e1c19a5f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.118:3300/0 says I am v2:172.21.15.118:54854/0 (socket says 172.21.15.118:54854) 2024-03-24T00:12:04.416 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:04.414+0000 7f1e1aa00700 1 -- 172.21.15.118:0/786010210 learned_addr learned my addr 172.21.15.118:0/786010210 (peer_addr_for_me v2:172.21.15.118:0/0) 2024-03-24T00:12:04.417 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:04.414+0000 7f1e1a000700 1 --2- 172.21.15.118:0/786010210 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1e1c105610 0x7f1e1c19ab30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:12:04.417 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:04.414+0000 7f1e1b400700 1 --2- 172.21.15.118:0/786010210 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f1e1c105fb0 0x7f1e1c1a1bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:12:04.417 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:04.415+0000 7f1e1aa00700 1 -- 172.21.15.118:0/786010210 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f1e1c105fb0 msgr2=0x7f1e1c1a1bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:12:04.417 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:04.415+0000 7f1e1aa00700 1 --2- 172.21.15.118:0/786010210 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f1e1c105fb0 0x7f1e1c1a1bb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:12:04.417 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:04.415+0000 7f1e1aa00700 1 -- 172.21.15.118:0/786010210 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1e1c105610 msgr2=0x7f1e1c19ab30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:12:04.417 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:04.415+0000 7f1e1aa00700 1 --2- 172.21.15.118:0/786010210 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1e1c105610 0x7f1e1c19ab30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:12:04.417 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:04.415+0000 7f1e1aa00700 1 -- 172.21.15.118:0/786010210 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1e1c1a2250 con 0x7f1e1c104410 2024-03-24T00:12:04.417 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:04.415+0000 7f1e1aa00700 1 --2- 172.21.15.118:0/786010210 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f1e1c104410 0x7f1e1c19a5f0 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7f1e0c00bf90 tx=0x7f1e0c00bfc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:12:04.417 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:04.415+0000 7f1e13400700 1 -- 172.21.15.118:0/786010210 <== mon.0 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1e0c009e60 con 0x7f1e1c104410 2024-03-24T00:12:04.417 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:04.415+0000 7f1e13400700 1 -- 172.21.15.118:0/786010210 <== mon.0 v2:172.21.15.118:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f1e0c003e30 con 0x7f1e1c104410 2024-03-24T00:12:04.417 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:04.415+0000 7f1e13400700 1 -- 172.21.15.118:0/786010210 <== mon.0 v2:172.21.15.118:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1e0c00fd70 con 0x7f1e1c104410 2024-03-24T00:12:04.418 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:04.415+0000 7f1e21600700 1 -- 172.21.15.118:0/786010210 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1e1c1a24e0 con 0x7f1e1c104410 2024-03-24T00:12:04.418 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:04.415+0000 7f1e21600700 1 -- 172.21.15.118:0/786010210 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1e1c1a2970 con 0x7f1e1c104410 2024-03-24T00:12:04.418 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:04.417+0000 7f1e21600700 1 -- 172.21.15.118:0/786010210 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1e1c068e50 con 0x7f1e1c104410 2024-03-24T00:12:04.418 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:04.419+0000 7f1e13400700 1 -- 172.21.15.118:0/786010210 <== mon.0 v2:172.21.15.118:3300/0 4 ==== mgrmap(e 16) v1 ==== 137291+0+0 (secure 0 0 0) 0x7f1e0c00f460 con 0x7f1e1c104410 2024-03-24T00:12:04.419 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:04.420+0000 7f1e13400700 1 --2- 172.21.15.118:0/786010210 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7f1e000a2890 0x7f1e000a4d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:12:04.419 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:04.420+0000 7f1e13400700 1 -- 172.21.15.118:0/786010210 <== mon.0 v2:172.21.15.118:3300/0 5 ==== osd_map(81..81 src has 1..81) v4 ==== 7127+0+0 (secure 0 0 0) 0x7f1e0c0cf3e0 con 0x7f1e1c104410 2024-03-24T00:12:04.420 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:04.420+0000 7f1e1a000700 1 --2- 172.21.15.118:0/786010210 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7f1e000a2890 0x7f1e000a4d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:12:04.420 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:04.421+0000 7f1e1a000700 1 --2- 172.21.15.118:0/786010210 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7f1e000a2890 0x7f1e000a4d50 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f1e08009680 tx=0x7f1e08009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:12:04.424 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:04.425+0000 7f1e13400700 1 -- 172.21.15.118:0/786010210 <== mon.0 v2:172.21.15.118:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f1e0c013070 con 0x7f1e1c104410 2024-03-24T00:12:04.646 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:04.646+0000 7f1e21600700 1 -- 172.21.15.118:0/786010210 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_command({"prefix": "osd stat", "format": "json"} v 0) v1 -- 0x7f1e1c062da0 con 0x7f1e1c104410 2024-03-24T00:12:04.646 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:04.647+0000 7f1e13400700 1 -- 172.21.15.118:0/786010210 <== mon.0 v2:172.21.15.118:3300/0 7 ==== mon_command_ack([{"prefix": "osd stat", "format": "json"}]=0 v81) v1 ==== 74+0+133 (secure 0 0 0) 0x7f1e0c09c910 con 0x7f1e1c104410 2024-03-24T00:12:04.647 INFO:teuthology.orchestra.run.smithi118.stdout: 2024-03-24T00:12:04.649 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:04.649+0000 7f1e21600700 1 -- 172.21.15.118:0/786010210 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7f1e000a2890 msgr2=0x7f1e000a4d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:12:04.650 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:04.649+0000 7f1e21600700 1 --2- 172.21.15.118:0/786010210 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7f1e000a2890 0x7f1e000a4d50 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f1e08009680 tx=0x7f1e08009260 comp rx=0 tx=0).stop 2024-03-24T00:12:04.650 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:04.649+0000 7f1e21600700 1 -- 172.21.15.118:0/786010210 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f1e1c104410 msgr2=0x7f1e1c19a5f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:12:04.650 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:04.649+0000 7f1e21600700 1 --2- 172.21.15.118:0/786010210 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f1e1c104410 0x7f1e1c19a5f0 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7f1e0c00bf90 tx=0x7f1e0c00bfc0 comp rx=0 tx=0).stop 2024-03-24T00:12:04.650 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:04.650+0000 7f1e21600700 1 -- 172.21.15.118:0/786010210 shutdown_connections 2024-03-24T00:12:04.650 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:04.650+0000 7f1e21600700 1 --2- 172.21.15.118:0/786010210 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7f1e000a2890 0x7f1e000a4d50 unknown :-1 s=CLOSED pgs=116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:12:04.650 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:04.650+0000 7f1e21600700 1 --2- 172.21.15.118:0/786010210 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f1e1c105fb0 0x7f1e1c1a1bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:12:04.650 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:04.650+0000 7f1e21600700 1 --2- 172.21.15.118:0/786010210 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f1e1c104410 0x7f1e1c19a5f0 unknown :-1 s=CLOSED pgs=131 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:12:04.650 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:04.650+0000 7f1e21600700 1 --2- 172.21.15.118:0/786010210 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1e1c105610 0x7f1e1c19ab30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:12:04.650 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:04.650+0000 7f1e21600700 1 -- 172.21.15.118:0/786010210 >> 172.21.15.118:0/786010210 conn(0x7f1e1c0ff9b0 msgr2=0x7f1e1c108910 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:12:04.650 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:04.650+0000 7f1e21600700 1 -- 172.21.15.118:0/786010210 shutdown_connections 2024-03-24T00:12:04.650 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:04.650+0000 7f1e21600700 1 -- 172.21.15.118:0/786010210 wait complete. 2024-03-24T00:12:04.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:12:04 smithi118 ceph-mon[31535]: purged_snaps scrub starts 2024-03-24T00:12:04.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:12:04 smithi118 ceph-mon[31535]: purged_snaps scrub ok 2024-03-24T00:12:04.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:12:04 smithi118 ceph-mon[31535]: osd.11 [v2:172.21.15.193:6824/3880515510,v1:172.21.15.193:6825/3880515510] boot 2024-03-24T00:12:04.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:12:04 smithi118 ceph-mon[31535]: osdmap e81: 12 total, 12 up, 12 in 2024-03-24T00:12:04.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:12:04 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-03-24T00:12:04.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:12:04 smithi118 ceph-mon[31535]: from='client.? 172.21.15.118:0/786010210' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-03-24T00:12:04.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:12:04 smithi193 ceph-mon[36878]: purged_snaps scrub starts 2024-03-24T00:12:04.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:12:04 smithi193 ceph-mon[36878]: purged_snaps scrub ok 2024-03-24T00:12:04.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:12:04 smithi193 ceph-mon[36878]: osd.11 [v2:172.21.15.193:6824/3880515510,v1:172.21.15.193:6825/3880515510] boot 2024-03-24T00:12:04.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:12:04 smithi193 ceph-mon[36878]: osdmap e81: 12 total, 12 up, 12 in 2024-03-24T00:12:04.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:12:04 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-03-24T00:12:04.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:12:04 smithi193 ceph-mon[36878]: from='client.? 172.21.15.118:0/786010210' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-03-24T00:12:05.203 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:12:04 smithi189 ceph-mon[36187]: purged_snaps scrub starts 2024-03-24T00:12:05.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:12:04 smithi189 ceph-mon[36187]: purged_snaps scrub ok 2024-03-24T00:12:05.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:12:04 smithi189 ceph-mon[36187]: osd.11 [v2:172.21.15.193:6824/3880515510,v1:172.21.15.193:6825/3880515510] boot 2024-03-24T00:12:05.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:12:04 smithi189 ceph-mon[36187]: osdmap e81: 12 total, 12 up, 12 in 2024-03-24T00:12:05.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:12:04 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "osd metadata", "id": 11}]: dispatch 2024-03-24T00:12:05.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:12:04 smithi189 ceph-mon[36187]: from='client.? 172.21.15.118:0/786010210' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-03-24T00:12:05.380 INFO:teuthology.orchestra.run.smithi118.stdout:{"epoch":81,"num_osds":12,"num_up_osds":12,"osd_up_since":1711239123,"num_in_osds":12,"osd_in_since":1711239104,"num_remapped_pgs":0} 2024-03-24T00:12:05.381 DEBUG:teuthology.orchestra.run.smithi118:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 shell --fsid d79a5420-e971-11ee-95cd-87774f69a715 -- ceph osd dump --format=json 2024-03-24T00:12:05.828 INFO:teuthology.orchestra.run.smithi118.stderr:Inferring config /var/lib/ceph/d79a5420-e971-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T00:12:06.175 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:12:05 smithi118 ceph-mon[31535]: osdmap e82: 12 total, 12 up, 12 in 2024-03-24T00:12:06.175 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:12:05 smithi118 ceph-mon[31535]: Detected new or changed devices on smithi193 2024-03-24T00:12:06.175 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:12:05 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T00:12:06.175 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:12:05 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T00:12:06.175 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:12:05 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-24T00:12:06.175 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:12:05 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-24T00:12:06.175 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:12:05 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-03-24T00:12:06.175 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:12:05 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-03-24T00:12:06.175 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:12:05 smithi118 ceph-mon[31535]: Adjusting osd_memory_target on smithi193 to 4270M 2024-03-24T00:12:06.175 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:12:05 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T00:12:06.175 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:12:05 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T00:12:06.175 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:12:05 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T00:12:06.176 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:12:05 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T00:12:06.176 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:12:05 smithi118 ceph-mon[31535]: pgmap v282: 1 pgs: 1 active+clean; 577 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-24T00:12:06.176 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:12:05 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T00:12:06.203 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:12:05 smithi189 ceph-mon[36187]: osdmap e82: 12 total, 12 up, 12 in 2024-03-24T00:12:06.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:12:05 smithi189 ceph-mon[36187]: Detected new or changed devices on smithi193 2024-03-24T00:12:06.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:12:05 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T00:12:06.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:12:05 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T00:12:06.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:12:05 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-24T00:12:06.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:12:05 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-24T00:12:06.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:12:05 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-03-24T00:12:06.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:12:05 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-03-24T00:12:06.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:12:05 smithi189 ceph-mon[36187]: Adjusting osd_memory_target on smithi193 to 4270M 2024-03-24T00:12:06.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:12:05 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T00:12:06.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:12:05 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T00:12:06.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:12:05 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T00:12:06.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:12:05 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T00:12:06.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:12:05 smithi189 ceph-mon[36187]: pgmap v282: 1 pgs: 1 active+clean; 577 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-24T00:12:06.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:12:05 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T00:12:06.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:12:05 smithi193 ceph-mon[36878]: osdmap e82: 12 total, 12 up, 12 in 2024-03-24T00:12:06.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:12:05 smithi193 ceph-mon[36878]: Detected new or changed devices on smithi193 2024-03-24T00:12:06.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:12:05 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T00:12:06.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:12:05 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T00:12:06.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:12:05 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-24T00:12:06.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:12:05 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-24T00:12:06.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:12:05 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-03-24T00:12:06.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:12:05 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-03-24T00:12:06.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:12:05 smithi193 ceph-mon[36878]: Adjusting osd_memory_target on smithi193 to 4270M 2024-03-24T00:12:06.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:12:05 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T00:12:06.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:12:05 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T00:12:06.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:12:05 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T00:12:06.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:12:05 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T00:12:06.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:12:05 smithi193 ceph-mon[36878]: pgmap v282: 1 pgs: 1 active+clean; 577 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-24T00:12:06.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:12:05 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T00:12:07.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:12:06 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T00:12:07.197 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:12:06 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T00:12:07.197 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:12:06 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T00:12:07.203 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:12:06 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T00:12:07.203 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:12:06 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T00:12:07.203 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:12:06 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T00:12:07.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:12:06 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T00:12:07.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:12:06 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T00:12:07.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:12:06 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T00:12:08.027 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:08.027+0000 7f4430c00700 1 -- 172.21.15.118:0/2767746898 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f442c104f90 msgr2=0x7f442c108fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:12:08.027 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:08.027+0000 7f4430c00700 1 --2- 172.21.15.118:0/2767746898 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f442c104f90 0x7f442c108fe0 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f4418009920 tx=0x7f4418009c30 comp rx=0 tx=0).stop 2024-03-24T00:12:08.027 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:08.027+0000 7f4430c00700 1 -- 172.21.15.118:0/2767746898 shutdown_connections 2024-03-24T00:12:08.028 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:08.027+0000 7f4430c00700 1 --2- 172.21.15.118:0/2767746898 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f442c1045e0 0x7f442c1049c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:12:08.028 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:08.027+0000 7f4430c00700 1 --2- 172.21.15.118:0/2767746898 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f442c1096c0 0x7f442c111290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:12:08.028 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:08.027+0000 7f4430c00700 1 --2- 172.21.15.118:0/2767746898 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f442c104f90 0x7f442c108fe0 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:12:08.028 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:08.027+0000 7f4430c00700 1 -- 172.21.15.118:0/2767746898 >> 172.21.15.118:0/2767746898 conn(0x7f442c0ffe90 msgr2=0x7f442c1022b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:12:08.028 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:08.027+0000 7f4430c00700 1 -- 172.21.15.118:0/2767746898 shutdown_connections 2024-03-24T00:12:08.028 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:08.027+0000 7f4430c00700 1 -- 172.21.15.118:0/2767746898 wait complete. 2024-03-24T00:12:08.030 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:08.028+0000 7f4430c00700 1 Processor -- start 2024-03-24T00:12:08.030 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:08.028+0000 7f4430c00700 1 -- start start 2024-03-24T00:12:08.030 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:08.029+0000 7f4430c00700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f442c1045e0 0x7f442c1a0890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:12:08.030 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:08.029+0000 7f4430c00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f442c104f90 0x7f442c1a0dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:12:08.030 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:08.029+0000 7f4430c00700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f442c1096c0 0x7f442c19aa50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:12:08.030 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:08.029+0000 7f4430c00700 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f442c1146d0 con 0x7f442c1096c0 2024-03-24T00:12:08.030 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:08.029+0000 7f4430c00700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f442c114550 con 0x7f442c104f90 2024-03-24T00:12:08.031 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:08.029+0000 7f4430c00700 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f442c114850 con 0x7f442c1045e0 2024-03-24T00:12:08.031 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:08.029+0000 7f4429400700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f442c104f90 0x7f442c1a0dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:12:08.031 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:08.029+0000 7f442a800700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f442c1096c0 0x7f442c19aa50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:12:08.031 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:08.029+0000 7f4429400700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f442c104f90 0x7f442c1a0dd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.118:33408/0 (socket says 172.21.15.118:33408) 2024-03-24T00:12:08.031 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:08.029+0000 7f4429400700 1 -- 172.21.15.118:0/1088682912 learned_addr learned my addr 172.21.15.118:0/1088682912 (peer_addr_for_me v2:172.21.15.118:0/0) 2024-03-24T00:12:08.031 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:08.029+0000 7f4429e00700 1 --2- 172.21.15.118:0/1088682912 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f442c1045e0 0x7f442c1a0890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:12:08.031 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:08.029+0000 7f4429400700 1 -- 172.21.15.118:0/1088682912 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f442c1045e0 msgr2=0x7f442c1a0890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:12:08.031 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:08.029+0000 7f4429400700 1 --2- 172.21.15.118:0/1088682912 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f442c1045e0 0x7f442c1a0890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:12:08.031 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:08.029+0000 7f4429400700 1 -- 172.21.15.118:0/1088682912 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f442c1096c0 msgr2=0x7f442c19aa50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:12:08.031 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:08.029+0000 7f4429400700 1 --2- 172.21.15.118:0/1088682912 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f442c1096c0 0x7f442c19aa50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:12:08.031 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:08.029+0000 7f4429400700 1 -- 172.21.15.118:0/1088682912 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f442c19b1b0 con 0x7f442c104f90 2024-03-24T00:12:08.031 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:08.030+0000 7f4429400700 1 --2- 172.21.15.118:0/1088682912 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f442c104f90 0x7f442c1a0dd0 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f441800a010 tx=0x7f441800fa70 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:12:08.032 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:08.030+0000 7f4422a00700 1 -- 172.21.15.118:0/1088682912 <== mon.2 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4418005180 con 0x7f442c104f90 2024-03-24T00:12:08.032 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:08.030+0000 7f4422a00700 1 -- 172.21.15.118:0/1088682912 <== mon.2 v2:172.21.15.189:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f441800b560 con 0x7f442c104f90 2024-03-24T00:12:08.032 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:08.030+0000 7f4430c00700 1 -- 172.21.15.118:0/1088682912 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f442c19b440 con 0x7f442c104f90 2024-03-24T00:12:08.032 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:08.030+0000 7f4422a00700 1 -- 172.21.15.118:0/1088682912 <== mon.2 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4418003a00 con 0x7f442c104f90 2024-03-24T00:12:08.032 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:08.030+0000 7f4430c00700 1 -- 172.21.15.118:0/1088682912 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f442c1a7760 con 0x7f442c104f90 2024-03-24T00:12:08.032 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:08.031+0000 7f4430c00700 1 -- 172.21.15.118:0/1088682912 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f442c105ac0 con 0x7f442c104f90 2024-03-24T00:12:08.033 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:08.033+0000 7f4422a00700 1 -- 172.21.15.118:0/1088682912 <== mon.2 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 16) v1 ==== 137291+0+0 (secure 0 0 0) 0x7f4418003cc0 con 0x7f442c104f90 2024-03-24T00:12:08.034 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:08.034+0000 7f4422a00700 1 --2- 172.21.15.118:0/1088682912 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7f44100a2890 0x7f44100a4d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:12:08.034 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:08.035+0000 7f4422a00700 1 -- 172.21.15.118:0/1088682912 <== mon.2 v2:172.21.15.189:3300/0 5 ==== osd_map(82..82 src has 1..82) v4 ==== 7127+0+0 (secure 0 0 0) 0x7f44180cf610 con 0x7f442c104f90 2024-03-24T00:12:08.034 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:08.035+0000 7f4429e00700 1 --2- 172.21.15.118:0/1088682912 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7f44100a2890 0x7f44100a4d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:12:08.035 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:08.035+0000 7f4429e00700 1 --2- 172.21.15.118:0/1088682912 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7f44100a2890 0x7f44100a4d50 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f442c19bb90 tx=0x7f441c009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:12:08.038 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:08.038+0000 7f4422a00700 1 -- 172.21.15.118:0/1088682912 <== mon.2 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f4418013030 con 0x7f442c104f90 2024-03-24T00:12:08.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:12:07 smithi118 ceph-mon[31535]: pgmap v283: 1 pgs: 1 active+clean; 577 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-24T00:12:08.203 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:12:07 smithi189 ceph-mon[36187]: pgmap v283: 1 pgs: 1 active+clean; 577 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-24T00:12:08.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:12:07 smithi193 ceph-mon[36878]: pgmap v283: 1 pgs: 1 active+clean; 577 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-24T00:12:08.264 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:08.265+0000 7f4430c00700 1 -- 172.21.15.118:0/1088682912 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f442c10c930 con 0x7f442c104f90 2024-03-24T00:12:08.266 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:08.266+0000 7f4422a00700 1 -- 172.21.15.118:0/1088682912 <== mon.2 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v82) v1 ==== 74+0+18524 (secure 0 0 0) 0x7f441809cb40 con 0x7f442c104f90 2024-03-24T00:12:08.267 INFO:teuthology.orchestra.run.smithi118.stdout: 2024-03-24T00:12:08.267 INFO:teuthology.orchestra.run.smithi118.stdout:{"epoch":82,"fsid":"d79a5420-e971-11ee-95cd-87774f69a715","created":"2024-03-24T00:03:58.648524+0000","modified":"2024-03-24T00:12:04.819275+0000","last_up_change":"2024-03-24T00:12:03.809000+0000","last_in_change":"2024-03-24T00:11:44.389344+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-24T00:07:50.190965+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"c9590426-3901-4f68-a5ef-d57aa8fbac92","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6802","nonce":557690061},{"type":"v1","addr":"172.21.15.118:6803","nonce":557690061}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6804","nonce":557690061},{"type":"v1","addr":"172.21.15.118:6805","nonce":557690061}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6808","nonce":557690061},{"type":"v1","addr":"172.21.15.118:6809","nonce":557690061}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6806","nonce":557690061},{"type":"v1","addr":"172.21.15.118:6807","nonce":557690061}]},"public_addr":"172.21.15.118:6803/557690061","cluster_addr":"172.21.15.118:6805/557690061","heartbeat_back_addr":"172.21.15.118:6809/557690061","heartbeat_front_addr":"172.21.15.118:6807/557690061","state":["exists","up"]},{"osd":1,"uuid":"fc7f827e-ecf6-4661-aad0-7a22951892e0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":42,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6800","nonce":4203029485},{"type":"v1","addr":"172.21.15.189:6801","nonce":4203029485}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6802","nonce":4203029485},{"type":"v1","addr":"172.21.15.189:6803","nonce":4203029485}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6806","nonce":4203029485},{"type":"v1","addr":"172.21.15.189:6807","nonce":4203029485}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6804","nonce":4203029485},{"type":"v1","addr":"172.21.15.189:6805","nonce":4203029485}]},"public_addr":"172.21.15.189:6801/4203029485","cluster_addr":"172.21.15.189:6803/4203029485","heartbeat_back_addr":"172.21.15.189:6807/4203029485","heartbeat_front_addr":"172.21.15.189:6805/4203029485","state":["exists","up"]},{"osd":2,"uuid":"e73ccfe9-0e5b-4734-a154-a2f0d829fcfb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6800","nonce":3699076971},{"type":"v1","addr":"172.21.15.193:6801","nonce":3699076971}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6802","nonce":3699076971},{"type":"v1","addr":"172.21.15.193:6803","nonce":3699076971}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6806","nonce":3699076971},{"type":"v1","addr":"172.21.15.193:6807","nonce":3699076971}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6804","nonce":3699076971},{"type":"v1","addr":"172.21.15.193:6805","nonce":3699076971}]},"public_addr":"172.21.15.193:6801/3699076971","cluster_addr":"172.21.15.193:6803/3699076971","heartbeat_back_addr":"172.21.15.193:6807/3699076971","heartbeat_front_addr":"172.21.15.193:6805/3699076971","state":["exists","up"]},{"osd":3,"uuid":"63fed43b-43ce-4031-8957-7fe94725c1ed","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6810","nonce":4059617010},{"type":"v1","addr":"172.21.15.118:6811","nonce":4059617010}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6812","nonce":4059617010},{"type":"v1","addr":"172.21.15.118:6813","nonce":4059617010}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6816","nonce":4059617010},{"type":"v1","addr":"172.21.15.118:6817","nonce":4059617010}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6814","nonce":4059617010},{"type":"v1","addr":"172.21.15.118:6815","nonce":4059617010}]},"public_addr":"172.21.15.118:6811/4059617010","cluster_addr":"172.21.15.118:6813/4059617010","heartbeat_back_addr":"172.21.15.118:6817/4059617010","heartbeat_front_addr":"172.21.15.118:6815/4059617010","state":["exists","up"]},{"osd":4,"uuid":"57377f66-c3b4-43c1-8e8e-bc873672906d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":34,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6808","nonce":4075247068},{"type":"v1","addr":"172.21.15.189:6809","nonce":4075247068}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6810","nonce":4075247068},{"type":"v1","addr":"172.21.15.189:6811","nonce":4075247068}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6814","nonce":4075247068},{"type":"v1","addr":"172.21.15.189:6815","nonce":4075247068}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6812","nonce":4075247068},{"type":"v1","addr":"172.21.15.189:6813","nonce":4075247068}]},"public_addr":"172.21.15.189:6809/4075247068","cluster_addr":"172.21.15.189:6811/4075247068","heartbeat_back_addr":"172.21.15.189:6815/4075247068","heartbeat_front_addr":"172.21.15.189:6813/4075247068","state":["exists","up"]},{"osd":5,"uuid":"91970976-2193-4224-81c0-dc96cb022c85","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":39,"up_thru":54,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6808","nonce":2920363402},{"type":"v1","addr":"172.21.15.193:6809","nonce":2920363402}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6810","nonce":2920363402},{"type":"v1","addr":"172.21.15.193:6811","nonce":2920363402}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6814","nonce":2920363402},{"type":"v1","addr":"172.21.15.193:6815","nonce":2920363402}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6812","nonce":2920363402},{"type":"v1","addr":"172.21.15.193:6813","nonce":2920363402}]},"public_addr":"172.21.15.193:6809/2920363402","cluster_addr":"172.21.15.193:6811/2920363402","heartbeat_back_addr":"172.21.15.193:6815/2920363402","heartbeat_front_addr":"172.21.15.193:6813/2920363402","state":["exists","up"]},{"osd":6,"uuid":"0b77b196-8674-41c6-ad3d-726b5d01c8fd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":47,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6818","nonce":431354576},{"type":"v1","addr":"172.21.15.118:6819","nonce":431354576}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6820","nonce":431354576},{"type":"v1","addr":"172.21.15.118:6821","nonce":431354576}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6824","nonce":431354576},{"type":"v1","addr":"172.21.15.118:6825","nonce":431354576}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6822","nonce":431354576},{"type":"v1","addr":"172.21.15.118:6823","nonce":431354576}]},"public_addr":"172.21.15.118:6819/431354576","cluster_addr":"172.21.15.118:6821/431354576","heartbeat_back_addr":"172.21.15.118:6825/431354576","heartbeat_front_addr":"172.21.15.118:6823/431354576","state":["exists","up"]},{"osd":7,"uuid":"1fbbd38f-4e8e-4d4c-8e98-66f6862bc4bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":52,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6816","nonce":3667611284},{"type":"v1","addr":"172.21.15.189:6817","nonce":3667611284}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6818","nonce":3667611284},{"type":"v1","addr":"172.21.15.189:6819","nonce":3667611284}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6822","nonce":3667611284},{"type":"v1","addr":"172.21.15.189:6823","nonce":3667611284}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6820","nonce":3667611284},{"type":"v1","addr":"172.21.15.189:6821","nonce":3667611284}]},"public_addr":"172.21.15.189:6817/3667611284","cluster_addr":"172.21.15.189:6819/3667611284","heartbeat_back_addr":"172.21.15.189:6823/3667611284","heartbeat_front_addr":"172.21.15.189:6821/3667611284","state":["exists","up"]},{"osd":8,"uuid":"6462c2b5-e4b3-4b0b-bcfa-4de2c89e93e0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":60,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6816","nonce":3853800910},{"type":"v1","addr":"172.21.15.193:6817","nonce":3853800910}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6818","nonce":3853800910},{"type":"v1","addr":"172.21.15.193:6819","nonce":3853800910}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6822","nonce":3853800910},{"type":"v1","addr":"172.21.15.193:6823","nonce":3853800910}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6820","nonce":3853800910},{"type":"v1","addr":"172.21.15.193:6821","nonce":3853800910}]},"public_addr":"172.21.15.193:6817/3853800910","cluster_addr":"172.21.15.193:6819/3853800910","heartbeat_back_addr":"172.21.15.193:6823/3853800910","heartbeat_front_addr":"172.21.15.193:6821/3853800910","state":["exists","up"]},{"osd":9,"uuid":"54be277f-3596-422f-9de5-9005a8571c6f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":68,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6826","nonce":3248293936},{"type":"v1","addr":"172.21.15.118:6827","nonce":3248293936}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6828","nonce":3248293936},{"type":"v1","addr":"172.21.15.118:6829","nonce":3248293936}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6832","nonce":3248293936},{"type":"v1","addr":"172.21.15.118:6833","nonce":3248293936}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6830","nonce":3248293936},{"type":"v1","addr":"172.21.15.118:6831","nonce":3248293936}]},"public_addr":"172.21.15.118:6827/3248293936","cluster_addr":"172.21.15.118:6829/3248293936","heartbeat_back_addr":"172.21.15.118:6833/3248293936","heartbeat_front_addr":"172.21.15.118:6831/3248293936","state":["exists","up"]},{"osd":10,"uuid":"7ba69d69-ac41-44ce-bc39-c049a8ca5900","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":73,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6824","nonce":2112195760},{"type":"v1","addr":"172.21.15.189:6825","nonce":2112195760}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6826","nonce":2112195760},{"type":"v1","addr":"172.21.15.189:6827","nonce":2112195760}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6830","nonce":2112195760},{"type":"v1","addr":"172.21.15.189:6831","nonce":2112195760}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6828","nonce":2112195760},{"type":"v1","addr":"172.21.15.189:6829","nonce":2112195760}]},"public_addr":"172.21.15.189:6825/2112195760","cluster_addr":"172.21.15.189:6827/2112195760","heartbeat_back_addr":"172.21.15.189:6831/2112195760","heartbeat_front_addr":"172.21.15.189:6829/2112195760","state":["exists","up"]},{"osd":11,"uuid":"186f0b18-6b22-4eae-83b7-b21bbcce0817","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":81,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6824","nonce":3880515510},{"type":"v1","addr":"172.21.15.193:6825","nonce":3880515510}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6826","nonce":3880515510},{"type":"v1","addr":"172.21.15.193:6827","nonce":3880515510}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6830","nonce":3880515510},{"type":"v1","addr":"172.21.15.193:6831","nonce":3880515510}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6828","nonce":3880515510},{"type":"v1","addr":"172.21.15.193:6829","nonce":3880515510}]},"public_addr":"172.21.15.193:6825/3880515510","cluster_addr":"172.21.15.193:6827/3880515510","heartbeat_back_addr":"172.21.15.193:6831/3880515510","heartbeat_front_addr":"172.21.15.193:6829/3880515510","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:06:58.105543+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:07:23.795866+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:07:47.426050+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:08:19.427073+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:08:45.816082+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:09:11.081523+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:09:44.485077+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:10:11.338972+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:10:36.234024+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:11:09.730432+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:11:37.059083+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:12:02.371947+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.118:6801/389249272":"2024-03-25T00:05:05.458796+0000","172.21.15.118:6800/389249272":"2024-03-25T00:05:05.458796+0000","172.21.15.118:0/2101854694":"2024-03-25T00:05:05.458796+0000","172.21.15.118:0/630779359":"2024-03-25T00:05:05.458796+0000","172.21.15.118:0/2158921029":"2024-03-25T00:04:31.283725+0000","172.21.15.118:0/1003670455":"2024-03-25T00:05:05.458796+0000","172.21.15.118:0/368712314":"2024-03-25T00:04:31.283725+0000","172.21.15.118:0/2004775597":"2024-03-25T00:04:31.283725+0000","172.21.15.118:6800/1463147955":"2024-03-25T00:04:31.283725+0000","172.21.15.118:6801/1463147955":"2024-03-25T00:04:31.283725+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-24T00:12:08.270 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:08.269+0000 7f4430c00700 1 -- 172.21.15.118:0/1088682912 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7f44100a2890 msgr2=0x7f44100a4d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:12:08.270 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:08.269+0000 7f4430c00700 1 --2- 172.21.15.118:0/1088682912 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7f44100a2890 0x7f44100a4d50 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f442c19bb90 tx=0x7f441c009260 comp rx=0 tx=0).stop 2024-03-24T00:12:08.270 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:08.270+0000 7f4430c00700 1 -- 172.21.15.118:0/1088682912 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f442c104f90 msgr2=0x7f442c1a0dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:12:08.270 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:08.270+0000 7f4430c00700 1 --2- 172.21.15.118:0/1088682912 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f442c104f90 0x7f442c1a0dd0 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f441800a010 tx=0x7f441800fa70 comp rx=0 tx=0).stop 2024-03-24T00:12:08.270 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:08.270+0000 7f4430c00700 1 -- 172.21.15.118:0/1088682912 shutdown_connections 2024-03-24T00:12:08.270 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:08.270+0000 7f4430c00700 1 --2- 172.21.15.118:0/1088682912 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7f44100a2890 0x7f44100a4d50 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:12:08.270 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:08.270+0000 7f4430c00700 1 --2- 172.21.15.118:0/1088682912 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f442c1096c0 0x7f442c19aa50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:12:08.270 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:08.270+0000 7f4430c00700 1 --2- 172.21.15.118:0/1088682912 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f442c1045e0 0x7f442c1a0890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:12:08.270 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:08.270+0000 7f4430c00700 1 --2- 172.21.15.118:0/1088682912 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f442c104f90 0x7f442c1a0dd0 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:12:08.270 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:08.270+0000 7f4430c00700 1 -- 172.21.15.118:0/1088682912 >> 172.21.15.118:0/1088682912 conn(0x7f442c0ffe90 msgr2=0x7f442c101300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:12:08.270 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:08.270+0000 7f4430c00700 1 -- 172.21.15.118:0/1088682912 shutdown_connections 2024-03-24T00:12:08.270 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:08.270+0000 7f4430c00700 1 -- 172.21.15.118:0/1088682912 wait complete. 2024-03-24T00:12:09.088 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:12:08 smithi118 ceph-mon[31535]: from='client.? 172.21.15.118:0/1088682912' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-24T00:12:09.167 INFO:tasks.cephadm.ceph_manager.ceph:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-24T00:07:50.190965+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-24T00:12:09.168 DEBUG:teuthology.orchestra.run.smithi118:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 shell --fsid d79a5420-e971-11ee-95cd-87774f69a715 -- ceph osd pool get .mgr pg_num 2024-03-24T00:12:09.203 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:12:08 smithi189 ceph-mon[36187]: from='client.? 172.21.15.118:0/1088682912' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-24T00:12:09.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:12:08 smithi193 ceph-mon[36878]: from='client.? 172.21.15.118:0/1088682912' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-24T00:12:09.616 INFO:teuthology.orchestra.run.smithi118.stderr:Inferring config /var/lib/ceph/d79a5420-e971-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T00:12:10.024 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:12:09 smithi118 ceph-mon[31535]: pgmap v284: 1 pgs: 1 active+clean; 577 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-24T00:12:10.203 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:12:09 smithi189 ceph-mon[36187]: pgmap v284: 1 pgs: 1 active+clean; 577 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-24T00:12:10.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:12:09 smithi193 ceph-mon[36878]: pgmap v284: 1 pgs: 1 active+clean; 577 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-24T00:12:11.801 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:11.801+0000 7f5fe4200700 1 -- 172.21.15.118:0/945968938 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f5fdc1096c0 msgr2=0x7f5fdc111290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:12:11.801 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:11.801+0000 7f5fe4200700 1 --2- 172.21.15.118:0/945968938 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f5fdc1096c0 0x7f5fdc111290 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7f5fd400b0d0 tx=0x7f5fd400b3e0 comp rx=0 tx=0).stop 2024-03-24T00:12:11.801 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:11.801+0000 7f5fe4200700 1 -- 172.21.15.118:0/945968938 shutdown_connections 2024-03-24T00:12:11.801 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:11.801+0000 7f5fe4200700 1 --2- 172.21.15.118:0/945968938 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f5fdc1096c0 0x7f5fdc111290 unknown :-1 s=CLOSED pgs=132 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:12:11.801 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:11.801+0000 7f5fe4200700 1 --2- 172.21.15.118:0/945968938 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f5fdc1045e0 0x7f5fdc1049c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:12:11.801 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:11.801+0000 7f5fe4200700 1 --2- 172.21.15.118:0/945968938 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5fdc104f90 0x7f5fdc108fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:12:11.801 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:11.801+0000 7f5fe4200700 1 -- 172.21.15.118:0/945968938 >> 172.21.15.118:0/945968938 conn(0x7f5fdc0ffe70 msgr2=0x7f5fdc102290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:12:11.802 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:11.801+0000 7f5fe4200700 1 -- 172.21.15.118:0/945968938 shutdown_connections 2024-03-24T00:12:11.802 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:11.801+0000 7f5fe4200700 1 -- 172.21.15.118:0/945968938 wait complete. 2024-03-24T00:12:11.803 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:11.802+0000 7f5fe4200700 1 Processor -- start 2024-03-24T00:12:11.803 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:11.802+0000 7f5fe4200700 1 -- start start 2024-03-24T00:12:11.804 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:11.803+0000 7f5fe4200700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f5fdc1045e0 0x7f5fdc1a07b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:12:11.804 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:11.803+0000 7f5fe4200700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5fdc104f90 0x7f5fdc1a0cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:12:11.804 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:11.803+0000 7f5fe4200700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f5fdc1096c0 0x7f5fdc19a9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:12:11.804 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:11.803+0000 7f5fe4200700 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5fdc114720 con 0x7f5fdc1096c0 2024-03-24T00:12:11.804 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:11.803+0000 7f5fe4200700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5fdc1145a0 con 0x7f5fdc104f90 2024-03-24T00:12:11.804 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:11.803+0000 7f5fe4200700 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5fdc1148a0 con 0x7f5fdc1045e0 2024-03-24T00:12:11.804 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:11.803+0000 7f5fe1800700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f5fdc1045e0 0x7f5fdc1a07b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:12:11.804 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:11.803+0000 7f5fe0e00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5fdc104f90 0x7f5fdc1a0cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:12:11.804 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:11.803+0000 7f5fe1800700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f5fdc1045e0 0x7f5fdc1a07b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.118:60604/0 (socket says 172.21.15.118:60604) 2024-03-24T00:12:11.804 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:11.803+0000 7f5fe1800700 1 -- 172.21.15.118:0/2971119888 learned_addr learned my addr 172.21.15.118:0/2971119888 (peer_addr_for_me v2:172.21.15.118:0/0) 2024-03-24T00:12:11.804 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:11.803+0000 7f5fe2200700 1 --2- 172.21.15.118:0/2971119888 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f5fdc1096c0 0x7f5fdc19a9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:12:11.804 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:11.804+0000 7f5fe0e00700 1 -- 172.21.15.118:0/2971119888 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f5fdc1045e0 msgr2=0x7f5fdc1a07b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:12:11.805 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:11.804+0000 7f5fe0e00700 1 --2- 172.21.15.118:0/2971119888 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f5fdc1045e0 0x7f5fdc1a07b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:12:11.805 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:11.804+0000 7f5fe0e00700 1 -- 172.21.15.118:0/2971119888 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f5fdc1096c0 msgr2=0x7f5fdc19a9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:12:11.805 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:11.804+0000 7f5fe0e00700 1 --2- 172.21.15.118:0/2971119888 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f5fdc1096c0 0x7f5fdc19a9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:12:11.805 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:11.804+0000 7f5fe0e00700 1 -- 172.21.15.118:0/2971119888 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5fdc19b120 con 0x7f5fdc104f90 2024-03-24T00:12:11.805 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:11.804+0000 7f5fe1800700 1 --2- 172.21.15.118:0/2971119888 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f5fdc1045e0 0x7f5fdc1a07b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T00:12:11.805 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:11.804+0000 7f5fe0e00700 1 --2- 172.21.15.118:0/2971119888 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5fdc104f90 0x7f5fdc1a0cf0 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f5fc800e9c0 tx=0x7f5fc800ecd0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:12:11.805 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:11.804+0000 7f5fda000700 1 -- 172.21.15.118:0/2971119888 <== mon.2 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5fc800cb70 con 0x7f5fdc104f90 2024-03-24T00:12:11.806 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:11.804+0000 7f5fda000700 1 -- 172.21.15.118:0/2971119888 <== mon.2 v2:172.21.15.189:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f5fc8004500 con 0x7f5fdc104f90 2024-03-24T00:12:11.806 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:11.804+0000 7f5fe4200700 1 -- 172.21.15.118:0/2971119888 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5fdc19b410 con 0x7f5fdc104f90 2024-03-24T00:12:11.806 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:11.804+0000 7f5fda000700 1 -- 172.21.15.118:0/2971119888 <== mon.2 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5fc8003de0 con 0x7f5fdc104f90 2024-03-24T00:12:11.806 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:11.804+0000 7f5fe4200700 1 -- 172.21.15.118:0/2971119888 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5fdc1a7680 con 0x7f5fdc104f90 2024-03-24T00:12:11.806 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:11.805+0000 7f5fe4200700 1 -- 172.21.15.118:0/2971119888 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5fdc068e50 con 0x7f5fdc104f90 2024-03-24T00:12:11.807 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:11.807+0000 7f5fda000700 1 -- 172.21.15.118:0/2971119888 <== mon.2 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 16) v1 ==== 137291+0+0 (secure 0 0 0) 0x7f5fc8003fc0 con 0x7f5fdc104f90 2024-03-24T00:12:11.809 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:11.809+0000 7f5fda000700 1 --2- 172.21.15.118:0/2971119888 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7f5fc40a2890 0x7f5fc40a4d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:12:11.809 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:11.809+0000 7f5fda000700 1 -- 172.21.15.118:0/2971119888 <== mon.2 v2:172.21.15.189:3300/0 5 ==== osd_map(82..82 src has 1..82) v4 ==== 7127+0+0 (secure 0 0 0) 0x7f5fc80ce6e0 con 0x7f5fdc104f90 2024-03-24T00:12:11.809 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:11.809+0000 7f5fe1800700 1 --2- 172.21.15.118:0/2971119888 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7f5fc40a2890 0x7f5fc40a4d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:12:11.809 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:11.810+0000 7f5fe1800700 1 --2- 172.21.15.118:0/2971119888 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7f5fc40a2890 0x7f5fc40a4d50 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f5fdc109470 tx=0x7f5fd0009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:12:11.811 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:11.811+0000 7f5fda000700 1 -- 172.21.15.118:0/2971119888 <== mon.2 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f5fc800ee60 con 0x7f5fdc104f90 2024-03-24T00:12:12.030 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:12.030+0000 7f5fe4200700 1 -- 172.21.15.118:0/2971119888 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f5fdc19bea0 con 0x7f5fdc104f90 2024-03-24T00:12:12.030 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:12.031+0000 7f5fda000700 1 -- 172.21.15.118:0/2971119888 <== mon.2 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v82) v1 ==== 93+0+10 (secure 0 0 0) 0x7f5fc809bb80 con 0x7f5fdc104f90 2024-03-24T00:12:12.031 INFO:teuthology.orchestra.run.smithi118.stdout:pg_num: 1 2024-03-24T00:12:12.034 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:12.034+0000 7f5fe4200700 1 -- 172.21.15.118:0/2971119888 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7f5fc40a2890 msgr2=0x7f5fc40a4d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:12:12.034 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:12.034+0000 7f5fe4200700 1 --2- 172.21.15.118:0/2971119888 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7f5fc40a2890 0x7f5fc40a4d50 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f5fdc109470 tx=0x7f5fd0009260 comp rx=0 tx=0).stop 2024-03-24T00:12:12.034 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:12.034+0000 7f5fe4200700 1 -- 172.21.15.118:0/2971119888 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5fdc104f90 msgr2=0x7f5fdc1a0cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:12:12.034 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:12.034+0000 7f5fe4200700 1 --2- 172.21.15.118:0/2971119888 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5fdc104f90 0x7f5fdc1a0cf0 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f5fc800e9c0 tx=0x7f5fc800ecd0 comp rx=0 tx=0).stop 2024-03-24T00:12:12.034 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:12.034+0000 7f5fe4200700 1 -- 172.21.15.118:0/2971119888 shutdown_connections 2024-03-24T00:12:12.034 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:12.034+0000 7f5fe4200700 1 --2- 172.21.15.118:0/2971119888 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7f5fc40a2890 0x7f5fc40a4d50 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:12:12.034 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:12.034+0000 7f5fe4200700 1 --2- 172.21.15.118:0/2971119888 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f5fdc1096c0 0x7f5fdc19a9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:12:12.034 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:12.034+0000 7f5fe4200700 1 --2- 172.21.15.118:0/2971119888 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f5fdc1045e0 0x7f5fdc1a07b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:12:12.034 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:12.034+0000 7f5fe4200700 1 --2- 172.21.15.118:0/2971119888 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5fdc104f90 0x7f5fdc1a0cf0 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:12:12.035 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:12.034+0000 7f5fe4200700 1 -- 172.21.15.118:0/2971119888 >> 172.21.15.118:0/2971119888 conn(0x7f5fdc0ffe70 msgr2=0x7f5fdc100940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:12:12.035 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:12.034+0000 7f5fe4200700 1 -- 172.21.15.118:0/2971119888 shutdown_connections 2024-03-24T00:12:12.035 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:12.034+0000 7f5fe4200700 1 -- 172.21.15.118:0/2971119888 wait complete. 2024-03-24T00:12:12.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:12:12 smithi118 ceph-mon[31535]: pgmap v285: 1 pgs: 1 active+clean; 577 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-24T00:12:12.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:12:12 smithi118 ceph-mon[31535]: from='client.? 172.21.15.118:0/2971119888' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-24T00:12:12.838 INFO:tasks.cephadm:Adding mds.a on smithi118 2024-03-24T00:12:12.838 INFO:tasks.cephadm:Adding mds.d on smithi118 2024-03-24T00:12:12.838 INFO:tasks.cephadm:Adding mds.g on smithi118 2024-03-24T00:12:12.838 INFO:tasks.cephadm:Adding mds.j on smithi118 2024-03-24T00:12:12.839 INFO:tasks.cephadm:Adding mds.b on smithi189 2024-03-24T00:12:12.839 INFO:tasks.cephadm:Adding mds.e on smithi189 2024-03-24T00:12:12.839 INFO:tasks.cephadm:Adding mds.h on smithi189 2024-03-24T00:12:12.839 INFO:tasks.cephadm:Adding mds.k on smithi189 2024-03-24T00:12:12.839 INFO:tasks.cephadm:Adding mds.c on smithi193 2024-03-24T00:12:12.839 INFO:tasks.cephadm:Adding mds.f on smithi193 2024-03-24T00:12:12.839 INFO:tasks.cephadm:Adding mds.i on smithi193 2024-03-24T00:12:12.839 INFO:tasks.cephadm:Adding mds.l on smithi193 2024-03-24T00:12:12.839 DEBUG:teuthology.orchestra.run.smithi193:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d79a5420-e971-11ee-95cd-87774f69a715 -- ceph orch apply mds all '12;smithi118=a;smithi118=d;smithi118=g;smithi118=j;smithi189=b;smithi189=e;smithi189=h;smithi189=k;smithi193=c;smithi193=f;smithi193=i;smithi193=l' 2024-03-24T00:12:12.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:12:12 smithi189 ceph-mon[36187]: pgmap v285: 1 pgs: 1 active+clean; 577 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-24T00:12:12.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:12:12 smithi189 ceph-mon[36187]: from='client.? 172.21.15.118:0/2971119888' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-24T00:12:12.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:12:12 smithi193 ceph-mon[36878]: pgmap v285: 1 pgs: 1 active+clean; 577 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-24T00:12:12.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:12:12 smithi193 ceph-mon[36878]: from='client.? 172.21.15.118:0/2971119888' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-24T00:12:13.087 INFO:teuthology.orchestra.run.smithi193.stderr:Inferring config /var/lib/ceph/d79a5420-e971-11ee-95cd-87774f69a715/mon.c/config 2024-03-24T00:12:14.715 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-24T00:12:14.715+0000 7f9332980700 1 -- 172.21.15.193:0/3498344103 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f932c1010c0 msgr2=0x7f932c1014e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:12:14.715 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-24T00:12:14.715+0000 7f9332980700 1 --2- 172.21.15.193:0/3498344103 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f932c1010c0 0x7f932c1014e0 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f931c009990 tx=0x7f931c009ca0 comp rx=0 tx=0).stop 2024-03-24T00:12:14.715 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-24T00:12:14.715+0000 7f9332980700 1 -- 172.21.15.193:0/3498344103 shutdown_connections 2024-03-24T00:12:14.716 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-24T00:12:14.715+0000 7f9332980700 1 --2- 172.21.15.193:0/3498344103 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f932c102c60 0x7f932c109510 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:12:14.716 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-24T00:12:14.715+0000 7f9332980700 1 --2- 172.21.15.193:0/3498344103 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f932c1010c0 0x7f932c1014e0 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:12:14.716 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-24T00:12:14.715+0000 7f9332980700 1 --2- 172.21.15.193:0/3498344103 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f932c1022c0 0x7f932c102720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:12:14.716 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-24T00:12:14.715+0000 7f9332980700 1 -- 172.21.15.193:0/3498344103 >> 172.21.15.193:0/3498344103 conn(0x7f932c0fc680 msgr2=0x7f932c0feaa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:12:14.716 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-24T00:12:14.715+0000 7f9332980700 1 -- 172.21.15.193:0/3498344103 shutdown_connections 2024-03-24T00:12:14.716 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-24T00:12:14.715+0000 7f9332980700 1 -- 172.21.15.193:0/3498344103 wait complete. 2024-03-24T00:12:14.716 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-24T00:12:14.715+0000 7f9332980700 1 Processor -- start 2024-03-24T00:12:14.716 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-24T00:12:14.715+0000 7f9332980700 1 -- start start 2024-03-24T00:12:14.718 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-24T00:12:14.716+0000 7f9332980700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f932c1010c0 0x7f932c06b420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:12:14.718 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-24T00:12:14.716+0000 7f9332980700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f932c1022c0 0x7f932c19d2a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:12:14.718 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-24T00:12:14.716+0000 7f9332980700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f932c102c60 0x7f932c19f8e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:12:14.718 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-24T00:12:14.716+0000 7f9332980700 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f932c10cd80 con 0x7f932c1022c0 2024-03-24T00:12:14.718 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-24T00:12:14.716+0000 7f9332980700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f932c10cc00 con 0x7f932c102c60 2024-03-24T00:12:14.719 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-24T00:12:14.716+0000 7f9332980700 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f932c10cf00 con 0x7f932c1010c0 2024-03-24T00:12:14.719 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-24T00:12:14.716+0000 7f932bfff700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f932c1010c0 0x7f932c06b420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:12:14.719 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-24T00:12:14.716+0000 7f932bfff700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f932c1010c0 0x7f932c06b420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.193:49538/0 (socket says 172.21.15.193:49538) 2024-03-24T00:12:14.719 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-24T00:12:14.716+0000 7f932bfff700 1 -- 172.21.15.193:0/1256704988 learned_addr learned my addr 172.21.15.193:0/1256704988 (peer_addr_for_me v2:172.21.15.193:0/0) 2024-03-24T00:12:14.719 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-24T00:12:14.716+0000 7f9330f1d700 1 --2- 172.21.15.193:0/1256704988 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f932c102c60 0x7f932c19f8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:12:14.719 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-24T00:12:14.716+0000 7f932bfff700 1 -- 172.21.15.193:0/1256704988 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f932c102c60 msgr2=0x7f932c19f8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:12:14.719 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-24T00:12:14.716+0000 7f932bfff700 1 --2- 172.21.15.193:0/1256704988 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f932c102c60 0x7f932c19f8e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:12:14.719 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-24T00:12:14.716+0000 7f932bfff700 1 -- 172.21.15.193:0/1256704988 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f932c1022c0 msgr2=0x7f932c19d2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:12:14.719 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-24T00:12:14.716+0000 7f932bfff700 1 --2- 172.21.15.193:0/1256704988 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f932c1022c0 0x7f932c19d2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:12:14.719 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-24T00:12:14.716+0000 7f932bfff700 1 -- 172.21.15.193:0/1256704988 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f932c1a01b0 con 0x7f932c1010c0 2024-03-24T00:12:14.719 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-24T00:12:14.716+0000 7f932bfff700 1 --2- 172.21.15.193:0/1256704988 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f932c1010c0 0x7f932c06b420 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f931c000c00 tx=0x7f931c0047b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:12:14.719 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-24T00:12:14.716+0000 7f93297fa700 1 -- 172.21.15.193:0/1256704988 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f931c003d30 con 0x7f932c1010c0 2024-03-24T00:12:14.720 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-24T00:12:14.716+0000 7f93297fa700 1 -- 172.21.15.193:0/1256704988 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f931c004830 con 0x7f932c1010c0 2024-03-24T00:12:14.720 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-24T00:12:14.716+0000 7f93297fa700 1 -- 172.21.15.193:0/1256704988 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f931c005580 con 0x7f932c1010c0 2024-03-24T00:12:14.720 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-24T00:12:14.716+0000 7f9332980700 1 -- 172.21.15.193:0/1256704988 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f932c1a0440 con 0x7f932c1010c0 2024-03-24T00:12:14.720 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-24T00:12:14.717+0000 7f9332980700 1 -- 172.21.15.193:0/1256704988 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f932c1a08b0 con 0x7f932c1010c0 2024-03-24T00:12:14.720 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-24T00:12:14.718+0000 7f9332980700 1 -- 172.21.15.193:0/1256704988 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f932c068e50 con 0x7f932c1010c0 2024-03-24T00:12:14.721 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-24T00:12:14.719+0000 7f93297fa700 1 -- 172.21.15.193:0/1256704988 <== mon.1 v2:172.21.15.193:3300/0 4 ==== mgrmap(e 16) v1 ==== 137291+0+0 (secure 0 0 0) 0x7f931c003ed0 con 0x7f932c1010c0 2024-03-24T00:12:14.721 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-24T00:12:14.719+0000 7f93297fa700 1 --2- 172.21.15.193:0/1256704988 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7f93140a2890 0x7f93140a4d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:12:14.721 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-24T00:12:14.719+0000 7f93297fa700 1 -- 172.21.15.193:0/1256704988 <== mon.1 v2:172.21.15.193:3300/0 5 ==== osd_map(82..82 src has 1..82) v4 ==== 7127+0+0 (secure 0 0 0) 0x7f931c0cf760 con 0x7f932c1010c0 2024-03-24T00:12:14.721 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-24T00:12:14.720+0000 7f932b7fe700 1 --2- 172.21.15.193:0/1256704988 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7f93140a2890 0x7f93140a4d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:12:14.721 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-24T00:12:14.721+0000 7f932b7fe700 1 --2- 172.21.15.193:0/1256704988 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7f93140a2890 0x7f93140a4d50 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f93200097b0 tx=0x7f9320006d20 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:12:14.723 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-24T00:12:14.724+0000 7f93297fa700 1 -- 172.21.15.193:0/1256704988 <== mon.1 v2:172.21.15.193:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f931c013070 con 0x7f932c1010c0 2024-03-24T00:12:14.928 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-24T00:12:14.928+0000 7f9332980700 1 -- 172.21.15.193:0/1256704988 --> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] -- mgr_command(tid 0: {"prefix": "orch apply mds", "fs_name": "all", "placement": "12;smithi118=a;smithi118=d;smithi118=g;smithi118=j;smithi189=b;smithi189=e;smithi189=h;smithi189=k;smithi193=c;smithi193=f;smithi193=i;smithi193=l", "target": ["mon-mgr", ""]}) v1 -- 0x7f932c0626d0 con 0x7f93140a2890 2024-03-24T00:12:14.929 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:12:14 smithi193 ceph-mon[36878]: pgmap v286: 1 pgs: 1 active+clean; 577 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-24T00:12:14.945 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-24T00:12:14.944+0000 7f93297fa700 1 -- 172.21.15.193:0/1256704988 <== mgr.14152 v2:172.21.15.118:6800/2112912943 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+28 (secure 0 0 0) 0x7f932c0626d0 con 0x7f93140a2890 2024-03-24T00:12:14.945 INFO:teuthology.orchestra.run.smithi193.stdout:Scheduled mds.all update... 2024-03-24T00:12:14.945 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:12:14 smithi118 ceph-mon[31535]: pgmap v286: 1 pgs: 1 active+clean; 577 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-24T00:12:14.947 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-24T00:12:14.947+0000 7f9332980700 1 -- 172.21.15.193:0/1256704988 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7f93140a2890 msgr2=0x7f93140a4d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:12:14.947 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-24T00:12:14.947+0000 7f9332980700 1 --2- 172.21.15.193:0/1256704988 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7f93140a2890 0x7f93140a4d50 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f93200097b0 tx=0x7f9320006d20 comp rx=0 tx=0).stop 2024-03-24T00:12:14.947 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-24T00:12:14.947+0000 7f9332980700 1 -- 172.21.15.193:0/1256704988 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f932c1010c0 msgr2=0x7f932c06b420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:12:14.947 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-24T00:12:14.947+0000 7f9332980700 1 --2- 172.21.15.193:0/1256704988 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f932c1010c0 0x7f932c06b420 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f931c000c00 tx=0x7f931c0047b0 comp rx=0 tx=0).stop 2024-03-24T00:12:14.947 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-24T00:12:14.947+0000 7f9332980700 1 -- 172.21.15.193:0/1256704988 shutdown_connections 2024-03-24T00:12:14.947 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-24T00:12:14.947+0000 7f9332980700 1 --2- 172.21.15.193:0/1256704988 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f932c102c60 0x7f932c19f8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:12:14.947 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-24T00:12:14.947+0000 7f9332980700 1 --2- 172.21.15.193:0/1256704988 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7f93140a2890 0x7f93140a4d50 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:12:14.947 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-24T00:12:14.947+0000 7f9332980700 1 --2- 172.21.15.193:0/1256704988 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f932c1010c0 0x7f932c06b420 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:12:14.948 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-24T00:12:14.947+0000 7f9332980700 1 --2- 172.21.15.193:0/1256704988 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f932c1022c0 0x7f932c19d2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:12:14.948 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-24T00:12:14.947+0000 7f9332980700 1 -- 172.21.15.193:0/1256704988 >> 172.21.15.193:0/1256704988 conn(0x7f932c0fc680 msgr2=0x7f932c1056e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:12:14.948 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-24T00:12:14.947+0000 7f9332980700 1 -- 172.21.15.193:0/1256704988 shutdown_connections 2024-03-24T00:12:14.948 INFO:teuthology.orchestra.run.smithi193.stderr:2024-03-24T00:12:14.947+0000 7f9332980700 1 -- 172.21.15.193:0/1256704988 wait complete. 2024-03-24T00:12:14.953 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:12:14 smithi189 ceph-mon[36187]: pgmap v286: 1 pgs: 1 active+clean; 577 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-24T00:12:15.568 DEBUG:teuthology.orchestra.run.smithi118:mds.a> sudo journalctl -f -n 0 -u ceph-d79a5420-e971-11ee-95cd-87774f69a715@mds.a.service 2024-03-24T00:12:15.576 DEBUG:teuthology.orchestra.run.smithi118:mds.d> sudo journalctl -f -n 0 -u ceph-d79a5420-e971-11ee-95cd-87774f69a715@mds.d.service 2024-03-24T00:12:15.621 DEBUG:teuthology.orchestra.run.smithi118:mds.g> sudo journalctl -f -n 0 -u ceph-d79a5420-e971-11ee-95cd-87774f69a715@mds.g.service 2024-03-24T00:12:15.666 DEBUG:teuthology.orchestra.run.smithi118:mds.j> sudo journalctl -f -n 0 -u ceph-d79a5420-e971-11ee-95cd-87774f69a715@mds.j.service 2024-03-24T00:12:15.713 DEBUG:teuthology.orchestra.run.smithi189:mds.b> sudo journalctl -f -n 0 -u ceph-d79a5420-e971-11ee-95cd-87774f69a715@mds.b.service 2024-03-24T00:12:15.716 DEBUG:teuthology.orchestra.run.smithi189:mds.e> sudo journalctl -f -n 0 -u ceph-d79a5420-e971-11ee-95cd-87774f69a715@mds.e.service 2024-03-24T00:12:15.759 DEBUG:teuthology.orchestra.run.smithi189:mds.h> sudo journalctl -f -n 0 -u ceph-d79a5420-e971-11ee-95cd-87774f69a715@mds.h.service 2024-03-24T00:12:15.802 DEBUG:teuthology.orchestra.run.smithi189:mds.k> sudo journalctl -f -n 0 -u ceph-d79a5420-e971-11ee-95cd-87774f69a715@mds.k.service 2024-03-24T00:12:15.845 DEBUG:teuthology.orchestra.run.smithi193:mds.c> sudo journalctl -f -n 0 -u ceph-d79a5420-e971-11ee-95cd-87774f69a715@mds.c.service 2024-03-24T00:12:15.847 DEBUG:teuthology.orchestra.run.smithi193:mds.f> sudo journalctl -f -n 0 -u ceph-d79a5420-e971-11ee-95cd-87774f69a715@mds.f.service 2024-03-24T00:12:15.890 DEBUG:teuthology.orchestra.run.smithi193:mds.i> sudo journalctl -f -n 0 -u ceph-d79a5420-e971-11ee-95cd-87774f69a715@mds.i.service 2024-03-24T00:12:15.933 DEBUG:teuthology.orchestra.run.smithi193:mds.l> sudo journalctl -f -n 0 -u ceph-d79a5420-e971-11ee-95cd-87774f69a715@mds.l.service 2024-03-24T00:12:15.976 INFO:tasks.cephadm:Setting up CephFS filesystem(s)... 2024-03-24T00:12:15.977 DEBUG:teuthology.orchestra.run.smithi118:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-24T00:12:16.203 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:12:15 smithi189 ceph-mon[36187]: from='client.24415 -' entity='client.admin' cmd=[{"prefix": "orch apply mds", "fs_name": "all", "placement": "12;smithi118=a;smithi118=d;smithi118=g;smithi118=j;smithi189=b;smithi189=e;smithi189=h;smithi189=k;smithi193=c;smithi193=f;smithi193=i;smithi193=l", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T00:12:16.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:12:15 smithi189 ceph-mon[36187]: Saving service mds.all spec with placement smithi118=a;smithi118=d;smithi118=g;smithi118=j;smithi189=b;smithi189=e;smithi189=h;smithi189=k;smithi193=c;smithi193=f;smithi193=i;smithi193=l;count:12 2024-03-24T00:12:16.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:12:15 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T00:12:16.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:12:15 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T00:12:16.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:12:15 smithi189 ceph-mon[36187]: pgmap v287: 1 pgs: 1 active+clean; 577 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-24T00:12:16.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:12:15 smithi193 ceph-mon[36878]: from='client.24415 -' entity='client.admin' cmd=[{"prefix": "orch apply mds", "fs_name": "all", "placement": "12;smithi118=a;smithi118=d;smithi118=g;smithi118=j;smithi189=b;smithi189=e;smithi189=h;smithi189=k;smithi193=c;smithi193=f;smithi193=i;smithi193=l", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T00:12:16.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:12:15 smithi193 ceph-mon[36878]: Saving service mds.all spec with placement smithi118=a;smithi118=d;smithi118=g;smithi118=j;smithi189=b;smithi189=e;smithi189=h;smithi189=k;smithi193=c;smithi193=f;smithi193=i;smithi193=l;count:12 2024-03-24T00:12:16.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:12:15 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T00:12:16.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:12:15 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T00:12:16.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:12:15 smithi193 ceph-mon[36878]: pgmap v287: 1 pgs: 1 active+clean; 577 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-24T00:12:16.238 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:16.239+0000 7fb2dbe00640 1 -- 172.21.15.118:0/1434875649 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb2d41038c0 msgr2=0x7fb2d4103ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:12:16.239 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:16.239+0000 7fb2dbe00640 1 --2- 172.21.15.118:0/1434875649 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb2d41038c0 0x7fb2d4103ce0 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7fb2c8009ca0 tx=0x7fb2c802c100 comp rx=0 tx=0).stop 2024-03-24T00:12:16.239 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:16.239+0000 7fb2dbe00640 1 -- 172.21.15.118:0/1434875649 shutdown_connections 2024-03-24T00:12:16.239 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:16.239+0000 7fb2dbe00640 1 --2- 172.21.15.118:0/1434875649 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb2d4105470 0x7fb2d410bd00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:12:16.239 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:16.239+0000 7fb2dbe00640 1 --2- 172.21.15.118:0/1434875649 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fb2d4104ab0 0x7fb2d4104f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:12:16.239 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:16.239+0000 7fb2dbe00640 1 --2- 172.21.15.118:0/1434875649 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb2d41038c0 0x7fb2d4103ce0 unknown :-1 s=CLOSED pgs=133 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:12:16.239 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:16.239+0000 7fb2dbe00640 1 -- 172.21.15.118:0/1434875649 >> 172.21.15.118:0/1434875649 conn(0x7fb2d40fee50 msgr2=0x7fb2d4101270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:12:16.239 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:16.239+0000 7fb2dbe00640 1 -- 172.21.15.118:0/1434875649 shutdown_connections 2024-03-24T00:12:16.239 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:16.239+0000 7fb2dbe00640 1 -- 172.21.15.118:0/1434875649 wait complete. 2024-03-24T00:12:16.241 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:16.240+0000 7fb2dbe00640 1 Processor -- start 2024-03-24T00:12:16.241 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:16.240+0000 7fb2dbe00640 1 -- start start 2024-03-24T00:12:16.241 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:16.240+0000 7fb2dbe00640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb2d41038c0 0x7fb2d4199bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:12:16.241 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:16.240+0000 7fb2dbe00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb2d4104ab0 0x7fb2d419a110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:12:16.241 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:16.240+0000 7fb2dbe00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fb2d4105470 0x7fb2d41a1180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:12:16.241 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:16.240+0000 7fb2dbe00640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb2d410f510 con 0x7fb2d41038c0 2024-03-24T00:12:16.241 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:16.240+0000 7fb2dbe00640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb2d410f390 con 0x7fb2d4104ab0 2024-03-24T00:12:16.242 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:16.240+0000 7fb2dbe00640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb2d410f690 con 0x7fb2d4105470 2024-03-24T00:12:16.242 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:16.240+0000 7fb2d9400640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb2d41038c0 0x7fb2d4199bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:12:16.242 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:16.240+0000 7fb2d8a00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb2d4104ab0 0x7fb2d419a110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:12:16.242 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:16.240+0000 7fb2d9e00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fb2d4105470 0x7fb2d41a1180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:12:16.242 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:16.241+0000 7fb2d9e00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fb2d4105470 0x7fb2d41a1180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.118:51070/0 (socket says 172.21.15.118:51070) 2024-03-24T00:12:16.242 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:16.241+0000 7fb2d8a00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb2d4104ab0 0x7fb2d419a110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.118:47640/0 (socket says 172.21.15.118:47640) 2024-03-24T00:12:16.242 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:16.241+0000 7fb2d9400640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb2d41038c0 0x7fb2d4199bd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.118:3300/0 says I am v2:172.21.15.118:39074/0 (socket says 172.21.15.118:39074) 2024-03-24T00:12:16.242 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:16.241+0000 7fb2d9e00640 1 -- 172.21.15.118:0/632274590 learned_addr learned my addr 172.21.15.118:0/632274590 (peer_addr_for_me v2:172.21.15.118:0/0) 2024-03-24T00:12:16.242 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:16.241+0000 7fb2d9400640 1 -- 172.21.15.118:0/632274590 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fb2d4105470 msgr2=0x7fb2d41a1180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:12:16.242 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:16.241+0000 7fb2d9400640 1 --2- 172.21.15.118:0/632274590 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fb2d4105470 0x7fb2d41a1180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:12:16.242 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:16.241+0000 7fb2d9400640 1 -- 172.21.15.118:0/632274590 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb2d4104ab0 msgr2=0x7fb2d419a110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:12:16.242 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:16.241+0000 7fb2d9400640 1 --2- 172.21.15.118:0/632274590 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb2d4104ab0 0x7fb2d419a110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:12:16.243 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:16.241+0000 7fb2d9400640 1 -- 172.21.15.118:0/632274590 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb2d41a17f0 con 0x7fb2d41038c0 2024-03-24T00:12:16.243 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:16.241+0000 7fb2d8a00640 1 --2- 172.21.15.118:0/632274590 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb2d4104ab0 0x7fb2d419a110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T00:12:16.243 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:16.241+0000 7fb2d9e00640 1 --2- 172.21.15.118:0/632274590 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fb2d4105470 0x7fb2d41a1180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T00:12:16.243 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:16.241+0000 7fb2d9400640 1 --2- 172.21.15.118:0/632274590 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb2d41038c0 0x7fb2d4199bd0 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7fb2c8037040 tx=0x7fb2c8035d10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:12:16.243 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:16.241+0000 7fb2d2000640 1 -- 172.21.15.118:0/632274590 <== mon.0 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb2c8039040 con 0x7fb2d41038c0 2024-03-24T00:12:16.243 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:16.242+0000 7fb2d2000640 1 -- 172.21.15.118:0/632274590 <== mon.0 v2:172.21.15.118:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fb2c8003970 con 0x7fb2d41038c0 2024-03-24T00:12:16.243 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:16.242+0000 7fb2dbe00640 1 -- 172.21.15.118:0/632274590 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb2d41a1a80 con 0x7fb2d41038c0 2024-03-24T00:12:16.243 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:16.242+0000 7fb2d2000640 1 -- 172.21.15.118:0/632274590 <== mon.0 v2:172.21.15.118:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb2c80043c0 con 0x7fb2d41038c0 2024-03-24T00:12:16.243 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:16.242+0000 7fb2dbe00640 1 -- 172.21.15.118:0/632274590 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb2d41a1f40 con 0x7fb2d41038c0 2024-03-24T00:12:16.243 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:16.243+0000 7fb2dbe00640 1 -- 172.21.15.118:0/632274590 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb2d410acb0 con 0x7fb2d41038c0 2024-03-24T00:12:16.244 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:16.245+0000 7fb2d2000640 1 -- 172.21.15.118:0/632274590 <== mon.0 v2:172.21.15.118:3300/0 4 ==== mgrmap(e 16) v1 ==== 137291+0+0 (secure 0 0 0) 0x7fb2c8046030 con 0x7fb2d41038c0 2024-03-24T00:12:16.246 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:16.246+0000 7fb2d2000640 1 --2- 172.21.15.118:0/632274590 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7fb2bc0a2800 0x7fb2bc0a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:12:16.246 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:16.246+0000 7fb2d2000640 1 -- 172.21.15.118:0/632274590 <== mon.0 v2:172.21.15.118:3300/0 5 ==== osd_map(82..82 src has 1..82) v4 ==== 7127+0+0 (secure 0 0 0) 0x7fb2c80efff0 con 0x7fb2d41038c0 2024-03-24T00:12:16.246 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:16.246+0000 7fb2d8a00640 1 --2- 172.21.15.118:0/632274590 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7fb2bc0a2800 0x7fb2bc0a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:12:16.246 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:16.247+0000 7fb2d8a00640 1 --2- 172.21.15.118:0/632274590 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7fb2bc0a2800 0x7fb2bc0a4cc0 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7fb2d419b0e0 tx=0x7fb2c0005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:12:16.250 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:16.251+0000 7fb2d2000640 1 -- 172.21.15.118:0/632274590 <== mon.0 v2:172.21.15.118:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7fb2c80bd540 con 0x7fb2d41038c0 2024-03-24T00:12:16.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:12:15 smithi118 ceph-mon[31535]: from='client.24415 -' entity='client.admin' cmd=[{"prefix": "orch apply mds", "fs_name": "all", "placement": "12;smithi118=a;smithi118=d;smithi118=g;smithi118=j;smithi189=b;smithi189=e;smithi189=h;smithi189=k;smithi193=c;smithi193=f;smithi193=i;smithi193=l", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T00:12:16.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:12:15 smithi118 ceph-mon[31535]: Saving service mds.all spec with placement smithi118=a;smithi118=d;smithi118=g;smithi118=j;smithi189=b;smithi189=e;smithi189=h;smithi189=k;smithi193=c;smithi193=f;smithi193=i;smithi193=l;count:12 2024-03-24T00:12:16.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:12:15 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T00:12:16.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:12:15 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T00:12:16.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:12:15 smithi118 ceph-mon[31535]: pgmap v287: 1 pgs: 1 active+clean; 577 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-24T00:12:16.449 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:16.450+0000 7fb2dbe00640 1 -- 172.21.15.118:0/632274590 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fb2d40622a0 con 0x7fb2d41038c0 2024-03-24T00:12:16.452 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:16.453+0000 7fb2d2000640 1 -- 172.21.15.118:0/632274590 <== mon.0 v2:172.21.15.118:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v82) v1 ==== 74+0+18524 (secure 0 0 0) 0x7fb2c80bce40 con 0x7fb2d41038c0 2024-03-24T00:12:16.453 INFO:teuthology.orchestra.run.smithi118.stdout: 2024-03-24T00:12:16.453 INFO:teuthology.orchestra.run.smithi118.stdout:{"epoch":82,"fsid":"d79a5420-e971-11ee-95cd-87774f69a715","created":"2024-03-24T00:03:58.648524+0000","modified":"2024-03-24T00:12:04.819275+0000","last_up_change":"2024-03-24T00:12:03.809000+0000","last_in_change":"2024-03-24T00:11:44.389344+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-24T00:07:50.190965+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"c9590426-3901-4f68-a5ef-d57aa8fbac92","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6802","nonce":557690061},{"type":"v1","addr":"172.21.15.118:6803","nonce":557690061}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6804","nonce":557690061},{"type":"v1","addr":"172.21.15.118:6805","nonce":557690061}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6808","nonce":557690061},{"type":"v1","addr":"172.21.15.118:6809","nonce":557690061}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6806","nonce":557690061},{"type":"v1","addr":"172.21.15.118:6807","nonce":557690061}]},"public_addr":"172.21.15.118:6803/557690061","cluster_addr":"172.21.15.118:6805/557690061","heartbeat_back_addr":"172.21.15.118:6809/557690061","heartbeat_front_addr":"172.21.15.118:6807/557690061","state":["exists","up"]},{"osd":1,"uuid":"fc7f827e-ecf6-4661-aad0-7a22951892e0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":42,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6800","nonce":4203029485},{"type":"v1","addr":"172.21.15.189:6801","nonce":4203029485}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6802","nonce":4203029485},{"type":"v1","addr":"172.21.15.189:6803","nonce":4203029485}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6806","nonce":4203029485},{"type":"v1","addr":"172.21.15.189:6807","nonce":4203029485}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6804","nonce":4203029485},{"type":"v1","addr":"172.21.15.189:6805","nonce":4203029485}]},"public_addr":"172.21.15.189:6801/4203029485","cluster_addr":"172.21.15.189:6803/4203029485","heartbeat_back_addr":"172.21.15.189:6807/4203029485","heartbeat_front_addr":"172.21.15.189:6805/4203029485","state":["exists","up"]},{"osd":2,"uuid":"e73ccfe9-0e5b-4734-a154-a2f0d829fcfb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6800","nonce":3699076971},{"type":"v1","addr":"172.21.15.193:6801","nonce":3699076971}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6802","nonce":3699076971},{"type":"v1","addr":"172.21.15.193:6803","nonce":3699076971}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6806","nonce":3699076971},{"type":"v1","addr":"172.21.15.193:6807","nonce":3699076971}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6804","nonce":3699076971},{"type":"v1","addr":"172.21.15.193:6805","nonce":3699076971}]},"public_addr":"172.21.15.193:6801/3699076971","cluster_addr":"172.21.15.193:6803/3699076971","heartbeat_back_addr":"172.21.15.193:6807/3699076971","heartbeat_front_addr":"172.21.15.193:6805/3699076971","state":["exists","up"]},{"osd":3,"uuid":"63fed43b-43ce-4031-8957-7fe94725c1ed","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6810","nonce":4059617010},{"type":"v1","addr":"172.21.15.118:6811","nonce":4059617010}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6812","nonce":4059617010},{"type":"v1","addr":"172.21.15.118:6813","nonce":4059617010}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6816","nonce":4059617010},{"type":"v1","addr":"172.21.15.118:6817","nonce":4059617010}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6814","nonce":4059617010},{"type":"v1","addr":"172.21.15.118:6815","nonce":4059617010}]},"public_addr":"172.21.15.118:6811/4059617010","cluster_addr":"172.21.15.118:6813/4059617010","heartbeat_back_addr":"172.21.15.118:6817/4059617010","heartbeat_front_addr":"172.21.15.118:6815/4059617010","state":["exists","up"]},{"osd":4,"uuid":"57377f66-c3b4-43c1-8e8e-bc873672906d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":34,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6808","nonce":4075247068},{"type":"v1","addr":"172.21.15.189:6809","nonce":4075247068}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6810","nonce":4075247068},{"type":"v1","addr":"172.21.15.189:6811","nonce":4075247068}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6814","nonce":4075247068},{"type":"v1","addr":"172.21.15.189:6815","nonce":4075247068}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6812","nonce":4075247068},{"type":"v1","addr":"172.21.15.189:6813","nonce":4075247068}]},"public_addr":"172.21.15.189:6809/4075247068","cluster_addr":"172.21.15.189:6811/4075247068","heartbeat_back_addr":"172.21.15.189:6815/4075247068","heartbeat_front_addr":"172.21.15.189:6813/4075247068","state":["exists","up"]},{"osd":5,"uuid":"91970976-2193-4224-81c0-dc96cb022c85","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":39,"up_thru":54,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6808","nonce":2920363402},{"type":"v1","addr":"172.21.15.193:6809","nonce":2920363402}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6810","nonce":2920363402},{"type":"v1","addr":"172.21.15.193:6811","nonce":2920363402}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6814","nonce":2920363402},{"type":"v1","addr":"172.21.15.193:6815","nonce":2920363402}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6812","nonce":2920363402},{"type":"v1","addr":"172.21.15.193:6813","nonce":2920363402}]},"public_addr":"172.21.15.193:6809/2920363402","cluster_addr":"172.21.15.193:6811/2920363402","heartbeat_back_addr":"172.21.15.193:6815/2920363402","heartbeat_front_addr":"172.21.15.193:6813/2920363402","state":["exists","up"]},{"osd":6,"uuid":"0b77b196-8674-41c6-ad3d-726b5d01c8fd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":47,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6818","nonce":431354576},{"type":"v1","addr":"172.21.15.118:6819","nonce":431354576}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6820","nonce":431354576},{"type":"v1","addr":"172.21.15.118:6821","nonce":431354576}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6824","nonce":431354576},{"type":"v1","addr":"172.21.15.118:6825","nonce":431354576}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6822","nonce":431354576},{"type":"v1","addr":"172.21.15.118:6823","nonce":431354576}]},"public_addr":"172.21.15.118:6819/431354576","cluster_addr":"172.21.15.118:6821/431354576","heartbeat_back_addr":"172.21.15.118:6825/431354576","heartbeat_front_addr":"172.21.15.118:6823/431354576","state":["exists","up"]},{"osd":7,"uuid":"1fbbd38f-4e8e-4d4c-8e98-66f6862bc4bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":52,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6816","nonce":3667611284},{"type":"v1","addr":"172.21.15.189:6817","nonce":3667611284}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6818","nonce":3667611284},{"type":"v1","addr":"172.21.15.189:6819","nonce":3667611284}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6822","nonce":3667611284},{"type":"v1","addr":"172.21.15.189:6823","nonce":3667611284}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6820","nonce":3667611284},{"type":"v1","addr":"172.21.15.189:6821","nonce":3667611284}]},"public_addr":"172.21.15.189:6817/3667611284","cluster_addr":"172.21.15.189:6819/3667611284","heartbeat_back_addr":"172.21.15.189:6823/3667611284","heartbeat_front_addr":"172.21.15.189:6821/3667611284","state":["exists","up"]},{"osd":8,"uuid":"6462c2b5-e4b3-4b0b-bcfa-4de2c89e93e0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":60,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6816","nonce":3853800910},{"type":"v1","addr":"172.21.15.193:6817","nonce":3853800910}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6818","nonce":3853800910},{"type":"v1","addr":"172.21.15.193:6819","nonce":3853800910}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6822","nonce":3853800910},{"type":"v1","addr":"172.21.15.193:6823","nonce":3853800910}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6820","nonce":3853800910},{"type":"v1","addr":"172.21.15.193:6821","nonce":3853800910}]},"public_addr":"172.21.15.193:6817/3853800910","cluster_addr":"172.21.15.193:6819/3853800910","heartbeat_back_addr":"172.21.15.193:6823/3853800910","heartbeat_front_addr":"172.21.15.193:6821/3853800910","state":["exists","up"]},{"osd":9,"uuid":"54be277f-3596-422f-9de5-9005a8571c6f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":68,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6826","nonce":3248293936},{"type":"v1","addr":"172.21.15.118:6827","nonce":3248293936}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6828","nonce":3248293936},{"type":"v1","addr":"172.21.15.118:6829","nonce":3248293936}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6832","nonce":3248293936},{"type":"v1","addr":"172.21.15.118:6833","nonce":3248293936}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6830","nonce":3248293936},{"type":"v1","addr":"172.21.15.118:6831","nonce":3248293936}]},"public_addr":"172.21.15.118:6827/3248293936","cluster_addr":"172.21.15.118:6829/3248293936","heartbeat_back_addr":"172.21.15.118:6833/3248293936","heartbeat_front_addr":"172.21.15.118:6831/3248293936","state":["exists","up"]},{"osd":10,"uuid":"7ba69d69-ac41-44ce-bc39-c049a8ca5900","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":73,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6824","nonce":2112195760},{"type":"v1","addr":"172.21.15.189:6825","nonce":2112195760}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6826","nonce":2112195760},{"type":"v1","addr":"172.21.15.189:6827","nonce":2112195760}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6830","nonce":2112195760},{"type":"v1","addr":"172.21.15.189:6831","nonce":2112195760}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6828","nonce":2112195760},{"type":"v1","addr":"172.21.15.189:6829","nonce":2112195760}]},"public_addr":"172.21.15.189:6825/2112195760","cluster_addr":"172.21.15.189:6827/2112195760","heartbeat_back_addr":"172.21.15.189:6831/2112195760","heartbeat_front_addr":"172.21.15.189:6829/2112195760","state":["exists","up"]},{"osd":11,"uuid":"186f0b18-6b22-4eae-83b7-b21bbcce0817","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":81,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6824","nonce":3880515510},{"type":"v1","addr":"172.21.15.193:6825","nonce":3880515510}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6826","nonce":3880515510},{"type":"v1","addr":"172.21.15.193:6827","nonce":3880515510}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6830","nonce":3880515510},{"type":"v1","addr":"172.21.15.193:6831","nonce":3880515510}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6828","nonce":3880515510},{"type":"v1","addr":"172.21.15.193:6829","nonce":3880515510}]},"public_addr":"172.21.15.193:6825/3880515510","cluster_addr":"172.21.15.193:6827/3880515510","heartbeat_back_addr":"172.21.15.193:6831/3880515510","heartbeat_front_addr":"172.21.15.193:6829/3880515510","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:06:58.105543+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:07:23.795866+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:07:47.426050+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:08:19.427073+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:08:45.816082+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:09:11.081523+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:09:44.485077+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:10:11.338972+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:10:36.234024+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:11:09.730432+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:11:37.059083+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:12:02.371947+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.118:6801/389249272":"2024-03-25T00:05:05.458796+0000","172.21.15.118:6800/389249272":"2024-03-25T00:05:05.458796+0000","172.21.15.118:0/2101854694":"2024-03-25T00:05:05.458796+0000","172.21.15.118:0/630779359":"2024-03-25T00:05:05.458796+0000","172.21.15.118:0/2158921029":"2024-03-25T00:04:31.283725+0000","172.21.15.118:0/1003670455":"2024-03-25T00:05:05.458796+0000","172.21.15.118:0/368712314":"2024-03-25T00:04:31.283725+0000","172.21.15.118:0/2004775597":"2024-03-25T00:04:31.283725+0000","172.21.15.118:6800/1463147955":"2024-03-25T00:04:31.283725+0000","172.21.15.118:6801/1463147955":"2024-03-25T00:04:31.283725+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-24T00:12:16.455 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:16.455+0000 7fb2dbe00640 1 -- 172.21.15.118:0/632274590 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7fb2bc0a2800 msgr2=0x7fb2bc0a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:12:16.455 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:16.455+0000 7fb2dbe00640 1 --2- 172.21.15.118:0/632274590 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7fb2bc0a2800 0x7fb2bc0a4cc0 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7fb2d419b0e0 tx=0x7fb2c0005f90 comp rx=0 tx=0).stop 2024-03-24T00:12:16.455 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:16.455+0000 7fb2dbe00640 1 -- 172.21.15.118:0/632274590 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb2d41038c0 msgr2=0x7fb2d4199bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:12:16.455 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:16.455+0000 7fb2dbe00640 1 --2- 172.21.15.118:0/632274590 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb2d41038c0 0x7fb2d4199bd0 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7fb2c8037040 tx=0x7fb2c8035d10 comp rx=0 tx=0).stop 2024-03-24T00:12:16.456 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:16.455+0000 7fb2dbe00640 1 -- 172.21.15.118:0/632274590 shutdown_connections 2024-03-24T00:12:16.456 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:16.455+0000 7fb2dbe00640 1 --2- 172.21.15.118:0/632274590 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7fb2bc0a2800 0x7fb2bc0a4cc0 unknown :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:12:16.456 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:16.455+0000 7fb2dbe00640 1 --2- 172.21.15.118:0/632274590 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fb2d4105470 0x7fb2d41a1180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:12:16.456 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:16.455+0000 7fb2dbe00640 1 --2- 172.21.15.118:0/632274590 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb2d4104ab0 0x7fb2d419a110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:12:16.456 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:16.455+0000 7fb2dbe00640 1 --2- 172.21.15.118:0/632274590 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb2d41038c0 0x7fb2d4199bd0 unknown :-1 s=CLOSED pgs=134 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:12:16.456 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:16.455+0000 7fb2dbe00640 1 -- 172.21.15.118:0/632274590 >> 172.21.15.118:0/632274590 conn(0x7fb2d40fee50 msgr2=0x7fb2d4107e80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:12:16.456 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:16.456+0000 7fb2dbe00640 1 -- 172.21.15.118:0/632274590 shutdown_connections 2024-03-24T00:12:16.456 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:16.456+0000 7fb2dbe00640 1 -- 172.21.15.118:0/632274590 wait complete. 2024-03-24T00:12:16.477 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-24T00:07:50.190965+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-24T00:12:16.478 DEBUG:teuthology.orchestra.run.smithi118:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-24T00:12:16.736 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:16.736+0000 7f130f400640 1 -- 172.21.15.118:0/1988835352 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f13081038c0 msgr2=0x7f1308103ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:12:16.736 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:16.736+0000 7f130f400640 1 --2- 172.21.15.118:0/1988835352 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f13081038c0 0x7f1308103ce0 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7f12fc009900 tx=0x7f12fc02bc50 comp rx=0 tx=0).stop 2024-03-24T00:12:16.736 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:16.736+0000 7f130f400640 1 -- 172.21.15.118:0/1988835352 shutdown_connections 2024-03-24T00:12:16.736 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:16.736+0000 7f130f400640 1 --2- 172.21.15.118:0/1988835352 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1308105470 0x7f130810bd00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:12:16.737 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:16.736+0000 7f130f400640 1 --2- 172.21.15.118:0/1988835352 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f1308104ab0 0x7f1308104f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:12:16.737 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:16.736+0000 7f130f400640 1 --2- 172.21.15.118:0/1988835352 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f13081038c0 0x7f1308103ce0 unknown :-1 s=CLOSED pgs=135 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:12:16.737 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:16.736+0000 7f130f400640 1 -- 172.21.15.118:0/1988835352 >> 172.21.15.118:0/1988835352 conn(0x7f13080fee50 msgr2=0x7f1308101270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:12:16.737 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:16.736+0000 7f130f400640 1 -- 172.21.15.118:0/1988835352 shutdown_connections 2024-03-24T00:12:16.737 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:16.736+0000 7f130f400640 1 -- 172.21.15.118:0/1988835352 wait complete. 2024-03-24T00:12:16.739 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:16.737+0000 7f130f400640 1 Processor -- start 2024-03-24T00:12:16.739 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:16.737+0000 7f130f400640 1 -- start start 2024-03-24T00:12:16.739 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:16.738+0000 7f130f400640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f13081038c0 0x7f1308199be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:12:16.740 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:16.738+0000 7f130f400640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f1308104ab0 0x7f130819a120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:12:16.740 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:16.738+0000 7f130f400640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f1308105470 0x7f13081a1190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:12:16.740 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:16.738+0000 7f130f400640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f130810f560 con 0x7f1308105470 2024-03-24T00:12:16.740 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:16.738+0000 7f130f400640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f130810f3e0 con 0x7f13081038c0 2024-03-24T00:12:16.740 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:16.738+0000 7f130f400640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f130810f6e0 con 0x7f1308104ab0 2024-03-24T00:12:16.740 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:16.738+0000 7f1307e00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f1308104ab0 0x7f130819a120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:12:16.740 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:16.738+0000 7f1307e00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f1308104ab0 0x7f130819a120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.118:51084/0 (socket says 172.21.15.118:51084) 2024-03-24T00:12:16.740 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:16.738+0000 7f1307e00640 1 -- 172.21.15.118:0/3498624058 learned_addr learned my addr 172.21.15.118:0/3498624058 (peer_addr_for_me v2:172.21.15.118:0/0) 2024-03-24T00:12:16.740 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:16.738+0000 7f130ca00640 1 --2- 172.21.15.118:0/3498624058 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f13081038c0 0x7f1308199be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:12:16.740 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:16.738+0000 7f130d400640 1 --2- 172.21.15.118:0/3498624058 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f1308105470 0x7f13081a1190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:12:16.740 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:16.738+0000 7f1307e00640 1 -- 172.21.15.118:0/3498624058 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f13081038c0 msgr2=0x7f1308199be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:12:16.740 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:16.738+0000 7f1307e00640 1 --2- 172.21.15.118:0/3498624058 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f13081038c0 0x7f1308199be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:12:16.740 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:16.738+0000 7f1307e00640 1 -- 172.21.15.118:0/3498624058 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f1308105470 msgr2=0x7f13081a1190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:12:16.741 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:16.738+0000 7f1307e00640 1 --2- 172.21.15.118:0/3498624058 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f1308105470 0x7f13081a1190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:12:16.741 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:16.738+0000 7f1307e00640 1 -- 172.21.15.118:0/3498624058 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f13081a1800 con 0x7f1308104ab0 2024-03-24T00:12:16.741 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:16.738+0000 7f1307e00640 1 --2- 172.21.15.118:0/3498624058 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f1308104ab0 0x7f130819a120 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f12f400ea60 tx=0x7f12f400ef20 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:12:16.741 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:16.739+0000 7f1305600640 1 -- 172.21.15.118:0/3498624058 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f12f400cde0 con 0x7f1308104ab0 2024-03-24T00:12:16.741 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:16.739+0000 7f130f400640 1 -- 172.21.15.118:0/3498624058 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f13081a1af0 con 0x7f1308104ab0 2024-03-24T00:12:16.741 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:16.739+0000 7f1305600640 1 -- 172.21.15.118:0/3498624058 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f12f40035f0 con 0x7f1308104ab0 2024-03-24T00:12:16.741 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:16.739+0000 7f1305600640 1 -- 172.21.15.118:0/3498624058 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f12f4010670 con 0x7f1308104ab0 2024-03-24T00:12:16.741 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:16.739+0000 7f130f400640 1 -- 172.21.15.118:0/3498624058 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f13081a2090 con 0x7f1308104ab0 2024-03-24T00:12:16.741 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:16.740+0000 7f130f400640 1 -- 172.21.15.118:0/3498624058 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f130810acb0 con 0x7f1308104ab0 2024-03-24T00:12:16.742 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:16.742+0000 7f1305600640 1 -- 172.21.15.118:0/3498624058 <== mon.1 v2:172.21.15.193:3300/0 4 ==== mgrmap(e 16) v1 ==== 137291+0+0 (secure 0 0 0) 0x7f12f40037a0 con 0x7f1308104ab0 2024-03-24T00:12:16.743 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:16.744+0000 7f1305600640 1 --2- 172.21.15.118:0/3498624058 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7f12f00a2800 0x7f12f00a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:12:16.743 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:16.744+0000 7f1305600640 1 -- 172.21.15.118:0/3498624058 <== mon.1 v2:172.21.15.193:3300/0 5 ==== osd_map(82..82 src has 1..82) v4 ==== 7127+0+0 (secure 0 0 0) 0x7f12f40ce940 con 0x7f1308104ab0 2024-03-24T00:12:16.743 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:16.744+0000 7f130ca00640 1 --2- 172.21.15.118:0/3498624058 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7f12f00a2800 0x7f12f00a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:12:16.744 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:16.745+0000 7f130ca00640 1 --2- 172.21.15.118:0/3498624058 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7f12f00a2800 0x7f12f00a4cc0 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f12fc02c190 tx=0x7f12fc000bc0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:12:16.747 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:16.747+0000 7f1305600640 1 -- 172.21.15.118:0/3498624058 <== mon.1 v2:172.21.15.193:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f12f409bde0 con 0x7f1308104ab0 2024-03-24T00:12:16.941 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:16.942+0000 7f130f400640 1 -- 172.21.15.118:0/3498624058 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f13080622a0 con 0x7f1308104ab0 2024-03-24T00:12:16.942 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:16.943+0000 7f1305600640 1 -- 172.21.15.118:0/3498624058 <== mon.1 v2:172.21.15.193:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v82) v1 ==== 93+0+10 (secure 0 0 0) 0x7f12f40860f0 con 0x7f1308104ab0 2024-03-24T00:12:16.943 INFO:teuthology.orchestra.run.smithi118.stdout:pg_num: 1 2024-03-24T00:12:16.945 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:16.945+0000 7f130f400640 1 -- 172.21.15.118:0/3498624058 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7f12f00a2800 msgr2=0x7f12f00a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:12:16.945 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:16.945+0000 7f130f400640 1 --2- 172.21.15.118:0/3498624058 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7f12f00a2800 0x7f12f00a4cc0 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f12fc02c190 tx=0x7f12fc000bc0 comp rx=0 tx=0).stop 2024-03-24T00:12:16.945 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:16.945+0000 7f130f400640 1 -- 172.21.15.118:0/3498624058 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f1308104ab0 msgr2=0x7f130819a120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:12:16.945 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:16.945+0000 7f130f400640 1 --2- 172.21.15.118:0/3498624058 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f1308104ab0 0x7f130819a120 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f12f400ea60 tx=0x7f12f400ef20 comp rx=0 tx=0).stop 2024-03-24T00:12:16.945 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:16.945+0000 7f130f400640 1 -- 172.21.15.118:0/3498624058 shutdown_connections 2024-03-24T00:12:16.945 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:16.945+0000 7f130f400640 1 --2- 172.21.15.118:0/3498624058 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7f12f00a2800 0x7f12f00a4cc0 unknown :-1 s=CLOSED pgs=121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:12:16.946 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:16.945+0000 7f130f400640 1 --2- 172.21.15.118:0/3498624058 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f1308105470 0x7f13081a1190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:12:16.946 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:16.945+0000 7f130f400640 1 --2- 172.21.15.118:0/3498624058 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f1308104ab0 0x7f130819a120 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:12:16.946 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:16.945+0000 7f130f400640 1 --2- 172.21.15.118:0/3498624058 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f13081038c0 0x7f1308199be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:12:16.946 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:16.945+0000 7f130f400640 1 -- 172.21.15.118:0/3498624058 >> 172.21.15.118:0/3498624058 conn(0x7f13080fee50 msgr2=0x7f1308107e80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:12:16.946 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:16.946+0000 7f130f400640 1 -- 172.21.15.118:0/3498624058 shutdown_connections 2024-03-24T00:12:16.946 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:16.946+0000 7f130f400640 1 -- 172.21.15.118:0/3498624058 wait complete. 2024-03-24T00:12:16.964 DEBUG:teuthology.orchestra.run.smithi118:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-24T00:12:17.078 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:12:16 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T00:12:17.078 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:12:16 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T00:12:17.078 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:12:16 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T00:12:17.078 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:12:16 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.l", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-24T00:12:17.078 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:12:16 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.l", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-24T00:12:17.078 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:12:16 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T00:12:17.078 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:12:16 smithi193 ceph-mon[36878]: Deploying daemon mds.l on smithi193 2024-03-24T00:12:17.078 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:12:16 smithi193 ceph-mon[36878]: from='client.? 172.21.15.118:0/632274590' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-24T00:12:17.078 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:12:16 smithi193 ceph-mon[36878]: from='client.? 172.21.15.118:0/3498624058' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-24T00:12:17.171 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:17.171+0000 7fb7dca00640 1 -- 172.21.15.118:0/4234181367 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7d8109580 msgr2=0x7fb7d810ba10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:12:17.172 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:17.171+0000 7fb7dca00640 1 --2- 172.21.15.118:0/4234181367 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7d8109580 0x7fb7d810ba10 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7fb7d0009ca0 tx=0x7fb7d002c0f0 comp rx=0 tx=0).stop 2024-03-24T00:12:17.172 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:17.171+0000 7fb7dca00640 1 -- 172.21.15.118:0/4234181367 shutdown_connections 2024-03-24T00:12:17.172 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:17.171+0000 7fb7dca00640 1 --2- 172.21.15.118:0/4234181367 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7d8109580 0x7fb7d810ba10 unknown :-1 s=CLOSED pgs=136 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:12:17.172 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:17.172+0000 7fb7dca00640 1 --2- 172.21.15.118:0/4234181367 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fb7d8074100 0x7fb7d8074580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:12:17.172 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:17.172+0000 7fb7dca00640 1 --2- 172.21.15.118:0/4234181367 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb7d80756e0 0x7fb7d8073b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:12:17.172 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:17.172+0000 7fb7dca00640 1 -- 172.21.15.118:0/4234181367 >> 172.21.15.118:0/4234181367 conn(0x7fb7d80fcc50 msgr2=0x7fb7d80ff070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:12:17.172 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:17.172+0000 7fb7dca00640 1 -- 172.21.15.118:0/4234181367 shutdown_connections 2024-03-24T00:12:17.172 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:17.172+0000 7fb7dca00640 1 -- 172.21.15.118:0/4234181367 wait complete. 2024-03-24T00:12:17.174 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:17.172+0000 7fb7dca00640 1 Processor -- start 2024-03-24T00:12:17.174 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:17.173+0000 7fb7dca00640 1 -- start start 2024-03-24T00:12:17.174 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:17.173+0000 7fb7dca00640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7d8074100 0x7fb7d8199b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:12:17.175 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:17.173+0000 7fb7dca00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb7d80756e0 0x7fb7d819a0b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:12:17.175 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:17.173+0000 7fb7dca00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fb7d8109580 0x7fb7d81a1120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:12:17.175 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:17.173+0000 7fb7dca00640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb7d810f520 con 0x7fb7d8074100 2024-03-24T00:12:17.175 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:17.173+0000 7fb7dca00640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb7d810f3a0 con 0x7fb7d80756e0 2024-03-24T00:12:17.175 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:17.173+0000 7fb7dca00640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb7d810f6a0 con 0x7fb7d8109580 2024-03-24T00:12:17.175 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:17.173+0000 7fb7d5e00640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7d8074100 0x7fb7d8199b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:12:17.175 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:17.173+0000 7fb7d5e00640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7d8074100 0x7fb7d8199b70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.118:3300/0 says I am v2:172.21.15.118:39100/0 (socket says 172.21.15.118:39100) 2024-03-24T00:12:17.175 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:17.173+0000 7fb7d5e00640 1 -- 172.21.15.118:0/1621358704 learned_addr learned my addr 172.21.15.118:0/1621358704 (peer_addr_for_me v2:172.21.15.118:0/0) 2024-03-24T00:12:17.175 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:17.173+0000 7fb7d6800640 1 --2- 172.21.15.118:0/1621358704 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fb7d8109580 0x7fb7d81a1120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:12:17.175 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:17.173+0000 7fb7d5400640 1 --2- 172.21.15.118:0/1621358704 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb7d80756e0 0x7fb7d819a0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:12:17.175 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:17.174+0000 7fb7d5e00640 1 -- 172.21.15.118:0/1621358704 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fb7d8109580 msgr2=0x7fb7d81a1120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:12:17.175 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:17.174+0000 7fb7d5e00640 1 --2- 172.21.15.118:0/1621358704 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fb7d8109580 0x7fb7d81a1120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:12:17.176 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:17.174+0000 7fb7d5e00640 1 -- 172.21.15.118:0/1621358704 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb7d80756e0 msgr2=0x7fb7d819a0b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:12:17.176 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:17.174+0000 7fb7d5e00640 1 --2- 172.21.15.118:0/1621358704 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb7d80756e0 0x7fb7d819a0b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:12:17.176 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:17.174+0000 7fb7d5e00640 1 -- 172.21.15.118:0/1621358704 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb7d81a1790 con 0x7fb7d8074100 2024-03-24T00:12:17.176 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:17.174+0000 7fb7d6800640 1 --2- 172.21.15.118:0/1621358704 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fb7d8109580 0x7fb7d81a1120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T00:12:17.176 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:17.174+0000 7fb7d5e00640 1 --2- 172.21.15.118:0/1621358704 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7d8074100 0x7fb7d8199b70 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7fb7c800e9f0 tx=0x7fb7c800eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:12:17.176 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:17.174+0000 7fb7cea00640 1 -- 172.21.15.118:0/1621358704 <== mon.0 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb7c800cd80 con 0x7fb7d8074100 2024-03-24T00:12:17.176 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:17.174+0000 7fb7cea00640 1 -- 172.21.15.118:0/1621358704 <== mon.0 v2:172.21.15.118:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fb7c8010910 con 0x7fb7d8074100 2024-03-24T00:12:17.176 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:17.174+0000 7fb7dca00640 1 -- 172.21.15.118:0/1621358704 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb7d81a1a80 con 0x7fb7d8074100 2024-03-24T00:12:17.176 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:17.174+0000 7fb7cea00640 1 -- 172.21.15.118:0/1621358704 <== mon.0 v2:172.21.15.118:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb7c8018990 con 0x7fb7d8074100 2024-03-24T00:12:17.176 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:17.174+0000 7fb7dca00640 1 -- 172.21.15.118:0/1621358704 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb7d81a1fd0 con 0x7fb7d8074100 2024-03-24T00:12:17.177 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:17.175+0000 7fb7dca00640 1 -- 172.21.15.118:0/1621358704 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb7d80688e0 con 0x7fb7d8074100 2024-03-24T00:12:17.177 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:17.177+0000 7fb7cea00640 1 -- 172.21.15.118:0/1621358704 <== mon.0 v2:172.21.15.118:3300/0 4 ==== mgrmap(e 16) v1 ==== 137291+0+0 (secure 0 0 0) 0x7fb7c8010ac0 con 0x7fb7d8074100 2024-03-24T00:12:17.178 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:17.179+0000 7fb7cea00640 1 --2- 172.21.15.118:0/1621358704 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7fb7bc0a2800 0x7fb7bc0a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:12:17.178 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:17.179+0000 7fb7cea00640 1 -- 172.21.15.118:0/1621358704 <== mon.0 v2:172.21.15.118:3300/0 5 ==== osd_map(82..82 src has 1..82) v4 ==== 7127+0+0 (secure 0 0 0) 0x7fb7c80ce9b0 con 0x7fb7d8074100 2024-03-24T00:12:17.178 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:17.179+0000 7fb7d5400640 1 --2- 172.21.15.118:0/1621358704 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7fb7bc0a2800 0x7fb7bc0a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:12:17.179 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:17.180+0000 7fb7d5400640 1 --2- 172.21.15.118:0/1621358704 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7fb7bc0a2800 0x7fb7bc0a4cc0 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7fb7d819b080 tx=0x7fb7c4005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:12:17.182 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:17.183+0000 7fb7cea00640 1 -- 172.21.15.118:0/1621358704 <== mon.0 v2:172.21.15.118:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7fb7c809b800 con 0x7fb7d8074100 2024-03-24T00:12:17.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:12:16 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T00:12:17.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:12:16 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T00:12:17.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:12:16 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T00:12:17.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:12:16 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.l", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-24T00:12:17.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:12:16 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.l", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-24T00:12:17.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:12:16 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T00:12:17.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:12:16 smithi189 ceph-mon[36187]: Deploying daemon mds.l on smithi193 2024-03-24T00:12:17.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:12:16 smithi189 ceph-mon[36187]: from='client.? 172.21.15.118:0/632274590' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-24T00:12:17.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:12:16 smithi189 ceph-mon[36187]: from='client.? 172.21.15.118:0/3498624058' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-24T00:12:17.422 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:17.423+0000 7fb7dca00640 1 -- 172.21.15.118:0/1621358704 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fb7d80622a0 con 0x7fb7d8074100 2024-03-24T00:12:17.423 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:17.423+0000 7fb7cea00640 1 -- 172.21.15.118:0/1621358704 <== mon.0 v2:172.21.15.118:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 1 v1) v1 ==== 93+0+535 (secure 0 0 0) 0x7fb7c809f8a0 con 0x7fb7d8074100 2024-03-24T00:12:17.423 INFO:teuthology.orchestra.run.smithi118.stdout: 2024-03-24T00:12:17.423 INFO:teuthology.orchestra.run.smithi118.stdout:{"epoch":1,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[]} 2024-03-24T00:12:17.423 INFO:teuthology.orchestra.run.smithi118.stderr:dumped fsmap epoch 1 2024-03-24T00:12:17.426 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:17.426+0000 7fb7dca00640 1 -- 172.21.15.118:0/1621358704 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7fb7bc0a2800 msgr2=0x7fb7bc0a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:12:17.426 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:17.426+0000 7fb7dca00640 1 --2- 172.21.15.118:0/1621358704 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7fb7bc0a2800 0x7fb7bc0a4cc0 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7fb7d819b080 tx=0x7fb7c4005f90 comp rx=0 tx=0).stop 2024-03-24T00:12:17.426 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:17.426+0000 7fb7dca00640 1 -- 172.21.15.118:0/1621358704 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7d8074100 msgr2=0x7fb7d8199b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:12:17.426 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:17.426+0000 7fb7dca00640 1 --2- 172.21.15.118:0/1621358704 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7d8074100 0x7fb7d8199b70 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7fb7c800e9f0 tx=0x7fb7c800eeb0 comp rx=0 tx=0).stop 2024-03-24T00:12:17.426 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:17.426+0000 7fb7dca00640 1 -- 172.21.15.118:0/1621358704 shutdown_connections 2024-03-24T00:12:17.426 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:17.426+0000 7fb7dca00640 1 --2- 172.21.15.118:0/1621358704 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7fb7bc0a2800 0x7fb7bc0a4cc0 unknown :-1 s=CLOSED pgs=122 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:12:17.426 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:17.426+0000 7fb7dca00640 1 --2- 172.21.15.118:0/1621358704 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fb7d8109580 0x7fb7d81a1120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:12:17.426 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:17.426+0000 7fb7dca00640 1 --2- 172.21.15.118:0/1621358704 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb7d80756e0 0x7fb7d819a0b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:12:17.426 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:17.426+0000 7fb7dca00640 1 --2- 172.21.15.118:0/1621358704 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb7d8074100 0x7fb7d8199b70 unknown :-1 s=CLOSED pgs=137 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:12:17.427 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:17.426+0000 7fb7dca00640 1 -- 172.21.15.118:0/1621358704 >> 172.21.15.118:0/1621358704 conn(0x7fb7d80fcc50 msgr2=0x7fb7d8105480 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:12:17.427 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:17.426+0000 7fb7dca00640 1 -- 172.21.15.118:0/1621358704 shutdown_connections 2024-03-24T00:12:17.427 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:17.426+0000 7fb7dca00640 1 -- 172.21.15.118:0/1621358704 wait complete. 2024-03-24T00:12:17.442 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:12:16 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T00:12:17.442 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:12:16 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T00:12:17.442 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:12:16 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T00:12:17.442 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:12:16 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.l", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-24T00:12:17.442 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:12:16 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.l", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-24T00:12:17.442 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:12:16 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T00:12:17.442 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:12:16 smithi118 ceph-mon[31535]: Deploying daemon mds.l on smithi193 2024-03-24T00:12:17.443 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:12:16 smithi118 ceph-mon[31535]: from='client.? 172.21.15.118:0/632274590' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-24T00:12:17.443 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:12:16 smithi118 ceph-mon[31535]: from='client.? 172.21.15.118:0/3498624058' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-24T00:12:18.089 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:12:17 smithi193 ceph-mon[36878]: from='client.? 172.21.15.118:0/1621358704' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-24T00:12:18.090 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:12:17 smithi193 ceph-mon[36878]: pgmap v288: 1 pgs: 1 active+clean; 577 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-24T00:12:18.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:12:17 smithi118 ceph-mon[31535]: from='client.? 172.21.15.118:0/1621358704' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-24T00:12:18.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:12:17 smithi118 ceph-mon[31535]: pgmap v288: 1 pgs: 1 active+clean; 577 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-24T00:12:18.453 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:12:17 smithi189 ceph-mon[36187]: from='client.? 172.21.15.118:0/1621358704' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-24T00:12:18.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:12:17 smithi189 ceph-mon[36187]: pgmap v288: 1 pgs: 1 active+clean; 577 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-24T00:12:18.460 INFO:journalctl@ceph.mds.l.smithi193.stdout:Mar 24 00:12:18 smithi193 systemd[1]: Starting Ceph mds.l for d79a5420-e971-11ee-95cd-87774f69a715... 2024-03-24T00:12:18.754 INFO:journalctl@ceph.mds.l.smithi193.stdout:Mar 24 00:12:18 smithi193 podman[67146]: 2024-03-24 00:12:18.393340307 +0000 UTC m=+0.017905475 image pull c664afe56bb01db4f9afa9532c20598e201e664f2e9ba2be50db0d52674a26d2 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc 2024-03-24T00:12:18.755 INFO:journalctl@ceph.mds.l.smithi193.stdout:Mar 24 00:12:18 smithi193 podman[67146]: 2024-03-24 00:12:18.50208476 +0000 UTC m=+0.126649913 container create af712abd88afa4d7323b2ec8d40f9e391e6b5a25c411254b91eb86f79dd3c586 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-l, org.label-schema.license=GPLv2, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240319, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, maintainer=Guillaume Abrioux , GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, io.buildah.version=1.33.5) 2024-03-24T00:12:18.755 INFO:journalctl@ceph.mds.l.smithi193.stdout:Mar 24 00:12:18 smithi193 podman[67146]: 2024-03-24 00:12:18.729571663 +0000 UTC m=+0.354136810 container init af712abd88afa4d7323b2ec8d40f9e391e6b5a25c411254b91eb86f79dd3c586 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-l, org.label-schema.vendor=CentOS, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.build-date=20240319, GIT_BRANCH=HEAD, ceph=True, io.buildah.version=1.33.5, GIT_CLEAN=True, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=) 2024-03-24T00:12:18.755 INFO:journalctl@ceph.mds.l.smithi193.stdout:Mar 24 00:12:18 smithi193 podman[67146]: 2024-03-24 00:12:18.732068372 +0000 UTC m=+0.356633517 container start af712abd88afa4d7323b2ec8d40f9e391e6b5a25c411254b91eb86f79dd3c586 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-l, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240319, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, ceph=True, org.label-schema.license=GPLv2, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-24T00:12:18.755 INFO:journalctl@ceph.mds.l.smithi193.stdout:Mar 24 00:12:18 smithi193 ceph-mds[67165]: -- 172.21.15.193:0/2378310355 <== mon.1 v2:172.21.15.193:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x5557d95eba40 con 0x5557d95bec00 2024-03-24T00:12:19.211 INFO:journalctl@ceph.mds.l.smithi193.stdout:Mar 24 00:12:18 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-l[67161]: starting mds.l at 2024-03-24T00:12:19.211 INFO:journalctl@ceph.mds.l.smithi193.stdout:Mar 24 00:12:18 smithi193 bash[67146]: af712abd88afa4d7323b2ec8d40f9e391e6b5a25c411254b91eb86f79dd3c586 2024-03-24T00:12:19.211 INFO:journalctl@ceph.mds.l.smithi193.stdout:Mar 24 00:12:18 smithi193 systemd[1]: Started Ceph mds.l for d79a5420-e971-11ee-95cd-87774f69a715. 2024-03-24T00:12:19.446 DEBUG:teuthology.orchestra.run.smithi118:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-24T00:12:19.667 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:19.667+0000 7f4610200640 1 -- 172.21.15.118:0/2761424422 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4608104ad0 msgr2=0x7f4608104f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:12:19.667 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:19.667+0000 7f4610200640 1 --2- 172.21.15.118:0/2761424422 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4608104ad0 0x7f4608104f50 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7f45f4009900 tx=0x7f45f402bc30 comp rx=0 tx=0).stop 2024-03-24T00:12:19.667 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:19.667+0000 7f4610200640 1 -- 172.21.15.118:0/2761424422 shutdown_connections 2024-03-24T00:12:19.667 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:19.667+0000 7f4610200640 1 --2- 172.21.15.118:0/2761424422 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f4608105490 0x7f460810bd20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:12:19.667 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:19.667+0000 7f4610200640 1 --2- 172.21.15.118:0/2761424422 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4608104ad0 0x7f4608104f50 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:12:19.667 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:19.667+0000 7f4610200640 1 --2- 172.21.15.118:0/2761424422 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f46081038e0 0x7f4608103d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:12:19.667 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:19.667+0000 7f4610200640 1 -- 172.21.15.118:0/2761424422 >> 172.21.15.118:0/2761424422 conn(0x7f46080fee50 msgr2=0x7f4608101290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:12:19.667 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:19.667+0000 7f4610200640 1 -- 172.21.15.118:0/2761424422 shutdown_connections 2024-03-24T00:12:19.667 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:19.667+0000 7f4610200640 1 -- 172.21.15.118:0/2761424422 wait complete. 2024-03-24T00:12:19.668 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:19.668+0000 7f4610200640 1 Processor -- start 2024-03-24T00:12:19.669 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:19.668+0000 7f4610200640 1 -- start start 2024-03-24T00:12:19.669 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:19.669+0000 7f4610200640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f46081038e0 0x7f4608199bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:12:19.669 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:19.669+0000 7f4610200640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f4608104ad0 0x7f460819a130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:12:19.669 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:19.669+0000 7f4610200640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f4608105490 0x7f46081a11a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:12:19.669 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:19.669+0000 7f4610200640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f460810f5b0 con 0x7f4608104ad0 2024-03-24T00:12:19.669 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:19.669+0000 7f4610200640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f460810f430 con 0x7f46081038e0 2024-03-24T00:12:19.669 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:19.669+0000 7f4610200640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f460810f730 con 0x7f4608105490 2024-03-24T00:12:19.669 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:19.669+0000 7f460ce00640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f4608104ad0 0x7f460819a130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:12:19.669 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:19.669+0000 7f460d800640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f46081038e0 0x7f4608199bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:12:19.669 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:19.669+0000 7f460d800640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f46081038e0 0x7f4608199bf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.118:47710/0 (socket says 172.21.15.118:47710) 2024-03-24T00:12:19.669 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:19.669+0000 7f460d800640 1 -- 172.21.15.118:0/3797746604 learned_addr learned my addr 172.21.15.118:0/3797746604 (peer_addr_for_me v2:172.21.15.118:0/0) 2024-03-24T00:12:19.669 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:19.669+0000 7f460e200640 1 --2- 172.21.15.118:0/3797746604 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f4608105490 0x7f46081a11a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:12:19.670 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:19.669+0000 7f460d800640 1 -- 172.21.15.118:0/3797746604 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f4608105490 msgr2=0x7f46081a11a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:12:19.671 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:19.669+0000 7f460d800640 1 --2- 172.21.15.118:0/3797746604 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f4608105490 0x7f46081a11a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:12:19.671 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:19.669+0000 7f460d800640 1 -- 172.21.15.118:0/3797746604 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f4608104ad0 msgr2=0x7f460819a130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:12:19.671 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:19.669+0000 7f460d800640 1 --2- 172.21.15.118:0/3797746604 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f4608104ad0 0x7f460819a130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:12:19.671 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:19.669+0000 7f460d800640 1 -- 172.21.15.118:0/3797746604 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f46081a1810 con 0x7f46081038e0 2024-03-24T00:12:19.671 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:19.670+0000 7f460d800640 1 --2- 172.21.15.118:0/3797746604 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f46081038e0 0x7f4608199bf0 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f45fc00e9c0 tx=0x7f45fc00ee80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:12:19.671 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:19.670+0000 7f4606000640 1 -- 172.21.15.118:0/3797746604 <== mon.2 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f45fc00cd80 con 0x7f46081038e0 2024-03-24T00:12:19.671 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:19.670+0000 7f4610200640 1 -- 172.21.15.118:0/3797746604 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f46081a1b00 con 0x7f46081038e0 2024-03-24T00:12:19.671 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:19.670+0000 7f4606000640 1 -- 172.21.15.118:0/3797746604 <== mon.2 v2:172.21.15.189:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f45fc0035f0 con 0x7f46081038e0 2024-03-24T00:12:19.671 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:19.670+0000 7f4606000640 1 -- 172.21.15.118:0/3797746604 <== mon.2 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f45fc010690 con 0x7f46081038e0 2024-03-24T00:12:19.671 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:19.670+0000 7f4610200640 1 -- 172.21.15.118:0/3797746604 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f46081a2070 con 0x7f46081038e0 2024-03-24T00:12:19.671 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:19.671+0000 7f4610200640 1 -- 172.21.15.118:0/3797746604 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f460810ac00 con 0x7f46081038e0 2024-03-24T00:12:19.672 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:19.673+0000 7f4606000640 1 -- 172.21.15.118:0/3797746604 <== mon.2 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 16) v1 ==== 137291+0+0 (secure 0 0 0) 0x7f45fc0037a0 con 0x7f46081038e0 2024-03-24T00:12:19.674 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:19.674+0000 7f4606000640 1 --2- 172.21.15.118:0/3797746604 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7f45f00a2880 0x7f45f00a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:12:19.674 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:19.675+0000 7f4606000640 1 -- 172.21.15.118:0/3797746604 <== mon.2 v2:172.21.15.189:3300/0 5 ==== osd_map(82..82 src has 1..82) v4 ==== 7127+0+0 (secure 0 0 0) 0x7f45fc0ce990 con 0x7f46081038e0 2024-03-24T00:12:19.674 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:19.675+0000 7f460ce00640 1 --2- 172.21.15.118:0/3797746604 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7f45f00a2880 0x7f45f00a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:12:19.675 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:19.676+0000 7f460ce00640 1 --2- 172.21.15.118:0/3797746604 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7f45f00a2880 0x7f45f00a4d40 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f45f402c240 tx=0x7f45f402df90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:12:19.678 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:19.679+0000 7f4606000640 1 -- 172.21.15.118:0/3797746604 <== mon.2 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f45fc09be30 con 0x7f46081038e0 2024-03-24T00:12:19.825 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:12:19 smithi189 ceph-mon[36187]: mds.? [v2:172.21.15.193:6832/1023428412,v1:172.21.15.193:6833/1023428412] up:boot 2024-03-24T00:12:19.825 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:12:19 smithi189 ceph-mon[36187]: fsmap 1 up:standby 2024-03-24T00:12:19.825 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:12:19 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "l"}]: dispatch 2024-03-24T00:12:19.825 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:12:19 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T00:12:19.825 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:12:19 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T00:12:19.825 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:12:19 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T00:12:19.825 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:12:19 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.k", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-24T00:12:19.825 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:12:19 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.k", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-24T00:12:19.826 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:12:19 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T00:12:19.826 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:12:19 smithi189 ceph-mon[36187]: Deploying daemon mds.k on smithi189 2024-03-24T00:12:19.826 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:12:19 smithi189 ceph-mon[36187]: pgmap v289: 1 pgs: 1 active+clean; 577 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-24T00:12:19.923 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:19.924+0000 7f4610200640 1 -- 172.21.15.118:0/3797746604 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f460802d070 con 0x7f46081038e0 2024-03-24T00:12:19.924 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:19.925+0000 7f4606000640 1 -- 172.21.15.118:0/3797746604 <== mon.2 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 2 v2) v1 ==== 93+0+1328 (secure 0 0 0) 0x7f45fc09b730 con 0x7f46081038e0 2024-03-24T00:12:19.924 INFO:teuthology.orchestra.run.smithi118.stdout: 2024-03-24T00:12:19.925 INFO:teuthology.orchestra.run.smithi118.stdout:{"epoch":2,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":24439,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.193:6833/1023428412","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6832","nonce":1023428412},{"type":"v1","addr":"172.21.15.193:6833","nonce":1023428412}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2}],"filesystems":[]} 2024-03-24T00:12:19.925 INFO:teuthology.orchestra.run.smithi118.stderr:dumped fsmap epoch 2 2024-03-24T00:12:19.927 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:19.928+0000 7f4610200640 1 -- 172.21.15.118:0/3797746604 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7f45f00a2880 msgr2=0x7f45f00a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:12:19.928 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:19.928+0000 7f4610200640 1 --2- 172.21.15.118:0/3797746604 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7f45f00a2880 0x7f45f00a4d40 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f45f402c240 tx=0x7f45f402df90 comp rx=0 tx=0).stop 2024-03-24T00:12:19.928 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:19.928+0000 7f4610200640 1 -- 172.21.15.118:0/3797746604 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f46081038e0 msgr2=0x7f4608199bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:12:19.928 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:19.928+0000 7f4610200640 1 --2- 172.21.15.118:0/3797746604 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f46081038e0 0x7f4608199bf0 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7f45fc00e9c0 tx=0x7f45fc00ee80 comp rx=0 tx=0).stop 2024-03-24T00:12:19.928 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:19.928+0000 7f4610200640 1 -- 172.21.15.118:0/3797746604 shutdown_connections 2024-03-24T00:12:19.928 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:19.928+0000 7f4610200640 1 --2- 172.21.15.118:0/3797746604 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7f45f00a2880 0x7f45f00a4d40 unknown :-1 s=CLOSED pgs=124 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:12:19.928 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:19.928+0000 7f4610200640 1 --2- 172.21.15.118:0/3797746604 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f4608105490 0x7f46081a11a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:12:19.928 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:19.928+0000 7f4610200640 1 --2- 172.21.15.118:0/3797746604 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f4608104ad0 0x7f460819a130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:12:19.928 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:19.928+0000 7f4610200640 1 --2- 172.21.15.118:0/3797746604 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f46081038e0 0x7f4608199bf0 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:12:19.928 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:19.928+0000 7f4610200640 1 -- 172.21.15.118:0/3797746604 >> 172.21.15.118:0/3797746604 conn(0x7f46080fee50 msgr2=0x7f4608107dd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:12:19.928 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:19.928+0000 7f4610200640 1 -- 172.21.15.118:0/3797746604 shutdown_connections 2024-03-24T00:12:19.928 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:19.928+0000 7f4610200640 1 -- 172.21.15.118:0/3797746604 wait complete. 2024-03-24T00:12:19.944 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:12:19 smithi118 ceph-mon[31535]: mds.? [v2:172.21.15.193:6832/1023428412,v1:172.21.15.193:6833/1023428412] up:boot 2024-03-24T00:12:19.944 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:12:19 smithi118 ceph-mon[31535]: fsmap 1 up:standby 2024-03-24T00:12:19.944 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:12:19 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "l"}]: dispatch 2024-03-24T00:12:19.944 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:12:19 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T00:12:19.944 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:12:19 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T00:12:19.944 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:12:19 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T00:12:19.944 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:12:19 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.k", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-24T00:12:19.944 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:12:19 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.k", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-24T00:12:19.944 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:12:19 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T00:12:19.944 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:12:19 smithi118 ceph-mon[31535]: Deploying daemon mds.k on smithi189 2024-03-24T00:12:19.944 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:12:19 smithi118 ceph-mon[31535]: pgmap v289: 1 pgs: 1 active+clean; 577 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-24T00:12:20.112 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:12:19 smithi193 ceph-mon[36878]: mds.? [v2:172.21.15.193:6832/1023428412,v1:172.21.15.193:6833/1023428412] up:boot 2024-03-24T00:12:20.112 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:12:19 smithi193 ceph-mon[36878]: fsmap 1 up:standby 2024-03-24T00:12:20.112 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:12:19 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "l"}]: dispatch 2024-03-24T00:12:20.113 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:12:19 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T00:12:20.113 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:12:19 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T00:12:20.113 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:12:19 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T00:12:20.113 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:12:19 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.k", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-24T00:12:20.113 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:12:19 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.k", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-24T00:12:20.113 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:12:19 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T00:12:20.113 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:12:19 smithi193 ceph-mon[36878]: Deploying daemon mds.k on smithi189 2024-03-24T00:12:20.113 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:12:19 smithi193 ceph-mon[36878]: pgmap v289: 1 pgs: 1 active+clean; 577 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-24T00:12:20.953 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:12:20 smithi189 ceph-mon[36187]: from='client.? 172.21.15.118:0/3797746604' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-24T00:12:21.124 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:12:20 smithi193 ceph-mon[36878]: from='client.? 172.21.15.118:0/3797746604' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-24T00:12:21.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:12:20 smithi118 ceph-mon[31535]: from='client.? 172.21.15.118:0/3797746604' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-24T00:12:21.427 INFO:journalctl@ceph.mds.k.smithi189.stdout:Mar 24 00:12:21 smithi189 systemd[1]: Starting Ceph mds.k for d79a5420-e971-11ee-95cd-87774f69a715... 2024-03-24T00:12:21.704 INFO:journalctl@ceph.mds.k.smithi189.stdout:Mar 24 00:12:21 smithi189 podman[66023]: 2024-03-24 00:12:21.330585434 +0000 UTC m=+0.017872881 image pull c664afe56bb01db4f9afa9532c20598e201e664f2e9ba2be50db0d52674a26d2 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc 2024-03-24T00:12:21.704 INFO:journalctl@ceph.mds.k.smithi189.stdout:Mar 24 00:12:21 smithi189 podman[66023]: 2024-03-24 00:12:21.436224392 +0000 UTC m=+0.123511870 container create 89f5182d7c26e85a3b40978255d4bf9f4b2b1e9d17632ed765e7e31e55c907d4 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-k, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240319, ceph=True) 2024-03-24T00:12:21.704 INFO:journalctl@ceph.mds.k.smithi189.stdout:Mar 24 00:12:21 smithi189 podman[66023]: 2024-03-24 00:12:21.647163491 +0000 UTC m=+0.334450943 container init 89f5182d7c26e85a3b40978255d4bf9f4b2b1e9d17632ed765e7e31e55c907d4 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-k, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240319, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_CLEAN=True, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-24T00:12:21.704 INFO:journalctl@ceph.mds.k.smithi189.stdout:Mar 24 00:12:21 smithi189 podman[66023]: 2024-03-24 00:12:21.649889514 +0000 UTC m=+0.337176955 container start 89f5182d7c26e85a3b40978255d4bf9f4b2b1e9d17632ed765e7e31e55c907d4 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-k, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.license=GPLv2, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.build-date=20240319, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=) 2024-03-24T00:12:21.704 INFO:journalctl@ceph.mds.k.smithi189.stdout:Mar 24 00:12:21 smithi189 ceph-mds[66042]: -- 172.21.15.189:0/3720310338 <== mon.2 v2:172.21.15.189:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x5617d913fa40 con 0x5617d9112800 2024-03-24T00:12:21.704 INFO:journalctl@ceph.mds.k.smithi189.stdout:Mar 24 00:12:21 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-k[66038]: starting mds.k at 2024-03-24T00:12:21.947 DEBUG:teuthology.orchestra.run.smithi118:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-24T00:12:22.145 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:22.145+0000 7ff40da00640 1 -- 172.21.15.118:0/4226134779 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7ff408103260 msgr2=0x7ff408109af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:12:22.145 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:22.145+0000 7ff40da00640 1 --2- 172.21.15.118:0/4226134779 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7ff408103260 0x7ff408109af0 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7ff400009be0 tx=0x7ff40002bef0 comp rx=0 tx=0).stop 2024-03-24T00:12:22.145 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:22.145+0000 7ff40da00640 1 -- 172.21.15.118:0/4226134779 shutdown_connections 2024-03-24T00:12:22.145 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:22.145+0000 7ff40da00640 1 --2- 172.21.15.118:0/4226134779 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7ff408103260 0x7ff408109af0 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:12:22.145 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:22.145+0000 7ff40da00640 1 --2- 172.21.15.118:0/4226134779 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff4081028a0 0x7ff408102d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:12:22.145 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:22.145+0000 7ff40da00640 1 --2- 172.21.15.118:0/4226134779 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7ff4081016b0 0x7ff408101ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:12:22.145 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:22.145+0000 7ff40da00640 1 -- 172.21.15.118:0/4226134779 >> 172.21.15.118:0/4226134779 conn(0x7ff4080fcc80 msgr2=0x7ff4080ff0a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:12:22.145 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:22.145+0000 7ff40da00640 1 -- 172.21.15.118:0/4226134779 shutdown_connections 2024-03-24T00:12:22.146 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:22.145+0000 7ff40da00640 1 -- 172.21.15.118:0/4226134779 wait complete. 2024-03-24T00:12:22.147 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:22.146+0000 7ff40da00640 1 Processor -- start 2024-03-24T00:12:22.147 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:22.147+0000 7ff40da00640 1 -- start start 2024-03-24T00:12:22.147 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:22.147+0000 7ff40da00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff4081016b0 0x7ff408075450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:12:22.147 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:22.147+0000 7ff40da00640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7ff4081028a0 0x7ff408073a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:12:22.147 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:22.147+0000 7ff40da00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7ff408103260 0x7ff408073fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:12:22.147 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:22.147+0000 7ff40da00640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff40810d310 con 0x7ff4081028a0 2024-03-24T00:12:22.147 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:22.147+0000 7ff40da00640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff40810d190 con 0x7ff4081016b0 2024-03-24T00:12:22.148 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:22.147+0000 7ff40da00640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff40810d490 con 0x7ff408103260 2024-03-24T00:12:22.148 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:22.147+0000 7ff406000640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7ff4081028a0 0x7ff408073a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:12:22.148 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:22.147+0000 7ff406a00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff4081016b0 0x7ff408075450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:12:22.148 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:22.147+0000 7ff406000640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7ff4081028a0 0x7ff408073a90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.118:3300/0 says I am v2:172.21.15.118:39116/0 (socket says 172.21.15.118:39116) 2024-03-24T00:12:22.148 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:22.147+0000 7ff407400640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7ff408103260 0x7ff408073fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:12:22.148 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:22.147+0000 7ff406000640 1 -- 172.21.15.118:0/126947587 learned_addr learned my addr 172.21.15.118:0/126947587 (peer_addr_for_me v2:172.21.15.118:0/0) 2024-03-24T00:12:22.148 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:22.148+0000 7ff406a00640 1 -- 172.21.15.118:0/126947587 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7ff408103260 msgr2=0x7ff408073fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:12:22.148 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:22.148+0000 7ff406a00640 1 --2- 172.21.15.118:0/126947587 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7ff408103260 0x7ff408073fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:12:22.148 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:22.148+0000 7ff406a00640 1 -- 172.21.15.118:0/126947587 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7ff4081028a0 msgr2=0x7ff408073a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:12:22.148 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:22.148+0000 7ff406a00640 1 --2- 172.21.15.118:0/126947587 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7ff4081028a0 0x7ff408073a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:12:22.148 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:22.148+0000 7ff406a00640 1 -- 172.21.15.118:0/126947587 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff408074860 con 0x7ff4081016b0 2024-03-24T00:12:22.148 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:22.148+0000 7ff406000640 1 --2- 172.21.15.118:0/126947587 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7ff4081028a0 0x7ff408073a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T00:12:22.149 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:22.148+0000 7ff406a00640 1 --2- 172.21.15.118:0/126947587 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff4081016b0 0x7ff408075450 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7ff3f000e9c0 tx=0x7ff3f000ee80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:12:22.149 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:22.148+0000 7ff3ff400640 1 -- 172.21.15.118:0/126947587 <== mon.2 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff3f000ccc0 con 0x7ff4081016b0 2024-03-24T00:12:22.149 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:22.148+0000 7ff3ff400640 1 -- 172.21.15.118:0/126947587 <== mon.2 v2:172.21.15.189:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7ff3f00035f0 con 0x7ff4081016b0 2024-03-24T00:12:22.150 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:22.148+0000 7ff40da00640 1 -- 172.21.15.118:0/126947587 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff4081a1a80 con 0x7ff4081016b0 2024-03-24T00:12:22.150 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:22.148+0000 7ff3ff400640 1 -- 172.21.15.118:0/126947587 <== mon.2 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff3f000ce60 con 0x7ff4081016b0 2024-03-24T00:12:22.150 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:22.149+0000 7ff40da00640 1 -- 172.21.15.118:0/126947587 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff4081a2020 con 0x7ff4081016b0 2024-03-24T00:12:22.150 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:22.150+0000 7ff40da00640 1 -- 172.21.15.118:0/126947587 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff4081089d0 con 0x7ff4081016b0 2024-03-24T00:12:22.150 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:22.151+0000 7ff3ff400640 1 -- 172.21.15.118:0/126947587 <== mon.2 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 16) v1 ==== 137291+0+0 (secure 0 0 0) 0x7ff3f0004020 con 0x7ff4081016b0 2024-03-24T00:12:22.152 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:22.153+0000 7ff3ff400640 1 --2- 172.21.15.118:0/126947587 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7ff3ec0a2880 0x7ff3ec0a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:12:22.152 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:22.153+0000 7ff3ff400640 1 -- 172.21.15.118:0/126947587 <== mon.2 v2:172.21.15.189:3300/0 5 ==== osd_map(82..82 src has 1..82) v4 ==== 7127+0+0 (secure 0 0 0) 0x7ff3f00ce9b0 con 0x7ff4081016b0 2024-03-24T00:12:22.152 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:22.153+0000 7ff406000640 1 --2- 172.21.15.118:0/126947587 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7ff3ec0a2880 0x7ff3ec0a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:12:22.153 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:22.154+0000 7ff406000640 1 --2- 172.21.15.118:0/126947587 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7ff3ec0a2880 0x7ff3ec0a4d40 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7ff4080751c0 tx=0x7ff3f8009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:12:22.155 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:22.156+0000 7ff3ff400640 1 -- 172.21.15.118:0/126947587 <== mon.2 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7ff3f009be50 con 0x7ff4081016b0 2024-03-24T00:12:22.195 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:12:21 smithi118 ceph-mon[31535]: pgmap v290: 1 pgs: 1 active+clean; 577 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-24T00:12:22.203 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:12:21 smithi189 ceph-mon[36187]: pgmap v290: 1 pgs: 1 active+clean; 577 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-24T00:12:22.203 INFO:journalctl@ceph.mds.k.smithi189.stdout:Mar 24 00:12:21 smithi189 bash[66023]: 89f5182d7c26e85a3b40978255d4bf9f4b2b1e9d17632ed765e7e31e55c907d4 2024-03-24T00:12:22.204 INFO:journalctl@ceph.mds.k.smithi189.stdout:Mar 24 00:12:21 smithi189 systemd[1]: Started Ceph mds.k for d79a5420-e971-11ee-95cd-87774f69a715. 2024-03-24T00:12:22.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:12:21 smithi193 ceph-mon[36878]: pgmap v290: 1 pgs: 1 active+clean; 577 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-24T00:12:22.395 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:22.395+0000 7ff40da00640 1 -- 172.21.15.118:0/126947587 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7ff40802d070 con 0x7ff4081016b0 2024-03-24T00:12:22.395 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:22.395+0000 7ff3ff400640 1 -- 172.21.15.118:0/126947587 <== mon.2 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 3 v3) v1 ==== 93+0+2122 (secure 0 0 0) 0x7ff40802d070 con 0x7ff4081016b0 2024-03-24T00:12:22.395 INFO:teuthology.orchestra.run.smithi118.stdout: 2024-03-24T00:12:22.396 INFO:teuthology.orchestra.run.smithi118.stdout:{"epoch":3,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":24439,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.193:6833/1023428412","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6832","nonce":1023428412},{"type":"v1","addr":"172.21.15.193:6833","nonce":1023428412}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":24440,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.189:6833/2740024078","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6832","nonce":2740024078},{"type":"v1","addr":"172.21.15.189:6833","nonce":2740024078}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[]} 2024-03-24T00:12:22.396 INFO:teuthology.orchestra.run.smithi118.stderr:dumped fsmap epoch 3 2024-03-24T00:12:22.398 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:22.398+0000 7ff40da00640 1 -- 172.21.15.118:0/126947587 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7ff3ec0a2880 msgr2=0x7ff3ec0a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:12:22.398 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:22.398+0000 7ff40da00640 1 --2- 172.21.15.118:0/126947587 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7ff3ec0a2880 0x7ff3ec0a4d40 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7ff4080751c0 tx=0x7ff3f8009260 comp rx=0 tx=0).stop 2024-03-24T00:12:22.398 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:22.398+0000 7ff40da00640 1 -- 172.21.15.118:0/126947587 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff4081016b0 msgr2=0x7ff408075450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:12:22.398 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:22.398+0000 7ff40da00640 1 --2- 172.21.15.118:0/126947587 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff4081016b0 0x7ff408075450 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7ff3f000e9c0 tx=0x7ff3f000ee80 comp rx=0 tx=0).stop 2024-03-24T00:12:22.398 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:22.398+0000 7ff40da00640 1 -- 172.21.15.118:0/126947587 shutdown_connections 2024-03-24T00:12:22.398 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:22.398+0000 7ff40da00640 1 --2- 172.21.15.118:0/126947587 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7ff3ec0a2880 0x7ff3ec0a4d40 unknown :-1 s=CLOSED pgs=127 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:12:22.398 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:22.398+0000 7ff40da00640 1 --2- 172.21.15.118:0/126947587 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7ff408103260 0x7ff408073fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:12:22.398 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:22.398+0000 7ff40da00640 1 --2- 172.21.15.118:0/126947587 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7ff4081028a0 0x7ff408073a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:12:22.398 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:22.398+0000 7ff40da00640 1 --2- 172.21.15.118:0/126947587 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff4081016b0 0x7ff408075450 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:12:22.399 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:22.398+0000 7ff40da00640 1 -- 172.21.15.118:0/126947587 >> 172.21.15.118:0/126947587 conn(0x7ff4080fcc80 msgr2=0x7ff408105ba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:12:22.399 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:22.398+0000 7ff40da00640 1 -- 172.21.15.118:0/126947587 shutdown_connections 2024-03-24T00:12:22.399 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:22.398+0000 7ff40da00640 1 -- 172.21.15.118:0/126947587 wait complete. 2024-03-24T00:12:23.056 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:12:22 smithi118 ceph-mon[31535]: mds.? [v2:172.21.15.189:6832/2740024078,v1:172.21.15.189:6833/2740024078] up:boot 2024-03-24T00:12:23.056 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:12:22 smithi118 ceph-mon[31535]: fsmap 2 up:standby 2024-03-24T00:12:23.056 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:12:22 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "k"}]: dispatch 2024-03-24T00:12:23.056 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:12:22 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T00:12:23.056 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:12:22 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T00:12:23.057 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:12:22 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T00:12:23.057 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:12:22 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.j", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-24T00:12:23.057 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:12:22 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.j", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-24T00:12:23.057 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:12:22 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T00:12:23.057 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:12:22 smithi118 ceph-mon[31535]: Deploying daemon mds.j on smithi118 2024-03-24T00:12:23.057 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:12:22 smithi118 ceph-mon[31535]: from='client.? 172.21.15.118:0/126947587' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-24T00:12:23.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:12:22 smithi189 ceph-mon[36187]: mds.? [v2:172.21.15.189:6832/2740024078,v1:172.21.15.189:6833/2740024078] up:boot 2024-03-24T00:12:23.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:12:22 smithi189 ceph-mon[36187]: fsmap 2 up:standby 2024-03-24T00:12:23.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:12:22 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "k"}]: dispatch 2024-03-24T00:12:23.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:12:22 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T00:12:23.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:12:22 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T00:12:23.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:12:22 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T00:12:23.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:12:22 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.j", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-24T00:12:23.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:12:22 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.j", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-24T00:12:23.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:12:22 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T00:12:23.205 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:12:22 smithi189 ceph-mon[36187]: Deploying daemon mds.j on smithi118 2024-03-24T00:12:23.205 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:12:22 smithi189 ceph-mon[36187]: from='client.? 172.21.15.118:0/126947587' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-24T00:12:23.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:12:22 smithi193 ceph-mon[36878]: mds.? [v2:172.21.15.189:6832/2740024078,v1:172.21.15.189:6833/2740024078] up:boot 2024-03-24T00:12:23.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:12:22 smithi193 ceph-mon[36878]: fsmap 2 up:standby 2024-03-24T00:12:23.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:12:22 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "k"}]: dispatch 2024-03-24T00:12:23.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:12:22 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T00:12:23.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:12:22 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T00:12:23.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:12:22 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T00:12:23.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:12:22 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.j", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-24T00:12:23.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:12:22 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.j", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-24T00:12:23.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:12:22 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T00:12:23.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:12:22 smithi193 ceph-mon[36878]: Deploying daemon mds.j on smithi118 2024-03-24T00:12:23.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:12:22 smithi193 ceph-mon[36878]: from='client.? 172.21.15.118:0/126947587' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-24T00:12:24.057 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:12:23 smithi189 ceph-mon[36187]: pgmap v291: 1 pgs: 1 active+clean; 577 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-24T00:12:24.112 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:12:23 smithi118 ceph-mon[31535]: pgmap v291: 1 pgs: 1 active+clean; 577 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-24T00:12:24.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:12:23 smithi193 ceph-mon[36878]: pgmap v291: 1 pgs: 1 active+clean; 577 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-24T00:12:24.418 DEBUG:teuthology.orchestra.run.smithi118:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-24T00:12:24.687 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:24.687+0000 7feff4800640 1 -- 172.21.15.118:0/3112514859 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fefec104a70 msgr2=0x7fefec106e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:12:24.688 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:24.687+0000 7feff4800640 1 --2- 172.21.15.118:0/3112514859 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fefec104a70 0x7fefec106e60 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7fefdc009900 tx=0x7fefdc02bc40 comp rx=0 tx=0).stop 2024-03-24T00:12:24.688 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:24.687+0000 7feff4800640 1 -- 172.21.15.118:0/3112514859 shutdown_connections 2024-03-24T00:12:24.688 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:24.687+0000 7feff4800640 1 --2- 172.21.15.118:0/3112514859 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fefec1073a0 0x7fefec1097f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:12:24.688 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:24.687+0000 7feff4800640 1 --2- 172.21.15.118:0/3112514859 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fefec104a70 0x7fefec106e60 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:12:24.688 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:24.687+0000 7feff4800640 1 --2- 172.21.15.118:0/3112514859 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fefec102140 0x7fefec104530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:12:24.688 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:24.687+0000 7feff4800640 1 -- 172.21.15.118:0/3112514859 >> 172.21.15.118:0/3112514859 conn(0x7fefec0fbb50 msgr2=0x7fefec0fdf70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:12:24.688 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:24.688+0000 7feff4800640 1 -- 172.21.15.118:0/3112514859 shutdown_connections 2024-03-24T00:12:24.688 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:24.688+0000 7feff4800640 1 -- 172.21.15.118:0/3112514859 wait complete. 2024-03-24T00:12:24.689 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:24.690+0000 7feff4800640 1 Processor -- start 2024-03-24T00:12:24.689 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:24.690+0000 7feff4800640 1 -- start start 2024-03-24T00:12:24.689 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:24.690+0000 7feff4800640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fefec102140 0x7fefec110a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:12:24.689 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:24.690+0000 7feff4800640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fefec104a70 0x7fefec110f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:12:24.690 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:24.690+0000 7feff4800640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fefec1073a0 0x7fefec1adfa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:12:24.690 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:24.690+0000 7feff4800640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fefec101c50 con 0x7fefec1073a0 2024-03-24T00:12:24.690 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:24.690+0000 7feff4800640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fefec101ad0 con 0x7fefec104a70 2024-03-24T00:12:24.690 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:24.690+0000 7feff4800640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fefec101dd0 con 0x7fefec102140 2024-03-24T00:12:24.691 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:24.691+0000 7feff1e00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fefec102140 0x7fefec110a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:12:24.691 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:24.691+0000 7feff1e00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fefec102140 0x7fefec110a30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.118:57274/0 (socket says 172.21.15.118:57274) 2024-03-24T00:12:24.691 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:24.691+0000 7feff1e00640 1 -- 172.21.15.118:0/1202300667 learned_addr learned my addr 172.21.15.118:0/1202300667 (peer_addr_for_me v2:172.21.15.118:0/0) 2024-03-24T00:12:24.691 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:24.691+0000 7feff1400640 1 --2- 172.21.15.118:0/1202300667 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fefec104a70 0x7fefec110f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:12:24.692 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:24.692+0000 7feff2800640 1 --2- 172.21.15.118:0/1202300667 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fefec1073a0 0x7fefec1adfa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:12:24.692 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:24.692+0000 7feff1e00640 1 -- 172.21.15.118:0/1202300667 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fefec104a70 msgr2=0x7fefec110f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:12:24.692 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:24.692+0000 7feff1e00640 1 --2- 172.21.15.118:0/1202300667 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fefec104a70 0x7fefec110f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:12:24.692 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:24.692+0000 7feff1e00640 1 -- 172.21.15.118:0/1202300667 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fefec1073a0 msgr2=0x7fefec1adfa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:12:24.692 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:24.692+0000 7feff1e00640 1 --2- 172.21.15.118:0/1202300667 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fefec1073a0 0x7fefec1adfa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:12:24.692 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:24.692+0000 7feff1e00640 1 -- 172.21.15.118:0/1202300667 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fefec1ae4e0 con 0x7fefec102140 2024-03-24T00:12:24.693 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:24.692+0000 7feff2800640 1 --2- 172.21.15.118:0/1202300667 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fefec1073a0 0x7fefec1adfa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T00:12:24.693 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:24.693+0000 7feff1400640 1 --2- 172.21.15.118:0/1202300667 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fefec104a70 0x7fefec110f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T00:12:24.693 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:24.693+0000 7feff1e00640 1 --2- 172.21.15.118:0/1202300667 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fefec102140 0x7fefec110a30 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7fefe000e9f0 tx=0x7fefe000eeb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:12:24.693 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:24.693+0000 7fefe6a00640 1 -- 172.21.15.118:0/1202300667 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fefe000cd80 con 0x7fefec102140 2024-03-24T00:12:24.693 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:24.693+0000 7feff4800640 1 -- 172.21.15.118:0/1202300667 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fefec1ae6b0 con 0x7fefec102140 2024-03-24T00:12:24.693 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:24.694+0000 7fefe6a00640 1 -- 172.21.15.118:0/1202300667 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fefe0010910 con 0x7fefec102140 2024-03-24T00:12:24.693 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:24.694+0000 7fefe6a00640 1 -- 172.21.15.118:0/1202300667 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fefe00188f0 con 0x7fefec102140 2024-03-24T00:12:24.694 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:24.694+0000 7feff4800640 1 -- 172.21.15.118:0/1202300667 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fefec1aebd0 con 0x7fefec102140 2024-03-24T00:12:24.696 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:24.697+0000 7feff4800640 1 -- 172.21.15.118:0/1202300667 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fefec0688e0 con 0x7fefec102140 2024-03-24T00:12:24.698 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:24.698+0000 7fefe6a00640 1 -- 172.21.15.118:0/1202300667 <== mon.1 v2:172.21.15.193:3300/0 4 ==== mgrmap(e 16) v1 ==== 137291+0+0 (secure 0 0 0) 0x7fefe0010ac0 con 0x7fefec102140 2024-03-24T00:12:24.699 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:24.700+0000 7fefe6a00640 1 --2- 172.21.15.118:0/1202300667 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7fefd40a2800 0x7fefd40a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:12:24.699 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:24.700+0000 7fefe6a00640 1 -- 172.21.15.118:0/1202300667 <== mon.1 v2:172.21.15.193:3300/0 5 ==== osd_map(82..82 src has 1..82) v4 ==== 7127+0+0 (secure 0 0 0) 0x7fefe00cea00 con 0x7fefec102140 2024-03-24T00:12:24.700 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:24.700+0000 7feff1400640 1 --2- 172.21.15.118:0/1202300667 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7fefd40a2800 0x7fefd40a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:12:24.700 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:24.701+0000 7feff1400640 1 --2- 172.21.15.118:0/1202300667 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7fefd40a2800 0x7fefd40a4cc0 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7fefdc02c180 tx=0x7fefdc02cf90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:12:24.704 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:24.705+0000 7fefe6a00640 1 -- 172.21.15.118:0/1202300667 <== mon.1 v2:172.21.15.193:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7fefe009bf40 con 0x7fefec102140 2024-03-24T00:12:24.945 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:24.945+0000 7feff4800640 1 -- 172.21.15.118:0/1202300667 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fefec02d3b0 con 0x7fefec102140 2024-03-24T00:12:24.946 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:24.946+0000 7fefe6a00640 1 -- 172.21.15.118:0/1202300667 <== mon.1 v2:172.21.15.193:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 3 v3) v1 ==== 93+0+2122 (secure 0 0 0) 0x7fefe009b840 con 0x7fefec102140 2024-03-24T00:12:24.946 INFO:teuthology.orchestra.run.smithi118.stdout: 2024-03-24T00:12:24.946 INFO:teuthology.orchestra.run.smithi118.stdout:{"epoch":3,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":24439,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.193:6833/1023428412","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6832","nonce":1023428412},{"type":"v1","addr":"172.21.15.193:6833","nonce":1023428412}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":24440,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.189:6833/2740024078","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6832","nonce":2740024078},{"type":"v1","addr":"172.21.15.189:6833","nonce":2740024078}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[]} 2024-03-24T00:12:24.946 INFO:teuthology.orchestra.run.smithi118.stderr:dumped fsmap epoch 3 2024-03-24T00:12:24.948 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:24.949+0000 7feff4800640 1 -- 172.21.15.118:0/1202300667 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7fefd40a2800 msgr2=0x7fefd40a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:12:24.949 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:24.949+0000 7feff4800640 1 --2- 172.21.15.118:0/1202300667 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7fefd40a2800 0x7fefd40a4cc0 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7fefdc02c180 tx=0x7fefdc02cf90 comp rx=0 tx=0).stop 2024-03-24T00:12:24.949 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:24.949+0000 7feff4800640 1 -- 172.21.15.118:0/1202300667 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fefec102140 msgr2=0x7fefec110a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:12:24.949 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:24.949+0000 7feff4800640 1 --2- 172.21.15.118:0/1202300667 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fefec102140 0x7fefec110a30 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7fefe000e9f0 tx=0x7fefe000eeb0 comp rx=0 tx=0).stop 2024-03-24T00:12:24.949 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:24.949+0000 7feff4800640 1 -- 172.21.15.118:0/1202300667 shutdown_connections 2024-03-24T00:12:24.949 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:24.949+0000 7feff4800640 1 --2- 172.21.15.118:0/1202300667 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7fefd40a2800 0x7fefd40a4cc0 unknown :-1 s=CLOSED pgs=129 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:12:24.949 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:24.949+0000 7feff4800640 1 --2- 172.21.15.118:0/1202300667 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fefec1073a0 0x7fefec1adfa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:12:24.949 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:24.949+0000 7feff4800640 1 --2- 172.21.15.118:0/1202300667 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fefec104a70 0x7fefec110f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:12:24.949 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:24.949+0000 7feff4800640 1 --2- 172.21.15.118:0/1202300667 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fefec102140 0x7fefec110a30 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:12:24.949 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:24.949+0000 7feff4800640 1 -- 172.21.15.118:0/1202300667 >> 172.21.15.118:0/1202300667 conn(0x7fefec0fbb50 msgr2=0x7fefec102a80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:12:24.949 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:24.949+0000 7feff4800640 1 -- 172.21.15.118:0/1202300667 shutdown_connections 2024-03-24T00:12:24.949 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:24.949+0000 7feff4800640 1 -- 172.21.15.118:0/1202300667 wait complete. 2024-03-24T00:12:25.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:12:24 smithi118 ceph-mon[31535]: from='client.? 172.21.15.118:0/1202300667' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-24T00:12:25.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:12:24 smithi189 ceph-mon[36187]: from='client.? 172.21.15.118:0/1202300667' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-24T00:12:25.460 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:12:24 smithi193 ceph-mon[36878]: from='client.? 172.21.15.118:0/1202300667' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-24T00:12:25.585 INFO:journalctl@ceph.mds.j.smithi118.stdout:Mar 24 00:12:25 smithi118 podman[72610]: 2024-03-24 00:12:25.330271234 +0000 UTC m=+0.184550984 container create 4b581125b505cb60f3ec5e80ce75c7ab756f93ffd0eadd1cd88571d4c2d74cc1 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-j, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.build-date=20240319, maintainer=Guillaume Abrioux , RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.schema-version=1.0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2) 2024-03-24T00:12:25.585 INFO:journalctl@ceph.mds.j.smithi118.stdout:Mar 24 00:12:25 smithi118 podman[72610]: 2024-03-24 00:12:25.584525031 +0000 UTC m=+0.438804788 container init 4b581125b505cb60f3ec5e80ce75c7ab756f93ffd0eadd1cd88571d4c2d74cc1 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-j, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240319, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-24T00:12:25.946 INFO:journalctl@ceph.mds.j.smithi118.stdout:Mar 24 00:12:25 smithi118 podman[72610]: 2024-03-24 00:12:25.594909984 +0000 UTC m=+0.449189724 container start 4b581125b505cb60f3ec5e80ce75c7ab756f93ffd0eadd1cd88571d4c2d74cc1 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-j, GIT_CLEAN=True, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.build-date=20240319, ceph=True, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5) 2024-03-24T00:12:25.946 INFO:journalctl@ceph.mds.j.smithi118.stdout:Mar 24 00:12:25 smithi118 ceph-mds[72629]: -- 172.21.15.118:0/1579920019 <== mon.0 v2:172.21.15.118:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x55a5dd737a40 con 0x55a5dd70ac00 2024-03-24T00:12:25.946 INFO:journalctl@ceph.mds.j.smithi118.stdout:Mar 24 00:12:25 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-j[72625]: starting mds.j at 2024-03-24T00:12:25.946 INFO:journalctl@ceph.mds.j.smithi118.stdout:Mar 24 00:12:25 smithi118 bash[72610]: 4b581125b505cb60f3ec5e80ce75c7ab756f93ffd0eadd1cd88571d4c2d74cc1 2024-03-24T00:12:25.946 INFO:journalctl@ceph.mds.j.smithi118.stdout:Mar 24 00:12:25 smithi118 systemd[1]: Started Ceph mds.j for d79a5420-e971-11ee-95cd-87774f69a715. 2024-03-24T00:12:26.355 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:12:26 smithi189 ceph-mon[36187]: pgmap v292: 1 pgs: 1 active+clean; 577 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-24T00:12:26.355 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:12:26 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T00:12:26.355 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:12:26 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T00:12:26.355 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:12:26 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T00:12:26.355 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:12:26 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.d", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-24T00:12:26.355 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:12:26 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.d", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-24T00:12:26.356 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:12:26 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T00:12:26.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:12:26 smithi118 ceph-mon[31535]: pgmap v292: 1 pgs: 1 active+clean; 577 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-24T00:12:26.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:12:26 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T00:12:26.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:12:26 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T00:12:26.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:12:26 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T00:12:26.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:12:26 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.d", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-24T00:12:26.447 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:12:26 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.d", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-24T00:12:26.447 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:12:26 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T00:12:26.460 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:12:26 smithi193 ceph-mon[36878]: pgmap v292: 1 pgs: 1 active+clean; 577 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-24T00:12:26.460 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:12:26 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T00:12:26.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:12:26 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T00:12:26.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:12:26 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T00:12:26.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:12:26 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.d", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-24T00:12:26.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:12:26 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.d", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-24T00:12:26.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:12:26 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T00:12:26.968 DEBUG:teuthology.orchestra.run.smithi118:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-24T00:12:27.146 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:12:27 smithi118 ceph-mon[31535]: Deploying daemon mds.d on smithi118 2024-03-24T00:12:27.146 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:12:27 smithi118 ceph-mon[31535]: mds.? [v2:172.21.15.118:6834/3194741373,v1:172.21.15.118:6835/3194741373] up:boot 2024-03-24T00:12:27.146 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:12:27 smithi118 ceph-mon[31535]: fsmap 3 up:standby 2024-03-24T00:12:27.146 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:12:27 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2024-03-24T00:12:27.163 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:27.162+0000 7fa599000640 1 -- 172.21.15.118:0/3113974164 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fa5941016c0 msgr2=0x7fa594101ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:12:27.163 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:27.162+0000 7fa599000640 1 --2- 172.21.15.118:0/3113974164 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fa5941016c0 0x7fa594101ae0 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7fa584009900 tx=0x7fa58402bc30 comp rx=0 tx=0).stop 2024-03-24T00:12:27.163 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:27.162+0000 7fa599000640 1 -- 172.21.15.118:0/3113974164 shutdown_connections 2024-03-24T00:12:27.163 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:27.162+0000 7fa599000640 1 --2- 172.21.15.118:0/3113974164 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa594103270 0x7fa594109b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:12:27.163 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:27.162+0000 7fa599000640 1 --2- 172.21.15.118:0/3113974164 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fa5941028b0 0x7fa594102d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:12:27.163 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:27.162+0000 7fa599000640 1 --2- 172.21.15.118:0/3113974164 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fa5941016c0 0x7fa594101ae0 unknown :-1 s=CLOSED pgs=140 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:12:27.163 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:27.162+0000 7fa599000640 1 -- 172.21.15.118:0/3113974164 >> 172.21.15.118:0/3113974164 conn(0x7fa5940fcc50 msgr2=0x7fa5940ff070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:12:27.163 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:27.162+0000 7fa599000640 1 -- 172.21.15.118:0/3113974164 shutdown_connections 2024-03-24T00:12:27.163 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:27.162+0000 7fa599000640 1 -- 172.21.15.118:0/3113974164 wait complete. 2024-03-24T00:12:27.164 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:27.164+0000 7fa599000640 1 Processor -- start 2024-03-24T00:12:27.164 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:27.164+0000 7fa599000640 1 -- start start 2024-03-24T00:12:27.166 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:27.164+0000 7fa599000640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fa5941016c0 0x7fa594199be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:12:27.167 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:27.164+0000 7fa599000640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fa5941028b0 0x7fa59419a120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:12:27.167 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:27.164+0000 7fa599000640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa594103270 0x7fa5941a1190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:12:27.167 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:27.164+0000 7fa599000640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa59410d380 con 0x7fa5941016c0 2024-03-24T00:12:27.167 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:27.164+0000 7fa599000640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa59410d200 con 0x7fa594103270 2024-03-24T00:12:27.167 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:27.164+0000 7fa599000640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa59410d500 con 0x7fa5941028b0 2024-03-24T00:12:27.167 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:27.164+0000 7fa591e00640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fa5941016c0 0x7fa594199be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:12:27.167 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:27.164+0000 7fa592800640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa594103270 0x7fa5941a1190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:12:27.167 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:27.164+0000 7fa592800640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa594103270 0x7fa5941a1190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.118:47034/0 (socket says 172.21.15.118:47034) 2024-03-24T00:12:27.167 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:27.164+0000 7fa592800640 1 -- 172.21.15.118:0/2115311333 learned_addr learned my addr 172.21.15.118:0/2115311333 (peer_addr_for_me v2:172.21.15.118:0/0) 2024-03-24T00:12:27.167 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:27.164+0000 7fa591e00640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fa5941016c0 0x7fa594199be0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.118:3300/0 says I am v2:172.21.15.118:33790/0 (socket says 172.21.15.118:33790) 2024-03-24T00:12:27.167 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:27.164+0000 7fa591400640 1 --2- 172.21.15.118:0/2115311333 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fa5941028b0 0x7fa59419a120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:12:27.167 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:27.165+0000 7fa592800640 1 -- 172.21.15.118:0/2115311333 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fa5941028b0 msgr2=0x7fa59419a120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:12:27.167 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:27.165+0000 7fa592800640 1 --2- 172.21.15.118:0/2115311333 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fa5941028b0 0x7fa59419a120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:12:27.167 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:27.165+0000 7fa592800640 1 -- 172.21.15.118:0/2115311333 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fa5941016c0 msgr2=0x7fa594199be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:12:27.168 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:27.165+0000 7fa592800640 1 --2- 172.21.15.118:0/2115311333 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fa5941016c0 0x7fa594199be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:12:27.168 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:27.165+0000 7fa592800640 1 -- 172.21.15.118:0/2115311333 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa5941a1800 con 0x7fa594103270 2024-03-24T00:12:27.168 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:27.165+0000 7fa591400640 1 --2- 172.21.15.118:0/2115311333 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fa5941028b0 0x7fa59419a120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T00:12:27.168 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:27.165+0000 7fa592800640 1 --2- 172.21.15.118:0/2115311333 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa594103270 0x7fa5941a1190 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7fa58c00eb10 tx=0x7fa58c00c2d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:12:27.168 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:27.166+0000 7fa58aa00640 1 -- 172.21.15.118:0/2115311333 <== mon.2 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa58c004b20 con 0x7fa594103270 2024-03-24T00:12:27.168 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:27.166+0000 7fa599000640 1 -- 172.21.15.118:0/2115311333 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa5941a1af0 con 0x7fa594103270 2024-03-24T00:12:27.168 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:27.166+0000 7fa58aa00640 1 -- 172.21.15.118:0/2115311333 <== mon.2 v2:172.21.15.189:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fa58c004cc0 con 0x7fa594103270 2024-03-24T00:12:27.168 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:27.166+0000 7fa58aa00640 1 -- 172.21.15.118:0/2115311333 <== mon.2 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa58c010760 con 0x7fa594103270 2024-03-24T00:12:27.168 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:27.166+0000 7fa599000640 1 -- 172.21.15.118:0/2115311333 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa5941a1ff0 con 0x7fa594103270 2024-03-24T00:12:27.168 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:27.167+0000 7fa599000640 1 -- 172.21.15.118:0/2115311333 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa5941a2280 con 0x7fa594103270 2024-03-24T00:12:27.169 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:27.169+0000 7fa58aa00640 1 -- 172.21.15.118:0/2115311333 <== mon.2 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 16) v1 ==== 137291+0+0 (secure 0 0 0) 0x7fa58c005020 con 0x7fa594103270 2024-03-24T00:12:27.170 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:27.170+0000 7fa58aa00640 1 --2- 172.21.15.118:0/2115311333 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7fa5780a2800 0x7fa5780a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:12:27.171 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:27.170+0000 7fa58aa00640 1 -- 172.21.15.118:0/2115311333 <== mon.2 v2:172.21.15.189:3300/0 5 ==== osd_map(82..82 src has 1..82) v4 ==== 7127+0+0 (secure 0 0 0) 0x7fa58c0cec80 con 0x7fa594103270 2024-03-24T00:12:27.171 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:27.170+0000 7fa591e00640 1 --2- 172.21.15.118:0/2115311333 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7fa5780a2800 0x7fa5780a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:12:27.171 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:27.171+0000 7fa591e00640 1 --2- 172.21.15.118:0/2115311333 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7fa5780a2800 0x7fa5780a4cc0 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7fa58402c170 tx=0x7fa58402df90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:12:27.175 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:27.174+0000 7fa58aa00640 1 -- 172.21.15.118:0/2115311333 <== mon.2 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7fa58c09c120 con 0x7fa594103270 2024-03-24T00:12:27.430 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:27.429+0000 7fa599000640 1 -- 172.21.15.118:0/2115311333 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fa5941a2450 con 0x7fa594103270 2024-03-24T00:12:27.430 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:27.430+0000 7fa58aa00640 1 -- 172.21.15.118:0/2115311333 <== mon.2 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 4 v4) v1 ==== 93+0+2916 (secure 0 0 0) 0x7fa58c09ba20 con 0x7fa594103270 2024-03-24T00:12:27.430 INFO:teuthology.orchestra.run.smithi118.stdout: 2024-03-24T00:12:27.431 INFO:teuthology.orchestra.run.smithi118.stdout:{"epoch":4,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14385,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.118:6835/3194741373","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6834","nonce":3194741373},{"type":"v1","addr":"172.21.15.118:6835","nonce":3194741373}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":4},{"gid":24439,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.193:6833/1023428412","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6832","nonce":1023428412},{"type":"v1","addr":"172.21.15.193:6833","nonce":1023428412}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":24440,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.189:6833/2740024078","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6832","nonce":2740024078},{"type":"v1","addr":"172.21.15.189:6833","nonce":2740024078}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[]} 2024-03-24T00:12:27.431 INFO:teuthology.orchestra.run.smithi118.stderr:dumped fsmap epoch 4 2024-03-24T00:12:27.433 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:27.433+0000 7fa599000640 1 -- 172.21.15.118:0/2115311333 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7fa5780a2800 msgr2=0x7fa5780a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:12:27.433 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:27.433+0000 7fa599000640 1 --2- 172.21.15.118:0/2115311333 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7fa5780a2800 0x7fa5780a4cc0 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7fa58402c170 tx=0x7fa58402df90 comp rx=0 tx=0).stop 2024-03-24T00:12:27.433 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:27.433+0000 7fa599000640 1 -- 172.21.15.118:0/2115311333 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa594103270 msgr2=0x7fa5941a1190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:12:27.433 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:27.433+0000 7fa599000640 1 --2- 172.21.15.118:0/2115311333 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa594103270 0x7fa5941a1190 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7fa58c00eb10 tx=0x7fa58c00c2d0 comp rx=0 tx=0).stop 2024-03-24T00:12:27.434 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:27.433+0000 7fa599000640 1 -- 172.21.15.118:0/2115311333 shutdown_connections 2024-03-24T00:12:27.434 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:27.433+0000 7fa599000640 1 --2- 172.21.15.118:0/2115311333 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7fa5780a2800 0x7fa5780a4cc0 unknown :-1 s=CLOSED pgs=131 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:12:27.434 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:27.433+0000 7fa599000640 1 --2- 172.21.15.118:0/2115311333 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa594103270 0x7fa5941a1190 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:12:27.434 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:27.433+0000 7fa599000640 1 --2- 172.21.15.118:0/2115311333 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fa5941028b0 0x7fa59419a120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:12:27.434 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:27.433+0000 7fa599000640 1 --2- 172.21.15.118:0/2115311333 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fa5941016c0 0x7fa594199be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:12:27.434 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:27.433+0000 7fa599000640 1 -- 172.21.15.118:0/2115311333 >> 172.21.15.118:0/2115311333 conn(0x7fa5940fcc50 msgr2=0x7fa5940738d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:12:27.434 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:27.433+0000 7fa599000640 1 -- 172.21.15.118:0/2115311333 shutdown_connections 2024-03-24T00:12:27.434 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:27.433+0000 7fa599000640 1 -- 172.21.15.118:0/2115311333 wait complete. 2024-03-24T00:12:27.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:12:27 smithi189 ceph-mon[36187]: Deploying daemon mds.d on smithi118 2024-03-24T00:12:27.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:12:27 smithi189 ceph-mon[36187]: mds.? [v2:172.21.15.118:6834/3194741373,v1:172.21.15.118:6835/3194741373] up:boot 2024-03-24T00:12:27.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:12:27 smithi189 ceph-mon[36187]: fsmap 3 up:standby 2024-03-24T00:12:27.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:12:27 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2024-03-24T00:12:27.460 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:12:27 smithi193 ceph-mon[36878]: Deploying daemon mds.d on smithi118 2024-03-24T00:12:27.460 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:12:27 smithi193 ceph-mon[36878]: mds.? [v2:172.21.15.118:6834/3194741373,v1:172.21.15.118:6835/3194741373] up:boot 2024-03-24T00:12:27.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:12:27 smithi193 ceph-mon[36878]: fsmap 3 up:standby 2024-03-24T00:12:27.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:12:27 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2024-03-24T00:12:28.085 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:12:28 smithi118 ceph-mon[31535]: from='client.? 172.21.15.118:0/2115311333' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-24T00:12:28.085 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:12:28 smithi118 ceph-mon[31535]: pgmap v293: 1 pgs: 1 active+clean; 577 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-24T00:12:28.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:12:28 smithi189 ceph-mon[36187]: from='client.? 172.21.15.118:0/2115311333' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-24T00:12:28.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:12:28 smithi189 ceph-mon[36187]: pgmap v293: 1 pgs: 1 active+clean; 577 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-24T00:12:28.460 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:12:28 smithi193 ceph-mon[36878]: from='client.? 172.21.15.118:0/2115311333' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-24T00:12:28.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:12:28 smithi193 ceph-mon[36878]: pgmap v293: 1 pgs: 1 active+clean; 577 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-24T00:12:29.197 INFO:journalctl@ceph.mds.d.smithi118.stdout:Mar 24 00:12:28 smithi118 systemd[1]: Starting Ceph mds.d for d79a5420-e971-11ee-95cd-87774f69a715... 2024-03-24T00:12:29.454 DEBUG:teuthology.orchestra.run.smithi118:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-24T00:12:29.531 INFO:journalctl@ceph.mds.d.smithi118.stdout:Mar 24 00:12:29 smithi118 podman[72969]: 2024-03-24 00:12:29.184149412 +0000 UTC m=+0.035555025 image pull c664afe56bb01db4f9afa9532c20598e201e664f2e9ba2be50db0d52674a26d2 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc 2024-03-24T00:12:29.531 INFO:journalctl@ceph.mds.d.smithi118.stdout:Mar 24 00:12:29 smithi118 podman[72969]: 2024-03-24 00:12:29.30471736 +0000 UTC m=+0.156123010 container create 92f7fd8f0d7a1188e7c2bdbc2ff52f5e634f544c8fb5568aa09480ae496c40bd (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-d, ceph=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, org.label-schema.build-date=20240319, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-24T00:12:29.659 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:29.658+0000 7f815ce00640 1 -- 172.21.15.118:0/4064749703 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f81581038e0 msgr2=0x7f8158103d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:12:29.659 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:29.658+0000 7f815ce00640 1 --2- 172.21.15.118:0/4064749703 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f81581038e0 0x7f8158103d00 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7f8148009860 tx=0x7f814802bb70 comp rx=0 tx=0).stop 2024-03-24T00:12:29.659 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:29.658+0000 7f815ce00640 1 -- 172.21.15.118:0/4064749703 shutdown_connections 2024-03-24T00:12:29.660 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:29.658+0000 7f815ce00640 1 --2- 172.21.15.118:0/4064749703 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8158105490 0x7f815810bd20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:12:29.660 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:29.658+0000 7f815ce00640 1 --2- 172.21.15.118:0/4064749703 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f8158104ad0 0x7f8158104f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:12:29.660 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:29.658+0000 7f815ce00640 1 --2- 172.21.15.118:0/4064749703 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f81581038e0 0x7f8158103d00 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:12:29.660 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:29.658+0000 7f815ce00640 1 -- 172.21.15.118:0/4064749703 >> 172.21.15.118:0/4064749703 conn(0x7f81580fee90 msgr2=0x7f81581012b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:12:29.660 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:29.658+0000 7f815ce00640 1 -- 172.21.15.118:0/4064749703 shutdown_connections 2024-03-24T00:12:29.660 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:29.658+0000 7f815ce00640 1 -- 172.21.15.118:0/4064749703 wait complete. 2024-03-24T00:12:29.662 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:29.659+0000 7f815ce00640 1 Processor -- start 2024-03-24T00:12:29.663 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:29.659+0000 7f815ce00640 1 -- start start 2024-03-24T00:12:29.663 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:29.660+0000 7f815ce00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f81581038e0 0x7f8158199c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:12:29.663 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:29.660+0000 7f815ce00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f8158104ad0 0x7f815819a160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:12:29.663 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:29.660+0000 7f815ce00640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f8158105490 0x7f81581a1190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:12:29.663 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:29.660+0000 7f815ce00640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f815810f560 con 0x7f8158105490 2024-03-24T00:12:29.663 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:29.660+0000 7f815ce00640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f815810f3e0 con 0x7f81581038e0 2024-03-24T00:12:29.663 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:29.660+0000 7f815ce00640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f815810f6e0 con 0x7f8158104ad0 2024-03-24T00:12:29.663 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:29.660+0000 7f8156800640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f8158105490 0x7f81581a1190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:12:29.663 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:29.660+0000 7f814fe00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f8158104ad0 0x7f815819a160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:12:29.663 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:29.660+0000 7f814fe00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f8158104ad0 0x7f815819a160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.118:57346/0 (socket says 172.21.15.118:57346) 2024-03-24T00:12:29.663 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:29.660+0000 7f814fe00640 1 -- 172.21.15.118:0/3577325675 learned_addr learned my addr 172.21.15.118:0/3577325675 (peer_addr_for_me v2:172.21.15.118:0/0) 2024-03-24T00:12:29.663 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:29.660+0000 7f8155e00640 1 --2- 172.21.15.118:0/3577325675 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f81581038e0 0x7f8158199c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:12:29.663 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:29.660+0000 7f814fe00640 1 -- 172.21.15.118:0/3577325675 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f81581038e0 msgr2=0x7f8158199c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:12:29.664 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:29.660+0000 7f814fe00640 1 --2- 172.21.15.118:0/3577325675 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f81581038e0 0x7f8158199c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:12:29.664 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:29.660+0000 7f814fe00640 1 -- 172.21.15.118:0/3577325675 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f8158105490 msgr2=0x7f81581a1190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:12:29.664 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:29.660+0000 7f814fe00640 1 --2- 172.21.15.118:0/3577325675 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f8158105490 0x7f81581a1190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:12:29.664 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:29.661+0000 7f814fe00640 1 -- 172.21.15.118:0/3577325675 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f81581a1830 con 0x7f8158104ad0 2024-03-24T00:12:29.664 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:29.661+0000 7f814fe00640 1 --2- 172.21.15.118:0/3577325675 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f8158104ad0 0x7f815819a160 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f814000e9c0 tx=0x7f814000ee80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:12:29.664 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:29.661+0000 7f814ea00640 1 -- 172.21.15.118:0/3577325675 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f814000ccc0 con 0x7f8158104ad0 2024-03-24T00:12:29.664 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:29.661+0000 7f814ea00640 1 -- 172.21.15.118:0/3577325675 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f8140005500 con 0x7f8158104ad0 2024-03-24T00:12:29.664 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:29.661+0000 7f815ce00640 1 -- 172.21.15.118:0/3577325675 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f81581a1b20 con 0x7f8158104ad0 2024-03-24T00:12:29.664 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:29.661+0000 7f814ea00640 1 -- 172.21.15.118:0/3577325675 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8140010430 con 0x7f8158104ad0 2024-03-24T00:12:29.664 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:29.661+0000 7f815ce00640 1 -- 172.21.15.118:0/3577325675 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f81581a2090 con 0x7f8158104ad0 2024-03-24T00:12:29.664 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:29.662+0000 7f815ce00640 1 -- 172.21.15.118:0/3577325675 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f815810ac00 con 0x7f8158104ad0 2024-03-24T00:12:29.665 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:29.664+0000 7f814ea00640 1 -- 172.21.15.118:0/3577325675 <== mon.1 v2:172.21.15.193:3300/0 4 ==== mgrmap(e 16) v1 ==== 137291+0+0 (secure 0 0 0) 0x7f81400046c0 con 0x7f8158104ad0 2024-03-24T00:12:29.665 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:29.665+0000 7f814ea00640 1 --2- 172.21.15.118:0/3577325675 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7f813c0a2880 0x7f813c0a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:12:29.666 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:29.665+0000 7f814ea00640 1 -- 172.21.15.118:0/3577325675 <== mon.1 v2:172.21.15.193:3300/0 5 ==== osd_map(82..82 src has 1..82) v4 ==== 7127+0+0 (secure 0 0 0) 0x7f81400ce950 con 0x7f8158104ad0 2024-03-24T00:12:29.666 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:29.665+0000 7f8155e00640 1 --2- 172.21.15.118:0/3577325675 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7f813c0a2880 0x7f813c0a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:12:29.666 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:29.666+0000 7f8155e00640 1 --2- 172.21.15.118:0/3577325675 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7f813c0a2880 0x7f813c0a4d40 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7f81480349e0 tx=0x7f8148034ab0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:12:29.670 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:29.669+0000 7f814ea00640 1 -- 172.21.15.118:0/3577325675 <== mon.1 v2:172.21.15.193:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f814009bdf0 con 0x7f8158104ad0 2024-03-24T00:12:29.929 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:29.929+0000 7f815ce00640 1 -- 172.21.15.118:0/3577325675 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f815802d450 con 0x7f8158104ad0 2024-03-24T00:12:29.930 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:29.929+0000 7f814ea00640 1 -- 172.21.15.118:0/3577325675 <== mon.1 v2:172.21.15.193:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 5 v5) v1 ==== 93+0+3710 (secure 0 0 0) 0x7f815802d450 con 0x7f8158104ad0 2024-03-24T00:12:29.930 INFO:teuthology.orchestra.run.smithi118.stdout: 2024-03-24T00:12:29.930 INFO:teuthology.orchestra.run.smithi118.stdout:{"epoch":5,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14385,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.118:6835/3194741373","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6834","nonce":3194741373},{"type":"v1","addr":"172.21.15.118:6835","nonce":3194741373}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":4},{"gid":14397,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.118:6837/1374462956","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6836","nonce":1374462956},{"type":"v1","addr":"172.21.15.118:6837","nonce":1374462956}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":5},{"gid":24439,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.193:6833/1023428412","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6832","nonce":1023428412},{"type":"v1","addr":"172.21.15.193:6833","nonce":1023428412}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":24440,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.189:6833/2740024078","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6832","nonce":2740024078},{"type":"v1","addr":"172.21.15.189:6833","nonce":2740024078}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[]} 2024-03-24T00:12:29.931 INFO:teuthology.orchestra.run.smithi118.stderr:dumped fsmap epoch 5 2024-03-24T00:12:29.933 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:29.932+0000 7f815ce00640 1 -- 172.21.15.118:0/3577325675 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7f813c0a2880 msgr2=0x7f813c0a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:12:29.933 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:29.932+0000 7f815ce00640 1 --2- 172.21.15.118:0/3577325675 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7f813c0a2880 0x7f813c0a4d40 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7f81480349e0 tx=0x7f8148034ab0 comp rx=0 tx=0).stop 2024-03-24T00:12:29.934 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:29.932+0000 7f815ce00640 1 -- 172.21.15.118:0/3577325675 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f8158104ad0 msgr2=0x7f815819a160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:12:29.934 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:29.933+0000 7f815ce00640 1 --2- 172.21.15.118:0/3577325675 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f8158104ad0 0x7f815819a160 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f814000e9c0 tx=0x7f814000ee80 comp rx=0 tx=0).stop 2024-03-24T00:12:29.934 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:29.933+0000 7f815ce00640 1 -- 172.21.15.118:0/3577325675 shutdown_connections 2024-03-24T00:12:29.934 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:29.933+0000 7f815ce00640 1 --2- 172.21.15.118:0/3577325675 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7f813c0a2880 0x7f813c0a4d40 unknown :-1 s=CLOSED pgs=132 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:12:29.934 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:29.933+0000 7f815ce00640 1 --2- 172.21.15.118:0/3577325675 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f8158105490 0x7f81581a1190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:12:29.934 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:29.933+0000 7f815ce00640 1 --2- 172.21.15.118:0/3577325675 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f8158104ad0 0x7f815819a160 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:12:29.934 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:29.933+0000 7f815ce00640 1 --2- 172.21.15.118:0/3577325675 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f81581038e0 0x7f8158199c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:12:29.934 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:29.933+0000 7f815ce00640 1 -- 172.21.15.118:0/3577325675 >> 172.21.15.118:0/3577325675 conn(0x7f81580fee90 msgr2=0x7f8158107dd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:12:29.934 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:29.933+0000 7f815ce00640 1 -- 172.21.15.118:0/3577325675 shutdown_connections 2024-03-24T00:12:29.934 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:29.933+0000 7f815ce00640 1 -- 172.21.15.118:0/3577325675 wait complete. 2024-03-24T00:12:29.945 INFO:journalctl@ceph.mds.d.smithi118.stdout:Mar 24 00:12:29 smithi118 podman[72969]: 2024-03-24 00:12:29.584062723 +0000 UTC m=+0.435468383 container init 92f7fd8f0d7a1188e7c2bdbc2ff52f5e634f544c8fb5568aa09480ae496c40bd (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-d, org.label-schema.build-date=20240319, GIT_CLEAN=True, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.license=GPLv2, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=) 2024-03-24T00:12:29.946 INFO:journalctl@ceph.mds.d.smithi118.stdout:Mar 24 00:12:29 smithi118 podman[72969]: 2024-03-24 00:12:29.59421053 +0000 UTC m=+0.445616190 container start 92f7fd8f0d7a1188e7c2bdbc2ff52f5e634f544c8fb5568aa09480ae496c40bd (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-d, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.schema-version=1.0, ceph=True, CEPH_POINT_RELEASE=, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, org.label-schema.build-date=20240319) 2024-03-24T00:12:29.946 INFO:journalctl@ceph.mds.d.smithi118.stdout:Mar 24 00:12:29 smithi118 ceph-mds[73007]: -- 172.21.15.118:0/1798986787 <== mon.0 v2:172.21.15.118:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x555eb0ba9a40 con 0x555eb0b7d000 2024-03-24T00:12:29.946 INFO:journalctl@ceph.mds.d.smithi118.stdout:Mar 24 00:12:29 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-d[72983]: starting mds.d at 2024-03-24T00:12:29.946 INFO:journalctl@ceph.mds.d.smithi118.stdout:Mar 24 00:12:29 smithi118 bash[72969]: 92f7fd8f0d7a1188e7c2bdbc2ff52f5e634f544c8fb5568aa09480ae496c40bd 2024-03-24T00:12:29.946 INFO:journalctl@ceph.mds.d.smithi118.stdout:Mar 24 00:12:29 smithi118 systemd[1]: Started Ceph mds.d for d79a5420-e971-11ee-95cd-87774f69a715. 2024-03-24T00:12:30.812 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:12:30 smithi189 ceph-mon[36187]: pgmap v294: 1 pgs: 1 active+clean; 577 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-24T00:12:30.812 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:12:30 smithi189 ceph-mon[36187]: mds.? [v2:172.21.15.118:6836/1374462956,v1:172.21.15.118:6837/1374462956] up:boot 2024-03-24T00:12:30.812 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:12:30 smithi189 ceph-mon[36187]: fsmap 4 up:standby 2024-03-24T00:12:30.812 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:12:30 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2024-03-24T00:12:30.812 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:12:30 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T00:12:30.812 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:12:30 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T00:12:30.813 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:12:30 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T00:12:30.813 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:12:30 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.e", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-24T00:12:30.813 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:12:30 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.e", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-24T00:12:30.813 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:12:30 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T00:12:30.813 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:12:30 smithi189 ceph-mon[36187]: from='client.? 172.21.15.118:0/3577325675' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-24T00:12:30.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:12:30 smithi193 ceph-mon[36878]: pgmap v294: 1 pgs: 1 active+clean; 577 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-24T00:12:30.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:12:30 smithi193 ceph-mon[36878]: mds.? [v2:172.21.15.118:6836/1374462956,v1:172.21.15.118:6837/1374462956] up:boot 2024-03-24T00:12:30.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:12:30 smithi193 ceph-mon[36878]: fsmap 4 up:standby 2024-03-24T00:12:30.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:12:30 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2024-03-24T00:12:30.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:12:30 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T00:12:30.962 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:12:30 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T00:12:30.962 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:12:30 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T00:12:30.962 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:12:30 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.e", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-24T00:12:30.962 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:12:30 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.e", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-24T00:12:30.962 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:12:30 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T00:12:30.962 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:12:30 smithi193 ceph-mon[36878]: from='client.? 172.21.15.118:0/3577325675' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-24T00:12:31.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:12:30 smithi118 ceph-mon[31535]: pgmap v294: 1 pgs: 1 active+clean; 577 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-24T00:12:31.197 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:12:30 smithi118 ceph-mon[31535]: mds.? [v2:172.21.15.118:6836/1374462956,v1:172.21.15.118:6837/1374462956] up:boot 2024-03-24T00:12:31.197 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:12:30 smithi118 ceph-mon[31535]: fsmap 4 up:standby 2024-03-24T00:12:31.197 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:12:30 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2024-03-24T00:12:31.197 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:12:30 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T00:12:31.197 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:12:30 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T00:12:31.197 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:12:30 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T00:12:31.197 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:12:30 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.e", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-24T00:12:31.197 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:12:30 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.e", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-24T00:12:31.197 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:12:30 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T00:12:31.197 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:12:30 smithi118 ceph-mon[31535]: from='client.? 172.21.15.118:0/3577325675' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-24T00:12:31.953 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:12:31 smithi189 ceph-mon[36187]: Deploying daemon mds.e on smithi189 2024-03-24T00:12:31.954 DEBUG:teuthology.orchestra.run.smithi118:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-24T00:12:31.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:12:31 smithi193 ceph-mon[36878]: Deploying daemon mds.e on smithi189 2024-03-24T00:12:32.043 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:12:31 smithi118 ceph-mon[31535]: Deploying daemon mds.e on smithi189 2024-03-24T00:12:32.180 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:32.179+0000 7f5ec5e00640 1 -- 172.21.15.118:0/3454826438 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5ec00772a0 msgr2=0x7f5ec0077700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:12:32.180 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:32.179+0000 7f5ec5e00640 1 --2- 172.21.15.118:0/3454826438 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5ec00772a0 0x7f5ec0077700 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f5eac009900 tx=0x7f5eac02bc30 comp rx=0 tx=0).stop 2024-03-24T00:12:32.181 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:32.180+0000 7f5ec5e00640 1 -- 172.21.15.118:0/3454826438 shutdown_connections 2024-03-24T00:12:32.181 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:32.180+0000 7f5ec5e00640 1 --2- 172.21.15.118:0/3454826438 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f5ec0077c40 0x7f5ec00784e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:12:32.181 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:32.180+0000 7f5ec5e00640 1 --2- 172.21.15.118:0/3454826438 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5ec00772a0 0x7f5ec0077700 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:12:32.181 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:32.180+0000 7f5ec5e00640 1 --2- 172.21.15.118:0/3454826438 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f5ec0076060 0x7f5ec0076480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:12:32.181 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:32.180+0000 7f5ec5e00640 1 -- 172.21.15.118:0/3454826438 >> 172.21.15.118:0/3454826438 conn(0x7f5ec00fee90 msgr2=0x7f5ec01012b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:12:32.181 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:32.180+0000 7f5ec5e00640 1 -- 172.21.15.118:0/3454826438 shutdown_connections 2024-03-24T00:12:32.181 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:32.180+0000 7f5ec5e00640 1 -- 172.21.15.118:0/3454826438 wait complete. 2024-03-24T00:12:32.183 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:32.181+0000 7f5ec5e00640 1 Processor -- start 2024-03-24T00:12:32.183 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:32.181+0000 7f5ec5e00640 1 -- start start 2024-03-24T00:12:32.183 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:32.181+0000 7f5ec5e00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5ec0076060 0x7f5ec019dfb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:12:32.183 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:32.181+0000 7f5ec5e00640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f5ec00772a0 0x7f5ec019e4f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:12:32.183 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:32.181+0000 7f5ec5e00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f5ec0077c40 0x7f5ec01a5560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:12:32.183 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:32.181+0000 7f5ec5e00640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5ec01139b0 con 0x7f5ec00772a0 2024-03-24T00:12:32.183 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:32.181+0000 7f5ec5e00640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5ec0113830 con 0x7f5ec0076060 2024-03-24T00:12:32.183 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:32.181+0000 7f5ec5e00640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5ec0113b30 con 0x7f5ec0077c40 2024-03-24T00:12:32.184 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:32.181+0000 7f5ebea00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5ec0076060 0x7f5ec019dfb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:12:32.184 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:32.181+0000 7f5ebe000640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f5ec00772a0 0x7f5ec019e4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:12:32.184 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:32.181+0000 7f5ebea00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5ec0076060 0x7f5ec019dfb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.118:47098/0 (socket says 172.21.15.118:47098) 2024-03-24T00:12:32.184 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:32.181+0000 7f5ebea00640 1 -- 172.21.15.118:0/1076427139 learned_addr learned my addr 172.21.15.118:0/1076427139 (peer_addr_for_me v2:172.21.15.118:0/0) 2024-03-24T00:12:32.184 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:32.181+0000 7f5ebf400640 1 --2- 172.21.15.118:0/1076427139 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f5ec0077c40 0x7f5ec01a5560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:12:32.184 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:32.182+0000 7f5ebea00640 1 -- 172.21.15.118:0/1076427139 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f5ec0077c40 msgr2=0x7f5ec01a5560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:12:32.184 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:32.182+0000 7f5ebea00640 1 --2- 172.21.15.118:0/1076427139 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f5ec0077c40 0x7f5ec01a5560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:12:32.184 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:32.182+0000 7f5ebea00640 1 -- 172.21.15.118:0/1076427139 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f5ec00772a0 msgr2=0x7f5ec019e4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:12:32.184 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:32.182+0000 7f5ebea00640 1 --2- 172.21.15.118:0/1076427139 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f5ec00772a0 0x7f5ec019e4f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:12:32.184 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:32.182+0000 7f5ebea00640 1 -- 172.21.15.118:0/1076427139 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5ec01a5bd0 con 0x7f5ec0076060 2024-03-24T00:12:32.184 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:32.182+0000 7f5ebea00640 1 --2- 172.21.15.118:0/1076427139 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5ec0076060 0x7f5ec019dfb0 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f5eb000e9f0 tx=0x7f5eb000edf0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:12:32.184 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:32.182+0000 7f5ec4a00640 1 -- 172.21.15.118:0/1076427139 <== mon.2 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5eb000cd80 con 0x7f5ec0076060 2024-03-24T00:12:32.184 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:32.182+0000 7f5ec4a00640 1 -- 172.21.15.118:0/1076427139 <== mon.2 v2:172.21.15.189:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f5eb0005500 con 0x7f5ec0076060 2024-03-24T00:12:32.184 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:32.182+0000 7f5ec4a00640 1 -- 172.21.15.118:0/1076427139 <== mon.2 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5eb0010690 con 0x7f5ec0076060 2024-03-24T00:12:32.184 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:32.182+0000 7f5ec5e00640 1 -- 172.21.15.118:0/1076427139 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5ec01a5ec0 con 0x7f5ec0076060 2024-03-24T00:12:32.185 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:32.182+0000 7f5ec5e00640 1 -- 172.21.15.118:0/1076427139 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5ec01a62d0 con 0x7f5ec0076060 2024-03-24T00:12:32.185 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:32.183+0000 7f5ec5e00640 1 -- 172.21.15.118:0/1076427139 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5ec0110250 con 0x7f5ec0076060 2024-03-24T00:12:32.185 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:32.185+0000 7f5ec4a00640 1 -- 172.21.15.118:0/1076427139 <== mon.2 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 16) v1 ==== 137291+0+0 (secure 0 0 0) 0x7f5eb00046c0 con 0x7f5ec0076060 2024-03-24T00:12:32.187 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:32.187+0000 7f5ec4a00640 1 --2- 172.21.15.118:0/1076427139 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7f5ea40a2800 0x7f5ea40a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:12:32.187 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:32.187+0000 7f5ec4a00640 1 -- 172.21.15.118:0/1076427139 <== mon.2 v2:172.21.15.189:3300/0 5 ==== osd_map(82..82 src has 1..82) v4 ==== 7127+0+0 (secure 0 0 0) 0x7f5eb00ceaf0 con 0x7f5ec0076060 2024-03-24T00:12:32.187 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:32.187+0000 7f5ebe000640 1 --2- 172.21.15.118:0/1076427139 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7f5ea40a2800 0x7f5ea40a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:12:32.188 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:32.188+0000 7f5ebe000640 1 --2- 172.21.15.118:0/1076427139 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7f5ea40a2800 0x7f5ea40a4cc0 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f5eac02c240 tx=0x7f5eac02df90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:12:32.191 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:32.191+0000 7f5ec4a00640 1 -- 172.21.15.118:0/1076427139 <== mon.2 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f5eb009bf90 con 0x7f5ec0076060 2024-03-24T00:12:32.372 INFO:journalctl@ceph.mds.e.smithi189.stdout:Mar 24 00:12:32 smithi189 systemd[1]: Starting Ceph mds.e for d79a5420-e971-11ee-95cd-87774f69a715... 2024-03-24T00:12:32.435 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:32.434+0000 7f5ec5e00640 1 -- 172.21.15.118:0/1076427139 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f5ec002d070 con 0x7f5ec0076060 2024-03-24T00:12:32.435 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:32.435+0000 7f5ec4a00640 1 -- 172.21.15.118:0/1076427139 <== mon.2 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 5 v5) v1 ==== 93+0+3710 (secure 0 0 0) 0x7f5eb0084e20 con 0x7f5ec0076060 2024-03-24T00:12:32.436 INFO:teuthology.orchestra.run.smithi118.stdout: 2024-03-24T00:12:32.436 INFO:teuthology.orchestra.run.smithi118.stdout:{"epoch":5,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14385,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.118:6835/3194741373","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6834","nonce":3194741373},{"type":"v1","addr":"172.21.15.118:6835","nonce":3194741373}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":4},{"gid":14397,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.118:6837/1374462956","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6836","nonce":1374462956},{"type":"v1","addr":"172.21.15.118:6837","nonce":1374462956}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":5},{"gid":24439,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.193:6833/1023428412","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6832","nonce":1023428412},{"type":"v1","addr":"172.21.15.193:6833","nonce":1023428412}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":24440,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.189:6833/2740024078","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6832","nonce":2740024078},{"type":"v1","addr":"172.21.15.189:6833","nonce":2740024078}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3}],"filesystems":[]} 2024-03-24T00:12:32.436 INFO:teuthology.orchestra.run.smithi118.stderr:dumped fsmap epoch 5 2024-03-24T00:12:32.439 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:32.438+0000 7f5ec5e00640 1 -- 172.21.15.118:0/1076427139 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7f5ea40a2800 msgr2=0x7f5ea40a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:12:32.439 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:32.438+0000 7f5ec5e00640 1 --2- 172.21.15.118:0/1076427139 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7f5ea40a2800 0x7f5ea40a4cc0 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f5eac02c240 tx=0x7f5eac02df90 comp rx=0 tx=0).stop 2024-03-24T00:12:32.439 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:32.438+0000 7f5ec5e00640 1 -- 172.21.15.118:0/1076427139 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5ec0076060 msgr2=0x7f5ec019dfb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:12:32.439 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:32.438+0000 7f5ec5e00640 1 --2- 172.21.15.118:0/1076427139 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5ec0076060 0x7f5ec019dfb0 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f5eb000e9f0 tx=0x7f5eb000edf0 comp rx=0 tx=0).stop 2024-03-24T00:12:32.439 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:32.438+0000 7f5ec5e00640 1 -- 172.21.15.118:0/1076427139 shutdown_connections 2024-03-24T00:12:32.439 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:32.438+0000 7f5ec5e00640 1 --2- 172.21.15.118:0/1076427139 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7f5ea40a2800 0x7f5ea40a4cc0 unknown :-1 s=CLOSED pgs=134 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:12:32.439 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:32.438+0000 7f5ec5e00640 1 --2- 172.21.15.118:0/1076427139 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f5ec0077c40 0x7f5ec01a5560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:12:32.439 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:32.438+0000 7f5ec5e00640 1 --2- 172.21.15.118:0/1076427139 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f5ec00772a0 0x7f5ec019e4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:12:32.439 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:32.438+0000 7f5ec5e00640 1 --2- 172.21.15.118:0/1076427139 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5ec0076060 0x7f5ec019dfb0 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:12:32.439 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:32.438+0000 7f5ec5e00640 1 -- 172.21.15.118:0/1076427139 >> 172.21.15.118:0/1076427139 conn(0x7f5ec00fee90 msgr2=0x7f5ec0100990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:12:32.439 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:32.438+0000 7f5ec5e00640 1 -- 172.21.15.118:0/1076427139 shutdown_connections 2024-03-24T00:12:32.439 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:32.438+0000 7f5ec5e00640 1 -- 172.21.15.118:0/1076427139 wait complete. 2024-03-24T00:12:32.653 INFO:journalctl@ceph.mds.e.smithi189.stdout:Mar 24 00:12:32 smithi189 podman[66346]: 2024-03-24 00:12:32.276257386 +0000 UTC m=+0.017087141 image pull c664afe56bb01db4f9afa9532c20598e201e664f2e9ba2be50db0d52674a26d2 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc 2024-03-24T00:12:32.654 INFO:journalctl@ceph.mds.e.smithi189.stdout:Mar 24 00:12:32 smithi189 podman[66346]: 2024-03-24 00:12:32.409191625 +0000 UTC m=+0.150021413 container create 489bbba1e9002927b94f19e01c0fe7b3970bc1f9840a73d8747daae773b971db (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-e, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240319, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-24T00:12:32.654 INFO:journalctl@ceph.mds.e.smithi189.stdout:Mar 24 00:12:32 smithi189 podman[66346]: 2024-03-24 00:12:32.6537014 +0000 UTC m=+0.394531141 container init 489bbba1e9002927b94f19e01c0fe7b3970bc1f9840a73d8747daae773b971db (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-e, GIT_BRANCH=HEAD, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240319, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, ceph=True, org.label-schema.schema-version=1.0) 2024-03-24T00:12:32.953 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:12:32 smithi189 ceph-mon[36187]: pgmap v295: 1 pgs: 1 active+clean; 577 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-24T00:12:32.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:12:32 smithi189 ceph-mon[36187]: from='client.? 172.21.15.118:0/1076427139' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-24T00:12:32.954 INFO:journalctl@ceph.mds.e.smithi189.stdout:Mar 24 00:12:32 smithi189 podman[66346]: 2024-03-24 00:12:32.656432655 +0000 UTC m=+0.397262413 container start 489bbba1e9002927b94f19e01c0fe7b3970bc1f9840a73d8747daae773b971db (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-e, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, org.label-schema.build-date=20240319, io.buildah.version=1.33.5, ceph=True, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_CLEAN=True, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0) 2024-03-24T00:12:32.954 INFO:journalctl@ceph.mds.e.smithi189.stdout:Mar 24 00:12:32 smithi189 ceph-mds[66365]: -- 172.21.15.189:0/4233163540 <== mon.2 v2:172.21.15.189:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x55dda7b71a40 con 0x55dda7b44c00 2024-03-24T00:12:32.954 INFO:journalctl@ceph.mds.e.smithi189.stdout:Mar 24 00:12:32 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-e[66361]: starting mds.e at 2024-03-24T00:12:32.954 INFO:journalctl@ceph.mds.e.smithi189.stdout:Mar 24 00:12:32 smithi189 bash[66346]: 489bbba1e9002927b94f19e01c0fe7b3970bc1f9840a73d8747daae773b971db 2024-03-24T00:12:32.954 INFO:journalctl@ceph.mds.e.smithi189.stdout:Mar 24 00:12:32 smithi189 systemd[1]: Started Ceph mds.e for d79a5420-e971-11ee-95cd-87774f69a715. 2024-03-24T00:12:33.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:12:32 smithi118 ceph-mon[31535]: pgmap v295: 1 pgs: 1 active+clean; 577 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-24T00:12:33.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:12:32 smithi118 ceph-mon[31535]: from='client.? 172.21.15.118:0/1076427139' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-24T00:12:33.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:12:32 smithi193 ceph-mon[36878]: pgmap v295: 1 pgs: 1 active+clean; 577 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-24T00:12:33.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:12:32 smithi193 ceph-mon[36878]: from='client.? 172.21.15.118:0/1076427139' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-24T00:12:33.764 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:12:33 smithi193 ceph-mon[36878]: mds.? [v2:172.21.15.189:6834/599473205,v1:172.21.15.189:6835/599473205] up:boot 2024-03-24T00:12:33.765 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:12:33 smithi193 ceph-mon[36878]: fsmap 5 up:standby 2024-03-24T00:12:33.765 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:12:33 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2024-03-24T00:12:33.765 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:12:33 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T00:12:33.765 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:12:33 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T00:12:33.765 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:12:33 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T00:12:33.765 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:12:33 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.f", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-24T00:12:33.765 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:12:33 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.f", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-24T00:12:33.765 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:12:33 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T00:12:34.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:12:33 smithi118 ceph-mon[31535]: mds.? [v2:172.21.15.189:6834/599473205,v1:172.21.15.189:6835/599473205] up:boot 2024-03-24T00:12:34.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:12:33 smithi118 ceph-mon[31535]: fsmap 5 up:standby 2024-03-24T00:12:34.197 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:12:33 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2024-03-24T00:12:34.197 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:12:33 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T00:12:34.197 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:12:33 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T00:12:34.197 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:12:33 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T00:12:34.197 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:12:33 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.f", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-24T00:12:34.197 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:12:33 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.f", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-24T00:12:34.197 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:12:33 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T00:12:34.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:12:33 smithi189 ceph-mon[36187]: mds.? [v2:172.21.15.189:6834/599473205,v1:172.21.15.189:6835/599473205] up:boot 2024-03-24T00:12:34.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:12:33 smithi189 ceph-mon[36187]: fsmap 5 up:standby 2024-03-24T00:12:34.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:12:33 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2024-03-24T00:12:34.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:12:33 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T00:12:34.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:12:33 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T00:12:34.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:12:33 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T00:12:34.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:12:33 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.f", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-24T00:12:34.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:12:33 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.f", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-24T00:12:34.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:12:33 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T00:12:34.458 DEBUG:teuthology.orchestra.run.smithi118:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-24T00:12:34.682 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:34.681+0000 7f3f50200640 1 -- 172.21.15.118:0/729887510 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f3f481038e0 msgr2=0x7f3f48103d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:12:34.683 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:34.681+0000 7f3f50200640 1 --2- 172.21.15.118:0/729887510 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f3f481038e0 0x7f3f48103d00 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7f3f40009ca0 tx=0x7f3f4002c0f0 comp rx=0 tx=0).stop 2024-03-24T00:12:34.683 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:34.681+0000 7f3f50200640 1 -- 172.21.15.118:0/729887510 shutdown_connections 2024-03-24T00:12:34.683 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:34.681+0000 7f3f50200640 1 --2- 172.21.15.118:0/729887510 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f3f48105490 0x7f3f4810bd20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:12:34.683 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:34.681+0000 7f3f50200640 1 --2- 172.21.15.118:0/729887510 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3f48104ad0 0x7f3f48104f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:12:34.683 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:34.681+0000 7f3f50200640 1 --2- 172.21.15.118:0/729887510 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f3f481038e0 0x7f3f48103d00 unknown :-1 s=CLOSED pgs=143 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:12:34.683 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:34.681+0000 7f3f50200640 1 -- 172.21.15.118:0/729887510 >> 172.21.15.118:0/729887510 conn(0x7f3f480fee90 msgr2=0x7f3f481012b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:12:34.683 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:34.682+0000 7f3f50200640 1 -- 172.21.15.118:0/729887510 shutdown_connections 2024-03-24T00:12:34.683 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:34.682+0000 7f3f50200640 1 -- 172.21.15.118:0/729887510 wait complete. 2024-03-24T00:12:34.685 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:34.682+0000 7f3f50200640 1 Processor -- start 2024-03-24T00:12:34.685 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:34.683+0000 7f3f50200640 1 -- start start 2024-03-24T00:12:34.685 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:34.683+0000 7f3f50200640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f3f481038e0 0x7f3f48199bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:12:34.685 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:34.683+0000 7f3f50200640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f3f48104ad0 0x7f3f4819a110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:12:34.685 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:34.683+0000 7f3f50200640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3f48105490 0x7f3f481a1180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:12:34.685 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:34.683+0000 7f3f50200640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3f4810f560 con 0x7f3f481038e0 2024-03-24T00:12:34.685 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:34.683+0000 7f3f50200640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3f4810f3e0 con 0x7f3f48105490 2024-03-24T00:12:34.686 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:34.683+0000 7f3f50200640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3f4810f6e0 con 0x7f3f48104ad0 2024-03-24T00:12:34.686 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:34.683+0000 7f3f4d800640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f3f481038e0 0x7f3f48199bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:12:34.686 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:34.683+0000 7f3f4d800640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f3f481038e0 0x7f3f48199bd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.118:3300/0 says I am v2:172.21.15.118:44126/0 (socket says 172.21.15.118:44126) 2024-03-24T00:12:34.686 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:34.683+0000 7f3f4d800640 1 -- 172.21.15.118:0/771446215 learned_addr learned my addr 172.21.15.118:0/771446215 (peer_addr_for_me v2:172.21.15.118:0/0) 2024-03-24T00:12:34.686 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:34.683+0000 7f3f4ce00640 1 --2- 172.21.15.118:0/771446215 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f3f48104ad0 0x7f3f4819a110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:12:34.686 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:34.683+0000 7f3f4e200640 1 --2- 172.21.15.118:0/771446215 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3f48105490 0x7f3f481a1180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:12:34.686 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:34.683+0000 7f3f4d800640 1 -- 172.21.15.118:0/771446215 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f3f48104ad0 msgr2=0x7f3f4819a110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:12:34.686 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:34.683+0000 7f3f4d800640 1 --2- 172.21.15.118:0/771446215 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f3f48104ad0 0x7f3f4819a110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:12:34.686 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:34.683+0000 7f3f4d800640 1 -- 172.21.15.118:0/771446215 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3f48105490 msgr2=0x7f3f481a1180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:12:34.686 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:34.683+0000 7f3f4d800640 1 --2- 172.21.15.118:0/771446215 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3f48105490 0x7f3f481a1180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:12:34.686 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:34.683+0000 7f3f4d800640 1 -- 172.21.15.118:0/771446215 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3f481a17f0 con 0x7f3f481038e0 2024-03-24T00:12:34.686 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:34.683+0000 7f3f4e200640 1 --2- 172.21.15.118:0/771446215 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3f48105490 0x7f3f481a1180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T00:12:34.686 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:34.684+0000 7f3f4d800640 1 --2- 172.21.15.118:0/771446215 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f3f481038e0 0x7f3f48199bd0 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7f3f40000f80 tx=0x7f3f40032ec0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:12:34.686 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:34.684+0000 7f3f46000640 1 -- 172.21.15.118:0/771446215 <== mon.0 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3f40041020 con 0x7f3f481038e0 2024-03-24T00:12:34.687 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:34.684+0000 7f3f46000640 1 -- 172.21.15.118:0/771446215 <== mon.0 v2:172.21.15.118:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f3f40034cd0 con 0x7f3f481038e0 2024-03-24T00:12:34.687 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:34.684+0000 7f3f46000640 1 -- 172.21.15.118:0/771446215 <== mon.0 v2:172.21.15.118:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3f40039040 con 0x7f3f481038e0 2024-03-24T00:12:34.687 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:34.684+0000 7f3f50200640 1 -- 172.21.15.118:0/771446215 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3f481a1a80 con 0x7f3f481038e0 2024-03-24T00:12:34.687 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:34.684+0000 7f3f50200640 1 -- 172.21.15.118:0/771446215 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3f481a1f40 con 0x7f3f481038e0 2024-03-24T00:12:34.687 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:34.685+0000 7f3f50200640 1 -- 172.21.15.118:0/771446215 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3f4810ac00 con 0x7f3f481038e0 2024-03-24T00:12:34.688 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:34.687+0000 7f3f46000640 1 -- 172.21.15.118:0/771446215 <== mon.0 v2:172.21.15.118:3300/0 4 ==== mgrmap(e 16) v1 ==== 137291+0+0 (secure 0 0 0) 0x7f3f40044020 con 0x7f3f481038e0 2024-03-24T00:12:34.689 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:34.688+0000 7f3f46000640 1 --2- 172.21.15.118:0/771446215 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7f3f300a2800 0x7f3f300a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:12:34.689 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:34.689+0000 7f3f46000640 1 -- 172.21.15.118:0/771446215 <== mon.0 v2:172.21.15.118:3300/0 5 ==== osd_map(82..82 src has 1..82) v4 ==== 7127+0+0 (secure 0 0 0) 0x7f3f400f0550 con 0x7f3f481038e0 2024-03-24T00:12:34.689 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:34.689+0000 7f3f4ce00640 1 --2- 172.21.15.118:0/771446215 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7f3f300a2800 0x7f3f300a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:12:34.690 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:34.690+0000 7f3f4ce00640 1 --2- 172.21.15.118:0/771446215 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7f3f300a2800 0x7f3f300a4cc0 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7f3f4819b0e0 tx=0x7f3f3400b390 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:12:34.694 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:34.693+0000 7f3f46000640 1 -- 172.21.15.118:0/771446215 <== mon.0 v2:172.21.15.118:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f3f4004b020 con 0x7f3f481038e0 2024-03-24T00:12:34.932 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:34.932+0000 7f3f50200640 1 -- 172.21.15.118:0/771446215 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f3f4802d070 con 0x7f3f481038e0 2024-03-24T00:12:34.933 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:34.933+0000 7f3f46000640 1 -- 172.21.15.118:0/771446215 <== mon.0 v2:172.21.15.118:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 6 v6) v1 ==== 93+0+4501 (secure 0 0 0) 0x7f3f400bd9f0 con 0x7f3f481038e0 2024-03-24T00:12:34.934 INFO:teuthology.orchestra.run.smithi118.stdout: 2024-03-24T00:12:34.934 INFO:teuthology.orchestra.run.smithi118.stdout:{"epoch":6,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14385,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.118:6835/3194741373","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6834","nonce":3194741373},{"type":"v1","addr":"172.21.15.118:6835","nonce":3194741373}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":4},{"gid":14397,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.118:6837/1374462956","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6836","nonce":1374462956},{"type":"v1","addr":"172.21.15.118:6837","nonce":1374462956}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":5},{"gid":24439,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.193:6833/1023428412","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6832","nonce":1023428412},{"type":"v1","addr":"172.21.15.193:6833","nonce":1023428412}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":24440,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.189:6833/2740024078","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6832","nonce":2740024078},{"type":"v1","addr":"172.21.15.189:6833","nonce":2740024078}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":24488,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.189:6835/599473205","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6834","nonce":599473205},{"type":"v1","addr":"172.21.15.189:6835","nonce":599473205}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":6}],"filesystems":[]} 2024-03-24T00:12:34.934 INFO:teuthology.orchestra.run.smithi118.stderr:dumped fsmap epoch 6 2024-03-24T00:12:34.936 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:34.935+0000 7f3f50200640 1 -- 172.21.15.118:0/771446215 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7f3f300a2800 msgr2=0x7f3f300a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:12:34.936 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:34.935+0000 7f3f50200640 1 --2- 172.21.15.118:0/771446215 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7f3f300a2800 0x7f3f300a4cc0 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7f3f4819b0e0 tx=0x7f3f3400b390 comp rx=0 tx=0).stop 2024-03-24T00:12:34.936 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:34.935+0000 7f3f50200640 1 -- 172.21.15.118:0/771446215 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f3f481038e0 msgr2=0x7f3f48199bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:12:34.936 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:34.935+0000 7f3f50200640 1 --2- 172.21.15.118:0/771446215 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f3f481038e0 0x7f3f48199bd0 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7f3f40000f80 tx=0x7f3f40032ec0 comp rx=0 tx=0).stop 2024-03-24T00:12:34.937 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:34.936+0000 7f3f50200640 1 -- 172.21.15.118:0/771446215 shutdown_connections 2024-03-24T00:12:34.937 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:34.936+0000 7f3f50200640 1 --2- 172.21.15.118:0/771446215 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7f3f300a2800 0x7f3f300a4cc0 unknown :-1 s=CLOSED pgs=137 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:12:34.937 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:34.936+0000 7f3f50200640 1 --2- 172.21.15.118:0/771446215 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3f48105490 0x7f3f481a1180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:12:34.937 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:34.936+0000 7f3f50200640 1 --2- 172.21.15.118:0/771446215 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f3f48104ad0 0x7f3f4819a110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:12:34.937 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:34.936+0000 7f3f50200640 1 --2- 172.21.15.118:0/771446215 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f3f481038e0 0x7f3f48199bd0 unknown :-1 s=CLOSED pgs=144 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:12:34.937 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:34.936+0000 7f3f50200640 1 -- 172.21.15.118:0/771446215 >> 172.21.15.118:0/771446215 conn(0x7f3f480fee90 msgr2=0x7f3f48107dd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:12:34.937 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:34.936+0000 7f3f50200640 1 -- 172.21.15.118:0/771446215 shutdown_connections 2024-03-24T00:12:34.937 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:34.936+0000 7f3f50200640 1 -- 172.21.15.118:0/771446215 wait complete. 2024-03-24T00:12:34.945 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:12:34 smithi118 ceph-mon[31535]: Deploying daemon mds.f on smithi193 2024-03-24T00:12:34.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:12:34 smithi118 ceph-mon[31535]: pgmap v296: 1 pgs: 1 active+clean; 577 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-24T00:12:34.953 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:12:34 smithi189 ceph-mon[36187]: Deploying daemon mds.f on smithi193 2024-03-24T00:12:34.953 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:12:34 smithi189 ceph-mon[36187]: pgmap v296: 1 pgs: 1 active+clean; 577 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-24T00:12:34.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:12:34 smithi193 ceph-mon[36878]: Deploying daemon mds.f on smithi193 2024-03-24T00:12:34.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:12:34 smithi193 ceph-mon[36878]: pgmap v296: 1 pgs: 1 active+clean; 577 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-24T00:12:35.404 INFO:journalctl@ceph.mds.f.smithi193.stdout:Mar 24 00:12:35 smithi193 systemd[1]: Starting Ceph mds.f for d79a5420-e971-11ee-95cd-87774f69a715... 2024-03-24T00:12:35.673 INFO:journalctl@ceph.mds.f.smithi193.stdout:Mar 24 00:12:35 smithi193 podman[67450]: 2024-03-24 00:12:35.404248673 +0000 UTC m=+0.111559149 container create b0ac59f98847bdf62c8c530bbedfda67e984890ecfc89cae5f599bd350b72480 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-f, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.build-date=20240319, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, ceph=True) 2024-03-24T00:12:35.674 INFO:journalctl@ceph.mds.f.smithi193.stdout:Mar 24 00:12:35 smithi193 podman[67450]: 2024-03-24 00:12:35.310747837 +0000 UTC m=+0.018058324 image pull c664afe56bb01db4f9afa9532c20598e201e664f2e9ba2be50db0d52674a26d2 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc 2024-03-24T00:12:35.674 INFO:journalctl@ceph.mds.f.smithi193.stdout:Mar 24 00:12:35 smithi193 podman[67450]: 2024-03-24 00:12:35.648656997 +0000 UTC m=+0.355967492 container init b0ac59f98847bdf62c8c530bbedfda67e984890ecfc89cae5f599bd350b72480 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-f, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, GIT_CLEAN=True, org.label-schema.license=GPLv2, ceph=True, org.label-schema.build-date=20240319, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30) 2024-03-24T00:12:35.674 INFO:journalctl@ceph.mds.f.smithi193.stdout:Mar 24 00:12:35 smithi193 podman[67450]: 2024-03-24 00:12:35.651297765 +0000 UTC m=+0.358608242 container start b0ac59f98847bdf62c8c530bbedfda67e984890ecfc89cae5f599bd350b72480 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-f, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240319, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, io.buildah.version=1.33.5, GIT_CLEAN=True, GIT_BRANCH=HEAD) 2024-03-24T00:12:35.960 INFO:journalctl@ceph.mds.f.smithi193.stdout:Mar 24 00:12:35 smithi193 ceph-mds[67469]: -- 172.21.15.193:0/1034870447 <== mon.1 v2:172.21.15.193:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x555b07033a40 con 0x555b07006c00 2024-03-24T00:12:35.961 INFO:journalctl@ceph.mds.f.smithi193.stdout:Mar 24 00:12:35 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-f[67465]: starting mds.f at 2024-03-24T00:12:35.961 INFO:journalctl@ceph.mds.f.smithi193.stdout:Mar 24 00:12:35 smithi193 bash[67450]: b0ac59f98847bdf62c8c530bbedfda67e984890ecfc89cae5f599bd350b72480 2024-03-24T00:12:35.961 INFO:journalctl@ceph.mds.f.smithi193.stdout:Mar 24 00:12:35 smithi193 systemd[1]: Started Ceph mds.f for d79a5420-e971-11ee-95cd-87774f69a715. 2024-03-24T00:12:35.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:12:35 smithi193 ceph-mon[36878]: from='client.? 172.21.15.118:0/771446215' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-24T00:12:36.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:12:35 smithi118 ceph-mon[31535]: from='client.? 172.21.15.118:0/771446215' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-24T00:12:36.203 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:12:35 smithi189 ceph-mon[36187]: from='client.? 172.21.15.118:0/771446215' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-24T00:12:36.776 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:12:36 smithi193 ceph-mon[36878]: pgmap v297: 1 pgs: 1 active+clean; 577 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-24T00:12:36.777 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:12:36 smithi193 ceph-mon[36878]: mds.? [v2:172.21.15.193:6834/2055310541,v1:172.21.15.193:6835/2055310541] up:boot 2024-03-24T00:12:36.777 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:12:36 smithi193 ceph-mon[36878]: fsmap 6 up:standby 2024-03-24T00:12:36.777 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:12:36 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-03-24T00:12:36.777 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:12:36 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T00:12:36.777 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:12:36 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T00:12:36.777 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:12:36 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T00:12:36.777 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:12:36 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.c", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-24T00:12:36.777 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:12:36 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.c", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-24T00:12:36.777 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:12:36 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T00:12:36.957 DEBUG:teuthology.orchestra.run.smithi118:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-24T00:12:37.043 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:12:36 smithi118 ceph-mon[31535]: pgmap v297: 1 pgs: 1 active+clean; 577 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-24T00:12:37.044 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:12:36 smithi118 ceph-mon[31535]: mds.? [v2:172.21.15.193:6834/2055310541,v1:172.21.15.193:6835/2055310541] up:boot 2024-03-24T00:12:37.044 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:12:36 smithi118 ceph-mon[31535]: fsmap 6 up:standby 2024-03-24T00:12:37.044 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:12:36 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-03-24T00:12:37.044 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:12:36 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T00:12:37.044 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:12:36 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T00:12:37.044 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:12:36 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T00:12:37.045 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:12:36 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.c", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-24T00:12:37.045 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:12:36 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.c", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-24T00:12:37.045 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:12:36 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T00:12:37.176 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:37.175+0000 7f675ae00640 1 -- 172.21.15.118:0/1761851595 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f6754109890 msgr2=0x7f6754110120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:12:37.176 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:37.175+0000 7f675ae00640 1 --2- 172.21.15.118:0/1761851595 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f6754109890 0x7f6754110120 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f674c009ca0 tx=0x7f674c02c100 comp rx=0 tx=0).stop 2024-03-24T00:12:37.177 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:37.175+0000 7f675ae00640 1 -- 172.21.15.118:0/1761851595 shutdown_connections 2024-03-24T00:12:37.177 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:37.175+0000 7f675ae00640 1 --2- 172.21.15.118:0/1761851595 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f6754109890 0x7f6754110120 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:12:37.177 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:37.175+0000 7f675ae00640 1 --2- 172.21.15.118:0/1761851595 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f6754108ed0 0x7f6754109350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:12:37.177 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:37.175+0000 7f675ae00640 1 --2- 172.21.15.118:0/1761851595 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6754107ce0 0x7f6754108100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:12:37.177 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:37.175+0000 7f675ae00640 1 -- 172.21.15.118:0/1761851595 >> 172.21.15.118:0/1761851595 conn(0x7f6754076cb0 msgr2=0x7f67540790d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:12:37.177 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:37.175+0000 7f675ae00640 1 -- 172.21.15.118:0/1761851595 shutdown_connections 2024-03-24T00:12:37.177 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:37.175+0000 7f675ae00640 1 -- 172.21.15.118:0/1761851595 wait complete. 2024-03-24T00:12:37.179 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:37.176+0000 7f675ae00640 1 Processor -- start 2024-03-24T00:12:37.179 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:37.176+0000 7f675ae00640 1 -- start start 2024-03-24T00:12:37.179 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:37.177+0000 7f675ae00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6754107ce0 0x7f675419df80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:12:37.179 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:37.177+0000 7f675ae00640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f6754108ed0 0x7f675419e4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:12:37.180 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:37.177+0000 7f675ae00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f6754109890 0x7f67541a5530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:12:37.180 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:37.177+0000 7f675ae00640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6754113960 con 0x7f6754108ed0 2024-03-24T00:12:37.180 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:37.177+0000 7f675ae00640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f67541137e0 con 0x7f6754107ce0 2024-03-24T00:12:37.180 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:37.177+0000 7f675ae00640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6754113ae0 con 0x7f6754109890 2024-03-24T00:12:37.180 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:37.177+0000 7f674b400640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f6754108ed0 0x7f675419e4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:12:37.180 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:37.177+0000 7f674b400640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f6754108ed0 0x7f675419e4c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.118:3300/0 says I am v2:172.21.15.118:44154/0 (socket says 172.21.15.118:44154) 2024-03-24T00:12:37.180 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:37.177+0000 7f674b400640 1 -- 172.21.15.118:0/3315188560 learned_addr learned my addr 172.21.15.118:0/3315188560 (peer_addr_for_me v2:172.21.15.118:0/0) 2024-03-24T00:12:37.180 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:37.177+0000 7f6753e00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6754107ce0 0x7f675419df80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:12:37.180 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:37.177+0000 7f6758e00640 1 --2- 172.21.15.118:0/3315188560 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f6754109890 0x7f67541a5530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:12:37.180 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:37.177+0000 7f674b400640 1 -- 172.21.15.118:0/3315188560 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f6754109890 msgr2=0x7f67541a5530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:12:37.180 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:37.177+0000 7f674b400640 1 --2- 172.21.15.118:0/3315188560 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f6754109890 0x7f67541a5530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:12:37.180 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:37.177+0000 7f674b400640 1 -- 172.21.15.118:0/3315188560 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6754107ce0 msgr2=0x7f675419df80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:12:37.180 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:37.177+0000 7f674b400640 1 --2- 172.21.15.118:0/3315188560 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6754107ce0 0x7f675419df80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:12:37.180 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:37.177+0000 7f674b400640 1 -- 172.21.15.118:0/3315188560 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f67541a5ba0 con 0x7f6754108ed0 2024-03-24T00:12:37.180 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:37.177+0000 7f6758e00640 1 --2- 172.21.15.118:0/3315188560 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f6754109890 0x7f67541a5530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T00:12:37.181 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:37.178+0000 7f674b400640 1 --2- 172.21.15.118:0/3315188560 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f6754108ed0 0x7f675419e4c0 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7f673c00e9f0 tx=0x7f673c00eeb0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:12:37.181 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:37.178+0000 7f6751600640 1 -- 172.21.15.118:0/3315188560 <== mon.0 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f673c00cd80 con 0x7f6754108ed0 2024-03-24T00:12:37.181 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:37.178+0000 7f6751600640 1 -- 172.21.15.118:0/3315188560 <== mon.0 v2:172.21.15.118:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f673c010910 con 0x7f6754108ed0 2024-03-24T00:12:37.181 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:37.178+0000 7f675ae00640 1 -- 172.21.15.118:0/3315188560 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f67541a5e90 con 0x7f6754108ed0 2024-03-24T00:12:37.181 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:37.178+0000 7f6751600640 1 -- 172.21.15.118:0/3315188560 <== mon.0 v2:172.21.15.118:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f673c018990 con 0x7f6754108ed0 2024-03-24T00:12:37.181 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:37.178+0000 7f675ae00640 1 -- 172.21.15.118:0/3315188560 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f67541a63a0 con 0x7f6754108ed0 2024-03-24T00:12:37.181 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:37.179+0000 7f675ae00640 1 -- 172.21.15.118:0/3315188560 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f675410f000 con 0x7f6754108ed0 2024-03-24T00:12:37.182 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:37.181+0000 7f6751600640 1 -- 172.21.15.118:0/3315188560 <== mon.0 v2:172.21.15.118:3300/0 4 ==== mgrmap(e 16) v1 ==== 137291+0+0 (secure 0 0 0) 0x7f673c010ac0 con 0x7f6754108ed0 2024-03-24T00:12:37.183 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:37.182+0000 7f6751600640 1 --2- 172.21.15.118:0/3315188560 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7f67380a2880 0x7f67380a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:12:37.183 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:37.183+0000 7f6751600640 1 -- 172.21.15.118:0/3315188560 <== mon.0 v2:172.21.15.118:3300/0 5 ==== osd_map(82..82 src has 1..82) v4 ==== 7127+0+0 (secure 0 0 0) 0x7f673c0cec80 con 0x7f6754108ed0 2024-03-24T00:12:37.183 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:37.183+0000 7f6753e00640 1 --2- 172.21.15.118:0/3315188560 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7f67380a2880 0x7f67380a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:12:37.184 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:37.184+0000 7f6753e00640 1 --2- 172.21.15.118:0/3315188560 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7f67380a2880 0x7f67380a4d40 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7f6754108d30 tx=0x7f6744005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:12:37.185 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:37.185+0000 7f6751600640 1 -- 172.21.15.118:0/3315188560 <== mon.0 v2:172.21.15.118:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f673c09bad0 con 0x7f6754108ed0 2024-03-24T00:12:37.203 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:12:36 smithi189 ceph-mon[36187]: pgmap v297: 1 pgs: 1 active+clean; 577 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-24T00:12:37.203 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:12:36 smithi189 ceph-mon[36187]: mds.? [v2:172.21.15.193:6834/2055310541,v1:172.21.15.193:6835/2055310541] up:boot 2024-03-24T00:12:37.203 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:12:36 smithi189 ceph-mon[36187]: fsmap 6 up:standby 2024-03-24T00:12:37.203 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:12:36 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-03-24T00:12:37.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:12:36 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T00:12:37.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:12:36 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T00:12:37.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:12:36 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T00:12:37.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:12:36 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.c", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-24T00:12:37.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:12:36 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.c", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-24T00:12:37.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:12:36 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T00:12:37.423 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:37.423+0000 7f675ae00640 1 -- 172.21.15.118:0/3315188560 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f675402d070 con 0x7f6754108ed0 2024-03-24T00:12:37.425 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:37.424+0000 7f6751600640 1 -- 172.21.15.118:0/3315188560 <== mon.0 v2:172.21.15.118:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 7 v7) v1 ==== 93+0+5295 (secure 0 0 0) 0x7f673c09fb70 con 0x7f6754108ed0 2024-03-24T00:12:37.425 INFO:teuthology.orchestra.run.smithi118.stdout: 2024-03-24T00:12:37.425 INFO:teuthology.orchestra.run.smithi118.stdout:{"epoch":7,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14385,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.118:6835/3194741373","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6834","nonce":3194741373},{"type":"v1","addr":"172.21.15.118:6835","nonce":3194741373}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":4},{"gid":14397,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.118:6837/1374462956","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6836","nonce":1374462956},{"type":"v1","addr":"172.21.15.118:6837","nonce":1374462956}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":5},{"gid":24439,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.193:6833/1023428412","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6832","nonce":1023428412},{"type":"v1","addr":"172.21.15.193:6833","nonce":1023428412}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":24440,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.189:6833/2740024078","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6832","nonce":2740024078},{"type":"v1","addr":"172.21.15.189:6833","nonce":2740024078}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":24488,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.189:6835/599473205","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6834","nonce":599473205},{"type":"v1","addr":"172.21.15.189:6835","nonce":599473205}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":6},{"gid":24499,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.193:6835/2055310541","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6834","nonce":2055310541},{"type":"v1","addr":"172.21.15.193:6835","nonce":2055310541}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":7}],"filesystems":[]} 2024-03-24T00:12:37.425 INFO:teuthology.orchestra.run.smithi118.stderr:dumped fsmap epoch 7 2024-03-24T00:12:37.427 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:37.427+0000 7f675ae00640 1 -- 172.21.15.118:0/3315188560 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7f67380a2880 msgr2=0x7f67380a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:12:37.428 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:37.427+0000 7f675ae00640 1 --2- 172.21.15.118:0/3315188560 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7f67380a2880 0x7f67380a4d40 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7f6754108d30 tx=0x7f6744005f90 comp rx=0 tx=0).stop 2024-03-24T00:12:37.428 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:37.427+0000 7f675ae00640 1 -- 172.21.15.118:0/3315188560 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f6754108ed0 msgr2=0x7f675419e4c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:12:37.428 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:37.427+0000 7f675ae00640 1 --2- 172.21.15.118:0/3315188560 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f6754108ed0 0x7f675419e4c0 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7f673c00e9f0 tx=0x7f673c00eeb0 comp rx=0 tx=0).stop 2024-03-24T00:12:37.428 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:37.427+0000 7f675ae00640 1 -- 172.21.15.118:0/3315188560 shutdown_connections 2024-03-24T00:12:37.428 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:37.427+0000 7f675ae00640 1 --2- 172.21.15.118:0/3315188560 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7f67380a2880 0x7f67380a4d40 unknown :-1 s=CLOSED pgs=140 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:12:37.428 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:37.427+0000 7f675ae00640 1 --2- 172.21.15.118:0/3315188560 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f6754109890 0x7f67541a5530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:12:37.428 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:37.427+0000 7f675ae00640 1 --2- 172.21.15.118:0/3315188560 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f6754108ed0 0x7f675419e4c0 unknown :-1 s=CLOSED pgs=145 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:12:37.428 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:37.427+0000 7f675ae00640 1 --2- 172.21.15.118:0/3315188560 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6754107ce0 0x7f675419df80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:12:37.428 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:37.427+0000 7f675ae00640 1 -- 172.21.15.118:0/3315188560 >> 172.21.15.118:0/3315188560 conn(0x7f6754076cb0 msgr2=0x7f675410c1d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:12:37.428 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:37.427+0000 7f675ae00640 1 -- 172.21.15.118:0/3315188560 shutdown_connections 2024-03-24T00:12:37.428 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:37.427+0000 7f675ae00640 1 -- 172.21.15.118:0/3315188560 wait complete. 2024-03-24T00:12:37.957 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:12:37 smithi193 ceph-mon[36878]: Deploying daemon mds.c on smithi193 2024-03-24T00:12:37.958 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:12:37 smithi193 ceph-mon[36878]: from='client.? 172.21.15.118:0/3315188560' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-24T00:12:38.166 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:12:37 smithi118 ceph-mon[31535]: Deploying daemon mds.c on smithi193 2024-03-24T00:12:38.166 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:12:37 smithi118 ceph-mon[31535]: from='client.? 172.21.15.118:0/3315188560' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-24T00:12:38.203 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:12:37 smithi189 ceph-mon[36187]: Deploying daemon mds.c on smithi193 2024-03-24T00:12:38.203 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:12:37 smithi189 ceph-mon[36187]: from='client.? 172.21.15.118:0/3315188560' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-24T00:12:38.210 INFO:journalctl@ceph.mds.c.smithi193.stdout:Mar 24 00:12:38 smithi193 systemd[1]: Starting Ceph mds.c for d79a5420-e971-11ee-95cd-87774f69a715... 2024-03-24T00:12:38.628 INFO:journalctl@ceph.mds.c.smithi193.stdout:Mar 24 00:12:38 smithi193 podman[67759]: 2024-03-24 00:12:38.259947675 +0000 UTC m=+0.017716841 image pull c664afe56bb01db4f9afa9532c20598e201e664f2e9ba2be50db0d52674a26d2 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc 2024-03-24T00:12:38.628 INFO:journalctl@ceph.mds.c.smithi193.stdout:Mar 24 00:12:38 smithi193 podman[67759]: 2024-03-24 00:12:38.376644521 +0000 UTC m=+0.134413697 container create f0210a5dc42f895c3146008a5346994f1d3b5b2168820fde275433feeaac32a4 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-c, CEPH_POINT_RELEASE=, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.build-date=20240319, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, ceph=True, io.buildah.version=1.33.5, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, maintainer=Guillaume Abrioux , GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2) 2024-03-24T00:12:38.628 INFO:journalctl@ceph.mds.c.smithi193.stdout:Mar 24 00:12:38 smithi193 podman[67759]: 2024-03-24 00:12:38.629294741 +0000 UTC m=+0.387063897 container init f0210a5dc42f895c3146008a5346994f1d3b5b2168820fde275433feeaac32a4 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-c, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, ceph=True, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.build-date=20240319, io.buildah.version=1.33.5) 2024-03-24T00:12:38.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:12:38 smithi193 ceph-mon[36878]: pgmap v298: 1 pgs: 1 active+clean; 577 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-24T00:12:38.961 INFO:journalctl@ceph.mds.c.smithi193.stdout:Mar 24 00:12:38 smithi193 podman[67759]: 2024-03-24 00:12:38.63221064 +0000 UTC m=+0.389979803 container start f0210a5dc42f895c3146008a5346994f1d3b5b2168820fde275433feeaac32a4 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-c, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, io.buildah.version=1.33.5, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, GIT_BRANCH=HEAD, org.label-schema.build-date=20240319, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30) 2024-03-24T00:12:38.961 INFO:journalctl@ceph.mds.c.smithi193.stdout:Mar 24 00:12:38 smithi193 ceph-mds[67778]: -- 172.21.15.193:0/2476035402 <== mon.1 v2:172.21.15.193:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x56195e5afa40 con 0x56195e582800 2024-03-24T00:12:38.961 INFO:journalctl@ceph.mds.c.smithi193.stdout:Mar 24 00:12:38 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-c[67774]: starting mds.c at 2024-03-24T00:12:38.961 INFO:journalctl@ceph.mds.c.smithi193.stdout:Mar 24 00:12:38 smithi193 bash[67759]: f0210a5dc42f895c3146008a5346994f1d3b5b2168820fde275433feeaac32a4 2024-03-24T00:12:38.961 INFO:journalctl@ceph.mds.c.smithi193.stdout:Mar 24 00:12:38 smithi193 systemd[1]: Started Ceph mds.c for d79a5420-e971-11ee-95cd-87774f69a715. 2024-03-24T00:12:39.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:12:38 smithi118 ceph-mon[31535]: pgmap v298: 1 pgs: 1 active+clean; 577 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-24T00:12:39.203 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:12:38 smithi189 ceph-mon[36187]: pgmap v298: 1 pgs: 1 active+clean; 577 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-24T00:12:39.446 DEBUG:teuthology.orchestra.run.smithi118:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-24T00:12:39.665 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:39.664+0000 7f3c74c00640 1 -- 172.21.15.118:0/689476192 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3c70107b00 msgr2=0x7f3c70107f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:12:39.665 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:39.664+0000 7f3c74c00640 1 --2- 172.21.15.118:0/689476192 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3c70107b00 0x7f3c70107f20 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f3c60009900 tx=0x7f3c6002bc30 comp rx=0 tx=0).stop 2024-03-24T00:12:39.665 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:39.664+0000 7f3c74c00640 1 -- 172.21.15.118:0/689476192 shutdown_connections 2024-03-24T00:12:39.665 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:39.664+0000 7f3c74c00640 1 --2- 172.21.15.118:0/689476192 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f3c701096b0 0x7f3c7010ff40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:12:39.666 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:39.664+0000 7f3c74c00640 1 --2- 172.21.15.118:0/689476192 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f3c70108cf0 0x7f3c70109170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:12:39.666 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:39.664+0000 7f3c74c00640 1 --2- 172.21.15.118:0/689476192 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3c70107b00 0x7f3c70107f20 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:12:39.666 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:39.664+0000 7f3c74c00640 1 -- 172.21.15.118:0/689476192 >> 172.21.15.118:0/689476192 conn(0x7f3c70076ab0 msgr2=0x7f3c70078ed0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:12:39.666 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:39.664+0000 7f3c74c00640 1 -- 172.21.15.118:0/689476192 shutdown_connections 2024-03-24T00:12:39.666 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:39.664+0000 7f3c74c00640 1 -- 172.21.15.118:0/689476192 wait complete. 2024-03-24T00:12:39.669 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:39.665+0000 7f3c74c00640 1 Processor -- start 2024-03-24T00:12:39.669 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:39.665+0000 7f3c74c00640 1 -- start start 2024-03-24T00:12:39.669 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:39.666+0000 7f3c74c00640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f3c70107b00 0x7f3c7019de30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:12:39.669 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:39.666+0000 7f3c74c00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f3c70108cf0 0x7f3c7019e370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:12:39.669 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:39.666+0000 7f3c74c00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3c701096b0 0x7f3c701a53e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:12:39.669 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:39.666+0000 7f3c74c00640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3c701137c0 con 0x7f3c70107b00 2024-03-24T00:12:39.669 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:39.666+0000 7f3c74c00640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3c70113640 con 0x7f3c701096b0 2024-03-24T00:12:39.669 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:39.666+0000 7f3c74c00640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3c70113940 con 0x7f3c70108cf0 2024-03-24T00:12:39.669 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:39.666+0000 7f3c6f400640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f3c70107b00 0x7f3c7019de30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:12:39.669 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:39.666+0000 7f3c6f400640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f3c70107b00 0x7f3c7019de30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.118:3300/0 says I am v2:172.21.15.118:44180/0 (socket says 172.21.15.118:44180) 2024-03-24T00:12:39.670 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:39.666+0000 7f3c6f400640 1 -- 172.21.15.118:0/958549239 learned_addr learned my addr 172.21.15.118:0/958549239 (peer_addr_for_me v2:172.21.15.118:0/0) 2024-03-24T00:12:39.670 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:39.666+0000 7f3c67e00640 1 --2- 172.21.15.118:0/958549239 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f3c70108cf0 0x7f3c7019e370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:12:39.670 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:39.666+0000 7f3c6fe00640 1 --2- 172.21.15.118:0/958549239 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3c701096b0 0x7f3c701a53e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:12:39.670 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:39.666+0000 7f3c6f400640 1 -- 172.21.15.118:0/958549239 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f3c70108cf0 msgr2=0x7f3c7019e370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:12:39.670 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:39.666+0000 7f3c6f400640 1 --2- 172.21.15.118:0/958549239 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f3c70108cf0 0x7f3c7019e370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:12:39.670 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:39.666+0000 7f3c6f400640 1 -- 172.21.15.118:0/958549239 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3c701096b0 msgr2=0x7f3c701a53e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:12:39.670 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:39.666+0000 7f3c6f400640 1 --2- 172.21.15.118:0/958549239 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3c701096b0 0x7f3c701a53e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:12:39.670 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:39.666+0000 7f3c6f400640 1 -- 172.21.15.118:0/958549239 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3c701a5a50 con 0x7f3c70107b00 2024-03-24T00:12:39.670 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:39.667+0000 7f3c6f400640 1 --2- 172.21.15.118:0/958549239 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f3c70107b00 0x7f3c7019de30 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7f3c60000c00 tx=0x7f3c6002c650 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:12:39.670 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:39.667+0000 7f3c6cc00640 1 -- 172.21.15.118:0/958549239 <== mon.0 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3c60043020 con 0x7f3c70107b00 2024-03-24T00:12:39.670 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:39.667+0000 7f3c6cc00640 1 -- 172.21.15.118:0/958549239 <== mon.0 v2:172.21.15.118:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f3c600054e0 con 0x7f3c70107b00 2024-03-24T00:12:39.670 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:39.667+0000 7f3c74c00640 1 -- 172.21.15.118:0/958549239 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3c701a5ce0 con 0x7f3c70107b00 2024-03-24T00:12:39.670 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:39.667+0000 7f3c6cc00640 1 -- 172.21.15.118:0/958549239 <== mon.0 v2:172.21.15.118:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3c60003c40 con 0x7f3c70107b00 2024-03-24T00:12:39.671 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:39.667+0000 7f3c74c00640 1 -- 172.21.15.118:0/958549239 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3c701a61d0 con 0x7f3c70107b00 2024-03-24T00:12:39.671 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:39.668+0000 7f3c74c00640 1 -- 172.21.15.118:0/958549239 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3c7010ee20 con 0x7f3c70107b00 2024-03-24T00:12:39.671 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:39.670+0000 7f3c6cc00640 1 -- 172.21.15.118:0/958549239 <== mon.0 v2:172.21.15.118:3300/0 4 ==== mgrmap(e 16) v1 ==== 137291+0+0 (secure 0 0 0) 0x7f3c60046020 con 0x7f3c70107b00 2024-03-24T00:12:39.672 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:39.671+0000 7f3c6cc00640 1 --2- 172.21.15.118:0/958549239 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7f3c540a2880 0x7f3c540a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:12:39.672 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:39.672+0000 7f3c6cc00640 1 -- 172.21.15.118:0/958549239 <== mon.0 v2:172.21.15.118:3300/0 5 ==== osd_map(82..82 src has 1..82) v4 ==== 7127+0+0 (secure 0 0 0) 0x7f3c600f0990 con 0x7f3c70107b00 2024-03-24T00:12:39.672 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:39.672+0000 7f3c67e00640 1 --2- 172.21.15.118:0/958549239 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7f3c540a2880 0x7f3c540a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:12:39.673 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:39.672+0000 7f3c67e00640 1 --2- 172.21.15.118:0/958549239 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7f3c540a2880 0x7f3c540a4d40 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7f3c7019f340 tx=0x7f3c58005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:12:39.675 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:39.675+0000 7f3c6cc00640 1 -- 172.21.15.118:0/958549239 <== mon.0 v2:172.21.15.118:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f3c6004d020 con 0x7f3c70107b00 2024-03-24T00:12:39.915 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:39.914+0000 7f3c74c00640 1 -- 172.21.15.118:0/958549239 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f3c7002d070 con 0x7f3c70107b00 2024-03-24T00:12:39.916 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:39.916+0000 7f3c6cc00640 1 -- 172.21.15.118:0/958549239 <== mon.0 v2:172.21.15.118:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 8 v8) v1 ==== 93+0+6089 (secure 0 0 0) 0x7f3c600bde30 con 0x7f3c70107b00 2024-03-24T00:12:39.917 INFO:teuthology.orchestra.run.smithi118.stdout: 2024-03-24T00:12:39.917 INFO:teuthology.orchestra.run.smithi118.stdout:{"epoch":8,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14385,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.118:6835/3194741373","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6834","nonce":3194741373},{"type":"v1","addr":"172.21.15.118:6835","nonce":3194741373}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":4},{"gid":14397,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.118:6837/1374462956","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6836","nonce":1374462956},{"type":"v1","addr":"172.21.15.118:6837","nonce":1374462956}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":5},{"gid":24439,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.193:6833/1023428412","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6832","nonce":1023428412},{"type":"v1","addr":"172.21.15.193:6833","nonce":1023428412}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":24440,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.189:6833/2740024078","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6832","nonce":2740024078},{"type":"v1","addr":"172.21.15.189:6833","nonce":2740024078}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":24488,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.189:6835/599473205","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6834","nonce":599473205},{"type":"v1","addr":"172.21.15.189:6835","nonce":599473205}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":6},{"gid":24499,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.193:6835/2055310541","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6834","nonce":2055310541},{"type":"v1","addr":"172.21.15.193:6835","nonce":2055310541}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":7},{"gid":24511,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.193:6837/2921520053","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6836","nonce":2921520053},{"type":"v1","addr":"172.21.15.193:6837","nonce":2921520053}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":8}],"filesystems":[]} 2024-03-24T00:12:39.917 INFO:teuthology.orchestra.run.smithi118.stderr:dumped fsmap epoch 8 2024-03-24T00:12:39.920 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:39.919+0000 7f3c74c00640 1 -- 172.21.15.118:0/958549239 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7f3c540a2880 msgr2=0x7f3c540a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:12:39.920 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:39.919+0000 7f3c74c00640 1 --2- 172.21.15.118:0/958549239 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7f3c540a2880 0x7f3c540a4d40 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7f3c7019f340 tx=0x7f3c58005f90 comp rx=0 tx=0).stop 2024-03-24T00:12:39.920 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:39.919+0000 7f3c74c00640 1 -- 172.21.15.118:0/958549239 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f3c70107b00 msgr2=0x7f3c7019de30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:12:39.920 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:39.919+0000 7f3c74c00640 1 --2- 172.21.15.118:0/958549239 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f3c70107b00 0x7f3c7019de30 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7f3c60000c00 tx=0x7f3c6002c650 comp rx=0 tx=0).stop 2024-03-24T00:12:39.920 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:39.919+0000 7f3c74c00640 1 -- 172.21.15.118:0/958549239 shutdown_connections 2024-03-24T00:12:39.920 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:39.919+0000 7f3c74c00640 1 --2- 172.21.15.118:0/958549239 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7f3c540a2880 0x7f3c540a4d40 unknown :-1 s=CLOSED pgs=142 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:12:39.920 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:39.919+0000 7f3c74c00640 1 --2- 172.21.15.118:0/958549239 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3c701096b0 0x7f3c701a53e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:12:39.920 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:39.919+0000 7f3c74c00640 1 --2- 172.21.15.118:0/958549239 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f3c70108cf0 0x7f3c7019e370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:12:39.920 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:39.919+0000 7f3c74c00640 1 --2- 172.21.15.118:0/958549239 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f3c70107b00 0x7f3c7019de30 unknown :-1 s=CLOSED pgs=146 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:12:39.920 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:39.919+0000 7f3c74c00640 1 -- 172.21.15.118:0/958549239 >> 172.21.15.118:0/958549239 conn(0x7f3c70076ab0 msgr2=0x7f3c7010bff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:12:39.920 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:39.919+0000 7f3c74c00640 1 -- 172.21.15.118:0/958549239 shutdown_connections 2024-03-24T00:12:39.920 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:39.919+0000 7f3c74c00640 1 -- 172.21.15.118:0/958549239 wait complete. 2024-03-24T00:12:39.936 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:12:39 smithi118 ceph-mon[31535]: mds.? [v2:172.21.15.193:6836/2921520053,v1:172.21.15.193:6837/2921520053] up:boot 2024-03-24T00:12:39.937 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:12:39 smithi118 ceph-mon[31535]: fsmap 7 up:standby 2024-03-24T00:12:39.937 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:12:39 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-03-24T00:12:39.937 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:12:39 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T00:12:39.937 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:12:39 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T00:12:39.937 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:12:39 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T00:12:39.937 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:12:39 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.h", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-24T00:12:39.937 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:12:39 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.h", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-24T00:12:39.937 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:12:39 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T00:12:39.937 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:12:39 smithi118 ceph-mon[31535]: Deploying daemon mds.h on smithi189 2024-03-24T00:12:39.937 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:12:39 smithi118 ceph-mon[31535]: pgmap v299: 1 pgs: 1 active+clean; 577 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-24T00:12:40.203 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:12:39 smithi189 ceph-mon[36187]: mds.? [v2:172.21.15.193:6836/2921520053,v1:172.21.15.193:6837/2921520053] up:boot 2024-03-24T00:12:40.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:12:39 smithi189 ceph-mon[36187]: fsmap 7 up:standby 2024-03-24T00:12:40.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:12:39 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-03-24T00:12:40.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:12:39 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T00:12:40.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:12:39 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T00:12:40.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:12:39 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T00:12:40.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:12:39 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.h", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-24T00:12:40.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:12:39 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.h", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-24T00:12:40.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:12:39 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T00:12:40.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:12:39 smithi189 ceph-mon[36187]: Deploying daemon mds.h on smithi189 2024-03-24T00:12:40.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:12:39 smithi189 ceph-mon[36187]: pgmap v299: 1 pgs: 1 active+clean; 577 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-24T00:12:40.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:12:39 smithi193 ceph-mon[36878]: mds.? [v2:172.21.15.193:6836/2921520053,v1:172.21.15.193:6837/2921520053] up:boot 2024-03-24T00:12:40.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:12:39 smithi193 ceph-mon[36878]: fsmap 7 up:standby 2024-03-24T00:12:40.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:12:39 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-03-24T00:12:40.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:12:39 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T00:12:40.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:12:39 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T00:12:40.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:12:39 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T00:12:40.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:12:39 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.h", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-24T00:12:40.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:12:39 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.h", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-24T00:12:40.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:12:39 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T00:12:40.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:12:39 smithi193 ceph-mon[36878]: Deploying daemon mds.h on smithi189 2024-03-24T00:12:40.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:12:39 smithi193 ceph-mon[36878]: pgmap v299: 1 pgs: 1 active+clean; 577 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-24T00:12:41.193 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:12:40 smithi189 ceph-mon[36187]: from='client.? 172.21.15.118:0/958549239' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-24T00:12:41.194 INFO:journalctl@ceph.mds.h.smithi189.stdout:Mar 24 00:12:40 smithi189 systemd[1]: Starting Ceph mds.h for d79a5420-e971-11ee-95cd-87774f69a715... 2024-03-24T00:12:41.195 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:12:40 smithi118 ceph-mon[31535]: from='client.? 172.21.15.118:0/958549239' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-24T00:12:41.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:12:40 smithi193 ceph-mon[36878]: from='client.? 172.21.15.118:0/958549239' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-24T00:12:41.453 INFO:journalctl@ceph.mds.h.smithi189.stdout:Mar 24 00:12:41 smithi189 podman[66656]: 2024-03-24 00:12:41.096971442 +0000 UTC m=+0.018430124 image pull c664afe56bb01db4f9afa9532c20598e201e664f2e9ba2be50db0d52674a26d2 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc 2024-03-24T00:12:41.454 INFO:journalctl@ceph.mds.h.smithi189.stdout:Mar 24 00:12:41 smithi189 podman[66656]: 2024-03-24 00:12:41.222835125 +0000 UTC m=+0.144293793 container create 8cde5e7f81746198147a07f104c44245acf3afed80c4f5b597e0a01e660f8672 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-h, maintainer=Guillaume Abrioux , GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240319, ceph=True, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2) 2024-03-24T00:12:41.940 DEBUG:teuthology.orchestra.run.smithi118:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-24T00:12:41.953 INFO:journalctl@ceph.mds.h.smithi189.stdout:Mar 24 00:12:41 smithi189 podman[66656]: 2024-03-24 00:12:41.458859425 +0000 UTC m=+0.380318101 container init 8cde5e7f81746198147a07f104c44245acf3afed80c4f5b597e0a01e660f8672 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-h, org.label-schema.license=GPLv2, org.label-schema.build-date=20240319, CEPH_POINT_RELEASE=, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, ceph=True, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux ) 2024-03-24T00:12:41.954 INFO:journalctl@ceph.mds.h.smithi189.stdout:Mar 24 00:12:41 smithi189 podman[66656]: 2024-03-24 00:12:41.461921317 +0000 UTC m=+0.383379986 container start 8cde5e7f81746198147a07f104c44245acf3afed80c4f5b597e0a01e660f8672 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-h, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, ceph=True, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240319, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2) 2024-03-24T00:12:41.954 INFO:journalctl@ceph.mds.h.smithi189.stdout:Mar 24 00:12:41 smithi189 ceph-mds[66675]: -- 172.21.15.189:0/4142176558 <== mon.2 v2:172.21.15.189:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x56516138ba40 con 0x56516135f000 2024-03-24T00:12:41.954 INFO:journalctl@ceph.mds.h.smithi189.stdout:Mar 24 00:12:41 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-h[66671]: starting mds.h at 2024-03-24T00:12:41.954 INFO:journalctl@ceph.mds.h.smithi189.stdout:Mar 24 00:12:41 smithi189 bash[66656]: 8cde5e7f81746198147a07f104c44245acf3afed80c4f5b597e0a01e660f8672 2024-03-24T00:12:41.954 INFO:journalctl@ceph.mds.h.smithi189.stdout:Mar 24 00:12:41 smithi189 systemd[1]: Started Ceph mds.h for d79a5420-e971-11ee-95cd-87774f69a715. 2024-03-24T00:12:42.162 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:42.161+0000 7f6d51e00640 1 -- 172.21.15.118:0/1091551342 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f6d4c1008f0 msgr2=0x7f6d4c100d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:12:42.163 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:42.161+0000 7f6d51e00640 1 --2- 172.21.15.118:0/1091551342 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f6d4c1008f0 0x7f6d4c100d50 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7f6d34009900 tx=0x7f6d3402bc30 comp rx=0 tx=0).stop 2024-03-24T00:12:42.163 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:42.162+0000 7f6d51e00640 1 -- 172.21.15.118:0/1091551342 shutdown_connections 2024-03-24T00:12:42.163 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:42.162+0000 7f6d51e00640 1 --2- 172.21.15.118:0/1091551342 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6d4c101290 0x7f6d4c101710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:12:42.163 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:42.162+0000 7f6d51e00640 1 --2- 172.21.15.118:0/1091551342 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f6d4c1008f0 0x7f6d4c100d50 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:12:42.163 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:42.162+0000 7f6d51e00640 1 --2- 172.21.15.118:0/1091551342 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f6d4c0fff90 0x7f6d4c1003b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:12:42.163 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:42.162+0000 7f6d51e00640 1 -- 172.21.15.118:0/1091551342 >> 172.21.15.118:0/1091551342 conn(0x7f6d4c0fbb30 msgr2=0x7f6d4c0fdf70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:12:42.163 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:42.162+0000 7f6d51e00640 1 -- 172.21.15.118:0/1091551342 shutdown_connections 2024-03-24T00:12:42.163 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:42.162+0000 7f6d51e00640 1 -- 172.21.15.118:0/1091551342 wait complete. 2024-03-24T00:12:42.165 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:42.163+0000 7f6d51e00640 1 Processor -- start 2024-03-24T00:12:42.165 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:42.163+0000 7f6d51e00640 1 -- start start 2024-03-24T00:12:42.166 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:42.163+0000 7f6d51e00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f6d4c0fff90 0x7f6d4c199bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:12:42.166 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:42.163+0000 7f6d51e00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6d4c1008f0 0x7f6d4c19a100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:12:42.166 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:42.163+0000 7f6d51e00640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f6d4c101290 0x7f6d4c1a1170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:12:42.166 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:42.163+0000 7f6d51e00640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6d4c10f550 con 0x7f6d4c101290 2024-03-24T00:12:42.166 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:42.163+0000 7f6d51e00640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6d4c10f3d0 con 0x7f6d4c1008f0 2024-03-24T00:12:42.166 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:42.163+0000 7f6d51e00640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6d4c10f6d0 con 0x7f6d4c0fff90 2024-03-24T00:12:42.166 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:42.163+0000 7f6d4aa00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f6d4c0fff90 0x7f6d4c199bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:12:42.166 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:42.163+0000 7f6d4b400640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f6d4c101290 0x7f6d4c1a1170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:12:42.166 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:42.163+0000 7f6d4aa00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f6d4c0fff90 0x7f6d4c199bc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.118:55526/0 (socket says 172.21.15.118:55526) 2024-03-24T00:12:42.166 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:42.163+0000 7f6d4aa00640 1 -- 172.21.15.118:0/4194293827 learned_addr learned my addr 172.21.15.118:0/4194293827 (peer_addr_for_me v2:172.21.15.118:0/0) 2024-03-24T00:12:42.166 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:42.163+0000 7f6d43e00640 1 --2- 172.21.15.118:0/4194293827 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6d4c1008f0 0x7f6d4c19a100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:12:42.166 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:42.164+0000 7f6d4aa00640 1 -- 172.21.15.118:0/4194293827 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6d4c1008f0 msgr2=0x7f6d4c19a100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:12:42.166 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:42.164+0000 7f6d4aa00640 1 --2- 172.21.15.118:0/4194293827 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6d4c1008f0 0x7f6d4c19a100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:12:42.167 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:42.164+0000 7f6d4aa00640 1 -- 172.21.15.118:0/4194293827 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f6d4c101290 msgr2=0x7f6d4c1a1170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:12:42.167 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:42.164+0000 7f6d4aa00640 1 --2- 172.21.15.118:0/4194293827 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f6d4c101290 0x7f6d4c1a1170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:12:42.167 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:42.164+0000 7f6d4aa00640 1 -- 172.21.15.118:0/4194293827 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6d4c1a17e0 con 0x7f6d4c0fff90 2024-03-24T00:12:42.167 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:42.164+0000 7f6d4b400640 1 --2- 172.21.15.118:0/4194293827 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f6d4c101290 0x7f6d4c1a1170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T00:12:42.167 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:42.164+0000 7f6d43e00640 1 --2- 172.21.15.118:0/4194293827 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6d4c1008f0 0x7f6d4c19a100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T00:12:42.167 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:42.164+0000 7f6d4aa00640 1 --2- 172.21.15.118:0/4194293827 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f6d4c0fff90 0x7f6d4c199bc0 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f6d3c00e9f0 tx=0x7f6d3c00edf0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:12:42.167 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:42.164+0000 7f6d50a00640 1 -- 172.21.15.118:0/4194293827 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6d3c00ccc0 con 0x7f6d4c0fff90 2024-03-24T00:12:42.167 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:42.164+0000 7f6d50a00640 1 -- 172.21.15.118:0/4194293827 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f6d3c010910 con 0x7f6d4c0fff90 2024-03-24T00:12:42.167 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:42.164+0000 7f6d51e00640 1 -- 172.21.15.118:0/4194293827 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6d4c1a1ad0 con 0x7f6d4c0fff90 2024-03-24T00:12:42.167 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:42.164+0000 7f6d50a00640 1 -- 172.21.15.118:0/4194293827 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6d3c0188a0 con 0x7f6d4c0fff90 2024-03-24T00:12:42.167 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:42.165+0000 7f6d51e00640 1 -- 172.21.15.118:0/4194293827 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6d4c1a1ff0 con 0x7f6d4c0fff90 2024-03-24T00:12:42.168 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:42.166+0000 7f6d51e00640 1 -- 172.21.15.118:0/4194293827 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6d4c10ae80 con 0x7f6d4c0fff90 2024-03-24T00:12:42.168 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:42.167+0000 7f6d50a00640 1 -- 172.21.15.118:0/4194293827 <== mon.1 v2:172.21.15.193:3300/0 4 ==== mgrmap(e 16) v1 ==== 137291+0+0 (secure 0 0 0) 0x7f6d3c010ac0 con 0x7f6d4c0fff90 2024-03-24T00:12:42.169 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:42.169+0000 7f6d50a00640 1 --2- 172.21.15.118:0/4194293827 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7f6d300a2800 0x7f6d300a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:12:42.169 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:42.169+0000 7f6d50a00640 1 -- 172.21.15.118:0/4194293827 <== mon.1 v2:172.21.15.193:3300/0 5 ==== osd_map(82..82 src has 1..82) v4 ==== 7127+0+0 (secure 0 0 0) 0x7f6d3c0cea80 con 0x7f6d4c0fff90 2024-03-24T00:12:42.169 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:42.169+0000 7f6d43e00640 1 --2- 172.21.15.118:0/4194293827 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7f6d300a2800 0x7f6d300a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:12:42.170 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:42.170+0000 7f6d43e00640 1 --2- 172.21.15.118:0/4194293827 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7f6d300a2800 0x7f6d300a4cc0 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7f6d3402c240 tx=0x7f6d3402df90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:12:42.172 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:42.172+0000 7f6d50a00640 1 -- 172.21.15.118:0/4194293827 <== mon.1 v2:172.21.15.193:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f6d3c09bf20 con 0x7f6d4c0fff90 2024-03-24T00:12:42.407 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:42.407+0000 7f6d51e00640 1 -- 172.21.15.118:0/4194293827 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f6d4c02d2e0 con 0x7f6d4c0fff90 2024-03-24T00:12:42.408 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:42.408+0000 7f6d50a00640 1 -- 172.21.15.118:0/4194293827 <== mon.1 v2:172.21.15.193:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 9 v9) v1 ==== 93+0+6883 (secure 0 0 0) 0x7f6d3c09b820 con 0x7f6d4c0fff90 2024-03-24T00:12:42.409 INFO:teuthology.orchestra.run.smithi118.stdout: 2024-03-24T00:12:42.409 INFO:teuthology.orchestra.run.smithi118.stdout:{"epoch":9,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14385,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.118:6835/3194741373","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6834","nonce":3194741373},{"type":"v1","addr":"172.21.15.118:6835","nonce":3194741373}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":4},{"gid":14397,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.118:6837/1374462956","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6836","nonce":1374462956},{"type":"v1","addr":"172.21.15.118:6837","nonce":1374462956}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":5},{"gid":24439,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.193:6833/1023428412","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6832","nonce":1023428412},{"type":"v1","addr":"172.21.15.193:6833","nonce":1023428412}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":24440,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.189:6833/2740024078","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6832","nonce":2740024078},{"type":"v1","addr":"172.21.15.189:6833","nonce":2740024078}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":24488,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.189:6835/599473205","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6834","nonce":599473205},{"type":"v1","addr":"172.21.15.189:6835","nonce":599473205}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":6},{"gid":24499,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.193:6835/2055310541","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6834","nonce":2055310541},{"type":"v1","addr":"172.21.15.193:6835","nonce":2055310541}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":7},{"gid":24511,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.193:6837/2921520053","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6836","nonce":2921520053},{"type":"v1","addr":"172.21.15.193:6837","nonce":2921520053}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":8},{"gid":24515,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.189:6837/3450187137","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6836","nonce":3450187137},{"type":"v1","addr":"172.21.15.189:6837","nonce":3450187137}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":9}],"filesystems":[]} 2024-03-24T00:12:42.409 INFO:teuthology.orchestra.run.smithi118.stderr:dumped fsmap epoch 9 2024-03-24T00:12:42.411 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:42.410+0000 7f6d51e00640 1 -- 172.21.15.118:0/4194293827 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7f6d300a2800 msgr2=0x7f6d300a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:12:42.411 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:42.410+0000 7f6d51e00640 1 --2- 172.21.15.118:0/4194293827 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7f6d300a2800 0x7f6d300a4cc0 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7f6d3402c240 tx=0x7f6d3402df90 comp rx=0 tx=0).stop 2024-03-24T00:12:42.411 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:42.410+0000 7f6d51e00640 1 -- 172.21.15.118:0/4194293827 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f6d4c0fff90 msgr2=0x7f6d4c199bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:12:42.411 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:42.410+0000 7f6d51e00640 1 --2- 172.21.15.118:0/4194293827 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f6d4c0fff90 0x7f6d4c199bc0 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f6d3c00e9f0 tx=0x7f6d3c00edf0 comp rx=0 tx=0).stop 2024-03-24T00:12:42.411 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:42.410+0000 7f6d51e00640 1 -- 172.21.15.118:0/4194293827 shutdown_connections 2024-03-24T00:12:42.411 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:42.410+0000 7f6d51e00640 1 --2- 172.21.15.118:0/4194293827 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7f6d300a2800 0x7f6d300a4cc0 unknown :-1 s=CLOSED pgs=144 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:12:42.412 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:42.410+0000 7f6d51e00640 1 --2- 172.21.15.118:0/4194293827 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f6d4c101290 0x7f6d4c1a1170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:12:42.412 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:42.410+0000 7f6d51e00640 1 --2- 172.21.15.118:0/4194293827 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6d4c1008f0 0x7f6d4c19a100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:12:42.412 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:42.410+0000 7f6d51e00640 1 --2- 172.21.15.118:0/4194293827 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f6d4c0fff90 0x7f6d4c199bc0 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:12:42.412 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:42.410+0000 7f6d51e00640 1 -- 172.21.15.118:0/4194293827 >> 172.21.15.118:0/4194293827 conn(0x7f6d4c0fbb30 msgr2=0x7f6d4c0fd220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:12:42.412 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:42.411+0000 7f6d51e00640 1 -- 172.21.15.118:0/4194293827 shutdown_connections 2024-03-24T00:12:42.412 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:42.411+0000 7f6d51e00640 1 -- 172.21.15.118:0/4194293827 wait complete. 2024-03-24T00:12:42.698 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:12:42 smithi118 ceph-mon[31535]: pgmap v300: 1 pgs: 1 active+clean; 577 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-24T00:12:42.698 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:12:42 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T00:12:42.698 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:12:42 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T00:12:42.698 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:12:42 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T00:12:42.698 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:12:42 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.i", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-24T00:12:42.698 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:12:42 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.i", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-24T00:12:42.698 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:12:42 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T00:12:42.698 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:12:42 smithi118 ceph-mon[31535]: Deploying daemon mds.i on smithi193 2024-03-24T00:12:42.698 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:12:42 smithi118 ceph-mon[31535]: mds.? [v2:172.21.15.189:6836/3450187137,v1:172.21.15.189:6837/3450187137] up:boot 2024-03-24T00:12:42.699 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:12:42 smithi118 ceph-mon[31535]: fsmap 8 up:standby 2024-03-24T00:12:42.699 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:12:42 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-03-24T00:12:42.699 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:12:42 smithi118 ceph-mon[31535]: from='client.? 172.21.15.118:0/4194293827' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-24T00:12:42.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:12:42 smithi189 ceph-mon[36187]: pgmap v300: 1 pgs: 1 active+clean; 577 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-24T00:12:42.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:12:42 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T00:12:42.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:12:42 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T00:12:42.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:12:42 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T00:12:42.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:12:42 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.i", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-24T00:12:42.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:12:42 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.i", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-24T00:12:42.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:12:42 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T00:12:42.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:12:42 smithi189 ceph-mon[36187]: Deploying daemon mds.i on smithi193 2024-03-24T00:12:42.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:12:42 smithi189 ceph-mon[36187]: mds.? [v2:172.21.15.189:6836/3450187137,v1:172.21.15.189:6837/3450187137] up:boot 2024-03-24T00:12:42.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:12:42 smithi189 ceph-mon[36187]: fsmap 8 up:standby 2024-03-24T00:12:42.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:12:42 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-03-24T00:12:42.955 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:12:42 smithi189 ceph-mon[36187]: from='client.? 172.21.15.118:0/4194293827' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-24T00:12:42.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:12:42 smithi193 ceph-mon[36878]: pgmap v300: 1 pgs: 1 active+clean; 577 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-24T00:12:42.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:12:42 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T00:12:42.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:12:42 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T00:12:42.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:12:42 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T00:12:42.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:12:42 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.i", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-24T00:12:42.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:12:42 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.i", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-24T00:12:42.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:12:42 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T00:12:42.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:12:42 smithi193 ceph-mon[36878]: Deploying daemon mds.i on smithi193 2024-03-24T00:12:42.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:12:42 smithi193 ceph-mon[36878]: mds.? [v2:172.21.15.189:6836/3450187137,v1:172.21.15.189:6837/3450187137] up:boot 2024-03-24T00:12:42.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:12:42 smithi193 ceph-mon[36878]: fsmap 8 up:standby 2024-03-24T00:12:42.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:12:42 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-03-24T00:12:42.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:12:42 smithi193 ceph-mon[36878]: from='client.? 172.21.15.118:0/4194293827' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-24T00:12:44.151 INFO:journalctl@ceph.mds.i.smithi193.stdout:Mar 24 00:12:43 smithi193 systemd[1]: Starting Ceph mds.i for d79a5420-e971-11ee-95cd-87774f69a715... 2024-03-24T00:12:44.430 DEBUG:teuthology.orchestra.run.smithi118:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-24T00:12:44.437 INFO:journalctl@ceph.mds.i.smithi193.stdout:Mar 24 00:12:44 smithi193 podman[68073]: 2024-03-24 00:12:44.055237361 +0000 UTC m=+0.018122476 image pull c664afe56bb01db4f9afa9532c20598e201e664f2e9ba2be50db0d52674a26d2 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc 2024-03-24T00:12:44.437 INFO:journalctl@ceph.mds.i.smithi193.stdout:Mar 24 00:12:44 smithi193 podman[68073]: 2024-03-24 00:12:44.19388106 +0000 UTC m=+0.156766153 container create 749e1d96d3972c0c41d21f00c3bff52d847c20bf10030537cb51011beac07fa6 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-i, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240319, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_CLEAN=True, org.label-schema.schema-version=1.0, ceph=True, GIT_BRANCH=HEAD, io.buildah.version=1.33.5) 2024-03-24T00:12:44.437 INFO:journalctl@ceph.mds.i.smithi193.stdout:Mar 24 00:12:44 smithi193 podman[68073]: 2024-03-24 00:12:44.438183142 +0000 UTC m=+0.401068244 container init 749e1d96d3972c0c41d21f00c3bff52d847c20bf10030537cb51011beac07fa6 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-i, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, CEPH_POINT_RELEASE=, ceph=True, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.build-date=20240319, GIT_CLEAN=True, org.label-schema.vendor=CentOS) 2024-03-24T00:12:44.661 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:44.661+0000 7f3c89800640 1 -- 172.21.15.118:0/1335268516 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f3c84104ad0 msgr2=0x7f3c84104f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:12:44.661 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:44.661+0000 7f3c89800640 1 --2- 172.21.15.118:0/1335268516 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f3c84104ad0 0x7f3c84104f50 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f3c70009900 tx=0x7f3c7002bc30 comp rx=0 tx=0).stop 2024-03-24T00:12:44.661 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:44.661+0000 7f3c89800640 1 -- 172.21.15.118:0/1335268516 shutdown_connections 2024-03-24T00:12:44.661 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:44.661+0000 7f3c89800640 1 --2- 172.21.15.118:0/1335268516 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f3c84105490 0x7f3c8410bd20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:12:44.661 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:44.661+0000 7f3c89800640 1 --2- 172.21.15.118:0/1335268516 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f3c84104ad0 0x7f3c84104f50 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:12:44.661 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:44.661+0000 7f3c89800640 1 --2- 172.21.15.118:0/1335268516 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3c841038e0 0x7f3c84103d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:12:44.661 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:44.661+0000 7f3c89800640 1 -- 172.21.15.118:0/1335268516 >> 172.21.15.118:0/1335268516 conn(0x7f3c840fee50 msgr2=0x7f3c84101290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:12:44.661 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:44.661+0000 7f3c89800640 1 -- 172.21.15.118:0/1335268516 shutdown_connections 2024-03-24T00:12:44.661 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:44.661+0000 7f3c89800640 1 -- 172.21.15.118:0/1335268516 wait complete. 2024-03-24T00:12:44.663 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:44.662+0000 7f3c89800640 1 Processor -- start 2024-03-24T00:12:44.663 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:44.662+0000 7f3c89800640 1 -- start start 2024-03-24T00:12:44.663 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:44.662+0000 7f3c89800640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f3c841038e0 0x7f3c84199bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:12:44.663 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:44.662+0000 7f3c89800640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f3c84104ad0 0x7f3c8419a130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:12:44.663 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:44.662+0000 7f3c89800640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3c84105490 0x7f3c841a11a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:12:44.664 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:44.662+0000 7f3c89800640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3c8410f5b0 con 0x7f3c841038e0 2024-03-24T00:12:44.664 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:44.662+0000 7f3c89800640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3c8410f430 con 0x7f3c84105490 2024-03-24T00:12:44.664 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:44.662+0000 7f3c89800640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3c8410f730 con 0x7f3c84104ad0 2024-03-24T00:12:44.664 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:44.663+0000 7f3c82000640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f3c84104ad0 0x7f3c8419a130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:12:44.664 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:44.663+0000 7f3c82a00640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f3c841038e0 0x7f3c84199bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:12:44.664 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:44.663+0000 7f3c82000640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f3c84104ad0 0x7f3c8419a130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.118:34178/0 (socket says 172.21.15.118:34178) 2024-03-24T00:12:44.664 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:44.663+0000 7f3c82000640 1 -- 172.21.15.118:0/3110879135 learned_addr learned my addr 172.21.15.118:0/3110879135 (peer_addr_for_me v2:172.21.15.118:0/0) 2024-03-24T00:12:44.664 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:44.663+0000 7f3c83400640 1 --2- 172.21.15.118:0/3110879135 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3c84105490 0x7f3c841a11a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:12:44.664 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:44.663+0000 7f3c82000640 1 -- 172.21.15.118:0/3110879135 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3c84105490 msgr2=0x7f3c841a11a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:12:44.664 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:44.663+0000 7f3c82000640 1 --2- 172.21.15.118:0/3110879135 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3c84105490 0x7f3c841a11a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:12:44.664 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:44.663+0000 7f3c82000640 1 -- 172.21.15.118:0/3110879135 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f3c841038e0 msgr2=0x7f3c84199bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:12:44.664 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:44.663+0000 7f3c82000640 1 --2- 172.21.15.118:0/3110879135 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f3c841038e0 0x7f3c84199bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:12:44.664 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:44.663+0000 7f3c82000640 1 -- 172.21.15.118:0/3110879135 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3c841a1810 con 0x7f3c84104ad0 2024-03-24T00:12:44.664 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:44.663+0000 7f3c82000640 1 --2- 172.21.15.118:0/3110879135 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f3c84104ad0 0x7f3c8419a130 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f3c70000c00 tx=0x7f3c7002c650 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:12:44.664 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:44.664+0000 7f3c7b400640 1 -- 172.21.15.118:0/3110879135 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3c70043020 con 0x7f3c84104ad0 2024-03-24T00:12:44.665 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:44.664+0000 7f3c7b400640 1 -- 172.21.15.118:0/3110879135 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f3c700054e0 con 0x7f3c84104ad0 2024-03-24T00:12:44.665 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:44.664+0000 7f3c7b400640 1 -- 172.21.15.118:0/3110879135 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3c70003c40 con 0x7f3c84104ad0 2024-03-24T00:12:44.665 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:44.664+0000 7f3c89800640 1 -- 172.21.15.118:0/3110879135 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3c841a1aa0 con 0x7f3c84104ad0 2024-03-24T00:12:44.665 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:44.664+0000 7f3c89800640 1 -- 172.21.15.118:0/3110879135 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3c841a2040 con 0x7f3c84104ad0 2024-03-24T00:12:44.665 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:44.665+0000 7f3c89800640 1 -- 172.21.15.118:0/3110879135 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3c8410ac00 con 0x7f3c84104ad0 2024-03-24T00:12:44.666 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:44.667+0000 7f3c7b400640 1 -- 172.21.15.118:0/3110879135 <== mon.1 v2:172.21.15.193:3300/0 4 ==== mgrmap(e 16) v1 ==== 137291+0+0 (secure 0 0 0) 0x7f3c70046020 con 0x7f3c84104ad0 2024-03-24T00:12:44.668 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:44.668+0000 7f3c7b400640 1 --2- 172.21.15.118:0/3110879135 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7f3c680a2880 0x7f3c680a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:12:44.668 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:44.669+0000 7f3c7b400640 1 -- 172.21.15.118:0/3110879135 <== mon.1 v2:172.21.15.193:3300/0 5 ==== osd_map(82..82 src has 1..82) v4 ==== 7127+0+0 (secure 0 0 0) 0x7f3c700f0650 con 0x7f3c84104ad0 2024-03-24T00:12:44.668 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:44.669+0000 7f3c82a00640 1 --2- 172.21.15.118:0/3110879135 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7f3c680a2880 0x7f3c680a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:12:44.669 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:44.669+0000 7f3c82a00640 1 --2- 172.21.15.118:0/3110879135 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7f3c680a2880 0x7f3c680a4d40 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f3c84104930 tx=0x7f3c74005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:12:44.672 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:44.672+0000 7f3c7b400640 1 -- 172.21.15.118:0/3110879135 <== mon.1 v2:172.21.15.193:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f3c7004d020 con 0x7f3c84104ad0 2024-03-24T00:12:44.710 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:12:44 smithi193 ceph-mon[36878]: pgmap v301: 1 pgs: 1 active+clean; 577 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-24T00:12:44.710 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:12:44 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T00:12:44.711 INFO:journalctl@ceph.mds.i.smithi193.stdout:Mar 24 00:12:44 smithi193 podman[68073]: 2024-03-24 00:12:44.441025201 +0000 UTC m=+0.403910304 container start 749e1d96d3972c0c41d21f00c3bff52d847c20bf10030537cb51011beac07fa6 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-i, ceph=True, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, GIT_BRANCH=HEAD, org.label-schema.build-date=20240319, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5) 2024-03-24T00:12:44.711 INFO:journalctl@ceph.mds.i.smithi193.stdout:Mar 24 00:12:44 smithi193 ceph-mds[68092]: -- 172.21.15.193:0/2361972972 <== mon.1 v2:172.21.15.193:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x5647ead47a40 con 0x5647ead1b000 2024-03-24T00:12:44.711 INFO:journalctl@ceph.mds.i.smithi193.stdout:Mar 24 00:12:44 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-i[68088]: starting mds.i at 2024-03-24T00:12:44.711 INFO:journalctl@ceph.mds.i.smithi193.stdout:Mar 24 00:12:44 smithi193 bash[68073]: 749e1d96d3972c0c41d21f00c3bff52d847c20bf10030537cb51011beac07fa6 2024-03-24T00:12:44.711 INFO:journalctl@ceph.mds.i.smithi193.stdout:Mar 24 00:12:44 smithi193 systemd[1]: Started Ceph mds.i for d79a5420-e971-11ee-95cd-87774f69a715. 2024-03-24T00:12:44.923 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:44.923+0000 7f3c89800640 1 -- 172.21.15.118:0/3110879135 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f3c8402d070 con 0x7f3c84104ad0 2024-03-24T00:12:44.924 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:44.924+0000 7f3c7b400640 1 -- 172.21.15.118:0/3110879135 <== mon.1 v2:172.21.15.193:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 10 v10) v1 ==== 94+0+7679 (secure 0 0 0) 0x7f3c700bdaf0 con 0x7f3c84104ad0 2024-03-24T00:12:44.924 INFO:teuthology.orchestra.run.smithi118.stdout: 2024-03-24T00:12:44.924 INFO:teuthology.orchestra.run.smithi118.stdout:{"epoch":10,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14385,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.118:6835/3194741373","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6834","nonce":3194741373},{"type":"v1","addr":"172.21.15.118:6835","nonce":3194741373}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":4},{"gid":14397,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.118:6837/1374462956","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6836","nonce":1374462956},{"type":"v1","addr":"172.21.15.118:6837","nonce":1374462956}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":5},{"gid":24439,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.193:6833/1023428412","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6832","nonce":1023428412},{"type":"v1","addr":"172.21.15.193:6833","nonce":1023428412}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":24440,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.189:6833/2740024078","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6832","nonce":2740024078},{"type":"v1","addr":"172.21.15.189:6833","nonce":2740024078}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":24488,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.189:6835/599473205","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6834","nonce":599473205},{"type":"v1","addr":"172.21.15.189:6835","nonce":599473205}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":6},{"gid":24499,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.193:6835/2055310541","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6834","nonce":2055310541},{"type":"v1","addr":"172.21.15.193:6835","nonce":2055310541}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":7},{"gid":24511,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.193:6837/2921520053","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6836","nonce":2921520053},{"type":"v1","addr":"172.21.15.193:6837","nonce":2921520053}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":8},{"gid":24515,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.189:6837/3450187137","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6836","nonce":3450187137},{"type":"v1","addr":"172.21.15.189:6837","nonce":3450187137}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":9},{"gid":24532,"name":"i","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.193:6839/2126842603","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6838","nonce":2126842603},{"type":"v1","addr":"172.21.15.193:6839","nonce":2126842603}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":10}],"filesystems":[]} 2024-03-24T00:12:44.925 INFO:teuthology.orchestra.run.smithi118.stderr:dumped fsmap epoch 10 2024-03-24T00:12:44.927 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:44.927+0000 7f3c89800640 1 -- 172.21.15.118:0/3110879135 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7f3c680a2880 msgr2=0x7f3c680a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:12:44.927 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:44.927+0000 7f3c89800640 1 --2- 172.21.15.118:0/3110879135 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7f3c680a2880 0x7f3c680a4d40 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f3c84104930 tx=0x7f3c74005f90 comp rx=0 tx=0).stop 2024-03-24T00:12:44.927 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:44.927+0000 7f3c89800640 1 -- 172.21.15.118:0/3110879135 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f3c84104ad0 msgr2=0x7f3c8419a130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:12:44.927 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:44.927+0000 7f3c89800640 1 --2- 172.21.15.118:0/3110879135 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f3c84104ad0 0x7f3c8419a130 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7f3c70000c00 tx=0x7f3c7002c650 comp rx=0 tx=0).stop 2024-03-24T00:12:44.927 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:44.927+0000 7f3c89800640 1 -- 172.21.15.118:0/3110879135 shutdown_connections 2024-03-24T00:12:44.927 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:44.927+0000 7f3c89800640 1 --2- 172.21.15.118:0/3110879135 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7f3c680a2880 0x7f3c680a4d40 unknown :-1 s=CLOSED pgs=147 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:12:44.927 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:44.927+0000 7f3c89800640 1 --2- 172.21.15.118:0/3110879135 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3c84105490 0x7f3c841a11a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:12:44.928 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:44.927+0000 7f3c89800640 1 --2- 172.21.15.118:0/3110879135 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f3c84104ad0 0x7f3c8419a130 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:12:44.928 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:44.927+0000 7f3c89800640 1 --2- 172.21.15.118:0/3110879135 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f3c841038e0 0x7f3c84199bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:12:44.928 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:44.927+0000 7f3c89800640 1 -- 172.21.15.118:0/3110879135 >> 172.21.15.118:0/3110879135 conn(0x7f3c840fee50 msgr2=0x7f3c84107dd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:12:44.928 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:44.927+0000 7f3c89800640 1 -- 172.21.15.118:0/3110879135 shutdown_connections 2024-03-24T00:12:44.928 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:44.928+0000 7f3c89800640 1 -- 172.21.15.118:0/3110879135 wait complete. 2024-03-24T00:12:44.942 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:12:44 smithi118 ceph-mon[31535]: pgmap v301: 1 pgs: 1 active+clean; 577 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-24T00:12:44.942 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:12:44 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T00:12:44.953 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:12:44 smithi189 ceph-mon[36187]: pgmap v301: 1 pgs: 1 active+clean; 577 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-24T00:12:44.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:12:44 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T00:12:45.868 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:12:45 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T00:12:45.868 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:12:45 smithi193 ceph-mon[36878]: mds.? [v2:172.21.15.193:6838/2126842603,v1:172.21.15.193:6839/2126842603] up:boot 2024-03-24T00:12:45.868 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:12:45 smithi193 ceph-mon[36878]: fsmap 9 up:standby 2024-03-24T00:12:45.868 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:12:45 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2024-03-24T00:12:45.868 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:12:45 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T00:12:45.868 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:12:45 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.g", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-24T00:12:45.868 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:12:45 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.g", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-24T00:12:45.868 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:12:45 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T00:12:45.868 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:12:45 smithi193 ceph-mon[36878]: Deploying daemon mds.g on smithi118 2024-03-24T00:12:45.869 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:12:45 smithi193 ceph-mon[36878]: from='client.? 172.21.15.118:0/3110879135' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-24T00:12:45.889 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:12:45 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T00:12:45.889 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:12:45 smithi118 ceph-mon[31535]: mds.? [v2:172.21.15.193:6838/2126842603,v1:172.21.15.193:6839/2126842603] up:boot 2024-03-24T00:12:45.889 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:12:45 smithi118 ceph-mon[31535]: fsmap 9 up:standby 2024-03-24T00:12:45.889 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:12:45 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2024-03-24T00:12:45.889 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:12:45 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T00:12:45.889 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:12:45 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.g", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-24T00:12:45.890 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:12:45 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.g", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-24T00:12:45.890 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:12:45 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T00:12:45.890 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:12:45 smithi118 ceph-mon[31535]: Deploying daemon mds.g on smithi118 2024-03-24T00:12:45.890 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:12:45 smithi118 ceph-mon[31535]: from='client.? 172.21.15.118:0/3110879135' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-24T00:12:45.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:12:45 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T00:12:45.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:12:45 smithi189 ceph-mon[36187]: mds.? [v2:172.21.15.193:6838/2126842603,v1:172.21.15.193:6839/2126842603] up:boot 2024-03-24T00:12:45.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:12:45 smithi189 ceph-mon[36187]: fsmap 9 up:standby 2024-03-24T00:12:45.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:12:45 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2024-03-24T00:12:45.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:12:45 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T00:12:45.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:12:45 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.g", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-24T00:12:45.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:12:45 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.g", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-24T00:12:45.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:12:45 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T00:12:45.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:12:45 smithi189 ceph-mon[36187]: Deploying daemon mds.g on smithi118 2024-03-24T00:12:45.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:12:45 smithi189 ceph-mon[36187]: from='client.? 172.21.15.118:0/3110879135' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-24T00:12:46.695 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:12:46 smithi118 ceph-mon[31535]: pgmap v302: 1 pgs: 1 active+clean; 577 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-24T00:12:46.946 DEBUG:teuthology.orchestra.run.smithi118:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-24T00:12:46.953 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:12:46 smithi189 ceph-mon[36187]: pgmap v302: 1 pgs: 1 active+clean; 577 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-24T00:12:46.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:12:46 smithi193 ceph-mon[36878]: pgmap v302: 1 pgs: 1 active+clean; 577 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-24T00:12:47.365 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:47.365+0000 7fe48b200640 1 -- 172.21.15.118:0/2504826239 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe484109000 msgr2=0x7fe48410f9e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:12:47.365 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:47.365+0000 7fe48b200640 1 --2- 172.21.15.118:0/2504826239 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe484109000 0x7fe48410f9e0 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7fe46c009860 tx=0x7fe46c02bb70 comp rx=0 tx=0).stop 2024-03-24T00:12:47.365 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:47.365+0000 7fe48b200640 1 -- 172.21.15.118:0/2504826239 shutdown_connections 2024-03-24T00:12:47.365 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:47.365+0000 7fe48b200640 1 --2- 172.21.15.118:0/2504826239 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fe48410ff20 0x7fe484112310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:12:47.365 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:47.365+0000 7fe48b200640 1 --2- 172.21.15.118:0/2504826239 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe484109000 0x7fe48410f9e0 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:12:47.365 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:47.365+0000 7fe48b200640 1 --2- 172.21.15.118:0/2504826239 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fe4841086a0 0x7fe484108ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:12:47.365 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:47.365+0000 7fe48b200640 1 -- 172.21.15.118:0/2504826239 >> 172.21.15.118:0/2504826239 conn(0x7fe48406e510 msgr2=0x7fe484070950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:12:47.365 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:47.365+0000 7fe48b200640 1 -- 172.21.15.118:0/2504826239 shutdown_connections 2024-03-24T00:12:47.365 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:47.365+0000 7fe48b200640 1 -- 172.21.15.118:0/2504826239 wait complete. 2024-03-24T00:12:47.367 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:47.366+0000 7fe48b200640 1 Processor -- start 2024-03-24T00:12:47.367 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:47.366+0000 7fe48b200640 1 -- start start 2024-03-24T00:12:47.367 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:47.366+0000 7fe48b200640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fe4841086a0 0x7fe48410b330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:12:47.367 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:47.366+0000 7fe48b200640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe484109000 0x7fe484109970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:12:47.367 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:47.366+0000 7fe48b200640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fe48410ff20 0x7fe484109eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:12:47.367 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:47.366+0000 7fe48b200640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe484115b20 con 0x7fe4841086a0 2024-03-24T00:12:47.367 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:47.366+0000 7fe48b200640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe4841159a0 con 0x7fe484109000 2024-03-24T00:12:47.367 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:47.366+0000 7fe48b200640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe484115ca0 con 0x7fe48410ff20 2024-03-24T00:12:47.367 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:47.367+0000 7fe483e00640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fe4841086a0 0x7fe48410b330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:12:47.367 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:47.367+0000 7fe483e00640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fe4841086a0 0x7fe48410b330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.118:3300/0 says I am v2:172.21.15.118:33566/0 (socket says 172.21.15.118:33566) 2024-03-24T00:12:47.367 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:47.367+0000 7fe483e00640 1 -- 172.21.15.118:0/2304101894 learned_addr learned my addr 172.21.15.118:0/2304101894 (peer_addr_for_me v2:172.21.15.118:0/0) 2024-03-24T00:12:47.367 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:47.367+0000 7fe483400640 1 --2- 172.21.15.118:0/2304101894 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe484109000 0x7fe484109970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:12:47.368 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:47.367+0000 7fe489200640 1 --2- 172.21.15.118:0/2304101894 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fe48410ff20 0x7fe484109eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:12:47.368 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:47.367+0000 7fe483e00640 1 -- 172.21.15.118:0/2304101894 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fe48410ff20 msgr2=0x7fe484109eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:12:47.368 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:47.367+0000 7fe483e00640 1 --2- 172.21.15.118:0/2304101894 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fe48410ff20 0x7fe484109eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:12:47.368 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:47.367+0000 7fe483e00640 1 -- 172.21.15.118:0/2304101894 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe484109000 msgr2=0x7fe484109970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:12:47.368 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:47.367+0000 7fe483e00640 1 --2- 172.21.15.118:0/2304101894 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe484109000 0x7fe484109970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:12:47.368 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:47.367+0000 7fe483e00640 1 -- 172.21.15.118:0/2304101894 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe48410a740 con 0x7fe4841086a0 2024-03-24T00:12:47.368 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:47.367+0000 7fe483400640 1 --2- 172.21.15.118:0/2304101894 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe484109000 0x7fe484109970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T00:12:47.369 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:47.368+0000 7fe483e00640 1 --2- 172.21.15.118:0/2304101894 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fe4841086a0 0x7fe48410b330 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7fe47c009d20 tx=0x7fe47c00c6a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:12:47.369 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:47.368+0000 7fe480c00640 1 -- 172.21.15.118:0/2304101894 <== mon.0 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe47c010630 con 0x7fe4841086a0 2024-03-24T00:12:47.369 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:47.368+0000 7fe48b200640 1 -- 172.21.15.118:0/2304101894 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe4841aa340 con 0x7fe4841086a0 2024-03-24T00:12:47.369 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:47.368+0000 7fe480c00640 1 -- 172.21.15.118:0/2304101894 <== mon.0 v2:172.21.15.118:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fe47c010cb0 con 0x7fe4841086a0 2024-03-24T00:12:47.369 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:47.368+0000 7fe480c00640 1 -- 172.21.15.118:0/2304101894 <== mon.0 v2:172.21.15.118:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe47c018de0 con 0x7fe4841086a0 2024-03-24T00:12:47.369 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:47.368+0000 7fe48b200640 1 -- 172.21.15.118:0/2304101894 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe4841aa8b0 con 0x7fe4841086a0 2024-03-24T00:12:47.369 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:47.369+0000 7fe48b200640 1 -- 172.21.15.118:0/2304101894 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe48410f0e0 con 0x7fe4841086a0 2024-03-24T00:12:47.371 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:47.371+0000 7fe480c00640 1 -- 172.21.15.118:0/2304101894 <== mon.0 v2:172.21.15.118:3300/0 4 ==== mgrmap(e 16) v1 ==== 137291+0+0 (secure 0 0 0) 0x7fe47c004b40 con 0x7fe4841086a0 2024-03-24T00:12:47.372 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:47.373+0000 7fe480c00640 1 --2- 172.21.15.118:0/2304101894 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7fe4680a2800 0x7fe4680a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:12:47.372 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:47.373+0000 7fe480c00640 1 -- 172.21.15.118:0/2304101894 <== mon.0 v2:172.21.15.118:3300/0 5 ==== osd_map(82..82 src has 1..82) v4 ==== 7127+0+0 (secure 0 0 0) 0x7fe47c0cf190 con 0x7fe4841086a0 2024-03-24T00:12:47.372 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:47.373+0000 7fe483400640 1 --2- 172.21.15.118:0/2304101894 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7fe4680a2800 0x7fe4680a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:12:47.373 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:47.374+0000 7fe483400640 1 --2- 172.21.15.118:0/2304101894 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7fe4680a2800 0x7fe4680a4cc0 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7fe46c009830 tx=0x7fe46c02e040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:12:47.380 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:47.380+0000 7fe480c00640 1 -- 172.21.15.118:0/2304101894 <== mon.0 v2:172.21.15.118:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7fe47c026020 con 0x7fe4841086a0 2024-03-24T00:12:47.643 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:47.643+0000 7fe48b200640 1 -- 172.21.15.118:0/2304101894 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fe48402d070 con 0x7fe4841086a0 2024-03-24T00:12:47.643 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:47.644+0000 7fe480c00640 1 -- 172.21.15.118:0/2304101894 <== mon.0 v2:172.21.15.118:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 10 v10) v1 ==== 94+0+7679 (secure 0 0 0) 0x7fe47c09c6d0 con 0x7fe4841086a0 2024-03-24T00:12:47.644 INFO:teuthology.orchestra.run.smithi118.stdout: 2024-03-24T00:12:47.644 INFO:teuthology.orchestra.run.smithi118.stdout:{"epoch":10,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14385,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.118:6835/3194741373","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6834","nonce":3194741373},{"type":"v1","addr":"172.21.15.118:6835","nonce":3194741373}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":4},{"gid":14397,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.118:6837/1374462956","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6836","nonce":1374462956},{"type":"v1","addr":"172.21.15.118:6837","nonce":1374462956}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":5},{"gid":24439,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.193:6833/1023428412","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6832","nonce":1023428412},{"type":"v1","addr":"172.21.15.193:6833","nonce":1023428412}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":24440,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.189:6833/2740024078","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6832","nonce":2740024078},{"type":"v1","addr":"172.21.15.189:6833","nonce":2740024078}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":24488,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.189:6835/599473205","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6834","nonce":599473205},{"type":"v1","addr":"172.21.15.189:6835","nonce":599473205}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":6},{"gid":24499,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.193:6835/2055310541","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6834","nonce":2055310541},{"type":"v1","addr":"172.21.15.193:6835","nonce":2055310541}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":7},{"gid":24511,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.193:6837/2921520053","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6836","nonce":2921520053},{"type":"v1","addr":"172.21.15.193:6837","nonce":2921520053}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":8},{"gid":24515,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.189:6837/3450187137","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6836","nonce":3450187137},{"type":"v1","addr":"172.21.15.189:6837","nonce":3450187137}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":9},{"gid":24532,"name":"i","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.193:6839/2126842603","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6838","nonce":2126842603},{"type":"v1","addr":"172.21.15.193:6839","nonce":2126842603}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":10}],"filesystems":[]} 2024-03-24T00:12:47.644 INFO:teuthology.orchestra.run.smithi118.stderr:dumped fsmap epoch 10 2024-03-24T00:12:47.646 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:47.647+0000 7fe48b200640 1 -- 172.21.15.118:0/2304101894 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7fe4680a2800 msgr2=0x7fe4680a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:12:47.646 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:47.647+0000 7fe48b200640 1 --2- 172.21.15.118:0/2304101894 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7fe4680a2800 0x7fe4680a4cc0 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7fe46c009830 tx=0x7fe46c02e040 comp rx=0 tx=0).stop 2024-03-24T00:12:47.647 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:47.647+0000 7fe48b200640 1 -- 172.21.15.118:0/2304101894 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fe4841086a0 msgr2=0x7fe48410b330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:12:47.647 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:47.647+0000 7fe48b200640 1 --2- 172.21.15.118:0/2304101894 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fe4841086a0 0x7fe48410b330 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7fe47c009d20 tx=0x7fe47c00c6a0 comp rx=0 tx=0).stop 2024-03-24T00:12:47.647 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:47.647+0000 7fe48b200640 1 -- 172.21.15.118:0/2304101894 shutdown_connections 2024-03-24T00:12:47.647 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:47.647+0000 7fe48b200640 1 --2- 172.21.15.118:0/2304101894 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7fe4680a2800 0x7fe4680a4cc0 unknown :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:12:47.647 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:47.647+0000 7fe48b200640 1 --2- 172.21.15.118:0/2304101894 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fe48410ff20 0x7fe484109eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:12:47.647 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:47.647+0000 7fe48b200640 1 --2- 172.21.15.118:0/2304101894 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe484109000 0x7fe484109970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:12:47.647 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:47.647+0000 7fe48b200640 1 --2- 172.21.15.118:0/2304101894 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fe4841086a0 0x7fe48410b330 unknown :-1 s=CLOSED pgs=147 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:12:47.647 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:47.647+0000 7fe48b200640 1 -- 172.21.15.118:0/2304101894 >> 172.21.15.118:0/2304101894 conn(0x7fe48406e510 msgr2=0x7fe4840702a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:12:47.647 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:47.647+0000 7fe48b200640 1 -- 172.21.15.118:0/2304101894 shutdown_connections 2024-03-24T00:12:47.647 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:47.647+0000 7fe48b200640 1 -- 172.21.15.118:0/2304101894 wait complete. 2024-03-24T00:12:47.881 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:12:47 smithi118 ceph-mon[31535]: from='client.? 172.21.15.118:0/2304101894' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-24T00:12:47.953 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:12:47 smithi189 ceph-mon[36187]: from='client.? 172.21.15.118:0/2304101894' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-24T00:12:47.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:12:47 smithi193 ceph-mon[36878]: from='client.? 172.21.15.118:0/2304101894' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-24T00:12:48.175 INFO:journalctl@ceph.mds.g.smithi118.stdout:Mar 24 00:12:47 smithi118 podman[73595]: 2024-03-24 00:12:47.990682559 +0000 UTC m=+0.185473944 container create 2943a8e74123a676960a943dae30cd5c28d14b4754589981df7f4d82ed679c74 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-g, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240319, CEPH_POINT_RELEASE=, GIT_CLEAN=True, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, ceph=True) 2024-03-24T00:12:48.661 INFO:journalctl@ceph.mds.g.smithi118.stdout:Mar 24 00:12:48 smithi118 podman[73595]: 2024-03-24 00:12:48.286585037 +0000 UTC m=+0.481376432 container init 2943a8e74123a676960a943dae30cd5c28d14b4754589981df7f4d82ed679c74 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-g, GIT_CLEAN=True, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, GIT_BRANCH=HEAD, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.build-date=20240319, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=) 2024-03-24T00:12:48.661 INFO:journalctl@ceph.mds.g.smithi118.stdout:Mar 24 00:12:48 smithi118 podman[73595]: 2024-03-24 00:12:48.293945782 +0000 UTC m=+0.488737187 container start 2943a8e74123a676960a943dae30cd5c28d14b4754589981df7f4d82ed679c74 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-g, CEPH_POINT_RELEASE=, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, org.label-schema.build-date=20240319, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, io.buildah.version=1.33.5, ceph=True, org.label-schema.schema-version=1.0, GIT_CLEAN=True) 2024-03-24T00:12:48.661 INFO:journalctl@ceph.mds.g.smithi118.stdout:Mar 24 00:12:48 smithi118 ceph-mds[73621]: -- 172.21.15.118:0/1612014924 <== mon.0 v2:172.21.15.118:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x561bcc94fa40 con 0x561bcc922c00 2024-03-24T00:12:48.662 INFO:journalctl@ceph.mds.g.smithi118.stdout:Mar 24 00:12:48 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-g[73614]: starting mds.g at 2024-03-24T00:12:48.662 INFO:journalctl@ceph.mds.g.smithi118.stdout:Mar 24 00:12:48 smithi118 bash[73595]: 2943a8e74123a676960a943dae30cd5c28d14b4754589981df7f4d82ed679c74 2024-03-24T00:12:48.662 INFO:journalctl@ceph.mds.g.smithi118.stdout:Mar 24 00:12:48 smithi118 systemd[1]: Started Ceph mds.g for d79a5420-e971-11ee-95cd-87774f69a715. 2024-03-24T00:12:48.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:12:48 smithi118 ceph-mon[31535]: pgmap v303: 1 pgs: 1 active+clean; 577 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-24T00:12:48.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:12:48 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T00:12:48.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:12:48 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T00:12:48.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:12:48 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T00:12:48.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:12:48 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.a", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-24T00:12:48.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:12:48 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.a", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-24T00:12:48.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:12:48 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T00:12:48.953 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:12:48 smithi189 ceph-mon[36187]: pgmap v303: 1 pgs: 1 active+clean; 577 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-24T00:12:48.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:12:48 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T00:12:48.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:12:48 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T00:12:48.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:12:48 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T00:12:48.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:12:48 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.a", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-24T00:12:48.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:12:48 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.a", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-24T00:12:48.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:12:48 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T00:12:48.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:12:48 smithi193 ceph-mon[36878]: pgmap v303: 1 pgs: 1 active+clean; 577 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-24T00:12:48.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:12:48 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T00:12:48.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:12:48 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T00:12:48.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:12:48 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T00:12:48.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:12:48 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.a", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-24T00:12:48.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:12:48 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.a", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-24T00:12:48.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:12:48 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T00:12:49.667 DEBUG:teuthology.orchestra.run.smithi118:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-24T00:12:49.716 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:12:49 smithi118 ceph-mon[31535]: Deploying daemon mds.a on smithi118 2024-03-24T00:12:49.716 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:12:49 smithi118 ceph-mon[31535]: mds.? [v2:172.21.15.118:6838/3285788503,v1:172.21.15.118:6839/3285788503] up:boot 2024-03-24T00:12:49.716 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:12:49 smithi118 ceph-mon[31535]: fsmap 10 up:standby 2024-03-24T00:12:49.716 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:12:49 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2024-03-24T00:12:49.872 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:49.872+0000 7ff8a5a00640 1 -- 172.21.15.118:0/3896983422 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7ff8a0109890 msgr2=0x7ff8a0110120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:12:49.872 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:49.872+0000 7ff8a5a00640 1 --2- 172.21.15.118:0/3896983422 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7ff8a0109890 0x7ff8a0110120 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7ff898009900 tx=0x7ff89802bc30 comp rx=0 tx=0).stop 2024-03-24T00:12:49.872 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:49.872+0000 7ff8a5a00640 1 -- 172.21.15.118:0/3896983422 shutdown_connections 2024-03-24T00:12:49.872 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:49.872+0000 7ff8a5a00640 1 --2- 172.21.15.118:0/3896983422 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7ff8a0109890 0x7ff8a0110120 unknown :-1 s=CLOSED pgs=149 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:12:49.872 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:49.872+0000 7ff8a5a00640 1 --2- 172.21.15.118:0/3896983422 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7ff8a0108ed0 0x7ff8a0109350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:12:49.872 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:49.872+0000 7ff8a5a00640 1 --2- 172.21.15.118:0/3896983422 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff8a0107ce0 0x7ff8a0108100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:12:49.872 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:49.872+0000 7ff8a5a00640 1 -- 172.21.15.118:0/3896983422 >> 172.21.15.118:0/3896983422 conn(0x7ff8a0076cb0 msgr2=0x7ff8a00790d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:12:49.872 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:49.872+0000 7ff8a5a00640 1 -- 172.21.15.118:0/3896983422 shutdown_connections 2024-03-24T00:12:49.872 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:49.872+0000 7ff8a5a00640 1 -- 172.21.15.118:0/3896983422 wait complete. 2024-03-24T00:12:49.874 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:49.873+0000 7ff8a5a00640 1 Processor -- start 2024-03-24T00:12:49.875 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:49.873+0000 7ff8a5a00640 1 -- start start 2024-03-24T00:12:49.875 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:49.873+0000 7ff8a5a00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7ff8a0107ce0 0x7ff8a019df70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:12:49.875 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:49.873+0000 7ff8a5a00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff8a0108ed0 0x7ff8a019e4b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:12:49.875 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:49.873+0000 7ff8a5a00640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7ff8a0109890 0x7ff8a01a5520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:12:49.875 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:49.873+0000 7ff8a5a00640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff8a01139b0 con 0x7ff8a0109890 2024-03-24T00:12:49.875 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:49.873+0000 7ff8a5a00640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff8a0113830 con 0x7ff8a0108ed0 2024-03-24T00:12:49.875 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:49.873+0000 7ff8a5a00640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff8a0113b30 con 0x7ff8a0107ce0 2024-03-24T00:12:49.875 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:49.874+0000 7ff89f400640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7ff8a0109890 0x7ff8a01a5520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:12:49.875 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:49.874+0000 7ff89e000640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff8a0108ed0 0x7ff8a019e4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:12:49.875 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:49.874+0000 7ff89f400640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7ff8a0109890 0x7ff8a01a5520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.118:3300/0 says I am v2:172.21.15.118:33592/0 (socket says 172.21.15.118:33592) 2024-03-24T00:12:49.875 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:49.874+0000 7ff89e000640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff8a0108ed0 0x7ff8a019e4b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.118:44018/0 (socket says 172.21.15.118:44018) 2024-03-24T00:12:49.875 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:49.874+0000 7ff89f400640 1 -- 172.21.15.118:0/3940489268 learned_addr learned my addr 172.21.15.118:0/3940489268 (peer_addr_for_me v2:172.21.15.118:0/0) 2024-03-24T00:12:49.875 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:49.874+0000 7ff89ea00640 1 --2- 172.21.15.118:0/3940489268 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7ff8a0107ce0 0x7ff8a019df70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:12:49.876 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:49.874+0000 7ff89e000640 1 -- 172.21.15.118:0/3940489268 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7ff8a0107ce0 msgr2=0x7ff8a019df70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:12:49.876 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:49.874+0000 7ff89e000640 1 --2- 172.21.15.118:0/3940489268 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7ff8a0107ce0 0x7ff8a019df70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:12:49.876 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:49.874+0000 7ff89e000640 1 -- 172.21.15.118:0/3940489268 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7ff8a0109890 msgr2=0x7ff8a01a5520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:12:49.876 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:49.874+0000 7ff89e000640 1 --2- 172.21.15.118:0/3940489268 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7ff8a0109890 0x7ff8a01a5520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:12:49.876 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:49.874+0000 7ff89e000640 1 -- 172.21.15.118:0/3940489268 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff8a01a5b90 con 0x7ff8a0108ed0 2024-03-24T00:12:49.876 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:49.874+0000 7ff89ea00640 1 --2- 172.21.15.118:0/3940489268 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7ff8a0107ce0 0x7ff8a019df70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T00:12:49.876 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:49.874+0000 7ff89f400640 1 --2- 172.21.15.118:0/3940489268 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7ff8a0109890 0x7ff8a01a5520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T00:12:49.876 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:49.874+0000 7ff89e000640 1 --2- 172.21.15.118:0/3940489268 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff8a0108ed0 0x7ff8a019e4b0 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7ff890009fd0 tx=0x7ff89000ef90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:12:49.876 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:49.875+0000 7ff897400640 1 -- 172.21.15.118:0/3940489268 <== mon.2 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff890004b20 con 0x7ff8a0108ed0 2024-03-24T00:12:49.876 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:49.875+0000 7ff897400640 1 -- 172.21.15.118:0/3940489268 <== mon.2 v2:172.21.15.189:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7ff890004cc0 con 0x7ff8a0108ed0 2024-03-24T00:12:49.876 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:49.875+0000 7ff897400640 1 -- 172.21.15.118:0/3940489268 <== mon.2 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff8900107b0 con 0x7ff8a0108ed0 2024-03-24T00:12:49.876 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:49.875+0000 7ff8a5a00640 1 -- 172.21.15.118:0/3940489268 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff8a01a5e80 con 0x7ff8a0108ed0 2024-03-24T00:12:49.876 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:49.875+0000 7ff8a5a00640 1 -- 172.21.15.118:0/3940489268 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff8a01a62f0 con 0x7ff8a0108ed0 2024-03-24T00:12:49.877 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:49.876+0000 7ff8a5a00640 1 -- 172.21.15.118:0/3940489268 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff8a010ef30 con 0x7ff8a0108ed0 2024-03-24T00:12:49.879 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:49.879+0000 7ff897400640 1 -- 172.21.15.118:0/3940489268 <== mon.2 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 16) v1 ==== 137291+0+0 (secure 0 0 0) 0x7ff890005020 con 0x7ff8a0108ed0 2024-03-24T00:12:49.880 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:49.881+0000 7ff897400640 1 --2- 172.21.15.118:0/3940489268 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7ff8840a2880 0x7ff8840a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:12:49.880 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:49.881+0000 7ff897400640 1 -- 172.21.15.118:0/3940489268 <== mon.2 v2:172.21.15.189:3300/0 5 ==== osd_map(82..82 src has 1..82) v4 ==== 7127+0+0 (secure 0 0 0) 0x7ff8900cede0 con 0x7ff8a0108ed0 2024-03-24T00:12:49.880 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:49.881+0000 7ff89ea00640 1 --2- 172.21.15.118:0/3940489268 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7ff8840a2880 0x7ff8840a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:12:49.881 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:49.881+0000 7ff89ea00640 1 --2- 172.21.15.118:0/3940489268 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7ff8840a2880 0x7ff8840a4d40 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7ff8a0108d30 tx=0x7ff888005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:12:49.884 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:49.885+0000 7ff897400640 1 -- 172.21.15.118:0/3940489268 <== mon.2 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7ff89009c280 con 0x7ff8a0108ed0 2024-03-24T00:12:49.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:12:49 smithi189 ceph-mon[36187]: Deploying daemon mds.a on smithi118 2024-03-24T00:12:49.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:12:49 smithi189 ceph-mon[36187]: mds.? [v2:172.21.15.118:6838/3285788503,v1:172.21.15.118:6839/3285788503] up:boot 2024-03-24T00:12:49.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:12:49 smithi189 ceph-mon[36187]: fsmap 10 up:standby 2024-03-24T00:12:49.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:12:49 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2024-03-24T00:12:49.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:12:49 smithi193 ceph-mon[36878]: Deploying daemon mds.a on smithi118 2024-03-24T00:12:49.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:12:49 smithi193 ceph-mon[36878]: mds.? [v2:172.21.15.118:6838/3285788503,v1:172.21.15.118:6839/3285788503] up:boot 2024-03-24T00:12:49.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:12:49 smithi193 ceph-mon[36878]: fsmap 10 up:standby 2024-03-24T00:12:49.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:12:49 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2024-03-24T00:12:50.123 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:50.123+0000 7ff8a5a00640 1 -- 172.21.15.118:0/3940489268 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7ff8a002d070 con 0x7ff8a0108ed0 2024-03-24T00:12:50.125 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:50.125+0000 7ff897400640 1 -- 172.21.15.118:0/3940489268 <== mon.2 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 11 v11) v1 ==== 94+0+8474 (secure 0 0 0) 0x7ff89009bb80 con 0x7ff8a0108ed0 2024-03-24T00:12:50.125 INFO:teuthology.orchestra.run.smithi118.stdout: 2024-03-24T00:12:50.125 INFO:teuthology.orchestra.run.smithi118.stdout:{"epoch":11,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14385,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.118:6835/3194741373","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6834","nonce":3194741373},{"type":"v1","addr":"172.21.15.118:6835","nonce":3194741373}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":4},{"gid":14397,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.118:6837/1374462956","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6836","nonce":1374462956},{"type":"v1","addr":"172.21.15.118:6837","nonce":1374462956}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":5},{"gid":24439,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.193:6833/1023428412","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6832","nonce":1023428412},{"type":"v1","addr":"172.21.15.193:6833","nonce":1023428412}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":24440,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.189:6833/2740024078","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6832","nonce":2740024078},{"type":"v1","addr":"172.21.15.189:6833","nonce":2740024078}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":24488,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.189:6835/599473205","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6834","nonce":599473205},{"type":"v1","addr":"172.21.15.189:6835","nonce":599473205}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":6},{"gid":24499,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.193:6835/2055310541","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6834","nonce":2055310541},{"type":"v1","addr":"172.21.15.193:6835","nonce":2055310541}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":7},{"gid":24511,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.193:6837/2921520053","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6836","nonce":2921520053},{"type":"v1","addr":"172.21.15.193:6837","nonce":2921520053}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":8},{"gid":24515,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.189:6837/3450187137","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6836","nonce":3450187137},{"type":"v1","addr":"172.21.15.189:6837","nonce":3450187137}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":9},{"gid":24532,"name":"i","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.193:6839/2126842603","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6838","nonce":2126842603},{"type":"v1","addr":"172.21.15.193:6839","nonce":2126842603}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":10},{"gid":24550,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.118:6839/3285788503","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6838","nonce":3285788503},{"type":"v1","addr":"172.21.15.118:6839","nonce":3285788503}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":11}],"filesystems":[]} 2024-03-24T00:12:50.125 INFO:teuthology.orchestra.run.smithi118.stderr:dumped fsmap epoch 11 2024-03-24T00:12:50.128 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:50.128+0000 7ff8a5a00640 1 -- 172.21.15.118:0/3940489268 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7ff8840a2880 msgr2=0x7ff8840a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:12:50.128 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:50.128+0000 7ff8a5a00640 1 --2- 172.21.15.118:0/3940489268 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7ff8840a2880 0x7ff8840a4d40 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7ff8a0108d30 tx=0x7ff888005f90 comp rx=0 tx=0).stop 2024-03-24T00:12:50.128 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:50.128+0000 7ff8a5a00640 1 -- 172.21.15.118:0/3940489268 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff8a0108ed0 msgr2=0x7ff8a019e4b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:12:50.128 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:50.128+0000 7ff8a5a00640 1 --2- 172.21.15.118:0/3940489268 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff8a0108ed0 0x7ff8a019e4b0 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7ff890009fd0 tx=0x7ff89000ef90 comp rx=0 tx=0).stop 2024-03-24T00:12:50.128 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:50.128+0000 7ff8a5a00640 1 -- 172.21.15.118:0/3940489268 shutdown_connections 2024-03-24T00:12:50.128 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:50.128+0000 7ff8a5a00640 1 --2- 172.21.15.118:0/3940489268 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7ff8840a2880 0x7ff8840a4d40 unknown :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:12:50.128 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:50.128+0000 7ff8a5a00640 1 --2- 172.21.15.118:0/3940489268 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7ff8a0109890 0x7ff8a01a5520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:12:50.128 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:50.128+0000 7ff8a5a00640 1 --2- 172.21.15.118:0/3940489268 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff8a0108ed0 0x7ff8a019e4b0 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:12:50.129 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:50.128+0000 7ff8a5a00640 1 --2- 172.21.15.118:0/3940489268 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7ff8a0107ce0 0x7ff8a019df70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:12:50.129 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:50.128+0000 7ff8a5a00640 1 -- 172.21.15.118:0/3940489268 >> 172.21.15.118:0/3940489268 conn(0x7ff8a0076cb0 msgr2=0x7ff8a010c100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:12:50.129 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:50.128+0000 7ff8a5a00640 1 -- 172.21.15.118:0/3940489268 shutdown_connections 2024-03-24T00:12:50.129 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:50.128+0000 7ff8a5a00640 1 -- 172.21.15.118:0/3940489268 wait complete. 2024-03-24T00:12:50.810 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:12:50 smithi118 ceph-mon[31535]: pgmap v304: 1 pgs: 1 active+clean; 577 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-24T00:12:50.810 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:12:50 smithi118 ceph-mon[31535]: from='client.? 172.21.15.118:0/3940489268' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-24T00:12:50.953 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:12:50 smithi189 ceph-mon[36187]: pgmap v304: 1 pgs: 1 active+clean; 577 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-24T00:12:50.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:12:50 smithi189 ceph-mon[36187]: from='client.? 172.21.15.118:0/3940489268' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-24T00:12:50.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:12:50 smithi193 ceph-mon[36878]: pgmap v304: 1 pgs: 1 active+clean; 577 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-24T00:12:50.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:12:50 smithi193 ceph-mon[36878]: from='client.? 172.21.15.118:0/3940489268' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-24T00:12:51.632 INFO:journalctl@ceph.mds.a.smithi118.stdout:Mar 24 00:12:51 smithi118 systemd[1]: Starting Ceph mds.a for d79a5420-e971-11ee-95cd-87774f69a715... 2024-03-24T00:12:52.147 DEBUG:teuthology.orchestra.run.smithi118:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-24T00:12:52.333 INFO:journalctl@ceph.mds.a.smithi118.stdout:Mar 24 00:12:51 smithi118 podman[73949]: 2024-03-24 00:12:51.894972349 +0000 UTC m=+0.035623733 image pull c664afe56bb01db4f9afa9532c20598e201e664f2e9ba2be50db0d52674a26d2 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc 2024-03-24T00:12:52.333 INFO:journalctl@ceph.mds.a.smithi118.stdout:Mar 24 00:12:52 smithi118 podman[73949]: 2024-03-24 00:12:52.021891341 +0000 UTC m=+0.162542761 container create 8b122c08c5eea799f91ee96444ffb9f22e8135af2a327414fac0fa3f34694326 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-a, org.label-schema.build-date=20240319, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, GIT_CLEAN=True, ceph=True, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, maintainer=Guillaume Abrioux ) 2024-03-24T00:12:52.334 INFO:journalctl@ceph.mds.a.smithi118.stdout:Mar 24 00:12:52 smithi118 podman[73949]: 2024-03-24 00:12:52.33258843 +0000 UTC m=+0.473239856 container init 8b122c08c5eea799f91ee96444ffb9f22e8135af2a327414fac0fa3f34694326 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-a, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, GIT_CLEAN=True, ceph=True, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.build-date=20240319, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c) 2024-03-24T00:12:52.359 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:52.359+0000 7f2ae8000640 1 -- 172.21.15.118:0/1109866265 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f2ae01038e0 msgr2=0x7f2ae0103d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:12:52.359 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:52.359+0000 7f2ae8000640 1 --2- 172.21.15.118:0/1109866265 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f2ae01038e0 0x7f2ae0103d00 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f2ad4009900 tx=0x7f2ad402bc30 comp rx=0 tx=0).stop 2024-03-24T00:12:52.359 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:52.359+0000 7f2ae8000640 1 -- 172.21.15.118:0/1109866265 shutdown_connections 2024-03-24T00:12:52.359 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:52.359+0000 7f2ae8000640 1 --2- 172.21.15.118:0/1109866265 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f2ae0105490 0x7f2ae010bd20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:12:52.359 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:52.359+0000 7f2ae8000640 1 --2- 172.21.15.118:0/1109866265 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2ae0104ad0 0x7f2ae0104f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:12:52.359 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:52.359+0000 7f2ae8000640 1 --2- 172.21.15.118:0/1109866265 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f2ae01038e0 0x7f2ae0103d00 unknown :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:12:52.359 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:52.359+0000 7f2ae8000640 1 -- 172.21.15.118:0/1109866265 >> 172.21.15.118:0/1109866265 conn(0x7f2ae00fee90 msgr2=0x7f2ae01012b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:12:52.359 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:52.359+0000 7f2ae8000640 1 -- 172.21.15.118:0/1109866265 shutdown_connections 2024-03-24T00:12:52.359 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:52.359+0000 7f2ae8000640 1 -- 172.21.15.118:0/1109866265 wait complete. 2024-03-24T00:12:52.361 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:52.360+0000 7f2ae8000640 1 Processor -- start 2024-03-24T00:12:52.362 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:52.360+0000 7f2ae8000640 1 -- start start 2024-03-24T00:12:52.362 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:52.360+0000 7f2ae8000640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f2ae01038e0 0x7f2ae0199c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:12:52.362 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:52.360+0000 7f2ae8000640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2ae0104ad0 0x7f2ae019a170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:12:52.362 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:52.360+0000 7f2ae8000640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f2ae0105490 0x7f2ae01a11a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:12:52.362 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:52.360+0000 7f2ae8000640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2ae010f560 con 0x7f2ae01038e0 2024-03-24T00:12:52.362 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:52.360+0000 7f2ae8000640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2ae010f3e0 con 0x7f2ae0104ad0 2024-03-24T00:12:52.362 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:52.360+0000 7f2ae8000640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2ae010f6e0 con 0x7f2ae0105490 2024-03-24T00:12:52.362 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:52.361+0000 7f2ae5600640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f2ae01038e0 0x7f2ae0199c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:12:52.362 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:52.361+0000 7f2ae5600640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f2ae01038e0 0x7f2ae0199c30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.118:3300/0 says I am v2:172.21.15.118:33616/0 (socket says 172.21.15.118:33616) 2024-03-24T00:12:52.362 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:52.361+0000 7f2ae5600640 1 -- 172.21.15.118:0/2582713969 learned_addr learned my addr 172.21.15.118:0/2582713969 (peer_addr_for_me v2:172.21.15.118:0/0) 2024-03-24T00:12:52.362 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:52.361+0000 7f2ae4c00640 1 --2- 172.21.15.118:0/2582713969 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2ae0104ad0 0x7f2ae019a170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:12:52.362 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:52.361+0000 7f2ae6000640 1 --2- 172.21.15.118:0/2582713969 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f2ae0105490 0x7f2ae01a11a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:12:52.362 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:52.361+0000 7f2ae5600640 1 -- 172.21.15.118:0/2582713969 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f2ae0105490 msgr2=0x7f2ae01a11a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:12:52.363 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:52.361+0000 7f2ae5600640 1 --2- 172.21.15.118:0/2582713969 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f2ae0105490 0x7f2ae01a11a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:12:52.363 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:52.361+0000 7f2ae5600640 1 -- 172.21.15.118:0/2582713969 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2ae0104ad0 msgr2=0x7f2ae019a170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:12:52.363 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:52.361+0000 7f2ae5600640 1 --2- 172.21.15.118:0/2582713969 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2ae0104ad0 0x7f2ae019a170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:12:52.363 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:52.361+0000 7f2ae5600640 1 -- 172.21.15.118:0/2582713969 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2ae01a1840 con 0x7f2ae01038e0 2024-03-24T00:12:52.363 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:52.361+0000 7f2ae4c00640 1 --2- 172.21.15.118:0/2582713969 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2ae0104ad0 0x7f2ae019a170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T00:12:52.363 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:52.361+0000 7f2ae5600640 1 --2- 172.21.15.118:0/2582713969 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f2ae01038e0 0x7f2ae0199c30 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f2ad4000c00 tx=0x7f2ad4005d50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:12:52.363 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:52.362+0000 7f2ade000640 1 -- 172.21.15.118:0/2582713969 <== mon.0 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2ad4041070 con 0x7f2ae01038e0 2024-03-24T00:12:52.363 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:52.362+0000 7f2ade000640 1 -- 172.21.15.118:0/2582713969 <== mon.0 v2:172.21.15.118:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f2ad40044d0 con 0x7f2ae01038e0 2024-03-24T00:12:52.363 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:52.362+0000 7f2ae8000640 1 -- 172.21.15.118:0/2582713969 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2ae01a1ad0 con 0x7f2ae01038e0 2024-03-24T00:12:52.363 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:52.362+0000 7f2ade000640 1 -- 172.21.15.118:0/2582713969 <== mon.0 v2:172.21.15.118:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2ad4034440 con 0x7f2ae01038e0 2024-03-24T00:12:52.363 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:52.362+0000 7f2ae8000640 1 -- 172.21.15.118:0/2582713969 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2ae01a1fc0 con 0x7f2ae01038e0 2024-03-24T00:12:52.364 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:52.363+0000 7f2ae8000640 1 -- 172.21.15.118:0/2582713969 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2ae010ac00 con 0x7f2ae01038e0 2024-03-24T00:12:52.364 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:52.365+0000 7f2ade000640 1 -- 172.21.15.118:0/2582713969 <== mon.0 v2:172.21.15.118:3300/0 4 ==== mgrmap(e 16) v1 ==== 137291+0+0 (secure 0 0 0) 0x7f2ad4044070 con 0x7f2ae01038e0 2024-03-24T00:12:52.367 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:52.367+0000 7f2ade000640 1 --2- 172.21.15.118:0/2582713969 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7f2ac80a2800 0x7f2ac80a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:12:52.367 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:52.368+0000 7f2ade000640 1 -- 172.21.15.118:0/2582713969 <== mon.0 v2:172.21.15.118:3300/0 5 ==== osd_map(82..82 src has 1..82) v4 ==== 7127+0+0 (secure 0 0 0) 0x7f2ad40f0340 con 0x7f2ae01038e0 2024-03-24T00:12:52.367 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:52.368+0000 7f2ae4c00640 1 --2- 172.21.15.118:0/2582713969 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7f2ac80a2800 0x7f2ac80a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:12:52.368 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:52.368+0000 7f2ae4c00640 1 --2- 172.21.15.118:0/2582713969 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7f2ac80a2800 0x7f2ac80a4cc0 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f2acc009c20 tx=0x7f2acc005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:12:52.372 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:52.372+0000 7f2ade000640 1 -- 172.21.15.118:0/2582713969 <== mon.0 v2:172.21.15.118:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f2ad40bd7e0 con 0x7f2ae01038e0 2024-03-24T00:12:52.623 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:52.623+0000 7f2ae8000640 1 -- 172.21.15.118:0/2582713969 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f2ae002d070 con 0x7f2ae01038e0 2024-03-24T00:12:52.623 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:52.624+0000 7f2ade000640 1 -- 172.21.15.118:0/2582713969 <== mon.0 v2:172.21.15.118:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 11 v11) v1 ==== 94+0+8474 (secure 0 0 0) 0x7f2ad40bd0e0 con 0x7f2ae01038e0 2024-03-24T00:12:52.624 INFO:teuthology.orchestra.run.smithi118.stdout: 2024-03-24T00:12:52.624 INFO:teuthology.orchestra.run.smithi118.stdout:{"epoch":11,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14385,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.118:6835/3194741373","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6834","nonce":3194741373},{"type":"v1","addr":"172.21.15.118:6835","nonce":3194741373}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":4},{"gid":14397,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.118:6837/1374462956","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6836","nonce":1374462956},{"type":"v1","addr":"172.21.15.118:6837","nonce":1374462956}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":5},{"gid":24439,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.193:6833/1023428412","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6832","nonce":1023428412},{"type":"v1","addr":"172.21.15.193:6833","nonce":1023428412}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":24440,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.189:6833/2740024078","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6832","nonce":2740024078},{"type":"v1","addr":"172.21.15.189:6833","nonce":2740024078}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":24488,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.189:6835/599473205","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6834","nonce":599473205},{"type":"v1","addr":"172.21.15.189:6835","nonce":599473205}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":6},{"gid":24499,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.193:6835/2055310541","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6834","nonce":2055310541},{"type":"v1","addr":"172.21.15.193:6835","nonce":2055310541}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":7},{"gid":24511,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.193:6837/2921520053","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6836","nonce":2921520053},{"type":"v1","addr":"172.21.15.193:6837","nonce":2921520053}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":8},{"gid":24515,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.189:6837/3450187137","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6836","nonce":3450187137},{"type":"v1","addr":"172.21.15.189:6837","nonce":3450187137}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":9},{"gid":24532,"name":"i","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.193:6839/2126842603","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6838","nonce":2126842603},{"type":"v1","addr":"172.21.15.193:6839","nonce":2126842603}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":10},{"gid":24550,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.118:6839/3285788503","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6838","nonce":3285788503},{"type":"v1","addr":"172.21.15.118:6839","nonce":3285788503}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":11}],"filesystems":[]} 2024-03-24T00:12:52.624 INFO:teuthology.orchestra.run.smithi118.stderr:dumped fsmap epoch 11 2024-03-24T00:12:52.626 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:52.627+0000 7f2ae8000640 1 -- 172.21.15.118:0/2582713969 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7f2ac80a2800 msgr2=0x7f2ac80a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:12:52.627 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:52.627+0000 7f2ae8000640 1 --2- 172.21.15.118:0/2582713969 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7f2ac80a2800 0x7f2ac80a4cc0 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f2acc009c20 tx=0x7f2acc005f90 comp rx=0 tx=0).stop 2024-03-24T00:12:52.627 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:52.627+0000 7f2ae8000640 1 -- 172.21.15.118:0/2582713969 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f2ae01038e0 msgr2=0x7f2ae0199c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:12:52.627 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:52.627+0000 7f2ae8000640 1 --2- 172.21.15.118:0/2582713969 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f2ae01038e0 0x7f2ae0199c30 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f2ad4000c00 tx=0x7f2ad4005d50 comp rx=0 tx=0).stop 2024-03-24T00:12:52.627 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:52.627+0000 7f2ae8000640 1 -- 172.21.15.118:0/2582713969 shutdown_connections 2024-03-24T00:12:52.627 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:52.627+0000 7f2ae8000640 1 --2- 172.21.15.118:0/2582713969 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7f2ac80a2800 0x7f2ac80a4cc0 unknown :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:12:52.627 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:52.627+0000 7f2ae8000640 1 --2- 172.21.15.118:0/2582713969 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f2ae0105490 0x7f2ae01a11a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:12:52.627 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:52.627+0000 7f2ae8000640 1 --2- 172.21.15.118:0/2582713969 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2ae0104ad0 0x7f2ae019a170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:12:52.627 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:52.627+0000 7f2ae8000640 1 --2- 172.21.15.118:0/2582713969 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f2ae01038e0 0x7f2ae0199c30 unknown :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:12:52.627 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:52.627+0000 7f2ae8000640 1 -- 172.21.15.118:0/2582713969 >> 172.21.15.118:0/2582713969 conn(0x7f2ae00fee90 msgr2=0x7f2ae0107dd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:12:52.627 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:52.627+0000 7f2ae8000640 1 -- 172.21.15.118:0/2582713969 shutdown_connections 2024-03-24T00:12:52.627 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:52.627+0000 7f2ae8000640 1 -- 172.21.15.118:0/2582713969 wait complete. 2024-03-24T00:12:52.642 INFO:journalctl@ceph.mds.a.smithi118.stdout:Mar 24 00:12:52 smithi118 podman[73949]: 2024-03-24 00:12:52.341396894 +0000 UTC m=+0.482048283 container start 8b122c08c5eea799f91ee96444ffb9f22e8135af2a327414fac0fa3f34694326 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-a, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240319, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_CLEAN=True, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-24T00:12:52.642 INFO:journalctl@ceph.mds.a.smithi118.stdout:Mar 24 00:12:52 smithi118 ceph-mds[73988]: -- 172.21.15.118:0/67276546 <== mon.0 v2:172.21.15.118:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x5570eb8fda40 con 0x5570eb8d0800 2024-03-24T00:12:52.642 INFO:journalctl@ceph.mds.a.smithi118.stdout:Mar 24 00:12:52 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-a[73967]: starting mds.a at 2024-03-24T00:12:52.643 INFO:journalctl@ceph.mds.a.smithi118.stdout:Mar 24 00:12:52 smithi118 bash[73949]: 8b122c08c5eea799f91ee96444ffb9f22e8135af2a327414fac0fa3f34694326 2024-03-24T00:12:52.643 INFO:journalctl@ceph.mds.a.smithi118.stdout:Mar 24 00:12:52 smithi118 systemd[1]: Started Ceph mds.a for d79a5420-e971-11ee-95cd-87774f69a715. 2024-03-24T00:12:52.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:12:52 smithi118 ceph-mon[31535]: pgmap v305: 1 pgs: 1 active+clean; 577 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-24T00:12:52.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:12:52 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T00:12:52.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:12:52 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T00:12:52.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:12:52 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T00:12:52.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:12:52 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.b", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-24T00:12:52.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:12:52 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.b", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-24T00:12:52.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:12:52 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T00:12:52.947 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:12:52 smithi118 ceph-mon[31535]: from='client.? 172.21.15.118:0/2582713969' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-24T00:12:52.953 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:12:52 smithi189 ceph-mon[36187]: pgmap v305: 1 pgs: 1 active+clean; 577 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-24T00:12:52.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:12:52 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T00:12:52.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:12:52 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T00:12:52.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:12:52 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T00:12:52.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:12:52 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.b", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-24T00:12:52.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:12:52 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.b", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-24T00:12:52.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:12:52 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T00:12:52.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:12:52 smithi189 ceph-mon[36187]: from='client.? 172.21.15.118:0/2582713969' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-24T00:12:52.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:12:52 smithi193 ceph-mon[36878]: pgmap v305: 1 pgs: 1 active+clean; 577 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-24T00:12:52.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:12:52 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T00:12:52.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:12:52 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T00:12:52.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:12:52 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T00:12:52.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:12:52 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "auth get-or-create", "entity": "mds.b", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-24T00:12:52.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:12:52 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd='[{"prefix": "auth get-or-create", "entity": "mds.b", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-24T00:12:52.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:12:52 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T00:12:52.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:12:52 smithi193 ceph-mon[36878]: from='client.? 172.21.15.118:0/2582713969' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-24T00:12:53.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:12:53 smithi118 ceph-mon[31535]: Deploying daemon mds.b on smithi189 2024-03-24T00:12:53.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:12:53 smithi118 ceph-mon[31535]: mds.? [v2:172.21.15.118:6840/3040946537,v1:172.21.15.118:6841/3040946537] up:boot 2024-03-24T00:12:53.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:12:53 smithi118 ceph-mon[31535]: fsmap 11 up:standby 2024-03-24T00:12:53.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:12:53 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "a"}]: dispatch 2024-03-24T00:12:53.953 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:12:53 smithi189 ceph-mon[36187]: Deploying daemon mds.b on smithi189 2024-03-24T00:12:53.953 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:12:53 smithi189 ceph-mon[36187]: mds.? [v2:172.21.15.118:6840/3040946537,v1:172.21.15.118:6841/3040946537] up:boot 2024-03-24T00:12:53.953 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:12:53 smithi189 ceph-mon[36187]: fsmap 11 up:standby 2024-03-24T00:12:53.953 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:12:53 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "a"}]: dispatch 2024-03-24T00:12:53.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:12:53 smithi193 ceph-mon[36878]: Deploying daemon mds.b on smithi189 2024-03-24T00:12:53.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:12:53 smithi193 ceph-mon[36878]: mds.? [v2:172.21.15.118:6840/3040946537,v1:172.21.15.118:6841/3040946537] up:boot 2024-03-24T00:12:53.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:12:53 smithi193 ceph-mon[36878]: fsmap 11 up:standby 2024-03-24T00:12:53.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:12:53 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "a"}]: dispatch 2024-03-24T00:12:54.646 DEBUG:teuthology.orchestra.run.smithi118:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-24T00:12:54.700 INFO:journalctl@ceph.mds.b.smithi189.stdout:Mar 24 00:12:54 smithi189 systemd[1]: Starting Ceph mds.b for d79a5420-e971-11ee-95cd-87774f69a715... 2024-03-24T00:12:54.862 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:54.862+0000 7fda7aa00640 1 -- 172.21.15.118:0/2603710981 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fda74108cf0 msgr2=0x7fda74109170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:12:54.862 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:54.862+0000 7fda7aa00640 1 --2- 172.21.15.118:0/2603710981 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fda74108cf0 0x7fda74109170 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7fda60009860 tx=0x7fda6002bb70 comp rx=0 tx=0).stop 2024-03-24T00:12:54.862 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:54.862+0000 7fda7aa00640 1 -- 172.21.15.118:0/2603710981 shutdown_connections 2024-03-24T00:12:54.862 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:54.862+0000 7fda7aa00640 1 --2- 172.21.15.118:0/2603710981 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fda741096b0 0x7fda7410ff40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:12:54.862 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:54.862+0000 7fda7aa00640 1 --2- 172.21.15.118:0/2603710981 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fda74108cf0 0x7fda74109170 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:12:54.862 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:54.862+0000 7fda7aa00640 1 --2- 172.21.15.118:0/2603710981 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fda74107b00 0x7fda74107f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:12:54.862 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:54.862+0000 7fda7aa00640 1 -- 172.21.15.118:0/2603710981 >> 172.21.15.118:0/2603710981 conn(0x7fda74076ab0 msgr2=0x7fda74078ed0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:12:54.862 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:54.862+0000 7fda7aa00640 1 -- 172.21.15.118:0/2603710981 shutdown_connections 2024-03-24T00:12:54.862 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:54.862+0000 7fda7aa00640 1 -- 172.21.15.118:0/2603710981 wait complete. 2024-03-24T00:12:54.864 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:54.863+0000 7fda7aa00640 1 Processor -- start 2024-03-24T00:12:54.864 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:54.863+0000 7fda7aa00640 1 -- start start 2024-03-24T00:12:54.864 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:54.864+0000 7fda7aa00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fda74107b00 0x7fda7419de00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:12:54.865 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:54.864+0000 7fda7aa00640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fda74108cf0 0x7fda7419e340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:12:54.865 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:54.864+0000 7fda7aa00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fda741096b0 0x7fda741a53b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:12:54.865 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:54.864+0000 7fda7aa00640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fda74113760 con 0x7fda74108cf0 2024-03-24T00:12:54.865 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:54.864+0000 7fda7aa00640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fda741135e0 con 0x7fda74107b00 2024-03-24T00:12:54.865 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:54.864+0000 7fda7aa00640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fda741138e0 con 0x7fda741096b0 2024-03-24T00:12:54.865 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:54.864+0000 7fda78c00640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fda74108cf0 0x7fda7419e340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:12:54.865 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:54.864+0000 7fda78c00640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fda74108cf0 0x7fda7419e340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.118:3300/0 says I am v2:172.21.15.118:50070/0 (socket says 172.21.15.118:50070) 2024-03-24T00:12:54.865 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:54.864+0000 7fda78c00640 1 -- 172.21.15.118:0/1088042803 learned_addr learned my addr 172.21.15.118:0/1088042803 (peer_addr_for_me v2:172.21.15.118:0/0) 2024-03-24T00:12:54.865 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:54.864+0000 7fda79600640 1 --2- 172.21.15.118:0/1088042803 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fda74107b00 0x7fda7419de00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:12:54.865 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:54.864+0000 7fda7a000640 1 --2- 172.21.15.118:0/1088042803 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fda741096b0 0x7fda741a53b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:12:54.865 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:54.864+0000 7fda78c00640 1 -- 172.21.15.118:0/1088042803 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fda741096b0 msgr2=0x7fda741a53b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:12:54.865 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:54.864+0000 7fda78c00640 1 --2- 172.21.15.118:0/1088042803 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fda741096b0 0x7fda741a53b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:12:54.865 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:54.864+0000 7fda78c00640 1 -- 172.21.15.118:0/1088042803 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fda74107b00 msgr2=0x7fda7419de00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:12:54.865 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:54.864+0000 7fda78c00640 1 --2- 172.21.15.118:0/1088042803 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fda74107b00 0x7fda7419de00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:12:54.866 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:54.864+0000 7fda78c00640 1 -- 172.21.15.118:0/1088042803 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fda741a5a20 con 0x7fda74108cf0 2024-03-24T00:12:54.866 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:54.865+0000 7fda78c00640 1 --2- 172.21.15.118:0/1088042803 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fda74108cf0 0x7fda7419e340 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7fda600349e0 tx=0x7fda60032450 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:12:54.866 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:54.865+0000 7fda72000640 1 -- 172.21.15.118:0/1088042803 <== mon.0 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fda60041030 con 0x7fda74108cf0 2024-03-24T00:12:54.866 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:54.865+0000 7fda7aa00640 1 -- 172.21.15.118:0/1088042803 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fda741a5cb0 con 0x7fda74108cf0 2024-03-24T00:12:54.866 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:54.865+0000 7fda72000640 1 -- 172.21.15.118:0/1088042803 <== mon.0 v2:172.21.15.118:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fda6002c510 con 0x7fda74108cf0 2024-03-24T00:12:54.866 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:54.865+0000 7fda72000640 1 -- 172.21.15.118:0/1088042803 <== mon.0 v2:172.21.15.118:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fda60039040 con 0x7fda74108cf0 2024-03-24T00:12:54.866 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:54.865+0000 7fda7aa00640 1 -- 172.21.15.118:0/1088042803 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fda741a61a0 con 0x7fda74108cf0 2024-03-24T00:12:54.866 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:54.866+0000 7fda7aa00640 1 -- 172.21.15.118:0/1088042803 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fda7410ee20 con 0x7fda74108cf0 2024-03-24T00:12:54.867 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:54.868+0000 7fda72000640 1 -- 172.21.15.118:0/1088042803 <== mon.0 v2:172.21.15.118:3300/0 4 ==== mgrmap(e 16) v1 ==== 137291+0+0 (secure 0 0 0) 0x7fda60044070 con 0x7fda74108cf0 2024-03-24T00:12:54.869 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:54.869+0000 7fda72000640 1 --2- 172.21.15.118:0/1088042803 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7fda5c0a2800 0x7fda5c0a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:12:54.869 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:54.870+0000 7fda72000640 1 -- 172.21.15.118:0/1088042803 <== mon.0 v2:172.21.15.118:3300/0 5 ==== osd_map(82..82 src has 1..82) v4 ==== 7127+0+0 (secure 0 0 0) 0x7fda600efbc0 con 0x7fda74108cf0 2024-03-24T00:12:54.869 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:54.870+0000 7fda79600640 1 --2- 172.21.15.118:0/1088042803 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7fda5c0a2800 0x7fda5c0a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:12:54.870 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:54.870+0000 7fda79600640 1 --2- 172.21.15.118:0/1088042803 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7fda5c0a2800 0x7fda5c0a4cc0 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7fda74108b50 tx=0x7fda68005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:12:54.873 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:54.874+0000 7fda72000640 1 -- 172.21.15.118:0/1088042803 <== mon.0 v2:172.21.15.118:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7fda600bd060 con 0x7fda74108cf0 2024-03-24T00:12:54.953 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:12:54 smithi189 ceph-mon[36187]: pgmap v306: 1 pgs: 1 active+clean; 577 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-24T00:12:54.953 INFO:journalctl@ceph.mds.b.smithi189.stdout:Mar 24 00:12:54 smithi189 podman[66960]: 2024-03-24 00:12:54.746689129 +0000 UTC m=+0.017492917 image pull c664afe56bb01db4f9afa9532c20598e201e664f2e9ba2be50db0d52674a26d2 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc 2024-03-24T00:12:54.954 INFO:journalctl@ceph.mds.b.smithi189.stdout:Mar 24 00:12:54 smithi189 podman[66960]: 2024-03-24 00:12:54.869678358 +0000 UTC m=+0.140482128 container create a872553a64922db566fa034c584c9492ef898423c008d98e4780f6e3fe67960b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-b, GIT_CLEAN=True, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240319, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, ceph=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, io.buildah.version=1.33.5) 2024-03-24T00:12:54.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:12:54 smithi193 ceph-mon[36878]: pgmap v306: 1 pgs: 1 active+clean; 577 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-24T00:12:55.108 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:55.109+0000 7fda7aa00640 1 -- 172.21.15.118:0/1088042803 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fda740622a0 con 0x7fda74108cf0 2024-03-24T00:12:55.110 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:55.110+0000 7fda72000640 1 -- 172.21.15.118:0/1088042803 <== mon.0 v2:172.21.15.118:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 12 v12) v1 ==== 94+0+9269 (secure 0 0 0) 0x7fda600bc960 con 0x7fda74108cf0 2024-03-24T00:12:55.110 INFO:teuthology.orchestra.run.smithi118.stdout: 2024-03-24T00:12:55.110 INFO:teuthology.orchestra.run.smithi118.stdout:{"epoch":12,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14385,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.118:6835/3194741373","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6834","nonce":3194741373},{"type":"v1","addr":"172.21.15.118:6835","nonce":3194741373}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":4},{"gid":14397,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.118:6837/1374462956","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6836","nonce":1374462956},{"type":"v1","addr":"172.21.15.118:6837","nonce":1374462956}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":5},{"gid":14469,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.118:6841/3040946537","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6840","nonce":3040946537},{"type":"v1","addr":"172.21.15.118:6841","nonce":3040946537}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":12},{"gid":24439,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.193:6833/1023428412","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6832","nonce":1023428412},{"type":"v1","addr":"172.21.15.193:6833","nonce":1023428412}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":24440,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.189:6833/2740024078","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6832","nonce":2740024078},{"type":"v1","addr":"172.21.15.189:6833","nonce":2740024078}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":24488,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.189:6835/599473205","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6834","nonce":599473205},{"type":"v1","addr":"172.21.15.189:6835","nonce":599473205}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":6},{"gid":24499,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.193:6835/2055310541","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6834","nonce":2055310541},{"type":"v1","addr":"172.21.15.193:6835","nonce":2055310541}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":7},{"gid":24511,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.193:6837/2921520053","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6836","nonce":2921520053},{"type":"v1","addr":"172.21.15.193:6837","nonce":2921520053}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":8},{"gid":24515,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.189:6837/3450187137","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6836","nonce":3450187137},{"type":"v1","addr":"172.21.15.189:6837","nonce":3450187137}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":9},{"gid":24532,"name":"i","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.193:6839/2126842603","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6838","nonce":2126842603},{"type":"v1","addr":"172.21.15.193:6839","nonce":2126842603}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":10},{"gid":24550,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.118:6839/3285788503","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6838","nonce":3285788503},{"type":"v1","addr":"172.21.15.118:6839","nonce":3285788503}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":11}],"filesystems":[]} 2024-03-24T00:12:55.110 INFO:teuthology.orchestra.run.smithi118.stderr:dumped fsmap epoch 12 2024-03-24T00:12:55.113 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:55.113+0000 7fda7aa00640 1 -- 172.21.15.118:0/1088042803 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7fda5c0a2800 msgr2=0x7fda5c0a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:12:55.113 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:55.113+0000 7fda7aa00640 1 --2- 172.21.15.118:0/1088042803 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7fda5c0a2800 0x7fda5c0a4cc0 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7fda74108b50 tx=0x7fda68005f90 comp rx=0 tx=0).stop 2024-03-24T00:12:55.113 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:55.113+0000 7fda7aa00640 1 -- 172.21.15.118:0/1088042803 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fda74108cf0 msgr2=0x7fda7419e340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:12:55.113 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:55.113+0000 7fda7aa00640 1 --2- 172.21.15.118:0/1088042803 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fda74108cf0 0x7fda7419e340 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7fda600349e0 tx=0x7fda60032450 comp rx=0 tx=0).stop 2024-03-24T00:12:55.113 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:55.113+0000 7fda7aa00640 1 -- 172.21.15.118:0/1088042803 shutdown_connections 2024-03-24T00:12:55.113 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:55.113+0000 7fda7aa00640 1 --2- 172.21.15.118:0/1088042803 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7fda5c0a2800 0x7fda5c0a4cc0 unknown :-1 s=CLOSED pgs=153 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:12:55.113 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:55.113+0000 7fda7aa00640 1 --2- 172.21.15.118:0/1088042803 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fda741096b0 0x7fda741a53b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:12:55.113 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:55.113+0000 7fda7aa00640 1 --2- 172.21.15.118:0/1088042803 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fda74108cf0 0x7fda7419e340 unknown :-1 s=CLOSED pgs=154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:12:55.113 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:55.113+0000 7fda7aa00640 1 --2- 172.21.15.118:0/1088042803 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fda74107b00 0x7fda7419de00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:12:55.113 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:55.113+0000 7fda7aa00640 1 -- 172.21.15.118:0/1088042803 >> 172.21.15.118:0/1088042803 conn(0x7fda74076ab0 msgr2=0x7fda7410bff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:12:55.113 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:55.113+0000 7fda7aa00640 1 -- 172.21.15.118:0/1088042803 shutdown_connections 2024-03-24T00:12:55.113 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:55.113+0000 7fda7aa00640 1 -- 172.21.15.118:0/1088042803 wait complete. 2024-03-24T00:12:55.128 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:12:54 smithi118 ceph-mon[31535]: pgmap v306: 1 pgs: 1 active+clean; 577 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-24T00:12:55.454 INFO:journalctl@ceph.mds.b.smithi189.stdout:Mar 24 00:12:55 smithi189 podman[66960]: 2024-03-24 00:12:55.106073204 +0000 UTC m=+0.376876999 container init a872553a64922db566fa034c584c9492ef898423c008d98e4780f6e3fe67960b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-b, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.build-date=20240319, maintainer=Guillaume Abrioux , GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5) 2024-03-24T00:12:55.454 INFO:journalctl@ceph.mds.b.smithi189.stdout:Mar 24 00:12:55 smithi189 podman[66960]: 2024-03-24 00:12:55.108813328 +0000 UTC m=+0.379617102 container start a872553a64922db566fa034c584c9492ef898423c008d98e4780f6e3fe67960b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-b, GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, ceph=True, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.build-date=20240319, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-24T00:12:55.454 INFO:journalctl@ceph.mds.b.smithi189.stdout:Mar 24 00:12:55 smithi189 ceph-mds[66979]: -- 172.21.15.189:0/1737172923 <== mon.2 v2:172.21.15.189:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x55fe84c61a40 con 0x55fe84c35000 2024-03-24T00:12:55.454 INFO:journalctl@ceph.mds.b.smithi189.stdout:Mar 24 00:12:55 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-b[66975]: starting mds.b at 2024-03-24T00:12:55.454 INFO:journalctl@ceph.mds.b.smithi189.stdout:Mar 24 00:12:55 smithi189 bash[66960]: a872553a64922db566fa034c584c9492ef898423c008d98e4780f6e3fe67960b 2024-03-24T00:12:55.454 INFO:journalctl@ceph.mds.b.smithi189.stdout:Mar 24 00:12:55 smithi189 systemd[1]: Started Ceph mds.b for d79a5420-e971-11ee-95cd-87774f69a715. 2024-03-24T00:12:55.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:12:55 smithi193 ceph-mon[36878]: from='client.? 172.21.15.118:0/1088042803' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-24T00:12:55.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:12:55 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T00:12:55.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:12:55 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T00:12:55.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:12:55 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T00:12:55.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:12:55 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T00:12:55.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:12:55 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T00:12:55.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:12:55 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T00:12:55.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:12:55 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T00:12:56.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:12:55 smithi118 ceph-mon[31535]: from='client.? 172.21.15.118:0/1088042803' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-24T00:12:56.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:12:55 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T00:12:56.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:12:55 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T00:12:56.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:12:55 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T00:12:56.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:12:55 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T00:12:56.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:12:55 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T00:12:56.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:12:55 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T00:12:56.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:12:55 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T00:12:56.203 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:12:55 smithi189 ceph-mon[36187]: from='client.? 172.21.15.118:0/1088042803' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-24T00:12:56.203 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:12:55 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T00:12:56.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:12:55 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T00:12:56.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:12:55 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T00:12:56.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:12:55 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T00:12:56.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:12:55 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T00:12:56.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:12:55 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T00:12:56.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:12:55 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T00:12:56.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:12:56 smithi189 ceph-mon[36187]: pgmap v307: 1 pgs: 1 active+clean; 577 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-24T00:12:56.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:12:56 smithi189 ceph-mon[36187]: mds.? [v2:172.21.15.189:6838/4074484168,v1:172.21.15.189:6839/4074484168] up:boot 2024-03-24T00:12:56.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:12:56 smithi189 ceph-mon[36187]: fsmap 12 up:standby 2024-03-24T00:12:56.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:12:56 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2024-03-24T00:12:56.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:12:56 smithi193 ceph-mon[36878]: pgmap v307: 1 pgs: 1 active+clean; 577 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-24T00:12:56.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:12:56 smithi193 ceph-mon[36878]: mds.? [v2:172.21.15.189:6838/4074484168,v1:172.21.15.189:6839/4074484168] up:boot 2024-03-24T00:12:56.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:12:56 smithi193 ceph-mon[36878]: fsmap 12 up:standby 2024-03-24T00:12:56.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:12:56 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2024-03-24T00:12:57.134 DEBUG:teuthology.orchestra.run.smithi118:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-24T00:12:57.155 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:12:56 smithi118 ceph-mon[31535]: pgmap v307: 1 pgs: 1 active+clean; 577 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-24T00:12:57.155 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:12:56 smithi118 ceph-mon[31535]: mds.? [v2:172.21.15.189:6838/4074484168,v1:172.21.15.189:6839/4074484168] up:boot 2024-03-24T00:12:57.155 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:12:56 smithi118 ceph-mon[31535]: fsmap 12 up:standby 2024-03-24T00:12:57.155 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:12:56 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2024-03-24T00:12:57.342 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:57.342+0000 7ff31aa00640 1 -- 172.21.15.118:0/655592310 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff314104a90 msgr2=0x7ff314106e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:12:57.342 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:57.342+0000 7ff31aa00640 1 --2- 172.21.15.118:0/655592310 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff314104a90 0x7ff314106e80 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7ff300009900 tx=0x7ff30002bc30 comp rx=0 tx=0).stop 2024-03-24T00:12:57.342 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:57.342+0000 7ff31aa00640 1 -- 172.21.15.118:0/655592310 shutdown_connections 2024-03-24T00:12:57.342 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:57.342+0000 7ff31aa00640 1 --2- 172.21.15.118:0/655592310 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7ff3141073c0 0x7ff314109810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:12:57.342 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:57.342+0000 7ff31aa00640 1 --2- 172.21.15.118:0/655592310 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff314104a90 0x7ff314106e80 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:12:57.342 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:57.342+0000 7ff31aa00640 1 --2- 172.21.15.118:0/655592310 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7ff314102160 0x7ff314104550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:12:57.342 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:57.342+0000 7ff31aa00640 1 -- 172.21.15.118:0/655592310 >> 172.21.15.118:0/655592310 conn(0x7ff3140fbb50 msgr2=0x7ff3140fdf90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:12:57.342 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:57.342+0000 7ff31aa00640 1 -- 172.21.15.118:0/655592310 shutdown_connections 2024-03-24T00:12:57.342 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:57.342+0000 7ff31aa00640 1 -- 172.21.15.118:0/655592310 wait complete. 2024-03-24T00:12:57.345 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:57.343+0000 7ff31aa00640 1 Processor -- start 2024-03-24T00:12:57.345 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:57.343+0000 7ff31aa00640 1 -- start start 2024-03-24T00:12:57.345 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:57.343+0000 7ff31aa00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7ff314102160 0x7ff314197a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:12:57.345 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:57.343+0000 7ff31aa00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff314104a90 0x7ff314197f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:12:57.345 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:57.343+0000 7ff31aa00640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7ff3141073c0 0x7ff31419efc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:12:57.345 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:57.343+0000 7ff31aa00640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff31410d3c0 con 0x7ff3141073c0 2024-03-24T00:12:57.345 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:57.343+0000 7ff31aa00640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff31410d240 con 0x7ff314104a90 2024-03-24T00:12:57.345 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:57.343+0000 7ff31aa00640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff31410d540 con 0x7ff314102160 2024-03-24T00:12:57.345 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:57.344+0000 7ff313400640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff314104a90 0x7ff314197f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:12:57.345 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:57.344+0000 7ff313400640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff314104a90 0x7ff314197f50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.118:50142/0 (socket says 172.21.15.118:50142) 2024-03-24T00:12:57.346 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:57.344+0000 7ff313400640 1 -- 172.21.15.118:0/723484770 learned_addr learned my addr 172.21.15.118:0/723484770 (peer_addr_for_me v2:172.21.15.118:0/0) 2024-03-24T00:12:57.346 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:57.344+0000 7ff318a00640 1 --2- 172.21.15.118:0/723484770 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7ff3141073c0 0x7ff31419efc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:12:57.346 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:57.344+0000 7ff313e00640 1 --2- 172.21.15.118:0/723484770 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7ff314102160 0x7ff314197a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:12:57.346 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:57.344+0000 7ff313400640 1 -- 172.21.15.118:0/723484770 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7ff314102160 msgr2=0x7ff314197a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:12:57.346 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:57.344+0000 7ff313400640 1 --2- 172.21.15.118:0/723484770 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7ff314102160 0x7ff314197a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:12:57.346 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:57.344+0000 7ff313400640 1 -- 172.21.15.118:0/723484770 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7ff3141073c0 msgr2=0x7ff31419efc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:12:57.346 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:57.344+0000 7ff313400640 1 --2- 172.21.15.118:0/723484770 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7ff3141073c0 0x7ff31419efc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:12:57.346 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:57.344+0000 7ff313400640 1 -- 172.21.15.118:0/723484770 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff31419f630 con 0x7ff314104a90 2024-03-24T00:12:57.346 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:57.344+0000 7ff313e00640 1 --2- 172.21.15.118:0/723484770 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7ff314102160 0x7ff314197a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T00:12:57.346 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:57.345+0000 7ff313400640 1 --2- 172.21.15.118:0/723484770 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff314104a90 0x7ff314197f50 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7ff300000c00 tx=0x7ff30002c650 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:12:57.346 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:57.345+0000 7ff310c00640 1 -- 172.21.15.118:0/723484770 <== mon.2 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff300043020 con 0x7ff314104a90 2024-03-24T00:12:57.346 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:57.345+0000 7ff310c00640 1 -- 172.21.15.118:0/723484770 <== mon.2 v2:172.21.15.189:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7ff3000054e0 con 0x7ff314104a90 2024-03-24T00:12:57.346 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:57.345+0000 7ff310c00640 1 -- 172.21.15.118:0/723484770 <== mon.2 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff300003c40 con 0x7ff314104a90 2024-03-24T00:12:57.346 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:57.345+0000 7ff31aa00640 1 -- 172.21.15.118:0/723484770 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff31419f8c0 con 0x7ff314104a90 2024-03-24T00:12:57.347 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:57.345+0000 7ff31aa00640 1 -- 172.21.15.118:0/723484770 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff31419fe60 con 0x7ff314104a90 2024-03-24T00:12:57.347 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:57.346+0000 7ff31aa00640 1 -- 172.21.15.118:0/723484770 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff3140fd640 con 0x7ff314104a90 2024-03-24T00:12:57.348 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:57.349+0000 7ff310c00640 1 -- 172.21.15.118:0/723484770 <== mon.2 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 16) v1 ==== 137291+0+0 (secure 0 0 0) 0x7ff300046020 con 0x7ff314104a90 2024-03-24T00:12:57.350 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:57.350+0000 7ff310c00640 1 --2- 172.21.15.118:0/723484770 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7ff2f80a2880 0x7ff2f80a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:12:57.350 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:57.350+0000 7ff310c00640 1 -- 172.21.15.118:0/723484770 <== mon.2 v2:172.21.15.189:3300/0 5 ==== osd_map(82..82 src has 1..82) v4 ==== 7127+0+0 (secure 0 0 0) 0x7ff3000f0510 con 0x7ff314104a90 2024-03-24T00:12:57.350 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:57.350+0000 7ff313e00640 1 --2- 172.21.15.118:0/723484770 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7ff2f80a2880 0x7ff2f80a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:12:57.350 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:57.351+0000 7ff313e00640 1 --2- 172.21.15.118:0/723484770 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7ff2f80a2880 0x7ff2f80a4d40 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7ff304009fc0 tx=0x7ff304005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:12:57.354 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:57.354+0000 7ff310c00640 1 -- 172.21.15.118:0/723484770 <== mon.2 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7ff30004d020 con 0x7ff314104a90 2024-03-24T00:12:57.607 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:57.607+0000 7ff31aa00640 1 -- 172.21.15.118:0/723484770 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7ff3140622a0 con 0x7ff314104a90 2024-03-24T00:12:57.608 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:57.608+0000 7ff310c00640 1 -- 172.21.15.118:0/723484770 <== mon.2 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 13 v13) v1 ==== 94+0+10064 (secure 0 0 0) 0x7ff3000bda50 con 0x7ff314104a90 2024-03-24T00:12:57.608 INFO:teuthology.orchestra.run.smithi118.stdout: 2024-03-24T00:12:57.608 INFO:teuthology.orchestra.run.smithi118.stdout:{"epoch":13,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14385,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.118:6835/3194741373","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6834","nonce":3194741373},{"type":"v1","addr":"172.21.15.118:6835","nonce":3194741373}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":4},{"gid":14397,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.118:6837/1374462956","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6836","nonce":1374462956},{"type":"v1","addr":"172.21.15.118:6837","nonce":1374462956}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":5},{"gid":14469,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.118:6841/3040946537","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6840","nonce":3040946537},{"type":"v1","addr":"172.21.15.118:6841","nonce":3040946537}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":12},{"gid":24439,"name":"l","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.193:6833/1023428412","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6832","nonce":1023428412},{"type":"v1","addr":"172.21.15.193:6833","nonce":1023428412}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":2},{"gid":24440,"name":"k","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.189:6833/2740024078","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6832","nonce":2740024078},{"type":"v1","addr":"172.21.15.189:6833","nonce":2740024078}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":3},{"gid":24488,"name":"e","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.189:6835/599473205","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6834","nonce":599473205},{"type":"v1","addr":"172.21.15.189:6835","nonce":599473205}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":6},{"gid":24499,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.193:6835/2055310541","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6834","nonce":2055310541},{"type":"v1","addr":"172.21.15.193:6835","nonce":2055310541}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":7},{"gid":24511,"name":"c","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.193:6837/2921520053","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6836","nonce":2921520053},{"type":"v1","addr":"172.21.15.193:6837","nonce":2921520053}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":8},{"gid":24515,"name":"h","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.189:6837/3450187137","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6836","nonce":3450187137},{"type":"v1","addr":"172.21.15.189:6837","nonce":3450187137}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":9},{"gid":24532,"name":"i","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.193:6839/2126842603","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6838","nonce":2126842603},{"type":"v1","addr":"172.21.15.193:6839","nonce":2126842603}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":10},{"gid":24550,"name":"g","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.118:6839/3285788503","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6838","nonce":3285788503},{"type":"v1","addr":"172.21.15.118:6839","nonce":3285788503}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":11},{"gid":24569,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.189:6839/4074484168","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6838","nonce":4074484168},{"type":"v1","addr":"172.21.15.189:6839","nonce":4074484168}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":13}],"filesystems":[]} 2024-03-24T00:12:57.609 INFO:teuthology.orchestra.run.smithi118.stderr:dumped fsmap epoch 13 2024-03-24T00:12:57.611 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:57.611+0000 7ff31aa00640 1 -- 172.21.15.118:0/723484770 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7ff2f80a2880 msgr2=0x7ff2f80a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:12:57.611 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:57.611+0000 7ff31aa00640 1 --2- 172.21.15.118:0/723484770 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7ff2f80a2880 0x7ff2f80a4d40 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7ff304009fc0 tx=0x7ff304005f90 comp rx=0 tx=0).stop 2024-03-24T00:12:57.611 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:57.611+0000 7ff31aa00640 1 -- 172.21.15.118:0/723484770 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff314104a90 msgr2=0x7ff314197f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:12:57.611 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:57.611+0000 7ff31aa00640 1 --2- 172.21.15.118:0/723484770 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff314104a90 0x7ff314197f50 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7ff300000c00 tx=0x7ff30002c650 comp rx=0 tx=0).stop 2024-03-24T00:12:57.611 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:57.611+0000 7ff31aa00640 1 -- 172.21.15.118:0/723484770 shutdown_connections 2024-03-24T00:12:57.611 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:57.611+0000 7ff31aa00640 1 --2- 172.21.15.118:0/723484770 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7ff2f80a2880 0x7ff2f80a4d40 unknown :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:12:57.611 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:57.611+0000 7ff31aa00640 1 --2- 172.21.15.118:0/723484770 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7ff3141073c0 0x7ff31419efc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:12:57.612 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:57.611+0000 7ff31aa00640 1 --2- 172.21.15.118:0/723484770 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff314104a90 0x7ff314197f50 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:12:57.612 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:57.611+0000 7ff31aa00640 1 --2- 172.21.15.118:0/723484770 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7ff314102160 0x7ff314197a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:12:57.612 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:57.611+0000 7ff31aa00640 1 -- 172.21.15.118:0/723484770 >> 172.21.15.118:0/723484770 conn(0x7ff3140fbb50 msgr2=0x7ff314102ba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:12:57.612 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:57.611+0000 7ff31aa00640 1 -- 172.21.15.118:0/723484770 shutdown_connections 2024-03-24T00:12:57.612 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:57.612+0000 7ff31aa00640 1 -- 172.21.15.118:0/723484770 wait complete. 2024-03-24T00:12:57.629 DEBUG:teuthology.orchestra.run.smithi118:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-24T00:12:57.858 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:57.858+0000 7f31bc400640 1 -- 172.21.15.118:0/3290294393 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f31b41038c0 msgr2=0x7f31b4103ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:12:57.858 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:57.858+0000 7f31bc400640 1 --2- 172.21.15.118:0/3290294393 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f31b41038c0 0x7f31b4103ce0 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f31a8009900 tx=0x7f31a802bc30 comp rx=0 tx=0).stop 2024-03-24T00:12:57.858 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:57.858+0000 7f31bc400640 1 -- 172.21.15.118:0/3290294393 shutdown_connections 2024-03-24T00:12:57.858 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:57.858+0000 7f31bc400640 1 --2- 172.21.15.118:0/3290294393 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f31b4105470 0x7f31b410bd00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:12:57.858 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:57.858+0000 7f31bc400640 1 --2- 172.21.15.118:0/3290294393 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f31b4104ab0 0x7f31b4104f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:12:57.858 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:57.858+0000 7f31bc400640 1 --2- 172.21.15.118:0/3290294393 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f31b41038c0 0x7f31b4103ce0 unknown :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:12:57.859 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:57.858+0000 7f31bc400640 1 -- 172.21.15.118:0/3290294393 >> 172.21.15.118:0/3290294393 conn(0x7f31b40fee50 msgr2=0x7f31b4101270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:12:57.859 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:57.858+0000 7f31bc400640 1 -- 172.21.15.118:0/3290294393 shutdown_connections 2024-03-24T00:12:57.859 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:57.858+0000 7f31bc400640 1 -- 172.21.15.118:0/3290294393 wait complete. 2024-03-24T00:12:57.861 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:57.859+0000 7f31bc400640 1 Processor -- start 2024-03-24T00:12:57.861 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:57.859+0000 7f31bc400640 1 -- start start 2024-03-24T00:12:57.861 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:57.860+0000 7f31bc400640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f31b41038c0 0x7f31b4199bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:12:57.861 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:57.860+0000 7f31bc400640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f31b4104ab0 0x7f31b419a110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:12:57.861 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:57.860+0000 7f31bc400640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f31b4105470 0x7f31b41a1180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:12:57.861 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:57.860+0000 7f31bc400640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f31b410f510 con 0x7f31b41038c0 2024-03-24T00:12:57.861 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:57.860+0000 7f31bc400640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f31b410f390 con 0x7f31b4104ab0 2024-03-24T00:12:57.862 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:57.860+0000 7f31bc400640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f31b410f690 con 0x7f31b4105470 2024-03-24T00:12:57.862 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:57.860+0000 7f31b9a00640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f31b41038c0 0x7f31b4199bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:12:57.862 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:57.860+0000 7f31b9a00640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f31b41038c0 0x7f31b4199bd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.118:3300/0 says I am v2:172.21.15.118:50110/0 (socket says 172.21.15.118:50110) 2024-03-24T00:12:57.862 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:57.860+0000 7f31b9a00640 1 -- 172.21.15.118:0/774815908 learned_addr learned my addr 172.21.15.118:0/774815908 (peer_addr_for_me v2:172.21.15.118:0/0) 2024-03-24T00:12:57.862 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:57.860+0000 7f31ba400640 1 --2- 172.21.15.118:0/774815908 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f31b4105470 0x7f31b41a1180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:12:57.862 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:57.860+0000 7f31b9000640 1 --2- 172.21.15.118:0/774815908 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f31b4104ab0 0x7f31b419a110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:12:57.862 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:57.860+0000 7f31b9a00640 1 -- 172.21.15.118:0/774815908 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f31b4105470 msgr2=0x7f31b41a1180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:12:57.862 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:57.860+0000 7f31b9a00640 1 --2- 172.21.15.118:0/774815908 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f31b4105470 0x7f31b41a1180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:12:57.862 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:57.860+0000 7f31b9a00640 1 -- 172.21.15.118:0/774815908 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f31b4104ab0 msgr2=0x7f31b419a110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:12:57.862 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:57.861+0000 7f31b9a00640 1 --2- 172.21.15.118:0/774815908 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f31b4104ab0 0x7f31b419a110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:12:57.862 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:57.861+0000 7f31b9a00640 1 -- 172.21.15.118:0/774815908 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f31b41a17f0 con 0x7f31b41038c0 2024-03-24T00:12:57.862 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:57.861+0000 7f31ba400640 1 --2- 172.21.15.118:0/774815908 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f31b4105470 0x7f31b41a1180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T00:12:57.863 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:57.861+0000 7f31b9a00640 1 --2- 172.21.15.118:0/774815908 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f31b41038c0 0x7f31b4199bd0 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7f31a8000c00 tx=0x7f31a8005d50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:12:57.863 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:57.861+0000 7f31b2000640 1 -- 172.21.15.118:0/774815908 <== mon.0 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f31a8041070 con 0x7f31b41038c0 2024-03-24T00:12:57.863 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:57.861+0000 7f31b2000640 1 -- 172.21.15.118:0/774815908 <== mon.0 v2:172.21.15.118:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f31a80044d0 con 0x7f31b41038c0 2024-03-24T00:12:57.863 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:57.861+0000 7f31bc400640 1 -- 172.21.15.118:0/774815908 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f31b41a1a80 con 0x7f31b41038c0 2024-03-24T00:12:57.863 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:57.861+0000 7f31b2000640 1 -- 172.21.15.118:0/774815908 <== mon.0 v2:172.21.15.118:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f31a8034440 con 0x7f31b41038c0 2024-03-24T00:12:57.863 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:57.861+0000 7f31bc400640 1 -- 172.21.15.118:0/774815908 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f31b41a1f70 con 0x7f31b41038c0 2024-03-24T00:12:57.863 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:57.863+0000 7f31bc400640 1 -- 172.21.15.118:0/774815908 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f31b410acb0 con 0x7f31b41038c0 2024-03-24T00:12:57.864 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:57.864+0000 7f31b2000640 1 -- 172.21.15.118:0/774815908 <== mon.0 v2:172.21.15.118:3300/0 4 ==== mgrmap(e 16) v1 ==== 137291+0+0 (secure 0 0 0) 0x7f31a8044070 con 0x7f31b41038c0 2024-03-24T00:12:57.865 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:57.866+0000 7f31b2000640 1 --2- 172.21.15.118:0/774815908 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7f319c0a2800 0x7f319c0a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:12:57.865 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:57.866+0000 7f31b2000640 1 -- 172.21.15.118:0/774815908 <== mon.0 v2:172.21.15.118:3300/0 5 ==== osd_map(82..82 src has 1..82) v4 ==== 7127+0+0 (secure 0 0 0) 0x7f31a80f0150 con 0x7f31b41038c0 2024-03-24T00:12:57.865 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:57.866+0000 7f31b9000640 1 --2- 172.21.15.118:0/774815908 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7f319c0a2800 0x7f319c0a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:12:57.866 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:57.867+0000 7f31b9000640 1 --2- 172.21.15.118:0/774815908 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7f319c0a2800 0x7f319c0a4cc0 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7f31b419b0e0 tx=0x7f31a0005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:12:57.869 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:57.870+0000 7f31b2000640 1 -- 172.21.15.118:0/774815908 <== mon.0 v2:172.21.15.118:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f31a80bcfa0 con 0x7f31b41038c0 2024-03-24T00:12:58.088 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:58.088+0000 7f31bc400640 1 -- 172.21.15.118:0/774815908 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f31b40622d0 con 0x7f31b41038c0 2024-03-24T00:12:58.089 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:58.089+0000 7f31b2000640 1 -- 172.21.15.118:0/774815908 <== mon.0 v2:172.21.15.118:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v82) v1 ==== 74+0+18524 (secure 0 0 0) 0x7f31a80c1040 con 0x7f31b41038c0 2024-03-24T00:12:58.089 INFO:teuthology.orchestra.run.smithi118.stdout: 2024-03-24T00:12:58.089 INFO:teuthology.orchestra.run.smithi118.stdout:{"epoch":82,"fsid":"d79a5420-e971-11ee-95cd-87774f69a715","created":"2024-03-24T00:03:58.648524+0000","modified":"2024-03-24T00:12:04.819275+0000","last_up_change":"2024-03-24T00:12:03.809000+0000","last_in_change":"2024-03-24T00:11:44.389344+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-24T00:07:50.190965+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"c9590426-3901-4f68-a5ef-d57aa8fbac92","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6802","nonce":557690061},{"type":"v1","addr":"172.21.15.118:6803","nonce":557690061}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6804","nonce":557690061},{"type":"v1","addr":"172.21.15.118:6805","nonce":557690061}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6808","nonce":557690061},{"type":"v1","addr":"172.21.15.118:6809","nonce":557690061}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6806","nonce":557690061},{"type":"v1","addr":"172.21.15.118:6807","nonce":557690061}]},"public_addr":"172.21.15.118:6803/557690061","cluster_addr":"172.21.15.118:6805/557690061","heartbeat_back_addr":"172.21.15.118:6809/557690061","heartbeat_front_addr":"172.21.15.118:6807/557690061","state":["exists","up"]},{"osd":1,"uuid":"fc7f827e-ecf6-4661-aad0-7a22951892e0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":42,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6800","nonce":4203029485},{"type":"v1","addr":"172.21.15.189:6801","nonce":4203029485}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6802","nonce":4203029485},{"type":"v1","addr":"172.21.15.189:6803","nonce":4203029485}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6806","nonce":4203029485},{"type":"v1","addr":"172.21.15.189:6807","nonce":4203029485}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6804","nonce":4203029485},{"type":"v1","addr":"172.21.15.189:6805","nonce":4203029485}]},"public_addr":"172.21.15.189:6801/4203029485","cluster_addr":"172.21.15.189:6803/4203029485","heartbeat_back_addr":"172.21.15.189:6807/4203029485","heartbeat_front_addr":"172.21.15.189:6805/4203029485","state":["exists","up"]},{"osd":2,"uuid":"e73ccfe9-0e5b-4734-a154-a2f0d829fcfb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6800","nonce":3699076971},{"type":"v1","addr":"172.21.15.193:6801","nonce":3699076971}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6802","nonce":3699076971},{"type":"v1","addr":"172.21.15.193:6803","nonce":3699076971}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6806","nonce":3699076971},{"type":"v1","addr":"172.21.15.193:6807","nonce":3699076971}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6804","nonce":3699076971},{"type":"v1","addr":"172.21.15.193:6805","nonce":3699076971}]},"public_addr":"172.21.15.193:6801/3699076971","cluster_addr":"172.21.15.193:6803/3699076971","heartbeat_back_addr":"172.21.15.193:6807/3699076971","heartbeat_front_addr":"172.21.15.193:6805/3699076971","state":["exists","up"]},{"osd":3,"uuid":"63fed43b-43ce-4031-8957-7fe94725c1ed","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6810","nonce":4059617010},{"type":"v1","addr":"172.21.15.118:6811","nonce":4059617010}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6812","nonce":4059617010},{"type":"v1","addr":"172.21.15.118:6813","nonce":4059617010}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6816","nonce":4059617010},{"type":"v1","addr":"172.21.15.118:6817","nonce":4059617010}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6814","nonce":4059617010},{"type":"v1","addr":"172.21.15.118:6815","nonce":4059617010}]},"public_addr":"172.21.15.118:6811/4059617010","cluster_addr":"172.21.15.118:6813/4059617010","heartbeat_back_addr":"172.21.15.118:6817/4059617010","heartbeat_front_addr":"172.21.15.118:6815/4059617010","state":["exists","up"]},{"osd":4,"uuid":"57377f66-c3b4-43c1-8e8e-bc873672906d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":34,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6808","nonce":4075247068},{"type":"v1","addr":"172.21.15.189:6809","nonce":4075247068}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6810","nonce":4075247068},{"type":"v1","addr":"172.21.15.189:6811","nonce":4075247068}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6814","nonce":4075247068},{"type":"v1","addr":"172.21.15.189:6815","nonce":4075247068}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6812","nonce":4075247068},{"type":"v1","addr":"172.21.15.189:6813","nonce":4075247068}]},"public_addr":"172.21.15.189:6809/4075247068","cluster_addr":"172.21.15.189:6811/4075247068","heartbeat_back_addr":"172.21.15.189:6815/4075247068","heartbeat_front_addr":"172.21.15.189:6813/4075247068","state":["exists","up"]},{"osd":5,"uuid":"91970976-2193-4224-81c0-dc96cb022c85","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":39,"up_thru":54,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6808","nonce":2920363402},{"type":"v1","addr":"172.21.15.193:6809","nonce":2920363402}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6810","nonce":2920363402},{"type":"v1","addr":"172.21.15.193:6811","nonce":2920363402}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6814","nonce":2920363402},{"type":"v1","addr":"172.21.15.193:6815","nonce":2920363402}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6812","nonce":2920363402},{"type":"v1","addr":"172.21.15.193:6813","nonce":2920363402}]},"public_addr":"172.21.15.193:6809/2920363402","cluster_addr":"172.21.15.193:6811/2920363402","heartbeat_back_addr":"172.21.15.193:6815/2920363402","heartbeat_front_addr":"172.21.15.193:6813/2920363402","state":["exists","up"]},{"osd":6,"uuid":"0b77b196-8674-41c6-ad3d-726b5d01c8fd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":47,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6818","nonce":431354576},{"type":"v1","addr":"172.21.15.118:6819","nonce":431354576}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6820","nonce":431354576},{"type":"v1","addr":"172.21.15.118:6821","nonce":431354576}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6824","nonce":431354576},{"type":"v1","addr":"172.21.15.118:6825","nonce":431354576}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6822","nonce":431354576},{"type":"v1","addr":"172.21.15.118:6823","nonce":431354576}]},"public_addr":"172.21.15.118:6819/431354576","cluster_addr":"172.21.15.118:6821/431354576","heartbeat_back_addr":"172.21.15.118:6825/431354576","heartbeat_front_addr":"172.21.15.118:6823/431354576","state":["exists","up"]},{"osd":7,"uuid":"1fbbd38f-4e8e-4d4c-8e98-66f6862bc4bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":52,"up_thru":76,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6816","nonce":3667611284},{"type":"v1","addr":"172.21.15.189:6817","nonce":3667611284}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6818","nonce":3667611284},{"type":"v1","addr":"172.21.15.189:6819","nonce":3667611284}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6822","nonce":3667611284},{"type":"v1","addr":"172.21.15.189:6823","nonce":3667611284}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6820","nonce":3667611284},{"type":"v1","addr":"172.21.15.189:6821","nonce":3667611284}]},"public_addr":"172.21.15.189:6817/3667611284","cluster_addr":"172.21.15.189:6819/3667611284","heartbeat_back_addr":"172.21.15.189:6823/3667611284","heartbeat_front_addr":"172.21.15.189:6821/3667611284","state":["exists","up"]},{"osd":8,"uuid":"6462c2b5-e4b3-4b0b-bcfa-4de2c89e93e0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":60,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6816","nonce":3853800910},{"type":"v1","addr":"172.21.15.193:6817","nonce":3853800910}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6818","nonce":3853800910},{"type":"v1","addr":"172.21.15.193:6819","nonce":3853800910}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6822","nonce":3853800910},{"type":"v1","addr":"172.21.15.193:6823","nonce":3853800910}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6820","nonce":3853800910},{"type":"v1","addr":"172.21.15.193:6821","nonce":3853800910}]},"public_addr":"172.21.15.193:6817/3853800910","cluster_addr":"172.21.15.193:6819/3853800910","heartbeat_back_addr":"172.21.15.193:6823/3853800910","heartbeat_front_addr":"172.21.15.193:6821/3853800910","state":["exists","up"]},{"osd":9,"uuid":"54be277f-3596-422f-9de5-9005a8571c6f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":68,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6826","nonce":3248293936},{"type":"v1","addr":"172.21.15.118:6827","nonce":3248293936}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6828","nonce":3248293936},{"type":"v1","addr":"172.21.15.118:6829","nonce":3248293936}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6832","nonce":3248293936},{"type":"v1","addr":"172.21.15.118:6833","nonce":3248293936}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6830","nonce":3248293936},{"type":"v1","addr":"172.21.15.118:6831","nonce":3248293936}]},"public_addr":"172.21.15.118:6827/3248293936","cluster_addr":"172.21.15.118:6829/3248293936","heartbeat_back_addr":"172.21.15.118:6833/3248293936","heartbeat_front_addr":"172.21.15.118:6831/3248293936","state":["exists","up"]},{"osd":10,"uuid":"7ba69d69-ac41-44ce-bc39-c049a8ca5900","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":73,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6824","nonce":2112195760},{"type":"v1","addr":"172.21.15.189:6825","nonce":2112195760}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6826","nonce":2112195760},{"type":"v1","addr":"172.21.15.189:6827","nonce":2112195760}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6830","nonce":2112195760},{"type":"v1","addr":"172.21.15.189:6831","nonce":2112195760}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6828","nonce":2112195760},{"type":"v1","addr":"172.21.15.189:6829","nonce":2112195760}]},"public_addr":"172.21.15.189:6825/2112195760","cluster_addr":"172.21.15.189:6827/2112195760","heartbeat_back_addr":"172.21.15.189:6831/2112195760","heartbeat_front_addr":"172.21.15.189:6829/2112195760","state":["exists","up"]},{"osd":11,"uuid":"186f0b18-6b22-4eae-83b7-b21bbcce0817","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":81,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6824","nonce":3880515510},{"type":"v1","addr":"172.21.15.193:6825","nonce":3880515510}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6826","nonce":3880515510},{"type":"v1","addr":"172.21.15.193:6827","nonce":3880515510}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6830","nonce":3880515510},{"type":"v1","addr":"172.21.15.193:6831","nonce":3880515510}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6828","nonce":3880515510},{"type":"v1","addr":"172.21.15.193:6829","nonce":3880515510}]},"public_addr":"172.21.15.193:6825/3880515510","cluster_addr":"172.21.15.193:6827/3880515510","heartbeat_back_addr":"172.21.15.193:6831/3880515510","heartbeat_front_addr":"172.21.15.193:6829/3880515510","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:06:58.105543+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:07:23.795866+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:07:47.426050+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:08:19.427073+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:08:45.816082+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:09:11.081523+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:09:44.485077+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:10:11.338972+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:10:36.234024+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:11:09.730432+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:11:37.059083+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:12:02.371947+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.118:6801/389249272":"2024-03-25T00:05:05.458796+0000","172.21.15.118:6800/389249272":"2024-03-25T00:05:05.458796+0000","172.21.15.118:0/2101854694":"2024-03-25T00:05:05.458796+0000","172.21.15.118:0/630779359":"2024-03-25T00:05:05.458796+0000","172.21.15.118:0/2158921029":"2024-03-25T00:04:31.283725+0000","172.21.15.118:0/1003670455":"2024-03-25T00:05:05.458796+0000","172.21.15.118:0/368712314":"2024-03-25T00:04:31.283725+0000","172.21.15.118:0/2004775597":"2024-03-25T00:04:31.283725+0000","172.21.15.118:6800/1463147955":"2024-03-25T00:04:31.283725+0000","172.21.15.118:6801/1463147955":"2024-03-25T00:04:31.283725+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-24T00:12:58.092 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:58.092+0000 7f31bc400640 1 -- 172.21.15.118:0/774815908 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7f319c0a2800 msgr2=0x7f319c0a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:12:58.092 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:58.092+0000 7f31bc400640 1 --2- 172.21.15.118:0/774815908 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7f319c0a2800 0x7f319c0a4cc0 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7f31b419b0e0 tx=0x7f31a0005f90 comp rx=0 tx=0).stop 2024-03-24T00:12:58.092 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:58.092+0000 7f31bc400640 1 -- 172.21.15.118:0/774815908 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f31b41038c0 msgr2=0x7f31b4199bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:12:58.092 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:58.092+0000 7f31bc400640 1 --2- 172.21.15.118:0/774815908 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f31b41038c0 0x7f31b4199bd0 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7f31a8000c00 tx=0x7f31a8005d50 comp rx=0 tx=0).stop 2024-03-24T00:12:58.093 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:58.092+0000 7f31bc400640 1 -- 172.21.15.118:0/774815908 shutdown_connections 2024-03-24T00:12:58.093 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:58.092+0000 7f31bc400640 1 --2- 172.21.15.118:0/774815908 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7f319c0a2800 0x7f319c0a4cc0 unknown :-1 s=CLOSED pgs=156 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:12:58.093 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:58.092+0000 7f31bc400640 1 --2- 172.21.15.118:0/774815908 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f31b4105470 0x7f31b41a1180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:12:58.093 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:58.092+0000 7f31bc400640 1 --2- 172.21.15.118:0/774815908 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f31b4104ab0 0x7f31b419a110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:12:58.093 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:58.092+0000 7f31bc400640 1 --2- 172.21.15.118:0/774815908 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f31b41038c0 0x7f31b4199bd0 unknown :-1 s=CLOSED pgs=156 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:12:58.093 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:58.092+0000 7f31bc400640 1 -- 172.21.15.118:0/774815908 >> 172.21.15.118:0/774815908 conn(0x7f31b40fee50 msgr2=0x7f31b4107e80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:12:58.093 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:58.093+0000 7f31bc400640 1 -- 172.21.15.118:0/774815908 shutdown_connections 2024-03-24T00:12:58.093 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:58.093+0000 7f31bc400640 1 -- 172.21.15.118:0/774815908 wait complete. 2024-03-24T00:12:58.111 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-24T00:07:50.190965+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-24T00:12:58.111 DEBUG:teuthology.orchestra.run.smithi118:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-24T00:12:58.316 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:58.316+0000 7f319be00640 1 -- 172.21.15.118:0/3372886733 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f319c0756e0 msgr2=0x7f319c073b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:12:58.316 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:58.316+0000 7f319be00640 1 --2- 172.21.15.118:0/3372886733 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f319c0756e0 0x7f319c073b30 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f318c009900 tx=0x7f318c02bc30 comp rx=0 tx=0).stop 2024-03-24T00:12:58.316 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:58.316+0000 7f319be00640 1 -- 172.21.15.118:0/3372886733 shutdown_connections 2024-03-24T00:12:58.316 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:58.316+0000 7f319be00640 1 --2- 172.21.15.118:0/3372886733 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f319c109460 0x7f319c10b8f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:12:58.316 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:58.316+0000 7f319be00640 1 --2- 172.21.15.118:0/3372886733 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f319c074100 0x7f319c074580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:12:58.317 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:58.316+0000 7f319be00640 1 --2- 172.21.15.118:0/3372886733 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f319c0756e0 0x7f319c073b30 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:12:58.317 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:58.316+0000 7f319be00640 1 -- 172.21.15.118:0/3372886733 >> 172.21.15.118:0/3372886733 conn(0x7f319c0fcb10 msgr2=0x7f319c0fef50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:12:58.317 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:58.316+0000 7f319be00640 1 -- 172.21.15.118:0/3372886733 shutdown_connections 2024-03-24T00:12:58.317 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:58.316+0000 7f319be00640 1 -- 172.21.15.118:0/3372886733 wait complete. 2024-03-24T00:12:58.319 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:58.317+0000 7f319be00640 1 Processor -- start 2024-03-24T00:12:58.319 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:58.317+0000 7f319be00640 1 -- start start 2024-03-24T00:12:58.319 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:58.318+0000 7f319be00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f319c074100 0x7f319c199a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:12:58.319 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:58.318+0000 7f319be00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f319c0756e0 0x7f319c199fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:12:58.319 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:58.318+0000 7f319be00640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f319c109460 0x7f319c1a1020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:12:58.319 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:58.318+0000 7f319be00640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f319c10f450 con 0x7f319c109460 2024-03-24T00:12:58.319 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:58.318+0000 7f319be00640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f319c10f2d0 con 0x7f319c074100 2024-03-24T00:12:58.319 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:58.318+0000 7f319be00640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f319c10f5d0 con 0x7f319c0756e0 2024-03-24T00:12:58.319 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:58.318+0000 7f319b400640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f319c109460 0x7f319c1a1020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:12:58.319 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:58.318+0000 7f319aa00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f319c074100 0x7f319c199a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:12:58.319 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:58.318+0000 7f319a000640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f319c0756e0 0x7f319c199fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:12:58.320 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:58.318+0000 7f319aa00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f319c074100 0x7f319c199a70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.118:50190/0 (socket says 172.21.15.118:50190) 2024-03-24T00:12:58.320 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:58.318+0000 7f319aa00640 1 -- 172.21.15.118:0/1226160420 learned_addr learned my addr 172.21.15.118:0/1226160420 (peer_addr_for_me v2:172.21.15.118:0/0) 2024-03-24T00:12:58.320 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:58.319+0000 7f319aa00640 1 -- 172.21.15.118:0/1226160420 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f319c0756e0 msgr2=0x7f319c199fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:12:58.320 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:58.319+0000 7f319aa00640 1 --2- 172.21.15.118:0/1226160420 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f319c0756e0 0x7f319c199fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:12:58.320 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:58.319+0000 7f319aa00640 1 -- 172.21.15.118:0/1226160420 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f319c109460 msgr2=0x7f319c1a1020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:12:58.320 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:58.319+0000 7f319aa00640 1 --2- 172.21.15.118:0/1226160420 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f319c109460 0x7f319c1a1020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:12:58.320 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:58.319+0000 7f319aa00640 1 -- 172.21.15.118:0/1226160420 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f319c1a1690 con 0x7f319c074100 2024-03-24T00:12:58.320 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:58.319+0000 7f319a000640 1 --2- 172.21.15.118:0/1226160420 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f319c0756e0 0x7f319c199fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T00:12:58.320 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:58.319+0000 7f319aa00640 1 --2- 172.21.15.118:0/1226160420 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f319c074100 0x7f319c199a70 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f318c000c00 tx=0x7f318c02c650 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:12:58.320 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:58.319+0000 7f3193400640 1 -- 172.21.15.118:0/1226160420 <== mon.2 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f318c043020 con 0x7f319c074100 2024-03-24T00:12:58.320 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:58.319+0000 7f3193400640 1 -- 172.21.15.118:0/1226160420 <== mon.2 v2:172.21.15.189:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f318c0054e0 con 0x7f319c074100 2024-03-24T00:12:58.320 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:58.319+0000 7f319be00640 1 -- 172.21.15.118:0/1226160420 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f319c1a1920 con 0x7f319c074100 2024-03-24T00:12:58.320 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:58.319+0000 7f3193400640 1 -- 172.21.15.118:0/1226160420 <== mon.2 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f318c003c40 con 0x7f319c074100 2024-03-24T00:12:58.320 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:58.319+0000 7f319be00640 1 -- 172.21.15.118:0/1226160420 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f319c1a1e70 con 0x7f319c074100 2024-03-24T00:12:58.321 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:58.321+0000 7f319be00640 1 -- 172.21.15.118:0/1226160420 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f319c0683b0 con 0x7f319c074100 2024-03-24T00:12:58.322 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:58.322+0000 7f3193400640 1 -- 172.21.15.118:0/1226160420 <== mon.2 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 16) v1 ==== 137291+0+0 (secure 0 0 0) 0x7f318c046020 con 0x7f319c074100 2024-03-24T00:12:58.323 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:58.323+0000 7f3193400640 1 --2- 172.21.15.118:0/1226160420 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7f31800a2800 0x7f31800a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:12:58.323 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:58.324+0000 7f3193400640 1 -- 172.21.15.118:0/1226160420 <== mon.2 v2:172.21.15.189:3300/0 5 ==== osd_map(82..82 src has 1..82) v4 ==== 7127+0+0 (secure 0 0 0) 0x7f318c0f06d0 con 0x7f319c074100 2024-03-24T00:12:58.323 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:58.324+0000 7f319a000640 1 --2- 172.21.15.118:0/1226160420 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7f31800a2800 0x7f31800a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:12:58.324 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:58.324+0000 7f319a000640 1 --2- 172.21.15.118:0/1226160420 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7f31800a2800 0x7f31800a4cc0 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f319c19af80 tx=0x7f3188005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:12:58.327 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:58.327+0000 7f3193400640 1 -- 172.21.15.118:0/1226160420 <== mon.2 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f318c04d020 con 0x7f319c074100 2024-03-24T00:12:58.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:12:58 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T00:12:58.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:12:58 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T00:12:58.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:12:58 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T00:12:58.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:12:58 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T00:12:58.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:12:58 smithi118 ceph-mon[31535]: pgmap v308: 1 pgs: 1 active+clean; 577 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-24T00:12:58.447 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:12:58 smithi118 ceph-mon[31535]: from='client.? 172.21.15.118:0/723484770' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-24T00:12:58.447 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:12:58 smithi118 ceph-mon[31535]: from='client.? 172.21.15.118:0/774815908' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-24T00:12:58.453 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:12:58 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T00:12:58.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:12:58 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T00:12:58.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:12:58 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T00:12:58.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:12:58 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T00:12:58.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:12:58 smithi189 ceph-mon[36187]: pgmap v308: 1 pgs: 1 active+clean; 577 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-24T00:12:58.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:12:58 smithi189 ceph-mon[36187]: from='client.? 172.21.15.118:0/723484770' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-24T00:12:58.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:12:58 smithi189 ceph-mon[36187]: from='client.? 172.21.15.118:0/774815908' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-24T00:12:58.460 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:12:58 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T00:12:58.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:12:58 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T00:12:58.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:12:58 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T00:12:58.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:12:58 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T00:12:58.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:12:58 smithi193 ceph-mon[36878]: pgmap v308: 1 pgs: 1 active+clean; 577 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-24T00:12:58.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:12:58 smithi193 ceph-mon[36878]: from='client.? 172.21.15.118:0/723484770' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-24T00:12:58.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:12:58 smithi193 ceph-mon[36878]: from='client.? 172.21.15.118:0/774815908' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-24T00:12:58.534 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:58.534+0000 7f319be00640 1 -- 172.21.15.118:0/1226160420 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f319c0622a0 con 0x7f319c074100 2024-03-24T00:12:58.534 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:58.535+0000 7f3193400640 1 -- 172.21.15.118:0/1226160420 <== mon.2 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v82) v1 ==== 93+0+10 (secure 0 0 0) 0x7f318c0bdb70 con 0x7f319c074100 2024-03-24T00:12:58.535 INFO:teuthology.orchestra.run.smithi118.stdout:pg_num: 1 2024-03-24T00:12:58.537 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:58.537+0000 7f319be00640 1 -- 172.21.15.118:0/1226160420 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7f31800a2800 msgr2=0x7f31800a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:12:58.537 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:58.537+0000 7f319be00640 1 --2- 172.21.15.118:0/1226160420 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7f31800a2800 0x7f31800a4cc0 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f319c19af80 tx=0x7f3188005f90 comp rx=0 tx=0).stop 2024-03-24T00:12:58.537 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:58.537+0000 7f319be00640 1 -- 172.21.15.118:0/1226160420 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f319c074100 msgr2=0x7f319c199a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:12:58.537 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:58.537+0000 7f319be00640 1 --2- 172.21.15.118:0/1226160420 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f319c074100 0x7f319c199a70 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f318c000c00 tx=0x7f318c02c650 comp rx=0 tx=0).stop 2024-03-24T00:12:58.537 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:58.537+0000 7f319be00640 1 -- 172.21.15.118:0/1226160420 shutdown_connections 2024-03-24T00:12:58.538 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:58.537+0000 7f319be00640 1 --2- 172.21.15.118:0/1226160420 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7f31800a2800 0x7f31800a4cc0 unknown :-1 s=CLOSED pgs=157 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:12:58.538 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:58.537+0000 7f319be00640 1 --2- 172.21.15.118:0/1226160420 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f319c109460 0x7f319c1a1020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:12:58.538 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:58.537+0000 7f319be00640 1 --2- 172.21.15.118:0/1226160420 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f319c0756e0 0x7f319c199fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:12:58.538 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:58.537+0000 7f319be00640 1 --2- 172.21.15.118:0/1226160420 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f319c074100 0x7f319c199a70 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:12:58.538 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:58.537+0000 7f319be00640 1 -- 172.21.15.118:0/1226160420 >> 172.21.15.118:0/1226160420 conn(0x7f319c0fcb10 msgr2=0x7f319c105360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:12:58.538 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:58.537+0000 7f319be00640 1 -- 172.21.15.118:0/1226160420 shutdown_connections 2024-03-24T00:12:58.538 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:58.537+0000 7f319be00640 1 -- 172.21.15.118:0/1226160420 wait complete. 2024-03-24T00:12:58.556 DEBUG:teuthology.orchestra.run.smithi118:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2024-03-24T00:12:58.739 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:58.739+0000 7f0a93800640 1 -- 172.21.15.118:0/282568865 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f0a8c1021c0 msgr2=0x7f0a8c1045b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:12:58.739 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:58.739+0000 7f0a93800640 1 --2- 172.21.15.118:0/282568865 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f0a8c1021c0 0x7f0a8c1045b0 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f0a80009900 tx=0x7f0a8002bc30 comp rx=0 tx=0).stop 2024-03-24T00:12:58.739 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:58.739+0000 7f0a93800640 1 -- 172.21.15.118:0/282568865 shutdown_connections 2024-03-24T00:12:58.739 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:58.739+0000 7f0a93800640 1 --2- 172.21.15.118:0/282568865 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f0a8c107420 0x7f0a8c109810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:12:58.740 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:58.739+0000 7f0a93800640 1 --2- 172.21.15.118:0/282568865 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0a8c104af0 0x7f0a8c106ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:12:58.740 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:58.739+0000 7f0a93800640 1 --2- 172.21.15.118:0/282568865 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f0a8c1021c0 0x7f0a8c1045b0 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:12:58.740 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:58.739+0000 7f0a93800640 1 -- 172.21.15.118:0/282568865 >> 172.21.15.118:0/282568865 conn(0x7f0a8c0fbb30 msgr2=0x7f0a8c0fdf70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:12:58.740 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:58.740+0000 7f0a93800640 1 -- 172.21.15.118:0/282568865 shutdown_connections 2024-03-24T00:12:58.740 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:58.740+0000 7f0a93800640 1 -- 172.21.15.118:0/282568865 wait complete. 2024-03-24T00:12:58.741 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:58.741+0000 7f0a93800640 1 Processor -- start 2024-03-24T00:12:58.741 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:58.741+0000 7f0a93800640 1 -- start start 2024-03-24T00:12:58.742 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:58.741+0000 7f0a93800640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0a8c1021c0 0x7f0a8c199bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:12:58.742 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:58.741+0000 7f0a93800640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f0a8c104af0 0x7f0a8c19a110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:12:58.742 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:58.741+0000 7f0a93800640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f0a8c107420 0x7f0a8c1a1180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:12:58.742 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:58.741+0000 7f0a93800640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0a8c10d380 con 0x7f0a8c107420 2024-03-24T00:12:58.742 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:58.741+0000 7f0a93800640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0a8c10d200 con 0x7f0a8c1021c0 2024-03-24T00:12:58.742 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:58.741+0000 7f0a93800640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0a8c10d500 con 0x7f0a8c104af0 2024-03-24T00:12:58.742 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:58.741+0000 7f0a90e00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0a8c1021c0 0x7f0a8c199bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:12:58.742 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:58.741+0000 7f0a90e00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0a8c1021c0 0x7f0a8c199bd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.118:50196/0 (socket says 172.21.15.118:50196) 2024-03-24T00:12:58.742 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:58.741+0000 7f0a8be00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f0a8c104af0 0x7f0a8c19a110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:12:58.742 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:58.741+0000 7f0a90e00640 1 -- 172.21.15.118:0/3703457155 learned_addr learned my addr 172.21.15.118:0/3703457155 (peer_addr_for_me v2:172.21.15.118:0/0) 2024-03-24T00:12:58.742 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:58.741+0000 7f0a91800640 1 --2- 172.21.15.118:0/3703457155 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f0a8c107420 0x7f0a8c1a1180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:12:58.742 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:58.742+0000 7f0a90e00640 1 -- 172.21.15.118:0/3703457155 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f0a8c104af0 msgr2=0x7f0a8c19a110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:12:58.742 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:58.742+0000 7f0a90e00640 1 --2- 172.21.15.118:0/3703457155 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f0a8c104af0 0x7f0a8c19a110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:12:58.743 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:58.742+0000 7f0a90e00640 1 -- 172.21.15.118:0/3703457155 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f0a8c107420 msgr2=0x7f0a8c1a1180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:12:58.743 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:58.742+0000 7f0a90e00640 1 --2- 172.21.15.118:0/3703457155 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f0a8c107420 0x7f0a8c1a1180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:12:58.743 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:58.742+0000 7f0a90e00640 1 -- 172.21.15.118:0/3703457155 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0a8c1a17f0 con 0x7f0a8c1021c0 2024-03-24T00:12:58.743 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:58.742+0000 7f0a8be00640 1 --2- 172.21.15.118:0/3703457155 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f0a8c104af0 0x7f0a8c19a110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T00:12:58.743 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:58.742+0000 7f0a90e00640 1 --2- 172.21.15.118:0/3703457155 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0a8c1021c0 0x7f0a8c199bd0 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f0a80000c00 tx=0x7f0a8002c650 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:12:58.744 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:58.742+0000 7f0a89600640 1 -- 172.21.15.118:0/3703457155 <== mon.2 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0a80043020 con 0x7f0a8c1021c0 2024-03-24T00:12:58.744 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:58.742+0000 7f0a89600640 1 -- 172.21.15.118:0/3703457155 <== mon.2 v2:172.21.15.189:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f0a800054e0 con 0x7f0a8c1021c0 2024-03-24T00:12:58.744 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:58.743+0000 7f0a89600640 1 -- 172.21.15.118:0/3703457155 <== mon.2 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0a80003c40 con 0x7f0a8c1021c0 2024-03-24T00:12:58.744 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:58.743+0000 7f0a93800640 1 -- 172.21.15.118:0/3703457155 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0a8c1a1a80 con 0x7f0a8c1021c0 2024-03-24T00:12:58.744 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:58.743+0000 7f0a93800640 1 -- 172.21.15.118:0/3703457155 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0a8c1a1f80 con 0x7f0a8c1021c0 2024-03-24T00:12:58.744 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:58.744+0000 7f0a93800640 1 -- 172.21.15.118:0/3703457155 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0a8c108f10 con 0x7f0a8c1021c0 2024-03-24T00:12:58.745 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:58.746+0000 7f0a89600640 1 -- 172.21.15.118:0/3703457155 <== mon.2 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 16) v1 ==== 137291+0+0 (secure 0 0 0) 0x7f0a80046020 con 0x7f0a8c1021c0 2024-03-24T00:12:58.747 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:58.747+0000 7f0a89600640 1 --2- 172.21.15.118:0/3703457155 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7f0a740a2800 0x7f0a740a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:12:58.747 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:58.747+0000 7f0a89600640 1 -- 172.21.15.118:0/3703457155 <== mon.2 v2:172.21.15.189:3300/0 5 ==== osd_map(82..82 src has 1..82) v4 ==== 7127+0+0 (secure 0 0 0) 0x7f0a800f0620 con 0x7f0a8c1021c0 2024-03-24T00:12:58.747 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:58.747+0000 7f0a8be00640 1 --2- 172.21.15.118:0/3703457155 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7f0a740a2800 0x7f0a740a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:12:58.747 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:58.748+0000 7f0a8be00640 1 --2- 172.21.15.118:0/3703457155 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7f0a740a2800 0x7f0a740a4cc0 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f0a8c19b0e0 tx=0x7f0a78005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:12:58.751 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:58.751+0000 7f0a89600640 1 -- 172.21.15.118:0/3703457155 <== mon.2 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f0a8004d020 con 0x7f0a8c1021c0 2024-03-24T00:12:58.954 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:58.955+0000 7f0a93800640 1 -- 172.21.15.118:0/3703457155 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) v1 -- 0x7f0a8c02d2c0 con 0x7f0a8c1021c0 2024-03-24T00:12:58.955 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:58.955+0000 7f0a89600640 1 -- 172.21.15.118:0/3703457155 <== mon.2 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v82) v1 ==== 84+0+65 (secure 0 0 0) 0x7f0a800bdb40 con 0x7f0a8c1021c0 2024-03-24T00:12:58.955 INFO:teuthology.orchestra.run.smithi118.stdout: 2024-03-24T00:12:58.955 INFO:teuthology.orchestra.run.smithi118.stdout:[ 2024-03-24T00:12:58.955 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-24T00:12:58.955 INFO:teuthology.orchestra.run.smithi118.stdout: "poolnum": 1, 2024-03-24T00:12:58.955 INFO:teuthology.orchestra.run.smithi118.stdout: "poolname": ".mgr" 2024-03-24T00:12:58.955 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-03-24T00:12:58.955 INFO:teuthology.orchestra.run.smithi118.stdout:] 2024-03-24T00:12:58.958 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:58.958+0000 7f0a93800640 1 -- 172.21.15.118:0/3703457155 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7f0a740a2800 msgr2=0x7f0a740a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:12:58.958 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:58.958+0000 7f0a93800640 1 --2- 172.21.15.118:0/3703457155 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7f0a740a2800 0x7f0a740a4cc0 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f0a8c19b0e0 tx=0x7f0a78005f90 comp rx=0 tx=0).stop 2024-03-24T00:12:58.958 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:58.958+0000 7f0a93800640 1 -- 172.21.15.118:0/3703457155 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0a8c1021c0 msgr2=0x7f0a8c199bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:12:58.958 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:58.958+0000 7f0a93800640 1 --2- 172.21.15.118:0/3703457155 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0a8c1021c0 0x7f0a8c199bd0 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f0a80000c00 tx=0x7f0a8002c650 comp rx=0 tx=0).stop 2024-03-24T00:12:58.958 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:58.958+0000 7f0a93800640 1 -- 172.21.15.118:0/3703457155 shutdown_connections 2024-03-24T00:12:58.958 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:58.958+0000 7f0a93800640 1 --2- 172.21.15.118:0/3703457155 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7f0a740a2800 0x7f0a740a4cc0 unknown :-1 s=CLOSED pgs=158 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:12:58.958 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:58.958+0000 7f0a93800640 1 --2- 172.21.15.118:0/3703457155 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f0a8c107420 0x7f0a8c1a1180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:12:58.959 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:58.958+0000 7f0a93800640 1 --2- 172.21.15.118:0/3703457155 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f0a8c104af0 0x7f0a8c19a110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:12:58.959 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:58.958+0000 7f0a93800640 1 --2- 172.21.15.118:0/3703457155 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0a8c1021c0 0x7f0a8c199bd0 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:12:58.959 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:58.958+0000 7f0a93800640 1 -- 172.21.15.118:0/3703457155 >> 172.21.15.118:0/3703457155 conn(0x7f0a8c0fbb30 msgr2=0x7f0a8c1018b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:12:58.959 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:58.958+0000 7f0a93800640 1 -- 172.21.15.118:0/3703457155 shutdown_connections 2024-03-24T00:12:58.959 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:58.959+0000 7f0a93800640 1 -- 172.21.15.118:0/3703457155 wait complete. 2024-03-24T00:12:58.976 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2024-03-24T00:12:58.976 DEBUG:teuthology.orchestra.run.smithi118:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_metadata --pg_num_min 64 2024-03-24T00:12:59.181 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:59.181+0000 7fd16a200640 1 -- 172.21.15.118:0/3718000591 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fd164104ad0 msgr2=0x7fd164104f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:12:59.182 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:59.181+0000 7fd16a200640 1 --2- 172.21.15.118:0/3718000591 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fd164104ad0 0x7fd164104f50 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7fd150009900 tx=0x7fd15002bc30 comp rx=0 tx=0).stop 2024-03-24T00:12:59.182 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:59.181+0000 7fd16a200640 1 -- 172.21.15.118:0/3718000591 shutdown_connections 2024-03-24T00:12:59.182 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:59.181+0000 7fd16a200640 1 --2- 172.21.15.118:0/3718000591 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd164105490 0x7fd16410bd20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:12:59.182 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:59.181+0000 7fd16a200640 1 --2- 172.21.15.118:0/3718000591 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fd164104ad0 0x7fd164104f50 unknown :-1 s=CLOSED pgs=157 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:12:59.182 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:59.181+0000 7fd16a200640 1 --2- 172.21.15.118:0/3718000591 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fd1641038e0 0x7fd164103d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:12:59.182 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:59.181+0000 7fd16a200640 1 -- 172.21.15.118:0/3718000591 >> 172.21.15.118:0/3718000591 conn(0x7fd1640fee50 msgr2=0x7fd164101290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:12:59.182 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:59.182+0000 7fd16a200640 1 -- 172.21.15.118:0/3718000591 shutdown_connections 2024-03-24T00:12:59.182 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:59.182+0000 7fd16a200640 1 -- 172.21.15.118:0/3718000591 wait complete. 2024-03-24T00:12:59.184 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:59.182+0000 7fd16a200640 1 Processor -- start 2024-03-24T00:12:59.184 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:59.183+0000 7fd16a200640 1 -- start start 2024-03-24T00:12:59.184 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:59.183+0000 7fd16a200640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fd1641038e0 0x7fd164199bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:12:59.185 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:59.183+0000 7fd16a200640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd164104ad0 0x7fd16419a130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:12:59.185 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:59.183+0000 7fd16a200640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fd164105490 0x7fd1641a11a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:12:59.185 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:59.183+0000 7fd16a200640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd16410f5b0 con 0x7fd1641038e0 2024-03-24T00:12:59.185 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:59.183+0000 7fd16a200640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd16410f430 con 0x7fd164104ad0 2024-03-24T00:12:59.185 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:59.183+0000 7fd16a200640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd16410f730 con 0x7fd164105490 2024-03-24T00:12:59.185 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:59.183+0000 7fd163400640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fd1641038e0 0x7fd164199bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:12:59.185 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:59.183+0000 7fd163e00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fd164105490 0x7fd1641a11a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:12:59.185 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:59.183+0000 7fd163400640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fd1641038e0 0x7fd164199bf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.118:3300/0 says I am v2:172.21.15.118:50166/0 (socket says 172.21.15.118:50166) 2024-03-24T00:12:59.185 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:59.183+0000 7fd163400640 1 -- 172.21.15.118:0/990972799 learned_addr learned my addr 172.21.15.118:0/990972799 (peer_addr_for_me v2:172.21.15.118:0/0) 2024-03-24T00:12:59.185 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:59.183+0000 7fd162a00640 1 --2- 172.21.15.118:0/990972799 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd164104ad0 0x7fd16419a130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:12:59.185 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:59.184+0000 7fd163e00640 1 -- 172.21.15.118:0/990972799 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd164104ad0 msgr2=0x7fd16419a130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:12:59.185 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:59.184+0000 7fd163e00640 1 --2- 172.21.15.118:0/990972799 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd164104ad0 0x7fd16419a130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:12:59.185 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:59.184+0000 7fd163e00640 1 -- 172.21.15.118:0/990972799 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fd1641038e0 msgr2=0x7fd164199bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:12:59.185 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:59.184+0000 7fd163e00640 1 --2- 172.21.15.118:0/990972799 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fd1641038e0 0x7fd164199bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:12:59.186 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:59.184+0000 7fd163e00640 1 -- 172.21.15.118:0/990972799 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd1641a1810 con 0x7fd164105490 2024-03-24T00:12:59.186 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:59.184+0000 7fd162a00640 1 --2- 172.21.15.118:0/990972799 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd164104ad0 0x7fd16419a130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T00:12:59.186 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:59.184+0000 7fd163e00640 1 --2- 172.21.15.118:0/990972799 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fd164105490 0x7fd1641a11a0 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7fd15c009fd0 tx=0x7fd15c00ef10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:12:59.186 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:59.184+0000 7fd15be00640 1 -- 172.21.15.118:0/990972799 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd15c00ce20 con 0x7fd164105490 2024-03-24T00:12:59.186 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:59.184+0000 7fd15be00640 1 -- 172.21.15.118:0/990972799 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fd15c005020 con 0x7fd164105490 2024-03-24T00:12:59.186 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:59.184+0000 7fd16a200640 1 -- 172.21.15.118:0/990972799 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd1641a1b00 con 0x7fd164105490 2024-03-24T00:12:59.186 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:59.184+0000 7fd15be00640 1 -- 172.21.15.118:0/990972799 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd15c010720 con 0x7fd164105490 2024-03-24T00:12:59.186 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:59.184+0000 7fd16a200640 1 -- 172.21.15.118:0/990972799 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd1641a2020 con 0x7fd164105490 2024-03-24T00:12:59.186 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:59.185+0000 7fd16a200640 1 -- 172.21.15.118:0/990972799 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd16410acd0 con 0x7fd164105490 2024-03-24T00:12:59.187 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:59.187+0000 7fd15be00640 1 -- 172.21.15.118:0/990972799 <== mon.1 v2:172.21.15.193:3300/0 4 ==== mgrmap(e 16) v1 ==== 137291+0+0 (secure 0 0 0) 0x7fd15c005540 con 0x7fd164105490 2024-03-24T00:12:59.188 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:59.188+0000 7fd15be00640 1 --2- 172.21.15.118:0/990972799 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7fd1480a2880 0x7fd1480a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:12:59.188 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:59.188+0000 7fd15be00640 1 -- 172.21.15.118:0/990972799 <== mon.1 v2:172.21.15.193:3300/0 5 ==== osd_map(82..82 src has 1..82) v4 ==== 7127+0+0 (secure 0 0 0) 0x7fd15c0cecd0 con 0x7fd164105490 2024-03-24T00:12:59.188 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:59.189+0000 7fd163400640 1 --2- 172.21.15.118:0/990972799 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7fd1480a2880 0x7fd1480a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:12:59.188 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:59.189+0000 7fd163400640 1 --2- 172.21.15.118:0/990972799 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7fd1480a2880 0x7fd1480a4d40 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7fd1540058c0 tx=0x7fd154005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:12:59.192 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:59.192+0000 7fd15be00640 1 -- 172.21.15.118:0/990972799 <== mon.1 v2:172.21.15.193:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7fd15c09c1f0 con 0x7fd164105490 2024-03-24T00:12:59.431 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:12:59.431+0000 7fd16a200640 1 -- 172.21.15.118:0/990972799 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64} v 0) v1 -- 0x7fd16402d070 con 0x7fd164105490 2024-03-24T00:12:59.445 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:12:59 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T00:12:59.445 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:12:59 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T00:12:59.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:12:59 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T00:12:59.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:12:59 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T00:12:59.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:12:59 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T00:12:59.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:12:59 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T00:12:59.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:12:59 smithi118 ceph-mon[31535]: from='client.? 172.21.15.118:0/1226160420' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-24T00:12:59.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:12:59 smithi118 ceph-mon[31535]: from='client.? 172.21.15.118:0/3703457155' entity='client.admin' cmd=[{"prefix": "osd lspools", "format": "json-pretty"}]: dispatch 2024-03-24T00:12:59.472 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:12:59 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T00:12:59.472 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:12:59 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T00:12:59.472 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:12:59 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T00:12:59.472 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:12:59 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T00:12:59.472 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:12:59 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T00:12:59.472 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:12:59 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T00:12:59.472 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:12:59 smithi189 ceph-mon[36187]: from='client.? 172.21.15.118:0/1226160420' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-24T00:12:59.473 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:12:59 smithi189 ceph-mon[36187]: from='client.? 172.21.15.118:0/3703457155' entity='client.admin' cmd=[{"prefix": "osd lspools", "format": "json-pretty"}]: dispatch 2024-03-24T00:12:59.562 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:12:59 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T00:12:59.562 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:12:59 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T00:12:59.562 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:12:59 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T00:12:59.562 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:12:59 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T00:12:59.562 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:12:59 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T00:12:59.562 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:12:59 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T00:12:59.562 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:12:59 smithi193 ceph-mon[36878]: from='client.? 172.21.15.118:0/1226160420' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-24T00:12:59.562 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:12:59 smithi193 ceph-mon[36878]: from='client.? 172.21.15.118:0/3703457155' entity='client.admin' cmd=[{"prefix": "osd lspools", "format": "json-pretty"}]: dispatch 2024-03-24T00:13:00.239 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:00.239+0000 7fd15be00640 1 -- 172.21.15.118:0/990972799 <== mon.1 v2:172.21.15.193:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]=0 pool 'cephfs_metadata' created v83) v1 ==== 138+0+0 (secure 0 0 0) 0x7fd15c09baf0 con 0x7fd164105490 2024-03-24T00:13:00.239 INFO:teuthology.orchestra.run.smithi118.stderr:pool 'cephfs_metadata' created 2024-03-24T00:13:00.245 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:00.246+0000 7fd16a200640 1 -- 172.21.15.118:0/990972799 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7fd1480a2880 msgr2=0x7fd1480a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:13:00.245 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:00.246+0000 7fd16a200640 1 --2- 172.21.15.118:0/990972799 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7fd1480a2880 0x7fd1480a4d40 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7fd1540058c0 tx=0x7fd154005f90 comp rx=0 tx=0).stop 2024-03-24T00:13:00.245 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:00.246+0000 7fd16a200640 1 -- 172.21.15.118:0/990972799 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fd164105490 msgr2=0x7fd1641a11a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:13:00.246 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:00.246+0000 7fd16a200640 1 --2- 172.21.15.118:0/990972799 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fd164105490 0x7fd1641a11a0 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7fd15c009fd0 tx=0x7fd15c00ef10 comp rx=0 tx=0).stop 2024-03-24T00:13:00.246 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:00.247+0000 7fd16a200640 1 -- 172.21.15.118:0/990972799 shutdown_connections 2024-03-24T00:13:00.246 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:00.247+0000 7fd16a200640 1 --2- 172.21.15.118:0/990972799 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7fd1480a2880 0x7fd1480a4d40 unknown :-1 s=CLOSED pgs=159 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:13:00.246 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:00.247+0000 7fd16a200640 1 --2- 172.21.15.118:0/990972799 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fd164105490 0x7fd1641a11a0 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:13:00.246 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:00.247+0000 7fd16a200640 1 --2- 172.21.15.118:0/990972799 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd164104ad0 0x7fd16419a130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:13:00.246 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:00.247+0000 7fd16a200640 1 --2- 172.21.15.118:0/990972799 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fd1641038e0 0x7fd164199bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:13:00.246 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:00.247+0000 7fd16a200640 1 -- 172.21.15.118:0/990972799 >> 172.21.15.118:0/990972799 conn(0x7fd1640fee50 msgr2=0x7fd164107ea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:13:00.247 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:00.247+0000 7fd16a200640 1 -- 172.21.15.118:0/990972799 shutdown_connections 2024-03-24T00:13:00.247 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:00.247+0000 7fd16a200640 1 -- 172.21.15.118:0/990972799 wait complete. 2024-03-24T00:13:00.273 DEBUG:teuthology.orchestra.run.smithi118:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data 64 --pg_num_min 64 --target_size_ratio 0.9 2024-03-24T00:13:00.469 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:00.469+0000 7ff3c2000640 1 -- 172.21.15.118:0/2052200319 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7ff3bc1015a0 msgr2=0x7ff3bc1019c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:13:00.469 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:00.469+0000 7ff3c2000640 1 --2- 172.21.15.118:0/2052200319 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7ff3bc1015a0 0x7ff3bc1019c0 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7ff3ac009860 tx=0x7ff3ac02bb70 comp rx=0 tx=0).stop 2024-03-24T00:13:00.469 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:00.469+0000 7ff3c2000640 1 -- 172.21.15.118:0/2052200319 shutdown_connections 2024-03-24T00:13:00.469 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:00.469+0000 7ff3c2000640 1 --2- 172.21.15.118:0/2052200319 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff3bc103150 0x7ff3bc1099e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:13:00.469 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:00.469+0000 7ff3c2000640 1 --2- 172.21.15.118:0/2052200319 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7ff3bc102790 0x7ff3bc102c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:13:00.469 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:00.469+0000 7ff3c2000640 1 --2- 172.21.15.118:0/2052200319 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7ff3bc1015a0 0x7ff3bc1019c0 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:13:00.470 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:00.469+0000 7ff3c2000640 1 -- 172.21.15.118:0/2052200319 >> 172.21.15.118:0/2052200319 conn(0x7ff3bc0fcb10 msgr2=0x7ff3bc0fef50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:13:00.470 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:00.469+0000 7ff3c2000640 1 -- 172.21.15.118:0/2052200319 shutdown_connections 2024-03-24T00:13:00.470 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:00.469+0000 7ff3c2000640 1 -- 172.21.15.118:0/2052200319 wait complete. 2024-03-24T00:13:00.472 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:00.470+0000 7ff3c2000640 1 Processor -- start 2024-03-24T00:13:00.472 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:00.470+0000 7ff3c2000640 1 -- start start 2024-03-24T00:13:00.472 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:00.471+0000 7ff3c2000640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7ff3bc1015a0 0x7ff3bc199ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:13:00.472 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:00.471+0000 7ff3c2000640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7ff3bc102790 0x7ff3bc19a010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:13:00.472 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:00.471+0000 7ff3c2000640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff3bc103150 0x7ff3bc1a1080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:13:00.472 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:00.471+0000 7ff3c2000640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff3bc10d230 con 0x7ff3bc102790 2024-03-24T00:13:00.472 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:00.471+0000 7ff3c2000640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff3bc10d0b0 con 0x7ff3bc103150 2024-03-24T00:13:00.472 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:00.471+0000 7ff3c2000640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff3bc10d3b0 con 0x7ff3bc1015a0 2024-03-24T00:13:00.472 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:00.471+0000 7ff3c0c00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7ff3bc1015a0 0x7ff3bc199ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:13:00.472 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:00.471+0000 7ff3c0c00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7ff3bc1015a0 0x7ff3bc199ad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.118:58870/0 (socket says 172.21.15.118:58870) 2024-03-24T00:13:00.473 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:00.471+0000 7ff3c0c00640 1 -- 172.21.15.118:0/1083336648 learned_addr learned my addr 172.21.15.118:0/1083336648 (peer_addr_for_me v2:172.21.15.118:0/0) 2024-03-24T00:13:00.473 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:00.471+0000 7ff3b3e00640 1 --2- 172.21.15.118:0/1083336648 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7ff3bc102790 0x7ff3bc19a010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:13:00.473 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:00.471+0000 7ff3c1600640 1 --2- 172.21.15.118:0/1083336648 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff3bc103150 0x7ff3bc1a1080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:13:00.473 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:00.471+0000 7ff3c0c00640 1 -- 172.21.15.118:0/1083336648 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff3bc103150 msgr2=0x7ff3bc1a1080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:13:00.473 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:00.471+0000 7ff3c0c00640 1 --2- 172.21.15.118:0/1083336648 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff3bc103150 0x7ff3bc1a1080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:13:00.473 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:00.471+0000 7ff3c0c00640 1 -- 172.21.15.118:0/1083336648 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7ff3bc102790 msgr2=0x7ff3bc19a010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:13:00.473 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:00.471+0000 7ff3c0c00640 1 --2- 172.21.15.118:0/1083336648 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7ff3bc102790 0x7ff3bc19a010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:13:00.473 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:00.471+0000 7ff3c0c00640 1 -- 172.21.15.118:0/1083336648 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff3bc1a16f0 con 0x7ff3bc1015a0 2024-03-24T00:13:00.473 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:00.472+0000 7ff3c0c00640 1 --2- 172.21.15.118:0/1083336648 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7ff3bc1015a0 0x7ff3bc199ad0 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7ff3ac0349e0 tx=0x7ff3ac032430 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:13:00.473 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:00.472+0000 7ff3ba000640 1 -- 172.21.15.118:0/1083336648 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff3ac041020 con 0x7ff3bc1015a0 2024-03-24T00:13:00.473 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:00.472+0000 7ff3ba000640 1 -- 172.21.15.118:0/1083336648 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7ff3ac02c9f0 con 0x7ff3bc1015a0 2024-03-24T00:13:00.473 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:00.472+0000 7ff3ba000640 1 -- 172.21.15.118:0/1083336648 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff3ac039040 con 0x7ff3bc1015a0 2024-03-24T00:13:00.473 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:00.472+0000 7ff3c2000640 1 -- 172.21.15.118:0/1083336648 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff3bc1a1980 con 0x7ff3bc1015a0 2024-03-24T00:13:00.473 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:00.472+0000 7ff3c2000640 1 -- 172.21.15.118:0/1083336648 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff3bc1a1e20 con 0x7ff3bc1015a0 2024-03-24T00:13:00.474 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:00.474+0000 7ff3c2000640 1 -- 172.21.15.118:0/1083336648 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff3bc1a20b0 con 0x7ff3bc1015a0 2024-03-24T00:13:00.474 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:00.475+0000 7ff3ba000640 1 -- 172.21.15.118:0/1083336648 <== mon.1 v2:172.21.15.193:3300/0 4 ==== mgrmap(e 16) v1 ==== 137291+0+0 (secure 0 0 0) 0x7ff3ac044020 con 0x7ff3bc1015a0 2024-03-24T00:13:00.476 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:00.476+0000 7ff3ba000640 1 --2- 172.21.15.118:0/1083336648 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7ff3a00a2880 0x7ff3a00a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:13:00.476 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:00.476+0000 7ff3ba000640 1 -- 172.21.15.118:0/1083336648 <== mon.1 v2:172.21.15.193:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 7497+0+0 (secure 0 0 0) 0x7ff3ac0eff80 con 0x7ff3bc1015a0 2024-03-24T00:13:00.476 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:00.477+0000 7ff3b3e00640 1 --2- 172.21.15.118:0/1083336648 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7ff3a00a2880 0x7ff3a00a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:13:00.476 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:00.477+0000 7ff3b3e00640 1 --2- 172.21.15.118:0/1083336648 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7ff3a00a2880 0x7ff3a00a4d40 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7ff3bc19afe0 tx=0x7ff3a4005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:13:00.480 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:00.480+0000 7ff3ba000640 1 -- 172.21.15.118:0/1083336648 <== mon.1 v2:172.21.15.193:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7ff3ac04b020 con 0x7ff3bc1015a0 2024-03-24T00:13:00.695 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:13:00 smithi118 ceph-mon[31535]: from='client.? 172.21.15.118:0/990972799' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]: dispatch 2024-03-24T00:13:00.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:13:00 smithi118 ceph-mon[31535]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]: dispatch 2024-03-24T00:13:00.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:13:00 smithi118 ceph-mon[31535]: pgmap v309: 1 pgs: 1 active+clean; 577 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-24T00:13:00.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:13:00 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T00:13:00.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:13:00 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T00:13:00.697 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:00.698+0000 7ff3c2000640 1 -- 172.21.15.118:0/1083336648 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9} v 0) v1 -- 0x7ff3bc1a26f0 con 0x7ff3bc1015a0 2024-03-24T00:13:00.703 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:13:00 smithi189 ceph-mon[36187]: from='client.? 172.21.15.118:0/990972799' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]: dispatch 2024-03-24T00:13:00.703 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:13:00 smithi189 ceph-mon[36187]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]: dispatch 2024-03-24T00:13:00.704 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:13:00 smithi189 ceph-mon[36187]: pgmap v309: 1 pgs: 1 active+clean; 577 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-24T00:13:00.704 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:13:00 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T00:13:00.704 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:13:00 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T00:13:00.710 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:13:00 smithi193 ceph-mon[36878]: from='client.? 172.21.15.118:0/990972799' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]: dispatch 2024-03-24T00:13:00.710 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:13:00 smithi193 ceph-mon[36878]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]: dispatch 2024-03-24T00:13:00.710 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:13:00 smithi193 ceph-mon[36878]: pgmap v309: 1 pgs: 1 active+clean; 577 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-24T00:13:00.710 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:13:00 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T00:13:00.711 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:13:00 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T00:13:01.288 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:01.289+0000 7ff3ba000640 1 -- 172.21.15.118:0/1083336648 <== mon.1 v2:172.21.15.193:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]=0 pool 'cephfs_data' created v84) v1 ==== 170+0+0 (secure 0 0 0) 0x7ff3ac0bd2b0 con 0x7ff3bc1015a0 2024-03-24T00:13:01.291 INFO:teuthology.orchestra.run.smithi118.stderr:pool 'cephfs_data' created 2024-03-24T00:13:01.295 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:01.296+0000 7ff3c2000640 1 -- 172.21.15.118:0/1083336648 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7ff3a00a2880 msgr2=0x7ff3a00a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:13:01.296 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:01.296+0000 7ff3c2000640 1 --2- 172.21.15.118:0/1083336648 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7ff3a00a2880 0x7ff3a00a4d40 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7ff3bc19afe0 tx=0x7ff3a4005f90 comp rx=0 tx=0).stop 2024-03-24T00:13:01.296 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:01.296+0000 7ff3c2000640 1 -- 172.21.15.118:0/1083336648 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7ff3bc1015a0 msgr2=0x7ff3bc199ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:13:01.296 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:01.296+0000 7ff3c2000640 1 --2- 172.21.15.118:0/1083336648 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7ff3bc1015a0 0x7ff3bc199ad0 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7ff3ac0349e0 tx=0x7ff3ac032430 comp rx=0 tx=0).stop 2024-03-24T00:13:01.296 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:01.296+0000 7ff3c2000640 1 -- 172.21.15.118:0/1083336648 shutdown_connections 2024-03-24T00:13:01.297 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:01.296+0000 7ff3c2000640 1 --2- 172.21.15.118:0/1083336648 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7ff3a00a2880 0x7ff3a00a4d40 secure :-1 s=CLOSED pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7ff3bc19afe0 tx=0x7ff3a4005f90 comp rx=0 tx=0).stop 2024-03-24T00:13:01.297 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:01.296+0000 7ff3c2000640 1 --2- 172.21.15.118:0/1083336648 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff3bc103150 0x7ff3bc1a1080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:13:01.297 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:01.296+0000 7ff3c2000640 1 --2- 172.21.15.118:0/1083336648 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7ff3bc102790 0x7ff3bc19a010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:13:01.297 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:01.296+0000 7ff3c2000640 1 --2- 172.21.15.118:0/1083336648 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7ff3bc1015a0 0x7ff3bc199ad0 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:13:01.297 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:01.296+0000 7ff3c2000640 1 -- 172.21.15.118:0/1083336648 >> 172.21.15.118:0/1083336648 conn(0x7ff3bc0fcb10 msgr2=0x7ff3bc107f60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:13:01.297 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:01.297+0000 7ff3c2000640 1 -- 172.21.15.118:0/1083336648 shutdown_connections 2024-03-24T00:13:01.297 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:01.298+0000 7ff3c2000640 1 -- 172.21.15.118:0/1083336648 wait complete. 2024-03-24T00:13:01.331 DEBUG:teuthology.orchestra.run.smithi118:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data 2024-03-24T00:13:01.519 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:01.519+0000 7f44f7200640 1 -- 172.21.15.118:0/3154958192 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f44f80772a0 msgr2=0x7f44f8077700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:13:01.519 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:01.519+0000 7f44f7200640 1 --2- 172.21.15.118:0/3154958192 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f44f80772a0 0x7f44f8077700 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f44e0009900 tx=0x7f44e002bc40 comp rx=0 tx=0).stop 2024-03-24T00:13:01.519 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:01.519+0000 7f44f7200640 1 -- 172.21.15.118:0/3154958192 shutdown_connections 2024-03-24T00:13:01.519 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:01.519+0000 7f44f7200640 1 --2- 172.21.15.118:0/3154958192 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f44f8077c40 0x7f44f80784e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:13:01.519 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:01.519+0000 7f44f7200640 1 --2- 172.21.15.118:0/3154958192 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f44f80772a0 0x7f44f8077700 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:13:01.519 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:01.519+0000 7f44f7200640 1 --2- 172.21.15.118:0/3154958192 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f44f8076060 0x7f44f8076480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:13:01.519 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:01.519+0000 7f44f7200640 1 -- 172.21.15.118:0/3154958192 >> 172.21.15.118:0/3154958192 conn(0x7f44f80fed50 msgr2=0x7f44f8101170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:13:01.519 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:01.519+0000 7f44f7200640 1 -- 172.21.15.118:0/3154958192 shutdown_connections 2024-03-24T00:13:01.519 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:01.519+0000 7f44f7200640 1 -- 172.21.15.118:0/3154958192 wait complete. 2024-03-24T00:13:01.520 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:01.520+0000 7f44f7200640 1 Processor -- start 2024-03-24T00:13:01.522 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:01.521+0000 7f44f7200640 1 -- start start 2024-03-24T00:13:01.522 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:01.521+0000 7f44f7200640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f44f8076060 0x7f44f819dea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:13:01.522 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:01.521+0000 7f44f7200640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f44f80772a0 0x7f44f819e3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:13:01.522 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:01.521+0000 7f44f7200640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f44f8077c40 0x7f44f81a5450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:13:01.522 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:01.521+0000 7f44f7200640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f44f8113870 con 0x7f44f8077c40 2024-03-24T00:13:01.522 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:01.521+0000 7f44f7200640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f44f81136f0 con 0x7f44f8076060 2024-03-24T00:13:01.522 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:01.521+0000 7f44f7200640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f44f81139f0 con 0x7f44f80772a0 2024-03-24T00:13:01.522 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:01.521+0000 7f44ed400640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f44f80772a0 0x7f44f819e3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:13:01.522 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:01.521+0000 7f44ed400640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f44f80772a0 0x7f44f819e3e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.118:58888/0 (socket says 172.21.15.118:58888) 2024-03-24T00:13:01.523 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:01.521+0000 7f44ed400640 1 -- 172.21.15.118:0/520319298 learned_addr learned my addr 172.21.15.118:0/520319298 (peer_addr_for_me v2:172.21.15.118:0/0) 2024-03-24T00:13:01.523 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:01.521+0000 7f44f6800640 1 --2- 172.21.15.118:0/520319298 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f44f8077c40 0x7f44f81a5450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:13:01.523 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:01.521+0000 7f44f5e00640 1 --2- 172.21.15.118:0/520319298 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f44f8076060 0x7f44f819dea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:13:01.523 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:01.522+0000 7f44ed400640 1 -- 172.21.15.118:0/520319298 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f44f8076060 msgr2=0x7f44f819dea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:13:01.523 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:01.522+0000 7f44ed400640 1 --2- 172.21.15.118:0/520319298 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f44f8076060 0x7f44f819dea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:13:01.523 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:01.522+0000 7f44ed400640 1 -- 172.21.15.118:0/520319298 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f44f8077c40 msgr2=0x7f44f81a5450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:13:01.523 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:01.522+0000 7f44ed400640 1 --2- 172.21.15.118:0/520319298 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f44f8077c40 0x7f44f81a5450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:13:01.523 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:01.522+0000 7f44ed400640 1 -- 172.21.15.118:0/520319298 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f44f81a5ac0 con 0x7f44f80772a0 2024-03-24T00:13:01.523 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:01.522+0000 7f44ed400640 1 --2- 172.21.15.118:0/520319298 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f44f80772a0 0x7f44f819e3e0 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f44e002c8f0 tx=0x7f44e002c660 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:13:01.523 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:01.522+0000 7f44ef400640 1 -- 172.21.15.118:0/520319298 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f44e0041070 con 0x7f44f80772a0 2024-03-24T00:13:01.523 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:01.522+0000 7f44f7200640 1 -- 172.21.15.118:0/520319298 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f44f81a5d50 con 0x7f44f80772a0 2024-03-24T00:13:01.523 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:01.522+0000 7f44ef400640 1 -- 172.21.15.118:0/520319298 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f44e0004450 con 0x7f44f80772a0 2024-03-24T00:13:01.523 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:01.522+0000 7f44ef400640 1 -- 172.21.15.118:0/520319298 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f44e0034440 con 0x7f44f80772a0 2024-03-24T00:13:01.523 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:01.522+0000 7f44f7200640 1 -- 172.21.15.118:0/520319298 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f44f81a61e0 con 0x7f44f80772a0 2024-03-24T00:13:01.524 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:01.523+0000 7f44f7200640 1 -- 172.21.15.118:0/520319298 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f44f8110110 con 0x7f44f80772a0 2024-03-24T00:13:01.525 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:01.525+0000 7f44ef400640 1 -- 172.21.15.118:0/520319298 <== mon.1 v2:172.21.15.193:3300/0 4 ==== mgrmap(e 16) v1 ==== 137291+0+0 (secure 0 0 0) 0x7f44e0044070 con 0x7f44f80772a0 2024-03-24T00:13:01.526 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:01.527+0000 7f44ef400640 1 --2- 172.21.15.118:0/520319298 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7f44dc0a2800 0x7f44dc0a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:13:01.526 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:01.527+0000 7f44ef400640 1 -- 172.21.15.118:0/520319298 <== mon.1 v2:172.21.15.193:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 7879+0+0 (secure 0 0 0) 0x7f44e00f0460 con 0x7f44f80772a0 2024-03-24T00:13:01.526 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:01.527+0000 7f44f5e00640 1 --2- 172.21.15.118:0/520319298 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7f44dc0a2800 0x7f44dc0a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:13:01.527 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:01.527+0000 7f44f5e00640 1 --2- 172.21.15.118:0/520319298 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7f44dc0a2800 0x7f44dc0a4cc0 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f44e8009bd0 tx=0x7f44e8005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:13:01.531 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:01.532+0000 7f44ef400640 1 -- 172.21.15.118:0/520319298 <== mon.1 v2:172.21.15.193:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f44e00bcfc0 con 0x7f44f80772a0 2024-03-24T00:13:01.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:13:01 smithi118 ceph-mon[31535]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]': finished 2024-03-24T00:13:01.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:13:01 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T00:13:01.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:13:01 smithi118 ceph-mon[31535]: osdmap e83: 12 total, 12 up, 12 in 2024-03-24T00:13:01.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:13:01 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T00:13:01.703 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:13:01 smithi189 ceph-mon[36187]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]': finished 2024-03-24T00:13:01.703 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:13:01 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T00:13:01.704 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:13:01 smithi189 ceph-mon[36187]: osdmap e83: 12 total, 12 up, 12 in 2024-03-24T00:13:01.704 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:13:01 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T00:13:01.710 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:13:01 smithi193 ceph-mon[36878]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num_min": 64}]': finished 2024-03-24T00:13:01.710 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:13:01 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T00:13:01.710 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:13:01 smithi193 ceph-mon[36878]: osdmap e83: 12 total, 12 up, 12 in 2024-03-24T00:13:01.710 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:13:01 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T00:13:01.778 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:01.779+0000 7f44f7200640 1 -- 172.21.15.118:0/520319298 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"} v 0) v1 -- 0x7f44f8062310 con 0x7f44f80772a0 2024-03-24T00:13:02.340 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:02.341+0000 7f44ef400640 1 -- 172.21.15.118:0/520319298 <== mon.1 v2:172.21.15.193:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]=0 new fs with metadata pool 2 and data pool 3 v14) v1 ==== 172+0+0 (secure 0 0 0) 0x7f44e00c1060 con 0x7f44f80772a0 2024-03-24T00:13:02.340 INFO:teuthology.orchestra.run.smithi118.stderr:new fs with metadata pool 2 and data pool 3 2024-03-24T00:13:02.345 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:02.346+0000 7f44f7200640 1 -- 172.21.15.118:0/520319298 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7f44dc0a2800 msgr2=0x7f44dc0a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:13:02.345 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:02.346+0000 7f44f7200640 1 --2- 172.21.15.118:0/520319298 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7f44dc0a2800 0x7f44dc0a4cc0 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f44e8009bd0 tx=0x7f44e8005f90 comp rx=0 tx=0).stop 2024-03-24T00:13:02.345 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:02.346+0000 7f44f7200640 1 -- 172.21.15.118:0/520319298 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f44f80772a0 msgr2=0x7f44f819e3e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:13:02.346 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:02.346+0000 7f44f7200640 1 --2- 172.21.15.118:0/520319298 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f44f80772a0 0x7f44f819e3e0 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f44e002c8f0 tx=0x7f44e002c660 comp rx=0 tx=0).stop 2024-03-24T00:13:02.346 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:02.346+0000 7f44f7200640 1 -- 172.21.15.118:0/520319298 shutdown_connections 2024-03-24T00:13:02.346 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:02.346+0000 7f44f7200640 1 --2- 172.21.15.118:0/520319298 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7f44dc0a2800 0x7f44dc0a4cc0 secure :-1 s=CLOSED pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f44e8009bd0 tx=0x7f44e8005f90 comp rx=0 tx=0).stop 2024-03-24T00:13:02.346 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:02.346+0000 7f44f7200640 1 --2- 172.21.15.118:0/520319298 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f44f8077c40 0x7f44f81a5450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:13:02.346 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:02.346+0000 7f44f7200640 1 --2- 172.21.15.118:0/520319298 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f44f80772a0 0x7f44f819e3e0 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:13:02.346 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:02.346+0000 7f44f7200640 1 --2- 172.21.15.118:0/520319298 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f44f8076060 0x7f44f819dea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:13:02.346 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:02.346+0000 7f44f7200640 1 -- 172.21.15.118:0/520319298 >> 172.21.15.118:0/520319298 conn(0x7f44f80fed50 msgr2=0x7f44f8100850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:13:02.347 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:02.347+0000 7f44f7200640 1 -- 172.21.15.118:0/520319298 shutdown_connections 2024-03-24T00:13:02.347 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:02.347+0000 7f44f7200640 1 -- 172.21.15.118:0/520319298 wait complete. 2024-03-24T00:13:02.367 DEBUG:teuthology.orchestra.run.smithi118:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-24T00:13:02.549 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:02.549+0000 7f94db000640 1 -- 172.21.15.118:0/1423960182 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f94d41036f0 msgr2=0x7f94d4103b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:13:02.549 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:02.549+0000 7f94db000640 1 --2- 172.21.15.118:0/1423960182 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f94d41036f0 0x7f94d4103b10 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f94c8009900 tx=0x7f94c802bc30 comp rx=0 tx=0).stop 2024-03-24T00:13:02.549 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:02.549+0000 7f94db000640 1 -- 172.21.15.118:0/1423960182 shutdown_connections 2024-03-24T00:13:02.549 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:02.549+0000 7f94db000640 1 --2- 172.21.15.118:0/1423960182 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f94d41052a0 0x7f94d410bb30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:13:02.549 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:02.549+0000 7f94db000640 1 --2- 172.21.15.118:0/1423960182 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f94d41048e0 0x7f94d4104d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:13:02.549 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:02.549+0000 7f94db000640 1 --2- 172.21.15.118:0/1423960182 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f94d41036f0 0x7f94d4103b10 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:13:02.549 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:02.549+0000 7f94db000640 1 -- 172.21.15.118:0/1423960182 >> 172.21.15.118:0/1423960182 conn(0x7f94d40fec80 msgr2=0x7f94d41010a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:13:02.549 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:02.549+0000 7f94db000640 1 -- 172.21.15.118:0/1423960182 shutdown_connections 2024-03-24T00:13:02.549 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:02.549+0000 7f94db000640 1 -- 172.21.15.118:0/1423960182 wait complete. 2024-03-24T00:13:02.551 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:02.550+0000 7f94db000640 1 Processor -- start 2024-03-24T00:13:02.552 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:02.550+0000 7f94db000640 1 -- start start 2024-03-24T00:13:02.552 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:02.550+0000 7f94db000640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f94d41036f0 0x7f94d41999d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:13:02.552 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:02.550+0000 7f94db000640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f94d41048e0 0x7f94d4199f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:13:02.552 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:02.550+0000 7f94db000640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f94d41052a0 0x7f94d41a0f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:13:02.552 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:02.550+0000 7f94db000640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f94d410f3c0 con 0x7f94d41036f0 2024-03-24T00:13:02.552 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:02.550+0000 7f94db000640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f94d410f240 con 0x7f94d41052a0 2024-03-24T00:13:02.552 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:02.550+0000 7f94db000640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f94d410f540 con 0x7f94d41048e0 2024-03-24T00:13:02.552 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:02.551+0000 7f94d9c00640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f94d41036f0 0x7f94d41999d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:13:02.552 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:02.551+0000 7f94d9c00640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f94d41036f0 0x7f94d41999d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.118:3300/0 says I am v2:172.21.15.118:50222/0 (socket says 172.21.15.118:50222) 2024-03-24T00:13:02.552 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:02.551+0000 7f94d9c00640 1 -- 172.21.15.118:0/3343535271 learned_addr learned my addr 172.21.15.118:0/3343535271 (peer_addr_for_me v2:172.21.15.118:0/0) 2024-03-24T00:13:02.552 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:02.551+0000 7f94da600640 1 --2- 172.21.15.118:0/3343535271 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f94d41052a0 0x7f94d41a0f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:13:02.552 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:02.551+0000 7f94d9200640 1 --2- 172.21.15.118:0/3343535271 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f94d41048e0 0x7f94d4199f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:13:02.552 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:02.551+0000 7f94d9c00640 1 -- 172.21.15.118:0/3343535271 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f94d41048e0 msgr2=0x7f94d4199f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:13:02.553 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:02.551+0000 7f94d9c00640 1 --2- 172.21.15.118:0/3343535271 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f94d41048e0 0x7f94d4199f10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:13:02.553 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:02.551+0000 7f94d9c00640 1 -- 172.21.15.118:0/3343535271 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f94d41052a0 msgr2=0x7f94d41a0f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:13:02.553 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:02.551+0000 7f94d9c00640 1 --2- 172.21.15.118:0/3343535271 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f94d41052a0 0x7f94d41a0f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:13:02.553 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:02.551+0000 7f94d9c00640 1 -- 172.21.15.118:0/3343535271 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f94d41a15f0 con 0x7f94d41036f0 2024-03-24T00:13:02.553 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:02.551+0000 7f94d9c00640 1 --2- 172.21.15.118:0/3343535271 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f94d41036f0 0x7f94d41999d0 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f94c8000c00 tx=0x7f94c802c650 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:13:02.553 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:02.552+0000 7f94d2000640 1 -- 172.21.15.118:0/3343535271 <== mon.0 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f94c8043020 con 0x7f94d41036f0 2024-03-24T00:13:02.553 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:02.552+0000 7f94db000640 1 -- 172.21.15.118:0/3343535271 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f94d41a1880 con 0x7f94d41036f0 2024-03-24T00:13:02.553 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:02.552+0000 7f94d2000640 1 -- 172.21.15.118:0/3343535271 <== mon.0 v2:172.21.15.118:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f94c80054e0 con 0x7f94d41036f0 2024-03-24T00:13:02.553 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:02.552+0000 7f94d2000640 1 -- 172.21.15.118:0/3343535271 <== mon.0 v2:172.21.15.118:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f94c8003c40 con 0x7f94d41036f0 2024-03-24T00:13:02.553 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:02.552+0000 7f94db000640 1 -- 172.21.15.118:0/3343535271 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f94d41a1dd0 con 0x7f94d41036f0 2024-03-24T00:13:02.554 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:02.553+0000 7f94db000640 1 -- 172.21.15.118:0/3343535271 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f94d410a9b0 con 0x7f94d41036f0 2024-03-24T00:13:02.554 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:02.555+0000 7f94d2000640 1 -- 172.21.15.118:0/3343535271 <== mon.0 v2:172.21.15.118:3300/0 4 ==== mgrmap(e 16) v1 ==== 137291+0+0 (secure 0 0 0) 0x7f94c8046020 con 0x7f94d41036f0 2024-03-24T00:13:02.556 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:02.557+0000 7f94d2000640 1 --2- 172.21.15.118:0/3343535271 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7f94bc0a2800 0x7f94bc0a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:13:02.556 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:02.557+0000 7f94d2000640 1 -- 172.21.15.118:0/3343535271 <== mon.0 v2:172.21.15.118:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 7979+0+0 (secure 0 0 0) 0x7f94c80f0d60 con 0x7f94d41036f0 2024-03-24T00:13:02.556 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:02.557+0000 7f94d9200640 1 --2- 172.21.15.118:0/3343535271 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7f94bc0a2800 0x7f94bc0a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:13:02.557 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:02.558+0000 7f94d9200640 1 --2- 172.21.15.118:0/3343535271 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7f94bc0a2800 0x7f94bc0a4cc0 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f94d419aee0 tx=0x7f94c0005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:13:02.559 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:02.560+0000 7f94d2000640 1 -- 172.21.15.118:0/3343535271 <== mon.0 v2:172.21.15.118:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f94c804d020 con 0x7f94d41036f0 2024-03-24T00:13:02.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:13:02 smithi118 ceph-mon[31535]: from='client.? 172.21.15.118:0/1083336648' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]: dispatch 2024-03-24T00:13:02.697 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:13:02 smithi118 ceph-mon[31535]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]: dispatch 2024-03-24T00:13:02.697 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:13:02 smithi118 ceph-mon[31535]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]': finished 2024-03-24T00:13:02.697 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:13:02 smithi118 ceph-mon[31535]: osdmap e84: 12 total, 12 up, 12 in 2024-03-24T00:13:02.697 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:13:02 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T00:13:02.697 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:13:02 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T00:13:02.697 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:13:02 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T00:13:02.697 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:13:02 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T00:13:02.697 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:13:02 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T00:13:02.697 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:13:02 smithi118 ceph-mon[31535]: pgmap v312: 97 pgs: 96 unknown, 1 active+clean; 577 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-24T00:13:02.697 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:13:02 smithi118 ceph-mon[31535]: from='client.? 172.21.15.118:0/520319298' entity='client.admin' cmd=[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]: dispatch 2024-03-24T00:13:02.697 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:13:02 smithi118 ceph-mon[31535]: from='client.? ' entity='client.admin' cmd=[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]: dispatch 2024-03-24T00:13:02.697 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:13:02 smithi118 ceph-mon[31535]: daemon mds.b assigned to filesystem cephfs as rank 0 2024-03-24T00:13:02.704 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:13:02 smithi189 ceph-mon[36187]: from='client.? 172.21.15.118:0/1083336648' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]: dispatch 2024-03-24T00:13:02.704 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:13:02 smithi189 ceph-mon[36187]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]: dispatch 2024-03-24T00:13:02.704 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:13:02 smithi189 ceph-mon[36187]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]': finished 2024-03-24T00:13:02.704 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:13:02 smithi189 ceph-mon[36187]: osdmap e84: 12 total, 12 up, 12 in 2024-03-24T00:13:02.704 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:13:02 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T00:13:02.704 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:13:02 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T00:13:02.704 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:13:02 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T00:13:02.704 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:13:02 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T00:13:02.704 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:13:02 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T00:13:02.705 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:13:02 smithi189 ceph-mon[36187]: pgmap v312: 97 pgs: 96 unknown, 1 active+clean; 577 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-24T00:13:02.705 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:13:02 smithi189 ceph-mon[36187]: from='client.? 172.21.15.118:0/520319298' entity='client.admin' cmd=[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]: dispatch 2024-03-24T00:13:02.705 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:13:02 smithi189 ceph-mon[36187]: from='client.? ' entity='client.admin' cmd=[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]: dispatch 2024-03-24T00:13:02.705 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:13:02 smithi189 ceph-mon[36187]: daemon mds.b assigned to filesystem cephfs as rank 0 2024-03-24T00:13:02.710 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:13:02 smithi193 ceph-mon[36878]: from='client.? 172.21.15.118:0/1083336648' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]: dispatch 2024-03-24T00:13:02.710 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:13:02 smithi193 ceph-mon[36878]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]: dispatch 2024-03-24T00:13:02.710 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:13:02 smithi193 ceph-mon[36878]: from='client.? ' entity='client.admin' cmd='[{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 64, "pg_num_min": 64, "target_size_ratio": 0.9}]': finished 2024-03-24T00:13:02.711 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:13:02 smithi193 ceph-mon[36878]: osdmap e84: 12 total, 12 up, 12 in 2024-03-24T00:13:02.711 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:13:02 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T00:13:02.711 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:13:02 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T00:13:02.711 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:13:02 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T00:13:02.711 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:13:02 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T00:13:02.711 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:13:02 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T00:13:02.711 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:13:02 smithi193 ceph-mon[36878]: pgmap v312: 97 pgs: 96 unknown, 1 active+clean; 577 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail 2024-03-24T00:13:02.711 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:13:02 smithi193 ceph-mon[36878]: from='client.? 172.21.15.118:0/520319298' entity='client.admin' cmd=[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]: dispatch 2024-03-24T00:13:02.711 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:13:02 smithi193 ceph-mon[36878]: from='client.? ' entity='client.admin' cmd=[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]: dispatch 2024-03-24T00:13:02.711 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:13:02 smithi193 ceph-mon[36878]: daemon mds.b assigned to filesystem cephfs as rank 0 2024-03-24T00:13:02.762 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:02.762+0000 7f94db000640 1 -- 172.21.15.118:0/3343535271 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f94d402d240 con 0x7f94d41036f0 2024-03-24T00:13:02.764 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:02.764+0000 7f94d2000640 1 -- 172.21.15.118:0/3343535271 <== mon.0 v2:172.21.15.118:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v85) v1 ==== 74+0+22300 (secure 0 0 0) 0x7f94c80bdeb0 con 0x7f94d41036f0 2024-03-24T00:13:02.765 INFO:teuthology.orchestra.run.smithi118.stdout: 2024-03-24T00:13:02.765 INFO:teuthology.orchestra.run.smithi118.stdout:{"epoch":85,"fsid":"d79a5420-e971-11ee-95cd-87774f69a715","created":"2024-03-24T00:03:58.648524+0000","modified":"2024-03-24T00:13:01.781414+0000","last_up_change":"2024-03-24T00:12:03.809000+0000","last_in_change":"2024-03-24T00:11:44.389344+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-24T00:07:50.190965+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-24T00:12:59.993723+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-24T00:13:01.257347+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"c9590426-3901-4f68-a5ef-d57aa8fbac92","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6802","nonce":557690061},{"type":"v1","addr":"172.21.15.118:6803","nonce":557690061}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6804","nonce":557690061},{"type":"v1","addr":"172.21.15.118:6805","nonce":557690061}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6808","nonce":557690061},{"type":"v1","addr":"172.21.15.118:6809","nonce":557690061}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6806","nonce":557690061},{"type":"v1","addr":"172.21.15.118:6807","nonce":557690061}]},"public_addr":"172.21.15.118:6803/557690061","cluster_addr":"172.21.15.118:6805/557690061","heartbeat_back_addr":"172.21.15.118:6809/557690061","heartbeat_front_addr":"172.21.15.118:6807/557690061","state":["exists","up"]},{"osd":1,"uuid":"fc7f827e-ecf6-4661-aad0-7a22951892e0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6800","nonce":4203029485},{"type":"v1","addr":"172.21.15.189:6801","nonce":4203029485}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6802","nonce":4203029485},{"type":"v1","addr":"172.21.15.189:6803","nonce":4203029485}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6806","nonce":4203029485},{"type":"v1","addr":"172.21.15.189:6807","nonce":4203029485}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6804","nonce":4203029485},{"type":"v1","addr":"172.21.15.189:6805","nonce":4203029485}]},"public_addr":"172.21.15.189:6801/4203029485","cluster_addr":"172.21.15.189:6803/4203029485","heartbeat_back_addr":"172.21.15.189:6807/4203029485","heartbeat_front_addr":"172.21.15.189:6805/4203029485","state":["exists","up"]},{"osd":2,"uuid":"e73ccfe9-0e5b-4734-a154-a2f0d829fcfb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6800","nonce":3699076971},{"type":"v1","addr":"172.21.15.193:6801","nonce":3699076971}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6802","nonce":3699076971},{"type":"v1","addr":"172.21.15.193:6803","nonce":3699076971}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6806","nonce":3699076971},{"type":"v1","addr":"172.21.15.193:6807","nonce":3699076971}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6804","nonce":3699076971},{"type":"v1","addr":"172.21.15.193:6805","nonce":3699076971}]},"public_addr":"172.21.15.193:6801/3699076971","cluster_addr":"172.21.15.193:6803/3699076971","heartbeat_back_addr":"172.21.15.193:6807/3699076971","heartbeat_front_addr":"172.21.15.193:6805/3699076971","state":["exists","up"]},{"osd":3,"uuid":"63fed43b-43ce-4031-8957-7fe94725c1ed","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6810","nonce":4059617010},{"type":"v1","addr":"172.21.15.118:6811","nonce":4059617010}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6812","nonce":4059617010},{"type":"v1","addr":"172.21.15.118:6813","nonce":4059617010}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6816","nonce":4059617010},{"type":"v1","addr":"172.21.15.118:6817","nonce":4059617010}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6814","nonce":4059617010},{"type":"v1","addr":"172.21.15.118:6815","nonce":4059617010}]},"public_addr":"172.21.15.118:6811/4059617010","cluster_addr":"172.21.15.118:6813/4059617010","heartbeat_back_addr":"172.21.15.118:6817/4059617010","heartbeat_front_addr":"172.21.15.118:6815/4059617010","state":["exists","up"]},{"osd":4,"uuid":"57377f66-c3b4-43c1-8e8e-bc873672906d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":34,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6808","nonce":4075247068},{"type":"v1","addr":"172.21.15.189:6809","nonce":4075247068}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6810","nonce":4075247068},{"type":"v1","addr":"172.21.15.189:6811","nonce":4075247068}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6814","nonce":4075247068},{"type":"v1","addr":"172.21.15.189:6815","nonce":4075247068}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6812","nonce":4075247068},{"type":"v1","addr":"172.21.15.189:6813","nonce":4075247068}]},"public_addr":"172.21.15.189:6809/4075247068","cluster_addr":"172.21.15.189:6811/4075247068","heartbeat_back_addr":"172.21.15.189:6815/4075247068","heartbeat_front_addr":"172.21.15.189:6813/4075247068","state":["exists","up"]},{"osd":5,"uuid":"91970976-2193-4224-81c0-dc96cb022c85","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":39,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6808","nonce":2920363402},{"type":"v1","addr":"172.21.15.193:6809","nonce":2920363402}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6810","nonce":2920363402},{"type":"v1","addr":"172.21.15.193:6811","nonce":2920363402}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6814","nonce":2920363402},{"type":"v1","addr":"172.21.15.193:6815","nonce":2920363402}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6812","nonce":2920363402},{"type":"v1","addr":"172.21.15.193:6813","nonce":2920363402}]},"public_addr":"172.21.15.193:6809/2920363402","cluster_addr":"172.21.15.193:6811/2920363402","heartbeat_back_addr":"172.21.15.193:6815/2920363402","heartbeat_front_addr":"172.21.15.193:6813/2920363402","state":["exists","up"]},{"osd":6,"uuid":"0b77b196-8674-41c6-ad3d-726b5d01c8fd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":47,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6818","nonce":431354576},{"type":"v1","addr":"172.21.15.118:6819","nonce":431354576}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6820","nonce":431354576},{"type":"v1","addr":"172.21.15.118:6821","nonce":431354576}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6824","nonce":431354576},{"type":"v1","addr":"172.21.15.118:6825","nonce":431354576}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6822","nonce":431354576},{"type":"v1","addr":"172.21.15.118:6823","nonce":431354576}]},"public_addr":"172.21.15.118:6819/431354576","cluster_addr":"172.21.15.118:6821/431354576","heartbeat_back_addr":"172.21.15.118:6825/431354576","heartbeat_front_addr":"172.21.15.118:6823/431354576","state":["exists","up"]},{"osd":7,"uuid":"1fbbd38f-4e8e-4d4c-8e98-66f6862bc4bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":52,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6816","nonce":3667611284},{"type":"v1","addr":"172.21.15.189:6817","nonce":3667611284}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6818","nonce":3667611284},{"type":"v1","addr":"172.21.15.189:6819","nonce":3667611284}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6822","nonce":3667611284},{"type":"v1","addr":"172.21.15.189:6823","nonce":3667611284}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6820","nonce":3667611284},{"type":"v1","addr":"172.21.15.189:6821","nonce":3667611284}]},"public_addr":"172.21.15.189:6817/3667611284","cluster_addr":"172.21.15.189:6819/3667611284","heartbeat_back_addr":"172.21.15.189:6823/3667611284","heartbeat_front_addr":"172.21.15.189:6821/3667611284","state":["exists","up"]},{"osd":8,"uuid":"6462c2b5-e4b3-4b0b-bcfa-4de2c89e93e0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":60,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6816","nonce":3853800910},{"type":"v1","addr":"172.21.15.193:6817","nonce":3853800910}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6818","nonce":3853800910},{"type":"v1","addr":"172.21.15.193:6819","nonce":3853800910}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6822","nonce":3853800910},{"type":"v1","addr":"172.21.15.193:6823","nonce":3853800910}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6820","nonce":3853800910},{"type":"v1","addr":"172.21.15.193:6821","nonce":3853800910}]},"public_addr":"172.21.15.193:6817/3853800910","cluster_addr":"172.21.15.193:6819/3853800910","heartbeat_back_addr":"172.21.15.193:6823/3853800910","heartbeat_front_addr":"172.21.15.193:6821/3853800910","state":["exists","up"]},{"osd":9,"uuid":"54be277f-3596-422f-9de5-9005a8571c6f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":68,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6826","nonce":3248293936},{"type":"v1","addr":"172.21.15.118:6827","nonce":3248293936}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6828","nonce":3248293936},{"type":"v1","addr":"172.21.15.118:6829","nonce":3248293936}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6832","nonce":3248293936},{"type":"v1","addr":"172.21.15.118:6833","nonce":3248293936}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6830","nonce":3248293936},{"type":"v1","addr":"172.21.15.118:6831","nonce":3248293936}]},"public_addr":"172.21.15.118:6827/3248293936","cluster_addr":"172.21.15.118:6829/3248293936","heartbeat_back_addr":"172.21.15.118:6833/3248293936","heartbeat_front_addr":"172.21.15.118:6831/3248293936","state":["exists","up"]},{"osd":10,"uuid":"7ba69d69-ac41-44ce-bc39-c049a8ca5900","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":73,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6824","nonce":2112195760},{"type":"v1","addr":"172.21.15.189:6825","nonce":2112195760}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6826","nonce":2112195760},{"type":"v1","addr":"172.21.15.189:6827","nonce":2112195760}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6830","nonce":2112195760},{"type":"v1","addr":"172.21.15.189:6831","nonce":2112195760}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6828","nonce":2112195760},{"type":"v1","addr":"172.21.15.189:6829","nonce":2112195760}]},"public_addr":"172.21.15.189:6825/2112195760","cluster_addr":"172.21.15.189:6827/2112195760","heartbeat_back_addr":"172.21.15.189:6831/2112195760","heartbeat_front_addr":"172.21.15.189:6829/2112195760","state":["exists","up"]},{"osd":11,"uuid":"186f0b18-6b22-4eae-83b7-b21bbcce0817","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":81,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6824","nonce":3880515510},{"type":"v1","addr":"172.21.15.193:6825","nonce":3880515510}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6826","nonce":3880515510},{"type":"v1","addr":"172.21.15.193:6827","nonce":3880515510}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6830","nonce":3880515510},{"type":"v1","addr":"172.21.15.193:6831","nonce":3880515510}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6828","nonce":3880515510},{"type":"v1","addr":"172.21.15.193:6829","nonce":3880515510}]},"public_addr":"172.21.15.193:6825/3880515510","cluster_addr":"172.21.15.193:6827/3880515510","heartbeat_back_addr":"172.21.15.193:6831/3880515510","heartbeat_front_addr":"172.21.15.193:6829/3880515510","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:06:58.105543+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:07:23.795866+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:07:47.426050+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:08:19.427073+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:08:45.816082+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:09:11.081523+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:09:44.485077+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:10:11.338972+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:10:36.234024+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:11:09.730432+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:11:37.059083+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:12:02.371947+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.118:6801/389249272":"2024-03-25T00:05:05.458796+0000","172.21.15.118:6800/389249272":"2024-03-25T00:05:05.458796+0000","172.21.15.118:0/2101854694":"2024-03-25T00:05:05.458796+0000","172.21.15.118:0/630779359":"2024-03-25T00:05:05.458796+0000","172.21.15.118:0/2158921029":"2024-03-25T00:04:31.283725+0000","172.21.15.118:0/1003670455":"2024-03-25T00:05:05.458796+0000","172.21.15.118:0/368712314":"2024-03-25T00:04:31.283725+0000","172.21.15.118:0/2004775597":"2024-03-25T00:04:31.283725+0000","172.21.15.118:6800/1463147955":"2024-03-25T00:04:31.283725+0000","172.21.15.118:6801/1463147955":"2024-03-25T00:04:31.283725+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-24T00:13:02.767 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:02.767+0000 7f94db000640 1 -- 172.21.15.118:0/3343535271 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7f94bc0a2800 msgr2=0x7f94bc0a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:13:02.768 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:02.767+0000 7f94db000640 1 --2- 172.21.15.118:0/3343535271 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7f94bc0a2800 0x7f94bc0a4cc0 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f94d419aee0 tx=0x7f94c0005f90 comp rx=0 tx=0).stop 2024-03-24T00:13:02.768 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:02.767+0000 7f94db000640 1 -- 172.21.15.118:0/3343535271 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f94d41036f0 msgr2=0x7f94d41999d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:13:02.768 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:02.767+0000 7f94db000640 1 --2- 172.21.15.118:0/3343535271 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f94d41036f0 0x7f94d41999d0 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f94c8000c00 tx=0x7f94c802c650 comp rx=0 tx=0).stop 2024-03-24T00:13:02.768 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:02.767+0000 7f94db000640 1 -- 172.21.15.118:0/3343535271 shutdown_connections 2024-03-24T00:13:02.768 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:02.767+0000 7f94db000640 1 --2- 172.21.15.118:0/3343535271 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7f94bc0a2800 0x7f94bc0a4cc0 unknown :-1 s=CLOSED pgs=162 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:13:02.768 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:02.767+0000 7f94db000640 1 --2- 172.21.15.118:0/3343535271 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f94d41052a0 0x7f94d41a0f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:13:02.768 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:02.767+0000 7f94db000640 1 --2- 172.21.15.118:0/3343535271 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f94d41048e0 0x7f94d4199f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:13:02.768 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:02.767+0000 7f94db000640 1 --2- 172.21.15.118:0/3343535271 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f94d41036f0 0x7f94d41999d0 unknown :-1 s=CLOSED pgs=158 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:13:02.768 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:02.767+0000 7f94db000640 1 -- 172.21.15.118:0/3343535271 >> 172.21.15.118:0/3343535271 conn(0x7f94d40fec80 msgr2=0x7f94d4107b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:13:02.768 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:02.768+0000 7f94db000640 1 -- 172.21.15.118:0/3343535271 shutdown_connections 2024-03-24T00:13:02.768 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:02.768+0000 7f94db000640 1 -- 172.21.15.118:0/3343535271 wait complete. 2024-03-24T00:13:02.786 DEBUG:teuthology.orchestra.run.smithi118:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-24T00:13:02.994 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:02.994+0000 7f89b4200640 1 -- 172.21.15.118:0/1825030983 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f89ac108ed0 msgr2=0x7f89ac109350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:13:02.994 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:02.994+0000 7f89b4200640 1 --2- 172.21.15.118:0/1825030983 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f89ac108ed0 0x7f89ac109350 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f8998009860 tx=0x7f899802bb70 comp rx=0 tx=0).stop 2024-03-24T00:13:02.994 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:02.994+0000 7f89b4200640 1 -- 172.21.15.118:0/1825030983 shutdown_connections 2024-03-24T00:13:02.994 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:02.994+0000 7f89b4200640 1 --2- 172.21.15.118:0/1825030983 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f89ac109890 0x7f89ac110120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:13:02.994 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:02.994+0000 7f89b4200640 1 --2- 172.21.15.118:0/1825030983 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f89ac108ed0 0x7f89ac109350 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:13:02.994 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:02.994+0000 7f89b4200640 1 --2- 172.21.15.118:0/1825030983 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f89ac107ce0 0x7f89ac108100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:13:02.994 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:02.994+0000 7f89b4200640 1 -- 172.21.15.118:0/1825030983 >> 172.21.15.118:0/1825030983 conn(0x7f89ac076cb0 msgr2=0x7f89ac0790d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:13:02.994 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:02.994+0000 7f89b4200640 1 -- 172.21.15.118:0/1825030983 shutdown_connections 2024-03-24T00:13:02.994 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:02.994+0000 7f89b4200640 1 -- 172.21.15.118:0/1825030983 wait complete. 2024-03-24T00:13:02.996 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:02.995+0000 7f89b4200640 1 Processor -- start 2024-03-24T00:13:02.996 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:02.995+0000 7f89b4200640 1 -- start start 2024-03-24T00:13:02.996 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:02.996+0000 7f89b4200640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f89ac107ce0 0x7f89ac19dfe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:13:02.996 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:02.996+0000 7f89b4200640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f89ac108ed0 0x7f89ac19e520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:13:02.996 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:02.996+0000 7f89b4200640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f89ac109890 0x7f89ac1a5590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:13:02.996 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:02.996+0000 7f89b4200640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f89ac113940 con 0x7f89ac107ce0 2024-03-24T00:13:02.996 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:02.996+0000 7f89b4200640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f89ac1137c0 con 0x7f89ac108ed0 2024-03-24T00:13:02.996 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:02.996+0000 7f89b4200640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f89ac113ac0 con 0x7f89ac109890 2024-03-24T00:13:02.996 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:02.996+0000 7f89b1800640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f89ac107ce0 0x7f89ac19dfe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:13:02.997 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:02.996+0000 7f89b1800640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f89ac107ce0 0x7f89ac19dfe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.118:3300/0 says I am v2:172.21.15.118:35264/0 (socket says 172.21.15.118:35264) 2024-03-24T00:13:02.997 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:02.996+0000 7f89b1800640 1 -- 172.21.15.118:0/793666982 learned_addr learned my addr 172.21.15.118:0/793666982 (peer_addr_for_me v2:172.21.15.118:0/0) 2024-03-24T00:13:02.997 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:02.996+0000 7f89b0e00640 1 --2- 172.21.15.118:0/793666982 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f89ac108ed0 0x7f89ac19e520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:13:02.997 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:02.996+0000 7f89b2200640 1 --2- 172.21.15.118:0/793666982 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f89ac109890 0x7f89ac1a5590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:13:02.997 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:02.996+0000 7f89b1800640 1 -- 172.21.15.118:0/793666982 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f89ac109890 msgr2=0x7f89ac1a5590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:13:02.997 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:02.996+0000 7f89b1800640 1 --2- 172.21.15.118:0/793666982 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f89ac109890 0x7f89ac1a5590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:13:02.997 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:02.996+0000 7f89b1800640 1 -- 172.21.15.118:0/793666982 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f89ac108ed0 msgr2=0x7f89ac19e520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:13:02.997 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:02.996+0000 7f89b1800640 1 --2- 172.21.15.118:0/793666982 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f89ac108ed0 0x7f89ac19e520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:13:02.997 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:02.996+0000 7f89b1800640 1 -- 172.21.15.118:0/793666982 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f89ac1a5c00 con 0x7f89ac107ce0 2024-03-24T00:13:02.997 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:02.997+0000 7f89b1800640 1 --2- 172.21.15.118:0/793666982 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f89ac107ce0 0x7f89ac19dfe0 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f89a000e9a0 tx=0x7f89a000eda0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:13:02.998 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:02.997+0000 7f89aa000640 1 -- 172.21.15.118:0/793666982 <== mon.0 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f89a000ccc0 con 0x7f89ac107ce0 2024-03-24T00:13:02.998 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:02.997+0000 7f89b4200640 1 -- 172.21.15.118:0/793666982 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f89ac1a5ef0 con 0x7f89ac107ce0 2024-03-24T00:13:02.998 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:02.997+0000 7f89aa000640 1 -- 172.21.15.118:0/793666982 <== mon.0 v2:172.21.15.118:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f89a0010910 con 0x7f89ac107ce0 2024-03-24T00:13:02.998 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:02.997+0000 7f89aa000640 1 -- 172.21.15.118:0/793666982 <== mon.0 v2:172.21.15.118:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f89a0018a30 con 0x7f89ac107ce0 2024-03-24T00:13:02.998 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:02.997+0000 7f89b4200640 1 -- 172.21.15.118:0/793666982 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f89ac1a6360 con 0x7f89ac107ce0 2024-03-24T00:13:02.998 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:02.999+0000 7f89b4200640 1 -- 172.21.15.118:0/793666982 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f89ac10f000 con 0x7f89ac107ce0 2024-03-24T00:13:03.000 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:03.000+0000 7f89aa000640 1 -- 172.21.15.118:0/793666982 <== mon.0 v2:172.21.15.118:3300/0 4 ==== mgrmap(e 16) v1 ==== 137291+0+0 (secure 0 0 0) 0x7f89a001f020 con 0x7f89ac107ce0 2024-03-24T00:13:03.001 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:03.002+0000 7f89aa000640 1 --2- 172.21.15.118:0/793666982 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7f89940a2800 0x7f89940a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:13:03.001 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:03.002+0000 7f89aa000640 1 -- 172.21.15.118:0/793666982 <== mon.0 v2:172.21.15.118:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 7979+0+0 (secure 0 0 0) 0x7f89a00ceb70 con 0x7f89ac107ce0 2024-03-24T00:13:03.001 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:03.002+0000 7f89b0e00640 1 --2- 172.21.15.118:0/793666982 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7f89940a2800 0x7f89940a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:13:03.002 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:03.003+0000 7f89b0e00640 1 --2- 172.21.15.118:0/793666982 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7f89940a2800 0x7f89940a4cc0 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7f8998009830 tx=0x7f8998034ab0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:13:03.006 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:03.006+0000 7f89aa000640 1 -- 172.21.15.118:0/793666982 <== mon.0 v2:172.21.15.118:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f89a009bcc0 con 0x7f89ac107ce0 2024-03-24T00:13:03.205 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:03.205+0000 7f89b4200640 1 -- 172.21.15.118:0/793666982 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f89ac062340 con 0x7f89ac107ce0 2024-03-24T00:13:03.206 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:03.207+0000 7f89aa000640 1 -- 172.21.15.118:0/793666982 <== mon.0 v2:172.21.15.118:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v85) v1 ==== 74+0+22300 (secure 0 0 0) 0x7f89a009b5c0 con 0x7f89ac107ce0 2024-03-24T00:13:03.207 INFO:teuthology.orchestra.run.smithi118.stdout: 2024-03-24T00:13:03.207 INFO:teuthology.orchestra.run.smithi118.stdout:{"epoch":85,"fsid":"d79a5420-e971-11ee-95cd-87774f69a715","created":"2024-03-24T00:03:58.648524+0000","modified":"2024-03-24T00:13:01.781414+0000","last_up_change":"2024-03-24T00:12:03.809000+0000","last_in_change":"2024-03-24T00:11:44.389344+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-24T00:07:50.190965+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-24T00:12:59.993723+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-24T00:13:01.257347+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"c9590426-3901-4f68-a5ef-d57aa8fbac92","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6802","nonce":557690061},{"type":"v1","addr":"172.21.15.118:6803","nonce":557690061}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6804","nonce":557690061},{"type":"v1","addr":"172.21.15.118:6805","nonce":557690061}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6808","nonce":557690061},{"type":"v1","addr":"172.21.15.118:6809","nonce":557690061}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6806","nonce":557690061},{"type":"v1","addr":"172.21.15.118:6807","nonce":557690061}]},"public_addr":"172.21.15.118:6803/557690061","cluster_addr":"172.21.15.118:6805/557690061","heartbeat_back_addr":"172.21.15.118:6809/557690061","heartbeat_front_addr":"172.21.15.118:6807/557690061","state":["exists","up"]},{"osd":1,"uuid":"fc7f827e-ecf6-4661-aad0-7a22951892e0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6800","nonce":4203029485},{"type":"v1","addr":"172.21.15.189:6801","nonce":4203029485}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6802","nonce":4203029485},{"type":"v1","addr":"172.21.15.189:6803","nonce":4203029485}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6806","nonce":4203029485},{"type":"v1","addr":"172.21.15.189:6807","nonce":4203029485}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6804","nonce":4203029485},{"type":"v1","addr":"172.21.15.189:6805","nonce":4203029485}]},"public_addr":"172.21.15.189:6801/4203029485","cluster_addr":"172.21.15.189:6803/4203029485","heartbeat_back_addr":"172.21.15.189:6807/4203029485","heartbeat_front_addr":"172.21.15.189:6805/4203029485","state":["exists","up"]},{"osd":2,"uuid":"e73ccfe9-0e5b-4734-a154-a2f0d829fcfb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6800","nonce":3699076971},{"type":"v1","addr":"172.21.15.193:6801","nonce":3699076971}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6802","nonce":3699076971},{"type":"v1","addr":"172.21.15.193:6803","nonce":3699076971}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6806","nonce":3699076971},{"type":"v1","addr":"172.21.15.193:6807","nonce":3699076971}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6804","nonce":3699076971},{"type":"v1","addr":"172.21.15.193:6805","nonce":3699076971}]},"public_addr":"172.21.15.193:6801/3699076971","cluster_addr":"172.21.15.193:6803/3699076971","heartbeat_back_addr":"172.21.15.193:6807/3699076971","heartbeat_front_addr":"172.21.15.193:6805/3699076971","state":["exists","up"]},{"osd":3,"uuid":"63fed43b-43ce-4031-8957-7fe94725c1ed","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6810","nonce":4059617010},{"type":"v1","addr":"172.21.15.118:6811","nonce":4059617010}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6812","nonce":4059617010},{"type":"v1","addr":"172.21.15.118:6813","nonce":4059617010}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6816","nonce":4059617010},{"type":"v1","addr":"172.21.15.118:6817","nonce":4059617010}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6814","nonce":4059617010},{"type":"v1","addr":"172.21.15.118:6815","nonce":4059617010}]},"public_addr":"172.21.15.118:6811/4059617010","cluster_addr":"172.21.15.118:6813/4059617010","heartbeat_back_addr":"172.21.15.118:6817/4059617010","heartbeat_front_addr":"172.21.15.118:6815/4059617010","state":["exists","up"]},{"osd":4,"uuid":"57377f66-c3b4-43c1-8e8e-bc873672906d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":34,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6808","nonce":4075247068},{"type":"v1","addr":"172.21.15.189:6809","nonce":4075247068}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6810","nonce":4075247068},{"type":"v1","addr":"172.21.15.189:6811","nonce":4075247068}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6814","nonce":4075247068},{"type":"v1","addr":"172.21.15.189:6815","nonce":4075247068}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6812","nonce":4075247068},{"type":"v1","addr":"172.21.15.189:6813","nonce":4075247068}]},"public_addr":"172.21.15.189:6809/4075247068","cluster_addr":"172.21.15.189:6811/4075247068","heartbeat_back_addr":"172.21.15.189:6815/4075247068","heartbeat_front_addr":"172.21.15.189:6813/4075247068","state":["exists","up"]},{"osd":5,"uuid":"91970976-2193-4224-81c0-dc96cb022c85","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":39,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6808","nonce":2920363402},{"type":"v1","addr":"172.21.15.193:6809","nonce":2920363402}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6810","nonce":2920363402},{"type":"v1","addr":"172.21.15.193:6811","nonce":2920363402}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6814","nonce":2920363402},{"type":"v1","addr":"172.21.15.193:6815","nonce":2920363402}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6812","nonce":2920363402},{"type":"v1","addr":"172.21.15.193:6813","nonce":2920363402}]},"public_addr":"172.21.15.193:6809/2920363402","cluster_addr":"172.21.15.193:6811/2920363402","heartbeat_back_addr":"172.21.15.193:6815/2920363402","heartbeat_front_addr":"172.21.15.193:6813/2920363402","state":["exists","up"]},{"osd":6,"uuid":"0b77b196-8674-41c6-ad3d-726b5d01c8fd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":47,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6818","nonce":431354576},{"type":"v1","addr":"172.21.15.118:6819","nonce":431354576}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6820","nonce":431354576},{"type":"v1","addr":"172.21.15.118:6821","nonce":431354576}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6824","nonce":431354576},{"type":"v1","addr":"172.21.15.118:6825","nonce":431354576}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6822","nonce":431354576},{"type":"v1","addr":"172.21.15.118:6823","nonce":431354576}]},"public_addr":"172.21.15.118:6819/431354576","cluster_addr":"172.21.15.118:6821/431354576","heartbeat_back_addr":"172.21.15.118:6825/431354576","heartbeat_front_addr":"172.21.15.118:6823/431354576","state":["exists","up"]},{"osd":7,"uuid":"1fbbd38f-4e8e-4d4c-8e98-66f6862bc4bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":52,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6816","nonce":3667611284},{"type":"v1","addr":"172.21.15.189:6817","nonce":3667611284}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6818","nonce":3667611284},{"type":"v1","addr":"172.21.15.189:6819","nonce":3667611284}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6822","nonce":3667611284},{"type":"v1","addr":"172.21.15.189:6823","nonce":3667611284}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6820","nonce":3667611284},{"type":"v1","addr":"172.21.15.189:6821","nonce":3667611284}]},"public_addr":"172.21.15.189:6817/3667611284","cluster_addr":"172.21.15.189:6819/3667611284","heartbeat_back_addr":"172.21.15.189:6823/3667611284","heartbeat_front_addr":"172.21.15.189:6821/3667611284","state":["exists","up"]},{"osd":8,"uuid":"6462c2b5-e4b3-4b0b-bcfa-4de2c89e93e0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":60,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6816","nonce":3853800910},{"type":"v1","addr":"172.21.15.193:6817","nonce":3853800910}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6818","nonce":3853800910},{"type":"v1","addr":"172.21.15.193:6819","nonce":3853800910}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6822","nonce":3853800910},{"type":"v1","addr":"172.21.15.193:6823","nonce":3853800910}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6820","nonce":3853800910},{"type":"v1","addr":"172.21.15.193:6821","nonce":3853800910}]},"public_addr":"172.21.15.193:6817/3853800910","cluster_addr":"172.21.15.193:6819/3853800910","heartbeat_back_addr":"172.21.15.193:6823/3853800910","heartbeat_front_addr":"172.21.15.193:6821/3853800910","state":["exists","up"]},{"osd":9,"uuid":"54be277f-3596-422f-9de5-9005a8571c6f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":68,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6826","nonce":3248293936},{"type":"v1","addr":"172.21.15.118:6827","nonce":3248293936}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6828","nonce":3248293936},{"type":"v1","addr":"172.21.15.118:6829","nonce":3248293936}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6832","nonce":3248293936},{"type":"v1","addr":"172.21.15.118:6833","nonce":3248293936}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6830","nonce":3248293936},{"type":"v1","addr":"172.21.15.118:6831","nonce":3248293936}]},"public_addr":"172.21.15.118:6827/3248293936","cluster_addr":"172.21.15.118:6829/3248293936","heartbeat_back_addr":"172.21.15.118:6833/3248293936","heartbeat_front_addr":"172.21.15.118:6831/3248293936","state":["exists","up"]},{"osd":10,"uuid":"7ba69d69-ac41-44ce-bc39-c049a8ca5900","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":73,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6824","nonce":2112195760},{"type":"v1","addr":"172.21.15.189:6825","nonce":2112195760}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6826","nonce":2112195760},{"type":"v1","addr":"172.21.15.189:6827","nonce":2112195760}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6830","nonce":2112195760},{"type":"v1","addr":"172.21.15.189:6831","nonce":2112195760}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6828","nonce":2112195760},{"type":"v1","addr":"172.21.15.189:6829","nonce":2112195760}]},"public_addr":"172.21.15.189:6825/2112195760","cluster_addr":"172.21.15.189:6827/2112195760","heartbeat_back_addr":"172.21.15.189:6831/2112195760","heartbeat_front_addr":"172.21.15.189:6829/2112195760","state":["exists","up"]},{"osd":11,"uuid":"186f0b18-6b22-4eae-83b7-b21bbcce0817","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":81,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6824","nonce":3880515510},{"type":"v1","addr":"172.21.15.193:6825","nonce":3880515510}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6826","nonce":3880515510},{"type":"v1","addr":"172.21.15.193:6827","nonce":3880515510}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6830","nonce":3880515510},{"type":"v1","addr":"172.21.15.193:6831","nonce":3880515510}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6828","nonce":3880515510},{"type":"v1","addr":"172.21.15.193:6829","nonce":3880515510}]},"public_addr":"172.21.15.193:6825/3880515510","cluster_addr":"172.21.15.193:6827/3880515510","heartbeat_back_addr":"172.21.15.193:6831/3880515510","heartbeat_front_addr":"172.21.15.193:6829/3880515510","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:06:58.105543+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:07:23.795866+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:07:47.426050+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:08:19.427073+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:08:45.816082+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:09:11.081523+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:09:44.485077+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:10:11.338972+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:10:36.234024+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:11:09.730432+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:11:37.059083+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:12:02.371947+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.118:6801/389249272":"2024-03-25T00:05:05.458796+0000","172.21.15.118:6800/389249272":"2024-03-25T00:05:05.458796+0000","172.21.15.118:0/2101854694":"2024-03-25T00:05:05.458796+0000","172.21.15.118:0/630779359":"2024-03-25T00:05:05.458796+0000","172.21.15.118:0/2158921029":"2024-03-25T00:04:31.283725+0000","172.21.15.118:0/1003670455":"2024-03-25T00:05:05.458796+0000","172.21.15.118:0/368712314":"2024-03-25T00:04:31.283725+0000","172.21.15.118:0/2004775597":"2024-03-25T00:04:31.283725+0000","172.21.15.118:6800/1463147955":"2024-03-25T00:04:31.283725+0000","172.21.15.118:6801/1463147955":"2024-03-25T00:04:31.283725+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-24T00:13:03.210 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:03.209+0000 7f89b4200640 1 -- 172.21.15.118:0/793666982 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7f89940a2800 msgr2=0x7f89940a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:13:03.210 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:03.209+0000 7f89b4200640 1 --2- 172.21.15.118:0/793666982 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7f89940a2800 0x7f89940a4cc0 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7f8998009830 tx=0x7f8998034ab0 comp rx=0 tx=0).stop 2024-03-24T00:13:03.210 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:03.210+0000 7f89b4200640 1 -- 172.21.15.118:0/793666982 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f89ac107ce0 msgr2=0x7f89ac19dfe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:13:03.210 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:03.210+0000 7f89b4200640 1 --2- 172.21.15.118:0/793666982 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f89ac107ce0 0x7f89ac19dfe0 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f89a000e9a0 tx=0x7f89a000eda0 comp rx=0 tx=0).stop 2024-03-24T00:13:03.210 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:03.210+0000 7f89b4200640 1 -- 172.21.15.118:0/793666982 shutdown_connections 2024-03-24T00:13:03.210 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:03.210+0000 7f89b4200640 1 --2- 172.21.15.118:0/793666982 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7f89940a2800 0x7f89940a4cc0 unknown :-1 s=CLOSED pgs=163 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:13:03.210 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:03.210+0000 7f89b4200640 1 --2- 172.21.15.118:0/793666982 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f89ac109890 0x7f89ac1a5590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:13:03.210 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:03.210+0000 7f89b4200640 1 --2- 172.21.15.118:0/793666982 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f89ac108ed0 0x7f89ac19e520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:13:03.210 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:03.210+0000 7f89b4200640 1 --2- 172.21.15.118:0/793666982 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f89ac107ce0 0x7f89ac19dfe0 unknown :-1 s=CLOSED pgs=159 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:13:03.210 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:03.210+0000 7f89b4200640 1 -- 172.21.15.118:0/793666982 >> 172.21.15.118:0/793666982 conn(0x7f89ac076cb0 msgr2=0x7f89ac10c1d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:13:03.211 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:03.210+0000 7f89b4200640 1 -- 172.21.15.118:0/793666982 shutdown_connections 2024-03-24T00:13:03.211 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:03.210+0000 7f89b4200640 1 -- 172.21.15.118:0/793666982 wait complete. 2024-03-24T00:13:03.228 DEBUG:teuthology.orchestra.run.smithi118:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2024-03-24T00:13:03.448 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:03.448+0000 7f4569000640 1 -- 172.21.15.118:0/2971722868 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f45641038c0 msgr2=0x7f4564103ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:13:03.448 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:03.448+0000 7f4569000640 1 --2- 172.21.15.118:0/2971722868 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f45641038c0 0x7f4564103ce0 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f4554009900 tx=0x7f455402bc30 comp rx=0 tx=0).stop 2024-03-24T00:13:03.449 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:03.448+0000 7f4569000640 1 -- 172.21.15.118:0/2971722868 shutdown_connections 2024-03-24T00:13:03.449 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:03.448+0000 7f4569000640 1 --2- 172.21.15.118:0/2971722868 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f4564105470 0x7f456410bd00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:13:03.449 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:03.448+0000 7f4569000640 1 --2- 172.21.15.118:0/2971722868 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f4564104ab0 0x7f4564104f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:13:03.449 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:03.448+0000 7f4569000640 1 --2- 172.21.15.118:0/2971722868 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f45641038c0 0x7f4564103ce0 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:13:03.449 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:03.448+0000 7f4569000640 1 -- 172.21.15.118:0/2971722868 >> 172.21.15.118:0/2971722868 conn(0x7f45640fee50 msgr2=0x7f4564101270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:13:03.449 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:03.448+0000 7f4569000640 1 -- 172.21.15.118:0/2971722868 shutdown_connections 2024-03-24T00:13:03.449 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:03.448+0000 7f4569000640 1 -- 172.21.15.118:0/2971722868 wait complete. 2024-03-24T00:13:03.451 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:03.449+0000 7f4569000640 1 Processor -- start 2024-03-24T00:13:03.451 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:03.449+0000 7f4569000640 1 -- start start 2024-03-24T00:13:03.451 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:03.450+0000 7f4569000640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f45641038c0 0x7f4564199bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:13:03.451 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:03.450+0000 7f4569000640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f4564104ab0 0x7f456419a0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:13:03.451 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:03.450+0000 7f4569000640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4564105470 0x7f45641a1160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:13:03.452 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:03.450+0000 7f4569000640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f456410f590 con 0x7f45641038c0 2024-03-24T00:13:03.452 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:03.450+0000 7f4569000640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f456410f410 con 0x7f4564105470 2024-03-24T00:13:03.452 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:03.450+0000 7f4569000640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f456410f710 con 0x7f4564104ab0 2024-03-24T00:13:03.452 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:03.450+0000 7f4561e00640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f45641038c0 0x7f4564199bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:13:03.452 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:03.450+0000 7f4561e00640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f45641038c0 0x7f4564199bb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.118:3300/0 says I am v2:172.21.15.118:35278/0 (socket says 172.21.15.118:35278) 2024-03-24T00:13:03.452 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:03.450+0000 7f4562800640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4564105470 0x7f45641a1160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:13:03.452 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:03.450+0000 7f4561e00640 1 -- 172.21.15.118:0/3518174268 learned_addr learned my addr 172.21.15.118:0/3518174268 (peer_addr_for_me v2:172.21.15.118:0/0) 2024-03-24T00:13:03.452 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:03.450+0000 7f4559400640 1 --2- 172.21.15.118:0/3518174268 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f4564104ab0 0x7f456419a0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:13:03.452 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:03.451+0000 7f4561e00640 1 -- 172.21.15.118:0/3518174268 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f4564104ab0 msgr2=0x7f456419a0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:13:03.452 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:03.451+0000 7f4561e00640 1 --2- 172.21.15.118:0/3518174268 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f4564104ab0 0x7f456419a0f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:13:03.452 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:03.451+0000 7f4561e00640 1 -- 172.21.15.118:0/3518174268 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4564105470 msgr2=0x7f45641a1160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:13:03.452 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:03.451+0000 7f4561e00640 1 --2- 172.21.15.118:0/3518174268 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4564105470 0x7f45641a1160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:13:03.452 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:03.451+0000 7f4561e00640 1 -- 172.21.15.118:0/3518174268 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f45641a17d0 con 0x7f45641038c0 2024-03-24T00:13:03.452 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:03.451+0000 7f4562800640 1 --2- 172.21.15.118:0/3518174268 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4564105470 0x7f45641a1160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T00:13:03.452 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:03.451+0000 7f4561e00640 1 --2- 172.21.15.118:0/3518174268 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f45641038c0 0x7f4564199bb0 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f4554000c00 tx=0x7f455402c650 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:13:03.452 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:03.451+0000 7f455b400640 1 -- 172.21.15.118:0/3518174268 <== mon.0 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4554043020 con 0x7f45641038c0 2024-03-24T00:13:03.453 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:03.451+0000 7f455b400640 1 -- 172.21.15.118:0/3518174268 <== mon.0 v2:172.21.15.118:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f45540054e0 con 0x7f45641038c0 2024-03-24T00:13:03.453 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:03.451+0000 7f4569000640 1 -- 172.21.15.118:0/3518174268 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f45641a1a60 con 0x7f45641038c0 2024-03-24T00:13:03.453 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:03.451+0000 7f455b400640 1 -- 172.21.15.118:0/3518174268 <== mon.0 v2:172.21.15.118:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4554003c40 con 0x7f45641038c0 2024-03-24T00:13:03.453 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:03.451+0000 7f4569000640 1 -- 172.21.15.118:0/3518174268 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f45641a1fb0 con 0x7f45641038c0 2024-03-24T00:13:03.453 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:03.453+0000 7f4569000640 1 -- 172.21.15.118:0/3518174268 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f456410abe0 con 0x7f45641038c0 2024-03-24T00:13:03.454 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:03.454+0000 7f455b400640 1 -- 172.21.15.118:0/3518174268 <== mon.0 v2:172.21.15.118:3300/0 4 ==== mgrmap(e 16) v1 ==== 137291+0+0 (secure 0 0 0) 0x7f4554046020 con 0x7f45641038c0 2024-03-24T00:13:03.455 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:03.455+0000 7f455b400640 1 --2- 172.21.15.118:0/3518174268 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7f45480a2800 0x7f45480a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:13:03.455 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:03.456+0000 7f455b400640 1 -- 172.21.15.118:0/3518174268 <== mon.0 v2:172.21.15.118:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 7979+0+0 (secure 0 0 0) 0x7f45540f0970 con 0x7f45641038c0 2024-03-24T00:13:03.455 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:03.456+0000 7f4559400640 1 --2- 172.21.15.118:0/3518174268 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7f45480a2800 0x7f45480a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:13:03.456 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:03.456+0000 7f4559400640 1 --2- 172.21.15.118:0/3518174268 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7f45480a2800 0x7f45480a4cc0 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f456419b0c0 tx=0x7f454c005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:13:03.459 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:03.460+0000 7f455b400640 1 -- 172.21.15.118:0/3518174268 <== mon.0 v2:172.21.15.118:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f455404d020 con 0x7f45641038c0 2024-03-24T00:13:03.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:13:03 smithi118 ceph-mon[31535]: osdmap e85: 12 total, 12 up, 12 in 2024-03-24T00:13:03.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:13:03 smithi118 ceph-mon[31535]: Health check failed: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-24T00:13:03.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:13:03 smithi118 ceph-mon[31535]: from='client.? ' entity='client.admin' cmd='[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]': finished 2024-03-24T00:13:03.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:13:03 smithi118 ceph-mon[31535]: fsmap cephfs:1 {0=b=up:creating} 11 up:standby 2024-03-24T00:13:03.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:13:03 smithi118 ceph-mon[31535]: daemon mds.b is now active in filesystem cephfs as rank 0 2024-03-24T00:13:03.697 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:13:03 smithi118 ceph-mon[31535]: from='client.? 172.21.15.118:0/3343535271' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-24T00:13:03.697 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:13:03 smithi118 ceph-mon[31535]: from='client.? 172.21.15.118:0/793666982' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-24T00:13:03.700 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:03.701+0000 7f4569000640 1 -- 172.21.15.118:0/3518174268 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"} v 0) v1 -- 0x7f456402d1f0 con 0x7f45641038c0 2024-03-24T00:13:03.703 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:13:03 smithi189 ceph-mon[36187]: osdmap e85: 12 total, 12 up, 12 in 2024-03-24T00:13:03.703 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:13:03 smithi189 ceph-mon[36187]: Health check failed: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-24T00:13:03.703 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:13:03 smithi189 ceph-mon[36187]: from='client.? ' entity='client.admin' cmd='[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]': finished 2024-03-24T00:13:03.703 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:13:03 smithi189 ceph-mon[36187]: fsmap cephfs:1 {0=b=up:creating} 11 up:standby 2024-03-24T00:13:03.704 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:13:03 smithi189 ceph-mon[36187]: daemon mds.b is now active in filesystem cephfs as rank 0 2024-03-24T00:13:03.704 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:13:03 smithi189 ceph-mon[36187]: from='client.? 172.21.15.118:0/3343535271' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-24T00:13:03.704 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:13:03 smithi189 ceph-mon[36187]: from='client.? 172.21.15.118:0/793666982' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-24T00:13:03.710 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:13:03 smithi193 ceph-mon[36878]: osdmap e85: 12 total, 12 up, 12 in 2024-03-24T00:13:03.710 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:13:03 smithi193 ceph-mon[36878]: Health check failed: 2 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-24T00:13:03.710 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:13:03 smithi193 ceph-mon[36878]: from='client.? ' entity='client.admin' cmd='[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]': finished 2024-03-24T00:13:03.711 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:13:03 smithi193 ceph-mon[36878]: fsmap cephfs:1 {0=b=up:creating} 11 up:standby 2024-03-24T00:13:03.711 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:13:03 smithi193 ceph-mon[36878]: daemon mds.b is now active in filesystem cephfs as rank 0 2024-03-24T00:13:03.711 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:13:03 smithi193 ceph-mon[36878]: from='client.? 172.21.15.118:0/3343535271' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-24T00:13:03.711 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:13:03 smithi193 ceph-mon[36878]: from='client.? 172.21.15.118:0/793666982' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-24T00:13:04.359 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:04.359+0000 7f455b400640 1 -- 172.21.15.118:0/3518174268 <== mon.0 v2:172.21.15.118:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]=0 v17) v1 ==== 118+0+0 (secure 0 0 0) 0x7f45540bdac0 con 0x7f45641038c0 2024-03-24T00:13:04.363 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:04.363+0000 7f4569000640 1 -- 172.21.15.118:0/3518174268 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7f45480a2800 msgr2=0x7f45480a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:13:04.363 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:04.363+0000 7f4569000640 1 --2- 172.21.15.118:0/3518174268 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7f45480a2800 0x7f45480a4cc0 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f456419b0c0 tx=0x7f454c005f90 comp rx=0 tx=0).stop 2024-03-24T00:13:04.363 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:04.363+0000 7f4569000640 1 -- 172.21.15.118:0/3518174268 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f45641038c0 msgr2=0x7f4564199bb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:13:04.363 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:04.363+0000 7f4569000640 1 --2- 172.21.15.118:0/3518174268 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f45641038c0 0x7f4564199bb0 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f4554000c00 tx=0x7f455402c650 comp rx=0 tx=0).stop 2024-03-24T00:13:04.364 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:04.364+0000 7f4569000640 1 -- 172.21.15.118:0/3518174268 shutdown_connections 2024-03-24T00:13:04.364 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:04.364+0000 7f4569000640 1 --2- 172.21.15.118:0/3518174268 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7f45480a2800 0x7f45480a4cc0 unknown :-1 s=CLOSED pgs=164 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:13:04.364 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:04.364+0000 7f4569000640 1 --2- 172.21.15.118:0/3518174268 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4564105470 0x7f45641a1160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:13:04.364 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:04.364+0000 7f4569000640 1 --2- 172.21.15.118:0/3518174268 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f4564104ab0 0x7f456419a0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:13:04.364 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:04.364+0000 7f4569000640 1 --2- 172.21.15.118:0/3518174268 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f45641038c0 0x7f4564199bb0 unknown :-1 s=CLOSED pgs=160 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:13:04.364 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:04.364+0000 7f4569000640 1 -- 172.21.15.118:0/3518174268 >> 172.21.15.118:0/3518174268 conn(0x7f45640fee50 msgr2=0x7f4564107db0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:13:04.364 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:04.365+0000 7f4569000640 1 -- 172.21.15.118:0/3518174268 shutdown_connections 2024-03-24T00:13:04.365 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:04.365+0000 7f4569000640 1 -- 172.21.15.118:0/3518174268 wait complete. 2024-03-24T00:13:04.384 DEBUG:tasks.cephfs.filesystem:fs_config: {'max_mds': 5, 'session_timeout': 300, 'standby_replay': True, 'subvols': {'create': 2}} 2024-03-24T00:13:04.385 DEBUG:teuthology.orchestra.run.smithi118:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs max_mds 5 2024-03-24T00:13:04.596 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:04.596+0000 7f5f6a400640 1 -- 172.21.15.118:0/1307064314 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5f641005b0 msgr2=0x7f5f641009d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:13:04.596 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:04.596+0000 7f5f6a400640 1 --2- 172.21.15.118:0/1307064314 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5f641005b0 0x7f5f641009d0 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f5f54009900 tx=0x7f5f5402bc50 comp rx=0 tx=0).stop 2024-03-24T00:13:04.596 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:04.596+0000 7f5f6a400640 1 -- 172.21.15.118:0/1307064314 shutdown_connections 2024-03-24T00:13:04.596 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:04.596+0000 7f5f6a400640 1 --2- 172.21.15.118:0/1307064314 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f5f64102160 0x7f5f641089f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:13:04.596 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:04.596+0000 7f5f6a400640 1 --2- 172.21.15.118:0/1307064314 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f5f641017a0 0x7f5f64101c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:13:04.596 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:04.596+0000 7f5f6a400640 1 --2- 172.21.15.118:0/1307064314 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5f641005b0 0x7f5f641009d0 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:13:04.597 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:04.596+0000 7f5f6a400640 1 -- 172.21.15.118:0/1307064314 >> 172.21.15.118:0/1307064314 conn(0x7f5f640fbb40 msgr2=0x7f5f640fdf60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:13:04.597 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:04.596+0000 7f5f6a400640 1 -- 172.21.15.118:0/1307064314 shutdown_connections 2024-03-24T00:13:04.597 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:04.596+0000 7f5f6a400640 1 -- 172.21.15.118:0/1307064314 wait complete. 2024-03-24T00:13:04.599 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:04.597+0000 7f5f6a400640 1 Processor -- start 2024-03-24T00:13:04.599 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:04.597+0000 7f5f6a400640 1 -- start start 2024-03-24T00:13:04.599 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:04.598+0000 7f5f6a400640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5f641005b0 0x7f5f64197ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:13:04.599 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:04.598+0000 7f5f6a400640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f5f641017a0 0x7f5f64198000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:13:04.599 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:04.598+0000 7f5f6a400640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f5f64102160 0x7f5f6419f000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:13:04.599 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:04.598+0000 7f5f6a400640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5f6410c270 con 0x7f5f641017a0 2024-03-24T00:13:04.599 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:04.598+0000 7f5f6a400640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5f6410c0f0 con 0x7f5f641005b0 2024-03-24T00:13:04.599 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:04.598+0000 7f5f6a400640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5f6410c3f0 con 0x7f5f64102160 2024-03-24T00:13:04.599 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:04.598+0000 7f5f62a00640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f5f641017a0 0x7f5f64198000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:13:04.600 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:04.598+0000 7f5f63400640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5f641005b0 0x7f5f64197ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:13:04.600 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:04.598+0000 7f5f63400640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5f641005b0 0x7f5f64197ac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.118:33940/0 (socket says 172.21.15.118:33940) 2024-03-24T00:13:04.600 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:04.598+0000 7f5f63400640 1 -- 172.21.15.118:0/1665417053 learned_addr learned my addr 172.21.15.118:0/1665417053 (peer_addr_for_me v2:172.21.15.118:0/0) 2024-03-24T00:13:04.600 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:04.598+0000 7f5f63e00640 1 --2- 172.21.15.118:0/1665417053 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f5f64102160 0x7f5f6419f000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:13:04.600 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:04.598+0000 7f5f63400640 1 -- 172.21.15.118:0/1665417053 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f5f64102160 msgr2=0x7f5f6419f000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:13:04.600 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:04.598+0000 7f5f63400640 1 --2- 172.21.15.118:0/1665417053 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f5f64102160 0x7f5f6419f000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:13:04.601 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:04.598+0000 7f5f63400640 1 -- 172.21.15.118:0/1665417053 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f5f641017a0 msgr2=0x7f5f64198000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:13:04.601 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:04.598+0000 7f5f63400640 1 --2- 172.21.15.118:0/1665417053 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f5f641017a0 0x7f5f64198000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:13:04.601 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:04.598+0000 7f5f63400640 1 -- 172.21.15.118:0/1665417053 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5f6419f670 con 0x7f5f641005b0 2024-03-24T00:13:04.601 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:04.598+0000 7f5f63e00640 1 --2- 172.21.15.118:0/1665417053 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f5f64102160 0x7f5f6419f000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-24T00:13:04.601 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:04.599+0000 7f5f63400640 1 --2- 172.21.15.118:0/1665417053 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5f641005b0 0x7f5f64197ac0 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f5f5402c260 tx=0x7f5f5402c670 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:13:04.601 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:04.599+0000 7f5f5be00640 1 -- 172.21.15.118:0/1665417053 <== mon.2 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5f54041070 con 0x7f5f641005b0 2024-03-24T00:13:04.602 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:04.599+0000 7f5f5be00640 1 -- 172.21.15.118:0/1665417053 <== mon.2 v2:172.21.15.189:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f5f54004450 con 0x7f5f641005b0 2024-03-24T00:13:04.602 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:04.599+0000 7f5f6a400640 1 -- 172.21.15.118:0/1665417053 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5f6419f900 con 0x7f5f641005b0 2024-03-24T00:13:04.602 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:04.599+0000 7f5f5be00640 1 -- 172.21.15.118:0/1665417053 <== mon.2 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5f54034440 con 0x7f5f641005b0 2024-03-24T00:13:04.602 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:04.599+0000 7f5f6a400640 1 -- 172.21.15.118:0/1665417053 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5f6419fea0 con 0x7f5f641005b0 2024-03-24T00:13:04.604 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:04.600+0000 7f5f6a400640 1 -- 172.21.15.118:0/1665417053 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5f641a00d0 con 0x7f5f641005b0 2024-03-24T00:13:04.604 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:04.602+0000 7f5f5be00640 1 -- 172.21.15.118:0/1665417053 <== mon.2 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 16) v1 ==== 137291+0+0 (secure 0 0 0) 0x7f5f54044070 con 0x7f5f641005b0 2024-03-24T00:13:04.604 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:04.603+0000 7f5f5be00640 1 --2- 172.21.15.118:0/1665417053 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7f5f480a2880 0x7f5f480a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:13:04.604 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:04.603+0000 7f5f5be00640 1 -- 172.21.15.118:0/1665417053 <== mon.2 v2:172.21.15.189:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 7979+0+0 (secure 0 0 0) 0x7f5f540f0460 con 0x7f5f641005b0 2024-03-24T00:13:04.604 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:04.603+0000 7f5f62a00640 1 --2- 172.21.15.118:0/1665417053 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7f5f480a2880 0x7f5f480a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:13:04.604 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:04.604+0000 7f5f62a00640 1 --2- 172.21.15.118:0/1665417053 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7f5f480a2880 0x7f5f480a4d40 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f5f64198f60 tx=0x7f5f50005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:13:04.606 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:04.607+0000 7f5f5be00640 1 -- 172.21.15.118:0/1665417053 <== mon.2 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f5f540bd5b0 con 0x7f5f641005b0 2024-03-24T00:13:04.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:13:04 smithi118 ceph-mon[31535]: osdmap e86: 12 total, 12 up, 12 in 2024-03-24T00:13:04.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:13:04 smithi118 ceph-mon[31535]: mds.? [v2:172.21.15.189:6838/4074484168,v1:172.21.15.189:6839/4074484168] up:active 2024-03-24T00:13:04.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:13:04 smithi118 ceph-mon[31535]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-03-24T00:13:04.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:13:04 smithi118 ceph-mon[31535]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-03-24T00:13:04.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:13:04 smithi118 ceph-mon[31535]: pgmap v315: 97 pgs: 4 creating+activating, 1 active, 6 creating+peering, 52 active+clean, 34 unknown; 577 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 1023 B/s wr, 7 op/s 2024-03-24T00:13:04.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:13:04 smithi118 ceph-mon[31535]: from='client.? 172.21.15.118:0/3518174268' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]: dispatch 2024-03-24T00:13:04.703 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:13:04 smithi189 ceph-mon[36187]: osdmap e86: 12 total, 12 up, 12 in 2024-03-24T00:13:04.703 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:13:04 smithi189 ceph-mon[36187]: mds.? [v2:172.21.15.189:6838/4074484168,v1:172.21.15.189:6839/4074484168] up:active 2024-03-24T00:13:04.703 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:13:04 smithi189 ceph-mon[36187]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-03-24T00:13:04.703 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:13:04 smithi189 ceph-mon[36187]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-03-24T00:13:04.704 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:13:04 smithi189 ceph-mon[36187]: pgmap v315: 97 pgs: 4 creating+activating, 1 active, 6 creating+peering, 52 active+clean, 34 unknown; 577 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 1023 B/s wr, 7 op/s 2024-03-24T00:13:04.704 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:13:04 smithi189 ceph-mon[36187]: from='client.? 172.21.15.118:0/3518174268' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]: dispatch 2024-03-24T00:13:04.710 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:13:04 smithi193 ceph-mon[36878]: osdmap e86: 12 total, 12 up, 12 in 2024-03-24T00:13:04.710 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:13:04 smithi193 ceph-mon[36878]: mds.? [v2:172.21.15.189:6838/4074484168,v1:172.21.15.189:6839/4074484168] up:active 2024-03-24T00:13:04.710 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:13:04 smithi193 ceph-mon[36878]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-03-24T00:13:04.711 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:13:04 smithi193 ceph-mon[36878]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-03-24T00:13:04.711 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:13:04 smithi193 ceph-mon[36878]: pgmap v315: 97 pgs: 4 creating+activating, 1 active, 6 creating+peering, 52 active+clean, 34 unknown; 577 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 1023 B/s wr, 7 op/s 2024-03-24T00:13:04.711 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:13:04 smithi193 ceph-mon[36878]: from='client.? 172.21.15.118:0/3518174268' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]: dispatch 2024-03-24T00:13:04.847 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:04.847+0000 7f5f6a400640 1 -- 172.21.15.118:0/1665417053 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "5"} v 0) v1 -- 0x7f5f641a02a0 con 0x7f5f641005b0 2024-03-24T00:13:05.374 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:05.374+0000 7f5f5be00640 1 -- 172.21.15.118:0/1665417053 <== mon.2 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "5"}]=0 v18) v1 ==== 105+0+0 (secure 0 0 0) 0x7f5f540bceb0 con 0x7f5f641005b0 2024-03-24T00:13:05.377 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:05.378+0000 7f5f6a400640 1 -- 172.21.15.118:0/1665417053 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7f5f480a2880 msgr2=0x7f5f480a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:13:05.378 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:05.378+0000 7f5f6a400640 1 --2- 172.21.15.118:0/1665417053 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7f5f480a2880 0x7f5f480a4d40 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f5f64198f60 tx=0x7f5f50005f90 comp rx=0 tx=0).stop 2024-03-24T00:13:05.378 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:05.378+0000 7f5f6a400640 1 -- 172.21.15.118:0/1665417053 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5f641005b0 msgr2=0x7f5f64197ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:13:05.378 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:05.378+0000 7f5f6a400640 1 --2- 172.21.15.118:0/1665417053 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5f641005b0 0x7f5f64197ac0 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f5f5402c260 tx=0x7f5f5402c670 comp rx=0 tx=0).stop 2024-03-24T00:13:05.378 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:05.378+0000 7f5f6a400640 1 -- 172.21.15.118:0/1665417053 shutdown_connections 2024-03-24T00:13:05.378 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:05.378+0000 7f5f6a400640 1 --2- 172.21.15.118:0/1665417053 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7f5f480a2880 0x7f5f480a4d40 unknown :-1 s=CLOSED pgs=165 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:13:05.378 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:05.378+0000 7f5f6a400640 1 --2- 172.21.15.118:0/1665417053 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f5f64102160 0x7f5f6419f000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:13:05.378 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:05.378+0000 7f5f6a400640 1 --2- 172.21.15.118:0/1665417053 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f5f641017a0 0x7f5f64198000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:13:05.378 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:05.378+0000 7f5f6a400640 1 --2- 172.21.15.118:0/1665417053 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5f641005b0 0x7f5f64197ac0 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:13:05.378 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:05.378+0000 7f5f6a400640 1 -- 172.21.15.118:0/1665417053 >> 172.21.15.118:0/1665417053 conn(0x7f5f640fbb40 msgr2=0x7f5f640fd5f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:13:05.378 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:05.378+0000 7f5f6a400640 1 -- 172.21.15.118:0/1665417053 shutdown_connections 2024-03-24T00:13:05.378 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:05.378+0000 7f5f6a400640 1 -- 172.21.15.118:0/1665417053 wait complete. 2024-03-24T00:13:05.403 DEBUG:teuthology.orchestra.run.smithi118:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs allow_standby_replay true 2024-03-24T00:13:05.595 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:05.595+0000 7f2207200640 1 -- 172.21.15.118:0/3109493766 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f2208103150 msgr2=0x7f22081099e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:13:05.596 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:05.595+0000 7f2207200640 1 --2- 172.21.15.118:0/3109493766 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f2208103150 0x7f22081099e0 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f2200009ca0 tx=0x7f220002c0f0 comp rx=0 tx=0).stop 2024-03-24T00:13:05.596 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:05.596+0000 7f2207200640 1 -- 172.21.15.118:0/3109493766 shutdown_connections 2024-03-24T00:13:05.596 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:05.596+0000 7f2207200640 1 --2- 172.21.15.118:0/3109493766 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f2208103150 0x7f22081099e0 unknown :-1 s=CLOSED pgs=161 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:13:05.596 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:05.596+0000 7f2207200640 1 --2- 172.21.15.118:0/3109493766 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f2208102790 0x7f2208102c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:13:05.596 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:05.596+0000 7f2207200640 1 --2- 172.21.15.118:0/3109493766 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f22081015a0 0x7f22081019c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:13:05.596 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:05.596+0000 7f2207200640 1 -- 172.21.15.118:0/3109493766 >> 172.21.15.118:0/3109493766 conn(0x7f22080fcb10 msgr2=0x7f22080fef50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:13:05.596 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:05.596+0000 7f2207200640 1 -- 172.21.15.118:0/3109493766 shutdown_connections 2024-03-24T00:13:05.596 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:05.596+0000 7f2207200640 1 -- 172.21.15.118:0/3109493766 wait complete. 2024-03-24T00:13:05.598 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:05.597+0000 7f2207200640 1 Processor -- start 2024-03-24T00:13:05.598 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:05.597+0000 7f2207200640 1 -- start start 2024-03-24T00:13:05.598 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:05.597+0000 7f2207200640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f22081015a0 0x7f2208199aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:13:05.599 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:05.597+0000 7f2207200640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f2208102790 0x7f2208199fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:13:05.599 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:05.597+0000 7f2207200640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f2208103150 0x7f22081a1050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:13:05.599 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:05.597+0000 7f2207200640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f220810d230 con 0x7f2208102790 2024-03-24T00:13:05.599 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:05.597+0000 7f2207200640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f220810d0b0 con 0x7f22081015a0 2024-03-24T00:13:05.599 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:05.597+0000 7f2207200640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f220810d3b0 con 0x7f2208103150 2024-03-24T00:13:05.599 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:05.598+0000 7f2205e00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f22081015a0 0x7f2208199aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:13:05.599 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:05.598+0000 7f2205400640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f2208102790 0x7f2208199fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:13:05.599 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:05.598+0000 7f2205e00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f22081015a0 0x7f2208199aa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.118:33970/0 (socket says 172.21.15.118:33970) 2024-03-24T00:13:05.599 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:05.598+0000 7f2205e00640 1 -- 172.21.15.118:0/3554217061 learned_addr learned my addr 172.21.15.118:0/3554217061 (peer_addr_for_me v2:172.21.15.118:0/0) 2024-03-24T00:13:05.599 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:05.598+0000 7f2206800640 1 --2- 172.21.15.118:0/3554217061 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f2208103150 0x7f22081a1050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:13:05.599 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:05.598+0000 7f2205e00640 1 -- 172.21.15.118:0/3554217061 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f2208103150 msgr2=0x7f22081a1050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:13:05.599 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:05.598+0000 7f2205e00640 1 --2- 172.21.15.118:0/3554217061 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f2208103150 0x7f22081a1050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:13:05.599 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:05.598+0000 7f2205e00640 1 -- 172.21.15.118:0/3554217061 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f2208102790 msgr2=0x7f2208199fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:13:05.600 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:05.598+0000 7f2205e00640 1 --2- 172.21.15.118:0/3554217061 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f2208102790 0x7f2208199fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:13:05.600 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:05.598+0000 7f2205e00640 1 -- 172.21.15.118:0/3554217061 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f22081a16c0 con 0x7f22081015a0 2024-03-24T00:13:05.600 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:05.598+0000 7f2205e00640 1 --2- 172.21.15.118:0/3554217061 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f22081015a0 0x7f2208199aa0 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f21f800e9c0 tx=0x7f21f800ee80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:13:05.600 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:05.599+0000 7f21fea00640 1 -- 172.21.15.118:0/3554217061 <== mon.2 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21f800cd80 con 0x7f22081015a0 2024-03-24T00:13:05.600 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:05.599+0000 7f21fea00640 1 -- 172.21.15.118:0/3554217061 <== mon.2 v2:172.21.15.189:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f21f80035f0 con 0x7f22081015a0 2024-03-24T00:13:05.600 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:05.599+0000 7f21fea00640 1 -- 172.21.15.118:0/3554217061 <== mon.2 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f21f8010690 con 0x7f22081015a0 2024-03-24T00:13:05.600 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:05.599+0000 7f2207200640 1 -- 172.21.15.118:0/3554217061 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f22081a19b0 con 0x7f22081015a0 2024-03-24T00:13:05.600 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:05.599+0000 7f2207200640 1 -- 172.21.15.118:0/3554217061 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f22081a1f00 con 0x7f22081015a0 2024-03-24T00:13:05.600 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:05.600+0000 7f2207200640 1 -- 172.21.15.118:0/3554217061 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f22081a2190 con 0x7f22081015a0 2024-03-24T00:13:05.601 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:05.602+0000 7f21fea00640 1 -- 172.21.15.118:0/3554217061 <== mon.2 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 16) v1 ==== 137291+0+0 (secure 0 0 0) 0x7f21f80037a0 con 0x7f22081015a0 2024-03-24T00:13:05.603 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:05.603+0000 7f21fea00640 1 --2- 172.21.15.118:0/3554217061 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7f21ec0a2800 0x7f21ec0a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:13:05.603 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:05.603+0000 7f21fea00640 1 -- 172.21.15.118:0/3554217061 <== mon.2 v2:172.21.15.189:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 7979+0+0 (secure 0 0 0) 0x7f21f80cee10 con 0x7f22081015a0 2024-03-24T00:13:05.603 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:05.603+0000 7f2205400640 1 --2- 172.21.15.118:0/3554217061 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7f21ec0a2800 0x7f21ec0a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:13:05.603 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:05.604+0000 7f2205400640 1 --2- 172.21.15.118:0/3554217061 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7f21ec0a2800 0x7f21ec0a4cc0 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7f220819afb0 tx=0x7f21f4005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:13:05.608 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:05.608+0000 7f21fea00640 1 -- 172.21.15.118:0/3554217061 <== mon.2 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f21f80d2030 con 0x7f22081015a0 2024-03-24T00:13:05.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:13:05 smithi118 ceph-mon[31535]: Health check cleared: POOL_APP_NOT_ENABLED (was: 2 pool(s) do not have an application enabled) 2024-03-24T00:13:05.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:13:05 smithi118 ceph-mon[31535]: Cluster is now healthy 2024-03-24T00:13:05.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:13:05 smithi118 ceph-mon[31535]: from='client.? 172.21.15.118:0/3518174268' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]': finished 2024-03-24T00:13:05.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:13:05 smithi118 ceph-mon[31535]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-03-24T00:13:05.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:13:05 smithi118 ceph-mon[31535]: from='client.? 172.21.15.118:0/1665417053' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "5"}]: dispatch 2024-03-24T00:13:05.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:13:05 smithi118 ceph-mon[31535]: from='client.? ' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "5"}]: dispatch 2024-03-24T00:13:05.703 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:13:05 smithi189 ceph-mon[36187]: Health check cleared: POOL_APP_NOT_ENABLED (was: 2 pool(s) do not have an application enabled) 2024-03-24T00:13:05.703 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:13:05 smithi189 ceph-mon[36187]: Cluster is now healthy 2024-03-24T00:13:05.703 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:13:05 smithi189 ceph-mon[36187]: from='client.? 172.21.15.118:0/3518174268' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]': finished 2024-03-24T00:13:05.703 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:13:05 smithi189 ceph-mon[36187]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-03-24T00:13:05.703 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:13:05 smithi189 ceph-mon[36187]: from='client.? 172.21.15.118:0/1665417053' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "5"}]: dispatch 2024-03-24T00:13:05.704 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:13:05 smithi189 ceph-mon[36187]: from='client.? ' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "5"}]: dispatch 2024-03-24T00:13:05.711 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:13:05 smithi193 ceph-mon[36878]: Health check cleared: POOL_APP_NOT_ENABLED (was: 2 pool(s) do not have an application enabled) 2024-03-24T00:13:05.711 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:13:05 smithi193 ceph-mon[36878]: Cluster is now healthy 2024-03-24T00:13:05.711 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:13:05 smithi193 ceph-mon[36878]: from='client.? 172.21.15.118:0/3518174268' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]': finished 2024-03-24T00:13:05.711 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:13:05 smithi193 ceph-mon[36878]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-03-24T00:13:05.711 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:13:05 smithi193 ceph-mon[36878]: from='client.? 172.21.15.118:0/1665417053' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "5"}]: dispatch 2024-03-24T00:13:05.711 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:13:05 smithi193 ceph-mon[36878]: from='client.? ' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "5"}]: dispatch 2024-03-24T00:13:05.849 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:05.849+0000 7f2207200640 1 -- 172.21.15.118:0/3554217061 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"} v 0) v1 -- 0x7f22081a2360 con 0x7f22081015a0 2024-03-24T00:13:06.203 INFO:journalctl@ceph.mds.b.smithi189.stdout:Mar 24 00:13:05 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-b[66975]: 2024-03-24T00:13:05.710+0000 7f4e07ab9700 -1 mds.pinger is_rank_lagging: rank=0 was never sent ping request. 2024-03-24T00:13:06.275 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:06.275+0000 7f21fea00640 1 -- 172.21.15.118:0/3554217061 <== mon.2 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]=0 v20) v1 ==== 121+0+0 (secure 0 0 0) 0x7f21f809bf60 con 0x7f22081015a0 2024-03-24T00:13:06.279 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:06.279+0000 7f2207200640 1 -- 172.21.15.118:0/3554217061 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7f21ec0a2800 msgr2=0x7f21ec0a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:13:06.279 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:06.279+0000 7f2207200640 1 --2- 172.21.15.118:0/3554217061 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7f21ec0a2800 0x7f21ec0a4cc0 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7f220819afb0 tx=0x7f21f4005f90 comp rx=0 tx=0).stop 2024-03-24T00:13:06.279 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:06.279+0000 7f2207200640 1 -- 172.21.15.118:0/3554217061 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f22081015a0 msgr2=0x7f2208199aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:13:06.279 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:06.279+0000 7f2207200640 1 --2- 172.21.15.118:0/3554217061 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f22081015a0 0x7f2208199aa0 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f21f800e9c0 tx=0x7f21f800ee80 comp rx=0 tx=0).stop 2024-03-24T00:13:06.279 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:06.279+0000 7f2207200640 1 -- 172.21.15.118:0/3554217061 shutdown_connections 2024-03-24T00:13:06.279 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:06.279+0000 7f2207200640 1 --2- 172.21.15.118:0/3554217061 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7f21ec0a2800 0x7f21ec0a4cc0 unknown :-1 s=CLOSED pgs=166 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:13:06.279 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:06.279+0000 7f2207200640 1 --2- 172.21.15.118:0/3554217061 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f2208103150 0x7f22081a1050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:13:06.279 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:06.279+0000 7f2207200640 1 --2- 172.21.15.118:0/3554217061 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f2208102790 0x7f2208199fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:13:06.280 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:06.279+0000 7f2207200640 1 --2- 172.21.15.118:0/3554217061 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f22081015a0 0x7f2208199aa0 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:13:06.280 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:06.279+0000 7f2207200640 1 -- 172.21.15.118:0/3554217061 >> 172.21.15.118:0/3554217061 conn(0x7f22080fcb10 msgr2=0x7f22080738d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:13:06.280 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:06.279+0000 7f2207200640 1 -- 172.21.15.118:0/3554217061 shutdown_connections 2024-03-24T00:13:06.280 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:06.279+0000 7f2207200640 1 -- 172.21.15.118:0/3554217061 wait complete. 2024-03-24T00:13:06.306 DEBUG:teuthology.orchestra.run.smithi118:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs session_timeout 300 2024-03-24T00:13:06.538 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:06.538+0000 7f97cd000640 1 -- 172.21.15.118:0/2980261206 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f97c8102120 msgr2=0x7f97c8104510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:13:06.539 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:06.538+0000 7f97cd000640 1 --2- 172.21.15.118:0/2980261206 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f97c8102120 0x7f97c8104510 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f97b8009900 tx=0x7f97b802bc30 comp rx=0 tx=0).stop 2024-03-24T00:13:06.539 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:06.539+0000 7f97cd000640 1 -- 172.21.15.118:0/2980261206 shutdown_connections 2024-03-24T00:13:06.539 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:06.539+0000 7f97cd000640 1 --2- 172.21.15.118:0/2980261206 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f97c8107380 0x7f97c81097d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:13:06.539 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:06.539+0000 7f97cd000640 1 --2- 172.21.15.118:0/2980261206 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f97c8104a50 0x7f97c8106e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:13:06.539 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:06.539+0000 7f97cd000640 1 --2- 172.21.15.118:0/2980261206 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f97c8102120 0x7f97c8104510 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:13:06.539 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:06.539+0000 7f97cd000640 1 -- 172.21.15.118:0/2980261206 >> 172.21.15.118:0/2980261206 conn(0x7f97c80fbb30 msgr2=0x7f97c80fdf50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:13:06.539 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:06.539+0000 7f97cd000640 1 -- 172.21.15.118:0/2980261206 shutdown_connections 2024-03-24T00:13:06.539 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:06.539+0000 7f97cd000640 1 -- 172.21.15.118:0/2980261206 wait complete. 2024-03-24T00:13:06.540 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:06.540+0000 7f97cd000640 1 Processor -- start 2024-03-24T00:13:06.540 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:06.540+0000 7f97cd000640 1 -- start start 2024-03-24T00:13:06.541 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:06.541+0000 7f97cd000640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f97c8102120 0x7f97c8101a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:13:06.541 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:06.541+0000 7f97cd000640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f97c8104a50 0x7f97c81000d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:13:06.541 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:06.541+0000 7f97cd000640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f97c8107380 0x7f97c8100610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:13:06.541 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:06.541+0000 7f97cd000640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f97c810d390 con 0x7f97c8104a50 2024-03-24T00:13:06.541 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:06.541+0000 7f97cd000640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f97c810d210 con 0x7f97c8107380 2024-03-24T00:13:06.542 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:06.541+0000 7f97cd000640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f97c810d510 con 0x7f97c8102120 2024-03-24T00:13:06.542 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:06.541+0000 7f97bfe00640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f97c8104a50 0x7f97c81000d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:13:06.542 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:06.541+0000 7f97c6800640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f97c8107380 0x7f97c8100610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:13:06.542 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:06.541+0000 7f97c5e00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f97c8102120 0x7f97c8101a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:13:06.542 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:06.541+0000 7f97c6800640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f97c8107380 0x7f97c8100610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.118:33980/0 (socket says 172.21.15.118:33980) 2024-03-24T00:13:06.542 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:06.541+0000 7f97c6800640 1 -- 172.21.15.118:0/491634331 learned_addr learned my addr 172.21.15.118:0/491634331 (peer_addr_for_me v2:172.21.15.118:0/0) 2024-03-24T00:13:06.542 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:06.542+0000 7f97bfe00640 1 -- 172.21.15.118:0/491634331 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f97c8102120 msgr2=0x7f97c8101a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:13:06.542 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:06.542+0000 7f97bfe00640 1 --2- 172.21.15.118:0/491634331 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f97c8102120 0x7f97c8101a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:13:06.542 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:06.542+0000 7f97bfe00640 1 -- 172.21.15.118:0/491634331 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f97c8107380 msgr2=0x7f97c8100610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:13:06.542 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:06.542+0000 7f97bfe00640 1 --2- 172.21.15.118:0/491634331 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f97c8107380 0x7f97c8100610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:13:06.542 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:06.542+0000 7f97bfe00640 1 -- 172.21.15.118:0/491634331 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f97c8100ea0 con 0x7f97c8104a50 2024-03-24T00:13:06.542 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:06.542+0000 7f97c6800640 1 --2- 172.21.15.118:0/491634331 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f97c8107380 0x7f97c8100610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T00:13:06.542 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:06.542+0000 7f97c5e00640 1 --2- 172.21.15.118:0/491634331 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f97c8102120 0x7f97c8101a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T00:13:06.542 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:06.542+0000 7f97bfe00640 1 --2- 172.21.15.118:0/491634331 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f97c8104a50 0x7f97c81000d0 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f97b000e9c0 tx=0x7f97b000ee80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:13:06.543 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:06.542+0000 7f97bea00640 1 -- 172.21.15.118:0/491634331 <== mon.0 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f97b000ccc0 con 0x7f97c8104a50 2024-03-24T00:13:06.543 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:06.542+0000 7f97cd000640 1 -- 172.21.15.118:0/491634331 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f97c81a1b30 con 0x7f97c8104a50 2024-03-24T00:13:06.544 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:06.542+0000 7f97bea00640 1 -- 172.21.15.118:0/491634331 <== mon.0 v2:172.21.15.118:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f97b0005020 con 0x7f97c8104a50 2024-03-24T00:13:06.544 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:06.542+0000 7f97bea00640 1 -- 172.21.15.118:0/491634331 <== mon.0 v2:172.21.15.118:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f97b0004a00 con 0x7f97c8104a50 2024-03-24T00:13:06.544 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:06.542+0000 7f97cd000640 1 -- 172.21.15.118:0/491634331 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f97c81a2080 con 0x7f97c8104a50 2024-03-24T00:13:06.544 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:06.544+0000 7f97cd000640 1 -- 172.21.15.118:0/491634331 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f97c80fd620 con 0x7f97c8104a50 2024-03-24T00:13:06.545 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:06.545+0000 7f97bea00640 1 -- 172.21.15.118:0/491634331 <== mon.0 v2:172.21.15.118:3300/0 4 ==== mgrmap(e 16) v1 ==== 137291+0+0 (secure 0 0 0) 0x7f97b00051d0 con 0x7f97c8104a50 2024-03-24T00:13:06.546 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:06.547+0000 7f97bea00640 1 --2- 172.21.15.118:0/491634331 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7f97ac0a2800 0x7f97ac0a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:13:06.547 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:06.547+0000 7f97bea00640 1 -- 172.21.15.118:0/491634331 <== mon.0 v2:172.21.15.118:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 7979+0+0 (secure 0 0 0) 0x7f97b00ce960 con 0x7f97c8104a50 2024-03-24T00:13:06.547 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:06.547+0000 7f97c5e00640 1 --2- 172.21.15.118:0/491634331 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7f97ac0a2800 0x7f97ac0a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:13:06.547 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:06.548+0000 7f97c5e00640 1 --2- 172.21.15.118:0/491634331 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7f97ac0a2800 0x7f97ac0a4cc0 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7f97b802c240 tx=0x7f97b802df90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:13:06.551 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:06.551+0000 7f97bea00640 1 -- 172.21.15.118:0/491634331 <== mon.0 v2:172.21.15.118:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f97b009bab0 con 0x7f97c8104a50 2024-03-24T00:13:06.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:13:06 smithi118 ceph-mon[31535]: Health check failed: 1 filesystem is online with fewer MDS than max_mds (MDS_UP_LESS_THAN_MAX) 2024-03-24T00:13:06.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:13:06 smithi118 ceph-mon[31535]: from='client.? ' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "5"}]': finished 2024-03-24T00:13:06.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:13:06 smithi118 ceph-mon[31535]: daemon mds.g assigned to filesystem cephfs as rank 1 (now has 2 ranks) 2024-03-24T00:13:06.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:13:06 smithi118 ceph-mon[31535]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-03-24T00:13:06.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:13:06 smithi118 ceph-mon[31535]: fsmap cephfs:2 {0=b=up:active,1=g=up:creating} 10 up:standby 2024-03-24T00:13:06.697 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:13:06 smithi118 ceph-mon[31535]: daemon mds.g is now active in filesystem cephfs as rank 1 2024-03-24T00:13:06.697 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:13:06 smithi118 ceph-mon[31535]: pgmap v316: 97 pgs: 4 creating+activating, 1 active, 6 creating+peering, 78 active+clean, 8 unknown; 579 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s wr, 7 op/s 2024-03-24T00:13:06.697 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:13:06 smithi118 ceph-mon[31535]: from='client.? 172.21.15.118:0/3554217061' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]: dispatch 2024-03-24T00:13:06.697 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:13:06 smithi118 ceph-mon[31535]: from='client.? ' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]: dispatch 2024-03-24T00:13:06.697 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:13:06 smithi118 ceph-mon[31535]: from='client.? ' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]': finished 2024-03-24T00:13:06.697 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:13:06 smithi118 ceph-mon[31535]: mds.? [v2:172.21.15.118:6838/3285788503,v1:172.21.15.118:6839/3285788503] up:active 2024-03-24T00:13:06.697 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:13:06 smithi118 ceph-mon[31535]: daemon mds.c assigned to filesystem cephfs as rank 2 (now has 3 ranks) 2024-03-24T00:13:06.697 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:13:06 smithi118 ceph-mon[31535]: fsmap cephfs:2 {0=b=up:active,1=g=up:active} 2 up:standby-replay 8 up:standby 2024-03-24T00:13:06.697 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:13:06 smithi118 ceph-mon[31535]: fsmap cephfs:3 {0=b=up:active,1=g=up:active,2=c=up:creating} 2 up:standby-replay 7 up:standby 2024-03-24T00:13:06.703 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:13:06 smithi189 ceph-mon[36187]: Health check failed: 1 filesystem is online with fewer MDS than max_mds (MDS_UP_LESS_THAN_MAX) 2024-03-24T00:13:06.703 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:13:06 smithi189 ceph-mon[36187]: from='client.? ' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "5"}]': finished 2024-03-24T00:13:06.704 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:13:06 smithi189 ceph-mon[36187]: daemon mds.g assigned to filesystem cephfs as rank 1 (now has 2 ranks) 2024-03-24T00:13:06.704 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:13:06 smithi189 ceph-mon[36187]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-03-24T00:13:06.704 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:13:06 smithi189 ceph-mon[36187]: fsmap cephfs:2 {0=b=up:active,1=g=up:creating} 10 up:standby 2024-03-24T00:13:06.704 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:13:06 smithi189 ceph-mon[36187]: daemon mds.g is now active in filesystem cephfs as rank 1 2024-03-24T00:13:06.704 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:13:06 smithi189 ceph-mon[36187]: pgmap v316: 97 pgs: 4 creating+activating, 1 active, 6 creating+peering, 78 active+clean, 8 unknown; 579 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s wr, 7 op/s 2024-03-24T00:13:06.704 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:13:06 smithi189 ceph-mon[36187]: from='client.? 172.21.15.118:0/3554217061' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]: dispatch 2024-03-24T00:13:06.704 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:13:06 smithi189 ceph-mon[36187]: from='client.? ' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]: dispatch 2024-03-24T00:13:06.704 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:13:06 smithi189 ceph-mon[36187]: from='client.? ' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]': finished 2024-03-24T00:13:06.704 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:13:06 smithi189 ceph-mon[36187]: mds.? [v2:172.21.15.118:6838/3285788503,v1:172.21.15.118:6839/3285788503] up:active 2024-03-24T00:13:06.704 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:13:06 smithi189 ceph-mon[36187]: daemon mds.c assigned to filesystem cephfs as rank 2 (now has 3 ranks) 2024-03-24T00:13:06.704 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:13:06 smithi189 ceph-mon[36187]: fsmap cephfs:2 {0=b=up:active,1=g=up:active} 2 up:standby-replay 8 up:standby 2024-03-24T00:13:06.704 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:13:06 smithi189 ceph-mon[36187]: fsmap cephfs:3 {0=b=up:active,1=g=up:active,2=c=up:creating} 2 up:standby-replay 7 up:standby 2024-03-24T00:13:06.711 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:13:06 smithi193 ceph-mon[36878]: Health check failed: 1 filesystem is online with fewer MDS than max_mds (MDS_UP_LESS_THAN_MAX) 2024-03-24T00:13:06.711 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:13:06 smithi193 ceph-mon[36878]: from='client.? ' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "5"}]': finished 2024-03-24T00:13:06.711 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:13:06 smithi193 ceph-mon[36878]: daemon mds.g assigned to filesystem cephfs as rank 1 (now has 2 ranks) 2024-03-24T00:13:06.711 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:13:06 smithi193 ceph-mon[36878]: fsmap cephfs:1 {0=b=up:active} 11 up:standby 2024-03-24T00:13:06.711 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:13:06 smithi193 ceph-mon[36878]: fsmap cephfs:2 {0=b=up:active,1=g=up:creating} 10 up:standby 2024-03-24T00:13:06.711 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:13:06 smithi193 ceph-mon[36878]: daemon mds.g is now active in filesystem cephfs as rank 1 2024-03-24T00:13:06.711 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:13:06 smithi193 ceph-mon[36878]: pgmap v316: 97 pgs: 4 creating+activating, 1 active, 6 creating+peering, 78 active+clean, 8 unknown; 579 KiB data, 343 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s wr, 7 op/s 2024-03-24T00:13:06.711 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:13:06 smithi193 ceph-mon[36878]: from='client.? 172.21.15.118:0/3554217061' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]: dispatch 2024-03-24T00:13:06.711 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:13:06 smithi193 ceph-mon[36878]: from='client.? ' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]: dispatch 2024-03-24T00:13:06.711 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:13:06 smithi193 ceph-mon[36878]: from='client.? ' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]': finished 2024-03-24T00:13:06.711 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:13:06 smithi193 ceph-mon[36878]: mds.? [v2:172.21.15.118:6838/3285788503,v1:172.21.15.118:6839/3285788503] up:active 2024-03-24T00:13:06.712 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:13:06 smithi193 ceph-mon[36878]: daemon mds.c assigned to filesystem cephfs as rank 2 (now has 3 ranks) 2024-03-24T00:13:06.712 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:13:06 smithi193 ceph-mon[36878]: fsmap cephfs:2 {0=b=up:active,1=g=up:active} 2 up:standby-replay 8 up:standby 2024-03-24T00:13:06.712 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:13:06 smithi193 ceph-mon[36878]: fsmap cephfs:3 {0=b=up:active,1=g=up:active,2=c=up:creating} 2 up:standby-replay 7 up:standby 2024-03-24T00:13:06.792 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:06.793+0000 7f97cd000640 1 -- 172.21.15.118:0/491634331 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"} v 0) v1 -- 0x7f97c80622a0 con 0x7f97c8104a50 2024-03-24T00:13:07.308 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:07.308+0000 7f97bea00640 1 -- 172.21.15.118:0/491634331 <== mon.0 v2:172.21.15.118:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]=0 v22) v1 ==== 115+0+0 (secure 0 0 0) 0x7f97b009b3b0 con 0x7f97c8104a50 2024-03-24T00:13:07.312 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:07.312+0000 7f97cd000640 1 -- 172.21.15.118:0/491634331 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7f97ac0a2800 msgr2=0x7f97ac0a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:13:07.312 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:07.312+0000 7f97cd000640 1 --2- 172.21.15.118:0/491634331 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7f97ac0a2800 0x7f97ac0a4cc0 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7f97b802c240 tx=0x7f97b802df90 comp rx=0 tx=0).stop 2024-03-24T00:13:07.312 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:07.312+0000 7f97cd000640 1 -- 172.21.15.118:0/491634331 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f97c8104a50 msgr2=0x7f97c81000d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:13:07.312 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:07.312+0000 7f97cd000640 1 --2- 172.21.15.118:0/491634331 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f97c8104a50 0x7f97c81000d0 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f97b000e9c0 tx=0x7f97b000ee80 comp rx=0 tx=0).stop 2024-03-24T00:13:07.312 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:07.312+0000 7f97cd000640 1 -- 172.21.15.118:0/491634331 shutdown_connections 2024-03-24T00:13:07.312 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:07.312+0000 7f97cd000640 1 --2- 172.21.15.118:0/491634331 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7f97ac0a2800 0x7f97ac0a4cc0 unknown :-1 s=CLOSED pgs=167 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:13:07.312 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:07.312+0000 7f97cd000640 1 --2- 172.21.15.118:0/491634331 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f97c8107380 0x7f97c8100610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:13:07.312 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:07.312+0000 7f97cd000640 1 --2- 172.21.15.118:0/491634331 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f97c8104a50 0x7f97c81000d0 unknown :-1 s=CLOSED pgs=162 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:13:07.312 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:07.312+0000 7f97cd000640 1 --2- 172.21.15.118:0/491634331 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f97c8102120 0x7f97c8101a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:13:07.312 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:07.312+0000 7f97cd000640 1 -- 172.21.15.118:0/491634331 >> 172.21.15.118:0/491634331 conn(0x7f97c80fbb30 msgr2=0x7f97c8102a60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:13:07.312 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:07.312+0000 7f97cd000640 1 -- 172.21.15.118:0/491634331 shutdown_connections 2024-03-24T00:13:07.312 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:07.312+0000 7f97cd000640 1 -- 172.21.15.118:0/491634331 wait complete. 2024-03-24T00:13:07.338 DEBUG:tasks.cephfs.filesystem:Creating {'create': 2} subvols for filesystem 'cephfs' 2024-03-24T00:13:07.338 DEBUG:teuthology.orchestra.run.smithi118:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolumegroup create cephfs qa 2024-03-24T00:13:07.528 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:07.528+0000 7f08b1600640 1 -- 172.21.15.118:0/3636848029 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f08ac102140 msgr2=0x7f08ac104530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:13:07.528 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:07.528+0000 7f08b1600640 1 --2- 172.21.15.118:0/3636848029 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f08ac102140 0x7f08ac104530 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f089c009900 tx=0x7f089c02bc40 comp rx=0 tx=0).stop 2024-03-24T00:13:07.528 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:07.528+0000 7f08b1600640 1 -- 172.21.15.118:0/3636848029 shutdown_connections 2024-03-24T00:13:07.528 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:07.528+0000 7f08b1600640 1 --2- 172.21.15.118:0/3636848029 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f08ac1073a0 0x7f08ac1097f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:13:07.528 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:07.528+0000 7f08b1600640 1 --2- 172.21.15.118:0/3636848029 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f08ac104a70 0x7f08ac106e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:13:07.528 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:07.528+0000 7f08b1600640 1 --2- 172.21.15.118:0/3636848029 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f08ac102140 0x7f08ac104530 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:13:07.528 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:07.528+0000 7f08b1600640 1 -- 172.21.15.118:0/3636848029 >> 172.21.15.118:0/3636848029 conn(0x7f08ac0fbb50 msgr2=0x7f08ac0fdf70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:13:07.528 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:07.528+0000 7f08b1600640 1 -- 172.21.15.118:0/3636848029 shutdown_connections 2024-03-24T00:13:07.528 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:07.528+0000 7f08b1600640 1 -- 172.21.15.118:0/3636848029 wait complete. 2024-03-24T00:13:07.529 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:07.530+0000 7f08b1600640 1 Processor -- start 2024-03-24T00:13:07.529 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:07.530+0000 7f08b1600640 1 -- start start 2024-03-24T00:13:07.531 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:07.530+0000 7f08b1600640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f08ac102140 0x7f08ac197990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:13:07.531 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:07.530+0000 7f08b1600640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f08ac104a70 0x7f08ac197ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:13:07.531 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:07.530+0000 7f08b1600640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f08ac1073a0 0x7f08ac19ef40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:13:07.531 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:07.530+0000 7f08b1600640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f08ac10d3b0 con 0x7f08ac102140 2024-03-24T00:13:07.531 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:07.530+0000 7f08b1600640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f08ac10d230 con 0x7f08ac104a70 2024-03-24T00:13:07.531 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:07.530+0000 7f08b1600640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f08ac10d530 con 0x7f08ac1073a0 2024-03-24T00:13:07.532 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:07.530+0000 7f08aaa00640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f08ac102140 0x7f08ac197990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:13:07.532 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:07.530+0000 7f08aa000640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f08ac104a70 0x7f08ac197ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:13:07.532 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:07.530+0000 7f08aa000640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f08ac104a70 0x7f08ac197ed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.118:34000/0 (socket says 172.21.15.118:34000) 2024-03-24T00:13:07.532 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:07.530+0000 7f08aa000640 1 -- 172.21.15.118:0/2365709365 learned_addr learned my addr 172.21.15.118:0/2365709365 (peer_addr_for_me v2:172.21.15.118:0/0) 2024-03-24T00:13:07.532 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:07.531+0000 7f08ab400640 1 --2- 172.21.15.118:0/2365709365 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f08ac1073a0 0x7f08ac19ef40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:13:07.532 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:07.531+0000 7f08aa000640 1 -- 172.21.15.118:0/2365709365 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f08ac1073a0 msgr2=0x7f08ac19ef40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:13:07.532 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:07.531+0000 7f08aa000640 1 --2- 172.21.15.118:0/2365709365 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f08ac1073a0 0x7f08ac19ef40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:13:07.532 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:07.531+0000 7f08aa000640 1 -- 172.21.15.118:0/2365709365 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f08ac102140 msgr2=0x7f08ac197990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:13:07.532 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:07.531+0000 7f08aa000640 1 --2- 172.21.15.118:0/2365709365 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f08ac102140 0x7f08ac197990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:13:07.532 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:07.531+0000 7f08aa000640 1 -- 172.21.15.118:0/2365709365 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f08ac19f5b0 con 0x7f08ac104a70 2024-03-24T00:13:07.532 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:07.531+0000 7f08aaa00640 1 --2- 172.21.15.118:0/2365709365 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f08ac102140 0x7f08ac197990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T00:13:07.532 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:07.531+0000 7f08ab400640 1 --2- 172.21.15.118:0/2365709365 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f08ac1073a0 0x7f08ac19ef40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T00:13:07.532 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:07.531+0000 7f08aa000640 1 --2- 172.21.15.118:0/2365709365 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f08ac104a70 0x7f08ac197ed0 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f089800e9c0 tx=0x7f089800ee80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:13:07.532 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:07.531+0000 7f08a3400640 1 -- 172.21.15.118:0/2365709365 <== mon.2 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f089800cd80 con 0x7f08ac104a70 2024-03-24T00:13:07.533 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:07.531+0000 7f08a3400640 1 -- 172.21.15.118:0/2365709365 <== mon.2 v2:172.21.15.189:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f0898005500 con 0x7f08ac104a70 2024-03-24T00:13:07.533 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:07.531+0000 7f08a3400640 1 -- 172.21.15.118:0/2365709365 <== mon.2 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0898010690 con 0x7f08ac104a70 2024-03-24T00:13:07.533 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:07.532+0000 7f08b1600640 1 -- 172.21.15.118:0/2365709365 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f08ac19f8a0 con 0x7f08ac104a70 2024-03-24T00:13:07.533 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:07.532+0000 7f08b1600640 1 -- 172.21.15.118:0/2365709365 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f08ac19fe40 con 0x7f08ac104a70 2024-03-24T00:13:07.533 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:07.533+0000 7f08b1600640 1 -- 172.21.15.118:0/2365709365 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f08ac0fd640 con 0x7f08ac104a70 2024-03-24T00:13:07.535 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:07.535+0000 7f08a3400640 1 -- 172.21.15.118:0/2365709365 <== mon.2 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 16) v1 ==== 137291+0+0 (secure 0 0 0) 0x7f08980046c0 con 0x7f08ac104a70 2024-03-24T00:13:07.536 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:07.537+0000 7f08a3400640 1 --2- 172.21.15.118:0/2365709365 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7f08900a2880 0x7f08900a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:13:07.536 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:07.537+0000 7f08a3400640 1 -- 172.21.15.118:0/2365709365 <== mon.2 v2:172.21.15.189:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 7979+0+0 (secure 0 0 0) 0x7f08980cec90 con 0x7f08ac104a70 2024-03-24T00:13:07.536 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:07.537+0000 7f08aaa00640 1 --2- 172.21.15.118:0/2365709365 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7f08900a2880 0x7f08900a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:13:07.537 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:07.538+0000 7f08aaa00640 1 --2- 172.21.15.118:0/2365709365 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7f08900a2880 0x7f08900a4d40 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7f089c02c250 tx=0x7f089c02cf90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:13:07.540 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:07.541+0000 7f08a3400640 1 -- 172.21.15.118:0/2365709365 <== mon.2 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f089809bde0 con 0x7f08ac104a70 2024-03-24T00:13:07.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:13:07 smithi118 ceph-mon[31535]: daemon mds.c is now active in filesystem cephfs as rank 2 2024-03-24T00:13:07.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:13:07 smithi118 ceph-mon[31535]: from='client.? 172.21.15.118:0/491634331' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]: dispatch 2024-03-24T00:13:07.703 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:13:07 smithi189 ceph-mon[36187]: daemon mds.c is now active in filesystem cephfs as rank 2 2024-03-24T00:13:07.703 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:13:07 smithi189 ceph-mon[36187]: from='client.? 172.21.15.118:0/491634331' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]: dispatch 2024-03-24T00:13:07.710 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:13:07 smithi193 ceph-mon[36878]: daemon mds.c is now active in filesystem cephfs as rank 2 2024-03-24T00:13:07.710 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:13:07 smithi193 ceph-mon[36878]: from='client.? 172.21.15.118:0/491634331' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]: dispatch 2024-03-24T00:13:07.817 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:07.817+0000 7f08b1600640 1 -- 172.21.15.118:0/2365709365 --> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] -- mgr_command(tid 0: {"prefix": "fs subvolumegroup create", "vol_name": "cephfs", "group_name": "qa", "target": ["mon-mgr", ""]}) v1 -- 0x7f08ac061ae0 con 0x7f08900a2880 2024-03-24T00:13:08.196 INFO:journalctl@ceph.mgr.x.smithi118.stdout:Mar 24 00:13:07 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mgr-x[31762]: 2024-03-24T00:13:07.827+0000 7f62ad800700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-24T00:13:08.197 INFO:journalctl@ceph.mgr.x.smithi118.stdout:Mar 24 00:13:07 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mgr-x[31762]: 2024-03-24T00:13:07.827+0000 7f62ad800700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-24T00:13:08.197 INFO:journalctl@ceph.mgr.x.smithi118.stdout:Mar 24 00:13:07 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mgr-x[31762]: 2024-03-24T00:13:07.827+0000 7f62ad800700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-24T00:13:08.197 INFO:journalctl@ceph.mgr.x.smithi118.stdout:Mar 24 00:13:07 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mgr-x[31762]: 2024-03-24T00:13:07.827+0000 7f62ad800700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-24T00:13:08.197 INFO:journalctl@ceph.mgr.x.smithi118.stdout:Mar 24 00:13:07 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mgr-x[31762]: 2024-03-24T00:13:07.827+0000 7f62ad800700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-24T00:13:08.352 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:08.353+0000 7f08a3400640 1 -- 172.21.15.118:0/2365709365 <== mgr.14152 v2:172.21.15.118:6800/2112912943 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7f08ac061ae0 con 0x7f08900a2880 2024-03-24T00:13:08.356 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:08.356+0000 7f08b1600640 1 -- 172.21.15.118:0/2365709365 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7f08900a2880 msgr2=0x7f08900a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:13:08.356 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:08.356+0000 7f08b1600640 1 --2- 172.21.15.118:0/2365709365 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7f08900a2880 0x7f08900a4d40 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7f089c02c250 tx=0x7f089c02cf90 comp rx=0 tx=0).stop 2024-03-24T00:13:08.356 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:08.356+0000 7f08b1600640 1 -- 172.21.15.118:0/2365709365 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f08ac104a70 msgr2=0x7f08ac197ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:13:08.356 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:08.356+0000 7f08b1600640 1 --2- 172.21.15.118:0/2365709365 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f08ac104a70 0x7f08ac197ed0 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f089800e9c0 tx=0x7f089800ee80 comp rx=0 tx=0).stop 2024-03-24T00:13:08.356 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:08.356+0000 7f08b1600640 1 -- 172.21.15.118:0/2365709365 shutdown_connections 2024-03-24T00:13:08.356 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:08.356+0000 7f08b1600640 1 --2- 172.21.15.118:0/2365709365 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7f08900a2880 0x7f08900a4d40 unknown :-1 s=CLOSED pgs=168 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:13:08.356 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:08.356+0000 7f08b1600640 1 --2- 172.21.15.118:0/2365709365 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f08ac1073a0 0x7f08ac19ef40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:13:08.356 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:08.356+0000 7f08b1600640 1 --2- 172.21.15.118:0/2365709365 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f08ac104a70 0x7f08ac197ed0 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:13:08.356 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:08.356+0000 7f08b1600640 1 --2- 172.21.15.118:0/2365709365 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f08ac102140 0x7f08ac197990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:13:08.357 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:08.356+0000 7f08b1600640 1 -- 172.21.15.118:0/2365709365 >> 172.21.15.118:0/2365709365 conn(0x7f08ac0fbb50 msgr2=0x7f08ac102aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:13:08.357 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:08.356+0000 7f08b1600640 1 -- 172.21.15.118:0/2365709365 shutdown_connections 2024-03-24T00:13:08.357 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:08.356+0000 7f08b1600640 1 -- 172.21.15.118:0/2365709365 wait complete. 2024-03-24T00:13:08.375 DEBUG:teuthology.orchestra.run.smithi118:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume create cephfs sv_0 --group_name qa 2024-03-24T00:13:08.575 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:08.575+0000 7fd8e1600640 1 -- 172.21.15.118:0/3357226948 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fd8dc104ab0 msgr2=0x7fd8dc106ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:13:08.575 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:08.575+0000 7fd8e1600640 1 --2- 172.21.15.118:0/3357226948 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fd8dc104ab0 0x7fd8dc106ea0 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7fd8c8009900 tx=0x7fd8c802bc30 comp rx=0 tx=0).stop 2024-03-24T00:13:08.575 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:08.575+0000 7fd8e1600640 1 -- 172.21.15.118:0/3357226948 shutdown_connections 2024-03-24T00:13:08.575 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:08.575+0000 7fd8e1600640 1 --2- 172.21.15.118:0/3357226948 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fd8dc1073e0 0x7fd8dc1097d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:13:08.575 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:08.575+0000 7fd8e1600640 1 --2- 172.21.15.118:0/3357226948 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fd8dc104ab0 0x7fd8dc106ea0 unknown :-1 s=CLOSED pgs=163 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:13:08.575 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:08.575+0000 7fd8e1600640 1 --2- 172.21.15.118:0/3357226948 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd8dc102180 0x7fd8dc104570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:13:08.575 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:08.575+0000 7fd8e1600640 1 -- 172.21.15.118:0/3357226948 >> 172.21.15.118:0/3357226948 conn(0x7fd8dc0fbb30 msgr2=0x7fd8dc0fdf50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:13:08.575 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:08.575+0000 7fd8e1600640 1 -- 172.21.15.118:0/3357226948 shutdown_connections 2024-03-24T00:13:08.575 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:08.575+0000 7fd8e1600640 1 -- 172.21.15.118:0/3357226948 wait complete. 2024-03-24T00:13:08.577 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:08.576+0000 7fd8e1600640 1 Processor -- start 2024-03-24T00:13:08.578 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:08.576+0000 7fd8e1600640 1 -- start start 2024-03-24T00:13:08.578 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:08.576+0000 7fd8e1600640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd8dc102180 0x7fd8dc197a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:13:08.578 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:08.576+0000 7fd8e1600640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fd8dc104ab0 0x7fd8dc197f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:13:08.578 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:08.576+0000 7fd8e1600640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fd8dc1073e0 0x7fd8dc19efc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:13:08.578 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:08.577+0000 7fd8e1600640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd8dc10d370 con 0x7fd8dc104ab0 2024-03-24T00:13:08.578 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:08.577+0000 7fd8e1600640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd8dc10d1f0 con 0x7fd8dc102180 2024-03-24T00:13:08.578 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:08.577+0000 7fd8e1600640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd8dc10d4f0 con 0x7fd8dc1073e0 2024-03-24T00:13:08.578 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:08.577+0000 7fd8daa00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd8dc102180 0x7fd8dc197a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:13:08.578 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:08.577+0000 7fd8da000640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fd8dc104ab0 0x7fd8dc197f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:13:08.578 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:08.577+0000 7fd8daa00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd8dc102180 0x7fd8dc197a10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.118:34046/0 (socket says 172.21.15.118:34046) 2024-03-24T00:13:08.578 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:08.577+0000 7fd8daa00640 1 -- 172.21.15.118:0/2987811672 learned_addr learned my addr 172.21.15.118:0/2987811672 (peer_addr_for_me v2:172.21.15.118:0/0) 2024-03-24T00:13:08.578 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:08.577+0000 7fd8db400640 1 --2- 172.21.15.118:0/2987811672 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fd8dc1073e0 0x7fd8dc19efc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:13:08.579 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:08.577+0000 7fd8daa00640 1 -- 172.21.15.118:0/2987811672 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fd8dc1073e0 msgr2=0x7fd8dc19efc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:13:08.579 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:08.577+0000 7fd8daa00640 1 --2- 172.21.15.118:0/2987811672 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fd8dc1073e0 0x7fd8dc19efc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:13:08.579 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:08.577+0000 7fd8daa00640 1 -- 172.21.15.118:0/2987811672 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fd8dc104ab0 msgr2=0x7fd8dc197f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:13:08.579 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:08.577+0000 7fd8daa00640 1 --2- 172.21.15.118:0/2987811672 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fd8dc104ab0 0x7fd8dc197f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:13:08.579 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:08.577+0000 7fd8daa00640 1 -- 172.21.15.118:0/2987811672 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd8dc19f630 con 0x7fd8dc102180 2024-03-24T00:13:08.579 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:08.577+0000 7fd8daa00640 1 --2- 172.21.15.118:0/2987811672 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd8dc102180 0x7fd8dc197a10 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7fd8cc00ea60 tx=0x7fd8cc00ef20 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:13:08.579 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:08.578+0000 7fd8d3400640 1 -- 172.21.15.118:0/2987811672 <== mon.2 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd8cc00cde0 con 0x7fd8dc102180 2024-03-24T00:13:08.579 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:08.578+0000 7fd8d3400640 1 -- 172.21.15.118:0/2987811672 <== mon.2 v2:172.21.15.189:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fd8cc0035f0 con 0x7fd8dc102180 2024-03-24T00:13:08.579 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:08.578+0000 7fd8e1600640 1 -- 172.21.15.118:0/2987811672 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd8dc19f920 con 0x7fd8dc102180 2024-03-24T00:13:08.579 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:08.578+0000 7fd8d3400640 1 -- 172.21.15.118:0/2987811672 <== mon.2 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd8cc010670 con 0x7fd8dc102180 2024-03-24T00:13:08.579 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:08.578+0000 7fd8e1600640 1 -- 172.21.15.118:0/2987811672 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd8dc19fe40 con 0x7fd8dc102180 2024-03-24T00:13:08.580 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:08.579+0000 7fd8e1600640 1 -- 172.21.15.118:0/2987811672 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd8dc0fd620 con 0x7fd8dc102180 2024-03-24T00:13:08.580 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:08.581+0000 7fd8d3400640 1 -- 172.21.15.118:0/2987811672 <== mon.2 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 16) v1 ==== 137291+0+0 (secure 0 0 0) 0x7fd8cc0037a0 con 0x7fd8dc102180 2024-03-24T00:13:08.581 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:08.582+0000 7fd8d3400640 1 --2- 172.21.15.118:0/2987811672 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7fd8c00a2800 0x7fd8c00a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:13:08.582 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:08.582+0000 7fd8d3400640 1 -- 172.21.15.118:0/2987811672 <== mon.2 v2:172.21.15.189:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 7979+0+0 (secure 0 0 0) 0x7fd8cc0cec60 con 0x7fd8dc102180 2024-03-24T00:13:08.582 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:08.582+0000 7fd8da000640 1 --2- 172.21.15.118:0/2987811672 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7fd8c00a2800 0x7fd8c00a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:13:08.582 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:08.583+0000 7fd8da000640 1 --2- 172.21.15.118:0/2987811672 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7fd8c00a2800 0x7fd8c00a4cc0 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7fd8c802c170 tx=0x7fd8c802df90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:13:08.583 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:08.584+0000 7fd8d3400640 1 -- 172.21.15.118:0/2987811672 <== mon.2 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7fd8cc09bdb0 con 0x7fd8dc102180 2024-03-24T00:13:08.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:13:08 smithi118 ceph-mon[31535]: from='client.? 172.21.15.118:0/491634331' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]': finished 2024-03-24T00:13:08.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:13:08 smithi118 ceph-mon[31535]: mds.? [v2:172.21.15.193:6836/2921520053,v1:172.21.15.193:6837/2921520053] up:active 2024-03-24T00:13:08.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:13:08 smithi118 ceph-mon[31535]: daemon mds.f assigned to filesystem cephfs as rank 3 (now has 4 ranks) 2024-03-24T00:13:08.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:13:08 smithi118 ceph-mon[31535]: fsmap cephfs:3 {0=b=up:active,1=g=up:active,2=c=up:active} 2 up:standby-replay 7 up:standby 2024-03-24T00:13:08.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:13:08 smithi118 ceph-mon[31535]: fsmap cephfs:4 {0=b=up:active,1=g=up:active,2=c=up:active,3=f=up:creating} 2 up:standby-replay 6 up:standby 2024-03-24T00:13:08.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:13:08 smithi118 ceph-mon[31535]: daemon mds.f is now active in filesystem cephfs as rank 3 2024-03-24T00:13:08.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:13:08 smithi118 ceph-mon[31535]: pgmap v317: 97 pgs: 97 active+clean; 581 KiB data, 344 MiB used, 1.0 TiB / 1.0 TiB avail; 647 B/s rd, 4.3 KiB/s wr, 12 op/s 2024-03-24T00:13:08.703 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:13:08 smithi189 ceph-mon[36187]: from='client.? 172.21.15.118:0/491634331' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]': finished 2024-03-24T00:13:08.703 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:13:08 smithi189 ceph-mon[36187]: mds.? [v2:172.21.15.193:6836/2921520053,v1:172.21.15.193:6837/2921520053] up:active 2024-03-24T00:13:08.704 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:13:08 smithi189 ceph-mon[36187]: daemon mds.f assigned to filesystem cephfs as rank 3 (now has 4 ranks) 2024-03-24T00:13:08.704 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:13:08 smithi189 ceph-mon[36187]: fsmap cephfs:3 {0=b=up:active,1=g=up:active,2=c=up:active} 2 up:standby-replay 7 up:standby 2024-03-24T00:13:08.704 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:13:08 smithi189 ceph-mon[36187]: fsmap cephfs:4 {0=b=up:active,1=g=up:active,2=c=up:active,3=f=up:creating} 2 up:standby-replay 6 up:standby 2024-03-24T00:13:08.704 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:13:08 smithi189 ceph-mon[36187]: daemon mds.f is now active in filesystem cephfs as rank 3 2024-03-24T00:13:08.704 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:13:08 smithi189 ceph-mon[36187]: pgmap v317: 97 pgs: 97 active+clean; 581 KiB data, 344 MiB used, 1.0 TiB / 1.0 TiB avail; 647 B/s rd, 4.3 KiB/s wr, 12 op/s 2024-03-24T00:13:08.710 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:13:08 smithi193 ceph-mon[36878]: from='client.? 172.21.15.118:0/491634331' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "session_timeout", "val": "300"}]': finished 2024-03-24T00:13:08.710 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:13:08 smithi193 ceph-mon[36878]: mds.? [v2:172.21.15.193:6836/2921520053,v1:172.21.15.193:6837/2921520053] up:active 2024-03-24T00:13:08.710 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:13:08 smithi193 ceph-mon[36878]: daemon mds.f assigned to filesystem cephfs as rank 3 (now has 4 ranks) 2024-03-24T00:13:08.710 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:13:08 smithi193 ceph-mon[36878]: fsmap cephfs:3 {0=b=up:active,1=g=up:active,2=c=up:active} 2 up:standby-replay 7 up:standby 2024-03-24T00:13:08.711 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:13:08 smithi193 ceph-mon[36878]: fsmap cephfs:4 {0=b=up:active,1=g=up:active,2=c=up:active,3=f=up:creating} 2 up:standby-replay 6 up:standby 2024-03-24T00:13:08.711 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:13:08 smithi193 ceph-mon[36878]: daemon mds.f is now active in filesystem cephfs as rank 3 2024-03-24T00:13:08.711 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:13:08 smithi193 ceph-mon[36878]: pgmap v317: 97 pgs: 97 active+clean; 581 KiB data, 344 MiB used, 1.0 TiB / 1.0 TiB avail; 647 B/s rd, 4.3 KiB/s wr, 12 op/s 2024-03-24T00:13:08.812 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:08.812+0000 7fd8e1600640 1 -- 172.21.15.118:0/2987811672 --> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] -- mgr_command(tid 0: {"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_0", "group_name": "qa", "target": ["mon-mgr", ""]}) v1 -- 0x7fd8dc061ae0 con 0x7fd8c00a2800 2024-03-24T00:13:08.845 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:08.845+0000 7fd8d3400640 1 -- 172.21.15.118:0/2987811672 <== mgr.14152 v2:172.21.15.118:6800/2112912943 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7fd8dc061ae0 con 0x7fd8c00a2800 2024-03-24T00:13:08.847 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:08.847+0000 7fd8e1600640 1 -- 172.21.15.118:0/2987811672 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7fd8c00a2800 msgr2=0x7fd8c00a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:13:08.847 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:08.847+0000 7fd8e1600640 1 --2- 172.21.15.118:0/2987811672 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7fd8c00a2800 0x7fd8c00a4cc0 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7fd8c802c170 tx=0x7fd8c802df90 comp rx=0 tx=0).stop 2024-03-24T00:13:08.848 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:08.848+0000 7fd8e1600640 1 -- 172.21.15.118:0/2987811672 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd8dc102180 msgr2=0x7fd8dc197a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:13:08.848 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:08.848+0000 7fd8e1600640 1 --2- 172.21.15.118:0/2987811672 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd8dc102180 0x7fd8dc197a10 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7fd8cc00ea60 tx=0x7fd8cc00ef20 comp rx=0 tx=0).stop 2024-03-24T00:13:08.848 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:08.848+0000 7fd8e1600640 1 -- 172.21.15.118:0/2987811672 shutdown_connections 2024-03-24T00:13:08.848 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:08.848+0000 7fd8e1600640 1 --2- 172.21.15.118:0/2987811672 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7fd8c00a2800 0x7fd8c00a4cc0 unknown :-1 s=CLOSED pgs=169 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:13:08.848 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:08.848+0000 7fd8e1600640 1 --2- 172.21.15.118:0/2987811672 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fd8dc1073e0 0x7fd8dc19efc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:13:08.848 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:08.848+0000 7fd8e1600640 1 --2- 172.21.15.118:0/2987811672 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fd8dc104ab0 0x7fd8dc197f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:13:08.848 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:08.848+0000 7fd8e1600640 1 --2- 172.21.15.118:0/2987811672 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd8dc102180 0x7fd8dc197a10 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:13:08.848 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:08.848+0000 7fd8e1600640 1 -- 172.21.15.118:0/2987811672 >> 172.21.15.118:0/2987811672 conn(0x7fd8dc0fbb30 msgr2=0x7fd8dc102ac0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:13:08.848 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:08.848+0000 7fd8e1600640 1 -- 172.21.15.118:0/2987811672 shutdown_connections 2024-03-24T00:13:08.848 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:08.848+0000 7fd8e1600640 1 -- 172.21.15.118:0/2987811672 wait complete. 2024-03-24T00:13:08.866 DEBUG:teuthology.orchestra.run.smithi118:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume getpath cephfs --group_name qa sv_0 2024-03-24T00:13:09.069 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:09.069+0000 7f8160600640 1 -- 172.21.15.118:0/1087247173 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8158105470 msgr2=0x7f815810bd00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:13:09.069 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:09.069+0000 7f8160600640 1 --2- 172.21.15.118:0/1087247173 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8158105470 0x7f815810bd00 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f8150009ca0 tx=0x7f815002c0f0 comp rx=0 tx=0).stop 2024-03-24T00:13:09.069 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:09.069+0000 7f8160600640 1 -- 172.21.15.118:0/1087247173 shutdown_connections 2024-03-24T00:13:09.069 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:09.069+0000 7f8160600640 1 --2- 172.21.15.118:0/1087247173 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8158105470 0x7f815810bd00 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:13:09.069 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:09.069+0000 7f8160600640 1 --2- 172.21.15.118:0/1087247173 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f8158104ab0 0x7f8158104f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:13:09.070 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:09.069+0000 7f8160600640 1 --2- 172.21.15.118:0/1087247173 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f81581038c0 0x7f8158103ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:13:09.070 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:09.069+0000 7f8160600640 1 -- 172.21.15.118:0/1087247173 >> 172.21.15.118:0/1087247173 conn(0x7f81580fee50 msgr2=0x7f8158101270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:13:09.070 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:09.069+0000 7f8160600640 1 -- 172.21.15.118:0/1087247173 shutdown_connections 2024-03-24T00:13:09.070 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:09.069+0000 7f8160600640 1 -- 172.21.15.118:0/1087247173 wait complete. 2024-03-24T00:13:09.071 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:09.070+0000 7f8160600640 1 Processor -- start 2024-03-24T00:13:09.072 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:09.070+0000 7f8160600640 1 -- start start 2024-03-24T00:13:09.072 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:09.070+0000 7f8160600640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f81581038c0 0x7f8158199bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:13:09.072 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:09.070+0000 7f8160600640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8158104ab0 0x7f815819a130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:13:09.072 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:09.070+0000 7f8160600640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f8158105470 0x7f81581a11a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:13:09.072 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:09.070+0000 7f8160600640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f815810f540 con 0x7f81581038c0 2024-03-24T00:13:09.072 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:09.070+0000 7f8160600640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f815810f3c0 con 0x7f8158104ab0 2024-03-24T00:13:09.072 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:09.070+0000 7f8160600640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f815810f6c0 con 0x7f8158105470 2024-03-24T00:13:09.072 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:09.071+0000 7f815dc00640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f81581038c0 0x7f8158199bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:13:09.072 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:09.071+0000 7f815d200640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8158104ab0 0x7f815819a130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:13:09.072 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:09.071+0000 7f815d200640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8158104ab0 0x7f815819a130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.118:34070/0 (socket says 172.21.15.118:34070) 2024-03-24T00:13:09.072 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:09.071+0000 7f815dc00640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f81581038c0 0x7f8158199bf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.118:3300/0 says I am v2:172.21.15.118:35426/0 (socket says 172.21.15.118:35426) 2024-03-24T00:13:09.072 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:09.071+0000 7f815d200640 1 -- 172.21.15.118:0/4137496827 learned_addr learned my addr 172.21.15.118:0/4137496827 (peer_addr_for_me v2:172.21.15.118:0/0) 2024-03-24T00:13:09.072 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:09.071+0000 7f815e600640 1 --2- 172.21.15.118:0/4137496827 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f8158105470 0x7f81581a11a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:13:09.073 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:09.071+0000 7f815d200640 1 -- 172.21.15.118:0/4137496827 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f8158105470 msgr2=0x7f81581a11a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:13:09.073 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:09.071+0000 7f815d200640 1 --2- 172.21.15.118:0/4137496827 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f8158105470 0x7f81581a11a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:13:09.073 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:09.071+0000 7f815d200640 1 -- 172.21.15.118:0/4137496827 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f81581038c0 msgr2=0x7f8158199bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:13:09.073 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:09.071+0000 7f815d200640 1 --2- 172.21.15.118:0/4137496827 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f81581038c0 0x7f8158199bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:13:09.073 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:09.071+0000 7f815d200640 1 -- 172.21.15.118:0/4137496827 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f81581a1810 con 0x7f8158104ab0 2024-03-24T00:13:09.073 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:09.071+0000 7f815d200640 1 --2- 172.21.15.118:0/4137496827 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8158104ab0 0x7f815819a130 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f814400e9f0 tx=0x7f814400edf0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:13:09.073 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:09.072+0000 7f8156000640 1 -- 172.21.15.118:0/4137496827 <== mon.2 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f814400cd80 con 0x7f8158104ab0 2024-03-24T00:13:09.073 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:09.072+0000 7f8156000640 1 -- 172.21.15.118:0/4137496827 <== mon.2 v2:172.21.15.189:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f8144005500 con 0x7f8158104ab0 2024-03-24T00:13:09.073 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:09.072+0000 7f8160600640 1 -- 172.21.15.118:0/4137496827 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f81581a1b00 con 0x7f8158104ab0 2024-03-24T00:13:09.073 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:09.072+0000 7f8156000640 1 -- 172.21.15.118:0/4137496827 <== mon.2 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8144010690 con 0x7f8158104ab0 2024-03-24T00:13:09.073 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:09.072+0000 7f8160600640 1 -- 172.21.15.118:0/4137496827 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f81581a2050 con 0x7f8158104ab0 2024-03-24T00:13:09.073 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:09.073+0000 7f8160600640 1 -- 172.21.15.118:0/4137496827 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f815810ab80 con 0x7f8158104ab0 2024-03-24T00:13:09.076 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:09.076+0000 7f8156000640 1 -- 172.21.15.118:0/4137496827 <== mon.2 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 16) v1 ==== 137291+0+0 (secure 0 0 0) 0x7f81440046c0 con 0x7f8158104ab0 2024-03-24T00:13:09.077 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:09.078+0000 7f8156000640 1 --2- 172.21.15.118:0/4137496827 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7f81400a2880 0x7f81400a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:13:09.077 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:09.078+0000 7f8156000640 1 -- 172.21.15.118:0/4137496827 <== mon.2 v2:172.21.15.189:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 7979+0+0 (secure 0 0 0) 0x7f81440ced10 con 0x7f8158104ab0 2024-03-24T00:13:09.077 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:09.078+0000 7f815dc00640 1 --2- 172.21.15.118:0/4137496827 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7f81400a2880 0x7f81400a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:13:09.078 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:09.078+0000 7f815dc00640 1 --2- 172.21.15.118:0/4137496827 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7f81400a2880 0x7f81400a4d40 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7f8158104910 tx=0x7f814c00b390 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:13:09.081 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:09.082+0000 7f8156000640 1 -- 172.21.15.118:0/4137496827 <== mon.2 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f81440d4020 con 0x7f8158104ab0 2024-03-24T00:13:09.307 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:09.308+0000 7f8160600640 1 -- 172.21.15.118:0/4137496827 --> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] -- mgr_command(tid 0: {"prefix": "fs subvolume getpath", "vol_name": "cephfs", "group_name": "qa", "sub_name": "sv_0", "target": ["mon-mgr", ""]}) v1 -- 0x7f8158061ae0 con 0x7f81400a2880 2024-03-24T00:13:09.318 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:09.318+0000 7f8156000640 1 -- 172.21.15.118:0/4137496827 <== mgr.14152 v2:172.21.15.118:6800/2112912943 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+54 (secure 0 0 0) 0x7f8158061ae0 con 0x7f81400a2880 2024-03-24T00:13:09.318 INFO:teuthology.orchestra.run.smithi118.stdout:/volumes/qa/sv_0/cfeed6c7-1e66-4200-b512-97b6fead6ae5 2024-03-24T00:13:09.321 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:09.321+0000 7f8160600640 1 -- 172.21.15.118:0/4137496827 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7f81400a2880 msgr2=0x7f81400a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:13:09.321 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:09.321+0000 7f8160600640 1 --2- 172.21.15.118:0/4137496827 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7f81400a2880 0x7f81400a4d40 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7f8158104910 tx=0x7f814c00b390 comp rx=0 tx=0).stop 2024-03-24T00:13:09.321 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:09.321+0000 7f8160600640 1 -- 172.21.15.118:0/4137496827 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8158104ab0 msgr2=0x7f815819a130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:13:09.321 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:09.321+0000 7f8160600640 1 --2- 172.21.15.118:0/4137496827 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8158104ab0 0x7f815819a130 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f814400e9f0 tx=0x7f814400edf0 comp rx=0 tx=0).stop 2024-03-24T00:13:09.321 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:09.321+0000 7f8160600640 1 -- 172.21.15.118:0/4137496827 shutdown_connections 2024-03-24T00:13:09.321 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:09.321+0000 7f8160600640 1 --2- 172.21.15.118:0/4137496827 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7f81400a2880 0x7f81400a4d40 unknown :-1 s=CLOSED pgs=170 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:13:09.321 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:09.321+0000 7f8160600640 1 --2- 172.21.15.118:0/4137496827 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f8158105470 0x7f81581a11a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:13:09.321 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:09.321+0000 7f8160600640 1 --2- 172.21.15.118:0/4137496827 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8158104ab0 0x7f815819a130 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:13:09.321 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:09.321+0000 7f8160600640 1 --2- 172.21.15.118:0/4137496827 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f81581038c0 0x7f8158199bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:13:09.321 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:09.321+0000 7f8160600640 1 -- 172.21.15.118:0/4137496827 >> 172.21.15.118:0/4137496827 conn(0x7f81580fee50 msgr2=0x7f8158107d50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:13:09.321 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:09.321+0000 7f8160600640 1 -- 172.21.15.118:0/4137496827 shutdown_connections 2024-03-24T00:13:09.321 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:09.321+0000 7f8160600640 1 -- 172.21.15.118:0/4137496827 wait complete. 2024-03-24T00:13:09.341 DEBUG:teuthology.orchestra.run.smithi118:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume create cephfs sv_1 --group_name qa 2024-03-24T00:13:09.527 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:09.527+0000 7f1e65e00640 1 -- 172.21.15.118:0/4026678283 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1e600756e0 msgr2=0x7f1e60073b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:13:09.527 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:09.527+0000 7f1e65e00640 1 --2- 172.21.15.118:0/4026678283 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1e600756e0 0x7f1e60073b30 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f1e54009930 tx=0x7f1e5402bc20 comp rx=0 tx=0).stop 2024-03-24T00:13:09.527 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:09.527+0000 7f1e65e00640 1 -- 172.21.15.118:0/4026678283 shutdown_connections 2024-03-24T00:13:09.527 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:09.527+0000 7f1e65e00640 1 --2- 172.21.15.118:0/4026678283 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f1e6010db70 0x7f1e6010ff60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:13:09.527 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:09.527+0000 7f1e65e00640 1 --2- 172.21.15.118:0/4026678283 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f1e60074100 0x7f1e60074580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:13:09.527 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:09.527+0000 7f1e65e00640 1 --2- 172.21.15.118:0/4026678283 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1e600756e0 0x7f1e60073b30 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:13:09.527 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:09.527+0000 7f1e65e00640 1 -- 172.21.15.118:0/4026678283 >> 172.21.15.118:0/4026678283 conn(0x7f1e600fcb10 msgr2=0x7f1e600fef50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:13:09.527 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:09.527+0000 7f1e65e00640 1 -- 172.21.15.118:0/4026678283 shutdown_connections 2024-03-24T00:13:09.527 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:09.527+0000 7f1e65e00640 1 -- 172.21.15.118:0/4026678283 wait complete. 2024-03-24T00:13:09.529 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:09.528+0000 7f1e65e00640 1 Processor -- start 2024-03-24T00:13:09.529 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:09.528+0000 7f1e65e00640 1 -- start start 2024-03-24T00:13:09.529 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:09.529+0000 7f1e65e00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f1e60074100 0x7f1e6019ded0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:13:09.529 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:09.529+0000 7f1e65e00640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f1e600756e0 0x7f1e6019e410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:13:09.529 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:09.529+0000 7f1e65e00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1e6010db70 0x7f1e601a5480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:13:09.529 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:09.529+0000 7f1e65e00640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1e60113840 con 0x7f1e600756e0 2024-03-24T00:13:09.529 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:09.529+0000 7f1e65e00640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1e601136c0 con 0x7f1e6010db70 2024-03-24T00:13:09.529 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:09.529+0000 7f1e65e00640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1e601139c0 con 0x7f1e60074100 2024-03-24T00:13:09.529 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:09.529+0000 7f1e64a00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f1e60074100 0x7f1e6019ded0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:13:09.529 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:09.529+0000 7f1e5fe00640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f1e600756e0 0x7f1e6019e410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:13:09.529 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:09.529+0000 7f1e65400640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1e6010db70 0x7f1e601a5480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:13:09.529 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:09.529+0000 7f1e5fe00640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f1e600756e0 0x7f1e6019e410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.118:3300/0 says I am v2:172.21.15.118:35436/0 (socket says 172.21.15.118:35436) 2024-03-24T00:13:09.529 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:09.529+0000 7f1e64a00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f1e60074100 0x7f1e6019ded0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.118:47684/0 (socket says 172.21.15.118:47684) 2024-03-24T00:13:09.530 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:09.529+0000 7f1e5fe00640 1 -- 172.21.15.118:0/193917015 learned_addr learned my addr 172.21.15.118:0/193917015 (peer_addr_for_me v2:172.21.15.118:0/0) 2024-03-24T00:13:09.530 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:09.529+0000 7f1e5fe00640 1 -- 172.21.15.118:0/193917015 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f1e60074100 msgr2=0x7f1e6019ded0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:13:09.530 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:09.529+0000 7f1e5fe00640 1 --2- 172.21.15.118:0/193917015 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f1e60074100 0x7f1e6019ded0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:13:09.530 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:09.529+0000 7f1e5fe00640 1 -- 172.21.15.118:0/193917015 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1e6010db70 msgr2=0x7f1e601a5480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:13:09.530 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:09.529+0000 7f1e5fe00640 1 --2- 172.21.15.118:0/193917015 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1e6010db70 0x7f1e601a5480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:13:09.530 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:09.529+0000 7f1e5fe00640 1 -- 172.21.15.118:0/193917015 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1e601a5af0 con 0x7f1e600756e0 2024-03-24T00:13:09.530 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:09.529+0000 7f1e64a00640 1 --2- 172.21.15.118:0/193917015 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f1e60074100 0x7f1e6019ded0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T00:13:09.531 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:09.530+0000 7f1e5fe00640 1 --2- 172.21.15.118:0/193917015 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f1e600756e0 0x7f1e6019e410 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f1e4c00e9f0 tx=0x7f1e4c00edf0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:13:09.531 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:09.530+0000 7f1e5d600640 1 -- 172.21.15.118:0/193917015 <== mon.0 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1e4c00ccc0 con 0x7f1e600756e0 2024-03-24T00:13:09.531 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:09.530+0000 7f1e65e00640 1 -- 172.21.15.118:0/193917015 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1e601a5de0 con 0x7f1e600756e0 2024-03-24T00:13:09.531 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:09.530+0000 7f1e5d600640 1 -- 172.21.15.118:0/193917015 <== mon.0 v2:172.21.15.118:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f1e4c010910 con 0x7f1e600756e0 2024-03-24T00:13:09.531 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:09.530+0000 7f1e5d600640 1 -- 172.21.15.118:0/193917015 <== mon.0 v2:172.21.15.118:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1e4c0188a0 con 0x7f1e600756e0 2024-03-24T00:13:09.531 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:09.530+0000 7f1e65e00640 1 -- 172.21.15.118:0/193917015 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1e601a62f0 con 0x7f1e600756e0 2024-03-24T00:13:09.531 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:09.531+0000 7f1e65e00640 1 -- 172.21.15.118:0/193917015 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1e60103d40 con 0x7f1e600756e0 2024-03-24T00:13:09.535 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:09.536+0000 7f1e5d600640 1 -- 172.21.15.118:0/193917015 <== mon.0 v2:172.21.15.118:3300/0 4 ==== mgrmap(e 16) v1 ==== 137291+0+0 (secure 0 0 0) 0x7f1e4c010ac0 con 0x7f1e600756e0 2024-03-24T00:13:09.537 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:09.537+0000 7f1e5d600640 1 --2- 172.21.15.118:0/193917015 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7f1e480a2800 0x7f1e480a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:13:09.537 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:09.538+0000 7f1e5d600640 1 -- 172.21.15.118:0/193917015 <== mon.0 v2:172.21.15.118:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 7979+0+0 (secure 0 0 0) 0x7f1e4c0cedc0 con 0x7f1e600756e0 2024-03-24T00:13:09.537 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:09.538+0000 7f1e64a00640 1 --2- 172.21.15.118:0/193917015 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7f1e480a2800 0x7f1e480a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:13:09.538 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:09.539+0000 7f1e64a00640 1 --2- 172.21.15.118:0/193917015 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7f1e480a2800 0x7f1e480a4cc0 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7f1e54036040 tx=0x7f1e54000bc0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:13:09.542 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:09.543+0000 7f1e5d600640 1 -- 172.21.15.118:0/193917015 <== mon.0 v2:172.21.15.118:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f1e4c09bf10 con 0x7f1e600756e0 2024-03-24T00:13:09.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:13:09 smithi118 ceph-mon[31535]: from='client.24653 -' entity='client.admin' cmd=[{"prefix": "fs subvolumegroup create", "vol_name": "cephfs", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T00:13:09.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:13:09 smithi118 ceph-mon[31535]: mds.? [v2:172.21.15.193:6834/2055310541,v1:172.21.15.193:6835/2055310541] up:active 2024-03-24T00:13:09.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:13:09 smithi118 ceph-mon[31535]: daemon mds.e assigned to filesystem cephfs as rank 4 (now has 5 ranks) 2024-03-24T00:13:09.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:13:09 smithi118 ceph-mon[31535]: Health check cleared: MDS_UP_LESS_THAN_MAX (was: 1 filesystem is online with fewer MDS than max_mds) 2024-03-24T00:13:09.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:13:09 smithi118 ceph-mon[31535]: Cluster is now healthy 2024-03-24T00:13:09.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:13:09 smithi118 ceph-mon[31535]: fsmap cephfs:4 {0=b=up:active,1=g=up:active,2=c=up:active,3=f=up:active} 2 up:standby-replay 6 up:standby 2024-03-24T00:13:09.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:13:09 smithi118 ceph-mon[31535]: fsmap cephfs:5 2 up:standby-replay 1 up:creating 5 up:standby 4 up:active 2024-03-24T00:13:09.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:13:09 smithi118 ceph-mon[31535]: daemon mds.e is now active in filesystem cephfs as rank 4 2024-03-24T00:13:09.703 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:13:09 smithi189 ceph-mon[36187]: from='client.24653 -' entity='client.admin' cmd=[{"prefix": "fs subvolumegroup create", "vol_name": "cephfs", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T00:13:09.703 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:13:09 smithi189 ceph-mon[36187]: mds.? [v2:172.21.15.193:6834/2055310541,v1:172.21.15.193:6835/2055310541] up:active 2024-03-24T00:13:09.703 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:13:09 smithi189 ceph-mon[36187]: daemon mds.e assigned to filesystem cephfs as rank 4 (now has 5 ranks) 2024-03-24T00:13:09.703 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:13:09 smithi189 ceph-mon[36187]: Health check cleared: MDS_UP_LESS_THAN_MAX (was: 1 filesystem is online with fewer MDS than max_mds) 2024-03-24T00:13:09.704 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:13:09 smithi189 ceph-mon[36187]: Cluster is now healthy 2024-03-24T00:13:09.704 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:13:09 smithi189 ceph-mon[36187]: fsmap cephfs:4 {0=b=up:active,1=g=up:active,2=c=up:active,3=f=up:active} 2 up:standby-replay 6 up:standby 2024-03-24T00:13:09.704 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:13:09 smithi189 ceph-mon[36187]: fsmap cephfs:5 2 up:standby-replay 1 up:creating 5 up:standby 4 up:active 2024-03-24T00:13:09.704 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:13:09 smithi189 ceph-mon[36187]: daemon mds.e is now active in filesystem cephfs as rank 4 2024-03-24T00:13:09.710 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:13:09 smithi193 ceph-mon[36878]: from='client.24653 -' entity='client.admin' cmd=[{"prefix": "fs subvolumegroup create", "vol_name": "cephfs", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T00:13:09.710 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:13:09 smithi193 ceph-mon[36878]: mds.? [v2:172.21.15.193:6834/2055310541,v1:172.21.15.193:6835/2055310541] up:active 2024-03-24T00:13:09.710 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:13:09 smithi193 ceph-mon[36878]: daemon mds.e assigned to filesystem cephfs as rank 4 (now has 5 ranks) 2024-03-24T00:13:09.710 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:13:09 smithi193 ceph-mon[36878]: Health check cleared: MDS_UP_LESS_THAN_MAX (was: 1 filesystem is online with fewer MDS than max_mds) 2024-03-24T00:13:09.710 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:13:09 smithi193 ceph-mon[36878]: Cluster is now healthy 2024-03-24T00:13:09.710 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:13:09 smithi193 ceph-mon[36878]: fsmap cephfs:4 {0=b=up:active,1=g=up:active,2=c=up:active,3=f=up:active} 2 up:standby-replay 6 up:standby 2024-03-24T00:13:09.711 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:13:09 smithi193 ceph-mon[36878]: fsmap cephfs:5 2 up:standby-replay 1 up:creating 5 up:standby 4 up:active 2024-03-24T00:13:09.711 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:13:09 smithi193 ceph-mon[36878]: daemon mds.e is now active in filesystem cephfs as rank 4 2024-03-24T00:13:09.772 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:09.771+0000 7f1e65e00640 1 -- 172.21.15.118:0/193917015 --> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] -- mgr_command(tid 0: {"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_1", "group_name": "qa", "target": ["mon-mgr", ""]}) v1 -- 0x7f1e60061b70 con 0x7f1e480a2800 2024-03-24T00:13:09.803 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:09.802+0000 7f1e5d600640 1 -- 172.21.15.118:0/193917015 <== mgr.14152 v2:172.21.15.118:6800/2112912943 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7f1e60061b70 con 0x7f1e480a2800 2024-03-24T00:13:09.806 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:09.805+0000 7f1e65e00640 1 -- 172.21.15.118:0/193917015 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7f1e480a2800 msgr2=0x7f1e480a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:13:09.806 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:09.805+0000 7f1e65e00640 1 --2- 172.21.15.118:0/193917015 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7f1e480a2800 0x7f1e480a4cc0 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7f1e54036040 tx=0x7f1e54000bc0 comp rx=0 tx=0).stop 2024-03-24T00:13:09.806 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:09.805+0000 7f1e65e00640 1 -- 172.21.15.118:0/193917015 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f1e600756e0 msgr2=0x7f1e6019e410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:13:09.806 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:09.805+0000 7f1e65e00640 1 --2- 172.21.15.118:0/193917015 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f1e600756e0 0x7f1e6019e410 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f1e4c00e9f0 tx=0x7f1e4c00edf0 comp rx=0 tx=0).stop 2024-03-24T00:13:09.806 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:09.805+0000 7f1e65e00640 1 -- 172.21.15.118:0/193917015 shutdown_connections 2024-03-24T00:13:09.807 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:09.805+0000 7f1e65e00640 1 --2- 172.21.15.118:0/193917015 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7f1e480a2800 0x7f1e480a4cc0 unknown :-1 s=CLOSED pgs=171 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:13:09.807 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:09.805+0000 7f1e65e00640 1 --2- 172.21.15.118:0/193917015 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1e6010db70 0x7f1e601a5480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:13:09.807 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:09.805+0000 7f1e65e00640 1 --2- 172.21.15.118:0/193917015 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f1e600756e0 0x7f1e6019e410 unknown :-1 s=CLOSED pgs=164 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:13:09.807 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:09.805+0000 7f1e65e00640 1 --2- 172.21.15.118:0/193917015 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f1e60074100 0x7f1e6019ded0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:13:09.807 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:09.805+0000 7f1e65e00640 1 -- 172.21.15.118:0/193917015 >> 172.21.15.118:0/193917015 conn(0x7f1e600fcb10 msgr2=0x7f1e60100f10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:13:09.807 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:09.806+0000 7f1e65e00640 1 -- 172.21.15.118:0/193917015 shutdown_connections 2024-03-24T00:13:09.807 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:09.806+0000 7f1e65e00640 1 -- 172.21.15.118:0/193917015 wait complete. 2024-03-24T00:13:09.823 DEBUG:teuthology.orchestra.run.smithi118:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs subvolume getpath cephfs --group_name qa sv_1 2024-03-24T00:13:10.040 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:10.039+0000 7f67f2600640 1 -- 172.21.15.118:0/3631955155 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f67ec104ad0 msgr2=0x7f67ec104f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:13:10.041 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:10.039+0000 7f67f2600640 1 --2- 172.21.15.118:0/3631955155 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f67ec104ad0 0x7f67ec104f50 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f67d8009900 tx=0x7f67d802bc40 comp rx=0 tx=0).stop 2024-03-24T00:13:10.041 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:10.040+0000 7f67f2600640 1 -- 172.21.15.118:0/3631955155 shutdown_connections 2024-03-24T00:13:10.041 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:10.040+0000 7f67f2600640 1 --2- 172.21.15.118:0/3631955155 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f67ec105490 0x7f67ec10bd20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:13:10.041 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:10.040+0000 7f67f2600640 1 --2- 172.21.15.118:0/3631955155 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f67ec104ad0 0x7f67ec104f50 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:13:10.041 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:10.040+0000 7f67f2600640 1 --2- 172.21.15.118:0/3631955155 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f67ec1038e0 0x7f67ec103d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:13:10.041 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:10.040+0000 7f67f2600640 1 -- 172.21.15.118:0/3631955155 >> 172.21.15.118:0/3631955155 conn(0x7f67ec0fee50 msgr2=0x7f67ec101290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:13:10.041 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:10.040+0000 7f67f2600640 1 -- 172.21.15.118:0/3631955155 shutdown_connections 2024-03-24T00:13:10.041 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:10.040+0000 7f67f2600640 1 -- 172.21.15.118:0/3631955155 wait complete. 2024-03-24T00:13:10.042 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:10.041+0000 7f67f2600640 1 Processor -- start 2024-03-24T00:13:10.042 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:10.041+0000 7f67f2600640 1 -- start start 2024-03-24T00:13:10.043 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:10.042+0000 7f67f2600640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f67ec1038e0 0x7f67ec199be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:13:10.043 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:10.042+0000 7f67f2600640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f67ec104ad0 0x7f67ec19a120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:13:10.043 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:10.042+0000 7f67f2600640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f67ec105490 0x7f67ec1a1190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:13:10.043 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:10.042+0000 7f67f2600640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f67ec10f560 con 0x7f67ec1038e0 2024-03-24T00:13:10.043 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:10.042+0000 7f67f2600640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f67ec10f3e0 con 0x7f67ec104ad0 2024-03-24T00:13:10.043 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:10.042+0000 7f67f2600640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f67ec10f6e0 con 0x7f67ec105490 2024-03-24T00:13:10.043 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:10.042+0000 7f67eaa00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f67ec104ad0 0x7f67ec19a120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:13:10.043 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:10.042+0000 7f67eb400640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f67ec1038e0 0x7f67ec199be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:13:10.044 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:10.042+0000 7f67eaa00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f67ec104ad0 0x7f67ec19a120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.118:34092/0 (socket says 172.21.15.118:34092) 2024-03-24T00:13:10.044 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:10.042+0000 7f67eaa00640 1 -- 172.21.15.118:0/3918880385 learned_addr learned my addr 172.21.15.118:0/3918880385 (peer_addr_for_me v2:172.21.15.118:0/0) 2024-03-24T00:13:10.044 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:10.042+0000 7f67ebe00640 1 --2- 172.21.15.118:0/3918880385 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f67ec105490 0x7f67ec1a1190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:13:10.044 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:10.042+0000 7f67eaa00640 1 -- 172.21.15.118:0/3918880385 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f67ec105490 msgr2=0x7f67ec1a1190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:13:10.044 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:10.042+0000 7f67eaa00640 1 --2- 172.21.15.118:0/3918880385 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f67ec105490 0x7f67ec1a1190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:13:10.044 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:10.042+0000 7f67eaa00640 1 -- 172.21.15.118:0/3918880385 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f67ec1038e0 msgr2=0x7f67ec199be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:13:10.044 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:10.042+0000 7f67eaa00640 1 --2- 172.21.15.118:0/3918880385 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f67ec1038e0 0x7f67ec199be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:13:10.044 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:10.042+0000 7f67eaa00640 1 -- 172.21.15.118:0/3918880385 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f67ec1a1800 con 0x7f67ec104ad0 2024-03-24T00:13:10.044 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:10.043+0000 7f67eaa00640 1 --2- 172.21.15.118:0/3918880385 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f67ec104ad0 0x7f67ec19a120 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f67d8009230 tx=0x7f67d802c660 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:13:10.045 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:10.043+0000 7f67e3e00640 1 -- 172.21.15.118:0/3918880385 <== mon.2 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f67d8041070 con 0x7f67ec104ad0 2024-03-24T00:13:10.045 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:10.043+0000 7f67e3e00640 1 -- 172.21.15.118:0/3918880385 <== mon.2 v2:172.21.15.189:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f67d8004450 con 0x7f67ec104ad0 2024-03-24T00:13:10.045 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:10.043+0000 7f67f2600640 1 -- 172.21.15.118:0/3918880385 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f67ec1a1a90 con 0x7f67ec104ad0 2024-03-24T00:13:10.045 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:10.043+0000 7f67e3e00640 1 -- 172.21.15.118:0/3918880385 <== mon.2 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f67d8003e00 con 0x7f67ec104ad0 2024-03-24T00:13:10.045 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:10.043+0000 7f67f2600640 1 -- 172.21.15.118:0/3918880385 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f67ec1a1f80 con 0x7f67ec104ad0 2024-03-24T00:13:10.045 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:10.044+0000 7f67f2600640 1 -- 172.21.15.118:0/3918880385 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f67ec10acd0 con 0x7f67ec104ad0 2024-03-24T00:13:10.046 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:10.046+0000 7f67e3e00640 1 -- 172.21.15.118:0/3918880385 <== mon.2 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 16) v1 ==== 137291+0+0 (secure 0 0 0) 0x7f67d8044070 con 0x7f67ec104ad0 2024-03-24T00:13:10.048 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:10.047+0000 7f67e3e00640 1 --2- 172.21.15.118:0/3918880385 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7f67d00a2800 0x7f67d00a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:13:10.048 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:10.048+0000 7f67e3e00640 1 -- 172.21.15.118:0/3918880385 <== mon.2 v2:172.21.15.189:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 7979+0+0 (secure 0 0 0) 0x7f67d80f08d0 con 0x7f67ec104ad0 2024-03-24T00:13:10.048 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:10.048+0000 7f67eb400640 1 --2- 172.21.15.118:0/3918880385 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7f67d00a2800 0x7f67d00a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:13:10.049 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:10.048+0000 7f67eb400640 1 --2- 172.21.15.118:0/3918880385 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7f67d00a2800 0x7f67d00a4cc0 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7f67ec104930 tx=0x7f67dc005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:13:10.050 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:10.050+0000 7f67e3e00640 1 -- 172.21.15.118:0/3918880385 <== mon.2 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f67d80bd3d0 con 0x7f67ec104ad0 2024-03-24T00:13:10.282 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:10.281+0000 7f67f2600640 1 -- 172.21.15.118:0/3918880385 --> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] -- mgr_command(tid 0: {"prefix": "fs subvolume getpath", "vol_name": "cephfs", "group_name": "qa", "sub_name": "sv_1", "target": ["mon-mgr", ""]}) v1 -- 0x7f67ec061ae0 con 0x7f67d00a2800 2024-03-24T00:13:10.290 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:10.289+0000 7f67e3e00640 1 -- 172.21.15.118:0/3918880385 <== mgr.14152 v2:172.21.15.118:6800/2112912943 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+54 (secure 0 0 0) 0x7f67ec061ae0 con 0x7f67d00a2800 2024-03-24T00:13:10.290 INFO:teuthology.orchestra.run.smithi118.stdout:/volumes/qa/sv_1/8e57c0ba-0773-4e6b-91f6-45f3c40de69d 2024-03-24T00:13:10.293 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:10.292+0000 7f67f2600640 1 -- 172.21.15.118:0/3918880385 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7f67d00a2800 msgr2=0x7f67d00a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:13:10.293 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:10.292+0000 7f67f2600640 1 --2- 172.21.15.118:0/3918880385 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7f67d00a2800 0x7f67d00a4cc0 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7f67ec104930 tx=0x7f67dc005f90 comp rx=0 tx=0).stop 2024-03-24T00:13:10.293 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:10.292+0000 7f67f2600640 1 -- 172.21.15.118:0/3918880385 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f67ec104ad0 msgr2=0x7f67ec19a120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:13:10.293 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:10.292+0000 7f67f2600640 1 --2- 172.21.15.118:0/3918880385 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f67ec104ad0 0x7f67ec19a120 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f67d8009230 tx=0x7f67d802c660 comp rx=0 tx=0).stop 2024-03-24T00:13:10.293 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:10.292+0000 7f67f2600640 1 -- 172.21.15.118:0/3918880385 shutdown_connections 2024-03-24T00:13:10.293 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:10.292+0000 7f67f2600640 1 --2- 172.21.15.118:0/3918880385 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7f67d00a2800 0x7f67d00a4cc0 unknown :-1 s=CLOSED pgs=172 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:13:10.293 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:10.292+0000 7f67f2600640 1 --2- 172.21.15.118:0/3918880385 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f67ec105490 0x7f67ec1a1190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:13:10.293 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:10.292+0000 7f67f2600640 1 --2- 172.21.15.118:0/3918880385 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f67ec104ad0 0x7f67ec19a120 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:13:10.293 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:10.292+0000 7f67f2600640 1 --2- 172.21.15.118:0/3918880385 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f67ec1038e0 0x7f67ec199be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:13:10.293 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:10.292+0000 7f67f2600640 1 -- 172.21.15.118:0/3918880385 >> 172.21.15.118:0/3918880385 conn(0x7f67ec0fee50 msgr2=0x7f67ec107ea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:13:10.293 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:10.292+0000 7f67f2600640 1 -- 172.21.15.118:0/3918880385 shutdown_connections 2024-03-24T00:13:10.293 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:10.292+0000 7f67f2600640 1 -- 172.21.15.118:0/3918880385 wait complete. 2024-03-24T00:13:10.311 DEBUG:teuthology.orchestra.run.smithi118:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-24T00:13:10.511 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:10.510+0000 7feab2a00640 1 -- 172.21.15.118:0/849944077 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7feaac104ad0 msgr2=0x7feaac104f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:13:10.511 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:10.510+0000 7feab2a00640 1 --2- 172.21.15.118:0/849944077 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7feaac104ad0 0x7feaac104f50 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7fea98009900 tx=0x7fea9802bc40 comp rx=0 tx=0).stop 2024-03-24T00:13:10.512 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:10.510+0000 7feab2a00640 1 -- 172.21.15.118:0/849944077 shutdown_connections 2024-03-24T00:13:10.512 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:10.510+0000 7feab2a00640 1 --2- 172.21.15.118:0/849944077 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7feaac105490 0x7feaac10bd20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:13:10.512 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:10.510+0000 7feab2a00640 1 --2- 172.21.15.118:0/849944077 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7feaac104ad0 0x7feaac104f50 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:13:10.512 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:10.510+0000 7feab2a00640 1 --2- 172.21.15.118:0/849944077 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7feaac1038e0 0x7feaac103d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:13:10.512 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:10.510+0000 7feab2a00640 1 -- 172.21.15.118:0/849944077 >> 172.21.15.118:0/849944077 conn(0x7feaac0fee50 msgr2=0x7feaac101290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:13:10.512 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:10.511+0000 7feab2a00640 1 -- 172.21.15.118:0/849944077 shutdown_connections 2024-03-24T00:13:10.512 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:10.511+0000 7feab2a00640 1 -- 172.21.15.118:0/849944077 wait complete. 2024-03-24T00:13:10.514 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:10.512+0000 7feab2a00640 1 Processor -- start 2024-03-24T00:13:10.514 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:10.512+0000 7feab2a00640 1 -- start start 2024-03-24T00:13:10.514 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:10.512+0000 7feab2a00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7feaac1038e0 0x7feaac199be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:13:10.514 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:10.512+0000 7feab2a00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7feaac104ad0 0x7feaac19a120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:13:10.515 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:10.512+0000 7feab2a00640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7feaac105490 0x7feaac1a1190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:13:10.515 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:10.512+0000 7feab2a00640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feaac10f540 con 0x7feaac105490 2024-03-24T00:13:10.515 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:10.512+0000 7feab2a00640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feaac10f3c0 con 0x7feaac104ad0 2024-03-24T00:13:10.515 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:10.512+0000 7feab2a00640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feaac10f6c0 con 0x7feaac1038e0 2024-03-24T00:13:10.515 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:10.512+0000 7feaab400640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7feaac104ad0 0x7feaac19a120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:13:10.515 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:10.512+0000 7feaabe00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7feaac1038e0 0x7feaac199be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:13:10.515 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:10.512+0000 7feaab400640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7feaac104ad0 0x7feaac19a120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.118:34114/0 (socket says 172.21.15.118:34114) 2024-03-24T00:13:10.515 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:10.512+0000 7feaab400640 1 -- 172.21.15.118:0/3373418970 learned_addr learned my addr 172.21.15.118:0/3373418970 (peer_addr_for_me v2:172.21.15.118:0/0) 2024-03-24T00:13:10.515 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:10.513+0000 7feab0a00640 1 --2- 172.21.15.118:0/3373418970 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7feaac105490 0x7feaac1a1190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:13:10.515 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:10.513+0000 7feaabe00640 1 -- 172.21.15.118:0/3373418970 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7feaac104ad0 msgr2=0x7feaac19a120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:13:10.515 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:10.513+0000 7feaabe00640 1 --2- 172.21.15.118:0/3373418970 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7feaac104ad0 0x7feaac19a120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:13:10.515 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:10.513+0000 7feaabe00640 1 -- 172.21.15.118:0/3373418970 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7feaac105490 msgr2=0x7feaac1a1190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:13:10.515 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:10.513+0000 7feaabe00640 1 --2- 172.21.15.118:0/3373418970 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7feaac105490 0x7feaac1a1190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:13:10.515 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:10.513+0000 7feaabe00640 1 -- 172.21.15.118:0/3373418970 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feaac1a1800 con 0x7feaac1038e0 2024-03-24T00:13:10.515 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:10.513+0000 7feab0a00640 1 --2- 172.21.15.118:0/3373418970 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7feaac105490 0x7feaac1a1190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T00:13:10.516 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:10.513+0000 7feaab400640 1 --2- 172.21.15.118:0/3373418970 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7feaac104ad0 0x7feaac19a120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T00:13:10.516 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:10.513+0000 7feaabe00640 1 --2- 172.21.15.118:0/3373418970 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7feaac1038e0 0x7feaac199be0 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7fea9c00e9c0 tx=0x7fea9c00ee80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:13:10.516 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:10.513+0000 7feaa8c00640 1 -- 172.21.15.118:0/3373418970 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fea9c00cd80 con 0x7feaac1038e0 2024-03-24T00:13:10.516 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:10.514+0000 7feab2a00640 1 -- 172.21.15.118:0/3373418970 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7feaac1a1af0 con 0x7feaac1038e0 2024-03-24T00:13:10.516 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:10.514+0000 7feaa8c00640 1 -- 172.21.15.118:0/3373418970 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fea9c010910 con 0x7feaac1038e0 2024-03-24T00:13:10.516 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:10.514+0000 7feaa8c00640 1 -- 172.21.15.118:0/3373418970 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fea9c018980 con 0x7feaac1038e0 2024-03-24T00:13:10.516 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:10.514+0000 7feab2a00640 1 -- 172.21.15.118:0/3373418970 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7feaac1a2010 con 0x7feaac1038e0 2024-03-24T00:13:10.516 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:10.515+0000 7feab2a00640 1 -- 172.21.15.118:0/3373418970 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7feaac10ac00 con 0x7feaac1038e0 2024-03-24T00:13:10.517 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:10.517+0000 7feaa8c00640 1 -- 172.21.15.118:0/3373418970 <== mon.1 v2:172.21.15.193:3300/0 4 ==== mgrmap(e 17) v1 ==== 137342+0+0 (secure 0 0 0) 0x7fea9c01f020 con 0x7feaac1038e0 2024-03-24T00:13:10.519 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:10.518+0000 7feaa8c00640 1 --2- 172.21.15.118:0/3373418970 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7fea900a2890 0x7fea900a4d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:13:10.519 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:10.519+0000 7feaa8c00640 1 -- 172.21.15.118:0/3373418970 <== mon.1 v2:172.21.15.193:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 7979+0+0 (secure 0 0 0) 0x7fea9c0ceba0 con 0x7feaac1038e0 2024-03-24T00:13:10.519 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:10.519+0000 7feaab400640 1 --2- 172.21.15.118:0/3373418970 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7fea900a2890 0x7fea900a4d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:13:10.519 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:10.519+0000 7feaab400640 1 --2- 172.21.15.118:0/3373418970 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7fea900a2890 0x7fea900a4d50 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7fea98000c00 tx=0x7fea9802c890 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:13:10.523 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:10.522+0000 7feaa8c00640 1 -- 172.21.15.118:0/3373418970 <== mon.1 v2:172.21.15.193:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7fea9c09bb90 con 0x7feaac1038e0 2024-03-24T00:13:10.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:13:10 smithi118 ceph-mon[31535]: from='client.24665 -' entity='client.admin' cmd=[{"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_0", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T00:13:10.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:13:10 smithi118 ceph-mon[31535]: from='client.24671 -' entity='client.admin' cmd=[{"prefix": "fs subvolume getpath", "vol_name": "cephfs", "group_name": "qa", "sub_name": "sv_0", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T00:13:10.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:13:10 smithi118 ceph-mon[31535]: mds.? [v2:172.21.15.189:6834/599473205,v1:172.21.15.189:6835/599473205] up:active 2024-03-24T00:13:10.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:13:10 smithi118 ceph-mon[31535]: fsmap cephfs:5 2 up:standby-replay 5 up:standby 5 up:active 2024-03-24T00:13:10.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:13:10 smithi118 ceph-mon[31535]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-03-24T00:13:10.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:13:10 smithi118 ceph-mon[31535]: pgmap v318: 97 pgs: 97 active+clean; 601 KiB data, 344 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 8.7 KiB/s wr, 23 op/s 2024-03-24T00:13:10.703 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:13:10 smithi189 ceph-mon[36187]: from='client.24665 -' entity='client.admin' cmd=[{"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_0", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T00:13:10.703 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:13:10 smithi189 ceph-mon[36187]: from='client.24671 -' entity='client.admin' cmd=[{"prefix": "fs subvolume getpath", "vol_name": "cephfs", "group_name": "qa", "sub_name": "sv_0", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T00:13:10.703 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:13:10 smithi189 ceph-mon[36187]: mds.? [v2:172.21.15.189:6834/599473205,v1:172.21.15.189:6835/599473205] up:active 2024-03-24T00:13:10.703 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:13:10 smithi189 ceph-mon[36187]: fsmap cephfs:5 2 up:standby-replay 5 up:standby 5 up:active 2024-03-24T00:13:10.703 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:13:10 smithi189 ceph-mon[36187]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-03-24T00:13:10.703 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:13:10 smithi189 ceph-mon[36187]: pgmap v318: 97 pgs: 97 active+clean; 601 KiB data, 344 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 8.7 KiB/s wr, 23 op/s 2024-03-24T00:13:10.710 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:13:10 smithi193 ceph-mon[36878]: from='client.24665 -' entity='client.admin' cmd=[{"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_0", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T00:13:10.710 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:13:10 smithi193 ceph-mon[36878]: from='client.24671 -' entity='client.admin' cmd=[{"prefix": "fs subvolume getpath", "vol_name": "cephfs", "group_name": "qa", "sub_name": "sv_0", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T00:13:10.710 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:13:10 smithi193 ceph-mon[36878]: mds.? [v2:172.21.15.189:6834/599473205,v1:172.21.15.189:6835/599473205] up:active 2024-03-24T00:13:10.710 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:13:10 smithi193 ceph-mon[36878]: fsmap cephfs:5 2 up:standby-replay 5 up:standby 5 up:active 2024-03-24T00:13:10.710 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:13:10 smithi193 ceph-mon[36878]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-03-24T00:13:10.711 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:13:10 smithi193 ceph-mon[36878]: pgmap v318: 97 pgs: 97 active+clean; 601 KiB data, 344 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 8.7 KiB/s wr, 23 op/s 2024-03-24T00:13:10.850 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:10.850+0000 7feab2a00640 1 -- 172.21.15.118:0/3373418970 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7feaac02d070 con 0x7feaac1038e0 2024-03-24T00:13:10.851 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:10.851+0000 7feaa8c00640 1 -- 172.21.15.118:0/3373418970 <== mon.1 v2:172.21.15.193:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 27 v27) v1 ==== 94+0+11478 (secure 0 0 0) 0x7fea9c09b490 con 0x7feaac1038e0 2024-03-24T00:13:10.852 INFO:teuthology.orchestra.run.smithi118.stdout: 2024-03-24T00:13:10.852 INFO:teuthology.orchestra.run.smithi118.stdout:{"epoch":27,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14385,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.118:6835/3194741373","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6834","nonce":3194741373},{"type":"v1","addr":"172.21.15.118:6835","nonce":3194741373}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":4},{"gid":14397,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.118:6837/1374462956","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6836","nonce":1374462956},{"type":"v1","addr":"172.21.15.118:6837","nonce":1374462956}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":27,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-03-24T00:13:01.782474+0000","modified":"2024-03-24T00:13:09.335361+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24569,"mds_1":24550,"mds_2":24511,"mds_3":24499,"mds_4":24488},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14469":{"gid":14469,"name":"a","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.118:6841/3040946537","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6840","nonce":3040946537},{"type":"v1","addr":"172.21.15.118:6841","nonce":3040946537}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24439":{"gid":24439,"name":"l","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.193:6833/1023428412","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6832","nonce":1023428412},{"type":"v1","addr":"172.21.15.193:6833","nonce":1023428412}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24440":{"gid":24440,"name":"k","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.189:6833/2740024078","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6832","nonce":2740024078},{"type":"v1","addr":"172.21.15.189:6833","nonce":2740024078}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24488":{"gid":24488,"name":"e","rank":4,"incarnation":25,"state":"up:active","state_seq":10,"addr":"172.21.15.189:6835/599473205","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6834","nonce":599473205},{"type":"v1","addr":"172.21.15.189:6835","nonce":599473205}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24499":{"gid":24499,"name":"f","rank":3,"incarnation":23,"state":"up:active","state_seq":9,"addr":"172.21.15.193:6835/2055310541","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6834","nonce":2055310541},{"type":"v1","addr":"172.21.15.193:6835","nonce":2055310541}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24511":{"gid":24511,"name":"c","rank":2,"incarnation":21,"state":"up:active","state_seq":8,"addr":"172.21.15.193:6837/2921520053","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6836","nonce":2921520053},{"type":"v1","addr":"172.21.15.193:6837","nonce":2921520053}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24515":{"gid":24515,"name":"h","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.189:6837/3450187137","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6836","nonce":3450187137},{"type":"v1","addr":"172.21.15.189:6837","nonce":3450187137}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24532":{"gid":24532,"name":"i","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.193:6839/2126842603","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6838","nonce":2126842603},{"type":"v1","addr":"172.21.15.193:6839","nonce":2126842603}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24550":{"gid":24550,"name":"g","rank":1,"incarnation":19,"state":"up:active","state_seq":6,"addr":"172.21.15.118:6839/3285788503","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6838","nonce":3285788503},{"type":"v1","addr":"172.21.15.118:6839","nonce":3285788503}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24569":{"gid":24569,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.189:6839/4074484168","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6838","nonce":4074484168},{"type":"v1","addr":"172.21.15.189:6839","nonce":4074484168}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24569,"qdb_cluster":[24511,24550,24488,24499,24569]},"id":1}]} 2024-03-24T00:13:10.852 INFO:teuthology.orchestra.run.smithi118.stderr:dumped fsmap epoch 27 2024-03-24T00:13:10.854 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:10.853+0000 7feab2a00640 1 -- 172.21.15.118:0/3373418970 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7fea900a2890 msgr2=0x7fea900a4d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:13:10.854 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:10.853+0000 7feab2a00640 1 --2- 172.21.15.118:0/3373418970 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7fea900a2890 0x7fea900a4d50 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7fea98000c00 tx=0x7fea9802c890 comp rx=0 tx=0).stop 2024-03-24T00:13:10.854 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:10.853+0000 7feab2a00640 1 -- 172.21.15.118:0/3373418970 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7feaac1038e0 msgr2=0x7feaac199be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:13:10.854 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:10.853+0000 7feab2a00640 1 --2- 172.21.15.118:0/3373418970 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7feaac1038e0 0x7feaac199be0 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7fea9c00e9c0 tx=0x7fea9c00ee80 comp rx=0 tx=0).stop 2024-03-24T00:13:10.854 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:10.853+0000 7feab2a00640 1 -- 172.21.15.118:0/3373418970 shutdown_connections 2024-03-24T00:13:10.855 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:10.853+0000 7feab2a00640 1 --2- 172.21.15.118:0/3373418970 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7fea900a2890 0x7fea900a4d50 unknown :-1 s=CLOSED pgs=173 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:13:10.855 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:10.853+0000 7feab2a00640 1 --2- 172.21.15.118:0/3373418970 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7feaac105490 0x7feaac1a1190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:13:10.855 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:10.853+0000 7feab2a00640 1 --2- 172.21.15.118:0/3373418970 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7feaac104ad0 0x7feaac19a120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:13:10.855 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:10.853+0000 7feab2a00640 1 --2- 172.21.15.118:0/3373418970 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7feaac1038e0 0x7feaac199be0 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:13:10.855 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:10.853+0000 7feab2a00640 1 -- 172.21.15.118:0/3373418970 >> 172.21.15.118:0/3373418970 conn(0x7feaac0fee50 msgr2=0x7feaac107dd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:13:10.855 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:10.854+0000 7feab2a00640 1 -- 172.21.15.118:0/3373418970 shutdown_connections 2024-03-24T00:13:10.855 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:10.854+0000 7feab2a00640 1 -- 172.21.15.118:0/3373418970 wait complete. 2024-03-24T00:13:10.873 DEBUG:teuthology.orchestra.run.smithi118:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-24T00:13:11.085 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:11.084+0000 7fbc62800640 1 -- 172.21.15.118:0/394751961 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fbc5c1008f0 msgr2=0x7fbc5c100d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:13:11.085 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:11.084+0000 7fbc62800640 1 --2- 172.21.15.118:0/394751961 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fbc5c1008f0 0x7fbc5c100d50 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7fbc48009900 tx=0x7fbc4802bc40 comp rx=0 tx=0).stop 2024-03-24T00:13:11.085 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:11.084+0000 7fbc62800640 1 -- 172.21.15.118:0/394751961 shutdown_connections 2024-03-24T00:13:11.086 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:11.084+0000 7fbc62800640 1 --2- 172.21.15.118:0/394751961 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fbc5c101290 0x7fbc5c101710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:13:11.086 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:11.084+0000 7fbc62800640 1 --2- 172.21.15.118:0/394751961 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fbc5c1008f0 0x7fbc5c100d50 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:13:11.086 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:11.084+0000 7fbc62800640 1 --2- 172.21.15.118:0/394751961 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fbc5c0fff90 0x7fbc5c1003b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:13:11.086 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:11.084+0000 7fbc62800640 1 -- 172.21.15.118:0/394751961 >> 172.21.15.118:0/394751961 conn(0x7fbc5c0fbb30 msgr2=0x7fbc5c0fdf70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:13:11.086 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:11.084+0000 7fbc62800640 1 -- 172.21.15.118:0/394751961 shutdown_connections 2024-03-24T00:13:11.086 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:11.084+0000 7fbc62800640 1 -- 172.21.15.118:0/394751961 wait complete. 2024-03-24T00:13:11.088 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:11.085+0000 7fbc62800640 1 Processor -- start 2024-03-24T00:13:11.088 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:11.086+0000 7fbc62800640 1 -- start start 2024-03-24T00:13:11.088 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:11.086+0000 7fbc62800640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fbc5c0fff90 0x7fbc5c199be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:13:11.088 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:11.086+0000 7fbc62800640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fbc5c1008f0 0x7fbc5c19a120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:13:11.088 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:11.086+0000 7fbc62800640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fbc5c101290 0x7fbc5c1a1190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:13:11.088 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:11.086+0000 7fbc62800640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbc5c10f570 con 0x7fbc5c1008f0 2024-03-24T00:13:11.089 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:11.086+0000 7fbc62800640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbc5c10f3f0 con 0x7fbc5c0fff90 2024-03-24T00:13:11.089 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:11.086+0000 7fbc62800640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbc5c10f6f0 con 0x7fbc5c101290 2024-03-24T00:13:11.089 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:11.086+0000 7fbc5b400640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fbc5c0fff90 0x7fbc5c199be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:13:11.089 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:11.086+0000 7fbc5b400640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fbc5c0fff90 0x7fbc5c199be0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.118:34124/0 (socket says 172.21.15.118:34124) 2024-03-24T00:13:11.089 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:11.086+0000 7fbc5b400640 1 -- 172.21.15.118:0/214859361 learned_addr learned my addr 172.21.15.118:0/214859361 (peer_addr_for_me v2:172.21.15.118:0/0) 2024-03-24T00:13:11.089 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:11.086+0000 7fbc5be00640 1 --2- 172.21.15.118:0/214859361 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fbc5c101290 0x7fbc5c1a1190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:13:11.089 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:11.086+0000 7fbc5aa00640 1 --2- 172.21.15.118:0/214859361 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fbc5c1008f0 0x7fbc5c19a120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:13:11.089 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:11.087+0000 7fbc5b400640 1 -- 172.21.15.118:0/214859361 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fbc5c101290 msgr2=0x7fbc5c1a1190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:13:11.089 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:11.087+0000 7fbc5b400640 1 --2- 172.21.15.118:0/214859361 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fbc5c101290 0x7fbc5c1a1190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:13:11.089 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:11.087+0000 7fbc5b400640 1 -- 172.21.15.118:0/214859361 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fbc5c1008f0 msgr2=0x7fbc5c19a120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:13:11.089 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:11.087+0000 7fbc5b400640 1 --2- 172.21.15.118:0/214859361 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fbc5c1008f0 0x7fbc5c19a120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:13:11.089 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:11.087+0000 7fbc5b400640 1 -- 172.21.15.118:0/214859361 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbc5c1a1800 con 0x7fbc5c0fff90 2024-03-24T00:13:11.089 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:11.087+0000 7fbc5be00640 1 --2- 172.21.15.118:0/214859361 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fbc5c101290 0x7fbc5c1a1190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T00:13:11.089 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:11.087+0000 7fbc5b400640 1 --2- 172.21.15.118:0/214859361 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fbc5c0fff90 0x7fbc5c199be0 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7fbc4c00e9f0 tx=0x7fbc4c00edf0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:13:11.089 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:11.087+0000 7fbc53e00640 1 -- 172.21.15.118:0/214859361 <== mon.2 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbc4c00cd80 con 0x7fbc5c0fff90 2024-03-24T00:13:11.090 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:11.087+0000 7fbc53e00640 1 -- 172.21.15.118:0/214859361 <== mon.2 v2:172.21.15.189:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fbc4c005500 con 0x7fbc5c0fff90 2024-03-24T00:13:11.090 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:11.087+0000 7fbc62800640 1 -- 172.21.15.118:0/214859361 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbc5c1a1af0 con 0x7fbc5c0fff90 2024-03-24T00:13:11.090 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:11.087+0000 7fbc53e00640 1 -- 172.21.15.118:0/214859361 <== mon.2 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbc4c010690 con 0x7fbc5c0fff90 2024-03-24T00:13:11.090 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:11.087+0000 7fbc62800640 1 -- 172.21.15.118:0/214859361 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbc5c1a2010 con 0x7fbc5c0fff90 2024-03-24T00:13:11.090 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:11.088+0000 7fbc62800640 1 -- 172.21.15.118:0/214859361 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbc5c10ae10 con 0x7fbc5c0fff90 2024-03-24T00:13:11.090 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:11.090+0000 7fbc53e00640 1 -- 172.21.15.118:0/214859361 <== mon.2 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 17) v1 ==== 137342+0+0 (secure 0 0 0) 0x7fbc4c0046c0 con 0x7fbc5c0fff90 2024-03-24T00:13:11.093 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:11.092+0000 7fbc53e00640 1 --2- 172.21.15.118:0/214859361 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7fbc400a2910 0x7fbc400a4dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:13:11.093 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:11.093+0000 7fbc5aa00640 1 --2- 172.21.15.118:0/214859361 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7fbc400a2910 0x7fbc400a4dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:13:11.093 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:11.093+0000 7fbc53e00640 1 -- 172.21.15.118:0/214859361 <== mon.2 v2:172.21.15.189:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 7979+0+0 (secure 0 0 0) 0x7fbc4c0ceea0 con 0x7fbc5c0fff90 2024-03-24T00:13:11.094 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:11.093+0000 7fbc5aa00640 1 --2- 172.21.15.118:0/214859361 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7fbc400a2910 0x7fbc400a4dd0 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7fbc4802c250 tx=0x7fbc4802cf90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:13:11.096 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:11.095+0000 7fbc53e00640 1 -- 172.21.15.118:0/214859361 <== mon.2 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7fbc4c09be90 con 0x7fbc5c0fff90 2024-03-24T00:13:11.203 INFO:journalctl@ceph.mds.b.smithi189.stdout:Mar 24 00:13:10 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-b[66975]: 2024-03-24T00:13:10.710+0000 7f4e07ab9700 -1 mds.pinger is_rank_lagging: rank=4 was never sent ping request. 2024-03-24T00:13:11.295 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:11.295+0000 7fbc62800640 1 -- 172.21.15.118:0/214859361 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fbc5c02d2b0 con 0x7fbc5c0fff90 2024-03-24T00:13:11.297 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:11.297+0000 7fbc53e00640 1 -- 172.21.15.118:0/214859361 <== mon.2 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v86) v1 ==== 74+0+22287 (secure 0 0 0) 0x7fbc4c09b790 con 0x7fbc5c0fff90 2024-03-24T00:13:11.298 INFO:teuthology.orchestra.run.smithi118.stdout: 2024-03-24T00:13:11.299 INFO:teuthology.orchestra.run.smithi118.stdout:{"epoch":86,"fsid":"d79a5420-e971-11ee-95cd-87774f69a715","created":"2024-03-24T00:03:58.648524+0000","modified":"2024-03-24T00:13:03.282705+0000","last_up_change":"2024-03-24T00:12:03.809000+0000","last_in_change":"2024-03-24T00:11:44.389344+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-24T00:07:50.190965+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-24T00:12:59.993723+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-24T00:13:01.257347+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"86","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"c9590426-3901-4f68-a5ef-d57aa8fbac92","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6802","nonce":557690061},{"type":"v1","addr":"172.21.15.118:6803","nonce":557690061}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6804","nonce":557690061},{"type":"v1","addr":"172.21.15.118:6805","nonce":557690061}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6808","nonce":557690061},{"type":"v1","addr":"172.21.15.118:6809","nonce":557690061}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6806","nonce":557690061},{"type":"v1","addr":"172.21.15.118:6807","nonce":557690061}]},"public_addr":"172.21.15.118:6803/557690061","cluster_addr":"172.21.15.118:6805/557690061","heartbeat_back_addr":"172.21.15.118:6809/557690061","heartbeat_front_addr":"172.21.15.118:6807/557690061","state":["exists","up"]},{"osd":1,"uuid":"fc7f827e-ecf6-4661-aad0-7a22951892e0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6800","nonce":4203029485},{"type":"v1","addr":"172.21.15.189:6801","nonce":4203029485}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6802","nonce":4203029485},{"type":"v1","addr":"172.21.15.189:6803","nonce":4203029485}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6806","nonce":4203029485},{"type":"v1","addr":"172.21.15.189:6807","nonce":4203029485}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6804","nonce":4203029485},{"type":"v1","addr":"172.21.15.189:6805","nonce":4203029485}]},"public_addr":"172.21.15.189:6801/4203029485","cluster_addr":"172.21.15.189:6803/4203029485","heartbeat_back_addr":"172.21.15.189:6807/4203029485","heartbeat_front_addr":"172.21.15.189:6805/4203029485","state":["exists","up"]},{"osd":2,"uuid":"e73ccfe9-0e5b-4734-a154-a2f0d829fcfb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6800","nonce":3699076971},{"type":"v1","addr":"172.21.15.193:6801","nonce":3699076971}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6802","nonce":3699076971},{"type":"v1","addr":"172.21.15.193:6803","nonce":3699076971}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6806","nonce":3699076971},{"type":"v1","addr":"172.21.15.193:6807","nonce":3699076971}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6804","nonce":3699076971},{"type":"v1","addr":"172.21.15.193:6805","nonce":3699076971}]},"public_addr":"172.21.15.193:6801/3699076971","cluster_addr":"172.21.15.193:6803/3699076971","heartbeat_back_addr":"172.21.15.193:6807/3699076971","heartbeat_front_addr":"172.21.15.193:6805/3699076971","state":["exists","up"]},{"osd":3,"uuid":"63fed43b-43ce-4031-8957-7fe94725c1ed","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6810","nonce":4059617010},{"type":"v1","addr":"172.21.15.118:6811","nonce":4059617010}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6812","nonce":4059617010},{"type":"v1","addr":"172.21.15.118:6813","nonce":4059617010}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6816","nonce":4059617010},{"type":"v1","addr":"172.21.15.118:6817","nonce":4059617010}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6814","nonce":4059617010},{"type":"v1","addr":"172.21.15.118:6815","nonce":4059617010}]},"public_addr":"172.21.15.118:6811/4059617010","cluster_addr":"172.21.15.118:6813/4059617010","heartbeat_back_addr":"172.21.15.118:6817/4059617010","heartbeat_front_addr":"172.21.15.118:6815/4059617010","state":["exists","up"]},{"osd":4,"uuid":"57377f66-c3b4-43c1-8e8e-bc873672906d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":34,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6808","nonce":4075247068},{"type":"v1","addr":"172.21.15.189:6809","nonce":4075247068}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6810","nonce":4075247068},{"type":"v1","addr":"172.21.15.189:6811","nonce":4075247068}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6814","nonce":4075247068},{"type":"v1","addr":"172.21.15.189:6815","nonce":4075247068}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6812","nonce":4075247068},{"type":"v1","addr":"172.21.15.189:6813","nonce":4075247068}]},"public_addr":"172.21.15.189:6809/4075247068","cluster_addr":"172.21.15.189:6811/4075247068","heartbeat_back_addr":"172.21.15.189:6815/4075247068","heartbeat_front_addr":"172.21.15.189:6813/4075247068","state":["exists","up"]},{"osd":5,"uuid":"91970976-2193-4224-81c0-dc96cb022c85","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":39,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6808","nonce":2920363402},{"type":"v1","addr":"172.21.15.193:6809","nonce":2920363402}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6810","nonce":2920363402},{"type":"v1","addr":"172.21.15.193:6811","nonce":2920363402}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6814","nonce":2920363402},{"type":"v1","addr":"172.21.15.193:6815","nonce":2920363402}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6812","nonce":2920363402},{"type":"v1","addr":"172.21.15.193:6813","nonce":2920363402}]},"public_addr":"172.21.15.193:6809/2920363402","cluster_addr":"172.21.15.193:6811/2920363402","heartbeat_back_addr":"172.21.15.193:6815/2920363402","heartbeat_front_addr":"172.21.15.193:6813/2920363402","state":["exists","up"]},{"osd":6,"uuid":"0b77b196-8674-41c6-ad3d-726b5d01c8fd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":47,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6818","nonce":431354576},{"type":"v1","addr":"172.21.15.118:6819","nonce":431354576}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6820","nonce":431354576},{"type":"v1","addr":"172.21.15.118:6821","nonce":431354576}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6824","nonce":431354576},{"type":"v1","addr":"172.21.15.118:6825","nonce":431354576}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6822","nonce":431354576},{"type":"v1","addr":"172.21.15.118:6823","nonce":431354576}]},"public_addr":"172.21.15.118:6819/431354576","cluster_addr":"172.21.15.118:6821/431354576","heartbeat_back_addr":"172.21.15.118:6825/431354576","heartbeat_front_addr":"172.21.15.118:6823/431354576","state":["exists","up"]},{"osd":7,"uuid":"1fbbd38f-4e8e-4d4c-8e98-66f6862bc4bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":52,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6816","nonce":3667611284},{"type":"v1","addr":"172.21.15.189:6817","nonce":3667611284}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6818","nonce":3667611284},{"type":"v1","addr":"172.21.15.189:6819","nonce":3667611284}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6822","nonce":3667611284},{"type":"v1","addr":"172.21.15.189:6823","nonce":3667611284}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6820","nonce":3667611284},{"type":"v1","addr":"172.21.15.189:6821","nonce":3667611284}]},"public_addr":"172.21.15.189:6817/3667611284","cluster_addr":"172.21.15.189:6819/3667611284","heartbeat_back_addr":"172.21.15.189:6823/3667611284","heartbeat_front_addr":"172.21.15.189:6821/3667611284","state":["exists","up"]},{"osd":8,"uuid":"6462c2b5-e4b3-4b0b-bcfa-4de2c89e93e0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":60,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6816","nonce":3853800910},{"type":"v1","addr":"172.21.15.193:6817","nonce":3853800910}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6818","nonce":3853800910},{"type":"v1","addr":"172.21.15.193:6819","nonce":3853800910}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6822","nonce":3853800910},{"type":"v1","addr":"172.21.15.193:6823","nonce":3853800910}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6820","nonce":3853800910},{"type":"v1","addr":"172.21.15.193:6821","nonce":3853800910}]},"public_addr":"172.21.15.193:6817/3853800910","cluster_addr":"172.21.15.193:6819/3853800910","heartbeat_back_addr":"172.21.15.193:6823/3853800910","heartbeat_front_addr":"172.21.15.193:6821/3853800910","state":["exists","up"]},{"osd":9,"uuid":"54be277f-3596-422f-9de5-9005a8571c6f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":68,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6826","nonce":3248293936},{"type":"v1","addr":"172.21.15.118:6827","nonce":3248293936}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6828","nonce":3248293936},{"type":"v1","addr":"172.21.15.118:6829","nonce":3248293936}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6832","nonce":3248293936},{"type":"v1","addr":"172.21.15.118:6833","nonce":3248293936}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6830","nonce":3248293936},{"type":"v1","addr":"172.21.15.118:6831","nonce":3248293936}]},"public_addr":"172.21.15.118:6827/3248293936","cluster_addr":"172.21.15.118:6829/3248293936","heartbeat_back_addr":"172.21.15.118:6833/3248293936","heartbeat_front_addr":"172.21.15.118:6831/3248293936","state":["exists","up"]},{"osd":10,"uuid":"7ba69d69-ac41-44ce-bc39-c049a8ca5900","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":73,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6824","nonce":2112195760},{"type":"v1","addr":"172.21.15.189:6825","nonce":2112195760}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6826","nonce":2112195760},{"type":"v1","addr":"172.21.15.189:6827","nonce":2112195760}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6830","nonce":2112195760},{"type":"v1","addr":"172.21.15.189:6831","nonce":2112195760}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6828","nonce":2112195760},{"type":"v1","addr":"172.21.15.189:6829","nonce":2112195760}]},"public_addr":"172.21.15.189:6825/2112195760","cluster_addr":"172.21.15.189:6827/2112195760","heartbeat_back_addr":"172.21.15.189:6831/2112195760","heartbeat_front_addr":"172.21.15.189:6829/2112195760","state":["exists","up"]},{"osd":11,"uuid":"186f0b18-6b22-4eae-83b7-b21bbcce0817","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":81,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6824","nonce":3880515510},{"type":"v1","addr":"172.21.15.193:6825","nonce":3880515510}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6826","nonce":3880515510},{"type":"v1","addr":"172.21.15.193:6827","nonce":3880515510}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6830","nonce":3880515510},{"type":"v1","addr":"172.21.15.193:6831","nonce":3880515510}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6828","nonce":3880515510},{"type":"v1","addr":"172.21.15.193:6829","nonce":3880515510}]},"public_addr":"172.21.15.193:6825/3880515510","cluster_addr":"172.21.15.193:6827/3880515510","heartbeat_back_addr":"172.21.15.193:6831/3880515510","heartbeat_front_addr":"172.21.15.193:6829/3880515510","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:06:58.105543+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:07:23.795866+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:07:47.426050+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:08:19.427073+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:08:45.816082+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:09:11.081523+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:09:44.485077+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:10:11.338972+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:10:36.234024+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:11:09.730432+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:11:37.059083+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:12:02.371947+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.118:6801/389249272":"2024-03-25T00:05:05.458796+0000","172.21.15.118:6800/389249272":"2024-03-25T00:05:05.458796+0000","172.21.15.118:0/2101854694":"2024-03-25T00:05:05.458796+0000","172.21.15.118:0/630779359":"2024-03-25T00:05:05.458796+0000","172.21.15.118:0/2158921029":"2024-03-25T00:04:31.283725+0000","172.21.15.118:0/1003670455":"2024-03-25T00:05:05.458796+0000","172.21.15.118:0/368712314":"2024-03-25T00:04:31.283725+0000","172.21.15.118:0/2004775597":"2024-03-25T00:04:31.283725+0000","172.21.15.118:6800/1463147955":"2024-03-25T00:04:31.283725+0000","172.21.15.118:6801/1463147955":"2024-03-25T00:04:31.283725+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-24T00:13:11.301 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:11.300+0000 7fbc62800640 1 -- 172.21.15.118:0/214859361 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7fbc400a2910 msgr2=0x7fbc400a4dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:13:11.301 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:11.300+0000 7fbc62800640 1 --2- 172.21.15.118:0/214859361 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7fbc400a2910 0x7fbc400a4dd0 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7fbc4802c250 tx=0x7fbc4802cf90 comp rx=0 tx=0).stop 2024-03-24T00:13:11.301 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:11.300+0000 7fbc62800640 1 -- 172.21.15.118:0/214859361 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fbc5c0fff90 msgr2=0x7fbc5c199be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:13:11.301 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:11.300+0000 7fbc62800640 1 --2- 172.21.15.118:0/214859361 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fbc5c0fff90 0x7fbc5c199be0 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7fbc4c00e9f0 tx=0x7fbc4c00edf0 comp rx=0 tx=0).stop 2024-03-24T00:13:11.301 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:11.300+0000 7fbc62800640 1 -- 172.21.15.118:0/214859361 shutdown_connections 2024-03-24T00:13:11.301 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:11.300+0000 7fbc62800640 1 --2- 172.21.15.118:0/214859361 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7fbc400a2910 0x7fbc400a4dd0 unknown :-1 s=CLOSED pgs=174 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:13:11.301 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:11.300+0000 7fbc62800640 1 --2- 172.21.15.118:0/214859361 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fbc5c101290 0x7fbc5c1a1190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:13:11.301 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:11.300+0000 7fbc62800640 1 --2- 172.21.15.118:0/214859361 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fbc5c1008f0 0x7fbc5c19a120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:13:11.301 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:11.300+0000 7fbc62800640 1 --2- 172.21.15.118:0/214859361 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fbc5c0fff90 0x7fbc5c199be0 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:13:11.301 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:11.300+0000 7fbc62800640 1 -- 172.21.15.118:0/214859361 >> 172.21.15.118:0/214859361 conn(0x7fbc5c0fbb30 msgr2=0x7fbc5c0fd200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:13:11.301 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:11.300+0000 7fbc62800640 1 -- 172.21.15.118:0/214859361 shutdown_connections 2024-03-24T00:13:11.301 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:11.300+0000 7fbc62800640 1 -- 172.21.15.118:0/214859361 wait complete. 2024-03-24T00:13:11.328 INFO:tasks.cephfs.filesystem.ceph_manager:waiting for clean 2024-03-24T00:13:11.328 DEBUG:teuthology.orchestra.run.smithi118:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-24T00:13:11.528 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:11.527+0000 7f505f600640 1 -- 172.21.15.118:0/2979359905 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f50581073a0 msgr2=0x7f50581097f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:13:11.528 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:11.527+0000 7f505f600640 1 --2- 172.21.15.118:0/2979359905 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f50581073a0 0x7f50581097f0 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f5050009ca0 tx=0x7f505002c0f0 comp rx=0 tx=0).stop 2024-03-24T00:13:11.528 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:11.527+0000 7f505f600640 1 -- 172.21.15.118:0/2979359905 shutdown_connections 2024-03-24T00:13:11.528 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:11.527+0000 7f505f600640 1 --2- 172.21.15.118:0/2979359905 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f50581073a0 0x7f50581097f0 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:13:11.528 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:11.527+0000 7f505f600640 1 --2- 172.21.15.118:0/2979359905 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f5058104a70 0x7f5058106e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:13:11.528 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:11.527+0000 7f505f600640 1 --2- 172.21.15.118:0/2979359905 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f5058102140 0x7f5058104530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:13:11.528 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:11.527+0000 7f505f600640 1 -- 172.21.15.118:0/2979359905 >> 172.21.15.118:0/2979359905 conn(0x7f50580fbb30 msgr2=0x7f50580fdf70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:13:11.528 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:11.527+0000 7f505f600640 1 -- 172.21.15.118:0/2979359905 shutdown_connections 2024-03-24T00:13:11.528 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:11.527+0000 7f505f600640 1 -- 172.21.15.118:0/2979359905 wait complete. 2024-03-24T00:13:11.529 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:11.528+0000 7f505f600640 1 Processor -- start 2024-03-24T00:13:11.530 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:11.528+0000 7f505f600640 1 -- start start 2024-03-24T00:13:11.530 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:11.529+0000 7f505f600640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5058102140 0x7f5058101ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:13:11.530 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:11.529+0000 7f505f600640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f5058104a70 0x7f50581000f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:13:11.530 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:11.529+0000 7f505f600640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f50581073a0 0x7f5058100630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:13:11.530 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:11.529+0000 7f505f600640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f505810d360 con 0x7f5058104a70 2024-03-24T00:13:11.530 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:11.529+0000 7f505f600640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f505810d1e0 con 0x7f5058102140 2024-03-24T00:13:11.530 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:11.529+0000 7f505f600640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f505810d4e0 con 0x7f50581073a0 2024-03-24T00:13:11.530 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:11.529+0000 7f505d600640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f50581073a0 0x7f5058100630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:13:11.530 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:11.529+0000 7f505cc00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5058102140 0x7f5058101ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:13:11.530 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:11.529+0000 7f504fe00640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f5058104a70 0x7f50581000f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:13:11.530 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:11.529+0000 7f505d600640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f50581073a0 0x7f5058100630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.118:47742/0 (socket says 172.21.15.118:47742) 2024-03-24T00:13:11.530 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:11.529+0000 7f505d600640 1 -- 172.21.15.118:0/1328022696 learned_addr learned my addr 172.21.15.118:0/1328022696 (peer_addr_for_me v2:172.21.15.118:0/0) 2024-03-24T00:13:11.531 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:11.529+0000 7f505d600640 1 -- 172.21.15.118:0/1328022696 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5058102140 msgr2=0x7f5058101ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:13:11.531 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:11.529+0000 7f505d600640 1 --2- 172.21.15.118:0/1328022696 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5058102140 0x7f5058101ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:13:11.532 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:11.529+0000 7f505d600640 1 -- 172.21.15.118:0/1328022696 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f5058104a70 msgr2=0x7f50581000f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:13:11.532 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:11.529+0000 7f505d600640 1 --2- 172.21.15.118:0/1328022696 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f5058104a70 0x7f50581000f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:13:11.532 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:11.529+0000 7f505d600640 1 -- 172.21.15.118:0/1328022696 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5058100ec0 con 0x7f50581073a0 2024-03-24T00:13:11.532 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:11.530+0000 7f505d600640 1 --2- 172.21.15.118:0/1328022696 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f50581073a0 0x7f5058100630 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f505002c630 tx=0x7f5050032ea0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:13:11.532 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:11.530+0000 7f5056000640 1 -- 172.21.15.118:0/1328022696 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5050041020 con 0x7f50581073a0 2024-03-24T00:13:11.532 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:11.530+0000 7f5056000640 1 -- 172.21.15.118:0/1328022696 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f5050034cd0 con 0x7f50581073a0 2024-03-24T00:13:11.532 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:11.530+0000 7f505f600640 1 -- 172.21.15.118:0/1328022696 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f50581a1a30 con 0x7f50581073a0 2024-03-24T00:13:11.532 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:11.530+0000 7f5056000640 1 -- 172.21.15.118:0/1328022696 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5050039040 con 0x7f50581073a0 2024-03-24T00:13:11.532 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:11.530+0000 7f505f600640 1 -- 172.21.15.118:0/1328022696 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f50581a1fd0 con 0x7f50581073a0 2024-03-24T00:13:11.532 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:11.531+0000 7f505f600640 1 -- 172.21.15.118:0/1328022696 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f50580fd620 con 0x7f50581073a0 2024-03-24T00:13:11.533 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:11.533+0000 7f5056000640 1 -- 172.21.15.118:0/1328022696 <== mon.1 v2:172.21.15.193:3300/0 4 ==== mgrmap(e 17) v1 ==== 137342+0+0 (secure 0 0 0) 0x7f5050044020 con 0x7f50581073a0 2024-03-24T00:13:11.534 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:11.534+0000 7f5056000640 1 --2- 172.21.15.118:0/1328022696 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7f503c0a2910 0x7f503c0a4dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:13:11.535 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:11.534+0000 7f5056000640 1 -- 172.21.15.118:0/1328022696 <== mon.1 v2:172.21.15.193:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 7979+0+0 (secure 0 0 0) 0x7f50500f07c0 con 0x7f50581073a0 2024-03-24T00:13:11.535 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:11.534+0000 7f505cc00640 1 --2- 172.21.15.118:0/1328022696 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7f503c0a2910 0x7f503c0a4dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:13:11.535 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:11.535+0000 7f505cc00640 1 --2- 172.21.15.118:0/1328022696 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7f503c0a2910 0x7f503c0a4dd0 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7f5048009bd0 tx=0x7f5048005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:13:11.538 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:11.538+0000 7f5056000640 1 -- 172.21.15.118:0/1328022696 <== mon.1 v2:172.21.15.193:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f505004b020 con 0x7f50581073a0 2024-03-24T00:13:11.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:13:11 smithi118 ceph-mon[31535]: from='client.14583 -' entity='client.admin' cmd=[{"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_1", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T00:13:11.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:13:11 smithi118 ceph-mon[31535]: from='client.24683 -' entity='client.admin' cmd=[{"prefix": "fs subvolume getpath", "vol_name": "cephfs", "group_name": "qa", "sub_name": "sv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T00:13:11.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:13:11 smithi118 ceph-mon[31535]: mgrmap e17: x(active, since 8m), standbys: z, y 2024-03-24T00:13:11.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:13:11 smithi118 ceph-mon[31535]: from='client.? 172.21.15.118:0/3373418970' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-24T00:13:11.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:13:11 smithi118 ceph-mon[31535]: from='client.? 172.21.15.118:0/214859361' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-24T00:13:11.703 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:13:11 smithi189 ceph-mon[36187]: from='client.14583 -' entity='client.admin' cmd=[{"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_1", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T00:13:11.703 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:13:11 smithi189 ceph-mon[36187]: from='client.24683 -' entity='client.admin' cmd=[{"prefix": "fs subvolume getpath", "vol_name": "cephfs", "group_name": "qa", "sub_name": "sv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T00:13:11.703 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:13:11 smithi189 ceph-mon[36187]: mgrmap e17: x(active, since 8m), standbys: z, y 2024-03-24T00:13:11.703 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:13:11 smithi189 ceph-mon[36187]: from='client.? 172.21.15.118:0/3373418970' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-24T00:13:11.703 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:13:11 smithi189 ceph-mon[36187]: from='client.? 172.21.15.118:0/214859361' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-24T00:13:11.710 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:13:11 smithi193 ceph-mon[36878]: from='client.14583 -' entity='client.admin' cmd=[{"prefix": "fs subvolume create", "vol_name": "cephfs", "sub_name": "sv_1", "group_name": "qa", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T00:13:11.710 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:13:11 smithi193 ceph-mon[36878]: from='client.24683 -' entity='client.admin' cmd=[{"prefix": "fs subvolume getpath", "vol_name": "cephfs", "group_name": "qa", "sub_name": "sv_1", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T00:13:11.710 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:13:11 smithi193 ceph-mon[36878]: mgrmap e17: x(active, since 8m), standbys: z, y 2024-03-24T00:13:11.710 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:13:11 smithi193 ceph-mon[36878]: from='client.? 172.21.15.118:0/3373418970' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-24T00:13:11.710 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:13:11 smithi193 ceph-mon[36878]: from='client.? 172.21.15.118:0/214859361' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-24T00:13:11.742 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:11.742+0000 7f505f600640 1 -- 172.21.15.118:0/1328022696 --> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f5058061ba0 con 0x7f503c0a2910 2024-03-24T00:13:11.771 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:11.771+0000 7f5056000640 1 -- 172.21.15.118:0/1328022696 <== mgr.14152 v2:172.21.15.118:6800/2112912943 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+258968 (secure 0 0 0) 0x7f5058061ba0 con 0x7f503c0a2910 2024-03-24T00:13:11.775 INFO:teuthology.orchestra.run.smithi118.stdout: 2024-03-24T00:13:11.777 INFO:teuthology.orchestra.run.smithi118.stderr:dumped all 2024-03-24T00:13:11.779 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:11.777+0000 7f505f600640 1 -- 172.21.15.118:0/1328022696 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7f503c0a2910 msgr2=0x7f503c0a4dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:13:11.779 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:11.777+0000 7f505f600640 1 --2- 172.21.15.118:0/1328022696 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7f503c0a2910 0x7f503c0a4dd0 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7f5048009bd0 tx=0x7f5048005f90 comp rx=0 tx=0).stop 2024-03-24T00:13:11.779 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:11.777+0000 7f505f600640 1 -- 172.21.15.118:0/1328022696 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f50581073a0 msgr2=0x7f5058100630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:13:11.779 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:11.777+0000 7f505f600640 1 --2- 172.21.15.118:0/1328022696 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f50581073a0 0x7f5058100630 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f505002c630 tx=0x7f5050032ea0 comp rx=0 tx=0).stop 2024-03-24T00:13:11.780 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:11.777+0000 7f505f600640 1 -- 172.21.15.118:0/1328022696 shutdown_connections 2024-03-24T00:13:11.780 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:11.777+0000 7f505f600640 1 --2- 172.21.15.118:0/1328022696 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7f503c0a2910 0x7f503c0a4dd0 unknown :-1 s=CLOSED pgs=175 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:13:11.780 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:11.777+0000 7f505f600640 1 --2- 172.21.15.118:0/1328022696 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f50581073a0 0x7f5058100630 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:13:11.780 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:11.777+0000 7f505f600640 1 --2- 172.21.15.118:0/1328022696 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f5058104a70 0x7f50581000f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:13:11.780 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:11.777+0000 7f505f600640 1 --2- 172.21.15.118:0/1328022696 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5058102140 0x7f5058101ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:13:11.780 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:11.777+0000 7f505f600640 1 -- 172.21.15.118:0/1328022696 >> 172.21.15.118:0/1328022696 conn(0x7f50580fbb30 msgr2=0x7f5058102a80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:13:11.780 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:11.777+0000 7f505f600640 1 -- 172.21.15.118:0/1328022696 shutdown_connections 2024-03-24T00:13:11.780 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:11.777+0000 7f505f600640 1 -- 172.21.15.118:0/1328022696 wait complete. 2024-03-24T00:13:11.798 INFO:teuthology.orchestra.run.smithi118.stdout:{"pg_ready":true,"pg_map":{"version":319,"stamp":"2024-03-24T00:13:11.604553+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":616821,"num_objects":85,"num_object_clones":0,"num_object_copies":255,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":85,"num_whiteouts":0,"num_read":155,"num_read_kb":132,"num_write":364,"num_write_kb":3012,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":55,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":295,"ondisk_log_size":295,"up":291,"acting":291,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":291,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":352088,"kb_used_data":11252,"kb_used_omap":18,"kb_used_meta":340717,"kb_avail":1124589736,"statfs":{"total":1151940427776,"available":1151579889664,"internally_reserved":0,"allocated":11522048,"data_stored":7116213,"data_compressed":108567,"data_compressed_allocated":135168,"data_compressed_original":1769472,"omap_allocated":19142,"internal_metadata":348894522},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":11,"apply_latency_ms":11,"commit_latency_ns":11000000,"apply_latency_ns":11000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":26453,"num_objects":83,"num_object_clones":0,"num_object_copies":249,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":83,"num_whiteouts":0,"num_read":29,"num_read_kb":23,"num_write":183,"num_write_kb":74,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":55,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"9.319843"},"pg_stats":[{"pgid":"3.3f","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-24T00:13:03.860443+0000","last_change":"2024-03-24T00:13:02.325815+0000","last_active":"2024-03-24T00:13:03.860443+0000","last_peered":"2024-03-24T00:13:03.860443+0000","last_clean":"2024-03-24T00:13:03.860443+0000","last_became_active":"2024-03-24T00:13:02.325450+0000","last_became_peered":"2024-03-24T00:13:02.325450+0000","last_unstale":"2024-03-24T00:13:03.860443+0000","last_undegraded":"2024-03-24T00:13:03.860443+0000","last_fullsized":"2024-03-24T00:13:03.860443+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:13:01.267198+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:13:01.267198+0000","last_clean_scrub_stamp":"2024-03-24T00:13:01.267198+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T02:18:21.994322+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-24T00:13:03.403477+0000","last_change":"2024-03-24T00:13:02.324015+0000","last_active":"2024-03-24T00:13:03.403477+0000","last_peered":"2024-03-24T00:13:03.403477+0000","last_clean":"2024-03-24T00:13:03.403477+0000","last_became_active":"2024-03-24T00:13:02.323668+0000","last_became_peered":"2024-03-24T00:13:02.323668+0000","last_unstale":"2024-03-24T00:13:03.403477+0000","last_undegraded":"2024-03-24T00:13:03.403477+0000","last_fullsized":"2024-03-24T00:13:03.403477+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:13:01.267198+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:13:01.267198+0000","last_clean_scrub_stamp":"2024-03-24T00:13:01.267198+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:36:19.442266+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11,10],"acting":[0,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-24T00:13:03.403525+0000","last_change":"2024-03-24T00:13:02.328911+0000","last_active":"2024-03-24T00:13:03.403525+0000","last_peered":"2024-03-24T00:13:03.403525+0000","last_clean":"2024-03-24T00:13:03.403525+0000","last_became_active":"2024-03-24T00:13:02.324333+0000","last_became_peered":"2024-03-24T00:13:02.324333+0000","last_unstale":"2024-03-24T00:13:03.403525+0000","last_undegraded":"2024-03-24T00:13:03.403525+0000","last_fullsized":"2024-03-24T00:13:03.403525+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:13:01.267198+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:13:01.267198+0000","last_clean_scrub_stamp":"2024-03-24T00:13:01.267198+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T02:49:02.318097+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-24T00:13:03.290642+0000","last_change":"2024-03-24T00:13:02.309057+0000","last_active":"2024-03-24T00:13:03.290642+0000","last_peered":"2024-03-24T00:13:03.290642+0000","last_clean":"2024-03-24T00:13:03.290642+0000","last_became_active":"2024-03-24T00:13:02.308403+0000","last_became_peered":"2024-03-24T00:13:02.308403+0000","last_unstale":"2024-03-24T00:13:03.290642+0000","last_undegraded":"2024-03-24T00:13:03.290642+0000","last_fullsized":"2024-03-24T00:13:03.290642+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:13:01.267198+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:13:01.267198+0000","last_clean_scrub_stamp":"2024-03-24T00:13:01.267198+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T07:54:44.310833+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,8,2],"acting":[4,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-24T00:13:03.597254+0000","last_change":"2024-03-24T00:13:02.330651+0000","last_active":"2024-03-24T00:13:03.597254+0000","last_peered":"2024-03-24T00:13:03.597254+0000","last_clean":"2024-03-24T00:13:03.597254+0000","last_became_active":"2024-03-24T00:13:02.330313+0000","last_became_peered":"2024-03-24T00:13:02.330313+0000","last_unstale":"2024-03-24T00:13:03.597254+0000","last_undegraded":"2024-03-24T00:13:03.597254+0000","last_fullsized":"2024-03-24T00:13:03.597254+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:13:01.267198+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:13:01.267198+0000","last_clean_scrub_stamp":"2024-03-24T00:13:01.267198+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T03:45:38.480929+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,6],"acting":[8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-24T00:13:04.365800+0000","last_change":"2024-03-24T00:13:02.313115+0000","last_active":"2024-03-24T00:13:04.365800+0000","last_peered":"2024-03-24T00:13:04.365800+0000","last_clean":"2024-03-24T00:13:04.365800+0000","last_became_active":"2024-03-24T00:13:02.312387+0000","last_became_peered":"2024-03-24T00:13:02.312387+0000","last_unstale":"2024-03-24T00:13:04.365800+0000","last_undegraded":"2024-03-24T00:13:04.365800+0000","last_fullsized":"2024-03-24T00:13:04.365800+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:13:01.267198+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:13:01.267198+0000","last_clean_scrub_stamp":"2024-03-24T00:13:01.267198+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T09:16:02.574920+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,4],"acting":[10,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-24T00:13:03.840695+0000","last_change":"2024-03-24T00:13:02.315117+0000","last_active":"2024-03-24T00:13:03.840695+0000","last_peered":"2024-03-24T00:13:03.840695+0000","last_clean":"2024-03-24T00:13:03.840695+0000","last_became_active":"2024-03-24T00:13:02.314834+0000","last_became_peered":"2024-03-24T00:13:02.314834+0000","last_unstale":"2024-03-24T00:13:03.840695+0000","last_undegraded":"2024-03-24T00:13:03.840695+0000","last_fullsized":"2024-03-24T00:13:03.840695+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:13:01.267198+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:13:01.267198+0000","last_clean_scrub_stamp":"2024-03-24T00:13:01.267198+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T08:05:17.021141+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-24T00:13:04.347277+0000","last_change":"2024-03-24T00:13:02.331381+0000","last_active":"2024-03-24T00:13:04.347277+0000","last_peered":"2024-03-24T00:13:04.347277+0000","last_clean":"2024-03-24T00:13:04.347277+0000","last_became_active":"2024-03-24T00:13:02.330883+0000","last_became_peered":"2024-03-24T00:13:02.330883+0000","last_unstale":"2024-03-24T00:13:04.347277+0000","last_undegraded":"2024-03-24T00:13:04.347277+0000","last_fullsized":"2024-03-24T00:13:04.347277+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:13:01.267198+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:13:01.267198+0000","last_clean_scrub_stamp":"2024-03-24T00:13:01.267198+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T00:21:17.021540+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,10],"acting":[3,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-24T00:13:04.366392+0000","last_change":"2024-03-24T00:13:02.305625+0000","last_active":"2024-03-24T00:13:04.366392+0000","last_peered":"2024-03-24T00:13:04.366392+0000","last_clean":"2024-03-24T00:13:04.366392+0000","last_became_active":"2024-03-24T00:13:02.305221+0000","last_became_peered":"2024-03-24T00:13:02.305221+0000","last_unstale":"2024-03-24T00:13:04.366392+0000","last_undegraded":"2024-03-24T00:13:04.366392+0000","last_fullsized":"2024-03-24T00:13:04.366392+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:13:01.267198+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:13:01.267198+0000","last_clean_scrub_stamp":"2024-03-24T00:13:01.267198+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T04:37:12.467387+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,11],"acting":[1,5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-24T00:13:04.366223+0000","last_change":"2024-03-24T00:13:02.307878+0000","last_active":"2024-03-24T00:13:04.366223+0000","last_peered":"2024-03-24T00:13:04.366223+0000","last_clean":"2024-03-24T00:13:04.366223+0000","last_became_active":"2024-03-24T00:13:02.307513+0000","last_became_peered":"2024-03-24T00:13:02.307513+0000","last_unstale":"2024-03-24T00:13:04.366223+0000","last_undegraded":"2024-03-24T00:13:04.366223+0000","last_fullsized":"2024-03-24T00:13:04.366223+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:13:01.267198+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:13:01.267198+0000","last_clean_scrub_stamp":"2024-03-24T00:13:01.267198+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T07:06:08.162416+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,4],"acting":[10,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-24T00:13:03.841008+0000","last_change":"2024-03-24T00:13:02.310883+0000","last_active":"2024-03-24T00:13:03.841008+0000","last_peered":"2024-03-24T00:13:03.841008+0000","last_clean":"2024-03-24T00:13:03.841008+0000","last_became_active":"2024-03-24T00:13:02.310559+0000","last_became_peered":"2024-03-24T00:13:02.310559+0000","last_unstale":"2024-03-24T00:13:03.841008+0000","last_undegraded":"2024-03-24T00:13:03.841008+0000","last_fullsized":"2024-03-24T00:13:03.841008+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:13:01.267198+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:13:01.267198+0000","last_clean_scrub_stamp":"2024-03-24T00:13:01.267198+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T04:04:25.801248+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8,1],"acting":[7,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-24T00:13:03.290545+0000","last_change":"2024-03-24T00:13:02.308747+0000","last_active":"2024-03-24T00:13:03.290545+0000","last_peered":"2024-03-24T00:13:03.290545+0000","last_clean":"2024-03-24T00:13:03.290545+0000","last_became_active":"2024-03-24T00:13:02.307791+0000","last_became_peered":"2024-03-24T00:13:02.307791+0000","last_unstale":"2024-03-24T00:13:03.290545+0000","last_undegraded":"2024-03-24T00:13:03.290545+0000","last_fullsized":"2024-03-24T00:13:03.290545+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:13:01.267198+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:13:01.267198+0000","last_clean_scrub_stamp":"2024-03-24T00:13:01.267198+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T03:28:08.780044+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,8],"acting":[4,10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-24T00:13:03.673934+0000","last_change":"2024-03-24T00:13:02.327061+0000","last_active":"2024-03-24T00:13:03.673934+0000","last_peered":"2024-03-24T00:13:03.673934+0000","last_clean":"2024-03-24T00:13:03.673934+0000","last_became_active":"2024-03-24T00:13:02.326696+0000","last_became_peered":"2024-03-24T00:13:02.326696+0000","last_unstale":"2024-03-24T00:13:03.673934+0000","last_undegraded":"2024-03-24T00:13:03.673934+0000","last_fullsized":"2024-03-24T00:13:03.673934+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:13:01.267198+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:13:01.267198+0000","last_clean_scrub_stamp":"2024-03-24T00:13:01.267198+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T08:55:17.325488+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,5],"acting":[6,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-24T00:13:03.402980+0000","last_change":"2024-03-24T00:13:02.321326+0000","last_active":"2024-03-24T00:13:03.402980+0000","last_peered":"2024-03-24T00:13:03.402980+0000","last_clean":"2024-03-24T00:13:03.402980+0000","last_became_active":"2024-03-24T00:13:02.320964+0000","last_became_peered":"2024-03-24T00:13:02.320964+0000","last_unstale":"2024-03-24T00:13:03.402980+0000","last_undegraded":"2024-03-24T00:13:03.402980+0000","last_fullsized":"2024-03-24T00:13:03.402980+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:13:01.267198+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:13:01.267198+0000","last_clean_scrub_stamp":"2024-03-24T00:13:01.267198+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T08:05:15.365809+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,10],"acting":[0,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-24T00:13:03.403404+0000","last_change":"2024-03-24T00:13:02.323922+0000","last_active":"2024-03-24T00:13:03.403404+0000","last_peered":"2024-03-24T00:13:03.403404+0000","last_clean":"2024-03-24T00:13:03.403404+0000","last_became_active":"2024-03-24T00:13:02.323440+0000","last_became_peered":"2024-03-24T00:13:02.323440+0000","last_unstale":"2024-03-24T00:13:03.403404+0000","last_undegraded":"2024-03-24T00:13:03.403404+0000","last_fullsized":"2024-03-24T00:13:03.403404+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:13:01.267198+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:13:01.267198+0000","last_clean_scrub_stamp":"2024-03-24T00:13:01.267198+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T01:26:49.115140+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,2],"acting":[0,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-24T00:13:04.365721+0000","last_change":"2024-03-24T00:13:02.325460+0000","last_active":"2024-03-24T00:13:04.365721+0000","last_peered":"2024-03-24T00:13:04.365721+0000","last_clean":"2024-03-24T00:13:04.365721+0000","last_became_active":"2024-03-24T00:13:02.325112+0000","last_became_peered":"2024-03-24T00:13:02.325112+0000","last_unstale":"2024-03-24T00:13:04.365721+0000","last_undegraded":"2024-03-24T00:13:04.365721+0000","last_fullsized":"2024-03-24T00:13:04.365721+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:13:01.267198+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:13:01.267198+0000","last_clean_scrub_stamp":"2024-03-24T00:13:01.267198+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T03:15:16.362970+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-24T00:13:03.860561+0000","last_change":"2024-03-24T00:13:02.325918+0000","last_active":"2024-03-24T00:13:03.860561+0000","last_peered":"2024-03-24T00:13:03.860561+0000","last_clean":"2024-03-24T00:13:03.860561+0000","last_became_active":"2024-03-24T00:13:02.325557+0000","last_became_peered":"2024-03-24T00:13:02.325557+0000","last_unstale":"2024-03-24T00:13:03.860561+0000","last_undegraded":"2024-03-24T00:13:03.860561+0000","last_fullsized":"2024-03-24T00:13:03.860561+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:13:01.267198+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:13:01.267198+0000","last_clean_scrub_stamp":"2024-03-24T00:13:01.267198+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T02:55:43.759674+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-24T00:13:03.288385+0000","last_change":"2024-03-24T00:13:02.308106+0000","last_active":"2024-03-24T00:13:03.288385+0000","last_peered":"2024-03-24T00:13:03.288385+0000","last_clean":"2024-03-24T00:13:03.288385+0000","last_became_active":"2024-03-24T00:13:02.306972+0000","last_became_peered":"2024-03-24T00:13:02.306972+0000","last_unstale":"2024-03-24T00:13:03.288385+0000","last_undegraded":"2024-03-24T00:13:03.288385+0000","last_fullsized":"2024-03-24T00:13:03.288385+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:13:01.267198+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:13:01.267198+0000","last_clean_scrub_stamp":"2024-03-24T00:13:01.267198+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T09:38:54.677255+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,10],"acting":[9,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-24T00:13:03.289954+0000","last_change":"2024-03-24T00:13:02.331742+0000","last_active":"2024-03-24T00:13:03.289954+0000","last_peered":"2024-03-24T00:13:03.289954+0000","last_clean":"2024-03-24T00:13:03.289954+0000","last_became_active":"2024-03-24T00:13:02.331103+0000","last_became_peered":"2024-03-24T00:13:02.331103+0000","last_unstale":"2024-03-24T00:13:03.289954+0000","last_undegraded":"2024-03-24T00:13:03.289954+0000","last_fullsized":"2024-03-24T00:13:03.289954+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:13:01.267198+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:13:01.267198+0000","last_clean_scrub_stamp":"2024-03-24T00:13:01.267198+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:25:58.116306+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1f","version":"86'11","reported_seq":27,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-24T00:13:07.331461+0000","last_change":"2024-03-24T00:13:01.310276+0000","last_active":"2024-03-24T00:13:07.331461+0000","last_peered":"2024-03-24T00:13:07.331461+0000","last_clean":"2024-03-24T00:13:07.331461+0000","last_became_active":"2024-03-24T00:13:01.308169+0000","last_became_peered":"2024-03-24T00:13:01.308169+0000","last_unstale":"2024-03-24T00:13:07.331461+0000","last_undegraded":"2024-03-24T00:13:07.331461+0000","last_fullsized":"2024-03-24T00:13:07.331461+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:13:00.222790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:13:00.222790+0000","last_clean_scrub_stamp":"2024-03-24T00:13:00.222790+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T05:41:02.833903+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":13,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,8],"acting":[9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-24T00:13:04.348213+0000","last_change":"2024-03-24T00:13:02.322461+0000","last_active":"2024-03-24T00:13:04.348213+0000","last_peered":"2024-03-24T00:13:04.348213+0000","last_clean":"2024-03-24T00:13:04.348213+0000","last_became_active":"2024-03-24T00:13:02.321752+0000","last_became_peered":"2024-03-24T00:13:02.321752+0000","last_unstale":"2024-03-24T00:13:04.348213+0000","last_undegraded":"2024-03-24T00:13:04.348213+0000","last_fullsized":"2024-03-24T00:13:04.348213+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:13:01.267198+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:13:01.267198+0000","last_clean_scrub_stamp":"2024-03-24T00:13:01.267198+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T06:37:05.825919+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1e","version":"86'2","reported_seq":17,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-24T00:13:04.348878+0000","last_change":"2024-03-24T00:13:01.324972+0000","last_active":"2024-03-24T00:13:04.348878+0000","last_peered":"2024-03-24T00:13:04.348878+0000","last_clean":"2024-03-24T00:13:04.348878+0000","last_became_active":"2024-03-24T00:13:01.323342+0000","last_became_peered":"2024-03-24T00:13:01.323342+0000","last_unstale":"2024-03-24T00:13:04.348878+0000","last_undegraded":"2024-03-24T00:13:04.348878+0000","last_fullsized":"2024-03-24T00:13:04.348878+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:13:00.222790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:13:00.222790+0000","last_clean_scrub_stamp":"2024-03-24T00:13:00.222790+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T05:34:35.827634+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-24T00:13:03.404090+0000","last_change":"2024-03-24T00:13:02.325455+0000","last_active":"2024-03-24T00:13:03.404090+0000","last_peered":"2024-03-24T00:13:03.404090+0000","last_clean":"2024-03-24T00:13:03.404090+0000","last_became_active":"2024-03-24T00:13:02.325061+0000","last_became_peered":"2024-03-24T00:13:02.325061+0000","last_unstale":"2024-03-24T00:13:03.404090+0000","last_undegraded":"2024-03-24T00:13:03.404090+0000","last_fullsized":"2024-03-24T00:13:03.404090+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:13:01.267198+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:13:01.267198+0000","last_clean_scrub_stamp":"2024-03-24T00:13:01.267198+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T06:13:33.223361+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,9],"acting":[0,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1d","version":"86'7","reported_seq":22,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-24T00:13:07.327587+0000","last_change":"2024-03-24T00:13:01.300541+0000","last_active":"2024-03-24T00:13:07.327587+0000","last_peered":"2024-03-24T00:13:07.327587+0000","last_clean":"2024-03-24T00:13:07.327587+0000","last_became_active":"2024-03-24T00:13:01.299686+0000","last_became_peered":"2024-03-24T00:13:01.299686+0000","last_unstale":"2024-03-24T00:13:07.327587+0000","last_undegraded":"2024-03-24T00:13:07.327587+0000","last_fullsized":"2024-03-24T00:13:07.327587+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:13:00.222790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:13:00.222790+0000","last_clean_scrub_stamp":"2024-03-24T00:13:00.222790+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T00:20:54.474131+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-24T00:13:04.365967+0000","last_change":"2024-03-24T00:13:02.325130+0000","last_active":"2024-03-24T00:13:04.365967+0000","last_peered":"2024-03-24T00:13:04.365967+0000","last_clean":"2024-03-24T00:13:04.365967+0000","last_became_active":"2024-03-24T00:13:02.324740+0000","last_became_peered":"2024-03-24T00:13:02.324740+0000","last_unstale":"2024-03-24T00:13:04.365967+0000","last_undegraded":"2024-03-24T00:13:04.365967+0000","last_fullsized":"2024-03-24T00:13:04.365967+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:13:01.267198+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:13:01.267198+0000","last_clean_scrub_stamp":"2024-03-24T00:13:01.267198+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T01:23:35.194788+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,6],"acting":[1,10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1c","version":"85'2","reported_seq":17,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-24T00:13:03.597568+0000","last_change":"2024-03-24T00:13:01.300572+0000","last_active":"2024-03-24T00:13:03.597568+0000","last_peered":"2024-03-24T00:13:03.597568+0000","last_clean":"2024-03-24T00:13:03.597568+0000","last_became_active":"2024-03-24T00:13:01.299801+0000","last_became_peered":"2024-03-24T00:13:01.299801+0000","last_unstale":"2024-03-24T00:13:03.597568+0000","last_undegraded":"2024-03-24T00:13:03.597568+0000","last_fullsized":"2024-03-24T00:13:03.597568+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:13:00.222790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:13:00.222790+0000","last_clean_scrub_stamp":"2024-03-24T00:13:00.222790+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T10:29:03.986969+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,5],"acting":[8,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-24T00:13:03.429608+0000","last_change":"2024-03-24T00:13:02.304079+0000","last_active":"2024-03-24T00:13:03.429608+0000","last_peered":"2024-03-24T00:13:03.429608+0000","last_clean":"2024-03-24T00:13:03.429608+0000","last_became_active":"2024-03-24T00:13:02.303700+0000","last_became_peered":"2024-03-24T00:13:02.303700+0000","last_unstale":"2024-03-24T00:13:03.429608+0000","last_undegraded":"2024-03-24T00:13:03.429608+0000","last_fullsized":"2024-03-24T00:13:03.429608+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:13:01.267198+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:13:01.267198+0000","last_clean_scrub_stamp":"2024-03-24T00:13:01.267198+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T10:19:07.701927+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1b","version":"86'5","reported_seq":24,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-24T00:13:06.280203+0000","last_change":"2024-03-24T00:13:01.300949+0000","last_active":"2024-03-24T00:13:06.280203+0000","last_peered":"2024-03-24T00:13:06.280203+0000","last_clean":"2024-03-24T00:13:06.280203+0000","last_became_active":"2024-03-24T00:13:01.300605+0000","last_became_peered":"2024-03-24T00:13:01.300605+0000","last_unstale":"2024-03-24T00:13:06.280203+0000","last_undegraded":"2024-03-24T00:13:06.280203+0000","last_fullsized":"2024-03-24T00:13:06.280203+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:13:00.222790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:13:00.222790+0000","last_clean_scrub_stamp":"2024-03-24T00:13:00.222790+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T02:47:58.283103+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":622,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":4,"num_read_kb":2,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-24T00:13:03.597357+0000","last_change":"2024-03-24T00:13:02.312884+0000","last_active":"2024-03-24T00:13:03.597357+0000","last_peered":"2024-03-24T00:13:03.597357+0000","last_clean":"2024-03-24T00:13:03.597357+0000","last_became_active":"2024-03-24T00:13:02.312533+0000","last_became_peered":"2024-03-24T00:13:02.312533+0000","last_unstale":"2024-03-24T00:13:03.597357+0000","last_undegraded":"2024-03-24T00:13:03.597357+0000","last_fullsized":"2024-03-24T00:13:03.597357+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:13:01.267198+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:13:01.267198+0000","last_clean_scrub_stamp":"2024-03-24T00:13:01.267198+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T10:54:48.398645+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,5],"acting":[8,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.1a","version":"86'7","reported_seq":22,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-24T00:13:07.331054+0000","last_change":"2024-03-24T00:13:01.306292+0000","last_active":"2024-03-24T00:13:07.331054+0000","last_peered":"2024-03-24T00:13:07.331054+0000","last_clean":"2024-03-24T00:13:07.331054+0000","last_became_active":"2024-03-24T00:13:01.305898+0000","last_became_peered":"2024-03-24T00:13:01.305898+0000","last_unstale":"2024-03-24T00:13:07.331054+0000","last_undegraded":"2024-03-24T00:13:07.331054+0000","last_fullsized":"2024-03-24T00:13:07.331054+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:13:00.222790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:13:00.222790+0000","last_clean_scrub_stamp":"2024-03-24T00:13:00.222790+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T07:41:17.003588+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8,6],"acting":[10,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-24T00:13:03.429437+0000","last_change":"2024-03-24T00:13:02.323763+0000","last_active":"2024-03-24T00:13:03.429437+0000","last_peered":"2024-03-24T00:13:03.429437+0000","last_clean":"2024-03-24T00:13:03.429437+0000","last_became_active":"2024-03-24T00:13:02.323367+0000","last_became_peered":"2024-03-24T00:13:02.323367+0000","last_unstale":"2024-03-24T00:13:03.429437+0000","last_undegraded":"2024-03-24T00:13:03.429437+0000","last_fullsized":"2024-03-24T00:13:03.429437+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:13:01.267198+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:13:01.267198+0000","last_clean_scrub_stamp":"2024-03-24T00:13:01.267198+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T08:53:21.671317+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,3],"acting":[5,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.19","version":"86'2","reported_seq":17,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-24T00:13:06.298333+0000","last_change":"2024-03-24T00:13:01.323111+0000","last_active":"2024-03-24T00:13:06.298333+0000","last_peered":"2024-03-24T00:13:06.298333+0000","last_clean":"2024-03-24T00:13:06.298333+0000","last_became_active":"2024-03-24T00:13:01.322772+0000","last_became_peered":"2024-03-24T00:13:01.322772+0000","last_unstale":"2024-03-24T00:13:06.298333+0000","last_undegraded":"2024-03-24T00:13:06.298333+0000","last_fullsized":"2024-03-24T00:13:06.298333+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:13:00.222790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:13:00.222790+0000","last_clean_scrub_stamp":"2024-03-24T00:13:00.222790+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T04:19:53.097933+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,9],"acting":[6,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-24T00:13:04.347578+0000","last_change":"2024-03-24T00:13:02.330469+0000","last_active":"2024-03-24T00:13:04.347578+0000","last_peered":"2024-03-24T00:13:04.347578+0000","last_clean":"2024-03-24T00:13:04.347578+0000","last_became_active":"2024-03-24T00:13:02.327019+0000","last_became_peered":"2024-03-24T00:13:02.327019+0000","last_unstale":"2024-03-24T00:13:04.347578+0000","last_undegraded":"2024-03-24T00:13:04.347578+0000","last_fullsized":"2024-03-24T00:13:04.347578+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:13:01.267198+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:13:01.267198+0000","last_clean_scrub_stamp":"2024-03-24T00:13:01.267198+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T07:30:37.155535+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,6],"acting":[3,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.18","version":"86'2","reported_seq":17,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-24T00:13:05.401573+0000","last_change":"2024-03-24T00:13:01.300089+0000","last_active":"2024-03-24T00:13:05.401573+0000","last_peered":"2024-03-24T00:13:05.401573+0000","last_clean":"2024-03-24T00:13:05.401573+0000","last_became_active":"2024-03-24T00:13:01.299747+0000","last_became_peered":"2024-03-24T00:13:01.299747+0000","last_unstale":"2024-03-24T00:13:05.401573+0000","last_undegraded":"2024-03-24T00:13:05.401573+0000","last_fullsized":"2024-03-24T00:13:05.401573+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:13:00.222790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:13:00.222790+0000","last_clean_scrub_stamp":"2024-03-24T00:13:00.222790+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T00:30:30.142785+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-24T00:13:03.287810+0000","last_change":"2024-03-24T00:13:02.332490+0000","last_active":"2024-03-24T00:13:03.287810+0000","last_peered":"2024-03-24T00:13:03.287810+0000","last_clean":"2024-03-24T00:13:03.287810+0000","last_became_active":"2024-03-24T00:13:02.331182+0000","last_became_peered":"2024-03-24T00:13:02.331182+0000","last_unstale":"2024-03-24T00:13:03.287810+0000","last_undegraded":"2024-03-24T00:13:03.287810+0000","last_fullsized":"2024-03-24T00:13:03.287810+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:13:01.267198+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:13:01.267198+0000","last_clean_scrub_stamp":"2024-03-24T00:13:01.267198+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T06:22:16.084879+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,10],"acting":[9,3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.17","version":"86'4","reported_seq":22,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-24T00:13:07.331208+0000","last_change":"2024-03-24T00:13:01.299818+0000","last_active":"2024-03-24T00:13:07.331208+0000","last_peered":"2024-03-24T00:13:07.331208+0000","last_clean":"2024-03-24T00:13:07.331208+0000","last_became_active":"2024-03-24T00:13:01.299478+0000","last_became_peered":"2024-03-24T00:13:01.299478+0000","last_unstale":"2024-03-24T00:13:07.331208+0000","last_undegraded":"2024-03-24T00:13:07.331208+0000","last_fullsized":"2024-03-24T00:13:07.331208+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:13:00.222790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:13:00.222790+0000","last_clean_scrub_stamp":"2024-03-24T00:13:00.222790+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:51:47.088677+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":180,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":3,"num_read_kb":3,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,9],"acting":[10,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-24T00:13:04.366406+0000","last_change":"2024-03-24T00:13:02.304971+0000","last_active":"2024-03-24T00:13:04.366406+0000","last_peered":"2024-03-24T00:13:04.366406+0000","last_clean":"2024-03-24T00:13:04.366406+0000","last_became_active":"2024-03-24T00:13:02.304577+0000","last_became_peered":"2024-03-24T00:13:02.304577+0000","last_unstale":"2024-03-24T00:13:04.366406+0000","last_undegraded":"2024-03-24T00:13:04.366406+0000","last_fullsized":"2024-03-24T00:13:04.366406+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:13:01.267198+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:13:01.267198+0000","last_clean_scrub_stamp":"2024-03-24T00:13:01.267198+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T05:08:03.444554+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,9],"acting":[1,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.16","version":"86'2","reported_seq":17,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-24T00:13:05.400208+0000","last_change":"2024-03-24T00:13:01.303075+0000","last_active":"2024-03-24T00:13:05.400208+0000","last_peered":"2024-03-24T00:13:05.400208+0000","last_clean":"2024-03-24T00:13:05.400208+0000","last_became_active":"2024-03-24T00:13:01.301517+0000","last_became_peered":"2024-03-24T00:13:01.301517+0000","last_unstale":"2024-03-24T00:13:05.400208+0000","last_undegraded":"2024-03-24T00:13:05.400208+0000","last_fullsized":"2024-03-24T00:13:05.400208+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:13:00.222790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:13:00.222790+0000","last_clean_scrub_stamp":"2024-03-24T00:13:00.222790+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:04:10.552510+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-24T00:13:03.403947+0000","last_change":"2024-03-24T00:13:02.320108+0000","last_active":"2024-03-24T00:13:03.403947+0000","last_peered":"2024-03-24T00:13:03.403947+0000","last_clean":"2024-03-24T00:13:03.403947+0000","last_became_active":"2024-03-24T00:13:02.319155+0000","last_became_peered":"2024-03-24T00:13:02.319155+0000","last_unstale":"2024-03-24T00:13:03.403947+0000","last_undegraded":"2024-03-24T00:13:03.403947+0000","last_fullsized":"2024-03-24T00:13:03.403947+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:13:01.267198+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:13:01.267198+0000","last_clean_scrub_stamp":"2024-03-24T00:13:01.267198+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T10:15:54.770304+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,5],"acting":[0,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.15","version":"86'5","reported_seq":21,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-24T00:13:06.279634+0000","last_change":"2024-03-24T00:13:01.323919+0000","last_active":"2024-03-24T00:13:06.279634+0000","last_peered":"2024-03-24T00:13:06.279634+0000","last_clean":"2024-03-24T00:13:06.279634+0000","last_became_active":"2024-03-24T00:13:01.323640+0000","last_became_peered":"2024-03-24T00:13:01.323640+0000","last_unstale":"2024-03-24T00:13:06.279634+0000","last_undegraded":"2024-03-24T00:13:06.279634+0000","last_fullsized":"2024-03-24T00:13:06.279634+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:13:00.222790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:13:00.222790+0000","last_clean_scrub_stamp":"2024-03-24T00:13:00.222790+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T06:16:49.018642+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":7,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,10],"acting":[5,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-24T00:13:04.366231+0000","last_change":"2024-03-24T00:13:02.304945+0000","last_active":"2024-03-24T00:13:04.366231+0000","last_peered":"2024-03-24T00:13:04.366231+0000","last_clean":"2024-03-24T00:13:04.366231+0000","last_became_active":"2024-03-24T00:13:02.304599+0000","last_became_peered":"2024-03-24T00:13:02.304599+0000","last_unstale":"2024-03-24T00:13:04.366231+0000","last_undegraded":"2024-03-24T00:13:04.366231+0000","last_fullsized":"2024-03-24T00:13:04.366231+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:13:01.267198+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:13:01.267198+0000","last_clean_scrub_stamp":"2024-03-24T00:13:01.267198+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T05:32:02.763946+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,4],"acting":[1,8,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.14","version":"86'17","reported_seq":40,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-24T00:13:08.834016+0000","last_change":"2024-03-24T00:13:01.301460+0000","last_active":"2024-03-24T00:13:08.834016+0000","last_peered":"2024-03-24T00:13:08.834016+0000","last_clean":"2024-03-24T00:13:08.834016+0000","last_became_active":"2024-03-24T00:13:01.301014+0000","last_became_peered":"2024-03-24T00:13:01.301014+0000","last_unstale":"2024-03-24T00:13:08.834016+0000","last_undegraded":"2024-03-24T00:13:08.834016+0000","last_fullsized":"2024-03-24T00:13:08.834016+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:13:00.222790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:13:00.222790+0000","last_clean_scrub_stamp":"2024-03-24T00:13:00.222790+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T01:05:15.984359+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20895,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":8,"num_read_kb":5,"num_write":19,"num_write_kb":24,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,2],"acting":[10,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-24T00:13:03.597622+0000","last_change":"2024-03-24T00:13:02.307653+0000","last_active":"2024-03-24T00:13:03.597622+0000","last_peered":"2024-03-24T00:13:03.597622+0000","last_clean":"2024-03-24T00:13:03.597622+0000","last_became_active":"2024-03-24T00:13:02.307287+0000","last_became_peered":"2024-03-24T00:13:02.307287+0000","last_unstale":"2024-03-24T00:13:03.597622+0000","last_undegraded":"2024-03-24T00:13:03.597622+0000","last_fullsized":"2024-03-24T00:13:03.597622+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:13:01.267198+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:13:01.267198+0000","last_clean_scrub_stamp":"2024-03-24T00:13:01.267198+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T05:59:27.701614+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.13","version":"86'9","reported_seq":26,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-24T00:13:08.345059+0000","last_change":"2024-03-24T00:13:01.316854+0000","last_active":"2024-03-24T00:13:08.345059+0000","last_peered":"2024-03-24T00:13:08.345059+0000","last_clean":"2024-03-24T00:13:08.345059+0000","last_became_active":"2024-03-24T00:13:01.316507+0000","last_became_peered":"2024-03-24T00:13:01.316507+0000","last_unstale":"2024-03-24T00:13:08.345059+0000","last_undegraded":"2024-03-24T00:13:08.345059+0000","last_fullsized":"2024-03-24T00:13:08.345059+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:13:00.222790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:13:00.222790+0000","last_clean_scrub_stamp":"2024-03-24T00:13:00.222790+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T07:58:41.888003+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":13,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,10,3],"acting":[2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-24T00:13:03.673410+0000","last_change":"2024-03-24T00:13:02.338811+0000","last_active":"2024-03-24T00:13:03.673410+0000","last_peered":"2024-03-24T00:13:03.673410+0000","last_clean":"2024-03-24T00:13:03.673410+0000","last_became_active":"2024-03-24T00:13:02.338438+0000","last_became_peered":"2024-03-24T00:13:02.338438+0000","last_unstale":"2024-03-24T00:13:03.673410+0000","last_undegraded":"2024-03-24T00:13:03.673410+0000","last_fullsized":"2024-03-24T00:13:03.673410+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:13:01.267198+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:13:01.267198+0000","last_clean_scrub_stamp":"2024-03-24T00:13:01.267198+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T03:55:16.531224+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.12","version":"86'4","reported_seq":19,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-24T00:13:05.403830+0000","last_change":"2024-03-24T00:13:01.313914+0000","last_active":"2024-03-24T00:13:05.403830+0000","last_peered":"2024-03-24T00:13:05.403830+0000","last_clean":"2024-03-24T00:13:05.403830+0000","last_became_active":"2024-03-24T00:13:01.313535+0000","last_became_peered":"2024-03-24T00:13:01.313535+0000","last_unstale":"2024-03-24T00:13:05.403830+0000","last_undegraded":"2024-03-24T00:13:05.403830+0000","last_fullsized":"2024-03-24T00:13:05.403830+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:13:00.222790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:13:00.222790+0000","last_clean_scrub_stamp":"2024-03-24T00:13:00.222790+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T03:23:53.011550+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-24T00:13:03.840909+0000","last_change":"2024-03-24T00:13:02.334755+0000","last_active":"2024-03-24T00:13:03.840909+0000","last_peered":"2024-03-24T00:13:03.840909+0000","last_clean":"2024-03-24T00:13:03.840909+0000","last_became_active":"2024-03-24T00:13:02.334444+0000","last_became_peered":"2024-03-24T00:13:02.334444+0000","last_unstale":"2024-03-24T00:13:03.840909+0000","last_undegraded":"2024-03-24T00:13:03.840909+0000","last_fullsized":"2024-03-24T00:13:03.840909+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:13:01.267198+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:13:01.267198+0000","last_clean_scrub_stamp":"2024-03-24T00:13:01.267198+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T06:05:44.006815+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.11","version":"86'5","reported_seq":20,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-24T00:13:07.324761+0000","last_change":"2024-03-24T00:13:01.297887+0000","last_active":"2024-03-24T00:13:07.324761+0000","last_peered":"2024-03-24T00:13:07.324761+0000","last_clean":"2024-03-24T00:13:07.324761+0000","last_became_active":"2024-03-24T00:13:01.297560+0000","last_became_peered":"2024-03-24T00:13:01.297560+0000","last_unstale":"2024-03-24T00:13:07.324761+0000","last_undegraded":"2024-03-24T00:13:07.324761+0000","last_fullsized":"2024-03-24T00:13:07.324761+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:13:00.222790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:13:00.222790+0000","last_clean_scrub_stamp":"2024-03-24T00:13:00.222790+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T01:06:28.927274+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":588,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,10],"acting":[4,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-24T00:13:03.840682+0000","last_change":"2024-03-24T00:13:02.306811+0000","last_active":"2024-03-24T00:13:03.840682+0000","last_peered":"2024-03-24T00:13:03.840682+0000","last_clean":"2024-03-24T00:13:03.840682+0000","last_became_active":"2024-03-24T00:13:02.306499+0000","last_became_peered":"2024-03-24T00:13:02.306499+0000","last_unstale":"2024-03-24T00:13:03.840682+0000","last_undegraded":"2024-03-24T00:13:03.840682+0000","last_fullsized":"2024-03-24T00:13:03.840682+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:13:01.267198+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:13:01.267198+0000","last_clean_scrub_stamp":"2024-03-24T00:13:01.267198+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:49:32.683005+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.10","version":"86'16","reported_seq":31,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-24T00:13:06.298946+0000","last_change":"2024-03-24T00:13:01.307058+0000","last_active":"2024-03-24T00:13:06.298946+0000","last_peered":"2024-03-24T00:13:06.298946+0000","last_clean":"2024-03-24T00:13:06.298946+0000","last_became_active":"2024-03-24T00:13:01.306699+0000","last_became_peered":"2024-03-24T00:13:01.306699+0000","last_unstale":"2024-03-24T00:13:06.298946+0000","last_undegraded":"2024-03-24T00:13:06.298946+0000","last_fullsized":"2024-03-24T00:13:06.298946+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:13:00.222790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:13:00.222790+0000","last_clean_scrub_stamp":"2024-03-24T00:13:00.222790+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T10:32:37.334514+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,2],"acting":[8,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-24T00:13:03.429225+0000","last_change":"2024-03-24T00:13:02.305052+0000","last_active":"2024-03-24T00:13:03.429225+0000","last_peered":"2024-03-24T00:13:03.429225+0000","last_clean":"2024-03-24T00:13:03.429225+0000","last_became_active":"2024-03-24T00:13:02.304700+0000","last_became_peered":"2024-03-24T00:13:02.304700+0000","last_unstale":"2024-03-24T00:13:03.429225+0000","last_undegraded":"2024-03-24T00:13:03.429225+0000","last_fullsized":"2024-03-24T00:13:03.429225+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:13:01.267198+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:13:01.267198+0000","last_clean_scrub_stamp":"2024-03-24T00:13:01.267198+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T00:43:10.413894+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.f","version":"85'1","reported_seq":17,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-24T00:13:03.597698+0000","last_change":"2024-03-24T00:13:01.295874+0000","last_active":"2024-03-24T00:13:03.597698+0000","last_peered":"2024-03-24T00:13:03.597698+0000","last_clean":"2024-03-24T00:13:03.597698+0000","last_became_active":"2024-03-24T00:13:01.295521+0000","last_became_peered":"2024-03-24T00:13:01.295521+0000","last_unstale":"2024-03-24T00:13:03.597698+0000","last_undegraded":"2024-03-24T00:13:03.597698+0000","last_fullsized":"2024-03-24T00:13:03.597698+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:13:00.222790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:13:00.222790+0000","last_clean_scrub_stamp":"2024-03-24T00:13:00.222790+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T06:47:15.749491+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5,4],"acting":[8,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-24T00:13:03.860319+0000","last_change":"2024-03-24T00:13:02.306617+0000","last_active":"2024-03-24T00:13:03.860319+0000","last_peered":"2024-03-24T00:13:03.860319+0000","last_clean":"2024-03-24T00:13:03.860319+0000","last_became_active":"2024-03-24T00:13:02.306224+0000","last_became_peered":"2024-03-24T00:13:02.306224+0000","last_unstale":"2024-03-24T00:13:03.860319+0000","last_undegraded":"2024-03-24T00:13:03.860319+0000","last_fullsized":"2024-03-24T00:13:03.860319+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:13:01.267198+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:13:01.267198+0000","last_clean_scrub_stamp":"2024-03-24T00:13:01.267198+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T06:49:16.032741+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,4],"acting":[11,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.4","version":"86'1","reported_seq":16,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-24T00:13:03.288098+0000","last_change":"2024-03-24T00:13:01.314428+0000","last_active":"2024-03-24T00:13:03.288098+0000","last_peered":"2024-03-24T00:13:03.288098+0000","last_clean":"2024-03-24T00:13:03.288098+0000","last_became_active":"2024-03-24T00:13:01.313421+0000","last_became_peered":"2024-03-24T00:13:01.313421+0000","last_unstale":"2024-03-24T00:13:03.288098+0000","last_undegraded":"2024-03-24T00:13:03.288098+0000","last_fullsized":"2024-03-24T00:13:03.288098+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:13:00.222790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:13:00.222790+0000","last_clean_scrub_stamp":"2024-03-24T00:13:00.222790+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T08:10:25.796718+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,0],"acting":[9,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.5","version":"86'1","reported_seq":14,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-24T00:13:04.366169+0000","last_change":"2024-03-24T00:13:02.331660+0000","last_active":"2024-03-24T00:13:04.366169+0000","last_peered":"2024-03-24T00:13:04.366169+0000","last_clean":"2024-03-24T00:13:04.366169+0000","last_became_active":"2024-03-24T00:13:02.331288+0000","last_became_peered":"2024-03-24T00:13:02.331288+0000","last_unstale":"2024-03-24T00:13:04.366169+0000","last_undegraded":"2024-03-24T00:13:04.366169+0000","last_fullsized":"2024-03-24T00:13:04.366169+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:13:01.267198+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:13:01.267198+0000","last_clean_scrub_stamp":"2024-03-24T00:13:01.267198+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:36:55.576510+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3,6],"acting":[10,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.2","version":"86'1","reported_seq":16,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-24T00:13:03.840849+0000","last_change":"2024-03-24T00:13:01.310538+0000","last_active":"2024-03-24T00:13:03.840849+0000","last_peered":"2024-03-24T00:13:03.840849+0000","last_clean":"2024-03-24T00:13:03.840849+0000","last_became_active":"2024-03-24T00:13:01.310124+0000","last_became_peered":"2024-03-24T00:13:01.310124+0000","last_unstale":"2024-03-24T00:13:03.840849+0000","last_undegraded":"2024-03-24T00:13:03.840849+0000","last_fullsized":"2024-03-24T00:13:03.840849+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:13:00.222790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:13:00.222790+0000","last_clean_scrub_stamp":"2024-03-24T00:13:00.222790+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T05:47:03.534824+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-24T00:13:04.365996+0000","last_change":"2024-03-24T00:13:02.324117+0000","last_active":"2024-03-24T00:13:04.365996+0000","last_peered":"2024-03-24T00:13:04.365996+0000","last_clean":"2024-03-24T00:13:04.365996+0000","last_became_active":"2024-03-24T00:13:02.323706+0000","last_became_peered":"2024-03-24T00:13:02.323706+0000","last_unstale":"2024-03-24T00:13:04.365996+0000","last_undegraded":"2024-03-24T00:13:04.365996+0000","last_fullsized":"2024-03-24T00:13:04.365996+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:13:01.267198+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:13:01.267198+0000","last_clean_scrub_stamp":"2024-03-24T00:13:01.267198+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T05:34:31.657324+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,4,0],"acting":[10,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-24T00:13:03.429464+0000","last_change":"2024-03-24T00:13:02.306030+0000","last_active":"2024-03-24T00:13:03.429464+0000","last_peered":"2024-03-24T00:13:03.429464+0000","last_clean":"2024-03-24T00:13:03.429464+0000","last_became_active":"2024-03-24T00:13:02.305588+0000","last_became_peered":"2024-03-24T00:13:02.305588+0000","last_unstale":"2024-03-24T00:13:03.429464+0000","last_undegraded":"2024-03-24T00:13:03.429464+0000","last_fullsized":"2024-03-24T00:13:03.429464+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:13:01.267198+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:13:01.267198+0000","last_clean_scrub_stamp":"2024-03-24T00:13:01.267198+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:20:58.206244+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,4],"acting":[5,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"86'4","reported_seq":19,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-24T00:13:06.298832+0000","last_change":"2024-03-24T00:13:01.318123+0000","last_active":"2024-03-24T00:13:06.298832+0000","last_peered":"2024-03-24T00:13:06.298832+0000","last_clean":"2024-03-24T00:13:06.298832+0000","last_became_active":"2024-03-24T00:13:01.317683+0000","last_became_peered":"2024-03-24T00:13:01.317683+0000","last_unstale":"2024-03-24T00:13:06.298832+0000","last_undegraded":"2024-03-24T00:13:06.298832+0000","last_fullsized":"2024-03-24T00:13:06.298832+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:13:00.222790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:13:00.222790+0000","last_clean_scrub_stamp":"2024-03-24T00:13:00.222790+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T06:33:02.669668+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-24T00:13:04.348063+0000","last_change":"2024-03-24T00:13:02.328599+0000","last_active":"2024-03-24T00:13:04.348063+0000","last_peered":"2024-03-24T00:13:04.348063+0000","last_clean":"2024-03-24T00:13:04.348063+0000","last_became_active":"2024-03-24T00:13:02.328165+0000","last_became_peered":"2024-03-24T00:13:02.328165+0000","last_unstale":"2024-03-24T00:13:04.348063+0000","last_undegraded":"2024-03-24T00:13:04.348063+0000","last_fullsized":"2024-03-24T00:13:04.348063+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:13:01.267198+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:13:01.267198+0000","last_clean_scrub_stamp":"2024-03-24T00:13:01.267198+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T07:22:39.646578+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"86'6","reported_seq":22,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-24T00:13:07.331671+0000","last_change":"2024-03-24T00:13:01.301932+0000","last_active":"2024-03-24T00:13:07.331671+0000","last_peered":"2024-03-24T00:13:07.331671+0000","last_clean":"2024-03-24T00:13:07.331671+0000","last_became_active":"2024-03-24T00:13:01.301464+0000","last_became_peered":"2024-03-24T00:13:01.301464+0000","last_unstale":"2024-03-24T00:13:07.331671+0000","last_undegraded":"2024-03-24T00:13:07.331671+0000","last_fullsized":"2024-03-24T00:13:07.331671+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:13:00.222790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:13:00.222790+0000","last_clean_scrub_stamp":"2024-03-24T00:13:00.222790+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T08:58:25.512421+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":8,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,9],"acting":[4,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.0","version":"21'137","reported_seq":340,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-24T00:13:03.840668+0000","last_change":"2024-03-24T00:11:43.008767+0000","last_active":"2024-03-24T00:13:03.840668+0000","last_peered":"2024-03-24T00:13:03.840668+0000","last_clean":"2024-03-24T00:13:03.840668+0000","last_became_active":"2024-03-24T00:11:43.008364+0000","last_became_peered":"2024-03-24T00:11:43.008364+0000","last_unstale":"2024-03-24T00:13:03.840668+0000","last_undegraded":"2024-03-24T00:13:03.840668+0000","last_fullsized":"2024-03-24T00:13:03.840668+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:07:50.191828+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:07:50.191828+0000","last_clean_scrub_stamp":"2024-03-24T00:07:50.191828+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T01:55:26.549348+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-24T00:13:03.672974+0000","last_change":"2024-03-24T00:13:02.325667+0000","last_active":"2024-03-24T00:13:03.672974+0000","last_peered":"2024-03-24T00:13:03.672974+0000","last_clean":"2024-03-24T00:13:03.672974+0000","last_became_active":"2024-03-24T00:13:02.322476+0000","last_became_peered":"2024-03-24T00:13:02.322476+0000","last_unstale":"2024-03-24T00:13:03.672974+0000","last_undegraded":"2024-03-24T00:13:03.672974+0000","last_fullsized":"2024-03-24T00:13:03.672974+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:13:01.267198+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:13:01.267198+0000","last_clean_scrub_stamp":"2024-03-24T00:13:01.267198+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T08:07:19.936708+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"86'3","reported_seq":18,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-24T00:13:06.292557+0000","last_change":"2024-03-24T00:13:01.298082+0000","last_active":"2024-03-24T00:13:06.292557+0000","last_peered":"2024-03-24T00:13:06.292557+0000","last_clean":"2024-03-24T00:13:06.292557+0000","last_became_active":"2024-03-24T00:13:01.297686+0000","last_became_peered":"2024-03-24T00:13:01.297686+0000","last_unstale":"2024-03-24T00:13:06.292557+0000","last_undegraded":"2024-03-24T00:13:06.292557+0000","last_fullsized":"2024-03-24T00:13:06.292557+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:13:00.222790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:13:00.222790+0000","last_clean_scrub_stamp":"2024-03-24T00:13:00.222790+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T09:25:07.204076+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.5","version":"86'3","reported_seq":19,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-24T00:13:07.332496+0000","last_change":"2024-03-24T00:13:01.321250+0000","last_active":"2024-03-24T00:13:07.332496+0000","last_peered":"2024-03-24T00:13:07.332496+0000","last_clean":"2024-03-24T00:13:07.332496+0000","last_became_active":"2024-03-24T00:13:01.320929+0000","last_became_peered":"2024-03-24T00:13:01.320929+0000","last_unstale":"2024-03-24T00:13:07.332496+0000","last_undegraded":"2024-03-24T00:13:07.332496+0000","last_fullsized":"2024-03-24T00:13:07.332496+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:13:00.222790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:13:00.222790+0000","last_clean_scrub_stamp":"2024-03-24T00:13:00.222790+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T05:22:05.490303+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,9],"acting":[8,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-24T00:13:03.673933+0000","last_change":"2024-03-24T00:13:02.323413+0000","last_active":"2024-03-24T00:13:03.673933+0000","last_peered":"2024-03-24T00:13:03.673933+0000","last_clean":"2024-03-24T00:13:03.673933+0000","last_became_active":"2024-03-24T00:13:02.322947+0000","last_became_peered":"2024-03-24T00:13:02.322947+0000","last_unstale":"2024-03-24T00:13:03.673933+0000","last_undegraded":"2024-03-24T00:13:03.673933+0000","last_fullsized":"2024-03-24T00:13:03.673933+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:13:01.267198+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:13:01.267198+0000","last_clean_scrub_stamp":"2024-03-24T00:13:01.267198+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T00:16:14.490270+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,9],"acting":[6,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.6","version":"86'7","reported_seq":22,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-24T00:13:07.334476+0000","last_change":"2024-03-24T00:13:01.320380+0000","last_active":"2024-03-24T00:13:07.334476+0000","last_peered":"2024-03-24T00:13:07.334476+0000","last_clean":"2024-03-24T00:13:07.334476+0000","last_became_active":"2024-03-24T00:13:01.319192+0000","last_became_peered":"2024-03-24T00:13:01.319192+0000","last_unstale":"2024-03-24T00:13:07.334476+0000","last_undegraded":"2024-03-24T00:13:07.334476+0000","last_fullsized":"2024-03-24T00:13:07.334476+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:13:00.222790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:13:00.222790+0000","last_clean_scrub_stamp":"2024-03-24T00:13:00.222790+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T02:09:27.085298+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,3],"acting":[9,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-24T00:13:04.348800+0000","last_change":"2024-03-24T00:13:02.328067+0000","last_active":"2024-03-24T00:13:04.348800+0000","last_peered":"2024-03-24T00:13:04.348800+0000","last_clean":"2024-03-24T00:13:04.348800+0000","last_became_active":"2024-03-24T00:13:02.327468+0000","last_became_peered":"2024-03-24T00:13:02.327468+0000","last_unstale":"2024-03-24T00:13:04.348800+0000","last_undegraded":"2024-03-24T00:13:04.348800+0000","last_fullsized":"2024-03-24T00:13:04.348800+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:13:01.267198+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:13:01.267198+0000","last_clean_scrub_stamp":"2024-03-24T00:13:01.267198+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T01:21:58.596941+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,7],"acting":[3,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"86'7","reported_seq":22,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-24T00:13:08.343221+0000","last_change":"2024-03-24T00:13:01.309189+0000","last_active":"2024-03-24T00:13:08.343221+0000","last_peered":"2024-03-24T00:13:08.343221+0000","last_clean":"2024-03-24T00:13:08.343221+0000","last_became_active":"2024-03-24T00:13:01.308871+0000","last_became_peered":"2024-03-24T00:13:01.308871+0000","last_unstale":"2024-03-24T00:13:08.343221+0000","last_undegraded":"2024-03-24T00:13:08.343221+0000","last_fullsized":"2024-03-24T00:13:08.343221+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:13:00.222790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:13:00.222790+0000","last_clean_scrub_stamp":"2024-03-24T00:13:00.222790+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T01:00:32.610036+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,6],"acting":[4,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-24T00:13:03.403801+0000","last_change":"2024-03-24T00:13:02.332238+0000","last_active":"2024-03-24T00:13:03.403801+0000","last_peered":"2024-03-24T00:13:03.403801+0000","last_clean":"2024-03-24T00:13:03.403801+0000","last_became_active":"2024-03-24T00:13:02.331909+0000","last_became_peered":"2024-03-24T00:13:02.331909+0000","last_unstale":"2024-03-24T00:13:03.403801+0000","last_undegraded":"2024-03-24T00:13:03.403801+0000","last_fullsized":"2024-03-24T00:13:03.403801+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:13:01.267198+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:13:01.267198+0000","last_clean_scrub_stamp":"2024-03-24T00:13:01.267198+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T04:41:14.762251+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"86'2","reported_seq":17,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-24T00:13:05.402584+0000","last_change":"2024-03-24T00:13:01.306883+0000","last_active":"2024-03-24T00:13:05.402584+0000","last_peered":"2024-03-24T00:13:05.402584+0000","last_clean":"2024-03-24T00:13:05.402584+0000","last_became_active":"2024-03-24T00:13:01.306629+0000","last_became_peered":"2024-03-24T00:13:01.306629+0000","last_unstale":"2024-03-24T00:13:05.402584+0000","last_undegraded":"2024-03-24T00:13:05.402584+0000","last_fullsized":"2024-03-24T00:13:05.402584+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:13:00.222790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:13:00.222790+0000","last_clean_scrub_stamp":"2024-03-24T00:13:00.222790+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T08:11:52.398783+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-24T00:13:03.597200+0000","last_change":"2024-03-24T00:13:02.308421+0000","last_active":"2024-03-24T00:13:03.597200+0000","last_peered":"2024-03-24T00:13:03.597200+0000","last_clean":"2024-03-24T00:13:03.597200+0000","last_became_active":"2024-03-24T00:13:02.308081+0000","last_became_peered":"2024-03-24T00:13:02.308081+0000","last_unstale":"2024-03-24T00:13:03.597200+0000","last_undegraded":"2024-03-24T00:13:03.597200+0000","last_fullsized":"2024-03-24T00:13:03.597200+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:13:01.267198+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:13:01.267198+0000","last_clean_scrub_stamp":"2024-03-24T00:13:01.267198+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T06:50:42.356293+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,10],"acting":[8,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.9","version":"86'5","reported_seq":20,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-24T00:13:07.331289+0000","last_change":"2024-03-24T00:13:01.324439+0000","last_active":"2024-03-24T00:13:07.331289+0000","last_peered":"2024-03-24T00:13:07.331289+0000","last_clean":"2024-03-24T00:13:07.331289+0000","last_became_active":"2024-03-24T00:13:01.324037+0000","last_became_peered":"2024-03-24T00:13:01.324037+0000","last_unstale":"2024-03-24T00:13:07.331289+0000","last_undegraded":"2024-03-24T00:13:07.331289+0000","last_fullsized":"2024-03-24T00:13:07.331289+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:13:00.222790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:13:00.222790+0000","last_clean_scrub_stamp":"2024-03-24T00:13:00.222790+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T01:37:56.767931+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,10],"acting":[3,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-24T00:13:03.860093+0000","last_change":"2024-03-24T00:13:02.313740+0000","last_active":"2024-03-24T00:13:03.860093+0000","last_peered":"2024-03-24T00:13:03.860093+0000","last_clean":"2024-03-24T00:13:03.860093+0000","last_became_active":"2024-03-24T00:13:02.313401+0000","last_became_peered":"2024-03-24T00:13:02.313401+0000","last_unstale":"2024-03-24T00:13:03.860093+0000","last_undegraded":"2024-03-24T00:13:03.860093+0000","last_fullsized":"2024-03-24T00:13:03.860093+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:13:01.267198+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:13:01.267198+0000","last_clean_scrub_stamp":"2024-03-24T00:13:01.267198+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T04:16:52.569135+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.a","version":"86'3","reported_seq":18,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-24T00:13:07.333350+0000","last_change":"2024-03-24T00:13:01.316870+0000","last_active":"2024-03-24T00:13:07.333350+0000","last_peered":"2024-03-24T00:13:07.333350+0000","last_clean":"2024-03-24T00:13:07.333350+0000","last_became_active":"2024-03-24T00:13:01.316512+0000","last_became_peered":"2024-03-24T00:13:01.316512+0000","last_unstale":"2024-03-24T00:13:07.333350+0000","last_undegraded":"2024-03-24T00:13:07.333350+0000","last_fullsized":"2024-03-24T00:13:07.333350+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:13:00.222790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:13:00.222790+0000","last_clean_scrub_stamp":"2024-03-24T00:13:00.222790+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T08:28:02.418185+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,8],"acting":[2,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-24T00:13:04.347976+0000","last_change":"2024-03-24T00:13:02.337413+0000","last_active":"2024-03-24T00:13:04.347976+0000","last_peered":"2024-03-24T00:13:04.347976+0000","last_clean":"2024-03-24T00:13:04.347976+0000","last_became_active":"2024-03-24T00:13:02.336910+0000","last_became_peered":"2024-03-24T00:13:02.336910+0000","last_unstale":"2024-03-24T00:13:04.347976+0000","last_undegraded":"2024-03-24T00:13:04.347976+0000","last_fullsized":"2024-03-24T00:13:04.347976+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:13:01.267198+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:13:01.267198+0000","last_clean_scrub_stamp":"2024-03-24T00:13:01.267198+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T10:51:06.549979+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,10],"acting":[3,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.b","version":"86'4","reported_seq":20,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-24T00:13:05.400768+0000","last_change":"2024-03-24T00:13:01.300201+0000","last_active":"2024-03-24T00:13:05.400768+0000","last_peered":"2024-03-24T00:13:05.400768+0000","last_clean":"2024-03-24T00:13:05.400768+0000","last_became_active":"2024-03-24T00:13:01.299750+0000","last_became_peered":"2024-03-24T00:13:01.299750+0000","last_unstale":"2024-03-24T00:13:05.400768+0000","last_undegraded":"2024-03-24T00:13:05.400768+0000","last_fullsized":"2024-03-24T00:13:05.400768+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:13:00.222790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:13:00.222790+0000","last_clean_scrub_stamp":"2024-03-24T00:13:00.222790+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T01:37:32.768926+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-24T00:13:03.840972+0000","last_change":"2024-03-24T00:13:02.331732+0000","last_active":"2024-03-24T00:13:03.840972+0000","last_peered":"2024-03-24T00:13:03.840972+0000","last_clean":"2024-03-24T00:13:03.840972+0000","last_became_active":"2024-03-24T00:13:02.331365+0000","last_became_peered":"2024-03-24T00:13:02.331365+0000","last_unstale":"2024-03-24T00:13:03.840972+0000","last_undegraded":"2024-03-24T00:13:03.840972+0000","last_fullsized":"2024-03-24T00:13:03.840972+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:13:01.267198+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:13:01.267198+0000","last_clean_scrub_stamp":"2024-03-24T00:13:01.267198+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:27:53.598158+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.c","version":"86'6","reported_seq":24,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-24T00:13:09.346129+0000","last_change":"2024-03-24T00:13:01.308725+0000","last_active":"2024-03-24T00:13:09.346129+0000","last_peered":"2024-03-24T00:13:09.346129+0000","last_clean":"2024-03-24T00:13:09.346129+0000","last_became_active":"2024-03-24T00:13:01.308324+0000","last_became_peered":"2024-03-24T00:13:01.308324+0000","last_unstale":"2024-03-24T00:13:09.346129+0000","last_undegraded":"2024-03-24T00:13:09.346129+0000","last_fullsized":"2024-03-24T00:13:09.346129+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:13:00.222790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:13:00.222790+0000","last_clean_scrub_stamp":"2024-03-24T00:13:00.222790+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T07:43:03.691683+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1561,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":3,"num_read_kb":2,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,11,0],"acting":[2,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-24T00:13:03.841100+0000","last_change":"2024-03-24T00:13:02.305749+0000","last_active":"2024-03-24T00:13:03.841100+0000","last_peered":"2024-03-24T00:13:03.841100+0000","last_clean":"2024-03-24T00:13:03.841100+0000","last_became_active":"2024-03-24T00:13:02.305413+0000","last_became_peered":"2024-03-24T00:13:02.305413+0000","last_unstale":"2024-03-24T00:13:03.841100+0000","last_undegraded":"2024-03-24T00:13:03.841100+0000","last_fullsized":"2024-03-24T00:13:03.841100+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:13:01.267198+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:13:01.267198+0000","last_clean_scrub_stamp":"2024-03-24T00:13:01.267198+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T00:54:56.437725+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,11],"acting":[7,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.d","version":"86'2","reported_seq":18,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-24T00:13:06.279022+0000","last_change":"2024-03-24T00:13:01.304883+0000","last_active":"2024-03-24T00:13:06.279022+0000","last_peered":"2024-03-24T00:13:06.279022+0000","last_clean":"2024-03-24T00:13:06.279022+0000","last_became_active":"2024-03-24T00:13:01.304459+0000","last_became_peered":"2024-03-24T00:13:01.304459+0000","last_unstale":"2024-03-24T00:13:06.279022+0000","last_undegraded":"2024-03-24T00:13:06.279022+0000","last_fullsized":"2024-03-24T00:13:06.279022+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:13:00.222790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:13:00.222790+0000","last_clean_scrub_stamp":"2024-03-24T00:13:00.222790+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T06:14:22.473571+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10,3],"acting":[5,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-24T00:13:03.840920+0000","last_change":"2024-03-24T00:13:02.331923+0000","last_active":"2024-03-24T00:13:03.840920+0000","last_peered":"2024-03-24T00:13:03.840920+0000","last_clean":"2024-03-24T00:13:03.840920+0000","last_became_active":"2024-03-24T00:13:02.331504+0000","last_became_peered":"2024-03-24T00:13:02.331504+0000","last_unstale":"2024-03-24T00:13:03.840920+0000","last_undegraded":"2024-03-24T00:13:03.840920+0000","last_fullsized":"2024-03-24T00:13:03.840920+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:13:01.267198+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:13:01.267198+0000","last_clean_scrub_stamp":"2024-03-24T00:13:01.267198+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T03:27:17.160248+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,5],"acting":[7,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"85'2","reported_seq":17,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-24T00:13:03.673400+0000","last_change":"2024-03-24T00:13:01.320223+0000","last_active":"2024-03-24T00:13:03.673400+0000","last_peered":"2024-03-24T00:13:03.673400+0000","last_clean":"2024-03-24T00:13:03.673400+0000","last_became_active":"2024-03-24T00:13:01.319342+0000","last_became_peered":"2024-03-24T00:13:01.319342+0000","last_unstale":"2024-03-24T00:13:03.673400+0000","last_undegraded":"2024-03-24T00:13:03.673400+0000","last_fullsized":"2024-03-24T00:13:03.673400+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:13:00.222790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:13:00.222790+0000","last_clean_scrub_stamp":"2024-03-24T00:13:00.222790+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T02:24:08.018521+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-24T00:13:03.841120+0000","last_change":"2024-03-24T00:13:02.307719+0000","last_active":"2024-03-24T00:13:03.841120+0000","last_peered":"2024-03-24T00:13:03.841120+0000","last_clean":"2024-03-24T00:13:03.841120+0000","last_became_active":"2024-03-24T00:13:02.307399+0000","last_became_peered":"2024-03-24T00:13:02.307399+0000","last_unstale":"2024-03-24T00:13:03.841120+0000","last_undegraded":"2024-03-24T00:13:03.841120+0000","last_fullsized":"2024-03-24T00:13:03.841120+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:13:01.267198+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:13:01.267198+0000","last_clean_scrub_stamp":"2024-03-24T00:13:01.267198+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T09:38:13.077843+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,8],"acting":[7,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-24T00:13:03.860199+0000","last_change":"2024-03-24T00:13:02.307469+0000","last_active":"2024-03-24T00:13:03.860199+0000","last_peered":"2024-03-24T00:13:03.860199+0000","last_clean":"2024-03-24T00:13:03.860199+0000","last_became_active":"2024-03-24T00:13:02.306873+0000","last_became_peered":"2024-03-24T00:13:02.306873+0000","last_unstale":"2024-03-24T00:13:03.860199+0000","last_undegraded":"2024-03-24T00:13:03.860199+0000","last_fullsized":"2024-03-24T00:13:03.860199+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:13:01.267198+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:13:01.267198+0000","last_clean_scrub_stamp":"2024-03-24T00:13:01.267198+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:19:51.163015+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,5],"acting":[11,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-24T00:13:03.290104+0000","last_change":"2024-03-24T00:13:02.307472+0000","last_active":"2024-03-24T00:13:03.290104+0000","last_peered":"2024-03-24T00:13:03.290104+0000","last_clean":"2024-03-24T00:13:03.290104+0000","last_became_active":"2024-03-24T00:13:02.307099+0000","last_became_peered":"2024-03-24T00:13:02.307099+0000","last_unstale":"2024-03-24T00:13:03.290104+0000","last_undegraded":"2024-03-24T00:13:03.290104+0000","last_fullsized":"2024-03-24T00:13:03.290104+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:13:01.267198+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:13:01.267198+0000","last_clean_scrub_stamp":"2024-03-24T00:13:01.267198+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T08:49:06.461235+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,4],"acting":[2,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-24T00:13:03.289812+0000","last_change":"2024-03-24T00:13:02.331133+0000","last_active":"2024-03-24T00:13:03.289812+0000","last_peered":"2024-03-24T00:13:03.289812+0000","last_clean":"2024-03-24T00:13:03.289812+0000","last_became_active":"2024-03-24T00:13:02.330780+0000","last_became_peered":"2024-03-24T00:13:02.330780+0000","last_unstale":"2024-03-24T00:13:03.289812+0000","last_undegraded":"2024-03-24T00:13:03.289812+0000","last_fullsized":"2024-03-24T00:13:03.289812+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:13:01.267198+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:13:01.267198+0000","last_clean_scrub_stamp":"2024-03-24T00:13:01.267198+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T10:46:07.933131+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-24T00:13:03.402944+0000","last_change":"2024-03-24T00:13:02.335406+0000","last_active":"2024-03-24T00:13:03.402944+0000","last_peered":"2024-03-24T00:13:03.402944+0000","last_clean":"2024-03-24T00:13:03.402944+0000","last_became_active":"2024-03-24T00:13:02.334994+0000","last_became_peered":"2024-03-24T00:13:02.334994+0000","last_unstale":"2024-03-24T00:13:03.402944+0000","last_undegraded":"2024-03-24T00:13:03.402944+0000","last_fullsized":"2024-03-24T00:13:03.402944+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:13:01.267198+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:13:01.267198+0000","last_clean_scrub_stamp":"2024-03-24T00:13:01.267198+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T03:24:59.547395+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,4],"acting":[0,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-24T00:13:03.674018+0000","last_change":"2024-03-24T00:13:02.332420+0000","last_active":"2024-03-24T00:13:03.674018+0000","last_peered":"2024-03-24T00:13:03.674018+0000","last_clean":"2024-03-24T00:13:03.674018+0000","last_became_active":"2024-03-24T00:13:02.331496+0000","last_became_peered":"2024-03-24T00:13:02.331496+0000","last_unstale":"2024-03-24T00:13:03.674018+0000","last_undegraded":"2024-03-24T00:13:03.674018+0000","last_fullsized":"2024-03-24T00:13:03.674018+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:13:01.267198+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:13:01.267198+0000","last_clean_scrub_stamp":"2024-03-24T00:13:01.267198+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T03:15:06.007947+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-24T00:13:03.860354+0000","last_change":"2024-03-24T00:13:02.309077+0000","last_active":"2024-03-24T00:13:03.860354+0000","last_peered":"2024-03-24T00:13:03.860354+0000","last_clean":"2024-03-24T00:13:03.860354+0000","last_became_active":"2024-03-24T00:13:02.308752+0000","last_became_peered":"2024-03-24T00:13:02.308752+0000","last_unstale":"2024-03-24T00:13:03.860354+0000","last_undegraded":"2024-03-24T00:13:03.860354+0000","last_fullsized":"2024-03-24T00:13:03.860354+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:13:01.267198+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:13:01.267198+0000","last_clean_scrub_stamp":"2024-03-24T00:13:01.267198+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T06:49:19.657988+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,5],"acting":[11,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-24T00:13:03.290284+0000","last_change":"2024-03-24T00:13:02.304962+0000","last_active":"2024-03-24T00:13:03.290284+0000","last_peered":"2024-03-24T00:13:03.290284+0000","last_clean":"2024-03-24T00:13:03.290284+0000","last_became_active":"2024-03-24T00:13:02.304580+0000","last_became_peered":"2024-03-24T00:13:02.304580+0000","last_unstale":"2024-03-24T00:13:03.290284+0000","last_undegraded":"2024-03-24T00:13:03.290284+0000","last_fullsized":"2024-03-24T00:13:03.290284+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:13:01.267198+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:13:01.267198+0000","last_clean_scrub_stamp":"2024-03-24T00:13:01.267198+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:45:26.779169+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8,1],"acting":[2,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-24T00:13:04.348365+0000","last_change":"2024-03-24T00:13:02.321908+0000","last_active":"2024-03-24T00:13:04.348365+0000","last_peered":"2024-03-24T00:13:04.348365+0000","last_clean":"2024-03-24T00:13:04.348365+0000","last_became_active":"2024-03-24T00:13:02.321495+0000","last_became_peered":"2024-03-24T00:13:02.321495+0000","last_unstale":"2024-03-24T00:13:04.348365+0000","last_undegraded":"2024-03-24T00:13:04.348365+0000","last_fullsized":"2024-03-24T00:13:04.348365+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:13:01.267198+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:13:01.267198+0000","last_clean_scrub_stamp":"2024-03-24T00:13:01.267198+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T08:43:29.261314+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-24T00:13:03.429452+0000","last_change":"2024-03-24T00:13:02.331966+0000","last_active":"2024-03-24T00:13:03.429452+0000","last_peered":"2024-03-24T00:13:03.429452+0000","last_clean":"2024-03-24T00:13:03.429452+0000","last_became_active":"2024-03-24T00:13:02.331633+0000","last_became_peered":"2024-03-24T00:13:02.331633+0000","last_unstale":"2024-03-24T00:13:03.429452+0000","last_undegraded":"2024-03-24T00:13:03.429452+0000","last_fullsized":"2024-03-24T00:13:03.429452+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:13:01.267198+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:13:01.267198+0000","last_clean_scrub_stamp":"2024-03-24T00:13:01.267198+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T05:04:01.574351+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,3],"acting":[5,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-24T00:13:04.348470+0000","last_change":"2024-03-24T00:13:02.330344+0000","last_active":"2024-03-24T00:13:04.348470+0000","last_peered":"2024-03-24T00:13:04.348470+0000","last_clean":"2024-03-24T00:13:04.348470+0000","last_became_active":"2024-03-24T00:13:02.328218+0000","last_became_peered":"2024-03-24T00:13:02.328218+0000","last_unstale":"2024-03-24T00:13:04.348470+0000","last_undegraded":"2024-03-24T00:13:04.348470+0000","last_fullsized":"2024-03-24T00:13:04.348470+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:13:01.267198+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:13:01.267198+0000","last_clean_scrub_stamp":"2024-03-24T00:13:01.267198+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T08:54:36.392397+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,11],"acting":[3,2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-24T00:13:04.366065+0000","last_change":"2024-03-24T00:13:02.330956+0000","last_active":"2024-03-24T00:13:04.366065+0000","last_peered":"2024-03-24T00:13:04.366065+0000","last_clean":"2024-03-24T00:13:04.366065+0000","last_became_active":"2024-03-24T00:13:02.330642+0000","last_became_peered":"2024-03-24T00:13:02.330642+0000","last_unstale":"2024-03-24T00:13:04.366065+0000","last_undegraded":"2024-03-24T00:13:04.366065+0000","last_fullsized":"2024-03-24T00:13:04.366065+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:13:01.267198+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:13:01.267198+0000","last_clean_scrub_stamp":"2024-03-24T00:13:01.267198+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:41:11.326722+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-24T00:13:03.673782+0000","last_change":"2024-03-24T00:13:02.333026+0000","last_active":"2024-03-24T00:13:03.673782+0000","last_peered":"2024-03-24T00:13:03.673782+0000","last_clean":"2024-03-24T00:13:03.673782+0000","last_became_active":"2024-03-24T00:13:02.331232+0000","last_became_peered":"2024-03-24T00:13:02.331232+0000","last_unstale":"2024-03-24T00:13:03.673782+0000","last_undegraded":"2024-03-24T00:13:03.673782+0000","last_fullsized":"2024-03-24T00:13:03.673782+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:13:01.267198+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:13:01.267198+0000","last_clean_scrub_stamp":"2024-03-24T00:13:01.267198+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T02:14:18.504012+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-24T00:13:03.860196+0000","last_change":"2024-03-24T00:13:02.309023+0000","last_active":"2024-03-24T00:13:03.860196+0000","last_peered":"2024-03-24T00:13:03.860196+0000","last_clean":"2024-03-24T00:13:03.860196+0000","last_became_active":"2024-03-24T00:13:02.308236+0000","last_became_peered":"2024-03-24T00:13:02.308236+0000","last_unstale":"2024-03-24T00:13:03.860196+0000","last_undegraded":"2024-03-24T00:13:03.860196+0000","last_fullsized":"2024-03-24T00:13:03.860196+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:13:01.267198+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:13:01.267198+0000","last_clean_scrub_stamp":"2024-03-24T00:13:01.267198+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T03:30:37.427095+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,8],"acting":[11,4,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1,"ondisk_log_size":1,"up":192,"acting":192,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":442368,"data_stored":4722944,"data_compressed":284592,"data_compressed_allocated":360448,"data_compressed_original":4669440,"omap_allocated":0,"internal_metadata":0},"log_size":137,"ondisk_log_size":137,"up":3,"acting":3,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":26336,"num_objects":82,"num_object_clones":0,"num_object_copies":246,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":82,"num_whiteouts":0,"num_read":29,"num_read_kb":23,"num_write":182,"num_write_kb":73,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":55,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":385024,"data_stored":83236,"data_compressed":1719,"data_compressed_allocated":4096,"data_compressed_original":8192,"omap_allocated":0,"internal_metadata":0},"log_size":157,"ondisk_log_size":157,"up":96,"acting":96,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":81,"seq":347892350990,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27928,"kb_used_data":912,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717224,"statfs":{"total":95995035648,"available":95966437376,"internally_reserved":0,"allocated":933888,"data_stored":443734,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":27654596},"hb_peers":[0,1,2,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":73,"seq":313532612627,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28016,"kb_used_data":1000,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717136,"statfs":{"total":95995035648,"available":95966347264,"internally_reserved":0,"allocated":1024000,"data_stored":1035833,"data_compressed":36189,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":1599,"internal_metadata":27654593},"hb_peers":[0,1,2,3,4,5,6,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":68,"seq":292057776152,"num_pgs":17,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32096,"kb_used_data":920,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713056,"statfs":{"total":95995035648,"available":95962169344,"internally_reserved":0,"allocated":942080,"data_stored":442969,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":31914432},"hb_peers":[0,1,2,3,4,5,6,7,8,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":8,"up_from":60,"seq":257698037791,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32096,"kb_used_data":920,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713056,"statfs":{"total":95995035648,"available":95962169344,"internally_reserved":0,"allocated":942080,"data_stored":442889,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":7,"up_from":52,"seq":223338299429,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27988,"kb_used_data":972,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717164,"statfs":{"total":95995035648,"available":95966375936,"internally_reserved":0,"allocated":995328,"data_stored":1034983,"data_compressed":36189,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":47,"seq":201863462955,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27920,"kb_used_data":904,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717232,"statfs":{"total":95995035648,"available":95966445568,"internally_reserved":0,"allocated":925696,"data_stored":442241,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574918,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27932,"kb_used_data":912,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717220,"statfs":{"total":95995035648,"available":95966433280,"internally_reserved":0,"allocated":933888,"data_stored":443763,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738445,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27956,"kb_used_data":940,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717196,"statfs":{"total":95995035648,"available":95966408704,"internally_reserved":0,"allocated":962560,"data_stored":444720,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411394,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27976,"kb_used_data":960,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717176,"statfs":{"total":95995035648,"available":95966388224,"internally_reserved":0,"allocated":983040,"data_stored":465800,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":26,"seq":111669149756,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32092,"kb_used_data":916,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713060,"statfs":{"total":95995035648,"available":95962173440,"internally_reserved":0,"allocated":937984,"data_stored":442435,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":4,"up_from":34,"seq":146028888118,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32096,"kb_used_data":920,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713056,"statfs":{"total":95995035648,"available":95962169344,"internally_reserved":0,"allocated":942080,"data_stored":442851,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":39,"seq":167503724592,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27992,"kb_used_data":976,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717160,"statfs":{"total":95995035648,"available":95966371840,"internally_reserved":0,"allocated":999424,"data_stored":1033995,"data_compressed":36189,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34549,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34549,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34549,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":36189,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":36189,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":36189,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":36189,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":36189,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":2569,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":81920,"data_stored":46882,"data_compressed":1719,"data_compressed_allocated":4096,"data_compressed_original":8192,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":700,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":2238,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":112,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":2855,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":738,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":908,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":22631,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1583,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-24T00:13:11.803 DEBUG:teuthology.orchestra.run.smithi118:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-24T00:13:12.056 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:12.055+0000 7f5dac200640 1 -- 172.21.15.118:0/2773906056 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f5da41038e0 msgr2=0x7f5da4103d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:13:12.057 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:12.055+0000 7f5dac200640 1 --2- 172.21.15.118:0/2773906056 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f5da41038e0 0x7f5da4103d00 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f5d98009900 tx=0x7f5d9802bc30 comp rx=0 tx=0).stop 2024-03-24T00:13:12.057 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:12.056+0000 7f5dac200640 1 -- 172.21.15.118:0/2773906056 shutdown_connections 2024-03-24T00:13:12.057 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:12.056+0000 7f5dac200640 1 --2- 172.21.15.118:0/2773906056 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5da4105490 0x7f5da410bd20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:13:12.057 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:12.056+0000 7f5dac200640 1 --2- 172.21.15.118:0/2773906056 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f5da4104ad0 0x7f5da4104f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:13:12.057 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:12.056+0000 7f5dac200640 1 --2- 172.21.15.118:0/2773906056 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f5da41038e0 0x7f5da4103d00 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:13:12.057 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:12.056+0000 7f5dac200640 1 -- 172.21.15.118:0/2773906056 >> 172.21.15.118:0/2773906056 conn(0x7f5da40fee90 msgr2=0x7f5da41012b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:13:12.057 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:12.056+0000 7f5dac200640 1 -- 172.21.15.118:0/2773906056 shutdown_connections 2024-03-24T00:13:12.057 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:12.056+0000 7f5dac200640 1 -- 172.21.15.118:0/2773906056 wait complete. 2024-03-24T00:13:12.058 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:12.057+0000 7f5dac200640 1 Processor -- start 2024-03-24T00:13:12.058 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:12.057+0000 7f5dac200640 1 -- start start 2024-03-24T00:13:12.058 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:12.057+0000 7f5dac200640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f5da41038e0 0x7f5da4199c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:13:12.058 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:12.057+0000 7f5dac200640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5da4104ad0 0x7f5da419a180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:13:12.059 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:12.057+0000 7f5dac200640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f5da4105490 0x7f5da41a11b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:13:12.059 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:12.057+0000 7f5dac200640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5da410f580 con 0x7f5da4105490 2024-03-24T00:13:12.059 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:12.057+0000 7f5dac200640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5da410f400 con 0x7f5da4104ad0 2024-03-24T00:13:12.059 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:12.057+0000 7f5dac200640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5da410f700 con 0x7f5da41038e0 2024-03-24T00:13:12.059 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:12.058+0000 7f5da9800640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f5da41038e0 0x7f5da4199c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:13:12.059 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:12.058+0000 7f5da9800640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f5da41038e0 0x7f5da4199c40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.118:47762/0 (socket says 172.21.15.118:47762) 2024-03-24T00:13:12.059 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:12.058+0000 7f5da9800640 1 -- 172.21.15.118:0/1918554656 learned_addr learned my addr 172.21.15.118:0/1918554656 (peer_addr_for_me v2:172.21.15.118:0/0) 2024-03-24T00:13:12.059 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:12.058+0000 7f5da8e00640 1 --2- 172.21.15.118:0/1918554656 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5da4104ad0 0x7f5da419a180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:13:12.059 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:12.058+0000 7f5daa200640 1 --2- 172.21.15.118:0/1918554656 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f5da4105490 0x7f5da41a11b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:13:12.060 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:12.058+0000 7f5da9800640 1 -- 172.21.15.118:0/1918554656 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5da4104ad0 msgr2=0x7f5da419a180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:13:12.060 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:12.058+0000 7f5da9800640 1 --2- 172.21.15.118:0/1918554656 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5da4104ad0 0x7f5da419a180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:13:12.060 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:12.058+0000 7f5da9800640 1 -- 172.21.15.118:0/1918554656 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f5da4105490 msgr2=0x7f5da41a11b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:13:12.060 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:12.058+0000 7f5da9800640 1 --2- 172.21.15.118:0/1918554656 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f5da4105490 0x7f5da41a11b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:13:12.060 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:12.058+0000 7f5da9800640 1 -- 172.21.15.118:0/1918554656 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5da41a1850 con 0x7f5da41038e0 2024-03-24T00:13:12.060 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:12.058+0000 7f5da8e00640 1 --2- 172.21.15.118:0/1918554656 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5da4104ad0 0x7f5da419a180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T00:13:12.060 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:12.058+0000 7f5da9800640 1 --2- 172.21.15.118:0/1918554656 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f5da41038e0 0x7f5da4199c40 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f5d98000c00 tx=0x7f5d9802c650 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:13:12.061 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:12.059+0000 7f5da2000640 1 -- 172.21.15.118:0/1918554656 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5d98043020 con 0x7f5da41038e0 2024-03-24T00:13:12.061 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:12.059+0000 7f5da2000640 1 -- 172.21.15.118:0/1918554656 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f5d980054e0 con 0x7f5da41038e0 2024-03-24T00:13:12.061 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:12.059+0000 7f5dac200640 1 -- 172.21.15.118:0/1918554656 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5da41a1ae0 con 0x7f5da41038e0 2024-03-24T00:13:12.061 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:12.059+0000 7f5da2000640 1 -- 172.21.15.118:0/1918554656 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5d98003bd0 con 0x7f5da41038e0 2024-03-24T00:13:12.061 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:12.059+0000 7f5dac200640 1 -- 172.21.15.118:0/1918554656 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5da41a2080 con 0x7f5da41038e0 2024-03-24T00:13:12.061 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:12.060+0000 7f5dac200640 1 -- 172.21.15.118:0/1918554656 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5da40688e0 con 0x7f5da41038e0 2024-03-24T00:13:12.062 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:12.062+0000 7f5da2000640 1 -- 172.21.15.118:0/1918554656 <== mon.1 v2:172.21.15.193:3300/0 4 ==== mgrmap(e 17) v1 ==== 137342+0+0 (secure 0 0 0) 0x7f5d98046020 con 0x7f5da41038e0 2024-03-24T00:13:12.064 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:12.063+0000 7f5da2000640 1 --2- 172.21.15.118:0/1918554656 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7f5d8c0a2910 0x7f5d8c0a4dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:13:12.064 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:12.063+0000 7f5da2000640 1 -- 172.21.15.118:0/1918554656 <== mon.1 v2:172.21.15.193:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 7979+0+0 (secure 0 0 0) 0x7f5d980f0a40 con 0x7f5da41038e0 2024-03-24T00:13:12.064 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:12.063+0000 7f5da8e00640 1 --2- 172.21.15.118:0/1918554656 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7f5d8c0a2910 0x7f5d8c0a4dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:13:12.064 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:12.064+0000 7f5da8e00640 1 --2- 172.21.15.118:0/1918554656 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7f5d8c0a2910 0x7f5d8c0a4dd0 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7f5d90009fc0 tx=0x7f5d90005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:13:12.067 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:12.067+0000 7f5da2000640 1 -- 172.21.15.118:0/1918554656 <== mon.1 v2:172.21.15.193:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f5d9804d020 con 0x7f5da41038e0 2024-03-24T00:13:12.264 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:12.263+0000 7f5dac200640 1 -- 172.21.15.118:0/1918554656 --> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f5da4061ba0 con 0x7f5d8c0a2910 2024-03-24T00:13:12.281 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:12.280+0000 7f5da2000640 1 -- 172.21.15.118:0/1918554656 <== mgr.14152 v2:172.21.15.118:6800/2112912943 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+258968 (secure 0 0 0) 0x7f5da4061ba0 con 0x7f5d8c0a2910 2024-03-24T00:13:12.284 INFO:teuthology.orchestra.run.smithi118.stdout: 2024-03-24T00:13:12.285 INFO:teuthology.orchestra.run.smithi118.stderr:dumped all 2024-03-24T00:13:12.287 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:12.285+0000 7f5dac200640 1 -- 172.21.15.118:0/1918554656 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7f5d8c0a2910 msgr2=0x7f5d8c0a4dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:13:12.287 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:12.285+0000 7f5dac200640 1 --2- 172.21.15.118:0/1918554656 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7f5d8c0a2910 0x7f5d8c0a4dd0 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7f5d90009fc0 tx=0x7f5d90005f90 comp rx=0 tx=0).stop 2024-03-24T00:13:12.287 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:12.285+0000 7f5dac200640 1 -- 172.21.15.118:0/1918554656 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f5da41038e0 msgr2=0x7f5da4199c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:13:12.287 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:12.285+0000 7f5dac200640 1 --2- 172.21.15.118:0/1918554656 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f5da41038e0 0x7f5da4199c40 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f5d98000c00 tx=0x7f5d9802c650 comp rx=0 tx=0).stop 2024-03-24T00:13:12.287 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:12.285+0000 7f5dac200640 1 -- 172.21.15.118:0/1918554656 shutdown_connections 2024-03-24T00:13:12.287 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:12.285+0000 7f5dac200640 1 --2- 172.21.15.118:0/1918554656 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7f5d8c0a2910 0x7f5d8c0a4dd0 unknown :-1 s=CLOSED pgs=176 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:13:12.287 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:12.285+0000 7f5dac200640 1 --2- 172.21.15.118:0/1918554656 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f5da4105490 0x7f5da41a11b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:13:12.288 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:12.285+0000 7f5dac200640 1 --2- 172.21.15.118:0/1918554656 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5da4104ad0 0x7f5da419a180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:13:12.288 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:12.285+0000 7f5dac200640 1 --2- 172.21.15.118:0/1918554656 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f5da41038e0 0x7f5da4199c40 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:13:12.288 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:12.285+0000 7f5dac200640 1 -- 172.21.15.118:0/1918554656 >> 172.21.15.118:0/1918554656 conn(0x7f5da40fee90 msgr2=0x7f5da4107dd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:13:12.288 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:12.285+0000 7f5dac200640 1 -- 172.21.15.118:0/1918554656 shutdown_connections 2024-03-24T00:13:12.288 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:12.285+0000 7f5dac200640 1 -- 172.21.15.118:0/1918554656 wait complete. 2024-03-24T00:13:12.303 INFO:teuthology.orchestra.run.smithi118.stdout:{"pg_ready":true,"pg_map":{"version":319,"stamp":"2024-03-24T00:13:11.604553+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":616821,"num_objects":85,"num_object_clones":0,"num_object_copies":255,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":85,"num_whiteouts":0,"num_read":155,"num_read_kb":132,"num_write":364,"num_write_kb":3012,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":55,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":295,"ondisk_log_size":295,"up":291,"acting":291,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":291,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":352088,"kb_used_data":11252,"kb_used_omap":18,"kb_used_meta":340717,"kb_avail":1124589736,"statfs":{"total":1151940427776,"available":1151579889664,"internally_reserved":0,"allocated":11522048,"data_stored":7116213,"data_compressed":108567,"data_compressed_allocated":135168,"data_compressed_original":1769472,"omap_allocated":19142,"internal_metadata":348894522},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":11,"apply_latency_ms":11,"commit_latency_ns":11000000,"apply_latency_ns":11000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":26453,"num_objects":83,"num_object_clones":0,"num_object_copies":249,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":83,"num_whiteouts":0,"num_read":29,"num_read_kb":23,"num_write":183,"num_write_kb":74,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":55,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"9.319843"},"pg_stats":[{"pgid":"3.3f","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-24T00:13:03.860443+0000","last_change":"2024-03-24T00:13:02.325815+0000","last_active":"2024-03-24T00:13:03.860443+0000","last_peered":"2024-03-24T00:13:03.860443+0000","last_clean":"2024-03-24T00:13:03.860443+0000","last_became_active":"2024-03-24T00:13:02.325450+0000","last_became_peered":"2024-03-24T00:13:02.325450+0000","last_unstale":"2024-03-24T00:13:03.860443+0000","last_undegraded":"2024-03-24T00:13:03.860443+0000","last_fullsized":"2024-03-24T00:13:03.860443+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:13:01.267198+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:13:01.267198+0000","last_clean_scrub_stamp":"2024-03-24T00:13:01.267198+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T02:18:21.994322+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.3e","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-24T00:13:03.403477+0000","last_change":"2024-03-24T00:13:02.324015+0000","last_active":"2024-03-24T00:13:03.403477+0000","last_peered":"2024-03-24T00:13:03.403477+0000","last_clean":"2024-03-24T00:13:03.403477+0000","last_became_active":"2024-03-24T00:13:02.323668+0000","last_became_peered":"2024-03-24T00:13:02.323668+0000","last_unstale":"2024-03-24T00:13:03.403477+0000","last_undegraded":"2024-03-24T00:13:03.403477+0000","last_fullsized":"2024-03-24T00:13:03.403477+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:13:01.267198+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:13:01.267198+0000","last_clean_scrub_stamp":"2024-03-24T00:13:01.267198+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:36:19.442266+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11,10],"acting":[0,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3d","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-24T00:13:03.403525+0000","last_change":"2024-03-24T00:13:02.328911+0000","last_active":"2024-03-24T00:13:03.403525+0000","last_peered":"2024-03-24T00:13:03.403525+0000","last_clean":"2024-03-24T00:13:03.403525+0000","last_became_active":"2024-03-24T00:13:02.324333+0000","last_became_peered":"2024-03-24T00:13:02.324333+0000","last_unstale":"2024-03-24T00:13:03.403525+0000","last_undegraded":"2024-03-24T00:13:03.403525+0000","last_fullsized":"2024-03-24T00:13:03.403525+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:13:01.267198+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:13:01.267198+0000","last_clean_scrub_stamp":"2024-03-24T00:13:01.267198+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T02:49:02.318097+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-24T00:13:03.290642+0000","last_change":"2024-03-24T00:13:02.309057+0000","last_active":"2024-03-24T00:13:03.290642+0000","last_peered":"2024-03-24T00:13:03.290642+0000","last_clean":"2024-03-24T00:13:03.290642+0000","last_became_active":"2024-03-24T00:13:02.308403+0000","last_became_peered":"2024-03-24T00:13:02.308403+0000","last_unstale":"2024-03-24T00:13:03.290642+0000","last_undegraded":"2024-03-24T00:13:03.290642+0000","last_fullsized":"2024-03-24T00:13:03.290642+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:13:01.267198+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:13:01.267198+0000","last_clean_scrub_stamp":"2024-03-24T00:13:01.267198+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T07:54:44.310833+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,8,2],"acting":[4,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.3b","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-24T00:13:03.597254+0000","last_change":"2024-03-24T00:13:02.330651+0000","last_active":"2024-03-24T00:13:03.597254+0000","last_peered":"2024-03-24T00:13:03.597254+0000","last_clean":"2024-03-24T00:13:03.597254+0000","last_became_active":"2024-03-24T00:13:02.330313+0000","last_became_peered":"2024-03-24T00:13:02.330313+0000","last_unstale":"2024-03-24T00:13:03.597254+0000","last_undegraded":"2024-03-24T00:13:03.597254+0000","last_fullsized":"2024-03-24T00:13:03.597254+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:13:01.267198+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:13:01.267198+0000","last_clean_scrub_stamp":"2024-03-24T00:13:01.267198+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T03:45:38.480929+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,6],"acting":[8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3a","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-24T00:13:04.365800+0000","last_change":"2024-03-24T00:13:02.313115+0000","last_active":"2024-03-24T00:13:04.365800+0000","last_peered":"2024-03-24T00:13:04.365800+0000","last_clean":"2024-03-24T00:13:04.365800+0000","last_became_active":"2024-03-24T00:13:02.312387+0000","last_became_peered":"2024-03-24T00:13:02.312387+0000","last_unstale":"2024-03-24T00:13:04.365800+0000","last_undegraded":"2024-03-24T00:13:04.365800+0000","last_fullsized":"2024-03-24T00:13:04.365800+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:13:01.267198+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:13:01.267198+0000","last_clean_scrub_stamp":"2024-03-24T00:13:01.267198+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T09:16:02.574920+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,4],"acting":[10,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.39","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-24T00:13:03.840695+0000","last_change":"2024-03-24T00:13:02.315117+0000","last_active":"2024-03-24T00:13:03.840695+0000","last_peered":"2024-03-24T00:13:03.840695+0000","last_clean":"2024-03-24T00:13:03.840695+0000","last_became_active":"2024-03-24T00:13:02.314834+0000","last_became_peered":"2024-03-24T00:13:02.314834+0000","last_unstale":"2024-03-24T00:13:03.840695+0000","last_undegraded":"2024-03-24T00:13:03.840695+0000","last_fullsized":"2024-03-24T00:13:03.840695+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:13:01.267198+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:13:01.267198+0000","last_clean_scrub_stamp":"2024-03-24T00:13:01.267198+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T08:05:17.021141+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,5],"acting":[7,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-24T00:13:04.347277+0000","last_change":"2024-03-24T00:13:02.331381+0000","last_active":"2024-03-24T00:13:04.347277+0000","last_peered":"2024-03-24T00:13:04.347277+0000","last_clean":"2024-03-24T00:13:04.347277+0000","last_became_active":"2024-03-24T00:13:02.330883+0000","last_became_peered":"2024-03-24T00:13:02.330883+0000","last_unstale":"2024-03-24T00:13:04.347277+0000","last_undegraded":"2024-03-24T00:13:04.347277+0000","last_fullsized":"2024-03-24T00:13:04.347277+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:13:01.267198+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:13:01.267198+0000","last_clean_scrub_stamp":"2024-03-24T00:13:01.267198+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T00:21:17.021540+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,10],"acting":[3,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.37","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-24T00:13:04.366392+0000","last_change":"2024-03-24T00:13:02.305625+0000","last_active":"2024-03-24T00:13:04.366392+0000","last_peered":"2024-03-24T00:13:04.366392+0000","last_clean":"2024-03-24T00:13:04.366392+0000","last_became_active":"2024-03-24T00:13:02.305221+0000","last_became_peered":"2024-03-24T00:13:02.305221+0000","last_unstale":"2024-03-24T00:13:04.366392+0000","last_undegraded":"2024-03-24T00:13:04.366392+0000","last_fullsized":"2024-03-24T00:13:04.366392+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:13:01.267198+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:13:01.267198+0000","last_clean_scrub_stamp":"2024-03-24T00:13:01.267198+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T04:37:12.467387+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,11],"acting":[1,5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.36","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-24T00:13:04.366223+0000","last_change":"2024-03-24T00:13:02.307878+0000","last_active":"2024-03-24T00:13:04.366223+0000","last_peered":"2024-03-24T00:13:04.366223+0000","last_clean":"2024-03-24T00:13:04.366223+0000","last_became_active":"2024-03-24T00:13:02.307513+0000","last_became_peered":"2024-03-24T00:13:02.307513+0000","last_unstale":"2024-03-24T00:13:04.366223+0000","last_undegraded":"2024-03-24T00:13:04.366223+0000","last_fullsized":"2024-03-24T00:13:04.366223+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:13:01.267198+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:13:01.267198+0000","last_clean_scrub_stamp":"2024-03-24T00:13:01.267198+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T07:06:08.162416+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,4],"acting":[10,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.35","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-24T00:13:03.841008+0000","last_change":"2024-03-24T00:13:02.310883+0000","last_active":"2024-03-24T00:13:03.841008+0000","last_peered":"2024-03-24T00:13:03.841008+0000","last_clean":"2024-03-24T00:13:03.841008+0000","last_became_active":"2024-03-24T00:13:02.310559+0000","last_became_peered":"2024-03-24T00:13:02.310559+0000","last_unstale":"2024-03-24T00:13:03.841008+0000","last_undegraded":"2024-03-24T00:13:03.841008+0000","last_fullsized":"2024-03-24T00:13:03.841008+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:13:01.267198+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:13:01.267198+0000","last_clean_scrub_stamp":"2024-03-24T00:13:01.267198+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T04:04:25.801248+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8,1],"acting":[7,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.34","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-24T00:13:03.290545+0000","last_change":"2024-03-24T00:13:02.308747+0000","last_active":"2024-03-24T00:13:03.290545+0000","last_peered":"2024-03-24T00:13:03.290545+0000","last_clean":"2024-03-24T00:13:03.290545+0000","last_became_active":"2024-03-24T00:13:02.307791+0000","last_became_peered":"2024-03-24T00:13:02.307791+0000","last_unstale":"2024-03-24T00:13:03.290545+0000","last_undegraded":"2024-03-24T00:13:03.290545+0000","last_fullsized":"2024-03-24T00:13:03.290545+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:13:01.267198+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:13:01.267198+0000","last_clean_scrub_stamp":"2024-03-24T00:13:01.267198+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T03:28:08.780044+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,8],"acting":[4,10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-24T00:13:03.673934+0000","last_change":"2024-03-24T00:13:02.327061+0000","last_active":"2024-03-24T00:13:03.673934+0000","last_peered":"2024-03-24T00:13:03.673934+0000","last_clean":"2024-03-24T00:13:03.673934+0000","last_became_active":"2024-03-24T00:13:02.326696+0000","last_became_peered":"2024-03-24T00:13:02.326696+0000","last_unstale":"2024-03-24T00:13:03.673934+0000","last_undegraded":"2024-03-24T00:13:03.673934+0000","last_fullsized":"2024-03-24T00:13:03.673934+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:13:01.267198+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:13:01.267198+0000","last_clean_scrub_stamp":"2024-03-24T00:13:01.267198+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T08:55:17.325488+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,5],"acting":[6,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.32","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-24T00:13:03.402980+0000","last_change":"2024-03-24T00:13:02.321326+0000","last_active":"2024-03-24T00:13:03.402980+0000","last_peered":"2024-03-24T00:13:03.402980+0000","last_clean":"2024-03-24T00:13:03.402980+0000","last_became_active":"2024-03-24T00:13:02.320964+0000","last_became_peered":"2024-03-24T00:13:02.320964+0000","last_unstale":"2024-03-24T00:13:03.402980+0000","last_undegraded":"2024-03-24T00:13:03.402980+0000","last_fullsized":"2024-03-24T00:13:03.402980+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:13:01.267198+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:13:01.267198+0000","last_clean_scrub_stamp":"2024-03-24T00:13:01.267198+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T08:05:15.365809+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,10],"acting":[0,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-24T00:13:03.403404+0000","last_change":"2024-03-24T00:13:02.323922+0000","last_active":"2024-03-24T00:13:03.403404+0000","last_peered":"2024-03-24T00:13:03.403404+0000","last_clean":"2024-03-24T00:13:03.403404+0000","last_became_active":"2024-03-24T00:13:02.323440+0000","last_became_peered":"2024-03-24T00:13:02.323440+0000","last_unstale":"2024-03-24T00:13:03.403404+0000","last_undegraded":"2024-03-24T00:13:03.403404+0000","last_fullsized":"2024-03-24T00:13:03.403404+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:13:01.267198+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:13:01.267198+0000","last_clean_scrub_stamp":"2024-03-24T00:13:01.267198+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T01:26:49.115140+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,2],"acting":[0,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.30","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-24T00:13:04.365721+0000","last_change":"2024-03-24T00:13:02.325460+0000","last_active":"2024-03-24T00:13:04.365721+0000","last_peered":"2024-03-24T00:13:04.365721+0000","last_clean":"2024-03-24T00:13:04.365721+0000","last_became_active":"2024-03-24T00:13:02.325112+0000","last_became_peered":"2024-03-24T00:13:02.325112+0000","last_unstale":"2024-03-24T00:13:04.365721+0000","last_undegraded":"2024-03-24T00:13:04.365721+0000","last_fullsized":"2024-03-24T00:13:04.365721+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:13:01.267198+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:13:01.267198+0000","last_clean_scrub_stamp":"2024-03-24T00:13:01.267198+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T03:15:16.362970+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2f","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-24T00:13:03.860561+0000","last_change":"2024-03-24T00:13:02.325918+0000","last_active":"2024-03-24T00:13:03.860561+0000","last_peered":"2024-03-24T00:13:03.860561+0000","last_clean":"2024-03-24T00:13:03.860561+0000","last_became_active":"2024-03-24T00:13:02.325557+0000","last_became_peered":"2024-03-24T00:13:02.325557+0000","last_unstale":"2024-03-24T00:13:03.860561+0000","last_undegraded":"2024-03-24T00:13:03.860561+0000","last_fullsized":"2024-03-24T00:13:03.860561+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:13:01.267198+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:13:01.267198+0000","last_clean_scrub_stamp":"2024-03-24T00:13:01.267198+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T02:55:43.759674+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2e","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-24T00:13:03.288385+0000","last_change":"2024-03-24T00:13:02.308106+0000","last_active":"2024-03-24T00:13:03.288385+0000","last_peered":"2024-03-24T00:13:03.288385+0000","last_clean":"2024-03-24T00:13:03.288385+0000","last_became_active":"2024-03-24T00:13:02.306972+0000","last_became_peered":"2024-03-24T00:13:02.306972+0000","last_unstale":"2024-03-24T00:13:03.288385+0000","last_undegraded":"2024-03-24T00:13:03.288385+0000","last_fullsized":"2024-03-24T00:13:03.288385+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:13:01.267198+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:13:01.267198+0000","last_clean_scrub_stamp":"2024-03-24T00:13:01.267198+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T09:38:54.677255+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,10],"acting":[9,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-24T00:13:03.289954+0000","last_change":"2024-03-24T00:13:02.331742+0000","last_active":"2024-03-24T00:13:03.289954+0000","last_peered":"2024-03-24T00:13:03.289954+0000","last_clean":"2024-03-24T00:13:03.289954+0000","last_became_active":"2024-03-24T00:13:02.331103+0000","last_became_peered":"2024-03-24T00:13:02.331103+0000","last_unstale":"2024-03-24T00:13:03.289954+0000","last_undegraded":"2024-03-24T00:13:03.289954+0000","last_fullsized":"2024-03-24T00:13:03.289954+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:13:01.267198+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:13:01.267198+0000","last_clean_scrub_stamp":"2024-03-24T00:13:01.267198+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:25:58.116306+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1f","version":"86'11","reported_seq":27,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-24T00:13:07.331461+0000","last_change":"2024-03-24T00:13:01.310276+0000","last_active":"2024-03-24T00:13:07.331461+0000","last_peered":"2024-03-24T00:13:07.331461+0000","last_clean":"2024-03-24T00:13:07.331461+0000","last_became_active":"2024-03-24T00:13:01.308169+0000","last_became_peered":"2024-03-24T00:13:01.308169+0000","last_unstale":"2024-03-24T00:13:07.331461+0000","last_undegraded":"2024-03-24T00:13:07.331461+0000","last_fullsized":"2024-03-24T00:13:07.331461+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:13:00.222790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:13:00.222790+0000","last_clean_scrub_stamp":"2024-03-24T00:13:00.222790+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T05:41:02.833903+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":13,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,8],"acting":[9,7,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.1e","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-24T00:13:04.348213+0000","last_change":"2024-03-24T00:13:02.322461+0000","last_active":"2024-03-24T00:13:04.348213+0000","last_peered":"2024-03-24T00:13:04.348213+0000","last_clean":"2024-03-24T00:13:04.348213+0000","last_became_active":"2024-03-24T00:13:02.321752+0000","last_became_peered":"2024-03-24T00:13:02.321752+0000","last_unstale":"2024-03-24T00:13:04.348213+0000","last_undegraded":"2024-03-24T00:13:04.348213+0000","last_fullsized":"2024-03-24T00:13:04.348213+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:13:01.267198+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:13:01.267198+0000","last_clean_scrub_stamp":"2024-03-24T00:13:01.267198+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T06:37:05.825919+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1e","version":"86'2","reported_seq":17,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-24T00:13:04.348878+0000","last_change":"2024-03-24T00:13:01.324972+0000","last_active":"2024-03-24T00:13:04.348878+0000","last_peered":"2024-03-24T00:13:04.348878+0000","last_clean":"2024-03-24T00:13:04.348878+0000","last_became_active":"2024-03-24T00:13:01.323342+0000","last_became_peered":"2024-03-24T00:13:01.323342+0000","last_unstale":"2024-03-24T00:13:04.348878+0000","last_undegraded":"2024-03-24T00:13:04.348878+0000","last_fullsized":"2024-03-24T00:13:04.348878+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:13:00.222790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:13:00.222790+0000","last_clean_scrub_stamp":"2024-03-24T00:13:00.222790+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T05:34:35.827634+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-24T00:13:03.404090+0000","last_change":"2024-03-24T00:13:02.325455+0000","last_active":"2024-03-24T00:13:03.404090+0000","last_peered":"2024-03-24T00:13:03.404090+0000","last_clean":"2024-03-24T00:13:03.404090+0000","last_became_active":"2024-03-24T00:13:02.325061+0000","last_became_peered":"2024-03-24T00:13:02.325061+0000","last_unstale":"2024-03-24T00:13:03.404090+0000","last_undegraded":"2024-03-24T00:13:03.404090+0000","last_fullsized":"2024-03-24T00:13:03.404090+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:13:01.267198+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:13:01.267198+0000","last_clean_scrub_stamp":"2024-03-24T00:13:01.267198+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T06:13:33.223361+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,9],"acting":[0,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1d","version":"86'7","reported_seq":22,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-24T00:13:07.327587+0000","last_change":"2024-03-24T00:13:01.300541+0000","last_active":"2024-03-24T00:13:07.327587+0000","last_peered":"2024-03-24T00:13:07.327587+0000","last_clean":"2024-03-24T00:13:07.327587+0000","last_became_active":"2024-03-24T00:13:01.299686+0000","last_became_peered":"2024-03-24T00:13:01.299686+0000","last_unstale":"2024-03-24T00:13:07.327587+0000","last_undegraded":"2024-03-24T00:13:07.327587+0000","last_fullsized":"2024-03-24T00:13:07.327587+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:13:00.222790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:13:00.222790+0000","last_clean_scrub_stamp":"2024-03-24T00:13:00.222790+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T00:20:54.474131+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1c","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-24T00:13:04.365967+0000","last_change":"2024-03-24T00:13:02.325130+0000","last_active":"2024-03-24T00:13:04.365967+0000","last_peered":"2024-03-24T00:13:04.365967+0000","last_clean":"2024-03-24T00:13:04.365967+0000","last_became_active":"2024-03-24T00:13:02.324740+0000","last_became_peered":"2024-03-24T00:13:02.324740+0000","last_unstale":"2024-03-24T00:13:04.365967+0000","last_undegraded":"2024-03-24T00:13:04.365967+0000","last_fullsized":"2024-03-24T00:13:04.365967+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:13:01.267198+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:13:01.267198+0000","last_clean_scrub_stamp":"2024-03-24T00:13:01.267198+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T01:23:35.194788+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,6],"acting":[1,10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1c","version":"85'2","reported_seq":17,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-24T00:13:03.597568+0000","last_change":"2024-03-24T00:13:01.300572+0000","last_active":"2024-03-24T00:13:03.597568+0000","last_peered":"2024-03-24T00:13:03.597568+0000","last_clean":"2024-03-24T00:13:03.597568+0000","last_became_active":"2024-03-24T00:13:01.299801+0000","last_became_peered":"2024-03-24T00:13:01.299801+0000","last_unstale":"2024-03-24T00:13:03.597568+0000","last_undegraded":"2024-03-24T00:13:03.597568+0000","last_fullsized":"2024-03-24T00:13:03.597568+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:13:00.222790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:13:00.222790+0000","last_clean_scrub_stamp":"2024-03-24T00:13:00.222790+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T10:29:03.986969+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,5],"acting":[8,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1d","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-24T00:13:03.429608+0000","last_change":"2024-03-24T00:13:02.304079+0000","last_active":"2024-03-24T00:13:03.429608+0000","last_peered":"2024-03-24T00:13:03.429608+0000","last_clean":"2024-03-24T00:13:03.429608+0000","last_became_active":"2024-03-24T00:13:02.303700+0000","last_became_peered":"2024-03-24T00:13:02.303700+0000","last_unstale":"2024-03-24T00:13:03.429608+0000","last_undegraded":"2024-03-24T00:13:03.429608+0000","last_fullsized":"2024-03-24T00:13:03.429608+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:13:01.267198+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:13:01.267198+0000","last_clean_scrub_stamp":"2024-03-24T00:13:01.267198+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T10:19:07.701927+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1b","version":"86'5","reported_seq":24,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-24T00:13:06.280203+0000","last_change":"2024-03-24T00:13:01.300949+0000","last_active":"2024-03-24T00:13:06.280203+0000","last_peered":"2024-03-24T00:13:06.280203+0000","last_clean":"2024-03-24T00:13:06.280203+0000","last_became_active":"2024-03-24T00:13:01.300605+0000","last_became_peered":"2024-03-24T00:13:01.300605+0000","last_unstale":"2024-03-24T00:13:06.280203+0000","last_undegraded":"2024-03-24T00:13:06.280203+0000","last_fullsized":"2024-03-24T00:13:06.280203+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:13:00.222790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:13:00.222790+0000","last_clean_scrub_stamp":"2024-03-24T00:13:00.222790+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T02:47:58.283103+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":622,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":4,"num_read_kb":2,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1a","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-24T00:13:03.597357+0000","last_change":"2024-03-24T00:13:02.312884+0000","last_active":"2024-03-24T00:13:03.597357+0000","last_peered":"2024-03-24T00:13:03.597357+0000","last_clean":"2024-03-24T00:13:03.597357+0000","last_became_active":"2024-03-24T00:13:02.312533+0000","last_became_peered":"2024-03-24T00:13:02.312533+0000","last_unstale":"2024-03-24T00:13:03.597357+0000","last_undegraded":"2024-03-24T00:13:03.597357+0000","last_fullsized":"2024-03-24T00:13:03.597357+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:13:01.267198+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:13:01.267198+0000","last_clean_scrub_stamp":"2024-03-24T00:13:01.267198+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T10:54:48.398645+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,5],"acting":[8,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.1a","version":"86'7","reported_seq":22,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-24T00:13:07.331054+0000","last_change":"2024-03-24T00:13:01.306292+0000","last_active":"2024-03-24T00:13:07.331054+0000","last_peered":"2024-03-24T00:13:07.331054+0000","last_clean":"2024-03-24T00:13:07.331054+0000","last_became_active":"2024-03-24T00:13:01.305898+0000","last_became_peered":"2024-03-24T00:13:01.305898+0000","last_unstale":"2024-03-24T00:13:07.331054+0000","last_undegraded":"2024-03-24T00:13:07.331054+0000","last_fullsized":"2024-03-24T00:13:07.331054+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:13:00.222790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:13:00.222790+0000","last_clean_scrub_stamp":"2024-03-24T00:13:00.222790+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T07:41:17.003588+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8,6],"acting":[10,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.1b","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-24T00:13:03.429437+0000","last_change":"2024-03-24T00:13:02.323763+0000","last_active":"2024-03-24T00:13:03.429437+0000","last_peered":"2024-03-24T00:13:03.429437+0000","last_clean":"2024-03-24T00:13:03.429437+0000","last_became_active":"2024-03-24T00:13:02.323367+0000","last_became_peered":"2024-03-24T00:13:02.323367+0000","last_unstale":"2024-03-24T00:13:03.429437+0000","last_undegraded":"2024-03-24T00:13:03.429437+0000","last_fullsized":"2024-03-24T00:13:03.429437+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:13:01.267198+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:13:01.267198+0000","last_clean_scrub_stamp":"2024-03-24T00:13:01.267198+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T08:53:21.671317+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,3],"acting":[5,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.19","version":"86'2","reported_seq":17,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-24T00:13:06.298333+0000","last_change":"2024-03-24T00:13:01.323111+0000","last_active":"2024-03-24T00:13:06.298333+0000","last_peered":"2024-03-24T00:13:06.298333+0000","last_clean":"2024-03-24T00:13:06.298333+0000","last_became_active":"2024-03-24T00:13:01.322772+0000","last_became_peered":"2024-03-24T00:13:01.322772+0000","last_unstale":"2024-03-24T00:13:06.298333+0000","last_undegraded":"2024-03-24T00:13:06.298333+0000","last_fullsized":"2024-03-24T00:13:06.298333+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:13:00.222790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:13:00.222790+0000","last_clean_scrub_stamp":"2024-03-24T00:13:00.222790+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T04:19:53.097933+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,9],"acting":[6,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-24T00:13:04.347578+0000","last_change":"2024-03-24T00:13:02.330469+0000","last_active":"2024-03-24T00:13:04.347578+0000","last_peered":"2024-03-24T00:13:04.347578+0000","last_clean":"2024-03-24T00:13:04.347578+0000","last_became_active":"2024-03-24T00:13:02.327019+0000","last_became_peered":"2024-03-24T00:13:02.327019+0000","last_unstale":"2024-03-24T00:13:04.347578+0000","last_undegraded":"2024-03-24T00:13:04.347578+0000","last_fullsized":"2024-03-24T00:13:04.347578+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:13:01.267198+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:13:01.267198+0000","last_clean_scrub_stamp":"2024-03-24T00:13:01.267198+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T07:30:37.155535+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,6],"acting":[3,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.18","version":"86'2","reported_seq":17,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-24T00:13:05.401573+0000","last_change":"2024-03-24T00:13:01.300089+0000","last_active":"2024-03-24T00:13:05.401573+0000","last_peered":"2024-03-24T00:13:05.401573+0000","last_clean":"2024-03-24T00:13:05.401573+0000","last_became_active":"2024-03-24T00:13:01.299747+0000","last_became_peered":"2024-03-24T00:13:01.299747+0000","last_unstale":"2024-03-24T00:13:05.401573+0000","last_undegraded":"2024-03-24T00:13:05.401573+0000","last_fullsized":"2024-03-24T00:13:05.401573+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:13:00.222790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:13:00.222790+0000","last_clean_scrub_stamp":"2024-03-24T00:13:00.222790+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T00:30:30.142785+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-24T00:13:03.287810+0000","last_change":"2024-03-24T00:13:02.332490+0000","last_active":"2024-03-24T00:13:03.287810+0000","last_peered":"2024-03-24T00:13:03.287810+0000","last_clean":"2024-03-24T00:13:03.287810+0000","last_became_active":"2024-03-24T00:13:02.331182+0000","last_became_peered":"2024-03-24T00:13:02.331182+0000","last_unstale":"2024-03-24T00:13:03.287810+0000","last_undegraded":"2024-03-24T00:13:03.287810+0000","last_fullsized":"2024-03-24T00:13:03.287810+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:13:01.267198+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:13:01.267198+0000","last_clean_scrub_stamp":"2024-03-24T00:13:01.267198+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T06:22:16.084879+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,10],"acting":[9,3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.17","version":"86'4","reported_seq":22,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-24T00:13:07.331208+0000","last_change":"2024-03-24T00:13:01.299818+0000","last_active":"2024-03-24T00:13:07.331208+0000","last_peered":"2024-03-24T00:13:07.331208+0000","last_clean":"2024-03-24T00:13:07.331208+0000","last_became_active":"2024-03-24T00:13:01.299478+0000","last_became_peered":"2024-03-24T00:13:01.299478+0000","last_unstale":"2024-03-24T00:13:07.331208+0000","last_undegraded":"2024-03-24T00:13:07.331208+0000","last_fullsized":"2024-03-24T00:13:07.331208+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:13:00.222790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:13:00.222790+0000","last_clean_scrub_stamp":"2024-03-24T00:13:00.222790+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:51:47.088677+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":180,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":3,"num_read_kb":3,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,9],"acting":[10,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.16","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-24T00:13:04.366406+0000","last_change":"2024-03-24T00:13:02.304971+0000","last_active":"2024-03-24T00:13:04.366406+0000","last_peered":"2024-03-24T00:13:04.366406+0000","last_clean":"2024-03-24T00:13:04.366406+0000","last_became_active":"2024-03-24T00:13:02.304577+0000","last_became_peered":"2024-03-24T00:13:02.304577+0000","last_unstale":"2024-03-24T00:13:04.366406+0000","last_undegraded":"2024-03-24T00:13:04.366406+0000","last_fullsized":"2024-03-24T00:13:04.366406+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:13:01.267198+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:13:01.267198+0000","last_clean_scrub_stamp":"2024-03-24T00:13:01.267198+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T05:08:03.444554+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,9],"acting":[1,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.16","version":"86'2","reported_seq":17,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-24T00:13:05.400208+0000","last_change":"2024-03-24T00:13:01.303075+0000","last_active":"2024-03-24T00:13:05.400208+0000","last_peered":"2024-03-24T00:13:05.400208+0000","last_clean":"2024-03-24T00:13:05.400208+0000","last_became_active":"2024-03-24T00:13:01.301517+0000","last_became_peered":"2024-03-24T00:13:01.301517+0000","last_unstale":"2024-03-24T00:13:05.400208+0000","last_undegraded":"2024-03-24T00:13:05.400208+0000","last_fullsized":"2024-03-24T00:13:05.400208+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:13:00.222790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:13:00.222790+0000","last_clean_scrub_stamp":"2024-03-24T00:13:00.222790+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:04:10.552510+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.17","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-24T00:13:03.403947+0000","last_change":"2024-03-24T00:13:02.320108+0000","last_active":"2024-03-24T00:13:03.403947+0000","last_peered":"2024-03-24T00:13:03.403947+0000","last_clean":"2024-03-24T00:13:03.403947+0000","last_became_active":"2024-03-24T00:13:02.319155+0000","last_became_peered":"2024-03-24T00:13:02.319155+0000","last_unstale":"2024-03-24T00:13:03.403947+0000","last_undegraded":"2024-03-24T00:13:03.403947+0000","last_fullsized":"2024-03-24T00:13:03.403947+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:13:01.267198+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:13:01.267198+0000","last_clean_scrub_stamp":"2024-03-24T00:13:01.267198+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T10:15:54.770304+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,5],"acting":[0,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.15","version":"86'5","reported_seq":21,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-24T00:13:06.279634+0000","last_change":"2024-03-24T00:13:01.323919+0000","last_active":"2024-03-24T00:13:06.279634+0000","last_peered":"2024-03-24T00:13:06.279634+0000","last_clean":"2024-03-24T00:13:06.279634+0000","last_became_active":"2024-03-24T00:13:01.323640+0000","last_became_peered":"2024-03-24T00:13:01.323640+0000","last_unstale":"2024-03-24T00:13:06.279634+0000","last_undegraded":"2024-03-24T00:13:06.279634+0000","last_fullsized":"2024-03-24T00:13:06.279634+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:13:00.222790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:13:00.222790+0000","last_clean_scrub_stamp":"2024-03-24T00:13:00.222790+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T06:16:49.018642+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":7,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,10],"acting":[5,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.14","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-24T00:13:04.366231+0000","last_change":"2024-03-24T00:13:02.304945+0000","last_active":"2024-03-24T00:13:04.366231+0000","last_peered":"2024-03-24T00:13:04.366231+0000","last_clean":"2024-03-24T00:13:04.366231+0000","last_became_active":"2024-03-24T00:13:02.304599+0000","last_became_peered":"2024-03-24T00:13:02.304599+0000","last_unstale":"2024-03-24T00:13:04.366231+0000","last_undegraded":"2024-03-24T00:13:04.366231+0000","last_fullsized":"2024-03-24T00:13:04.366231+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:13:01.267198+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:13:01.267198+0000","last_clean_scrub_stamp":"2024-03-24T00:13:01.267198+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T05:32:02.763946+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,4],"acting":[1,8,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.14","version":"86'17","reported_seq":40,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-24T00:13:08.834016+0000","last_change":"2024-03-24T00:13:01.301460+0000","last_active":"2024-03-24T00:13:08.834016+0000","last_peered":"2024-03-24T00:13:08.834016+0000","last_clean":"2024-03-24T00:13:08.834016+0000","last_became_active":"2024-03-24T00:13:01.301014+0000","last_became_peered":"2024-03-24T00:13:01.301014+0000","last_unstale":"2024-03-24T00:13:08.834016+0000","last_undegraded":"2024-03-24T00:13:08.834016+0000","last_fullsized":"2024-03-24T00:13:08.834016+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:13:00.222790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:13:00.222790+0000","last_clean_scrub_stamp":"2024-03-24T00:13:00.222790+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T01:05:15.984359+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":20895,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":8,"num_read_kb":5,"num_write":19,"num_write_kb":24,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,2],"acting":[10,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.15","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-24T00:13:03.597622+0000","last_change":"2024-03-24T00:13:02.307653+0000","last_active":"2024-03-24T00:13:03.597622+0000","last_peered":"2024-03-24T00:13:03.597622+0000","last_clean":"2024-03-24T00:13:03.597622+0000","last_became_active":"2024-03-24T00:13:02.307287+0000","last_became_peered":"2024-03-24T00:13:02.307287+0000","last_unstale":"2024-03-24T00:13:03.597622+0000","last_undegraded":"2024-03-24T00:13:03.597622+0000","last_fullsized":"2024-03-24T00:13:03.597622+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:13:01.267198+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:13:01.267198+0000","last_clean_scrub_stamp":"2024-03-24T00:13:01.267198+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T05:59:27.701614+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.13","version":"86'9","reported_seq":26,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-24T00:13:08.345059+0000","last_change":"2024-03-24T00:13:01.316854+0000","last_active":"2024-03-24T00:13:08.345059+0000","last_peered":"2024-03-24T00:13:08.345059+0000","last_clean":"2024-03-24T00:13:08.345059+0000","last_became_active":"2024-03-24T00:13:01.316507+0000","last_became_peered":"2024-03-24T00:13:01.316507+0000","last_unstale":"2024-03-24T00:13:08.345059+0000","last_undegraded":"2024-03-24T00:13:08.345059+0000","last_fullsized":"2024-03-24T00:13:08.345059+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:13:00.222790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:13:00.222790+0000","last_clean_scrub_stamp":"2024-03-24T00:13:00.222790+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T07:58:41.888003+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":13,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,10,3],"acting":[2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.12","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-24T00:13:03.673410+0000","last_change":"2024-03-24T00:13:02.338811+0000","last_active":"2024-03-24T00:13:03.673410+0000","last_peered":"2024-03-24T00:13:03.673410+0000","last_clean":"2024-03-24T00:13:03.673410+0000","last_became_active":"2024-03-24T00:13:02.338438+0000","last_became_peered":"2024-03-24T00:13:02.338438+0000","last_unstale":"2024-03-24T00:13:03.673410+0000","last_undegraded":"2024-03-24T00:13:03.673410+0000","last_fullsized":"2024-03-24T00:13:03.673410+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:13:01.267198+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:13:01.267198+0000","last_clean_scrub_stamp":"2024-03-24T00:13:01.267198+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T03:55:16.531224+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.12","version":"86'4","reported_seq":19,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-24T00:13:05.403830+0000","last_change":"2024-03-24T00:13:01.313914+0000","last_active":"2024-03-24T00:13:05.403830+0000","last_peered":"2024-03-24T00:13:05.403830+0000","last_clean":"2024-03-24T00:13:05.403830+0000","last_became_active":"2024-03-24T00:13:01.313535+0000","last_became_peered":"2024-03-24T00:13:01.313535+0000","last_unstale":"2024-03-24T00:13:05.403830+0000","last_undegraded":"2024-03-24T00:13:05.403830+0000","last_fullsized":"2024-03-24T00:13:05.403830+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:13:00.222790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:13:00.222790+0000","last_clean_scrub_stamp":"2024-03-24T00:13:00.222790+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T03:23:53.011550+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-24T00:13:03.840909+0000","last_change":"2024-03-24T00:13:02.334755+0000","last_active":"2024-03-24T00:13:03.840909+0000","last_peered":"2024-03-24T00:13:03.840909+0000","last_clean":"2024-03-24T00:13:03.840909+0000","last_became_active":"2024-03-24T00:13:02.334444+0000","last_became_peered":"2024-03-24T00:13:02.334444+0000","last_unstale":"2024-03-24T00:13:03.840909+0000","last_undegraded":"2024-03-24T00:13:03.840909+0000","last_fullsized":"2024-03-24T00:13:03.840909+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:13:01.267198+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:13:01.267198+0000","last_clean_scrub_stamp":"2024-03-24T00:13:01.267198+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T06:05:44.006815+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,4,6],"acting":[7,4,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.11","version":"86'5","reported_seq":20,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-24T00:13:07.324761+0000","last_change":"2024-03-24T00:13:01.297887+0000","last_active":"2024-03-24T00:13:07.324761+0000","last_peered":"2024-03-24T00:13:07.324761+0000","last_clean":"2024-03-24T00:13:07.324761+0000","last_became_active":"2024-03-24T00:13:01.297560+0000","last_became_peered":"2024-03-24T00:13:01.297560+0000","last_unstale":"2024-03-24T00:13:07.324761+0000","last_undegraded":"2024-03-24T00:13:07.324761+0000","last_fullsized":"2024-03-24T00:13:07.324761+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:13:00.222790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:13:00.222790+0000","last_clean_scrub_stamp":"2024-03-24T00:13:00.222790+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T01:06:28.927274+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":588,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,10],"acting":[4,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.10","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-24T00:13:03.840682+0000","last_change":"2024-03-24T00:13:02.306811+0000","last_active":"2024-03-24T00:13:03.840682+0000","last_peered":"2024-03-24T00:13:03.840682+0000","last_clean":"2024-03-24T00:13:03.840682+0000","last_became_active":"2024-03-24T00:13:02.306499+0000","last_became_peered":"2024-03-24T00:13:02.306499+0000","last_unstale":"2024-03-24T00:13:03.840682+0000","last_undegraded":"2024-03-24T00:13:03.840682+0000","last_fullsized":"2024-03-24T00:13:03.840682+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:13:01.267198+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:13:01.267198+0000","last_clean_scrub_stamp":"2024-03-24T00:13:01.267198+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:49:32.683005+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.10","version":"86'16","reported_seq":31,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-24T00:13:06.298946+0000","last_change":"2024-03-24T00:13:01.307058+0000","last_active":"2024-03-24T00:13:06.298946+0000","last_peered":"2024-03-24T00:13:06.298946+0000","last_clean":"2024-03-24T00:13:06.298946+0000","last_became_active":"2024-03-24T00:13:01.306699+0000","last_became_peered":"2024-03-24T00:13:01.306699+0000","last_unstale":"2024-03-24T00:13:06.298946+0000","last_undegraded":"2024-03-24T00:13:06.298946+0000","last_fullsized":"2024-03-24T00:13:06.298946+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:13:00.222790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:13:00.222790+0000","last_clean_scrub_stamp":"2024-03-24T00:13:00.222790+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T10:32:37.334514+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,2],"acting":[8,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.11","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-24T00:13:03.429225+0000","last_change":"2024-03-24T00:13:02.305052+0000","last_active":"2024-03-24T00:13:03.429225+0000","last_peered":"2024-03-24T00:13:03.429225+0000","last_clean":"2024-03-24T00:13:03.429225+0000","last_became_active":"2024-03-24T00:13:02.304700+0000","last_became_peered":"2024-03-24T00:13:02.304700+0000","last_unstale":"2024-03-24T00:13:03.429225+0000","last_undegraded":"2024-03-24T00:13:03.429225+0000","last_fullsized":"2024-03-24T00:13:03.429225+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:13:01.267198+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:13:01.267198+0000","last_clean_scrub_stamp":"2024-03-24T00:13:01.267198+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T00:43:10.413894+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.f","version":"85'1","reported_seq":17,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-24T00:13:03.597698+0000","last_change":"2024-03-24T00:13:01.295874+0000","last_active":"2024-03-24T00:13:03.597698+0000","last_peered":"2024-03-24T00:13:03.597698+0000","last_clean":"2024-03-24T00:13:03.597698+0000","last_became_active":"2024-03-24T00:13:01.295521+0000","last_became_peered":"2024-03-24T00:13:01.295521+0000","last_unstale":"2024-03-24T00:13:03.597698+0000","last_undegraded":"2024-03-24T00:13:03.597698+0000","last_fullsized":"2024-03-24T00:13:03.597698+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:13:00.222790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:13:00.222790+0000","last_clean_scrub_stamp":"2024-03-24T00:13:00.222790+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T06:47:15.749491+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5,4],"acting":[8,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.e","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-24T00:13:03.860319+0000","last_change":"2024-03-24T00:13:02.306617+0000","last_active":"2024-03-24T00:13:03.860319+0000","last_peered":"2024-03-24T00:13:03.860319+0000","last_clean":"2024-03-24T00:13:03.860319+0000","last_became_active":"2024-03-24T00:13:02.306224+0000","last_became_peered":"2024-03-24T00:13:02.306224+0000","last_unstale":"2024-03-24T00:13:03.860319+0000","last_undegraded":"2024-03-24T00:13:03.860319+0000","last_fullsized":"2024-03-24T00:13:03.860319+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:13:01.267198+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:13:01.267198+0000","last_clean_scrub_stamp":"2024-03-24T00:13:01.267198+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T06:49:16.032741+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,4],"acting":[11,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.4","version":"86'1","reported_seq":16,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-24T00:13:03.288098+0000","last_change":"2024-03-24T00:13:01.314428+0000","last_active":"2024-03-24T00:13:03.288098+0000","last_peered":"2024-03-24T00:13:03.288098+0000","last_clean":"2024-03-24T00:13:03.288098+0000","last_became_active":"2024-03-24T00:13:01.313421+0000","last_became_peered":"2024-03-24T00:13:01.313421+0000","last_unstale":"2024-03-24T00:13:03.288098+0000","last_undegraded":"2024-03-24T00:13:03.288098+0000","last_fullsized":"2024-03-24T00:13:03.288098+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:13:00.222790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:13:00.222790+0000","last_clean_scrub_stamp":"2024-03-24T00:13:00.222790+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T08:10:25.796718+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,0],"acting":[9,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.5","version":"86'1","reported_seq":14,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-24T00:13:04.366169+0000","last_change":"2024-03-24T00:13:02.331660+0000","last_active":"2024-03-24T00:13:04.366169+0000","last_peered":"2024-03-24T00:13:04.366169+0000","last_clean":"2024-03-24T00:13:04.366169+0000","last_became_active":"2024-03-24T00:13:02.331288+0000","last_became_peered":"2024-03-24T00:13:02.331288+0000","last_unstale":"2024-03-24T00:13:04.366169+0000","last_undegraded":"2024-03-24T00:13:04.366169+0000","last_fullsized":"2024-03-24T00:13:04.366169+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:13:01.267198+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:13:01.267198+0000","last_clean_scrub_stamp":"2024-03-24T00:13:01.267198+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:36:55.576510+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3,6],"acting":[10,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.2","version":"86'1","reported_seq":16,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-24T00:13:03.840849+0000","last_change":"2024-03-24T00:13:01.310538+0000","last_active":"2024-03-24T00:13:03.840849+0000","last_peered":"2024-03-24T00:13:03.840849+0000","last_clean":"2024-03-24T00:13:03.840849+0000","last_became_active":"2024-03-24T00:13:01.310124+0000","last_became_peered":"2024-03-24T00:13:01.310124+0000","last_unstale":"2024-03-24T00:13:03.840849+0000","last_undegraded":"2024-03-24T00:13:03.840849+0000","last_fullsized":"2024-03-24T00:13:03.840849+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:13:00.222790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:13:00.222790+0000","last_clean_scrub_stamp":"2024-03-24T00:13:00.222790+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T05:47:03.534824+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-24T00:13:04.365996+0000","last_change":"2024-03-24T00:13:02.324117+0000","last_active":"2024-03-24T00:13:04.365996+0000","last_peered":"2024-03-24T00:13:04.365996+0000","last_clean":"2024-03-24T00:13:04.365996+0000","last_became_active":"2024-03-24T00:13:02.323706+0000","last_became_peered":"2024-03-24T00:13:02.323706+0000","last_unstale":"2024-03-24T00:13:04.365996+0000","last_undegraded":"2024-03-24T00:13:04.365996+0000","last_fullsized":"2024-03-24T00:13:04.365996+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:13:01.267198+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:13:01.267198+0000","last_clean_scrub_stamp":"2024-03-24T00:13:01.267198+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T05:34:31.657324+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,4,0],"acting":[10,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.0","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-24T00:13:03.429464+0000","last_change":"2024-03-24T00:13:02.306030+0000","last_active":"2024-03-24T00:13:03.429464+0000","last_peered":"2024-03-24T00:13:03.429464+0000","last_clean":"2024-03-24T00:13:03.429464+0000","last_became_active":"2024-03-24T00:13:02.305588+0000","last_became_peered":"2024-03-24T00:13:02.305588+0000","last_unstale":"2024-03-24T00:13:03.429464+0000","last_undegraded":"2024-03-24T00:13:03.429464+0000","last_fullsized":"2024-03-24T00:13:03.429464+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:13:01.267198+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:13:01.267198+0000","last_clean_scrub_stamp":"2024-03-24T00:13:01.267198+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:20:58.206244+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,4],"acting":[5,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"86'4","reported_seq":19,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-24T00:13:06.298832+0000","last_change":"2024-03-24T00:13:01.318123+0000","last_active":"2024-03-24T00:13:06.298832+0000","last_peered":"2024-03-24T00:13:06.298832+0000","last_clean":"2024-03-24T00:13:06.298832+0000","last_became_active":"2024-03-24T00:13:01.317683+0000","last_became_peered":"2024-03-24T00:13:01.317683+0000","last_unstale":"2024-03-24T00:13:06.298832+0000","last_undegraded":"2024-03-24T00:13:06.298832+0000","last_fullsized":"2024-03-24T00:13:06.298832+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:13:00.222790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:13:00.222790+0000","last_clean_scrub_stamp":"2024-03-24T00:13:00.222790+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T06:33:02.669668+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-24T00:13:04.348063+0000","last_change":"2024-03-24T00:13:02.328599+0000","last_active":"2024-03-24T00:13:04.348063+0000","last_peered":"2024-03-24T00:13:04.348063+0000","last_clean":"2024-03-24T00:13:04.348063+0000","last_became_active":"2024-03-24T00:13:02.328165+0000","last_became_peered":"2024-03-24T00:13:02.328165+0000","last_unstale":"2024-03-24T00:13:04.348063+0000","last_undegraded":"2024-03-24T00:13:04.348063+0000","last_fullsized":"2024-03-24T00:13:04.348063+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:13:01.267198+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:13:01.267198+0000","last_clean_scrub_stamp":"2024-03-24T00:13:01.267198+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T07:22:39.646578+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"86'6","reported_seq":22,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-24T00:13:07.331671+0000","last_change":"2024-03-24T00:13:01.301932+0000","last_active":"2024-03-24T00:13:07.331671+0000","last_peered":"2024-03-24T00:13:07.331671+0000","last_clean":"2024-03-24T00:13:07.331671+0000","last_became_active":"2024-03-24T00:13:01.301464+0000","last_became_peered":"2024-03-24T00:13:01.301464+0000","last_unstale":"2024-03-24T00:13:07.331671+0000","last_undegraded":"2024-03-24T00:13:07.331671+0000","last_fullsized":"2024-03-24T00:13:07.331671+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:13:00.222790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:13:00.222790+0000","last_clean_scrub_stamp":"2024-03-24T00:13:00.222790+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T08:58:25.512421+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":8,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,9],"acting":[4,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.0","version":"21'137","reported_seq":340,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-24T00:13:03.840668+0000","last_change":"2024-03-24T00:11:43.008767+0000","last_active":"2024-03-24T00:13:03.840668+0000","last_peered":"2024-03-24T00:13:03.840668+0000","last_clean":"2024-03-24T00:13:03.840668+0000","last_became_active":"2024-03-24T00:11:43.008364+0000","last_became_peered":"2024-03-24T00:11:43.008364+0000","last_unstale":"2024-03-24T00:13:03.840668+0000","last_undegraded":"2024-03-24T00:13:03.840668+0000","last_fullsized":"2024-03-24T00:13:03.840668+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:07:50.191828+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:07:50.191828+0000","last_clean_scrub_stamp":"2024-03-24T00:07:50.191828+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T01:55:26.549348+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-24T00:13:03.672974+0000","last_change":"2024-03-24T00:13:02.325667+0000","last_active":"2024-03-24T00:13:03.672974+0000","last_peered":"2024-03-24T00:13:03.672974+0000","last_clean":"2024-03-24T00:13:03.672974+0000","last_became_active":"2024-03-24T00:13:02.322476+0000","last_became_peered":"2024-03-24T00:13:02.322476+0000","last_unstale":"2024-03-24T00:13:03.672974+0000","last_undegraded":"2024-03-24T00:13:03.672974+0000","last_fullsized":"2024-03-24T00:13:03.672974+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:13:01.267198+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:13:01.267198+0000","last_clean_scrub_stamp":"2024-03-24T00:13:01.267198+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T08:07:19.936708+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"86'3","reported_seq":18,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-24T00:13:06.292557+0000","last_change":"2024-03-24T00:13:01.298082+0000","last_active":"2024-03-24T00:13:06.292557+0000","last_peered":"2024-03-24T00:13:06.292557+0000","last_clean":"2024-03-24T00:13:06.292557+0000","last_became_active":"2024-03-24T00:13:01.297686+0000","last_became_peered":"2024-03-24T00:13:01.297686+0000","last_unstale":"2024-03-24T00:13:06.292557+0000","last_undegraded":"2024-03-24T00:13:06.292557+0000","last_fullsized":"2024-03-24T00:13:06.292557+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:13:00.222790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:13:00.222790+0000","last_clean_scrub_stamp":"2024-03-24T00:13:00.222790+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T09:25:07.204076+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.5","version":"86'3","reported_seq":19,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-24T00:13:07.332496+0000","last_change":"2024-03-24T00:13:01.321250+0000","last_active":"2024-03-24T00:13:07.332496+0000","last_peered":"2024-03-24T00:13:07.332496+0000","last_clean":"2024-03-24T00:13:07.332496+0000","last_became_active":"2024-03-24T00:13:01.320929+0000","last_became_peered":"2024-03-24T00:13:01.320929+0000","last_unstale":"2024-03-24T00:13:07.332496+0000","last_undegraded":"2024-03-24T00:13:07.332496+0000","last_fullsized":"2024-03-24T00:13:07.332496+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:13:00.222790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:13:00.222790+0000","last_clean_scrub_stamp":"2024-03-24T00:13:00.222790+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T05:22:05.490303+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,9],"acting":[8,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.4","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-24T00:13:03.673933+0000","last_change":"2024-03-24T00:13:02.323413+0000","last_active":"2024-03-24T00:13:03.673933+0000","last_peered":"2024-03-24T00:13:03.673933+0000","last_clean":"2024-03-24T00:13:03.673933+0000","last_became_active":"2024-03-24T00:13:02.322947+0000","last_became_peered":"2024-03-24T00:13:02.322947+0000","last_unstale":"2024-03-24T00:13:03.673933+0000","last_undegraded":"2024-03-24T00:13:03.673933+0000","last_fullsized":"2024-03-24T00:13:03.673933+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:13:01.267198+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:13:01.267198+0000","last_clean_scrub_stamp":"2024-03-24T00:13:01.267198+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T00:16:14.490270+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,9],"acting":[6,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.6","version":"86'7","reported_seq":22,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-24T00:13:07.334476+0000","last_change":"2024-03-24T00:13:01.320380+0000","last_active":"2024-03-24T00:13:07.334476+0000","last_peered":"2024-03-24T00:13:07.334476+0000","last_clean":"2024-03-24T00:13:07.334476+0000","last_became_active":"2024-03-24T00:13:01.319192+0000","last_became_peered":"2024-03-24T00:13:01.319192+0000","last_unstale":"2024-03-24T00:13:07.334476+0000","last_undegraded":"2024-03-24T00:13:07.334476+0000","last_fullsized":"2024-03-24T00:13:07.334476+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:13:00.222790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:13:00.222790+0000","last_clean_scrub_stamp":"2024-03-24T00:13:00.222790+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T02:09:27.085298+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,3],"acting":[9,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.7","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-24T00:13:04.348800+0000","last_change":"2024-03-24T00:13:02.328067+0000","last_active":"2024-03-24T00:13:04.348800+0000","last_peered":"2024-03-24T00:13:04.348800+0000","last_clean":"2024-03-24T00:13:04.348800+0000","last_became_active":"2024-03-24T00:13:02.327468+0000","last_became_peered":"2024-03-24T00:13:02.327468+0000","last_unstale":"2024-03-24T00:13:04.348800+0000","last_undegraded":"2024-03-24T00:13:04.348800+0000","last_fullsized":"2024-03-24T00:13:04.348800+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:13:01.267198+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:13:01.267198+0000","last_clean_scrub_stamp":"2024-03-24T00:13:01.267198+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T01:21:58.596941+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,7],"acting":[3,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"86'7","reported_seq":22,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-24T00:13:08.343221+0000","last_change":"2024-03-24T00:13:01.309189+0000","last_active":"2024-03-24T00:13:08.343221+0000","last_peered":"2024-03-24T00:13:08.343221+0000","last_clean":"2024-03-24T00:13:08.343221+0000","last_became_active":"2024-03-24T00:13:01.308871+0000","last_became_peered":"2024-03-24T00:13:01.308871+0000","last_unstale":"2024-03-24T00:13:08.343221+0000","last_undegraded":"2024-03-24T00:13:08.343221+0000","last_fullsized":"2024-03-24T00:13:08.343221+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:13:00.222790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:13:00.222790+0000","last_clean_scrub_stamp":"2024-03-24T00:13:00.222790+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T01:00:32.610036+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,6],"acting":[4,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-24T00:13:03.403801+0000","last_change":"2024-03-24T00:13:02.332238+0000","last_active":"2024-03-24T00:13:03.403801+0000","last_peered":"2024-03-24T00:13:03.403801+0000","last_clean":"2024-03-24T00:13:03.403801+0000","last_became_active":"2024-03-24T00:13:02.331909+0000","last_became_peered":"2024-03-24T00:13:02.331909+0000","last_unstale":"2024-03-24T00:13:03.403801+0000","last_undegraded":"2024-03-24T00:13:03.403801+0000","last_fullsized":"2024-03-24T00:13:03.403801+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:13:01.267198+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:13:01.267198+0000","last_clean_scrub_stamp":"2024-03-24T00:13:01.267198+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T04:41:14.762251+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"86'2","reported_seq":17,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-24T00:13:05.402584+0000","last_change":"2024-03-24T00:13:01.306883+0000","last_active":"2024-03-24T00:13:05.402584+0000","last_peered":"2024-03-24T00:13:05.402584+0000","last_clean":"2024-03-24T00:13:05.402584+0000","last_became_active":"2024-03-24T00:13:01.306629+0000","last_became_peered":"2024-03-24T00:13:01.306629+0000","last_unstale":"2024-03-24T00:13:05.402584+0000","last_undegraded":"2024-03-24T00:13:05.402584+0000","last_fullsized":"2024-03-24T00:13:05.402584+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:13:00.222790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:13:00.222790+0000","last_clean_scrub_stamp":"2024-03-24T00:13:00.222790+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T08:11:52.398783+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-24T00:13:03.597200+0000","last_change":"2024-03-24T00:13:02.308421+0000","last_active":"2024-03-24T00:13:03.597200+0000","last_peered":"2024-03-24T00:13:03.597200+0000","last_clean":"2024-03-24T00:13:03.597200+0000","last_became_active":"2024-03-24T00:13:02.308081+0000","last_became_peered":"2024-03-24T00:13:02.308081+0000","last_unstale":"2024-03-24T00:13:03.597200+0000","last_undegraded":"2024-03-24T00:13:03.597200+0000","last_fullsized":"2024-03-24T00:13:03.597200+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:13:01.267198+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:13:01.267198+0000","last_clean_scrub_stamp":"2024-03-24T00:13:01.267198+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T06:50:42.356293+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,10],"acting":[8,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.9","version":"86'5","reported_seq":20,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-24T00:13:07.331289+0000","last_change":"2024-03-24T00:13:01.324439+0000","last_active":"2024-03-24T00:13:07.331289+0000","last_peered":"2024-03-24T00:13:07.331289+0000","last_clean":"2024-03-24T00:13:07.331289+0000","last_became_active":"2024-03-24T00:13:01.324037+0000","last_became_peered":"2024-03-24T00:13:01.324037+0000","last_unstale":"2024-03-24T00:13:07.331289+0000","last_undegraded":"2024-03-24T00:13:07.331289+0000","last_fullsized":"2024-03-24T00:13:07.331289+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:13:00.222790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:13:00.222790+0000","last_clean_scrub_stamp":"2024-03-24T00:13:00.222790+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T01:37:56.767931+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,10],"acting":[3,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.8","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-24T00:13:03.860093+0000","last_change":"2024-03-24T00:13:02.313740+0000","last_active":"2024-03-24T00:13:03.860093+0000","last_peered":"2024-03-24T00:13:03.860093+0000","last_clean":"2024-03-24T00:13:03.860093+0000","last_became_active":"2024-03-24T00:13:02.313401+0000","last_became_peered":"2024-03-24T00:13:02.313401+0000","last_unstale":"2024-03-24T00:13:03.860093+0000","last_undegraded":"2024-03-24T00:13:03.860093+0000","last_fullsized":"2024-03-24T00:13:03.860093+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:13:01.267198+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:13:01.267198+0000","last_clean_scrub_stamp":"2024-03-24T00:13:01.267198+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T04:16:52.569135+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.a","version":"86'3","reported_seq":18,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-24T00:13:07.333350+0000","last_change":"2024-03-24T00:13:01.316870+0000","last_active":"2024-03-24T00:13:07.333350+0000","last_peered":"2024-03-24T00:13:07.333350+0000","last_clean":"2024-03-24T00:13:07.333350+0000","last_became_active":"2024-03-24T00:13:01.316512+0000","last_became_peered":"2024-03-24T00:13:01.316512+0000","last_unstale":"2024-03-24T00:13:07.333350+0000","last_undegraded":"2024-03-24T00:13:07.333350+0000","last_fullsized":"2024-03-24T00:13:07.333350+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:13:00.222790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:13:00.222790+0000","last_clean_scrub_stamp":"2024-03-24T00:13:00.222790+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T08:28:02.418185+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,8],"acting":[2,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.b","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-24T00:13:04.347976+0000","last_change":"2024-03-24T00:13:02.337413+0000","last_active":"2024-03-24T00:13:04.347976+0000","last_peered":"2024-03-24T00:13:04.347976+0000","last_clean":"2024-03-24T00:13:04.347976+0000","last_became_active":"2024-03-24T00:13:02.336910+0000","last_became_peered":"2024-03-24T00:13:02.336910+0000","last_unstale":"2024-03-24T00:13:04.347976+0000","last_undegraded":"2024-03-24T00:13:04.347976+0000","last_fullsized":"2024-03-24T00:13:04.347976+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:13:01.267198+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:13:01.267198+0000","last_clean_scrub_stamp":"2024-03-24T00:13:01.267198+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T10:51:06.549979+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,10],"acting":[3,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.b","version":"86'4","reported_seq":20,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-24T00:13:05.400768+0000","last_change":"2024-03-24T00:13:01.300201+0000","last_active":"2024-03-24T00:13:05.400768+0000","last_peered":"2024-03-24T00:13:05.400768+0000","last_clean":"2024-03-24T00:13:05.400768+0000","last_became_active":"2024-03-24T00:13:01.299750+0000","last_became_peered":"2024-03-24T00:13:01.299750+0000","last_unstale":"2024-03-24T00:13:05.400768+0000","last_undegraded":"2024-03-24T00:13:05.400768+0000","last_fullsized":"2024-03-24T00:13:05.400768+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:13:00.222790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:13:00.222790+0000","last_clean_scrub_stamp":"2024-03-24T00:13:00.222790+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T01:37:32.768926+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.a","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-24T00:13:03.840972+0000","last_change":"2024-03-24T00:13:02.331732+0000","last_active":"2024-03-24T00:13:03.840972+0000","last_peered":"2024-03-24T00:13:03.840972+0000","last_clean":"2024-03-24T00:13:03.840972+0000","last_became_active":"2024-03-24T00:13:02.331365+0000","last_became_peered":"2024-03-24T00:13:02.331365+0000","last_unstale":"2024-03-24T00:13:03.840972+0000","last_undegraded":"2024-03-24T00:13:03.840972+0000","last_fullsized":"2024-03-24T00:13:03.840972+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:13:01.267198+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:13:01.267198+0000","last_clean_scrub_stamp":"2024-03-24T00:13:01.267198+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:27:53.598158+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.c","version":"86'6","reported_seq":24,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-24T00:13:09.346129+0000","last_change":"2024-03-24T00:13:01.308725+0000","last_active":"2024-03-24T00:13:09.346129+0000","last_peered":"2024-03-24T00:13:09.346129+0000","last_clean":"2024-03-24T00:13:09.346129+0000","last_became_active":"2024-03-24T00:13:01.308324+0000","last_became_peered":"2024-03-24T00:13:01.308324+0000","last_unstale":"2024-03-24T00:13:09.346129+0000","last_undegraded":"2024-03-24T00:13:09.346129+0000","last_fullsized":"2024-03-24T00:13:09.346129+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:13:00.222790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:13:00.222790+0000","last_clean_scrub_stamp":"2024-03-24T00:13:00.222790+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T07:43:03.691683+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1561,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":3,"num_read_kb":2,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,11,0],"acting":[2,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-24T00:13:03.841100+0000","last_change":"2024-03-24T00:13:02.305749+0000","last_active":"2024-03-24T00:13:03.841100+0000","last_peered":"2024-03-24T00:13:03.841100+0000","last_clean":"2024-03-24T00:13:03.841100+0000","last_became_active":"2024-03-24T00:13:02.305413+0000","last_became_peered":"2024-03-24T00:13:02.305413+0000","last_unstale":"2024-03-24T00:13:03.841100+0000","last_undegraded":"2024-03-24T00:13:03.841100+0000","last_fullsized":"2024-03-24T00:13:03.841100+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:13:01.267198+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:13:01.267198+0000","last_clean_scrub_stamp":"2024-03-24T00:13:01.267198+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T00:54:56.437725+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,11],"acting":[7,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.d","version":"86'2","reported_seq":18,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-24T00:13:06.279022+0000","last_change":"2024-03-24T00:13:01.304883+0000","last_active":"2024-03-24T00:13:06.279022+0000","last_peered":"2024-03-24T00:13:06.279022+0000","last_clean":"2024-03-24T00:13:06.279022+0000","last_became_active":"2024-03-24T00:13:01.304459+0000","last_became_peered":"2024-03-24T00:13:01.304459+0000","last_unstale":"2024-03-24T00:13:06.279022+0000","last_undegraded":"2024-03-24T00:13:06.279022+0000","last_fullsized":"2024-03-24T00:13:06.279022+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:13:00.222790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:13:00.222790+0000","last_clean_scrub_stamp":"2024-03-24T00:13:00.222790+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T06:14:22.473571+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10,3],"acting":[5,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.c","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-24T00:13:03.840920+0000","last_change":"2024-03-24T00:13:02.331923+0000","last_active":"2024-03-24T00:13:03.840920+0000","last_peered":"2024-03-24T00:13:03.840920+0000","last_clean":"2024-03-24T00:13:03.840920+0000","last_became_active":"2024-03-24T00:13:02.331504+0000","last_became_peered":"2024-03-24T00:13:02.331504+0000","last_unstale":"2024-03-24T00:13:03.840920+0000","last_undegraded":"2024-03-24T00:13:03.840920+0000","last_fullsized":"2024-03-24T00:13:03.840920+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:13:01.267198+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:13:01.267198+0000","last_clean_scrub_stamp":"2024-03-24T00:13:01.267198+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T03:27:17.160248+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,5],"acting":[7,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"85'2","reported_seq":17,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-24T00:13:03.673400+0000","last_change":"2024-03-24T00:13:01.320223+0000","last_active":"2024-03-24T00:13:03.673400+0000","last_peered":"2024-03-24T00:13:03.673400+0000","last_clean":"2024-03-24T00:13:03.673400+0000","last_became_active":"2024-03-24T00:13:01.319342+0000","last_became_peered":"2024-03-24T00:13:01.319342+0000","last_unstale":"2024-03-24T00:13:03.673400+0000","last_undegraded":"2024-03-24T00:13:03.673400+0000","last_fullsized":"2024-03-24T00:13:03.673400+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:13:00.222790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:13:00.222790+0000","last_clean_scrub_stamp":"2024-03-24T00:13:00.222790+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T02:24:08.018521+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.f","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-24T00:13:03.841120+0000","last_change":"2024-03-24T00:13:02.307719+0000","last_active":"2024-03-24T00:13:03.841120+0000","last_peered":"2024-03-24T00:13:03.841120+0000","last_clean":"2024-03-24T00:13:03.841120+0000","last_became_active":"2024-03-24T00:13:02.307399+0000","last_became_peered":"2024-03-24T00:13:02.307399+0000","last_unstale":"2024-03-24T00:13:03.841120+0000","last_undegraded":"2024-03-24T00:13:03.841120+0000","last_fullsized":"2024-03-24T00:13:03.841120+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:13:01.267198+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:13:01.267198+0000","last_clean_scrub_stamp":"2024-03-24T00:13:01.267198+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T09:38:13.077843+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,8],"acting":[7,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.20","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-24T00:13:03.860199+0000","last_change":"2024-03-24T00:13:02.307469+0000","last_active":"2024-03-24T00:13:03.860199+0000","last_peered":"2024-03-24T00:13:03.860199+0000","last_clean":"2024-03-24T00:13:03.860199+0000","last_became_active":"2024-03-24T00:13:02.306873+0000","last_became_peered":"2024-03-24T00:13:02.306873+0000","last_unstale":"2024-03-24T00:13:03.860199+0000","last_undegraded":"2024-03-24T00:13:03.860199+0000","last_fullsized":"2024-03-24T00:13:03.860199+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:13:01.267198+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:13:01.267198+0000","last_clean_scrub_stamp":"2024-03-24T00:13:01.267198+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:19:51.163015+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,5],"acting":[11,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-24T00:13:03.290104+0000","last_change":"2024-03-24T00:13:02.307472+0000","last_active":"2024-03-24T00:13:03.290104+0000","last_peered":"2024-03-24T00:13:03.290104+0000","last_clean":"2024-03-24T00:13:03.290104+0000","last_became_active":"2024-03-24T00:13:02.307099+0000","last_became_peered":"2024-03-24T00:13:02.307099+0000","last_unstale":"2024-03-24T00:13:03.290104+0000","last_undegraded":"2024-03-24T00:13:03.290104+0000","last_fullsized":"2024-03-24T00:13:03.290104+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:13:01.267198+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:13:01.267198+0000","last_clean_scrub_stamp":"2024-03-24T00:13:01.267198+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T08:49:06.461235+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,4],"acting":[2,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-24T00:13:03.289812+0000","last_change":"2024-03-24T00:13:02.331133+0000","last_active":"2024-03-24T00:13:03.289812+0000","last_peered":"2024-03-24T00:13:03.289812+0000","last_clean":"2024-03-24T00:13:03.289812+0000","last_became_active":"2024-03-24T00:13:02.330780+0000","last_became_peered":"2024-03-24T00:13:02.330780+0000","last_unstale":"2024-03-24T00:13:03.289812+0000","last_undegraded":"2024-03-24T00:13:03.289812+0000","last_fullsized":"2024-03-24T00:13:03.289812+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:13:01.267198+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:13:01.267198+0000","last_clean_scrub_stamp":"2024-03-24T00:13:01.267198+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T10:46:07.933131+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.23","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-24T00:13:03.402944+0000","last_change":"2024-03-24T00:13:02.335406+0000","last_active":"2024-03-24T00:13:03.402944+0000","last_peered":"2024-03-24T00:13:03.402944+0000","last_clean":"2024-03-24T00:13:03.402944+0000","last_became_active":"2024-03-24T00:13:02.334994+0000","last_became_peered":"2024-03-24T00:13:02.334994+0000","last_unstale":"2024-03-24T00:13:03.402944+0000","last_undegraded":"2024-03-24T00:13:03.402944+0000","last_fullsized":"2024-03-24T00:13:03.402944+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:13:01.267198+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:13:01.267198+0000","last_clean_scrub_stamp":"2024-03-24T00:13:01.267198+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T03:24:59.547395+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,4],"acting":[0,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-24T00:13:03.674018+0000","last_change":"2024-03-24T00:13:02.332420+0000","last_active":"2024-03-24T00:13:03.674018+0000","last_peered":"2024-03-24T00:13:03.674018+0000","last_clean":"2024-03-24T00:13:03.674018+0000","last_became_active":"2024-03-24T00:13:02.331496+0000","last_became_peered":"2024-03-24T00:13:02.331496+0000","last_unstale":"2024-03-24T00:13:03.674018+0000","last_undegraded":"2024-03-24T00:13:03.674018+0000","last_fullsized":"2024-03-24T00:13:03.674018+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:13:01.267198+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:13:01.267198+0000","last_clean_scrub_stamp":"2024-03-24T00:13:01.267198+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T03:15:06.007947+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-24T00:13:03.860354+0000","last_change":"2024-03-24T00:13:02.309077+0000","last_active":"2024-03-24T00:13:03.860354+0000","last_peered":"2024-03-24T00:13:03.860354+0000","last_clean":"2024-03-24T00:13:03.860354+0000","last_became_active":"2024-03-24T00:13:02.308752+0000","last_became_peered":"2024-03-24T00:13:02.308752+0000","last_unstale":"2024-03-24T00:13:03.860354+0000","last_undegraded":"2024-03-24T00:13:03.860354+0000","last_fullsized":"2024-03-24T00:13:03.860354+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:13:01.267198+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:13:01.267198+0000","last_clean_scrub_stamp":"2024-03-24T00:13:01.267198+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T06:49:19.657988+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,5],"acting":[11,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-24T00:13:03.290284+0000","last_change":"2024-03-24T00:13:02.304962+0000","last_active":"2024-03-24T00:13:03.290284+0000","last_peered":"2024-03-24T00:13:03.290284+0000","last_clean":"2024-03-24T00:13:03.290284+0000","last_became_active":"2024-03-24T00:13:02.304580+0000","last_became_peered":"2024-03-24T00:13:02.304580+0000","last_unstale":"2024-03-24T00:13:03.290284+0000","last_undegraded":"2024-03-24T00:13:03.290284+0000","last_fullsized":"2024-03-24T00:13:03.290284+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:13:01.267198+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:13:01.267198+0000","last_clean_scrub_stamp":"2024-03-24T00:13:01.267198+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:45:26.779169+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8,1],"acting":[2,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.27","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-24T00:13:04.348365+0000","last_change":"2024-03-24T00:13:02.321908+0000","last_active":"2024-03-24T00:13:04.348365+0000","last_peered":"2024-03-24T00:13:04.348365+0000","last_clean":"2024-03-24T00:13:04.348365+0000","last_became_active":"2024-03-24T00:13:02.321495+0000","last_became_peered":"2024-03-24T00:13:02.321495+0000","last_unstale":"2024-03-24T00:13:04.348365+0000","last_undegraded":"2024-03-24T00:13:04.348365+0000","last_fullsized":"2024-03-24T00:13:04.348365+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:13:01.267198+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:13:01.267198+0000","last_clean_scrub_stamp":"2024-03-24T00:13:01.267198+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T08:43:29.261314+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-24T00:13:03.429452+0000","last_change":"2024-03-24T00:13:02.331966+0000","last_active":"2024-03-24T00:13:03.429452+0000","last_peered":"2024-03-24T00:13:03.429452+0000","last_clean":"2024-03-24T00:13:03.429452+0000","last_became_active":"2024-03-24T00:13:02.331633+0000","last_became_peered":"2024-03-24T00:13:02.331633+0000","last_unstale":"2024-03-24T00:13:03.429452+0000","last_undegraded":"2024-03-24T00:13:03.429452+0000","last_fullsized":"2024-03-24T00:13:03.429452+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:13:01.267198+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:13:01.267198+0000","last_clean_scrub_stamp":"2024-03-24T00:13:01.267198+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T05:04:01.574351+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,3],"acting":[5,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-24T00:13:04.348470+0000","last_change":"2024-03-24T00:13:02.330344+0000","last_active":"2024-03-24T00:13:04.348470+0000","last_peered":"2024-03-24T00:13:04.348470+0000","last_clean":"2024-03-24T00:13:04.348470+0000","last_became_active":"2024-03-24T00:13:02.328218+0000","last_became_peered":"2024-03-24T00:13:02.328218+0000","last_unstale":"2024-03-24T00:13:04.348470+0000","last_undegraded":"2024-03-24T00:13:04.348470+0000","last_fullsized":"2024-03-24T00:13:04.348470+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:13:01.267198+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:13:01.267198+0000","last_clean_scrub_stamp":"2024-03-24T00:13:01.267198+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T08:54:36.392397+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,11],"acting":[3,2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-24T00:13:04.366065+0000","last_change":"2024-03-24T00:13:02.330956+0000","last_active":"2024-03-24T00:13:04.366065+0000","last_peered":"2024-03-24T00:13:04.366065+0000","last_clean":"2024-03-24T00:13:04.366065+0000","last_became_active":"2024-03-24T00:13:02.330642+0000","last_became_peered":"2024-03-24T00:13:02.330642+0000","last_unstale":"2024-03-24T00:13:04.366065+0000","last_undegraded":"2024-03-24T00:13:04.366065+0000","last_fullsized":"2024-03-24T00:13:04.366065+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:13:01.267198+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:13:01.267198+0000","last_clean_scrub_stamp":"2024-03-24T00:13:01.267198+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:41:11.326722+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2b","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-24T00:13:03.673782+0000","last_change":"2024-03-24T00:13:02.333026+0000","last_active":"2024-03-24T00:13:03.673782+0000","last_peered":"2024-03-24T00:13:03.673782+0000","last_clean":"2024-03-24T00:13:03.673782+0000","last_became_active":"2024-03-24T00:13:02.331232+0000","last_became_peered":"2024-03-24T00:13:02.331232+0000","last_unstale":"2024-03-24T00:13:03.673782+0000","last_undegraded":"2024-03-24T00:13:03.673782+0000","last_fullsized":"2024-03-24T00:13:03.673782+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:13:01.267198+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:13:01.267198+0000","last_clean_scrub_stamp":"2024-03-24T00:13:01.267198+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T02:14:18.504012+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2c","version":"0'0","reported_seq":13,"reported_epoch":86,"state":"active+clean","last_fresh":"2024-03-24T00:13:03.860196+0000","last_change":"2024-03-24T00:13:02.309023+0000","last_active":"2024-03-24T00:13:03.860196+0000","last_peered":"2024-03-24T00:13:03.860196+0000","last_clean":"2024-03-24T00:13:03.860196+0000","last_became_active":"2024-03-24T00:13:02.308236+0000","last_became_peered":"2024-03-24T00:13:02.308236+0000","last_unstale":"2024-03-24T00:13:03.860196+0000","last_undegraded":"2024-03-24T00:13:03.860196+0000","last_fullsized":"2024-03-24T00:13:03.860196+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:13:01.267198+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:13:01.267198+0000","last_clean_scrub_stamp":"2024-03-24T00:13:01.267198+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T03:30:37.427095+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,8],"acting":[11,4,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":1,"ondisk_log_size":1,"up":192,"acting":192,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":442368,"data_stored":4722944,"data_compressed":284592,"data_compressed_allocated":360448,"data_compressed_original":4669440,"omap_allocated":0,"internal_metadata":0},"log_size":137,"ondisk_log_size":137,"up":3,"acting":3,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":26336,"num_objects":82,"num_object_clones":0,"num_object_copies":246,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":82,"num_whiteouts":0,"num_read":29,"num_read_kb":23,"num_write":182,"num_write_kb":73,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":55,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":385024,"data_stored":83236,"data_compressed":1719,"data_compressed_allocated":4096,"data_compressed_original":8192,"omap_allocated":0,"internal_metadata":0},"log_size":157,"ondisk_log_size":157,"up":96,"acting":96,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":81,"seq":347892350990,"num_pgs":18,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27928,"kb_used_data":912,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717224,"statfs":{"total":95995035648,"available":95966437376,"internally_reserved":0,"allocated":933888,"data_stored":443734,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1596,"internal_metadata":27654596},"hb_peers":[0,1,2,4,5,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":73,"seq":313532612627,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28016,"kb_used_data":1000,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717136,"statfs":{"total":95995035648,"available":95966347264,"internally_reserved":0,"allocated":1024000,"data_stored":1035833,"data_compressed":36189,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":1599,"internal_metadata":27654593},"hb_peers":[0,1,2,3,4,5,6,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":68,"seq":292057776152,"num_pgs":17,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32096,"kb_used_data":920,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713056,"statfs":{"total":95995035648,"available":95962169344,"internally_reserved":0,"allocated":942080,"data_stored":442969,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1600,"internal_metadata":31914432},"hb_peers":[0,1,2,3,4,5,6,7,8,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":8,"up_from":60,"seq":257698037791,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32096,"kb_used_data":920,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713056,"statfs":{"total":95995035648,"available":95962169344,"internally_reserved":0,"allocated":942080,"data_stored":442889,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[0,1,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":7,"up_from":52,"seq":223338299429,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27988,"kb_used_data":972,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717164,"statfs":{"total":95995035648,"available":95966375936,"internally_reserved":0,"allocated":995328,"data_stored":1034983,"data_compressed":36189,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":47,"seq":201863462955,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27920,"kb_used_data":904,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717232,"statfs":{"total":95995035648,"available":95966445568,"internally_reserved":0,"allocated":925696,"data_stored":442241,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574918,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27932,"kb_used_data":912,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717220,"statfs":{"total":95995035648,"available":95966433280,"internally_reserved":0,"allocated":933888,"data_stored":443763,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738445,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27956,"kb_used_data":940,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717196,"statfs":{"total":95995035648,"available":95966408704,"internally_reserved":0,"allocated":962560,"data_stored":444720,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411394,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27976,"kb_used_data":960,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717176,"statfs":{"total":95995035648,"available":95966388224,"internally_reserved":0,"allocated":983040,"data_stored":465800,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":26,"seq":111669149756,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32092,"kb_used_data":916,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713060,"statfs":{"total":95995035648,"available":95962173440,"internally_reserved":0,"allocated":937984,"data_stored":442435,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":4,"up_from":34,"seq":146028888118,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32096,"kb_used_data":920,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713056,"statfs":{"total":95995035648,"available":95962169344,"internally_reserved":0,"allocated":942080,"data_stored":442851,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":39,"seq":167503724592,"num_pgs":30,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27992,"kb_used_data":976,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717160,"statfs":{"total":95995035648,"available":95966371840,"internally_reserved":0,"allocated":999424,"data_stored":1033995,"data_compressed":36189,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34549,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34549,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34549,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":36189,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":36189,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":36189,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":36189,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":36189,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":2569,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1612,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":81920,"data_stored":46882,"data_compressed":1719,"data_compressed_allocated":4096,"data_compressed_original":8192,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":408,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":700,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":2238,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":112,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":2855,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":20480,"data_stored":738,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":908,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":73728,"data_stored":22631,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1583,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-24T00:13:12.308 INFO:tasks.cephfs.filesystem.ceph_manager:clean! 2024-03-24T00:13:12.309 INFO:tasks.cephadm:Setting up client nodes... 2024-03-24T00:13:12.309 DEBUG:teuthology.orchestra.run.smithi118:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d79a5420-e971-11ee-95cd-87774f69a715 -- ceph auth get-or-create client.0 mon 'allow *' osd 'allow *' mds 'allow *' mgr 'allow *' 2024-03-24T00:13:12.695 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:13:12 smithi118 ceph-mon[31535]: pgmap v319: 97 pgs: 97 active+clean; 602 KiB data, 344 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 7.9 KiB/s wr, 22 op/s 2024-03-24T00:13:12.695 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:13:12 smithi118 ceph-mon[31535]: from='client.24706 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-24T00:13:12.703 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:13:12 smithi189 ceph-mon[36187]: pgmap v319: 97 pgs: 97 active+clean; 602 KiB data, 344 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 7.9 KiB/s wr, 22 op/s 2024-03-24T00:13:12.703 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:13:12 smithi189 ceph-mon[36187]: from='client.24706 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-24T00:13:12.710 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:13:12 smithi193 ceph-mon[36878]: pgmap v319: 97 pgs: 97 active+clean; 602 KiB data, 344 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 7.9 KiB/s wr, 22 op/s 2024-03-24T00:13:12.710 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:13:12 smithi193 ceph-mon[36878]: from='client.24706 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-24T00:13:12.793 INFO:teuthology.orchestra.run.smithi118.stderr:Inferring config /var/lib/ceph/d79a5420-e971-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T00:13:13.695 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:13:13 smithi118 ceph-mon[31535]: from='client.24712 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-24T00:13:13.703 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:13:13 smithi189 ceph-mon[36187]: from='client.24712 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-24T00:13:13.710 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:13:13 smithi193 ceph-mon[36878]: from='client.24712 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-24T00:13:14.596 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:13:14 smithi118 ceph-mon[31535]: pgmap v320: 97 pgs: 97 active+clean; 604 KiB data, 344 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 8.2 KiB/s wr, 23 op/s 2024-03-24T00:13:14.703 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:13:14 smithi189 ceph-mon[36187]: pgmap v320: 97 pgs: 97 active+clean; 604 KiB data, 344 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 8.2 KiB/s wr, 23 op/s 2024-03-24T00:13:14.710 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:13:14 smithi193 ceph-mon[36878]: pgmap v320: 97 pgs: 97 active+clean; 604 KiB data, 344 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 8.2 KiB/s wr, 23 op/s 2024-03-24T00:13:15.179 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:15.178+0000 7f811b200700 1 -- 172.21.15.118:0/2994173264 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f811c1008a0 msgr2=0x7f811c102cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:13:15.179 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:15.178+0000 7f811b200700 1 --2- 172.21.15.118:0/2994173264 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f811c1008a0 0x7f811c102cd0 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f810c009990 tx=0x7f810c009ca0 comp rx=0 tx=0).stop 2024-03-24T00:13:15.179 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:15.178+0000 7f811b200700 1 -- 172.21.15.118:0/2994173264 shutdown_connections 2024-03-24T00:13:15.179 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:15.178+0000 7f811b200700 1 --2- 172.21.15.118:0/2994173264 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f811c105be0 0x7f811c106060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:13:15.179 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:15.178+0000 7f811b200700 1 --2- 172.21.15.118:0/2994173264 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f811c1008a0 0x7f811c102cd0 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:13:15.180 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:15.178+0000 7f811b200700 1 --2- 172.21.15.118:0/2994173264 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f811c103210 0x7f811c1056a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:13:15.180 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:15.178+0000 7f811b200700 1 -- 172.21.15.118:0/2994173264 >> 172.21.15.118:0/2994173264 conn(0x7f811c0fc4c0 msgr2=0x7f811c0fe900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:13:15.180 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:15.178+0000 7f811b200700 1 -- 172.21.15.118:0/2994173264 shutdown_connections 2024-03-24T00:13:15.180 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:15.178+0000 7f811b200700 1 -- 172.21.15.118:0/2994173264 wait complete. 2024-03-24T00:13:15.181 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:15.179+0000 7f811b200700 1 Processor -- start 2024-03-24T00:13:15.181 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:15.179+0000 7f811b200700 1 -- start start 2024-03-24T00:13:15.181 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:15.180+0000 7f811b200700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f811c1008a0 0x7f811c19e700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:13:15.181 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:15.180+0000 7f811b200700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f811c103210 0x7f811c19ec40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:13:15.181 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:15.180+0000 7f811b200700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f811c105be0 0x7f811c1a5cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:13:15.181 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:15.180+0000 7f811b200700 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f811c1141b0 con 0x7f811c1008a0 2024-03-24T00:13:15.182 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:15.180+0000 7f811b200700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f811c114030 con 0x7f811c105be0 2024-03-24T00:13:15.182 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:15.180+0000 7f811b200700 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f811c114330 con 0x7f811c103210 2024-03-24T00:13:15.182 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:15.180+0000 7f811a800700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f811c105be0 0x7f811c1a5cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:13:15.182 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:15.180+0000 7f8119400700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f811c103210 0x7f811c19ec40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:13:15.182 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:15.180+0000 7f811a800700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f811c105be0 0x7f811c1a5cc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.118:43342/0 (socket says 172.21.15.118:43342) 2024-03-24T00:13:15.182 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:15.180+0000 7f811a800700 1 -- 172.21.15.118:0/2880341731 learned_addr learned my addr 172.21.15.118:0/2880341731 (peer_addr_for_me v2:172.21.15.118:0/0) 2024-03-24T00:13:15.182 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:15.180+0000 7f8119e00700 1 --2- 172.21.15.118:0/2880341731 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f811c1008a0 0x7f811c19e700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:13:15.182 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:15.181+0000 7f811a800700 1 -- 172.21.15.118:0/2880341731 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f811c103210 msgr2=0x7f811c19ec40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:13:15.182 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:15.181+0000 7f811a800700 1 --2- 172.21.15.118:0/2880341731 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f811c103210 0x7f811c19ec40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:13:15.182 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:15.181+0000 7f811a800700 1 -- 172.21.15.118:0/2880341731 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f811c1008a0 msgr2=0x7f811c19e700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:13:15.182 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:15.181+0000 7f811a800700 1 --2- 172.21.15.118:0/2880341731 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f811c1008a0 0x7f811c19e700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:13:15.182 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:15.181+0000 7f811a800700 1 -- 172.21.15.118:0/2880341731 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f811c1a6360 con 0x7f811c105be0 2024-03-24T00:13:15.183 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:15.181+0000 7f811a800700 1 --2- 172.21.15.118:0/2880341731 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f811c105be0 0x7f811c1a5cc0 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f811400ef30 tx=0x7f811400c3a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:13:15.183 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:15.181+0000 7f8112a00700 1 -- 172.21.15.118:0/2880341731 <== mon.2 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f81140092e0 con 0x7f811c105be0 2024-03-24T00:13:15.183 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:15.181+0000 7f8112a00700 1 -- 172.21.15.118:0/2880341731 <== mon.2 v2:172.21.15.189:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f8114004bb0 con 0x7f811c105be0 2024-03-24T00:13:15.183 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:15.181+0000 7f811b200700 1 -- 172.21.15.118:0/2880341731 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f811c1a65f0 con 0x7f811c105be0 2024-03-24T00:13:15.184 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:15.181+0000 7f8112a00700 1 -- 172.21.15.118:0/2880341731 <== mon.2 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8114004280 con 0x7f811c105be0 2024-03-24T00:13:15.184 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:15.181+0000 7f811b200700 1 -- 172.21.15.118:0/2880341731 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f811c1a6b30 con 0x7f811c105be0 2024-03-24T00:13:15.184 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:15.183+0000 7f811b200700 1 -- 172.21.15.118:0/2880341731 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f811c0637c0 con 0x7f811c105be0 2024-03-24T00:13:15.185 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:15.184+0000 7f8112a00700 1 -- 172.21.15.118:0/2880341731 <== mon.2 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 17) v1 ==== 137342+0+0 (secure 0 0 0) 0x7f811401b030 con 0x7f811c105be0 2024-03-24T00:13:15.186 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:15.186+0000 7f8112a00700 1 --2- 172.21.15.118:0/2880341731 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7f81000a2a50 0x7f81000a4f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:13:15.186 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:15.186+0000 7f8112a00700 1 -- 172.21.15.118:0/2880341731 <== mon.2 v2:172.21.15.189:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 7979+0+0 (secure 0 0 0) 0x7f81140ce630 con 0x7f811c105be0 2024-03-24T00:13:15.187 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:15.186+0000 7f8119e00700 1 --2- 172.21.15.118:0/2880341731 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7f81000a2a50 0x7f81000a4f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:13:15.187 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:15.187+0000 7f8119e00700 1 --2- 172.21.15.118:0/2880341731 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7f81000a2a50 0x7f81000a4f10 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7f810c006010 tx=0x7f810c01a040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:13:15.195 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:15.194+0000 7f8112a00700 1 -- 172.21.15.118:0/2880341731 <== mon.2 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f8114022020 con 0x7f811c105be0 2024-03-24T00:13:15.524 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:15.523+0000 7f811b200700 1 -- 172.21.15.118:0/2880341731 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]} v 0) v1 -- 0x7f811c02dad0 con 0x7f811c105be0 2024-03-24T00:13:15.532 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:15.531+0000 7f8112a00700 1 -- 172.21.15.118:0/2880341731 <== mon.2 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]=0 v32) v1 ==== 170+0+59 (secure 0 0 0) 0x7f811c02dad0 con 0x7f811c105be0 2024-03-24T00:13:15.532 INFO:teuthology.orchestra.run.smithi118.stdout:[client.0] 2024-03-24T00:13:15.532 INFO:teuthology.orchestra.run.smithi118.stdout: key = AQAbcP9l7mxhHxAAGOaP7OFiB4BINcLnK+6INw== 2024-03-24T00:13:15.535 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:15.534+0000 7f811b200700 1 -- 172.21.15.118:0/2880341731 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7f81000a2a50 msgr2=0x7f81000a4f10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:13:15.535 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:15.534+0000 7f811b200700 1 --2- 172.21.15.118:0/2880341731 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7f81000a2a50 0x7f81000a4f10 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7f810c006010 tx=0x7f810c01a040 comp rx=0 tx=0).stop 2024-03-24T00:13:15.535 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:15.534+0000 7f811b200700 1 -- 172.21.15.118:0/2880341731 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f811c105be0 msgr2=0x7f811c1a5cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:13:15.535 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:15.534+0000 7f811b200700 1 --2- 172.21.15.118:0/2880341731 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f811c105be0 0x7f811c1a5cc0 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f811400ef30 tx=0x7f811400c3a0 comp rx=0 tx=0).stop 2024-03-24T00:13:15.535 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:15.534+0000 7f811b200700 1 -- 172.21.15.118:0/2880341731 shutdown_connections 2024-03-24T00:13:15.535 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:15.534+0000 7f811b200700 1 --2- 172.21.15.118:0/2880341731 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f811c105be0 0x7f811c1a5cc0 unknown :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:13:15.535 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:15.534+0000 7f811b200700 1 --2- 172.21.15.118:0/2880341731 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7f81000a2a50 0x7f81000a4f10 unknown :-1 s=CLOSED pgs=177 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:13:15.535 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:15.534+0000 7f811b200700 1 --2- 172.21.15.118:0/2880341731 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f811c1008a0 0x7f811c19e700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:13:15.535 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:15.534+0000 7f811b200700 1 --2- 172.21.15.118:0/2880341731 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f811c103210 0x7f811c19ec40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:13:15.536 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:15.534+0000 7f811b200700 1 -- 172.21.15.118:0/2880341731 >> 172.21.15.118:0/2880341731 conn(0x7f811c0fc4c0 msgr2=0x7f811c103f20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:13:15.536 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:15.534+0000 7f811b200700 1 -- 172.21.15.118:0/2880341731 shutdown_connections 2024-03-24T00:13:15.536 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:15.535+0000 7f811b200700 1 -- 172.21.15.118:0/2880341731 wait complete. 2024-03-24T00:13:15.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:13:15 smithi118 ceph-mon[31535]: from='client.? 172.21.15.118:0/2880341731' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-03-24T00:13:15.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:13:15 smithi118 ceph-mon[31535]: from='client.? ' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-03-24T00:13:15.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:13:15 smithi118 ceph-mon[31535]: from='client.? ' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-03-24T00:13:15.953 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:13:15 smithi189 ceph-mon[36187]: from='client.? 172.21.15.118:0/2880341731' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-03-24T00:13:15.953 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:13:15 smithi189 ceph-mon[36187]: from='client.? ' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-03-24T00:13:15.953 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:13:15 smithi189 ceph-mon[36187]: from='client.? ' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-03-24T00:13:15.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:13:15 smithi193 ceph-mon[36878]: from='client.? 172.21.15.118:0/2880341731' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-03-24T00:13:15.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:13:15 smithi193 ceph-mon[36878]: from='client.? ' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-03-24T00:13:15.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:13:15 smithi193 ceph-mon[36878]: from='client.? ' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-03-24T00:13:16.484 DEBUG:teuthology.orchestra.run.smithi118:> set -ex 2024-03-24T00:13:16.484 DEBUG:teuthology.orchestra.run.smithi118:> sudo dd of=/etc/ceph/ceph.client.0.keyring 2024-03-24T00:13:16.484 DEBUG:teuthology.orchestra.run.smithi118:> sudo chmod 0644 /etc/ceph/ceph.client.0.keyring 2024-03-24T00:13:16.640 INFO:tasks.ceph:Waiting until ceph daemons up and pgs clean... 2024-03-24T00:13:16.640 INFO:tasks.cephadm.ceph_manager.ceph:waiting for mgr available 2024-03-24T00:13:16.640 DEBUG:teuthology.orchestra.run.smithi118:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 shell --fsid d79a5420-e971-11ee-95cd-87774f69a715 -- ceph mgr dump --format=json 2024-03-24T00:13:16.695 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:13:16 smithi118 ceph-mon[31535]: pgmap v321: 97 pgs: 97 active+clean; 632 KiB data, 344 MiB used, 1.0 TiB / 1.0 TiB avail; 9.0 KiB/s rd, 9.6 KiB/s wr, 24 op/s 2024-03-24T00:13:16.953 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:13:16 smithi189 ceph-mon[36187]: pgmap v321: 97 pgs: 97 active+clean; 632 KiB data, 344 MiB used, 1.0 TiB / 1.0 TiB avail; 9.0 KiB/s rd, 9.6 KiB/s wr, 24 op/s 2024-03-24T00:13:16.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:13:16 smithi193 ceph-mon[36878]: pgmap v321: 97 pgs: 97 active+clean; 632 KiB data, 344 MiB used, 1.0 TiB / 1.0 TiB avail; 9.0 KiB/s rd, 9.6 KiB/s wr, 24 op/s 2024-03-24T00:13:17.145 INFO:teuthology.orchestra.run.smithi118.stderr:Inferring config /var/lib/ceph/d79a5420-e971-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T00:13:18.945 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:13:18 smithi118 ceph-mon[31535]: pgmap v322: 97 pgs: 97 active+clean; 632 KiB data, 344 MiB used, 1.0 TiB / 1.0 TiB avail; 9.4 KiB/s rd, 8.7 KiB/s wr, 23 op/s 2024-03-24T00:13:18.953 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:13:18 smithi189 ceph-mon[36187]: pgmap v322: 97 pgs: 97 active+clean; 632 KiB data, 344 MiB used, 1.0 TiB / 1.0 TiB avail; 9.4 KiB/s rd, 8.7 KiB/s wr, 23 op/s 2024-03-24T00:13:18.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:13:18 smithi193 ceph-mon[36878]: pgmap v322: 97 pgs: 97 active+clean; 632 KiB data, 344 MiB used, 1.0 TiB / 1.0 TiB avail; 9.4 KiB/s rd, 8.7 KiB/s wr, 23 op/s 2024-03-24T00:13:19.530 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:19.529+0000 7efebbe00700 1 -- 172.21.15.118:0/1757980337 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7efebc10e8b0 msgr2=0x7efebc110d20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:13:19.530 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:19.529+0000 7efebbe00700 1 --2- 172.21.15.118:0/1757980337 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7efebc10e8b0 0x7efebc110d20 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7efeb4009f90 tx=0x7efeb400b330 comp rx=0 tx=0).stop 2024-03-24T00:13:19.531 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:19.529+0000 7efebbe00700 1 -- 172.21.15.118:0/1757980337 shutdown_connections 2024-03-24T00:13:19.531 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:19.529+0000 7efebbe00700 1 --2- 172.21.15.118:0/1757980337 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7efebc0745c0 0x7efebc0749a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:13:19.531 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:19.529+0000 7efebbe00700 1 --2- 172.21.15.118:0/1757980337 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7efebc10e8b0 0x7efebc110d20 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:13:19.531 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:19.529+0000 7efebbe00700 1 --2- 172.21.15.118:0/1757980337 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7efebc074ee0 0x7efebc10e370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:13:19.531 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:19.529+0000 7efebbe00700 1 -- 172.21.15.118:0/1757980337 >> 172.21.15.118:0/1757980337 conn(0x7efebc0fdb60 msgr2=0x7efebc0fff80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:13:19.531 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:19.529+0000 7efebbe00700 1 -- 172.21.15.118:0/1757980337 shutdown_connections 2024-03-24T00:13:19.531 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:19.529+0000 7efebbe00700 1 -- 172.21.15.118:0/1757980337 wait complete. 2024-03-24T00:13:19.532 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:19.531+0000 7efebbe00700 1 Processor -- start 2024-03-24T00:13:19.532 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:19.531+0000 7efebbe00700 1 -- start start 2024-03-24T00:13:19.533 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:19.531+0000 7efebbe00700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7efebc0745c0 0x7efebc19aa60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:13:19.533 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:19.531+0000 7efebbe00700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7efebc074ee0 0x7efebc19afa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:13:19.533 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:19.531+0000 7efebbe00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7efebc10e8b0 0x7efebc19f360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:13:19.533 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:19.531+0000 7efebbe00700 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efebc1145d0 con 0x7efebc0745c0 2024-03-24T00:13:19.533 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:19.531+0000 7efebbe00700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efebc114450 con 0x7efebc10e8b0 2024-03-24T00:13:19.533 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:19.531+0000 7efebbe00700 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efebc114750 con 0x7efebc074ee0 2024-03-24T00:13:19.533 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:19.532+0000 7efebaa00700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7efebc0745c0 0x7efebc19aa60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:13:19.533 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:19.532+0000 7efeba000700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7efebc074ee0 0x7efebc19afa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:13:19.533 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:19.532+0000 7efeba000700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7efebc074ee0 0x7efebc19afa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.118:53936/0 (socket says 172.21.15.118:53936) 2024-03-24T00:13:19.533 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:19.532+0000 7efeba000700 1 -- 172.21.15.118:0/1470171871 learned_addr learned my addr 172.21.15.118:0/1470171871 (peer_addr_for_me v2:172.21.15.118:0/0) 2024-03-24T00:13:19.533 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:19.532+0000 7efebb400700 1 --2- 172.21.15.118:0/1470171871 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7efebc10e8b0 0x7efebc19f360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:13:19.533 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:19.532+0000 7efeba000700 1 -- 172.21.15.118:0/1470171871 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7efebc10e8b0 msgr2=0x7efebc19f360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:13:19.534 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:19.532+0000 7efeba000700 1 --2- 172.21.15.118:0/1470171871 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7efebc10e8b0 0x7efebc19f360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:13:19.534 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:19.532+0000 7efeba000700 1 -- 172.21.15.118:0/1470171871 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7efebc0745c0 msgr2=0x7efebc19aa60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:13:19.534 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:19.532+0000 7efeba000700 1 --2- 172.21.15.118:0/1470171871 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7efebc0745c0 0x7efebc19aa60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:13:19.534 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:19.532+0000 7efeba000700 1 -- 172.21.15.118:0/1470171871 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efebc19fa60 con 0x7efebc074ee0 2024-03-24T00:13:19.535 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:19.532+0000 7efeba000700 1 --2- 172.21.15.118:0/1470171871 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7efebc074ee0 0x7efebc19afa0 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7efea800e9a0 tx=0x7efea800ecb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:13:19.535 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:19.532+0000 7efeb3400700 1 -- 172.21.15.118:0/1470171871 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efea800ca10 con 0x7efebc074ee0 2024-03-24T00:13:19.535 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:19.533+0000 7efeb3400700 1 -- 172.21.15.118:0/1470171871 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7efea800cbb0 con 0x7efebc074ee0 2024-03-24T00:13:19.535 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:19.533+0000 7efeb3400700 1 -- 172.21.15.118:0/1470171871 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efea8018680 con 0x7efebc074ee0 2024-03-24T00:13:19.535 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:19.533+0000 7efebbe00700 1 -- 172.21.15.118:0/1470171871 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efebc19fd50 con 0x7efebc074ee0 2024-03-24T00:13:19.535 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:19.533+0000 7efebbe00700 1 -- 172.21.15.118:0/1470171871 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efebc1a7610 con 0x7efebc074ee0 2024-03-24T00:13:19.535 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:19.534+0000 7efebbe00700 1 -- 172.21.15.118:0/1470171871 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efebc068e50 con 0x7efebc074ee0 2024-03-24T00:13:19.536 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:19.536+0000 7efeb3400700 1 -- 172.21.15.118:0/1470171871 <== mon.1 v2:172.21.15.193:3300/0 4 ==== mgrmap(e 17) v1 ==== 137342+0+0 (secure 0 0 0) 0x7efea8047a90 con 0x7efebc074ee0 2024-03-24T00:13:19.537 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:19.537+0000 7efeb3400700 1 --2- 172.21.15.118:0/1470171871 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7efea00a2920 0x7efea00a4de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:13:19.538 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:19.537+0000 7efeb3400700 1 -- 172.21.15.118:0/1470171871 <== mon.1 v2:172.21.15.193:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 7979+0+0 (secure 0 0 0) 0x7efea80cecf0 con 0x7efebc074ee0 2024-03-24T00:13:19.538 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:19.537+0000 7efebaa00700 1 --2- 172.21.15.118:0/1470171871 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7efea00a2920 0x7efea00a4de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:13:19.538 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:19.538+0000 7efebaa00700 1 --2- 172.21.15.118:0/1470171871 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7efea00a2920 0x7efea00a4de0 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7efeac009750 tx=0x7efeac009310 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:13:19.540 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:19.539+0000 7efeb3400700 1 -- 172.21.15.118:0/1470171871 <== mon.1 v2:172.21.15.193:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7efea809bce0 con 0x7efebc074ee0 2024-03-24T00:13:19.826 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:19.825+0000 7efebbe00700 1 -- 172.21.15.118:0/1470171871 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_command({"prefix": "mgr dump", "format": "json"} v 0) v1 -- 0x7efebc1a77e0 con 0x7efebc074ee0 2024-03-24T00:13:19.830 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:19.830+0000 7efeb3400700 1 -- 172.21.15.118:0/1470171871 <== mon.1 v2:172.21.15.193:3300/0 7 ==== mon_command_ack([{"prefix": "mgr dump", "format": "json"}]=0 v17) v1 ==== 74+0+262668 (secure 0 0 0) 0x7efea809b5e0 con 0x7efebc074ee0 2024-03-24T00:13:19.832 INFO:teuthology.orchestra.run.smithi118.stdout: 2024-03-24T00:13:19.837 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:19.837+0000 7efebbe00700 1 -- 172.21.15.118:0/1470171871 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7efea00a2920 msgr2=0x7efea00a4de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:13:19.837 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:19.837+0000 7efebbe00700 1 --2- 172.21.15.118:0/1470171871 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7efea00a2920 0x7efea00a4de0 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7efeac009750 tx=0x7efeac009310 comp rx=0 tx=0).stop 2024-03-24T00:13:19.837 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:19.837+0000 7efebbe00700 1 -- 172.21.15.118:0/1470171871 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7efebc074ee0 msgr2=0x7efebc19afa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:13:19.837 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:19.837+0000 7efebbe00700 1 --2- 172.21.15.118:0/1470171871 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7efebc074ee0 0x7efebc19afa0 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7efea800e9a0 tx=0x7efea800ecb0 comp rx=0 tx=0).stop 2024-03-24T00:13:19.838 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:19.837+0000 7efebbe00700 1 -- 172.21.15.118:0/1470171871 shutdown_connections 2024-03-24T00:13:19.838 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:19.837+0000 7efebbe00700 1 --2- 172.21.15.118:0/1470171871 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7efebc10e8b0 0x7efebc19f360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:13:19.838 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:19.837+0000 7efebbe00700 1 --2- 172.21.15.118:0/1470171871 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7efea00a2920 0x7efea00a4de0 unknown :-1 s=CLOSED pgs=178 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:13:19.838 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:19.837+0000 7efebbe00700 1 --2- 172.21.15.118:0/1470171871 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7efebc0745c0 0x7efebc19aa60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:13:19.838 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:19.837+0000 7efebbe00700 1 --2- 172.21.15.118:0/1470171871 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7efebc074ee0 0x7efebc19afa0 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:13:19.839 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:19.837+0000 7efebbe00700 1 -- 172.21.15.118:0/1470171871 >> 172.21.15.118:0/1470171871 conn(0x7efebc0fdb60 msgr2=0x7efebc110b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:13:19.839 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:19.837+0000 7efebbe00700 1 -- 172.21.15.118:0/1470171871 shutdown_connections 2024-03-24T00:13:19.839 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:19.837+0000 7efebbe00700 1 -- 172.21.15.118:0/1470171871 wait complete. 2024-03-24T00:13:20.690 INFO:teuthology.orchestra.run.smithi118.stdout:{"epoch":17,"flags":0,"active_gid":14152,"active_name":"x","active_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6800","nonce":2112912943},{"type":"v1","addr":"172.21.15.118:6801","nonce":2112912943}]},"active_addr":"172.21.15.118:6801/2112912943","active_change":"2024-03-24T00:05:05.459020+0000","active_mgr_features":4540701547738038271,"available":true,"standbys":[{"gid":24112,"name":"z","mgr_features":4540701547738038271,"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","read","upmap","upmap-read"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:9.4.12","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:1.0.0","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"MULTICLUSTER_CONFIG":{"name":"MULTICLUSTER_CONFIG","type":"str","level":"advanced","flags":0,"default_value":"{}","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"k8sevents","can_run":true,"error_string":"","module_options":{"ceph_event_retention_days":{"name":"ceph_event_retention_days","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"Days to hold ceph event information within local cache","long_desc":"","tags":[],"see_also":[]},"config_check_secs":{"name":"config_check_secs","type":"int","level":"advanced","flags":0,"default_value":"10","min":"10","max":"","enum_allowed":[],"desc":"interval (secs) to check for cluster configuration changes","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_no_wait":{"name":"snapshot_clone_no_wait","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Reject subvolume clone request when cloner threads are busy","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}]},{"gid":24116,"name":"y","mgr_features":4540701547738038271,"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","read","upmap","upmap-read"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:9.4.12","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:1.0.0","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"MULTICLUSTER_CONFIG":{"name":"MULTICLUSTER_CONFIG","type":"str","level":"advanced","flags":0,"default_value":"{}","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"k8sevents","can_run":true,"error_string":"","module_options":{"ceph_event_retention_days":{"name":"ceph_event_retention_days","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"Days to hold ceph event information within local cache","long_desc":"","tags":[],"see_also":[]},"config_check_secs":{"name":"config_check_secs","type":"int","level":"advanced","flags":0,"default_value":"10","min":"10","max":"","enum_allowed":[],"desc":"interval (secs) to check for cluster configuration changes","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_no_wait":{"name":"snapshot_clone_no_wait","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Reject subvolume clone request when cloner threads are busy","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}]}],"modules":["cephadm","dashboard","iostat","nfs","restful"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","read","upmap","upmap-read"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:9.4.12","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:1.0.0","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"MULTICLUSTER_CONFIG":{"name":"MULTICLUSTER_CONFIG","type":"str","level":"advanced","flags":0,"default_value":"{}","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"k8sevents","can_run":true,"error_string":"","module_options":{"ceph_event_retention_days":{"name":"ceph_event_retention_days","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"Days to hold ceph event information within local cache","long_desc":"","tags":[],"see_also":[]},"config_check_secs":{"name":"config_check_secs","type":"int","level":"advanced","flags":0,"default_value":"10","min":"10","max":"","enum_allowed":[],"desc":"interval (secs) to check for cluster configuration changes","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_no_wait":{"name":"snapshot_clone_no_wait","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Reject subvolume clone request when cloner threads are busy","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{"dashboard":"https://172.21.15.118:8443/"},"always_on_modules":{"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"quincy":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"reef":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"squid":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"]},"last_failure_osd_epoch":3,"active_clients":[{"name":"devicehealth","addrvec":[{"type":"v2","addr":"172.21.15.118:0","nonce":1128391920}]},{"name":"libcephsqlite","addrvec":[{"type":"v2","addr":"172.21.15.118:0","nonce":533985636}]},{"name":"rbd_support","addrvec":[{"type":"v2","addr":"172.21.15.118:0","nonce":1198929553}]},{"name":"volumes","addrvec":[{"type":"v2","addr":"172.21.15.118:0","nonce":2229730490}]},{"name":"volumes","addrvec":[{"type":"v2","addr":"172.21.15.118:0","nonce":857617396}]}]} 2024-03-24T00:13:20.695 INFO:tasks.cephadm.ceph_manager.ceph:mgr available! 2024-03-24T00:13:20.695 INFO:tasks.cephadm.ceph_manager.ceph:waiting for all up 2024-03-24T00:13:20.696 DEBUG:teuthology.orchestra.run.smithi118:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 shell --fsid d79a5420-e971-11ee-95cd-87774f69a715 -- ceph osd dump --format=json 2024-03-24T00:13:20.945 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:13:20 smithi118 ceph-mon[31535]: pgmap v323: 97 pgs: 97 active+clean; 636 KiB data, 344 MiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 8.2 KiB/s wr, 24 op/s 2024-03-24T00:13:20.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:13:20 smithi118 ceph-mon[31535]: from='client.? 172.21.15.118:0/1470171871' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2024-03-24T00:13:20.953 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:13:20 smithi189 ceph-mon[36187]: pgmap v323: 97 pgs: 97 active+clean; 636 KiB data, 344 MiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 8.2 KiB/s wr, 24 op/s 2024-03-24T00:13:20.953 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:13:20 smithi189 ceph-mon[36187]: from='client.? 172.21.15.118:0/1470171871' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2024-03-24T00:13:20.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:13:20 smithi193 ceph-mon[36878]: pgmap v323: 97 pgs: 97 active+clean; 636 KiB data, 344 MiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 8.2 KiB/s wr, 24 op/s 2024-03-24T00:13:20.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:13:20 smithi193 ceph-mon[36878]: from='client.? 172.21.15.118:0/1470171871' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2024-03-24T00:13:21.220 INFO:teuthology.orchestra.run.smithi118.stderr:Inferring config /var/lib/ceph/d79a5420-e971-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T00:13:22.934 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:13:22 smithi118 ceph-mon[31535]: pgmap v324: 97 pgs: 97 active+clean; 636 KiB data, 344 MiB used, 1.0 TiB / 1.0 TiB avail; 9.8 KiB/s rd, 4.6 KiB/s wr, 15 op/s 2024-03-24T00:13:22.953 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:13:22 smithi189 ceph-mon[36187]: pgmap v324: 97 pgs: 97 active+clean; 636 KiB data, 344 MiB used, 1.0 TiB / 1.0 TiB avail; 9.8 KiB/s rd, 4.6 KiB/s wr, 15 op/s 2024-03-24T00:13:22.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:13:22 smithi193 ceph-mon[36878]: pgmap v324: 97 pgs: 97 active+clean; 636 KiB data, 344 MiB used, 1.0 TiB / 1.0 TiB avail; 9.8 KiB/s rd, 4.6 KiB/s wr, 15 op/s 2024-03-24T00:13:23.450 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:23.449+0000 7f3862400700 1 -- 172.21.15.118:0/701054154 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f385c1070d0 msgr2=0x7f385c1074b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:13:23.450 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:23.449+0000 7f3862400700 1 --2- 172.21.15.118:0/701054154 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f385c1070d0 0x7f385c1074b0 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f384c0098f0 tx=0x7f384c009c00 comp rx=0 tx=0).stop 2024-03-24T00:13:23.450 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:23.449+0000 7f3862400700 1 -- 172.21.15.118:0/701054154 shutdown_connections 2024-03-24T00:13:23.451 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:23.449+0000 7f3862400700 1 --2- 172.21.15.118:0/701054154 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f385c101a30 0x7f385c10df10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:13:23.451 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:23.449+0000 7f3862400700 1 --2- 172.21.15.118:0/701054154 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f385c1070d0 0x7f385c1074b0 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:13:23.451 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:23.449+0000 7f3862400700 1 --2- 172.21.15.118:0/701054154 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f385c101090 0x7f385c1014f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:13:23.451 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:23.449+0000 7f3862400700 1 -- 172.21.15.118:0/701054154 >> 172.21.15.118:0/701054154 conn(0x7f385c077fc0 msgr2=0x7f385c0783d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:13:23.451 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:23.449+0000 7f3862400700 1 -- 172.21.15.118:0/701054154 shutdown_connections 2024-03-24T00:13:23.451 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:23.449+0000 7f3862400700 1 -- 172.21.15.118:0/701054154 wait complete. 2024-03-24T00:13:23.452 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:23.450+0000 7f3862400700 1 Processor -- start 2024-03-24T00:13:23.452 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:23.451+0000 7f3862400700 1 -- start start 2024-03-24T00:13:23.452 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:23.451+0000 7f3862400700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f385c101090 0x7f385c1a07b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:13:23.452 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:23.451+0000 7f3862400700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f385c101a30 0x7f385c1a0cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:13:23.452 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:23.451+0000 7f3862400700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f385c1070d0 0x7f385c19a970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:13:23.453 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:23.451+0000 7f3862400700 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f385c06a5a0 con 0x7f385c101090 2024-03-24T00:13:23.453 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:23.451+0000 7f3862400700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f385c06a420 con 0x7f385c1070d0 2024-03-24T00:13:23.453 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:23.451+0000 7f3862400700 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f385c06a720 con 0x7f385c101a30 2024-03-24T00:13:23.453 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:23.451+0000 7f385aa00700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f385c101a30 0x7f385c1a0cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:13:23.453 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:23.451+0000 7f385aa00700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f385c101a30 0x7f385c1a0cf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.118:50408/0 (socket says 172.21.15.118:50408) 2024-03-24T00:13:23.453 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:23.451+0000 7f385b400700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f385c101090 0x7f385c1a07b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:13:23.453 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:23.451+0000 7f385aa00700 1 -- 172.21.15.118:0/854968645 learned_addr learned my addr 172.21.15.118:0/854968645 (peer_addr_for_me v2:172.21.15.118:0/0) 2024-03-24T00:13:23.453 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:23.451+0000 7f385be00700 1 --2- 172.21.15.118:0/854968645 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f385c1070d0 0x7f385c19a970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:13:23.453 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:23.452+0000 7f385aa00700 1 -- 172.21.15.118:0/854968645 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f385c1070d0 msgr2=0x7f385c19a970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:13:23.453 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:23.452+0000 7f385aa00700 1 --2- 172.21.15.118:0/854968645 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f385c1070d0 0x7f385c19a970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:13:23.453 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:23.452+0000 7f385aa00700 1 -- 172.21.15.118:0/854968645 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f385c101090 msgr2=0x7f385c1a07b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:13:23.453 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:23.452+0000 7f385aa00700 1 --2- 172.21.15.118:0/854968645 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f385c101090 0x7f385c1a07b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:13:23.453 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:23.452+0000 7f385aa00700 1 -- 172.21.15.118:0/854968645 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f385c19b0d0 con 0x7f385c101a30 2024-03-24T00:13:23.454 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:23.452+0000 7f385aa00700 1 --2- 172.21.15.118:0/854968645 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f385c101a30 0x7f385c1a0cf0 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f384800eb30 tx=0x7f384800ee40 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:13:23.454 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:23.452+0000 7f3853e00700 1 -- 172.21.15.118:0/854968645 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f384800cbd0 con 0x7f385c101a30 2024-03-24T00:13:23.454 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:23.452+0000 7f3853e00700 1 -- 172.21.15.118:0/854968645 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f3848004500 con 0x7f385c101a30 2024-03-24T00:13:23.454 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:23.452+0000 7f3853e00700 1 -- 172.21.15.118:0/854968645 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3848010430 con 0x7f385c101a30 2024-03-24T00:13:23.455 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:23.452+0000 7f3862400700 1 -- 172.21.15.118:0/854968645 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f385c19b3c0 con 0x7f385c101a30 2024-03-24T00:13:23.455 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:23.452+0000 7f3862400700 1 -- 172.21.15.118:0/854968645 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f385c1a7680 con 0x7f385c101a30 2024-03-24T00:13:23.455 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:23.453+0000 7f3862400700 1 -- 172.21.15.118:0/854968645 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f385c068e50 con 0x7f385c101a30 2024-03-24T00:13:23.456 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:23.455+0000 7f3853e00700 1 -- 172.21.15.118:0/854968645 <== mon.1 v2:172.21.15.193:3300/0 4 ==== mgrmap(e 17) v1 ==== 137342+0+0 (secure 0 0 0) 0x7f3848003f00 con 0x7f385c101a30 2024-03-24T00:13:23.457 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:23.457+0000 7f3853e00700 1 --2- 172.21.15.118:0/854968645 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7f38400a29a0 0x7f38400a4e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:13:23.457 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:23.457+0000 7f3853e00700 1 -- 172.21.15.118:0/854968645 <== mon.1 v2:172.21.15.193:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 7979+0+0 (secure 0 0 0) 0x7f38480cedf0 con 0x7f385c101a30 2024-03-24T00:13:23.457 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:23.457+0000 7f385b400700 1 --2- 172.21.15.118:0/854968645 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7f38400a29a0 0x7f38400a4e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:13:23.458 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:23.457+0000 7f385b400700 1 --2- 172.21.15.118:0/854968645 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7f38400a29a0 0x7f38400a4e60 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7f384c0098c0 tx=0x7f384c019040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:13:23.461 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:23.460+0000 7f3853e00700 1 -- 172.21.15.118:0/854968645 <== mon.1 v2:172.21.15.193:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f384809bde0 con 0x7f385c101a30 2024-03-24T00:13:23.685 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:23.684+0000 7f3862400700 1 -- 172.21.15.118:0/854968645 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f385c19bea0 con 0x7f385c101a30 2024-03-24T00:13:23.686 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:23.685+0000 7f3853e00700 1 -- 172.21.15.118:0/854968645 <== mon.1 v2:172.21.15.193:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v86) v1 ==== 74+0+22287 (secure 0 0 0) 0x7f385c19bea0 con 0x7f385c101a30 2024-03-24T00:13:23.687 INFO:teuthology.orchestra.run.smithi118.stdout: 2024-03-24T00:13:23.687 INFO:teuthology.orchestra.run.smithi118.stdout:{"epoch":86,"fsid":"d79a5420-e971-11ee-95cd-87774f69a715","created":"2024-03-24T00:03:58.648524+0000","modified":"2024-03-24T00:13:03.282705+0000","last_up_change":"2024-03-24T00:12:03.809000+0000","last_in_change":"2024-03-24T00:11:44.389344+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-24T00:07:50.190965+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-24T00:12:59.993723+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-24T00:13:01.257347+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"86","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"c9590426-3901-4f68-a5ef-d57aa8fbac92","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6802","nonce":557690061},{"type":"v1","addr":"172.21.15.118:6803","nonce":557690061}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6804","nonce":557690061},{"type":"v1","addr":"172.21.15.118:6805","nonce":557690061}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6808","nonce":557690061},{"type":"v1","addr":"172.21.15.118:6809","nonce":557690061}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6806","nonce":557690061},{"type":"v1","addr":"172.21.15.118:6807","nonce":557690061}]},"public_addr":"172.21.15.118:6803/557690061","cluster_addr":"172.21.15.118:6805/557690061","heartbeat_back_addr":"172.21.15.118:6809/557690061","heartbeat_front_addr":"172.21.15.118:6807/557690061","state":["exists","up"]},{"osd":1,"uuid":"fc7f827e-ecf6-4661-aad0-7a22951892e0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6800","nonce":4203029485},{"type":"v1","addr":"172.21.15.189:6801","nonce":4203029485}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6802","nonce":4203029485},{"type":"v1","addr":"172.21.15.189:6803","nonce":4203029485}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6806","nonce":4203029485},{"type":"v1","addr":"172.21.15.189:6807","nonce":4203029485}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6804","nonce":4203029485},{"type":"v1","addr":"172.21.15.189:6805","nonce":4203029485}]},"public_addr":"172.21.15.189:6801/4203029485","cluster_addr":"172.21.15.189:6803/4203029485","heartbeat_back_addr":"172.21.15.189:6807/4203029485","heartbeat_front_addr":"172.21.15.189:6805/4203029485","state":["exists","up"]},{"osd":2,"uuid":"e73ccfe9-0e5b-4734-a154-a2f0d829fcfb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6800","nonce":3699076971},{"type":"v1","addr":"172.21.15.193:6801","nonce":3699076971}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6802","nonce":3699076971},{"type":"v1","addr":"172.21.15.193:6803","nonce":3699076971}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6806","nonce":3699076971},{"type":"v1","addr":"172.21.15.193:6807","nonce":3699076971}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6804","nonce":3699076971},{"type":"v1","addr":"172.21.15.193:6805","nonce":3699076971}]},"public_addr":"172.21.15.193:6801/3699076971","cluster_addr":"172.21.15.193:6803/3699076971","heartbeat_back_addr":"172.21.15.193:6807/3699076971","heartbeat_front_addr":"172.21.15.193:6805/3699076971","state":["exists","up"]},{"osd":3,"uuid":"63fed43b-43ce-4031-8957-7fe94725c1ed","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6810","nonce":4059617010},{"type":"v1","addr":"172.21.15.118:6811","nonce":4059617010}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6812","nonce":4059617010},{"type":"v1","addr":"172.21.15.118:6813","nonce":4059617010}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6816","nonce":4059617010},{"type":"v1","addr":"172.21.15.118:6817","nonce":4059617010}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6814","nonce":4059617010},{"type":"v1","addr":"172.21.15.118:6815","nonce":4059617010}]},"public_addr":"172.21.15.118:6811/4059617010","cluster_addr":"172.21.15.118:6813/4059617010","heartbeat_back_addr":"172.21.15.118:6817/4059617010","heartbeat_front_addr":"172.21.15.118:6815/4059617010","state":["exists","up"]},{"osd":4,"uuid":"57377f66-c3b4-43c1-8e8e-bc873672906d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":34,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6808","nonce":4075247068},{"type":"v1","addr":"172.21.15.189:6809","nonce":4075247068}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6810","nonce":4075247068},{"type":"v1","addr":"172.21.15.189:6811","nonce":4075247068}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6814","nonce":4075247068},{"type":"v1","addr":"172.21.15.189:6815","nonce":4075247068}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6812","nonce":4075247068},{"type":"v1","addr":"172.21.15.189:6813","nonce":4075247068}]},"public_addr":"172.21.15.189:6809/4075247068","cluster_addr":"172.21.15.189:6811/4075247068","heartbeat_back_addr":"172.21.15.189:6815/4075247068","heartbeat_front_addr":"172.21.15.189:6813/4075247068","state":["exists","up"]},{"osd":5,"uuid":"91970976-2193-4224-81c0-dc96cb022c85","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":39,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6808","nonce":2920363402},{"type":"v1","addr":"172.21.15.193:6809","nonce":2920363402}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6810","nonce":2920363402},{"type":"v1","addr":"172.21.15.193:6811","nonce":2920363402}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6814","nonce":2920363402},{"type":"v1","addr":"172.21.15.193:6815","nonce":2920363402}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6812","nonce":2920363402},{"type":"v1","addr":"172.21.15.193:6813","nonce":2920363402}]},"public_addr":"172.21.15.193:6809/2920363402","cluster_addr":"172.21.15.193:6811/2920363402","heartbeat_back_addr":"172.21.15.193:6815/2920363402","heartbeat_front_addr":"172.21.15.193:6813/2920363402","state":["exists","up"]},{"osd":6,"uuid":"0b77b196-8674-41c6-ad3d-726b5d01c8fd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":47,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6818","nonce":431354576},{"type":"v1","addr":"172.21.15.118:6819","nonce":431354576}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6820","nonce":431354576},{"type":"v1","addr":"172.21.15.118:6821","nonce":431354576}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6824","nonce":431354576},{"type":"v1","addr":"172.21.15.118:6825","nonce":431354576}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6822","nonce":431354576},{"type":"v1","addr":"172.21.15.118:6823","nonce":431354576}]},"public_addr":"172.21.15.118:6819/431354576","cluster_addr":"172.21.15.118:6821/431354576","heartbeat_back_addr":"172.21.15.118:6825/431354576","heartbeat_front_addr":"172.21.15.118:6823/431354576","state":["exists","up"]},{"osd":7,"uuid":"1fbbd38f-4e8e-4d4c-8e98-66f6862bc4bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":52,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6816","nonce":3667611284},{"type":"v1","addr":"172.21.15.189:6817","nonce":3667611284}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6818","nonce":3667611284},{"type":"v1","addr":"172.21.15.189:6819","nonce":3667611284}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6822","nonce":3667611284},{"type":"v1","addr":"172.21.15.189:6823","nonce":3667611284}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6820","nonce":3667611284},{"type":"v1","addr":"172.21.15.189:6821","nonce":3667611284}]},"public_addr":"172.21.15.189:6817/3667611284","cluster_addr":"172.21.15.189:6819/3667611284","heartbeat_back_addr":"172.21.15.189:6823/3667611284","heartbeat_front_addr":"172.21.15.189:6821/3667611284","state":["exists","up"]},{"osd":8,"uuid":"6462c2b5-e4b3-4b0b-bcfa-4de2c89e93e0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":60,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6816","nonce":3853800910},{"type":"v1","addr":"172.21.15.193:6817","nonce":3853800910}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6818","nonce":3853800910},{"type":"v1","addr":"172.21.15.193:6819","nonce":3853800910}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6822","nonce":3853800910},{"type":"v1","addr":"172.21.15.193:6823","nonce":3853800910}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6820","nonce":3853800910},{"type":"v1","addr":"172.21.15.193:6821","nonce":3853800910}]},"public_addr":"172.21.15.193:6817/3853800910","cluster_addr":"172.21.15.193:6819/3853800910","heartbeat_back_addr":"172.21.15.193:6823/3853800910","heartbeat_front_addr":"172.21.15.193:6821/3853800910","state":["exists","up"]},{"osd":9,"uuid":"54be277f-3596-422f-9de5-9005a8571c6f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":68,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6826","nonce":3248293936},{"type":"v1","addr":"172.21.15.118:6827","nonce":3248293936}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6828","nonce":3248293936},{"type":"v1","addr":"172.21.15.118:6829","nonce":3248293936}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6832","nonce":3248293936},{"type":"v1","addr":"172.21.15.118:6833","nonce":3248293936}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6830","nonce":3248293936},{"type":"v1","addr":"172.21.15.118:6831","nonce":3248293936}]},"public_addr":"172.21.15.118:6827/3248293936","cluster_addr":"172.21.15.118:6829/3248293936","heartbeat_back_addr":"172.21.15.118:6833/3248293936","heartbeat_front_addr":"172.21.15.118:6831/3248293936","state":["exists","up"]},{"osd":10,"uuid":"7ba69d69-ac41-44ce-bc39-c049a8ca5900","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":73,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6824","nonce":2112195760},{"type":"v1","addr":"172.21.15.189:6825","nonce":2112195760}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6826","nonce":2112195760},{"type":"v1","addr":"172.21.15.189:6827","nonce":2112195760}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6830","nonce":2112195760},{"type":"v1","addr":"172.21.15.189:6831","nonce":2112195760}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6828","nonce":2112195760},{"type":"v1","addr":"172.21.15.189:6829","nonce":2112195760}]},"public_addr":"172.21.15.189:6825/2112195760","cluster_addr":"172.21.15.189:6827/2112195760","heartbeat_back_addr":"172.21.15.189:6831/2112195760","heartbeat_front_addr":"172.21.15.189:6829/2112195760","state":["exists","up"]},{"osd":11,"uuid":"186f0b18-6b22-4eae-83b7-b21bbcce0817","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":81,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6824","nonce":3880515510},{"type":"v1","addr":"172.21.15.193:6825","nonce":3880515510}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6826","nonce":3880515510},{"type":"v1","addr":"172.21.15.193:6827","nonce":3880515510}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6830","nonce":3880515510},{"type":"v1","addr":"172.21.15.193:6831","nonce":3880515510}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6828","nonce":3880515510},{"type":"v1","addr":"172.21.15.193:6829","nonce":3880515510}]},"public_addr":"172.21.15.193:6825/3880515510","cluster_addr":"172.21.15.193:6827/3880515510","heartbeat_back_addr":"172.21.15.193:6831/3880515510","heartbeat_front_addr":"172.21.15.193:6829/3880515510","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:06:58.105543+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:07:23.795866+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:07:47.426050+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:08:19.427073+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:08:45.816082+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:09:11.081523+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:09:44.485077+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:10:11.338972+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:10:36.234024+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:11:09.730432+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:11:37.059083+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:12:02.371947+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.118:6801/389249272":"2024-03-25T00:05:05.458796+0000","172.21.15.118:6800/389249272":"2024-03-25T00:05:05.458796+0000","172.21.15.118:0/2101854694":"2024-03-25T00:05:05.458796+0000","172.21.15.118:0/630779359":"2024-03-25T00:05:05.458796+0000","172.21.15.118:0/2158921029":"2024-03-25T00:04:31.283725+0000","172.21.15.118:0/1003670455":"2024-03-25T00:05:05.458796+0000","172.21.15.118:0/368712314":"2024-03-25T00:04:31.283725+0000","172.21.15.118:0/2004775597":"2024-03-25T00:04:31.283725+0000","172.21.15.118:6800/1463147955":"2024-03-25T00:04:31.283725+0000","172.21.15.118:6801/1463147955":"2024-03-25T00:04:31.283725+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-24T00:13:23.690 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:23.689+0000 7f3862400700 1 -- 172.21.15.118:0/854968645 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7f38400a29a0 msgr2=0x7f38400a4e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:13:23.690 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:23.689+0000 7f3862400700 1 --2- 172.21.15.118:0/854968645 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7f38400a29a0 0x7f38400a4e60 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7f384c0098c0 tx=0x7f384c019040 comp rx=0 tx=0).stop 2024-03-24T00:13:23.691 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:23.689+0000 7f3862400700 1 -- 172.21.15.118:0/854968645 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f385c101a30 msgr2=0x7f385c1a0cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:13:23.691 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:23.689+0000 7f3862400700 1 --2- 172.21.15.118:0/854968645 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f385c101a30 0x7f385c1a0cf0 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f384800eb30 tx=0x7f384800ee40 comp rx=0 tx=0).stop 2024-03-24T00:13:23.691 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:23.689+0000 7f3862400700 1 -- 172.21.15.118:0/854968645 shutdown_connections 2024-03-24T00:13:23.691 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:23.689+0000 7f3862400700 1 --2- 172.21.15.118:0/854968645 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f385c1070d0 0x7f385c19a970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:13:23.691 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:23.689+0000 7f3862400700 1 --2- 172.21.15.118:0/854968645 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7f38400a29a0 0x7f38400a4e60 unknown :-1 s=CLOSED pgs=179 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:13:23.691 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:23.690+0000 7f3862400700 1 --2- 172.21.15.118:0/854968645 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f385c101090 0x7f385c1a07b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:13:23.691 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:23.690+0000 7f3862400700 1 --2- 172.21.15.118:0/854968645 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f385c101a30 0x7f385c1a0cf0 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:13:23.691 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:23.690+0000 7f3862400700 1 -- 172.21.15.118:0/854968645 >> 172.21.15.118:0/854968645 conn(0x7f385c077fc0 msgr2=0x7f385c10c340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:13:23.691 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:23.690+0000 7f3862400700 1 -- 172.21.15.118:0/854968645 shutdown_connections 2024-03-24T00:13:23.691 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:23.690+0000 7f3862400700 1 -- 172.21.15.118:0/854968645 wait complete. 2024-03-24T00:13:24.621 INFO:tasks.cephadm.ceph_manager.ceph:all up! 2024-03-24T00:13:24.621 DEBUG:teuthology.orchestra.run.smithi118:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 shell --fsid d79a5420-e971-11ee-95cd-87774f69a715 -- ceph osd dump --format=json 2024-03-24T00:13:24.945 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:13:24 smithi118 ceph-mon[31535]: pgmap v325: 97 pgs: 97 active+clean; 636 KiB data, 344 MiB used, 1.0 TiB / 1.0 TiB avail; 10 KiB/s rd, 4.2 KiB/s wr, 15 op/s 2024-03-24T00:13:24.945 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:13:24 smithi118 ceph-mon[31535]: from='client.? 172.21.15.118:0/854968645' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-24T00:13:24.953 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:13:24 smithi189 ceph-mon[36187]: pgmap v325: 97 pgs: 97 active+clean; 636 KiB data, 344 MiB used, 1.0 TiB / 1.0 TiB avail; 10 KiB/s rd, 4.2 KiB/s wr, 15 op/s 2024-03-24T00:13:24.953 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:13:24 smithi189 ceph-mon[36187]: from='client.? 172.21.15.118:0/854968645' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-24T00:13:24.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:13:24 smithi193 ceph-mon[36878]: pgmap v325: 97 pgs: 97 active+clean; 636 KiB data, 344 MiB used, 1.0 TiB / 1.0 TiB avail; 10 KiB/s rd, 4.2 KiB/s wr, 15 op/s 2024-03-24T00:13:24.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:13:24 smithi193 ceph-mon[36878]: from='client.? 172.21.15.118:0/854968645' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-24T00:13:25.076 INFO:teuthology.orchestra.run.smithi118.stderr:Inferring config /var/lib/ceph/d79a5420-e971-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T00:13:26.928 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:13:26 smithi118 ceph-mon[31535]: pgmap v326: 97 pgs: 97 active+clean; 640 KiB data, 344 MiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 3.5 KiB/s wr, 12 op/s 2024-03-24T00:13:26.953 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:13:26 smithi189 ceph-mon[36187]: pgmap v326: 97 pgs: 97 active+clean; 640 KiB data, 344 MiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 3.5 KiB/s wr, 12 op/s 2024-03-24T00:13:26.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:13:26 smithi193 ceph-mon[36878]: pgmap v326: 97 pgs: 97 active+clean; 640 KiB data, 344 MiB used, 1.0 TiB / 1.0 TiB avail; 11 KiB/s rd, 3.5 KiB/s wr, 12 op/s 2024-03-24T00:13:27.502 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:27.501+0000 7ff898200700 1 -- 172.21.15.118:0/3842941379 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7ff890107510 msgr2=0x7ff89010f0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:13:27.502 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:27.501+0000 7ff898200700 1 --2- 172.21.15.118:0/3842941379 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7ff890107510 0x7ff89010f0e0 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7ff88800b0d0 tx=0x7ff88800b3e0 comp rx=0 tx=0).stop 2024-03-24T00:13:27.502 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:27.501+0000 7ff898200700 1 -- 172.21.15.118:0/3842941379 shutdown_connections 2024-03-24T00:13:27.502 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:27.501+0000 7ff898200700 1 --2- 172.21.15.118:0/3842941379 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff890102440 0x7ff890102820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:13:27.502 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:27.501+0000 7ff898200700 1 --2- 172.21.15.118:0/3842941379 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7ff890107510 0x7ff89010f0e0 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:13:27.502 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:27.501+0000 7ff898200700 1 --2- 172.21.15.118:0/3842941379 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7ff890102df0 0x7ff890106e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:13:27.502 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:27.501+0000 7ff898200700 1 -- 172.21.15.118:0/3842941379 >> 172.21.15.118:0/3842941379 conn(0x7ff8900792f0 msgr2=0x7ff890079700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:13:27.502 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:27.502+0000 7ff898200700 1 -- 172.21.15.118:0/3842941379 shutdown_connections 2024-03-24T00:13:27.502 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:27.502+0000 7ff898200700 1 -- 172.21.15.118:0/3842941379 wait complete. 2024-03-24T00:13:27.503 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:27.503+0000 7ff898200700 1 Processor -- start 2024-03-24T00:13:27.503 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:27.503+0000 7ff898200700 1 -- start start 2024-03-24T00:13:27.503 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:27.503+0000 7ff898200700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7ff890102440 0x7ff8901a07d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:13:27.504 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:27.503+0000 7ff898200700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff890102df0 0x7ff8901a0d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:13:27.504 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:27.503+0000 7ff898200700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7ff890107510 0x7ff89019a990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:13:27.504 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:27.504+0000 7ff898200700 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff890112560 con 0x7ff890102440 2024-03-24T00:13:27.504 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:27.504+0000 7ff898200700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff8901123e0 con 0x7ff890102df0 2024-03-24T00:13:27.504 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:27.504+0000 7ff898200700 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff8901126e0 con 0x7ff890107510 2024-03-24T00:13:27.504 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:27.504+0000 7ff894e00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff890102df0 0x7ff8901a0d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:13:27.504 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:27.504+0000 7ff895800700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7ff890102440 0x7ff8901a07d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:13:27.504 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:27.504+0000 7ff896200700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7ff890107510 0x7ff89019a990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:13:27.504 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:27.504+0000 7ff894e00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff890102df0 0x7ff8901a0d10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.118:57772/0 (socket says 172.21.15.118:57772) 2024-03-24T00:13:27.504 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:27.504+0000 7ff894e00700 1 -- 172.21.15.118:0/3412575577 learned_addr learned my addr 172.21.15.118:0/3412575577 (peer_addr_for_me v2:172.21.15.118:0/0) 2024-03-24T00:13:27.505 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:27.504+0000 7ff896200700 1 -- 172.21.15.118:0/3412575577 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff890102df0 msgr2=0x7ff8901a0d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:13:27.505 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:27.504+0000 7ff896200700 1 --2- 172.21.15.118:0/3412575577 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff890102df0 0x7ff8901a0d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:13:27.505 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:27.504+0000 7ff896200700 1 -- 172.21.15.118:0/3412575577 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7ff890102440 msgr2=0x7ff8901a07d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:13:27.505 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:27.504+0000 7ff896200700 1 --2- 172.21.15.118:0/3412575577 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7ff890102440 0x7ff8901a07d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:13:27.505 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:27.504+0000 7ff896200700 1 -- 172.21.15.118:0/3412575577 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff89019b0f0 con 0x7ff890107510 2024-03-24T00:13:27.506 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:27.504+0000 7ff896200700 1 --2- 172.21.15.118:0/3412575577 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7ff890107510 0x7ff89019a990 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7ff88800d010 tx=0x7ff8880079a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:13:27.506 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:27.505+0000 7ff88e000700 1 -- 172.21.15.118:0/3412575577 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff888003c00 con 0x7ff890107510 2024-03-24T00:13:27.506 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:27.505+0000 7ff88e000700 1 -- 172.21.15.118:0/3412575577 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7ff888022070 con 0x7ff890107510 2024-03-24T00:13:27.506 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:27.505+0000 7ff88e000700 1 -- 172.21.15.118:0/3412575577 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff88801b460 con 0x7ff890107510 2024-03-24T00:13:27.506 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:27.505+0000 7ff898200700 1 -- 172.21.15.118:0/3412575577 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff89019b380 con 0x7ff890107510 2024-03-24T00:13:27.506 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:27.505+0000 7ff898200700 1 -- 172.21.15.118:0/3412575577 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff8901a76a0 con 0x7ff890107510 2024-03-24T00:13:27.506 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:27.506+0000 7ff898200700 1 -- 172.21.15.118:0/3412575577 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff890068e50 con 0x7ff890107510 2024-03-24T00:13:27.507 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:27.508+0000 7ff88e000700 1 -- 172.21.15.118:0/3412575577 <== mon.1 v2:172.21.15.193:3300/0 4 ==== mgrmap(e 17) v1 ==== 137342+0+0 (secure 0 0 0) 0x7ff888009b80 con 0x7ff890107510 2024-03-24T00:13:27.510 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:27.510+0000 7ff88e000700 1 --2- 172.21.15.118:0/3412575577 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7ff8780a29a0 0x7ff8780a4e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:13:27.510 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:27.511+0000 7ff88e000700 1 -- 172.21.15.118:0/3412575577 <== mon.1 v2:172.21.15.193:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 7979+0+0 (secure 0 0 0) 0x7ff88800f020 con 0x7ff890107510 2024-03-24T00:13:27.510 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:27.511+0000 7ff895800700 1 --2- 172.21.15.118:0/3412575577 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7ff8780a29a0 0x7ff8780a4e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:13:27.511 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:27.511+0000 7ff895800700 1 --2- 172.21.15.118:0/3412575577 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7ff8780a29a0 0x7ff8780a4e60 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7ff89019bad0 tx=0x7ff884009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:13:27.515 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:27.516+0000 7ff88e000700 1 -- 172.21.15.118:0/3412575577 <== mon.1 v2:172.21.15.193:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7ff88802b020 con 0x7ff890107510 2024-03-24T00:13:27.739 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:27.739+0000 7ff898200700 1 -- 172.21.15.118:0/3412575577 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff8901a7870 con 0x7ff890107510 2024-03-24T00:13:27.741 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:27.742+0000 7ff88e000700 1 -- 172.21.15.118:0/3412575577 <== mon.1 v2:172.21.15.193:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v86) v1 ==== 74+0+22287 (secure 0 0 0) 0x7ff88809c430 con 0x7ff890107510 2024-03-24T00:13:27.742 INFO:teuthology.orchestra.run.smithi118.stdout: 2024-03-24T00:13:27.742 INFO:teuthology.orchestra.run.smithi118.stdout:{"epoch":86,"fsid":"d79a5420-e971-11ee-95cd-87774f69a715","created":"2024-03-24T00:03:58.648524+0000","modified":"2024-03-24T00:13:03.282705+0000","last_up_change":"2024-03-24T00:12:03.809000+0000","last_in_change":"2024-03-24T00:11:44.389344+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-24T00:07:50.190965+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-24T00:12:59.993723+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-24T00:13:01.257347+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"86","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"c9590426-3901-4f68-a5ef-d57aa8fbac92","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6802","nonce":557690061},{"type":"v1","addr":"172.21.15.118:6803","nonce":557690061}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6804","nonce":557690061},{"type":"v1","addr":"172.21.15.118:6805","nonce":557690061}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6808","nonce":557690061},{"type":"v1","addr":"172.21.15.118:6809","nonce":557690061}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6806","nonce":557690061},{"type":"v1","addr":"172.21.15.118:6807","nonce":557690061}]},"public_addr":"172.21.15.118:6803/557690061","cluster_addr":"172.21.15.118:6805/557690061","heartbeat_back_addr":"172.21.15.118:6809/557690061","heartbeat_front_addr":"172.21.15.118:6807/557690061","state":["exists","up"]},{"osd":1,"uuid":"fc7f827e-ecf6-4661-aad0-7a22951892e0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6800","nonce":4203029485},{"type":"v1","addr":"172.21.15.189:6801","nonce":4203029485}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6802","nonce":4203029485},{"type":"v1","addr":"172.21.15.189:6803","nonce":4203029485}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6806","nonce":4203029485},{"type":"v1","addr":"172.21.15.189:6807","nonce":4203029485}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6804","nonce":4203029485},{"type":"v1","addr":"172.21.15.189:6805","nonce":4203029485}]},"public_addr":"172.21.15.189:6801/4203029485","cluster_addr":"172.21.15.189:6803/4203029485","heartbeat_back_addr":"172.21.15.189:6807/4203029485","heartbeat_front_addr":"172.21.15.189:6805/4203029485","state":["exists","up"]},{"osd":2,"uuid":"e73ccfe9-0e5b-4734-a154-a2f0d829fcfb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6800","nonce":3699076971},{"type":"v1","addr":"172.21.15.193:6801","nonce":3699076971}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6802","nonce":3699076971},{"type":"v1","addr":"172.21.15.193:6803","nonce":3699076971}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6806","nonce":3699076971},{"type":"v1","addr":"172.21.15.193:6807","nonce":3699076971}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6804","nonce":3699076971},{"type":"v1","addr":"172.21.15.193:6805","nonce":3699076971}]},"public_addr":"172.21.15.193:6801/3699076971","cluster_addr":"172.21.15.193:6803/3699076971","heartbeat_back_addr":"172.21.15.193:6807/3699076971","heartbeat_front_addr":"172.21.15.193:6805/3699076971","state":["exists","up"]},{"osd":3,"uuid":"63fed43b-43ce-4031-8957-7fe94725c1ed","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6810","nonce":4059617010},{"type":"v1","addr":"172.21.15.118:6811","nonce":4059617010}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6812","nonce":4059617010},{"type":"v1","addr":"172.21.15.118:6813","nonce":4059617010}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6816","nonce":4059617010},{"type":"v1","addr":"172.21.15.118:6817","nonce":4059617010}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6814","nonce":4059617010},{"type":"v1","addr":"172.21.15.118:6815","nonce":4059617010}]},"public_addr":"172.21.15.118:6811/4059617010","cluster_addr":"172.21.15.118:6813/4059617010","heartbeat_back_addr":"172.21.15.118:6817/4059617010","heartbeat_front_addr":"172.21.15.118:6815/4059617010","state":["exists","up"]},{"osd":4,"uuid":"57377f66-c3b4-43c1-8e8e-bc873672906d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":34,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6808","nonce":4075247068},{"type":"v1","addr":"172.21.15.189:6809","nonce":4075247068}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6810","nonce":4075247068},{"type":"v1","addr":"172.21.15.189:6811","nonce":4075247068}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6814","nonce":4075247068},{"type":"v1","addr":"172.21.15.189:6815","nonce":4075247068}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6812","nonce":4075247068},{"type":"v1","addr":"172.21.15.189:6813","nonce":4075247068}]},"public_addr":"172.21.15.189:6809/4075247068","cluster_addr":"172.21.15.189:6811/4075247068","heartbeat_back_addr":"172.21.15.189:6815/4075247068","heartbeat_front_addr":"172.21.15.189:6813/4075247068","state":["exists","up"]},{"osd":5,"uuid":"91970976-2193-4224-81c0-dc96cb022c85","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":39,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6808","nonce":2920363402},{"type":"v1","addr":"172.21.15.193:6809","nonce":2920363402}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6810","nonce":2920363402},{"type":"v1","addr":"172.21.15.193:6811","nonce":2920363402}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6814","nonce":2920363402},{"type":"v1","addr":"172.21.15.193:6815","nonce":2920363402}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6812","nonce":2920363402},{"type":"v1","addr":"172.21.15.193:6813","nonce":2920363402}]},"public_addr":"172.21.15.193:6809/2920363402","cluster_addr":"172.21.15.193:6811/2920363402","heartbeat_back_addr":"172.21.15.193:6815/2920363402","heartbeat_front_addr":"172.21.15.193:6813/2920363402","state":["exists","up"]},{"osd":6,"uuid":"0b77b196-8674-41c6-ad3d-726b5d01c8fd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":47,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6818","nonce":431354576},{"type":"v1","addr":"172.21.15.118:6819","nonce":431354576}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6820","nonce":431354576},{"type":"v1","addr":"172.21.15.118:6821","nonce":431354576}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6824","nonce":431354576},{"type":"v1","addr":"172.21.15.118:6825","nonce":431354576}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6822","nonce":431354576},{"type":"v1","addr":"172.21.15.118:6823","nonce":431354576}]},"public_addr":"172.21.15.118:6819/431354576","cluster_addr":"172.21.15.118:6821/431354576","heartbeat_back_addr":"172.21.15.118:6825/431354576","heartbeat_front_addr":"172.21.15.118:6823/431354576","state":["exists","up"]},{"osd":7,"uuid":"1fbbd38f-4e8e-4d4c-8e98-66f6862bc4bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":52,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6816","nonce":3667611284},{"type":"v1","addr":"172.21.15.189:6817","nonce":3667611284}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6818","nonce":3667611284},{"type":"v1","addr":"172.21.15.189:6819","nonce":3667611284}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6822","nonce":3667611284},{"type":"v1","addr":"172.21.15.189:6823","nonce":3667611284}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6820","nonce":3667611284},{"type":"v1","addr":"172.21.15.189:6821","nonce":3667611284}]},"public_addr":"172.21.15.189:6817/3667611284","cluster_addr":"172.21.15.189:6819/3667611284","heartbeat_back_addr":"172.21.15.189:6823/3667611284","heartbeat_front_addr":"172.21.15.189:6821/3667611284","state":["exists","up"]},{"osd":8,"uuid":"6462c2b5-e4b3-4b0b-bcfa-4de2c89e93e0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":60,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6816","nonce":3853800910},{"type":"v1","addr":"172.21.15.193:6817","nonce":3853800910}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6818","nonce":3853800910},{"type":"v1","addr":"172.21.15.193:6819","nonce":3853800910}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6822","nonce":3853800910},{"type":"v1","addr":"172.21.15.193:6823","nonce":3853800910}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6820","nonce":3853800910},{"type":"v1","addr":"172.21.15.193:6821","nonce":3853800910}]},"public_addr":"172.21.15.193:6817/3853800910","cluster_addr":"172.21.15.193:6819/3853800910","heartbeat_back_addr":"172.21.15.193:6823/3853800910","heartbeat_front_addr":"172.21.15.193:6821/3853800910","state":["exists","up"]},{"osd":9,"uuid":"54be277f-3596-422f-9de5-9005a8571c6f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":68,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6826","nonce":3248293936},{"type":"v1","addr":"172.21.15.118:6827","nonce":3248293936}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6828","nonce":3248293936},{"type":"v1","addr":"172.21.15.118:6829","nonce":3248293936}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6832","nonce":3248293936},{"type":"v1","addr":"172.21.15.118:6833","nonce":3248293936}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6830","nonce":3248293936},{"type":"v1","addr":"172.21.15.118:6831","nonce":3248293936}]},"public_addr":"172.21.15.118:6827/3248293936","cluster_addr":"172.21.15.118:6829/3248293936","heartbeat_back_addr":"172.21.15.118:6833/3248293936","heartbeat_front_addr":"172.21.15.118:6831/3248293936","state":["exists","up"]},{"osd":10,"uuid":"7ba69d69-ac41-44ce-bc39-c049a8ca5900","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":73,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6824","nonce":2112195760},{"type":"v1","addr":"172.21.15.189:6825","nonce":2112195760}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6826","nonce":2112195760},{"type":"v1","addr":"172.21.15.189:6827","nonce":2112195760}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6830","nonce":2112195760},{"type":"v1","addr":"172.21.15.189:6831","nonce":2112195760}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6828","nonce":2112195760},{"type":"v1","addr":"172.21.15.189:6829","nonce":2112195760}]},"public_addr":"172.21.15.189:6825/2112195760","cluster_addr":"172.21.15.189:6827/2112195760","heartbeat_back_addr":"172.21.15.189:6831/2112195760","heartbeat_front_addr":"172.21.15.189:6829/2112195760","state":["exists","up"]},{"osd":11,"uuid":"186f0b18-6b22-4eae-83b7-b21bbcce0817","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":81,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6824","nonce":3880515510},{"type":"v1","addr":"172.21.15.193:6825","nonce":3880515510}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6826","nonce":3880515510},{"type":"v1","addr":"172.21.15.193:6827","nonce":3880515510}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6830","nonce":3880515510},{"type":"v1","addr":"172.21.15.193:6831","nonce":3880515510}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6828","nonce":3880515510},{"type":"v1","addr":"172.21.15.193:6829","nonce":3880515510}]},"public_addr":"172.21.15.193:6825/3880515510","cluster_addr":"172.21.15.193:6827/3880515510","heartbeat_back_addr":"172.21.15.193:6831/3880515510","heartbeat_front_addr":"172.21.15.193:6829/3880515510","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:06:58.105543+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:07:23.795866+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:07:47.426050+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:08:19.427073+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:08:45.816082+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:09:11.081523+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:09:44.485077+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:10:11.338972+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:10:36.234024+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:11:09.730432+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:11:37.059083+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:12:02.371947+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.118:6801/389249272":"2024-03-25T00:05:05.458796+0000","172.21.15.118:6800/389249272":"2024-03-25T00:05:05.458796+0000","172.21.15.118:0/2101854694":"2024-03-25T00:05:05.458796+0000","172.21.15.118:0/630779359":"2024-03-25T00:05:05.458796+0000","172.21.15.118:0/2158921029":"2024-03-25T00:04:31.283725+0000","172.21.15.118:0/1003670455":"2024-03-25T00:05:05.458796+0000","172.21.15.118:0/368712314":"2024-03-25T00:04:31.283725+0000","172.21.15.118:0/2004775597":"2024-03-25T00:04:31.283725+0000","172.21.15.118:6800/1463147955":"2024-03-25T00:04:31.283725+0000","172.21.15.118:6801/1463147955":"2024-03-25T00:04:31.283725+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-24T00:13:27.746 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:27.746+0000 7ff898200700 1 -- 172.21.15.118:0/3412575577 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7ff8780a29a0 msgr2=0x7ff8780a4e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:13:27.746 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:27.746+0000 7ff898200700 1 --2- 172.21.15.118:0/3412575577 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7ff8780a29a0 0x7ff8780a4e60 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7ff89019bad0 tx=0x7ff884009260 comp rx=0 tx=0).stop 2024-03-24T00:13:27.746 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:27.746+0000 7ff898200700 1 -- 172.21.15.118:0/3412575577 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7ff890107510 msgr2=0x7ff89019a990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:13:27.746 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:27.746+0000 7ff898200700 1 --2- 172.21.15.118:0/3412575577 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7ff890107510 0x7ff89019a990 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7ff88800d010 tx=0x7ff8880079a0 comp rx=0 tx=0).stop 2024-03-24T00:13:27.746 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:27.746+0000 7ff898200700 1 -- 172.21.15.118:0/3412575577 shutdown_connections 2024-03-24T00:13:27.747 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:27.746+0000 7ff898200700 1 --2- 172.21.15.118:0/3412575577 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7ff8780a29a0 0x7ff8780a4e60 unknown :-1 s=CLOSED pgs=180 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:13:27.747 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:27.746+0000 7ff898200700 1 --2- 172.21.15.118:0/3412575577 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7ff890107510 0x7ff89019a990 unknown :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:13:27.747 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:27.746+0000 7ff898200700 1 --2- 172.21.15.118:0/3412575577 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7ff890102440 0x7ff8901a07d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:13:27.747 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:27.746+0000 7ff898200700 1 --2- 172.21.15.118:0/3412575577 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff890102df0 0x7ff8901a0d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:13:27.747 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:27.746+0000 7ff898200700 1 -- 172.21.15.118:0/3412575577 >> 172.21.15.118:0/3412575577 conn(0x7ff8900792f0 msgr2=0x7ff89010d780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:13:27.747 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:27.746+0000 7ff898200700 1 -- 172.21.15.118:0/3412575577 shutdown_connections 2024-03-24T00:13:27.747 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:27.746+0000 7ff898200700 1 -- 172.21.15.118:0/3412575577 wait complete. 2024-03-24T00:13:28.734 DEBUG:teuthology.orchestra.run.smithi118:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 shell --fsid d79a5420-e971-11ee-95cd-87774f69a715 -- ceph tell osd.0 flush_pg_stats 2024-03-24T00:13:28.735 DEBUG:teuthology.orchestra.run.smithi118:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 shell --fsid d79a5420-e971-11ee-95cd-87774f69a715 -- ceph tell osd.1 flush_pg_stats 2024-03-24T00:13:28.735 DEBUG:teuthology.orchestra.run.smithi118:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 shell --fsid d79a5420-e971-11ee-95cd-87774f69a715 -- ceph tell osd.2 flush_pg_stats 2024-03-24T00:13:28.736 DEBUG:teuthology.orchestra.run.smithi118:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 shell --fsid d79a5420-e971-11ee-95cd-87774f69a715 -- ceph tell osd.3 flush_pg_stats 2024-03-24T00:13:28.736 DEBUG:teuthology.orchestra.run.smithi118:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 shell --fsid d79a5420-e971-11ee-95cd-87774f69a715 -- ceph tell osd.4 flush_pg_stats 2024-03-24T00:13:28.736 DEBUG:teuthology.orchestra.run.smithi118:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 shell --fsid d79a5420-e971-11ee-95cd-87774f69a715 -- ceph tell osd.5 flush_pg_stats 2024-03-24T00:13:28.737 DEBUG:teuthology.orchestra.run.smithi118:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 shell --fsid d79a5420-e971-11ee-95cd-87774f69a715 -- ceph tell osd.6 flush_pg_stats 2024-03-24T00:13:28.737 DEBUG:teuthology.orchestra.run.smithi118:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 shell --fsid d79a5420-e971-11ee-95cd-87774f69a715 -- ceph tell osd.7 flush_pg_stats 2024-03-24T00:13:28.738 DEBUG:teuthology.orchestra.run.smithi118:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 shell --fsid d79a5420-e971-11ee-95cd-87774f69a715 -- ceph tell osd.8 flush_pg_stats 2024-03-24T00:13:28.738 DEBUG:teuthology.orchestra.run.smithi118:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 shell --fsid d79a5420-e971-11ee-95cd-87774f69a715 -- ceph tell osd.9 flush_pg_stats 2024-03-24T00:13:28.738 DEBUG:teuthology.orchestra.run.smithi118:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 shell --fsid d79a5420-e971-11ee-95cd-87774f69a715 -- ceph tell osd.10 flush_pg_stats 2024-03-24T00:13:28.739 DEBUG:teuthology.orchestra.run.smithi118:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 shell --fsid d79a5420-e971-11ee-95cd-87774f69a715 -- ceph tell osd.11 flush_pg_stats 2024-03-24T00:13:28.935 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:13:28 smithi118 ceph-mon[31535]: pgmap v327: 97 pgs: 97 active+clean; 640 KiB data, 344 MiB used, 1.0 TiB / 1.0 TiB avail; 5.3 KiB/s rd, 1023 B/s wr, 10 op/s 2024-03-24T00:13:28.935 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:13:28 smithi118 ceph-mon[31535]: from='client.? 172.21.15.118:0/3412575577' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-24T00:13:28.953 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:13:28 smithi189 ceph-mon[36187]: pgmap v327: 97 pgs: 97 active+clean; 640 KiB data, 344 MiB used, 1.0 TiB / 1.0 TiB avail; 5.3 KiB/s rd, 1023 B/s wr, 10 op/s 2024-03-24T00:13:28.953 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:13:28 smithi189 ceph-mon[36187]: from='client.? 172.21.15.118:0/3412575577' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-24T00:13:28.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:13:28 smithi193 ceph-mon[36878]: pgmap v327: 97 pgs: 97 active+clean; 640 KiB data, 344 MiB used, 1.0 TiB / 1.0 TiB avail; 5.3 KiB/s rd, 1023 B/s wr, 10 op/s 2024-03-24T00:13:28.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:13:28 smithi193 ceph-mon[36878]: from='client.? 172.21.15.118:0/3412575577' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-24T00:13:29.769 INFO:teuthology.orchestra.run.smithi118.stderr:Inferring config /var/lib/ceph/d79a5420-e971-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T00:13:29.916 INFO:teuthology.orchestra.run.smithi118.stderr:Inferring config /var/lib/ceph/d79a5420-e971-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T00:13:30.022 INFO:teuthology.orchestra.run.smithi118.stderr:Inferring config /var/lib/ceph/d79a5420-e971-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T00:13:30.030 INFO:teuthology.orchestra.run.smithi118.stderr:Inferring config /var/lib/ceph/d79a5420-e971-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T00:13:30.042 INFO:teuthology.orchestra.run.smithi118.stderr:Inferring config /var/lib/ceph/d79a5420-e971-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T00:13:30.043 INFO:teuthology.orchestra.run.smithi118.stderr:Inferring config /var/lib/ceph/d79a5420-e971-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T00:13:30.049 INFO:teuthology.orchestra.run.smithi118.stderr:Inferring config /var/lib/ceph/d79a5420-e971-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T00:13:30.052 INFO:teuthology.orchestra.run.smithi118.stderr:Inferring config /var/lib/ceph/d79a5420-e971-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T00:13:30.277 INFO:teuthology.orchestra.run.smithi118.stderr:Inferring config /var/lib/ceph/d79a5420-e971-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T00:13:30.279 INFO:teuthology.orchestra.run.smithi118.stderr:Inferring config /var/lib/ceph/d79a5420-e971-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T00:13:30.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:13:30 smithi118 ceph-mon[31535]: pgmap v328: 97 pgs: 97 active+clean; 642 KiB data, 344 MiB used, 1.0 TiB / 1.0 TiB avail; 6.8 KiB/s rd, 1.3 KiB/s wr, 12 op/s 2024-03-24T00:13:30.953 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:13:30 smithi189 ceph-mon[36187]: pgmap v328: 97 pgs: 97 active+clean; 642 KiB data, 344 MiB used, 1.0 TiB / 1.0 TiB avail; 6.8 KiB/s rd, 1.3 KiB/s wr, 12 op/s 2024-03-24T00:13:30.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:13:30 smithi193 ceph-mon[36878]: pgmap v328: 97 pgs: 97 active+clean; 642 KiB data, 344 MiB used, 1.0 TiB / 1.0 TiB avail; 6.8 KiB/s rd, 1.3 KiB/s wr, 12 op/s 2024-03-24T00:13:31.055 INFO:teuthology.orchestra.run.smithi118.stderr:Inferring config /var/lib/ceph/d79a5420-e971-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T00:13:32.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:13:32 smithi118 ceph-mon[31535]: pgmap v329: 97 pgs: 97 active+clean; 642 KiB data, 344 MiB used, 1.0 TiB / 1.0 TiB avail; 4.8 KiB/s rd, 853 B/s wr, 9 op/s 2024-03-24T00:13:32.953 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:13:32 smithi189 ceph-mon[36187]: pgmap v329: 97 pgs: 97 active+clean; 642 KiB data, 344 MiB used, 1.0 TiB / 1.0 TiB avail; 4.8 KiB/s rd, 853 B/s wr, 9 op/s 2024-03-24T00:13:32.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:13:32 smithi193 ceph-mon[36878]: pgmap v329: 97 pgs: 97 active+clean; 642 KiB data, 344 MiB used, 1.0 TiB / 1.0 TiB avail; 4.8 KiB/s rd, 853 B/s wr, 9 op/s 2024-03-24T00:13:34.893 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:13:34 smithi118 ceph-mon[31535]: pgmap v330: 97 pgs: 97 active+clean; 642 KiB data, 344 MiB used, 1.0 TiB / 1.0 TiB avail; 5.7 KiB/s rd, 853 B/s wr, 10 op/s 2024-03-24T00:13:34.953 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:13:34 smithi189 ceph-mon[36187]: pgmap v330: 97 pgs: 97 active+clean; 642 KiB data, 344 MiB used, 1.0 TiB / 1.0 TiB avail; 5.7 KiB/s rd, 853 B/s wr, 10 op/s 2024-03-24T00:13:34.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:13:34 smithi193 ceph-mon[36878]: pgmap v330: 97 pgs: 97 active+clean; 642 KiB data, 344 MiB used, 1.0 TiB / 1.0 TiB avail; 5.7 KiB/s rd, 853 B/s wr, 10 op/s 2024-03-24T00:13:36.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:13:36 smithi118 ceph-mon[31535]: pgmap v331: 97 pgs: 97 active+clean; 642 KiB data, 344 MiB used, 1.0 TiB / 1.0 TiB avail; 6.1 KiB/s rd, 853 B/s wr, 11 op/s 2024-03-24T00:13:36.953 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:13:36 smithi189 ceph-mon[36187]: pgmap v331: 97 pgs: 97 active+clean; 642 KiB data, 344 MiB used, 1.0 TiB / 1.0 TiB avail; 6.1 KiB/s rd, 853 B/s wr, 11 op/s 2024-03-24T00:13:36.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:13:36 smithi193 ceph-mon[36878]: pgmap v331: 97 pgs: 97 active+clean; 642 KiB data, 344 MiB used, 1.0 TiB / 1.0 TiB avail; 6.1 KiB/s rd, 853 B/s wr, 11 op/s 2024-03-24T00:13:38.842 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:13:38 smithi118 ceph-mon[31535]: pgmap v332: 97 pgs: 97 active+clean; 642 KiB data, 344 MiB used, 1.0 TiB / 1.0 TiB avail; 4.8 KiB/s rd, 341 B/s wr, 9 op/s 2024-03-24T00:13:38.953 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:13:38 smithi189 ceph-mon[36187]: pgmap v332: 97 pgs: 97 active+clean; 642 KiB data, 344 MiB used, 1.0 TiB / 1.0 TiB avail; 4.8 KiB/s rd, 341 B/s wr, 9 op/s 2024-03-24T00:13:38.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:13:38 smithi193 ceph-mon[36878]: pgmap v332: 97 pgs: 97 active+clean; 642 KiB data, 344 MiB used, 1.0 TiB / 1.0 TiB avail; 4.8 KiB/s rd, 341 B/s wr, 9 op/s 2024-03-24T00:13:40.835 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:13:40 smithi118 ceph-mon[31535]: pgmap v333: 97 pgs: 97 active+clean; 644 KiB data, 344 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 596 B/s wr, 11 op/s 2024-03-24T00:13:40.953 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:13:40 smithi189 ceph-mon[36187]: pgmap v333: 97 pgs: 97 active+clean; 644 KiB data, 344 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 596 B/s wr, 11 op/s 2024-03-24T00:13:40.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:13:40 smithi193 ceph-mon[36878]: pgmap v333: 97 pgs: 97 active+clean; 644 KiB data, 344 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 596 B/s wr, 11 op/s 2024-03-24T00:13:42.352 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:42.352+0000 7f0d10a00700 1 -- 172.21.15.118:0/3122708657 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f0d0c1081e0 msgr2=0x7f0d0c1085c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:13:42.352 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:42.352+0000 7f0d10a00700 1 --2- 172.21.15.118:0/3122708657 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f0d0c1081e0 0x7f0d0c1085c0 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f0cfc0099c0 tx=0x7f0cfc009cd0 comp rx=0 tx=0).stop 2024-03-24T00:13:42.352 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:42.352+0000 7f0d10a00700 1 -- 172.21.15.118:0/3122708657 shutdown_connections 2024-03-24T00:13:42.352 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:42.352+0000 7f0d10a00700 1 --2- 172.21.15.118:0/3122708657 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0d0c06a7e0 0x7f0d0c10f020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:13:42.352 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:42.352+0000 7f0d10a00700 1 --2- 172.21.15.118:0/3122708657 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f0d0c1081e0 0x7f0d0c1085c0 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:13:42.353 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:42.352+0000 7f0d10a00700 1 --2- 172.21.15.118:0/3122708657 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f0d0c069e40 0x7f0d0c06a2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:13:42.353 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:42.352+0000 7f0d10a00700 1 -- 172.21.15.118:0/3122708657 >> 172.21.15.118:0/3122708657 conn(0x7f0d0c077fc0 msgr2=0x7f0d0c0783d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:13:42.353 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:42.352+0000 7f0d10a00700 1 -- 172.21.15.118:0/3122708657 shutdown_connections 2024-03-24T00:13:42.353 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:42.352+0000 7f0d10a00700 1 -- 172.21.15.118:0/3122708657 wait complete. 2024-03-24T00:13:42.354 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:42.353+0000 7f0d10a00700 1 Processor -- start 2024-03-24T00:13:42.354 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:42.353+0000 7f0d10a00700 1 -- start start 2024-03-24T00:13:42.354 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:42.354+0000 7f0d10a00700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f0d0c069e40 0x7f0d0c19e680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:13:42.354 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:42.354+0000 7f0d10a00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0d0c06a7e0 0x7f0d0c19ebc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:13:42.354 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:42.354+0000 7f0d10a00700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f0d0c1081e0 0x7f0d0c198750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:13:42.354 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:42.354+0000 7f0d10a00700 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0d0c112470 con 0x7f0d0c069e40 2024-03-24T00:13:42.354 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:42.354+0000 7f0d10a00700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0d0c1122f0 con 0x7f0d0c06a7e0 2024-03-24T00:13:42.354 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:42.354+0000 7f0d10a00700 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0d0c1125f0 con 0x7f0d0c1081e0 2024-03-24T00:13:42.355 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:42.354+0000 7f0d09400700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0d0c06a7e0 0x7f0d0c19ebc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:13:42.355 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:42.354+0000 7f0d09e00700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f0d0c069e40 0x7f0d0c19e680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:13:42.355 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:42.354+0000 7f0d09400700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0d0c06a7e0 0x7f0d0c19ebc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.118:38320/0 (socket says 172.21.15.118:38320) 2024-03-24T00:13:42.355 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:42.354+0000 7f0d09400700 1 -- 172.21.15.118:0/3445567176 learned_addr learned my addr 172.21.15.118:0/3445567176 (peer_addr_for_me v2:172.21.15.118:0/0) 2024-03-24T00:13:42.355 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:42.354+0000 7f0d0a800700 1 --2- 172.21.15.118:0/3445567176 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f0d0c1081e0 0x7f0d0c198750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:13:42.355 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:42.354+0000 7f0d09400700 1 -- 172.21.15.118:0/3445567176 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f0d0c1081e0 msgr2=0x7f0d0c198750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:13:42.355 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:42.354+0000 7f0d09400700 1 --2- 172.21.15.118:0/3445567176 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f0d0c1081e0 0x7f0d0c198750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:13:42.355 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:42.354+0000 7f0d09400700 1 -- 172.21.15.118:0/3445567176 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f0d0c069e40 msgr2=0x7f0d0c19e680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:13:42.355 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:42.354+0000 7f0d09400700 1 --2- 172.21.15.118:0/3445567176 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f0d0c069e40 0x7f0d0c19e680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:13:42.355 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:42.354+0000 7f0d09400700 1 -- 172.21.15.118:0/3445567176 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0d0c199030 con 0x7f0d0c06a7e0 2024-03-24T00:13:42.356 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:42.355+0000 7f0d09400700 1 --2- 172.21.15.118:0/3445567176 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0d0c06a7e0 0x7f0d0c19ebc0 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f0cf800e9a0 tx=0x7f0cf800ecb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:13:42.356 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:42.355+0000 7f0d02a00700 1 -- 172.21.15.118:0/3445567176 <== mon.2 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0cf800ca10 con 0x7f0d0c06a7e0 2024-03-24T00:13:42.356 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:42.355+0000 7f0d02a00700 1 -- 172.21.15.118:0/3445567176 <== mon.2 v2:172.21.15.189:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f0cf800cbb0 con 0x7f0d0c06a7e0 2024-03-24T00:13:42.356 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:42.355+0000 7f0d10a00700 1 -- 172.21.15.118:0/3445567176 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0d0c199320 con 0x7f0d0c06a7e0 2024-03-24T00:13:42.356 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:42.355+0000 7f0d02a00700 1 -- 172.21.15.118:0/3445567176 <== mon.2 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0cf8018680 con 0x7f0d0c06a7e0 2024-03-24T00:13:42.356 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:42.355+0000 7f0d10a00700 1 -- 172.21.15.118:0/3445567176 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0d0c1a5600 con 0x7f0d0c06a7e0 2024-03-24T00:13:42.356 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:42.356+0000 7f0d10a00700 1 -- 172.21.15.118:0/3445567176 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f0d0c0628e0 con 0x7f0d0c06a7e0 2024-03-24T00:13:42.357 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:42.358+0000 7f0d02a00700 1 -- 172.21.15.118:0/3445567176 <== mon.2 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 17) v1 ==== 137342+0+0 (secure 0 0 0) 0x7f0cf8010430 con 0x7f0d0c06a7e0 2024-03-24T00:13:42.359 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:42.359+0000 7f0d02a00700 1 --2- 172.21.15.118:0/3445567176 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7f0cf00a29a0 0x7f0cf00a4e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:13:42.359 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:42.360+0000 7f0d02a00700 1 -- 172.21.15.118:0/3445567176 <== mon.2 v2:172.21.15.189:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 7979+0+0 (secure 0 0 0) 0x7f0cf80cecf0 con 0x7f0d0c06a7e0 2024-03-24T00:13:42.359 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:42.360+0000 7f0d09e00700 1 --2- 172.21.15.118:0/3445567176 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7f0cf00a29a0 0x7f0cf00a4e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:13:42.360 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:42.360+0000 7f0d02a00700 1 --2- 172.21.15.118:0/3445567176 >> [v2:172.21.15.189:6808/4075247068,v1:172.21.15.189:6809/4075247068] conn(0x7f0cf00ac180 0x7f0cf00ae5a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:13:42.360 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:42.360+0000 7f0d02a00700 1 -- 172.21.15.118:0/3445567176 --> [v2:172.21.15.189:6808/4075247068,v1:172.21.15.189:6809/4075247068] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f0cf8015e80 con 0x7f0cf00ac180 2024-03-24T00:13:42.360 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:42.360+0000 7f0d02a00700 1 -- 172.21.15.118:0/3445567176 <== mon.2 v2:172.21.15.189:3300/0 6 ==== mon_get_version_reply(handle=1 version=86) v2 ==== 24+0+0 (secure 0 0 0) 0x7f0cf80d1310 con 0x7f0d0c06a7e0 2024-03-24T00:13:42.360 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:42.360+0000 7f0d0a800700 1 --2- 172.21.15.118:0/3445567176 >> [v2:172.21.15.189:6808/4075247068,v1:172.21.15.189:6809/4075247068] conn(0x7f0cf00ac180 0x7f0cf00ae5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:13:42.360 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:42.360+0000 7f0d09e00700 1 --2- 172.21.15.118:0/3445567176 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7f0cf00a29a0 0x7f0cf00a4e60 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7f0cfc000c00 tx=0x7f0cfc005fb0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:13:42.361 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:42.361+0000 7f0d0a800700 1 --2- 172.21.15.118:0/3445567176 >> [v2:172.21.15.189:6808/4075247068,v1:172.21.15.189:6809/4075247068] conn(0x7f0cf00ac180 0x7f0cf00ae5a0 crc :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:13:42.361 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:42.362+0000 7f0d02a00700 1 -- 172.21.15.118:0/3445567176 <== osd.4 v2:172.21.15.189:6808/4075247068 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7f0cf8015e80 con 0x7f0cf00ac180 2024-03-24T00:13:42.380 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:42.380+0000 7f0d10a00700 1 -- 172.21.15.118:0/3445567176 --> [v2:172.21.15.189:6808/4075247068,v1:172.21.15.189:6809/4075247068] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f0d0c062ce0 con 0x7f0cf00ac180 2024-03-24T00:13:42.380 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:42.380+0000 7f0d02a00700 1 -- 172.21.15.118:0/3445567176 <== osd.4 v2:172.21.15.189:6808/4075247068 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+12 (crc 0 0 0) 0x7f0d0c062ce0 con 0x7f0cf00ac180 2024-03-24T00:13:42.381 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:42.381+0000 7f0d10a00700 1 -- 172.21.15.118:0/3445567176 >> [v2:172.21.15.189:6808/4075247068,v1:172.21.15.189:6809/4075247068] conn(0x7f0cf00ac180 msgr2=0x7f0cf00ae5a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:13:42.381 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:42.381+0000 7f0d10a00700 1 --2- 172.21.15.118:0/3445567176 >> [v2:172.21.15.189:6808/4075247068,v1:172.21.15.189:6809/4075247068] conn(0x7f0cf00ac180 0x7f0cf00ae5a0 crc :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:13:42.381 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:42.381+0000 7f0d10a00700 1 -- 172.21.15.118:0/3445567176 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7f0cf00a29a0 msgr2=0x7f0cf00a4e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:13:42.381 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:42.381+0000 7f0d10a00700 1 --2- 172.21.15.118:0/3445567176 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7f0cf00a29a0 0x7f0cf00a4e60 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7f0cfc000c00 tx=0x7f0cfc005fb0 comp rx=0 tx=0).stop 2024-03-24T00:13:42.381 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:42.381+0000 7f0d10a00700 1 -- 172.21.15.118:0/3445567176 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0d0c06a7e0 msgr2=0x7f0d0c19ebc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:13:42.381 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:42.381+0000 7f0d10a00700 1 --2- 172.21.15.118:0/3445567176 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0d0c06a7e0 0x7f0d0c19ebc0 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f0cf800e9a0 tx=0x7f0cf800ecb0 comp rx=0 tx=0).stop 2024-03-24T00:13:42.382 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:42.381+0000 7f0d0a800700 1 -- 172.21.15.118:0/3445567176 reap_dead start 2024-03-24T00:13:42.382 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:42.381+0000 7f0d10a00700 1 -- 172.21.15.118:0/3445567176 shutdown_connections 2024-03-24T00:13:42.382 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:42.381+0000 7f0d10a00700 1 -- 172.21.15.118:0/3445567176 >> 172.21.15.118:0/3445567176 conn(0x7f0d0c077fc0 msgr2=0x7f0d0c0ffcd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:13:42.382 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:42.381+0000 7f0d10a00700 1 -- 172.21.15.118:0/3445567176 shutdown_connections 2024-03-24T00:13:42.382 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:42.381+0000 7f0d10a00700 1 -- 172.21.15.118:0/3445567176 wait complete. 2024-03-24T00:13:42.945 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:13:42 smithi118 ceph-mon[31535]: pgmap v334: 97 pgs: 97 active+clean; 644 KiB data, 344 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 255 B/s wr, 8 op/s 2024-03-24T00:13:42.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:13:42 smithi193 ceph-mon[36878]: pgmap v334: 97 pgs: 97 active+clean; 644 KiB data, 344 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 255 B/s wr, 8 op/s 2024-03-24T00:13:43.203 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:13:42 smithi189 ceph-mon[36187]: pgmap v334: 97 pgs: 97 active+clean; 644 KiB data, 344 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 255 B/s wr, 8 op/s 2024-03-24T00:13:43.908 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:43.333+0000 7f6855800700 1 -- 172.21.15.118:0/2433413749 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f6850113360 msgr2=0x7f6850115750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:13:43.908 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:43.333+0000 7f6855800700 1 --2- 172.21.15.118:0/2433413749 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f6850113360 0x7f6850115750 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f684800b0d0 tx=0x7f684800b3e0 comp rx=0 tx=0).stop 2024-03-24T00:13:43.908 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:43.333+0000 7f6855800700 1 -- 172.21.15.118:0/2433413749 shutdown_connections 2024-03-24T00:13:43.908 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:43.333+0000 7f6855800700 1 --2- 172.21.15.118:0/2433413749 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f68500743c0 0x7f68500747a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:13:43.908 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:43.333+0000 7f6855800700 1 --2- 172.21.15.118:0/2433413749 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f6850113360 0x7f6850115750 unknown :-1 s=CLOSED pgs=102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:13:43.908 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:43.333+0000 7f6855800700 1 --2- 172.21.15.118:0/2433413749 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f6850074ce0 0x7f6850112e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:13:43.908 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:43.333+0000 7f6855800700 1 -- 172.21.15.118:0/2433413749 >> 172.21.15.118:0/2433413749 conn(0x7f68500792f0 msgr2=0x7f6850079700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:13:43.908 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:43.333+0000 7f6855800700 1 -- 172.21.15.118:0/2433413749 shutdown_connections 2024-03-24T00:13:43.908 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:43.333+0000 7f6855800700 1 -- 172.21.15.118:0/2433413749 wait complete. 2024-03-24T00:13:43.908 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:43.335+0000 7f6855800700 1 Processor -- start 2024-03-24T00:13:43.908 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:43.335+0000 7f6855800700 1 -- start start 2024-03-24T00:13:43.909 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:43.335+0000 7f6855800700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f68500743c0 0x7f685019ef10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:13:43.909 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:43.335+0000 7f6855800700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f6850074ce0 0x7f685019f450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:13:43.909 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:43.335+0000 7f6855800700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6850113360 0x7f68501a37e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:13:43.909 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:43.335+0000 7f6855800700 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6850118ba0 con 0x7f6850074ce0 2024-03-24T00:13:43.909 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:43.335+0000 7f6855800700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6850118a20 con 0x7f6850113360 2024-03-24T00:13:43.909 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:43.335+0000 7f6855800700 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6850118d20 con 0x7f68500743c0 2024-03-24T00:13:43.909 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:43.335+0000 7f684ea00700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f68500743c0 0x7f685019ef10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:13:43.909 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:43.335+0000 7f684ea00700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f68500743c0 0x7f685019ef10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.118:54212/0 (socket says 172.21.15.118:54212) 2024-03-24T00:13:43.909 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:43.335+0000 7f684ea00700 1 -- 172.21.15.118:0/3804535778 learned_addr learned my addr 172.21.15.118:0/3804535778 (peer_addr_for_me v2:172.21.15.118:0/0) 2024-03-24T00:13:43.909 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:43.335+0000 7f684f400700 1 --2- 172.21.15.118:0/3804535778 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6850113360 0x7f68501a37e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:13:43.909 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:43.336+0000 7f684e000700 1 --2- 172.21.15.118:0/3804535778 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f6850074ce0 0x7f685019f450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:13:43.909 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:43.336+0000 7f684ea00700 1 -- 172.21.15.118:0/3804535778 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6850113360 msgr2=0x7f68501a37e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:13:43.910 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:43.336+0000 7f684ea00700 1 --2- 172.21.15.118:0/3804535778 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6850113360 0x7f68501a37e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:13:43.910 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:43.336+0000 7f684ea00700 1 -- 172.21.15.118:0/3804535778 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f6850074ce0 msgr2=0x7f685019f450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:13:43.910 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:43.336+0000 7f684ea00700 1 --2- 172.21.15.118:0/3804535778 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f6850074ce0 0x7f685019f450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:13:43.910 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:43.336+0000 7f684ea00700 1 -- 172.21.15.118:0/3804535778 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f68501a3f20 con 0x7f68500743c0 2024-03-24T00:13:43.910 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:43.336+0000 7f684f400700 1 --2- 172.21.15.118:0/3804535778 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6850113360 0x7f68501a37e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T00:13:43.910 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:43.336+0000 7f684ea00700 1 --2- 172.21.15.118:0/3804535778 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f68500743c0 0x7f685019ef10 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f684000e9a0 tx=0x7f684000ecb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:13:43.910 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:43.336+0000 7f6847400700 1 -- 172.21.15.118:0/3804535778 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f684000ca10 con 0x7f68500743c0 2024-03-24T00:13:43.910 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:43.336+0000 7f6847400700 1 -- 172.21.15.118:0/3804535778 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f684000cbb0 con 0x7f68500743c0 2024-03-24T00:13:43.910 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:43.336+0000 7f6847400700 1 -- 172.21.15.118:0/3804535778 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6840018680 con 0x7f68500743c0 2024-03-24T00:13:43.910 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:43.337+0000 7f6855800700 1 -- 172.21.15.118:0/3804535778 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f68501a4210 con 0x7f68500743c0 2024-03-24T00:13:43.910 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:43.337+0000 7f6855800700 1 -- 172.21.15.118:0/3804535778 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f68501aba90 con 0x7f68500743c0 2024-03-24T00:13:43.910 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:43.338+0000 7f6855800700 1 -- 172.21.15.118:0/3804535778 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f6850112a30 con 0x7f68500743c0 2024-03-24T00:13:43.910 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:43.340+0000 7f6847400700 1 -- 172.21.15.118:0/3804535778 <== mon.1 v2:172.21.15.193:3300/0 4 ==== mgrmap(e 17) v1 ==== 137342+0+0 (secure 0 0 0) 0x7f6840047a90 con 0x7f68500743c0 2024-03-24T00:13:43.910 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:43.342+0000 7f6847400700 1 --2- 172.21.15.118:0/3804535778 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7f68340a29a0 0x7f68340a4e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:13:43.910 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:43.342+0000 7f684e000700 1 --2- 172.21.15.118:0/3804535778 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7f68340a29a0 0x7f68340a4e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:13:43.911 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:43.342+0000 7f6847400700 1 -- 172.21.15.118:0/3804535778 <== mon.1 v2:172.21.15.193:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 7979+0+0 (secure 0 0 0) 0x7f68400ceea0 con 0x7f68500743c0 2024-03-24T00:13:43.911 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:43.343+0000 7f6847400700 1 --2- 172.21.15.118:0/3804535778 >> [v2:172.21.15.118:6818/431354576,v1:172.21.15.118:6819/431354576] conn(0x7f68340ac180 0x7f68340ae5a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:13:43.911 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:43.343+0000 7f6847400700 1 -- 172.21.15.118:0/3804535778 --> [v2:172.21.15.118:6818/431354576,v1:172.21.15.118:6819/431354576] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f68340aec50 con 0x7f68340ac180 2024-03-24T00:13:43.911 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:43.343+0000 7f6847400700 1 -- 172.21.15.118:0/3804535778 <== mon.1 v2:172.21.15.193:3300/0 6 ==== mon_get_version_reply(handle=1 version=86) v2 ==== 24+0+0 (secure 0 0 0) 0x7f68400cf250 con 0x7f68500743c0 2024-03-24T00:13:43.911 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:43.343+0000 7f684e000700 1 --2- 172.21.15.118:0/3804535778 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7f68340a29a0 0x7f68340a4e60 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7f6850115270 tx=0x7f683c009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:13:43.911 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:43.343+0000 7f684f400700 1 --2- 172.21.15.118:0/3804535778 >> [v2:172.21.15.118:6818/431354576,v1:172.21.15.118:6819/431354576] conn(0x7f68340ac180 0x7f68340ae5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:13:43.912 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:43.343+0000 7f684f400700 1 --2- 172.21.15.118:0/3804535778 >> [v2:172.21.15.118:6818/431354576,v1:172.21.15.118:6819/431354576] conn(0x7f68340ac180 0x7f68340ae5a0 crc :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:13:43.912 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:43.345+0000 7f6847400700 1 -- 172.21.15.118:0/3804535778 <== osd.6 v2:172.21.15.118:6818/431354576 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7f68340aec50 con 0x7f68340ac180 2024-03-24T00:13:43.912 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:43.362+0000 7f6855800700 1 -- 172.21.15.118:0/3804535778 --> [v2:172.21.15.118:6818/431354576,v1:172.21.15.118:6819/431354576] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f6850075af0 con 0x7f68340ac180 2024-03-24T00:13:43.912 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:43.363+0000 7f6847400700 1 -- 172.21.15.118:0/3804535778 <== osd.6 v2:172.21.15.118:6818/431354576 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+12 (crc 0 0 0) 0x7f6850075af0 con 0x7f68340ac180 2024-03-24T00:13:43.912 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:43.363+0000 7f6855800700 1 -- 172.21.15.118:0/3804535778 >> [v2:172.21.15.118:6818/431354576,v1:172.21.15.118:6819/431354576] conn(0x7f68340ac180 msgr2=0x7f68340ae5a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:13:43.912 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:43.363+0000 7f6855800700 1 --2- 172.21.15.118:0/3804535778 >> [v2:172.21.15.118:6818/431354576,v1:172.21.15.118:6819/431354576] conn(0x7f68340ac180 0x7f68340ae5a0 crc :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:13:43.912 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:43.364+0000 7f6855800700 1 -- 172.21.15.118:0/3804535778 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7f68340a29a0 msgr2=0x7f68340a4e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:13:43.912 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:43.364+0000 7f6855800700 1 --2- 172.21.15.118:0/3804535778 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7f68340a29a0 0x7f68340a4e60 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7f6850115270 tx=0x7f683c009260 comp rx=0 tx=0).stop 2024-03-24T00:13:43.912 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:43.364+0000 7f6855800700 1 -- 172.21.15.118:0/3804535778 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f68500743c0 msgr2=0x7f685019ef10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:13:43.912 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:43.364+0000 7f6855800700 1 --2- 172.21.15.118:0/3804535778 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f68500743c0 0x7f685019ef10 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f684000e9a0 tx=0x7f684000ecb0 comp rx=0 tx=0).stop 2024-03-24T00:13:43.912 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:43.364+0000 7f684f400700 1 -- 172.21.15.118:0/3804535778 reap_dead start 2024-03-24T00:13:43.912 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:43.364+0000 7f6855800700 1 -- 172.21.15.118:0/3804535778 shutdown_connections 2024-03-24T00:13:43.913 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:43.364+0000 7f6855800700 1 -- 172.21.15.118:0/3804535778 >> 172.21.15.118:0/3804535778 conn(0x7f68500792f0 msgr2=0x7f6850112050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:13:43.913 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:43.364+0000 7f6855800700 1 -- 172.21.15.118:0/3804535778 shutdown_connections 2024-03-24T00:13:43.913 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:43.364+0000 7f6855800700 1 -- 172.21.15.118:0/3804535778 wait complete. 2024-03-24T00:13:44.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:13:44 smithi118 ceph-mon[31535]: pgmap v335: 97 pgs: 97 active+clean; 644 KiB data, 344 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 255 B/s wr, 10 op/s 2024-03-24T00:13:44.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:13:44 smithi193 ceph-mon[36878]: pgmap v335: 97 pgs: 97 active+clean; 644 KiB data, 344 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 255 B/s wr, 10 op/s 2024-03-24T00:13:45.203 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:13:44 smithi189 ceph-mon[36187]: pgmap v335: 97 pgs: 97 active+clean; 644 KiB data, 344 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 255 B/s wr, 10 op/s 2024-03-24T00:13:45.403 INFO:teuthology.orchestra.run.smithi118.stdout:201863462963 2024-03-24T00:13:45.404 DEBUG:teuthology.orchestra.run.smithi118:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 shell --fsid d79a5420-e971-11ee-95cd-87774f69a715 -- ceph osd last-stat-seq osd.6 2024-03-24T00:13:46.016 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:46.009+0000 7f1a4b400700 1 -- 172.21.15.118:0/1823167961 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1a44105f90 msgr2=0x7f1a44106410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:13:46.016 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:46.009+0000 7f1a4b400700 1 --2- 172.21.15.118:0/1823167961 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1a44105f90 0x7f1a44106410 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f1a30009990 tx=0x7f1a30009ca0 comp rx=0 tx=0).stop 2024-03-24T00:13:46.016 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:46.009+0000 7f1a4b400700 1 -- 172.21.15.118:0/1823167961 shutdown_connections 2024-03-24T00:13:46.017 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:46.009+0000 7f1a4b400700 1 --2- 172.21.15.118:0/1823167961 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f1a44106950 0x7f1a44111330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:13:46.017 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:46.009+0000 7f1a4b400700 1 --2- 172.21.15.118:0/1823167961 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f1a441046a0 0x7f1a44104a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:13:46.017 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:46.009+0000 7f1a4b400700 1 --2- 172.21.15.118:0/1823167961 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1a44105f90 0x7f1a44106410 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:13:46.017 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:46.009+0000 7f1a4b400700 1 -- 172.21.15.118:0/1823167961 >> 172.21.15.118:0/1823167961 conn(0x7f1a440ffe30 msgr2=0x7f1a44102250 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:13:46.017 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:46.009+0000 7f1a4b400700 1 -- 172.21.15.118:0/1823167961 shutdown_connections 2024-03-24T00:13:46.017 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:46.009+0000 7f1a4b400700 1 -- 172.21.15.118:0/1823167961 wait complete. 2024-03-24T00:13:46.017 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:46.011+0000 7f1a4b400700 1 Processor -- start 2024-03-24T00:13:46.017 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:46.011+0000 7f1a4b400700 1 -- start start 2024-03-24T00:13:46.017 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:46.011+0000 7f1a4b400700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f1a441046a0 0x7f1a441a0870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:13:46.017 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:46.011+0000 7f1a4b400700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1a44105f90 0x7f1a441a0db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:13:46.018 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:46.011+0000 7f1a4b400700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f1a44106950 0x7f1a4419aa80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:13:46.018 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:46.011+0000 7f1a4b400700 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1a441147c0 con 0x7f1a44106950 2024-03-24T00:13:46.018 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:46.011+0000 7f1a4b400700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1a44114640 con 0x7f1a44105f90 2024-03-24T00:13:46.018 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:46.011+0000 7f1a4b400700 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1a44114940 con 0x7f1a441046a0 2024-03-24T00:13:46.018 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:46.012+0000 7f1a43e00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1a44105f90 0x7f1a441a0db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:13:46.018 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:46.012+0000 7f1a48a00700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f1a441046a0 0x7f1a441a0870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:13:46.018 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:46.012+0000 7f1a43e00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1a44105f90 0x7f1a441a0db0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.118:42780/0 (socket says 172.21.15.118:42780) 2024-03-24T00:13:46.018 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:46.012+0000 7f1a43e00700 1 -- 172.21.15.118:0/114750459 learned_addr learned my addr 172.21.15.118:0/114750459 (peer_addr_for_me v2:172.21.15.118:0/0) 2024-03-24T00:13:46.018 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:46.012+0000 7f1a48a00700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f1a441046a0 0x7f1a441a0870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.118:54228/0 (socket says 172.21.15.118:54228) 2024-03-24T00:13:46.018 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:46.012+0000 7f1a49400700 1 --2- 172.21.15.118:0/114750459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f1a44106950 0x7f1a4419aa80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:13:46.018 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:46.012+0000 7f1a43e00700 1 -- 172.21.15.118:0/114750459 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f1a441046a0 msgr2=0x7f1a441a0870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:13:46.018 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:46.012+0000 7f1a43e00700 1 --2- 172.21.15.118:0/114750459 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f1a441046a0 0x7f1a441a0870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:13:46.019 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:46.012+0000 7f1a43e00700 1 -- 172.21.15.118:0/114750459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f1a44106950 msgr2=0x7f1a4419aa80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:13:46.019 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:46.012+0000 7f1a43e00700 1 --2- 172.21.15.118:0/114750459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f1a44106950 0x7f1a4419aa80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:13:46.019 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:46.012+0000 7f1a43e00700 1 -- 172.21.15.118:0/114750459 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1a4419b1e0 con 0x7f1a44105f90 2024-03-24T00:13:46.019 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:46.012+0000 7f1a49400700 1 --2- 172.21.15.118:0/114750459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f1a44106950 0x7f1a4419aa80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T00:13:46.019 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:46.012+0000 7f1a48a00700 1 --2- 172.21.15.118:0/114750459 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f1a441046a0 0x7f1a441a0870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T00:13:46.019 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:46.012+0000 7f1a43e00700 1 --2- 172.21.15.118:0/114750459 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1a44105f90 0x7f1a441a0db0 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f1a30000c00 tx=0x7f1a30005590 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:13:46.019 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:46.013+0000 7f1a41600700 1 -- 172.21.15.118:0/114750459 <== mon.2 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1a30027070 con 0x7f1a44105f90 2024-03-24T00:13:46.019 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:46.013+0000 7f1a41600700 1 -- 172.21.15.118:0/114750459 <== mon.2 v2:172.21.15.189:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f1a300056b0 con 0x7f1a44105f90 2024-03-24T00:13:46.019 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:46.013+0000 7f1a41600700 1 -- 172.21.15.118:0/114750459 <== mon.2 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1a30017650 con 0x7f1a44105f90 2024-03-24T00:13:46.019 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:46.013+0000 7f1a4b400700 1 -- 172.21.15.118:0/114750459 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1a4419b470 con 0x7f1a44105f90 2024-03-24T00:13:46.019 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:46.013+0000 7f1a4b400700 1 -- 172.21.15.118:0/114750459 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1a441a7740 con 0x7f1a44105f90 2024-03-24T00:13:46.019 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:46.014+0000 7f1a4b400700 1 -- 172.21.15.118:0/114750459 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f1a44062920 con 0x7f1a44105f90 2024-03-24T00:13:46.019 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:46.016+0000 7f1a41600700 1 -- 172.21.15.118:0/114750459 <== mon.2 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 17) v1 ==== 137342+0+0 (secure 0 0 0) 0x7f1a3002a070 con 0x7f1a44105f90 2024-03-24T00:13:46.021 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:46.017+0000 7f1a41600700 1 --2- 172.21.15.118:0/114750459 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7f1a2c0a29a0 0x7f1a2c0a4e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:13:46.021 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:46.017+0000 7f1a41600700 1 -- 172.21.15.118:0/114750459 <== mon.2 v2:172.21.15.189:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 7979+0+0 (secure 0 0 0) 0x7f1a300cfaa0 con 0x7f1a44105f90 2024-03-24T00:13:46.021 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:46.018+0000 7f1a48a00700 1 --2- 172.21.15.118:0/114750459 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7f1a2c0a29a0 0x7f1a2c0a4e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:13:46.021 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:46.018+0000 7f1a41600700 1 --2- 172.21.15.118:0/114750459 >> [v2:172.21.15.189:6800/4203029485,v1:172.21.15.189:6801/4203029485] conn(0x7f1a2c0ac050 0x7f1a2c0ae470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:13:46.021 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:46.018+0000 7f1a41600700 1 -- 172.21.15.118:0/114750459 --> [v2:172.21.15.189:6800/4203029485,v1:172.21.15.189:6801/4203029485] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f1a2c0aeb20 con 0x7f1a2c0ac050 2024-03-24T00:13:46.021 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:46.018+0000 7f1a41600700 1 -- 172.21.15.118:0/114750459 <== mon.2 v2:172.21.15.189:3300/0 6 ==== mon_get_version_reply(handle=1 version=86) v2 ==== 24+0+0 (secure 0 0 0) 0x7f1a300d42a0 con 0x7f1a44105f90 2024-03-24T00:13:46.021 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:46.018+0000 7f1a49400700 1 --2- 172.21.15.118:0/114750459 >> [v2:172.21.15.189:6800/4203029485,v1:172.21.15.189:6801/4203029485] conn(0x7f1a2c0ac050 0x7f1a2c0ae470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:13:46.021 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:46.018+0000 7f1a48a00700 1 --2- 172.21.15.118:0/114750459 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7f1a2c0a29a0 0x7f1a2c0a4e60 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7f1a44101e10 tx=0x7f1a38009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:13:46.021 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:46.019+0000 7f1a49400700 1 --2- 172.21.15.118:0/114750459 >> [v2:172.21.15.189:6800/4203029485,v1:172.21.15.189:6801/4203029485] conn(0x7f1a2c0ac050 0x7f1a2c0ae470 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:13:46.021 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:46.019+0000 7f1a41600700 1 -- 172.21.15.118:0/114750459 <== osd.1 v2:172.21.15.189:6800/4203029485 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7f1a2c0aeb20 con 0x7f1a2c0ac050 2024-03-24T00:13:46.037 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:46.038+0000 7f1a4b400700 1 -- 172.21.15.118:0/114750459 --> [v2:172.21.15.189:6800/4203029485,v1:172.21.15.189:6801/4203029485] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f1a44062d20 con 0x7f1a2c0ac050 2024-03-24T00:13:46.037 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:46.038+0000 7f1a41600700 1 -- 172.21.15.118:0/114750459 <== osd.1 v2:172.21.15.189:6800/4203029485 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7f1a44062d20 con 0x7f1a2c0ac050 2024-03-24T00:13:46.038 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:46.039+0000 7f1a4b400700 1 -- 172.21.15.118:0/114750459 >> [v2:172.21.15.189:6800/4203029485,v1:172.21.15.189:6801/4203029485] conn(0x7f1a2c0ac050 msgr2=0x7f1a2c0ae470 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:13:46.038 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:46.039+0000 7f1a4b400700 1 --2- 172.21.15.118:0/114750459 >> [v2:172.21.15.189:6800/4203029485,v1:172.21.15.189:6801/4203029485] conn(0x7f1a2c0ac050 0x7f1a2c0ae470 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:13:46.038 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:46.039+0000 7f1a4b400700 1 -- 172.21.15.118:0/114750459 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7f1a2c0a29a0 msgr2=0x7f1a2c0a4e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:13:46.038 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:46.039+0000 7f1a4b400700 1 --2- 172.21.15.118:0/114750459 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7f1a2c0a29a0 0x7f1a2c0a4e60 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7f1a44101e10 tx=0x7f1a38009260 comp rx=0 tx=0).stop 2024-03-24T00:13:46.039 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:46.039+0000 7f1a4b400700 1 -- 172.21.15.118:0/114750459 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1a44105f90 msgr2=0x7f1a441a0db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:13:46.039 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:46.039+0000 7f1a4b400700 1 --2- 172.21.15.118:0/114750459 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1a44105f90 0x7f1a441a0db0 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f1a30000c00 tx=0x7f1a30005590 comp rx=0 tx=0).stop 2024-03-24T00:13:46.039 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:46.039+0000 7f1a49400700 1 -- 172.21.15.118:0/114750459 reap_dead start 2024-03-24T00:13:46.039 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:46.039+0000 7f1a4b400700 1 -- 172.21.15.118:0/114750459 shutdown_connections 2024-03-24T00:13:46.039 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:46.039+0000 7f1a4b400700 1 -- 172.21.15.118:0/114750459 >> 172.21.15.118:0/114750459 conn(0x7f1a440ffe30 msgr2=0x7f1a44100410 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:13:46.039 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:46.039+0000 7f1a4b400700 1 -- 172.21.15.118:0/114750459 shutdown_connections 2024-03-24T00:13:46.039 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:46.039+0000 7f1a4b400700 1 -- 172.21.15.118:0/114750459 wait complete. 2024-03-24T00:13:46.796 INFO:teuthology.orchestra.run.smithi118.stdout:146028888125 2024-03-24T00:13:46.796 DEBUG:teuthology.orchestra.run.smithi118:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 shell --fsid d79a5420-e971-11ee-95cd-87774f69a715 -- ceph osd last-stat-seq osd.4 2024-03-24T00:13:47.039 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:40.900+0000 7f0d38a00700 1 -- 172.21.15.118:0/3095375120 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f0d34102c80 msgr2=0x7f0d34106cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:13:47.039 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:40.900+0000 7f0d38a00700 1 --2- 172.21.15.118:0/3095375120 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f0d34102c80 0x7f0d34106cd0 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f0d200099c0 tx=0x7f0d20009cd0 comp rx=0 tx=0).stop 2024-03-24T00:13:47.039 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:40.900+0000 7f0d38a00700 1 -- 172.21.15.118:0/3095375120 shutdown_connections 2024-03-24T00:13:47.039 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:40.900+0000 7f0d38a00700 1 --2- 172.21.15.118:0/3095375120 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0d341022d0 0x7f0d341026b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:13:47.039 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:40.900+0000 7f0d38a00700 1 --2- 172.21.15.118:0/3095375120 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f0d341073b0 0x7f0d3410ef80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:13:47.039 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:40.900+0000 7f0d38a00700 1 --2- 172.21.15.118:0/3095375120 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f0d34102c80 0x7f0d34106cd0 unknown :-1 s=CLOSED pgs=99 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:13:47.039 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:40.900+0000 7f0d38a00700 1 -- 172.21.15.118:0/3095375120 >> 172.21.15.118:0/3095375120 conn(0x7f0d340fdb80 msgr2=0x7f0d340fffa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:13:47.040 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:40.900+0000 7f0d38a00700 1 -- 172.21.15.118:0/3095375120 shutdown_connections 2024-03-24T00:13:47.040 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:40.900+0000 7f0d38a00700 1 -- 172.21.15.118:0/3095375120 wait complete. 2024-03-24T00:13:47.040 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:40.902+0000 7f0d38a00700 1 Processor -- start 2024-03-24T00:13:47.040 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:40.902+0000 7f0d38a00700 1 -- start start 2024-03-24T00:13:47.040 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:40.902+0000 7f0d38a00700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f0d341022d0 0x7f0d3419e630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:13:47.040 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:40.902+0000 7f0d38a00700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f0d34102c80 0x7f0d3419eb70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:13:47.040 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:40.902+0000 7f0d38a00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0d341073b0 0x7f0d34198700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:13:47.040 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:40.902+0000 7f0d38a00700 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0d341123c0 con 0x7f0d341022d0 2024-03-24T00:13:47.040 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:40.902+0000 7f0d38a00700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0d34112240 con 0x7f0d341073b0 2024-03-24T00:13:47.040 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:40.902+0000 7f0d38a00700 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0d34112540 con 0x7f0d34102c80 2024-03-24T00:13:47.040 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:40.903+0000 7f0d32a00700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f0d34102c80 0x7f0d3419eb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:13:47.040 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:40.903+0000 7f0d33e00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0d341073b0 0x7f0d34198700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:13:47.040 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:40.903+0000 7f0d32a00700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f0d34102c80 0x7f0d3419eb70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.118:35360/0 (socket says 172.21.15.118:35360) 2024-03-24T00:13:47.040 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:40.903+0000 7f0d32a00700 1 -- 172.21.15.118:0/2900990561 learned_addr learned my addr 172.21.15.118:0/2900990561 (peer_addr_for_me v2:172.21.15.118:0/0) 2024-03-24T00:13:47.040 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:40.903+0000 7f0d33400700 1 --2- 172.21.15.118:0/2900990561 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f0d341022d0 0x7f0d3419e630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:13:47.041 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:40.903+0000 7f0d32a00700 1 -- 172.21.15.118:0/2900990561 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0d341073b0 msgr2=0x7f0d34198700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:13:47.041 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:40.903+0000 7f0d32a00700 1 --2- 172.21.15.118:0/2900990561 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0d341073b0 0x7f0d34198700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:13:47.041 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:40.903+0000 7f0d32a00700 1 -- 172.21.15.118:0/2900990561 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f0d341022d0 msgr2=0x7f0d3419e630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:13:47.041 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:40.903+0000 7f0d32a00700 1 --2- 172.21.15.118:0/2900990561 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f0d341022d0 0x7f0d3419e630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:13:47.041 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:40.903+0000 7f0d32a00700 1 -- 172.21.15.118:0/2900990561 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0d34198fe0 con 0x7f0d34102c80 2024-03-24T00:13:47.041 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:40.903+0000 7f0d33e00700 1 --2- 172.21.15.118:0/2900990561 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0d341073b0 0x7f0d34198700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T00:13:47.041 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:40.903+0000 7f0d32a00700 1 --2- 172.21.15.118:0/2900990561 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f0d34102c80 0x7f0d3419eb70 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f0d20005f50 tx=0x7f0d20003a90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:13:47.041 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:40.904+0000 7f0d2be00700 1 -- 172.21.15.118:0/2900990561 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0d20003680 con 0x7f0d34102c80 2024-03-24T00:13:47.041 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:40.904+0000 7f0d2be00700 1 -- 172.21.15.118:0/2900990561 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f0d200055a0 con 0x7f0d34102c80 2024-03-24T00:13:47.041 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:40.904+0000 7f0d2be00700 1 -- 172.21.15.118:0/2900990561 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0d2000fce0 con 0x7f0d34102c80 2024-03-24T00:13:47.041 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:40.904+0000 7f0d38a00700 1 -- 172.21.15.118:0/2900990561 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0d34199270 con 0x7f0d34102c80 2024-03-24T00:13:47.041 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:40.904+0000 7f0d38a00700 1 -- 172.21.15.118:0/2900990561 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0d341a55b0 con 0x7f0d34102c80 2024-03-24T00:13:47.041 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:40.905+0000 7f0d38a00700 1 -- 172.21.15.118:0/2900990561 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f0d340628e0 con 0x7f0d34102c80 2024-03-24T00:13:47.041 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:40.907+0000 7f0d2be00700 1 -- 172.21.15.118:0/2900990561 <== mon.1 v2:172.21.15.193:3300/0 4 ==== mgrmap(e 17) v1 ==== 137342+0+0 (secure 0 0 0) 0x7f0d2000f460 con 0x7f0d34102c80 2024-03-24T00:13:47.042 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:40.908+0000 7f0d2be00700 1 --2- 172.21.15.118:0/2900990561 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7f0d180a2b40 0x7f0d180a5000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:13:47.042 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:40.908+0000 7f0d2be00700 1 -- 172.21.15.118:0/2900990561 <== mon.1 v2:172.21.15.193:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 7979+0+0 (secure 0 0 0) 0x7f0d200cfc10 con 0x7f0d34102c80 2024-03-24T00:13:47.042 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:40.908+0000 7f0d33400700 1 --2- 172.21.15.118:0/2900990561 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7f0d180a2b40 0x7f0d180a5000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:13:47.042 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:40.909+0000 7f0d2be00700 1 --2- 172.21.15.118:0/2900990561 >> [v2:172.21.15.118:6810/4059617010,v1:172.21.15.118:6811/4059617010] conn(0x7f0d180ac320 0x7f0d180ae740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:13:47.043 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:40.909+0000 7f0d2be00700 1 -- 172.21.15.118:0/2900990561 --> [v2:172.21.15.118:6810/4059617010,v1:172.21.15.118:6811/4059617010] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f0d20003680 con 0x7f0d180ac320 2024-03-24T00:13:47.043 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:40.909+0000 7f0d2be00700 1 -- 172.21.15.118:0/2900990561 <== mon.1 v2:172.21.15.193:3300/0 6 ==== mon_get_version_reply(handle=1 version=86) v2 ==== 24+0+0 (secure 0 0 0) 0x7f0d200cffc0 con 0x7f0d34102c80 2024-03-24T00:13:47.043 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:40.909+0000 7f0d33e00700 1 --2- 172.21.15.118:0/2900990561 >> [v2:172.21.15.118:6810/4059617010,v1:172.21.15.118:6811/4059617010] conn(0x7f0d180ac320 0x7f0d180ae740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:13:47.043 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:40.909+0000 7f0d33400700 1 --2- 172.21.15.118:0/2900990561 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7f0d180a2b40 0x7f0d180a5000 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7f0d24009680 tx=0x7f0d24009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:13:47.043 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:40.910+0000 7f0d33e00700 1 --2- 172.21.15.118:0/2900990561 >> [v2:172.21.15.118:6810/4059617010,v1:172.21.15.118:6811/4059617010] conn(0x7f0d180ac320 0x7f0d180ae740 crc :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:13:47.043 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:40.911+0000 7f0d2be00700 1 -- 172.21.15.118:0/2900990561 <== osd.3 v2:172.21.15.118:6810/4059617010 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7f0d20003680 con 0x7f0d180ac320 2024-03-24T00:13:47.043 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:40.931+0000 7f0d38a00700 1 -- 172.21.15.118:0/2900990561 --> [v2:172.21.15.118:6810/4059617010,v1:172.21.15.118:6811/4059617010] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f0d34062ce0 con 0x7f0d180ac320 2024-03-24T00:13:47.043 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:40.931+0000 7f0d2be00700 1 -- 172.21.15.118:0/2900990561 <== osd.3 v2:172.21.15.118:6810/4059617010 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+12 (crc 0 0 0) 0x7f0d34062ce0 con 0x7f0d180ac320 2024-03-24T00:13:47.043 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:40.932+0000 7f0d38a00700 1 -- 172.21.15.118:0/2900990561 >> [v2:172.21.15.118:6810/4059617010,v1:172.21.15.118:6811/4059617010] conn(0x7f0d180ac320 msgr2=0x7f0d180ae740 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:13:47.043 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:40.932+0000 7f0d38a00700 1 --2- 172.21.15.118:0/2900990561 >> [v2:172.21.15.118:6810/4059617010,v1:172.21.15.118:6811/4059617010] conn(0x7f0d180ac320 0x7f0d180ae740 crc :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:13:47.043 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:40.932+0000 7f0d38a00700 1 -- 172.21.15.118:0/2900990561 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7f0d180a2b40 msgr2=0x7f0d180a5000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:13:47.043 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:40.932+0000 7f0d38a00700 1 --2- 172.21.15.118:0/2900990561 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7f0d180a2b40 0x7f0d180a5000 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7f0d24009680 tx=0x7f0d24009260 comp rx=0 tx=0).stop 2024-03-24T00:13:47.044 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:40.932+0000 7f0d38a00700 1 -- 172.21.15.118:0/2900990561 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f0d34102c80 msgr2=0x7f0d3419eb70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:13:47.044 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:40.932+0000 7f0d38a00700 1 --2- 172.21.15.118:0/2900990561 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f0d34102c80 0x7f0d3419eb70 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f0d20005f50 tx=0x7f0d20003a90 comp rx=0 tx=0).stop 2024-03-24T00:13:47.044 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:40.932+0000 7f0d33e00700 1 -- 172.21.15.118:0/2900990561 reap_dead start 2024-03-24T00:13:47.044 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:40.932+0000 7f0d38a00700 1 -- 172.21.15.118:0/2900990561 shutdown_connections 2024-03-24T00:13:47.044 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:40.932+0000 7f0d38a00700 1 -- 172.21.15.118:0/2900990561 >> 172.21.15.118:0/2900990561 conn(0x7f0d340fdb80 msgr2=0x7f0d340ffeb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:13:47.044 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:40.932+0000 7f0d38a00700 1 -- 172.21.15.118:0/2900990561 shutdown_connections 2024-03-24T00:13:47.044 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:40.932+0000 7f0d38a00700 1 -- 172.21.15.118:0/2900990561 wait complete. 2024-03-24T00:13:47.044 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:13:46 smithi118 ceph-mon[31535]: pgmap v336: 97 pgs: 97 active+clean; 644 KiB data, 344 MiB used, 1.0 TiB / 1.0 TiB avail; 5.7 KiB/s rd, 255 B/s wr, 11 op/s 2024-03-24T00:13:47.203 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:13:46 smithi189 ceph-mon[36187]: pgmap v336: 97 pgs: 97 active+clean; 644 KiB data, 344 MiB used, 1.0 TiB / 1.0 TiB avail; 5.7 KiB/s rd, 255 B/s wr, 11 op/s 2024-03-24T00:13:47.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:13:46 smithi193 ceph-mon[36878]: pgmap v336: 97 pgs: 97 active+clean; 644 KiB data, 344 MiB used, 1.0 TiB / 1.0 TiB avail; 5.7 KiB/s rd, 255 B/s wr, 11 op/s 2024-03-24T00:13:47.737 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:46.328+0000 7fcdb3c00700 1 -- 172.21.15.118:0/131482395 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fcdac0743c0 msgr2=0x7fcdac0747a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:13:47.738 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:46.328+0000 7fcdb3c00700 1 --2- 172.21.15.118:0/131482395 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fcdac0743c0 0x7fcdac0747a0 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7fcda0009990 tx=0x7fcda0009ca0 comp rx=0 tx=0).stop 2024-03-24T00:13:47.738 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:46.328+0000 7fcdb3c00700 1 -- 172.21.15.118:0/131482395 shutdown_connections 2024-03-24T00:13:47.738 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:46.328+0000 7fcdb3c00700 1 --2- 172.21.15.118:0/131482395 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fcdac10eb90 0x7fcdac110f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:13:47.738 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:46.328+0000 7fcdb3c00700 1 --2- 172.21.15.118:0/131482395 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fcdac0743c0 0x7fcdac0747a0 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:13:47.738 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:46.328+0000 7fcdb3c00700 1 --2- 172.21.15.118:0/131482395 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fcdac074ce0 0x7fcdac10e650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:13:47.738 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:46.328+0000 7fcdb3c00700 1 -- 172.21.15.118:0/131482395 >> 172.21.15.118:0/131482395 conn(0x7fcdac0792f0 msgr2=0x7fcdac079700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:13:47.738 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:46.328+0000 7fcdb3c00700 1 -- 172.21.15.118:0/131482395 shutdown_connections 2024-03-24T00:13:47.738 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:46.329+0000 7fcdb3c00700 1 -- 172.21.15.118:0/131482395 wait complete. 2024-03-24T00:13:47.738 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:46.330+0000 7fcdb3c00700 1 Processor -- start 2024-03-24T00:13:47.738 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:46.330+0000 7fcdb3c00700 1 -- start start 2024-03-24T00:13:47.738 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:46.330+0000 7fcdb3c00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fcdac0743c0 0x7fcdac1a0860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:13:47.739 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:46.330+0000 7fcdb3c00700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fcdac074ce0 0x7fcdac1a0da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:13:47.739 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:46.330+0000 7fcdb3c00700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fcdac10eb90 0x7fcdac19aa70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:13:47.739 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:46.330+0000 7fcdb3c00700 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcdac1147b0 con 0x7fcdac10eb90 2024-03-24T00:13:47.739 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:46.330+0000 7fcdb3c00700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcdac114630 con 0x7fcdac0743c0 2024-03-24T00:13:47.739 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:46.330+0000 7fcdb3c00700 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcdac114930 con 0x7fcdac074ce0 2024-03-24T00:13:47.739 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:46.330+0000 7fcdb1200700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fcdac0743c0 0x7fcdac1a0860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:13:47.739 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:46.330+0000 7fcdabe00700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fcdac074ce0 0x7fcdac1a0da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:13:47.739 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:46.330+0000 7fcdb1200700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fcdac0743c0 0x7fcdac1a0860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.118:42798/0 (socket says 172.21.15.118:42798) 2024-03-24T00:13:47.739 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:46.330+0000 7fcdb1200700 1 -- 172.21.15.118:0/3562510053 learned_addr learned my addr 172.21.15.118:0/3562510053 (peer_addr_for_me v2:172.21.15.118:0/0) 2024-03-24T00:13:47.739 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:46.331+0000 7fcdb1c00700 1 --2- 172.21.15.118:0/3562510053 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fcdac10eb90 0x7fcdac19aa70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:13:47.739 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:46.331+0000 7fcdabe00700 1 -- 172.21.15.118:0/3562510053 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fcdac0743c0 msgr2=0x7fcdac1a0860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:13:47.739 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:46.331+0000 7fcdabe00700 1 --2- 172.21.15.118:0/3562510053 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fcdac0743c0 0x7fcdac1a0860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:13:47.739 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:46.331+0000 7fcdabe00700 1 -- 172.21.15.118:0/3562510053 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fcdac10eb90 msgr2=0x7fcdac19aa70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:13:47.739 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:46.331+0000 7fcdabe00700 1 --2- 172.21.15.118:0/3562510053 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fcdac10eb90 0x7fcdac19aa70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:13:47.740 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:46.331+0000 7fcdabe00700 1 -- 172.21.15.118:0/3562510053 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcdac19b1d0 con 0x7fcdac074ce0 2024-03-24T00:13:47.740 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:46.331+0000 7fcdabe00700 1 --2- 172.21.15.118:0/3562510053 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fcdac074ce0 0x7fcdac1a0da0 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7fcd9800e9f0 tx=0x7fcd9800ed00 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:13:47.740 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:46.331+0000 7fcda9600700 1 -- 172.21.15.118:0/3562510053 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcd9800cbd0 con 0x7fcdac074ce0 2024-03-24T00:13:47.740 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:46.331+0000 7fcda9600700 1 -- 172.21.15.118:0/3562510053 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fcd98004500 con 0x7fcdac074ce0 2024-03-24T00:13:47.740 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:46.331+0000 7fcda9600700 1 -- 172.21.15.118:0/3562510053 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcd98005280 con 0x7fcdac074ce0 2024-03-24T00:13:47.740 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:46.331+0000 7fcdb3c00700 1 -- 172.21.15.118:0/3562510053 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcdac19b4c0 con 0x7fcdac074ce0 2024-03-24T00:13:47.740 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:46.332+0000 7fcdb3c00700 1 -- 172.21.15.118:0/3562510053 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcdac1a7730 con 0x7fcdac074ce0 2024-03-24T00:13:47.740 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:46.332+0000 7fcdb3c00700 1 -- 172.21.15.118:0/3562510053 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7fcdac0ffd00 con 0x7fcdac074ce0 2024-03-24T00:13:47.740 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:46.335+0000 7fcda9600700 1 -- 172.21.15.118:0/3562510053 <== mon.1 v2:172.21.15.193:3300/0 4 ==== mgrmap(e 17) v1 ==== 137342+0+0 (secure 0 0 0) 0x7fcd98005420 con 0x7fcdac074ce0 2024-03-24T00:13:47.740 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:46.336+0000 7fcda9600700 1 --2- 172.21.15.118:0/3562510053 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7fcd940a2920 0x7fcd940a4de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:13:47.740 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:46.336+0000 7fcda9600700 1 -- 172.21.15.118:0/3562510053 <== mon.1 v2:172.21.15.193:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 7979+0+0 (secure 0 0 0) 0x7fcd980ceda0 con 0x7fcdac074ce0 2024-03-24T00:13:47.740 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:46.336+0000 7fcdb1200700 1 --2- 172.21.15.118:0/3562510053 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7fcd940a2920 0x7fcd940a4de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:13:47.741 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:46.336+0000 7fcda9600700 1 --2- 172.21.15.118:0/3562510053 >> [v2:172.21.15.193:6824/3880515510,v1:172.21.15.193:6825/3880515510] conn(0x7fcd940ac100 0x7fcd940ae520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:13:47.741 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:46.336+0000 7fcda9600700 1 -- 172.21.15.118:0/3562510053 --> [v2:172.21.15.193:6824/3880515510,v1:172.21.15.193:6825/3880515510] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fcd9800be50 con 0x7fcd940ac100 2024-03-24T00:13:47.741 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:46.336+0000 7fcda9600700 1 -- 172.21.15.118:0/3562510053 <== mon.1 v2:172.21.15.193:3300/0 6 ==== mon_get_version_reply(handle=1 version=86) v2 ==== 24+0+0 (secure 0 0 0) 0x7fcd980cf150 con 0x7fcdac074ce0 2024-03-24T00:13:47.741 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:46.337+0000 7fcdb1c00700 1 --2- 172.21.15.118:0/3562510053 >> [v2:172.21.15.193:6824/3880515510,v1:172.21.15.193:6825/3880515510] conn(0x7fcd940ac100 0x7fcd940ae520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:13:47.741 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:46.337+0000 7fcdb1200700 1 --2- 172.21.15.118:0/3562510053 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7fcd940a2920 0x7fcd940a4de0 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7fcda0009750 tx=0x7fcda000b280 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:13:47.741 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:46.337+0000 7fcdb1c00700 1 --2- 172.21.15.118:0/3562510053 >> [v2:172.21.15.193:6824/3880515510,v1:172.21.15.193:6825/3880515510] conn(0x7fcd940ac100 0x7fcd940ae520 crc :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.11 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:13:47.741 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:46.338+0000 7fcda9600700 1 -- 172.21.15.118:0/3562510053 <== osd.11 v2:172.21.15.193:6824/3880515510 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7fcd9800be50 con 0x7fcd940ac100 2024-03-24T00:13:47.742 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:46.357+0000 7fcdb3c00700 1 -- 172.21.15.118:0/3562510053 --> [v2:172.21.15.193:6824/3880515510,v1:172.21.15.193:6825/3880515510] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7fcdac100100 con 0x7fcd940ac100 2024-03-24T00:13:47.742 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:46.357+0000 7fcda9600700 1 -- 172.21.15.118:0/3562510053 <== osd.11 v2:172.21.15.193:6824/3880515510 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+12 (crc 0 0 0) 0x7fcdac100100 con 0x7fcd940ac100 2024-03-24T00:13:47.742 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:46.358+0000 7fcdb3c00700 1 -- 172.21.15.118:0/3562510053 >> [v2:172.21.15.193:6824/3880515510,v1:172.21.15.193:6825/3880515510] conn(0x7fcd940ac100 msgr2=0x7fcd940ae520 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:13:47.742 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:46.358+0000 7fcdb3c00700 1 --2- 172.21.15.118:0/3562510053 >> [v2:172.21.15.193:6824/3880515510,v1:172.21.15.193:6825/3880515510] conn(0x7fcd940ac100 0x7fcd940ae520 crc :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:13:47.742 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:46.358+0000 7fcdb3c00700 1 -- 172.21.15.118:0/3562510053 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7fcd940a2920 msgr2=0x7fcd940a4de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:13:47.742 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:46.358+0000 7fcdb3c00700 1 --2- 172.21.15.118:0/3562510053 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7fcd940a2920 0x7fcd940a4de0 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7fcda0009750 tx=0x7fcda000b280 comp rx=0 tx=0).stop 2024-03-24T00:13:47.742 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:46.358+0000 7fcdb3c00700 1 -- 172.21.15.118:0/3562510053 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fcdac074ce0 msgr2=0x7fcdac1a0da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:13:47.742 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:46.358+0000 7fcdb3c00700 1 --2- 172.21.15.118:0/3562510053 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fcdac074ce0 0x7fcdac1a0da0 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7fcd9800e9f0 tx=0x7fcd9800ed00 comp rx=0 tx=0).stop 2024-03-24T00:13:47.742 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:46.358+0000 7fcdb1c00700 1 -- 172.21.15.118:0/3562510053 reap_dead start 2024-03-24T00:13:47.742 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:46.358+0000 7fcdb3c00700 1 -- 172.21.15.118:0/3562510053 shutdown_connections 2024-03-24T00:13:47.742 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:46.358+0000 7fcdb3c00700 1 -- 172.21.15.118:0/3562510053 >> 172.21.15.118:0/3562510053 conn(0x7fcdac0792f0 msgr2=0x7fcdac0ff3a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:13:47.742 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:46.358+0000 7fcdb3c00700 1 -- 172.21.15.118:0/3562510053 shutdown_connections 2024-03-24T00:13:47.743 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:46.358+0000 7fcdb3c00700 1 -- 172.21.15.118:0/3562510053 wait complete. 2024-03-24T00:13:48.848 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:48.667+0000 7f4105800700 1 -- 172.21.15.118:0/333096446 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4100104f90 msgr2=0x7f4100108fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:13:48.848 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:48.667+0000 7f4105800700 1 --2- 172.21.15.118:0/333096446 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4100104f90 0x7f4100108fe0 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f40ec0098f0 tx=0x7f40ec009c00 comp rx=0 tx=0).stop 2024-03-24T00:13:48.848 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:48.667+0000 7f4105800700 1 -- 172.21.15.118:0/333096446 shutdown_connections 2024-03-24T00:13:48.848 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:48.667+0000 7f4105800700 1 --2- 172.21.15.118:0/333096446 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f41001096c0 0x7f4100111290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:13:48.848 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:48.667+0000 7f4105800700 1 --2- 172.21.15.118:0/333096446 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f41001045e0 0x7f41001049c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:13:48.848 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:48.667+0000 7f4105800700 1 --2- 172.21.15.118:0/333096446 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4100104f90 0x7f4100108fe0 unknown :-1 s=CLOSED pgs=102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:13:48.848 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:48.667+0000 7f4105800700 1 -- 172.21.15.118:0/333096446 >> 172.21.15.118:0/333096446 conn(0x7f41000ffe90 msgr2=0x7f41001022b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:13:48.848 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:48.667+0000 7f4105800700 1 -- 172.21.15.118:0/333096446 shutdown_connections 2024-03-24T00:13:48.848 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:48.668+0000 7f4105800700 1 -- 172.21.15.118:0/333096446 wait complete. 2024-03-24T00:13:48.848 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:48.669+0000 7f4105800700 1 Processor -- start 2024-03-24T00:13:48.848 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:48.669+0000 7f4105800700 1 -- start start 2024-03-24T00:13:48.848 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:48.669+0000 7f4105800700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f41001045e0 0x7f41001a0890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:13:48.849 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:48.669+0000 7f4105800700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f4100104f90 0x7f41001a0dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:13:48.849 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:48.669+0000 7f4105800700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f41001096c0 0x7f410019aa50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:13:48.849 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:48.669+0000 7f4105800700 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f41001146f0 con 0x7f4100104f90 2024-03-24T00:13:48.849 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:48.669+0000 7f4105800700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4100114570 con 0x7f41001045e0 2024-03-24T00:13:48.849 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:48.669+0000 7f4105800700 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4100114870 con 0x7f41001096c0 2024-03-24T00:13:48.849 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:48.669+0000 7f40fea00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f41001045e0 0x7f41001a0890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:13:48.849 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:48.669+0000 7f40fea00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f41001045e0 0x7f41001a0890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.118:42820/0 (socket says 172.21.15.118:42820) 2024-03-24T00:13:48.849 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:48.669+0000 7f40fea00700 1 -- 172.21.15.118:0/1768186446 learned_addr learned my addr 172.21.15.118:0/1768186446 (peer_addr_for_me v2:172.21.15.118:0/0) 2024-03-24T00:13:48.849 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:48.670+0000 7f40ff400700 1 --2- 172.21.15.118:0/1768186446 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f41001096c0 0x7f410019aa50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:13:48.849 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:48.670+0000 7f40fe000700 1 --2- 172.21.15.118:0/1768186446 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f4100104f90 0x7f41001a0dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:13:48.849 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:48.670+0000 7f40fea00700 1 -- 172.21.15.118:0/1768186446 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f41001096c0 msgr2=0x7f410019aa50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:13:48.849 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:48.670+0000 7f40fea00700 1 --2- 172.21.15.118:0/1768186446 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f41001096c0 0x7f410019aa50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:13:48.849 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:48.670+0000 7f40fea00700 1 -- 172.21.15.118:0/1768186446 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f4100104f90 msgr2=0x7f41001a0dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:13:48.850 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:48.670+0000 7f40fea00700 1 --2- 172.21.15.118:0/1768186446 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f4100104f90 0x7f41001a0dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:13:48.850 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:48.670+0000 7f40fea00700 1 -- 172.21.15.118:0/1768186446 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f410019b1b0 con 0x7f41001045e0 2024-03-24T00:13:48.850 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:48.670+0000 7f40fea00700 1 --2- 172.21.15.118:0/1768186446 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f41001045e0 0x7f41001a0890 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f40f000e9c0 tx=0x7f40f000ecd0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:13:48.850 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:48.670+0000 7f40f7400700 1 -- 172.21.15.118:0/1768186446 <== mon.2 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f40f000cbd0 con 0x7f41001045e0 2024-03-24T00:13:48.850 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:48.670+0000 7f40f7400700 1 -- 172.21.15.118:0/1768186446 <== mon.2 v2:172.21.15.189:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f40f000cd70 con 0x7f41001045e0 2024-03-24T00:13:48.850 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:48.670+0000 7f4105800700 1 -- 172.21.15.118:0/1768186446 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f410019b4a0 con 0x7f41001045e0 2024-03-24T00:13:48.850 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:48.670+0000 7f40f7400700 1 -- 172.21.15.118:0/1768186446 <== mon.2 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f40f0005660 con 0x7f41001045e0 2024-03-24T00:13:48.850 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:48.671+0000 7f4105800700 1 -- 172.21.15.118:0/1768186446 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f41001a7760 con 0x7f41001045e0 2024-03-24T00:13:48.850 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:48.672+0000 7f4105800700 1 -- 172.21.15.118:0/1768186446 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f40e0001030 con 0x7f41001045e0 2024-03-24T00:13:48.850 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:48.674+0000 7f40f7400700 1 -- 172.21.15.118:0/1768186446 <== mon.2 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 17) v1 ==== 137342+0+0 (secure 0 0 0) 0x7f40f0004830 con 0x7f41001045e0 2024-03-24T00:13:48.850 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:48.676+0000 7f40f7400700 1 --2- 172.21.15.118:0/1768186446 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7f40e40a29a0 0x7f40e40a4e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:13:48.850 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:48.676+0000 7f40f7400700 1 -- 172.21.15.118:0/1768186446 <== mon.2 v2:172.21.15.189:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 7979+0+0 (secure 0 0 0) 0x7f40f00ceb10 con 0x7f41001045e0 2024-03-24T00:13:48.850 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:48.676+0000 7f40fe000700 1 --2- 172.21.15.118:0/1768186446 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7f40e40a29a0 0x7f40e40a4e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:13:48.850 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:48.677+0000 7f40f7400700 1 --2- 172.21.15.118:0/1768186446 >> [v2:172.21.15.118:6802/557690061,v1:172.21.15.118:6803/557690061] conn(0x7f40e40ac180 0x7f40e40ae5a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:13:48.851 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:48.677+0000 7f40f7400700 1 -- 172.21.15.118:0/1768186446 --> [v2:172.21.15.118:6802/557690061,v1:172.21.15.118:6803/557690061] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f40f000e1a0 con 0x7f40e40ac180 2024-03-24T00:13:48.851 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:48.677+0000 7f40f7400700 1 -- 172.21.15.118:0/1768186446 <== mon.2 v2:172.21.15.189:3300/0 6 ==== mon_get_version_reply(handle=1 version=86) v2 ==== 24+0+0 (secure 0 0 0) 0x7f40f00d32a0 con 0x7f41001045e0 2024-03-24T00:13:48.851 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:48.677+0000 7f40ff400700 1 --2- 172.21.15.118:0/1768186446 >> [v2:172.21.15.118:6802/557690061,v1:172.21.15.118:6803/557690061] conn(0x7f40e40ac180 0x7f40e40ae5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:13:48.851 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:48.677+0000 7f40fe000700 1 --2- 172.21.15.118:0/1768186446 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7f40e40a29a0 0x7f40e40a4e60 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7f40ec0098c0 tx=0x7f40ec019040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:13:48.851 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:48.678+0000 7f40ff400700 1 --2- 172.21.15.118:0/1768186446 >> [v2:172.21.15.118:6802/557690061,v1:172.21.15.118:6803/557690061] conn(0x7f40e40ac180 0x7f40e40ae5a0 crc :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:13:48.851 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:48.678+0000 7f40f7400700 1 -- 172.21.15.118:0/1768186446 <== osd.0 v2:172.21.15.118:6802/557690061 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7f40f000e1a0 con 0x7f40e40ac180 2024-03-24T00:13:48.851 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:48.697+0000 7f4105800700 1 -- 172.21.15.118:0/1768186446 --> [v2:172.21.15.118:6802/557690061,v1:172.21.15.118:6803/557690061] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f40e0002e60 con 0x7f40e40ac180 2024-03-24T00:13:48.851 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:48.697+0000 7f40f7400700 1 -- 172.21.15.118:0/1768186446 <== osd.0 v2:172.21.15.118:6802/557690061 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7f40e0002e60 con 0x7f40e40ac180 2024-03-24T00:13:48.851 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:48.698+0000 7f4105800700 1 -- 172.21.15.118:0/1768186446 >> [v2:172.21.15.118:6802/557690061,v1:172.21.15.118:6803/557690061] conn(0x7f40e40ac180 msgr2=0x7f40e40ae5a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:13:48.851 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:48.698+0000 7f4105800700 1 --2- 172.21.15.118:0/1768186446 >> [v2:172.21.15.118:6802/557690061,v1:172.21.15.118:6803/557690061] conn(0x7f40e40ac180 0x7f40e40ae5a0 crc :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:13:48.851 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:48.698+0000 7f4105800700 1 -- 172.21.15.118:0/1768186446 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7f40e40a29a0 msgr2=0x7f40e40a4e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:13:48.851 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:48.698+0000 7f4105800700 1 --2- 172.21.15.118:0/1768186446 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7f40e40a29a0 0x7f40e40a4e60 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7f40ec0098c0 tx=0x7f40ec019040 comp rx=0 tx=0).stop 2024-03-24T00:13:48.851 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:48.698+0000 7f4105800700 1 -- 172.21.15.118:0/1768186446 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f41001045e0 msgr2=0x7f41001a0890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:13:48.852 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:48.698+0000 7f4105800700 1 --2- 172.21.15.118:0/1768186446 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f41001045e0 0x7f41001a0890 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f40f000e9c0 tx=0x7f40f000ecd0 comp rx=0 tx=0).stop 2024-03-24T00:13:48.852 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:48.698+0000 7f40ff400700 1 -- 172.21.15.118:0/1768186446 reap_dead start 2024-03-24T00:13:48.852 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:48.698+0000 7f4105800700 1 -- 172.21.15.118:0/1768186446 shutdown_connections 2024-03-24T00:13:48.852 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:48.698+0000 7f4105800700 1 -- 172.21.15.118:0/1768186446 >> 172.21.15.118:0/1768186446 conn(0x7f41000ffe90 msgr2=0x7f41001021e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:13:48.852 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:48.698+0000 7f4105800700 1 -- 172.21.15.118:0/1768186446 shutdown_connections 2024-03-24T00:13:48.852 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:48.698+0000 7f4105800700 1 -- 172.21.15.118:0/1768186446 wait complete. 2024-03-24T00:13:48.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:13:48 smithi118 ceph-mon[31535]: pgmap v337: 97 pgs: 97 active+clean; 644 KiB data, 344 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 255 B/s wr, 9 op/s 2024-03-24T00:13:49.066 INFO:teuthology.orchestra.run.smithi118.stdout:111669149763 2024-03-24T00:13:49.067 DEBUG:teuthology.orchestra.run.smithi118:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 shell --fsid d79a5420-e971-11ee-95cd-87774f69a715 -- ceph osd last-stat-seq osd.3 2024-03-24T00:13:49.203 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:13:48 smithi189 ceph-mon[36187]: pgmap v337: 97 pgs: 97 active+clean; 644 KiB data, 344 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 255 B/s wr, 9 op/s 2024-03-24T00:13:49.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:13:48 smithi193 ceph-mon[36878]: pgmap v337: 97 pgs: 97 active+clean; 644 KiB data, 344 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 255 B/s wr, 9 op/s 2024-03-24T00:13:49.295 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:49.294+0000 7f9e84000700 1 -- 172.21.15.118:0/2097323775 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9e7c102df0 msgr2=0x7f9e7c106e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:13:49.295 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:49.294+0000 7f9e84000700 1 --2- 172.21.15.118:0/2097323775 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9e7c102df0 0x7f9e7c106e30 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f9e68009990 tx=0x7f9e68009ca0 comp rx=0 tx=0).stop 2024-03-24T00:13:49.295 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:49.294+0000 7f9e84000700 1 -- 172.21.15.118:0/2097323775 shutdown_connections 2024-03-24T00:13:49.295 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:49.294+0000 7f9e84000700 1 --2- 172.21.15.118:0/2097323775 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f9e7c107510 0x7f9e7c10f0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:13:49.295 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:49.294+0000 7f9e84000700 1 --2- 172.21.15.118:0/2097323775 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f9e7c102440 0x7f9e7c102820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:13:49.295 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:49.294+0000 7f9e84000700 1 --2- 172.21.15.118:0/2097323775 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9e7c102df0 0x7f9e7c106e30 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:13:49.295 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:49.294+0000 7f9e84000700 1 -- 172.21.15.118:0/2097323775 >> 172.21.15.118:0/2097323775 conn(0x7f9e7c0792f0 msgr2=0x7f9e7c079700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:13:49.295 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:49.294+0000 7f9e84000700 1 -- 172.21.15.118:0/2097323775 shutdown_connections 2024-03-24T00:13:49.295 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:49.294+0000 7f9e84000700 1 -- 172.21.15.118:0/2097323775 wait complete. 2024-03-24T00:13:49.296 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:49.295+0000 7f9e84000700 1 Processor -- start 2024-03-24T00:13:49.296 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:49.296+0000 7f9e84000700 1 -- start start 2024-03-24T00:13:49.296 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:49.296+0000 7f9e84000700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9e7c102440 0x7f9e7c1a0880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:13:49.296 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:49.296+0000 7f9e84000700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f9e7c102df0 0x7f9e7c1a0dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:13:49.296 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:49.296+0000 7f9e84000700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f9e7c107510 0x7f9e7c19aa40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:13:49.296 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:49.296+0000 7f9e84000700 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9e7c112520 con 0x7f9e7c107510 2024-03-24T00:13:49.297 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:49.296+0000 7f9e84000700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9e7c1123a0 con 0x7f9e7c102440 2024-03-24T00:13:49.297 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:49.296+0000 7f9e84000700 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9e7c1126a0 con 0x7f9e7c102df0 2024-03-24T00:13:49.297 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:49.297+0000 7f9e81600700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9e7c102440 0x7f9e7c1a0880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:13:49.297 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:49.297+0000 7f9e81600700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9e7c102440 0x7f9e7c1a0880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.118:42850/0 (socket says 172.21.15.118:42850) 2024-03-24T00:13:49.297 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:49.297+0000 7f9e81600700 1 -- 172.21.15.118:0/2625398501 learned_addr learned my addr 172.21.15.118:0/2625398501 (peer_addr_for_me v2:172.21.15.118:0/0) 2024-03-24T00:13:49.298 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:49.297+0000 7f9e80c00700 1 --2- 172.21.15.118:0/2625398501 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f9e7c102df0 0x7f9e7c1a0dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:13:49.298 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:49.297+0000 7f9e82000700 1 --2- 172.21.15.118:0/2625398501 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f9e7c107510 0x7f9e7c19aa40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:13:49.298 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:49.297+0000 7f9e81600700 1 -- 172.21.15.118:0/2625398501 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f9e7c102df0 msgr2=0x7f9e7c1a0dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:13:49.298 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:49.297+0000 7f9e81600700 1 --2- 172.21.15.118:0/2625398501 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f9e7c102df0 0x7f9e7c1a0dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:13:49.298 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:49.297+0000 7f9e81600700 1 -- 172.21.15.118:0/2625398501 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f9e7c107510 msgr2=0x7f9e7c19aa40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:13:49.298 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:49.297+0000 7f9e81600700 1 --2- 172.21.15.118:0/2625398501 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f9e7c107510 0x7f9e7c19aa40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:13:49.298 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:49.297+0000 7f9e81600700 1 -- 172.21.15.118:0/2625398501 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9e7c19b1a0 con 0x7f9e7c102440 2024-03-24T00:13:49.298 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:49.298+0000 7f9e81600700 1 --2- 172.21.15.118:0/2625398501 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9e7c102440 0x7f9e7c1a0880 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f9e7000e9a0 tx=0x7f9e7000ed60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:13:49.298 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:49.298+0000 7f9e7a000700 1 -- 172.21.15.118:0/2625398501 <== mon.2 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9e7000cbd0 con 0x7f9e7c102440 2024-03-24T00:13:49.298 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:49.298+0000 7f9e7a000700 1 -- 172.21.15.118:0/2625398501 <== mon.2 v2:172.21.15.189:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f9e7000cd70 con 0x7f9e7c102440 2024-03-24T00:13:49.299 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:49.298+0000 7f9e7a000700 1 -- 172.21.15.118:0/2625398501 <== mon.2 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9e700187c0 con 0x7f9e7c102440 2024-03-24T00:13:49.299 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:49.298+0000 7f9e84000700 1 -- 172.21.15.118:0/2625398501 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9e7c19b490 con 0x7f9e7c102440 2024-03-24T00:13:49.299 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:49.298+0000 7f9e84000700 1 -- 172.21.15.118:0/2625398501 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9e7c1a7750 con 0x7f9e7c102440 2024-03-24T00:13:49.299 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:49.299+0000 7f9e84000700 1 -- 172.21.15.118:0/2625398501 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f9e7c10e740 con 0x7f9e7c102440 2024-03-24T00:13:49.302 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:49.303+0000 7f9e7a000700 1 -- 172.21.15.118:0/2625398501 <== mon.2 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 17) v1 ==== 137342+0+0 (secure 0 0 0) 0x7f9e70010430 con 0x7f9e7c102440 2024-03-24T00:13:49.304 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:49.305+0000 7f9e7a000700 1 --2- 172.21.15.118:0/2625398501 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7f9e640a29a0 0x7f9e640a4e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:13:49.305 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:49.305+0000 7f9e7a000700 1 -- 172.21.15.118:0/2625398501 <== mon.2 v2:172.21.15.189:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 7979+0+0 (secure 0 0 0) 0x7f9e700cedd0 con 0x7f9e7c102440 2024-03-24T00:13:49.305 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:49.305+0000 7f9e80c00700 1 --2- 172.21.15.118:0/2625398501 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7f9e640a29a0 0x7f9e640a4e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:13:49.305 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:49.306+0000 7f9e7a000700 1 --2- 172.21.15.118:0/2625398501 >> [v2:172.21.15.193:6816/3853800910,v1:172.21.15.193:6817/3853800910] conn(0x7f9e640ac180 0x7f9e640ae5a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:13:49.306 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:49.306+0000 7f9e7a000700 1 -- 172.21.15.118:0/2625398501 --> [v2:172.21.15.193:6816/3853800910,v1:172.21.15.193:6817/3853800910] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f9e640aec50 con 0x7f9e640ac180 2024-03-24T00:13:49.306 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:49.306+0000 7f9e7a000700 1 -- 172.21.15.118:0/2625398501 <== mon.2 v2:172.21.15.189:3300/0 6 ==== mon_get_version_reply(handle=1 version=86) v2 ==== 24+0+0 (secure 0 0 0) 0x7f9e700d2020 con 0x7f9e7c102440 2024-03-24T00:13:49.306 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:49.306+0000 7f9e82000700 1 --2- 172.21.15.118:0/2625398501 >> [v2:172.21.15.193:6816/3853800910,v1:172.21.15.193:6817/3853800910] conn(0x7f9e640ac180 0x7f9e640ae5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:13:49.306 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:49.307+0000 7f9e80c00700 1 --2- 172.21.15.118:0/2625398501 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7f9e640a29a0 0x7f9e640a4e60 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7f9e6800b2e0 tx=0x7f9e6801a040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:13:49.307 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:49.307+0000 7f9e82000700 1 --2- 172.21.15.118:0/2625398501 >> [v2:172.21.15.193:6816/3853800910,v1:172.21.15.193:6817/3853800910] conn(0x7f9e640ac180 0x7f9e640ae5a0 crc :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.8 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:13:49.307 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:49.308+0000 7f9e7a000700 1 -- 172.21.15.118:0/2625398501 <== osd.8 v2:172.21.15.193:6816/3853800910 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7f9e640aec50 con 0x7f9e640ac180 2024-03-24T00:13:49.343 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:49.343+0000 7f9e84000700 1 -- 172.21.15.118:0/2625398501 --> [v2:172.21.15.193:6816/3853800910,v1:172.21.15.193:6817/3853800910] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f9e7c103910 con 0x7f9e640ac180 2024-03-24T00:13:49.343 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:49.344+0000 7f9e7a000700 1 -- 172.21.15.118:0/2625398501 <== osd.8 v2:172.21.15.193:6816/3853800910 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+12 (crc 0 0 0) 0x7f9e7c103910 con 0x7f9e640ac180 2024-03-24T00:13:49.344 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:49.345+0000 7f9e84000700 1 -- 172.21.15.118:0/2625398501 >> [v2:172.21.15.193:6816/3853800910,v1:172.21.15.193:6817/3853800910] conn(0x7f9e640ac180 msgr2=0x7f9e640ae5a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:13:49.344 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:49.345+0000 7f9e84000700 1 --2- 172.21.15.118:0/2625398501 >> [v2:172.21.15.193:6816/3853800910,v1:172.21.15.193:6817/3853800910] conn(0x7f9e640ac180 0x7f9e640ae5a0 crc :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:13:49.345 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:49.345+0000 7f9e84000700 1 -- 172.21.15.118:0/2625398501 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7f9e640a29a0 msgr2=0x7f9e640a4e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:13:49.345 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:49.345+0000 7f9e84000700 1 --2- 172.21.15.118:0/2625398501 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7f9e640a29a0 0x7f9e640a4e60 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7f9e6800b2e0 tx=0x7f9e6801a040 comp rx=0 tx=0).stop 2024-03-24T00:13:49.345 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:49.345+0000 7f9e84000700 1 -- 172.21.15.118:0/2625398501 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9e7c102440 msgr2=0x7f9e7c1a0880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:13:49.345 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:49.345+0000 7f9e84000700 1 --2- 172.21.15.118:0/2625398501 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9e7c102440 0x7f9e7c1a0880 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f9e7000e9a0 tx=0x7f9e7000ed60 comp rx=0 tx=0).stop 2024-03-24T00:13:49.345 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:49.345+0000 7f9e82000700 1 -- 172.21.15.118:0/2625398501 reap_dead start 2024-03-24T00:13:49.346 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:49.345+0000 7f9e84000700 1 -- 172.21.15.118:0/2625398501 shutdown_connections 2024-03-24T00:13:49.346 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:49.345+0000 7f9e84000700 1 -- 172.21.15.118:0/2625398501 >> 172.21.15.118:0/2625398501 conn(0x7f9e7c0792f0 msgr2=0x7f9e7c100570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:13:49.346 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:49.345+0000 7f9e84000700 1 -- 172.21.15.118:0/2625398501 shutdown_connections 2024-03-24T00:13:49.346 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:49.345+0000 7f9e84000700 1 -- 172.21.15.118:0/2625398501 wait complete. 2024-03-24T00:13:50.265 INFO:teuthology.orchestra.run.smithi118.stdout:347892350998 2024-03-24T00:13:50.265 DEBUG:teuthology.orchestra.run.smithi118:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 shell --fsid d79a5420-e971-11ee-95cd-87774f69a715 -- ceph osd last-stat-seq osd.11 2024-03-24T00:13:50.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:13:50 smithi118 ceph-mon[31535]: pgmap v338: 97 pgs: 97 active+clean; 644 KiB data, 344 MiB used, 1.0 TiB / 1.0 TiB avail; 6.0 KiB/s rd, 255 B/s wr, 11 op/s 2024-03-24T00:13:51.012 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:13:50 smithi193 ceph-mon[36878]: pgmap v338: 97 pgs: 97 active+clean; 644 KiB data, 344 MiB used, 1.0 TiB / 1.0 TiB avail; 6.0 KiB/s rd, 255 B/s wr, 11 op/s 2024-03-24T00:13:51.203 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:13:50 smithi189 ceph-mon[36187]: pgmap v338: 97 pgs: 97 active+clean; 644 KiB data, 344 MiB used, 1.0 TiB / 1.0 TiB avail; 6.0 KiB/s rd, 255 B/s wr, 11 op/s 2024-03-24T00:13:51.348 INFO:teuthology.orchestra.run.smithi118.stdout:257698037800 2024-03-24T00:13:51.348 DEBUG:teuthology.orchestra.run.smithi118:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 shell --fsid d79a5420-e971-11ee-95cd-87774f69a715 -- ceph osd last-stat-seq osd.8 2024-03-24T00:13:52.550 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:52.545+0000 7fe487e00700 1 -- 172.21.15.118:0/1323411155 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fe480102440 msgr2=0x7fe480102820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:13:52.550 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:52.545+0000 7fe487e00700 1 --2- 172.21.15.118:0/1323411155 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fe480102440 0x7fe480102820 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7fe4740099c0 tx=0x7fe474009cd0 comp rx=0 tx=0).stop 2024-03-24T00:13:52.550 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:52.546+0000 7fe487e00700 1 -- 172.21.15.118:0/1323411155 shutdown_connections 2024-03-24T00:13:52.550 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:52.546+0000 7fe487e00700 1 --2- 172.21.15.118:0/1323411155 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fe480107510 0x7fe48010f0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:13:52.550 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:52.546+0000 7fe487e00700 1 --2- 172.21.15.118:0/1323411155 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fe480102440 0x7fe480102820 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:13:52.551 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:52.546+0000 7fe487e00700 1 --2- 172.21.15.118:0/1323411155 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe480102df0 0x7fe480106e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:13:52.551 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:52.546+0000 7fe487e00700 1 -- 172.21.15.118:0/1323411155 >> 172.21.15.118:0/1323411155 conn(0x7fe4800792f0 msgr2=0x7fe480079700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:13:52.551 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:52.546+0000 7fe487e00700 1 -- 172.21.15.118:0/1323411155 shutdown_connections 2024-03-24T00:13:52.551 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:52.546+0000 7fe487e00700 1 -- 172.21.15.118:0/1323411155 wait complete. 2024-03-24T00:13:52.551 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:52.547+0000 7fe487e00700 1 Processor -- start 2024-03-24T00:13:52.552 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:52.547+0000 7fe487e00700 1 -- start start 2024-03-24T00:13:52.552 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:52.547+0000 7fe487e00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe480102440 0x7fe4801a0900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:13:52.552 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:52.547+0000 7fe487e00700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fe480102df0 0x7fe4801a0e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:13:52.552 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:52.547+0000 7fe487e00700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fe480107510 0x7fe48019aac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:13:52.552 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:52.547+0000 7fe487e00700 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe480112540 con 0x7fe480102df0 2024-03-24T00:13:52.552 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:52.547+0000 7fe487e00700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe4801123c0 con 0x7fe480102440 2024-03-24T00:13:52.553 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:52.547+0000 7fe487e00700 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe4801126c0 con 0x7fe480107510 2024-03-24T00:13:52.553 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:52.548+0000 7fe485400700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe480102440 0x7fe4801a0900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:13:52.553 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:52.548+0000 7fe485400700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe480102440 0x7fe4801a0900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.118:42866/0 (socket says 172.21.15.118:42866) 2024-03-24T00:13:52.553 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:52.548+0000 7fe485400700 1 -- 172.21.15.118:0/1415332210 learned_addr learned my addr 172.21.15.118:0/1415332210 (peer_addr_for_me v2:172.21.15.118:0/0) 2024-03-24T00:13:52.553 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:52.548+0000 7fe485e00700 1 --2- 172.21.15.118:0/1415332210 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fe480107510 0x7fe48019aac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:13:52.553 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:52.548+0000 7fe484a00700 1 --2- 172.21.15.118:0/1415332210 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fe480102df0 0x7fe4801a0e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:13:52.554 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:52.548+0000 7fe485400700 1 -- 172.21.15.118:0/1415332210 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fe480107510 msgr2=0x7fe48019aac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:13:52.554 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:52.548+0000 7fe485400700 1 --2- 172.21.15.118:0/1415332210 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fe480107510 0x7fe48019aac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:13:52.554 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:52.548+0000 7fe485400700 1 -- 172.21.15.118:0/1415332210 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fe480102df0 msgr2=0x7fe4801a0e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:13:52.554 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:52.548+0000 7fe485400700 1 --2- 172.21.15.118:0/1415332210 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fe480102df0 0x7fe4801a0e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:13:52.554 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:52.548+0000 7fe485400700 1 -- 172.21.15.118:0/1415332210 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe48019b220 con 0x7fe480102440 2024-03-24T00:13:52.554 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:52.548+0000 7fe484a00700 1 --2- 172.21.15.118:0/1415332210 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fe480102df0 0x7fe4801a0e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T00:13:52.555 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:52.548+0000 7fe485400700 1 --2- 172.21.15.118:0/1415332210 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe480102440 0x7fe4801a0900 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7fe474005910 tx=0x7fe47400f690 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:13:52.555 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:52.549+0000 7fe47e000700 1 -- 172.21.15.118:0/1415332210 <== mon.2 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe474027070 con 0x7fe480102440 2024-03-24T00:13:52.555 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:52.549+0000 7fe47e000700 1 -- 172.21.15.118:0/1415332210 <== mon.2 v2:172.21.15.189:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fe474003680 con 0x7fe480102440 2024-03-24T00:13:52.555 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:52.549+0000 7fe487e00700 1 -- 172.21.15.118:0/1415332210 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe48019b4b0 con 0x7fe480102440 2024-03-24T00:13:52.555 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:52.549+0000 7fe47e000700 1 -- 172.21.15.118:0/1415332210 <== mon.2 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe4740173f0 con 0x7fe480102440 2024-03-24T00:13:52.556 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:52.549+0000 7fe487e00700 1 -- 172.21.15.118:0/1415332210 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe4801a77d0 con 0x7fe480102440 2024-03-24T00:13:52.558 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:52.550+0000 7fe487e00700 1 -- 172.21.15.118:0/1415332210 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7fe4800756a0 con 0x7fe480102440 2024-03-24T00:13:52.558 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:52.552+0000 7fe47e000700 1 -- 172.21.15.118:0/1415332210 <== mon.2 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 17) v1 ==== 137342+0+0 (secure 0 0 0) 0x7fe47400fcd0 con 0x7fe480102440 2024-03-24T00:13:52.558 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:52.553+0000 7fe47e000700 1 --2- 172.21.15.118:0/1415332210 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7fe4680a2920 0x7fe4680a4de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:13:52.558 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:52.553+0000 7fe47e000700 1 -- 172.21.15.118:0/1415332210 <== mon.2 v2:172.21.15.189:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 7979+0+0 (secure 0 0 0) 0x7fe4740cfb30 con 0x7fe480102440 2024-03-24T00:13:52.559 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:52.553+0000 7fe484a00700 1 --2- 172.21.15.118:0/1415332210 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7fe4680a2920 0x7fe4680a4de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:13:52.559 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:52.553+0000 7fe47e000700 1 --2- 172.21.15.118:0/1415332210 >> [v2:172.21.15.189:6824/2112195760,v1:172.21.15.189:6825/2112195760] conn(0x7fe4680ac100 0x7fe4680ae520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:13:52.559 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:52.553+0000 7fe47e000700 1 -- 172.21.15.118:0/1415332210 --> [v2:172.21.15.189:6824/2112195760,v1:172.21.15.189:6825/2112195760] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fe4680aebd0 con 0x7fe4680ac100 2024-03-24T00:13:52.559 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:52.553+0000 7fe47e000700 1 -- 172.21.15.118:0/1415332210 <== mon.2 v2:172.21.15.189:3300/0 6 ==== mon_get_version_reply(handle=1 version=86) v2 ==== 24+0+0 (secure 0 0 0) 0x7fe4740cfee0 con 0x7fe480102440 2024-03-24T00:13:52.559 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:52.554+0000 7fe485e00700 1 --2- 172.21.15.118:0/1415332210 >> [v2:172.21.15.189:6824/2112195760,v1:172.21.15.189:6825/2112195760] conn(0x7fe4680ac100 0x7fe4680ae520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:13:52.559 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:52.554+0000 7fe484a00700 1 --2- 172.21.15.118:0/1415332210 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7fe4680a2920 0x7fe4680a4de0 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7fe480102170 tx=0x7fe46c009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:13:52.560 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:52.554+0000 7fe485e00700 1 --2- 172.21.15.118:0/1415332210 >> [v2:172.21.15.189:6824/2112195760,v1:172.21.15.189:6825/2112195760] conn(0x7fe4680ac100 0x7fe4680ae520 crc :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.10 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:13:52.560 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:52.555+0000 7fe47e000700 1 -- 172.21.15.118:0/1415332210 <== osd.10 v2:172.21.15.189:6824/2112195760 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7fe4680aebd0 con 0x7fe4680ac100 2024-03-24T00:13:52.574 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:52.573+0000 7fe487e00700 1 -- 172.21.15.118:0/1415332210 --> [v2:172.21.15.189:6824/2112195760,v1:172.21.15.189:6825/2112195760] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7fe480103910 con 0x7fe4680ac100 2024-03-24T00:13:52.574 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:52.573+0000 7fe47e000700 1 -- 172.21.15.118:0/1415332210 <== osd.10 v2:172.21.15.189:6824/2112195760 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+12 (crc 0 0 0) 0x7fe480103910 con 0x7fe4680ac100 2024-03-24T00:13:52.575 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:52.573+0000 7fe487e00700 1 -- 172.21.15.118:0/1415332210 >> [v2:172.21.15.189:6824/2112195760,v1:172.21.15.189:6825/2112195760] conn(0x7fe4680ac100 msgr2=0x7fe4680ae520 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:13:52.576 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:52.573+0000 7fe487e00700 1 --2- 172.21.15.118:0/1415332210 >> [v2:172.21.15.189:6824/2112195760,v1:172.21.15.189:6825/2112195760] conn(0x7fe4680ac100 0x7fe4680ae520 crc :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:13:52.576 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:52.574+0000 7fe487e00700 1 -- 172.21.15.118:0/1415332210 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7fe4680a2920 msgr2=0x7fe4680a4de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:13:52.576 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:52.574+0000 7fe487e00700 1 --2- 172.21.15.118:0/1415332210 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7fe4680a2920 0x7fe4680a4de0 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7fe480102170 tx=0x7fe46c009260 comp rx=0 tx=0).stop 2024-03-24T00:13:52.576 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:52.574+0000 7fe487e00700 1 -- 172.21.15.118:0/1415332210 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe480102440 msgr2=0x7fe4801a0900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:13:52.577 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:52.574+0000 7fe487e00700 1 --2- 172.21.15.118:0/1415332210 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe480102440 0x7fe4801a0900 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7fe474005910 tx=0x7fe47400f690 comp rx=0 tx=0).stop 2024-03-24T00:13:52.577 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:52.574+0000 7fe485e00700 1 -- 172.21.15.118:0/1415332210 reap_dead start 2024-03-24T00:13:52.577 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:52.574+0000 7fe487e00700 1 -- 172.21.15.118:0/1415332210 shutdown_connections 2024-03-24T00:13:52.577 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:52.574+0000 7fe487e00700 1 -- 172.21.15.118:0/1415332210 >> 172.21.15.118:0/1415332210 conn(0x7fe4800792f0 msgr2=0x7fe480100590 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:13:52.577 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:52.574+0000 7fe487e00700 1 -- 172.21.15.118:0/1415332210 shutdown_connections 2024-03-24T00:13:52.577 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:52.574+0000 7fe487e00700 1 -- 172.21.15.118:0/1415332210 wait complete. 2024-03-24T00:13:53.020 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:13:52 smithi118 ceph-mon[31535]: pgmap v339: 97 pgs: 97 active+clean; 644 KiB data, 344 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-24T00:13:53.203 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:13:52 smithi189 ceph-mon[36187]: pgmap v339: 97 pgs: 97 active+clean; 644 KiB data, 344 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-24T00:13:53.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:13:52 smithi193 ceph-mon[36878]: pgmap v339: 97 pgs: 97 active+clean; 644 KiB data, 344 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-24T00:13:53.410 INFO:teuthology.orchestra.run.smithi118.stdout:55834574926 2024-03-24T00:13:53.411 DEBUG:teuthology.orchestra.run.smithi118:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 shell --fsid d79a5420-e971-11ee-95cd-87774f69a715 -- ceph osd last-stat-seq osd.1 2024-03-24T00:13:54.153 INFO:teuthology.orchestra.run.smithi118.stdout:34359738453 2024-03-24T00:13:54.153 DEBUG:teuthology.orchestra.run.smithi118:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 shell --fsid d79a5420-e971-11ee-95cd-87774f69a715 -- ceph osd last-stat-seq osd.0 2024-03-24T00:13:54.581 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:54.546+0000 7fecc5a00700 1 -- 172.21.15.118:0/232115562 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fecc0102440 msgr2=0x7fecc0102820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:13:54.581 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:54.546+0000 7fecc5a00700 1 --2- 172.21.15.118:0/232115562 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fecc0102440 0x7fecc0102820 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7fecb00099c0 tx=0x7fecb0009cd0 comp rx=0 tx=0).stop 2024-03-24T00:13:54.581 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:54.546+0000 7fecc5a00700 1 -- 172.21.15.118:0/232115562 shutdown_connections 2024-03-24T00:13:54.581 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:54.546+0000 7fecc5a00700 1 --2- 172.21.15.118:0/232115562 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fecc0107510 0x7fecc010f0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:13:54.581 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:54.546+0000 7fecc5a00700 1 --2- 172.21.15.118:0/232115562 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fecc0102440 0x7fecc0102820 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:13:54.582 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:54.546+0000 7fecc5a00700 1 --2- 172.21.15.118:0/232115562 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fecc0102df0 0x7fecc0106e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:13:54.582 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:54.546+0000 7fecc5a00700 1 -- 172.21.15.118:0/232115562 >> 172.21.15.118:0/232115562 conn(0x7fecc00792f0 msgr2=0x7fecc0079700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:13:54.582 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:54.546+0000 7fecc5a00700 1 -- 172.21.15.118:0/232115562 shutdown_connections 2024-03-24T00:13:54.582 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:54.546+0000 7fecc5a00700 1 -- 172.21.15.118:0/232115562 wait complete. 2024-03-24T00:13:54.582 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:54.548+0000 7fecc5a00700 1 Processor -- start 2024-03-24T00:13:54.582 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:54.548+0000 7fecc5a00700 1 -- start start 2024-03-24T00:13:54.582 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:54.549+0000 7fecc5a00700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fecc0102440 0x7fecc0074490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:13:54.582 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:54.549+0000 7fecc5a00700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fecc0102df0 0x7fecc00749d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:13:54.583 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:54.549+0000 7fecc5a00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fecc0107510 0x7fecc0074f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:13:54.583 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:54.549+0000 7fecc5a00700 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fecc0112540 con 0x7fecc0102440 2024-03-24T00:13:54.583 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:54.549+0000 7fecc5a00700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fecc01123c0 con 0x7fecc0107510 2024-03-24T00:13:54.583 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:54.549+0000 7fecc5a00700 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fecc01126c0 con 0x7fecc0102df0 2024-03-24T00:13:54.583 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:54.549+0000 7fecbe000700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fecc0102df0 0x7fecc00749d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:13:54.583 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:54.549+0000 7fecbe000700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fecc0102df0 0x7fecc00749d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.118:43962/0 (socket says 172.21.15.118:43962) 2024-03-24T00:13:54.583 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:54.549+0000 7fecbe000700 1 -- 172.21.15.118:0/3221827999 learned_addr learned my addr 172.21.15.118:0/3221827999 (peer_addr_for_me v2:172.21.15.118:0/0) 2024-03-24T00:13:54.583 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:54.549+0000 7fecbf400700 1 --2- 172.21.15.118:0/3221827999 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fecc0107510 0x7fecc0074f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:13:54.583 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:54.549+0000 7fecbea00700 1 --2- 172.21.15.118:0/3221827999 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fecc0102440 0x7fecc0074490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:13:54.584 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:54.549+0000 7fecbe000700 1 -- 172.21.15.118:0/3221827999 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fecc0107510 msgr2=0x7fecc0074f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:13:54.584 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:54.549+0000 7fecbe000700 1 --2- 172.21.15.118:0/3221827999 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fecc0107510 0x7fecc0074f10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:13:54.584 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:54.549+0000 7fecbe000700 1 -- 172.21.15.118:0/3221827999 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fecc0102440 msgr2=0x7fecc0074490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:13:54.584 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:54.549+0000 7fecbe000700 1 --2- 172.21.15.118:0/3221827999 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fecc0102440 0x7fecc0074490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:13:54.584 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:54.550+0000 7fecbe000700 1 -- 172.21.15.118:0/3221827999 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fecc01a6ef0 con 0x7fecc0102df0 2024-03-24T00:13:54.584 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:54.550+0000 7fecbf400700 1 --2- 172.21.15.118:0/3221827999 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fecc0107510 0x7fecc0074f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T00:13:54.584 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:54.550+0000 7fecbe000700 1 --2- 172.21.15.118:0/3221827999 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fecc0102df0 0x7fecc00749d0 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7fecac00d980 tx=0x7fecac00dc90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:13:54.584 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:54.550+0000 7fecb7400700 1 -- 172.21.15.118:0/3221827999 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fecac017070 con 0x7fecc0102df0 2024-03-24T00:13:54.584 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:54.550+0000 7fecb7400700 1 -- 172.21.15.118:0/3221827999 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fecac00fd70 con 0x7fecc0102df0 2024-03-24T00:13:54.584 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:54.550+0000 7fecc5a00700 1 -- 172.21.15.118:0/3221827999 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fecc01a71e0 con 0x7fecc0102df0 2024-03-24T00:13:54.585 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:54.550+0000 7fecb7400700 1 -- 172.21.15.118:0/3221827999 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fecac00e3b0 con 0x7fecc0102df0 2024-03-24T00:13:54.585 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:54.550+0000 7fecc5a00700 1 -- 172.21.15.118:0/3221827999 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fecc01a7720 con 0x7fecc0102df0 2024-03-24T00:13:54.585 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:54.551+0000 7fecc5a00700 1 -- 172.21.15.118:0/3221827999 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7fecc00628a0 con 0x7fecc0102df0 2024-03-24T00:13:54.585 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:54.553+0000 7fecb7400700 1 -- 172.21.15.118:0/3221827999 <== mon.1 v2:172.21.15.193:3300/0 4 ==== mgrmap(e 17) v1 ==== 137342+0+0 (secure 0 0 0) 0x7fecac00f890 con 0x7fecc0102df0 2024-03-24T00:13:54.585 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:54.554+0000 7fecb7400700 1 --2- 172.21.15.118:0/3221827999 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7feca40a2920 0x7feca40a4de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:13:54.585 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:54.555+0000 7fecb7400700 1 -- 172.21.15.118:0/3221827999 <== mon.1 v2:172.21.15.193:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 7979+0+0 (secure 0 0 0) 0x7fecac0ce5b0 con 0x7fecc0102df0 2024-03-24T00:13:54.585 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:54.555+0000 7fecbea00700 1 --2- 172.21.15.118:0/3221827999 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7feca40a2920 0x7feca40a4de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:13:54.585 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:54.555+0000 7fecb7400700 1 --2- 172.21.15.118:0/3221827999 >> [v2:172.21.15.118:6826/3248293936,v1:172.21.15.118:6827/3248293936] conn(0x7feca40ac100 0x7feca40ae520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:13:54.585 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:54.555+0000 7fecb7400700 1 -- 172.21.15.118:0/3221827999 --> [v2:172.21.15.118:6826/3248293936,v1:172.21.15.118:6827/3248293936] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fecac014ec0 con 0x7feca40ac100 2024-03-24T00:13:54.585 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:54.555+0000 7fecb7400700 1 -- 172.21.15.118:0/3221827999 <== mon.1 v2:172.21.15.193:3300/0 6 ==== mon_get_version_reply(handle=1 version=86) v2 ==== 24+0+0 (secure 0 0 0) 0x7fecac0d5050 con 0x7fecc0102df0 2024-03-24T00:13:54.585 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:54.555+0000 7fecbf400700 1 --2- 172.21.15.118:0/3221827999 >> [v2:172.21.15.118:6826/3248293936,v1:172.21.15.118:6827/3248293936] conn(0x7feca40ac100 0x7feca40ae520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:13:54.585 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:54.555+0000 7fecbea00700 1 --2- 172.21.15.118:0/3221827999 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7feca40a2920 0x7feca40a4de0 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7fecb0000c00 tx=0x7fecb0005fb0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:13:54.585 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:54.556+0000 7fecbf400700 1 --2- 172.21.15.118:0/3221827999 >> [v2:172.21.15.118:6826/3248293936,v1:172.21.15.118:6827/3248293936] conn(0x7feca40ac100 0x7feca40ae520 crc :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.9 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:13:54.586 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:54.557+0000 7fecb7400700 1 -- 172.21.15.118:0/3221827999 <== osd.9 v2:172.21.15.118:6826/3248293936 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7fecac014ec0 con 0x7feca40ac100 2024-03-24T00:13:54.586 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:54.576+0000 7fecc5a00700 1 -- 172.21.15.118:0/3221827999 --> [v2:172.21.15.118:6826/3248293936,v1:172.21.15.118:6827/3248293936] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7fecc0062ca0 con 0x7feca40ac100 2024-03-24T00:13:54.586 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:54.576+0000 7fecb7400700 1 -- 172.21.15.118:0/3221827999 <== osd.9 v2:172.21.15.118:6826/3248293936 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+12 (crc 0 0 0) 0x7fecc0062ca0 con 0x7feca40ac100 2024-03-24T00:13:54.586 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:54.577+0000 7fecc5a00700 1 -- 172.21.15.118:0/3221827999 >> [v2:172.21.15.118:6826/3248293936,v1:172.21.15.118:6827/3248293936] conn(0x7feca40ac100 msgr2=0x7feca40ae520 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:13:54.586 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:54.577+0000 7fecc5a00700 1 --2- 172.21.15.118:0/3221827999 >> [v2:172.21.15.118:6826/3248293936,v1:172.21.15.118:6827/3248293936] conn(0x7feca40ac100 0x7feca40ae520 crc :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:13:54.586 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:54.577+0000 7fecc5a00700 1 -- 172.21.15.118:0/3221827999 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7feca40a2920 msgr2=0x7feca40a4de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:13:54.586 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:54.577+0000 7fecc5a00700 1 --2- 172.21.15.118:0/3221827999 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7feca40a2920 0x7feca40a4de0 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7fecb0000c00 tx=0x7fecb0005fb0 comp rx=0 tx=0).stop 2024-03-24T00:13:54.586 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:54.577+0000 7fecc5a00700 1 -- 172.21.15.118:0/3221827999 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fecc0102df0 msgr2=0x7fecc00749d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:13:54.586 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:54.577+0000 7fecc5a00700 1 --2- 172.21.15.118:0/3221827999 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fecc0102df0 0x7fecc00749d0 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7fecac00d980 tx=0x7fecac00dc90 comp rx=0 tx=0).stop 2024-03-24T00:13:54.586 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:54.577+0000 7fecbf400700 1 -- 172.21.15.118:0/3221827999 reap_dead start 2024-03-24T00:13:54.586 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:54.577+0000 7fecc5a00700 1 -- 172.21.15.118:0/3221827999 shutdown_connections 2024-03-24T00:13:54.586 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:54.577+0000 7fecc5a00700 1 -- 172.21.15.118:0/3221827999 >> 172.21.15.118:0/3221827999 conn(0x7fecc00792f0 msgr2=0x7fecc0100450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:13:54.586 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:54.577+0000 7fecc5a00700 1 -- 172.21.15.118:0/3221827999 shutdown_connections 2024-03-24T00:13:54.587 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:54.577+0000 7fecc5a00700 1 -- 172.21.15.118:0/3221827999 wait complete. 2024-03-24T00:13:54.986 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:13:54 smithi118 ceph-mon[31535]: pgmap v340: 97 pgs: 97 active+clean; 644 KiB data, 344 MiB used, 1.0 TiB / 1.0 TiB avail; 6.0 KiB/s rd, 11 op/s 2024-03-24T00:13:55.122 INFO:teuthology.orchestra.run.smithi118.stderr:Inferring config /var/lib/ceph/d79a5420-e971-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T00:13:55.165 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:55.164+0000 7f5216400700 1 -- 172.21.15.118:0/1354498050 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f5210104b90 msgr2=0x7f5210105020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:13:55.167 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:55.165+0000 7f5216400700 1 --2- 172.21.15.118:0/1354498050 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f5210104b90 0x7f5210105020 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f520800b0a0 tx=0x7f520800b3b0 comp rx=0 tx=0).stop 2024-03-24T00:13:55.167 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:55.165+0000 7f5216400700 1 -- 172.21.15.118:0/1354498050 shutdown_connections 2024-03-24T00:13:55.167 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:55.165+0000 7f5216400700 1 --2- 172.21.15.118:0/1354498050 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5210107080 0x7f5210069e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:13:55.167 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:55.165+0000 7f5216400700 1 --2- 172.21.15.118:0/1354498050 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f5210104b90 0x7f5210105020 unknown :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:13:55.167 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:55.165+0000 7f5216400700 1 --2- 172.21.15.118:0/1354498050 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f521006a380 0x7f521006a7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:13:55.167 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:55.165+0000 7f5216400700 1 -- 172.21.15.118:0/1354498050 >> 172.21.15.118:0/1354498050 conn(0x7f5210077fc0 msgr2=0x7f52100783d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:13:55.167 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:55.165+0000 7f5216400700 1 -- 172.21.15.118:0/1354498050 shutdown_connections 2024-03-24T00:13:55.167 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:55.165+0000 7f5216400700 1 -- 172.21.15.118:0/1354498050 wait complete. 2024-03-24T00:13:55.168 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:55.167+0000 7f5216400700 1 Processor -- start 2024-03-24T00:13:55.168 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:55.167+0000 7f5216400700 1 -- start start 2024-03-24T00:13:55.169 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:55.168+0000 7f5216400700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f521006a380 0x7f5210073880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:13:55.169 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:55.168+0000 7f5216400700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f5210104b90 0x7f5210073dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:13:55.169 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:55.168+0000 7f5216400700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5210107080 0x7f521006caa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:13:55.170 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:55.168+0000 7f5216400700 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5210118b00 con 0x7f5210104b90 2024-03-24T00:13:55.170 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:55.168+0000 7f5216400700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5210118980 con 0x7f5210107080 2024-03-24T00:13:55.170 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:55.168+0000 7f5216400700 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5210118c80 con 0x7f521006a380 2024-03-24T00:13:55.170 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:55.168+0000 7f520ea00700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f5210104b90 0x7f5210073dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:13:55.170 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:55.168+0000 7f520ea00700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f5210104b90 0x7f5210073dc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.118:3300/0 says I am v2:172.21.15.118:53590/0 (socket says 172.21.15.118:53590) 2024-03-24T00:13:55.171 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:55.169+0000 7f520ea00700 1 -- 172.21.15.118:0/4089495462 learned_addr learned my addr 172.21.15.118:0/4089495462 (peer_addr_for_me v2:172.21.15.118:0/0) 2024-03-24T00:13:55.171 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:55.169+0000 7f520ea00700 1 -- 172.21.15.118:0/4089495462 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f521006a380 msgr2=0x7f5210073880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:13:55.174 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:55.170+0000 7f520f400700 1 --2- 172.21.15.118:0/4089495462 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f521006a380 0x7f5210073880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:13:55.174 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:55.170+0000 7f520fe00700 1 --2- 172.21.15.118:0/4089495462 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5210107080 0x7f521006caa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:13:55.174 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:55.170+0000 7f520ea00700 1 --2- 172.21.15.118:0/4089495462 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f521006a380 0x7f5210073880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:13:55.174 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:55.170+0000 7f520ea00700 1 -- 172.21.15.118:0/4089495462 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5210107080 msgr2=0x7f521006caa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:13:55.174 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:55.170+0000 7f520ea00700 1 --2- 172.21.15.118:0/4089495462 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5210107080 0x7f521006caa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:13:55.174 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:55.170+0000 7f520ea00700 1 -- 172.21.15.118:0/4089495462 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f521006d3a0 con 0x7f5210104b90 2024-03-24T00:13:55.174 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:55.170+0000 7f520fe00700 1 --2- 172.21.15.118:0/4089495462 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5210107080 0x7f521006caa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-24T00:13:55.174 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:55.171+0000 7f520f400700 1 --2- 172.21.15.118:0/4089495462 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f521006a380 0x7f5210073880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-24T00:13:55.174 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:55.171+0000 7f520ea00700 1 --2- 172.21.15.118:0/4089495462 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f5210104b90 0x7f5210073dc0 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f51f800e9a0 tx=0x7f51f800ed60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:13:55.175 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:55.171+0000 7f5207e00700 1 -- 172.21.15.118:0/4089495462 <== mon.0 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f51f800cbd0 con 0x7f5210104b90 2024-03-24T00:13:55.175 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:55.171+0000 7f5207e00700 1 -- 172.21.15.118:0/4089495462 <== mon.0 v2:172.21.15.118:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f51f800cd70 con 0x7f5210104b90 2024-03-24T00:13:55.175 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:55.171+0000 7f5216400700 1 -- 172.21.15.118:0/4089495462 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f521006d690 con 0x7f5210104b90 2024-03-24T00:13:55.175 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:55.172+0000 7f5216400700 1 -- 172.21.15.118:0/4089495462 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5210071630 con 0x7f5210104b90 2024-03-24T00:13:55.175 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:55.172+0000 7f5207e00700 1 -- 172.21.15.118:0/4089495462 <== mon.0 v2:172.21.15.118:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f51f80187c0 con 0x7f5210104b90 2024-03-24T00:13:55.175 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:55.173+0000 7f5216400700 1 -- 172.21.15.118:0/4089495462 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f51ec001060 con 0x7f5210104b90 2024-03-24T00:13:55.177 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:55.176+0000 7f5207e00700 1 -- 172.21.15.118:0/4089495462 <== mon.0 v2:172.21.15.118:3300/0 4 ==== mgrmap(e 17) v1 ==== 137342+0+0 (secure 0 0 0) 0x7f51f8018960 con 0x7f5210104b90 2024-03-24T00:13:55.180 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:55.179+0000 7f5207e00700 1 --2- 172.21.15.118:0/4089495462 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7f51f40a2a70 0x7f51f40a4f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:13:55.181 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:55.179+0000 7f5207e00700 1 -- 172.21.15.118:0/4089495462 <== mon.0 v2:172.21.15.118:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 7979+0+0 (secure 0 0 0) 0x7f51f80cf480 con 0x7f5210104b90 2024-03-24T00:13:55.181 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:55.179+0000 7f5207e00700 1 --2- 172.21.15.118:0/4089495462 >> [v2:172.21.15.193:6800/3699076971,v1:172.21.15.193:6801/3699076971] conn(0x7f51f40ac250 0x7f51f40ae670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:13:55.181 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:55.180+0000 7f5207e00700 1 -- 172.21.15.118:0/4089495462 --> [v2:172.21.15.193:6800/3699076971,v1:172.21.15.193:6801/3699076971] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f51f40aed20 con 0x7f51f40ac250 2024-03-24T00:13:55.181 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:55.180+0000 7f5207e00700 1 -- 172.21.15.118:0/4089495462 <== mon.0 v2:172.21.15.118:3300/0 6 ==== mon_get_version_reply(handle=1 version=86) v2 ==== 24+0+0 (secure 0 0 0) 0x7f51f80cf830 con 0x7f5210104b90 2024-03-24T00:13:55.185 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:55.183+0000 7f520f400700 1 --2- 172.21.15.118:0/4089495462 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7f51f40a2a70 0x7f51f40a4f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:13:55.185 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:55.183+0000 7f520fe00700 1 --2- 172.21.15.118:0/4089495462 >> [v2:172.21.15.193:6800/3699076971,v1:172.21.15.193:6801/3699076971] conn(0x7f51f40ac250 0x7f51f40ae670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:13:55.185 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:55.184+0000 7f520fe00700 1 --2- 172.21.15.118:0/4089495462 >> [v2:172.21.15.193:6800/3699076971,v1:172.21.15.193:6801/3699076971] conn(0x7f51f40ac250 0x7f51f40ae670 crc :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:13:55.185 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:55.184+0000 7f520f400700 1 --2- 172.21.15.118:0/4089495462 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7f51f40a2a70 0x7f51f40a4f30 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7f5200006fd0 tx=0x7f5200008040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:13:55.187 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:55.185+0000 7f5207e00700 1 -- 172.21.15.118:0/4089495462 <== osd.2 v2:172.21.15.193:6800/3699076971 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7f51f40aed20 con 0x7f51f40ac250 2024-03-24T00:13:55.203 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:13:54 smithi189 ceph-mon[36187]: pgmap v340: 97 pgs: 97 active+clean; 644 KiB data, 344 MiB used, 1.0 TiB / 1.0 TiB avail; 6.0 KiB/s rd, 11 op/s 2024-03-24T00:13:55.210 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:55.209+0000 7f5216400700 1 -- 172.21.15.118:0/4089495462 --> [v2:172.21.15.193:6800/3699076971,v1:172.21.15.193:6801/3699076971] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f51ec002e90 con 0x7f51f40ac250 2024-03-24T00:13:55.211 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:55.210+0000 7f5207e00700 1 -- 172.21.15.118:0/4089495462 <== osd.2 v2:172.21.15.193:6800/3699076971 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7f51ec002e90 con 0x7f51f40ac250 2024-03-24T00:13:55.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:13:54 smithi193 ceph-mon[36878]: pgmap v340: 97 pgs: 97 active+clean; 644 KiB data, 344 MiB used, 1.0 TiB / 1.0 TiB avail; 6.0 KiB/s rd, 11 op/s 2024-03-24T00:13:55.212 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:55.210+0000 7f5216400700 1 -- 172.21.15.118:0/4089495462 >> [v2:172.21.15.193:6800/3699076971,v1:172.21.15.193:6801/3699076971] conn(0x7f51f40ac250 msgr2=0x7f51f40ae670 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:13:55.212 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:55.210+0000 7f5216400700 1 --2- 172.21.15.118:0/4089495462 >> [v2:172.21.15.193:6800/3699076971,v1:172.21.15.193:6801/3699076971] conn(0x7f51f40ac250 0x7f51f40ae670 crc :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:13:55.212 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:55.210+0000 7f5216400700 1 -- 172.21.15.118:0/4089495462 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7f51f40a2a70 msgr2=0x7f51f40a4f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:13:55.212 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:55.210+0000 7f5216400700 1 --2- 172.21.15.118:0/4089495462 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7f51f40a2a70 0x7f51f40a4f30 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7f5200006fd0 tx=0x7f5200008040 comp rx=0 tx=0).stop 2024-03-24T00:13:55.212 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:55.210+0000 7f5216400700 1 -- 172.21.15.118:0/4089495462 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f5210104b90 msgr2=0x7f5210073dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:13:55.212 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:55.210+0000 7f5216400700 1 --2- 172.21.15.118:0/4089495462 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f5210104b90 0x7f5210073dc0 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f51f800e9a0 tx=0x7f51f800ed60 comp rx=0 tx=0).stop 2024-03-24T00:13:55.212 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:55.211+0000 7f520fe00700 1 -- 172.21.15.118:0/4089495462 reap_dead start 2024-03-24T00:13:55.212 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:55.211+0000 7f5216400700 1 -- 172.21.15.118:0/4089495462 shutdown_connections 2024-03-24T00:13:55.212 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:55.211+0000 7f5216400700 1 -- 172.21.15.118:0/4089495462 >> 172.21.15.118:0/4089495462 conn(0x7f5210077fc0 msgr2=0x7f5210103b70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:13:55.213 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:55.211+0000 7f5216400700 1 -- 172.21.15.118:0/4089495462 shutdown_connections 2024-03-24T00:13:55.213 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:55.211+0000 7f5216400700 1 -- 172.21.15.118:0/4089495462 wait complete. 2024-03-24T00:13:56.112 INFO:teuthology.orchestra.run.smithi118.stdout:313532612636 2024-03-24T00:13:56.113 DEBUG:teuthology.orchestra.run.smithi118:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 shell --fsid d79a5420-e971-11ee-95cd-87774f69a715 -- ceph osd last-stat-seq osd.10 2024-03-24T00:13:57.005 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:13:56 smithi189 ceph-mon[36187]: pgmap v341: 97 pgs: 97 active+clean; 644 KiB data, 344 MiB used, 1.0 TiB / 1.0 TiB avail; 5.5 KiB/s rd, 10 op/s 2024-03-24T00:13:57.024 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:57.023+0000 7fb18e200700 1 -- 172.21.15.118:0/4235854034 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb188102210 msgr2=0x7fb188102690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:13:57.024 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:57.023+0000 7fb18e200700 1 --2- 172.21.15.118:0/4235854034 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb188102210 0x7fb188102690 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7fb1740099c0 tx=0x7fb174009cd0 comp rx=0 tx=0).stop 2024-03-24T00:13:57.025 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:13:56 smithi118 ceph-mon[31535]: pgmap v341: 97 pgs: 97 active+clean; 644 KiB data, 344 MiB used, 1.0 TiB / 1.0 TiB avail; 5.5 KiB/s rd, 10 op/s 2024-03-24T00:13:57.026 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:57.023+0000 7fb18e200700 1 -- 172.21.15.118:0/4235854034 shutdown_connections 2024-03-24T00:13:57.026 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:57.023+0000 7fb18e200700 1 --2- 172.21.15.118:0/4235854034 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb188102bd0 0x7fb18810f040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:13:57.026 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:57.023+0000 7fb18e200700 1 --2- 172.21.15.118:0/4235854034 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fb188108220 0x7fb188108600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:13:57.026 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:57.023+0000 7fb18e200700 1 --2- 172.21.15.118:0/4235854034 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb188102210 0x7fb188102690 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:13:57.026 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:57.023+0000 7fb18e200700 1 -- 172.21.15.118:0/4235854034 >> 172.21.15.118:0/4235854034 conn(0x7fb1880792f0 msgr2=0x7fb188079700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:13:57.026 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:57.023+0000 7fb18e200700 1 -- 172.21.15.118:0/4235854034 shutdown_connections 2024-03-24T00:13:57.026 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:57.023+0000 7fb18e200700 1 -- 172.21.15.118:0/4235854034 wait complete. 2024-03-24T00:13:57.026 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:57.024+0000 7fb18e200700 1 Processor -- start 2024-03-24T00:13:57.026 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:57.024+0000 7fb18e200700 1 -- start start 2024-03-24T00:13:57.027 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:57.025+0000 7fb18e200700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb188102210 0x7fb18810b660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:13:57.027 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:57.025+0000 7fb18e200700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb188102bd0 0x7fb18810bba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:13:57.027 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:57.025+0000 7fb18e200700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fb188108220 0x7fb1881aacb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:13:57.027 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:57.025+0000 7fb18e200700 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb188075580 con 0x7fb188102bd0 2024-03-24T00:13:57.027 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:57.025+0000 7fb18e200700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb188075400 con 0x7fb188102210 2024-03-24T00:13:57.027 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:57.025+0000 7fb18e200700 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb188075700 con 0x7fb188108220 2024-03-24T00:13:57.027 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:57.025+0000 7fb187400700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb188102210 0x7fb18810b660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:13:57.027 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:57.025+0000 7fb187400700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb188102210 0x7fb18810b660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.118:58946/0 (socket says 172.21.15.118:58946) 2024-03-24T00:13:57.027 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:57.025+0000 7fb187400700 1 -- 172.21.15.118:0/186527546 learned_addr learned my addr 172.21.15.118:0/186527546 (peer_addr_for_me v2:172.21.15.118:0/0) 2024-03-24T00:13:57.027 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:57.025+0000 7fb187e00700 1 --2- 172.21.15.118:0/186527546 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fb188108220 0x7fb1881aacb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:13:57.027 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:57.025+0000 7fb186a00700 1 --2- 172.21.15.118:0/186527546 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb188102bd0 0x7fb18810bba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:13:57.028 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:57.025+0000 7fb187400700 1 -- 172.21.15.118:0/186527546 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fb188108220 msgr2=0x7fb1881aacb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:13:57.029 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:57.025+0000 7fb187400700 1 --2- 172.21.15.118:0/186527546 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fb188108220 0x7fb1881aacb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:13:57.029 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:57.025+0000 7fb187400700 1 -- 172.21.15.118:0/186527546 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb188102bd0 msgr2=0x7fb18810bba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:13:57.029 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:57.025+0000 7fb187400700 1 --2- 172.21.15.118:0/186527546 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb188102bd0 0x7fb18810bba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:13:57.029 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:57.026+0000 7fb187400700 1 -- 172.21.15.118:0/186527546 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb1881ab1f0 con 0x7fb188102210 2024-03-24T00:13:57.029 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:57.026+0000 7fb187e00700 1 --2- 172.21.15.118:0/186527546 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fb188108220 0x7fb1881aacb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T00:13:57.029 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:57.026+0000 7fb186a00700 1 --2- 172.21.15.118:0/186527546 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb188102bd0 0x7fb18810bba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-24T00:13:57.029 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:57.026+0000 7fb187400700 1 --2- 172.21.15.118:0/186527546 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb188102210 0x7fb18810b660 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7fb17800eb30 tx=0x7fb17800ee40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:13:57.029 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:57.026+0000 7fb17fe00700 1 -- 172.21.15.118:0/186527546 <== mon.2 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb17800cc30 con 0x7fb188102210 2024-03-24T00:13:57.029 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:57.026+0000 7fb17fe00700 1 -- 172.21.15.118:0/186527546 <== mon.2 v2:172.21.15.189:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fb178004500 con 0x7fb188102210 2024-03-24T00:13:57.029 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:57.026+0000 7fb17fe00700 1 -- 172.21.15.118:0/186527546 <== mon.2 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb178010430 con 0x7fb188102210 2024-03-24T00:13:57.029 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:57.026+0000 7fb18e200700 1 -- 172.21.15.118:0/186527546 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb1881ab480 con 0x7fb188102210 2024-03-24T00:13:57.029 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:57.026+0000 7fb18e200700 1 -- 172.21.15.118:0/186527546 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb1881ab990 con 0x7fb188102210 2024-03-24T00:13:57.029 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:57.027+0000 7fb18e200700 1 -- 172.21.15.118:0/186527546 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7fb188107130 con 0x7fb188102210 2024-03-24T00:13:57.030 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:57.029+0000 7fb17fe00700 1 -- 172.21.15.118:0/186527546 <== mon.2 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 17) v1 ==== 137342+0+0 (secure 0 0 0) 0x7fb178003f20 con 0x7fb188102210 2024-03-24T00:13:57.031 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:57.030+0000 7fb17fe00700 1 --2- 172.21.15.118:0/186527546 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7fb16c0a2920 0x7fb16c0a4de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:13:57.031 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:57.031+0000 7fb17fe00700 1 -- 172.21.15.118:0/186527546 <== mon.2 v2:172.21.15.189:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 7979+0+0 (secure 0 0 0) 0x7fb1780ced60 con 0x7fb188102210 2024-03-24T00:13:57.032 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:57.031+0000 7fb186a00700 1 --2- 172.21.15.118:0/186527546 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7fb16c0a2920 0x7fb16c0a4de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:13:57.032 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:57.031+0000 7fb17fe00700 1 --2- 172.21.15.118:0/186527546 >> [v2:172.21.15.189:6816/3667611284,v1:172.21.15.189:6817/3667611284] conn(0x7fb16c0ac100 0x7fb16c0ae520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:13:57.032 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:57.031+0000 7fb17fe00700 1 -- 172.21.15.118:0/186527546 --> [v2:172.21.15.189:6816/3667611284,v1:172.21.15.189:6817/3667611284] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fb178018e50 con 0x7fb16c0ac100 2024-03-24T00:13:57.032 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:57.031+0000 7fb17fe00700 1 -- 172.21.15.118:0/186527546 <== mon.2 v2:172.21.15.189:3300/0 6 ==== mon_get_version_reply(handle=1 version=86) v2 ==== 24+0+0 (secure 0 0 0) 0x7fb1780d14b0 con 0x7fb188102210 2024-03-24T00:13:57.032 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:57.031+0000 7fb187e00700 1 --2- 172.21.15.118:0/186527546 >> [v2:172.21.15.189:6816/3667611284,v1:172.21.15.189:6817/3667611284] conn(0x7fb16c0ac100 0x7fb16c0ae520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:13:57.032 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:57.032+0000 7fb186a00700 1 --2- 172.21.15.118:0/186527546 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7fb16c0a2920 0x7fb16c0a4de0 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7fb174006010 tx=0x7fb174000bc0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:13:57.033 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:57.032+0000 7fb187e00700 1 --2- 172.21.15.118:0/186527546 >> [v2:172.21.15.189:6816/3667611284,v1:172.21.15.189:6817/3667611284] conn(0x7fb16c0ac100 0x7fb16c0ae520 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:13:57.033 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:57.033+0000 7fb17fe00700 1 -- 172.21.15.118:0/186527546 <== osd.7 v2:172.21.15.189:6816/3667611284 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7fb178018e50 con 0x7fb16c0ac100 2024-03-24T00:13:57.051 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:57.050+0000 7fb18e200700 1 -- 172.21.15.118:0/186527546 --> [v2:172.21.15.189:6816/3667611284,v1:172.21.15.189:6817/3667611284] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7fb1881038d0 con 0x7fb16c0ac100 2024-03-24T00:13:57.051 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:57.051+0000 7fb17fe00700 1 -- 172.21.15.118:0/186527546 <== osd.7 v2:172.21.15.189:6816/3667611284 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+12 (crc 0 0 0) 0x7fb1881038d0 con 0x7fb16c0ac100 2024-03-24T00:13:57.052 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:57.051+0000 7fb18e200700 1 -- 172.21.15.118:0/186527546 >> [v2:172.21.15.189:6816/3667611284,v1:172.21.15.189:6817/3667611284] conn(0x7fb16c0ac100 msgr2=0x7fb16c0ae520 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:13:57.052 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:57.051+0000 7fb18e200700 1 --2- 172.21.15.118:0/186527546 >> [v2:172.21.15.189:6816/3667611284,v1:172.21.15.189:6817/3667611284] conn(0x7fb16c0ac100 0x7fb16c0ae520 crc :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:13:57.052 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:57.051+0000 7fb18e200700 1 -- 172.21.15.118:0/186527546 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7fb16c0a2920 msgr2=0x7fb16c0a4de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:13:57.052 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:57.051+0000 7fb18e200700 1 --2- 172.21.15.118:0/186527546 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7fb16c0a2920 0x7fb16c0a4de0 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7fb174006010 tx=0x7fb174000bc0 comp rx=0 tx=0).stop 2024-03-24T00:13:57.052 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:57.051+0000 7fb18e200700 1 -- 172.21.15.118:0/186527546 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb188102210 msgr2=0x7fb18810b660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:13:57.052 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:57.051+0000 7fb18e200700 1 --2- 172.21.15.118:0/186527546 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb188102210 0x7fb18810b660 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7fb17800eb30 tx=0x7fb17800ee40 comp rx=0 tx=0).stop 2024-03-24T00:13:57.052 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:57.051+0000 7fb187e00700 1 -- 172.21.15.118:0/186527546 reap_dead start 2024-03-24T00:13:57.053 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:57.051+0000 7fb18e200700 1 -- 172.21.15.118:0/186527546 shutdown_connections 2024-03-24T00:13:57.053 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:57.051+0000 7fb18e200700 1 -- 172.21.15.118:0/186527546 >> 172.21.15.118:0/186527546 conn(0x7fb1880792f0 msgr2=0x7fb188100bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:13:57.053 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:57.052+0000 7fb18e200700 1 -- 172.21.15.118:0/186527546 shutdown_connections 2024-03-24T00:13:57.053 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:13:57.052+0000 7fb18e200700 1 -- 172.21.15.118:0/186527546 wait complete. 2024-03-24T00:13:57.154 INFO:teuthology.orchestra.run.smithi118.stdout:77309411404 2024-03-24T00:13:57.154 DEBUG:teuthology.orchestra.run.smithi118:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 shell --fsid d79a5420-e971-11ee-95cd-87774f69a715 -- ceph osd last-stat-seq osd.2 2024-03-24T00:13:57.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:13:56 smithi193 ceph-mon[36878]: pgmap v341: 97 pgs: 97 active+clean; 644 KiB data, 344 MiB used, 1.0 TiB / 1.0 TiB avail; 5.5 KiB/s rd, 10 op/s 2024-03-24T00:13:57.927 INFO:teuthology.orchestra.run.smithi118.stderr:Inferring config /var/lib/ceph/d79a5420-e971-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T00:13:57.931 INFO:teuthology.orchestra.run.smithi118.stderr:Inferring config /var/lib/ceph/d79a5420-e971-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T00:13:57.933 INFO:teuthology.orchestra.run.smithi118.stderr:Inferring config /var/lib/ceph/d79a5420-e971-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T00:13:57.990 INFO:teuthology.orchestra.run.smithi118.stdout:292057776162 2024-03-24T00:13:57.992 DEBUG:teuthology.orchestra.run.smithi118:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 shell --fsid d79a5420-e971-11ee-95cd-87774f69a715 -- ceph osd last-stat-seq osd.9 2024-03-24T00:13:58.420 INFO:teuthology.orchestra.run.smithi118.stdout:223338299439 2024-03-24T00:13:58.420 DEBUG:teuthology.orchestra.run.smithi118:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 shell --fsid d79a5420-e971-11ee-95cd-87774f69a715 -- ceph osd last-stat-seq osd.7 2024-03-24T00:13:58.459 INFO:teuthology.orchestra.run.smithi118.stderr:Inferring config /var/lib/ceph/d79a5420-e971-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T00:13:59.013 INFO:teuthology.orchestra.run.smithi118.stderr:Inferring config /var/lib/ceph/d79a5420-e971-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T00:13:59.024 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:13:58 smithi118 ceph-mon[31535]: pgmap v342: 97 pgs: 97 active+clean; 644 KiB data, 344 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 9 op/s 2024-03-24T00:13:59.203 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:13:58 smithi189 ceph-mon[36187]: pgmap v342: 97 pgs: 97 active+clean; 644 KiB data, 344 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 9 op/s 2024-03-24T00:13:59.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:13:58 smithi193 ceph-mon[36878]: pgmap v342: 97 pgs: 97 active+clean; 644 KiB data, 344 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 9 op/s 2024-03-24T00:14:01.014 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:14:00 smithi118 ceph-mon[31535]: pgmap v343: 97 pgs: 97 active+clean; 644 KiB data, 344 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 11 op/s 2024-03-24T00:14:01.203 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:14:00 smithi189 ceph-mon[36187]: pgmap v343: 97 pgs: 97 active+clean; 644 KiB data, 344 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 11 op/s 2024-03-24T00:14:01.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:14:00 smithi193 ceph-mon[36878]: pgmap v343: 97 pgs: 97 active+clean; 644 KiB data, 344 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 11 op/s 2024-03-24T00:14:01.337 INFO:teuthology.orchestra.run.smithi118.stderr:Inferring config /var/lib/ceph/d79a5420-e971-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T00:14:02.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:14:01 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T00:14:02.203 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:14:01 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T00:14:02.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:14:01 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T00:14:02.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:14:02 smithi118 ceph-mon[31535]: pgmap v344: 97 pgs: 97 active+clean; 644 KiB data, 344 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-24T00:14:02.953 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:14:02 smithi189 ceph-mon[36187]: pgmap v344: 97 pgs: 97 active+clean; 644 KiB data, 344 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-24T00:14:03.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:14:02 smithi193 ceph-mon[36878]: pgmap v344: 97 pgs: 97 active+clean; 644 KiB data, 344 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-24T00:14:03.375 INFO:teuthology.orchestra.run.smithi118.stderr:Inferring config /var/lib/ceph/d79a5420-e971-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T00:14:03.443 INFO:teuthology.orchestra.run.smithi118.stderr:Inferring config /var/lib/ceph/d79a5420-e971-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T00:14:03.444 INFO:teuthology.orchestra.run.smithi118.stderr:Inferring config /var/lib/ceph/d79a5420-e971-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T00:14:04.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:14:04 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T00:14:04.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:14:04 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T00:14:04.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:14:04 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T00:14:04.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:14:04 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T00:14:04.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:14:04 smithi118 ceph-mon[31535]: pgmap v345: 97 pgs: 97 active+clean; 644 KiB data, 344 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-03-24T00:14:04.706 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:14:04 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T00:14:04.706 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:14:04 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T00:14:04.706 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:14:04 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T00:14:04.706 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:14:04 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T00:14:04.706 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:14:04 smithi189 ceph-mon[36187]: pgmap v345: 97 pgs: 97 active+clean; 644 KiB data, 344 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-03-24T00:14:04.710 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:14:04 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T00:14:04.711 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:14:04 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T00:14:04.711 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:14:04 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T00:14:04.711 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:14:04 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T00:14:04.711 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:14:04 smithi193 ceph-mon[36878]: pgmap v345: 97 pgs: 97 active+clean; 644 KiB data, 344 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-03-24T00:14:04.988 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:04.987+0000 7fccacc00700 1 -- 172.21.15.118:0/2029321329 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fcca8107510 msgr2=0x7fcca810f0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:14:04.988 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:04.987+0000 7fccacc00700 1 --2- 172.21.15.118:0/2029321329 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fcca8107510 0x7fcca810f0e0 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7fcca000b0a0 tx=0x7fcca000b3b0 comp rx=0 tx=0).stop 2024-03-24T00:14:04.988 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:04.987+0000 7fccacc00700 1 -- 172.21.15.118:0/2029321329 shutdown_connections 2024-03-24T00:14:04.989 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:04.987+0000 7fccacc00700 1 --2- 172.21.15.118:0/2029321329 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fcca8107510 0x7fcca810f0e0 unknown :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:14:04.989 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:04.987+0000 7fccacc00700 1 --2- 172.21.15.118:0/2029321329 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fcca8102440 0x7fcca8102820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:14:04.989 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:04.987+0000 7fccacc00700 1 --2- 172.21.15.118:0/2029321329 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fcca8102df0 0x7fcca8106e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:14:04.989 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:04.987+0000 7fccacc00700 1 -- 172.21.15.118:0/2029321329 >> 172.21.15.118:0/2029321329 conn(0x7fcca80792f0 msgr2=0x7fcca8079700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:14:04.989 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:04.987+0000 7fccacc00700 1 -- 172.21.15.118:0/2029321329 shutdown_connections 2024-03-24T00:14:04.989 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:04.987+0000 7fccacc00700 1 -- 172.21.15.118:0/2029321329 wait complete. 2024-03-24T00:14:04.989 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:04.989+0000 7fccacc00700 1 Processor -- start 2024-03-24T00:14:04.990 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:04.989+0000 7fccacc00700 1 -- start start 2024-03-24T00:14:04.990 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:04.989+0000 7fccacc00700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fcca8102440 0x7fcca81a0800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:14:04.990 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:04.989+0000 7fccacc00700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fcca8102df0 0x7fcca81a0d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:14:04.991 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:04.989+0000 7fccacc00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fcca8107510 0x7fcca819a9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:14:04.991 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:04.989+0000 7fccacc00700 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcca8112560 con 0x7fcca8102440 2024-03-24T00:14:04.991 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:04.989+0000 7fccacc00700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcca81123e0 con 0x7fcca8107510 2024-03-24T00:14:04.991 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:04.989+0000 7fccacc00700 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcca81126e0 con 0x7fcca8102df0 2024-03-24T00:14:04.991 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:04.990+0000 7fcc9fe00700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fcca8102df0 0x7fcca81a0d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:14:04.991 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:04.990+0000 7fcc9fe00700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fcca8102df0 0x7fcca81a0d40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.118:47618/0 (socket says 172.21.15.118:47618) 2024-03-24T00:14:04.991 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:04.990+0000 7fcc9fe00700 1 -- 172.21.15.118:0/2940940341 learned_addr learned my addr 172.21.15.118:0/2940940341 (peer_addr_for_me v2:172.21.15.118:0/0) 2024-03-24T00:14:04.991 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:04.990+0000 7fcca6800700 1 --2- 172.21.15.118:0/2940940341 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fcca8107510 0x7fcca819a9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:14:04.991 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:04.990+0000 7fcca5e00700 1 --2- 172.21.15.118:0/2940940341 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fcca8102440 0x7fcca81a0800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:14:04.991 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:04.990+0000 7fcca6800700 1 -- 172.21.15.118:0/2940940341 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fcca8102df0 msgr2=0x7fcca81a0d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:14:04.991 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:04.990+0000 7fcca6800700 1 --2- 172.21.15.118:0/2940940341 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fcca8102df0 0x7fcca81a0d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:14:04.991 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:04.990+0000 7fcca6800700 1 -- 172.21.15.118:0/2940940341 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fcca8102440 msgr2=0x7fcca81a0800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:14:04.991 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:04.990+0000 7fcca6800700 1 --2- 172.21.15.118:0/2940940341 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fcca8102440 0x7fcca81a0800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:14:04.991 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:04.990+0000 7fcca6800700 1 -- 172.21.15.118:0/2940940341 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcca819b120 con 0x7fcca8107510 2024-03-24T00:14:04.992 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:04.990+0000 7fcc9fe00700 1 --2- 172.21.15.118:0/2940940341 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fcca8102df0 0x7fcca81a0d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T00:14:04.992 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:04.990+0000 7fcca6800700 1 --2- 172.21.15.118:0/2940940341 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fcca8107510 0x7fcca819a9c0 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7fcca00062a0 tx=0x7fcca0004c60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:14:04.992 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:04.991+0000 7fcc9ea00700 1 -- 172.21.15.118:0/2940940341 <== mon.2 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcca0021070 con 0x7fcca8107510 2024-03-24T00:14:04.992 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:04.991+0000 7fcc9ea00700 1 -- 172.21.15.118:0/2940940341 <== mon.2 v2:172.21.15.189:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fcca0003c00 con 0x7fcca8107510 2024-03-24T00:14:04.993 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:04.991+0000 7fcc9ea00700 1 -- 172.21.15.118:0/2940940341 <== mon.2 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcca00045a0 con 0x7fcca8107510 2024-03-24T00:14:04.993 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:04.991+0000 7fccacc00700 1 -- 172.21.15.118:0/2940940341 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcca819b3b0 con 0x7fcca8107510 2024-03-24T00:14:04.993 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:04.991+0000 7fccacc00700 1 -- 172.21.15.118:0/2940940341 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcca81a76d0 con 0x7fcca8107510 2024-03-24T00:14:04.993 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:04.992+0000 7fccacc00700 1 -- 172.21.15.118:0/2940940341 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcca8103910 con 0x7fcca8107510 2024-03-24T00:14:04.994 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:04.994+0000 7fcc9ea00700 1 -- 172.21.15.118:0/2940940341 <== mon.2 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 17) v1 ==== 137342+0+0 (secure 0 0 0) 0x7fcca0004740 con 0x7fcca8107510 2024-03-24T00:14:04.996 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:04.995+0000 7fcc9ea00700 1 --2- 172.21.15.118:0/2940940341 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7fcc8c0a29a0 0x7fcc8c0a4e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:14:04.996 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:04.995+0000 7fcc9ea00700 1 -- 172.21.15.118:0/2940940341 <== mon.2 v2:172.21.15.189:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 7979+0+0 (secure 0 0 0) 0x7fcca000f020 con 0x7fcca8107510 2024-03-24T00:14:04.996 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:04.995+0000 7fcca5e00700 1 --2- 172.21.15.118:0/2940940341 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7fcc8c0a29a0 0x7fcc8c0a4e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:14:04.997 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:04.996+0000 7fcca5e00700 1 --2- 172.21.15.118:0/2940940341 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7fcc8c0a29a0 0x7fcc8c0a4e60 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7fcca819bb00 tx=0x7fcc98009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:14:04.999 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:04.999+0000 7fcc9ea00700 1 -- 172.21.15.118:0/2940940341 <== mon.2 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7fcca002a020 con 0x7fcca8107510 2024-03-24T00:14:05.250 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:05.249+0000 7fccacc00700 1 -- 172.21.15.118:0/2940940341 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 8} v 0) v1 -- 0x7fcca819bef0 con 0x7fcca8107510 2024-03-24T00:14:05.251 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:05.250+0000 7fcc9ea00700 1 -- 172.21.15.118:0/2940940341 <== mon.2 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 8}]=0 v0) v1 ==== 74+0+13 (secure 0 0 0) 0x7fcca009cd60 con 0x7fcca8107510 2024-03-24T00:14:05.251 INFO:teuthology.orchestra.run.smithi118.stdout:257698037803 2024-03-24T00:14:05.253 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:05.253+0000 7fccacc00700 1 -- 172.21.15.118:0/2940940341 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7fcc8c0a29a0 msgr2=0x7fcc8c0a4e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:14:05.254 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:05.253+0000 7fccacc00700 1 --2- 172.21.15.118:0/2940940341 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7fcc8c0a29a0 0x7fcc8c0a4e60 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7fcca819bb00 tx=0x7fcc98009260 comp rx=0 tx=0).stop 2024-03-24T00:14:05.254 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:05.253+0000 7fccacc00700 1 -- 172.21.15.118:0/2940940341 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fcca8107510 msgr2=0x7fcca819a9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:14:05.254 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:05.253+0000 7fccacc00700 1 --2- 172.21.15.118:0/2940940341 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fcca8107510 0x7fcca819a9c0 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7fcca00062a0 tx=0x7fcca0004c60 comp rx=0 tx=0).stop 2024-03-24T00:14:05.254 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:05.253+0000 7fccacc00700 1 -- 172.21.15.118:0/2940940341 shutdown_connections 2024-03-24T00:14:05.254 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:05.253+0000 7fccacc00700 1 --2- 172.21.15.118:0/2940940341 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fcca8107510 0x7fcca819a9c0 unknown :-1 s=CLOSED pgs=110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:14:05.254 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:05.253+0000 7fccacc00700 1 --2- 172.21.15.118:0/2940940341 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7fcc8c0a29a0 0x7fcc8c0a4e60 unknown :-1 s=CLOSED pgs=192 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:14:05.254 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:05.253+0000 7fccacc00700 1 --2- 172.21.15.118:0/2940940341 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fcca8102440 0x7fcca81a0800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:14:05.254 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:05.253+0000 7fccacc00700 1 --2- 172.21.15.118:0/2940940341 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fcca8102df0 0x7fcca81a0d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:14:05.254 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:05.253+0000 7fccacc00700 1 -- 172.21.15.118:0/2940940341 >> 172.21.15.118:0/2940940341 conn(0x7fcca80792f0 msgr2=0x7fcca8100390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:14:05.254 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:05.253+0000 7fccacc00700 1 -- 172.21.15.118:0/2940940341 shutdown_connections 2024-03-24T00:14:05.254 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:05.253+0000 7fccacc00700 1 -- 172.21.15.118:0/2940940341 wait complete. 2024-03-24T00:14:05.695 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:14:05 smithi118 ceph-mon[31535]: from='client.? 172.21.15.118:0/2940940341' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 8}]: dispatch 2024-03-24T00:14:05.953 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:14:05 smithi189 ceph-mon[36187]: from='client.? 172.21.15.118:0/2940940341' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 8}]: dispatch 2024-03-24T00:14:05.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:14:05 smithi193 ceph-mon[36878]: from='client.? 172.21.15.118:0/2940940341' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 8}]: dispatch 2024-03-24T00:14:06.441 INFO:teuthology.orchestra.run.smithi118.stderr:Inferring config /var/lib/ceph/d79a5420-e971-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T00:14:06.782 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:14:06 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.a", "id": [2, 11]}]: dispatch 2024-03-24T00:14:06.783 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:14:06 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.14", "id": [2, 4]}]: dispatch 2024-03-24T00:14:06.783 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:14:06 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1f", "id": [8, 1]}]: dispatch 2024-03-24T00:14:06.783 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:14:06 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.11", "id": [4, 8, 5, 9]}]: dispatch 2024-03-24T00:14:06.783 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:14:06 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.13", "id": [4, 9]}]: dispatch 2024-03-24T00:14:06.783 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:14:06 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.14", "id": [4, 6]}]: dispatch 2024-03-24T00:14:06.783 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:14:06 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.31", "id": [4, 7]}]: dispatch 2024-03-24T00:14:06.783 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:14:06 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.38", "id": [3, 9]}]: dispatch 2024-03-24T00:14:06.783 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:14:06 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.39", "id": [5, 8]}]: dispatch 2024-03-24T00:14:06.783 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:14:06 smithi118 ceph-mon[31535]: pgmap v346: 97 pgs: 97 active+clean; 644 KiB data, 344 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 9 op/s 2024-03-24T00:14:06.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:14:06 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.a", "id": [2, 11]}]: dispatch 2024-03-24T00:14:06.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:14:06 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.14", "id": [2, 4]}]: dispatch 2024-03-24T00:14:06.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:14:06 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1f", "id": [8, 1]}]: dispatch 2024-03-24T00:14:06.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:14:06 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.11", "id": [4, 8, 5, 9]}]: dispatch 2024-03-24T00:14:06.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:14:06 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.13", "id": [4, 9]}]: dispatch 2024-03-24T00:14:06.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:14:06 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.14", "id": [4, 6]}]: dispatch 2024-03-24T00:14:06.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:14:06 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.31", "id": [4, 7]}]: dispatch 2024-03-24T00:14:06.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:14:06 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.38", "id": [3, 9]}]: dispatch 2024-03-24T00:14:06.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:14:06 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.39", "id": [5, 8]}]: dispatch 2024-03-24T00:14:06.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:14:06 smithi189 ceph-mon[36187]: pgmap v346: 97 pgs: 97 active+clean; 644 KiB data, 344 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 9 op/s 2024-03-24T00:14:06.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:14:06 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.a", "id": [2, 11]}]: dispatch 2024-03-24T00:14:06.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:14:06 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.14", "id": [2, 4]}]: dispatch 2024-03-24T00:14:06.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:14:06 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1f", "id": [8, 1]}]: dispatch 2024-03-24T00:14:06.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:14:06 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.11", "id": [4, 8, 5, 9]}]: dispatch 2024-03-24T00:14:06.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:14:06 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.13", "id": [4, 9]}]: dispatch 2024-03-24T00:14:06.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:14:06 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.14", "id": [4, 6]}]: dispatch 2024-03-24T00:14:06.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:14:06 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.31", "id": [4, 7]}]: dispatch 2024-03-24T00:14:06.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:14:06 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.38", "id": [3, 9]}]: dispatch 2024-03-24T00:14:06.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:14:06 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.39", "id": [5, 8]}]: dispatch 2024-03-24T00:14:06.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:14:06 smithi193 ceph-mon[36878]: pgmap v346: 97 pgs: 97 active+clean; 644 KiB data, 344 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 9 op/s 2024-03-24T00:14:07.780 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:14:07 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.a", "id": [2, 11]}]': finished 2024-03-24T00:14:07.780 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:14:07 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.14", "id": [2, 4]}]': finished 2024-03-24T00:14:07.781 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:14:07 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1f", "id": [8, 1]}]': finished 2024-03-24T00:14:07.781 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:14:07 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.11", "id": [4, 8, 5, 9]}]': finished 2024-03-24T00:14:07.781 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:14:07 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.13", "id": [4, 9]}]': finished 2024-03-24T00:14:07.781 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:14:07 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.14", "id": [4, 6]}]': finished 2024-03-24T00:14:07.781 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:14:07 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.31", "id": [4, 7]}]': finished 2024-03-24T00:14:07.781 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:14:07 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.38", "id": [3, 9]}]': finished 2024-03-24T00:14:07.781 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:14:07 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.39", "id": [5, 8]}]': finished 2024-03-24T00:14:07.781 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:14:07 smithi118 ceph-mon[31535]: osdmap e87: 12 total, 12 up, 12 in 2024-03-24T00:14:07.953 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:14:07 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.a", "id": [2, 11]}]': finished 2024-03-24T00:14:07.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:14:07 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.14", "id": [2, 4]}]': finished 2024-03-24T00:14:07.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:14:07 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1f", "id": [8, 1]}]': finished 2024-03-24T00:14:07.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:14:07 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.11", "id": [4, 8, 5, 9]}]': finished 2024-03-24T00:14:07.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:14:07 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.13", "id": [4, 9]}]': finished 2024-03-24T00:14:07.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:14:07 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.14", "id": [4, 6]}]': finished 2024-03-24T00:14:07.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:14:07 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.31", "id": [4, 7]}]': finished 2024-03-24T00:14:07.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:14:07 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.38", "id": [3, 9]}]': finished 2024-03-24T00:14:07.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:14:07 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.39", "id": [5, 8]}]': finished 2024-03-24T00:14:07.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:14:07 smithi189 ceph-mon[36187]: osdmap e87: 12 total, 12 up, 12 in 2024-03-24T00:14:07.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:14:07 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.a", "id": [2, 11]}]': finished 2024-03-24T00:14:07.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:14:07 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.14", "id": [2, 4]}]': finished 2024-03-24T00:14:07.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:14:07 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1f", "id": [8, 1]}]': finished 2024-03-24T00:14:07.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:14:07 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.11", "id": [4, 8, 5, 9]}]': finished 2024-03-24T00:14:07.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:14:07 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.13", "id": [4, 9]}]': finished 2024-03-24T00:14:07.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:14:07 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.14", "id": [4, 6]}]': finished 2024-03-24T00:14:07.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:14:07 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.31", "id": [4, 7]}]': finished 2024-03-24T00:14:07.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:14:07 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.38", "id": [3, 9]}]': finished 2024-03-24T00:14:07.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:14:07 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "3.39", "id": [5, 8]}]': finished 2024-03-24T00:14:07.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:14:07 smithi193 ceph-mon[36878]: osdmap e87: 12 total, 12 up, 12 in 2024-03-24T00:14:08.527 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:14:08 smithi118 ceph-mon[31535]: osdmap e88: 12 total, 12 up, 12 in 2024-03-24T00:14:08.527 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:14:08 smithi118 ceph-mon[31535]: pgmap v349: 97 pgs: 1 peering, 96 active+clean; 644 KiB data, 344 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-03-24T00:14:08.953 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:14:08 smithi189 ceph-mon[36187]: osdmap e88: 12 total, 12 up, 12 in 2024-03-24T00:14:08.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:14:08 smithi189 ceph-mon[36187]: pgmap v349: 97 pgs: 1 peering, 96 active+clean; 644 KiB data, 344 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-03-24T00:14:08.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:14:08 smithi193 ceph-mon[36878]: osdmap e88: 12 total, 12 up, 12 in 2024-03-24T00:14:08.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:14:08 smithi193 ceph-mon[36878]: pgmap v349: 97 pgs: 1 peering, 96 active+clean; 644 KiB data, 344 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 8 op/s 2024-03-24T00:14:09.523 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:09.523+0000 7f7c26800700 1 -- 172.21.15.118:0/2533655594 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7c20104e70 msgr2=0x7f7c20108ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:14:09.523 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:09.523+0000 7f7c26800700 1 --2- 172.21.15.118:0/2533655594 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7c20104e70 0x7f7c20108ec0 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7f7c0c009990 tx=0x7f7c0c009ca0 comp rx=0 tx=0).stop 2024-03-24T00:14:09.523 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:09.523+0000 7f7c26800700 1 -- 172.21.15.118:0/2533655594 shutdown_connections 2024-03-24T00:14:09.523 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:09.523+0000 7f7c26800700 1 --2- 172.21.15.118:0/2533655594 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f7c201095a0 0x7f7c20111170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:14:09.523 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:09.523+0000 7f7c26800700 1 --2- 172.21.15.118:0/2533655594 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f7c201044c0 0x7f7c201048a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:14:09.523 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:09.523+0000 7f7c26800700 1 --2- 172.21.15.118:0/2533655594 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7c20104e70 0x7f7c20108ec0 unknown :-1 s=CLOSED pgs=112 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:14:09.523 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:09.523+0000 7f7c26800700 1 -- 172.21.15.118:0/2533655594 >> 172.21.15.118:0/2533655594 conn(0x7f7c200ffd70 msgr2=0x7f7c20102190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:14:09.524 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:09.523+0000 7f7c26800700 1 -- 172.21.15.118:0/2533655594 shutdown_connections 2024-03-24T00:14:09.524 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:09.523+0000 7f7c26800700 1 -- 172.21.15.118:0/2533655594 wait complete. 2024-03-24T00:14:09.524 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:09.525+0000 7f7c26800700 1 Processor -- start 2024-03-24T00:14:09.524 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:09.525+0000 7f7c26800700 1 -- start start 2024-03-24T00:14:09.525 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:09.526+0000 7f7c26800700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f7c201044c0 0x7f7c201a06e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:14:09.525 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:09.526+0000 7f7c26800700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f7c20104e70 0x7f7c201a0c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:14:09.526 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:09.526+0000 7f7c26800700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7c201095a0 0x7f7c2019a8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:14:09.526 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:09.526+0000 7f7c26800700 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7c201145d0 con 0x7f7c201044c0 2024-03-24T00:14:09.526 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:09.526+0000 7f7c26800700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7c20114450 con 0x7f7c201095a0 2024-03-24T00:14:09.526 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:09.526+0000 7f7c26800700 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7c20114750 con 0x7f7c20104e70 2024-03-24T00:14:09.526 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:09.526+0000 7f7c25400700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f7c201044c0 0x7f7c201a06e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:14:09.526 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:09.526+0000 7f7c24a00700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f7c20104e70 0x7f7c201a0c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:14:09.526 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:09.526+0000 7f7c24a00700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f7c20104e70 0x7f7c201a0c20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.118:47666/0 (socket says 172.21.15.118:47666) 2024-03-24T00:14:09.526 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:09.526+0000 7f7c24a00700 1 -- 172.21.15.118:0/266201547 learned_addr learned my addr 172.21.15.118:0/266201547 (peer_addr_for_me v2:172.21.15.118:0/0) 2024-03-24T00:14:09.527 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:09.526+0000 7f7c25e00700 1 --2- 172.21.15.118:0/266201547 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7c201095a0 0x7f7c2019a8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:14:09.527 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:09.526+0000 7f7c24a00700 1 -- 172.21.15.118:0/266201547 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7c201095a0 msgr2=0x7f7c2019a8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:14:09.527 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:09.526+0000 7f7c24a00700 1 --2- 172.21.15.118:0/266201547 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7c201095a0 0x7f7c2019a8f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:14:09.527 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:09.526+0000 7f7c24a00700 1 -- 172.21.15.118:0/266201547 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f7c201044c0 msgr2=0x7f7c201a06e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:14:09.527 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:09.526+0000 7f7c24a00700 1 --2- 172.21.15.118:0/266201547 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f7c201044c0 0x7f7c201a06e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:14:09.528 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:09.526+0000 7f7c24a00700 1 -- 172.21.15.118:0/266201547 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7c2019b050 con 0x7f7c20104e70 2024-03-24T00:14:09.528 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:09.527+0000 7f7c25400700 1 --2- 172.21.15.118:0/266201547 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f7c201044c0 0x7f7c201a06e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T00:14:09.528 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:09.527+0000 7f7c24a00700 1 --2- 172.21.15.118:0/266201547 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f7c20104e70 0x7f7c201a0c20 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7f7c0c000c00 tx=0x7f7c0c005590 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:14:09.528 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:09.527+0000 7f7c1e000700 1 -- 172.21.15.118:0/266201547 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7c0c027070 con 0x7f7c20104e70 2024-03-24T00:14:09.528 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:09.527+0000 7f7c1e000700 1 -- 172.21.15.118:0/266201547 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f7c0c0056b0 con 0x7f7c20104e70 2024-03-24T00:14:09.528 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:09.527+0000 7f7c1e000700 1 -- 172.21.15.118:0/266201547 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7c0c0173f0 con 0x7f7c20104e70 2024-03-24T00:14:09.528 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:09.527+0000 7f7c26800700 1 -- 172.21.15.118:0/266201547 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7c2019b2e0 con 0x7f7c20104e70 2024-03-24T00:14:09.528 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:09.527+0000 7f7c26800700 1 -- 172.21.15.118:0/266201547 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7c201a75b0 con 0x7f7c20104e70 2024-03-24T00:14:09.528 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:09.529+0000 7f7c26800700 1 -- 172.21.15.118:0/266201547 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7c20068e50 con 0x7f7c20104e70 2024-03-24T00:14:09.530 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:09.530+0000 7f7c1e000700 1 -- 172.21.15.118:0/266201547 <== mon.1 v2:172.21.15.193:3300/0 4 ==== mgrmap(e 17) v1 ==== 137342+0+0 (secure 0 0 0) 0x7f7c0c02a070 con 0x7f7c20104e70 2024-03-24T00:14:09.531 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:09.532+0000 7f7c1e000700 1 --2- 172.21.15.118:0/266201547 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7f7c080a29a0 0x7f7c080a4e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:14:09.531 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:09.532+0000 7f7c25400700 1 --2- 172.21.15.118:0/266201547 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7f7c080a29a0 0x7f7c080a4e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:14:09.532 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:09.532+0000 7f7c1e000700 1 -- 172.21.15.118:0/266201547 <== mon.1 v2:172.21.15.193:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 8248+0+0 (secure 0 0 0) 0x7f7c0c0cf8f0 con 0x7f7c20104e70 2024-03-24T00:14:09.532 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:09.533+0000 7f7c25400700 1 --2- 172.21.15.118:0/266201547 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7f7c080a29a0 0x7f7c080a4e60 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7f7c201001c0 tx=0x7f7c14009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:14:09.535 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:09.536+0000 7f7c1e000700 1 -- 172.21.15.118:0/266201547 <== mon.1 v2:172.21.15.193:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f7c0c013070 con 0x7f7c20104e70 2024-03-24T00:14:09.620 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:14:09 smithi118 ceph-mon[31535]: Health check failed: Reduced data availability: 1 pg peering (PG_AVAILABILITY) 2024-03-24T00:14:09.763 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:09.763+0000 7f7c26800700 1 -- 172.21.15.118:0/266201547 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 11} v 0) v1 -- 0x7f7c2002da30 con 0x7f7c20104e70 2024-03-24T00:14:09.763 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:09.764+0000 7f7c1e000700 1 -- 172.21.15.118:0/266201547 <== mon.1 v2:172.21.15.193:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 11}]=0 v0) v1 ==== 75+0+13 (secure 0 0 0) 0x7f7c0c09c850 con 0x7f7c20104e70 2024-03-24T00:14:09.764 INFO:teuthology.orchestra.run.smithi118.stdout:347892351002 2024-03-24T00:14:09.766 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:09.766+0000 7f7c26800700 1 -- 172.21.15.118:0/266201547 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7f7c080a29a0 msgr2=0x7f7c080a4e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:14:09.766 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:09.766+0000 7f7c26800700 1 --2- 172.21.15.118:0/266201547 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7f7c080a29a0 0x7f7c080a4e60 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7f7c201001c0 tx=0x7f7c14009260 comp rx=0 tx=0).stop 2024-03-24T00:14:09.767 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:09.766+0000 7f7c26800700 1 -- 172.21.15.118:0/266201547 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f7c20104e70 msgr2=0x7f7c201a0c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:14:09.767 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:09.766+0000 7f7c26800700 1 --2- 172.21.15.118:0/266201547 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f7c20104e70 0x7f7c201a0c20 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7f7c0c000c00 tx=0x7f7c0c005590 comp rx=0 tx=0).stop 2024-03-24T00:14:09.767 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:09.767+0000 7f7c26800700 1 -- 172.21.15.118:0/266201547 shutdown_connections 2024-03-24T00:14:09.767 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:09.767+0000 7f7c26800700 1 --2- 172.21.15.118:0/266201547 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7c201095a0 0x7f7c2019a8f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:14:09.767 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:09.767+0000 7f7c26800700 1 --2- 172.21.15.118:0/266201547 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7f7c080a29a0 0x7f7c080a4e60 unknown :-1 s=CLOSED pgs=194 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:14:09.767 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:09.767+0000 7f7c26800700 1 --2- 172.21.15.118:0/266201547 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f7c201044c0 0x7f7c201a06e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:14:09.767 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:09.767+0000 7f7c26800700 1 --2- 172.21.15.118:0/266201547 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f7c20104e70 0x7f7c201a0c20 unknown :-1 s=CLOSED pgs=111 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:14:09.767 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:09.767+0000 7f7c26800700 1 -- 172.21.15.118:0/266201547 >> 172.21.15.118:0/266201547 conn(0x7f7c200ffd70 msgr2=0x7f7c201020a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:14:09.767 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:09.767+0000 7f7c26800700 1 -- 172.21.15.118:0/266201547 shutdown_connections 2024-03-24T00:14:09.767 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:09.767+0000 7f7c26800700 1 -- 172.21.15.118:0/266201547 wait complete. 2024-03-24T00:14:09.953 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:14:09 smithi189 ceph-mon[36187]: Health check failed: Reduced data availability: 1 pg peering (PG_AVAILABILITY) 2024-03-24T00:14:09.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:14:09 smithi193 ceph-mon[36878]: Health check failed: Reduced data availability: 1 pg peering (PG_AVAILABILITY) 2024-03-24T00:14:10.767 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:14:10 smithi118 ceph-mon[31535]: pgmap v350: 97 pgs: 1 activating, 1 peering, 95 active+clean; 644 KiB data, 344 MiB used, 1.0 TiB / 1.0 TiB avail; 5.7 KiB/s rd, 11 op/s; 7.6 KiB/s, 1 objects/s recovering 2024-03-24T00:14:10.767 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:14:10 smithi118 ceph-mon[31535]: from='client.? 172.21.15.118:0/266201547' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 11}]: dispatch 2024-03-24T00:14:10.953 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:14:10 smithi189 ceph-mon[36187]: pgmap v350: 97 pgs: 1 activating, 1 peering, 95 active+clean; 644 KiB data, 344 MiB used, 1.0 TiB / 1.0 TiB avail; 5.7 KiB/s rd, 11 op/s; 7.6 KiB/s, 1 objects/s recovering 2024-03-24T00:14:10.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:14:10 smithi189 ceph-mon[36187]: from='client.? 172.21.15.118:0/266201547' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 11}]: dispatch 2024-03-24T00:14:10.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:14:10 smithi193 ceph-mon[36878]: pgmap v350: 97 pgs: 1 activating, 1 peering, 95 active+clean; 644 KiB data, 344 MiB used, 1.0 TiB / 1.0 TiB avail; 5.7 KiB/s rd, 11 op/s; 7.6 KiB/s, 1 objects/s recovering 2024-03-24T00:14:10.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:14:10 smithi193 ceph-mon[36878]: from='client.? 172.21.15.118:0/266201547' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 11}]: dispatch 2024-03-24T00:14:11.283 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:09.029+0000 7fc9bc000700 1 -- 172.21.15.118:0/3507311584 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fc9b4069e40 msgr2=0x7fc9b406a2a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:14:11.284 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:09.029+0000 7fc9bc000700 1 --2- 172.21.15.118:0/3507311584 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fc9b4069e40 0x7fc9b406a2a0 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7fc9a00099c0 tx=0x7fc9a0009cd0 comp rx=0 tx=0).stop 2024-03-24T00:14:11.284 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:09.030+0000 7fc9bc000700 1 -- 172.21.15.118:0/3507311584 shutdown_connections 2024-03-24T00:14:11.284 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:09.030+0000 7fc9bc000700 1 --2- 172.21.15.118:0/3507311584 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc9b41081e0 0x7fc9b41085c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:14:11.284 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:09.030+0000 7fc9bc000700 1 --2- 172.21.15.118:0/3507311584 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fc9b406a7e0 0x7fc9b410f020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:14:11.284 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:09.030+0000 7fc9bc000700 1 --2- 172.21.15.118:0/3507311584 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fc9b4069e40 0x7fc9b406a2a0 unknown :-1 s=CLOSED pgs=110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:14:11.284 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:09.030+0000 7fc9bc000700 1 -- 172.21.15.118:0/3507311584 >> 172.21.15.118:0/3507311584 conn(0x7fc9b4077fc0 msgr2=0x7fc9b40783d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:14:11.284 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:09.030+0000 7fc9bc000700 1 -- 172.21.15.118:0/3507311584 shutdown_connections 2024-03-24T00:14:11.284 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:09.030+0000 7fc9bc000700 1 -- 172.21.15.118:0/3507311584 wait complete. 2024-03-24T00:14:11.285 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:09.031+0000 7fc9bc000700 1 Processor -- start 2024-03-24T00:14:11.285 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:09.031+0000 7fc9bc000700 1 -- start start 2024-03-24T00:14:11.285 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:09.032+0000 7fc9bc000700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc9b4069e40 0x7fc9b41a0890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:14:11.285 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:09.032+0000 7fc9bc000700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fc9b406a7e0 0x7fc9b41a0dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:14:11.285 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:09.032+0000 7fc9bc000700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fc9b41081e0 0x7fc9b419aa50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:14:11.285 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:09.032+0000 7fc9bc000700 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc9b4112490 con 0x7fc9b406a7e0 2024-03-24T00:14:11.285 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:09.032+0000 7fc9bc000700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc9b4112310 con 0x7fc9b4069e40 2024-03-24T00:14:11.285 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:09.032+0000 7fc9b9600700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc9b4069e40 0x7fc9b41a0890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:14:11.285 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:09.032+0000 7fc9b9600700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc9b4069e40 0x7fc9b41a0890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.118:36026/0 (socket says 172.21.15.118:36026) 2024-03-24T00:14:11.285 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:09.032+0000 7fc9bc000700 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc9b4112610 con 0x7fc9b41081e0 2024-03-24T00:14:11.285 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:09.032+0000 7fc9b9600700 1 -- 172.21.15.118:0/2528565552 learned_addr learned my addr 172.21.15.118:0/2528565552 (peer_addr_for_me v2:172.21.15.118:0/0) 2024-03-24T00:14:11.285 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:09.032+0000 7fc9ba000700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fc9b41081e0 0x7fc9b419aa50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:14:11.286 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:09.032+0000 7fc9b8c00700 1 --2- 172.21.15.118:0/2528565552 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fc9b406a7e0 0x7fc9b41a0dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:14:11.286 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:09.033+0000 7fc9b9600700 1 -- 172.21.15.118:0/2528565552 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fc9b41081e0 msgr2=0x7fc9b419aa50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:14:11.286 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:09.033+0000 7fc9b9600700 1 --2- 172.21.15.118:0/2528565552 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fc9b41081e0 0x7fc9b419aa50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:14:11.286 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:09.033+0000 7fc9b9600700 1 -- 172.21.15.118:0/2528565552 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fc9b406a7e0 msgr2=0x7fc9b41a0dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:14:11.286 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:09.033+0000 7fc9b9600700 1 --2- 172.21.15.118:0/2528565552 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fc9b406a7e0 0x7fc9b41a0dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:14:11.286 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:09.033+0000 7fc9b9600700 1 -- 172.21.15.118:0/2528565552 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc9b419b1b0 con 0x7fc9b4069e40 2024-03-24T00:14:11.286 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:09.033+0000 7fc9ba000700 1 --2- 172.21.15.118:0/2528565552 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fc9b41081e0 0x7fc9b419aa50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T00:14:11.286 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:09.033+0000 7fc9b9600700 1 --2- 172.21.15.118:0/2528565552 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc9b4069e40 0x7fc9b41a0890 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7fc9a800eb30 tx=0x7fc9a800ee40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:14:11.286 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:09.033+0000 7fc9b2000700 1 -- 172.21.15.118:0/2528565552 <== mon.2 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc9a800cbd0 con 0x7fc9b4069e40 2024-03-24T00:14:11.286 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:09.033+0000 7fc9bc000700 1 -- 172.21.15.118:0/2528565552 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc9b419b4a0 con 0x7fc9b4069e40 2024-03-24T00:14:11.286 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:09.033+0000 7fc9b2000700 1 -- 172.21.15.118:0/2528565552 <== mon.2 v2:172.21.15.189:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fc9a8004500 con 0x7fc9b4069e40 2024-03-24T00:14:11.287 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:09.033+0000 7fc9b2000700 1 -- 172.21.15.118:0/2528565552 <== mon.2 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc9a8010430 con 0x7fc9b4069e40 2024-03-24T00:14:11.287 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:09.033+0000 7fc9bc000700 1 -- 172.21.15.118:0/2528565552 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc9b41a7760 con 0x7fc9b4069e40 2024-03-24T00:14:11.287 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:09.034+0000 7fc9bc000700 1 -- 172.21.15.118:0/2528565552 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7fc9b410dd50 con 0x7fc9b4069e40 2024-03-24T00:14:11.287 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:09.036+0000 7fc9b2000700 1 -- 172.21.15.118:0/2528565552 <== mon.2 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 17) v1 ==== 137342+0+0 (secure 0 0 0) 0x7fc9a8003f00 con 0x7fc9b4069e40 2024-03-24T00:14:11.287 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:09.038+0000 7fc9b2000700 1 --2- 172.21.15.118:0/2528565552 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7fc99c0a2920 0x7fc99c0a4de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:14:11.287 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:09.038+0000 7fc9b2000700 1 -- 172.21.15.118:0/2528565552 <== mon.2 v2:172.21.15.189:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 8248+0+0 (secure 0 0 0) 0x7fc9a80cf1b0 con 0x7fc9b4069e40 2024-03-24T00:14:11.287 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:09.038+0000 7fc9b8c00700 1 --2- 172.21.15.118:0/2528565552 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7fc99c0a2920 0x7fc99c0a4de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:14:11.287 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:09.038+0000 7fc9b2000700 1 --2- 172.21.15.118:0/2528565552 >> [v2:172.21.15.193:6808/2920363402,v1:172.21.15.193:6809/2920363402] conn(0x7fc99c0ac4e0 0x7fc99c0ae980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:14:11.287 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:09.038+0000 7fc9b2000700 1 -- 172.21.15.118:0/2528565552 --> [v2:172.21.15.193:6808/2920363402,v1:172.21.15.193:6809/2920363402] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fc9a8012ec0 con 0x7fc99c0ac4e0 2024-03-24T00:14:11.287 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:09.038+0000 7fc9b2000700 1 -- 172.21.15.118:0/2528565552 <== mon.2 v2:172.21.15.189:3300/0 6 ==== mon_get_version_reply(handle=1 version=88) v2 ==== 24+0+0 (secure 0 0 0) 0x7fc9a80cf560 con 0x7fc9b4069e40 2024-03-24T00:14:11.287 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:09.038+0000 7fc9ba000700 1 --2- 172.21.15.118:0/2528565552 >> [v2:172.21.15.193:6808/2920363402,v1:172.21.15.193:6809/2920363402] conn(0x7fc99c0ac4e0 0x7fc99c0ae980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:14:11.287 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:09.039+0000 7fc9b8c00700 1 --2- 172.21.15.118:0/2528565552 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7fc99c0a2920 0x7fc99c0a4de0 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7fc9b4103150 tx=0x7fc9a0005fb0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:14:11.287 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:09.039+0000 7fc9ba000700 1 --2- 172.21.15.118:0/2528565552 >> [v2:172.21.15.193:6808/2920363402,v1:172.21.15.193:6809/2920363402] conn(0x7fc99c0ac4e0 0x7fc99c0ae980 crc :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:14:11.288 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:09.040+0000 7fc9b2000700 1 -- 172.21.15.118:0/2528565552 <== osd.5 v2:172.21.15.193:6808/2920363402 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+27193 (crc 0 0 0) 0x7fc9a8012ec0 con 0x7fc99c0ac4e0 2024-03-24T00:14:11.288 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:09.058+0000 7fc9bc000700 1 -- 172.21.15.118:0/2528565552 --> [v2:172.21.15.193:6808/2920363402,v1:172.21.15.193:6809/2920363402] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7fc9b4103890 con 0x7fc99c0ac4e0 2024-03-24T00:14:11.288 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:09.058+0000 7fc9b2000700 1 -- 172.21.15.118:0/2528565552 <== osd.5 v2:172.21.15.193:6808/2920363402 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+12 (crc 0 0 0) 0x7fc9b4103890 con 0x7fc99c0ac4e0 2024-03-24T00:14:11.288 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:09.058+0000 7fc9bc000700 1 -- 172.21.15.118:0/2528565552 >> [v2:172.21.15.193:6808/2920363402,v1:172.21.15.193:6809/2920363402] conn(0x7fc99c0ac4e0 msgr2=0x7fc99c0ae980 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:14:11.288 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:09.058+0000 7fc9bc000700 1 --2- 172.21.15.118:0/2528565552 >> [v2:172.21.15.193:6808/2920363402,v1:172.21.15.193:6809/2920363402] conn(0x7fc99c0ac4e0 0x7fc99c0ae980 crc :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:14:11.288 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:09.058+0000 7fc9bc000700 1 -- 172.21.15.118:0/2528565552 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7fc99c0a2920 msgr2=0x7fc99c0a4de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:14:11.288 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:09.058+0000 7fc9bc000700 1 --2- 172.21.15.118:0/2528565552 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7fc99c0a2920 0x7fc99c0a4de0 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7fc9b4103150 tx=0x7fc9a0005fb0 comp rx=0 tx=0).stop 2024-03-24T00:14:11.289 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:09.059+0000 7fc9bc000700 1 -- 172.21.15.118:0/2528565552 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc9b4069e40 msgr2=0x7fc9b41a0890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:14:11.289 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:09.059+0000 7fc9bc000700 1 --2- 172.21.15.118:0/2528565552 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc9b4069e40 0x7fc9b41a0890 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7fc9a800eb30 tx=0x7fc9a800ee40 comp rx=0 tx=0).stop 2024-03-24T00:14:11.289 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:09.059+0000 7fc9ba000700 1 -- 172.21.15.118:0/2528565552 reap_dead start 2024-03-24T00:14:11.289 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:09.059+0000 7fc9bc000700 1 -- 172.21.15.118:0/2528565552 shutdown_connections 2024-03-24T00:14:11.289 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:09.059+0000 7fc9bc000700 1 -- 172.21.15.118:0/2528565552 >> 172.21.15.118:0/2528565552 conn(0x7fc9b4077fc0 msgr2=0x7fc9b410d370 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:14:11.289 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:09.059+0000 7fc9bc000700 1 -- 172.21.15.118:0/2528565552 shutdown_connections 2024-03-24T00:14:11.289 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:09.059+0000 7fc9bc000700 1 -- 172.21.15.118:0/2528565552 wait complete. 2024-03-24T00:14:12.334 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:12.334+0000 7f68c0800700 1 -- 172.21.15.118:0/3138545556 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f68b8069e40 msgr2=0x7f68b806a220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:14:12.335 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:12.334+0000 7f68c0800700 1 --2- 172.21.15.118:0/3138545556 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f68b8069e40 0x7f68b806a220 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7f68ac009990 tx=0x7f68ac009ca0 comp rx=0 tx=0).stop 2024-03-24T00:14:12.335 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:12.335+0000 7f68c0800700 1 -- 172.21.15.118:0/3138545556 shutdown_connections 2024-03-24T00:14:12.335 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:12.335+0000 7f68c0800700 1 --2- 172.21.15.118:0/3138545556 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f68b8069e40 0x7f68b806a220 unknown :-1 s=CLOSED pgs=113 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:14:12.335 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:12.335+0000 7f68c0800700 1 --2- 172.21.15.118:0/3138545556 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f68b810ec40 0x7f68b8111030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:14:12.335 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:12.335+0000 7f68c0800700 1 --2- 172.21.15.118:0/3138545556 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f68b806a760 0x7f68b810e700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:14:12.335 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:12.335+0000 7f68c0800700 1 -- 172.21.15.118:0/3138545556 >> 172.21.15.118:0/3138545556 conn(0x7f68b8077fc0 msgr2=0x7f68b80783d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:14:12.335 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:12.335+0000 7f68c0800700 1 -- 172.21.15.118:0/3138545556 shutdown_connections 2024-03-24T00:14:12.335 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:12.335+0000 7f68c0800700 1 -- 172.21.15.118:0/3138545556 wait complete. 2024-03-24T00:14:12.336 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:12.336+0000 7f68c0800700 1 Processor -- start 2024-03-24T00:14:12.336 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:12.336+0000 7f68c0800700 1 -- start start 2024-03-24T00:14:12.336 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:12.337+0000 7f68c0800700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f68b8069e40 0x7f68b819ac70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:14:12.336 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:12.337+0000 7f68c0800700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f68b806a760 0x7f68b819b1b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:14:12.337 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:12.337+0000 7f68c0800700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f68b810ec40 0x7f68b819f570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:14:12.337 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:12.337+0000 7f68c0800700 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f68b81146c0 con 0x7f68b806a760 2024-03-24T00:14:12.337 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:12.337+0000 7f68c0800700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f68b8114540 con 0x7f68b8069e40 2024-03-24T00:14:12.337 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:12.337+0000 7f68c0800700 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f68b8114840 con 0x7f68b810ec40 2024-03-24T00:14:12.338 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:12.337+0000 7f68bde00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f68b8069e40 0x7f68b819ac70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:14:12.338 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:12.337+0000 7f68bde00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f68b8069e40 0x7f68b819ac70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.118:36074/0 (socket says 172.21.15.118:36074) 2024-03-24T00:14:12.338 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:12.337+0000 7f68bde00700 1 -- 172.21.15.118:0/3813785898 learned_addr learned my addr 172.21.15.118:0/3813785898 (peer_addr_for_me v2:172.21.15.118:0/0) 2024-03-24T00:14:12.338 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:12.337+0000 7f68bd400700 1 --2- 172.21.15.118:0/3813785898 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f68b806a760 0x7f68b819b1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:14:12.338 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:12.337+0000 7f68be800700 1 --2- 172.21.15.118:0/3813785898 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f68b810ec40 0x7f68b819f570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:14:12.338 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:12.338+0000 7f68bde00700 1 -- 172.21.15.118:0/3813785898 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f68b810ec40 msgr2=0x7f68b819f570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:14:12.338 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:12.338+0000 7f68bde00700 1 --2- 172.21.15.118:0/3813785898 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f68b810ec40 0x7f68b819f570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:14:12.338 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:12.338+0000 7f68bde00700 1 -- 172.21.15.118:0/3813785898 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f68b806a760 msgr2=0x7f68b819b1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:14:12.338 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:12.338+0000 7f68bde00700 1 --2- 172.21.15.118:0/3813785898 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f68b806a760 0x7f68b819b1b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:14:12.338 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:12.338+0000 7f68bde00700 1 -- 172.21.15.118:0/3813785898 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f68b819fc70 con 0x7f68b8069e40 2024-03-24T00:14:12.339 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:12.338+0000 7f68be800700 1 --2- 172.21.15.118:0/3813785898 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f68b810ec40 0x7f68b819f570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T00:14:12.339 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:12.338+0000 7f68bde00700 1 --2- 172.21.15.118:0/3813785898 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f68b8069e40 0x7f68b819ac70 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7f68ac005910 tx=0x7f68ac00bab0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:14:12.339 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:12.338+0000 7f68b2a00700 1 -- 172.21.15.118:0/3813785898 <== mon.2 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f68ac027070 con 0x7f68b8069e40 2024-03-24T00:14:12.339 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:12.338+0000 7f68b2a00700 1 -- 172.21.15.118:0/3813785898 <== mon.2 v2:172.21.15.189:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f68ac004650 con 0x7f68b8069e40 2024-03-24T00:14:12.339 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:12.338+0000 7f68c0800700 1 -- 172.21.15.118:0/3813785898 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f68b819ff00 con 0x7f68b8069e40 2024-03-24T00:14:12.339 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:12.338+0000 7f68b2a00700 1 -- 172.21.15.118:0/3813785898 <== mon.2 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f68ac00f460 con 0x7f68b8069e40 2024-03-24T00:14:12.339 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:12.339+0000 7f68c0800700 1 -- 172.21.15.118:0/3813785898 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f68b81a7820 con 0x7f68b8069e40 2024-03-24T00:14:12.339 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:12.340+0000 7f68c0800700 1 -- 172.21.15.118:0/3813785898 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f68b8068e50 con 0x7f68b8069e40 2024-03-24T00:14:12.341 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:12.341+0000 7f68b2a00700 1 -- 172.21.15.118:0/3813785898 <== mon.2 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 17) v1 ==== 137342+0+0 (secure 0 0 0) 0x7f68ac02a070 con 0x7f68b8069e40 2024-03-24T00:14:12.342 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:12.343+0000 7f68b2a00700 1 --2- 172.21.15.118:0/3813785898 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7f68a00a2920 0x7f68a00a4de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:14:12.342 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:12.343+0000 7f68b2a00700 1 -- 172.21.15.118:0/3813785898 <== mon.2 v2:172.21.15.189:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 8248+0+0 (secure 0 0 0) 0x7f68ac0cfb50 con 0x7f68b8069e40 2024-03-24T00:14:12.343 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:12.343+0000 7f68bd400700 1 --2- 172.21.15.118:0/3813785898 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7f68a00a2920 0x7f68a00a4de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:14:12.343 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:12.344+0000 7f68bd400700 1 --2- 172.21.15.118:0/3813785898 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7f68a00a2920 0x7f68a00a4de0 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7f68b8102520 tx=0x7f68a8009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:14:12.346 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:12.347+0000 7f68b2a00700 1 -- 172.21.15.118:0/3813785898 <== mon.2 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f68ac05e740 con 0x7f68b8069e40 2024-03-24T00:14:12.482 INFO:teuthology.orchestra.run.smithi118.stdout:167503724605 2024-03-24T00:14:12.482 DEBUG:teuthology.orchestra.run.smithi118:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 shell --fsid d79a5420-e971-11ee-95cd-87774f69a715 -- ceph osd last-stat-seq osd.5 2024-03-24T00:14:12.636 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:12.636+0000 7f68c0800700 1 -- 172.21.15.118:0/3813785898 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 3} v 0) v1 -- 0x7f68b802da30 con 0x7f68b8069e40 2024-03-24T00:14:12.636 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:12.636+0000 7f68b2a00700 1 -- 172.21.15.118:0/3813785898 <== mon.2 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 3}]=0 v0) v1 ==== 74+0+13 (secure 0 0 0) 0x7f68ac0bae20 con 0x7f68b8069e40 2024-03-24T00:14:12.637 INFO:teuthology.orchestra.run.smithi118.stdout:111669149769 2024-03-24T00:14:12.638 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:12.639+0000 7f68c0800700 1 -- 172.21.15.118:0/3813785898 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7f68a00a2920 msgr2=0x7f68a00a4de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:14:12.639 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:12.639+0000 7f68c0800700 1 --2- 172.21.15.118:0/3813785898 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7f68a00a2920 0x7f68a00a4de0 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7f68b8102520 tx=0x7f68a8009260 comp rx=0 tx=0).stop 2024-03-24T00:14:12.639 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:12.639+0000 7f68c0800700 1 -- 172.21.15.118:0/3813785898 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f68b8069e40 msgr2=0x7f68b819ac70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:14:12.639 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:12.639+0000 7f68c0800700 1 --2- 172.21.15.118:0/3813785898 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f68b8069e40 0x7f68b819ac70 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7f68ac005910 tx=0x7f68ac00bab0 comp rx=0 tx=0).stop 2024-03-24T00:14:12.639 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:12.639+0000 7f68c0800700 1 -- 172.21.15.118:0/3813785898 shutdown_connections 2024-03-24T00:14:12.639 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:12.639+0000 7f68c0800700 1 --2- 172.21.15.118:0/3813785898 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f68b8069e40 0x7f68b819ac70 unknown :-1 s=CLOSED pgs=114 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:14:12.640 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:12.639+0000 7f68c0800700 1 --2- 172.21.15.118:0/3813785898 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7f68a00a2920 0x7f68a00a4de0 unknown :-1 s=CLOSED pgs=195 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:14:12.640 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:12.639+0000 7f68c0800700 1 --2- 172.21.15.118:0/3813785898 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f68b810ec40 0x7f68b819f570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:14:12.640 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:12.639+0000 7f68c0800700 1 --2- 172.21.15.118:0/3813785898 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f68b806a760 0x7f68b819b1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:14:12.640 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:12.639+0000 7f68c0800700 1 -- 172.21.15.118:0/3813785898 >> 172.21.15.118:0/3813785898 conn(0x7f68b8077fc0 msgr2=0x7f68b8110150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:14:12.640 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:12.639+0000 7f68c0800700 1 -- 172.21.15.118:0/3813785898 shutdown_connections 2024-03-24T00:14:12.640 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:12.640+0000 7f68c0800700 1 -- 172.21.15.118:0/3813785898 wait complete. 2024-03-24T00:14:12.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:14:12 smithi118 ceph-mon[31535]: pgmap v351: 97 pgs: 1 activating, 1 peering, 95 active+clean; 644 KiB data, 344 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 9 op/s; 7.6 KiB/s, 1 objects/s recovering 2024-03-24T00:14:12.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:14:12 smithi118 ceph-mon[31535]: from='client.? 172.21.15.118:0/3813785898' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2024-03-24T00:14:12.953 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:14:12 smithi189 ceph-mon[36187]: pgmap v351: 97 pgs: 1 activating, 1 peering, 95 active+clean; 644 KiB data, 344 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 9 op/s; 7.6 KiB/s, 1 objects/s recovering 2024-03-24T00:14:12.953 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:14:12 smithi189 ceph-mon[36187]: from='client.? 172.21.15.118:0/3813785898' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2024-03-24T00:14:12.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:14:12 smithi193 ceph-mon[36878]: pgmap v351: 97 pgs: 1 activating, 1 peering, 95 active+clean; 644 KiB data, 344 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 9 op/s; 7.6 KiB/s, 1 objects/s recovering 2024-03-24T00:14:12.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:14:12 smithi193 ceph-mon[36878]: from='client.? 172.21.15.118:0/3813785898' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2024-03-24T00:14:13.857 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:14:13 smithi118 ceph-mon[31535]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg peering) 2024-03-24T00:14:13.857 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:14:13 smithi118 ceph-mon[31535]: Cluster is now healthy 2024-03-24T00:14:14.203 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:14:13 smithi189 ceph-mon[36187]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg peering) 2024-03-24T00:14:14.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:14:13 smithi189 ceph-mon[36187]: Cluster is now healthy 2024-03-24T00:14:14.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:14:13 smithi193 ceph-mon[36878]: Health check cleared: PG_AVAILABILITY (was: Reduced data availability: 1 pg peering) 2024-03-24T00:14:14.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:14:13 smithi193 ceph-mon[36878]: Cluster is now healthy 2024-03-24T00:14:14.228 INFO:tasks.cephadm.ceph_manager.ceph:need seq 257698037800 got 257698037803 for osd.8 2024-03-24T00:14:14.228 DEBUG:teuthology.parallel:result is None 2024-03-24T00:14:14.425 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:13.857+0000 7f2a0b200700 1 -- 172.21.15.118:0/1872246537 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2a04104fb0 msgr2=0x7f2a04109000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:14:14.425 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:13.857+0000 7f2a0b200700 1 --2- 172.21.15.118:0/1872246537 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2a04104fb0 0x7f2a04109000 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f29f0009990 tx=0x7f29f0009ca0 comp rx=0 tx=0).stop 2024-03-24T00:14:14.425 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:13.857+0000 7f2a0b200700 1 -- 172.21.15.118:0/1872246537 shutdown_connections 2024-03-24T00:14:14.425 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:13.857+0000 7f2a0b200700 1 --2- 172.21.15.118:0/1872246537 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f2a041096e0 0x7f2a041112b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:14:14.425 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:13.857+0000 7f2a0b200700 1 --2- 172.21.15.118:0/1872246537 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f2a04104600 0x7f2a041049e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:14:14.425 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:13.857+0000 7f2a0b200700 1 --2- 172.21.15.118:0/1872246537 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2a04104fb0 0x7f2a04109000 unknown :-1 s=CLOSED pgs=115 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:14:14.425 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:13.857+0000 7f2a0b200700 1 -- 172.21.15.118:0/1872246537 >> 172.21.15.118:0/1872246537 conn(0x7f2a040ffe90 msgr2=0x7f2a041022b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:14:14.425 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:13.857+0000 7f2a0b200700 1 -- 172.21.15.118:0/1872246537 shutdown_connections 2024-03-24T00:14:14.426 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:13.857+0000 7f2a0b200700 1 -- 172.21.15.118:0/1872246537 wait complete. 2024-03-24T00:14:14.426 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:13.858+0000 7f2a0b200700 1 Processor -- start 2024-03-24T00:14:14.426 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:13.858+0000 7f2a0b200700 1 -- start start 2024-03-24T00:14:14.426 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:13.859+0000 7f2a0b200700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2a04104600 0x7f2a041a0820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:14:14.426 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:13.859+0000 7f2a0b200700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f2a04104fb0 0x7f2a041a0d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:14:14.426 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:13.859+0000 7f2a0b200700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f2a041096e0 0x7f2a0419a9e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:14:14.426 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:13.859+0000 7f2a0b200700 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2a041146c0 con 0x7f2a04104fb0 2024-03-24T00:14:14.426 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:13.859+0000 7f2a0b200700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2a04114540 con 0x7f2a04104600 2024-03-24T00:14:14.426 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:13.859+0000 7f2a0b200700 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2a04114840 con 0x7f2a041096e0 2024-03-24T00:14:14.426 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:13.859+0000 7f2a03e00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2a04104600 0x7f2a041a0820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:14:14.426 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:13.859+0000 7f2a09200700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f2a041096e0 0x7f2a0419a9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:14:14.426 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:13.859+0000 7f2a03400700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f2a04104fb0 0x7f2a041a0d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:14:14.426 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:13.859+0000 7f2a03e00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2a04104600 0x7f2a041a0820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.118:51400/0 (socket says 172.21.15.118:51400) 2024-03-24T00:14:14.427 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:13.859+0000 7f2a03400700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f2a04104fb0 0x7f2a041a0d60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.118:3300/0 says I am v2:172.21.15.118:56726/0 (socket says 172.21.15.118:56726) 2024-03-24T00:14:14.427 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:13.859+0000 7f2a09200700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f2a041096e0 0x7f2a0419a9e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.118:33010/0 (socket says 172.21.15.118:33010) 2024-03-24T00:14:14.427 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:13.859+0000 7f2a03e00700 1 -- 172.21.15.118:0/2027908098 learned_addr learned my addr 172.21.15.118:0/2027908098 (peer_addr_for_me v2:172.21.15.118:0/0) 2024-03-24T00:14:14.427 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:13.860+0000 7f2a03e00700 1 -- 172.21.15.118:0/2027908098 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f2a041096e0 msgr2=0x7f2a0419a9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:14:14.427 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:13.860+0000 7f2a03e00700 1 --2- 172.21.15.118:0/2027908098 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f2a041096e0 0x7f2a0419a9e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:14:14.427 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:13.860+0000 7f2a03e00700 1 -- 172.21.15.118:0/2027908098 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f2a04104fb0 msgr2=0x7f2a041a0d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:14:14.427 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:13.860+0000 7f2a03e00700 1 --2- 172.21.15.118:0/2027908098 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f2a04104fb0 0x7f2a041a0d60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:14:14.427 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:13.860+0000 7f2a03e00700 1 -- 172.21.15.118:0/2027908098 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2a0419b140 con 0x7f2a04104600 2024-03-24T00:14:14.427 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:13.860+0000 7f2a09200700 1 --2- 172.21.15.118:0/2027908098 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f2a041096e0 0x7f2a0419a9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T00:14:14.427 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:13.860+0000 7f2a03e00700 1 --2- 172.21.15.118:0/2027908098 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2a04104600 0x7f2a041a0820 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f29f400eb30 tx=0x7f29f400ee40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:14:14.427 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:13.860+0000 7f2a00c00700 1 -- 172.21.15.118:0/2027908098 <== mon.2 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f29f400cbd0 con 0x7f2a04104600 2024-03-24T00:14:14.427 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:13.860+0000 7f2a00c00700 1 -- 172.21.15.118:0/2027908098 <== mon.2 v2:172.21.15.189:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f29f4004500 con 0x7f2a04104600 2024-03-24T00:14:14.427 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:13.860+0000 7f2a0b200700 1 -- 172.21.15.118:0/2027908098 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2a0419b430 con 0x7f2a04104600 2024-03-24T00:14:14.427 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:13.860+0000 7f2a00c00700 1 -- 172.21.15.118:0/2027908098 <== mon.2 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f29f4010430 con 0x7f2a04104600 2024-03-24T00:14:14.428 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:13.860+0000 7f2a0b200700 1 -- 172.21.15.118:0/2027908098 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2a041a76f0 con 0x7f2a04104600 2024-03-24T00:14:14.428 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:13.861+0000 7f2a0b200700 1 -- 172.21.15.118:0/2027908098 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2a04068e50 con 0x7f2a04104600 2024-03-24T00:14:14.428 INFO:teuthology.orchestra.run.smithi118.stdout:313532612640 2024-03-24T00:14:14.428 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:13.863+0000 7f2a00c00700 1 -- 172.21.15.118:0/2027908098 <== mon.2 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 17) v1 ==== 137342+0+0 (secure 0 0 0) 0x7f29f4003f00 con 0x7f2a04104600 2024-03-24T00:14:14.428 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:13.864+0000 7f2a00c00700 1 --2- 172.21.15.118:0/2027908098 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7f29e80a2920 0x7f29e80a4de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:14:14.429 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:13.865+0000 7f2a00c00700 1 -- 172.21.15.118:0/2027908098 <== mon.2 v2:172.21.15.189:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 8248+0+0 (secure 0 0 0) 0x7f29f40cef90 con 0x7f2a04104600 2024-03-24T00:14:14.429 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:13.865+0000 7f2a03400700 1 --2- 172.21.15.118:0/2027908098 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7f29e80a2920 0x7f29e80a4de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:14:14.429 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:13.865+0000 7f2a03400700 1 --2- 172.21.15.118:0/2027908098 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7f29e80a2920 0x7f29e80a4de0 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7f29f000b2e0 tx=0x7f29f0005960 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:14:14.429 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:13.873+0000 7f2a00c00700 1 -- 172.21.15.118:0/2027908098 <== mon.2 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f29f409be60 con 0x7f2a04104600 2024-03-24T00:14:14.429 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:14.124+0000 7f2a0b200700 1 -- 172.21.15.118:0/2027908098 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 10} v 0) v1 -- 0x7f2a0419bf10 con 0x7f2a04104600 2024-03-24T00:14:14.429 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:14.124+0000 7f2a00c00700 1 -- 172.21.15.118:0/2027908098 <== mon.2 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 10}]=0 v0) v1 ==== 75+0+13 (secure 0 0 0) 0x7f29f409b760 con 0x7f2a04104600 2024-03-24T00:14:14.429 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:14.127+0000 7f2a0b200700 1 -- 172.21.15.118:0/2027908098 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7f29e80a2920 msgr2=0x7f29e80a4de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:14:14.429 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:14.127+0000 7f2a0b200700 1 --2- 172.21.15.118:0/2027908098 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7f29e80a2920 0x7f29e80a4de0 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7f29f000b2e0 tx=0x7f29f0005960 comp rx=0 tx=0).stop 2024-03-24T00:14:14.429 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:14.127+0000 7f2a0b200700 1 -- 172.21.15.118:0/2027908098 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2a04104600 msgr2=0x7f2a041a0820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:14:14.429 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:14.127+0000 7f2a0b200700 1 --2- 172.21.15.118:0/2027908098 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2a04104600 0x7f2a041a0820 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f29f400eb30 tx=0x7f29f400ee40 comp rx=0 tx=0).stop 2024-03-24T00:14:14.429 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:14.127+0000 7f2a0b200700 1 -- 172.21.15.118:0/2027908098 shutdown_connections 2024-03-24T00:14:14.429 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:14.127+0000 7f2a0b200700 1 --2- 172.21.15.118:0/2027908098 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2a04104600 0x7f2a041a0820 unknown :-1 s=CLOSED pgs=116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:14:14.429 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:14.127+0000 7f2a0b200700 1 --2- 172.21.15.118:0/2027908098 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7f29e80a2920 0x7f29e80a4de0 unknown :-1 s=CLOSED pgs=196 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:14:14.429 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:14.127+0000 7f2a0b200700 1 --2- 172.21.15.118:0/2027908098 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f2a041096e0 0x7f2a0419a9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:14:14.430 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:14.127+0000 7f2a0b200700 1 --2- 172.21.15.118:0/2027908098 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f2a04104fb0 0x7f2a041a0d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:14:14.430 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:14.127+0000 7f2a0b200700 1 -- 172.21.15.118:0/2027908098 >> 172.21.15.118:0/2027908098 conn(0x7f2a040ffe90 msgr2=0x7f2a04100a80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:14:14.430 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:14.128+0000 7f2a0b200700 1 -- 172.21.15.118:0/2027908098 shutdown_connections 2024-03-24T00:14:14.430 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:14.128+0000 7f2a0b200700 1 -- 172.21.15.118:0/2027908098 wait complete. 2024-03-24T00:14:14.921 INFO:teuthology.orchestra.run.smithi118.stderr:Inferring config /var/lib/ceph/d79a5420-e971-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T00:14:15.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:14:14 smithi118 ceph-mon[31535]: pgmap v352: 97 pgs: 97 active+clean; 644 KiB data, 344 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 7 op/s; 7.6 KiB/s, 1 objects/s recovering 2024-03-24T00:14:15.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:14:14 smithi118 ceph-mon[31535]: from='client.? 172.21.15.118:0/2027908098' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 10}]: dispatch 2024-03-24T00:14:15.203 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:14:14 smithi189 ceph-mon[36187]: pgmap v352: 97 pgs: 97 active+clean; 644 KiB data, 344 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 7 op/s; 7.6 KiB/s, 1 objects/s recovering 2024-03-24T00:14:15.203 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:14:14 smithi189 ceph-mon[36187]: from='client.? 172.21.15.118:0/2027908098' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 10}]: dispatch 2024-03-24T00:14:15.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:14:14 smithi193 ceph-mon[36878]: pgmap v352: 97 pgs: 97 active+clean; 644 KiB data, 344 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 7 op/s; 7.6 KiB/s, 1 objects/s recovering 2024-03-24T00:14:15.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:14:14 smithi193 ceph-mon[36878]: from='client.? 172.21.15.118:0/2027908098' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 10}]: dispatch 2024-03-24T00:14:16.581 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:16.581+0000 7f6bc4000700 1 -- 172.21.15.118:0/1584282413 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6bbc06aad0 msgr2=0x7f6bbc06aeb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:14:16.581 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:16.581+0000 7f6bc4000700 1 --2- 172.21.15.118:0/1584282413 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6bbc06aad0 0x7f6bbc06aeb0 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f6bb00098f0 tx=0x7f6bb0009c00 comp rx=0 tx=0).stop 2024-03-24T00:14:16.582 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:16.581+0000 7f6bc4000700 1 -- 172.21.15.118:0/1584282413 shutdown_connections 2024-03-24T00:14:16.582 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:16.581+0000 7f6bc4000700 1 --2- 172.21.15.118:0/1584282413 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6bbc06aad0 0x7f6bbc06aeb0 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:14:16.582 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:16.581+0000 7f6bc4000700 1 --2- 172.21.15.118:0/1584282413 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f6bbc107540 0x7f6bbc10f0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:14:16.582 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:16.581+0000 7f6bc4000700 1 --2- 172.21.15.118:0/1584282413 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f6bbc06b480 0x7f6bbc106e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:14:16.582 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:16.581+0000 7f6bc4000700 1 -- 172.21.15.118:0/1584282413 >> 172.21.15.118:0/1584282413 conn(0x7f6bbc0770e0 msgr2=0x7f6bbc0774f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:14:16.582 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:16.581+0000 7f6bc4000700 1 -- 172.21.15.118:0/1584282413 shutdown_connections 2024-03-24T00:14:16.582 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:16.581+0000 7f6bc4000700 1 -- 172.21.15.118:0/1584282413 wait complete. 2024-03-24T00:14:16.584 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:16.582+0000 7f6bc4000700 1 Processor -- start 2024-03-24T00:14:16.584 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:16.583+0000 7f6bc4000700 1 -- start start 2024-03-24T00:14:16.584 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:16.583+0000 7f6bc4000700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6bbc06aad0 0x7f6bbc1a0820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:14:16.584 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:16.583+0000 7f6bc4000700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f6bbc06b480 0x7f6bbc1a0d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:14:16.585 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:16.583+0000 7f6bc4000700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f6bbc107540 0x7f6bbc19a9e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:14:16.585 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:16.583+0000 7f6bc4000700 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6bbc112520 con 0x7f6bbc107540 2024-03-24T00:14:16.585 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:16.583+0000 7f6bc4000700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6bbc1123a0 con 0x7f6bbc06aad0 2024-03-24T00:14:16.585 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:16.583+0000 7f6bc4000700 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6bbc1126a0 con 0x7f6bbc06b480 2024-03-24T00:14:16.585 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:16.583+0000 7f6bc1600700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6bbc06aad0 0x7f6bbc1a0820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:14:16.585 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:16.583+0000 7f6bc0c00700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f6bbc06b480 0x7f6bbc1a0d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:14:16.585 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:16.583+0000 7f6bc1600700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6bbc06aad0 0x7f6bbc1a0820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.118:51424/0 (socket says 172.21.15.118:51424) 2024-03-24T00:14:16.585 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:16.583+0000 7f6bc1600700 1 -- 172.21.15.118:0/1313356030 learned_addr learned my addr 172.21.15.118:0/1313356030 (peer_addr_for_me v2:172.21.15.118:0/0) 2024-03-24T00:14:16.585 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:16.584+0000 7f6bc2000700 1 --2- 172.21.15.118:0/1313356030 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f6bbc107540 0x7f6bbc19a9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:14:16.585 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:16.584+0000 7f6bc0c00700 1 -- 172.21.15.118:0/1313356030 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6bbc06aad0 msgr2=0x7f6bbc1a0820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:14:16.585 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:16.584+0000 7f6bc0c00700 1 --2- 172.21.15.118:0/1313356030 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6bbc06aad0 0x7f6bbc1a0820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:14:16.586 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:16.584+0000 7f6bc0c00700 1 -- 172.21.15.118:0/1313356030 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f6bbc107540 msgr2=0x7f6bbc19a9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:14:16.586 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:16.584+0000 7f6bc0c00700 1 --2- 172.21.15.118:0/1313356030 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f6bbc107540 0x7f6bbc19a9e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:14:16.586 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:16.584+0000 7f6bc0c00700 1 -- 172.21.15.118:0/1313356030 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6bbc19b140 con 0x7f6bbc06b480 2024-03-24T00:14:16.586 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:16.584+0000 7f6bc1600700 1 --2- 172.21.15.118:0/1313356030 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6bbc06aad0 0x7f6bbc1a0820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T00:14:16.586 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:16.584+0000 7f6bc2000700 1 --2- 172.21.15.118:0/1313356030 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f6bbc107540 0x7f6bbc19a9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T00:14:16.586 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:16.584+0000 7f6bc0c00700 1 --2- 172.21.15.118:0/1313356030 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f6bbc06b480 0x7f6bbc1a0d60 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7f6bbc103640 tx=0x7f6ba800ee70 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:14:16.586 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:16.584+0000 7f6bba000700 1 -- 172.21.15.118:0/1313356030 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6ba800cc60 con 0x7f6bbc06b480 2024-03-24T00:14:16.586 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:16.585+0000 7f6bba000700 1 -- 172.21.15.118:0/1313356030 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f6ba8004500 con 0x7f6bbc06b480 2024-03-24T00:14:16.586 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:16.585+0000 7f6bc4000700 1 -- 172.21.15.118:0/1313356030 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6bbc19b430 con 0x7f6bbc06b480 2024-03-24T00:14:16.586 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:16.585+0000 7f6bba000700 1 -- 172.21.15.118:0/1313356030 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6ba8010690 con 0x7f6bbc06b480 2024-03-24T00:14:16.587 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:16.585+0000 7f6bc4000700 1 -- 172.21.15.118:0/1313356030 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6bbc1a76f0 con 0x7f6bbc06b480 2024-03-24T00:14:16.587 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:16.586+0000 7f6bc4000700 1 -- 172.21.15.118:0/1313356030 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6bbc1038f0 con 0x7f6bbc06b480 2024-03-24T00:14:16.587 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:16.588+0000 7f6bba000700 1 -- 172.21.15.118:0/1313356030 <== mon.1 v2:172.21.15.193:3300/0 4 ==== mgrmap(e 17) v1 ==== 137342+0+0 (secure 0 0 0) 0x7f6ba8003e70 con 0x7f6bbc06b480 2024-03-24T00:14:16.589 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:16.589+0000 7f6bba000700 1 --2- 172.21.15.118:0/1313356030 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7f6ba40a29a0 0x7f6ba40a4e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:14:16.589 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:16.590+0000 7f6bba000700 1 -- 172.21.15.118:0/1313356030 <== mon.1 v2:172.21.15.193:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 8248+0+0 (secure 0 0 0) 0x7f6ba80ceeb0 con 0x7f6bbc06b480 2024-03-24T00:14:16.589 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:16.590+0000 7f6bc1600700 1 --2- 172.21.15.118:0/1313356030 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7f6ba40a29a0 0x7f6ba40a4e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:14:16.590 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:16.590+0000 7f6bc1600700 1 --2- 172.21.15.118:0/1313356030 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7f6ba40a29a0 0x7f6ba40a4e60 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7f6bb0009680 tx=0x7f6bb001a040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:14:16.597 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:16.598+0000 7f6bba000700 1 -- 172.21.15.118:0/1313356030 <== mon.1 v2:172.21.15.193:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f6ba809bd80 con 0x7f6bbc06b480 2024-03-24T00:14:16.886 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:16.887+0000 7f6bc4000700 1 -- 172.21.15.118:0/1313356030 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 9} v 0) v1 -- 0x7f6bbc1a7980 con 0x7f6bbc06b480 2024-03-24T00:14:16.887 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:16.887+0000 7f6bba000700 1 -- 172.21.15.118:0/1313356030 <== mon.1 v2:172.21.15.193:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 9}]=0 v0) v1 ==== 74+0+13 (secure 0 0 0) 0x7f6ba809b680 con 0x7f6bbc06b480 2024-03-24T00:14:16.887 INFO:teuthology.orchestra.run.smithi118.stdout:292057776166 2024-03-24T00:14:16.889 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:16.890+0000 7f6bc4000700 1 -- 172.21.15.118:0/1313356030 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7f6ba40a29a0 msgr2=0x7f6ba40a4e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:14:16.890 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:16.890+0000 7f6bc4000700 1 --2- 172.21.15.118:0/1313356030 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7f6ba40a29a0 0x7f6ba40a4e60 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7f6bb0009680 tx=0x7f6bb001a040 comp rx=0 tx=0).stop 2024-03-24T00:14:16.890 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:16.890+0000 7f6bc4000700 1 -- 172.21.15.118:0/1313356030 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f6bbc06b480 msgr2=0x7f6bbc1a0d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:14:16.890 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:16.890+0000 7f6bc4000700 1 --2- 172.21.15.118:0/1313356030 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f6bbc06b480 0x7f6bbc1a0d60 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7f6bbc103640 tx=0x7f6ba800ee70 comp rx=0 tx=0).stop 2024-03-24T00:14:16.890 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:16.890+0000 7f6bc4000700 1 -- 172.21.15.118:0/1313356030 shutdown_connections 2024-03-24T00:14:16.890 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:16.890+0000 7f6bc4000700 1 --2- 172.21.15.118:0/1313356030 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6bbc06aad0 0x7f6bbc1a0820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:14:16.890 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:16.890+0000 7f6bc4000700 1 --2- 172.21.15.118:0/1313356030 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7f6ba40a29a0 0x7f6ba40a4e60 unknown :-1 s=CLOSED pgs=197 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:14:16.890 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:16.890+0000 7f6bc4000700 1 --2- 172.21.15.118:0/1313356030 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f6bbc107540 0x7f6bbc19a9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:14:16.890 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:16.890+0000 7f6bc4000700 1 --2- 172.21.15.118:0/1313356030 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f6bbc06b480 0x7f6bbc1a0d60 unknown :-1 s=CLOSED pgs=112 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:14:16.890 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:16.890+0000 7f6bc4000700 1 -- 172.21.15.118:0/1313356030 >> 172.21.15.118:0/1313356030 conn(0x7f6bbc0770e0 msgr2=0x7f6bbc0ff150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:14:16.891 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:16.890+0000 7f6bc4000700 1 -- 172.21.15.118:0/1313356030 shutdown_connections 2024-03-24T00:14:16.891 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:16.890+0000 7f6bc4000700 1 -- 172.21.15.118:0/1313356030 wait complete. 2024-03-24T00:14:16.919 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:14:16 smithi118 ceph-mon[31535]: pgmap v353: 97 pgs: 97 active+clean; 644 KiB data, 344 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 10 op/s; 6.7 KiB/s, 1 objects/s recovering 2024-03-24T00:14:17.203 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:14:16 smithi189 ceph-mon[36187]: pgmap v353: 97 pgs: 97 active+clean; 644 KiB data, 344 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 10 op/s; 6.7 KiB/s, 1 objects/s recovering 2024-03-24T00:14:17.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:14:16 smithi193 ceph-mon[36878]: pgmap v353: 97 pgs: 97 active+clean; 644 KiB data, 344 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 10 op/s; 6.7 KiB/s, 1 objects/s recovering 2024-03-24T00:14:18.014 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:14:17 smithi118 ceph-mon[31535]: from='client.? 172.21.15.118:0/1313356030' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 9}]: dispatch 2024-03-24T00:14:18.203 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:14:17 smithi189 ceph-mon[36187]: from='client.? 172.21.15.118:0/1313356030' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 9}]: dispatch 2024-03-24T00:14:18.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:14:17 smithi193 ceph-mon[36878]: from='client.? 172.21.15.118:0/1313356030' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 9}]: dispatch 2024-03-24T00:14:18.323 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:18.314+0000 7f5b73800700 1 -- 172.21.15.118:0/2508389870 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f5b6c108800 msgr2=0x7f5b6c108c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:14:18.323 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:18.314+0000 7f5b73800700 1 --2- 172.21.15.118:0/2508389870 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f5b6c108800 0x7f5b6c108c80 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7f5b58009990 tx=0x7f5b58009ca0 comp rx=0 tx=0).stop 2024-03-24T00:14:18.323 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:18.314+0000 7f5b73800700 1 -- 172.21.15.118:0/2508389870 shutdown_connections 2024-03-24T00:14:18.323 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:18.314+0000 7f5b73800700 1 --2- 172.21.15.118:0/2508389870 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5b6c10e800 0x7f5b6c10ebe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:14:18.323 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:18.314+0000 7f5b73800700 1 --2- 172.21.15.118:0/2508389870 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f5b6c1091c0 0x7f5b6c1156d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:14:18.323 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:18.314+0000 7f5b73800700 1 --2- 172.21.15.118:0/2508389870 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f5b6c108800 0x7f5b6c108c80 unknown :-1 s=CLOSED pgs=113 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:14:18.323 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:18.314+0000 7f5b73800700 1 -- 172.21.15.118:0/2508389870 >> 172.21.15.118:0/2508389870 conn(0x7f5b6c0767e0 msgr2=0x7f5b6c078c00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:14:18.323 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:18.314+0000 7f5b73800700 1 -- 172.21.15.118:0/2508389870 shutdown_connections 2024-03-24T00:14:18.324 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:18.314+0000 7f5b73800700 1 -- 172.21.15.118:0/2508389870 wait complete. 2024-03-24T00:14:18.324 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:18.315+0000 7f5b73800700 1 Processor -- start 2024-03-24T00:14:18.324 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:18.316+0000 7f5b73800700 1 -- start start 2024-03-24T00:14:18.324 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:18.316+0000 7f5b73800700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f5b6c108800 0x7f5b6c19f010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:14:18.324 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:18.316+0000 7f5b73800700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f5b6c1091c0 0x7f5b6c19f550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:14:18.324 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:18.316+0000 7f5b73800700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5b6c10e800 0x7f5b6c1a38e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:14:18.324 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:18.316+0000 7f5b73800700 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5b6c118be0 con 0x7f5b6c108800 2024-03-24T00:14:18.324 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:18.316+0000 7f5b73800700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5b6c118a60 con 0x7f5b6c10e800 2024-03-24T00:14:18.324 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:18.316+0000 7f5b73800700 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5b6c118d60 con 0x7f5b6c1091c0 2024-03-24T00:14:18.324 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:18.316+0000 7f5b6be00700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f5b6c1091c0 0x7f5b6c19f550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:14:18.324 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:18.316+0000 7f5b6be00700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f5b6c1091c0 0x7f5b6c19f550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.118:33044/0 (socket says 172.21.15.118:33044) 2024-03-24T00:14:18.324 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:18.316+0000 7f5b6be00700 1 -- 172.21.15.118:0/1289454349 learned_addr learned my addr 172.21.15.118:0/1289454349 (peer_addr_for_me v2:172.21.15.118:0/0) 2024-03-24T00:14:18.324 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:18.316+0000 7f5b71800700 1 --2- 172.21.15.118:0/1289454349 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5b6c10e800 0x7f5b6c1a38e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:14:18.325 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:18.316+0000 7f5b70e00700 1 --2- 172.21.15.118:0/1289454349 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f5b6c108800 0x7f5b6c19f010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:14:18.325 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:18.317+0000 7f5b6be00700 1 -- 172.21.15.118:0/1289454349 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5b6c10e800 msgr2=0x7f5b6c1a38e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:14:18.325 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:18.317+0000 7f5b6be00700 1 --2- 172.21.15.118:0/1289454349 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5b6c10e800 0x7f5b6c1a38e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:14:18.325 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:18.317+0000 7f5b6be00700 1 -- 172.21.15.118:0/1289454349 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f5b6c108800 msgr2=0x7f5b6c19f010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:14:18.325 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:18.317+0000 7f5b6be00700 1 --2- 172.21.15.118:0/1289454349 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f5b6c108800 0x7f5b6c19f010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:14:18.325 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:18.317+0000 7f5b6be00700 1 -- 172.21.15.118:0/1289454349 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5b6c1a4020 con 0x7f5b6c1091c0 2024-03-24T00:14:18.325 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:18.317+0000 7f5b6be00700 1 --2- 172.21.15.118:0/1289454349 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f5b6c1091c0 0x7f5b6c19f550 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7f5b5800b2e0 tx=0x7f5b5800b840 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:14:18.325 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:18.317+0000 7f5b69600700 1 -- 172.21.15.118:0/1289454349 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5b58005030 con 0x7f5b6c1091c0 2024-03-24T00:14:18.325 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:18.317+0000 7f5b69600700 1 -- 172.21.15.118:0/1289454349 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f5b580046d0 con 0x7f5b6c1091c0 2024-03-24T00:14:18.325 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:18.317+0000 7f5b73800700 1 -- 172.21.15.118:0/1289454349 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5b6c1a42b0 con 0x7f5b6c1091c0 2024-03-24T00:14:18.325 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:18.317+0000 7f5b69600700 1 -- 172.21.15.118:0/1289454349 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5b5800fab0 con 0x7f5b6c1091c0 2024-03-24T00:14:18.325 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:18.318+0000 7f5b73800700 1 -- 172.21.15.118:0/1289454349 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5b6c1abb90 con 0x7f5b6c1091c0 2024-03-24T00:14:18.325 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:18.319+0000 7f5b73800700 1 -- 172.21.15.118:0/1289454349 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5b6c068e50 con 0x7f5b6c1091c0 2024-03-24T00:14:18.325 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:18.321+0000 7f5b69600700 1 -- 172.21.15.118:0/1289454349 <== mon.1 v2:172.21.15.193:3300/0 4 ==== mgrmap(e 17) v1 ==== 137342+0+0 (secure 0 0 0) 0x7f5b58009d10 con 0x7f5b6c1091c0 2024-03-24T00:14:18.325 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:18.322+0000 7f5b69600700 1 --2- 172.21.15.118:0/1289454349 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7f5b540a29a0 0x7f5b540a4e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:14:18.326 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:18.323+0000 7f5b69600700 1 -- 172.21.15.118:0/1289454349 <== mon.1 v2:172.21.15.193:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 8248+0+0 (secure 0 0 0) 0x7f5b580cf8a0 con 0x7f5b6c1091c0 2024-03-24T00:14:18.326 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:18.323+0000 7f5b70e00700 1 --2- 172.21.15.118:0/1289454349 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7f5b540a29a0 0x7f5b540a4e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:14:18.326 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:18.324+0000 7f5b70e00700 1 --2- 172.21.15.118:0/1289454349 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7f5b540a29a0 0x7f5b540a4e60 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7f5b60009be0 tx=0x7f5b60009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:14:18.326 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:18.324+0000 7f5b69600700 1 -- 172.21.15.118:0/1289454349 <== mon.1 v2:172.21.15.193:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f5b58013030 con 0x7f5b6c1091c0 2024-03-24T00:14:18.565 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:18.565+0000 7f5b73800700 1 -- 172.21.15.118:0/1289454349 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 1} v 0) v1 -- 0x7f5b6c110db0 con 0x7f5b6c1091c0 2024-03-24T00:14:18.565 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:18.565+0000 7f5b69600700 1 -- 172.21.15.118:0/1289454349 <== mon.1 v2:172.21.15.193:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 1}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f5b5809c920 con 0x7f5b6c1091c0 2024-03-24T00:14:18.566 INFO:teuthology.orchestra.run.smithi118.stdout:55834574933 2024-03-24T00:14:18.568 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:18.568+0000 7f5b73800700 1 -- 172.21.15.118:0/1289454349 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7f5b540a29a0 msgr2=0x7f5b540a4e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:14:18.568 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:18.568+0000 7f5b73800700 1 --2- 172.21.15.118:0/1289454349 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7f5b540a29a0 0x7f5b540a4e60 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7f5b60009be0 tx=0x7f5b60009260 comp rx=0 tx=0).stop 2024-03-24T00:14:18.568 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:18.568+0000 7f5b73800700 1 -- 172.21.15.118:0/1289454349 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f5b6c1091c0 msgr2=0x7f5b6c19f550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:14:18.568 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:18.568+0000 7f5b73800700 1 --2- 172.21.15.118:0/1289454349 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f5b6c1091c0 0x7f5b6c19f550 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7f5b5800b2e0 tx=0x7f5b5800b840 comp rx=0 tx=0).stop 2024-03-24T00:14:18.569 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:18.569+0000 7f5b73800700 1 -- 172.21.15.118:0/1289454349 shutdown_connections 2024-03-24T00:14:18.569 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:18.569+0000 7f5b73800700 1 --2- 172.21.15.118:0/1289454349 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5b6c10e800 0x7f5b6c1a38e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:14:18.569 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:18.569+0000 7f5b73800700 1 --2- 172.21.15.118:0/1289454349 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7f5b540a29a0 0x7f5b540a4e60 unknown :-1 s=CLOSED pgs=198 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:14:18.569 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:18.569+0000 7f5b73800700 1 --2- 172.21.15.118:0/1289454349 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f5b6c108800 0x7f5b6c19f010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:14:18.569 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:18.569+0000 7f5b73800700 1 --2- 172.21.15.118:0/1289454349 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f5b6c1091c0 0x7f5b6c19f550 unknown :-1 s=CLOSED pgs=114 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:14:18.569 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:18.569+0000 7f5b73800700 1 -- 172.21.15.118:0/1289454349 >> 172.21.15.118:0/1289454349 conn(0x7f5b6c0767e0 msgr2=0x7f5b6c076e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:14:18.569 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:18.569+0000 7f5b73800700 1 -- 172.21.15.118:0/1289454349 shutdown_connections 2024-03-24T00:14:18.569 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:18.569+0000 7f5b73800700 1 -- 172.21.15.118:0/1289454349 wait complete. 2024-03-24T00:14:18.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:14:18 smithi118 ceph-mon[31535]: pgmap v354: 97 pgs: 97 active+clean; 644 KiB data, 344 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 9 op/s; 6.0 KiB/s, 1 objects/s recovering 2024-03-24T00:14:18.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:14:18 smithi118 ceph-mon[31535]: from='client.? 172.21.15.118:0/1289454349' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2024-03-24T00:14:19.203 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:14:18 smithi189 ceph-mon[36187]: pgmap v354: 97 pgs: 97 active+clean; 644 KiB data, 344 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 9 op/s; 6.0 KiB/s, 1 objects/s recovering 2024-03-24T00:14:19.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:14:18 smithi189 ceph-mon[36187]: from='client.? 172.21.15.118:0/1289454349' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2024-03-24T00:14:19.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:14:18 smithi193 ceph-mon[36878]: pgmap v354: 97 pgs: 97 active+clean; 644 KiB data, 344 MiB used, 1.0 TiB / 1.0 TiB avail; 4.5 KiB/s rd, 9 op/s; 6.0 KiB/s, 1 objects/s recovering 2024-03-24T00:14:19.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:14:18 smithi193 ceph-mon[36878]: from='client.? 172.21.15.118:0/1289454349' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2024-03-24T00:14:20.780 INFO:tasks.cephadm.ceph_manager.ceph:need seq 313532612636 got 313532612640 for osd.10 2024-03-24T00:14:20.780 DEBUG:teuthology.parallel:result is None 2024-03-24T00:14:21.012 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:14:20 smithi118 ceph-mon[31535]: pgmap v355: 97 pgs: 97 active+clean; 644 KiB data, 344 MiB used, 1.0 TiB / 1.0 TiB avail; 5.5 KiB/s rd, 10 op/s; 5.1 KiB/s, 1 objects/s recovering 2024-03-24T00:14:21.012 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:14:20 smithi193 ceph-mon[36878]: pgmap v355: 97 pgs: 97 active+clean; 644 KiB data, 344 MiB used, 1.0 TiB / 1.0 TiB avail; 5.5 KiB/s rd, 10 op/s; 5.1 KiB/s, 1 objects/s recovering 2024-03-24T00:14:21.203 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:14:20 smithi189 ceph-mon[36187]: pgmap v355: 97 pgs: 97 active+clean; 644 KiB data, 344 MiB used, 1.0 TiB / 1.0 TiB avail; 5.5 KiB/s rd, 10 op/s; 5.1 KiB/s, 1 objects/s recovering 2024-03-24T00:14:22.402 INFO:tasks.cephadm.ceph_manager.ceph:need seq 292057776162 got 292057776166 for osd.9 2024-03-24T00:14:22.402 DEBUG:teuthology.parallel:result is None 2024-03-24T00:14:23.064 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:14:22 smithi118 ceph-mon[31535]: pgmap v356: 97 pgs: 97 active+clean; 644 KiB data, 344 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-24T00:14:23.203 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:14:22 smithi189 ceph-mon[36187]: pgmap v356: 97 pgs: 97 active+clean; 644 KiB data, 344 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-24T00:14:23.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:14:22 smithi193 ceph-mon[36878]: pgmap v356: 97 pgs: 97 active+clean; 644 KiB data, 344 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-24T00:14:23.375 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:22.998+0000 7f7079200700 1 -- 172.21.15.118:0/1775253484 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f70741074b0 msgr2=0x7f707410f080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:14:23.375 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:22.998+0000 7f7079200700 1 --2- 172.21.15.118:0/1775253484 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f70741074b0 0x7f707410f080 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f706c00b0a0 tx=0x7f706c00b3b0 comp rx=0 tx=0).stop 2024-03-24T00:14:23.375 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:22.998+0000 7f7079200700 1 -- 172.21.15.118:0/1775253484 shutdown_connections 2024-03-24T00:14:23.375 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:22.998+0000 7f7079200700 1 --2- 172.21.15.118:0/1775253484 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f70741074b0 0x7f707410f080 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:14:23.375 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:22.998+0000 7f7079200700 1 --2- 172.21.15.118:0/1775253484 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f70741023d0 0x7f70741027b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:14:23.375 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:22.998+0000 7f7079200700 1 --2- 172.21.15.118:0/1775253484 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f7074102d80 0x7f7074106dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:14:23.375 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:22.998+0000 7f7079200700 1 -- 172.21.15.118:0/1775253484 >> 172.21.15.118:0/1775253484 conn(0x7f70740fdc40 msgr2=0x7f7074100060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:14:23.375 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:22.998+0000 7f7079200700 1 -- 172.21.15.118:0/1775253484 shutdown_connections 2024-03-24T00:14:23.376 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:22.998+0000 7f7079200700 1 -- 172.21.15.118:0/1775253484 wait complete. 2024-03-24T00:14:23.376 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:22.999+0000 7f7079200700 1 Processor -- start 2024-03-24T00:14:23.376 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:23.000+0000 7f7079200700 1 -- start start 2024-03-24T00:14:23.376 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:23.000+0000 7f7079200700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f70741023d0 0x7f70741a07b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:14:23.376 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:23.000+0000 7f7079200700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7074102d80 0x7f70741a0cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:14:23.376 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:23.000+0000 7f7079200700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f70741074b0 0x7f707419a970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:14:23.376 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:23.000+0000 7f7079200700 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7074112510 con 0x7f70741074b0 2024-03-24T00:14:23.376 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:23.000+0000 7f7079200700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7074112390 con 0x7f7074102d80 2024-03-24T00:14:23.376 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:23.000+0000 7f7079200700 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7074112690 con 0x7f70741023d0 2024-03-24T00:14:23.376 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:23.000+0000 7f7071e00700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f70741023d0 0x7f70741a07b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:14:23.376 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:23.000+0000 7f7071e00700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f70741023d0 0x7f70741a07b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.118:52866/0 (socket says 172.21.15.118:52866) 2024-03-24T00:14:23.376 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:23.000+0000 7f7071e00700 1 -- 172.21.15.118:0/524983775 learned_addr learned my addr 172.21.15.118:0/524983775 (peer_addr_for_me v2:172.21.15.118:0/0) 2024-03-24T00:14:23.376 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:23.000+0000 7f7072800700 1 --2- 172.21.15.118:0/524983775 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f70741074b0 0x7f707419a970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:14:23.377 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:23.000+0000 7f7071400700 1 --2- 172.21.15.118:0/524983775 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7074102d80 0x7f70741a0cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:14:23.377 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:23.001+0000 7f7071e00700 1 -- 172.21.15.118:0/524983775 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7074102d80 msgr2=0x7f70741a0cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:14:23.377 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:23.001+0000 7f7071e00700 1 --2- 172.21.15.118:0/524983775 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7074102d80 0x7f70741a0cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:14:23.377 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:23.001+0000 7f7071e00700 1 -- 172.21.15.118:0/524983775 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f70741074b0 msgr2=0x7f707419a970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:14:23.377 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:23.001+0000 7f7071e00700 1 --2- 172.21.15.118:0/524983775 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f70741074b0 0x7f707419a970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:14:23.377 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:23.001+0000 7f7071e00700 1 -- 172.21.15.118:0/524983775 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f707419b0d0 con 0x7f70741023d0 2024-03-24T00:14:23.377 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:23.001+0000 7f7071e00700 1 --2- 172.21.15.118:0/524983775 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f70741023d0 0x7f70741a07b0 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f706400e9c0 tx=0x7f706400ecd0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:14:23.377 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:23.001+0000 7f706aa00700 1 -- 172.21.15.118:0/524983775 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f706400cbd0 con 0x7f70741023d0 2024-03-24T00:14:23.377 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:23.001+0000 7f706aa00700 1 -- 172.21.15.118:0/524983775 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f706400ee60 con 0x7f70741023d0 2024-03-24T00:14:23.377 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:23.001+0000 7f706aa00700 1 -- 172.21.15.118:0/524983775 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7064005630 con 0x7f70741023d0 2024-03-24T00:14:23.378 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:23.001+0000 7f7079200700 1 -- 172.21.15.118:0/524983775 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f707419b3c0 con 0x7f70741023d0 2024-03-24T00:14:23.378 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:23.002+0000 7f7079200700 1 -- 172.21.15.118:0/524983775 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f70741a7680 con 0x7f70741023d0 2024-03-24T00:14:23.378 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:23.003+0000 7f7079200700 1 -- 172.21.15.118:0/524983775 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7074068e50 con 0x7f70741023d0 2024-03-24T00:14:23.378 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:23.004+0000 7f706aa00700 1 -- 172.21.15.118:0/524983775 <== mon.1 v2:172.21.15.193:3300/0 4 ==== mgrmap(e 17) v1 ==== 137342+0+0 (secure 0 0 0) 0x7f7064044940 con 0x7f70741023d0 2024-03-24T00:14:23.378 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:23.006+0000 7f706aa00700 1 --2- 172.21.15.118:0/524983775 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7f70580a29a0 0x7f70580a4e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:14:23.378 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:23.006+0000 7f706aa00700 1 -- 172.21.15.118:0/524983775 <== mon.1 v2:172.21.15.193:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 8248+0+0 (secure 0 0 0) 0x7f70640cee90 con 0x7f70741023d0 2024-03-24T00:14:23.378 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:23.006+0000 7f7071400700 1 --2- 172.21.15.118:0/524983775 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7f70580a29a0 0x7f70580a4e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:14:23.378 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:23.007+0000 7f7071400700 1 --2- 172.21.15.118:0/524983775 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7f70580a29a0 0x7f70580a4e60 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7f7074100080 tx=0x7f7060009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:14:23.378 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:23.008+0000 7f706aa00700 1 -- 172.21.15.118:0/524983775 <== mon.1 v2:172.21.15.193:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f706409bd60 con 0x7f70741023d0 2024-03-24T00:14:23.378 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:23.235+0000 7f7079200700 1 -- 172.21.15.118:0/524983775 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 2} v 0) v1 -- 0x7f707419bea0 con 0x7f70741023d0 2024-03-24T00:14:23.379 INFO:teuthology.orchestra.run.smithi118.stdout:77309411409 2024-03-24T00:14:23.379 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:23.235+0000 7f706aa00700 1 -- 172.21.15.118:0/524983775 <== mon.1 v2:172.21.15.193:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 2}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f706409b660 con 0x7f70741023d0 2024-03-24T00:14:23.379 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:23.238+0000 7f7079200700 1 -- 172.21.15.118:0/524983775 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7f70580a29a0 msgr2=0x7f70580a4e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:14:23.379 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:23.238+0000 7f7079200700 1 --2- 172.21.15.118:0/524983775 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7f70580a29a0 0x7f70580a4e60 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7f7074100080 tx=0x7f7060009260 comp rx=0 tx=0).stop 2024-03-24T00:14:23.379 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:23.238+0000 7f7079200700 1 -- 172.21.15.118:0/524983775 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f70741023d0 msgr2=0x7f70741a07b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:14:23.379 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:23.238+0000 7f7079200700 1 --2- 172.21.15.118:0/524983775 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f70741023d0 0x7f70741a07b0 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f706400e9c0 tx=0x7f706400ecd0 comp rx=0 tx=0).stop 2024-03-24T00:14:23.379 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:23.238+0000 7f7079200700 1 -- 172.21.15.118:0/524983775 shutdown_connections 2024-03-24T00:14:23.380 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:23.238+0000 7f7079200700 1 --2- 172.21.15.118:0/524983775 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7f70580a29a0 0x7f70580a4e60 unknown :-1 s=CLOSED pgs=199 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:14:23.380 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:23.238+0000 7f7079200700 1 --2- 172.21.15.118:0/524983775 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f70741074b0 0x7f707419a970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:14:23.380 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:23.238+0000 7f7079200700 1 --2- 172.21.15.118:0/524983775 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f70741023d0 0x7f70741a07b0 unknown :-1 s=CLOSED pgs=115 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:14:23.380 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:23.238+0000 7f7079200700 1 --2- 172.21.15.118:0/524983775 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7074102d80 0x7f70741a0cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:14:23.380 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:23.238+0000 7f7079200700 1 -- 172.21.15.118:0/524983775 >> 172.21.15.118:0/524983775 conn(0x7f70740fdc40 msgr2=0x7f707410d6b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:14:23.380 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:23.239+0000 7f7079200700 1 -- 172.21.15.118:0/524983775 shutdown_connections 2024-03-24T00:14:23.380 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:23.239+0000 7f7079200700 1 -- 172.21.15.118:0/524983775 wait complete. 2024-03-24T00:14:24.036 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:14:23 smithi118 ceph-mon[31535]: from='client.? 172.21.15.118:0/524983775' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2024-03-24T00:14:24.203 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:14:23 smithi189 ceph-mon[36187]: from='client.? 172.21.15.118:0/524983775' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2024-03-24T00:14:24.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:14:23 smithi193 ceph-mon[36878]: from='client.? 172.21.15.118:0/524983775' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2024-03-24T00:14:24.913 INFO:tasks.cephadm.ceph_manager.ceph:need seq 111669149763 got 111669149769 for osd.3 2024-03-24T00:14:24.913 DEBUG:teuthology.parallel:result is None 2024-03-24T00:14:25.054 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:14:24 smithi118 ceph-mon[31535]: pgmap v357: 97 pgs: 97 active+clean; 644 KiB data, 344 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-03-24T00:14:25.142 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:25.141+0000 7f5c39000700 1 -- 172.21.15.118:0/4172717569 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f5c3410e7a0 msgr2=0x7f5c3410eb80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:14:25.142 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:25.141+0000 7f5c39000700 1 --2- 172.21.15.118:0/4172717569 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f5c3410e7a0 0x7f5c3410eb80 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f5c24009990 tx=0x7f5c24009ca0 comp rx=0 tx=0).stop 2024-03-24T00:14:25.142 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:25.141+0000 7f5c39000700 1 -- 172.21.15.118:0/4172717569 shutdown_connections 2024-03-24T00:14:25.142 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:25.141+0000 7f5c39000700 1 --2- 172.21.15.118:0/4172717569 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f5c34109160 0x7f5c34115670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:14:25.142 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:25.141+0000 7f5c39000700 1 --2- 172.21.15.118:0/4172717569 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f5c3410e7a0 0x7f5c3410eb80 unknown :-1 s=CLOSED pgs=116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:14:25.142 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:25.141+0000 7f5c39000700 1 --2- 172.21.15.118:0/4172717569 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5c341087a0 0x7f5c34108c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:14:25.142 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:25.142+0000 7f5c39000700 1 -- 172.21.15.118:0/4172717569 >> 172.21.15.118:0/4172717569 conn(0x7f5c34075960 msgr2=0x7f5c34077d80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:14:25.142 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:25.142+0000 7f5c39000700 1 -- 172.21.15.118:0/4172717569 shutdown_connections 2024-03-24T00:14:25.142 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:25.142+0000 7f5c39000700 1 -- 172.21.15.118:0/4172717569 wait complete. 2024-03-24T00:14:25.144 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:25.143+0000 7f5c39000700 1 Processor -- start 2024-03-24T00:14:25.144 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:25.143+0000 7f5c39000700 1 -- start start 2024-03-24T00:14:25.144 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:25.143+0000 7f5c39000700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f5c341087a0 0x7f5c3419efb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:14:25.144 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:25.143+0000 7f5c39000700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f5c34109160 0x7f5c3419f4f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:14:25.144 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:25.143+0000 7f5c39000700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5c3410e7a0 0x7f5c341a3830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:14:25.144 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:25.143+0000 7f5c39000700 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5c34118b20 con 0x7f5c341087a0 2024-03-24T00:14:25.144 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:25.143+0000 7f5c39000700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5c341189a0 con 0x7f5c3410e7a0 2024-03-24T00:14:25.144 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:25.143+0000 7f5c39000700 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5c34118ca0 con 0x7f5c34109160 2024-03-24T00:14:25.144 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:25.144+0000 7f5c31400700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f5c34109160 0x7f5c3419f4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:14:25.144 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:25.144+0000 7f5c31e00700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f5c341087a0 0x7f5c3419efb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:14:25.144 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:25.144+0000 7f5c31e00700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f5c341087a0 0x7f5c3419efb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.118:3300/0 says I am v2:172.21.15.118:44454/0 (socket says 172.21.15.118:44454) 2024-03-24T00:14:25.145 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:25.144+0000 7f5c31400700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f5c34109160 0x7f5c3419f4f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.118:52888/0 (socket says 172.21.15.118:52888) 2024-03-24T00:14:25.145 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:25.144+0000 7f5c31e00700 1 -- 172.21.15.118:0/4002659418 learned_addr learned my addr 172.21.15.118:0/4002659418 (peer_addr_for_me v2:172.21.15.118:0/0) 2024-03-24T00:14:25.145 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:25.144+0000 7f5c32800700 1 --2- 172.21.15.118:0/4002659418 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5c3410e7a0 0x7f5c341a3830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:14:25.145 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:25.144+0000 7f5c31400700 1 -- 172.21.15.118:0/4002659418 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5c3410e7a0 msgr2=0x7f5c341a3830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:14:25.145 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:25.144+0000 7f5c31400700 1 --2- 172.21.15.118:0/4002659418 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5c3410e7a0 0x7f5c341a3830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:14:25.145 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:25.144+0000 7f5c31400700 1 -- 172.21.15.118:0/4002659418 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f5c341087a0 msgr2=0x7f5c3419efb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:14:25.145 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:25.144+0000 7f5c31400700 1 --2- 172.21.15.118:0/4002659418 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f5c341087a0 0x7f5c3419efb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:14:25.145 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:25.144+0000 7f5c31400700 1 -- 172.21.15.118:0/4002659418 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5c341a3f70 con 0x7f5c34109160 2024-03-24T00:14:25.145 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:25.144+0000 7f5c32800700 1 --2- 172.21.15.118:0/4002659418 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5c3410e7a0 0x7f5c341a3830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T00:14:25.145 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:25.144+0000 7f5c31400700 1 --2- 172.21.15.118:0/4002659418 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f5c34109160 0x7f5c3419f4f0 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f5c2000e9f0 tx=0x7f5c2000ed00 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:14:25.146 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:25.145+0000 7f5c2aa00700 1 -- 172.21.15.118:0/4002659418 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5c2000c9c0 con 0x7f5c34109160 2024-03-24T00:14:25.146 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:25.145+0000 7f5c2aa00700 1 -- 172.21.15.118:0/4002659418 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f5c2000cb60 con 0x7f5c34109160 2024-03-24T00:14:25.146 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:25.145+0000 7f5c2aa00700 1 -- 172.21.15.118:0/4002659418 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5c20018680 con 0x7f5c34109160 2024-03-24T00:14:25.146 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:25.145+0000 7f5c39000700 1 -- 172.21.15.118:0/4002659418 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5c341a4260 con 0x7f5c34109160 2024-03-24T00:14:25.146 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:25.145+0000 7f5c39000700 1 -- 172.21.15.118:0/4002659418 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5c341abae0 con 0x7f5c34109160 2024-03-24T00:14:25.146 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:25.146+0000 7f5c39000700 1 -- 172.21.15.118:0/4002659418 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5c34110d30 con 0x7f5c34109160 2024-03-24T00:14:25.147 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:25.148+0000 7f5c2aa00700 1 -- 172.21.15.118:0/4002659418 <== mon.1 v2:172.21.15.193:3300/0 4 ==== mgrmap(e 17) v1 ==== 137342+0+0 (secure 0 0 0) 0x7f5c20010640 con 0x7f5c34109160 2024-03-24T00:14:25.149 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:25.149+0000 7f5c2aa00700 1 --2- 172.21.15.118:0/4002659418 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7f5c180a29a0 0x7f5c180a4e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:14:25.149 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:25.150+0000 7f5c2aa00700 1 -- 172.21.15.118:0/4002659418 <== mon.1 v2:172.21.15.193:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 8248+0+0 (secure 0 0 0) 0x7f5c200cf130 con 0x7f5c34109160 2024-03-24T00:14:25.149 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:25.150+0000 7f5c31e00700 1 --2- 172.21.15.118:0/4002659418 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7f5c180a29a0 0x7f5c180a4e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:14:25.150 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:25.150+0000 7f5c31e00700 1 --2- 172.21.15.118:0/4002659418 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7f5c180a29a0 0x7f5c180a4e60 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7f5c24009750 tx=0x7f5c2400b280 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:14:25.151 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:25.151+0000 7f5c2aa00700 1 -- 172.21.15.118:0/4002659418 <== mon.1 v2:172.21.15.193:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f5c2009c000 con 0x7f5c34109160 2024-03-24T00:14:25.203 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:14:24 smithi189 ceph-mon[36187]: pgmap v357: 97 pgs: 97 active+clean; 644 KiB data, 344 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-03-24T00:14:25.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:14:24 smithi193 ceph-mon[36878]: pgmap v357: 97 pgs: 97 active+clean; 644 KiB data, 344 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-03-24T00:14:25.385 INFO:tasks.cephadm.ceph_manager.ceph:need seq 347892350998 got 347892351002 for osd.11 2024-03-24T00:14:25.385 DEBUG:teuthology.parallel:result is None 2024-03-24T00:14:25.452 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:25.452+0000 7f5c39000700 1 -- 172.21.15.118:0/4002659418 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 6} v 0) v1 -- 0x7f5c341a02c0 con 0x7f5c34109160 2024-03-24T00:14:25.453 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:25.453+0000 7f5c2aa00700 1 -- 172.21.15.118:0/4002659418 <== mon.1 v2:172.21.15.193:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 6}]=0 v0) v1 ==== 74+0+13 (secure 0 0 0) 0x7f5c2009b900 con 0x7f5c34109160 2024-03-24T00:14:25.453 INFO:teuthology.orchestra.run.smithi118.stdout:201863462971 2024-03-24T00:14:25.456 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:25.456+0000 7f5c39000700 1 -- 172.21.15.118:0/4002659418 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7f5c180a29a0 msgr2=0x7f5c180a4e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:14:25.456 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:25.456+0000 7f5c39000700 1 --2- 172.21.15.118:0/4002659418 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7f5c180a29a0 0x7f5c180a4e60 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7f5c24009750 tx=0x7f5c2400b280 comp rx=0 tx=0).stop 2024-03-24T00:14:25.456 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:25.456+0000 7f5c39000700 1 -- 172.21.15.118:0/4002659418 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f5c34109160 msgr2=0x7f5c3419f4f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:14:25.456 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:25.456+0000 7f5c39000700 1 --2- 172.21.15.118:0/4002659418 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f5c34109160 0x7f5c3419f4f0 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f5c2000e9f0 tx=0x7f5c2000ed00 comp rx=0 tx=0).stop 2024-03-24T00:14:25.456 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:25.456+0000 7f5c39000700 1 -- 172.21.15.118:0/4002659418 shutdown_connections 2024-03-24T00:14:25.456 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:25.456+0000 7f5c39000700 1 --2- 172.21.15.118:0/4002659418 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5c3410e7a0 0x7f5c341a3830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:14:25.456 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:25.456+0000 7f5c39000700 1 --2- 172.21.15.118:0/4002659418 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7f5c180a29a0 0x7f5c180a4e60 unknown :-1 s=CLOSED pgs=200 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:14:25.456 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:25.456+0000 7f5c39000700 1 --2- 172.21.15.118:0/4002659418 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f5c341087a0 0x7f5c3419efb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:14:25.456 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:25.456+0000 7f5c39000700 1 --2- 172.21.15.118:0/4002659418 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f5c34109160 0x7f5c3419f4f0 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:14:25.456 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:25.456+0000 7f5c39000700 1 -- 172.21.15.118:0/4002659418 >> 172.21.15.118:0/4002659418 conn(0x7f5c34075960 msgr2=0x7f5c34076150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:14:25.456 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:25.456+0000 7f5c39000700 1 -- 172.21.15.118:0/4002659418 shutdown_connections 2024-03-24T00:14:25.457 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:25.456+0000 7f5c39000700 1 -- 172.21.15.118:0/4002659418 wait complete. 2024-03-24T00:14:25.945 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:14:25 smithi118 ceph-mon[31535]: from='client.? 172.21.15.118:0/4002659418' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 6}]: dispatch 2024-03-24T00:14:26.203 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:14:25 smithi189 ceph-mon[36187]: from='client.? 172.21.15.118:0/4002659418' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 6}]: dispatch 2024-03-24T00:14:26.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:14:25 smithi193 ceph-mon[36878]: from='client.? 172.21.15.118:0/4002659418' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 6}]: dispatch 2024-03-24T00:14:26.315 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:25.732+0000 7f35cae00700 1 -- 172.21.15.118:0/3797056607 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f35c41081e0 msgr2=0x7f35c41085c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:14:26.315 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:25.732+0000 7f35cae00700 1 --2- 172.21.15.118:0/3797056607 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f35c41081e0 0x7f35c41085c0 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f35b40099c0 tx=0x7f35b4009cd0 comp rx=0 tx=0).stop 2024-03-24T00:14:26.315 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:25.732+0000 7f35cae00700 1 -- 172.21.15.118:0/3797056607 shutdown_connections 2024-03-24T00:14:26.315 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:25.732+0000 7f35cae00700 1 --2- 172.21.15.118:0/3797056607 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f35c406a7e0 0x7f35c410f020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:14:26.315 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:25.732+0000 7f35cae00700 1 --2- 172.21.15.118:0/3797056607 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f35c41081e0 0x7f35c41085c0 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:14:26.316 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:25.732+0000 7f35cae00700 1 --2- 172.21.15.118:0/3797056607 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f35c4069e40 0x7f35c406a2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:14:26.316 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:25.732+0000 7f35cae00700 1 -- 172.21.15.118:0/3797056607 >> 172.21.15.118:0/3797056607 conn(0x7f35c4077fc0 msgr2=0x7f35c40783d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:14:26.316 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:25.732+0000 7f35cae00700 1 -- 172.21.15.118:0/3797056607 shutdown_connections 2024-03-24T00:14:26.316 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:25.732+0000 7f35cae00700 1 -- 172.21.15.118:0/3797056607 wait complete. 2024-03-24T00:14:26.316 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:25.733+0000 7f35cae00700 1 Processor -- start 2024-03-24T00:14:26.316 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:25.733+0000 7f35cae00700 1 -- start start 2024-03-24T00:14:26.316 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:25.734+0000 7f35cae00700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f35c4069e40 0x7f35c419e6a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:14:26.316 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:25.734+0000 7f35cae00700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f35c406a7e0 0x7f35c419ebe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:14:26.316 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:25.734+0000 7f35cae00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f35c41081e0 0x7f35c4198770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:14:26.316 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:25.734+0000 7f35cae00700 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f35c4112470 con 0x7f35c406a7e0 2024-03-24T00:14:26.316 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:25.734+0000 7f35cae00700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f35c41122f0 con 0x7f35c41081e0 2024-03-24T00:14:26.316 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:25.734+0000 7f35cae00700 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f35c41125f0 con 0x7f35c4069e40 2024-03-24T00:14:26.316 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:25.734+0000 7f35c3e00700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f35c4069e40 0x7f35c419e6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:14:26.317 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:25.734+0000 7f35c3400700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f35c406a7e0 0x7f35c419ebe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:14:26.317 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:25.734+0000 7f35c3e00700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f35c4069e40 0x7f35c419e6a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.118:52900/0 (socket says 172.21.15.118:52900) 2024-03-24T00:14:26.317 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:25.734+0000 7f35c3e00700 1 -- 172.21.15.118:0/1793856462 learned_addr learned my addr 172.21.15.118:0/1793856462 (peer_addr_for_me v2:172.21.15.118:0/0) 2024-03-24T00:14:26.317 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:25.734+0000 7f35c8e00700 1 --2- 172.21.15.118:0/1793856462 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f35c41081e0 0x7f35c4198770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:14:26.317 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:25.734+0000 7f35c3e00700 1 -- 172.21.15.118:0/1793856462 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f35c41081e0 msgr2=0x7f35c4198770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:14:26.317 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:25.734+0000 7f35c3e00700 1 --2- 172.21.15.118:0/1793856462 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f35c41081e0 0x7f35c4198770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:14:26.317 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:25.734+0000 7f35c3e00700 1 -- 172.21.15.118:0/1793856462 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f35c406a7e0 msgr2=0x7f35c419ebe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:14:26.317 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:25.734+0000 7f35c3e00700 1 --2- 172.21.15.118:0/1793856462 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f35c406a7e0 0x7f35c419ebe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:14:26.317 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:25.734+0000 7f35c3e00700 1 -- 172.21.15.118:0/1793856462 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f35c4199050 con 0x7f35c4069e40 2024-03-24T00:14:26.317 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:25.735+0000 7f35c8e00700 1 --2- 172.21.15.118:0/1793856462 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f35c41081e0 0x7f35c4198770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T00:14:26.317 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:25.735+0000 7f35c3e00700 1 --2- 172.21.15.118:0/1793856462 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f35c4069e40 0x7f35c419e6a0 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f35b4009990 tx=0x7f35b4009f80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:14:26.317 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:25.735+0000 7f35c0c00700 1 -- 172.21.15.118:0/1793856462 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f35b4027070 con 0x7f35c4069e40 2024-03-24T00:14:26.317 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:25.735+0000 7f35c0c00700 1 -- 172.21.15.118:0/1793856462 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f35b4003e40 con 0x7f35c4069e40 2024-03-24T00:14:26.317 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:25.735+0000 7f35c0c00700 1 -- 172.21.15.118:0/1793856462 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f35b4022bf0 con 0x7f35c4069e40 2024-03-24T00:14:26.317 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:25.735+0000 7f35cae00700 1 -- 172.21.15.118:0/1793856462 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f35c41992e0 con 0x7f35c4069e40 2024-03-24T00:14:26.318 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:25.735+0000 7f35cae00700 1 -- 172.21.15.118:0/1793856462 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f35c41a5450 con 0x7f35c4069e40 2024-03-24T00:14:26.318 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:25.736+0000 7f35cae00700 1 -- 172.21.15.118:0/1793856462 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f35c4103890 con 0x7f35c4069e40 2024-03-24T00:14:26.318 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:25.738+0000 7f35c0c00700 1 -- 172.21.15.118:0/1793856462 <== mon.1 v2:172.21.15.193:3300/0 4 ==== mgrmap(e 17) v1 ==== 137342+0+0 (secure 0 0 0) 0x7f35b402a070 con 0x7f35c4069e40 2024-03-24T00:14:26.318 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:25.740+0000 7f35c0c00700 1 --2- 172.21.15.118:0/1793856462 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7f35a80a29a0 0x7f35a80a4e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:14:26.318 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:25.740+0000 7f35c0c00700 1 -- 172.21.15.118:0/1793856462 <== mon.1 v2:172.21.15.193:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 8248+0+0 (secure 0 0 0) 0x7f35b40cff60 con 0x7f35c4069e40 2024-03-24T00:14:26.318 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:25.740+0000 7f35c3400700 1 --2- 172.21.15.118:0/1793856462 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7f35a80a29a0 0x7f35a80a4e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:14:26.318 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:25.740+0000 7f35c3400700 1 --2- 172.21.15.118:0/1793856462 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7f35a80a29a0 0x7f35a80a4e60 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7f35c4199f30 tx=0x7f35b00073d0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:14:26.318 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:25.742+0000 7f35c0c00700 1 -- 172.21.15.118:0/1793856462 <== mon.1 v2:172.21.15.193:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f35b4031070 con 0x7f35c4069e40 2024-03-24T00:14:26.318 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:25.997+0000 7f35cae00700 1 -- 172.21.15.118:0/1793856462 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 7} v 0) v1 -- 0x7f35c402da30 con 0x7f35c4069e40 2024-03-24T00:14:26.318 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:25.998+0000 7f35c0c00700 1 -- 172.21.15.118:0/1793856462 <== mon.1 v2:172.21.15.193:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 7}]=0 v0) v1 ==== 74+0+13 (secure 0 0 0) 0x7f35b4013070 con 0x7f35c4069e40 2024-03-24T00:14:26.319 INFO:teuthology.orchestra.run.smithi118.stdout:223338299445 2024-03-24T00:14:26.319 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:26.002+0000 7f35cae00700 1 -- 172.21.15.118:0/1793856462 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7f35a80a29a0 msgr2=0x7f35a80a4e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:14:26.319 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:26.002+0000 7f35cae00700 1 --2- 172.21.15.118:0/1793856462 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7f35a80a29a0 0x7f35a80a4e60 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7f35c4199f30 tx=0x7f35b00073d0 comp rx=0 tx=0).stop 2024-03-24T00:14:26.319 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:26.002+0000 7f35cae00700 1 -- 172.21.15.118:0/1793856462 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f35c4069e40 msgr2=0x7f35c419e6a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:14:26.319 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:26.002+0000 7f35cae00700 1 --2- 172.21.15.118:0/1793856462 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f35c4069e40 0x7f35c419e6a0 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f35b4009990 tx=0x7f35b4009f80 comp rx=0 tx=0).stop 2024-03-24T00:14:26.319 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:26.002+0000 7f35cae00700 1 -- 172.21.15.118:0/1793856462 shutdown_connections 2024-03-24T00:14:26.319 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:26.002+0000 7f35cae00700 1 --2- 172.21.15.118:0/1793856462 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f35c41081e0 0x7f35c4198770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:14:26.319 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:26.002+0000 7f35cae00700 1 --2- 172.21.15.118:0/1793856462 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7f35a80a29a0 0x7f35a80a4e60 unknown :-1 s=CLOSED pgs=201 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:14:26.320 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:26.003+0000 7f35cae00700 1 --2- 172.21.15.118:0/1793856462 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f35c4069e40 0x7f35c419e6a0 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:14:26.320 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:26.003+0000 7f35cae00700 1 --2- 172.21.15.118:0/1793856462 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f35c406a7e0 0x7f35c419ebe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:14:26.320 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:26.003+0000 7f35cae00700 1 -- 172.21.15.118:0/1793856462 >> 172.21.15.118:0/1793856462 conn(0x7f35c4077fc0 msgr2=0x7f35c40ffc90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:14:26.320 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:26.003+0000 7f35cae00700 1 -- 172.21.15.118:0/1793856462 shutdown_connections 2024-03-24T00:14:26.320 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:26.003+0000 7f35cae00700 1 -- 172.21.15.118:0/1793856462 wait complete. 2024-03-24T00:14:26.936 INFO:tasks.cephadm.ceph_manager.ceph:need seq 55834574926 got 55834574933 for osd.1 2024-03-24T00:14:26.936 DEBUG:teuthology.parallel:result is None 2024-03-24T00:14:27.104 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:14:26 smithi189 ceph-mon[36187]: pgmap v358: 97 pgs: 97 active+clean; 644 KiB data, 344 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 10 op/s 2024-03-24T00:14:27.105 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:14:26 smithi189 ceph-mon[36187]: from='client.? 172.21.15.118:0/1793856462' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 7}]: dispatch 2024-03-24T00:14:27.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:14:26 smithi118 ceph-mon[31535]: pgmap v358: 97 pgs: 97 active+clean; 644 KiB data, 344 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 10 op/s 2024-03-24T00:14:27.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:14:26 smithi118 ceph-mon[31535]: from='client.? 172.21.15.118:0/1793856462' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 7}]: dispatch 2024-03-24T00:14:27.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:14:26 smithi193 ceph-mon[36878]: pgmap v358: 97 pgs: 97 active+clean; 644 KiB data, 344 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 10 op/s 2024-03-24T00:14:27.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:14:26 smithi193 ceph-mon[36878]: from='client.? 172.21.15.118:0/1793856462' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 7}]: dispatch 2024-03-24T00:14:27.287 INFO:tasks.cephadm.ceph_manager.ceph:need seq 223338299439 got 223338299445 for osd.7 2024-03-24T00:14:27.287 DEBUG:teuthology.parallel:result is None 2024-03-24T00:14:27.921 INFO:tasks.cephadm.ceph_manager.ceph:need seq 201863462963 got 201863462971 for osd.6 2024-03-24T00:14:27.921 DEBUG:teuthology.parallel:result is None 2024-03-24T00:14:27.995 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:27.938+0000 7fbe6a400700 1 -- 172.21.15.118:0/3175395147 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fbe64102df0 msgr2=0x7fbe64106e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:14:27.995 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:27.938+0000 7fbe6a400700 1 --2- 172.21.15.118:0/3175395147 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fbe64102df0 0x7fbe64106e30 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7fbe500099c0 tx=0x7fbe50009cd0 comp rx=0 tx=0).stop 2024-03-24T00:14:27.995 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:27.938+0000 7fbe6a400700 1 -- 172.21.15.118:0/3175395147 shutdown_connections 2024-03-24T00:14:27.995 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:27.938+0000 7fbe6a400700 1 --2- 172.21.15.118:0/3175395147 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fbe64107510 0x7fbe6410f0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:14:27.995 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:27.938+0000 7fbe6a400700 1 --2- 172.21.15.118:0/3175395147 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fbe64102440 0x7fbe64102820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:14:27.996 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:27.938+0000 7fbe6a400700 1 --2- 172.21.15.118:0/3175395147 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fbe64102df0 0x7fbe64106e30 unknown :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:14:27.996 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:27.938+0000 7fbe6a400700 1 -- 172.21.15.118:0/3175395147 >> 172.21.15.118:0/3175395147 conn(0x7fbe640792f0 msgr2=0x7fbe64079700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:14:27.996 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:27.938+0000 7fbe6a400700 1 -- 172.21.15.118:0/3175395147 shutdown_connections 2024-03-24T00:14:27.996 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:27.939+0000 7fbe6a400700 1 -- 172.21.15.118:0/3175395147 wait complete. 2024-03-24T00:14:27.996 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:27.940+0000 7fbe6a400700 1 Processor -- start 2024-03-24T00:14:27.996 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:27.940+0000 7fbe6a400700 1 -- start start 2024-03-24T00:14:27.996 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:27.941+0000 7fbe6a400700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fbe64102440 0x7fbe6419e6a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:14:27.996 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:27.941+0000 7fbe6a400700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fbe64102df0 0x7fbe6419ebe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:14:27.996 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:27.941+0000 7fbe6a400700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fbe64107510 0x7fbe64198820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:14:27.996 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:27.941+0000 7fbe6a400700 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbe64112520 con 0x7fbe64107510 2024-03-24T00:14:27.996 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:27.941+0000 7fbe6a400700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbe641123a0 con 0x7fbe64102df0 2024-03-24T00:14:27.996 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:27.941+0000 7fbe6a400700 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbe641126a0 con 0x7fbe64102440 2024-03-24T00:14:27.996 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:27.941+0000 7fbe63400700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fbe64102440 0x7fbe6419e6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:14:27.996 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:27.941+0000 7fbe62a00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fbe64102df0 0x7fbe6419ebe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:14:27.997 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:27.941+0000 7fbe63400700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fbe64102440 0x7fbe6419e6a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.118:52920/0 (socket says 172.21.15.118:52920) 2024-03-24T00:14:27.997 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:27.941+0000 7fbe63400700 1 -- 172.21.15.118:0/1313933499 learned_addr learned my addr 172.21.15.118:0/1313933499 (peer_addr_for_me v2:172.21.15.118:0/0) 2024-03-24T00:14:27.997 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:27.941+0000 7fbe63e00700 1 --2- 172.21.15.118:0/1313933499 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fbe64107510 0x7fbe64198820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:14:27.997 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:27.941+0000 7fbe62a00700 1 -- 172.21.15.118:0/1313933499 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fbe64102440 msgr2=0x7fbe6419e6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:14:27.997 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:27.941+0000 7fbe62a00700 1 --2- 172.21.15.118:0/1313933499 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fbe64102440 0x7fbe6419e6a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:14:27.997 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:27.941+0000 7fbe62a00700 1 -- 172.21.15.118:0/1313933499 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fbe64107510 msgr2=0x7fbe64198820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:14:27.997 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:27.941+0000 7fbe62a00700 1 --2- 172.21.15.118:0/1313933499 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fbe64107510 0x7fbe64198820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:14:27.997 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:27.941+0000 7fbe62a00700 1 -- 172.21.15.118:0/1313933499 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbe64199000 con 0x7fbe64102df0 2024-03-24T00:14:27.997 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:27.941+0000 7fbe63400700 1 --2- 172.21.15.118:0/1313933499 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fbe64102440 0x7fbe6419e6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T00:14:27.997 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:27.942+0000 7fbe62a00700 1 --2- 172.21.15.118:0/1313933499 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fbe64102df0 0x7fbe6419ebe0 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7fbe500098e0 tx=0x7fbe5000fa00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:14:27.997 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:27.942+0000 7fbe5be00700 1 -- 172.21.15.118:0/1313933499 <== mon.2 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbe50027070 con 0x7fbe64102df0 2024-03-24T00:14:27.997 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:27.942+0000 7fbe5be00700 1 -- 172.21.15.118:0/1313933499 <== mon.2 v2:172.21.15.189:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fbe50003680 con 0x7fbe64102df0 2024-03-24T00:14:27.997 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:27.942+0000 7fbe6a400700 1 -- 172.21.15.118:0/1313933499 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbe64199290 con 0x7fbe64102df0 2024-03-24T00:14:27.997 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:27.942+0000 7fbe5be00700 1 -- 172.21.15.118:0/1313933499 <== mon.2 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbe500173f0 con 0x7fbe64102df0 2024-03-24T00:14:27.997 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:27.942+0000 7fbe6a400700 1 -- 172.21.15.118:0/1313933499 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbe641a5500 con 0x7fbe64102df0 2024-03-24T00:14:27.998 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:27.944+0000 7fbe6a400700 1 -- 172.21.15.118:0/1313933499 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbe64103910 con 0x7fbe64102df0 2024-03-24T00:14:27.998 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:27.945+0000 7fbe5be00700 1 -- 172.21.15.118:0/1313933499 <== mon.2 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 17) v1 ==== 137342+0+0 (secure 0 0 0) 0x7fbe5002a070 con 0x7fbe64102df0 2024-03-24T00:14:27.998 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:27.947+0000 7fbe5be00700 1 --2- 172.21.15.118:0/1313933499 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7fbe480a2920 0x7fbe480a4de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:14:27.998 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:27.947+0000 7fbe5be00700 1 -- 172.21.15.118:0/1313933499 <== mon.2 v2:172.21.15.189:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 8248+0+0 (secure 0 0 0) 0x7fbe500cf820 con 0x7fbe64102df0 2024-03-24T00:14:27.998 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:27.947+0000 7fbe63400700 1 --2- 172.21.15.118:0/1313933499 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7fbe480a2920 0x7fbe480a4de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:14:27.998 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:27.947+0000 7fbe63400700 1 --2- 172.21.15.118:0/1313933499 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7fbe480a2920 0x7fbe480a4de0 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7fbe54005950 tx=0x7fbe5400b500 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:14:27.998 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:27.954+0000 7fbe5be00700 1 -- 172.21.15.118:0/1313933499 <== mon.2 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7fbe5005e740 con 0x7fbe64102df0 2024-03-24T00:14:28.187 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:14:27 smithi118 ceph-mon[31535]: pgmap v359: 97 pgs: 97 active+clean; 644 KiB data, 344 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-03-24T00:14:28.196 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:28.196+0000 7fbe6a400700 1 -- 172.21.15.118:0/1313933499 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 0} v 0) v1 -- 0x7fbe64199d50 con 0x7fbe64102df0 2024-03-24T00:14:28.197 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:28.197+0000 7fbe5be00700 1 -- 172.21.15.118:0/1313933499 <== mon.2 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 0}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7fbe50013070 con 0x7fbe64102df0 2024-03-24T00:14:28.197 INFO:teuthology.orchestra.run.smithi118.stdout:34359738461 2024-03-24T00:14:28.200 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:28.200+0000 7fbe6a400700 1 -- 172.21.15.118:0/1313933499 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7fbe480a2920 msgr2=0x7fbe480a4de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:14:28.200 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:28.200+0000 7fbe6a400700 1 --2- 172.21.15.118:0/1313933499 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7fbe480a2920 0x7fbe480a4de0 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7fbe54005950 tx=0x7fbe5400b500 comp rx=0 tx=0).stop 2024-03-24T00:14:28.200 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:28.200+0000 7fbe6a400700 1 -- 172.21.15.118:0/1313933499 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fbe64102df0 msgr2=0x7fbe6419ebe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:14:28.200 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:28.200+0000 7fbe6a400700 1 --2- 172.21.15.118:0/1313933499 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fbe64102df0 0x7fbe6419ebe0 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7fbe500098e0 tx=0x7fbe5000fa00 comp rx=0 tx=0).stop 2024-03-24T00:14:28.200 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:28.200+0000 7fbe6a400700 1 -- 172.21.15.118:0/1313933499 shutdown_connections 2024-03-24T00:14:28.200 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:28.200+0000 7fbe6a400700 1 --2- 172.21.15.118:0/1313933499 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7fbe480a2920 0x7fbe480a4de0 unknown :-1 s=CLOSED pgs=202 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:14:28.200 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:28.200+0000 7fbe6a400700 1 --2- 172.21.15.118:0/1313933499 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fbe64107510 0x7fbe64198820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:14:28.200 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:28.200+0000 7fbe6a400700 1 --2- 172.21.15.118:0/1313933499 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fbe64102440 0x7fbe6419e6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:14:28.200 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:28.200+0000 7fbe6a400700 1 --2- 172.21.15.118:0/1313933499 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fbe64102df0 0x7fbe6419ebe0 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:14:28.200 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:28.200+0000 7fbe6a400700 1 -- 172.21.15.118:0/1313933499 >> 172.21.15.118:0/1313933499 conn(0x7fbe640792f0 msgr2=0x7fbe64100230 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:14:28.200 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:28.200+0000 7fbe6a400700 1 -- 172.21.15.118:0/1313933499 shutdown_connections 2024-03-24T00:14:28.201 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:28.200+0000 7fbe6a400700 1 -- 172.21.15.118:0/1313933499 wait complete. 2024-03-24T00:14:28.203 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:14:27 smithi189 ceph-mon[36187]: pgmap v359: 97 pgs: 97 active+clean; 644 KiB data, 344 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-03-24T00:14:28.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:14:27 smithi193 ceph-mon[36878]: pgmap v359: 97 pgs: 97 active+clean; 644 KiB data, 344 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-03-24T00:14:29.071 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:14:28 smithi118 ceph-mon[31535]: from='client.? 172.21.15.118:0/1313933499' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2024-03-24T00:14:29.203 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:14:28 smithi189 ceph-mon[36187]: from='client.? 172.21.15.118:0/1313933499' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2024-03-24T00:14:29.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:14:28 smithi193 ceph-mon[36878]: from='client.? 172.21.15.118:0/1313933499' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2024-03-24T00:14:29.408 INFO:tasks.cephadm.ceph_manager.ceph:need seq 77309411404 got 77309411409 for osd.2 2024-03-24T00:14:29.408 DEBUG:teuthology.parallel:result is None 2024-03-24T00:14:29.490 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:28.803+0000 7f59a7800700 1 -- 172.21.15.118:0/1194963323 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f59a01023f0 msgr2=0x7f59a01027d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:14:29.490 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:28.803+0000 7f59a7800700 1 --2- 172.21.15.118:0/1194963323 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f59a01023f0 0x7f59a01027d0 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f59940099c0 tx=0x7f5994009cd0 comp rx=0 tx=0).stop 2024-03-24T00:14:29.490 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:28.803+0000 7f59a7800700 1 -- 172.21.15.118:0/1194963323 shutdown_connections 2024-03-24T00:14:29.490 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:28.803+0000 7f59a7800700 1 --2- 172.21.15.118:0/1194963323 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f59a01074d0 0x7f59a010f0a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:14:29.490 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:28.803+0000 7f59a7800700 1 --2- 172.21.15.118:0/1194963323 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f59a01023f0 0x7f59a01027d0 unknown :-1 s=CLOSED pgs=121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:14:29.490 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:28.803+0000 7f59a7800700 1 --2- 172.21.15.118:0/1194963323 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f59a0102da0 0x7f59a0106df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:14:29.490 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:28.803+0000 7f59a7800700 1 -- 172.21.15.118:0/1194963323 >> 172.21.15.118:0/1194963323 conn(0x7f59a00fdc60 msgr2=0x7f59a0100080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:14:29.490 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:28.803+0000 7f59a7800700 1 -- 172.21.15.118:0/1194963323 shutdown_connections 2024-03-24T00:14:29.491 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:28.803+0000 7f59a7800700 1 -- 172.21.15.118:0/1194963323 wait complete. 2024-03-24T00:14:29.491 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:28.804+0000 7f59a7800700 1 Processor -- start 2024-03-24T00:14:29.491 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:28.804+0000 7f59a7800700 1 -- start start 2024-03-24T00:14:29.491 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:28.804+0000 7f59a7800700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f59a01023f0 0x7f59a01a0790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:14:29.491 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:28.804+0000 7f59a7800700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f59a0102da0 0x7f59a01a0cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:14:29.491 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:28.804+0000 7f59a7800700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f59a01074d0 0x7f59a019a950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:14:29.491 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:28.804+0000 7f59a7800700 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f59a01124e0 con 0x7f59a0102da0 2024-03-24T00:14:29.491 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:28.804+0000 7f59a7800700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f59a0112360 con 0x7f59a01023f0 2024-03-24T00:14:29.491 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:28.804+0000 7f59a7800700 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f59a0112660 con 0x7f59a01074d0 2024-03-24T00:14:29.491 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:28.805+0000 7f599fe00700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f59a0102da0 0x7f59a01a0cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:14:29.491 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:28.805+0000 7f59a5800700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f59a01074d0 0x7f59a019a950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:14:29.491 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:28.805+0000 7f59a5800700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f59a01074d0 0x7f59a019a950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.118:52938/0 (socket says 172.21.15.118:52938) 2024-03-24T00:14:29.491 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:28.805+0000 7f59a5800700 1 -- 172.21.15.118:0/280481422 learned_addr learned my addr 172.21.15.118:0/280481422 (peer_addr_for_me v2:172.21.15.118:0/0) 2024-03-24T00:14:29.492 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:28.805+0000 7f59a4e00700 1 --2- 172.21.15.118:0/280481422 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f59a01023f0 0x7f59a01a0790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:14:29.492 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:28.805+0000 7f59a5800700 1 -- 172.21.15.118:0/280481422 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f59a01023f0 msgr2=0x7f59a01a0790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:14:29.492 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:28.805+0000 7f59a5800700 1 --2- 172.21.15.118:0/280481422 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f59a01023f0 0x7f59a01a0790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:14:29.492 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:28.805+0000 7f59a5800700 1 -- 172.21.15.118:0/280481422 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f59a0102da0 msgr2=0x7f59a01a0cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:14:29.492 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:28.805+0000 7f59a5800700 1 --2- 172.21.15.118:0/280481422 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f59a0102da0 0x7f59a01a0cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:14:29.492 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:28.805+0000 7f59a5800700 1 -- 172.21.15.118:0/280481422 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f59a019b0b0 con 0x7f59a01074d0 2024-03-24T00:14:29.492 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:28.805+0000 7f59a4e00700 1 --2- 172.21.15.118:0/280481422 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f59a01023f0 0x7f59a01a0790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T00:14:29.492 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:28.805+0000 7f59a5800700 1 --2- 172.21.15.118:0/280481422 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f59a01074d0 0x7f59a019a950 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f599800ef30 tx=0x7f599800c3a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:14:29.492 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:28.806+0000 7f599d600700 1 -- 172.21.15.118:0/280481422 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f59980092e0 con 0x7f59a01074d0 2024-03-24T00:14:29.492 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:28.806+0000 7f599d600700 1 -- 172.21.15.118:0/280481422 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f5998004bb0 con 0x7f59a01074d0 2024-03-24T00:14:29.492 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:28.806+0000 7f59a7800700 1 -- 172.21.15.118:0/280481422 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f59a019b340 con 0x7f59a01074d0 2024-03-24T00:14:29.492 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:28.806+0000 7f599d600700 1 -- 172.21.15.118:0/280481422 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5998004280 con 0x7f59a01074d0 2024-03-24T00:14:29.492 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:28.806+0000 7f59a7800700 1 -- 172.21.15.118:0/280481422 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f59a01a7660 con 0x7f59a01074d0 2024-03-24T00:14:29.492 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:28.807+0000 7f59a7800700 1 -- 172.21.15.118:0/280481422 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f59a0068e50 con 0x7f59a01074d0 2024-03-24T00:14:29.493 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:28.809+0000 7f599d600700 1 -- 172.21.15.118:0/280481422 <== mon.1 v2:172.21.15.193:3300/0 4 ==== mgrmap(e 17) v1 ==== 137342+0+0 (secure 0 0 0) 0x7f599801b030 con 0x7f59a01074d0 2024-03-24T00:14:29.493 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:28.810+0000 7f599d600700 1 --2- 172.21.15.118:0/280481422 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7f59880a29a0 0x7f59880a4e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:14:29.493 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:28.810+0000 7f599d600700 1 -- 172.21.15.118:0/280481422 <== mon.1 v2:172.21.15.193:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 8248+0+0 (secure 0 0 0) 0x7f59980ce670 con 0x7f59a01074d0 2024-03-24T00:14:29.493 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:28.810+0000 7f59a4e00700 1 --2- 172.21.15.118:0/280481422 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7f59880a29a0 0x7f59880a4e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:14:29.493 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:28.811+0000 7f59a4e00700 1 --2- 172.21.15.118:0/280481422 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7f59880a29a0 0x7f59880a4e60 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7f5994000c00 tx=0x7f5994005ab0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:14:29.493 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:28.814+0000 7f599d600700 1 -- 172.21.15.118:0/280481422 <== mon.1 v2:172.21.15.193:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f5998022020 con 0x7f59a01074d0 2024-03-24T00:14:29.493 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:29.071+0000 7f59a7800700 1 -- 172.21.15.118:0/280481422 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 4} v 0) v1 -- 0x7f59a01a7900 con 0x7f59a01074d0 2024-03-24T00:14:29.493 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:29.071+0000 7f599d600700 1 -- 172.21.15.118:0/280481422 <== mon.1 v2:172.21.15.193:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 4}]=0 v0) v1 ==== 74+0+13 (secure 0 0 0) 0x7f5998010040 con 0x7f59a01074d0 2024-03-24T00:14:29.493 INFO:teuthology.orchestra.run.smithi118.stdout:146028888134 2024-03-24T00:14:29.494 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:29.074+0000 7f59a7800700 1 -- 172.21.15.118:0/280481422 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7f59880a29a0 msgr2=0x7f59880a4e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:14:29.494 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:29.075+0000 7f59a7800700 1 --2- 172.21.15.118:0/280481422 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7f59880a29a0 0x7f59880a4e60 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7f5994000c00 tx=0x7f5994005ab0 comp rx=0 tx=0).stop 2024-03-24T00:14:29.494 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:29.075+0000 7f59a7800700 1 -- 172.21.15.118:0/280481422 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f59a01074d0 msgr2=0x7f59a019a950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:14:29.494 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:29.075+0000 7f59a7800700 1 --2- 172.21.15.118:0/280481422 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f59a01074d0 0x7f59a019a950 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f599800ef30 tx=0x7f599800c3a0 comp rx=0 tx=0).stop 2024-03-24T00:14:29.494 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:29.075+0000 7f59a7800700 1 -- 172.21.15.118:0/280481422 shutdown_connections 2024-03-24T00:14:29.494 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:29.075+0000 7f59a7800700 1 --2- 172.21.15.118:0/280481422 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f59a01023f0 0x7f59a01a0790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:14:29.494 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:29.075+0000 7f59a7800700 1 --2- 172.21.15.118:0/280481422 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7f59880a29a0 0x7f59880a4e60 unknown :-1 s=CLOSED pgs=203 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:14:29.494 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:29.075+0000 7f59a7800700 1 --2- 172.21.15.118:0/280481422 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f59a01074d0 0x7f59a019a950 unknown :-1 s=CLOSED pgs=122 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:14:29.494 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:29.075+0000 7f59a7800700 1 --2- 172.21.15.118:0/280481422 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f59a0102da0 0x7f59a01a0cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:14:29.494 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:29.075+0000 7f59a7800700 1 -- 172.21.15.118:0/280481422 >> 172.21.15.118:0/280481422 conn(0x7f59a00fdc60 msgr2=0x7f59a010e370 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:14:29.494 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:29.075+0000 7f59a7800700 1 -- 172.21.15.118:0/280481422 shutdown_connections 2024-03-24T00:14:29.494 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:29.075+0000 7f59a7800700 1 -- 172.21.15.118:0/280481422 wait complete. 2024-03-24T00:14:29.756 INFO:teuthology.orchestra.run.smithi118.stderr:Inferring config /var/lib/ceph/d79a5420-e971-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T00:14:30.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:14:29 smithi118 ceph-mon[31535]: from='client.? 172.21.15.118:0/280481422' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2024-03-24T00:14:30.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:14:29 smithi118 ceph-mon[31535]: pgmap v360: 97 pgs: 97 active+clean; 644 KiB data, 344 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 11 op/s 2024-03-24T00:14:30.203 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:14:29 smithi189 ceph-mon[36187]: from='client.? 172.21.15.118:0/280481422' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2024-03-24T00:14:30.203 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:14:29 smithi189 ceph-mon[36187]: pgmap v360: 97 pgs: 97 active+clean; 644 KiB data, 344 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 11 op/s 2024-03-24T00:14:30.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:14:29 smithi193 ceph-mon[36878]: from='client.? 172.21.15.118:0/280481422' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2024-03-24T00:14:30.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:14:29 smithi193 ceph-mon[36878]: pgmap v360: 97 pgs: 97 active+clean; 644 KiB data, 344 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 11 op/s 2024-03-24T00:14:30.387 INFO:tasks.cephadm.ceph_manager.ceph:need seq 34359738453 got 34359738461 for osd.0 2024-03-24T00:14:30.387 DEBUG:teuthology.parallel:result is None 2024-03-24T00:14:32.953 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:14:32 smithi189 ceph-mon[36187]: pgmap v361: 97 pgs: 97 active+clean; 644 KiB data, 344 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-24T00:14:32.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:14:32 smithi193 ceph-mon[36878]: pgmap v361: 97 pgs: 97 active+clean; 644 KiB data, 344 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-24T00:14:33.071 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:14:32 smithi118 ceph-mon[31535]: pgmap v361: 97 pgs: 97 active+clean; 644 KiB data, 344 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-24T00:14:33.141 INFO:tasks.cephadm.ceph_manager.ceph:need seq 146028888125 got 146028888134 for osd.4 2024-03-24T00:14:33.141 DEBUG:teuthology.parallel:result is None 2024-03-24T00:14:34.379 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:34.238+0000 7fc5d9c00700 1 -- 172.21.15.118:0/2887048077 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fc5d41096c0 msgr2=0x7fc5d4111290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:14:34.379 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:34.238+0000 7fc5d9c00700 1 --2- 172.21.15.118:0/2887048077 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fc5d41096c0 0x7fc5d4111290 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7fc5cc00b0d0 tx=0x7fc5cc00b3e0 comp rx=0 tx=0).stop 2024-03-24T00:14:34.380 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:34.238+0000 7fc5d9c00700 1 -- 172.21.15.118:0/2887048077 shutdown_connections 2024-03-24T00:14:34.380 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:34.238+0000 7fc5d9c00700 1 --2- 172.21.15.118:0/2887048077 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc5d41045e0 0x7fc5d41049c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:14:34.380 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:34.238+0000 7fc5d9c00700 1 --2- 172.21.15.118:0/2887048077 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fc5d41096c0 0x7fc5d4111290 unknown :-1 s=CLOSED pgs=123 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:14:34.380 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:34.238+0000 7fc5d9c00700 1 --2- 172.21.15.118:0/2887048077 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fc5d4104f90 0x7fc5d4108fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:14:34.380 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:34.238+0000 7fc5d9c00700 1 -- 172.21.15.118:0/2887048077 >> 172.21.15.118:0/2887048077 conn(0x7fc5d40ffe70 msgr2=0x7fc5d4102290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:14:34.380 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:34.238+0000 7fc5d9c00700 1 -- 172.21.15.118:0/2887048077 shutdown_connections 2024-03-24T00:14:34.380 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:34.238+0000 7fc5d9c00700 1 -- 172.21.15.118:0/2887048077 wait complete. 2024-03-24T00:14:34.380 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:34.240+0000 7fc5d9c00700 1 Processor -- start 2024-03-24T00:14:34.380 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:34.240+0000 7fc5d9c00700 1 -- start start 2024-03-24T00:14:34.380 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:34.240+0000 7fc5d9c00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc5d41045e0 0x7fc5d41a0720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:14:34.380 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:34.240+0000 7fc5d9c00700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fc5d4104f90 0x7fc5d41a0c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:14:34.380 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:34.240+0000 7fc5d9c00700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fc5d41096c0 0x7fc5d419a8e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:14:34.380 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:34.240+0000 7fc5d9c00700 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc5d4114700 con 0x7fc5d4104f90 2024-03-24T00:14:34.381 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:34.240+0000 7fc5d9c00700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc5d4114580 con 0x7fc5d41045e0 2024-03-24T00:14:34.381 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:34.240+0000 7fc5d9c00700 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc5d4114880 con 0x7fc5d41096c0 2024-03-24T00:14:34.381 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:34.240+0000 7fc5d2a00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc5d41045e0 0x7fc5d41a0720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:14:34.381 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:34.240+0000 7fc5d2a00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc5d41045e0 0x7fc5d41a0720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.118:49000/0 (socket says 172.21.15.118:49000) 2024-03-24T00:14:34.381 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:34.240+0000 7fc5d2a00700 1 -- 172.21.15.118:0/171919525 learned_addr learned my addr 172.21.15.118:0/171919525 (peer_addr_for_me v2:172.21.15.118:0/0) 2024-03-24T00:14:34.381 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:34.241+0000 7fc5cbe00700 1 --2- 172.21.15.118:0/171919525 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fc5d4104f90 0x7fc5d41a0c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:14:34.381 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:34.241+0000 7fc5d3400700 1 --2- 172.21.15.118:0/171919525 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fc5d41096c0 0x7fc5d419a8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:14:34.381 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:34.241+0000 7fc5d2a00700 1 -- 172.21.15.118:0/171919525 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fc5d41096c0 msgr2=0x7fc5d419a8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:14:34.381 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:34.241+0000 7fc5d2a00700 1 --2- 172.21.15.118:0/171919525 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fc5d41096c0 0x7fc5d419a8e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:14:34.381 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:34.241+0000 7fc5d2a00700 1 -- 172.21.15.118:0/171919525 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fc5d4104f90 msgr2=0x7fc5d41a0c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:14:34.381 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:34.241+0000 7fc5d2a00700 1 --2- 172.21.15.118:0/171919525 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fc5d4104f90 0x7fc5d41a0c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:14:34.381 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:34.241+0000 7fc5d2a00700 1 -- 172.21.15.118:0/171919525 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc5d419b040 con 0x7fc5d41045e0 2024-03-24T00:14:34.382 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:34.241+0000 7fc5d2a00700 1 --2- 172.21.15.118:0/171919525 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc5d41045e0 0x7fc5d41a0720 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7fc5c400e9a0 tx=0x7fc5c400ecb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:14:34.382 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:34.242+0000 7fc5cb400700 1 -- 172.21.15.118:0/171919525 <== mon.2 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc5c400ca10 con 0x7fc5d41045e0 2024-03-24T00:14:34.382 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:34.242+0000 7fc5cb400700 1 -- 172.21.15.118:0/171919525 <== mon.2 v2:172.21.15.189:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fc5c400cbb0 con 0x7fc5d41045e0 2024-03-24T00:14:34.382 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:34.242+0000 7fc5d9c00700 1 -- 172.21.15.118:0/171919525 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc5d419b330 con 0x7fc5d41045e0 2024-03-24T00:14:34.382 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:34.242+0000 7fc5cb400700 1 -- 172.21.15.118:0/171919525 <== mon.2 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc5c4018680 con 0x7fc5d41045e0 2024-03-24T00:14:34.382 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:34.242+0000 7fc5d9c00700 1 -- 172.21.15.118:0/171919525 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc5d41a75f0 con 0x7fc5d41045e0 2024-03-24T00:14:34.382 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:34.243+0000 7fc5d9c00700 1 -- 172.21.15.118:0/171919525 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc5d4068e50 con 0x7fc5d41045e0 2024-03-24T00:14:34.382 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:34.245+0000 7fc5cb400700 1 -- 172.21.15.118:0/171919525 <== mon.2 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 17) v1 ==== 137342+0+0 (secure 0 0 0) 0x7fc5c4010430 con 0x7fc5d41045e0 2024-03-24T00:14:34.382 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:34.247+0000 7fc5cb400700 1 --2- 172.21.15.118:0/171919525 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7fc5b80a29a0 0x7fc5b80a4e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:14:34.382 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:34.247+0000 7fc5cb400700 1 -- 172.21.15.118:0/171919525 <== mon.2 v2:172.21.15.189:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 8248+0+0 (secure 0 0 0) 0x7fc5c40cee10 con 0x7fc5d41045e0 2024-03-24T00:14:34.382 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:34.247+0000 7fc5cbe00700 1 --2- 172.21.15.118:0/171919525 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7fc5b80a29a0 0x7fc5b80a4e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:14:34.382 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:34.248+0000 7fc5cbe00700 1 --2- 172.21.15.118:0/171919525 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7fc5b80a29a0 0x7fc5b80a4e60 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7fc5d41034c0 tx=0x7fc5bc009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:14:34.382 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:34.250+0000 7fc5cb400700 1 -- 172.21.15.118:0/171919525 <== mon.2 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7fc5c40d2030 con 0x7fc5d41045e0 2024-03-24T00:14:34.552 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:34.552+0000 7fc5d9c00700 1 -- 172.21.15.118:0/171919525 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 5} v 0) v1 -- 0x7fc5d41a7820 con 0x7fc5d41045e0 2024-03-24T00:14:34.552 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:34.553+0000 7fc5cb400700 1 -- 172.21.15.118:0/171919525 <== mon.2 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 5}]=0 v0) v1 ==== 74+0+13 (secure 0 0 0) 0x7fc5c409bce0 con 0x7fc5d41045e0 2024-03-24T00:14:34.553 INFO:teuthology.orchestra.run.smithi118.stdout:167503724610 2024-03-24T00:14:34.556 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:34.556+0000 7fc5d9c00700 1 -- 172.21.15.118:0/171919525 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7fc5b80a29a0 msgr2=0x7fc5b80a4e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:14:34.556 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:34.556+0000 7fc5d9c00700 1 --2- 172.21.15.118:0/171919525 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7fc5b80a29a0 0x7fc5b80a4e60 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7fc5d41034c0 tx=0x7fc5bc009260 comp rx=0 tx=0).stop 2024-03-24T00:14:34.556 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:34.556+0000 7fc5d9c00700 1 -- 172.21.15.118:0/171919525 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc5d41045e0 msgr2=0x7fc5d41a0720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:14:34.557 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:34.556+0000 7fc5d9c00700 1 --2- 172.21.15.118:0/171919525 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc5d41045e0 0x7fc5d41a0720 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7fc5c400e9a0 tx=0x7fc5c400ecb0 comp rx=0 tx=0).stop 2024-03-24T00:14:34.557 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:34.556+0000 7fc5d9c00700 1 -- 172.21.15.118:0/171919525 shutdown_connections 2024-03-24T00:14:34.557 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:34.556+0000 7fc5d9c00700 1 --2- 172.21.15.118:0/171919525 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc5d41045e0 0x7fc5d41a0720 unknown :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:14:34.557 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:34.556+0000 7fc5d9c00700 1 --2- 172.21.15.118:0/171919525 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7fc5b80a29a0 0x7fc5b80a4e60 unknown :-1 s=CLOSED pgs=204 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:14:34.557 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:34.556+0000 7fc5d9c00700 1 --2- 172.21.15.118:0/171919525 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fc5d41096c0 0x7fc5d419a8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:14:34.557 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:34.556+0000 7fc5d9c00700 1 --2- 172.21.15.118:0/171919525 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fc5d4104f90 0x7fc5d41a0c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:14:34.557 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:34.556+0000 7fc5d9c00700 1 -- 172.21.15.118:0/171919525 >> 172.21.15.118:0/171919525 conn(0x7fc5d40ffe70 msgr2=0x7fc5d4100810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:14:34.557 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:34.557+0000 7fc5d9c00700 1 -- 172.21.15.118:0/171919525 shutdown_connections 2024-03-24T00:14:34.557 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:34.557+0000 7fc5d9c00700 1 -- 172.21.15.118:0/171919525 wait complete. 2024-03-24T00:14:34.953 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:14:34 smithi189 ceph-mon[36187]: pgmap v362: 97 pgs: 97 active+clean; 644 KiB data, 344 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-03-24T00:14:34.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:14:34 smithi189 ceph-mon[36187]: from='client.? 172.21.15.118:0/171919525' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2024-03-24T00:14:34.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:14:34 smithi193 ceph-mon[36878]: pgmap v362: 97 pgs: 97 active+clean; 644 KiB data, 344 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-03-24T00:14:34.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:14:34 smithi193 ceph-mon[36878]: from='client.? 172.21.15.118:0/171919525' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2024-03-24T00:14:35.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:14:34 smithi118 ceph-mon[31535]: pgmap v362: 97 pgs: 97 active+clean; 644 KiB data, 344 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-03-24T00:14:35.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:14:34 smithi118 ceph-mon[31535]: from='client.? 172.21.15.118:0/171919525' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2024-03-24T00:14:35.803 INFO:tasks.cephadm.ceph_manager.ceph:need seq 167503724605 got 167503724610 for osd.5 2024-03-24T00:14:35.803 DEBUG:teuthology.parallel:result is None 2024-03-24T00:14:35.803 INFO:tasks.cephadm.ceph_manager.ceph:waiting for clean 2024-03-24T00:14:35.803 DEBUG:teuthology.orchestra.run.smithi118:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 shell --fsid d79a5420-e971-11ee-95cd-87774f69a715 -- ceph pg dump --format=json 2024-03-24T00:14:36.257 INFO:teuthology.orchestra.run.smithi118.stderr:Inferring config /var/lib/ceph/d79a5420-e971-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T00:14:36.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:14:36 smithi193 ceph-mon[36878]: pgmap v363: 97 pgs: 97 active+clean; 644 KiB data, 344 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 10 op/s 2024-03-24T00:14:36.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:14:36 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T00:14:36.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:14:36 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T00:14:37.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:14:36 smithi118 ceph-mon[31535]: pgmap v363: 97 pgs: 97 active+clean; 644 KiB data, 344 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 10 op/s 2024-03-24T00:14:37.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:14:36 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T00:14:37.197 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:14:36 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T00:14:37.203 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:14:36 smithi189 ceph-mon[36187]: pgmap v363: 97 pgs: 97 active+clean; 644 KiB data, 344 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 10 op/s 2024-03-24T00:14:37.203 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:14:36 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T00:14:37.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:14:36 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T00:14:38.197 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:14:37 smithi118 ceph-mon[31535]: mgrmap e18: x(active, since 9m), standbys: z, y 2024-03-24T00:14:38.203 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:14:37 smithi189 ceph-mon[36187]: mgrmap e18: x(active, since 9m), standbys: z, y 2024-03-24T00:14:38.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:14:37 smithi193 ceph-mon[36878]: mgrmap e18: x(active, since 9m), standbys: z, y 2024-03-24T00:14:39.037 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:14:38 smithi118 ceph-mon[31535]: pgmap v364: 97 pgs: 97 active+clean; 644 KiB data, 344 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 85 B/s wr, 9 op/s 2024-03-24T00:14:39.038 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:14:38 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T00:14:39.038 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:14:38 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T00:14:39.038 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:14:38 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T00:14:39.203 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:14:38 smithi189 ceph-mon[36187]: pgmap v364: 97 pgs: 97 active+clean; 644 KiB data, 344 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 85 B/s wr, 9 op/s 2024-03-24T00:14:39.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:14:38 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T00:14:39.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:14:38 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T00:14:39.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:14:38 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T00:14:39.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:14:38 smithi193 ceph-mon[36878]: pgmap v364: 97 pgs: 97 active+clean; 644 KiB data, 344 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 85 B/s wr, 9 op/s 2024-03-24T00:14:39.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:14:38 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T00:14:39.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:14:38 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T00:14:39.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:14:38 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T00:14:39.583 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:39.582+0000 7f642f200700 1 -- 172.21.15.118:0/1252319895 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f64301022d0 msgr2=0x7f64301026b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:14:39.584 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:39.582+0000 7f642f200700 1 --2- 172.21.15.118:0/1252319895 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f64301022d0 0x7f64301026b0 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f6420009990 tx=0x7f6420009ca0 comp rx=0 tx=0).stop 2024-03-24T00:14:39.584 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:39.582+0000 7f642f200700 1 -- 172.21.15.118:0/1252319895 shutdown_connections 2024-03-24T00:14:39.584 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:39.582+0000 7f642f200700 1 --2- 172.21.15.118:0/1252319895 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f64301022d0 0x7f64301026b0 unknown :-1 s=CLOSED pgs=121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:14:39.584 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:39.582+0000 7f642f200700 1 --2- 172.21.15.118:0/1252319895 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f64301073b0 0x7f643010ef80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:14:39.584 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:39.582+0000 7f642f200700 1 --2- 172.21.15.118:0/1252319895 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f6430102c80 0x7f6430106cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:14:39.584 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:39.582+0000 7f642f200700 1 -- 172.21.15.118:0/1252319895 >> 172.21.15.118:0/1252319895 conn(0x7f64300fdb60 msgr2=0x7f64300fff80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:14:39.584 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:39.582+0000 7f642f200700 1 -- 172.21.15.118:0/1252319895 shutdown_connections 2024-03-24T00:14:39.585 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:39.582+0000 7f642f200700 1 -- 172.21.15.118:0/1252319895 wait complete. 2024-03-24T00:14:39.586 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:39.584+0000 7f642f200700 1 Processor -- start 2024-03-24T00:14:39.587 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:39.584+0000 7f642f200700 1 -- start start 2024-03-24T00:14:39.587 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:39.584+0000 7f642f200700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f64301022d0 0x7f6430074490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:14:39.587 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:39.585+0000 7f642f200700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f6430102c80 0x7f64300749d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:14:39.587 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:39.585+0000 7f642f200700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f64301073b0 0x7f6430074f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:14:39.587 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:39.585+0000 7f642f200700 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6430112390 con 0x7f64301022d0 2024-03-24T00:14:39.587 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:39.585+0000 7f642f200700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6430112210 con 0x7f64301073b0 2024-03-24T00:14:39.587 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:39.585+0000 7f642f200700 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6430112510 con 0x7f6430102c80 2024-03-24T00:14:39.587 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:39.585+0000 7f642d400700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f6430102c80 0x7f64300749d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:14:39.587 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:39.585+0000 7f642de00700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f64301022d0 0x7f6430074490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:14:39.587 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:39.585+0000 7f642d400700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f6430102c80 0x7f64300749d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.118:49480/0 (socket says 172.21.15.118:49480) 2024-03-24T00:14:39.587 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:39.585+0000 7f642d400700 1 -- 172.21.15.118:0/422349816 learned_addr learned my addr 172.21.15.118:0/422349816 (peer_addr_for_me v2:172.21.15.118:0/0) 2024-03-24T00:14:39.587 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:39.585+0000 7f642e800700 1 --2- 172.21.15.118:0/422349816 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f64301073b0 0x7f6430074f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:14:39.587 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:39.585+0000 7f642d400700 1 -- 172.21.15.118:0/422349816 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f64301073b0 msgr2=0x7f6430074f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:14:39.588 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:39.585+0000 7f642d400700 1 --2- 172.21.15.118:0/422349816 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f64301073b0 0x7f6430074f10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:14:39.588 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:39.585+0000 7f642d400700 1 -- 172.21.15.118:0/422349816 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f64301022d0 msgr2=0x7f6430074490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:14:39.588 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:39.585+0000 7f642d400700 1 --2- 172.21.15.118:0/422349816 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f64301022d0 0x7f6430074490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:14:39.588 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:39.585+0000 7f642d400700 1 -- 172.21.15.118:0/422349816 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f64301a6d20 con 0x7f6430102c80 2024-03-24T00:14:39.588 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:39.585+0000 7f642e800700 1 --2- 172.21.15.118:0/422349816 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f64301073b0 0x7f6430074f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T00:14:39.588 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:39.586+0000 7f642d400700 1 --2- 172.21.15.118:0/422349816 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f6430102c80 0x7f64300749d0 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f641c00ea30 tx=0x7f641c00ed40 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:14:39.588 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:39.586+0000 7f6426a00700 1 -- 172.21.15.118:0/422349816 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f641c00cbd0 con 0x7f6430102c80 2024-03-24T00:14:39.588 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:39.586+0000 7f6426a00700 1 -- 172.21.15.118:0/422349816 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f641c004500 con 0x7f6430102c80 2024-03-24T00:14:39.588 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:39.586+0000 7f642f200700 1 -- 172.21.15.118:0/422349816 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f64301a6fb0 con 0x7f6430102c80 2024-03-24T00:14:39.588 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:39.586+0000 7f6426a00700 1 -- 172.21.15.118:0/422349816 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f641c003e20 con 0x7f6430102c80 2024-03-24T00:14:39.588 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:39.586+0000 7f642f200700 1 -- 172.21.15.118:0/422349816 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f64301a74c0 con 0x7f6430102c80 2024-03-24T00:14:39.589 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:39.587+0000 7f642f200700 1 -- 172.21.15.118:0/422349816 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6430068e50 con 0x7f6430102c80 2024-03-24T00:14:39.589 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:39.589+0000 7f6426a00700 1 -- 172.21.15.118:0/422349816 <== mon.1 v2:172.21.15.193:3300/0 4 ==== mgrmap(e 18) v1 ==== 137291+0+0 (secure 0 0 0) 0x7f641c004000 con 0x7f6430102c80 2024-03-24T00:14:39.591 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:39.590+0000 7f6426a00700 1 --2- 172.21.15.118:0/422349816 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7f64140a2890 0x7f64140a4d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:14:39.591 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:39.590+0000 7f6426a00700 1 -- 172.21.15.118:0/422349816 <== mon.1 v2:172.21.15.193:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 8248+0+0 (secure 0 0 0) 0x7f641c0cebf0 con 0x7f6430102c80 2024-03-24T00:14:39.591 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:39.591+0000 7f642de00700 1 --2- 172.21.15.118:0/422349816 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7f64140a2890 0x7f64140a4d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:14:39.592 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:39.591+0000 7f642de00700 1 --2- 172.21.15.118:0/422349816 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7f64140a2890 0x7f64140a4d50 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7f64200098e0 tx=0x7f6420005960 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:14:39.599 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:39.598+0000 7f6426a00700 1 -- 172.21.15.118:0/422349816 <== mon.1 v2:172.21.15.193:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f641c016330 con 0x7f6430102c80 2024-03-24T00:14:39.821 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:39.820+0000 7f642f200700 1 -- 172.21.15.118:0/422349816 --> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f64300625d0 con 0x7f64140a2890 2024-03-24T00:14:39.830 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:39.829+0000 7f6426a00700 1 -- 172.21.15.118:0/422349816 <== mgr.14152 v2:172.21.15.118:6800/2112912943 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+259325 (secure 0 0 0) 0x7f64300625d0 con 0x7f64140a2890 2024-03-24T00:14:39.832 INFO:teuthology.orchestra.run.smithi118.stdout: 2024-03-24T00:14:39.841 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:39.837+0000 7f642f200700 1 -- 172.21.15.118:0/422349816 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7f64140a2890 msgr2=0x7f64140a4d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:14:39.842 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:39.837+0000 7f642f200700 1 --2- 172.21.15.118:0/422349816 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7f64140a2890 0x7f64140a4d50 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7f64200098e0 tx=0x7f6420005960 comp rx=0 tx=0).stop 2024-03-24T00:14:39.842 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:39.837+0000 7f642f200700 1 -- 172.21.15.118:0/422349816 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f6430102c80 msgr2=0x7f64300749d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:14:39.842 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:39.837+0000 7f642f200700 1 --2- 172.21.15.118:0/422349816 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f6430102c80 0x7f64300749d0 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f641c00ea30 tx=0x7f641c00ed40 comp rx=0 tx=0).stop 2024-03-24T00:14:39.842 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:39.837+0000 7f642f200700 1 -- 172.21.15.118:0/422349816 shutdown_connections 2024-03-24T00:14:39.842 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:39.837+0000 7f642f200700 1 --2- 172.21.15.118:0/422349816 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f64301073b0 0x7f6430074f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:14:39.842 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:39.837+0000 7f642f200700 1 --2- 172.21.15.118:0/422349816 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7f64140a2890 0x7f64140a4d50 unknown :-1 s=CLOSED pgs=205 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:14:39.842 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:39.837+0000 7f642f200700 1 --2- 172.21.15.118:0/422349816 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f64301022d0 0x7f6430074490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:14:39.842 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:39.837+0000 7f642f200700 1 --2- 172.21.15.118:0/422349816 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f6430102c80 0x7f64300749d0 unknown :-1 s=CLOSED pgs=124 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:14:39.842 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:39.837+0000 7f642f200700 1 -- 172.21.15.118:0/422349816 >> 172.21.15.118:0/422349816 conn(0x7f64300fdb60 msgr2=0x7f6430106540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:14:39.842 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:39.837+0000 7f642f200700 1 -- 172.21.15.118:0/422349816 shutdown_connections 2024-03-24T00:14:39.842 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:39.837+0000 7f642f200700 1 -- 172.21.15.118:0/422349816 wait complete. 2024-03-24T00:14:39.843 INFO:teuthology.orchestra.run.smithi118.stderr:dumped all 2024-03-24T00:14:40.758 INFO:teuthology.orchestra.run.smithi118.stdout:{"pg_ready":true,"pg_map":{"version":365,"stamp":"2024-03-24T00:14:39.644444+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":659897,"num_objects":102,"num_object_clones":0,"num_object_copies":306,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":102,"num_whiteouts":0,"num_read":1064,"num_read_kb":654,"num_write":428,"num_write_kb":3082,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":1243109,"num_keys_recovered":0,"num_objects_omap":66,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":811,"ondisk_log_size":811,"up":291,"acting":291,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":291,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":356760,"kb_used_data":11756,"kb_used_omap":18,"kb_used_meta":344877,"kb_avail":1124585064,"statfs":{"total":1151940427776,"available":1151575105536,"internally_reserved":0,"allocated":12038144,"data_stored":7492059,"data_compressed":191142,"data_compressed_allocated":253952,"data_compressed_original":2039808,"omap_allocated":19142,"internal_metadata":353154362},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":45,"apply_latency_ms":45,"commit_latency_ns":45000000,"apply_latency_ns":45000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":390,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":137,"num_read_kb":72,"num_write":8,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004717"},"pg_stats":[{"pgid":"3.3f","version":"88'8","reported_seq":25,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-24T00:14:35.620777+0000","last_change":"2024-03-24T00:13:02.325815+0000","last_active":"2024-03-24T00:14:35.620777+0000","last_peered":"2024-03-24T00:14:35.620777+0000","last_clean":"2024-03-24T00:14:35.620777+0000","last_became_active":"2024-03-24T00:13:02.325450+0000","last_became_peered":"2024-03-24T00:13:02.325450+0000","last_unstale":"2024-03-24T00:14:35.620777+0000","last_undegraded":"2024-03-24T00:14:35.620777+0000","last_fullsized":"2024-03-24T00:14:35.620777+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:13:01.267198+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:13:01.267198+0000","last_clean_scrub_stamp":"2024-03-24T00:13:01.267198+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T02:18:21.994322+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.3e","version":"88'4","reported_seq":21,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-24T00:14:35.611588+0000","last_change":"2024-03-24T00:13:02.324015+0000","last_active":"2024-03-24T00:14:35.611588+0000","last_peered":"2024-03-24T00:14:35.611588+0000","last_clean":"2024-03-24T00:14:35.611588+0000","last_became_active":"2024-03-24T00:13:02.323668+0000","last_became_peered":"2024-03-24T00:13:02.323668+0000","last_unstale":"2024-03-24T00:14:35.611588+0000","last_undegraded":"2024-03-24T00:14:35.611588+0000","last_fullsized":"2024-03-24T00:14:35.611588+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:13:01.267198+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:13:01.267198+0000","last_clean_scrub_stamp":"2024-03-24T00:13:01.267198+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:36:19.442266+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11,10],"acting":[0,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3d","version":"88'7","reported_seq":24,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-24T00:14:35.616284+0000","last_change":"2024-03-24T00:13:02.328911+0000","last_active":"2024-03-24T00:14:35.616284+0000","last_peered":"2024-03-24T00:14:35.616284+0000","last_clean":"2024-03-24T00:14:35.616284+0000","last_became_active":"2024-03-24T00:13:02.324333+0000","last_became_peered":"2024-03-24T00:13:02.324333+0000","last_unstale":"2024-03-24T00:14:35.616284+0000","last_undegraded":"2024-03-24T00:14:35.616284+0000","last_fullsized":"2024-03-24T00:14:35.616284+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:13:01.267198+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:13:01.267198+0000","last_clean_scrub_stamp":"2024-03-24T00:13:01.267198+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T02:49:02.318097+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"88'8","reported_seq":25,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-24T00:14:35.602705+0000","last_change":"2024-03-24T00:13:02.309057+0000","last_active":"2024-03-24T00:14:35.602705+0000","last_peered":"2024-03-24T00:14:35.602705+0000","last_clean":"2024-03-24T00:14:35.602705+0000","last_became_active":"2024-03-24T00:13:02.308403+0000","last_became_peered":"2024-03-24T00:13:02.308403+0000","last_unstale":"2024-03-24T00:14:35.602705+0000","last_undegraded":"2024-03-24T00:14:35.602705+0000","last_fullsized":"2024-03-24T00:14:35.602705+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:13:01.267198+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:13:01.267198+0000","last_clean_scrub_stamp":"2024-03-24T00:13:01.267198+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T07:54:44.310833+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,8,2],"acting":[4,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.3b","version":"88'4","reported_seq":21,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-24T00:14:35.622268+0000","last_change":"2024-03-24T00:13:02.330651+0000","last_active":"2024-03-24T00:14:35.622268+0000","last_peered":"2024-03-24T00:14:35.622268+0000","last_clean":"2024-03-24T00:14:35.622268+0000","last_became_active":"2024-03-24T00:13:02.330313+0000","last_became_peered":"2024-03-24T00:13:02.330313+0000","last_unstale":"2024-03-24T00:14:35.622268+0000","last_undegraded":"2024-03-24T00:14:35.622268+0000","last_fullsized":"2024-03-24T00:14:35.622268+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:13:01.267198+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:13:01.267198+0000","last_clean_scrub_stamp":"2024-03-24T00:13:01.267198+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T03:45:38.480929+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,6],"acting":[8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3a","version":"88'6","reported_seq":23,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-24T00:14:35.607384+0000","last_change":"2024-03-24T00:13:02.313115+0000","last_active":"2024-03-24T00:14:35.607384+0000","last_peered":"2024-03-24T00:14:35.607384+0000","last_clean":"2024-03-24T00:14:35.607384+0000","last_became_active":"2024-03-24T00:13:02.312387+0000","last_became_peered":"2024-03-24T00:13:02.312387+0000","last_unstale":"2024-03-24T00:14:35.607384+0000","last_undegraded":"2024-03-24T00:14:35.607384+0000","last_fullsized":"2024-03-24T00:14:35.607384+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:13:01.267198+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:13:01.267198+0000","last_clean_scrub_stamp":"2024-03-24T00:13:01.267198+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T09:16:02.574920+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,4],"acting":[10,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.39","version":"88'5","reported_seq":28,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-24T00:14:35.605231+0000","last_change":"2024-03-24T00:14:07.503364+0000","last_active":"2024-03-24T00:14:35.605231+0000","last_peered":"2024-03-24T00:14:35.605231+0000","last_clean":"2024-03-24T00:14:35.605231+0000","last_became_active":"2024-03-24T00:14:07.503103+0000","last_became_peered":"2024-03-24T00:14:07.503103+0000","last_unstale":"2024-03-24T00:14:35.605231+0000","last_undegraded":"2024-03-24T00:14:35.605231+0000","last_fullsized":"2024-03-24T00:14:35.605231+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:13:01.267198+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:13:01.267198+0000","last_clean_scrub_stamp":"2024-03-24T00:13:01.267198+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T03:22:04.225634+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,8],"acting":[7,9,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"88'9","reported_seq":20,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-24T00:14:35.608741+0000","last_change":"2024-03-24T00:14:07.681011+0000","last_active":"2024-03-24T00:14:35.608741+0000","last_peered":"2024-03-24T00:14:35.608741+0000","last_clean":"2024-03-24T00:14:35.608741+0000","last_became_active":"2024-03-24T00:14:07.680726+0000","last_became_peered":"2024-03-24T00:14:07.680726+0000","last_unstale":"2024-03-24T00:14:35.608741+0000","last_undegraded":"2024-03-24T00:14:35.608741+0000","last_fullsized":"2024-03-24T00:14:35.608741+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:13:01.267198+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:13:01.267198+0000","last_clean_scrub_stamp":"2024-03-24T00:13:01.267198+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T01:05:03.379937+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,5,10],"acting":[9,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.37","version":"88'11","reported_seq":28,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-24T00:14:35.619612+0000","last_change":"2024-03-24T00:13:02.305625+0000","last_active":"2024-03-24T00:14:35.619612+0000","last_peered":"2024-03-24T00:14:35.619612+0000","last_clean":"2024-03-24T00:14:35.619612+0000","last_became_active":"2024-03-24T00:13:02.305221+0000","last_became_peered":"2024-03-24T00:13:02.305221+0000","last_unstale":"2024-03-24T00:14:35.619612+0000","last_undegraded":"2024-03-24T00:14:35.619612+0000","last_fullsized":"2024-03-24T00:14:35.619612+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:13:01.267198+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:13:01.267198+0000","last_clean_scrub_stamp":"2024-03-24T00:13:01.267198+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T04:37:12.467387+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,11],"acting":[1,5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.36","version":"88'16","reported_seq":33,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-24T00:14:35.610392+0000","last_change":"2024-03-24T00:13:02.307878+0000","last_active":"2024-03-24T00:14:35.610392+0000","last_peered":"2024-03-24T00:14:35.610392+0000","last_clean":"2024-03-24T00:14:35.610392+0000","last_became_active":"2024-03-24T00:13:02.307513+0000","last_became_peered":"2024-03-24T00:13:02.307513+0000","last_unstale":"2024-03-24T00:14:35.610392+0000","last_undegraded":"2024-03-24T00:14:35.610392+0000","last_fullsized":"2024-03-24T00:14:35.610392+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:13:01.267198+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:13:01.267198+0000","last_clean_scrub_stamp":"2024-03-24T00:13:01.267198+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T07:06:08.162416+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,4],"acting":[10,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.35","version":"88'6","reported_seq":23,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-24T00:14:35.616059+0000","last_change":"2024-03-24T00:13:02.310883+0000","last_active":"2024-03-24T00:14:35.616059+0000","last_peered":"2024-03-24T00:14:35.616059+0000","last_clean":"2024-03-24T00:14:35.616059+0000","last_became_active":"2024-03-24T00:13:02.310559+0000","last_became_peered":"2024-03-24T00:13:02.310559+0000","last_unstale":"2024-03-24T00:14:35.616059+0000","last_undegraded":"2024-03-24T00:14:35.616059+0000","last_fullsized":"2024-03-24T00:14:35.616059+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:13:01.267198+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:13:01.267198+0000","last_clean_scrub_stamp":"2024-03-24T00:13:01.267198+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T04:04:25.801248+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8,1],"acting":[7,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.34","version":"88'7","reported_seq":24,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-24T00:14:35.602511+0000","last_change":"2024-03-24T00:13:02.308747+0000","last_active":"2024-03-24T00:14:35.602511+0000","last_peered":"2024-03-24T00:14:35.602511+0000","last_clean":"2024-03-24T00:14:35.602511+0000","last_became_active":"2024-03-24T00:13:02.307791+0000","last_became_peered":"2024-03-24T00:13:02.307791+0000","last_unstale":"2024-03-24T00:14:35.602511+0000","last_undegraded":"2024-03-24T00:14:35.602511+0000","last_fullsized":"2024-03-24T00:14:35.602511+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:13:01.267198+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:13:01.267198+0000","last_clean_scrub_stamp":"2024-03-24T00:13:01.267198+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T03:28:08.780044+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,8],"acting":[4,10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"88'10","reported_seq":27,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-24T00:14:35.625613+0000","last_change":"2024-03-24T00:13:02.327061+0000","last_active":"2024-03-24T00:14:35.625613+0000","last_peered":"2024-03-24T00:14:35.625613+0000","last_clean":"2024-03-24T00:14:35.625613+0000","last_became_active":"2024-03-24T00:13:02.326696+0000","last_became_peered":"2024-03-24T00:13:02.326696+0000","last_unstale":"2024-03-24T00:14:35.625613+0000","last_undegraded":"2024-03-24T00:14:35.625613+0000","last_fullsized":"2024-03-24T00:14:35.625613+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:13:01.267198+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:13:01.267198+0000","last_clean_scrub_stamp":"2024-03-24T00:13:01.267198+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T08:55:17.325488+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,5],"acting":[6,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.32","version":"88'7","reported_seq":24,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-24T00:14:35.610502+0000","last_change":"2024-03-24T00:13:02.321326+0000","last_active":"2024-03-24T00:14:35.610502+0000","last_peered":"2024-03-24T00:14:35.610502+0000","last_clean":"2024-03-24T00:14:35.610502+0000","last_became_active":"2024-03-24T00:13:02.320964+0000","last_became_peered":"2024-03-24T00:13:02.320964+0000","last_unstale":"2024-03-24T00:14:35.610502+0000","last_undegraded":"2024-03-24T00:14:35.610502+0000","last_fullsized":"2024-03-24T00:14:35.610502+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:13:01.267198+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:13:01.267198+0000","last_clean_scrub_stamp":"2024-03-24T00:13:01.267198+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T08:05:15.365809+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,10],"acting":[0,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"88'7","reported_seq":30,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-24T00:14:35.605221+0000","last_change":"2024-03-24T00:14:07.505774+0000","last_active":"2024-03-24T00:14:35.605221+0000","last_peered":"2024-03-24T00:14:35.605221+0000","last_clean":"2024-03-24T00:14:35.605221+0000","last_became_active":"2024-03-24T00:14:07.504800+0000","last_became_peered":"2024-03-24T00:14:07.504800+0000","last_unstale":"2024-03-24T00:14:35.605221+0000","last_undegraded":"2024-03-24T00:14:35.605221+0000","last_fullsized":"2024-03-24T00:14:35.605221+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:13:01.267198+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:13:01.267198+0000","last_clean_scrub_stamp":"2024-03-24T00:13:01.267198+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T06:50:30.686330+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.30","version":"88'9","reported_seq":26,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-24T00:14:35.621361+0000","last_change":"2024-03-24T00:13:02.325460+0000","last_active":"2024-03-24T00:14:35.621361+0000","last_peered":"2024-03-24T00:14:35.621361+0000","last_clean":"2024-03-24T00:14:35.621361+0000","last_became_active":"2024-03-24T00:13:02.325112+0000","last_became_peered":"2024-03-24T00:13:02.325112+0000","last_unstale":"2024-03-24T00:14:35.621361+0000","last_undegraded":"2024-03-24T00:14:35.621361+0000","last_fullsized":"2024-03-24T00:14:35.621361+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:13:01.267198+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:13:01.267198+0000","last_clean_scrub_stamp":"2024-03-24T00:13:01.267198+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T03:15:16.362970+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2f","version":"88'8","reported_seq":25,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-24T00:14:35.620799+0000","last_change":"2024-03-24T00:13:02.325918+0000","last_active":"2024-03-24T00:14:35.620799+0000","last_peered":"2024-03-24T00:14:35.620799+0000","last_clean":"2024-03-24T00:14:35.620799+0000","last_became_active":"2024-03-24T00:13:02.325557+0000","last_became_peered":"2024-03-24T00:13:02.325557+0000","last_unstale":"2024-03-24T00:14:35.620799+0000","last_undegraded":"2024-03-24T00:14:35.620799+0000","last_fullsized":"2024-03-24T00:14:35.620799+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:13:01.267198+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:13:01.267198+0000","last_clean_scrub_stamp":"2024-03-24T00:13:01.267198+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T02:55:43.759674+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2e","version":"88'10","reported_seq":27,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-24T00:14:35.619330+0000","last_change":"2024-03-24T00:13:02.308106+0000","last_active":"2024-03-24T00:14:35.619330+0000","last_peered":"2024-03-24T00:14:35.619330+0000","last_clean":"2024-03-24T00:14:35.619330+0000","last_became_active":"2024-03-24T00:13:02.306972+0000","last_became_peered":"2024-03-24T00:13:02.306972+0000","last_unstale":"2024-03-24T00:14:35.619330+0000","last_undegraded":"2024-03-24T00:14:35.619330+0000","last_fullsized":"2024-03-24T00:14:35.619330+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:13:01.267198+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:13:01.267198+0000","last_clean_scrub_stamp":"2024-03-24T00:13:01.267198+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T09:38:54.677255+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,10],"acting":[9,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.2d","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-24T00:14:07.495228+0000","last_change":"2024-03-24T00:13:02.331742+0000","last_active":"2024-03-24T00:14:07.495228+0000","last_peered":"2024-03-24T00:14:07.495228+0000","last_clean":"2024-03-24T00:14:07.495228+0000","last_became_active":"2024-03-24T00:13:02.331103+0000","last_became_peered":"2024-03-24T00:13:02.331103+0000","last_unstale":"2024-03-24T00:14:07.495228+0000","last_undegraded":"2024-03-24T00:14:07.495228+0000","last_fullsized":"2024-03-24T00:14:07.495228+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:13:01.267198+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:13:01.267198+0000","last_clean_scrub_stamp":"2024-03-24T00:13:01.267198+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:25:58.116306+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1f","version":"86'13","reported_seq":142,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-24T00:14:36.564387+0000","last_change":"2024-03-24T00:14:07.520827+0000","last_active":"2024-03-24T00:14:36.564387+0000","last_peered":"2024-03-24T00:14:36.564387+0000","last_clean":"2024-03-24T00:14:36.564387+0000","last_became_active":"2024-03-24T00:14:07.505406+0000","last_became_peered":"2024-03-24T00:14:07.505406+0000","last_unstale":"2024-03-24T00:14:36.564387+0000","last_undegraded":"2024-03-24T00:14:36.564387+0000","last_fullsized":"2024-03-24T00:14:36.564387+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:13:00.222790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:13:00.222790+0000","last_clean_scrub_stamp":"2024-03-24T00:13:00.222790+0000","objects_scrubbed":0,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T01:05:01.235978+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":88,"num_read_kb":88,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":672,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,1],"acting":[9,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.1e","version":"88'5","reported_seq":22,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-24T00:14:35.613946+0000","last_change":"2024-03-24T00:13:02.322461+0000","last_active":"2024-03-24T00:14:35.613946+0000","last_peered":"2024-03-24T00:14:35.613946+0000","last_clean":"2024-03-24T00:14:35.613946+0000","last_became_active":"2024-03-24T00:13:02.321752+0000","last_became_peered":"2024-03-24T00:13:02.321752+0000","last_unstale":"2024-03-24T00:14:35.613946+0000","last_undegraded":"2024-03-24T00:14:35.613946+0000","last_fullsized":"2024-03-24T00:14:35.613946+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:13:01.267198+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:13:01.267198+0000","last_clean_scrub_stamp":"2024-03-24T00:13:01.267198+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T06:37:05.825919+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1e","version":"86'2","reported_seq":21,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-24T00:14:07.504657+0000","last_change":"2024-03-24T00:13:01.324972+0000","last_active":"2024-03-24T00:14:07.504657+0000","last_peered":"2024-03-24T00:14:07.504657+0000","last_clean":"2024-03-24T00:14:07.504657+0000","last_became_active":"2024-03-24T00:13:01.323342+0000","last_became_peered":"2024-03-24T00:13:01.323342+0000","last_unstale":"2024-03-24T00:14:07.504657+0000","last_undegraded":"2024-03-24T00:14:07.504657+0000","last_fullsized":"2024-03-24T00:14:07.504657+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:13:00.222790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:13:00.222790+0000","last_clean_scrub_stamp":"2024-03-24T00:13:00.222790+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T05:34:35.827634+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"88'6","reported_seq":23,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-24T00:14:35.622169+0000","last_change":"2024-03-24T00:13:02.325455+0000","last_active":"2024-03-24T00:14:35.622169+0000","last_peered":"2024-03-24T00:14:35.622169+0000","last_clean":"2024-03-24T00:14:35.622169+0000","last_became_active":"2024-03-24T00:13:02.325061+0000","last_became_peered":"2024-03-24T00:13:02.325061+0000","last_unstale":"2024-03-24T00:14:35.622169+0000","last_undegraded":"2024-03-24T00:14:35.622169+0000","last_fullsized":"2024-03-24T00:14:35.622169+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:13:01.267198+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:13:01.267198+0000","last_clean_scrub_stamp":"2024-03-24T00:13:01.267198+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T06:13:33.223361+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,9],"acting":[0,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1d","version":"86'12","reported_seq":32,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-24T00:14:07.499436+0000","last_change":"2024-03-24T00:13:01.300541+0000","last_active":"2024-03-24T00:14:07.499436+0000","last_peered":"2024-03-24T00:14:07.499436+0000","last_clean":"2024-03-24T00:14:07.499436+0000","last_became_active":"2024-03-24T00:13:01.299686+0000","last_became_peered":"2024-03-24T00:13:01.299686+0000","last_unstale":"2024-03-24T00:14:07.499436+0000","last_undegraded":"2024-03-24T00:14:07.499436+0000","last_fullsized":"2024-03-24T00:14:07.499436+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:13:00.222790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:13:00.222790+0000","last_clean_scrub_stamp":"2024-03-24T00:13:00.222790+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T00:20:54.474131+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":16,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":7,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1c","version":"88'9","reported_seq":26,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-24T00:14:35.624592+0000","last_change":"2024-03-24T00:13:02.325130+0000","last_active":"2024-03-24T00:14:35.624592+0000","last_peered":"2024-03-24T00:14:35.624592+0000","last_clean":"2024-03-24T00:14:35.624592+0000","last_became_active":"2024-03-24T00:13:02.324740+0000","last_became_peered":"2024-03-24T00:13:02.324740+0000","last_unstale":"2024-03-24T00:14:35.624592+0000","last_undegraded":"2024-03-24T00:14:35.624592+0000","last_fullsized":"2024-03-24T00:14:35.624592+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:13:01.267198+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:13:01.267198+0000","last_clean_scrub_stamp":"2024-03-24T00:13:01.267198+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T01:23:35.194788+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,6],"acting":[1,10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1c","version":"86'5","reported_seq":25,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-24T00:14:07.499266+0000","last_change":"2024-03-24T00:13:01.300572+0000","last_active":"2024-03-24T00:14:07.499266+0000","last_peered":"2024-03-24T00:14:07.499266+0000","last_clean":"2024-03-24T00:14:07.499266+0000","last_became_active":"2024-03-24T00:13:01.299801+0000","last_became_peered":"2024-03-24T00:13:01.299801+0000","last_unstale":"2024-03-24T00:14:07.499266+0000","last_undegraded":"2024-03-24T00:14:07.499266+0000","last_fullsized":"2024-03-24T00:14:07.499266+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:13:00.222790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:13:00.222790+0000","last_clean_scrub_stamp":"2024-03-24T00:13:00.222790+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T10:29:03.986969+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":7,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,5],"acting":[8,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1d","version":"88'15","reported_seq":32,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-24T00:14:35.613940+0000","last_change":"2024-03-24T00:13:02.304079+0000","last_active":"2024-03-24T00:14:35.613940+0000","last_peered":"2024-03-24T00:14:35.613940+0000","last_clean":"2024-03-24T00:14:35.613940+0000","last_became_active":"2024-03-24T00:13:02.303700+0000","last_became_peered":"2024-03-24T00:13:02.303700+0000","last_unstale":"2024-03-24T00:14:35.613940+0000","last_undegraded":"2024-03-24T00:14:35.613940+0000","last_fullsized":"2024-03-24T00:14:35.613940+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:13:01.267198+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:13:01.267198+0000","last_clean_scrub_stamp":"2024-03-24T00:13:01.267198+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T10:19:07.701927+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1b","version":"88'7","reported_seq":122,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-24T00:14:36.369831+0000","last_change":"2024-03-24T00:13:01.300949+0000","last_active":"2024-03-24T00:14:36.369831+0000","last_peered":"2024-03-24T00:14:36.369831+0000","last_clean":"2024-03-24T00:14:36.369831+0000","last_became_active":"2024-03-24T00:13:01.300605+0000","last_became_peered":"2024-03-24T00:13:01.300605+0000","last_unstale":"2024-03-24T00:14:36.369831+0000","last_undegraded":"2024-03-24T00:14:36.369831+0000","last_fullsized":"2024-03-24T00:14:36.369831+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:13:00.222790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:13:00.222790+0000","last_clean_scrub_stamp":"2024-03-24T00:13:00.222790+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T02:47:58.283103+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1154,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":96,"num_read_kb":4,"num_write":4,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1a","version":"88'7","reported_seq":24,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-24T00:14:35.603317+0000","last_change":"2024-03-24T00:13:02.312884+0000","last_active":"2024-03-24T00:14:35.603317+0000","last_peered":"2024-03-24T00:14:35.603317+0000","last_clean":"2024-03-24T00:14:35.603317+0000","last_became_active":"2024-03-24T00:13:02.312533+0000","last_became_peered":"2024-03-24T00:13:02.312533+0000","last_unstale":"2024-03-24T00:14:35.603317+0000","last_undegraded":"2024-03-24T00:14:35.603317+0000","last_fullsized":"2024-03-24T00:14:35.603317+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:13:01.267198+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:13:01.267198+0000","last_clean_scrub_stamp":"2024-03-24T00:13:01.267198+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T10:54:48.398645+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,5],"acting":[8,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.1a","version":"86'7","reported_seq":26,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-24T00:14:07.492787+0000","last_change":"2024-03-24T00:13:01.306292+0000","last_active":"2024-03-24T00:14:07.492787+0000","last_peered":"2024-03-24T00:14:07.492787+0000","last_clean":"2024-03-24T00:14:07.492787+0000","last_became_active":"2024-03-24T00:13:01.305898+0000","last_became_peered":"2024-03-24T00:13:01.305898+0000","last_unstale":"2024-03-24T00:14:07.492787+0000","last_undegraded":"2024-03-24T00:14:07.492787+0000","last_fullsized":"2024-03-24T00:14:07.492787+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:13:00.222790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:13:00.222790+0000","last_clean_scrub_stamp":"2024-03-24T00:13:00.222790+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T07:41:17.003588+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8,6],"acting":[10,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.1b","version":"88'8","reported_seq":25,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-24T00:14:35.622308+0000","last_change":"2024-03-24T00:13:02.323763+0000","last_active":"2024-03-24T00:14:35.622308+0000","last_peered":"2024-03-24T00:14:35.622308+0000","last_clean":"2024-03-24T00:14:35.622308+0000","last_became_active":"2024-03-24T00:13:02.323367+0000","last_became_peered":"2024-03-24T00:13:02.323367+0000","last_unstale":"2024-03-24T00:14:35.622308+0000","last_undegraded":"2024-03-24T00:14:35.622308+0000","last_fullsized":"2024-03-24T00:14:35.622308+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:13:01.267198+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:13:01.267198+0000","last_clean_scrub_stamp":"2024-03-24T00:13:01.267198+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T08:53:21.671317+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,3],"acting":[5,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.19","version":"86'3","reported_seq":22,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-24T00:14:07.632380+0000","last_change":"2024-03-24T00:13:01.323111+0000","last_active":"2024-03-24T00:14:07.632380+0000","last_peered":"2024-03-24T00:14:07.632380+0000","last_clean":"2024-03-24T00:14:07.632380+0000","last_became_active":"2024-03-24T00:13:01.322772+0000","last_became_peered":"2024-03-24T00:13:01.322772+0000","last_unstale":"2024-03-24T00:14:07.632380+0000","last_undegraded":"2024-03-24T00:14:07.632380+0000","last_fullsized":"2024-03-24T00:14:07.632380+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:13:00.222790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:13:00.222790+0000","last_clean_scrub_stamp":"2024-03-24T00:13:00.222790+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T04:19:53.097933+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,9],"acting":[6,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"88'8","reported_seq":25,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-24T00:14:35.624504+0000","last_change":"2024-03-24T00:13:02.330469+0000","last_active":"2024-03-24T00:14:35.624504+0000","last_peered":"2024-03-24T00:14:35.624504+0000","last_clean":"2024-03-24T00:14:35.624504+0000","last_became_active":"2024-03-24T00:13:02.327019+0000","last_became_peered":"2024-03-24T00:13:02.327019+0000","last_unstale":"2024-03-24T00:14:35.624504+0000","last_undegraded":"2024-03-24T00:14:35.624504+0000","last_fullsized":"2024-03-24T00:14:35.624504+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:13:01.267198+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:13:01.267198+0000","last_clean_scrub_stamp":"2024-03-24T00:13:01.267198+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T07:30:37.155535+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,6],"acting":[3,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.18","version":"86'3","reported_seq":23,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-24T00:14:07.679914+0000","last_change":"2024-03-24T00:13:01.300089+0000","last_active":"2024-03-24T00:14:07.679914+0000","last_peered":"2024-03-24T00:14:07.679914+0000","last_clean":"2024-03-24T00:14:07.679914+0000","last_became_active":"2024-03-24T00:13:01.299747+0000","last_became_peered":"2024-03-24T00:13:01.299747+0000","last_unstale":"2024-03-24T00:14:07.679914+0000","last_undegraded":"2024-03-24T00:14:07.679914+0000","last_fullsized":"2024-03-24T00:14:07.679914+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:13:00.222790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:13:00.222790+0000","last_clean_scrub_stamp":"2024-03-24T00:13:00.222790+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T00:30:30.142785+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"88'11","reported_seq":28,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-24T00:14:35.625608+0000","last_change":"2024-03-24T00:13:02.332490+0000","last_active":"2024-03-24T00:14:35.625608+0000","last_peered":"2024-03-24T00:14:35.625608+0000","last_clean":"2024-03-24T00:14:35.625608+0000","last_became_active":"2024-03-24T00:13:02.331182+0000","last_became_peered":"2024-03-24T00:13:02.331182+0000","last_unstale":"2024-03-24T00:14:35.625608+0000","last_undegraded":"2024-03-24T00:14:35.625608+0000","last_fullsized":"2024-03-24T00:14:35.625608+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:13:01.267198+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:13:01.267198+0000","last_clean_scrub_stamp":"2024-03-24T00:13:01.267198+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T06:22:16.084879+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,10],"acting":[9,3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.17","version":"88'6","reported_seq":208,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-24T00:14:38.371517+0000","last_change":"2024-03-24T00:13:01.299818+0000","last_active":"2024-03-24T00:14:38.371517+0000","last_peered":"2024-03-24T00:14:38.371517+0000","last_clean":"2024-03-24T00:14:38.371517+0000","last_became_active":"2024-03-24T00:13:01.299478+0000","last_became_peered":"2024-03-24T00:13:01.299478+0000","last_unstale":"2024-03-24T00:14:38.371517+0000","last_undegraded":"2024-03-24T00:14:38.371517+0000","last_fullsized":"2024-03-24T00:14:38.371517+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:13:00.222790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:13:00.222790+0000","last_clean_scrub_stamp":"2024-03-24T00:13:00.222790+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:51:47.088677+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":180,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":183,"num_read_kb":183,"num_write":7,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,9],"acting":[10,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.16","version":"88'10","reported_seq":27,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-24T00:14:35.621993+0000","last_change":"2024-03-24T00:13:02.304971+0000","last_active":"2024-03-24T00:14:35.621993+0000","last_peered":"2024-03-24T00:14:35.621993+0000","last_clean":"2024-03-24T00:14:35.621993+0000","last_became_active":"2024-03-24T00:13:02.304577+0000","last_became_peered":"2024-03-24T00:13:02.304577+0000","last_unstale":"2024-03-24T00:14:35.621993+0000","last_undegraded":"2024-03-24T00:14:35.621993+0000","last_fullsized":"2024-03-24T00:14:35.621993+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:13:01.267198+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:13:01.267198+0000","last_clean_scrub_stamp":"2024-03-24T00:13:01.267198+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T05:08:03.444554+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,9],"acting":[1,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.16","version":"86'4","reported_seq":23,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-24T00:14:07.496701+0000","last_change":"2024-03-24T00:13:01.303075+0000","last_active":"2024-03-24T00:14:07.496701+0000","last_peered":"2024-03-24T00:14:07.496701+0000","last_clean":"2024-03-24T00:14:07.496701+0000","last_became_active":"2024-03-24T00:13:01.301517+0000","last_became_peered":"2024-03-24T00:13:01.301517+0000","last_unstale":"2024-03-24T00:14:07.496701+0000","last_undegraded":"2024-03-24T00:14:07.496701+0000","last_fullsized":"2024-03-24T00:14:07.496701+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:13:00.222790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:13:00.222790+0000","last_clean_scrub_stamp":"2024-03-24T00:13:00.222790+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:04:10.552510+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.17","version":"88'9","reported_seq":26,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-24T00:14:35.618608+0000","last_change":"2024-03-24T00:13:02.320108+0000","last_active":"2024-03-24T00:14:35.618608+0000","last_peered":"2024-03-24T00:14:35.618608+0000","last_clean":"2024-03-24T00:14:35.618608+0000","last_became_active":"2024-03-24T00:13:02.319155+0000","last_became_peered":"2024-03-24T00:13:02.319155+0000","last_unstale":"2024-03-24T00:14:35.618608+0000","last_undegraded":"2024-03-24T00:14:35.618608+0000","last_fullsized":"2024-03-24T00:14:35.618608+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:13:01.267198+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:13:01.267198+0000","last_clean_scrub_stamp":"2024-03-24T00:13:01.267198+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T10:15:54.770304+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,5],"acting":[0,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.15","version":"86'5","reported_seq":25,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-24T00:14:07.680187+0000","last_change":"2024-03-24T00:13:01.323919+0000","last_active":"2024-03-24T00:14:07.680187+0000","last_peered":"2024-03-24T00:14:07.680187+0000","last_clean":"2024-03-24T00:14:07.680187+0000","last_became_active":"2024-03-24T00:13:01.323640+0000","last_became_peered":"2024-03-24T00:13:01.323640+0000","last_unstale":"2024-03-24T00:14:07.680187+0000","last_undegraded":"2024-03-24T00:14:07.680187+0000","last_fullsized":"2024-03-24T00:14:07.680187+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:13:00.222790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:13:00.222790+0000","last_clean_scrub_stamp":"2024-03-24T00:13:00.222790+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T06:16:49.018642+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":7,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,10],"acting":[5,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.14","version":"88'11","reported_seq":34,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-24T00:14:35.624710+0000","last_change":"2024-03-24T00:14:07.635409+0000","last_active":"2024-03-24T00:14:35.624710+0000","last_peered":"2024-03-24T00:14:35.624710+0000","last_clean":"2024-03-24T00:14:35.624710+0000","last_became_active":"2024-03-24T00:14:07.635093+0000","last_became_peered":"2024-03-24T00:14:07.635093+0000","last_unstale":"2024-03-24T00:14:35.624710+0000","last_undegraded":"2024-03-24T00:14:35.624710+0000","last_fullsized":"2024-03-24T00:14:35.624710+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:13:01.267198+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:13:01.267198+0000","last_clean_scrub_stamp":"2024-03-24T00:13:01.267198+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T10:31:10.205874+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,6],"acting":[1,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.14","version":"88'35","reported_seq":269,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-24T00:14:38.835685+0000","last_change":"2024-03-24T00:14:07.807393+0000","last_active":"2024-03-24T00:14:38.835685+0000","last_peered":"2024-03-24T00:14:38.835685+0000","last_clean":"2024-03-24T00:14:38.835685+0000","last_became_active":"2024-03-24T00:14:07.801140+0000","last_became_peered":"2024-03-24T00:14:07.801140+0000","last_unstale":"2024-03-24T00:14:38.835685+0000","last_undegraded":"2024-03-24T00:14:38.835685+0000","last_fullsized":"2024-03-24T00:14:38.835685+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:13:00.222790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:13:00.222790+0000","last_clean_scrub_stamp":"2024-03-24T00:13:00.222790+0000","objects_scrubbed":0,"log_size":35,"log_dups_size":0,"ondisk_log_size":35,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T10:35:25.627828+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":61950,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":191,"num_read_kb":155,"num_write":37,"num_write_kb":76,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":61701,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,4],"acting":[10,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.15","version":"88'6","reported_seq":23,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-24T00:14:35.600331+0000","last_change":"2024-03-24T00:13:02.307653+0000","last_active":"2024-03-24T00:14:35.600331+0000","last_peered":"2024-03-24T00:14:35.600331+0000","last_clean":"2024-03-24T00:14:35.600331+0000","last_became_active":"2024-03-24T00:13:02.307287+0000","last_became_peered":"2024-03-24T00:13:02.307287+0000","last_unstale":"2024-03-24T00:14:35.600331+0000","last_undegraded":"2024-03-24T00:14:35.600331+0000","last_fullsized":"2024-03-24T00:14:35.600331+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:13:01.267198+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:13:01.267198+0000","last_clean_scrub_stamp":"2024-03-24T00:13:01.267198+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T05:59:27.701614+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.13","version":"86'9","reported_seq":30,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-24T00:14:07.495111+0000","last_change":"2024-03-24T00:13:01.316854+0000","last_active":"2024-03-24T00:14:07.495111+0000","last_peered":"2024-03-24T00:14:07.495111+0000","last_clean":"2024-03-24T00:14:07.495111+0000","last_became_active":"2024-03-24T00:13:01.316507+0000","last_became_peered":"2024-03-24T00:13:01.316507+0000","last_unstale":"2024-03-24T00:14:07.495111+0000","last_undegraded":"2024-03-24T00:14:07.495111+0000","last_fullsized":"2024-03-24T00:14:07.495111+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:13:00.222790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:13:00.222790+0000","last_clean_scrub_stamp":"2024-03-24T00:13:00.222790+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T07:58:41.888003+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":13,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,10,3],"acting":[2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.12","version":"88'9","reported_seq":26,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-24T00:14:35.626094+0000","last_change":"2024-03-24T00:13:02.338811+0000","last_active":"2024-03-24T00:14:35.626094+0000","last_peered":"2024-03-24T00:14:35.626094+0000","last_clean":"2024-03-24T00:14:35.626094+0000","last_became_active":"2024-03-24T00:13:02.338438+0000","last_became_peered":"2024-03-24T00:13:02.338438+0000","last_unstale":"2024-03-24T00:14:35.626094+0000","last_undegraded":"2024-03-24T00:14:35.626094+0000","last_fullsized":"2024-03-24T00:14:35.626094+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:13:01.267198+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:13:01.267198+0000","last_clean_scrub_stamp":"2024-03-24T00:13:01.267198+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T03:55:16.531224+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.12","version":"86'5","reported_seq":25,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-24T00:14:07.679758+0000","last_change":"2024-03-24T00:13:01.313914+0000","last_active":"2024-03-24T00:14:07.679758+0000","last_peered":"2024-03-24T00:14:07.679758+0000","last_clean":"2024-03-24T00:14:07.679758+0000","last_became_active":"2024-03-24T00:13:01.313535+0000","last_became_peered":"2024-03-24T00:13:01.313535+0000","last_unstale":"2024-03-24T00:14:07.679758+0000","last_undegraded":"2024-03-24T00:14:07.679758+0000","last_fullsized":"2024-03-24T00:14:07.679758+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:13:00.222790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:13:00.222790+0000","last_clean_scrub_stamp":"2024-03-24T00:13:00.222790+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T03:23:53.011550+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":5,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"88'15","reported_seq":38,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-24T00:14:35.623013+0000","last_change":"2024-03-24T00:14:07.632978+0000","last_active":"2024-03-24T00:14:35.623013+0000","last_peered":"2024-03-24T00:14:35.623013+0000","last_clean":"2024-03-24T00:14:35.623013+0000","last_became_active":"2024-03-24T00:14:07.632719+0000","last_became_peered":"2024-03-24T00:14:07.632719+0000","last_unstale":"2024-03-24T00:14:35.623013+0000","last_undegraded":"2024-03-24T00:14:35.623013+0000","last_fullsized":"2024-03-24T00:14:35.623013+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:13:01.267198+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:13:01.267198+0000","last_clean_scrub_stamp":"2024-03-24T00:13:01.267198+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T09:59:38.993218+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,6],"acting":[7,9,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.11","version":"86'5","reported_seq":116,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-24T00:14:37.463705+0000","last_change":"2024-03-24T00:13:01.297887+0000","last_active":"2024-03-24T00:14:37.463705+0000","last_peered":"2024-03-24T00:14:37.463705+0000","last_clean":"2024-03-24T00:14:37.463705+0000","last_became_active":"2024-03-24T00:13:01.297560+0000","last_became_peered":"2024-03-24T00:13:01.297560+0000","last_unstale":"2024-03-24T00:14:37.463705+0000","last_undegraded":"2024-03-24T00:14:37.463705+0000","last_fullsized":"2024-03-24T00:14:37.463705+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:13:00.222790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:13:00.222790+0000","last_clean_scrub_stamp":"2024-03-24T00:13:00.222790+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T01:06:28.927274+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":588,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":93,"num_read_kb":3,"num_write":5,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,10],"acting":[4,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.10","version":"88'6","reported_seq":23,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-24T00:14:35.604519+0000","last_change":"2024-03-24T00:13:02.306811+0000","last_active":"2024-03-24T00:14:35.604519+0000","last_peered":"2024-03-24T00:14:35.604519+0000","last_clean":"2024-03-24T00:14:35.604519+0000","last_became_active":"2024-03-24T00:13:02.306499+0000","last_became_peered":"2024-03-24T00:13:02.306499+0000","last_unstale":"2024-03-24T00:14:35.604519+0000","last_undegraded":"2024-03-24T00:14:35.604519+0000","last_fullsized":"2024-03-24T00:14:35.604519+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:13:01.267198+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:13:01.267198+0000","last_clean_scrub_stamp":"2024-03-24T00:13:01.267198+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:49:32.683005+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.10","version":"86'17","reported_seq":37,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-24T00:14:07.498949+0000","last_change":"2024-03-24T00:13:01.307058+0000","last_active":"2024-03-24T00:14:07.498949+0000","last_peered":"2024-03-24T00:14:07.498949+0000","last_clean":"2024-03-24T00:14:07.498949+0000","last_became_active":"2024-03-24T00:13:01.306699+0000","last_became_peered":"2024-03-24T00:13:01.306699+0000","last_unstale":"2024-03-24T00:14:07.498949+0000","last_undegraded":"2024-03-24T00:14:07.498949+0000","last_fullsized":"2024-03-24T00:14:07.498949+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:13:00.222790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:13:00.222790+0000","last_clean_scrub_stamp":"2024-03-24T00:13:00.222790+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T10:32:37.334514+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,2],"acting":[8,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.11","version":"88'6","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-24T00:14:35.620207+0000","last_change":"2024-03-24T00:14:07.503357+0000","last_active":"2024-03-24T00:14:35.620207+0000","last_peered":"2024-03-24T00:14:35.620207+0000","last_clean":"2024-03-24T00:14:35.620207+0000","last_became_active":"2024-03-24T00:14:07.502511+0000","last_became_peered":"2024-03-24T00:14:07.502511+0000","last_unstale":"2024-03-24T00:14:35.620207+0000","last_undegraded":"2024-03-24T00:14:35.620207+0000","last_fullsized":"2024-03-24T00:14:35.620207+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:13:01.267198+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:13:01.267198+0000","last_clean_scrub_stamp":"2024-03-24T00:13:01.267198+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T04:18:09.351579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,1],"acting":[9,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.f","version":"86'2","reported_seq":22,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-24T00:14:07.499518+0000","last_change":"2024-03-24T00:13:01.295874+0000","last_active":"2024-03-24T00:14:07.499518+0000","last_peered":"2024-03-24T00:14:07.499518+0000","last_clean":"2024-03-24T00:14:07.499518+0000","last_became_active":"2024-03-24T00:13:01.295521+0000","last_became_peered":"2024-03-24T00:13:01.295521+0000","last_unstale":"2024-03-24T00:14:07.499518+0000","last_undegraded":"2024-03-24T00:14:07.499518+0000","last_fullsized":"2024-03-24T00:14:07.499518+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:13:00.222790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:13:00.222790+0000","last_clean_scrub_stamp":"2024-03-24T00:13:00.222790+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T06:47:15.749491+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5,4],"acting":[8,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.e","version":"88'5","reported_seq":22,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-24T00:14:35.603168+0000","last_change":"2024-03-24T00:13:02.306617+0000","last_active":"2024-03-24T00:14:35.603168+0000","last_peered":"2024-03-24T00:14:35.603168+0000","last_clean":"2024-03-24T00:14:35.603168+0000","last_became_active":"2024-03-24T00:13:02.306224+0000","last_became_peered":"2024-03-24T00:13:02.306224+0000","last_unstale":"2024-03-24T00:14:35.603168+0000","last_undegraded":"2024-03-24T00:14:35.603168+0000","last_fullsized":"2024-03-24T00:14:35.603168+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:13:01.267198+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:13:01.267198+0000","last_clean_scrub_stamp":"2024-03-24T00:13:01.267198+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T06:49:16.032741+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,4],"acting":[11,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.4","version":"86'1","reported_seq":20,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-24T00:14:07.495528+0000","last_change":"2024-03-24T00:13:01.314428+0000","last_active":"2024-03-24T00:14:07.495528+0000","last_peered":"2024-03-24T00:14:07.495528+0000","last_clean":"2024-03-24T00:14:07.495528+0000","last_became_active":"2024-03-24T00:13:01.313421+0000","last_became_peered":"2024-03-24T00:13:01.313421+0000","last_unstale":"2024-03-24T00:14:07.495528+0000","last_undegraded":"2024-03-24T00:14:07.495528+0000","last_fullsized":"2024-03-24T00:14:07.495528+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:13:00.222790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:13:00.222790+0000","last_clean_scrub_stamp":"2024-03-24T00:13:00.222790+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T08:10:25.796718+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,0],"acting":[9,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.5","version":"88'5","reported_seq":22,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-24T00:14:35.617044+0000","last_change":"2024-03-24T00:13:02.331660+0000","last_active":"2024-03-24T00:14:35.617044+0000","last_peered":"2024-03-24T00:14:35.617044+0000","last_clean":"2024-03-24T00:14:35.617044+0000","last_became_active":"2024-03-24T00:13:02.331288+0000","last_became_peered":"2024-03-24T00:13:02.331288+0000","last_unstale":"2024-03-24T00:14:35.617044+0000","last_undegraded":"2024-03-24T00:14:35.617044+0000","last_fullsized":"2024-03-24T00:14:35.617044+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:13:01.267198+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:13:01.267198+0000","last_clean_scrub_stamp":"2024-03-24T00:13:01.267198+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:36:55.576510+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3,6],"acting":[10,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.2","version":"86'5","reported_seq":114,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-24T00:14:36.565175+0000","last_change":"2024-03-24T00:13:01.310538+0000","last_active":"2024-03-24T00:14:36.565175+0000","last_peered":"2024-03-24T00:14:36.565175+0000","last_clean":"2024-03-24T00:14:36.565175+0000","last_became_active":"2024-03-24T00:13:01.310124+0000","last_became_peered":"2024-03-24T00:13:01.310124+0000","last_unstale":"2024-03-24T00:14:36.565175+0000","last_undegraded":"2024-03-24T00:14:36.565175+0000","last_fullsized":"2024-03-24T00:14:36.565175+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:13:00.222790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:13:00.222790+0000","last_clean_scrub_stamp":"2024-03-24T00:13:00.222790+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T05:47:03.534824+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":588,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":90,"num_read_kb":1,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3","version":"88'7","reported_seq":24,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-24T00:14:35.614536+0000","last_change":"2024-03-24T00:13:02.324117+0000","last_active":"2024-03-24T00:14:35.614536+0000","last_peered":"2024-03-24T00:14:35.614536+0000","last_clean":"2024-03-24T00:14:35.614536+0000","last_became_active":"2024-03-24T00:13:02.323706+0000","last_became_peered":"2024-03-24T00:13:02.323706+0000","last_unstale":"2024-03-24T00:14:35.614536+0000","last_undegraded":"2024-03-24T00:14:35.614536+0000","last_fullsized":"2024-03-24T00:14:35.614536+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:13:01.267198+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:13:01.267198+0000","last_clean_scrub_stamp":"2024-03-24T00:13:01.267198+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T05:34:31.657324+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,4,0],"acting":[10,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.0","version":"88'8","reported_seq":25,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-24T00:14:35.596304+0000","last_change":"2024-03-24T00:13:02.306030+0000","last_active":"2024-03-24T00:14:35.596304+0000","last_peered":"2024-03-24T00:14:35.596304+0000","last_clean":"2024-03-24T00:14:35.596304+0000","last_became_active":"2024-03-24T00:13:02.305588+0000","last_became_peered":"2024-03-24T00:13:02.305588+0000","last_unstale":"2024-03-24T00:14:35.596304+0000","last_undegraded":"2024-03-24T00:14:35.596304+0000","last_fullsized":"2024-03-24T00:14:35.596304+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:13:01.267198+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:13:01.267198+0000","last_clean_scrub_stamp":"2024-03-24T00:13:01.267198+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:20:58.206244+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,4],"acting":[5,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"86'5","reported_seq":113,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-24T00:14:35.460822+0000","last_change":"2024-03-24T00:13:01.318123+0000","last_active":"2024-03-24T00:14:35.460822+0000","last_peered":"2024-03-24T00:14:35.460822+0000","last_clean":"2024-03-24T00:14:35.460822+0000","last_became_active":"2024-03-24T00:13:01.317683+0000","last_became_peered":"2024-03-24T00:13:01.317683+0000","last_unstale":"2024-03-24T00:14:35.460822+0000","last_undegraded":"2024-03-24T00:14:35.460822+0000","last_fullsized":"2024-03-24T00:14:35.460822+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:13:00.222790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:13:00.222790+0000","last_clean_scrub_stamp":"2024-03-24T00:13:00.222790+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T06:33:02.669668+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":89,"num_read_kb":89,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"88'8","reported_seq":25,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-24T00:14:35.626644+0000","last_change":"2024-03-24T00:13:02.328599+0000","last_active":"2024-03-24T00:14:35.626644+0000","last_peered":"2024-03-24T00:14:35.626644+0000","last_clean":"2024-03-24T00:14:35.626644+0000","last_became_active":"2024-03-24T00:13:02.328165+0000","last_became_peered":"2024-03-24T00:13:02.328165+0000","last_unstale":"2024-03-24T00:14:35.626644+0000","last_undegraded":"2024-03-24T00:14:35.626644+0000","last_fullsized":"2024-03-24T00:14:35.626644+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:13:01.267198+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:13:01.267198+0000","last_clean_scrub_stamp":"2024-03-24T00:13:01.267198+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T07:22:39.646578+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"86'6","reported_seq":28,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-24T00:14:07.800384+0000","last_change":"2024-03-24T00:13:01.301932+0000","last_active":"2024-03-24T00:14:07.800384+0000","last_peered":"2024-03-24T00:14:07.800384+0000","last_clean":"2024-03-24T00:14:07.800384+0000","last_became_active":"2024-03-24T00:13:01.301464+0000","last_became_peered":"2024-03-24T00:13:01.301464+0000","last_unstale":"2024-03-24T00:14:07.800384+0000","last_undegraded":"2024-03-24T00:14:07.800384+0000","last_fullsized":"2024-03-24T00:14:07.800384+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:13:00.222790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:13:00.222790+0000","last_clean_scrub_stamp":"2024-03-24T00:13:00.222790+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T08:58:25.512421+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":4,"num_read_kb":4,"num_write":8,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,9],"acting":[4,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.0","version":"21'137","reported_seq":345,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-24T00:14:07.496358+0000","last_change":"2024-03-24T00:11:43.008767+0000","last_active":"2024-03-24T00:14:07.496358+0000","last_peered":"2024-03-24T00:14:07.496358+0000","last_clean":"2024-03-24T00:14:07.496358+0000","last_became_active":"2024-03-24T00:11:43.008364+0000","last_became_peered":"2024-03-24T00:11:43.008364+0000","last_unstale":"2024-03-24T00:14:07.496358+0000","last_undegraded":"2024-03-24T00:14:07.496358+0000","last_fullsized":"2024-03-24T00:14:07.496358+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:07:50.191828+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:07:50.191828+0000","last_clean_scrub_stamp":"2024-03-24T00:07:50.191828+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T01:55:26.549348+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"88'5","reported_seq":22,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-24T00:14:35.621140+0000","last_change":"2024-03-24T00:13:02.325667+0000","last_active":"2024-03-24T00:14:35.621140+0000","last_peered":"2024-03-24T00:14:35.621140+0000","last_clean":"2024-03-24T00:14:35.621140+0000","last_became_active":"2024-03-24T00:13:02.322476+0000","last_became_peered":"2024-03-24T00:13:02.322476+0000","last_unstale":"2024-03-24T00:14:35.621140+0000","last_undegraded":"2024-03-24T00:14:35.621140+0000","last_fullsized":"2024-03-24T00:14:35.621140+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:13:01.267198+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:13:01.267198+0000","last_clean_scrub_stamp":"2024-03-24T00:13:01.267198+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T08:07:19.936708+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"86'4","reported_seq":23,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-24T00:14:07.496426+0000","last_change":"2024-03-24T00:13:01.298082+0000","last_active":"2024-03-24T00:14:07.496426+0000","last_peered":"2024-03-24T00:14:07.496426+0000","last_clean":"2024-03-24T00:14:07.496426+0000","last_became_active":"2024-03-24T00:13:01.297686+0000","last_became_peered":"2024-03-24T00:13:01.297686+0000","last_unstale":"2024-03-24T00:14:07.496426+0000","last_undegraded":"2024-03-24T00:14:07.496426+0000","last_fullsized":"2024-03-24T00:14:07.496426+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:13:00.222790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:13:00.222790+0000","last_clean_scrub_stamp":"2024-03-24T00:13:00.222790+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T09:25:07.204076+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.5","version":"86'4","reported_seq":24,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-24T00:14:07.499345+0000","last_change":"2024-03-24T00:13:01.321250+0000","last_active":"2024-03-24T00:14:07.499345+0000","last_peered":"2024-03-24T00:14:07.499345+0000","last_clean":"2024-03-24T00:14:07.499345+0000","last_became_active":"2024-03-24T00:13:01.320929+0000","last_became_peered":"2024-03-24T00:13:01.320929+0000","last_unstale":"2024-03-24T00:14:07.499345+0000","last_undegraded":"2024-03-24T00:14:07.499345+0000","last_fullsized":"2024-03-24T00:14:07.499345+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:13:00.222790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:13:00.222790+0000","last_clean_scrub_stamp":"2024-03-24T00:13:00.222790+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T05:22:05.490303+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,9],"acting":[8,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.4","version":"88'9","reported_seq":26,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-24T00:14:35.621019+0000","last_change":"2024-03-24T00:13:02.323413+0000","last_active":"2024-03-24T00:14:35.621019+0000","last_peered":"2024-03-24T00:14:35.621019+0000","last_clean":"2024-03-24T00:14:35.621019+0000","last_became_active":"2024-03-24T00:13:02.322947+0000","last_became_peered":"2024-03-24T00:13:02.322947+0000","last_unstale":"2024-03-24T00:14:35.621019+0000","last_undegraded":"2024-03-24T00:14:35.621019+0000","last_fullsized":"2024-03-24T00:14:35.621019+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:13:01.267198+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:13:01.267198+0000","last_clean_scrub_stamp":"2024-03-24T00:13:01.267198+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T00:16:14.490270+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,9],"acting":[6,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.6","version":"88'12","reported_seq":31,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-24T00:14:07.496543+0000","last_change":"2024-03-24T00:13:01.320380+0000","last_active":"2024-03-24T00:14:07.496543+0000","last_peered":"2024-03-24T00:14:07.496543+0000","last_clean":"2024-03-24T00:14:07.496543+0000","last_became_active":"2024-03-24T00:13:01.319192+0000","last_became_peered":"2024-03-24T00:13:01.319192+0000","last_unstale":"2024-03-24T00:14:07.496543+0000","last_undegraded":"2024-03-24T00:14:07.496543+0000","last_fullsized":"2024-03-24T00:14:07.496543+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:13:00.222790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:13:00.222790+0000","last_clean_scrub_stamp":"2024-03-24T00:13:00.222790+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T02:09:27.085298+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,3],"acting":[9,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.7","version":"88'5","reported_seq":22,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-24T00:14:35.613666+0000","last_change":"2024-03-24T00:13:02.328067+0000","last_active":"2024-03-24T00:14:35.613666+0000","last_peered":"2024-03-24T00:14:35.613666+0000","last_clean":"2024-03-24T00:14:35.613666+0000","last_became_active":"2024-03-24T00:13:02.327468+0000","last_became_peered":"2024-03-24T00:13:02.327468+0000","last_unstale":"2024-03-24T00:14:35.613666+0000","last_undegraded":"2024-03-24T00:14:35.613666+0000","last_fullsized":"2024-03-24T00:14:35.613666+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:13:01.267198+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:13:01.267198+0000","last_clean_scrub_stamp":"2024-03-24T00:13:01.267198+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T01:21:58.596941+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,7],"acting":[3,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"86'7","reported_seq":27,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-24T00:14:07.800125+0000","last_change":"2024-03-24T00:13:01.309189+0000","last_active":"2024-03-24T00:14:07.800125+0000","last_peered":"2024-03-24T00:14:07.800125+0000","last_clean":"2024-03-24T00:14:07.800125+0000","last_became_active":"2024-03-24T00:13:01.308871+0000","last_became_peered":"2024-03-24T00:13:01.308871+0000","last_unstale":"2024-03-24T00:14:07.800125+0000","last_undegraded":"2024-03-24T00:14:07.800125+0000","last_fullsized":"2024-03-24T00:14:07.800125+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:13:00.222790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:13:00.222790+0000","last_clean_scrub_stamp":"2024-03-24T00:13:00.222790+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T01:00:32.610036+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,6],"acting":[4,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"88'7","reported_seq":24,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-24T00:14:35.614946+0000","last_change":"2024-03-24T00:13:02.332238+0000","last_active":"2024-03-24T00:14:35.614946+0000","last_peered":"2024-03-24T00:14:35.614946+0000","last_clean":"2024-03-24T00:14:35.614946+0000","last_became_active":"2024-03-24T00:13:02.331909+0000","last_became_peered":"2024-03-24T00:13:02.331909+0000","last_unstale":"2024-03-24T00:14:35.614946+0000","last_undegraded":"2024-03-24T00:14:35.614946+0000","last_fullsized":"2024-03-24T00:14:35.614946+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:13:01.267198+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:13:01.267198+0000","last_clean_scrub_stamp":"2024-03-24T00:13:01.267198+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T04:41:14.762251+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"86'2","reported_seq":21,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-24T00:14:07.494700+0000","last_change":"2024-03-24T00:13:01.306883+0000","last_active":"2024-03-24T00:14:07.494700+0000","last_peered":"2024-03-24T00:14:07.494700+0000","last_clean":"2024-03-24T00:14:07.494700+0000","last_became_active":"2024-03-24T00:13:01.306629+0000","last_became_peered":"2024-03-24T00:13:01.306629+0000","last_unstale":"2024-03-24T00:14:07.494700+0000","last_undegraded":"2024-03-24T00:14:07.494700+0000","last_fullsized":"2024-03-24T00:14:07.494700+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:13:00.222790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:13:00.222790+0000","last_clean_scrub_stamp":"2024-03-24T00:13:00.222790+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T08:11:52.398783+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"88'2","reported_seq":19,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-24T00:14:35.584250+0000","last_change":"2024-03-24T00:13:02.308421+0000","last_active":"2024-03-24T00:14:35.584250+0000","last_peered":"2024-03-24T00:14:35.584250+0000","last_clean":"2024-03-24T00:14:35.584250+0000","last_became_active":"2024-03-24T00:13:02.308081+0000","last_became_peered":"2024-03-24T00:13:02.308081+0000","last_unstale":"2024-03-24T00:14:35.584250+0000","last_undegraded":"2024-03-24T00:14:35.584250+0000","last_fullsized":"2024-03-24T00:14:35.584250+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:13:01.267198+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:13:01.267198+0000","last_clean_scrub_stamp":"2024-03-24T00:13:01.267198+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T06:50:42.356293+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,10],"acting":[8,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.9","version":"86'7","reported_seq":26,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-24T00:14:07.508158+0000","last_change":"2024-03-24T00:13:01.324439+0000","last_active":"2024-03-24T00:14:07.508158+0000","last_peered":"2024-03-24T00:14:07.508158+0000","last_clean":"2024-03-24T00:14:07.508158+0000","last_became_active":"2024-03-24T00:13:01.324037+0000","last_became_peered":"2024-03-24T00:13:01.324037+0000","last_unstale":"2024-03-24T00:14:07.508158+0000","last_undegraded":"2024-03-24T00:14:07.508158+0000","last_fullsized":"2024-03-24T00:14:07.508158+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:13:00.222790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:13:00.222790+0000","last_clean_scrub_stamp":"2024-03-24T00:13:00.222790+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T01:37:56.767931+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,10],"acting":[3,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.8","version":"88'9","reported_seq":26,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-24T00:14:35.605813+0000","last_change":"2024-03-24T00:13:02.313740+0000","last_active":"2024-03-24T00:14:35.605813+0000","last_peered":"2024-03-24T00:14:35.605813+0000","last_clean":"2024-03-24T00:14:35.605813+0000","last_became_active":"2024-03-24T00:13:02.313401+0000","last_became_peered":"2024-03-24T00:13:02.313401+0000","last_unstale":"2024-03-24T00:14:35.605813+0000","last_undegraded":"2024-03-24T00:14:35.605813+0000","last_fullsized":"2024-03-24T00:14:35.605813+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:13:01.267198+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:13:01.267198+0000","last_clean_scrub_stamp":"2024-03-24T00:13:01.267198+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T04:16:52.569135+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.a","version":"86'3","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-24T00:14:07.514496+0000","last_change":"2024-03-24T00:14:07.514050+0000","last_active":"2024-03-24T00:14:07.514496+0000","last_peered":"2024-03-24T00:14:07.514496+0000","last_clean":"2024-03-24T00:14:07.514496+0000","last_became_active":"2024-03-24T00:14:07.509219+0000","last_became_peered":"2024-03-24T00:14:07.509219+0000","last_unstale":"2024-03-24T00:14:07.514496+0000","last_undegraded":"2024-03-24T00:14:07.514496+0000","last_fullsized":"2024-03-24T00:14:07.514496+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:13:00.222790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:13:00.222790+0000","last_clean_scrub_stamp":"2024-03-24T00:13:00.222790+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T08:05:33.888846+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,3,8],"acting":[11,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.b","version":"88'8","reported_seq":25,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-24T00:14:35.617423+0000","last_change":"2024-03-24T00:13:02.337413+0000","last_active":"2024-03-24T00:14:35.617423+0000","last_peered":"2024-03-24T00:14:35.617423+0000","last_clean":"2024-03-24T00:14:35.617423+0000","last_became_active":"2024-03-24T00:13:02.336910+0000","last_became_peered":"2024-03-24T00:13:02.336910+0000","last_unstale":"2024-03-24T00:14:35.617423+0000","last_undegraded":"2024-03-24T00:14:35.617423+0000","last_fullsized":"2024-03-24T00:14:35.617423+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:13:01.267198+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:13:01.267198+0000","last_clean_scrub_stamp":"2024-03-24T00:13:01.267198+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T10:51:06.549979+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,10],"acting":[3,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.b","version":"86'4","reported_seq":24,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-24T00:14:07.499164+0000","last_change":"2024-03-24T00:13:01.300201+0000","last_active":"2024-03-24T00:14:07.499164+0000","last_peered":"2024-03-24T00:14:07.499164+0000","last_clean":"2024-03-24T00:14:07.499164+0000","last_became_active":"2024-03-24T00:13:01.299750+0000","last_became_peered":"2024-03-24T00:13:01.299750+0000","last_unstale":"2024-03-24T00:14:07.499164+0000","last_undegraded":"2024-03-24T00:14:07.499164+0000","last_fullsized":"2024-03-24T00:14:07.499164+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:13:00.222790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:13:00.222790+0000","last_clean_scrub_stamp":"2024-03-24T00:13:00.222790+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T01:37:32.768926+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.a","version":"88'5","reported_seq":22,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-24T00:14:35.622160+0000","last_change":"2024-03-24T00:13:02.331732+0000","last_active":"2024-03-24T00:14:35.622160+0000","last_peered":"2024-03-24T00:14:35.622160+0000","last_clean":"2024-03-24T00:14:35.622160+0000","last_became_active":"2024-03-24T00:13:02.331365+0000","last_became_peered":"2024-03-24T00:13:02.331365+0000","last_unstale":"2024-03-24T00:14:35.622160+0000","last_undegraded":"2024-03-24T00:14:35.622160+0000","last_fullsized":"2024-03-24T00:14:35.622160+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:13:01.267198+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:13:01.267198+0000","last_clean_scrub_stamp":"2024-03-24T00:13:01.267198+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:27:53.598158+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.c","version":"86'6","reported_seq":113,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-24T00:14:33.424217+0000","last_change":"2024-03-24T00:13:01.308725+0000","last_active":"2024-03-24T00:14:33.424217+0000","last_peered":"2024-03-24T00:14:33.424217+0000","last_clean":"2024-03-24T00:14:33.424217+0000","last_became_active":"2024-03-24T00:13:01.308324+0000","last_became_peered":"2024-03-24T00:13:01.308324+0000","last_unstale":"2024-03-24T00:14:33.424217+0000","last_undegraded":"2024-03-24T00:14:33.424217+0000","last_fullsized":"2024-03-24T00:14:33.424217+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:13:00.222790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:13:00.222790+0000","last_clean_scrub_stamp":"2024-03-24T00:13:00.222790+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T07:43:03.691683+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1561,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":88,"num_read_kb":2,"num_write":6,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,11,0],"acting":[2,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"88'7","reported_seq":24,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-24T00:14:35.616970+0000","last_change":"2024-03-24T00:13:02.305749+0000","last_active":"2024-03-24T00:14:35.616970+0000","last_peered":"2024-03-24T00:14:35.616970+0000","last_clean":"2024-03-24T00:14:35.616970+0000","last_became_active":"2024-03-24T00:13:02.305413+0000","last_became_peered":"2024-03-24T00:13:02.305413+0000","last_unstale":"2024-03-24T00:14:35.616970+0000","last_undegraded":"2024-03-24T00:14:35.616970+0000","last_fullsized":"2024-03-24T00:14:35.616970+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:13:01.267198+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:13:01.267198+0000","last_clean_scrub_stamp":"2024-03-24T00:13:01.267198+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T00:54:56.437725+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,11],"acting":[7,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.d","version":"86'3","reported_seq":23,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-24T00:14:07.680179+0000","last_change":"2024-03-24T00:13:01.304883+0000","last_active":"2024-03-24T00:14:07.680179+0000","last_peered":"2024-03-24T00:14:07.680179+0000","last_clean":"2024-03-24T00:14:07.680179+0000","last_became_active":"2024-03-24T00:13:01.304459+0000","last_became_peered":"2024-03-24T00:13:01.304459+0000","last_unstale":"2024-03-24T00:14:07.680179+0000","last_undegraded":"2024-03-24T00:14:07.680179+0000","last_fullsized":"2024-03-24T00:14:07.680179+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:13:00.222790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:13:00.222790+0000","last_clean_scrub_stamp":"2024-03-24T00:13:00.222790+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T06:14:22.473571+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10,3],"acting":[5,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.c","version":"88'7","reported_seq":24,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-24T00:14:35.613996+0000","last_change":"2024-03-24T00:13:02.331923+0000","last_active":"2024-03-24T00:14:35.613996+0000","last_peered":"2024-03-24T00:14:35.613996+0000","last_clean":"2024-03-24T00:14:35.613996+0000","last_became_active":"2024-03-24T00:13:02.331504+0000","last_became_peered":"2024-03-24T00:13:02.331504+0000","last_unstale":"2024-03-24T00:14:35.613996+0000","last_undegraded":"2024-03-24T00:14:35.613996+0000","last_fullsized":"2024-03-24T00:14:35.613996+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:13:01.267198+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:13:01.267198+0000","last_clean_scrub_stamp":"2024-03-24T00:13:01.267198+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T03:27:17.160248+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,5],"acting":[7,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"86'5","reported_seq":24,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-24T00:14:07.632695+0000","last_change":"2024-03-24T00:13:01.320223+0000","last_active":"2024-03-24T00:14:07.632695+0000","last_peered":"2024-03-24T00:14:07.632695+0000","last_clean":"2024-03-24T00:14:07.632695+0000","last_became_active":"2024-03-24T00:13:01.319342+0000","last_became_peered":"2024-03-24T00:13:01.319342+0000","last_unstale":"2024-03-24T00:14:07.632695+0000","last_undegraded":"2024-03-24T00:14:07.632695+0000","last_fullsized":"2024-03-24T00:14:07.632695+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:13:00.222790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:13:00.222790+0000","last_clean_scrub_stamp":"2024-03-24T00:13:00.222790+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T02:24:08.018521+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.f","version":"88'9","reported_seq":26,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-24T00:14:35.606455+0000","last_change":"2024-03-24T00:13:02.307719+0000","last_active":"2024-03-24T00:14:35.606455+0000","last_peered":"2024-03-24T00:14:35.606455+0000","last_clean":"2024-03-24T00:14:35.606455+0000","last_became_active":"2024-03-24T00:13:02.307399+0000","last_became_peered":"2024-03-24T00:13:02.307399+0000","last_unstale":"2024-03-24T00:14:35.606455+0000","last_undegraded":"2024-03-24T00:14:35.606455+0000","last_fullsized":"2024-03-24T00:14:35.606455+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:13:01.267198+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:13:01.267198+0000","last_clean_scrub_stamp":"2024-03-24T00:13:01.267198+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T09:38:13.077843+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,8],"acting":[7,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.20","version":"88'9","reported_seq":26,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-24T00:14:35.607423+0000","last_change":"2024-03-24T00:13:02.307469+0000","last_active":"2024-03-24T00:14:35.607423+0000","last_peered":"2024-03-24T00:14:35.607423+0000","last_clean":"2024-03-24T00:14:35.607423+0000","last_became_active":"2024-03-24T00:13:02.306873+0000","last_became_peered":"2024-03-24T00:13:02.306873+0000","last_unstale":"2024-03-24T00:14:35.607423+0000","last_undegraded":"2024-03-24T00:14:35.607423+0000","last_fullsized":"2024-03-24T00:14:35.607423+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:13:01.267198+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:13:01.267198+0000","last_clean_scrub_stamp":"2024-03-24T00:13:01.267198+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:19:51.163015+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,5],"acting":[11,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.21","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-24T00:14:07.494967+0000","last_change":"2024-03-24T00:13:02.307472+0000","last_active":"2024-03-24T00:14:07.494967+0000","last_peered":"2024-03-24T00:14:07.494967+0000","last_clean":"2024-03-24T00:14:07.494967+0000","last_became_active":"2024-03-24T00:13:02.307099+0000","last_became_peered":"2024-03-24T00:13:02.307099+0000","last_unstale":"2024-03-24T00:14:07.494967+0000","last_undegraded":"2024-03-24T00:14:07.494967+0000","last_fullsized":"2024-03-24T00:14:07.494967+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:13:01.267198+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:13:01.267198+0000","last_clean_scrub_stamp":"2024-03-24T00:13:01.267198+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T08:49:06.461235+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,4],"acting":[2,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-24T00:14:07.494954+0000","last_change":"2024-03-24T00:13:02.331133+0000","last_active":"2024-03-24T00:14:07.494954+0000","last_peered":"2024-03-24T00:14:07.494954+0000","last_clean":"2024-03-24T00:14:07.494954+0000","last_became_active":"2024-03-24T00:13:02.330780+0000","last_became_peered":"2024-03-24T00:13:02.330780+0000","last_unstale":"2024-03-24T00:14:07.494954+0000","last_undegraded":"2024-03-24T00:14:07.494954+0000","last_fullsized":"2024-03-24T00:14:07.494954+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:13:01.267198+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:13:01.267198+0000","last_clean_scrub_stamp":"2024-03-24T00:13:01.267198+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T10:46:07.933131+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.23","version":"88'7","reported_seq":24,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-24T00:14:35.623964+0000","last_change":"2024-03-24T00:13:02.335406+0000","last_active":"2024-03-24T00:14:35.623964+0000","last_peered":"2024-03-24T00:14:35.623964+0000","last_clean":"2024-03-24T00:14:35.623964+0000","last_became_active":"2024-03-24T00:13:02.334994+0000","last_became_peered":"2024-03-24T00:13:02.334994+0000","last_unstale":"2024-03-24T00:14:35.623964+0000","last_undegraded":"2024-03-24T00:14:35.623964+0000","last_fullsized":"2024-03-24T00:14:35.623964+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:13:01.267198+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:13:01.267198+0000","last_clean_scrub_stamp":"2024-03-24T00:13:01.267198+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T03:24:59.547395+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,4],"acting":[0,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"88'4","reported_seq":21,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-24T00:14:35.621871+0000","last_change":"2024-03-24T00:13:02.332420+0000","last_active":"2024-03-24T00:14:35.621871+0000","last_peered":"2024-03-24T00:14:35.621871+0000","last_clean":"2024-03-24T00:14:35.621871+0000","last_became_active":"2024-03-24T00:13:02.331496+0000","last_became_peered":"2024-03-24T00:13:02.331496+0000","last_unstale":"2024-03-24T00:14:35.621871+0000","last_undegraded":"2024-03-24T00:14:35.621871+0000","last_fullsized":"2024-03-24T00:14:35.621871+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:13:01.267198+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:13:01.267198+0000","last_clean_scrub_stamp":"2024-03-24T00:13:01.267198+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T03:15:06.007947+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"88'6","reported_seq":23,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-24T00:14:35.604153+0000","last_change":"2024-03-24T00:13:02.309077+0000","last_active":"2024-03-24T00:14:35.604153+0000","last_peered":"2024-03-24T00:14:35.604153+0000","last_clean":"2024-03-24T00:14:35.604153+0000","last_became_active":"2024-03-24T00:13:02.308752+0000","last_became_peered":"2024-03-24T00:13:02.308752+0000","last_unstale":"2024-03-24T00:14:35.604153+0000","last_undegraded":"2024-03-24T00:14:35.604153+0000","last_fullsized":"2024-03-24T00:14:35.604153+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:13:01.267198+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:13:01.267198+0000","last_clean_scrub_stamp":"2024-03-24T00:13:01.267198+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T06:49:19.657988+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,5],"acting":[11,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.26","version":"0'0","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-24T00:14:07.495426+0000","last_change":"2024-03-24T00:13:02.304962+0000","last_active":"2024-03-24T00:14:07.495426+0000","last_peered":"2024-03-24T00:14:07.495426+0000","last_clean":"2024-03-24T00:14:07.495426+0000","last_became_active":"2024-03-24T00:13:02.304580+0000","last_became_peered":"2024-03-24T00:13:02.304580+0000","last_unstale":"2024-03-24T00:14:07.495426+0000","last_undegraded":"2024-03-24T00:14:07.495426+0000","last_fullsized":"2024-03-24T00:14:07.495426+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:13:01.267198+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:13:01.267198+0000","last_clean_scrub_stamp":"2024-03-24T00:13:01.267198+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:45:26.779169+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8,1],"acting":[2,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.27","version":"88'2","reported_seq":19,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-24T00:14:35.612439+0000","last_change":"2024-03-24T00:13:02.321908+0000","last_active":"2024-03-24T00:14:35.612439+0000","last_peered":"2024-03-24T00:14:35.612439+0000","last_clean":"2024-03-24T00:14:35.612439+0000","last_became_active":"2024-03-24T00:13:02.321495+0000","last_became_peered":"2024-03-24T00:13:02.321495+0000","last_unstale":"2024-03-24T00:14:35.612439+0000","last_undegraded":"2024-03-24T00:14:35.612439+0000","last_fullsized":"2024-03-24T00:14:35.612439+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:13:01.267198+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:13:01.267198+0000","last_clean_scrub_stamp":"2024-03-24T00:13:01.267198+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T08:43:29.261314+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"88'6","reported_seq":23,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-24T00:14:35.621983+0000","last_change":"2024-03-24T00:13:02.331966+0000","last_active":"2024-03-24T00:14:35.621983+0000","last_peered":"2024-03-24T00:14:35.621983+0000","last_clean":"2024-03-24T00:14:35.621983+0000","last_became_active":"2024-03-24T00:13:02.331633+0000","last_became_peered":"2024-03-24T00:13:02.331633+0000","last_unstale":"2024-03-24T00:14:35.621983+0000","last_undegraded":"2024-03-24T00:14:35.621983+0000","last_fullsized":"2024-03-24T00:14:35.621983+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:13:01.267198+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:13:01.267198+0000","last_clean_scrub_stamp":"2024-03-24T00:13:01.267198+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T05:04:01.574351+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,3],"acting":[5,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"88'8","reported_seq":25,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-24T00:14:35.617213+0000","last_change":"2024-03-24T00:13:02.330344+0000","last_active":"2024-03-24T00:14:35.617213+0000","last_peered":"2024-03-24T00:14:35.617213+0000","last_clean":"2024-03-24T00:14:35.617213+0000","last_became_active":"2024-03-24T00:13:02.328218+0000","last_became_peered":"2024-03-24T00:13:02.328218+0000","last_unstale":"2024-03-24T00:14:35.617213+0000","last_undegraded":"2024-03-24T00:14:35.617213+0000","last_fullsized":"2024-03-24T00:14:35.617213+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:13:01.267198+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:13:01.267198+0000","last_clean_scrub_stamp":"2024-03-24T00:13:01.267198+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T08:54:36.392397+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,11],"acting":[3,2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"88'10","reported_seq":27,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-24T00:14:35.627624+0000","last_change":"2024-03-24T00:13:02.330956+0000","last_active":"2024-03-24T00:14:35.627624+0000","last_peered":"2024-03-24T00:14:35.627624+0000","last_clean":"2024-03-24T00:14:35.627624+0000","last_became_active":"2024-03-24T00:13:02.330642+0000","last_became_peered":"2024-03-24T00:13:02.330642+0000","last_unstale":"2024-03-24T00:14:35.627624+0000","last_undegraded":"2024-03-24T00:14:35.627624+0000","last_fullsized":"2024-03-24T00:14:35.627624+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:13:01.267198+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:13:01.267198+0000","last_clean_scrub_stamp":"2024-03-24T00:13:01.267198+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:41:11.326722+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2b","version":"88'11","reported_seq":28,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-24T00:14:35.623864+0000","last_change":"2024-03-24T00:13:02.333026+0000","last_active":"2024-03-24T00:14:35.623864+0000","last_peered":"2024-03-24T00:14:35.623864+0000","last_clean":"2024-03-24T00:14:35.623864+0000","last_became_active":"2024-03-24T00:13:02.331232+0000","last_became_peered":"2024-03-24T00:13:02.331232+0000","last_unstale":"2024-03-24T00:14:35.623864+0000","last_undegraded":"2024-03-24T00:14:35.623864+0000","last_fullsized":"2024-03-24T00:14:35.623864+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:13:01.267198+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:13:01.267198+0000","last_clean_scrub_stamp":"2024-03-24T00:13:01.267198+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T02:14:18.504012+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2c","version":"88'8","reported_seq":25,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-24T00:14:35.602677+0000","last_change":"2024-03-24T00:13:02.309023+0000","last_active":"2024-03-24T00:14:35.602677+0000","last_peered":"2024-03-24T00:14:35.602677+0000","last_clean":"2024-03-24T00:14:35.602677+0000","last_became_active":"2024-03-24T00:13:02.308236+0000","last_became_peered":"2024-03-24T00:13:02.308236+0000","last_unstale":"2024-03-24T00:14:35.602677+0000","last_undegraded":"2024-03-24T00:14:35.602677+0000","last_fullsized":"2024-03-24T00:14:35.602677+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:13:01.267198+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:13:01.267198+0000","last_clean_scrub_stamp":"2024-03-24T00:13:01.267198+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T03:30:37.427095+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,8],"acting":[11,4,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":234,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":702,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":460,"ondisk_log_size":460,"up":192,"acting":192,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":442368,"data_stored":4722944,"data_compressed":284592,"data_compressed_allocated":360448,"data_compressed_original":4669440,"omap_allocated":0,"internal_metadata":0},"log_size":137,"ondisk_log_size":137,"up":3,"acting":3,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":69295,"num_objects":98,"num_object_clones":0,"num_object_copies":294,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":98,"num_whiteouts":0,"num_read":938,"num_read_kb":545,"num_write":245,"num_write_kb":142,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":62373,"num_keys_recovered":0,"num_objects_omap":66,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":512000,"data_stored":208026,"data_compressed":13143,"data_compressed_allocated":20480,"data_compressed_original":73728,"omap_allocated":0,"internal_metadata":0},"log_size":214,"ondisk_log_size":214,"up":96,"acting":96,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":81,"seq":347892351009,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27952,"kb_used_data":936,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717200,"statfs":{"total":95995035648,"available":95966412800,"internally_reserved":0,"allocated":958464,"data_stored":461088,"data_compressed":5786,"data_compressed_allocated":8192,"data_compressed_original":16384,"omap_allocated":1596,"internal_metadata":27654596},"hb_peers":[0,1,2,3,4,5,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":10,"up_from":73,"seq":313532612646,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28100,"kb_used_data":1084,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717052,"statfs":{"total":95995035648,"available":95966261248,"internally_reserved":0,"allocated":1110016,"data_stored":1113535,"data_compressed":43694,"data_compressed_allocated":57344,"data_compressed_original":614400,"omap_allocated":1599,"internal_metadata":27654593},"hb_peers":[0,1,2,3,4,5,6,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":9,"up_from":68,"seq":292057776171,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32136,"kb_used_data":960,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713016,"statfs":{"total":95995035648,"available":95962128384,"internally_reserved":0,"allocated":983040,"data_stored":460486,"data_compressed":5786,"data_compressed_allocated":8192,"data_compressed_original":16384,"omap_allocated":1600,"internal_metadata":31914432},"hb_peers":[0,1,2,3,4,5,6,7,8,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":8,"up_from":60,"seq":257698037810,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36284,"kb_used_data":948,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708868,"statfs":{"total":95995035648,"available":95957880832,"internally_reserved":0,"allocated":970752,"data_stored":460192,"data_compressed":5786,"data_compressed_allocated":8192,"data_compressed_original":16384,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":7,"up_from":52,"seq":223338299448,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28024,"kb_used_data":1008,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717128,"statfs":{"total":95995035648,"available":95966339072,"internally_reserved":0,"allocated":1032192,"data_stored":1053523,"data_compressed":41975,"data_compressed_allocated":53248,"data_compressed_original":606208,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":6,"up_from":47,"seq":201863462974,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27960,"kb_used_data":944,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717192,"statfs":{"total":95995035648,"available":95966404608,"internally_reserved":0,"allocated":966656,"data_stored":460203,"data_compressed":5786,"data_compressed_allocated":8192,"data_compressed_original":16384,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":12,"apply_latency_ms":12,"commit_latency_ns":12000000,"apply_latency_ns":12000000},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574937,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28024,"kb_used_data":1008,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717128,"statfs":{"total":95995035648,"available":95966339072,"internally_reserved":0,"allocated":1032192,"data_stored":523738,"data_compressed":7505,"data_compressed_allocated":12288,"data_compressed_original":24576,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738463,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27988,"kb_used_data":964,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717164,"statfs":{"total":95995035648,"available":95966375936,"internally_reserved":0,"allocated":987136,"data_stored":461933,"data_compressed":5786,"data_compressed_allocated":8192,"data_compressed_original":16384,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411412,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27976,"kb_used_data":956,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717176,"statfs":{"total":95995035648,"available":95966388224,"internally_reserved":0,"allocated":978944,"data_stored":462118,"data_compressed":5786,"data_compressed_allocated":8192,"data_compressed_original":16384,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":26,"seq":111669149775,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32136,"kb_used_data":960,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713016,"statfs":{"total":95995035648,"available":95962128384,"internally_reserved":0,"allocated":983040,"data_stored":460028,"data_compressed":5786,"data_compressed_allocated":8192,"data_compressed_original":16384,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":11,"apply_latency_ms":11,"commit_latency_ns":11000000,"apply_latency_ns":11000000},"alerts":[]},{"osd":4,"up_from":34,"seq":146028888137,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32144,"kb_used_data":968,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713008,"statfs":{"total":95995035648,"available":95962120192,"internally_reserved":0,"allocated":991232,"data_stored":522014,"data_compressed":15491,"data_compressed_allocated":20480,"data_compressed_original":73728,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":5,"up_from":39,"seq":167503724611,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28036,"kb_used_data":1020,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717116,"statfs":{"total":95995035648,"available":95966326784,"internally_reserved":0,"allocated":1044480,"data_stored":1053201,"data_compressed":41975,"data_compressed_allocated":53248,"data_compressed_original":606208,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34549,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34549,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34549,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":36189,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":36189,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":36189,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":36189,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":36189,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":2710,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":86016,"data_stored":64374,"data_compressed":1719,"data_compressed_allocated":4096,"data_compressed_original":8192,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":2754,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":430,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":62650,"data_compressed":9705,"data_compressed_allocated":12288,"data_compressed_original":57344,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":3352,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":722,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":3674,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":828,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":1122,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":110592,"data_stored":63686,"data_compressed":1719,"data_compressed_allocated":4096,"data_compressed_original":8192,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1724,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":234,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-24T00:14:40.763 DEBUG:teuthology.orchestra.run.smithi118:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 shell --fsid d79a5420-e971-11ee-95cd-87774f69a715 -- ceph pg dump --format=json 2024-03-24T00:14:40.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:14:40 smithi118 ceph-mon[31535]: pgmap v365: 97 pgs: 97 active+clean; 644 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 6.0 KiB/s rd, 511 B/s wr, 12 op/s 2024-03-24T00:14:41.203 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:14:40 smithi189 ceph-mon[36187]: pgmap v365: 97 pgs: 97 active+clean; 644 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 6.0 KiB/s rd, 511 B/s wr, 12 op/s 2024-03-24T00:14:41.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:14:40 smithi193 ceph-mon[36878]: pgmap v365: 97 pgs: 97 active+clean; 644 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 6.0 KiB/s rd, 511 B/s wr, 12 op/s 2024-03-24T00:14:41.261 INFO:teuthology.orchestra.run.smithi118.stderr:Inferring config /var/lib/ceph/d79a5420-e971-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T00:14:42.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:14:41 smithi118 ceph-mon[31535]: from='client.24883 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-24T00:14:42.203 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:14:41 smithi189 ceph-mon[36187]: from='client.24883 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-24T00:14:42.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:14:41 smithi193 ceph-mon[36878]: from='client.24883 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-24T00:14:43.172 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:14:42 smithi118 ceph-mon[31535]: pgmap v366: 97 pgs: 97 active+clean; 645 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 597 B/s wr, 9 op/s 2024-03-24T00:14:43.203 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:14:42 smithi189 ceph-mon[36187]: pgmap v366: 97 pgs: 97 active+clean; 645 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 597 B/s wr, 9 op/s 2024-03-24T00:14:43.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:14:42 smithi193 ceph-mon[36878]: pgmap v366: 97 pgs: 97 active+clean; 645 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 597 B/s wr, 9 op/s 2024-03-24T00:14:43.776 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:43.774+0000 7fd0b1000700 1 -- 172.21.15.118:0/2473790943 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fd0ac06a380 msgr2=0x7fd0ac06a7e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:14:43.776 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:43.774+0000 7fd0b1000700 1 --2- 172.21.15.118:0/2473790943 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fd0ac06a380 0x7fd0ac06a7e0 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7fd098009990 tx=0x7fd098009ca0 comp rx=0 tx=0).stop 2024-03-24T00:14:43.776 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:43.774+0000 7fd0b1000700 1 -- 172.21.15.118:0/2473790943 shutdown_connections 2024-03-24T00:14:43.776 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:43.774+0000 7fd0b1000700 1 --2- 172.21.15.118:0/2473790943 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd0ac107080 0x7fd0ac069e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:14:43.776 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:43.774+0000 7fd0b1000700 1 --2- 172.21.15.118:0/2473790943 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fd0ac104b90 0x7fd0ac105020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:14:43.776 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:43.774+0000 7fd0b1000700 1 --2- 172.21.15.118:0/2473790943 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fd0ac06a380 0x7fd0ac06a7e0 unknown :-1 s=CLOSED pgs=125 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:14:43.776 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:43.774+0000 7fd0b1000700 1 -- 172.21.15.118:0/2473790943 >> 172.21.15.118:0/2473790943 conn(0x7fd0ac077fc0 msgr2=0x7fd0ac0783d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:14:43.776 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:43.774+0000 7fd0b1000700 1 -- 172.21.15.118:0/2473790943 shutdown_connections 2024-03-24T00:14:43.776 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:43.775+0000 7fd0b1000700 1 -- 172.21.15.118:0/2473790943 wait complete. 2024-03-24T00:14:43.778 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:43.776+0000 7fd0b1000700 1 Processor -- start 2024-03-24T00:14:43.778 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:43.776+0000 7fd0b1000700 1 -- start start 2024-03-24T00:14:43.778 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:43.776+0000 7fd0b1000700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd0ac06a380 0x7fd0ac19efb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:14:43.778 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:43.776+0000 7fd0b1000700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fd0ac104b90 0x7fd0ac19f4f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:14:43.778 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:43.776+0000 7fd0b1000700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fd0ac107080 0x7fd0ac1a3880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:14:43.778 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:43.776+0000 7fd0b1000700 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd0ac118a80 con 0x7fd0ac104b90 2024-03-24T00:14:43.779 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:43.776+0000 7fd0b1000700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd0ac118900 con 0x7fd0ac06a380 2024-03-24T00:14:43.779 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:43.776+0000 7fd0b1000700 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd0ac118c00 con 0x7fd0ac107080 2024-03-24T00:14:43.779 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:43.776+0000 7fd0a9e00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd0ac06a380 0x7fd0ac19efb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:14:43.779 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:43.776+0000 7fd0a9e00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd0ac06a380 0x7fd0ac19efb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.118:55614/0 (socket says 172.21.15.118:55614) 2024-03-24T00:14:43.779 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:43.776+0000 7fd0a9e00700 1 -- 172.21.15.118:0/2709811081 learned_addr learned my addr 172.21.15.118:0/2709811081 (peer_addr_for_me v2:172.21.15.118:0/0) 2024-03-24T00:14:43.779 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:43.776+0000 7fd0aa800700 1 --2- 172.21.15.118:0/2709811081 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fd0ac107080 0x7fd0ac1a3880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:14:43.780 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:43.777+0000 7fd0a9400700 1 --2- 172.21.15.118:0/2709811081 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fd0ac104b90 0x7fd0ac19f4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:14:43.780 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:43.777+0000 7fd0a9e00700 1 -- 172.21.15.118:0/2709811081 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fd0ac107080 msgr2=0x7fd0ac1a3880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:14:43.780 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:43.777+0000 7fd0a9e00700 1 --2- 172.21.15.118:0/2709811081 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fd0ac107080 0x7fd0ac1a3880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:14:43.780 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:43.777+0000 7fd0a9e00700 1 -- 172.21.15.118:0/2709811081 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fd0ac104b90 msgr2=0x7fd0ac19f4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:14:43.780 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:43.777+0000 7fd0a9e00700 1 --2- 172.21.15.118:0/2709811081 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fd0ac104b90 0x7fd0ac19f4f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:14:43.780 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:43.777+0000 7fd0a9e00700 1 -- 172.21.15.118:0/2709811081 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd0ac1a3fc0 con 0x7fd0ac06a380 2024-03-24T00:14:43.781 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:43.777+0000 7fd0aa800700 1 --2- 172.21.15.118:0/2709811081 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fd0ac107080 0x7fd0ac1a3880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T00:14:43.781 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:43.777+0000 7fd0a9400700 1 --2- 172.21.15.118:0/2709811081 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fd0ac104b90 0x7fd0ac19f4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T00:14:43.781 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:43.777+0000 7fd0a9e00700 1 --2- 172.21.15.118:0/2709811081 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd0ac06a380 0x7fd0ac19efb0 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7fd0a400eb30 tx=0x7fd0a400eef0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:14:43.782 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:43.777+0000 7fd0a2a00700 1 -- 172.21.15.118:0/2709811081 <== mon.2 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd0a400cc80 con 0x7fd0ac06a380 2024-03-24T00:14:43.783 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:43.778+0000 7fd0a2a00700 1 -- 172.21.15.118:0/2709811081 <== mon.2 v2:172.21.15.189:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fd0a4004500 con 0x7fd0ac06a380 2024-03-24T00:14:43.783 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:43.778+0000 7fd0a2a00700 1 -- 172.21.15.118:0/2709811081 <== mon.2 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd0a4010430 con 0x7fd0ac06a380 2024-03-24T00:14:43.783 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:43.778+0000 7fd0b1000700 1 -- 172.21.15.118:0/2709811081 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd0ac1a42b0 con 0x7fd0ac06a380 2024-03-24T00:14:43.783 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:43.778+0000 7fd0b1000700 1 -- 172.21.15.118:0/2709811081 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd0ac1abb30 con 0x7fd0ac06a380 2024-03-24T00:14:43.783 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:43.779+0000 7fd0b1000700 1 -- 172.21.15.118:0/2709811081 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd0ac068e50 con 0x7fd0ac06a380 2024-03-24T00:14:43.783 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:43.782+0000 7fd0a2a00700 1 -- 172.21.15.118:0/2709811081 <== mon.2 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 18) v1 ==== 137291+0+0 (secure 0 0 0) 0x7fd0a4003f50 con 0x7fd0ac06a380 2024-03-24T00:14:43.785 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:43.783+0000 7fd0a2a00700 1 --2- 172.21.15.118:0/2709811081 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7fd0900a2910 0x7fd0900a4dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:14:43.785 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:43.783+0000 7fd0a2a00700 1 -- 172.21.15.118:0/2709811081 <== mon.2 v2:172.21.15.189:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 8248+0+0 (secure 0 0 0) 0x7fd0a40cec80 con 0x7fd0ac06a380 2024-03-24T00:14:43.785 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:43.783+0000 7fd0a9400700 1 --2- 172.21.15.118:0/2709811081 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7fd0900a2910 0x7fd0900a4dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:14:43.785 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:43.784+0000 7fd0a9400700 1 --2- 172.21.15.118:0/2709811081 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7fd0900a2910 0x7fd0900a4dd0 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7fd098006010 tx=0x7fd09801a040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:14:43.787 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:43.787+0000 7fd0a2a00700 1 -- 172.21.15.118:0/2709811081 <== mon.2 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7fd0a40d5080 con 0x7fd0ac06a380 2024-03-24T00:14:44.011 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:44.010+0000 7fd0b1000700 1 -- 172.21.15.118:0/2709811081 --> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fd0ac103280 con 0x7fd0900a2910 2024-03-24T00:14:44.034 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:44.032+0000 7fd0a2a00700 1 -- 172.21.15.118:0/2709811081 <== mgr.14152 v2:172.21.15.118:6800/2112912943 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+259240 (secure 0 0 0) 0x7fd0ac103280 con 0x7fd0900a2910 2024-03-24T00:14:44.042 INFO:teuthology.orchestra.run.smithi118.stdout: 2024-03-24T00:14:44.044 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:44.038+0000 7fd0b1000700 1 -- 172.21.15.118:0/2709811081 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7fd0900a2910 msgr2=0x7fd0900a4dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:14:44.044 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:44.038+0000 7fd0b1000700 1 --2- 172.21.15.118:0/2709811081 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7fd0900a2910 0x7fd0900a4dd0 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7fd098006010 tx=0x7fd09801a040 comp rx=0 tx=0).stop 2024-03-24T00:14:44.045 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:44.038+0000 7fd0b1000700 1 -- 172.21.15.118:0/2709811081 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd0ac06a380 msgr2=0x7fd0ac19efb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:14:44.045 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:44.038+0000 7fd0b1000700 1 --2- 172.21.15.118:0/2709811081 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd0ac06a380 0x7fd0ac19efb0 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7fd0a400eb30 tx=0x7fd0a400eef0 comp rx=0 tx=0).stop 2024-03-24T00:14:44.045 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:44.038+0000 7fd0b1000700 1 -- 172.21.15.118:0/2709811081 shutdown_connections 2024-03-24T00:14:44.045 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:44.038+0000 7fd0b1000700 1 --2- 172.21.15.118:0/2709811081 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd0ac06a380 0x7fd0ac19efb0 unknown :-1 s=CLOSED pgs=122 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:14:44.045 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:44.038+0000 7fd0b1000700 1 --2- 172.21.15.118:0/2709811081 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7fd0900a2910 0x7fd0900a4dd0 unknown :-1 s=CLOSED pgs=206 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:14:44.045 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:44.038+0000 7fd0b1000700 1 --2- 172.21.15.118:0/2709811081 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fd0ac107080 0x7fd0ac1a3880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:14:44.045 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:44.038+0000 7fd0b1000700 1 --2- 172.21.15.118:0/2709811081 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fd0ac104b90 0x7fd0ac19f4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:14:44.045 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:44.038+0000 7fd0b1000700 1 -- 172.21.15.118:0/2709811081 >> 172.21.15.118:0/2709811081 conn(0x7fd0ac077fc0 msgr2=0x7fd0ac101b60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:14:44.046 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:44.038+0000 7fd0b1000700 1 -- 172.21.15.118:0/2709811081 shutdown_connections 2024-03-24T00:14:44.046 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:44.039+0000 7fd0b1000700 1 -- 172.21.15.118:0/2709811081 wait complete. 2024-03-24T00:14:44.046 INFO:teuthology.orchestra.run.smithi118.stderr:dumped all 2024-03-24T00:14:45.006 INFO:teuthology.orchestra.run.smithi118.stdout:{"pg_ready":true,"pg_map":{"version":367,"stamp":"2024-03-24T00:14:43.645813+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":660038,"num_objects":102,"num_object_clones":0,"num_object_copies":306,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":102,"num_whiteouts":0,"num_read":1089,"num_read_kb":665,"num_write":429,"num_write_kb":3083,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18,"num_bytes_recovered":1243109,"num_keys_recovered":0,"num_objects_omap":66,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":854,"ondisk_log_size":854,"up":291,"acting":291,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":291,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":356748,"kb_used_data":11756,"kb_used_omap":18,"kb_used_meta":344877,"kb_avail":1124585076,"statfs":{"total":1151940427776,"available":1151575117824,"internally_reserved":0,"allocated":12038144,"data_stored":7492482,"data_compressed":191142,"data_compressed_allocated":253952,"data_compressed_original":2039808,"omap_allocated":19142,"internal_metadata":353154362},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":531,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":122,"num_read_kb":63,"num_write":9,"num_write_kb":7,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004082"},"pg_stats":[{"pgid":"3.3f","version":"88'8","reported_seq":25,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-24T00:14:35.620777+0000","last_change":"2024-03-24T00:13:02.325815+0000","last_active":"2024-03-24T00:14:35.620777+0000","last_peered":"2024-03-24T00:14:35.620777+0000","last_clean":"2024-03-24T00:14:35.620777+0000","last_became_active":"2024-03-24T00:13:02.325450+0000","last_became_peered":"2024-03-24T00:13:02.325450+0000","last_unstale":"2024-03-24T00:14:35.620777+0000","last_undegraded":"2024-03-24T00:14:35.620777+0000","last_fullsized":"2024-03-24T00:14:35.620777+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:13:01.267198+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:13:01.267198+0000","last_clean_scrub_stamp":"2024-03-24T00:13:01.267198+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T02:18:21.994322+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.3e","version":"88'4","reported_seq":21,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-24T00:14:35.611588+0000","last_change":"2024-03-24T00:13:02.324015+0000","last_active":"2024-03-24T00:14:35.611588+0000","last_peered":"2024-03-24T00:14:35.611588+0000","last_clean":"2024-03-24T00:14:35.611588+0000","last_became_active":"2024-03-24T00:13:02.323668+0000","last_became_peered":"2024-03-24T00:13:02.323668+0000","last_unstale":"2024-03-24T00:14:35.611588+0000","last_undegraded":"2024-03-24T00:14:35.611588+0000","last_fullsized":"2024-03-24T00:14:35.611588+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:13:01.267198+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:13:01.267198+0000","last_clean_scrub_stamp":"2024-03-24T00:13:01.267198+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:36:19.442266+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11,10],"acting":[0,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3d","version":"88'7","reported_seq":24,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-24T00:14:35.616284+0000","last_change":"2024-03-24T00:13:02.328911+0000","last_active":"2024-03-24T00:14:35.616284+0000","last_peered":"2024-03-24T00:14:35.616284+0000","last_clean":"2024-03-24T00:14:35.616284+0000","last_became_active":"2024-03-24T00:13:02.324333+0000","last_became_peered":"2024-03-24T00:13:02.324333+0000","last_unstale":"2024-03-24T00:14:35.616284+0000","last_undegraded":"2024-03-24T00:14:35.616284+0000","last_fullsized":"2024-03-24T00:14:35.616284+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:13:01.267198+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:13:01.267198+0000","last_clean_scrub_stamp":"2024-03-24T00:13:01.267198+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T02:49:02.318097+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"88'8","reported_seq":25,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-24T00:14:35.602705+0000","last_change":"2024-03-24T00:13:02.309057+0000","last_active":"2024-03-24T00:14:35.602705+0000","last_peered":"2024-03-24T00:14:35.602705+0000","last_clean":"2024-03-24T00:14:35.602705+0000","last_became_active":"2024-03-24T00:13:02.308403+0000","last_became_peered":"2024-03-24T00:13:02.308403+0000","last_unstale":"2024-03-24T00:14:35.602705+0000","last_undegraded":"2024-03-24T00:14:35.602705+0000","last_fullsized":"2024-03-24T00:14:35.602705+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:13:01.267198+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:13:01.267198+0000","last_clean_scrub_stamp":"2024-03-24T00:13:01.267198+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T07:54:44.310833+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,8,2],"acting":[4,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.3b","version":"88'4","reported_seq":21,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-24T00:14:35.622268+0000","last_change":"2024-03-24T00:13:02.330651+0000","last_active":"2024-03-24T00:14:35.622268+0000","last_peered":"2024-03-24T00:14:35.622268+0000","last_clean":"2024-03-24T00:14:35.622268+0000","last_became_active":"2024-03-24T00:13:02.330313+0000","last_became_peered":"2024-03-24T00:13:02.330313+0000","last_unstale":"2024-03-24T00:14:35.622268+0000","last_undegraded":"2024-03-24T00:14:35.622268+0000","last_fullsized":"2024-03-24T00:14:35.622268+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:13:01.267198+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:13:01.267198+0000","last_clean_scrub_stamp":"2024-03-24T00:13:01.267198+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T03:45:38.480929+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,6],"acting":[8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3a","version":"88'6","reported_seq":23,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-24T00:14:35.607384+0000","last_change":"2024-03-24T00:13:02.313115+0000","last_active":"2024-03-24T00:14:35.607384+0000","last_peered":"2024-03-24T00:14:35.607384+0000","last_clean":"2024-03-24T00:14:35.607384+0000","last_became_active":"2024-03-24T00:13:02.312387+0000","last_became_peered":"2024-03-24T00:13:02.312387+0000","last_unstale":"2024-03-24T00:14:35.607384+0000","last_undegraded":"2024-03-24T00:14:35.607384+0000","last_fullsized":"2024-03-24T00:14:35.607384+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:13:01.267198+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:13:01.267198+0000","last_clean_scrub_stamp":"2024-03-24T00:13:01.267198+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T09:16:02.574920+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,4],"acting":[10,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.39","version":"88'5","reported_seq":28,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-24T00:14:35.605231+0000","last_change":"2024-03-24T00:14:07.503364+0000","last_active":"2024-03-24T00:14:35.605231+0000","last_peered":"2024-03-24T00:14:35.605231+0000","last_clean":"2024-03-24T00:14:35.605231+0000","last_became_active":"2024-03-24T00:14:07.503103+0000","last_became_peered":"2024-03-24T00:14:07.503103+0000","last_unstale":"2024-03-24T00:14:35.605231+0000","last_undegraded":"2024-03-24T00:14:35.605231+0000","last_fullsized":"2024-03-24T00:14:35.605231+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:13:01.267198+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:13:01.267198+0000","last_clean_scrub_stamp":"2024-03-24T00:13:01.267198+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T03:22:04.225634+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,8],"acting":[7,9,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"88'9","reported_seq":20,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-24T00:14:35.608741+0000","last_change":"2024-03-24T00:14:07.681011+0000","last_active":"2024-03-24T00:14:35.608741+0000","last_peered":"2024-03-24T00:14:35.608741+0000","last_clean":"2024-03-24T00:14:35.608741+0000","last_became_active":"2024-03-24T00:14:07.680726+0000","last_became_peered":"2024-03-24T00:14:07.680726+0000","last_unstale":"2024-03-24T00:14:35.608741+0000","last_undegraded":"2024-03-24T00:14:35.608741+0000","last_fullsized":"2024-03-24T00:14:35.608741+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:13:01.267198+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:13:01.267198+0000","last_clean_scrub_stamp":"2024-03-24T00:13:01.267198+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T01:05:03.379937+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,5,10],"acting":[9,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.37","version":"88'11","reported_seq":28,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-24T00:14:35.619612+0000","last_change":"2024-03-24T00:13:02.305625+0000","last_active":"2024-03-24T00:14:35.619612+0000","last_peered":"2024-03-24T00:14:35.619612+0000","last_clean":"2024-03-24T00:14:35.619612+0000","last_became_active":"2024-03-24T00:13:02.305221+0000","last_became_peered":"2024-03-24T00:13:02.305221+0000","last_unstale":"2024-03-24T00:14:35.619612+0000","last_undegraded":"2024-03-24T00:14:35.619612+0000","last_fullsized":"2024-03-24T00:14:35.619612+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:13:01.267198+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:13:01.267198+0000","last_clean_scrub_stamp":"2024-03-24T00:13:01.267198+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T04:37:12.467387+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,11],"acting":[1,5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.36","version":"88'16","reported_seq":33,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-24T00:14:35.610392+0000","last_change":"2024-03-24T00:13:02.307878+0000","last_active":"2024-03-24T00:14:35.610392+0000","last_peered":"2024-03-24T00:14:35.610392+0000","last_clean":"2024-03-24T00:14:35.610392+0000","last_became_active":"2024-03-24T00:13:02.307513+0000","last_became_peered":"2024-03-24T00:13:02.307513+0000","last_unstale":"2024-03-24T00:14:35.610392+0000","last_undegraded":"2024-03-24T00:14:35.610392+0000","last_fullsized":"2024-03-24T00:14:35.610392+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:13:01.267198+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:13:01.267198+0000","last_clean_scrub_stamp":"2024-03-24T00:13:01.267198+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T07:06:08.162416+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,4],"acting":[10,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.35","version":"88'6","reported_seq":23,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-24T00:14:35.616059+0000","last_change":"2024-03-24T00:13:02.310883+0000","last_active":"2024-03-24T00:14:35.616059+0000","last_peered":"2024-03-24T00:14:35.616059+0000","last_clean":"2024-03-24T00:14:35.616059+0000","last_became_active":"2024-03-24T00:13:02.310559+0000","last_became_peered":"2024-03-24T00:13:02.310559+0000","last_unstale":"2024-03-24T00:14:35.616059+0000","last_undegraded":"2024-03-24T00:14:35.616059+0000","last_fullsized":"2024-03-24T00:14:35.616059+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:13:01.267198+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:13:01.267198+0000","last_clean_scrub_stamp":"2024-03-24T00:13:01.267198+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T04:04:25.801248+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8,1],"acting":[7,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.34","version":"88'7","reported_seq":24,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-24T00:14:35.602511+0000","last_change":"2024-03-24T00:13:02.308747+0000","last_active":"2024-03-24T00:14:35.602511+0000","last_peered":"2024-03-24T00:14:35.602511+0000","last_clean":"2024-03-24T00:14:35.602511+0000","last_became_active":"2024-03-24T00:13:02.307791+0000","last_became_peered":"2024-03-24T00:13:02.307791+0000","last_unstale":"2024-03-24T00:14:35.602511+0000","last_undegraded":"2024-03-24T00:14:35.602511+0000","last_fullsized":"2024-03-24T00:14:35.602511+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:13:01.267198+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:13:01.267198+0000","last_clean_scrub_stamp":"2024-03-24T00:13:01.267198+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T03:28:08.780044+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,8],"acting":[4,10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"88'10","reported_seq":27,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-24T00:14:35.625613+0000","last_change":"2024-03-24T00:13:02.327061+0000","last_active":"2024-03-24T00:14:35.625613+0000","last_peered":"2024-03-24T00:14:35.625613+0000","last_clean":"2024-03-24T00:14:35.625613+0000","last_became_active":"2024-03-24T00:13:02.326696+0000","last_became_peered":"2024-03-24T00:13:02.326696+0000","last_unstale":"2024-03-24T00:14:35.625613+0000","last_undegraded":"2024-03-24T00:14:35.625613+0000","last_fullsized":"2024-03-24T00:14:35.625613+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:13:01.267198+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:13:01.267198+0000","last_clean_scrub_stamp":"2024-03-24T00:13:01.267198+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T08:55:17.325488+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,5],"acting":[6,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.32","version":"88'7","reported_seq":24,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-24T00:14:35.610502+0000","last_change":"2024-03-24T00:13:02.321326+0000","last_active":"2024-03-24T00:14:35.610502+0000","last_peered":"2024-03-24T00:14:35.610502+0000","last_clean":"2024-03-24T00:14:35.610502+0000","last_became_active":"2024-03-24T00:13:02.320964+0000","last_became_peered":"2024-03-24T00:13:02.320964+0000","last_unstale":"2024-03-24T00:14:35.610502+0000","last_undegraded":"2024-03-24T00:14:35.610502+0000","last_fullsized":"2024-03-24T00:14:35.610502+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:13:01.267198+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:13:01.267198+0000","last_clean_scrub_stamp":"2024-03-24T00:13:01.267198+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T08:05:15.365809+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,10],"acting":[0,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"88'7","reported_seq":30,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-24T00:14:35.605221+0000","last_change":"2024-03-24T00:14:07.505774+0000","last_active":"2024-03-24T00:14:35.605221+0000","last_peered":"2024-03-24T00:14:35.605221+0000","last_clean":"2024-03-24T00:14:35.605221+0000","last_became_active":"2024-03-24T00:14:07.504800+0000","last_became_peered":"2024-03-24T00:14:07.504800+0000","last_unstale":"2024-03-24T00:14:35.605221+0000","last_undegraded":"2024-03-24T00:14:35.605221+0000","last_fullsized":"2024-03-24T00:14:35.605221+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:13:01.267198+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:13:01.267198+0000","last_clean_scrub_stamp":"2024-03-24T00:13:01.267198+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T06:50:30.686330+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.30","version":"88'9","reported_seq":26,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-24T00:14:35.621361+0000","last_change":"2024-03-24T00:13:02.325460+0000","last_active":"2024-03-24T00:14:35.621361+0000","last_peered":"2024-03-24T00:14:35.621361+0000","last_clean":"2024-03-24T00:14:35.621361+0000","last_became_active":"2024-03-24T00:13:02.325112+0000","last_became_peered":"2024-03-24T00:13:02.325112+0000","last_unstale":"2024-03-24T00:14:35.621361+0000","last_undegraded":"2024-03-24T00:14:35.621361+0000","last_fullsized":"2024-03-24T00:14:35.621361+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:13:01.267198+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:13:01.267198+0000","last_clean_scrub_stamp":"2024-03-24T00:13:01.267198+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T03:15:16.362970+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2f","version":"88'8","reported_seq":25,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-24T00:14:35.620799+0000","last_change":"2024-03-24T00:13:02.325918+0000","last_active":"2024-03-24T00:14:35.620799+0000","last_peered":"2024-03-24T00:14:35.620799+0000","last_clean":"2024-03-24T00:14:35.620799+0000","last_became_active":"2024-03-24T00:13:02.325557+0000","last_became_peered":"2024-03-24T00:13:02.325557+0000","last_unstale":"2024-03-24T00:14:35.620799+0000","last_undegraded":"2024-03-24T00:14:35.620799+0000","last_fullsized":"2024-03-24T00:14:35.620799+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:13:01.267198+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:13:01.267198+0000","last_clean_scrub_stamp":"2024-03-24T00:13:01.267198+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T02:55:43.759674+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2e","version":"88'10","reported_seq":27,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-24T00:14:35.619330+0000","last_change":"2024-03-24T00:13:02.308106+0000","last_active":"2024-03-24T00:14:35.619330+0000","last_peered":"2024-03-24T00:14:35.619330+0000","last_clean":"2024-03-24T00:14:35.619330+0000","last_became_active":"2024-03-24T00:13:02.306972+0000","last_became_peered":"2024-03-24T00:13:02.306972+0000","last_unstale":"2024-03-24T00:14:35.619330+0000","last_undegraded":"2024-03-24T00:14:35.619330+0000","last_fullsized":"2024-03-24T00:14:35.619330+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:13:01.267198+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:13:01.267198+0000","last_clean_scrub_stamp":"2024-03-24T00:13:01.267198+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T09:38:54.677255+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,10],"acting":[9,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.2d","version":"88'7","reported_seq":24,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-24T00:14:35.624901+0000","last_change":"2024-03-24T00:13:02.331742+0000","last_active":"2024-03-24T00:14:35.624901+0000","last_peered":"2024-03-24T00:14:35.624901+0000","last_clean":"2024-03-24T00:14:35.624901+0000","last_became_active":"2024-03-24T00:13:02.331103+0000","last_became_peered":"2024-03-24T00:13:02.331103+0000","last_unstale":"2024-03-24T00:14:35.624901+0000","last_undegraded":"2024-03-24T00:14:35.624901+0000","last_fullsized":"2024-03-24T00:14:35.624901+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:13:01.267198+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:13:01.267198+0000","last_clean_scrub_stamp":"2024-03-24T00:13:01.267198+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:25:58.116306+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1f","version":"86'13","reported_seq":147,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-24T00:14:41.573936+0000","last_change":"2024-03-24T00:14:07.520827+0000","last_active":"2024-03-24T00:14:41.573936+0000","last_peered":"2024-03-24T00:14:41.573936+0000","last_clean":"2024-03-24T00:14:41.573936+0000","last_became_active":"2024-03-24T00:14:07.505406+0000","last_became_peered":"2024-03-24T00:14:07.505406+0000","last_unstale":"2024-03-24T00:14:41.573936+0000","last_undegraded":"2024-03-24T00:14:41.573936+0000","last_fullsized":"2024-03-24T00:14:41.573936+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:13:00.222790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:13:00.222790+0000","last_clean_scrub_stamp":"2024-03-24T00:13:00.222790+0000","objects_scrubbed":0,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T01:05:01.235978+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":93,"num_read_kb":93,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":672,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,1],"acting":[9,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.1e","version":"88'5","reported_seq":22,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-24T00:14:35.613946+0000","last_change":"2024-03-24T00:13:02.322461+0000","last_active":"2024-03-24T00:14:35.613946+0000","last_peered":"2024-03-24T00:14:35.613946+0000","last_clean":"2024-03-24T00:14:35.613946+0000","last_became_active":"2024-03-24T00:13:02.321752+0000","last_became_peered":"2024-03-24T00:13:02.321752+0000","last_unstale":"2024-03-24T00:14:35.613946+0000","last_undegraded":"2024-03-24T00:14:35.613946+0000","last_fullsized":"2024-03-24T00:14:35.613946+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:13:01.267198+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:13:01.267198+0000","last_clean_scrub_stamp":"2024-03-24T00:13:01.267198+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T06:37:05.825919+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1e","version":"86'2","reported_seq":21,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-24T00:14:07.504657+0000","last_change":"2024-03-24T00:13:01.324972+0000","last_active":"2024-03-24T00:14:07.504657+0000","last_peered":"2024-03-24T00:14:07.504657+0000","last_clean":"2024-03-24T00:14:07.504657+0000","last_became_active":"2024-03-24T00:13:01.323342+0000","last_became_peered":"2024-03-24T00:13:01.323342+0000","last_unstale":"2024-03-24T00:14:07.504657+0000","last_undegraded":"2024-03-24T00:14:07.504657+0000","last_fullsized":"2024-03-24T00:14:07.504657+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:13:00.222790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:13:00.222790+0000","last_clean_scrub_stamp":"2024-03-24T00:13:00.222790+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T05:34:35.827634+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"88'6","reported_seq":23,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-24T00:14:35.622169+0000","last_change":"2024-03-24T00:13:02.325455+0000","last_active":"2024-03-24T00:14:35.622169+0000","last_peered":"2024-03-24T00:14:35.622169+0000","last_clean":"2024-03-24T00:14:35.622169+0000","last_became_active":"2024-03-24T00:13:02.325061+0000","last_became_peered":"2024-03-24T00:13:02.325061+0000","last_unstale":"2024-03-24T00:14:35.622169+0000","last_undegraded":"2024-03-24T00:14:35.622169+0000","last_fullsized":"2024-03-24T00:14:35.622169+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:13:01.267198+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:13:01.267198+0000","last_clean_scrub_stamp":"2024-03-24T00:13:01.267198+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T06:13:33.223361+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,9],"acting":[0,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1d","version":"86'12","reported_seq":32,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-24T00:14:07.499436+0000","last_change":"2024-03-24T00:13:01.300541+0000","last_active":"2024-03-24T00:14:07.499436+0000","last_peered":"2024-03-24T00:14:07.499436+0000","last_clean":"2024-03-24T00:14:07.499436+0000","last_became_active":"2024-03-24T00:13:01.299686+0000","last_became_peered":"2024-03-24T00:13:01.299686+0000","last_unstale":"2024-03-24T00:14:07.499436+0000","last_undegraded":"2024-03-24T00:14:07.499436+0000","last_fullsized":"2024-03-24T00:14:07.499436+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:13:00.222790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:13:00.222790+0000","last_clean_scrub_stamp":"2024-03-24T00:13:00.222790+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T00:20:54.474131+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":16,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":7,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1c","version":"88'9","reported_seq":26,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-24T00:14:35.624592+0000","last_change":"2024-03-24T00:13:02.325130+0000","last_active":"2024-03-24T00:14:35.624592+0000","last_peered":"2024-03-24T00:14:35.624592+0000","last_clean":"2024-03-24T00:14:35.624592+0000","last_became_active":"2024-03-24T00:13:02.324740+0000","last_became_peered":"2024-03-24T00:13:02.324740+0000","last_unstale":"2024-03-24T00:14:35.624592+0000","last_undegraded":"2024-03-24T00:14:35.624592+0000","last_fullsized":"2024-03-24T00:14:35.624592+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:13:01.267198+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:13:01.267198+0000","last_clean_scrub_stamp":"2024-03-24T00:13:01.267198+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T01:23:35.194788+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,6],"acting":[1,10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1c","version":"86'5","reported_seq":25,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-24T00:14:07.499266+0000","last_change":"2024-03-24T00:13:01.300572+0000","last_active":"2024-03-24T00:14:07.499266+0000","last_peered":"2024-03-24T00:14:07.499266+0000","last_clean":"2024-03-24T00:14:07.499266+0000","last_became_active":"2024-03-24T00:13:01.299801+0000","last_became_peered":"2024-03-24T00:13:01.299801+0000","last_unstale":"2024-03-24T00:14:07.499266+0000","last_undegraded":"2024-03-24T00:14:07.499266+0000","last_fullsized":"2024-03-24T00:14:07.499266+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:13:00.222790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:13:00.222790+0000","last_clean_scrub_stamp":"2024-03-24T00:13:00.222790+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T10:29:03.986969+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":7,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,5],"acting":[8,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1d","version":"88'15","reported_seq":32,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-24T00:14:35.613940+0000","last_change":"2024-03-24T00:13:02.304079+0000","last_active":"2024-03-24T00:14:35.613940+0000","last_peered":"2024-03-24T00:14:35.613940+0000","last_clean":"2024-03-24T00:14:35.613940+0000","last_became_active":"2024-03-24T00:13:02.303700+0000","last_became_peered":"2024-03-24T00:13:02.303700+0000","last_unstale":"2024-03-24T00:14:35.613940+0000","last_undegraded":"2024-03-24T00:14:35.613940+0000","last_fullsized":"2024-03-24T00:14:35.613940+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:13:01.267198+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:13:01.267198+0000","last_clean_scrub_stamp":"2024-03-24T00:13:01.267198+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T10:19:07.701927+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1b","version":"88'7","reported_seq":126,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-24T00:14:40.374043+0000","last_change":"2024-03-24T00:13:01.300949+0000","last_active":"2024-03-24T00:14:40.374043+0000","last_peered":"2024-03-24T00:14:40.374043+0000","last_clean":"2024-03-24T00:14:40.374043+0000","last_became_active":"2024-03-24T00:13:01.300605+0000","last_became_peered":"2024-03-24T00:13:01.300605+0000","last_unstale":"2024-03-24T00:14:40.374043+0000","last_undegraded":"2024-03-24T00:14:40.374043+0000","last_fullsized":"2024-03-24T00:14:40.374043+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:13:00.222790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:13:00.222790+0000","last_clean_scrub_stamp":"2024-03-24T00:13:00.222790+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T02:47:58.283103+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1154,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":100,"num_read_kb":4,"num_write":4,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1a","version":"88'7","reported_seq":24,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-24T00:14:35.603317+0000","last_change":"2024-03-24T00:13:02.312884+0000","last_active":"2024-03-24T00:14:35.603317+0000","last_peered":"2024-03-24T00:14:35.603317+0000","last_clean":"2024-03-24T00:14:35.603317+0000","last_became_active":"2024-03-24T00:13:02.312533+0000","last_became_peered":"2024-03-24T00:13:02.312533+0000","last_unstale":"2024-03-24T00:14:35.603317+0000","last_undegraded":"2024-03-24T00:14:35.603317+0000","last_fullsized":"2024-03-24T00:14:35.603317+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:13:01.267198+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:13:01.267198+0000","last_clean_scrub_stamp":"2024-03-24T00:13:01.267198+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T10:54:48.398645+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,5],"acting":[8,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.1a","version":"86'7","reported_seq":26,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-24T00:14:07.492787+0000","last_change":"2024-03-24T00:13:01.306292+0000","last_active":"2024-03-24T00:14:07.492787+0000","last_peered":"2024-03-24T00:14:07.492787+0000","last_clean":"2024-03-24T00:14:07.492787+0000","last_became_active":"2024-03-24T00:13:01.305898+0000","last_became_peered":"2024-03-24T00:13:01.305898+0000","last_unstale":"2024-03-24T00:14:07.492787+0000","last_undegraded":"2024-03-24T00:14:07.492787+0000","last_fullsized":"2024-03-24T00:14:07.492787+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:13:00.222790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:13:00.222790+0000","last_clean_scrub_stamp":"2024-03-24T00:13:00.222790+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T07:41:17.003588+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8,6],"acting":[10,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.1b","version":"88'8","reported_seq":25,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-24T00:14:35.622308+0000","last_change":"2024-03-24T00:13:02.323763+0000","last_active":"2024-03-24T00:14:35.622308+0000","last_peered":"2024-03-24T00:14:35.622308+0000","last_clean":"2024-03-24T00:14:35.622308+0000","last_became_active":"2024-03-24T00:13:02.323367+0000","last_became_peered":"2024-03-24T00:13:02.323367+0000","last_unstale":"2024-03-24T00:14:35.622308+0000","last_undegraded":"2024-03-24T00:14:35.622308+0000","last_fullsized":"2024-03-24T00:14:35.622308+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:13:01.267198+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:13:01.267198+0000","last_clean_scrub_stamp":"2024-03-24T00:13:01.267198+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T08:53:21.671317+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,3],"acting":[5,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.19","version":"86'3","reported_seq":22,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-24T00:14:07.632380+0000","last_change":"2024-03-24T00:13:01.323111+0000","last_active":"2024-03-24T00:14:07.632380+0000","last_peered":"2024-03-24T00:14:07.632380+0000","last_clean":"2024-03-24T00:14:07.632380+0000","last_became_active":"2024-03-24T00:13:01.322772+0000","last_became_peered":"2024-03-24T00:13:01.322772+0000","last_unstale":"2024-03-24T00:14:07.632380+0000","last_undegraded":"2024-03-24T00:14:07.632380+0000","last_fullsized":"2024-03-24T00:14:07.632380+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:13:00.222790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:13:00.222790+0000","last_clean_scrub_stamp":"2024-03-24T00:13:00.222790+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T04:19:53.097933+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,9],"acting":[6,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"88'8","reported_seq":25,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-24T00:14:35.624504+0000","last_change":"2024-03-24T00:13:02.330469+0000","last_active":"2024-03-24T00:14:35.624504+0000","last_peered":"2024-03-24T00:14:35.624504+0000","last_clean":"2024-03-24T00:14:35.624504+0000","last_became_active":"2024-03-24T00:13:02.327019+0000","last_became_peered":"2024-03-24T00:13:02.327019+0000","last_unstale":"2024-03-24T00:14:35.624504+0000","last_undegraded":"2024-03-24T00:14:35.624504+0000","last_fullsized":"2024-03-24T00:14:35.624504+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:13:01.267198+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:13:01.267198+0000","last_clean_scrub_stamp":"2024-03-24T00:13:01.267198+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T07:30:37.155535+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,6],"acting":[3,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.18","version":"86'3","reported_seq":23,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-24T00:14:07.679914+0000","last_change":"2024-03-24T00:13:01.300089+0000","last_active":"2024-03-24T00:14:07.679914+0000","last_peered":"2024-03-24T00:14:07.679914+0000","last_clean":"2024-03-24T00:14:07.679914+0000","last_became_active":"2024-03-24T00:13:01.299747+0000","last_became_peered":"2024-03-24T00:13:01.299747+0000","last_unstale":"2024-03-24T00:14:07.679914+0000","last_undegraded":"2024-03-24T00:14:07.679914+0000","last_fullsized":"2024-03-24T00:14:07.679914+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:13:00.222790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:13:00.222790+0000","last_clean_scrub_stamp":"2024-03-24T00:13:00.222790+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T00:30:30.142785+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"88'11","reported_seq":28,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-24T00:14:35.625608+0000","last_change":"2024-03-24T00:13:02.332490+0000","last_active":"2024-03-24T00:14:35.625608+0000","last_peered":"2024-03-24T00:14:35.625608+0000","last_clean":"2024-03-24T00:14:35.625608+0000","last_became_active":"2024-03-24T00:13:02.331182+0000","last_became_peered":"2024-03-24T00:13:02.331182+0000","last_unstale":"2024-03-24T00:14:35.625608+0000","last_undegraded":"2024-03-24T00:14:35.625608+0000","last_fullsized":"2024-03-24T00:14:35.625608+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:13:01.267198+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:13:01.267198+0000","last_clean_scrub_stamp":"2024-03-24T00:13:01.267198+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T06:22:16.084879+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,10],"acting":[9,3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.17","version":"88'6","reported_seq":208,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-24T00:14:38.371517+0000","last_change":"2024-03-24T00:13:01.299818+0000","last_active":"2024-03-24T00:14:38.371517+0000","last_peered":"2024-03-24T00:14:38.371517+0000","last_clean":"2024-03-24T00:14:38.371517+0000","last_became_active":"2024-03-24T00:13:01.299478+0000","last_became_peered":"2024-03-24T00:13:01.299478+0000","last_unstale":"2024-03-24T00:14:38.371517+0000","last_undegraded":"2024-03-24T00:14:38.371517+0000","last_fullsized":"2024-03-24T00:14:38.371517+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:13:00.222790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:13:00.222790+0000","last_clean_scrub_stamp":"2024-03-24T00:13:00.222790+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:51:47.088677+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":180,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":183,"num_read_kb":183,"num_write":7,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,9],"acting":[10,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.16","version":"88'10","reported_seq":27,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-24T00:14:35.621993+0000","last_change":"2024-03-24T00:13:02.304971+0000","last_active":"2024-03-24T00:14:35.621993+0000","last_peered":"2024-03-24T00:14:35.621993+0000","last_clean":"2024-03-24T00:14:35.621993+0000","last_became_active":"2024-03-24T00:13:02.304577+0000","last_became_peered":"2024-03-24T00:13:02.304577+0000","last_unstale":"2024-03-24T00:14:35.621993+0000","last_undegraded":"2024-03-24T00:14:35.621993+0000","last_fullsized":"2024-03-24T00:14:35.621993+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:13:01.267198+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:13:01.267198+0000","last_clean_scrub_stamp":"2024-03-24T00:13:01.267198+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T05:08:03.444554+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,9],"acting":[1,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.16","version":"86'4","reported_seq":23,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-24T00:14:07.496701+0000","last_change":"2024-03-24T00:13:01.303075+0000","last_active":"2024-03-24T00:14:07.496701+0000","last_peered":"2024-03-24T00:14:07.496701+0000","last_clean":"2024-03-24T00:14:07.496701+0000","last_became_active":"2024-03-24T00:13:01.301517+0000","last_became_peered":"2024-03-24T00:13:01.301517+0000","last_unstale":"2024-03-24T00:14:07.496701+0000","last_undegraded":"2024-03-24T00:14:07.496701+0000","last_fullsized":"2024-03-24T00:14:07.496701+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:13:00.222790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:13:00.222790+0000","last_clean_scrub_stamp":"2024-03-24T00:13:00.222790+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:04:10.552510+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.17","version":"88'9","reported_seq":26,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-24T00:14:35.618608+0000","last_change":"2024-03-24T00:13:02.320108+0000","last_active":"2024-03-24T00:14:35.618608+0000","last_peered":"2024-03-24T00:14:35.618608+0000","last_clean":"2024-03-24T00:14:35.618608+0000","last_became_active":"2024-03-24T00:13:02.319155+0000","last_became_peered":"2024-03-24T00:13:02.319155+0000","last_unstale":"2024-03-24T00:14:35.618608+0000","last_undegraded":"2024-03-24T00:14:35.618608+0000","last_fullsized":"2024-03-24T00:14:35.618608+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:13:01.267198+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:13:01.267198+0000","last_clean_scrub_stamp":"2024-03-24T00:13:01.267198+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T10:15:54.770304+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,5],"acting":[0,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.15","version":"86'5","reported_seq":25,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-24T00:14:07.680187+0000","last_change":"2024-03-24T00:13:01.323919+0000","last_active":"2024-03-24T00:14:07.680187+0000","last_peered":"2024-03-24T00:14:07.680187+0000","last_clean":"2024-03-24T00:14:07.680187+0000","last_became_active":"2024-03-24T00:13:01.323640+0000","last_became_peered":"2024-03-24T00:13:01.323640+0000","last_unstale":"2024-03-24T00:14:07.680187+0000","last_undegraded":"2024-03-24T00:14:07.680187+0000","last_fullsized":"2024-03-24T00:14:07.680187+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:13:00.222790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:13:00.222790+0000","last_clean_scrub_stamp":"2024-03-24T00:13:00.222790+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T06:16:49.018642+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":7,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,10],"acting":[5,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.14","version":"88'11","reported_seq":34,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-24T00:14:35.624710+0000","last_change":"2024-03-24T00:14:07.635409+0000","last_active":"2024-03-24T00:14:35.624710+0000","last_peered":"2024-03-24T00:14:35.624710+0000","last_clean":"2024-03-24T00:14:35.624710+0000","last_became_active":"2024-03-24T00:14:07.635093+0000","last_became_peered":"2024-03-24T00:14:07.635093+0000","last_unstale":"2024-03-24T00:14:35.624710+0000","last_undegraded":"2024-03-24T00:14:35.624710+0000","last_fullsized":"2024-03-24T00:14:35.624710+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:13:01.267198+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:13:01.267198+0000","last_clean_scrub_stamp":"2024-03-24T00:13:01.267198+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T10:31:10.205874+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,6],"acting":[1,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.14","version":"88'35","reported_seq":269,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-24T00:14:38.835685+0000","last_change":"2024-03-24T00:14:07.807393+0000","last_active":"2024-03-24T00:14:38.835685+0000","last_peered":"2024-03-24T00:14:38.835685+0000","last_clean":"2024-03-24T00:14:38.835685+0000","last_became_active":"2024-03-24T00:14:07.801140+0000","last_became_peered":"2024-03-24T00:14:07.801140+0000","last_unstale":"2024-03-24T00:14:38.835685+0000","last_undegraded":"2024-03-24T00:14:38.835685+0000","last_fullsized":"2024-03-24T00:14:38.835685+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:13:00.222790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:13:00.222790+0000","last_clean_scrub_stamp":"2024-03-24T00:13:00.222790+0000","objects_scrubbed":0,"log_size":35,"log_dups_size":0,"ondisk_log_size":35,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T10:35:25.627828+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":61950,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":191,"num_read_kb":155,"num_write":37,"num_write_kb":76,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":61701,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,4],"acting":[10,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.15","version":"88'6","reported_seq":23,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-24T00:14:35.600331+0000","last_change":"2024-03-24T00:13:02.307653+0000","last_active":"2024-03-24T00:14:35.600331+0000","last_peered":"2024-03-24T00:14:35.600331+0000","last_clean":"2024-03-24T00:14:35.600331+0000","last_became_active":"2024-03-24T00:13:02.307287+0000","last_became_peered":"2024-03-24T00:13:02.307287+0000","last_unstale":"2024-03-24T00:14:35.600331+0000","last_undegraded":"2024-03-24T00:14:35.600331+0000","last_fullsized":"2024-03-24T00:14:35.600331+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:13:01.267198+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:13:01.267198+0000","last_clean_scrub_stamp":"2024-03-24T00:13:01.267198+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T05:59:27.701614+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.13","version":"86'9","reported_seq":30,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-24T00:14:07.495111+0000","last_change":"2024-03-24T00:13:01.316854+0000","last_active":"2024-03-24T00:14:07.495111+0000","last_peered":"2024-03-24T00:14:07.495111+0000","last_clean":"2024-03-24T00:14:07.495111+0000","last_became_active":"2024-03-24T00:13:01.316507+0000","last_became_peered":"2024-03-24T00:13:01.316507+0000","last_unstale":"2024-03-24T00:14:07.495111+0000","last_undegraded":"2024-03-24T00:14:07.495111+0000","last_fullsized":"2024-03-24T00:14:07.495111+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:13:00.222790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:13:00.222790+0000","last_clean_scrub_stamp":"2024-03-24T00:13:00.222790+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T07:58:41.888003+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":13,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,10,3],"acting":[2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.12","version":"88'9","reported_seq":26,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-24T00:14:35.626094+0000","last_change":"2024-03-24T00:13:02.338811+0000","last_active":"2024-03-24T00:14:35.626094+0000","last_peered":"2024-03-24T00:14:35.626094+0000","last_clean":"2024-03-24T00:14:35.626094+0000","last_became_active":"2024-03-24T00:13:02.338438+0000","last_became_peered":"2024-03-24T00:13:02.338438+0000","last_unstale":"2024-03-24T00:14:35.626094+0000","last_undegraded":"2024-03-24T00:14:35.626094+0000","last_fullsized":"2024-03-24T00:14:35.626094+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:13:01.267198+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:13:01.267198+0000","last_clean_scrub_stamp":"2024-03-24T00:13:01.267198+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T03:55:16.531224+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.12","version":"86'5","reported_seq":25,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-24T00:14:07.679758+0000","last_change":"2024-03-24T00:13:01.313914+0000","last_active":"2024-03-24T00:14:07.679758+0000","last_peered":"2024-03-24T00:14:07.679758+0000","last_clean":"2024-03-24T00:14:07.679758+0000","last_became_active":"2024-03-24T00:13:01.313535+0000","last_became_peered":"2024-03-24T00:13:01.313535+0000","last_unstale":"2024-03-24T00:14:07.679758+0000","last_undegraded":"2024-03-24T00:14:07.679758+0000","last_fullsized":"2024-03-24T00:14:07.679758+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:13:00.222790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:13:00.222790+0000","last_clean_scrub_stamp":"2024-03-24T00:13:00.222790+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T03:23:53.011550+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":5,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"88'15","reported_seq":38,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-24T00:14:35.623013+0000","last_change":"2024-03-24T00:14:07.632978+0000","last_active":"2024-03-24T00:14:35.623013+0000","last_peered":"2024-03-24T00:14:35.623013+0000","last_clean":"2024-03-24T00:14:35.623013+0000","last_became_active":"2024-03-24T00:14:07.632719+0000","last_became_peered":"2024-03-24T00:14:07.632719+0000","last_unstale":"2024-03-24T00:14:35.623013+0000","last_undegraded":"2024-03-24T00:14:35.623013+0000","last_fullsized":"2024-03-24T00:14:35.623013+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:13:01.267198+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:13:01.267198+0000","last_clean_scrub_stamp":"2024-03-24T00:13:01.267198+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T09:59:38.993218+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,6],"acting":[7,9,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.11","version":"86'5","reported_seq":116,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-24T00:14:37.463705+0000","last_change":"2024-03-24T00:13:01.297887+0000","last_active":"2024-03-24T00:14:37.463705+0000","last_peered":"2024-03-24T00:14:37.463705+0000","last_clean":"2024-03-24T00:14:37.463705+0000","last_became_active":"2024-03-24T00:13:01.297560+0000","last_became_peered":"2024-03-24T00:13:01.297560+0000","last_unstale":"2024-03-24T00:14:37.463705+0000","last_undegraded":"2024-03-24T00:14:37.463705+0000","last_fullsized":"2024-03-24T00:14:37.463705+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:13:00.222790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:13:00.222790+0000","last_clean_scrub_stamp":"2024-03-24T00:13:00.222790+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T01:06:28.927274+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":588,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":93,"num_read_kb":3,"num_write":5,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,10],"acting":[4,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.10","version":"88'6","reported_seq":23,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-24T00:14:35.604519+0000","last_change":"2024-03-24T00:13:02.306811+0000","last_active":"2024-03-24T00:14:35.604519+0000","last_peered":"2024-03-24T00:14:35.604519+0000","last_clean":"2024-03-24T00:14:35.604519+0000","last_became_active":"2024-03-24T00:13:02.306499+0000","last_became_peered":"2024-03-24T00:13:02.306499+0000","last_unstale":"2024-03-24T00:14:35.604519+0000","last_undegraded":"2024-03-24T00:14:35.604519+0000","last_fullsized":"2024-03-24T00:14:35.604519+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:13:01.267198+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:13:01.267198+0000","last_clean_scrub_stamp":"2024-03-24T00:13:01.267198+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:49:32.683005+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.10","version":"86'17","reported_seq":37,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-24T00:14:07.498949+0000","last_change":"2024-03-24T00:13:01.307058+0000","last_active":"2024-03-24T00:14:07.498949+0000","last_peered":"2024-03-24T00:14:07.498949+0000","last_clean":"2024-03-24T00:14:07.498949+0000","last_became_active":"2024-03-24T00:13:01.306699+0000","last_became_peered":"2024-03-24T00:13:01.306699+0000","last_unstale":"2024-03-24T00:14:07.498949+0000","last_undegraded":"2024-03-24T00:14:07.498949+0000","last_fullsized":"2024-03-24T00:14:07.498949+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:13:00.222790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:13:00.222790+0000","last_clean_scrub_stamp":"2024-03-24T00:13:00.222790+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T10:32:37.334514+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,2],"acting":[8,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.11","version":"88'6","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-24T00:14:35.620207+0000","last_change":"2024-03-24T00:14:07.503357+0000","last_active":"2024-03-24T00:14:35.620207+0000","last_peered":"2024-03-24T00:14:35.620207+0000","last_clean":"2024-03-24T00:14:35.620207+0000","last_became_active":"2024-03-24T00:14:07.502511+0000","last_became_peered":"2024-03-24T00:14:07.502511+0000","last_unstale":"2024-03-24T00:14:35.620207+0000","last_undegraded":"2024-03-24T00:14:35.620207+0000","last_fullsized":"2024-03-24T00:14:35.620207+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:13:01.267198+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:13:01.267198+0000","last_clean_scrub_stamp":"2024-03-24T00:13:01.267198+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T04:18:09.351579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,1],"acting":[9,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.f","version":"86'2","reported_seq":22,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-24T00:14:07.499518+0000","last_change":"2024-03-24T00:13:01.295874+0000","last_active":"2024-03-24T00:14:07.499518+0000","last_peered":"2024-03-24T00:14:07.499518+0000","last_clean":"2024-03-24T00:14:07.499518+0000","last_became_active":"2024-03-24T00:13:01.295521+0000","last_became_peered":"2024-03-24T00:13:01.295521+0000","last_unstale":"2024-03-24T00:14:07.499518+0000","last_undegraded":"2024-03-24T00:14:07.499518+0000","last_fullsized":"2024-03-24T00:14:07.499518+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:13:00.222790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:13:00.222790+0000","last_clean_scrub_stamp":"2024-03-24T00:13:00.222790+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T06:47:15.749491+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5,4],"acting":[8,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.e","version":"88'5","reported_seq":22,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-24T00:14:35.603168+0000","last_change":"2024-03-24T00:13:02.306617+0000","last_active":"2024-03-24T00:14:35.603168+0000","last_peered":"2024-03-24T00:14:35.603168+0000","last_clean":"2024-03-24T00:14:35.603168+0000","last_became_active":"2024-03-24T00:13:02.306224+0000","last_became_peered":"2024-03-24T00:13:02.306224+0000","last_unstale":"2024-03-24T00:14:35.603168+0000","last_undegraded":"2024-03-24T00:14:35.603168+0000","last_fullsized":"2024-03-24T00:14:35.603168+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:13:01.267198+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:13:01.267198+0000","last_clean_scrub_stamp":"2024-03-24T00:13:01.267198+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T06:49:16.032741+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,4],"acting":[11,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.4","version":"86'1","reported_seq":20,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-24T00:14:07.495528+0000","last_change":"2024-03-24T00:13:01.314428+0000","last_active":"2024-03-24T00:14:07.495528+0000","last_peered":"2024-03-24T00:14:07.495528+0000","last_clean":"2024-03-24T00:14:07.495528+0000","last_became_active":"2024-03-24T00:13:01.313421+0000","last_became_peered":"2024-03-24T00:13:01.313421+0000","last_unstale":"2024-03-24T00:14:07.495528+0000","last_undegraded":"2024-03-24T00:14:07.495528+0000","last_fullsized":"2024-03-24T00:14:07.495528+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:13:00.222790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:13:00.222790+0000","last_clean_scrub_stamp":"2024-03-24T00:13:00.222790+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T08:10:25.796718+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,0],"acting":[9,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.5","version":"88'5","reported_seq":22,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-24T00:14:35.617044+0000","last_change":"2024-03-24T00:13:02.331660+0000","last_active":"2024-03-24T00:14:35.617044+0000","last_peered":"2024-03-24T00:14:35.617044+0000","last_clean":"2024-03-24T00:14:35.617044+0000","last_became_active":"2024-03-24T00:13:02.331288+0000","last_became_peered":"2024-03-24T00:13:02.331288+0000","last_unstale":"2024-03-24T00:14:35.617044+0000","last_undegraded":"2024-03-24T00:14:35.617044+0000","last_fullsized":"2024-03-24T00:14:35.617044+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:13:01.267198+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:13:01.267198+0000","last_clean_scrub_stamp":"2024-03-24T00:13:01.267198+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:36:55.576510+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3,6],"acting":[10,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.2","version":"86'5","reported_seq":119,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-24T00:14:41.574452+0000","last_change":"2024-03-24T00:13:01.310538+0000","last_active":"2024-03-24T00:14:41.574452+0000","last_peered":"2024-03-24T00:14:41.574452+0000","last_clean":"2024-03-24T00:14:41.574452+0000","last_became_active":"2024-03-24T00:13:01.310124+0000","last_became_peered":"2024-03-24T00:13:01.310124+0000","last_unstale":"2024-03-24T00:14:41.574452+0000","last_undegraded":"2024-03-24T00:14:41.574452+0000","last_fullsized":"2024-03-24T00:14:41.574452+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:13:00.222790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:13:00.222790+0000","last_clean_scrub_stamp":"2024-03-24T00:13:00.222790+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T05:47:03.534824+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":588,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":95,"num_read_kb":1,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3","version":"88'7","reported_seq":24,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-24T00:14:35.614536+0000","last_change":"2024-03-24T00:13:02.324117+0000","last_active":"2024-03-24T00:14:35.614536+0000","last_peered":"2024-03-24T00:14:35.614536+0000","last_clean":"2024-03-24T00:14:35.614536+0000","last_became_active":"2024-03-24T00:13:02.323706+0000","last_became_peered":"2024-03-24T00:13:02.323706+0000","last_unstale":"2024-03-24T00:14:35.614536+0000","last_undegraded":"2024-03-24T00:14:35.614536+0000","last_fullsized":"2024-03-24T00:14:35.614536+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:13:01.267198+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:13:01.267198+0000","last_clean_scrub_stamp":"2024-03-24T00:13:01.267198+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T05:34:31.657324+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,4,0],"acting":[10,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.0","version":"88'8","reported_seq":25,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-24T00:14:35.596304+0000","last_change":"2024-03-24T00:13:02.306030+0000","last_active":"2024-03-24T00:14:35.596304+0000","last_peered":"2024-03-24T00:14:35.596304+0000","last_clean":"2024-03-24T00:14:35.596304+0000","last_became_active":"2024-03-24T00:13:02.305588+0000","last_became_peered":"2024-03-24T00:13:02.305588+0000","last_unstale":"2024-03-24T00:14:35.596304+0000","last_undegraded":"2024-03-24T00:14:35.596304+0000","last_fullsized":"2024-03-24T00:14:35.596304+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:13:01.267198+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:13:01.267198+0000","last_clean_scrub_stamp":"2024-03-24T00:13:01.267198+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:20:58.206244+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,4],"acting":[5,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"86'5","reported_seq":118,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-24T00:14:40.467360+0000","last_change":"2024-03-24T00:13:01.318123+0000","last_active":"2024-03-24T00:14:40.467360+0000","last_peered":"2024-03-24T00:14:40.467360+0000","last_clean":"2024-03-24T00:14:40.467360+0000","last_became_active":"2024-03-24T00:13:01.317683+0000","last_became_peered":"2024-03-24T00:13:01.317683+0000","last_unstale":"2024-03-24T00:14:40.467360+0000","last_undegraded":"2024-03-24T00:14:40.467360+0000","last_fullsized":"2024-03-24T00:14:40.467360+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:13:00.222790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:13:00.222790+0000","last_clean_scrub_stamp":"2024-03-24T00:13:00.222790+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T06:33:02.669668+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":94,"num_read_kb":94,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"88'8","reported_seq":25,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-24T00:14:35.626644+0000","last_change":"2024-03-24T00:13:02.328599+0000","last_active":"2024-03-24T00:14:35.626644+0000","last_peered":"2024-03-24T00:14:35.626644+0000","last_clean":"2024-03-24T00:14:35.626644+0000","last_became_active":"2024-03-24T00:13:02.328165+0000","last_became_peered":"2024-03-24T00:13:02.328165+0000","last_unstale":"2024-03-24T00:14:35.626644+0000","last_undegraded":"2024-03-24T00:14:35.626644+0000","last_fullsized":"2024-03-24T00:14:35.626644+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:13:01.267198+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:13:01.267198+0000","last_clean_scrub_stamp":"2024-03-24T00:13:01.267198+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T07:22:39.646578+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"86'6","reported_seq":28,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-24T00:14:07.800384+0000","last_change":"2024-03-24T00:13:01.301932+0000","last_active":"2024-03-24T00:14:07.800384+0000","last_peered":"2024-03-24T00:14:07.800384+0000","last_clean":"2024-03-24T00:14:07.800384+0000","last_became_active":"2024-03-24T00:13:01.301464+0000","last_became_peered":"2024-03-24T00:13:01.301464+0000","last_unstale":"2024-03-24T00:14:07.800384+0000","last_undegraded":"2024-03-24T00:14:07.800384+0000","last_fullsized":"2024-03-24T00:14:07.800384+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:13:00.222790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:13:00.222790+0000","last_clean_scrub_stamp":"2024-03-24T00:13:00.222790+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T08:58:25.512421+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":4,"num_read_kb":4,"num_write":8,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,9],"acting":[4,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.0","version":"21'137","reported_seq":345,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-24T00:14:07.496358+0000","last_change":"2024-03-24T00:11:43.008767+0000","last_active":"2024-03-24T00:14:07.496358+0000","last_peered":"2024-03-24T00:14:07.496358+0000","last_clean":"2024-03-24T00:14:07.496358+0000","last_became_active":"2024-03-24T00:11:43.008364+0000","last_became_peered":"2024-03-24T00:11:43.008364+0000","last_unstale":"2024-03-24T00:14:07.496358+0000","last_undegraded":"2024-03-24T00:14:07.496358+0000","last_fullsized":"2024-03-24T00:14:07.496358+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:07:50.191828+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:07:50.191828+0000","last_clean_scrub_stamp":"2024-03-24T00:07:50.191828+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T01:55:26.549348+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"88'5","reported_seq":22,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-24T00:14:35.621140+0000","last_change":"2024-03-24T00:13:02.325667+0000","last_active":"2024-03-24T00:14:35.621140+0000","last_peered":"2024-03-24T00:14:35.621140+0000","last_clean":"2024-03-24T00:14:35.621140+0000","last_became_active":"2024-03-24T00:13:02.322476+0000","last_became_peered":"2024-03-24T00:13:02.322476+0000","last_unstale":"2024-03-24T00:14:35.621140+0000","last_undegraded":"2024-03-24T00:14:35.621140+0000","last_fullsized":"2024-03-24T00:14:35.621140+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:13:01.267198+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:13:01.267198+0000","last_clean_scrub_stamp":"2024-03-24T00:13:01.267198+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T08:07:19.936708+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"86'4","reported_seq":23,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-24T00:14:07.496426+0000","last_change":"2024-03-24T00:13:01.298082+0000","last_active":"2024-03-24T00:14:07.496426+0000","last_peered":"2024-03-24T00:14:07.496426+0000","last_clean":"2024-03-24T00:14:07.496426+0000","last_became_active":"2024-03-24T00:13:01.297686+0000","last_became_peered":"2024-03-24T00:13:01.297686+0000","last_unstale":"2024-03-24T00:14:07.496426+0000","last_undegraded":"2024-03-24T00:14:07.496426+0000","last_fullsized":"2024-03-24T00:14:07.496426+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:13:00.222790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:13:00.222790+0000","last_clean_scrub_stamp":"2024-03-24T00:13:00.222790+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T09:25:07.204076+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.5","version":"86'4","reported_seq":24,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-24T00:14:07.499345+0000","last_change":"2024-03-24T00:13:01.321250+0000","last_active":"2024-03-24T00:14:07.499345+0000","last_peered":"2024-03-24T00:14:07.499345+0000","last_clean":"2024-03-24T00:14:07.499345+0000","last_became_active":"2024-03-24T00:13:01.320929+0000","last_became_peered":"2024-03-24T00:13:01.320929+0000","last_unstale":"2024-03-24T00:14:07.499345+0000","last_undegraded":"2024-03-24T00:14:07.499345+0000","last_fullsized":"2024-03-24T00:14:07.499345+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:13:00.222790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:13:00.222790+0000","last_clean_scrub_stamp":"2024-03-24T00:13:00.222790+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T05:22:05.490303+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,9],"acting":[8,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.4","version":"88'9","reported_seq":26,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-24T00:14:35.621019+0000","last_change":"2024-03-24T00:13:02.323413+0000","last_active":"2024-03-24T00:14:35.621019+0000","last_peered":"2024-03-24T00:14:35.621019+0000","last_clean":"2024-03-24T00:14:35.621019+0000","last_became_active":"2024-03-24T00:13:02.322947+0000","last_became_peered":"2024-03-24T00:13:02.322947+0000","last_unstale":"2024-03-24T00:14:35.621019+0000","last_undegraded":"2024-03-24T00:14:35.621019+0000","last_fullsized":"2024-03-24T00:14:35.621019+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:13:01.267198+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:13:01.267198+0000","last_clean_scrub_stamp":"2024-03-24T00:13:01.267198+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T00:16:14.490270+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,9],"acting":[6,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.6","version":"88'12","reported_seq":31,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-24T00:14:07.496543+0000","last_change":"2024-03-24T00:13:01.320380+0000","last_active":"2024-03-24T00:14:07.496543+0000","last_peered":"2024-03-24T00:14:07.496543+0000","last_clean":"2024-03-24T00:14:07.496543+0000","last_became_active":"2024-03-24T00:13:01.319192+0000","last_became_peered":"2024-03-24T00:13:01.319192+0000","last_unstale":"2024-03-24T00:14:07.496543+0000","last_undegraded":"2024-03-24T00:14:07.496543+0000","last_fullsized":"2024-03-24T00:14:07.496543+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:13:00.222790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:13:00.222790+0000","last_clean_scrub_stamp":"2024-03-24T00:13:00.222790+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T02:09:27.085298+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,3],"acting":[9,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.7","version":"88'5","reported_seq":22,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-24T00:14:35.613666+0000","last_change":"2024-03-24T00:13:02.328067+0000","last_active":"2024-03-24T00:14:35.613666+0000","last_peered":"2024-03-24T00:14:35.613666+0000","last_clean":"2024-03-24T00:14:35.613666+0000","last_became_active":"2024-03-24T00:13:02.327468+0000","last_became_peered":"2024-03-24T00:13:02.327468+0000","last_unstale":"2024-03-24T00:14:35.613666+0000","last_undegraded":"2024-03-24T00:14:35.613666+0000","last_fullsized":"2024-03-24T00:14:35.613666+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:13:01.267198+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:13:01.267198+0000","last_clean_scrub_stamp":"2024-03-24T00:13:01.267198+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T01:21:58.596941+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,7],"acting":[3,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"86'7","reported_seq":27,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-24T00:14:07.800125+0000","last_change":"2024-03-24T00:13:01.309189+0000","last_active":"2024-03-24T00:14:07.800125+0000","last_peered":"2024-03-24T00:14:07.800125+0000","last_clean":"2024-03-24T00:14:07.800125+0000","last_became_active":"2024-03-24T00:13:01.308871+0000","last_became_peered":"2024-03-24T00:13:01.308871+0000","last_unstale":"2024-03-24T00:14:07.800125+0000","last_undegraded":"2024-03-24T00:14:07.800125+0000","last_fullsized":"2024-03-24T00:14:07.800125+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:13:00.222790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:13:00.222790+0000","last_clean_scrub_stamp":"2024-03-24T00:13:00.222790+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T01:00:32.610036+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,6],"acting":[4,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"88'7","reported_seq":24,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-24T00:14:35.614946+0000","last_change":"2024-03-24T00:13:02.332238+0000","last_active":"2024-03-24T00:14:35.614946+0000","last_peered":"2024-03-24T00:14:35.614946+0000","last_clean":"2024-03-24T00:14:35.614946+0000","last_became_active":"2024-03-24T00:13:02.331909+0000","last_became_peered":"2024-03-24T00:13:02.331909+0000","last_unstale":"2024-03-24T00:14:35.614946+0000","last_undegraded":"2024-03-24T00:14:35.614946+0000","last_fullsized":"2024-03-24T00:14:35.614946+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:13:01.267198+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:13:01.267198+0000","last_clean_scrub_stamp":"2024-03-24T00:13:01.267198+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T04:41:14.762251+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"86'2","reported_seq":21,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-24T00:14:07.494700+0000","last_change":"2024-03-24T00:13:01.306883+0000","last_active":"2024-03-24T00:14:07.494700+0000","last_peered":"2024-03-24T00:14:07.494700+0000","last_clean":"2024-03-24T00:14:07.494700+0000","last_became_active":"2024-03-24T00:13:01.306629+0000","last_became_peered":"2024-03-24T00:13:01.306629+0000","last_unstale":"2024-03-24T00:14:07.494700+0000","last_undegraded":"2024-03-24T00:14:07.494700+0000","last_fullsized":"2024-03-24T00:14:07.494700+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:13:00.222790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:13:00.222790+0000","last_clean_scrub_stamp":"2024-03-24T00:13:00.222790+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T08:11:52.398783+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"88'2","reported_seq":19,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-24T00:14:35.584250+0000","last_change":"2024-03-24T00:13:02.308421+0000","last_active":"2024-03-24T00:14:35.584250+0000","last_peered":"2024-03-24T00:14:35.584250+0000","last_clean":"2024-03-24T00:14:35.584250+0000","last_became_active":"2024-03-24T00:13:02.308081+0000","last_became_peered":"2024-03-24T00:13:02.308081+0000","last_unstale":"2024-03-24T00:14:35.584250+0000","last_undegraded":"2024-03-24T00:14:35.584250+0000","last_fullsized":"2024-03-24T00:14:35.584250+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:13:01.267198+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:13:01.267198+0000","last_clean_scrub_stamp":"2024-03-24T00:13:01.267198+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T06:50:42.356293+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,10],"acting":[8,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.9","version":"86'7","reported_seq":26,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-24T00:14:07.508158+0000","last_change":"2024-03-24T00:13:01.324439+0000","last_active":"2024-03-24T00:14:07.508158+0000","last_peered":"2024-03-24T00:14:07.508158+0000","last_clean":"2024-03-24T00:14:07.508158+0000","last_became_active":"2024-03-24T00:13:01.324037+0000","last_became_peered":"2024-03-24T00:13:01.324037+0000","last_unstale":"2024-03-24T00:14:07.508158+0000","last_undegraded":"2024-03-24T00:14:07.508158+0000","last_fullsized":"2024-03-24T00:14:07.508158+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:13:00.222790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:13:00.222790+0000","last_clean_scrub_stamp":"2024-03-24T00:13:00.222790+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T01:37:56.767931+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,10],"acting":[3,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.8","version":"88'9","reported_seq":26,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-24T00:14:35.605813+0000","last_change":"2024-03-24T00:13:02.313740+0000","last_active":"2024-03-24T00:14:35.605813+0000","last_peered":"2024-03-24T00:14:35.605813+0000","last_clean":"2024-03-24T00:14:35.605813+0000","last_became_active":"2024-03-24T00:13:02.313401+0000","last_became_peered":"2024-03-24T00:13:02.313401+0000","last_unstale":"2024-03-24T00:14:35.605813+0000","last_undegraded":"2024-03-24T00:14:35.605813+0000","last_fullsized":"2024-03-24T00:14:35.605813+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:13:01.267198+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:13:01.267198+0000","last_clean_scrub_stamp":"2024-03-24T00:13:01.267198+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T04:16:52.569135+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.a","version":"86'3","reported_seq":17,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-24T00:14:07.514496+0000","last_change":"2024-03-24T00:14:07.514050+0000","last_active":"2024-03-24T00:14:07.514496+0000","last_peered":"2024-03-24T00:14:07.514496+0000","last_clean":"2024-03-24T00:14:07.514496+0000","last_became_active":"2024-03-24T00:14:07.509219+0000","last_became_peered":"2024-03-24T00:14:07.509219+0000","last_unstale":"2024-03-24T00:14:07.514496+0000","last_undegraded":"2024-03-24T00:14:07.514496+0000","last_fullsized":"2024-03-24T00:14:07.514496+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:13:00.222790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:13:00.222790+0000","last_clean_scrub_stamp":"2024-03-24T00:13:00.222790+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T08:05:33.888846+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,3,8],"acting":[11,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.b","version":"88'8","reported_seq":25,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-24T00:14:35.617423+0000","last_change":"2024-03-24T00:13:02.337413+0000","last_active":"2024-03-24T00:14:35.617423+0000","last_peered":"2024-03-24T00:14:35.617423+0000","last_clean":"2024-03-24T00:14:35.617423+0000","last_became_active":"2024-03-24T00:13:02.336910+0000","last_became_peered":"2024-03-24T00:13:02.336910+0000","last_unstale":"2024-03-24T00:14:35.617423+0000","last_undegraded":"2024-03-24T00:14:35.617423+0000","last_fullsized":"2024-03-24T00:14:35.617423+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:13:01.267198+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:13:01.267198+0000","last_clean_scrub_stamp":"2024-03-24T00:13:01.267198+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T10:51:06.549979+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,10],"acting":[3,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.b","version":"86'4","reported_seq":24,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-24T00:14:07.499164+0000","last_change":"2024-03-24T00:13:01.300201+0000","last_active":"2024-03-24T00:14:07.499164+0000","last_peered":"2024-03-24T00:14:07.499164+0000","last_clean":"2024-03-24T00:14:07.499164+0000","last_became_active":"2024-03-24T00:13:01.299750+0000","last_became_peered":"2024-03-24T00:13:01.299750+0000","last_unstale":"2024-03-24T00:14:07.499164+0000","last_undegraded":"2024-03-24T00:14:07.499164+0000","last_fullsized":"2024-03-24T00:14:07.499164+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:13:00.222790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:13:00.222790+0000","last_clean_scrub_stamp":"2024-03-24T00:13:00.222790+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T01:37:32.768926+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.a","version":"88'5","reported_seq":22,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-24T00:14:35.622160+0000","last_change":"2024-03-24T00:13:02.331732+0000","last_active":"2024-03-24T00:14:35.622160+0000","last_peered":"2024-03-24T00:14:35.622160+0000","last_clean":"2024-03-24T00:14:35.622160+0000","last_became_active":"2024-03-24T00:13:02.331365+0000","last_became_peered":"2024-03-24T00:13:02.331365+0000","last_unstale":"2024-03-24T00:14:35.622160+0000","last_undegraded":"2024-03-24T00:14:35.622160+0000","last_fullsized":"2024-03-24T00:14:35.622160+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:13:01.267198+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:13:01.267198+0000","last_clean_scrub_stamp":"2024-03-24T00:13:01.267198+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:27:53.598158+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.c","version":"88'7","reported_seq":120,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-24T00:14:38.429238+0000","last_change":"2024-03-24T00:13:01.308725+0000","last_active":"2024-03-24T00:14:38.429238+0000","last_peered":"2024-03-24T00:14:38.429238+0000","last_clean":"2024-03-24T00:14:38.429238+0000","last_became_active":"2024-03-24T00:13:01.308324+0000","last_became_peered":"2024-03-24T00:13:01.308324+0000","last_unstale":"2024-03-24T00:14:38.429238+0000","last_undegraded":"2024-03-24T00:14:38.429238+0000","last_fullsized":"2024-03-24T00:14:38.429238+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:13:00.222790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:13:00.222790+0000","last_clean_scrub_stamp":"2024-03-24T00:13:00.222790+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T07:43:03.691683+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1702,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":94,"num_read_kb":3,"num_write":7,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,11,0],"acting":[2,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"88'7","reported_seq":24,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-24T00:14:35.616970+0000","last_change":"2024-03-24T00:13:02.305749+0000","last_active":"2024-03-24T00:14:35.616970+0000","last_peered":"2024-03-24T00:14:35.616970+0000","last_clean":"2024-03-24T00:14:35.616970+0000","last_became_active":"2024-03-24T00:13:02.305413+0000","last_became_peered":"2024-03-24T00:13:02.305413+0000","last_unstale":"2024-03-24T00:14:35.616970+0000","last_undegraded":"2024-03-24T00:14:35.616970+0000","last_fullsized":"2024-03-24T00:14:35.616970+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:13:01.267198+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:13:01.267198+0000","last_clean_scrub_stamp":"2024-03-24T00:13:01.267198+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T00:54:56.437725+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,11],"acting":[7,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.d","version":"86'3","reported_seq":23,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-24T00:14:07.680179+0000","last_change":"2024-03-24T00:13:01.304883+0000","last_active":"2024-03-24T00:14:07.680179+0000","last_peered":"2024-03-24T00:14:07.680179+0000","last_clean":"2024-03-24T00:14:07.680179+0000","last_became_active":"2024-03-24T00:13:01.304459+0000","last_became_peered":"2024-03-24T00:13:01.304459+0000","last_unstale":"2024-03-24T00:14:07.680179+0000","last_undegraded":"2024-03-24T00:14:07.680179+0000","last_fullsized":"2024-03-24T00:14:07.680179+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:13:00.222790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:13:00.222790+0000","last_clean_scrub_stamp":"2024-03-24T00:13:00.222790+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T06:14:22.473571+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10,3],"acting":[5,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.c","version":"88'7","reported_seq":24,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-24T00:14:35.613996+0000","last_change":"2024-03-24T00:13:02.331923+0000","last_active":"2024-03-24T00:14:35.613996+0000","last_peered":"2024-03-24T00:14:35.613996+0000","last_clean":"2024-03-24T00:14:35.613996+0000","last_became_active":"2024-03-24T00:13:02.331504+0000","last_became_peered":"2024-03-24T00:13:02.331504+0000","last_unstale":"2024-03-24T00:14:35.613996+0000","last_undegraded":"2024-03-24T00:14:35.613996+0000","last_fullsized":"2024-03-24T00:14:35.613996+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:13:01.267198+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:13:01.267198+0000","last_clean_scrub_stamp":"2024-03-24T00:13:01.267198+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T03:27:17.160248+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,5],"acting":[7,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"86'5","reported_seq":24,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-24T00:14:07.632695+0000","last_change":"2024-03-24T00:13:01.320223+0000","last_active":"2024-03-24T00:14:07.632695+0000","last_peered":"2024-03-24T00:14:07.632695+0000","last_clean":"2024-03-24T00:14:07.632695+0000","last_became_active":"2024-03-24T00:13:01.319342+0000","last_became_peered":"2024-03-24T00:13:01.319342+0000","last_unstale":"2024-03-24T00:14:07.632695+0000","last_undegraded":"2024-03-24T00:14:07.632695+0000","last_fullsized":"2024-03-24T00:14:07.632695+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:13:00.222790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:13:00.222790+0000","last_clean_scrub_stamp":"2024-03-24T00:13:00.222790+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T02:24:08.018521+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.f","version":"88'9","reported_seq":26,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-24T00:14:35.606455+0000","last_change":"2024-03-24T00:13:02.307719+0000","last_active":"2024-03-24T00:14:35.606455+0000","last_peered":"2024-03-24T00:14:35.606455+0000","last_clean":"2024-03-24T00:14:35.606455+0000","last_became_active":"2024-03-24T00:13:02.307399+0000","last_became_peered":"2024-03-24T00:13:02.307399+0000","last_unstale":"2024-03-24T00:14:35.606455+0000","last_undegraded":"2024-03-24T00:14:35.606455+0000","last_fullsized":"2024-03-24T00:14:35.606455+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:13:01.267198+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:13:01.267198+0000","last_clean_scrub_stamp":"2024-03-24T00:13:01.267198+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T09:38:13.077843+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,8],"acting":[7,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.20","version":"88'9","reported_seq":26,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-24T00:14:35.607423+0000","last_change":"2024-03-24T00:13:02.307469+0000","last_active":"2024-03-24T00:14:35.607423+0000","last_peered":"2024-03-24T00:14:35.607423+0000","last_clean":"2024-03-24T00:14:35.607423+0000","last_became_active":"2024-03-24T00:13:02.306873+0000","last_became_peered":"2024-03-24T00:13:02.306873+0000","last_unstale":"2024-03-24T00:14:35.607423+0000","last_undegraded":"2024-03-24T00:14:35.607423+0000","last_fullsized":"2024-03-24T00:14:35.607423+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:13:01.267198+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:13:01.267198+0000","last_clean_scrub_stamp":"2024-03-24T00:13:01.267198+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:19:51.163015+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,5],"acting":[11,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.21","version":"88'12","reported_seq":29,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-24T00:14:35.606261+0000","last_change":"2024-03-24T00:13:02.307472+0000","last_active":"2024-03-24T00:14:35.606261+0000","last_peered":"2024-03-24T00:14:35.606261+0000","last_clean":"2024-03-24T00:14:35.606261+0000","last_became_active":"2024-03-24T00:13:02.307099+0000","last_became_peered":"2024-03-24T00:13:02.307099+0000","last_unstale":"2024-03-24T00:14:35.606261+0000","last_undegraded":"2024-03-24T00:14:35.606261+0000","last_fullsized":"2024-03-24T00:14:35.606261+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:13:01.267198+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:13:01.267198+0000","last_clean_scrub_stamp":"2024-03-24T00:13:01.267198+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T08:49:06.461235+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,4],"acting":[2,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"88'8","reported_seq":25,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-24T00:14:35.624327+0000","last_change":"2024-03-24T00:13:02.331133+0000","last_active":"2024-03-24T00:14:35.624327+0000","last_peered":"2024-03-24T00:14:35.624327+0000","last_clean":"2024-03-24T00:14:35.624327+0000","last_became_active":"2024-03-24T00:13:02.330780+0000","last_became_peered":"2024-03-24T00:13:02.330780+0000","last_unstale":"2024-03-24T00:14:35.624327+0000","last_undegraded":"2024-03-24T00:14:35.624327+0000","last_fullsized":"2024-03-24T00:14:35.624327+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:13:01.267198+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:13:01.267198+0000","last_clean_scrub_stamp":"2024-03-24T00:13:01.267198+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T10:46:07.933131+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.23","version":"88'7","reported_seq":24,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-24T00:14:35.623964+0000","last_change":"2024-03-24T00:13:02.335406+0000","last_active":"2024-03-24T00:14:35.623964+0000","last_peered":"2024-03-24T00:14:35.623964+0000","last_clean":"2024-03-24T00:14:35.623964+0000","last_became_active":"2024-03-24T00:13:02.334994+0000","last_became_peered":"2024-03-24T00:13:02.334994+0000","last_unstale":"2024-03-24T00:14:35.623964+0000","last_undegraded":"2024-03-24T00:14:35.623964+0000","last_fullsized":"2024-03-24T00:14:35.623964+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:13:01.267198+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:13:01.267198+0000","last_clean_scrub_stamp":"2024-03-24T00:13:01.267198+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T03:24:59.547395+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,4],"acting":[0,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"88'4","reported_seq":21,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-24T00:14:35.621871+0000","last_change":"2024-03-24T00:13:02.332420+0000","last_active":"2024-03-24T00:14:35.621871+0000","last_peered":"2024-03-24T00:14:35.621871+0000","last_clean":"2024-03-24T00:14:35.621871+0000","last_became_active":"2024-03-24T00:13:02.331496+0000","last_became_peered":"2024-03-24T00:13:02.331496+0000","last_unstale":"2024-03-24T00:14:35.621871+0000","last_undegraded":"2024-03-24T00:14:35.621871+0000","last_fullsized":"2024-03-24T00:14:35.621871+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:13:01.267198+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:13:01.267198+0000","last_clean_scrub_stamp":"2024-03-24T00:13:01.267198+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T03:15:06.007947+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"88'6","reported_seq":23,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-24T00:14:35.604153+0000","last_change":"2024-03-24T00:13:02.309077+0000","last_active":"2024-03-24T00:14:35.604153+0000","last_peered":"2024-03-24T00:14:35.604153+0000","last_clean":"2024-03-24T00:14:35.604153+0000","last_became_active":"2024-03-24T00:13:02.308752+0000","last_became_peered":"2024-03-24T00:13:02.308752+0000","last_unstale":"2024-03-24T00:14:35.604153+0000","last_undegraded":"2024-03-24T00:14:35.604153+0000","last_fullsized":"2024-03-24T00:14:35.604153+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:13:01.267198+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:13:01.267198+0000","last_clean_scrub_stamp":"2024-03-24T00:13:01.267198+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T06:49:19.657988+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,5],"acting":[11,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.26","version":"88'15","reported_seq":32,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-24T00:14:35.619206+0000","last_change":"2024-03-24T00:13:02.304962+0000","last_active":"2024-03-24T00:14:35.619206+0000","last_peered":"2024-03-24T00:14:35.619206+0000","last_clean":"2024-03-24T00:14:35.619206+0000","last_became_active":"2024-03-24T00:13:02.304580+0000","last_became_peered":"2024-03-24T00:13:02.304580+0000","last_unstale":"2024-03-24T00:14:35.619206+0000","last_undegraded":"2024-03-24T00:14:35.619206+0000","last_fullsized":"2024-03-24T00:14:35.619206+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:13:01.267198+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:13:01.267198+0000","last_clean_scrub_stamp":"2024-03-24T00:13:01.267198+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:45:26.779169+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8,1],"acting":[2,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.27","version":"88'2","reported_seq":19,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-24T00:14:35.612439+0000","last_change":"2024-03-24T00:13:02.321908+0000","last_active":"2024-03-24T00:14:35.612439+0000","last_peered":"2024-03-24T00:14:35.612439+0000","last_clean":"2024-03-24T00:14:35.612439+0000","last_became_active":"2024-03-24T00:13:02.321495+0000","last_became_peered":"2024-03-24T00:13:02.321495+0000","last_unstale":"2024-03-24T00:14:35.612439+0000","last_undegraded":"2024-03-24T00:14:35.612439+0000","last_fullsized":"2024-03-24T00:14:35.612439+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:13:01.267198+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:13:01.267198+0000","last_clean_scrub_stamp":"2024-03-24T00:13:01.267198+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T08:43:29.261314+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"88'6","reported_seq":23,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-24T00:14:35.621983+0000","last_change":"2024-03-24T00:13:02.331966+0000","last_active":"2024-03-24T00:14:35.621983+0000","last_peered":"2024-03-24T00:14:35.621983+0000","last_clean":"2024-03-24T00:14:35.621983+0000","last_became_active":"2024-03-24T00:13:02.331633+0000","last_became_peered":"2024-03-24T00:13:02.331633+0000","last_unstale":"2024-03-24T00:14:35.621983+0000","last_undegraded":"2024-03-24T00:14:35.621983+0000","last_fullsized":"2024-03-24T00:14:35.621983+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:13:01.267198+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:13:01.267198+0000","last_clean_scrub_stamp":"2024-03-24T00:13:01.267198+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T05:04:01.574351+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,3],"acting":[5,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"88'8","reported_seq":25,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-24T00:14:35.617213+0000","last_change":"2024-03-24T00:13:02.330344+0000","last_active":"2024-03-24T00:14:35.617213+0000","last_peered":"2024-03-24T00:14:35.617213+0000","last_clean":"2024-03-24T00:14:35.617213+0000","last_became_active":"2024-03-24T00:13:02.328218+0000","last_became_peered":"2024-03-24T00:13:02.328218+0000","last_unstale":"2024-03-24T00:14:35.617213+0000","last_undegraded":"2024-03-24T00:14:35.617213+0000","last_fullsized":"2024-03-24T00:14:35.617213+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:13:01.267198+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:13:01.267198+0000","last_clean_scrub_stamp":"2024-03-24T00:13:01.267198+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T08:54:36.392397+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,11],"acting":[3,2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"88'10","reported_seq":27,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-24T00:14:35.627624+0000","last_change":"2024-03-24T00:13:02.330956+0000","last_active":"2024-03-24T00:14:35.627624+0000","last_peered":"2024-03-24T00:14:35.627624+0000","last_clean":"2024-03-24T00:14:35.627624+0000","last_became_active":"2024-03-24T00:13:02.330642+0000","last_became_peered":"2024-03-24T00:13:02.330642+0000","last_unstale":"2024-03-24T00:14:35.627624+0000","last_undegraded":"2024-03-24T00:14:35.627624+0000","last_fullsized":"2024-03-24T00:14:35.627624+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:13:01.267198+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:13:01.267198+0000","last_clean_scrub_stamp":"2024-03-24T00:13:01.267198+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:41:11.326722+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2b","version":"88'11","reported_seq":28,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-24T00:14:35.623864+0000","last_change":"2024-03-24T00:13:02.333026+0000","last_active":"2024-03-24T00:14:35.623864+0000","last_peered":"2024-03-24T00:14:35.623864+0000","last_clean":"2024-03-24T00:14:35.623864+0000","last_became_active":"2024-03-24T00:13:02.331232+0000","last_became_peered":"2024-03-24T00:13:02.331232+0000","last_unstale":"2024-03-24T00:14:35.623864+0000","last_undegraded":"2024-03-24T00:14:35.623864+0000","last_fullsized":"2024-03-24T00:14:35.623864+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:13:01.267198+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:13:01.267198+0000","last_clean_scrub_stamp":"2024-03-24T00:13:01.267198+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T02:14:18.504012+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2c","version":"88'8","reported_seq":25,"reported_epoch":88,"state":"active+clean","last_fresh":"2024-03-24T00:14:35.602677+0000","last_change":"2024-03-24T00:13:02.309023+0000","last_active":"2024-03-24T00:14:35.602677+0000","last_peered":"2024-03-24T00:14:35.602677+0000","last_clean":"2024-03-24T00:14:35.602677+0000","last_became_active":"2024-03-24T00:13:02.308236+0000","last_became_peered":"2024-03-24T00:13:02.308236+0000","last_unstale":"2024-03-24T00:14:35.602677+0000","last_undegraded":"2024-03-24T00:14:35.602677+0000","last_fullsized":"2024-03-24T00:14:35.602677+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:13:01.267198+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:13:01.267198+0000","last_clean_scrub_stamp":"2024-03-24T00:13:01.267198+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T03:30:37.427095+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,8],"acting":[11,4,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":234,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":702,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":502,"ondisk_log_size":502,"up":192,"acting":192,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":442368,"data_stored":4722944,"data_compressed":284592,"data_compressed_allocated":360448,"data_compressed_original":4669440,"omap_allocated":0,"internal_metadata":0},"log_size":137,"ondisk_log_size":137,"up":3,"acting":3,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":69436,"num_objects":98,"num_object_clones":0,"num_object_copies":294,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":98,"num_whiteouts":0,"num_read":963,"num_read_kb":556,"num_write":246,"num_write_kb":143,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14,"num_bytes_recovered":62373,"num_keys_recovered":0,"num_objects_omap":66,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":512000,"data_stored":208308,"data_compressed":13143,"data_compressed_allocated":20480,"data_compressed_original":73728,"omap_allocated":0,"internal_metadata":0},"log_size":215,"ondisk_log_size":215,"up":96,"acting":96,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":81,"seq":347892351009,"num_pgs":19,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27952,"kb_used_data":936,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717200,"statfs":{"total":95995035648,"available":95966412800,"internally_reserved":0,"allocated":958464,"data_stored":461088,"data_compressed":5786,"data_compressed_allocated":8192,"data_compressed_original":16384,"omap_allocated":1596,"internal_metadata":27654596},"hb_peers":[0,1,2,3,4,5,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":10,"up_from":73,"seq":313532612646,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28100,"kb_used_data":1084,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717052,"statfs":{"total":95995035648,"available":95966261248,"internally_reserved":0,"allocated":1110016,"data_stored":1113535,"data_compressed":43694,"data_compressed_allocated":57344,"data_compressed_original":614400,"omap_allocated":1599,"internal_metadata":27654593},"hb_peers":[0,1,2,3,4,5,6,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":9,"up_from":68,"seq":292057776172,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32136,"kb_used_data":960,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713016,"statfs":{"total":95995035648,"available":95962128384,"internally_reserved":0,"allocated":983040,"data_stored":460486,"data_compressed":5786,"data_compressed_allocated":8192,"data_compressed_original":16384,"omap_allocated":1600,"internal_metadata":31914432},"hb_peers":[0,1,2,3,4,5,6,7,8,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":60,"seq":257698037811,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36284,"kb_used_data":948,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708868,"statfs":{"total":95995035648,"available":95957880832,"internally_reserved":0,"allocated":970752,"data_stored":460192,"data_compressed":5786,"data_compressed_allocated":8192,"data_compressed_original":16384,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":52,"seq":223338299449,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28024,"kb_used_data":1008,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717128,"statfs":{"total":95995035648,"available":95966339072,"internally_reserved":0,"allocated":1032192,"data_stored":1053523,"data_compressed":41975,"data_compressed_allocated":53248,"data_compressed_original":606208,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":47,"seq":201863462975,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27960,"kb_used_data":944,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717192,"statfs":{"total":95995035648,"available":95966404608,"internally_reserved":0,"allocated":966656,"data_stored":460203,"data_compressed":5786,"data_compressed_allocated":8192,"data_compressed_original":16384,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574938,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28024,"kb_used_data":1008,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717128,"statfs":{"total":95995035648,"available":95966339072,"internally_reserved":0,"allocated":1032192,"data_stored":523738,"data_compressed":7505,"data_compressed_allocated":12288,"data_compressed_original":24576,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738464,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27980,"kb_used_data":964,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717172,"statfs":{"total":95995035648,"available":95966384128,"internally_reserved":0,"allocated":987136,"data_stored":462074,"data_compressed":5786,"data_compressed_allocated":8192,"data_compressed_original":16384,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411413,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27972,"kb_used_data":956,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717180,"statfs":{"total":95995035648,"available":95966392320,"internally_reserved":0,"allocated":978944,"data_stored":462400,"data_compressed":5786,"data_compressed_allocated":8192,"data_compressed_original":16384,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":3,"up_from":26,"seq":111669149776,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32136,"kb_used_data":960,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713016,"statfs":{"total":95995035648,"available":95962128384,"internally_reserved":0,"allocated":983040,"data_stored":460028,"data_compressed":5786,"data_compressed_allocated":8192,"data_compressed_original":16384,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":34,"seq":146028888137,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32144,"kb_used_data":968,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93713008,"statfs":{"total":95995035648,"available":95962120192,"internally_reserved":0,"allocated":991232,"data_stored":522014,"data_compressed":15491,"data_compressed_allocated":20480,"data_compressed_original":73728,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":5,"up_from":39,"seq":167503724612,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28036,"kb_used_data":1020,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717116,"statfs":{"total":95995035648,"available":95966326784,"internally_reserved":0,"allocated":1044480,"data_stored":1053201,"data_compressed":41975,"data_compressed_allocated":53248,"data_compressed_original":606208,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34549,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34549,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34549,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":36189,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":36189,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":36189,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":36189,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":36189,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":2710,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":86016,"data_stored":64374,"data_compressed":1719,"data_compressed_allocated":4096,"data_compressed_original":8192,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":3036,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":430,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":62650,"data_compressed":9705,"data_compressed_allocated":12288,"data_compressed_original":57344,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":3352,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":722,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":3674,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":828,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":1122,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":110592,"data_stored":63686,"data_compressed":1719,"data_compressed_allocated":4096,"data_compressed_original":8192,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1724,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":234,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-24T00:14:45.010 INFO:tasks.cephadm.ceph_manager.ceph:clean! 2024-03-24T00:14:45.010 INFO:tasks.ceph:Waiting until ceph cluster ceph is healthy... 2024-03-24T00:14:45.010 INFO:tasks.cephadm.ceph_manager.ceph:wait_until_healthy 2024-03-24T00:14:45.010 DEBUG:teuthology.orchestra.run.smithi118:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 shell --fsid d79a5420-e971-11ee-95cd-87774f69a715 -- ceph health --format=json 2024-03-24T00:14:45.087 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:14:44 smithi118 ceph-mon[31535]: pgmap v367: 97 pgs: 97 active+clean; 645 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 597 B/s wr, 10 op/s 2024-03-24T00:14:45.203 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:14:44 smithi189 ceph-mon[36187]: pgmap v367: 97 pgs: 97 active+clean; 645 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 597 B/s wr, 10 op/s 2024-03-24T00:14:45.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:14:44 smithi193 ceph-mon[36878]: pgmap v367: 97 pgs: 97 active+clean; 645 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 597 B/s wr, 10 op/s 2024-03-24T00:14:45.446 INFO:teuthology.orchestra.run.smithi118.stderr:Inferring config /var/lib/ceph/d79a5420-e971-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T00:14:45.849 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:14:45 smithi118 ceph-mon[31535]: from='client.24881 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-24T00:14:46.203 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:14:45 smithi189 ceph-mon[36187]: from='client.24881 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-24T00:14:46.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:14:45 smithi193 ceph-mon[36878]: from='client.24881 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-24T00:14:46.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:14:46 smithi118 ceph-mon[31535]: pgmap v368: 97 pgs: 97 active+clean; 645 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 5.7 KiB/s rd, 597 B/s wr, 11 op/s 2024-03-24T00:14:47.203 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:14:46 smithi189 ceph-mon[36187]: pgmap v368: 97 pgs: 97 active+clean; 645 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 5.7 KiB/s rd, 597 B/s wr, 11 op/s 2024-03-24T00:14:47.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:14:46 smithi193 ceph-mon[36878]: pgmap v368: 97 pgs: 97 active+clean; 645 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 5.7 KiB/s rd, 597 B/s wr, 11 op/s 2024-03-24T00:14:48.094 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:14:47 smithi118 ceph-mon[31535]: pgmap v369: 97 pgs: 97 active+clean; 645 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 4.8 KiB/s rd, 597 B/s wr, 10 op/s 2024-03-24T00:14:48.138 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:48.137+0000 7f9e5d400700 1 -- 172.21.15.118:0/579950013 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f9e581045f0 msgr2=0x7f9e581049d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:14:48.139 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:48.137+0000 7f9e5d400700 1 --2- 172.21.15.118:0/579950013 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f9e581045f0 0x7f9e581049d0 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7f9e480099c0 tx=0x7f9e48009cd0 comp rx=0 tx=0).stop 2024-03-24T00:14:48.139 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:48.137+0000 7f9e5d400700 1 -- 172.21.15.118:0/579950013 shutdown_connections 2024-03-24T00:14:48.139 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:48.137+0000 7f9e5d400700 1 --2- 172.21.15.118:0/579950013 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9e581096d0 0x7f9e581112a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:14:48.139 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:48.137+0000 7f9e5d400700 1 --2- 172.21.15.118:0/579950013 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f9e581045f0 0x7f9e581049d0 unknown :-1 s=CLOSED pgs=166 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:14:48.139 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:48.137+0000 7f9e5d400700 1 --2- 172.21.15.118:0/579950013 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f9e58104fa0 0x7f9e58108ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:14:48.139 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:48.137+0000 7f9e5d400700 1 -- 172.21.15.118:0/579950013 >> 172.21.15.118:0/579950013 conn(0x7f9e580ffe60 msgr2=0x7f9e58102280 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:14:48.139 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:48.137+0000 7f9e5d400700 1 -- 172.21.15.118:0/579950013 shutdown_connections 2024-03-24T00:14:48.139 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:48.137+0000 7f9e5d400700 1 -- 172.21.15.118:0/579950013 wait complete. 2024-03-24T00:14:48.140 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:48.138+0000 7f9e5d400700 1 Processor -- start 2024-03-24T00:14:48.140 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:48.139+0000 7f9e5d400700 1 -- start start 2024-03-24T00:14:48.140 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:48.139+0000 7f9e5d400700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f9e581045f0 0x7f9e580766a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:14:48.141 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:48.139+0000 7f9e5d400700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f9e58104fa0 0x7f9e58076be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:14:48.141 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:48.139+0000 7f9e5d400700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9e581096d0 0x7f9e58077120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:14:48.141 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:48.139+0000 7f9e5d400700 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9e581146b0 con 0x7f9e58104fa0 2024-03-24T00:14:48.141 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:48.139+0000 7f9e5d400700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9e58114530 con 0x7f9e581096d0 2024-03-24T00:14:48.141 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:48.139+0000 7f9e5d400700 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9e58114830 con 0x7f9e581045f0 2024-03-24T00:14:48.141 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:48.139+0000 7f9e56800700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f9e581045f0 0x7f9e580766a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:14:48.141 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:48.139+0000 7f9e56800700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f9e581045f0 0x7f9e580766a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.118:60348/0 (socket says 172.21.15.118:60348) 2024-03-24T00:14:48.141 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:48.139+0000 7f9e56800700 1 -- 172.21.15.118:0/4024009606 learned_addr learned my addr 172.21.15.118:0/4024009606 (peer_addr_for_me v2:172.21.15.118:0/0) 2024-03-24T00:14:48.141 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:48.139+0000 7f9e57200700 1 --2- 172.21.15.118:0/4024009606 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9e581096d0 0x7f9e58077120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:14:48.141 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:48.139+0000 7f9e55e00700 1 --2- 172.21.15.118:0/4024009606 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f9e58104fa0 0x7f9e58076be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:14:48.141 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:48.140+0000 7f9e56800700 1 -- 172.21.15.118:0/4024009606 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9e581096d0 msgr2=0x7f9e58077120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:14:48.141 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:48.140+0000 7f9e56800700 1 --2- 172.21.15.118:0/4024009606 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9e581096d0 0x7f9e58077120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:14:48.141 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:48.140+0000 7f9e56800700 1 -- 172.21.15.118:0/4024009606 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f9e58104fa0 msgr2=0x7f9e58076be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:14:48.141 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:48.140+0000 7f9e56800700 1 --2- 172.21.15.118:0/4024009606 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f9e58104fa0 0x7f9e58076be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:14:48.142 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:48.140+0000 7f9e56800700 1 -- 172.21.15.118:0/4024009606 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9e581ab1d0 con 0x7f9e581045f0 2024-03-24T00:14:48.142 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:48.140+0000 7f9e56800700 1 --2- 172.21.15.118:0/4024009606 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f9e581045f0 0x7f9e580766a0 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f9e48000c00 tx=0x7f9e4800fa00 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:14:48.143 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:48.140+0000 7f9e4f400700 1 -- 172.21.15.118:0/4024009606 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9e48027070 con 0x7f9e581045f0 2024-03-24T00:14:48.143 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:48.140+0000 7f9e4f400700 1 -- 172.21.15.118:0/4024009606 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f9e48003680 con 0x7f9e581045f0 2024-03-24T00:14:48.143 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:48.140+0000 7f9e4f400700 1 -- 172.21.15.118:0/4024009606 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9e480173f0 con 0x7f9e581045f0 2024-03-24T00:14:48.143 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:48.140+0000 7f9e5d400700 1 -- 172.21.15.118:0/4024009606 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9e581ab460 con 0x7f9e581045f0 2024-03-24T00:14:48.143 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:48.141+0000 7f9e5d400700 1 -- 172.21.15.118:0/4024009606 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9e581ab860 con 0x7f9e581045f0 2024-03-24T00:14:48.143 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:48.142+0000 7f9e5d400700 1 -- 172.21.15.118:0/4024009606 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9e58105ad0 con 0x7f9e581045f0 2024-03-24T00:14:48.144 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:48.143+0000 7f9e4f400700 1 -- 172.21.15.118:0/4024009606 <== mon.1 v2:172.21.15.193:3300/0 4 ==== mgrmap(e 18) v1 ==== 137291+0+0 (secure 0 0 0) 0x7f9e4802a070 con 0x7f9e581045f0 2024-03-24T00:14:48.145 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:48.145+0000 7f9e4f400700 1 --2- 172.21.15.118:0/4024009606 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7f9e3c0a2910 0x7f9e3c0a4dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:14:48.145 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:48.145+0000 7f9e4f400700 1 -- 172.21.15.118:0/4024009606 <== mon.1 v2:172.21.15.193:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 8248+0+0 (secure 0 0 0) 0x7f9e480cf5b0 con 0x7f9e581045f0 2024-03-24T00:14:48.146 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:48.145+0000 7f9e55e00700 1 --2- 172.21.15.118:0/4024009606 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7f9e3c0a2910 0x7f9e3c0a4dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:14:48.146 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:48.146+0000 7f9e55e00700 1 --2- 172.21.15.118:0/4024009606 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7f9e3c0a2910 0x7f9e3c0a4dd0 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7f9e58077e60 tx=0x7f9e44009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:14:48.149 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:48.148+0000 7f9e4f400700 1 -- 172.21.15.118:0/4024009606 <== mon.1 v2:172.21.15.193:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f9e48013070 con 0x7f9e581045f0 2024-03-24T00:14:48.203 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:14:47 smithi189 ceph-mon[36187]: pgmap v369: 97 pgs: 97 active+clean; 645 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 4.8 KiB/s rd, 597 B/s wr, 10 op/s 2024-03-24T00:14:48.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:14:47 smithi193 ceph-mon[36878]: pgmap v369: 97 pgs: 97 active+clean; 645 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 4.8 KiB/s rd, 597 B/s wr, 10 op/s 2024-03-24T00:14:48.418 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:48.418+0000 7f9e5d400700 1 -- 172.21.15.118:0/4024009606 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f9e5802da30 con 0x7f9e581045f0 2024-03-24T00:14:48.419 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:48.418+0000 7f9e4f400700 1 -- 172.21.15.118:0/4024009606 <== mon.1 v2:172.21.15.193:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+46 (secure 0 0 0) 0x7f9e4809c670 con 0x7f9e581045f0 2024-03-24T00:14:48.419 INFO:teuthology.orchestra.run.smithi118.stdout: 2024-03-24T00:14:48.419 INFO:teuthology.orchestra.run.smithi118.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2024-03-24T00:14:48.423 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:48.422+0000 7f9e5d400700 1 -- 172.21.15.118:0/4024009606 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7f9e3c0a2910 msgr2=0x7f9e3c0a4dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:14:48.423 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:48.422+0000 7f9e5d400700 1 --2- 172.21.15.118:0/4024009606 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7f9e3c0a2910 0x7f9e3c0a4dd0 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7f9e58077e60 tx=0x7f9e44009260 comp rx=0 tx=0).stop 2024-03-24T00:14:48.423 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:48.422+0000 7f9e5d400700 1 -- 172.21.15.118:0/4024009606 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f9e581045f0 msgr2=0x7f9e580766a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:14:48.423 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:48.422+0000 7f9e5d400700 1 --2- 172.21.15.118:0/4024009606 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f9e581045f0 0x7f9e580766a0 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7f9e48000c00 tx=0x7f9e4800fa00 comp rx=0 tx=0).stop 2024-03-24T00:14:48.423 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:48.422+0000 7f9e5d400700 1 -- 172.21.15.118:0/4024009606 shutdown_connections 2024-03-24T00:14:48.423 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:48.422+0000 7f9e5d400700 1 --2- 172.21.15.118:0/4024009606 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9e581096d0 0x7f9e58077120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:14:48.423 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:48.422+0000 7f9e5d400700 1 --2- 172.21.15.118:0/4024009606 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7f9e3c0a2910 0x7f9e3c0a4dd0 unknown :-1 s=CLOSED pgs=207 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:14:48.424 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:48.422+0000 7f9e5d400700 1 --2- 172.21.15.118:0/4024009606 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f9e581045f0 0x7f9e580766a0 unknown :-1 s=CLOSED pgs=126 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:14:48.424 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:48.422+0000 7f9e5d400700 1 --2- 172.21.15.118:0/4024009606 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f9e58104fa0 0x7f9e58076be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:14:48.424 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:48.422+0000 7f9e5d400700 1 -- 172.21.15.118:0/4024009606 >> 172.21.15.118:0/4024009606 conn(0x7f9e580ffe60 msgr2=0x7f9e58102120 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:14:48.424 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:48.422+0000 7f9e5d400700 1 -- 172.21.15.118:0/4024009606 shutdown_connections 2024-03-24T00:14:48.424 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:48.423+0000 7f9e5d400700 1 -- 172.21.15.118:0/4024009606 wait complete. 2024-03-24T00:14:49.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:14:48 smithi118 ceph-mon[31535]: from='client.? 172.21.15.118:0/4024009606' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-03-24T00:14:49.203 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:14:48 smithi189 ceph-mon[36187]: from='client.? 172.21.15.118:0/4024009606' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-03-24T00:14:49.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:14:48 smithi193 ceph-mon[36878]: from='client.? 172.21.15.118:0/4024009606' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-03-24T00:14:49.611 INFO:tasks.cephadm.ceph_manager.ceph:wait_until_healthy done 2024-03-24T00:14:49.612 DEBUG:teuthology.orchestra.run.smithi118:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-24T00:14:49.835 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:49.834+0000 7f5479400640 1 -- 172.21.15.118:0/1922880723 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f5474077c40 msgr2=0x7f54740784e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:14:49.835 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:49.834+0000 7f5479400640 1 --2- 172.21.15.118:0/1922880723 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f5474077c40 0x7f54740784e0 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f546c009900 tx=0x7f546c02bc30 comp rx=0 tx=0).stop 2024-03-24T00:14:49.835 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:49.834+0000 7f5479400640 1 -- 172.21.15.118:0/1922880723 shutdown_connections 2024-03-24T00:14:49.835 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:49.834+0000 7f5479400640 1 --2- 172.21.15.118:0/1922880723 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f5474077c40 0x7f54740784e0 unknown :-1 s=CLOSED pgs=127 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:14:49.835 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:49.834+0000 7f5479400640 1 --2- 172.21.15.118:0/1922880723 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f54740772a0 0x7f5474077700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:14:49.835 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:49.834+0000 7f5479400640 1 --2- 172.21.15.118:0/1922880723 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f5474076060 0x7f5474076480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:14:49.835 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:49.834+0000 7f5479400640 1 -- 172.21.15.118:0/1922880723 >> 172.21.15.118:0/1922880723 conn(0x7f54740fee50 msgr2=0x7f5474101290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:14:49.835 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:49.834+0000 7f5479400640 1 -- 172.21.15.118:0/1922880723 shutdown_connections 2024-03-24T00:14:49.835 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:49.834+0000 7f5479400640 1 -- 172.21.15.118:0/1922880723 wait complete. 2024-03-24T00:14:49.837 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:49.835+0000 7f5479400640 1 Processor -- start 2024-03-24T00:14:49.837 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:49.835+0000 7f5479400640 1 -- start start 2024-03-24T00:14:49.837 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:49.835+0000 7f5479400640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5474076060 0x7f547419df40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:14:49.838 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:49.835+0000 7f5479400640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f54740772a0 0x7f547419e480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:14:49.838 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:49.835+0000 7f5479400640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f5474077c40 0x7f54741a54f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:14:49.838 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:49.835+0000 7f5479400640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f54741139b0 con 0x7f54740772a0 2024-03-24T00:14:49.838 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:49.835+0000 7f5479400640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5474113830 con 0x7f5474076060 2024-03-24T00:14:49.838 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:49.835+0000 7f5479400640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5474113b30 con 0x7f5474077c40 2024-03-24T00:14:49.838 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:49.836+0000 7f5472800640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5474076060 0x7f547419df40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:14:49.838 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:49.836+0000 7f5473200640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f5474077c40 0x7f54741a54f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:14:49.838 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:49.836+0000 7f5469e00640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f54740772a0 0x7f547419e480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:14:49.838 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:49.836+0000 7f5472800640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5474076060 0x7f547419df40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.118:55650/0 (socket says 172.21.15.118:55650) 2024-03-24T00:14:49.838 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:49.836+0000 7f5472800640 1 -- 172.21.15.118:0/3436596878 learned_addr learned my addr 172.21.15.118:0/3436596878 (peer_addr_for_me v2:172.21.15.118:0/0) 2024-03-24T00:14:49.838 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:49.836+0000 7f5472800640 1 -- 172.21.15.118:0/3436596878 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f5474077c40 msgr2=0x7f54741a54f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:14:49.838 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:49.836+0000 7f5472800640 1 --2- 172.21.15.118:0/3436596878 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f5474077c40 0x7f54741a54f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:14:49.838 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:49.836+0000 7f5472800640 1 -- 172.21.15.118:0/3436596878 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f54740772a0 msgr2=0x7f547419e480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:14:49.838 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:49.836+0000 7f5472800640 1 --2- 172.21.15.118:0/3436596878 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f54740772a0 0x7f547419e480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:14:49.839 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:49.836+0000 7f5472800640 1 -- 172.21.15.118:0/3436596878 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f54741a5b60 con 0x7f5474076060 2024-03-24T00:14:49.839 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:49.836+0000 7f5472800640 1 --2- 172.21.15.118:0/3436596878 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5474076060 0x7f547419df40 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f546400e9f0 tx=0x7f546400eeb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:14:49.839 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:49.837+0000 7f546be00640 1 -- 172.21.15.118:0/3436596878 <== mon.2 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f546400cd80 con 0x7f5474076060 2024-03-24T00:14:49.839 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:49.837+0000 7f546be00640 1 -- 172.21.15.118:0/3436596878 <== mon.2 v2:172.21.15.189:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f5464010910 con 0x7f5474076060 2024-03-24T00:14:49.839 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:49.837+0000 7f5479400640 1 -- 172.21.15.118:0/3436596878 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f54741a5e50 con 0x7f5474076060 2024-03-24T00:14:49.839 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:49.837+0000 7f546be00640 1 -- 172.21.15.118:0/3436596878 <== mon.2 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5464018990 con 0x7f5474076060 2024-03-24T00:14:49.839 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:49.837+0000 7f5479400640 1 -- 172.21.15.118:0/3436596878 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f54741a62c0 con 0x7f5474076060 2024-03-24T00:14:49.839 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:49.838+0000 7f5479400640 1 -- 172.21.15.118:0/3436596878 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5474110320 con 0x7f5474076060 2024-03-24T00:14:49.840 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:49.840+0000 7f546be00640 1 -- 172.21.15.118:0/3436596878 <== mon.2 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 18) v1 ==== 137291+0+0 (secure 0 0 0) 0x7f5464010ac0 con 0x7f5474076060 2024-03-24T00:14:49.841 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:49.841+0000 7f546be00640 1 --2- 172.21.15.118:0/3436596878 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7f54580a2880 0x7f54580a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:14:49.842 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:49.841+0000 7f546be00640 1 -- 172.21.15.118:0/3436596878 <== mon.2 v2:172.21.15.189:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 8248+0+0 (secure 0 0 0) 0x7f54640ced00 con 0x7f5474076060 2024-03-24T00:14:49.842 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:49.841+0000 7f5469e00640 1 --2- 172.21.15.118:0/3436596878 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7f54580a2880 0x7f54580a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:14:49.842 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:49.842+0000 7f5469e00640 1 --2- 172.21.15.118:0/3436596878 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7f54580a2880 0x7f54580a4d40 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7f547419f450 tx=0x7f545c009500 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:14:49.850 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:49.849+0000 7f546be00640 1 -- 172.21.15.118:0/3436596878 <== mon.2 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f546409b740 con 0x7f5474076060 2024-03-24T00:14:50.058 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:50.057+0000 7f5479400640 1 -- 172.21.15.118:0/3436596878 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f54740623a0 con 0x7f5474076060 2024-03-24T00:14:50.059 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:50.059+0000 7f546be00640 1 -- 172.21.15.118:0/3436596878 <== mon.2 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v88) v1 ==== 74+0+22730 (secure 0 0 0) 0x7f546409f7e0 con 0x7f5474076060 2024-03-24T00:14:50.060 INFO:teuthology.orchestra.run.smithi118.stdout: 2024-03-24T00:14:50.060 INFO:teuthology.orchestra.run.smithi118.stdout:{"epoch":88,"fsid":"d79a5420-e971-11ee-95cd-87774f69a715","created":"2024-03-24T00:03:58.648524+0000","modified":"2024-03-24T00:14:07.487993+0000","last_up_change":"2024-03-24T00:12:03.809000+0000","last_in_change":"2024-03-24T00:11:44.389344+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-24T00:07:50.190965+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-24T00:12:59.993723+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-24T00:13:01.257347+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"86","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"c9590426-3901-4f68-a5ef-d57aa8fbac92","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6802","nonce":557690061},{"type":"v1","addr":"172.21.15.118:6803","nonce":557690061}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6804","nonce":557690061},{"type":"v1","addr":"172.21.15.118:6805","nonce":557690061}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6808","nonce":557690061},{"type":"v1","addr":"172.21.15.118:6809","nonce":557690061}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6806","nonce":557690061},{"type":"v1","addr":"172.21.15.118:6807","nonce":557690061}]},"public_addr":"172.21.15.118:6803/557690061","cluster_addr":"172.21.15.118:6805/557690061","heartbeat_back_addr":"172.21.15.118:6809/557690061","heartbeat_front_addr":"172.21.15.118:6807/557690061","state":["exists","up"]},{"osd":1,"uuid":"fc7f827e-ecf6-4661-aad0-7a22951892e0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6800","nonce":4203029485},{"type":"v1","addr":"172.21.15.189:6801","nonce":4203029485}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6802","nonce":4203029485},{"type":"v1","addr":"172.21.15.189:6803","nonce":4203029485}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6806","nonce":4203029485},{"type":"v1","addr":"172.21.15.189:6807","nonce":4203029485}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6804","nonce":4203029485},{"type":"v1","addr":"172.21.15.189:6805","nonce":4203029485}]},"public_addr":"172.21.15.189:6801/4203029485","cluster_addr":"172.21.15.189:6803/4203029485","heartbeat_back_addr":"172.21.15.189:6807/4203029485","heartbeat_front_addr":"172.21.15.189:6805/4203029485","state":["exists","up"]},{"osd":2,"uuid":"e73ccfe9-0e5b-4734-a154-a2f0d829fcfb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6800","nonce":3699076971},{"type":"v1","addr":"172.21.15.193:6801","nonce":3699076971}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6802","nonce":3699076971},{"type":"v1","addr":"172.21.15.193:6803","nonce":3699076971}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6806","nonce":3699076971},{"type":"v1","addr":"172.21.15.193:6807","nonce":3699076971}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6804","nonce":3699076971},{"type":"v1","addr":"172.21.15.193:6805","nonce":3699076971}]},"public_addr":"172.21.15.193:6801/3699076971","cluster_addr":"172.21.15.193:6803/3699076971","heartbeat_back_addr":"172.21.15.193:6807/3699076971","heartbeat_front_addr":"172.21.15.193:6805/3699076971","state":["exists","up"]},{"osd":3,"uuid":"63fed43b-43ce-4031-8957-7fe94725c1ed","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6810","nonce":4059617010},{"type":"v1","addr":"172.21.15.118:6811","nonce":4059617010}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6812","nonce":4059617010},{"type":"v1","addr":"172.21.15.118:6813","nonce":4059617010}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6816","nonce":4059617010},{"type":"v1","addr":"172.21.15.118:6817","nonce":4059617010}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6814","nonce":4059617010},{"type":"v1","addr":"172.21.15.118:6815","nonce":4059617010}]},"public_addr":"172.21.15.118:6811/4059617010","cluster_addr":"172.21.15.118:6813/4059617010","heartbeat_back_addr":"172.21.15.118:6817/4059617010","heartbeat_front_addr":"172.21.15.118:6815/4059617010","state":["exists","up"]},{"osd":4,"uuid":"57377f66-c3b4-43c1-8e8e-bc873672906d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":34,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6808","nonce":4075247068},{"type":"v1","addr":"172.21.15.189:6809","nonce":4075247068}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6810","nonce":4075247068},{"type":"v1","addr":"172.21.15.189:6811","nonce":4075247068}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6814","nonce":4075247068},{"type":"v1","addr":"172.21.15.189:6815","nonce":4075247068}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6812","nonce":4075247068},{"type":"v1","addr":"172.21.15.189:6813","nonce":4075247068}]},"public_addr":"172.21.15.189:6809/4075247068","cluster_addr":"172.21.15.189:6811/4075247068","heartbeat_back_addr":"172.21.15.189:6815/4075247068","heartbeat_front_addr":"172.21.15.189:6813/4075247068","state":["exists","up"]},{"osd":5,"uuid":"91970976-2193-4224-81c0-dc96cb022c85","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":39,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6808","nonce":2920363402},{"type":"v1","addr":"172.21.15.193:6809","nonce":2920363402}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6810","nonce":2920363402},{"type":"v1","addr":"172.21.15.193:6811","nonce":2920363402}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6814","nonce":2920363402},{"type":"v1","addr":"172.21.15.193:6815","nonce":2920363402}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6812","nonce":2920363402},{"type":"v1","addr":"172.21.15.193:6813","nonce":2920363402}]},"public_addr":"172.21.15.193:6809/2920363402","cluster_addr":"172.21.15.193:6811/2920363402","heartbeat_back_addr":"172.21.15.193:6815/2920363402","heartbeat_front_addr":"172.21.15.193:6813/2920363402","state":["exists","up"]},{"osd":6,"uuid":"0b77b196-8674-41c6-ad3d-726b5d01c8fd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":47,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6818","nonce":431354576},{"type":"v1","addr":"172.21.15.118:6819","nonce":431354576}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6820","nonce":431354576},{"type":"v1","addr":"172.21.15.118:6821","nonce":431354576}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6824","nonce":431354576},{"type":"v1","addr":"172.21.15.118:6825","nonce":431354576}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6822","nonce":431354576},{"type":"v1","addr":"172.21.15.118:6823","nonce":431354576}]},"public_addr":"172.21.15.118:6819/431354576","cluster_addr":"172.21.15.118:6821/431354576","heartbeat_back_addr":"172.21.15.118:6825/431354576","heartbeat_front_addr":"172.21.15.118:6823/431354576","state":["exists","up"]},{"osd":7,"uuid":"1fbbd38f-4e8e-4d4c-8e98-66f6862bc4bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":52,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6816","nonce":3667611284},{"type":"v1","addr":"172.21.15.189:6817","nonce":3667611284}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6818","nonce":3667611284},{"type":"v1","addr":"172.21.15.189:6819","nonce":3667611284}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6822","nonce":3667611284},{"type":"v1","addr":"172.21.15.189:6823","nonce":3667611284}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6820","nonce":3667611284},{"type":"v1","addr":"172.21.15.189:6821","nonce":3667611284}]},"public_addr":"172.21.15.189:6817/3667611284","cluster_addr":"172.21.15.189:6819/3667611284","heartbeat_back_addr":"172.21.15.189:6823/3667611284","heartbeat_front_addr":"172.21.15.189:6821/3667611284","state":["exists","up"]},{"osd":8,"uuid":"6462c2b5-e4b3-4b0b-bcfa-4de2c89e93e0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":60,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6816","nonce":3853800910},{"type":"v1","addr":"172.21.15.193:6817","nonce":3853800910}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6818","nonce":3853800910},{"type":"v1","addr":"172.21.15.193:6819","nonce":3853800910}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6822","nonce":3853800910},{"type":"v1","addr":"172.21.15.193:6823","nonce":3853800910}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6820","nonce":3853800910},{"type":"v1","addr":"172.21.15.193:6821","nonce":3853800910}]},"public_addr":"172.21.15.193:6817/3853800910","cluster_addr":"172.21.15.193:6819/3853800910","heartbeat_back_addr":"172.21.15.193:6823/3853800910","heartbeat_front_addr":"172.21.15.193:6821/3853800910","state":["exists","up"]},{"osd":9,"uuid":"54be277f-3596-422f-9de5-9005a8571c6f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":68,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6826","nonce":3248293936},{"type":"v1","addr":"172.21.15.118:6827","nonce":3248293936}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6828","nonce":3248293936},{"type":"v1","addr":"172.21.15.118:6829","nonce":3248293936}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6832","nonce":3248293936},{"type":"v1","addr":"172.21.15.118:6833","nonce":3248293936}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6830","nonce":3248293936},{"type":"v1","addr":"172.21.15.118:6831","nonce":3248293936}]},"public_addr":"172.21.15.118:6827/3248293936","cluster_addr":"172.21.15.118:6829/3248293936","heartbeat_back_addr":"172.21.15.118:6833/3248293936","heartbeat_front_addr":"172.21.15.118:6831/3248293936","state":["exists","up"]},{"osd":10,"uuid":"7ba69d69-ac41-44ce-bc39-c049a8ca5900","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":73,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6824","nonce":2112195760},{"type":"v1","addr":"172.21.15.189:6825","nonce":2112195760}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6826","nonce":2112195760},{"type":"v1","addr":"172.21.15.189:6827","nonce":2112195760}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6830","nonce":2112195760},{"type":"v1","addr":"172.21.15.189:6831","nonce":2112195760}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6828","nonce":2112195760},{"type":"v1","addr":"172.21.15.189:6829","nonce":2112195760}]},"public_addr":"172.21.15.189:6825/2112195760","cluster_addr":"172.21.15.189:6827/2112195760","heartbeat_back_addr":"172.21.15.189:6831/2112195760","heartbeat_front_addr":"172.21.15.189:6829/2112195760","state":["exists","up"]},{"osd":11,"uuid":"186f0b18-6b22-4eae-83b7-b21bbcce0817","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":81,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6824","nonce":3880515510},{"type":"v1","addr":"172.21.15.193:6825","nonce":3880515510}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6826","nonce":3880515510},{"type":"v1","addr":"172.21.15.193:6827","nonce":3880515510}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6830","nonce":3880515510},{"type":"v1","addr":"172.21.15.193:6831","nonce":3880515510}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6828","nonce":3880515510},{"type":"v1","addr":"172.21.15.193:6829","nonce":3880515510}]},"public_addr":"172.21.15.193:6825/3880515510","cluster_addr":"172.21.15.193:6827/3880515510","heartbeat_back_addr":"172.21.15.193:6831/3880515510","heartbeat_front_addr":"172.21.15.193:6829/3880515510","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:06:58.105543+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:07:23.795866+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:07:47.426050+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:08:19.427073+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:08:45.816082+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:09:11.081523+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:09:44.485077+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:10:11.338972+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:10:36.234024+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:11:09.730432+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:11:37.059083+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:12:02.371947+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.a","mappings":[{"from":2,"to":11}]},{"pgid":"2.14","mappings":[{"from":2,"to":4}]},{"pgid":"2.1f","mappings":[{"from":8,"to":1}]},{"pgid":"3.11","mappings":[{"from":4,"to":8},{"from":5,"to":9}]},{"pgid":"3.13","mappings":[{"from":4,"to":9}]},{"pgid":"3.14","mappings":[{"from":4,"to":6}]},{"pgid":"3.31","mappings":[{"from":4,"to":7}]},{"pgid":"3.38","mappings":[{"from":3,"to":9}]},{"pgid":"3.39","mappings":[{"from":5,"to":8}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.118:6801/389249272":"2024-03-25T00:05:05.458796+0000","172.21.15.118:6800/389249272":"2024-03-25T00:05:05.458796+0000","172.21.15.118:0/2101854694":"2024-03-25T00:05:05.458796+0000","172.21.15.118:0/630779359":"2024-03-25T00:05:05.458796+0000","172.21.15.118:0/2158921029":"2024-03-25T00:04:31.283725+0000","172.21.15.118:0/1003670455":"2024-03-25T00:05:05.458796+0000","172.21.15.118:0/368712314":"2024-03-25T00:04:31.283725+0000","172.21.15.118:0/2004775597":"2024-03-25T00:04:31.283725+0000","172.21.15.118:6800/1463147955":"2024-03-25T00:04:31.283725+0000","172.21.15.118:6801/1463147955":"2024-03-25T00:04:31.283725+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-24T00:14:50.063 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:50.062+0000 7f5479400640 1 -- 172.21.15.118:0/3436596878 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7f54580a2880 msgr2=0x7f54580a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:14:50.063 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:50.062+0000 7f5479400640 1 --2- 172.21.15.118:0/3436596878 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7f54580a2880 0x7f54580a4d40 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7f547419f450 tx=0x7f545c009500 comp rx=0 tx=0).stop 2024-03-24T00:14:50.063 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:50.062+0000 7f5479400640 1 -- 172.21.15.118:0/3436596878 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5474076060 msgr2=0x7f547419df40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:14:50.063 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:50.062+0000 7f5479400640 1 --2- 172.21.15.118:0/3436596878 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5474076060 0x7f547419df40 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f546400e9f0 tx=0x7f546400eeb0 comp rx=0 tx=0).stop 2024-03-24T00:14:50.063 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:50.062+0000 7f5479400640 1 -- 172.21.15.118:0/3436596878 shutdown_connections 2024-03-24T00:14:50.063 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:50.062+0000 7f5479400640 1 --2- 172.21.15.118:0/3436596878 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7f54580a2880 0x7f54580a4d40 unknown :-1 s=CLOSED pgs=208 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:14:50.063 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:50.062+0000 7f5479400640 1 --2- 172.21.15.118:0/3436596878 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f5474077c40 0x7f54741a54f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:14:50.063 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:50.062+0000 7f5479400640 1 --2- 172.21.15.118:0/3436596878 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f54740772a0 0x7f547419e480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:14:50.063 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:50.062+0000 7f5479400640 1 --2- 172.21.15.118:0/3436596878 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5474076060 0x7f547419df40 unknown :-1 s=CLOSED pgs=123 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:14:50.063 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:50.062+0000 7f5479400640 1 -- 172.21.15.118:0/3436596878 >> 172.21.15.118:0/3436596878 conn(0x7f54740fee50 msgr2=0x7f5474100470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:14:50.063 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:50.062+0000 7f5479400640 1 -- 172.21.15.118:0/3436596878 shutdown_connections 2024-03-24T00:14:50.063 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:50.062+0000 7f5479400640 1 -- 172.21.15.118:0/3436596878 wait complete. 2024-03-24T00:14:50.078 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:14:49 smithi118 ceph-mon[31535]: pgmap v370: 97 pgs: 97 active+clean; 645 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 6.0 KiB/s rd, 511 B/s wr, 11 op/s 2024-03-24T00:14:50.083 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-24T00:07:50.190965+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-24T00:12:59.993723+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '85', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.25, 'score_stable': 2.25, 'optimal_score': 1, 'raw_score_acting': 2.25, 'raw_score_stable': 2.25, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-03-24T00:13:01.257347+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '86', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-24T00:14:50.083 DEBUG:teuthology.orchestra.run.smithi118:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-24T00:14:50.203 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:14:49 smithi189 ceph-mon[36187]: pgmap v370: 97 pgs: 97 active+clean; 645 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 6.0 KiB/s rd, 511 B/s wr, 11 op/s 2024-03-24T00:14:50.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:14:49 smithi193 ceph-mon[36878]: pgmap v370: 97 pgs: 97 active+clean; 645 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 6.0 KiB/s rd, 511 B/s wr, 11 op/s 2024-03-24T00:14:50.290 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:50.289+0000 7f6769600640 1 -- 172.21.15.118:0/3294798528 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f67641038e0 msgr2=0x7f6764103d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:14:50.290 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:50.289+0000 7f6769600640 1 --2- 172.21.15.118:0/3294798528 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f67641038e0 0x7f6764103d00 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f6754009900 tx=0x7f675402bc30 comp rx=0 tx=0).stop 2024-03-24T00:14:50.290 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:50.289+0000 7f6769600640 1 -- 172.21.15.118:0/3294798528 shutdown_connections 2024-03-24T00:14:50.290 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:50.289+0000 7f6769600640 1 --2- 172.21.15.118:0/3294798528 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6764105490 0x7f676410bd20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:14:50.290 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:50.289+0000 7f6769600640 1 --2- 172.21.15.118:0/3294798528 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f6764104ad0 0x7f6764104f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:14:50.290 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:50.289+0000 7f6769600640 1 --2- 172.21.15.118:0/3294798528 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f67641038e0 0x7f6764103d00 unknown :-1 s=CLOSED pgs=128 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:14:50.290 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:50.289+0000 7f6769600640 1 -- 172.21.15.118:0/3294798528 >> 172.21.15.118:0/3294798528 conn(0x7f67640fee50 msgr2=0x7f6764101290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:14:50.290 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:50.289+0000 7f6769600640 1 -- 172.21.15.118:0/3294798528 shutdown_connections 2024-03-24T00:14:50.290 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:50.289+0000 7f6769600640 1 -- 172.21.15.118:0/3294798528 wait complete. 2024-03-24T00:14:50.292 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:50.290+0000 7f6769600640 1 Processor -- start 2024-03-24T00:14:50.292 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:50.290+0000 7f6769600640 1 -- start start 2024-03-24T00:14:50.292 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:50.291+0000 7f6769600640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f67641038e0 0x7f6764199bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:14:50.292 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:50.291+0000 7f6769600640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6764104ad0 0x7f676419a130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:14:50.292 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:50.291+0000 7f6769600640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f6764105490 0x7f67641a11a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:14:50.292 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:50.291+0000 7f6769600640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f676410f5b0 con 0x7f67641038e0 2024-03-24T00:14:50.292 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:50.291+0000 7f6769600640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f676410f430 con 0x7f6764104ad0 2024-03-24T00:14:50.293 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:50.291+0000 7f6769600640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f676410f730 con 0x7f6764105490 2024-03-24T00:14:50.293 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:50.291+0000 7f6762000640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6764104ad0 0x7f676419a130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:14:50.293 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:50.291+0000 7f6763400640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f6764105490 0x7f67641a11a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:14:50.293 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:50.291+0000 7f6762000640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6764104ad0 0x7f676419a130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.118:55668/0 (socket says 172.21.15.118:55668) 2024-03-24T00:14:50.293 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:50.291+0000 7f6762000640 1 -- 172.21.15.118:0/536650464 learned_addr learned my addr 172.21.15.118:0/536650464 (peer_addr_for_me v2:172.21.15.118:0/0) 2024-03-24T00:14:50.293 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:50.291+0000 7f6762a00640 1 --2- 172.21.15.118:0/536650464 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f67641038e0 0x7f6764199bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:14:50.293 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:50.291+0000 7f6763400640 1 -- 172.21.15.118:0/536650464 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6764104ad0 msgr2=0x7f676419a130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:14:50.293 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:50.291+0000 7f6763400640 1 --2- 172.21.15.118:0/536650464 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6764104ad0 0x7f676419a130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:14:50.293 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:50.291+0000 7f6763400640 1 -- 172.21.15.118:0/536650464 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f67641038e0 msgr2=0x7f6764199bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:14:50.293 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:50.291+0000 7f6763400640 1 --2- 172.21.15.118:0/536650464 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f67641038e0 0x7f6764199bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:14:50.293 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:50.292+0000 7f6763400640 1 -- 172.21.15.118:0/536650464 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f67641a1810 con 0x7f6764105490 2024-03-24T00:14:50.293 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:50.292+0000 7f6762a00640 1 --2- 172.21.15.118:0/536650464 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f67641038e0 0x7f6764199bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T00:14:50.293 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:50.292+0000 7f6762000640 1 --2- 172.21.15.118:0/536650464 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6764104ad0 0x7f676419a130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T00:14:50.293 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:50.292+0000 7f6763400640 1 --2- 172.21.15.118:0/536650464 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f6764105490 0x7f67641a11a0 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f675c009fd0 tx=0x7f675c00ef10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:14:50.294 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:50.292+0000 7f675b400640 1 -- 172.21.15.118:0/536650464 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f675c00ce20 con 0x7f6764105490 2024-03-24T00:14:50.294 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:50.292+0000 7f675b400640 1 -- 172.21.15.118:0/536650464 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f675c005020 con 0x7f6764105490 2024-03-24T00:14:50.294 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:50.292+0000 7f675b400640 1 -- 172.21.15.118:0/536650464 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f675c010720 con 0x7f6764105490 2024-03-24T00:14:50.294 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:50.292+0000 7f6769600640 1 -- 172.21.15.118:0/536650464 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f67641a1b00 con 0x7f6764105490 2024-03-24T00:14:50.294 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:50.292+0000 7f6769600640 1 -- 172.21.15.118:0/536650464 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f67641a2050 con 0x7f6764105490 2024-03-24T00:14:50.295 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:50.293+0000 7f6769600640 1 -- 172.21.15.118:0/536650464 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f676410aba0 con 0x7f6764105490 2024-03-24T00:14:50.297 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:50.297+0000 7f675b400640 1 -- 172.21.15.118:0/536650464 <== mon.1 v2:172.21.15.193:3300/0 4 ==== mgrmap(e 18) v1 ==== 137291+0+0 (secure 0 0 0) 0x7f675c005540 con 0x7f6764105490 2024-03-24T00:14:50.298 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:50.298+0000 7f675b400640 1 --2- 172.21.15.118:0/536650464 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7f67480a2800 0x7f67480a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:14:50.299 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:50.298+0000 7f675b400640 1 -- 172.21.15.118:0/536650464 <== mon.1 v2:172.21.15.193:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 8248+0+0 (secure 0 0 0) 0x7f675c0cf1b0 con 0x7f6764105490 2024-03-24T00:14:50.299 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:50.298+0000 7f6762a00640 1 --2- 172.21.15.118:0/536650464 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7f67480a2800 0x7f67480a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:14:50.299 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:50.299+0000 7f6762a00640 1 --2- 172.21.15.118:0/536650464 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7f67480a2800 0x7f67480a4cc0 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7f675402c170 tx=0x7f675402df90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:14:50.302 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:50.302+0000 7f675b400640 1 -- 172.21.15.118:0/536650464 <== mon.1 v2:172.21.15.193:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f675c09c1e0 con 0x7f6764105490 2024-03-24T00:14:50.527 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:50.526+0000 7f6769600640 1 -- 172.21.15.118:0/536650464 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f6764062340 con 0x7f6764105490 2024-03-24T00:14:50.527 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:50.527+0000 7f675b400640 1 -- 172.21.15.118:0/536650464 <== mon.1 v2:172.21.15.193:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v88) v1 ==== 93+0+10 (secure 0 0 0) 0x7f675c09bae0 con 0x7f6764105490 2024-03-24T00:14:50.528 INFO:teuthology.orchestra.run.smithi118.stdout:pg_num: 1 2024-03-24T00:14:50.530 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:50.529+0000 7f6769600640 1 -- 172.21.15.118:0/536650464 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7f67480a2800 msgr2=0x7f67480a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:14:50.530 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:50.529+0000 7f6769600640 1 --2- 172.21.15.118:0/536650464 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7f67480a2800 0x7f67480a4cc0 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7f675402c170 tx=0x7f675402df90 comp rx=0 tx=0).stop 2024-03-24T00:14:50.530 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:50.529+0000 7f6769600640 1 -- 172.21.15.118:0/536650464 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f6764105490 msgr2=0x7f67641a11a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:14:50.530 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:50.529+0000 7f6769600640 1 --2- 172.21.15.118:0/536650464 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f6764105490 0x7f67641a11a0 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f675c009fd0 tx=0x7f675c00ef10 comp rx=0 tx=0).stop 2024-03-24T00:14:50.531 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:50.530+0000 7f6769600640 1 -- 172.21.15.118:0/536650464 shutdown_connections 2024-03-24T00:14:50.531 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:50.530+0000 7f6769600640 1 --2- 172.21.15.118:0/536650464 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7f67480a2800 0x7f67480a4cc0 unknown :-1 s=CLOSED pgs=209 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:14:50.531 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:50.530+0000 7f6769600640 1 --2- 172.21.15.118:0/536650464 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f6764105490 0x7f67641a11a0 unknown :-1 s=CLOSED pgs=129 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:14:50.531 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:50.530+0000 7f6769600640 1 --2- 172.21.15.118:0/536650464 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6764104ad0 0x7f676419a130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:14:50.531 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:50.530+0000 7f6769600640 1 --2- 172.21.15.118:0/536650464 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f67641038e0 0x7f6764199bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:14:50.531 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:50.530+0000 7f6769600640 1 -- 172.21.15.118:0/536650464 >> 172.21.15.118:0/536650464 conn(0x7f67640fee50 msgr2=0x7f6764107d70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:14:50.531 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:50.530+0000 7f6769600640 1 -- 172.21.15.118:0/536650464 shutdown_connections 2024-03-24T00:14:50.531 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:50.530+0000 7f6769600640 1 -- 172.21.15.118:0/536650464 wait complete. 2024-03-24T00:14:50.547 DEBUG:teuthology.orchestra.run.smithi118:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-24T00:14:50.745 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:50.744+0000 7f96dfc00640 1 -- 172.21.15.118:0/268767683 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f96d81038e0 msgr2=0x7f96d8103d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:14:50.745 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:50.744+0000 7f96dfc00640 1 --2- 172.21.15.118:0/268767683 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f96d81038e0 0x7f96d8103d00 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7f96cc009860 tx=0x7f96cc02bb70 comp rx=0 tx=0).stop 2024-03-24T00:14:50.745 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:50.744+0000 7f96dfc00640 1 -- 172.21.15.118:0/268767683 shutdown_connections 2024-03-24T00:14:50.745 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:50.744+0000 7f96dfc00640 1 --2- 172.21.15.118:0/268767683 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f96d8105490 0x7f96d810bd20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:14:50.745 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:50.744+0000 7f96dfc00640 1 --2- 172.21.15.118:0/268767683 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f96d8104ad0 0x7f96d8104f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:14:50.745 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:50.744+0000 7f96dfc00640 1 --2- 172.21.15.118:0/268767683 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f96d81038e0 0x7f96d8103d00 unknown :-1 s=CLOSED pgs=167 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:14:50.745 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:50.744+0000 7f96dfc00640 1 -- 172.21.15.118:0/268767683 >> 172.21.15.118:0/268767683 conn(0x7f96d80fee50 msgr2=0x7f96d8101290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:14:50.746 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:50.744+0000 7f96dfc00640 1 -- 172.21.15.118:0/268767683 shutdown_connections 2024-03-24T00:14:50.746 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:50.744+0000 7f96dfc00640 1 -- 172.21.15.118:0/268767683 wait complete. 2024-03-24T00:14:50.747 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:50.745+0000 7f96dfc00640 1 Processor -- start 2024-03-24T00:14:50.747 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:50.745+0000 7f96dfc00640 1 -- start start 2024-03-24T00:14:50.748 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:50.745+0000 7f96dfc00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f96d81038e0 0x7f96d8199c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:14:50.748 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:50.745+0000 7f96dfc00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f96d8104ad0 0x7f96d819a190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:14:50.748 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:50.745+0000 7f96dfc00640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f96d8105490 0x7f96d81a11c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:14:50.748 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:50.745+0000 7f96dfc00640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f96d810f530 con 0x7f96d8105490 2024-03-24T00:14:50.749 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:50.745+0000 7f96dfc00640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f96d810f3b0 con 0x7f96d81038e0 2024-03-24T00:14:50.749 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:50.745+0000 7f96dfc00640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f96d810f6b0 con 0x7f96d8104ad0 2024-03-24T00:14:50.749 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:50.746+0000 7f96dd200640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f96d81038e0 0x7f96d8199c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:14:50.749 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:50.746+0000 7f96dd200640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f96d81038e0 0x7f96d8199c50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.118:55680/0 (socket says 172.21.15.118:55680) 2024-03-24T00:14:50.749 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:50.746+0000 7f96dd200640 1 -- 172.21.15.118:0/4146363460 learned_addr learned my addr 172.21.15.118:0/4146363460 (peer_addr_for_me v2:172.21.15.118:0/0) 2024-03-24T00:14:50.749 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:50.746+0000 7f96d7e00640 1 --2- 172.21.15.118:0/4146363460 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f96d8104ad0 0x7f96d819a190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:14:50.749 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:50.746+0000 7f96ddc00640 1 --2- 172.21.15.118:0/4146363460 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f96d8105490 0x7f96d81a11c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:14:50.749 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:50.746+0000 7f96dd200640 1 -- 172.21.15.118:0/4146363460 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f96d8104ad0 msgr2=0x7f96d819a190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:14:50.749 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:50.746+0000 7f96dd200640 1 --2- 172.21.15.118:0/4146363460 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f96d8104ad0 0x7f96d819a190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:14:50.749 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:50.746+0000 7f96dd200640 1 -- 172.21.15.118:0/4146363460 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f96d8105490 msgr2=0x7f96d81a11c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:14:50.749 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:50.746+0000 7f96dd200640 1 --2- 172.21.15.118:0/4146363460 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f96d8105490 0x7f96d81a11c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:14:50.749 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:50.746+0000 7f96dd200640 1 -- 172.21.15.118:0/4146363460 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f96d81a1860 con 0x7f96d81038e0 2024-03-24T00:14:50.750 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:50.746+0000 7f96dd200640 1 --2- 172.21.15.118:0/4146363460 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f96d81038e0 0x7f96d8199c50 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f96cc0349e0 tx=0x7f96cc032430 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:14:50.750 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:50.747+0000 7f96d5600640 1 -- 172.21.15.118:0/4146363460 <== mon.2 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f96cc041020 con 0x7f96d81038e0 2024-03-24T00:14:50.750 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:50.747+0000 7f96d5600640 1 -- 172.21.15.118:0/4146363460 <== mon.2 v2:172.21.15.189:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f96cc02c9f0 con 0x7f96d81038e0 2024-03-24T00:14:50.750 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:50.747+0000 7f96d5600640 1 -- 172.21.15.118:0/4146363460 <== mon.2 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f96cc039040 con 0x7f96d81038e0 2024-03-24T00:14:50.750 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:50.747+0000 7f96dfc00640 1 -- 172.21.15.118:0/4146363460 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f96d81a1af0 con 0x7f96d81038e0 2024-03-24T00:14:50.750 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:50.747+0000 7f96dfc00640 1 -- 172.21.15.118:0/4146363460 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f96d81a1f90 con 0x7f96d81038e0 2024-03-24T00:14:50.750 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:50.748+0000 7f96dfc00640 1 -- 172.21.15.118:0/4146363460 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f96d810ac80 con 0x7f96d81038e0 2024-03-24T00:14:50.751 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:50.751+0000 7f96d5600640 1 -- 172.21.15.118:0/4146363460 <== mon.2 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 18) v1 ==== 137291+0+0 (secure 0 0 0) 0x7f96cc044020 con 0x7f96d81038e0 2024-03-24T00:14:50.754 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:50.753+0000 7f96d5600640 1 --2- 172.21.15.118:0/4146363460 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7f96c00a2880 0x7f96c00a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:14:50.754 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:50.753+0000 7f96d5600640 1 -- 172.21.15.118:0/4146363460 <== mon.2 v2:172.21.15.189:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 8248+0+0 (secure 0 0 0) 0x7f96cc0f0310 con 0x7f96d81038e0 2024-03-24T00:14:50.754 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:50.753+0000 7f96d7e00640 1 --2- 172.21.15.118:0/4146363460 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7f96c00a2880 0x7f96c00a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:14:50.754 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:50.754+0000 7f96d7e00640 1 --2- 172.21.15.118:0/4146363460 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7f96c00a2880 0x7f96c00a4d40 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7f96c4009bd0 tx=0x7f96c4005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:14:50.757 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:50.757+0000 7f96d5600640 1 -- 172.21.15.118:0/4146363460 <== mon.2 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f96cc04b020 con 0x7f96d81038e0 2024-03-24T00:14:50.954 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:50.954+0000 7f96dfc00640 1 -- 172.21.15.118:0/4146363460 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f96d802d2b0 con 0x7f96d81038e0 2024-03-24T00:14:50.955 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:50.954+0000 7f96d5600640 1 -- 172.21.15.118:0/4146363460 <== mon.2 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v88) v1 ==== 104+0+11 (secure 0 0 0) 0x7f96cc0bd340 con 0x7f96d81038e0 2024-03-24T00:14:50.955 INFO:teuthology.orchestra.run.smithi118.stdout:pg_num: 32 2024-03-24T00:14:50.959 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:50.958+0000 7f96dfc00640 1 -- 172.21.15.118:0/4146363460 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7f96c00a2880 msgr2=0x7f96c00a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:14:50.959 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:50.958+0000 7f96dfc00640 1 --2- 172.21.15.118:0/4146363460 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7f96c00a2880 0x7f96c00a4d40 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7f96c4009bd0 tx=0x7f96c4005f90 comp rx=0 tx=0).stop 2024-03-24T00:14:50.959 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:50.958+0000 7f96dfc00640 1 -- 172.21.15.118:0/4146363460 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f96d81038e0 msgr2=0x7f96d8199c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:14:50.959 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:50.958+0000 7f96dfc00640 1 --2- 172.21.15.118:0/4146363460 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f96d81038e0 0x7f96d8199c50 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f96cc0349e0 tx=0x7f96cc032430 comp rx=0 tx=0).stop 2024-03-24T00:14:50.959 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:50.958+0000 7f96dfc00640 1 -- 172.21.15.118:0/4146363460 shutdown_connections 2024-03-24T00:14:50.959 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:50.958+0000 7f96dfc00640 1 --2- 172.21.15.118:0/4146363460 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7f96c00a2880 0x7f96c00a4d40 unknown :-1 s=CLOSED pgs=210 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:14:50.959 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:50.958+0000 7f96dfc00640 1 --2- 172.21.15.118:0/4146363460 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f96d8105490 0x7f96d81a11c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:14:50.959 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:50.958+0000 7f96dfc00640 1 --2- 172.21.15.118:0/4146363460 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f96d8104ad0 0x7f96d819a190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:14:50.959 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:50.958+0000 7f96dfc00640 1 --2- 172.21.15.118:0/4146363460 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f96d81038e0 0x7f96d8199c50 unknown :-1 s=CLOSED pgs=124 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:14:50.960 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:50.958+0000 7f96dfc00640 1 -- 172.21.15.118:0/4146363460 >> 172.21.15.118:0/4146363460 conn(0x7f96d80fee50 msgr2=0x7f96d8107e50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:14:50.960 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:50.958+0000 7f96dfc00640 1 -- 172.21.15.118:0/4146363460 shutdown_connections 2024-03-24T00:14:50.960 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:50.958+0000 7f96dfc00640 1 -- 172.21.15.118:0/4146363460 wait complete. 2024-03-24T00:14:50.976 DEBUG:teuthology.orchestra.run.smithi118:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-24T00:14:51.044 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:14:50 smithi118 ceph-mon[31535]: from='client.? 172.21.15.118:0/3436596878' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-24T00:14:51.045 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:14:50 smithi118 ceph-mon[31535]: from='client.? 172.21.15.118:0/536650464' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-24T00:14:51.111 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:14:50 smithi193 ceph-mon[36878]: from='client.? 172.21.15.118:0/3436596878' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-24T00:14:51.112 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:14:50 smithi193 ceph-mon[36878]: from='client.? 172.21.15.118:0/536650464' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-24T00:14:51.174 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:51.173+0000 7f1ebbe00640 1 -- 172.21.15.118:0/1439983211 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1ebc1015a0 msgr2=0x7f1ebc1019c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:14:51.174 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:51.173+0000 7f1ebbe00640 1 --2- 172.21.15.118:0/1439983211 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1ebc1015a0 0x7f1ebc1019c0 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7f1eac009900 tx=0x7f1eac02bc40 comp rx=0 tx=0).stop 2024-03-24T00:14:51.174 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:51.173+0000 7f1ebbe00640 1 -- 172.21.15.118:0/1439983211 shutdown_connections 2024-03-24T00:14:51.174 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:51.173+0000 7f1ebbe00640 1 --2- 172.21.15.118:0/1439983211 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f1ebc103150 0x7f1ebc1099e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:14:51.174 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:51.173+0000 7f1ebbe00640 1 --2- 172.21.15.118:0/1439983211 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f1ebc102790 0x7f1ebc102c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:14:51.175 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:51.173+0000 7f1ebbe00640 1 --2- 172.21.15.118:0/1439983211 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1ebc1015a0 0x7f1ebc1019c0 unknown :-1 s=CLOSED pgs=125 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:14:51.175 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:51.173+0000 7f1ebbe00640 1 -- 172.21.15.118:0/1439983211 >> 172.21.15.118:0/1439983211 conn(0x7f1ebc0fcb10 msgr2=0x7f1ebc0fef50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:14:51.175 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:51.173+0000 7f1ebbe00640 1 -- 172.21.15.118:0/1439983211 shutdown_connections 2024-03-24T00:14:51.175 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:51.173+0000 7f1ebbe00640 1 -- 172.21.15.118:0/1439983211 wait complete. 2024-03-24T00:14:51.177 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:51.174+0000 7f1ebbe00640 1 Processor -- start 2024-03-24T00:14:51.177 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:51.174+0000 7f1ebbe00640 1 -- start start 2024-03-24T00:14:51.177 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:51.175+0000 7f1ebbe00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f1ebc1015a0 0x7f1ebc199b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:14:51.177 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:51.175+0000 7f1ebbe00640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f1ebc102790 0x7f1ebc19a060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:14:51.177 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:51.175+0000 7f1ebbe00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1ebc103150 0x7f1ebc1a10d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:14:51.177 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:51.175+0000 7f1ebbe00640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1ebc10d240 con 0x7f1ebc102790 2024-03-24T00:14:51.177 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:51.175+0000 7f1ebbe00640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1ebc10d0c0 con 0x7f1ebc103150 2024-03-24T00:14:51.177 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:51.175+0000 7f1ebbe00640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1ebc10d3c0 con 0x7f1ebc1015a0 2024-03-24T00:14:51.177 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:51.175+0000 7f1ebaa00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f1ebc1015a0 0x7f1ebc199b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:14:51.178 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:51.175+0000 7f1ebaa00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f1ebc1015a0 0x7f1ebc199b20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.118:60422/0 (socket says 172.21.15.118:60422) 2024-03-24T00:14:51.178 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:51.175+0000 7f1ebb400640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1ebc103150 0x7f1ebc1a10d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:14:51.178 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:51.175+0000 7f1ebaa00640 1 -- 172.21.15.118:0/1651444317 learned_addr learned my addr 172.21.15.118:0/1651444317 (peer_addr_for_me v2:172.21.15.118:0/0) 2024-03-24T00:14:51.178 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:51.175+0000 7f1eba000640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f1ebc102790 0x7f1ebc19a060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:14:51.178 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:51.175+0000 7f1ebaa00640 1 -- 172.21.15.118:0/1651444317 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1ebc103150 msgr2=0x7f1ebc1a10d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:14:51.178 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:51.175+0000 7f1ebaa00640 1 --2- 172.21.15.118:0/1651444317 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1ebc103150 0x7f1ebc1a10d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:14:51.178 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:51.175+0000 7f1ebaa00640 1 -- 172.21.15.118:0/1651444317 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f1ebc102790 msgr2=0x7f1ebc19a060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:14:51.178 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:51.175+0000 7f1ebaa00640 1 --2- 172.21.15.118:0/1651444317 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f1ebc102790 0x7f1ebc19a060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:14:51.179 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:51.175+0000 7f1ebaa00640 1 -- 172.21.15.118:0/1651444317 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1ebc1a1740 con 0x7f1ebc1015a0 2024-03-24T00:14:51.179 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:51.176+0000 7f1ebaa00640 1 --2- 172.21.15.118:0/1651444317 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f1ebc1015a0 0x7f1ebc199b20 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f1eac02efd0 tx=0x7f1eac02c660 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:14:51.179 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:51.176+0000 7f1eb3400640 1 -- 172.21.15.118:0/1651444317 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1eac041070 con 0x7f1ebc1015a0 2024-03-24T00:14:51.179 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:51.176+0000 7f1ebbe00640 1 -- 172.21.15.118:0/1651444317 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1ebc1a19d0 con 0x7f1ebc1015a0 2024-03-24T00:14:51.179 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:51.176+0000 7f1eb3400640 1 -- 172.21.15.118:0/1651444317 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f1eac004450 con 0x7f1ebc1015a0 2024-03-24T00:14:51.179 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:51.176+0000 7f1eb3400640 1 -- 172.21.15.118:0/1651444317 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1eac003e50 con 0x7f1ebc1015a0 2024-03-24T00:14:51.179 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:51.176+0000 7f1ebbe00640 1 -- 172.21.15.118:0/1651444317 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1ebc1a1e90 con 0x7f1ebc1015a0 2024-03-24T00:14:51.179 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:51.177+0000 7f1ebbe00640 1 -- 172.21.15.118:0/1651444317 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1ebc1a2060 con 0x7f1ebc1015a0 2024-03-24T00:14:51.180 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:51.179+0000 7f1eb3400640 1 -- 172.21.15.118:0/1651444317 <== mon.1 v2:172.21.15.193:3300/0 4 ==== mgrmap(e 18) v1 ==== 137291+0+0 (secure 0 0 0) 0x7f1eac044070 con 0x7f1ebc1015a0 2024-03-24T00:14:51.181 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:51.181+0000 7f1eb3400640 1 --2- 172.21.15.118:0/1651444317 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7f1ea00a29a0 0x7f1ea00a4e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:14:51.182 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:51.181+0000 7f1eb3400640 1 -- 172.21.15.118:0/1651444317 <== mon.1 v2:172.21.15.193:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 8248+0+0 (secure 0 0 0) 0x7f1eac0f0610 con 0x7f1ebc1015a0 2024-03-24T00:14:51.182 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:51.181+0000 7f1eba000640 1 --2- 172.21.15.118:0/1651444317 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7f1ea00a29a0 0x7f1ea00a4e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:14:51.183 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:51.182+0000 7f1eba000640 1 --2- 172.21.15.118:0/1651444317 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7f1ea00a29a0 0x7f1ea00a4e60 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7f1ebc19b030 tx=0x7f1ea8005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:14:51.184 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:51.184+0000 7f1eb3400640 1 -- 172.21.15.118:0/1651444317 <== mon.1 v2:172.21.15.193:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f1eac0bcff0 con 0x7f1ebc1015a0 2024-03-24T00:14:51.203 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:14:50 smithi189 ceph-mon[36187]: from='client.? 172.21.15.118:0/3436596878' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-24T00:14:51.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:14:50 smithi189 ceph-mon[36187]: from='client.? 172.21.15.118:0/536650464' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-24T00:14:51.489 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:51.489+0000 7f1ebbe00640 1 -- 172.21.15.118:0/1651444317 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f1ebc1a2350 con 0x7f1ebc1015a0 2024-03-24T00:14:51.490 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:51.490+0000 7f1eb3400640 1 -- 172.21.15.118:0/1651444317 <== mon.1 v2:172.21.15.193:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v88) v1 ==== 100+0+11 (secure 0 0 0) 0x7f1eac0c1090 con 0x7f1ebc1015a0 2024-03-24T00:14:51.490 INFO:teuthology.orchestra.run.smithi118.stdout:pg_num: 64 2024-03-24T00:14:51.492 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:51.492+0000 7f1ebbe00640 1 -- 172.21.15.118:0/1651444317 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7f1ea00a29a0 msgr2=0x7f1ea00a4e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:14:51.492 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:51.492+0000 7f1ebbe00640 1 --2- 172.21.15.118:0/1651444317 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7f1ea00a29a0 0x7f1ea00a4e60 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7f1ebc19b030 tx=0x7f1ea8005f90 comp rx=0 tx=0).stop 2024-03-24T00:14:51.492 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:51.492+0000 7f1ebbe00640 1 -- 172.21.15.118:0/1651444317 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f1ebc1015a0 msgr2=0x7f1ebc199b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:14:51.493 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:51.492+0000 7f1ebbe00640 1 --2- 172.21.15.118:0/1651444317 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f1ebc1015a0 0x7f1ebc199b20 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f1eac02efd0 tx=0x7f1eac02c660 comp rx=0 tx=0).stop 2024-03-24T00:14:51.493 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:51.493+0000 7f1ebbe00640 1 -- 172.21.15.118:0/1651444317 shutdown_connections 2024-03-24T00:14:51.493 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:51.493+0000 7f1ebbe00640 1 --2- 172.21.15.118:0/1651444317 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7f1ea00a29a0 0x7f1ea00a4e60 unknown :-1 s=CLOSED pgs=211 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:14:51.493 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:51.493+0000 7f1ebbe00640 1 --2- 172.21.15.118:0/1651444317 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1ebc103150 0x7f1ebc1a10d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:14:51.493 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:51.493+0000 7f1ebbe00640 1 --2- 172.21.15.118:0/1651444317 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f1ebc102790 0x7f1ebc19a060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:14:51.493 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:51.493+0000 7f1ebbe00640 1 --2- 172.21.15.118:0/1651444317 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f1ebc1015a0 0x7f1ebc199b20 unknown :-1 s=CLOSED pgs=130 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:14:51.493 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:51.493+0000 7f1ebbe00640 1 -- 172.21.15.118:0/1651444317 >> 172.21.15.118:0/1651444317 conn(0x7f1ebc0fcb10 msgr2=0x7f1ebc0738d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:14:51.493 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:51.493+0000 7f1ebbe00640 1 -- 172.21.15.118:0/1651444317 shutdown_connections 2024-03-24T00:14:51.493 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:51.493+0000 7f1ebbe00640 1 -- 172.21.15.118:0/1651444317 wait complete. 2024-03-24T00:14:51.511 DEBUG:teuthology.orchestra.run.smithi118:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-24T00:14:51.695 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:51.695+0000 7ff852600640 1 -- 172.21.15.118:0/1864644134 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff84c1026f0 msgr2=0x7ff84c102b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:14:51.695 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:51.695+0000 7ff852600640 1 --2- 172.21.15.118:0/1864644134 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff84c1026f0 0x7ff84c102b70 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7ff838009900 tx=0x7ff83802bc30 comp rx=0 tx=0).stop 2024-03-24T00:14:51.695 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:51.695+0000 7ff852600640 1 -- 172.21.15.118:0/1864644134 shutdown_connections 2024-03-24T00:14:51.695 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:51.695+0000 7ff852600640 1 --2- 172.21.15.118:0/1864644134 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7ff84c1030b0 0x7ff84c109940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:14:51.695 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:51.695+0000 7ff852600640 1 --2- 172.21.15.118:0/1864644134 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff84c1026f0 0x7ff84c102b70 unknown :-1 s=CLOSED pgs=126 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:14:51.695 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:51.695+0000 7ff852600640 1 --2- 172.21.15.118:0/1864644134 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7ff84c101500 0x7ff84c101920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:14:51.696 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:51.695+0000 7ff852600640 1 -- 172.21.15.118:0/1864644134 >> 172.21.15.118:0/1864644134 conn(0x7ff84c0fca70 msgr2=0x7ff84c0feeb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:14:51.696 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:51.695+0000 7ff852600640 1 -- 172.21.15.118:0/1864644134 shutdown_connections 2024-03-24T00:14:51.696 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:51.695+0000 7ff852600640 1 -- 172.21.15.118:0/1864644134 wait complete. 2024-03-24T00:14:51.698 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:51.696+0000 7ff852600640 1 Processor -- start 2024-03-24T00:14:51.698 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:51.696+0000 7ff852600640 1 -- start start 2024-03-24T00:14:51.698 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:51.697+0000 7ff852600640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7ff84c101500 0x7ff84c199990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:14:51.698 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:51.697+0000 7ff852600640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff84c1026f0 0x7ff84c199ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:14:51.698 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:51.697+0000 7ff852600640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7ff84c1030b0 0x7ff84c1a0f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:14:51.698 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:51.697+0000 7ff852600640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff84c10d1e0 con 0x7ff84c1030b0 2024-03-24T00:14:51.698 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:51.697+0000 7ff852600640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff84c10d060 con 0x7ff84c1026f0 2024-03-24T00:14:51.699 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:51.697+0000 7ff852600640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff84c10d360 con 0x7ff84c101500 2024-03-24T00:14:51.699 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:51.697+0000 7ff851200640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7ff84c101500 0x7ff84c199990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:14:51.699 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:51.697+0000 7ff84be00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff84c1026f0 0x7ff84c199ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:14:51.699 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:51.697+0000 7ff851200640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7ff84c101500 0x7ff84c199990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.118:60446/0 (socket says 172.21.15.118:60446) 2024-03-24T00:14:51.699 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:51.697+0000 7ff851200640 1 -- 172.21.15.118:0/202068537 learned_addr learned my addr 172.21.15.118:0/202068537 (peer_addr_for_me v2:172.21.15.118:0/0) 2024-03-24T00:14:51.699 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:51.697+0000 7ff851c00640 1 --2- 172.21.15.118:0/202068537 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7ff84c1030b0 0x7ff84c1a0f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:14:51.699 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:51.697+0000 7ff851200640 1 -- 172.21.15.118:0/202068537 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff84c1026f0 msgr2=0x7ff84c199ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:14:51.699 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:51.697+0000 7ff851200640 1 --2- 172.21.15.118:0/202068537 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff84c1026f0 0x7ff84c199ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:14:51.699 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:51.697+0000 7ff851200640 1 -- 172.21.15.118:0/202068537 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7ff84c1030b0 msgr2=0x7ff84c1a0f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:14:51.699 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:51.697+0000 7ff851200640 1 --2- 172.21.15.118:0/202068537 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7ff84c1030b0 0x7ff84c1a0f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:14:51.699 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:51.697+0000 7ff851200640 1 -- 172.21.15.118:0/202068537 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff84c1a15b0 con 0x7ff84c101500 2024-03-24T00:14:51.699 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:51.697+0000 7ff851c00640 1 --2- 172.21.15.118:0/202068537 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7ff84c1030b0 0x7ff84c1a0f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T00:14:51.699 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:51.697+0000 7ff84be00640 1 --2- 172.21.15.118:0/202068537 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff84c1026f0 0x7ff84c199ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T00:14:51.699 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:51.698+0000 7ff851200640 1 --2- 172.21.15.118:0/202068537 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7ff84c101500 0x7ff84c199990 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7ff84000e9c0 tx=0x7ff84000ee80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:14:51.699 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:51.698+0000 7ff849600640 1 -- 172.21.15.118:0/202068537 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff84000cd80 con 0x7ff84c101500 2024-03-24T00:14:51.700 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:51.698+0000 7ff849600640 1 -- 172.21.15.118:0/202068537 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7ff8400035f0 con 0x7ff84c101500 2024-03-24T00:14:51.700 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:51.698+0000 7ff849600640 1 -- 172.21.15.118:0/202068537 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff840010690 con 0x7ff84c101500 2024-03-24T00:14:51.700 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:51.698+0000 7ff852600640 1 -- 172.21.15.118:0/202068537 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff84c1a18a0 con 0x7ff84c101500 2024-03-24T00:14:51.700 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:51.698+0000 7ff852600640 1 -- 172.21.15.118:0/202068537 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff84c1a1dc0 con 0x7ff84c101500 2024-03-24T00:14:51.700 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:51.699+0000 7ff852600640 1 -- 172.21.15.118:0/202068537 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff84c1a2020 con 0x7ff84c101500 2024-03-24T00:14:51.700 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:51.701+0000 7ff849600640 1 -- 172.21.15.118:0/202068537 <== mon.1 v2:172.21.15.193:3300/0 4 ==== mgrmap(e 18) v1 ==== 137291+0+0 (secure 0 0 0) 0x7ff8400037a0 con 0x7ff84c101500 2024-03-24T00:14:51.702 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:51.703+0000 7ff849600640 1 --2- 172.21.15.118:0/202068537 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7ff8340a2800 0x7ff8340a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:14:51.703 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:51.703+0000 7ff849600640 1 -- 172.21.15.118:0/202068537 <== mon.1 v2:172.21.15.193:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 8248+0+0 (secure 0 0 0) 0x7ff8400cef30 con 0x7ff84c101500 2024-03-24T00:14:51.703 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:51.703+0000 7ff84be00640 1 --2- 172.21.15.118:0/202068537 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7ff8340a2800 0x7ff8340a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:14:51.703 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:51.704+0000 7ff84be00640 1 --2- 172.21.15.118:0/202068537 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7ff8340a2800 0x7ff8340a4cc0 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7ff83802c240 tx=0x7ff83802df90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:14:51.705 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:51.706+0000 7ff849600640 1 -- 172.21.15.118:0/202068537 <== mon.1 v2:172.21.15.193:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7ff84009bf60 con 0x7ff84c101500 2024-03-24T00:14:51.960 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:51.960+0000 7ff852600640 1 -- 172.21.15.118:0/202068537 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7ff84c1a21f0 con 0x7ff84c101500 2024-03-24T00:14:51.960 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:51.961+0000 7ff849600640 1 -- 172.21.15.118:0/202068537 <== mon.1 v2:172.21.15.193:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 27 v27) v1 ==== 94+0+11478 (secure 0 0 0) 0x7ff84009b860 con 0x7ff84c101500 2024-03-24T00:14:51.961 INFO:teuthology.orchestra.run.smithi118.stdout: 2024-03-24T00:14:51.961 INFO:teuthology.orchestra.run.smithi118.stdout:{"epoch":27,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14385,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.118:6835/3194741373","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6834","nonce":3194741373},{"type":"v1","addr":"172.21.15.118:6835","nonce":3194741373}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":4},{"gid":14397,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.118:6837/1374462956","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6836","nonce":1374462956},{"type":"v1","addr":"172.21.15.118:6837","nonce":1374462956}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":27,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-03-24T00:13:01.782474+0000","modified":"2024-03-24T00:13:09.335361+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24569,"mds_1":24550,"mds_2":24511,"mds_3":24499,"mds_4":24488},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14469":{"gid":14469,"name":"a","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.118:6841/3040946537","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6840","nonce":3040946537},{"type":"v1","addr":"172.21.15.118:6841","nonce":3040946537}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24439":{"gid":24439,"name":"l","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.193:6833/1023428412","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6832","nonce":1023428412},{"type":"v1","addr":"172.21.15.193:6833","nonce":1023428412}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24440":{"gid":24440,"name":"k","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.189:6833/2740024078","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6832","nonce":2740024078},{"type":"v1","addr":"172.21.15.189:6833","nonce":2740024078}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24488":{"gid":24488,"name":"e","rank":4,"incarnation":25,"state":"up:active","state_seq":10,"addr":"172.21.15.189:6835/599473205","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6834","nonce":599473205},{"type":"v1","addr":"172.21.15.189:6835","nonce":599473205}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24499":{"gid":24499,"name":"f","rank":3,"incarnation":23,"state":"up:active","state_seq":9,"addr":"172.21.15.193:6835/2055310541","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6834","nonce":2055310541},{"type":"v1","addr":"172.21.15.193:6835","nonce":2055310541}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24511":{"gid":24511,"name":"c","rank":2,"incarnation":21,"state":"up:active","state_seq":8,"addr":"172.21.15.193:6837/2921520053","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6836","nonce":2921520053},{"type":"v1","addr":"172.21.15.193:6837","nonce":2921520053}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24515":{"gid":24515,"name":"h","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.189:6837/3450187137","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6836","nonce":3450187137},{"type":"v1","addr":"172.21.15.189:6837","nonce":3450187137}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24532":{"gid":24532,"name":"i","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.193:6839/2126842603","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6838","nonce":2126842603},{"type":"v1","addr":"172.21.15.193:6839","nonce":2126842603}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24550":{"gid":24550,"name":"g","rank":1,"incarnation":19,"state":"up:active","state_seq":6,"addr":"172.21.15.118:6839/3285788503","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6838","nonce":3285788503},{"type":"v1","addr":"172.21.15.118:6839","nonce":3285788503}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24569":{"gid":24569,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.189:6839/4074484168","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6838","nonce":4074484168},{"type":"v1","addr":"172.21.15.189:6839","nonce":4074484168}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24569,"qdb_cluster":[24511,24550,24488,24499,24569]},"id":1}]} 2024-03-24T00:14:51.961 INFO:teuthology.orchestra.run.smithi118.stderr:dumped fsmap epoch 27 2024-03-24T00:14:51.963 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:51.964+0000 7ff852600640 1 -- 172.21.15.118:0/202068537 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7ff8340a2800 msgr2=0x7ff8340a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:14:51.963 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:51.964+0000 7ff852600640 1 --2- 172.21.15.118:0/202068537 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7ff8340a2800 0x7ff8340a4cc0 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7ff83802c240 tx=0x7ff83802df90 comp rx=0 tx=0).stop 2024-03-24T00:14:51.964 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:51.964+0000 7ff852600640 1 -- 172.21.15.118:0/202068537 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7ff84c101500 msgr2=0x7ff84c199990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:14:51.964 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:51.964+0000 7ff852600640 1 --2- 172.21.15.118:0/202068537 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7ff84c101500 0x7ff84c199990 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7ff84000e9c0 tx=0x7ff84000ee80 comp rx=0 tx=0).stop 2024-03-24T00:14:51.964 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:51.964+0000 7ff852600640 1 -- 172.21.15.118:0/202068537 shutdown_connections 2024-03-24T00:14:51.964 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:51.964+0000 7ff852600640 1 --2- 172.21.15.118:0/202068537 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7ff8340a2800 0x7ff8340a4cc0 unknown :-1 s=CLOSED pgs=212 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:14:51.964 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:51.964+0000 7ff852600640 1 --2- 172.21.15.118:0/202068537 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7ff84c1030b0 0x7ff84c1a0f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:14:51.964 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:51.964+0000 7ff852600640 1 --2- 172.21.15.118:0/202068537 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff84c1026f0 0x7ff84c199ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:14:51.964 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:51.964+0000 7ff852600640 1 --2- 172.21.15.118:0/202068537 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7ff84c101500 0x7ff84c199990 unknown :-1 s=CLOSED pgs=131 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:14:51.964 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:51.964+0000 7ff852600640 1 -- 172.21.15.118:0/202068537 >> 172.21.15.118:0/202068537 conn(0x7ff84c0fca70 msgr2=0x7ff84c0738d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:14:51.964 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:51.964+0000 7ff852600640 1 -- 172.21.15.118:0/202068537 shutdown_connections 2024-03-24T00:14:51.964 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:51.964+0000 7ff852600640 1 -- 172.21.15.118:0/202068537 wait complete. 2024-03-24T00:14:51.982 DEBUG:teuthology.orchestra.run.smithi118:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-24T00:14:52.182 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:52.182+0000 7f9d63a00640 1 -- 172.21.15.118:0/3359748026 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f9d5c1017d0 msgr2=0x7f9d5c101c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:14:52.182 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:52.182+0000 7f9d63a00640 1 --2- 172.21.15.118:0/3359748026 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f9d5c1017d0 0x7f9d5c101c50 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7f9d44009900 tx=0x7f9d4402bc30 comp rx=0 tx=0).stop 2024-03-24T00:14:52.183 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:52.182+0000 7f9d63a00640 1 -- 172.21.15.118:0/3359748026 shutdown_connections 2024-03-24T00:14:52.183 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:52.182+0000 7f9d63a00640 1 --2- 172.21.15.118:0/3359748026 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f9d5c102190 0x7f9d5c108a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:14:52.183 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:52.182+0000 7f9d63a00640 1 --2- 172.21.15.118:0/3359748026 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f9d5c1017d0 0x7f9d5c101c50 unknown :-1 s=CLOSED pgs=168 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:14:52.183 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:52.182+0000 7f9d63a00640 1 --2- 172.21.15.118:0/3359748026 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9d5c1005e0 0x7f9d5c100a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:14:52.183 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:52.182+0000 7f9d63a00640 1 -- 172.21.15.118:0/3359748026 >> 172.21.15.118:0/3359748026 conn(0x7f9d5c0fbb50 msgr2=0x7f9d5c0fdf90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:14:52.183 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:52.182+0000 7f9d63a00640 1 -- 172.21.15.118:0/3359748026 shutdown_connections 2024-03-24T00:14:52.183 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:52.183+0000 7f9d63a00640 1 -- 172.21.15.118:0/3359748026 wait complete. 2024-03-24T00:14:52.183 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:52.184+0000 7f9d63a00640 1 Processor -- start 2024-03-24T00:14:52.184 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:52.185+0000 7f9d63a00640 1 -- start start 2024-03-24T00:14:52.185 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:52.185+0000 7f9d63a00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9d5c1005e0 0x7f9d5c06a840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:14:52.185 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:52.185+0000 7f9d63a00640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f9d5c1017d0 0x7f9d5c19c6f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:14:52.185 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:52.185+0000 7f9d63a00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f9d5c102190 0x7f9d5c19ed10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:14:52.185 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:52.185+0000 7f9d63a00640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9d5c10c260 con 0x7f9d5c1017d0 2024-03-24T00:14:52.185 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:52.185+0000 7f9d63a00640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9d5c10c0e0 con 0x7f9d5c1005e0 2024-03-24T00:14:52.186 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:52.185+0000 7f9d63a00640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9d5c10c3e0 con 0x7f9d5c102190 2024-03-24T00:14:52.186 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:52.185+0000 7f9d61000640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9d5c1005e0 0x7f9d5c06a840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:14:52.186 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:52.185+0000 7f9d61a00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f9d5c102190 0x7f9d5c19ed10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:14:52.186 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:52.185+0000 7f9d61000640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9d5c1005e0 0x7f9d5c06a840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.118:55734/0 (socket says 172.21.15.118:55734) 2024-03-24T00:14:52.186 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:52.185+0000 7f9d61000640 1 -- 172.21.15.118:0/3463017120 learned_addr learned my addr 172.21.15.118:0/3463017120 (peer_addr_for_me v2:172.21.15.118:0/0) 2024-03-24T00:14:52.186 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:52.185+0000 7f9d5be00640 1 --2- 172.21.15.118:0/3463017120 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f9d5c1017d0 0x7f9d5c19c6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:14:52.186 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:52.186+0000 7f9d61000640 1 -- 172.21.15.118:0/3463017120 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f9d5c102190 msgr2=0x7f9d5c19ed10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:14:52.186 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:52.186+0000 7f9d61000640 1 --2- 172.21.15.118:0/3463017120 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f9d5c102190 0x7f9d5c19ed10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:14:52.186 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:52.186+0000 7f9d61000640 1 -- 172.21.15.118:0/3463017120 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f9d5c1017d0 msgr2=0x7f9d5c19c6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:14:52.186 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:52.186+0000 7f9d61000640 1 --2- 172.21.15.118:0/3463017120 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f9d5c1017d0 0x7f9d5c19c6f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:14:52.186 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:52.186+0000 7f9d61000640 1 -- 172.21.15.118:0/3463017120 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9d5c19f620 con 0x7f9d5c1005e0 2024-03-24T00:14:52.186 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:52.186+0000 7f9d5be00640 1 --2- 172.21.15.118:0/3463017120 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f9d5c1017d0 0x7f9d5c19c6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T00:14:52.187 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:52.186+0000 7f9d61000640 1 --2- 172.21.15.118:0/3463017120 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9d5c1005e0 0x7f9d5c06a840 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f9d4c009d20 tx=0x7f9d4c00c6a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:14:52.187 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:52.186+0000 7f9d59600640 1 -- 172.21.15.118:0/3463017120 <== mon.2 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9d4c010630 con 0x7f9d5c1005e0 2024-03-24T00:14:52.187 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:52.186+0000 7f9d59600640 1 -- 172.21.15.118:0/3463017120 <== mon.2 v2:172.21.15.189:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f9d4c010cb0 con 0x7f9d5c1005e0 2024-03-24T00:14:52.187 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:52.186+0000 7f9d63a00640 1 -- 172.21.15.118:0/3463017120 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9d5c19f8b0 con 0x7f9d5c1005e0 2024-03-24T00:14:52.188 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:52.186+0000 7f9d59600640 1 -- 172.21.15.118:0/3463017120 <== mon.2 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9d4c018de0 con 0x7f9d5c1005e0 2024-03-24T00:14:52.188 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:52.187+0000 7f9d63a00640 1 -- 172.21.15.118:0/3463017120 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9d5c19fdd0 con 0x7f9d5c1005e0 2024-03-24T00:14:52.188 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:52.188+0000 7f9d63a00640 1 -- 172.21.15.118:0/3463017120 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9d5c107900 con 0x7f9d5c1005e0 2024-03-24T00:14:52.189 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:52.190+0000 7f9d59600640 1 -- 172.21.15.118:0/3463017120 <== mon.2 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 18) v1 ==== 137291+0+0 (secure 0 0 0) 0x7f9d4c01f020 con 0x7f9d5c1005e0 2024-03-24T00:14:52.190 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:52.191+0000 7f9d59600640 1 --2- 172.21.15.118:0/3463017120 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7f9d400a2800 0x7f9d400a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:14:52.190 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:52.191+0000 7f9d59600640 1 -- 172.21.15.118:0/3463017120 <== mon.2 v2:172.21.15.189:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 8248+0+0 (secure 0 0 0) 0x7f9d4c0cefb0 con 0x7f9d5c1005e0 2024-03-24T00:14:52.191 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:52.191+0000 7f9d5be00640 1 --2- 172.21.15.118:0/3463017120 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7f9d400a2800 0x7f9d400a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:14:52.191 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:52.192+0000 7f9d5be00640 1 --2- 172.21.15.118:0/3463017120 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7f9d400a2800 0x7f9d400a4cc0 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7f9d44006010 tx=0x7f9d4402c8a0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:14:52.195 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:14:51 smithi118 ceph-mon[31535]: from='client.? 172.21.15.118:0/4146363460' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-24T00:14:52.195 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:14:51 smithi118 ceph-mon[31535]: from='client.? 172.21.15.118:0/1651444317' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-24T00:14:52.195 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:14:51 smithi118 ceph-mon[31535]: pgmap v371: 97 pgs: 97 active+clean; 645 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 85 B/s wr, 8 op/s 2024-03-24T00:14:52.197 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:52.197+0000 7f9d59600640 1 -- 172.21.15.118:0/3463017120 <== mon.2 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f9d4c09bfe0 con 0x7f9d5c1005e0 2024-03-24T00:14:52.203 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:14:51 smithi189 ceph-mon[36187]: from='client.? 172.21.15.118:0/4146363460' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-24T00:14:52.203 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:14:51 smithi189 ceph-mon[36187]: from='client.? 172.21.15.118:0/1651444317' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-24T00:14:52.203 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:14:51 smithi189 ceph-mon[36187]: pgmap v371: 97 pgs: 97 active+clean; 645 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 85 B/s wr, 8 op/s 2024-03-24T00:14:52.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:14:51 smithi193 ceph-mon[36878]: from='client.? 172.21.15.118:0/4146363460' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-24T00:14:52.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:14:51 smithi193 ceph-mon[36878]: from='client.? 172.21.15.118:0/1651444317' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-24T00:14:52.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:14:51 smithi193 ceph-mon[36878]: pgmap v371: 97 pgs: 97 active+clean; 645 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 85 B/s wr, 8 op/s 2024-03-24T00:14:52.399 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:52.400+0000 7f9d63a00640 1 -- 172.21.15.118:0/3463017120 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f9d5c02d300 con 0x7f9d5c1005e0 2024-03-24T00:14:52.401 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:52.401+0000 7f9d59600640 1 -- 172.21.15.118:0/3463017120 <== mon.2 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v88) v1 ==== 74+0+22730 (secure 0 0 0) 0x7f9d4c09b8e0 con 0x7f9d5c1005e0 2024-03-24T00:14:52.403 INFO:teuthology.orchestra.run.smithi118.stdout: 2024-03-24T00:14:52.403 INFO:teuthology.orchestra.run.smithi118.stdout:{"epoch":88,"fsid":"d79a5420-e971-11ee-95cd-87774f69a715","created":"2024-03-24T00:03:58.648524+0000","modified":"2024-03-24T00:14:07.487993+0000","last_up_change":"2024-03-24T00:12:03.809000+0000","last_in_change":"2024-03-24T00:11:44.389344+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-24T00:07:50.190965+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-24T00:12:59.993723+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-24T00:13:01.257347+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"86","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"c9590426-3901-4f68-a5ef-d57aa8fbac92","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6802","nonce":557690061},{"type":"v1","addr":"172.21.15.118:6803","nonce":557690061}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6804","nonce":557690061},{"type":"v1","addr":"172.21.15.118:6805","nonce":557690061}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6808","nonce":557690061},{"type":"v1","addr":"172.21.15.118:6809","nonce":557690061}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6806","nonce":557690061},{"type":"v1","addr":"172.21.15.118:6807","nonce":557690061}]},"public_addr":"172.21.15.118:6803/557690061","cluster_addr":"172.21.15.118:6805/557690061","heartbeat_back_addr":"172.21.15.118:6809/557690061","heartbeat_front_addr":"172.21.15.118:6807/557690061","state":["exists","up"]},{"osd":1,"uuid":"fc7f827e-ecf6-4661-aad0-7a22951892e0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6800","nonce":4203029485},{"type":"v1","addr":"172.21.15.189:6801","nonce":4203029485}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6802","nonce":4203029485},{"type":"v1","addr":"172.21.15.189:6803","nonce":4203029485}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6806","nonce":4203029485},{"type":"v1","addr":"172.21.15.189:6807","nonce":4203029485}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6804","nonce":4203029485},{"type":"v1","addr":"172.21.15.189:6805","nonce":4203029485}]},"public_addr":"172.21.15.189:6801/4203029485","cluster_addr":"172.21.15.189:6803/4203029485","heartbeat_back_addr":"172.21.15.189:6807/4203029485","heartbeat_front_addr":"172.21.15.189:6805/4203029485","state":["exists","up"]},{"osd":2,"uuid":"e73ccfe9-0e5b-4734-a154-a2f0d829fcfb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6800","nonce":3699076971},{"type":"v1","addr":"172.21.15.193:6801","nonce":3699076971}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6802","nonce":3699076971},{"type":"v1","addr":"172.21.15.193:6803","nonce":3699076971}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6806","nonce":3699076971},{"type":"v1","addr":"172.21.15.193:6807","nonce":3699076971}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6804","nonce":3699076971},{"type":"v1","addr":"172.21.15.193:6805","nonce":3699076971}]},"public_addr":"172.21.15.193:6801/3699076971","cluster_addr":"172.21.15.193:6803/3699076971","heartbeat_back_addr":"172.21.15.193:6807/3699076971","heartbeat_front_addr":"172.21.15.193:6805/3699076971","state":["exists","up"]},{"osd":3,"uuid":"63fed43b-43ce-4031-8957-7fe94725c1ed","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6810","nonce":4059617010},{"type":"v1","addr":"172.21.15.118:6811","nonce":4059617010}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6812","nonce":4059617010},{"type":"v1","addr":"172.21.15.118:6813","nonce":4059617010}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6816","nonce":4059617010},{"type":"v1","addr":"172.21.15.118:6817","nonce":4059617010}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6814","nonce":4059617010},{"type":"v1","addr":"172.21.15.118:6815","nonce":4059617010}]},"public_addr":"172.21.15.118:6811/4059617010","cluster_addr":"172.21.15.118:6813/4059617010","heartbeat_back_addr":"172.21.15.118:6817/4059617010","heartbeat_front_addr":"172.21.15.118:6815/4059617010","state":["exists","up"]},{"osd":4,"uuid":"57377f66-c3b4-43c1-8e8e-bc873672906d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":34,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6808","nonce":4075247068},{"type":"v1","addr":"172.21.15.189:6809","nonce":4075247068}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6810","nonce":4075247068},{"type":"v1","addr":"172.21.15.189:6811","nonce":4075247068}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6814","nonce":4075247068},{"type":"v1","addr":"172.21.15.189:6815","nonce":4075247068}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6812","nonce":4075247068},{"type":"v1","addr":"172.21.15.189:6813","nonce":4075247068}]},"public_addr":"172.21.15.189:6809/4075247068","cluster_addr":"172.21.15.189:6811/4075247068","heartbeat_back_addr":"172.21.15.189:6815/4075247068","heartbeat_front_addr":"172.21.15.189:6813/4075247068","state":["exists","up"]},{"osd":5,"uuid":"91970976-2193-4224-81c0-dc96cb022c85","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":39,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6808","nonce":2920363402},{"type":"v1","addr":"172.21.15.193:6809","nonce":2920363402}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6810","nonce":2920363402},{"type":"v1","addr":"172.21.15.193:6811","nonce":2920363402}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6814","nonce":2920363402},{"type":"v1","addr":"172.21.15.193:6815","nonce":2920363402}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6812","nonce":2920363402},{"type":"v1","addr":"172.21.15.193:6813","nonce":2920363402}]},"public_addr":"172.21.15.193:6809/2920363402","cluster_addr":"172.21.15.193:6811/2920363402","heartbeat_back_addr":"172.21.15.193:6815/2920363402","heartbeat_front_addr":"172.21.15.193:6813/2920363402","state":["exists","up"]},{"osd":6,"uuid":"0b77b196-8674-41c6-ad3d-726b5d01c8fd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":47,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6818","nonce":431354576},{"type":"v1","addr":"172.21.15.118:6819","nonce":431354576}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6820","nonce":431354576},{"type":"v1","addr":"172.21.15.118:6821","nonce":431354576}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6824","nonce":431354576},{"type":"v1","addr":"172.21.15.118:6825","nonce":431354576}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6822","nonce":431354576},{"type":"v1","addr":"172.21.15.118:6823","nonce":431354576}]},"public_addr":"172.21.15.118:6819/431354576","cluster_addr":"172.21.15.118:6821/431354576","heartbeat_back_addr":"172.21.15.118:6825/431354576","heartbeat_front_addr":"172.21.15.118:6823/431354576","state":["exists","up"]},{"osd":7,"uuid":"1fbbd38f-4e8e-4d4c-8e98-66f6862bc4bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":52,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6816","nonce":3667611284},{"type":"v1","addr":"172.21.15.189:6817","nonce":3667611284}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6818","nonce":3667611284},{"type":"v1","addr":"172.21.15.189:6819","nonce":3667611284}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6822","nonce":3667611284},{"type":"v1","addr":"172.21.15.189:6823","nonce":3667611284}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6820","nonce":3667611284},{"type":"v1","addr":"172.21.15.189:6821","nonce":3667611284}]},"public_addr":"172.21.15.189:6817/3667611284","cluster_addr":"172.21.15.189:6819/3667611284","heartbeat_back_addr":"172.21.15.189:6823/3667611284","heartbeat_front_addr":"172.21.15.189:6821/3667611284","state":["exists","up"]},{"osd":8,"uuid":"6462c2b5-e4b3-4b0b-bcfa-4de2c89e93e0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":60,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6816","nonce":3853800910},{"type":"v1","addr":"172.21.15.193:6817","nonce":3853800910}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6818","nonce":3853800910},{"type":"v1","addr":"172.21.15.193:6819","nonce":3853800910}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6822","nonce":3853800910},{"type":"v1","addr":"172.21.15.193:6823","nonce":3853800910}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6820","nonce":3853800910},{"type":"v1","addr":"172.21.15.193:6821","nonce":3853800910}]},"public_addr":"172.21.15.193:6817/3853800910","cluster_addr":"172.21.15.193:6819/3853800910","heartbeat_back_addr":"172.21.15.193:6823/3853800910","heartbeat_front_addr":"172.21.15.193:6821/3853800910","state":["exists","up"]},{"osd":9,"uuid":"54be277f-3596-422f-9de5-9005a8571c6f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":68,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6826","nonce":3248293936},{"type":"v1","addr":"172.21.15.118:6827","nonce":3248293936}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6828","nonce":3248293936},{"type":"v1","addr":"172.21.15.118:6829","nonce":3248293936}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6832","nonce":3248293936},{"type":"v1","addr":"172.21.15.118:6833","nonce":3248293936}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6830","nonce":3248293936},{"type":"v1","addr":"172.21.15.118:6831","nonce":3248293936}]},"public_addr":"172.21.15.118:6827/3248293936","cluster_addr":"172.21.15.118:6829/3248293936","heartbeat_back_addr":"172.21.15.118:6833/3248293936","heartbeat_front_addr":"172.21.15.118:6831/3248293936","state":["exists","up"]},{"osd":10,"uuid":"7ba69d69-ac41-44ce-bc39-c049a8ca5900","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":73,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6824","nonce":2112195760},{"type":"v1","addr":"172.21.15.189:6825","nonce":2112195760}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6826","nonce":2112195760},{"type":"v1","addr":"172.21.15.189:6827","nonce":2112195760}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6830","nonce":2112195760},{"type":"v1","addr":"172.21.15.189:6831","nonce":2112195760}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6828","nonce":2112195760},{"type":"v1","addr":"172.21.15.189:6829","nonce":2112195760}]},"public_addr":"172.21.15.189:6825/2112195760","cluster_addr":"172.21.15.189:6827/2112195760","heartbeat_back_addr":"172.21.15.189:6831/2112195760","heartbeat_front_addr":"172.21.15.189:6829/2112195760","state":["exists","up"]},{"osd":11,"uuid":"186f0b18-6b22-4eae-83b7-b21bbcce0817","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":81,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6824","nonce":3880515510},{"type":"v1","addr":"172.21.15.193:6825","nonce":3880515510}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6826","nonce":3880515510},{"type":"v1","addr":"172.21.15.193:6827","nonce":3880515510}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6830","nonce":3880515510},{"type":"v1","addr":"172.21.15.193:6831","nonce":3880515510}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6828","nonce":3880515510},{"type":"v1","addr":"172.21.15.193:6829","nonce":3880515510}]},"public_addr":"172.21.15.193:6825/3880515510","cluster_addr":"172.21.15.193:6827/3880515510","heartbeat_back_addr":"172.21.15.193:6831/3880515510","heartbeat_front_addr":"172.21.15.193:6829/3880515510","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:06:58.105543+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:07:23.795866+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:07:47.426050+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:08:19.427073+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:08:45.816082+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:09:11.081523+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:09:44.485077+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:10:11.338972+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:10:36.234024+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:11:09.730432+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:11:37.059083+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:12:02.371947+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.a","mappings":[{"from":2,"to":11}]},{"pgid":"2.14","mappings":[{"from":2,"to":4}]},{"pgid":"2.1f","mappings":[{"from":8,"to":1}]},{"pgid":"3.11","mappings":[{"from":4,"to":8},{"from":5,"to":9}]},{"pgid":"3.13","mappings":[{"from":4,"to":9}]},{"pgid":"3.14","mappings":[{"from":4,"to":6}]},{"pgid":"3.31","mappings":[{"from":4,"to":7}]},{"pgid":"3.38","mappings":[{"from":3,"to":9}]},{"pgid":"3.39","mappings":[{"from":5,"to":8}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.118:6801/389249272":"2024-03-25T00:05:05.458796+0000","172.21.15.118:6800/389249272":"2024-03-25T00:05:05.458796+0000","172.21.15.118:0/2101854694":"2024-03-25T00:05:05.458796+0000","172.21.15.118:0/630779359":"2024-03-25T00:05:05.458796+0000","172.21.15.118:0/2158921029":"2024-03-25T00:04:31.283725+0000","172.21.15.118:0/1003670455":"2024-03-25T00:05:05.458796+0000","172.21.15.118:0/368712314":"2024-03-25T00:04:31.283725+0000","172.21.15.118:0/2004775597":"2024-03-25T00:04:31.283725+0000","172.21.15.118:6800/1463147955":"2024-03-25T00:04:31.283725+0000","172.21.15.118:6801/1463147955":"2024-03-25T00:04:31.283725+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-24T00:14:52.405 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:52.405+0000 7f9d63a00640 1 -- 172.21.15.118:0/3463017120 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7f9d400a2800 msgr2=0x7f9d400a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:14:52.406 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:52.405+0000 7f9d63a00640 1 --2- 172.21.15.118:0/3463017120 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7f9d400a2800 0x7f9d400a4cc0 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7f9d44006010 tx=0x7f9d4402c8a0 comp rx=0 tx=0).stop 2024-03-24T00:14:52.406 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:52.405+0000 7f9d63a00640 1 -- 172.21.15.118:0/3463017120 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9d5c1005e0 msgr2=0x7f9d5c06a840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:14:52.406 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:52.405+0000 7f9d63a00640 1 --2- 172.21.15.118:0/3463017120 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9d5c1005e0 0x7f9d5c06a840 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f9d4c009d20 tx=0x7f9d4c00c6a0 comp rx=0 tx=0).stop 2024-03-24T00:14:52.406 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:52.405+0000 7f9d63a00640 1 -- 172.21.15.118:0/3463017120 shutdown_connections 2024-03-24T00:14:52.406 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:52.406+0000 7f9d63a00640 1 --2- 172.21.15.118:0/3463017120 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7f9d400a2800 0x7f9d400a4cc0 unknown :-1 s=CLOSED pgs=213 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:14:52.406 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:52.406+0000 7f9d63a00640 1 --2- 172.21.15.118:0/3463017120 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f9d5c102190 0x7f9d5c19ed10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:14:52.407 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:52.406+0000 7f9d63a00640 1 --2- 172.21.15.118:0/3463017120 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f9d5c1017d0 0x7f9d5c19c6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:14:52.407 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:52.406+0000 7f9d63a00640 1 --2- 172.21.15.118:0/3463017120 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9d5c1005e0 0x7f9d5c06a840 unknown :-1 s=CLOSED pgs=127 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:14:52.407 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:52.406+0000 7f9d63a00640 1 -- 172.21.15.118:0/3463017120 >> 172.21.15.118:0/3463017120 conn(0x7f9d5c0fbb50 msgr2=0x7f9d5c104ad0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:14:52.407 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:52.406+0000 7f9d63a00640 1 -- 172.21.15.118:0/3463017120 shutdown_connections 2024-03-24T00:14:52.407 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:52.406+0000 7f9d63a00640 1 -- 172.21.15.118:0/3463017120 wait complete. 2024-03-24T00:14:52.426 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-24T00:07:50.190965+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-24T00:12:59.993723+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '85', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 2.25, 'score_stable': 2.25, 'optimal_score': 1, 'raw_score_acting': 2.25, 'raw_score_stable': 2.25, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-03-24T00:13:01.257347+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '86', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-24T00:14:52.427 DEBUG:teuthology.orchestra.run.smithi118:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-24T00:14:52.643 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:52.644+0000 7f497a600640 1 -- 172.21.15.118:0/3678550124 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f49741005c0 msgr2=0x7f49741009e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:14:52.644 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:52.644+0000 7f497a600640 1 --2- 172.21.15.118:0/3678550124 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f49741005c0 0x7f49741009e0 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7f4964009900 tx=0x7f496402bc30 comp rx=0 tx=0).stop 2024-03-24T00:14:52.644 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:52.644+0000 7f497a600640 1 -- 172.21.15.118:0/3678550124 shutdown_connections 2024-03-24T00:14:52.644 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:52.644+0000 7f497a600640 1 --2- 172.21.15.118:0/3678550124 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f4974102170 0x7f4974108a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:14:52.644 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:52.644+0000 7f497a600640 1 --2- 172.21.15.118:0/3678550124 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f49741017b0 0x7f4974101c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:14:52.644 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:52.644+0000 7f497a600640 1 --2- 172.21.15.118:0/3678550124 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f49741005c0 0x7f49741009e0 unknown :-1 s=CLOSED pgs=169 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:14:52.644 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:52.644+0000 7f497a600640 1 -- 172.21.15.118:0/3678550124 >> 172.21.15.118:0/3678550124 conn(0x7f49740fbb50 msgr2=0x7f49740fdf70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:14:52.644 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:52.644+0000 7f497a600640 1 -- 172.21.15.118:0/3678550124 shutdown_connections 2024-03-24T00:14:52.644 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:52.644+0000 7f497a600640 1 -- 172.21.15.118:0/3678550124 wait complete. 2024-03-24T00:14:52.647 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:52.645+0000 7f497a600640 1 Processor -- start 2024-03-24T00:14:52.647 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:52.645+0000 7f497a600640 1 -- start start 2024-03-24T00:14:52.647 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:52.645+0000 7f497a600640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f49741005c0 0x7f4974199c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:14:52.647 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:52.645+0000 7f497a600640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f49741017b0 0x7f497419a1b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:14:52.647 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:52.645+0000 7f497a600640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4974102170 0x7f49741a11e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:14:52.647 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:52.645+0000 7f497a600640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f497410c280 con 0x7f49741005c0 2024-03-24T00:14:52.647 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:52.645+0000 7f497a600640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f497410c100 con 0x7f4974102170 2024-03-24T00:14:52.647 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:52.645+0000 7f497a600640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f497410c400 con 0x7f49741017b0 2024-03-24T00:14:52.647 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:52.646+0000 7f4973400640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f49741005c0 0x7f4974199c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:14:52.648 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:52.646+0000 7f4972a00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f49741017b0 0x7f497419a1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:14:52.648 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:52.646+0000 7f4973e00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4974102170 0x7f49741a11e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:14:52.648 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:52.646+0000 7f4972a00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f49741017b0 0x7f497419a1b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.118:60472/0 (socket says 172.21.15.118:60472) 2024-03-24T00:14:52.648 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:52.646+0000 7f4973400640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f49741005c0 0x7f4974199c70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.118:3300/0 says I am v2:172.21.15.118:41640/0 (socket says 172.21.15.118:41640) 2024-03-24T00:14:52.648 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:52.646+0000 7f4973400640 1 -- 172.21.15.118:0/1108749811 learned_addr learned my addr 172.21.15.118:0/1108749811 (peer_addr_for_me v2:172.21.15.118:0/0) 2024-03-24T00:14:52.648 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:52.646+0000 7f4973e00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4974102170 0x7f49741a11e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.118:55742/0 (socket says 172.21.15.118:55742) 2024-03-24T00:14:52.648 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:52.646+0000 7f4973e00640 1 -- 172.21.15.118:0/1108749811 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f49741017b0 msgr2=0x7f497419a1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:14:52.648 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:52.646+0000 7f4973e00640 1 --2- 172.21.15.118:0/1108749811 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f49741017b0 0x7f497419a1b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:14:52.648 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:52.646+0000 7f4973e00640 1 -- 172.21.15.118:0/1108749811 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f49741005c0 msgr2=0x7f4974199c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:14:52.648 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:52.646+0000 7f4973e00640 1 --2- 172.21.15.118:0/1108749811 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f49741005c0 0x7f4974199c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:14:52.648 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:52.646+0000 7f4973e00640 1 -- 172.21.15.118:0/1108749811 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f49741a1880 con 0x7f4974102170 2024-03-24T00:14:52.648 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:52.646+0000 7f4973e00640 1 --2- 172.21.15.118:0/1108749811 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4974102170 0x7f49741a11e0 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f496c009fd0 tx=0x7f496c00ef10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:14:52.648 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:52.646+0000 7f496be00640 1 -- 172.21.15.118:0/1108749811 <== mon.2 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f496c00ce20 con 0x7f4974102170 2024-03-24T00:14:52.648 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:52.647+0000 7f497a600640 1 -- 172.21.15.118:0/1108749811 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f49741a1b70 con 0x7f4974102170 2024-03-24T00:14:52.649 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:52.647+0000 7f496be00640 1 -- 172.21.15.118:0/1108749811 <== mon.2 v2:172.21.15.189:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f496c005020 con 0x7f4974102170 2024-03-24T00:14:52.649 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:52.647+0000 7f496be00640 1 -- 172.21.15.118:0/1108749811 <== mon.2 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f496c010720 con 0x7f4974102170 2024-03-24T00:14:52.649 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:52.647+0000 7f497a600640 1 -- 172.21.15.118:0/1108749811 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f49741a2070 con 0x7f4974102170 2024-03-24T00:14:52.649 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:52.648+0000 7f497a600640 1 -- 172.21.15.118:0/1108749811 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f49741a22a0 con 0x7f4974102170 2024-03-24T00:14:52.649 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:52.650+0000 7f496be00640 1 -- 172.21.15.118:0/1108749811 <== mon.2 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 18) v1 ==== 137291+0+0 (secure 0 0 0) 0x7f496c005540 con 0x7f4974102170 2024-03-24T00:14:52.650 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:52.651+0000 7f496be00640 1 --2- 172.21.15.118:0/1108749811 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7f49580a2800 0x7f49580a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:14:52.650 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:52.651+0000 7f496be00640 1 -- 172.21.15.118:0/1108749811 <== mon.2 v2:172.21.15.189:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 8248+0+0 (secure 0 0 0) 0x7f496c0cf240 con 0x7f4974102170 2024-03-24T00:14:52.651 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:52.651+0000 7f4973400640 1 --2- 172.21.15.118:0/1108749811 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7f49580a2800 0x7f49580a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:14:52.651 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:52.652+0000 7f4973400640 1 --2- 172.21.15.118:0/1108749811 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7f49580a2800 0x7f49580a4cc0 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7f497419b140 tx=0x7f496402df90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:14:52.658 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:52.659+0000 7f496be00640 1 -- 172.21.15.118:0/1108749811 <== mon.2 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f496c09c270 con 0x7f4974102170 2024-03-24T00:14:52.858 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:52.859+0000 7f497a600640 1 -- 172.21.15.118:0/1108749811 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f49741a2470 con 0x7f4974102170 2024-03-24T00:14:52.859 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:52.860+0000 7f496be00640 1 -- 172.21.15.118:0/1108749811 <== mon.2 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v88) v1 ==== 93+0+10 (secure 0 0 0) 0x7f496c09bb70 con 0x7f4974102170 2024-03-24T00:14:52.859 INFO:teuthology.orchestra.run.smithi118.stdout:pg_num: 1 2024-03-24T00:14:52.865 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:52.863+0000 7f497a600640 1 -- 172.21.15.118:0/1108749811 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7f49580a2800 msgr2=0x7f49580a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:14:52.865 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:52.863+0000 7f497a600640 1 --2- 172.21.15.118:0/1108749811 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7f49580a2800 0x7f49580a4cc0 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7f497419b140 tx=0x7f496402df90 comp rx=0 tx=0).stop 2024-03-24T00:14:52.865 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:52.863+0000 7f497a600640 1 -- 172.21.15.118:0/1108749811 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4974102170 msgr2=0x7f49741a11e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:14:52.866 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:52.863+0000 7f497a600640 1 --2- 172.21.15.118:0/1108749811 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4974102170 0x7f49741a11e0 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f496c009fd0 tx=0x7f496c00ef10 comp rx=0 tx=0).stop 2024-03-24T00:14:52.866 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:52.863+0000 7f497a600640 1 -- 172.21.15.118:0/1108749811 shutdown_connections 2024-03-24T00:14:52.866 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:52.863+0000 7f497a600640 1 --2- 172.21.15.118:0/1108749811 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7f49580a2800 0x7f49580a4cc0 unknown :-1 s=CLOSED pgs=214 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:14:52.866 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:52.863+0000 7f497a600640 1 --2- 172.21.15.118:0/1108749811 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f4974102170 0x7f49741a11e0 unknown :-1 s=CLOSED pgs=128 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:14:52.866 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:52.863+0000 7f497a600640 1 --2- 172.21.15.118:0/1108749811 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f49741017b0 0x7f497419a1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:14:52.866 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:52.863+0000 7f497a600640 1 --2- 172.21.15.118:0/1108749811 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f49741005c0 0x7f4974199c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:14:52.866 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:52.863+0000 7f497a600640 1 -- 172.21.15.118:0/1108749811 >> 172.21.15.118:0/1108749811 conn(0x7f49740fbb50 msgr2=0x7f49740fd610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:14:52.866 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:52.863+0000 7f497a600640 1 -- 172.21.15.118:0/1108749811 shutdown_connections 2024-03-24T00:14:52.866 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:52.863+0000 7f497a600640 1 -- 172.21.15.118:0/1108749811 wait complete. 2024-03-24T00:14:52.882 DEBUG:teuthology.orchestra.run.smithi118:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-24T00:14:53.081 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:53.081+0000 7f6993200640 1 -- 172.21.15.118:0/4282731010 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f698c105490 msgr2=0x7f698c10bd20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:14:53.081 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:53.081+0000 7f6993200640 1 --2- 172.21.15.118:0/4282731010 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f698c105490 0x7f698c10bd20 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7f6984009ca0 tx=0x7f698402c0f0 comp rx=0 tx=0).stop 2024-03-24T00:14:53.081 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:53.081+0000 7f6993200640 1 -- 172.21.15.118:0/4282731010 shutdown_connections 2024-03-24T00:14:53.081 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:53.081+0000 7f6993200640 1 --2- 172.21.15.118:0/4282731010 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f698c105490 0x7f698c10bd20 unknown :-1 s=CLOSED pgs=132 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:14:53.082 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:53.081+0000 7f6993200640 1 --2- 172.21.15.118:0/4282731010 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f698c104ad0 0x7f698c104f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:14:53.082 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:53.081+0000 7f6993200640 1 --2- 172.21.15.118:0/4282731010 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f698c1038e0 0x7f698c103d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:14:53.082 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:53.081+0000 7f6993200640 1 -- 172.21.15.118:0/4282731010 >> 172.21.15.118:0/4282731010 conn(0x7f698c0fee50 msgr2=0x7f698c101290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:14:53.082 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:53.081+0000 7f6993200640 1 -- 172.21.15.118:0/4282731010 shutdown_connections 2024-03-24T00:14:53.082 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:53.082+0000 7f6993200640 1 -- 172.21.15.118:0/4282731010 wait complete. 2024-03-24T00:14:53.083 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:53.083+0000 7f6993200640 1 Processor -- start 2024-03-24T00:14:53.083 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:53.083+0000 7f6993200640 1 -- start start 2024-03-24T00:14:53.084 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:53.083+0000 7f6993200640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f698c1038e0 0x7f698c199bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:14:53.084 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:53.083+0000 7f6993200640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f698c104ad0 0x7f698c19a100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:14:53.084 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:53.083+0000 7f6993200640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f698c105490 0x7f698c1a1170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:14:53.084 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:53.083+0000 7f6993200640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f698c10f560 con 0x7f698c105490 2024-03-24T00:14:53.084 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:53.083+0000 7f6993200640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f698c10f3e0 con 0x7f698c104ad0 2024-03-24T00:14:53.084 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:53.083+0000 7f6993200640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f698c10f6e0 con 0x7f698c1038e0 2024-03-24T00:14:53.084 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:53.083+0000 7f698be00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f698c1038e0 0x7f698c199bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:14:53.084 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:53.083+0000 7f698b400640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f698c104ad0 0x7f698c19a100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:14:53.084 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:53.083+0000 7f698be00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f698c1038e0 0x7f698c199bc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.118:42680/0 (socket says 172.21.15.118:42680) 2024-03-24T00:14:53.084 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:53.083+0000 7f698be00640 1 -- 172.21.15.118:0/3523736110 learned_addr learned my addr 172.21.15.118:0/3523736110 (peer_addr_for_me v2:172.21.15.118:0/0) 2024-03-24T00:14:53.084 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:53.084+0000 7f6991200640 1 --2- 172.21.15.118:0/3523736110 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f698c105490 0x7f698c1a1170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:14:53.084 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:53.084+0000 7f698be00640 1 -- 172.21.15.118:0/3523736110 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f698c104ad0 msgr2=0x7f698c19a100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:14:53.084 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:53.084+0000 7f698be00640 1 --2- 172.21.15.118:0/3523736110 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f698c104ad0 0x7f698c19a100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:14:53.085 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:53.084+0000 7f698be00640 1 -- 172.21.15.118:0/3523736110 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f698c105490 msgr2=0x7f698c1a1170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:14:53.085 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:53.084+0000 7f698be00640 1 --2- 172.21.15.118:0/3523736110 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f698c105490 0x7f698c1a1170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:14:53.085 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:53.084+0000 7f698be00640 1 -- 172.21.15.118:0/3523736110 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f698c1a17e0 con 0x7f698c1038e0 2024-03-24T00:14:53.085 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:53.084+0000 7f698b400640 1 --2- 172.21.15.118:0/3523736110 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f698c104ad0 0x7f698c19a100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T00:14:53.085 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:53.084+0000 7f698be00640 1 --2- 172.21.15.118:0/3523736110 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f698c1038e0 0x7f698c199bc0 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7f697c00e9c0 tx=0x7f697c00ee80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:14:53.085 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:53.084+0000 7f6988c00640 1 -- 172.21.15.118:0/3523736110 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f697c00cd80 con 0x7f698c1038e0 2024-03-24T00:14:53.086 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:53.085+0000 7f6988c00640 1 -- 172.21.15.118:0/3523736110 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f697c0035f0 con 0x7f698c1038e0 2024-03-24T00:14:53.086 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:53.085+0000 7f6988c00640 1 -- 172.21.15.118:0/3523736110 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f697c010690 con 0x7f698c1038e0 2024-03-24T00:14:53.086 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:53.085+0000 7f6993200640 1 -- 172.21.15.118:0/3523736110 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f698c1a1ad0 con 0x7f698c1038e0 2024-03-24T00:14:53.086 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:53.085+0000 7f6993200640 1 -- 172.21.15.118:0/3523736110 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f698c1a1fd0 con 0x7f698c1038e0 2024-03-24T00:14:53.086 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:53.086+0000 7f6993200640 1 -- 172.21.15.118:0/3523736110 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f698c10aba0 con 0x7f698c1038e0 2024-03-24T00:14:53.087 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:53.087+0000 7f6988c00640 1 -- 172.21.15.118:0/3523736110 <== mon.1 v2:172.21.15.193:3300/0 4 ==== mgrmap(e 18) v1 ==== 137291+0+0 (secure 0 0 0) 0x7f697c0037a0 con 0x7f698c1038e0 2024-03-24T00:14:53.089 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:53.090+0000 7f6988c00640 1 --2- 172.21.15.118:0/3523736110 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7f69700a2880 0x7f69700a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:14:53.089 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:53.090+0000 7f698b400640 1 --2- 172.21.15.118:0/3523736110 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7f69700a2880 0x7f69700a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:14:53.089 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:53.090+0000 7f6988c00640 1 -- 172.21.15.118:0/3523736110 <== mon.1 v2:172.21.15.193:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 8248+0+0 (secure 0 0 0) 0x7f697c0cee00 con 0x7f698c1038e0 2024-03-24T00:14:53.090 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:53.091+0000 7f698b400640 1 --2- 172.21.15.118:0/3523736110 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7f69700a2880 0x7f69700a4d40 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7f698c19b0d0 tx=0x7f6978005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:14:53.092 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:53.093+0000 7f6988c00640 1 -- 172.21.15.118:0/3523736110 <== mon.1 v2:172.21.15.193:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f697c09be30 con 0x7f698c1038e0 2024-03-24T00:14:53.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:14:52 smithi118 ceph-mon[31535]: from='client.? 172.21.15.118:0/202068537' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-24T00:14:53.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:14:52 smithi118 ceph-mon[31535]: from='client.? 172.21.15.118:0/3463017120' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-24T00:14:53.203 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:14:52 smithi189 ceph-mon[36187]: from='client.? 172.21.15.118:0/202068537' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-24T00:14:53.203 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:14:52 smithi189 ceph-mon[36187]: from='client.? 172.21.15.118:0/3463017120' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-24T00:14:53.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:14:52 smithi193 ceph-mon[36878]: from='client.? 172.21.15.118:0/202068537' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-24T00:14:53.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:14:52 smithi193 ceph-mon[36878]: from='client.? 172.21.15.118:0/3463017120' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-24T00:14:53.291 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:53.292+0000 7f6993200640 1 -- 172.21.15.118:0/3523736110 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f698c062340 con 0x7f698c1038e0 2024-03-24T00:14:53.292 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:53.292+0000 7f6988c00640 1 -- 172.21.15.118:0/3523736110 <== mon.1 v2:172.21.15.193:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v88) v1 ==== 104+0+11 (secure 0 0 0) 0x7f697c09b730 con 0x7f698c1038e0 2024-03-24T00:14:53.292 INFO:teuthology.orchestra.run.smithi118.stdout:pg_num: 32 2024-03-24T00:14:53.295 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:53.295+0000 7f6993200640 1 -- 172.21.15.118:0/3523736110 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7f69700a2880 msgr2=0x7f69700a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:14:53.295 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:53.295+0000 7f6993200640 1 --2- 172.21.15.118:0/3523736110 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7f69700a2880 0x7f69700a4d40 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7f698c19b0d0 tx=0x7f6978005f90 comp rx=0 tx=0).stop 2024-03-24T00:14:53.295 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:53.295+0000 7f6993200640 1 -- 172.21.15.118:0/3523736110 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f698c1038e0 msgr2=0x7f698c199bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:14:53.295 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:53.295+0000 7f6993200640 1 --2- 172.21.15.118:0/3523736110 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f698c1038e0 0x7f698c199bc0 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7f697c00e9c0 tx=0x7f697c00ee80 comp rx=0 tx=0).stop 2024-03-24T00:14:53.295 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:53.296+0000 7f6993200640 1 -- 172.21.15.118:0/3523736110 shutdown_connections 2024-03-24T00:14:53.296 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:53.296+0000 7f6993200640 1 --2- 172.21.15.118:0/3523736110 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7f69700a2880 0x7f69700a4d40 unknown :-1 s=CLOSED pgs=215 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:14:53.296 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:53.296+0000 7f6993200640 1 --2- 172.21.15.118:0/3523736110 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f698c105490 0x7f698c1a1170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:14:53.296 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:53.296+0000 7f6993200640 1 --2- 172.21.15.118:0/3523736110 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f698c104ad0 0x7f698c19a100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:14:53.296 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:53.296+0000 7f6993200640 1 --2- 172.21.15.118:0/3523736110 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f698c1038e0 0x7f698c199bc0 unknown :-1 s=CLOSED pgs=133 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:14:53.296 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:53.296+0000 7f6993200640 1 -- 172.21.15.118:0/3523736110 >> 172.21.15.118:0/3523736110 conn(0x7f698c0fee50 msgr2=0x7f698c107d70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:14:53.296 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:53.296+0000 7f6993200640 1 -- 172.21.15.118:0/3523736110 shutdown_connections 2024-03-24T00:14:53.296 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:53.296+0000 7f6993200640 1 -- 172.21.15.118:0/3523736110 wait complete. 2024-03-24T00:14:53.313 DEBUG:teuthology.orchestra.run.smithi118:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-24T00:14:53.526 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:53.526+0000 7f67a3800640 1 -- 172.21.15.118:0/1545350500 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f679c1038e0 msgr2=0x7f679c103d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:14:53.527 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:53.526+0000 7f67a3800640 1 --2- 172.21.15.118:0/1545350500 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f679c1038e0 0x7f679c103d00 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f678c009900 tx=0x7f678c02bc30 comp rx=0 tx=0).stop 2024-03-24T00:14:53.527 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:53.527+0000 7f67a3800640 1 -- 172.21.15.118:0/1545350500 shutdown_connections 2024-03-24T00:14:53.527 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:53.527+0000 7f67a3800640 1 --2- 172.21.15.118:0/1545350500 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f679c105490 0x7f679c10bd20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:14:53.527 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:53.527+0000 7f67a3800640 1 --2- 172.21.15.118:0/1545350500 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f679c104ad0 0x7f679c104f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:14:53.527 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:53.527+0000 7f67a3800640 1 --2- 172.21.15.118:0/1545350500 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f679c1038e0 0x7f679c103d00 unknown :-1 s=CLOSED pgs=134 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:14:53.527 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:53.527+0000 7f67a3800640 1 -- 172.21.15.118:0/1545350500 >> 172.21.15.118:0/1545350500 conn(0x7f679c0fee50 msgr2=0x7f679c101290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:14:53.527 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:53.527+0000 7f67a3800640 1 -- 172.21.15.118:0/1545350500 shutdown_connections 2024-03-24T00:14:53.527 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:53.527+0000 7f67a3800640 1 -- 172.21.15.118:0/1545350500 wait complete. 2024-03-24T00:14:53.527 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:53.528+0000 7f67a3800640 1 Processor -- start 2024-03-24T00:14:53.528 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:53.528+0000 7f67a3800640 1 -- start start 2024-03-24T00:14:53.529 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:53.529+0000 7f67a3800640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f679c1038e0 0x7f679c199c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:14:53.529 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:53.529+0000 7f67a3800640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f679c104ad0 0x7f679c19a1b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:14:53.529 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:53.529+0000 7f67a3800640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f679c105490 0x7f679c1a11e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:14:53.529 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:53.529+0000 7f67a3800640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f679c10f580 con 0x7f679c105490 2024-03-24T00:14:53.529 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:53.529+0000 7f67a3800640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f679c10f400 con 0x7f679c104ad0 2024-03-24T00:14:53.529 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:53.529+0000 7f67a3800640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f679c10f700 con 0x7f679c1038e0 2024-03-24T00:14:53.530 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:53.529+0000 7f67a0e00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f679c1038e0 0x7f679c199c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:14:53.530 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:53.529+0000 7f6793e00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f679c104ad0 0x7f679c19a1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:14:53.530 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:53.529+0000 7f67a0e00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f679c1038e0 0x7f679c199c70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.118:42688/0 (socket says 172.21.15.118:42688) 2024-03-24T00:14:53.530 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:53.529+0000 7f67a0e00640 1 -- 172.21.15.118:0/1886094495 learned_addr learned my addr 172.21.15.118:0/1886094495 (peer_addr_for_me v2:172.21.15.118:0/0) 2024-03-24T00:14:53.530 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:53.529+0000 7f67a1800640 1 --2- 172.21.15.118:0/1886094495 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f679c105490 0x7f679c1a11e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:14:53.530 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:53.529+0000 7f67a0e00640 1 -- 172.21.15.118:0/1886094495 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f679c104ad0 msgr2=0x7f679c19a1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:14:53.530 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:53.529+0000 7f67a0e00640 1 --2- 172.21.15.118:0/1886094495 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f679c104ad0 0x7f679c19a1b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:14:53.530 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:53.529+0000 7f67a0e00640 1 -- 172.21.15.118:0/1886094495 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f679c105490 msgr2=0x7f679c1a11e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:14:53.530 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:53.529+0000 7f67a0e00640 1 --2- 172.21.15.118:0/1886094495 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f679c105490 0x7f679c1a11e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:14:53.530 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:53.529+0000 7f67a0e00640 1 -- 172.21.15.118:0/1886094495 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f679c1a1880 con 0x7f679c1038e0 2024-03-24T00:14:53.530 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:53.530+0000 7f67a0e00640 1 --2- 172.21.15.118:0/1886094495 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f679c1038e0 0x7f679c199c70 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7f678c000c00 tx=0x7f678c02c650 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:14:53.531 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:53.530+0000 7f679a000640 1 -- 172.21.15.118:0/1886094495 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f678c043020 con 0x7f679c1038e0 2024-03-24T00:14:53.531 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:53.530+0000 7f679a000640 1 -- 172.21.15.118:0/1886094495 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f678c0054e0 con 0x7f679c1038e0 2024-03-24T00:14:53.531 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:53.530+0000 7f679a000640 1 -- 172.21.15.118:0/1886094495 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f678c003bd0 con 0x7f679c1038e0 2024-03-24T00:14:53.531 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:53.530+0000 7f67a3800640 1 -- 172.21.15.118:0/1886094495 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f679c1a1b10 con 0x7f679c1038e0 2024-03-24T00:14:53.531 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:53.530+0000 7f67a3800640 1 -- 172.21.15.118:0/1886094495 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f679c1a2010 con 0x7f679c1038e0 2024-03-24T00:14:53.531 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:53.532+0000 7f67a3800640 1 -- 172.21.15.118:0/1886094495 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f679c0688e0 con 0x7f679c1038e0 2024-03-24T00:14:53.532 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:53.533+0000 7f679a000640 1 -- 172.21.15.118:0/1886094495 <== mon.1 v2:172.21.15.193:3300/0 4 ==== mgrmap(e 18) v1 ==== 137291+0+0 (secure 0 0 0) 0x7f678c046020 con 0x7f679c1038e0 2024-03-24T00:14:53.534 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:53.534+0000 7f679a000640 1 --2- 172.21.15.118:0/1886094495 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7f67800a2880 0x7f67800a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:14:53.534 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:53.535+0000 7f679a000640 1 -- 172.21.15.118:0/1886094495 <== mon.1 v2:172.21.15.193:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 8248+0+0 (secure 0 0 0) 0x7f678c0f0ad0 con 0x7f679c1038e0 2024-03-24T00:14:53.534 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:53.535+0000 7f6793e00640 1 --2- 172.21.15.118:0/1886094495 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7f67800a2880 0x7f67800a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:14:53.535 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:53.535+0000 7f6793e00640 1 --2- 172.21.15.118:0/1886094495 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7f67800a2880 0x7f67800a4d40 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7f6784009bd0 tx=0x7f6784005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:14:53.538 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:53.538+0000 7f679a000640 1 -- 172.21.15.118:0/1886094495 <== mon.1 v2:172.21.15.193:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f678c04d020 con 0x7f679c1038e0 2024-03-24T00:14:53.754 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:53.754+0000 7f67a3800640 1 -- 172.21.15.118:0/1886094495 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f679c0622a0 con 0x7f679c1038e0 2024-03-24T00:14:53.754 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:53.755+0000 7f679a000640 1 -- 172.21.15.118:0/1886094495 <== mon.1 v2:172.21.15.193:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v88) v1 ==== 100+0+11 (secure 0 0 0) 0x7f678c0bdb00 con 0x7f679c1038e0 2024-03-24T00:14:53.754 INFO:teuthology.orchestra.run.smithi118.stdout:pg_num: 64 2024-03-24T00:14:53.758 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:53.758+0000 7f67a3800640 1 -- 172.21.15.118:0/1886094495 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7f67800a2880 msgr2=0x7f67800a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:14:53.758 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:53.758+0000 7f67a3800640 1 --2- 172.21.15.118:0/1886094495 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7f67800a2880 0x7f67800a4d40 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7f6784009bd0 tx=0x7f6784005f90 comp rx=0 tx=0).stop 2024-03-24T00:14:53.758 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:53.758+0000 7f67a3800640 1 -- 172.21.15.118:0/1886094495 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f679c1038e0 msgr2=0x7f679c199c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:14:53.758 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:53.758+0000 7f67a3800640 1 --2- 172.21.15.118:0/1886094495 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f679c1038e0 0x7f679c199c70 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7f678c000c00 tx=0x7f678c02c650 comp rx=0 tx=0).stop 2024-03-24T00:14:53.758 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:53.758+0000 7f67a3800640 1 -- 172.21.15.118:0/1886094495 shutdown_connections 2024-03-24T00:14:53.758 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:53.758+0000 7f67a3800640 1 --2- 172.21.15.118:0/1886094495 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7f67800a2880 0x7f67800a4d40 unknown :-1 s=CLOSED pgs=216 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:14:53.758 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:53.758+0000 7f67a3800640 1 --2- 172.21.15.118:0/1886094495 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f679c105490 0x7f679c1a11e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:14:53.758 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:53.758+0000 7f67a3800640 1 --2- 172.21.15.118:0/1886094495 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f679c104ad0 0x7f679c19a1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:14:53.759 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:53.758+0000 7f67a3800640 1 --2- 172.21.15.118:0/1886094495 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f679c1038e0 0x7f679c199c70 unknown :-1 s=CLOSED pgs=135 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:14:53.759 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:53.758+0000 7f67a3800640 1 -- 172.21.15.118:0/1886094495 >> 172.21.15.118:0/1886094495 conn(0x7f679c0fee50 msgr2=0x7f679c107df0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:14:53.759 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:53.758+0000 7f67a3800640 1 -- 172.21.15.118:0/1886094495 shutdown_connections 2024-03-24T00:14:53.759 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:53.758+0000 7f67a3800640 1 -- 172.21.15.118:0/1886094495 wait complete. 2024-03-24T00:14:53.776 DEBUG:teuthology.orchestra.run.smithi118:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-24T00:14:53.984 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:53.984+0000 7fa5aac00640 1 -- 172.21.15.118:0/2397759739 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fa5a40756e0 msgr2=0x7fa5a4073b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:14:53.984 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:53.984+0000 7fa5aac00640 1 --2- 172.21.15.118:0/2397759739 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fa5a40756e0 0x7fa5a4073b30 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7fa598009900 tx=0x7fa59802bc40 comp rx=0 tx=0).stop 2024-03-24T00:14:53.984 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:53.984+0000 7fa5aac00640 1 -- 172.21.15.118:0/2397759739 shutdown_connections 2024-03-24T00:14:53.984 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:53.984+0000 7fa5aac00640 1 --2- 172.21.15.118:0/2397759739 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa5a410dae0 0x7fa5a410fed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:14:53.984 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:53.984+0000 7fa5aac00640 1 --2- 172.21.15.118:0/2397759739 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fa5a4074100 0x7fa5a4074580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:14:53.984 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:53.984+0000 7fa5aac00640 1 --2- 172.21.15.118:0/2397759739 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fa5a40756e0 0x7fa5a4073b30 unknown :-1 s=CLOSED pgs=136 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:14:53.985 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:53.984+0000 7fa5aac00640 1 -- 172.21.15.118:0/2397759739 >> 172.21.15.118:0/2397759739 conn(0x7fa5a40fca80 msgr2=0x7fa5a40feec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:14:53.985 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:53.984+0000 7fa5aac00640 1 -- 172.21.15.118:0/2397759739 shutdown_connections 2024-03-24T00:14:53.985 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:53.984+0000 7fa5aac00640 1 -- 172.21.15.118:0/2397759739 wait complete. 2024-03-24T00:14:53.987 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:53.985+0000 7fa5aac00640 1 Processor -- start 2024-03-24T00:14:53.987 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:53.985+0000 7fa5aac00640 1 -- start start 2024-03-24T00:14:53.987 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:53.985+0000 7fa5aac00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa5a4074100 0x7fa5a419dde0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:14:53.987 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:53.985+0000 7fa5aac00640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fa5a40756e0 0x7fa5a419e320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:14:53.987 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:53.986+0000 7fa5aac00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fa5a410dae0 0x7fa5a41a5390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:14:53.987 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:53.986+0000 7fa5aac00640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa5a41137b0 con 0x7fa5a40756e0 2024-03-24T00:14:53.987 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:53.986+0000 7fa5aac00640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa5a4113630 con 0x7fa5a4074100 2024-03-24T00:14:53.987 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:53.986+0000 7fa5aac00640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa5a4113930 con 0x7fa5a410dae0 2024-03-24T00:14:53.987 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:53.986+0000 7fa5a9800640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa5a4074100 0x7fa5a419dde0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:14:53.987 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:53.986+0000 7fa5a9800640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa5a4074100 0x7fa5a419dde0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.118:39082/0 (socket says 172.21.15.118:39082) 2024-03-24T00:14:53.987 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:53.986+0000 7fa5a9800640 1 -- 172.21.15.118:0/1860918744 learned_addr learned my addr 172.21.15.118:0/1860918744 (peer_addr_for_me v2:172.21.15.118:0/0) 2024-03-24T00:14:53.988 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:53.986+0000 7fa5a8e00640 1 --2- 172.21.15.118:0/1860918744 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fa5a40756e0 0x7fa5a419e320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:14:53.988 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:53.986+0000 7fa5aa200640 1 --2- 172.21.15.118:0/1860918744 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fa5a410dae0 0x7fa5a41a5390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:14:53.988 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:53.986+0000 7fa5a9800640 1 -- 172.21.15.118:0/1860918744 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fa5a410dae0 msgr2=0x7fa5a41a5390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:14:53.988 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:53.986+0000 7fa5a9800640 1 --2- 172.21.15.118:0/1860918744 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fa5a410dae0 0x7fa5a41a5390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:14:53.988 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:53.986+0000 7fa5a9800640 1 -- 172.21.15.118:0/1860918744 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fa5a40756e0 msgr2=0x7fa5a419e320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:14:53.988 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:53.986+0000 7fa5a9800640 1 --2- 172.21.15.118:0/1860918744 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fa5a40756e0 0x7fa5a419e320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:14:53.988 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:53.986+0000 7fa5a9800640 1 -- 172.21.15.118:0/1860918744 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa5a41a5a00 con 0x7fa5a4074100 2024-03-24T00:14:53.988 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:53.986+0000 7fa5a9800640 1 --2- 172.21.15.118:0/1860918744 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa5a4074100 0x7fa5a419dde0 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7fa59802c920 tx=0x7fa598005d50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:14:53.988 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:53.987+0000 7fa5a2000640 1 -- 172.21.15.118:0/1860918744 <== mon.2 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa598041070 con 0x7fa5a4074100 2024-03-24T00:14:53.988 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:53.987+0000 7fa5a2000640 1 -- 172.21.15.118:0/1860918744 <== mon.2 v2:172.21.15.189:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fa5980044d0 con 0x7fa5a4074100 2024-03-24T00:14:53.988 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:53.987+0000 7fa5aac00640 1 -- 172.21.15.118:0/1860918744 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa5a41a5c90 con 0x7fa5a4074100 2024-03-24T00:14:53.988 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:53.987+0000 7fa5a2000640 1 -- 172.21.15.118:0/1860918744 <== mon.2 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa598034440 con 0x7fa5a4074100 2024-03-24T00:14:53.988 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:53.987+0000 7fa5aac00640 1 -- 172.21.15.118:0/1860918744 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa5a41a6180 con 0x7fa5a4074100 2024-03-24T00:14:53.989 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:53.988+0000 7fa5aac00640 1 -- 172.21.15.118:0/1860918744 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa5a4103cb0 con 0x7fa5a4074100 2024-03-24T00:14:53.989 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:53.990+0000 7fa5a2000640 1 -- 172.21.15.118:0/1860918744 <== mon.2 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 18) v1 ==== 137291+0+0 (secure 0 0 0) 0x7fa598044070 con 0x7fa5a4074100 2024-03-24T00:14:53.991 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:53.991+0000 7fa5a2000640 1 --2- 172.21.15.118:0/1860918744 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7fa58c0a2800 0x7fa58c0a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:14:53.991 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:53.991+0000 7fa5a2000640 1 -- 172.21.15.118:0/1860918744 <== mon.2 v2:172.21.15.189:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 8248+0+0 (secure 0 0 0) 0x7fa5980f0630 con 0x7fa5a4074100 2024-03-24T00:14:53.991 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:53.991+0000 7fa5a8e00640 1 --2- 172.21.15.118:0/1860918744 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7fa58c0a2800 0x7fa58c0a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:14:53.991 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:53.992+0000 7fa5a8e00640 1 --2- 172.21.15.118:0/1860918744 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7fa58c0a2800 0x7fa58c0a4cc0 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7fa5a419f2f0 tx=0x7fa590005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:14:53.994 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:53.995+0000 7fa5a2000640 1 -- 172.21.15.118:0/1860918744 <== mon.2 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7fa5980bd010 con 0x7fa5a4074100 2024-03-24T00:14:54.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:14:53 smithi118 ceph-mon[31535]: from='client.? 172.21.15.118:0/1108749811' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-24T00:14:54.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:14:53 smithi118 ceph-mon[31535]: from='client.? 172.21.15.118:0/3523736110' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-24T00:14:54.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:14:53 smithi118 ceph-mon[31535]: pgmap v372: 97 pgs: 97 active+clean; 645 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-03-24T00:14:54.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:14:53 smithi118 ceph-mon[31535]: from='client.? 172.21.15.118:0/1886094495' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-24T00:14:54.203 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:14:53 smithi189 ceph-mon[36187]: from='client.? 172.21.15.118:0/1108749811' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-24T00:14:54.203 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:14:53 smithi189 ceph-mon[36187]: from='client.? 172.21.15.118:0/3523736110' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-24T00:14:54.203 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:14:53 smithi189 ceph-mon[36187]: pgmap v372: 97 pgs: 97 active+clean; 645 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-03-24T00:14:54.203 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:14:53 smithi189 ceph-mon[36187]: from='client.? 172.21.15.118:0/1886094495' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-24T00:14:54.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:14:53 smithi193 ceph-mon[36878]: from='client.? 172.21.15.118:0/1108749811' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-24T00:14:54.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:14:53 smithi193 ceph-mon[36878]: from='client.? 172.21.15.118:0/3523736110' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-24T00:14:54.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:14:53 smithi193 ceph-mon[36878]: pgmap v372: 97 pgs: 97 active+clean; 645 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-03-24T00:14:54.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:14:53 smithi193 ceph-mon[36878]: from='client.? 172.21.15.118:0/1886094495' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-24T00:14:54.290 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:54.290+0000 7fa5aac00640 1 -- 172.21.15.118:0/1860918744 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fa5a40622a0 con 0x7fa5a4074100 2024-03-24T00:14:54.291 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:54.291+0000 7fa5a2000640 1 -- 172.21.15.118:0/1860918744 <== mon.2 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 27 v27) v1 ==== 94+0+11478 (secure 0 0 0) 0x7fa5980c10b0 con 0x7fa5a4074100 2024-03-24T00:14:54.291 INFO:teuthology.orchestra.run.smithi118.stdout: 2024-03-24T00:14:54.291 INFO:teuthology.orchestra.run.smithi118.stdout:{"epoch":27,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14385,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.118:6835/3194741373","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6834","nonce":3194741373},{"type":"v1","addr":"172.21.15.118:6835","nonce":3194741373}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":4},{"gid":14397,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.118:6837/1374462956","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6836","nonce":1374462956},{"type":"v1","addr":"172.21.15.118:6837","nonce":1374462956}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":27,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-03-24T00:13:01.782474+0000","modified":"2024-03-24T00:13:09.335361+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24569,"mds_1":24550,"mds_2":24511,"mds_3":24499,"mds_4":24488},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14469":{"gid":14469,"name":"a","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.118:6841/3040946537","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6840","nonce":3040946537},{"type":"v1","addr":"172.21.15.118:6841","nonce":3040946537}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24439":{"gid":24439,"name":"l","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.193:6833/1023428412","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6832","nonce":1023428412},{"type":"v1","addr":"172.21.15.193:6833","nonce":1023428412}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24440":{"gid":24440,"name":"k","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.189:6833/2740024078","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6832","nonce":2740024078},{"type":"v1","addr":"172.21.15.189:6833","nonce":2740024078}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24488":{"gid":24488,"name":"e","rank":4,"incarnation":25,"state":"up:active","state_seq":10,"addr":"172.21.15.189:6835/599473205","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6834","nonce":599473205},{"type":"v1","addr":"172.21.15.189:6835","nonce":599473205}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24499":{"gid":24499,"name":"f","rank":3,"incarnation":23,"state":"up:active","state_seq":9,"addr":"172.21.15.193:6835/2055310541","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6834","nonce":2055310541},{"type":"v1","addr":"172.21.15.193:6835","nonce":2055310541}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24511":{"gid":24511,"name":"c","rank":2,"incarnation":21,"state":"up:active","state_seq":8,"addr":"172.21.15.193:6837/2921520053","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6836","nonce":2921520053},{"type":"v1","addr":"172.21.15.193:6837","nonce":2921520053}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24515":{"gid":24515,"name":"h","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.189:6837/3450187137","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6836","nonce":3450187137},{"type":"v1","addr":"172.21.15.189:6837","nonce":3450187137}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24532":{"gid":24532,"name":"i","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.193:6839/2126842603","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6838","nonce":2126842603},{"type":"v1","addr":"172.21.15.193:6839","nonce":2126842603}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24550":{"gid":24550,"name":"g","rank":1,"incarnation":19,"state":"up:active","state_seq":6,"addr":"172.21.15.118:6839/3285788503","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6838","nonce":3285788503},{"type":"v1","addr":"172.21.15.118:6839","nonce":3285788503}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24569":{"gid":24569,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.189:6839/4074484168","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6838","nonce":4074484168},{"type":"v1","addr":"172.21.15.189:6839","nonce":4074484168}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24569,"qdb_cluster":[24511,24550,24488,24499,24569]},"id":1}]} 2024-03-24T00:14:54.292 INFO:teuthology.orchestra.run.smithi118.stderr:dumped fsmap epoch 27 2024-03-24T00:14:54.294 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:54.294+0000 7fa5aac00640 1 -- 172.21.15.118:0/1860918744 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7fa58c0a2800 msgr2=0x7fa58c0a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:14:54.294 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:54.294+0000 7fa5aac00640 1 --2- 172.21.15.118:0/1860918744 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7fa58c0a2800 0x7fa58c0a4cc0 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7fa5a419f2f0 tx=0x7fa590005f90 comp rx=0 tx=0).stop 2024-03-24T00:14:54.294 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:54.294+0000 7fa5aac00640 1 -- 172.21.15.118:0/1860918744 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa5a4074100 msgr2=0x7fa5a419dde0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:14:54.294 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:54.294+0000 7fa5aac00640 1 --2- 172.21.15.118:0/1860918744 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa5a4074100 0x7fa5a419dde0 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7fa59802c920 tx=0x7fa598005d50 comp rx=0 tx=0).stop 2024-03-24T00:14:54.294 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:54.294+0000 7fa5aac00640 1 -- 172.21.15.118:0/1860918744 shutdown_connections 2024-03-24T00:14:54.294 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:54.294+0000 7fa5aac00640 1 --2- 172.21.15.118:0/1860918744 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7fa58c0a2800 0x7fa58c0a4cc0 unknown :-1 s=CLOSED pgs=217 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:14:54.294 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:54.294+0000 7fa5aac00640 1 --2- 172.21.15.118:0/1860918744 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fa5a410dae0 0x7fa5a41a5390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:14:54.294 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:54.294+0000 7fa5aac00640 1 --2- 172.21.15.118:0/1860918744 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fa5a40756e0 0x7fa5a419e320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:14:54.294 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:54.294+0000 7fa5aac00640 1 --2- 172.21.15.118:0/1860918744 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa5a4074100 0x7fa5a419dde0 unknown :-1 s=CLOSED pgs=129 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:14:54.295 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:54.295+0000 7fa5aac00640 1 -- 172.21.15.118:0/1860918744 >> 172.21.15.118:0/1860918744 conn(0x7fa5a40fca80 msgr2=0x7fa5a4100e80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:14:54.295 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:54.295+0000 7fa5aac00640 1 -- 172.21.15.118:0/1860918744 shutdown_connections 2024-03-24T00:14:54.295 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:54.295+0000 7fa5aac00640 1 -- 172.21.15.118:0/1860918744 wait complete. 2024-03-24T00:14:54.314 DEBUG:teuthology.orchestra.run.smithi118:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-24T00:14:54.515 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:54.516+0000 7f3bd8a00640 1 -- 172.21.15.118:0/1856411270 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3bd4074100 msgr2=0x7f3bd4074580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:14:54.516 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:54.516+0000 7f3bd8a00640 1 --2- 172.21.15.118:0/1856411270 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3bd4074100 0x7f3bd4074580 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f3bbc009930 tx=0x7f3bbc02bc30 comp rx=0 tx=0).stop 2024-03-24T00:14:54.516 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:54.516+0000 7f3bd8a00640 1 -- 172.21.15.118:0/1856411270 shutdown_connections 2024-03-24T00:14:54.516 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:54.516+0000 7f3bd8a00640 1 --2- 172.21.15.118:0/1856411270 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f3bd410dae0 0x7f3bd410fed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:14:54.516 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:54.516+0000 7f3bd8a00640 1 --2- 172.21.15.118:0/1856411270 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3bd4074100 0x7f3bd4074580 unknown :-1 s=CLOSED pgs=130 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:14:54.516 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:54.516+0000 7f3bd8a00640 1 --2- 172.21.15.118:0/1856411270 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f3bd40756e0 0x7f3bd4073b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:14:54.516 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:54.516+0000 7f3bd8a00640 1 -- 172.21.15.118:0/1856411270 >> 172.21.15.118:0/1856411270 conn(0x7f3bd40fca80 msgr2=0x7f3bd40feec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:14:54.516 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:54.516+0000 7f3bd8a00640 1 -- 172.21.15.118:0/1856411270 shutdown_connections 2024-03-24T00:14:54.516 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:54.516+0000 7f3bd8a00640 1 -- 172.21.15.118:0/1856411270 wait complete. 2024-03-24T00:14:54.518 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:54.517+0000 7f3bd8a00640 1 Processor -- start 2024-03-24T00:14:54.518 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:54.517+0000 7f3bd8a00640 1 -- start start 2024-03-24T00:14:54.518 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:54.517+0000 7f3bd8a00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3bd4074100 0x7f3bd419de40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:14:54.518 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:54.517+0000 7f3bd8a00640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f3bd40756e0 0x7f3bd419e380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:14:54.518 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:54.517+0000 7f3bd8a00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f3bd410dae0 0x7f3bd41a53f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:14:54.518 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:54.517+0000 7f3bd8a00640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3bd41137b0 con 0x7f3bd40756e0 2024-03-24T00:14:54.518 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:54.517+0000 7f3bd8a00640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3bd4113630 con 0x7f3bd4074100 2024-03-24T00:14:54.518 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:54.517+0000 7f3bd8a00640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3bd4113930 con 0x7f3bd410dae0 2024-03-24T00:14:54.518 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:54.517+0000 7f3bd3400640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3bd4074100 0x7f3bd419de40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:14:54.519 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:54.517+0000 7f3bd3400640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3bd4074100 0x7f3bd419de40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.118:39090/0 (socket says 172.21.15.118:39090) 2024-03-24T00:14:54.519 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:54.517+0000 7f3bcbe00640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f3bd40756e0 0x7f3bd419e380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:14:54.519 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:54.517+0000 7f3bd3400640 1 -- 172.21.15.118:0/1133924754 learned_addr learned my addr 172.21.15.118:0/1133924754 (peer_addr_for_me v2:172.21.15.118:0/0) 2024-03-24T00:14:54.519 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:54.518+0000 7f3bd3e00640 1 --2- 172.21.15.118:0/1133924754 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f3bd410dae0 0x7f3bd41a53f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:14:54.519 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:54.518+0000 7f3bd3400640 1 -- 172.21.15.118:0/1133924754 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f3bd410dae0 msgr2=0x7f3bd41a53f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:14:54.519 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:54.518+0000 7f3bd3400640 1 --2- 172.21.15.118:0/1133924754 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f3bd410dae0 0x7f3bd41a53f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:14:54.519 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:54.518+0000 7f3bd3400640 1 -- 172.21.15.118:0/1133924754 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f3bd40756e0 msgr2=0x7f3bd419e380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:14:54.519 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:54.518+0000 7f3bd3400640 1 --2- 172.21.15.118:0/1133924754 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f3bd40756e0 0x7f3bd419e380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:14:54.519 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:54.518+0000 7f3bd3400640 1 -- 172.21.15.118:0/1133924754 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3bd41a5a60 con 0x7f3bd4074100 2024-03-24T00:14:54.519 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:54.518+0000 7f3bd3400640 1 --2- 172.21.15.118:0/1133924754 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3bd4074100 0x7f3bd419de40 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7f3bc400e9f0 tx=0x7f3bc400eeb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:14:54.519 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:54.518+0000 7f3bd0c00640 1 -- 172.21.15.118:0/1133924754 <== mon.2 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3bc400cd80 con 0x7f3bd4074100 2024-03-24T00:14:54.519 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:54.518+0000 7f3bd0c00640 1 -- 172.21.15.118:0/1133924754 <== mon.2 v2:172.21.15.189:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f3bc4010910 con 0x7f3bd4074100 2024-03-24T00:14:54.519 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:54.518+0000 7f3bd0c00640 1 -- 172.21.15.118:0/1133924754 <== mon.2 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3bc4018990 con 0x7f3bd4074100 2024-03-24T00:14:54.519 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:54.518+0000 7f3bd8a00640 1 -- 172.21.15.118:0/1133924754 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3bd41a5d50 con 0x7f3bd4074100 2024-03-24T00:14:54.520 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:54.519+0000 7f3bd8a00640 1 -- 172.21.15.118:0/1133924754 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3bd41a61c0 con 0x7f3bd4074100 2024-03-24T00:14:54.520 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:54.520+0000 7f3bd8a00640 1 -- 172.21.15.118:0/1133924754 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3bd4103cb0 con 0x7f3bd4074100 2024-03-24T00:14:54.521 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:54.521+0000 7f3bd0c00640 1 -- 172.21.15.118:0/1133924754 <== mon.2 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 18) v1 ==== 137291+0+0 (secure 0 0 0) 0x7f3bc4010ac0 con 0x7f3bd4074100 2024-03-24T00:14:54.522 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:54.523+0000 7f3bd0c00640 1 --2- 172.21.15.118:0/1133924754 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7f3bb80a2800 0x7f3bb80a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:14:54.522 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:54.523+0000 7f3bd0c00640 1 -- 172.21.15.118:0/1133924754 <== mon.2 v2:172.21.15.189:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 8248+0+0 (secure 0 0 0) 0x7f3bc40ceda0 con 0x7f3bd4074100 2024-03-24T00:14:54.522 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:54.523+0000 7f3bcbe00640 1 --2- 172.21.15.118:0/1133924754 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7f3bb80a2800 0x7f3bb80a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:14:54.523 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:54.524+0000 7f3bcbe00640 1 --2- 172.21.15.118:0/1133924754 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7f3bb80a2800 0x7f3bb80a4cc0 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7f3bbc036040 tx=0x7f3bbc02df90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:14:54.526 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:54.527+0000 7f3bd0c00640 1 -- 172.21.15.118:0/1133924754 <== mon.2 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f3bc409b7e0 con 0x7f3bd4074100 2024-03-24T00:14:54.733 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:54.733+0000 7f3bd8a00640 1 -- 172.21.15.118:0/1133924754 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f3bd40623e0 con 0x7f3bd4074100 2024-03-24T00:14:54.734 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:54.734+0000 7f3bd0c00640 1 -- 172.21.15.118:0/1133924754 <== mon.2 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v88) v1 ==== 74+0+22730 (secure 0 0 0) 0x7f3bc409f880 con 0x7f3bd4074100 2024-03-24T00:14:54.735 INFO:teuthology.orchestra.run.smithi118.stdout: 2024-03-24T00:14:54.735 INFO:teuthology.orchestra.run.smithi118.stdout:{"epoch":88,"fsid":"d79a5420-e971-11ee-95cd-87774f69a715","created":"2024-03-24T00:03:58.648524+0000","modified":"2024-03-24T00:14:07.487993+0000","last_up_change":"2024-03-24T00:12:03.809000+0000","last_in_change":"2024-03-24T00:11:44.389344+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-24T00:07:50.190965+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-24T00:12:59.993723+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":2.25,"score_stable":2.25,"optimal_score":1,"raw_score_acting":2.25,"raw_score_stable":2.25,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-24T00:13:01.257347+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"86","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"c9590426-3901-4f68-a5ef-d57aa8fbac92","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6802","nonce":557690061},{"type":"v1","addr":"172.21.15.118:6803","nonce":557690061}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6804","nonce":557690061},{"type":"v1","addr":"172.21.15.118:6805","nonce":557690061}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6808","nonce":557690061},{"type":"v1","addr":"172.21.15.118:6809","nonce":557690061}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6806","nonce":557690061},{"type":"v1","addr":"172.21.15.118:6807","nonce":557690061}]},"public_addr":"172.21.15.118:6803/557690061","cluster_addr":"172.21.15.118:6805/557690061","heartbeat_back_addr":"172.21.15.118:6809/557690061","heartbeat_front_addr":"172.21.15.118:6807/557690061","state":["exists","up"]},{"osd":1,"uuid":"fc7f827e-ecf6-4661-aad0-7a22951892e0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6800","nonce":4203029485},{"type":"v1","addr":"172.21.15.189:6801","nonce":4203029485}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6802","nonce":4203029485},{"type":"v1","addr":"172.21.15.189:6803","nonce":4203029485}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6806","nonce":4203029485},{"type":"v1","addr":"172.21.15.189:6807","nonce":4203029485}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6804","nonce":4203029485},{"type":"v1","addr":"172.21.15.189:6805","nonce":4203029485}]},"public_addr":"172.21.15.189:6801/4203029485","cluster_addr":"172.21.15.189:6803/4203029485","heartbeat_back_addr":"172.21.15.189:6807/4203029485","heartbeat_front_addr":"172.21.15.189:6805/4203029485","state":["exists","up"]},{"osd":2,"uuid":"e73ccfe9-0e5b-4734-a154-a2f0d829fcfb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6800","nonce":3699076971},{"type":"v1","addr":"172.21.15.193:6801","nonce":3699076971}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6802","nonce":3699076971},{"type":"v1","addr":"172.21.15.193:6803","nonce":3699076971}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6806","nonce":3699076971},{"type":"v1","addr":"172.21.15.193:6807","nonce":3699076971}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6804","nonce":3699076971},{"type":"v1","addr":"172.21.15.193:6805","nonce":3699076971}]},"public_addr":"172.21.15.193:6801/3699076971","cluster_addr":"172.21.15.193:6803/3699076971","heartbeat_back_addr":"172.21.15.193:6807/3699076971","heartbeat_front_addr":"172.21.15.193:6805/3699076971","state":["exists","up"]},{"osd":3,"uuid":"63fed43b-43ce-4031-8957-7fe94725c1ed","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6810","nonce":4059617010},{"type":"v1","addr":"172.21.15.118:6811","nonce":4059617010}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6812","nonce":4059617010},{"type":"v1","addr":"172.21.15.118:6813","nonce":4059617010}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6816","nonce":4059617010},{"type":"v1","addr":"172.21.15.118:6817","nonce":4059617010}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6814","nonce":4059617010},{"type":"v1","addr":"172.21.15.118:6815","nonce":4059617010}]},"public_addr":"172.21.15.118:6811/4059617010","cluster_addr":"172.21.15.118:6813/4059617010","heartbeat_back_addr":"172.21.15.118:6817/4059617010","heartbeat_front_addr":"172.21.15.118:6815/4059617010","state":["exists","up"]},{"osd":4,"uuid":"57377f66-c3b4-43c1-8e8e-bc873672906d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":34,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6808","nonce":4075247068},{"type":"v1","addr":"172.21.15.189:6809","nonce":4075247068}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6810","nonce":4075247068},{"type":"v1","addr":"172.21.15.189:6811","nonce":4075247068}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6814","nonce":4075247068},{"type":"v1","addr":"172.21.15.189:6815","nonce":4075247068}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6812","nonce":4075247068},{"type":"v1","addr":"172.21.15.189:6813","nonce":4075247068}]},"public_addr":"172.21.15.189:6809/4075247068","cluster_addr":"172.21.15.189:6811/4075247068","heartbeat_back_addr":"172.21.15.189:6815/4075247068","heartbeat_front_addr":"172.21.15.189:6813/4075247068","state":["exists","up"]},{"osd":5,"uuid":"91970976-2193-4224-81c0-dc96cb022c85","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":39,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6808","nonce":2920363402},{"type":"v1","addr":"172.21.15.193:6809","nonce":2920363402}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6810","nonce":2920363402},{"type":"v1","addr":"172.21.15.193:6811","nonce":2920363402}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6814","nonce":2920363402},{"type":"v1","addr":"172.21.15.193:6815","nonce":2920363402}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6812","nonce":2920363402},{"type":"v1","addr":"172.21.15.193:6813","nonce":2920363402}]},"public_addr":"172.21.15.193:6809/2920363402","cluster_addr":"172.21.15.193:6811/2920363402","heartbeat_back_addr":"172.21.15.193:6815/2920363402","heartbeat_front_addr":"172.21.15.193:6813/2920363402","state":["exists","up"]},{"osd":6,"uuid":"0b77b196-8674-41c6-ad3d-726b5d01c8fd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":47,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6818","nonce":431354576},{"type":"v1","addr":"172.21.15.118:6819","nonce":431354576}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6820","nonce":431354576},{"type":"v1","addr":"172.21.15.118:6821","nonce":431354576}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6824","nonce":431354576},{"type":"v1","addr":"172.21.15.118:6825","nonce":431354576}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6822","nonce":431354576},{"type":"v1","addr":"172.21.15.118:6823","nonce":431354576}]},"public_addr":"172.21.15.118:6819/431354576","cluster_addr":"172.21.15.118:6821/431354576","heartbeat_back_addr":"172.21.15.118:6825/431354576","heartbeat_front_addr":"172.21.15.118:6823/431354576","state":["exists","up"]},{"osd":7,"uuid":"1fbbd38f-4e8e-4d4c-8e98-66f6862bc4bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":52,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6816","nonce":3667611284},{"type":"v1","addr":"172.21.15.189:6817","nonce":3667611284}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6818","nonce":3667611284},{"type":"v1","addr":"172.21.15.189:6819","nonce":3667611284}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6822","nonce":3667611284},{"type":"v1","addr":"172.21.15.189:6823","nonce":3667611284}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6820","nonce":3667611284},{"type":"v1","addr":"172.21.15.189:6821","nonce":3667611284}]},"public_addr":"172.21.15.189:6817/3667611284","cluster_addr":"172.21.15.189:6819/3667611284","heartbeat_back_addr":"172.21.15.189:6823/3667611284","heartbeat_front_addr":"172.21.15.189:6821/3667611284","state":["exists","up"]},{"osd":8,"uuid":"6462c2b5-e4b3-4b0b-bcfa-4de2c89e93e0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":60,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6816","nonce":3853800910},{"type":"v1","addr":"172.21.15.193:6817","nonce":3853800910}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6818","nonce":3853800910},{"type":"v1","addr":"172.21.15.193:6819","nonce":3853800910}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6822","nonce":3853800910},{"type":"v1","addr":"172.21.15.193:6823","nonce":3853800910}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6820","nonce":3853800910},{"type":"v1","addr":"172.21.15.193:6821","nonce":3853800910}]},"public_addr":"172.21.15.193:6817/3853800910","cluster_addr":"172.21.15.193:6819/3853800910","heartbeat_back_addr":"172.21.15.193:6823/3853800910","heartbeat_front_addr":"172.21.15.193:6821/3853800910","state":["exists","up"]},{"osd":9,"uuid":"54be277f-3596-422f-9de5-9005a8571c6f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":68,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6826","nonce":3248293936},{"type":"v1","addr":"172.21.15.118:6827","nonce":3248293936}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6828","nonce":3248293936},{"type":"v1","addr":"172.21.15.118:6829","nonce":3248293936}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6832","nonce":3248293936},{"type":"v1","addr":"172.21.15.118:6833","nonce":3248293936}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6830","nonce":3248293936},{"type":"v1","addr":"172.21.15.118:6831","nonce":3248293936}]},"public_addr":"172.21.15.118:6827/3248293936","cluster_addr":"172.21.15.118:6829/3248293936","heartbeat_back_addr":"172.21.15.118:6833/3248293936","heartbeat_front_addr":"172.21.15.118:6831/3248293936","state":["exists","up"]},{"osd":10,"uuid":"7ba69d69-ac41-44ce-bc39-c049a8ca5900","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":73,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6824","nonce":2112195760},{"type":"v1","addr":"172.21.15.189:6825","nonce":2112195760}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6826","nonce":2112195760},{"type":"v1","addr":"172.21.15.189:6827","nonce":2112195760}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6830","nonce":2112195760},{"type":"v1","addr":"172.21.15.189:6831","nonce":2112195760}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6828","nonce":2112195760},{"type":"v1","addr":"172.21.15.189:6829","nonce":2112195760}]},"public_addr":"172.21.15.189:6825/2112195760","cluster_addr":"172.21.15.189:6827/2112195760","heartbeat_back_addr":"172.21.15.189:6831/2112195760","heartbeat_front_addr":"172.21.15.189:6829/2112195760","state":["exists","up"]},{"osd":11,"uuid":"186f0b18-6b22-4eae-83b7-b21bbcce0817","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":81,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6824","nonce":3880515510},{"type":"v1","addr":"172.21.15.193:6825","nonce":3880515510}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6826","nonce":3880515510},{"type":"v1","addr":"172.21.15.193:6827","nonce":3880515510}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6830","nonce":3880515510},{"type":"v1","addr":"172.21.15.193:6831","nonce":3880515510}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6828","nonce":3880515510},{"type":"v1","addr":"172.21.15.193:6829","nonce":3880515510}]},"public_addr":"172.21.15.193:6825/3880515510","cluster_addr":"172.21.15.193:6827/3880515510","heartbeat_back_addr":"172.21.15.193:6831/3880515510","heartbeat_front_addr":"172.21.15.193:6829/3880515510","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:06:58.105543+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:07:23.795866+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:07:47.426050+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:08:19.427073+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:08:45.816082+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:09:11.081523+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:09:44.485077+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:10:11.338972+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:10:36.234024+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:11:09.730432+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:11:37.059083+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:12:02.371947+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.a","mappings":[{"from":2,"to":11}]},{"pgid":"2.14","mappings":[{"from":2,"to":4}]},{"pgid":"2.1f","mappings":[{"from":8,"to":1}]},{"pgid":"3.11","mappings":[{"from":4,"to":8},{"from":5,"to":9}]},{"pgid":"3.13","mappings":[{"from":4,"to":9}]},{"pgid":"3.14","mappings":[{"from":4,"to":6}]},{"pgid":"3.31","mappings":[{"from":4,"to":7}]},{"pgid":"3.38","mappings":[{"from":3,"to":9}]},{"pgid":"3.39","mappings":[{"from":5,"to":8}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.118:6801/389249272":"2024-03-25T00:05:05.458796+0000","172.21.15.118:6800/389249272":"2024-03-25T00:05:05.458796+0000","172.21.15.118:0/2101854694":"2024-03-25T00:05:05.458796+0000","172.21.15.118:0/630779359":"2024-03-25T00:05:05.458796+0000","172.21.15.118:0/2158921029":"2024-03-25T00:04:31.283725+0000","172.21.15.118:0/1003670455":"2024-03-25T00:05:05.458796+0000","172.21.15.118:0/368712314":"2024-03-25T00:04:31.283725+0000","172.21.15.118:0/2004775597":"2024-03-25T00:04:31.283725+0000","172.21.15.118:6800/1463147955":"2024-03-25T00:04:31.283725+0000","172.21.15.118:6801/1463147955":"2024-03-25T00:04:31.283725+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-24T00:14:54.738 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:54.737+0000 7f3bd8a00640 1 -- 172.21.15.118:0/1133924754 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7f3bb80a2800 msgr2=0x7f3bb80a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:14:54.738 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:54.737+0000 7f3bd8a00640 1 --2- 172.21.15.118:0/1133924754 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7f3bb80a2800 0x7f3bb80a4cc0 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7f3bbc036040 tx=0x7f3bbc02df90 comp rx=0 tx=0).stop 2024-03-24T00:14:54.738 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:54.737+0000 7f3bd8a00640 1 -- 172.21.15.118:0/1133924754 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3bd4074100 msgr2=0x7f3bd419de40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:14:54.738 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:54.737+0000 7f3bd8a00640 1 --2- 172.21.15.118:0/1133924754 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3bd4074100 0x7f3bd419de40 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7f3bc400e9f0 tx=0x7f3bc400eeb0 comp rx=0 tx=0).stop 2024-03-24T00:14:54.738 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:54.737+0000 7f3bd8a00640 1 -- 172.21.15.118:0/1133924754 shutdown_connections 2024-03-24T00:14:54.739 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:54.738+0000 7f3bd8a00640 1 --2- 172.21.15.118:0/1133924754 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7f3bb80a2800 0x7f3bb80a4cc0 unknown :-1 s=CLOSED pgs=218 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:14:54.739 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:54.738+0000 7f3bd8a00640 1 --2- 172.21.15.118:0/1133924754 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f3bd410dae0 0x7f3bd41a53f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:14:54.739 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:54.738+0000 7f3bd8a00640 1 --2- 172.21.15.118:0/1133924754 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f3bd40756e0 0x7f3bd419e380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:14:54.740 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:54.738+0000 7f3bd8a00640 1 --2- 172.21.15.118:0/1133924754 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3bd4074100 0x7f3bd419de40 unknown :-1 s=CLOSED pgs=131 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:14:54.740 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:54.738+0000 7f3bd8a00640 1 -- 172.21.15.118:0/1133924754 >> 172.21.15.118:0/1133924754 conn(0x7f3bd40fca80 msgr2=0x7f3bd4100e80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:14:54.740 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:54.738+0000 7f3bd8a00640 1 -- 172.21.15.118:0/1133924754 shutdown_connections 2024-03-24T00:14:54.740 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:54.738+0000 7f3bd8a00640 1 -- 172.21.15.118:0/1133924754 wait complete. 2024-03-24T00:14:54.756 DEBUG:teuthology.orchestra.run.smithi118:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-24T00:14:54.953 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:54.953+0000 7fbd67000640 1 -- 172.21.15.118:0/1773161172 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fbd60103700 msgr2=0x7fbd60103b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:14:54.953 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:54.953+0000 7fbd67000640 1 --2- 172.21.15.118:0/1773161172 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fbd60103700 0x7fbd60103b20 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7fbd54009900 tx=0x7fbd5402bc40 comp rx=0 tx=0).stop 2024-03-24T00:14:54.953 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:54.953+0000 7fbd67000640 1 -- 172.21.15.118:0/1773161172 shutdown_connections 2024-03-24T00:14:54.953 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:54.953+0000 7fbd67000640 1 --2- 172.21.15.118:0/1773161172 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fbd601052b0 0x7fbd6010bb40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:14:54.953 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:54.953+0000 7fbd67000640 1 --2- 172.21.15.118:0/1773161172 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fbd601048f0 0x7fbd60104d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:14:54.954 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:54.953+0000 7fbd67000640 1 --2- 172.21.15.118:0/1773161172 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fbd60103700 0x7fbd60103b20 unknown :-1 s=CLOSED pgs=132 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:14:54.954 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:54.953+0000 7fbd67000640 1 -- 172.21.15.118:0/1773161172 >> 172.21.15.118:0/1773161172 conn(0x7fbd600fec70 msgr2=0x7fbd601010b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:14:54.954 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:54.953+0000 7fbd67000640 1 -- 172.21.15.118:0/1773161172 shutdown_connections 2024-03-24T00:14:54.954 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:54.953+0000 7fbd67000640 1 -- 172.21.15.118:0/1773161172 wait complete. 2024-03-24T00:14:54.956 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:54.954+0000 7fbd67000640 1 Processor -- start 2024-03-24T00:14:54.956 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:54.954+0000 7fbd67000640 1 -- start start 2024-03-24T00:14:54.956 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:54.955+0000 7fbd67000640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fbd60103700 0x7fbd60199a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:14:54.956 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:54.955+0000 7fbd67000640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fbd601048f0 0x7fbd60199fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:14:54.956 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:54.955+0000 7fbd67000640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fbd601052b0 0x7fbd601a1040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:14:54.956 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:54.955+0000 7fbd67000640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbd6010f3c0 con 0x7fbd60103700 2024-03-24T00:14:54.956 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:54.955+0000 7fbd67000640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbd6010f240 con 0x7fbd601048f0 2024-03-24T00:14:54.957 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:54.955+0000 7fbd67000640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbd6010f540 con 0x7fbd601052b0 2024-03-24T00:14:54.957 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:54.955+0000 7fbd66600640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fbd601052b0 0x7fbd601a1040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:14:54.957 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:54.955+0000 7fbd65200640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fbd601048f0 0x7fbd60199fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:14:54.957 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:54.955+0000 7fbd65c00640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fbd60103700 0x7fbd60199a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:14:54.957 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:54.955+0000 7fbd66600640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fbd601052b0 0x7fbd601a1040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.118:42742/0 (socket says 172.21.15.118:42742) 2024-03-24T00:14:54.957 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:54.955+0000 7fbd66600640 1 -- 172.21.15.118:0/1969623487 learned_addr learned my addr 172.21.15.118:0/1969623487 (peer_addr_for_me v2:172.21.15.118:0/0) 2024-03-24T00:14:54.957 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:54.955+0000 7fbd65200640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fbd601048f0 0x7fbd60199fd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.118:39094/0 (socket says 172.21.15.118:39094) 2024-03-24T00:14:54.957 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:54.955+0000 7fbd66600640 1 -- 172.21.15.118:0/1969623487 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fbd601048f0 msgr2=0x7fbd60199fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:14:54.957 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:54.955+0000 7fbd66600640 1 --2- 172.21.15.118:0/1969623487 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fbd601048f0 0x7fbd60199fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:14:54.957 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:54.955+0000 7fbd66600640 1 -- 172.21.15.118:0/1969623487 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fbd60103700 msgr2=0x7fbd60199a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:14:54.957 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:54.955+0000 7fbd66600640 1 --2- 172.21.15.118:0/1969623487 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fbd60103700 0x7fbd60199a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:14:54.957 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:54.955+0000 7fbd66600640 1 -- 172.21.15.118:0/1969623487 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbd601a16b0 con 0x7fbd601052b0 2024-03-24T00:14:54.957 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:54.955+0000 7fbd65200640 1 --2- 172.21.15.118:0/1969623487 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fbd601048f0 0x7fbd60199fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T00:14:54.957 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:54.956+0000 7fbd66600640 1 --2- 172.21.15.118:0/1969623487 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fbd601052b0 0x7fbd601a1040 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7fbd5800eb10 tx=0x7fbd5800c2d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:14:54.958 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:54.956+0000 7fbd5e000640 1 -- 172.21.15.118:0/1969623487 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbd58004b20 con 0x7fbd601052b0 2024-03-24T00:14:54.958 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:54.956+0000 7fbd5e000640 1 -- 172.21.15.118:0/1969623487 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fbd58004cc0 con 0x7fbd601052b0 2024-03-24T00:14:54.958 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:54.956+0000 7fbd67000640 1 -- 172.21.15.118:0/1969623487 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbd601a19a0 con 0x7fbd601052b0 2024-03-24T00:14:54.958 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:54.956+0000 7fbd5e000640 1 -- 172.21.15.118:0/1969623487 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbd58010760 con 0x7fbd601052b0 2024-03-24T00:14:54.958 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:54.956+0000 7fbd67000640 1 -- 172.21.15.118:0/1969623487 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbd601a1ea0 con 0x7fbd601052b0 2024-03-24T00:14:54.958 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:54.957+0000 7fbd67000640 1 -- 172.21.15.118:0/1969623487 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbd6010aa40 con 0x7fbd601052b0 2024-03-24T00:14:54.958 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:54.959+0000 7fbd5e000640 1 -- 172.21.15.118:0/1969623487 <== mon.1 v2:172.21.15.193:3300/0 4 ==== mgrmap(e 18) v1 ==== 137291+0+0 (secure 0 0 0) 0x7fbd58005020 con 0x7fbd601052b0 2024-03-24T00:14:54.959 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:54.960+0000 7fbd5e000640 1 --2- 172.21.15.118:0/1969623487 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7fbd480a2800 0x7fbd480a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:14:54.960 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:54.960+0000 7fbd5e000640 1 -- 172.21.15.118:0/1969623487 <== mon.1 v2:172.21.15.193:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 8248+0+0 (secure 0 0 0) 0x7fbd580cefe0 con 0x7fbd601052b0 2024-03-24T00:14:54.960 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:54.960+0000 7fbd65c00640 1 --2- 172.21.15.118:0/1969623487 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7fbd480a2800 0x7fbd480a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:14:54.960 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:54.961+0000 7fbd65c00640 1 --2- 172.21.15.118:0/1969623487 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7fbd480a2800 0x7fbd480a4cc0 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7fbd5402c180 tx=0x7fbd5402cf90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:14:54.968 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:54.968+0000 7fbd5e000640 1 -- 172.21.15.118:0/1969623487 <== mon.1 v2:172.21.15.193:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7fbd5809c070 con 0x7fbd601052b0 2024-03-24T00:14:55.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:14:54 smithi118 ceph-mon[31535]: from='client.? 172.21.15.118:0/1860918744' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-24T00:14:55.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:14:54 smithi118 ceph-mon[31535]: from='client.? 172.21.15.118:0/1133924754' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-24T00:14:55.203 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:14:54 smithi189 ceph-mon[36187]: from='client.? 172.21.15.118:0/1860918744' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-24T00:14:55.203 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:14:54 smithi189 ceph-mon[36187]: from='client.? 172.21.15.118:0/1133924754' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-24T00:14:55.206 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:55.207+0000 7fbd67000640 1 -- 172.21.15.118:0/1969623487 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fbd6002d070 con 0x7fbd601052b0 2024-03-24T00:14:55.207 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:55.208+0000 7fbd5e000640 1 -- 172.21.15.118:0/1969623487 <== mon.1 v2:172.21.15.193:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 27 v27) v1 ==== 94+0+11478 (secure 0 0 0) 0x7fbd5809b970 con 0x7fbd601052b0 2024-03-24T00:14:55.208 INFO:teuthology.orchestra.run.smithi118.stdout: 2024-03-24T00:14:55.208 INFO:teuthology.orchestra.run.smithi118.stdout:{"epoch":27,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14385,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.118:6835/3194741373","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6834","nonce":3194741373},{"type":"v1","addr":"172.21.15.118:6835","nonce":3194741373}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":4},{"gid":14397,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.118:6837/1374462956","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6836","nonce":1374462956},{"type":"v1","addr":"172.21.15.118:6837","nonce":1374462956}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":27,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-03-24T00:13:01.782474+0000","modified":"2024-03-24T00:13:09.335361+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24569,"mds_1":24550,"mds_2":24511,"mds_3":24499,"mds_4":24488},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14469":{"gid":14469,"name":"a","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.118:6841/3040946537","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6840","nonce":3040946537},{"type":"v1","addr":"172.21.15.118:6841","nonce":3040946537}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24439":{"gid":24439,"name":"l","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.193:6833/1023428412","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6832","nonce":1023428412},{"type":"v1","addr":"172.21.15.193:6833","nonce":1023428412}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24440":{"gid":24440,"name":"k","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.189:6833/2740024078","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6832","nonce":2740024078},{"type":"v1","addr":"172.21.15.189:6833","nonce":2740024078}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24488":{"gid":24488,"name":"e","rank":4,"incarnation":25,"state":"up:active","state_seq":10,"addr":"172.21.15.189:6835/599473205","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6834","nonce":599473205},{"type":"v1","addr":"172.21.15.189:6835","nonce":599473205}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24499":{"gid":24499,"name":"f","rank":3,"incarnation":23,"state":"up:active","state_seq":9,"addr":"172.21.15.193:6835/2055310541","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6834","nonce":2055310541},{"type":"v1","addr":"172.21.15.193:6835","nonce":2055310541}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24511":{"gid":24511,"name":"c","rank":2,"incarnation":21,"state":"up:active","state_seq":8,"addr":"172.21.15.193:6837/2921520053","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6836","nonce":2921520053},{"type":"v1","addr":"172.21.15.193:6837","nonce":2921520053}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24515":{"gid":24515,"name":"h","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.189:6837/3450187137","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6836","nonce":3450187137},{"type":"v1","addr":"172.21.15.189:6837","nonce":3450187137}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24532":{"gid":24532,"name":"i","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.193:6839/2126842603","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6838","nonce":2126842603},{"type":"v1","addr":"172.21.15.193:6839","nonce":2126842603}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24550":{"gid":24550,"name":"g","rank":1,"incarnation":19,"state":"up:active","state_seq":6,"addr":"172.21.15.118:6839/3285788503","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6838","nonce":3285788503},{"type":"v1","addr":"172.21.15.118:6839","nonce":3285788503}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24569":{"gid":24569,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.189:6839/4074484168","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6838","nonce":4074484168},{"type":"v1","addr":"172.21.15.189:6839","nonce":4074484168}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24569,"qdb_cluster":[24511,24550,24488,24499,24569]},"id":1}]} 2024-03-24T00:14:55.208 INFO:teuthology.orchestra.run.smithi118.stderr:dumped fsmap epoch 27 2024-03-24T00:14:55.210 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:55.210+0000 7fbd67000640 1 -- 172.21.15.118:0/1969623487 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7fbd480a2800 msgr2=0x7fbd480a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:14:55.211 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:55.210+0000 7fbd67000640 1 --2- 172.21.15.118:0/1969623487 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7fbd480a2800 0x7fbd480a4cc0 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7fbd5402c180 tx=0x7fbd5402cf90 comp rx=0 tx=0).stop 2024-03-24T00:14:55.211 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:55.210+0000 7fbd67000640 1 -- 172.21.15.118:0/1969623487 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fbd601052b0 msgr2=0x7fbd601a1040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:14:55.211 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:55.210+0000 7fbd67000640 1 --2- 172.21.15.118:0/1969623487 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fbd601052b0 0x7fbd601a1040 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7fbd5800eb10 tx=0x7fbd5800c2d0 comp rx=0 tx=0).stop 2024-03-24T00:14:55.211 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:55.211+0000 7fbd67000640 1 -- 172.21.15.118:0/1969623487 shutdown_connections 2024-03-24T00:14:55.211 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:55.211+0000 7fbd67000640 1 --2- 172.21.15.118:0/1969623487 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7fbd480a2800 0x7fbd480a4cc0 unknown :-1 s=CLOSED pgs=219 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:14:55.211 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:55.211+0000 7fbd67000640 1 --2- 172.21.15.118:0/1969623487 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fbd601052b0 0x7fbd601a1040 unknown :-1 s=CLOSED pgs=137 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:14:55.211 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:55.211+0000 7fbd67000640 1 --2- 172.21.15.118:0/1969623487 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fbd601048f0 0x7fbd60199fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:14:55.211 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:55.211+0000 7fbd67000640 1 --2- 172.21.15.118:0/1969623487 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fbd60103700 0x7fbd60199a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:14:55.211 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:55.211+0000 7fbd67000640 1 -- 172.21.15.118:0/1969623487 >> 172.21.15.118:0/1969623487 conn(0x7fbd600fec70 msgr2=0x7fbd60107c10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:14:55.211 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:55.211+0000 7fbd67000640 1 -- 172.21.15.118:0/1969623487 shutdown_connections 2024-03-24T00:14:55.211 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:55.211+0000 7fbd67000640 1 -- 172.21.15.118:0/1969623487 wait complete. 2024-03-24T00:14:55.212 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:14:54 smithi193 ceph-mon[36878]: from='client.? 172.21.15.118:0/1860918744' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-24T00:14:55.212 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:14:54 smithi193 ceph-mon[36878]: from='client.? 172.21.15.118:0/1133924754' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-24T00:14:55.229 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 27, 'flags': 50, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': True, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 32, 'explicitly_allowed_features': 32, 'created': '2024-03-24T00:13:01.782474+0000', 'modified': '2024-03-24T00:13:09.335361+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 24569, 'mds_1': 24550, 'mds_2': 24511, 'mds_3': 24499, 'mds_4': 24488}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_14469': {'gid': 14469, 'name': 'a', 'rank': 4, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.118:6841/3040946537', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.118:6840', 'nonce': 3040946537}, {'type': 'v1', 'addr': '172.21.15.118:6841', 'nonce': 3040946537}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_24439': {'gid': 24439, 'name': 'l', 'rank': 3, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.193:6833/1023428412', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.193:6832', 'nonce': 1023428412}, {'type': 'v1', 'addr': '172.21.15.193:6833', 'nonce': 1023428412}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_24440': {'gid': 24440, 'name': 'k', 'rank': 2, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.189:6833/2740024078', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.189:6832', 'nonce': 2740024078}, {'type': 'v1', 'addr': '172.21.15.189:6833', 'nonce': 2740024078}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_24488': {'gid': 24488, 'name': 'e', 'rank': 4, 'incarnation': 25, 'state': 'up:active', 'state_seq': 10, 'addr': '172.21.15.189:6835/599473205', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.189:6834', 'nonce': 599473205}, {'type': 'v1', 'addr': '172.21.15.189:6835', 'nonce': 599473205}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_24499': {'gid': 24499, 'name': 'f', 'rank': 3, 'incarnation': 23, 'state': 'up:active', 'state_seq': 9, 'addr': '172.21.15.193:6835/2055310541', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.193:6834', 'nonce': 2055310541}, {'type': 'v1', 'addr': '172.21.15.193:6835', 'nonce': 2055310541}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_24511': {'gid': 24511, 'name': 'c', 'rank': 2, 'incarnation': 21, 'state': 'up:active', 'state_seq': 8, 'addr': '172.21.15.193:6837/2921520053', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.193:6836', 'nonce': 2921520053}, {'type': 'v1', 'addr': '172.21.15.193:6837', 'nonce': 2921520053}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_24515': {'gid': 24515, 'name': 'h', 'rank': 1, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.189:6837/3450187137', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.189:6836', 'nonce': 3450187137}, {'type': 'v1', 'addr': '172.21.15.189:6837', 'nonce': 3450187137}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_24532': {'gid': 24532, 'name': 'i', 'rank': 0, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.193:6839/2126842603', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.193:6838', 'nonce': 2126842603}, {'type': 'v1', 'addr': '172.21.15.193:6839', 'nonce': 2126842603}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_24550': {'gid': 24550, 'name': 'g', 'rank': 1, 'incarnation': 19, 'state': 'up:active', 'state_seq': 6, 'addr': '172.21.15.118:6839/3285788503', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.118:6838', 'nonce': 3285788503}, {'type': 'v1', 'addr': '172.21.15.118:6839', 'nonce': 3285788503}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_24569': {'gid': 24569, 'name': 'b', 'rank': 0, 'incarnation': 14, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.189:6839/4074484168', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.189:6838', 'nonce': 4074484168}, {'type': 'v1', 'addr': '172.21.15.189:6839', 'nonce': 4074484168}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 24569, 'qdb_cluster': [24511, 24550, 24488, 24499, 24569]} 2024-03-24T00:14:55.229 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 5/5 2024-03-24T00:14:55.229 DEBUG:teuthology.orchestra.run.smithi118:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.e status 2024-03-24T00:14:55.442 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:55.442+0000 7f614e200640 1 -- 172.21.15.118:0/926818130 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6148105490 msgr2=0x7f614810bd20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:14:55.443 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:55.442+0000 7f614e200640 1 --2- 172.21.15.118:0/926818130 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6148105490 0x7f614810bd20 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7f6140009be0 tx=0x7f614002bef0 comp rx=0 tx=0).stop 2024-03-24T00:14:55.443 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:55.443+0000 7f614e200640 1 -- 172.21.15.118:0/926818130 shutdown_connections 2024-03-24T00:14:55.443 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:55.443+0000 7f614e200640 1 --2- 172.21.15.118:0/926818130 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6148105490 0x7f614810bd20 unknown :-1 s=CLOSED pgs=133 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:14:55.443 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:55.443+0000 7f614e200640 1 --2- 172.21.15.118:0/926818130 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f6148104ad0 0x7f6148104f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:14:55.443 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:55.443+0000 7f614e200640 1 --2- 172.21.15.118:0/926818130 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f61481038e0 0x7f6148103d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:14:55.443 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:55.443+0000 7f614e200640 1 -- 172.21.15.118:0/926818130 >> 172.21.15.118:0/926818130 conn(0x7f61480fee50 msgr2=0x7f6148101290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:14:55.443 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:55.443+0000 7f614e200640 1 -- 172.21.15.118:0/926818130 shutdown_connections 2024-03-24T00:14:55.443 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:55.443+0000 7f614e200640 1 -- 172.21.15.118:0/926818130 wait complete. 2024-03-24T00:14:55.446 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:55.444+0000 7f614e200640 1 Processor -- start 2024-03-24T00:14:55.446 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:55.444+0000 7f614e200640 1 -- start start 2024-03-24T00:14:55.446 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:55.444+0000 7f614e200640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f61481038e0 0x7f6148199c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:14:55.446 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:55.444+0000 7f614e200640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f6148104ad0 0x7f614819a180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:14:55.446 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:55.444+0000 7f614e200640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6148105490 0x7f61481a11b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:14:55.446 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:55.444+0000 7f614e200640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f614810f5b0 con 0x7f6148104ad0 2024-03-24T00:14:55.446 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:55.444+0000 7f614e200640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f614810f430 con 0x7f6148105490 2024-03-24T00:14:55.446 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:55.444+0000 7f614e200640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f614810f730 con 0x7f61481038e0 2024-03-24T00:14:55.446 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:55.445+0000 7f6147400640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f61481038e0 0x7f6148199c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:14:55.446 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:55.445+0000 7f6147400640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f61481038e0 0x7f6148199c40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.118:42770/0 (socket says 172.21.15.118:42770) 2024-03-24T00:14:55.446 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:55.445+0000 7f6147e00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6148105490 0x7f61481a11b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:14:55.446 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:55.445+0000 7f6147400640 1 -- 172.21.15.118:0/1669966247 learned_addr learned my addr 172.21.15.118:0/1669966247 (peer_addr_for_me v2:172.21.15.118:0/0) 2024-03-24T00:14:55.447 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:55.445+0000 7f613fe00640 1 --2- 172.21.15.118:0/1669966247 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f6148104ad0 0x7f614819a180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:14:55.447 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:55.445+0000 7f6147400640 1 -- 172.21.15.118:0/1669966247 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6148105490 msgr2=0x7f61481a11b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:14:55.447 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:55.445+0000 7f6147400640 1 --2- 172.21.15.118:0/1669966247 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6148105490 0x7f61481a11b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:14:55.447 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:55.445+0000 7f6147400640 1 -- 172.21.15.118:0/1669966247 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f6148104ad0 msgr2=0x7f614819a180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:14:55.447 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:55.445+0000 7f6147400640 1 --2- 172.21.15.118:0/1669966247 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f6148104ad0 0x7f614819a180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:14:55.447 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:55.445+0000 7f6147400640 1 -- 172.21.15.118:0/1669966247 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f61481a1850 con 0x7f61481038e0 2024-03-24T00:14:55.447 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:55.445+0000 7f6147e00640 1 --2- 172.21.15.118:0/1669966247 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6148105490 0x7f61481a11b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T00:14:55.447 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:55.445+0000 7f6147400640 1 --2- 172.21.15.118:0/1669966247 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f61481038e0 0x7f6148199c40 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7f613800d950 tx=0x7f613800dd50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:14:55.447 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:55.446+0000 7f6144c00640 1 -- 172.21.15.118:0/1669966247 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6138017020 con 0x7f61481038e0 2024-03-24T00:14:55.447 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:55.446+0000 7f6144c00640 1 -- 172.21.15.118:0/1669966247 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f6138009c30 con 0x7f61481038e0 2024-03-24T00:14:55.447 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:55.446+0000 7f614e200640 1 -- 172.21.15.118:0/1669966247 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f61481a1b40 con 0x7f61481038e0 2024-03-24T00:14:55.447 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:55.446+0000 7f6144c00640 1 -- 172.21.15.118:0/1669966247 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f613800f6b0 con 0x7f61481038e0 2024-03-24T00:14:55.447 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:55.446+0000 7f614e200640 1 -- 172.21.15.118:0/1669966247 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f61481a2090 con 0x7f61481038e0 2024-03-24T00:14:55.450 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:55.450+0000 7f6150fc3740 1 Processor -- start 2024-03-24T00:14:55.450 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:55.450+0000 7f6144c00640 1 -- 172.21.15.118:0/1669966247 <== mon.1 v2:172.21.15.193:3300/0 4 ==== mgrmap(e 18) v1 ==== 137291+0+0 (secure 0 0 0) 0x7f613801a020 con 0x7f61481038e0 2024-03-24T00:14:55.450 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:55.450+0000 7f6150fc3740 1 -- start start 2024-03-24T00:14:55.450 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:55.451+0000 7f6150fc3740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55e970f9cba0 0x55e970f9cfc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:14:55.450 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:55.451+0000 7f6150fc3740 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55e970f9d590 0x55e970fa2090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:14:55.450 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:55.451+0000 7f6150fc3740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55e970fa2700 0x55e970fa4b90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:14:55.450 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:55.451+0000 7f6150fc3740 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e970f01c90 con 0x55e970f9d590 2024-03-24T00:14:55.451 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:55.451+0000 7f6150fc3740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e970eaa980 con 0x55e970fa2700 2024-03-24T00:14:55.451 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:55.451+0000 7f6150fc3740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e970efdd30 con 0x55e970f9cba0 2024-03-24T00:14:55.453 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:55.451+0000 7f6147e00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55e970f9cba0 0x55e970f9cfc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:14:55.453 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:55.451+0000 7f6147400640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55e970f9d590 0x55e970fa2090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:14:55.453 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:55.451+0000 7f6147e00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55e970f9cba0 0x55e970f9cfc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.118:42784/0 (socket says 172.21.15.118:42784) 2024-03-24T00:14:55.453 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:55.451+0000 7f613fe00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55e970fa2700 0x55e970fa4b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:14:55.453 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:55.451+0000 7f6147e00640 1 -- 172.21.15.118:0/243663017 learned_addr learned my addr 172.21.15.118:0/243663017 (peer_addr_for_me v2:172.21.15.118:0/0) 2024-03-24T00:14:55.453 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:55.451+0000 7f6147e00640 1 -- 172.21.15.118:0/243663017 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55e970fa2700 msgr2=0x55e970fa4b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:14:55.454 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:55.451+0000 7f6147e00640 1 --2- 172.21.15.118:0/243663017 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55e970fa2700 0x55e970fa4b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:14:55.454 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:55.451+0000 7f6147e00640 1 -- 172.21.15.118:0/243663017 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55e970f9d590 msgr2=0x55e970fa2090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:14:55.454 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:55.451+0000 7f6147e00640 1 --2- 172.21.15.118:0/243663017 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55e970f9d590 0x55e970fa2090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:14:55.454 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:55.451+0000 7f6147e00640 1 -- 172.21.15.118:0/243663017 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55e970ec2510 con 0x55e970f9cba0 2024-03-24T00:14:55.454 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:55.451+0000 7f613fe00640 1 --2- 172.21.15.118:0/243663017 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55e970fa2700 0x55e970fa4b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T00:14:55.454 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:55.452+0000 7f6147e00640 1 --2- 172.21.15.118:0/243663017 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55e970f9cba0 0x55e970f9cfc0 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f614002c500 tx=0x7f6140033fa0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=e6792aa09fda3270 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:14:55.454 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:55.452+0000 7f6144c00640 1 --2- 172.21.15.118:0/1669966247 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7f612c0a2800 0x7f612c0a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:14:55.454 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:55.452+0000 7f6137e00640 1 -- 172.21.15.118:0/243663017 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6140043030 con 0x55e970f9cba0 2024-03-24T00:14:55.454 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:55.452+0000 7f6137e00640 1 -- 172.21.15.118:0/243663017 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f6140003970 con 0x55e970f9cba0 2024-03-24T00:14:55.454 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:55.452+0000 7f6137e00640 1 -- 172.21.15.118:0/243663017 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f614002cdb0 con 0x55e970f9cba0 2024-03-24T00:14:55.454 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:55.452+0000 7f6144c00640 1 -- 172.21.15.118:0/1669966247 <== mon.1 v2:172.21.15.193:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 8248+0+0 (secure 0 0 0) 0x7f61380ce3a0 con 0x7f61481038e0 2024-03-24T00:14:55.455 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:55.452+0000 7f6147400640 1 --2- 172.21.15.118:0/1669966247 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7f612c0a2800 0x7f612c0a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:14:55.455 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:55.453+0000 7f6150fc3740 1 -- 172.21.15.118:0/243663017 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55e970f9cba0 msgr2=0x55e970f9cfc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:14:55.455 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:55.453+0000 7f6150fc3740 1 --2- 172.21.15.118:0/243663017 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55e970f9cba0 0x55e970f9cfc0 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f614002c500 tx=0x7f6140033fa0 comp rx=0 tx=0).stop 2024-03-24T00:14:55.455 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:55.453+0000 7f6150fc3740 1 -- 172.21.15.118:0/243663017 shutdown_connections 2024-03-24T00:14:55.455 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:55.453+0000 7f6150fc3740 1 --2- 172.21.15.118:0/243663017 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55e970fa2700 0x55e970fa4b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:14:55.455 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:55.453+0000 7f6150fc3740 1 --2- 172.21.15.118:0/243663017 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55e970f9d590 0x55e970fa2090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:14:55.455 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:55.453+0000 7f6150fc3740 1 --2- 172.21.15.118:0/243663017 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55e970f9cba0 0x55e970f9cfc0 unknown :-1 s=CLOSED pgs=139 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:14:55.455 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:55.453+0000 7f6150fc3740 1 -- 172.21.15.118:0/243663017 >> 172.21.15.118:0/243663017 conn(0x55e970f98110 msgr2=0x55e970f98510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:14:55.455 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:55.453+0000 7f6147400640 1 --2- 172.21.15.118:0/1669966247 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7f612c0a2800 0x7f612c0a4cc0 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x55e970f9e000 tx=0x7f61380d1e80 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:14:55.455 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:55.453+0000 7f6150fc3740 1 -- 172.21.15.118:0/243663017 shutdown_connections 2024-03-24T00:14:55.455 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:55.453+0000 7f6150fc3740 1 -- 172.21.15.118:0/243663017 wait complete. 2024-03-24T00:14:55.455 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:55.453+0000 7f6150fc3740 1 -- start start 2024-03-24T00:14:55.455 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:55.454+0000 7f6150fc3740 1 Processor -- start 2024-03-24T00:14:55.459 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:55.454+0000 7f6150fc3740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55e970f9cba0 0x55e970fa7080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:14:55.459 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:55.454+0000 7f6150fc3740 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55e970f9d590 0x55e970fa75c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:14:55.459 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:55.454+0000 7f6150fc3740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55e970fa2700 0x55e970fae600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:14:55.459 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:55.454+0000 7f6150fc3740 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e970fa5650 con 0x55e970f9d590 2024-03-24T00:14:55.459 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:55.454+0000 7f6150fc3740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e970ebddd0 con 0x55e970f9cba0 2024-03-24T00:14:55.460 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:55.454+0000 7f6150fc3740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e970fa57d0 con 0x55e970fa2700 2024-03-24T00:14:55.460 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:55.454+0000 7f6150fc3740 10 client.0 _ll_register_callbacks cb 0x55e970c15ac0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T00:14:55.460 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:55.454+0000 7f6147e00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55e970f9cba0 0x55e970fa7080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:14:55.460 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:55.454+0000 7f6147e00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55e970f9cba0 0x55e970fa7080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.118:39110/0 (socket says 172.21.15.118:39110) 2024-03-24T00:14:55.460 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:55.454+0000 7f6147e00640 1 -- 172.21.15.118:0/1908735758 learned_addr learned my addr 172.21.15.118:0/1908735758 (peer_addr_for_me v2:172.21.15.118:0/0) 2024-03-24T00:14:55.460 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:55.454+0000 7f613fe00640 1 --2- 172.21.15.118:0/1908735758 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55e970f9d590 0x55e970fa75c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:14:55.460 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:55.454+0000 7f6147e00640 1 --2- 172.21.15.118:0/1908735758 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55e970fa2700 0x55e970fae600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:14:55.460 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:55.455+0000 7f6147e00640 1 -- 172.21.15.118:0/1908735758 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55e970fa2700 msgr2=0x55e970fae600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:14:55.460 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:55.455+0000 7f6147e00640 1 --2- 172.21.15.118:0/1908735758 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55e970fa2700 0x55e970fae600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:14:55.460 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:55.455+0000 7f6147e00640 1 -- 172.21.15.118:0/1908735758 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55e970f9d590 msgr2=0x55e970fa75c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:14:55.460 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:55.455+0000 7f6147e00640 1 --2- 172.21.15.118:0/1908735758 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55e970f9d590 0x55e970fa75c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:14:55.460 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:55.455+0000 7f6147e00640 1 -- 172.21.15.118:0/1908735758 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55e970c162d0 con 0x55e970f9cba0 2024-03-24T00:14:55.460 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:55.455+0000 7f6147e00640 1 --2- 172.21.15.118:0/1908735758 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55e970f9cba0 0x55e970fa7080 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f614002cf20 tx=0x7f614002cf90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=11eaf4786904a85d server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:14:55.461 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:55.455+0000 7f6134c00640 10 client.0 ms_handle_connect on v2:172.21.15.189:3300/0 2024-03-24T00:14:55.461 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:55.455+0000 7f6134c00640 1 -- 172.21.15.118:0/1908735758 <== mon.2 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6140043040 con 0x55e970f9cba0 2024-03-24T00:14:55.461 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:55.455+0000 7f6134c00640 1 -- 172.21.15.118:0/1908735758 <== mon.2 v2:172.21.15.189:3300/0 2 ==== osd_map(88..88 src has 1..88) v4 ==== 8248+0+0 (secure 0 0 0) 0x7f6140033630 con 0x55e970f9cba0 2024-03-24T00:14:55.461 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:55.455+0000 7f6150fc3740 1 -- 172.21.15.118:0/1908735758 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f6140003970 con 0x55e970f9cba0 2024-03-24T00:14:55.461 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:55.456+0000 7f6134c00640 1 -- 172.21.15.118:0/1908735758 <== mon.2 v2:172.21.15.189:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) v2 ==== 24+0+0 (secure 0 0 0) 0x7f614005c050 con 0x55e970f9cba0 2024-03-24T00:14:55.461 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:55.456+0000 7f6150fc3740 10 client.24971 fetch_fsmap learned FSMap version 27 2024-03-24T00:14:55.461 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:55.456+0000 7f6150fc3740 1 -- 172.21.15.118:0/1908735758 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) v3 -- 0x55e970ce06b0 con 0x55e970f9cba0 2024-03-24T00:14:55.461 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:55.456+0000 7f6134c00640 1 -- 172.21.15.118:0/1908735758 <== mon.2 v2:172.21.15.189:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f6140003970 con 0x55e970f9cba0 2024-03-24T00:14:55.461 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:55.456+0000 7f6134c00640 1 -- 172.21.15.118:0/1908735758 <== mon.2 v2:172.21.15.189:3300/0 5 ==== fsmap(e 27) v1 ==== 7934+0+0 (secure 0 0 0) 0x7f6140034510 con 0x55e970f9cba0 2024-03-24T00:14:55.461 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:55.456+0000 7f6150fc3740 10 client.24971 fetch_fsmap finished waiting for FSMap version 27 2024-03-24T00:14:55.461 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:55.456+0000 7f6150fc3740 10 client.24971 resolve_mds: resolved name 'e' to daemon mds.e 2024-03-24T00:14:55.461 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:55.456+0000 7f6150fc3740 20 client.24971 populate_metadata read hostname 'smithi118' 2024-03-24T00:14:55.461 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:55.456+0000 7f6150fc3740 1 --2- 172.21.15.118:0/1908735758 >> [v2:172.21.15.189:6834/599473205,v1:172.21.15.189:6835/599473205] conn(0x55e970fb2760 0x55e970fb77e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:14:55.461 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:55.456+0000 7f6150fc3740 4 client.24971 mds_command: new command op to 24488 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-24T00:14:55.462 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:55.456+0000 7f6150fc3740 1 -- 172.21.15.118:0/1908735758 --> [v2:172.21.15.189:6834/599473205,v1:172.21.15.189:6835/599473205] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55e970ea3520 con 0x55e970fb2760 2024-03-24T00:14:55.462 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:55.457+0000 7f613fe00640 1 --2- 172.21.15.118:0/1908735758 >> [v2:172.21.15.189:6834/599473205,v1:172.21.15.189:6835/599473205] conn(0x55e970fb2760 0x55e970fb77e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:14:55.462 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:55.457+0000 7f613fe00640 1 --2- 172.21.15.118:0/1908735758 >> [v2:172.21.15.189:6834/599473205,v1:172.21.15.189:6835/599473205] conn(0x55e970fb2760 0x55e970fb77e0 crc :-1 s=READY pgs=16 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=3eee5e14ab0cb4ec server_cookie=320f48c54aa8d03d in_seq=0 out_seq=0 2024-03-24T00:14:55.462 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:55.457+0000 7f6134c00640 10 client.24971 ms_handle_connect on v2:172.21.15.189:6834/599473205 2024-03-24T00:14:55.462 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:55.458+0000 7f6134c00640 1 -- 172.21.15.118:0/1908735758 <== mds.4 v2:172.21.15.189:6834/599473205 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x55e970ea3520 con 0x55e970fb2760 2024-03-24T00:14:55.462 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:55.458+0000 7f6134c00640 10 client.24971 handle_command_reply: tid=0 2024-03-24T00:14:55.462 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:55.458+0000 7f6150fc3740 1 client.24971 shutdown 2024-03-24T00:14:55.462 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:55.458+0000 7f6150fc3740 1 -- 172.21.15.118:0/1908735758 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55e970f9cba0 msgr2=0x55e970fa7080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:14:55.462 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:55.458+0000 7f6150fc3740 1 --2- 172.21.15.118:0/1908735758 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55e970f9cba0 0x55e970fa7080 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f614002cf20 tx=0x7f614002cf90 comp rx=0 tx=0).stop 2024-03-24T00:14:55.462 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:55.458+0000 7f6150fc3740 1 -- 172.21.15.118:0/1908735758 shutdown_connections 2024-03-24T00:14:55.462 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:55.458+0000 7f6150fc3740 1 --2- 172.21.15.118:0/1908735758 >> [v2:172.21.15.189:6834/599473205,v1:172.21.15.189:6835/599473205] conn(0x55e970fb2760 0x55e970fb77e0 crc :-1 s=READY pgs=16 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:14:55.462 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:55.458+0000 7f6150fc3740 1 --2- 172.21.15.118:0/1908735758 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55e970fa2700 0x55e970fae600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:14:55.462 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:55.458+0000 7f6150fc3740 1 --2- 172.21.15.118:0/1908735758 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55e970f9d590 0x55e970fa75c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:14:55.462 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:55.458+0000 7f6150fc3740 1 --2- 172.21.15.118:0/1908735758 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55e970f9cba0 0x55e970fa7080 unknown :-1 s=CLOSED pgs=134 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:14:55.463 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:55.458+0000 7f6150fc3740 1 -- 172.21.15.118:0/1908735758 >> 172.21.15.118:0/1908735758 conn(0x55e970f98110 msgr2=0x55e970f97310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:14:55.463 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:55.458+0000 7f6134c00640 0 client.24971 ms_handle_reset on v2:172.21.15.189:6834/599473205 2024-03-24T00:14:55.463 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:55.458+0000 7f6150fc3740 1 -- 172.21.15.118:0/1908735758 shutdown_connections 2024-03-24T00:14:55.463 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:55.458+0000 7f6150fc3740 1 -- 172.21.15.118:0/1908735758 wait complete. 2024-03-24T00:14:55.463 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:55.458+0000 7f6150fc3740 20 client.24971 trim_cache size 0 max 16384 2024-03-24T00:14:55.477 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:55.476+0000 7f6150fc3740 1 Processor -- start 2024-03-24T00:14:55.477 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:55.477+0000 7f6150fc3740 1 -- start start 2024-03-24T00:14:55.477 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:55.477+0000 7f6150fc3740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55e970f9cba0 0x55e970f92fc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:14:55.477 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:55.477+0000 7f6150fc3740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55e970f9d590 0x55e970fae590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:14:55.477 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:55.477+0000 7f6150fc3740 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55e970fa2700 0x55e970f9e030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:14:55.478 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:55.477+0000 7f6150fc3740 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e970fb2b80 con 0x55e970fa2700 2024-03-24T00:14:55.478 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:55.477+0000 7f6150fc3740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e970fb31e0 con 0x55e970f9d590 2024-03-24T00:14:55.478 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:55.477+0000 7f6150fc3740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e970fb3840 con 0x55e970f9cba0 2024-03-24T00:14:55.478 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:55.477+0000 7f613fe00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55e970f9d590 0x55e970fae590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:14:55.478 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:55.477+0000 7f613fe00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55e970f9d590 0x55e970fae590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.118:39118/0 (socket says 172.21.15.118:39118) 2024-03-24T00:14:55.478 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:55.477+0000 7f613fe00640 1 -- 172.21.15.118:0/1879346786 learned_addr learned my addr 172.21.15.118:0/1879346786 (peer_addr_for_me v2:172.21.15.118:0/0) 2024-03-24T00:14:55.478 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:55.477+0000 7f6147e00640 1 --2- 172.21.15.118:0/1879346786 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55e970f9cba0 0x55e970f92fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:14:55.478 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:55.477+0000 7f6147e00640 1 --2- 172.21.15.118:0/1879346786 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55e970fa2700 0x55e970f9e030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:14:55.478 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:55.477+0000 7f613fe00640 1 -- 172.21.15.118:0/1879346786 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55e970f9cba0 msgr2=0x55e970f92fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:14:55.478 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:55.477+0000 7f613fe00640 1 --2- 172.21.15.118:0/1879346786 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55e970f9cba0 0x55e970f92fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:14:55.478 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:55.477+0000 7f613fe00640 1 -- 172.21.15.118:0/1879346786 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55e970fa2700 msgr2=0x55e970f9e030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:14:55.478 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:55.477+0000 7f613fe00640 1 --2- 172.21.15.118:0/1879346786 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55e970fa2700 0x55e970f9e030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:14:55.479 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:55.477+0000 7f613fe00640 1 -- 172.21.15.118:0/1879346786 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55e970fb2990 con 0x55e970f9d590 2024-03-24T00:14:55.479 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:55.477+0000 7f6147e00640 1 --2- 172.21.15.118:0/1879346786 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55e970fa2700 0x55e970f9e030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T00:14:55.479 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:55.478+0000 7f613fe00640 1 --2- 172.21.15.118:0/1879346786 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55e970f9d590 0x55e970fae590 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7f6130005fd0 tx=0x7f6130009260 comp rx=0 tx=0).ready entity=mon.2 client_cookie=50a3c07f1ad7fb06 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:14:55.483 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:55.478+0000 7f6137400640 1 -- 172.21.15.118:0/1879346786 <== mon.2 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f613001f250 con 0x55e970f9d590 2024-03-24T00:14:55.483 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:55.478+0000 7f6137400640 1 -- 172.21.15.118:0/1879346786 <== mon.2 v2:172.21.15.189:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f613001f700 con 0x55e970f9d590 2024-03-24T00:14:55.483 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:55.478+0000 7f6137400640 1 -- 172.21.15.118:0/1879346786 <== mon.2 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6130010690 con 0x55e970f9d590 2024-03-24T00:14:55.483 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:55.478+0000 7f6150fc3740 1 -- 172.21.15.118:0/1879346786 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55e970f9d590 msgr2=0x55e970fae590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:14:55.483 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:55.478+0000 7f6150fc3740 1 --2- 172.21.15.118:0/1879346786 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55e970f9d590 0x55e970fae590 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7f6130005fd0 tx=0x7f6130009260 comp rx=0 tx=0).stop 2024-03-24T00:14:55.483 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:55.479+0000 7f6150fc3740 1 -- 172.21.15.118:0/1879346786 shutdown_connections 2024-03-24T00:14:55.483 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:55.479+0000 7f6150fc3740 1 --2- 172.21.15.118:0/1879346786 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55e970fa2700 0x55e970f9e030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:14:55.483 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:55.479+0000 7f6150fc3740 1 --2- 172.21.15.118:0/1879346786 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55e970f9d590 0x55e970fae590 unknown :-1 s=CLOSED pgs=135 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:14:55.483 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:55.479+0000 7f6150fc3740 1 --2- 172.21.15.118:0/1879346786 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55e970f9cba0 0x55e970f92fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:14:55.483 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:55.479+0000 7f6150fc3740 1 -- 172.21.15.118:0/1879346786 >> 172.21.15.118:0/1879346786 conn(0x55e970f98110 msgr2=0x55e970f984f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:14:55.484 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:55.479+0000 7f6150fc3740 1 -- 172.21.15.118:0/1879346786 shutdown_connections 2024-03-24T00:14:55.484 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:55.479+0000 7f6150fc3740 1 -- 172.21.15.118:0/1879346786 wait complete. 2024-03-24T00:14:55.484 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:55.479+0000 7f6150fc3740 1 -- start start 2024-03-24T00:14:55.484 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:55.479+0000 7f6150fc3740 1 Processor -- start 2024-03-24T00:14:55.484 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:55.480+0000 7f6150fc3740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55e970f9cba0 0x55e970fa9be0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:14:55.484 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:55.480+0000 7f6150fc3740 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55e970f9d590 0x55e970faa120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:14:55.484 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:55.480+0000 7f6150fc3740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55e970fa2700 0x55e970fad0d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:14:55.484 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:55.480+0000 7f6150fc3740 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e970fb4500 con 0x55e970f9d590 2024-03-24T00:14:55.484 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:55.480+0000 7f6150fc3740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e970fb3ea0 con 0x55e970f9cba0 2024-03-24T00:14:55.484 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:55.480+0000 7f6150fc3740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e970fb4b60 con 0x55e970fa2700 2024-03-24T00:14:55.484 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:55.480+0000 7f6150fc3740 10 client.0 _ll_register_callbacks cb 0x7f61380d1d50 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T00:14:55.484 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:55.480+0000 7f6147e00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55e970f9cba0 0x55e970fa9be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:14:55.484 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:55.480+0000 7f6147e00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55e970f9cba0 0x55e970fa9be0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.118:39130/0 (socket says 172.21.15.118:39130) 2024-03-24T00:14:55.484 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:55.480+0000 7f6147e00640 1 -- 172.21.15.118:0/2519547338 learned_addr learned my addr 172.21.15.118:0/2519547338 (peer_addr_for_me v2:172.21.15.118:0/0) 2024-03-24T00:14:55.485 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:55.480+0000 7f613fe00640 1 --2- 172.21.15.118:0/2519547338 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55e970f9d590 0x55e970faa120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:14:55.485 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:55.480+0000 7f6147e00640 1 --2- 172.21.15.118:0/2519547338 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55e970fa2700 0x55e970fad0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:14:55.485 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:55.480+0000 7f6147e00640 1 -- 172.21.15.118:0/2519547338 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55e970fa2700 msgr2=0x55e970fad0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:14:55.485 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:55.480+0000 7f6147e00640 1 --2- 172.21.15.118:0/2519547338 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55e970fa2700 0x55e970fad0d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:14:55.485 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:55.480+0000 7f6147e00640 1 -- 172.21.15.118:0/2519547338 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55e970f9d590 msgr2=0x55e970faa120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:14:55.485 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:55.481+0000 7f6147e00640 1 --2- 172.21.15.118:0/2519547338 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55e970f9d590 0x55e970faa120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:14:55.485 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:55.481+0000 7f6147e00640 1 -- 172.21.15.118:0/2519547338 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55e970fad6c0 con 0x55e970f9cba0 2024-03-24T00:14:55.485 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:55.481+0000 7f6147e00640 1 --2- 172.21.15.118:0/2519547338 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55e970f9cba0 0x55e970fa9be0 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f614002c4d0 tx=0x7f614002cd30 comp rx=0 tx=0).ready entity=mon.2 client_cookie=6a3a1246ec2552bb server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:14:55.485 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:55.481+0000 7f6134c00640 10 client.0 ms_handle_connect on v2:172.21.15.189:3300/0 2024-03-24T00:14:55.485 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:55.481+0000 7f6134c00640 1 -- 172.21.15.118:0/2519547338 <== mon.2 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6140043040 con 0x55e970f9cba0 2024-03-24T00:14:55.485 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:55.481+0000 7f6134c00640 1 -- 172.21.15.118:0/2519547338 <== mon.2 v2:172.21.15.189:3300/0 2 ==== osd_map(88..88 src has 1..88) v4 ==== 8248+0+0 (secure 0 0 0) 0x7f6140034a80 con 0x55e970f9cba0 2024-03-24T00:14:55.485 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:55.481+0000 7f6150fc3740 1 -- 172.21.15.118:0/2519547338 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f613001f700 con 0x55e970f9cba0 2024-03-24T00:14:55.485 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:55.481+0000 7f6134c00640 1 -- 172.21.15.118:0/2519547338 <== mon.2 v2:172.21.15.189:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) v2 ==== 24+0+0 (secure 0 0 0) 0x7f6140034c80 con 0x55e970f9cba0 2024-03-24T00:14:55.486 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:55.481+0000 7f6150fc3740 10 client.24977 fetch_fsmap learned FSMap version 27 2024-03-24T00:14:55.486 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:55.481+0000 7f6150fc3740 1 -- 172.21.15.118:0/2519547338 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) v3 -- 0x55e97100e900 con 0x55e970f9cba0 2024-03-24T00:14:55.486 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:55.481+0000 7f6134c00640 1 -- 172.21.15.118:0/2519547338 <== mon.2 v2:172.21.15.189:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f613001f700 con 0x55e970f9cba0 2024-03-24T00:14:55.486 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:55.482+0000 7f6134c00640 1 -- 172.21.15.118:0/2519547338 <== mon.2 v2:172.21.15.189:3300/0 5 ==== fsmap(e 27) v1 ==== 7934+0+0 (secure 0 0 0) 0x7f61400042b0 con 0x55e970f9cba0 2024-03-24T00:14:55.486 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:55.482+0000 7f6150fc3740 10 client.24977 fetch_fsmap finished waiting for FSMap version 27 2024-03-24T00:14:55.486 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:55.482+0000 7f6150fc3740 10 client.24977 resolve_mds: resolved name 'e' to daemon mds.e 2024-03-24T00:14:55.486 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:55.482+0000 7f6150fc3740 20 client.24977 populate_metadata read hostname 'smithi118' 2024-03-24T00:14:55.486 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:55.482+0000 7f6150fc3740 1 --2- 172.21.15.118:0/2519547338 >> [v2:172.21.15.189:6834/599473205,v1:172.21.15.189:6835/599473205] conn(0x55e971012cb0 0x55e97100ead0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:14:55.486 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:55.482+0000 7f6150fc3740 4 client.24977 mds_command: new command op to 24488 tid=0 multi_id=0 [{"prefix": "status"}] 2024-03-24T00:14:55.486 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:55.482+0000 7f6150fc3740 1 -- 172.21.15.118:0/2519547338 --> [v2:172.21.15.189:6834/599473205,v1:172.21.15.189:6835/599473205] -- command(tid 0: {"prefix": "status"}) v1 -- 0x55e970f0f1d0 con 0x55e971012cb0 2024-03-24T00:14:55.486 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:55.482+0000 7f613fe00640 1 --2- 172.21.15.118:0/2519547338 >> [v2:172.21.15.189:6834/599473205,v1:172.21.15.189:6835/599473205] conn(0x55e971012cb0 0x55e97100ead0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:14:55.486 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:55.483+0000 7f613fe00640 1 --2- 172.21.15.118:0/2519547338 >> [v2:172.21.15.189:6834/599473205,v1:172.21.15.189:6835/599473205] conn(0x55e971012cb0 0x55e97100ead0 crc :-1 s=READY pgs=17 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=228a62654c5ac6ec server_cookie=8f92da46a6278ab1 in_seq=0 out_seq=0 2024-03-24T00:14:55.486 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:55.483+0000 7f6134c00640 10 client.24977 ms_handle_connect on v2:172.21.15.189:6834/599473205 2024-03-24T00:14:55.487 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:55.483+0000 7f6134c00640 1 -- 172.21.15.118:0/2519547338 <== mds.4 v2:172.21.15.189:6834/599473205 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+327 (crc 0 0 0) 0x55e970f0f1d0 con 0x55e971012cb0 2024-03-24T00:14:55.487 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:55.483+0000 7f6134c00640 10 client.24977 handle_command_reply: tid=0 2024-03-24T00:14:55.487 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:55.483+0000 7f6150fc3740 1 client.24977 shutdown 2024-03-24T00:14:55.487 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:55.483+0000 7f6150fc3740 1 -- 172.21.15.118:0/2519547338 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55e970f9cba0 msgr2=0x55e970fa9be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:14:55.487 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:55.483+0000 7f6150fc3740 1 --2- 172.21.15.118:0/2519547338 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55e970f9cba0 0x55e970fa9be0 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f614002c4d0 tx=0x7f614002cd30 comp rx=0 tx=0).stop 2024-03-24T00:14:55.487 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:55.483+0000 7f6150fc3740 1 -- 172.21.15.118:0/2519547338 shutdown_connections 2024-03-24T00:14:55.487 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:55.483+0000 7f6150fc3740 1 --2- 172.21.15.118:0/2519547338 >> [v2:172.21.15.189:6834/599473205,v1:172.21.15.189:6835/599473205] conn(0x55e971012cb0 0x55e97100ead0 crc :-1 s=READY pgs=17 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:14:55.487 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:55.483+0000 7f6150fc3740 1 --2- 172.21.15.118:0/2519547338 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55e970fa2700 0x55e970fad0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:14:55.487 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:55.483+0000 7f6134c00640 0 client.24977 ms_handle_reset on v2:172.21.15.189:6834/599473205 2024-03-24T00:14:55.487 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:55.483+0000 7f6150fc3740 1 --2- 172.21.15.118:0/2519547338 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55e970f9d590 0x55e970faa120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:14:55.487 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:55.483+0000 7f6150fc3740 1 --2- 172.21.15.118:0/2519547338 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55e970f9cba0 0x55e970fa9be0 unknown :-1 s=CLOSED pgs=136 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:14:55.487 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:55.483+0000 7f6150fc3740 1 -- 172.21.15.118:0/2519547338 >> 172.21.15.118:0/2519547338 conn(0x55e970f98110 msgr2=0x55e970f92e00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:14:55.488 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:55.483+0000 7f6150fc3740 1 -- 172.21.15.118:0/2519547338 shutdown_connections 2024-03-24T00:14:55.488 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:55.483+0000 7f6150fc3740 1 -- 172.21.15.118:0/2519547338 wait complete. 2024-03-24T00:14:55.488 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:55.483+0000 7f6150fc3740 20 client.24977 trim_cache size 0 max 16384 2024-03-24T00:14:55.488 INFO:teuthology.orchestra.run.smithi118.stdout:{ 2024-03-24T00:14:55.488 INFO:teuthology.orchestra.run.smithi118.stdout: "cluster_fsid": "d79a5420-e971-11ee-95cd-87774f69a715", 2024-03-24T00:14:55.488 INFO:teuthology.orchestra.run.smithi118.stdout: "whoami": 4, 2024-03-24T00:14:55.488 INFO:teuthology.orchestra.run.smithi118.stdout: "id": 24488, 2024-03-24T00:14:55.488 INFO:teuthology.orchestra.run.smithi118.stdout: "want_state": "up:active", 2024-03-24T00:14:55.489 INFO:teuthology.orchestra.run.smithi118.stdout: "state": "up:active", 2024-03-24T00:14:55.489 INFO:teuthology.orchestra.run.smithi118.stdout: "fs_name": "cephfs", 2024-03-24T00:14:55.489 INFO:teuthology.orchestra.run.smithi118.stdout: "rank_uptime": 107.15174365999999, 2024-03-24T00:14:55.489 INFO:teuthology.orchestra.run.smithi118.stdout: "mdsmap_epoch": 27, 2024-03-24T00:14:55.489 INFO:teuthology.orchestra.run.smithi118.stdout: "osdmap_epoch": 88, 2024-03-24T00:14:55.489 INFO:teuthology.orchestra.run.smithi118.stdout: "osdmap_epoch_barrier": 86, 2024-03-24T00:14:55.489 INFO:teuthology.orchestra.run.smithi118.stdout: "uptime": 142.802067588 2024-03-24T00:14:55.489 INFO:teuthology.orchestra.run.smithi118.stdout:} 2024-03-24T00:14:55.490 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:55.484+0000 7f614e200640 1 -- 172.21.15.118:0/1669966247 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7f612c0a2800 msgr2=0x7f612c0a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:14:55.490 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:55.484+0000 7f614e200640 1 --2- 172.21.15.118:0/1669966247 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7f612c0a2800 0x7f612c0a4cc0 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x55e970f9e000 tx=0x7f61380d1e80 comp rx=0 tx=0).stop 2024-03-24T00:14:55.490 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:55.484+0000 7f614e200640 1 -- 172.21.15.118:0/1669966247 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f61481038e0 msgr2=0x7f6148199c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:14:55.490 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:55.484+0000 7f614e200640 1 --2- 172.21.15.118:0/1669966247 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f61481038e0 0x7f6148199c40 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7f613800d950 tx=0x7f613800dd50 comp rx=0 tx=0).stop 2024-03-24T00:14:55.490 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:55.484+0000 7f614e200640 1 -- 172.21.15.118:0/1669966247 shutdown_connections 2024-03-24T00:14:55.490 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:55.484+0000 7f614e200640 1 --2- 172.21.15.118:0/1669966247 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7f612c0a2800 0x7f612c0a4cc0 unknown :-1 s=CLOSED pgs=220 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:14:55.490 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:55.484+0000 7f614e200640 1 --2- 172.21.15.118:0/1669966247 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6148105490 0x7f61481a11b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:14:55.490 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:55.484+0000 7f614e200640 1 --2- 172.21.15.118:0/1669966247 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f6148104ad0 0x7f614819a180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:14:55.490 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:55.484+0000 7f614e200640 1 --2- 172.21.15.118:0/1669966247 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f61481038e0 0x7f6148199c40 unknown :-1 s=CLOSED pgs=138 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:14:55.490 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:55.484+0000 7f614e200640 1 -- 172.21.15.118:0/1669966247 >> 172.21.15.118:0/1669966247 conn(0x7f61480fee50 msgr2=0x7f6148107d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:14:55.490 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:55.484+0000 7f614e200640 1 -- 172.21.15.118:0/1669966247 shutdown_connections 2024-03-24T00:14:55.491 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:55.484+0000 7f614e200640 1 -- 172.21.15.118:0/1669966247 wait complete. 2024-03-24T00:14:55.501 DEBUG:teuthology.orchestra.run.smithi118:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.f status 2024-03-24T00:14:55.713 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:55.713+0000 7f41bda00640 1 -- 172.21.15.118:0/4099905703 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f41b8108eb0 msgr2=0x7f41b8109330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:14:55.713 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:55.713+0000 7f41bda00640 1 --2- 172.21.15.118:0/4099905703 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f41b8108eb0 0x7f41b8109330 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7f41a0009860 tx=0x7f41a002bb70 comp rx=0 tx=0).stop 2024-03-24T00:14:55.713 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:55.713+0000 7f41bda00640 1 -- 172.21.15.118:0/4099905703 shutdown_connections 2024-03-24T00:14:55.713 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:55.713+0000 7f41bda00640 1 --2- 172.21.15.118:0/4099905703 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f41b8109870 0x7f41b8110100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:14:55.713 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:55.713+0000 7f41bda00640 1 --2- 172.21.15.118:0/4099905703 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f41b8108eb0 0x7f41b8109330 unknown :-1 s=CLOSED pgs=140 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:14:55.713 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:55.713+0000 7f41bda00640 1 --2- 172.21.15.118:0/4099905703 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f41b8107cc0 0x7f41b81080e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:14:55.713 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:55.713+0000 7f41bda00640 1 -- 172.21.15.118:0/4099905703 >> 172.21.15.118:0/4099905703 conn(0x7f41b8076c60 msgr2=0x7f41b80790a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:14:55.713 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:55.713+0000 7f41bda00640 1 -- 172.21.15.118:0/4099905703 shutdown_connections 2024-03-24T00:14:55.714 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:55.713+0000 7f41bda00640 1 -- 172.21.15.118:0/4099905703 wait complete. 2024-03-24T00:14:55.715 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:55.714+0000 7f41bda00640 1 Processor -- start 2024-03-24T00:14:55.715 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:55.714+0000 7f41bda00640 1 -- start start 2024-03-24T00:14:55.716 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:55.714+0000 7f41bda00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f41b8107cc0 0x7f41b819e010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:14:55.716 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:55.715+0000 7f41bda00640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f41b8108eb0 0x7f41b819e550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:14:55.716 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:55.715+0000 7f41bda00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f41b8109870 0x7f41b81a55c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:14:55.716 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:55.715+0000 7f41bda00640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f41b8113940 con 0x7f41b8108eb0 2024-03-24T00:14:55.716 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:55.715+0000 7f41bda00640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f41b81137c0 con 0x7f41b8109870 2024-03-24T00:14:55.716 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:55.715+0000 7f41bda00640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f41b8113ac0 con 0x7f41b8107cc0 2024-03-24T00:14:55.716 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:55.715+0000 7f41b6a00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f41b8107cc0 0x7f41b819e010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:14:55.716 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:55.715+0000 7f41ae000640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f41b8108eb0 0x7f41b819e550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:14:55.716 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:55.715+0000 7f41b6a00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f41b8107cc0 0x7f41b819e010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.118:42824/0 (socket says 172.21.15.118:42824) 2024-03-24T00:14:55.716 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:55.715+0000 7f41b6a00640 1 -- 172.21.15.118:0/2375320278 learned_addr learned my addr 172.21.15.118:0/2375320278 (peer_addr_for_me v2:172.21.15.118:0/0) 2024-03-24T00:14:55.716 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:55.715+0000 7f41b7400640 1 --2- 172.21.15.118:0/2375320278 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f41b8109870 0x7f41b81a55c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:14:55.716 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:55.715+0000 7f41b7400640 1 -- 172.21.15.118:0/2375320278 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f41b8107cc0 msgr2=0x7f41b819e010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:14:55.716 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:55.715+0000 7f41b7400640 1 --2- 172.21.15.118:0/2375320278 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f41b8107cc0 0x7f41b819e010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:14:55.717 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:55.715+0000 7f41b7400640 1 -- 172.21.15.118:0/2375320278 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f41b8108eb0 msgr2=0x7f41b819e550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:14:55.717 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:55.715+0000 7f41b7400640 1 --2- 172.21.15.118:0/2375320278 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f41b8108eb0 0x7f41b819e550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:14:55.717 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:55.715+0000 7f41b7400640 1 -- 172.21.15.118:0/2375320278 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f41b81a5c30 con 0x7f41b8109870 2024-03-24T00:14:55.717 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:55.715+0000 7f41b6a00640 1 --2- 172.21.15.118:0/2375320278 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f41b8107cc0 0x7f41b819e010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T00:14:55.717 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:55.716+0000 7f41b7400640 1 --2- 172.21.15.118:0/2375320278 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f41b8109870 0x7f41b81a55c0 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7f41b000d6b0 tx=0x7f41b000db70 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:14:55.717 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:55.716+0000 7f41afe00640 1 -- 172.21.15.118:0/2375320278 <== mon.2 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f41b0017070 con 0x7f41b8109870 2024-03-24T00:14:55.717 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:55.716+0000 7f41afe00640 1 -- 172.21.15.118:0/2375320278 <== mon.2 v2:172.21.15.189:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f41b0009d00 con 0x7f41b8109870 2024-03-24T00:14:55.717 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:55.716+0000 7f41bda00640 1 -- 172.21.15.118:0/2375320278 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f41b81a5f20 con 0x7f41b8109870 2024-03-24T00:14:55.717 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:55.716+0000 7f41afe00640 1 -- 172.21.15.118:0/2375320278 <== mon.2 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f41b0003d60 con 0x7f41b8109870 2024-03-24T00:14:55.717 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:55.716+0000 7f41bda00640 1 -- 172.21.15.118:0/2375320278 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f41b81a63e0 con 0x7f41b8109870 2024-03-24T00:14:55.718 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:55.719+0000 7f41afe00640 1 -- 172.21.15.118:0/2375320278 <== mon.2 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 18) v1 ==== 137291+0+0 (secure 0 0 0) 0x7f41b0009820 con 0x7f41b8109870 2024-03-24T00:14:55.720 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:55.720+0000 7f41afe00640 1 --2- 172.21.15.118:0/2375320278 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7f419c0a2880 0x7f419c0a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:14:55.720 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:55.720+0000 7f41afe00640 1 -- 172.21.15.118:0/2375320278 <== mon.2 v2:172.21.15.189:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 8248+0+0 (secure 0 0 0) 0x7f41b00ce250 con 0x7f41b8109870 2024-03-24T00:14:55.720 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:55.721+0000 7f41b6a00640 1 --2- 172.21.15.118:0/2375320278 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7f419c0a2880 0x7f419c0a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:14:55.720 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:55.721+0000 7f41c07a2740 1 Processor -- start 2024-03-24T00:14:55.721 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:55.721+0000 7f41c07a2740 1 -- start start 2024-03-24T00:14:55.721 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:55.721+0000 7f41c07a2740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55a1e40acba0 0x55a1e40acfc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:14:55.722 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:55.721+0000 7f41c07a2740 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55a1e40ad590 0x55a1e40b2090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:14:55.722 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:55.721+0000 7f41c07a2740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55a1e40b2700 0x55a1e40b4b90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:14:55.722 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:55.721+0000 7f41c07a2740 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a1e4011c90 con 0x55a1e40ad590 2024-03-24T00:14:55.722 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:55.721+0000 7f41c07a2740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a1e3fba980 con 0x55a1e40acba0 2024-03-24T00:14:55.722 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:55.721+0000 7f41c07a2740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a1e400dd30 con 0x55a1e40b2700 2024-03-24T00:14:55.722 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:55.721+0000 7f41b6a00640 1 --2- 172.21.15.118:0/2375320278 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7f419c0a2880 0x7f419c0a4d40 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7f41a800e8f0 tx=0x7f41a80095c0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:14:55.722 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:55.721+0000 7f41b6a00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55a1e40acba0 0x55a1e40acfc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:14:55.722 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:55.721+0000 7f41ae000640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55a1e40ad590 0x55a1e40b2090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:14:55.722 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:55.721+0000 7f41b6a00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55a1e40acba0 0x55a1e40acfc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.118:39164/0 (socket says 172.21.15.118:39164) 2024-03-24T00:14:55.722 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:55.721+0000 7f41b6a00640 1 -- 172.21.15.118:0/3361012442 learned_addr learned my addr 172.21.15.118:0/3361012442 (peer_addr_for_me v2:172.21.15.118:0/0) 2024-03-24T00:14:55.723 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:55.721+0000 7f41b7400640 1 --2- 172.21.15.118:0/3361012442 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55a1e40b2700 0x55a1e40b4b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:14:55.723 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:55.722+0000 7f41b6a00640 1 -- 172.21.15.118:0/3361012442 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55a1e40b2700 msgr2=0x55a1e40b4b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:14:55.723 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:55.722+0000 7f41b6a00640 1 --2- 172.21.15.118:0/3361012442 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55a1e40b2700 0x55a1e40b4b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:14:55.723 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:55.722+0000 7f41b6a00640 1 -- 172.21.15.118:0/3361012442 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55a1e40ad590 msgr2=0x55a1e40b2090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:14:55.723 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:55.722+0000 7f41b6a00640 1 --2- 172.21.15.118:0/3361012442 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55a1e40ad590 0x55a1e40b2090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:14:55.723 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:55.722+0000 7f41b6a00640 1 -- 172.21.15.118:0/3361012442 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55a1e3fd2510 con 0x55a1e40acba0 2024-03-24T00:14:55.723 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:55.722+0000 7f41b7400640 1 --2- 172.21.15.118:0/3361012442 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55a1e40b2700 0x55a1e40b4b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T00:14:55.723 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:55.722+0000 7f41b6a00640 1 --2- 172.21.15.118:0/3361012442 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55a1e40acba0 0x55a1e40acfc0 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7f41a800ccc0 tx=0x7f41a8015000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=971c56532e60152f server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:14:55.727 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:55.722+0000 7f41a7e00640 1 -- 172.21.15.118:0/3361012442 <== mon.2 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f41a802a070 con 0x55a1e40acba0 2024-03-24T00:14:55.727 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:55.722+0000 7f41a7e00640 1 -- 172.21.15.118:0/3361012442 <== mon.2 v2:172.21.15.189:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f41a801ce10 con 0x55a1e40acba0 2024-03-24T00:14:55.727 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:55.722+0000 7f41a7e00640 1 -- 172.21.15.118:0/3361012442 <== mon.2 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f41a8025af0 con 0x55a1e40acba0 2024-03-24T00:14:55.727 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:55.723+0000 7f41c07a2740 1 -- 172.21.15.118:0/3361012442 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55a1e40acba0 msgr2=0x55a1e40acfc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:14:55.727 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:55.723+0000 7f41c07a2740 1 --2- 172.21.15.118:0/3361012442 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55a1e40acba0 0x55a1e40acfc0 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7f41a800ccc0 tx=0x7f41a8015000 comp rx=0 tx=0).stop 2024-03-24T00:14:55.727 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:55.723+0000 7f41c07a2740 1 -- 172.21.15.118:0/3361012442 shutdown_connections 2024-03-24T00:14:55.727 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:55.723+0000 7f41c07a2740 1 --2- 172.21.15.118:0/3361012442 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55a1e40b2700 0x55a1e40b4b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:14:55.727 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:55.723+0000 7f41c07a2740 1 --2- 172.21.15.118:0/3361012442 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55a1e40ad590 0x55a1e40b2090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:14:55.727 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:55.723+0000 7f41c07a2740 1 --2- 172.21.15.118:0/3361012442 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55a1e40acba0 0x55a1e40acfc0 unknown :-1 s=CLOSED pgs=138 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:14:55.727 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:55.723+0000 7f41c07a2740 1 -- 172.21.15.118:0/3361012442 >> 172.21.15.118:0/3361012442 conn(0x55a1e40a8110 msgr2=0x55a1e40a8510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:14:55.727 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:55.724+0000 7f41c07a2740 1 -- 172.21.15.118:0/3361012442 shutdown_connections 2024-03-24T00:14:55.727 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:55.724+0000 7f41c07a2740 1 -- 172.21.15.118:0/3361012442 wait complete. 2024-03-24T00:14:55.728 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:55.724+0000 7f41c07a2740 1 -- start start 2024-03-24T00:14:55.728 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:55.724+0000 7f41c07a2740 1 Processor -- start 2024-03-24T00:14:55.728 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:55.725+0000 7f41c07a2740 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55a1e40acba0 0x55a1e40b7060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:14:55.728 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:55.725+0000 7f41c07a2740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55a1e40ad590 0x55a1e40b75a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:14:55.728 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:55.725+0000 7f41c07a2740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55a1e40b2700 0x55a1e40be5e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:14:55.728 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:55.725+0000 7f41c07a2740 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a1e40b56b0 con 0x55a1e40acba0 2024-03-24T00:14:55.729 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:55.725+0000 7f41c07a2740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a1e3fcddd0 con 0x55a1e40b2700 2024-03-24T00:14:55.729 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:55.725+0000 7f41c07a2740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a1e40b5830 con 0x55a1e40ad590 2024-03-24T00:14:55.729 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:55.725+0000 7f41c07a2740 10 client.0 _ll_register_callbacks cb 0x55a1e3d25ac0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T00:14:55.729 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:55.725+0000 7f41b6a00640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55a1e40acba0 0x55a1e40b7060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:14:55.729 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:55.725+0000 7f41ae000640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55a1e40ad590 0x55a1e40b75a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:14:55.729 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:55.725+0000 7f41b6a00640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55a1e40acba0 0x55a1e40b7060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.118:3300/0 says I am v2:172.21.15.118:56726/0 (socket says 172.21.15.118:56726) 2024-03-24T00:14:55.729 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:55.725+0000 7f41b6a00640 1 -- 172.21.15.118:0/1299857886 learned_addr learned my addr 172.21.15.118:0/1299857886 (peer_addr_for_me v2:172.21.15.118:0/0) 2024-03-24T00:14:55.729 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:55.725+0000 7f41b7400640 1 --2- 172.21.15.118:0/1299857886 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55a1e40b2700 0x55a1e40be5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:14:55.729 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:55.725+0000 7f41ae000640 1 -- 172.21.15.118:0/1299857886 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55a1e40b2700 msgr2=0x55a1e40be5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:14:55.729 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:55.725+0000 7f41ae000640 1 --2- 172.21.15.118:0/1299857886 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55a1e40b2700 0x55a1e40be5e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:14:55.730 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:55.725+0000 7f41ae000640 1 -- 172.21.15.118:0/1299857886 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55a1e40acba0 msgr2=0x55a1e40b7060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:14:55.730 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:55.725+0000 7f41ae000640 1 --2- 172.21.15.118:0/1299857886 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55a1e40acba0 0x55a1e40b7060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:14:55.730 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:55.725+0000 7f41ae000640 1 -- 172.21.15.118:0/1299857886 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55a1e3d262d0 con 0x55a1e40ad590 2024-03-24T00:14:55.730 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:55.725+0000 7f41b6a00640 1 --2- 172.21.15.118:0/1299857886 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55a1e40acba0 0x55a1e40b7060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T00:14:55.730 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:55.726+0000 7f41ae000640 1 --2- 172.21.15.118:0/1299857886 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55a1e40ad590 0x55a1e40b75a0 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7f41a002c0b0 tx=0x7f41a00324a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=f0ce8acc4cbe8dc2 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:14:55.730 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:55.726+0000 7f41a4c00640 10 client.0 ms_handle_connect on v2:172.21.15.193:3300/0 2024-03-24T00:14:55.730 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:55.726+0000 7f41a4c00640 1 -- 172.21.15.118:0/1299857886 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f41a0043070 con 0x55a1e40ad590 2024-03-24T00:14:55.730 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:55.726+0000 7f41a4c00640 1 -- 172.21.15.118:0/1299857886 <== mon.1 v2:172.21.15.193:3300/0 2 ==== osd_map(88..88 src has 1..88) v4 ==== 8248+0+0 (secure 0 0 0) 0x7f41a0004040 con 0x55a1e40ad590 2024-03-24T00:14:55.730 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:55.726+0000 7f41c07a2740 1 -- 172.21.15.118:0/1299857886 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f41a801ce10 con 0x55a1e40ad590 2024-03-24T00:14:55.730 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:55.726+0000 7f41a4c00640 1 -- 172.21.15.118:0/1299857886 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) v2 ==== 24+0+0 (secure 0 0 0) 0x7f41a004f050 con 0x55a1e40ad590 2024-03-24T00:14:55.730 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:55.726+0000 7f41c07a2740 10 client.24997 fetch_fsmap learned FSMap version 27 2024-03-24T00:14:55.730 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:55.726+0000 7f41c07a2740 1 -- 172.21.15.118:0/1299857886 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) v3 -- 0x55a1e3df06b0 con 0x55a1e40ad590 2024-03-24T00:14:55.730 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:55.727+0000 7f41a4c00640 1 -- 172.21.15.118:0/1299857886 <== mon.1 v2:172.21.15.193:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f41a801ce10 con 0x55a1e40ad590 2024-03-24T00:14:55.730 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:55.727+0000 7f41a4c00640 1 -- 172.21.15.118:0/1299857886 <== mon.1 v2:172.21.15.193:3300/0 5 ==== fsmap(e 27) v1 ==== 7934+0+0 (secure 0 0 0) 0x7f41a00326b0 con 0x55a1e40ad590 2024-03-24T00:14:55.731 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:55.727+0000 7f41c07a2740 10 client.24997 fetch_fsmap finished waiting for FSMap version 27 2024-03-24T00:14:55.731 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:55.727+0000 7f41c07a2740 10 client.24997 resolve_mds: resolved name 'f' to daemon mds.f 2024-03-24T00:14:55.731 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:55.727+0000 7f41c07a2740 20 client.24997 populate_metadata read hostname 'smithi118' 2024-03-24T00:14:55.731 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:55.727+0000 7f41c07a2740 1 --2- 172.21.15.118:0/1299857886 >> [v2:172.21.15.193:6834/2055310541,v1:172.21.15.193:6835/2055310541] conn(0x55a1e40c38c0 0x55a1e40c2920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:14:55.731 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:55.727+0000 7f41c07a2740 4 client.24997 mds_command: new command op to 24499 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-24T00:14:55.731 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:55.727+0000 7f41c07a2740 1 -- 172.21.15.118:0/1299857886 --> [v2:172.21.15.193:6834/2055310541,v1:172.21.15.193:6835/2055310541] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55a1e3fb3520 con 0x55a1e40c38c0 2024-03-24T00:14:55.731 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:55.727+0000 7f41b6a00640 1 --2- 172.21.15.118:0/1299857886 >> [v2:172.21.15.193:6834/2055310541,v1:172.21.15.193:6835/2055310541] conn(0x55a1e40c38c0 0x55a1e40c2920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:14:55.732 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:55.728+0000 7f41a4c00640 10 client.24997 ms_handle_connect on v2:172.21.15.193:6834/2055310541 2024-03-24T00:14:55.732 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:55.728+0000 7f41b6a00640 1 --2- 172.21.15.118:0/1299857886 >> [v2:172.21.15.193:6834/2055310541,v1:172.21.15.193:6835/2055310541] conn(0x55a1e40c38c0 0x55a1e40c2920 crc :-1 s=READY pgs=18 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=dac9d4710610ac93 server_cookie=886299e1e14178c5 in_seq=0 out_seq=0 2024-03-24T00:14:55.732 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:55.728+0000 7f41a4c00640 1 -- 172.21.15.118:0/1299857886 <== mds.3 v2:172.21.15.193:6834/2055310541 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x55a1e3fb3520 con 0x55a1e40c38c0 2024-03-24T00:14:55.732 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:55.728+0000 7f41a4c00640 10 client.24997 handle_command_reply: tid=0 2024-03-24T00:14:55.732 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:55.729+0000 7f41c07a2740 1 client.24997 shutdown 2024-03-24T00:14:55.732 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:55.729+0000 7f41c07a2740 1 -- 172.21.15.118:0/1299857886 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55a1e40ad590 msgr2=0x55a1e40b75a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:14:55.733 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:55.729+0000 7f41c07a2740 1 --2- 172.21.15.118:0/1299857886 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55a1e40ad590 0x55a1e40b75a0 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7f41a002c0b0 tx=0x7f41a00324a0 comp rx=0 tx=0).stop 2024-03-24T00:14:55.733 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:55.729+0000 7f41c07a2740 1 -- 172.21.15.118:0/1299857886 shutdown_connections 2024-03-24T00:14:55.733 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:55.729+0000 7f41c07a2740 1 --2- 172.21.15.118:0/1299857886 >> [v2:172.21.15.193:6834/2055310541,v1:172.21.15.193:6835/2055310541] conn(0x55a1e40c38c0 0x55a1e40c2920 crc :-1 s=READY pgs=18 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:14:55.733 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:55.729+0000 7f41c07a2740 1 --2- 172.21.15.118:0/1299857886 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55a1e40b2700 0x55a1e40be5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:14:55.733 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:55.729+0000 7f41a4c00640 0 client.24997 ms_handle_reset on v2:172.21.15.193:6834/2055310541 2024-03-24T00:14:55.733 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:55.729+0000 7f41c07a2740 1 --2- 172.21.15.118:0/1299857886 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55a1e40ad590 0x55a1e40b75a0 unknown :-1 s=CLOSED pgs=141 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:14:55.733 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:55.729+0000 7f41c07a2740 1 --2- 172.21.15.118:0/1299857886 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55a1e40acba0 0x55a1e40b7060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:14:55.733 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:55.729+0000 7f41c07a2740 1 -- 172.21.15.118:0/1299857886 >> 172.21.15.118:0/1299857886 conn(0x55a1e40a8110 msgr2=0x55a1e40a7310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:14:55.733 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:55.729+0000 7f41c07a2740 1 -- 172.21.15.118:0/1299857886 shutdown_connections 2024-03-24T00:14:55.733 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:55.729+0000 7f41c07a2740 1 -- 172.21.15.118:0/1299857886 wait complete. 2024-03-24T00:14:55.733 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:55.729+0000 7f41c07a2740 20 client.24997 trim_cache size 0 max 16384 2024-03-24T00:14:55.748 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:55.749+0000 7f41c07a2740 1 Processor -- start 2024-03-24T00:14:55.748 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:55.749+0000 7f41c07a2740 1 -- start start 2024-03-24T00:14:55.750 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:55.749+0000 7f41c07a2740 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55a1e40acba0 0x55a1e40a84f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:14:55.750 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:55.749+0000 7f41c07a2740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55a1e40ad590 0x55a1e40be5a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:14:55.750 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:55.749+0000 7f41c07a2740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55a1e40b2700 0x55a1e40c2920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:14:55.750 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:55.749+0000 7f41c07a2740 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a1e40c30e0 con 0x55a1e40acba0 2024-03-24T00:14:55.750 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:55.749+0000 7f41c07a2740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a1e40c3740 con 0x55a1e40ad590 2024-03-24T00:14:55.750 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:55.749+0000 7f41c07a2740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a1e40c3da0 con 0x55a1e40b2700 2024-03-24T00:14:55.750 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:55.750+0000 7f41b6a00640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55a1e40acba0 0x55a1e40a84f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:14:55.750 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:55.750+0000 7f41b6a00640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55a1e40acba0 0x55a1e40a84f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.118:3300/0 says I am v2:172.21.15.118:56740/0 (socket says 172.21.15.118:56740) 2024-03-24T00:14:55.750 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:55.750+0000 7f41b6a00640 1 -- 172.21.15.118:0/483512041 learned_addr learned my addr 172.21.15.118:0/483512041 (peer_addr_for_me v2:172.21.15.118:0/0) 2024-03-24T00:14:55.750 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:55.750+0000 7f41b7400640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55a1e40b2700 0x55a1e40c2920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:14:55.750 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:55.750+0000 7f41ae000640 1 --2- 172.21.15.118:0/483512041 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55a1e40ad590 0x55a1e40be5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:14:55.750 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:55.750+0000 7f41b6a00640 1 -- 172.21.15.118:0/483512041 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55a1e40b2700 msgr2=0x55a1e40c2920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:14:55.751 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:55.750+0000 7f41b6a00640 1 --2- 172.21.15.118:0/483512041 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55a1e40b2700 0x55a1e40c2920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:14:55.751 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:55.750+0000 7f41b6a00640 1 -- 172.21.15.118:0/483512041 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55a1e40ad590 msgr2=0x55a1e40be5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:14:55.751 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:55.750+0000 7f41b6a00640 1 --2- 172.21.15.118:0/483512041 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55a1e40ad590 0x55a1e40be5a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:14:55.751 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:55.750+0000 7f41b6a00640 1 -- 172.21.15.118:0/483512041 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55a1e40a7f40 con 0x55a1e40acba0 2024-03-24T00:14:55.751 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:55.750+0000 7f41ae000640 1 --2- 172.21.15.118:0/483512041 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55a1e40ad590 0x55a1e40be5a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T00:14:55.751 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:55.750+0000 7f41b6a00640 1 --2- 172.21.15.118:0/483512041 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55a1e40acba0 0x55a1e40a84f0 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7f41a800ccc0 tx=0x7f41a80254b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5f465612886202d2 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:14:55.753 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:55.751+0000 7f41a7400640 1 -- 172.21.15.118:0/483512041 <== mon.0 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f41a802a040 con 0x55a1e40acba0 2024-03-24T00:14:55.753 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:55.751+0000 7f41a7400640 1 -- 172.21.15.118:0/483512041 <== mon.0 v2:172.21.15.118:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f41a800c910 con 0x55a1e40acba0 2024-03-24T00:14:55.753 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:55.751+0000 7f41a7400640 1 -- 172.21.15.118:0/483512041 <== mon.0 v2:172.21.15.118:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f41a8025730 con 0x55a1e40acba0 2024-03-24T00:14:55.753 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:55.751+0000 7f41c07a2740 1 -- 172.21.15.118:0/483512041 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55a1e40acba0 msgr2=0x55a1e40a84f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:14:55.754 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:55.751+0000 7f41c07a2740 1 --2- 172.21.15.118:0/483512041 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55a1e40acba0 0x55a1e40a84f0 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7f41a800ccc0 tx=0x7f41a80254b0 comp rx=0 tx=0).stop 2024-03-24T00:14:55.754 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:55.752+0000 7f41c07a2740 1 -- 172.21.15.118:0/483512041 shutdown_connections 2024-03-24T00:14:55.754 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:55.752+0000 7f41c07a2740 1 --2- 172.21.15.118:0/483512041 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55a1e40b2700 0x55a1e40c2920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:14:55.754 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:55.752+0000 7f41c07a2740 1 --2- 172.21.15.118:0/483512041 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55a1e40ad590 0x55a1e40be5a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:14:55.754 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:55.752+0000 7f41c07a2740 1 --2- 172.21.15.118:0/483512041 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55a1e40acba0 0x55a1e40a84f0 unknown :-1 s=CLOSED pgs=170 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:14:55.754 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:55.752+0000 7f41c07a2740 1 -- 172.21.15.118:0/483512041 >> 172.21.15.118:0/483512041 conn(0x55a1e40a8110 msgr2=0x55a1e40b5c00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:14:55.754 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:55.752+0000 7f41c07a2740 1 -- 172.21.15.118:0/483512041 shutdown_connections 2024-03-24T00:14:55.754 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:55.752+0000 7f41c07a2740 1 -- 172.21.15.118:0/483512041 wait complete. 2024-03-24T00:14:55.754 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:55.752+0000 7f41c07a2740 1 -- start start 2024-03-24T00:14:55.754 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:55.752+0000 7f41c07a2740 1 Processor -- start 2024-03-24T00:14:55.754 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:55.753+0000 7f41c07a2740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55a1e40acba0 0x55a1e40ba7d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:14:55.754 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:55.753+0000 7f41c07a2740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55a1e40ad590 0x55a1e40bad10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:14:55.755 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:55.753+0000 7f41c07a2740 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55a1e40b2700 0x55a1e40bd8d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:14:55.755 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:55.753+0000 7f41c07a2740 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a1e40c4a60 con 0x55a1e40b2700 2024-03-24T00:14:55.755 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:55.753+0000 7f41c07a2740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a1e40c4400 con 0x55a1e40ad590 2024-03-24T00:14:55.755 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:55.753+0000 7f41c07a2740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a1e40c50c0 con 0x55a1e40acba0 2024-03-24T00:14:55.755 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:55.753+0000 7f41c07a2740 10 client.0 _ll_register_callbacks cb 0x7f41a00096b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T00:14:55.755 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:55.753+0000 7f41b6a00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55a1e40acba0 0x55a1e40ba7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:14:55.755 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:55.753+0000 7f41ae000640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55a1e40ad590 0x55a1e40bad10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:14:55.755 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:55.753+0000 7f41ae000640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55a1e40ad590 0x55a1e40bad10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.118:39186/0 (socket says 172.21.15.118:39186) 2024-03-24T00:14:55.755 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:55.753+0000 7f41b6a00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55a1e40acba0 0x55a1e40ba7d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.118:42852/0 (socket says 172.21.15.118:42852) 2024-03-24T00:14:55.755 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:55.753+0000 7f41ae000640 1 -- 172.21.15.118:0/1604076885 learned_addr learned my addr 172.21.15.118:0/1604076885 (peer_addr_for_me v2:172.21.15.118:0/0) 2024-03-24T00:14:55.755 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:55.753+0000 7f41b7400640 1 --2- 172.21.15.118:0/1604076885 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55a1e40b2700 0x55a1e40bd8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:14:55.755 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:55.753+0000 7f41ae000640 1 -- 172.21.15.118:0/1604076885 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55a1e40acba0 msgr2=0x55a1e40ba7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:14:55.755 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:55.753+0000 7f41ae000640 1 --2- 172.21.15.118:0/1604076885 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55a1e40acba0 0x55a1e40ba7d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:14:55.755 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:55.753+0000 7f41ae000640 1 -- 172.21.15.118:0/1604076885 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55a1e40b2700 msgr2=0x55a1e40bd8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:14:55.755 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:55.753+0000 7f41ae000640 1 --2- 172.21.15.118:0/1604076885 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55a1e40b2700 0x55a1e40bd8d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:14:55.756 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:55.753+0000 7f41ae000640 1 -- 172.21.15.118:0/1604076885 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55a1e40bdec0 con 0x55a1e40ad590 2024-03-24T00:14:55.756 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:55.753+0000 7f41b6a00640 1 --2- 172.21.15.118:0/1604076885 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55a1e40acba0 0x55a1e40ba7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T00:14:55.756 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:55.753+0000 7f41ae000640 1 --2- 172.21.15.118:0/1604076885 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55a1e40ad590 0x55a1e40bad10 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f41a0009780 tx=0x7f41a0005f10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=6675293e91fa98e8 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:14:55.756 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:55.753+0000 7f41a4c00640 10 client.0 ms_handle_connect on v2:172.21.15.189:3300/0 2024-03-24T00:14:55.756 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:55.754+0000 7f41a4c00640 1 -- 172.21.15.118:0/1604076885 <== mon.2 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f41a0043040 con 0x55a1e40ad590 2024-03-24T00:14:55.756 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:55.754+0000 7f41a4c00640 1 -- 172.21.15.118:0/1604076885 <== mon.2 v2:172.21.15.189:3300/0 2 ==== osd_map(88..88 src has 1..88) v4 ==== 8248+0+0 (secure 0 0 0) 0x7f41a0004030 con 0x55a1e40ad590 2024-03-24T00:14:55.756 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:55.754+0000 7f41c07a2740 1 -- 172.21.15.118:0/1604076885 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f41a800c910 con 0x55a1e40ad590 2024-03-24T00:14:55.756 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:55.754+0000 7f41a4c00640 1 -- 172.21.15.118:0/1604076885 <== mon.2 v2:172.21.15.189:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) v2 ==== 24+0+0 (secure 0 0 0) 0x7f41a0039040 con 0x55a1e40ad590 2024-03-24T00:14:55.756 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:55.754+0000 7f41c07a2740 10 client.24995 fetch_fsmap learned FSMap version 27 2024-03-24T00:14:55.756 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:55.754+0000 7f41c07a2740 1 -- 172.21.15.118:0/1604076885 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) v3 -- 0x55a1e411f0a0 con 0x55a1e40ad590 2024-03-24T00:14:55.758 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:55.754+0000 7f41a4c00640 1 -- 172.21.15.118:0/1604076885 <== mon.2 v2:172.21.15.189:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f41a800c910 con 0x55a1e40ad590 2024-03-24T00:14:55.758 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:55.754+0000 7f41a4c00640 1 -- 172.21.15.118:0/1604076885 <== mon.2 v2:172.21.15.189:3300/0 5 ==== fsmap(e 27) v1 ==== 7934+0+0 (secure 0 0 0) 0x7f41a00042e0 con 0x55a1e40ad590 2024-03-24T00:14:55.758 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:55.754+0000 7f41c07a2740 10 client.24995 fetch_fsmap finished waiting for FSMap version 27 2024-03-24T00:14:55.758 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:55.755+0000 7f41c07a2740 10 client.24995 resolve_mds: resolved name 'f' to daemon mds.f 2024-03-24T00:14:55.758 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:55.755+0000 7f41c07a2740 20 client.24995 populate_metadata read hostname 'smithi118' 2024-03-24T00:14:55.758 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:55.755+0000 7f41c07a2740 1 --2- 172.21.15.118:0/1604076885 >> [v2:172.21.15.193:6834/2055310541,v1:172.21.15.193:6835/2055310541] conn(0x55a1e4123690 0x55a1e411f2d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:14:55.759 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:55.755+0000 7f41c07a2740 4 client.24995 mds_command: new command op to 24499 tid=0 multi_id=0 [{"prefix": "status"}] 2024-03-24T00:14:55.759 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:55.755+0000 7f41c07a2740 1 -- 172.21.15.118:0/1604076885 --> [v2:172.21.15.193:6834/2055310541,v1:172.21.15.193:6835/2055310541] -- command(tid 0: {"prefix": "status"}) v1 -- 0x55a1e401f1d0 con 0x55a1e4123690 2024-03-24T00:14:55.759 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:55.755+0000 7f41b6a00640 1 --2- 172.21.15.118:0/1604076885 >> [v2:172.21.15.193:6834/2055310541,v1:172.21.15.193:6835/2055310541] conn(0x55a1e4123690 0x55a1e411f2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:14:55.759 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:55.755+0000 7f41b6a00640 1 --2- 172.21.15.118:0/1604076885 >> [v2:172.21.15.193:6834/2055310541,v1:172.21.15.193:6835/2055310541] conn(0x55a1e4123690 0x55a1e411f2d0 crc :-1 s=READY pgs=19 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=f99b9f59c98af030 server_cookie=1516ce19440ef728 in_seq=0 out_seq=0 2024-03-24T00:14:55.759 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:55.755+0000 7f41a4c00640 10 client.24995 ms_handle_connect on v2:172.21.15.193:6834/2055310541 2024-03-24T00:14:55.759 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:55.755+0000 7f41a4c00640 1 -- 172.21.15.118:0/1604076885 <== mds.3 v2:172.21.15.193:6834/2055310541 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+327 (crc 0 0 0) 0x55a1e401f1d0 con 0x55a1e4123690 2024-03-24T00:14:55.759 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:55.755+0000 7f41a4c00640 10 client.24995 handle_command_reply: tid=0 2024-03-24T00:14:55.759 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:55.756+0000 7f41c07a2740 1 client.24995 shutdown 2024-03-24T00:14:55.759 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:55.756+0000 7f41c07a2740 1 -- 172.21.15.118:0/1604076885 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55a1e40ad590 msgr2=0x55a1e40bad10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:14:55.759 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:55.756+0000 7f41c07a2740 1 --2- 172.21.15.118:0/1604076885 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55a1e40ad590 0x55a1e40bad10 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7f41a0009780 tx=0x7f41a0005f10 comp rx=0 tx=0).stop 2024-03-24T00:14:55.759 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:55.756+0000 7f41c07a2740 1 -- 172.21.15.118:0/1604076885 shutdown_connections 2024-03-24T00:14:55.759 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:55.756+0000 7f41c07a2740 1 --2- 172.21.15.118:0/1604076885 >> [v2:172.21.15.193:6834/2055310541,v1:172.21.15.193:6835/2055310541] conn(0x55a1e4123690 0x55a1e411f2d0 crc :-1 s=READY pgs=19 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:14:55.759 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:55.756+0000 7f41c07a2740 1 --2- 172.21.15.118:0/1604076885 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55a1e40b2700 0x55a1e40bd8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:14:55.759 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:55.756+0000 7f41c07a2740 1 --2- 172.21.15.118:0/1604076885 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55a1e40ad590 0x55a1e40bad10 unknown :-1 s=CLOSED pgs=139 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:14:55.760 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:55.756+0000 7f41c07a2740 1 --2- 172.21.15.118:0/1604076885 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55a1e40acba0 0x55a1e40ba7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:14:55.760 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:55.756+0000 7f41a4c00640 0 client.24995 ms_handle_reset on v2:172.21.15.193:6834/2055310541 2024-03-24T00:14:55.760 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:55.756+0000 7f41c07a2740 1 -- 172.21.15.118:0/1604076885 >> 172.21.15.118:0/1604076885 conn(0x55a1e40a8110 msgr2=0x55a1e40c2920 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:14:55.760 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:55.756+0000 7f41c07a2740 1 -- 172.21.15.118:0/1604076885 shutdown_connections 2024-03-24T00:14:55.760 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:55.756+0000 7f41c07a2740 1 -- 172.21.15.118:0/1604076885 wait complete. 2024-03-24T00:14:55.760 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:55.756+0000 7f41c07a2740 20 client.24995 trim_cache size 0 max 16384 2024-03-24T00:14:55.760 INFO:teuthology.orchestra.run.smithi118.stdout:{ 2024-03-24T00:14:55.760 INFO:teuthology.orchestra.run.smithi118.stdout: "cluster_fsid": "d79a5420-e971-11ee-95cd-87774f69a715", 2024-03-24T00:14:55.760 INFO:teuthology.orchestra.run.smithi118.stdout: "whoami": 3, 2024-03-24T00:14:55.760 INFO:teuthology.orchestra.run.smithi118.stdout: "id": 24499, 2024-03-24T00:14:55.760 INFO:teuthology.orchestra.run.smithi118.stdout: "want_state": "up:active", 2024-03-24T00:14:55.761 INFO:teuthology.orchestra.run.smithi118.stdout: "state": "up:active", 2024-03-24T00:14:55.761 INFO:teuthology.orchestra.run.smithi118.stdout: "fs_name": "cephfs", 2024-03-24T00:14:55.761 INFO:teuthology.orchestra.run.smithi118.stdout: "rank_uptime": 108.438338144, 2024-03-24T00:14:55.761 INFO:teuthology.orchestra.run.smithi118.stdout: "mdsmap_epoch": 27, 2024-03-24T00:14:55.761 INFO:teuthology.orchestra.run.smithi118.stdout: "osdmap_epoch": 88, 2024-03-24T00:14:55.761 INFO:teuthology.orchestra.run.smithi118.stdout: "osdmap_epoch_barrier": 86, 2024-03-24T00:14:55.761 INFO:teuthology.orchestra.run.smithi118.stdout: "uptime": 140.07967821700001 2024-03-24T00:14:55.761 INFO:teuthology.orchestra.run.smithi118.stdout:} 2024-03-24T00:14:55.762 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:55.757+0000 7f41bda00640 1 -- 172.21.15.118:0/2375320278 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7f419c0a2880 msgr2=0x7f419c0a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:14:55.762 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:55.757+0000 7f41bda00640 1 --2- 172.21.15.118:0/2375320278 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7f419c0a2880 0x7f419c0a4d40 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7f41a800e8f0 tx=0x7f41a80095c0 comp rx=0 tx=0).stop 2024-03-24T00:14:55.762 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:55.757+0000 7f41bda00640 1 -- 172.21.15.118:0/2375320278 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f41b8109870 msgr2=0x7f41b81a55c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:14:55.762 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:55.757+0000 7f41bda00640 1 --2- 172.21.15.118:0/2375320278 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f41b8109870 0x7f41b81a55c0 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7f41b000d6b0 tx=0x7f41b000db70 comp rx=0 tx=0).stop 2024-03-24T00:14:55.762 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:55.757+0000 7f41bda00640 1 -- 172.21.15.118:0/2375320278 shutdown_connections 2024-03-24T00:14:55.762 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:55.757+0000 7f41bda00640 1 --2- 172.21.15.118:0/2375320278 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7f419c0a2880 0x7f419c0a4d40 unknown :-1 s=CLOSED pgs=221 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:14:55.762 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:55.757+0000 7f41bda00640 1 --2- 172.21.15.118:0/2375320278 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f41b8109870 0x7f41b81a55c0 unknown :-1 s=CLOSED pgs=137 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:14:55.762 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:55.757+0000 7f41bda00640 1 --2- 172.21.15.118:0/2375320278 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f41b8108eb0 0x7f41b819e550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:14:55.763 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:55.757+0000 7f41bda00640 1 --2- 172.21.15.118:0/2375320278 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f41b8107cc0 0x7f41b819e010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:14:55.763 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:55.757+0000 7f41bda00640 1 -- 172.21.15.118:0/2375320278 >> 172.21.15.118:0/2375320278 conn(0x7f41b8076c60 msgr2=0x7f41b810c280 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:14:55.763 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:55.757+0000 7f41bda00640 1 -- 172.21.15.118:0/2375320278 shutdown_connections 2024-03-24T00:14:55.763 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:55.757+0000 7f41bda00640 1 -- 172.21.15.118:0/2375320278 wait complete. 2024-03-24T00:14:55.774 DEBUG:teuthology.orchestra.run.smithi118:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.c status 2024-03-24T00:14:55.993 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:55.994+0000 7f03d7600640 1 -- 172.21.15.118:0/1398322433 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f03d0102130 msgr2=0x7f03d0104520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:14:55.994 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:55.994+0000 7f03d7600640 1 --2- 172.21.15.118:0/1398322433 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f03d0102130 0x7f03d0104520 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7f03c0009900 tx=0x7f03c002bc40 comp rx=0 tx=0).stop 2024-03-24T00:14:55.994 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:55.994+0000 7f03d7600640 1 -- 172.21.15.118:0/1398322433 shutdown_connections 2024-03-24T00:14:55.994 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:55.994+0000 7f03d7600640 1 --2- 172.21.15.118:0/1398322433 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f03d0107390 0x7f03d01097e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:14:55.994 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:55.994+0000 7f03d7600640 1 --2- 172.21.15.118:0/1398322433 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f03d0104a60 0x7f03d0106e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:14:55.994 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:55.994+0000 7f03d7600640 1 --2- 172.21.15.118:0/1398322433 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f03d0102130 0x7f03d0104520 unknown :-1 s=CLOSED pgs=140 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:14:55.994 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:55.994+0000 7f03d7600640 1 -- 172.21.15.118:0/1398322433 >> 172.21.15.118:0/1398322433 conn(0x7f03d00fbb40 msgr2=0x7f03d00fdf60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:14:55.994 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:55.994+0000 7f03d7600640 1 -- 172.21.15.118:0/1398322433 shutdown_connections 2024-03-24T00:14:55.994 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:55.994+0000 7f03d7600640 1 -- 172.21.15.118:0/1398322433 wait complete. 2024-03-24T00:14:55.996 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:55.995+0000 7f03d7600640 1 Processor -- start 2024-03-24T00:14:55.997 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:55.995+0000 7f03d7600640 1 -- start start 2024-03-24T00:14:55.997 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:55.996+0000 7f03d7600640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f03d0102130 0x7f03d0199ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:14:55.997 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:55.996+0000 7f03d7600640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f03d0104a60 0x7f03d019a1e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:14:55.997 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:55.996+0000 7f03d7600640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f03d0107390 0x7f03d01a1240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:14:55.997 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:55.996+0000 7f03d7600640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f03d010d380 con 0x7f03d0107390 2024-03-24T00:14:55.997 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:55.996+0000 7f03d7600640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f03d010d200 con 0x7f03d0104a60 2024-03-24T00:14:55.997 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:55.996+0000 7f03d7600640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f03d010d500 con 0x7f03d0102130 2024-03-24T00:14:55.997 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:55.996+0000 7f03d4c00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f03d0102130 0x7f03d0199ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:14:55.997 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:55.996+0000 7f03c7e00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f03d0104a60 0x7f03d019a1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:14:55.997 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:55.996+0000 7f03d4c00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f03d0102130 0x7f03d0199ca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.118:42864/0 (socket says 172.21.15.118:42864) 2024-03-24T00:14:55.997 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:55.996+0000 7f03c7e00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f03d0104a60 0x7f03d019a1e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.118:39208/0 (socket says 172.21.15.118:39208) 2024-03-24T00:14:55.997 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:55.996+0000 7f03d4c00640 1 -- 172.21.15.118:0/4104920205 learned_addr learned my addr 172.21.15.118:0/4104920205 (peer_addr_for_me v2:172.21.15.118:0/0) 2024-03-24T00:14:55.997 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:55.996+0000 7f03d5600640 1 --2- 172.21.15.118:0/4104920205 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f03d0107390 0x7f03d01a1240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:14:55.998 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:55.996+0000 7f03d4c00640 1 -- 172.21.15.118:0/4104920205 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f03d0104a60 msgr2=0x7f03d019a1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:14:55.998 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:55.996+0000 7f03d4c00640 1 --2- 172.21.15.118:0/4104920205 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f03d0104a60 0x7f03d019a1e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:14:55.998 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:55.996+0000 7f03d4c00640 1 -- 172.21.15.118:0/4104920205 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f03d0107390 msgr2=0x7f03d01a1240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:14:55.998 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:55.996+0000 7f03d4c00640 1 --2- 172.21.15.118:0/4104920205 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f03d0107390 0x7f03d01a1240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:14:55.998 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:55.996+0000 7f03d4c00640 1 -- 172.21.15.118:0/4104920205 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f03d01a18e0 con 0x7f03d0102130 2024-03-24T00:14:55.998 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:55.997+0000 7f03d4c00640 1 --2- 172.21.15.118:0/4104920205 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f03d0102130 0x7f03d0199ca0 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7f03c002c8f0 tx=0x7f03c002c660 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:14:55.998 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:55.997+0000 7f03ce000640 1 -- 172.21.15.118:0/4104920205 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f03c0041070 con 0x7f03d0102130 2024-03-24T00:14:55.998 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:55.997+0000 7f03ce000640 1 -- 172.21.15.118:0/4104920205 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f03c0004450 con 0x7f03d0102130 2024-03-24T00:14:55.998 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:55.997+0000 7f03d7600640 1 -- 172.21.15.118:0/4104920205 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f03d01a1b70 con 0x7f03d0102130 2024-03-24T00:14:55.998 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:55.997+0000 7f03ce000640 1 -- 172.21.15.118:0/4104920205 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f03c0034440 con 0x7f03d0102130 2024-03-24T00:14:55.998 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:55.997+0000 7f03d7600640 1 -- 172.21.15.118:0/4104920205 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f03d01a2060 con 0x7f03d0102130 2024-03-24T00:14:55.999 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:56.000+0000 7f03ce000640 1 -- 172.21.15.118:0/4104920205 <== mon.1 v2:172.21.15.193:3300/0 4 ==== mgrmap(e 18) v1 ==== 137291+0+0 (secure 0 0 0) 0x7f03c0044070 con 0x7f03d0102130 2024-03-24T00:14:56.000 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:56.000+0000 7f03da38c740 1 Processor -- start 2024-03-24T00:14:56.001 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:56.000+0000 7f03da38c740 1 -- start start 2024-03-24T00:14:56.001 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:56.001+0000 7f03da38c740 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x56347f9abba0 0x56347f9abfc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:14:56.001 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:56.001+0000 7f03da38c740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56347f9ac590 0x56347f9b1090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:14:56.001 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:56.001+0000 7f03da38c740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56347f9b1700 0x56347f9b3b90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:14:56.001 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:56.001+0000 7f03da38c740 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x56347f910c90 con 0x56347f9abba0 2024-03-24T00:14:56.002 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:56.001+0000 7f03da38c740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x56347f8b9980 con 0x56347f9b1700 2024-03-24T00:14:56.002 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:56.001+0000 7f03da38c740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x56347f90cd30 con 0x56347f9ac590 2024-03-24T00:14:56.002 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:56.001+0000 7f03c7e00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56347f9b1700 0x56347f9b3b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:14:56.002 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:56.001+0000 7f03d4c00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56347f9ac590 0x56347f9b1090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:14:56.002 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:56.001+0000 7f03d5600640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x56347f9abba0 0x56347f9abfc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:14:56.002 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:56.001+0000 7f03c7e00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56347f9b1700 0x56347f9b3b90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.118:39214/0 (socket says 172.21.15.118:39214) 2024-03-24T00:14:56.002 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:56.001+0000 7f03c7e00640 1 -- 172.21.15.118:0/3581876523 learned_addr learned my addr 172.21.15.118:0/3581876523 (peer_addr_for_me v2:172.21.15.118:0/0) 2024-03-24T00:14:56.002 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:56.001+0000 7f03d5600640 1 -- 172.21.15.118:0/3581876523 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56347f9ac590 msgr2=0x56347f9b1090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:14:56.002 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:56.001+0000 7f03d5600640 1 --2- 172.21.15.118:0/3581876523 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56347f9ac590 0x56347f9b1090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:14:56.002 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:56.001+0000 7f03d5600640 1 -- 172.21.15.118:0/3581876523 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56347f9b1700 msgr2=0x56347f9b3b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:14:56.002 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:56.001+0000 7f03d5600640 1 --2- 172.21.15.118:0/3581876523 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56347f9b1700 0x56347f9b3b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:14:56.002 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:56.002+0000 7f03d5600640 1 -- 172.21.15.118:0/3581876523 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x56347f8d1510 con 0x56347f9abba0 2024-03-24T00:14:56.002 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:56.002+0000 7f03c7e00640 1 --2- 172.21.15.118:0/3581876523 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56347f9b1700 0x56347f9b3b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T00:14:56.003 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:56.002+0000 7f03ce000640 1 --2- 172.21.15.118:0/4104920205 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7f03b40a2800 0x7f03b40a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:14:56.003 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:56.002+0000 7f03d5600640 1 --2- 172.21.15.118:0/3581876523 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x56347f9abba0 0x56347f9abfc0 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7f03c8009f00 tx=0x7f03c8011810 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e1b8f3b3642ec469 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:14:56.003 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:56.002+0000 7f03ce000640 1 -- 172.21.15.118:0/4104920205 <== mon.1 v2:172.21.15.193:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 8248+0+0 (secure 0 0 0) 0x7f03c00f0640 con 0x7f03d0102130 2024-03-24T00:14:56.003 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:56.002+0000 7f03d4c00640 1 --2- 172.21.15.118:0/4104920205 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7f03b40a2800 0x7f03b40a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:14:56.006 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:56.002+0000 7f03c5c00640 1 -- 172.21.15.118:0/3581876523 <== mon.0 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f03c8004d20 con 0x56347f9abba0 2024-03-24T00:14:56.006 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:56.002+0000 7f03c5c00640 1 -- 172.21.15.118:0/3581876523 <== mon.0 v2:172.21.15.118:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f03c8004030 con 0x56347f9abba0 2024-03-24T00:14:56.006 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:56.002+0000 7f03c5c00640 1 -- 172.21.15.118:0/3581876523 <== mon.0 v2:172.21.15.118:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f03c8005280 con 0x56347f9abba0 2024-03-24T00:14:56.006 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:56.003+0000 7f03d4c00640 1 --2- 172.21.15.118:0/4104920205 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7f03b40a2800 0x7f03b40a4cc0 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7f03c00b9480 tx=0x7f03c00dd040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:14:56.006 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:56.003+0000 7f03da38c740 1 -- 172.21.15.118:0/3581876523 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x56347f9abba0 msgr2=0x56347f9abfc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:14:56.006 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:56.003+0000 7f03da38c740 1 --2- 172.21.15.118:0/3581876523 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x56347f9abba0 0x56347f9abfc0 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7f03c8009f00 tx=0x7f03c8011810 comp rx=0 tx=0).stop 2024-03-24T00:14:56.006 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:56.003+0000 7f03da38c740 1 -- 172.21.15.118:0/3581876523 shutdown_connections 2024-03-24T00:14:56.006 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:56.003+0000 7f03da38c740 1 --2- 172.21.15.118:0/3581876523 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56347f9b1700 0x56347f9b3b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:14:56.006 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:56.003+0000 7f03da38c740 1 --2- 172.21.15.118:0/3581876523 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56347f9ac590 0x56347f9b1090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:14:56.007 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:56.003+0000 7f03da38c740 1 --2- 172.21.15.118:0/3581876523 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x56347f9abba0 0x56347f9abfc0 unknown :-1 s=CLOSED pgs=171 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:14:56.007 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:56.003+0000 7f03da38c740 1 -- 172.21.15.118:0/3581876523 >> 172.21.15.118:0/3581876523 conn(0x56347f9a7110 msgr2=0x56347f9a7510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:14:56.007 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:56.003+0000 7f03da38c740 1 -- 172.21.15.118:0/3581876523 shutdown_connections 2024-03-24T00:14:56.007 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:56.003+0000 7f03da38c740 1 -- 172.21.15.118:0/3581876523 wait complete. 2024-03-24T00:14:56.007 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:56.004+0000 7f03da38c740 1 -- start start 2024-03-24T00:14:56.007 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:56.004+0000 7f03da38c740 1 Processor -- start 2024-03-24T00:14:56.007 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:56.004+0000 7f03da38c740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56347f9abba0 0x56347f9b6140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:14:56.007 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:56.004+0000 7f03da38c740 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x56347f9ac590 0x56347f9b6680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:14:56.007 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:56.004+0000 7f03da38c740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56347f9b1700 0x56347f9bd6c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:14:56.007 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:56.004+0000 7f03da38c740 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x56347f9b46b0 con 0x56347f9ac590 2024-03-24T00:14:56.007 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:56.004+0000 7f03da38c740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x56347f8ccdd0 con 0x56347f9b1700 2024-03-24T00:14:56.007 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:56.004+0000 7f03da38c740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x56347f9b4830 con 0x56347f9abba0 2024-03-24T00:14:56.007 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:56.004+0000 7f03da38c740 10 client.0 _ll_register_callbacks cb 0x56347f624ac0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T00:14:56.008 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:56.005+0000 7f03c7e00640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x56347f9ac590 0x56347f9b6680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:14:56.008 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:56.005+0000 7f03d5600640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56347f9abba0 0x56347f9b6140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:14:56.008 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:56.005+0000 7f03c7e00640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x56347f9ac590 0x56347f9b6680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.118:3300/0 says I am v2:172.21.15.118:56778/0 (socket says 172.21.15.118:56778) 2024-03-24T00:14:56.008 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:56.005+0000 7f03c7e00640 1 -- 172.21.15.118:0/4059088790 learned_addr learned my addr 172.21.15.118:0/4059088790 (peer_addr_for_me v2:172.21.15.118:0/0) 2024-03-24T00:14:56.008 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:56.005+0000 7f03d5600640 1 --2- 172.21.15.118:0/4059088790 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56347f9b1700 0x56347f9bd6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:14:56.008 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:56.005+0000 7f03c7e00640 1 -- 172.21.15.118:0/4059088790 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56347f9abba0 msgr2=0x56347f9b6140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:14:56.008 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:56.005+0000 7f03c7e00640 1 --2- 172.21.15.118:0/4059088790 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56347f9abba0 0x56347f9b6140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:14:56.008 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:56.005+0000 7f03c7e00640 1 -- 172.21.15.118:0/4059088790 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56347f9b1700 msgr2=0x56347f9bd6c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:14:56.008 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:56.005+0000 7f03c7e00640 1 --2- 172.21.15.118:0/4059088790 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56347f9b1700 0x56347f9bd6c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:14:56.008 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:56.005+0000 7f03c7e00640 1 -- 172.21.15.118:0/4059088790 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x56347f6252d0 con 0x56347f9ac590 2024-03-24T00:14:56.008 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:56.005+0000 7f03d5600640 1 --2- 172.21.15.118:0/4059088790 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56347f9abba0 0x56347f9b6140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T00:14:56.008 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:56.005+0000 7f03c7e00640 1 --2- 172.21.15.118:0/4059088790 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x56347f9ac590 0x56347f9b6680 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7f03b80126e0 tx=0x7f03b8012ba0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b7f12b8314fd66bc server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:14:56.008 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:56.005+0000 7f03be000640 10 client.0 ms_handle_connect on v2:172.21.15.118:3300/0 2024-03-24T00:14:56.008 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:56.006+0000 7f03be000640 1 -- 172.21.15.118:0/4059088790 <== mon.0 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f03b80121d0 con 0x56347f9ac590 2024-03-24T00:14:56.009 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:56.006+0000 7f03be000640 1 -- 172.21.15.118:0/4059088790 <== mon.0 v2:172.21.15.118:3300/0 2 ==== osd_map(88..88 src has 1..88) v4 ==== 8248+0+0 (secure 0 0 0) 0x7f03b8004050 con 0x56347f9ac590 2024-03-24T00:14:56.009 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:56.006+0000 7f03da38c740 1 -- 172.21.15.118:0/4059088790 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f03c8004030 con 0x56347f9ac590 2024-03-24T00:14:56.009 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:56.006+0000 7f03be000640 1 -- 172.21.15.118:0/4059088790 <== mon.0 v2:172.21.15.118:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) v2 ==== 24+0+0 (secure 0 0 0) 0x7f03b8004250 con 0x56347f9ac590 2024-03-24T00:14:56.009 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:56.006+0000 7f03da38c740 10 client.14916 fetch_fsmap learned FSMap version 27 2024-03-24T00:14:56.009 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:56.006+0000 7f03da38c740 1 -- 172.21.15.118:0/4059088790 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) v3 -- 0x56347f6ef6b0 con 0x56347f9ac590 2024-03-24T00:14:56.009 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:56.006+0000 7f03be000640 1 -- 172.21.15.118:0/4059088790 <== mon.0 v2:172.21.15.118:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f03c8004030 con 0x56347f9ac590 2024-03-24T00:14:56.011 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:56.007+0000 7f03be000640 1 -- 172.21.15.118:0/4059088790 <== mon.0 v2:172.21.15.118:3300/0 5 ==== fsmap(e 27) v1 ==== 7934+0+0 (secure 0 0 0) 0x7f03b8003a40 con 0x56347f9ac590 2024-03-24T00:14:56.011 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:56.007+0000 7f03da38c740 10 client.14916 fetch_fsmap finished waiting for FSMap version 27 2024-03-24T00:14:56.011 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:56.007+0000 7f03da38c740 10 client.14916 resolve_mds: resolved name 'c' to daemon mds.c 2024-03-24T00:14:56.011 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:56.007+0000 7f03da38c740 20 client.14916 populate_metadata read hostname 'smithi118' 2024-03-24T00:14:56.011 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:56.007+0000 7f03da38c740 1 --2- 172.21.15.118:0/4059088790 >> [v2:172.21.15.193:6836/2921520053,v1:172.21.15.193:6837/2921520053] conn(0x56347f9c2980 0x56347f9c1a60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:14:56.011 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:56.007+0000 7f03da38c740 4 client.14916 mds_command: new command op to 24511 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-24T00:14:56.011 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:56.007+0000 7f03da38c740 1 -- 172.21.15.118:0/4059088790 --> [v2:172.21.15.193:6836/2921520053,v1:172.21.15.193:6837/2921520053] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x56347f8b2520 con 0x56347f9c2980 2024-03-24T00:14:56.011 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:56.007+0000 7f03d5600640 1 --2- 172.21.15.118:0/4059088790 >> [v2:172.21.15.193:6836/2921520053,v1:172.21.15.193:6837/2921520053] conn(0x56347f9c2980 0x56347f9c1a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:14:56.011 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:56.007+0000 7f03d5600640 1 --2- 172.21.15.118:0/4059088790 >> [v2:172.21.15.193:6836/2921520053,v1:172.21.15.193:6837/2921520053] conn(0x56347f9c2980 0x56347f9c1a60 crc :-1 s=READY pgs=14 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=2f71e7b980819c16 server_cookie=51e96b3c1f5a0df6 in_seq=0 out_seq=0 2024-03-24T00:14:56.011 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:56.007+0000 7f03be000640 10 client.14916 ms_handle_connect on v2:172.21.15.193:6836/2921520053 2024-03-24T00:14:56.011 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:56.008+0000 7f03be000640 1 -- 172.21.15.118:0/4059088790 <== mds.2 v2:172.21.15.193:6836/2921520053 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x56347f8b2520 con 0x56347f9c2980 2024-03-24T00:14:56.012 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:56.008+0000 7f03be000640 10 client.14916 handle_command_reply: tid=0 2024-03-24T00:14:56.012 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:56.008+0000 7f03da38c740 1 client.14916 shutdown 2024-03-24T00:14:56.012 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:56.009+0000 7f03da38c740 1 -- 172.21.15.118:0/4059088790 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x56347f9ac590 msgr2=0x56347f9b6680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:14:56.012 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:56.009+0000 7f03da38c740 1 --2- 172.21.15.118:0/4059088790 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x56347f9ac590 0x56347f9b6680 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7f03b80126e0 tx=0x7f03b8012ba0 comp rx=0 tx=0).stop 2024-03-24T00:14:56.012 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:56.009+0000 7f03da38c740 1 -- 172.21.15.118:0/4059088790 shutdown_connections 2024-03-24T00:14:56.012 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:56.009+0000 7f03da38c740 1 --2- 172.21.15.118:0/4059088790 >> [v2:172.21.15.193:6836/2921520053,v1:172.21.15.193:6837/2921520053] conn(0x56347f9c2980 0x56347f9c1a60 crc :-1 s=READY pgs=14 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:14:56.012 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:56.009+0000 7f03da38c740 1 --2- 172.21.15.118:0/4059088790 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56347f9b1700 0x56347f9bd6c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:14:56.012 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:56.009+0000 7f03da38c740 1 --2- 172.21.15.118:0/4059088790 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x56347f9ac590 0x56347f9b6680 unknown :-1 s=CLOSED pgs=172 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:14:56.012 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:56.009+0000 7f03da38c740 1 --2- 172.21.15.118:0/4059088790 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56347f9abba0 0x56347f9b6140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:14:56.012 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:56.009+0000 7f03be000640 0 client.14916 ms_handle_reset on v2:172.21.15.193:6836/2921520053 2024-03-24T00:14:56.012 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:56.009+0000 7f03da38c740 1 -- 172.21.15.118:0/4059088790 >> 172.21.15.118:0/4059088790 conn(0x56347f9a7110 msgr2=0x56347f9a6310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:14:56.012 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:56.009+0000 7f03da38c740 1 -- 172.21.15.118:0/4059088790 shutdown_connections 2024-03-24T00:14:56.012 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:56.009+0000 7f03da38c740 1 -- 172.21.15.118:0/4059088790 wait complete. 2024-03-24T00:14:56.012 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:56.009+0000 7f03da38c740 20 client.14916 trim_cache size 0 max 16384 2024-03-24T00:14:56.027 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:56.027+0000 7f03da38c740 1 Processor -- start 2024-03-24T00:14:56.027 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:56.027+0000 7f03da38c740 1 -- start start 2024-03-24T00:14:56.027 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:56.027+0000 7f03da38c740 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x56347f9abba0 0x56347f9b4c00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:14:56.027 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:56.027+0000 7f03da38c740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56347f9ac590 0x56347f9bd680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:14:56.027 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:56.027+0000 7f03da38c740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56347f9b1700 0x56347f9a1e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:14:56.027 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:56.027+0000 7f03da38c740 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x56347f9c21a0 con 0x56347f9abba0 2024-03-24T00:14:56.027 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:56.027+0000 7f03da38c740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x56347f9c2800 con 0x56347f9b1700 2024-03-24T00:14:56.027 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:56.027+0000 7f03da38c740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x56347f9c2e60 con 0x56347f9ac590 2024-03-24T00:14:56.029 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:56.028+0000 7f03c7e00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56347f9ac590 0x56347f9bd680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:14:56.029 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:56.028+0000 7f03c7e00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56347f9ac590 0x56347f9bd680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.118:42886/0 (socket says 172.21.15.118:42886) 2024-03-24T00:14:56.029 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:56.028+0000 7f03c7e00640 1 -- 172.21.15.118:0/3692343883 learned_addr learned my addr 172.21.15.118:0/3692343883 (peer_addr_for_me v2:172.21.15.118:0/0) 2024-03-24T00:14:56.029 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:56.028+0000 7f03d5600640 1 --2- 172.21.15.118:0/3692343883 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x56347f9abba0 0x56347f9b4c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:14:56.029 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:56.028+0000 7f03c7e00640 1 -- 172.21.15.118:0/3692343883 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56347f9b1700 msgr2=0x56347f9a1e00 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-03-24T00:14:56.029 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:56.028+0000 7f03c7e00640 1 --2- 172.21.15.118:0/3692343883 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56347f9b1700 0x56347f9a1e00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:14:56.029 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:56.028+0000 7f03c7e00640 1 -- 172.21.15.118:0/3692343883 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x56347f9abba0 msgr2=0x56347f9b4c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:14:56.029 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:56.028+0000 7f03c7e00640 1 --2- 172.21.15.118:0/3692343883 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x56347f9abba0 0x56347f9b4c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:14:56.029 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:56.028+0000 7f03c7e00640 1 -- 172.21.15.118:0/3692343883 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x56347f9b7270 con 0x56347f9ac590 2024-03-24T00:14:56.029 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:56.028+0000 7f03d5600640 1 --2- 172.21.15.118:0/3692343883 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x56347f9abba0 0x56347f9b4c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-24T00:14:56.029 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:56.028+0000 7f03c7e00640 1 --2- 172.21.15.118:0/3692343883 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56347f9ac590 0x56347f9bd680 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7f03b8004210 tx=0x7f03b8009850 comp rx=0 tx=0).ready entity=mon.1 client_cookie=2731c051c9cec9c server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:14:56.030 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:56.028+0000 7f03c5200640 1 -- 172.21.15.118:0/3692343883 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f03b8012250 con 0x56347f9ac590 2024-03-24T00:14:56.030 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:56.029+0000 7f03c5200640 1 -- 172.21.15.118:0/3692343883 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f03b8003ad0 con 0x56347f9ac590 2024-03-24T00:14:56.030 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:56.029+0000 7f03c5200640 1 -- 172.21.15.118:0/3692343883 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f03b80107e0 con 0x56347f9ac590 2024-03-24T00:14:56.030 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:56.029+0000 7f03da38c740 1 -- 172.21.15.118:0/3692343883 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56347f9ac590 msgr2=0x56347f9bd680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:14:56.030 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:56.029+0000 7f03da38c740 1 --2- 172.21.15.118:0/3692343883 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56347f9ac590 0x56347f9bd680 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7f03b8004210 tx=0x7f03b8009850 comp rx=0 tx=0).stop 2024-03-24T00:14:56.030 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:56.029+0000 7f03da38c740 1 -- 172.21.15.118:0/3692343883 shutdown_connections 2024-03-24T00:14:56.030 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:56.029+0000 7f03da38c740 1 --2- 172.21.15.118:0/3692343883 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56347f9b1700 0x56347f9a1e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:14:56.030 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:56.029+0000 7f03da38c740 1 --2- 172.21.15.118:0/3692343883 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56347f9ac590 0x56347f9bd680 unknown :-1 s=CLOSED pgs=143 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:14:56.030 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:56.029+0000 7f03da38c740 1 --2- 172.21.15.118:0/3692343883 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x56347f9abba0 0x56347f9b4c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:14:56.030 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:56.029+0000 7f03da38c740 1 -- 172.21.15.118:0/3692343883 >> 172.21.15.118:0/3692343883 conn(0x56347f9a7110 msgr2=0x56347f9aac00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:14:56.030 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:56.029+0000 7f03da38c740 1 -- 172.21.15.118:0/3692343883 shutdown_connections 2024-03-24T00:14:56.031 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:56.029+0000 7f03da38c740 1 -- 172.21.15.118:0/3692343883 wait complete. 2024-03-24T00:14:56.034 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:56.030+0000 7f03da38c740 1 -- start start 2024-03-24T00:14:56.034 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:56.030+0000 7f03da38c740 1 Processor -- start 2024-03-24T00:14:56.034 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:56.030+0000 7f03da38c740 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x56347f9abba0 0x56347f9b9750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:14:56.034 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:56.030+0000 7f03da38c740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56347f9ac590 0x56347f9b9c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:14:56.034 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:56.030+0000 7f03da38c740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56347f9b1700 0x56347f9bccb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:14:56.034 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:56.030+0000 7f03da38c740 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x56347f9c3b20 con 0x56347f9abba0 2024-03-24T00:14:56.034 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:56.030+0000 7f03da38c740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x56347f9c34c0 con 0x56347f9b1700 2024-03-24T00:14:56.034 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:56.030+0000 7f03da38c740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x56347f9c4180 con 0x56347f9ac590 2024-03-24T00:14:56.035 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:56.030+0000 7f03da38c740 10 client.0 _ll_register_callbacks cb 0x7f03b8009930 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T00:14:56.035 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:56.030+0000 7f03c7e00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56347f9ac590 0x56347f9b9c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:14:56.035 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:56.031+0000 7f03c7e00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56347f9ac590 0x56347f9b9c90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.118:42888/0 (socket says 172.21.15.118:42888) 2024-03-24T00:14:56.035 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:56.031+0000 7f03d5600640 1 --2- 172.21.15.118:0/75151711 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x56347f9abba0 0x56347f9b9750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:14:56.035 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:56.031+0000 7f03c7e00640 1 -- 172.21.15.118:0/75151711 learned_addr learned my addr 172.21.15.118:0/75151711 (peer_addr_for_me v2:172.21.15.118:0/0) 2024-03-24T00:14:56.035 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:56.031+0000 7f03d5600640 1 --2- 172.21.15.118:0/75151711 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56347f9b1700 0x56347f9bccb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:14:56.035 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:56.031+0000 7f03c7e00640 1 -- 172.21.15.118:0/75151711 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56347f9b1700 msgr2=0x56347f9bccb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:14:56.035 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:56.031+0000 7f03c7e00640 1 --2- 172.21.15.118:0/75151711 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56347f9b1700 0x56347f9bccb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:14:56.035 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:56.031+0000 7f03c7e00640 1 -- 172.21.15.118:0/75151711 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x56347f9abba0 msgr2=0x56347f9b9750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:14:56.035 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:56.031+0000 7f03c7e00640 1 --2- 172.21.15.118:0/75151711 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x56347f9abba0 0x56347f9b9750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:14:56.035 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:56.031+0000 7f03c7e00640 1 -- 172.21.15.118:0/75151711 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x56347f9bd2a0 con 0x56347f9ac590 2024-03-24T00:14:56.035 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:56.031+0000 7f03c7e00640 1 --2- 172.21.15.118:0/75151711 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56347f9ac590 0x56347f9b9c90 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7f03b8003e30 tx=0x7f03b8005760 comp rx=0 tx=0).ready entity=mon.1 client_cookie=b274e2fce5f3d860 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:14:56.035 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:56.031+0000 7f03be000640 10 client.0 ms_handle_connect on v2:172.21.15.193:3300/0 2024-03-24T00:14:56.035 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:56.031+0000 7f03be000640 1 -- 172.21.15.118:0/75151711 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f03b8012250 con 0x56347f9ac590 2024-03-24T00:14:56.036 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:56.031+0000 7f03be000640 1 -- 172.21.15.118:0/75151711 <== mon.1 v2:172.21.15.193:3300/0 2 ==== osd_map(88..88 src has 1..88) v4 ==== 8248+0+0 (secure 0 0 0) 0x7f03b800f9a0 con 0x56347f9ac590 2024-03-24T00:14:56.036 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:56.031+0000 7f03da38c740 1 -- 172.21.15.118:0/75151711 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f03b8003ad0 con 0x56347f9ac590 2024-03-24T00:14:56.036 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:56.032+0000 7f03be000640 1 -- 172.21.15.118:0/75151711 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) v2 ==== 24+0+0 (secure 0 0 0) 0x7f03b800fc20 con 0x56347f9ac590 2024-03-24T00:14:56.036 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:56.032+0000 7f03da38c740 10 client.25021 fetch_fsmap learned FSMap version 27 2024-03-24T00:14:56.036 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:56.032+0000 7f03da38c740 1 -- 172.21.15.118:0/75151711 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) v3 -- 0x56347fa1e330 con 0x56347f9ac590 2024-03-24T00:14:56.036 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:56.032+0000 7f03be000640 1 -- 172.21.15.118:0/75151711 <== mon.1 v2:172.21.15.193:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f03b8003ad0 con 0x56347f9ac590 2024-03-24T00:14:56.036 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:56.032+0000 7f03be000640 1 -- 172.21.15.118:0/75151711 <== mon.1 v2:172.21.15.193:3300/0 5 ==== fsmap(e 27) v1 ==== 7934+0+0 (secure 0 0 0) 0x7f03b8010430 con 0x56347f9ac590 2024-03-24T00:14:56.036 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:56.032+0000 7f03da38c740 10 client.25021 fetch_fsmap finished waiting for FSMap version 27 2024-03-24T00:14:56.036 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:56.032+0000 7f03da38c740 10 client.25021 resolve_mds: resolved name 'c' to daemon mds.c 2024-03-24T00:14:56.036 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:56.032+0000 7f03da38c740 20 client.25021 populate_metadata read hostname 'smithi118' 2024-03-24T00:14:56.036 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:56.032+0000 7f03da38c740 1 --2- 172.21.15.118:0/75151711 >> [v2:172.21.15.193:6836/2921520053,v1:172.21.15.193:6837/2921520053] conn(0x56347fa22780 0x56347fa1e500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:14:56.036 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:56.032+0000 7f03da38c740 4 client.25021 mds_command: new command op to 24511 tid=0 multi_id=0 [{"prefix": "status"}] 2024-03-24T00:14:56.036 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:56.032+0000 7f03da38c740 1 -- 172.21.15.118:0/75151711 --> [v2:172.21.15.193:6836/2921520053,v1:172.21.15.193:6837/2921520053] -- command(tid 0: {"prefix": "status"}) v1 -- 0x56347f91e1d0 con 0x56347fa22780 2024-03-24T00:14:56.037 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:56.033+0000 7f03d5600640 1 --2- 172.21.15.118:0/75151711 >> [v2:172.21.15.193:6836/2921520053,v1:172.21.15.193:6837/2921520053] conn(0x56347fa22780 0x56347fa1e500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:14:56.037 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:56.033+0000 7f03d5600640 1 --2- 172.21.15.118:0/75151711 >> [v2:172.21.15.193:6836/2921520053,v1:172.21.15.193:6837/2921520053] conn(0x56347fa22780 0x56347fa1e500 crc :-1 s=READY pgs=15 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=2f144fd280b87df0 server_cookie=6a202a4da2f39801 in_seq=0 out_seq=0 2024-03-24T00:14:56.037 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:56.033+0000 7f03be000640 10 client.25021 ms_handle_connect on v2:172.21.15.193:6836/2921520053 2024-03-24T00:14:56.037 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:56.033+0000 7f03be000640 1 -- 172.21.15.118:0/75151711 <== mds.2 v2:172.21.15.193:6836/2921520053 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+327 (crc 0 0 0) 0x56347f91e1d0 con 0x56347fa22780 2024-03-24T00:14:56.037 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:56.033+0000 7f03be000640 10 client.25021 handle_command_reply: tid=0 2024-03-24T00:14:56.037 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:56.033+0000 7f03da38c740 1 client.25021 shutdown 2024-03-24T00:14:56.037 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:56.033+0000 7f03da38c740 1 -- 172.21.15.118:0/75151711 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56347f9ac590 msgr2=0x56347f9b9c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:14:56.037 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:56.033+0000 7f03da38c740 1 --2- 172.21.15.118:0/75151711 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56347f9ac590 0x56347f9b9c90 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7f03b8003e30 tx=0x7f03b8005760 comp rx=0 tx=0).stop 2024-03-24T00:14:56.037 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:56.033+0000 7f03da38c740 1 -- 172.21.15.118:0/75151711 shutdown_connections 2024-03-24T00:14:56.037 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:56.033+0000 7f03da38c740 1 --2- 172.21.15.118:0/75151711 >> [v2:172.21.15.193:6836/2921520053,v1:172.21.15.193:6837/2921520053] conn(0x56347fa22780 0x56347fa1e500 crc :-1 s=READY pgs=15 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:14:56.037 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:56.033+0000 7f03da38c740 1 --2- 172.21.15.118:0/75151711 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56347f9b1700 0x56347f9bccb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:14:56.037 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:56.033+0000 7f03be000640 0 client.25021 ms_handle_reset on v2:172.21.15.193:6836/2921520053 2024-03-24T00:14:56.037 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:56.033+0000 7f03da38c740 1 --2- 172.21.15.118:0/75151711 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56347f9ac590 0x56347f9b9c90 unknown :-1 s=CLOSED pgs=144 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:14:56.038 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:56.033+0000 7f03da38c740 1 --2- 172.21.15.118:0/75151711 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x56347f9abba0 0x56347f9b9750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:14:56.038 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:56.033+0000 7f03da38c740 1 -- 172.21.15.118:0/75151711 >> 172.21.15.118:0/75151711 conn(0x56347f9a7110 msgr2=0x56347f9aac00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:14:56.038 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:56.034+0000 7f03da38c740 1 -- 172.21.15.118:0/75151711 shutdown_connections 2024-03-24T00:14:56.038 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:56.034+0000 7f03da38c740 1 -- 172.21.15.118:0/75151711 wait complete. 2024-03-24T00:14:56.038 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:56.034+0000 7f03da38c740 20 client.25021 trim_cache size 0 max 16384 2024-03-24T00:14:56.038 INFO:teuthology.orchestra.run.smithi118.stdout:{ 2024-03-24T00:14:56.038 INFO:teuthology.orchestra.run.smithi118.stdout: "cluster_fsid": "d79a5420-e971-11ee-95cd-87774f69a715", 2024-03-24T00:14:56.038 INFO:teuthology.orchestra.run.smithi118.stdout: "whoami": 2, 2024-03-24T00:14:56.039 INFO:teuthology.orchestra.run.smithi118.stdout: "id": 24511, 2024-03-24T00:14:56.039 INFO:teuthology.orchestra.run.smithi118.stdout: "want_state": "up:active", 2024-03-24T00:14:56.039 INFO:teuthology.orchestra.run.smithi118.stdout: "state": "up:active", 2024-03-24T00:14:56.039 INFO:teuthology.orchestra.run.smithi118.stdout: "fs_name": "cephfs", 2024-03-24T00:14:56.039 INFO:teuthology.orchestra.run.smithi118.stdout: "rank_uptime": 109.749723222, 2024-03-24T00:14:56.039 INFO:teuthology.orchestra.run.smithi118.stdout: "mdsmap_epoch": 27, 2024-03-24T00:14:56.039 INFO:teuthology.orchestra.run.smithi118.stdout: "osdmap_epoch": 88, 2024-03-24T00:14:56.039 INFO:teuthology.orchestra.run.smithi118.stdout: "osdmap_epoch_barrier": 86, 2024-03-24T00:14:56.039 INFO:teuthology.orchestra.run.smithi118.stdout: "uptime": 137.37695621899999 2024-03-24T00:14:56.039 INFO:teuthology.orchestra.run.smithi118.stdout:} 2024-03-24T00:14:56.040 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:56.034+0000 7f03d7600640 1 -- 172.21.15.118:0/4104920205 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7f03b40a2800 msgr2=0x7f03b40a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:14:56.040 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:56.034+0000 7f03d7600640 1 --2- 172.21.15.118:0/4104920205 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7f03b40a2800 0x7f03b40a4cc0 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7f03c00b9480 tx=0x7f03c00dd040 comp rx=0 tx=0).stop 2024-03-24T00:14:56.040 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:56.034+0000 7f03d7600640 1 -- 172.21.15.118:0/4104920205 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f03d0102130 msgr2=0x7f03d0199ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:14:56.040 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:56.034+0000 7f03d7600640 1 --2- 172.21.15.118:0/4104920205 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f03d0102130 0x7f03d0199ca0 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7f03c002c8f0 tx=0x7f03c002c660 comp rx=0 tx=0).stop 2024-03-24T00:14:56.040 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:56.034+0000 7f03d7600640 1 -- 172.21.15.118:0/4104920205 shutdown_connections 2024-03-24T00:14:56.040 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:56.034+0000 7f03d7600640 1 --2- 172.21.15.118:0/4104920205 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7f03b40a2800 0x7f03b40a4cc0 unknown :-1 s=CLOSED pgs=222 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:14:56.040 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:56.034+0000 7f03d7600640 1 --2- 172.21.15.118:0/4104920205 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f03d0107390 0x7f03d01a1240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:14:56.040 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:56.034+0000 7f03d7600640 1 --2- 172.21.15.118:0/4104920205 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f03d0104a60 0x7f03d019a1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:14:56.040 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:56.034+0000 7f03d7600640 1 --2- 172.21.15.118:0/4104920205 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f03d0102130 0x7f03d0199ca0 unknown :-1 s=CLOSED pgs=142 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:14:56.040 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:56.034+0000 7f03d7600640 1 -- 172.21.15.118:0/4104920205 >> 172.21.15.118:0/4104920205 conn(0x7f03d00fbb40 msgr2=0x7f03d01018e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:14:56.040 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:56.035+0000 7f03d7600640 1 -- 172.21.15.118:0/4104920205 shutdown_connections 2024-03-24T00:14:56.041 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:56.035+0000 7f03d7600640 1 -- 172.21.15.118:0/4104920205 wait complete. 2024-03-24T00:14:56.048 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:14:55 smithi118 ceph-mon[31535]: from='client.? 172.21.15.118:0/1969623487' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-24T00:14:56.048 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:14:55 smithi118 ceph-mon[31535]: pgmap v373: 97 pgs: 97 active+clean; 645 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 10 op/s 2024-03-24T00:14:56.051 DEBUG:teuthology.orchestra.run.smithi118:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.g status 2024-03-24T00:14:56.203 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:14:55 smithi189 ceph-mon[36187]: from='client.? 172.21.15.118:0/1969623487' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-24T00:14:56.203 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:14:55 smithi189 ceph-mon[36187]: pgmap v373: 97 pgs: 97 active+clean; 645 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 10 op/s 2024-03-24T00:14:56.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:14:55 smithi193 ceph-mon[36878]: from='client.? 172.21.15.118:0/1969623487' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-24T00:14:56.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:14:55 smithi193 ceph-mon[36878]: pgmap v373: 97 pgs: 97 active+clean; 645 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 10 op/s 2024-03-24T00:14:56.248 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:56.249+0000 7fa80f400640 1 -- 172.21.15.118:0/729338412 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fa808102150 msgr2=0x7fa8081089e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:14:56.249 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:56.249+0000 7fa80f400640 1 --2- 172.21.15.118:0/729338412 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fa808102150 0x7fa8081089e0 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7fa800009ca0 tx=0x7fa80002c100 comp rx=0 tx=0).stop 2024-03-24T00:14:56.249 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:56.249+0000 7fa80f400640 1 -- 172.21.15.118:0/729338412 shutdown_connections 2024-03-24T00:14:56.249 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:56.249+0000 7fa80f400640 1 --2- 172.21.15.118:0/729338412 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fa808102150 0x7fa8081089e0 unknown :-1 s=CLOSED pgs=145 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:14:56.249 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:56.249+0000 7fa80f400640 1 --2- 172.21.15.118:0/729338412 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa808101790 0x7fa808101c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:14:56.249 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:56.249+0000 7fa80f400640 1 --2- 172.21.15.118:0/729338412 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fa8081005a0 0x7fa8081009c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:14:56.249 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:56.249+0000 7fa80f400640 1 -- 172.21.15.118:0/729338412 >> 172.21.15.118:0/729338412 conn(0x7fa8080fbb10 msgr2=0x7fa8080fdf50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:14:56.249 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:56.249+0000 7fa80f400640 1 -- 172.21.15.118:0/729338412 shutdown_connections 2024-03-24T00:14:56.249 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:56.249+0000 7fa80f400640 1 -- 172.21.15.118:0/729338412 wait complete. 2024-03-24T00:14:56.250 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:56.250+0000 7fa80f400640 1 Processor -- start 2024-03-24T00:14:56.250 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:56.251+0000 7fa80f400640 1 -- start start 2024-03-24T00:14:56.251 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:56.251+0000 7fa80f400640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa8081005a0 0x7fa808199be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:14:56.251 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:56.251+0000 7fa80f400640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fa808101790 0x7fa80819a120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:14:56.251 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:56.251+0000 7fa80f400640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fa808102150 0x7fa8081a1190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:14:56.251 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:56.251+0000 7fa80f400640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa80810c230 con 0x7fa808101790 2024-03-24T00:14:56.251 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:56.251+0000 7fa80f400640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa80810c0b0 con 0x7fa8081005a0 2024-03-24T00:14:56.251 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:56.251+0000 7fa80f400640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa80810c3b0 con 0x7fa808102150 2024-03-24T00:14:56.252 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:56.251+0000 7fa80ca00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa8081005a0 0x7fa808199be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:14:56.252 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:56.251+0000 7fa80ca00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa8081005a0 0x7fa808199be0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.118:39270/0 (socket says 172.21.15.118:39270) 2024-03-24T00:14:56.252 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:56.251+0000 7fa80ca00640 1 -- 172.21.15.118:0/2646603242 learned_addr learned my addr 172.21.15.118:0/2646603242 (peer_addr_for_me v2:172.21.15.118:0/0) 2024-03-24T00:14:56.252 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:56.251+0000 7fa80d400640 1 --2- 172.21.15.118:0/2646603242 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fa808102150 0x7fa8081a1190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:14:56.252 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:56.251+0000 7fa807e00640 1 --2- 172.21.15.118:0/2646603242 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fa808101790 0x7fa80819a120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:14:56.252 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:56.252+0000 7fa80ca00640 1 -- 172.21.15.118:0/2646603242 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fa808102150 msgr2=0x7fa8081a1190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:14:56.252 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:56.252+0000 7fa80ca00640 1 --2- 172.21.15.118:0/2646603242 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fa808102150 0x7fa8081a1190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:14:56.252 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:56.252+0000 7fa80ca00640 1 -- 172.21.15.118:0/2646603242 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fa808101790 msgr2=0x7fa80819a120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:14:56.252 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:56.252+0000 7fa80ca00640 1 --2- 172.21.15.118:0/2646603242 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fa808101790 0x7fa80819a120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:14:56.252 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:56.252+0000 7fa80ca00640 1 -- 172.21.15.118:0/2646603242 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa8081a1800 con 0x7fa8081005a0 2024-03-24T00:14:56.252 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:56.252+0000 7fa80d400640 1 --2- 172.21.15.118:0/2646603242 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fa808102150 0x7fa8081a1190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T00:14:56.252 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:56.252+0000 7fa80ca00640 1 --2- 172.21.15.118:0/2646603242 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa8081005a0 0x7fa808199be0 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7fa7fc00d770 tx=0x7fa7fc00dc30 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:14:56.253 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:56.252+0000 7fa805600640 1 -- 172.21.15.118:0/2646603242 <== mon.2 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa7fc017070 con 0x7fa8081005a0 2024-03-24T00:14:56.253 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:56.252+0000 7fa805600640 1 -- 172.21.15.118:0/2646603242 <== mon.2 v2:172.21.15.189:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fa7fc009c30 con 0x7fa8081005a0 2024-03-24T00:14:56.253 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:56.252+0000 7fa80f400640 1 -- 172.21.15.118:0/2646603242 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa8081a1af0 con 0x7fa8081005a0 2024-03-24T00:14:56.253 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:56.252+0000 7fa805600640 1 -- 172.21.15.118:0/2646603242 <== mon.2 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa7fc003dd0 con 0x7fa8081005a0 2024-03-24T00:14:56.253 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:56.252+0000 7fa80f400640 1 -- 172.21.15.118:0/2646603242 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa8081a2090 con 0x7fa8081005a0 2024-03-24T00:14:56.254 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:56.255+0000 7fa805600640 1 -- 172.21.15.118:0/2646603242 <== mon.2 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 18) v1 ==== 137291+0+0 (secure 0 0 0) 0x7fa7fc003fb0 con 0x7fa8081005a0 2024-03-24T00:14:56.258 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:56.255+0000 7fa812218740 1 Processor -- start 2024-03-24T00:14:56.259 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:56.256+0000 7fa812218740 1 -- start start 2024-03-24T00:14:56.259 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:56.256+0000 7fa812218740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56341dffbba0 0x56341dffbfc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:14:56.259 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:56.256+0000 7fa812218740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56341dffc590 0x56341e001090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:14:56.259 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:56.256+0000 7fa812218740 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x56341e001700 0x56341e003b90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:14:56.259 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:56.256+0000 7fa812218740 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x56341df60c90 con 0x56341e001700 2024-03-24T00:14:56.259 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:56.256+0000 7fa812218740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x56341df09980 con 0x56341dffbba0 2024-03-24T00:14:56.259 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:56.256+0000 7fa812218740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x56341df5cd30 con 0x56341dffc590 2024-03-24T00:14:56.259 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:56.256+0000 7fa80d400640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56341dffbba0 0x56341dffbfc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:14:56.260 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:56.256+0000 7fa80ca00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56341dffc590 0x56341e001090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:14:56.260 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:56.256+0000 7fa80d400640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56341dffbba0 0x56341dffbfc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.118:39272/0 (socket says 172.21.15.118:39272) 2024-03-24T00:14:56.260 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:56.256+0000 7fa80d400640 1 -- 172.21.15.118:0/3557900288 learned_addr learned my addr 172.21.15.118:0/3557900288 (peer_addr_for_me v2:172.21.15.118:0/0) 2024-03-24T00:14:56.260 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:56.256+0000 7fa807e00640 1 --2- 172.21.15.118:0/3557900288 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x56341e001700 0x56341e003b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:14:56.260 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:56.257+0000 7fa80d400640 1 -- 172.21.15.118:0/3557900288 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56341dffc590 msgr2=0x56341e001090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:14:56.260 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:56.257+0000 7fa80d400640 1 --2- 172.21.15.118:0/3557900288 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56341dffc590 0x56341e001090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:14:56.260 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:56.257+0000 7fa80d400640 1 -- 172.21.15.118:0/3557900288 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x56341e001700 msgr2=0x56341e003b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:14:56.260 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:56.257+0000 7fa80d400640 1 --2- 172.21.15.118:0/3557900288 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x56341e001700 0x56341e003b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:14:56.261 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:56.257+0000 7fa80d400640 1 -- 172.21.15.118:0/3557900288 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x56341df21510 con 0x56341dffbba0 2024-03-24T00:14:56.261 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:56.257+0000 7fa80d400640 1 --2- 172.21.15.118:0/3557900288 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56341dffbba0 0x56341dffbfc0 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7fa80002c710 tx=0x7fa8000323d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f02ac231cdbf80f1 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:14:56.261 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:56.257+0000 7fa805600640 1 --2- 172.21.15.118:0/2646603242 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7fa7f00a2880 0x7fa7f00a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:14:56.261 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:56.257+0000 7fa7f9c00640 1 -- 172.21.15.118:0/3557900288 <== mon.2 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa800043070 con 0x56341dffbba0 2024-03-24T00:14:56.261 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:56.257+0000 7fa80ca00640 1 --2- 172.21.15.118:0/2646603242 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7fa7f00a2880 0x7fa7f00a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:14:56.261 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:56.257+0000 7fa805600640 1 -- 172.21.15.118:0/2646603242 <== mon.2 v2:172.21.15.189:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 8248+0+0 (secure 0 0 0) 0x7fa7fc0ce2d0 con 0x7fa8081005a0 2024-03-24T00:14:56.261 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:56.257+0000 7fa7f9c00640 1 -- 172.21.15.118:0/3557900288 <== mon.2 v2:172.21.15.189:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fa800032c90 con 0x56341dffbba0 2024-03-24T00:14:56.261 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:56.257+0000 7fa7f9c00640 1 -- 172.21.15.118:0/3557900288 <== mon.2 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa800039040 con 0x56341dffbba0 2024-03-24T00:14:56.261 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:56.258+0000 7fa80ca00640 1 --2- 172.21.15.118:0/2646603242 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7fa7f00a2880 0x7fa7f00a4d40 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7fa7fc0cc130 tx=0x7fa7fc0d1db0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:14:56.262 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:56.258+0000 7fa812218740 1 -- 172.21.15.118:0/3557900288 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56341dffbba0 msgr2=0x56341dffbfc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:14:56.262 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:56.258+0000 7fa812218740 1 --2- 172.21.15.118:0/3557900288 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56341dffbba0 0x56341dffbfc0 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7fa80002c710 tx=0x7fa8000323d0 comp rx=0 tx=0).stop 2024-03-24T00:14:56.262 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:56.258+0000 7fa812218740 1 -- 172.21.15.118:0/3557900288 shutdown_connections 2024-03-24T00:14:56.262 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:56.258+0000 7fa812218740 1 --2- 172.21.15.118:0/3557900288 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x56341e001700 0x56341e003b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:14:56.262 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:56.258+0000 7fa812218740 1 --2- 172.21.15.118:0/3557900288 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56341dffc590 0x56341e001090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:14:56.262 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:56.258+0000 7fa812218740 1 --2- 172.21.15.118:0/3557900288 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56341dffbba0 0x56341dffbfc0 unknown :-1 s=CLOSED pgs=142 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:14:56.262 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:56.258+0000 7fa812218740 1 -- 172.21.15.118:0/3557900288 >> 172.21.15.118:0/3557900288 conn(0x56341dff7110 msgr2=0x56341dff7510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:14:56.262 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:56.258+0000 7fa812218740 1 -- 172.21.15.118:0/3557900288 shutdown_connections 2024-03-24T00:14:56.263 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:56.258+0000 7fa812218740 1 -- 172.21.15.118:0/3557900288 wait complete. 2024-03-24T00:14:56.263 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:56.259+0000 7fa812218740 1 -- start start 2024-03-24T00:14:56.267 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:56.259+0000 7fa812218740 1 Processor -- start 2024-03-24T00:14:56.267 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:56.260+0000 7fa812218740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56341dffbba0 0x56341e006050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:14:56.267 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:56.260+0000 7fa812218740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56341dffc590 0x56341e006590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:14:56.267 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:56.260+0000 7fa812218740 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x56341e001700 0x56341e00d5d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:14:56.267 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:56.260+0000 7fa812218740 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x56341e004650 con 0x56341e001700 2024-03-24T00:14:56.267 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:56.260+0000 7fa812218740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x56341df1cdd0 con 0x56341dffbba0 2024-03-24T00:14:56.267 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:56.260+0000 7fa812218740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x56341e0047d0 con 0x56341dffc590 2024-03-24T00:14:56.268 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:56.260+0000 7fa812218740 10 client.0 _ll_register_callbacks cb 0x56341dc74ac0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T00:14:56.268 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:56.260+0000 7fa80d400640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56341dffbba0 0x56341e006050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:14:56.268 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:56.260+0000 7fa80d400640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x56341e001700 0x56341e00d5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:14:56.268 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:56.260+0000 7fa807e00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56341dffc590 0x56341e006590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:14:56.268 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:56.260+0000 7fa80d400640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56341dffbba0 0x56341e006050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.118:39276/0 (socket says 172.21.15.118:39276) 2024-03-24T00:14:56.268 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:56.260+0000 7fa807e00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56341dffc590 0x56341e006590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.118:42908/0 (socket says 172.21.15.118:42908) 2024-03-24T00:14:56.268 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:56.260+0000 7fa80d400640 1 -- 172.21.15.118:0/922602412 learned_addr learned my addr 172.21.15.118:0/922602412 (peer_addr_for_me v2:172.21.15.118:0/0) 2024-03-24T00:14:56.268 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:56.261+0000 7fa807e00640 1 -- 172.21.15.118:0/922602412 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56341dffbba0 msgr2=0x56341e006050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:14:56.268 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:56.261+0000 7fa807e00640 1 --2- 172.21.15.118:0/922602412 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56341dffbba0 0x56341e006050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:14:56.268 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:56.261+0000 7fa807e00640 1 -- 172.21.15.118:0/922602412 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x56341e001700 msgr2=0x56341e00d5d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:14:56.268 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:56.261+0000 7fa807e00640 1 --2- 172.21.15.118:0/922602412 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x56341e001700 0x56341e00d5d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:14:56.268 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:56.261+0000 7fa807e00640 1 -- 172.21.15.118:0/922602412 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x56341dc752d0 con 0x56341dffc590 2024-03-24T00:14:56.268 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:56.261+0000 7fa807e00640 1 --2- 172.21.15.118:0/922602412 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56341dffc590 0x56341e006590 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7fa80819b0f0 tx=0x7fa7f4012b60 comp rx=0 tx=0).ready entity=mon.1 client_cookie=283421814c375ed3 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:14:56.268 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:56.261+0000 7fa7ee000640 10 client.0 ms_handle_connect on v2:172.21.15.193:3300/0 2024-03-24T00:14:56.268 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:56.261+0000 7fa7ee000640 1 -- 172.21.15.118:0/922602412 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa7f4004030 con 0x56341dffc590 2024-03-24T00:14:56.269 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:56.261+0000 7fa812218740 1 -- 172.21.15.118:0/922602412 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fa800032c90 con 0x56341dffc590 2024-03-24T00:14:56.269 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:56.261+0000 7fa7ee000640 1 -- 172.21.15.118:0/922602412 <== mon.1 v2:172.21.15.193:3300/0 2 ==== osd_map(88..88 src has 1..88) v4 ==== 8248+0+0 (secure 0 0 0) 0x7fa7f40041d0 con 0x56341dffc590 2024-03-24T00:14:56.269 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:56.261+0000 7fa7ee000640 1 -- 172.21.15.118:0/922602412 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) v2 ==== 24+0+0 (secure 0 0 0) 0x7fa7f40043d0 con 0x56341dffc590 2024-03-24T00:14:56.269 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:56.262+0000 7fa812218740 10 client.25033 fetch_fsmap learned FSMap version 27 2024-03-24T00:14:56.269 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:56.262+0000 7fa812218740 1 -- 172.21.15.118:0/922602412 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) v3 -- 0x56341dd3f6b0 con 0x56341dffc590 2024-03-24T00:14:56.269 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:56.262+0000 7fa7ee000640 1 -- 172.21.15.118:0/922602412 <== mon.1 v2:172.21.15.193:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fa800032c90 con 0x56341dffc590 2024-03-24T00:14:56.269 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:56.262+0000 7fa7ee000640 1 -- 172.21.15.118:0/922602412 <== mon.1 v2:172.21.15.193:3300/0 5 ==== fsmap(e 27) v1 ==== 7934+0+0 (secure 0 0 0) 0x7fa7f4003b90 con 0x56341dffc590 2024-03-24T00:14:56.269 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:56.262+0000 7fa812218740 10 client.25033 fetch_fsmap finished waiting for FSMap version 27 2024-03-24T00:14:56.269 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:56.262+0000 7fa812218740 10 client.25033 resolve_mds: resolved name 'g' to daemon mds.g 2024-03-24T00:14:56.269 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:56.262+0000 7fa812218740 20 client.25033 populate_metadata read hostname 'smithi118' 2024-03-24T00:14:56.269 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:56.262+0000 7fa812218740 1 --2- 172.21.15.118:0/922602412 >> [v2:172.21.15.118:6838/3285788503,v1:172.21.15.118:6839/3285788503] conn(0x56341e0128c0 0x56341e011970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:14:56.269 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:56.262+0000 7fa812218740 4 client.25033 mds_command: new command op to 24550 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-24T00:14:56.269 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:56.262+0000 7fa812218740 1 -- 172.21.15.118:0/922602412 --> [v2:172.21.15.118:6838/3285788503,v1:172.21.15.118:6839/3285788503] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x56341df02520 con 0x56341e0128c0 2024-03-24T00:14:56.269 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:56.262+0000 7fa80d400640 1 --2- 172.21.15.118:0/922602412 >> [v2:172.21.15.118:6838/3285788503,v1:172.21.15.118:6839/3285788503] conn(0x56341e0128c0 0x56341e011970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:14:56.269 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:56.263+0000 7fa80d400640 1 --2- 172.21.15.118:0/922602412 >> [v2:172.21.15.118:6838/3285788503,v1:172.21.15.118:6839/3285788503] conn(0x56341e0128c0 0x56341e011970 crc :-1 s=READY pgs=16 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=a9c5a80a38226c39 server_cookie=be0485703d99f41f in_seq=0 out_seq=0 2024-03-24T00:14:56.270 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:56.263+0000 7fa7ee000640 10 client.25033 ms_handle_connect on v2:172.21.15.118:6838/3285788503 2024-03-24T00:14:56.270 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:56.264+0000 7fa7ee000640 1 -- 172.21.15.118:0/922602412 <== mds.1 v2:172.21.15.118:6838/3285788503 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x56341df02520 con 0x56341e0128c0 2024-03-24T00:14:56.270 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:56.264+0000 7fa7ee000640 10 client.25033 handle_command_reply: tid=0 2024-03-24T00:14:56.270 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:56.264+0000 7fa812218740 1 client.25033 shutdown 2024-03-24T00:14:56.270 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:56.264+0000 7fa812218740 1 -- 172.21.15.118:0/922602412 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56341dffc590 msgr2=0x56341e006590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:14:56.270 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:56.264+0000 7fa812218740 1 --2- 172.21.15.118:0/922602412 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56341dffc590 0x56341e006590 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7fa80819b0f0 tx=0x7fa7f4012b60 comp rx=0 tx=0).stop 2024-03-24T00:14:56.270 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:56.264+0000 7fa812218740 1 -- 172.21.15.118:0/922602412 shutdown_connections 2024-03-24T00:14:56.270 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:56.264+0000 7fa812218740 1 --2- 172.21.15.118:0/922602412 >> [v2:172.21.15.118:6838/3285788503,v1:172.21.15.118:6839/3285788503] conn(0x56341e0128c0 0x56341e011970 crc :-1 s=READY pgs=16 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:14:56.270 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:56.264+0000 7fa812218740 1 --2- 172.21.15.118:0/922602412 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x56341e001700 0x56341e00d5d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:14:56.270 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:56.264+0000 7fa812218740 1 --2- 172.21.15.118:0/922602412 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56341dffc590 0x56341e006590 unknown :-1 s=CLOSED pgs=146 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:14:56.270 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:56.264+0000 7fa812218740 1 --2- 172.21.15.118:0/922602412 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56341dffbba0 0x56341e006050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:14:56.270 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:56.264+0000 7fa812218740 1 -- 172.21.15.118:0/922602412 >> 172.21.15.118:0/922602412 conn(0x56341dff7110 msgr2=0x56341dff6310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:14:56.270 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:56.264+0000 7fa7ee000640 0 client.25033 ms_handle_reset on v2:172.21.15.118:6838/3285788503 2024-03-24T00:14:56.271 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:56.264+0000 7fa812218740 1 -- 172.21.15.118:0/922602412 shutdown_connections 2024-03-24T00:14:56.271 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:56.264+0000 7fa812218740 1 -- 172.21.15.118:0/922602412 wait complete. 2024-03-24T00:14:56.271 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:56.264+0000 7fa812218740 20 client.25033 trim_cache size 0 max 16384 2024-03-24T00:14:56.289 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:56.290+0000 7fa812218740 1 Processor -- start 2024-03-24T00:14:56.289 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:56.290+0000 7fa812218740 1 -- start start 2024-03-24T00:14:56.291 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:56.290+0000 7fa812218740 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x56341dffbba0 0x56341dffaa80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:14:56.291 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:56.290+0000 7fa812218740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56341dffc590 0x56341e00d590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:14:56.291 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:56.290+0000 7fa812218740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56341e001700 0x56341e01ad60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:14:56.291 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:56.290+0000 7fa812218740 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x56341e0120e0 con 0x56341dffbba0 2024-03-24T00:14:56.291 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:56.290+0000 7fa812218740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x56341e012740 con 0x56341dffc590 2024-03-24T00:14:56.291 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:56.290+0000 7fa812218740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x56341e012da0 con 0x56341e001700 2024-03-24T00:14:56.291 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:56.290+0000 7fa807e00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56341dffc590 0x56341e00d590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:14:56.291 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:56.291+0000 7fa807e00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56341dffc590 0x56341e00d590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.118:39280/0 (socket says 172.21.15.118:39280) 2024-03-24T00:14:56.292 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:56.291+0000 7fa807e00640 1 -- 172.21.15.118:0/2622338608 learned_addr learned my addr 172.21.15.118:0/2622338608 (peer_addr_for_me v2:172.21.15.118:0/0) 2024-03-24T00:14:56.292 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:56.291+0000 7fa80d400640 1 --2- 172.21.15.118:0/2622338608 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x56341dffbba0 0x56341dffaa80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:14:56.292 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:56.291+0000 7fa80d400640 1 --2- 172.21.15.118:0/2622338608 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56341e001700 0x56341e01ad60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:14:56.292 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:56.291+0000 7fa807e00640 1 -- 172.21.15.118:0/2622338608 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56341e001700 msgr2=0x56341e01ad60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:14:56.292 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:56.291+0000 7fa807e00640 1 --2- 172.21.15.118:0/2622338608 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56341e001700 0x56341e01ad60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:14:56.292 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:56.291+0000 7fa807e00640 1 -- 172.21.15.118:0/2622338608 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x56341dffbba0 msgr2=0x56341dffaa80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:14:56.292 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:56.291+0000 7fa807e00640 1 --2- 172.21.15.118:0/2622338608 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x56341dffbba0 0x56341dffaa80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:14:56.292 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:56.291+0000 7fa807e00640 1 -- 172.21.15.118:0/2622338608 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x56341dff7e80 con 0x56341dffc590 2024-03-24T00:14:56.292 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:56.291+0000 7fa807e00640 1 --2- 172.21.15.118:0/2622338608 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56341dffc590 0x56341e00d590 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7fa7f4001eb0 tx=0x7fa7f4009d40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=397bc7e57512f71a server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:14:56.292 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:56.291+0000 7fa7f9200640 1 -- 172.21.15.118:0/2622338608 <== mon.2 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa7f40105f0 con 0x56341dffc590 2024-03-24T00:14:56.292 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:56.291+0000 7fa7f9200640 1 -- 172.21.15.118:0/2622338608 <== mon.2 v2:172.21.15.189:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fa7f4010e30 con 0x56341dffc590 2024-03-24T00:14:56.292 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:56.291+0000 7fa7f9200640 1 -- 172.21.15.118:0/2622338608 <== mon.2 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa7f4005620 con 0x56341dffc590 2024-03-24T00:14:56.297 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:56.292+0000 7fa812218740 1 -- 172.21.15.118:0/2622338608 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56341dffc590 msgr2=0x56341e00d590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:14:56.297 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:56.292+0000 7fa812218740 1 --2- 172.21.15.118:0/2622338608 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56341dffc590 0x56341e00d590 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7fa7f4001eb0 tx=0x7fa7f4009d40 comp rx=0 tx=0).stop 2024-03-24T00:14:56.297 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:56.292+0000 7fa812218740 1 -- 172.21.15.118:0/2622338608 shutdown_connections 2024-03-24T00:14:56.297 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:56.292+0000 7fa812218740 1 --2- 172.21.15.118:0/2622338608 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56341e001700 0x56341e01ad60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:14:56.297 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:56.292+0000 7fa812218740 1 --2- 172.21.15.118:0/2622338608 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56341dffc590 0x56341e00d590 unknown :-1 s=CLOSED pgs=143 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:14:56.297 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:56.292+0000 7fa812218740 1 --2- 172.21.15.118:0/2622338608 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x56341dffbba0 0x56341dffaa80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:14:56.297 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:56.292+0000 7fa812218740 1 -- 172.21.15.118:0/2622338608 >> 172.21.15.118:0/2622338608 conn(0x56341dff7110 msgr2=0x56341dffa370 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:14:56.297 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:56.292+0000 7fa812218740 1 -- 172.21.15.118:0/2622338608 shutdown_connections 2024-03-24T00:14:56.297 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:56.292+0000 7fa812218740 1 -- 172.21.15.118:0/2622338608 wait complete. 2024-03-24T00:14:56.297 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:56.292+0000 7fa812218740 1 -- start start 2024-03-24T00:14:56.297 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:56.293+0000 7fa812218740 1 Processor -- start 2024-03-24T00:14:56.297 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:56.293+0000 7fa812218740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56341dffbba0 0x56341e00a200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:14:56.298 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:56.293+0000 7fa812218740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56341dffc590 0x56341e00a740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:14:56.298 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:56.293+0000 7fa812218740 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x56341e001700 0x56341e06d920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:14:56.298 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:56.293+0000 7fa812218740 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x56341e013a60 con 0x56341e001700 2024-03-24T00:14:56.298 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:56.293+0000 7fa812218740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x56341e013400 con 0x56341dffbba0 2024-03-24T00:14:56.298 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:56.293+0000 7fa812218740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x56341e0140c0 con 0x56341dffc590 2024-03-24T00:14:56.298 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:56.293+0000 7fa812218740 10 client.0 _ll_register_callbacks cb 0x7fa7f4005a70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T00:14:56.298 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:56.293+0000 7fa807e00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56341dffc590 0x56341e00a740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:14:56.298 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:56.293+0000 7fa807e00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56341dffc590 0x56341e00a740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.118:42932/0 (socket says 172.21.15.118:42932) 2024-03-24T00:14:56.298 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:56.293+0000 7fa807e00640 1 -- 172.21.15.118:0/528843018 learned_addr learned my addr 172.21.15.118:0/528843018 (peer_addr_for_me v2:172.21.15.118:0/0) 2024-03-24T00:14:56.298 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:56.293+0000 7fa80d400640 1 --2- 172.21.15.118:0/528843018 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56341dffbba0 0x56341e00a200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:14:56.298 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:56.293+0000 7fa80d400640 1 --2- 172.21.15.118:0/528843018 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x56341e001700 0x56341e06d920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:14:56.298 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:56.293+0000 7fa807e00640 1 -- 172.21.15.118:0/528843018 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56341dffbba0 msgr2=0x56341e00a200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:14:56.299 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:56.294+0000 7fa807e00640 1 --2- 172.21.15.118:0/528843018 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56341dffbba0 0x56341e00a200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:14:56.299 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:56.294+0000 7fa807e00640 1 -- 172.21.15.118:0/528843018 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x56341e001700 msgr2=0x56341e06d920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:14:56.299 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:56.294+0000 7fa807e00640 1 --2- 172.21.15.118:0/528843018 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x56341e001700 0x56341e06d920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:14:56.299 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:56.294+0000 7fa807e00640 1 -- 172.21.15.118:0/528843018 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x56341e00d7c0 con 0x56341dffc590 2024-03-24T00:14:56.299 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:56.294+0000 7fa80d400640 1 --2- 172.21.15.118:0/528843018 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x56341e001700 0x56341e06d920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-24T00:14:56.299 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:56.294+0000 7fa807e00640 1 --2- 172.21.15.118:0/528843018 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56341dffc590 0x56341e00a740 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7fa7f40126a0 tx=0x7fa7f4005860 comp rx=0 tx=0).ready entity=mon.1 client_cookie=2eb9cdc6960303d server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:14:56.299 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:56.294+0000 7fa7ee000640 10 client.0 ms_handle_connect on v2:172.21.15.193:3300/0 2024-03-24T00:14:56.299 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:56.294+0000 7fa7ee000640 1 -- 172.21.15.118:0/528843018 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa7f4033020 con 0x56341dffc590 2024-03-24T00:14:56.299 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:56.294+0000 7fa812218740 1 -- 172.21.15.118:0/528843018 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fa7f4010e30 con 0x56341dffc590 2024-03-24T00:14:56.299 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:56.294+0000 7fa7ee000640 1 -- 172.21.15.118:0/528843018 <== mon.1 v2:172.21.15.193:3300/0 2 ==== osd_map(88..88 src has 1..88) v4 ==== 8248+0+0 (secure 0 0 0) 0x7fa7f4010b40 con 0x56341dffc590 2024-03-24T00:14:56.299 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:56.294+0000 7fa7ee000640 1 -- 172.21.15.118:0/528843018 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) v2 ==== 24+0+0 (secure 0 0 0) 0x7fa7f40041c0 con 0x56341dffc590 2024-03-24T00:14:56.299 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:56.294+0000 7fa812218740 10 client.25036 fetch_fsmap learned FSMap version 27 2024-03-24T00:14:56.299 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:56.294+0000 7fa812218740 1 -- 172.21.15.118:0/528843018 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) v3 -- 0x56341e06edd0 con 0x56341dffc590 2024-03-24T00:14:56.299 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:56.295+0000 7fa7ee000640 1 -- 172.21.15.118:0/528843018 <== mon.1 v2:172.21.15.193:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fa7f4010e30 con 0x56341dffc590 2024-03-24T00:14:56.300 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:56.295+0000 7fa7ee000640 1 -- 172.21.15.118:0/528843018 <== mon.1 v2:172.21.15.193:3300/0 5 ==== fsmap(e 27) v1 ==== 7934+0+0 (secure 0 0 0) 0x7fa7f4038030 con 0x56341dffc590 2024-03-24T00:14:56.300 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:56.295+0000 7fa812218740 10 client.25036 fetch_fsmap finished waiting for FSMap version 27 2024-03-24T00:14:56.300 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:56.295+0000 7fa812218740 10 client.25036 resolve_mds: resolved name 'g' to daemon mds.g 2024-03-24T00:14:56.300 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:56.295+0000 7fa812218740 20 client.25036 populate_metadata read hostname 'smithi118' 2024-03-24T00:14:56.300 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:56.295+0000 7fa812218740 1 --2- 172.21.15.118:0/528843018 >> [v2:172.21.15.118:6838/3285788503,v1:172.21.15.118:6839/3285788503] conn(0x56341e0732d0 0x56341e06f060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:14:56.300 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:56.295+0000 7fa812218740 4 client.25036 mds_command: new command op to 24550 tid=0 multi_id=0 [{"prefix": "status"}] 2024-03-24T00:14:56.300 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:56.295+0000 7fa812218740 1 -- 172.21.15.118:0/528843018 --> [v2:172.21.15.118:6838/3285788503,v1:172.21.15.118:6839/3285788503] -- command(tid 0: {"prefix": "status"}) v1 -- 0x56341df6e1d0 con 0x56341e0732d0 2024-03-24T00:14:56.300 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:56.295+0000 7fa80d400640 1 --2- 172.21.15.118:0/528843018 >> [v2:172.21.15.118:6838/3285788503,v1:172.21.15.118:6839/3285788503] conn(0x56341e0732d0 0x56341e06f060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:14:56.300 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:56.296+0000 7fa80d400640 1 --2- 172.21.15.118:0/528843018 >> [v2:172.21.15.118:6838/3285788503,v1:172.21.15.118:6839/3285788503] conn(0x56341e0732d0 0x56341e06f060 crc :-1 s=READY pgs=17 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=20cbd1acc736864d server_cookie=415af4e5ef9d8479 in_seq=0 out_seq=0 2024-03-24T00:14:56.300 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:56.296+0000 7fa7ee000640 10 client.25036 ms_handle_connect on v2:172.21.15.118:6838/3285788503 2024-03-24T00:14:56.300 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:56.296+0000 7fa7ee000640 1 -- 172.21.15.118:0/528843018 <== mds.1 v2:172.21.15.118:6838/3285788503 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+322 (crc 0 0 0) 0x56341df6e1d0 con 0x56341e0732d0 2024-03-24T00:14:56.300 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:56.296+0000 7fa7ee000640 10 client.25036 handle_command_reply: tid=0 2024-03-24T00:14:56.300 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:56.296+0000 7fa812218740 1 client.25036 shutdown 2024-03-24T00:14:56.301 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:56.296+0000 7fa812218740 1 -- 172.21.15.118:0/528843018 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56341dffc590 msgr2=0x56341e00a740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:14:56.301 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:56.296+0000 7fa812218740 1 --2- 172.21.15.118:0/528843018 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56341dffc590 0x56341e00a740 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7fa7f40126a0 tx=0x7fa7f4005860 comp rx=0 tx=0).stop 2024-03-24T00:14:56.301 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:56.296+0000 7fa812218740 1 -- 172.21.15.118:0/528843018 shutdown_connections 2024-03-24T00:14:56.301 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:56.296+0000 7fa812218740 1 --2- 172.21.15.118:0/528843018 >> [v2:172.21.15.118:6838/3285788503,v1:172.21.15.118:6839/3285788503] conn(0x56341e0732d0 0x56341e06f060 crc :-1 s=READY pgs=17 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:14:56.301 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:56.296+0000 7fa812218740 1 --2- 172.21.15.118:0/528843018 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x56341e001700 0x56341e06d920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:14:56.301 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:56.296+0000 7fa812218740 1 --2- 172.21.15.118:0/528843018 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56341dffc590 0x56341e00a740 unknown :-1 s=CLOSED pgs=147 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:14:56.301 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:56.296+0000 7fa7ee000640 0 client.25036 ms_handle_reset on v2:172.21.15.118:6838/3285788503 2024-03-24T00:14:56.301 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:56.296+0000 7fa812218740 1 --2- 172.21.15.118:0/528843018 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56341dffbba0 0x56341e00a200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:14:56.301 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:56.296+0000 7fa812218740 1 -- 172.21.15.118:0/528843018 >> 172.21.15.118:0/528843018 conn(0x56341dff7110 msgr2=0x56341dffac00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:14:56.301 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:56.296+0000 7fa812218740 1 -- 172.21.15.118:0/528843018 shutdown_connections 2024-03-24T00:14:56.301 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:56.296+0000 7fa812218740 1 -- 172.21.15.118:0/528843018 wait complete. 2024-03-24T00:14:56.301 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:56.297+0000 7fa812218740 20 client.25036 trim_cache size 0 max 16384 2024-03-24T00:14:56.302 INFO:teuthology.orchestra.run.smithi118.stdout:{ 2024-03-24T00:14:56.302 INFO:teuthology.orchestra.run.smithi118.stdout: "cluster_fsid": "d79a5420-e971-11ee-95cd-87774f69a715", 2024-03-24T00:14:56.302 INFO:teuthology.orchestra.run.smithi118.stdout: "whoami": 1, 2024-03-24T00:14:56.302 INFO:teuthology.orchestra.run.smithi118.stdout: "id": 24550, 2024-03-24T00:14:56.302 INFO:teuthology.orchestra.run.smithi118.stdout: "want_state": "up:active", 2024-03-24T00:14:56.302 INFO:teuthology.orchestra.run.smithi118.stdout: "state": "up:active", 2024-03-24T00:14:56.302 INFO:teuthology.orchestra.run.smithi118.stdout: "fs_name": "cephfs", 2024-03-24T00:14:56.302 INFO:teuthology.orchestra.run.smithi118.stdout: "rank_uptime": 110.912397814, 2024-03-24T00:14:56.302 INFO:teuthology.orchestra.run.smithi118.stdout: "mdsmap_epoch": 27, 2024-03-24T00:14:56.302 INFO:teuthology.orchestra.run.smithi118.stdout: "osdmap_epoch": 88, 2024-03-24T00:14:56.302 INFO:teuthology.orchestra.run.smithi118.stdout: "osdmap_epoch_barrier": 86, 2024-03-24T00:14:56.303 INFO:teuthology.orchestra.run.smithi118.stdout: "uptime": 127.954729427 2024-03-24T00:14:56.303 INFO:teuthology.orchestra.run.smithi118.stdout:} 2024-03-24T00:14:56.303 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:56.297+0000 7fa80f400640 1 -- 172.21.15.118:0/2646603242 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7fa7f00a2880 msgr2=0x7fa7f00a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:14:56.303 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:56.297+0000 7fa80f400640 1 --2- 172.21.15.118:0/2646603242 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7fa7f00a2880 0x7fa7f00a4d40 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7fa7fc0cc130 tx=0x7fa7fc0d1db0 comp rx=0 tx=0).stop 2024-03-24T00:14:56.303 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:56.297+0000 7fa80f400640 1 -- 172.21.15.118:0/2646603242 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa8081005a0 msgr2=0x7fa808199be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:14:56.303 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:56.297+0000 7fa80f400640 1 --2- 172.21.15.118:0/2646603242 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa8081005a0 0x7fa808199be0 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7fa7fc00d770 tx=0x7fa7fc00dc30 comp rx=0 tx=0).stop 2024-03-24T00:14:56.303 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:56.297+0000 7fa80f400640 1 -- 172.21.15.118:0/2646603242 shutdown_connections 2024-03-24T00:14:56.304 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:56.297+0000 7fa80f400640 1 --2- 172.21.15.118:0/2646603242 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7fa7f00a2880 0x7fa7f00a4d40 unknown :-1 s=CLOSED pgs=223 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:14:56.304 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:56.297+0000 7fa80f400640 1 --2- 172.21.15.118:0/2646603242 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fa808102150 0x7fa8081a1190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:14:56.304 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:56.297+0000 7fa80f400640 1 --2- 172.21.15.118:0/2646603242 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fa808101790 0x7fa80819a120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:14:56.304 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:56.297+0000 7fa80f400640 1 --2- 172.21.15.118:0/2646603242 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa8081005a0 0x7fa808199be0 unknown :-1 s=CLOSED pgs=141 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:14:56.304 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:56.297+0000 7fa80f400640 1 -- 172.21.15.118:0/2646603242 >> 172.21.15.118:0/2646603242 conn(0x7fa8080fbb10 msgr2=0x7fa8080fd5c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:14:56.304 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:56.297+0000 7fa80f400640 1 -- 172.21.15.118:0/2646603242 shutdown_connections 2024-03-24T00:14:56.304 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:56.297+0000 7fa80f400640 1 -- 172.21.15.118:0/2646603242 wait complete. 2024-03-24T00:14:56.315 DEBUG:teuthology.orchestra.run.smithi118:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.b status 2024-03-24T00:14:56.521 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:56.521+0000 7fc013200640 1 -- 172.21.15.118:0/1727620606 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fc014105350 msgr2=0x7fc01410bbe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:14:56.521 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:56.521+0000 7fc013200640 1 --2- 172.21.15.118:0/1727620606 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fc014105350 0x7fc01410bbe0 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7fc00c009ca0 tx=0x7fc00c02c100 comp rx=0 tx=0).stop 2024-03-24T00:14:56.521 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:56.521+0000 7fc013200640 1 -- 172.21.15.118:0/1727620606 shutdown_connections 2024-03-24T00:14:56.521 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:56.521+0000 7fc013200640 1 --2- 172.21.15.118:0/1727620606 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fc014105350 0x7fc01410bbe0 unknown :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:14:56.521 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:56.521+0000 7fc013200640 1 --2- 172.21.15.118:0/1727620606 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc014104990 0x7fc014104e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:14:56.522 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:56.521+0000 7fc013200640 1 --2- 172.21.15.118:0/1727620606 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fc0141037a0 0x7fc014103bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:14:56.522 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:56.521+0000 7fc013200640 1 -- 172.21.15.118:0/1727620606 >> 172.21.15.118:0/1727620606 conn(0x7fc0140fed10 msgr2=0x7fc014101150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:14:56.522 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:56.521+0000 7fc013200640 1 -- 172.21.15.118:0/1727620606 shutdown_connections 2024-03-24T00:14:56.522 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:56.522+0000 7fc013200640 1 -- 172.21.15.118:0/1727620606 wait complete. 2024-03-24T00:14:56.524 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:56.522+0000 7fc013200640 1 Processor -- start 2024-03-24T00:14:56.524 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:56.523+0000 7fc013200640 1 -- start start 2024-03-24T00:14:56.524 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:56.523+0000 7fc013200640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fc0141037a0 0x7fc0141999e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:14:56.524 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:56.523+0000 7fc013200640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fc014104990 0x7fc014199f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:14:56.524 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:56.523+0000 7fc013200640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc014105350 0x7fc0141a0f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:14:56.524 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:56.523+0000 7fc013200640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc01410f420 con 0x7fc0141037a0 2024-03-24T00:14:56.524 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:56.523+0000 7fc013200640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc01410f2a0 con 0x7fc014105350 2024-03-24T00:14:56.524 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:56.523+0000 7fc013200640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc01410f5a0 con 0x7fc014104990 2024-03-24T00:14:56.524 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:56.523+0000 7fc011e00640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fc0141037a0 0x7fc0141999e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:14:56.525 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:56.523+0000 7fc011400640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fc014104990 0x7fc014199f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:14:56.525 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:56.523+0000 7fc011e00640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fc0141037a0 0x7fc0141999e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.118:3300/0 says I am v2:172.21.15.118:56886/0 (socket says 172.21.15.118:56886) 2024-03-24T00:14:56.525 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:56.523+0000 7fc011400640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fc014104990 0x7fc014199f20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.118:42948/0 (socket says 172.21.15.118:42948) 2024-03-24T00:14:56.525 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:56.523+0000 7fc011e00640 1 -- 172.21.15.118:0/201383769 learned_addr learned my addr 172.21.15.118:0/201383769 (peer_addr_for_me v2:172.21.15.118:0/0) 2024-03-24T00:14:56.525 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:56.523+0000 7fc012800640 1 --2- 172.21.15.118:0/201383769 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc014105350 0x7fc0141a0f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:14:56.525 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:56.523+0000 7fc012800640 1 -- 172.21.15.118:0/201383769 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fc014104990 msgr2=0x7fc014199f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:14:56.525 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:56.523+0000 7fc012800640 1 --2- 172.21.15.118:0/201383769 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fc014104990 0x7fc014199f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:14:56.525 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:56.523+0000 7fc012800640 1 -- 172.21.15.118:0/201383769 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fc0141037a0 msgr2=0x7fc0141999e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:14:56.525 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:56.523+0000 7fc012800640 1 --2- 172.21.15.118:0/201383769 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fc0141037a0 0x7fc0141999e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:14:56.525 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:56.523+0000 7fc012800640 1 -- 172.21.15.118:0/201383769 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc0141a1600 con 0x7fc014105350 2024-03-24T00:14:56.525 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:56.524+0000 7fc011e00640 1 --2- 172.21.15.118:0/201383769 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fc0141037a0 0x7fc0141999e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T00:14:56.525 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:56.524+0000 7fc011400640 1 --2- 172.21.15.118:0/201383769 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fc014104990 0x7fc014199f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T00:14:56.525 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:56.524+0000 7fc012800640 1 --2- 172.21.15.118:0/201383769 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc014105350 0x7fc0141a0f90 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7fc00c037040 tx=0x7fc00c035d10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:14:56.525 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:56.524+0000 7fc00aa00640 1 -- 172.21.15.118:0/201383769 <== mon.2 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc00c039040 con 0x7fc014105350 2024-03-24T00:14:56.526 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:56.524+0000 7fc00aa00640 1 -- 172.21.15.118:0/201383769 <== mon.2 v2:172.21.15.189:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fc00c044070 con 0x7fc014105350 2024-03-24T00:14:56.526 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:56.524+0000 7fc013200640 1 -- 172.21.15.118:0/201383769 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc0141a1890 con 0x7fc014105350 2024-03-24T00:14:56.526 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:56.524+0000 7fc00aa00640 1 -- 172.21.15.118:0/201383769 <== mon.2 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc00c004350 con 0x7fc014105350 2024-03-24T00:14:56.526 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:56.524+0000 7fc013200640 1 -- 172.21.15.118:0/201383769 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc0141a1d50 con 0x7fc014105350 2024-03-24T00:14:56.528 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:56.528+0000 7fc01bdc4740 1 Processor -- start 2024-03-24T00:14:56.528 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:56.529+0000 7fc00aa00640 1 -- 172.21.15.118:0/201383769 <== mon.2 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 18) v1 ==== 137291+0+0 (secure 0 0 0) 0x7fc00c0044f0 con 0x7fc014105350 2024-03-24T00:14:56.528 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:56.529+0000 7fc01bdc4740 1 -- start start 2024-03-24T00:14:56.530 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:56.529+0000 7fc01bdc4740 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55babc766d80 0x55babc7671a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:14:56.530 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:56.529+0000 7fc01bdc4740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55babc767770 0x55babc76c270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:14:56.530 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:56.529+0000 7fc01bdc4740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55babc76c8e0 0x55babc76ed70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:14:56.530 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:56.529+0000 7fc01bdc4740 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x55babc6cbc90 con 0x55babc766d80 2024-03-24T00:14:56.530 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:56.529+0000 7fc01bdc4740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x55babc674980 con 0x55babc76c8e0 2024-03-24T00:14:56.530 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:56.529+0000 7fc01bdc4740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x55babc6c7d30 con 0x55babc767770 2024-03-24T00:14:56.530 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:56.529+0000 7fc011e00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55babc767770 0x55babc76c270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:14:56.530 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:56.529+0000 7fc011e00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55babc767770 0x55babc76c270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.118:42962/0 (socket says 172.21.15.118:42962) 2024-03-24T00:14:56.531 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:56.529+0000 7fc011e00640 1 -- 172.21.15.118:0/2944940547 learned_addr learned my addr 172.21.15.118:0/2944940547 (peer_addr_for_me v2:172.21.15.118:0/0) 2024-03-24T00:14:56.531 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:56.529+0000 7fc011400640 1 --2- 172.21.15.118:0/2944940547 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55babc766d80 0x55babc7671a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:14:56.531 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:56.530+0000 7fc011e00640 1 -- 172.21.15.118:0/2944940547 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55babc76c8e0 msgr2=0x55babc76ed70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:14:56.531 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:56.530+0000 7fc011400640 1 --2- 172.21.15.118:0/2944940547 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55babc76c8e0 0x55babc76ed70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:14:56.531 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:56.530+0000 7fc011e00640 1 --2- 172.21.15.118:0/2944940547 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55babc76c8e0 0x55babc76ed70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:14:56.531 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:56.530+0000 7fc011e00640 1 -- 172.21.15.118:0/2944940547 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55babc766d80 msgr2=0x55babc7671a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:14:56.531 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:56.530+0000 7fc011e00640 1 --2- 172.21.15.118:0/2944940547 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55babc766d80 0x55babc7671a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:14:56.531 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:56.530+0000 7fc011e00640 1 -- 172.21.15.118:0/2944940547 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55babc68c510 con 0x55babc767770 2024-03-24T00:14:56.531 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:56.530+0000 7fc011400640 1 --2- 172.21.15.118:0/2944940547 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55babc76c8e0 0x55babc76ed70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-24T00:14:56.531 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:56.530+0000 7fc011e00640 1 --2- 172.21.15.118:0/2944940547 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55babc767770 0x55babc76c270 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7fc0040115c0 tx=0x7fc004011a80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=f838760f24bfdfbb server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:14:56.531 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:56.530+0000 7fc00aa00640 1 --2- 172.21.15.118:0/201383769 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7fbff80a2800 0x7fbff80a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:14:56.531 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:56.530+0000 7fc00aa00640 1 -- 172.21.15.118:0/201383769 <== mon.2 v2:172.21.15.189:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 8248+0+0 (secure 0 0 0) 0x7fc00c0f0fe0 con 0x7fc014105350 2024-03-24T00:14:56.531 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:56.530+0000 7fbfff400640 1 -- 172.21.15.118:0/2944940547 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc004005020 con 0x55babc767770 2024-03-24T00:14:56.531 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:56.530+0000 7fbfff400640 1 -- 172.21.15.118:0/2944940547 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fc0040047a0 con 0x55babc767770 2024-03-24T00:14:56.531 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:56.530+0000 7fbfff400640 1 -- 172.21.15.118:0/2944940547 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc00400fa00 con 0x55babc767770 2024-03-24T00:14:56.532 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:56.530+0000 7fc011400640 1 --2- 172.21.15.118:0/201383769 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7fbff80a2800 0x7fbff80a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:14:56.535 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:56.531+0000 7fc01bdc4740 1 -- 172.21.15.118:0/2944940547 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55babc767770 msgr2=0x55babc76c270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:14:56.536 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:56.531+0000 7fc01bdc4740 1 --2- 172.21.15.118:0/2944940547 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55babc767770 0x55babc76c270 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7fc0040115c0 tx=0x7fc004011a80 comp rx=0 tx=0).stop 2024-03-24T00:14:56.536 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:56.531+0000 7fc011400640 1 --2- 172.21.15.118:0/201383769 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7fbff80a2800 0x7fbff80a4cc0 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x55babc767fd0 tx=0x7fc000015040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:14:56.536 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:56.531+0000 7fc01bdc4740 1 -- 172.21.15.118:0/2944940547 shutdown_connections 2024-03-24T00:14:56.536 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:56.531+0000 7fc01bdc4740 1 --2- 172.21.15.118:0/2944940547 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55babc76c8e0 0x55babc76ed70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:14:56.536 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:56.531+0000 7fc01bdc4740 1 --2- 172.21.15.118:0/2944940547 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55babc767770 0x55babc76c270 unknown :-1 s=CLOSED pgs=149 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:14:56.536 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:56.531+0000 7fc01bdc4740 1 --2- 172.21.15.118:0/2944940547 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55babc766d80 0x55babc7671a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:14:56.536 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:56.531+0000 7fc01bdc4740 1 -- 172.21.15.118:0/2944940547 >> 172.21.15.118:0/2944940547 conn(0x55babc7622f0 msgr2=0x55babc7626f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:14:56.536 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:56.531+0000 7fc01bdc4740 1 -- 172.21.15.118:0/2944940547 shutdown_connections 2024-03-24T00:14:56.536 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:56.531+0000 7fc01bdc4740 1 -- 172.21.15.118:0/2944940547 wait complete. 2024-03-24T00:14:56.536 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:56.532+0000 7fc01bdc4740 1 -- start start 2024-03-24T00:14:56.536 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:56.532+0000 7fc01bdc4740 1 Processor -- start 2024-03-24T00:14:56.536 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:56.532+0000 7fc01bdc4740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55babc766d80 0x55babc771290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:14:56.537 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:56.532+0000 7fc01bdc4740 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55babc767770 0x55babc7717d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:14:56.537 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:56.532+0000 7fc01bdc4740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55babc76c8e0 0x55babc778810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:14:56.537 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:56.532+0000 7fc01bdc4740 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x55babc76f890 con 0x55babc767770 2024-03-24T00:14:56.537 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:56.532+0000 7fc01bdc4740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x55babc687dd0 con 0x55babc766d80 2024-03-24T00:14:56.537 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:56.532+0000 7fc01bdc4740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x55babc76fa10 con 0x55babc76c8e0 2024-03-24T00:14:56.537 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:56.532+0000 7fc01bdc4740 10 client.0 _ll_register_callbacks cb 0x55babc3dfac0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T00:14:56.537 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:56.533+0000 7fc011e00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55babc766d80 0x55babc771290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:14:56.537 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:56.533+0000 7fc011400640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55babc767770 0x55babc7717d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:14:56.537 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:56.533+0000 7fc011e00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55babc766d80 0x55babc771290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.118:39332/0 (socket says 172.21.15.118:39332) 2024-03-24T00:14:56.537 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:56.533+0000 7fc011e00640 1 -- 172.21.15.118:0/3327512709 learned_addr learned my addr 172.21.15.118:0/3327512709 (peer_addr_for_me v2:172.21.15.118:0/0) 2024-03-24T00:14:56.537 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:56.533+0000 7fc012800640 1 --2- 172.21.15.118:0/3327512709 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55babc76c8e0 0x55babc778810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:14:56.537 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:56.533+0000 7fc011e00640 1 -- 172.21.15.118:0/3327512709 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55babc76c8e0 msgr2=0x55babc778810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:14:56.537 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:56.533+0000 7fc011e00640 1 --2- 172.21.15.118:0/3327512709 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55babc76c8e0 0x55babc778810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:14:56.538 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:56.533+0000 7fc011e00640 1 -- 172.21.15.118:0/3327512709 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55babc767770 msgr2=0x55babc7717d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:14:56.538 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:56.533+0000 7fc011e00640 1 --2- 172.21.15.118:0/3327512709 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55babc767770 0x55babc7717d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:14:56.538 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:56.533+0000 7fc011e00640 1 -- 172.21.15.118:0/3327512709 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55babc3e02d0 con 0x55babc766d80 2024-03-24T00:14:56.538 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:56.533+0000 7fc011400640 1 --2- 172.21.15.118:0/3327512709 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55babc767770 0x55babc7717d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T00:14:56.538 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:56.533+0000 7fc011e00640 1 --2- 172.21.15.118:0/3327512709 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55babc766d80 0x55babc771290 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7fc00401b040 tx=0x7fc004005230 comp rx=0 tx=0).ready entity=mon.2 client_cookie=8781c722146c4c5e server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:14:56.538 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:56.533+0000 7fbff3e00640 10 client.0 ms_handle_connect on v2:172.21.15.189:3300/0 2024-03-24T00:14:56.538 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:56.534+0000 7fbff3e00640 1 -- 172.21.15.118:0/3327512709 <== mon.2 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc004028020 con 0x55babc766d80 2024-03-24T00:14:56.538 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:56.534+0000 7fbff3e00640 1 -- 172.21.15.118:0/3327512709 <== mon.2 v2:172.21.15.189:3300/0 2 ==== osd_map(88..88 src has 1..88) v4 ==== 8248+0+0 (secure 0 0 0) 0x7fc0040053a0 con 0x55babc766d80 2024-03-24T00:14:56.538 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:56.534+0000 7fc01bdc4740 1 -- 172.21.15.118:0/3327512709 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fc0040047a0 con 0x55babc766d80 2024-03-24T00:14:56.538 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:56.534+0000 7fbff3e00640 1 -- 172.21.15.118:0/3327512709 <== mon.2 v2:172.21.15.189:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) v2 ==== 24+0+0 (secure 0 0 0) 0x7fc004033050 con 0x55babc766d80 2024-03-24T00:14:56.539 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:56.534+0000 7fc01bdc4740 10 client.25037 fetch_fsmap learned FSMap version 27 2024-03-24T00:14:56.539 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:56.534+0000 7fc01bdc4740 1 -- 172.21.15.118:0/3327512709 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) v3 -- 0x55babc4aa6b0 con 0x55babc766d80 2024-03-24T00:14:56.539 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:56.534+0000 7fbff3e00640 1 -- 172.21.15.118:0/3327512709 <== mon.2 v2:172.21.15.189:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fc0040047a0 con 0x55babc766d80 2024-03-24T00:14:56.539 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:56.535+0000 7fbff3e00640 1 -- 172.21.15.118:0/3327512709 <== mon.2 v2:172.21.15.189:3300/0 5 ==== fsmap(e 27) v1 ==== 7934+0+0 (secure 0 0 0) 0x7fc0040236b0 con 0x55babc766d80 2024-03-24T00:14:56.539 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:56.535+0000 7fc01bdc4740 10 client.25037 fetch_fsmap finished waiting for FSMap version 27 2024-03-24T00:14:56.539 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:56.535+0000 7fc01bdc4740 10 client.25037 resolve_mds: resolved name 'b' to daemon mds.b 2024-03-24T00:14:56.539 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:56.535+0000 7fc01bdc4740 20 client.25037 populate_metadata read hostname 'smithi118' 2024-03-24T00:14:56.539 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:56.535+0000 7fc01bdc4740 1 --2- 172.21.15.118:0/3327512709 >> [v2:172.21.15.189:6838/4074484168,v1:172.21.15.189:6839/4074484168] conn(0x55babc77d7a0 0x55babc77c9f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:14:56.539 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:56.535+0000 7fc01bdc4740 4 client.25037 mds_command: new command op to 24569 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-24T00:14:56.539 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:56.535+0000 7fc01bdc4740 1 -- 172.21.15.118:0/3327512709 --> [v2:172.21.15.189:6838/4074484168,v1:172.21.15.189:6839/4074484168] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55babc66d520 con 0x55babc77d7a0 2024-03-24T00:14:56.539 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:56.535+0000 7fc011400640 1 --2- 172.21.15.118:0/3327512709 >> [v2:172.21.15.189:6838/4074484168,v1:172.21.15.189:6839/4074484168] conn(0x55babc77d7a0 0x55babc77c9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:14:56.539 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:56.535+0000 7fbff3e00640 10 client.25037 ms_handle_connect on v2:172.21.15.189:6838/4074484168 2024-03-24T00:14:56.539 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:56.535+0000 7fc011400640 1 --2- 172.21.15.118:0/3327512709 >> [v2:172.21.15.189:6838/4074484168,v1:172.21.15.189:6839/4074484168] conn(0x55babc77d7a0 0x55babc77c9f0 crc :-1 s=READY pgs=27 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=ce06d6918dc9de61 server_cookie=761b287c89932ae1 in_seq=0 out_seq=0 2024-03-24T00:14:56.540 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:56.536+0000 7fbff3e00640 1 -- 172.21.15.118:0/3327512709 <== mds.0 v2:172.21.15.189:6838/4074484168 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x55babc66d520 con 0x55babc77d7a0 2024-03-24T00:14:56.540 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:56.536+0000 7fbff3e00640 10 client.25037 handle_command_reply: tid=0 2024-03-24T00:14:56.540 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:56.536+0000 7fc01bdc4740 1 client.25037 shutdown 2024-03-24T00:14:56.541 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:56.536+0000 7fc01bdc4740 1 -- 172.21.15.118:0/3327512709 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55babc766d80 msgr2=0x55babc771290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:14:56.541 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:56.536+0000 7fc01bdc4740 1 --2- 172.21.15.118:0/3327512709 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55babc766d80 0x55babc771290 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7fc00401b040 tx=0x7fc004005230 comp rx=0 tx=0).stop 2024-03-24T00:14:56.541 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:56.537+0000 7fc01bdc4740 1 -- 172.21.15.118:0/3327512709 shutdown_connections 2024-03-24T00:14:56.541 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:56.537+0000 7fc01bdc4740 1 --2- 172.21.15.118:0/3327512709 >> [v2:172.21.15.189:6838/4074484168,v1:172.21.15.189:6839/4074484168] conn(0x55babc77d7a0 0x55babc77c9f0 crc :-1 s=READY pgs=27 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:14:56.541 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:56.537+0000 7fc01bdc4740 1 --2- 172.21.15.118:0/3327512709 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55babc76c8e0 0x55babc778810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:14:56.541 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:56.537+0000 7fc01bdc4740 1 --2- 172.21.15.118:0/3327512709 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55babc767770 0x55babc7717d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:14:56.541 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:56.537+0000 7fbff3e00640 0 client.25037 ms_handle_reset on v2:172.21.15.189:6838/4074484168 2024-03-24T00:14:56.541 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:56.537+0000 7fc01bdc4740 1 --2- 172.21.15.118:0/3327512709 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55babc766d80 0x55babc771290 unknown :-1 s=CLOSED pgs=145 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:14:56.541 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:56.537+0000 7fc01bdc4740 1 -- 172.21.15.118:0/3327512709 >> 172.21.15.118:0/3327512709 conn(0x55babc7622f0 msgr2=0x55babc7614f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:14:56.541 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:56.537+0000 7fc01bdc4740 1 -- 172.21.15.118:0/3327512709 shutdown_connections 2024-03-24T00:14:56.541 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:56.537+0000 7fc01bdc4740 1 -- 172.21.15.118:0/3327512709 wait complete. 2024-03-24T00:14:56.541 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:56.537+0000 7fc01bdc4740 20 client.25037 trim_cache size 0 max 16384 2024-03-24T00:14:56.554 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:56.555+0000 7fc01bdc4740 1 Processor -- start 2024-03-24T00:14:56.555 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:56.555+0000 7fc01bdc4740 1 -- start start 2024-03-24T00:14:56.555 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:56.555+0000 7fc01bdc4740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55babc766d80 0x55babc77c9f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:14:56.555 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:56.555+0000 7fc01bdc4740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55babc767770 0x55babc7681e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:14:56.555 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:56.555+0000 7fc01bdc4740 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55babc76c8e0 0x55babc7626d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:14:56.555 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:56.555+0000 7fc01bdc4740 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x55babc77cfc0 con 0x55babc76c8e0 2024-03-24T00:14:56.555 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:56.555+0000 7fc01bdc4740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x55babc77d620 con 0x55babc767770 2024-03-24T00:14:56.555 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:56.555+0000 7fc01bdc4740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x55babc77dc80 con 0x55babc766d80 2024-03-24T00:14:56.557 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:56.555+0000 7fc011e00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55babc766d80 0x55babc77c9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:14:56.557 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:56.555+0000 7fc011400640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55babc767770 0x55babc7681e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:14:56.557 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:56.555+0000 7fc011400640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55babc767770 0x55babc7681e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.118:39348/0 (socket says 172.21.15.118:39348) 2024-03-24T00:14:56.557 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:56.555+0000 7fc011400640 1 -- 172.21.15.118:0/2503268981 learned_addr learned my addr 172.21.15.118:0/2503268981 (peer_addr_for_me v2:172.21.15.118:0/0) 2024-03-24T00:14:56.557 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:56.555+0000 7fc011e00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55babc766d80 0x55babc77c9f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.118:42984/0 (socket says 172.21.15.118:42984) 2024-03-24T00:14:56.557 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:56.556+0000 7fc012800640 1 --2- 172.21.15.118:0/2503268981 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55babc76c8e0 0x55babc7626d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:14:56.558 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:56.556+0000 7fc011400640 1 -- 172.21.15.118:0/2503268981 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55babc766d80 msgr2=0x55babc77c9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:14:56.558 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:56.556+0000 7fc011400640 1 --2- 172.21.15.118:0/2503268981 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55babc766d80 0x55babc77c9f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:14:56.558 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:56.556+0000 7fc011400640 1 -- 172.21.15.118:0/2503268981 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55babc76c8e0 msgr2=0x55babc7626d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:14:56.558 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:56.556+0000 7fc011400640 1 --2- 172.21.15.118:0/2503268981 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55babc76c8e0 0x55babc7626d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:14:56.558 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:56.556+0000 7fc011400640 1 -- 172.21.15.118:0/2503268981 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55babc767b50 con 0x55babc767770 2024-03-24T00:14:56.558 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:56.556+0000 7fc012800640 1 --2- 172.21.15.118:0/2503268981 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55babc76c8e0 0x55babc7626d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T00:14:56.558 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:56.556+0000 7fc011400640 1 --2- 172.21.15.118:0/2503268981 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55babc767770 0x55babc7681e0 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7fc01419aef0 tx=0x7fc000017000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b7d1f44d12b6cf00 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:14:56.558 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:56.556+0000 7fbffea00640 1 -- 172.21.15.118:0/2503268981 <== mon.2 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc00002c0d0 con 0x55babc767770 2024-03-24T00:14:56.558 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:56.556+0000 7fbffea00640 1 -- 172.21.15.118:0/2503268981 <== mon.2 v2:172.21.15.189:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fc00002c270 con 0x55babc767770 2024-03-24T00:14:56.558 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:56.556+0000 7fbffea00640 1 -- 172.21.15.118:0/2503268981 <== mon.2 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc00002c970 con 0x55babc767770 2024-03-24T00:14:56.558 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:56.557+0000 7fc01bdc4740 1 -- 172.21.15.118:0/2503268981 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55babc767770 msgr2=0x55babc7681e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:14:56.558 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:56.557+0000 7fc01bdc4740 1 --2- 172.21.15.118:0/2503268981 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55babc767770 0x55babc7681e0 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7fc01419aef0 tx=0x7fc000017000 comp rx=0 tx=0).stop 2024-03-24T00:14:56.559 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:56.557+0000 7fc01bdc4740 1 -- 172.21.15.118:0/2503268981 shutdown_connections 2024-03-24T00:14:56.559 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:56.557+0000 7fc01bdc4740 1 --2- 172.21.15.118:0/2503268981 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55babc76c8e0 0x55babc7626d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:14:56.559 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:56.557+0000 7fc01bdc4740 1 --2- 172.21.15.118:0/2503268981 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55babc767770 0x55babc7681e0 unknown :-1 s=CLOSED pgs=146 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:14:56.559 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:56.557+0000 7fc01bdc4740 1 --2- 172.21.15.118:0/2503268981 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55babc766d80 0x55babc77c9f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:14:56.559 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:56.557+0000 7fc01bdc4740 1 -- 172.21.15.118:0/2503268981 >> 172.21.15.118:0/2503268981 conn(0x55babc7622f0 msgr2=0x55babc76fde0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:14:56.559 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:56.557+0000 7fc01bdc4740 1 -- 172.21.15.118:0/2503268981 shutdown_connections 2024-03-24T00:14:56.559 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:56.557+0000 7fc01bdc4740 1 -- 172.21.15.118:0/2503268981 wait complete. 2024-03-24T00:14:56.559 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:56.558+0000 7fc01bdc4740 1 -- start start 2024-03-24T00:14:56.559 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:56.558+0000 7fc01bdc4740 1 Processor -- start 2024-03-24T00:14:56.562 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:56.558+0000 7fc01bdc4740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55babc766d80 0x55babc773cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:14:56.562 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:56.558+0000 7fc01bdc4740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55babc767770 0x55babc774200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:14:56.562 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:56.558+0000 7fc01bdc4740 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55babc76c8e0 0x55babc7771d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:14:56.563 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:56.558+0000 7fc01bdc4740 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x55babc77e940 con 0x55babc76c8e0 2024-03-24T00:14:56.563 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:56.558+0000 7fc01bdc4740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x55babc77e2e0 con 0x55babc767770 2024-03-24T00:14:56.563 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:56.558+0000 7fc01bdc4740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x55babc77efa0 con 0x55babc766d80 2024-03-24T00:14:56.563 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:56.558+0000 7fc01bdc4740 10 client.0 _ll_register_callbacks cb 0x7fc004011440 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T00:14:56.563 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:56.558+0000 7fc011e00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55babc766d80 0x55babc773cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:14:56.563 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:56.558+0000 7fc011400640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55babc767770 0x55babc774200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:14:56.563 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:56.558+0000 7fc011e00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55babc766d80 0x55babc773cc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.118:43000/0 (socket says 172.21.15.118:43000) 2024-03-24T00:14:56.563 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:56.558+0000 7fc011400640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55babc767770 0x55babc774200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.118:39352/0 (socket says 172.21.15.118:39352) 2024-03-24T00:14:56.563 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:56.558+0000 7fc011e00640 1 -- 172.21.15.118:0/1890867457 learned_addr learned my addr 172.21.15.118:0/1890867457 (peer_addr_for_me v2:172.21.15.118:0/0) 2024-03-24T00:14:56.563 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:56.558+0000 7fc012800640 1 --2- 172.21.15.118:0/1890867457 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55babc76c8e0 0x55babc7771d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:14:56.563 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:56.559+0000 7fc011e00640 1 -- 172.21.15.118:0/1890867457 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55babc767770 msgr2=0x55babc774200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:14:56.563 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:56.559+0000 7fc011e00640 1 --2- 172.21.15.118:0/1890867457 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55babc767770 0x55babc774200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:14:56.563 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:56.559+0000 7fc011e00640 1 -- 172.21.15.118:0/1890867457 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55babc76c8e0 msgr2=0x55babc7771d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:14:56.564 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:56.559+0000 7fc011e00640 1 --2- 172.21.15.118:0/1890867457 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55babc76c8e0 0x55babc7771d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:14:56.564 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:56.559+0000 7fc011e00640 1 -- 172.21.15.118:0/1890867457 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55babc7777c0 con 0x55babc766d80 2024-03-24T00:14:56.564 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:56.559+0000 7fc011400640 1 --2- 172.21.15.118:0/1890867457 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55babc767770 0x55babc774200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T00:14:56.564 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:56.559+0000 7fc011e00640 1 --2- 172.21.15.118:0/1890867457 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55babc766d80 0x55babc773cc0 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7fc004001eb0 tx=0x7fc0040114f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=882c258273f6f672 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:14:56.564 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:56.559+0000 7fbff3e00640 10 client.0 ms_handle_connect on v2:172.21.15.193:3300/0 2024-03-24T00:14:56.564 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:56.559+0000 7fbff3e00640 1 -- 172.21.15.118:0/1890867457 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc00403c060 con 0x55babc766d80 2024-03-24T00:14:56.564 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:56.559+0000 7fbff3e00640 1 -- 172.21.15.118:0/1890867457 <== mon.1 v2:172.21.15.193:3300/0 2 ==== osd_map(88..88 src has 1..88) v4 ==== 8248+0+0 (secure 0 0 0) 0x7fc0040052a0 con 0x55babc766d80 2024-03-24T00:14:56.564 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:56.559+0000 7fc01bdc4740 1 -- 172.21.15.118:0/1890867457 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fc00002c270 con 0x55babc766d80 2024-03-24T00:14:56.564 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:56.560+0000 7fbff3e00640 1 -- 172.21.15.118:0/1890867457 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) v2 ==== 24+0+0 (secure 0 0 0) 0x7fc004028040 con 0x55babc766d80 2024-03-24T00:14:56.564 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:56.560+0000 7fc01bdc4740 10 client.25054 fetch_fsmap learned FSMap version 27 2024-03-24T00:14:56.564 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:56.560+0000 7fc01bdc4740 1 -- 172.21.15.118:0/1890867457 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) v3 -- 0x55babc7d8e00 con 0x55babc766d80 2024-03-24T00:14:56.564 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:56.560+0000 7fbff3e00640 1 -- 172.21.15.118:0/1890867457 <== mon.1 v2:172.21.15.193:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fc00002c270 con 0x55babc766d80 2024-03-24T00:14:56.564 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:56.560+0000 7fbff3e00640 1 -- 172.21.15.118:0/1890867457 <== mon.1 v2:172.21.15.193:3300/0 5 ==== fsmap(e 27) v1 ==== 7934+0+0 (secure 0 0 0) 0x7fc004004450 con 0x55babc766d80 2024-03-24T00:14:56.564 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:56.560+0000 7fc01bdc4740 10 client.25054 fetch_fsmap finished waiting for FSMap version 27 2024-03-24T00:14:56.565 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:56.560+0000 7fc01bdc4740 10 client.25054 resolve_mds: resolved name 'b' to daemon mds.b 2024-03-24T00:14:56.565 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:56.560+0000 7fc01bdc4740 20 client.25054 populate_metadata read hostname 'smithi118' 2024-03-24T00:14:56.565 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:56.560+0000 7fc01bdc4740 1 --2- 172.21.15.118:0/1890867457 >> [v2:172.21.15.189:6838/4074484168,v1:172.21.15.189:6839/4074484168] conn(0x55babc7dd450 0x55babc7d9030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:14:56.565 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:56.560+0000 7fc01bdc4740 4 client.25054 mds_command: new command op to 24569 tid=0 multi_id=0 [{"prefix": "status"}] 2024-03-24T00:14:56.565 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:56.560+0000 7fc01bdc4740 1 -- 172.21.15.118:0/1890867457 --> [v2:172.21.15.189:6838/4074484168,v1:172.21.15.189:6839/4074484168] -- command(tid 0: {"prefix": "status"}) v1 -- 0x55babc6d91d0 con 0x55babc7dd450 2024-03-24T00:14:56.565 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:56.560+0000 7fc011400640 1 --2- 172.21.15.118:0/1890867457 >> [v2:172.21.15.189:6838/4074484168,v1:172.21.15.189:6839/4074484168] conn(0x55babc7dd450 0x55babc7d9030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:14:56.565 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:56.561+0000 7fc011400640 1 --2- 172.21.15.118:0/1890867457 >> [v2:172.21.15.189:6838/4074484168,v1:172.21.15.189:6839/4074484168] conn(0x55babc7dd450 0x55babc7d9030 crc :-1 s=READY pgs=28 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=64399ebef4e54f43 server_cookie=c8418050e79be851 in_seq=0 out_seq=0 2024-03-24T00:14:56.565 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:56.561+0000 7fbff3e00640 10 client.25054 ms_handle_connect on v2:172.21.15.189:6838/4074484168 2024-03-24T00:14:56.565 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:56.561+0000 7fbff3e00640 1 -- 172.21.15.118:0/1890867457 <== mds.0 v2:172.21.15.189:6838/4074484168 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+322 (crc 0 0 0) 0x55babc6d91d0 con 0x55babc7dd450 2024-03-24T00:14:56.565 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:56.561+0000 7fbff3e00640 10 client.25054 handle_command_reply: tid=0 2024-03-24T00:14:56.565 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:56.561+0000 7fc01bdc4740 1 client.25054 shutdown 2024-03-24T00:14:56.565 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:56.561+0000 7fc01bdc4740 1 -- 172.21.15.118:0/1890867457 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55babc766d80 msgr2=0x55babc773cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:14:56.565 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:56.561+0000 7fc01bdc4740 1 --2- 172.21.15.118:0/1890867457 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55babc766d80 0x55babc773cc0 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7fc004001eb0 tx=0x7fc0040114f0 comp rx=0 tx=0).stop 2024-03-24T00:14:56.566 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:56.561+0000 7fc01bdc4740 1 -- 172.21.15.118:0/1890867457 shutdown_connections 2024-03-24T00:14:56.566 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:56.561+0000 7fc01bdc4740 1 --2- 172.21.15.118:0/1890867457 >> [v2:172.21.15.189:6838/4074484168,v1:172.21.15.189:6839/4074484168] conn(0x55babc7dd450 0x55babc7d9030 crc :-1 s=READY pgs=28 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:14:56.566 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:56.561+0000 7fc01bdc4740 1 --2- 172.21.15.118:0/1890867457 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55babc76c8e0 0x55babc7771d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:14:56.566 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:56.561+0000 7fc01bdc4740 1 --2- 172.21.15.118:0/1890867457 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55babc767770 0x55babc774200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:14:56.566 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:56.561+0000 7fc01bdc4740 1 --2- 172.21.15.118:0/1890867457 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55babc766d80 0x55babc773cc0 unknown :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:14:56.566 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:56.561+0000 7fbff3e00640 0 client.25054 ms_handle_reset on v2:172.21.15.189:6838/4074484168 2024-03-24T00:14:56.566 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:56.561+0000 7fc01bdc4740 1 -- 172.21.15.118:0/1890867457 >> 172.21.15.118:0/1890867457 conn(0x55babc7622f0 msgr2=0x55babc768170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:14:56.566 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:56.562+0000 7fc01bdc4740 1 -- 172.21.15.118:0/1890867457 shutdown_connections 2024-03-24T00:14:56.566 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:56.562+0000 7fc01bdc4740 1 -- 172.21.15.118:0/1890867457 wait complete. 2024-03-24T00:14:56.566 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:56.562+0000 7fc01bdc4740 20 client.25054 trim_cache size 0 max 16384 2024-03-24T00:14:56.567 INFO:teuthology.orchestra.run.smithi118.stdout:{ 2024-03-24T00:14:56.567 INFO:teuthology.orchestra.run.smithi118.stdout: "cluster_fsid": "d79a5420-e971-11ee-95cd-87774f69a715", 2024-03-24T00:14:56.567 INFO:teuthology.orchestra.run.smithi118.stdout: "whoami": 0, 2024-03-24T00:14:56.567 INFO:teuthology.orchestra.run.smithi118.stdout: "id": 24569, 2024-03-24T00:14:56.567 INFO:teuthology.orchestra.run.smithi118.stdout: "want_state": "up:active", 2024-03-24T00:14:56.567 INFO:teuthology.orchestra.run.smithi118.stdout: "state": "up:active", 2024-03-24T00:14:56.567 INFO:teuthology.orchestra.run.smithi118.stdout: "fs_name": "cephfs", 2024-03-24T00:14:56.567 INFO:teuthology.orchestra.run.smithi118.stdout: "rank_uptime": 114.222584458, 2024-03-24T00:14:56.567 INFO:teuthology.orchestra.run.smithi118.stdout: "mdsmap_epoch": 27, 2024-03-24T00:14:56.567 INFO:teuthology.orchestra.run.smithi118.stdout: "osdmap_epoch": 88, 2024-03-24T00:14:56.568 INFO:teuthology.orchestra.run.smithi118.stdout: "osdmap_epoch_barrier": 86, 2024-03-24T00:14:56.568 INFO:teuthology.orchestra.run.smithi118.stdout: "uptime": 121.427588224 2024-03-24T00:14:56.568 INFO:teuthology.orchestra.run.smithi118.stdout:} 2024-03-24T00:14:56.568 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:56.562+0000 7fc013200640 1 -- 172.21.15.118:0/201383769 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7fbff80a2800 msgr2=0x7fbff80a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:14:56.568 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:56.562+0000 7fc013200640 1 --2- 172.21.15.118:0/201383769 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7fbff80a2800 0x7fbff80a4cc0 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x55babc767fd0 tx=0x7fc000015040 comp rx=0 tx=0).stop 2024-03-24T00:14:56.569 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:56.562+0000 7fc013200640 1 -- 172.21.15.118:0/201383769 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc014105350 msgr2=0x7fc0141a0f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:14:56.569 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:56.562+0000 7fc013200640 1 --2- 172.21.15.118:0/201383769 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc014105350 0x7fc0141a0f90 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7fc00c037040 tx=0x7fc00c035d10 comp rx=0 tx=0).stop 2024-03-24T00:14:56.569 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:56.562+0000 7fc013200640 1 -- 172.21.15.118:0/201383769 shutdown_connections 2024-03-24T00:14:56.569 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:56.562+0000 7fc013200640 1 --2- 172.21.15.118:0/201383769 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7fbff80a2800 0x7fbff80a4cc0 unknown :-1 s=CLOSED pgs=224 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:14:56.569 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:56.562+0000 7fc013200640 1 --2- 172.21.15.118:0/201383769 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc014105350 0x7fc0141a0f90 unknown :-1 s=CLOSED pgs=144 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:14:56.569 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:56.562+0000 7fc013200640 1 --2- 172.21.15.118:0/201383769 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fc014104990 0x7fc014199f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:14:56.569 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:56.562+0000 7fc013200640 1 --2- 172.21.15.118:0/201383769 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fc0141037a0 0x7fc0141999e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:14:56.569 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:56.562+0000 7fc013200640 1 -- 172.21.15.118:0/201383769 >> 172.21.15.118:0/201383769 conn(0x7fc0140fed10 msgr2=0x7fc014107c30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:14:56.569 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:56.562+0000 7fc013200640 1 -- 172.21.15.118:0/201383769 shutdown_connections 2024-03-24T00:14:56.569 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:56.563+0000 7fc013200640 1 -- 172.21.15.118:0/201383769 wait complete. 2024-03-24T00:14:56.580 INFO:tasks.cephadm:Setup complete, yielding 2024-03-24T00:14:56.581 INFO:teuthology.run_tasks:Running task cephadm.shell... 2024-03-24T00:14:56.591 INFO:tasks.cephadm:Running commands on role mon.a host ubuntu@smithi118.front.sepia.ceph.com 2024-03-24T00:14:56.591 DEBUG:teuthology.orchestra.run.smithi118:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d79a5420-e971-11ee-95cd-87774f69a715 -- bash -c 'ceph orch status' 2024-03-24T00:14:57.044 INFO:teuthology.orchestra.run.smithi118.stderr:Inferring config /var/lib/ceph/d79a5420-e971-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T00:14:58.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:14:58 smithi193 ceph-mon[36878]: pgmap v374: 97 pgs: 97 active+clean; 645 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-03-24T00:14:58.979 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:14:58 smithi118 ceph-mon[31535]: pgmap v374: 97 pgs: 97 active+clean; 645 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-03-24T00:14:59.203 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:14:58 smithi189 ceph-mon[36187]: pgmap v374: 97 pgs: 97 active+clean; 645 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-03-24T00:14:59.525 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:59.524+0000 7fee96000700 1 -- 172.21.15.118:0/412399036 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fee90103d70 msgr2=0x7fee9010a620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:14:59.525 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:59.524+0000 7fee96000700 1 --2- 172.21.15.118:0/412399036 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fee90103d70 0x7fee9010a620 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7fee8800b0d0 tx=0x7fee8800b3e0 comp rx=0 tx=0).stop 2024-03-24T00:14:59.525 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:59.524+0000 7fee96000700 1 -- 172.21.15.118:0/412399036 shutdown_connections 2024-03-24T00:14:59.525 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:59.524+0000 7fee96000700 1 --2- 172.21.15.118:0/412399036 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fee90103d70 0x7fee9010a620 unknown :-1 s=CLOSED pgs=147 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:14:59.525 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:59.524+0000 7fee96000700 1 --2- 172.21.15.118:0/412399036 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fee901021d0 0x7fee901025f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:14:59.525 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:59.524+0000 7fee96000700 1 --2- 172.21.15.118:0/412399036 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fee901033d0 0x7fee90103830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:14:59.525 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:59.524+0000 7fee96000700 1 -- 172.21.15.118:0/412399036 >> 172.21.15.118:0/412399036 conn(0x7fee900fd730 msgr2=0x7fee900ffbb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:14:59.525 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:59.524+0000 7fee96000700 1 -- 172.21.15.118:0/412399036 shutdown_connections 2024-03-24T00:14:59.525 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:59.524+0000 7fee96000700 1 -- 172.21.15.118:0/412399036 wait complete. 2024-03-24T00:14:59.527 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:59.525+0000 7fee96000700 1 Processor -- start 2024-03-24T00:14:59.527 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:59.526+0000 7fee96000700 1 -- start start 2024-03-24T00:14:59.527 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:59.526+0000 7fee96000700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fee901021d0 0x7fee901983d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:14:59.527 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:59.526+0000 7fee96000700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fee901033d0 0x7fee90198910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:14:59.527 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:59.526+0000 7fee96000700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fee90103d70 0x7fee9019f990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:14:59.527 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:59.526+0000 7fee96000700 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fee9010de20 con 0x7fee901033d0 2024-03-24T00:14:59.527 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:59.526+0000 7fee96000700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fee9010dca0 con 0x7fee90103d70 2024-03-24T00:14:59.527 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:59.526+0000 7fee96000700 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fee9010dfa0 con 0x7fee901021d0 2024-03-24T00:14:59.528 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:59.526+0000 7fee8f400700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fee901021d0 0x7fee901983d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:14:59.528 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:59.526+0000 7fee8f400700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fee901021d0 0x7fee901983d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.118:43018/0 (socket says 172.21.15.118:43018) 2024-03-24T00:14:59.528 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:59.526+0000 7fee8f400700 1 -- 172.21.15.118:0/1389060969 learned_addr learned my addr 172.21.15.118:0/1389060969 (peer_addr_for_me v2:172.21.15.118:0/0) 2024-03-24T00:14:59.528 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:59.526+0000 7fee8ea00700 1 --2- 172.21.15.118:0/1389060969 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fee901033d0 0x7fee90198910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:14:59.528 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:59.526+0000 7fee8fe00700 1 --2- 172.21.15.118:0/1389060969 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fee90103d70 0x7fee9019f990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:14:59.528 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:59.527+0000 7fee8fe00700 1 -- 172.21.15.118:0/1389060969 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fee901021d0 msgr2=0x7fee901983d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:14:59.528 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:59.527+0000 7fee8fe00700 1 --2- 172.21.15.118:0/1389060969 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fee901021d0 0x7fee901983d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:14:59.528 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:59.527+0000 7fee8fe00700 1 -- 172.21.15.118:0/1389060969 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fee901033d0 msgr2=0x7fee90198910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:14:59.528 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:59.527+0000 7fee8fe00700 1 --2- 172.21.15.118:0/1389060969 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fee901033d0 0x7fee90198910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:14:59.528 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:59.527+0000 7fee8fe00700 1 -- 172.21.15.118:0/1389060969 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fee901a0030 con 0x7fee90103d70 2024-03-24T00:14:59.528 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:59.527+0000 7fee8f400700 1 --2- 172.21.15.118:0/1389060969 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fee901021d0 0x7fee901983d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T00:14:59.528 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:59.527+0000 7fee8fe00700 1 --2- 172.21.15.118:0/1389060969 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fee90103d70 0x7fee9019f990 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7fee8800d010 tx=0x7fee880079a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:14:59.528 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:59.527+0000 7fee87e00700 1 -- 172.21.15.118:0/1389060969 <== mon.2 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fee88021070 con 0x7fee90103d70 2024-03-24T00:14:59.528 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:59.527+0000 7fee87e00700 1 -- 172.21.15.118:0/1389060969 <== mon.2 v2:172.21.15.189:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fee8801db30 con 0x7fee90103d70 2024-03-24T00:14:59.528 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:59.527+0000 7fee96000700 1 -- 172.21.15.118:0/1389060969 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fee901a02c0 con 0x7fee90103d70 2024-03-24T00:14:59.529 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:59.527+0000 7fee87e00700 1 -- 172.21.15.118:0/1389060969 <== mon.2 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fee8801bba0 con 0x7fee90103d70 2024-03-24T00:14:59.529 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:59.528+0000 7fee96000700 1 -- 172.21.15.118:0/1389060969 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fee901a0770 con 0x7fee90103d70 2024-03-24T00:14:59.529 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:59.529+0000 7fee96000700 1 -- 172.21.15.118:0/1389060969 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fee90068e50 con 0x7fee90103d70 2024-03-24T00:14:59.530 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:59.530+0000 7fee87e00700 1 -- 172.21.15.118:0/1389060969 <== mon.2 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 18) v1 ==== 137291+0+0 (secure 0 0 0) 0x7fee88024070 con 0x7fee90103d70 2024-03-24T00:14:59.531 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:59.532+0000 7fee87e00700 1 --2- 172.21.15.118:0/1389060969 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7fee740a2910 0x7fee740a4dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:14:59.531 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:59.532+0000 7fee87e00700 1 -- 172.21.15.118:0/1389060969 <== mon.2 v2:172.21.15.189:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 8248+0+0 (secure 0 0 0) 0x7fee880cfa10 con 0x7fee90103d70 2024-03-24T00:14:59.531 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:59.532+0000 7fee8f400700 1 --2- 172.21.15.118:0/1389060969 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7fee740a2910 0x7fee740a4dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:14:59.532 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:59.533+0000 7fee8f400700 1 --2- 172.21.15.118:0/1389060969 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7fee740a2910 0x7fee740a4dd0 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7fee80005950 tx=0x7fee80009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:14:59.535 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:59.535+0000 7fee87e00700 1 -- 172.21.15.118:0/1389060969 <== mon.2 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7fee8809cad0 con 0x7fee90103d70 2024-03-24T00:14:59.776 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:59.776+0000 7fee96000700 1 -- 172.21.15.118:0/1389060969 --> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] -- mgr_command(tid 0: {"prefix": "orch status", "target": ["mon-mgr", ""]}) v1 -- 0x7fee900625d0 con 0x7fee740a2910 2024-03-24T00:14:59.778 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:59.778+0000 7fee87e00700 1 -- 172.21.15.118:0/1389060969 <== mgr.14152 v2:172.21.15.118:6800/2112912943 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+43 (secure 0 0 0) 0x7fee900625d0 con 0x7fee740a2910 2024-03-24T00:14:59.778 INFO:teuthology.orchestra.run.smithi118.stdout:Backend: cephadm 2024-03-24T00:14:59.778 INFO:teuthology.orchestra.run.smithi118.stdout:Available: Yes 2024-03-24T00:14:59.778 INFO:teuthology.orchestra.run.smithi118.stdout:Paused: No 2024-03-24T00:14:59.781 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:59.782+0000 7fee96000700 1 -- 172.21.15.118:0/1389060969 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7fee740a2910 msgr2=0x7fee740a4dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:14:59.782 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:59.782+0000 7fee96000700 1 --2- 172.21.15.118:0/1389060969 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7fee740a2910 0x7fee740a4dd0 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7fee80005950 tx=0x7fee80009260 comp rx=0 tx=0).stop 2024-03-24T00:14:59.782 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:59.782+0000 7fee96000700 1 -- 172.21.15.118:0/1389060969 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fee90103d70 msgr2=0x7fee9019f990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:14:59.782 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:59.782+0000 7fee96000700 1 --2- 172.21.15.118:0/1389060969 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fee90103d70 0x7fee9019f990 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7fee8800d010 tx=0x7fee880079a0 comp rx=0 tx=0).stop 2024-03-24T00:14:59.782 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:59.782+0000 7fee96000700 1 -- 172.21.15.118:0/1389060969 shutdown_connections 2024-03-24T00:14:59.782 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:59.782+0000 7fee96000700 1 --2- 172.21.15.118:0/1389060969 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fee90103d70 0x7fee9019f990 unknown :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:14:59.782 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:59.782+0000 7fee96000700 1 --2- 172.21.15.118:0/1389060969 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7fee740a2910 0x7fee740a4dd0 unknown :-1 s=CLOSED pgs=225 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:14:59.782 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:59.782+0000 7fee96000700 1 --2- 172.21.15.118:0/1389060969 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fee901021d0 0x7fee901983d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:14:59.782 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:59.782+0000 7fee96000700 1 --2- 172.21.15.118:0/1389060969 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fee901033d0 0x7fee90198910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:14:59.782 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:59.782+0000 7fee96000700 1 -- 172.21.15.118:0/1389060969 >> 172.21.15.118:0/1389060969 conn(0x7fee900fd730 msgr2=0x7fee90106670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:14:59.782 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:59.782+0000 7fee96000700 1 -- 172.21.15.118:0/1389060969 shutdown_connections 2024-03-24T00:14:59.782 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:14:59.782+0000 7fee96000700 1 -- 172.21.15.118:0/1389060969 wait complete. 2024-03-24T00:15:00.640 DEBUG:teuthology.orchestra.run.smithi118:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d79a5420-e971-11ee-95cd-87774f69a715 -- bash -c 'ceph orch ps' 2024-03-24T00:15:00.945 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:15:00 smithi118 ceph-mon[31535]: pgmap v375: 97 pgs: 97 active+clean; 645 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 11 op/s 2024-03-24T00:15:00.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:15:00 smithi118 ceph-mon[31535]: from='client.25049 -' entity='client.admin' cmd=[{"prefix": "orch status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T00:15:00.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:15:00 smithi193 ceph-mon[36878]: pgmap v375: 97 pgs: 97 active+clean; 645 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 11 op/s 2024-03-24T00:15:00.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:15:00 smithi193 ceph-mon[36878]: from='client.25049 -' entity='client.admin' cmd=[{"prefix": "orch status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T00:15:01.081 INFO:teuthology.orchestra.run.smithi118.stderr:Inferring config /var/lib/ceph/d79a5420-e971-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T00:15:01.203 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:15:00 smithi189 ceph-mon[36187]: pgmap v375: 97 pgs: 97 active+clean; 645 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 11 op/s 2024-03-24T00:15:01.203 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:15:00 smithi189 ceph-mon[36187]: from='client.25049 -' entity='client.admin' cmd=[{"prefix": "orch status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T00:15:03.050 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:15:02 smithi118 ceph-mon[31535]: pgmap v376: 97 pgs: 97 active+clean; 645 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-24T00:15:03.203 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:15:02 smithi189 ceph-mon[36187]: pgmap v376: 97 pgs: 97 active+clean; 645 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-24T00:15:03.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:15:02 smithi193 ceph-mon[36878]: pgmap v376: 97 pgs: 97 active+clean; 645 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-24T00:15:03.212 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:03.213+0000 7fdc10c00700 1 -- 172.21.15.118:0/2838140060 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fdc0c1034b0 msgr2=0x7fdc0c103910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:15:03.213 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:03.213+0000 7fdc10c00700 1 --2- 172.21.15.118:0/2838140060 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fdc0c1034b0 0x7fdc0c103910 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7fdbf8009990 tx=0x7fdbf8009ca0 comp rx=0 tx=0).stop 2024-03-24T00:15:03.213 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:03.213+0000 7fdc10c00700 1 -- 172.21.15.118:0/2838140060 shutdown_connections 2024-03-24T00:15:03.213 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:03.213+0000 7fdc10c00700 1 --2- 172.21.15.118:0/2838140060 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fdc0c103e50 0x7fdc0c10a700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:15:03.213 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:03.213+0000 7fdc10c00700 1 --2- 172.21.15.118:0/2838140060 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fdc0c1004d0 0x7fdc0c1008f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:15:03.213 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:03.213+0000 7fdc10c00700 1 --2- 172.21.15.118:0/2838140060 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fdc0c1034b0 0x7fdc0c103910 unknown :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:15:03.214 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:03.213+0000 7fdc10c00700 1 -- 172.21.15.118:0/2838140060 >> 172.21.15.118:0/2838140060 conn(0x7fdc0c0fbe30 msgr2=0x7fdc0c0fe270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:15:03.214 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:03.213+0000 7fdc10c00700 1 -- 172.21.15.118:0/2838140060 shutdown_connections 2024-03-24T00:15:03.214 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:03.213+0000 7fdc10c00700 1 -- 172.21.15.118:0/2838140060 wait complete. 2024-03-24T00:15:03.214 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:03.215+0000 7fdc10c00700 1 Processor -- start 2024-03-24T00:15:03.214 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:03.215+0000 7fdc10c00700 1 -- start start 2024-03-24T00:15:03.215 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:03.216+0000 7fdc10c00700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fdc0c1004d0 0x7fdc0c198420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:15:03.215 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:03.216+0000 7fdc10c00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fdc0c1034b0 0x7fdc0c198960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:15:03.216 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:03.216+0000 7fdc10c00700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fdc0c103e50 0x7fdc0c19f9e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:15:03.216 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:03.216+0000 7fdc10c00700 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdc0c10de30 con 0x7fdc0c1004d0 2024-03-24T00:15:03.216 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:03.216+0000 7fdc10c00700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdc0c10dcb0 con 0x7fdc0c1034b0 2024-03-24T00:15:03.216 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:03.216+0000 7fdc10c00700 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdc0c10dfb0 con 0x7fdc0c103e50 2024-03-24T00:15:03.216 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:03.216+0000 7fdc09400700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fdc0c1034b0 0x7fdc0c198960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:15:03.216 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:03.216+0000 7fdc09e00700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fdc0c1004d0 0x7fdc0c198420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:15:03.216 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:03.216+0000 7fdc09400700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fdc0c1034b0 0x7fdc0c198960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.118:51242/0 (socket says 172.21.15.118:51242) 2024-03-24T00:15:03.216 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:03.216+0000 7fdc09400700 1 -- 172.21.15.118:0/2585817212 learned_addr learned my addr 172.21.15.118:0/2585817212 (peer_addr_for_me v2:172.21.15.118:0/0) 2024-03-24T00:15:03.216 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:03.216+0000 7fdc0a800700 1 --2- 172.21.15.118:0/2585817212 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fdc0c103e50 0x7fdc0c19f9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:15:03.217 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:03.216+0000 7fdc09400700 1 -- 172.21.15.118:0/2585817212 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fdc0c103e50 msgr2=0x7fdc0c19f9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:15:03.218 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:03.216+0000 7fdc09400700 1 --2- 172.21.15.118:0/2585817212 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fdc0c103e50 0x7fdc0c19f9e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:15:03.218 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:03.216+0000 7fdc09400700 1 -- 172.21.15.118:0/2585817212 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fdc0c1004d0 msgr2=0x7fdc0c198420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:15:03.218 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:03.216+0000 7fdc09400700 1 --2- 172.21.15.118:0/2585817212 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fdc0c1004d0 0x7fdc0c198420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:15:03.218 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:03.216+0000 7fdc09400700 1 -- 172.21.15.118:0/2585817212 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdc0c1a0080 con 0x7fdc0c1034b0 2024-03-24T00:15:03.218 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:03.217+0000 7fdc09400700 1 --2- 172.21.15.118:0/2585817212 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fdc0c1034b0 0x7fdc0c198960 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7fdbf8000c00 tx=0x7fdbf800bbb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:15:03.218 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:03.217+0000 7fdc02a00700 1 -- 172.21.15.118:0/2585817212 <== mon.2 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdbf8026070 con 0x7fdc0c1034b0 2024-03-24T00:15:03.218 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:03.217+0000 7fdc02a00700 1 -- 172.21.15.118:0/2585817212 <== mon.2 v2:172.21.15.189:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fdbf8003d70 con 0x7fdc0c1034b0 2024-03-24T00:15:03.218 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:03.217+0000 7fdc02a00700 1 -- 172.21.15.118:0/2585817212 <== mon.2 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdbf8021410 con 0x7fdc0c1034b0 2024-03-24T00:15:03.218 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:03.217+0000 7fdc10c00700 1 -- 172.21.15.118:0/2585817212 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdc0c1a0310 con 0x7fdc0c1034b0 2024-03-24T00:15:03.219 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:03.217+0000 7fdc10c00700 1 -- 172.21.15.118:0/2585817212 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdc0c1a07c0 con 0x7fdc0c1034b0 2024-03-24T00:15:03.219 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:03.218+0000 7fdc10c00700 1 -- 172.21.15.118:0/2585817212 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdc0c068e50 con 0x7fdc0c1034b0 2024-03-24T00:15:03.220 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:03.220+0000 7fdc02a00700 1 -- 172.21.15.118:0/2585817212 <== mon.2 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 18) v1 ==== 137291+0+0 (secure 0 0 0) 0x7fdbf8009d10 con 0x7fdc0c1034b0 2024-03-24T00:15:03.222 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:03.222+0000 7fdc02a00700 1 --2- 172.21.15.118:0/2585817212 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7fdbf00a2910 0x7fdbf00a4dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:15:03.222 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:03.222+0000 7fdc02a00700 1 -- 172.21.15.118:0/2585817212 <== mon.2 v2:172.21.15.189:3300/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 8248+0+0 (secure 0 0 0) 0x7fdbf80cf850 con 0x7fdc0c1034b0 2024-03-24T00:15:03.222 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:03.222+0000 7fdc09e00700 1 --2- 172.21.15.118:0/2585817212 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7fdbf00a2910 0x7fdbf00a4dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:15:03.222 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:03.223+0000 7fdc09e00700 1 --2- 172.21.15.118:0/2585817212 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7fdbf00a2910 0x7fdbf00a4dd0 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7fdbfc009680 tx=0x7fdbfc009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:15:03.223 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:03.224+0000 7fdc02a00700 1 -- 172.21.15.118:0/2585817212 <== mon.2 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7fdbf8013070 con 0x7fdc0c1034b0 2024-03-24T00:15:03.458 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:03.458+0000 7fdc10c00700 1 -- 172.21.15.118:0/2585817212 --> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7fdc0c0625d0 con 0x7fdbf00a2910 2024-03-24T00:15:03.480 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:03.480+0000 7fdc02a00700 1 -- 172.21.15.118:0/2585817212 <== mgr.14152 v2:172.21.15.118:6800/2112912943 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+4123 (secure 0 0 0) 0x7fdc0c0625d0 con 0x7fdbf00a2910 2024-03-24T00:15:03.480 INFO:teuthology.orchestra.run.smithi118.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-03-24T00:15:03.480 INFO:teuthology.orchestra.run.smithi118.stdout:mds.a smithi118 running (2m) 27s ago 2m 15.7M - 19.0.0-2145-g3be9a5c9 c664afe56bb0 8b122c08c5ee 2024-03-24T00:15:03.480 INFO:teuthology.orchestra.run.smithi118.stdout:mds.b smithi189 running (2m) 60s ago 2m 26.1M - 19.0.0-2145-g3be9a5c9 c664afe56bb0 a872553a6492 2024-03-24T00:15:03.481 INFO:teuthology.orchestra.run.smithi118.stdout:mds.c smithi193 running (2m) 60s ago 2m 18.1M - 19.0.0-2145-g3be9a5c9 c664afe56bb0 f0210a5dc42f 2024-03-24T00:15:03.481 INFO:teuthology.orchestra.run.smithi118.stdout:mds.d smithi118 running (2m) 27s ago 2m 12.4M - 19.0.0-2145-g3be9a5c9 c664afe56bb0 92f7fd8f0d7a 2024-03-24T00:15:03.481 INFO:teuthology.orchestra.run.smithi118.stdout:mds.e smithi189 running (2m) 60s ago 2m 17.7M - 19.0.0-2145-g3be9a5c9 c664afe56bb0 489bbba1e900 2024-03-24T00:15:03.481 INFO:teuthology.orchestra.run.smithi118.stdout:mds.f smithi193 running (2m) 60s ago 2m 18.1M - 19.0.0-2145-g3be9a5c9 c664afe56bb0 b0ac59f98847 2024-03-24T00:15:03.481 INFO:teuthology.orchestra.run.smithi118.stdout:mds.g smithi118 running (2m) 27s ago 2m 16.9M - 19.0.0-2145-g3be9a5c9 c664afe56bb0 2943a8e74123 2024-03-24T00:15:03.481 INFO:teuthology.orchestra.run.smithi118.stdout:mds.h smithi189 running (2m) 60s ago 2m 17.1M - 19.0.0-2145-g3be9a5c9 c664afe56bb0 8cde5e7f8174 2024-03-24T00:15:03.481 INFO:teuthology.orchestra.run.smithi118.stdout:mds.i smithi193 running (2m) 60s ago 2m 18.0M - 19.0.0-2145-g3be9a5c9 c664afe56bb0 749e1d96d397 2024-03-24T00:15:03.481 INFO:teuthology.orchestra.run.smithi118.stdout:mds.j smithi118 running (2m) 27s ago 2m 12.4M - 19.0.0-2145-g3be9a5c9 c664afe56bb0 4b581125b505 2024-03-24T00:15:03.481 INFO:teuthology.orchestra.run.smithi118.stdout:mds.k smithi189 running (2m) 60s ago 2m 17.0M - 19.0.0-2145-g3be9a5c9 c664afe56bb0 89f5182d7c26 2024-03-24T00:15:03.481 INFO:teuthology.orchestra.run.smithi118.stdout:mds.l smithi193 running (2m) 60s ago 2m 19.3M - 19.0.0-2145-g3be9a5c9 c664afe56bb0 af712abd88af 2024-03-24T00:15:03.481 INFO:teuthology.orchestra.run.smithi118.stdout:mgr.x smithi118 *:9283,8765 running (10m) 27s ago 10m 514M - 19.0.0-2145-g3be9a5c9 c664afe56bb0 16a004216cba 2024-03-24T00:15:03.481 INFO:teuthology.orchestra.run.smithi118.stdout:mgr.y smithi189 *:8443,8765 running (8m) 60s ago 8m 451M - 19.0.0-2145-g3be9a5c9 c664afe56bb0 93099988c06e 2024-03-24T00:15:03.481 INFO:teuthology.orchestra.run.smithi118.stdout:mgr.z smithi193 *:8443,8765 running (8m) 60s ago 8m 453M - 19.0.0-2145-g3be9a5c9 c664afe56bb0 df9c7647eda0 2024-03-24T00:15:03.481 INFO:teuthology.orchestra.run.smithi118.stdout:mon.a smithi118 running (10m) 27s ago 11m 47.6M 2048M 19.0.0-2145-g3be9a5c9 c664afe56bb0 9a3b56cb7cc8 2024-03-24T00:15:03.482 INFO:teuthology.orchestra.run.smithi118.stdout:mon.b smithi189 running (8m) 60s ago 8m 43.0M 2048M 19.0.0-2145-g3be9a5c9 c664afe56bb0 332ea823179e 2024-03-24T00:15:03.482 INFO:teuthology.orchestra.run.smithi118.stdout:mon.c smithi193 running (9m) 60s ago 9m 39.5M 2048M 19.0.0-2145-g3be9a5c9 c664afe56bb0 0284545a88c5 2024-03-24T00:15:03.482 INFO:teuthology.orchestra.run.smithi118.stdout:osd.0 smithi118 running (8m) 27s ago 8m 44.1M 4272M 19.0.0-2145-g3be9a5c9 c664afe56bb0 796fc494a238 2024-03-24T00:15:03.482 INFO:teuthology.orchestra.run.smithi118.stdout:osd.1 smithi189 running (7m) 60s ago 7m 47.4M 4270M 19.0.0-2145-g3be9a5c9 c664afe56bb0 5b082a48fda4 2024-03-24T00:15:03.482 INFO:teuthology.orchestra.run.smithi118.stdout:osd.2 smithi193 running (7m) 60s ago 7m 45.3M 4270M 19.0.0-2145-g3be9a5c9 c664afe56bb0 c5b2b368e96b 2024-03-24T00:15:03.482 INFO:teuthology.orchestra.run.smithi118.stdout:osd.3 smithi118 running (6m) 27s ago 6m 53.0M 4272M 19.0.0-2145-g3be9a5c9 c664afe56bb0 622b727a5611 2024-03-24T00:15:03.482 INFO:teuthology.orchestra.run.smithi118.stdout:osd.4 smithi189 running (6m) 60s ago 6m 52.9M 4270M 19.0.0-2145-g3be9a5c9 c664afe56bb0 f019d42b8ff1 2024-03-24T00:15:03.482 INFO:teuthology.orchestra.run.smithi118.stdout:osd.5 smithi193 running (5m) 60s ago 5m 46.2M 4270M 19.0.0-2145-g3be9a5c9 c664afe56bb0 ba9e56c4eb16 2024-03-24T00:15:03.482 INFO:teuthology.orchestra.run.smithi118.stdout:osd.6 smithi118 running (5m) 27s ago 5m 40.9M 4272M 19.0.0-2145-g3be9a5c9 c664afe56bb0 2d383b462099 2024-03-24T00:15:03.482 INFO:teuthology.orchestra.run.smithi118.stdout:osd.7 smithi189 running (4m) 60s ago 4m 43.6M 4270M 19.0.0-2145-g3be9a5c9 c664afe56bb0 0a8b745009ba 2024-03-24T00:15:03.482 INFO:teuthology.orchestra.run.smithi118.stdout:osd.8 smithi193 running (4m) 60s ago 4m 55.2M 4270M 19.0.0-2145-g3be9a5c9 c664afe56bb0 c2109ee696bd 2024-03-24T00:15:03.483 INFO:teuthology.orchestra.run.smithi118.stdout:osd.9 smithi118 running (3m) 27s ago 3m 51.3M 4272M 19.0.0-2145-g3be9a5c9 c664afe56bb0 f7f5e9ce5086 2024-03-24T00:15:03.483 INFO:teuthology.orchestra.run.smithi118.stdout:osd.10 smithi189 running (3m) 60s ago 3m 42.8M 4270M 19.0.0-2145-g3be9a5c9 c664afe56bb0 0775da42b099 2024-03-24T00:15:03.483 INFO:teuthology.orchestra.run.smithi118.stdout:osd.11 smithi193 running (3m) 60s ago 3m 41.1M 4270M 19.0.0-2145-g3be9a5c9 c664afe56bb0 175169a7efde 2024-03-24T00:15:03.484 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:03.483+0000 7fdc10c00700 1 -- 172.21.15.118:0/2585817212 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7fdbf00a2910 msgr2=0x7fdbf00a4dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:15:03.484 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:03.483+0000 7fdc10c00700 1 --2- 172.21.15.118:0/2585817212 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7fdbf00a2910 0x7fdbf00a4dd0 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7fdbfc009680 tx=0x7fdbfc009260 comp rx=0 tx=0).stop 2024-03-24T00:15:03.484 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:03.483+0000 7fdc10c00700 1 -- 172.21.15.118:0/2585817212 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fdc0c1034b0 msgr2=0x7fdc0c198960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:15:03.484 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:03.483+0000 7fdc10c00700 1 --2- 172.21.15.118:0/2585817212 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fdc0c1034b0 0x7fdc0c198960 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7fdbf8000c00 tx=0x7fdbf800bbb0 comp rx=0 tx=0).stop 2024-03-24T00:15:03.484 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:03.483+0000 7fdc10c00700 1 -- 172.21.15.118:0/2585817212 shutdown_connections 2024-03-24T00:15:03.484 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:03.483+0000 7fdc10c00700 1 --2- 172.21.15.118:0/2585817212 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7fdbf00a2910 0x7fdbf00a4dd0 unknown :-1 s=CLOSED pgs=226 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:15:03.484 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:03.483+0000 7fdc10c00700 1 --2- 172.21.15.118:0/2585817212 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fdc0c103e50 0x7fdc0c19f9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:15:03.484 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:03.483+0000 7fdc10c00700 1 --2- 172.21.15.118:0/2585817212 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fdc0c1004d0 0x7fdc0c198420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:15:03.484 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:03.483+0000 7fdc10c00700 1 --2- 172.21.15.118:0/2585817212 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fdc0c1034b0 0x7fdc0c198960 unknown :-1 s=CLOSED pgs=149 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:15:03.484 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:03.483+0000 7fdc10c00700 1 -- 172.21.15.118:0/2585817212 >> 172.21.15.118:0/2585817212 conn(0x7fdc0c0fbe30 msgr2=0x7fdc0c106750 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:15:03.484 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:03.484+0000 7fdc10c00700 1 -- 172.21.15.118:0/2585817212 shutdown_connections 2024-03-24T00:15:03.485 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:03.484+0000 7fdc10c00700 1 -- 172.21.15.118:0/2585817212 wait complete. 2024-03-24T00:15:04.271 DEBUG:teuthology.orchestra.run.smithi118:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d79a5420-e971-11ee-95cd-87774f69a715 -- bash -c 'ceph orch ls' 2024-03-24T00:15:04.729 INFO:teuthology.orchestra.run.smithi118.stderr:Inferring config /var/lib/ceph/d79a5420-e971-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T00:15:05.089 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:15:04 smithi118 ceph-mon[31535]: from='client.25055 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T00:15:05.089 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:15:04 smithi118 ceph-mon[31535]: pgmap v377: 97 pgs: 97 active+clean; 645 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-03-24T00:15:05.203 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:15:04 smithi189 ceph-mon[36187]: from='client.25055 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T00:15:05.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:15:04 smithi189 ceph-mon[36187]: pgmap v377: 97 pgs: 97 active+clean; 645 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-03-24T00:15:05.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:15:04 smithi193 ceph-mon[36878]: from='client.25055 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T00:15:05.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:15:04 smithi193 ceph-mon[36878]: pgmap v377: 97 pgs: 97 active+clean; 645 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-03-24T00:15:06.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:15:06 smithi118 ceph-mon[31535]: pgmap v378: 97 pgs: 97 active+clean; 645 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 10 op/s 2024-03-24T00:15:06.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:15:06 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1d", "id": [8, 11]}]: dispatch 2024-03-24T00:15:07.017 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:07.017+0000 7f7f0ca00700 1 -- 172.21.15.118:0/1506626738 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f7f08105510 msgr2=0x7f7f08107900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:15:07.017 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:07.017+0000 7f7f0ca00700 1 --2- 172.21.15.118:0/1506626738 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f7f08105510 0x7f7f08107900 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7f7ef40098f0 tx=0x7f7ef4009c00 comp rx=0 tx=0).stop 2024-03-24T00:15:07.018 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:07.018+0000 7f7f0ca00700 1 -- 172.21.15.118:0/1506626738 shutdown_connections 2024-03-24T00:15:07.018 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:07.018+0000 7f7f0ca00700 1 --2- 172.21.15.118:0/1506626738 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7f08107e40 0x7f7f0810a250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:15:07.018 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:07.018+0000 7f7f0ca00700 1 --2- 172.21.15.118:0/1506626738 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f7f08102be0 0x7f7f08104fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:15:07.018 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:07.018+0000 7f7f0ca00700 1 --2- 172.21.15.118:0/1506626738 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f7f08105510 0x7f7f08107900 unknown :-1 s=CLOSED pgs=152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:15:07.018 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:07.018+0000 7f7f0ca00700 1 -- 172.21.15.118:0/1506626738 >> 172.21.15.118:0/1506626738 conn(0x7f7f080fc5b0 msgr2=0x7f7f080fe9d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:15:07.018 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:07.018+0000 7f7f0ca00700 1 -- 172.21.15.118:0/1506626738 shutdown_connections 2024-03-24T00:15:07.018 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:07.018+0000 7f7f0ca00700 1 -- 172.21.15.118:0/1506626738 wait complete. 2024-03-24T00:15:07.019 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:07.019+0000 7f7f0ca00700 1 Processor -- start 2024-03-24T00:15:07.019 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:07.019+0000 7f7f0ca00700 1 -- start start 2024-03-24T00:15:07.019 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:07.020+0000 7f7f0ca00700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f7f08102be0 0x7f7f0819a550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:15:07.020 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:07.020+0000 7f7f0ca00700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f7f08105510 0x7f7f0819aa90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:15:07.020 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:07.020+0000 7f7f0ca00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7f08107e40 0x7f7f081a1b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:15:07.020 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:07.020+0000 7f7f0ca00700 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7f0810dd50 con 0x7f7f08102be0 2024-03-24T00:15:07.020 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:07.020+0000 7f7f0ca00700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7f0810dbd0 con 0x7f7f08107e40 2024-03-24T00:15:07.020 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:07.020+0000 7f7f0ca00700 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7f0810ded0 con 0x7f7f08105510 2024-03-24T00:15:07.020 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:07.020+0000 7f7f06a00700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f7f08105510 0x7f7f0819aa90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:15:07.020 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:07.020+0000 7f7f06a00700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f7f08105510 0x7f7f0819aa90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.118:53616/0 (socket says 172.21.15.118:53616) 2024-03-24T00:15:07.020 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:07.020+0000 7f7f06a00700 1 -- 172.21.15.118:0/82756701 learned_addr learned my addr 172.21.15.118:0/82756701 (peer_addr_for_me v2:172.21.15.118:0/0) 2024-03-24T00:15:07.020 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:07.020+0000 7f7f07e00700 1 --2- 172.21.15.118:0/82756701 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7f08107e40 0x7f7f081a1b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:15:07.020 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:07.020+0000 7f7f07400700 1 --2- 172.21.15.118:0/82756701 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f7f08102be0 0x7f7f0819a550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:15:07.021 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:07.020+0000 7f7f06a00700 1 -- 172.21.15.118:0/82756701 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7f08107e40 msgr2=0x7f7f081a1b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:15:07.021 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:07.020+0000 7f7f06a00700 1 --2- 172.21.15.118:0/82756701 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7f08107e40 0x7f7f081a1b10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:15:07.021 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:07.020+0000 7f7f06a00700 1 -- 172.21.15.118:0/82756701 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f7f08102be0 msgr2=0x7f7f0819a550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:15:07.021 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:07.020+0000 7f7f06a00700 1 --2- 172.21.15.118:0/82756701 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f7f08102be0 0x7f7f0819a550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:15:07.021 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:07.020+0000 7f7f06a00700 1 -- 172.21.15.118:0/82756701 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7f081a21b0 con 0x7f7f08105510 2024-03-24T00:15:07.022 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:07.020+0000 7f7f07400700 1 --2- 172.21.15.118:0/82756701 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f7f08102be0 0x7f7f0819a550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T00:15:07.022 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:07.021+0000 7f7f06a00700 1 --2- 172.21.15.118:0/82756701 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f7f08105510 0x7f7f0819aa90 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7f7ef400fca0 tx=0x7f7ef4005250 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:15:07.022 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:07.021+0000 7f7effe00700 1 -- 172.21.15.118:0/82756701 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7ef4025070 con 0x7f7f08105510 2024-03-24T00:15:07.022 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:07.021+0000 7f7effe00700 1 -- 172.21.15.118:0/82756701 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f7ef4005650 con 0x7f7f08105510 2024-03-24T00:15:07.022 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:07.021+0000 7f7f0ca00700 1 -- 172.21.15.118:0/82756701 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7f081a2440 con 0x7f7f08105510 2024-03-24T00:15:07.022 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:07.021+0000 7f7effe00700 1 -- 172.21.15.118:0/82756701 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7ef4003d20 con 0x7f7f08105510 2024-03-24T00:15:07.022 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:07.021+0000 7f7f0ca00700 1 -- 172.21.15.118:0/82756701 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7f081a28f0 con 0x7f7f08105510 2024-03-24T00:15:07.022 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:07.022+0000 7f7f0ca00700 1 -- 172.21.15.118:0/82756701 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7f08068e50 con 0x7f7f08105510 2024-03-24T00:15:07.023 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:07.024+0000 7f7effe00700 1 -- 172.21.15.118:0/82756701 <== mon.1 v2:172.21.15.193:3300/0 4 ==== mgrmap(e 18) v1 ==== 137291+0+0 (secure 0 0 0) 0x7f7ef4028070 con 0x7f7f08105510 2024-03-24T00:15:07.025 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:07.025+0000 7f7effe00700 1 --2- 172.21.15.118:0/82756701 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7f7eec0a2910 0x7f7eec0a4dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:15:07.025 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:07.025+0000 7f7effe00700 1 -- 172.21.15.118:0/82756701 <== mon.1 v2:172.21.15.193:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 8277+0+0 (secure 0 0 0) 0x7f7ef40cf410 con 0x7f7f08105510 2024-03-24T00:15:07.025 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:07.025+0000 7f7f07400700 1 --2- 172.21.15.118:0/82756701 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7f7eec0a2910 0x7f7eec0a4dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:15:07.025 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:07.026+0000 7f7f07400700 1 --2- 172.21.15.118:0/82756701 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7f7eec0a2910 0x7f7eec0a4dd0 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7f7ef8005fd0 tx=0x7f7ef8009310 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:15:07.029 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:07.029+0000 7f7effe00700 1 -- 172.21.15.118:0/82756701 <== mon.1 v2:172.21.15.193:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f7ef405c740 con 0x7f7f08105510 2024-03-24T00:15:07.203 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:15:06 smithi189 ceph-mon[36187]: pgmap v378: 97 pgs: 97 active+clean; 645 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 10 op/s 2024-03-24T00:15:07.203 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:15:06 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1d", "id": [8, 11]}]: dispatch 2024-03-24T00:15:07.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:15:06 smithi193 ceph-mon[36878]: pgmap v378: 97 pgs: 97 active+clean; 645 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 10 op/s 2024-03-24T00:15:07.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:15:06 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1d", "id": [8, 11]}]: dispatch 2024-03-24T00:15:07.261 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:07.261+0000 7f7f0ca00700 1 -- 172.21.15.118:0/82756701 --> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] -- mgr_command(tid 0: {"prefix": "orch ls", "target": ["mon-mgr", ""]}) v1 -- 0x7f7f080625d0 con 0x7f7eec0a2910 2024-03-24T00:15:07.270 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:07.271+0000 7f7effe00700 1 -- 172.21.15.118:0/82756701 <== mgr.14152 v2:172.21.15.118:6800/2112912943 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+980 (secure 0 0 0) 0x7f7f080625d0 con 0x7f7eec0a2910 2024-03-24T00:15:07.270 INFO:teuthology.orchestra.run.smithi118.stdout:NAME PORTS RUNNING REFRESHED AGE PLACEMENT 2024-03-24T00:15:07.271 INFO:teuthology.orchestra.run.smithi118.stdout:mds.all 12/12 63s ago 2m smithi118=a;smithi118=d;smithi118=g;smithi118=j;smithi189=b;smithi189=e;smithi189=h;smithi189=k;smithi193=c;smithi193=f;smithi193=i;smithi193=l;count:12 2024-03-24T00:15:07.271 INFO:teuthology.orchestra.run.smithi118.stdout:mgr 3/3 63s ago 8m smithi118=x;smithi189=y;smithi193=z;count:3 2024-03-24T00:15:07.271 INFO:teuthology.orchestra.run.smithi118.stdout:mon 3/3 63s ago 9m smithi118:172.21.15.118=a;smithi189:172.21.15.189=b;smithi193:172.21.15.193=c;count:3 2024-03-24T00:15:07.271 INFO:teuthology.orchestra.run.smithi118.stdout:osd 12 63s ago - 2024-03-24T00:15:07.273 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:07.273+0000 7f7f0ca00700 1 -- 172.21.15.118:0/82756701 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7f7eec0a2910 msgr2=0x7f7eec0a4dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:15:07.273 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:07.273+0000 7f7f0ca00700 1 --2- 172.21.15.118:0/82756701 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7f7eec0a2910 0x7f7eec0a4dd0 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7f7ef8005fd0 tx=0x7f7ef8009310 comp rx=0 tx=0).stop 2024-03-24T00:15:07.273 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:07.273+0000 7f7f0ca00700 1 -- 172.21.15.118:0/82756701 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f7f08105510 msgr2=0x7f7f0819aa90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:15:07.273 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:07.273+0000 7f7f0ca00700 1 --2- 172.21.15.118:0/82756701 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f7f08105510 0x7f7f0819aa90 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7f7ef400fca0 tx=0x7f7ef4005250 comp rx=0 tx=0).stop 2024-03-24T00:15:07.273 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:07.273+0000 7f7f0ca00700 1 -- 172.21.15.118:0/82756701 shutdown_connections 2024-03-24T00:15:07.273 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:07.273+0000 7f7f0ca00700 1 --2- 172.21.15.118:0/82756701 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7f08107e40 0x7f7f081a1b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:15:07.274 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:07.273+0000 7f7f0ca00700 1 --2- 172.21.15.118:0/82756701 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7f7eec0a2910 0x7f7eec0a4dd0 unknown :-1 s=CLOSED pgs=227 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:15:07.274 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:07.273+0000 7f7f0ca00700 1 --2- 172.21.15.118:0/82756701 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f7f08102be0 0x7f7f0819a550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:15:07.274 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:07.273+0000 7f7f0ca00700 1 --2- 172.21.15.118:0/82756701 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f7f08105510 0x7f7f0819aa90 unknown :-1 s=CLOSED pgs=153 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:15:07.274 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:07.273+0000 7f7f0ca00700 1 -- 172.21.15.118:0/82756701 >> 172.21.15.118:0/82756701 conn(0x7f7f080fc5b0 msgr2=0x7f7f08102320 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:15:07.274 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:07.274+0000 7f7f0ca00700 1 -- 172.21.15.118:0/82756701 shutdown_connections 2024-03-24T00:15:07.274 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:07.274+0000 7f7f0ca00700 1 -- 172.21.15.118:0/82756701 wait complete. 2024-03-24T00:15:07.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:15:07 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1d", "id": [8, 11]}]': finished 2024-03-24T00:15:07.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:15:07 smithi118 ceph-mon[31535]: osdmap e89: 12 total, 12 up, 12 in 2024-03-24T00:15:08.081 DEBUG:teuthology.orchestra.run.smithi118:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d79a5420-e971-11ee-95cd-87774f69a715 -- bash -c 'ceph orch host ls' 2024-03-24T00:15:08.203 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:15:07 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1d", "id": [8, 11]}]': finished 2024-03-24T00:15:08.203 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:15:07 smithi189 ceph-mon[36187]: osdmap e89: 12 total, 12 up, 12 in 2024-03-24T00:15:08.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:15:07 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd='[{"prefix": "osd pg-upmap-items", "format": "json", "pgid": "2.1d", "id": [8, 11]}]': finished 2024-03-24T00:15:08.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:15:07 smithi193 ceph-mon[36878]: osdmap e89: 12 total, 12 up, 12 in 2024-03-24T00:15:08.534 INFO:teuthology.orchestra.run.smithi118.stderr:Inferring config /var/lib/ceph/d79a5420-e971-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T00:15:08.945 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:15:08 smithi118 ceph-mon[31535]: from='client.25072 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T00:15:08.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:15:08 smithi118 ceph-mon[31535]: pgmap v380: 97 pgs: 97 active+clean; 645 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-03-24T00:15:08.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:15:08 smithi118 ceph-mon[31535]: osdmap e90: 12 total, 12 up, 12 in 2024-03-24T00:15:09.203 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:15:08 smithi189 ceph-mon[36187]: from='client.25072 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T00:15:09.203 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:15:08 smithi189 ceph-mon[36187]: pgmap v380: 97 pgs: 97 active+clean; 645 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-03-24T00:15:09.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:15:08 smithi189 ceph-mon[36187]: osdmap e90: 12 total, 12 up, 12 in 2024-03-24T00:15:09.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:15:08 smithi193 ceph-mon[36878]: from='client.25072 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T00:15:09.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:15:08 smithi193 ceph-mon[36878]: pgmap v380: 97 pgs: 97 active+clean; 645 KiB data, 348 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-03-24T00:15:09.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:15:08 smithi193 ceph-mon[36878]: osdmap e90: 12 total, 12 up, 12 in 2024-03-24T00:15:10.816 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:10.815+0000 7f6e6c400700 1 -- 172.21.15.118:0/3880551579 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6e641033d0 msgr2=0x7f6e64103830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:15:10.816 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:10.815+0000 7f6e6c400700 1 --2- 172.21.15.118:0/3880551579 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6e641033d0 0x7f6e64103830 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f6e500099c0 tx=0x7f6e50009cd0 comp rx=0 tx=0).stop 2024-03-24T00:15:10.816 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:10.816+0000 7f6e6c400700 1 -- 172.21.15.118:0/3880551579 shutdown_connections 2024-03-24T00:15:10.817 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:10.816+0000 7f6e6c400700 1 --2- 172.21.15.118:0/3880551579 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f6e64103d70 0x7f6e6410a620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:15:10.817 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:10.816+0000 7f6e6c400700 1 --2- 172.21.15.118:0/3880551579 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f6e641021d0 0x7f6e641025f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:15:10.817 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:10.816+0000 7f6e6c400700 1 --2- 172.21.15.118:0/3880551579 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6e641033d0 0x7f6e64103830 unknown :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:15:10.817 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:10.816+0000 7f6e6c400700 1 -- 172.21.15.118:0/3880551579 >> 172.21.15.118:0/3880551579 conn(0x7f6e640fd730 msgr2=0x7f6e640ffbb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:15:10.817 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:10.816+0000 7f6e6c400700 1 -- 172.21.15.118:0/3880551579 shutdown_connections 2024-03-24T00:15:10.817 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:10.816+0000 7f6e6c400700 1 -- 172.21.15.118:0/3880551579 wait complete. 2024-03-24T00:15:10.819 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:10.817+0000 7f6e6c400700 1 Processor -- start 2024-03-24T00:15:10.819 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:10.817+0000 7f6e6c400700 1 -- start start 2024-03-24T00:15:10.819 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:10.818+0000 7f6e6c400700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f6e641021d0 0x7f6e6419a660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:15:10.819 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:10.818+0000 7f6e6c400700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6e641033d0 0x7f6e6419aba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:15:10.819 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:10.818+0000 7f6e6c400700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f6e64103d70 0x7f6e641a1c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:15:10.819 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:10.818+0000 7f6e6c400700 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6e6410dde0 con 0x7f6e641021d0 2024-03-24T00:15:10.819 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:10.818+0000 7f6e6c400700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6e6410dc60 con 0x7f6e641033d0 2024-03-24T00:15:10.819 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:10.818+0000 7f6e6c400700 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6e6410df60 con 0x7f6e64103d70 2024-03-24T00:15:10.819 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:10.818+0000 7f6e69000700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6e641033d0 0x7f6e6419aba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:15:10.819 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:10.818+0000 7f6e69a00700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f6e641021d0 0x7f6e6419a660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:15:10.820 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:10.818+0000 7f6e69000700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6e641033d0 0x7f6e6419aba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.118:51278/0 (socket says 172.21.15.118:51278) 2024-03-24T00:15:10.820 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:10.818+0000 7f6e69000700 1 -- 172.21.15.118:0/1114160769 learned_addr learned my addr 172.21.15.118:0/1114160769 (peer_addr_for_me v2:172.21.15.118:0/0) 2024-03-24T00:15:10.820 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:10.818+0000 7f6e6a400700 1 --2- 172.21.15.118:0/1114160769 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f6e64103d70 0x7f6e641a1c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:15:10.820 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:10.818+0000 7f6e69000700 1 -- 172.21.15.118:0/1114160769 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f6e64103d70 msgr2=0x7f6e641a1c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:15:10.820 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:10.818+0000 7f6e69000700 1 --2- 172.21.15.118:0/1114160769 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f6e64103d70 0x7f6e641a1c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:15:10.820 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:10.818+0000 7f6e69000700 1 -- 172.21.15.118:0/1114160769 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f6e641021d0 msgr2=0x7f6e6419a660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:15:10.820 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:10.818+0000 7f6e69000700 1 --2- 172.21.15.118:0/1114160769 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f6e641021d0 0x7f6e6419a660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:15:10.820 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:10.818+0000 7f6e69000700 1 -- 172.21.15.118:0/1114160769 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6e641a22c0 con 0x7f6e641033d0 2024-03-24T00:15:10.820 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:10.818+0000 7f6e6a400700 1 --2- 172.21.15.118:0/1114160769 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f6e64103d70 0x7f6e641a1c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T00:15:10.820 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:10.819+0000 7f6e69000700 1 --2- 172.21.15.118:0/1114160769 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6e641033d0 0x7f6e6419aba0 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f6e5000fa00 tx=0x7f6e5000fa30 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:15:10.820 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:10.819+0000 7f6e62000700 1 -- 172.21.15.118:0/1114160769 <== mon.2 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6e50027070 con 0x7f6e641033d0 2024-03-24T00:15:10.820 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:10.819+0000 7f6e6c400700 1 -- 172.21.15.118:0/1114160769 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6e641a2550 con 0x7f6e641033d0 2024-03-24T00:15:10.820 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:10.819+0000 7f6e62000700 1 -- 172.21.15.118:0/1114160769 <== mon.2 v2:172.21.15.189:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f6e50022950 con 0x7f6e641033d0 2024-03-24T00:15:10.820 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:10.819+0000 7f6e62000700 1 -- 172.21.15.118:0/1114160769 <== mon.2 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6e50017650 con 0x7f6e641033d0 2024-03-24T00:15:10.821 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:10.819+0000 7f6e6c400700 1 -- 172.21.15.118:0/1114160769 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6e641a29a0 con 0x7f6e641033d0 2024-03-24T00:15:10.821 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:10.821+0000 7f6e6c400700 1 -- 172.21.15.118:0/1114160769 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6e64068e50 con 0x7f6e641033d0 2024-03-24T00:15:10.822 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:10.823+0000 7f6e62000700 1 -- 172.21.15.118:0/1114160769 <== mon.2 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 18) v1 ==== 137291+0+0 (secure 0 0 0) 0x7f6e5002a070 con 0x7f6e641033d0 2024-03-24T00:15:10.823 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:10.824+0000 7f6e62000700 1 --2- 172.21.15.118:0/1114160769 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7f6e4c0a2890 0x7f6e4c0a4d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:15:10.824 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:10.824+0000 7f6e62000700 1 -- 172.21.15.118:0/1114160769 <== mon.2 v2:172.21.15.189:3300/0 5 ==== osd_map(90..90 src has 1..90) v4 ==== 8277+0+0 (secure 0 0 0) 0x7f6e500cf840 con 0x7f6e641033d0 2024-03-24T00:15:10.824 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:10.824+0000 7f6e69a00700 1 --2- 172.21.15.118:0/1114160769 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7f6e4c0a2890 0x7f6e4c0a4d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:15:10.824 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:10.825+0000 7f6e69a00700 1 --2- 172.21.15.118:0/1114160769 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7f6e4c0a2890 0x7f6e4c0a4d50 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7f6e64103230 tx=0x7f6e58009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:15:10.827 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:10.828+0000 7f6e62000700 1 -- 172.21.15.118:0/1114160769 <== mon.2 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f6e50013070 con 0x7f6e641033d0 2024-03-24T00:15:11.062 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:11.063+0000 7f6e6c400700 1 -- 172.21.15.118:0/1114160769 --> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] -- mgr_command(tid 0: {"prefix": "orch host ls", "target": ["mon-mgr", ""]}) v1 -- 0x7f6e64075ae0 con 0x7f6e4c0a2890 2024-03-24T00:15:11.063 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:15:10 smithi118 ceph-mon[31535]: pgmap v382: 97 pgs: 97 active+clean; 645 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 6.0 KiB/s rd, 11 op/s; 0 B/s, 1 keys/s, 0 objects/s recovering 2024-03-24T00:15:11.065 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:11.065+0000 7f6e62000700 1 -- 172.21.15.118:0/1114160769 <== mgr.14152 v2:172.21.15.118:6800/2112912943 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+191 (secure 0 0 0) 0x7f6e64075ae0 con 0x7f6e4c0a2890 2024-03-24T00:15:11.065 INFO:teuthology.orchestra.run.smithi118.stdout:HOST ADDR LABELS STATUS 2024-03-24T00:15:11.065 INFO:teuthology.orchestra.run.smithi118.stdout:smithi118 172.21.15.118 2024-03-24T00:15:11.065 INFO:teuthology.orchestra.run.smithi118.stdout:smithi189 172.21.15.189 2024-03-24T00:15:11.065 INFO:teuthology.orchestra.run.smithi118.stdout:smithi193 172.21.15.193 2024-03-24T00:15:11.065 INFO:teuthology.orchestra.run.smithi118.stdout:3 hosts in cluster 2024-03-24T00:15:11.068 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:11.068+0000 7f6e6c400700 1 -- 172.21.15.118:0/1114160769 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7f6e4c0a2890 msgr2=0x7f6e4c0a4d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:15:11.068 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:11.068+0000 7f6e6c400700 1 --2- 172.21.15.118:0/1114160769 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7f6e4c0a2890 0x7f6e4c0a4d50 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7f6e64103230 tx=0x7f6e58009260 comp rx=0 tx=0).stop 2024-03-24T00:15:11.068 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:11.068+0000 7f6e6c400700 1 -- 172.21.15.118:0/1114160769 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6e641033d0 msgr2=0x7f6e6419aba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:15:11.068 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:11.068+0000 7f6e6c400700 1 --2- 172.21.15.118:0/1114160769 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6e641033d0 0x7f6e6419aba0 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 crypto rx=0x7f6e5000fa00 tx=0x7f6e5000fa30 comp rx=0 tx=0).stop 2024-03-24T00:15:11.069 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:11.069+0000 7f6e6c400700 1 -- 172.21.15.118:0/1114160769 shutdown_connections 2024-03-24T00:15:11.069 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:11.069+0000 7f6e6c400700 1 --2- 172.21.15.118:0/1114160769 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7f6e4c0a2890 0x7f6e4c0a4d50 unknown :-1 s=CLOSED pgs=228 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:15:11.069 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:11.069+0000 7f6e6c400700 1 --2- 172.21.15.118:0/1114160769 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f6e64103d70 0x7f6e641a1c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:15:11.069 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:11.069+0000 7f6e6c400700 1 --2- 172.21.15.118:0/1114160769 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f6e641021d0 0x7f6e6419a660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:15:11.069 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:11.069+0000 7f6e6c400700 1 --2- 172.21.15.118:0/1114160769 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6e641033d0 0x7f6e6419aba0 unknown :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:15:11.069 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:11.069+0000 7f6e6c400700 1 -- 172.21.15.118:0/1114160769 >> 172.21.15.118:0/1114160769 conn(0x7f6e640fd730 msgr2=0x7f6e640743c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:15:11.069 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:11.069+0000 7f6e6c400700 1 -- 172.21.15.118:0/1114160769 shutdown_connections 2024-03-24T00:15:11.069 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:11.069+0000 7f6e6c400700 1 -- 172.21.15.118:0/1114160769 wait complete. 2024-03-24T00:15:11.203 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:15:10 smithi189 ceph-mon[36187]: pgmap v382: 97 pgs: 97 active+clean; 645 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 6.0 KiB/s rd, 11 op/s; 0 B/s, 1 keys/s, 0 objects/s recovering 2024-03-24T00:15:11.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:15:10 smithi193 ceph-mon[36878]: pgmap v382: 97 pgs: 97 active+clean; 645 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 6.0 KiB/s rd, 11 op/s; 0 B/s, 1 keys/s, 0 objects/s recovering 2024-03-24T00:15:11.918 DEBUG:teuthology.orchestra.run.smithi118:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d79a5420-e971-11ee-95cd-87774f69a715 -- bash -c 'ceph orch device ls' 2024-03-24T00:15:12.390 INFO:teuthology.orchestra.run.smithi118.stderr:Inferring config /var/lib/ceph/d79a5420-e971-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T00:15:12.788 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:15:12 smithi118 ceph-mon[31535]: from='client.25067 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T00:15:12.788 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:15:12 smithi118 ceph-mon[31535]: pgmap v383: 97 pgs: 97 active+clean; 645 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 9 op/s; 0 B/s, 1 keys/s, 0 objects/s recovering 2024-03-24T00:15:13.203 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:15:12 smithi189 ceph-mon[36187]: from='client.25067 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T00:15:13.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:15:12 smithi189 ceph-mon[36187]: pgmap v383: 97 pgs: 97 active+clean; 645 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 9 op/s; 0 B/s, 1 keys/s, 0 objects/s recovering 2024-03-24T00:15:13.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:15:12 smithi193 ceph-mon[36878]: from='client.25067 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T00:15:13.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:15:12 smithi193 ceph-mon[36878]: pgmap v383: 97 pgs: 97 active+clean; 645 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 9 op/s; 0 B/s, 1 keys/s, 0 objects/s recovering 2024-03-24T00:15:14.716 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:14.715+0000 7f0766c00700 1 -- 172.21.15.118:0/1546767513 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f076010a0f0 msgr2=0x7f076010c560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:15:14.716 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:14.715+0000 7f0766c00700 1 --2- 172.21.15.118:0/1546767513 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f076010a0f0 0x7f076010c560 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f075800b0a0 tx=0x7f075800b3b0 comp rx=0 tx=0).stop 2024-03-24T00:15:14.716 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:14.715+0000 7f0766c00700 1 -- 172.21.15.118:0/1546767513 shutdown_connections 2024-03-24T00:15:14.716 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:14.715+0000 7f0766c00700 1 --2- 172.21.15.118:0/1546767513 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0760076140 0x7f07600745a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:15:14.716 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:14.715+0000 7f0766c00700 1 --2- 172.21.15.118:0/1546767513 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f076010a0f0 0x7f076010c560 unknown :-1 s=CLOSED pgs=154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:15:14.716 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:14.715+0000 7f0766c00700 1 --2- 172.21.15.118:0/1546767513 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f0760074b70 0x7f0760074ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:15:14.716 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:14.715+0000 7f0766c00700 1 -- 172.21.15.118:0/1546767513 >> 172.21.15.118:0/1546767513 conn(0x7f07600fd790 msgr2=0x7f07600ffbd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:15:14.716 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:14.716+0000 7f0766c00700 1 -- 172.21.15.118:0/1546767513 shutdown_connections 2024-03-24T00:15:14.716 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:14.716+0000 7f0766c00700 1 -- 172.21.15.118:0/1546767513 wait complete. 2024-03-24T00:15:14.717 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:14.717+0000 7f0766c00700 1 Processor -- start 2024-03-24T00:15:14.717 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:14.717+0000 7f0766c00700 1 -- start start 2024-03-24T00:15:14.718 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:14.718+0000 7f0766c00700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f0760074b70 0x7f076019a4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:15:14.718 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:14.718+0000 7f0766c00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0760076140 0x7f076019aa20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:15:14.718 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:14.718+0000 7f0766c00700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f076010a0f0 0x7f07601a1aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:15:14.718 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:14.718+0000 7f0766c00700 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f076010ffb0 con 0x7f0760074b70 2024-03-24T00:15:14.718 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:14.718+0000 7f0766c00700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f076010fe30 con 0x7f0760076140 2024-03-24T00:15:14.718 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:14.718+0000 7f0766c00700 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0760110130 con 0x7f076010a0f0 2024-03-24T00:15:14.718 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:14.718+0000 7f0757e00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0760076140 0x7f076019aa20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:15:14.718 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:14.718+0000 7f075fe00700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f0760074b70 0x7f076019a4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:15:14.718 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:14.718+0000 7f0764c00700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f076010a0f0 0x7f07601a1aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:15:14.719 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:14.718+0000 7f075fe00700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f0760074b70 0x7f076019a4e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.118:3300/0 says I am v2:172.21.15.118:42518/0 (socket says 172.21.15.118:42518) 2024-03-24T00:15:14.719 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:14.718+0000 7f0757e00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0760076140 0x7f076019aa20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.118:44082/0 (socket says 172.21.15.118:44082) 2024-03-24T00:15:14.719 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:14.718+0000 7f0764c00700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f076010a0f0 0x7f07601a1aa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.118:45070/0 (socket says 172.21.15.118:45070) 2024-03-24T00:15:14.719 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:14.718+0000 7f075fe00700 1 -- 172.21.15.118:0/3767474589 learned_addr learned my addr 172.21.15.118:0/3767474589 (peer_addr_for_me v2:172.21.15.118:0/0) 2024-03-24T00:15:14.719 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:14.718+0000 7f0764c00700 1 -- 172.21.15.118:0/3767474589 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0760076140 msgr2=0x7f076019aa20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:15:14.719 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:14.718+0000 7f0764c00700 1 --2- 172.21.15.118:0/3767474589 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0760076140 0x7f076019aa20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:15:14.719 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:14.718+0000 7f0764c00700 1 -- 172.21.15.118:0/3767474589 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f0760074b70 msgr2=0x7f076019a4e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:15:14.719 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:14.718+0000 7f0764c00700 1 --2- 172.21.15.118:0/3767474589 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f0760074b70 0x7f076019a4e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:15:14.719 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:14.718+0000 7f0764c00700 1 -- 172.21.15.118:0/3767474589 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f07601a2140 con 0x7f076010a0f0 2024-03-24T00:15:14.720 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:14.719+0000 7f0764c00700 1 --2- 172.21.15.118:0/3767474589 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f076010a0f0 0x7f07601a1aa0 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f075800b070 tx=0x7f0758004ce0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:15:14.720 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:14.719+0000 7f075d600700 1 -- 172.21.15.118:0/3767474589 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0758021070 con 0x7f076010a0f0 2024-03-24T00:15:14.720 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:14.719+0000 7f075d600700 1 -- 172.21.15.118:0/3767474589 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f0758007b80 con 0x7f076010a0f0 2024-03-24T00:15:14.720 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:14.719+0000 7f075d600700 1 -- 172.21.15.118:0/3767474589 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0758004800 con 0x7f076010a0f0 2024-03-24T00:15:14.720 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:14.720+0000 7f0766c00700 1 -- 172.21.15.118:0/3767474589 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f07601a23d0 con 0x7f076010a0f0 2024-03-24T00:15:14.720 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:14.720+0000 7f0766c00700 1 -- 172.21.15.118:0/3767474589 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f07601a2830 con 0x7f076010a0f0 2024-03-24T00:15:14.721 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:14.721+0000 7f0766c00700 1 -- 172.21.15.118:0/3767474589 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0760068e50 con 0x7f076010a0f0 2024-03-24T00:15:14.722 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:14.723+0000 7f075d600700 1 -- 172.21.15.118:0/3767474589 <== mon.1 v2:172.21.15.193:3300/0 4 ==== mgrmap(e 18) v1 ==== 137291+0+0 (secure 0 0 0) 0x7f0758003a00 con 0x7f076010a0f0 2024-03-24T00:15:14.724 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:14.725+0000 7f075d600700 1 --2- 172.21.15.118:0/3767474589 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7f07440a2910 0x7f07440a4dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:15:14.725 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:14.725+0000 7f075d600700 1 -- 172.21.15.118:0/3767474589 <== mon.1 v2:172.21.15.193:3300/0 5 ==== osd_map(90..90 src has 1..90) v4 ==== 8277+0+0 (secure 0 0 0) 0x7f075800f070 con 0x7f076010a0f0 2024-03-24T00:15:14.725 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:14.725+0000 7f075fe00700 1 --2- 172.21.15.118:0/3767474589 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7f07440a2910 0x7f07440a4dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:15:14.725 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:14.726+0000 7f075fe00700 1 --2- 172.21.15.118:0/3767474589 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7f07440a2910 0x7f07440a4dd0 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7f0750009d60 tx=0x7f0750009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:15:14.732 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:14.733+0000 7f075d600700 1 -- 172.21.15.118:0/3767474589 <== mon.1 v2:172.21.15.193:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f075802a020 con 0x7f076010a0f0 2024-03-24T00:15:14.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:15:14 smithi118 ceph-mon[31535]: pgmap v384: 97 pgs: 97 active+clean; 645 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s; 0 B/s, 1 keys/s, 0 objects/s recovering 2024-03-24T00:15:14.961 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:14.962+0000 7f0766c00700 1 -- 172.21.15.118:0/3767474589 --> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] -- mgr_command(tid 0: {"prefix": "orch device ls", "target": ["mon-mgr", ""]}) v1 -- 0x7f07600625d0 con 0x7f07440a2910 2024-03-24T00:15:14.964 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:14.965+0000 7f075d600700 1 -- 172.21.15.118:0/3767474589 <== mgr.14152 v2:172.21.15.118:6800/2112912943 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+532 (secure 0 0 0) 0x7f07600625d0 con 0x7f07440a2910 2024-03-24T00:15:14.965 INFO:teuthology.orchestra.run.smithi118.stdout:HOST PATH TYPE DEVICE ID SIZE AVAILABLE REFRESHED REJECT REASONS 2024-03-24T00:15:14.965 INFO:teuthology.orchestra.run.smithi118.stdout:smithi118 /dev/nvme0n1 ssd INTEL_SSDPED1D480GA_PHMB747500JS480DGN 447G No 3m ago Has a FileSystem, LVM detected 2024-03-24T00:15:14.965 INFO:teuthology.orchestra.run.smithi118.stdout:smithi189 /dev/nvme0n1 ssd INTEL_SSDPEDMD400G4_CVFT6232004L400BGN 372G No 3m ago Has a FileSystem, LVM detected 2024-03-24T00:15:14.965 INFO:teuthology.orchestra.run.smithi118.stdout:smithi193 /dev/nvme0n1 ssd INTEL_SSDPEDMD400G4_PHFT620500HB400BGN 372G No 3m ago Has a FileSystem, LVM detected 2024-03-24T00:15:14.967 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:14.968+0000 7f0766c00700 1 -- 172.21.15.118:0/3767474589 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7f07440a2910 msgr2=0x7f07440a4dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:15:14.967 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:14.968+0000 7f0766c00700 1 --2- 172.21.15.118:0/3767474589 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7f07440a2910 0x7f07440a4dd0 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7f0750009d60 tx=0x7f0750009260 comp rx=0 tx=0).stop 2024-03-24T00:15:14.967 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:14.968+0000 7f0766c00700 1 -- 172.21.15.118:0/3767474589 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f076010a0f0 msgr2=0x7f07601a1aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:15:14.967 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:14.968+0000 7f0766c00700 1 --2- 172.21.15.118:0/3767474589 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f076010a0f0 0x7f07601a1aa0 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f075800b070 tx=0x7f0758004ce0 comp rx=0 tx=0).stop 2024-03-24T00:15:14.968 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:14.968+0000 7f0766c00700 1 -- 172.21.15.118:0/3767474589 shutdown_connections 2024-03-24T00:15:14.968 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:14.968+0000 7f0766c00700 1 --2- 172.21.15.118:0/3767474589 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7f07440a2910 0x7f07440a4dd0 unknown :-1 s=CLOSED pgs=229 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:15:14.968 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:14.968+0000 7f0766c00700 1 --2- 172.21.15.118:0/3767474589 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f076010a0f0 0x7f07601a1aa0 unknown :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:15:14.968 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:14.968+0000 7f0766c00700 1 --2- 172.21.15.118:0/3767474589 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f0760074b70 0x7f076019a4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:15:14.968 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:14.968+0000 7f0766c00700 1 --2- 172.21.15.118:0/3767474589 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0760076140 0x7f076019aa20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:15:14.968 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:14.968+0000 7f0766c00700 1 -- 172.21.15.118:0/3767474589 >> 172.21.15.118:0/3767474589 conn(0x7f07600fd790 msgr2=0x7f0760106040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:15:14.968 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:14.968+0000 7f0766c00700 1 -- 172.21.15.118:0/3767474589 shutdown_connections 2024-03-24T00:15:14.969 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:14.968+0000 7f0766c00700 1 -- 172.21.15.118:0/3767474589 wait complete. 2024-03-24T00:15:15.203 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:15:14 smithi189 ceph-mon[36187]: pgmap v384: 97 pgs: 97 active+clean; 645 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s; 0 B/s, 1 keys/s, 0 objects/s recovering 2024-03-24T00:15:15.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:15:14 smithi193 ceph-mon[36878]: pgmap v384: 97 pgs: 97 active+clean; 645 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 4.1 KiB/s rd, 8 op/s; 0 B/s, 1 keys/s, 0 objects/s recovering 2024-03-24T00:15:15.883 INFO:teuthology.run_tasks:Running task cephadm.shell... 2024-03-24T00:15:15.893 INFO:tasks.cephadm:Running commands on role mon.a host ubuntu@smithi118.front.sepia.ceph.com 2024-03-24T00:15:15.893 DEBUG:teuthology.orchestra.run.smithi118:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d79a5420-e971-11ee-95cd-87774f69a715 -- bash -c 'ceph fs dump' 2024-03-24T00:15:16.367 INFO:teuthology.orchestra.run.smithi118.stderr:Inferring config /var/lib/ceph/d79a5420-e971-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T00:15:16.875 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:15:16 smithi118 ceph-mon[31535]: from='client.25084 -' entity='client.admin' cmd=[{"prefix": "orch device ls", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T00:15:16.876 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:15:16 smithi118 ceph-mon[31535]: pgmap v385: 97 pgs: 97 active+clean; 645 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 10 op/s; 0 B/s, 1 keys/s, 0 objects/s recovering 2024-03-24T00:15:17.203 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:15:16 smithi189 ceph-mon[36187]: from='client.25084 -' entity='client.admin' cmd=[{"prefix": "orch device ls", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T00:15:17.203 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:15:16 smithi189 ceph-mon[36187]: pgmap v385: 97 pgs: 97 active+clean; 645 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 10 op/s; 0 B/s, 1 keys/s, 0 objects/s recovering 2024-03-24T00:15:17.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:15:16 smithi193 ceph-mon[36878]: from='client.25084 -' entity='client.admin' cmd=[{"prefix": "orch device ls", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T00:15:17.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:15:16 smithi193 ceph-mon[36878]: pgmap v385: 97 pgs: 97 active+clean; 645 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 10 op/s; 0 B/s, 1 keys/s, 0 objects/s recovering 2024-03-24T00:15:18.616 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:18.615+0000 7ff735800700 1 -- 172.21.15.118:0/482806602 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff730103db0 msgr2=0x7ff73010a660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:15:18.618 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:18.615+0000 7ff735800700 1 --2- 172.21.15.118:0/482806602 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff730103db0 0x7ff73010a660 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 crypto rx=0x7ff72800b0d0 tx=0x7ff72800b3e0 comp rx=0 tx=0).stop 2024-03-24T00:15:18.618 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:18.616+0000 7ff735800700 1 -- 172.21.15.118:0/482806602 shutdown_connections 2024-03-24T00:15:18.618 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:18.616+0000 7ff735800700 1 --2- 172.21.15.118:0/482806602 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff730103db0 0x7ff73010a660 unknown :-1 s=CLOSED pgs=152 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:15:18.618 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:18.616+0000 7ff735800700 1 --2- 172.21.15.118:0/482806602 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7ff730102210 0x7ff730102630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:15:18.618 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:18.616+0000 7ff735800700 1 --2- 172.21.15.118:0/482806602 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7ff730103410 0x7ff730103870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:15:18.618 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:18.616+0000 7ff735800700 1 -- 172.21.15.118:0/482806602 >> 172.21.15.118:0/482806602 conn(0x7ff7300fd7b0 msgr2=0x7ff7300ffbf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:15:18.618 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:18.616+0000 7ff735800700 1 -- 172.21.15.118:0/482806602 shutdown_connections 2024-03-24T00:15:18.618 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:18.616+0000 7ff735800700 1 -- 172.21.15.118:0/482806602 wait complete. 2024-03-24T00:15:18.620 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:18.617+0000 7ff735800700 1 Processor -- start 2024-03-24T00:15:18.620 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:18.617+0000 7ff735800700 1 -- start start 2024-03-24T00:15:18.620 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:18.618+0000 7ff735800700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7ff730102210 0x7ff730198370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:15:18.620 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:18.618+0000 7ff735800700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7ff730103410 0x7ff7301988b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:15:18.620 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:18.618+0000 7ff735800700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff730103db0 0x7ff73019f930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:15:18.620 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:18.618+0000 7ff735800700 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff73010de20 con 0x7ff730103410 2024-03-24T00:15:18.620 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:18.618+0000 7ff735800700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff73010dca0 con 0x7ff730103db0 2024-03-24T00:15:18.621 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:18.618+0000 7ff735800700 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff73010dfa0 con 0x7ff730102210 2024-03-24T00:15:18.621 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:18.618+0000 7ff72ea00700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7ff730102210 0x7ff730198370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:15:18.621 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:18.618+0000 7ff72ea00700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7ff730102210 0x7ff730198370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.118:45090/0 (socket says 172.21.15.118:45090) 2024-03-24T00:15:18.621 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:18.618+0000 7ff72ea00700 1 -- 172.21.15.118:0/2757524837 learned_addr learned my addr 172.21.15.118:0/2757524837 (peer_addr_for_me v2:172.21.15.118:0/0) 2024-03-24T00:15:18.621 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:18.618+0000 7ff72f400700 1 --2- 172.21.15.118:0/2757524837 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff730103db0 0x7ff73019f930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:15:18.621 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:18.618+0000 7ff72e000700 1 --2- 172.21.15.118:0/2757524837 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7ff730103410 0x7ff7301988b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:15:18.621 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:18.619+0000 7ff72ea00700 1 -- 172.21.15.118:0/2757524837 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff730103db0 msgr2=0x7ff73019f930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:15:18.621 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:18.619+0000 7ff72ea00700 1 --2- 172.21.15.118:0/2757524837 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff730103db0 0x7ff73019f930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:15:18.621 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:18.619+0000 7ff72ea00700 1 -- 172.21.15.118:0/2757524837 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7ff730103410 msgr2=0x7ff7301988b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:15:18.621 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:18.619+0000 7ff72ea00700 1 --2- 172.21.15.118:0/2757524837 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7ff730103410 0x7ff7301988b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:15:18.621 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:18.619+0000 7ff72ea00700 1 -- 172.21.15.118:0/2757524837 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff73019ffd0 con 0x7ff730102210 2024-03-24T00:15:18.621 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:18.619+0000 7ff72ea00700 1 --2- 172.21.15.118:0/2757524837 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7ff730102210 0x7ff730198370 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7ff72000e9a0 tx=0x7ff72000ecb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:15:18.621 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:18.619+0000 7ff727400700 1 -- 172.21.15.118:0/2757524837 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff72000cbd0 con 0x7ff730102210 2024-03-24T00:15:18.621 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:18.619+0000 7ff727400700 1 -- 172.21.15.118:0/2757524837 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7ff720004500 con 0x7ff730102210 2024-03-24T00:15:18.622 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:18.619+0000 7ff735800700 1 -- 172.21.15.118:0/2757524837 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff7301a02c0 con 0x7ff730102210 2024-03-24T00:15:18.622 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:18.619+0000 7ff727400700 1 -- 172.21.15.118:0/2757524837 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff720003e00 con 0x7ff730102210 2024-03-24T00:15:18.622 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:18.619+0000 7ff735800700 1 -- 172.21.15.118:0/2757524837 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff7301a0800 con 0x7ff730102210 2024-03-24T00:15:18.622 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:18.621+0000 7ff735800700 1 -- 172.21.15.118:0/2757524837 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff730068e50 con 0x7ff730102210 2024-03-24T00:15:18.623 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:18.622+0000 7ff727400700 1 -- 172.21.15.118:0/2757524837 <== mon.1 v2:172.21.15.193:3300/0 4 ==== mgrmap(e 18) v1 ==== 137291+0+0 (secure 0 0 0) 0x7ff720003fe0 con 0x7ff730102210 2024-03-24T00:15:18.624 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:18.624+0000 7ff727400700 1 --2- 172.21.15.118:0/2757524837 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7ff7140a2890 0x7ff7140a4d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:15:18.625 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:18.624+0000 7ff727400700 1 -- 172.21.15.118:0/2757524837 <== mon.1 v2:172.21.15.193:3300/0 5 ==== osd_map(90..90 src has 1..90) v4 ==== 8277+0+0 (secure 0 0 0) 0x7ff7200cedd0 con 0x7ff730102210 2024-03-24T00:15:18.625 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:18.624+0000 7ff72e000700 1 --2- 172.21.15.118:0/2757524837 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7ff7140a2890 0x7ff7140a4d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:15:18.625 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:18.625+0000 7ff72e000700 1 --2- 172.21.15.118:0/2757524837 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7ff7140a2890 0x7ff7140a4d50 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7ff71c009bf0 tx=0x7ff71c009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:15:18.628 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:18.628+0000 7ff727400700 1 -- 172.21.15.118:0/2757524837 <== mon.1 v2:172.21.15.193:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7ff72009bde0 con 0x7ff730102210 2024-03-24T00:15:18.901 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:18.901+0000 7ff735800700 1 -- 172.21.15.118:0/2757524837 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7ff730062da0 con 0x7ff730102210 2024-03-24T00:15:18.902 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:18.901+0000 7ff727400700 1 -- 172.21.15.118:0/2757524837 <== mon.1 v2:172.21.15.193:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 27 v27) v1 ==== 76+0+3049 (secure 0 0 0) 0x7ff72009b6e0 con 0x7ff730102210 2024-03-24T00:15:18.903 INFO:teuthology.orchestra.run.smithi118.stdout:e27 2024-03-24T00:15:18.903 INFO:teuthology.orchestra.run.smithi118.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-03-24T00:15:18.903 INFO:teuthology.orchestra.run.smithi118.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2,11=minor log segments} 2024-03-24T00:15:18.904 INFO:teuthology.orchestra.run.smithi118.stdout:legacy client fscid: 1 2024-03-24T00:15:18.904 INFO:teuthology.orchestra.run.smithi118.stdout: 2024-03-24T00:15:18.904 INFO:teuthology.orchestra.run.smithi118.stdout:Filesystem 'cephfs' (1) 2024-03-24T00:15:18.904 INFO:teuthology.orchestra.run.smithi118.stdout:fs_name cephfs 2024-03-24T00:15:18.904 INFO:teuthology.orchestra.run.smithi118.stdout:epoch 27 2024-03-24T00:15:18.904 INFO:teuthology.orchestra.run.smithi118.stdout:flags 32 joinable allow_snaps allow_multimds_snaps allow_standby_replay 2024-03-24T00:15:18.904 INFO:teuthology.orchestra.run.smithi118.stdout:created 2024-03-24T00:13:01.782474+0000 2024-03-24T00:15:18.904 INFO:teuthology.orchestra.run.smithi118.stdout:modified 2024-03-24T00:13:09.335361+0000 2024-03-24T00:15:18.904 INFO:teuthology.orchestra.run.smithi118.stdout:tableserver 0 2024-03-24T00:15:18.904 INFO:teuthology.orchestra.run.smithi118.stdout:root 0 2024-03-24T00:15:18.904 INFO:teuthology.orchestra.run.smithi118.stdout:session_timeout 300 2024-03-24T00:15:18.904 INFO:teuthology.orchestra.run.smithi118.stdout:session_autoclose 300 2024-03-24T00:15:18.904 INFO:teuthology.orchestra.run.smithi118.stdout:max_file_size 1099511627776 2024-03-24T00:15:18.904 INFO:teuthology.orchestra.run.smithi118.stdout:max_xattr_size 65536 2024-03-24T00:15:18.904 INFO:teuthology.orchestra.run.smithi118.stdout:required_client_features {} 2024-03-24T00:15:18.905 INFO:teuthology.orchestra.run.smithi118.stdout:last_failure 0 2024-03-24T00:15:18.905 INFO:teuthology.orchestra.run.smithi118.stdout:last_failure_osd_epoch 0 2024-03-24T00:15:18.905 INFO:teuthology.orchestra.run.smithi118.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2,11=minor log segments} 2024-03-24T00:15:18.905 INFO:teuthology.orchestra.run.smithi118.stdout:max_mds 5 2024-03-24T00:15:18.905 INFO:teuthology.orchestra.run.smithi118.stdout:in 0,1,2,3,4 2024-03-24T00:15:18.905 INFO:teuthology.orchestra.run.smithi118.stdout:up {0=24569,1=24550,2=24511,3=24499,4=24488} 2024-03-24T00:15:18.905 INFO:teuthology.orchestra.run.smithi118.stdout:failed 2024-03-24T00:15:18.905 INFO:teuthology.orchestra.run.smithi118.stdout:damaged 2024-03-24T00:15:18.905 INFO:teuthology.orchestra.run.smithi118.stdout:stopped 2024-03-24T00:15:18.905 INFO:teuthology.orchestra.run.smithi118.stdout:data_pools [3] 2024-03-24T00:15:18.905 INFO:teuthology.orchestra.run.smithi118.stdout:metadata_pool 2 2024-03-24T00:15:18.905 INFO:teuthology.orchestra.run.smithi118.stdout:inline_data disabled 2024-03-24T00:15:18.905 INFO:teuthology.orchestra.run.smithi118.stdout:balancer 2024-03-24T00:15:18.905 INFO:teuthology.orchestra.run.smithi118.stdout:bal_rank_mask -1 2024-03-24T00:15:18.905 INFO:teuthology.orchestra.run.smithi118.stdout:standby_count_wanted 0 2024-03-24T00:15:18.905 INFO:teuthology.orchestra.run.smithi118.stdout:qdb_cluster leader: 24569 members: 24511,24550,24488,24499,24569 2024-03-24T00:15:18.906 INFO:teuthology.orchestra.run.smithi118.stdout:[mds.b{0:24569} state up:active seq 3 addr [v2:172.21.15.189:6838/4074484168,v1:172.21.15.189:6839/4074484168] compat {c=[1],r=[1],i=[fff]}] 2024-03-24T00:15:18.906 INFO:teuthology.orchestra.run.smithi118.stdout:[mds.i{0:24532} state up:standby-replay seq 1 addr [v2:172.21.15.193:6838/2126842603,v1:172.21.15.193:6839/2126842603] compat {c=[1],r=[1],i=[fff]}] 2024-03-24T00:15:18.906 INFO:teuthology.orchestra.run.smithi118.stdout:[mds.g{1:24550} state up:active seq 6 addr [v2:172.21.15.118:6838/3285788503,v1:172.21.15.118:6839/3285788503] compat {c=[1],r=[1],i=[fff]}] 2024-03-24T00:15:18.906 INFO:teuthology.orchestra.run.smithi118.stdout:[mds.h{1:24515} state up:standby-replay seq 1 addr [v2:172.21.15.189:6836/3450187137,v1:172.21.15.189:6837/3450187137] compat {c=[1],r=[1],i=[fff]}] 2024-03-24T00:15:18.906 INFO:teuthology.orchestra.run.smithi118.stdout:[mds.c{2:24511} state up:active seq 8 addr [v2:172.21.15.193:6836/2921520053,v1:172.21.15.193:6837/2921520053] compat {c=[1],r=[1],i=[fff]}] 2024-03-24T00:15:18.906 INFO:teuthology.orchestra.run.smithi118.stdout:[mds.k{2:24440} state up:standby-replay seq 1 addr [v2:172.21.15.189:6832/2740024078,v1:172.21.15.189:6833/2740024078] compat {c=[1],r=[1],i=[fff]}] 2024-03-24T00:15:18.906 INFO:teuthology.orchestra.run.smithi118.stdout:[mds.f{3:24499} state up:active seq 9 addr [v2:172.21.15.193:6834/2055310541,v1:172.21.15.193:6835/2055310541] compat {c=[1],r=[1],i=[fff]}] 2024-03-24T00:15:18.906 INFO:teuthology.orchestra.run.smithi118.stdout:[mds.l{3:24439} state up:standby-replay seq 1 addr [v2:172.21.15.193:6832/1023428412,v1:172.21.15.193:6833/1023428412] compat {c=[1],r=[1],i=[fff]}] 2024-03-24T00:15:18.906 INFO:teuthology.orchestra.run.smithi118.stdout:[mds.e{4:24488} state up:active seq 10 addr [v2:172.21.15.189:6834/599473205,v1:172.21.15.189:6835/599473205] compat {c=[1],r=[1],i=[fff]}] 2024-03-24T00:15:18.906 INFO:teuthology.orchestra.run.smithi118.stdout:[mds.a{4:14469} state up:standby-replay seq 1 addr [v2:172.21.15.118:6840/3040946537,v1:172.21.15.118:6841/3040946537] compat {c=[1],r=[1],i=[fff]}] 2024-03-24T00:15:18.906 INFO:teuthology.orchestra.run.smithi118.stdout: 2024-03-24T00:15:18.906 INFO:teuthology.orchestra.run.smithi118.stdout: 2024-03-24T00:15:18.906 INFO:teuthology.orchestra.run.smithi118.stdout:Standby daemons: 2024-03-24T00:15:18.906 INFO:teuthology.orchestra.run.smithi118.stdout: 2024-03-24T00:15:18.907 INFO:teuthology.orchestra.run.smithi118.stdout:[mds.j{-1:14385} state up:standby seq 1 addr [v2:172.21.15.118:6834/3194741373,v1:172.21.15.118:6835/3194741373] compat {c=[1],r=[1],i=[fff]}] 2024-03-24T00:15:18.907 INFO:teuthology.orchestra.run.smithi118.stdout:[mds.d{-1:14397} state up:standby seq 1 addr [v2:172.21.15.118:6836/1374462956,v1:172.21.15.118:6837/1374462956] compat {c=[1],r=[1],i=[fff]}] 2024-03-24T00:15:18.907 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:18.904+0000 7ff735800700 1 -- 172.21.15.118:0/2757524837 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7ff7140a2890 msgr2=0x7ff7140a4d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:15:18.908 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:18.904+0000 7ff735800700 1 --2- 172.21.15.118:0/2757524837 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7ff7140a2890 0x7ff7140a4d50 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7ff71c009bf0 tx=0x7ff71c009260 comp rx=0 tx=0).stop 2024-03-24T00:15:18.908 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:18.905+0000 7ff735800700 1 -- 172.21.15.118:0/2757524837 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7ff730102210 msgr2=0x7ff730198370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:15:18.908 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:18.905+0000 7ff735800700 1 --2- 172.21.15.118:0/2757524837 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7ff730102210 0x7ff730198370 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7ff72000e9a0 tx=0x7ff72000ecb0 comp rx=0 tx=0).stop 2024-03-24T00:15:18.908 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:18.905+0000 7ff735800700 1 -- 172.21.15.118:0/2757524837 shutdown_connections 2024-03-24T00:15:18.908 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:18.905+0000 7ff735800700 1 --2- 172.21.15.118:0/2757524837 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff730103db0 0x7ff73019f930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:15:18.908 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:18.905+0000 7ff735800700 1 --2- 172.21.15.118:0/2757524837 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7ff7140a2890 0x7ff7140a4d50 unknown :-1 s=CLOSED pgs=230 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:15:18.908 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:18.905+0000 7ff735800700 1 --2- 172.21.15.118:0/2757524837 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7ff730102210 0x7ff730198370 unknown :-1 s=CLOSED pgs=156 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:15:18.908 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:18.905+0000 7ff735800700 1 --2- 172.21.15.118:0/2757524837 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7ff730103410 0x7ff7301988b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:15:18.908 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:18.905+0000 7ff735800700 1 -- 172.21.15.118:0/2757524837 >> 172.21.15.118:0/2757524837 conn(0x7ff7300fd7b0 msgr2=0x7ff730106640 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:15:18.908 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:18.905+0000 7ff735800700 1 -- 172.21.15.118:0/2757524837 shutdown_connections 2024-03-24T00:15:18.908 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:18.905+0000 7ff735800700 1 -- 172.21.15.118:0/2757524837 wait complete. 2024-03-24T00:15:18.908 INFO:teuthology.orchestra.run.smithi118.stderr:dumped fsmap epoch 27 2024-03-24T00:15:19.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:15:18 smithi118 ceph-mon[31535]: pgmap v386: 97 pgs: 97 active+clean; 645 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 4.8 KiB/s rd, 9 op/s; 0 B/s, 0 keys/s, 0 objects/s recovering 2024-03-24T00:15:19.203 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:15:18 smithi189 ceph-mon[36187]: pgmap v386: 97 pgs: 97 active+clean; 645 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 4.8 KiB/s rd, 9 op/s; 0 B/s, 0 keys/s, 0 objects/s recovering 2024-03-24T00:15:19.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:15:18 smithi193 ceph-mon[36878]: pgmap v386: 97 pgs: 97 active+clean; 645 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 4.8 KiB/s rd, 9 op/s; 0 B/s, 0 keys/s, 0 objects/s recovering 2024-03-24T00:15:19.794 DEBUG:teuthology.orchestra.run.smithi118:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid d79a5420-e971-11ee-95cd-87774f69a715 -- bash -c 'ceph osd dump' 2024-03-24T00:15:20.195 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:15:19 smithi118 ceph-mon[31535]: from='client.? 172.21.15.118:0/2757524837' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-03-24T00:15:20.203 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:15:19 smithi189 ceph-mon[36187]: from='client.? 172.21.15.118:0/2757524837' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-03-24T00:15:20.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:15:19 smithi193 ceph-mon[36878]: from='client.? 172.21.15.118:0/2757524837' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-03-24T00:15:20.253 INFO:teuthology.orchestra.run.smithi118.stderr:Inferring config /var/lib/ceph/d79a5420-e971-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T00:15:21.112 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:15:20 smithi193 ceph-mon[36878]: pgmap v387: 97 pgs: 97 active+clean; 645 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 5.3 KiB/s rd, 10 op/s; 0 B/s, 0 keys/s, 0 objects/s recovering 2024-03-24T00:15:21.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:15:20 smithi118 ceph-mon[31535]: pgmap v387: 97 pgs: 97 active+clean; 645 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 5.3 KiB/s rd, 10 op/s; 0 B/s, 0 keys/s, 0 objects/s recovering 2024-03-24T00:15:21.203 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:15:20 smithi189 ceph-mon[36187]: pgmap v387: 97 pgs: 97 active+clean; 645 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 5.3 KiB/s rd, 10 op/s; 0 B/s, 0 keys/s, 0 objects/s recovering 2024-03-24T00:15:22.519 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:22.517+0000 7fe104a00700 1 -- 172.21.15.118:0/3227614281 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe100109fd0 msgr2=0x7fe10010c460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:15:22.519 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:22.517+0000 7fe104a00700 1 --2- 172.21.15.118:0/3227614281 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe100109fd0 0x7fe10010c460 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 crypto rx=0x7fe0f800b0a0 tx=0x7fe0f800b3b0 comp rx=0 tx=0).stop 2024-03-24T00:15:22.519 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:22.517+0000 7fe104a00700 1 -- 172.21.15.118:0/3227614281 shutdown_connections 2024-03-24T00:15:22.519 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:22.517+0000 7fe104a00700 1 --2- 172.21.15.118:0/3227614281 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe100109fd0 0x7fe10010c460 unknown :-1 s=CLOSED pgs=153 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:15:22.519 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:22.517+0000 7fe104a00700 1 --2- 172.21.15.118:0/3227614281 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fe1000761c0 0x7fe100074620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:15:22.519 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:22.517+0000 7fe104a00700 1 --2- 172.21.15.118:0/3227614281 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fe100074bf0 0x7fe100075070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:15:22.519 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:22.517+0000 7fe104a00700 1 -- 172.21.15.118:0/3227614281 >> 172.21.15.118:0/3227614281 conn(0x7fe1000fd6a0 msgr2=0x7fe1000ffae0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:15:22.520 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:22.517+0000 7fe104a00700 1 -- 172.21.15.118:0/3227614281 shutdown_connections 2024-03-24T00:15:22.520 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:22.517+0000 7fe104a00700 1 -- 172.21.15.118:0/3227614281 wait complete. 2024-03-24T00:15:22.520 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:22.518+0000 7fe104a00700 1 Processor -- start 2024-03-24T00:15:22.520 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:22.518+0000 7fe104a00700 1 -- start start 2024-03-24T00:15:22.521 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:22.519+0000 7fe104a00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe100074bf0 0x7fe10019a4b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:15:22.521 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:22.519+0000 7fe104a00700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fe1000761c0 0x7fe10019a9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:15:22.521 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:22.519+0000 7fe104a00700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fe100109fd0 0x7fe1001a1a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:15:22.521 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:22.519+0000 7fe104a00700 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe10010ff30 con 0x7fe100109fd0 2024-03-24T00:15:22.521 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:22.519+0000 7fe104a00700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe10010fdb0 con 0x7fe100074bf0 2024-03-24T00:15:22.522 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:22.519+0000 7fe104a00700 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe1001100b0 con 0x7fe1000761c0 2024-03-24T00:15:22.522 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:22.519+0000 7fe0fea00700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fe1000761c0 0x7fe10019a9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:15:22.522 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:22.519+0000 7fe0ff400700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe100074bf0 0x7fe10019a4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:15:22.522 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:22.519+0000 7fe0fea00700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fe1000761c0 0x7fe10019a9f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.118:45106/0 (socket says 172.21.15.118:45106) 2024-03-24T00:15:22.522 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:22.519+0000 7fe0fea00700 1 -- 172.21.15.118:0/820801929 learned_addr learned my addr 172.21.15.118:0/820801929 (peer_addr_for_me v2:172.21.15.118:0/0) 2024-03-24T00:15:22.522 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:22.519+0000 7fe0ff400700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe100074bf0 0x7fe10019a4b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.118:44110/0 (socket says 172.21.15.118:44110) 2024-03-24T00:15:22.522 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:22.519+0000 7fe0ffe00700 1 --2- 172.21.15.118:0/820801929 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fe100109fd0 0x7fe1001a1a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:15:22.522 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:22.519+0000 7fe0fea00700 1 -- 172.21.15.118:0/820801929 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe100074bf0 msgr2=0x7fe10019a4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:15:22.522 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:22.519+0000 7fe0fea00700 1 --2- 172.21.15.118:0/820801929 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe100074bf0 0x7fe10019a4b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:15:22.522 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:22.519+0000 7fe0fea00700 1 -- 172.21.15.118:0/820801929 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fe100109fd0 msgr2=0x7fe1001a1a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:15:22.522 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:22.519+0000 7fe0fea00700 1 --2- 172.21.15.118:0/820801929 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fe100109fd0 0x7fe1001a1a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:15:22.522 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:22.519+0000 7fe0fea00700 1 -- 172.21.15.118:0/820801929 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe1001a2110 con 0x7fe1000761c0 2024-03-24T00:15:22.523 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:22.519+0000 7fe0ffe00700 1 --2- 172.21.15.118:0/820801929 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fe100109fd0 0x7fe1001a1a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T00:15:22.523 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:22.520+0000 7fe0fea00700 1 --2- 172.21.15.118:0/820801929 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fe1000761c0 0x7fe10019a9f0 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7fe0ec00e9a0 tx=0x7fe0ec00ecb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:15:22.523 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:22.520+0000 7fe0f7e00700 1 -- 172.21.15.118:0/820801929 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe0ec00ca10 con 0x7fe1000761c0 2024-03-24T00:15:22.523 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:22.520+0000 7fe0f7e00700 1 -- 172.21.15.118:0/820801929 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fe0ec00cbb0 con 0x7fe1000761c0 2024-03-24T00:15:22.523 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:22.520+0000 7fe104a00700 1 -- 172.21.15.118:0/820801929 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe1001a2400 con 0x7fe1000761c0 2024-03-24T00:15:22.523 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:22.520+0000 7fe0f7e00700 1 -- 172.21.15.118:0/820801929 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe0ec018680 con 0x7fe1000761c0 2024-03-24T00:15:22.523 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:22.520+0000 7fe104a00700 1 -- 172.21.15.118:0/820801929 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe1001a2940 con 0x7fe1000761c0 2024-03-24T00:15:22.523 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:22.521+0000 7fe104a00700 1 -- 172.21.15.118:0/820801929 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe100068e50 con 0x7fe1000761c0 2024-03-24T00:15:22.524 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:22.523+0000 7fe0f7e00700 1 -- 172.21.15.118:0/820801929 <== mon.1 v2:172.21.15.193:3300/0 4 ==== mgrmap(e 18) v1 ==== 137291+0+0 (secure 0 0 0) 0x7fe0ec01f020 con 0x7fe1000761c0 2024-03-24T00:15:22.525 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:22.524+0000 7fe0f7e00700 1 --2- 172.21.15.118:0/820801929 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7fe0e40a2890 0x7fe0e40a4d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:15:22.525 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:22.525+0000 7fe0f7e00700 1 -- 172.21.15.118:0/820801929 <== mon.1 v2:172.21.15.193:3300/0 5 ==== osd_map(90..90 src has 1..90) v4 ==== 8277+0+0 (secure 0 0 0) 0x7fe0ec0cea10 con 0x7fe1000761c0 2024-03-24T00:15:22.525 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:22.525+0000 7fe0ff400700 1 --2- 172.21.15.118:0/820801929 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7fe0e40a2890 0x7fe0e40a4d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:15:22.526 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:22.525+0000 7fe0ff400700 1 --2- 172.21.15.118:0/820801929 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7fe0e40a2890 0x7fe0e40a4d50 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7fe1000765a0 tx=0x7fe0f0009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:15:22.533 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:22.533+0000 7fe0f7e00700 1 -- 172.21.15.118:0/820801929 <== mon.1 v2:172.21.15.193:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7fe0ec01e2c0 con 0x7fe1000761c0 2024-03-24T00:15:22.782 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:22.781+0000 7fe104a00700 1 -- 172.21.15.118:0/820801929 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_command({"prefix": "osd dump"} v 0) v1 -- 0x7fe100062da0 con 0x7fe1000761c0 2024-03-24T00:15:22.782 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:22.782+0000 7fe0f7e00700 1 -- 172.21.15.118:0/820801929 <== mon.1 v2:172.21.15.193:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump"}]=0 v90) v1 ==== 56+0+5369 (secure 0 0 0) 0x7fe0ec0b9e20 con 0x7fe1000761c0 2024-03-24T00:15:22.783 INFO:teuthology.orchestra.run.smithi118.stdout:epoch 90 2024-03-24T00:15:22.783 INFO:teuthology.orchestra.run.smithi118.stdout:fsid d79a5420-e971-11ee-95cd-87774f69a715 2024-03-24T00:15:22.783 INFO:teuthology.orchestra.run.smithi118.stdout:created 2024-03-24T00:03:58.648524+0000 2024-03-24T00:15:22.784 INFO:teuthology.orchestra.run.smithi118.stdout:modified 2024-03-24T00:15:07.721960+0000 2024-03-24T00:15:22.784 INFO:teuthology.orchestra.run.smithi118.stdout:flags sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit 2024-03-24T00:15:22.784 INFO:teuthology.orchestra.run.smithi118.stdout:crush_version 26 2024-03-24T00:15:22.784 INFO:teuthology.orchestra.run.smithi118.stdout:full_ratio 0.95 2024-03-24T00:15:22.784 INFO:teuthology.orchestra.run.smithi118.stdout:backfillfull_ratio 0.9 2024-03-24T00:15:22.784 INFO:teuthology.orchestra.run.smithi118.stdout:nearfull_ratio 0.85 2024-03-24T00:15:22.784 INFO:teuthology.orchestra.run.smithi118.stdout:require_min_compat_client luminous 2024-03-24T00:15:22.784 INFO:teuthology.orchestra.run.smithi118.stdout:min_compat_client luminous 2024-03-24T00:15:22.784 INFO:teuthology.orchestra.run.smithi118.stdout:require_osd_release squid 2024-03-24T00:15:22.784 INFO:teuthology.orchestra.run.smithi118.stdout:stretch_mode_enabled false 2024-03-24T00:15:22.784 INFO:teuthology.orchestra.run.smithi118.stdout:pool 1 '.mgr' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 1 pgp_num 1 autoscale_mode off last_change 21 flags hashpspool stripe_width 0 pg_num_max 32 pg_num_min 1 application mgr read_balance_score 12.00 2024-03-24T00:15:22.784 INFO:teuthology.orchestra.run.smithi118.stdout:pool 2 'cephfs_metadata' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 32 pgp_num 32 autoscale_mode off last_change 85 flags hashpspool stripe_width 0 pg_autoscale_bias 4 pg_num_min 16 recovery_priority 5 application cephfs read_balance_score 1.88 2024-03-24T00:15:22.784 INFO:teuthology.orchestra.run.smithi118.stdout:pool 3 'cephfs_data' replicated size 3 min_size 2 crush_rule 0 object_hash rjenkins pg_num 64 pgp_num 64 autoscale_mode off last_change 86 flags hashpspool stripe_width 0 pg_num_min 64 target_size_ratio 0.9 application cephfs read_balance_score 1.50 2024-03-24T00:15:22.784 INFO:teuthology.orchestra.run.smithi118.stdout:max_osd 12 2024-03-24T00:15:22.785 INFO:teuthology.orchestra.run.smithi118.stdout:osd.0 up in weight 1 up_from 8 up_thru 87 down_at 0 last_clean_interval [0,0) [v2:172.21.15.118:6802/557690061,v1:172.21.15.118:6803/557690061] [v2:172.21.15.118:6804/557690061,v1:172.21.15.118:6805/557690061] exists,up c9590426-3901-4f68-a5ef-d57aa8fbac92 2024-03-24T00:15:22.785 INFO:teuthology.orchestra.run.smithi118.stdout:osd.1 up in weight 1 up_from 13 up_thru 87 down_at 0 last_clean_interval [0,0) [v2:172.21.15.189:6800/4203029485,v1:172.21.15.189:6801/4203029485] [v2:172.21.15.189:6802/4203029485,v1:172.21.15.189:6803/4203029485] exists,up fc7f827e-ecf6-4661-aad0-7a22951892e0 2024-03-24T00:15:22.785 INFO:teuthology.orchestra.run.smithi118.stdout:osd.2 up in weight 1 up_from 18 up_thru 84 down_at 0 last_clean_interval [0,0) [v2:172.21.15.193:6800/3699076971,v1:172.21.15.193:6801/3699076971] [v2:172.21.15.193:6802/3699076971,v1:172.21.15.193:6803/3699076971] exists,up e73ccfe9-0e5b-4734-a154-a2f0d829fcfb 2024-03-24T00:15:22.785 INFO:teuthology.orchestra.run.smithi118.stdout:osd.3 up in weight 1 up_from 26 up_thru 84 down_at 0 last_clean_interval [0,0) [v2:172.21.15.118:6810/4059617010,v1:172.21.15.118:6811/4059617010] [v2:172.21.15.118:6812/4059617010,v1:172.21.15.118:6813/4059617010] exists,up 63fed43b-43ce-4031-8957-7fe94725c1ed 2024-03-24T00:15:22.785 INFO:teuthology.orchestra.run.smithi118.stdout:osd.4 up in weight 1 up_from 34 up_thru 84 down_at 0 last_clean_interval [0,0) [v2:172.21.15.189:6808/4075247068,v1:172.21.15.189:6809/4075247068] [v2:172.21.15.189:6810/4075247068,v1:172.21.15.189:6811/4075247068] exists,up 57377f66-c3b4-43c1-8e8e-bc873672906d 2024-03-24T00:15:22.785 INFO:teuthology.orchestra.run.smithi118.stdout:osd.5 up in weight 1 up_from 39 up_thru 84 down_at 0 last_clean_interval [0,0) [v2:172.21.15.193:6808/2920363402,v1:172.21.15.193:6809/2920363402] [v2:172.21.15.193:6810/2920363402,v1:172.21.15.193:6811/2920363402] exists,up 91970976-2193-4224-81c0-dc96cb022c85 2024-03-24T00:15:22.785 INFO:teuthology.orchestra.run.smithi118.stdout:osd.6 up in weight 1 up_from 47 up_thru 84 down_at 0 last_clean_interval [0,0) [v2:172.21.15.118:6818/431354576,v1:172.21.15.118:6819/431354576] [v2:172.21.15.118:6820/431354576,v1:172.21.15.118:6821/431354576] exists,up 0b77b196-8674-41c6-ad3d-726b5d01c8fd 2024-03-24T00:15:22.785 INFO:teuthology.orchestra.run.smithi118.stdout:osd.7 up in weight 1 up_from 52 up_thru 87 down_at 0 last_clean_interval [0,0) [v2:172.21.15.189:6816/3667611284,v1:172.21.15.189:6817/3667611284] [v2:172.21.15.189:6818/3667611284,v1:172.21.15.189:6819/3667611284] exists,up 1fbbd38f-4e8e-4d4c-8e98-66f6862bc4bd 2024-03-24T00:15:22.785 INFO:teuthology.orchestra.run.smithi118.stdout:osd.8 up in weight 1 up_from 60 up_thru 84 down_at 0 last_clean_interval [0,0) [v2:172.21.15.193:6816/3853800910,v1:172.21.15.193:6817/3853800910] [v2:172.21.15.193:6818/3853800910,v1:172.21.15.193:6819/3853800910] exists,up 6462c2b5-e4b3-4b0b-bcfa-4de2c89e93e0 2024-03-24T00:15:22.785 INFO:teuthology.orchestra.run.smithi118.stdout:osd.9 up in weight 1 up_from 68 up_thru 87 down_at 0 last_clean_interval [0,0) [v2:172.21.15.118:6826/3248293936,v1:172.21.15.118:6827/3248293936] [v2:172.21.15.118:6828/3248293936,v1:172.21.15.118:6829/3248293936] exists,up 54be277f-3596-422f-9de5-9005a8571c6f 2024-03-24T00:15:22.785 INFO:teuthology.orchestra.run.smithi118.stdout:osd.10 up in weight 1 up_from 73 up_thru 87 down_at 0 last_clean_interval [0,0) [v2:172.21.15.189:6824/2112195760,v1:172.21.15.189:6825/2112195760] [v2:172.21.15.189:6826/2112195760,v1:172.21.15.189:6827/2112195760] exists,up 7ba69d69-ac41-44ce-bc39-c049a8ca5900 2024-03-24T00:15:22.785 INFO:teuthology.orchestra.run.smithi118.stdout:osd.11 up in weight 1 up_from 81 up_thru 89 down_at 0 last_clean_interval [0,0) [v2:172.21.15.193:6824/3880515510,v1:172.21.15.193:6825/3880515510] [v2:172.21.15.193:6826/3880515510,v1:172.21.15.193:6827/3880515510] exists,up 186f0b18-6b22-4eae-83b7-b21bbcce0817 2024-03-24T00:15:22.785 INFO:teuthology.orchestra.run.smithi118.stdout:pg_upmap_items 2.a [2,11] 2024-03-24T00:15:22.785 INFO:teuthology.orchestra.run.smithi118.stdout:pg_upmap_items 2.14 [2,4] 2024-03-24T00:15:22.786 INFO:teuthology.orchestra.run.smithi118.stdout:pg_upmap_items 2.1d [8,11] 2024-03-24T00:15:22.786 INFO:teuthology.orchestra.run.smithi118.stdout:pg_upmap_items 2.1f [8,1] 2024-03-24T00:15:22.786 INFO:teuthology.orchestra.run.smithi118.stdout:pg_upmap_items 3.11 [4,8,5,9] 2024-03-24T00:15:22.786 INFO:teuthology.orchestra.run.smithi118.stdout:pg_upmap_items 3.13 [4,9] 2024-03-24T00:15:22.786 INFO:teuthology.orchestra.run.smithi118.stdout:pg_upmap_items 3.14 [4,6] 2024-03-24T00:15:22.786 INFO:teuthology.orchestra.run.smithi118.stdout:pg_upmap_items 3.31 [4,7] 2024-03-24T00:15:22.786 INFO:teuthology.orchestra.run.smithi118.stdout:pg_upmap_items 3.38 [3,9] 2024-03-24T00:15:22.786 INFO:teuthology.orchestra.run.smithi118.stdout:pg_upmap_items 3.39 [5,8] 2024-03-24T00:15:22.786 INFO:teuthology.orchestra.run.smithi118.stdout:blocklist 172.21.15.118:6801/389249272 expires 2024-03-25T00:05:05.458796+0000 2024-03-24T00:15:22.786 INFO:teuthology.orchestra.run.smithi118.stdout:blocklist 172.21.15.118:6800/389249272 expires 2024-03-25T00:05:05.458796+0000 2024-03-24T00:15:22.786 INFO:teuthology.orchestra.run.smithi118.stdout:blocklist 172.21.15.118:0/2101854694 expires 2024-03-25T00:05:05.458796+0000 2024-03-24T00:15:22.786 INFO:teuthology.orchestra.run.smithi118.stdout:blocklist 172.21.15.118:0/630779359 expires 2024-03-25T00:05:05.458796+0000 2024-03-24T00:15:22.786 INFO:teuthology.orchestra.run.smithi118.stdout:blocklist 172.21.15.118:0/2158921029 expires 2024-03-25T00:04:31.283725+0000 2024-03-24T00:15:22.786 INFO:teuthology.orchestra.run.smithi118.stdout:blocklist 172.21.15.118:0/1003670455 expires 2024-03-25T00:05:05.458796+0000 2024-03-24T00:15:22.787 INFO:teuthology.orchestra.run.smithi118.stdout:blocklist 172.21.15.118:0/368712314 expires 2024-03-25T00:04:31.283725+0000 2024-03-24T00:15:22.787 INFO:teuthology.orchestra.run.smithi118.stdout:blocklist 172.21.15.118:0/2004775597 expires 2024-03-25T00:04:31.283725+0000 2024-03-24T00:15:22.787 INFO:teuthology.orchestra.run.smithi118.stdout:blocklist 172.21.15.118:6800/1463147955 expires 2024-03-25T00:04:31.283725+0000 2024-03-24T00:15:22.787 INFO:teuthology.orchestra.run.smithi118.stdout:blocklist 172.21.15.118:6801/1463147955 expires 2024-03-25T00:04:31.283725+0000 2024-03-24T00:15:22.787 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:22.785+0000 7fe104a00700 1 -- 172.21.15.118:0/820801929 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7fe0e40a2890 msgr2=0x7fe0e40a4d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:15:22.787 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:22.785+0000 7fe104a00700 1 --2- 172.21.15.118:0/820801929 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7fe0e40a2890 0x7fe0e40a4d50 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7fe1000765a0 tx=0x7fe0f0009260 comp rx=0 tx=0).stop 2024-03-24T00:15:22.787 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:22.785+0000 7fe104a00700 1 -- 172.21.15.118:0/820801929 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fe1000761c0 msgr2=0x7fe10019a9f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:15:22.787 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:22.785+0000 7fe104a00700 1 --2- 172.21.15.118:0/820801929 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fe1000761c0 0x7fe10019a9f0 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7fe0ec00e9a0 tx=0x7fe0ec00ecb0 comp rx=0 tx=0).stop 2024-03-24T00:15:22.787 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:22.785+0000 7fe104a00700 1 -- 172.21.15.118:0/820801929 shutdown_connections 2024-03-24T00:15:22.788 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:22.785+0000 7fe104a00700 1 --2- 172.21.15.118:0/820801929 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe100074bf0 0x7fe10019a4b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:15:22.788 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:22.785+0000 7fe104a00700 1 --2- 172.21.15.118:0/820801929 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7fe0e40a2890 0x7fe0e40a4d50 unknown :-1 s=CLOSED pgs=231 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:15:22.788 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:22.785+0000 7fe104a00700 1 --2- 172.21.15.118:0/820801929 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fe100109fd0 0x7fe1001a1a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:15:22.788 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:22.785+0000 7fe104a00700 1 --2- 172.21.15.118:0/820801929 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fe1000761c0 0x7fe10019a9f0 unknown :-1 s=CLOSED pgs=157 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:15:22.788 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:22.785+0000 7fe104a00700 1 -- 172.21.15.118:0/820801929 >> 172.21.15.118:0/820801929 conn(0x7fe1000fd6a0 msgr2=0x7fe100105ed0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:15:22.788 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:22.785+0000 7fe104a00700 1 -- 172.21.15.118:0/820801929 shutdown_connections 2024-03-24T00:15:22.788 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:22.785+0000 7fe104a00700 1 -- 172.21.15.118:0/820801929 wait complete. 2024-03-24T00:15:23.178 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:15:22 smithi118 ceph-mon[31535]: pgmap v388: 97 pgs: 97 active+clean; 645 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-24T00:15:23.203 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:15:22 smithi189 ceph-mon[36187]: pgmap v388: 97 pgs: 97 active+clean; 645 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-24T00:15:23.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:15:22 smithi193 ceph-mon[36878]: pgmap v388: 97 pgs: 97 active+clean; 645 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-24T00:15:23.620 INFO:teuthology.run_tasks:Running task fs.ready... 2024-03-24T00:15:23.631 DEBUG:teuthology.orchestra.run.smithi118:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-24T00:15:23.866 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:23.865+0000 7f3d2fe00640 1 -- 172.21.15.118:0/942695586 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f3d301015e0 msgr2=0x7f3d30101a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:15:23.867 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:23.865+0000 7f3d2fe00640 1 --2- 172.21.15.118:0/942695586 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f3d301015e0 0x7f3d30101a60 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7f3d1c009900 tx=0x7f3d1c02bc50 comp rx=0 tx=0).stop 2024-03-24T00:15:23.867 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:23.866+0000 7f3d2fe00640 1 -- 172.21.15.118:0/942695586 shutdown_connections 2024-03-24T00:15:23.867 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:23.866+0000 7f3d2fe00640 1 --2- 172.21.15.118:0/942695586 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f3d30101fa0 0x7f3d30108830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:15:23.867 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:23.866+0000 7f3d2fe00640 1 --2- 172.21.15.118:0/942695586 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f3d301015e0 0x7f3d30101a60 unknown :-1 s=CLOSED pgs=158 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:15:23.867 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:23.866+0000 7f3d2fe00640 1 --2- 172.21.15.118:0/942695586 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3d301003f0 0x7f3d30100810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:15:23.867 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:23.866+0000 7f3d2fe00640 1 -- 172.21.15.118:0/942695586 >> 172.21.15.118:0/942695586 conn(0x7f3d300fb980 msgr2=0x7f3d300fdda0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:15:23.867 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:23.866+0000 7f3d2fe00640 1 -- 172.21.15.118:0/942695586 shutdown_connections 2024-03-24T00:15:23.867 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:23.866+0000 7f3d2fe00640 1 -- 172.21.15.118:0/942695586 wait complete. 2024-03-24T00:15:23.871 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:23.867+0000 7f3d2fe00640 1 Processor -- start 2024-03-24T00:15:23.871 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:23.867+0000 7f3d2fe00640 1 -- start start 2024-03-24T00:15:23.871 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:23.867+0000 7f3d2fe00640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f3d301003f0 0x7f3d301999f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:15:23.871 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:23.867+0000 7f3d2fe00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3d301015e0 0x7f3d30199f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:15:23.872 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:23.867+0000 7f3d2fe00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f3d30101fa0 0x7f3d301a0fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:15:23.872 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:23.867+0000 7f3d2fe00640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3d3010c0d0 con 0x7f3d301003f0 2024-03-24T00:15:23.872 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:23.867+0000 7f3d2fe00640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3d3010bf50 con 0x7f3d301015e0 2024-03-24T00:15:23.872 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:23.867+0000 7f3d2fe00640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3d3010c250 con 0x7f3d30101fa0 2024-03-24T00:15:23.872 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:23.868+0000 7f3d2e000640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3d301015e0 0x7f3d30199f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:15:23.872 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:23.868+0000 7f3d2e000640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3d301015e0 0x7f3d30199f30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.118:54730/0 (socket says 172.21.15.118:54730) 2024-03-24T00:15:23.872 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:23.868+0000 7f3d2e000640 1 -- 172.21.15.118:0/1031125045 learned_addr learned my addr 172.21.15.118:0/1031125045 (peer_addr_for_me v2:172.21.15.118:0/0) 2024-03-24T00:15:23.872 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:23.868+0000 7f3d2f400640 1 --2- 172.21.15.118:0/1031125045 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f3d30101fa0 0x7f3d301a0fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:15:23.872 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:23.868+0000 7f3d2ea00640 1 --2- 172.21.15.118:0/1031125045 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f3d301003f0 0x7f3d301999f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:15:23.872 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:23.868+0000 7f3d2e000640 1 -- 172.21.15.118:0/1031125045 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f3d30101fa0 msgr2=0x7f3d301a0fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:15:23.872 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:23.868+0000 7f3d2e000640 1 --2- 172.21.15.118:0/1031125045 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f3d30101fa0 0x7f3d301a0fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:15:23.872 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:23.868+0000 7f3d2e000640 1 -- 172.21.15.118:0/1031125045 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f3d301003f0 msgr2=0x7f3d301999f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:15:23.872 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:23.868+0000 7f3d2e000640 1 --2- 172.21.15.118:0/1031125045 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f3d301003f0 0x7f3d301999f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:15:23.873 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:23.868+0000 7f3d2e000640 1 -- 172.21.15.118:0/1031125045 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3d301a1610 con 0x7f3d301015e0 2024-03-24T00:15:23.873 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:23.868+0000 7f3d2f400640 1 --2- 172.21.15.118:0/1031125045 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f3d30101fa0 0x7f3d301a0fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T00:15:23.873 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:23.868+0000 7f3d2e000640 1 --2- 172.21.15.118:0/1031125045 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3d301015e0 0x7f3d30199f30 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f3d1c02c260 tx=0x7f3d1c02c670 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:15:23.873 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:23.869+0000 7f3d27400640 1 -- 172.21.15.118:0/1031125045 <== mon.2 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3d1c041070 con 0x7f3d301015e0 2024-03-24T00:15:23.873 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:23.869+0000 7f3d27400640 1 -- 172.21.15.118:0/1031125045 <== mon.2 v2:172.21.15.189:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f3d1c004450 con 0x7f3d301015e0 2024-03-24T00:15:23.873 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:23.869+0000 7f3d2fe00640 1 -- 172.21.15.118:0/1031125045 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3d301a18a0 con 0x7f3d301015e0 2024-03-24T00:15:23.873 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:23.869+0000 7f3d27400640 1 -- 172.21.15.118:0/1031125045 <== mon.2 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3d1c034440 con 0x7f3d301015e0 2024-03-24T00:15:23.873 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:23.869+0000 7f3d2fe00640 1 -- 172.21.15.118:0/1031125045 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3d301a1df0 con 0x7f3d301015e0 2024-03-24T00:15:23.873 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:23.870+0000 7f3d2fe00640 1 -- 172.21.15.118:0/1031125045 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3d301a20b0 con 0x7f3d301015e0 2024-03-24T00:15:23.873 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:23.872+0000 7f3d27400640 1 -- 172.21.15.118:0/1031125045 <== mon.2 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 18) v1 ==== 137291+0+0 (secure 0 0 0) 0x7f3d1c044070 con 0x7f3d301015e0 2024-03-24T00:15:23.874 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:23.873+0000 7f3d27400640 1 --2- 172.21.15.118:0/1031125045 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7f3d140a2800 0x7f3d140a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:15:23.874 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:23.874+0000 7f3d27400640 1 -- 172.21.15.118:0/1031125045 <== mon.2 v2:172.21.15.189:3300/0 5 ==== osd_map(90..90 src has 1..90) v4 ==== 8277+0+0 (secure 0 0 0) 0x7f3d1c0f05e0 con 0x7f3d301015e0 2024-03-24T00:15:23.874 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:23.874+0000 7f3d2ea00640 1 --2- 172.21.15.118:0/1031125045 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7f3d140a2800 0x7f3d140a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:15:23.874 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:23.874+0000 7f3d2ea00640 1 --2- 172.21.15.118:0/1031125045 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7f3d140a2800 0x7f3d140a4cc0 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7f3d30101440 tx=0x7f3d20005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:15:23.882 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:23.877+0000 7f3d27400640 1 -- 172.21.15.118:0/1031125045 <== mon.2 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f3d1c0bcfa0 con 0x7f3d301015e0 2024-03-24T00:15:23.945 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:15:23 smithi118 ceph-mon[31535]: from='client.? 172.21.15.118:0/820801929' entity='client.admin' cmd=[{"prefix": "osd dump"}]: dispatch 2024-03-24T00:15:24.103 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:24.103+0000 7f3d2fe00640 1 -- 172.21.15.118:0/1031125045 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f3d301a2770 con 0x7f3d301015e0 2024-03-24T00:15:24.105 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:24.105+0000 7f3d27400640 1 -- 172.21.15.118:0/1031125045 <== mon.2 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v90) v1 ==== 74+0+22834 (secure 0 0 0) 0x7f3d1c0c1040 con 0x7f3d301015e0 2024-03-24T00:15:24.106 INFO:teuthology.orchestra.run.smithi118.stdout: 2024-03-24T00:15:24.106 INFO:teuthology.orchestra.run.smithi118.stdout:{"epoch":90,"fsid":"d79a5420-e971-11ee-95cd-87774f69a715","created":"2024-03-24T00:03:58.648524+0000","modified":"2024-03-24T00:15:07.721960+0000","last_up_change":"2024-03-24T00:12:03.809000+0000","last_in_change":"2024-03-24T00:11:44.389344+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-24T00:07:50.190965+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-24T00:12:59.993723+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-24T00:13:01.257347+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"86","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"c9590426-3901-4f68-a5ef-d57aa8fbac92","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6802","nonce":557690061},{"type":"v1","addr":"172.21.15.118:6803","nonce":557690061}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6804","nonce":557690061},{"type":"v1","addr":"172.21.15.118:6805","nonce":557690061}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6808","nonce":557690061},{"type":"v1","addr":"172.21.15.118:6809","nonce":557690061}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6806","nonce":557690061},{"type":"v1","addr":"172.21.15.118:6807","nonce":557690061}]},"public_addr":"172.21.15.118:6803/557690061","cluster_addr":"172.21.15.118:6805/557690061","heartbeat_back_addr":"172.21.15.118:6809/557690061","heartbeat_front_addr":"172.21.15.118:6807/557690061","state":["exists","up"]},{"osd":1,"uuid":"fc7f827e-ecf6-4661-aad0-7a22951892e0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6800","nonce":4203029485},{"type":"v1","addr":"172.21.15.189:6801","nonce":4203029485}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6802","nonce":4203029485},{"type":"v1","addr":"172.21.15.189:6803","nonce":4203029485}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6806","nonce":4203029485},{"type":"v1","addr":"172.21.15.189:6807","nonce":4203029485}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6804","nonce":4203029485},{"type":"v1","addr":"172.21.15.189:6805","nonce":4203029485}]},"public_addr":"172.21.15.189:6801/4203029485","cluster_addr":"172.21.15.189:6803/4203029485","heartbeat_back_addr":"172.21.15.189:6807/4203029485","heartbeat_front_addr":"172.21.15.189:6805/4203029485","state":["exists","up"]},{"osd":2,"uuid":"e73ccfe9-0e5b-4734-a154-a2f0d829fcfb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6800","nonce":3699076971},{"type":"v1","addr":"172.21.15.193:6801","nonce":3699076971}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6802","nonce":3699076971},{"type":"v1","addr":"172.21.15.193:6803","nonce":3699076971}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6806","nonce":3699076971},{"type":"v1","addr":"172.21.15.193:6807","nonce":3699076971}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6804","nonce":3699076971},{"type":"v1","addr":"172.21.15.193:6805","nonce":3699076971}]},"public_addr":"172.21.15.193:6801/3699076971","cluster_addr":"172.21.15.193:6803/3699076971","heartbeat_back_addr":"172.21.15.193:6807/3699076971","heartbeat_front_addr":"172.21.15.193:6805/3699076971","state":["exists","up"]},{"osd":3,"uuid":"63fed43b-43ce-4031-8957-7fe94725c1ed","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6810","nonce":4059617010},{"type":"v1","addr":"172.21.15.118:6811","nonce":4059617010}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6812","nonce":4059617010},{"type":"v1","addr":"172.21.15.118:6813","nonce":4059617010}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6816","nonce":4059617010},{"type":"v1","addr":"172.21.15.118:6817","nonce":4059617010}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6814","nonce":4059617010},{"type":"v1","addr":"172.21.15.118:6815","nonce":4059617010}]},"public_addr":"172.21.15.118:6811/4059617010","cluster_addr":"172.21.15.118:6813/4059617010","heartbeat_back_addr":"172.21.15.118:6817/4059617010","heartbeat_front_addr":"172.21.15.118:6815/4059617010","state":["exists","up"]},{"osd":4,"uuid":"57377f66-c3b4-43c1-8e8e-bc873672906d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":34,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6808","nonce":4075247068},{"type":"v1","addr":"172.21.15.189:6809","nonce":4075247068}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6810","nonce":4075247068},{"type":"v1","addr":"172.21.15.189:6811","nonce":4075247068}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6814","nonce":4075247068},{"type":"v1","addr":"172.21.15.189:6815","nonce":4075247068}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6812","nonce":4075247068},{"type":"v1","addr":"172.21.15.189:6813","nonce":4075247068}]},"public_addr":"172.21.15.189:6809/4075247068","cluster_addr":"172.21.15.189:6811/4075247068","heartbeat_back_addr":"172.21.15.189:6815/4075247068","heartbeat_front_addr":"172.21.15.189:6813/4075247068","state":["exists","up"]},{"osd":5,"uuid":"91970976-2193-4224-81c0-dc96cb022c85","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":39,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6808","nonce":2920363402},{"type":"v1","addr":"172.21.15.193:6809","nonce":2920363402}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6810","nonce":2920363402},{"type":"v1","addr":"172.21.15.193:6811","nonce":2920363402}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6814","nonce":2920363402},{"type":"v1","addr":"172.21.15.193:6815","nonce":2920363402}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6812","nonce":2920363402},{"type":"v1","addr":"172.21.15.193:6813","nonce":2920363402}]},"public_addr":"172.21.15.193:6809/2920363402","cluster_addr":"172.21.15.193:6811/2920363402","heartbeat_back_addr":"172.21.15.193:6815/2920363402","heartbeat_front_addr":"172.21.15.193:6813/2920363402","state":["exists","up"]},{"osd":6,"uuid":"0b77b196-8674-41c6-ad3d-726b5d01c8fd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":47,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6818","nonce":431354576},{"type":"v1","addr":"172.21.15.118:6819","nonce":431354576}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6820","nonce":431354576},{"type":"v1","addr":"172.21.15.118:6821","nonce":431354576}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6824","nonce":431354576},{"type":"v1","addr":"172.21.15.118:6825","nonce":431354576}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6822","nonce":431354576},{"type":"v1","addr":"172.21.15.118:6823","nonce":431354576}]},"public_addr":"172.21.15.118:6819/431354576","cluster_addr":"172.21.15.118:6821/431354576","heartbeat_back_addr":"172.21.15.118:6825/431354576","heartbeat_front_addr":"172.21.15.118:6823/431354576","state":["exists","up"]},{"osd":7,"uuid":"1fbbd38f-4e8e-4d4c-8e98-66f6862bc4bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":52,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6816","nonce":3667611284},{"type":"v1","addr":"172.21.15.189:6817","nonce":3667611284}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6818","nonce":3667611284},{"type":"v1","addr":"172.21.15.189:6819","nonce":3667611284}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6822","nonce":3667611284},{"type":"v1","addr":"172.21.15.189:6823","nonce":3667611284}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6820","nonce":3667611284},{"type":"v1","addr":"172.21.15.189:6821","nonce":3667611284}]},"public_addr":"172.21.15.189:6817/3667611284","cluster_addr":"172.21.15.189:6819/3667611284","heartbeat_back_addr":"172.21.15.189:6823/3667611284","heartbeat_front_addr":"172.21.15.189:6821/3667611284","state":["exists","up"]},{"osd":8,"uuid":"6462c2b5-e4b3-4b0b-bcfa-4de2c89e93e0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":60,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6816","nonce":3853800910},{"type":"v1","addr":"172.21.15.193:6817","nonce":3853800910}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6818","nonce":3853800910},{"type":"v1","addr":"172.21.15.193:6819","nonce":3853800910}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6822","nonce":3853800910},{"type":"v1","addr":"172.21.15.193:6823","nonce":3853800910}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6820","nonce":3853800910},{"type":"v1","addr":"172.21.15.193:6821","nonce":3853800910}]},"public_addr":"172.21.15.193:6817/3853800910","cluster_addr":"172.21.15.193:6819/3853800910","heartbeat_back_addr":"172.21.15.193:6823/3853800910","heartbeat_front_addr":"172.21.15.193:6821/3853800910","state":["exists","up"]},{"osd":9,"uuid":"54be277f-3596-422f-9de5-9005a8571c6f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":68,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6826","nonce":3248293936},{"type":"v1","addr":"172.21.15.118:6827","nonce":3248293936}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6828","nonce":3248293936},{"type":"v1","addr":"172.21.15.118:6829","nonce":3248293936}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6832","nonce":3248293936},{"type":"v1","addr":"172.21.15.118:6833","nonce":3248293936}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6830","nonce":3248293936},{"type":"v1","addr":"172.21.15.118:6831","nonce":3248293936}]},"public_addr":"172.21.15.118:6827/3248293936","cluster_addr":"172.21.15.118:6829/3248293936","heartbeat_back_addr":"172.21.15.118:6833/3248293936","heartbeat_front_addr":"172.21.15.118:6831/3248293936","state":["exists","up"]},{"osd":10,"uuid":"7ba69d69-ac41-44ce-bc39-c049a8ca5900","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":73,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6824","nonce":2112195760},{"type":"v1","addr":"172.21.15.189:6825","nonce":2112195760}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6826","nonce":2112195760},{"type":"v1","addr":"172.21.15.189:6827","nonce":2112195760}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6830","nonce":2112195760},{"type":"v1","addr":"172.21.15.189:6831","nonce":2112195760}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6828","nonce":2112195760},{"type":"v1","addr":"172.21.15.189:6829","nonce":2112195760}]},"public_addr":"172.21.15.189:6825/2112195760","cluster_addr":"172.21.15.189:6827/2112195760","heartbeat_back_addr":"172.21.15.189:6831/2112195760","heartbeat_front_addr":"172.21.15.189:6829/2112195760","state":["exists","up"]},{"osd":11,"uuid":"186f0b18-6b22-4eae-83b7-b21bbcce0817","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":81,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6824","nonce":3880515510},{"type":"v1","addr":"172.21.15.193:6825","nonce":3880515510}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6826","nonce":3880515510},{"type":"v1","addr":"172.21.15.193:6827","nonce":3880515510}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6830","nonce":3880515510},{"type":"v1","addr":"172.21.15.193:6831","nonce":3880515510}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6828","nonce":3880515510},{"type":"v1","addr":"172.21.15.193:6829","nonce":3880515510}]},"public_addr":"172.21.15.193:6825/3880515510","cluster_addr":"172.21.15.193:6827/3880515510","heartbeat_back_addr":"172.21.15.193:6831/3880515510","heartbeat_front_addr":"172.21.15.193:6829/3880515510","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:06:58.105543+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:07:23.795866+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:07:47.426050+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:08:19.427073+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:08:45.816082+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:09:11.081523+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:09:44.485077+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:10:11.338972+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:10:36.234024+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:11:09.730432+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:11:37.059083+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:12:02.371947+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.a","mappings":[{"from":2,"to":11}]},{"pgid":"2.14","mappings":[{"from":2,"to":4}]},{"pgid":"2.1d","mappings":[{"from":8,"to":11}]},{"pgid":"2.1f","mappings":[{"from":8,"to":1}]},{"pgid":"3.11","mappings":[{"from":4,"to":8},{"from":5,"to":9}]},{"pgid":"3.13","mappings":[{"from":4,"to":9}]},{"pgid":"3.14","mappings":[{"from":4,"to":6}]},{"pgid":"3.31","mappings":[{"from":4,"to":7}]},{"pgid":"3.38","mappings":[{"from":3,"to":9}]},{"pgid":"3.39","mappings":[{"from":5,"to":8}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.118:6801/389249272":"2024-03-25T00:05:05.458796+0000","172.21.15.118:6800/389249272":"2024-03-25T00:05:05.458796+0000","172.21.15.118:0/2101854694":"2024-03-25T00:05:05.458796+0000","172.21.15.118:0/630779359":"2024-03-25T00:05:05.458796+0000","172.21.15.118:0/2158921029":"2024-03-25T00:04:31.283725+0000","172.21.15.118:0/1003670455":"2024-03-25T00:05:05.458796+0000","172.21.15.118:0/368712314":"2024-03-25T00:04:31.283725+0000","172.21.15.118:0/2004775597":"2024-03-25T00:04:31.283725+0000","172.21.15.118:6800/1463147955":"2024-03-25T00:04:31.283725+0000","172.21.15.118:6801/1463147955":"2024-03-25T00:04:31.283725+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-24T00:15:24.108 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:24.107+0000 7f3d2fe00640 1 -- 172.21.15.118:0/1031125045 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7f3d140a2800 msgr2=0x7f3d140a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:15:24.108 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:24.107+0000 7f3d2fe00640 1 --2- 172.21.15.118:0/1031125045 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7f3d140a2800 0x7f3d140a4cc0 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7f3d30101440 tx=0x7f3d20005f90 comp rx=0 tx=0).stop 2024-03-24T00:15:24.108 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:24.107+0000 7f3d2fe00640 1 -- 172.21.15.118:0/1031125045 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3d301015e0 msgr2=0x7f3d30199f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:15:24.109 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:24.107+0000 7f3d2fe00640 1 --2- 172.21.15.118:0/1031125045 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3d301015e0 0x7f3d30199f30 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 crypto rx=0x7f3d1c02c260 tx=0x7f3d1c02c670 comp rx=0 tx=0).stop 2024-03-24T00:15:24.109 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:24.107+0000 7f3d2fe00640 1 -- 172.21.15.118:0/1031125045 shutdown_connections 2024-03-24T00:15:24.109 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:24.107+0000 7f3d2fe00640 1 --2- 172.21.15.118:0/1031125045 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7f3d140a2800 0x7f3d140a4cc0 unknown :-1 s=CLOSED pgs=232 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:15:24.109 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:24.107+0000 7f3d2fe00640 1 --2- 172.21.15.118:0/1031125045 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f3d30101fa0 0x7f3d301a0fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:15:24.109 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:24.107+0000 7f3d2fe00640 1 --2- 172.21.15.118:0/1031125045 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3d301015e0 0x7f3d30199f30 unknown :-1 s=CLOSED pgs=154 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:15:24.109 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:24.107+0000 7f3d2fe00640 1 --2- 172.21.15.118:0/1031125045 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f3d301003f0 0x7f3d301999f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:15:24.109 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:24.107+0000 7f3d2fe00640 1 -- 172.21.15.118:0/1031125045 >> 172.21.15.118:0/1031125045 conn(0x7f3d300fb980 msgr2=0x7f3d300fd3a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:15:24.109 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:24.108+0000 7f3d2fe00640 1 -- 172.21.15.118:0/1031125045 shutdown_connections 2024-03-24T00:15:24.109 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:24.108+0000 7f3d2fe00640 1 -- 172.21.15.118:0/1031125045 wait complete. 2024-03-24T00:15:24.128 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-24T00:07:50.190965+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-24T00:12:59.993723+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '85', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.8799999952316284, 'score_stable': 1.8799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.8799999952316284, 'raw_score_stable': 1.8799999952316284, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-03-24T00:13:01.257347+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '86', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-24T00:15:24.128 DEBUG:teuthology.orchestra.run.smithi118:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-24T00:15:24.203 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:15:23 smithi189 ceph-mon[36187]: from='client.? 172.21.15.118:0/820801929' entity='client.admin' cmd=[{"prefix": "osd dump"}]: dispatch 2024-03-24T00:15:24.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:15:23 smithi193 ceph-mon[36878]: from='client.? 172.21.15.118:0/820801929' entity='client.admin' cmd=[{"prefix": "osd dump"}]: dispatch 2024-03-24T00:15:24.327 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:24.326+0000 7f6c85a00640 1 -- 172.21.15.118:0/979845100 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6c801038e0 msgr2=0x7f6c80103d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:15:24.328 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:24.326+0000 7f6c85a00640 1 --2- 172.21.15.118:0/979845100 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6c801038e0 0x7f6c80103d00 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 crypto rx=0x7f6c70009900 tx=0x7f6c7002bc40 comp rx=0 tx=0).stop 2024-03-24T00:15:24.328 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:24.326+0000 7f6c85a00640 1 -- 172.21.15.118:0/979845100 shutdown_connections 2024-03-24T00:15:24.328 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:24.326+0000 7f6c85a00640 1 --2- 172.21.15.118:0/979845100 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f6c80105490 0x7f6c8010bd20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:15:24.328 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:24.326+0000 7f6c85a00640 1 --2- 172.21.15.118:0/979845100 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f6c80104ad0 0x7f6c80104f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:15:24.328 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:24.327+0000 7f6c85a00640 1 --2- 172.21.15.118:0/979845100 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6c801038e0 0x7f6c80103d00 unknown :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:15:24.328 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:24.327+0000 7f6c85a00640 1 -- 172.21.15.118:0/979845100 >> 172.21.15.118:0/979845100 conn(0x7f6c800fee50 msgr2=0x7f6c80101290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:15:24.328 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:24.327+0000 7f6c85a00640 1 -- 172.21.15.118:0/979845100 shutdown_connections 2024-03-24T00:15:24.328 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:24.327+0000 7f6c85a00640 1 -- 172.21.15.118:0/979845100 wait complete. 2024-03-24T00:15:24.330 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:24.328+0000 7f6c85a00640 1 Processor -- start 2024-03-24T00:15:24.330 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:24.328+0000 7f6c85a00640 1 -- start start 2024-03-24T00:15:24.330 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:24.328+0000 7f6c85a00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6c801038e0 0x7f6c80199bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:15:24.330 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:24.328+0000 7f6c85a00640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f6c80104ad0 0x7f6c8019a130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:15:24.330 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:24.328+0000 7f6c85a00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f6c80105490 0x7f6c801a11a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:15:24.330 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:24.328+0000 7f6c85a00640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6c8010f5b0 con 0x7f6c80104ad0 2024-03-24T00:15:24.330 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:24.328+0000 7f6c85a00640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6c8010f430 con 0x7f6c801038e0 2024-03-24T00:15:24.330 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:24.328+0000 7f6c85a00640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6c8010f730 con 0x7f6c80105490 2024-03-24T00:15:24.331 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:24.328+0000 7f6c7ea00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6c801038e0 0x7f6c80199bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:15:24.331 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:24.328+0000 7f6c7f400640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f6c80105490 0x7f6c801a11a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:15:24.331 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:24.328+0000 7f6c7ea00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6c801038e0 0x7f6c80199bf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.118:54748/0 (socket says 172.21.15.118:54748) 2024-03-24T00:15:24.331 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:24.328+0000 7f6c7f400640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f6c80105490 0x7f6c801a11a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.118:46758/0 (socket says 172.21.15.118:46758) 2024-03-24T00:15:24.331 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:24.328+0000 7f6c7ea00640 1 -- 172.21.15.118:0/4242383814 learned_addr learned my addr 172.21.15.118:0/4242383814 (peer_addr_for_me v2:172.21.15.118:0/0) 2024-03-24T00:15:24.331 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:24.329+0000 7f6c7e000640 1 --2- 172.21.15.118:0/4242383814 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f6c80104ad0 0x7f6c8019a130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:15:24.331 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:24.329+0000 7f6c7f400640 1 -- 172.21.15.118:0/4242383814 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6c801038e0 msgr2=0x7f6c80199bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:15:24.331 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:24.329+0000 7f6c7f400640 1 --2- 172.21.15.118:0/4242383814 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6c801038e0 0x7f6c80199bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:15:24.331 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:24.329+0000 7f6c7f400640 1 -- 172.21.15.118:0/4242383814 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f6c80104ad0 msgr2=0x7f6c8019a130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:15:24.331 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:24.329+0000 7f6c7f400640 1 --2- 172.21.15.118:0/4242383814 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f6c80104ad0 0x7f6c8019a130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:15:24.331 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:24.329+0000 7f6c7f400640 1 -- 172.21.15.118:0/4242383814 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6c801a1810 con 0x7f6c80105490 2024-03-24T00:15:24.331 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:24.329+0000 7f6c7f400640 1 --2- 172.21.15.118:0/4242383814 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f6c80105490 0x7f6c801a11a0 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f6c78009fd0 tx=0x7f6c7800ef10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:15:24.331 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:24.329+0000 7f6c77400640 1 -- 172.21.15.118:0/4242383814 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6c7800ce20 con 0x7f6c80105490 2024-03-24T00:15:24.331 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:24.329+0000 7f6c85a00640 1 -- 172.21.15.118:0/4242383814 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6c801a1b00 con 0x7f6c80105490 2024-03-24T00:15:24.331 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:24.329+0000 7f6c77400640 1 -- 172.21.15.118:0/4242383814 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f6c78005020 con 0x7f6c80105490 2024-03-24T00:15:24.331 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:24.329+0000 7f6c77400640 1 -- 172.21.15.118:0/4242383814 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6c78010720 con 0x7f6c80105490 2024-03-24T00:15:24.332 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:24.330+0000 7f6c85a00640 1 -- 172.21.15.118:0/4242383814 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6c801a20a0 con 0x7f6c80105490 2024-03-24T00:15:24.332 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:24.331+0000 7f6c85a00640 1 -- 172.21.15.118:0/4242383814 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6c8010aba0 con 0x7f6c80105490 2024-03-24T00:15:24.333 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:24.333+0000 7f6c77400640 1 -- 172.21.15.118:0/4242383814 <== mon.1 v2:172.21.15.193:3300/0 4 ==== mgrmap(e 18) v1 ==== 137291+0+0 (secure 0 0 0) 0x7f6c78005540 con 0x7f6c80105490 2024-03-24T00:15:24.335 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:24.334+0000 7f6c77400640 1 --2- 172.21.15.118:0/4242383814 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7f6c640a2880 0x7f6c640a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:15:24.335 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:24.334+0000 7f6c77400640 1 -- 172.21.15.118:0/4242383814 <== mon.1 v2:172.21.15.193:3300/0 5 ==== osd_map(90..90 src has 1..90) v4 ==== 8277+0+0 (secure 0 0 0) 0x7f6c780cf090 con 0x7f6c80105490 2024-03-24T00:15:24.335 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:24.335+0000 7f6c7ea00640 1 --2- 172.21.15.118:0/4242383814 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7f6c640a2880 0x7f6c640a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:15:24.336 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:24.335+0000 7f6c7ea00640 1 --2- 172.21.15.118:0/4242383814 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7f6c640a2880 0x7f6c640a4d40 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7f6c70005ab0 tx=0x7f6c7002cf90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:15:24.338 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:24.338+0000 7f6c77400640 1 -- 172.21.15.118:0/4242383814 <== mon.1 v2:172.21.15.193:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f6c7809c0a0 con 0x7f6c80105490 2024-03-24T00:15:24.538 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:24.538+0000 7f6c85a00640 1 -- 172.21.15.118:0/4242383814 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f6c800622a0 con 0x7f6c80105490 2024-03-24T00:15:24.539 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:24.538+0000 7f6c77400640 1 -- 172.21.15.118:0/4242383814 <== mon.1 v2:172.21.15.193:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v90) v1 ==== 93+0+10 (secure 0 0 0) 0x7f6c7809b9a0 con 0x7f6c80105490 2024-03-24T00:15:24.539 INFO:teuthology.orchestra.run.smithi118.stdout:pg_num: 1 2024-03-24T00:15:24.542 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:24.541+0000 7f6c85a00640 1 -- 172.21.15.118:0/4242383814 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7f6c640a2880 msgr2=0x7f6c640a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:15:24.542 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:24.541+0000 7f6c85a00640 1 --2- 172.21.15.118:0/4242383814 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7f6c640a2880 0x7f6c640a4d40 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7f6c70005ab0 tx=0x7f6c7002cf90 comp rx=0 tx=0).stop 2024-03-24T00:15:24.542 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:24.541+0000 7f6c85a00640 1 -- 172.21.15.118:0/4242383814 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f6c80105490 msgr2=0x7f6c801a11a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:15:24.542 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:24.541+0000 7f6c85a00640 1 --2- 172.21.15.118:0/4242383814 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f6c80105490 0x7f6c801a11a0 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f6c78009fd0 tx=0x7f6c7800ef10 comp rx=0 tx=0).stop 2024-03-24T00:15:24.543 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:24.541+0000 7f6c85a00640 1 -- 172.21.15.118:0/4242383814 shutdown_connections 2024-03-24T00:15:24.543 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:24.541+0000 7f6c85a00640 1 --2- 172.21.15.118:0/4242383814 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7f6c640a2880 0x7f6c640a4d40 unknown :-1 s=CLOSED pgs=233 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:15:24.543 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:24.541+0000 7f6c85a00640 1 --2- 172.21.15.118:0/4242383814 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f6c80105490 0x7f6c801a11a0 unknown :-1 s=CLOSED pgs=159 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:15:24.543 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:24.541+0000 7f6c85a00640 1 --2- 172.21.15.118:0/4242383814 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f6c80104ad0 0x7f6c8019a130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:15:24.543 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:24.541+0000 7f6c85a00640 1 --2- 172.21.15.118:0/4242383814 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6c801038e0 0x7f6c80199bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:15:24.543 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:24.541+0000 7f6c85a00640 1 -- 172.21.15.118:0/4242383814 >> 172.21.15.118:0/4242383814 conn(0x7f6c800fee50 msgr2=0x7f6c80107d70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:15:24.543 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:24.542+0000 7f6c85a00640 1 -- 172.21.15.118:0/4242383814 shutdown_connections 2024-03-24T00:15:24.543 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:24.542+0000 7f6c85a00640 1 -- 172.21.15.118:0/4242383814 wait complete. 2024-03-24T00:15:24.563 DEBUG:teuthology.orchestra.run.smithi118:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-24T00:15:24.787 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:24.785+0000 7fb5cfe00640 1 -- 172.21.15.118:0/2150172574 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fb5d01049b0 msgr2=0x7fb5d0104e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:15:24.787 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:24.785+0000 7fb5cfe00640 1 --2- 172.21.15.118:0/2150172574 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fb5d01049b0 0x7fb5d0104e30 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7fb5bc009900 tx=0x7fb5bc02bc40 comp rx=0 tx=0).stop 2024-03-24T00:15:24.787 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:24.785+0000 7fb5cfe00640 1 -- 172.21.15.118:0/2150172574 shutdown_connections 2024-03-24T00:15:24.787 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:24.785+0000 7fb5cfe00640 1 --2- 172.21.15.118:0/2150172574 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb5d0105370 0x7fb5d010bc00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:15:24.787 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:24.785+0000 7fb5cfe00640 1 --2- 172.21.15.118:0/2150172574 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fb5d01049b0 0x7fb5d0104e30 unknown :-1 s=CLOSED pgs=160 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:15:24.787 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:24.785+0000 7fb5cfe00640 1 --2- 172.21.15.118:0/2150172574 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb5d01037c0 0x7fb5d0103be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:15:24.787 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:24.785+0000 7fb5cfe00640 1 -- 172.21.15.118:0/2150172574 >> 172.21.15.118:0/2150172574 conn(0x7fb5d00fed50 msgr2=0x7fb5d0101170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:15:24.787 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:24.786+0000 7fb5cfe00640 1 -- 172.21.15.118:0/2150172574 shutdown_connections 2024-03-24T00:15:24.788 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:24.786+0000 7fb5cfe00640 1 -- 172.21.15.118:0/2150172574 wait complete. 2024-03-24T00:15:24.788 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:24.786+0000 7fb5cfe00640 1 Processor -- start 2024-03-24T00:15:24.790 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:24.787+0000 7fb5cfe00640 1 -- start start 2024-03-24T00:15:24.790 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:24.787+0000 7fb5cfe00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fb5d01037c0 0x7fb5d0199b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:15:24.790 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:24.787+0000 7fb5cfe00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb5d01049b0 0x7fb5d019a040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:15:24.790 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:24.787+0000 7fb5cfe00640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb5d0105370 0x7fb5d01a10b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:15:24.790 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:24.787+0000 7fb5cfe00640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb5d010f490 con 0x7fb5d0105370 2024-03-24T00:15:24.790 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:24.787+0000 7fb5cfe00640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb5d010f310 con 0x7fb5d01049b0 2024-03-24T00:15:24.790 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:24.787+0000 7fb5cfe00640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb5d010f610 con 0x7fb5d01037c0 2024-03-24T00:15:24.790 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:24.787+0000 7fb5cea00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fb5d01037c0 0x7fb5d0199b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:15:24.790 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:24.787+0000 7fb5cf400640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb5d0105370 0x7fb5d01a10b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:15:24.790 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:24.787+0000 7fb5ce000640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb5d01049b0 0x7fb5d019a040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:15:24.791 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:24.787+0000 7fb5cea00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fb5d01037c0 0x7fb5d0199b00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.118:46772/0 (socket says 172.21.15.118:46772) 2024-03-24T00:15:24.791 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:24.787+0000 7fb5ce000640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb5d01049b0 0x7fb5d019a040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.118:54766/0 (socket says 172.21.15.118:54766) 2024-03-24T00:15:24.791 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:24.787+0000 7fb5cea00640 1 -- 172.21.15.118:0/2760328151 learned_addr learned my addr 172.21.15.118:0/2760328151 (peer_addr_for_me v2:172.21.15.118:0/0) 2024-03-24T00:15:24.791 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:24.787+0000 7fb5cea00640 1 -- 172.21.15.118:0/2760328151 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb5d01049b0 msgr2=0x7fb5d019a040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:15:24.791 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:24.787+0000 7fb5cea00640 1 --2- 172.21.15.118:0/2760328151 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb5d01049b0 0x7fb5d019a040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:15:24.791 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:24.787+0000 7fb5cea00640 1 -- 172.21.15.118:0/2760328151 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb5d0105370 msgr2=0x7fb5d01a10b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:15:24.791 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:24.787+0000 7fb5cea00640 1 --2- 172.21.15.118:0/2760328151 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb5d0105370 0x7fb5d01a10b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:15:24.791 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:24.788+0000 7fb5cea00640 1 -- 172.21.15.118:0/2760328151 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb5d01a1720 con 0x7fb5d01037c0 2024-03-24T00:15:24.791 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:24.788+0000 7fb5cf400640 1 --2- 172.21.15.118:0/2760328151 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb5d0105370 0x7fb5d01a10b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T00:15:24.791 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:24.788+0000 7fb5cea00640 1 --2- 172.21.15.118:0/2760328151 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fb5d01037c0 0x7fb5d0199b00 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7fb5c000e9c0 tx=0x7fb5c000edc0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:15:24.791 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:24.788+0000 7fb5c7400640 1 -- 172.21.15.118:0/2760328151 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb5c000ccc0 con 0x7fb5d01037c0 2024-03-24T00:15:24.791 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:24.788+0000 7fb5c7400640 1 -- 172.21.15.118:0/2760328151 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fb5c00035f0 con 0x7fb5d01037c0 2024-03-24T00:15:24.791 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:24.788+0000 7fb5c7400640 1 -- 172.21.15.118:0/2760328151 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb5c000ce60 con 0x7fb5d01037c0 2024-03-24T00:15:24.791 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:24.788+0000 7fb5cfe00640 1 -- 172.21.15.118:0/2760328151 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb5d01a1a10 con 0x7fb5d01037c0 2024-03-24T00:15:24.791 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:24.788+0000 7fb5cfe00640 1 -- 172.21.15.118:0/2760328151 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb5d01a1ee0 con 0x7fb5d01037c0 2024-03-24T00:15:24.792 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:24.789+0000 7fb5cfe00640 1 -- 172.21.15.118:0/2760328151 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb5d010aae0 con 0x7fb5d01037c0 2024-03-24T00:15:24.792 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:24.791+0000 7fb5c7400640 1 -- 172.21.15.118:0/2760328151 <== mon.1 v2:172.21.15.193:3300/0 4 ==== mgrmap(e 18) v1 ==== 137291+0+0 (secure 0 0 0) 0x7fb5c00037a0 con 0x7fb5d01037c0 2024-03-24T00:15:24.793 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:24.792+0000 7fb5c7400640 1 --2- 172.21.15.118:0/2760328151 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7fb5b40a2880 0x7fb5b40a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:15:24.793 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:24.793+0000 7fb5c7400640 1 -- 172.21.15.118:0/2760328151 <== mon.1 v2:172.21.15.193:3300/0 5 ==== osd_map(90..90 src has 1..90) v4 ==== 8277+0+0 (secure 0 0 0) 0x7fb5c00cedc0 con 0x7fb5d01037c0 2024-03-24T00:15:24.793 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:24.793+0000 7fb5ce000640 1 --2- 172.21.15.118:0/2760328151 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7fb5b40a2880 0x7fb5b40a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:15:24.793 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:24.793+0000 7fb5ce000640 1 --2- 172.21.15.118:0/2760328151 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7fb5b40a2880 0x7fb5b40a4d40 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7fb5bc02c250 tx=0x7fb5bc02cf90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:15:24.797 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:24.796+0000 7fb5c7400640 1 -- 172.21.15.118:0/2760328151 <== mon.1 v2:172.21.15.193:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7fb5c009bdd0 con 0x7fb5d01037c0 2024-03-24T00:15:24.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:15:24 smithi118 ceph-mon[31535]: pgmap v389: 97 pgs: 97 active+clean; 645 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-03-24T00:15:24.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:15:24 smithi118 ceph-mon[31535]: from='client.? 172.21.15.118:0/1031125045' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-24T00:15:24.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:15:24 smithi118 ceph-mon[31535]: from='client.? 172.21.15.118:0/4242383814' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-24T00:15:24.993 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:24.992+0000 7fb5cfe00640 1 -- 172.21.15.118:0/2760328151 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7fb5d002d2d0 con 0x7fb5d01037c0 2024-03-24T00:15:24.993 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:24.993+0000 7fb5c7400640 1 -- 172.21.15.118:0/2760328151 <== mon.1 v2:172.21.15.193:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v90) v1 ==== 104+0+11 (secure 0 0 0) 0x7fb5c009b6d0 con 0x7fb5d01037c0 2024-03-24T00:15:24.993 INFO:teuthology.orchestra.run.smithi118.stdout:pg_num: 32 2024-03-24T00:15:24.996 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:24.995+0000 7fb5cfe00640 1 -- 172.21.15.118:0/2760328151 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7fb5b40a2880 msgr2=0x7fb5b40a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:15:24.996 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:24.995+0000 7fb5cfe00640 1 --2- 172.21.15.118:0/2760328151 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7fb5b40a2880 0x7fb5b40a4d40 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7fb5bc02c250 tx=0x7fb5bc02cf90 comp rx=0 tx=0).stop 2024-03-24T00:15:24.996 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:24.995+0000 7fb5cfe00640 1 -- 172.21.15.118:0/2760328151 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fb5d01037c0 msgr2=0x7fb5d0199b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:15:24.997 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:24.995+0000 7fb5cfe00640 1 --2- 172.21.15.118:0/2760328151 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fb5d01037c0 0x7fb5d0199b00 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7fb5c000e9c0 tx=0x7fb5c000edc0 comp rx=0 tx=0).stop 2024-03-24T00:15:24.997 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:24.995+0000 7fb5cfe00640 1 -- 172.21.15.118:0/2760328151 shutdown_connections 2024-03-24T00:15:24.997 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:24.995+0000 7fb5cfe00640 1 --2- 172.21.15.118:0/2760328151 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7fb5b40a2880 0x7fb5b40a4d40 unknown :-1 s=CLOSED pgs=234 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:15:24.997 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:24.995+0000 7fb5cfe00640 1 --2- 172.21.15.118:0/2760328151 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb5d0105370 0x7fb5d01a10b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:15:24.997 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:24.995+0000 7fb5cfe00640 1 --2- 172.21.15.118:0/2760328151 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb5d01049b0 0x7fb5d019a040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:15:24.997 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:24.995+0000 7fb5cfe00640 1 --2- 172.21.15.118:0/2760328151 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fb5d01037c0 0x7fb5d0199b00 unknown :-1 s=CLOSED pgs=161 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:15:24.997 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:24.995+0000 7fb5cfe00640 1 -- 172.21.15.118:0/2760328151 >> 172.21.15.118:0/2760328151 conn(0x7fb5d00fed50 msgr2=0x7fb5d0107cb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:15:24.997 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:24.996+0000 7fb5cfe00640 1 -- 172.21.15.118:0/2760328151 shutdown_connections 2024-03-24T00:15:24.997 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:24.996+0000 7fb5cfe00640 1 -- 172.21.15.118:0/2760328151 wait complete. 2024-03-24T00:15:25.014 DEBUG:teuthology.orchestra.run.smithi118:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-24T00:15:25.203 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:15:24 smithi189 ceph-mon[36187]: pgmap v389: 97 pgs: 97 active+clean; 645 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-03-24T00:15:25.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:15:24 smithi189 ceph-mon[36187]: from='client.? 172.21.15.118:0/1031125045' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-24T00:15:25.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:15:24 smithi189 ceph-mon[36187]: from='client.? 172.21.15.118:0/4242383814' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-24T00:15:25.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:15:24 smithi193 ceph-mon[36878]: pgmap v389: 97 pgs: 97 active+clean; 645 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-03-24T00:15:25.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:15:24 smithi193 ceph-mon[36878]: from='client.? 172.21.15.118:0/1031125045' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-24T00:15:25.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:15:24 smithi193 ceph-mon[36878]: from='client.? 172.21.15.118:0/4242383814' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-24T00:15:25.232 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:25.231+0000 7fb06f200640 1 -- 172.21.15.118:0/3964739070 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb070100480 msgr2=0x7fb0701008a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:15:25.232 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:25.231+0000 7fb06f200640 1 --2- 172.21.15.118:0/3964739070 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb070100480 0x7fb0701008a0 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7fb060009900 tx=0x7fb06002bc30 comp rx=0 tx=0).stop 2024-03-24T00:15:25.232 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:25.231+0000 7fb06f200640 1 -- 172.21.15.118:0/3964739070 shutdown_connections 2024-03-24T00:15:25.232 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:25.231+0000 7fb06f200640 1 --2- 172.21.15.118:0/3964739070 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fb070102030 0x7fb0701088c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:15:25.232 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:25.231+0000 7fb06f200640 1 --2- 172.21.15.118:0/3964739070 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb070101670 0x7fb070101af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:15:25.232 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:25.231+0000 7fb06f200640 1 --2- 172.21.15.118:0/3964739070 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb070100480 0x7fb0701008a0 unknown :-1 s=CLOSED pgs=156 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:15:25.232 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:25.231+0000 7fb06f200640 1 -- 172.21.15.118:0/3964739070 >> 172.21.15.118:0/3964739070 conn(0x7fb0700fba10 msgr2=0x7fb0700fde30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:15:25.232 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:25.231+0000 7fb06f200640 1 -- 172.21.15.118:0/3964739070 shutdown_connections 2024-03-24T00:15:25.232 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:25.231+0000 7fb06f200640 1 -- 172.21.15.118:0/3964739070 wait complete. 2024-03-24T00:15:25.233 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:25.233+0000 7fb06f200640 1 Processor -- start 2024-03-24T00:15:25.233 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:25.233+0000 7fb06f200640 1 -- start start 2024-03-24T00:15:25.235 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:25.233+0000 7fb06f200640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb070100480 0x7fb070199ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:15:25.235 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:25.233+0000 7fb06f200640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb070101670 0x7fb070199ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:15:25.235 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:25.233+0000 7fb06f200640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fb070102030 0x7fb0701a1060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:15:25.235 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:25.233+0000 7fb06f200640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb07010c160 con 0x7fb070100480 2024-03-24T00:15:25.235 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:25.233+0000 7fb06f200640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb07010bfe0 con 0x7fb070101670 2024-03-24T00:15:25.235 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:25.233+0000 7fb06f200640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb07010c2e0 con 0x7fb070102030 2024-03-24T00:15:25.235 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:25.233+0000 7fb067e00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb070101670 0x7fb070199ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:15:25.235 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:25.233+0000 7fb067e00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb070101670 0x7fb070199ff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.118:54788/0 (socket says 172.21.15.118:54788) 2024-03-24T00:15:25.235 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:25.233+0000 7fb067e00640 1 -- 172.21.15.118:0/2447638366 learned_addr learned my addr 172.21.15.118:0/2447638366 (peer_addr_for_me v2:172.21.15.118:0/0) 2024-03-24T00:15:25.236 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:25.233+0000 7fb06e800640 1 --2- 172.21.15.118:0/2447638366 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fb070102030 0x7fb0701a1060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:15:25.236 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:25.233+0000 7fb06de00640 1 --2- 172.21.15.118:0/2447638366 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb070100480 0x7fb070199ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:15:25.236 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:25.234+0000 7fb067e00640 1 -- 172.21.15.118:0/2447638366 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fb070102030 msgr2=0x7fb0701a1060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:15:25.236 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:25.234+0000 7fb067e00640 1 --2- 172.21.15.118:0/2447638366 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fb070102030 0x7fb0701a1060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:15:25.236 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:25.234+0000 7fb067e00640 1 -- 172.21.15.118:0/2447638366 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb070100480 msgr2=0x7fb070199ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:15:25.236 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:25.234+0000 7fb067e00640 1 --2- 172.21.15.118:0/2447638366 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb070100480 0x7fb070199ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:15:25.236 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:25.234+0000 7fb067e00640 1 -- 172.21.15.118:0/2447638366 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb0701a16d0 con 0x7fb070101670 2024-03-24T00:15:25.236 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:25.234+0000 7fb067e00640 1 --2- 172.21.15.118:0/2447638366 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb070101670 0x7fb070199ff0 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7fb05800e9f0 tx=0x7fb05800eeb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:15:25.236 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:25.234+0000 7fb066a00640 1 -- 172.21.15.118:0/2447638366 <== mon.2 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb05800cd80 con 0x7fb070101670 2024-03-24T00:15:25.236 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:25.234+0000 7fb066a00640 1 -- 172.21.15.118:0/2447638366 <== mon.2 v2:172.21.15.189:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fb058010910 con 0x7fb070101670 2024-03-24T00:15:25.236 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:25.234+0000 7fb06f200640 1 -- 172.21.15.118:0/2447638366 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb0701a19c0 con 0x7fb070101670 2024-03-24T00:15:25.236 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:25.234+0000 7fb066a00640 1 -- 172.21.15.118:0/2447638366 <== mon.2 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb058018990 con 0x7fb070101670 2024-03-24T00:15:25.236 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:25.235+0000 7fb06f200640 1 -- 172.21.15.118:0/2447638366 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb0701a1f10 con 0x7fb070101670 2024-03-24T00:15:25.237 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:25.236+0000 7fb06f200640 1 -- 172.21.15.118:0/2447638366 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb0701a2140 con 0x7fb070101670 2024-03-24T00:15:25.237 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:25.237+0000 7fb066a00640 1 -- 172.21.15.118:0/2447638366 <== mon.2 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 18) v1 ==== 137291+0+0 (secure 0 0 0) 0x7fb058010ac0 con 0x7fb070101670 2024-03-24T00:15:25.239 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:25.239+0000 7fb066a00640 1 --2- 172.21.15.118:0/2447638366 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7fb0540a2800 0x7fb0540a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:15:25.239 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:25.239+0000 7fb066a00640 1 -- 172.21.15.118:0/2447638366 <== mon.2 v2:172.21.15.189:3300/0 5 ==== osd_map(90..90 src has 1..90) v4 ==== 8277+0+0 (secure 0 0 0) 0x7fb0580cee40 con 0x7fb070101670 2024-03-24T00:15:25.239 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:25.239+0000 7fb06de00640 1 --2- 172.21.15.118:0/2447638366 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7fb0540a2800 0x7fb0540a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:15:25.240 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:25.239+0000 7fb06de00640 1 --2- 172.21.15.118:0/2447638366 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7fb0540a2800 0x7fb0540a4cc0 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7fb06002c170 tx=0x7fb06002df90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:15:25.243 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:25.242+0000 7fb066a00640 1 -- 172.21.15.118:0/2447638366 <== mon.2 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7fb05809b800 con 0x7fb070101670 2024-03-24T00:15:25.491 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:25.490+0000 7fb06f200640 1 -- 172.21.15.118:0/2447638366 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7fb0701a24d0 con 0x7fb070101670 2024-03-24T00:15:25.491 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:25.491+0000 7fb066a00640 1 -- 172.21.15.118:0/2447638366 <== mon.2 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v90) v1 ==== 100+0+11 (secure 0 0 0) 0x7fb05809f8a0 con 0x7fb070101670 2024-03-24T00:15:25.492 INFO:teuthology.orchestra.run.smithi118.stdout:pg_num: 64 2024-03-24T00:15:25.494 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:25.493+0000 7fb06f200640 1 -- 172.21.15.118:0/2447638366 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7fb0540a2800 msgr2=0x7fb0540a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:15:25.494 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:25.493+0000 7fb06f200640 1 --2- 172.21.15.118:0/2447638366 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7fb0540a2800 0x7fb0540a4cc0 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7fb06002c170 tx=0x7fb06002df90 comp rx=0 tx=0).stop 2024-03-24T00:15:25.494 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:25.493+0000 7fb06f200640 1 -- 172.21.15.118:0/2447638366 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb070101670 msgr2=0x7fb070199ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:15:25.494 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:25.493+0000 7fb06f200640 1 --2- 172.21.15.118:0/2447638366 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb070101670 0x7fb070199ff0 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7fb05800e9f0 tx=0x7fb05800eeb0 comp rx=0 tx=0).stop 2024-03-24T00:15:25.494 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:25.493+0000 7fb06f200640 1 -- 172.21.15.118:0/2447638366 shutdown_connections 2024-03-24T00:15:25.494 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:25.493+0000 7fb06f200640 1 --2- 172.21.15.118:0/2447638366 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7fb0540a2800 0x7fb0540a4cc0 unknown :-1 s=CLOSED pgs=235 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:15:25.494 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:25.493+0000 7fb06f200640 1 --2- 172.21.15.118:0/2447638366 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fb070102030 0x7fb0701a1060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:15:25.495 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:25.493+0000 7fb06f200640 1 --2- 172.21.15.118:0/2447638366 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb070101670 0x7fb070199ff0 unknown :-1 s=CLOSED pgs=157 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:15:25.495 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:25.493+0000 7fb06f200640 1 --2- 172.21.15.118:0/2447638366 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb070100480 0x7fb070199ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:15:25.495 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:25.493+0000 7fb06f200640 1 -- 172.21.15.118:0/2447638366 >> 172.21.15.118:0/2447638366 conn(0x7fb0700fba10 msgr2=0x7fb0700fd490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:15:25.495 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:25.494+0000 7fb06f200640 1 -- 172.21.15.118:0/2447638366 shutdown_connections 2024-03-24T00:15:25.495 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:25.494+0000 7fb06f200640 1 -- 172.21.15.118:0/2447638366 wait complete. 2024-03-24T00:15:25.514 DEBUG:teuthology.orchestra.run.smithi118:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-24T00:15:25.712 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:25.711+0000 7ff169c00640 1 -- 172.21.15.118:0/220887583 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff1641052a0 msgr2=0x7ff16410bb30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:15:25.712 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:25.711+0000 7ff169c00640 1 --2- 172.21.15.118:0/220887583 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff1641052a0 0x7ff16410bb30 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7ff15c009ca0 tx=0x7ff15c02c100 comp rx=0 tx=0).stop 2024-03-24T00:15:25.712 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:25.711+0000 7ff169c00640 1 -- 172.21.15.118:0/220887583 shutdown_connections 2024-03-24T00:15:25.713 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:25.711+0000 7ff169c00640 1 --2- 172.21.15.118:0/220887583 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff1641052a0 0x7ff16410bb30 unknown :-1 s=CLOSED pgs=158 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:15:25.713 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:25.711+0000 7ff169c00640 1 --2- 172.21.15.118:0/220887583 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7ff1641048e0 0x7ff164104d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:15:25.713 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:25.711+0000 7ff169c00640 1 --2- 172.21.15.118:0/220887583 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7ff1641036f0 0x7ff164103b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:15:25.713 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:25.711+0000 7ff169c00640 1 -- 172.21.15.118:0/220887583 >> 172.21.15.118:0/220887583 conn(0x7ff1640fec80 msgr2=0x7ff1641010a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:15:25.713 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:25.712+0000 7ff169c00640 1 -- 172.21.15.118:0/220887583 shutdown_connections 2024-03-24T00:15:25.713 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:25.712+0000 7ff169c00640 1 -- 172.21.15.118:0/220887583 wait complete. 2024-03-24T00:15:25.715 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:25.713+0000 7ff169c00640 1 Processor -- start 2024-03-24T00:15:25.715 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:25.713+0000 7ff169c00640 1 -- start start 2024-03-24T00:15:25.715 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:25.713+0000 7ff169c00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7ff1641036f0 0x7ff164199970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:15:25.715 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:25.713+0000 7ff169c00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff1641048e0 0x7ff164199eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:15:25.715 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:25.713+0000 7ff169c00640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7ff1641052a0 0x7ff1641a0f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:15:25.715 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:25.713+0000 7ff169c00640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff16410f350 con 0x7ff1641052a0 2024-03-24T00:15:25.715 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:25.713+0000 7ff169c00640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff16410f1d0 con 0x7ff1641048e0 2024-03-24T00:15:25.715 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:25.713+0000 7ff169c00640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff16410f4d0 con 0x7ff1641036f0 2024-03-24T00:15:25.716 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:25.713+0000 7ff163e00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7ff1641036f0 0x7ff164199970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:15:25.716 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:25.713+0000 7ff163400640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff1641048e0 0x7ff164199eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:15:25.716 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:25.713+0000 7ff163e00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7ff1641036f0 0x7ff164199970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.118:46802/0 (socket says 172.21.15.118:46802) 2024-03-24T00:15:25.716 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:25.713+0000 7ff169200640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7ff1641052a0 0x7ff1641a0f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:15:25.716 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:25.713+0000 7ff163e00640 1 -- 172.21.15.118:0/3314792481 learned_addr learned my addr 172.21.15.118:0/3314792481 (peer_addr_for_me v2:172.21.15.118:0/0) 2024-03-24T00:15:25.716 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:25.714+0000 7ff163e00640 1 -- 172.21.15.118:0/3314792481 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff1641048e0 msgr2=0x7ff164199eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:15:25.716 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:25.714+0000 7ff163e00640 1 --2- 172.21.15.118:0/3314792481 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff1641048e0 0x7ff164199eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:15:25.716 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:25.714+0000 7ff163e00640 1 -- 172.21.15.118:0/3314792481 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7ff1641052a0 msgr2=0x7ff1641a0f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:15:25.716 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:25.714+0000 7ff163e00640 1 --2- 172.21.15.118:0/3314792481 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7ff1641052a0 0x7ff1641a0f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:15:25.716 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:25.714+0000 7ff163e00640 1 -- 172.21.15.118:0/3314792481 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff1641a1590 con 0x7ff1641036f0 2024-03-24T00:15:25.716 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:25.714+0000 7ff169200640 1 --2- 172.21.15.118:0/3314792481 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7ff1641052a0 0x7ff1641a0f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T00:15:25.716 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:25.714+0000 7ff163400640 1 --2- 172.21.15.118:0/3314792481 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff1641048e0 0x7ff164199eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T00:15:25.716 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:25.714+0000 7ff163e00640 1 --2- 172.21.15.118:0/3314792481 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7ff1641036f0 0x7ff164199970 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7ff15400e9c0 tx=0x7ff15400ee80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:15:25.716 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:25.714+0000 7ff160c00640 1 -- 172.21.15.118:0/3314792481 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff15400cd80 con 0x7ff1641036f0 2024-03-24T00:15:25.717 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:25.714+0000 7ff160c00640 1 -- 172.21.15.118:0/3314792481 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7ff1540035f0 con 0x7ff1641036f0 2024-03-24T00:15:25.717 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:25.714+0000 7ff169c00640 1 -- 172.21.15.118:0/3314792481 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff1641a1880 con 0x7ff1641036f0 2024-03-24T00:15:25.717 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:25.714+0000 7ff160c00640 1 -- 172.21.15.118:0/3314792481 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff154010690 con 0x7ff1641036f0 2024-03-24T00:15:25.717 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:25.714+0000 7ff169c00640 1 -- 172.21.15.118:0/3314792481 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff1641a1e20 con 0x7ff1641036f0 2024-03-24T00:15:25.717 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:25.715+0000 7ff169c00640 1 -- 172.21.15.118:0/3314792481 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff16410a9b0 con 0x7ff1641036f0 2024-03-24T00:15:25.718 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:25.717+0000 7ff160c00640 1 -- 172.21.15.118:0/3314792481 <== mon.1 v2:172.21.15.193:3300/0 4 ==== mgrmap(e 18) v1 ==== 137291+0+0 (secure 0 0 0) 0x7ff1540037a0 con 0x7ff1641036f0 2024-03-24T00:15:25.719 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:25.719+0000 7ff160c00640 1 --2- 172.21.15.118:0/3314792481 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7ff1480a2880 0x7ff1480a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:15:25.719 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:25.719+0000 7ff160c00640 1 -- 172.21.15.118:0/3314792481 <== mon.1 v2:172.21.15.193:3300/0 5 ==== osd_map(90..90 src has 1..90) v4 ==== 8277+0+0 (secure 0 0 0) 0x7ff1540cee00 con 0x7ff1641036f0 2024-03-24T00:15:25.720 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:25.719+0000 7ff163400640 1 --2- 172.21.15.118:0/3314792481 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7ff1480a2880 0x7ff1480a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:15:25.720 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:25.720+0000 7ff163400640 1 --2- 172.21.15.118:0/3314792481 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7ff1480a2880 0x7ff1480a4d40 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7ff16419ae80 tx=0x7ff150005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:15:25.721 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:25.721+0000 7ff160c00640 1 -- 172.21.15.118:0/3314792481 <== mon.1 v2:172.21.15.193:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7ff15409be10 con 0x7ff1641036f0 2024-03-24T00:15:25.967 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:25.967+0000 7ff169c00640 1 -- 172.21.15.118:0/3314792481 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7ff16402d2d0 con 0x7ff1641036f0 2024-03-24T00:15:25.968 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:25.968+0000 7ff160c00640 1 -- 172.21.15.118:0/3314792481 <== mon.1 v2:172.21.15.193:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 27 v27) v1 ==== 94+0+11478 (secure 0 0 0) 0x7ff15409b710 con 0x7ff1641036f0 2024-03-24T00:15:25.968 INFO:teuthology.orchestra.run.smithi118.stdout: 2024-03-24T00:15:25.969 INFO:teuthology.orchestra.run.smithi118.stdout:{"epoch":27,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14385,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.118:6835/3194741373","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6834","nonce":3194741373},{"type":"v1","addr":"172.21.15.118:6835","nonce":3194741373}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":4},{"gid":14397,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.118:6837/1374462956","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6836","nonce":1374462956},{"type":"v1","addr":"172.21.15.118:6837","nonce":1374462956}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":27,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-03-24T00:13:01.782474+0000","modified":"2024-03-24T00:13:09.335361+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24569,"mds_1":24550,"mds_2":24511,"mds_3":24499,"mds_4":24488},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14469":{"gid":14469,"name":"a","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.118:6841/3040946537","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6840","nonce":3040946537},{"type":"v1","addr":"172.21.15.118:6841","nonce":3040946537}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24439":{"gid":24439,"name":"l","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.193:6833/1023428412","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6832","nonce":1023428412},{"type":"v1","addr":"172.21.15.193:6833","nonce":1023428412}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24440":{"gid":24440,"name":"k","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.189:6833/2740024078","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6832","nonce":2740024078},{"type":"v1","addr":"172.21.15.189:6833","nonce":2740024078}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24488":{"gid":24488,"name":"e","rank":4,"incarnation":25,"state":"up:active","state_seq":10,"addr":"172.21.15.189:6835/599473205","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6834","nonce":599473205},{"type":"v1","addr":"172.21.15.189:6835","nonce":599473205}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24499":{"gid":24499,"name":"f","rank":3,"incarnation":23,"state":"up:active","state_seq":9,"addr":"172.21.15.193:6835/2055310541","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6834","nonce":2055310541},{"type":"v1","addr":"172.21.15.193:6835","nonce":2055310541}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24511":{"gid":24511,"name":"c","rank":2,"incarnation":21,"state":"up:active","state_seq":8,"addr":"172.21.15.193:6837/2921520053","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6836","nonce":2921520053},{"type":"v1","addr":"172.21.15.193:6837","nonce":2921520053}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24515":{"gid":24515,"name":"h","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.189:6837/3450187137","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6836","nonce":3450187137},{"type":"v1","addr":"172.21.15.189:6837","nonce":3450187137}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24532":{"gid":24532,"name":"i","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.193:6839/2126842603","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6838","nonce":2126842603},{"type":"v1","addr":"172.21.15.193:6839","nonce":2126842603}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24550":{"gid":24550,"name":"g","rank":1,"incarnation":19,"state":"up:active","state_seq":6,"addr":"172.21.15.118:6839/3285788503","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6838","nonce":3285788503},{"type":"v1","addr":"172.21.15.118:6839","nonce":3285788503}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24569":{"gid":24569,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.189:6839/4074484168","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6838","nonce":4074484168},{"type":"v1","addr":"172.21.15.189:6839","nonce":4074484168}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24569,"qdb_cluster":[24511,24550,24488,24499,24569]},"id":1}]} 2024-03-24T00:15:25.969 INFO:teuthology.orchestra.run.smithi118.stderr:dumped fsmap epoch 27 2024-03-24T00:15:25.971 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:25.970+0000 7ff169c00640 1 -- 172.21.15.118:0/3314792481 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7ff1480a2880 msgr2=0x7ff1480a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:15:25.971 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:25.970+0000 7ff169c00640 1 --2- 172.21.15.118:0/3314792481 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7ff1480a2880 0x7ff1480a4d40 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7ff16419ae80 tx=0x7ff150005f90 comp rx=0 tx=0).stop 2024-03-24T00:15:25.971 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:25.970+0000 7ff169c00640 1 -- 172.21.15.118:0/3314792481 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7ff1641036f0 msgr2=0x7ff164199970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:15:25.972 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:25.970+0000 7ff169c00640 1 --2- 172.21.15.118:0/3314792481 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7ff1641036f0 0x7ff164199970 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7ff15400e9c0 tx=0x7ff15400ee80 comp rx=0 tx=0).stop 2024-03-24T00:15:25.972 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:25.971+0000 7ff169c00640 1 -- 172.21.15.118:0/3314792481 shutdown_connections 2024-03-24T00:15:25.972 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:25.971+0000 7ff169c00640 1 --2- 172.21.15.118:0/3314792481 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7ff1480a2880 0x7ff1480a4d40 unknown :-1 s=CLOSED pgs=236 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:15:25.972 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:25.971+0000 7ff169c00640 1 --2- 172.21.15.118:0/3314792481 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7ff1641052a0 0x7ff1641a0f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:15:25.972 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:25.971+0000 7ff169c00640 1 --2- 172.21.15.118:0/3314792481 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff1641048e0 0x7ff164199eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:15:25.972 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:25.971+0000 7ff169c00640 1 --2- 172.21.15.118:0/3314792481 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7ff1641036f0 0x7ff164199970 unknown :-1 s=CLOSED pgs=162 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:15:25.972 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:25.971+0000 7ff169c00640 1 -- 172.21.15.118:0/3314792481 >> 172.21.15.118:0/3314792481 conn(0x7ff1640fec80 msgr2=0x7ff164107b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:15:25.972 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:25.971+0000 7ff169c00640 1 -- 172.21.15.118:0/3314792481 shutdown_connections 2024-03-24T00:15:25.972 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:25.971+0000 7ff169c00640 1 -- 172.21.15.118:0/3314792481 wait complete. 2024-03-24T00:15:25.994 DEBUG:teuthology.orchestra.run.smithi118:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-24T00:15:26.070 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:15:25 smithi118 ceph-mon[31535]: from='client.? 172.21.15.118:0/2760328151' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-24T00:15:26.070 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:15:25 smithi118 ceph-mon[31535]: from='client.? 172.21.15.118:0/2447638366' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-24T00:15:26.198 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:26.197+0000 7f46e4800640 1 -- 172.21.15.118:0/1436420954 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f46dc1038e0 msgr2=0x7f46dc103d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:15:26.198 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:26.197+0000 7f46e4800640 1 --2- 172.21.15.118:0/1436420954 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f46dc1038e0 0x7f46dc103d00 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7f46d0009900 tx=0x7f46d002bc30 comp rx=0 tx=0).stop 2024-03-24T00:15:26.198 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:26.197+0000 7f46e4800640 1 -- 172.21.15.118:0/1436420954 shutdown_connections 2024-03-24T00:15:26.198 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:26.197+0000 7f46e4800640 1 --2- 172.21.15.118:0/1436420954 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f46dc105490 0x7f46dc10bd20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:15:26.198 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:26.197+0000 7f46e4800640 1 --2- 172.21.15.118:0/1436420954 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f46dc104ad0 0x7f46dc104f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:15:26.198 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:26.197+0000 7f46e4800640 1 --2- 172.21.15.118:0/1436420954 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f46dc1038e0 0x7f46dc103d00 unknown :-1 s=CLOSED pgs=163 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:15:26.198 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:26.197+0000 7f46e4800640 1 -- 172.21.15.118:0/1436420954 >> 172.21.15.118:0/1436420954 conn(0x7f46dc0fee90 msgr2=0x7f46dc1012b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:15:26.199 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:26.197+0000 7f46e4800640 1 -- 172.21.15.118:0/1436420954 shutdown_connections 2024-03-24T00:15:26.199 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:26.197+0000 7f46e4800640 1 -- 172.21.15.118:0/1436420954 wait complete. 2024-03-24T00:15:26.201 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:26.198+0000 7f46e4800640 1 Processor -- start 2024-03-24T00:15:26.201 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:26.198+0000 7f46e4800640 1 -- start start 2024-03-24T00:15:26.201 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:26.199+0000 7f46e4800640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f46dc1038e0 0x7f46dc199bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:15:26.201 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:26.199+0000 7f46e4800640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f46dc104ad0 0x7f46dc19a130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:15:26.201 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:26.199+0000 7f46e4800640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f46dc105490 0x7f46dc1a11a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:15:26.201 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:26.199+0000 7f46e4800640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f46dc10f580 con 0x7f46dc1038e0 2024-03-24T00:15:26.201 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:26.199+0000 7f46e4800640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f46dc10f400 con 0x7f46dc105490 2024-03-24T00:15:26.202 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:26.199+0000 7f46e4800640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f46dc10f700 con 0x7f46dc104ad0 2024-03-24T00:15:26.202 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:26.199+0000 7f46e1e00640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f46dc1038e0 0x7f46dc199bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:15:26.202 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:26.199+0000 7f46e1400640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f46dc104ad0 0x7f46dc19a130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:15:26.202 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:26.199+0000 7f46e1400640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f46dc104ad0 0x7f46dc19a130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.118:46816/0 (socket says 172.21.15.118:46816) 2024-03-24T00:15:26.202 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:26.199+0000 7f46e1400640 1 -- 172.21.15.118:0/4148933722 learned_addr learned my addr 172.21.15.118:0/4148933722 (peer_addr_for_me v2:172.21.15.118:0/0) 2024-03-24T00:15:26.202 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:26.199+0000 7f46e1e00640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f46dc1038e0 0x7f46dc199bf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.118:3300/0 says I am v2:172.21.15.118:34400/0 (socket says 172.21.15.118:34400) 2024-03-24T00:15:26.202 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:26.199+0000 7f46e2800640 1 --2- 172.21.15.118:0/4148933722 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f46dc105490 0x7f46dc1a11a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:15:26.202 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:26.199+0000 7f46e1400640 1 -- 172.21.15.118:0/4148933722 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f46dc105490 msgr2=0x7f46dc1a11a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:15:26.202 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:26.199+0000 7f46e1400640 1 --2- 172.21.15.118:0/4148933722 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f46dc105490 0x7f46dc1a11a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:15:26.202 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:26.199+0000 7f46e1400640 1 -- 172.21.15.118:0/4148933722 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f46dc1038e0 msgr2=0x7f46dc199bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:15:26.202 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:26.199+0000 7f46e1400640 1 --2- 172.21.15.118:0/4148933722 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f46dc1038e0 0x7f46dc199bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:15:26.202 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:26.199+0000 7f46e1400640 1 -- 172.21.15.118:0/4148933722 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f46dc1a1810 con 0x7f46dc104ad0 2024-03-24T00:15:26.202 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:26.199+0000 7f46e2800640 1 --2- 172.21.15.118:0/4148933722 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f46dc105490 0x7f46dc1a11a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T00:15:26.203 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:26.199+0000 7f46e1400640 1 --2- 172.21.15.118:0/4148933722 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f46dc104ad0 0x7f46dc19a130 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f46cc00e9f0 tx=0x7f46cc00eeb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:15:26.203 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:26.200+0000 7f46d6a00640 1 -- 172.21.15.118:0/4148933722 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f46cc00cd80 con 0x7f46dc104ad0 2024-03-24T00:15:26.203 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:26.200+0000 7f46d6a00640 1 -- 172.21.15.118:0/4148933722 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f46cc010910 con 0x7f46dc104ad0 2024-03-24T00:15:26.203 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:26.200+0000 7f46d6a00640 1 -- 172.21.15.118:0/4148933722 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f46cc018990 con 0x7f46dc104ad0 2024-03-24T00:15:26.203 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:26.200+0000 7f46e4800640 1 -- 172.21.15.118:0/4148933722 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f46dc1a1b00 con 0x7f46dc104ad0 2024-03-24T00:15:26.203 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:26.200+0000 7f46e4800640 1 -- 172.21.15.118:0/4148933722 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f46dc1a2000 con 0x7f46dc104ad0 2024-03-24T00:15:26.203 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:15:25 smithi189 ceph-mon[36187]: from='client.? 172.21.15.118:0/2760328151' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-24T00:15:26.203 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:15:25 smithi189 ceph-mon[36187]: from='client.? 172.21.15.118:0/2447638366' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-24T00:15:26.204 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:26.201+0000 7f46e4800640 1 -- 172.21.15.118:0/4148933722 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f46dc10ac00 con 0x7f46dc104ad0 2024-03-24T00:15:26.206 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:26.205+0000 7f46d6a00640 1 -- 172.21.15.118:0/4148933722 <== mon.1 v2:172.21.15.193:3300/0 4 ==== mgrmap(e 18) v1 ==== 137291+0+0 (secure 0 0 0) 0x7f46cc010ac0 con 0x7f46dc104ad0 2024-03-24T00:15:26.207 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:26.207+0000 7f46d6a00640 1 --2- 172.21.15.118:0/4148933722 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7f46c40a2880 0x7f46c40a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:15:26.207 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:26.207+0000 7f46d6a00640 1 -- 172.21.15.118:0/4148933722 <== mon.1 v2:172.21.15.193:3300/0 5 ==== osd_map(90..90 src has 1..90) v4 ==== 8277+0+0 (secure 0 0 0) 0x7f46cc0cee30 con 0x7f46dc104ad0 2024-03-24T00:15:26.208 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:26.207+0000 7f46e1e00640 1 --2- 172.21.15.118:0/4148933722 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7f46c40a2880 0x7f46c40a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:15:26.208 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:26.208+0000 7f46e1e00640 1 --2- 172.21.15.118:0/4148933722 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7f46c40a2880 0x7f46c40a4d40 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7f46d002c170 tx=0x7f46d002df90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:15:26.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:15:25 smithi193 ceph-mon[36878]: from='client.? 172.21.15.118:0/2760328151' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-24T00:15:26.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:15:25 smithi193 ceph-mon[36878]: from='client.? 172.21.15.118:0/2447638366' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-24T00:15:26.211 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:26.211+0000 7f46d6a00640 1 -- 172.21.15.118:0/4148933722 <== mon.1 v2:172.21.15.193:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f46cc09be40 con 0x7f46dc104ad0 2024-03-24T00:15:26.437 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:26.436+0000 7f46e4800640 1 -- 172.21.15.118:0/4148933722 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f46dc0622a0 con 0x7f46dc104ad0 2024-03-24T00:15:26.439 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:26.438+0000 7f46d6a00640 1 -- 172.21.15.118:0/4148933722 <== mon.1 v2:172.21.15.193:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v90) v1 ==== 74+0+22834 (secure 0 0 0) 0x7f46cc09b740 con 0x7f46dc104ad0 2024-03-24T00:15:26.439 INFO:teuthology.orchestra.run.smithi118.stdout: 2024-03-24T00:15:26.440 INFO:teuthology.orchestra.run.smithi118.stdout:{"epoch":90,"fsid":"d79a5420-e971-11ee-95cd-87774f69a715","created":"2024-03-24T00:03:58.648524+0000","modified":"2024-03-24T00:15:07.721960+0000","last_up_change":"2024-03-24T00:12:03.809000+0000","last_in_change":"2024-03-24T00:11:44.389344+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-24T00:07:50.190965+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-24T00:12:59.993723+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-24T00:13:01.257347+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"86","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"c9590426-3901-4f68-a5ef-d57aa8fbac92","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6802","nonce":557690061},{"type":"v1","addr":"172.21.15.118:6803","nonce":557690061}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6804","nonce":557690061},{"type":"v1","addr":"172.21.15.118:6805","nonce":557690061}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6808","nonce":557690061},{"type":"v1","addr":"172.21.15.118:6809","nonce":557690061}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6806","nonce":557690061},{"type":"v1","addr":"172.21.15.118:6807","nonce":557690061}]},"public_addr":"172.21.15.118:6803/557690061","cluster_addr":"172.21.15.118:6805/557690061","heartbeat_back_addr":"172.21.15.118:6809/557690061","heartbeat_front_addr":"172.21.15.118:6807/557690061","state":["exists","up"]},{"osd":1,"uuid":"fc7f827e-ecf6-4661-aad0-7a22951892e0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6800","nonce":4203029485},{"type":"v1","addr":"172.21.15.189:6801","nonce":4203029485}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6802","nonce":4203029485},{"type":"v1","addr":"172.21.15.189:6803","nonce":4203029485}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6806","nonce":4203029485},{"type":"v1","addr":"172.21.15.189:6807","nonce":4203029485}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6804","nonce":4203029485},{"type":"v1","addr":"172.21.15.189:6805","nonce":4203029485}]},"public_addr":"172.21.15.189:6801/4203029485","cluster_addr":"172.21.15.189:6803/4203029485","heartbeat_back_addr":"172.21.15.189:6807/4203029485","heartbeat_front_addr":"172.21.15.189:6805/4203029485","state":["exists","up"]},{"osd":2,"uuid":"e73ccfe9-0e5b-4734-a154-a2f0d829fcfb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6800","nonce":3699076971},{"type":"v1","addr":"172.21.15.193:6801","nonce":3699076971}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6802","nonce":3699076971},{"type":"v1","addr":"172.21.15.193:6803","nonce":3699076971}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6806","nonce":3699076971},{"type":"v1","addr":"172.21.15.193:6807","nonce":3699076971}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6804","nonce":3699076971},{"type":"v1","addr":"172.21.15.193:6805","nonce":3699076971}]},"public_addr":"172.21.15.193:6801/3699076971","cluster_addr":"172.21.15.193:6803/3699076971","heartbeat_back_addr":"172.21.15.193:6807/3699076971","heartbeat_front_addr":"172.21.15.193:6805/3699076971","state":["exists","up"]},{"osd":3,"uuid":"63fed43b-43ce-4031-8957-7fe94725c1ed","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6810","nonce":4059617010},{"type":"v1","addr":"172.21.15.118:6811","nonce":4059617010}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6812","nonce":4059617010},{"type":"v1","addr":"172.21.15.118:6813","nonce":4059617010}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6816","nonce":4059617010},{"type":"v1","addr":"172.21.15.118:6817","nonce":4059617010}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6814","nonce":4059617010},{"type":"v1","addr":"172.21.15.118:6815","nonce":4059617010}]},"public_addr":"172.21.15.118:6811/4059617010","cluster_addr":"172.21.15.118:6813/4059617010","heartbeat_back_addr":"172.21.15.118:6817/4059617010","heartbeat_front_addr":"172.21.15.118:6815/4059617010","state":["exists","up"]},{"osd":4,"uuid":"57377f66-c3b4-43c1-8e8e-bc873672906d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":34,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6808","nonce":4075247068},{"type":"v1","addr":"172.21.15.189:6809","nonce":4075247068}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6810","nonce":4075247068},{"type":"v1","addr":"172.21.15.189:6811","nonce":4075247068}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6814","nonce":4075247068},{"type":"v1","addr":"172.21.15.189:6815","nonce":4075247068}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6812","nonce":4075247068},{"type":"v1","addr":"172.21.15.189:6813","nonce":4075247068}]},"public_addr":"172.21.15.189:6809/4075247068","cluster_addr":"172.21.15.189:6811/4075247068","heartbeat_back_addr":"172.21.15.189:6815/4075247068","heartbeat_front_addr":"172.21.15.189:6813/4075247068","state":["exists","up"]},{"osd":5,"uuid":"91970976-2193-4224-81c0-dc96cb022c85","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":39,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6808","nonce":2920363402},{"type":"v1","addr":"172.21.15.193:6809","nonce":2920363402}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6810","nonce":2920363402},{"type":"v1","addr":"172.21.15.193:6811","nonce":2920363402}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6814","nonce":2920363402},{"type":"v1","addr":"172.21.15.193:6815","nonce":2920363402}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6812","nonce":2920363402},{"type":"v1","addr":"172.21.15.193:6813","nonce":2920363402}]},"public_addr":"172.21.15.193:6809/2920363402","cluster_addr":"172.21.15.193:6811/2920363402","heartbeat_back_addr":"172.21.15.193:6815/2920363402","heartbeat_front_addr":"172.21.15.193:6813/2920363402","state":["exists","up"]},{"osd":6,"uuid":"0b77b196-8674-41c6-ad3d-726b5d01c8fd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":47,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6818","nonce":431354576},{"type":"v1","addr":"172.21.15.118:6819","nonce":431354576}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6820","nonce":431354576},{"type":"v1","addr":"172.21.15.118:6821","nonce":431354576}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6824","nonce":431354576},{"type":"v1","addr":"172.21.15.118:6825","nonce":431354576}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6822","nonce":431354576},{"type":"v1","addr":"172.21.15.118:6823","nonce":431354576}]},"public_addr":"172.21.15.118:6819/431354576","cluster_addr":"172.21.15.118:6821/431354576","heartbeat_back_addr":"172.21.15.118:6825/431354576","heartbeat_front_addr":"172.21.15.118:6823/431354576","state":["exists","up"]},{"osd":7,"uuid":"1fbbd38f-4e8e-4d4c-8e98-66f6862bc4bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":52,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6816","nonce":3667611284},{"type":"v1","addr":"172.21.15.189:6817","nonce":3667611284}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6818","nonce":3667611284},{"type":"v1","addr":"172.21.15.189:6819","nonce":3667611284}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6822","nonce":3667611284},{"type":"v1","addr":"172.21.15.189:6823","nonce":3667611284}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6820","nonce":3667611284},{"type":"v1","addr":"172.21.15.189:6821","nonce":3667611284}]},"public_addr":"172.21.15.189:6817/3667611284","cluster_addr":"172.21.15.189:6819/3667611284","heartbeat_back_addr":"172.21.15.189:6823/3667611284","heartbeat_front_addr":"172.21.15.189:6821/3667611284","state":["exists","up"]},{"osd":8,"uuid":"6462c2b5-e4b3-4b0b-bcfa-4de2c89e93e0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":60,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6816","nonce":3853800910},{"type":"v1","addr":"172.21.15.193:6817","nonce":3853800910}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6818","nonce":3853800910},{"type":"v1","addr":"172.21.15.193:6819","nonce":3853800910}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6822","nonce":3853800910},{"type":"v1","addr":"172.21.15.193:6823","nonce":3853800910}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6820","nonce":3853800910},{"type":"v1","addr":"172.21.15.193:6821","nonce":3853800910}]},"public_addr":"172.21.15.193:6817/3853800910","cluster_addr":"172.21.15.193:6819/3853800910","heartbeat_back_addr":"172.21.15.193:6823/3853800910","heartbeat_front_addr":"172.21.15.193:6821/3853800910","state":["exists","up"]},{"osd":9,"uuid":"54be277f-3596-422f-9de5-9005a8571c6f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":68,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6826","nonce":3248293936},{"type":"v1","addr":"172.21.15.118:6827","nonce":3248293936}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6828","nonce":3248293936},{"type":"v1","addr":"172.21.15.118:6829","nonce":3248293936}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6832","nonce":3248293936},{"type":"v1","addr":"172.21.15.118:6833","nonce":3248293936}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6830","nonce":3248293936},{"type":"v1","addr":"172.21.15.118:6831","nonce":3248293936}]},"public_addr":"172.21.15.118:6827/3248293936","cluster_addr":"172.21.15.118:6829/3248293936","heartbeat_back_addr":"172.21.15.118:6833/3248293936","heartbeat_front_addr":"172.21.15.118:6831/3248293936","state":["exists","up"]},{"osd":10,"uuid":"7ba69d69-ac41-44ce-bc39-c049a8ca5900","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":73,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6824","nonce":2112195760},{"type":"v1","addr":"172.21.15.189:6825","nonce":2112195760}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6826","nonce":2112195760},{"type":"v1","addr":"172.21.15.189:6827","nonce":2112195760}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6830","nonce":2112195760},{"type":"v1","addr":"172.21.15.189:6831","nonce":2112195760}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6828","nonce":2112195760},{"type":"v1","addr":"172.21.15.189:6829","nonce":2112195760}]},"public_addr":"172.21.15.189:6825/2112195760","cluster_addr":"172.21.15.189:6827/2112195760","heartbeat_back_addr":"172.21.15.189:6831/2112195760","heartbeat_front_addr":"172.21.15.189:6829/2112195760","state":["exists","up"]},{"osd":11,"uuid":"186f0b18-6b22-4eae-83b7-b21bbcce0817","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":81,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6824","nonce":3880515510},{"type":"v1","addr":"172.21.15.193:6825","nonce":3880515510}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6826","nonce":3880515510},{"type":"v1","addr":"172.21.15.193:6827","nonce":3880515510}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6830","nonce":3880515510},{"type":"v1","addr":"172.21.15.193:6831","nonce":3880515510}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6828","nonce":3880515510},{"type":"v1","addr":"172.21.15.193:6829","nonce":3880515510}]},"public_addr":"172.21.15.193:6825/3880515510","cluster_addr":"172.21.15.193:6827/3880515510","heartbeat_back_addr":"172.21.15.193:6831/3880515510","heartbeat_front_addr":"172.21.15.193:6829/3880515510","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:06:58.105543+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:07:23.795866+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:07:47.426050+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:08:19.427073+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:08:45.816082+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:09:11.081523+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:09:44.485077+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:10:11.338972+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:10:36.234024+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:11:09.730432+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:11:37.059083+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:12:02.371947+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.a","mappings":[{"from":2,"to":11}]},{"pgid":"2.14","mappings":[{"from":2,"to":4}]},{"pgid":"2.1d","mappings":[{"from":8,"to":11}]},{"pgid":"2.1f","mappings":[{"from":8,"to":1}]},{"pgid":"3.11","mappings":[{"from":4,"to":8},{"from":5,"to":9}]},{"pgid":"3.13","mappings":[{"from":4,"to":9}]},{"pgid":"3.14","mappings":[{"from":4,"to":6}]},{"pgid":"3.31","mappings":[{"from":4,"to":7}]},{"pgid":"3.38","mappings":[{"from":3,"to":9}]},{"pgid":"3.39","mappings":[{"from":5,"to":8}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.118:6801/389249272":"2024-03-25T00:05:05.458796+0000","172.21.15.118:6800/389249272":"2024-03-25T00:05:05.458796+0000","172.21.15.118:0/2101854694":"2024-03-25T00:05:05.458796+0000","172.21.15.118:0/630779359":"2024-03-25T00:05:05.458796+0000","172.21.15.118:0/2158921029":"2024-03-25T00:04:31.283725+0000","172.21.15.118:0/1003670455":"2024-03-25T00:05:05.458796+0000","172.21.15.118:0/368712314":"2024-03-25T00:04:31.283725+0000","172.21.15.118:0/2004775597":"2024-03-25T00:04:31.283725+0000","172.21.15.118:6800/1463147955":"2024-03-25T00:04:31.283725+0000","172.21.15.118:6801/1463147955":"2024-03-25T00:04:31.283725+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-24T00:15:26.443 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:26.442+0000 7f46e4800640 1 -- 172.21.15.118:0/4148933722 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7f46c40a2880 msgr2=0x7f46c40a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:15:26.443 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:26.442+0000 7f46e4800640 1 --2- 172.21.15.118:0/4148933722 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7f46c40a2880 0x7f46c40a4d40 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7f46d002c170 tx=0x7f46d002df90 comp rx=0 tx=0).stop 2024-03-24T00:15:26.443 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:26.442+0000 7f46e4800640 1 -- 172.21.15.118:0/4148933722 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f46dc104ad0 msgr2=0x7f46dc19a130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:15:26.443 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:26.442+0000 7f46e4800640 1 --2- 172.21.15.118:0/4148933722 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f46dc104ad0 0x7f46dc19a130 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f46cc00e9f0 tx=0x7f46cc00eeb0 comp rx=0 tx=0).stop 2024-03-24T00:15:26.443 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:26.442+0000 7f46e4800640 1 -- 172.21.15.118:0/4148933722 shutdown_connections 2024-03-24T00:15:26.443 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:26.442+0000 7f46e4800640 1 --2- 172.21.15.118:0/4148933722 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7f46c40a2880 0x7f46c40a4d40 unknown :-1 s=CLOSED pgs=237 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:15:26.443 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:26.442+0000 7f46e4800640 1 --2- 172.21.15.118:0/4148933722 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f46dc105490 0x7f46dc1a11a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:15:26.443 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:26.442+0000 7f46e4800640 1 --2- 172.21.15.118:0/4148933722 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f46dc104ad0 0x7f46dc19a130 unknown :-1 s=CLOSED pgs=164 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:15:26.443 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:26.442+0000 7f46e4800640 1 --2- 172.21.15.118:0/4148933722 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f46dc1038e0 0x7f46dc199bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:15:26.443 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:26.442+0000 7f46e4800640 1 -- 172.21.15.118:0/4148933722 >> 172.21.15.118:0/4148933722 conn(0x7f46dc0fee90 msgr2=0x7f46dc107dd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:15:26.443 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:26.442+0000 7f46e4800640 1 -- 172.21.15.118:0/4148933722 shutdown_connections 2024-03-24T00:15:26.444 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:26.442+0000 7f46e4800640 1 -- 172.21.15.118:0/4148933722 wait complete. 2024-03-24T00:15:26.463 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-24T00:07:50.190965+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-24T00:12:59.993723+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '85', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.8799999952316284, 'score_stable': 1.8799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.8799999952316284, 'raw_score_stable': 1.8799999952316284, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-03-24T00:13:01.257347+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '86', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-24T00:15:26.463 DEBUG:teuthology.orchestra.run.smithi118:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-24T00:15:26.679 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:26.674+0000 7f806d800640 1 -- 172.21.15.118:0/1974185060 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f8068101b50 msgr2=0x7f8068101f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:15:26.679 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:26.674+0000 7f806d800640 1 --2- 172.21.15.118:0/1974185060 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f8068101b50 0x7f8068101f70 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f8058009860 tx=0x7f805802bb70 comp rx=0 tx=0).stop 2024-03-24T00:15:26.679 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:26.674+0000 7f806d800640 1 -- 172.21.15.118:0/1974185060 shutdown_connections 2024-03-24T00:15:26.679 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:26.674+0000 7f806d800640 1 --2- 172.21.15.118:0/1974185060 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8068101240 0x7f8068108ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:15:26.680 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:26.674+0000 7f806d800640 1 --2- 172.21.15.118:0/1974185060 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f8068100880 0x7f8068100d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:15:26.680 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:26.674+0000 7f806d800640 1 --2- 172.21.15.118:0/1974185060 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f8068101b50 0x7f8068101f70 unknown :-1 s=CLOSED pgs=165 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:15:26.680 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:26.674+0000 7f806d800640 1 -- 172.21.15.118:0/1974185060 >> 172.21.15.118:0/1974185060 conn(0x7f80680fb290 msgr2=0x7f80680fd6f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:15:26.680 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:26.674+0000 7f806d800640 1 -- 172.21.15.118:0/1974185060 shutdown_connections 2024-03-24T00:15:26.680 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:26.675+0000 7f806d800640 1 -- 172.21.15.118:0/1974185060 wait complete. 2024-03-24T00:15:26.680 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:26.675+0000 7f806d800640 1 Processor -- start 2024-03-24T00:15:26.681 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:26.676+0000 7f806d800640 1 -- start start 2024-03-24T00:15:26.681 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:26.676+0000 7f806d800640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8068100880 0x7f8068197a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:15:26.681 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:26.676+0000 7f806d800640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f8068101240 0x7f8068197f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:15:26.681 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:26.676+0000 7f806d800640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f8068101b50 0x7f806819efd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:15:26.681 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:26.676+0000 7f806d800640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f806810d370 con 0x7f8068101b50 2024-03-24T00:15:26.681 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:26.676+0000 7f806d800640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f806810d1f0 con 0x7f8068100880 2024-03-24T00:15:26.681 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:26.676+0000 7f806d800640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f806810d4f0 con 0x7f8068101240 2024-03-24T00:15:26.682 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:26.676+0000 7f8066a00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8068100880 0x7f8068197a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:15:26.682 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:26.676+0000 7f8066000640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f8068101240 0x7f8068197f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:15:26.682 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:26.676+0000 7f8066a00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8068100880 0x7f8068197a20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.118:54828/0 (socket says 172.21.15.118:54828) 2024-03-24T00:15:26.682 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:26.676+0000 7f8066a00640 1 -- 172.21.15.118:0/3914240704 learned_addr learned my addr 172.21.15.118:0/3914240704 (peer_addr_for_me v2:172.21.15.118:0/0) 2024-03-24T00:15:26.682 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:26.676+0000 7f8067400640 1 --2- 172.21.15.118:0/3914240704 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f8068101b50 0x7f806819efd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:15:26.682 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:26.677+0000 7f8066a00640 1 -- 172.21.15.118:0/3914240704 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f8068101240 msgr2=0x7f8068197f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:15:26.682 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:26.677+0000 7f8066a00640 1 --2- 172.21.15.118:0/3914240704 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f8068101240 0x7f8068197f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:15:26.682 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:26.677+0000 7f8066a00640 1 -- 172.21.15.118:0/3914240704 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f8068101b50 msgr2=0x7f806819efd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:15:26.683 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:26.677+0000 7f8066a00640 1 --2- 172.21.15.118:0/3914240704 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f8068101b50 0x7f806819efd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:15:26.683 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:26.677+0000 7f8066a00640 1 -- 172.21.15.118:0/3914240704 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f806819f640 con 0x7f8068100880 2024-03-24T00:15:26.683 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:26.677+0000 7f8066000640 1 --2- 172.21.15.118:0/3914240704 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f8068101240 0x7f8068197f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T00:15:26.683 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:26.677+0000 7f8066a00640 1 --2- 172.21.15.118:0/3914240704 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8068100880 0x7f8068197a20 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f80580349e0 tx=0x7f8058032430 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:15:26.683 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:26.677+0000 7f805f400640 1 -- 172.21.15.118:0/3914240704 <== mon.2 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8058041020 con 0x7f8068100880 2024-03-24T00:15:26.683 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:26.677+0000 7f805f400640 1 -- 172.21.15.118:0/3914240704 <== mon.2 v2:172.21.15.189:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f805802c9f0 con 0x7f8068100880 2024-03-24T00:15:26.683 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:26.677+0000 7f806d800640 1 -- 172.21.15.118:0/3914240704 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f80681093a0 con 0x7f8068100880 2024-03-24T00:15:26.684 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:26.677+0000 7f805f400640 1 -- 172.21.15.118:0/3914240704 <== mon.2 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8058039040 con 0x7f8068100880 2024-03-24T00:15:26.684 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:26.678+0000 7f806d800640 1 -- 172.21.15.118:0/3914240704 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8068109890 con 0x7f8068100880 2024-03-24T00:15:26.685 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:26.679+0000 7f806d800640 1 -- 172.21.15.118:0/3914240704 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8068107ab0 con 0x7f8068100880 2024-03-24T00:15:26.686 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:26.682+0000 7f805f400640 1 -- 172.21.15.118:0/3914240704 <== mon.2 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 18) v1 ==== 137291+0+0 (secure 0 0 0) 0x7f8058044020 con 0x7f8068100880 2024-03-24T00:15:26.686 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:26.683+0000 7f805f400640 1 --2- 172.21.15.118:0/3914240704 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7f804c0a2800 0x7f804c0a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:15:26.686 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:26.683+0000 7f805f400640 1 -- 172.21.15.118:0/3914240704 <== mon.2 v2:172.21.15.189:3300/0 5 ==== osd_map(90..90 src has 1..90) v4 ==== 8277+0+0 (secure 0 0 0) 0x7f80580f02d0 con 0x7f8068100880 2024-03-24T00:15:26.686 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:26.683+0000 7f8066000640 1 --2- 172.21.15.118:0/3914240704 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7f804c0a2800 0x7f804c0a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:15:26.686 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:26.684+0000 7f8066000640 1 --2- 172.21.15.118:0/3914240704 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7f804c0a2800 0x7f804c0a4cc0 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7f8068198f30 tx=0x7f8054005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:15:26.687 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:26.687+0000 7f805f400640 1 -- 172.21.15.118:0/3914240704 <== mon.2 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f805804b020 con 0x7f8068100880 2024-03-24T00:15:26.887 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:26.886+0000 7f806d800640 1 -- 172.21.15.118:0/3914240704 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f806802d290 con 0x7f8068100880 2024-03-24T00:15:26.887 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:26.887+0000 7f805f400640 1 -- 172.21.15.118:0/3914240704 <== mon.2 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v90) v1 ==== 93+0+10 (secure 0 0 0) 0x7f80580bd340 con 0x7f8068100880 2024-03-24T00:15:26.888 INFO:teuthology.orchestra.run.smithi118.stdout:pg_num: 1 2024-03-24T00:15:26.890 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:26.889+0000 7f806d800640 1 -- 172.21.15.118:0/3914240704 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7f804c0a2800 msgr2=0x7f804c0a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:15:26.890 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:26.889+0000 7f806d800640 1 --2- 172.21.15.118:0/3914240704 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7f804c0a2800 0x7f804c0a4cc0 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7f8068198f30 tx=0x7f8054005f90 comp rx=0 tx=0).stop 2024-03-24T00:15:26.890 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:26.889+0000 7f806d800640 1 -- 172.21.15.118:0/3914240704 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8068100880 msgr2=0x7f8068197a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:15:26.890 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:26.889+0000 7f806d800640 1 --2- 172.21.15.118:0/3914240704 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8068100880 0x7f8068197a20 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7f80580349e0 tx=0x7f8058032430 comp rx=0 tx=0).stop 2024-03-24T00:15:26.890 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:26.889+0000 7f806d800640 1 -- 172.21.15.118:0/3914240704 shutdown_connections 2024-03-24T00:15:26.891 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:26.889+0000 7f806d800640 1 --2- 172.21.15.118:0/3914240704 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7f804c0a2800 0x7f804c0a4cc0 unknown :-1 s=CLOSED pgs=238 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:15:26.891 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:26.889+0000 7f806d800640 1 --2- 172.21.15.118:0/3914240704 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f8068101b50 0x7f806819efd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:15:26.891 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:26.889+0000 7f806d800640 1 --2- 172.21.15.118:0/3914240704 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f8068101240 0x7f8068197f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:15:26.891 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:26.889+0000 7f806d800640 1 --2- 172.21.15.118:0/3914240704 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8068100880 0x7f8068197a20 unknown :-1 s=CLOSED pgs=159 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:15:26.891 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:26.889+0000 7f806d800640 1 -- 172.21.15.118:0/3914240704 >> 172.21.15.118:0/3914240704 conn(0x7f80680fb290 msgr2=0x7f8068104c80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:15:26.891 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:26.889+0000 7f806d800640 1 -- 172.21.15.118:0/3914240704 shutdown_connections 2024-03-24T00:15:26.891 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:26.890+0000 7f806d800640 1 -- 172.21.15.118:0/3914240704 wait complete. 2024-03-24T00:15:26.912 DEBUG:teuthology.orchestra.run.smithi118:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-24T00:15:27.115 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:27.114+0000 7f536ca00640 1 -- 172.21.15.118:0/1670860229 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5368101210 msgr2=0x7f5368101690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:15:27.115 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:27.114+0000 7f536ca00640 1 --2- 172.21.15.118:0/1670860229 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5368101210 0x7f5368101690 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 crypto rx=0x7f5360009ca0 tx=0x7f536002c100 comp rx=0 tx=0).stop 2024-03-24T00:15:27.115 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:27.114+0000 7f536ca00640 1 -- 172.21.15.118:0/1670860229 shutdown_connections 2024-03-24T00:15:27.115 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:27.114+0000 7f536ca00640 1 --2- 172.21.15.118:0/1670860229 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5368101210 0x7f5368101690 unknown :-1 s=CLOSED pgs=160 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:15:27.115 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:27.114+0000 7f536ca00640 1 --2- 172.21.15.118:0/1670860229 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f5368100870 0x7f5368100cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:15:27.115 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:27.114+0000 7f536ca00640 1 --2- 172.21.15.118:0/1670860229 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f53680fff10 0x7f5368100330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:15:27.116 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:27.114+0000 7f536ca00640 1 -- 172.21.15.118:0/1670860229 >> 172.21.15.118:0/1670860229 conn(0x7f53680fbb30 msgr2=0x7f53680fdf50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:15:27.116 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:27.114+0000 7f536ca00640 1 -- 172.21.15.118:0/1670860229 shutdown_connections 2024-03-24T00:15:27.116 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:27.114+0000 7f536ca00640 1 -- 172.21.15.118:0/1670860229 wait complete. 2024-03-24T00:15:27.117 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:27.115+0000 7f536ca00640 1 Processor -- start 2024-03-24T00:15:27.117 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:27.116+0000 7f536ca00640 1 -- start start 2024-03-24T00:15:27.117 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:27.116+0000 7f536ca00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f53680fff10 0x7f5368199b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:15:27.117 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:27.116+0000 7f536ca00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f5368100870 0x7f536819a0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:15:27.117 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:27.116+0000 7f536ca00640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f5368101210 0x7f53681a1130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:15:27.118 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:27.116+0000 7f536ca00640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f536810f510 con 0x7f5368101210 2024-03-24T00:15:27.118 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:27.116+0000 7f536ca00640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f536810f390 con 0x7f53680fff10 2024-03-24T00:15:27.118 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:27.116+0000 7f536ca00640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f536810f690 con 0x7f5368100870 2024-03-24T00:15:27.118 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:27.116+0000 7f5365e00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f53680fff10 0x7f5368199b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:15:27.118 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:27.116+0000 7f535fe00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f5368100870 0x7f536819a0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:15:27.118 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:27.116+0000 7f5365e00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f53680fff10 0x7f5368199b80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.118:54838/0 (socket says 172.21.15.118:54838) 2024-03-24T00:15:27.118 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:27.116+0000 7f5365e00640 1 -- 172.21.15.118:0/2420781385 learned_addr learned my addr 172.21.15.118:0/2420781385 (peer_addr_for_me v2:172.21.15.118:0/0) 2024-03-24T00:15:27.118 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:27.116+0000 7f5366800640 1 --2- 172.21.15.118:0/2420781385 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f5368101210 0x7f53681a1130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:15:27.118 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:27.117+0000 7f535fe00640 1 -- 172.21.15.118:0/2420781385 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f53680fff10 msgr2=0x7f5368199b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:15:27.118 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:27.117+0000 7f535fe00640 1 --2- 172.21.15.118:0/2420781385 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f53680fff10 0x7f5368199b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:15:27.118 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:27.117+0000 7f535fe00640 1 -- 172.21.15.118:0/2420781385 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f5368101210 msgr2=0x7f53681a1130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:15:27.118 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:27.117+0000 7f535fe00640 1 --2- 172.21.15.118:0/2420781385 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f5368101210 0x7f53681a1130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:15:27.118 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:27.117+0000 7f535fe00640 1 -- 172.21.15.118:0/2420781385 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f53681a17a0 con 0x7f5368100870 2024-03-24T00:15:27.118 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:27.117+0000 7f5365e00640 1 --2- 172.21.15.118:0/2420781385 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f53680fff10 0x7f5368199b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T00:15:27.119 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:27.117+0000 7f535fe00640 1 --2- 172.21.15.118:0/2420781385 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f5368100870 0x7f536819a0c0 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7f535000d6d0 tx=0x7f535000db90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:15:27.119 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:27.117+0000 7f535ea00640 1 -- 172.21.15.118:0/2420781385 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5350017070 con 0x7f5368100870 2024-03-24T00:15:27.119 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:27.117+0000 7f536ca00640 1 -- 172.21.15.118:0/2420781385 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f53681a1a90 con 0x7f5368100870 2024-03-24T00:15:27.120 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:27.117+0000 7f535ea00640 1 -- 172.21.15.118:0/2420781385 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f5350009820 con 0x7f5368100870 2024-03-24T00:15:27.120 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:27.117+0000 7f535ea00640 1 -- 172.21.15.118:0/2420781385 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5350004cd0 con 0x7f5368100870 2024-03-24T00:15:27.120 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:27.117+0000 7f536ca00640 1 -- 172.21.15.118:0/2420781385 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f53681a1fe0 con 0x7f5368100870 2024-03-24T00:15:27.120 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:27.119+0000 7f536ca00640 1 -- 172.21.15.118:0/2420781385 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f536810ae00 con 0x7f5368100870 2024-03-24T00:15:27.121 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:27.121+0000 7f535ea00640 1 -- 172.21.15.118:0/2420781385 <== mon.1 v2:172.21.15.193:3300/0 4 ==== mgrmap(e 18) v1 ==== 137291+0+0 (secure 0 0 0) 0x7f53500099d0 con 0x7f5368100870 2024-03-24T00:15:27.122 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:27.122+0000 7f535ea00640 1 --2- 172.21.15.118:0/2420781385 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7f534c0a2800 0x7f534c0a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:15:27.123 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:27.122+0000 7f535ea00640 1 -- 172.21.15.118:0/2420781385 <== mon.1 v2:172.21.15.193:3300/0 5 ==== osd_map(90..90 src has 1..90) v4 ==== 8277+0+0 (secure 0 0 0) 0x7f53500ce5c0 con 0x7f5368100870 2024-03-24T00:15:27.123 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:27.122+0000 7f5365e00640 1 --2- 172.21.15.118:0/2420781385 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7f534c0a2800 0x7f534c0a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:15:27.123 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:27.123+0000 7f5365e00640 1 --2- 172.21.15.118:0/2420781385 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7f534c0a2800 0x7f534c0a4cc0 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7f5358011500 tx=0x7f5358005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:15:27.131 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:27.130+0000 7f535ea00640 1 -- 172.21.15.118:0/2420781385 <== mon.1 v2:172.21.15.193:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f535009af80 con 0x7f5368100870 2024-03-24T00:15:27.195 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:15:26 smithi118 ceph-mon[31535]: pgmap v390: 97 pgs: 97 active+clean; 645 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 10 op/s 2024-03-24T00:15:27.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:15:26 smithi118 ceph-mon[31535]: from='client.? 172.21.15.118:0/3314792481' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-24T00:15:27.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:15:26 smithi118 ceph-mon[31535]: from='client.? 172.21.15.118:0/4148933722' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-24T00:15:27.203 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:15:26 smithi189 ceph-mon[36187]: pgmap v390: 97 pgs: 97 active+clean; 645 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 10 op/s 2024-03-24T00:15:27.203 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:15:26 smithi189 ceph-mon[36187]: from='client.? 172.21.15.118:0/3314792481' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-24T00:15:27.203 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:15:26 smithi189 ceph-mon[36187]: from='client.? 172.21.15.118:0/4148933722' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-24T00:15:27.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:15:26 smithi193 ceph-mon[36878]: pgmap v390: 97 pgs: 97 active+clean; 645 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 10 op/s 2024-03-24T00:15:27.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:15:26 smithi193 ceph-mon[36878]: from='client.? 172.21.15.118:0/3314792481' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-24T00:15:27.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:15:26 smithi193 ceph-mon[36878]: from='client.? 172.21.15.118:0/4148933722' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-24T00:15:27.332 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:27.331+0000 7f536ca00640 1 -- 172.21.15.118:0/2420781385 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f53680622a0 con 0x7f5368100870 2024-03-24T00:15:27.332 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:27.332+0000 7f535ea00640 1 -- 172.21.15.118:0/2420781385 <== mon.1 v2:172.21.15.193:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v90) v1 ==== 104+0+11 (secure 0 0 0) 0x7f535009f020 con 0x7f5368100870 2024-03-24T00:15:27.332 INFO:teuthology.orchestra.run.smithi118.stdout:pg_num: 32 2024-03-24T00:15:27.335 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:27.334+0000 7f536ca00640 1 -- 172.21.15.118:0/2420781385 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7f534c0a2800 msgr2=0x7f534c0a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:15:27.335 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:27.334+0000 7f536ca00640 1 --2- 172.21.15.118:0/2420781385 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7f534c0a2800 0x7f534c0a4cc0 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7f5358011500 tx=0x7f5358005f90 comp rx=0 tx=0).stop 2024-03-24T00:15:27.335 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:27.334+0000 7f536ca00640 1 -- 172.21.15.118:0/2420781385 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f5368100870 msgr2=0x7f536819a0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:15:27.335 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:27.334+0000 7f536ca00640 1 --2- 172.21.15.118:0/2420781385 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f5368100870 0x7f536819a0c0 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7f535000d6d0 tx=0x7f535000db90 comp rx=0 tx=0).stop 2024-03-24T00:15:27.335 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:27.334+0000 7f536ca00640 1 -- 172.21.15.118:0/2420781385 shutdown_connections 2024-03-24T00:15:27.335 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:27.334+0000 7f536ca00640 1 --2- 172.21.15.118:0/2420781385 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7f534c0a2800 0x7f534c0a4cc0 unknown :-1 s=CLOSED pgs=239 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:15:27.336 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:27.334+0000 7f536ca00640 1 --2- 172.21.15.118:0/2420781385 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f5368101210 0x7f53681a1130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:15:27.336 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:27.334+0000 7f536ca00640 1 --2- 172.21.15.118:0/2420781385 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f5368100870 0x7f536819a0c0 unknown :-1 s=CLOSED pgs=166 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:15:27.336 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:27.334+0000 7f536ca00640 1 --2- 172.21.15.118:0/2420781385 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f53680fff10 0x7f5368199b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:15:27.336 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:27.334+0000 7f536ca00640 1 -- 172.21.15.118:0/2420781385 >> 172.21.15.118:0/2420781385 conn(0x7f53680fbb30 msgr2=0x7f53680fd240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:15:27.336 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:27.335+0000 7f536ca00640 1 -- 172.21.15.118:0/2420781385 shutdown_connections 2024-03-24T00:15:27.336 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:27.335+0000 7f536ca00640 1 -- 172.21.15.118:0/2420781385 wait complete. 2024-03-24T00:15:27.357 DEBUG:teuthology.orchestra.run.smithi118:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-24T00:15:27.574 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:27.573+0000 7f977be00640 1 -- 172.21.15.118:0/3619822027 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f977c101670 msgr2=0x7f977c101af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:15:27.575 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:27.573+0000 7f977be00640 1 --2- 172.21.15.118:0/3619822027 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f977c101670 0x7f977c101af0 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 crypto rx=0x7f9768009900 tx=0x7f976802bc50 comp rx=0 tx=0).stop 2024-03-24T00:15:27.575 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:27.574+0000 7f977be00640 1 -- 172.21.15.118:0/3619822027 shutdown_connections 2024-03-24T00:15:27.575 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:27.574+0000 7f977be00640 1 --2- 172.21.15.118:0/3619822027 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f977c102030 0x7f977c1088c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:15:27.575 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:27.574+0000 7f977be00640 1 --2- 172.21.15.118:0/3619822027 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f977c101670 0x7f977c101af0 unknown :-1 s=CLOSED pgs=161 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:15:27.575 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:27.574+0000 7f977be00640 1 --2- 172.21.15.118:0/3619822027 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f977c100480 0x7f977c1008a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:15:27.575 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:27.574+0000 7f977be00640 1 -- 172.21.15.118:0/3619822027 >> 172.21.15.118:0/3619822027 conn(0x7f977c0fba10 msgr2=0x7f977c0fde30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:15:27.575 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:27.574+0000 7f977be00640 1 -- 172.21.15.118:0/3619822027 shutdown_connections 2024-03-24T00:15:27.575 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:27.574+0000 7f977be00640 1 -- 172.21.15.118:0/3619822027 wait complete. 2024-03-24T00:15:27.577 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:27.574+0000 7f977be00640 1 Processor -- start 2024-03-24T00:15:27.577 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:27.575+0000 7f977be00640 1 -- start start 2024-03-24T00:15:27.577 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:27.575+0000 7f977be00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f977c100480 0x7f977c199ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:15:27.577 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:27.575+0000 7f977be00640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f977c101670 0x7f977c199ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:15:27.577 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:27.575+0000 7f977be00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f977c102030 0x7f977c1a1060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:15:27.578 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:27.575+0000 7f977be00640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f977c10c160 con 0x7f977c101670 2024-03-24T00:15:27.578 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:27.575+0000 7f977be00640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f977c10bfe0 con 0x7f977c100480 2024-03-24T00:15:27.578 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:27.575+0000 7f977be00640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f977c10c2e0 con 0x7f977c102030 2024-03-24T00:15:27.578 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:27.575+0000 7f977aa00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f977c100480 0x7f977c199ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:15:27.578 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:27.575+0000 7f977a000640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f977c101670 0x7f977c199ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:15:27.578 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:27.575+0000 7f977b400640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f977c102030 0x7f977c1a1060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:15:27.578 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:27.575+0000 7f977aa00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f977c100480 0x7f977c199ab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.118:54852/0 (socket says 172.21.15.118:54852) 2024-03-24T00:15:27.578 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:27.575+0000 7f977aa00640 1 -- 172.21.15.118:0/2051213444 learned_addr learned my addr 172.21.15.118:0/2051213444 (peer_addr_for_me v2:172.21.15.118:0/0) 2024-03-24T00:15:27.578 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:27.576+0000 7f977aa00640 1 -- 172.21.15.118:0/2051213444 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f977c102030 msgr2=0x7f977c1a1060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:15:27.578 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:27.576+0000 7f977aa00640 1 --2- 172.21.15.118:0/2051213444 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f977c102030 0x7f977c1a1060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:15:27.578 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:27.576+0000 7f977aa00640 1 -- 172.21.15.118:0/2051213444 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f977c101670 msgr2=0x7f977c199ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:15:27.578 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:27.576+0000 7f977aa00640 1 --2- 172.21.15.118:0/2051213444 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f977c101670 0x7f977c199ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:15:27.578 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:27.576+0000 7f977aa00640 1 -- 172.21.15.118:0/2051213444 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f977c1a16d0 con 0x7f977c100480 2024-03-24T00:15:27.578 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:27.576+0000 7f977b400640 1 --2- 172.21.15.118:0/2051213444 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f977c102030 0x7f977c1a1060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T00:15:27.579 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:27.576+0000 7f977aa00640 1 --2- 172.21.15.118:0/2051213444 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f977c100480 0x7f977c199ab0 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f976c00e9c0 tx=0x7f976c00ee80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:15:27.579 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:27.576+0000 7f9773400640 1 -- 172.21.15.118:0/2051213444 <== mon.2 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f976c00cd80 con 0x7f977c100480 2024-03-24T00:15:27.579 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:27.576+0000 7f9773400640 1 -- 172.21.15.118:0/2051213444 <== mon.2 v2:172.21.15.189:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f976c0035f0 con 0x7f977c100480 2024-03-24T00:15:27.579 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:27.576+0000 7f9773400640 1 -- 172.21.15.118:0/2051213444 <== mon.2 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f976c010690 con 0x7f977c100480 2024-03-24T00:15:27.579 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:27.576+0000 7f977be00640 1 -- 172.21.15.118:0/2051213444 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f977c1a19c0 con 0x7f977c100480 2024-03-24T00:15:27.579 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:27.576+0000 7f977be00640 1 -- 172.21.15.118:0/2051213444 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f977c1a1e90 con 0x7f977c100480 2024-03-24T00:15:27.579 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:27.577+0000 7f977be00640 1 -- 172.21.15.118:0/2051213444 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f977c1a20c0 con 0x7f977c100480 2024-03-24T00:15:27.579 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:27.579+0000 7f9773400640 1 -- 172.21.15.118:0/2051213444 <== mon.2 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 18) v1 ==== 137291+0+0 (secure 0 0 0) 0x7f976c0037a0 con 0x7f977c100480 2024-03-24T00:15:27.581 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:27.580+0000 7f9773400640 1 --2- 172.21.15.118:0/2051213444 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7f97600a2800 0x7f97600a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:15:27.581 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:27.581+0000 7f9773400640 1 -- 172.21.15.118:0/2051213444 <== mon.2 v2:172.21.15.189:3300/0 5 ==== osd_map(90..90 src has 1..90) v4 ==== 8277+0+0 (secure 0 0 0) 0x7f976c0cee50 con 0x7f977c100480 2024-03-24T00:15:27.581 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:27.581+0000 7f977a000640 1 --2- 172.21.15.118:0/2051213444 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7f97600a2800 0x7f97600a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:15:27.582 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:27.581+0000 7f977a000640 1 --2- 172.21.15.118:0/2051213444 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7f97600a2800 0x7f97600a4cc0 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7f976802c260 tx=0x7f9768000bc0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:15:27.585 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:27.584+0000 7f9773400640 1 -- 172.21.15.118:0/2051213444 <== mon.2 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f976c09be60 con 0x7f977c100480 2024-03-24T00:15:27.817 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:27.816+0000 7f977be00640 1 -- 172.21.15.118:0/2051213444 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f977c1a2290 con 0x7f977c100480 2024-03-24T00:15:27.817 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:27.817+0000 7f9773400640 1 -- 172.21.15.118:0/2051213444 <== mon.2 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v90) v1 ==== 100+0+11 (secure 0 0 0) 0x7f976c09b760 con 0x7f977c100480 2024-03-24T00:15:27.817 INFO:teuthology.orchestra.run.smithi118.stdout:pg_num: 64 2024-03-24T00:15:27.820 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:27.819+0000 7f977be00640 1 -- 172.21.15.118:0/2051213444 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7f97600a2800 msgr2=0x7f97600a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:15:27.820 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:27.819+0000 7f977be00640 1 --2- 172.21.15.118:0/2051213444 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7f97600a2800 0x7f97600a4cc0 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7f976802c260 tx=0x7f9768000bc0 comp rx=0 tx=0).stop 2024-03-24T00:15:27.820 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:27.819+0000 7f977be00640 1 -- 172.21.15.118:0/2051213444 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f977c100480 msgr2=0x7f977c199ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:15:27.820 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:27.819+0000 7f977be00640 1 --2- 172.21.15.118:0/2051213444 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f977c100480 0x7f977c199ab0 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 crypto rx=0x7f976c00e9c0 tx=0x7f976c00ee80 comp rx=0 tx=0).stop 2024-03-24T00:15:27.820 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:27.819+0000 7f977be00640 1 -- 172.21.15.118:0/2051213444 shutdown_connections 2024-03-24T00:15:27.821 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:27.819+0000 7f977be00640 1 --2- 172.21.15.118:0/2051213444 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7f97600a2800 0x7f97600a4cc0 unknown :-1 s=CLOSED pgs=240 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:15:27.821 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:27.819+0000 7f977be00640 1 --2- 172.21.15.118:0/2051213444 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f977c102030 0x7f977c1a1060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:15:27.821 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:27.819+0000 7f977be00640 1 --2- 172.21.15.118:0/2051213444 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f977c101670 0x7f977c199ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:15:27.821 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:27.819+0000 7f977be00640 1 --2- 172.21.15.118:0/2051213444 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f977c100480 0x7f977c199ab0 unknown :-1 s=CLOSED pgs=162 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:15:27.821 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:27.819+0000 7f977be00640 1 -- 172.21.15.118:0/2051213444 >> 172.21.15.118:0/2051213444 conn(0x7f977c0fba10 msgr2=0x7f977c0fd480 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:15:27.821 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:27.820+0000 7f977be00640 1 -- 172.21.15.118:0/2051213444 shutdown_connections 2024-03-24T00:15:27.821 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:27.820+0000 7f977be00640 1 -- 172.21.15.118:0/2051213444 wait complete. 2024-03-24T00:15:27.839 DEBUG:teuthology.orchestra.run.smithi118:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-24T00:15:28.084 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:28.083+0000 7f1173200640 1 -- 172.21.15.118:0/818590450 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f11741003f0 msgr2=0x7f1174100810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:15:28.084 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:28.083+0000 7f1173200640 1 --2- 172.21.15.118:0/818590450 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f11741003f0 0x7f1174100810 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 crypto rx=0x7f1164009900 tx=0x7f116402bc30 comp rx=0 tx=0).stop 2024-03-24T00:15:28.084 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:28.083+0000 7f1173200640 1 -- 172.21.15.118:0/818590450 shutdown_connections 2024-03-24T00:15:28.084 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:28.083+0000 7f1173200640 1 --2- 172.21.15.118:0/818590450 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f1174101fa0 0x7f1174108830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:15:28.084 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:28.083+0000 7f1173200640 1 --2- 172.21.15.118:0/818590450 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f11741015e0 0x7f1174101a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:15:28.084 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:28.083+0000 7f1173200640 1 --2- 172.21.15.118:0/818590450 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f11741003f0 0x7f1174100810 unknown :-1 s=CLOSED pgs=163 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:15:28.084 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:28.083+0000 7f1173200640 1 -- 172.21.15.118:0/818590450 >> 172.21.15.118:0/818590450 conn(0x7f11740fb980 msgr2=0x7f11740fdda0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:15:28.084 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:28.083+0000 7f1173200640 1 -- 172.21.15.118:0/818590450 shutdown_connections 2024-03-24T00:15:28.084 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:28.083+0000 7f1173200640 1 -- 172.21.15.118:0/818590450 wait complete. 2024-03-24T00:15:28.086 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:28.084+0000 7f1173200640 1 Processor -- start 2024-03-24T00:15:28.086 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:28.084+0000 7f1173200640 1 -- start start 2024-03-24T00:15:28.087 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:28.085+0000 7f1173200640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f11741003f0 0x7f1174104380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:15:28.087 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:28.085+0000 7f1173200640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f11741015e0 0x7f11741029c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:15:28.087 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:28.085+0000 7f1173200640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f1174101fa0 0x7f1174102f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:15:28.087 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:28.085+0000 7f1173200640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f117410c0d0 con 0x7f1174101fa0 2024-03-24T00:15:28.087 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:28.085+0000 7f1173200640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f117410bf50 con 0x7f11741003f0 2024-03-24T00:15:28.087 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:28.085+0000 7f1173200640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f117410c250 con 0x7f11741015e0 2024-03-24T00:15:28.087 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:28.085+0000 7f1171e00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f11741003f0 0x7f1174104380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:15:28.087 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:28.085+0000 7f1171e00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f11741003f0 0x7f1174104380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.118:54874/0 (socket says 172.21.15.118:54874) 2024-03-24T00:15:28.087 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:28.085+0000 7f1171e00640 1 -- 172.21.15.118:0/2495320211 learned_addr learned my addr 172.21.15.118:0/2495320211 (peer_addr_for_me v2:172.21.15.118:0/0) 2024-03-24T00:15:28.087 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:28.085+0000 7f1171400640 1 --2- 172.21.15.118:0/2495320211 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f11741015e0 0x7f11741029c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:15:28.087 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:28.085+0000 7f1172800640 1 --2- 172.21.15.118:0/2495320211 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f1174101fa0 0x7f1174102f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:15:28.087 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:28.085+0000 7f1171e00640 1 -- 172.21.15.118:0/2495320211 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f11741015e0 msgr2=0x7f11741029c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:15:28.087 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:28.085+0000 7f1171e00640 1 --2- 172.21.15.118:0/2495320211 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f11741015e0 0x7f11741029c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:15:28.087 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:28.085+0000 7f1171e00640 1 -- 172.21.15.118:0/2495320211 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f1174101fa0 msgr2=0x7f1174102f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:15:28.088 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:28.085+0000 7f1171e00640 1 --2- 172.21.15.118:0/2495320211 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f1174101fa0 0x7f1174102f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:15:28.088 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:28.085+0000 7f1171e00640 1 -- 172.21.15.118:0/2495320211 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1174103790 con 0x7f11741003f0 2024-03-24T00:15:28.088 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:28.085+0000 7f1171e00640 1 --2- 172.21.15.118:0/2495320211 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f11741003f0 0x7f1174104380 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f1164000c00 tx=0x7f116402c650 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:15:28.088 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:28.086+0000 7f116aa00640 1 -- 172.21.15.118:0/2495320211 <== mon.2 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1164043020 con 0x7f11741003f0 2024-03-24T00:15:28.088 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:28.086+0000 7f116aa00640 1 -- 172.21.15.118:0/2495320211 <== mon.2 v2:172.21.15.189:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f11640054e0 con 0x7f11741003f0 2024-03-24T00:15:28.088 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:28.086+0000 7f1173200640 1 -- 172.21.15.118:0/2495320211 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f11741a1950 con 0x7f11741003f0 2024-03-24T00:15:28.088 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:28.086+0000 7f116aa00640 1 -- 172.21.15.118:0/2495320211 <== mon.2 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1164003c40 con 0x7f11741003f0 2024-03-24T00:15:28.088 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:28.086+0000 7f1173200640 1 -- 172.21.15.118:0/2495320211 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f11741a1ef0 con 0x7f11741003f0 2024-03-24T00:15:28.088 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:28.087+0000 7f1173200640 1 -- 172.21.15.118:0/2495320211 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f11741a20c0 con 0x7f11741003f0 2024-03-24T00:15:28.091 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:28.090+0000 7f116aa00640 1 -- 172.21.15.118:0/2495320211 <== mon.2 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 18) v1 ==== 137291+0+0 (secure 0 0 0) 0x7f1164046020 con 0x7f11741003f0 2024-03-24T00:15:28.092 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:28.092+0000 7f116aa00640 1 --2- 172.21.15.118:0/2495320211 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7f11580a2880 0x7f11580a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:15:28.092 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:28.092+0000 7f116aa00640 1 -- 172.21.15.118:0/2495320211 <== mon.2 v2:172.21.15.189:3300/0 5 ==== osd_map(90..90 src has 1..90) v4 ==== 8277+0+0 (secure 0 0 0) 0x7f11640f0e30 con 0x7f11741003f0 2024-03-24T00:15:28.092 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:28.092+0000 7f1171400640 1 --2- 172.21.15.118:0/2495320211 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7f11580a2880 0x7f11580a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:15:28.093 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:28.092+0000 7f116aa00640 1 -- 172.21.15.118:0/2495320211 <== mon.2 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f116404d020 con 0x7f11741003f0 2024-03-24T00:15:28.093 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:28.093+0000 7f1171400640 1 --2- 172.21.15.118:0/2495320211 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7f11580a2880 0x7f11580a4d40 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7f11741040f0 tx=0x7f115c005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:15:28.104 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:15:27 smithi189 ceph-mon[36187]: from='client.? 172.21.15.118:0/3914240704' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-24T00:15:28.104 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:15:27 smithi189 ceph-mon[36187]: from='client.? 172.21.15.118:0/2420781385' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-24T00:15:28.104 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:15:27 smithi189 ceph-mon[36187]: pgmap v391: 97 pgs: 97 active+clean; 645 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-03-24T00:15:28.195 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:15:27 smithi118 ceph-mon[31535]: from='client.? 172.21.15.118:0/3914240704' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-24T00:15:28.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:15:27 smithi118 ceph-mon[31535]: from='client.? 172.21.15.118:0/2420781385' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-24T00:15:28.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:15:27 smithi118 ceph-mon[31535]: pgmap v391: 97 pgs: 97 active+clean; 645 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-03-24T00:15:28.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:15:27 smithi193 ceph-mon[36878]: from='client.? 172.21.15.118:0/3914240704' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-24T00:15:28.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:15:27 smithi193 ceph-mon[36878]: from='client.? 172.21.15.118:0/2420781385' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-24T00:15:28.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:15:27 smithi193 ceph-mon[36878]: pgmap v391: 97 pgs: 97 active+clean; 645 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-03-24T00:15:28.329 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:28.328+0000 7f1173200640 1 -- 172.21.15.118:0/2495320211 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f11741a2290 con 0x7f11741003f0 2024-03-24T00:15:28.330 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:28.329+0000 7f116aa00640 1 -- 172.21.15.118:0/2495320211 <== mon.2 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 27 v27) v1 ==== 94+0+11478 (secure 0 0 0) 0x7f11640bde40 con 0x7f11741003f0 2024-03-24T00:15:28.330 INFO:teuthology.orchestra.run.smithi118.stdout: 2024-03-24T00:15:28.330 INFO:teuthology.orchestra.run.smithi118.stdout:{"epoch":27,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14385,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.118:6835/3194741373","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6834","nonce":3194741373},{"type":"v1","addr":"172.21.15.118:6835","nonce":3194741373}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":4},{"gid":14397,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.118:6837/1374462956","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6836","nonce":1374462956},{"type":"v1","addr":"172.21.15.118:6837","nonce":1374462956}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":27,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-03-24T00:13:01.782474+0000","modified":"2024-03-24T00:13:09.335361+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24569,"mds_1":24550,"mds_2":24511,"mds_3":24499,"mds_4":24488},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14469":{"gid":14469,"name":"a","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.118:6841/3040946537","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6840","nonce":3040946537},{"type":"v1","addr":"172.21.15.118:6841","nonce":3040946537}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24439":{"gid":24439,"name":"l","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.193:6833/1023428412","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6832","nonce":1023428412},{"type":"v1","addr":"172.21.15.193:6833","nonce":1023428412}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24440":{"gid":24440,"name":"k","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.189:6833/2740024078","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6832","nonce":2740024078},{"type":"v1","addr":"172.21.15.189:6833","nonce":2740024078}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24488":{"gid":24488,"name":"e","rank":4,"incarnation":25,"state":"up:active","state_seq":10,"addr":"172.21.15.189:6835/599473205","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6834","nonce":599473205},{"type":"v1","addr":"172.21.15.189:6835","nonce":599473205}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24499":{"gid":24499,"name":"f","rank":3,"incarnation":23,"state":"up:active","state_seq":9,"addr":"172.21.15.193:6835/2055310541","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6834","nonce":2055310541},{"type":"v1","addr":"172.21.15.193:6835","nonce":2055310541}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24511":{"gid":24511,"name":"c","rank":2,"incarnation":21,"state":"up:active","state_seq":8,"addr":"172.21.15.193:6837/2921520053","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6836","nonce":2921520053},{"type":"v1","addr":"172.21.15.193:6837","nonce":2921520053}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24515":{"gid":24515,"name":"h","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.189:6837/3450187137","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6836","nonce":3450187137},{"type":"v1","addr":"172.21.15.189:6837","nonce":3450187137}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24532":{"gid":24532,"name":"i","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.193:6839/2126842603","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6838","nonce":2126842603},{"type":"v1","addr":"172.21.15.193:6839","nonce":2126842603}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24550":{"gid":24550,"name":"g","rank":1,"incarnation":19,"state":"up:active","state_seq":6,"addr":"172.21.15.118:6839/3285788503","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6838","nonce":3285788503},{"type":"v1","addr":"172.21.15.118:6839","nonce":3285788503}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24569":{"gid":24569,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.189:6839/4074484168","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6838","nonce":4074484168},{"type":"v1","addr":"172.21.15.189:6839","nonce":4074484168}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24569,"qdb_cluster":[24511,24550,24488,24499,24569]},"id":1}]} 2024-03-24T00:15:28.331 INFO:teuthology.orchestra.run.smithi118.stderr:dumped fsmap epoch 27 2024-03-24T00:15:28.333 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:28.332+0000 7f1173200640 1 -- 172.21.15.118:0/2495320211 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7f11580a2880 msgr2=0x7f11580a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:15:28.333 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:28.332+0000 7f1173200640 1 --2- 172.21.15.118:0/2495320211 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7f11580a2880 0x7f11580a4d40 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7f11741040f0 tx=0x7f115c005f90 comp rx=0 tx=0).stop 2024-03-24T00:15:28.333 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:28.332+0000 7f1173200640 1 -- 172.21.15.118:0/2495320211 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f11741003f0 msgr2=0x7f1174104380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:15:28.333 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:28.332+0000 7f1173200640 1 --2- 172.21.15.118:0/2495320211 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f11741003f0 0x7f1174104380 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 crypto rx=0x7f1164000c00 tx=0x7f116402c650 comp rx=0 tx=0).stop 2024-03-24T00:15:28.333 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:28.332+0000 7f1173200640 1 -- 172.21.15.118:0/2495320211 shutdown_connections 2024-03-24T00:15:28.333 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:28.332+0000 7f1173200640 1 --2- 172.21.15.118:0/2495320211 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7f11580a2880 0x7f11580a4d40 unknown :-1 s=CLOSED pgs=241 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:15:28.333 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:28.332+0000 7f1173200640 1 --2- 172.21.15.118:0/2495320211 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f1174101fa0 0x7f1174102f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:15:28.333 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:28.332+0000 7f1173200640 1 --2- 172.21.15.118:0/2495320211 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f11741015e0 0x7f11741029c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:15:28.333 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:28.332+0000 7f1173200640 1 --2- 172.21.15.118:0/2495320211 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f11741003f0 0x7f1174104380 unknown :-1 s=CLOSED pgs=164 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:15:28.333 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:28.332+0000 7f1173200640 1 -- 172.21.15.118:0/2495320211 >> 172.21.15.118:0/2495320211 conn(0x7f11740fb980 msgr2=0x7f11740fd3d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:15:28.333 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:28.332+0000 7f1173200640 1 -- 172.21.15.118:0/2495320211 shutdown_connections 2024-03-24T00:15:28.333 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:28.332+0000 7f1173200640 1 -- 172.21.15.118:0/2495320211 wait complete. 2024-03-24T00:15:28.350 DEBUG:teuthology.orchestra.run.smithi118:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-24T00:15:28.576 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:28.575+0000 7f8744600640 1 -- 172.21.15.118:0/1569796289 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f873c1028b0 msgr2=0x7f873c102d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:15:28.576 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:28.575+0000 7f8744600640 1 --2- 172.21.15.118:0/1569796289 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f873c1028b0 0x7f873c102d30 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7f8728009900 tx=0x7f872802bc30 comp rx=0 tx=0).stop 2024-03-24T00:15:28.576 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:28.575+0000 7f8744600640 1 -- 172.21.15.118:0/1569796289 shutdown_connections 2024-03-24T00:15:28.576 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:28.575+0000 7f8744600640 1 --2- 172.21.15.118:0/1569796289 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f873c103270 0x7f873c109b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:15:28.577 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:28.575+0000 7f8744600640 1 --2- 172.21.15.118:0/1569796289 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f873c1028b0 0x7f873c102d30 unknown :-1 s=CLOSED pgs=165 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:15:28.577 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:28.575+0000 7f8744600640 1 --2- 172.21.15.118:0/1569796289 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f873c1016c0 0x7f873c101ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:15:28.577 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:28.575+0000 7f8744600640 1 -- 172.21.15.118:0/1569796289 >> 172.21.15.118:0/1569796289 conn(0x7f873c0fcc50 msgr2=0x7f873c0ff070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:15:28.577 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:28.575+0000 7f8744600640 1 -- 172.21.15.118:0/1569796289 shutdown_connections 2024-03-24T00:15:28.577 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:28.575+0000 7f8744600640 1 -- 172.21.15.118:0/1569796289 wait complete. 2024-03-24T00:15:28.579 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:28.576+0000 7f8744600640 1 Processor -- start 2024-03-24T00:15:28.579 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:28.576+0000 7f8744600640 1 -- start start 2024-03-24T00:15:28.579 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:28.577+0000 7f8744600640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f873c1016c0 0x7f873c1979b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:15:28.579 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:28.577+0000 7f8744600640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f873c1028b0 0x7f873c197ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:15:28.579 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:28.577+0000 7f8744600640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f873c103270 0x7f873c19ef60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:15:28.579 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:28.577+0000 7f8744600640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f873c10d370 con 0x7f873c103270 2024-03-24T00:15:28.579 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:28.577+0000 7f8744600640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f873c10d1f0 con 0x7f873c1028b0 2024-03-24T00:15:28.579 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:28.577+0000 7f8744600640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f873c10d4f0 con 0x7f873c1016c0 2024-03-24T00:15:28.579 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:28.577+0000 7f8741c00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f873c1016c0 0x7f873c1979b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:15:28.579 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:28.577+0000 7f8741c00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f873c1016c0 0x7f873c1979b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.118:46896/0 (socket says 172.21.15.118:46896) 2024-03-24T00:15:28.579 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:28.577+0000 7f8741c00640 1 -- 172.21.15.118:0/1271532583 learned_addr learned my addr 172.21.15.118:0/1271532583 (peer_addr_for_me v2:172.21.15.118:0/0) 2024-03-24T00:15:28.579 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:28.577+0000 7f8741200640 1 --2- 172.21.15.118:0/1271532583 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f873c1028b0 0x7f873c197ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:15:28.580 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:28.577+0000 7f8742600640 1 --2- 172.21.15.118:0/1271532583 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f873c103270 0x7f873c19ef60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:15:28.580 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:28.577+0000 7f8741c00640 1 -- 172.21.15.118:0/1271532583 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f873c1028b0 msgr2=0x7f873c197ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:15:28.580 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:28.577+0000 7f8741c00640 1 --2- 172.21.15.118:0/1271532583 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f873c1028b0 0x7f873c197ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:15:28.580 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:28.577+0000 7f8741c00640 1 -- 172.21.15.118:0/1271532583 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f873c103270 msgr2=0x7f873c19ef60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:15:28.580 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:28.577+0000 7f8741c00640 1 --2- 172.21.15.118:0/1271532583 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f873c103270 0x7f873c19ef60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:15:28.580 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:28.577+0000 7f8741c00640 1 -- 172.21.15.118:0/1271532583 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f873c19f5d0 con 0x7f873c1016c0 2024-03-24T00:15:28.580 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:28.578+0000 7f8741c00640 1 --2- 172.21.15.118:0/1271532583 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f873c1016c0 0x7f873c1979b0 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7f873000e9c0 tx=0x7f873000edc0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:15:28.580 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:28.578+0000 7f873a000640 1 -- 172.21.15.118:0/1271532583 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f873000ccc0 con 0x7f873c1016c0 2024-03-24T00:15:28.580 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:28.578+0000 7f873a000640 1 -- 172.21.15.118:0/1271532583 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f87300035f0 con 0x7f873c1016c0 2024-03-24T00:15:28.580 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:28.578+0000 7f873a000640 1 -- 172.21.15.118:0/1271532583 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f873000ce60 con 0x7f873c1016c0 2024-03-24T00:15:28.580 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:28.578+0000 7f8744600640 1 -- 172.21.15.118:0/1271532583 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f873c19f8c0 con 0x7f873c1016c0 2024-03-24T00:15:28.580 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:28.578+0000 7f8744600640 1 -- 172.21.15.118:0/1271532583 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f873c19fd90 con 0x7f873c1016c0 2024-03-24T00:15:28.581 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:28.579+0000 7f8744600640 1 -- 172.21.15.118:0/1271532583 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f873c1089e0 con 0x7f873c1016c0 2024-03-24T00:15:28.581 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:28.581+0000 7f873a000640 1 -- 172.21.15.118:0/1271532583 <== mon.1 v2:172.21.15.193:3300/0 4 ==== mgrmap(e 18) v1 ==== 137291+0+0 (secure 0 0 0) 0x7f87300037a0 con 0x7f873c1016c0 2024-03-24T00:15:28.583 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:28.582+0000 7f873a000640 1 --2- 172.21.15.118:0/1271532583 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7f87240a2800 0x7f87240a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:15:28.583 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:28.582+0000 7f873a000640 1 -- 172.21.15.118:0/1271532583 <== mon.1 v2:172.21.15.193:3300/0 5 ==== osd_map(90..90 src has 1..90) v4 ==== 8277+0+0 (secure 0 0 0) 0x7f87300cee50 con 0x7f873c1016c0 2024-03-24T00:15:28.583 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:28.583+0000 7f8741200640 1 --2- 172.21.15.118:0/1271532583 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7f87240a2800 0x7f87240a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:15:28.584 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:28.583+0000 7f8741200640 1 --2- 172.21.15.118:0/1271532583 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7f87240a2800 0x7f87240a4cc0 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7f872802c240 tx=0x7f872802df90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:15:28.586 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:28.586+0000 7f873a000640 1 -- 172.21.15.118:0/1271532583 <== mon.1 v2:172.21.15.193:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f873009be60 con 0x7f873c1016c0 2024-03-24T00:15:28.789 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:28.788+0000 7f8744600640 1 -- 172.21.15.118:0/1271532583 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f873c02d070 con 0x7f873c1016c0 2024-03-24T00:15:28.791 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:28.791+0000 7f873a000640 1 -- 172.21.15.118:0/1271532583 <== mon.1 v2:172.21.15.193:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v90) v1 ==== 74+0+22834 (secure 0 0 0) 0x7f873009b760 con 0x7f873c1016c0 2024-03-24T00:15:28.792 INFO:teuthology.orchestra.run.smithi118.stdout: 2024-03-24T00:15:28.792 INFO:teuthology.orchestra.run.smithi118.stdout:{"epoch":90,"fsid":"d79a5420-e971-11ee-95cd-87774f69a715","created":"2024-03-24T00:03:58.648524+0000","modified":"2024-03-24T00:15:07.721960+0000","last_up_change":"2024-03-24T00:12:03.809000+0000","last_in_change":"2024-03-24T00:11:44.389344+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-24T00:07:50.190965+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-24T00:12:59.993723+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-24T00:13:01.257347+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"86","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"c9590426-3901-4f68-a5ef-d57aa8fbac92","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6802","nonce":557690061},{"type":"v1","addr":"172.21.15.118:6803","nonce":557690061}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6804","nonce":557690061},{"type":"v1","addr":"172.21.15.118:6805","nonce":557690061}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6808","nonce":557690061},{"type":"v1","addr":"172.21.15.118:6809","nonce":557690061}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6806","nonce":557690061},{"type":"v1","addr":"172.21.15.118:6807","nonce":557690061}]},"public_addr":"172.21.15.118:6803/557690061","cluster_addr":"172.21.15.118:6805/557690061","heartbeat_back_addr":"172.21.15.118:6809/557690061","heartbeat_front_addr":"172.21.15.118:6807/557690061","state":["exists","up"]},{"osd":1,"uuid":"fc7f827e-ecf6-4661-aad0-7a22951892e0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6800","nonce":4203029485},{"type":"v1","addr":"172.21.15.189:6801","nonce":4203029485}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6802","nonce":4203029485},{"type":"v1","addr":"172.21.15.189:6803","nonce":4203029485}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6806","nonce":4203029485},{"type":"v1","addr":"172.21.15.189:6807","nonce":4203029485}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6804","nonce":4203029485},{"type":"v1","addr":"172.21.15.189:6805","nonce":4203029485}]},"public_addr":"172.21.15.189:6801/4203029485","cluster_addr":"172.21.15.189:6803/4203029485","heartbeat_back_addr":"172.21.15.189:6807/4203029485","heartbeat_front_addr":"172.21.15.189:6805/4203029485","state":["exists","up"]},{"osd":2,"uuid":"e73ccfe9-0e5b-4734-a154-a2f0d829fcfb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6800","nonce":3699076971},{"type":"v1","addr":"172.21.15.193:6801","nonce":3699076971}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6802","nonce":3699076971},{"type":"v1","addr":"172.21.15.193:6803","nonce":3699076971}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6806","nonce":3699076971},{"type":"v1","addr":"172.21.15.193:6807","nonce":3699076971}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6804","nonce":3699076971},{"type":"v1","addr":"172.21.15.193:6805","nonce":3699076971}]},"public_addr":"172.21.15.193:6801/3699076971","cluster_addr":"172.21.15.193:6803/3699076971","heartbeat_back_addr":"172.21.15.193:6807/3699076971","heartbeat_front_addr":"172.21.15.193:6805/3699076971","state":["exists","up"]},{"osd":3,"uuid":"63fed43b-43ce-4031-8957-7fe94725c1ed","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6810","nonce":4059617010},{"type":"v1","addr":"172.21.15.118:6811","nonce":4059617010}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6812","nonce":4059617010},{"type":"v1","addr":"172.21.15.118:6813","nonce":4059617010}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6816","nonce":4059617010},{"type":"v1","addr":"172.21.15.118:6817","nonce":4059617010}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6814","nonce":4059617010},{"type":"v1","addr":"172.21.15.118:6815","nonce":4059617010}]},"public_addr":"172.21.15.118:6811/4059617010","cluster_addr":"172.21.15.118:6813/4059617010","heartbeat_back_addr":"172.21.15.118:6817/4059617010","heartbeat_front_addr":"172.21.15.118:6815/4059617010","state":["exists","up"]},{"osd":4,"uuid":"57377f66-c3b4-43c1-8e8e-bc873672906d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":34,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6808","nonce":4075247068},{"type":"v1","addr":"172.21.15.189:6809","nonce":4075247068}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6810","nonce":4075247068},{"type":"v1","addr":"172.21.15.189:6811","nonce":4075247068}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6814","nonce":4075247068},{"type":"v1","addr":"172.21.15.189:6815","nonce":4075247068}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6812","nonce":4075247068},{"type":"v1","addr":"172.21.15.189:6813","nonce":4075247068}]},"public_addr":"172.21.15.189:6809/4075247068","cluster_addr":"172.21.15.189:6811/4075247068","heartbeat_back_addr":"172.21.15.189:6815/4075247068","heartbeat_front_addr":"172.21.15.189:6813/4075247068","state":["exists","up"]},{"osd":5,"uuid":"91970976-2193-4224-81c0-dc96cb022c85","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":39,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6808","nonce":2920363402},{"type":"v1","addr":"172.21.15.193:6809","nonce":2920363402}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6810","nonce":2920363402},{"type":"v1","addr":"172.21.15.193:6811","nonce":2920363402}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6814","nonce":2920363402},{"type":"v1","addr":"172.21.15.193:6815","nonce":2920363402}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6812","nonce":2920363402},{"type":"v1","addr":"172.21.15.193:6813","nonce":2920363402}]},"public_addr":"172.21.15.193:6809/2920363402","cluster_addr":"172.21.15.193:6811/2920363402","heartbeat_back_addr":"172.21.15.193:6815/2920363402","heartbeat_front_addr":"172.21.15.193:6813/2920363402","state":["exists","up"]},{"osd":6,"uuid":"0b77b196-8674-41c6-ad3d-726b5d01c8fd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":47,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6818","nonce":431354576},{"type":"v1","addr":"172.21.15.118:6819","nonce":431354576}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6820","nonce":431354576},{"type":"v1","addr":"172.21.15.118:6821","nonce":431354576}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6824","nonce":431354576},{"type":"v1","addr":"172.21.15.118:6825","nonce":431354576}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6822","nonce":431354576},{"type":"v1","addr":"172.21.15.118:6823","nonce":431354576}]},"public_addr":"172.21.15.118:6819/431354576","cluster_addr":"172.21.15.118:6821/431354576","heartbeat_back_addr":"172.21.15.118:6825/431354576","heartbeat_front_addr":"172.21.15.118:6823/431354576","state":["exists","up"]},{"osd":7,"uuid":"1fbbd38f-4e8e-4d4c-8e98-66f6862bc4bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":52,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6816","nonce":3667611284},{"type":"v1","addr":"172.21.15.189:6817","nonce":3667611284}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6818","nonce":3667611284},{"type":"v1","addr":"172.21.15.189:6819","nonce":3667611284}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6822","nonce":3667611284},{"type":"v1","addr":"172.21.15.189:6823","nonce":3667611284}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6820","nonce":3667611284},{"type":"v1","addr":"172.21.15.189:6821","nonce":3667611284}]},"public_addr":"172.21.15.189:6817/3667611284","cluster_addr":"172.21.15.189:6819/3667611284","heartbeat_back_addr":"172.21.15.189:6823/3667611284","heartbeat_front_addr":"172.21.15.189:6821/3667611284","state":["exists","up"]},{"osd":8,"uuid":"6462c2b5-e4b3-4b0b-bcfa-4de2c89e93e0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":60,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6816","nonce":3853800910},{"type":"v1","addr":"172.21.15.193:6817","nonce":3853800910}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6818","nonce":3853800910},{"type":"v1","addr":"172.21.15.193:6819","nonce":3853800910}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6822","nonce":3853800910},{"type":"v1","addr":"172.21.15.193:6823","nonce":3853800910}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6820","nonce":3853800910},{"type":"v1","addr":"172.21.15.193:6821","nonce":3853800910}]},"public_addr":"172.21.15.193:6817/3853800910","cluster_addr":"172.21.15.193:6819/3853800910","heartbeat_back_addr":"172.21.15.193:6823/3853800910","heartbeat_front_addr":"172.21.15.193:6821/3853800910","state":["exists","up"]},{"osd":9,"uuid":"54be277f-3596-422f-9de5-9005a8571c6f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":68,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6826","nonce":3248293936},{"type":"v1","addr":"172.21.15.118:6827","nonce":3248293936}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6828","nonce":3248293936},{"type":"v1","addr":"172.21.15.118:6829","nonce":3248293936}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6832","nonce":3248293936},{"type":"v1","addr":"172.21.15.118:6833","nonce":3248293936}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6830","nonce":3248293936},{"type":"v1","addr":"172.21.15.118:6831","nonce":3248293936}]},"public_addr":"172.21.15.118:6827/3248293936","cluster_addr":"172.21.15.118:6829/3248293936","heartbeat_back_addr":"172.21.15.118:6833/3248293936","heartbeat_front_addr":"172.21.15.118:6831/3248293936","state":["exists","up"]},{"osd":10,"uuid":"7ba69d69-ac41-44ce-bc39-c049a8ca5900","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":73,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6824","nonce":2112195760},{"type":"v1","addr":"172.21.15.189:6825","nonce":2112195760}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6826","nonce":2112195760},{"type":"v1","addr":"172.21.15.189:6827","nonce":2112195760}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6830","nonce":2112195760},{"type":"v1","addr":"172.21.15.189:6831","nonce":2112195760}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6828","nonce":2112195760},{"type":"v1","addr":"172.21.15.189:6829","nonce":2112195760}]},"public_addr":"172.21.15.189:6825/2112195760","cluster_addr":"172.21.15.189:6827/2112195760","heartbeat_back_addr":"172.21.15.189:6831/2112195760","heartbeat_front_addr":"172.21.15.189:6829/2112195760","state":["exists","up"]},{"osd":11,"uuid":"186f0b18-6b22-4eae-83b7-b21bbcce0817","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":81,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6824","nonce":3880515510},{"type":"v1","addr":"172.21.15.193:6825","nonce":3880515510}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6826","nonce":3880515510},{"type":"v1","addr":"172.21.15.193:6827","nonce":3880515510}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6830","nonce":3880515510},{"type":"v1","addr":"172.21.15.193:6831","nonce":3880515510}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6828","nonce":3880515510},{"type":"v1","addr":"172.21.15.193:6829","nonce":3880515510}]},"public_addr":"172.21.15.193:6825/3880515510","cluster_addr":"172.21.15.193:6827/3880515510","heartbeat_back_addr":"172.21.15.193:6831/3880515510","heartbeat_front_addr":"172.21.15.193:6829/3880515510","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:06:58.105543+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:07:23.795866+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:07:47.426050+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:08:19.427073+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:08:45.816082+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:09:11.081523+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:09:44.485077+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:10:11.338972+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:10:36.234024+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:11:09.730432+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:11:37.059083+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:12:02.371947+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.a","mappings":[{"from":2,"to":11}]},{"pgid":"2.14","mappings":[{"from":2,"to":4}]},{"pgid":"2.1d","mappings":[{"from":8,"to":11}]},{"pgid":"2.1f","mappings":[{"from":8,"to":1}]},{"pgid":"3.11","mappings":[{"from":4,"to":8},{"from":5,"to":9}]},{"pgid":"3.13","mappings":[{"from":4,"to":9}]},{"pgid":"3.14","mappings":[{"from":4,"to":6}]},{"pgid":"3.31","mappings":[{"from":4,"to":7}]},{"pgid":"3.38","mappings":[{"from":3,"to":9}]},{"pgid":"3.39","mappings":[{"from":5,"to":8}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.118:6801/389249272":"2024-03-25T00:05:05.458796+0000","172.21.15.118:6800/389249272":"2024-03-25T00:05:05.458796+0000","172.21.15.118:0/2101854694":"2024-03-25T00:05:05.458796+0000","172.21.15.118:0/630779359":"2024-03-25T00:05:05.458796+0000","172.21.15.118:0/2158921029":"2024-03-25T00:04:31.283725+0000","172.21.15.118:0/1003670455":"2024-03-25T00:05:05.458796+0000","172.21.15.118:0/368712314":"2024-03-25T00:04:31.283725+0000","172.21.15.118:0/2004775597":"2024-03-25T00:04:31.283725+0000","172.21.15.118:6800/1463147955":"2024-03-25T00:04:31.283725+0000","172.21.15.118:6801/1463147955":"2024-03-25T00:04:31.283725+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-24T00:15:28.795 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:28.794+0000 7f8744600640 1 -- 172.21.15.118:0/1271532583 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7f87240a2800 msgr2=0x7f87240a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:15:28.795 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:28.794+0000 7f8744600640 1 --2- 172.21.15.118:0/1271532583 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7f87240a2800 0x7f87240a4cc0 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7f872802c240 tx=0x7f872802df90 comp rx=0 tx=0).stop 2024-03-24T00:15:28.795 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:28.794+0000 7f8744600640 1 -- 172.21.15.118:0/1271532583 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f873c1016c0 msgr2=0x7f873c1979b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:15:28.795 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:28.794+0000 7f8744600640 1 --2- 172.21.15.118:0/1271532583 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f873c1016c0 0x7f873c1979b0 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7f873000e9c0 tx=0x7f873000edc0 comp rx=0 tx=0).stop 2024-03-24T00:15:28.795 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:28.794+0000 7f8744600640 1 -- 172.21.15.118:0/1271532583 shutdown_connections 2024-03-24T00:15:28.795 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:28.794+0000 7f8744600640 1 --2- 172.21.15.118:0/1271532583 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7f87240a2800 0x7f87240a4cc0 unknown :-1 s=CLOSED pgs=242 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:15:28.795 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:28.794+0000 7f8744600640 1 --2- 172.21.15.118:0/1271532583 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f873c103270 0x7f873c19ef60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:15:28.795 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:28.794+0000 7f8744600640 1 --2- 172.21.15.118:0/1271532583 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f873c1028b0 0x7f873c197ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:15:28.795 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:28.794+0000 7f8744600640 1 --2- 172.21.15.118:0/1271532583 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f873c1016c0 0x7f873c1979b0 unknown :-1 s=CLOSED pgs=167 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:15:28.795 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:28.794+0000 7f8744600640 1 -- 172.21.15.118:0/1271532583 >> 172.21.15.118:0/1271532583 conn(0x7f873c0fcc50 msgr2=0x7f873c105bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:15:28.795 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:28.794+0000 7f8744600640 1 -- 172.21.15.118:0/1271532583 shutdown_connections 2024-03-24T00:15:28.795 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:28.794+0000 7f8744600640 1 -- 172.21.15.118:0/1271532583 wait complete. 2024-03-24T00:15:28.815 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 27, 'flags': 50, 'flags_state': {'joinable': True, 'allow_snaps': True, 'allow_multimds_snaps': True, 'allow_standby_replay': True, 'refuse_client_session': False, 'refuse_standby_for_another_fs': False, 'balance_automate': False}, 'ever_allowed_features': 32, 'explicitly_allowed_features': 32, 'created': '2024-03-24T00:13:01.782474+0000', 'modified': '2024-03-24T00:13:09.335361+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 300, 'session_autoclose': 300, 'required_client_features': {}, 'max_file_size': 1099511627776, 'max_xattr_size': 65536, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}, 'max_mds': 5, 'in': [0, 1, 2, 3, 4], 'up': {'mds_0': 24569, 'mds_1': 24550, 'mds_2': 24511, 'mds_3': 24499, 'mds_4': 24488}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_14469': {'gid': 14469, 'name': 'a', 'rank': 4, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.118:6841/3040946537', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.118:6840', 'nonce': 3040946537}, {'type': 'v1', 'addr': '172.21.15.118:6841', 'nonce': 3040946537}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_24439': {'gid': 24439, 'name': 'l', 'rank': 3, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.193:6833/1023428412', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.193:6832', 'nonce': 1023428412}, {'type': 'v1', 'addr': '172.21.15.193:6833', 'nonce': 1023428412}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_24440': {'gid': 24440, 'name': 'k', 'rank': 2, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.189:6833/2740024078', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.189:6832', 'nonce': 2740024078}, {'type': 'v1', 'addr': '172.21.15.189:6833', 'nonce': 2740024078}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_24488': {'gid': 24488, 'name': 'e', 'rank': 4, 'incarnation': 25, 'state': 'up:active', 'state_seq': 10, 'addr': '172.21.15.189:6835/599473205', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.189:6834', 'nonce': 599473205}, {'type': 'v1', 'addr': '172.21.15.189:6835', 'nonce': 599473205}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_24499': {'gid': 24499, 'name': 'f', 'rank': 3, 'incarnation': 23, 'state': 'up:active', 'state_seq': 9, 'addr': '172.21.15.193:6835/2055310541', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.193:6834', 'nonce': 2055310541}, {'type': 'v1', 'addr': '172.21.15.193:6835', 'nonce': 2055310541}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_24511': {'gid': 24511, 'name': 'c', 'rank': 2, 'incarnation': 21, 'state': 'up:active', 'state_seq': 8, 'addr': '172.21.15.193:6837/2921520053', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.193:6836', 'nonce': 2921520053}, {'type': 'v1', 'addr': '172.21.15.193:6837', 'nonce': 2921520053}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_24515': {'gid': 24515, 'name': 'h', 'rank': 1, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.189:6837/3450187137', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.189:6836', 'nonce': 3450187137}, {'type': 'v1', 'addr': '172.21.15.189:6837', 'nonce': 3450187137}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_24532': {'gid': 24532, 'name': 'i', 'rank': 0, 'incarnation': 0, 'state': 'up:standby-replay', 'state_seq': 1, 'addr': '172.21.15.193:6839/2126842603', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.193:6838', 'nonce': 2126842603}, {'type': 'v1', 'addr': '172.21.15.193:6839', 'nonce': 2126842603}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_24550': {'gid': 24550, 'name': 'g', 'rank': 1, 'incarnation': 19, 'state': 'up:active', 'state_seq': 6, 'addr': '172.21.15.118:6839/3285788503', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.118:6838', 'nonce': 3285788503}, {'type': 'v1', 'addr': '172.21.15.118:6839', 'nonce': 3285788503}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}, 'gid_24569': {'gid': 24569, 'name': 'b', 'rank': 0, 'incarnation': 14, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.189:6839/4074484168', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.189:6838', 'nonce': 4074484168}, {'type': 'v1', 'addr': '172.21.15.189:6839', 'nonce': 4074484168}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540701547738038271, 'flags': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_7': 'mds uses inline data', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2', 'feature_11': 'minor log segments'}}}}, 'data_pools': [3], 'metadata_pool': 2, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'bal_rank_mask': '-1', 'standby_count_wanted': 0, 'qdb_leader': 24569, 'qdb_cluster': [24511, 24550, 24488, 24499, 24569]} 2024-03-24T00:15:28.815 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 5/5 2024-03-24T00:15:28.815 DEBUG:teuthology.orchestra.run.smithi118:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.e status 2024-03-24T00:15:29.018 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:29.016+0000 7fccbf000640 1 -- 172.21.15.118:0/192699540 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fccb8102130 msgr2=0x7fccb8104520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:15:29.018 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:29.016+0000 7fccbf000640 1 --2- 172.21.15.118:0/192699540 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fccb8102130 0x7fccb8104520 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7fcca8009900 tx=0x7fcca802bc30 comp rx=0 tx=0).stop 2024-03-24T00:15:29.018 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:29.016+0000 7fccbf000640 1 -- 172.21.15.118:0/192699540 shutdown_connections 2024-03-24T00:15:29.018 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:29.016+0000 7fccbf000640 1 --2- 172.21.15.118:0/192699540 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fccb8107390 0x7fccb81097e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:15:29.018 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:29.016+0000 7fccbf000640 1 --2- 172.21.15.118:0/192699540 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fccb8104a60 0x7fccb8106e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:15:29.018 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:29.016+0000 7fccbf000640 1 --2- 172.21.15.118:0/192699540 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fccb8102130 0x7fccb8104520 unknown :-1 s=CLOSED pgs=166 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:15:29.019 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:29.016+0000 7fccbf000640 1 -- 172.21.15.118:0/192699540 >> 172.21.15.118:0/192699540 conn(0x7fccb80fbb40 msgr2=0x7fccb80fdf60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:15:29.019 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:29.016+0000 7fccbf000640 1 -- 172.21.15.118:0/192699540 shutdown_connections 2024-03-24T00:15:29.019 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:29.016+0000 7fccbf000640 1 -- 172.21.15.118:0/192699540 wait complete. 2024-03-24T00:15:29.022 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:29.018+0000 7fccbf000640 1 Processor -- start 2024-03-24T00:15:29.022 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:29.018+0000 7fccbf000640 1 -- start start 2024-03-24T00:15:29.022 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:29.018+0000 7fccbf000640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fccb8102130 0x7fccb8199c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:15:29.022 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:29.019+0000 7fccbf000640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fccb8104a60 0x7fccb819a1b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:15:29.023 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:29.019+0000 7fccbf000640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fccb8107390 0x7fccb81a1210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:15:29.023 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:29.019+0000 7fccbf000640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fccb810d3a0 con 0x7fccb8102130 2024-03-24T00:15:29.023 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:29.019+0000 7fccbf000640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fccb810d220 con 0x7fccb8104a60 2024-03-24T00:15:29.023 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:29.019+0000 7fccbf000640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fccb810d520 con 0x7fccb8107390 2024-03-24T00:15:29.023 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:29.019+0000 7fccb7400640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fccb8104a60 0x7fccb819a1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:15:29.023 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:29.019+0000 7fccb7400640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fccb8104a60 0x7fccb819a1b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.118:54904/0 (socket says 172.21.15.118:54904) 2024-03-24T00:15:29.023 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:29.019+0000 7fccb7400640 1 -- 172.21.15.118:0/398713640 learned_addr learned my addr 172.21.15.118:0/398713640 (peer_addr_for_me v2:172.21.15.118:0/0) 2024-03-24T00:15:29.023 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:29.019+0000 7fccb7e00640 1 --2- 172.21.15.118:0/398713640 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fccb8102130 0x7fccb8199c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:15:29.023 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:29.019+0000 7fccbd000640 1 --2- 172.21.15.118:0/398713640 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fccb8107390 0x7fccb81a1210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:15:29.023 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:29.019+0000 7fccb7400640 1 -- 172.21.15.118:0/398713640 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fccb8107390 msgr2=0x7fccb81a1210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:15:29.023 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:29.019+0000 7fccb7400640 1 --2- 172.21.15.118:0/398713640 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fccb8107390 0x7fccb81a1210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:15:29.023 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:29.019+0000 7fccb7400640 1 -- 172.21.15.118:0/398713640 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fccb8102130 msgr2=0x7fccb8199c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:15:29.023 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:29.019+0000 7fccb7400640 1 --2- 172.21.15.118:0/398713640 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fccb8102130 0x7fccb8199c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:15:29.023 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:29.019+0000 7fccb7400640 1 -- 172.21.15.118:0/398713640 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fccb81a18b0 con 0x7fccb8104a60 2024-03-24T00:15:29.024 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:29.019+0000 7fccb7400640 1 --2- 172.21.15.118:0/398713640 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fccb8104a60 0x7fccb819a1b0 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7fcca400e9c0 tx=0x7fcca400ee80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:15:29.024 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:29.020+0000 7fccb4c00640 1 -- 172.21.15.118:0/398713640 <== mon.2 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcca400cd80 con 0x7fccb8104a60 2024-03-24T00:15:29.024 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:29.020+0000 7fccb4c00640 1 -- 172.21.15.118:0/398713640 <== mon.2 v2:172.21.15.189:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fcca4005500 con 0x7fccb8104a60 2024-03-24T00:15:29.024 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:29.020+0000 7fccb4c00640 1 -- 172.21.15.118:0/398713640 <== mon.2 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcca4010690 con 0x7fccb8104a60 2024-03-24T00:15:29.024 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:29.020+0000 7fccbf000640 1 -- 172.21.15.118:0/398713640 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fccb81a1ba0 con 0x7fccb8104a60 2024-03-24T00:15:29.024 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:29.020+0000 7fccbf000640 1 -- 172.21.15.118:0/398713640 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fccb81a2110 con 0x7fccb8104a60 2024-03-24T00:15:29.027 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:29.023+0000 7fccc1da4740 1 Processor -- start 2024-03-24T00:15:29.027 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:29.023+0000 7fccc1da4740 1 -- start start 2024-03-24T00:15:29.027 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:29.023+0000 7fccc1da4740 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55a6819ecba0 0x55a6819ecfc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:15:29.027 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:29.023+0000 7fccc1da4740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55a6819ed590 0x55a6819f2090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:15:29.027 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:29.023+0000 7fccc1da4740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55a6819f2700 0x55a6819f4b90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:15:29.027 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:29.023+0000 7fccc1da4740 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a681951c90 con 0x55a6819ecba0 2024-03-24T00:15:29.027 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:29.023+0000 7fccc1da4740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a6818fa980 con 0x55a6819ed590 2024-03-24T00:15:29.027 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:29.023+0000 7fccc1da4740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a68194dd30 con 0x55a6819f2700 2024-03-24T00:15:29.027 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:29.024+0000 7fccb4c00640 1 -- 172.21.15.118:0/398713640 <== mon.2 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 18) v1 ==== 137291+0+0 (secure 0 0 0) 0x7fcca40046c0 con 0x7fccb8104a60 2024-03-24T00:15:29.027 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:29.024+0000 7fccb7e00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55a6819ed590 0x55a6819f2090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:15:29.027 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:29.024+0000 7fccbd000640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55a6819ecba0 0x55a6819ecfc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:15:29.027 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:29.024+0000 7fccb7e00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55a6819ed590 0x55a6819f2090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.118:54914/0 (socket says 172.21.15.118:54914) 2024-03-24T00:15:29.028 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:29.024+0000 7fccb7e00640 1 -- 172.21.15.118:0/1730061565 learned_addr learned my addr 172.21.15.118:0/1730061565 (peer_addr_for_me v2:172.21.15.118:0/0) 2024-03-24T00:15:29.028 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:29.024+0000 7fccb7400640 1 --2- 172.21.15.118:0/1730061565 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55a6819f2700 0x55a6819f4b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:15:29.028 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:29.024+0000 7fccb7e00640 1 -- 172.21.15.118:0/1730061565 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55a6819f2700 msgr2=0x55a6819f4b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:15:29.028 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:29.024+0000 7fccb7e00640 1 --2- 172.21.15.118:0/1730061565 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55a6819f2700 0x55a6819f4b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:15:29.028 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:29.024+0000 7fccb7e00640 1 -- 172.21.15.118:0/1730061565 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55a6819ecba0 msgr2=0x55a6819ecfc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:15:29.028 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:29.024+0000 7fccb7e00640 1 --2- 172.21.15.118:0/1730061565 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55a6819ecba0 0x55a6819ecfc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:15:29.028 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:29.024+0000 7fccb7e00640 1 -- 172.21.15.118:0/1730061565 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55a681912510 con 0x55a6819ed590 2024-03-24T00:15:29.028 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:29.024+0000 7fccbd000640 1 --2- 172.21.15.118:0/1730061565 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55a6819ecba0 0x55a6819ecfc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T00:15:29.028 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:29.025+0000 7fccb7e00640 1 --2- 172.21.15.118:0/1730061565 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55a6819ed590 0x55a6819f2090 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7fcca8005ab0 tx=0x7fcca8009560 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ff793904613ba0fd server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:15:29.028 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:29.025+0000 7fccad600640 1 -- 172.21.15.118:0/1730061565 <== mon.2 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcca8045070 con 0x55a6819ed590 2024-03-24T00:15:29.028 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:29.025+0000 7fccad600640 1 -- 172.21.15.118:0/1730061565 <== mon.2 v2:172.21.15.189:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fcca8004040 con 0x55a6819ed590 2024-03-24T00:15:29.028 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:29.025+0000 7fccad600640 1 -- 172.21.15.118:0/1730061565 <== mon.2 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcca8004eb0 con 0x55a6819ed590 2024-03-24T00:15:29.028 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:29.025+0000 7fccb4c00640 1 --2- 172.21.15.118:0/398713640 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7fcc9c0a2880 0x7fcc9c0a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:15:29.028 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:29.026+0000 7fccb4c00640 1 -- 172.21.15.118:0/398713640 <== mon.2 v2:172.21.15.189:3300/0 5 ==== osd_map(90..90 src has 1..90) v4 ==== 8277+0+0 (secure 0 0 0) 0x7fcca40ced40 con 0x7fccb8104a60 2024-03-24T00:15:29.029 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:29.026+0000 7fccc1da4740 1 -- 172.21.15.118:0/1730061565 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55a6819ed590 msgr2=0x55a6819f2090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:15:29.029 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:29.026+0000 7fccc1da4740 1 --2- 172.21.15.118:0/1730061565 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55a6819ed590 0x55a6819f2090 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7fcca8005ab0 tx=0x7fcca8009560 comp rx=0 tx=0).stop 2024-03-24T00:15:29.029 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:29.026+0000 7fccc1da4740 1 -- 172.21.15.118:0/1730061565 shutdown_connections 2024-03-24T00:15:29.029 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:29.026+0000 7fccc1da4740 1 --2- 172.21.15.118:0/1730061565 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55a6819f2700 0x55a6819f4b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:15:29.029 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:29.026+0000 7fccc1da4740 1 --2- 172.21.15.118:0/1730061565 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55a6819ed590 0x55a6819f2090 unknown :-1 s=CLOSED pgs=168 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:15:29.029 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:29.026+0000 7fccc1da4740 1 --2- 172.21.15.118:0/1730061565 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55a6819ecba0 0x55a6819ecfc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:15:29.029 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:29.026+0000 7fccc1da4740 1 -- 172.21.15.118:0/1730061565 >> 172.21.15.118:0/1730061565 conn(0x55a6819e8110 msgr2=0x55a6819e8510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:15:29.029 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:29.026+0000 7fccbd000640 1 --2- 172.21.15.118:0/398713640 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7fcc9c0a2880 0x7fcc9c0a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:15:29.029 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:29.026+0000 7fccc1da4740 1 -- 172.21.15.118:0/1730061565 shutdown_connections 2024-03-24T00:15:29.029 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:29.026+0000 7fccc1da4740 1 -- 172.21.15.118:0/1730061565 wait complete. 2024-03-24T00:15:29.033 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:29.027+0000 7fccc1da4740 1 -- start start 2024-03-24T00:15:29.034 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:29.027+0000 7fccbd000640 1 --2- 172.21.15.118:0/398713640 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7fcc9c0a2880 0x7fcc9c0a4d40 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x55a6819eddf0 tx=0x7fccb0009500 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:15:29.034 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:29.027+0000 7fccc1da4740 1 Processor -- start 2024-03-24T00:15:29.034 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:29.027+0000 7fccc1da4740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55a6819ecba0 0x55a6819f70b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:15:29.034 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:29.027+0000 7fccc1da4740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55a6819ed590 0x55a6819f75f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:15:29.034 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:29.027+0000 7fccc1da4740 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55a6819f2700 0x55a6819fe630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:15:29.034 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:29.027+0000 7fccc1da4740 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a6819f5650 con 0x55a6819f2700 2024-03-24T00:15:29.034 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:29.027+0000 7fccc1da4740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a68190ddd0 con 0x55a6819ecba0 2024-03-24T00:15:29.034 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:29.027+0000 7fccc1da4740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a6819f57d0 con 0x55a6819ed590 2024-03-24T00:15:29.034 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:29.027+0000 7fccc1da4740 10 client.0 _ll_register_callbacks cb 0x55a681665ac0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T00:15:29.034 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:29.027+0000 7fccb7e00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55a6819ecba0 0x55a6819f70b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:15:29.034 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:29.027+0000 7fccb7e00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55a6819ecba0 0x55a6819f70b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.118:54922/0 (socket says 172.21.15.118:54922) 2024-03-24T00:15:29.034 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:29.027+0000 7fccb7e00640 1 -- 172.21.15.118:0/1514913323 learned_addr learned my addr 172.21.15.118:0/1514913323 (peer_addr_for_me v2:172.21.15.118:0/0) 2024-03-24T00:15:29.035 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:29.027+0000 7fccb7400640 1 --2- 172.21.15.118:0/1514913323 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55a6819ed590 0x55a6819f75f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:15:29.035 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:29.028+0000 7fccbd000640 1 --2- 172.21.15.118:0/1514913323 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55a6819f2700 0x55a6819fe630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:15:29.035 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:29.028+0000 7fccb7e00640 1 -- 172.21.15.118:0/1514913323 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55a6819ed590 msgr2=0x55a6819f75f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:15:29.035 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:29.028+0000 7fccb7e00640 1 --2- 172.21.15.118:0/1514913323 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55a6819ed590 0x55a6819f75f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:15:29.035 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:29.028+0000 7fccb7e00640 1 -- 172.21.15.118:0/1514913323 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55a6819f2700 msgr2=0x55a6819fe630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:15:29.035 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:29.028+0000 7fccb7e00640 1 --2- 172.21.15.118:0/1514913323 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55a6819f2700 0x55a6819fe630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:15:29.035 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:29.028+0000 7fccb7e00640 1 -- 172.21.15.118:0/1514913323 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55a6816662d0 con 0x55a6819ecba0 2024-03-24T00:15:29.035 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:29.028+0000 7fccb7400640 1 --2- 172.21.15.118:0/1514913323 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55a6819ed590 0x55a6819f75f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T00:15:29.035 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:29.028+0000 7fccb7e00640 1 --2- 172.21.15.118:0/1514913323 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55a6819ecba0 0x55a6819f70b0 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7fcca802c7b0 tx=0x7fcca8004420 comp rx=0 tx=0).ready entity=mon.2 client_cookie=cca8870ae5d50c99 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:15:29.035 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:29.028+0000 7fcca2000640 10 client.0 ms_handle_connect on v2:172.21.15.189:3300/0 2024-03-24T00:15:29.035 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:29.028+0000 7fcca2000640 1 -- 172.21.15.118:0/1514913323 <== mon.2 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcca8045040 con 0x55a6819ecba0 2024-03-24T00:15:29.035 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:29.028+0000 7fcca2000640 1 -- 172.21.15.118:0/1514913323 <== mon.2 v2:172.21.15.189:3300/0 2 ==== osd_map(90..90 src has 1..90) v4 ==== 8277+0+0 (secure 0 0 0) 0x7fcca8004eb0 con 0x55a6819ecba0 2024-03-24T00:15:29.035 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:29.028+0000 7fccc1da4740 1 -- 172.21.15.118:0/1514913323 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fcca8004040 con 0x55a6819ecba0 2024-03-24T00:15:29.035 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:29.029+0000 7fcca2000640 1 -- 172.21.15.118:0/1514913323 <== mon.2 v2:172.21.15.189:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) v2 ==== 24+0+0 (secure 0 0 0) 0x7fcca8058050 con 0x55a6819ecba0 2024-03-24T00:15:29.036 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:29.029+0000 7fccc1da4740 10 client.25163 fetch_fsmap learned FSMap version 27 2024-03-24T00:15:29.036 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:29.029+0000 7fccc1da4740 1 -- 172.21.15.118:0/1514913323 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) v3 -- 0x55a6817306b0 con 0x55a6819ecba0 2024-03-24T00:15:29.036 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:29.029+0000 7fcca2000640 1 -- 172.21.15.118:0/1514913323 <== mon.2 v2:172.21.15.189:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fcca8004040 con 0x55a6819ecba0 2024-03-24T00:15:29.036 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:29.029+0000 7fcca2000640 1 -- 172.21.15.118:0/1514913323 <== mon.2 v2:172.21.15.189:3300/0 5 ==== fsmap(e 27) v1 ==== 7934+0+0 (secure 0 0 0) 0x7fcca8003b10 con 0x55a6819ecba0 2024-03-24T00:15:29.036 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:29.029+0000 7fccc1da4740 10 client.25163 fetch_fsmap finished waiting for FSMap version 27 2024-03-24T00:15:29.036 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:29.029+0000 7fccc1da4740 10 client.25163 resolve_mds: resolved name 'e' to daemon mds.e 2024-03-24T00:15:29.036 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:29.029+0000 7fccc1da4740 20 client.25163 populate_metadata read hostname 'smithi118' 2024-03-24T00:15:29.036 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:29.029+0000 7fccc1da4740 1 --2- 172.21.15.118:0/1514913323 >> [v2:172.21.15.189:6834/599473205,v1:172.21.15.189:6835/599473205] conn(0x55a681a02790 0x55a681a07830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:15:29.036 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:29.029+0000 7fccc1da4740 4 client.25163 mds_command: new command op to 24488 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-24T00:15:29.036 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:29.029+0000 7fccc1da4740 1 -- 172.21.15.118:0/1514913323 --> [v2:172.21.15.189:6834/599473205,v1:172.21.15.189:6835/599473205] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55a6818f3520 con 0x55a681a02790 2024-03-24T00:15:29.036 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:29.030+0000 7fccb7400640 1 --2- 172.21.15.118:0/1514913323 >> [v2:172.21.15.189:6834/599473205,v1:172.21.15.189:6835/599473205] conn(0x55a681a02790 0x55a681a07830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:15:29.036 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:29.030+0000 7fcca2000640 10 client.25163 ms_handle_connect on v2:172.21.15.189:6834/599473205 2024-03-24T00:15:29.036 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:29.030+0000 7fccb7400640 1 --2- 172.21.15.118:0/1514913323 >> [v2:172.21.15.189:6834/599473205,v1:172.21.15.189:6835/599473205] conn(0x55a681a02790 0x55a681a07830 crc :-1 s=READY pgs=18 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=1811b94e86f31688 server_cookie=9586ea4242e223a5 in_seq=0 out_seq=0 2024-03-24T00:15:29.036 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:29.031+0000 7fcca2000640 1 -- 172.21.15.118:0/1514913323 <== mds.4 v2:172.21.15.189:6834/599473205 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x55a6818f3520 con 0x55a681a02790 2024-03-24T00:15:29.037 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:29.031+0000 7fcca2000640 10 client.25163 handle_command_reply: tid=0 2024-03-24T00:15:29.037 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:29.031+0000 7fccc1da4740 1 client.25163 shutdown 2024-03-24T00:15:29.037 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:29.031+0000 7fccc1da4740 1 -- 172.21.15.118:0/1514913323 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55a6819ecba0 msgr2=0x55a6819f70b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:15:29.037 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:29.031+0000 7fccc1da4740 1 --2- 172.21.15.118:0/1514913323 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55a6819ecba0 0x55a6819f70b0 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7fcca802c7b0 tx=0x7fcca8004420 comp rx=0 tx=0).stop 2024-03-24T00:15:29.037 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:29.031+0000 7fccc1da4740 1 -- 172.21.15.118:0/1514913323 shutdown_connections 2024-03-24T00:15:29.037 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:29.031+0000 7fccc1da4740 1 --2- 172.21.15.118:0/1514913323 >> [v2:172.21.15.189:6834/599473205,v1:172.21.15.189:6835/599473205] conn(0x55a681a02790 0x55a681a07830 crc :-1 s=READY pgs=18 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:15:29.037 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:29.031+0000 7fccc1da4740 1 --2- 172.21.15.118:0/1514913323 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55a6819f2700 0x55a6819fe630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:15:29.037 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:29.031+0000 7fccc1da4740 1 --2- 172.21.15.118:0/1514913323 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55a6819ed590 0x55a6819f75f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:15:29.037 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:29.031+0000 7fcca2000640 0 client.25163 ms_handle_reset on v2:172.21.15.189:6834/599473205 2024-03-24T00:15:29.037 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:29.031+0000 7fccc1da4740 1 --2- 172.21.15.118:0/1514913323 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55a6819ecba0 0x55a6819f70b0 unknown :-1 s=CLOSED pgs=169 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:15:29.037 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:29.031+0000 7fccc1da4740 1 -- 172.21.15.118:0/1514913323 >> 172.21.15.118:0/1514913323 conn(0x55a6819e8110 msgr2=0x55a6819e7310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:15:29.037 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:29.032+0000 7fccc1da4740 1 -- 172.21.15.118:0/1514913323 shutdown_connections 2024-03-24T00:15:29.037 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:29.032+0000 7fccc1da4740 1 -- 172.21.15.118:0/1514913323 wait complete. 2024-03-24T00:15:29.038 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:29.032+0000 7fccc1da4740 20 client.25163 trim_cache size 0 max 16384 2024-03-24T00:15:29.051 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:29.050+0000 7fccc1da4740 1 Processor -- start 2024-03-24T00:15:29.051 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:29.050+0000 7fccc1da4740 1 -- start start 2024-03-24T00:15:29.051 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:29.050+0000 7fccc1da4740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55a6819ecba0 0x55a6819fecc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:15:29.051 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:29.050+0000 7fccc1da4740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55a6819ed590 0x55a6819f5c00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:15:29.051 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:29.050+0000 7fccc1da4740 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55a6819f2700 0x55a6819e84f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:15:29.051 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:29.050+0000 7fccc1da4740 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a681a02bd0 con 0x55a6819f2700 2024-03-24T00:15:29.051 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:29.050+0000 7fccc1da4740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a681a03230 con 0x55a6819ecba0 2024-03-24T00:15:29.051 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:29.050+0000 7fccc1da4740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a681a03890 con 0x55a6819ed590 2024-03-24T00:15:29.052 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:29.050+0000 7fccb7e00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55a6819ecba0 0x55a6819fecc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:15:29.052 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:29.050+0000 7fccb7e00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55a6819ecba0 0x55a6819fecc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.118:54938/0 (socket says 172.21.15.118:54938) 2024-03-24T00:15:29.052 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:29.050+0000 7fccb7e00640 1 -- 172.21.15.118:0/2697556161 learned_addr learned my addr 172.21.15.118:0/2697556161 (peer_addr_for_me v2:172.21.15.118:0/0) 2024-03-24T00:15:29.052 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:29.050+0000 7fccbd000640 1 --2- 172.21.15.118:0/2697556161 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55a6819f2700 0x55a6819e84f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:15:29.052 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:29.050+0000 7fccb7400640 1 --2- 172.21.15.118:0/2697556161 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55a6819ed590 0x55a6819f5c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:15:29.052 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:29.050+0000 7fccb7e00640 1 -- 172.21.15.118:0/2697556161 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55a6819ed590 msgr2=0x55a6819f5c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:15:29.052 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:29.050+0000 7fccb7e00640 1 --2- 172.21.15.118:0/2697556161 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55a6819ed590 0x55a6819f5c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:15:29.052 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:29.050+0000 7fccb7e00640 1 -- 172.21.15.118:0/2697556161 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55a6819f2700 msgr2=0x55a6819e84f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:15:29.052 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:29.050+0000 7fccb7e00640 1 --2- 172.21.15.118:0/2697556161 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55a6819f2700 0x55a6819e84f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:15:29.052 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:29.050+0000 7fccb7e00640 1 -- 172.21.15.118:0/2697556161 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55a6819f5710 con 0x55a6819ecba0 2024-03-24T00:15:29.052 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:29.050+0000 7fccb7400640 1 --2- 172.21.15.118:0/2697556161 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55a6819ed590 0x55a6819f5c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T00:15:29.052 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:29.051+0000 7fccb7e00640 1 --2- 172.21.15.118:0/2697556161 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55a6819ecba0 0x55a6819fecc0 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7fcca802c170 tx=0x7fcca802c880 comp rx=0 tx=0).ready entity=mon.2 client_cookie=fddcf7b974a75ea8 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:15:29.057 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:29.051+0000 7fccacc00640 1 -- 172.21.15.118:0/2697556161 <== mon.2 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcca8045030 con 0x55a6819ecba0 2024-03-24T00:15:29.057 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:29.051+0000 7fccacc00640 1 -- 172.21.15.118:0/2697556161 <== mon.2 v2:172.21.15.189:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fcca8004490 con 0x55a6819ecba0 2024-03-24T00:15:29.057 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:29.051+0000 7fccacc00640 1 -- 172.21.15.118:0/2697556161 <== mon.2 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcca8004900 con 0x55a6819ecba0 2024-03-24T00:15:29.057 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:29.051+0000 7fccc1da4740 1 -- 172.21.15.118:0/2697556161 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55a6819ecba0 msgr2=0x55a6819fecc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:15:29.057 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:29.051+0000 7fccc1da4740 1 --2- 172.21.15.118:0/2697556161 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55a6819ecba0 0x55a6819fecc0 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7fcca802c170 tx=0x7fcca802c880 comp rx=0 tx=0).stop 2024-03-24T00:15:29.057 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:29.052+0000 7fccc1da4740 1 -- 172.21.15.118:0/2697556161 shutdown_connections 2024-03-24T00:15:29.057 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:29.052+0000 7fccc1da4740 1 --2- 172.21.15.118:0/2697556161 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55a6819f2700 0x55a6819e84f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:15:29.057 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:29.052+0000 7fccc1da4740 1 --2- 172.21.15.118:0/2697556161 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55a6819ed590 0x55a6819f5c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:15:29.058 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:29.052+0000 7fccc1da4740 1 --2- 172.21.15.118:0/2697556161 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55a6819ecba0 0x55a6819fecc0 unknown :-1 s=CLOSED pgs=170 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:15:29.058 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:29.052+0000 7fccc1da4740 1 -- 172.21.15.118:0/2697556161 >> 172.21.15.118:0/2697556161 conn(0x55a6819e8110 msgr2=0x55a6819ee000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:15:29.058 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:29.052+0000 7fccc1da4740 1 -- 172.21.15.118:0/2697556161 shutdown_connections 2024-03-24T00:15:29.058 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:29.052+0000 7fccc1da4740 1 -- 172.21.15.118:0/2697556161 wait complete. 2024-03-24T00:15:29.058 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:29.052+0000 7fccc1da4740 1 -- start start 2024-03-24T00:15:29.058 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:29.052+0000 7fccc1da4740 1 Processor -- start 2024-03-24T00:15:29.058 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:29.053+0000 7fccc1da4740 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55a6819ecba0 0x55a6819f91f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:15:29.058 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:29.053+0000 7fccc1da4740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55a6819ed590 0x55a6819f9730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:15:29.058 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:29.053+0000 7fccc1da4740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55a6819f2700 0x55a6819fc700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:15:29.058 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:29.053+0000 7fccc1da4740 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a681a04550 con 0x55a6819ecba0 2024-03-24T00:15:29.058 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:29.053+0000 7fccc1da4740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a681a03ef0 con 0x55a6819ed590 2024-03-24T00:15:29.058 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:29.053+0000 7fccc1da4740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x55a681a04bb0 con 0x55a6819f2700 2024-03-24T00:15:29.058 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:29.053+0000 7fccc1da4740 10 client.0 _ll_register_callbacks cb 0x7fccb0009b30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T00:15:29.059 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:29.053+0000 7fccb7e00640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55a6819ecba0 0x55a6819f91f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:15:29.059 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:29.053+0000 7fccbd000640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55a6819f2700 0x55a6819fc700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:15:29.059 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:29.053+0000 7fccb7400640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55a6819ed590 0x55a6819f9730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:15:29.059 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:29.053+0000 7fccbd000640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55a6819f2700 0x55a6819fc700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.118:46946/0 (socket says 172.21.15.118:46946) 2024-03-24T00:15:29.059 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:29.053+0000 7fccb7400640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55a6819ed590 0x55a6819f9730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.118:54952/0 (socket says 172.21.15.118:54952) 2024-03-24T00:15:29.059 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:29.053+0000 7fccbd000640 1 -- 172.21.15.118:0/4090496706 learned_addr learned my addr 172.21.15.118:0/4090496706 (peer_addr_for_me v2:172.21.15.118:0/0) 2024-03-24T00:15:29.059 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:29.053+0000 7fccb7e00640 1 -- 172.21.15.118:0/4090496706 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55a6819f2700 msgr2=0x55a6819fc700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:15:29.059 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:29.053+0000 7fccb7e00640 1 --2- 172.21.15.118:0/4090496706 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55a6819f2700 0x55a6819fc700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:15:29.059 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:29.053+0000 7fccb7e00640 1 -- 172.21.15.118:0/4090496706 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55a6819ed590 msgr2=0x55a6819f9730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:15:29.059 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:29.053+0000 7fccb7e00640 1 --2- 172.21.15.118:0/4090496706 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55a6819ed590 0x55a6819f9730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:15:29.059 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:29.053+0000 7fccb7e00640 1 -- 172.21.15.118:0/4090496706 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55a6819fccf0 con 0x55a6819ecba0 2024-03-24T00:15:29.059 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:29.053+0000 7fccb7400640 1 --2- 172.21.15.118:0/4090496706 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55a6819ed590 0x55a6819f9730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T00:15:29.059 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:29.053+0000 7fccbd000640 1 --2- 172.21.15.118:0/4090496706 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55a6819f2700 0x55a6819fc700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T00:15:29.060 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:29.054+0000 7fccb7e00640 1 --2- 172.21.15.118:0/4090496706 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55a6819ecba0 0x55a6819f91f0 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x55a6819e6a20 tx=0x7fcca802c2b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9a253e171d8c62f4 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:15:29.060 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:29.054+0000 7fcca2000640 10 client.0 ms_handle_connect on v2:172.21.15.118:3300/0 2024-03-24T00:15:29.060 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:29.054+0000 7fcca2000640 1 -- 172.21.15.118:0/4090496706 <== mon.0 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcca8045040 con 0x55a6819ecba0 2024-03-24T00:15:29.060 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:29.054+0000 7fcca2000640 1 -- 172.21.15.118:0/4090496706 <== mon.0 v2:172.21.15.118:3300/0 2 ==== osd_map(90..90 src has 1..90) v4 ==== 8277+0+0 (secure 0 0 0) 0x7fcca804a5f0 con 0x55a6819ecba0 2024-03-24T00:15:29.060 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:29.054+0000 7fccc1da4740 1 -- 172.21.15.118:0/4090496706 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fcca8004490 con 0x55a6819ecba0 2024-03-24T00:15:29.060 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:29.054+0000 7fcca2000640 1 -- 172.21.15.118:0/4090496706 <== mon.0 v2:172.21.15.118:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) v2 ==== 24+0+0 (secure 0 0 0) 0x7fcca80048c0 con 0x55a6819ecba0 2024-03-24T00:15:29.060 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:29.054+0000 7fccc1da4740 10 client.15078 fetch_fsmap learned FSMap version 27 2024-03-24T00:15:29.060 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:29.054+0000 7fccc1da4740 1 -- 172.21.15.118:0/4090496706 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) v3 -- 0x55a6819fdff0 con 0x55a6819ecba0 2024-03-24T00:15:29.060 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:29.054+0000 7fcca2000640 1 -- 172.21.15.118:0/4090496706 <== mon.0 v2:172.21.15.118:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fcca8004490 con 0x55a6819ecba0 2024-03-24T00:15:29.060 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:29.055+0000 7fcca2000640 1 -- 172.21.15.118:0/4090496706 <== mon.0 v2:172.21.15.118:3300/0 5 ==== fsmap(e 27) v1 ==== 7934+0+0 (secure 0 0 0) 0x7fcca8004eb0 con 0x55a6819ecba0 2024-03-24T00:15:29.060 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:29.055+0000 7fccc1da4740 10 client.15078 fetch_fsmap finished waiting for FSMap version 27 2024-03-24T00:15:29.060 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:29.055+0000 7fccc1da4740 10 client.15078 resolve_mds: resolved name 'e' to daemon mds.e 2024-03-24T00:15:29.060 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:29.055+0000 7fccc1da4740 20 client.15078 populate_metadata read hostname 'smithi118' 2024-03-24T00:15:29.060 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:29.055+0000 7fccc1da4740 1 --2- 172.21.15.118:0/4090496706 >> [v2:172.21.15.189:6834/599473205,v1:172.21.15.189:6835/599473205] conn(0x55a681a62810 0x55a681a5e950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:15:29.061 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:29.055+0000 7fccc1da4740 4 client.15078 mds_command: new command op to 24488 tid=0 multi_id=0 [{"prefix": "status"}] 2024-03-24T00:15:29.061 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:29.055+0000 7fccc1da4740 1 -- 172.21.15.118:0/4090496706 --> [v2:172.21.15.189:6834/599473205,v1:172.21.15.189:6835/599473205] -- command(tid 0: {"prefix": "status"}) v1 -- 0x55a68195f1d0 con 0x55a681a62810 2024-03-24T00:15:29.061 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:29.055+0000 7fccb7400640 1 --2- 172.21.15.118:0/4090496706 >> [v2:172.21.15.189:6834/599473205,v1:172.21.15.189:6835/599473205] conn(0x55a681a62810 0x55a681a5e950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:15:29.061 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:29.055+0000 7fccb7400640 1 --2- 172.21.15.118:0/4090496706 >> [v2:172.21.15.189:6834/599473205,v1:172.21.15.189:6835/599473205] conn(0x55a681a62810 0x55a681a5e950 crc :-1 s=READY pgs=19 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=a6c536349fc140ee server_cookie=8dc95dd4873b39e8 in_seq=0 out_seq=0 2024-03-24T00:15:29.061 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:29.055+0000 7fcca2000640 10 client.15078 ms_handle_connect on v2:172.21.15.189:6834/599473205 2024-03-24T00:15:29.061 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:29.056+0000 7fcca2000640 1 -- 172.21.15.118:0/4090496706 <== mds.4 v2:172.21.15.189:6834/599473205 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+327 (crc 0 0 0) 0x55a68195f1d0 con 0x55a681a62810 2024-03-24T00:15:29.061 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:29.056+0000 7fcca2000640 10 client.15078 handle_command_reply: tid=0 2024-03-24T00:15:29.061 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:29.056+0000 7fccc1da4740 1 client.15078 shutdown 2024-03-24T00:15:29.061 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:29.056+0000 7fccc1da4740 1 -- 172.21.15.118:0/4090496706 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55a6819ecba0 msgr2=0x55a6819f91f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:15:29.061 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:29.056+0000 7fccc1da4740 1 --2- 172.21.15.118:0/4090496706 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55a6819ecba0 0x55a6819f91f0 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x55a6819e6a20 tx=0x7fcca802c2b0 comp rx=0 tx=0).stop 2024-03-24T00:15:29.061 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:29.056+0000 7fccc1da4740 1 -- 172.21.15.118:0/4090496706 shutdown_connections 2024-03-24T00:15:29.061 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:29.056+0000 7fccc1da4740 1 --2- 172.21.15.118:0/4090496706 >> [v2:172.21.15.189:6834/599473205,v1:172.21.15.189:6835/599473205] conn(0x55a681a62810 0x55a681a5e950 crc :-1 s=READY pgs=19 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:15:29.061 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:29.056+0000 7fccc1da4740 1 --2- 172.21.15.118:0/4090496706 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55a6819f2700 0x55a6819fc700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:15:29.061 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:29.056+0000 7fccc1da4740 1 --2- 172.21.15.118:0/4090496706 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55a6819ed590 0x55a6819f9730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:15:29.062 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:29.056+0000 7fccc1da4740 1 --2- 172.21.15.118:0/4090496706 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55a6819ecba0 0x55a6819f91f0 unknown :-1 s=CLOSED pgs=173 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:15:29.062 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:29.056+0000 7fcca2000640 0 client.15078 ms_handle_reset on v2:172.21.15.189:6834/599473205 2024-03-24T00:15:29.062 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:29.056+0000 7fccc1da4740 1 -- 172.21.15.118:0/4090496706 >> 172.21.15.118:0/4090496706 conn(0x55a6819e8110 msgr2=0x55a6819f5c00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:15:29.062 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:29.056+0000 7fccc1da4740 1 -- 172.21.15.118:0/4090496706 shutdown_connections 2024-03-24T00:15:29.062 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:29.056+0000 7fccc1da4740 1 -- 172.21.15.118:0/4090496706 wait complete. 2024-03-24T00:15:29.062 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:29.056+0000 7fccc1da4740 20 client.15078 trim_cache size 0 max 16384 2024-03-24T00:15:29.062 INFO:teuthology.orchestra.run.smithi118.stdout:{ 2024-03-24T00:15:29.062 INFO:teuthology.orchestra.run.smithi118.stdout: "cluster_fsid": "d79a5420-e971-11ee-95cd-87774f69a715", 2024-03-24T00:15:29.062 INFO:teuthology.orchestra.run.smithi118.stdout: "whoami": 4, 2024-03-24T00:15:29.062 INFO:teuthology.orchestra.run.smithi118.stdout: "id": 24488, 2024-03-24T00:15:29.063 INFO:teuthology.orchestra.run.smithi118.stdout: "want_state": "up:active", 2024-03-24T00:15:29.063 INFO:teuthology.orchestra.run.smithi118.stdout: "state": "up:active", 2024-03-24T00:15:29.063 INFO:teuthology.orchestra.run.smithi118.stdout: "fs_name": "cephfs", 2024-03-24T00:15:29.063 INFO:teuthology.orchestra.run.smithi118.stdout: "rank_uptime": 140.72564546000001, 2024-03-24T00:15:29.063 INFO:teuthology.orchestra.run.smithi118.stdout: "mdsmap_epoch": 27, 2024-03-24T00:15:29.063 INFO:teuthology.orchestra.run.smithi118.stdout: "osdmap_epoch": 90, 2024-03-24T00:15:29.063 INFO:teuthology.orchestra.run.smithi118.stdout: "osdmap_epoch_barrier": 86, 2024-03-24T00:15:29.063 INFO:teuthology.orchestra.run.smithi118.stdout: "uptime": 176.375973262 2024-03-24T00:15:29.063 INFO:teuthology.orchestra.run.smithi118.stdout:} 2024-03-24T00:15:29.064 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:29.057+0000 7fccbf000640 1 -- 172.21.15.118:0/398713640 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7fcc9c0a2880 msgr2=0x7fcc9c0a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:15:29.064 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:29.057+0000 7fccbf000640 1 --2- 172.21.15.118:0/398713640 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7fcc9c0a2880 0x7fcc9c0a4d40 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x55a6819eddf0 tx=0x7fccb0009500 comp rx=0 tx=0).stop 2024-03-24T00:15:29.064 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:29.057+0000 7fccbf000640 1 -- 172.21.15.118:0/398713640 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fccb8104a60 msgr2=0x7fccb819a1b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:15:29.064 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:29.057+0000 7fccbf000640 1 --2- 172.21.15.118:0/398713640 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fccb8104a60 0x7fccb819a1b0 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7fcca400e9c0 tx=0x7fcca400ee80 comp rx=0 tx=0).stop 2024-03-24T00:15:29.064 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:29.057+0000 7fccbf000640 1 -- 172.21.15.118:0/398713640 shutdown_connections 2024-03-24T00:15:29.064 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:29.057+0000 7fccbf000640 1 --2- 172.21.15.118:0/398713640 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7fcc9c0a2880 0x7fcc9c0a4d40 unknown :-1 s=CLOSED pgs=243 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:15:29.064 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:29.057+0000 7fccbf000640 1 --2- 172.21.15.118:0/398713640 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fccb8107390 0x7fccb81a1210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:15:29.064 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:29.057+0000 7fccbf000640 1 --2- 172.21.15.118:0/398713640 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fccb8104a60 0x7fccb819a1b0 unknown :-1 s=CLOSED pgs=167 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:15:29.064 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:29.057+0000 7fccbf000640 1 --2- 172.21.15.118:0/398713640 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fccb8102130 0x7fccb8199c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:15:29.064 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:29.057+0000 7fccbf000640 1 -- 172.21.15.118:0/398713640 >> 172.21.15.118:0/398713640 conn(0x7fccb80fbb40 msgr2=0x7fccb8101910 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:15:29.064 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:29.057+0000 7fccbf000640 1 -- 172.21.15.118:0/398713640 shutdown_connections 2024-03-24T00:15:29.065 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:29.057+0000 7fccbf000640 1 -- 172.21.15.118:0/398713640 wait complete. 2024-03-24T00:15:29.073 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:15:28 smithi118 ceph-mon[31535]: from='client.? 172.21.15.118:0/2051213444' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-24T00:15:29.073 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:15:28 smithi118 ceph-mon[31535]: from='client.? 172.21.15.118:0/2495320211' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-24T00:15:29.073 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:15:28 smithi118 ceph-mon[31535]: from='client.? 172.21.15.118:0/1271532583' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-24T00:15:29.077 DEBUG:teuthology.orchestra.run.smithi118:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.f status 2024-03-24T00:15:29.203 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:15:28 smithi189 ceph-mon[36187]: from='client.? 172.21.15.118:0/2051213444' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-24T00:15:29.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:15:28 smithi189 ceph-mon[36187]: from='client.? 172.21.15.118:0/2495320211' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-24T00:15:29.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:15:28 smithi189 ceph-mon[36187]: from='client.? 172.21.15.118:0/1271532583' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-24T00:15:29.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:15:28 smithi193 ceph-mon[36878]: from='client.? 172.21.15.118:0/2051213444' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-24T00:15:29.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:15:28 smithi193 ceph-mon[36878]: from='client.? 172.21.15.118:0/2495320211' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-24T00:15:29.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:15:28 smithi193 ceph-mon[36878]: from='client.? 172.21.15.118:0/1271532583' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-24T00:15:29.272 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:29.271+0000 7f3013200640 1 -- 172.21.15.118:0/3443447383 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3014107260 msgr2=0x7f30141096b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:15:29.272 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:29.271+0000 7f3013200640 1 --2- 172.21.15.118:0/3443447383 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3014107260 0x7f30141096b0 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7f300c009ca0 tx=0x7f300c02c100 comp rx=0 tx=0).stop 2024-03-24T00:15:29.272 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:29.271+0000 7f3013200640 1 -- 172.21.15.118:0/3443447383 shutdown_connections 2024-03-24T00:15:29.272 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:29.271+0000 7f3013200640 1 --2- 172.21.15.118:0/3443447383 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3014107260 0x7f30141096b0 unknown :-1 s=CLOSED pgs=171 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:15:29.272 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:29.271+0000 7f3013200640 1 --2- 172.21.15.118:0/3443447383 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f3014104930 0x7f3014106d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:15:29.272 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:29.271+0000 7f3013200640 1 --2- 172.21.15.118:0/3443447383 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f3014102000 0x7f30141043f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:15:29.272 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:29.271+0000 7f3013200640 1 -- 172.21.15.118:0/3443447383 >> 172.21.15.118:0/3443447383 conn(0x7f30140fba10 msgr2=0x7f30140fde30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:15:29.272 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:29.271+0000 7f3013200640 1 -- 172.21.15.118:0/3443447383 shutdown_connections 2024-03-24T00:15:29.272 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:29.271+0000 7f3013200640 1 -- 172.21.15.118:0/3443447383 wait complete. 2024-03-24T00:15:29.273 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:29.272+0000 7f3013200640 1 Processor -- start 2024-03-24T00:15:29.273 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:29.273+0000 7f3013200640 1 -- start start 2024-03-24T00:15:29.275 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:29.273+0000 7f3013200640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f3014102000 0x7f3014101970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:15:29.275 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:29.273+0000 7f3013200640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f3014104930 0x7f30140fffb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:15:29.275 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:29.273+0000 7f3013200640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3014107260 0x7f30141004f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:15:29.275 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:29.273+0000 7f3013200640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f301410d210 con 0x7f3014104930 2024-03-24T00:15:29.275 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:29.273+0000 7f3013200640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f301410d090 con 0x7f3014107260 2024-03-24T00:15:29.275 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:29.273+0000 7f3013200640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f301410d390 con 0x7f3014102000 2024-03-24T00:15:29.275 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:29.273+0000 7f3011e00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f3014102000 0x7f3014101970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:15:29.275 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:29.273+0000 7f3011400640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f3014104930 0x7f30140fffb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:15:29.275 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:29.273+0000 7f3011e00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f3014102000 0x7f3014101970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.118:46964/0 (socket says 172.21.15.118:46964) 2024-03-24T00:15:29.276 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:29.273+0000 7f3012800640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3014107260 0x7f30141004f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:15:29.276 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:29.273+0000 7f3011e00640 1 -- 172.21.15.118:0/1191067842 learned_addr learned my addr 172.21.15.118:0/1191067842 (peer_addr_for_me v2:172.21.15.118:0/0) 2024-03-24T00:15:29.276 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:29.274+0000 7f3012800640 1 -- 172.21.15.118:0/1191067842 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f3014102000 msgr2=0x7f3014101970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:15:29.276 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:29.274+0000 7f3012800640 1 --2- 172.21.15.118:0/1191067842 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f3014102000 0x7f3014101970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:15:29.276 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:29.274+0000 7f3012800640 1 -- 172.21.15.118:0/1191067842 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f3014104930 msgr2=0x7f30140fffb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:15:29.276 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:29.274+0000 7f3012800640 1 --2- 172.21.15.118:0/1191067842 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f3014104930 0x7f30140fffb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:15:29.276 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:29.274+0000 7f3012800640 1 -- 172.21.15.118:0/1191067842 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3014100d80 con 0x7f3014107260 2024-03-24T00:15:29.276 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:29.274+0000 7f3012800640 1 --2- 172.21.15.118:0/1191067842 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3014107260 0x7f30141004f0 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7f300c037040 tx=0x7f300c035d10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:15:29.276 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:29.274+0000 7f300aa00640 1 -- 172.21.15.118:0/1191067842 <== mon.2 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f300c039040 con 0x7f3014107260 2024-03-24T00:15:29.276 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:29.274+0000 7f300aa00640 1 -- 172.21.15.118:0/1191067842 <== mon.2 v2:172.21.15.189:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f300c044070 con 0x7f3014107260 2024-03-24T00:15:29.276 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:29.274+0000 7f3013200640 1 -- 172.21.15.118:0/1191067842 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f30141a1910 con 0x7f3014107260 2024-03-24T00:15:29.276 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:29.274+0000 7f300aa00640 1 -- 172.21.15.118:0/1191067842 <== mon.2 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f300c004450 con 0x7f3014107260 2024-03-24T00:15:29.276 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:29.274+0000 7f3013200640 1 -- 172.21.15.118:0/1191067842 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f30141a1e80 con 0x7f3014107260 2024-03-24T00:15:29.278 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:29.277+0000 7f300aa00640 1 -- 172.21.15.118:0/1191067842 <== mon.2 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 18) v1 ==== 137291+0+0 (secure 0 0 0) 0x7f300c0045f0 con 0x7f3014107260 2024-03-24T00:15:29.278 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:29.277+0000 7f301bbd7740 1 Processor -- start 2024-03-24T00:15:29.278 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:29.278+0000 7f301bbd7740 1 -- start start 2024-03-24T00:15:29.279 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:29.278+0000 7f301bbd7740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5588ea93cd80 0x5588ea93d1a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:15:29.280 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:29.278+0000 7f301bbd7740 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5588ea93d770 0x5588ea942270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:15:29.280 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:29.278+0000 7f301bbd7740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5588ea9428e0 0x5588ea944d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:15:29.280 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:29.278+0000 7f301bbd7740 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x5588ea8a1c90 con 0x5588ea93d770 2024-03-24T00:15:29.280 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:29.278+0000 7f301bbd7740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x5588ea84a980 con 0x5588ea9428e0 2024-03-24T00:15:29.280 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:29.278+0000 7f301bbd7740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x5588ea89dd30 con 0x5588ea93cd80 2024-03-24T00:15:29.280 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:29.278+0000 7f3011400640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5588ea93cd80 0x5588ea93d1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:15:29.280 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:29.278+0000 7f3011400640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5588ea93cd80 0x5588ea93d1a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.118:46972/0 (socket says 172.21.15.118:46972) 2024-03-24T00:15:29.280 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:29.278+0000 7f3011400640 1 -- 172.21.15.118:0/198132416 learned_addr learned my addr 172.21.15.118:0/198132416 (peer_addr_for_me v2:172.21.15.118:0/0) 2024-03-24T00:15:29.280 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:29.278+0000 7f3011e00640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5588ea93d770 0x5588ea942270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:15:29.280 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:29.278+0000 7f3011400640 1 --2- 172.21.15.118:0/198132416 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5588ea9428e0 0x5588ea944d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:15:29.280 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:29.279+0000 7f3011400640 1 -- 172.21.15.118:0/198132416 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5588ea9428e0 msgr2=0x5588ea944d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:15:29.280 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:29.279+0000 7f3011400640 1 --2- 172.21.15.118:0/198132416 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5588ea9428e0 0x5588ea944d70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:15:29.280 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:29.279+0000 7f3011400640 1 -- 172.21.15.118:0/198132416 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5588ea93d770 msgr2=0x5588ea942270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:15:29.281 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:29.279+0000 7f3011400640 1 --2- 172.21.15.118:0/198132416 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5588ea93d770 0x5588ea942270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:15:29.281 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:29.279+0000 7f3011400640 1 -- 172.21.15.118:0/198132416 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5588ea862510 con 0x5588ea93cd80 2024-03-24T00:15:29.281 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:29.279+0000 7f3011400640 1 --2- 172.21.15.118:0/198132416 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5588ea93cd80 0x5588ea93d1a0 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7f3000011740 tx=0x7f3000004730 comp rx=0 tx=0).ready entity=mon.1 client_cookie=9e77b74d916a54a4 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:15:29.281 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:29.279+0000 7f2fff400640 1 -- 172.21.15.118:0/198132416 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3000004050 con 0x5588ea93cd80 2024-03-24T00:15:29.281 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:29.279+0000 7f2fff400640 1 -- 172.21.15.118:0/198132416 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f3000005500 con 0x5588ea93cd80 2024-03-24T00:15:29.281 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:29.279+0000 7f2fff400640 1 -- 172.21.15.118:0/198132416 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3000004cb0 con 0x5588ea93cd80 2024-03-24T00:15:29.281 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:29.279+0000 7f300aa00640 1 --2- 172.21.15.118:0/1191067842 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7f2ff80a2880 0x7f2ff80a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:15:29.286 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:29.279+0000 7f3011e00640 1 --2- 172.21.15.118:0/1191067842 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7f2ff80a2880 0x7f2ff80a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:15:29.286 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:29.279+0000 7f300aa00640 1 -- 172.21.15.118:0/1191067842 <== mon.2 v2:172.21.15.189:3300/0 5 ==== osd_map(90..90 src has 1..90) v4 ==== 8277+0+0 (secure 0 0 0) 0x7f300c0f0ff0 con 0x7f3014107260 2024-03-24T00:15:29.286 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:29.280+0000 7f301bbd7740 1 -- 172.21.15.118:0/198132416 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5588ea93cd80 msgr2=0x5588ea93d1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:15:29.286 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:29.280+0000 7f301bbd7740 1 --2- 172.21.15.118:0/198132416 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5588ea93cd80 0x5588ea93d1a0 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 crypto rx=0x7f3000011740 tx=0x7f3000004730 comp rx=0 tx=0).stop 2024-03-24T00:15:29.286 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:29.280+0000 7f301bbd7740 1 -- 172.21.15.118:0/198132416 shutdown_connections 2024-03-24T00:15:29.286 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:29.280+0000 7f301bbd7740 1 --2- 172.21.15.118:0/198132416 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5588ea9428e0 0x5588ea944d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:15:29.286 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:29.280+0000 7f301bbd7740 1 --2- 172.21.15.118:0/198132416 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5588ea93d770 0x5588ea942270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:15:29.286 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:29.280+0000 7f301bbd7740 1 --2- 172.21.15.118:0/198132416 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5588ea93cd80 0x5588ea93d1a0 unknown :-1 s=CLOSED pgs=168 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:15:29.286 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:29.280+0000 7f301bbd7740 1 -- 172.21.15.118:0/198132416 >> 172.21.15.118:0/198132416 conn(0x5588ea9382f0 msgr2=0x5588ea9386f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:15:29.286 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:29.280+0000 7f301bbd7740 1 -- 172.21.15.118:0/198132416 shutdown_connections 2024-03-24T00:15:29.287 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:29.280+0000 7f301bbd7740 1 -- 172.21.15.118:0/198132416 wait complete. 2024-03-24T00:15:29.287 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:29.280+0000 7f3011e00640 1 --2- 172.21.15.118:0/1191067842 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7f2ff80a2880 0x7f2ff80a4d40 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x5588ea93db70 tx=0x7f300400b4a0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:15:29.287 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:29.280+0000 7f301bbd7740 1 -- start start 2024-03-24T00:15:29.287 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:29.281+0000 7f301bbd7740 1 Processor -- start 2024-03-24T00:15:29.287 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:29.281+0000 7f301bbd7740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5588ea93cd80 0x5588ea9472f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:15:29.287 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:29.281+0000 7f301bbd7740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5588ea93d770 0x5588ea947830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:15:29.287 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:29.281+0000 7f301bbd7740 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5588ea9428e0 0x5588ea94e870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:15:29.287 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:29.281+0000 7f301bbd7740 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x5588ea945890 con 0x5588ea9428e0 2024-03-24T00:15:29.287 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:29.281+0000 7f301bbd7740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x5588ea85ddd0 con 0x5588ea93d770 2024-03-24T00:15:29.287 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:29.281+0000 7f301bbd7740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x5588ea945a10 con 0x5588ea93cd80 2024-03-24T00:15:29.287 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:29.281+0000 7f301bbd7740 10 client.0 _ll_register_callbacks cb 0x5588ea5b5ac0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T00:15:29.287 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:29.281+0000 7f3011e00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5588ea93cd80 0x5588ea9472f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:15:29.287 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:29.281+0000 7f3011400640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5588ea93d770 0x5588ea947830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:15:29.287 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:29.281+0000 7f3011400640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5588ea93d770 0x5588ea947830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.118:54996/0 (socket says 172.21.15.118:54996) 2024-03-24T00:15:29.288 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:29.281+0000 7f3011e00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5588ea93cd80 0x5588ea9472f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.118:46978/0 (socket says 172.21.15.118:46978) 2024-03-24T00:15:29.288 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:29.281+0000 7f3011400640 1 -- 172.21.15.118:0/3639728222 learned_addr learned my addr 172.21.15.118:0/3639728222 (peer_addr_for_me v2:172.21.15.118:0/0) 2024-03-24T00:15:29.288 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:29.281+0000 7f3012800640 1 --2- 172.21.15.118:0/3639728222 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5588ea9428e0 0x5588ea94e870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:15:29.288 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:29.282+0000 7f3011400640 1 -- 172.21.15.118:0/3639728222 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5588ea93cd80 msgr2=0x5588ea9472f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:15:29.288 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:29.282+0000 7f3011400640 1 --2- 172.21.15.118:0/3639728222 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5588ea93cd80 0x5588ea9472f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:15:29.288 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:29.282+0000 7f3011400640 1 -- 172.21.15.118:0/3639728222 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5588ea9428e0 msgr2=0x5588ea94e870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:15:29.288 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:29.282+0000 7f3011400640 1 --2- 172.21.15.118:0/3639728222 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5588ea9428e0 0x5588ea94e870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:15:29.288 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:29.282+0000 7f3011400640 1 -- 172.21.15.118:0/3639728222 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5588ea5b62d0 con 0x5588ea93d770 2024-03-24T00:15:29.288 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:29.282+0000 7f3011400640 1 --2- 172.21.15.118:0/3639728222 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5588ea93d770 0x5588ea947830 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7f3000009c10 tx=0x7f3000005400 comp rx=0 tx=0).ready entity=mon.2 client_cookie=2fb3c4a6c7990181 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:15:29.288 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:29.282+0000 7f2ff3e00640 10 client.0 ms_handle_connect on v2:172.21.15.189:3300/0 2024-03-24T00:15:29.288 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:29.282+0000 7f2ff3e00640 1 -- 172.21.15.118:0/3639728222 <== mon.2 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3000004bc0 con 0x5588ea93d770 2024-03-24T00:15:29.288 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:29.282+0000 7f2ff3e00640 1 -- 172.21.15.118:0/3639728222 <== mon.2 v2:172.21.15.189:3300/0 2 ==== osd_map(90..90 src has 1..90) v4 ==== 8277+0+0 (secure 0 0 0) 0x7f3000004d60 con 0x5588ea93d770 2024-03-24T00:15:29.288 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:29.282+0000 7f301bbd7740 1 -- 172.21.15.118:0/3639728222 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f3000005500 con 0x5588ea93d770 2024-03-24T00:15:29.288 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:29.283+0000 7f2ff3e00640 1 -- 172.21.15.118:0/3639728222 <== mon.2 v2:172.21.15.189:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) v2 ==== 24+0+0 (secure 0 0 0) 0x7f300002f030 con 0x5588ea93d770 2024-03-24T00:15:29.288 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:29.283+0000 7f301bbd7740 10 client.25181 fetch_fsmap learned FSMap version 27 2024-03-24T00:15:29.289 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:29.283+0000 7f301bbd7740 1 -- 172.21.15.118:0/3639728222 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) v3 -- 0x5588ea6806b0 con 0x5588ea93d770 2024-03-24T00:15:29.289 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:29.283+0000 7f2ff3e00640 1 -- 172.21.15.118:0/3639728222 <== mon.2 v2:172.21.15.189:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f3000005500 con 0x5588ea93d770 2024-03-24T00:15:29.289 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:29.283+0000 7f2ff3e00640 1 -- 172.21.15.118:0/3639728222 <== mon.2 v2:172.21.15.189:3300/0 5 ==== fsmap(e 27) v1 ==== 7934+0+0 (secure 0 0 0) 0x7f300001d020 con 0x5588ea93d770 2024-03-24T00:15:29.289 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:29.283+0000 7f301bbd7740 10 client.25181 fetch_fsmap finished waiting for FSMap version 27 2024-03-24T00:15:29.289 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:29.283+0000 7f301bbd7740 10 client.25181 resolve_mds: resolved name 'f' to daemon mds.f 2024-03-24T00:15:29.289 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:29.283+0000 7f301bbd7740 20 client.25181 populate_metadata read hostname 'smithi118' 2024-03-24T00:15:29.289 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:29.283+0000 7f301bbd7740 1 --2- 172.21.15.118:0/3639728222 >> [v2:172.21.15.193:6834/2055310541,v1:172.21.15.193:6835/2055310541] conn(0x5588ea953770 0x5588ea957bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:15:29.289 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:29.283+0000 7f301bbd7740 4 client.25181 mds_command: new command op to 24499 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-24T00:15:29.289 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:29.283+0000 7f301bbd7740 1 -- 172.21.15.118:0/3639728222 --> [v2:172.21.15.193:6834/2055310541,v1:172.21.15.193:6835/2055310541] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x5588ea843520 con 0x5588ea953770 2024-03-24T00:15:29.289 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:29.284+0000 7f3011e00640 1 --2- 172.21.15.118:0/3639728222 >> [v2:172.21.15.193:6834/2055310541,v1:172.21.15.193:6835/2055310541] conn(0x5588ea953770 0x5588ea957bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:15:29.289 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:29.284+0000 7f2ff3e00640 10 client.25181 ms_handle_connect on v2:172.21.15.193:6834/2055310541 2024-03-24T00:15:29.289 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:29.284+0000 7f3011e00640 1 --2- 172.21.15.118:0/3639728222 >> [v2:172.21.15.193:6834/2055310541,v1:172.21.15.193:6835/2055310541] conn(0x5588ea953770 0x5588ea957bf0 crc :-1 s=READY pgs=20 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=979c60ccd8bf4951 server_cookie=d75389abfd9c3d14 in_seq=0 out_seq=0 2024-03-24T00:15:29.289 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:29.285+0000 7f2ff3e00640 1 -- 172.21.15.118:0/3639728222 <== mds.3 v2:172.21.15.193:6834/2055310541 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x5588ea843520 con 0x5588ea953770 2024-03-24T00:15:29.289 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:29.285+0000 7f2ff3e00640 10 client.25181 handle_command_reply: tid=0 2024-03-24T00:15:29.290 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:29.285+0000 7f301bbd7740 1 client.25181 shutdown 2024-03-24T00:15:29.290 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:29.285+0000 7f301bbd7740 1 -- 172.21.15.118:0/3639728222 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5588ea93d770 msgr2=0x5588ea947830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:15:29.290 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:29.285+0000 7f301bbd7740 1 --2- 172.21.15.118:0/3639728222 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5588ea93d770 0x5588ea947830 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7f3000009c10 tx=0x7f3000005400 comp rx=0 tx=0).stop 2024-03-24T00:15:29.290 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:29.285+0000 7f301bbd7740 1 -- 172.21.15.118:0/3639728222 shutdown_connections 2024-03-24T00:15:29.290 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:29.285+0000 7f301bbd7740 1 --2- 172.21.15.118:0/3639728222 >> [v2:172.21.15.193:6834/2055310541,v1:172.21.15.193:6835/2055310541] conn(0x5588ea953770 0x5588ea957bf0 crc :-1 s=READY pgs=20 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:15:29.290 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:29.285+0000 7f301bbd7740 1 --2- 172.21.15.118:0/3639728222 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5588ea9428e0 0x5588ea94e870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:15:29.290 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:29.285+0000 7f301bbd7740 1 --2- 172.21.15.118:0/3639728222 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5588ea93d770 0x5588ea947830 unknown :-1 s=CLOSED pgs=173 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:15:29.290 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:29.285+0000 7f301bbd7740 1 --2- 172.21.15.118:0/3639728222 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5588ea93cd80 0x5588ea9472f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:15:29.290 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:29.285+0000 7f2ff3e00640 0 client.25181 ms_handle_reset on v2:172.21.15.193:6834/2055310541 2024-03-24T00:15:29.290 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:29.285+0000 7f301bbd7740 1 -- 172.21.15.118:0/3639728222 >> 172.21.15.118:0/3639728222 conn(0x5588ea9382f0 msgr2=0x5588ea9374f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:15:29.290 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:29.285+0000 7f301bbd7740 1 -- 172.21.15.118:0/3639728222 shutdown_connections 2024-03-24T00:15:29.290 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:29.285+0000 7f301bbd7740 1 -- 172.21.15.118:0/3639728222 wait complete. 2024-03-24T00:15:29.290 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:29.285+0000 7f301bbd7740 20 client.25181 trim_cache size 0 max 16384 2024-03-24T00:15:29.305 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:29.304+0000 7f301bbd7740 1 Processor -- start 2024-03-24T00:15:29.305 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:29.304+0000 7f301bbd7740 1 -- start start 2024-03-24T00:15:29.306 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:29.304+0000 7f301bbd7740 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5588ea93cd80 0x5588ea936c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:15:29.306 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:29.304+0000 7f301bbd7740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5588ea93d770 0x5588ea94e830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:15:29.306 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:29.304+0000 7f301bbd7740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5588ea9428e0 0x5588ea9371a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:15:29.306 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:29.304+0000 7f301bbd7740 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x5588ea952f90 con 0x5588ea93cd80 2024-03-24T00:15:29.306 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:29.304+0000 7f301bbd7740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x5588ea9535f0 con 0x5588ea93d770 2024-03-24T00:15:29.306 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:29.304+0000 7f301bbd7740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x5588ea953c50 con 0x5588ea9428e0 2024-03-24T00:15:29.306 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:29.304+0000 7f3011e00640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5588ea93cd80 0x5588ea936c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:15:29.306 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:29.304+0000 7f3011e00640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5588ea93cd80 0x5588ea936c60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.118:3300/0 says I am v2:172.21.15.118:34590/0 (socket says 172.21.15.118:34590) 2024-03-24T00:15:29.306 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:29.304+0000 7f3012800640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5588ea9428e0 0x5588ea9371a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:15:29.306 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:29.304+0000 7f3011e00640 1 -- 172.21.15.118:0/2807416620 learned_addr learned my addr 172.21.15.118:0/2807416620 (peer_addr_for_me v2:172.21.15.118:0/0) 2024-03-24T00:15:29.306 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:29.304+0000 7f3011400640 1 --2- 172.21.15.118:0/2807416620 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5588ea93d770 0x5588ea94e830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:15:29.306 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:29.305+0000 7f3012800640 1 -- 172.21.15.118:0/2807416620 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5588ea93d770 msgr2=0x5588ea94e830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:15:29.306 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:29.305+0000 7f3012800640 1 --2- 172.21.15.118:0/2807416620 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5588ea93d770 0x5588ea94e830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:15:29.307 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:29.305+0000 7f3012800640 1 -- 172.21.15.118:0/2807416620 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5588ea93cd80 msgr2=0x5588ea936c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:15:29.307 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:29.305+0000 7f3012800640 1 --2- 172.21.15.118:0/2807416620 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5588ea93cd80 0x5588ea936c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:15:29.307 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:29.305+0000 7f3012800640 1 -- 172.21.15.118:0/2807416620 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5588ea92a170 con 0x5588ea9428e0 2024-03-24T00:15:29.307 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:29.305+0000 7f3011e00640 1 --2- 172.21.15.118:0/2807416620 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5588ea93cd80 0x5588ea936c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T00:15:29.307 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:29.305+0000 7f3012800640 1 --2- 172.21.15.118:0/2807416620 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5588ea9428e0 0x5588ea9371a0 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7f300c02c710 tx=0x7f300c035f00 comp rx=0 tx=0).ready entity=mon.1 client_cookie=1046332320760376 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:15:29.307 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:29.305+0000 7f2ffea00640 1 -- 172.21.15.118:0/2807416620 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f300c0098b0 con 0x5588ea9428e0 2024-03-24T00:15:29.307 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:29.305+0000 7f2ffea00640 1 -- 172.21.15.118:0/2807416620 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f300c0bd960 con 0x5588ea9428e0 2024-03-24T00:15:29.307 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:29.305+0000 7f2ffea00640 1 -- 172.21.15.118:0/2807416620 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f300c0bdbc0 con 0x5588ea9428e0 2024-03-24T00:15:29.311 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:29.306+0000 7f301bbd7740 1 -- 172.21.15.118:0/2807416620 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5588ea9428e0 msgr2=0x5588ea9371a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:15:29.311 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:29.306+0000 7f301bbd7740 1 --2- 172.21.15.118:0/2807416620 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5588ea9428e0 0x5588ea9371a0 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 crypto rx=0x7f300c02c710 tx=0x7f300c035f00 comp rx=0 tx=0).stop 2024-03-24T00:15:29.311 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:29.306+0000 7f301bbd7740 1 -- 172.21.15.118:0/2807416620 shutdown_connections 2024-03-24T00:15:29.311 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:29.306+0000 7f301bbd7740 1 --2- 172.21.15.118:0/2807416620 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5588ea9428e0 0x5588ea9371a0 unknown :-1 s=CLOSED pgs=169 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:15:29.311 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:29.306+0000 7f301bbd7740 1 --2- 172.21.15.118:0/2807416620 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5588ea93d770 0x5588ea94e830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:15:29.311 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:29.306+0000 7f301bbd7740 1 --2- 172.21.15.118:0/2807416620 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5588ea93cd80 0x5588ea936c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:15:29.311 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:29.306+0000 7f301bbd7740 1 -- 172.21.15.118:0/2807416620 >> 172.21.15.118:0/2807416620 conn(0x5588ea9382f0 msgr2=0x5588ea9386d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:15:29.312 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:29.306+0000 7f301bbd7740 1 -- 172.21.15.118:0/2807416620 shutdown_connections 2024-03-24T00:15:29.312 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:29.306+0000 7f301bbd7740 1 -- 172.21.15.118:0/2807416620 wait complete. 2024-03-24T00:15:29.312 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:29.306+0000 7f301bbd7740 1 -- start start 2024-03-24T00:15:29.312 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:29.306+0000 7f301bbd7740 1 Processor -- start 2024-03-24T00:15:29.312 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:29.307+0000 7f301bbd7740 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5588ea93cd80 0x5588ea949ed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:15:29.312 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:29.307+0000 7f301bbd7740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5588ea93d770 0x5588ea94a410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:15:29.312 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:29.307+0000 7f301bbd7740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5588ea9428e0 0x5588ea94d400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:15:29.312 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:29.307+0000 7f301bbd7740 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x5588ea954910 con 0x5588ea93cd80 2024-03-24T00:15:29.312 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:29.307+0000 7f301bbd7740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x5588ea9542b0 con 0x5588ea9428e0 2024-03-24T00:15:29.312 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:29.307+0000 7f301bbd7740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x5588ea954f70 con 0x5588ea93d770 2024-03-24T00:15:29.312 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:29.307+0000 7f301bbd7740 10 client.0 _ll_register_callbacks cb 0x7f3004004060 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T00:15:29.312 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:29.307+0000 7f3011400640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5588ea93d770 0x5588ea94a410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:15:29.312 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:29.307+0000 7f3011e00640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5588ea93cd80 0x5588ea949ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:15:29.312 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:29.307+0000 7f3011400640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5588ea93d770 0x5588ea94a410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.118:46996/0 (socket says 172.21.15.118:46996) 2024-03-24T00:15:29.313 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:29.307+0000 7f3011400640 1 -- 172.21.15.118:0/3009814112 learned_addr learned my addr 172.21.15.118:0/3009814112 (peer_addr_for_me v2:172.21.15.118:0/0) 2024-03-24T00:15:29.313 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:29.307+0000 7f3012800640 1 --2- 172.21.15.118:0/3009814112 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5588ea9428e0 0x5588ea94d400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:15:29.313 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:29.307+0000 7f3011400640 1 -- 172.21.15.118:0/3009814112 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5588ea9428e0 msgr2=0x5588ea94d400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:15:29.313 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:29.307+0000 7f3011400640 1 --2- 172.21.15.118:0/3009814112 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5588ea9428e0 0x5588ea94d400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:15:29.313 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:29.307+0000 7f3011400640 1 -- 172.21.15.118:0/3009814112 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5588ea93cd80 msgr2=0x5588ea949ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:15:29.313 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:29.307+0000 7f3011400640 1 --2- 172.21.15.118:0/3009814112 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5588ea93cd80 0x5588ea949ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:15:29.313 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:29.307+0000 7f3011400640 1 -- 172.21.15.118:0/3009814112 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5588ea94d9f0 con 0x5588ea93d770 2024-03-24T00:15:29.313 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:29.308+0000 7f3011400640 1 --2- 172.21.15.118:0/3009814112 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5588ea93d770 0x5588ea94a410 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7f3000009460 tx=0x7f3000005dc0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=b906b8f7ec9b2aab server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:15:29.313 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:29.308+0000 7f2ff3e00640 10 client.0 ms_handle_connect on v2:172.21.15.193:3300/0 2024-03-24T00:15:29.313 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:29.308+0000 7f2ff3e00640 1 -- 172.21.15.118:0/3009814112 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3000009c40 con 0x5588ea93d770 2024-03-24T00:15:29.313 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:29.308+0000 7f2ff3e00640 1 -- 172.21.15.118:0/3009814112 <== mon.1 v2:172.21.15.193:3300/0 2 ==== osd_map(90..90 src has 1..90) v4 ==== 8277+0+0 (secure 0 0 0) 0x7f3000030070 con 0x5588ea93d770 2024-03-24T00:15:29.313 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:29.308+0000 7f301bbd7740 1 -- 172.21.15.118:0/3009814112 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f300c0bd960 con 0x5588ea93d770 2024-03-24T00:15:29.313 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:29.308+0000 7f2ff3e00640 1 -- 172.21.15.118:0/3009814112 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) v2 ==== 24+0+0 (secure 0 0 0) 0x7f3000042030 con 0x5588ea93d770 2024-03-24T00:15:29.314 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:29.308+0000 7f301bbd7740 10 client.25198 fetch_fsmap learned FSMap version 27 2024-03-24T00:15:29.314 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:29.308+0000 7f301bbd7740 1 -- 172.21.15.118:0/3009814112 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) v3 -- 0x5588ea9aed10 con 0x5588ea93d770 2024-03-24T00:15:29.314 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:29.308+0000 7f2ff3e00640 1 -- 172.21.15.118:0/3009814112 <== mon.1 v2:172.21.15.193:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f300c0bd960 con 0x5588ea93d770 2024-03-24T00:15:29.314 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:29.309+0000 7f2ff3e00640 1 -- 172.21.15.118:0/3009814112 <== mon.1 v2:172.21.15.193:3300/0 5 ==== fsmap(e 27) v1 ==== 7934+0+0 (secure 0 0 0) 0x7f300001d050 con 0x5588ea93d770 2024-03-24T00:15:29.314 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:29.309+0000 7f301bbd7740 10 client.25198 fetch_fsmap finished waiting for FSMap version 27 2024-03-24T00:15:29.314 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:29.309+0000 7f301bbd7740 10 client.25198 resolve_mds: resolved name 'f' to daemon mds.f 2024-03-24T00:15:29.314 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:29.309+0000 7f301bbd7740 20 client.25198 populate_metadata read hostname 'smithi118' 2024-03-24T00:15:29.314 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:29.309+0000 7f301bbd7740 1 --2- 172.21.15.118:0/3009814112 >> [v2:172.21.15.193:6834/2055310541,v1:172.21.15.193:6835/2055310541] conn(0x5588ea9b3360 0x5588ea9aef40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:15:29.314 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:29.309+0000 7f301bbd7740 4 client.25198 mds_command: new command op to 24499 tid=0 multi_id=0 [{"prefix": "status"}] 2024-03-24T00:15:29.314 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:29.309+0000 7f301bbd7740 1 -- 172.21.15.118:0/3009814112 --> [v2:172.21.15.193:6834/2055310541,v1:172.21.15.193:6835/2055310541] -- command(tid 0: {"prefix": "status"}) v1 -- 0x5588ea8af1d0 con 0x5588ea9b3360 2024-03-24T00:15:29.314 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:29.309+0000 7f3011e00640 1 --2- 172.21.15.118:0/3009814112 >> [v2:172.21.15.193:6834/2055310541,v1:172.21.15.193:6835/2055310541] conn(0x5588ea9b3360 0x5588ea9aef40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:15:29.314 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:29.310+0000 7f2ff3e00640 10 client.25198 ms_handle_connect on v2:172.21.15.193:6834/2055310541 2024-03-24T00:15:29.314 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:29.310+0000 7f3011e00640 1 --2- 172.21.15.118:0/3009814112 >> [v2:172.21.15.193:6834/2055310541,v1:172.21.15.193:6835/2055310541] conn(0x5588ea9b3360 0x5588ea9aef40 crc :-1 s=READY pgs=21 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=12e4e87a1921f64c server_cookie=d68d467c4feb820e in_seq=0 out_seq=0 2024-03-24T00:15:29.315 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:29.310+0000 7f2ff3e00640 1 -- 172.21.15.118:0/3009814112 <== mds.3 v2:172.21.15.193:6834/2055310541 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+332 (crc 0 0 0) 0x5588ea8af1d0 con 0x5588ea9b3360 2024-03-24T00:15:29.315 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:29.310+0000 7f2ff3e00640 10 client.25198 handle_command_reply: tid=0 2024-03-24T00:15:29.315 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:29.310+0000 7f301bbd7740 1 client.25198 shutdown 2024-03-24T00:15:29.315 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:29.310+0000 7f301bbd7740 1 -- 172.21.15.118:0/3009814112 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5588ea93d770 msgr2=0x5588ea94a410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:15:29.315 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:29.310+0000 7f301bbd7740 1 --2- 172.21.15.118:0/3009814112 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5588ea93d770 0x5588ea94a410 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 crypto rx=0x7f3000009460 tx=0x7f3000005dc0 comp rx=0 tx=0).stop 2024-03-24T00:15:29.315 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:29.310+0000 7f301bbd7740 1 -- 172.21.15.118:0/3009814112 shutdown_connections 2024-03-24T00:15:29.315 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:29.310+0000 7f301bbd7740 1 --2- 172.21.15.118:0/3009814112 >> [v2:172.21.15.193:6834/2055310541,v1:172.21.15.193:6835/2055310541] conn(0x5588ea9b3360 0x5588ea9aef40 crc :-1 s=READY pgs=21 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:15:29.315 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:29.310+0000 7f301bbd7740 1 --2- 172.21.15.118:0/3009814112 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5588ea9428e0 0x5588ea94d400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:15:29.315 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:29.310+0000 7f301bbd7740 1 --2- 172.21.15.118:0/3009814112 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5588ea93d770 0x5588ea94a410 unknown :-1 s=CLOSED pgs=170 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:15:29.315 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:29.310+0000 7f301bbd7740 1 --2- 172.21.15.118:0/3009814112 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5588ea93cd80 0x5588ea949ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:15:29.315 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:29.310+0000 7f2ff3e00640 0 client.25198 ms_handle_reset on v2:172.21.15.193:6834/2055310541 2024-03-24T00:15:29.315 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:29.310+0000 7f301bbd7740 1 -- 172.21.15.118:0/3009814112 >> 172.21.15.118:0/3009814112 conn(0x5588ea9382f0 msgr2=0x5588ea93b550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:15:29.315 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:29.310+0000 7f301bbd7740 1 -- 172.21.15.118:0/3009814112 shutdown_connections 2024-03-24T00:15:29.315 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:29.310+0000 7f301bbd7740 1 -- 172.21.15.118:0/3009814112 wait complete. 2024-03-24T00:15:29.316 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:29.310+0000 7f301bbd7740 20 client.25198 trim_cache size 0 max 16384 2024-03-24T00:15:29.316 INFO:teuthology.orchestra.run.smithi118.stdout:{ 2024-03-24T00:15:29.316 INFO:teuthology.orchestra.run.smithi118.stdout: "cluster_fsid": "d79a5420-e971-11ee-95cd-87774f69a715", 2024-03-24T00:15:29.316 INFO:teuthology.orchestra.run.smithi118.stdout: "whoami": 3, 2024-03-24T00:15:29.316 INFO:teuthology.orchestra.run.smithi118.stdout: "id": 24499, 2024-03-24T00:15:29.316 INFO:teuthology.orchestra.run.smithi118.stdout: "want_state": "up:active", 2024-03-24T00:15:29.316 INFO:teuthology.orchestra.run.smithi118.stdout: "state": "up:active", 2024-03-24T00:15:29.316 INFO:teuthology.orchestra.run.smithi118.stdout: "fs_name": "cephfs", 2024-03-24T00:15:29.316 INFO:teuthology.orchestra.run.smithi118.stdout: "rank_uptime": 141.99363026500001, 2024-03-24T00:15:29.316 INFO:teuthology.orchestra.run.smithi118.stdout: "mdsmap_epoch": 27, 2024-03-24T00:15:29.316 INFO:teuthology.orchestra.run.smithi118.stdout: "osdmap_epoch": 90, 2024-03-24T00:15:29.317 INFO:teuthology.orchestra.run.smithi118.stdout: "osdmap_epoch_barrier": 86, 2024-03-24T00:15:29.317 INFO:teuthology.orchestra.run.smithi118.stdout: "uptime": 173.63497501200001 2024-03-24T00:15:29.317 INFO:teuthology.orchestra.run.smithi118.stdout:} 2024-03-24T00:15:29.317 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:29.311+0000 7f3013200640 1 -- 172.21.15.118:0/1191067842 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7f2ff80a2880 msgr2=0x7f2ff80a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:15:29.317 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:29.311+0000 7f3013200640 1 --2- 172.21.15.118:0/1191067842 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7f2ff80a2880 0x7f2ff80a4d40 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x5588ea93db70 tx=0x7f300400b4a0 comp rx=0 tx=0).stop 2024-03-24T00:15:29.317 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:29.311+0000 7f3013200640 1 -- 172.21.15.118:0/1191067842 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3014107260 msgr2=0x7f30141004f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:15:29.318 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:29.311+0000 7f3013200640 1 --2- 172.21.15.118:0/1191067842 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3014107260 0x7f30141004f0 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7f300c037040 tx=0x7f300c035d10 comp rx=0 tx=0).stop 2024-03-24T00:15:29.318 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:29.311+0000 7f3013200640 1 -- 172.21.15.118:0/1191067842 shutdown_connections 2024-03-24T00:15:29.318 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:29.311+0000 7f3013200640 1 --2- 172.21.15.118:0/1191067842 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7f2ff80a2880 0x7f2ff80a4d40 unknown :-1 s=CLOSED pgs=244 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:15:29.318 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:29.311+0000 7f3013200640 1 --2- 172.21.15.118:0/1191067842 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3014107260 0x7f30141004f0 unknown :-1 s=CLOSED pgs=172 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:15:29.318 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:29.311+0000 7f3013200640 1 --2- 172.21.15.118:0/1191067842 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f3014104930 0x7f30140fffb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:15:29.318 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:29.311+0000 7f3013200640 1 --2- 172.21.15.118:0/1191067842 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f3014102000 0x7f3014101970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:15:29.318 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:29.311+0000 7f3013200640 1 -- 172.21.15.118:0/1191067842 >> 172.21.15.118:0/1191067842 conn(0x7f30140fba10 msgr2=0x7f3014102940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:15:29.318 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:29.311+0000 7f3013200640 1 -- 172.21.15.118:0/1191067842 shutdown_connections 2024-03-24T00:15:29.318 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:29.311+0000 7f3013200640 1 -- 172.21.15.118:0/1191067842 wait complete. 2024-03-24T00:15:29.330 DEBUG:teuthology.orchestra.run.smithi118:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.c status 2024-03-24T00:15:29.525 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:29.524+0000 7f61dfe00640 1 -- 172.21.15.118:0/1697521902 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f61e0102000 msgr2=0x7f61e01043f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:15:29.526 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:29.524+0000 7f61dfe00640 1 --2- 172.21.15.118:0/1697521902 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f61e0102000 0x7f61e01043f0 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 crypto rx=0x7f61d0009900 tx=0x7f61d002bc30 comp rx=0 tx=0).stop 2024-03-24T00:15:29.526 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:29.525+0000 7f61dfe00640 1 -- 172.21.15.118:0/1697521902 shutdown_connections 2024-03-24T00:15:29.526 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:29.525+0000 7f61dfe00640 1 --2- 172.21.15.118:0/1697521902 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f61e0107260 0x7f61e01096b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:15:29.526 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:29.525+0000 7f61dfe00640 1 --2- 172.21.15.118:0/1697521902 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f61e0104930 0x7f61e0106d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:15:29.526 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:29.525+0000 7f61dfe00640 1 --2- 172.21.15.118:0/1697521902 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f61e0102000 0x7f61e01043f0 unknown :-1 s=CLOSED pgs=171 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:15:29.526 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:29.525+0000 7f61dfe00640 1 -- 172.21.15.118:0/1697521902 >> 172.21.15.118:0/1697521902 conn(0x7f61e00fba10 msgr2=0x7f61e00fde30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:15:29.526 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:29.525+0000 7f61dfe00640 1 -- 172.21.15.118:0/1697521902 shutdown_connections 2024-03-24T00:15:29.526 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:29.525+0000 7f61dfe00640 1 -- 172.21.15.118:0/1697521902 wait complete. 2024-03-24T00:15:29.528 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:29.526+0000 7f61dfe00640 1 Processor -- start 2024-03-24T00:15:29.528 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:29.526+0000 7f61dfe00640 1 -- start start 2024-03-24T00:15:29.528 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:29.526+0000 7f61dfe00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f61e0102000 0x7f61e0199af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:15:29.528 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:29.526+0000 7f61dfe00640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f61e0104930 0x7f61e019a030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:15:29.528 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:29.526+0000 7f61dfe00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f61e0107260 0x7f61e01a10a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:15:29.528 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:29.526+0000 7f61dfe00640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f61e010d260 con 0x7f61e0104930 2024-03-24T00:15:29.528 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:29.526+0000 7f61dfe00640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f61e010d0e0 con 0x7f61e0102000 2024-03-24T00:15:29.529 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:29.526+0000 7f61dfe00640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f61e010d3e0 con 0x7f61e0107260 2024-03-24T00:15:29.529 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:29.527+0000 7f61dea00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f61e0102000 0x7f61e0199af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:15:29.529 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:29.527+0000 7f61dea00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f61e0102000 0x7f61e0199af0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.118:55022/0 (socket says 172.21.15.118:55022) 2024-03-24T00:15:29.529 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:29.527+0000 7f61dea00640 1 -- 172.21.15.118:0/2920053321 learned_addr learned my addr 172.21.15.118:0/2920053321 (peer_addr_for_me v2:172.21.15.118:0/0) 2024-03-24T00:15:29.529 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:29.527+0000 7f61df400640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f61e0107260 0x7f61e01a10a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:15:29.529 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:29.527+0000 7f61d7e00640 1 --2- 172.21.15.118:0/2920053321 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f61e0104930 0x7f61e019a030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:15:29.529 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:29.527+0000 7f61dea00640 1 -- 172.21.15.118:0/2920053321 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f61e0107260 msgr2=0x7f61e01a10a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:15:29.529 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:29.527+0000 7f61dea00640 1 --2- 172.21.15.118:0/2920053321 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f61e0107260 0x7f61e01a10a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:15:29.529 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:29.527+0000 7f61dea00640 1 -- 172.21.15.118:0/2920053321 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f61e0104930 msgr2=0x7f61e019a030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:15:29.529 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:29.527+0000 7f61dea00640 1 --2- 172.21.15.118:0/2920053321 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f61e0104930 0x7f61e019a030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:15:29.529 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:29.527+0000 7f61dea00640 1 -- 172.21.15.118:0/2920053321 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f61e01a1710 con 0x7f61e0102000 2024-03-24T00:15:29.529 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:29.527+0000 7f61dea00640 1 --2- 172.21.15.118:0/2920053321 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f61e0102000 0x7f61e0199af0 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7f61d0005ab0 tx=0x7f61d002c6c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:15:29.529 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:29.527+0000 7f61d7400640 1 -- 172.21.15.118:0/2920053321 <== mon.2 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f61d0043020 con 0x7f61e0102000 2024-03-24T00:15:29.529 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:29.527+0000 7f61d7400640 1 -- 172.21.15.118:0/2920053321 <== mon.2 v2:172.21.15.189:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f61d00054e0 con 0x7f61e0102000 2024-03-24T00:15:29.529 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:29.528+0000 7f61d7400640 1 -- 172.21.15.118:0/2920053321 <== mon.2 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f61d0003ce0 con 0x7f61e0102000 2024-03-24T00:15:29.530 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:29.528+0000 7f61dfe00640 1 -- 172.21.15.118:0/2920053321 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f61e01a19a0 con 0x7f61e0102000 2024-03-24T00:15:29.530 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:29.528+0000 7f61dfe00640 1 -- 172.21.15.118:0/2920053321 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f61e01a1ea0 con 0x7f61e0102000 2024-03-24T00:15:29.531 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:29.531+0000 7f61d7400640 1 -- 172.21.15.118:0/2920053321 <== mon.2 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 18) v1 ==== 137291+0+0 (secure 0 0 0) 0x7f61d0046020 con 0x7f61e0102000 2024-03-24T00:15:29.531 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:29.531+0000 7f61e89ab740 1 Processor -- start 2024-03-24T00:15:29.531 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:29.531+0000 7f61e89ab740 1 -- start start 2024-03-24T00:15:29.533 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:29.531+0000 7f61e89ab740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5600654dad80 0x5600654db1a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:15:29.533 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:29.531+0000 7f61e89ab740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5600654db770 0x5600654e0270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:15:29.533 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:29.531+0000 7f61e89ab740 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5600654e08e0 0x5600654e2d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:15:29.533 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:29.531+0000 7f61e89ab740 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x56006543fc90 con 0x5600654e08e0 2024-03-24T00:15:29.533 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:29.531+0000 7f61e89ab740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x5600653e8980 con 0x5600654db770 2024-03-24T00:15:29.534 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:29.531+0000 7f61e89ab740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x56006543bd30 con 0x5600654dad80 2024-03-24T00:15:29.534 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:29.531+0000 7f61df400640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5600654dad80 0x5600654db1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:15:29.534 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:29.531+0000 7f61dea00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5600654db770 0x5600654e0270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:15:29.534 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:29.532+0000 7f61df400640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5600654dad80 0x5600654db1a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.118:47018/0 (socket says 172.21.15.118:47018) 2024-03-24T00:15:29.534 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:29.532+0000 7f61df400640 1 -- 172.21.15.118:0/3383289362 learned_addr learned my addr 172.21.15.118:0/3383289362 (peer_addr_for_me v2:172.21.15.118:0/0) 2024-03-24T00:15:29.534 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:29.532+0000 7f61d7e00640 1 --2- 172.21.15.118:0/3383289362 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5600654e08e0 0x5600654e2d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:15:29.534 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:29.532+0000 7f61df400640 1 -- 172.21.15.118:0/3383289362 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5600654db770 msgr2=0x5600654e0270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:15:29.534 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:29.532+0000 7f61df400640 1 --2- 172.21.15.118:0/3383289362 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5600654db770 0x5600654e0270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:15:29.534 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:29.532+0000 7f61df400640 1 -- 172.21.15.118:0/3383289362 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5600654e08e0 msgr2=0x5600654e2d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:15:29.534 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:29.532+0000 7f61df400640 1 --2- 172.21.15.118:0/3383289362 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5600654e08e0 0x5600654e2d70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:15:29.534 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:29.532+0000 7f61df400640 1 -- 172.21.15.118:0/3383289362 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x560065400510 con 0x5600654dad80 2024-03-24T00:15:29.534 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:29.532+0000 7f61d7e00640 1 --2- 172.21.15.118:0/3383289362 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5600654e08e0 0x5600654e2d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T00:15:29.535 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:29.532+0000 7f61df400640 1 --2- 172.21.15.118:0/3383289362 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5600654dad80 0x5600654db1a0 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7f61d8010370 tx=0x7f61d8010830 comp rx=0 tx=0).ready entity=mon.1 client_cookie=8b706900d43c37dd server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:15:29.535 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:29.532+0000 7f61d7400640 1 --2- 172.21.15.118:0/2920053321 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7f61c40a2800 0x7f61c40a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:15:29.535 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:29.532+0000 7f61d7400640 1 -- 172.21.15.118:0/2920053321 <== mon.2 v2:172.21.15.189:3300/0 5 ==== osd_map(90..90 src has 1..90) v4 ==== 8277+0+0 (secure 0 0 0) 0x7f61d00f0be0 con 0x7f61e0102000 2024-03-24T00:15:29.535 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:29.532+0000 7f61cfe00640 1 -- 172.21.15.118:0/3383289362 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f61d8016070 con 0x5600654dad80 2024-03-24T00:15:29.535 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:29.533+0000 7f61cfe00640 1 -- 172.21.15.118:0/3383289362 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f61d8004d20 con 0x5600654dad80 2024-03-24T00:15:29.535 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:29.533+0000 7f61cfe00640 1 -- 172.21.15.118:0/3383289362 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f61d8005480 con 0x5600654dad80 2024-03-24T00:15:29.535 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:29.533+0000 7f61dea00640 1 --2- 172.21.15.118:0/2920053321 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7f61c40a2800 0x7f61c40a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:15:29.538 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:29.533+0000 7f61dea00640 1 --2- 172.21.15.118:0/2920053321 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7f61c40a2800 0x7f61c40a4cc0 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7f61d00bd480 tx=0x7f61d00f4620 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:15:29.539 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:29.533+0000 7f61e89ab740 1 -- 172.21.15.118:0/3383289362 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5600654dad80 msgr2=0x5600654db1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:15:29.539 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:29.533+0000 7f61e89ab740 1 --2- 172.21.15.118:0/3383289362 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5600654dad80 0x5600654db1a0 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 crypto rx=0x7f61d8010370 tx=0x7f61d8010830 comp rx=0 tx=0).stop 2024-03-24T00:15:29.539 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:29.533+0000 7f61e89ab740 1 -- 172.21.15.118:0/3383289362 shutdown_connections 2024-03-24T00:15:29.539 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:29.533+0000 7f61e89ab740 1 --2- 172.21.15.118:0/3383289362 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5600654e08e0 0x5600654e2d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:15:29.539 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:29.533+0000 7f61e89ab740 1 --2- 172.21.15.118:0/3383289362 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5600654db770 0x5600654e0270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:15:29.539 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:29.533+0000 7f61e89ab740 1 --2- 172.21.15.118:0/3383289362 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5600654dad80 0x5600654db1a0 unknown :-1 s=CLOSED pgs=172 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:15:29.539 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:29.533+0000 7f61e89ab740 1 -- 172.21.15.118:0/3383289362 >> 172.21.15.118:0/3383289362 conn(0x5600654d62f0 msgr2=0x5600654d66f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:15:29.539 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:29.534+0000 7f61e89ab740 1 -- 172.21.15.118:0/3383289362 shutdown_connections 2024-03-24T00:15:29.539 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:29.534+0000 7f61e89ab740 1 -- 172.21.15.118:0/3383289362 wait complete. 2024-03-24T00:15:29.539 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:29.534+0000 7f61e89ab740 1 -- start start 2024-03-24T00:15:29.539 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:29.535+0000 7f61e89ab740 1 Processor -- start 2024-03-24T00:15:29.539 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:29.535+0000 7f61e89ab740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5600654dad80 0x5600654edb30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:15:29.539 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:29.535+0000 7f61e89ab740 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5600654db770 0x5600654ee070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:15:29.540 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:29.535+0000 7f61e89ab740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5600654e08e0 0x5600654f50b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:15:29.540 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:29.535+0000 7f61e89ab740 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x5600654ec060 con 0x5600654db770 2024-03-24T00:15:29.540 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:29.535+0000 7f61e89ab740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x5600653fbdd0 con 0x5600654dad80 2024-03-24T00:15:29.540 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:29.535+0000 7f61e89ab740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x5600654ec1e0 con 0x5600654e08e0 2024-03-24T00:15:29.540 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:29.535+0000 7f61df400640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5600654dad80 0x5600654edb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:15:29.540 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:29.535+0000 7f61df400640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5600654dad80 0x5600654edb30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.118:55032/0 (socket says 172.21.15.118:55032) 2024-03-24T00:15:29.540 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:29.535+0000 7f61df400640 1 -- 172.21.15.118:0/1926261018 learned_addr learned my addr 172.21.15.118:0/1926261018 (peer_addr_for_me v2:172.21.15.118:0/0) 2024-03-24T00:15:29.540 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:29.535+0000 7f61d7e00640 1 --2- 172.21.15.118:0/1926261018 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5600654db770 0x5600654ee070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:15:29.540 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:29.535+0000 7f61df400640 1 --2- 172.21.15.118:0/1926261018 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5600654e08e0 0x5600654f50b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:15:29.540 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:29.536+0000 7f61e89ab740 10 client.0 _ll_register_callbacks cb 0x560065153ac0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T00:15:29.540 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:29.536+0000 7f61df400640 1 -- 172.21.15.118:0/1926261018 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5600654e08e0 msgr2=0x5600654f50b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:15:29.540 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:29.536+0000 7f61df400640 1 --2- 172.21.15.118:0/1926261018 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5600654e08e0 0x5600654f50b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:15:29.540 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:29.536+0000 7f61df400640 1 -- 172.21.15.118:0/1926261018 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5600654db770 msgr2=0x5600654ee070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:15:29.540 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:29.536+0000 7f61df400640 1 --2- 172.21.15.118:0/1926261018 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5600654db770 0x5600654ee070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:15:29.541 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:29.536+0000 7f61df400640 1 -- 172.21.15.118:0/1926261018 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5600651542d0 con 0x5600654dad80 2024-03-24T00:15:29.541 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:29.536+0000 7f61df400640 1 --2- 172.21.15.118:0/1926261018 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5600654dad80 0x5600654edb30 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7f61d80056d0 tx=0x7f61d8009bb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b2b1411b5b90a8ef server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:15:29.541 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:29.536+0000 7f61ccc00640 10 client.0 ms_handle_connect on v2:172.21.15.189:3300/0 2024-03-24T00:15:29.541 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:29.536+0000 7f61ccc00640 1 -- 172.21.15.118:0/1926261018 <== mon.2 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f61d8016070 con 0x5600654dad80 2024-03-24T00:15:29.541 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:29.536+0000 7f61e89ab740 1 -- 172.21.15.118:0/1926261018 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f61d8004d20 con 0x5600654dad80 2024-03-24T00:15:29.541 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:29.536+0000 7f61ccc00640 1 -- 172.21.15.118:0/1926261018 <== mon.2 v2:172.21.15.189:3300/0 2 ==== osd_map(90..90 src has 1..90) v4 ==== 8277+0+0 (secure 0 0 0) 0x7f61d8005940 con 0x5600654dad80 2024-03-24T00:15:29.541 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:29.537+0000 7f61ccc00640 1 -- 172.21.15.118:0/1926261018 <== mon.2 v2:172.21.15.189:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) v2 ==== 24+0+0 (secure 0 0 0) 0x7f61d8003970 con 0x5600654dad80 2024-03-24T00:15:29.541 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:29.537+0000 7f61e89ab740 10 client.25199 fetch_fsmap learned FSMap version 27 2024-03-24T00:15:29.541 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:29.537+0000 7f61e89ab740 1 -- 172.21.15.118:0/1926261018 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) v3 -- 0x56006521e6b0 con 0x5600654dad80 2024-03-24T00:15:29.541 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:29.537+0000 7f61ccc00640 1 -- 172.21.15.118:0/1926261018 <== mon.2 v2:172.21.15.189:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f61d8004d20 con 0x5600654dad80 2024-03-24T00:15:29.541 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:29.537+0000 7f61ccc00640 1 -- 172.21.15.118:0/1926261018 <== mon.2 v2:172.21.15.189:3300/0 5 ==== fsmap(e 27) v1 ==== 7934+0+0 (secure 0 0 0) 0x7f61d8031070 con 0x5600654dad80 2024-03-24T00:15:29.541 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:29.537+0000 7f61e89ab740 10 client.25199 fetch_fsmap finished waiting for FSMap version 27 2024-03-24T00:15:29.541 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:29.537+0000 7f61e89ab740 10 client.25199 resolve_mds: resolved name 'c' to daemon mds.c 2024-03-24T00:15:29.541 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:29.537+0000 7f61e89ab740 20 client.25199 populate_metadata read hostname 'smithi118' 2024-03-24T00:15:29.541 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:29.537+0000 7f61e89ab740 1 --2- 172.21.15.118:0/1926261018 >> [v2:172.21.15.193:6836/2921520053,v1:172.21.15.193:6837/2921520053] conn(0x5600654f9e60 0x5600654fe2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:15:29.542 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:29.537+0000 7f61e89ab740 4 client.25199 mds_command: new command op to 24511 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-24T00:15:29.542 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:29.537+0000 7f61e89ab740 1 -- 172.21.15.118:0/1926261018 --> [v2:172.21.15.193:6836/2921520053,v1:172.21.15.193:6837/2921520053] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x5600653e1520 con 0x5600654f9e60 2024-03-24T00:15:29.542 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:29.537+0000 7f61d7e00640 1 --2- 172.21.15.118:0/1926261018 >> [v2:172.21.15.193:6836/2921520053,v1:172.21.15.193:6837/2921520053] conn(0x5600654f9e60 0x5600654fe2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:15:29.542 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:29.538+0000 7f61d7e00640 1 --2- 172.21.15.118:0/1926261018 >> [v2:172.21.15.193:6836/2921520053,v1:172.21.15.193:6837/2921520053] conn(0x5600654f9e60 0x5600654fe2e0 crc :-1 s=READY pgs=16 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=55108f539022867d server_cookie=a11206d348b6a466 in_seq=0 out_seq=0 2024-03-24T00:15:29.542 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:29.538+0000 7f61ccc00640 10 client.25199 ms_handle_connect on v2:172.21.15.193:6836/2921520053 2024-03-24T00:15:29.543 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:29.539+0000 7f61ccc00640 1 -- 172.21.15.118:0/1926261018 <== mds.2 v2:172.21.15.193:6836/2921520053 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x5600653e1520 con 0x5600654f9e60 2024-03-24T00:15:29.543 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:29.539+0000 7f61ccc00640 10 client.25199 handle_command_reply: tid=0 2024-03-24T00:15:29.544 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:29.539+0000 7f61e89ab740 1 client.25199 shutdown 2024-03-24T00:15:29.544 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:29.539+0000 7f61e89ab740 1 -- 172.21.15.118:0/1926261018 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5600654dad80 msgr2=0x5600654edb30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:15:29.544 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:29.539+0000 7f61e89ab740 1 --2- 172.21.15.118:0/1926261018 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5600654dad80 0x5600654edb30 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7f61d80056d0 tx=0x7f61d8009bb0 comp rx=0 tx=0).stop 2024-03-24T00:15:29.544 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:29.539+0000 7f61e89ab740 1 -- 172.21.15.118:0/1926261018 shutdown_connections 2024-03-24T00:15:29.544 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:29.539+0000 7f61e89ab740 1 --2- 172.21.15.118:0/1926261018 >> [v2:172.21.15.193:6836/2921520053,v1:172.21.15.193:6837/2921520053] conn(0x5600654f9e60 0x5600654fe2e0 crc :-1 s=READY pgs=16 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:15:29.544 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:29.539+0000 7f61e89ab740 1 --2- 172.21.15.118:0/1926261018 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5600654e08e0 0x5600654f50b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:15:29.544 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:29.539+0000 7f61e89ab740 1 --2- 172.21.15.118:0/1926261018 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5600654db770 0x5600654ee070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:15:29.544 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:29.539+0000 7f61ccc00640 0 client.25199 ms_handle_reset on v2:172.21.15.193:6836/2921520053 2024-03-24T00:15:29.544 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:29.539+0000 7f61e89ab740 1 --2- 172.21.15.118:0/1926261018 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5600654dad80 0x5600654edb30 unknown :-1 s=CLOSED pgs=175 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:15:29.544 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:29.539+0000 7f61e89ab740 1 -- 172.21.15.118:0/1926261018 >> 172.21.15.118:0/1926261018 conn(0x5600654d62f0 msgr2=0x5600654d54f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:15:29.544 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:29.539+0000 7f61e89ab740 1 -- 172.21.15.118:0/1926261018 shutdown_connections 2024-03-24T00:15:29.544 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:29.539+0000 7f61e89ab740 1 -- 172.21.15.118:0/1926261018 wait complete. 2024-03-24T00:15:29.544 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:29.539+0000 7f61e89ab740 20 client.25199 trim_cache size 0 max 16384 2024-03-24T00:15:29.568 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:29.568+0000 7f61e89ab740 1 Processor -- start 2024-03-24T00:15:29.568 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:29.568+0000 7f61e89ab740 1 -- start start 2024-03-24T00:15:29.570 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:29.568+0000 7f61e89ab740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5600654dad80 0x5600654ed150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:15:29.570 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:29.568+0000 7f61e89ab740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5600654db770 0x5600654edfd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:15:29.570 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:29.568+0000 7f61e89ab740 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5600654e08e0 0x5600654f5040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:15:29.570 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:29.568+0000 7f61e89ab740 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x5600654f9680 con 0x5600654e08e0 2024-03-24T00:15:29.570 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:29.568+0000 7f61e89ab740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x5600654f9ce0 con 0x5600654dad80 2024-03-24T00:15:29.570 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:29.568+0000 7f61e89ab740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x5600654fa340 con 0x5600654db770 2024-03-24T00:15:29.571 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:29.569+0000 7f61df400640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5600654dad80 0x5600654ed150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:15:29.571 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:29.569+0000 7f61df400640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5600654e08e0 0x5600654f5040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:15:29.571 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:29.569+0000 7f61df400640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5600654dad80 0x5600654ed150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.118:55034/0 (socket says 172.21.15.118:55034) 2024-03-24T00:15:29.571 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:29.569+0000 7f61df400640 1 -- 172.21.15.118:0/1345209653 learned_addr learned my addr 172.21.15.118:0/1345209653 (peer_addr_for_me v2:172.21.15.118:0/0) 2024-03-24T00:15:29.571 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:29.569+0000 7f61d7e00640 1 --2- 172.21.15.118:0/1345209653 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5600654db770 0x5600654edfd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:15:29.571 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:29.569+0000 7f61df400640 1 -- 172.21.15.118:0/1345209653 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5600654db770 msgr2=0x5600654edfd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:15:29.571 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:29.569+0000 7f61df400640 1 --2- 172.21.15.118:0/1345209653 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5600654db770 0x5600654edfd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:15:29.571 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:29.569+0000 7f61df400640 1 -- 172.21.15.118:0/1345209653 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5600654e08e0 msgr2=0x5600654f5040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:15:29.571 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:29.569+0000 7f61df400640 1 --2- 172.21.15.118:0/1345209653 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5600654e08e0 0x5600654f5040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:15:29.571 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:29.569+0000 7f61df400640 1 -- 172.21.15.118:0/1345209653 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5600654dff00 con 0x5600654dad80 2024-03-24T00:15:29.571 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:29.569+0000 7f61df400640 1 --2- 172.21.15.118:0/1345209653 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5600654dad80 0x5600654ed150 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7f61d80056a0 tx=0x7f61d80096f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=6ea1e9f9558a0859 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:15:29.571 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:29.570+0000 7f61cf400640 1 -- 172.21.15.118:0/1345209653 <== mon.2 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f61d8016040 con 0x5600654dad80 2024-03-24T00:15:29.571 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:29.570+0000 7f61cf400640 1 -- 172.21.15.118:0/1345209653 <== mon.2 v2:172.21.15.189:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f61d802e070 con 0x5600654dad80 2024-03-24T00:15:29.571 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:29.570+0000 7f61cf400640 1 -- 172.21.15.118:0/1345209653 <== mon.2 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f61d8028430 con 0x5600654dad80 2024-03-24T00:15:29.575 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:29.570+0000 7f61e89ab740 1 -- 172.21.15.118:0/1345209653 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5600654dad80 msgr2=0x5600654ed150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:15:29.575 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:29.570+0000 7f61e89ab740 1 --2- 172.21.15.118:0/1345209653 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5600654dad80 0x5600654ed150 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7f61d80056a0 tx=0x7f61d80096f0 comp rx=0 tx=0).stop 2024-03-24T00:15:29.575 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:29.570+0000 7f61e89ab740 1 -- 172.21.15.118:0/1345209653 shutdown_connections 2024-03-24T00:15:29.575 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:29.570+0000 7f61e89ab740 1 --2- 172.21.15.118:0/1345209653 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5600654e08e0 0x5600654f5040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:15:29.575 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:29.570+0000 7f61e89ab740 1 --2- 172.21.15.118:0/1345209653 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5600654db770 0x5600654edfd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:15:29.575 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:29.570+0000 7f61e89ab740 1 --2- 172.21.15.118:0/1345209653 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5600654dad80 0x5600654ed150 unknown :-1 s=CLOSED pgs=176 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:15:29.575 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:29.570+0000 7f61e89ab740 1 -- 172.21.15.118:0/1345209653 >> 172.21.15.118:0/1345209653 conn(0x5600654d62f0 msgr2=0x5600654ec680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:15:29.575 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:29.570+0000 7f61e89ab740 1 -- 172.21.15.118:0/1345209653 shutdown_connections 2024-03-24T00:15:29.575 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:29.571+0000 7f61e89ab740 1 -- 172.21.15.118:0/1345209653 wait complete. 2024-03-24T00:15:29.575 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:29.571+0000 7f61e89ab740 1 -- start start 2024-03-24T00:15:29.575 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:29.571+0000 7f61e89ab740 1 Processor -- start 2024-03-24T00:15:29.575 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:29.572+0000 7f61e89ab740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5600654dad80 0x5600654f0900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:15:29.575 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:29.572+0000 7f61e89ab740 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5600654db770 0x5600654f0e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:15:29.575 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:29.572+0000 7f61e89ab740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5600654e08e0 0x5600654f3e60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:15:29.576 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:29.572+0000 7f61e89ab740 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x5600654fb000 con 0x5600654db770 2024-03-24T00:15:29.576 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:29.572+0000 7f61e89ab740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x5600654fa9a0 con 0x5600654dad80 2024-03-24T00:15:29.576 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:29.572+0000 7f61e89ab740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x5600654fb660 con 0x5600654e08e0 2024-03-24T00:15:29.576 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:29.572+0000 7f61e89ab740 10 client.0 _ll_register_callbacks cb 0x7f61d00f44f0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T00:15:29.576 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:29.572+0000 7f61df400640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5600654dad80 0x5600654f0900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:15:29.576 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:29.572+0000 7f61df400640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5600654dad80 0x5600654f0900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.118:55048/0 (socket says 172.21.15.118:55048) 2024-03-24T00:15:29.576 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:29.572+0000 7f61df400640 1 -- 172.21.15.118:0/3691268046 learned_addr learned my addr 172.21.15.118:0/3691268046 (peer_addr_for_me v2:172.21.15.118:0/0) 2024-03-24T00:15:29.576 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:29.572+0000 7f61d7e00640 1 --2- 172.21.15.118:0/3691268046 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5600654db770 0x5600654f0e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:15:29.576 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:29.572+0000 7f61df400640 1 --2- 172.21.15.118:0/3691268046 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5600654e08e0 0x5600654f3e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:15:29.576 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:29.572+0000 7f61d7e00640 1 -- 172.21.15.118:0/3691268046 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5600654e08e0 msgr2=0x5600654f3e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:15:29.576 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:29.572+0000 7f61d7e00640 1 --2- 172.21.15.118:0/3691268046 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5600654e08e0 0x5600654f3e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:15:29.576 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:29.572+0000 7f61d7e00640 1 -- 172.21.15.118:0/3691268046 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5600654dad80 msgr2=0x5600654f0900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:15:29.576 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:29.572+0000 7f61d7e00640 1 --2- 172.21.15.118:0/3691268046 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5600654dad80 0x5600654f0900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:15:29.577 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:29.572+0000 7f61d7e00640 1 -- 172.21.15.118:0/3691268046 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5600654f4450 con 0x5600654db770 2024-03-24T00:15:29.577 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:29.573+0000 7f61d7e00640 1 --2- 172.21.15.118:0/3691268046 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5600654db770 0x5600654f0e40 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x5600654fb850 tx=0x7f61c8020b40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8e977302b7a3e407 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:15:29.577 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:29.573+0000 7f61ccc00640 10 client.0 ms_handle_connect on v2:172.21.15.118:3300/0 2024-03-24T00:15:29.577 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:29.573+0000 7f61ccc00640 1 -- 172.21.15.118:0/3691268046 <== mon.0 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f61c8020270 con 0x5600654db770 2024-03-24T00:15:29.577 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:29.573+0000 7f61ccc00640 1 -- 172.21.15.118:0/3691268046 <== mon.0 v2:172.21.15.118:3300/0 2 ==== osd_map(90..90 src has 1..90) v4 ==== 8277+0+0 (secure 0 0 0) 0x7f61c8010900 con 0x5600654db770 2024-03-24T00:15:29.577 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:29.573+0000 7f61e89ab740 1 -- 172.21.15.118:0/3691268046 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f61d802e070 con 0x5600654db770 2024-03-24T00:15:29.577 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:29.573+0000 7f61ccc00640 1 -- 172.21.15.118:0/3691268046 <== mon.0 v2:172.21.15.118:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) v2 ==== 24+0+0 (secure 0 0 0) 0x7f61c8010b00 con 0x5600654db770 2024-03-24T00:15:29.577 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:29.573+0000 7f61e89ab740 10 client.15114 fetch_fsmap learned FSMap version 27 2024-03-24T00:15:29.577 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:29.573+0000 7f61e89ab740 1 -- 172.21.15.118:0/3691268046 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) v3 -- 0x5600655555e0 con 0x5600654db770 2024-03-24T00:15:29.577 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:29.574+0000 7f61ccc00640 1 -- 172.21.15.118:0/3691268046 <== mon.0 v2:172.21.15.118:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f61d802e070 con 0x5600654db770 2024-03-24T00:15:29.577 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:29.574+0000 7f61ccc00640 1 -- 172.21.15.118:0/3691268046 <== mon.0 v2:172.21.15.118:3300/0 5 ==== fsmap(e 27) v1 ==== 7934+0+0 (secure 0 0 0) 0x7f61c801e390 con 0x5600654db770 2024-03-24T00:15:29.577 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:29.574+0000 7f61e89ab740 10 client.15114 fetch_fsmap finished waiting for FSMap version 27 2024-03-24T00:15:29.577 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:29.574+0000 7f61e89ab740 10 client.15114 resolve_mds: resolved name 'c' to daemon mds.c 2024-03-24T00:15:29.578 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:29.574+0000 7f61e89ab740 20 client.15114 populate_metadata read hostname 'smithi118' 2024-03-24T00:15:29.578 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:29.574+0000 7f61e89ab740 1 --2- 172.21.15.118:0/3691268046 >> [v2:172.21.15.193:6836/2921520053,v1:172.21.15.193:6837/2921520053] conn(0x560065559cb0 0x5600655558d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:15:29.578 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:29.574+0000 7f61e89ab740 4 client.15114 mds_command: new command op to 24511 tid=0 multi_id=0 [{"prefix": "status"}] 2024-03-24T00:15:29.578 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:29.574+0000 7f61e89ab740 1 -- 172.21.15.118:0/3691268046 --> [v2:172.21.15.193:6836/2921520053,v1:172.21.15.193:6837/2921520053] -- command(tid 0: {"prefix": "status"}) v1 -- 0x56006544d1d0 con 0x560065559cb0 2024-03-24T00:15:29.578 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:29.574+0000 7f61df400640 1 --2- 172.21.15.118:0/3691268046 >> [v2:172.21.15.193:6836/2921520053,v1:172.21.15.193:6837/2921520053] conn(0x560065559cb0 0x5600655558d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:15:29.579 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:29.575+0000 7f61ccc00640 10 client.15114 ms_handle_connect on v2:172.21.15.193:6836/2921520053 2024-03-24T00:15:29.579 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:29.574+0000 7f61df400640 1 --2- 172.21.15.118:0/3691268046 >> [v2:172.21.15.193:6836/2921520053,v1:172.21.15.193:6837/2921520053] conn(0x560065559cb0 0x5600655558d0 crc :-1 s=READY pgs=17 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=e0353db70dfc3404 server_cookie=af024d9d4464c8bc in_seq=0 out_seq=0 2024-03-24T00:15:29.579 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:29.575+0000 7f61ccc00640 1 -- 172.21.15.118:0/3691268046 <== mds.2 v2:172.21.15.193:6836/2921520053 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+332 (crc 0 0 0) 0x56006544d1d0 con 0x560065559cb0 2024-03-24T00:15:29.579 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:29.575+0000 7f61ccc00640 10 client.15114 handle_command_reply: tid=0 2024-03-24T00:15:29.579 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:29.575+0000 7f61e89ab740 1 client.15114 shutdown 2024-03-24T00:15:29.579 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:29.575+0000 7f61e89ab740 1 -- 172.21.15.118:0/3691268046 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5600654db770 msgr2=0x5600654f0e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:15:29.579 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:29.575+0000 7f61e89ab740 1 --2- 172.21.15.118:0/3691268046 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5600654db770 0x5600654f0e40 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x5600654fb850 tx=0x7f61c8020b40 comp rx=0 tx=0).stop 2024-03-24T00:15:29.579 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:29.575+0000 7f61e89ab740 1 -- 172.21.15.118:0/3691268046 shutdown_connections 2024-03-24T00:15:29.580 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:29.575+0000 7f61e89ab740 1 --2- 172.21.15.118:0/3691268046 >> [v2:172.21.15.193:6836/2921520053,v1:172.21.15.193:6837/2921520053] conn(0x560065559cb0 0x5600655558d0 crc :-1 s=READY pgs=17 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:15:29.580 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:29.575+0000 7f61e89ab740 1 --2- 172.21.15.118:0/3691268046 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5600654e08e0 0x5600654f3e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:15:29.580 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:29.575+0000 7f61e89ab740 1 --2- 172.21.15.118:0/3691268046 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5600654db770 0x5600654f0e40 unknown :-1 s=CLOSED pgs=174 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:15:29.580 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:29.575+0000 7f61e89ab740 1 --2- 172.21.15.118:0/3691268046 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5600654dad80 0x5600654f0900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:15:29.580 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:29.575+0000 7f61ccc00640 0 client.15114 ms_handle_reset on v2:172.21.15.193:6836/2921520053 2024-03-24T00:15:29.580 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:29.575+0000 7f61e89ab740 1 -- 172.21.15.118:0/3691268046 >> 172.21.15.118:0/3691268046 conn(0x5600654d62f0 msgr2=0x5600654d66d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:15:29.580 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:29.575+0000 7f61e89ab740 1 -- 172.21.15.118:0/3691268046 shutdown_connections 2024-03-24T00:15:29.580 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:29.575+0000 7f61e89ab740 1 -- 172.21.15.118:0/3691268046 wait complete. 2024-03-24T00:15:29.580 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:29.575+0000 7f61e89ab740 20 client.15114 trim_cache size 0 max 16384 2024-03-24T00:15:29.580 INFO:teuthology.orchestra.run.smithi118.stdout:{ 2024-03-24T00:15:29.580 INFO:teuthology.orchestra.run.smithi118.stdout: "cluster_fsid": "d79a5420-e971-11ee-95cd-87774f69a715", 2024-03-24T00:15:29.580 INFO:teuthology.orchestra.run.smithi118.stdout: "whoami": 2, 2024-03-24T00:15:29.581 INFO:teuthology.orchestra.run.smithi118.stdout: "id": 24511, 2024-03-24T00:15:29.581 INFO:teuthology.orchestra.run.smithi118.stdout: "want_state": "up:active", 2024-03-24T00:15:29.581 INFO:teuthology.orchestra.run.smithi118.stdout: "state": "up:active", 2024-03-24T00:15:29.581 INFO:teuthology.orchestra.run.smithi118.stdout: "fs_name": "cephfs", 2024-03-24T00:15:29.581 INFO:teuthology.orchestra.run.smithi118.stdout: "rank_uptime": 143.29233456899999, 2024-03-24T00:15:29.581 INFO:teuthology.orchestra.run.smithi118.stdout: "mdsmap_epoch": 27, 2024-03-24T00:15:29.581 INFO:teuthology.orchestra.run.smithi118.stdout: "osdmap_epoch": 90, 2024-03-24T00:15:29.581 INFO:teuthology.orchestra.run.smithi118.stdout: "osdmap_epoch_barrier": 86, 2024-03-24T00:15:29.581 INFO:teuthology.orchestra.run.smithi118.stdout: "uptime": 170.91956746100001 2024-03-24T00:15:29.581 INFO:teuthology.orchestra.run.smithi118.stdout:} 2024-03-24T00:15:29.582 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:29.576+0000 7f61dfe00640 1 -- 172.21.15.118:0/2920053321 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7f61c40a2800 msgr2=0x7f61c40a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:15:29.582 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:29.576+0000 7f61dfe00640 1 --2- 172.21.15.118:0/2920053321 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7f61c40a2800 0x7f61c40a4cc0 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7f61d00bd480 tx=0x7f61d00f4620 comp rx=0 tx=0).stop 2024-03-24T00:15:29.582 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:29.576+0000 7f61dfe00640 1 -- 172.21.15.118:0/2920053321 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f61e0102000 msgr2=0x7f61e0199af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:15:29.582 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:29.576+0000 7f61dfe00640 1 --2- 172.21.15.118:0/2920053321 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f61e0102000 0x7f61e0199af0 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7f61d0005ab0 tx=0x7f61d002c6c0 comp rx=0 tx=0).stop 2024-03-24T00:15:29.582 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:29.576+0000 7f61dfe00640 1 -- 172.21.15.118:0/2920053321 shutdown_connections 2024-03-24T00:15:29.582 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:29.576+0000 7f61dfe00640 1 --2- 172.21.15.118:0/2920053321 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7f61c40a2800 0x7f61c40a4cc0 unknown :-1 s=CLOSED pgs=245 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:15:29.582 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:29.576+0000 7f61dfe00640 1 --2- 172.21.15.118:0/2920053321 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f61e0107260 0x7f61e01a10a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:15:29.582 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:29.576+0000 7f61dfe00640 1 --2- 172.21.15.118:0/2920053321 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f61e0104930 0x7f61e019a030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:15:29.583 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:29.576+0000 7f61dfe00640 1 --2- 172.21.15.118:0/2920053321 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f61e0102000 0x7f61e0199af0 unknown :-1 s=CLOSED pgs=174 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:15:29.583 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:29.576+0000 7f61dfe00640 1 -- 172.21.15.118:0/2920053321 >> 172.21.15.118:0/2920053321 conn(0x7f61e00fba10 msgr2=0x7f61e0101720 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:15:29.583 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:29.576+0000 7f61dfe00640 1 -- 172.21.15.118:0/2920053321 shutdown_connections 2024-03-24T00:15:29.583 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:29.576+0000 7f61dfe00640 1 -- 172.21.15.118:0/2920053321 wait complete. 2024-03-24T00:15:29.593 DEBUG:teuthology.orchestra.run.smithi118:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.g status 2024-03-24T00:15:29.805 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:29.804+0000 7fcde6600640 1 -- 172.21.15.118:0/1559894778 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fcde01038e0 msgr2=0x7fcde0103d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:15:29.805 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:29.804+0000 7fcde6600640 1 --2- 172.21.15.118:0/1559894778 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fcde01038e0 0x7fcde0103d00 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7fcdd0009900 tx=0x7fcdd002bc30 comp rx=0 tx=0).stop 2024-03-24T00:15:29.805 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:29.804+0000 7fcde6600640 1 -- 172.21.15.118:0/1559894778 shutdown_connections 2024-03-24T00:15:29.805 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:29.804+0000 7fcde6600640 1 --2- 172.21.15.118:0/1559894778 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fcde0105490 0x7fcde010bd20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:15:29.805 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:29.804+0000 7fcde6600640 1 --2- 172.21.15.118:0/1559894778 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fcde0104ad0 0x7fcde0104f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:15:29.805 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:29.804+0000 7fcde6600640 1 --2- 172.21.15.118:0/1559894778 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fcde01038e0 0x7fcde0103d00 unknown :-1 s=CLOSED pgs=177 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:15:29.805 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:29.804+0000 7fcde6600640 1 -- 172.21.15.118:0/1559894778 >> 172.21.15.118:0/1559894778 conn(0x7fcde00fee50 msgr2=0x7fcde0101290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:15:29.805 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:29.804+0000 7fcde6600640 1 -- 172.21.15.118:0/1559894778 shutdown_connections 2024-03-24T00:15:29.805 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:29.804+0000 7fcde6600640 1 -- 172.21.15.118:0/1559894778 wait complete. 2024-03-24T00:15:29.808 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:29.805+0000 7fcde6600640 1 Processor -- start 2024-03-24T00:15:29.808 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:29.805+0000 7fcde6600640 1 -- start start 2024-03-24T00:15:29.808 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:29.805+0000 7fcde6600640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fcde01038e0 0x7fcde0199c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:15:29.808 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:29.805+0000 7fcde6600640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fcde0104ad0 0x7fcde019a1b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:15:29.808 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:29.805+0000 7fcde6600640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fcde0105490 0x7fcde01a11e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:15:29.809 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:29.805+0000 7fcde6600640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcde010f580 con 0x7fcde0104ad0 2024-03-24T00:15:29.809 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:29.805+0000 7fcde6600640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcde010f400 con 0x7fcde01038e0 2024-03-24T00:15:29.809 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:29.805+0000 7fcde6600640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcde010f700 con 0x7fcde0105490 2024-03-24T00:15:29.809 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:29.806+0000 7fcddf400640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fcde01038e0 0x7fcde0199c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:15:29.809 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:29.806+0000 7fcddf400640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fcde01038e0 0x7fcde0199c70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.118:55056/0 (socket says 172.21.15.118:55056) 2024-03-24T00:15:29.809 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:29.806+0000 7fcddf400640 1 -- 172.21.15.118:0/2297781045 learned_addr learned my addr 172.21.15.118:0/2297781045 (peer_addr_for_me v2:172.21.15.118:0/0) 2024-03-24T00:15:29.810 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:29.806+0000 7fcddea00640 1 --2- 172.21.15.118:0/2297781045 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fcde0104ad0 0x7fcde019a1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:15:29.810 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:29.806+0000 7fcddfe00640 1 --2- 172.21.15.118:0/2297781045 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fcde0105490 0x7fcde01a11e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:15:29.810 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:29.806+0000 7fcddf400640 1 -- 172.21.15.118:0/2297781045 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fcde0105490 msgr2=0x7fcde01a11e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:15:29.810 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:29.806+0000 7fcddf400640 1 --2- 172.21.15.118:0/2297781045 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fcde0105490 0x7fcde01a11e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:15:29.810 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:29.806+0000 7fcddf400640 1 -- 172.21.15.118:0/2297781045 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fcde0104ad0 msgr2=0x7fcde019a1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:15:29.810 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:29.806+0000 7fcddf400640 1 --2- 172.21.15.118:0/2297781045 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fcde0104ad0 0x7fcde019a1b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:15:29.810 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:29.806+0000 7fcddf400640 1 -- 172.21.15.118:0/2297781045 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcde01a1880 con 0x7fcde01038e0 2024-03-24T00:15:29.810 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:29.806+0000 7fcddf400640 1 --2- 172.21.15.118:0/2297781045 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fcde01038e0 0x7fcde0199c70 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7fcdd0000c00 tx=0x7fcdd002c650 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:15:29.810 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:29.807+0000 7fcdd7e00640 1 -- 172.21.15.118:0/2297781045 <== mon.2 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcdd0043020 con 0x7fcde01038e0 2024-03-24T00:15:29.811 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:29.807+0000 7fcdd7e00640 1 -- 172.21.15.118:0/2297781045 <== mon.2 v2:172.21.15.189:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fcdd00054e0 con 0x7fcde01038e0 2024-03-24T00:15:29.811 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:29.807+0000 7fcde6600640 1 -- 172.21.15.118:0/2297781045 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcde01a1b10 con 0x7fcde01038e0 2024-03-24T00:15:29.811 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:29.807+0000 7fcdd7e00640 1 -- 172.21.15.118:0/2297781045 <== mon.2 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcdd0003c40 con 0x7fcde01038e0 2024-03-24T00:15:29.811 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:29.807+0000 7fcde6600640 1 -- 172.21.15.118:0/2297781045 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcde01a2060 con 0x7fcde01038e0 2024-03-24T00:15:29.813 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:29.810+0000 7fcdd7e00640 1 -- 172.21.15.118:0/2297781045 <== mon.2 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 18) v1 ==== 137291+0+0 (secure 0 0 0) 0x7fcdd0046020 con 0x7fcde01038e0 2024-03-24T00:15:29.813 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:29.810+0000 7fcde9266740 1 Processor -- start 2024-03-24T00:15:29.813 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:29.810+0000 7fcde9266740 1 -- start start 2024-03-24T00:15:29.813 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:29.811+0000 7fcde9266740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55defc254ba0 0x55defc254fc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:15:29.813 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:29.811+0000 7fcde9266740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55defc255590 0x55defc25a090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:15:29.813 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:29.811+0000 7fcde9266740 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55defc25a700 0x55defc25cb90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:15:29.814 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:29.811+0000 7fcde9266740 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x55defc1b9c90 con 0x55defc25a700 2024-03-24T00:15:29.814 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:29.811+0000 7fcde9266740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x55defc162980 con 0x55defc255590 2024-03-24T00:15:29.814 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:29.811+0000 7fcde9266740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x55defc1b5d30 con 0x55defc254ba0 2024-03-24T00:15:29.814 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:29.811+0000 7fcdd7e00640 1 --2- 172.21.15.118:0/2297781045 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7fcdc40a2800 0x7fcdc40a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:15:29.814 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:29.811+0000 7fcddfe00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55defc254ba0 0x55defc254fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:15:29.814 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:29.811+0000 7fcddfe00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55defc254ba0 0x55defc254fc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.118:47074/0 (socket says 172.21.15.118:47074) 2024-03-24T00:15:29.814 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:29.811+0000 7fcddfe00640 1 -- 172.21.15.118:0/434328087 learned_addr learned my addr 172.21.15.118:0/434328087 (peer_addr_for_me v2:172.21.15.118:0/0) 2024-03-24T00:15:29.814 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:29.811+0000 7fcddf400640 1 --2- 172.21.15.118:0/434328087 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55defc255590 0x55defc25a090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:15:29.814 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:29.811+0000 7fcddea00640 1 --2- 172.21.15.118:0/434328087 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55defc25a700 0x55defc25cb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:15:29.814 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:29.811+0000 7fcdd7e00640 1 -- 172.21.15.118:0/2297781045 <== mon.2 v2:172.21.15.189:3300/0 5 ==== osd_map(90..90 src has 1..90) v4 ==== 8277+0+0 (secure 0 0 0) 0x7fcdd00f0b50 con 0x7fcde01038e0 2024-03-24T00:15:29.814 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:29.812+0000 7fcddfe00640 1 --2- 172.21.15.118:0/2297781045 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7fcdc40a2800 0x7fcdc40a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:15:29.814 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:29.812+0000 7fcddf400640 1 -- 172.21.15.118:0/434328087 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55defc254ba0 msgr2=0x55defc254fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:15:29.814 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:29.812+0000 7fcddf400640 1 --2- 172.21.15.118:0/434328087 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55defc254ba0 0x55defc254fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:15:29.815 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:29.812+0000 7fcddf400640 1 -- 172.21.15.118:0/434328087 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55defc25a700 msgr2=0x55defc25cb90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:15:29.815 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:29.812+0000 7fcddf400640 1 --2- 172.21.15.118:0/434328087 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55defc25a700 0x55defc25cb90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:15:29.815 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:29.812+0000 7fcddf400640 1 -- 172.21.15.118:0/434328087 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55defc17a510 con 0x55defc255590 2024-03-24T00:15:29.815 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:29.812+0000 7fcddea00640 1 --2- 172.21.15.118:0/434328087 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55defc25a700 0x55defc25cb90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T00:15:29.815 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:29.812+0000 7fcddf400640 1 --2- 172.21.15.118:0/434328087 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55defc255590 0x55defc25a090 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7fcdd00ee590 tx=0x7fcdd0005e50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a0307c421d9c6295 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:15:29.815 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:29.812+0000 7fcdd4c00640 1 -- 172.21.15.118:0/434328087 <== mon.2 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcdd00bcdc0 con 0x55defc255590 2024-03-24T00:15:29.815 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:29.812+0000 7fcddfe00640 1 --2- 172.21.15.118:0/2297781045 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7fcdc40a2800 0x7fcdc40a4cc0 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x55defc255df0 tx=0x7fcdd8004ab0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:15:29.815 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:29.812+0000 7fcdd4c00640 1 -- 172.21.15.118:0/434328087 <== mon.2 v2:172.21.15.189:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fcdd00035f0 con 0x55defc255590 2024-03-24T00:15:29.815 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:29.812+0000 7fcdd4c00640 1 -- 172.21.15.118:0/434328087 <== mon.2 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcdd00bb5a0 con 0x55defc255590 2024-03-24T00:15:29.819 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:29.813+0000 7fcde9266740 1 -- 172.21.15.118:0/434328087 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55defc255590 msgr2=0x55defc25a090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:15:29.820 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:29.813+0000 7fcde9266740 1 --2- 172.21.15.118:0/434328087 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55defc255590 0x55defc25a090 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7fcdd00ee590 tx=0x7fcdd0005e50 comp rx=0 tx=0).stop 2024-03-24T00:15:29.820 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:29.813+0000 7fcde9266740 1 -- 172.21.15.118:0/434328087 shutdown_connections 2024-03-24T00:15:29.820 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:29.813+0000 7fcde9266740 1 --2- 172.21.15.118:0/434328087 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55defc25a700 0x55defc25cb90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:15:29.820 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:29.813+0000 7fcde9266740 1 --2- 172.21.15.118:0/434328087 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55defc255590 0x55defc25a090 unknown :-1 s=CLOSED pgs=179 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:15:29.820 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:29.813+0000 7fcde9266740 1 --2- 172.21.15.118:0/434328087 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55defc254ba0 0x55defc254fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:15:29.820 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:29.813+0000 7fcde9266740 1 -- 172.21.15.118:0/434328087 >> 172.21.15.118:0/434328087 conn(0x55defc250110 msgr2=0x55defc250510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:15:29.820 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:29.813+0000 7fcde9266740 1 -- 172.21.15.118:0/434328087 shutdown_connections 2024-03-24T00:15:29.820 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:29.814+0000 7fcde9266740 1 -- 172.21.15.118:0/434328087 wait complete. 2024-03-24T00:15:29.820 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:29.814+0000 7fcde9266740 1 -- start start 2024-03-24T00:15:29.820 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:29.814+0000 7fcde9266740 1 Processor -- start 2024-03-24T00:15:29.820 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:29.815+0000 7fcde9266740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55defc254ba0 0x55defc25ee50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:15:29.820 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:29.815+0000 7fcde9266740 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55defc255590 0x55defc25f390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:15:29.820 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:29.815+0000 7fcde9266740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55defc25a700 0x55defc2663d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:15:29.820 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:29.815+0000 7fcde9266740 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x55defc25d470 con 0x55defc255590 2024-03-24T00:15:29.821 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:29.815+0000 7fcde9266740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x55defc175dd0 con 0x55defc254ba0 2024-03-24T00:15:29.821 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:29.815+0000 7fcde9266740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x55defc25d5f0 con 0x55defc25a700 2024-03-24T00:15:29.821 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:29.815+0000 7fcde9266740 10 client.0 _ll_register_callbacks cb 0x55defbecdac0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T00:15:29.821 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:29.815+0000 7fcddf400640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55defc254ba0 0x55defc25ee50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:15:29.821 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:29.815+0000 7fcddea00640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55defc255590 0x55defc25f390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:15:29.821 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:29.815+0000 7fcddf400640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55defc254ba0 0x55defc25ee50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.118:55060/0 (socket says 172.21.15.118:55060) 2024-03-24T00:15:29.821 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:29.815+0000 7fcddfe00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55defc25a700 0x55defc2663d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:15:29.821 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:29.815+0000 7fcddea00640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55defc255590 0x55defc25f390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.118:3300/0 says I am v2:172.21.15.118:34692/0 (socket says 172.21.15.118:34692) 2024-03-24T00:15:29.821 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:29.815+0000 7fcddea00640 1 -- 172.21.15.118:0/3301783838 learned_addr learned my addr 172.21.15.118:0/3301783838 (peer_addr_for_me v2:172.21.15.118:0/0) 2024-03-24T00:15:29.821 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:29.816+0000 7fcddf400640 1 -- 172.21.15.118:0/3301783838 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55defc25a700 msgr2=0x55defc2663d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:15:29.821 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:29.816+0000 7fcddf400640 1 --2- 172.21.15.118:0/3301783838 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55defc25a700 0x55defc2663d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:15:29.821 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:29.816+0000 7fcddf400640 1 -- 172.21.15.118:0/3301783838 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55defc255590 msgr2=0x55defc25f390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:15:29.821 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:29.816+0000 7fcddf400640 1 --2- 172.21.15.118:0/3301783838 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55defc255590 0x55defc25f390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:15:29.821 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:29.816+0000 7fcddf400640 1 -- 172.21.15.118:0/3301783838 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55defbece2d0 con 0x55defc254ba0 2024-03-24T00:15:29.821 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:29.816+0000 7fcddea00640 1 --2- 172.21.15.118:0/3301783838 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55defc255590 0x55defc25f390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T00:15:29.822 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:29.816+0000 7fcddf400640 1 --2- 172.21.15.118:0/3301783838 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55defc254ba0 0x55defc25ee50 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x55defc25d300 tx=0x7fcdd00c5d00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ffd6fa2bf62cad48 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:15:29.822 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:29.816+0000 7fcdc9600640 10 client.0 ms_handle_connect on v2:172.21.15.189:3300/0 2024-03-24T00:15:29.822 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:29.816+0000 7fcdc9600640 1 -- 172.21.15.118:0/3301783838 <== mon.2 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcdd00bb5a0 con 0x55defc254ba0 2024-03-24T00:15:29.822 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:29.816+0000 7fcdc9600640 1 -- 172.21.15.118:0/3301783838 <== mon.2 v2:172.21.15.189:3300/0 2 ==== osd_map(90..90 src has 1..90) v4 ==== 8277+0+0 (secure 0 0 0) 0x7fcdd00c5980 con 0x55defc254ba0 2024-03-24T00:15:29.822 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:29.816+0000 7fcde9266740 1 -- 172.21.15.118:0/3301783838 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55defc15b520 con 0x55defc254ba0 2024-03-24T00:15:29.822 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:29.816+0000 7fcdc9600640 1 -- 172.21.15.118:0/3301783838 <== mon.2 v2:172.21.15.189:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) v2 ==== 24+0+0 (secure 0 0 0) 0x7fcdd00bcdc0 con 0x55defc254ba0 2024-03-24T00:15:29.822 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:29.816+0000 7fcde9266740 10 client.25217 fetch_fsmap learned FSMap version 27 2024-03-24T00:15:29.822 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:29.816+0000 7fcde9266740 1 -- 172.21.15.118:0/3301783838 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) v3 -- 0x55defbf986b0 con 0x55defc254ba0 2024-03-24T00:15:29.822 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:29.817+0000 7fcdc9600640 1 -- 172.21.15.118:0/3301783838 <== mon.2 v2:172.21.15.189:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x55defc15b520 con 0x55defc254ba0 2024-03-24T00:15:29.822 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:29.817+0000 7fcdc9600640 1 -- 172.21.15.118:0/3301783838 <== mon.2 v2:172.21.15.189:3300/0 5 ==== fsmap(e 27) v1 ==== 7934+0+0 (secure 0 0 0) 0x7fcdd00bc9d0 con 0x55defc254ba0 2024-03-24T00:15:29.822 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:29.817+0000 7fcde9266740 10 client.25217 fetch_fsmap finished waiting for FSMap version 27 2024-03-24T00:15:29.822 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:29.817+0000 7fcde9266740 10 client.25217 resolve_mds: resolved name 'g' to daemon mds.g 2024-03-24T00:15:29.822 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:29.817+0000 7fcde9266740 20 client.25217 populate_metadata read hostname 'smithi118' 2024-03-24T00:15:29.823 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:29.817+0000 7fcde9266740 1 --2- 172.21.15.118:0/3301783838 >> [v2:172.21.15.118:6838/3285788503,v1:172.21.15.118:6839/3285788503] conn(0x55defc26b330 0x55defc26a530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:15:29.823 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:29.817+0000 7fcde9266740 4 client.25217 mds_command: new command op to 24550 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-24T00:15:29.823 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:29.817+0000 7fcde9266740 1 -- 172.21.15.118:0/3301783838 --> [v2:172.21.15.118:6838/3285788503,v1:172.21.15.118:6839/3285788503] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55defc1c71d0 con 0x55defc26b330 2024-03-24T00:15:29.823 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:29.817+0000 7fcddea00640 1 --2- 172.21.15.118:0/3301783838 >> [v2:172.21.15.118:6838/3285788503,v1:172.21.15.118:6839/3285788503] conn(0x55defc26b330 0x55defc26a530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:15:29.823 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:29.818+0000 7fcddea00640 1 --2- 172.21.15.118:0/3301783838 >> [v2:172.21.15.118:6838/3285788503,v1:172.21.15.118:6839/3285788503] conn(0x55defc26b330 0x55defc26a530 crc :-1 s=READY pgs=18 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=6b7fa3ed63c9a1aa server_cookie=9583a230140dd6c8 in_seq=0 out_seq=0 2024-03-24T00:15:29.823 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:29.818+0000 7fcdc9600640 10 client.25217 ms_handle_connect on v2:172.21.15.118:6838/3285788503 2024-03-24T00:15:29.823 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:29.818+0000 7fcdc9600640 1 -- 172.21.15.118:0/3301783838 <== mds.1 v2:172.21.15.118:6838/3285788503 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x55defc1c71d0 con 0x55defc26b330 2024-03-24T00:15:29.823 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:29.818+0000 7fcdc9600640 10 client.25217 handle_command_reply: tid=0 2024-03-24T00:15:29.823 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:29.819+0000 7fcde9266740 1 client.25217 shutdown 2024-03-24T00:15:29.823 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:29.819+0000 7fcde9266740 1 -- 172.21.15.118:0/3301783838 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55defc254ba0 msgr2=0x55defc25ee50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:15:29.823 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:29.819+0000 7fcde9266740 1 --2- 172.21.15.118:0/3301783838 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55defc254ba0 0x55defc25ee50 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x55defc25d300 tx=0x7fcdd00c5d00 comp rx=0 tx=0).stop 2024-03-24T00:15:29.823 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:29.819+0000 7fcde9266740 1 -- 172.21.15.118:0/3301783838 shutdown_connections 2024-03-24T00:15:29.823 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:29.819+0000 7fcde9266740 1 --2- 172.21.15.118:0/3301783838 >> [v2:172.21.15.118:6838/3285788503,v1:172.21.15.118:6839/3285788503] conn(0x55defc26b330 0x55defc26a530 crc :-1 s=READY pgs=18 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:15:29.823 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:29.819+0000 7fcde9266740 1 --2- 172.21.15.118:0/3301783838 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55defc25a700 0x55defc2663d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:15:29.823 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:29.819+0000 7fcde9266740 1 --2- 172.21.15.118:0/3301783838 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55defc255590 0x55defc25f390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:15:29.824 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:29.819+0000 7fcde9266740 1 --2- 172.21.15.118:0/3301783838 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55defc254ba0 0x55defc25ee50 unknown :-1 s=CLOSED pgs=180 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:15:29.824 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:29.819+0000 7fcdc9600640 0 client.25217 ms_handle_reset on v2:172.21.15.118:6838/3285788503 2024-03-24T00:15:29.824 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:29.819+0000 7fcde9266740 1 -- 172.21.15.118:0/3301783838 >> 172.21.15.118:0/3301783838 conn(0x55defc250110 msgr2=0x55defc24f310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:15:29.824 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:29.819+0000 7fcde9266740 1 -- 172.21.15.118:0/3301783838 shutdown_connections 2024-03-24T00:15:29.824 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:29.819+0000 7fcde9266740 1 -- 172.21.15.118:0/3301783838 wait complete. 2024-03-24T00:15:29.824 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:29.819+0000 7fcde9266740 20 client.25217 trim_cache size 0 max 16384 2024-03-24T00:15:29.838 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:29.837+0000 7fcde9266740 1 Processor -- start 2024-03-24T00:15:29.838 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:29.837+0000 7fcde9266740 1 -- start start 2024-03-24T00:15:29.838 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:29.838+0000 7fcde9266740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55defc254ba0 0x55defc24ae00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:15:29.838 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:29.838+0000 7fcde9266740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55defc255590 0x55defc266390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:15:29.838 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:29.838+0000 7fcde9266740 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55defc25a700 0x55defc2504f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:15:29.838 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:29.838+0000 7fcde9266740 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x55defc26ab50 con 0x55defc25a700 2024-03-24T00:15:29.839 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:29.838+0000 7fcde9266740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x55defc26b1b0 con 0x55defc254ba0 2024-03-24T00:15:29.839 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:29.838+0000 7fcde9266740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x55defc26b810 con 0x55defc255590 2024-03-24T00:15:29.840 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:29.838+0000 7fcddf400640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55defc254ba0 0x55defc24ae00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:15:29.840 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:29.838+0000 7fcddf400640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55defc254ba0 0x55defc24ae00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.118:55062/0 (socket says 172.21.15.118:55062) 2024-03-24T00:15:29.840 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:29.838+0000 7fcddf400640 1 -- 172.21.15.118:0/995417576 learned_addr learned my addr 172.21.15.118:0/995417576 (peer_addr_for_me v2:172.21.15.118:0/0) 2024-03-24T00:15:29.841 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:29.838+0000 7fcddfe00640 1 --2- 172.21.15.118:0/995417576 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55defc25a700 0x55defc2504f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:15:29.841 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:29.838+0000 7fcddea00640 1 --2- 172.21.15.118:0/995417576 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55defc255590 0x55defc266390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:15:29.841 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:29.838+0000 7fcddf400640 1 -- 172.21.15.118:0/995417576 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55defc255590 msgr2=0x55defc266390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:15:29.841 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:29.838+0000 7fcddf400640 1 --2- 172.21.15.118:0/995417576 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55defc255590 0x55defc266390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:15:29.841 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:29.838+0000 7fcddf400640 1 -- 172.21.15.118:0/995417576 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55defc25a700 msgr2=0x55defc2504f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:15:29.841 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:29.838+0000 7fcddf400640 1 --2- 172.21.15.118:0/995417576 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55defc25a700 0x55defc2504f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:15:29.841 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:29.838+0000 7fcddf400640 1 -- 172.21.15.118:0/995417576 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55defc26a8a0 con 0x55defc254ba0 2024-03-24T00:15:29.841 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:29.838+0000 7fcddea00640 1 --2- 172.21.15.118:0/995417576 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55defc255590 0x55defc266390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T00:15:29.841 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:29.838+0000 7fcddf400640 1 --2- 172.21.15.118:0/995417576 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55defc254ba0 0x55defc24ae00 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7fcdd00bd3a0 tx=0x7fcdd00bb970 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f02f57bda7991dc8 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:15:29.841 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:29.839+0000 7fcdcbe00640 1 -- 172.21.15.118:0/995417576 <== mon.2 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcdd00bb5a0 con 0x55defc254ba0 2024-03-24T00:15:29.841 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:29.839+0000 7fcdcbe00640 1 -- 172.21.15.118:0/995417576 <== mon.2 v2:172.21.15.189:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fcdd00bdb60 con 0x55defc254ba0 2024-03-24T00:15:29.841 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:29.839+0000 7fcdcbe00640 1 -- 172.21.15.118:0/995417576 <== mon.2 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcdd00b9670 con 0x55defc254ba0 2024-03-24T00:15:29.841 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:29.839+0000 7fcde9266740 1 -- 172.21.15.118:0/995417576 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55defc254ba0 msgr2=0x55defc24ae00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:15:29.842 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:29.839+0000 7fcde9266740 1 --2- 172.21.15.118:0/995417576 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55defc254ba0 0x55defc24ae00 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7fcdd00bd3a0 tx=0x7fcdd00bb970 comp rx=0 tx=0).stop 2024-03-24T00:15:29.842 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:29.839+0000 7fcde9266740 1 -- 172.21.15.118:0/995417576 shutdown_connections 2024-03-24T00:15:29.842 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:29.839+0000 7fcde9266740 1 --2- 172.21.15.118:0/995417576 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55defc25a700 0x55defc2504f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:15:29.842 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:29.839+0000 7fcde9266740 1 --2- 172.21.15.118:0/995417576 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55defc255590 0x55defc266390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:15:29.842 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:29.839+0000 7fcde9266740 1 --2- 172.21.15.118:0/995417576 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55defc254ba0 0x55defc24ae00 unknown :-1 s=CLOSED pgs=181 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:15:29.842 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:29.839+0000 7fcde9266740 1 -- 172.21.15.118:0/995417576 >> 172.21.15.118:0/995417576 conn(0x55defc250110 msgr2=0x55defc25d990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:15:29.842 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:29.839+0000 7fcde9266740 1 -- 172.21.15.118:0/995417576 shutdown_connections 2024-03-24T00:15:29.842 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:29.839+0000 7fcde9266740 1 -- 172.21.15.118:0/995417576 wait complete. 2024-03-24T00:15:29.842 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:29.840+0000 7fcde9266740 1 -- start start 2024-03-24T00:15:29.842 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:29.840+0000 7fcde9266740 1 Processor -- start 2024-03-24T00:15:29.846 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:29.840+0000 7fcde9266740 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55defc254ba0 0x55defc261920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:15:29.846 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:29.840+0000 7fcde9266740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55defc255590 0x55defc261e60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:15:29.846 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:29.840+0000 7fcde9266740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55defc25a700 0x55defc264e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:15:29.846 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:29.841+0000 7fcde9266740 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x55defc26c4d0 con 0x55defc254ba0 2024-03-24T00:15:29.846 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:29.841+0000 7fcde9266740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x55defc26be70 con 0x55defc255590 2024-03-24T00:15:29.846 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:29.841+0000 7fcde9266740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x55defc26cb30 con 0x55defc25a700 2024-03-24T00:15:29.846 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:29.841+0000 7fcde9266740 10 client.0 _ll_register_callbacks cb 0x7fcdcc0097a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T00:15:29.846 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:29.841+0000 7fcddf400640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55defc254ba0 0x55defc261920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:15:29.846 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:29.841+0000 7fcddea00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55defc255590 0x55defc261e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:15:29.846 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:29.841+0000 7fcddfe00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55defc25a700 0x55defc264e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:15:29.846 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:29.841+0000 7fcddea00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55defc255590 0x55defc261e60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.118:55072/0 (socket says 172.21.15.118:55072) 2024-03-24T00:15:29.847 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:29.841+0000 7fcddea00640 1 -- 172.21.15.118:0/4172220831 learned_addr learned my addr 172.21.15.118:0/4172220831 (peer_addr_for_me v2:172.21.15.118:0/0) 2024-03-24T00:15:29.847 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:29.841+0000 7fcddea00640 1 -- 172.21.15.118:0/4172220831 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55defc25a700 msgr2=0x55defc264e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:15:29.847 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:29.841+0000 7fcddea00640 1 --2- 172.21.15.118:0/4172220831 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55defc25a700 0x55defc264e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:15:29.847 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:29.841+0000 7fcddea00640 1 -- 172.21.15.118:0/4172220831 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55defc254ba0 msgr2=0x55defc261920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:15:29.847 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:29.841+0000 7fcddea00640 1 --2- 172.21.15.118:0/4172220831 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55defc254ba0 0x55defc261920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:15:29.847 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:29.841+0000 7fcddea00640 1 -- 172.21.15.118:0/4172220831 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55defc265470 con 0x55defc255590 2024-03-24T00:15:29.847 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:29.841+0000 7fcddea00640 1 --2- 172.21.15.118:0/4172220831 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55defc255590 0x55defc261e60 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7fcdcc0125e0 tx=0x7fcdcc020b40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=74123501de6517e2 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:15:29.847 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:29.841+0000 7fcdc9600640 10 client.0 ms_handle_connect on v2:172.21.15.189:3300/0 2024-03-24T00:15:29.847 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:29.842+0000 7fcdc9600640 1 -- 172.21.15.118:0/4172220831 <== mon.2 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcdcc0202f0 con 0x55defc255590 2024-03-24T00:15:29.847 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:29.842+0000 7fcdc9600640 1 -- 172.21.15.118:0/4172220831 <== mon.2 v2:172.21.15.189:3300/0 2 ==== osd_map(90..90 src has 1..90) v4 ==== 8277+0+0 (secure 0 0 0) 0x7fcdcc012b20 con 0x55defc255590 2024-03-24T00:15:29.847 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:29.842+0000 7fcde9266740 1 -- 172.21.15.118:0/4172220831 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fcdd00bdb60 con 0x55defc255590 2024-03-24T00:15:29.847 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:29.842+0000 7fcdc9600640 1 -- 172.21.15.118:0/4172220831 <== mon.2 v2:172.21.15.189:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) v2 ==== 24+0+0 (secure 0 0 0) 0x7fcdcc030050 con 0x55defc255590 2024-03-24T00:15:29.847 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:29.842+0000 7fcde9266740 10 client.25223 fetch_fsmap learned FSMap version 27 2024-03-24T00:15:29.848 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:29.842+0000 7fcde9266740 1 -- 172.21.15.118:0/4172220831 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) v3 -- 0x55defc2c6390 con 0x55defc255590 2024-03-24T00:15:29.848 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:29.842+0000 7fcdc9600640 1 -- 172.21.15.118:0/4172220831 <== mon.2 v2:172.21.15.189:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fcdd00bdb60 con 0x55defc255590 2024-03-24T00:15:29.848 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:29.842+0000 7fcdc9600640 1 -- 172.21.15.118:0/4172220831 <== mon.2 v2:172.21.15.189:3300/0 5 ==== fsmap(e 27) v1 ==== 7934+0+0 (secure 0 0 0) 0x7fcdcc01e3f0 con 0x55defc255590 2024-03-24T00:15:29.848 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:29.842+0000 7fcde9266740 10 client.25223 fetch_fsmap finished waiting for FSMap version 27 2024-03-24T00:15:29.848 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:29.842+0000 7fcde9266740 10 client.25223 resolve_mds: resolved name 'g' to daemon mds.g 2024-03-24T00:15:29.848 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:29.842+0000 7fcde9266740 20 client.25223 populate_metadata read hostname 'smithi118' 2024-03-24T00:15:29.848 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:29.842+0000 7fcde9266740 1 --2- 172.21.15.118:0/4172220831 >> [v2:172.21.15.118:6838/3285788503,v1:172.21.15.118:6839/3285788503] conn(0x55defc2ca9d0 0x55defc2c6610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:15:29.848 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:29.842+0000 7fcde9266740 4 client.25223 mds_command: new command op to 24550 tid=0 multi_id=0 [{"prefix": "status"}] 2024-03-24T00:15:29.848 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:29.842+0000 7fcde9266740 1 -- 172.21.15.118:0/4172220831 --> [v2:172.21.15.118:6838/3285788503,v1:172.21.15.118:6839/3285788503] -- command(tid 0: {"prefix": "status"}) v1 -- 0x55defc1c1520 con 0x55defc2ca9d0 2024-03-24T00:15:29.848 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:29.843+0000 7fcddf400640 1 --2- 172.21.15.118:0/4172220831 >> [v2:172.21.15.118:6838/3285788503,v1:172.21.15.118:6839/3285788503] conn(0x55defc2ca9d0 0x55defc2c6610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:15:29.848 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:29.843+0000 7fcddf400640 1 --2- 172.21.15.118:0/4172220831 >> [v2:172.21.15.118:6838/3285788503,v1:172.21.15.118:6839/3285788503] conn(0x55defc2ca9d0 0x55defc2c6610 crc :-1 s=READY pgs=19 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=cce1d644c9f05ff2 server_cookie=4f4eaf8eb09602a3 in_seq=0 out_seq=0 2024-03-24T00:15:29.848 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:29.843+0000 7fcdc9600640 10 client.25223 ms_handle_connect on v2:172.21.15.118:6838/3285788503 2024-03-24T00:15:29.848 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:29.843+0000 7fcdc9600640 1 -- 172.21.15.118:0/4172220831 <== mds.1 v2:172.21.15.118:6838/3285788503 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+322 (crc 0 0 0) 0x55defc1c1520 con 0x55defc2ca9d0 2024-03-24T00:15:29.848 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:29.843+0000 7fcdc9600640 10 client.25223 handle_command_reply: tid=0 2024-03-24T00:15:29.849 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:29.843+0000 7fcde9266740 1 client.25223 shutdown 2024-03-24T00:15:29.849 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:29.844+0000 7fcde9266740 1 -- 172.21.15.118:0/4172220831 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55defc255590 msgr2=0x55defc261e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:15:29.849 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:29.844+0000 7fcde9266740 1 --2- 172.21.15.118:0/4172220831 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55defc255590 0x55defc261e60 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7fcdcc0125e0 tx=0x7fcdcc020b40 comp rx=0 tx=0).stop 2024-03-24T00:15:29.849 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:29.844+0000 7fcde9266740 1 -- 172.21.15.118:0/4172220831 shutdown_connections 2024-03-24T00:15:29.849 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:29.844+0000 7fcde9266740 1 --2- 172.21.15.118:0/4172220831 >> [v2:172.21.15.118:6838/3285788503,v1:172.21.15.118:6839/3285788503] conn(0x55defc2ca9d0 0x55defc2c6610 crc :-1 s=READY pgs=19 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:15:29.849 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:29.844+0000 7fcde9266740 1 --2- 172.21.15.118:0/4172220831 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55defc25a700 0x55defc264e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:15:29.849 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:29.844+0000 7fcde9266740 1 --2- 172.21.15.118:0/4172220831 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55defc255590 0x55defc261e60 unknown :-1 s=CLOSED pgs=182 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:15:29.849 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:29.844+0000 7fcde9266740 1 --2- 172.21.15.118:0/4172220831 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55defc254ba0 0x55defc261920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:15:29.849 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:29.844+0000 7fcdc9600640 0 client.25223 ms_handle_reset on v2:172.21.15.118:6838/3285788503 2024-03-24T00:15:29.849 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:29.844+0000 7fcde9266740 1 -- 172.21.15.118:0/4172220831 >> 172.21.15.118:0/4172220831 conn(0x55defc250110 msgr2=0x55defc24ae00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:15:29.849 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:29.844+0000 7fcde9266740 1 -- 172.21.15.118:0/4172220831 shutdown_connections 2024-03-24T00:15:29.849 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:29.844+0000 7fcde9266740 1 -- 172.21.15.118:0/4172220831 wait complete. 2024-03-24T00:15:29.849 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:29.844+0000 7fcde9266740 20 client.25223 trim_cache size 0 max 16384 2024-03-24T00:15:29.850 INFO:teuthology.orchestra.run.smithi118.stdout:{ 2024-03-24T00:15:29.850 INFO:teuthology.orchestra.run.smithi118.stdout: "cluster_fsid": "d79a5420-e971-11ee-95cd-87774f69a715", 2024-03-24T00:15:29.850 INFO:teuthology.orchestra.run.smithi118.stdout: "whoami": 1, 2024-03-24T00:15:29.850 INFO:teuthology.orchestra.run.smithi118.stdout: "id": 24550, 2024-03-24T00:15:29.850 INFO:teuthology.orchestra.run.smithi118.stdout: "want_state": "up:active", 2024-03-24T00:15:29.850 INFO:teuthology.orchestra.run.smithi118.stdout: "state": "up:active", 2024-03-24T00:15:29.850 INFO:teuthology.orchestra.run.smithi118.stdout: "fs_name": "cephfs", 2024-03-24T00:15:29.850 INFO:teuthology.orchestra.run.smithi118.stdout: "rank_uptime": 144.460752404, 2024-03-24T00:15:29.850 INFO:teuthology.orchestra.run.smithi118.stdout: "mdsmap_epoch": 27, 2024-03-24T00:15:29.850 INFO:teuthology.orchestra.run.smithi118.stdout: "osdmap_epoch": 90, 2024-03-24T00:15:29.851 INFO:teuthology.orchestra.run.smithi118.stdout: "osdmap_epoch_barrier": 86, 2024-03-24T00:15:29.851 INFO:teuthology.orchestra.run.smithi118.stdout: "uptime": 161.503087975 2024-03-24T00:15:29.851 INFO:teuthology.orchestra.run.smithi118.stdout:} 2024-03-24T00:15:29.851 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:29.845+0000 7fcde6600640 1 -- 172.21.15.118:0/2297781045 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7fcdc40a2800 msgr2=0x7fcdc40a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:15:29.852 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:29.845+0000 7fcde6600640 1 --2- 172.21.15.118:0/2297781045 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7fcdc40a2800 0x7fcdc40a4cc0 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x55defc255df0 tx=0x7fcdd8004ab0 comp rx=0 tx=0).stop 2024-03-24T00:15:29.852 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:29.845+0000 7fcde6600640 1 -- 172.21.15.118:0/2297781045 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fcde01038e0 msgr2=0x7fcde0199c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:15:29.852 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:29.845+0000 7fcde6600640 1 --2- 172.21.15.118:0/2297781045 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fcde01038e0 0x7fcde0199c70 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7fcdd0000c00 tx=0x7fcdd002c650 comp rx=0 tx=0).stop 2024-03-24T00:15:29.852 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:29.845+0000 7fcde6600640 1 -- 172.21.15.118:0/2297781045 shutdown_connections 2024-03-24T00:15:29.852 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:29.845+0000 7fcde6600640 1 --2- 172.21.15.118:0/2297781045 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7fcdc40a2800 0x7fcdc40a4cc0 unknown :-1 s=CLOSED pgs=246 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:15:29.852 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:29.845+0000 7fcde6600640 1 --2- 172.21.15.118:0/2297781045 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fcde0105490 0x7fcde01a11e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:15:29.852 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:29.845+0000 7fcde6600640 1 --2- 172.21.15.118:0/2297781045 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fcde0104ad0 0x7fcde019a1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:15:29.852 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:29.845+0000 7fcde6600640 1 --2- 172.21.15.118:0/2297781045 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fcde01038e0 0x7fcde0199c70 unknown :-1 s=CLOSED pgs=178 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:15:29.852 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:29.845+0000 7fcde6600640 1 -- 172.21.15.118:0/2297781045 >> 172.21.15.118:0/2297781045 conn(0x7fcde00fee50 msgr2=0x7fcde0107df0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:15:29.852 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:29.845+0000 7fcde6600640 1 -- 172.21.15.118:0/2297781045 shutdown_connections 2024-03-24T00:15:29.852 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:29.845+0000 7fcde6600640 1 -- 172.21.15.118:0/2297781045 wait complete. 2024-03-24T00:15:29.862 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:15:29 smithi118 ceph-mon[31535]: pgmap v392: 97 pgs: 97 active+clean; 645 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 11 op/s 2024-03-24T00:15:29.866 DEBUG:teuthology.orchestra.run.smithi118:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.b status 2024-03-24T00:15:30.072 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:30.070+0000 7f38dda00640 1 -- 172.21.15.118:0/2534906031 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f38d81038e0 msgr2=0x7f38d8103d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:15:30.072 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:30.070+0000 7f38dda00640 1 --2- 172.21.15.118:0/2534906031 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f38d81038e0 0x7f38d8103d00 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 crypto rx=0x7f38c8009900 tx=0x7f38c802bc30 comp rx=0 tx=0).stop 2024-03-24T00:15:30.072 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:30.071+0000 7f38dda00640 1 -- 172.21.15.118:0/2534906031 shutdown_connections 2024-03-24T00:15:30.072 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:30.071+0000 7f38dda00640 1 --2- 172.21.15.118:0/2534906031 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f38d8105490 0x7f38d810bd20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:15:30.072 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:30.071+0000 7f38dda00640 1 --2- 172.21.15.118:0/2534906031 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f38d8104ad0 0x7f38d8104f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:15:30.072 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:30.071+0000 7f38dda00640 1 --2- 172.21.15.118:0/2534906031 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f38d81038e0 0x7f38d8103d00 unknown :-1 s=CLOSED pgs=173 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:15:30.072 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:30.071+0000 7f38dda00640 1 -- 172.21.15.118:0/2534906031 >> 172.21.15.118:0/2534906031 conn(0x7f38d80fee50 msgr2=0x7f38d8101290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:15:30.072 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:30.071+0000 7f38dda00640 1 -- 172.21.15.118:0/2534906031 shutdown_connections 2024-03-24T00:15:30.072 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:30.071+0000 7f38dda00640 1 -- 172.21.15.118:0/2534906031 wait complete. 2024-03-24T00:15:30.074 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:30.072+0000 7f38dda00640 1 Processor -- start 2024-03-24T00:15:30.074 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:30.072+0000 7f38dda00640 1 -- start start 2024-03-24T00:15:30.074 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:30.073+0000 7f38dda00640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f38d81038e0 0x7f38d8199c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:15:30.074 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:30.073+0000 7f38dda00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f38d8104ad0 0x7f38d819a180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:15:30.074 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:30.073+0000 7f38dda00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f38d8105490 0x7f38d81a11b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:15:30.074 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:30.073+0000 7f38dda00640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f38d810f580 con 0x7f38d81038e0 2024-03-24T00:15:30.074 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:30.073+0000 7f38dda00640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f38d810f400 con 0x7f38d8104ad0 2024-03-24T00:15:30.075 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:30.073+0000 7f38dda00640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f38d810f700 con 0x7f38d8105490 2024-03-24T00:15:30.075 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:30.073+0000 7f38d6a00640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f38d81038e0 0x7f38d8199c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:15:30.075 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:30.073+0000 7f38ce000640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f38d8104ad0 0x7f38d819a180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:15:30.075 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:30.073+0000 7f38d6a00640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f38d81038e0 0x7f38d8199c40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.118:3300/0 says I am v2:172.21.15.118:34730/0 (socket says 172.21.15.118:34730) 2024-03-24T00:15:30.075 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:30.073+0000 7f38ce000640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f38d8104ad0 0x7f38d819a180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.118:55096/0 (socket says 172.21.15.118:55096) 2024-03-24T00:15:30.075 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:30.073+0000 7f38d7400640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f38d8105490 0x7f38d81a11b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:15:30.075 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:30.073+0000 7f38d6a00640 1 -- 172.21.15.118:0/805042186 learned_addr learned my addr 172.21.15.118:0/805042186 (peer_addr_for_me v2:172.21.15.118:0/0) 2024-03-24T00:15:30.075 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:30.073+0000 7f38ce000640 1 -- 172.21.15.118:0/805042186 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f38d8105490 msgr2=0x7f38d81a11b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:15:30.075 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:30.073+0000 7f38ce000640 1 --2- 172.21.15.118:0/805042186 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f38d8105490 0x7f38d81a11b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:15:30.075 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:30.073+0000 7f38ce000640 1 -- 172.21.15.118:0/805042186 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f38d81038e0 msgr2=0x7f38d8199c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:15:30.075 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:30.073+0000 7f38ce000640 1 --2- 172.21.15.118:0/805042186 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f38d81038e0 0x7f38d8199c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:15:30.075 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:30.073+0000 7f38ce000640 1 -- 172.21.15.118:0/805042186 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f38d81a1850 con 0x7f38d8104ad0 2024-03-24T00:15:30.075 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:30.074+0000 7f38ce000640 1 --2- 172.21.15.118:0/805042186 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f38d8104ad0 0x7f38d819a180 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7f38c000e9f0 tx=0x7f38c000eeb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:15:30.076 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:30.074+0000 7f38cfe00640 1 -- 172.21.15.118:0/805042186 <== mon.2 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f38c000ccc0 con 0x7f38d8104ad0 2024-03-24T00:15:30.076 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:30.074+0000 7f38cfe00640 1 -- 172.21.15.118:0/805042186 <== mon.2 v2:172.21.15.189:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f38c0010910 con 0x7f38d8104ad0 2024-03-24T00:15:30.076 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:30.074+0000 7f38dda00640 1 -- 172.21.15.118:0/805042186 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f38d81a1b40 con 0x7f38d8104ad0 2024-03-24T00:15:30.076 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:30.074+0000 7f38cfe00640 1 -- 172.21.15.118:0/805042186 <== mon.2 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f38c0018960 con 0x7f38d8104ad0 2024-03-24T00:15:30.076 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:30.074+0000 7f38dda00640 1 -- 172.21.15.118:0/805042186 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f38d81a2090 con 0x7f38d8104ad0 2024-03-24T00:15:30.077 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:30.077+0000 7f38cfe00640 1 -- 172.21.15.118:0/805042186 <== mon.2 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 18) v1 ==== 137291+0+0 (secure 0 0 0) 0x7f38c0010ac0 con 0x7f38d8104ad0 2024-03-24T00:15:30.078 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:30.077+0000 7f38e07ce740 1 Processor -- start 2024-03-24T00:15:30.081 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:30.077+0000 7f38e07ce740 1 -- start start 2024-03-24T00:15:30.081 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:30.078+0000 7f38e07ce740 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55e59fd81ba0 0x55e59fd81fc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:15:30.081 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:30.078+0000 7f38e07ce740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55e59fd82590 0x55e59fd87090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:15:30.081 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:30.078+0000 7f38e07ce740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55e59fd87700 0x55e59fd89b90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:15:30.081 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:30.078+0000 7f38e07ce740 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e59fce6c90 con 0x55e59fd81ba0 2024-03-24T00:15:30.081 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:30.078+0000 7f38e07ce740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e59fc8f980 con 0x55e59fd82590 2024-03-24T00:15:30.081 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:30.078+0000 7f38e07ce740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e59fce2d30 con 0x55e59fd87700 2024-03-24T00:15:30.081 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:30.078+0000 7f38d7400640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55e59fd81ba0 0x55e59fd81fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:15:30.082 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:30.078+0000 7f38ce000640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55e59fd87700 0x55e59fd89b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:15:30.082 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:30.078+0000 7f38d6a00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55e59fd82590 0x55e59fd87090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:15:30.082 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:30.078+0000 7f38ce000640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55e59fd87700 0x55e59fd89b90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.118:47124/0 (socket says 172.21.15.118:47124) 2024-03-24T00:15:30.082 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:30.078+0000 7f38ce000640 1 -- 172.21.15.118:0/779014025 learned_addr learned my addr 172.21.15.118:0/779014025 (peer_addr_for_me v2:172.21.15.118:0/0) 2024-03-24T00:15:30.082 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:30.078+0000 7f38d7400640 1 --2- 172.21.15.118:0/779014025 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55e59fd81ba0 0x55e59fd81fc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.118:3300/0 says I am v2:172.21.15.118:34738/0 (socket says 172.21.15.118:34738) 2024-03-24T00:15:30.082 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:30.078+0000 7f38d6a00640 1 -- 172.21.15.118:0/779014025 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55e59fd87700 msgr2=0x55e59fd89b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:15:30.082 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:30.078+0000 7f38d6a00640 1 --2- 172.21.15.118:0/779014025 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55e59fd87700 0x55e59fd89b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:15:30.082 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:30.078+0000 7f38d6a00640 1 -- 172.21.15.118:0/779014025 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55e59fd81ba0 msgr2=0x55e59fd81fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:15:30.082 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:30.078+0000 7f38d6a00640 1 --2- 172.21.15.118:0/779014025 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55e59fd81ba0 0x55e59fd81fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:15:30.082 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:30.079+0000 7f38d6a00640 1 -- 172.21.15.118:0/779014025 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55e59fca7510 con 0x55e59fd82590 2024-03-24T00:15:30.082 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:30.079+0000 7f38cfe00640 1 --2- 172.21.15.118:0/805042186 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7f38bc0a2800 0x7f38bc0a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:15:30.082 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:30.079+0000 7f38d6a00640 1 --2- 172.21.15.118:0/779014025 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55e59fd82590 0x55e59fd87090 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7f38c8000c00 tx=0x7f38c8009560 comp rx=0 tx=0).ready entity=mon.2 client_cookie=4e2432dc9c7edffd server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:15:30.082 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:30.079+0000 7f38cfe00640 1 -- 172.21.15.118:0/805042186 <== mon.2 v2:172.21.15.189:3300/0 5 ==== osd_map(90..90 src has 1..90) v4 ==== 8277+0+0 (secure 0 0 0) 0x7f38c00cef30 con 0x7f38d8104ad0 2024-03-24T00:15:30.083 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:30.079+0000 7f38d7400640 1 --2- 172.21.15.118:0/805042186 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7f38bc0a2800 0x7f38bc0a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:15:30.083 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:30.079+0000 7f38c7e00640 1 -- 172.21.15.118:0/779014025 <== mon.2 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f38c8045070 con 0x55e59fd82590 2024-03-24T00:15:30.083 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:30.079+0000 7f38c7e00640 1 -- 172.21.15.118:0/779014025 <== mon.2 v2:172.21.15.189:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f38c8004040 con 0x55e59fd82590 2024-03-24T00:15:30.083 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:30.079+0000 7f38c7e00640 1 -- 172.21.15.118:0/779014025 <== mon.2 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f38c8004ef0 con 0x55e59fd82590 2024-03-24T00:15:30.083 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:30.080+0000 7f38e07ce740 1 -- 172.21.15.118:0/779014025 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55e59fd82590 msgr2=0x55e59fd87090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:15:30.083 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:30.080+0000 7f38e07ce740 1 --2- 172.21.15.118:0/779014025 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55e59fd82590 0x55e59fd87090 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7f38c8000c00 tx=0x7f38c8009560 comp rx=0 tx=0).stop 2024-03-24T00:15:30.083 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:30.080+0000 7f38d7400640 1 --2- 172.21.15.118:0/805042186 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7f38bc0a2800 0x7f38bc0a4cc0 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x55e59fd82df0 tx=0x7f38d0009500 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:15:30.083 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:30.080+0000 7f38e07ce740 1 -- 172.21.15.118:0/779014025 shutdown_connections 2024-03-24T00:15:30.083 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:30.080+0000 7f38e07ce740 1 --2- 172.21.15.118:0/779014025 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55e59fd87700 0x55e59fd89b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:15:30.083 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:30.080+0000 7f38e07ce740 1 --2- 172.21.15.118:0/779014025 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55e59fd82590 0x55e59fd87090 unknown :-1 s=CLOSED pgs=184 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:15:30.083 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:30.080+0000 7f38e07ce740 1 --2- 172.21.15.118:0/779014025 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55e59fd81ba0 0x55e59fd81fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:15:30.083 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:30.080+0000 7f38e07ce740 1 -- 172.21.15.118:0/779014025 >> 172.21.15.118:0/779014025 conn(0x55e59fd7d110 msgr2=0x55e59fd7d510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:15:30.083 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:30.080+0000 7f38e07ce740 1 -- 172.21.15.118:0/779014025 shutdown_connections 2024-03-24T00:15:30.083 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:30.080+0000 7f38e07ce740 1 -- 172.21.15.118:0/779014025 wait complete. 2024-03-24T00:15:30.083 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:30.080+0000 7f38e07ce740 1 -- start start 2024-03-24T00:15:30.084 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:30.081+0000 7f38e07ce740 1 Processor -- start 2024-03-24T00:15:30.087 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:30.081+0000 7f38e07ce740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55e59fd81ba0 0x55e59fd8c090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:15:30.087 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:30.081+0000 7f38e07ce740 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55e59fd82590 0x55e59fd8c5d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:15:30.087 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:30.081+0000 7f38e07ce740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55e59fd87700 0x55e59fd93610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:15:30.088 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:30.081+0000 7f38e07ce740 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e59fd8a650 con 0x55e59fd82590 2024-03-24T00:15:30.088 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:30.081+0000 7f38e07ce740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e59fca2dd0 con 0x55e59fd87700 2024-03-24T00:15:30.088 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:30.081+0000 7f38e07ce740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e59fd8a7d0 con 0x55e59fd81ba0 2024-03-24T00:15:30.088 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:30.081+0000 7f38e07ce740 10 client.0 _ll_register_callbacks cb 0x55e59f9faac0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T00:15:30.088 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:30.081+0000 7f38d6a00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55e59fd81ba0 0x55e59fd8c090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:15:30.088 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:30.081+0000 7f38d6a00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55e59fd81ba0 0x55e59fd8c090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.118:47136/0 (socket says 172.21.15.118:47136) 2024-03-24T00:15:30.088 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:30.081+0000 7f38d7400640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55e59fd87700 0x55e59fd93610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:15:30.088 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:30.081+0000 7f38d6a00640 1 -- 172.21.15.118:0/3814146056 learned_addr learned my addr 172.21.15.118:0/3814146056 (peer_addr_for_me v2:172.21.15.118:0/0) 2024-03-24T00:15:30.088 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:30.081+0000 7f38ce000640 1 --2- 172.21.15.118:0/3814146056 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55e59fd82590 0x55e59fd8c5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:15:30.088 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:30.082+0000 7f38d6a00640 1 -- 172.21.15.118:0/3814146056 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55e59fd87700 msgr2=0x55e59fd93610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:15:30.088 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:30.082+0000 7f38d6a00640 1 --2- 172.21.15.118:0/3814146056 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55e59fd87700 0x55e59fd93610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:15:30.088 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:30.082+0000 7f38d6a00640 1 -- 172.21.15.118:0/3814146056 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55e59fd82590 msgr2=0x55e59fd8c5d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:15:30.088 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:30.082+0000 7f38d6a00640 1 --2- 172.21.15.118:0/3814146056 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55e59fd82590 0x55e59fd8c5d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:15:30.088 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:30.082+0000 7f38d6a00640 1 -- 172.21.15.118:0/3814146056 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55e59f9fb2d0 con 0x55e59fd81ba0 2024-03-24T00:15:30.088 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:30.082+0000 7f38ce000640 1 --2- 172.21.15.118:0/3814146056 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55e59fd82590 0x55e59fd8c5d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T00:15:30.089 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:30.082+0000 7f38d7400640 1 --2- 172.21.15.118:0/3814146056 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55e59fd87700 0x55e59fd93610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T00:15:30.089 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:30.082+0000 7f38d6a00640 1 --2- 172.21.15.118:0/3814146056 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55e59fd81ba0 0x55e59fd8c090 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7f38c802c240 tx=0x7f38c8005fd0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=5ec28f367387b10d server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:15:30.089 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:30.082+0000 7f38c4c00640 10 client.0 ms_handle_connect on v2:172.21.15.193:3300/0 2024-03-24T00:15:30.089 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:30.082+0000 7f38c4c00640 1 -- 172.21.15.118:0/3814146056 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f38c8045040 con 0x55e59fd81ba0 2024-03-24T00:15:30.089 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:30.082+0000 7f38c4c00640 1 -- 172.21.15.118:0/3814146056 <== mon.1 v2:172.21.15.193:3300/0 2 ==== osd_map(90..90 src has 1..90) v4 ==== 8277+0+0 (secure 0 0 0) 0x7f38c8004ef0 con 0x55e59fd81ba0 2024-03-24T00:15:30.089 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:30.082+0000 7f38e07ce740 1 -- 172.21.15.118:0/3814146056 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f38c8004040 con 0x55e59fd81ba0 2024-03-24T00:15:30.089 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:30.083+0000 7f38c4c00640 1 -- 172.21.15.118:0/3814146056 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) v2 ==== 24+0+0 (secure 0 0 0) 0x7f38c805b050 con 0x55e59fd81ba0 2024-03-24T00:15:30.089 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:30.083+0000 7f38e07ce740 10 client.25246 fetch_fsmap learned FSMap version 27 2024-03-24T00:15:30.089 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:30.083+0000 7f38e07ce740 1 -- 172.21.15.118:0/3814146056 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) v3 -- 0x55e59fac56b0 con 0x55e59fd81ba0 2024-03-24T00:15:30.089 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:30.083+0000 7f38c4c00640 1 -- 172.21.15.118:0/3814146056 <== mon.1 v2:172.21.15.193:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f38c8004040 con 0x55e59fd81ba0 2024-03-24T00:15:30.089 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:30.083+0000 7f38c4c00640 1 -- 172.21.15.118:0/3814146056 <== mon.1 v2:172.21.15.193:3300/0 5 ==== fsmap(e 27) v1 ==== 7934+0+0 (secure 0 0 0) 0x7f38c8003990 con 0x55e59fd81ba0 2024-03-24T00:15:30.089 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:30.083+0000 7f38e07ce740 10 client.25246 fetch_fsmap finished waiting for FSMap version 27 2024-03-24T00:15:30.089 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:30.083+0000 7f38e07ce740 10 client.25246 resolve_mds: resolved name 'b' to daemon mds.b 2024-03-24T00:15:30.089 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:30.083+0000 7f38e07ce740 20 client.25246 populate_metadata read hostname 'smithi118' 2024-03-24T00:15:30.089 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:30.083+0000 7f38e07ce740 1 --2- 172.21.15.118:0/3814146056 >> [v2:172.21.15.189:6838/4074484168,v1:172.21.15.189:6839/4074484168] conn(0x55e59fd97770 0x55e59fd9c7e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:15:30.089 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:30.083+0000 7f38e07ce740 4 client.25246 mds_command: new command op to 24569 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-24T00:15:30.090 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:30.083+0000 7f38e07ce740 1 -- 172.21.15.118:0/3814146056 --> [v2:172.21.15.189:6838/4074484168,v1:172.21.15.189:6839/4074484168] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55e59fc88520 con 0x55e59fd97770 2024-03-24T00:15:30.090 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:30.083+0000 7f38ce000640 1 --2- 172.21.15.118:0/3814146056 >> [v2:172.21.15.189:6838/4074484168,v1:172.21.15.189:6839/4074484168] conn(0x55e59fd97770 0x55e59fd9c7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:15:30.090 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:30.084+0000 7f38ce000640 1 --2- 172.21.15.118:0/3814146056 >> [v2:172.21.15.189:6838/4074484168,v1:172.21.15.189:6839/4074484168] conn(0x55e59fd97770 0x55e59fd9c7e0 crc :-1 s=READY pgs=29 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=b4d29af28a2da844 server_cookie=968a0d33a8abd2fe in_seq=0 out_seq=0 2024-03-24T00:15:30.090 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:30.084+0000 7f38c4c00640 10 client.25246 ms_handle_connect on v2:172.21.15.189:6838/4074484168 2024-03-24T00:15:30.090 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:30.085+0000 7f38c4c00640 1 -- 172.21.15.118:0/3814146056 <== mds.0 v2:172.21.15.189:6838/4074484168 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x55e59fc88520 con 0x55e59fd97770 2024-03-24T00:15:30.090 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:30.085+0000 7f38c4c00640 10 client.25246 handle_command_reply: tid=0 2024-03-24T00:15:30.090 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:30.085+0000 7f38e07ce740 1 client.25246 shutdown 2024-03-24T00:15:30.090 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:30.085+0000 7f38e07ce740 1 -- 172.21.15.118:0/3814146056 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55e59fd81ba0 msgr2=0x55e59fd8c090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:15:30.090 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:30.085+0000 7f38e07ce740 1 --2- 172.21.15.118:0/3814146056 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55e59fd81ba0 0x55e59fd8c090 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 crypto rx=0x7f38c802c240 tx=0x7f38c8005fd0 comp rx=0 tx=0).stop 2024-03-24T00:15:30.090 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:30.085+0000 7f38e07ce740 1 -- 172.21.15.118:0/3814146056 shutdown_connections 2024-03-24T00:15:30.090 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:30.085+0000 7f38e07ce740 1 --2- 172.21.15.118:0/3814146056 >> [v2:172.21.15.189:6838/4074484168,v1:172.21.15.189:6839/4074484168] conn(0x55e59fd97770 0x55e59fd9c7e0 crc :-1 s=READY pgs=29 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:15:30.090 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:30.085+0000 7f38e07ce740 1 --2- 172.21.15.118:0/3814146056 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55e59fd87700 0x55e59fd93610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:15:30.090 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:30.085+0000 7f38e07ce740 1 --2- 172.21.15.118:0/3814146056 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55e59fd82590 0x55e59fd8c5d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:15:30.090 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:30.085+0000 7f38e07ce740 1 --2- 172.21.15.118:0/3814146056 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55e59fd81ba0 0x55e59fd8c090 unknown :-1 s=CLOSED pgs=174 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:15:30.091 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:30.085+0000 7f38c4c00640 0 client.25246 ms_handle_reset on v2:172.21.15.189:6838/4074484168 2024-03-24T00:15:30.091 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:30.085+0000 7f38e07ce740 1 -- 172.21.15.118:0/3814146056 >> 172.21.15.118:0/3814146056 conn(0x55e59fd7d110 msgr2=0x55e59fd7c310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:15:30.091 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:30.085+0000 7f38e07ce740 1 -- 172.21.15.118:0/3814146056 shutdown_connections 2024-03-24T00:15:30.091 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:30.085+0000 7f38e07ce740 1 -- 172.21.15.118:0/3814146056 wait complete. 2024-03-24T00:15:30.091 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:30.085+0000 7f38e07ce740 20 client.25246 trim_cache size 0 max 16384 2024-03-24T00:15:30.105 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:30.103+0000 7f38e07ce740 1 Processor -- start 2024-03-24T00:15:30.105 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:30.103+0000 7f38e07ce740 1 -- start start 2024-03-24T00:15:30.105 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:30.103+0000 7f38e07ce740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55e59fd81ba0 0x55e59fd93ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:15:30.105 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:30.103+0000 7f38e07ce740 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55e59fd82590 0x55e59fd8abd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:15:30.105 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:30.103+0000 7f38e07ce740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55e59fd87700 0x55e59fd7d4f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:15:30.105 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:30.103+0000 7f38e07ce740 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e59fd97b80 con 0x55e59fd82590 2024-03-24T00:15:30.105 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:30.103+0000 7f38e07ce740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e59fd981e0 con 0x55e59fd87700 2024-03-24T00:15:30.105 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:30.103+0000 7f38e07ce740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e59fd98840 con 0x55e59fd81ba0 2024-03-24T00:15:30.105 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:30.104+0000 7f38ce000640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55e59fd82590 0x55e59fd8abd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:15:30.105 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:30.104+0000 7f38d6a00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55e59fd81ba0 0x55e59fd93ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:15:30.105 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:30.104+0000 7f38d6a00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55e59fd81ba0 0x55e59fd93ca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.118:47152/0 (socket says 172.21.15.118:47152) 2024-03-24T00:15:30.105 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:30.104+0000 7f38d6a00640 1 -- 172.21.15.118:0/2119012372 learned_addr learned my addr 172.21.15.118:0/2119012372 (peer_addr_for_me v2:172.21.15.118:0/0) 2024-03-24T00:15:30.105 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:30.104+0000 7f38d7400640 1 --2- 172.21.15.118:0/2119012372 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55e59fd87700 0x55e59fd7d4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:15:30.105 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:30.104+0000 7f38ce000640 1 -- 172.21.15.118:0/2119012372 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55e59fd81ba0 msgr2=0x55e59fd93ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:15:30.106 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:30.104+0000 7f38ce000640 1 --2- 172.21.15.118:0/2119012372 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55e59fd81ba0 0x55e59fd93ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:15:30.106 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:30.104+0000 7f38ce000640 1 -- 172.21.15.118:0/2119012372 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55e59fd87700 msgr2=0x55e59fd7d4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:15:30.106 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:30.104+0000 7f38ce000640 1 --2- 172.21.15.118:0/2119012372 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55e59fd87700 0x55e59fd7d4f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:15:30.106 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:30.104+0000 7f38ce000640 1 -- 172.21.15.118:0/2119012372 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55e59fd86d40 con 0x55e59fd82590 2024-03-24T00:15:30.106 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:30.104+0000 7f38d6a00640 1 --2- 172.21.15.118:0/2119012372 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55e59fd81ba0 0x55e59fd93ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T00:15:30.106 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:30.104+0000 7f38ce000640 1 --2- 172.21.15.118:0/2119012372 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55e59fd82590 0x55e59fd8abd0 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7f38c009b880 tx=0x7f38c00cc870 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b231f61b9363fa6c server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:15:30.107 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:30.105+0000 7f38c7400640 1 -- 172.21.15.118:0/2119012372 <== mon.0 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f38c00b4070 con 0x55e59fd82590 2024-03-24T00:15:30.107 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:30.105+0000 7f38c7400640 1 -- 172.21.15.118:0/2119012372 <== mon.0 v2:172.21.15.118:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f38c009b8f0 con 0x55e59fd82590 2024-03-24T00:15:30.107 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:30.105+0000 7f38c7400640 1 -- 172.21.15.118:0/2119012372 <== mon.0 v2:172.21.15.118:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f38c00cce60 con 0x55e59fd82590 2024-03-24T00:15:30.107 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:30.105+0000 7f38e07ce740 1 -- 172.21.15.118:0/2119012372 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55e59fd82590 msgr2=0x55e59fd8abd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:15:30.107 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:30.105+0000 7f38e07ce740 1 --2- 172.21.15.118:0/2119012372 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55e59fd82590 0x55e59fd8abd0 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7f38c009b880 tx=0x7f38c00cc870 comp rx=0 tx=0).stop 2024-03-24T00:15:30.107 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:30.106+0000 7f38e07ce740 1 -- 172.21.15.118:0/2119012372 shutdown_connections 2024-03-24T00:15:30.107 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:30.106+0000 7f38e07ce740 1 --2- 172.21.15.118:0/2119012372 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55e59fd87700 0x55e59fd7d4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:15:30.107 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:30.106+0000 7f38e07ce740 1 --2- 172.21.15.118:0/2119012372 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55e59fd82590 0x55e59fd8abd0 unknown :-1 s=CLOSED pgs=175 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:15:30.108 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:30.106+0000 7f38e07ce740 1 --2- 172.21.15.118:0/2119012372 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55e59fd81ba0 0x55e59fd93ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:15:30.108 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:30.106+0000 7f38e07ce740 1 -- 172.21.15.118:0/2119012372 >> 172.21.15.118:0/2119012372 conn(0x55e59fd7d110 msgr2=0x55e59fd83000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:15:30.108 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:30.106+0000 7f38e07ce740 1 -- 172.21.15.118:0/2119012372 shutdown_connections 2024-03-24T00:15:30.108 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:30.106+0000 7f38e07ce740 1 -- 172.21.15.118:0/2119012372 wait complete. 2024-03-24T00:15:30.108 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:30.106+0000 7f38e07ce740 1 -- start start 2024-03-24T00:15:30.108 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:30.106+0000 7f38e07ce740 1 Processor -- start 2024-03-24T00:15:30.111 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:30.107+0000 7f38e07ce740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55e59fd81ba0 0x55e59fd8e170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:15:30.111 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:30.107+0000 7f38e07ce740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55e59fd82590 0x55e59fd8e6b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:15:30.111 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:30.107+0000 7f38e07ce740 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55e59fd87700 0x55e59fd916d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:15:30.111 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:30.107+0000 7f38e07ce740 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e59fd99500 con 0x55e59fd87700 2024-03-24T00:15:30.111 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:30.107+0000 7f38e07ce740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e59fd98ea0 con 0x55e59fd82590 2024-03-24T00:15:30.111 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:30.107+0000 7f38e07ce740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e59fd99b60 con 0x55e59fd81ba0 2024-03-24T00:15:30.111 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:30.107+0000 7f38e07ce740 10 client.0 _ll_register_callbacks cb 0x7f38d0009b30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T00:15:30.111 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:30.107+0000 7f38d6a00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55e59fd81ba0 0x55e59fd8e170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:15:30.111 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:30.107+0000 7f38d6a00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55e59fd81ba0 0x55e59fd8e170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.118:47166/0 (socket says 172.21.15.118:47166) 2024-03-24T00:15:30.113 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:30.107+0000 7f38ce000640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55e59fd82590 0x55e59fd8e6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:15:30.113 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:30.107+0000 7f38d6a00640 1 -- 172.21.15.118:0/3798175009 learned_addr learned my addr 172.21.15.118:0/3798175009 (peer_addr_for_me v2:172.21.15.118:0/0) 2024-03-24T00:15:30.113 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:30.107+0000 7f38d7400640 1 --2- 172.21.15.118:0/3798175009 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55e59fd87700 0x55e59fd916d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:15:30.113 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:30.108+0000 7f38d6a00640 1 -- 172.21.15.118:0/3798175009 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55e59fd82590 msgr2=0x55e59fd8e6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:15:30.113 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:30.108+0000 7f38d6a00640 1 --2- 172.21.15.118:0/3798175009 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55e59fd82590 0x55e59fd8e6b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:15:30.113 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:30.108+0000 7f38d6a00640 1 -- 172.21.15.118:0/3798175009 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55e59fd87700 msgr2=0x55e59fd916d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:15:30.113 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:30.108+0000 7f38d6a00640 1 --2- 172.21.15.118:0/3798175009 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55e59fd87700 0x55e59fd916d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:15:30.113 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:30.108+0000 7f38d6a00640 1 -- 172.21.15.118:0/3798175009 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55e59fd91cc0 con 0x55e59fd81ba0 2024-03-24T00:15:30.114 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:30.108+0000 7f38ce000640 1 --2- 172.21.15.118:0/3798175009 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55e59fd82590 0x55e59fd8e6b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T00:15:30.114 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:30.108+0000 7f38d6a00640 1 --2- 172.21.15.118:0/3798175009 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55e59fd81ba0 0x55e59fd8e170 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7f38c802c170 tx=0x7f38c8005dd0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=cc40cac1b27f026f server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:15:30.114 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:30.108+0000 7f38c4c00640 10 client.0 ms_handle_connect on v2:172.21.15.193:3300/0 2024-03-24T00:15:30.114 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:30.108+0000 7f38c4c00640 1 -- 172.21.15.118:0/3798175009 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f38c804e020 con 0x55e59fd81ba0 2024-03-24T00:15:30.114 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:30.108+0000 7f38c4c00640 1 -- 172.21.15.118:0/3798175009 <== mon.1 v2:172.21.15.193:3300/0 2 ==== osd_map(90..90 src has 1..90) v4 ==== 8277+0+0 (secure 0 0 0) 0x7f38c80046c0 con 0x55e59fd81ba0 2024-03-24T00:15:30.114 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:30.108+0000 7f38e07ce740 1 -- 172.21.15.118:0/3798175009 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f38c009b8f0 con 0x55e59fd81ba0 2024-03-24T00:15:30.114 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:30.108+0000 7f38c4c00640 1 -- 172.21.15.118:0/3798175009 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_get_version_reply(handle=1 version=27) v2 ==== 24+0+0 (secure 0 0 0) 0x7f38c8045040 con 0x55e59fd81ba0 2024-03-24T00:15:30.114 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:30.108+0000 7f38e07ce740 10 client.25252 fetch_fsmap learned FSMap version 27 2024-03-24T00:15:30.114 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:30.108+0000 7f38e07ce740 1 -- 172.21.15.118:0/3798175009 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,fsmap=27,monmap=4+}) v3 -- 0x55e59fd92fc0 con 0x55e59fd81ba0 2024-03-24T00:15:30.114 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:30.109+0000 7f38c4c00640 1 -- 172.21.15.118:0/3798175009 <== mon.1 v2:172.21.15.193:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f38c009b8f0 con 0x55e59fd81ba0 2024-03-24T00:15:30.114 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:30.109+0000 7f38c4c00640 1 -- 172.21.15.118:0/3798175009 <== mon.1 v2:172.21.15.193:3300/0 5 ==== fsmap(e 27) v1 ==== 7934+0+0 (secure 0 0 0) 0x7f38c804a9e0 con 0x55e59fd81ba0 2024-03-24T00:15:30.114 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:30.109+0000 7f38e07ce740 10 client.25252 fetch_fsmap finished waiting for FSMap version 27 2024-03-24T00:15:30.114 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:30.109+0000 7f38e07ce740 10 client.25252 resolve_mds: resolved name 'b' to daemon mds.b 2024-03-24T00:15:30.114 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:30.109+0000 7f38e07ce740 20 client.25252 populate_metadata read hostname 'smithi118' 2024-03-24T00:15:30.114 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:30.109+0000 7f38e07ce740 1 --2- 172.21.15.118:0/3798175009 >> [v2:172.21.15.189:6838/4074484168,v1:172.21.15.189:6839/4074484168] conn(0x55e59fdf7b30 0x55e59fdf3900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:15:30.115 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:30.109+0000 7f38e07ce740 4 client.25252 mds_command: new command op to 24569 tid=0 multi_id=0 [{"prefix": "status"}] 2024-03-24T00:15:30.115 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:30.109+0000 7f38e07ce740 1 -- 172.21.15.118:0/3798175009 --> [v2:172.21.15.189:6838/4074484168,v1:172.21.15.189:6839/4074484168] -- command(tid 0: {"prefix": "status"}) v1 -- 0x55e59fcf41d0 con 0x55e59fdf7b30 2024-03-24T00:15:30.115 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:30.109+0000 7f38ce000640 1 --2- 172.21.15.118:0/3798175009 >> [v2:172.21.15.189:6838/4074484168,v1:172.21.15.189:6839/4074484168] conn(0x55e59fdf7b30 0x55e59fdf3900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:15:30.115 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:30.110+0000 7f38ce000640 1 --2- 172.21.15.118:0/3798175009 >> [v2:172.21.15.189:6838/4074484168,v1:172.21.15.189:6839/4074484168] conn(0x55e59fdf7b30 0x55e59fdf3900 crc :-1 s=READY pgs=30 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=6060945b47d71e68 server_cookie=50528c9af5d14ecb in_seq=0 out_seq=0 2024-03-24T00:15:30.115 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:30.110+0000 7f38c4c00640 10 client.25252 ms_handle_connect on v2:172.21.15.189:6838/4074484168 2024-03-24T00:15:30.115 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:30.110+0000 7f38c4c00640 1 -- 172.21.15.118:0/3798175009 <== mds.0 v2:172.21.15.189:6838/4074484168 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+327 (crc 0 0 0) 0x55e59fcf41d0 con 0x55e59fdf7b30 2024-03-24T00:15:30.115 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:30.110+0000 7f38c4c00640 10 client.25252 handle_command_reply: tid=0 2024-03-24T00:15:30.115 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:30.110+0000 7f38e07ce740 1 client.25252 shutdown 2024-03-24T00:15:30.115 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:30.110+0000 7f38e07ce740 1 -- 172.21.15.118:0/3798175009 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55e59fd81ba0 msgr2=0x55e59fd8e170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:15:30.115 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:30.110+0000 7f38e07ce740 1 --2- 172.21.15.118:0/3798175009 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55e59fd81ba0 0x55e59fd8e170 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 crypto rx=0x7f38c802c170 tx=0x7f38c8005dd0 comp rx=0 tx=0).stop 2024-03-24T00:15:30.115 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:30.110+0000 7f38e07ce740 1 -- 172.21.15.118:0/3798175009 shutdown_connections 2024-03-24T00:15:30.115 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:30.110+0000 7f38e07ce740 1 --2- 172.21.15.118:0/3798175009 >> [v2:172.21.15.189:6838/4074484168,v1:172.21.15.189:6839/4074484168] conn(0x55e59fdf7b30 0x55e59fdf3900 crc :-1 s=READY pgs=30 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:15:30.115 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:30.110+0000 7f38e07ce740 1 --2- 172.21.15.118:0/3798175009 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55e59fd87700 0x55e59fd916d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:15:30.115 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:30.110+0000 7f38e07ce740 1 --2- 172.21.15.118:0/3798175009 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55e59fd82590 0x55e59fd8e6b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:15:30.115 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:30.110+0000 7f38c4c00640 0 client.25252 ms_handle_reset on v2:172.21.15.189:6838/4074484168 2024-03-24T00:15:30.116 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:30.110+0000 7f38e07ce740 1 --2- 172.21.15.118:0/3798175009 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55e59fd81ba0 0x55e59fd8e170 unknown :-1 s=CLOSED pgs=175 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:15:30.116 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:30.110+0000 7f38e07ce740 1 -- 172.21.15.118:0/3798175009 >> 172.21.15.118:0/3798175009 conn(0x55e59fd7d110 msgr2=0x55e59fd82f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:15:30.116 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:30.110+0000 7f38e07ce740 1 -- 172.21.15.118:0/3798175009 shutdown_connections 2024-03-24T00:15:30.116 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:30.110+0000 7f38e07ce740 1 -- 172.21.15.118:0/3798175009 wait complete. 2024-03-24T00:15:30.116 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:30.110+0000 7f38e07ce740 20 client.25252 trim_cache size 0 max 16384 2024-03-24T00:15:30.116 INFO:teuthology.orchestra.run.smithi118.stdout:{ 2024-03-24T00:15:30.116 INFO:teuthology.orchestra.run.smithi118.stdout: "cluster_fsid": "d79a5420-e971-11ee-95cd-87774f69a715", 2024-03-24T00:15:30.116 INFO:teuthology.orchestra.run.smithi118.stdout: "whoami": 0, 2024-03-24T00:15:30.116 INFO:teuthology.orchestra.run.smithi118.stdout: "id": 24569, 2024-03-24T00:15:30.116 INFO:teuthology.orchestra.run.smithi118.stdout: "want_state": "up:active", 2024-03-24T00:15:30.116 INFO:teuthology.orchestra.run.smithi118.stdout: "state": "up:active", 2024-03-24T00:15:30.116 INFO:teuthology.orchestra.run.smithi118.stdout: "fs_name": "cephfs", 2024-03-24T00:15:30.117 INFO:teuthology.orchestra.run.smithi118.stdout: "rank_uptime": 147.772265428, 2024-03-24T00:15:30.117 INFO:teuthology.orchestra.run.smithi118.stdout: "mdsmap_epoch": 27, 2024-03-24T00:15:30.117 INFO:teuthology.orchestra.run.smithi118.stdout: "osdmap_epoch": 90, 2024-03-24T00:15:30.117 INFO:teuthology.orchestra.run.smithi118.stdout: "osdmap_epoch_barrier": 86, 2024-03-24T00:15:30.117 INFO:teuthology.orchestra.run.smithi118.stdout: "uptime": 154.97726860200001 2024-03-24T00:15:30.117 INFO:teuthology.orchestra.run.smithi118.stdout:} 2024-03-24T00:15:30.118 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:30.111+0000 7f38dda00640 1 -- 172.21.15.118:0/805042186 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7f38bc0a2800 msgr2=0x7f38bc0a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:15:30.118 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:30.111+0000 7f38dda00640 1 --2- 172.21.15.118:0/805042186 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7f38bc0a2800 0x7f38bc0a4cc0 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x55e59fd82df0 tx=0x7f38d0009500 comp rx=0 tx=0).stop 2024-03-24T00:15:30.118 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:30.111+0000 7f38dda00640 1 -- 172.21.15.118:0/805042186 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f38d8104ad0 msgr2=0x7f38d819a180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:15:30.118 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:30.111+0000 7f38dda00640 1 --2- 172.21.15.118:0/805042186 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f38d8104ad0 0x7f38d819a180 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7f38c000e9f0 tx=0x7f38c000eeb0 comp rx=0 tx=0).stop 2024-03-24T00:15:30.118 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:30.111+0000 7f38dda00640 1 -- 172.21.15.118:0/805042186 shutdown_connections 2024-03-24T00:15:30.118 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:30.111+0000 7f38dda00640 1 --2- 172.21.15.118:0/805042186 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7f38bc0a2800 0x7f38bc0a4cc0 unknown :-1 s=CLOSED pgs=247 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:15:30.118 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:30.111+0000 7f38dda00640 1 --2- 172.21.15.118:0/805042186 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f38d8105490 0x7f38d81a11b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:15:30.118 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:30.111+0000 7f38dda00640 1 --2- 172.21.15.118:0/805042186 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f38d8104ad0 0x7f38d819a180 unknown :-1 s=CLOSED pgs=183 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:15:30.118 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:30.111+0000 7f38dda00640 1 --2- 172.21.15.118:0/805042186 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f38d81038e0 0x7f38d8199c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:15:30.118 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:30.111+0000 7f38dda00640 1 -- 172.21.15.118:0/805042186 >> 172.21.15.118:0/805042186 conn(0x7f38d80fee50 msgr2=0x7f38d8107df0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:15:30.118 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:30.111+0000 7f38dda00640 1 -- 172.21.15.118:0/805042186 shutdown_connections 2024-03-24T00:15:30.118 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:30.111+0000 7f38dda00640 1 -- 172.21.15.118:0/805042186 wait complete. 2024-03-24T00:15:30.133 INFO:teuthology.run_tasks:Running task kclient... 2024-03-24T00:15:30.155 INFO:tasks.kclient:Mounting kernel clients... 2024-03-24T00:15:30.156 INFO:tasks.kclient:config is {'client.0': {}} 2024-03-24T00:15:30.156 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T00:15:30.156 DEBUG:teuthology.orchestra.run.smithi118:> ip netns list 2024-03-24T00:15:30.203 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:15:29 smithi189 ceph-mon[36187]: pgmap v392: 97 pgs: 97 active+clean; 645 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 11 op/s 2024-03-24T00:15:30.205 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T00:15:30.205 DEBUG:teuthology.orchestra.run.smithi118:> sudo ip link delete ceph-brx 2024-03-24T00:15:30.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:15:29 smithi193 ceph-mon[36878]: pgmap v392: 97 pgs: 97 active+clean; 645 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 5.8 KiB/s rd, 11 op/s 2024-03-24T00:15:30.292 INFO:teuthology.orchestra.run.smithi118.stderr:Cannot find device "ceph-brx" 2024-03-24T00:15:30.296 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T00:15:30.297 INFO:tasks.kclient:client.0 config is {'mntopts': ['ms_mode=crc', 'nowsync'], 'syntax': 'v2', 'mount_subvol_num': 1} 2024-03-24T00:15:30.297 INFO:tasks.cephfs.mount:cephfs_mntpt = None 2024-03-24T00:15:30.297 INFO:tasks.cephfs.mount:hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-03-24T00:15:30.297 DEBUG:teuthology.orchestra.run.smithi118:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 shell --fsid d79a5420-e971-11ee-95cd-87774f69a715 -- ceph fs ls 2024-03-24T00:15:30.731 INFO:teuthology.orchestra.run.smithi118.stderr:Inferring config /var/lib/ceph/d79a5420-e971-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T00:15:32.772 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:15:32 smithi118 ceph-mon[31535]: pgmap v393: 97 pgs: 97 active+clean; 645 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-24T00:15:33.091 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:33.090+0000 7fbd08200700 1 -- 172.21.15.118:0/1902838660 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fbd00101090 msgr2=0x7fbd001014f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:15:33.092 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:33.090+0000 7fbd08200700 1 --2- 172.21.15.118:0/1902838660 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fbd00101090 0x7fbd001014f0 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7fbcec009990 tx=0x7fbcec009ca0 comp rx=0 tx=0).stop 2024-03-24T00:15:33.092 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:33.090+0000 7fbd08200700 1 -- 172.21.15.118:0/1902838660 shutdown_connections 2024-03-24T00:15:33.092 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:33.091+0000 7fbd08200700 1 --2- 172.21.15.118:0/1902838660 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fbd00101a30 0x7fbd0010df10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:15:33.092 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:33.091+0000 7fbd08200700 1 --2- 172.21.15.118:0/1902838660 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fbd001070d0 0x7fbd001074b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:15:33.092 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:33.091+0000 7fbd08200700 1 --2- 172.21.15.118:0/1902838660 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fbd00101090 0x7fbd001014f0 unknown :-1 s=CLOSED pgs=185 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:15:33.092 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:33.091+0000 7fbd08200700 1 -- 172.21.15.118:0/1902838660 >> 172.21.15.118:0/1902838660 conn(0x7fbd00077fc0 msgr2=0x7fbd000783d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:15:33.092 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:33.091+0000 7fbd08200700 1 -- 172.21.15.118:0/1902838660 shutdown_connections 2024-03-24T00:15:33.092 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:33.091+0000 7fbd08200700 1 -- 172.21.15.118:0/1902838660 wait complete. 2024-03-24T00:15:33.093 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:33.092+0000 7fbd08200700 1 Processor -- start 2024-03-24T00:15:33.093 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:33.093+0000 7fbd08200700 1 -- start start 2024-03-24T00:15:33.094 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:33.093+0000 7fbd08200700 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fbd00101090 0x7fbd0019e630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:15:33.094 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:33.093+0000 7fbd08200700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fbd00101a30 0x7fbd0019eb70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:15:33.095 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:33.093+0000 7fbd08200700 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fbd001070d0 0x7fbd001987b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:15:33.095 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:33.093+0000 7fbd08200700 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbd00111380 con 0x7fbd00101090 2024-03-24T00:15:33.095 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:33.093+0000 7fbd08200700 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbd00111200 con 0x7fbd00101a30 2024-03-24T00:15:33.095 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:33.093+0000 7fbd08200700 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbd00111500 con 0x7fbd001070d0 2024-03-24T00:15:33.095 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:33.094+0000 7fbd04e00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fbd00101a30 0x7fbd0019eb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:15:33.095 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:33.094+0000 7fbd04e00700 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fbd00101a30 0x7fbd0019eb70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.118:46860/0 (socket says 172.21.15.118:46860) 2024-03-24T00:15:33.095 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:33.094+0000 7fbd04e00700 1 -- 172.21.15.118:0/3868909919 learned_addr learned my addr 172.21.15.118:0/3868909919 (peer_addr_for_me v2:172.21.15.118:0/0) 2024-03-24T00:15:33.095 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:33.094+0000 7fbd05800700 1 --2- 172.21.15.118:0/3868909919 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fbd00101090 0x7fbd0019e630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:15:33.095 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:33.094+0000 7fbd06200700 1 --2- 172.21.15.118:0/3868909919 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fbd001070d0 0x7fbd001987b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:15:33.095 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:33.094+0000 7fbd04e00700 1 -- 172.21.15.118:0/3868909919 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fbd001070d0 msgr2=0x7fbd001987b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:15:33.095 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:33.094+0000 7fbd04e00700 1 --2- 172.21.15.118:0/3868909919 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fbd001070d0 0x7fbd001987b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:15:33.096 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:33.094+0000 7fbd04e00700 1 -- 172.21.15.118:0/3868909919 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fbd00101090 msgr2=0x7fbd0019e630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:15:33.096 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:33.094+0000 7fbd04e00700 1 --2- 172.21.15.118:0/3868909919 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fbd00101090 0x7fbd0019e630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:15:33.096 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:33.094+0000 7fbd04e00700 1 -- 172.21.15.118:0/3868909919 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbd00198f90 con 0x7fbd00101a30 2024-03-24T00:15:33.096 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:33.094+0000 7fbd04e00700 1 --2- 172.21.15.118:0/3868909919 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fbd00101a30 0x7fbd0019eb70 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7fbcec000c00 tx=0x7fbcec00bab0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:15:33.097 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:33.095+0000 7fbcfe000700 1 -- 172.21.15.118:0/3868909919 <== mon.2 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbcec027070 con 0x7fbd00101a30 2024-03-24T00:15:33.097 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:33.095+0000 7fbcfe000700 1 -- 172.21.15.118:0/3868909919 <== mon.2 v2:172.21.15.189:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fbcec003d70 con 0x7fbd00101a30 2024-03-24T00:15:33.097 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:33.095+0000 7fbd08200700 1 -- 172.21.15.118:0/3868909919 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbd00199220 con 0x7fbd00101a30 2024-03-24T00:15:33.097 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:33.095+0000 7fbcfe000700 1 -- 172.21.15.118:0/3868909919 <== mon.2 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbcec00f460 con 0x7fbd00101a30 2024-03-24T00:15:33.097 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:33.095+0000 7fbd08200700 1 -- 172.21.15.118:0/3868909919 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbd00102c70 con 0x7fbd00101a30 2024-03-24T00:15:33.097 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:33.096+0000 7fbd08200700 1 -- 172.21.15.118:0/3868909919 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbd00102780 con 0x7fbd00101a30 2024-03-24T00:15:33.098 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:33.098+0000 7fbcfe000700 1 -- 172.21.15.118:0/3868909919 <== mon.2 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 18) v1 ==== 137291+0+0 (secure 0 0 0) 0x7fbcec02a070 con 0x7fbd00101a30 2024-03-24T00:15:33.100 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:33.099+0000 7fbcfe000700 1 --2- 172.21.15.118:0/3868909919 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7fbce80a2890 0x7fbce80a4d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:15:33.100 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:33.099+0000 7fbcfe000700 1 -- 172.21.15.118:0/3868909919 <== mon.2 v2:172.21.15.189:3300/0 5 ==== osd_map(90..90 src has 1..90) v4 ==== 8277+0+0 (secure 0 0 0) 0x7fbcec0cf9b0 con 0x7fbd00101a30 2024-03-24T00:15:33.100 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:33.099+0000 7fbd05800700 1 --2- 172.21.15.118:0/3868909919 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7fbce80a2890 0x7fbce80a4d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:15:33.100 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:33.100+0000 7fbd05800700 1 --2- 172.21.15.118:0/3868909919 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7fbce80a2890 0x7fbce80a4d50 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7fbcf4009be0 tx=0x7fbcf4009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:15:33.106 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:33.106+0000 7fbcfe000700 1 -- 172.21.15.118:0/3868909919 <== mon.2 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7fbcec05e740 con 0x7fbd00101a30 2024-03-24T00:15:33.203 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:15:32 smithi189 ceph-mon[36187]: pgmap v393: 97 pgs: 97 active+clean; 645 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-24T00:15:33.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:15:32 smithi193 ceph-mon[36878]: pgmap v393: 97 pgs: 97 active+clean; 645 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 8 op/s 2024-03-24T00:15:33.407 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:33.406+0000 7fbd08200700 1 -- 172.21.15.118:0/3868909919 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "fs ls"} v 0) v1 -- 0x7fbd00199ce0 con 0x7fbd00101a30 2024-03-24T00:15:33.407 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:33.407+0000 7fbcfe000700 1 -- 172.21.15.118:0/3868909919 <== mon.2 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v27) v1 ==== 53+0+73 (secure 0 0 0) 0x7fbcec013070 con 0x7fbd00101a30 2024-03-24T00:15:33.408 INFO:teuthology.orchestra.run.smithi118.stdout:name: cephfs, metadata pool: cephfs_metadata, data pools: [cephfs_data ] 2024-03-24T00:15:33.410 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:33.409+0000 7fbd08200700 1 -- 172.21.15.118:0/3868909919 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7fbce80a2890 msgr2=0x7fbce80a4d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:15:33.410 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:33.409+0000 7fbd08200700 1 --2- 172.21.15.118:0/3868909919 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7fbce80a2890 0x7fbce80a4d50 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7fbcf4009be0 tx=0x7fbcf4009260 comp rx=0 tx=0).stop 2024-03-24T00:15:33.410 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:33.409+0000 7fbd08200700 1 -- 172.21.15.118:0/3868909919 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fbd00101a30 msgr2=0x7fbd0019eb70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:15:33.411 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:33.409+0000 7fbd08200700 1 --2- 172.21.15.118:0/3868909919 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fbd00101a30 0x7fbd0019eb70 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7fbcec000c00 tx=0x7fbcec00bab0 comp rx=0 tx=0).stop 2024-03-24T00:15:33.411 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:33.410+0000 7fbd08200700 1 -- 172.21.15.118:0/3868909919 shutdown_connections 2024-03-24T00:15:33.411 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:33.410+0000 7fbd08200700 1 --2- 172.21.15.118:0/3868909919 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7fbce80a2890 0x7fbce80a4d50 unknown :-1 s=CLOSED pgs=248 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:15:33.411 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:33.410+0000 7fbd08200700 1 --2- 172.21.15.118:0/3868909919 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fbd001070d0 0x7fbd001987b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:15:33.411 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:33.410+0000 7fbd08200700 1 --2- 172.21.15.118:0/3868909919 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fbd00101090 0x7fbd0019e630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:15:33.411 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:33.410+0000 7fbd08200700 1 --2- 172.21.15.118:0/3868909919 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fbd00101a30 0x7fbd0019eb70 unknown :-1 s=CLOSED pgs=186 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:15:33.411 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:33.410+0000 7fbd08200700 1 -- 172.21.15.118:0/3868909919 >> 172.21.15.118:0/3868909919 conn(0x7fbd00077fc0 msgr2=0x7fbd00069fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:15:33.411 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:33.410+0000 7fbd08200700 1 -- 172.21.15.118:0/3868909919 shutdown_connections 2024-03-24T00:15:33.412 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:33.410+0000 7fbd08200700 1 -- 172.21.15.118:0/3868909919 wait complete. 2024-03-24T00:15:34.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:15:33 smithi118 ceph-mon[31535]: from='client.? 172.21.15.118:0/3868909919' entity='client.admin' cmd=[{"prefix": "fs ls"}]: dispatch 2024-03-24T00:15:34.206 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:15:33 smithi189 ceph-mon[36187]: from='client.? 172.21.15.118:0/3868909919' entity='client.admin' cmd=[{"prefix": "fs ls"}]: dispatch 2024-03-24T00:15:34.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:15:33 smithi193 ceph-mon[36878]: from='client.? 172.21.15.118:0/3868909919' entity='client.admin' cmd=[{"prefix": "fs ls"}]: dispatch 2024-03-24T00:15:34.254 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2024-03-24T00:15:34.255 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-03-24T00:15:34.255 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi118.front.sepia.ceph.com 2024-03-24T00:15:34.255 INFO:tasks.cephfs.mount:self.client.name = client.0 2024-03-24T00:15:34.255 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-03-24T00:15:34.255 INFO:tasks.cephfs.mount:self.cephfs_name = None 2024-03-24T00:15:34.255 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2024-03-24T00:15:34.255 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-03-24T00:15:34.255 INFO:tasks.cephfs.mount:Setting the 'None' netns for '/home/ubuntu/cephtest/mnt.0' 2024-03-24T00:15:34.255 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T00:15:34.255 DEBUG:teuthology.orchestra.run.smithi118:> ip addr 2024-03-24T00:15:34.309 INFO:teuthology.orchestra.run.smithi118.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-03-24T00:15:34.309 INFO:teuthology.orchestra.run.smithi118.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-03-24T00:15:34.309 INFO:teuthology.orchestra.run.smithi118.stdout: inet 127.0.0.1/8 scope host lo 2024-03-24T00:15:34.309 INFO:teuthology.orchestra.run.smithi118.stdout: valid_lft forever preferred_lft forever 2024-03-24T00:15:34.309 INFO:teuthology.orchestra.run.smithi118.stdout: inet6 ::1/128 scope host 2024-03-24T00:15:34.309 INFO:teuthology.orchestra.run.smithi118.stdout: valid_lft forever preferred_lft forever 2024-03-24T00:15:34.309 INFO:teuthology.orchestra.run.smithi118.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-24T00:15:34.309 INFO:teuthology.orchestra.run.smithi118.stdout: link/ether 0c:c4:7a:d8:a4:cc brd ff:ff:ff:ff:ff:ff 2024-03-24T00:15:34.309 INFO:teuthology.orchestra.run.smithi118.stdout: altname enp7s0f0 2024-03-24T00:15:34.309 INFO:teuthology.orchestra.run.smithi118.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-24T00:15:34.310 INFO:teuthology.orchestra.run.smithi118.stdout: link/ether 0c:c4:7a:d8:a4:cd brd ff:ff:ff:ff:ff:ff 2024-03-24T00:15:34.310 INFO:teuthology.orchestra.run.smithi118.stdout: altname enp7s0f1 2024-03-24T00:15:34.310 INFO:teuthology.orchestra.run.smithi118.stdout:4: enp3s0f0: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-24T00:15:34.310 INFO:teuthology.orchestra.run.smithi118.stdout: link/ether 0c:c4:7a:8f:d7:24 brd ff:ff:ff:ff:ff:ff 2024-03-24T00:15:34.310 INFO:teuthology.orchestra.run.smithi118.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2024-03-24T00:15:34.310 INFO:teuthology.orchestra.run.smithi118.stdout: link/ether 0c:c4:7a:8f:d7:25 brd ff:ff:ff:ff:ff:ff 2024-03-24T00:15:34.310 INFO:teuthology.orchestra.run.smithi118.stdout: inet 172.21.15.118/20 brd 172.21.15.255 scope global dynamic noprefixroute enp3s0f1 2024-03-24T00:15:34.310 INFO:teuthology.orchestra.run.smithi118.stdout: valid_lft 41911sec preferred_lft 41911sec 2024-03-24T00:15:34.310 INFO:teuthology.orchestra.run.smithi118.stdout: inet6 fe80::ec4:7aff:fe8f:d725/64 scope link 2024-03-24T00:15:34.310 INFO:teuthology.orchestra.run.smithi118.stdout: valid_lft forever preferred_lft forever 2024-03-24T00:15:34.311 INFO:tasks.cephfs.mount:Setuping the 'ceph-brx' with 192.168.255.254/16 2024-03-24T00:15:34.311 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T00:15:34.312 DEBUG:teuthology.orchestra.run.smithi118:> (cd / && exec bash -c ' 2024-03-24T00:15:34.312 DEBUG:teuthology.orchestra.run.smithi118:> set -e 2024-03-24T00:15:34.312 DEBUG:teuthology.orchestra.run.smithi118:> sudo ip link add name ceph-brx type bridge 2024-03-24T00:15:34.312 DEBUG:teuthology.orchestra.run.smithi118:> sudo ip addr flush dev ceph-brx 2024-03-24T00:15:34.312 DEBUG:teuthology.orchestra.run.smithi118:> sudo ip link set ceph-brx up 2024-03-24T00:15:34.312 DEBUG:teuthology.orchestra.run.smithi118:> sudo ip addr add 192.168.255.254/16 brd 192.168.255.255 dev ceph-brx 2024-03-24T00:15:34.312 DEBUG:teuthology.orchestra.run.smithi118:> ') 2024-03-24T00:15:34.734 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T00:15:34.735 DEBUG:teuthology.orchestra.run.smithi118:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-03-24T00:15:34.821 INFO:teuthology.orchestra.run.smithi118.stdout:1 2024-03-24T00:15:34.826 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T00:15:34.826 DEBUG:teuthology.orchestra.run.smithi118:> route 2024-03-24T00:15:34.904 INFO:teuthology.orchestra.run.smithi118.stdout:Kernel IP routing table 2024-03-24T00:15:34.904 INFO:teuthology.orchestra.run.smithi118.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-03-24T00:15:34.904 INFO:teuthology.orchestra.run.smithi118.stdout:default _gateway 0.0.0.0 UG 100 0 0 enp3s0f1 2024-03-24T00:15:34.904 INFO:teuthology.orchestra.run.smithi118.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 enp3s0f1 2024-03-24T00:15:34.904 INFO:teuthology.orchestra.run.smithi118.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-03-24T00:15:34.905 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T00:15:34.905 DEBUG:teuthology.orchestra.run.smithi118:> (cd / && exec bash -c ' 2024-03-24T00:15:34.905 DEBUG:teuthology.orchestra.run.smithi118:> set -e 2024-03-24T00:15:34.905 DEBUG:teuthology.orchestra.run.smithi118:> sudo iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT 2024-03-24T00:15:34.905 DEBUG:teuthology.orchestra.run.smithi118:> sudo iptables -A FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT 2024-03-24T00:15:34.905 DEBUG:teuthology.orchestra.run.smithi118:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE 2024-03-24T00:15:34.905 DEBUG:teuthology.orchestra.run.smithi118:> ') 2024-03-24T00:15:34.945 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:15:34 smithi118 ceph-mon[31535]: pgmap v394: 97 pgs: 97 active+clean; 645 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-03-24T00:15:35.203 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:15:34 smithi189 ceph-mon[36187]: pgmap v394: 97 pgs: 97 active+clean; 645 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-03-24T00:15:35.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:15:34 smithi193 ceph-mon[36878]: pgmap v394: 97 pgs: 97 active+clean; 645 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 9 op/s 2024-03-24T00:15:35.612 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T00:15:35.613 DEBUG:teuthology.orchestra.run.smithi118:> ip netns list 2024-03-24T00:15:35.710 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T00:15:35.711 DEBUG:teuthology.orchestra.run.smithi118:> ip netns list-id 2024-03-24T00:15:35.759 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T00:15:35.759 DEBUG:teuthology.orchestra.run.smithi118:> (cd / && exec bash -c ' 2024-03-24T00:15:35.759 DEBUG:teuthology.orchestra.run.smithi118:> set -e 2024-03-24T00:15:35.759 DEBUG:teuthology.orchestra.run.smithi118:> sudo ip netns add ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-24T00:15:35.759 DEBUG:teuthology.orchestra.run.smithi118:> sudo ip netns set ceph-ns--home-ubuntu-cephtest-mnt.0 0 2024-03-24T00:15:35.760 DEBUG:teuthology.orchestra.run.smithi118:> ') 2024-03-24T00:15:35.885 INFO:tasks.cephfs.mount:Setuping the netns 'ceph-ns--home-ubuntu-cephtest-mnt.0' with 192.168.0.1/16 2024-03-24T00:15:35.885 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T00:15:35.885 DEBUG:teuthology.orchestra.run.smithi118:> (cd / && exec bash -c ' 2024-03-24T00:15:35.885 DEBUG:teuthology.orchestra.run.smithi118:> set -e 2024-03-24T00:15:35.885 DEBUG:teuthology.orchestra.run.smithi118:> sudo ip link add veth0 netns ceph-ns--home-ubuntu-cephtest-mnt.0 type veth peer name brx.0 2024-03-24T00:15:35.886 DEBUG:teuthology.orchestra.run.smithi118:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip addr add 192.168.0.1/16 brd 192.168.255.255 dev veth0 2024-03-24T00:15:35.886 DEBUG:teuthology.orchestra.run.smithi118:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip link set veth0 up 2024-03-24T00:15:35.886 DEBUG:teuthology.orchestra.run.smithi118:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip link set lo up 2024-03-24T00:15:35.886 DEBUG:teuthology.orchestra.run.smithi118:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip route add default via 192.168.255.254 2024-03-24T00:15:35.886 DEBUG:teuthology.orchestra.run.smithi118:> ') 2024-03-24T00:15:36.178 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T00:15:36.178 DEBUG:teuthology.orchestra.run.smithi118:> (cd / && exec bash -c ' 2024-03-24T00:15:36.178 DEBUG:teuthology.orchestra.run.smithi118:> set -e 2024-03-24T00:15:36.178 DEBUG:teuthology.orchestra.run.smithi118:> sudo ip link set brx.0 up 2024-03-24T00:15:36.178 DEBUG:teuthology.orchestra.run.smithi118:> sudo ip link set dev brx.0 master ceph-brx 2024-03-24T00:15:36.178 DEBUG:teuthology.orchestra.run.smithi118:> ') 2024-03-24T00:15:36.328 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-03-24T00:15:36.328 DEBUG:teuthology.orchestra.run.smithi118:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2024-03-24T00:15:36.370 INFO:teuthology.orchestra.run.smithi118.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2024-03-24T00:15:36.370 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-03-24T00:15:36.370 DEBUG:teuthology.orchestra.run.smithi118:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2024-03-24T00:15:36.467 WARNING:tasks.cephfs.mount:You cannot specify both: cephfs_mntpt and mount_subvol_num 2024-03-24T00:15:36.467 INFO:tasks.cephfs.mount:Mounting subvol 1 for now 2024-03-24T00:15:36.467 INFO:tasks.cephfs.kernel_mount:mounting using device: 0@.cephfs=/volumes/qa/sv_1/8e57c0ba-0773-4e6b-91f6-45f3c40de69d 2024-03-24T00:15:36.467 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T00:15:36.467 DEBUG:teuthology.orchestra.run.smithi118:> sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage /bin/mount -t ceph 0@.cephfs=/volumes/qa/sv_1/8e57c0ba-0773-4e6b-91f6-45f3c40de69d /home/ubuntu/cephtest/mnt.0 -v -o norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,ms_mode=crc,nowsync,nofallback 2024-03-24T00:15:36.590 INFO:teuthology.orchestra.run.smithi118.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,ms_mode=crc,nowsync,nofallback 2024-03-24T00:15:36.590 INFO:teuthology.orchestra.run.smithi118.stdout:mount.ceph: options "norequire_active_mds,norbytes,ms_mode=crc,nowsync". 2024-03-24T00:15:36.827 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:15:36 smithi118 ceph-mon[31535]: pgmap v395: 97 pgs: 97 active+clean; 645 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 10 op/s 2024-03-24T00:15:37.203 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:15:36 smithi189 ceph-mon[36187]: pgmap v395: 97 pgs: 97 active+clean; 645 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 10 op/s 2024-03-24T00:15:37.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:15:36 smithi193 ceph-mon[36878]: pgmap v395: 97 pgs: 97 active+clean; 645 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 10 op/s 2024-03-24T00:15:37.269 INFO:teuthology.orchestra.run.smithi118.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,ms_mode=crc,nowsync,nofallback 2024-03-24T00:15:37.270 INFO:teuthology.orchestra.run.smithi118.stdout:mount.ceph: options "norequire_active_mds,norbytes,ms_mode=crc,nowsync". 2024-03-24T00:15:37.270 INFO:teuthology.orchestra.run.smithi118.stdout:mount.ceph: resolved to: "172.21.15.118:3300,172.21.15.189:3300,172.21.15.193:3300" 2024-03-24T00:15:37.270 INFO:teuthology.orchestra.run.smithi118.stdout:mount.ceph: trying mount with new device syntax: 0@d79a5420-e971-11ee-95cd-87774f69a715.cephfs=/volumes/qa/sv_1/8e57c0ba-0773-4e6b-91f6-45f3c40de69d 2024-03-24T00:15:37.270 INFO:teuthology.orchestra.run.smithi118.stdout:mount.ceph: options "norequire_active_mds,norbytes,ms_mode=crc,nowsync,name=0,key=0,mon_addr=172.21.15.118:3300/172.21.15.189:3300/172.21.15.193:3300" will pass to kernel 2024-03-24T00:15:37.274 INFO:teuthology.orchestra.run.smithi118.stdout:mount: /home/ubuntu/cephtest/mnt.0 does not contain SELinux labels. 2024-03-24T00:15:37.274 INFO:teuthology.orchestra.run.smithi118.stdout: You just mounted a file system that supports labels which does not 2024-03-24T00:15:37.274 INFO:teuthology.orchestra.run.smithi118.stdout: contain labels, onto an SELinux box. It is likely that confined 2024-03-24T00:15:37.275 INFO:teuthology.orchestra.run.smithi118.stdout: applications will generate AVC messages and not be allowed access to 2024-03-24T00:15:37.275 INFO:teuthology.orchestra.run.smithi118.stdout: this file system. For more details see restorecon(8) and mount(8). 2024-03-24T00:15:37.279 INFO:tasks.cephfs.kernel_mount:mount command passed 2024-03-24T00:15:37.279 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T00:15:37.279 DEBUG:teuthology.orchestra.run.smithi118:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2024-03-24T00:15:37.375 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T00:15:37.375 DEBUG:teuthology.orchestra.run.smithi118:> (cd /home/ubuntu/cephtest/mnt.0 && exec bash -c 'getfattr --only-values -n ceph.client_id .') 2024-03-24T00:15:37.425 INFO:teuthology.orchestra.run.smithi118.stdout:client15159 2024-03-24T00:15:37.425 DEBUG:teuthology.orchestra.run.smithi118:> set -ex 2024-03-24T00:15:37.425 DEBUG:teuthology.orchestra.run.smithi118:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-24T00:15:37.475 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T00:15:37.475 DEBUG:teuthology.orchestra.run.smithi118:> (cd /home/ubuntu/cephtest/mnt.0 && exec bash -c 'getfattr --only-values -n ceph.client_id .') 2024-03-24T00:15:37.570 INFO:teuthology.orchestra.run.smithi118.stdout:client15159 2024-03-24T00:15:37.570 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T00:15:37.570 DEBUG:teuthology.orchestra.run.smithi118:> (cd /home/ubuntu/cephtest/mnt.0 && exec bash -c 'getfattr --only-values -n ceph.client_id .') 2024-03-24T00:15:37.616 INFO:teuthology.orchestra.run.smithi118.stdout:client15159 2024-03-24T00:15:37.617 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T00:15:37.617 DEBUG:teuthology.orchestra.run.smithi118:> (cd /home/ubuntu/cephtest/mnt.0 && exec bash -c 'sudo dd if=/sys/kernel/debug/ceph/d79a5420-e971-11ee-95cd-87774f69a715.client15159/status') 2024-03-24T00:15:37.744 INFO:teuthology.orchestra.run.smithi118.stdout:instance: client.15159 (3)192.168.0.1:0/3463739313 2024-03-24T00:15:37.744 INFO:teuthology.orchestra.run.smithi118.stdout:blocklisted: false 2024-03-24T00:15:37.744 INFO:teuthology.orchestra.run.smithi118.stderr:0+1 records in 2024-03-24T00:15:37.744 INFO:teuthology.orchestra.run.smithi118.stderr:0+1 records out 2024-03-24T00:15:37.744 INFO:teuthology.orchestra.run.smithi118.stderr:70 bytes copied, 6.4389e-05 s, 1.1 MB/s 2024-03-24T00:15:37.748 INFO:teuthology.run_tasks:Running task exec... 2024-03-24T00:15:37.758 INFO:teuthology.task.exec:Executing custom commands... 2024-03-24T00:15:37.758 INFO:teuthology.task.exec:Running commands on role mon.a host ubuntu@smithi118.front.sepia.ceph.com 2024-03-24T00:15:37.758 DEBUG:teuthology.orchestra.run.smithi118:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph fs set cephfs balance_automate false' 2024-03-24T00:15:37.979 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:37.979+0000 7f897e200640 1 -- 172.21.15.118:0/3337559325 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8978074080 msgr2=0x7f89780744e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:15:37.979 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:37.979+0000 7f897e200640 1 --2- 172.21.15.118:0/3337559325 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8978074080 0x7f89780744e0 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7f8964009900 tx=0x7f896402bcc0 comp rx=0 tx=0).stop 2024-03-24T00:15:37.979 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:37.979+0000 7f897e200640 1 -- 172.21.15.118:0/3337559325 shutdown_connections 2024-03-24T00:15:37.979 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:37.979+0000 7f897e200640 1 --2- 172.21.15.118:0/3337559325 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f8978109580 0x7f897810ba10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:15:37.979 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:37.979+0000 7f897e200640 1 --2- 172.21.15.118:0/3337559325 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8978074080 0x7f89780744e0 unknown :-1 s=CLOSED pgs=187 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:15:37.979 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:37.979+0000 7f897e200640 1 --2- 172.21.15.118:0/3337559325 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f8978075660 0x7f8978073ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:15:37.979 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:37.979+0000 7f897e200640 1 -- 172.21.15.118:0/3337559325 >> 172.21.15.118:0/3337559325 conn(0x7f89780fcc60 msgr2=0x7f89780ff080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:15:37.979 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:37.979+0000 7f897e200640 1 -- 172.21.15.118:0/3337559325 shutdown_connections 2024-03-24T00:15:37.979 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:37.979+0000 7f897e200640 1 -- 172.21.15.118:0/3337559325 wait complete. 2024-03-24T00:15:37.981 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:37.980+0000 7f897e200640 1 Processor -- start 2024-03-24T00:15:37.981 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:37.980+0000 7f897e200640 1 -- start start 2024-03-24T00:15:37.981 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:37.981+0000 7f897e200640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f8978074080 0x7f8978104a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:15:37.981 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:37.981+0000 7f897e200640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8978075660 0x7f8978104fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:15:37.981 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:37.981+0000 7f897e200640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f8978109580 0x7f89781015e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:15:37.981 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:37.981+0000 7f897e200640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f897810f5a0 con 0x7f8978074080 2024-03-24T00:15:37.981 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:37.981+0000 7f897e200640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f897810f420 con 0x7f8978075660 2024-03-24T00:15:37.981 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:37.981+0000 7f897e200640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f897810f720 con 0x7f8978109580 2024-03-24T00:15:37.982 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:37.981+0000 7f8976a00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8978075660 0x7f8978104fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:15:37.982 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:37.981+0000 7f8976a00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8978075660 0x7f8978104fb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.118:46880/0 (socket says 172.21.15.118:46880) 2024-03-24T00:15:37.982 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:37.981+0000 7f8976a00640 1 -- 172.21.15.118:0/3961778018 learned_addr learned my addr 172.21.15.118:0/3961778018 (peer_addr_for_me v2:172.21.15.118:0/0) 2024-03-24T00:15:37.982 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:37.981+0000 7f8977e00640 1 --2- 172.21.15.118:0/3961778018 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f8978109580 0x7f89781015e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:15:37.982 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:37.981+0000 7f8977400640 1 --2- 172.21.15.118:0/3961778018 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f8978074080 0x7f8978104a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:15:37.982 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:37.981+0000 7f8976a00640 1 -- 172.21.15.118:0/3961778018 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f8978109580 msgr2=0x7f89781015e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:15:37.982 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:37.981+0000 7f8976a00640 1 --2- 172.21.15.118:0/3961778018 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f8978109580 0x7f89781015e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:15:37.982 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:37.981+0000 7f8976a00640 1 -- 172.21.15.118:0/3961778018 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f8978074080 msgr2=0x7f8978104a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:15:37.982 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:37.981+0000 7f8976a00640 1 --2- 172.21.15.118:0/3961778018 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f8978074080 0x7f8978104a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:15:37.982 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:37.981+0000 7f8976a00640 1 -- 172.21.15.118:0/3961778018 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8978101ea0 con 0x7f8978075660 2024-03-24T00:15:37.982 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:37.981+0000 7f8977400640 1 --2- 172.21.15.118:0/3961778018 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f8978074080 0x7f8978104a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T00:15:37.982 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:37.981+0000 7f8977e00640 1 --2- 172.21.15.118:0/3961778018 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f8978109580 0x7f89781015e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T00:15:37.982 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:37.982+0000 7f8976a00640 1 --2- 172.21.15.118:0/3961778018 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8978075660 0x7f8978104fb0 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7f8964005ab0 tx=0x7f896402c750 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:15:37.983 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:37.982+0000 7f896fe00640 1 -- 172.21.15.118:0/3961778018 <== mon.2 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8964043020 con 0x7f8978075660 2024-03-24T00:15:37.983 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:37.982+0000 7f896fe00640 1 -- 172.21.15.118:0/3961778018 <== mon.2 v2:172.21.15.189:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f89640054e0 con 0x7f8978075660 2024-03-24T00:15:37.983 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:37.982+0000 7f897e200640 1 -- 172.21.15.118:0/3961778018 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f89781a5f40 con 0x7f8978075660 2024-03-24T00:15:37.984 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:37.982+0000 7f896fe00640 1 -- 172.21.15.118:0/3961778018 <== mon.2 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8964003cc0 con 0x7f8978075660 2024-03-24T00:15:37.984 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:37.982+0000 7f897e200640 1 -- 172.21.15.118:0/3961778018 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f89781a62f0 con 0x7f8978075660 2024-03-24T00:15:37.984 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:37.983+0000 7f897e200640 1 -- 172.21.15.118:0/3961778018 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8978108340 con 0x7f8978075660 2024-03-24T00:15:37.984 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:37.985+0000 7f896fe00640 1 -- 172.21.15.118:0/3961778018 <== mon.2 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 18) v1 ==== 137291+0+0 (secure 0 0 0) 0x7f8964046020 con 0x7f8978075660 2024-03-24T00:15:37.986 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:37.986+0000 7f896fe00640 1 --2- 172.21.15.118:0/3961778018 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7f895c0a2920 0x7f895c0a4de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:15:37.986 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:37.986+0000 7f896fe00640 1 -- 172.21.15.118:0/3961778018 <== mon.2 v2:172.21.15.189:3300/0 5 ==== osd_map(90..90 src has 1..90) v4 ==== 8277+0+0 (secure 0 0 0) 0x7f89640f0bf0 con 0x7f8978075660 2024-03-24T00:15:37.986 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:37.986+0000 7f8977400640 1 --2- 172.21.15.118:0/3961778018 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7f895c0a2920 0x7f895c0a4de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:15:37.986 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:37.987+0000 7f8977400640 1 --2- 172.21.15.118:0/3961778018 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7f895c0a2920 0x7f895c0a4de0 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7f8968009c20 tx=0x7f8968005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:15:37.989 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:37.990+0000 7f896fe00640 1 -- 172.21.15.118:0/3961778018 <== mon.2 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f896404d020 con 0x7f8978075660 2024-03-24T00:15:38.233 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:38.233+0000 7f897e200640 1 -- 172.21.15.118:0/3961778018 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "false"} v 0) v1 -- 0x7f89780622a0 con 0x7f8978075660 2024-03-24T00:15:38.740 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:38.740+0000 7f896fe00640 1 -- 172.21.15.118:0/3961778018 <== mon.2 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "false"}]=0 v28) v1 ==== 118+0+0 (secure 0 0 0) 0x7f89640bdc00 con 0x7f8978075660 2024-03-24T00:15:38.743 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:38.743+0000 7f897e200640 1 -- 172.21.15.118:0/3961778018 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7f895c0a2920 msgr2=0x7f895c0a4de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:15:38.743 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:38.743+0000 7f897e200640 1 --2- 172.21.15.118:0/3961778018 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7f895c0a2920 0x7f895c0a4de0 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7f8968009c20 tx=0x7f8968005f90 comp rx=0 tx=0).stop 2024-03-24T00:15:38.744 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:38.743+0000 7f897e200640 1 -- 172.21.15.118:0/3961778018 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8978075660 msgr2=0x7f8978104fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:15:38.744 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:38.743+0000 7f897e200640 1 --2- 172.21.15.118:0/3961778018 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8978075660 0x7f8978104fb0 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7f8964005ab0 tx=0x7f896402c750 comp rx=0 tx=0).stop 2024-03-24T00:15:38.744 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:38.744+0000 7f897e200640 1 -- 172.21.15.118:0/3961778018 shutdown_connections 2024-03-24T00:15:38.744 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:38.744+0000 7f897e200640 1 --2- 172.21.15.118:0/3961778018 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7f895c0a2920 0x7f895c0a4de0 unknown :-1 s=CLOSED pgs=249 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:15:38.744 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:38.744+0000 7f897e200640 1 --2- 172.21.15.118:0/3961778018 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f8978109580 0x7f89781015e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:15:38.744 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:38.744+0000 7f897e200640 1 --2- 172.21.15.118:0/3961778018 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8978075660 0x7f8978104fb0 unknown :-1 s=CLOSED pgs=188 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:15:38.744 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:38.744+0000 7f897e200640 1 --2- 172.21.15.118:0/3961778018 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f8978074080 0x7f8978104a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:15:38.744 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:38.744+0000 7f897e200640 1 -- 172.21.15.118:0/3961778018 >> 172.21.15.118:0/3961778018 conn(0x7f89780fcc60 msgr2=0x7f8978105510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:15:38.744 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:38.744+0000 7f897e200640 1 -- 172.21.15.118:0/3961778018 shutdown_connections 2024-03-24T00:15:38.744 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:38.744+0000 7f897e200640 1 -- 172.21.15.118:0/3961778018 wait complete. 2024-03-24T00:15:38.765 DEBUG:teuthology.orchestra.run.smithi118:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'ceph fs subvolumegroup pin cephfs qa random 0.10' 2024-03-24T00:15:38.975 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:38.975+0000 7f1adc800640 1 -- 172.21.15.118:0/888215888 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f1ad4104af0 msgr2=0x7f1ad4104f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:15:38.975 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:38.975+0000 7f1adc800640 1 --2- 172.21.15.118:0/888215888 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f1ad4104af0 0x7f1ad4104f70 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 crypto rx=0x7f1ac4009900 tx=0x7f1ac402bc40 comp rx=0 tx=0).stop 2024-03-24T00:15:38.975 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:38.975+0000 7f1adc800640 1 -- 172.21.15.118:0/888215888 shutdown_connections 2024-03-24T00:15:38.976 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:38.975+0000 7f1adc800640 1 --2- 172.21.15.118:0/888215888 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1ad41054b0 0x7f1ad410bd40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:15:38.976 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:38.975+0000 7f1adc800640 1 --2- 172.21.15.118:0/888215888 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f1ad4104af0 0x7f1ad4104f70 unknown :-1 s=CLOSED pgs=176 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:15:38.976 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:38.975+0000 7f1adc800640 1 --2- 172.21.15.118:0/888215888 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f1ad4103900 0x7f1ad4103d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:15:38.976 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:38.975+0000 7f1adc800640 1 -- 172.21.15.118:0/888215888 >> 172.21.15.118:0/888215888 conn(0x7f1ad40fee90 msgr2=0x7f1ad41012b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:15:38.976 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:38.975+0000 7f1adc800640 1 -- 172.21.15.118:0/888215888 shutdown_connections 2024-03-24T00:15:38.976 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:38.975+0000 7f1adc800640 1 -- 172.21.15.118:0/888215888 wait complete. 2024-03-24T00:15:38.978 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:38.976+0000 7f1adc800640 1 Processor -- start 2024-03-24T00:15:38.978 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:38.977+0000 7f1adc800640 1 -- start start 2024-03-24T00:15:38.978 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:38.977+0000 7f1adc800640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f1ad4103900 0x7f1ad4199b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:15:38.978 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:38.977+0000 7f1adc800640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1ad4104af0 0x7f1ad419a060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:15:38.978 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:38.977+0000 7f1adc800640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f1ad41054b0 0x7f1ad41a10d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:15:38.978 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:38.977+0000 7f1adc800640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1ad410f5d0 con 0x7f1ad41054b0 2024-03-24T00:15:38.979 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:38.977+0000 7f1adc800640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1ad410f450 con 0x7f1ad4104af0 2024-03-24T00:15:38.979 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:38.977+0000 7f1adc800640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1ad410f750 con 0x7f1ad4103900 2024-03-24T00:15:38.979 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:38.977+0000 7f1ad9e00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f1ad4103900 0x7f1ad4199b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:15:38.979 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:38.977+0000 7f1ad9e00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f1ad4103900 0x7f1ad4199b20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.118:57208/0 (socket says 172.21.15.118:57208) 2024-03-24T00:15:38.979 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:38.977+0000 7f1ad9e00640 1 -- 172.21.15.118:0/1926094651 learned_addr learned my addr 172.21.15.118:0/1926094651 (peer_addr_for_me v2:172.21.15.118:0/0) 2024-03-24T00:15:38.979 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:38.977+0000 7f1ada800640 1 --2- 172.21.15.118:0/1926094651 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f1ad41054b0 0x7f1ad41a10d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:15:38.979 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:38.977+0000 7f1ad9400640 1 --2- 172.21.15.118:0/1926094651 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1ad4104af0 0x7f1ad419a060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:15:38.979 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:38.978+0000 7f1ad9e00640 1 -- 172.21.15.118:0/1926094651 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1ad4104af0 msgr2=0x7f1ad419a060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:15:38.979 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:38.978+0000 7f1ad9e00640 1 --2- 172.21.15.118:0/1926094651 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1ad4104af0 0x7f1ad419a060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:15:38.979 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:38.978+0000 7f1ad9e00640 1 -- 172.21.15.118:0/1926094651 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f1ad41054b0 msgr2=0x7f1ad41a10d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:15:38.979 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:38.978+0000 7f1ad9e00640 1 --2- 172.21.15.118:0/1926094651 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f1ad41054b0 0x7f1ad41a10d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:15:38.979 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:38.978+0000 7f1ad9e00640 1 -- 172.21.15.118:0/1926094651 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1ad41a1740 con 0x7f1ad4103900 2024-03-24T00:15:38.979 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:38.978+0000 7f1ad9400640 1 --2- 172.21.15.118:0/1926094651 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1ad4104af0 0x7f1ad419a060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T00:15:38.980 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:38.978+0000 7f1ada800640 1 --2- 172.21.15.118:0/1926094651 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f1ad41054b0 0x7f1ad41a10d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T00:15:38.980 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:38.978+0000 7f1ad9e00640 1 --2- 172.21.15.118:0/1926094651 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f1ad4103900 0x7f1ad4199b20 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7f1ac800e9c0 tx=0x7f1ac800ee80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:15:38.980 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:38.978+0000 7f1acea00640 1 -- 172.21.15.118:0/1926094651 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1ac800cd80 con 0x7f1ad4103900 2024-03-24T00:15:38.980 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:38.978+0000 7f1acea00640 1 -- 172.21.15.118:0/1926094651 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f1ac80035f0 con 0x7f1ad4103900 2024-03-24T00:15:38.980 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:38.978+0000 7f1acea00640 1 -- 172.21.15.118:0/1926094651 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1ac8010690 con 0x7f1ad4103900 2024-03-24T00:15:38.980 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:38.978+0000 7f1adc800640 1 -- 172.21.15.118:0/1926094651 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1ad41a1a30 con 0x7f1ad4103900 2024-03-24T00:15:38.980 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:38.979+0000 7f1adc800640 1 -- 172.21.15.118:0/1926094651 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1ad41a1f00 con 0x7f1ad4103900 2024-03-24T00:15:38.980 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:38.980+0000 7f1adc800640 1 -- 172.21.15.118:0/1926094651 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1ad404f400 con 0x7f1ad4103900 2024-03-24T00:15:38.981 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:38.982+0000 7f1acea00640 1 -- 172.21.15.118:0/1926094651 <== mon.1 v2:172.21.15.193:3300/0 4 ==== mgrmap(e 18) v1 ==== 137291+0+0 (secure 0 0 0) 0x7f1ac80037a0 con 0x7f1ad4103900 2024-03-24T00:15:38.983 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:38.984+0000 7f1acea00640 1 --2- 172.21.15.118:0/1926094651 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7f1abc0a2920 0x7f1abc0a4de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:15:38.984 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:38.984+0000 7f1acea00640 1 -- 172.21.15.118:0/1926094651 <== mon.1 v2:172.21.15.193:3300/0 5 ==== osd_map(90..90 src has 1..90) v4 ==== 8277+0+0 (secure 0 0 0) 0x7f1ac80cef90 con 0x7f1ad4103900 2024-03-24T00:15:38.984 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:38.984+0000 7f1ad9400640 1 --2- 172.21.15.118:0/1926094651 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7f1abc0a2920 0x7f1abc0a4de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:15:38.984 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:38.985+0000 7f1ad9400640 1 --2- 172.21.15.118:0/1926094651 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7f1abc0a2920 0x7f1abc0a4de0 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7f1ac402c250 tx=0x7f1ac4005fd0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:15:38.986 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:38.987+0000 7f1acea00640 1 -- 172.21.15.118:0/1926094651 <== mon.1 v2:172.21.15.193:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f1ac809bfa0 con 0x7f1ad4103900 2024-03-24T00:15:39.109 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:15:38 smithi193 ceph-mon[36878]: pgmap v396: 97 pgs: 97 active+clean; 645 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-03-24T00:15:39.110 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:15:38 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T00:15:39.110 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:15:38 smithi193 ceph-mon[36878]: from='client.? 172.21.15.118:0/3961778018' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "false"}]: dispatch 2024-03-24T00:15:39.110 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:15:38 smithi193 ceph-mon[36878]: from='client.? ' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "false"}]: dispatch 2024-03-24T00:15:39.195 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:15:38 smithi118 ceph-mon[31535]: pgmap v396: 97 pgs: 97 active+clean; 645 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-03-24T00:15:39.195 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:15:38 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T00:15:39.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:15:38 smithi118 ceph-mon[31535]: from='client.? 172.21.15.118:0/3961778018' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "false"}]: dispatch 2024-03-24T00:15:39.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:15:38 smithi118 ceph-mon[31535]: from='client.? ' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "false"}]: dispatch 2024-03-24T00:15:39.203 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:15:38 smithi189 ceph-mon[36187]: pgmap v396: 97 pgs: 97 active+clean; 645 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-03-24T00:15:39.203 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:15:38 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T00:15:39.203 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:15:38 smithi189 ceph-mon[36187]: from='client.? 172.21.15.118:0/3961778018' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "false"}]: dispatch 2024-03-24T00:15:39.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:15:38 smithi189 ceph-mon[36187]: from='client.? ' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "false"}]: dispatch 2024-03-24T00:15:39.220 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:39.220+0000 7f1adc800640 1 -- 172.21.15.118:0/1926094651 --> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] -- mgr_command(tid 0: {"prefix": "fs subvolumegroup pin", "vol_name": "cephfs", "group_name": "qa", "pin_type": "random", "pin_setting": "0.10", "target": ["mon-mgr", ""]}) v1 -- 0x7f1ad4061ae0 con 0x7f1abc0a2920 2024-03-24T00:15:39.243 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:39.243+0000 7f1acea00640 1 -- 172.21.15.118:0/1926094651 <== mgr.14152 v2:172.21.15.118:6800/2112912943 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3 (secure 0 0 0) 0x7f1ad4061ae0 con 0x7f1abc0a2920 2024-03-24T00:15:39.243 INFO:teuthology.orchestra.run.smithi118.stdout:{} 2024-03-24T00:15:39.246 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:39.246+0000 7f1adc800640 1 -- 172.21.15.118:0/1926094651 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7f1abc0a2920 msgr2=0x7f1abc0a4de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:15:39.246 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:39.246+0000 7f1adc800640 1 --2- 172.21.15.118:0/1926094651 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7f1abc0a2920 0x7f1abc0a4de0 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7f1ac402c250 tx=0x7f1ac4005fd0 comp rx=0 tx=0).stop 2024-03-24T00:15:39.246 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:39.246+0000 7f1adc800640 1 -- 172.21.15.118:0/1926094651 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f1ad4103900 msgr2=0x7f1ad4199b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:15:39.246 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:39.246+0000 7f1adc800640 1 --2- 172.21.15.118:0/1926094651 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f1ad4103900 0x7f1ad4199b20 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7f1ac800e9c0 tx=0x7f1ac800ee80 comp rx=0 tx=0).stop 2024-03-24T00:15:39.246 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:39.246+0000 7f1adc800640 1 -- 172.21.15.118:0/1926094651 shutdown_connections 2024-03-24T00:15:39.246 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:39.246+0000 7f1adc800640 1 --2- 172.21.15.118:0/1926094651 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7f1abc0a2920 0x7f1abc0a4de0 unknown :-1 s=CLOSED pgs=250 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:15:39.246 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:39.246+0000 7f1adc800640 1 --2- 172.21.15.118:0/1926094651 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f1ad41054b0 0x7f1ad41a10d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:15:39.246 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:39.246+0000 7f1adc800640 1 --2- 172.21.15.118:0/1926094651 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1ad4104af0 0x7f1ad419a060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:15:39.246 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:39.246+0000 7f1adc800640 1 --2- 172.21.15.118:0/1926094651 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f1ad4103900 0x7f1ad4199b20 unknown :-1 s=CLOSED pgs=177 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:15:39.246 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:39.246+0000 7f1adc800640 1 -- 172.21.15.118:0/1926094651 >> 172.21.15.118:0/1926094651 conn(0x7f1ad40fee90 msgr2=0x7f1ad4107d20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:15:39.246 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:39.246+0000 7f1adc800640 1 -- 172.21.15.118:0/1926094651 shutdown_connections 2024-03-24T00:15:39.246 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:39.246+0000 7f1adc800640 1 -- 172.21.15.118:0/1926094651 wait complete. 2024-03-24T00:15:39.266 INFO:teuthology.run_tasks:Running task check-counter... 2024-03-24T00:15:39.276 INFO:teuthology.run_tasks:Running task fwd_scrub... 2024-03-24T00:15:39.290 DEBUG:teuthology.orchestra.run.smithi118:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-24T00:15:39.489 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:39.489+0000 7f25f0400640 1 -- 172.21.15.118:0/647263797 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f25e8109570 msgr2=0x7f25e810b9e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:15:39.489 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:39.489+0000 7f25f0400640 1 --2- 172.21.15.118:0/647263797 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f25e8109570 0x7f25e810b9e0 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7f25e0009be0 tx=0x7f25e002bef0 comp rx=0 tx=0).stop 2024-03-24T00:15:39.489 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:39.489+0000 7f25f0400640 1 -- 172.21.15.118:0/647263797 shutdown_connections 2024-03-24T00:15:39.489 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:39.489+0000 7f25f0400640 1 --2- 172.21.15.118:0/647263797 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f25e8109570 0x7f25e810b9e0 unknown :-1 s=CLOSED pgs=179 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:15:39.489 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:39.489+0000 7f25f0400640 1 --2- 172.21.15.118:0/647263797 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f25e80740c0 0x7f25e8074540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:15:39.490 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:39.489+0000 7f25f0400640 1 --2- 172.21.15.118:0/647263797 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f25e80756a0 0x7f25e8073af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:15:39.490 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:39.489+0000 7f25f0400640 1 -- 172.21.15.118:0/647263797 >> 172.21.15.118:0/647263797 conn(0x7f25e80fcc50 msgr2=0x7f25e80ff070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:15:39.490 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:39.489+0000 7f25f0400640 1 -- 172.21.15.118:0/647263797 shutdown_connections 2024-03-24T00:15:39.490 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:39.489+0000 7f25f0400640 1 -- 172.21.15.118:0/647263797 wait complete. 2024-03-24T00:15:39.492 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:39.491+0000 7f25f0400640 1 Processor -- start 2024-03-24T00:15:39.492 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:39.491+0000 7f25f0400640 1 -- start start 2024-03-24T00:15:39.492 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:39.491+0000 7f25f0400640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f25e80740c0 0x7f25e8199b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:15:39.492 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:39.491+0000 7f25f0400640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f25e80756a0 0x7f25e819a080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:15:39.493 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:39.491+0000 7f25f0400640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f25e8109570 0x7f25e81a10f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:15:39.493 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:39.491+0000 7f25f0400640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f25e810f550 con 0x7f25e80756a0 2024-03-24T00:15:39.493 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:39.491+0000 7f25f0400640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f25e810f3d0 con 0x7f25e8109570 2024-03-24T00:15:39.493 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:39.491+0000 7f25f0400640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f25e810f6d0 con 0x7f25e80740c0 2024-03-24T00:15:39.493 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:39.492+0000 7f25eda00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f25e80740c0 0x7f25e8199b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:15:39.493 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:39.492+0000 7f25eda00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f25e80740c0 0x7f25e8199b40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.118:57240/0 (socket says 172.21.15.118:57240) 2024-03-24T00:15:39.493 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:39.492+0000 7f25eda00640 1 -- 172.21.15.118:0/2933795393 learned_addr learned my addr 172.21.15.118:0/2933795393 (peer_addr_for_me v2:172.21.15.118:0/0) 2024-03-24T00:15:39.493 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:39.492+0000 7f25ee400640 1 --2- 172.21.15.118:0/2933795393 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f25e8109570 0x7f25e81a10f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:15:39.493 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:39.492+0000 7f25ed000640 1 --2- 172.21.15.118:0/2933795393 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f25e80756a0 0x7f25e819a080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:15:39.493 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:39.492+0000 7f25eda00640 1 -- 172.21.15.118:0/2933795393 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f25e8109570 msgr2=0x7f25e81a10f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:15:39.493 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:39.492+0000 7f25eda00640 1 --2- 172.21.15.118:0/2933795393 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f25e8109570 0x7f25e81a10f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:15:39.493 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:39.492+0000 7f25eda00640 1 -- 172.21.15.118:0/2933795393 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f25e80756a0 msgr2=0x7f25e819a080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:15:39.493 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:39.492+0000 7f25eda00640 1 --2- 172.21.15.118:0/2933795393 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f25e80756a0 0x7f25e819a080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:15:39.494 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:39.492+0000 7f25eda00640 1 -- 172.21.15.118:0/2933795393 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f25e81a1760 con 0x7f25e80740c0 2024-03-24T00:15:39.494 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:39.492+0000 7f25eda00640 1 --2- 172.21.15.118:0/2933795393 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f25e80740c0 0x7f25e8199b40 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7f25dc00e9f0 tx=0x7f25dc00eeb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:15:39.494 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:39.493+0000 7f25e6000640 1 -- 172.21.15.118:0/2933795393 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f25dc00cd80 con 0x7f25e80740c0 2024-03-24T00:15:39.494 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:39.493+0000 7f25e6000640 1 -- 172.21.15.118:0/2933795393 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f25dc010910 con 0x7f25e80740c0 2024-03-24T00:15:39.494 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:39.493+0000 7f25e6000640 1 -- 172.21.15.118:0/2933795393 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f25dc018990 con 0x7f25e80740c0 2024-03-24T00:15:39.494 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:39.493+0000 7f25f0400640 1 -- 172.21.15.118:0/2933795393 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f25e81a1a50 con 0x7f25e80740c0 2024-03-24T00:15:39.494 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:39.493+0000 7f25f0400640 1 -- 172.21.15.118:0/2933795393 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f25e81a1f50 con 0x7f25e80740c0 2024-03-24T00:15:39.494 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:39.494+0000 7f25f0400640 1 -- 172.21.15.118:0/2933795393 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f25e80688e0 con 0x7f25e80740c0 2024-03-24T00:15:39.495 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:39.496+0000 7f25e6000640 1 -- 172.21.15.118:0/2933795393 <== mon.1 v2:172.21.15.193:3300/0 4 ==== mgrmap(e 18) v1 ==== 137291+0+0 (secure 0 0 0) 0x7f25dc010ac0 con 0x7f25e80740c0 2024-03-24T00:15:39.498 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:39.498+0000 7f25e6000640 1 --2- 172.21.15.118:0/2933795393 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7f25d00a2800 0x7f25d00a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:15:39.498 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:39.498+0000 7f25e6000640 1 -- 172.21.15.118:0/2933795393 <== mon.1 v2:172.21.15.193:3300/0 5 ==== osd_map(90..90 src has 1..90) v4 ==== 8277+0+0 (secure 0 0 0) 0x7f25dc0ceef0 con 0x7f25e80740c0 2024-03-24T00:15:39.498 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:39.498+0000 7f25ed000640 1 --2- 172.21.15.118:0/2933795393 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7f25d00a2800 0x7f25d00a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:15:39.499 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:39.499+0000 7f25ed000640 1 --2- 172.21.15.118:0/2933795393 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7f25d00a2800 0x7f25d00a4cc0 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7f25e819b050 tx=0x7f25d4009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:15:39.500 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:39.501+0000 7f25e6000640 1 -- 172.21.15.118:0/2933795393 <== mon.1 v2:172.21.15.193:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f25dc09bf00 con 0x7f25e80740c0 2024-03-24T00:15:39.696 INFO:journalctl@ceph.mgr.x.smithi118.stdout:Mar 24 00:15:39 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mgr-x[31762]: 2024-03-24T00:15:39.228+0000 7f62ad800700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-24T00:15:39.696 INFO:journalctl@ceph.mgr.x.smithi118.stdout:Mar 24 00:15:39 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mgr-x[31762]: 2024-03-24T00:15:39.228+0000 7f62ad800700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-24T00:15:39.696 INFO:journalctl@ceph.mgr.x.smithi118.stdout:Mar 24 00:15:39 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mgr-x[31762]: 2024-03-24T00:15:39.228+0000 7f62ad800700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-24T00:15:39.696 INFO:journalctl@ceph.mgr.x.smithi118.stdout:Mar 24 00:15:39 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mgr-x[31762]: 2024-03-24T00:15:39.228+0000 7f62ad800700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-24T00:15:39.696 INFO:journalctl@ceph.mgr.x.smithi118.stdout:Mar 24 00:15:39 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mgr-x[31762]: 2024-03-24T00:15:39.228+0000 7f62ad800700 -1 client.0 error registering admin socket command: (17) File exists 2024-03-24T00:15:39.733 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:39.734+0000 7f25f0400640 1 -- 172.21.15.118:0/2933795393 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f25e80622d0 con 0x7f25e80740c0 2024-03-24T00:15:39.736 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:39.736+0000 7f25e6000640 1 -- 172.21.15.118:0/2933795393 <== mon.1 v2:172.21.15.193:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v90) v1 ==== 74+0+22834 (secure 0 0 0) 0x7f25dc09b800 con 0x7f25e80740c0 2024-03-24T00:15:39.736 INFO:teuthology.orchestra.run.smithi118.stdout: 2024-03-24T00:15:39.737 INFO:teuthology.orchestra.run.smithi118.stdout:{"epoch":90,"fsid":"d79a5420-e971-11ee-95cd-87774f69a715","created":"2024-03-24T00:03:58.648524+0000","modified":"2024-03-24T00:15:07.721960+0000","last_up_change":"2024-03-24T00:12:03.809000+0000","last_in_change":"2024-03-24T00:11:44.389344+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-24T00:07:50.190965+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-24T00:12:59.993723+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-24T00:13:01.257347+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"86","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"c9590426-3901-4f68-a5ef-d57aa8fbac92","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6802","nonce":557690061},{"type":"v1","addr":"172.21.15.118:6803","nonce":557690061}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6804","nonce":557690061},{"type":"v1","addr":"172.21.15.118:6805","nonce":557690061}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6808","nonce":557690061},{"type":"v1","addr":"172.21.15.118:6809","nonce":557690061}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6806","nonce":557690061},{"type":"v1","addr":"172.21.15.118:6807","nonce":557690061}]},"public_addr":"172.21.15.118:6803/557690061","cluster_addr":"172.21.15.118:6805/557690061","heartbeat_back_addr":"172.21.15.118:6809/557690061","heartbeat_front_addr":"172.21.15.118:6807/557690061","state":["exists","up"]},{"osd":1,"uuid":"fc7f827e-ecf6-4661-aad0-7a22951892e0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6800","nonce":4203029485},{"type":"v1","addr":"172.21.15.189:6801","nonce":4203029485}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6802","nonce":4203029485},{"type":"v1","addr":"172.21.15.189:6803","nonce":4203029485}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6806","nonce":4203029485},{"type":"v1","addr":"172.21.15.189:6807","nonce":4203029485}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6804","nonce":4203029485},{"type":"v1","addr":"172.21.15.189:6805","nonce":4203029485}]},"public_addr":"172.21.15.189:6801/4203029485","cluster_addr":"172.21.15.189:6803/4203029485","heartbeat_back_addr":"172.21.15.189:6807/4203029485","heartbeat_front_addr":"172.21.15.189:6805/4203029485","state":["exists","up"]},{"osd":2,"uuid":"e73ccfe9-0e5b-4734-a154-a2f0d829fcfb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6800","nonce":3699076971},{"type":"v1","addr":"172.21.15.193:6801","nonce":3699076971}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6802","nonce":3699076971},{"type":"v1","addr":"172.21.15.193:6803","nonce":3699076971}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6806","nonce":3699076971},{"type":"v1","addr":"172.21.15.193:6807","nonce":3699076971}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6804","nonce":3699076971},{"type":"v1","addr":"172.21.15.193:6805","nonce":3699076971}]},"public_addr":"172.21.15.193:6801/3699076971","cluster_addr":"172.21.15.193:6803/3699076971","heartbeat_back_addr":"172.21.15.193:6807/3699076971","heartbeat_front_addr":"172.21.15.193:6805/3699076971","state":["exists","up"]},{"osd":3,"uuid":"63fed43b-43ce-4031-8957-7fe94725c1ed","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6810","nonce":4059617010},{"type":"v1","addr":"172.21.15.118:6811","nonce":4059617010}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6812","nonce":4059617010},{"type":"v1","addr":"172.21.15.118:6813","nonce":4059617010}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6816","nonce":4059617010},{"type":"v1","addr":"172.21.15.118:6817","nonce":4059617010}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6814","nonce":4059617010},{"type":"v1","addr":"172.21.15.118:6815","nonce":4059617010}]},"public_addr":"172.21.15.118:6811/4059617010","cluster_addr":"172.21.15.118:6813/4059617010","heartbeat_back_addr":"172.21.15.118:6817/4059617010","heartbeat_front_addr":"172.21.15.118:6815/4059617010","state":["exists","up"]},{"osd":4,"uuid":"57377f66-c3b4-43c1-8e8e-bc873672906d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":34,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6808","nonce":4075247068},{"type":"v1","addr":"172.21.15.189:6809","nonce":4075247068}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6810","nonce":4075247068},{"type":"v1","addr":"172.21.15.189:6811","nonce":4075247068}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6814","nonce":4075247068},{"type":"v1","addr":"172.21.15.189:6815","nonce":4075247068}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6812","nonce":4075247068},{"type":"v1","addr":"172.21.15.189:6813","nonce":4075247068}]},"public_addr":"172.21.15.189:6809/4075247068","cluster_addr":"172.21.15.189:6811/4075247068","heartbeat_back_addr":"172.21.15.189:6815/4075247068","heartbeat_front_addr":"172.21.15.189:6813/4075247068","state":["exists","up"]},{"osd":5,"uuid":"91970976-2193-4224-81c0-dc96cb022c85","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":39,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6808","nonce":2920363402},{"type":"v1","addr":"172.21.15.193:6809","nonce":2920363402}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6810","nonce":2920363402},{"type":"v1","addr":"172.21.15.193:6811","nonce":2920363402}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6814","nonce":2920363402},{"type":"v1","addr":"172.21.15.193:6815","nonce":2920363402}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6812","nonce":2920363402},{"type":"v1","addr":"172.21.15.193:6813","nonce":2920363402}]},"public_addr":"172.21.15.193:6809/2920363402","cluster_addr":"172.21.15.193:6811/2920363402","heartbeat_back_addr":"172.21.15.193:6815/2920363402","heartbeat_front_addr":"172.21.15.193:6813/2920363402","state":["exists","up"]},{"osd":6,"uuid":"0b77b196-8674-41c6-ad3d-726b5d01c8fd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":47,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6818","nonce":431354576},{"type":"v1","addr":"172.21.15.118:6819","nonce":431354576}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6820","nonce":431354576},{"type":"v1","addr":"172.21.15.118:6821","nonce":431354576}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6824","nonce":431354576},{"type":"v1","addr":"172.21.15.118:6825","nonce":431354576}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6822","nonce":431354576},{"type":"v1","addr":"172.21.15.118:6823","nonce":431354576}]},"public_addr":"172.21.15.118:6819/431354576","cluster_addr":"172.21.15.118:6821/431354576","heartbeat_back_addr":"172.21.15.118:6825/431354576","heartbeat_front_addr":"172.21.15.118:6823/431354576","state":["exists","up"]},{"osd":7,"uuid":"1fbbd38f-4e8e-4d4c-8e98-66f6862bc4bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":52,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6816","nonce":3667611284},{"type":"v1","addr":"172.21.15.189:6817","nonce":3667611284}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6818","nonce":3667611284},{"type":"v1","addr":"172.21.15.189:6819","nonce":3667611284}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6822","nonce":3667611284},{"type":"v1","addr":"172.21.15.189:6823","nonce":3667611284}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6820","nonce":3667611284},{"type":"v1","addr":"172.21.15.189:6821","nonce":3667611284}]},"public_addr":"172.21.15.189:6817/3667611284","cluster_addr":"172.21.15.189:6819/3667611284","heartbeat_back_addr":"172.21.15.189:6823/3667611284","heartbeat_front_addr":"172.21.15.189:6821/3667611284","state":["exists","up"]},{"osd":8,"uuid":"6462c2b5-e4b3-4b0b-bcfa-4de2c89e93e0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":60,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6816","nonce":3853800910},{"type":"v1","addr":"172.21.15.193:6817","nonce":3853800910}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6818","nonce":3853800910},{"type":"v1","addr":"172.21.15.193:6819","nonce":3853800910}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6822","nonce":3853800910},{"type":"v1","addr":"172.21.15.193:6823","nonce":3853800910}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6820","nonce":3853800910},{"type":"v1","addr":"172.21.15.193:6821","nonce":3853800910}]},"public_addr":"172.21.15.193:6817/3853800910","cluster_addr":"172.21.15.193:6819/3853800910","heartbeat_back_addr":"172.21.15.193:6823/3853800910","heartbeat_front_addr":"172.21.15.193:6821/3853800910","state":["exists","up"]},{"osd":9,"uuid":"54be277f-3596-422f-9de5-9005a8571c6f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":68,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6826","nonce":3248293936},{"type":"v1","addr":"172.21.15.118:6827","nonce":3248293936}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6828","nonce":3248293936},{"type":"v1","addr":"172.21.15.118:6829","nonce":3248293936}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6832","nonce":3248293936},{"type":"v1","addr":"172.21.15.118:6833","nonce":3248293936}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6830","nonce":3248293936},{"type":"v1","addr":"172.21.15.118:6831","nonce":3248293936}]},"public_addr":"172.21.15.118:6827/3248293936","cluster_addr":"172.21.15.118:6829/3248293936","heartbeat_back_addr":"172.21.15.118:6833/3248293936","heartbeat_front_addr":"172.21.15.118:6831/3248293936","state":["exists","up"]},{"osd":10,"uuid":"7ba69d69-ac41-44ce-bc39-c049a8ca5900","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":73,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6824","nonce":2112195760},{"type":"v1","addr":"172.21.15.189:6825","nonce":2112195760}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6826","nonce":2112195760},{"type":"v1","addr":"172.21.15.189:6827","nonce":2112195760}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6830","nonce":2112195760},{"type":"v1","addr":"172.21.15.189:6831","nonce":2112195760}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6828","nonce":2112195760},{"type":"v1","addr":"172.21.15.189:6829","nonce":2112195760}]},"public_addr":"172.21.15.189:6825/2112195760","cluster_addr":"172.21.15.189:6827/2112195760","heartbeat_back_addr":"172.21.15.189:6831/2112195760","heartbeat_front_addr":"172.21.15.189:6829/2112195760","state":["exists","up"]},{"osd":11,"uuid":"186f0b18-6b22-4eae-83b7-b21bbcce0817","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":81,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6824","nonce":3880515510},{"type":"v1","addr":"172.21.15.193:6825","nonce":3880515510}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6826","nonce":3880515510},{"type":"v1","addr":"172.21.15.193:6827","nonce":3880515510}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6830","nonce":3880515510},{"type":"v1","addr":"172.21.15.193:6831","nonce":3880515510}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6828","nonce":3880515510},{"type":"v1","addr":"172.21.15.193:6829","nonce":3880515510}]},"public_addr":"172.21.15.193:6825/3880515510","cluster_addr":"172.21.15.193:6827/3880515510","heartbeat_back_addr":"172.21.15.193:6831/3880515510","heartbeat_front_addr":"172.21.15.193:6829/3880515510","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:06:58.105543+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:07:23.795866+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:07:47.426050+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:08:19.427073+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:08:45.816082+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:09:11.081523+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:09:44.485077+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:10:11.338972+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:10:36.234024+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:11:09.730432+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:11:37.059083+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:12:02.371947+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.a","mappings":[{"from":2,"to":11}]},{"pgid":"2.14","mappings":[{"from":2,"to":4}]},{"pgid":"2.1d","mappings":[{"from":8,"to":11}]},{"pgid":"2.1f","mappings":[{"from":8,"to":1}]},{"pgid":"3.11","mappings":[{"from":4,"to":8},{"from":5,"to":9}]},{"pgid":"3.13","mappings":[{"from":4,"to":9}]},{"pgid":"3.14","mappings":[{"from":4,"to":6}]},{"pgid":"3.31","mappings":[{"from":4,"to":7}]},{"pgid":"3.38","mappings":[{"from":3,"to":9}]},{"pgid":"3.39","mappings":[{"from":5,"to":8}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.118:6801/389249272":"2024-03-25T00:05:05.458796+0000","172.21.15.118:6800/389249272":"2024-03-25T00:05:05.458796+0000","172.21.15.118:0/2101854694":"2024-03-25T00:05:05.458796+0000","172.21.15.118:0/630779359":"2024-03-25T00:05:05.458796+0000","172.21.15.118:0/2158921029":"2024-03-25T00:04:31.283725+0000","172.21.15.118:0/1003670455":"2024-03-25T00:05:05.458796+0000","172.21.15.118:0/368712314":"2024-03-25T00:04:31.283725+0000","172.21.15.118:0/2004775597":"2024-03-25T00:04:31.283725+0000","172.21.15.118:6800/1463147955":"2024-03-25T00:04:31.283725+0000","172.21.15.118:6801/1463147955":"2024-03-25T00:04:31.283725+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-24T00:15:39.739 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:39.739+0000 7f25f0400640 1 -- 172.21.15.118:0/2933795393 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7f25d00a2800 msgr2=0x7f25d00a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:15:39.739 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:39.739+0000 7f25f0400640 1 --2- 172.21.15.118:0/2933795393 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7f25d00a2800 0x7f25d00a4cc0 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7f25e819b050 tx=0x7f25d4009260 comp rx=0 tx=0).stop 2024-03-24T00:15:39.739 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:39.739+0000 7f25f0400640 1 -- 172.21.15.118:0/2933795393 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f25e80740c0 msgr2=0x7f25e8199b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:15:39.739 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:39.739+0000 7f25f0400640 1 --2- 172.21.15.118:0/2933795393 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f25e80740c0 0x7f25e8199b40 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7f25dc00e9f0 tx=0x7f25dc00eeb0 comp rx=0 tx=0).stop 2024-03-24T00:15:39.739 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:39.739+0000 7f25f0400640 1 -- 172.21.15.118:0/2933795393 shutdown_connections 2024-03-24T00:15:39.739 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:39.739+0000 7f25f0400640 1 --2- 172.21.15.118:0/2933795393 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7f25d00a2800 0x7f25d00a4cc0 unknown :-1 s=CLOSED pgs=251 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:15:39.739 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:39.739+0000 7f25f0400640 1 --2- 172.21.15.118:0/2933795393 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f25e8109570 0x7f25e81a10f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:15:39.739 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:39.739+0000 7f25f0400640 1 --2- 172.21.15.118:0/2933795393 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f25e80756a0 0x7f25e819a080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:15:39.739 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:39.739+0000 7f25f0400640 1 --2- 172.21.15.118:0/2933795393 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f25e80740c0 0x7f25e8199b40 unknown :-1 s=CLOSED pgs=180 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:15:39.739 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:39.739+0000 7f25f0400640 1 -- 172.21.15.118:0/2933795393 >> 172.21.15.118:0/2933795393 conn(0x7f25e80fcc50 msgr2=0x7f25e81054c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:15:39.740 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:39.739+0000 7f25f0400640 1 -- 172.21.15.118:0/2933795393 shutdown_connections 2024-03-24T00:15:39.740 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:39.739+0000 7f25f0400640 1 -- 172.21.15.118:0/2933795393 wait complete. 2024-03-24T00:15:39.758 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-24T00:07:50.190965+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-24T00:12:59.993723+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '85', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.8799999952316284, 'score_stable': 1.8799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.8799999952316284, 'raw_score_stable': 1.8799999952316284, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-03-24T00:13:01.257347+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '86', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-24T00:15:39.759 DEBUG:teuthology.orchestra.run.smithi118:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-24T00:15:39.953 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:15:39 smithi189 ceph-mon[36187]: from='client.? ' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "false"}]': finished 2024-03-24T00:15:39.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:15:39 smithi189 ceph-mon[36187]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-03-24T00:15:39.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:15:39 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T00:15:39.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:15:39 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T00:15:39.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:15:39 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T00:15:39.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:15:39 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T00:15:39.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:15:39 smithi189 ceph-mon[36187]: from='client.? 172.21.15.118:0/2933795393' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-24T00:15:39.966 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:39.966+0000 7fbda1000640 1 -- 172.21.15.118:0/1616594056 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fbd9c102130 msgr2=0x7fbd9c104520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:15:39.966 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:39.966+0000 7fbda1000640 1 --2- 172.21.15.118:0/1616594056 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fbd9c102130 0x7fbd9c104520 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7fbd8c009900 tx=0x7fbd8c02bc40 comp rx=0 tx=0).stop 2024-03-24T00:15:39.966 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:39.966+0000 7fbda1000640 1 -- 172.21.15.118:0/1616594056 shutdown_connections 2024-03-24T00:15:39.966 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:39.966+0000 7fbda1000640 1 --2- 172.21.15.118:0/1616594056 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fbd9c107390 0x7fbd9c1097e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:15:39.966 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:39.966+0000 7fbda1000640 1 --2- 172.21.15.118:0/1616594056 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fbd9c104a60 0x7fbd9c106e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:15:39.966 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:39.966+0000 7fbda1000640 1 --2- 172.21.15.118:0/1616594056 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fbd9c102130 0x7fbd9c104520 unknown :-1 s=CLOSED pgs=177 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:15:39.966 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:39.966+0000 7fbda1000640 1 -- 172.21.15.118:0/1616594056 >> 172.21.15.118:0/1616594056 conn(0x7fbd9c0fbb40 msgr2=0x7fbd9c0fdf60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:15:39.966 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:39.966+0000 7fbda1000640 1 -- 172.21.15.118:0/1616594056 shutdown_connections 2024-03-24T00:15:39.966 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:39.966+0000 7fbda1000640 1 -- 172.21.15.118:0/1616594056 wait complete. 2024-03-24T00:15:39.968 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:39.967+0000 7fbda1000640 1 Processor -- start 2024-03-24T00:15:39.968 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:39.967+0000 7fbda1000640 1 -- start start 2024-03-24T00:15:39.968 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:39.967+0000 7fbda1000640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fbd9c102130 0x7fbd9c199cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:15:39.968 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:39.967+0000 7fbda1000640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fbd9c104a60 0x7fbd9c19a210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:15:39.969 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:39.967+0000 7fbda1000640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fbd9c107390 0x7fbd9c1a1220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:15:39.969 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:39.967+0000 7fbda1000640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbd9c10d340 con 0x7fbd9c102130 2024-03-24T00:15:39.969 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:39.967+0000 7fbda1000640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbd9c10d1c0 con 0x7fbd9c104a60 2024-03-24T00:15:39.969 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:39.967+0000 7fbda1000640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbd9c10d4c0 con 0x7fbd9c107390 2024-03-24T00:15:39.969 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:39.967+0000 7fbd99e00640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fbd9c102130 0x7fbd9c199cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:15:39.969 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:39.968+0000 7fbd9a800640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fbd9c107390 0x7fbd9c1a1220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:15:39.969 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:39.968+0000 7fbd99e00640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fbd9c102130 0x7fbd9c199cd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.118:3300/0 says I am v2:172.21.15.118:58388/0 (socket says 172.21.15.118:58388) 2024-03-24T00:15:39.969 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:39.968+0000 7fbd99e00640 1 -- 172.21.15.118:0/2070133610 learned_addr learned my addr 172.21.15.118:0/2070133610 (peer_addr_for_me v2:172.21.15.118:0/0) 2024-03-24T00:15:39.969 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:39.968+0000 7fbd99400640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fbd9c104a60 0x7fbd9c19a210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:15:39.969 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:39.968+0000 7fbd9a800640 1 -- 172.21.15.118:0/2070133610 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fbd9c104a60 msgr2=0x7fbd9c19a210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:15:39.969 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:39.968+0000 7fbd9a800640 1 --2- 172.21.15.118:0/2070133610 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fbd9c104a60 0x7fbd9c19a210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:15:39.969 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:39.968+0000 7fbd9a800640 1 -- 172.21.15.118:0/2070133610 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fbd9c102130 msgr2=0x7fbd9c199cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:15:39.969 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:39.968+0000 7fbd9a800640 1 --2- 172.21.15.118:0/2070133610 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fbd9c102130 0x7fbd9c199cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:15:39.969 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:39.968+0000 7fbd9a800640 1 -- 172.21.15.118:0/2070133610 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbd9c1a18c0 con 0x7fbd9c107390 2024-03-24T00:15:39.970 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:39.968+0000 7fbd99400640 1 --2- 172.21.15.118:0/2070133610 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fbd9c104a60 0x7fbd9c19a210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T00:15:39.970 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:39.968+0000 7fbd99e00640 1 --2- 172.21.15.118:0/2070133610 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fbd9c102130 0x7fbd9c199cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T00:15:39.970 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:39.968+0000 7fbd9a800640 1 --2- 172.21.15.118:0/2070133610 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fbd9c107390 0x7fbd9c1a1220 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7fbd9400da10 tx=0x7fbd9400ded0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:15:39.970 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:39.969+0000 7fbd92a00640 1 -- 172.21.15.118:0/2070133610 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbd9400bd70 con 0x7fbd9c107390 2024-03-24T00:15:39.970 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:39.969+0000 7fbda1000640 1 -- 172.21.15.118:0/2070133610 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbd9c1a1bb0 con 0x7fbd9c107390 2024-03-24T00:15:39.970 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:39.969+0000 7fbd92a00640 1 -- 172.21.15.118:0/2070133610 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fbd94003e50 con 0x7fbd9c107390 2024-03-24T00:15:39.970 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:39.969+0000 7fbd92a00640 1 -- 172.21.15.118:0/2070133610 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbd940106a0 con 0x7fbd9c107390 2024-03-24T00:15:39.970 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:39.969+0000 7fbda1000640 1 -- 172.21.15.118:0/2070133610 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbd9c1a20b0 con 0x7fbd9c107390 2024-03-24T00:15:39.970 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:39.970+0000 7fbda1000640 1 -- 172.21.15.118:0/2070133610 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbd9c108ee0 con 0x7fbd9c107390 2024-03-24T00:15:39.972 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:39.973+0000 7fbd92a00640 1 -- 172.21.15.118:0/2070133610 <== mon.1 v2:172.21.15.193:3300/0 4 ==== mgrmap(e 18) v1 ==== 137291+0+0 (secure 0 0 0) 0x7fbd94003970 con 0x7fbd9c107390 2024-03-24T00:15:39.973 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:39.974+0000 7fbd92a00640 1 --2- 172.21.15.118:0/2070133610 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7fbd800a2800 0x7fbd800a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:15:39.974 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:39.974+0000 7fbd92a00640 1 -- 172.21.15.118:0/2070133610 <== mon.1 v2:172.21.15.193:3300/0 5 ==== osd_map(90..90 src has 1..90) v4 ==== 8277+0+0 (secure 0 0 0) 0x7fbd940ceec0 con 0x7fbd9c107390 2024-03-24T00:15:39.974 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:39.974+0000 7fbd99e00640 1 --2- 172.21.15.118:0/2070133610 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7fbd800a2800 0x7fbd800a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:15:39.974 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:39.975+0000 7fbd99e00640 1 --2- 172.21.15.118:0/2070133610 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7fbd800a2800 0x7fbd800a4cc0 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7fbd8c006010 tx=0x7fbd8c02c890 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:15:39.979 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:39.980+0000 7fbd92a00640 1 -- 172.21.15.118:0/2070133610 <== mon.1 v2:172.21.15.193:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7fbd9409bed0 con 0x7fbd9c107390 2024-03-24T00:15:40.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:15:39 smithi118 ceph-mon[31535]: from='client.? ' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "false"}]': finished 2024-03-24T00:15:40.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:15:39 smithi118 ceph-mon[31535]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-03-24T00:15:40.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:15:39 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T00:15:40.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:15:39 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T00:15:40.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:15:39 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T00:15:40.197 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:15:39 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T00:15:40.197 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:15:39 smithi118 ceph-mon[31535]: from='client.? 172.21.15.118:0/2933795393' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-24T00:15:40.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:15:39 smithi193 ceph-mon[36878]: from='client.? ' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "balance_automate", "val": "false"}]': finished 2024-03-24T00:15:40.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:15:39 smithi193 ceph-mon[36878]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-03-24T00:15:40.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:15:39 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T00:15:40.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:15:39 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T00:15:40.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:15:39 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T00:15:40.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:15:39 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T00:15:40.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:15:39 smithi193 ceph-mon[36878]: from='client.? 172.21.15.118:0/2933795393' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-24T00:15:40.223 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:40.223+0000 7fbda1000640 1 -- 172.21.15.118:0/2070133610 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fbd9c062340 con 0x7fbd9c107390 2024-03-24T00:15:40.223 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:40.224+0000 7fbd92a00640 1 -- 172.21.15.118:0/2070133610 <== mon.1 v2:172.21.15.193:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v90) v1 ==== 93+0+10 (secure 0 0 0) 0x7fbd9409b7d0 con 0x7fbd9c107390 2024-03-24T00:15:40.224 INFO:teuthology.orchestra.run.smithi118.stdout:pg_num: 1 2024-03-24T00:15:40.226 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:40.227+0000 7fbda1000640 1 -- 172.21.15.118:0/2070133610 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7fbd800a2800 msgr2=0x7fbd800a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:15:40.227 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:40.227+0000 7fbda1000640 1 --2- 172.21.15.118:0/2070133610 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7fbd800a2800 0x7fbd800a4cc0 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7fbd8c006010 tx=0x7fbd8c02c890 comp rx=0 tx=0).stop 2024-03-24T00:15:40.227 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:40.227+0000 7fbda1000640 1 -- 172.21.15.118:0/2070133610 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fbd9c107390 msgr2=0x7fbd9c1a1220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:15:40.227 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:40.227+0000 7fbda1000640 1 --2- 172.21.15.118:0/2070133610 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fbd9c107390 0x7fbd9c1a1220 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7fbd9400da10 tx=0x7fbd9400ded0 comp rx=0 tx=0).stop 2024-03-24T00:15:40.227 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:40.227+0000 7fbda1000640 1 -- 172.21.15.118:0/2070133610 shutdown_connections 2024-03-24T00:15:40.227 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:40.227+0000 7fbda1000640 1 --2- 172.21.15.118:0/2070133610 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7fbd800a2800 0x7fbd800a4cc0 unknown :-1 s=CLOSED pgs=252 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:15:40.227 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:40.227+0000 7fbda1000640 1 --2- 172.21.15.118:0/2070133610 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fbd9c107390 0x7fbd9c1a1220 unknown :-1 s=CLOSED pgs=181 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:15:40.227 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:40.227+0000 7fbda1000640 1 --2- 172.21.15.118:0/2070133610 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fbd9c104a60 0x7fbd9c19a210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:15:40.227 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:40.227+0000 7fbda1000640 1 --2- 172.21.15.118:0/2070133610 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fbd9c102130 0x7fbd9c199cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:15:40.227 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:40.227+0000 7fbda1000640 1 -- 172.21.15.118:0/2070133610 >> 172.21.15.118:0/2070133610 conn(0x7fbd9c0fbb40 msgr2=0x7fbd9c100870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:15:40.227 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:40.227+0000 7fbda1000640 1 -- 172.21.15.118:0/2070133610 shutdown_connections 2024-03-24T00:15:40.227 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:40.227+0000 7fbda1000640 1 -- 172.21.15.118:0/2070133610 wait complete. 2024-03-24T00:15:40.244 DEBUG:teuthology.orchestra.run.smithi118:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-24T00:15:40.455 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:40.455+0000 7f7a1c000640 1 -- 172.21.15.118:0/3702008339 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f7a141016d0 msgr2=0x7f7a14101af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:15:40.455 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:40.455+0000 7f7a1c000640 1 --2- 172.21.15.118:0/3702008339 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f7a141016d0 0x7f7a14101af0 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7f7a00009900 tx=0x7f7a0002bc30 comp rx=0 tx=0).stop 2024-03-24T00:15:40.455 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:40.455+0000 7f7a1c000640 1 -- 172.21.15.118:0/3702008339 shutdown_connections 2024-03-24T00:15:40.455 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:40.455+0000 7f7a1c000640 1 --2- 172.21.15.118:0/3702008339 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7a14103280 0x7f7a14109b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:15:40.455 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:40.455+0000 7f7a1c000640 1 --2- 172.21.15.118:0/3702008339 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f7a141028c0 0x7f7a14102d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:15:40.456 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:40.455+0000 7f7a1c000640 1 --2- 172.21.15.118:0/3702008339 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f7a141016d0 0x7f7a14101af0 unknown :-1 s=CLOSED pgs=182 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:15:40.456 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:40.455+0000 7f7a1c000640 1 -- 172.21.15.118:0/3702008339 >> 172.21.15.118:0/3702008339 conn(0x7f7a140fcc40 msgr2=0x7f7a140ff080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:15:40.456 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:40.455+0000 7f7a1c000640 1 -- 172.21.15.118:0/3702008339 shutdown_connections 2024-03-24T00:15:40.456 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:40.455+0000 7f7a1c000640 1 -- 172.21.15.118:0/3702008339 wait complete. 2024-03-24T00:15:40.458 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:40.456+0000 7f7a1c000640 1 Processor -- start 2024-03-24T00:15:40.458 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:40.457+0000 7f7a1c000640 1 -- start start 2024-03-24T00:15:40.458 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:40.457+0000 7f7a1c000640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7a141016d0 0x7f7a14075450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:15:40.458 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:40.457+0000 7f7a1c000640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f7a141028c0 0x7f7a14073a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:15:40.458 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:40.457+0000 7f7a1c000640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f7a14103280 0x7f7a14073fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:15:40.459 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:40.457+0000 7f7a1c000640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7a1410d370 con 0x7f7a14103280 2024-03-24T00:15:40.459 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:40.457+0000 7f7a1c000640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7a1410d1f0 con 0x7f7a141016d0 2024-03-24T00:15:40.459 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:40.457+0000 7f7a1c000640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7a1410d4f0 con 0x7f7a141028c0 2024-03-24T00:15:40.459 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:40.457+0000 7f7a18c00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f7a141028c0 0x7f7a14073a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:15:40.459 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:40.457+0000 7f7a18c00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f7a141028c0 0x7f7a14073a90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.118:57288/0 (socket says 172.21.15.118:57288) 2024-03-24T00:15:40.459 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:40.457+0000 7f7a18c00640 1 -- 172.21.15.118:0/483533557 learned_addr learned my addr 172.21.15.118:0/483533557 (peer_addr_for_me v2:172.21.15.118:0/0) 2024-03-24T00:15:40.459 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:40.457+0000 7f7a19600640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7a141016d0 0x7f7a14075450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:15:40.459 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:40.457+0000 7f7a1a000640 1 --2- 172.21.15.118:0/483533557 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f7a14103280 0x7f7a14073fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:15:40.459 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:40.458+0000 7f7a18c00640 1 -- 172.21.15.118:0/483533557 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7a141016d0 msgr2=0x7f7a14075450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:15:40.459 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:40.458+0000 7f7a18c00640 1 --2- 172.21.15.118:0/483533557 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7a141016d0 0x7f7a14075450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:15:40.459 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:40.458+0000 7f7a18c00640 1 -- 172.21.15.118:0/483533557 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f7a14103280 msgr2=0x7f7a14073fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:15:40.459 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:40.458+0000 7f7a18c00640 1 --2- 172.21.15.118:0/483533557 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f7a14103280 0x7f7a14073fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:15:40.459 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:40.458+0000 7f7a18c00640 1 -- 172.21.15.118:0/483533557 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7a14074860 con 0x7f7a141028c0 2024-03-24T00:15:40.459 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:40.458+0000 7f7a1a000640 1 --2- 172.21.15.118:0/483533557 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f7a14103280 0x7f7a14073fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T00:15:40.460 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:40.458+0000 7f7a18c00640 1 --2- 172.21.15.118:0/483533557 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f7a141028c0 0x7f7a14073a90 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7f7a0400eab0 tx=0x7f7a0400ef70 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:15:40.460 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:40.458+0000 7f7a12000640 1 -- 172.21.15.118:0/483533557 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7a0400ce40 con 0x7f7a141028c0 2024-03-24T00:15:40.460 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:40.458+0000 7f7a12000640 1 -- 172.21.15.118:0/483533557 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f7a04011950 con 0x7f7a141028c0 2024-03-24T00:15:40.460 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:40.458+0000 7f7a12000640 1 -- 172.21.15.118:0/483533557 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7a04010ae0 con 0x7f7a141028c0 2024-03-24T00:15:40.460 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:40.458+0000 7f7a1c000640 1 -- 172.21.15.118:0/483533557 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7a141a1b00 con 0x7f7a141028c0 2024-03-24T00:15:40.460 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:40.459+0000 7f7a1c000640 1 -- 172.21.15.118:0/483533557 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7a141a2050 con 0x7f7a141028c0 2024-03-24T00:15:40.460 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:40.460+0000 7f7a1c000640 1 -- 172.21.15.118:0/483533557 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7a141089f0 con 0x7f7a141028c0 2024-03-24T00:15:40.461 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:40.462+0000 7f7a12000640 1 -- 172.21.15.118:0/483533557 <== mon.1 v2:172.21.15.193:3300/0 4 ==== mgrmap(e 18) v1 ==== 137291+0+0 (secure 0 0 0) 0x7f7a04011b00 con 0x7f7a141028c0 2024-03-24T00:15:40.463 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:40.463+0000 7f7a12000640 1 --2- 172.21.15.118:0/483533557 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7f79f80a2800 0x7f79f80a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:15:40.463 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:40.463+0000 7f7a12000640 1 -- 172.21.15.118:0/483533557 <== mon.1 v2:172.21.15.193:3300/0 5 ==== osd_map(90..90 src has 1..90) v4 ==== 8277+0+0 (secure 0 0 0) 0x7f7a040cf200 con 0x7f7a141028c0 2024-03-24T00:15:40.463 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:40.463+0000 7f7a19600640 1 --2- 172.21.15.118:0/483533557 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7f79f80a2800 0x7f79f80a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:15:40.464 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:40.464+0000 7f7a19600640 1 --2- 172.21.15.118:0/483533557 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7f79f80a2800 0x7f79f80a4cc0 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7f7a0002c170 tx=0x7f7a0002df90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:15:40.471 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:40.472+0000 7f7a12000640 1 -- 172.21.15.118:0/483533557 <== mon.1 v2:172.21.15.193:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f7a04016030 con 0x7f7a141028c0 2024-03-24T00:15:40.685 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:40.685+0000 7f7a1c000640 1 -- 172.21.15.118:0/483533557 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f7a140622d0 con 0x7f7a141028c0 2024-03-24T00:15:40.685 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:40.686+0000 7f7a12000640 1 -- 172.21.15.118:0/483533557 <== mon.1 v2:172.21.15.193:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v90) v1 ==== 104+0+11 (secure 0 0 0) 0x7f7a0409c210 con 0x7f7a141028c0 2024-03-24T00:15:40.685 INFO:teuthology.orchestra.run.smithi118.stdout:pg_num: 32 2024-03-24T00:15:40.688 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:40.688+0000 7f7a1c000640 1 -- 172.21.15.118:0/483533557 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7f79f80a2800 msgr2=0x7f79f80a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:15:40.689 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:40.689+0000 7f7a1c000640 1 --2- 172.21.15.118:0/483533557 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7f79f80a2800 0x7f79f80a4cc0 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7f7a0002c170 tx=0x7f7a0002df90 comp rx=0 tx=0).stop 2024-03-24T00:15:40.689 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:40.689+0000 7f7a1c000640 1 -- 172.21.15.118:0/483533557 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f7a141028c0 msgr2=0x7f7a14073a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:15:40.689 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:40.689+0000 7f7a1c000640 1 --2- 172.21.15.118:0/483533557 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f7a141028c0 0x7f7a14073a90 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7f7a0400eab0 tx=0x7f7a0400ef70 comp rx=0 tx=0).stop 2024-03-24T00:15:40.689 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:40.689+0000 7f7a1c000640 1 -- 172.21.15.118:0/483533557 shutdown_connections 2024-03-24T00:15:40.689 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:40.689+0000 7f7a1c000640 1 --2- 172.21.15.118:0/483533557 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7f79f80a2800 0x7f79f80a4cc0 unknown :-1 s=CLOSED pgs=253 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:15:40.689 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:40.689+0000 7f7a1c000640 1 --2- 172.21.15.118:0/483533557 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f7a14103280 0x7f7a14073fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:15:40.689 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:40.689+0000 7f7a1c000640 1 --2- 172.21.15.118:0/483533557 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f7a141028c0 0x7f7a14073a90 unknown :-1 s=CLOSED pgs=183 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:15:40.689 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:40.689+0000 7f7a1c000640 1 --2- 172.21.15.118:0/483533557 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7a141016d0 0x7f7a14075450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:15:40.689 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:40.689+0000 7f7a1c000640 1 -- 172.21.15.118:0/483533557 >> 172.21.15.118:0/483533557 conn(0x7f7a140fcc40 msgr2=0x7f7a14105bc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:15:40.689 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:40.689+0000 7f7a1c000640 1 -- 172.21.15.118:0/483533557 shutdown_connections 2024-03-24T00:15:40.689 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:40.689+0000 7f7a1c000640 1 -- 172.21.15.118:0/483533557 wait complete. 2024-03-24T00:15:40.711 DEBUG:teuthology.orchestra.run.smithi118:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-24T00:15:40.931 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:40.931+0000 7f6e63a00640 1 -- 172.21.15.118:0/1794190005 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6e5c1038e0 msgr2=0x7f6e5c103d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:15:40.931 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:40.931+0000 7f6e63a00640 1 --2- 172.21.15.118:0/1794190005 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6e5c1038e0 0x7f6e5c103d00 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7f6e50009900 tx=0x7f6e5002bc30 comp rx=0 tx=0).stop 2024-03-24T00:15:40.931 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:40.931+0000 7f6e63a00640 1 -- 172.21.15.118:0/1794190005 shutdown_connections 2024-03-24T00:15:40.931 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:40.931+0000 7f6e63a00640 1 --2- 172.21.15.118:0/1794190005 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f6e5c105490 0x7f6e5c10bd20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:15:40.931 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:40.931+0000 7f6e63a00640 1 --2- 172.21.15.118:0/1794190005 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f6e5c104ad0 0x7f6e5c104f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:15:40.931 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:40.931+0000 7f6e63a00640 1 --2- 172.21.15.118:0/1794190005 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6e5c1038e0 0x7f6e5c103d00 unknown :-1 s=CLOSED pgs=190 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:15:40.931 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:40.931+0000 7f6e63a00640 1 -- 172.21.15.118:0/1794190005 >> 172.21.15.118:0/1794190005 conn(0x7f6e5c0fee50 msgr2=0x7f6e5c101290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:15:40.931 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:40.931+0000 7f6e63a00640 1 -- 172.21.15.118:0/1794190005 shutdown_connections 2024-03-24T00:15:40.931 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:40.931+0000 7f6e63a00640 1 -- 172.21.15.118:0/1794190005 wait complete. 2024-03-24T00:15:40.932 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:40.932+0000 7f6e63a00640 1 Processor -- start 2024-03-24T00:15:40.932 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:40.932+0000 7f6e63a00640 1 -- start start 2024-03-24T00:15:40.934 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:40.933+0000 7f6e63a00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6e5c1038e0 0x7f6e5c199c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:15:40.934 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:40.933+0000 7f6e63a00640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f6e5c104ad0 0x7f6e5c19a180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:15:40.934 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:40.933+0000 7f6e63a00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f6e5c105490 0x7f6e5c1a11b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:15:40.934 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:40.933+0000 7f6e63a00640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6e5c10f580 con 0x7f6e5c104ad0 2024-03-24T00:15:40.934 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:40.933+0000 7f6e63a00640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6e5c10f400 con 0x7f6e5c1038e0 2024-03-24T00:15:40.934 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:40.933+0000 7f6e63a00640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6e5c10f700 con 0x7f6e5c105490 2024-03-24T00:15:40.934 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:40.933+0000 7f6e61000640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6e5c1038e0 0x7f6e5c199c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:15:40.934 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:40.933+0000 7f6e61a00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f6e5c105490 0x7f6e5c1a11b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:15:40.935 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:40.933+0000 7f6e61000640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6e5c1038e0 0x7f6e5c199c40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.118:46998/0 (socket says 172.21.15.118:46998) 2024-03-24T00:15:40.935 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:40.933+0000 7f6e61000640 1 -- 172.21.15.118:0/1112173243 learned_addr learned my addr 172.21.15.118:0/1112173243 (peer_addr_for_me v2:172.21.15.118:0/0) 2024-03-24T00:15:40.935 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:40.933+0000 7f6e5be00640 1 --2- 172.21.15.118:0/1112173243 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f6e5c104ad0 0x7f6e5c19a180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:15:40.935 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:40.934+0000 7f6e61a00640 1 -- 172.21.15.118:0/1112173243 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6e5c1038e0 msgr2=0x7f6e5c199c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:15:40.935 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:40.934+0000 7f6e61a00640 1 --2- 172.21.15.118:0/1112173243 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6e5c1038e0 0x7f6e5c199c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:15:40.935 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:40.934+0000 7f6e61a00640 1 -- 172.21.15.118:0/1112173243 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f6e5c104ad0 msgr2=0x7f6e5c19a180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:15:40.935 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:40.934+0000 7f6e61a00640 1 --2- 172.21.15.118:0/1112173243 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f6e5c104ad0 0x7f6e5c19a180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:15:40.935 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:40.934+0000 7f6e61a00640 1 -- 172.21.15.118:0/1112173243 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6e5c1a1850 con 0x7f6e5c105490 2024-03-24T00:15:40.935 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:40.934+0000 7f6e61000640 1 --2- 172.21.15.118:0/1112173243 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6e5c1038e0 0x7f6e5c199c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T00:15:40.935 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:40.934+0000 7f6e61a00640 1 --2- 172.21.15.118:0/1112173243 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f6e5c105490 0x7f6e5c1a11b0 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7f6e5400ec50 tx=0x7f6e5400c6a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:15:40.935 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:40.934+0000 7f6e59600640 1 -- 172.21.15.118:0/1112173243 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6e54003b70 con 0x7f6e5c105490 2024-03-24T00:15:40.935 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:40.934+0000 7f6e59600640 1 -- 172.21.15.118:0/1112173243 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f6e54003d10 con 0x7f6e5c105490 2024-03-24T00:15:40.935 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:40.934+0000 7f6e59600640 1 -- 172.21.15.118:0/1112173243 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6e54010970 con 0x7f6e5c105490 2024-03-24T00:15:40.936 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:40.934+0000 7f6e63a00640 1 -- 172.21.15.118:0/1112173243 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6e5c1a1b40 con 0x7f6e5c105490 2024-03-24T00:15:40.936 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:40.934+0000 7f6e63a00640 1 -- 172.21.15.118:0/1112173243 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6e5c1a2040 con 0x7f6e5c105490 2024-03-24T00:15:40.936 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:40.936+0000 7f6e63a00640 1 -- 172.21.15.118:0/1112173243 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6e5c10ac00 con 0x7f6e5c105490 2024-03-24T00:15:40.936 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:40.937+0000 7f6e59600640 1 -- 172.21.15.118:0/1112173243 <== mon.1 v2:172.21.15.193:3300/0 4 ==== mgrmap(e 19) v1 ==== 137342+0+0 (secure 0 0 0) 0x7f6e54003ec0 con 0x7f6e5c105490 2024-03-24T00:15:40.939 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:40.939+0000 7f6e59600640 1 --2- 172.21.15.118:0/1112173243 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7f6e440a2890 0x7f6e440a4d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:15:40.939 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:40.940+0000 7f6e59600640 1 -- 172.21.15.118:0/1112173243 <== mon.1 v2:172.21.15.193:3300/0 5 ==== osd_map(90..90 src has 1..90) v4 ==== 8277+0+0 (secure 0 0 0) 0x7f6e540cf210 con 0x7f6e5c105490 2024-03-24T00:15:40.939 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:40.940+0000 7f6e61000640 1 --2- 172.21.15.118:0/1112173243 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7f6e440a2890 0x7f6e440a4d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:15:40.940 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:40.940+0000 7f6e61000640 1 --2- 172.21.15.118:0/1112173243 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7f6e440a2890 0x7f6e440a4d50 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7f6e5c19b110 tx=0x7f6e5002df90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:15:40.942 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:40.942+0000 7f6e59600640 1 -- 172.21.15.118:0/1112173243 <== mon.1 v2:172.21.15.193:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f6e5409c0c0 con 0x7f6e5c105490 2024-03-24T00:15:41.180 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:41.180+0000 7f6e63a00640 1 -- 172.21.15.118:0/1112173243 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f6e5c02d220 con 0x7f6e5c105490 2024-03-24T00:15:41.181 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:41.181+0000 7f6e59600640 1 -- 172.21.15.118:0/1112173243 <== mon.1 v2:172.21.15.193:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v90) v1 ==== 100+0+11 (secure 0 0 0) 0x7f6e5409b9c0 con 0x7f6e5c105490 2024-03-24T00:15:41.181 INFO:teuthology.orchestra.run.smithi118.stdout:pg_num: 64 2024-03-24T00:15:41.183 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:41.184+0000 7f6e63a00640 1 -- 172.21.15.118:0/1112173243 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7f6e440a2890 msgr2=0x7f6e440a4d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:15:41.184 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:41.184+0000 7f6e63a00640 1 --2- 172.21.15.118:0/1112173243 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7f6e440a2890 0x7f6e440a4d50 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7f6e5c19b110 tx=0x7f6e5002df90 comp rx=0 tx=0).stop 2024-03-24T00:15:41.184 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:41.184+0000 7f6e63a00640 1 -- 172.21.15.118:0/1112173243 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f6e5c105490 msgr2=0x7f6e5c1a11b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:15:41.184 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:41.184+0000 7f6e63a00640 1 --2- 172.21.15.118:0/1112173243 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f6e5c105490 0x7f6e5c1a11b0 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7f6e5400ec50 tx=0x7f6e5400c6a0 comp rx=0 tx=0).stop 2024-03-24T00:15:41.184 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:41.184+0000 7f6e63a00640 1 -- 172.21.15.118:0/1112173243 shutdown_connections 2024-03-24T00:15:41.184 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:41.184+0000 7f6e63a00640 1 --2- 172.21.15.118:0/1112173243 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7f6e440a2890 0x7f6e440a4d50 unknown :-1 s=CLOSED pgs=254 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:15:41.184 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:41.184+0000 7f6e63a00640 1 --2- 172.21.15.118:0/1112173243 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f6e5c105490 0x7f6e5c1a11b0 unknown :-1 s=CLOSED pgs=184 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:15:41.184 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:41.184+0000 7f6e63a00640 1 --2- 172.21.15.118:0/1112173243 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f6e5c104ad0 0x7f6e5c19a180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:15:41.184 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:41.184+0000 7f6e63a00640 1 --2- 172.21.15.118:0/1112173243 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6e5c1038e0 0x7f6e5c199c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:15:41.184 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:41.184+0000 7f6e63a00640 1 -- 172.21.15.118:0/1112173243 >> 172.21.15.118:0/1112173243 conn(0x7f6e5c0fee50 msgr2=0x7f6e5c107dd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:15:41.184 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:41.184+0000 7f6e63a00640 1 -- 172.21.15.118:0/1112173243 shutdown_connections 2024-03-24T00:15:41.184 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:41.184+0000 7f6e63a00640 1 -- 172.21.15.118:0/1112173243 wait complete. 2024-03-24T00:15:41.195 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:15:40 smithi118 ceph-mon[31535]: from='client.25270 -' entity='client.admin' cmd=[{"prefix": "fs subvolumegroup pin", "vol_name": "cephfs", "group_name": "qa", "pin_type": "random", "pin_setting": "0.10", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T00:15:41.195 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:15:40 smithi118 ceph-mon[31535]: pgmap v397: 97 pgs: 97 active+clean; 645 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 5.9 KiB/s rd, 255 B/s wr, 11 op/s 2024-03-24T00:15:41.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:15:40 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T00:15:41.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:15:40 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T00:15:41.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:15:40 smithi118 ceph-mon[31535]: from='client.? 172.21.15.118:0/2070133610' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-24T00:15:41.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:15:40 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T00:15:41.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:15:40 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T00:15:41.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:15:40 smithi118 ceph-mon[31535]: from='client.? 172.21.15.118:0/483533557' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-24T00:15:41.203 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:15:40 smithi189 ceph-mon[36187]: from='client.25270 -' entity='client.admin' cmd=[{"prefix": "fs subvolumegroup pin", "vol_name": "cephfs", "group_name": "qa", "pin_type": "random", "pin_setting": "0.10", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T00:15:41.203 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:15:40 smithi189 ceph-mon[36187]: pgmap v397: 97 pgs: 97 active+clean; 645 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 5.9 KiB/s rd, 255 B/s wr, 11 op/s 2024-03-24T00:15:41.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:15:40 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T00:15:41.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:15:40 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T00:15:41.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:15:40 smithi189 ceph-mon[36187]: from='client.? 172.21.15.118:0/2070133610' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-24T00:15:41.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:15:40 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T00:15:41.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:15:40 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T00:15:41.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:15:40 smithi189 ceph-mon[36187]: from='client.? 172.21.15.118:0/483533557' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-24T00:15:41.205 DEBUG:teuthology.orchestra.run.smithi118:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-24T00:15:41.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:15:40 smithi193 ceph-mon[36878]: from='client.25270 -' entity='client.admin' cmd=[{"prefix": "fs subvolumegroup pin", "vol_name": "cephfs", "group_name": "qa", "pin_type": "random", "pin_setting": "0.10", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T00:15:41.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:15:40 smithi193 ceph-mon[36878]: pgmap v397: 97 pgs: 97 active+clean; 645 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 5.9 KiB/s rd, 255 B/s wr, 11 op/s 2024-03-24T00:15:41.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:15:40 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T00:15:41.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:15:40 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T00:15:41.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:15:40 smithi193 ceph-mon[36878]: from='client.? 172.21.15.118:0/2070133610' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-24T00:15:41.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:15:40 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T00:15:41.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:15:40 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T00:15:41.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:15:40 smithi193 ceph-mon[36878]: from='client.? 172.21.15.118:0/483533557' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-24T00:15:41.396 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:41.396+0000 7f031ae00640 1 -- 172.21.15.118:0/3945124546 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f03141016b0 msgr2=0x7f0314101ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:15:41.396 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:41.396+0000 7f031ae00640 1 --2- 172.21.15.118:0/3945124546 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f03141016b0 0x7f0314101ad0 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7f0304009900 tx=0x7f030402bc30 comp rx=0 tx=0).stop 2024-03-24T00:15:41.396 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:41.396+0000 7f031ae00640 1 -- 172.21.15.118:0/3945124546 shutdown_connections 2024-03-24T00:15:41.396 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:41.396+0000 7f031ae00640 1 --2- 172.21.15.118:0/3945124546 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f0314103260 0x7f0314109af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:15:41.396 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:41.396+0000 7f031ae00640 1 --2- 172.21.15.118:0/3945124546 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f03141028a0 0x7f0314102d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:15:41.397 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:41.396+0000 7f031ae00640 1 --2- 172.21.15.118:0/3945124546 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f03141016b0 0x7f0314101ad0 unknown :-1 s=CLOSED pgs=191 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:15:41.397 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:41.396+0000 7f031ae00640 1 -- 172.21.15.118:0/3945124546 >> 172.21.15.118:0/3945124546 conn(0x7f03140fcc80 msgr2=0x7f03140ff0a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:15:41.397 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:41.396+0000 7f031ae00640 1 -- 172.21.15.118:0/3945124546 shutdown_connections 2024-03-24T00:15:41.397 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:41.396+0000 7f031ae00640 1 -- 172.21.15.118:0/3945124546 wait complete. 2024-03-24T00:15:41.399 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:41.397+0000 7f031ae00640 1 Processor -- start 2024-03-24T00:15:41.399 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:41.398+0000 7f031ae00640 1 -- start start 2024-03-24T00:15:41.399 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:41.398+0000 7f031ae00640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f03141016b0 0x7f03141979c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:15:41.399 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:41.398+0000 7f031ae00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f03141028a0 0x7f0314197f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:15:41.399 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:41.398+0000 7f031ae00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0314103260 0x7f031419ef70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:15:41.399 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:41.398+0000 7f031ae00640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f031410d350 con 0x7f03141016b0 2024-03-24T00:15:41.400 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:41.398+0000 7f031ae00640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f031410d1d0 con 0x7f0314103260 2024-03-24T00:15:41.400 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:41.398+0000 7f031ae00640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f031410d4d0 con 0x7f03141028a0 2024-03-24T00:15:41.400 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:41.398+0000 7f0313400640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f03141028a0 0x7f0314197f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:15:41.400 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:41.398+0000 7f0313400640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f03141028a0 0x7f0314197f00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.118:57332/0 (socket says 172.21.15.118:57332) 2024-03-24T00:15:41.400 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:41.398+0000 7f0313e00640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f03141016b0 0x7f03141979c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:15:41.400 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:41.398+0000 7f0313400640 1 -- 172.21.15.118:0/3511182168 learned_addr learned my addr 172.21.15.118:0/3511182168 (peer_addr_for_me v2:172.21.15.118:0/0) 2024-03-24T00:15:41.400 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:41.398+0000 7f0318e00640 1 --2- 172.21.15.118:0/3511182168 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0314103260 0x7f031419ef70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:15:41.400 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:41.399+0000 7f0313400640 1 -- 172.21.15.118:0/3511182168 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0314103260 msgr2=0x7f031419ef70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:15:41.400 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:41.399+0000 7f0313400640 1 --2- 172.21.15.118:0/3511182168 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0314103260 0x7f031419ef70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:15:41.400 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:41.399+0000 7f0313400640 1 -- 172.21.15.118:0/3511182168 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f03141016b0 msgr2=0x7f03141979c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:15:41.400 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:41.399+0000 7f0313400640 1 --2- 172.21.15.118:0/3511182168 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f03141016b0 0x7f03141979c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:15:41.400 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:41.399+0000 7f0313400640 1 -- 172.21.15.118:0/3511182168 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f031419f5e0 con 0x7f03141028a0 2024-03-24T00:15:41.401 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:41.399+0000 7f0313400640 1 --2- 172.21.15.118:0/3511182168 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f03141028a0 0x7f0314197f00 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7f030000ea60 tx=0x7f030000ef20 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:15:41.401 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:41.399+0000 7f0310c00640 1 -- 172.21.15.118:0/3511182168 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f030000cde0 con 0x7f03141028a0 2024-03-24T00:15:41.401 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:41.399+0000 7f0310c00640 1 -- 172.21.15.118:0/3511182168 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f03000035f0 con 0x7f03141028a0 2024-03-24T00:15:41.401 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:41.399+0000 7f031ae00640 1 -- 172.21.15.118:0/3511182168 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f031419f8d0 con 0x7f03141028a0 2024-03-24T00:15:41.401 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:41.399+0000 7f0310c00640 1 -- 172.21.15.118:0/3511182168 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0300010670 con 0x7f03141028a0 2024-03-24T00:15:41.401 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:41.400+0000 7f031ae00640 1 -- 172.21.15.118:0/3511182168 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f031419fe70 con 0x7f03141028a0 2024-03-24T00:15:41.401 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:41.401+0000 7f031ae00640 1 -- 172.21.15.118:0/3511182168 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f03141089d0 con 0x7f03141028a0 2024-03-24T00:15:41.403 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:41.403+0000 7f0310c00640 1 -- 172.21.15.118:0/3511182168 <== mon.1 v2:172.21.15.193:3300/0 4 ==== mgrmap(e 19) v1 ==== 137342+0+0 (secure 0 0 0) 0x7f03000037a0 con 0x7f03141028a0 2024-03-24T00:15:41.404 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:41.405+0000 7f0310c00640 1 --2- 172.21.15.118:0/3511182168 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7f02f80a2910 0x7f02f80a4dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:15:41.404 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:41.405+0000 7f0310c00640 1 -- 172.21.15.118:0/3511182168 <== mon.1 v2:172.21.15.193:3300/0 5 ==== osd_map(90..90 src has 1..90) v4 ==== 8277+0+0 (secure 0 0 0) 0x7f03000cef40 con 0x7f03141028a0 2024-03-24T00:15:41.404 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:41.405+0000 7f0313e00640 1 --2- 172.21.15.118:0/3511182168 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7f02f80a2910 0x7f02f80a4dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:15:41.405 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:41.406+0000 7f0313e00640 1 --2- 172.21.15.118:0/3511182168 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7f02f80a2910 0x7f02f80a4dd0 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7f030402c170 tx=0x7f030402df90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:15:41.413 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:41.413+0000 7f0310c00640 1 -- 172.21.15.118:0/3511182168 <== mon.1 v2:172.21.15.193:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f030009be50 con 0x7f03141028a0 2024-03-24T00:15:41.671 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:41.671+0000 7f031ae00640 1 -- 172.21.15.118:0/3511182168 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f031402d2c0 con 0x7f03141028a0 2024-03-24T00:15:41.673 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:41.674+0000 7f0310c00640 1 -- 172.21.15.118:0/3511182168 <== mon.1 v2:172.21.15.193:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v90) v1 ==== 74+0+22834 (secure 0 0 0) 0x7f030009b750 con 0x7f03141028a0 2024-03-24T00:15:41.674 INFO:teuthology.orchestra.run.smithi118.stdout: 2024-03-24T00:15:41.674 INFO:teuthology.orchestra.run.smithi118.stdout:{"epoch":90,"fsid":"d79a5420-e971-11ee-95cd-87774f69a715","created":"2024-03-24T00:03:58.648524+0000","modified":"2024-03-24T00:15:07.721960+0000","last_up_change":"2024-03-24T00:12:03.809000+0000","last_in_change":"2024-03-24T00:11:44.389344+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-24T00:07:50.190965+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-24T00:12:59.993723+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-24T00:13:01.257347+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"86","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"c9590426-3901-4f68-a5ef-d57aa8fbac92","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6802","nonce":557690061},{"type":"v1","addr":"172.21.15.118:6803","nonce":557690061}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6804","nonce":557690061},{"type":"v1","addr":"172.21.15.118:6805","nonce":557690061}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6808","nonce":557690061},{"type":"v1","addr":"172.21.15.118:6809","nonce":557690061}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6806","nonce":557690061},{"type":"v1","addr":"172.21.15.118:6807","nonce":557690061}]},"public_addr":"172.21.15.118:6803/557690061","cluster_addr":"172.21.15.118:6805/557690061","heartbeat_back_addr":"172.21.15.118:6809/557690061","heartbeat_front_addr":"172.21.15.118:6807/557690061","state":["exists","up"]},{"osd":1,"uuid":"fc7f827e-ecf6-4661-aad0-7a22951892e0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6800","nonce":4203029485},{"type":"v1","addr":"172.21.15.189:6801","nonce":4203029485}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6802","nonce":4203029485},{"type":"v1","addr":"172.21.15.189:6803","nonce":4203029485}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6806","nonce":4203029485},{"type":"v1","addr":"172.21.15.189:6807","nonce":4203029485}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6804","nonce":4203029485},{"type":"v1","addr":"172.21.15.189:6805","nonce":4203029485}]},"public_addr":"172.21.15.189:6801/4203029485","cluster_addr":"172.21.15.189:6803/4203029485","heartbeat_back_addr":"172.21.15.189:6807/4203029485","heartbeat_front_addr":"172.21.15.189:6805/4203029485","state":["exists","up"]},{"osd":2,"uuid":"e73ccfe9-0e5b-4734-a154-a2f0d829fcfb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6800","nonce":3699076971},{"type":"v1","addr":"172.21.15.193:6801","nonce":3699076971}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6802","nonce":3699076971},{"type":"v1","addr":"172.21.15.193:6803","nonce":3699076971}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6806","nonce":3699076971},{"type":"v1","addr":"172.21.15.193:6807","nonce":3699076971}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6804","nonce":3699076971},{"type":"v1","addr":"172.21.15.193:6805","nonce":3699076971}]},"public_addr":"172.21.15.193:6801/3699076971","cluster_addr":"172.21.15.193:6803/3699076971","heartbeat_back_addr":"172.21.15.193:6807/3699076971","heartbeat_front_addr":"172.21.15.193:6805/3699076971","state":["exists","up"]},{"osd":3,"uuid":"63fed43b-43ce-4031-8957-7fe94725c1ed","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6810","nonce":4059617010},{"type":"v1","addr":"172.21.15.118:6811","nonce":4059617010}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6812","nonce":4059617010},{"type":"v1","addr":"172.21.15.118:6813","nonce":4059617010}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6816","nonce":4059617010},{"type":"v1","addr":"172.21.15.118:6817","nonce":4059617010}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6814","nonce":4059617010},{"type":"v1","addr":"172.21.15.118:6815","nonce":4059617010}]},"public_addr":"172.21.15.118:6811/4059617010","cluster_addr":"172.21.15.118:6813/4059617010","heartbeat_back_addr":"172.21.15.118:6817/4059617010","heartbeat_front_addr":"172.21.15.118:6815/4059617010","state":["exists","up"]},{"osd":4,"uuid":"57377f66-c3b4-43c1-8e8e-bc873672906d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":34,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6808","nonce":4075247068},{"type":"v1","addr":"172.21.15.189:6809","nonce":4075247068}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6810","nonce":4075247068},{"type":"v1","addr":"172.21.15.189:6811","nonce":4075247068}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6814","nonce":4075247068},{"type":"v1","addr":"172.21.15.189:6815","nonce":4075247068}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6812","nonce":4075247068},{"type":"v1","addr":"172.21.15.189:6813","nonce":4075247068}]},"public_addr":"172.21.15.189:6809/4075247068","cluster_addr":"172.21.15.189:6811/4075247068","heartbeat_back_addr":"172.21.15.189:6815/4075247068","heartbeat_front_addr":"172.21.15.189:6813/4075247068","state":["exists","up"]},{"osd":5,"uuid":"91970976-2193-4224-81c0-dc96cb022c85","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":39,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6808","nonce":2920363402},{"type":"v1","addr":"172.21.15.193:6809","nonce":2920363402}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6810","nonce":2920363402},{"type":"v1","addr":"172.21.15.193:6811","nonce":2920363402}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6814","nonce":2920363402},{"type":"v1","addr":"172.21.15.193:6815","nonce":2920363402}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6812","nonce":2920363402},{"type":"v1","addr":"172.21.15.193:6813","nonce":2920363402}]},"public_addr":"172.21.15.193:6809/2920363402","cluster_addr":"172.21.15.193:6811/2920363402","heartbeat_back_addr":"172.21.15.193:6815/2920363402","heartbeat_front_addr":"172.21.15.193:6813/2920363402","state":["exists","up"]},{"osd":6,"uuid":"0b77b196-8674-41c6-ad3d-726b5d01c8fd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":47,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6818","nonce":431354576},{"type":"v1","addr":"172.21.15.118:6819","nonce":431354576}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6820","nonce":431354576},{"type":"v1","addr":"172.21.15.118:6821","nonce":431354576}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6824","nonce":431354576},{"type":"v1","addr":"172.21.15.118:6825","nonce":431354576}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6822","nonce":431354576},{"type":"v1","addr":"172.21.15.118:6823","nonce":431354576}]},"public_addr":"172.21.15.118:6819/431354576","cluster_addr":"172.21.15.118:6821/431354576","heartbeat_back_addr":"172.21.15.118:6825/431354576","heartbeat_front_addr":"172.21.15.118:6823/431354576","state":["exists","up"]},{"osd":7,"uuid":"1fbbd38f-4e8e-4d4c-8e98-66f6862bc4bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":52,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6816","nonce":3667611284},{"type":"v1","addr":"172.21.15.189:6817","nonce":3667611284}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6818","nonce":3667611284},{"type":"v1","addr":"172.21.15.189:6819","nonce":3667611284}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6822","nonce":3667611284},{"type":"v1","addr":"172.21.15.189:6823","nonce":3667611284}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6820","nonce":3667611284},{"type":"v1","addr":"172.21.15.189:6821","nonce":3667611284}]},"public_addr":"172.21.15.189:6817/3667611284","cluster_addr":"172.21.15.189:6819/3667611284","heartbeat_back_addr":"172.21.15.189:6823/3667611284","heartbeat_front_addr":"172.21.15.189:6821/3667611284","state":["exists","up"]},{"osd":8,"uuid":"6462c2b5-e4b3-4b0b-bcfa-4de2c89e93e0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":60,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6816","nonce":3853800910},{"type":"v1","addr":"172.21.15.193:6817","nonce":3853800910}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6818","nonce":3853800910},{"type":"v1","addr":"172.21.15.193:6819","nonce":3853800910}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6822","nonce":3853800910},{"type":"v1","addr":"172.21.15.193:6823","nonce":3853800910}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6820","nonce":3853800910},{"type":"v1","addr":"172.21.15.193:6821","nonce":3853800910}]},"public_addr":"172.21.15.193:6817/3853800910","cluster_addr":"172.21.15.193:6819/3853800910","heartbeat_back_addr":"172.21.15.193:6823/3853800910","heartbeat_front_addr":"172.21.15.193:6821/3853800910","state":["exists","up"]},{"osd":9,"uuid":"54be277f-3596-422f-9de5-9005a8571c6f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":68,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6826","nonce":3248293936},{"type":"v1","addr":"172.21.15.118:6827","nonce":3248293936}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6828","nonce":3248293936},{"type":"v1","addr":"172.21.15.118:6829","nonce":3248293936}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6832","nonce":3248293936},{"type":"v1","addr":"172.21.15.118:6833","nonce":3248293936}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6830","nonce":3248293936},{"type":"v1","addr":"172.21.15.118:6831","nonce":3248293936}]},"public_addr":"172.21.15.118:6827/3248293936","cluster_addr":"172.21.15.118:6829/3248293936","heartbeat_back_addr":"172.21.15.118:6833/3248293936","heartbeat_front_addr":"172.21.15.118:6831/3248293936","state":["exists","up"]},{"osd":10,"uuid":"7ba69d69-ac41-44ce-bc39-c049a8ca5900","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":73,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6824","nonce":2112195760},{"type":"v1","addr":"172.21.15.189:6825","nonce":2112195760}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6826","nonce":2112195760},{"type":"v1","addr":"172.21.15.189:6827","nonce":2112195760}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6830","nonce":2112195760},{"type":"v1","addr":"172.21.15.189:6831","nonce":2112195760}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6828","nonce":2112195760},{"type":"v1","addr":"172.21.15.189:6829","nonce":2112195760}]},"public_addr":"172.21.15.189:6825/2112195760","cluster_addr":"172.21.15.189:6827/2112195760","heartbeat_back_addr":"172.21.15.189:6831/2112195760","heartbeat_front_addr":"172.21.15.189:6829/2112195760","state":["exists","up"]},{"osd":11,"uuid":"186f0b18-6b22-4eae-83b7-b21bbcce0817","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":81,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6824","nonce":3880515510},{"type":"v1","addr":"172.21.15.193:6825","nonce":3880515510}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6826","nonce":3880515510},{"type":"v1","addr":"172.21.15.193:6827","nonce":3880515510}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6830","nonce":3880515510},{"type":"v1","addr":"172.21.15.193:6831","nonce":3880515510}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6828","nonce":3880515510},{"type":"v1","addr":"172.21.15.193:6829","nonce":3880515510}]},"public_addr":"172.21.15.193:6825/3880515510","cluster_addr":"172.21.15.193:6827/3880515510","heartbeat_back_addr":"172.21.15.193:6831/3880515510","heartbeat_front_addr":"172.21.15.193:6829/3880515510","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:06:58.105543+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:07:23.795866+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:07:47.426050+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:08:19.427073+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:08:45.816082+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:09:11.081523+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:09:44.485077+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:10:11.338972+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:10:36.234024+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:11:09.730432+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:11:37.059083+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:12:02.371947+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.a","mappings":[{"from":2,"to":11}]},{"pgid":"2.14","mappings":[{"from":2,"to":4}]},{"pgid":"2.1d","mappings":[{"from":8,"to":11}]},{"pgid":"2.1f","mappings":[{"from":8,"to":1}]},{"pgid":"3.11","mappings":[{"from":4,"to":8},{"from":5,"to":9}]},{"pgid":"3.13","mappings":[{"from":4,"to":9}]},{"pgid":"3.14","mappings":[{"from":4,"to":6}]},{"pgid":"3.31","mappings":[{"from":4,"to":7}]},{"pgid":"3.38","mappings":[{"from":3,"to":9}]},{"pgid":"3.39","mappings":[{"from":5,"to":8}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.118:6801/389249272":"2024-03-25T00:05:05.458796+0000","172.21.15.118:6800/389249272":"2024-03-25T00:05:05.458796+0000","172.21.15.118:0/2101854694":"2024-03-25T00:05:05.458796+0000","172.21.15.118:0/630779359":"2024-03-25T00:05:05.458796+0000","172.21.15.118:0/2158921029":"2024-03-25T00:04:31.283725+0000","172.21.15.118:0/1003670455":"2024-03-25T00:05:05.458796+0000","172.21.15.118:0/368712314":"2024-03-25T00:04:31.283725+0000","172.21.15.118:0/2004775597":"2024-03-25T00:04:31.283725+0000","172.21.15.118:6800/1463147955":"2024-03-25T00:04:31.283725+0000","172.21.15.118:6801/1463147955":"2024-03-25T00:04:31.283725+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-24T00:15:41.677 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:41.677+0000 7f031ae00640 1 -- 172.21.15.118:0/3511182168 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7f02f80a2910 msgr2=0x7f02f80a4dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:15:41.677 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:41.677+0000 7f031ae00640 1 --2- 172.21.15.118:0/3511182168 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7f02f80a2910 0x7f02f80a4dd0 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7f030402c170 tx=0x7f030402df90 comp rx=0 tx=0).stop 2024-03-24T00:15:41.677 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:41.677+0000 7f031ae00640 1 -- 172.21.15.118:0/3511182168 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f03141028a0 msgr2=0x7f0314197f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:15:41.677 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:41.677+0000 7f031ae00640 1 --2- 172.21.15.118:0/3511182168 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f03141028a0 0x7f0314197f00 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7f030000ea60 tx=0x7f030000ef20 comp rx=0 tx=0).stop 2024-03-24T00:15:41.677 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:41.677+0000 7f031ae00640 1 -- 172.21.15.118:0/3511182168 shutdown_connections 2024-03-24T00:15:41.678 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:41.677+0000 7f031ae00640 1 --2- 172.21.15.118:0/3511182168 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7f02f80a2910 0x7f02f80a4dd0 unknown :-1 s=CLOSED pgs=255 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:15:41.678 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:41.677+0000 7f031ae00640 1 --2- 172.21.15.118:0/3511182168 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0314103260 0x7f031419ef70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:15:41.678 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:41.677+0000 7f031ae00640 1 --2- 172.21.15.118:0/3511182168 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f03141028a0 0x7f0314197f00 unknown :-1 s=CLOSED pgs=185 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:15:41.678 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:41.677+0000 7f031ae00640 1 --2- 172.21.15.118:0/3511182168 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f03141016b0 0x7f03141979c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:15:41.678 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:41.677+0000 7f031ae00640 1 -- 172.21.15.118:0/3511182168 >> 172.21.15.118:0/3511182168 conn(0x7f03140fcc80 msgr2=0x7f0314105ba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:15:41.678 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:41.678+0000 7f031ae00640 1 -- 172.21.15.118:0/3511182168 shutdown_connections 2024-03-24T00:15:41.678 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:41.678+0000 7f031ae00640 1 -- 172.21.15.118:0/3511182168 wait complete. 2024-03-24T00:15:41.698 INFO:tasks.fwd_scrub.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-24T00:07:50.190965+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-24T00:12:59.993723+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '85', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.8799999952316284, 'score_stable': 1.8799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.8799999952316284, 'raw_score_stable': 1.8799999952316284, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-03-24T00:13:01.257347+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '86', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-24T00:15:41.698 DEBUG:teuthology.orchestra.run.smithi118:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-24T00:15:41.899 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:41.899+0000 7fd571e00640 1 -- 172.21.15.118:0/1512873542 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fd56c100480 msgr2=0x7fd56c1008a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:15:41.900 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:41.899+0000 7fd571e00640 1 --2- 172.21.15.118:0/1512873542 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fd56c100480 0x7fd56c1008a0 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7fd560009900 tx=0x7fd56002bc30 comp rx=0 tx=0).stop 2024-03-24T00:15:41.900 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:41.899+0000 7fd571e00640 1 -- 172.21.15.118:0/1512873542 shutdown_connections 2024-03-24T00:15:41.900 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:41.899+0000 7fd571e00640 1 --2- 172.21.15.118:0/1512873542 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd56c102030 0x7fd56c1088c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:15:41.900 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:41.899+0000 7fd571e00640 1 --2- 172.21.15.118:0/1512873542 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fd56c101670 0x7fd56c101af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:15:41.900 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:41.899+0000 7fd571e00640 1 --2- 172.21.15.118:0/1512873542 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fd56c100480 0x7fd56c1008a0 unknown :-1 s=CLOSED pgs=186 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:15:41.900 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:41.899+0000 7fd571e00640 1 -- 172.21.15.118:0/1512873542 >> 172.21.15.118:0/1512873542 conn(0x7fd56c0fba10 msgr2=0x7fd56c0fde30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:15:41.900 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:41.900+0000 7fd571e00640 1 -- 172.21.15.118:0/1512873542 shutdown_connections 2024-03-24T00:15:41.900 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:41.900+0000 7fd571e00640 1 -- 172.21.15.118:0/1512873542 wait complete. 2024-03-24T00:15:41.901 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:41.901+0000 7fd571e00640 1 Processor -- start 2024-03-24T00:15:41.901 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:41.901+0000 7fd571e00640 1 -- start start 2024-03-24T00:15:41.902 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:41.902+0000 7fd571e00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd56c100480 0x7fd56c104410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:15:41.902 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:41.902+0000 7fd571e00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fd56c101670 0x7fd56c102a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:15:41.902 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:41.902+0000 7fd571e00640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fd56c102030 0x7fd56c102f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:15:41.902 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:41.902+0000 7fd571e00640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd56c10c160 con 0x7fd56c102030 2024-03-24T00:15:41.902 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:41.902+0000 7fd571e00640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd56c10bfe0 con 0x7fd56c100480 2024-03-24T00:15:41.902 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:41.902+0000 7fd571e00640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd56c10c2e0 con 0x7fd56c101670 2024-03-24T00:15:41.903 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:41.902+0000 7fd570a00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd56c100480 0x7fd56c104410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:15:41.903 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:41.902+0000 7fd570a00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd56c100480 0x7fd56c104410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.118:47038/0 (socket says 172.21.15.118:47038) 2024-03-24T00:15:41.903 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:41.902+0000 7fd570a00640 1 -- 172.21.15.118:0/3456630921 learned_addr learned my addr 172.21.15.118:0/3456630921 (peer_addr_for_me v2:172.21.15.118:0/0) 2024-03-24T00:15:41.903 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:41.902+0000 7fd56be00640 1 --2- 172.21.15.118:0/3456630921 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fd56c101670 0x7fd56c102a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:15:41.903 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:41.902+0000 7fd571400640 1 --2- 172.21.15.118:0/3456630921 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fd56c102030 0x7fd56c102f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:15:41.903 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:41.902+0000 7fd570a00640 1 -- 172.21.15.118:0/3456630921 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fd56c101670 msgr2=0x7fd56c102a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:15:41.903 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:41.902+0000 7fd570a00640 1 --2- 172.21.15.118:0/3456630921 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fd56c101670 0x7fd56c102a50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:15:41.903 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:41.902+0000 7fd570a00640 1 -- 172.21.15.118:0/3456630921 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fd56c102030 msgr2=0x7fd56c102f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:15:41.903 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:41.902+0000 7fd570a00640 1 --2- 172.21.15.118:0/3456630921 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fd56c102030 0x7fd56c102f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:15:41.903 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:41.902+0000 7fd570a00640 1 -- 172.21.15.118:0/3456630921 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd56c103820 con 0x7fd56c100480 2024-03-24T00:15:41.903 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:41.902+0000 7fd56be00640 1 --2- 172.21.15.118:0/3456630921 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fd56c101670 0x7fd56c102a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T00:15:41.903 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:41.903+0000 7fd570a00640 1 --2- 172.21.15.118:0/3456630921 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd56c100480 0x7fd56c104410 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7fd560000c00 tx=0x7fd56002c650 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:15:41.904 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:41.903+0000 7fd569600640 1 -- 172.21.15.118:0/3456630921 <== mon.2 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd560043020 con 0x7fd56c100480 2024-03-24T00:15:41.904 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:41.903+0000 7fd569600640 1 -- 172.21.15.118:0/3456630921 <== mon.2 v2:172.21.15.189:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fd5600054e0 con 0x7fd56c100480 2024-03-24T00:15:41.904 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:41.903+0000 7fd571e00640 1 -- 172.21.15.118:0/3456630921 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd56c1a19d0 con 0x7fd56c100480 2024-03-24T00:15:41.904 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:41.903+0000 7fd569600640 1 -- 172.21.15.118:0/3456630921 <== mon.2 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd560003c40 con 0x7fd56c100480 2024-03-24T00:15:41.904 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:41.903+0000 7fd571e00640 1 -- 172.21.15.118:0/3456630921 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd56c1a1f20 con 0x7fd56c100480 2024-03-24T00:15:41.904 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:41.905+0000 7fd571e00640 1 -- 172.21.15.118:0/3456630921 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd56c1a20f0 con 0x7fd56c100480 2024-03-24T00:15:41.907 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:41.907+0000 7fd569600640 1 -- 172.21.15.118:0/3456630921 <== mon.2 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 19) v1 ==== 137342+0+0 (secure 0 0 0) 0x7fd560046020 con 0x7fd56c100480 2024-03-24T00:15:41.908 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:41.909+0000 7fd569600640 1 --2- 172.21.15.118:0/3456630921 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7fd5540a2890 0x7fd5540a4d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:15:41.909 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:41.909+0000 7fd569600640 1 -- 172.21.15.118:0/3456630921 <== mon.2 v2:172.21.15.189:3300/0 5 ==== osd_map(90..90 src has 1..90) v4 ==== 8277+0+0 (secure 0 0 0) 0x7fd5600f0b90 con 0x7fd56c100480 2024-03-24T00:15:41.909 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:41.909+0000 7fd56be00640 1 --2- 172.21.15.118:0/3456630921 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7fd5540a2890 0x7fd5540a4d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:15:41.909 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:41.910+0000 7fd56be00640 1 --2- 172.21.15.118:0/3456630921 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7fd5540a2890 0x7fd5540a4d50 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7fd56c104180 tx=0x7fd558005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:15:41.917 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:41.917+0000 7fd569600640 1 -- 172.21.15.118:0/3456630921 <== mon.2 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7fd56004d020 con 0x7fd56c100480 2024-03-24T00:15:41.945 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:15:41 smithi118 ceph-mon[31535]: mgrmap e19: x(active, since 10m), standbys: z, y 2024-03-24T00:15:41.945 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:15:41 smithi118 ceph-mon[31535]: from='client.? 172.21.15.118:0/1112173243' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-24T00:15:41.945 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:15:41 smithi118 ceph-mon[31535]: from='client.? 172.21.15.118:0/3511182168' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-24T00:15:42.134 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:42.135+0000 7fd571e00640 1 -- 172.21.15.118:0/3456630921 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fd56c1a2700 con 0x7fd56c100480 2024-03-24T00:15:42.135 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:42.135+0000 7fd569600640 1 -- 172.21.15.118:0/3456630921 <== mon.2 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v90) v1 ==== 93+0+10 (secure 0 0 0) 0x7fd5600bdbb0 con 0x7fd56c100480 2024-03-24T00:15:42.135 INFO:teuthology.orchestra.run.smithi118.stdout:pg_num: 1 2024-03-24T00:15:42.138 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:42.138+0000 7fd571e00640 1 -- 172.21.15.118:0/3456630921 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7fd5540a2890 msgr2=0x7fd5540a4d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:15:42.138 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:42.138+0000 7fd571e00640 1 --2- 172.21.15.118:0/3456630921 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7fd5540a2890 0x7fd5540a4d50 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7fd56c104180 tx=0x7fd558005f90 comp rx=0 tx=0).stop 2024-03-24T00:15:42.138 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:42.138+0000 7fd571e00640 1 -- 172.21.15.118:0/3456630921 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd56c100480 msgr2=0x7fd56c104410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:15:42.138 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:42.138+0000 7fd571e00640 1 --2- 172.21.15.118:0/3456630921 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd56c100480 0x7fd56c104410 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7fd560000c00 tx=0x7fd56002c650 comp rx=0 tx=0).stop 2024-03-24T00:15:42.138 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:42.138+0000 7fd571e00640 1 -- 172.21.15.118:0/3456630921 shutdown_connections 2024-03-24T00:15:42.138 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:42.138+0000 7fd571e00640 1 --2- 172.21.15.118:0/3456630921 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7fd5540a2890 0x7fd5540a4d50 unknown :-1 s=CLOSED pgs=256 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:15:42.138 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:42.138+0000 7fd571e00640 1 --2- 172.21.15.118:0/3456630921 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fd56c102030 0x7fd56c102f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:15:42.139 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:42.138+0000 7fd571e00640 1 --2- 172.21.15.118:0/3456630921 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fd56c101670 0x7fd56c102a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:15:42.139 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:42.138+0000 7fd571e00640 1 --2- 172.21.15.118:0/3456630921 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd56c100480 0x7fd56c104410 unknown :-1 s=CLOSED pgs=192 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:15:42.139 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:42.138+0000 7fd571e00640 1 -- 172.21.15.118:0/3456630921 >> 172.21.15.118:0/3456630921 conn(0x7fd56c0fba10 msgr2=0x7fd56c0fd490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:15:42.139 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:42.138+0000 7fd571e00640 1 -- 172.21.15.118:0/3456630921 shutdown_connections 2024-03-24T00:15:42.139 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:42.138+0000 7fd571e00640 1 -- 172.21.15.118:0/3456630921 wait complete. 2024-03-24T00:15:42.157 DEBUG:teuthology.orchestra.run.smithi118:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-24T00:15:42.203 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:15:41 smithi189 ceph-mon[36187]: mgrmap e19: x(active, since 10m), standbys: z, y 2024-03-24T00:15:42.203 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:15:41 smithi189 ceph-mon[36187]: from='client.? 172.21.15.118:0/1112173243' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-24T00:15:42.203 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:15:41 smithi189 ceph-mon[36187]: from='client.? 172.21.15.118:0/3511182168' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-24T00:15:42.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:15:41 smithi193 ceph-mon[36878]: mgrmap e19: x(active, since 10m), standbys: z, y 2024-03-24T00:15:42.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:15:41 smithi193 ceph-mon[36878]: from='client.? 172.21.15.118:0/1112173243' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-24T00:15:42.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:15:41 smithi193 ceph-mon[36878]: from='client.? 172.21.15.118:0/3511182168' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-24T00:15:42.354 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:42.354+0000 7f5fa4400640 1 -- 172.21.15.118:0/3732503783 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f5f9c105490 msgr2=0x7f5f9c10bd20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:15:42.355 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:42.354+0000 7f5fa4400640 1 --2- 172.21.15.118:0/3732503783 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f5f9c105490 0x7f5f9c10bd20 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7f5f94009ca0 tx=0x7f5f9402c0f0 comp rx=0 tx=0).stop 2024-03-24T00:15:42.355 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:42.354+0000 7f5fa4400640 1 -- 172.21.15.118:0/3732503783 shutdown_connections 2024-03-24T00:15:42.355 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:42.354+0000 7f5fa4400640 1 --2- 172.21.15.118:0/3732503783 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f5f9c105490 0x7f5f9c10bd20 unknown :-1 s=CLOSED pgs=187 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:15:42.355 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:42.354+0000 7f5fa4400640 1 --2- 172.21.15.118:0/3732503783 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5f9c104ad0 0x7f5f9c104f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:15:42.355 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:42.354+0000 7f5fa4400640 1 --2- 172.21.15.118:0/3732503783 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f5f9c1038e0 0x7f5f9c103d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:15:42.355 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:42.354+0000 7f5fa4400640 1 -- 172.21.15.118:0/3732503783 >> 172.21.15.118:0/3732503783 conn(0x7f5f9c0fee50 msgr2=0x7f5f9c101290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:15:42.355 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:42.355+0000 7f5fa4400640 1 -- 172.21.15.118:0/3732503783 shutdown_connections 2024-03-24T00:15:42.355 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:42.355+0000 7f5fa4400640 1 -- 172.21.15.118:0/3732503783 wait complete. 2024-03-24T00:15:42.356 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:42.356+0000 7f5fa4400640 1 Processor -- start 2024-03-24T00:15:42.356 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:42.356+0000 7f5fa4400640 1 -- start start 2024-03-24T00:15:42.356 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:42.357+0000 7f5fa4400640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f5f9c1038e0 0x7f5f9c199bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:15:42.357 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:42.357+0000 7f5fa4400640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f5f9c104ad0 0x7f5f9c19a100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:15:42.357 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:42.357+0000 7f5fa4400640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5f9c105490 0x7f5f9c1a1170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:15:42.357 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:42.357+0000 7f5fa4400640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5f9c10f560 con 0x7f5f9c104ad0 2024-03-24T00:15:42.357 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:42.357+0000 7f5fa4400640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5f9c10f3e0 con 0x7f5f9c105490 2024-03-24T00:15:42.357 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:42.357+0000 7f5fa4400640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5f9c10f6e0 con 0x7f5f9c1038e0 2024-03-24T00:15:42.357 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:42.357+0000 7f5fa1a00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f5f9c1038e0 0x7f5f9c199bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:15:42.357 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:42.357+0000 7f5fa1a00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f5f9c1038e0 0x7f5f9c199bc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.118:57350/0 (socket says 172.21.15.118:57350) 2024-03-24T00:15:42.357 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:42.357+0000 7f5fa1a00640 1 -- 172.21.15.118:0/4277448487 learned_addr learned my addr 172.21.15.118:0/4277448487 (peer_addr_for_me v2:172.21.15.118:0/0) 2024-03-24T00:15:42.357 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:42.357+0000 7f5fa2400640 1 --2- 172.21.15.118:0/4277448487 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5f9c105490 0x7f5f9c1a1170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:15:42.357 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:42.357+0000 7f5fa1000640 1 --2- 172.21.15.118:0/4277448487 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f5f9c104ad0 0x7f5f9c19a100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:15:42.358 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:42.357+0000 7f5fa1a00640 1 -- 172.21.15.118:0/4277448487 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5f9c105490 msgr2=0x7f5f9c1a1170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:15:42.358 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:42.357+0000 7f5fa1a00640 1 --2- 172.21.15.118:0/4277448487 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5f9c105490 0x7f5f9c1a1170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:15:42.358 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:42.357+0000 7f5fa1a00640 1 -- 172.21.15.118:0/4277448487 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f5f9c104ad0 msgr2=0x7f5f9c19a100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:15:42.358 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:42.357+0000 7f5fa1a00640 1 --2- 172.21.15.118:0/4277448487 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f5f9c104ad0 0x7f5f9c19a100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:15:42.358 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:42.357+0000 7f5fa1a00640 1 -- 172.21.15.118:0/4277448487 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5f9c1a17e0 con 0x7f5f9c1038e0 2024-03-24T00:15:42.359 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:42.358+0000 7f5fa1a00640 1 --2- 172.21.15.118:0/4277448487 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f5f9c1038e0 0x7f5f9c199bc0 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7f5f9000d950 tx=0x7f5f9000de10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:15:42.359 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:42.358+0000 7f5f9a000640 1 -- 172.21.15.118:0/4277448487 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5f90017020 con 0x7f5f9c1038e0 2024-03-24T00:15:42.359 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:42.358+0000 7f5f9a000640 1 -- 172.21.15.118:0/4277448487 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f5f90009c30 con 0x7f5f9c1038e0 2024-03-24T00:15:42.359 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:42.358+0000 7f5f9a000640 1 -- 172.21.15.118:0/4277448487 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5f9000f720 con 0x7f5f9c1038e0 2024-03-24T00:15:42.359 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:42.358+0000 7f5fa4400640 1 -- 172.21.15.118:0/4277448487 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5f9c1a1ad0 con 0x7f5f9c1038e0 2024-03-24T00:15:42.359 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:42.358+0000 7f5fa4400640 1 -- 172.21.15.118:0/4277448487 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5f9c1a2070 con 0x7f5f9c1038e0 2024-03-24T00:15:42.359 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:42.359+0000 7f5fa4400640 1 -- 172.21.15.118:0/4277448487 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5f9c10aba0 con 0x7f5f9c1038e0 2024-03-24T00:15:42.362 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:42.363+0000 7f5f9a000640 1 -- 172.21.15.118:0/4277448487 <== mon.1 v2:172.21.15.193:3300/0 4 ==== mgrmap(e 19) v1 ==== 137342+0+0 (secure 0 0 0) 0x7f5f9001a020 con 0x7f5f9c1038e0 2024-03-24T00:15:42.363 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:42.364+0000 7f5f9a000640 1 --2- 172.21.15.118:0/4277448487 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7f5f840a2910 0x7f5f840a4dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:15:42.364 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:42.364+0000 7f5f9a000640 1 -- 172.21.15.118:0/4277448487 <== mon.1 v2:172.21.15.193:3300/0 5 ==== osd_map(90..90 src has 1..90) v4 ==== 8277+0+0 (secure 0 0 0) 0x7f5f900ceac0 con 0x7f5f9c1038e0 2024-03-24T00:15:42.364 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:42.364+0000 7f5fa1000640 1 --2- 172.21.15.118:0/4277448487 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7f5f840a2910 0x7f5f840a4dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:15:42.364 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:42.365+0000 7f5fa1000640 1 --2- 172.21.15.118:0/4277448487 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7f5f840a2910 0x7f5f840a4dd0 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7f5f9c19b0d0 tx=0x7f5f88005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:15:42.368 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:42.368+0000 7f5f9a000640 1 -- 172.21.15.118:0/4277448487 <== mon.1 v2:172.21.15.193:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f5f90021020 con 0x7f5f9c1038e0 2024-03-24T00:15:42.569 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:42.569+0000 7f5fa4400640 1 -- 172.21.15.118:0/4277448487 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f5f9c02d070 con 0x7f5f9c1038e0 2024-03-24T00:15:42.570 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:42.570+0000 7f5f9a000640 1 -- 172.21.15.118:0/4277448487 <== mon.1 v2:172.21.15.193:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v90) v1 ==== 104+0+11 (secure 0 0 0) 0x7f5f9c02d070 con 0x7f5f9c1038e0 2024-03-24T00:15:42.570 INFO:teuthology.orchestra.run.smithi118.stdout:pg_num: 32 2024-03-24T00:15:42.572 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:42.572+0000 7f5fa4400640 1 -- 172.21.15.118:0/4277448487 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7f5f840a2910 msgr2=0x7f5f840a4dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:15:42.572 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:42.572+0000 7f5fa4400640 1 --2- 172.21.15.118:0/4277448487 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7f5f840a2910 0x7f5f840a4dd0 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7f5f9c19b0d0 tx=0x7f5f88005f90 comp rx=0 tx=0).stop 2024-03-24T00:15:42.573 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:42.572+0000 7f5fa4400640 1 -- 172.21.15.118:0/4277448487 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f5f9c1038e0 msgr2=0x7f5f9c199bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:15:42.573 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:42.572+0000 7f5fa4400640 1 --2- 172.21.15.118:0/4277448487 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f5f9c1038e0 0x7f5f9c199bc0 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7f5f9000d950 tx=0x7f5f9000de10 comp rx=0 tx=0).stop 2024-03-24T00:15:42.573 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:42.573+0000 7f5fa4400640 1 -- 172.21.15.118:0/4277448487 shutdown_connections 2024-03-24T00:15:42.573 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:42.573+0000 7f5fa4400640 1 --2- 172.21.15.118:0/4277448487 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7f5f840a2910 0x7f5f840a4dd0 unknown :-1 s=CLOSED pgs=257 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:15:42.573 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:42.573+0000 7f5fa4400640 1 --2- 172.21.15.118:0/4277448487 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5f9c105490 0x7f5f9c1a1170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:15:42.573 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:42.573+0000 7f5fa4400640 1 --2- 172.21.15.118:0/4277448487 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f5f9c104ad0 0x7f5f9c19a100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:15:42.573 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:42.573+0000 7f5fa4400640 1 --2- 172.21.15.118:0/4277448487 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f5f9c1038e0 0x7f5f9c199bc0 unknown :-1 s=CLOSED pgs=188 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:15:42.573 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:42.573+0000 7f5fa4400640 1 -- 172.21.15.118:0/4277448487 >> 172.21.15.118:0/4277448487 conn(0x7f5f9c0fee50 msgr2=0x7f5f9c107d70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:15:42.573 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:42.573+0000 7f5fa4400640 1 -- 172.21.15.118:0/4277448487 shutdown_connections 2024-03-24T00:15:42.573 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:42.573+0000 7f5fa4400640 1 -- 172.21.15.118:0/4277448487 wait complete. 2024-03-24T00:15:42.589 DEBUG:teuthology.orchestra.run.smithi118:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-24T00:15:42.779 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:42.779+0000 7fa1d8a00640 1 -- 172.21.15.118:0/800510298 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fa1d4102120 msgr2=0x7fa1d4104510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:15:42.779 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:42.779+0000 7fa1d8a00640 1 --2- 172.21.15.118:0/800510298 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fa1d4102120 0x7fa1d4104510 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7fa1c4009900 tx=0x7fa1c402bc50 comp rx=0 tx=0).stop 2024-03-24T00:15:42.779 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:42.779+0000 7fa1d8a00640 1 -- 172.21.15.118:0/800510298 shutdown_connections 2024-03-24T00:15:42.779 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:42.779+0000 7fa1d8a00640 1 --2- 172.21.15.118:0/800510298 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fa1d4107380 0x7fa1d41097d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:15:42.779 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:42.779+0000 7fa1d8a00640 1 --2- 172.21.15.118:0/800510298 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa1d4104a50 0x7fa1d4106e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:15:42.779 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:42.779+0000 7fa1d8a00640 1 --2- 172.21.15.118:0/800510298 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fa1d4102120 0x7fa1d4104510 unknown :-1 s=CLOSED pgs=189 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:15:42.779 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:42.779+0000 7fa1d8a00640 1 -- 172.21.15.118:0/800510298 >> 172.21.15.118:0/800510298 conn(0x7fa1d40fbb30 msgr2=0x7fa1d40fdf50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:15:42.780 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:42.779+0000 7fa1d8a00640 1 -- 172.21.15.118:0/800510298 shutdown_connections 2024-03-24T00:15:42.780 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:42.779+0000 7fa1d8a00640 1 -- 172.21.15.118:0/800510298 wait complete. 2024-03-24T00:15:42.782 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:42.780+0000 7fa1d8a00640 1 Processor -- start 2024-03-24T00:15:42.782 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:42.780+0000 7fa1d8a00640 1 -- start start 2024-03-24T00:15:42.782 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:42.781+0000 7fa1d8a00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fa1d4102120 0x7fa1d4101a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:15:42.782 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:42.781+0000 7fa1d8a00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa1d4104a50 0x7fa1d41000d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:15:42.782 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:42.781+0000 7fa1d8a00640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fa1d4107380 0x7fa1d4100610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:15:42.782 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:42.781+0000 7fa1d8a00640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa1d410d380 con 0x7fa1d4107380 2024-03-24T00:15:42.782 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:42.781+0000 7fa1d8a00640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa1d410d200 con 0x7fa1d4104a50 2024-03-24T00:15:42.782 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:42.781+0000 7fa1d8a00640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa1d410d500 con 0x7fa1d4102120 2024-03-24T00:15:42.783 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:42.781+0000 7fa1d1e00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fa1d4102120 0x7fa1d4101a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:15:42.783 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:42.781+0000 7fa1d1e00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fa1d4102120 0x7fa1d4101a90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.118:57374/0 (socket says 172.21.15.118:57374) 2024-03-24T00:15:42.783 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:42.781+0000 7fa1d1e00640 1 -- 172.21.15.118:0/754829360 learned_addr learned my addr 172.21.15.118:0/754829360 (peer_addr_for_me v2:172.21.15.118:0/0) 2024-03-24T00:15:42.783 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:42.781+0000 7fa1d1400640 1 --2- 172.21.15.118:0/754829360 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa1d4104a50 0x7fa1d41000d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:15:42.783 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:42.781+0000 7fa1d2800640 1 --2- 172.21.15.118:0/754829360 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fa1d4107380 0x7fa1d4100610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:15:42.783 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:42.781+0000 7fa1d1e00640 1 -- 172.21.15.118:0/754829360 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa1d4104a50 msgr2=0x7fa1d41000d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:15:42.783 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:42.781+0000 7fa1d1e00640 1 --2- 172.21.15.118:0/754829360 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa1d4104a50 0x7fa1d41000d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:15:42.783 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:42.781+0000 7fa1d1e00640 1 -- 172.21.15.118:0/754829360 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fa1d4107380 msgr2=0x7fa1d4100610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:15:42.783 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:42.781+0000 7fa1d1e00640 1 --2- 172.21.15.118:0/754829360 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fa1d4107380 0x7fa1d4100610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:15:42.783 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:42.781+0000 7fa1d1e00640 1 -- 172.21.15.118:0/754829360 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa1d4100ea0 con 0x7fa1d4102120 2024-03-24T00:15:42.783 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:42.781+0000 7fa1d1400640 1 --2- 172.21.15.118:0/754829360 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa1d4104a50 0x7fa1d41000d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T00:15:42.783 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:42.781+0000 7fa1d2800640 1 --2- 172.21.15.118:0/754829360 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fa1d4107380 0x7fa1d4100610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T00:15:42.783 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:42.782+0000 7fa1d1e00640 1 --2- 172.21.15.118:0/754829360 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fa1d4102120 0x7fa1d4101a90 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7fa1c4005ab0 tx=0x7fa1c402c6e0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:15:42.784 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:42.782+0000 7fa1caa00640 1 -- 172.21.15.118:0/754829360 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa1c4041070 con 0x7fa1d4102120 2024-03-24T00:15:42.784 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:42.782+0000 7fa1caa00640 1 -- 172.21.15.118:0/754829360 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fa1c4004450 con 0x7fa1d4102120 2024-03-24T00:15:42.784 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:42.782+0000 7fa1d8a00640 1 -- 172.21.15.118:0/754829360 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa1d41a1b40 con 0x7fa1d4102120 2024-03-24T00:15:42.784 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:42.782+0000 7fa1caa00640 1 -- 172.21.15.118:0/754829360 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa1c4034440 con 0x7fa1d4102120 2024-03-24T00:15:42.784 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:42.782+0000 7fa1d8a00640 1 -- 172.21.15.118:0/754829360 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa1d41a20e0 con 0x7fa1d4102120 2024-03-24T00:15:42.784 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:42.783+0000 7fa1d8a00640 1 -- 172.21.15.118:0/754829360 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa1d40fd620 con 0x7fa1d4102120 2024-03-24T00:15:42.785 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:42.786+0000 7fa1caa00640 1 -- 172.21.15.118:0/754829360 <== mon.1 v2:172.21.15.193:3300/0 4 ==== mgrmap(e 19) v1 ==== 137342+0+0 (secure 0 0 0) 0x7fa1c4044070 con 0x7fa1d4102120 2024-03-24T00:15:42.787 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:42.787+0000 7fa1caa00640 1 --2- 172.21.15.118:0/754829360 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7fa1b80a2910 0x7fa1b80a4dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:15:42.787 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:42.787+0000 7fa1caa00640 1 -- 172.21.15.118:0/754829360 <== mon.1 v2:172.21.15.193:3300/0 5 ==== osd_map(90..90 src has 1..90) v4 ==== 8277+0+0 (secure 0 0 0) 0x7fa1c40f06f0 con 0x7fa1d4102120 2024-03-24T00:15:42.787 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:42.788+0000 7fa1d1400640 1 --2- 172.21.15.118:0/754829360 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7fa1b80a2910 0x7fa1b80a4dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:15:42.787 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:42.788+0000 7fa1d1400640 1 --2- 172.21.15.118:0/754829360 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7fa1b80a2910 0x7fa1b80a4dd0 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7fa1c00095f0 tx=0x7fa1c0005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:15:42.795 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:42.796+0000 7fa1caa00640 1 -- 172.21.15.118:0/754829360 <== mon.1 v2:172.21.15.193:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7fa1c40bd710 con 0x7fa1d4102120 2024-03-24T00:15:42.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:15:42 smithi118 ceph-mon[31535]: pgmap v398: 97 pgs: 97 active+clean; 645 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 341 B/s wr, 9 op/s 2024-03-24T00:15:42.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:15:42 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T00:15:42.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:15:42 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T00:15:42.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:15:42 smithi118 ceph-mon[31535]: from='client.? 172.21.15.118:0/3456630921' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-24T00:15:42.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:15:42 smithi118 ceph-mon[31535]: from='client.? 172.21.15.118:0/4277448487' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-24T00:15:42.997 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:42.997+0000 7fa1d8a00640 1 -- 172.21.15.118:0/754829360 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7fa1d402d2c0 con 0x7fa1d4102120 2024-03-24T00:15:42.998 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:42.997+0000 7fa1caa00640 1 -- 172.21.15.118:0/754829360 <== mon.1 v2:172.21.15.193:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v90) v1 ==== 100+0+11 (secure 0 0 0) 0x7fa1c40bd010 con 0x7fa1d4102120 2024-03-24T00:15:42.998 INFO:teuthology.orchestra.run.smithi118.stdout:pg_num: 64 2024-03-24T00:15:42.999 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:43.000+0000 7fa1d8a00640 1 -- 172.21.15.118:0/754829360 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7fa1b80a2910 msgr2=0x7fa1b80a4dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:15:42.999 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:43.000+0000 7fa1d8a00640 1 --2- 172.21.15.118:0/754829360 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7fa1b80a2910 0x7fa1b80a4dd0 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7fa1c00095f0 tx=0x7fa1c0005f90 comp rx=0 tx=0).stop 2024-03-24T00:15:43.000 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:43.000+0000 7fa1d8a00640 1 -- 172.21.15.118:0/754829360 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fa1d4102120 msgr2=0x7fa1d4101a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:15:43.000 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:43.000+0000 7fa1d8a00640 1 --2- 172.21.15.118:0/754829360 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fa1d4102120 0x7fa1d4101a90 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7fa1c4005ab0 tx=0x7fa1c402c6e0 comp rx=0 tx=0).stop 2024-03-24T00:15:43.000 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:43.000+0000 7fa1d8a00640 1 -- 172.21.15.118:0/754829360 shutdown_connections 2024-03-24T00:15:43.000 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:43.000+0000 7fa1d8a00640 1 --2- 172.21.15.118:0/754829360 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7fa1b80a2910 0x7fa1b80a4dd0 unknown :-1 s=CLOSED pgs=258 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:15:43.000 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:43.000+0000 7fa1d8a00640 1 --2- 172.21.15.118:0/754829360 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fa1d4107380 0x7fa1d4100610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:15:43.000 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:43.000+0000 7fa1d8a00640 1 --2- 172.21.15.118:0/754829360 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa1d4104a50 0x7fa1d41000d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:15:43.000 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:43.000+0000 7fa1d8a00640 1 --2- 172.21.15.118:0/754829360 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fa1d4102120 0x7fa1d4101a90 unknown :-1 s=CLOSED pgs=190 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:15:43.000 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:43.000+0000 7fa1d8a00640 1 -- 172.21.15.118:0/754829360 >> 172.21.15.118:0/754829360 conn(0x7fa1d40fbb30 msgr2=0x7fa1d4102a60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:15:43.000 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:43.000+0000 7fa1d8a00640 1 -- 172.21.15.118:0/754829360 shutdown_connections 2024-03-24T00:15:43.000 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:43.000+0000 7fa1d8a00640 1 -- 172.21.15.118:0/754829360 wait complete. 2024-03-24T00:15:43.018 INFO:tasks.fwd_scrub:Wait for all MDSs to reach steady state... 2024-03-24T00:15:43.019 DEBUG:teuthology.orchestra.run.smithi118:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-24T00:15:43.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:15:42 smithi189 ceph-mon[36187]: pgmap v398: 97 pgs: 97 active+clean; 645 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 341 B/s wr, 9 op/s 2024-03-24T00:15:43.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:15:42 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T00:15:43.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:15:42 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T00:15:43.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:15:42 smithi189 ceph-mon[36187]: from='client.? 172.21.15.118:0/3456630921' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-24T00:15:43.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:15:42 smithi189 ceph-mon[36187]: from='client.? 172.21.15.118:0/4277448487' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-24T00:15:43.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:15:42 smithi193 ceph-mon[36878]: pgmap v398: 97 pgs: 97 active+clean; 645 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 341 B/s wr, 9 op/s 2024-03-24T00:15:43.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:15:42 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T00:15:43.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:15:42 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T00:15:43.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:15:42 smithi193 ceph-mon[36878]: from='client.? 172.21.15.118:0/3456630921' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-24T00:15:43.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:15:42 smithi193 ceph-mon[36878]: from='client.? 172.21.15.118:0/4277448487' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-24T00:15:43.217 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:43.218+0000 7f3b3a200640 1 -- 172.21.15.118:0/657858669 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3b341026d0 msgr2=0x7f3b34104b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:15:43.218 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:43.218+0000 7f3b3a200640 1 --2- 172.21.15.118:0/657858669 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3b341026d0 0x7f3b34104b40 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7f3b20009900 tx=0x7f3b2002bc30 comp rx=0 tx=0).stop 2024-03-24T00:15:43.218 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:43.218+0000 7f3b3a200640 1 -- 172.21.15.118:0/657858669 shutdown_connections 2024-03-24T00:15:43.218 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:43.218+0000 7f3b3a200640 1 --2- 172.21.15.118:0/657858669 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f3b34105080 0x7f3b34105500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:15:43.218 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:43.218+0000 7f3b3a200640 1 --2- 172.21.15.118:0/657858669 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3b341026d0 0x7f3b34104b40 unknown :-1 s=CLOSED pgs=193 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:15:43.218 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:43.218+0000 7f3b3a200640 1 --2- 172.21.15.118:0/657858669 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f3b340ffd60 0x7f3b34102190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:15:43.218 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:43.218+0000 7f3b3a200640 1 -- 172.21.15.118:0/657858669 >> 172.21.15.118:0/657858669 conn(0x7f3b340fb980 msgr2=0x7f3b340fdda0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:15:43.218 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:43.218+0000 7f3b3a200640 1 -- 172.21.15.118:0/657858669 shutdown_connections 2024-03-24T00:15:43.218 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:43.218+0000 7f3b3a200640 1 -- 172.21.15.118:0/657858669 wait complete. 2024-03-24T00:15:43.219 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:43.220+0000 7f3b3a200640 1 Processor -- start 2024-03-24T00:15:43.220 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:43.220+0000 7f3b3a200640 1 -- start start 2024-03-24T00:15:43.222 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:43.221+0000 7f3b3a200640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f3b340ffd60 0x7f3b3419ddf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:15:43.222 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:43.221+0000 7f3b3a200640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3b341026d0 0x7f3b3419e330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:15:43.222 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:43.221+0000 7f3b3a200640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f3b34105080 0x7f3b341a53a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:15:43.222 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:43.221+0000 7f3b3a200640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3b341137b0 con 0x7f3b340ffd60 2024-03-24T00:15:43.222 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:43.221+0000 7f3b3a200640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3b34113630 con 0x7f3b341026d0 2024-03-24T00:15:43.222 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:43.221+0000 7f3b3a200640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3b34113930 con 0x7f3b34105080 2024-03-24T00:15:43.223 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:43.221+0000 7f3b39800640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f3b34105080 0x7f3b341a53a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:15:43.223 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:43.221+0000 7f3b33e00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3b341026d0 0x7f3b3419e330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:15:43.223 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:43.221+0000 7f3b38e00640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f3b340ffd60 0x7f3b3419ddf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:15:43.223 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:43.221+0000 7f3b39800640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f3b34105080 0x7f3b341a53a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.118:33928/0 (socket says 172.21.15.118:33928) 2024-03-24T00:15:43.223 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:43.221+0000 7f3b39800640 1 -- 172.21.15.118:0/1702815803 learned_addr learned my addr 172.21.15.118:0/1702815803 (peer_addr_for_me v2:172.21.15.118:0/0) 2024-03-24T00:15:43.223 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:43.222+0000 7f3b33e00640 1 -- 172.21.15.118:0/1702815803 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f3b34105080 msgr2=0x7f3b341a53a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:15:43.223 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:43.222+0000 7f3b33e00640 1 --2- 172.21.15.118:0/1702815803 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f3b34105080 0x7f3b341a53a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:15:43.223 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:43.222+0000 7f3b33e00640 1 -- 172.21.15.118:0/1702815803 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f3b340ffd60 msgr2=0x7f3b3419ddf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:15:43.223 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:43.222+0000 7f3b33e00640 1 --2- 172.21.15.118:0/1702815803 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f3b340ffd60 0x7f3b3419ddf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:15:43.223 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:43.222+0000 7f3b33e00640 1 -- 172.21.15.118:0/1702815803 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3b341a5a10 con 0x7f3b341026d0 2024-03-24T00:15:43.223 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:43.222+0000 7f3b38e00640 1 --2- 172.21.15.118:0/1702815803 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f3b340ffd60 0x7f3b3419ddf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T00:15:43.223 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:43.222+0000 7f3b33e00640 1 --2- 172.21.15.118:0/1702815803 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3b341026d0 0x7f3b3419e330 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7f3b20036040 tx=0x7f3b2002c6c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:15:43.224 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:43.222+0000 7f3b31600640 1 -- 172.21.15.118:0/1702815803 <== mon.2 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3b20043020 con 0x7f3b341026d0 2024-03-24T00:15:43.224 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:43.222+0000 7f3b31600640 1 -- 172.21.15.118:0/1702815803 <== mon.2 v2:172.21.15.189:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f3b200054e0 con 0x7f3b341026d0 2024-03-24T00:15:43.224 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:43.222+0000 7f3b3a200640 1 -- 172.21.15.118:0/1702815803 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3b341a5ca0 con 0x7f3b341026d0 2024-03-24T00:15:43.224 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:43.222+0000 7f3b31600640 1 -- 172.21.15.118:0/1702815803 <== mon.2 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3b20003ca0 con 0x7f3b341026d0 2024-03-24T00:15:43.224 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:43.223+0000 7f3b3a200640 1 -- 172.21.15.118:0/1702815803 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3b341a61c0 con 0x7f3b341026d0 2024-03-24T00:15:43.224 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:43.223+0000 7f3b3a200640 1 -- 172.21.15.118:0/1702815803 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3b340fd470 con 0x7f3b341026d0 2024-03-24T00:15:43.225 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:43.226+0000 7f3b31600640 1 -- 172.21.15.118:0/1702815803 <== mon.2 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 19) v1 ==== 137342+0+0 (secure 0 0 0) 0x7f3b20046020 con 0x7f3b341026d0 2024-03-24T00:15:43.226 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:43.227+0000 7f3b31600640 1 --2- 172.21.15.118:0/1702815803 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7f3b1c0a2910 0x7f3b1c0a4dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:15:43.226 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:43.227+0000 7f3b31600640 1 -- 172.21.15.118:0/1702815803 <== mon.2 v2:172.21.15.189:3300/0 5 ==== osd_map(90..90 src has 1..90) v4 ==== 8277+0+0 (secure 0 0 0) 0x7f3b200f11b0 con 0x7f3b341026d0 2024-03-24T00:15:43.227 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:43.227+0000 7f3b38e00640 1 --2- 172.21.15.118:0/1702815803 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7f3b1c0a2910 0x7f3b1c0a4dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:15:43.227 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:43.228+0000 7f3b38e00640 1 --2- 172.21.15.118:0/1702815803 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7f3b1c0a2910 0x7f3b1c0a4dd0 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7f3b28009c20 tx=0x7f3b28005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:15:43.234 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:43.234+0000 7f3b31600640 1 -- 172.21.15.118:0/1702815803 <== mon.2 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f3b2004d020 con 0x7f3b341026d0 2024-03-24T00:15:43.460 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:43.460+0000 7f3b3a200640 1 -- 172.21.15.118:0/1702815803 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f3b3402d070 con 0x7f3b341026d0 2024-03-24T00:15:43.461 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:43.461+0000 7f3b31600640 1 -- 172.21.15.118:0/1702815803 <== mon.2 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+11478 (secure 0 0 0) 0x7f3b200be170 con 0x7f3b341026d0 2024-03-24T00:15:43.461 INFO:teuthology.orchestra.run.smithi118.stdout: 2024-03-24T00:15:43.461 INFO:teuthology.orchestra.run.smithi118.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14385,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.118:6835/3194741373","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6834","nonce":3194741373},{"type":"v1","addr":"172.21.15.118:6835","nonce":3194741373}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":4},{"gid":14397,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.118:6837/1374462956","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6836","nonce":1374462956},{"type":"v1","addr":"172.21.15.118:6837","nonce":1374462956}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":28,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-03-24T00:13:01.782474+0000","modified":"2024-03-24T00:15:38.235450+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24569,"mds_1":24550,"mds_2":24511,"mds_3":24499,"mds_4":24488},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14469":{"gid":14469,"name":"a","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.118:6841/3040946537","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6840","nonce":3040946537},{"type":"v1","addr":"172.21.15.118:6841","nonce":3040946537}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24439":{"gid":24439,"name":"l","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.193:6833/1023428412","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6832","nonce":1023428412},{"type":"v1","addr":"172.21.15.193:6833","nonce":1023428412}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24440":{"gid":24440,"name":"k","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.189:6833/2740024078","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6832","nonce":2740024078},{"type":"v1","addr":"172.21.15.189:6833","nonce":2740024078}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24488":{"gid":24488,"name":"e","rank":4,"incarnation":25,"state":"up:active","state_seq":10,"addr":"172.21.15.189:6835/599473205","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6834","nonce":599473205},{"type":"v1","addr":"172.21.15.189:6835","nonce":599473205}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24499":{"gid":24499,"name":"f","rank":3,"incarnation":23,"state":"up:active","state_seq":9,"addr":"172.21.15.193:6835/2055310541","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6834","nonce":2055310541},{"type":"v1","addr":"172.21.15.193:6835","nonce":2055310541}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24511":{"gid":24511,"name":"c","rank":2,"incarnation":21,"state":"up:active","state_seq":8,"addr":"172.21.15.193:6837/2921520053","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6836","nonce":2921520053},{"type":"v1","addr":"172.21.15.193:6837","nonce":2921520053}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24515":{"gid":24515,"name":"h","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.189:6837/3450187137","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6836","nonce":3450187137},{"type":"v1","addr":"172.21.15.189:6837","nonce":3450187137}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24532":{"gid":24532,"name":"i","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.193:6839/2126842603","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6838","nonce":2126842603},{"type":"v1","addr":"172.21.15.193:6839","nonce":2126842603}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24550":{"gid":24550,"name":"g","rank":1,"incarnation":19,"state":"up:active","state_seq":6,"addr":"172.21.15.118:6839/3285788503","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6838","nonce":3285788503},{"type":"v1","addr":"172.21.15.118:6839","nonce":3285788503}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24569":{"gid":24569,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.189:6839/4074484168","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6838","nonce":4074484168},{"type":"v1","addr":"172.21.15.189:6839","nonce":4074484168}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24569,"qdb_cluster":[24569,24499,24488,24511,24550]},"id":1}]} 2024-03-24T00:15:43.462 INFO:teuthology.orchestra.run.smithi118.stderr:dumped fsmap epoch 28 2024-03-24T00:15:43.463 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:43.464+0000 7f3b3a200640 1 -- 172.21.15.118:0/1702815803 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7f3b1c0a2910 msgr2=0x7f3b1c0a4dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:15:43.464 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:43.464+0000 7f3b3a200640 1 --2- 172.21.15.118:0/1702815803 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7f3b1c0a2910 0x7f3b1c0a4dd0 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7f3b28009c20 tx=0x7f3b28005f90 comp rx=0 tx=0).stop 2024-03-24T00:15:43.464 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:43.464+0000 7f3b3a200640 1 -- 172.21.15.118:0/1702815803 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3b341026d0 msgr2=0x7f3b3419e330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:15:43.464 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:43.464+0000 7f3b3a200640 1 --2- 172.21.15.118:0/1702815803 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3b341026d0 0x7f3b3419e330 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7f3b20036040 tx=0x7f3b2002c6c0 comp rx=0 tx=0).stop 2024-03-24T00:15:43.464 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:43.464+0000 7f3b3a200640 1 -- 172.21.15.118:0/1702815803 shutdown_connections 2024-03-24T00:15:43.464 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:43.464+0000 7f3b3a200640 1 --2- 172.21.15.118:0/1702815803 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7f3b1c0a2910 0x7f3b1c0a4dd0 unknown :-1 s=CLOSED pgs=259 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:15:43.464 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:43.464+0000 7f3b3a200640 1 --2- 172.21.15.118:0/1702815803 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f3b34105080 0x7f3b341a53a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:15:43.464 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:43.464+0000 7f3b3a200640 1 --2- 172.21.15.118:0/1702815803 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3b341026d0 0x7f3b3419e330 unknown :-1 s=CLOSED pgs=194 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:15:43.464 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:43.464+0000 7f3b3a200640 1 --2- 172.21.15.118:0/1702815803 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f3b340ffd60 0x7f3b3419ddf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:15:43.464 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:43.464+0000 7f3b3a200640 1 -- 172.21.15.118:0/1702815803 >> 172.21.15.118:0/1702815803 conn(0x7f3b340fb980 msgr2=0x7f3b34100a10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:15:43.464 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:43.464+0000 7f3b3a200640 1 -- 172.21.15.118:0/1702815803 shutdown_connections 2024-03-24T00:15:43.464 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:43.464+0000 7f3b3a200640 1 -- 172.21.15.118:0/1702815803 wait complete. 2024-03-24T00:15:43.483 INFO:tasks.fwd_scrub:Ready to start scrub thrashing 2024-03-24T00:15:43.483 INFO:tasks.fwd_scrub.ceph_manager:waiting for clean 2024-03-24T00:15:43.483 DEBUG:teuthology.orchestra.run.smithi118:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-24T00:15:43.682 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:43.682+0000 7f7d36e00640 1 -- 172.21.15.118:0/550562964 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7d300756e0 msgr2=0x7f7d30073b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:15:43.682 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:43.682+0000 7f7d36e00640 1 --2- 172.21.15.118:0/550562964 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7d300756e0 0x7f7d30073b30 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7f7d24009930 tx=0x7f7d2402bc20 comp rx=0 tx=0).stop 2024-03-24T00:15:43.682 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:43.682+0000 7f7d36e00640 1 -- 172.21.15.118:0/550562964 shutdown_connections 2024-03-24T00:15:43.682 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:43.682+0000 7f7d36e00640 1 --2- 172.21.15.118:0/550562964 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f7d3010dab0 0x7f7d3010fea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:15:43.682 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:43.682+0000 7f7d36e00640 1 --2- 172.21.15.118:0/550562964 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f7d30074100 0x7f7d30074580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:15:43.682 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:43.682+0000 7f7d36e00640 1 --2- 172.21.15.118:0/550562964 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7d300756e0 0x7f7d30073b30 unknown :-1 s=CLOSED pgs=195 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:15:43.682 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:43.682+0000 7f7d36e00640 1 -- 172.21.15.118:0/550562964 >> 172.21.15.118:0/550562964 conn(0x7f7d300fca30 msgr2=0x7f7d300fee90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:15:43.683 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:43.682+0000 7f7d36e00640 1 -- 172.21.15.118:0/550562964 shutdown_connections 2024-03-24T00:15:43.683 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:43.682+0000 7f7d36e00640 1 -- 172.21.15.118:0/550562964 wait complete. 2024-03-24T00:15:43.684 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:43.683+0000 7f7d36e00640 1 Processor -- start 2024-03-24T00:15:43.684 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:43.684+0000 7f7d36e00640 1 -- start start 2024-03-24T00:15:43.684 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:43.684+0000 7f7d36e00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f7d30074100 0x7f7d3019de70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:15:43.684 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:43.684+0000 7f7d36e00640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f7d300756e0 0x7f7d3019e3b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:15:43.684 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:43.684+0000 7f7d36e00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7d3010dab0 0x7f7d301a5420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:15:43.684 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:43.684+0000 7f7d36e00640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7d30113780 con 0x7f7d300756e0 2024-03-24T00:15:43.684 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:43.684+0000 7f7d36e00640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7d30113600 con 0x7f7d3010dab0 2024-03-24T00:15:43.684 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:43.684+0000 7f7d36e00640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7d30113900 con 0x7f7d30074100 2024-03-24T00:15:43.684 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:43.684+0000 7f7d35a00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f7d30074100 0x7f7d3019de70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:15:43.684 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:43.684+0000 7f7d35a00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f7d30074100 0x7f7d3019de70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.118:33942/0 (socket says 172.21.15.118:33942) 2024-03-24T00:15:43.684 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:43.684+0000 7f7d35a00640 1 -- 172.21.15.118:0/2198692274 learned_addr learned my addr 172.21.15.118:0/2198692274 (peer_addr_for_me v2:172.21.15.118:0/0) 2024-03-24T00:15:43.684 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:43.684+0000 7f7d36400640 1 --2- 172.21.15.118:0/2198692274 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7d3010dab0 0x7f7d301a5420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:15:43.686 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:43.685+0000 7f7d35000640 1 --2- 172.21.15.118:0/2198692274 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f7d300756e0 0x7f7d3019e3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:15:43.686 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:43.685+0000 7f7d36400640 1 -- 172.21.15.118:0/2198692274 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f7d30074100 msgr2=0x7f7d3019de70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:15:43.686 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:43.685+0000 7f7d36400640 1 --2- 172.21.15.118:0/2198692274 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f7d30074100 0x7f7d3019de70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:15:43.686 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:43.685+0000 7f7d36400640 1 -- 172.21.15.118:0/2198692274 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f7d300756e0 msgr2=0x7f7d3019e3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:15:43.686 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:43.685+0000 7f7d36400640 1 --2- 172.21.15.118:0/2198692274 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f7d300756e0 0x7f7d3019e3b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:15:43.686 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:43.685+0000 7f7d36400640 1 -- 172.21.15.118:0/2198692274 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7d301a5a90 con 0x7f7d3010dab0 2024-03-24T00:15:43.686 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:43.685+0000 7f7d35000640 1 --2- 172.21.15.118:0/2198692274 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f7d300756e0 0x7f7d3019e3b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-24T00:15:43.686 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:43.685+0000 7f7d36400640 1 --2- 172.21.15.118:0/2198692274 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7d3010dab0 0x7f7d301a5420 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7f7d28009fd0 tx=0x7f7d2800ef10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:15:43.686 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:43.685+0000 7f7d2e000640 1 -- 172.21.15.118:0/2198692274 <== mon.2 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7d2800ce20 con 0x7f7d3010dab0 2024-03-24T00:15:43.686 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:43.685+0000 7f7d2e000640 1 -- 172.21.15.118:0/2198692274 <== mon.2 v2:172.21.15.189:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f7d28005020 con 0x7f7d3010dab0 2024-03-24T00:15:43.686 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:43.686+0000 7f7d2e000640 1 -- 172.21.15.118:0/2198692274 <== mon.2 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7d28010720 con 0x7f7d3010dab0 2024-03-24T00:15:43.687 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:43.686+0000 7f7d36e00640 1 -- 172.21.15.118:0/2198692274 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7d301a5d80 con 0x7f7d3010dab0 2024-03-24T00:15:43.687 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:43.686+0000 7f7d36e00640 1 -- 172.21.15.118:0/2198692274 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7d301a6270 con 0x7f7d3010dab0 2024-03-24T00:15:43.687 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:43.687+0000 7f7d36e00640 1 -- 172.21.15.118:0/2198692274 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7d30103c80 con 0x7f7d3010dab0 2024-03-24T00:15:43.689 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:43.690+0000 7f7d2e000640 1 -- 172.21.15.118:0/2198692274 <== mon.2 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 19) v1 ==== 137342+0+0 (secure 0 0 0) 0x7f7d28005540 con 0x7f7d3010dab0 2024-03-24T00:15:43.691 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:43.691+0000 7f7d2e000640 1 --2- 172.21.15.118:0/2198692274 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7f7d180a2910 0x7f7d180a4dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:15:43.691 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:43.692+0000 7f7d2e000640 1 -- 172.21.15.118:0/2198692274 <== mon.2 v2:172.21.15.189:3300/0 5 ==== osd_map(90..90 src has 1..90) v4 ==== 8277+0+0 (secure 0 0 0) 0x7f7d280cf370 con 0x7f7d3010dab0 2024-03-24T00:15:43.691 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:43.692+0000 7f7d35a00640 1 --2- 172.21.15.118:0/2198692274 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7f7d180a2910 0x7f7d180a4dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:15:43.692 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:43.692+0000 7f7d35a00640 1 --2- 172.21.15.118:0/2198692274 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7f7d180a2910 0x7f7d180a4dd0 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7f7d24006010 tx=0x7f7d2402c870 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:15:43.695 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:43.695+0000 7f7d2e000640 1 -- 172.21.15.118:0/2198692274 <== mon.2 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f7d2809c220 con 0x7f7d3010dab0 2024-03-24T00:15:43.894 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:43.895+0000 7f7d36e00640 1 -- 172.21.15.118:0/2198692274 --> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f7d30061ae0 con 0x7f7d180a2910 2024-03-24T00:15:43.908 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:43.908+0000 7f7d2e000640 1 -- 172.21.15.118:0/2198692274 <== mgr.14152 v2:172.21.15.118:6800/2112912943 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+259216 (secure 0 0 0) 0x7f7d30061ae0 con 0x7f7d180a2910 2024-03-24T00:15:43.912 INFO:teuthology.orchestra.run.smithi118.stdout: 2024-03-24T00:15:43.913 INFO:teuthology.orchestra.run.smithi118.stderr:dumped all 2024-03-24T00:15:43.915 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:43.913+0000 7f7d36e00640 1 -- 172.21.15.118:0/2198692274 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7f7d180a2910 msgr2=0x7f7d180a4dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:15:43.915 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:43.913+0000 7f7d36e00640 1 --2- 172.21.15.118:0/2198692274 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7f7d180a2910 0x7f7d180a4dd0 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7f7d24006010 tx=0x7f7d2402c870 comp rx=0 tx=0).stop 2024-03-24T00:15:43.915 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:43.913+0000 7f7d36e00640 1 -- 172.21.15.118:0/2198692274 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7d3010dab0 msgr2=0x7f7d301a5420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:15:43.915 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:43.913+0000 7f7d36e00640 1 --2- 172.21.15.118:0/2198692274 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7d3010dab0 0x7f7d301a5420 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7f7d28009fd0 tx=0x7f7d2800ef10 comp rx=0 tx=0).stop 2024-03-24T00:15:43.915 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:43.913+0000 7f7d36e00640 1 -- 172.21.15.118:0/2198692274 shutdown_connections 2024-03-24T00:15:43.915 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:43.913+0000 7f7d36e00640 1 --2- 172.21.15.118:0/2198692274 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7f7d180a2910 0x7f7d180a4dd0 unknown :-1 s=CLOSED pgs=260 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:15:43.916 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:43.913+0000 7f7d36e00640 1 --2- 172.21.15.118:0/2198692274 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7d3010dab0 0x7f7d301a5420 unknown :-1 s=CLOSED pgs=196 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:15:43.916 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:43.913+0000 7f7d36e00640 1 --2- 172.21.15.118:0/2198692274 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f7d300756e0 0x7f7d3019e3b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:15:43.916 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:43.913+0000 7f7d36e00640 1 --2- 172.21.15.118:0/2198692274 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f7d30074100 0x7f7d3019de70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:15:43.916 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:43.913+0000 7f7d36e00640 1 -- 172.21.15.118:0/2198692274 >> 172.21.15.118:0/2198692274 conn(0x7f7d300fca30 msgr2=0x7f7d30100e50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:15:43.916 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:43.913+0000 7f7d36e00640 1 -- 172.21.15.118:0/2198692274 shutdown_connections 2024-03-24T00:15:43.916 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:43.913+0000 7f7d36e00640 1 -- 172.21.15.118:0/2198692274 wait complete. 2024-03-24T00:15:43.934 INFO:teuthology.orchestra.run.smithi118.stdout:{"pg_ready":true,"pg_map":{"version":399,"stamp":"2024-03-24T00:15:43.668833+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":661126,"num_objects":102,"num_object_clones":0,"num_object_copies":306,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":102,"num_whiteouts":0,"num_read":1689,"num_read_kb":965,"num_write":437,"num_write_kb":3089,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":1243109,"num_keys_recovered":10,"num_objects_omap":66,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":860,"ondisk_log_size":860,"up":291,"acting":291,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":291,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":357084,"kb_used_data":12064,"kb_used_omap":18,"kb_used_meta":344877,"kb_avail":1124584740,"statfs":{"total":1151940427776,"available":1151574773760,"internally_reserved":0,"allocated":12353536,"data_stored":7707531,"data_compressed":260586,"data_compressed_allocated":352256,"data_compressed_original":2236416,"omap_allocated":19142,"internal_metadata":353154362},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":1088,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":122,"num_read_kb":62,"num_write":8,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004105"},"pg_stats":[{"pgid":"3.3f","version":"88'8","reported_seq":30,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-24T00:15:07.728049+0000","last_change":"2024-03-24T00:13:02.325815+0000","last_active":"2024-03-24T00:15:07.728049+0000","last_peered":"2024-03-24T00:15:07.728049+0000","last_clean":"2024-03-24T00:15:07.728049+0000","last_became_active":"2024-03-24T00:13:02.325450+0000","last_became_peered":"2024-03-24T00:13:02.325450+0000","last_unstale":"2024-03-24T00:15:07.728049+0000","last_undegraded":"2024-03-24T00:15:07.728049+0000","last_fullsized":"2024-03-24T00:15:07.728049+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:13:01.267198+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:13:01.267198+0000","last_clean_scrub_stamp":"2024-03-24T00:13:01.267198+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T02:18:21.994322+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.3e","version":"88'4","reported_seq":26,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-24T00:15:07.750213+0000","last_change":"2024-03-24T00:13:02.324015+0000","last_active":"2024-03-24T00:15:07.750213+0000","last_peered":"2024-03-24T00:15:07.750213+0000","last_clean":"2024-03-24T00:15:07.750213+0000","last_became_active":"2024-03-24T00:13:02.323668+0000","last_became_peered":"2024-03-24T00:13:02.323668+0000","last_unstale":"2024-03-24T00:15:07.750213+0000","last_undegraded":"2024-03-24T00:15:07.750213+0000","last_fullsized":"2024-03-24T00:15:07.750213+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:13:01.267198+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:13:01.267198+0000","last_clean_scrub_stamp":"2024-03-24T00:13:01.267198+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:36:19.442266+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11,10],"acting":[0,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3d","version":"88'7","reported_seq":29,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-24T00:15:07.749167+0000","last_change":"2024-03-24T00:13:02.328911+0000","last_active":"2024-03-24T00:15:07.749167+0000","last_peered":"2024-03-24T00:15:07.749167+0000","last_clean":"2024-03-24T00:15:07.749167+0000","last_became_active":"2024-03-24T00:13:02.324333+0000","last_became_peered":"2024-03-24T00:13:02.324333+0000","last_unstale":"2024-03-24T00:15:07.749167+0000","last_undegraded":"2024-03-24T00:15:07.749167+0000","last_fullsized":"2024-03-24T00:15:07.749167+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:13:01.267198+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:13:01.267198+0000","last_clean_scrub_stamp":"2024-03-24T00:13:01.267198+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T02:49:02.318097+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"88'8","reported_seq":30,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-24T00:15:07.748082+0000","last_change":"2024-03-24T00:13:02.309057+0000","last_active":"2024-03-24T00:15:07.748082+0000","last_peered":"2024-03-24T00:15:07.748082+0000","last_clean":"2024-03-24T00:15:07.748082+0000","last_became_active":"2024-03-24T00:13:02.308403+0000","last_became_peered":"2024-03-24T00:13:02.308403+0000","last_unstale":"2024-03-24T00:15:07.748082+0000","last_undegraded":"2024-03-24T00:15:07.748082+0000","last_fullsized":"2024-03-24T00:15:07.748082+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:13:01.267198+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:13:01.267198+0000","last_clean_scrub_stamp":"2024-03-24T00:13:01.267198+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T07:54:44.310833+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,8,2],"acting":[4,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.3b","version":"88'4","reported_seq":26,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-24T00:15:07.729685+0000","last_change":"2024-03-24T00:13:02.330651+0000","last_active":"2024-03-24T00:15:07.729685+0000","last_peered":"2024-03-24T00:15:07.729685+0000","last_clean":"2024-03-24T00:15:07.729685+0000","last_became_active":"2024-03-24T00:13:02.330313+0000","last_became_peered":"2024-03-24T00:13:02.330313+0000","last_unstale":"2024-03-24T00:15:07.729685+0000","last_undegraded":"2024-03-24T00:15:07.729685+0000","last_fullsized":"2024-03-24T00:15:07.729685+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:13:01.267198+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:13:01.267198+0000","last_clean_scrub_stamp":"2024-03-24T00:13:01.267198+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T03:45:38.480929+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,6],"acting":[8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3a","version":"88'6","reported_seq":28,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-24T00:15:07.747612+0000","last_change":"2024-03-24T00:13:02.313115+0000","last_active":"2024-03-24T00:15:07.747612+0000","last_peered":"2024-03-24T00:15:07.747612+0000","last_clean":"2024-03-24T00:15:07.747612+0000","last_became_active":"2024-03-24T00:13:02.312387+0000","last_became_peered":"2024-03-24T00:13:02.312387+0000","last_unstale":"2024-03-24T00:15:07.747612+0000","last_undegraded":"2024-03-24T00:15:07.747612+0000","last_fullsized":"2024-03-24T00:15:07.747612+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:13:01.267198+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:13:01.267198+0000","last_clean_scrub_stamp":"2024-03-24T00:13:01.267198+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T09:16:02.574920+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,4],"acting":[10,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.39","version":"88'5","reported_seq":33,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-24T00:15:07.729862+0000","last_change":"2024-03-24T00:14:07.503364+0000","last_active":"2024-03-24T00:15:07.729862+0000","last_peered":"2024-03-24T00:15:07.729862+0000","last_clean":"2024-03-24T00:15:07.729862+0000","last_became_active":"2024-03-24T00:14:07.503103+0000","last_became_peered":"2024-03-24T00:14:07.503103+0000","last_unstale":"2024-03-24T00:15:07.729862+0000","last_undegraded":"2024-03-24T00:15:07.729862+0000","last_fullsized":"2024-03-24T00:15:07.729862+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:13:01.267198+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:13:01.267198+0000","last_clean_scrub_stamp":"2024-03-24T00:13:01.267198+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T03:22:04.225634+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,8],"acting":[7,9,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"88'9","reported_seq":24,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-24T00:15:07.729082+0000","last_change":"2024-03-24T00:14:07.681011+0000","last_active":"2024-03-24T00:15:07.729082+0000","last_peered":"2024-03-24T00:15:07.729082+0000","last_clean":"2024-03-24T00:15:07.729082+0000","last_became_active":"2024-03-24T00:14:07.680726+0000","last_became_peered":"2024-03-24T00:14:07.680726+0000","last_unstale":"2024-03-24T00:15:07.729082+0000","last_undegraded":"2024-03-24T00:15:07.729082+0000","last_fullsized":"2024-03-24T00:15:07.729082+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:13:01.267198+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:13:01.267198+0000","last_clean_scrub_stamp":"2024-03-24T00:13:01.267198+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T01:05:03.379937+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,5,10],"acting":[9,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.37","version":"88'11","reported_seq":33,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-24T00:15:07.727279+0000","last_change":"2024-03-24T00:13:02.305625+0000","last_active":"2024-03-24T00:15:07.727279+0000","last_peered":"2024-03-24T00:15:07.727279+0000","last_clean":"2024-03-24T00:15:07.727279+0000","last_became_active":"2024-03-24T00:13:02.305221+0000","last_became_peered":"2024-03-24T00:13:02.305221+0000","last_unstale":"2024-03-24T00:15:07.727279+0000","last_undegraded":"2024-03-24T00:15:07.727279+0000","last_fullsized":"2024-03-24T00:15:07.727279+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:13:01.267198+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:13:01.267198+0000","last_clean_scrub_stamp":"2024-03-24T00:13:01.267198+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T04:37:12.467387+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,11],"acting":[1,5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.36","version":"88'16","reported_seq":38,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-24T00:15:07.747823+0000","last_change":"2024-03-24T00:13:02.307878+0000","last_active":"2024-03-24T00:15:07.747823+0000","last_peered":"2024-03-24T00:15:07.747823+0000","last_clean":"2024-03-24T00:15:07.747823+0000","last_became_active":"2024-03-24T00:13:02.307513+0000","last_became_peered":"2024-03-24T00:13:02.307513+0000","last_unstale":"2024-03-24T00:15:07.747823+0000","last_undegraded":"2024-03-24T00:15:07.747823+0000","last_fullsized":"2024-03-24T00:15:07.747823+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:13:01.267198+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:13:01.267198+0000","last_clean_scrub_stamp":"2024-03-24T00:13:01.267198+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T07:06:08.162416+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,4],"acting":[10,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.35","version":"88'6","reported_seq":28,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-24T00:15:07.729986+0000","last_change":"2024-03-24T00:13:02.310883+0000","last_active":"2024-03-24T00:15:07.729986+0000","last_peered":"2024-03-24T00:15:07.729986+0000","last_clean":"2024-03-24T00:15:07.729986+0000","last_became_active":"2024-03-24T00:13:02.310559+0000","last_became_peered":"2024-03-24T00:13:02.310559+0000","last_unstale":"2024-03-24T00:15:07.729986+0000","last_undegraded":"2024-03-24T00:15:07.729986+0000","last_fullsized":"2024-03-24T00:15:07.729986+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:13:01.267198+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:13:01.267198+0000","last_clean_scrub_stamp":"2024-03-24T00:13:01.267198+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T04:04:25.801248+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8,1],"acting":[7,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.34","version":"88'7","reported_seq":29,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-24T00:15:07.748067+0000","last_change":"2024-03-24T00:13:02.308747+0000","last_active":"2024-03-24T00:15:07.748067+0000","last_peered":"2024-03-24T00:15:07.748067+0000","last_clean":"2024-03-24T00:15:07.748067+0000","last_became_active":"2024-03-24T00:13:02.307791+0000","last_became_peered":"2024-03-24T00:13:02.307791+0000","last_unstale":"2024-03-24T00:15:07.748067+0000","last_undegraded":"2024-03-24T00:15:07.748067+0000","last_fullsized":"2024-03-24T00:15:07.748067+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:13:01.267198+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:13:01.267198+0000","last_clean_scrub_stamp":"2024-03-24T00:13:01.267198+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T03:28:08.780044+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,8],"acting":[4,10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"88'10","reported_seq":32,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-24T00:15:07.751605+0000","last_change":"2024-03-24T00:13:02.327061+0000","last_active":"2024-03-24T00:15:07.751605+0000","last_peered":"2024-03-24T00:15:07.751605+0000","last_clean":"2024-03-24T00:15:07.751605+0000","last_became_active":"2024-03-24T00:13:02.326696+0000","last_became_peered":"2024-03-24T00:13:02.326696+0000","last_unstale":"2024-03-24T00:15:07.751605+0000","last_undegraded":"2024-03-24T00:15:07.751605+0000","last_fullsized":"2024-03-24T00:15:07.751605+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:13:01.267198+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:13:01.267198+0000","last_clean_scrub_stamp":"2024-03-24T00:13:01.267198+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T08:55:17.325488+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,5],"acting":[6,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.32","version":"88'7","reported_seq":29,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-24T00:15:07.748893+0000","last_change":"2024-03-24T00:13:02.321326+0000","last_active":"2024-03-24T00:15:07.748893+0000","last_peered":"2024-03-24T00:15:07.748893+0000","last_clean":"2024-03-24T00:15:07.748893+0000","last_became_active":"2024-03-24T00:13:02.320964+0000","last_became_peered":"2024-03-24T00:13:02.320964+0000","last_unstale":"2024-03-24T00:15:07.748893+0000","last_undegraded":"2024-03-24T00:15:07.748893+0000","last_fullsized":"2024-03-24T00:15:07.748893+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:13:01.267198+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:13:01.267198+0000","last_clean_scrub_stamp":"2024-03-24T00:13:01.267198+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T08:05:15.365809+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,10],"acting":[0,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"88'7","reported_seq":35,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-24T00:15:07.748624+0000","last_change":"2024-03-24T00:14:07.505774+0000","last_active":"2024-03-24T00:15:07.748624+0000","last_peered":"2024-03-24T00:15:07.748624+0000","last_clean":"2024-03-24T00:15:07.748624+0000","last_became_active":"2024-03-24T00:14:07.504800+0000","last_became_peered":"2024-03-24T00:14:07.504800+0000","last_unstale":"2024-03-24T00:15:07.748624+0000","last_undegraded":"2024-03-24T00:15:07.748624+0000","last_fullsized":"2024-03-24T00:15:07.748624+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:13:01.267198+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:13:01.267198+0000","last_clean_scrub_stamp":"2024-03-24T00:13:01.267198+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T06:50:30.686330+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.30","version":"88'9","reported_seq":31,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-24T00:15:07.726732+0000","last_change":"2024-03-24T00:13:02.325460+0000","last_active":"2024-03-24T00:15:07.726732+0000","last_peered":"2024-03-24T00:15:07.726732+0000","last_clean":"2024-03-24T00:15:07.726732+0000","last_became_active":"2024-03-24T00:13:02.325112+0000","last_became_peered":"2024-03-24T00:13:02.325112+0000","last_unstale":"2024-03-24T00:15:07.726732+0000","last_undegraded":"2024-03-24T00:15:07.726732+0000","last_fullsized":"2024-03-24T00:15:07.726732+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:13:01.267198+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:13:01.267198+0000","last_clean_scrub_stamp":"2024-03-24T00:13:01.267198+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T03:15:16.362970+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2f","version":"88'8","reported_seq":30,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-24T00:15:07.728294+0000","last_change":"2024-03-24T00:13:02.325918+0000","last_active":"2024-03-24T00:15:07.728294+0000","last_peered":"2024-03-24T00:15:07.728294+0000","last_clean":"2024-03-24T00:15:07.728294+0000","last_became_active":"2024-03-24T00:13:02.325557+0000","last_became_peered":"2024-03-24T00:13:02.325557+0000","last_unstale":"2024-03-24T00:15:07.728294+0000","last_undegraded":"2024-03-24T00:15:07.728294+0000","last_fullsized":"2024-03-24T00:15:07.728294+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:13:01.267198+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:13:01.267198+0000","last_clean_scrub_stamp":"2024-03-24T00:13:01.267198+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T02:55:43.759674+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2e","version":"88'10","reported_seq":32,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-24T00:15:07.729972+0000","last_change":"2024-03-24T00:13:02.308106+0000","last_active":"2024-03-24T00:15:07.729972+0000","last_peered":"2024-03-24T00:15:07.729972+0000","last_clean":"2024-03-24T00:15:07.729972+0000","last_became_active":"2024-03-24T00:13:02.306972+0000","last_became_peered":"2024-03-24T00:13:02.306972+0000","last_unstale":"2024-03-24T00:15:07.729972+0000","last_undegraded":"2024-03-24T00:15:07.729972+0000","last_fullsized":"2024-03-24T00:15:07.729972+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:13:01.267198+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:13:01.267198+0000","last_clean_scrub_stamp":"2024-03-24T00:13:01.267198+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T09:38:54.677255+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,10],"acting":[9,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.2d","version":"88'7","reported_seq":29,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-24T00:15:07.726431+0000","last_change":"2024-03-24T00:13:02.331742+0000","last_active":"2024-03-24T00:15:07.726431+0000","last_peered":"2024-03-24T00:15:07.726431+0000","last_clean":"2024-03-24T00:15:07.726431+0000","last_became_active":"2024-03-24T00:13:02.331103+0000","last_became_peered":"2024-03-24T00:13:02.331103+0000","last_unstale":"2024-03-24T00:15:07.726431+0000","last_undegraded":"2024-03-24T00:15:07.726431+0000","last_fullsized":"2024-03-24T00:15:07.726431+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:13:01.267198+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:13:01.267198+0000","last_clean_scrub_stamp":"2024-03-24T00:13:01.267198+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:25:58.116306+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1f","version":"86'13","reported_seq":211,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-24T00:15:40.682427+0000","last_change":"2024-03-24T00:14:07.520827+0000","last_active":"2024-03-24T00:15:40.682427+0000","last_peered":"2024-03-24T00:15:40.682427+0000","last_clean":"2024-03-24T00:15:40.682427+0000","last_became_active":"2024-03-24T00:14:07.505406+0000","last_became_peered":"2024-03-24T00:14:07.505406+0000","last_unstale":"2024-03-24T00:15:40.682427+0000","last_undegraded":"2024-03-24T00:15:40.682427+0000","last_fullsized":"2024-03-24T00:15:40.682427+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:13:00.222790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:13:00.222790+0000","last_clean_scrub_stamp":"2024-03-24T00:13:00.222790+0000","objects_scrubbed":0,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T01:05:01.235978+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":152,"num_read_kb":152,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":672,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,1],"acting":[9,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.1e","version":"88'5","reported_seq":27,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-24T00:15:07.751376+0000","last_change":"2024-03-24T00:13:02.322461+0000","last_active":"2024-03-24T00:15:07.751376+0000","last_peered":"2024-03-24T00:15:07.751376+0000","last_clean":"2024-03-24T00:15:07.751376+0000","last_became_active":"2024-03-24T00:13:02.321752+0000","last_became_peered":"2024-03-24T00:13:02.321752+0000","last_unstale":"2024-03-24T00:15:07.751376+0000","last_undegraded":"2024-03-24T00:15:07.751376+0000","last_fullsized":"2024-03-24T00:15:07.751376+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:13:01.267198+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:13:01.267198+0000","last_clean_scrub_stamp":"2024-03-24T00:13:01.267198+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T06:37:05.825919+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1e","version":"86'2","reported_seq":26,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-24T00:15:07.752241+0000","last_change":"2024-03-24T00:13:01.324972+0000","last_active":"2024-03-24T00:15:07.752241+0000","last_peered":"2024-03-24T00:15:07.752241+0000","last_clean":"2024-03-24T00:15:07.752241+0000","last_became_active":"2024-03-24T00:13:01.323342+0000","last_became_peered":"2024-03-24T00:13:01.323342+0000","last_unstale":"2024-03-24T00:15:07.752241+0000","last_undegraded":"2024-03-24T00:15:07.752241+0000","last_fullsized":"2024-03-24T00:15:07.752241+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:13:00.222790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:13:00.222790+0000","last_clean_scrub_stamp":"2024-03-24T00:13:00.222790+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T05:34:35.827634+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"88'6","reported_seq":28,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-24T00:15:07.749057+0000","last_change":"2024-03-24T00:13:02.325455+0000","last_active":"2024-03-24T00:15:07.749057+0000","last_peered":"2024-03-24T00:15:07.749057+0000","last_clean":"2024-03-24T00:15:07.749057+0000","last_became_active":"2024-03-24T00:13:02.325061+0000","last_became_peered":"2024-03-24T00:13:02.325061+0000","last_unstale":"2024-03-24T00:15:07.749057+0000","last_undegraded":"2024-03-24T00:15:07.749057+0000","last_fullsized":"2024-03-24T00:15:07.749057+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:13:01.267198+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:13:01.267198+0000","last_clean_scrub_stamp":"2024-03-24T00:13:01.267198+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T06:13:33.223361+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,9],"acting":[0,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1d","version":"86'12","reported_seq":27,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-24T00:15:07.746267+0000","last_change":"2024-03-24T00:15:07.745753+0000","last_active":"2024-03-24T00:15:07.746267+0000","last_peered":"2024-03-24T00:15:07.746267+0000","last_clean":"2024-03-24T00:15:07.746267+0000","last_became_active":"2024-03-24T00:15:07.730583+0000","last_became_peered":"2024-03-24T00:15:07.730583+0000","last_unstale":"2024-03-24T00:15:07.746267+0000","last_undegraded":"2024-03-24T00:15:07.746267+0000","last_fullsized":"2024-03-24T00:15:07.746267+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:13:00.222790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:13:00.222790+0000","last_clean_scrub_stamp":"2024-03-24T00:13:00.222790+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T07:33:32.085470+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":0,"num_keys_recovered":10,"num_objects_omap":7,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,7],"acting":[11,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.1c","version":"88'9","reported_seq":31,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-24T00:15:07.726909+0000","last_change":"2024-03-24T00:13:02.325130+0000","last_active":"2024-03-24T00:15:07.726909+0000","last_peered":"2024-03-24T00:15:07.726909+0000","last_clean":"2024-03-24T00:15:07.726909+0000","last_became_active":"2024-03-24T00:13:02.324740+0000","last_became_peered":"2024-03-24T00:13:02.324740+0000","last_unstale":"2024-03-24T00:15:07.726909+0000","last_undegraded":"2024-03-24T00:15:07.726909+0000","last_fullsized":"2024-03-24T00:15:07.726909+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:13:01.267198+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:13:01.267198+0000","last_clean_scrub_stamp":"2024-03-24T00:13:01.267198+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T01:23:35.194788+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,6],"acting":[1,10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1c","version":"86'5","reported_seq":30,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-24T00:15:07.729993+0000","last_change":"2024-03-24T00:13:01.300572+0000","last_active":"2024-03-24T00:15:07.729993+0000","last_peered":"2024-03-24T00:15:07.729993+0000","last_clean":"2024-03-24T00:15:07.729993+0000","last_became_active":"2024-03-24T00:13:01.299801+0000","last_became_peered":"2024-03-24T00:13:01.299801+0000","last_unstale":"2024-03-24T00:15:07.729993+0000","last_undegraded":"2024-03-24T00:15:07.729993+0000","last_fullsized":"2024-03-24T00:15:07.729993+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:13:00.222790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:13:00.222790+0000","last_clean_scrub_stamp":"2024-03-24T00:13:00.222790+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T10:29:03.986969+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":7,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,5],"acting":[8,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1d","version":"88'15","reported_seq":37,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-24T00:15:07.747448+0000","last_change":"2024-03-24T00:13:02.304079+0000","last_active":"2024-03-24T00:15:07.747448+0000","last_peered":"2024-03-24T00:15:07.747448+0000","last_clean":"2024-03-24T00:15:07.747448+0000","last_became_active":"2024-03-24T00:13:02.303700+0000","last_became_peered":"2024-03-24T00:13:02.303700+0000","last_unstale":"2024-03-24T00:15:07.747448+0000","last_undegraded":"2024-03-24T00:15:07.747448+0000","last_fullsized":"2024-03-24T00:15:07.747448+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:13:01.267198+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:13:01.267198+0000","last_clean_scrub_stamp":"2024-03-24T00:13:01.267198+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T10:19:07.701927+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1b","version":"90'8","reported_seq":193,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-24T00:15:40.437307+0000","last_change":"2024-03-24T00:13:01.300949+0000","last_active":"2024-03-24T00:15:40.437307+0000","last_peered":"2024-03-24T00:15:40.437307+0000","last_clean":"2024-03-24T00:15:40.437307+0000","last_became_active":"2024-03-24T00:13:01.300605+0000","last_became_peered":"2024-03-24T00:13:01.300605+0000","last_unstale":"2024-03-24T00:15:40.437307+0000","last_undegraded":"2024-03-24T00:15:40.437307+0000","last_fullsized":"2024-03-24T00:15:40.437307+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:13:00.222790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:13:00.222790+0000","last_clean_scrub_stamp":"2024-03-24T00:13:00.222790+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T02:47:58.283103+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1545,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":161,"num_read_kb":5,"num_write":5,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1a","version":"88'7","reported_seq":29,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-24T00:15:07.729749+0000","last_change":"2024-03-24T00:13:02.312884+0000","last_active":"2024-03-24T00:15:07.729749+0000","last_peered":"2024-03-24T00:15:07.729749+0000","last_clean":"2024-03-24T00:15:07.729749+0000","last_became_active":"2024-03-24T00:13:02.312533+0000","last_became_peered":"2024-03-24T00:13:02.312533+0000","last_unstale":"2024-03-24T00:15:07.729749+0000","last_undegraded":"2024-03-24T00:15:07.729749+0000","last_fullsized":"2024-03-24T00:15:07.729749+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:13:01.267198+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:13:01.267198+0000","last_clean_scrub_stamp":"2024-03-24T00:13:01.267198+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T10:54:48.398645+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,5],"acting":[8,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.1a","version":"86'7","reported_seq":31,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-24T00:15:07.747575+0000","last_change":"2024-03-24T00:13:01.306292+0000","last_active":"2024-03-24T00:15:07.747575+0000","last_peered":"2024-03-24T00:15:07.747575+0000","last_clean":"2024-03-24T00:15:07.747575+0000","last_became_active":"2024-03-24T00:13:01.305898+0000","last_became_peered":"2024-03-24T00:13:01.305898+0000","last_unstale":"2024-03-24T00:15:07.747575+0000","last_undegraded":"2024-03-24T00:15:07.747575+0000","last_fullsized":"2024-03-24T00:15:07.747575+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:13:00.222790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:13:00.222790+0000","last_clean_scrub_stamp":"2024-03-24T00:13:00.222790+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T07:41:17.003588+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8,6],"acting":[10,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.1b","version":"88'8","reported_seq":30,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-24T00:15:07.747429+0000","last_change":"2024-03-24T00:13:02.323763+0000","last_active":"2024-03-24T00:15:07.747429+0000","last_peered":"2024-03-24T00:15:07.747429+0000","last_clean":"2024-03-24T00:15:07.747429+0000","last_became_active":"2024-03-24T00:13:02.323367+0000","last_became_peered":"2024-03-24T00:13:02.323367+0000","last_unstale":"2024-03-24T00:15:07.747429+0000","last_undegraded":"2024-03-24T00:15:07.747429+0000","last_fullsized":"2024-03-24T00:15:07.747429+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:13:01.267198+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:13:01.267198+0000","last_clean_scrub_stamp":"2024-03-24T00:13:01.267198+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T08:53:21.671317+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,3],"acting":[5,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.19","version":"86'3","reported_seq":27,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-24T00:15:07.751196+0000","last_change":"2024-03-24T00:13:01.323111+0000","last_active":"2024-03-24T00:15:07.751196+0000","last_peered":"2024-03-24T00:15:07.751196+0000","last_clean":"2024-03-24T00:15:07.751196+0000","last_became_active":"2024-03-24T00:13:01.322772+0000","last_became_peered":"2024-03-24T00:13:01.322772+0000","last_unstale":"2024-03-24T00:15:07.751196+0000","last_undegraded":"2024-03-24T00:15:07.751196+0000","last_fullsized":"2024-03-24T00:15:07.751196+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:13:00.222790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:13:00.222790+0000","last_clean_scrub_stamp":"2024-03-24T00:13:00.222790+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T04:19:53.097933+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,9],"acting":[6,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"88'8","reported_seq":30,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-24T00:15:07.750057+0000","last_change":"2024-03-24T00:13:02.330469+0000","last_active":"2024-03-24T00:15:07.750057+0000","last_peered":"2024-03-24T00:15:07.750057+0000","last_clean":"2024-03-24T00:15:07.750057+0000","last_became_active":"2024-03-24T00:13:02.327019+0000","last_became_peered":"2024-03-24T00:13:02.327019+0000","last_unstale":"2024-03-24T00:15:07.750057+0000","last_undegraded":"2024-03-24T00:15:07.750057+0000","last_fullsized":"2024-03-24T00:15:07.750057+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:13:01.267198+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:13:01.267198+0000","last_clean_scrub_stamp":"2024-03-24T00:13:01.267198+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T07:30:37.155535+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,6],"acting":[3,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.18","version":"86'3","reported_seq":28,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-24T00:15:07.747073+0000","last_change":"2024-03-24T00:13:01.300089+0000","last_active":"2024-03-24T00:15:07.747073+0000","last_peered":"2024-03-24T00:15:07.747073+0000","last_clean":"2024-03-24T00:15:07.747073+0000","last_became_active":"2024-03-24T00:13:01.299747+0000","last_became_peered":"2024-03-24T00:13:01.299747+0000","last_unstale":"2024-03-24T00:15:07.747073+0000","last_undegraded":"2024-03-24T00:15:07.747073+0000","last_fullsized":"2024-03-24T00:15:07.747073+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:13:00.222790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:13:00.222790+0000","last_clean_scrub_stamp":"2024-03-24T00:13:00.222790+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T00:30:30.142785+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"88'11","reported_seq":33,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-24T00:15:07.729674+0000","last_change":"2024-03-24T00:13:02.332490+0000","last_active":"2024-03-24T00:15:07.729674+0000","last_peered":"2024-03-24T00:15:07.729674+0000","last_clean":"2024-03-24T00:15:07.729674+0000","last_became_active":"2024-03-24T00:13:02.331182+0000","last_became_peered":"2024-03-24T00:13:02.331182+0000","last_unstale":"2024-03-24T00:15:07.729674+0000","last_undegraded":"2024-03-24T00:15:07.729674+0000","last_fullsized":"2024-03-24T00:15:07.729674+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:13:01.267198+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:13:01.267198+0000","last_clean_scrub_stamp":"2024-03-24T00:13:01.267198+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T06:22:16.084879+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,10],"acting":[9,3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.17","version":"88'6","reported_seq":333,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-24T00:15:38.433780+0000","last_change":"2024-03-24T00:13:01.299818+0000","last_active":"2024-03-24T00:15:38.433780+0000","last_peered":"2024-03-24T00:15:38.433780+0000","last_clean":"2024-03-24T00:15:38.433780+0000","last_became_active":"2024-03-24T00:13:01.299478+0000","last_became_peered":"2024-03-24T00:13:01.299478+0000","last_unstale":"2024-03-24T00:15:38.433780+0000","last_undegraded":"2024-03-24T00:15:38.433780+0000","last_fullsized":"2024-03-24T00:15:38.433780+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:13:00.222790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:13:00.222790+0000","last_clean_scrub_stamp":"2024-03-24T00:13:00.222790+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:51:47.088677+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":180,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":303,"num_read_kb":303,"num_write":7,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,9],"acting":[10,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.16","version":"88'10","reported_seq":32,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-24T00:15:07.727190+0000","last_change":"2024-03-24T00:13:02.304971+0000","last_active":"2024-03-24T00:15:07.727190+0000","last_peered":"2024-03-24T00:15:07.727190+0000","last_clean":"2024-03-24T00:15:07.727190+0000","last_became_active":"2024-03-24T00:13:02.304577+0000","last_became_peered":"2024-03-24T00:13:02.304577+0000","last_unstale":"2024-03-24T00:15:07.727190+0000","last_undegraded":"2024-03-24T00:15:07.727190+0000","last_fullsized":"2024-03-24T00:15:07.727190+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:13:01.267198+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:13:01.267198+0000","last_clean_scrub_stamp":"2024-03-24T00:13:01.267198+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T05:08:03.444554+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,9],"acting":[1,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.16","version":"86'4","reported_seq":28,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-24T00:15:07.729906+0000","last_change":"2024-03-24T00:13:01.303075+0000","last_active":"2024-03-24T00:15:07.729906+0000","last_peered":"2024-03-24T00:15:07.729906+0000","last_clean":"2024-03-24T00:15:07.729906+0000","last_became_active":"2024-03-24T00:13:01.301517+0000","last_became_peered":"2024-03-24T00:13:01.301517+0000","last_unstale":"2024-03-24T00:15:07.729906+0000","last_undegraded":"2024-03-24T00:15:07.729906+0000","last_fullsized":"2024-03-24T00:15:07.729906+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:13:00.222790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:13:00.222790+0000","last_clean_scrub_stamp":"2024-03-24T00:13:00.222790+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:04:10.552510+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.17","version":"88'9","reported_seq":31,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-24T00:15:07.749184+0000","last_change":"2024-03-24T00:13:02.320108+0000","last_active":"2024-03-24T00:15:07.749184+0000","last_peered":"2024-03-24T00:15:07.749184+0000","last_clean":"2024-03-24T00:15:07.749184+0000","last_became_active":"2024-03-24T00:13:02.319155+0000","last_became_peered":"2024-03-24T00:13:02.319155+0000","last_unstale":"2024-03-24T00:15:07.749184+0000","last_undegraded":"2024-03-24T00:15:07.749184+0000","last_fullsized":"2024-03-24T00:15:07.749184+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:13:01.267198+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:13:01.267198+0000","last_clean_scrub_stamp":"2024-03-24T00:13:01.267198+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T10:15:54.770304+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,5],"acting":[0,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.15","version":"86'5","reported_seq":30,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-24T00:15:07.747498+0000","last_change":"2024-03-24T00:13:01.323919+0000","last_active":"2024-03-24T00:15:07.747498+0000","last_peered":"2024-03-24T00:15:07.747498+0000","last_clean":"2024-03-24T00:15:07.747498+0000","last_became_active":"2024-03-24T00:13:01.323640+0000","last_became_peered":"2024-03-24T00:13:01.323640+0000","last_unstale":"2024-03-24T00:15:07.747498+0000","last_undegraded":"2024-03-24T00:15:07.747498+0000","last_fullsized":"2024-03-24T00:15:07.747498+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:13:00.222790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:13:00.222790+0000","last_clean_scrub_stamp":"2024-03-24T00:13:00.222790+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T06:16:49.018642+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":7,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,10],"acting":[5,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.14","version":"88'11","reported_seq":39,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-24T00:15:07.727032+0000","last_change":"2024-03-24T00:14:07.635409+0000","last_active":"2024-03-24T00:15:07.727032+0000","last_peered":"2024-03-24T00:15:07.727032+0000","last_clean":"2024-03-24T00:15:07.727032+0000","last_became_active":"2024-03-24T00:14:07.635093+0000","last_became_peered":"2024-03-24T00:14:07.635093+0000","last_unstale":"2024-03-24T00:15:07.727032+0000","last_undegraded":"2024-03-24T00:15:07.727032+0000","last_fullsized":"2024-03-24T00:15:07.727032+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:13:01.267198+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:13:01.267198+0000","last_clean_scrub_stamp":"2024-03-24T00:13:01.267198+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T10:31:10.205874+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,6],"acting":[1,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.14","version":"90'37","reported_seq":397,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-24T00:15:38.895751+0000","last_change":"2024-03-24T00:14:07.807393+0000","last_active":"2024-03-24T00:15:38.895751+0000","last_peered":"2024-03-24T00:15:38.895751+0000","last_clean":"2024-03-24T00:15:38.895751+0000","last_became_active":"2024-03-24T00:14:07.801140+0000","last_became_peered":"2024-03-24T00:14:07.801140+0000","last_unstale":"2024-03-24T00:15:38.895751+0000","last_undegraded":"2024-03-24T00:15:38.895751+0000","last_fullsized":"2024-03-24T00:15:38.895751+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:13:00.222790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:13:00.222790+0000","last_clean_scrub_stamp":"2024-03-24T00:13:00.222790+0000","objects_scrubbed":0,"log_size":37,"log_dups_size":0,"ondisk_log_size":37,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T10:35:25.627828+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":62256,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":312,"num_read_kb":216,"num_write":39,"num_write_kb":78,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":61701,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,4],"acting":[10,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.15","version":"88'6","reported_seq":28,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-24T00:15:07.729922+0000","last_change":"2024-03-24T00:13:02.307653+0000","last_active":"2024-03-24T00:15:07.729922+0000","last_peered":"2024-03-24T00:15:07.729922+0000","last_clean":"2024-03-24T00:15:07.729922+0000","last_became_active":"2024-03-24T00:13:02.307287+0000","last_became_peered":"2024-03-24T00:13:02.307287+0000","last_unstale":"2024-03-24T00:15:07.729922+0000","last_undegraded":"2024-03-24T00:15:07.729922+0000","last_fullsized":"2024-03-24T00:15:07.729922+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:13:01.267198+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:13:01.267198+0000","last_clean_scrub_stamp":"2024-03-24T00:13:01.267198+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T05:59:27.701614+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.13","version":"86'9","reported_seq":35,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-24T00:15:07.726557+0000","last_change":"2024-03-24T00:13:01.316854+0000","last_active":"2024-03-24T00:15:07.726557+0000","last_peered":"2024-03-24T00:15:07.726557+0000","last_clean":"2024-03-24T00:15:07.726557+0000","last_became_active":"2024-03-24T00:13:01.316507+0000","last_became_peered":"2024-03-24T00:13:01.316507+0000","last_unstale":"2024-03-24T00:15:07.726557+0000","last_undegraded":"2024-03-24T00:15:07.726557+0000","last_fullsized":"2024-03-24T00:15:07.726557+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:13:00.222790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:13:00.222790+0000","last_clean_scrub_stamp":"2024-03-24T00:13:00.222790+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T07:58:41.888003+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":13,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,10,3],"acting":[2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.12","version":"88'9","reported_seq":31,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-24T00:15:07.751081+0000","last_change":"2024-03-24T00:13:02.338811+0000","last_active":"2024-03-24T00:15:07.751081+0000","last_peered":"2024-03-24T00:15:07.751081+0000","last_clean":"2024-03-24T00:15:07.751081+0000","last_became_active":"2024-03-24T00:13:02.338438+0000","last_became_peered":"2024-03-24T00:13:02.338438+0000","last_unstale":"2024-03-24T00:15:07.751081+0000","last_undegraded":"2024-03-24T00:15:07.751081+0000","last_fullsized":"2024-03-24T00:15:07.751081+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:13:01.267198+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:13:01.267198+0000","last_clean_scrub_stamp":"2024-03-24T00:13:01.267198+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T03:55:16.531224+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.12","version":"86'5","reported_seq":30,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-24T00:15:07.747178+0000","last_change":"2024-03-24T00:13:01.313914+0000","last_active":"2024-03-24T00:15:07.747178+0000","last_peered":"2024-03-24T00:15:07.747178+0000","last_clean":"2024-03-24T00:15:07.747178+0000","last_became_active":"2024-03-24T00:13:01.313535+0000","last_became_peered":"2024-03-24T00:13:01.313535+0000","last_unstale":"2024-03-24T00:15:07.747178+0000","last_undegraded":"2024-03-24T00:15:07.747178+0000","last_fullsized":"2024-03-24T00:15:07.747178+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:13:00.222790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:13:00.222790+0000","last_clean_scrub_stamp":"2024-03-24T00:13:00.222790+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T03:23:53.011550+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":5,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"88'15","reported_seq":43,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-24T00:15:07.729806+0000","last_change":"2024-03-24T00:14:07.632978+0000","last_active":"2024-03-24T00:15:07.729806+0000","last_peered":"2024-03-24T00:15:07.729806+0000","last_clean":"2024-03-24T00:15:07.729806+0000","last_became_active":"2024-03-24T00:14:07.632719+0000","last_became_peered":"2024-03-24T00:14:07.632719+0000","last_unstale":"2024-03-24T00:15:07.729806+0000","last_undegraded":"2024-03-24T00:15:07.729806+0000","last_fullsized":"2024-03-24T00:15:07.729806+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:13:01.267198+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:13:01.267198+0000","last_clean_scrub_stamp":"2024-03-24T00:13:01.267198+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T09:59:38.993218+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,6],"acting":[7,9,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.11","version":"86'5","reported_seq":181,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-24T00:15:37.543087+0000","last_change":"2024-03-24T00:13:01.297887+0000","last_active":"2024-03-24T00:15:37.543087+0000","last_peered":"2024-03-24T00:15:37.543087+0000","last_clean":"2024-03-24T00:15:37.543087+0000","last_became_active":"2024-03-24T00:13:01.297560+0000","last_became_peered":"2024-03-24T00:13:01.297560+0000","last_unstale":"2024-03-24T00:15:37.543087+0000","last_undegraded":"2024-03-24T00:15:37.543087+0000","last_fullsized":"2024-03-24T00:15:37.543087+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:13:00.222790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:13:00.222790+0000","last_clean_scrub_stamp":"2024-03-24T00:13:00.222790+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T01:06:28.927274+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":588,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":153,"num_read_kb":3,"num_write":5,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,10],"acting":[4,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.10","version":"88'6","reported_seq":28,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-24T00:15:07.729530+0000","last_change":"2024-03-24T00:13:02.306811+0000","last_active":"2024-03-24T00:15:07.729530+0000","last_peered":"2024-03-24T00:15:07.729530+0000","last_clean":"2024-03-24T00:15:07.729530+0000","last_became_active":"2024-03-24T00:13:02.306499+0000","last_became_peered":"2024-03-24T00:13:02.306499+0000","last_unstale":"2024-03-24T00:15:07.729530+0000","last_undegraded":"2024-03-24T00:15:07.729530+0000","last_fullsized":"2024-03-24T00:15:07.729530+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:13:01.267198+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:13:01.267198+0000","last_clean_scrub_stamp":"2024-03-24T00:13:01.267198+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:49:32.683005+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.10","version":"86'17","reported_seq":42,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-24T00:15:07.729583+0000","last_change":"2024-03-24T00:13:01.307058+0000","last_active":"2024-03-24T00:15:07.729583+0000","last_peered":"2024-03-24T00:15:07.729583+0000","last_clean":"2024-03-24T00:15:07.729583+0000","last_became_active":"2024-03-24T00:13:01.306699+0000","last_became_peered":"2024-03-24T00:13:01.306699+0000","last_unstale":"2024-03-24T00:15:07.729583+0000","last_undegraded":"2024-03-24T00:15:07.729583+0000","last_fullsized":"2024-03-24T00:15:07.729583+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:13:00.222790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:13:00.222790+0000","last_clean_scrub_stamp":"2024-03-24T00:13:00.222790+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T10:32:37.334514+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,2],"acting":[8,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.11","version":"88'6","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-24T00:15:07.729239+0000","last_change":"2024-03-24T00:14:07.503357+0000","last_active":"2024-03-24T00:15:07.729239+0000","last_peered":"2024-03-24T00:15:07.729239+0000","last_clean":"2024-03-24T00:15:07.729239+0000","last_became_active":"2024-03-24T00:14:07.502511+0000","last_became_peered":"2024-03-24T00:14:07.502511+0000","last_unstale":"2024-03-24T00:15:07.729239+0000","last_undegraded":"2024-03-24T00:15:07.729239+0000","last_fullsized":"2024-03-24T00:15:07.729239+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:13:01.267198+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:13:01.267198+0000","last_clean_scrub_stamp":"2024-03-24T00:13:01.267198+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T04:18:09.351579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,1],"acting":[9,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.f","version":"86'2","reported_seq":27,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-24T00:15:07.730062+0000","last_change":"2024-03-24T00:13:01.295874+0000","last_active":"2024-03-24T00:15:07.730062+0000","last_peered":"2024-03-24T00:15:07.730062+0000","last_clean":"2024-03-24T00:15:07.730062+0000","last_became_active":"2024-03-24T00:13:01.295521+0000","last_became_peered":"2024-03-24T00:13:01.295521+0000","last_unstale":"2024-03-24T00:15:07.730062+0000","last_undegraded":"2024-03-24T00:15:07.730062+0000","last_fullsized":"2024-03-24T00:15:07.730062+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:13:00.222790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:13:00.222790+0000","last_clean_scrub_stamp":"2024-03-24T00:13:00.222790+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T06:47:15.749491+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5,4],"acting":[8,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.e","version":"88'5","reported_seq":27,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-24T00:15:07.728056+0000","last_change":"2024-03-24T00:13:02.306617+0000","last_active":"2024-03-24T00:15:07.728056+0000","last_peered":"2024-03-24T00:15:07.728056+0000","last_clean":"2024-03-24T00:15:07.728056+0000","last_became_active":"2024-03-24T00:13:02.306224+0000","last_became_peered":"2024-03-24T00:13:02.306224+0000","last_unstale":"2024-03-24T00:15:07.728056+0000","last_undegraded":"2024-03-24T00:15:07.728056+0000","last_fullsized":"2024-03-24T00:15:07.728056+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:13:01.267198+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:13:01.267198+0000","last_clean_scrub_stamp":"2024-03-24T00:13:01.267198+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T06:49:16.032741+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,4],"acting":[11,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.4","version":"86'1","reported_seq":25,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-24T00:15:07.729939+0000","last_change":"2024-03-24T00:13:01.314428+0000","last_active":"2024-03-24T00:15:07.729939+0000","last_peered":"2024-03-24T00:15:07.729939+0000","last_clean":"2024-03-24T00:15:07.729939+0000","last_became_active":"2024-03-24T00:13:01.313421+0000","last_became_peered":"2024-03-24T00:13:01.313421+0000","last_unstale":"2024-03-24T00:15:07.729939+0000","last_undegraded":"2024-03-24T00:15:07.729939+0000","last_fullsized":"2024-03-24T00:15:07.729939+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:13:00.222790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:13:00.222790+0000","last_clean_scrub_stamp":"2024-03-24T00:13:00.222790+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T08:10:25.796718+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,0],"acting":[9,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.5","version":"88'5","reported_seq":27,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-24T00:15:07.747827+0000","last_change":"2024-03-24T00:13:02.331660+0000","last_active":"2024-03-24T00:15:07.747827+0000","last_peered":"2024-03-24T00:15:07.747827+0000","last_clean":"2024-03-24T00:15:07.747827+0000","last_became_active":"2024-03-24T00:13:02.331288+0000","last_became_peered":"2024-03-24T00:13:02.331288+0000","last_unstale":"2024-03-24T00:15:07.747827+0000","last_undegraded":"2024-03-24T00:15:07.747827+0000","last_fullsized":"2024-03-24T00:15:07.747827+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:13:01.267198+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:13:01.267198+0000","last_clean_scrub_stamp":"2024-03-24T00:13:01.267198+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:36:55.576510+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3,6],"acting":[10,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.2","version":"86'5","reported_seq":184,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-24T00:15:41.684981+0000","last_change":"2024-03-24T00:13:01.310538+0000","last_active":"2024-03-24T00:15:41.684981+0000","last_peered":"2024-03-24T00:15:41.684981+0000","last_clean":"2024-03-24T00:15:41.684981+0000","last_became_active":"2024-03-24T00:13:01.310124+0000","last_became_peered":"2024-03-24T00:13:01.310124+0000","last_unstale":"2024-03-24T00:15:41.684981+0000","last_undegraded":"2024-03-24T00:15:41.684981+0000","last_fullsized":"2024-03-24T00:15:41.684981+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:13:00.222790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:13:00.222790+0000","last_clean_scrub_stamp":"2024-03-24T00:13:00.222790+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T05:47:03.534824+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":588,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":155,"num_read_kb":1,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3","version":"88'7","reported_seq":29,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-24T00:15:07.747783+0000","last_change":"2024-03-24T00:13:02.324117+0000","last_active":"2024-03-24T00:15:07.747783+0000","last_peered":"2024-03-24T00:15:07.747783+0000","last_clean":"2024-03-24T00:15:07.747783+0000","last_became_active":"2024-03-24T00:13:02.323706+0000","last_became_peered":"2024-03-24T00:13:02.323706+0000","last_unstale":"2024-03-24T00:15:07.747783+0000","last_undegraded":"2024-03-24T00:15:07.747783+0000","last_fullsized":"2024-03-24T00:15:07.747783+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:13:01.267198+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:13:01.267198+0000","last_clean_scrub_stamp":"2024-03-24T00:13:01.267198+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T05:34:31.657324+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,4,0],"acting":[10,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.0","version":"88'8","reported_seq":30,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-24T00:15:07.747186+0000","last_change":"2024-03-24T00:13:02.306030+0000","last_active":"2024-03-24T00:15:07.747186+0000","last_peered":"2024-03-24T00:15:07.747186+0000","last_clean":"2024-03-24T00:15:07.747186+0000","last_became_active":"2024-03-24T00:13:02.305588+0000","last_became_peered":"2024-03-24T00:13:02.305588+0000","last_unstale":"2024-03-24T00:15:07.747186+0000","last_undegraded":"2024-03-24T00:15:07.747186+0000","last_fullsized":"2024-03-24T00:15:07.747186+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:13:01.267198+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:13:01.267198+0000","last_clean_scrub_stamp":"2024-03-24T00:13:01.267198+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:20:58.206244+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,4],"acting":[5,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"86'5","reported_seq":183,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-24T00:15:40.546451+0000","last_change":"2024-03-24T00:13:01.318123+0000","last_active":"2024-03-24T00:15:40.546451+0000","last_peered":"2024-03-24T00:15:40.546451+0000","last_clean":"2024-03-24T00:15:40.546451+0000","last_became_active":"2024-03-24T00:13:01.317683+0000","last_became_peered":"2024-03-24T00:13:01.317683+0000","last_unstale":"2024-03-24T00:15:40.546451+0000","last_undegraded":"2024-03-24T00:15:40.546451+0000","last_fullsized":"2024-03-24T00:15:40.546451+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:13:00.222790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:13:00.222790+0000","last_clean_scrub_stamp":"2024-03-24T00:13:00.222790+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T06:33:02.669668+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":154,"num_read_kb":154,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"88'8","reported_seq":30,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-24T00:15:07.750327+0000","last_change":"2024-03-24T00:13:02.328599+0000","last_active":"2024-03-24T00:15:07.750327+0000","last_peered":"2024-03-24T00:15:07.750327+0000","last_clean":"2024-03-24T00:15:07.750327+0000","last_became_active":"2024-03-24T00:13:02.328165+0000","last_became_peered":"2024-03-24T00:13:02.328165+0000","last_unstale":"2024-03-24T00:15:07.750327+0000","last_undegraded":"2024-03-24T00:15:07.750327+0000","last_fullsized":"2024-03-24T00:15:07.750327+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:13:01.267198+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:13:01.267198+0000","last_clean_scrub_stamp":"2024-03-24T00:13:01.267198+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T07:22:39.646578+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"86'6","reported_seq":33,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-24T00:15:07.748021+0000","last_change":"2024-03-24T00:13:01.301932+0000","last_active":"2024-03-24T00:15:07.748021+0000","last_peered":"2024-03-24T00:15:07.748021+0000","last_clean":"2024-03-24T00:15:07.748021+0000","last_became_active":"2024-03-24T00:13:01.301464+0000","last_became_peered":"2024-03-24T00:13:01.301464+0000","last_unstale":"2024-03-24T00:15:07.748021+0000","last_undegraded":"2024-03-24T00:15:07.748021+0000","last_fullsized":"2024-03-24T00:15:07.748021+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:13:00.222790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:13:00.222790+0000","last_clean_scrub_stamp":"2024-03-24T00:13:00.222790+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T08:58:25.512421+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":4,"num_read_kb":4,"num_write":8,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,9],"acting":[4,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.0","version":"21'137","reported_seq":349,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-24T00:15:07.729560+0000","last_change":"2024-03-24T00:11:43.008767+0000","last_active":"2024-03-24T00:15:07.729560+0000","last_peered":"2024-03-24T00:15:07.729560+0000","last_clean":"2024-03-24T00:15:07.729560+0000","last_became_active":"2024-03-24T00:11:43.008364+0000","last_became_peered":"2024-03-24T00:11:43.008364+0000","last_unstale":"2024-03-24T00:15:07.729560+0000","last_undegraded":"2024-03-24T00:15:07.729560+0000","last_fullsized":"2024-03-24T00:15:07.729560+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:07:50.191828+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:07:50.191828+0000","last_clean_scrub_stamp":"2024-03-24T00:07:50.191828+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T01:55:26.549348+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"88'5","reported_seq":27,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-24T00:15:07.750385+0000","last_change":"2024-03-24T00:13:02.325667+0000","last_active":"2024-03-24T00:15:07.750385+0000","last_peered":"2024-03-24T00:15:07.750385+0000","last_clean":"2024-03-24T00:15:07.750385+0000","last_became_active":"2024-03-24T00:13:02.322476+0000","last_became_peered":"2024-03-24T00:13:02.322476+0000","last_unstale":"2024-03-24T00:15:07.750385+0000","last_undegraded":"2024-03-24T00:15:07.750385+0000","last_fullsized":"2024-03-24T00:15:07.750385+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:13:01.267198+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:13:01.267198+0000","last_clean_scrub_stamp":"2024-03-24T00:13:01.267198+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T08:07:19.936708+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"86'4","reported_seq":28,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-24T00:15:07.729687+0000","last_change":"2024-03-24T00:13:01.298082+0000","last_active":"2024-03-24T00:15:07.729687+0000","last_peered":"2024-03-24T00:15:07.729687+0000","last_clean":"2024-03-24T00:15:07.729687+0000","last_became_active":"2024-03-24T00:13:01.297686+0000","last_became_peered":"2024-03-24T00:13:01.297686+0000","last_unstale":"2024-03-24T00:15:07.729687+0000","last_undegraded":"2024-03-24T00:15:07.729687+0000","last_fullsized":"2024-03-24T00:15:07.729687+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:13:00.222790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:13:00.222790+0000","last_clean_scrub_stamp":"2024-03-24T00:13:00.222790+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T09:25:07.204076+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.5","version":"86'4","reported_seq":29,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-24T00:15:07.729914+0000","last_change":"2024-03-24T00:13:01.321250+0000","last_active":"2024-03-24T00:15:07.729914+0000","last_peered":"2024-03-24T00:15:07.729914+0000","last_clean":"2024-03-24T00:15:07.729914+0000","last_became_active":"2024-03-24T00:13:01.320929+0000","last_became_peered":"2024-03-24T00:13:01.320929+0000","last_unstale":"2024-03-24T00:15:07.729914+0000","last_undegraded":"2024-03-24T00:15:07.729914+0000","last_fullsized":"2024-03-24T00:15:07.729914+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:13:00.222790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:13:00.222790+0000","last_clean_scrub_stamp":"2024-03-24T00:13:00.222790+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T05:22:05.490303+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,9],"acting":[8,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.4","version":"88'9","reported_seq":31,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-24T00:15:07.751893+0000","last_change":"2024-03-24T00:13:02.323413+0000","last_active":"2024-03-24T00:15:07.751893+0000","last_peered":"2024-03-24T00:15:07.751893+0000","last_clean":"2024-03-24T00:15:07.751893+0000","last_became_active":"2024-03-24T00:13:02.322947+0000","last_became_peered":"2024-03-24T00:13:02.322947+0000","last_unstale":"2024-03-24T00:15:07.751893+0000","last_undegraded":"2024-03-24T00:15:07.751893+0000","last_fullsized":"2024-03-24T00:15:07.751893+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:13:01.267198+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:13:01.267198+0000","last_clean_scrub_stamp":"2024-03-24T00:13:01.267198+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T00:16:14.490270+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,9],"acting":[6,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.6","version":"90'14","reported_seq":38,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-24T00:15:37.357286+0000","last_change":"2024-03-24T00:13:01.320380+0000","last_active":"2024-03-24T00:15:37.357286+0000","last_peered":"2024-03-24T00:15:37.357286+0000","last_clean":"2024-03-24T00:15:37.357286+0000","last_became_active":"2024-03-24T00:13:01.319192+0000","last_became_peered":"2024-03-24T00:13:01.319192+0000","last_unstale":"2024-03-24T00:15:37.357286+0000","last_undegraded":"2024-03-24T00:15:37.357286+0000","last_fullsized":"2024-03-24T00:15:37.357286+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:13:00.222790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:13:00.222790+0000","last_clean_scrub_stamp":"2024-03-24T00:13:00.222790+0000","objects_scrubbed":0,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T02:09:27.085298+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":24,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,3],"acting":[9,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.7","version":"88'5","reported_seq":27,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-24T00:15:07.753013+0000","last_change":"2024-03-24T00:13:02.328067+0000","last_active":"2024-03-24T00:15:07.753013+0000","last_peered":"2024-03-24T00:15:07.753013+0000","last_clean":"2024-03-24T00:15:07.753013+0000","last_became_active":"2024-03-24T00:13:02.327468+0000","last_became_peered":"2024-03-24T00:13:02.327468+0000","last_unstale":"2024-03-24T00:15:07.753013+0000","last_undegraded":"2024-03-24T00:15:07.753013+0000","last_fullsized":"2024-03-24T00:15:07.753013+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:13:01.267198+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:13:01.267198+0000","last_clean_scrub_stamp":"2024-03-24T00:13:01.267198+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T01:21:58.596941+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,7],"acting":[3,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"86'7","reported_seq":32,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-24T00:15:07.748240+0000","last_change":"2024-03-24T00:13:01.309189+0000","last_active":"2024-03-24T00:15:07.748240+0000","last_peered":"2024-03-24T00:15:07.748240+0000","last_clean":"2024-03-24T00:15:07.748240+0000","last_became_active":"2024-03-24T00:13:01.308871+0000","last_became_peered":"2024-03-24T00:13:01.308871+0000","last_unstale":"2024-03-24T00:15:07.748240+0000","last_undegraded":"2024-03-24T00:15:07.748240+0000","last_fullsized":"2024-03-24T00:15:07.748240+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:13:00.222790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:13:00.222790+0000","last_clean_scrub_stamp":"2024-03-24T00:13:00.222790+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T01:00:32.610036+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,6],"acting":[4,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"88'7","reported_seq":29,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-24T00:15:07.750291+0000","last_change":"2024-03-24T00:13:02.332238+0000","last_active":"2024-03-24T00:15:07.750291+0000","last_peered":"2024-03-24T00:15:07.750291+0000","last_clean":"2024-03-24T00:15:07.750291+0000","last_became_active":"2024-03-24T00:13:02.331909+0000","last_became_peered":"2024-03-24T00:13:02.331909+0000","last_unstale":"2024-03-24T00:15:07.750291+0000","last_undegraded":"2024-03-24T00:15:07.750291+0000","last_fullsized":"2024-03-24T00:15:07.750291+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:13:01.267198+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:13:01.267198+0000","last_clean_scrub_stamp":"2024-03-24T00:13:01.267198+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T04:41:14.762251+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"86'2","reported_seq":26,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-24T00:15:07.726733+0000","last_change":"2024-03-24T00:13:01.306883+0000","last_active":"2024-03-24T00:15:07.726733+0000","last_peered":"2024-03-24T00:15:07.726733+0000","last_clean":"2024-03-24T00:15:07.726733+0000","last_became_active":"2024-03-24T00:13:01.306629+0000","last_became_peered":"2024-03-24T00:13:01.306629+0000","last_unstale":"2024-03-24T00:15:07.726733+0000","last_undegraded":"2024-03-24T00:15:07.726733+0000","last_fullsized":"2024-03-24T00:15:07.726733+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:13:00.222790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:13:00.222790+0000","last_clean_scrub_stamp":"2024-03-24T00:13:00.222790+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T08:11:52.398783+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"88'2","reported_seq":24,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-24T00:15:07.729642+0000","last_change":"2024-03-24T00:13:02.308421+0000","last_active":"2024-03-24T00:15:07.729642+0000","last_peered":"2024-03-24T00:15:07.729642+0000","last_clean":"2024-03-24T00:15:07.729642+0000","last_became_active":"2024-03-24T00:13:02.308081+0000","last_became_peered":"2024-03-24T00:13:02.308081+0000","last_unstale":"2024-03-24T00:15:07.729642+0000","last_undegraded":"2024-03-24T00:15:07.729642+0000","last_fullsized":"2024-03-24T00:15:07.729642+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:13:01.267198+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:13:01.267198+0000","last_clean_scrub_stamp":"2024-03-24T00:13:01.267198+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T06:50:42.356293+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,10],"acting":[8,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.9","version":"86'7","reported_seq":31,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-24T00:15:07.750083+0000","last_change":"2024-03-24T00:13:01.324439+0000","last_active":"2024-03-24T00:15:07.750083+0000","last_peered":"2024-03-24T00:15:07.750083+0000","last_clean":"2024-03-24T00:15:07.750083+0000","last_became_active":"2024-03-24T00:13:01.324037+0000","last_became_peered":"2024-03-24T00:13:01.324037+0000","last_unstale":"2024-03-24T00:15:07.750083+0000","last_undegraded":"2024-03-24T00:15:07.750083+0000","last_fullsized":"2024-03-24T00:15:07.750083+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:13:00.222790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:13:00.222790+0000","last_clean_scrub_stamp":"2024-03-24T00:13:00.222790+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T01:37:56.767931+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,10],"acting":[3,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.8","version":"88'9","reported_seq":31,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-24T00:15:07.727799+0000","last_change":"2024-03-24T00:13:02.313740+0000","last_active":"2024-03-24T00:15:07.727799+0000","last_peered":"2024-03-24T00:15:07.727799+0000","last_clean":"2024-03-24T00:15:07.727799+0000","last_became_active":"2024-03-24T00:13:02.313401+0000","last_became_peered":"2024-03-24T00:13:02.313401+0000","last_unstale":"2024-03-24T00:15:07.727799+0000","last_undegraded":"2024-03-24T00:15:07.727799+0000","last_fullsized":"2024-03-24T00:15:07.727799+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:13:01.267198+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:13:01.267198+0000","last_clean_scrub_stamp":"2024-03-24T00:13:01.267198+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T04:16:52.569135+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.a","version":"86'3","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-24T00:15:07.727832+0000","last_change":"2024-03-24T00:14:07.514050+0000","last_active":"2024-03-24T00:15:07.727832+0000","last_peered":"2024-03-24T00:15:07.727832+0000","last_clean":"2024-03-24T00:15:07.727832+0000","last_became_active":"2024-03-24T00:14:07.509219+0000","last_became_peered":"2024-03-24T00:14:07.509219+0000","last_unstale":"2024-03-24T00:15:07.727832+0000","last_undegraded":"2024-03-24T00:15:07.727832+0000","last_fullsized":"2024-03-24T00:15:07.727832+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:13:00.222790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:13:00.222790+0000","last_clean_scrub_stamp":"2024-03-24T00:13:00.222790+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T08:05:33.888846+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,3,8],"acting":[11,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.b","version":"88'8","reported_seq":30,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-24T00:15:07.752804+0000","last_change":"2024-03-24T00:13:02.337413+0000","last_active":"2024-03-24T00:15:07.752804+0000","last_peered":"2024-03-24T00:15:07.752804+0000","last_clean":"2024-03-24T00:15:07.752804+0000","last_became_active":"2024-03-24T00:13:02.336910+0000","last_became_peered":"2024-03-24T00:13:02.336910+0000","last_unstale":"2024-03-24T00:15:07.752804+0000","last_undegraded":"2024-03-24T00:15:07.752804+0000","last_fullsized":"2024-03-24T00:15:07.752804+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:13:01.267198+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:13:01.267198+0000","last_clean_scrub_stamp":"2024-03-24T00:13:01.267198+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T10:51:06.549979+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,10],"acting":[3,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.b","version":"86'4","reported_seq":29,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-24T00:15:07.729799+0000","last_change":"2024-03-24T00:13:01.300201+0000","last_active":"2024-03-24T00:15:07.729799+0000","last_peered":"2024-03-24T00:15:07.729799+0000","last_clean":"2024-03-24T00:15:07.729799+0000","last_became_active":"2024-03-24T00:13:01.299750+0000","last_became_peered":"2024-03-24T00:13:01.299750+0000","last_unstale":"2024-03-24T00:15:07.729799+0000","last_undegraded":"2024-03-24T00:15:07.729799+0000","last_fullsized":"2024-03-24T00:15:07.729799+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:13:00.222790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:13:00.222790+0000","last_clean_scrub_stamp":"2024-03-24T00:13:00.222790+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T01:37:32.768926+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.a","version":"88'5","reported_seq":27,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-24T00:15:07.729769+0000","last_change":"2024-03-24T00:13:02.331732+0000","last_active":"2024-03-24T00:15:07.729769+0000","last_peered":"2024-03-24T00:15:07.729769+0000","last_clean":"2024-03-24T00:15:07.729769+0000","last_became_active":"2024-03-24T00:13:02.331365+0000","last_became_peered":"2024-03-24T00:13:02.331365+0000","last_unstale":"2024-03-24T00:15:07.729769+0000","last_undegraded":"2024-03-24T00:15:07.729769+0000","last_fullsized":"2024-03-24T00:15:07.729769+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:13:01.267198+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:13:01.267198+0000","last_clean_scrub_stamp":"2024-03-24T00:13:01.267198+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:27:53.598158+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.c","version":"90'8","reported_seq":187,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-24T00:15:39.488276+0000","last_change":"2024-03-24T00:13:01.308725+0000","last_active":"2024-03-24T00:15:39.488276+0000","last_peered":"2024-03-24T00:15:39.488276+0000","last_clean":"2024-03-24T00:15:39.488276+0000","last_became_active":"2024-03-24T00:13:01.308324+0000","last_became_peered":"2024-03-24T00:13:01.308324+0000","last_unstale":"2024-03-24T00:15:39.488276+0000","last_undegraded":"2024-03-24T00:15:39.488276+0000","last_fullsized":"2024-03-24T00:15:39.488276+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:13:00.222790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:13:00.222790+0000","last_clean_scrub_stamp":"2024-03-24T00:13:00.222790+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T07:43:03.691683+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2093,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":155,"num_read_kb":4,"num_write":8,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,11,0],"acting":[2,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"88'7","reported_seq":29,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-24T00:15:07.730048+0000","last_change":"2024-03-24T00:13:02.305749+0000","last_active":"2024-03-24T00:15:07.730048+0000","last_peered":"2024-03-24T00:15:07.730048+0000","last_clean":"2024-03-24T00:15:07.730048+0000","last_became_active":"2024-03-24T00:13:02.305413+0000","last_became_peered":"2024-03-24T00:13:02.305413+0000","last_unstale":"2024-03-24T00:15:07.730048+0000","last_undegraded":"2024-03-24T00:15:07.730048+0000","last_fullsized":"2024-03-24T00:15:07.730048+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:13:01.267198+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:13:01.267198+0000","last_clean_scrub_stamp":"2024-03-24T00:13:01.267198+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T00:54:56.437725+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,11],"acting":[7,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.d","version":"86'3","reported_seq":28,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-24T00:15:07.747344+0000","last_change":"2024-03-24T00:13:01.304883+0000","last_active":"2024-03-24T00:15:07.747344+0000","last_peered":"2024-03-24T00:15:07.747344+0000","last_clean":"2024-03-24T00:15:07.747344+0000","last_became_active":"2024-03-24T00:13:01.304459+0000","last_became_peered":"2024-03-24T00:13:01.304459+0000","last_unstale":"2024-03-24T00:15:07.747344+0000","last_undegraded":"2024-03-24T00:15:07.747344+0000","last_fullsized":"2024-03-24T00:15:07.747344+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:13:00.222790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:13:00.222790+0000","last_clean_scrub_stamp":"2024-03-24T00:13:00.222790+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T06:14:22.473571+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10,3],"acting":[5,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.c","version":"88'7","reported_seq":29,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-24T00:15:07.729746+0000","last_change":"2024-03-24T00:13:02.331923+0000","last_active":"2024-03-24T00:15:07.729746+0000","last_peered":"2024-03-24T00:15:07.729746+0000","last_clean":"2024-03-24T00:15:07.729746+0000","last_became_active":"2024-03-24T00:13:02.331504+0000","last_became_peered":"2024-03-24T00:13:02.331504+0000","last_unstale":"2024-03-24T00:15:07.729746+0000","last_undegraded":"2024-03-24T00:15:07.729746+0000","last_fullsized":"2024-03-24T00:15:07.729746+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:13:01.267198+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:13:01.267198+0000","last_clean_scrub_stamp":"2024-03-24T00:13:01.267198+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T03:27:17.160248+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,5],"acting":[7,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"86'5","reported_seq":29,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-24T00:15:07.751611+0000","last_change":"2024-03-24T00:13:01.320223+0000","last_active":"2024-03-24T00:15:07.751611+0000","last_peered":"2024-03-24T00:15:07.751611+0000","last_clean":"2024-03-24T00:15:07.751611+0000","last_became_active":"2024-03-24T00:13:01.319342+0000","last_became_peered":"2024-03-24T00:13:01.319342+0000","last_unstale":"2024-03-24T00:15:07.751611+0000","last_undegraded":"2024-03-24T00:15:07.751611+0000","last_fullsized":"2024-03-24T00:15:07.751611+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:13:00.222790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:13:00.222790+0000","last_clean_scrub_stamp":"2024-03-24T00:13:00.222790+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T02:24:08.018521+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.f","version":"88'9","reported_seq":31,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-24T00:15:07.729949+0000","last_change":"2024-03-24T00:13:02.307719+0000","last_active":"2024-03-24T00:15:07.729949+0000","last_peered":"2024-03-24T00:15:07.729949+0000","last_clean":"2024-03-24T00:15:07.729949+0000","last_became_active":"2024-03-24T00:13:02.307399+0000","last_became_peered":"2024-03-24T00:13:02.307399+0000","last_unstale":"2024-03-24T00:15:07.729949+0000","last_undegraded":"2024-03-24T00:15:07.729949+0000","last_fullsized":"2024-03-24T00:15:07.729949+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:13:01.267198+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:13:01.267198+0000","last_clean_scrub_stamp":"2024-03-24T00:13:01.267198+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T09:38:13.077843+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,8],"acting":[7,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.20","version":"88'9","reported_seq":31,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-24T00:15:07.727941+0000","last_change":"2024-03-24T00:13:02.307469+0000","last_active":"2024-03-24T00:15:07.727941+0000","last_peered":"2024-03-24T00:15:07.727941+0000","last_clean":"2024-03-24T00:15:07.727941+0000","last_became_active":"2024-03-24T00:13:02.306873+0000","last_became_peered":"2024-03-24T00:13:02.306873+0000","last_unstale":"2024-03-24T00:15:07.727941+0000","last_undegraded":"2024-03-24T00:15:07.727941+0000","last_fullsized":"2024-03-24T00:15:07.727941+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:13:01.267198+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:13:01.267198+0000","last_clean_scrub_stamp":"2024-03-24T00:13:01.267198+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:19:51.163015+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,5],"acting":[11,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.21","version":"88'12","reported_seq":34,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-24T00:15:07.726362+0000","last_change":"2024-03-24T00:13:02.307472+0000","last_active":"2024-03-24T00:15:07.726362+0000","last_peered":"2024-03-24T00:15:07.726362+0000","last_clean":"2024-03-24T00:15:07.726362+0000","last_became_active":"2024-03-24T00:13:02.307099+0000","last_became_peered":"2024-03-24T00:13:02.307099+0000","last_unstale":"2024-03-24T00:15:07.726362+0000","last_undegraded":"2024-03-24T00:15:07.726362+0000","last_fullsized":"2024-03-24T00:15:07.726362+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:13:01.267198+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:13:01.267198+0000","last_clean_scrub_stamp":"2024-03-24T00:13:01.267198+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T08:49:06.461235+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,4],"acting":[2,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"88'8","reported_seq":30,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-24T00:15:07.726483+0000","last_change":"2024-03-24T00:13:02.331133+0000","last_active":"2024-03-24T00:15:07.726483+0000","last_peered":"2024-03-24T00:15:07.726483+0000","last_clean":"2024-03-24T00:15:07.726483+0000","last_became_active":"2024-03-24T00:13:02.330780+0000","last_became_peered":"2024-03-24T00:13:02.330780+0000","last_unstale":"2024-03-24T00:15:07.726483+0000","last_undegraded":"2024-03-24T00:15:07.726483+0000","last_fullsized":"2024-03-24T00:15:07.726483+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:13:01.267198+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:13:01.267198+0000","last_clean_scrub_stamp":"2024-03-24T00:13:01.267198+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T10:46:07.933131+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.23","version":"88'7","reported_seq":29,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-24T00:15:07.749059+0000","last_change":"2024-03-24T00:13:02.335406+0000","last_active":"2024-03-24T00:15:07.749059+0000","last_peered":"2024-03-24T00:15:07.749059+0000","last_clean":"2024-03-24T00:15:07.749059+0000","last_became_active":"2024-03-24T00:13:02.334994+0000","last_became_peered":"2024-03-24T00:13:02.334994+0000","last_unstale":"2024-03-24T00:15:07.749059+0000","last_undegraded":"2024-03-24T00:15:07.749059+0000","last_fullsized":"2024-03-24T00:15:07.749059+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:13:01.267198+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:13:01.267198+0000","last_clean_scrub_stamp":"2024-03-24T00:13:01.267198+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T03:24:59.547395+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,4],"acting":[0,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"88'4","reported_seq":26,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-24T00:15:07.752325+0000","last_change":"2024-03-24T00:13:02.332420+0000","last_active":"2024-03-24T00:15:07.752325+0000","last_peered":"2024-03-24T00:15:07.752325+0000","last_clean":"2024-03-24T00:15:07.752325+0000","last_became_active":"2024-03-24T00:13:02.331496+0000","last_became_peered":"2024-03-24T00:13:02.331496+0000","last_unstale":"2024-03-24T00:15:07.752325+0000","last_undegraded":"2024-03-24T00:15:07.752325+0000","last_fullsized":"2024-03-24T00:15:07.752325+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:13:01.267198+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:13:01.267198+0000","last_clean_scrub_stamp":"2024-03-24T00:13:01.267198+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T03:15:06.007947+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"88'6","reported_seq":28,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-24T00:15:07.728441+0000","last_change":"2024-03-24T00:13:02.309077+0000","last_active":"2024-03-24T00:15:07.728441+0000","last_peered":"2024-03-24T00:15:07.728441+0000","last_clean":"2024-03-24T00:15:07.728441+0000","last_became_active":"2024-03-24T00:13:02.308752+0000","last_became_peered":"2024-03-24T00:13:02.308752+0000","last_unstale":"2024-03-24T00:15:07.728441+0000","last_undegraded":"2024-03-24T00:15:07.728441+0000","last_fullsized":"2024-03-24T00:15:07.728441+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:13:01.267198+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:13:01.267198+0000","last_clean_scrub_stamp":"2024-03-24T00:13:01.267198+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T06:49:19.657988+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,5],"acting":[11,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.26","version":"88'15","reported_seq":37,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-24T00:15:07.726661+0000","last_change":"2024-03-24T00:13:02.304962+0000","last_active":"2024-03-24T00:15:07.726661+0000","last_peered":"2024-03-24T00:15:07.726661+0000","last_clean":"2024-03-24T00:15:07.726661+0000","last_became_active":"2024-03-24T00:13:02.304580+0000","last_became_peered":"2024-03-24T00:13:02.304580+0000","last_unstale":"2024-03-24T00:15:07.726661+0000","last_undegraded":"2024-03-24T00:15:07.726661+0000","last_fullsized":"2024-03-24T00:15:07.726661+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:13:01.267198+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:13:01.267198+0000","last_clean_scrub_stamp":"2024-03-24T00:13:01.267198+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:45:26.779169+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8,1],"acting":[2,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.27","version":"88'2","reported_seq":24,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-24T00:15:07.752778+0000","last_change":"2024-03-24T00:13:02.321908+0000","last_active":"2024-03-24T00:15:07.752778+0000","last_peered":"2024-03-24T00:15:07.752778+0000","last_clean":"2024-03-24T00:15:07.752778+0000","last_became_active":"2024-03-24T00:13:02.321495+0000","last_became_peered":"2024-03-24T00:13:02.321495+0000","last_unstale":"2024-03-24T00:15:07.752778+0000","last_undegraded":"2024-03-24T00:15:07.752778+0000","last_fullsized":"2024-03-24T00:15:07.752778+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:13:01.267198+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:13:01.267198+0000","last_clean_scrub_stamp":"2024-03-24T00:13:01.267198+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T08:43:29.261314+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"88'6","reported_seq":28,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-24T00:15:07.747128+0000","last_change":"2024-03-24T00:13:02.331966+0000","last_active":"2024-03-24T00:15:07.747128+0000","last_peered":"2024-03-24T00:15:07.747128+0000","last_clean":"2024-03-24T00:15:07.747128+0000","last_became_active":"2024-03-24T00:13:02.331633+0000","last_became_peered":"2024-03-24T00:13:02.331633+0000","last_unstale":"2024-03-24T00:15:07.747128+0000","last_undegraded":"2024-03-24T00:15:07.747128+0000","last_fullsized":"2024-03-24T00:15:07.747128+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:13:01.267198+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:13:01.267198+0000","last_clean_scrub_stamp":"2024-03-24T00:13:01.267198+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T05:04:01.574351+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,3],"acting":[5,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"88'8","reported_seq":30,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-24T00:15:07.750202+0000","last_change":"2024-03-24T00:13:02.330344+0000","last_active":"2024-03-24T00:15:07.750202+0000","last_peered":"2024-03-24T00:15:07.750202+0000","last_clean":"2024-03-24T00:15:07.750202+0000","last_became_active":"2024-03-24T00:13:02.328218+0000","last_became_peered":"2024-03-24T00:13:02.328218+0000","last_unstale":"2024-03-24T00:15:07.750202+0000","last_undegraded":"2024-03-24T00:15:07.750202+0000","last_fullsized":"2024-03-24T00:15:07.750202+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:13:01.267198+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:13:01.267198+0000","last_clean_scrub_stamp":"2024-03-24T00:13:01.267198+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T08:54:36.392397+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,11],"acting":[3,2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"88'10","reported_seq":32,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-24T00:15:07.726927+0000","last_change":"2024-03-24T00:13:02.330956+0000","last_active":"2024-03-24T00:15:07.726927+0000","last_peered":"2024-03-24T00:15:07.726927+0000","last_clean":"2024-03-24T00:15:07.726927+0000","last_became_active":"2024-03-24T00:13:02.330642+0000","last_became_peered":"2024-03-24T00:13:02.330642+0000","last_unstale":"2024-03-24T00:15:07.726927+0000","last_undegraded":"2024-03-24T00:15:07.726927+0000","last_fullsized":"2024-03-24T00:15:07.726927+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:13:01.267198+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:13:01.267198+0000","last_clean_scrub_stamp":"2024-03-24T00:13:01.267198+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:41:11.326722+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2b","version":"88'11","reported_seq":33,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-24T00:15:07.751540+0000","last_change":"2024-03-24T00:13:02.333026+0000","last_active":"2024-03-24T00:15:07.751540+0000","last_peered":"2024-03-24T00:15:07.751540+0000","last_clean":"2024-03-24T00:15:07.751540+0000","last_became_active":"2024-03-24T00:13:02.331232+0000","last_became_peered":"2024-03-24T00:13:02.331232+0000","last_unstale":"2024-03-24T00:15:07.751540+0000","last_undegraded":"2024-03-24T00:15:07.751540+0000","last_fullsized":"2024-03-24T00:15:07.751540+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:13:01.267198+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:13:01.267198+0000","last_clean_scrub_stamp":"2024-03-24T00:13:01.267198+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T02:14:18.504012+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2c","version":"88'8","reported_seq":30,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-24T00:15:07.727828+0000","last_change":"2024-03-24T00:13:02.309023+0000","last_active":"2024-03-24T00:15:07.727828+0000","last_peered":"2024-03-24T00:15:07.727828+0000","last_clean":"2024-03-24T00:15:07.727828+0000","last_became_active":"2024-03-24T00:13:02.308236+0000","last_became_peered":"2024-03-24T00:13:02.308236+0000","last_unstale":"2024-03-24T00:15:07.727828+0000","last_undegraded":"2024-03-24T00:15:07.727828+0000","last_fullsized":"2024-03-24T00:15:07.727828+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:13:01.267198+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:13:01.267198+0000","last_clean_scrub_stamp":"2024-03-24T00:13:01.267198+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T03:30:37.427095+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,8],"acting":[11,4,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":234,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":702,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":502,"ondisk_log_size":502,"up":192,"acting":192,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":442368,"data_stored":4722944,"data_compressed":284592,"data_compressed_allocated":360448,"data_compressed_original":4669440,"omap_allocated":0,"internal_metadata":0},"log_size":137,"ondisk_log_size":137,"up":3,"acting":3,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":70524,"num_objects":98,"num_object_clones":0,"num_object_copies":294,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":98,"num_whiteouts":0,"num_read":1563,"num_read_kb":856,"num_write":254,"num_write_kb":149,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21,"num_bytes_recovered":62373,"num_keys_recovered":10,"num_objects_omap":66,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":219465,"data_compressed":13143,"data_compressed_allocated":20480,"data_compressed_original":73728,"omap_allocated":0,"internal_metadata":0},"log_size":221,"ondisk_log_size":221,"up":96,"acting":96,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":81,"seq":347892351021,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27980,"kb_used_data":960,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717172,"statfs":{"total":95995035648,"available":95966384128,"internally_reserved":0,"allocated":983040,"data_stored":478079,"data_compressed":11573,"data_compressed_allocated":16384,"data_compressed_original":32768,"omap_allocated":1596,"internal_metadata":27654596},"hb_peers":[0,1,2,3,4,5,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":73,"seq":313532612658,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28128,"kb_used_data":1112,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717024,"statfs":{"total":95995035648,"available":95966232576,"internally_reserved":0,"allocated":1138688,"data_stored":1130832,"data_compressed":49481,"data_compressed_allocated":65536,"data_compressed_original":630784,"omap_allocated":1599,"internal_metadata":27654593},"hb_peers":[0,1,2,3,4,5,6,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":68,"seq":292057776184,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32160,"kb_used_data":984,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712992,"statfs":{"total":95995035648,"available":95962103808,"internally_reserved":0,"allocated":1007616,"data_stored":477477,"data_compressed":11573,"data_compressed_allocated":16384,"data_compressed_original":32768,"omap_allocated":1600,"internal_metadata":31914432},"hb_peers":[0,1,2,3,4,5,6,7,8,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":60,"seq":257698037823,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36308,"kb_used_data":972,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708844,"statfs":{"total":95995035648,"available":95957856256,"internally_reserved":0,"allocated":995328,"data_stored":477183,"data_compressed":11573,"data_compressed_allocated":16384,"data_compressed_original":32768,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":52,"seq":223338299461,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28056,"kb_used_data":1032,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717096,"statfs":{"total":95995035648,"available":95966306304,"internally_reserved":0,"allocated":1056768,"data_stored":1070905,"data_compressed":47762,"data_compressed_allocated":61440,"data_compressed_original":622592,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":47,"seq":201863462987,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27992,"kb_used_data":968,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717160,"statfs":{"total":95995035648,"available":95966371840,"internally_reserved":0,"allocated":991232,"data_stored":477194,"data_compressed":11573,"data_compressed_allocated":16384,"data_compressed_original":32768,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574950,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28060,"kb_used_data":1044,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717092,"statfs":{"total":95995035648,"available":95966302208,"internally_reserved":0,"allocated":1069056,"data_stored":549319,"data_compressed":13292,"data_compressed_allocated":20480,"data_compressed_original":40960,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738476,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28012,"kb_used_data":988,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717140,"statfs":{"total":95995035648,"available":95966351360,"internally_reserved":0,"allocated":1011712,"data_stored":479456,"data_compressed":11573,"data_compressed_allocated":16384,"data_compressed_original":32768,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411425,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27996,"kb_used_data":980,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717156,"statfs":{"total":95995035648,"available":95966367744,"internally_reserved":0,"allocated":1003520,"data_stored":480173,"data_compressed":11573,"data_compressed_allocated":16384,"data_compressed_original":32768,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":26,"seq":111669149788,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32160,"kb_used_data":984,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712992,"statfs":{"total":95995035648,"available":95962103808,"internally_reserved":0,"allocated":1007616,"data_stored":477019,"data_compressed":11573,"data_compressed_allocated":16384,"data_compressed_original":32768,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":34,"seq":146028888149,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32172,"kb_used_data":996,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712980,"statfs":{"total":95995035648,"available":95962091520,"internally_reserved":0,"allocated":1019904,"data_stored":539311,"data_compressed":21278,"data_compressed_allocated":28672,"data_compressed_original":90112,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":39,"seq":167503724624,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28060,"kb_used_data":1044,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717092,"statfs":{"total":95995035648,"available":95966302208,"internally_reserved":0,"allocated":1069056,"data_stored":1070583,"data_compressed":47762,"data_compressed_allocated":61440,"data_compressed_original":622592,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34549,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34549,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34549,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":36189,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":36189,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":36189,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":36189,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":36189,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":3101,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":98304,"data_stored":72964,"data_compressed":1719,"data_compressed_allocated":4096,"data_compressed_original":8192,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":3818,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":430,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":62956,"data_compressed":9705,"data_compressed_allocated":12288,"data_compressed_original":57344,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":3743,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":722,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":4065,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":828,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":1122,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":114688,"data_stored":63992,"data_compressed":1719,"data_compressed_allocated":4096,"data_compressed_original":8192,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1724,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":234,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-24T00:15:43.938 DEBUG:teuthology.orchestra.run.smithi118:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-24T00:15:44.049 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:15:43 smithi118 ceph-mon[31535]: from='client.? 172.21.15.118:0/754829360' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-24T00:15:44.049 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:15:43 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T00:15:44.049 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:15:43 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T00:15:44.049 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:15:43 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T00:15:44.049 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:15:43 smithi118 ceph-mon[31535]: from='client.? 172.21.15.118:0/1702815803' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-24T00:15:44.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:15:43 smithi189 ceph-mon[36187]: from='client.? 172.21.15.118:0/754829360' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-24T00:15:44.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:15:43 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T00:15:44.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:15:43 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T00:15:44.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:15:43 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T00:15:44.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:15:43 smithi189 ceph-mon[36187]: from='client.? 172.21.15.118:0/1702815803' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-24T00:15:44.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:15:43 smithi193 ceph-mon[36878]: from='client.? 172.21.15.118:0/754829360' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-24T00:15:44.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:15:43 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T00:15:44.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:15:43 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T00:15:44.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:15:43 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T00:15:44.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:15:43 smithi193 ceph-mon[36878]: from='client.? 172.21.15.118:0/1702815803' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-24T00:15:44.238 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:44.235+0000 7f0a2e000640 1 -- 172.21.15.118:0/667579657 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0a28104a90 msgr2=0x7f0a28106e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:15:44.238 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:44.235+0000 7f0a2e000640 1 --2- 172.21.15.118:0/667579657 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0a28104a90 0x7f0a28106e80 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7f0a14009900 tx=0x7f0a1402bc30 comp rx=0 tx=0).stop 2024-03-24T00:15:44.238 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:44.235+0000 7f0a2e000640 1 -- 172.21.15.118:0/667579657 shutdown_connections 2024-03-24T00:15:44.238 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:44.235+0000 7f0a2e000640 1 --2- 172.21.15.118:0/667579657 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f0a281073c0 0x7f0a28109810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:15:44.239 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:44.235+0000 7f0a2e000640 1 --2- 172.21.15.118:0/667579657 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0a28104a90 0x7f0a28106e80 unknown :-1 s=CLOSED pgs=197 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:15:44.239 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:44.235+0000 7f0a2e000640 1 --2- 172.21.15.118:0/667579657 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f0a28102160 0x7f0a28104550 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:15:44.239 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:44.235+0000 7f0a2e000640 1 -- 172.21.15.118:0/667579657 >> 172.21.15.118:0/667579657 conn(0x7f0a280fbb50 msgr2=0x7f0a280fdf90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:15:44.239 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:44.235+0000 7f0a2e000640 1 -- 172.21.15.118:0/667579657 shutdown_connections 2024-03-24T00:15:44.239 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:44.235+0000 7f0a2e000640 1 -- 172.21.15.118:0/667579657 wait complete. 2024-03-24T00:15:44.239 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:44.236+0000 7f0a2e000640 1 Processor -- start 2024-03-24T00:15:44.239 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:44.236+0000 7f0a2e000640 1 -- start start 2024-03-24T00:15:44.239 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:44.236+0000 7f0a2e000640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f0a28102160 0x7f0a28101ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:15:44.240 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:44.236+0000 7f0a2e000640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f0a28104a90 0x7f0a28100110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:15:44.240 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:44.236+0000 7f0a2e000640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0a281073c0 0x7f0a28100650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:15:44.240 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:44.236+0000 7f0a2e000640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0a2810d3c0 con 0x7f0a28102160 2024-03-24T00:15:44.240 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:44.236+0000 7f0a2e000640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0a2810d240 con 0x7f0a281073c0 2024-03-24T00:15:44.240 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:44.236+0000 7f0a2e000640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0a2810d540 con 0x7f0a28104a90 2024-03-24T00:15:44.240 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:44.237+0000 7f0a27400640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f0a28102160 0x7f0a28101ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:15:44.240 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:44.237+0000 7f0a26a00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f0a28104a90 0x7f0a28100110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:15:44.240 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:44.237+0000 7f0a26a00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f0a28104a90 0x7f0a28100110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.118:33960/0 (socket says 172.21.15.118:33960) 2024-03-24T00:15:44.240 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:44.237+0000 7f0a26a00640 1 -- 172.21.15.118:0/659544044 learned_addr learned my addr 172.21.15.118:0/659544044 (peer_addr_for_me v2:172.21.15.118:0/0) 2024-03-24T00:15:44.241 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:44.237+0000 7f0a27e00640 1 --2- 172.21.15.118:0/659544044 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0a281073c0 0x7f0a28100650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:15:44.241 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:44.237+0000 7f0a26a00640 1 -- 172.21.15.118:0/659544044 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0a281073c0 msgr2=0x7f0a28100650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:15:44.241 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:44.237+0000 7f0a26a00640 1 --2- 172.21.15.118:0/659544044 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0a281073c0 0x7f0a28100650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:15:44.241 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:44.237+0000 7f0a26a00640 1 -- 172.21.15.118:0/659544044 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f0a28102160 msgr2=0x7f0a28101ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:15:44.241 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:44.237+0000 7f0a26a00640 1 --2- 172.21.15.118:0/659544044 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f0a28102160 0x7f0a28101ad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:15:44.241 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:44.237+0000 7f0a26a00640 1 -- 172.21.15.118:0/659544044 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0a28100ee0 con 0x7f0a28104a90 2024-03-24T00:15:44.242 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:44.237+0000 7f0a26a00640 1 --2- 172.21.15.118:0/659544044 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f0a28104a90 0x7f0a28100110 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7f0a14000c00 tx=0x7f0a1402c6c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:15:44.242 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:44.238+0000 7f0a1fe00640 1 -- 172.21.15.118:0/659544044 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0a14043020 con 0x7f0a28104a90 2024-03-24T00:15:44.242 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:44.238+0000 7f0a1fe00640 1 -- 172.21.15.118:0/659544044 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f0a140054e0 con 0x7f0a28104a90 2024-03-24T00:15:44.242 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:44.238+0000 7f0a1fe00640 1 -- 172.21.15.118:0/659544044 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0a14003ce0 con 0x7f0a28104a90 2024-03-24T00:15:44.242 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:44.238+0000 7f0a2e000640 1 -- 172.21.15.118:0/659544044 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0a281a1b60 con 0x7f0a28104a90 2024-03-24T00:15:44.243 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:44.238+0000 7f0a2e000640 1 -- 172.21.15.118:0/659544044 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0a281a2030 con 0x7f0a28104a90 2024-03-24T00:15:44.244 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:44.239+0000 7f0a2e000640 1 -- 172.21.15.118:0/659544044 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0a280fd640 con 0x7f0a28104a90 2024-03-24T00:15:44.244 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:44.241+0000 7f0a1fe00640 1 -- 172.21.15.118:0/659544044 <== mon.1 v2:172.21.15.193:3300/0 4 ==== mgrmap(e 19) v1 ==== 137342+0+0 (secure 0 0 0) 0x7f0a14046020 con 0x7f0a28104a90 2024-03-24T00:15:44.245 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:44.242+0000 7f0a1fe00640 1 --2- 172.21.15.118:0/659544044 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7f0a0c0a2910 0x7f0a0c0a4dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:15:44.245 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:44.242+0000 7f0a1fe00640 1 -- 172.21.15.118:0/659544044 <== mon.1 v2:172.21.15.193:3300/0 5 ==== osd_map(90..90 src has 1..90) v4 ==== 8277+0+0 (secure 0 0 0) 0x7f0a140f0be0 con 0x7f0a28104a90 2024-03-24T00:15:44.245 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:44.243+0000 7f0a27400640 1 --2- 172.21.15.118:0/659544044 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7f0a0c0a2910 0x7f0a0c0a4dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:15:44.245 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:44.243+0000 7f0a27400640 1 --2- 172.21.15.118:0/659544044 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7f0a0c0a2910 0x7f0a0c0a4dd0 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7f0a18009bd0 tx=0x7f0a18005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:15:44.250 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:44.250+0000 7f0a1fe00640 1 -- 172.21.15.118:0/659544044 <== mon.1 v2:172.21.15.193:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f0a1404d020 con 0x7f0a28104a90 2024-03-24T00:15:44.501 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:44.501+0000 7f0a2e000640 1 -- 172.21.15.118:0/659544044 --> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f0a28061ae0 con 0x7f0a0c0a2910 2024-03-24T00:15:44.514 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:44.515+0000 7f0a1fe00640 1 -- 172.21.15.118:0/659544044 <== mgr.14152 v2:172.21.15.118:6800/2112912943 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+259216 (secure 0 0 0) 0x7f0a28061ae0 con 0x7f0a0c0a2910 2024-03-24T00:15:44.518 INFO:teuthology.orchestra.run.smithi118.stdout: 2024-03-24T00:15:44.519 INFO:teuthology.orchestra.run.smithi118.stderr:dumped all 2024-03-24T00:15:44.521 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:44.519+0000 7f0a2e000640 1 -- 172.21.15.118:0/659544044 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7f0a0c0a2910 msgr2=0x7f0a0c0a4dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:15:44.521 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:44.519+0000 7f0a2e000640 1 --2- 172.21.15.118:0/659544044 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7f0a0c0a2910 0x7f0a0c0a4dd0 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7f0a18009bd0 tx=0x7f0a18005f90 comp rx=0 tx=0).stop 2024-03-24T00:15:44.521 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:44.519+0000 7f0a2e000640 1 -- 172.21.15.118:0/659544044 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f0a28104a90 msgr2=0x7f0a28100110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:15:44.521 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:44.519+0000 7f0a2e000640 1 --2- 172.21.15.118:0/659544044 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f0a28104a90 0x7f0a28100110 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7f0a14000c00 tx=0x7f0a1402c6c0 comp rx=0 tx=0).stop 2024-03-24T00:15:44.521 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:44.519+0000 7f0a2e000640 1 -- 172.21.15.118:0/659544044 shutdown_connections 2024-03-24T00:15:44.521 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:44.519+0000 7f0a2e000640 1 --2- 172.21.15.118:0/659544044 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7f0a0c0a2910 0x7f0a0c0a4dd0 unknown :-1 s=CLOSED pgs=261 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:15:44.522 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:44.519+0000 7f0a2e000640 1 --2- 172.21.15.118:0/659544044 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0a281073c0 0x7f0a28100650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:15:44.522 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:44.519+0000 7f0a2e000640 1 --2- 172.21.15.118:0/659544044 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f0a28104a90 0x7f0a28100110 unknown :-1 s=CLOSED pgs=191 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:15:44.522 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:44.519+0000 7f0a2e000640 1 --2- 172.21.15.118:0/659544044 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f0a28102160 0x7f0a28101ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:15:44.522 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:44.519+0000 7f0a2e000640 1 -- 172.21.15.118:0/659544044 >> 172.21.15.118:0/659544044 conn(0x7f0a280fbb50 msgr2=0x7f0a28102aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:15:44.522 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:44.520+0000 7f0a2e000640 1 -- 172.21.15.118:0/659544044 shutdown_connections 2024-03-24T00:15:44.522 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:44.520+0000 7f0a2e000640 1 -- 172.21.15.118:0/659544044 wait complete. 2024-03-24T00:15:44.537 INFO:teuthology.orchestra.run.smithi118.stdout:{"pg_ready":true,"pg_map":{"version":399,"stamp":"2024-03-24T00:15:43.668833+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":661126,"num_objects":102,"num_object_clones":0,"num_object_copies":306,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":102,"num_whiteouts":0,"num_read":1689,"num_read_kb":965,"num_write":437,"num_write_kb":3089,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":1243109,"num_keys_recovered":10,"num_objects_omap":66,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":860,"ondisk_log_size":860,"up":291,"acting":291,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":291,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":357084,"kb_used_data":12064,"kb_used_omap":18,"kb_used_meta":344877,"kb_avail":1124584740,"statfs":{"total":1151940427776,"available":1151574773760,"internally_reserved":0,"allocated":12353536,"data_stored":7707531,"data_compressed":260586,"data_compressed_allocated":352256,"data_compressed_original":2236416,"omap_allocated":19142,"internal_metadata":353154362},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":1088,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":122,"num_read_kb":62,"num_write":8,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004105"},"pg_stats":[{"pgid":"3.3f","version":"88'8","reported_seq":30,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-24T00:15:07.728049+0000","last_change":"2024-03-24T00:13:02.325815+0000","last_active":"2024-03-24T00:15:07.728049+0000","last_peered":"2024-03-24T00:15:07.728049+0000","last_clean":"2024-03-24T00:15:07.728049+0000","last_became_active":"2024-03-24T00:13:02.325450+0000","last_became_peered":"2024-03-24T00:13:02.325450+0000","last_unstale":"2024-03-24T00:15:07.728049+0000","last_undegraded":"2024-03-24T00:15:07.728049+0000","last_fullsized":"2024-03-24T00:15:07.728049+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:13:01.267198+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:13:01.267198+0000","last_clean_scrub_stamp":"2024-03-24T00:13:01.267198+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T02:18:21.994322+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.3e","version":"88'4","reported_seq":26,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-24T00:15:07.750213+0000","last_change":"2024-03-24T00:13:02.324015+0000","last_active":"2024-03-24T00:15:07.750213+0000","last_peered":"2024-03-24T00:15:07.750213+0000","last_clean":"2024-03-24T00:15:07.750213+0000","last_became_active":"2024-03-24T00:13:02.323668+0000","last_became_peered":"2024-03-24T00:13:02.323668+0000","last_unstale":"2024-03-24T00:15:07.750213+0000","last_undegraded":"2024-03-24T00:15:07.750213+0000","last_fullsized":"2024-03-24T00:15:07.750213+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:13:01.267198+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:13:01.267198+0000","last_clean_scrub_stamp":"2024-03-24T00:13:01.267198+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:36:19.442266+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11,10],"acting":[0,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3d","version":"88'7","reported_seq":29,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-24T00:15:07.749167+0000","last_change":"2024-03-24T00:13:02.328911+0000","last_active":"2024-03-24T00:15:07.749167+0000","last_peered":"2024-03-24T00:15:07.749167+0000","last_clean":"2024-03-24T00:15:07.749167+0000","last_became_active":"2024-03-24T00:13:02.324333+0000","last_became_peered":"2024-03-24T00:13:02.324333+0000","last_unstale":"2024-03-24T00:15:07.749167+0000","last_undegraded":"2024-03-24T00:15:07.749167+0000","last_fullsized":"2024-03-24T00:15:07.749167+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:13:01.267198+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:13:01.267198+0000","last_clean_scrub_stamp":"2024-03-24T00:13:01.267198+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T02:49:02.318097+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"88'8","reported_seq":30,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-24T00:15:07.748082+0000","last_change":"2024-03-24T00:13:02.309057+0000","last_active":"2024-03-24T00:15:07.748082+0000","last_peered":"2024-03-24T00:15:07.748082+0000","last_clean":"2024-03-24T00:15:07.748082+0000","last_became_active":"2024-03-24T00:13:02.308403+0000","last_became_peered":"2024-03-24T00:13:02.308403+0000","last_unstale":"2024-03-24T00:15:07.748082+0000","last_undegraded":"2024-03-24T00:15:07.748082+0000","last_fullsized":"2024-03-24T00:15:07.748082+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:13:01.267198+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:13:01.267198+0000","last_clean_scrub_stamp":"2024-03-24T00:13:01.267198+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T07:54:44.310833+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,8,2],"acting":[4,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.3b","version":"88'4","reported_seq":26,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-24T00:15:07.729685+0000","last_change":"2024-03-24T00:13:02.330651+0000","last_active":"2024-03-24T00:15:07.729685+0000","last_peered":"2024-03-24T00:15:07.729685+0000","last_clean":"2024-03-24T00:15:07.729685+0000","last_became_active":"2024-03-24T00:13:02.330313+0000","last_became_peered":"2024-03-24T00:13:02.330313+0000","last_unstale":"2024-03-24T00:15:07.729685+0000","last_undegraded":"2024-03-24T00:15:07.729685+0000","last_fullsized":"2024-03-24T00:15:07.729685+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:13:01.267198+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:13:01.267198+0000","last_clean_scrub_stamp":"2024-03-24T00:13:01.267198+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T03:45:38.480929+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,6],"acting":[8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3a","version":"88'6","reported_seq":28,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-24T00:15:07.747612+0000","last_change":"2024-03-24T00:13:02.313115+0000","last_active":"2024-03-24T00:15:07.747612+0000","last_peered":"2024-03-24T00:15:07.747612+0000","last_clean":"2024-03-24T00:15:07.747612+0000","last_became_active":"2024-03-24T00:13:02.312387+0000","last_became_peered":"2024-03-24T00:13:02.312387+0000","last_unstale":"2024-03-24T00:15:07.747612+0000","last_undegraded":"2024-03-24T00:15:07.747612+0000","last_fullsized":"2024-03-24T00:15:07.747612+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:13:01.267198+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:13:01.267198+0000","last_clean_scrub_stamp":"2024-03-24T00:13:01.267198+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T09:16:02.574920+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,4],"acting":[10,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.39","version":"88'5","reported_seq":33,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-24T00:15:07.729862+0000","last_change":"2024-03-24T00:14:07.503364+0000","last_active":"2024-03-24T00:15:07.729862+0000","last_peered":"2024-03-24T00:15:07.729862+0000","last_clean":"2024-03-24T00:15:07.729862+0000","last_became_active":"2024-03-24T00:14:07.503103+0000","last_became_peered":"2024-03-24T00:14:07.503103+0000","last_unstale":"2024-03-24T00:15:07.729862+0000","last_undegraded":"2024-03-24T00:15:07.729862+0000","last_fullsized":"2024-03-24T00:15:07.729862+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:13:01.267198+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:13:01.267198+0000","last_clean_scrub_stamp":"2024-03-24T00:13:01.267198+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T03:22:04.225634+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,8],"acting":[7,9,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"88'9","reported_seq":24,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-24T00:15:07.729082+0000","last_change":"2024-03-24T00:14:07.681011+0000","last_active":"2024-03-24T00:15:07.729082+0000","last_peered":"2024-03-24T00:15:07.729082+0000","last_clean":"2024-03-24T00:15:07.729082+0000","last_became_active":"2024-03-24T00:14:07.680726+0000","last_became_peered":"2024-03-24T00:14:07.680726+0000","last_unstale":"2024-03-24T00:15:07.729082+0000","last_undegraded":"2024-03-24T00:15:07.729082+0000","last_fullsized":"2024-03-24T00:15:07.729082+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:13:01.267198+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:13:01.267198+0000","last_clean_scrub_stamp":"2024-03-24T00:13:01.267198+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T01:05:03.379937+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,5,10],"acting":[9,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.37","version":"88'11","reported_seq":33,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-24T00:15:07.727279+0000","last_change":"2024-03-24T00:13:02.305625+0000","last_active":"2024-03-24T00:15:07.727279+0000","last_peered":"2024-03-24T00:15:07.727279+0000","last_clean":"2024-03-24T00:15:07.727279+0000","last_became_active":"2024-03-24T00:13:02.305221+0000","last_became_peered":"2024-03-24T00:13:02.305221+0000","last_unstale":"2024-03-24T00:15:07.727279+0000","last_undegraded":"2024-03-24T00:15:07.727279+0000","last_fullsized":"2024-03-24T00:15:07.727279+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:13:01.267198+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:13:01.267198+0000","last_clean_scrub_stamp":"2024-03-24T00:13:01.267198+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T04:37:12.467387+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,11],"acting":[1,5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.36","version":"88'16","reported_seq":38,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-24T00:15:07.747823+0000","last_change":"2024-03-24T00:13:02.307878+0000","last_active":"2024-03-24T00:15:07.747823+0000","last_peered":"2024-03-24T00:15:07.747823+0000","last_clean":"2024-03-24T00:15:07.747823+0000","last_became_active":"2024-03-24T00:13:02.307513+0000","last_became_peered":"2024-03-24T00:13:02.307513+0000","last_unstale":"2024-03-24T00:15:07.747823+0000","last_undegraded":"2024-03-24T00:15:07.747823+0000","last_fullsized":"2024-03-24T00:15:07.747823+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:13:01.267198+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:13:01.267198+0000","last_clean_scrub_stamp":"2024-03-24T00:13:01.267198+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T07:06:08.162416+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,4],"acting":[10,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.35","version":"88'6","reported_seq":28,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-24T00:15:07.729986+0000","last_change":"2024-03-24T00:13:02.310883+0000","last_active":"2024-03-24T00:15:07.729986+0000","last_peered":"2024-03-24T00:15:07.729986+0000","last_clean":"2024-03-24T00:15:07.729986+0000","last_became_active":"2024-03-24T00:13:02.310559+0000","last_became_peered":"2024-03-24T00:13:02.310559+0000","last_unstale":"2024-03-24T00:15:07.729986+0000","last_undegraded":"2024-03-24T00:15:07.729986+0000","last_fullsized":"2024-03-24T00:15:07.729986+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:13:01.267198+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:13:01.267198+0000","last_clean_scrub_stamp":"2024-03-24T00:13:01.267198+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T04:04:25.801248+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8,1],"acting":[7,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.34","version":"88'7","reported_seq":29,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-24T00:15:07.748067+0000","last_change":"2024-03-24T00:13:02.308747+0000","last_active":"2024-03-24T00:15:07.748067+0000","last_peered":"2024-03-24T00:15:07.748067+0000","last_clean":"2024-03-24T00:15:07.748067+0000","last_became_active":"2024-03-24T00:13:02.307791+0000","last_became_peered":"2024-03-24T00:13:02.307791+0000","last_unstale":"2024-03-24T00:15:07.748067+0000","last_undegraded":"2024-03-24T00:15:07.748067+0000","last_fullsized":"2024-03-24T00:15:07.748067+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:13:01.267198+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:13:01.267198+0000","last_clean_scrub_stamp":"2024-03-24T00:13:01.267198+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T03:28:08.780044+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,8],"acting":[4,10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"88'10","reported_seq":32,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-24T00:15:07.751605+0000","last_change":"2024-03-24T00:13:02.327061+0000","last_active":"2024-03-24T00:15:07.751605+0000","last_peered":"2024-03-24T00:15:07.751605+0000","last_clean":"2024-03-24T00:15:07.751605+0000","last_became_active":"2024-03-24T00:13:02.326696+0000","last_became_peered":"2024-03-24T00:13:02.326696+0000","last_unstale":"2024-03-24T00:15:07.751605+0000","last_undegraded":"2024-03-24T00:15:07.751605+0000","last_fullsized":"2024-03-24T00:15:07.751605+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:13:01.267198+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:13:01.267198+0000","last_clean_scrub_stamp":"2024-03-24T00:13:01.267198+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T08:55:17.325488+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,5],"acting":[6,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.32","version":"88'7","reported_seq":29,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-24T00:15:07.748893+0000","last_change":"2024-03-24T00:13:02.321326+0000","last_active":"2024-03-24T00:15:07.748893+0000","last_peered":"2024-03-24T00:15:07.748893+0000","last_clean":"2024-03-24T00:15:07.748893+0000","last_became_active":"2024-03-24T00:13:02.320964+0000","last_became_peered":"2024-03-24T00:13:02.320964+0000","last_unstale":"2024-03-24T00:15:07.748893+0000","last_undegraded":"2024-03-24T00:15:07.748893+0000","last_fullsized":"2024-03-24T00:15:07.748893+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:13:01.267198+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:13:01.267198+0000","last_clean_scrub_stamp":"2024-03-24T00:13:01.267198+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T08:05:15.365809+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,10],"acting":[0,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"88'7","reported_seq":35,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-24T00:15:07.748624+0000","last_change":"2024-03-24T00:14:07.505774+0000","last_active":"2024-03-24T00:15:07.748624+0000","last_peered":"2024-03-24T00:15:07.748624+0000","last_clean":"2024-03-24T00:15:07.748624+0000","last_became_active":"2024-03-24T00:14:07.504800+0000","last_became_peered":"2024-03-24T00:14:07.504800+0000","last_unstale":"2024-03-24T00:15:07.748624+0000","last_undegraded":"2024-03-24T00:15:07.748624+0000","last_fullsized":"2024-03-24T00:15:07.748624+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:13:01.267198+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:13:01.267198+0000","last_clean_scrub_stamp":"2024-03-24T00:13:01.267198+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T06:50:30.686330+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.30","version":"88'9","reported_seq":31,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-24T00:15:07.726732+0000","last_change":"2024-03-24T00:13:02.325460+0000","last_active":"2024-03-24T00:15:07.726732+0000","last_peered":"2024-03-24T00:15:07.726732+0000","last_clean":"2024-03-24T00:15:07.726732+0000","last_became_active":"2024-03-24T00:13:02.325112+0000","last_became_peered":"2024-03-24T00:13:02.325112+0000","last_unstale":"2024-03-24T00:15:07.726732+0000","last_undegraded":"2024-03-24T00:15:07.726732+0000","last_fullsized":"2024-03-24T00:15:07.726732+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:13:01.267198+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:13:01.267198+0000","last_clean_scrub_stamp":"2024-03-24T00:13:01.267198+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T03:15:16.362970+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2f","version":"88'8","reported_seq":30,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-24T00:15:07.728294+0000","last_change":"2024-03-24T00:13:02.325918+0000","last_active":"2024-03-24T00:15:07.728294+0000","last_peered":"2024-03-24T00:15:07.728294+0000","last_clean":"2024-03-24T00:15:07.728294+0000","last_became_active":"2024-03-24T00:13:02.325557+0000","last_became_peered":"2024-03-24T00:13:02.325557+0000","last_unstale":"2024-03-24T00:15:07.728294+0000","last_undegraded":"2024-03-24T00:15:07.728294+0000","last_fullsized":"2024-03-24T00:15:07.728294+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:13:01.267198+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:13:01.267198+0000","last_clean_scrub_stamp":"2024-03-24T00:13:01.267198+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T02:55:43.759674+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2e","version":"88'10","reported_seq":32,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-24T00:15:07.729972+0000","last_change":"2024-03-24T00:13:02.308106+0000","last_active":"2024-03-24T00:15:07.729972+0000","last_peered":"2024-03-24T00:15:07.729972+0000","last_clean":"2024-03-24T00:15:07.729972+0000","last_became_active":"2024-03-24T00:13:02.306972+0000","last_became_peered":"2024-03-24T00:13:02.306972+0000","last_unstale":"2024-03-24T00:15:07.729972+0000","last_undegraded":"2024-03-24T00:15:07.729972+0000","last_fullsized":"2024-03-24T00:15:07.729972+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:13:01.267198+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:13:01.267198+0000","last_clean_scrub_stamp":"2024-03-24T00:13:01.267198+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T09:38:54.677255+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,10],"acting":[9,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.2d","version":"88'7","reported_seq":29,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-24T00:15:07.726431+0000","last_change":"2024-03-24T00:13:02.331742+0000","last_active":"2024-03-24T00:15:07.726431+0000","last_peered":"2024-03-24T00:15:07.726431+0000","last_clean":"2024-03-24T00:15:07.726431+0000","last_became_active":"2024-03-24T00:13:02.331103+0000","last_became_peered":"2024-03-24T00:13:02.331103+0000","last_unstale":"2024-03-24T00:15:07.726431+0000","last_undegraded":"2024-03-24T00:15:07.726431+0000","last_fullsized":"2024-03-24T00:15:07.726431+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:13:01.267198+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:13:01.267198+0000","last_clean_scrub_stamp":"2024-03-24T00:13:01.267198+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:25:58.116306+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1f","version":"86'13","reported_seq":211,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-24T00:15:40.682427+0000","last_change":"2024-03-24T00:14:07.520827+0000","last_active":"2024-03-24T00:15:40.682427+0000","last_peered":"2024-03-24T00:15:40.682427+0000","last_clean":"2024-03-24T00:15:40.682427+0000","last_became_active":"2024-03-24T00:14:07.505406+0000","last_became_peered":"2024-03-24T00:14:07.505406+0000","last_unstale":"2024-03-24T00:15:40.682427+0000","last_undegraded":"2024-03-24T00:15:40.682427+0000","last_fullsized":"2024-03-24T00:15:40.682427+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:13:00.222790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:13:00.222790+0000","last_clean_scrub_stamp":"2024-03-24T00:13:00.222790+0000","objects_scrubbed":0,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T01:05:01.235978+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":152,"num_read_kb":152,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":672,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,1],"acting":[9,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.1e","version":"88'5","reported_seq":27,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-24T00:15:07.751376+0000","last_change":"2024-03-24T00:13:02.322461+0000","last_active":"2024-03-24T00:15:07.751376+0000","last_peered":"2024-03-24T00:15:07.751376+0000","last_clean":"2024-03-24T00:15:07.751376+0000","last_became_active":"2024-03-24T00:13:02.321752+0000","last_became_peered":"2024-03-24T00:13:02.321752+0000","last_unstale":"2024-03-24T00:15:07.751376+0000","last_undegraded":"2024-03-24T00:15:07.751376+0000","last_fullsized":"2024-03-24T00:15:07.751376+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:13:01.267198+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:13:01.267198+0000","last_clean_scrub_stamp":"2024-03-24T00:13:01.267198+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T06:37:05.825919+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1e","version":"86'2","reported_seq":26,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-24T00:15:07.752241+0000","last_change":"2024-03-24T00:13:01.324972+0000","last_active":"2024-03-24T00:15:07.752241+0000","last_peered":"2024-03-24T00:15:07.752241+0000","last_clean":"2024-03-24T00:15:07.752241+0000","last_became_active":"2024-03-24T00:13:01.323342+0000","last_became_peered":"2024-03-24T00:13:01.323342+0000","last_unstale":"2024-03-24T00:15:07.752241+0000","last_undegraded":"2024-03-24T00:15:07.752241+0000","last_fullsized":"2024-03-24T00:15:07.752241+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:13:00.222790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:13:00.222790+0000","last_clean_scrub_stamp":"2024-03-24T00:13:00.222790+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T05:34:35.827634+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"88'6","reported_seq":28,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-24T00:15:07.749057+0000","last_change":"2024-03-24T00:13:02.325455+0000","last_active":"2024-03-24T00:15:07.749057+0000","last_peered":"2024-03-24T00:15:07.749057+0000","last_clean":"2024-03-24T00:15:07.749057+0000","last_became_active":"2024-03-24T00:13:02.325061+0000","last_became_peered":"2024-03-24T00:13:02.325061+0000","last_unstale":"2024-03-24T00:15:07.749057+0000","last_undegraded":"2024-03-24T00:15:07.749057+0000","last_fullsized":"2024-03-24T00:15:07.749057+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:13:01.267198+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:13:01.267198+0000","last_clean_scrub_stamp":"2024-03-24T00:13:01.267198+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T06:13:33.223361+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,9],"acting":[0,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1d","version":"86'12","reported_seq":27,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-24T00:15:07.746267+0000","last_change":"2024-03-24T00:15:07.745753+0000","last_active":"2024-03-24T00:15:07.746267+0000","last_peered":"2024-03-24T00:15:07.746267+0000","last_clean":"2024-03-24T00:15:07.746267+0000","last_became_active":"2024-03-24T00:15:07.730583+0000","last_became_peered":"2024-03-24T00:15:07.730583+0000","last_unstale":"2024-03-24T00:15:07.746267+0000","last_undegraded":"2024-03-24T00:15:07.746267+0000","last_fullsized":"2024-03-24T00:15:07.746267+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:13:00.222790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:13:00.222790+0000","last_clean_scrub_stamp":"2024-03-24T00:13:00.222790+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T07:33:32.085470+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":0,"num_keys_recovered":10,"num_objects_omap":7,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,7],"acting":[11,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.1c","version":"88'9","reported_seq":31,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-24T00:15:07.726909+0000","last_change":"2024-03-24T00:13:02.325130+0000","last_active":"2024-03-24T00:15:07.726909+0000","last_peered":"2024-03-24T00:15:07.726909+0000","last_clean":"2024-03-24T00:15:07.726909+0000","last_became_active":"2024-03-24T00:13:02.324740+0000","last_became_peered":"2024-03-24T00:13:02.324740+0000","last_unstale":"2024-03-24T00:15:07.726909+0000","last_undegraded":"2024-03-24T00:15:07.726909+0000","last_fullsized":"2024-03-24T00:15:07.726909+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:13:01.267198+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:13:01.267198+0000","last_clean_scrub_stamp":"2024-03-24T00:13:01.267198+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T01:23:35.194788+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,6],"acting":[1,10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1c","version":"86'5","reported_seq":30,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-24T00:15:07.729993+0000","last_change":"2024-03-24T00:13:01.300572+0000","last_active":"2024-03-24T00:15:07.729993+0000","last_peered":"2024-03-24T00:15:07.729993+0000","last_clean":"2024-03-24T00:15:07.729993+0000","last_became_active":"2024-03-24T00:13:01.299801+0000","last_became_peered":"2024-03-24T00:13:01.299801+0000","last_unstale":"2024-03-24T00:15:07.729993+0000","last_undegraded":"2024-03-24T00:15:07.729993+0000","last_fullsized":"2024-03-24T00:15:07.729993+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:13:00.222790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:13:00.222790+0000","last_clean_scrub_stamp":"2024-03-24T00:13:00.222790+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T10:29:03.986969+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":7,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,5],"acting":[8,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1d","version":"88'15","reported_seq":37,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-24T00:15:07.747448+0000","last_change":"2024-03-24T00:13:02.304079+0000","last_active":"2024-03-24T00:15:07.747448+0000","last_peered":"2024-03-24T00:15:07.747448+0000","last_clean":"2024-03-24T00:15:07.747448+0000","last_became_active":"2024-03-24T00:13:02.303700+0000","last_became_peered":"2024-03-24T00:13:02.303700+0000","last_unstale":"2024-03-24T00:15:07.747448+0000","last_undegraded":"2024-03-24T00:15:07.747448+0000","last_fullsized":"2024-03-24T00:15:07.747448+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:13:01.267198+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:13:01.267198+0000","last_clean_scrub_stamp":"2024-03-24T00:13:01.267198+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T10:19:07.701927+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1b","version":"90'8","reported_seq":193,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-24T00:15:40.437307+0000","last_change":"2024-03-24T00:13:01.300949+0000","last_active":"2024-03-24T00:15:40.437307+0000","last_peered":"2024-03-24T00:15:40.437307+0000","last_clean":"2024-03-24T00:15:40.437307+0000","last_became_active":"2024-03-24T00:13:01.300605+0000","last_became_peered":"2024-03-24T00:13:01.300605+0000","last_unstale":"2024-03-24T00:15:40.437307+0000","last_undegraded":"2024-03-24T00:15:40.437307+0000","last_fullsized":"2024-03-24T00:15:40.437307+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:13:00.222790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:13:00.222790+0000","last_clean_scrub_stamp":"2024-03-24T00:13:00.222790+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T02:47:58.283103+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1545,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":161,"num_read_kb":5,"num_write":5,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1a","version":"88'7","reported_seq":29,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-24T00:15:07.729749+0000","last_change":"2024-03-24T00:13:02.312884+0000","last_active":"2024-03-24T00:15:07.729749+0000","last_peered":"2024-03-24T00:15:07.729749+0000","last_clean":"2024-03-24T00:15:07.729749+0000","last_became_active":"2024-03-24T00:13:02.312533+0000","last_became_peered":"2024-03-24T00:13:02.312533+0000","last_unstale":"2024-03-24T00:15:07.729749+0000","last_undegraded":"2024-03-24T00:15:07.729749+0000","last_fullsized":"2024-03-24T00:15:07.729749+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:13:01.267198+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:13:01.267198+0000","last_clean_scrub_stamp":"2024-03-24T00:13:01.267198+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T10:54:48.398645+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,5],"acting":[8,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.1a","version":"86'7","reported_seq":31,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-24T00:15:07.747575+0000","last_change":"2024-03-24T00:13:01.306292+0000","last_active":"2024-03-24T00:15:07.747575+0000","last_peered":"2024-03-24T00:15:07.747575+0000","last_clean":"2024-03-24T00:15:07.747575+0000","last_became_active":"2024-03-24T00:13:01.305898+0000","last_became_peered":"2024-03-24T00:13:01.305898+0000","last_unstale":"2024-03-24T00:15:07.747575+0000","last_undegraded":"2024-03-24T00:15:07.747575+0000","last_fullsized":"2024-03-24T00:15:07.747575+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:13:00.222790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:13:00.222790+0000","last_clean_scrub_stamp":"2024-03-24T00:13:00.222790+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T07:41:17.003588+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8,6],"acting":[10,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.1b","version":"88'8","reported_seq":30,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-24T00:15:07.747429+0000","last_change":"2024-03-24T00:13:02.323763+0000","last_active":"2024-03-24T00:15:07.747429+0000","last_peered":"2024-03-24T00:15:07.747429+0000","last_clean":"2024-03-24T00:15:07.747429+0000","last_became_active":"2024-03-24T00:13:02.323367+0000","last_became_peered":"2024-03-24T00:13:02.323367+0000","last_unstale":"2024-03-24T00:15:07.747429+0000","last_undegraded":"2024-03-24T00:15:07.747429+0000","last_fullsized":"2024-03-24T00:15:07.747429+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:13:01.267198+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:13:01.267198+0000","last_clean_scrub_stamp":"2024-03-24T00:13:01.267198+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T08:53:21.671317+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,3],"acting":[5,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.19","version":"86'3","reported_seq":27,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-24T00:15:07.751196+0000","last_change":"2024-03-24T00:13:01.323111+0000","last_active":"2024-03-24T00:15:07.751196+0000","last_peered":"2024-03-24T00:15:07.751196+0000","last_clean":"2024-03-24T00:15:07.751196+0000","last_became_active":"2024-03-24T00:13:01.322772+0000","last_became_peered":"2024-03-24T00:13:01.322772+0000","last_unstale":"2024-03-24T00:15:07.751196+0000","last_undegraded":"2024-03-24T00:15:07.751196+0000","last_fullsized":"2024-03-24T00:15:07.751196+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:13:00.222790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:13:00.222790+0000","last_clean_scrub_stamp":"2024-03-24T00:13:00.222790+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T04:19:53.097933+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,9],"acting":[6,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"88'8","reported_seq":30,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-24T00:15:07.750057+0000","last_change":"2024-03-24T00:13:02.330469+0000","last_active":"2024-03-24T00:15:07.750057+0000","last_peered":"2024-03-24T00:15:07.750057+0000","last_clean":"2024-03-24T00:15:07.750057+0000","last_became_active":"2024-03-24T00:13:02.327019+0000","last_became_peered":"2024-03-24T00:13:02.327019+0000","last_unstale":"2024-03-24T00:15:07.750057+0000","last_undegraded":"2024-03-24T00:15:07.750057+0000","last_fullsized":"2024-03-24T00:15:07.750057+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:13:01.267198+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:13:01.267198+0000","last_clean_scrub_stamp":"2024-03-24T00:13:01.267198+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T07:30:37.155535+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,6],"acting":[3,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.18","version":"86'3","reported_seq":28,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-24T00:15:07.747073+0000","last_change":"2024-03-24T00:13:01.300089+0000","last_active":"2024-03-24T00:15:07.747073+0000","last_peered":"2024-03-24T00:15:07.747073+0000","last_clean":"2024-03-24T00:15:07.747073+0000","last_became_active":"2024-03-24T00:13:01.299747+0000","last_became_peered":"2024-03-24T00:13:01.299747+0000","last_unstale":"2024-03-24T00:15:07.747073+0000","last_undegraded":"2024-03-24T00:15:07.747073+0000","last_fullsized":"2024-03-24T00:15:07.747073+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:13:00.222790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:13:00.222790+0000","last_clean_scrub_stamp":"2024-03-24T00:13:00.222790+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T00:30:30.142785+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"88'11","reported_seq":33,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-24T00:15:07.729674+0000","last_change":"2024-03-24T00:13:02.332490+0000","last_active":"2024-03-24T00:15:07.729674+0000","last_peered":"2024-03-24T00:15:07.729674+0000","last_clean":"2024-03-24T00:15:07.729674+0000","last_became_active":"2024-03-24T00:13:02.331182+0000","last_became_peered":"2024-03-24T00:13:02.331182+0000","last_unstale":"2024-03-24T00:15:07.729674+0000","last_undegraded":"2024-03-24T00:15:07.729674+0000","last_fullsized":"2024-03-24T00:15:07.729674+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:13:01.267198+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:13:01.267198+0000","last_clean_scrub_stamp":"2024-03-24T00:13:01.267198+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T06:22:16.084879+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,10],"acting":[9,3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.17","version":"88'6","reported_seq":333,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-24T00:15:38.433780+0000","last_change":"2024-03-24T00:13:01.299818+0000","last_active":"2024-03-24T00:15:38.433780+0000","last_peered":"2024-03-24T00:15:38.433780+0000","last_clean":"2024-03-24T00:15:38.433780+0000","last_became_active":"2024-03-24T00:13:01.299478+0000","last_became_peered":"2024-03-24T00:13:01.299478+0000","last_unstale":"2024-03-24T00:15:38.433780+0000","last_undegraded":"2024-03-24T00:15:38.433780+0000","last_fullsized":"2024-03-24T00:15:38.433780+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:13:00.222790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:13:00.222790+0000","last_clean_scrub_stamp":"2024-03-24T00:13:00.222790+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:51:47.088677+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":180,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":303,"num_read_kb":303,"num_write":7,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,9],"acting":[10,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.16","version":"88'10","reported_seq":32,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-24T00:15:07.727190+0000","last_change":"2024-03-24T00:13:02.304971+0000","last_active":"2024-03-24T00:15:07.727190+0000","last_peered":"2024-03-24T00:15:07.727190+0000","last_clean":"2024-03-24T00:15:07.727190+0000","last_became_active":"2024-03-24T00:13:02.304577+0000","last_became_peered":"2024-03-24T00:13:02.304577+0000","last_unstale":"2024-03-24T00:15:07.727190+0000","last_undegraded":"2024-03-24T00:15:07.727190+0000","last_fullsized":"2024-03-24T00:15:07.727190+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:13:01.267198+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:13:01.267198+0000","last_clean_scrub_stamp":"2024-03-24T00:13:01.267198+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T05:08:03.444554+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,9],"acting":[1,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.16","version":"86'4","reported_seq":28,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-24T00:15:07.729906+0000","last_change":"2024-03-24T00:13:01.303075+0000","last_active":"2024-03-24T00:15:07.729906+0000","last_peered":"2024-03-24T00:15:07.729906+0000","last_clean":"2024-03-24T00:15:07.729906+0000","last_became_active":"2024-03-24T00:13:01.301517+0000","last_became_peered":"2024-03-24T00:13:01.301517+0000","last_unstale":"2024-03-24T00:15:07.729906+0000","last_undegraded":"2024-03-24T00:15:07.729906+0000","last_fullsized":"2024-03-24T00:15:07.729906+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:13:00.222790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:13:00.222790+0000","last_clean_scrub_stamp":"2024-03-24T00:13:00.222790+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:04:10.552510+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.17","version":"88'9","reported_seq":31,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-24T00:15:07.749184+0000","last_change":"2024-03-24T00:13:02.320108+0000","last_active":"2024-03-24T00:15:07.749184+0000","last_peered":"2024-03-24T00:15:07.749184+0000","last_clean":"2024-03-24T00:15:07.749184+0000","last_became_active":"2024-03-24T00:13:02.319155+0000","last_became_peered":"2024-03-24T00:13:02.319155+0000","last_unstale":"2024-03-24T00:15:07.749184+0000","last_undegraded":"2024-03-24T00:15:07.749184+0000","last_fullsized":"2024-03-24T00:15:07.749184+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:13:01.267198+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:13:01.267198+0000","last_clean_scrub_stamp":"2024-03-24T00:13:01.267198+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T10:15:54.770304+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,5],"acting":[0,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.15","version":"86'5","reported_seq":30,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-24T00:15:07.747498+0000","last_change":"2024-03-24T00:13:01.323919+0000","last_active":"2024-03-24T00:15:07.747498+0000","last_peered":"2024-03-24T00:15:07.747498+0000","last_clean":"2024-03-24T00:15:07.747498+0000","last_became_active":"2024-03-24T00:13:01.323640+0000","last_became_peered":"2024-03-24T00:13:01.323640+0000","last_unstale":"2024-03-24T00:15:07.747498+0000","last_undegraded":"2024-03-24T00:15:07.747498+0000","last_fullsized":"2024-03-24T00:15:07.747498+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:13:00.222790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:13:00.222790+0000","last_clean_scrub_stamp":"2024-03-24T00:13:00.222790+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T06:16:49.018642+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":7,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,10],"acting":[5,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.14","version":"88'11","reported_seq":39,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-24T00:15:07.727032+0000","last_change":"2024-03-24T00:14:07.635409+0000","last_active":"2024-03-24T00:15:07.727032+0000","last_peered":"2024-03-24T00:15:07.727032+0000","last_clean":"2024-03-24T00:15:07.727032+0000","last_became_active":"2024-03-24T00:14:07.635093+0000","last_became_peered":"2024-03-24T00:14:07.635093+0000","last_unstale":"2024-03-24T00:15:07.727032+0000","last_undegraded":"2024-03-24T00:15:07.727032+0000","last_fullsized":"2024-03-24T00:15:07.727032+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:13:01.267198+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:13:01.267198+0000","last_clean_scrub_stamp":"2024-03-24T00:13:01.267198+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T10:31:10.205874+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,6],"acting":[1,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.14","version":"90'37","reported_seq":397,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-24T00:15:38.895751+0000","last_change":"2024-03-24T00:14:07.807393+0000","last_active":"2024-03-24T00:15:38.895751+0000","last_peered":"2024-03-24T00:15:38.895751+0000","last_clean":"2024-03-24T00:15:38.895751+0000","last_became_active":"2024-03-24T00:14:07.801140+0000","last_became_peered":"2024-03-24T00:14:07.801140+0000","last_unstale":"2024-03-24T00:15:38.895751+0000","last_undegraded":"2024-03-24T00:15:38.895751+0000","last_fullsized":"2024-03-24T00:15:38.895751+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:13:00.222790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:13:00.222790+0000","last_clean_scrub_stamp":"2024-03-24T00:13:00.222790+0000","objects_scrubbed":0,"log_size":37,"log_dups_size":0,"ondisk_log_size":37,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T10:35:25.627828+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":62256,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":312,"num_read_kb":216,"num_write":39,"num_write_kb":78,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":61701,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,4],"acting":[10,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.15","version":"88'6","reported_seq":28,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-24T00:15:07.729922+0000","last_change":"2024-03-24T00:13:02.307653+0000","last_active":"2024-03-24T00:15:07.729922+0000","last_peered":"2024-03-24T00:15:07.729922+0000","last_clean":"2024-03-24T00:15:07.729922+0000","last_became_active":"2024-03-24T00:13:02.307287+0000","last_became_peered":"2024-03-24T00:13:02.307287+0000","last_unstale":"2024-03-24T00:15:07.729922+0000","last_undegraded":"2024-03-24T00:15:07.729922+0000","last_fullsized":"2024-03-24T00:15:07.729922+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:13:01.267198+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:13:01.267198+0000","last_clean_scrub_stamp":"2024-03-24T00:13:01.267198+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T05:59:27.701614+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.13","version":"86'9","reported_seq":35,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-24T00:15:07.726557+0000","last_change":"2024-03-24T00:13:01.316854+0000","last_active":"2024-03-24T00:15:07.726557+0000","last_peered":"2024-03-24T00:15:07.726557+0000","last_clean":"2024-03-24T00:15:07.726557+0000","last_became_active":"2024-03-24T00:13:01.316507+0000","last_became_peered":"2024-03-24T00:13:01.316507+0000","last_unstale":"2024-03-24T00:15:07.726557+0000","last_undegraded":"2024-03-24T00:15:07.726557+0000","last_fullsized":"2024-03-24T00:15:07.726557+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:13:00.222790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:13:00.222790+0000","last_clean_scrub_stamp":"2024-03-24T00:13:00.222790+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T07:58:41.888003+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":13,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,10,3],"acting":[2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.12","version":"88'9","reported_seq":31,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-24T00:15:07.751081+0000","last_change":"2024-03-24T00:13:02.338811+0000","last_active":"2024-03-24T00:15:07.751081+0000","last_peered":"2024-03-24T00:15:07.751081+0000","last_clean":"2024-03-24T00:15:07.751081+0000","last_became_active":"2024-03-24T00:13:02.338438+0000","last_became_peered":"2024-03-24T00:13:02.338438+0000","last_unstale":"2024-03-24T00:15:07.751081+0000","last_undegraded":"2024-03-24T00:15:07.751081+0000","last_fullsized":"2024-03-24T00:15:07.751081+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:13:01.267198+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:13:01.267198+0000","last_clean_scrub_stamp":"2024-03-24T00:13:01.267198+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T03:55:16.531224+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.12","version":"86'5","reported_seq":30,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-24T00:15:07.747178+0000","last_change":"2024-03-24T00:13:01.313914+0000","last_active":"2024-03-24T00:15:07.747178+0000","last_peered":"2024-03-24T00:15:07.747178+0000","last_clean":"2024-03-24T00:15:07.747178+0000","last_became_active":"2024-03-24T00:13:01.313535+0000","last_became_peered":"2024-03-24T00:13:01.313535+0000","last_unstale":"2024-03-24T00:15:07.747178+0000","last_undegraded":"2024-03-24T00:15:07.747178+0000","last_fullsized":"2024-03-24T00:15:07.747178+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:13:00.222790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:13:00.222790+0000","last_clean_scrub_stamp":"2024-03-24T00:13:00.222790+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T03:23:53.011550+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":5,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"88'15","reported_seq":43,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-24T00:15:07.729806+0000","last_change":"2024-03-24T00:14:07.632978+0000","last_active":"2024-03-24T00:15:07.729806+0000","last_peered":"2024-03-24T00:15:07.729806+0000","last_clean":"2024-03-24T00:15:07.729806+0000","last_became_active":"2024-03-24T00:14:07.632719+0000","last_became_peered":"2024-03-24T00:14:07.632719+0000","last_unstale":"2024-03-24T00:15:07.729806+0000","last_undegraded":"2024-03-24T00:15:07.729806+0000","last_fullsized":"2024-03-24T00:15:07.729806+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:13:01.267198+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:13:01.267198+0000","last_clean_scrub_stamp":"2024-03-24T00:13:01.267198+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T09:59:38.993218+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,6],"acting":[7,9,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.11","version":"86'5","reported_seq":181,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-24T00:15:37.543087+0000","last_change":"2024-03-24T00:13:01.297887+0000","last_active":"2024-03-24T00:15:37.543087+0000","last_peered":"2024-03-24T00:15:37.543087+0000","last_clean":"2024-03-24T00:15:37.543087+0000","last_became_active":"2024-03-24T00:13:01.297560+0000","last_became_peered":"2024-03-24T00:13:01.297560+0000","last_unstale":"2024-03-24T00:15:37.543087+0000","last_undegraded":"2024-03-24T00:15:37.543087+0000","last_fullsized":"2024-03-24T00:15:37.543087+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:13:00.222790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:13:00.222790+0000","last_clean_scrub_stamp":"2024-03-24T00:13:00.222790+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T01:06:28.927274+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":588,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":153,"num_read_kb":3,"num_write":5,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,10],"acting":[4,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.10","version":"88'6","reported_seq":28,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-24T00:15:07.729530+0000","last_change":"2024-03-24T00:13:02.306811+0000","last_active":"2024-03-24T00:15:07.729530+0000","last_peered":"2024-03-24T00:15:07.729530+0000","last_clean":"2024-03-24T00:15:07.729530+0000","last_became_active":"2024-03-24T00:13:02.306499+0000","last_became_peered":"2024-03-24T00:13:02.306499+0000","last_unstale":"2024-03-24T00:15:07.729530+0000","last_undegraded":"2024-03-24T00:15:07.729530+0000","last_fullsized":"2024-03-24T00:15:07.729530+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:13:01.267198+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:13:01.267198+0000","last_clean_scrub_stamp":"2024-03-24T00:13:01.267198+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:49:32.683005+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.10","version":"86'17","reported_seq":42,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-24T00:15:07.729583+0000","last_change":"2024-03-24T00:13:01.307058+0000","last_active":"2024-03-24T00:15:07.729583+0000","last_peered":"2024-03-24T00:15:07.729583+0000","last_clean":"2024-03-24T00:15:07.729583+0000","last_became_active":"2024-03-24T00:13:01.306699+0000","last_became_peered":"2024-03-24T00:13:01.306699+0000","last_unstale":"2024-03-24T00:15:07.729583+0000","last_undegraded":"2024-03-24T00:15:07.729583+0000","last_fullsized":"2024-03-24T00:15:07.729583+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:13:00.222790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:13:00.222790+0000","last_clean_scrub_stamp":"2024-03-24T00:13:00.222790+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T10:32:37.334514+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,2],"acting":[8,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.11","version":"88'6","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-24T00:15:07.729239+0000","last_change":"2024-03-24T00:14:07.503357+0000","last_active":"2024-03-24T00:15:07.729239+0000","last_peered":"2024-03-24T00:15:07.729239+0000","last_clean":"2024-03-24T00:15:07.729239+0000","last_became_active":"2024-03-24T00:14:07.502511+0000","last_became_peered":"2024-03-24T00:14:07.502511+0000","last_unstale":"2024-03-24T00:15:07.729239+0000","last_undegraded":"2024-03-24T00:15:07.729239+0000","last_fullsized":"2024-03-24T00:15:07.729239+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:13:01.267198+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:13:01.267198+0000","last_clean_scrub_stamp":"2024-03-24T00:13:01.267198+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T04:18:09.351579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,1],"acting":[9,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.f","version":"86'2","reported_seq":27,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-24T00:15:07.730062+0000","last_change":"2024-03-24T00:13:01.295874+0000","last_active":"2024-03-24T00:15:07.730062+0000","last_peered":"2024-03-24T00:15:07.730062+0000","last_clean":"2024-03-24T00:15:07.730062+0000","last_became_active":"2024-03-24T00:13:01.295521+0000","last_became_peered":"2024-03-24T00:13:01.295521+0000","last_unstale":"2024-03-24T00:15:07.730062+0000","last_undegraded":"2024-03-24T00:15:07.730062+0000","last_fullsized":"2024-03-24T00:15:07.730062+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:13:00.222790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:13:00.222790+0000","last_clean_scrub_stamp":"2024-03-24T00:13:00.222790+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T06:47:15.749491+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5,4],"acting":[8,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.e","version":"88'5","reported_seq":27,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-24T00:15:07.728056+0000","last_change":"2024-03-24T00:13:02.306617+0000","last_active":"2024-03-24T00:15:07.728056+0000","last_peered":"2024-03-24T00:15:07.728056+0000","last_clean":"2024-03-24T00:15:07.728056+0000","last_became_active":"2024-03-24T00:13:02.306224+0000","last_became_peered":"2024-03-24T00:13:02.306224+0000","last_unstale":"2024-03-24T00:15:07.728056+0000","last_undegraded":"2024-03-24T00:15:07.728056+0000","last_fullsized":"2024-03-24T00:15:07.728056+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:13:01.267198+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:13:01.267198+0000","last_clean_scrub_stamp":"2024-03-24T00:13:01.267198+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T06:49:16.032741+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,4],"acting":[11,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.4","version":"86'1","reported_seq":25,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-24T00:15:07.729939+0000","last_change":"2024-03-24T00:13:01.314428+0000","last_active":"2024-03-24T00:15:07.729939+0000","last_peered":"2024-03-24T00:15:07.729939+0000","last_clean":"2024-03-24T00:15:07.729939+0000","last_became_active":"2024-03-24T00:13:01.313421+0000","last_became_peered":"2024-03-24T00:13:01.313421+0000","last_unstale":"2024-03-24T00:15:07.729939+0000","last_undegraded":"2024-03-24T00:15:07.729939+0000","last_fullsized":"2024-03-24T00:15:07.729939+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:13:00.222790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:13:00.222790+0000","last_clean_scrub_stamp":"2024-03-24T00:13:00.222790+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T08:10:25.796718+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,0],"acting":[9,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.5","version":"88'5","reported_seq":27,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-24T00:15:07.747827+0000","last_change":"2024-03-24T00:13:02.331660+0000","last_active":"2024-03-24T00:15:07.747827+0000","last_peered":"2024-03-24T00:15:07.747827+0000","last_clean":"2024-03-24T00:15:07.747827+0000","last_became_active":"2024-03-24T00:13:02.331288+0000","last_became_peered":"2024-03-24T00:13:02.331288+0000","last_unstale":"2024-03-24T00:15:07.747827+0000","last_undegraded":"2024-03-24T00:15:07.747827+0000","last_fullsized":"2024-03-24T00:15:07.747827+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:13:01.267198+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:13:01.267198+0000","last_clean_scrub_stamp":"2024-03-24T00:13:01.267198+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:36:55.576510+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3,6],"acting":[10,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.2","version":"86'5","reported_seq":184,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-24T00:15:41.684981+0000","last_change":"2024-03-24T00:13:01.310538+0000","last_active":"2024-03-24T00:15:41.684981+0000","last_peered":"2024-03-24T00:15:41.684981+0000","last_clean":"2024-03-24T00:15:41.684981+0000","last_became_active":"2024-03-24T00:13:01.310124+0000","last_became_peered":"2024-03-24T00:13:01.310124+0000","last_unstale":"2024-03-24T00:15:41.684981+0000","last_undegraded":"2024-03-24T00:15:41.684981+0000","last_fullsized":"2024-03-24T00:15:41.684981+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:13:00.222790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:13:00.222790+0000","last_clean_scrub_stamp":"2024-03-24T00:13:00.222790+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T05:47:03.534824+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":588,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":155,"num_read_kb":1,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3","version":"88'7","reported_seq":29,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-24T00:15:07.747783+0000","last_change":"2024-03-24T00:13:02.324117+0000","last_active":"2024-03-24T00:15:07.747783+0000","last_peered":"2024-03-24T00:15:07.747783+0000","last_clean":"2024-03-24T00:15:07.747783+0000","last_became_active":"2024-03-24T00:13:02.323706+0000","last_became_peered":"2024-03-24T00:13:02.323706+0000","last_unstale":"2024-03-24T00:15:07.747783+0000","last_undegraded":"2024-03-24T00:15:07.747783+0000","last_fullsized":"2024-03-24T00:15:07.747783+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:13:01.267198+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:13:01.267198+0000","last_clean_scrub_stamp":"2024-03-24T00:13:01.267198+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T05:34:31.657324+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,4,0],"acting":[10,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.0","version":"88'8","reported_seq":30,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-24T00:15:07.747186+0000","last_change":"2024-03-24T00:13:02.306030+0000","last_active":"2024-03-24T00:15:07.747186+0000","last_peered":"2024-03-24T00:15:07.747186+0000","last_clean":"2024-03-24T00:15:07.747186+0000","last_became_active":"2024-03-24T00:13:02.305588+0000","last_became_peered":"2024-03-24T00:13:02.305588+0000","last_unstale":"2024-03-24T00:15:07.747186+0000","last_undegraded":"2024-03-24T00:15:07.747186+0000","last_fullsized":"2024-03-24T00:15:07.747186+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:13:01.267198+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:13:01.267198+0000","last_clean_scrub_stamp":"2024-03-24T00:13:01.267198+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:20:58.206244+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,4],"acting":[5,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"86'5","reported_seq":183,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-24T00:15:40.546451+0000","last_change":"2024-03-24T00:13:01.318123+0000","last_active":"2024-03-24T00:15:40.546451+0000","last_peered":"2024-03-24T00:15:40.546451+0000","last_clean":"2024-03-24T00:15:40.546451+0000","last_became_active":"2024-03-24T00:13:01.317683+0000","last_became_peered":"2024-03-24T00:13:01.317683+0000","last_unstale":"2024-03-24T00:15:40.546451+0000","last_undegraded":"2024-03-24T00:15:40.546451+0000","last_fullsized":"2024-03-24T00:15:40.546451+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:13:00.222790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:13:00.222790+0000","last_clean_scrub_stamp":"2024-03-24T00:13:00.222790+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T06:33:02.669668+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":154,"num_read_kb":154,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"88'8","reported_seq":30,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-24T00:15:07.750327+0000","last_change":"2024-03-24T00:13:02.328599+0000","last_active":"2024-03-24T00:15:07.750327+0000","last_peered":"2024-03-24T00:15:07.750327+0000","last_clean":"2024-03-24T00:15:07.750327+0000","last_became_active":"2024-03-24T00:13:02.328165+0000","last_became_peered":"2024-03-24T00:13:02.328165+0000","last_unstale":"2024-03-24T00:15:07.750327+0000","last_undegraded":"2024-03-24T00:15:07.750327+0000","last_fullsized":"2024-03-24T00:15:07.750327+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:13:01.267198+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:13:01.267198+0000","last_clean_scrub_stamp":"2024-03-24T00:13:01.267198+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T07:22:39.646578+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"86'6","reported_seq":33,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-24T00:15:07.748021+0000","last_change":"2024-03-24T00:13:01.301932+0000","last_active":"2024-03-24T00:15:07.748021+0000","last_peered":"2024-03-24T00:15:07.748021+0000","last_clean":"2024-03-24T00:15:07.748021+0000","last_became_active":"2024-03-24T00:13:01.301464+0000","last_became_peered":"2024-03-24T00:13:01.301464+0000","last_unstale":"2024-03-24T00:15:07.748021+0000","last_undegraded":"2024-03-24T00:15:07.748021+0000","last_fullsized":"2024-03-24T00:15:07.748021+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:13:00.222790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:13:00.222790+0000","last_clean_scrub_stamp":"2024-03-24T00:13:00.222790+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T08:58:25.512421+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":4,"num_read_kb":4,"num_write":8,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,9],"acting":[4,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.0","version":"21'137","reported_seq":349,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-24T00:15:07.729560+0000","last_change":"2024-03-24T00:11:43.008767+0000","last_active":"2024-03-24T00:15:07.729560+0000","last_peered":"2024-03-24T00:15:07.729560+0000","last_clean":"2024-03-24T00:15:07.729560+0000","last_became_active":"2024-03-24T00:11:43.008364+0000","last_became_peered":"2024-03-24T00:11:43.008364+0000","last_unstale":"2024-03-24T00:15:07.729560+0000","last_undegraded":"2024-03-24T00:15:07.729560+0000","last_fullsized":"2024-03-24T00:15:07.729560+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:07:50.191828+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:07:50.191828+0000","last_clean_scrub_stamp":"2024-03-24T00:07:50.191828+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T01:55:26.549348+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"88'5","reported_seq":27,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-24T00:15:07.750385+0000","last_change":"2024-03-24T00:13:02.325667+0000","last_active":"2024-03-24T00:15:07.750385+0000","last_peered":"2024-03-24T00:15:07.750385+0000","last_clean":"2024-03-24T00:15:07.750385+0000","last_became_active":"2024-03-24T00:13:02.322476+0000","last_became_peered":"2024-03-24T00:13:02.322476+0000","last_unstale":"2024-03-24T00:15:07.750385+0000","last_undegraded":"2024-03-24T00:15:07.750385+0000","last_fullsized":"2024-03-24T00:15:07.750385+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:13:01.267198+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:13:01.267198+0000","last_clean_scrub_stamp":"2024-03-24T00:13:01.267198+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T08:07:19.936708+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"86'4","reported_seq":28,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-24T00:15:07.729687+0000","last_change":"2024-03-24T00:13:01.298082+0000","last_active":"2024-03-24T00:15:07.729687+0000","last_peered":"2024-03-24T00:15:07.729687+0000","last_clean":"2024-03-24T00:15:07.729687+0000","last_became_active":"2024-03-24T00:13:01.297686+0000","last_became_peered":"2024-03-24T00:13:01.297686+0000","last_unstale":"2024-03-24T00:15:07.729687+0000","last_undegraded":"2024-03-24T00:15:07.729687+0000","last_fullsized":"2024-03-24T00:15:07.729687+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:13:00.222790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:13:00.222790+0000","last_clean_scrub_stamp":"2024-03-24T00:13:00.222790+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T09:25:07.204076+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.5","version":"86'4","reported_seq":29,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-24T00:15:07.729914+0000","last_change":"2024-03-24T00:13:01.321250+0000","last_active":"2024-03-24T00:15:07.729914+0000","last_peered":"2024-03-24T00:15:07.729914+0000","last_clean":"2024-03-24T00:15:07.729914+0000","last_became_active":"2024-03-24T00:13:01.320929+0000","last_became_peered":"2024-03-24T00:13:01.320929+0000","last_unstale":"2024-03-24T00:15:07.729914+0000","last_undegraded":"2024-03-24T00:15:07.729914+0000","last_fullsized":"2024-03-24T00:15:07.729914+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:13:00.222790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:13:00.222790+0000","last_clean_scrub_stamp":"2024-03-24T00:13:00.222790+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T05:22:05.490303+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,9],"acting":[8,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.4","version":"88'9","reported_seq":31,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-24T00:15:07.751893+0000","last_change":"2024-03-24T00:13:02.323413+0000","last_active":"2024-03-24T00:15:07.751893+0000","last_peered":"2024-03-24T00:15:07.751893+0000","last_clean":"2024-03-24T00:15:07.751893+0000","last_became_active":"2024-03-24T00:13:02.322947+0000","last_became_peered":"2024-03-24T00:13:02.322947+0000","last_unstale":"2024-03-24T00:15:07.751893+0000","last_undegraded":"2024-03-24T00:15:07.751893+0000","last_fullsized":"2024-03-24T00:15:07.751893+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:13:01.267198+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:13:01.267198+0000","last_clean_scrub_stamp":"2024-03-24T00:13:01.267198+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T00:16:14.490270+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,9],"acting":[6,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.6","version":"90'14","reported_seq":38,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-24T00:15:37.357286+0000","last_change":"2024-03-24T00:13:01.320380+0000","last_active":"2024-03-24T00:15:37.357286+0000","last_peered":"2024-03-24T00:15:37.357286+0000","last_clean":"2024-03-24T00:15:37.357286+0000","last_became_active":"2024-03-24T00:13:01.319192+0000","last_became_peered":"2024-03-24T00:13:01.319192+0000","last_unstale":"2024-03-24T00:15:37.357286+0000","last_undegraded":"2024-03-24T00:15:37.357286+0000","last_fullsized":"2024-03-24T00:15:37.357286+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:13:00.222790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:13:00.222790+0000","last_clean_scrub_stamp":"2024-03-24T00:13:00.222790+0000","objects_scrubbed":0,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T02:09:27.085298+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":24,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,3],"acting":[9,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.7","version":"88'5","reported_seq":27,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-24T00:15:07.753013+0000","last_change":"2024-03-24T00:13:02.328067+0000","last_active":"2024-03-24T00:15:07.753013+0000","last_peered":"2024-03-24T00:15:07.753013+0000","last_clean":"2024-03-24T00:15:07.753013+0000","last_became_active":"2024-03-24T00:13:02.327468+0000","last_became_peered":"2024-03-24T00:13:02.327468+0000","last_unstale":"2024-03-24T00:15:07.753013+0000","last_undegraded":"2024-03-24T00:15:07.753013+0000","last_fullsized":"2024-03-24T00:15:07.753013+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:13:01.267198+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:13:01.267198+0000","last_clean_scrub_stamp":"2024-03-24T00:13:01.267198+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T01:21:58.596941+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,7],"acting":[3,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"86'7","reported_seq":32,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-24T00:15:07.748240+0000","last_change":"2024-03-24T00:13:01.309189+0000","last_active":"2024-03-24T00:15:07.748240+0000","last_peered":"2024-03-24T00:15:07.748240+0000","last_clean":"2024-03-24T00:15:07.748240+0000","last_became_active":"2024-03-24T00:13:01.308871+0000","last_became_peered":"2024-03-24T00:13:01.308871+0000","last_unstale":"2024-03-24T00:15:07.748240+0000","last_undegraded":"2024-03-24T00:15:07.748240+0000","last_fullsized":"2024-03-24T00:15:07.748240+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:13:00.222790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:13:00.222790+0000","last_clean_scrub_stamp":"2024-03-24T00:13:00.222790+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T01:00:32.610036+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,6],"acting":[4,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"88'7","reported_seq":29,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-24T00:15:07.750291+0000","last_change":"2024-03-24T00:13:02.332238+0000","last_active":"2024-03-24T00:15:07.750291+0000","last_peered":"2024-03-24T00:15:07.750291+0000","last_clean":"2024-03-24T00:15:07.750291+0000","last_became_active":"2024-03-24T00:13:02.331909+0000","last_became_peered":"2024-03-24T00:13:02.331909+0000","last_unstale":"2024-03-24T00:15:07.750291+0000","last_undegraded":"2024-03-24T00:15:07.750291+0000","last_fullsized":"2024-03-24T00:15:07.750291+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:13:01.267198+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:13:01.267198+0000","last_clean_scrub_stamp":"2024-03-24T00:13:01.267198+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T04:41:14.762251+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"86'2","reported_seq":26,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-24T00:15:07.726733+0000","last_change":"2024-03-24T00:13:01.306883+0000","last_active":"2024-03-24T00:15:07.726733+0000","last_peered":"2024-03-24T00:15:07.726733+0000","last_clean":"2024-03-24T00:15:07.726733+0000","last_became_active":"2024-03-24T00:13:01.306629+0000","last_became_peered":"2024-03-24T00:13:01.306629+0000","last_unstale":"2024-03-24T00:15:07.726733+0000","last_undegraded":"2024-03-24T00:15:07.726733+0000","last_fullsized":"2024-03-24T00:15:07.726733+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:13:00.222790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:13:00.222790+0000","last_clean_scrub_stamp":"2024-03-24T00:13:00.222790+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T08:11:52.398783+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"88'2","reported_seq":24,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-24T00:15:07.729642+0000","last_change":"2024-03-24T00:13:02.308421+0000","last_active":"2024-03-24T00:15:07.729642+0000","last_peered":"2024-03-24T00:15:07.729642+0000","last_clean":"2024-03-24T00:15:07.729642+0000","last_became_active":"2024-03-24T00:13:02.308081+0000","last_became_peered":"2024-03-24T00:13:02.308081+0000","last_unstale":"2024-03-24T00:15:07.729642+0000","last_undegraded":"2024-03-24T00:15:07.729642+0000","last_fullsized":"2024-03-24T00:15:07.729642+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:13:01.267198+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:13:01.267198+0000","last_clean_scrub_stamp":"2024-03-24T00:13:01.267198+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T06:50:42.356293+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,10],"acting":[8,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.9","version":"86'7","reported_seq":31,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-24T00:15:07.750083+0000","last_change":"2024-03-24T00:13:01.324439+0000","last_active":"2024-03-24T00:15:07.750083+0000","last_peered":"2024-03-24T00:15:07.750083+0000","last_clean":"2024-03-24T00:15:07.750083+0000","last_became_active":"2024-03-24T00:13:01.324037+0000","last_became_peered":"2024-03-24T00:13:01.324037+0000","last_unstale":"2024-03-24T00:15:07.750083+0000","last_undegraded":"2024-03-24T00:15:07.750083+0000","last_fullsized":"2024-03-24T00:15:07.750083+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:13:00.222790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:13:00.222790+0000","last_clean_scrub_stamp":"2024-03-24T00:13:00.222790+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T01:37:56.767931+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,10],"acting":[3,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.8","version":"88'9","reported_seq":31,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-24T00:15:07.727799+0000","last_change":"2024-03-24T00:13:02.313740+0000","last_active":"2024-03-24T00:15:07.727799+0000","last_peered":"2024-03-24T00:15:07.727799+0000","last_clean":"2024-03-24T00:15:07.727799+0000","last_became_active":"2024-03-24T00:13:02.313401+0000","last_became_peered":"2024-03-24T00:13:02.313401+0000","last_unstale":"2024-03-24T00:15:07.727799+0000","last_undegraded":"2024-03-24T00:15:07.727799+0000","last_fullsized":"2024-03-24T00:15:07.727799+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:13:01.267198+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:13:01.267198+0000","last_clean_scrub_stamp":"2024-03-24T00:13:01.267198+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T04:16:52.569135+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.a","version":"86'3","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-24T00:15:07.727832+0000","last_change":"2024-03-24T00:14:07.514050+0000","last_active":"2024-03-24T00:15:07.727832+0000","last_peered":"2024-03-24T00:15:07.727832+0000","last_clean":"2024-03-24T00:15:07.727832+0000","last_became_active":"2024-03-24T00:14:07.509219+0000","last_became_peered":"2024-03-24T00:14:07.509219+0000","last_unstale":"2024-03-24T00:15:07.727832+0000","last_undegraded":"2024-03-24T00:15:07.727832+0000","last_fullsized":"2024-03-24T00:15:07.727832+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:13:00.222790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:13:00.222790+0000","last_clean_scrub_stamp":"2024-03-24T00:13:00.222790+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T08:05:33.888846+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,3,8],"acting":[11,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.b","version":"88'8","reported_seq":30,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-24T00:15:07.752804+0000","last_change":"2024-03-24T00:13:02.337413+0000","last_active":"2024-03-24T00:15:07.752804+0000","last_peered":"2024-03-24T00:15:07.752804+0000","last_clean":"2024-03-24T00:15:07.752804+0000","last_became_active":"2024-03-24T00:13:02.336910+0000","last_became_peered":"2024-03-24T00:13:02.336910+0000","last_unstale":"2024-03-24T00:15:07.752804+0000","last_undegraded":"2024-03-24T00:15:07.752804+0000","last_fullsized":"2024-03-24T00:15:07.752804+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:13:01.267198+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:13:01.267198+0000","last_clean_scrub_stamp":"2024-03-24T00:13:01.267198+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T10:51:06.549979+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,10],"acting":[3,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.b","version":"86'4","reported_seq":29,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-24T00:15:07.729799+0000","last_change":"2024-03-24T00:13:01.300201+0000","last_active":"2024-03-24T00:15:07.729799+0000","last_peered":"2024-03-24T00:15:07.729799+0000","last_clean":"2024-03-24T00:15:07.729799+0000","last_became_active":"2024-03-24T00:13:01.299750+0000","last_became_peered":"2024-03-24T00:13:01.299750+0000","last_unstale":"2024-03-24T00:15:07.729799+0000","last_undegraded":"2024-03-24T00:15:07.729799+0000","last_fullsized":"2024-03-24T00:15:07.729799+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:13:00.222790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:13:00.222790+0000","last_clean_scrub_stamp":"2024-03-24T00:13:00.222790+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T01:37:32.768926+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.a","version":"88'5","reported_seq":27,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-24T00:15:07.729769+0000","last_change":"2024-03-24T00:13:02.331732+0000","last_active":"2024-03-24T00:15:07.729769+0000","last_peered":"2024-03-24T00:15:07.729769+0000","last_clean":"2024-03-24T00:15:07.729769+0000","last_became_active":"2024-03-24T00:13:02.331365+0000","last_became_peered":"2024-03-24T00:13:02.331365+0000","last_unstale":"2024-03-24T00:15:07.729769+0000","last_undegraded":"2024-03-24T00:15:07.729769+0000","last_fullsized":"2024-03-24T00:15:07.729769+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:13:01.267198+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:13:01.267198+0000","last_clean_scrub_stamp":"2024-03-24T00:13:01.267198+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:27:53.598158+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.c","version":"90'8","reported_seq":187,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-24T00:15:39.488276+0000","last_change":"2024-03-24T00:13:01.308725+0000","last_active":"2024-03-24T00:15:39.488276+0000","last_peered":"2024-03-24T00:15:39.488276+0000","last_clean":"2024-03-24T00:15:39.488276+0000","last_became_active":"2024-03-24T00:13:01.308324+0000","last_became_peered":"2024-03-24T00:13:01.308324+0000","last_unstale":"2024-03-24T00:15:39.488276+0000","last_undegraded":"2024-03-24T00:15:39.488276+0000","last_fullsized":"2024-03-24T00:15:39.488276+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:13:00.222790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:13:00.222790+0000","last_clean_scrub_stamp":"2024-03-24T00:13:00.222790+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T07:43:03.691683+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2093,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":155,"num_read_kb":4,"num_write":8,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,11,0],"acting":[2,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"88'7","reported_seq":29,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-24T00:15:07.730048+0000","last_change":"2024-03-24T00:13:02.305749+0000","last_active":"2024-03-24T00:15:07.730048+0000","last_peered":"2024-03-24T00:15:07.730048+0000","last_clean":"2024-03-24T00:15:07.730048+0000","last_became_active":"2024-03-24T00:13:02.305413+0000","last_became_peered":"2024-03-24T00:13:02.305413+0000","last_unstale":"2024-03-24T00:15:07.730048+0000","last_undegraded":"2024-03-24T00:15:07.730048+0000","last_fullsized":"2024-03-24T00:15:07.730048+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:13:01.267198+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:13:01.267198+0000","last_clean_scrub_stamp":"2024-03-24T00:13:01.267198+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T00:54:56.437725+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,11],"acting":[7,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.d","version":"86'3","reported_seq":28,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-24T00:15:07.747344+0000","last_change":"2024-03-24T00:13:01.304883+0000","last_active":"2024-03-24T00:15:07.747344+0000","last_peered":"2024-03-24T00:15:07.747344+0000","last_clean":"2024-03-24T00:15:07.747344+0000","last_became_active":"2024-03-24T00:13:01.304459+0000","last_became_peered":"2024-03-24T00:13:01.304459+0000","last_unstale":"2024-03-24T00:15:07.747344+0000","last_undegraded":"2024-03-24T00:15:07.747344+0000","last_fullsized":"2024-03-24T00:15:07.747344+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:13:00.222790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:13:00.222790+0000","last_clean_scrub_stamp":"2024-03-24T00:13:00.222790+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T06:14:22.473571+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10,3],"acting":[5,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.c","version":"88'7","reported_seq":29,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-24T00:15:07.729746+0000","last_change":"2024-03-24T00:13:02.331923+0000","last_active":"2024-03-24T00:15:07.729746+0000","last_peered":"2024-03-24T00:15:07.729746+0000","last_clean":"2024-03-24T00:15:07.729746+0000","last_became_active":"2024-03-24T00:13:02.331504+0000","last_became_peered":"2024-03-24T00:13:02.331504+0000","last_unstale":"2024-03-24T00:15:07.729746+0000","last_undegraded":"2024-03-24T00:15:07.729746+0000","last_fullsized":"2024-03-24T00:15:07.729746+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:13:01.267198+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:13:01.267198+0000","last_clean_scrub_stamp":"2024-03-24T00:13:01.267198+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T03:27:17.160248+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,5],"acting":[7,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"86'5","reported_seq":29,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-24T00:15:07.751611+0000","last_change":"2024-03-24T00:13:01.320223+0000","last_active":"2024-03-24T00:15:07.751611+0000","last_peered":"2024-03-24T00:15:07.751611+0000","last_clean":"2024-03-24T00:15:07.751611+0000","last_became_active":"2024-03-24T00:13:01.319342+0000","last_became_peered":"2024-03-24T00:13:01.319342+0000","last_unstale":"2024-03-24T00:15:07.751611+0000","last_undegraded":"2024-03-24T00:15:07.751611+0000","last_fullsized":"2024-03-24T00:15:07.751611+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:13:00.222790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:13:00.222790+0000","last_clean_scrub_stamp":"2024-03-24T00:13:00.222790+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T02:24:08.018521+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.f","version":"88'9","reported_seq":31,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-24T00:15:07.729949+0000","last_change":"2024-03-24T00:13:02.307719+0000","last_active":"2024-03-24T00:15:07.729949+0000","last_peered":"2024-03-24T00:15:07.729949+0000","last_clean":"2024-03-24T00:15:07.729949+0000","last_became_active":"2024-03-24T00:13:02.307399+0000","last_became_peered":"2024-03-24T00:13:02.307399+0000","last_unstale":"2024-03-24T00:15:07.729949+0000","last_undegraded":"2024-03-24T00:15:07.729949+0000","last_fullsized":"2024-03-24T00:15:07.729949+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:13:01.267198+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:13:01.267198+0000","last_clean_scrub_stamp":"2024-03-24T00:13:01.267198+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T09:38:13.077843+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,8],"acting":[7,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.20","version":"88'9","reported_seq":31,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-24T00:15:07.727941+0000","last_change":"2024-03-24T00:13:02.307469+0000","last_active":"2024-03-24T00:15:07.727941+0000","last_peered":"2024-03-24T00:15:07.727941+0000","last_clean":"2024-03-24T00:15:07.727941+0000","last_became_active":"2024-03-24T00:13:02.306873+0000","last_became_peered":"2024-03-24T00:13:02.306873+0000","last_unstale":"2024-03-24T00:15:07.727941+0000","last_undegraded":"2024-03-24T00:15:07.727941+0000","last_fullsized":"2024-03-24T00:15:07.727941+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:13:01.267198+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:13:01.267198+0000","last_clean_scrub_stamp":"2024-03-24T00:13:01.267198+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:19:51.163015+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,5],"acting":[11,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.21","version":"88'12","reported_seq":34,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-24T00:15:07.726362+0000","last_change":"2024-03-24T00:13:02.307472+0000","last_active":"2024-03-24T00:15:07.726362+0000","last_peered":"2024-03-24T00:15:07.726362+0000","last_clean":"2024-03-24T00:15:07.726362+0000","last_became_active":"2024-03-24T00:13:02.307099+0000","last_became_peered":"2024-03-24T00:13:02.307099+0000","last_unstale":"2024-03-24T00:15:07.726362+0000","last_undegraded":"2024-03-24T00:15:07.726362+0000","last_fullsized":"2024-03-24T00:15:07.726362+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:13:01.267198+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:13:01.267198+0000","last_clean_scrub_stamp":"2024-03-24T00:13:01.267198+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T08:49:06.461235+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,4],"acting":[2,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"88'8","reported_seq":30,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-24T00:15:07.726483+0000","last_change":"2024-03-24T00:13:02.331133+0000","last_active":"2024-03-24T00:15:07.726483+0000","last_peered":"2024-03-24T00:15:07.726483+0000","last_clean":"2024-03-24T00:15:07.726483+0000","last_became_active":"2024-03-24T00:13:02.330780+0000","last_became_peered":"2024-03-24T00:13:02.330780+0000","last_unstale":"2024-03-24T00:15:07.726483+0000","last_undegraded":"2024-03-24T00:15:07.726483+0000","last_fullsized":"2024-03-24T00:15:07.726483+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:13:01.267198+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:13:01.267198+0000","last_clean_scrub_stamp":"2024-03-24T00:13:01.267198+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T10:46:07.933131+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.23","version":"88'7","reported_seq":29,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-24T00:15:07.749059+0000","last_change":"2024-03-24T00:13:02.335406+0000","last_active":"2024-03-24T00:15:07.749059+0000","last_peered":"2024-03-24T00:15:07.749059+0000","last_clean":"2024-03-24T00:15:07.749059+0000","last_became_active":"2024-03-24T00:13:02.334994+0000","last_became_peered":"2024-03-24T00:13:02.334994+0000","last_unstale":"2024-03-24T00:15:07.749059+0000","last_undegraded":"2024-03-24T00:15:07.749059+0000","last_fullsized":"2024-03-24T00:15:07.749059+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:13:01.267198+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:13:01.267198+0000","last_clean_scrub_stamp":"2024-03-24T00:13:01.267198+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T03:24:59.547395+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,4],"acting":[0,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"88'4","reported_seq":26,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-24T00:15:07.752325+0000","last_change":"2024-03-24T00:13:02.332420+0000","last_active":"2024-03-24T00:15:07.752325+0000","last_peered":"2024-03-24T00:15:07.752325+0000","last_clean":"2024-03-24T00:15:07.752325+0000","last_became_active":"2024-03-24T00:13:02.331496+0000","last_became_peered":"2024-03-24T00:13:02.331496+0000","last_unstale":"2024-03-24T00:15:07.752325+0000","last_undegraded":"2024-03-24T00:15:07.752325+0000","last_fullsized":"2024-03-24T00:15:07.752325+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:13:01.267198+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:13:01.267198+0000","last_clean_scrub_stamp":"2024-03-24T00:13:01.267198+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T03:15:06.007947+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"88'6","reported_seq":28,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-24T00:15:07.728441+0000","last_change":"2024-03-24T00:13:02.309077+0000","last_active":"2024-03-24T00:15:07.728441+0000","last_peered":"2024-03-24T00:15:07.728441+0000","last_clean":"2024-03-24T00:15:07.728441+0000","last_became_active":"2024-03-24T00:13:02.308752+0000","last_became_peered":"2024-03-24T00:13:02.308752+0000","last_unstale":"2024-03-24T00:15:07.728441+0000","last_undegraded":"2024-03-24T00:15:07.728441+0000","last_fullsized":"2024-03-24T00:15:07.728441+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:13:01.267198+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:13:01.267198+0000","last_clean_scrub_stamp":"2024-03-24T00:13:01.267198+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T06:49:19.657988+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,5],"acting":[11,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.26","version":"88'15","reported_seq":37,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-24T00:15:07.726661+0000","last_change":"2024-03-24T00:13:02.304962+0000","last_active":"2024-03-24T00:15:07.726661+0000","last_peered":"2024-03-24T00:15:07.726661+0000","last_clean":"2024-03-24T00:15:07.726661+0000","last_became_active":"2024-03-24T00:13:02.304580+0000","last_became_peered":"2024-03-24T00:13:02.304580+0000","last_unstale":"2024-03-24T00:15:07.726661+0000","last_undegraded":"2024-03-24T00:15:07.726661+0000","last_fullsized":"2024-03-24T00:15:07.726661+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:13:01.267198+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:13:01.267198+0000","last_clean_scrub_stamp":"2024-03-24T00:13:01.267198+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:45:26.779169+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8,1],"acting":[2,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.27","version":"88'2","reported_seq":24,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-24T00:15:07.752778+0000","last_change":"2024-03-24T00:13:02.321908+0000","last_active":"2024-03-24T00:15:07.752778+0000","last_peered":"2024-03-24T00:15:07.752778+0000","last_clean":"2024-03-24T00:15:07.752778+0000","last_became_active":"2024-03-24T00:13:02.321495+0000","last_became_peered":"2024-03-24T00:13:02.321495+0000","last_unstale":"2024-03-24T00:15:07.752778+0000","last_undegraded":"2024-03-24T00:15:07.752778+0000","last_fullsized":"2024-03-24T00:15:07.752778+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:13:01.267198+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:13:01.267198+0000","last_clean_scrub_stamp":"2024-03-24T00:13:01.267198+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T08:43:29.261314+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"88'6","reported_seq":28,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-24T00:15:07.747128+0000","last_change":"2024-03-24T00:13:02.331966+0000","last_active":"2024-03-24T00:15:07.747128+0000","last_peered":"2024-03-24T00:15:07.747128+0000","last_clean":"2024-03-24T00:15:07.747128+0000","last_became_active":"2024-03-24T00:13:02.331633+0000","last_became_peered":"2024-03-24T00:13:02.331633+0000","last_unstale":"2024-03-24T00:15:07.747128+0000","last_undegraded":"2024-03-24T00:15:07.747128+0000","last_fullsized":"2024-03-24T00:15:07.747128+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:13:01.267198+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:13:01.267198+0000","last_clean_scrub_stamp":"2024-03-24T00:13:01.267198+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T05:04:01.574351+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,3],"acting":[5,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"88'8","reported_seq":30,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-24T00:15:07.750202+0000","last_change":"2024-03-24T00:13:02.330344+0000","last_active":"2024-03-24T00:15:07.750202+0000","last_peered":"2024-03-24T00:15:07.750202+0000","last_clean":"2024-03-24T00:15:07.750202+0000","last_became_active":"2024-03-24T00:13:02.328218+0000","last_became_peered":"2024-03-24T00:13:02.328218+0000","last_unstale":"2024-03-24T00:15:07.750202+0000","last_undegraded":"2024-03-24T00:15:07.750202+0000","last_fullsized":"2024-03-24T00:15:07.750202+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:13:01.267198+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:13:01.267198+0000","last_clean_scrub_stamp":"2024-03-24T00:13:01.267198+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T08:54:36.392397+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,11],"acting":[3,2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"88'10","reported_seq":32,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-24T00:15:07.726927+0000","last_change":"2024-03-24T00:13:02.330956+0000","last_active":"2024-03-24T00:15:07.726927+0000","last_peered":"2024-03-24T00:15:07.726927+0000","last_clean":"2024-03-24T00:15:07.726927+0000","last_became_active":"2024-03-24T00:13:02.330642+0000","last_became_peered":"2024-03-24T00:13:02.330642+0000","last_unstale":"2024-03-24T00:15:07.726927+0000","last_undegraded":"2024-03-24T00:15:07.726927+0000","last_fullsized":"2024-03-24T00:15:07.726927+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:13:01.267198+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:13:01.267198+0000","last_clean_scrub_stamp":"2024-03-24T00:13:01.267198+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:41:11.326722+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2b","version":"88'11","reported_seq":33,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-24T00:15:07.751540+0000","last_change":"2024-03-24T00:13:02.333026+0000","last_active":"2024-03-24T00:15:07.751540+0000","last_peered":"2024-03-24T00:15:07.751540+0000","last_clean":"2024-03-24T00:15:07.751540+0000","last_became_active":"2024-03-24T00:13:02.331232+0000","last_became_peered":"2024-03-24T00:13:02.331232+0000","last_unstale":"2024-03-24T00:15:07.751540+0000","last_undegraded":"2024-03-24T00:15:07.751540+0000","last_fullsized":"2024-03-24T00:15:07.751540+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:13:01.267198+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:13:01.267198+0000","last_clean_scrub_stamp":"2024-03-24T00:13:01.267198+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T02:14:18.504012+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2c","version":"88'8","reported_seq":30,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-24T00:15:07.727828+0000","last_change":"2024-03-24T00:13:02.309023+0000","last_active":"2024-03-24T00:15:07.727828+0000","last_peered":"2024-03-24T00:15:07.727828+0000","last_clean":"2024-03-24T00:15:07.727828+0000","last_became_active":"2024-03-24T00:13:02.308236+0000","last_became_peered":"2024-03-24T00:13:02.308236+0000","last_unstale":"2024-03-24T00:15:07.727828+0000","last_undegraded":"2024-03-24T00:15:07.727828+0000","last_fullsized":"2024-03-24T00:15:07.727828+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:13:01.267198+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:13:01.267198+0000","last_clean_scrub_stamp":"2024-03-24T00:13:01.267198+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T03:30:37.427095+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,8],"acting":[11,4,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":234,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":702,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":502,"ondisk_log_size":502,"up":192,"acting":192,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":442368,"data_stored":4722944,"data_compressed":284592,"data_compressed_allocated":360448,"data_compressed_original":4669440,"omap_allocated":0,"internal_metadata":0},"log_size":137,"ondisk_log_size":137,"up":3,"acting":3,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":70524,"num_objects":98,"num_object_clones":0,"num_object_copies":294,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":98,"num_whiteouts":0,"num_read":1563,"num_read_kb":856,"num_write":254,"num_write_kb":149,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21,"num_bytes_recovered":62373,"num_keys_recovered":10,"num_objects_omap":66,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":219465,"data_compressed":13143,"data_compressed_allocated":20480,"data_compressed_original":73728,"omap_allocated":0,"internal_metadata":0},"log_size":221,"ondisk_log_size":221,"up":96,"acting":96,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":81,"seq":347892351021,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27980,"kb_used_data":960,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717172,"statfs":{"total":95995035648,"available":95966384128,"internally_reserved":0,"allocated":983040,"data_stored":478079,"data_compressed":11573,"data_compressed_allocated":16384,"data_compressed_original":32768,"omap_allocated":1596,"internal_metadata":27654596},"hb_peers":[0,1,2,3,4,5,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":73,"seq":313532612658,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28128,"kb_used_data":1112,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717024,"statfs":{"total":95995035648,"available":95966232576,"internally_reserved":0,"allocated":1138688,"data_stored":1130832,"data_compressed":49481,"data_compressed_allocated":65536,"data_compressed_original":630784,"omap_allocated":1599,"internal_metadata":27654593},"hb_peers":[0,1,2,3,4,5,6,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":68,"seq":292057776184,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32160,"kb_used_data":984,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712992,"statfs":{"total":95995035648,"available":95962103808,"internally_reserved":0,"allocated":1007616,"data_stored":477477,"data_compressed":11573,"data_compressed_allocated":16384,"data_compressed_original":32768,"omap_allocated":1600,"internal_metadata":31914432},"hb_peers":[0,1,2,3,4,5,6,7,8,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":60,"seq":257698037823,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36308,"kb_used_data":972,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708844,"statfs":{"total":95995035648,"available":95957856256,"internally_reserved":0,"allocated":995328,"data_stored":477183,"data_compressed":11573,"data_compressed_allocated":16384,"data_compressed_original":32768,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":52,"seq":223338299461,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28056,"kb_used_data":1032,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717096,"statfs":{"total":95995035648,"available":95966306304,"internally_reserved":0,"allocated":1056768,"data_stored":1070905,"data_compressed":47762,"data_compressed_allocated":61440,"data_compressed_original":622592,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":47,"seq":201863462987,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27992,"kb_used_data":968,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717160,"statfs":{"total":95995035648,"available":95966371840,"internally_reserved":0,"allocated":991232,"data_stored":477194,"data_compressed":11573,"data_compressed_allocated":16384,"data_compressed_original":32768,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574950,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28060,"kb_used_data":1044,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717092,"statfs":{"total":95995035648,"available":95966302208,"internally_reserved":0,"allocated":1069056,"data_stored":549319,"data_compressed":13292,"data_compressed_allocated":20480,"data_compressed_original":40960,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738476,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28012,"kb_used_data":988,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717140,"statfs":{"total":95995035648,"available":95966351360,"internally_reserved":0,"allocated":1011712,"data_stored":479456,"data_compressed":11573,"data_compressed_allocated":16384,"data_compressed_original":32768,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411425,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27996,"kb_used_data":980,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717156,"statfs":{"total":95995035648,"available":95966367744,"internally_reserved":0,"allocated":1003520,"data_stored":480173,"data_compressed":11573,"data_compressed_allocated":16384,"data_compressed_original":32768,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":26,"seq":111669149788,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32160,"kb_used_data":984,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712992,"statfs":{"total":95995035648,"available":95962103808,"internally_reserved":0,"allocated":1007616,"data_stored":477019,"data_compressed":11573,"data_compressed_allocated":16384,"data_compressed_original":32768,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":34,"seq":146028888149,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32172,"kb_used_data":996,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712980,"statfs":{"total":95995035648,"available":95962091520,"internally_reserved":0,"allocated":1019904,"data_stored":539311,"data_compressed":21278,"data_compressed_allocated":28672,"data_compressed_original":90112,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":39,"seq":167503724624,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28060,"kb_used_data":1044,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717092,"statfs":{"total":95995035648,"available":95966302208,"internally_reserved":0,"allocated":1069056,"data_stored":1070583,"data_compressed":47762,"data_compressed_allocated":61440,"data_compressed_original":622592,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34549,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34549,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34549,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":36189,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":36189,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":36189,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":36189,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":36189,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":3101,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":98304,"data_stored":72964,"data_compressed":1719,"data_compressed_allocated":4096,"data_compressed_original":8192,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":3818,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":430,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":62956,"data_compressed":9705,"data_compressed_allocated":12288,"data_compressed_original":57344,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":3743,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":722,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":4065,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":828,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":1122,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":114688,"data_stored":63992,"data_compressed":1719,"data_compressed_allocated":4096,"data_compressed_original":8192,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1724,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":234,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-24T00:15:44.540 INFO:tasks.fwd_scrub.ceph_manager:clean! 2024-03-24T00:15:44.541 DEBUG:teuthology.orchestra.run.smithi118:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2024-03-24T00:15:44.777 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:44.777+0000 7f8302e00640 1 -- 172.21.15.118:0/1458440851 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f82fc103270 msgr2=0x7f82fc109b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:15:44.777 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:44.777+0000 7f8302e00640 1 --2- 172.21.15.118:0/1458440851 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f82fc103270 0x7f82fc109b00 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7f82f4009ca0 tx=0x7f82f402c0f0 comp rx=0 tx=0).stop 2024-03-24T00:15:44.778 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:44.777+0000 7f8302e00640 1 -- 172.21.15.118:0/1458440851 shutdown_connections 2024-03-24T00:15:44.778 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:44.777+0000 7f8302e00640 1 --2- 172.21.15.118:0/1458440851 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f82fc103270 0x7f82fc109b00 unknown :-1 s=CLOSED pgs=192 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:15:44.778 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:44.777+0000 7f8302e00640 1 --2- 172.21.15.118:0/1458440851 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f82fc1028b0 0x7f82fc102d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:15:44.778 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:44.777+0000 7f8302e00640 1 --2- 172.21.15.118:0/1458440851 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f82fc1016c0 0x7f82fc101ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:15:44.778 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:44.777+0000 7f8302e00640 1 -- 172.21.15.118:0/1458440851 >> 172.21.15.118:0/1458440851 conn(0x7f82fc0fcc50 msgr2=0x7f82fc0ff070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:15:44.778 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:44.777+0000 7f8302e00640 1 -- 172.21.15.118:0/1458440851 shutdown_connections 2024-03-24T00:15:44.778 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:44.777+0000 7f8302e00640 1 -- 172.21.15.118:0/1458440851 wait complete. 2024-03-24T00:15:44.780 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:44.779+0000 7f8302e00640 1 Processor -- start 2024-03-24T00:15:44.780 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:44.779+0000 7f8302e00640 1 -- start start 2024-03-24T00:15:44.780 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:44.779+0000 7f8302e00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f82fc1016c0 0x7f82fc199b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:15:44.780 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:44.779+0000 7f8302e00640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f82fc1028b0 0x7f82fc19a0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:15:44.780 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:44.779+0000 7f8302e00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f82fc103270 0x7f82fc1a1130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:15:44.780 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:44.779+0000 7f8302e00640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f82fc10d340 con 0x7f82fc1028b0 2024-03-24T00:15:44.780 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:44.779+0000 7f8302e00640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f82fc10d1c0 con 0x7f82fc1016c0 2024-03-24T00:15:44.780 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:44.779+0000 7f8302e00640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f82fc10d4c0 con 0x7f82fc103270 2024-03-24T00:15:44.780 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:44.780+0000 7f82fbe00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f82fc1016c0 0x7f82fc199b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:15:44.780 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:44.780+0000 7f82fbe00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f82fc1016c0 0x7f82fc199b80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.118:36344/0 (socket says 172.21.15.118:36344) 2024-03-24T00:15:44.780 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:44.780+0000 7f82fbe00640 1 -- 172.21.15.118:0/1215905736 learned_addr learned my addr 172.21.15.118:0/1215905736 (peer_addr_for_me v2:172.21.15.118:0/0) 2024-03-24T00:15:44.780 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:44.780+0000 7f82f3400640 1 --2- 172.21.15.118:0/1215905736 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f82fc1028b0 0x7f82fc19a0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:15:44.781 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:44.780+0000 7f8300e00640 1 --2- 172.21.15.118:0/1215905736 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f82fc103270 0x7f82fc1a1130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:15:44.781 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:44.780+0000 7f82fbe00640 1 -- 172.21.15.118:0/1215905736 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f82fc103270 msgr2=0x7f82fc1a1130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:15:44.781 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:44.780+0000 7f82fbe00640 1 --2- 172.21.15.118:0/1215905736 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f82fc103270 0x7f82fc1a1130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:15:44.781 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:44.780+0000 7f82fbe00640 1 -- 172.21.15.118:0/1215905736 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f82fc1028b0 msgr2=0x7f82fc19a0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:15:44.781 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:44.780+0000 7f82fbe00640 1 --2- 172.21.15.118:0/1215905736 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f82fc1028b0 0x7f82fc19a0c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:15:44.781 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:44.780+0000 7f82fbe00640 1 -- 172.21.15.118:0/1215905736 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f82fc1a17a0 con 0x7f82fc1016c0 2024-03-24T00:15:44.781 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:44.780+0000 7f82fbe00640 1 --2- 172.21.15.118:0/1215905736 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f82fc1016c0 0x7f82fc199b80 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7f82ec00e9c0 tx=0x7f82ec00ee80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:15:44.782 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:44.782+0000 7f82f9600640 1 -- 172.21.15.118:0/1215905736 <== mon.2 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f82ec00cd80 con 0x7f82fc1016c0 2024-03-24T00:15:44.782 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:44.782+0000 7f82f9600640 1 -- 172.21.15.118:0/1215905736 <== mon.2 v2:172.21.15.189:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f82ec0035f0 con 0x7f82fc1016c0 2024-03-24T00:15:44.782 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:44.782+0000 7f82f9600640 1 -- 172.21.15.118:0/1215905736 <== mon.2 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f82ec010690 con 0x7f82fc1016c0 2024-03-24T00:15:44.782 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:44.782+0000 7f8302e00640 1 -- 172.21.15.118:0/1215905736 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f82fc1a1a90 con 0x7f82fc1016c0 2024-03-24T00:15:44.782 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:44.782+0000 7f8302e00640 1 -- 172.21.15.118:0/1215905736 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f82fc1a1f90 con 0x7f82fc1016c0 2024-03-24T00:15:44.782 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:44.783+0000 7f8302e00640 1 -- 172.21.15.118:0/1215905736 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f82fc1a2220 con 0x7f82fc1016c0 2024-03-24T00:15:44.786 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:44.786+0000 7f82f9600640 1 -- 172.21.15.118:0/1215905736 <== mon.2 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 19) v1 ==== 137342+0+0 (secure 0 0 0) 0x7f82ec0037a0 con 0x7f82fc1016c0 2024-03-24T00:15:44.787 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:44.787+0000 7f82f9600640 1 --2- 172.21.15.118:0/1215905736 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7f82e00a2910 0x7f82e00a4dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:15:44.787 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:44.788+0000 7f82f9600640 1 -- 172.21.15.118:0/1215905736 <== mon.2 v2:172.21.15.189:3300/0 5 ==== osd_map(90..90 src has 1..90) v4 ==== 8277+0+0 (secure 0 0 0) 0x7f82ec0cef50 con 0x7f82fc1016c0 2024-03-24T00:15:44.787 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:44.788+0000 7f82f3400640 1 --2- 172.21.15.118:0/1215905736 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7f82e00a2910 0x7f82e00a4dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:15:44.788 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:44.788+0000 7f82f3400640 1 --2- 172.21.15.118:0/1215905736 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7f82e00a2910 0x7f82e00a4dd0 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7f82fc19b090 tx=0x7f82e4005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:15:44.791 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:44.791+0000 7f82f9600640 1 -- 172.21.15.118:0/1215905736 <== mon.2 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f82ec09be60 con 0x7f82fc1016c0 2024-03-24T00:15:44.813 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:15:44 smithi118 ceph-mon[31535]: pgmap v399: 97 pgs: 97 active+clean; 646 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 511 B/s wr, 10 op/s 2024-03-24T00:15:44.989 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:44.990+0000 7f8302e00640 1 -- 172.21.15.118:0/1215905736 --> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f82fc1a23f0 con 0x7f82e00a2910 2024-03-24T00:15:45.011 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:45.012+0000 7f82f9600640 1 -- 172.21.15.118:0/1215905736 <== mgr.14152 v2:172.21.15.118:6800/2112912943 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+259216 (secure 0 0 0) 0x7f82fc1a23f0 con 0x7f82e00a2910 2024-03-24T00:15:45.015 INFO:teuthology.orchestra.run.smithi118.stdout: 2024-03-24T00:15:45.016 INFO:teuthology.orchestra.run.smithi118.stderr:dumped all 2024-03-24T00:15:45.018 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:45.018+0000 7f8302e00640 1 -- 172.21.15.118:0/1215905736 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7f82e00a2910 msgr2=0x7f82e00a4dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:15:45.018 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:45.018+0000 7f8302e00640 1 --2- 172.21.15.118:0/1215905736 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7f82e00a2910 0x7f82e00a4dd0 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7f82fc19b090 tx=0x7f82e4005f90 comp rx=0 tx=0).stop 2024-03-24T00:15:45.018 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:45.018+0000 7f8302e00640 1 -- 172.21.15.118:0/1215905736 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f82fc1016c0 msgr2=0x7f82fc199b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:15:45.018 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:45.018+0000 7f8302e00640 1 --2- 172.21.15.118:0/1215905736 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f82fc1016c0 0x7f82fc199b80 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7f82ec00e9c0 tx=0x7f82ec00ee80 comp rx=0 tx=0).stop 2024-03-24T00:15:45.018 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:45.018+0000 7f8302e00640 1 -- 172.21.15.118:0/1215905736 shutdown_connections 2024-03-24T00:15:45.018 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:45.018+0000 7f8302e00640 1 --2- 172.21.15.118:0/1215905736 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7f82e00a2910 0x7f82e00a4dd0 unknown :-1 s=CLOSED pgs=262 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:15:45.019 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:45.018+0000 7f8302e00640 1 --2- 172.21.15.118:0/1215905736 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f82fc103270 0x7f82fc1a1130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:15:45.019 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:45.018+0000 7f8302e00640 1 --2- 172.21.15.118:0/1215905736 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f82fc1028b0 0x7f82fc19a0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:15:45.019 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:45.018+0000 7f8302e00640 1 --2- 172.21.15.118:0/1215905736 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f82fc1016c0 0x7f82fc199b80 unknown :-1 s=CLOSED pgs=198 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:15:45.019 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:45.018+0000 7f8302e00640 1 -- 172.21.15.118:0/1215905736 >> 172.21.15.118:0/1215905736 conn(0x7f82fc0fcc50 msgr2=0x7f82fc0738d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:15:45.019 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:45.018+0000 7f8302e00640 1 -- 172.21.15.118:0/1215905736 shutdown_connections 2024-03-24T00:15:45.019 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:45.018+0000 7f8302e00640 1 -- 172.21.15.118:0/1215905736 wait complete. 2024-03-24T00:15:45.045 INFO:teuthology.orchestra.run.smithi118.stdout:{"pg_ready":true,"pg_map":{"version":399,"stamp":"2024-03-24T00:15:43.668833+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":661126,"num_objects":102,"num_object_clones":0,"num_object_copies":306,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":102,"num_whiteouts":0,"num_read":1689,"num_read_kb":965,"num_write":437,"num_write_kb":3089,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":25,"num_bytes_recovered":1243109,"num_keys_recovered":10,"num_objects_omap":66,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":860,"ondisk_log_size":860,"up":291,"acting":291,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":291,"num_osds":12,"num_per_pool_osds":12,"num_per_pool_omap_osds":12,"kb":1124941824,"kb_used":357084,"kb_used_data":12064,"kb_used_omap":18,"kb_used_meta":344877,"kb_avail":1124584740,"statfs":{"total":1151940427776,"available":1151574773760,"internally_reserved":0,"allocated":12353536,"data_stored":7707531,"data_compressed":260586,"data_compressed_allocated":352256,"data_compressed_original":2236416,"omap_allocated":19142,"internal_metadata":353154362},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":1088,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":122,"num_read_kb":62,"num_write":8,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.004105"},"pg_stats":[{"pgid":"3.3f","version":"88'8","reported_seq":30,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-24T00:15:07.728049+0000","last_change":"2024-03-24T00:13:02.325815+0000","last_active":"2024-03-24T00:15:07.728049+0000","last_peered":"2024-03-24T00:15:07.728049+0000","last_clean":"2024-03-24T00:15:07.728049+0000","last_became_active":"2024-03-24T00:13:02.325450+0000","last_became_peered":"2024-03-24T00:13:02.325450+0000","last_unstale":"2024-03-24T00:15:07.728049+0000","last_undegraded":"2024-03-24T00:15:07.728049+0000","last_fullsized":"2024-03-24T00:15:07.728049+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:13:01.267198+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:13:01.267198+0000","last_clean_scrub_stamp":"2024-03-24T00:13:01.267198+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T02:18:21.994322+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.3e","version":"88'4","reported_seq":26,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-24T00:15:07.750213+0000","last_change":"2024-03-24T00:13:02.324015+0000","last_active":"2024-03-24T00:15:07.750213+0000","last_peered":"2024-03-24T00:15:07.750213+0000","last_clean":"2024-03-24T00:15:07.750213+0000","last_became_active":"2024-03-24T00:13:02.323668+0000","last_became_peered":"2024-03-24T00:13:02.323668+0000","last_unstale":"2024-03-24T00:15:07.750213+0000","last_undegraded":"2024-03-24T00:15:07.750213+0000","last_fullsized":"2024-03-24T00:15:07.750213+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:13:01.267198+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:13:01.267198+0000","last_clean_scrub_stamp":"2024-03-24T00:13:01.267198+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:36:19.442266+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,11,10],"acting":[0,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3d","version":"88'7","reported_seq":29,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-24T00:15:07.749167+0000","last_change":"2024-03-24T00:13:02.328911+0000","last_active":"2024-03-24T00:15:07.749167+0000","last_peered":"2024-03-24T00:15:07.749167+0000","last_clean":"2024-03-24T00:15:07.749167+0000","last_became_active":"2024-03-24T00:13:02.324333+0000","last_became_peered":"2024-03-24T00:13:02.324333+0000","last_unstale":"2024-03-24T00:15:07.749167+0000","last_undegraded":"2024-03-24T00:15:07.749167+0000","last_fullsized":"2024-03-24T00:15:07.749167+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:13:01.267198+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:13:01.267198+0000","last_clean_scrub_stamp":"2024-03-24T00:13:01.267198+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T02:49:02.318097+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,2,1],"acting":[0,2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.3c","version":"88'8","reported_seq":30,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-24T00:15:07.748082+0000","last_change":"2024-03-24T00:13:02.309057+0000","last_active":"2024-03-24T00:15:07.748082+0000","last_peered":"2024-03-24T00:15:07.748082+0000","last_clean":"2024-03-24T00:15:07.748082+0000","last_became_active":"2024-03-24T00:13:02.308403+0000","last_became_peered":"2024-03-24T00:13:02.308403+0000","last_unstale":"2024-03-24T00:15:07.748082+0000","last_undegraded":"2024-03-24T00:15:07.748082+0000","last_fullsized":"2024-03-24T00:15:07.748082+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:13:01.267198+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:13:01.267198+0000","last_clean_scrub_stamp":"2024-03-24T00:13:01.267198+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T07:54:44.310833+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,8,2],"acting":[4,8,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.3b","version":"88'4","reported_seq":26,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-24T00:15:07.729685+0000","last_change":"2024-03-24T00:13:02.330651+0000","last_active":"2024-03-24T00:15:07.729685+0000","last_peered":"2024-03-24T00:15:07.729685+0000","last_clean":"2024-03-24T00:15:07.729685+0000","last_became_active":"2024-03-24T00:13:02.330313+0000","last_became_peered":"2024-03-24T00:13:02.330313+0000","last_unstale":"2024-03-24T00:15:07.729685+0000","last_undegraded":"2024-03-24T00:15:07.729685+0000","last_fullsized":"2024-03-24T00:15:07.729685+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:13:01.267198+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:13:01.267198+0000","last_clean_scrub_stamp":"2024-03-24T00:13:01.267198+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T03:45:38.480929+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,3,6],"acting":[8,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.3a","version":"88'6","reported_seq":28,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-24T00:15:07.747612+0000","last_change":"2024-03-24T00:13:02.313115+0000","last_active":"2024-03-24T00:15:07.747612+0000","last_peered":"2024-03-24T00:15:07.747612+0000","last_clean":"2024-03-24T00:15:07.747612+0000","last_became_active":"2024-03-24T00:13:02.312387+0000","last_became_peered":"2024-03-24T00:13:02.312387+0000","last_unstale":"2024-03-24T00:15:07.747612+0000","last_undegraded":"2024-03-24T00:15:07.747612+0000","last_fullsized":"2024-03-24T00:15:07.747612+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:13:01.267198+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:13:01.267198+0000","last_clean_scrub_stamp":"2024-03-24T00:13:01.267198+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T09:16:02.574920+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,9,4],"acting":[10,9,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.39","version":"88'5","reported_seq":33,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-24T00:15:07.729862+0000","last_change":"2024-03-24T00:14:07.503364+0000","last_active":"2024-03-24T00:15:07.729862+0000","last_peered":"2024-03-24T00:15:07.729862+0000","last_clean":"2024-03-24T00:15:07.729862+0000","last_became_active":"2024-03-24T00:14:07.503103+0000","last_became_peered":"2024-03-24T00:14:07.503103+0000","last_unstale":"2024-03-24T00:15:07.729862+0000","last_undegraded":"2024-03-24T00:15:07.729862+0000","last_fullsized":"2024-03-24T00:15:07.729862+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:13:01.267198+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:13:01.267198+0000","last_clean_scrub_stamp":"2024-03-24T00:13:01.267198+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T03:22:04.225634+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,8],"acting":[7,9,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.38","version":"88'9","reported_seq":24,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-24T00:15:07.729082+0000","last_change":"2024-03-24T00:14:07.681011+0000","last_active":"2024-03-24T00:15:07.729082+0000","last_peered":"2024-03-24T00:15:07.729082+0000","last_clean":"2024-03-24T00:15:07.729082+0000","last_became_active":"2024-03-24T00:14:07.680726+0000","last_became_peered":"2024-03-24T00:14:07.680726+0000","last_unstale":"2024-03-24T00:15:07.729082+0000","last_undegraded":"2024-03-24T00:15:07.729082+0000","last_fullsized":"2024-03-24T00:15:07.729082+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:13:01.267198+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:13:01.267198+0000","last_clean_scrub_stamp":"2024-03-24T00:13:01.267198+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T01:05:03.379937+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,5,10],"acting":[9,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.37","version":"88'11","reported_seq":33,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-24T00:15:07.727279+0000","last_change":"2024-03-24T00:13:02.305625+0000","last_active":"2024-03-24T00:15:07.727279+0000","last_peered":"2024-03-24T00:15:07.727279+0000","last_clean":"2024-03-24T00:15:07.727279+0000","last_became_active":"2024-03-24T00:13:02.305221+0000","last_became_peered":"2024-03-24T00:13:02.305221+0000","last_unstale":"2024-03-24T00:15:07.727279+0000","last_undegraded":"2024-03-24T00:15:07.727279+0000","last_fullsized":"2024-03-24T00:15:07.727279+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:13:01.267198+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:13:01.267198+0000","last_clean_scrub_stamp":"2024-03-24T00:13:01.267198+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T04:37:12.467387+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5,11],"acting":[1,5,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.36","version":"88'16","reported_seq":38,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-24T00:15:07.747823+0000","last_change":"2024-03-24T00:13:02.307878+0000","last_active":"2024-03-24T00:15:07.747823+0000","last_peered":"2024-03-24T00:15:07.747823+0000","last_clean":"2024-03-24T00:15:07.747823+0000","last_became_active":"2024-03-24T00:13:02.307513+0000","last_became_peered":"2024-03-24T00:13:02.307513+0000","last_unstale":"2024-03-24T00:15:07.747823+0000","last_undegraded":"2024-03-24T00:15:07.747823+0000","last_fullsized":"2024-03-24T00:15:07.747823+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:13:01.267198+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:13:01.267198+0000","last_clean_scrub_stamp":"2024-03-24T00:13:01.267198+0000","objects_scrubbed":0,"log_size":16,"log_dups_size":0,"ondisk_log_size":16,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T07:06:08.162416+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,2,4],"acting":[10,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.35","version":"88'6","reported_seq":28,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-24T00:15:07.729986+0000","last_change":"2024-03-24T00:13:02.310883+0000","last_active":"2024-03-24T00:15:07.729986+0000","last_peered":"2024-03-24T00:15:07.729986+0000","last_clean":"2024-03-24T00:15:07.729986+0000","last_became_active":"2024-03-24T00:13:02.310559+0000","last_became_peered":"2024-03-24T00:13:02.310559+0000","last_unstale":"2024-03-24T00:15:07.729986+0000","last_undegraded":"2024-03-24T00:15:07.729986+0000","last_fullsized":"2024-03-24T00:15:07.729986+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:13:01.267198+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:13:01.267198+0000","last_clean_scrub_stamp":"2024-03-24T00:13:01.267198+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T04:04:25.801248+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,8,1],"acting":[7,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.34","version":"88'7","reported_seq":29,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-24T00:15:07.748067+0000","last_change":"2024-03-24T00:13:02.308747+0000","last_active":"2024-03-24T00:15:07.748067+0000","last_peered":"2024-03-24T00:15:07.748067+0000","last_clean":"2024-03-24T00:15:07.748067+0000","last_became_active":"2024-03-24T00:13:02.307791+0000","last_became_peered":"2024-03-24T00:13:02.307791+0000","last_unstale":"2024-03-24T00:15:07.748067+0000","last_undegraded":"2024-03-24T00:15:07.748067+0000","last_fullsized":"2024-03-24T00:15:07.748067+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:13:01.267198+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:13:01.267198+0000","last_clean_scrub_stamp":"2024-03-24T00:13:01.267198+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T03:28:08.780044+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,8],"acting":[4,10,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.33","version":"88'10","reported_seq":32,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-24T00:15:07.751605+0000","last_change":"2024-03-24T00:13:02.327061+0000","last_active":"2024-03-24T00:15:07.751605+0000","last_peered":"2024-03-24T00:15:07.751605+0000","last_clean":"2024-03-24T00:15:07.751605+0000","last_became_active":"2024-03-24T00:13:02.326696+0000","last_became_peered":"2024-03-24T00:13:02.326696+0000","last_unstale":"2024-03-24T00:15:07.751605+0000","last_undegraded":"2024-03-24T00:15:07.751605+0000","last_fullsized":"2024-03-24T00:15:07.751605+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:13:01.267198+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:13:01.267198+0000","last_clean_scrub_stamp":"2024-03-24T00:13:01.267198+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T08:55:17.325488+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,10,5],"acting":[6,10,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.32","version":"88'7","reported_seq":29,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-24T00:15:07.748893+0000","last_change":"2024-03-24T00:13:02.321326+0000","last_active":"2024-03-24T00:15:07.748893+0000","last_peered":"2024-03-24T00:15:07.748893+0000","last_clean":"2024-03-24T00:15:07.748893+0000","last_became_active":"2024-03-24T00:13:02.320964+0000","last_became_peered":"2024-03-24T00:13:02.320964+0000","last_unstale":"2024-03-24T00:15:07.748893+0000","last_undegraded":"2024-03-24T00:15:07.748893+0000","last_fullsized":"2024-03-24T00:15:07.748893+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:13:01.267198+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:13:01.267198+0000","last_clean_scrub_stamp":"2024-03-24T00:13:01.267198+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T08:05:15.365809+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,8,10],"acting":[0,8,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.31","version":"88'7","reported_seq":35,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-24T00:15:07.748624+0000","last_change":"2024-03-24T00:14:07.505774+0000","last_active":"2024-03-24T00:15:07.748624+0000","last_peered":"2024-03-24T00:15:07.748624+0000","last_clean":"2024-03-24T00:15:07.748624+0000","last_became_active":"2024-03-24T00:14:07.504800+0000","last_became_peered":"2024-03-24T00:14:07.504800+0000","last_unstale":"2024-03-24T00:15:07.748624+0000","last_undegraded":"2024-03-24T00:15:07.748624+0000","last_fullsized":"2024-03-24T00:15:07.748624+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:13:01.267198+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:13:01.267198+0000","last_clean_scrub_stamp":"2024-03-24T00:13:01.267198+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T06:50:30.686330+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,7,2],"acting":[0,7,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.30","version":"88'9","reported_seq":31,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-24T00:15:07.726732+0000","last_change":"2024-03-24T00:13:02.325460+0000","last_active":"2024-03-24T00:15:07.726732+0000","last_peered":"2024-03-24T00:15:07.726732+0000","last_clean":"2024-03-24T00:15:07.726732+0000","last_became_active":"2024-03-24T00:13:02.325112+0000","last_became_peered":"2024-03-24T00:13:02.325112+0000","last_unstale":"2024-03-24T00:15:07.726732+0000","last_undegraded":"2024-03-24T00:15:07.726732+0000","last_fullsized":"2024-03-24T00:15:07.726732+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:13:01.267198+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:13:01.267198+0000","last_clean_scrub_stamp":"2024-03-24T00:13:01.267198+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T03:15:16.362970+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,0],"acting":[1,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2f","version":"88'8","reported_seq":30,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-24T00:15:07.728294+0000","last_change":"2024-03-24T00:13:02.325918+0000","last_active":"2024-03-24T00:15:07.728294+0000","last_peered":"2024-03-24T00:15:07.728294+0000","last_clean":"2024-03-24T00:15:07.728294+0000","last_became_active":"2024-03-24T00:13:02.325557+0000","last_became_peered":"2024-03-24T00:13:02.325557+0000","last_unstale":"2024-03-24T00:15:07.728294+0000","last_undegraded":"2024-03-24T00:15:07.728294+0000","last_fullsized":"2024-03-24T00:15:07.728294+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:13:01.267198+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:13:01.267198+0000","last_clean_scrub_stamp":"2024-03-24T00:13:01.267198+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T02:55:43.759674+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,1,0],"acting":[11,1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.2e","version":"88'10","reported_seq":32,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-24T00:15:07.729972+0000","last_change":"2024-03-24T00:13:02.308106+0000","last_active":"2024-03-24T00:15:07.729972+0000","last_peered":"2024-03-24T00:15:07.729972+0000","last_clean":"2024-03-24T00:15:07.729972+0000","last_became_active":"2024-03-24T00:13:02.306972+0000","last_became_peered":"2024-03-24T00:13:02.306972+0000","last_unstale":"2024-03-24T00:15:07.729972+0000","last_undegraded":"2024-03-24T00:15:07.729972+0000","last_fullsized":"2024-03-24T00:15:07.729972+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:13:01.267198+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:13:01.267198+0000","last_clean_scrub_stamp":"2024-03-24T00:13:01.267198+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T09:38:54.677255+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,1,10],"acting":[9,1,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.2d","version":"88'7","reported_seq":29,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-24T00:15:07.726431+0000","last_change":"2024-03-24T00:13:02.331742+0000","last_active":"2024-03-24T00:15:07.726431+0000","last_peered":"2024-03-24T00:15:07.726431+0000","last_clean":"2024-03-24T00:15:07.726431+0000","last_became_active":"2024-03-24T00:13:02.331103+0000","last_became_peered":"2024-03-24T00:13:02.331103+0000","last_unstale":"2024-03-24T00:15:07.726431+0000","last_undegraded":"2024-03-24T00:15:07.726431+0000","last_fullsized":"2024-03-24T00:15:07.726431+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:13:01.267198+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:13:01.267198+0000","last_clean_scrub_stamp":"2024-03-24T00:13:01.267198+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:25:58.116306+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,6],"acting":[2,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.1f","version":"86'13","reported_seq":211,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-24T00:15:40.682427+0000","last_change":"2024-03-24T00:14:07.520827+0000","last_active":"2024-03-24T00:15:40.682427+0000","last_peered":"2024-03-24T00:15:40.682427+0000","last_clean":"2024-03-24T00:15:40.682427+0000","last_became_active":"2024-03-24T00:14:07.505406+0000","last_became_peered":"2024-03-24T00:14:07.505406+0000","last_unstale":"2024-03-24T00:15:40.682427+0000","last_undegraded":"2024-03-24T00:15:40.682427+0000","last_fullsized":"2024-03-24T00:15:40.682427+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:13:00.222790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:13:00.222790+0000","last_clean_scrub_stamp":"2024-03-24T00:13:00.222790+0000","objects_scrubbed":0,"log_size":13,"log_dups_size":0,"ondisk_log_size":13,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T01:05:01.235978+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":152,"num_read_kb":152,"num_write":14,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":672,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,7,1],"acting":[9,7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.1e","version":"88'5","reported_seq":27,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-24T00:15:07.751376+0000","last_change":"2024-03-24T00:13:02.322461+0000","last_active":"2024-03-24T00:15:07.751376+0000","last_peered":"2024-03-24T00:15:07.751376+0000","last_clean":"2024-03-24T00:15:07.751376+0000","last_became_active":"2024-03-24T00:13:02.321752+0000","last_became_peered":"2024-03-24T00:13:02.321752+0000","last_unstale":"2024-03-24T00:15:07.751376+0000","last_undegraded":"2024-03-24T00:15:07.751376+0000","last_fullsized":"2024-03-24T00:15:07.751376+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:13:01.267198+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:13:01.267198+0000","last_clean_scrub_stamp":"2024-03-24T00:13:01.267198+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T06:37:05.825919+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,5],"acting":[3,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.1e","version":"86'2","reported_seq":26,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-24T00:15:07.752241+0000","last_change":"2024-03-24T00:13:01.324972+0000","last_active":"2024-03-24T00:15:07.752241+0000","last_peered":"2024-03-24T00:15:07.752241+0000","last_clean":"2024-03-24T00:15:07.752241+0000","last_became_active":"2024-03-24T00:13:01.323342+0000","last_became_peered":"2024-03-24T00:13:01.323342+0000","last_unstale":"2024-03-24T00:15:07.752241+0000","last_undegraded":"2024-03-24T00:15:07.752241+0000","last_fullsized":"2024-03-24T00:15:07.752241+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:13:00.222790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:13:00.222790+0000","last_clean_scrub_stamp":"2024-03-24T00:13:00.222790+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T05:34:35.827634+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,5],"acting":[3,0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.1f","version":"88'6","reported_seq":28,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-24T00:15:07.749057+0000","last_change":"2024-03-24T00:13:02.325455+0000","last_active":"2024-03-24T00:15:07.749057+0000","last_peered":"2024-03-24T00:15:07.749057+0000","last_clean":"2024-03-24T00:15:07.749057+0000","last_became_active":"2024-03-24T00:13:02.325061+0000","last_became_peered":"2024-03-24T00:13:02.325061+0000","last_unstale":"2024-03-24T00:15:07.749057+0000","last_undegraded":"2024-03-24T00:15:07.749057+0000","last_fullsized":"2024-03-24T00:15:07.749057+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:13:01.267198+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:13:01.267198+0000","last_clean_scrub_stamp":"2024-03-24T00:13:01.267198+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T06:13:33.223361+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1,9],"acting":[0,1,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1d","version":"86'12","reported_seq":27,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-24T00:15:07.746267+0000","last_change":"2024-03-24T00:15:07.745753+0000","last_active":"2024-03-24T00:15:07.746267+0000","last_peered":"2024-03-24T00:15:07.746267+0000","last_clean":"2024-03-24T00:15:07.746267+0000","last_became_active":"2024-03-24T00:15:07.730583+0000","last_became_peered":"2024-03-24T00:15:07.730583+0000","last_unstale":"2024-03-24T00:15:07.746267+0000","last_undegraded":"2024-03-24T00:15:07.746267+0000","last_fullsized":"2024-03-24T00:15:07.746267+0000","mapping_epoch":89,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":90,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:13:00.222790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:13:00.222790+0000","last_clean_scrub_stamp":"2024-03-24T00:13:00.222790+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T07:33:32.085470+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":7,"num_object_clones":0,"num_object_copies":21,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":16,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7,"num_bytes_recovered":0,"num_keys_recovered":10,"num_objects_omap":7,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,7],"acting":[11,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.1c","version":"88'9","reported_seq":31,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-24T00:15:07.726909+0000","last_change":"2024-03-24T00:13:02.325130+0000","last_active":"2024-03-24T00:15:07.726909+0000","last_peered":"2024-03-24T00:15:07.726909+0000","last_clean":"2024-03-24T00:15:07.726909+0000","last_became_active":"2024-03-24T00:13:02.324740+0000","last_became_peered":"2024-03-24T00:13:02.324740+0000","last_unstale":"2024-03-24T00:15:07.726909+0000","last_undegraded":"2024-03-24T00:15:07.726909+0000","last_fullsized":"2024-03-24T00:15:07.726909+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:13:01.267198+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:13:01.267198+0000","last_clean_scrub_stamp":"2024-03-24T00:13:01.267198+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T01:23:35.194788+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,6],"acting":[1,10,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1c","version":"86'5","reported_seq":30,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-24T00:15:07.729993+0000","last_change":"2024-03-24T00:13:01.300572+0000","last_active":"2024-03-24T00:15:07.729993+0000","last_peered":"2024-03-24T00:15:07.729993+0000","last_clean":"2024-03-24T00:15:07.729993+0000","last_became_active":"2024-03-24T00:13:01.299801+0000","last_became_peered":"2024-03-24T00:13:01.299801+0000","last_unstale":"2024-03-24T00:15:07.729993+0000","last_undegraded":"2024-03-24T00:15:07.729993+0000","last_fullsized":"2024-03-24T00:15:07.729993+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:13:00.222790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:13:00.222790+0000","last_clean_scrub_stamp":"2024-03-24T00:13:00.222790+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T10:29:03.986969+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":7,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,1,5],"acting":[8,1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.1d","version":"88'15","reported_seq":37,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-24T00:15:07.747448+0000","last_change":"2024-03-24T00:13:02.304079+0000","last_active":"2024-03-24T00:15:07.747448+0000","last_peered":"2024-03-24T00:15:07.747448+0000","last_clean":"2024-03-24T00:15:07.747448+0000","last_became_active":"2024-03-24T00:13:02.303700+0000","last_became_peered":"2024-03-24T00:13:02.303700+0000","last_unstale":"2024-03-24T00:15:07.747448+0000","last_undegraded":"2024-03-24T00:15:07.747448+0000","last_fullsized":"2024-03-24T00:15:07.747448+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:13:01.267198+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:13:01.267198+0000","last_clean_scrub_stamp":"2024-03-24T00:13:01.267198+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T10:19:07.701927+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,1],"acting":[5,4,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1b","version":"90'8","reported_seq":193,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-24T00:15:40.437307+0000","last_change":"2024-03-24T00:13:01.300949+0000","last_active":"2024-03-24T00:15:40.437307+0000","last_peered":"2024-03-24T00:15:40.437307+0000","last_clean":"2024-03-24T00:15:40.437307+0000","last_became_active":"2024-03-24T00:13:01.300605+0000","last_became_peered":"2024-03-24T00:13:01.300605+0000","last_unstale":"2024-03-24T00:15:40.437307+0000","last_undegraded":"2024-03-24T00:15:40.437307+0000","last_fullsized":"2024-03-24T00:15:40.437307+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:13:00.222790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:13:00.222790+0000","last_clean_scrub_stamp":"2024-03-24T00:13:00.222790+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T02:47:58.283103+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":1545,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":161,"num_read_kb":5,"num_write":5,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,7],"acting":[5,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.1a","version":"88'7","reported_seq":29,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-24T00:15:07.729749+0000","last_change":"2024-03-24T00:13:02.312884+0000","last_active":"2024-03-24T00:15:07.729749+0000","last_peered":"2024-03-24T00:15:07.729749+0000","last_clean":"2024-03-24T00:15:07.729749+0000","last_became_active":"2024-03-24T00:13:02.312533+0000","last_became_peered":"2024-03-24T00:13:02.312533+0000","last_unstale":"2024-03-24T00:15:07.729749+0000","last_undegraded":"2024-03-24T00:15:07.729749+0000","last_fullsized":"2024-03-24T00:15:07.729749+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:13:01.267198+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:13:01.267198+0000","last_clean_scrub_stamp":"2024-03-24T00:13:01.267198+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T10:54:48.398645+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,9,5],"acting":[8,9,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.1a","version":"86'7","reported_seq":31,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-24T00:15:07.747575+0000","last_change":"2024-03-24T00:13:01.306292+0000","last_active":"2024-03-24T00:15:07.747575+0000","last_peered":"2024-03-24T00:15:07.747575+0000","last_clean":"2024-03-24T00:15:07.747575+0000","last_became_active":"2024-03-24T00:13:01.305898+0000","last_became_peered":"2024-03-24T00:13:01.305898+0000","last_unstale":"2024-03-24T00:15:07.747575+0000","last_undegraded":"2024-03-24T00:15:07.747575+0000","last_fullsized":"2024-03-24T00:15:07.747575+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:13:00.222790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:13:00.222790+0000","last_clean_scrub_stamp":"2024-03-24T00:13:00.222790+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T07:41:17.003588+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,8,6],"acting":[10,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.1b","version":"88'8","reported_seq":30,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-24T00:15:07.747429+0000","last_change":"2024-03-24T00:13:02.323763+0000","last_active":"2024-03-24T00:15:07.747429+0000","last_peered":"2024-03-24T00:15:07.747429+0000","last_clean":"2024-03-24T00:15:07.747429+0000","last_became_active":"2024-03-24T00:13:02.323367+0000","last_became_peered":"2024-03-24T00:13:02.323367+0000","last_unstale":"2024-03-24T00:15:07.747429+0000","last_undegraded":"2024-03-24T00:15:07.747429+0000","last_fullsized":"2024-03-24T00:15:07.747429+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:13:01.267198+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:13:01.267198+0000","last_clean_scrub_stamp":"2024-03-24T00:13:01.267198+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T08:53:21.671317+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,4,3],"acting":[5,4,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.19","version":"86'3","reported_seq":27,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-24T00:15:07.751196+0000","last_change":"2024-03-24T00:13:01.323111+0000","last_active":"2024-03-24T00:15:07.751196+0000","last_peered":"2024-03-24T00:15:07.751196+0000","last_clean":"2024-03-24T00:15:07.751196+0000","last_became_active":"2024-03-24T00:13:01.322772+0000","last_became_peered":"2024-03-24T00:13:01.322772+0000","last_unstale":"2024-03-24T00:15:07.751196+0000","last_undegraded":"2024-03-24T00:15:07.751196+0000","last_fullsized":"2024-03-24T00:15:07.751196+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:13:00.222790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:13:00.222790+0000","last_clean_scrub_stamp":"2024-03-24T00:13:00.222790+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T04:19:53.097933+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":5,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,9],"acting":[6,4,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.18","version":"88'8","reported_seq":30,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-24T00:15:07.750057+0000","last_change":"2024-03-24T00:13:02.330469+0000","last_active":"2024-03-24T00:15:07.750057+0000","last_peered":"2024-03-24T00:15:07.750057+0000","last_clean":"2024-03-24T00:15:07.750057+0000","last_became_active":"2024-03-24T00:13:02.327019+0000","last_became_peered":"2024-03-24T00:13:02.327019+0000","last_unstale":"2024-03-24T00:15:07.750057+0000","last_undegraded":"2024-03-24T00:15:07.750057+0000","last_fullsized":"2024-03-24T00:15:07.750057+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:13:01.267198+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:13:01.267198+0000","last_clean_scrub_stamp":"2024-03-24T00:13:01.267198+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T07:30:37.155535+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,6],"acting":[3,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.18","version":"86'3","reported_seq":28,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-24T00:15:07.747073+0000","last_change":"2024-03-24T00:13:01.300089+0000","last_active":"2024-03-24T00:15:07.747073+0000","last_peered":"2024-03-24T00:15:07.747073+0000","last_clean":"2024-03-24T00:15:07.747073+0000","last_became_active":"2024-03-24T00:13:01.299747+0000","last_became_peered":"2024-03-24T00:13:01.299747+0000","last_unstale":"2024-03-24T00:15:07.747073+0000","last_undegraded":"2024-03-24T00:15:07.747073+0000","last_fullsized":"2024-03-24T00:15:07.747073+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:13:00.222790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:13:00.222790+0000","last_clean_scrub_stamp":"2024-03-24T00:13:00.222790+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T00:30:30.142785+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":582,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1,7],"acting":[5,1,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.19","version":"88'11","reported_seq":33,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-24T00:15:07.729674+0000","last_change":"2024-03-24T00:13:02.332490+0000","last_active":"2024-03-24T00:15:07.729674+0000","last_peered":"2024-03-24T00:15:07.729674+0000","last_clean":"2024-03-24T00:15:07.729674+0000","last_became_active":"2024-03-24T00:13:02.331182+0000","last_became_peered":"2024-03-24T00:13:02.331182+0000","last_unstale":"2024-03-24T00:15:07.729674+0000","last_undegraded":"2024-03-24T00:15:07.729674+0000","last_fullsized":"2024-03-24T00:15:07.729674+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:13:01.267198+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:13:01.267198+0000","last_clean_scrub_stamp":"2024-03-24T00:13:01.267198+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T06:22:16.084879+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,3,10],"acting":[9,3,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.17","version":"88'6","reported_seq":333,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-24T00:15:38.433780+0000","last_change":"2024-03-24T00:13:01.299818+0000","last_active":"2024-03-24T00:15:38.433780+0000","last_peered":"2024-03-24T00:15:38.433780+0000","last_clean":"2024-03-24T00:15:38.433780+0000","last_became_active":"2024-03-24T00:13:01.299478+0000","last_became_peered":"2024-03-24T00:13:01.299478+0000","last_unstale":"2024-03-24T00:15:38.433780+0000","last_undegraded":"2024-03-24T00:15:38.433780+0000","last_fullsized":"2024-03-24T00:15:38.433780+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:13:00.222790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:13:00.222790+0000","last_clean_scrub_stamp":"2024-03-24T00:13:00.222790+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:51:47.088677+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":180,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":303,"num_read_kb":303,"num_write":7,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,5,9],"acting":[10,5,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.16","version":"88'10","reported_seq":32,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-24T00:15:07.727190+0000","last_change":"2024-03-24T00:13:02.304971+0000","last_active":"2024-03-24T00:15:07.727190+0000","last_peered":"2024-03-24T00:15:07.727190+0000","last_clean":"2024-03-24T00:15:07.727190+0000","last_became_active":"2024-03-24T00:13:02.304577+0000","last_became_peered":"2024-03-24T00:13:02.304577+0000","last_unstale":"2024-03-24T00:15:07.727190+0000","last_undegraded":"2024-03-24T00:15:07.727190+0000","last_fullsized":"2024-03-24T00:15:07.727190+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:13:01.267198+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:13:01.267198+0000","last_clean_scrub_stamp":"2024-03-24T00:13:01.267198+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T05:08:03.444554+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,10,9],"acting":[1,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.16","version":"86'4","reported_seq":28,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-24T00:15:07.729906+0000","last_change":"2024-03-24T00:13:01.303075+0000","last_active":"2024-03-24T00:15:07.729906+0000","last_peered":"2024-03-24T00:15:07.729906+0000","last_clean":"2024-03-24T00:15:07.729906+0000","last_became_active":"2024-03-24T00:13:01.301517+0000","last_became_peered":"2024-03-24T00:13:01.301517+0000","last_unstale":"2024-03-24T00:15:07.729906+0000","last_undegraded":"2024-03-24T00:15:07.729906+0000","last_fullsized":"2024-03-24T00:15:07.729906+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:13:00.222790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:13:00.222790+0000","last_clean_scrub_stamp":"2024-03-24T00:13:00.222790+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:04:10.552510+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,2],"acting":[7,1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.17","version":"88'9","reported_seq":31,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-24T00:15:07.749184+0000","last_change":"2024-03-24T00:13:02.320108+0000","last_active":"2024-03-24T00:15:07.749184+0000","last_peered":"2024-03-24T00:15:07.749184+0000","last_clean":"2024-03-24T00:15:07.749184+0000","last_became_active":"2024-03-24T00:13:02.319155+0000","last_became_peered":"2024-03-24T00:13:02.319155+0000","last_unstale":"2024-03-24T00:15:07.749184+0000","last_undegraded":"2024-03-24T00:15:07.749184+0000","last_fullsized":"2024-03-24T00:15:07.749184+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:13:01.267198+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:13:01.267198+0000","last_clean_scrub_stamp":"2024-03-24T00:13:01.267198+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T10:15:54.770304+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4,5],"acting":[0,4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.15","version":"86'5","reported_seq":30,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-24T00:15:07.747498+0000","last_change":"2024-03-24T00:13:01.323919+0000","last_active":"2024-03-24T00:15:07.747498+0000","last_peered":"2024-03-24T00:15:07.747498+0000","last_clean":"2024-03-24T00:15:07.747498+0000","last_became_active":"2024-03-24T00:13:01.323640+0000","last_became_peered":"2024-03-24T00:13:01.323640+0000","last_unstale":"2024-03-24T00:15:07.747498+0000","last_undegraded":"2024-03-24T00:15:07.747498+0000","last_fullsized":"2024-03-24T00:15:07.747498+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:13:00.222790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:13:00.222790+0000","last_clean_scrub_stamp":"2024-03-24T00:13:00.222790+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T06:16:49.018642+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":672,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":7,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,0,10],"acting":[5,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.14","version":"88'11","reported_seq":39,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-24T00:15:07.727032+0000","last_change":"2024-03-24T00:14:07.635409+0000","last_active":"2024-03-24T00:15:07.727032+0000","last_peered":"2024-03-24T00:15:07.727032+0000","last_clean":"2024-03-24T00:15:07.727032+0000","last_became_active":"2024-03-24T00:14:07.635093+0000","last_became_peered":"2024-03-24T00:14:07.635093+0000","last_unstale":"2024-03-24T00:15:07.727032+0000","last_undegraded":"2024-03-24T00:15:07.727032+0000","last_fullsized":"2024-03-24T00:15:07.727032+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:13:01.267198+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:13:01.267198+0000","last_clean_scrub_stamp":"2024-03-24T00:13:01.267198+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T10:31:10.205874+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,8,6],"acting":[1,8,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.14","version":"90'37","reported_seq":397,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-24T00:15:38.895751+0000","last_change":"2024-03-24T00:14:07.807393+0000","last_active":"2024-03-24T00:15:38.895751+0000","last_peered":"2024-03-24T00:15:38.895751+0000","last_clean":"2024-03-24T00:15:38.895751+0000","last_became_active":"2024-03-24T00:14:07.801140+0000","last_became_peered":"2024-03-24T00:14:07.801140+0000","last_unstale":"2024-03-24T00:15:38.895751+0000","last_undegraded":"2024-03-24T00:15:38.895751+0000","last_fullsized":"2024-03-24T00:15:38.895751+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:13:00.222790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:13:00.222790+0000","last_clean_scrub_stamp":"2024-03-24T00:13:00.222790+0000","objects_scrubbed":0,"log_size":37,"log_dups_size":0,"ondisk_log_size":37,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T10:35:25.627828+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":62256,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":312,"num_read_kb":216,"num_write":39,"num_write_kb":78,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":6,"num_bytes_recovered":61701,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,1,4],"acting":[10,1,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.15","version":"88'6","reported_seq":28,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-24T00:15:07.729922+0000","last_change":"2024-03-24T00:13:02.307653+0000","last_active":"2024-03-24T00:15:07.729922+0000","last_peered":"2024-03-24T00:15:07.729922+0000","last_clean":"2024-03-24T00:15:07.729922+0000","last_became_active":"2024-03-24T00:13:02.307287+0000","last_became_peered":"2024-03-24T00:13:02.307287+0000","last_unstale":"2024-03-24T00:15:07.729922+0000","last_undegraded":"2024-03-24T00:15:07.729922+0000","last_fullsized":"2024-03-24T00:15:07.729922+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:13:01.267198+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:13:01.267198+0000","last_clean_scrub_stamp":"2024-03-24T00:13:01.267198+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T05:59:27.701614+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,7],"acting":[8,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.13","version":"86'9","reported_seq":35,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-24T00:15:07.726557+0000","last_change":"2024-03-24T00:13:01.316854+0000","last_active":"2024-03-24T00:15:07.726557+0000","last_peered":"2024-03-24T00:15:07.726557+0000","last_clean":"2024-03-24T00:15:07.726557+0000","last_became_active":"2024-03-24T00:13:01.316507+0000","last_became_peered":"2024-03-24T00:13:01.316507+0000","last_unstale":"2024-03-24T00:15:07.726557+0000","last_undegraded":"2024-03-24T00:15:07.726557+0000","last_fullsized":"2024-03-24T00:15:07.726557+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:13:00.222790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:13:00.222790+0000","last_clean_scrub_stamp":"2024-03-24T00:13:00.222790+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T07:58:41.888003+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68,"num_objects":6,"num_object_clones":0,"num_object_copies":18,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":13,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,10,3],"acting":[2,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.12","version":"88'9","reported_seq":31,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-24T00:15:07.751081+0000","last_change":"2024-03-24T00:13:02.338811+0000","last_active":"2024-03-24T00:15:07.751081+0000","last_peered":"2024-03-24T00:15:07.751081+0000","last_clean":"2024-03-24T00:15:07.751081+0000","last_became_active":"2024-03-24T00:13:02.338438+0000","last_became_peered":"2024-03-24T00:13:02.338438+0000","last_unstale":"2024-03-24T00:15:07.751081+0000","last_undegraded":"2024-03-24T00:15:07.751081+0000","last_fullsized":"2024-03-24T00:15:07.751081+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:13:01.267198+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:13:01.267198+0000","last_clean_scrub_stamp":"2024-03-24T00:13:01.267198+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T03:55:16.531224+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,3],"acting":[6,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.12","version":"86'5","reported_seq":30,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-24T00:15:07.747178+0000","last_change":"2024-03-24T00:13:01.313914+0000","last_active":"2024-03-24T00:15:07.747178+0000","last_peered":"2024-03-24T00:15:07.747178+0000","last_clean":"2024-03-24T00:15:07.747178+0000","last_became_active":"2024-03-24T00:13:01.313535+0000","last_became_peered":"2024-03-24T00:13:01.313535+0000","last_unstale":"2024-03-24T00:15:07.747178+0000","last_undegraded":"2024-03-24T00:15:07.747178+0000","last_fullsized":"2024-03-24T00:15:07.747178+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:13:00.222790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:13:00.222790+0000","last_clean_scrub_stamp":"2024-03-24T00:13:00.222790+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T03:23:53.011550+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":5,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3,7],"acting":[5,3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.13","version":"88'15","reported_seq":43,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-24T00:15:07.729806+0000","last_change":"2024-03-24T00:14:07.632978+0000","last_active":"2024-03-24T00:15:07.729806+0000","last_peered":"2024-03-24T00:15:07.729806+0000","last_clean":"2024-03-24T00:15:07.729806+0000","last_became_active":"2024-03-24T00:14:07.632719+0000","last_became_peered":"2024-03-24T00:14:07.632719+0000","last_unstale":"2024-03-24T00:15:07.729806+0000","last_undegraded":"2024-03-24T00:15:07.729806+0000","last_fullsized":"2024-03-24T00:15:07.729806+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:13:01.267198+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:13:01.267198+0000","last_clean_scrub_stamp":"2024-03-24T00:13:01.267198+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T09:59:38.993218+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,9,6],"acting":[7,9,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.11","version":"86'5","reported_seq":181,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-24T00:15:37.543087+0000","last_change":"2024-03-24T00:13:01.297887+0000","last_active":"2024-03-24T00:15:37.543087+0000","last_peered":"2024-03-24T00:15:37.543087+0000","last_clean":"2024-03-24T00:15:37.543087+0000","last_became_active":"2024-03-24T00:13:01.297560+0000","last_became_peered":"2024-03-24T00:13:01.297560+0000","last_unstale":"2024-03-24T00:15:37.543087+0000","last_undegraded":"2024-03-24T00:15:37.543087+0000","last_fullsized":"2024-03-24T00:15:37.543087+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:13:00.222790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:13:00.222790+0000","last_clean_scrub_stamp":"2024-03-24T00:13:00.222790+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T01:06:28.927274+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":588,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":153,"num_read_kb":3,"num_write":5,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,7,10],"acting":[4,7,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.10","version":"88'6","reported_seq":28,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-24T00:15:07.729530+0000","last_change":"2024-03-24T00:13:02.306811+0000","last_active":"2024-03-24T00:15:07.729530+0000","last_peered":"2024-03-24T00:15:07.729530+0000","last_clean":"2024-03-24T00:15:07.729530+0000","last_became_active":"2024-03-24T00:13:02.306499+0000","last_became_peered":"2024-03-24T00:13:02.306499+0000","last_unstale":"2024-03-24T00:15:07.729530+0000","last_undegraded":"2024-03-24T00:15:07.729530+0000","last_fullsized":"2024-03-24T00:15:07.729530+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:13:01.267198+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:13:01.267198+0000","last_clean_scrub_stamp":"2024-03-24T00:13:01.267198+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:49:32.683005+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.10","version":"86'17","reported_seq":42,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-24T00:15:07.729583+0000","last_change":"2024-03-24T00:13:01.307058+0000","last_active":"2024-03-24T00:15:07.729583+0000","last_peered":"2024-03-24T00:15:07.729583+0000","last_clean":"2024-03-24T00:15:07.729583+0000","last_became_active":"2024-03-24T00:13:01.306699+0000","last_became_peered":"2024-03-24T00:13:01.306699+0000","last_unstale":"2024-03-24T00:15:07.729583+0000","last_undegraded":"2024-03-24T00:15:07.729583+0000","last_fullsized":"2024-03-24T00:15:07.729583+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:13:00.222790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:13:00.222790+0000","last_clean_scrub_stamp":"2024-03-24T00:13:00.222790+0000","objects_scrubbed":0,"log_size":17,"log_dups_size":0,"ondisk_log_size":17,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T10:32:37.334514+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":8,"num_object_clones":0,"num_object_copies":24,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":8,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":20,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":6,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,2],"acting":[8,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.11","version":"88'6","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-24T00:15:07.729239+0000","last_change":"2024-03-24T00:14:07.503357+0000","last_active":"2024-03-24T00:15:07.729239+0000","last_peered":"2024-03-24T00:15:07.729239+0000","last_clean":"2024-03-24T00:15:07.729239+0000","last_became_active":"2024-03-24T00:14:07.502511+0000","last_became_peered":"2024-03-24T00:14:07.502511+0000","last_unstale":"2024-03-24T00:15:07.729239+0000","last_undegraded":"2024-03-24T00:15:07.729239+0000","last_fullsized":"2024-03-24T00:15:07.729239+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:13:01.267198+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:13:01.267198+0000","last_clean_scrub_stamp":"2024-03-24T00:13:01.267198+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T04:18:09.351579+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,1],"acting":[9,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"2.f","version":"86'2","reported_seq":27,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-24T00:15:07.730062+0000","last_change":"2024-03-24T00:13:01.295874+0000","last_active":"2024-03-24T00:15:07.730062+0000","last_peered":"2024-03-24T00:15:07.730062+0000","last_clean":"2024-03-24T00:15:07.730062+0000","last_became_active":"2024-03-24T00:13:01.295521+0000","last_became_peered":"2024-03-24T00:13:01.295521+0000","last_unstale":"2024-03-24T00:15:07.730062+0000","last_undegraded":"2024-03-24T00:15:07.730062+0000","last_fullsized":"2024-03-24T00:15:07.730062+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:13:00.222790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:13:00.222790+0000","last_clean_scrub_stamp":"2024-03-24T00:13:00.222790+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T06:47:15.749491+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":2,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,5,4],"acting":[8,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.e","version":"88'5","reported_seq":27,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-24T00:15:07.728056+0000","last_change":"2024-03-24T00:13:02.306617+0000","last_active":"2024-03-24T00:15:07.728056+0000","last_peered":"2024-03-24T00:15:07.728056+0000","last_clean":"2024-03-24T00:15:07.728056+0000","last_became_active":"2024-03-24T00:13:02.306224+0000","last_became_peered":"2024-03-24T00:13:02.306224+0000","last_unstale":"2024-03-24T00:15:07.728056+0000","last_undegraded":"2024-03-24T00:15:07.728056+0000","last_fullsized":"2024-03-24T00:15:07.728056+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:13:01.267198+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:13:01.267198+0000","last_clean_scrub_stamp":"2024-03-24T00:13:01.267198+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T06:49:16.032741+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,5,4],"acting":[11,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.4","version":"86'1","reported_seq":25,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-24T00:15:07.729939+0000","last_change":"2024-03-24T00:13:01.314428+0000","last_active":"2024-03-24T00:15:07.729939+0000","last_peered":"2024-03-24T00:15:07.729939+0000","last_clean":"2024-03-24T00:15:07.729939+0000","last_became_active":"2024-03-24T00:13:01.313421+0000","last_became_peered":"2024-03-24T00:13:01.313421+0000","last_unstale":"2024-03-24T00:15:07.729939+0000","last_undegraded":"2024-03-24T00:15:07.729939+0000","last_fullsized":"2024-03-24T00:15:07.729939+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:13:00.222790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:13:00.222790+0000","last_clean_scrub_stamp":"2024-03-24T00:13:00.222790+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T08:10:25.796718+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,2,0],"acting":[9,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.5","version":"88'5","reported_seq":27,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-24T00:15:07.747827+0000","last_change":"2024-03-24T00:13:02.331660+0000","last_active":"2024-03-24T00:15:07.747827+0000","last_peered":"2024-03-24T00:15:07.747827+0000","last_clean":"2024-03-24T00:15:07.747827+0000","last_became_active":"2024-03-24T00:13:02.331288+0000","last_became_peered":"2024-03-24T00:13:02.331288+0000","last_unstale":"2024-03-24T00:15:07.747827+0000","last_undegraded":"2024-03-24T00:15:07.747827+0000","last_fullsized":"2024-03-24T00:15:07.747827+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:13:01.267198+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:13:01.267198+0000","last_clean_scrub_stamp":"2024-03-24T00:13:01.267198+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:36:55.576510+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,3,6],"acting":[10,3,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"2.2","version":"86'5","reported_seq":184,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-24T00:15:41.684981+0000","last_change":"2024-03-24T00:13:01.310538+0000","last_active":"2024-03-24T00:15:41.684981+0000","last_peered":"2024-03-24T00:15:41.684981+0000","last_clean":"2024-03-24T00:15:41.684981+0000","last_became_active":"2024-03-24T00:13:01.310124+0000","last_became_peered":"2024-03-24T00:13:01.310124+0000","last_unstale":"2024-03-24T00:15:41.684981+0000","last_undegraded":"2024-03-24T00:15:41.684981+0000","last_fullsized":"2024-03-24T00:15:41.684981+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:13:00.222790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:13:00.222790+0000","last_clean_scrub_stamp":"2024-03-24T00:13:00.222790+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T05:47:03.534824+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":588,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":155,"num_read_kb":1,"num_write":4,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,6,1],"acting":[7,6,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.3","version":"88'7","reported_seq":29,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-24T00:15:07.747783+0000","last_change":"2024-03-24T00:13:02.324117+0000","last_active":"2024-03-24T00:15:07.747783+0000","last_peered":"2024-03-24T00:15:07.747783+0000","last_clean":"2024-03-24T00:15:07.747783+0000","last_became_active":"2024-03-24T00:13:02.323706+0000","last_became_peered":"2024-03-24T00:13:02.323706+0000","last_unstale":"2024-03-24T00:15:07.747783+0000","last_undegraded":"2024-03-24T00:15:07.747783+0000","last_fullsized":"2024-03-24T00:15:07.747783+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:13:01.267198+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:13:01.267198+0000","last_clean_scrub_stamp":"2024-03-24T00:13:01.267198+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T05:34:31.657324+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[10,4,0],"acting":[10,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":10,"acting_primary":10,"purged_snaps":[]},{"pgid":"3.0","version":"88'8","reported_seq":30,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-24T00:15:07.747186+0000","last_change":"2024-03-24T00:13:02.306030+0000","last_active":"2024-03-24T00:15:07.747186+0000","last_peered":"2024-03-24T00:15:07.747186+0000","last_clean":"2024-03-24T00:15:07.747186+0000","last_became_active":"2024-03-24T00:13:02.305588+0000","last_became_peered":"2024-03-24T00:13:02.305588+0000","last_unstale":"2024-03-24T00:15:07.747186+0000","last_undegraded":"2024-03-24T00:15:07.747186+0000","last_fullsized":"2024-03-24T00:15:07.747186+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:13:01.267198+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:13:01.267198+0000","last_clean_scrub_stamp":"2024-03-24T00:13:01.267198+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:20:58.206244+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2,4],"acting":[5,2,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.1","version":"86'5","reported_seq":183,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-24T00:15:40.546451+0000","last_change":"2024-03-24T00:13:01.318123+0000","last_active":"2024-03-24T00:15:40.546451+0000","last_peered":"2024-03-24T00:15:40.546451+0000","last_clean":"2024-03-24T00:15:40.546451+0000","last_became_active":"2024-03-24T00:13:01.317683+0000","last_became_peered":"2024-03-24T00:13:01.317683+0000","last_unstale":"2024-03-24T00:15:40.546451+0000","last_undegraded":"2024-03-24T00:15:40.546451+0000","last_fullsized":"2024-03-24T00:15:40.546451+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:13:00.222790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:13:00.222790+0000","last_clean_scrub_stamp":"2024-03-24T00:13:00.222790+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T06:33:02.669668+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":154,"num_read_kb":154,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,0],"acting":[6,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2","version":"88'8","reported_seq":30,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-24T00:15:07.750327+0000","last_change":"2024-03-24T00:13:02.328599+0000","last_active":"2024-03-24T00:15:07.750327+0000","last_peered":"2024-03-24T00:15:07.750327+0000","last_clean":"2024-03-24T00:15:07.750327+0000","last_became_active":"2024-03-24T00:13:02.328165+0000","last_became_peered":"2024-03-24T00:13:02.328165+0000","last_unstale":"2024-03-24T00:15:07.750327+0000","last_undegraded":"2024-03-24T00:15:07.750327+0000","last_fullsized":"2024-03-24T00:15:07.750327+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:13:01.267198+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:13:01.267198+0000","last_clean_scrub_stamp":"2024-03-24T00:13:01.267198+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T07:22:39.646578+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5,1],"acting":[3,5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.3","version":"86'6","reported_seq":33,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-24T00:15:07.748021+0000","last_change":"2024-03-24T00:13:01.301932+0000","last_active":"2024-03-24T00:15:07.748021+0000","last_peered":"2024-03-24T00:15:07.748021+0000","last_clean":"2024-03-24T00:15:07.748021+0000","last_became_active":"2024-03-24T00:13:01.301464+0000","last_became_peered":"2024-03-24T00:13:01.301464+0000","last_unstale":"2024-03-24T00:15:07.748021+0000","last_undegraded":"2024-03-24T00:15:07.748021+0000","last_fullsized":"2024-03-24T00:15:07.748021+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:13:00.222790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:13:00.222790+0000","last_clean_scrub_stamp":"2024-03-24T00:13:00.222790+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T08:58:25.512421+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":68,"num_objects":5,"num_object_clones":0,"num_object_copies":15,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":4,"num_read_kb":4,"num_write":8,"num_write_kb":8,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,10,9],"acting":[4,10,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"1.0","version":"21'137","reported_seq":349,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-24T00:15:07.729560+0000","last_change":"2024-03-24T00:11:43.008767+0000","last_active":"2024-03-24T00:15:07.729560+0000","last_peered":"2024-03-24T00:15:07.729560+0000","last_clean":"2024-03-24T00:15:07.729560+0000","last_became_active":"2024-03-24T00:11:43.008364+0000","last_became_peered":"2024-03-24T00:11:43.008364+0000","last_unstale":"2024-03-24T00:15:07.729560+0000","last_undegraded":"2024-03-24T00:15:07.729560+0000","last_fullsized":"2024-03-24T00:15:07.729560+0000","mapping_epoch":76,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":77,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:07:50.191828+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:07:50.191828+0000","last_clean_scrub_stamp":"2024-03-24T00:07:50.191828+0000","objects_scrubbed":0,"log_size":137,"log_dups_size":0,"ondisk_log_size":137,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T01:55:26.549348+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,10],"acting":[7,5,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"88'5","reported_seq":27,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-24T00:15:07.750385+0000","last_change":"2024-03-24T00:13:02.325667+0000","last_active":"2024-03-24T00:15:07.750385+0000","last_peered":"2024-03-24T00:15:07.750385+0000","last_clean":"2024-03-24T00:15:07.750385+0000","last_became_active":"2024-03-24T00:13:02.322476+0000","last_became_peered":"2024-03-24T00:13:02.322476+0000","last_unstale":"2024-03-24T00:15:07.750385+0000","last_undegraded":"2024-03-24T00:15:07.750385+0000","last_fullsized":"2024-03-24T00:15:07.750385+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:13:01.267198+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:13:01.267198+0000","last_clean_scrub_stamp":"2024-03-24T00:13:01.267198+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T08:07:19.936708+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,2],"acting":[6,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.0","version":"86'4","reported_seq":28,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-24T00:15:07.729687+0000","last_change":"2024-03-24T00:13:01.298082+0000","last_active":"2024-03-24T00:15:07.729687+0000","last_peered":"2024-03-24T00:15:07.729687+0000","last_clean":"2024-03-24T00:15:07.729687+0000","last_became_active":"2024-03-24T00:13:01.297686+0000","last_became_peered":"2024-03-24T00:13:01.297686+0000","last_unstale":"2024-03-24T00:15:07.729687+0000","last_undegraded":"2024-03-24T00:15:07.729687+0000","last_fullsized":"2024-03-24T00:15:07.729687+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:13:00.222790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:13:00.222790+0000","last_clean_scrub_stamp":"2024-03-24T00:13:00.222790+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T09:25:07.204076+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,5,8],"acting":[7,5,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.5","version":"86'4","reported_seq":29,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-24T00:15:07.729914+0000","last_change":"2024-03-24T00:13:01.321250+0000","last_active":"2024-03-24T00:15:07.729914+0000","last_peered":"2024-03-24T00:15:07.729914+0000","last_clean":"2024-03-24T00:15:07.729914+0000","last_became_active":"2024-03-24T00:13:01.320929+0000","last_became_peered":"2024-03-24T00:13:01.320929+0000","last_unstale":"2024-03-24T00:15:07.729914+0000","last_undegraded":"2024-03-24T00:15:07.729914+0000","last_fullsized":"2024-03-24T00:15:07.729914+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:13:00.222790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:13:00.222790+0000","last_clean_scrub_stamp":"2024-03-24T00:13:00.222790+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T05:22:05.490303+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":112,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":5,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,0,9],"acting":[8,0,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.4","version":"88'9","reported_seq":31,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-24T00:15:07.751893+0000","last_change":"2024-03-24T00:13:02.323413+0000","last_active":"2024-03-24T00:15:07.751893+0000","last_peered":"2024-03-24T00:15:07.751893+0000","last_clean":"2024-03-24T00:15:07.751893+0000","last_became_active":"2024-03-24T00:13:02.322947+0000","last_became_peered":"2024-03-24T00:13:02.322947+0000","last_unstale":"2024-03-24T00:15:07.751893+0000","last_undegraded":"2024-03-24T00:15:07.751893+0000","last_fullsized":"2024-03-24T00:15:07.751893+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:13:01.267198+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:13:01.267198+0000","last_clean_scrub_stamp":"2024-03-24T00:13:01.267198+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T00:16:14.490270+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,2,9],"acting":[6,2,9],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"2.6","version":"90'14","reported_seq":38,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-24T00:15:37.357286+0000","last_change":"2024-03-24T00:13:01.320380+0000","last_active":"2024-03-24T00:15:37.357286+0000","last_peered":"2024-03-24T00:15:37.357286+0000","last_clean":"2024-03-24T00:15:37.357286+0000","last_became_active":"2024-03-24T00:13:01.319192+0000","last_became_peered":"2024-03-24T00:13:01.319192+0000","last_unstale":"2024-03-24T00:15:37.357286+0000","last_undegraded":"2024-03-24T00:15:37.357286+0000","last_fullsized":"2024-03-24T00:15:37.357286+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:13:00.222790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:13:00.222790+0000","last_clean_scrub_stamp":"2024-03-24T00:13:00.222790+0000","objects_scrubbed":0,"log_size":14,"log_dups_size":0,"ondisk_log_size":14,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T02:09:27.085298+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":24,"num_write_kb":4,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[9,8,3],"acting":[9,8,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":9,"acting_primary":9,"purged_snaps":[]},{"pgid":"3.7","version":"88'5","reported_seq":27,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-24T00:15:07.753013+0000","last_change":"2024-03-24T00:13:02.328067+0000","last_active":"2024-03-24T00:15:07.753013+0000","last_peered":"2024-03-24T00:15:07.753013+0000","last_clean":"2024-03-24T00:15:07.753013+0000","last_became_active":"2024-03-24T00:13:02.327468+0000","last_became_peered":"2024-03-24T00:13:02.327468+0000","last_unstale":"2024-03-24T00:15:07.753013+0000","last_undegraded":"2024-03-24T00:15:07.753013+0000","last_fullsized":"2024-03-24T00:15:07.753013+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:13:01.267198+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:13:01.267198+0000","last_clean_scrub_stamp":"2024-03-24T00:13:01.267198+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T01:21:58.596941+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,11,7],"acting":[3,11,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.7","version":"86'7","reported_seq":32,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-24T00:15:07.748240+0000","last_change":"2024-03-24T00:13:01.309189+0000","last_active":"2024-03-24T00:15:07.748240+0000","last_peered":"2024-03-24T00:15:07.748240+0000","last_clean":"2024-03-24T00:15:07.748240+0000","last_became_active":"2024-03-24T00:13:01.308871+0000","last_became_peered":"2024-03-24T00:13:01.308871+0000","last_unstale":"2024-03-24T00:15:07.748240+0000","last_undegraded":"2024-03-24T00:15:07.748240+0000","last_fullsized":"2024-03-24T00:15:07.748240+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:13:00.222790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:13:00.222790+0000","last_clean_scrub_stamp":"2024-03-24T00:13:00.222790+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T01:00:32.610036+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":22,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,11,6],"acting":[4,11,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"88'7","reported_seq":29,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-24T00:15:07.750291+0000","last_change":"2024-03-24T00:13:02.332238+0000","last_active":"2024-03-24T00:15:07.750291+0000","last_peered":"2024-03-24T00:15:07.750291+0000","last_clean":"2024-03-24T00:15:07.750291+0000","last_became_active":"2024-03-24T00:13:02.331909+0000","last_became_peered":"2024-03-24T00:13:02.331909+0000","last_unstale":"2024-03-24T00:15:07.750291+0000","last_undegraded":"2024-03-24T00:15:07.750291+0000","last_fullsized":"2024-03-24T00:15:07.750291+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:13:01.267198+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:13:01.267198+0000","last_clean_scrub_stamp":"2024-03-24T00:13:01.267198+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T04:41:14.762251+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,3,4],"acting":[0,3,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.8","version":"86'2","reported_seq":26,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-24T00:15:07.726733+0000","last_change":"2024-03-24T00:13:01.306883+0000","last_active":"2024-03-24T00:15:07.726733+0000","last_peered":"2024-03-24T00:15:07.726733+0000","last_clean":"2024-03-24T00:15:07.726733+0000","last_became_active":"2024-03-24T00:13:01.306629+0000","last_became_peered":"2024-03-24T00:13:01.306629+0000","last_unstale":"2024-03-24T00:15:07.726733+0000","last_undegraded":"2024-03-24T00:15:07.726733+0000","last_fullsized":"2024-03-24T00:15:07.726733+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:13:00.222790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:13:00.222790+0000","last_clean_scrub_stamp":"2024-03-24T00:13:00.222790+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T08:11:52.398783+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":3,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2,0],"acting":[1,2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.9","version":"88'2","reported_seq":24,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-24T00:15:07.729642+0000","last_change":"2024-03-24T00:13:02.308421+0000","last_active":"2024-03-24T00:15:07.729642+0000","last_peered":"2024-03-24T00:15:07.729642+0000","last_clean":"2024-03-24T00:15:07.729642+0000","last_became_active":"2024-03-24T00:13:02.308081+0000","last_became_peered":"2024-03-24T00:13:02.308081+0000","last_unstale":"2024-03-24T00:15:07.729642+0000","last_undegraded":"2024-03-24T00:15:07.729642+0000","last_fullsized":"2024-03-24T00:15:07.729642+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:13:01.267198+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:13:01.267198+0000","last_clean_scrub_stamp":"2024-03-24T00:13:01.267198+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T06:50:42.356293+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,11,10],"acting":[8,11,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"2.9","version":"86'7","reported_seq":31,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-24T00:15:07.750083+0000","last_change":"2024-03-24T00:13:01.324439+0000","last_active":"2024-03-24T00:15:07.750083+0000","last_peered":"2024-03-24T00:15:07.750083+0000","last_clean":"2024-03-24T00:15:07.750083+0000","last_became_active":"2024-03-24T00:13:01.324037+0000","last_became_peered":"2024-03-24T00:13:01.324037+0000","last_unstale":"2024-03-24T00:15:07.750083+0000","last_undegraded":"2024-03-24T00:15:07.750083+0000","last_fullsized":"2024-03-24T00:15:07.750083+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:13:00.222790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:13:00.222790+0000","last_clean_scrub_stamp":"2024-03-24T00:13:00.222790+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T01:37:56.767931+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":90,"num_objects":4,"num_object_clones":0,"num_object_copies":12,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":10,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":3,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,9,10],"acting":[3,9,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.8","version":"88'9","reported_seq":31,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-24T00:15:07.727799+0000","last_change":"2024-03-24T00:13:02.313740+0000","last_active":"2024-03-24T00:15:07.727799+0000","last_peered":"2024-03-24T00:15:07.727799+0000","last_clean":"2024-03-24T00:15:07.727799+0000","last_became_active":"2024-03-24T00:13:02.313401+0000","last_became_peered":"2024-03-24T00:13:02.313401+0000","last_unstale":"2024-03-24T00:15:07.727799+0000","last_undegraded":"2024-03-24T00:15:07.727799+0000","last_fullsized":"2024-03-24T00:15:07.727799+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:13:01.267198+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:13:01.267198+0000","last_clean_scrub_stamp":"2024-03-24T00:13:01.267198+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T04:16:52.569135+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,0,2],"acting":[11,0,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"2.a","version":"86'3","reported_seq":21,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-24T00:15:07.727832+0000","last_change":"2024-03-24T00:14:07.514050+0000","last_active":"2024-03-24T00:15:07.727832+0000","last_peered":"2024-03-24T00:15:07.727832+0000","last_clean":"2024-03-24T00:15:07.727832+0000","last_became_active":"2024-03-24T00:14:07.509219+0000","last_became_peered":"2024-03-24T00:14:07.509219+0000","last_unstale":"2024-03-24T00:15:07.727832+0000","last_undegraded":"2024-03-24T00:15:07.727832+0000","last_fullsized":"2024-03-24T00:15:07.727832+0000","mapping_epoch":87,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":88,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:13:00.222790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:13:00.222790+0000","last_clean_scrub_stamp":"2024-03-24T00:13:00.222790+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T08:05:33.888846+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,3,8],"acting":[11,3,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.b","version":"88'8","reported_seq":30,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-24T00:15:07.752804+0000","last_change":"2024-03-24T00:13:02.337413+0000","last_active":"2024-03-24T00:15:07.752804+0000","last_peered":"2024-03-24T00:15:07.752804+0000","last_clean":"2024-03-24T00:15:07.752804+0000","last_became_active":"2024-03-24T00:13:02.336910+0000","last_became_peered":"2024-03-24T00:13:02.336910+0000","last_unstale":"2024-03-24T00:15:07.752804+0000","last_undegraded":"2024-03-24T00:15:07.752804+0000","last_fullsized":"2024-03-24T00:15:07.752804+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:13:01.267198+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:13:01.267198+0000","last_clean_scrub_stamp":"2024-03-24T00:13:01.267198+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T10:51:06.549979+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,10],"acting":[3,0,10],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.b","version":"86'4","reported_seq":29,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-24T00:15:07.729799+0000","last_change":"2024-03-24T00:13:01.300201+0000","last_active":"2024-03-24T00:15:07.729799+0000","last_peered":"2024-03-24T00:15:07.729799+0000","last_clean":"2024-03-24T00:15:07.729799+0000","last_became_active":"2024-03-24T00:13:01.299750+0000","last_became_peered":"2024-03-24T00:13:01.299750+0000","last_unstale":"2024-03-24T00:15:07.729799+0000","last_undegraded":"2024-03-24T00:15:07.729799+0000","last_fullsized":"2024-03-24T00:15:07.729799+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:13:00.222790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:13:00.222790+0000","last_clean_scrub_stamp":"2024-03-24T00:13:00.222790+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T01:37:32.768926+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":2,"num_read_kb":2,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[8,2,7],"acting":[8,2,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":8,"acting_primary":8,"purged_snaps":[]},{"pgid":"3.a","version":"88'5","reported_seq":27,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-24T00:15:07.729769+0000","last_change":"2024-03-24T00:13:02.331732+0000","last_active":"2024-03-24T00:15:07.729769+0000","last_peered":"2024-03-24T00:15:07.729769+0000","last_clean":"2024-03-24T00:15:07.729769+0000","last_became_active":"2024-03-24T00:13:02.331365+0000","last_became_peered":"2024-03-24T00:13:02.331365+0000","last_unstale":"2024-03-24T00:15:07.729769+0000","last_undegraded":"2024-03-24T00:15:07.729769+0000","last_fullsized":"2024-03-24T00:15:07.729769+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:13:01.267198+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:13:01.267198+0000","last_clean_scrub_stamp":"2024-03-24T00:13:01.267198+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:27:53.598158+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,6],"acting":[7,1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.c","version":"90'8","reported_seq":187,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-24T00:15:39.488276+0000","last_change":"2024-03-24T00:13:01.308725+0000","last_active":"2024-03-24T00:15:39.488276+0000","last_peered":"2024-03-24T00:15:39.488276+0000","last_clean":"2024-03-24T00:15:39.488276+0000","last_became_active":"2024-03-24T00:13:01.308324+0000","last_became_peered":"2024-03-24T00:13:01.308324+0000","last_unstale":"2024-03-24T00:15:39.488276+0000","last_undegraded":"2024-03-24T00:15:39.488276+0000","last_fullsized":"2024-03-24T00:15:39.488276+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:13:00.222790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:13:00.222790+0000","last_clean_scrub_stamp":"2024-03-24T00:13:00.222790+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T07:43:03.691683+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":2093,"num_objects":3,"num_object_clones":0,"num_object_copies":9,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":155,"num_read_kb":4,"num_write":8,"num_write_kb":5,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":1,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,11,0],"acting":[2,11,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.d","version":"88'7","reported_seq":29,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-24T00:15:07.730048+0000","last_change":"2024-03-24T00:13:02.305749+0000","last_active":"2024-03-24T00:15:07.730048+0000","last_peered":"2024-03-24T00:15:07.730048+0000","last_clean":"2024-03-24T00:15:07.730048+0000","last_became_active":"2024-03-24T00:13:02.305413+0000","last_became_peered":"2024-03-24T00:13:02.305413+0000","last_unstale":"2024-03-24T00:15:07.730048+0000","last_undegraded":"2024-03-24T00:15:07.730048+0000","last_fullsized":"2024-03-24T00:15:07.730048+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:13:01.267198+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:13:01.267198+0000","last_clean_scrub_stamp":"2024-03-24T00:13:01.267198+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T00:54:56.437725+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1,11],"acting":[7,1,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.d","version":"86'3","reported_seq":28,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-24T00:15:07.747344+0000","last_change":"2024-03-24T00:13:01.304883+0000","last_active":"2024-03-24T00:15:07.747344+0000","last_peered":"2024-03-24T00:15:07.747344+0000","last_clean":"2024-03-24T00:15:07.747344+0000","last_became_active":"2024-03-24T00:13:01.304459+0000","last_became_peered":"2024-03-24T00:13:01.304459+0000","last_unstale":"2024-03-24T00:15:07.747344+0000","last_undegraded":"2024-03-24T00:15:07.747344+0000","last_fullsized":"2024-03-24T00:15:07.747344+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:13:00.222790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:13:00.222790+0000","last_clean_scrub_stamp":"2024-03-24T00:13:00.222790+0000","objects_scrubbed":0,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T06:14:22.473571+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":70,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":1,"num_read_kb":1,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,10,3],"acting":[5,10,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.c","version":"88'7","reported_seq":29,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-24T00:15:07.729746+0000","last_change":"2024-03-24T00:13:02.331923+0000","last_active":"2024-03-24T00:15:07.729746+0000","last_peered":"2024-03-24T00:15:07.729746+0000","last_clean":"2024-03-24T00:15:07.729746+0000","last_became_active":"2024-03-24T00:13:02.331504+0000","last_became_peered":"2024-03-24T00:13:02.331504+0000","last_unstale":"2024-03-24T00:15:07.729746+0000","last_undegraded":"2024-03-24T00:15:07.729746+0000","last_fullsized":"2024-03-24T00:15:07.729746+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:13:01.267198+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:13:01.267198+0000","last_clean_scrub_stamp":"2024-03-24T00:13:01.267198+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T03:27:17.160248+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":117,"num_objects":1,"num_object_clones":0,"num_object_copies":3,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":1,"num_write_kb":1,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,3,5],"acting":[7,3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.e","version":"86'5","reported_seq":29,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-24T00:15:07.751611+0000","last_change":"2024-03-24T00:13:01.320223+0000","last_active":"2024-03-24T00:15:07.751611+0000","last_peered":"2024-03-24T00:15:07.751611+0000","last_clean":"2024-03-24T00:15:07.751611+0000","last_became_active":"2024-03-24T00:13:01.319342+0000","last_became_peered":"2024-03-24T00:13:01.319342+0000","last_unstale":"2024-03-24T00:15:07.751611+0000","last_undegraded":"2024-03-24T00:15:07.751611+0000","last_fullsized":"2024-03-24T00:15:07.751611+0000","mapping_epoch":83,"log_start":"0'0","ondisk_log_start":"0'0","created":83,"last_epoch_clean":84,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:13:00.222790+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:13:00.222790+0000","last_clean_scrub_stamp":"2024-03-24T00:13:00.222790+0000","objects_scrubbed":0,"log_size":5,"log_dups_size":0,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T02:24:08.018521+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,3,2],"acting":[6,3,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.f","version":"88'9","reported_seq":31,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-24T00:15:07.729949+0000","last_change":"2024-03-24T00:13:02.307719+0000","last_active":"2024-03-24T00:15:07.729949+0000","last_peered":"2024-03-24T00:15:07.729949+0000","last_clean":"2024-03-24T00:15:07.729949+0000","last_became_active":"2024-03-24T00:13:02.307399+0000","last_became_peered":"2024-03-24T00:13:02.307399+0000","last_unstale":"2024-03-24T00:15:07.729949+0000","last_undegraded":"2024-03-24T00:15:07.729949+0000","last_fullsized":"2024-03-24T00:15:07.729949+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:13:01.267198+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:13:01.267198+0000","last_clean_scrub_stamp":"2024-03-24T00:13:01.267198+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T09:38:13.077843+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,11,8],"acting":[7,11,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.20","version":"88'9","reported_seq":31,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-24T00:15:07.727941+0000","last_change":"2024-03-24T00:13:02.307469+0000","last_active":"2024-03-24T00:15:07.727941+0000","last_peered":"2024-03-24T00:15:07.727941+0000","last_clean":"2024-03-24T00:15:07.727941+0000","last_became_active":"2024-03-24T00:13:02.306873+0000","last_became_peered":"2024-03-24T00:13:02.306873+0000","last_unstale":"2024-03-24T00:15:07.727941+0000","last_undegraded":"2024-03-24T00:15:07.727941+0000","last_fullsized":"2024-03-24T00:15:07.727941+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:13:01.267198+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:13:01.267198+0000","last_clean_scrub_stamp":"2024-03-24T00:13:01.267198+0000","objects_scrubbed":0,"log_size":9,"log_dups_size":0,"ondisk_log_size":9,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:19:51.163015+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,2,5],"acting":[11,2,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.21","version":"88'12","reported_seq":34,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-24T00:15:07.726362+0000","last_change":"2024-03-24T00:13:02.307472+0000","last_active":"2024-03-24T00:15:07.726362+0000","last_peered":"2024-03-24T00:15:07.726362+0000","last_clean":"2024-03-24T00:15:07.726362+0000","last_became_active":"2024-03-24T00:13:02.307099+0000","last_became_peered":"2024-03-24T00:13:02.307099+0000","last_unstale":"2024-03-24T00:15:07.726362+0000","last_undegraded":"2024-03-24T00:15:07.726362+0000","last_fullsized":"2024-03-24T00:15:07.726362+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:13:01.267198+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:13:01.267198+0000","last_clean_scrub_stamp":"2024-03-24T00:13:01.267198+0000","objects_scrubbed":0,"log_size":12,"log_dups_size":0,"ondisk_log_size":12,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T08:49:06.461235+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,5,4],"acting":[2,5,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.22","version":"88'8","reported_seq":30,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-24T00:15:07.726483+0000","last_change":"2024-03-24T00:13:02.331133+0000","last_active":"2024-03-24T00:15:07.726483+0000","last_peered":"2024-03-24T00:15:07.726483+0000","last_clean":"2024-03-24T00:15:07.726483+0000","last_became_active":"2024-03-24T00:13:02.330780+0000","last_became_peered":"2024-03-24T00:13:02.330780+0000","last_unstale":"2024-03-24T00:15:07.726483+0000","last_undegraded":"2024-03-24T00:15:07.726483+0000","last_fullsized":"2024-03-24T00:15:07.726483+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:13:01.267198+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:13:01.267198+0000","last_clean_scrub_stamp":"2024-03-24T00:13:01.267198+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T10:46:07.933131+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3,0],"acting":[2,3,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.23","version":"88'7","reported_seq":29,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-24T00:15:07.749059+0000","last_change":"2024-03-24T00:13:02.335406+0000","last_active":"2024-03-24T00:15:07.749059+0000","last_peered":"2024-03-24T00:15:07.749059+0000","last_clean":"2024-03-24T00:15:07.749059+0000","last_became_active":"2024-03-24T00:13:02.334994+0000","last_became_peered":"2024-03-24T00:13:02.334994+0000","last_unstale":"2024-03-24T00:15:07.749059+0000","last_undegraded":"2024-03-24T00:15:07.749059+0000","last_fullsized":"2024-03-24T00:15:07.749059+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:13:01.267198+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:13:01.267198+0000","last_clean_scrub_stamp":"2024-03-24T00:13:01.267198+0000","objects_scrubbed":0,"log_size":7,"log_dups_size":0,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T03:24:59.547395+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,6,4],"acting":[0,6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.24","version":"88'4","reported_seq":26,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-24T00:15:07.752325+0000","last_change":"2024-03-24T00:13:02.332420+0000","last_active":"2024-03-24T00:15:07.752325+0000","last_peered":"2024-03-24T00:15:07.752325+0000","last_clean":"2024-03-24T00:15:07.752325+0000","last_became_active":"2024-03-24T00:13:02.331496+0000","last_became_peered":"2024-03-24T00:13:02.331496+0000","last_unstale":"2024-03-24T00:15:07.752325+0000","last_undegraded":"2024-03-24T00:15:07.752325+0000","last_fullsized":"2024-03-24T00:15:07.752325+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:13:01.267198+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:13:01.267198+0000","last_clean_scrub_stamp":"2024-03-24T00:13:01.267198+0000","objects_scrubbed":0,"log_size":4,"log_dups_size":0,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T03:15:06.007947+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4,0],"acting":[6,4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.25","version":"88'6","reported_seq":28,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-24T00:15:07.728441+0000","last_change":"2024-03-24T00:13:02.309077+0000","last_active":"2024-03-24T00:15:07.728441+0000","last_peered":"2024-03-24T00:15:07.728441+0000","last_clean":"2024-03-24T00:15:07.728441+0000","last_became_active":"2024-03-24T00:13:02.308752+0000","last_became_peered":"2024-03-24T00:13:02.308752+0000","last_unstale":"2024-03-24T00:15:07.728441+0000","last_undegraded":"2024-03-24T00:15:07.728441+0000","last_fullsized":"2024-03-24T00:15:07.728441+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:13:01.267198+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:13:01.267198+0000","last_clean_scrub_stamp":"2024-03-24T00:13:01.267198+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T06:49:19.657988+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,8,5],"acting":[11,8,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]},{"pgid":"3.26","version":"88'15","reported_seq":37,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-24T00:15:07.726661+0000","last_change":"2024-03-24T00:13:02.304962+0000","last_active":"2024-03-24T00:15:07.726661+0000","last_peered":"2024-03-24T00:15:07.726661+0000","last_clean":"2024-03-24T00:15:07.726661+0000","last_became_active":"2024-03-24T00:13:02.304580+0000","last_became_peered":"2024-03-24T00:13:02.304580+0000","last_unstale":"2024-03-24T00:15:07.726661+0000","last_undegraded":"2024-03-24T00:15:07.726661+0000","last_fullsized":"2024-03-24T00:15:07.726661+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:13:01.267198+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:13:01.267198+0000","last_clean_scrub_stamp":"2024-03-24T00:13:01.267198+0000","objects_scrubbed":0,"log_size":15,"log_dups_size":0,"ondisk_log_size":15,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:45:26.779169+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,8,1],"acting":[2,8,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.27","version":"88'2","reported_seq":24,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-24T00:15:07.752778+0000","last_change":"2024-03-24T00:13:02.321908+0000","last_active":"2024-03-24T00:15:07.752778+0000","last_peered":"2024-03-24T00:15:07.752778+0000","last_clean":"2024-03-24T00:15:07.752778+0000","last_became_active":"2024-03-24T00:13:02.321495+0000","last_became_peered":"2024-03-24T00:13:02.321495+0000","last_unstale":"2024-03-24T00:15:07.752778+0000","last_undegraded":"2024-03-24T00:15:07.752778+0000","last_fullsized":"2024-03-24T00:15:07.752778+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:13:01.267198+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:13:01.267198+0000","last_clean_scrub_stamp":"2024-03-24T00:13:01.267198+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T08:43:29.261314+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,4,2],"acting":[3,4,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.28","version":"88'6","reported_seq":28,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-24T00:15:07.747128+0000","last_change":"2024-03-24T00:13:02.331966+0000","last_active":"2024-03-24T00:15:07.747128+0000","last_peered":"2024-03-24T00:15:07.747128+0000","last_clean":"2024-03-24T00:15:07.747128+0000","last_became_active":"2024-03-24T00:13:02.331633+0000","last_became_peered":"2024-03-24T00:13:02.331633+0000","last_unstale":"2024-03-24T00:15:07.747128+0000","last_undegraded":"2024-03-24T00:15:07.747128+0000","last_fullsized":"2024-03-24T00:15:07.747128+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:13:01.267198+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:13:01.267198+0000","last_clean_scrub_stamp":"2024-03-24T00:13:01.267198+0000","objects_scrubbed":0,"log_size":6,"log_dups_size":0,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T05:04:01.574351+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,9,3],"acting":[5,9,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.29","version":"88'8","reported_seq":30,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-24T00:15:07.750202+0000","last_change":"2024-03-24T00:13:02.330344+0000","last_active":"2024-03-24T00:15:07.750202+0000","last_peered":"2024-03-24T00:15:07.750202+0000","last_clean":"2024-03-24T00:15:07.750202+0000","last_became_active":"2024-03-24T00:13:02.328218+0000","last_became_peered":"2024-03-24T00:13:02.328218+0000","last_unstale":"2024-03-24T00:15:07.750202+0000","last_undegraded":"2024-03-24T00:15:07.750202+0000","last_fullsized":"2024-03-24T00:15:07.750202+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:13:01.267198+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:13:01.267198+0000","last_clean_scrub_stamp":"2024-03-24T00:13:01.267198+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T08:54:36.392397+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,2,11],"acting":[3,2,11],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.2a","version":"88'10","reported_seq":32,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-24T00:15:07.726927+0000","last_change":"2024-03-24T00:13:02.330956+0000","last_active":"2024-03-24T00:15:07.726927+0000","last_peered":"2024-03-24T00:15:07.726927+0000","last_clean":"2024-03-24T00:15:07.726927+0000","last_became_active":"2024-03-24T00:13:02.330642+0000","last_became_peered":"2024-03-24T00:13:02.330642+0000","last_unstale":"2024-03-24T00:15:07.726927+0000","last_undegraded":"2024-03-24T00:15:07.726927+0000","last_fullsized":"2024-03-24T00:15:07.726927+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:13:01.267198+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:13:01.267198+0000","last_clean_scrub_stamp":"2024-03-24T00:13:01.267198+0000","objects_scrubbed":0,"log_size":10,"log_dups_size":0,"ondisk_log_size":10,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T11:41:11.326722+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,7,3],"acting":[1,7,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.2b","version":"88'11","reported_seq":33,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-24T00:15:07.751540+0000","last_change":"2024-03-24T00:13:02.333026+0000","last_active":"2024-03-24T00:15:07.751540+0000","last_peered":"2024-03-24T00:15:07.751540+0000","last_clean":"2024-03-24T00:15:07.751540+0000","last_became_active":"2024-03-24T00:13:02.331232+0000","last_became_peered":"2024-03-24T00:13:02.331232+0000","last_unstale":"2024-03-24T00:15:07.751540+0000","last_undegraded":"2024-03-24T00:15:07.751540+0000","last_fullsized":"2024-03-24T00:15:07.751540+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:13:01.267198+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:13:01.267198+0000","last_clean_scrub_stamp":"2024-03-24T00:13:01.267198+0000","objects_scrubbed":0,"log_size":11,"log_dups_size":0,"ondisk_log_size":11,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T02:14:18.504012+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7,0],"acting":[6,7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"3.2c","version":"88'8","reported_seq":30,"reported_epoch":90,"state":"active+clean","last_fresh":"2024-03-24T00:15:07.727828+0000","last_change":"2024-03-24T00:13:02.309023+0000","last_active":"2024-03-24T00:15:07.727828+0000","last_peered":"2024-03-24T00:15:07.727828+0000","last_clean":"2024-03-24T00:15:07.727828+0000","last_became_active":"2024-03-24T00:13:02.308236+0000","last_became_peered":"2024-03-24T00:13:02.308236+0000","last_unstale":"2024-03-24T00:15:07.727828+0000","last_undegraded":"2024-03-24T00:15:07.727828+0000","last_fullsized":"2024-03-24T00:15:07.727828+0000","mapping_epoch":84,"log_start":"0'0","ondisk_log_start":"0'0","created":84,"last_epoch_clean":85,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T00:13:01.267198+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T00:13:01.267198+0000","last_clean_scrub_stamp":"2024-03-24T00:13:01.267198+0000","objects_scrubbed":0,"log_size":8,"log_dups_size":0,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T03:30:37.427095+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[11,4,8],"acting":[11,4,8],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":11,"acting_primary":11,"purged_snaps":[]}],"pool_stats":[{"poolid":3,"num_pg":64,"stat_sum":{"num_bytes":234,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":702,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":502,"ondisk_log_size":502,"up":192,"acting":192,"num_store_stats":12},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":126,"num_read_kb":109,"num_write":181,"num_write_kb":2938,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4,"num_bytes_recovered":1180736,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":442368,"data_stored":4722944,"data_compressed":284592,"data_compressed_allocated":360448,"data_compressed_original":4669440,"omap_allocated":0,"internal_metadata":0},"log_size":137,"ondisk_log_size":137,"up":3,"acting":3,"num_store_stats":8},{"poolid":2,"num_pg":32,"stat_sum":{"num_bytes":70524,"num_objects":98,"num_object_clones":0,"num_object_copies":294,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":98,"num_whiteouts":0,"num_read":1563,"num_read_kb":856,"num_write":254,"num_write_kb":149,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21,"num_bytes_recovered":62373,"num_keys_recovered":10,"num_objects_omap":66,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":532480,"data_stored":219465,"data_compressed":13143,"data_compressed_allocated":20480,"data_compressed_original":73728,"omap_allocated":0,"internal_metadata":0},"log_size":221,"ondisk_log_size":221,"up":96,"acting":96,"num_store_stats":12}],"osd_stats":[{"osd":11,"up_from":81,"seq":347892351021,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27980,"kb_used_data":960,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717172,"statfs":{"total":95995035648,"available":95966384128,"internally_reserved":0,"allocated":983040,"data_stored":478079,"data_compressed":11573,"data_compressed_allocated":16384,"data_compressed_original":32768,"omap_allocated":1596,"internal_metadata":27654596},"hb_peers":[0,1,2,3,4,5,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":10,"up_from":73,"seq":313532612658,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28128,"kb_used_data":1112,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717024,"statfs":{"total":95995035648,"available":95966232576,"internally_reserved":0,"allocated":1138688,"data_stored":1130832,"data_compressed":49481,"data_compressed_allocated":65536,"data_compressed_original":630784,"omap_allocated":1599,"internal_metadata":27654593},"hb_peers":[0,1,2,3,4,5,6,8,9,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":9,"up_from":68,"seq":292057776184,"num_pgs":20,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32160,"kb_used_data":984,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712992,"statfs":{"total":95995035648,"available":95962103808,"internally_reserved":0,"allocated":1007616,"data_stored":477477,"data_compressed":11573,"data_compressed_allocated":16384,"data_compressed_original":32768,"omap_allocated":1600,"internal_metadata":31914432},"hb_peers":[0,1,2,3,4,5,6,7,8,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":8,"up_from":60,"seq":257698037823,"num_pgs":24,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":36308,"kb_used_data":972,"kb_used_omap":1,"kb_used_meta":35326,"kb_avail":93708844,"statfs":{"total":95995035648,"available":95957856256,"internally_reserved":0,"allocated":995328,"data_stored":477183,"data_compressed":11573,"data_compressed_allocated":16384,"data_compressed_original":32768,"omap_allocated":1595,"internal_metadata":36174277},"hb_peers":[0,1,2,3,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":7,"up_from":52,"seq":223338299461,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28056,"kb_used_data":1032,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717096,"statfs":{"total":95995035648,"available":95966306304,"internally_reserved":0,"allocated":1056768,"data_stored":1070905,"data_compressed":47762,"data_compressed_allocated":61440,"data_compressed_original":622592,"omap_allocated":1590,"internal_metadata":27654602},"hb_peers":[1,2,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":6,"up_from":47,"seq":201863462987,"num_pgs":21,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27992,"kb_used_data":968,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717160,"statfs":{"total":95995035648,"available":95966371840,"internally_reserved":0,"allocated":991232,"data_stored":477194,"data_compressed":11573,"data_compressed_allocated":16384,"data_compressed_original":32768,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1,2,3,4,5,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":13,"seq":55834574950,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28060,"kb_used_data":1044,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717092,"statfs":{"total":95995035648,"available":95966302208,"internally_reserved":0,"allocated":1069056,"data_stored":549319,"data_compressed":13292,"data_compressed_allocated":20480,"data_compressed_original":40960,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738476,"num_pgs":25,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28012,"kb_used_data":988,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717140,"statfs":{"total":95995035648,"available":95966351360,"internally_reserved":0,"allocated":1011712,"data_stored":479456,"data_compressed":11573,"data_compressed_allocated":16384,"data_compressed_original":32768,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2,3,4,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":2,"up_from":18,"seq":77309411425,"num_pgs":26,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":27996,"kb_used_data":980,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717156,"statfs":{"total":95995035648,"available":95966367744,"internally_reserved":0,"allocated":1003520,"data_stored":480173,"data_compressed":11573,"data_compressed_allocated":16384,"data_compressed_original":32768,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,3,4,5,6,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":3,"up_from":26,"seq":111669149788,"num_pgs":27,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32160,"kb_used_data":984,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712992,"statfs":{"total":95995035648,"available":95962103808,"internally_reserved":0,"allocated":1007616,"data_stored":477019,"data_compressed":11573,"data_compressed_allocated":16384,"data_compressed_original":32768,"omap_allocated":1593,"internal_metadata":31914439},"hb_peers":[0,1,2,4,5,6,7,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":4,"up_from":34,"seq":146028888149,"num_pgs":23,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":32172,"kb_used_data":996,"kb_used_omap":1,"kb_used_meta":31166,"kb_avail":93712980,"statfs":{"total":95995035648,"available":95962091520,"internally_reserved":0,"allocated":1019904,"data_stored":539311,"data_compressed":21278,"data_compressed_allocated":28672,"data_compressed_original":90112,"omap_allocated":1595,"internal_metadata":31914437},"hb_peers":[1,2,3,5,6,7,8,9,10,11],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":5,"up_from":39,"seq":167503724624,"num_pgs":28,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":28060,"kb_used_data":1044,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":93717092,"statfs":{"total":95995035648,"available":95966302208,"internally_reserved":0,"allocated":1069056,"data_stored":1070583,"data_compressed":47762,"data_compressed_allocated":61440,"data_compressed_original":622592,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1,2,3,4,6,7,8,9,10],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34549,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34549,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":65536,"data_stored":590368,"data_compressed":34549,"data_compressed_allocated":45056,"data_compressed_original":573440,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":36189,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":36189,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":36189,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":36189,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":590368,"data_compressed":36189,"data_compressed_allocated":45056,"data_compressed_original":589824,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":40960,"data_stored":3101,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":98304,"data_stored":72964,"data_compressed":1719,"data_compressed_allocated":4096,"data_compressed_original":8192,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":3818,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":28672,"data_stored":430,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":49152,"data_stored":62956,"data_compressed":9705,"data_compressed_allocated":12288,"data_compressed_original":57344,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":45056,"data_stored":3743,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":16384,"data_stored":722,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":32768,"data_stored":4065,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":24576,"data_stored":828,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":36864,"data_stored":1122,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":114688,"data_stored":63992,"data_compressed":1719,"data_compressed_allocated":4096,"data_compressed_original":8192,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":12288,"data_stored":1724,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":234,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":4,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":5,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":6,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":7,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":8,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":9,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":10,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":117,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":3,"osd":11,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-24T00:15:45.050 DEBUG:teuthology.orchestra.run.smithi118:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-24T00:15:45.203 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:15:44 smithi189 ceph-mon[36187]: pgmap v399: 97 pgs: 97 active+clean; 646 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 511 B/s wr, 10 op/s 2024-03-24T00:15:45.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:15:44 smithi193 ceph-mon[36878]: pgmap v399: 97 pgs: 97 active+clean; 646 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 511 B/s wr, 10 op/s 2024-03-24T00:15:45.295 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:45.295+0000 7fe8f1800640 1 -- 172.21.15.118:0/2238200473 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe8ec076060 msgr2=0x7fe8ec076480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:15:45.295 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:45.295+0000 7fe8f1800640 1 --2- 172.21.15.118:0/2238200473 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe8ec076060 0x7fe8ec076480 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7fe8dc009ca0 tx=0x7fe8dc02c100 comp rx=0 tx=0).stop 2024-03-24T00:15:45.295 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:45.295+0000 7fe8f1800640 1 -- 172.21.15.118:0/2238200473 shutdown_connections 2024-03-24T00:15:45.295 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:45.295+0000 7fe8f1800640 1 --2- 172.21.15.118:0/2238200473 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fe8ec077c40 0x7fe8ec0784e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:15:45.296 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:45.295+0000 7fe8f1800640 1 --2- 172.21.15.118:0/2238200473 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fe8ec0772a0 0x7fe8ec077700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:15:45.296 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:45.295+0000 7fe8f1800640 1 --2- 172.21.15.118:0/2238200473 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe8ec076060 0x7fe8ec076480 unknown :-1 s=CLOSED pgs=199 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:15:45.296 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:45.295+0000 7fe8f1800640 1 -- 172.21.15.118:0/2238200473 >> 172.21.15.118:0/2238200473 conn(0x7fe8ec0fee90 msgr2=0x7fe8ec1012b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:15:45.296 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:45.296+0000 7fe8f1800640 1 -- 172.21.15.118:0/2238200473 shutdown_connections 2024-03-24T00:15:45.296 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:45.296+0000 7fe8f1800640 1 -- 172.21.15.118:0/2238200473 wait complete. 2024-03-24T00:15:45.296 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:45.297+0000 7fe8f1800640 1 Processor -- start 2024-03-24T00:15:45.296 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:45.297+0000 7fe8f1800640 1 -- start start 2024-03-24T00:15:45.298 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:45.297+0000 7fe8f1800640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fe8ec076060 0x7fe8ec19e030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:15:45.298 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:45.297+0000 7fe8f1800640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe8ec0772a0 0x7fe8ec19e570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:15:45.298 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:45.297+0000 7fe8f1800640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fe8ec077c40 0x7fe8ec1a55e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:15:45.299 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:45.297+0000 7fe8f1800640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe8ec113970 con 0x7fe8ec077c40 2024-03-24T00:15:45.299 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:45.297+0000 7fe8f1800640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe8ec1137f0 con 0x7fe8ec0772a0 2024-03-24T00:15:45.299 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:45.297+0000 7fe8f1800640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe8ec113af0 con 0x7fe8ec076060 2024-03-24T00:15:45.299 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:45.298+0000 7fe8eaa00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fe8ec076060 0x7fe8ec19e030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:15:45.299 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:45.298+0000 7fe8eaa00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fe8ec076060 0x7fe8ec19e030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.118:33992/0 (socket says 172.21.15.118:33992) 2024-03-24T00:15:45.299 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:45.298+0000 7fe8eaa00640 1 -- 172.21.15.118:0/3207048194 learned_addr learned my addr 172.21.15.118:0/3207048194 (peer_addr_for_me v2:172.21.15.118:0/0) 2024-03-24T00:15:45.299 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:45.298+0000 7fe8eb400640 1 --2- 172.21.15.118:0/3207048194 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fe8ec077c40 0x7fe8ec1a55e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:15:45.299 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:45.298+0000 7fe8ea000640 1 --2- 172.21.15.118:0/3207048194 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe8ec0772a0 0x7fe8ec19e570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:15:45.299 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:45.298+0000 7fe8eaa00640 1 -- 172.21.15.118:0/3207048194 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe8ec0772a0 msgr2=0x7fe8ec19e570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:15:45.299 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:45.298+0000 7fe8eaa00640 1 --2- 172.21.15.118:0/3207048194 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe8ec0772a0 0x7fe8ec19e570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:15:45.299 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:45.298+0000 7fe8eaa00640 1 -- 172.21.15.118:0/3207048194 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fe8ec077c40 msgr2=0x7fe8ec1a55e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:15:45.299 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:45.298+0000 7fe8eaa00640 1 --2- 172.21.15.118:0/3207048194 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fe8ec077c40 0x7fe8ec1a55e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:15:45.299 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:45.298+0000 7fe8eaa00640 1 -- 172.21.15.118:0/3207048194 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe8ec1a5c50 con 0x7fe8ec076060 2024-03-24T00:15:45.299 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:45.298+0000 7fe8eb400640 1 --2- 172.21.15.118:0/3207048194 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fe8ec077c40 0x7fe8ec1a55e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T00:15:45.300 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:45.298+0000 7fe8eaa00640 1 --2- 172.21.15.118:0/3207048194 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fe8ec076060 0x7fe8ec19e030 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7fe8dc0325f0 tx=0x7fe8dc035d10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:15:45.300 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:45.298+0000 7fe8e3400640 1 -- 172.21.15.118:0/3207048194 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe8dc039040 con 0x7fe8ec076060 2024-03-24T00:15:45.300 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:45.299+0000 7fe8e3400640 1 -- 172.21.15.118:0/3207048194 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fe8dc044070 con 0x7fe8ec076060 2024-03-24T00:15:45.300 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:45.299+0000 7fe8f1800640 1 -- 172.21.15.118:0/3207048194 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe8ec1a5ee0 con 0x7fe8ec076060 2024-03-24T00:15:45.300 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:45.299+0000 7fe8e3400640 1 -- 172.21.15.118:0/3207048194 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe8dc004540 con 0x7fe8ec076060 2024-03-24T00:15:45.300 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:45.299+0000 7fe8f1800640 1 -- 172.21.15.118:0/3207048194 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe8ec1a6290 con 0x7fe8ec076060 2024-03-24T00:15:45.300 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:45.300+0000 7fe8f1800640 1 -- 172.21.15.118:0/3207048194 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe8ec110420 con 0x7fe8ec076060 2024-03-24T00:15:45.301 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:45.301+0000 7fe8e3400640 1 -- 172.21.15.118:0/3207048194 <== mon.1 v2:172.21.15.193:3300/0 4 ==== mgrmap(e 19) v1 ==== 137342+0+0 (secure 0 0 0) 0x7fe8dc032890 con 0x7fe8ec076060 2024-03-24T00:15:45.302 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:45.303+0000 7fe8e3400640 1 --2- 172.21.15.118:0/3207048194 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7fe8d00a2890 0x7fe8d00a4d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:15:45.302 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:45.303+0000 7fe8e3400640 1 -- 172.21.15.118:0/3207048194 <== mon.1 v2:172.21.15.193:3300/0 5 ==== osd_map(90..90 src has 1..90) v4 ==== 8277+0+0 (secure 0 0 0) 0x7fe8dc0f1210 con 0x7fe8ec076060 2024-03-24T00:15:45.302 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:45.303+0000 7fe8ea000640 1 --2- 172.21.15.118:0/3207048194 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7fe8d00a2890 0x7fe8d00a4d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:15:45.303 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:45.304+0000 7fe8ea000640 1 --2- 172.21.15.118:0/3207048194 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7fe8d00a2890 0x7fe8d00a4d50 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7fe8ec19f540 tx=0x7fe8d4005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:15:45.306 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:45.307+0000 7fe8e3400640 1 -- 172.21.15.118:0/3207048194 <== mon.1 v2:172.21.15.193:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7fe8dc0be1d0 con 0x7fe8ec076060 2024-03-24T00:15:45.509 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:45.509+0000 7fe8f1800640 1 -- 172.21.15.118:0/3207048194 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fe8ec02d220 con 0x7fe8ec076060 2024-03-24T00:15:45.511 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:45.512+0000 7fe8e3400640 1 -- 172.21.15.118:0/3207048194 <== mon.1 v2:172.21.15.193:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v90) v1 ==== 74+0+22834 (secure 0 0 0) 0x7fe8dc0bdad0 con 0x7fe8ec076060 2024-03-24T00:15:45.512 INFO:teuthology.orchestra.run.smithi118.stdout: 2024-03-24T00:15:45.512 INFO:teuthology.orchestra.run.smithi118.stdout:{"epoch":90,"fsid":"d79a5420-e971-11ee-95cd-87774f69a715","created":"2024-03-24T00:03:58.648524+0000","modified":"2024-03-24T00:15:07.721960+0000","last_up_change":"2024-03-24T00:12:03.809000+0000","last_in_change":"2024-03-24T00:11:44.389344+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-24T00:07:50.190965+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-24T00:12:59.993723+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-24T00:13:01.257347+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"86","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"c9590426-3901-4f68-a5ef-d57aa8fbac92","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6802","nonce":557690061},{"type":"v1","addr":"172.21.15.118:6803","nonce":557690061}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6804","nonce":557690061},{"type":"v1","addr":"172.21.15.118:6805","nonce":557690061}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6808","nonce":557690061},{"type":"v1","addr":"172.21.15.118:6809","nonce":557690061}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6806","nonce":557690061},{"type":"v1","addr":"172.21.15.118:6807","nonce":557690061}]},"public_addr":"172.21.15.118:6803/557690061","cluster_addr":"172.21.15.118:6805/557690061","heartbeat_back_addr":"172.21.15.118:6809/557690061","heartbeat_front_addr":"172.21.15.118:6807/557690061","state":["exists","up"]},{"osd":1,"uuid":"fc7f827e-ecf6-4661-aad0-7a22951892e0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6800","nonce":4203029485},{"type":"v1","addr":"172.21.15.189:6801","nonce":4203029485}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6802","nonce":4203029485},{"type":"v1","addr":"172.21.15.189:6803","nonce":4203029485}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6806","nonce":4203029485},{"type":"v1","addr":"172.21.15.189:6807","nonce":4203029485}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6804","nonce":4203029485},{"type":"v1","addr":"172.21.15.189:6805","nonce":4203029485}]},"public_addr":"172.21.15.189:6801/4203029485","cluster_addr":"172.21.15.189:6803/4203029485","heartbeat_back_addr":"172.21.15.189:6807/4203029485","heartbeat_front_addr":"172.21.15.189:6805/4203029485","state":["exists","up"]},{"osd":2,"uuid":"e73ccfe9-0e5b-4734-a154-a2f0d829fcfb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6800","nonce":3699076971},{"type":"v1","addr":"172.21.15.193:6801","nonce":3699076971}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6802","nonce":3699076971},{"type":"v1","addr":"172.21.15.193:6803","nonce":3699076971}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6806","nonce":3699076971},{"type":"v1","addr":"172.21.15.193:6807","nonce":3699076971}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6804","nonce":3699076971},{"type":"v1","addr":"172.21.15.193:6805","nonce":3699076971}]},"public_addr":"172.21.15.193:6801/3699076971","cluster_addr":"172.21.15.193:6803/3699076971","heartbeat_back_addr":"172.21.15.193:6807/3699076971","heartbeat_front_addr":"172.21.15.193:6805/3699076971","state":["exists","up"]},{"osd":3,"uuid":"63fed43b-43ce-4031-8957-7fe94725c1ed","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6810","nonce":4059617010},{"type":"v1","addr":"172.21.15.118:6811","nonce":4059617010}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6812","nonce":4059617010},{"type":"v1","addr":"172.21.15.118:6813","nonce":4059617010}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6816","nonce":4059617010},{"type":"v1","addr":"172.21.15.118:6817","nonce":4059617010}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6814","nonce":4059617010},{"type":"v1","addr":"172.21.15.118:6815","nonce":4059617010}]},"public_addr":"172.21.15.118:6811/4059617010","cluster_addr":"172.21.15.118:6813/4059617010","heartbeat_back_addr":"172.21.15.118:6817/4059617010","heartbeat_front_addr":"172.21.15.118:6815/4059617010","state":["exists","up"]},{"osd":4,"uuid":"57377f66-c3b4-43c1-8e8e-bc873672906d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":34,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6808","nonce":4075247068},{"type":"v1","addr":"172.21.15.189:6809","nonce":4075247068}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6810","nonce":4075247068},{"type":"v1","addr":"172.21.15.189:6811","nonce":4075247068}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6814","nonce":4075247068},{"type":"v1","addr":"172.21.15.189:6815","nonce":4075247068}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6812","nonce":4075247068},{"type":"v1","addr":"172.21.15.189:6813","nonce":4075247068}]},"public_addr":"172.21.15.189:6809/4075247068","cluster_addr":"172.21.15.189:6811/4075247068","heartbeat_back_addr":"172.21.15.189:6815/4075247068","heartbeat_front_addr":"172.21.15.189:6813/4075247068","state":["exists","up"]},{"osd":5,"uuid":"91970976-2193-4224-81c0-dc96cb022c85","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":39,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6808","nonce":2920363402},{"type":"v1","addr":"172.21.15.193:6809","nonce":2920363402}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6810","nonce":2920363402},{"type":"v1","addr":"172.21.15.193:6811","nonce":2920363402}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6814","nonce":2920363402},{"type":"v1","addr":"172.21.15.193:6815","nonce":2920363402}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6812","nonce":2920363402},{"type":"v1","addr":"172.21.15.193:6813","nonce":2920363402}]},"public_addr":"172.21.15.193:6809/2920363402","cluster_addr":"172.21.15.193:6811/2920363402","heartbeat_back_addr":"172.21.15.193:6815/2920363402","heartbeat_front_addr":"172.21.15.193:6813/2920363402","state":["exists","up"]},{"osd":6,"uuid":"0b77b196-8674-41c6-ad3d-726b5d01c8fd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":47,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6818","nonce":431354576},{"type":"v1","addr":"172.21.15.118:6819","nonce":431354576}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6820","nonce":431354576},{"type":"v1","addr":"172.21.15.118:6821","nonce":431354576}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6824","nonce":431354576},{"type":"v1","addr":"172.21.15.118:6825","nonce":431354576}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6822","nonce":431354576},{"type":"v1","addr":"172.21.15.118:6823","nonce":431354576}]},"public_addr":"172.21.15.118:6819/431354576","cluster_addr":"172.21.15.118:6821/431354576","heartbeat_back_addr":"172.21.15.118:6825/431354576","heartbeat_front_addr":"172.21.15.118:6823/431354576","state":["exists","up"]},{"osd":7,"uuid":"1fbbd38f-4e8e-4d4c-8e98-66f6862bc4bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":52,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6816","nonce":3667611284},{"type":"v1","addr":"172.21.15.189:6817","nonce":3667611284}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6818","nonce":3667611284},{"type":"v1","addr":"172.21.15.189:6819","nonce":3667611284}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6822","nonce":3667611284},{"type":"v1","addr":"172.21.15.189:6823","nonce":3667611284}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6820","nonce":3667611284},{"type":"v1","addr":"172.21.15.189:6821","nonce":3667611284}]},"public_addr":"172.21.15.189:6817/3667611284","cluster_addr":"172.21.15.189:6819/3667611284","heartbeat_back_addr":"172.21.15.189:6823/3667611284","heartbeat_front_addr":"172.21.15.189:6821/3667611284","state":["exists","up"]},{"osd":8,"uuid":"6462c2b5-e4b3-4b0b-bcfa-4de2c89e93e0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":60,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6816","nonce":3853800910},{"type":"v1","addr":"172.21.15.193:6817","nonce":3853800910}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6818","nonce":3853800910},{"type":"v1","addr":"172.21.15.193:6819","nonce":3853800910}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6822","nonce":3853800910},{"type":"v1","addr":"172.21.15.193:6823","nonce":3853800910}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6820","nonce":3853800910},{"type":"v1","addr":"172.21.15.193:6821","nonce":3853800910}]},"public_addr":"172.21.15.193:6817/3853800910","cluster_addr":"172.21.15.193:6819/3853800910","heartbeat_back_addr":"172.21.15.193:6823/3853800910","heartbeat_front_addr":"172.21.15.193:6821/3853800910","state":["exists","up"]},{"osd":9,"uuid":"54be277f-3596-422f-9de5-9005a8571c6f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":68,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6826","nonce":3248293936},{"type":"v1","addr":"172.21.15.118:6827","nonce":3248293936}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6828","nonce":3248293936},{"type":"v1","addr":"172.21.15.118:6829","nonce":3248293936}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6832","nonce":3248293936},{"type":"v1","addr":"172.21.15.118:6833","nonce":3248293936}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6830","nonce":3248293936},{"type":"v1","addr":"172.21.15.118:6831","nonce":3248293936}]},"public_addr":"172.21.15.118:6827/3248293936","cluster_addr":"172.21.15.118:6829/3248293936","heartbeat_back_addr":"172.21.15.118:6833/3248293936","heartbeat_front_addr":"172.21.15.118:6831/3248293936","state":["exists","up"]},{"osd":10,"uuid":"7ba69d69-ac41-44ce-bc39-c049a8ca5900","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":73,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6824","nonce":2112195760},{"type":"v1","addr":"172.21.15.189:6825","nonce":2112195760}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6826","nonce":2112195760},{"type":"v1","addr":"172.21.15.189:6827","nonce":2112195760}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6830","nonce":2112195760},{"type":"v1","addr":"172.21.15.189:6831","nonce":2112195760}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6828","nonce":2112195760},{"type":"v1","addr":"172.21.15.189:6829","nonce":2112195760}]},"public_addr":"172.21.15.189:6825/2112195760","cluster_addr":"172.21.15.189:6827/2112195760","heartbeat_back_addr":"172.21.15.189:6831/2112195760","heartbeat_front_addr":"172.21.15.189:6829/2112195760","state":["exists","up"]},{"osd":11,"uuid":"186f0b18-6b22-4eae-83b7-b21bbcce0817","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":81,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6824","nonce":3880515510},{"type":"v1","addr":"172.21.15.193:6825","nonce":3880515510}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6826","nonce":3880515510},{"type":"v1","addr":"172.21.15.193:6827","nonce":3880515510}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6830","nonce":3880515510},{"type":"v1","addr":"172.21.15.193:6831","nonce":3880515510}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6828","nonce":3880515510},{"type":"v1","addr":"172.21.15.193:6829","nonce":3880515510}]},"public_addr":"172.21.15.193:6825/3880515510","cluster_addr":"172.21.15.193:6827/3880515510","heartbeat_back_addr":"172.21.15.193:6831/3880515510","heartbeat_front_addr":"172.21.15.193:6829/3880515510","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:06:58.105543+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:07:23.795866+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:07:47.426050+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:08:19.427073+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:08:45.816082+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:09:11.081523+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:09:44.485077+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:10:11.338972+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:10:36.234024+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:11:09.730432+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:11:37.059083+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:12:02.371947+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.a","mappings":[{"from":2,"to":11}]},{"pgid":"2.14","mappings":[{"from":2,"to":4}]},{"pgid":"2.1d","mappings":[{"from":8,"to":11}]},{"pgid":"2.1f","mappings":[{"from":8,"to":1}]},{"pgid":"3.11","mappings":[{"from":4,"to":8},{"from":5,"to":9}]},{"pgid":"3.13","mappings":[{"from":4,"to":9}]},{"pgid":"3.14","mappings":[{"from":4,"to":6}]},{"pgid":"3.31","mappings":[{"from":4,"to":7}]},{"pgid":"3.38","mappings":[{"from":3,"to":9}]},{"pgid":"3.39","mappings":[{"from":5,"to":8}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.118:6801/389249272":"2024-03-25T00:05:05.458796+0000","172.21.15.118:6800/389249272":"2024-03-25T00:05:05.458796+0000","172.21.15.118:0/2101854694":"2024-03-25T00:05:05.458796+0000","172.21.15.118:0/630779359":"2024-03-25T00:05:05.458796+0000","172.21.15.118:0/2158921029":"2024-03-25T00:04:31.283725+0000","172.21.15.118:0/1003670455":"2024-03-25T00:05:05.458796+0000","172.21.15.118:0/368712314":"2024-03-25T00:04:31.283725+0000","172.21.15.118:0/2004775597":"2024-03-25T00:04:31.283725+0000","172.21.15.118:6800/1463147955":"2024-03-25T00:04:31.283725+0000","172.21.15.118:6801/1463147955":"2024-03-25T00:04:31.283725+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-24T00:15:45.514 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:45.514+0000 7fe8f1800640 1 -- 172.21.15.118:0/3207048194 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7fe8d00a2890 msgr2=0x7fe8d00a4d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:15:45.514 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:45.514+0000 7fe8f1800640 1 --2- 172.21.15.118:0/3207048194 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7fe8d00a2890 0x7fe8d00a4d50 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7fe8ec19f540 tx=0x7fe8d4005f90 comp rx=0 tx=0).stop 2024-03-24T00:15:45.514 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:45.514+0000 7fe8f1800640 1 -- 172.21.15.118:0/3207048194 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fe8ec076060 msgr2=0x7fe8ec19e030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:15:45.514 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:45.514+0000 7fe8f1800640 1 --2- 172.21.15.118:0/3207048194 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fe8ec076060 0x7fe8ec19e030 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7fe8dc0325f0 tx=0x7fe8dc035d10 comp rx=0 tx=0).stop 2024-03-24T00:15:45.514 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:45.514+0000 7fe8f1800640 1 -- 172.21.15.118:0/3207048194 shutdown_connections 2024-03-24T00:15:45.515 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:45.514+0000 7fe8f1800640 1 --2- 172.21.15.118:0/3207048194 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7fe8d00a2890 0x7fe8d00a4d50 unknown :-1 s=CLOSED pgs=263 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:15:45.515 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:45.514+0000 7fe8f1800640 1 --2- 172.21.15.118:0/3207048194 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fe8ec077c40 0x7fe8ec1a55e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:15:45.515 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:45.514+0000 7fe8f1800640 1 --2- 172.21.15.118:0/3207048194 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe8ec0772a0 0x7fe8ec19e570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:15:45.515 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:45.514+0000 7fe8f1800640 1 --2- 172.21.15.118:0/3207048194 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fe8ec076060 0x7fe8ec19e030 unknown :-1 s=CLOSED pgs=193 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:15:45.515 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:45.514+0000 7fe8f1800640 1 -- 172.21.15.118:0/3207048194 >> 172.21.15.118:0/3207048194 conn(0x7fe8ec0fee90 msgr2=0x7fe8ec1005d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:15:45.515 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:45.515+0000 7fe8f1800640 1 -- 172.21.15.118:0/3207048194 shutdown_connections 2024-03-24T00:15:45.515 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:45.515+0000 7fe8f1800640 1 -- 172.21.15.118:0/3207048194 wait complete. 2024-03-24T00:15:45.541 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-24T00:07:50.190965+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-24T00:12:59.993723+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '85', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.8799999952316284, 'score_stable': 1.8799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.8799999952316284, 'raw_score_stable': 1.8799999952316284, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-03-24T00:13:01.257347+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '86', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-24T00:15:45.542 DEBUG:teuthology.orchestra.run.smithi118:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-24T00:15:45.749 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:45.749+0000 7fecb4800640 1 -- 172.21.15.118:0/4150086270 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fecac109560 msgr2=0x7fecac10b9f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:15:45.749 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:45.749+0000 7fecb4800640 1 --2- 172.21.15.118:0/4150086270 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fecac109560 0x7fecac10b9f0 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7feca8009ca0 tx=0x7feca802c190 comp rx=0 tx=0).stop 2024-03-24T00:15:45.749 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:45.749+0000 7fecb4800640 1 -- 172.21.15.118:0/4150086270 shutdown_connections 2024-03-24T00:15:45.749 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:45.749+0000 7fecb4800640 1 --2- 172.21.15.118:0/4150086270 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fecac109560 0x7fecac10b9f0 unknown :-1 s=CLOSED pgs=200 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:15:45.749 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:45.749+0000 7fecb4800640 1 --2- 172.21.15.118:0/4150086270 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fecac074080 0x7fecac0744e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:15:45.749 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:45.749+0000 7fecb4800640 1 --2- 172.21.15.118:0/4150086270 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fecac075660 0x7fecac073ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:15:45.750 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:45.749+0000 7fecb4800640 1 -- 172.21.15.118:0/4150086270 >> 172.21.15.118:0/4150086270 conn(0x7fecac0fcc40 msgr2=0x7fecac0ff060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:15:45.750 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:45.749+0000 7fecb4800640 1 -- 172.21.15.118:0/4150086270 shutdown_connections 2024-03-24T00:15:45.750 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:45.749+0000 7fecb4800640 1 -- 172.21.15.118:0/4150086270 wait complete. 2024-03-24T00:15:45.751 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:45.750+0000 7fecb4800640 1 Processor -- start 2024-03-24T00:15:45.752 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:45.750+0000 7fecb4800640 1 -- start start 2024-03-24T00:15:45.752 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:45.750+0000 7fecb4800640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fecac074080 0x7fecac199be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:15:45.752 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:45.750+0000 7fecb4800640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fecac075660 0x7fecac19a120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:15:45.752 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:45.750+0000 7fecb4800640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fecac109560 0x7fecac1a1190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:15:45.752 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:45.750+0000 7fecb4800640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fecac10f590 con 0x7fecac075660 2024-03-24T00:15:45.752 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:45.750+0000 7fecb4800640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fecac10f410 con 0x7fecac074080 2024-03-24T00:15:45.752 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:45.751+0000 7fecb4800640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fecac10f710 con 0x7fecac109560 2024-03-24T00:15:45.752 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:45.751+0000 7fecb1e00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fecac074080 0x7fecac199be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:15:45.752 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:45.751+0000 7fecb1400640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fecac075660 0x7fecac19a120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:15:45.752 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:45.751+0000 7fecb2800640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fecac109560 0x7fecac1a1190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:15:45.752 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:45.751+0000 7fecb2800640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fecac109560 0x7fecac1a1190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.118:34004/0 (socket says 172.21.15.118:34004) 2024-03-24T00:15:45.752 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:45.751+0000 7fecb2800640 1 -- 172.21.15.118:0/2438878040 learned_addr learned my addr 172.21.15.118:0/2438878040 (peer_addr_for_me v2:172.21.15.118:0/0) 2024-03-24T00:15:45.752 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:45.751+0000 7fecb1400640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fecac075660 0x7fecac19a120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.118:3300/0 says I am v2:172.21.15.118:43226/0 (socket says 172.21.15.118:43226) 2024-03-24T00:15:45.752 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:45.751+0000 7fecb1e00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fecac074080 0x7fecac199be0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.118:36378/0 (socket says 172.21.15.118:36378) 2024-03-24T00:15:45.753 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:45.751+0000 7fecb2800640 1 -- 172.21.15.118:0/2438878040 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fecac074080 msgr2=0x7fecac199be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:15:45.753 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:45.751+0000 7fecb2800640 1 --2- 172.21.15.118:0/2438878040 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fecac074080 0x7fecac199be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:15:45.753 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:45.751+0000 7fecb2800640 1 -- 172.21.15.118:0/2438878040 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fecac075660 msgr2=0x7fecac19a120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:15:45.753 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:45.751+0000 7fecb2800640 1 --2- 172.21.15.118:0/2438878040 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fecac075660 0x7fecac19a120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:15:45.753 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:45.751+0000 7fecb2800640 1 -- 172.21.15.118:0/2438878040 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fecac1a1800 con 0x7fecac109560 2024-03-24T00:15:45.753 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:45.751+0000 7fecb2800640 1 --2- 172.21.15.118:0/2438878040 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fecac109560 0x7fecac1a1190 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7feca802c6d0 tx=0x7feca8003ba0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:15:45.753 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:45.752+0000 7feca6a00640 1 -- 172.21.15.118:0/2438878040 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feca8039040 con 0x7fecac109560 2024-03-24T00:15:45.753 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:45.752+0000 7fecb4800640 1 -- 172.21.15.118:0/2438878040 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fecac1a1a90 con 0x7fecac109560 2024-03-24T00:15:45.753 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:45.752+0000 7feca6a00640 1 -- 172.21.15.118:0/2438878040 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7feca8044070 con 0x7fecac109560 2024-03-24T00:15:45.753 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:45.752+0000 7feca6a00640 1 -- 172.21.15.118:0/2438878040 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7feca8004370 con 0x7fecac109560 2024-03-24T00:15:45.753 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:45.752+0000 7fecb4800640 1 -- 172.21.15.118:0/2438878040 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fecac1a2000 con 0x7fecac109560 2024-03-24T00:15:45.754 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:45.753+0000 7fecb4800640 1 -- 172.21.15.118:0/2438878040 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fecac108320 con 0x7fecac109560 2024-03-24T00:15:45.754 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:45.755+0000 7feca6a00640 1 -- 172.21.15.118:0/2438878040 <== mon.1 v2:172.21.15.193:3300/0 4 ==== mgrmap(e 19) v1 ==== 137342+0+0 (secure 0 0 0) 0x7feca8004510 con 0x7fecac109560 2024-03-24T00:15:45.755 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:45.756+0000 7feca6a00640 1 --2- 172.21.15.118:0/2438878040 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7fec940a2890 0x7fec940a4d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:15:45.755 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:45.756+0000 7feca6a00640 1 -- 172.21.15.118:0/2438878040 <== mon.1 v2:172.21.15.193:3300/0 5 ==== osd_map(90..90 src has 1..90) v4 ==== 8277+0+0 (secure 0 0 0) 0x7feca80f11e0 con 0x7fecac109560 2024-03-24T00:15:45.756 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:45.756+0000 7fecb1e00640 1 --2- 172.21.15.118:0/2438878040 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7fec940a2890 0x7fec940a4d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:15:45.756 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:45.757+0000 7fecb1e00640 1 --2- 172.21.15.118:0/2438878040 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7fec940a2890 0x7fec940a4d50 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7feca0009990 tx=0x7feca0009110 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:15:45.759 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:45.760+0000 7feca6a00640 1 -- 172.21.15.118:0/2438878040 <== mon.1 v2:172.21.15.193:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7feca80be1a0 con 0x7fecac109560 2024-03-24T00:15:45.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:15:45 smithi118 ceph-mon[31535]: from='client.25325 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-24T00:15:45.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:15:45 smithi118 ceph-mon[31535]: from='client.25342 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-24T00:15:45.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:15:45 smithi118 ceph-mon[31535]: from='client.? 172.21.15.118:0/3207048194' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-24T00:15:45.973 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:45.974+0000 7fecb4800640 1 -- 172.21.15.118:0/2438878040 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fecac0622a0 con 0x7fecac109560 2024-03-24T00:15:45.974 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:45.975+0000 7feca6a00640 1 -- 172.21.15.118:0/2438878040 <== mon.1 v2:172.21.15.193:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v90) v1 ==== 93+0+10 (secure 0 0 0) 0x7feca80bdaa0 con 0x7fecac109560 2024-03-24T00:15:45.974 INFO:teuthology.orchestra.run.smithi118.stdout:pg_num: 1 2024-03-24T00:15:45.977 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:45.977+0000 7fecb4800640 1 -- 172.21.15.118:0/2438878040 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7fec940a2890 msgr2=0x7fec940a4d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:15:45.977 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:45.977+0000 7fecb4800640 1 --2- 172.21.15.118:0/2438878040 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7fec940a2890 0x7fec940a4d50 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7feca0009990 tx=0x7feca0009110 comp rx=0 tx=0).stop 2024-03-24T00:15:45.977 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:45.977+0000 7fecb4800640 1 -- 172.21.15.118:0/2438878040 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fecac109560 msgr2=0x7fecac1a1190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:15:45.977 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:45.977+0000 7fecb4800640 1 --2- 172.21.15.118:0/2438878040 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fecac109560 0x7fecac1a1190 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7feca802c6d0 tx=0x7feca8003ba0 comp rx=0 tx=0).stop 2024-03-24T00:15:45.977 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:45.977+0000 7fecb4800640 1 -- 172.21.15.118:0/2438878040 shutdown_connections 2024-03-24T00:15:45.977 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:45.977+0000 7fecb4800640 1 --2- 172.21.15.118:0/2438878040 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7fec940a2890 0x7fec940a4d50 unknown :-1 s=CLOSED pgs=264 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:15:45.977 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:45.977+0000 7fecb4800640 1 --2- 172.21.15.118:0/2438878040 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fecac109560 0x7fecac1a1190 unknown :-1 s=CLOSED pgs=194 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:15:45.977 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:45.977+0000 7fecb4800640 1 --2- 172.21.15.118:0/2438878040 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fecac075660 0x7fecac19a120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:15:45.978 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:45.977+0000 7fecb4800640 1 --2- 172.21.15.118:0/2438878040 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fecac074080 0x7fecac199be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:15:45.978 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:45.977+0000 7fecb4800640 1 -- 172.21.15.118:0/2438878040 >> 172.21.15.118:0/2438878040 conn(0x7fecac0fcc40 msgr2=0x7fecac1054f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:15:45.978 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:45.978+0000 7fecb4800640 1 -- 172.21.15.118:0/2438878040 shutdown_connections 2024-03-24T00:15:45.978 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:45.978+0000 7fecb4800640 1 -- 172.21.15.118:0/2438878040 wait complete. 2024-03-24T00:15:45.994 DEBUG:teuthology.orchestra.run.smithi118:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-24T00:15:46.199 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:46.199+0000 7f243d800640 1 -- 172.21.15.118:0/1812619252 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2438103260 msgr2=0x7f2438109af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:15:46.200 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:46.199+0000 7f243d800640 1 --2- 172.21.15.118:0/1812619252 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2438103260 0x7f2438109af0 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7f2430009ca0 tx=0x7f243002c0f0 comp rx=0 tx=0).stop 2024-03-24T00:15:46.200 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:46.200+0000 7f243d800640 1 -- 172.21.15.118:0/1812619252 shutdown_connections 2024-03-24T00:15:46.200 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:46.200+0000 7f243d800640 1 --2- 172.21.15.118:0/1812619252 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2438103260 0x7f2438109af0 unknown :-1 s=CLOSED pgs=201 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:15:46.200 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:46.200+0000 7f243d800640 1 --2- 172.21.15.118:0/1812619252 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f24381028a0 0x7f2438102d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:15:46.200 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:46.200+0000 7f243d800640 1 --2- 172.21.15.118:0/1812619252 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f24381016b0 0x7f2438101ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:15:46.200 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:46.200+0000 7f243d800640 1 -- 172.21.15.118:0/1812619252 >> 172.21.15.118:0/1812619252 conn(0x7f24380fcc40 msgr2=0x7f24380ff060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:15:46.200 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:46.200+0000 7f243d800640 1 -- 172.21.15.118:0/1812619252 shutdown_connections 2024-03-24T00:15:46.200 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:46.200+0000 7f243d800640 1 -- 172.21.15.118:0/1812619252 wait complete. 2024-03-24T00:15:46.201 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:46.201+0000 7f243d800640 1 Processor -- start 2024-03-24T00:15:46.201 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:46.201+0000 7f243d800640 1 -- start start 2024-03-24T00:15:46.202 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:46.202+0000 7f243d800640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f24381016b0 0x7f2438199b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:15:46.202 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:46.202+0000 7f243d800640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f24381028a0 0x7f243819a080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:15:46.202 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:46.202+0000 7f243d800640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2438103260 0x7f24381a10f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:15:46.202 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:46.202+0000 7f243d800640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f243810d310 con 0x7f24381016b0 2024-03-24T00:15:46.202 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:46.202+0000 7f243d800640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f243810d190 con 0x7f2438103260 2024-03-24T00:15:46.203 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:46.202+0000 7f243d800640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f243810d490 con 0x7f24381028a0 2024-03-24T00:15:46.203 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:46.202+0000 7f2436000640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f24381028a0 0x7f243819a080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:15:46.203 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:46.202+0000 7f2436000640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f24381028a0 0x7f243819a080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.118:34024/0 (socket says 172.21.15.118:34024) 2024-03-24T00:15:46.203 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:46.202+0000 7f2436000640 1 -- 172.21.15.118:0/2938459653 learned_addr learned my addr 172.21.15.118:0/2938459653 (peer_addr_for_me v2:172.21.15.118:0/0) 2024-03-24T00:15:46.203 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:46.202+0000 7f2436a00640 1 --2- 172.21.15.118:0/2938459653 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f24381016b0 0x7f2438199b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:15:46.203 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:46.202+0000 7f2437400640 1 --2- 172.21.15.118:0/2938459653 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2438103260 0x7f24381a10f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:15:46.203 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:46.203+0000 7f2436000640 1 -- 172.21.15.118:0/2938459653 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2438103260 msgr2=0x7f24381a10f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:15:46.203 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:46.203+0000 7f2436000640 1 --2- 172.21.15.118:0/2938459653 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2438103260 0x7f24381a10f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:15:46.203 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:46.203+0000 7f2436000640 1 -- 172.21.15.118:0/2938459653 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f24381016b0 msgr2=0x7f2438199b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:15:46.203 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:46.203+0000 7f2436000640 1 --2- 172.21.15.118:0/2938459653 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f24381016b0 0x7f2438199b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:15:46.203 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:46.203+0000 7f2436000640 1 -- 172.21.15.118:0/2938459653 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f24381a1760 con 0x7f24381028a0 2024-03-24T00:15:46.203 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:46.203+0000 7f2437400640 1 --2- 172.21.15.118:0/2938459653 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2438103260 0x7f24381a10f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T00:15:46.203 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:46.203+0000 7f2436000640 1 --2- 172.21.15.118:0/2938459653 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f24381028a0 0x7f243819a080 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7f242400e9f0 tx=0x7f242400eeb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:15:46.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:15:45 smithi189 ceph-mon[36187]: from='client.25325 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-24T00:15:46.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:15:45 smithi189 ceph-mon[36187]: from='client.25342 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-24T00:15:46.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:15:45 smithi189 ceph-mon[36187]: from='client.? 172.21.15.118:0/3207048194' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-24T00:15:46.204 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:46.203+0000 7f242f400640 1 -- 172.21.15.118:0/2938459653 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f242400cd80 con 0x7f24381028a0 2024-03-24T00:15:46.205 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:46.203+0000 7f243d800640 1 -- 172.21.15.118:0/2938459653 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f24381a1a50 con 0x7f24381028a0 2024-03-24T00:15:46.205 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:46.203+0000 7f242f400640 1 -- 172.21.15.118:0/2938459653 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f2424010910 con 0x7f24381028a0 2024-03-24T00:15:46.205 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:46.203+0000 7f242f400640 1 -- 172.21.15.118:0/2938459653 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f24240188f0 con 0x7f24381028a0 2024-03-24T00:15:46.205 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:46.203+0000 7f243d800640 1 -- 172.21.15.118:0/2938459653 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f24381a1fa0 con 0x7f24381028a0 2024-03-24T00:15:46.205 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:46.204+0000 7f243d800640 1 -- 172.21.15.118:0/2938459653 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f24381a2230 con 0x7f24381028a0 2024-03-24T00:15:46.209 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:46.209+0000 7f242f400640 1 -- 172.21.15.118:0/2938459653 <== mon.1 v2:172.21.15.193:3300/0 4 ==== mgrmap(e 19) v1 ==== 137342+0+0 (secure 0 0 0) 0x7f2424010ac0 con 0x7f24381028a0 2024-03-24T00:15:46.210 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:46.211+0000 7f242f400640 1 --2- 172.21.15.118:0/2938459653 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7f241c0a2890 0x7f241c0a4d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:15:46.210 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:46.211+0000 7f242f400640 1 -- 172.21.15.118:0/2938459653 <== mon.1 v2:172.21.15.193:3300/0 5 ==== osd_map(90..90 src has 1..90) v4 ==== 8277+0+0 (secure 0 0 0) 0x7f24240ceff0 con 0x7f24381028a0 2024-03-24T00:15:46.210 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:46.211+0000 7f2436a00640 1 --2- 172.21.15.118:0/2938459653 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7f241c0a2890 0x7f241c0a4d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:15:46.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:15:45 smithi193 ceph-mon[36878]: from='client.25325 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-24T00:15:46.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:15:45 smithi193 ceph-mon[36878]: from='client.25342 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-24T00:15:46.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:15:45 smithi193 ceph-mon[36878]: from='client.? 172.21.15.118:0/3207048194' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-24T00:15:46.211 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:46.211+0000 7f2436a00640 1 --2- 172.21.15.118:0/2938459653 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7f241c0a2890 0x7f241c0a4d50 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7f2438102700 tx=0x7f2428005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:15:46.215 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:46.216+0000 7f242f400640 1 -- 172.21.15.118:0/2938459653 <== mon.1 v2:172.21.15.193:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f242409b890 con 0x7f24381028a0 2024-03-24T00:15:46.402 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:46.402+0000 7f243d800640 1 -- 172.21.15.118:0/2938459653 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f24381a2400 con 0x7f24381028a0 2024-03-24T00:15:46.402 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:46.403+0000 7f242f400640 1 -- 172.21.15.118:0/2938459653 <== mon.1 v2:172.21.15.193:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v90) v1 ==== 104+0+11 (secure 0 0 0) 0x7f2424084e20 con 0x7f24381028a0 2024-03-24T00:15:46.403 INFO:teuthology.orchestra.run.smithi118.stdout:pg_num: 32 2024-03-24T00:15:46.405 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:46.405+0000 7f243d800640 1 -- 172.21.15.118:0/2938459653 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7f241c0a2890 msgr2=0x7f241c0a4d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:15:46.405 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:46.405+0000 7f243d800640 1 --2- 172.21.15.118:0/2938459653 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7f241c0a2890 0x7f241c0a4d50 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7f2438102700 tx=0x7f2428005f90 comp rx=0 tx=0).stop 2024-03-24T00:15:46.405 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:46.405+0000 7f243d800640 1 -- 172.21.15.118:0/2938459653 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f24381028a0 msgr2=0x7f243819a080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:15:46.405 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:46.405+0000 7f243d800640 1 --2- 172.21.15.118:0/2938459653 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f24381028a0 0x7f243819a080 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7f242400e9f0 tx=0x7f242400eeb0 comp rx=0 tx=0).stop 2024-03-24T00:15:46.406 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:46.405+0000 7f243d800640 1 -- 172.21.15.118:0/2938459653 shutdown_connections 2024-03-24T00:15:46.406 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:46.405+0000 7f243d800640 1 --2- 172.21.15.118:0/2938459653 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7f241c0a2890 0x7f241c0a4d50 unknown :-1 s=CLOSED pgs=265 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:15:46.406 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:46.405+0000 7f243d800640 1 --2- 172.21.15.118:0/2938459653 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2438103260 0x7f24381a10f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:15:46.406 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:46.405+0000 7f243d800640 1 --2- 172.21.15.118:0/2938459653 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f24381028a0 0x7f243819a080 unknown :-1 s=CLOSED pgs=195 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:15:46.406 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:46.405+0000 7f243d800640 1 --2- 172.21.15.118:0/2938459653 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f24381016b0 0x7f2438199b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:15:46.406 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:46.405+0000 7f243d800640 1 -- 172.21.15.118:0/2938459653 >> 172.21.15.118:0/2938459653 conn(0x7f24380fcc40 msgr2=0x7f24380738d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:15:46.406 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:46.406+0000 7f243d800640 1 -- 172.21.15.118:0/2938459653 shutdown_connections 2024-03-24T00:15:46.406 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:46.406+0000 7f243d800640 1 -- 172.21.15.118:0/2938459653 wait complete. 2024-03-24T00:15:46.422 DEBUG:teuthology.orchestra.run.smithi118:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-24T00:15:46.623 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:46.623+0000 7f7b91800640 1 -- 172.21.15.118:0/2421793792 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f7b8c107bc0 msgr2=0x7f7b8c107fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:15:46.623 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:46.623+0000 7f7b91800640 1 --2- 172.21.15.118:0/2421793792 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f7b8c107bc0 0x7f7b8c107fe0 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7f7b7c009900 tx=0x7f7b7c02bc40 comp rx=0 tx=0).stop 2024-03-24T00:15:46.623 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:46.623+0000 7f7b91800640 1 -- 172.21.15.118:0/2421793792 shutdown_connections 2024-03-24T00:15:46.623 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:46.623+0000 7f7b91800640 1 --2- 172.21.15.118:0/2421793792 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f7b8c109770 0x7f7b8c110000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:15:46.623 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:46.623+0000 7f7b91800640 1 --2- 172.21.15.118:0/2421793792 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7b8c108db0 0x7f7b8c109230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:15:46.624 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:46.623+0000 7f7b91800640 1 --2- 172.21.15.118:0/2421793792 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f7b8c107bc0 0x7f7b8c107fe0 unknown :-1 s=CLOSED pgs=196 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:15:46.624 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:46.623+0000 7f7b91800640 1 -- 172.21.15.118:0/2421793792 >> 172.21.15.118:0/2421793792 conn(0x7f7b8c076b70 msgr2=0x7f7b8c078f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:15:46.624 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:46.623+0000 7f7b91800640 1 -- 172.21.15.118:0/2421793792 shutdown_connections 2024-03-24T00:15:46.624 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:46.623+0000 7f7b91800640 1 -- 172.21.15.118:0/2421793792 wait complete. 2024-03-24T00:15:46.626 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:46.625+0000 7f7b91800640 1 Processor -- start 2024-03-24T00:15:46.626 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:46.625+0000 7f7b91800640 1 -- start start 2024-03-24T00:15:46.626 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:46.625+0000 7f7b91800640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7b8c107bc0 0x7f7b8c19ded0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:15:46.626 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:46.625+0000 7f7b91800640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f7b8c108db0 0x7f7b8c19e410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:15:46.626 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:46.625+0000 7f7b91800640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f7b8c109770 0x7f7b8c1a5480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:15:46.626 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:46.625+0000 7f7b91800640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7b8c113860 con 0x7f7b8c109770 2024-03-24T00:15:46.626 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:46.625+0000 7f7b91800640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7b8c1136e0 con 0x7f7b8c107bc0 2024-03-24T00:15:46.626 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:46.625+0000 7f7b91800640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7b8c1139e0 con 0x7f7b8c108db0 2024-03-24T00:15:46.626 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:46.625+0000 7f7b8be00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7b8c107bc0 0x7f7b8c19ded0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:15:46.626 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:46.625+0000 7f7b8be00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7b8c107bc0 0x7f7b8c19ded0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.118:36412/0 (socket says 172.21.15.118:36412) 2024-03-24T00:15:46.626 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:46.625+0000 7f7b8be00640 1 -- 172.21.15.118:0/2478782400 learned_addr learned my addr 172.21.15.118:0/2478782400 (peer_addr_for_me v2:172.21.15.118:0/0) 2024-03-24T00:15:46.626 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:46.625+0000 7f7b83e00640 1 --2- 172.21.15.118:0/2478782400 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f7b8c108db0 0x7f7b8c19e410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:15:46.626 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:46.625+0000 7f7b90e00640 1 --2- 172.21.15.118:0/2478782400 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f7b8c109770 0x7f7b8c1a5480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:15:46.627 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:46.626+0000 7f7b8be00640 1 -- 172.21.15.118:0/2478782400 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f7b8c108db0 msgr2=0x7f7b8c19e410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:15:46.627 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:46.626+0000 7f7b8be00640 1 --2- 172.21.15.118:0/2478782400 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f7b8c108db0 0x7f7b8c19e410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:15:46.627 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:46.626+0000 7f7b8be00640 1 -- 172.21.15.118:0/2478782400 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f7b8c109770 msgr2=0x7f7b8c1a5480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:15:46.627 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:46.626+0000 7f7b8be00640 1 --2- 172.21.15.118:0/2478782400 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f7b8c109770 0x7f7b8c1a5480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:15:46.627 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:46.626+0000 7f7b8be00640 1 -- 172.21.15.118:0/2478782400 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7b8c1a5af0 con 0x7f7b8c107bc0 2024-03-24T00:15:46.627 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:46.626+0000 7f7b83e00640 1 --2- 172.21.15.118:0/2478782400 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f7b8c108db0 0x7f7b8c19e410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T00:15:46.627 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:46.626+0000 7f7b8be00640 1 --2- 172.21.15.118:0/2478782400 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7b8c107bc0 0x7f7b8c19ded0 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7f7b7c02efd0 tx=0x7f7b7c005d50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:15:46.628 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:46.627+0000 7f7b89600640 1 -- 172.21.15.118:0/2478782400 <== mon.2 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7b7c041070 con 0x7f7b8c107bc0 2024-03-24T00:15:46.628 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:46.627+0000 7f7b89600640 1 -- 172.21.15.118:0/2478782400 <== mon.2 v2:172.21.15.189:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f7b7c0044d0 con 0x7f7b8c107bc0 2024-03-24T00:15:46.628 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:46.627+0000 7f7b91800640 1 -- 172.21.15.118:0/2478782400 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7b8c1a5d80 con 0x7f7b8c107bc0 2024-03-24T00:15:46.628 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:46.627+0000 7f7b89600640 1 -- 172.21.15.118:0/2478782400 <== mon.2 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7b7c034440 con 0x7f7b8c107bc0 2024-03-24T00:15:46.628 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:46.627+0000 7f7b91800640 1 -- 172.21.15.118:0/2478782400 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7b8c1a61d0 con 0x7f7b8c107bc0 2024-03-24T00:15:46.628 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:46.628+0000 7f7b91800640 1 -- 172.21.15.118:0/2478782400 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7b8c10eee0 con 0x7f7b8c107bc0 2024-03-24T00:15:46.629 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:46.630+0000 7f7b89600640 1 -- 172.21.15.118:0/2478782400 <== mon.2 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 19) v1 ==== 137342+0+0 (secure 0 0 0) 0x7f7b7c044070 con 0x7f7b8c107bc0 2024-03-24T00:15:46.632 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:46.632+0000 7f7b89600640 1 --2- 172.21.15.118:0/2478782400 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7f7b700a2910 0x7f7b700a4dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:15:46.632 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:46.633+0000 7f7b89600640 1 -- 172.21.15.118:0/2478782400 <== mon.2 v2:172.21.15.189:3300/0 5 ==== osd_map(90..90 src has 1..90) v4 ==== 8277+0+0 (secure 0 0 0) 0x7f7b7c0f0990 con 0x7f7b8c107bc0 2024-03-24T00:15:46.632 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:46.633+0000 7f7b83e00640 1 --2- 172.21.15.118:0/2478782400 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7f7b700a2910 0x7f7b700a4dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:15:46.632 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:46.633+0000 7f7b83e00640 1 --2- 172.21.15.118:0/2478782400 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7f7b700a2910 0x7f7b700a4dd0 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7f7b8c19f3e0 tx=0x7f7b74005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:15:46.633 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:46.633+0000 7f7b89600640 1 -- 172.21.15.118:0/2478782400 <== mon.2 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f7b7c0bd300 con 0x7f7b8c107bc0 2024-03-24T00:15:46.862 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:46.862+0000 7f7b91800640 1 -- 172.21.15.118:0/2478782400 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f7b8c0622a0 con 0x7f7b8c107bc0 2024-03-24T00:15:46.862 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:46.863+0000 7f7b89600640 1 -- 172.21.15.118:0/2478782400 <== mon.2 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v90) v1 ==== 100+0+11 (secure 0 0 0) 0x7f7b7c0c13a0 con 0x7f7b8c107bc0 2024-03-24T00:15:46.863 INFO:teuthology.orchestra.run.smithi118.stdout:pg_num: 64 2024-03-24T00:15:46.865 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:46.865+0000 7f7b91800640 1 -- 172.21.15.118:0/2478782400 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7f7b700a2910 msgr2=0x7f7b700a4dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:15:46.865 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:46.865+0000 7f7b91800640 1 --2- 172.21.15.118:0/2478782400 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7f7b700a2910 0x7f7b700a4dd0 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7f7b8c19f3e0 tx=0x7f7b74005f90 comp rx=0 tx=0).stop 2024-03-24T00:15:46.865 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:46.865+0000 7f7b91800640 1 -- 172.21.15.118:0/2478782400 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7b8c107bc0 msgr2=0x7f7b8c19ded0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:15:46.865 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:46.865+0000 7f7b91800640 1 --2- 172.21.15.118:0/2478782400 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7b8c107bc0 0x7f7b8c19ded0 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7f7b7c02efd0 tx=0x7f7b7c005d50 comp rx=0 tx=0).stop 2024-03-24T00:15:46.865 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:46.865+0000 7f7b91800640 1 -- 172.21.15.118:0/2478782400 shutdown_connections 2024-03-24T00:15:46.866 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:46.865+0000 7f7b91800640 1 --2- 172.21.15.118:0/2478782400 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7f7b700a2910 0x7f7b700a4dd0 unknown :-1 s=CLOSED pgs=266 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:15:46.866 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:46.865+0000 7f7b91800640 1 --2- 172.21.15.118:0/2478782400 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f7b8c109770 0x7f7b8c1a5480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:15:46.866 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:46.865+0000 7f7b91800640 1 --2- 172.21.15.118:0/2478782400 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f7b8c108db0 0x7f7b8c19e410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:15:46.866 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:46.865+0000 7f7b91800640 1 --2- 172.21.15.118:0/2478782400 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7b8c107bc0 0x7f7b8c19ded0 unknown :-1 s=CLOSED pgs=202 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:15:46.866 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:46.866+0000 7f7b91800640 1 -- 172.21.15.118:0/2478782400 >> 172.21.15.118:0/2478782400 conn(0x7f7b8c076b70 msgr2=0x7f7b8c10c0b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:15:46.866 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:46.866+0000 7f7b91800640 1 -- 172.21.15.118:0/2478782400 shutdown_connections 2024-03-24T00:15:46.866 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:46.866+0000 7f7b91800640 1 -- 172.21.15.118:0/2478782400 wait complete. 2024-03-24T00:15:46.885 DEBUG:teuthology.orchestra.run.smithi118:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-24T00:15:47.076 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:47.076+0000 7f95cb200640 1 -- 172.21.15.118:0/4211743814 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f95c4103090 msgr2=0x7f95c4109920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:15:47.076 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:47.076+0000 7f95cb200640 1 --2- 172.21.15.118:0/4211743814 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f95c4103090 0x7f95c4109920 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7f95c0009be0 tx=0x7f95c002bef0 comp rx=0 tx=0).stop 2024-03-24T00:15:47.076 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:47.076+0000 7f95cb200640 1 -- 172.21.15.118:0/4211743814 shutdown_connections 2024-03-24T00:15:47.076 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:47.076+0000 7f95cb200640 1 --2- 172.21.15.118:0/4211743814 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f95c4103090 0x7f95c4109920 unknown :-1 s=CLOSED pgs=197 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:15:47.076 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:47.076+0000 7f95cb200640 1 --2- 172.21.15.118:0/4211743814 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f95c41026d0 0x7f95c4102b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:15:47.076 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:47.076+0000 7f95cb200640 1 --2- 172.21.15.118:0/4211743814 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f95c41014e0 0x7f95c4101900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:15:47.077 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:47.076+0000 7f95cb200640 1 -- 172.21.15.118:0/4211743814 >> 172.21.15.118:0/4211743814 conn(0x7f95c40fca30 msgr2=0x7f95c40fee90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:15:47.077 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:47.076+0000 7f95cb200640 1 -- 172.21.15.118:0/4211743814 shutdown_connections 2024-03-24T00:15:47.077 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:47.076+0000 7f95cb200640 1 -- 172.21.15.118:0/4211743814 wait complete. 2024-03-24T00:15:47.079 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:47.077+0000 7f95cb200640 1 Processor -- start 2024-03-24T00:15:47.079 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:47.077+0000 7f95cb200640 1 -- start start 2024-03-24T00:15:47.079 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:47.077+0000 7f95cb200640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f95c41014e0 0x7f95c41999b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:15:47.079 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:47.077+0000 7f95cb200640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f95c41026d0 0x7f95c4199ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:15:47.079 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:47.078+0000 7f95cb200640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f95c4103090 0x7f95c41a0f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:15:47.079 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:47.078+0000 7f95cb200640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f95c410d1c0 con 0x7f95c4103090 2024-03-24T00:15:47.079 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:47.078+0000 7f95cb200640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f95c410d040 con 0x7f95c41014e0 2024-03-24T00:15:47.079 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:47.078+0000 7f95cb200640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f95c410d340 con 0x7f95c41026d0 2024-03-24T00:15:47.079 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:47.078+0000 7f95c9e00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f95c41014e0 0x7f95c41999b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:15:47.079 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:47.078+0000 7f95ca800640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f95c4103090 0x7f95c41a0f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:15:47.079 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:47.078+0000 7f95c9e00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f95c41014e0 0x7f95c41999b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.118:36420/0 (socket says 172.21.15.118:36420) 2024-03-24T00:15:47.080 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:47.078+0000 7f95c9e00640 1 -- 172.21.15.118:0/3201303141 learned_addr learned my addr 172.21.15.118:0/3201303141 (peer_addr_for_me v2:172.21.15.118:0/0) 2024-03-24T00:15:47.080 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:47.078+0000 7f95c9400640 1 --2- 172.21.15.118:0/3201303141 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f95c41026d0 0x7f95c4199ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:15:47.080 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:47.078+0000 7f95c9e00640 1 -- 172.21.15.118:0/3201303141 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f95c41026d0 msgr2=0x7f95c4199ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:15:47.080 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:47.078+0000 7f95c9e00640 1 --2- 172.21.15.118:0/3201303141 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f95c41026d0 0x7f95c4199ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:15:47.080 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:47.078+0000 7f95c9e00640 1 -- 172.21.15.118:0/3201303141 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f95c4103090 msgr2=0x7f95c41a0f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:15:47.080 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:47.078+0000 7f95c9e00640 1 --2- 172.21.15.118:0/3201303141 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f95c4103090 0x7f95c41a0f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:15:47.080 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:47.078+0000 7f95c9e00640 1 -- 172.21.15.118:0/3201303141 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f95c41a15d0 con 0x7f95c41014e0 2024-03-24T00:15:47.080 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:47.078+0000 7f95c9e00640 1 --2- 172.21.15.118:0/3201303141 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f95c41014e0 0x7f95c41999b0 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7f95b800e9a0 tx=0x7f95b800ee60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:15:47.080 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:47.079+0000 7f95bea00640 1 -- 172.21.15.118:0/3201303141 <== mon.2 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f95b800cd80 con 0x7f95c41014e0 2024-03-24T00:15:47.080 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:47.079+0000 7f95bea00640 1 -- 172.21.15.118:0/3201303141 <== mon.2 v2:172.21.15.189:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f95b8010910 con 0x7f95c41014e0 2024-03-24T00:15:47.080 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:47.079+0000 7f95bea00640 1 -- 172.21.15.118:0/3201303141 <== mon.2 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f95b8018a30 con 0x7f95c41014e0 2024-03-24T00:15:47.080 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:47.079+0000 7f95cb200640 1 -- 172.21.15.118:0/3201303141 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f95c41a18c0 con 0x7f95c41014e0 2024-03-24T00:15:47.080 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:47.079+0000 7f95cb200640 1 -- 172.21.15.118:0/3201303141 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f95c41a1dc0 con 0x7f95c41014e0 2024-03-24T00:15:47.081 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:47.080+0000 7f95cb200640 1 -- 172.21.15.118:0/3201303141 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f95c41a2020 con 0x7f95c41014e0 2024-03-24T00:15:47.081 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:47.081+0000 7f95bea00640 1 -- 172.21.15.118:0/3201303141 <== mon.2 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 19) v1 ==== 137342+0+0 (secure 0 0 0) 0x7f95b8010ac0 con 0x7f95c41014e0 2024-03-24T00:15:47.082 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:47.083+0000 7f95bea00640 1 --2- 172.21.15.118:0/3201303141 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7f95ac0a2910 0x7f95ac0a4dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:15:47.082 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:47.083+0000 7f95bea00640 1 -- 172.21.15.118:0/3201303141 <== mon.2 v2:172.21.15.189:3300/0 5 ==== osd_map(90..90 src has 1..90) v4 ==== 8277+0+0 (secure 0 0 0) 0x7f95b80cef60 con 0x7f95c41014e0 2024-03-24T00:15:47.082 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:47.083+0000 7f95c9400640 1 --2- 172.21.15.118:0/3201303141 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7f95ac0a2910 0x7f95ac0a4dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:15:47.083 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:47.083+0000 7f95c9400640 1 --2- 172.21.15.118:0/3201303141 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7f95ac0a2910 0x7f95ac0a4dd0 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7f95c419aec0 tx=0x7f95b0005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:15:47.086 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:47.087+0000 7f95bea00640 1 -- 172.21.15.118:0/3201303141 <== mon.2 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f95b8053740 con 0x7f95c41014e0 2024-03-24T00:15:47.195 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:15:46 smithi118 ceph-mon[31535]: from='client.25337 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-24T00:15:47.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:15:46 smithi118 ceph-mon[31535]: pgmap v400: 97 pgs: 97 active+clean; 654 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 1.5 KiB/s wr, 12 op/s 2024-03-24T00:15:47.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:15:46 smithi118 ceph-mon[31535]: from='client.? 172.21.15.118:0/2438878040' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-24T00:15:47.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:15:46 smithi118 ceph-mon[31535]: from='client.? 172.21.15.118:0/2938459653' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-24T00:15:47.203 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:15:46 smithi189 ceph-mon[36187]: from='client.25337 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-24T00:15:47.203 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:15:46 smithi189 ceph-mon[36187]: pgmap v400: 97 pgs: 97 active+clean; 654 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 1.5 KiB/s wr, 12 op/s 2024-03-24T00:15:47.203 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:15:46 smithi189 ceph-mon[36187]: from='client.? 172.21.15.118:0/2438878040' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-24T00:15:47.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:15:46 smithi189 ceph-mon[36187]: from='client.? 172.21.15.118:0/2938459653' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-24T00:15:47.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:15:46 smithi193 ceph-mon[36878]: from='client.25337 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-24T00:15:47.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:15:46 smithi193 ceph-mon[36878]: pgmap v400: 97 pgs: 97 active+clean; 654 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 1.5 KiB/s wr, 12 op/s 2024-03-24T00:15:47.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:15:46 smithi193 ceph-mon[36878]: from='client.? 172.21.15.118:0/2438878040' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-24T00:15:47.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:15:46 smithi193 ceph-mon[36878]: from='client.? 172.21.15.118:0/2938459653' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-24T00:15:47.364 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:47.364+0000 7f95cb200640 1 -- 172.21.15.118:0/3201303141 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f95c41a2310 con 0x7f95c41014e0 2024-03-24T00:15:47.364 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:47.365+0000 7f95bea00640 1 -- 172.21.15.118:0/3201303141 <== mon.2 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+11478 (secure 0 0 0) 0x7f95b809be10 con 0x7f95c41014e0 2024-03-24T00:15:47.365 INFO:teuthology.orchestra.run.smithi118.stdout: 2024-03-24T00:15:47.365 INFO:teuthology.orchestra.run.smithi118.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14385,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.118:6835/3194741373","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6834","nonce":3194741373},{"type":"v1","addr":"172.21.15.118:6835","nonce":3194741373}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":4},{"gid":14397,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.118:6837/1374462956","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6836","nonce":1374462956},{"type":"v1","addr":"172.21.15.118:6837","nonce":1374462956}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":28,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-03-24T00:13:01.782474+0000","modified":"2024-03-24T00:15:38.235450+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24569,"mds_1":24550,"mds_2":24511,"mds_3":24499,"mds_4":24488},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14469":{"gid":14469,"name":"a","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.118:6841/3040946537","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6840","nonce":3040946537},{"type":"v1","addr":"172.21.15.118:6841","nonce":3040946537}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24439":{"gid":24439,"name":"l","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.193:6833/1023428412","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6832","nonce":1023428412},{"type":"v1","addr":"172.21.15.193:6833","nonce":1023428412}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24440":{"gid":24440,"name":"k","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.189:6833/2740024078","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6832","nonce":2740024078},{"type":"v1","addr":"172.21.15.189:6833","nonce":2740024078}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24488":{"gid":24488,"name":"e","rank":4,"incarnation":25,"state":"up:active","state_seq":10,"addr":"172.21.15.189:6835/599473205","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6834","nonce":599473205},{"type":"v1","addr":"172.21.15.189:6835","nonce":599473205}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24499":{"gid":24499,"name":"f","rank":3,"incarnation":23,"state":"up:active","state_seq":9,"addr":"172.21.15.193:6835/2055310541","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6834","nonce":2055310541},{"type":"v1","addr":"172.21.15.193:6835","nonce":2055310541}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24511":{"gid":24511,"name":"c","rank":2,"incarnation":21,"state":"up:active","state_seq":8,"addr":"172.21.15.193:6837/2921520053","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6836","nonce":2921520053},{"type":"v1","addr":"172.21.15.193:6837","nonce":2921520053}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24515":{"gid":24515,"name":"h","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.189:6837/3450187137","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6836","nonce":3450187137},{"type":"v1","addr":"172.21.15.189:6837","nonce":3450187137}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24532":{"gid":24532,"name":"i","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.193:6839/2126842603","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6838","nonce":2126842603},{"type":"v1","addr":"172.21.15.193:6839","nonce":2126842603}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24550":{"gid":24550,"name":"g","rank":1,"incarnation":19,"state":"up:active","state_seq":6,"addr":"172.21.15.118:6839/3285788503","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6838","nonce":3285788503},{"type":"v1","addr":"172.21.15.118:6839","nonce":3285788503}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24569":{"gid":24569,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.189:6839/4074484168","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6838","nonce":4074484168},{"type":"v1","addr":"172.21.15.189:6839","nonce":4074484168}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24569,"qdb_cluster":[24569,24499,24488,24511,24550]},"id":1}]} 2024-03-24T00:15:47.365 INFO:teuthology.orchestra.run.smithi118.stderr:dumped fsmap epoch 28 2024-03-24T00:15:47.367 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:47.367+0000 7f95cb200640 1 -- 172.21.15.118:0/3201303141 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7f95ac0a2910 msgr2=0x7f95ac0a4dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:15:47.367 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:47.367+0000 7f95cb200640 1 --2- 172.21.15.118:0/3201303141 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7f95ac0a2910 0x7f95ac0a4dd0 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7f95c419aec0 tx=0x7f95b0005f90 comp rx=0 tx=0).stop 2024-03-24T00:15:47.367 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:47.367+0000 7f95cb200640 1 -- 172.21.15.118:0/3201303141 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f95c41014e0 msgr2=0x7f95c41999b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:15:47.367 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:47.367+0000 7f95cb200640 1 --2- 172.21.15.118:0/3201303141 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f95c41014e0 0x7f95c41999b0 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7f95b800e9a0 tx=0x7f95b800ee60 comp rx=0 tx=0).stop 2024-03-24T00:15:47.368 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:47.367+0000 7f95cb200640 1 -- 172.21.15.118:0/3201303141 shutdown_connections 2024-03-24T00:15:47.368 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:47.367+0000 7f95cb200640 1 --2- 172.21.15.118:0/3201303141 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7f95ac0a2910 0x7f95ac0a4dd0 unknown :-1 s=CLOSED pgs=267 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:15:47.368 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:47.367+0000 7f95cb200640 1 --2- 172.21.15.118:0/3201303141 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f95c4103090 0x7f95c41a0f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:15:47.368 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:47.367+0000 7f95cb200640 1 --2- 172.21.15.118:0/3201303141 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f95c41026d0 0x7f95c4199ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:15:47.368 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:47.367+0000 7f95cb200640 1 --2- 172.21.15.118:0/3201303141 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f95c41014e0 0x7f95c41999b0 unknown :-1 s=CLOSED pgs=203 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:15:47.368 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:47.367+0000 7f95cb200640 1 -- 172.21.15.118:0/3201303141 >> 172.21.15.118:0/3201303141 conn(0x7f95c40fca30 msgr2=0x7f95c40738d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:15:47.368 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:47.367+0000 7f95cb200640 1 -- 172.21.15.118:0/3201303141 shutdown_connections 2024-03-24T00:15:47.368 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:47.368+0000 7f95cb200640 1 -- 172.21.15.118:0/3201303141 wait complete. 2024-03-24T00:15:47.385 DEBUG:teuthology.orchestra.run.smithi118:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-24T00:15:47.602 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:47.602+0000 7f6125000640 1 -- 172.21.15.118:0/4190775149 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f61201093d0 msgr2=0x7f612010b860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:15:47.603 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:47.602+0000 7f6125000640 1 --2- 172.21.15.118:0/4190775149 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f61201093d0 0x7f612010b860 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7f6118009ca0 tx=0x7f611802c100 comp rx=0 tx=0).stop 2024-03-24T00:15:47.603 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:47.603+0000 7f6125000640 1 -- 172.21.15.118:0/4190775149 shutdown_connections 2024-03-24T00:15:47.603 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:47.603+0000 7f6125000640 1 --2- 172.21.15.118:0/4190775149 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f61201093d0 0x7f612010b860 unknown :-1 s=CLOSED pgs=204 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:15:47.603 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:47.603+0000 7f6125000640 1 --2- 172.21.15.118:0/4190775149 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f6120074100 0x7f6120074580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:15:47.603 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:47.603+0000 7f6125000640 1 --2- 172.21.15.118:0/4190775149 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f61200756e0 0x7f6120073b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:15:47.603 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:47.603+0000 7f6125000640 1 -- 172.21.15.118:0/4190775149 >> 172.21.15.118:0/4190775149 conn(0x7f61200fca80 msgr2=0x7f61200feec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:15:47.603 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:47.603+0000 7f6125000640 1 -- 172.21.15.118:0/4190775149 shutdown_connections 2024-03-24T00:15:47.603 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:47.603+0000 7f6125000640 1 -- 172.21.15.118:0/4190775149 wait complete. 2024-03-24T00:15:47.606 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:47.604+0000 7f6125000640 1 Processor -- start 2024-03-24T00:15:47.607 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:47.604+0000 7f6125000640 1 -- start start 2024-03-24T00:15:47.607 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:47.604+0000 7f6125000640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f6120074100 0x7f61201999a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:15:47.607 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:47.604+0000 7f6125000640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f61200756e0 0x7f6120199ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:15:47.607 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:47.604+0000 7f6125000640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f61201093d0 0x7f61201a0f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:15:47.607 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:47.604+0000 7f6125000640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f612010f350 con 0x7f6120074100 2024-03-24T00:15:47.607 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:47.604+0000 7f6125000640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f612010f1d0 con 0x7f61201093d0 2024-03-24T00:15:47.607 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:47.604+0000 7f6125000640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f612010f4d0 con 0x7f61200756e0 2024-03-24T00:15:47.607 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:47.604+0000 7f611f400640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f6120074100 0x7f61201999a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:15:47.607 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:47.605+0000 7f611f400640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f6120074100 0x7f61201999a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.118:3300/0 says I am v2:172.21.15.118:43320/0 (socket says 172.21.15.118:43320) 2024-03-24T00:15:47.607 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:47.605+0000 7f611f400640 1 -- 172.21.15.118:0/2948502676 learned_addr learned my addr 172.21.15.118:0/2948502676 (peer_addr_for_me v2:172.21.15.118:0/0) 2024-03-24T00:15:47.607 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:47.605+0000 7f611ea00640 1 --2- 172.21.15.118:0/2948502676 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f61200756e0 0x7f6120199ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:15:47.607 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:47.605+0000 7f611fe00640 1 --2- 172.21.15.118:0/2948502676 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f61201093d0 0x7f61201a0f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:15:47.607 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:47.605+0000 7f611f400640 1 -- 172.21.15.118:0/2948502676 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f61200756e0 msgr2=0x7f6120199ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:15:47.608 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:47.605+0000 7f611f400640 1 --2- 172.21.15.118:0/2948502676 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f61200756e0 0x7f6120199ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:15:47.608 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:47.605+0000 7f611f400640 1 -- 172.21.15.118:0/2948502676 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f61201093d0 msgr2=0x7f61201a0f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:15:47.608 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:47.605+0000 7f611f400640 1 --2- 172.21.15.118:0/2948502676 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f61201093d0 0x7f61201a0f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:15:47.608 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:47.605+0000 7f611f400640 1 -- 172.21.15.118:0/2948502676 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f61201a15c0 con 0x7f6120074100 2024-03-24T00:15:47.608 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:47.605+0000 7f611ea00640 1 --2- 172.21.15.118:0/2948502676 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f61200756e0 0x7f6120199ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T00:15:47.608 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:47.605+0000 7f611f400640 1 --2- 172.21.15.118:0/2948502676 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f6120074100 0x7f61201999a0 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7f611000d950 tx=0x7f611000de10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:15:47.608 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:47.606+0000 7f6117e00640 1 -- 172.21.15.118:0/2948502676 <== mon.0 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6110017070 con 0x7f6120074100 2024-03-24T00:15:47.608 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:47.606+0000 7f6117e00640 1 -- 172.21.15.118:0/2948502676 <== mon.0 v2:172.21.15.118:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f6110009c30 con 0x7f6120074100 2024-03-24T00:15:47.608 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:47.606+0000 7f6125000640 1 -- 172.21.15.118:0/2948502676 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f61201a18b0 con 0x7f6120074100 2024-03-24T00:15:47.608 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:47.606+0000 7f6117e00640 1 -- 172.21.15.118:0/2948502676 <== mon.0 v2:172.21.15.118:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f611000f6b0 con 0x7f6120074100 2024-03-24T00:15:47.608 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:47.606+0000 7f6125000640 1 -- 172.21.15.118:0/2948502676 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f61201a1e00 con 0x7f6120074100 2024-03-24T00:15:47.608 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:47.607+0000 7f6125000640 1 -- 172.21.15.118:0/2948502676 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6120108100 con 0x7f6120074100 2024-03-24T00:15:47.609 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:47.609+0000 7f6117e00640 1 -- 172.21.15.118:0/2948502676 <== mon.0 v2:172.21.15.118:3300/0 4 ==== mgrmap(e 19) v1 ==== 137342+0+0 (secure 0 0 0) 0x7f61100109d0 con 0x7f6120074100 2024-03-24T00:15:47.610 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:47.610+0000 7f6117e00640 1 --2- 172.21.15.118:0/2948502676 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7f61040a2890 0x7f61040a4d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:15:47.610 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:47.610+0000 7f6117e00640 1 -- 172.21.15.118:0/2948502676 <== mon.0 v2:172.21.15.118:3300/0 5 ==== osd_map(90..90 src has 1..90) v4 ==== 8277+0+0 (secure 0 0 0) 0x7f61100ce770 con 0x7f6120074100 2024-03-24T00:15:47.610 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:47.610+0000 7f611ea00640 1 --2- 172.21.15.118:0/2948502676 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7f61040a2890 0x7f61040a4d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:15:47.610 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:47.611+0000 7f611ea00640 1 --2- 172.21.15.118:0/2948502676 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7f61040a2890 0x7f61040a4d50 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7f612019aeb0 tx=0x7f610c005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:15:47.613 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:47.614+0000 7f6117e00640 1 -- 172.21.15.118:0/2948502676 <== mon.0 v2:172.21.15.118:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f6110009de0 con 0x7f6120074100 2024-03-24T00:15:47.923 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:47.923+0000 7f6125000640 1 -- 172.21.15.118:0/2948502676 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f6120062340 con 0x7f6120074100 2024-03-24T00:15:47.926 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:47.927+0000 7f6117e00640 1 -- 172.21.15.118:0/2948502676 <== mon.0 v2:172.21.15.118:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v90) v1 ==== 74+0+22834 (secure 0 0 0) 0x7f611009b030 con 0x7f6120074100 2024-03-24T00:15:47.927 INFO:teuthology.orchestra.run.smithi118.stdout: 2024-03-24T00:15:47.928 INFO:teuthology.orchestra.run.smithi118.stdout:{"epoch":90,"fsid":"d79a5420-e971-11ee-95cd-87774f69a715","created":"2024-03-24T00:03:58.648524+0000","modified":"2024-03-24T00:15:07.721960+0000","last_up_change":"2024-03-24T00:12:03.809000+0000","last_in_change":"2024-03-24T00:11:44.389344+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-24T00:07:50.190965+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-24T00:12:59.993723+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-24T00:13:01.257347+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"86","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"c9590426-3901-4f68-a5ef-d57aa8fbac92","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6802","nonce":557690061},{"type":"v1","addr":"172.21.15.118:6803","nonce":557690061}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6804","nonce":557690061},{"type":"v1","addr":"172.21.15.118:6805","nonce":557690061}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6808","nonce":557690061},{"type":"v1","addr":"172.21.15.118:6809","nonce":557690061}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6806","nonce":557690061},{"type":"v1","addr":"172.21.15.118:6807","nonce":557690061}]},"public_addr":"172.21.15.118:6803/557690061","cluster_addr":"172.21.15.118:6805/557690061","heartbeat_back_addr":"172.21.15.118:6809/557690061","heartbeat_front_addr":"172.21.15.118:6807/557690061","state":["exists","up"]},{"osd":1,"uuid":"fc7f827e-ecf6-4661-aad0-7a22951892e0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6800","nonce":4203029485},{"type":"v1","addr":"172.21.15.189:6801","nonce":4203029485}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6802","nonce":4203029485},{"type":"v1","addr":"172.21.15.189:6803","nonce":4203029485}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6806","nonce":4203029485},{"type":"v1","addr":"172.21.15.189:6807","nonce":4203029485}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6804","nonce":4203029485},{"type":"v1","addr":"172.21.15.189:6805","nonce":4203029485}]},"public_addr":"172.21.15.189:6801/4203029485","cluster_addr":"172.21.15.189:6803/4203029485","heartbeat_back_addr":"172.21.15.189:6807/4203029485","heartbeat_front_addr":"172.21.15.189:6805/4203029485","state":["exists","up"]},{"osd":2,"uuid":"e73ccfe9-0e5b-4734-a154-a2f0d829fcfb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6800","nonce":3699076971},{"type":"v1","addr":"172.21.15.193:6801","nonce":3699076971}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6802","nonce":3699076971},{"type":"v1","addr":"172.21.15.193:6803","nonce":3699076971}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6806","nonce":3699076971},{"type":"v1","addr":"172.21.15.193:6807","nonce":3699076971}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6804","nonce":3699076971},{"type":"v1","addr":"172.21.15.193:6805","nonce":3699076971}]},"public_addr":"172.21.15.193:6801/3699076971","cluster_addr":"172.21.15.193:6803/3699076971","heartbeat_back_addr":"172.21.15.193:6807/3699076971","heartbeat_front_addr":"172.21.15.193:6805/3699076971","state":["exists","up"]},{"osd":3,"uuid":"63fed43b-43ce-4031-8957-7fe94725c1ed","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6810","nonce":4059617010},{"type":"v1","addr":"172.21.15.118:6811","nonce":4059617010}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6812","nonce":4059617010},{"type":"v1","addr":"172.21.15.118:6813","nonce":4059617010}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6816","nonce":4059617010},{"type":"v1","addr":"172.21.15.118:6817","nonce":4059617010}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6814","nonce":4059617010},{"type":"v1","addr":"172.21.15.118:6815","nonce":4059617010}]},"public_addr":"172.21.15.118:6811/4059617010","cluster_addr":"172.21.15.118:6813/4059617010","heartbeat_back_addr":"172.21.15.118:6817/4059617010","heartbeat_front_addr":"172.21.15.118:6815/4059617010","state":["exists","up"]},{"osd":4,"uuid":"57377f66-c3b4-43c1-8e8e-bc873672906d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":34,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6808","nonce":4075247068},{"type":"v1","addr":"172.21.15.189:6809","nonce":4075247068}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6810","nonce":4075247068},{"type":"v1","addr":"172.21.15.189:6811","nonce":4075247068}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6814","nonce":4075247068},{"type":"v1","addr":"172.21.15.189:6815","nonce":4075247068}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6812","nonce":4075247068},{"type":"v1","addr":"172.21.15.189:6813","nonce":4075247068}]},"public_addr":"172.21.15.189:6809/4075247068","cluster_addr":"172.21.15.189:6811/4075247068","heartbeat_back_addr":"172.21.15.189:6815/4075247068","heartbeat_front_addr":"172.21.15.189:6813/4075247068","state":["exists","up"]},{"osd":5,"uuid":"91970976-2193-4224-81c0-dc96cb022c85","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":39,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6808","nonce":2920363402},{"type":"v1","addr":"172.21.15.193:6809","nonce":2920363402}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6810","nonce":2920363402},{"type":"v1","addr":"172.21.15.193:6811","nonce":2920363402}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6814","nonce":2920363402},{"type":"v1","addr":"172.21.15.193:6815","nonce":2920363402}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6812","nonce":2920363402},{"type":"v1","addr":"172.21.15.193:6813","nonce":2920363402}]},"public_addr":"172.21.15.193:6809/2920363402","cluster_addr":"172.21.15.193:6811/2920363402","heartbeat_back_addr":"172.21.15.193:6815/2920363402","heartbeat_front_addr":"172.21.15.193:6813/2920363402","state":["exists","up"]},{"osd":6,"uuid":"0b77b196-8674-41c6-ad3d-726b5d01c8fd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":47,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6818","nonce":431354576},{"type":"v1","addr":"172.21.15.118:6819","nonce":431354576}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6820","nonce":431354576},{"type":"v1","addr":"172.21.15.118:6821","nonce":431354576}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6824","nonce":431354576},{"type":"v1","addr":"172.21.15.118:6825","nonce":431354576}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6822","nonce":431354576},{"type":"v1","addr":"172.21.15.118:6823","nonce":431354576}]},"public_addr":"172.21.15.118:6819/431354576","cluster_addr":"172.21.15.118:6821/431354576","heartbeat_back_addr":"172.21.15.118:6825/431354576","heartbeat_front_addr":"172.21.15.118:6823/431354576","state":["exists","up"]},{"osd":7,"uuid":"1fbbd38f-4e8e-4d4c-8e98-66f6862bc4bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":52,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6816","nonce":3667611284},{"type":"v1","addr":"172.21.15.189:6817","nonce":3667611284}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6818","nonce":3667611284},{"type":"v1","addr":"172.21.15.189:6819","nonce":3667611284}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6822","nonce":3667611284},{"type":"v1","addr":"172.21.15.189:6823","nonce":3667611284}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6820","nonce":3667611284},{"type":"v1","addr":"172.21.15.189:6821","nonce":3667611284}]},"public_addr":"172.21.15.189:6817/3667611284","cluster_addr":"172.21.15.189:6819/3667611284","heartbeat_back_addr":"172.21.15.189:6823/3667611284","heartbeat_front_addr":"172.21.15.189:6821/3667611284","state":["exists","up"]},{"osd":8,"uuid":"6462c2b5-e4b3-4b0b-bcfa-4de2c89e93e0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":60,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6816","nonce":3853800910},{"type":"v1","addr":"172.21.15.193:6817","nonce":3853800910}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6818","nonce":3853800910},{"type":"v1","addr":"172.21.15.193:6819","nonce":3853800910}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6822","nonce":3853800910},{"type":"v1","addr":"172.21.15.193:6823","nonce":3853800910}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6820","nonce":3853800910},{"type":"v1","addr":"172.21.15.193:6821","nonce":3853800910}]},"public_addr":"172.21.15.193:6817/3853800910","cluster_addr":"172.21.15.193:6819/3853800910","heartbeat_back_addr":"172.21.15.193:6823/3853800910","heartbeat_front_addr":"172.21.15.193:6821/3853800910","state":["exists","up"]},{"osd":9,"uuid":"54be277f-3596-422f-9de5-9005a8571c6f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":68,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6826","nonce":3248293936},{"type":"v1","addr":"172.21.15.118:6827","nonce":3248293936}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6828","nonce":3248293936},{"type":"v1","addr":"172.21.15.118:6829","nonce":3248293936}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6832","nonce":3248293936},{"type":"v1","addr":"172.21.15.118:6833","nonce":3248293936}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6830","nonce":3248293936},{"type":"v1","addr":"172.21.15.118:6831","nonce":3248293936}]},"public_addr":"172.21.15.118:6827/3248293936","cluster_addr":"172.21.15.118:6829/3248293936","heartbeat_back_addr":"172.21.15.118:6833/3248293936","heartbeat_front_addr":"172.21.15.118:6831/3248293936","state":["exists","up"]},{"osd":10,"uuid":"7ba69d69-ac41-44ce-bc39-c049a8ca5900","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":73,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6824","nonce":2112195760},{"type":"v1","addr":"172.21.15.189:6825","nonce":2112195760}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6826","nonce":2112195760},{"type":"v1","addr":"172.21.15.189:6827","nonce":2112195760}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6830","nonce":2112195760},{"type":"v1","addr":"172.21.15.189:6831","nonce":2112195760}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6828","nonce":2112195760},{"type":"v1","addr":"172.21.15.189:6829","nonce":2112195760}]},"public_addr":"172.21.15.189:6825/2112195760","cluster_addr":"172.21.15.189:6827/2112195760","heartbeat_back_addr":"172.21.15.189:6831/2112195760","heartbeat_front_addr":"172.21.15.189:6829/2112195760","state":["exists","up"]},{"osd":11,"uuid":"186f0b18-6b22-4eae-83b7-b21bbcce0817","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":81,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6824","nonce":3880515510},{"type":"v1","addr":"172.21.15.193:6825","nonce":3880515510}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6826","nonce":3880515510},{"type":"v1","addr":"172.21.15.193:6827","nonce":3880515510}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6830","nonce":3880515510},{"type":"v1","addr":"172.21.15.193:6831","nonce":3880515510}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6828","nonce":3880515510},{"type":"v1","addr":"172.21.15.193:6829","nonce":3880515510}]},"public_addr":"172.21.15.193:6825/3880515510","cluster_addr":"172.21.15.193:6827/3880515510","heartbeat_back_addr":"172.21.15.193:6831/3880515510","heartbeat_front_addr":"172.21.15.193:6829/3880515510","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:06:58.105543+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:07:23.795866+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:07:47.426050+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:08:19.427073+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:08:45.816082+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:09:11.081523+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:09:44.485077+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:10:11.338972+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:10:36.234024+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:11:09.730432+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:11:37.059083+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:12:02.371947+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.a","mappings":[{"from":2,"to":11}]},{"pgid":"2.14","mappings":[{"from":2,"to":4}]},{"pgid":"2.1d","mappings":[{"from":8,"to":11}]},{"pgid":"2.1f","mappings":[{"from":8,"to":1}]},{"pgid":"3.11","mappings":[{"from":4,"to":8},{"from":5,"to":9}]},{"pgid":"3.13","mappings":[{"from":4,"to":9}]},{"pgid":"3.14","mappings":[{"from":4,"to":6}]},{"pgid":"3.31","mappings":[{"from":4,"to":7}]},{"pgid":"3.38","mappings":[{"from":3,"to":9}]},{"pgid":"3.39","mappings":[{"from":5,"to":8}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.118:6801/389249272":"2024-03-25T00:05:05.458796+0000","172.21.15.118:6800/389249272":"2024-03-25T00:05:05.458796+0000","172.21.15.118:0/2101854694":"2024-03-25T00:05:05.458796+0000","172.21.15.118:0/630779359":"2024-03-25T00:05:05.458796+0000","172.21.15.118:0/2158921029":"2024-03-25T00:04:31.283725+0000","172.21.15.118:0/1003670455":"2024-03-25T00:05:05.458796+0000","172.21.15.118:0/368712314":"2024-03-25T00:04:31.283725+0000","172.21.15.118:0/2004775597":"2024-03-25T00:04:31.283725+0000","172.21.15.118:6800/1463147955":"2024-03-25T00:04:31.283725+0000","172.21.15.118:6801/1463147955":"2024-03-25T00:04:31.283725+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-24T00:15:47.930 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:47.930+0000 7f6125000640 1 -- 172.21.15.118:0/2948502676 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7f61040a2890 msgr2=0x7f61040a4d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:15:47.930 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:47.930+0000 7f6125000640 1 --2- 172.21.15.118:0/2948502676 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7f61040a2890 0x7f61040a4d50 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7f612019aeb0 tx=0x7f610c005f90 comp rx=0 tx=0).stop 2024-03-24T00:15:47.930 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:47.930+0000 7f6125000640 1 -- 172.21.15.118:0/2948502676 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f6120074100 msgr2=0x7f61201999a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:15:47.930 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:47.930+0000 7f6125000640 1 --2- 172.21.15.118:0/2948502676 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f6120074100 0x7f61201999a0 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7f611000d950 tx=0x7f611000de10 comp rx=0 tx=0).stop 2024-03-24T00:15:47.930 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:47.930+0000 7f6125000640 1 -- 172.21.15.118:0/2948502676 shutdown_connections 2024-03-24T00:15:47.931 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:47.930+0000 7f6125000640 1 --2- 172.21.15.118:0/2948502676 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7f61040a2890 0x7f61040a4d50 unknown :-1 s=CLOSED pgs=268 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:15:47.931 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:47.930+0000 7f6125000640 1 --2- 172.21.15.118:0/2948502676 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f61201093d0 0x7f61201a0f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:15:47.931 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:47.930+0000 7f6125000640 1 --2- 172.21.15.118:0/2948502676 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f61200756e0 0x7f6120199ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:15:47.931 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:47.930+0000 7f6125000640 1 --2- 172.21.15.118:0/2948502676 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f6120074100 0x7f61201999a0 unknown :-1 s=CLOSED pgs=178 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:15:47.931 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:47.930+0000 7f6125000640 1 -- 172.21.15.118:0/2948502676 >> 172.21.15.118:0/2948502676 conn(0x7f61200fca80 msgr2=0x7f61201052d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:15:47.931 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:47.931+0000 7f6125000640 1 -- 172.21.15.118:0/2948502676 shutdown_connections 2024-03-24T00:15:47.931 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:47.931+0000 7f6125000640 1 -- 172.21.15.118:0/2948502676 wait complete. 2024-03-24T00:15:47.948 DEBUG:tasks.fwd_scrub:Yielding 2024-03-24T00:15:47.948 INFO:teuthology.run_tasks:Running task background_exec... 2024-03-24T00:15:47.958 INFO:teuthology.task.background_exec:Running background command on role mon.a host ubuntu@smithi118.front.sepia.ceph.com 2024-03-24T00:15:47.958 DEBUG:teuthology.orchestra.run.smithi118:> sudo TESTDIR=/home/ubuntu/cephtest daemon-helper kill --kill-group bash -c 'while sleep 13; do ceph tell mds.cephfs:0 flush journal; done' 2024-03-24T00:15:47.959 INFO:tasks.fwd_scrub.fs.[cephfs]:start scrubbing fs: cephfs 2024-03-24T00:15:47.959 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-03-24T00:15:47.959 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T00:15:47.959 DEBUG:teuthology.orchestra.run.smithi118:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2024-03-24T00:15:47.965 INFO:teuthology.run_tasks:Running task workunit... 2024-03-24T00:15:47.978 INFO:tasks.workunit:Pulling workunits from ref 3be9a5c9bc793e11e2800a8c0c696e8b46742033 2024-03-24T00:15:47.978 INFO:tasks.workunit:Making a separate scratch dir for every client... 2024-03-24T00:15:47.978 INFO:tasks.workunit:timeout=3h 2024-03-24T00:15:47.978 INFO:tasks.workunit:cleanup=True 2024-03-24T00:15:47.978 DEBUG:teuthology.orchestra.run.smithi118:> stat -- /home/ubuntu/cephtest/mnt.0 2024-03-24T00:15:48.020 INFO:teuthology.orchestra.run.smithi118.stdout: File: /home/ubuntu/cephtest/mnt.0 2024-03-24T00:15:48.020 INFO:teuthology.orchestra.run.smithi118.stdout: Size: 0 Blocks: 0 IO Block: 65536 directory 2024-03-24T00:15:48.020 INFO:teuthology.orchestra.run.smithi118.stdout:Device: 76h/118d Inode: 1099511628283 Links: 2 2024-03-24T00:15:48.020 INFO:teuthology.orchestra.run.smithi118.stdout:Access: (1777/drwxrwxrwt) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-24T00:15:48.020 INFO:teuthology.orchestra.run.smithi118.stdout:Context: system_u:object_r:unlabeled_t:s0 2024-03-24T00:15:48.020 INFO:teuthology.orchestra.run.smithi118.stdout:Access: 2024-03-24 00:13:09.781881030 +0000 2024-03-24T00:15:48.020 INFO:teuthology.orchestra.run.smithi118.stdout:Modify: 2024-03-24 00:13:09.781881030 +0000 2024-03-24T00:15:48.020 INFO:teuthology.orchestra.run.smithi118.stdout:Change: 2024-03-24 00:15:37.367555559 +0000 2024-03-24T00:15:48.020 INFO:teuthology.orchestra.run.smithi118.stdout: Birth: 2024-03-24 00:13:09.781881030 +0000 2024-03-24T00:15:48.020 INFO:tasks.workunit:Did not need to create dir /home/ubuntu/cephtest/mnt.0 2024-03-24T00:15:48.021 DEBUG:teuthology.orchestra.run.smithi118:> cd -- /home/ubuntu/cephtest/mnt.0 && sudo install -d -m 0755 --owner=ubuntu -- client.0 2024-03-24T00:15:48.127 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:15:47 smithi118 ceph-mon[31535]: from='client.? 172.21.15.118:0/2478782400' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-24T00:15:48.127 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:15:47 smithi118 ceph-mon[31535]: from='client.? 172.21.15.118:0/3201303141' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-24T00:15:48.127 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:15:47 smithi118 ceph-mon[31535]: pgmap v401: 97 pgs: 97 active+clean; 654 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 5.5 KiB/s rd, 1.5 KiB/s wr, 10 op/s 2024-03-24T00:15:48.151 DEBUG:teuthology.orchestra.run.smithi118:> rm -rf /home/ubuntu/cephtest/clone.client.0 && git clone https://git.ceph.com/ceph-ci.git /home/ubuntu/cephtest/clone.client.0 && cd /home/ubuntu/cephtest/clone.client.0 && git checkout 3be9a5c9bc793e11e2800a8c0c696e8b46742033 2024-03-24T00:15:48.178 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:48.178+0000 7f2d8ba00640 1 -- 172.21.15.118:0/2762406031 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f2d841016b0 msgr2=0x7f2d84101ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:15:48.178 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:48.178+0000 7f2d8ba00640 1 --2- 172.21.15.118:0/2762406031 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f2d841016b0 0x7f2d84101ad0 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7f2d78009860 tx=0x7f2d7802bb70 comp rx=0 tx=0).stop 2024-03-24T00:15:48.179 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:48.178+0000 7f2d8ba00640 1 -- 172.21.15.118:0/2762406031 shutdown_connections 2024-03-24T00:15:48.179 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:48.178+0000 7f2d8ba00640 1 --2- 172.21.15.118:0/2762406031 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f2d84103260 0x7f2d84109af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:15:48.179 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:48.178+0000 7f2d8ba00640 1 --2- 172.21.15.118:0/2762406031 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2d841028a0 0x7f2d84102d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:15:48.179 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:48.178+0000 7f2d8ba00640 1 --2- 172.21.15.118:0/2762406031 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f2d841016b0 0x7f2d84101ad0 unknown :-1 s=CLOSED pgs=198 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:15:48.179 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:48.178+0000 7f2d8ba00640 1 -- 172.21.15.118:0/2762406031 >> 172.21.15.118:0/2762406031 conn(0x7f2d840fcc40 msgr2=0x7f2d840ff060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:15:48.179 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:48.179+0000 7f2d8ba00640 1 -- 172.21.15.118:0/2762406031 shutdown_connections 2024-03-24T00:15:48.179 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:48.179+0000 7f2d8ba00640 1 -- 172.21.15.118:0/2762406031 wait complete. 2024-03-24T00:15:48.181 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:48.179+0000 7f2d8ba00640 1 Processor -- start 2024-03-24T00:15:48.181 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:48.180+0000 7f2d8ba00640 1 -- start start 2024-03-24T00:15:48.181 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:48.180+0000 7f2d8ba00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f2d841016b0 0x7f2d84197a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:15:48.181 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:48.180+0000 7f2d8ba00640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f2d841028a0 0x7f2d84197f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:15:48.181 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:48.180+0000 7f2d8ba00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2d84103260 0x7f2d8419efd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:15:48.181 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:48.180+0000 7f2d8ba00640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2d8410d300 con 0x7f2d841028a0 2024-03-24T00:15:48.181 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:48.180+0000 7f2d8ba00640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2d8410d180 con 0x7f2d84103260 2024-03-24T00:15:48.181 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:48.180+0000 7f2d8ba00640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2d8410d480 con 0x7f2d841016b0 2024-03-24T00:15:48.182 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:48.180+0000 7f2d89000640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f2d841016b0 0x7f2d84197a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:15:48.182 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:48.180+0000 7f2d89000640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f2d841016b0 0x7f2d84197a20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.118:34100/0 (socket says 172.21.15.118:34100) 2024-03-24T00:15:48.182 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:48.180+0000 7f2d89000640 1 -- 172.21.15.118:0/3861220039 learned_addr learned my addr 172.21.15.118:0/3861220039 (peer_addr_for_me v2:172.21.15.118:0/0) 2024-03-24T00:15:48.182 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:48.180+0000 7f2d89a00640 1 --2- 172.21.15.118:0/3861220039 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2d84103260 0x7f2d8419efd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:15:48.182 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:48.180+0000 7f2d83e00640 1 --2- 172.21.15.118:0/3861220039 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f2d841028a0 0x7f2d84197f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:15:48.182 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:48.181+0000 7f2d89000640 1 -- 172.21.15.118:0/3861220039 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2d84103260 msgr2=0x7f2d8419efd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:15:48.182 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:48.181+0000 7f2d89000640 1 --2- 172.21.15.118:0/3861220039 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2d84103260 0x7f2d8419efd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:15:48.182 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:48.181+0000 7f2d89000640 1 -- 172.21.15.118:0/3861220039 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f2d841028a0 msgr2=0x7f2d84197f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:15:48.182 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:48.181+0000 7f2d89000640 1 --2- 172.21.15.118:0/3861220039 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f2d841028a0 0x7f2d84197f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:15:48.182 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:48.181+0000 7f2d89000640 1 -- 172.21.15.118:0/3861220039 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2d8419f640 con 0x7f2d841016b0 2024-03-24T00:15:48.182 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:48.181+0000 7f2d89a00640 1 --2- 172.21.15.118:0/3861220039 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2d84103260 0x7f2d8419efd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T00:15:48.182 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:48.181+0000 7f2d89000640 1 --2- 172.21.15.118:0/3861220039 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f2d841016b0 0x7f2d84197a20 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7f2d780349e0 tx=0x7f2d780042f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:15:48.182 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:48.181+0000 7f2d81600640 1 -- 172.21.15.118:0/3861220039 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2d78041020 con 0x7f2d841016b0 2024-03-24T00:15:48.182 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:48.181+0000 7f2d81600640 1 -- 172.21.15.118:0/3861220039 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f2d780323b0 con 0x7f2d841016b0 2024-03-24T00:15:48.183 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:48.181+0000 7f2d8ba00640 1 -- 172.21.15.118:0/3861220039 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2d8419f8d0 con 0x7f2d841016b0 2024-03-24T00:15:48.183 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:48.181+0000 7f2d81600640 1 -- 172.21.15.118:0/3861220039 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2d78039040 con 0x7f2d841016b0 2024-03-24T00:15:48.183 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:48.181+0000 7f2d8ba00640 1 -- 172.21.15.118:0/3861220039 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2d8419fdc0 con 0x7f2d841016b0 2024-03-24T00:15:48.185 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:48.185+0000 7f2d8e7b2740 1 Processor -- start 2024-03-24T00:15:48.185 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:48.185+0000 7f2d8e7b2740 1 -- start start 2024-03-24T00:15:48.185 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:48.185+0000 7f2d81600640 1 -- 172.21.15.118:0/3861220039 <== mon.1 v2:172.21.15.193:3300/0 4 ==== mgrmap(e 19) v1 ==== 137342+0+0 (secure 0 0 0) 0x7f2d78044020 con 0x7f2d841016b0 2024-03-24T00:15:48.186 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:48.186+0000 7f2d8e7b2740 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55e2788c30a0 0x55e2788c34c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:15:48.186 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:48.186+0000 7f2d8e7b2740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55e2788c3a90 0x55e2788c8590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:15:48.186 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:48.186+0000 7f2d8e7b2740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55e2788c8c00 0x55e2788cb090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:15:48.186 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:48.186+0000 7f2d8e7b2740 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e278828eb0 con 0x55e2788c30a0 2024-03-24T00:15:48.186 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:48.186+0000 7f2d8e7b2740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e2787d1a60 con 0x55e2788c8c00 2024-03-24T00:15:48.186 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:48.186+0000 7f2d8e7b2740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e278824f50 con 0x55e2788c3a90 2024-03-24T00:15:48.187 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:48.186+0000 7f2d89000640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55e2788c3a90 0x55e2788c8590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:15:48.187 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:48.186+0000 7f2d89a00640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55e2788c30a0 0x55e2788c34c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:15:48.187 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:48.186+0000 7f2d89a00640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55e2788c30a0 0x55e2788c34c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.118:3300/0 says I am v2:172.21.15.118:43350/0 (socket says 172.21.15.118:43350) 2024-03-24T00:15:48.187 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:48.186+0000 7f2d89a00640 1 -- 172.21.15.118:0/403064818 learned_addr learned my addr 172.21.15.118:0/403064818 (peer_addr_for_me v2:172.21.15.118:0/0) 2024-03-24T00:15:48.187 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:48.186+0000 7f2d83e00640 1 --2- 172.21.15.118:0/403064818 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55e2788c8c00 0x55e2788cb090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:15:48.187 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:48.186+0000 7f2d89a00640 1 -- 172.21.15.118:0/403064818 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55e2788c3a90 msgr2=0x55e2788c8590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:15:48.187 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:48.186+0000 7f2d89a00640 1 --2- 172.21.15.118:0/403064818 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55e2788c3a90 0x55e2788c8590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:15:48.187 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:48.186+0000 7f2d89a00640 1 -- 172.21.15.118:0/403064818 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55e2788c8c00 msgr2=0x55e2788cb090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:15:48.187 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:48.186+0000 7f2d89a00640 1 --2- 172.21.15.118:0/403064818 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55e2788c8c00 0x55e2788cb090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:15:48.187 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:48.187+0000 7f2d89a00640 1 -- 172.21.15.118:0/403064818 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55e2786057e0 con 0x55e2788c30a0 2024-03-24T00:15:48.187 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:48.187+0000 7f2d89a00640 1 --2- 172.21.15.118:0/403064818 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55e2788c30a0 0x55e2788c34c0 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7f2d7c00f410 tx=0x7f2d7c00f8d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5f81808d6038f9f7 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:15:48.189 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:48.187+0000 7f2d76000640 1 -- 172.21.15.118:0/403064818 <== mon.0 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2d7c015070 con 0x55e2788c30a0 2024-03-24T00:15:48.189 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:48.187+0000 7f2d76000640 1 -- 172.21.15.118:0/403064818 <== mon.0 v2:172.21.15.118:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f2d7c011be0 con 0x55e2788c30a0 2024-03-24T00:15:48.189 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:48.187+0000 7f2d76000640 1 -- 172.21.15.118:0/403064818 <== mon.0 v2:172.21.15.118:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2d7c010cd0 con 0x55e2788c30a0 2024-03-24T00:15:48.189 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:48.187+0000 7f2d81600640 1 --2- 172.21.15.118:0/3861220039 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7f2d6c0a2890 0x7f2d6c0a4d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:15:48.189 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:48.188+0000 7f2d81600640 1 -- 172.21.15.118:0/3861220039 <== mon.1 v2:172.21.15.193:3300/0 5 ==== osd_map(90..90 src has 1..90) v4 ==== 8277+0+0 (secure 0 0 0) 0x7f2d780f0550 con 0x7f2d841016b0 2024-03-24T00:15:48.189 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:48.188+0000 7f2d89000640 1 --2- 172.21.15.118:0/3861220039 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7f2d6c0a2890 0x7f2d6c0a4d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:15:48.189 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:48.188+0000 7f2d8e7b2740 1 -- 172.21.15.118:0/403064818 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55e2788c30a0 msgr2=0x55e2788c34c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:15:48.189 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:48.188+0000 7f2d8e7b2740 1 --2- 172.21.15.118:0/403064818 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55e2788c30a0 0x55e2788c34c0 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7f2d7c00f410 tx=0x7f2d7c00f8d0 comp rx=0 tx=0).stop 2024-03-24T00:15:48.189 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:48.188+0000 7f2d8e7b2740 1 -- 172.21.15.118:0/403064818 shutdown_connections 2024-03-24T00:15:48.190 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:48.188+0000 7f2d8e7b2740 1 --2- 172.21.15.118:0/403064818 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55e2788c8c00 0x55e2788cb090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:15:48.190 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:48.188+0000 7f2d8e7b2740 1 --2- 172.21.15.118:0/403064818 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55e2788c3a90 0x55e2788c8590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:15:48.190 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:48.188+0000 7f2d8e7b2740 1 --2- 172.21.15.118:0/403064818 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55e2788c30a0 0x55e2788c34c0 unknown :-1 s=CLOSED pgs=179 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:15:48.190 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:48.188+0000 7f2d8e7b2740 1 -- 172.21.15.118:0/403064818 >> 172.21.15.118:0/403064818 conn(0x55e2788be610 msgr2=0x55e2788bea10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:15:48.190 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:48.188+0000 7f2d8e7b2740 1 -- 172.21.15.118:0/403064818 shutdown_connections 2024-03-24T00:15:48.190 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:48.188+0000 7f2d89000640 1 --2- 172.21.15.118:0/3861220039 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7f2d6c0a2890 0x7f2d6c0a4d50 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x55e2788c4500 tx=0x7f2d780f6110 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:15:48.190 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:48.188+0000 7f2d8e7b2740 1 -- 172.21.15.118:0/403064818 wait complete. 2024-03-24T00:15:48.190 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:48.189+0000 7f2d8e7b2740 1 -- start start 2024-03-24T00:15:48.190 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:48.189+0000 7f2d8e7b2740 1 Processor -- start 2024-03-24T00:15:48.192 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:48.190+0000 7f2d8e7b2740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55e2788c30a0 0x55e2788cd260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:15:48.192 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:48.190+0000 7f2d8e7b2740 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55e2788c3a90 0x55e2788cd7a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:15:48.193 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:48.190+0000 7f2d8e7b2740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55e2788c8c00 0x55e2788d47e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:15:48.193 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:48.190+0000 7f2d8e7b2740 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e2788cbbb0 con 0x55e2788c3a90 2024-03-24T00:15:48.193 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:48.190+0000 7f2d8e7b2740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e2787e4eb0 con 0x55e2788c30a0 2024-03-24T00:15:48.193 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:48.190+0000 7f2d8e7b2740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e2788cbd30 con 0x55e2788c8c00 2024-03-24T00:15:48.193 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:48.190+0000 7f2d8e7b2740 10 client.0 _ll_register_callbacks cb 0x55e2787d9150 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T00:15:48.193 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:48.190+0000 7f2d89a00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55e2788c30a0 0x55e2788cd260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:15:48.193 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:48.190+0000 7f2d89a00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55e2788c30a0 0x55e2788cd260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.118:36492/0 (socket says 172.21.15.118:36492) 2024-03-24T00:15:48.193 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:48.190+0000 7f2d89a00640 1 -- 172.21.15.118:0/550242013 learned_addr learned my addr 172.21.15.118:0/550242013 (peer_addr_for_me v2:172.21.15.118:0/0) 2024-03-24T00:15:48.193 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:48.190+0000 7f2d89a00640 1 --2- 172.21.15.118:0/550242013 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55e2788c8c00 0x55e2788d47e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:15:48.193 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:48.190+0000 7f2d83e00640 1 --2- 172.21.15.118:0/550242013 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55e2788c3a90 0x55e2788cd7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:15:48.193 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:48.191+0000 7f2d89a00640 1 -- 172.21.15.118:0/550242013 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55e2788c8c00 msgr2=0x55e2788d47e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:15:48.193 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:48.191+0000 7f2d89a00640 1 --2- 172.21.15.118:0/550242013 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55e2788c8c00 0x55e2788d47e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:15:48.193 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:48.191+0000 7f2d89a00640 1 -- 172.21.15.118:0/550242013 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55e2788c3a90 msgr2=0x55e2788cd7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:15:48.193 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:48.191+0000 7f2d89a00640 1 --2- 172.21.15.118:0/550242013 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55e2788c3a90 0x55e2788cd7a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:15:48.194 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:48.191+0000 7f2d89a00640 1 -- 172.21.15.118:0/550242013 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55e278665f70 con 0x55e2788c30a0 2024-03-24T00:15:48.194 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:48.191+0000 7f2d83e00640 1 --2- 172.21.15.118:0/550242013 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55e2788c3a90 0x55e2788cd7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-24T00:15:48.194 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:48.191+0000 7f2d89a00640 1 --2- 172.21.15.118:0/550242013 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55e2788c30a0 0x55e2788cd260 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7f2d7c011f40 tx=0x7f2d7c009940 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c1aef88205c1b2bd server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:15:48.194 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:48.191+0000 7f2d6aa00640 10 client.0 ms_handle_connect on v2:172.21.15.189:3300/0 2024-03-24T00:15:48.194 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:48.191+0000 7f2d6aa00640 1 -- 172.21.15.118:0/550242013 <== mon.2 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2d7c015040 con 0x55e2788c30a0 2024-03-24T00:15:48.194 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:48.191+0000 7f2d6aa00640 1 -- 172.21.15.118:0/550242013 <== mon.2 v2:172.21.15.189:3300/0 2 ==== osd_map(90..90 src has 1..90) v4 ==== 8277+0+0 (secure 0 0 0) 0x7f2d7c010c80 con 0x55e2788c30a0 2024-03-24T00:15:48.194 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:48.191+0000 7f2d8e7b2740 1 -- 172.21.15.118:0/550242013 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f2d7c011be0 con 0x55e2788c30a0 2024-03-24T00:15:48.194 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:48.191+0000 7f2d6aa00640 1 -- 172.21.15.118:0/550242013 <== mon.2 v2:172.21.15.189:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) v2 ==== 24+0+0 (secure 0 0 0) 0x7f2d7c030050 con 0x55e2788c30a0 2024-03-24T00:15:48.194 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:48.191+0000 7f2d8e7b2740 10 client.25385 fetch_fsmap learned FSMap version 28 2024-03-24T00:15:48.194 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:48.191+0000 7f2d8e7b2740 1 -- 172.21.15.118:0/550242013 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) v3 -- 0x55e2784fd170 con 0x55e2788c30a0 2024-03-24T00:15:48.194 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:48.192+0000 7f2d6aa00640 1 -- 172.21.15.118:0/550242013 <== mon.2 v2:172.21.15.189:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f2d7c011be0 con 0x55e2788c30a0 2024-03-24T00:15:48.194 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:48.192+0000 7f2d6aa00640 1 -- 172.21.15.118:0/550242013 <== mon.2 v2:172.21.15.189:3300/0 5 ==== fsmap(e 28) v1 ==== 7934+0+0 (secure 0 0 0) 0x7f2d7c028020 con 0x55e2788c30a0 2024-03-24T00:15:48.194 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:48.192+0000 7f2d8e7b2740 10 client.25385 fetch_fsmap finished waiting for FSMap version 28 2024-03-24T00:15:48.195 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:48.192+0000 7f2d8e7b2740 10 client.25385 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-24T00:15:48.195 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:48.192+0000 7f2d8e7b2740 20 client.25385 populate_metadata read hostname 'smithi118' 2024-03-24T00:15:48.195 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:48.192+0000 7f2d8e7b2740 1 --2- 172.21.15.118:0/550242013 >> [v2:172.21.15.189:6838/4074484168,v1:172.21.15.189:6839/4074484168] conn(0x55e2788d8e20 0x55e2788db230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:15:48.195 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:48.192+0000 7f2d8e7b2740 4 client.25385 mds_command: new command op to 24569 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-24T00:15:48.195 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:48.192+0000 7f2d8e7b2740 1 -- 172.21.15.118:0/550242013 --> [v2:172.21.15.189:6838/4074484168,v1:172.21.15.189:6839/4074484168] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55e2788363f0 con 0x55e2788d8e20 2024-03-24T00:15:48.195 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:48.192+0000 7f2d83e00640 1 --2- 172.21.15.118:0/550242013 >> [v2:172.21.15.189:6838/4074484168,v1:172.21.15.189:6839/4074484168] conn(0x55e2788d8e20 0x55e2788db230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:15:48.195 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:48.193+0000 7f2d83e00640 1 --2- 172.21.15.118:0/550242013 >> [v2:172.21.15.189:6838/4074484168,v1:172.21.15.189:6839/4074484168] conn(0x55e2788d8e20 0x55e2788db230 crc :-1 s=READY pgs=33 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=b556a6f0c27c5bdd server_cookie=9f8581f76e1ba737 in_seq=0 out_seq=0 2024-03-24T00:15:48.195 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:48.193+0000 7f2d6aa00640 10 client.25385 ms_handle_connect on v2:172.21.15.189:6838/4074484168 2024-03-24T00:15:48.196 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:48.193+0000 7f2d6aa00640 1 -- 172.21.15.118:0/550242013 <== mds.0 v2:172.21.15.189:6838/4074484168 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x55e2788363f0 con 0x55e2788d8e20 2024-03-24T00:15:48.196 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:48.193+0000 7f2d6aa00640 10 client.25385 handle_command_reply: tid=0 2024-03-24T00:15:48.196 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:48.193+0000 7f2d8e7b2740 1 client.25385 shutdown 2024-03-24T00:15:48.196 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:48.194+0000 7f2d8e7b2740 1 -- 172.21.15.118:0/550242013 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55e2788c30a0 msgr2=0x55e2788cd260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:15:48.197 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:48.194+0000 7f2d8e7b2740 1 --2- 172.21.15.118:0/550242013 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55e2788c30a0 0x55e2788cd260 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7f2d7c011f40 tx=0x7f2d7c009940 comp rx=0 tx=0).stop 2024-03-24T00:15:48.197 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:48.194+0000 7f2d8e7b2740 1 -- 172.21.15.118:0/550242013 shutdown_connections 2024-03-24T00:15:48.197 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:48.194+0000 7f2d8e7b2740 1 --2- 172.21.15.118:0/550242013 >> [v2:172.21.15.189:6838/4074484168,v1:172.21.15.189:6839/4074484168] conn(0x55e2788d8e20 0x55e2788db230 crc :-1 s=READY pgs=33 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:15:48.197 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:48.194+0000 7f2d8e7b2740 1 --2- 172.21.15.118:0/550242013 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55e2788c8c00 0x55e2788d47e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:15:48.197 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:48.194+0000 7f2d8e7b2740 1 --2- 172.21.15.118:0/550242013 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55e2788c3a90 0x55e2788cd7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:15:48.197 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:48.194+0000 7f2d6aa00640 0 client.25385 ms_handle_reset on v2:172.21.15.189:6838/4074484168 2024-03-24T00:15:48.197 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:48.194+0000 7f2d8e7b2740 1 --2- 172.21.15.118:0/550242013 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55e2788c30a0 0x55e2788cd260 unknown :-1 s=CLOSED pgs=205 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:15:48.197 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:48.194+0000 7f2d8e7b2740 1 -- 172.21.15.118:0/550242013 >> 172.21.15.118:0/550242013 conn(0x55e2788be610 msgr2=0x55e2788bd830 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:15:48.197 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:48.194+0000 7f2d8e7b2740 1 -- 172.21.15.118:0/550242013 shutdown_connections 2024-03-24T00:15:48.197 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:48.194+0000 7f2d8e7b2740 1 -- 172.21.15.118:0/550242013 wait complete. 2024-03-24T00:15:48.197 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:48.194+0000 7f2d8e7b2740 20 client.25385 trim_cache size 0 max 16384 2024-03-24T00:15:48.197 INFO:tasks.workunit.client.0.smithi118.stderr:Cloning into '/home/ubuntu/cephtest/clone.client.0'... 2024-03-24T00:15:48.203 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:15:47 smithi189 ceph-mon[36187]: from='client.? 172.21.15.118:0/2478782400' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-24T00:15:48.203 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:15:47 smithi189 ceph-mon[36187]: from='client.? 172.21.15.118:0/3201303141' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-24T00:15:48.203 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:15:47 smithi189 ceph-mon[36187]: pgmap v401: 97 pgs: 97 active+clean; 654 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 5.5 KiB/s rd, 1.5 KiB/s wr, 10 op/s 2024-03-24T00:15:48.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:15:47 smithi193 ceph-mon[36878]: from='client.? 172.21.15.118:0/2478782400' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-24T00:15:48.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:15:47 smithi193 ceph-mon[36878]: from='client.? 172.21.15.118:0/3201303141' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-24T00:15:48.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:15:47 smithi193 ceph-mon[36878]: pgmap v401: 97 pgs: 97 active+clean; 654 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 5.5 KiB/s rd, 1.5 KiB/s wr, 10 op/s 2024-03-24T00:15:48.212 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:48.211+0000 7f2d8e7b2740 1 Processor -- start 2024-03-24T00:15:48.212 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:48.211+0000 7f2d8e7b2740 1 -- start start 2024-03-24T00:15:48.212 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:48.211+0000 7f2d8e7b2740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55e2788c30a0 0x55e2788bd6b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:15:48.212 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:48.212+0000 7f2d8e7b2740 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55e2788c3a90 0x55e2788bdbf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:15:48.212 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:48.212+0000 7f2d8e7b2740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55e2788c8c00 0x55e2788d4770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:15:48.213 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:48.212+0000 7f2d8e7b2740 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2d841a6850 con 0x55e2788c3a90 2024-03-24T00:15:48.213 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:48.212+0000 7f2d8e7b2740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2d841a81d0 con 0x55e2788c30a0 2024-03-24T00:15:48.213 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:48.212+0000 7f2d8e7b2740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2d841a6eb0 con 0x55e2788c8c00 2024-03-24T00:15:48.213 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:48.212+0000 7f2d89a00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55e2788c30a0 0x55e2788bd6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:15:48.213 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:48.212+0000 7f2d89a00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55e2788c30a0 0x55e2788bd6b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.118:36496/0 (socket says 172.21.15.118:36496) 2024-03-24T00:15:48.213 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:48.212+0000 7f2d89a00640 1 -- 172.21.15.118:0/1637463459 learned_addr learned my addr 172.21.15.118:0/1637463459 (peer_addr_for_me v2:172.21.15.118:0/0) 2024-03-24T00:15:48.213 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:48.212+0000 7f2d83e00640 1 --2- 172.21.15.118:0/1637463459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55e2788c3a90 0x55e2788bdbf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:15:48.213 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:48.212+0000 7f2d89a00640 1 --2- 172.21.15.118:0/1637463459 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55e2788c8c00 0x55e2788d4770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:15:48.213 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:48.212+0000 7f2d89a00640 1 -- 172.21.15.118:0/1637463459 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55e2788c8c00 msgr2=0x55e2788d4770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:15:48.213 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:48.212+0000 7f2d89a00640 1 --2- 172.21.15.118:0/1637463459 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55e2788c8c00 0x55e2788d4770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:15:48.213 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:48.212+0000 7f2d89a00640 1 -- 172.21.15.118:0/1637463459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55e2788c3a90 msgr2=0x55e2788bdbf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:15:48.213 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:48.212+0000 7f2d89a00640 1 --2- 172.21.15.118:0/1637463459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55e2788c3a90 0x55e2788bdbf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:15:48.213 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:48.212+0000 7f2d89a00640 1 -- 172.21.15.118:0/1637463459 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55e2788c5540 con 0x55e2788c30a0 2024-03-24T00:15:48.213 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:48.212+0000 7f2d89a00640 1 --2- 172.21.15.118:0/1637463459 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55e2788c30a0 0x55e2788bd6b0 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7f2d7c010720 tx=0x7f2d7c00f8d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=16225cff7b85f8b6 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:15:48.214 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:48.213+0000 7f2d75600640 1 -- 172.21.15.118:0/1637463459 <== mon.2 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2d7c015070 con 0x55e2788c30a0 2024-03-24T00:15:48.214 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:48.213+0000 7f2d75600640 1 -- 172.21.15.118:0/1637463459 <== mon.2 v2:172.21.15.189:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f2d7c0051a0 con 0x55e2788c30a0 2024-03-24T00:15:48.214 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:48.213+0000 7f2d75600640 1 -- 172.21.15.118:0/1637463459 <== mon.2 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2d7c010c80 con 0x55e2788c30a0 2024-03-24T00:15:48.217 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:48.213+0000 7f2d8e7b2740 1 -- 172.21.15.118:0/1637463459 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55e2788c30a0 msgr2=0x55e2788bd6b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:15:48.217 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:48.213+0000 7f2d8e7b2740 1 --2- 172.21.15.118:0/1637463459 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55e2788c30a0 0x55e2788bd6b0 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7f2d7c010720 tx=0x7f2d7c00f8d0 comp rx=0 tx=0).stop 2024-03-24T00:15:48.217 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:48.213+0000 7f2d8e7b2740 1 -- 172.21.15.118:0/1637463459 shutdown_connections 2024-03-24T00:15:48.217 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:48.213+0000 7f2d8e7b2740 1 --2- 172.21.15.118:0/1637463459 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55e2788c8c00 0x55e2788d4770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:15:48.217 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:48.213+0000 7f2d8e7b2740 1 --2- 172.21.15.118:0/1637463459 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55e2788c3a90 0x55e2788bdbf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:15:48.217 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:48.213+0000 7f2d8e7b2740 1 --2- 172.21.15.118:0/1637463459 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55e2788c30a0 0x55e2788bd6b0 unknown :-1 s=CLOSED pgs=206 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:15:48.217 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:48.213+0000 7f2d8e7b2740 1 -- 172.21.15.118:0/1637463459 >> 172.21.15.118:0/1637463459 conn(0x55e2788be610 msgr2=0x55e2788bcfa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:15:48.218 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:48.214+0000 7f2d8e7b2740 1 -- 172.21.15.118:0/1637463459 shutdown_connections 2024-03-24T00:15:48.218 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:48.214+0000 7f2d8e7b2740 1 -- 172.21.15.118:0/1637463459 wait complete. 2024-03-24T00:15:48.218 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:48.214+0000 7f2d8e7b2740 1 -- start start 2024-03-24T00:15:48.218 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:48.214+0000 7f2d8e7b2740 1 Processor -- start 2024-03-24T00:15:48.218 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:48.215+0000 7f2d8e7b2740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55e2788c30a0 0x55e2788d4510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:15:48.218 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:48.215+0000 7f2d8e7b2740 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55e2788c3a90 0x55e2788d2b20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:15:48.218 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:48.215+0000 7f2d8e7b2740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55e2788c8c00 0x55e2788d3060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:15:48.218 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:48.215+0000 7f2d8e7b2740 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2d841a7b70 con 0x55e2788c3a90 2024-03-24T00:15:48.218 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:48.215+0000 7f2d8e7b2740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2d841a7510 con 0x55e2788c30a0 2024-03-24T00:15:48.218 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:48.215+0000 7f2d8e7b2740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2d841a4a10 con 0x55e2788c8c00 2024-03-24T00:15:48.218 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:48.215+0000 7f2d8e7b2740 10 client.0 _ll_register_callbacks cb 0x7f2d70009520 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T00:15:48.218 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:48.215+0000 7f2d89a00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55e2788c30a0 0x55e2788d4510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:15:48.218 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:48.215+0000 7f2d89a00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55e2788c30a0 0x55e2788d4510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.118:36498/0 (socket says 172.21.15.118:36498) 2024-03-24T00:15:48.219 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:48.215+0000 7f2d89a00640 1 -- 172.21.15.118:0/496178958 learned_addr learned my addr 172.21.15.118:0/496178958 (peer_addr_for_me v2:172.21.15.118:0/0) 2024-03-24T00:15:48.219 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:48.215+0000 7f2d83e00640 1 --2- 172.21.15.118:0/496178958 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55e2788c3a90 0x55e2788d2b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:15:48.219 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:48.215+0000 7f2d89a00640 1 --2- 172.21.15.118:0/496178958 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55e2788c8c00 0x55e2788d3060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:15:48.219 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:48.215+0000 7f2d89a00640 1 -- 172.21.15.118:0/496178958 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55e2788c8c00 msgr2=0x55e2788d3060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:15:48.219 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:48.215+0000 7f2d89a00640 1 --2- 172.21.15.118:0/496178958 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55e2788c8c00 0x55e2788d3060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:15:48.219 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:48.215+0000 7f2d89a00640 1 -- 172.21.15.118:0/496178958 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55e2788c3a90 msgr2=0x55e2788d2b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:15:48.219 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:48.215+0000 7f2d89a00640 1 --2- 172.21.15.118:0/496178958 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55e2788c3a90 0x55e2788d2b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:15:48.219 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:48.215+0000 7f2d89a00640 1 -- 172.21.15.118:0/496178958 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55e2788d3620 con 0x55e2788c30a0 2024-03-24T00:15:48.219 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:48.215+0000 7f2d89a00640 1 --2- 172.21.15.118:0/496178958 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55e2788c30a0 0x55e2788d4510 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7f2d7c010470 tx=0x7f2d7c010750 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c0e35bc319b0a5b6 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:15:48.219 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:48.216+0000 7f2d6aa00640 10 client.0 ms_handle_connect on v2:172.21.15.189:3300/0 2024-03-24T00:15:48.219 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:48.216+0000 7f2d6aa00640 1 -- 172.21.15.118:0/496178958 <== mon.2 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2d7c015070 con 0x55e2788c30a0 2024-03-24T00:15:48.219 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:48.216+0000 7f2d6aa00640 1 -- 172.21.15.118:0/496178958 <== mon.2 v2:172.21.15.189:3300/0 2 ==== osd_map(90..90 src has 1..90) v4 ==== 8277+0+0 (secure 0 0 0) 0x7f2d7c025040 con 0x55e2788c30a0 2024-03-24T00:15:48.219 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:48.216+0000 7f2d8e7b2740 1 -- 172.21.15.118:0/496178958 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f2d7c0051a0 con 0x55e2788c30a0 2024-03-24T00:15:48.219 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:48.216+0000 7f2d6aa00640 1 -- 172.21.15.118:0/496178958 <== mon.2 v2:172.21.15.189:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) v2 ==== 24+0+0 (secure 0 0 0) 0x7f2d7c045050 con 0x55e2788c30a0 2024-03-24T00:15:48.220 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:48.216+0000 7f2d8e7b2740 10 client.25391 fetch_fsmap learned FSMap version 28 2024-03-24T00:15:48.220 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:48.216+0000 7f2d8e7b2740 1 -- 172.21.15.118:0/496178958 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) v3 -- 0x55e2789354d0 con 0x55e2788c30a0 2024-03-24T00:15:48.220 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:48.216+0000 7f2d6aa00640 1 -- 172.21.15.118:0/496178958 <== mon.2 v2:172.21.15.189:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f2d7c0051a0 con 0x55e2788c30a0 2024-03-24T00:15:48.220 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:48.216+0000 7f2d6aa00640 1 -- 172.21.15.118:0/496178958 <== mon.2 v2:172.21.15.189:3300/0 5 ==== fsmap(e 28) v1 ==== 7934+0+0 (secure 0 0 0) 0x7f2d7c028070 con 0x55e2788c30a0 2024-03-24T00:15:48.220 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:48.217+0000 7f2d8e7b2740 10 client.25391 fetch_fsmap finished waiting for FSMap version 28 2024-03-24T00:15:48.220 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:48.217+0000 7f2d8e7b2740 10 client.25391 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-24T00:15:48.220 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:48.217+0000 7f2d8e7b2740 20 client.25391 populate_metadata read hostname 'smithi118' 2024-03-24T00:15:48.220 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:48.217+0000 7f2d8e7b2740 1 --2- 172.21.15.118:0/496178958 >> [v2:172.21.15.189:6838/4074484168,v1:172.21.15.189:6839/4074484168] conn(0x55e278935dc0 0x55e2789381e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:15:48.220 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:48.217+0000 7f2d8e7b2740 4 client.25391 mds_command: new command op to 24569 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-03-24T00:15:48.220 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:48.217+0000 7f2d8e7b2740 1 -- 172.21.15.118:0/496178958 --> [v2:172.21.15.189:6838/4074484168,v1:172.21.15.189:6839/4074484168] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) v1 -- 0x55e278830740 con 0x55e278935dc0 2024-03-24T00:15:48.220 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:48.217+0000 7f2d83e00640 1 --2- 172.21.15.118:0/496178958 >> [v2:172.21.15.189:6838/4074484168,v1:172.21.15.189:6839/4074484168] conn(0x55e278935dc0 0x55e2789381e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:15:48.220 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:48.217+0000 7f2d83e00640 1 --2- 172.21.15.118:0/496178958 >> [v2:172.21.15.189:6838/4074484168,v1:172.21.15.189:6839/4074484168] conn(0x55e278935dc0 0x55e2789381e0 crc :-1 s=READY pgs=34 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=a2126abd45f44511 server_cookie=6e75db86bd6901ed in_seq=0 out_seq=0 2024-03-24T00:15:48.220 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:48.217+0000 7f2d6aa00640 10 client.25391 ms_handle_connect on v2:172.21.15.189:6838/4074484168 2024-03-24T00:15:48.221 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:48.218+0000 7f2d6aa00640 1 -- 172.21.15.118:0/496178958 <== mds.0 v2:172.21.15.189:6838/4074484168 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+110 (crc 0 0 0) 0x55e278830740 con 0x55e278935dc0 2024-03-24T00:15:48.222 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:48.218+0000 7f2d6aa00640 10 client.25391 handle_command_reply: tid=0 2024-03-24T00:15:48.222 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:48.218+0000 7f2d8e7b2740 1 client.25391 shutdown 2024-03-24T00:15:48.222 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:48.219+0000 7f2d8e7b2740 1 -- 172.21.15.118:0/496178958 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55e2788c30a0 msgr2=0x55e2788d4510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:15:48.222 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:48.219+0000 7f2d8e7b2740 1 --2- 172.21.15.118:0/496178958 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55e2788c30a0 0x55e2788d4510 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7f2d7c010470 tx=0x7f2d7c010750 comp rx=0 tx=0).stop 2024-03-24T00:15:48.222 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:48.219+0000 7f2d8e7b2740 1 -- 172.21.15.118:0/496178958 shutdown_connections 2024-03-24T00:15:48.222 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:48.219+0000 7f2d8e7b2740 1 --2- 172.21.15.118:0/496178958 >> [v2:172.21.15.189:6838/4074484168,v1:172.21.15.189:6839/4074484168] conn(0x55e278935dc0 0x55e2789381e0 crc :-1 s=READY pgs=34 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:15:48.222 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:48.219+0000 7f2d8e7b2740 1 --2- 172.21.15.118:0/496178958 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55e2788c8c00 0x55e2788d3060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:15:48.222 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:48.219+0000 7f2d8e7b2740 1 --2- 172.21.15.118:0/496178958 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55e2788c3a90 0x55e2788d2b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:15:48.222 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:48.219+0000 7f2d8e7b2740 1 --2- 172.21.15.118:0/496178958 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55e2788c30a0 0x55e2788d4510 unknown :-1 s=CLOSED pgs=207 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:15:48.222 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:48.219+0000 7f2d8e7b2740 1 -- 172.21.15.118:0/496178958 >> 172.21.15.118:0/496178958 conn(0x55e2788be610 msgr2=0x55e2788c4dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:15:48.222 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:48.219+0000 7f2d6aa00640 0 client.25391 ms_handle_reset on v2:172.21.15.189:6838/4074484168 2024-03-24T00:15:48.222 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:48.219+0000 7f2d8e7b2740 1 -- 172.21.15.118:0/496178958 shutdown_connections 2024-03-24T00:15:48.222 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:48.219+0000 7f2d8e7b2740 1 -- 172.21.15.118:0/496178958 wait complete. 2024-03-24T00:15:48.223 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:48.219+0000 7f2d8e7b2740 20 client.25391 trim_cache size 0 max 16384 2024-03-24T00:15:48.223 INFO:teuthology.orchestra.run.smithi118.stdout:{ 2024-03-24T00:15:48.223 INFO:teuthology.orchestra.run.smithi118.stdout: "return_code": 0, 2024-03-24T00:15:48.223 INFO:teuthology.orchestra.run.smithi118.stdout: "scrub_tag": "89b9d04e-be57-42e0-abce-bad803301c34", 2024-03-24T00:15:48.223 INFO:teuthology.orchestra.run.smithi118.stdout: "mode": "asynchronous" 2024-03-24T00:15:48.223 INFO:teuthology.orchestra.run.smithi118.stdout:} 2024-03-24T00:15:48.224 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:48.220+0000 7f2d8ba00640 1 -- 172.21.15.118:0/3861220039 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7f2d6c0a2890 msgr2=0x7f2d6c0a4d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:15:48.224 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:48.220+0000 7f2d8ba00640 1 --2- 172.21.15.118:0/3861220039 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7f2d6c0a2890 0x7f2d6c0a4d50 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x55e2788c4500 tx=0x7f2d780f6110 comp rx=0 tx=0).stop 2024-03-24T00:15:48.224 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:48.220+0000 7f2d8ba00640 1 -- 172.21.15.118:0/3861220039 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f2d841016b0 msgr2=0x7f2d84197a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:15:48.224 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:48.220+0000 7f2d8ba00640 1 --2- 172.21.15.118:0/3861220039 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f2d841016b0 0x7f2d84197a20 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7f2d780349e0 tx=0x7f2d780042f0 comp rx=0 tx=0).stop 2024-03-24T00:15:48.224 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:48.220+0000 7f2d8ba00640 1 -- 172.21.15.118:0/3861220039 shutdown_connections 2024-03-24T00:15:48.224 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:48.220+0000 7f2d8ba00640 1 --2- 172.21.15.118:0/3861220039 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7f2d6c0a2890 0x7f2d6c0a4d50 unknown :-1 s=CLOSED pgs=269 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:15:48.224 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:48.220+0000 7f2d8ba00640 1 --2- 172.21.15.118:0/3861220039 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2d84103260 0x7f2d8419efd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:15:48.224 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:48.220+0000 7f2d8ba00640 1 --2- 172.21.15.118:0/3861220039 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f2d841028a0 0x7f2d84197f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:15:48.224 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:48.220+0000 7f2d8ba00640 1 --2- 172.21.15.118:0/3861220039 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f2d841016b0 0x7f2d84197a20 unknown :-1 s=CLOSED pgs=199 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:15:48.224 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:48.220+0000 7f2d8ba00640 1 -- 172.21.15.118:0/3861220039 >> 172.21.15.118:0/3861220039 conn(0x7f2d840fcc40 msgr2=0x7f2d84105cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:15:48.224 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:48.220+0000 7f2d8ba00640 1 -- 172.21.15.118:0/3861220039 shutdown_connections 2024-03-24T00:15:48.225 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:48.220+0000 7f2d8ba00640 1 -- 172.21.15.118:0/3861220039 wait complete. 2024-03-24T00:15:48.237 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-03-24T00:15:48.237 DEBUG:teuthology.orchestra.run.smithi118:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-24T00:15:48.487 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:48.487+0000 7fa905c00640 1 -- 172.21.15.118:0/3552262600 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fa900102790 msgr2=0x7fa900102c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:15:48.487 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:48.487+0000 7fa905c00640 1 --2- 172.21.15.118:0/3552262600 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fa900102790 0x7fa900102c10 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7fa8ec009900 tx=0x7fa8ec02bc30 comp rx=0 tx=0).stop 2024-03-24T00:15:48.487 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:48.487+0000 7fa905c00640 1 -- 172.21.15.118:0/3552262600 shutdown_connections 2024-03-24T00:15:48.487 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:48.487+0000 7fa905c00640 1 --2- 172.21.15.118:0/3552262600 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa900103150 0x7fa9001099e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:15:48.487 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:48.487+0000 7fa905c00640 1 --2- 172.21.15.118:0/3552262600 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fa900102790 0x7fa900102c10 unknown :-1 s=CLOSED pgs=200 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:15:48.487 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:48.487+0000 7fa905c00640 1 --2- 172.21.15.118:0/3552262600 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fa9001015a0 0x7fa9001019c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:15:48.488 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:48.487+0000 7fa905c00640 1 -- 172.21.15.118:0/3552262600 >> 172.21.15.118:0/3552262600 conn(0x7fa9000fcb10 msgr2=0x7fa9000fef50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:15:48.488 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:48.487+0000 7fa905c00640 1 -- 172.21.15.118:0/3552262600 shutdown_connections 2024-03-24T00:15:48.488 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:48.487+0000 7fa905c00640 1 -- 172.21.15.118:0/3552262600 wait complete. 2024-03-24T00:15:48.489 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:48.488+0000 7fa905c00640 1 Processor -- start 2024-03-24T00:15:48.489 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:48.488+0000 7fa905c00640 1 -- start start 2024-03-24T00:15:48.489 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:48.489+0000 7fa905c00640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fa9001015a0 0x7fa900199a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:15:48.489 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:48.489+0000 7fa905c00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fa900102790 0x7fa900199fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:15:48.490 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:48.489+0000 7fa905c00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa900103150 0x7fa9001a1010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:15:48.490 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:48.489+0000 7fa905c00640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa90010d250 con 0x7fa9001015a0 2024-03-24T00:15:48.490 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:48.489+0000 7fa905c00640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa90010d0d0 con 0x7fa900103150 2024-03-24T00:15:48.490 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:48.489+0000 7fa905c00640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa90010d3d0 con 0x7fa900102790 2024-03-24T00:15:48.490 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:48.489+0000 7fa8ff400640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fa900102790 0x7fa900199fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:15:48.490 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:48.489+0000 7fa905200640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa900103150 0x7fa9001a1010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:15:48.490 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:48.489+0000 7fa8ff400640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fa900102790 0x7fa900199fa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.118:34146/0 (socket says 172.21.15.118:34146) 2024-03-24T00:15:48.490 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:48.489+0000 7fa8ff400640 1 -- 172.21.15.118:0/273912562 learned_addr learned my addr 172.21.15.118:0/273912562 (peer_addr_for_me v2:172.21.15.118:0/0) 2024-03-24T00:15:48.490 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:48.489+0000 7fa8ffe00640 1 --2- 172.21.15.118:0/273912562 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fa9001015a0 0x7fa900199a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:15:48.490 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:48.489+0000 7fa8ff400640 1 -- 172.21.15.118:0/273912562 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa900103150 msgr2=0x7fa9001a1010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:15:48.490 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:48.490+0000 7fa8ff400640 1 --2- 172.21.15.118:0/273912562 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa900103150 0x7fa9001a1010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:15:48.490 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:48.490+0000 7fa8ff400640 1 -- 172.21.15.118:0/273912562 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fa9001015a0 msgr2=0x7fa900199a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:15:48.491 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:48.490+0000 7fa8ff400640 1 --2- 172.21.15.118:0/273912562 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fa9001015a0 0x7fa900199a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:15:48.491 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:48.490+0000 7fa8ff400640 1 -- 172.21.15.118:0/273912562 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa9001a1680 con 0x7fa900102790 2024-03-24T00:15:48.491 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:48.490+0000 7fa905200640 1 --2- 172.21.15.118:0/273912562 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa900103150 0x7fa9001a1010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T00:15:48.491 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:48.490+0000 7fa8ff400640 1 --2- 172.21.15.118:0/273912562 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fa900102790 0x7fa900199fa0 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7fa8ec005ab0 tx=0x7fa8ec02c6c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:15:48.492 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:48.490+0000 7fa8fcc00640 1 -- 172.21.15.118:0/273912562 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa8ec043020 con 0x7fa900102790 2024-03-24T00:15:48.492 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:48.490+0000 7fa8fcc00640 1 -- 172.21.15.118:0/273912562 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fa8ec0054e0 con 0x7fa900102790 2024-03-24T00:15:48.492 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:48.490+0000 7fa8fcc00640 1 -- 172.21.15.118:0/273912562 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa8ec003ce0 con 0x7fa900102790 2024-03-24T00:15:48.492 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:48.490+0000 7fa905c00640 1 -- 172.21.15.118:0/273912562 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa9001a1910 con 0x7fa900102790 2024-03-24T00:15:48.492 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:48.490+0000 7fa905c00640 1 -- 172.21.15.118:0/273912562 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa9001a1e10 con 0x7fa900102790 2024-03-24T00:15:48.494 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:48.494+0000 7fa909e6a740 1 Processor -- start 2024-03-24T00:15:48.494 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:48.494+0000 7fa909e6a740 1 -- start start 2024-03-24T00:15:48.494 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:48.494+0000 7fa8fcc00640 1 -- 172.21.15.118:0/273912562 <== mon.1 v2:172.21.15.193:3300/0 4 ==== mgrmap(e 19) v1 ==== 137342+0+0 (secure 0 0 0) 0x7fa8ec046020 con 0x7fa900102790 2024-03-24T00:15:48.494 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:48.494+0000 7fa909e6a740 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x560d9c743050 0x560d9c743470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:15:48.495 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:48.494+0000 7fa909e6a740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x560d9c743a40 0x560d9c748540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:15:48.495 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:48.495+0000 7fa909e6a740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x560d9c748bb0 0x560d9c74b040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:15:48.495 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:48.495+0000 7fa909e6a740 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x560d9c6a8250 con 0x560d9c743050 2024-03-24T00:15:48.495 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:48.495+0000 7fa909e6a740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x560d9c650a30 con 0x560d9c748bb0 2024-03-24T00:15:48.495 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:48.495+0000 7fa909e6a740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x560d9c6a42f0 con 0x560d9c743a40 2024-03-24T00:15:48.495 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:48.495+0000 7fa8ffe00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x560d9c743a40 0x560d9c748540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:15:48.495 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:48.495+0000 7fa905200640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x560d9c743050 0x560d9c743470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:15:48.495 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:48.495+0000 7fa8ffe00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x560d9c743a40 0x560d9c748540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.118:34160/0 (socket says 172.21.15.118:34160) 2024-03-24T00:15:48.495 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:48.495+0000 7fa905200640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x560d9c743050 0x560d9c743470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.118:3300/0 says I am v2:172.21.15.118:43400/0 (socket says 172.21.15.118:43400) 2024-03-24T00:15:48.495 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:48.495+0000 7fa8ffe00640 1 -- 172.21.15.118:0/2479290391 learned_addr learned my addr 172.21.15.118:0/2479290391 (peer_addr_for_me v2:172.21.15.118:0/0) 2024-03-24T00:15:48.495 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:48.495+0000 7fa8ff400640 1 --2- 172.21.15.118:0/2479290391 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x560d9c748bb0 0x560d9c74b040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:15:48.497 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:48.495+0000 7fa8ffe00640 1 -- 172.21.15.118:0/2479290391 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x560d9c748bb0 msgr2=0x560d9c74b040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:15:48.497 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:48.495+0000 7fa8ffe00640 1 --2- 172.21.15.118:0/2479290391 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x560d9c748bb0 0x560d9c74b040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:15:48.497 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:48.495+0000 7fa8ffe00640 1 -- 172.21.15.118:0/2479290391 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x560d9c743050 msgr2=0x560d9c743470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:15:48.497 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:48.495+0000 7fa8ffe00640 1 --2- 172.21.15.118:0/2479290391 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x560d9c743050 0x560d9c743470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:15:48.498 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:48.495+0000 7fa8ffe00640 1 -- 172.21.15.118:0/2479290391 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x560d9c63b690 con 0x560d9c743a40 2024-03-24T00:15:48.498 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:48.495+0000 7fa905200640 1 --2- 172.21.15.118:0/2479290391 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x560d9c743050 0x560d9c743470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T00:15:48.498 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:48.496+0000 7fa8ffe00640 1 --2- 172.21.15.118:0/2479290391 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x560d9c743a40 0x560d9c748540 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7fa8f000bd10 tx=0x7fa8f000f730 comp rx=0 tx=0).ready entity=mon.1 client_cookie=331a17111f5915a1 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:15:48.498 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:48.496+0000 7fa8f5600640 1 -- 172.21.15.118:0/2479290391 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa8f0015070 con 0x560d9c743a40 2024-03-24T00:15:48.498 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:48.496+0000 7fa8f5600640 1 -- 172.21.15.118:0/2479290391 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fa8f001a940 con 0x560d9c743a40 2024-03-24T00:15:48.498 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:48.496+0000 7fa8f5600640 1 -- 172.21.15.118:0/2479290391 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa8f00119a0 con 0x560d9c743a40 2024-03-24T00:15:48.498 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:48.496+0000 7fa8fcc00640 1 --2- 172.21.15.118:0/273912562 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7fa8e40a2890 0x7fa8e40a4d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:15:48.498 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:48.497+0000 7fa8fcc00640 1 -- 172.21.15.118:0/273912562 <== mon.1 v2:172.21.15.193:3300/0 5 ==== osd_map(90..90 src has 1..90) v4 ==== 8277+0+0 (secure 0 0 0) 0x7fa8ec0f0d10 con 0x7fa900102790 2024-03-24T00:15:48.498 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:48.497+0000 7fa905200640 1 --2- 172.21.15.118:0/273912562 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7fa8e40a2890 0x7fa8e40a4d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:15:48.498 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:48.497+0000 7fa909e6a740 1 -- 172.21.15.118:0/2479290391 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x560d9c743a40 msgr2=0x560d9c748540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:15:48.498 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:48.497+0000 7fa909e6a740 1 --2- 172.21.15.118:0/2479290391 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x560d9c743a40 0x560d9c748540 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7fa8f000bd10 tx=0x7fa8f000f730 comp rx=0 tx=0).stop 2024-03-24T00:15:48.498 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:48.497+0000 7fa909e6a740 1 -- 172.21.15.118:0/2479290391 shutdown_connections 2024-03-24T00:15:48.498 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:48.497+0000 7fa909e6a740 1 --2- 172.21.15.118:0/2479290391 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x560d9c748bb0 0x560d9c74b040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:15:48.499 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:48.497+0000 7fa909e6a740 1 --2- 172.21.15.118:0/2479290391 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x560d9c743a40 0x560d9c748540 unknown :-1 s=CLOSED pgs=202 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:15:48.499 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:48.497+0000 7fa909e6a740 1 --2- 172.21.15.118:0/2479290391 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x560d9c743050 0x560d9c743470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:15:48.499 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:48.497+0000 7fa909e6a740 1 -- 172.21.15.118:0/2479290391 >> 172.21.15.118:0/2479290391 conn(0x560d9c70b200 msgr2=0x560d9c73e9c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:15:48.499 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:48.497+0000 7fa909e6a740 1 -- 172.21.15.118:0/2479290391 shutdown_connections 2024-03-24T00:15:48.499 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:48.497+0000 7fa909e6a740 1 -- 172.21.15.118:0/2479290391 wait complete. 2024-03-24T00:15:48.499 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:48.497+0000 7fa905200640 1 --2- 172.21.15.118:0/273912562 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7fa8e40a2890 0x7fa8e40a4d50 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x560d9c7442a0 tx=0x7fa8f8009500 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:15:48.499 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:48.498+0000 7fa909e6a740 1 -- start start 2024-03-24T00:15:48.499 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:48.498+0000 7fa909e6a740 1 Processor -- start 2024-03-24T00:15:48.502 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:48.498+0000 7fa909e6a740 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x560d9c743050 0x560d9c74d090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:15:48.503 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:48.498+0000 7fa909e6a740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x560d9c743a40 0x560d9c74d5d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:15:48.503 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:48.498+0000 7fa909e6a740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x560d9c748bb0 0x560d9c754610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:15:48.503 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:48.498+0000 7fa909e6a740 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x560d9c74bb60 con 0x560d9c743050 2024-03-24T00:15:48.503 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:48.498+0000 7fa909e6a740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x560d9c663e80 con 0x560d9c743a40 2024-03-24T00:15:48.503 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:48.498+0000 7fa909e6a740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x560d9c74bce0 con 0x560d9c748bb0 2024-03-24T00:15:48.503 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:48.498+0000 7fa909e6a740 10 client.0 _ll_register_callbacks cb 0x560d9c676130 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T00:15:48.503 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:48.498+0000 7fa905200640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x560d9c748bb0 0x560d9c754610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:15:48.503 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:48.498+0000 7fa8ffe00640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x560d9c743050 0x560d9c74d090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:15:48.503 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:48.498+0000 7fa8ff400640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x560d9c743a40 0x560d9c74d5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:15:48.503 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:48.499+0000 7fa8ff400640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x560d9c743a40 0x560d9c74d5d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.118:36536/0 (socket says 172.21.15.118:36536) 2024-03-24T00:15:48.503 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:48.499+0000 7fa8ff400640 1 -- 172.21.15.118:0/2422641590 learned_addr learned my addr 172.21.15.118:0/2422641590 (peer_addr_for_me v2:172.21.15.118:0/0) 2024-03-24T00:15:48.503 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:48.499+0000 7fa8ff400640 1 -- 172.21.15.118:0/2422641590 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x560d9c748bb0 msgr2=0x560d9c754610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:15:48.503 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:48.499+0000 7fa8ff400640 1 --2- 172.21.15.118:0/2422641590 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x560d9c748bb0 0x560d9c754610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:15:48.504 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:48.499+0000 7fa8ff400640 1 -- 172.21.15.118:0/2422641590 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x560d9c743050 msgr2=0x560d9c74d090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:15:48.504 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:48.499+0000 7fa8ff400640 1 --2- 172.21.15.118:0/2422641590 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x560d9c743050 0x560d9c74d090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:15:48.504 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:48.499+0000 7fa8ff400640 1 -- 172.21.15.118:0/2422641590 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x560d9c4867d0 con 0x560d9c743a40 2024-03-24T00:15:48.504 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:48.499+0000 7fa8ff400640 1 --2- 172.21.15.118:0/2422641590 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x560d9c743a40 0x560d9c74d5d0 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7fa8ec0b9a30 tx=0x7fa8ec0b9270 comp rx=0 tx=0).ready entity=mon.2 client_cookie=84f79f7a2631db33 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:15:48.504 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:48.499+0000 7fa8ea000640 10 client.0 ms_handle_connect on v2:172.21.15.189:3300/0 2024-03-24T00:15:48.504 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:48.499+0000 7fa8ea000640 1 -- 172.21.15.118:0/2422641590 <== mon.2 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa8ec0bd3a0 con 0x560d9c743a40 2024-03-24T00:15:48.504 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:48.499+0000 7fa8ea000640 1 -- 172.21.15.118:0/2422641590 <== mon.2 v2:172.21.15.189:3300/0 2 ==== osd_map(90..90 src has 1..90) v4 ==== 8277+0+0 (secure 0 0 0) 0x7fa8ec0ee880 con 0x560d9c743a40 2024-03-24T00:15:48.504 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:48.499+0000 7fa909e6a740 1 -- 172.21.15.118:0/2422641590 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fa8f001a940 con 0x560d9c743a40 2024-03-24T00:15:48.504 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:48.500+0000 7fa8ea000640 1 -- 172.21.15.118:0/2422641590 <== mon.2 v2:172.21.15.189:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) v2 ==== 24+0+0 (secure 0 0 0) 0x7fa8ec0bcf30 con 0x560d9c743a40 2024-03-24T00:15:48.504 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:48.500+0000 7fa909e6a740 10 client.25403 fetch_fsmap learned FSMap version 28 2024-03-24T00:15:48.504 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:48.500+0000 7fa909e6a740 1 -- 172.21.15.118:0/2422641590 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) v3 -- 0x560d9c3bc3f0 con 0x560d9c743a40 2024-03-24T00:15:48.504 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:48.500+0000 7fa8ea000640 1 -- 172.21.15.118:0/2422641590 <== mon.2 v2:172.21.15.189:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fa8f001a940 con 0x560d9c743a40 2024-03-24T00:15:48.504 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:48.500+0000 7fa8ea000640 1 -- 172.21.15.118:0/2422641590 <== mon.2 v2:172.21.15.189:3300/0 5 ==== fsmap(e 28) v1 ==== 7934+0+0 (secure 0 0 0) 0x7fa8ec0f5040 con 0x560d9c743a40 2024-03-24T00:15:48.504 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:48.500+0000 7fa909e6a740 10 client.25403 fetch_fsmap finished waiting for FSMap version 28 2024-03-24T00:15:48.505 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:48.500+0000 7fa909e6a740 10 client.25403 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-24T00:15:48.505 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:48.500+0000 7fa909e6a740 20 client.25403 populate_metadata read hostname 'smithi118' 2024-03-24T00:15:48.505 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:48.500+0000 7fa909e6a740 1 --2- 172.21.15.118:0/2422641590 >> [v2:172.21.15.189:6838/4074484168,v1:172.21.15.189:6839/4074484168] conn(0x560d9c758e90 0x560d9c75b2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:15:48.505 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:48.500+0000 7fa909e6a740 4 client.25403 mds_command: new command op to 24569 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-24T00:15:48.505 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:48.500+0000 7fa909e6a740 1 -- 172.21.15.118:0/2422641590 --> [v2:172.21.15.189:6838/4074484168,v1:172.21.15.189:6839/4074484168] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x560d9c6b5790 con 0x560d9c758e90 2024-03-24T00:15:48.505 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:48.500+0000 7fa8ffe00640 1 --2- 172.21.15.118:0/2422641590 >> [v2:172.21.15.189:6838/4074484168,v1:172.21.15.189:6839/4074484168] conn(0x560d9c758e90 0x560d9c75b2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:15:48.505 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:48.501+0000 7fa8ea000640 10 client.25403 ms_handle_connect on v2:172.21.15.189:6838/4074484168 2024-03-24T00:15:48.505 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:48.501+0000 7fa8ffe00640 1 --2- 172.21.15.118:0/2422641590 >> [v2:172.21.15.189:6838/4074484168,v1:172.21.15.189:6839/4074484168] conn(0x560d9c758e90 0x560d9c75b2e0 crc :-1 s=READY pgs=35 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=4054ee398e7591b5 server_cookie=73cd096e651a0478 in_seq=0 out_seq=0 2024-03-24T00:15:48.505 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:48.502+0000 7fa8ea000640 1 -- 172.21.15.118:0/2422641590 <== mds.0 v2:172.21.15.189:6838/4074484168 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x560d9c6b5790 con 0x560d9c758e90 2024-03-24T00:15:48.505 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:48.502+0000 7fa8ea000640 10 client.25403 handle_command_reply: tid=0 2024-03-24T00:15:48.505 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:48.502+0000 7fa909e6a740 1 client.25403 shutdown 2024-03-24T00:15:48.505 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:48.502+0000 7fa909e6a740 1 -- 172.21.15.118:0/2422641590 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x560d9c743a40 msgr2=0x560d9c74d5d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:15:48.505 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:48.502+0000 7fa909e6a740 1 --2- 172.21.15.118:0/2422641590 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x560d9c743a40 0x560d9c74d5d0 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7fa8ec0b9a30 tx=0x7fa8ec0b9270 comp rx=0 tx=0).stop 2024-03-24T00:15:48.505 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:48.502+0000 7fa909e6a740 1 -- 172.21.15.118:0/2422641590 shutdown_connections 2024-03-24T00:15:48.505 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:48.502+0000 7fa909e6a740 1 --2- 172.21.15.118:0/2422641590 >> [v2:172.21.15.189:6838/4074484168,v1:172.21.15.189:6839/4074484168] conn(0x560d9c758e90 0x560d9c75b2e0 crc :-1 s=READY pgs=35 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:15:48.506 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:48.502+0000 7fa909e6a740 1 --2- 172.21.15.118:0/2422641590 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x560d9c748bb0 0x560d9c754610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:15:48.506 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:48.502+0000 7fa909e6a740 1 --2- 172.21.15.118:0/2422641590 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x560d9c743a40 0x560d9c74d5d0 unknown :-1 s=CLOSED pgs=208 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:15:48.506 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:48.502+0000 7fa909e6a740 1 --2- 172.21.15.118:0/2422641590 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x560d9c743050 0x560d9c74d090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:15:48.506 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:48.502+0000 7fa8ea000640 0 client.25403 ms_handle_reset on v2:172.21.15.189:6838/4074484168 2024-03-24T00:15:48.506 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:48.502+0000 7fa909e6a740 1 -- 172.21.15.118:0/2422641590 >> 172.21.15.118:0/2422641590 conn(0x560d9c70b200 msgr2=0x560d9c73dbc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:15:48.506 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:48.502+0000 7fa909e6a740 1 -- 172.21.15.118:0/2422641590 shutdown_connections 2024-03-24T00:15:48.506 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:48.502+0000 7fa909e6a740 1 -- 172.21.15.118:0/2422641590 wait complete. 2024-03-24T00:15:48.506 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:48.502+0000 7fa909e6a740 20 client.25403 trim_cache size 0 max 16384 2024-03-24T00:15:48.523 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:48.521+0000 7fa909e6a740 1 Processor -- start 2024-03-24T00:15:48.523 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:48.521+0000 7fa909e6a740 1 -- start start 2024-03-24T00:15:48.523 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:48.521+0000 7fa909e6a740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x560d9c743050 0x560d9c744bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:15:48.524 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:48.521+0000 7fa909e6a740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x560d9c743a40 0x560d9c7545d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:15:48.524 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:48.521+0000 7fa909e6a740 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x560d9c748bb0 0x560d9c741c00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:15:48.524 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:48.521+0000 7fa909e6a740 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa9001aaa60 con 0x560d9c748bb0 2024-03-24T00:15:48.524 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:48.521+0000 7fa909e6a740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa9001ac3e0 con 0x560d9c743a40 2024-03-24T00:15:48.524 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:48.521+0000 7fa909e6a740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa9001ab0c0 con 0x560d9c743050 2024-03-24T00:15:48.524 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:48.522+0000 7fa8ff400640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x560d9c743a40 0x560d9c7545d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:15:48.524 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:48.522+0000 7fa8ff400640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x560d9c743a40 0x560d9c7545d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.118:36550/0 (socket says 172.21.15.118:36550) 2024-03-24T00:15:48.524 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:48.522+0000 7fa8ff400640 1 -- 172.21.15.118:0/3476742457 learned_addr learned my addr 172.21.15.118:0/3476742457 (peer_addr_for_me v2:172.21.15.118:0/0) 2024-03-24T00:15:48.524 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:48.522+0000 7fa8ffe00640 1 --2- 172.21.15.118:0/3476742457 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x560d9c743050 0x560d9c744bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:15:48.524 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:48.522+0000 7fa905200640 1 --2- 172.21.15.118:0/3476742457 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x560d9c748bb0 0x560d9c741c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:15:48.524 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:48.522+0000 7fa8ff400640 1 -- 172.21.15.118:0/3476742457 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x560d9c743050 msgr2=0x560d9c744bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:15:48.524 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:48.522+0000 7fa8ff400640 1 --2- 172.21.15.118:0/3476742457 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x560d9c743050 0x560d9c744bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:15:48.524 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:48.522+0000 7fa8ff400640 1 -- 172.21.15.118:0/3476742457 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x560d9c748bb0 msgr2=0x560d9c741c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:15:48.524 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:48.522+0000 7fa8ff400640 1 --2- 172.21.15.118:0/3476742457 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x560d9c748bb0 0x560d9c741c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:15:48.525 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:48.522+0000 7fa8ff400640 1 -- 172.21.15.118:0/3476742457 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x560d9c37c0f0 con 0x560d9c743a40 2024-03-24T00:15:48.525 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:48.522+0000 7fa8ff400640 1 --2- 172.21.15.118:0/3476742457 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x560d9c743a40 0x560d9c7545d0 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7fa8ec0bd2d0 tx=0x7fa8ec0bd710 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d353c2c9f2e715a6 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:15:48.525 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:48.522+0000 7fa8f4c00640 1 -- 172.21.15.118:0/3476742457 <== mon.2 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa8ec0bd110 con 0x560d9c743a40 2024-03-24T00:15:48.525 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:48.522+0000 7fa8f4c00640 1 -- 172.21.15.118:0/3476742457 <== mon.2 v2:172.21.15.189:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fa8ec0044a0 con 0x560d9c743a40 2024-03-24T00:15:48.525 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:48.523+0000 7fa8f4c00640 1 -- 172.21.15.118:0/3476742457 <== mon.2 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa8ec0ee7f0 con 0x560d9c743a40 2024-03-24T00:15:48.525 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:48.523+0000 7fa909e6a740 1 -- 172.21.15.118:0/3476742457 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x560d9c743a40 msgr2=0x560d9c7545d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:15:48.525 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:48.523+0000 7fa909e6a740 1 --2- 172.21.15.118:0/3476742457 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x560d9c743a40 0x560d9c7545d0 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7fa8ec0bd2d0 tx=0x7fa8ec0bd710 comp rx=0 tx=0).stop 2024-03-24T00:15:48.525 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:48.523+0000 7fa909e6a740 1 -- 172.21.15.118:0/3476742457 shutdown_connections 2024-03-24T00:15:48.525 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:48.523+0000 7fa909e6a740 1 --2- 172.21.15.118:0/3476742457 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x560d9c748bb0 0x560d9c741c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:15:48.525 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:48.523+0000 7fa909e6a740 1 --2- 172.21.15.118:0/3476742457 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x560d9c743a40 0x560d9c7545d0 unknown :-1 s=CLOSED pgs=209 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:15:48.525 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:48.523+0000 7fa909e6a740 1 --2- 172.21.15.118:0/3476742457 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x560d9c743050 0x560d9c744bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:15:48.525 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:48.523+0000 7fa909e6a740 1 -- 172.21.15.118:0/3476742457 >> 172.21.15.118:0/3476742457 conn(0x560d9c70b200 msgr2=0x560d9c7444e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:15:48.525 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:48.523+0000 7fa909e6a740 1 -- 172.21.15.118:0/3476742457 shutdown_connections 2024-03-24T00:15:48.525 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:48.523+0000 7fa909e6a740 1 -- 172.21.15.118:0/3476742457 wait complete. 2024-03-24T00:15:48.526 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:48.524+0000 7fa909e6a740 1 -- start start 2024-03-24T00:15:48.526 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:48.524+0000 7fa909e6a740 1 Processor -- start 2024-03-24T00:15:48.529 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:48.524+0000 7fa909e6a740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x560d9c743050 0x560d9c751fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:15:48.529 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:48.524+0000 7fa909e6a740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x560d9c743a40 0x560d9c7524f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:15:48.529 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:48.524+0000 7fa909e6a740 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x560d9c748bb0 0x560d9c753500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:15:48.529 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:48.525+0000 7fa909e6a740 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa9001abd80 con 0x560d9c748bb0 2024-03-24T00:15:48.529 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:48.525+0000 7fa909e6a740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa9001ab720 con 0x560d9c743050 2024-03-24T00:15:48.529 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:48.525+0000 7fa909e6a740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa9001a8c20 con 0x560d9c743a40 2024-03-24T00:15:48.529 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:48.525+0000 7fa909e6a740 10 client.0 _ll_register_callbacks cb 0x7fa8ec005b50 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T00:15:48.530 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:48.525+0000 7fa8ffe00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x560d9c743050 0x560d9c751fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:15:48.530 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:48.525+0000 7fa905200640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x560d9c748bb0 0x560d9c753500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:15:48.530 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:48.525+0000 7fa8ffe00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x560d9c743050 0x560d9c751fb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.118:36552/0 (socket says 172.21.15.118:36552) 2024-03-24T00:15:48.530 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:48.525+0000 7fa8ffe00640 1 -- 172.21.15.118:0/2749030060 learned_addr learned my addr 172.21.15.118:0/2749030060 (peer_addr_for_me v2:172.21.15.118:0/0) 2024-03-24T00:15:48.530 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:48.525+0000 7fa8ff400640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x560d9c743a40 0x560d9c7524f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:15:48.530 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:48.525+0000 7fa8ffe00640 1 -- 172.21.15.118:0/2749030060 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x560d9c743a40 msgr2=0x560d9c7524f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:15:48.530 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:48.525+0000 7fa8ffe00640 1 --2- 172.21.15.118:0/2749030060 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x560d9c743a40 0x560d9c7524f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:15:48.530 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:48.525+0000 7fa8ffe00640 1 -- 172.21.15.118:0/2749030060 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x560d9c748bb0 msgr2=0x560d9c753500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:15:48.530 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:48.525+0000 7fa8ffe00640 1 --2- 172.21.15.118:0/2749030060 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x560d9c748bb0 0x560d9c753500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:15:48.530 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:48.525+0000 7fa8ffe00640 1 -- 172.21.15.118:0/2749030060 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x560d9c753af0 con 0x560d9c743050 2024-03-24T00:15:48.530 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:48.525+0000 7fa8ff400640 1 --2- 172.21.15.118:0/2749030060 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x560d9c743a40 0x560d9c7524f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T00:15:48.530 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:48.525+0000 7fa905200640 1 --2- 172.21.15.118:0/2749030060 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x560d9c748bb0 0x560d9c753500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T00:15:48.530 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:48.525+0000 7fa8ffe00640 1 --2- 172.21.15.118:0/2749030060 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x560d9c743050 0x560d9c751fb0 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7fa8f000bc30 tx=0x7fa8f000b420 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d529a5ca6de3f72d server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:15:48.531 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:48.525+0000 7fa8ea000640 10 client.0 ms_handle_connect on v2:172.21.15.189:3300/0 2024-03-24T00:15:48.531 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:48.526+0000 7fa8ea000640 1 -- 172.21.15.118:0/2749030060 <== mon.2 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa8f0015040 con 0x560d9c743050 2024-03-24T00:15:48.531 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:48.526+0000 7fa8ea000640 1 -- 172.21.15.118:0/2749030060 <== mon.2 v2:172.21.15.189:3300/0 2 ==== osd_map(90..90 src has 1..90) v4 ==== 8277+0+0 (secure 0 0 0) 0x7fa8f0005240 con 0x560d9c743050 2024-03-24T00:15:48.531 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:48.526+0000 7fa909e6a740 1 -- 172.21.15.118:0/2749030060 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x560d9c6afae0 con 0x560d9c743050 2024-03-24T00:15:48.531 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:48.526+0000 7fa8ea000640 1 -- 172.21.15.118:0/2749030060 <== mon.2 v2:172.21.15.189:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) v2 ==== 24+0+0 (secure 0 0 0) 0x7fa8f00272e0 con 0x560d9c743050 2024-03-24T00:15:48.531 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:48.526+0000 7fa909e6a740 10 client.25409 fetch_fsmap learned FSMap version 28 2024-03-24T00:15:48.531 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:48.526+0000 7fa909e6a740 1 -- 172.21.15.118:0/2749030060 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) v3 -- 0x560d9c7b4ab0 con 0x560d9c743050 2024-03-24T00:15:48.531 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:48.526+0000 7fa8ea000640 1 -- 172.21.15.118:0/2749030060 <== mon.2 v2:172.21.15.189:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x560d9c6afae0 con 0x560d9c743050 2024-03-24T00:15:48.531 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:48.526+0000 7fa8ea000640 1 -- 172.21.15.118:0/2749030060 <== mon.2 v2:172.21.15.189:3300/0 5 ==== fsmap(e 28) v1 ==== 7934+0+0 (secure 0 0 0) 0x7fa8f0004760 con 0x560d9c743050 2024-03-24T00:15:48.531 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:48.526+0000 7fa909e6a740 10 client.25409 fetch_fsmap finished waiting for FSMap version 28 2024-03-24T00:15:48.531 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:48.526+0000 7fa909e6a740 10 client.25409 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-24T00:15:48.531 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:48.526+0000 7fa909e6a740 20 client.25409 populate_metadata read hostname 'smithi118' 2024-03-24T00:15:48.531 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:48.526+0000 7fa909e6a740 1 --2- 172.21.15.118:0/2749030060 >> [v2:172.21.15.189:6838/4074484168,v1:172.21.15.189:6839/4074484168] conn(0x560d9c7b5150 0x560d9c7b7570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:15:48.531 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:48.526+0000 7fa909e6a740 4 client.25409 mds_command: new command op to 24569 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-24T00:15:48.532 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:48.526+0000 7fa909e6a740 1 -- 172.21.15.118:0/2749030060 --> [v2:172.21.15.189:6838/4074484168,v1:172.21.15.189:6839/4074484168] -- command(tid 0: {"prefix": "scrub status"}) v1 -- 0x560d9c62d3a0 con 0x560d9c7b5150 2024-03-24T00:15:48.532 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:48.527+0000 7fa8ff400640 1 --2- 172.21.15.118:0/2749030060 >> [v2:172.21.15.189:6838/4074484168,v1:172.21.15.189:6839/4074484168] conn(0x560d9c7b5150 0x560d9c7b7570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:15:48.532 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:48.527+0000 7fa8ea000640 10 client.25409 ms_handle_connect on v2:172.21.15.189:6838/4074484168 2024-03-24T00:15:48.532 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:48.527+0000 7fa8ff400640 1 --2- 172.21.15.118:0/2749030060 >> [v2:172.21.15.189:6838/4074484168,v1:172.21.15.189:6839/4074484168] conn(0x560d9c7b5150 0x560d9c7b7570 crc :-1 s=READY pgs=36 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=d0741b712b9e27f9 server_cookie=703179e53faf682b in_seq=0 out_seq=0 2024-03-24T00:15:48.532 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:48.527+0000 7fa8ea000640 1 -- 172.21.15.118:0/2749030060 <== mds.0 v2:172.21.15.189:6838/4074484168 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+259 (crc 0 0 0) 0x560d9c62d3a0 con 0x560d9c7b5150 2024-03-24T00:15:48.532 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:48.527+0000 7fa8ea000640 10 client.25409 handle_command_reply: tid=0 2024-03-24T00:15:48.532 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:48.527+0000 7fa909e6a740 1 client.25409 shutdown 2024-03-24T00:15:48.532 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:48.528+0000 7fa909e6a740 1 -- 172.21.15.118:0/2749030060 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x560d9c743050 msgr2=0x560d9c751fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:15:48.532 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:48.528+0000 7fa909e6a740 1 --2- 172.21.15.118:0/2749030060 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x560d9c743050 0x560d9c751fb0 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7fa8f000bc30 tx=0x7fa8f000b420 comp rx=0 tx=0).stop 2024-03-24T00:15:48.532 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:48.528+0000 7fa909e6a740 1 -- 172.21.15.118:0/2749030060 shutdown_connections 2024-03-24T00:15:48.532 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:48.528+0000 7fa909e6a740 1 --2- 172.21.15.118:0/2749030060 >> [v2:172.21.15.189:6838/4074484168,v1:172.21.15.189:6839/4074484168] conn(0x560d9c7b5150 0x560d9c7b7570 crc :-1 s=READY pgs=36 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:15:48.532 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:48.528+0000 7fa909e6a740 1 --2- 172.21.15.118:0/2749030060 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x560d9c748bb0 0x560d9c753500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:15:48.532 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:48.528+0000 7fa8ea000640 0 client.25409 ms_handle_reset on v2:172.21.15.189:6838/4074484168 2024-03-24T00:15:48.532 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:48.528+0000 7fa909e6a740 1 --2- 172.21.15.118:0/2749030060 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x560d9c743a40 0x560d9c7524f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:15:48.533 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:48.528+0000 7fa909e6a740 1 --2- 172.21.15.118:0/2749030060 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x560d9c743050 0x560d9c751fb0 unknown :-1 s=CLOSED pgs=210 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:15:48.533 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:48.528+0000 7fa909e6a740 1 -- 172.21.15.118:0/2749030060 >> 172.21.15.118:0/2749030060 conn(0x560d9c70b200 msgr2=0x560d9c744d70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:15:48.533 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:48.528+0000 7fa909e6a740 1 -- 172.21.15.118:0/2749030060 shutdown_connections 2024-03-24T00:15:48.533 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:48.528+0000 7fa909e6a740 1 -- 172.21.15.118:0/2749030060 wait complete. 2024-03-24T00:15:48.533 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:48.528+0000 7fa909e6a740 20 client.25409 trim_cache size 0 max 16384 2024-03-24T00:15:48.533 INFO:teuthology.orchestra.run.smithi118.stdout:{ 2024-03-24T00:15:48.533 INFO:teuthology.orchestra.run.smithi118.stdout: "status": "IDLE (waiting for more scrubs)", 2024-03-24T00:15:48.533 INFO:teuthology.orchestra.run.smithi118.stdout: "scrubs": { 2024-03-24T00:15:48.533 INFO:teuthology.orchestra.run.smithi118.stdout: "89b9d04e-be57-42e0-abce-bad803301c34": { 2024-03-24T00:15:48.533 INFO:teuthology.orchestra.run.smithi118.stdout: "path": "/", 2024-03-24T00:15:48.533 INFO:teuthology.orchestra.run.smithi118.stdout: "tag": "89b9d04e-be57-42e0-abce-bad803301c34", 2024-03-24T00:15:48.534 INFO:teuthology.orchestra.run.smithi118.stdout: "options": "recursive,force" 2024-03-24T00:15:48.534 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-03-24T00:15:48.534 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-03-24T00:15:48.534 INFO:teuthology.orchestra.run.smithi118.stdout:} 2024-03-24T00:15:48.534 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:48.529+0000 7fa905c00640 1 -- 172.21.15.118:0/273912562 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7fa8e40a2890 msgr2=0x7fa8e40a4d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:15:48.535 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:48.529+0000 7fa905c00640 1 --2- 172.21.15.118:0/273912562 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7fa8e40a2890 0x7fa8e40a4d50 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x560d9c7442a0 tx=0x7fa8f8009500 comp rx=0 tx=0).stop 2024-03-24T00:15:48.535 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:48.529+0000 7fa905c00640 1 -- 172.21.15.118:0/273912562 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fa900102790 msgr2=0x7fa900199fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:15:48.535 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:48.529+0000 7fa905c00640 1 --2- 172.21.15.118:0/273912562 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fa900102790 0x7fa900199fa0 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7fa8ec005ab0 tx=0x7fa8ec02c6c0 comp rx=0 tx=0).stop 2024-03-24T00:15:48.535 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:48.529+0000 7fa905c00640 1 -- 172.21.15.118:0/273912562 shutdown_connections 2024-03-24T00:15:48.535 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:48.529+0000 7fa905c00640 1 --2- 172.21.15.118:0/273912562 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7fa8e40a2890 0x7fa8e40a4d50 unknown :-1 s=CLOSED pgs=270 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:15:48.535 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:48.529+0000 7fa905c00640 1 --2- 172.21.15.118:0/273912562 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa900103150 0x7fa9001a1010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:15:48.535 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:48.529+0000 7fa905c00640 1 --2- 172.21.15.118:0/273912562 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fa900102790 0x7fa900199fa0 unknown :-1 s=CLOSED pgs=201 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:15:48.535 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:48.529+0000 7fa905c00640 1 --2- 172.21.15.118:0/273912562 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fa9001015a0 0x7fa900199a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:15:48.535 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:48.529+0000 7fa905c00640 1 -- 172.21.15.118:0/273912562 >> 172.21.15.118:0/273912562 conn(0x7fa9000fcb10 msgr2=0x7fa9000738d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:15:48.535 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:48.529+0000 7fa905c00640 1 -- 172.21.15.118:0/273912562 shutdown_connections 2024-03-24T00:15:48.535 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:15:48.529+0000 7fa905c00640 1 -- 172.21.15.118:0/273912562 wait complete. 2024-03-24T00:15:48.545 INFO:tasks.cephfs.filesystem:scrub status for tag:89b9d04e-be57-42e0-abce-bad803301c34 - {'path': '/', 'tag': '89b9d04e-be57-42e0-abce-bad803301c34', 'options': 'recursive,force'} 2024-03-24T00:15:49.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:15:48 smithi118 ceph-mon[31535]: from='client.? 172.21.15.118:0/2948502676' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-24T00:15:49.203 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:15:48 smithi189 ceph-mon[36187]: from='client.? 172.21.15.118:0/2948502676' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-24T00:15:49.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:15:48 smithi193 ceph-mon[36878]: from='client.? 172.21.15.118:0/2948502676' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-24T00:15:50.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:15:49 smithi118 ceph-mon[31535]: scrub queued for path: / 2024-03-24T00:15:50.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:15:49 smithi118 ceph-mon[31535]: scrub summary: idle+waiting paths [/] 2024-03-24T00:15:50.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:15:49 smithi118 ceph-mon[31535]: scrub summary: active paths [/] 2024-03-24T00:15:50.197 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:15:49 smithi118 ceph-mon[31535]: scrub summary: idle+waiting paths [/] 2024-03-24T00:15:50.197 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:15:49 smithi118 ceph-mon[31535]: pgmap v402: 97 pgs: 97 active+clean; 659 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 7.3 KiB/s rd, 2.1 KiB/s wr, 13 op/s 2024-03-24T00:15:50.203 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:15:49 smithi189 ceph-mon[36187]: scrub queued for path: / 2024-03-24T00:15:50.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:15:49 smithi189 ceph-mon[36187]: scrub summary: idle+waiting paths [/] 2024-03-24T00:15:50.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:15:49 smithi189 ceph-mon[36187]: scrub summary: active paths [/] 2024-03-24T00:15:50.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:15:49 smithi189 ceph-mon[36187]: scrub summary: idle+waiting paths [/] 2024-03-24T00:15:50.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:15:49 smithi189 ceph-mon[36187]: pgmap v402: 97 pgs: 97 active+clean; 659 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 7.3 KiB/s rd, 2.1 KiB/s wr, 13 op/s 2024-03-24T00:15:50.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:15:49 smithi193 ceph-mon[36878]: scrub queued for path: / 2024-03-24T00:15:50.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:15:49 smithi193 ceph-mon[36878]: scrub summary: idle+waiting paths [/] 2024-03-24T00:15:50.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:15:49 smithi193 ceph-mon[36878]: scrub summary: active paths [/] 2024-03-24T00:15:50.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:15:49 smithi193 ceph-mon[36878]: scrub summary: idle+waiting paths [/] 2024-03-24T00:15:50.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:15:49 smithi193 ceph-mon[36878]: pgmap v402: 97 pgs: 97 active+clean; 659 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 7.3 KiB/s rd, 2.1 KiB/s wr, 13 op/s 2024-03-24T00:15:53.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:15:52 smithi118 ceph-mon[31535]: pgmap v403: 97 pgs: 97 active+clean; 659 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 5.6 KiB/s rd, 1.8 KiB/s wr, 10 op/s 2024-03-24T00:15:53.203 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:15:52 smithi189 ceph-mon[36187]: pgmap v403: 97 pgs: 97 active+clean; 659 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 5.6 KiB/s rd, 1.8 KiB/s wr, 10 op/s 2024-03-24T00:15:53.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:15:52 smithi193 ceph-mon[36878]: pgmap v403: 97 pgs: 97 active+clean; 659 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 5.6 KiB/s rd, 1.8 KiB/s wr, 10 op/s 2024-03-24T00:15:55.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:15:54 smithi118 ceph-mon[31535]: pgmap v404: 97 pgs: 97 active+clean; 660 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 6.6 KiB/s rd, 2.1 KiB/s wr, 11 op/s 2024-03-24T00:15:55.203 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:15:54 smithi189 ceph-mon[36187]: pgmap v404: 97 pgs: 97 active+clean; 660 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 6.6 KiB/s rd, 2.1 KiB/s wr, 11 op/s 2024-03-24T00:15:55.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:15:54 smithi193 ceph-mon[36878]: pgmap v404: 97 pgs: 97 active+clean; 660 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 6.6 KiB/s rd, 2.1 KiB/s wr, 11 op/s 2024-03-24T00:15:56.998 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:15:56 smithi118 ceph-mon[31535]: pgmap v405: 97 pgs: 97 active+clean; 660 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 7.0 KiB/s rd, 1.9 KiB/s wr, 12 op/s 2024-03-24T00:15:57.203 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:15:56 smithi189 ceph-mon[36187]: pgmap v405: 97 pgs: 97 active+clean; 660 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 7.0 KiB/s rd, 1.9 KiB/s wr, 12 op/s 2024-03-24T00:15:57.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:15:56 smithi193 ceph-mon[36878]: pgmap v405: 97 pgs: 97 active+clean; 660 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 7.0 KiB/s rd, 1.9 KiB/s wr, 12 op/s 2024-03-24T00:15:59.006 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:15:58 smithi189 ceph-mon[36187]: pgmap v406: 97 pgs: 97 active+clean; 660 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 938 B/s wr, 10 op/s 2024-03-24T00:15:59.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:15:58 smithi118 ceph-mon[31535]: pgmap v406: 97 pgs: 97 active+clean; 660 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 938 B/s wr, 10 op/s 2024-03-24T00:15:59.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:15:58 smithi193 ceph-mon[36878]: pgmap v406: 97 pgs: 97 active+clean; 660 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 KiB/s rd, 938 B/s wr, 10 op/s 2024-03-24T00:16:01.195 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:16:00 smithi118 ceph-mon[31535]: pgmap v407: 97 pgs: 97 active+clean; 660 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 6.7 KiB/s rd, 938 B/s wr, 12 op/s 2024-03-24T00:16:01.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:16:00 smithi118 ceph-mon[31535]: scrub summary: idle 2024-03-24T00:16:01.203 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:16:00 smithi189 ceph-mon[36187]: pgmap v407: 97 pgs: 97 active+clean; 660 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 6.7 KiB/s rd, 938 B/s wr, 12 op/s 2024-03-24T00:16:01.203 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:16:00 smithi189 ceph-mon[36187]: scrub summary: idle 2024-03-24T00:16:01.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:16:00 smithi193 ceph-mon[36878]: pgmap v407: 97 pgs: 97 active+clean; 660 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 6.7 KiB/s rd, 938 B/s wr, 12 op/s 2024-03-24T00:16:01.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:16:00 smithi193 ceph-mon[36878]: scrub summary: idle 2024-03-24T00:16:01.212 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:16:01.211+0000 7fbe55200640 1 -- 172.21.15.118:0/3544209980 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fbe501098b0 msgr2=0x7fbe50110140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:16:01.212 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:16:01.211+0000 7fbe55200640 1 --2- 172.21.15.118:0/3544209980 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fbe501098b0 0x7fbe50110140 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7fbe48009ca0 tx=0x7fbe4802c0f0 comp rx=0 tx=0).stop 2024-03-24T00:16:01.212 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:16:01.211+0000 7fbe55200640 1 -- 172.21.15.118:0/3544209980 shutdown_connections 2024-03-24T00:16:01.212 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:16:01.211+0000 7fbe55200640 1 --2- 172.21.15.118:0/3544209980 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fbe501098b0 0x7fbe50110140 unknown :-1 s=CLOSED pgs=203 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:16:01.212 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:16:01.211+0000 7fbe55200640 1 --2- 172.21.15.118:0/3544209980 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fbe50108ef0 0x7fbe50109370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:16:01.212 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:16:01.211+0000 7fbe55200640 1 --2- 172.21.15.118:0/3544209980 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fbe50107d00 0x7fbe50108120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:16:01.213 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:16:01.211+0000 7fbe55200640 1 -- 172.21.15.118:0/3544209980 >> 172.21.15.118:0/3544209980 conn(0x7fbe50076cb0 msgr2=0x7fbe500790d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:16:01.213 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:16:01.211+0000 7fbe55200640 1 -- 172.21.15.118:0/3544209980 shutdown_connections 2024-03-24T00:16:01.213 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:16:01.211+0000 7fbe55200640 1 -- 172.21.15.118:0/3544209980 wait complete. 2024-03-24T00:16:01.215 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:16:01.212+0000 7fbe55200640 1 Processor -- start 2024-03-24T00:16:01.215 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:16:01.212+0000 7fbe55200640 1 -- start start 2024-03-24T00:16:01.215 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:16:01.213+0000 7fbe55200640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fbe50107d00 0x7fbe5019df30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:16:01.215 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:16:01.213+0000 7fbe55200640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fbe50108ef0 0x7fbe5019e470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:16:01.215 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:16:01.213+0000 7fbe55200640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fbe501098b0 0x7fbe501a54e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:16:01.215 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:16:01.213+0000 7fbe55200640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbe50113980 con 0x7fbe50108ef0 2024-03-24T00:16:01.215 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:16:01.213+0000 7fbe55200640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbe50113800 con 0x7fbe50107d00 2024-03-24T00:16:01.215 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:16:01.213+0000 7fbe55200640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbe50113b00 con 0x7fbe501098b0 2024-03-24T00:16:01.215 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:16:01.213+0000 7fbe4de00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fbe50107d00 0x7fbe5019df30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:16:01.215 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:16:01.213+0000 7fbe4de00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fbe50107d00 0x7fbe5019df30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.118:59300/0 (socket says 172.21.15.118:59300) 2024-03-24T00:16:01.216 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:16:01.213+0000 7fbe4de00640 1 -- 172.21.15.118:0/2694958437 learned_addr learned my addr 172.21.15.118:0/2694958437 (peer_addr_for_me v2:172.21.15.118:0/0) 2024-03-24T00:16:01.216 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:16:01.213+0000 7fbe4d400640 1 --2- 172.21.15.118:0/2694958437 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fbe50108ef0 0x7fbe5019e470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:16:01.216 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:16:01.213+0000 7fbe4e800640 1 --2- 172.21.15.118:0/2694958437 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fbe501098b0 0x7fbe501a54e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:16:01.216 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:16:01.213+0000 7fbe4de00640 1 -- 172.21.15.118:0/2694958437 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fbe501098b0 msgr2=0x7fbe501a54e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:16:01.216 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:16:01.213+0000 7fbe4de00640 1 --2- 172.21.15.118:0/2694958437 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fbe501098b0 0x7fbe501a54e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:16:01.216 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:16:01.213+0000 7fbe4de00640 1 -- 172.21.15.118:0/2694958437 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fbe50108ef0 msgr2=0x7fbe5019e470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:16:01.216 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:16:01.213+0000 7fbe4de00640 1 --2- 172.21.15.118:0/2694958437 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fbe50108ef0 0x7fbe5019e470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:16:01.216 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:16:01.213+0000 7fbe4de00640 1 -- 172.21.15.118:0/2694958437 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbe501a5b50 con 0x7fbe50107d00 2024-03-24T00:16:01.216 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:16:01.214+0000 7fbe4de00640 1 --2- 172.21.15.118:0/2694958437 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fbe50107d00 0x7fbe5019df30 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7fbe4000e9c0 tx=0x7fbe4000ee80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:16:01.216 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:16:01.214+0000 7fbe46a00640 1 -- 172.21.15.118:0/2694958437 <== mon.2 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbe4000cde0 con 0x7fbe50107d00 2024-03-24T00:16:01.216 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:16:01.214+0000 7fbe46a00640 1 -- 172.21.15.118:0/2694958437 <== mon.2 v2:172.21.15.189:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fbe40010910 con 0x7fbe50107d00 2024-03-24T00:16:01.216 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:16:01.214+0000 7fbe46a00640 1 -- 172.21.15.118:0/2694958437 <== mon.2 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbe40018a00 con 0x7fbe50107d00 2024-03-24T00:16:01.216 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:16:01.214+0000 7fbe55200640 1 -- 172.21.15.118:0/2694958437 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbe501a5e40 con 0x7fbe50107d00 2024-03-24T00:16:01.216 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:16:01.214+0000 7fbe55200640 1 -- 172.21.15.118:0/2694958437 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbe501a6250 con 0x7fbe50107d00 2024-03-24T00:16:01.218 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:16:01.217+0000 7fbe46a00640 1 -- 172.21.15.118:0/2694958437 <== mon.2 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 19) v1 ==== 137342+0+0 (secure 0 0 0) 0x7fbe4001f020 con 0x7fbe50107d00 2024-03-24T00:16:01.220 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:16:01.218+0000 7fbe57e74740 1 Processor -- start 2024-03-24T00:16:01.220 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:16:01.219+0000 7fbe46a00640 1 --2- 172.21.15.118:0/2694958437 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7fbe340a29c0 0x7fbe340a4e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:16:01.221 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:16:01.219+0000 7fbe57e74740 1 -- start start 2024-03-24T00:16:01.221 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:16:01.219+0000 7fbe46a00640 1 -- 172.21.15.118:0/2694958437 <== mon.2 v2:172.21.15.189:3300/0 5 ==== osd_map(90..90 src has 1..90) v4 ==== 8277+0+0 (secure 0 0 0) 0x7fbe400cede0 con 0x7fbe50107d00 2024-03-24T00:16:01.221 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:16:01.219+0000 7fbe4e800640 1 --2- 172.21.15.118:0/2694958437 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7fbe340a29c0 0x7fbe340a4e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:16:01.221 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:16:01.219+0000 7fbe57e74740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55ed0d4c61e0 0x55ed0d4c6600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:16:01.221 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:16:01.219+0000 7fbe57e74740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55ed0d4c6bd0 0x55ed0d4cb6d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:16:01.221 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:16:01.219+0000 7fbe57e74740 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55ed0d4cbd40 0x55ed0d4ce1d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:16:01.221 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:16:01.219+0000 7fbe57e74740 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ed0d4893b0 con 0x55ed0d4cbd40 2024-03-24T00:16:01.221 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:16:01.219+0000 7fbe57e74740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ed0d42b830 con 0x55ed0d4c61e0 2024-03-24T00:16:01.221 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:16:01.219+0000 7fbe57e74740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ed0d3d45d0 con 0x55ed0d4c6bd0 2024-03-24T00:16:01.221 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:16:01.219+0000 7fbe4de00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55ed0d4c61e0 0x55ed0d4c6600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:16:01.221 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:16:01.219+0000 7fbe4de00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55ed0d4c61e0 0x55ed0d4c6600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.118:59314/0 (socket says 172.21.15.118:59314) 2024-03-24T00:16:01.221 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:16:01.219+0000 7fbe4de00640 1 -- 172.21.15.118:0/1635275209 learned_addr learned my addr 172.21.15.118:0/1635275209 (peer_addr_for_me v2:172.21.15.118:0/0) 2024-03-24T00:16:01.221 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:16:01.220+0000 7fbe4e800640 1 --2- 172.21.15.118:0/1635275209 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55ed0d4cbd40 0x55ed0d4ce1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:16:01.222 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:16:01.220+0000 7fbe4de00640 1 -- 172.21.15.118:0/1635275209 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55ed0d4c6bd0 msgr2=0x55ed0d4cb6d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:16:01.222 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:16:01.220+0000 7fbe4d400640 1 --2- 172.21.15.118:0/1635275209 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55ed0d4c6bd0 0x55ed0d4cb6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:16:01.222 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:16:01.220+0000 7fbe4de00640 1 --2- 172.21.15.118:0/1635275209 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55ed0d4c6bd0 0x55ed0d4cb6d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:16:01.222 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:16:01.220+0000 7fbe4de00640 1 -- 172.21.15.118:0/1635275209 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55ed0d4cbd40 msgr2=0x55ed0d4ce1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:16:01.222 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:16:01.220+0000 7fbe4de00640 1 --2- 172.21.15.118:0/1635275209 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55ed0d4cbd40 0x55ed0d4ce1d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:16:01.222 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:16:01.220+0000 7fbe4de00640 1 -- 172.21.15.118:0/1635275209 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55ed0d3ec2c0 con 0x55ed0d4c61e0 2024-03-24T00:16:01.222 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:16:01.220+0000 7fbe4d400640 1 --2- 172.21.15.118:0/1635275209 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55ed0d4c6bd0 0x55ed0d4cb6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-24T00:16:01.222 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:16:01.220+0000 7fbe4de00640 1 --2- 172.21.15.118:0/1635275209 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55ed0d4c61e0 0x55ed0d4c6600 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7fbe40097f60 tx=0x7fbe4009afc0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f78db6189328fca5 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:16:01.222 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:16:01.220+0000 7fbe4e800640 1 --2- 172.21.15.118:0/2694958437 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7fbe340a29c0 0x7fbe340a4e80 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7fbe4802c630 tx=0x7fbe4802ce50 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:16:01.226 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:16:01.220+0000 7fbe3b400640 1 -- 172.21.15.118:0/1635275209 <== mon.2 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbe4009b590 con 0x55ed0d4c61e0 2024-03-24T00:16:01.226 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:16:01.220+0000 7fbe3b400640 1 -- 172.21.15.118:0/1635275209 <== mon.2 v2:172.21.15.189:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fbe4009f6e0 con 0x55ed0d4c61e0 2024-03-24T00:16:01.226 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:16:01.220+0000 7fbe3b400640 1 -- 172.21.15.118:0/1635275209 <== mon.2 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbe40010ac0 con 0x55ed0d4c61e0 2024-03-24T00:16:01.226 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:16:01.221+0000 7fbe57e74740 1 -- 172.21.15.118:0/1635275209 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55ed0d4c61e0 msgr2=0x55ed0d4c6600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:16:01.226 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:16:01.221+0000 7fbe57e74740 1 --2- 172.21.15.118:0/1635275209 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55ed0d4c61e0 0x55ed0d4c6600 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7fbe40097f60 tx=0x7fbe4009afc0 comp rx=0 tx=0).stop 2024-03-24T00:16:01.226 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:16:01.221+0000 7fbe57e74740 1 -- 172.21.15.118:0/1635275209 shutdown_connections 2024-03-24T00:16:01.226 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:16:01.221+0000 7fbe57e74740 1 --2- 172.21.15.118:0/1635275209 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55ed0d4cbd40 0x55ed0d4ce1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:16:01.226 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:16:01.221+0000 7fbe57e74740 1 --2- 172.21.15.118:0/1635275209 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55ed0d4c6bd0 0x55ed0d4cb6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:16:01.226 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:16:01.221+0000 7fbe57e74740 1 --2- 172.21.15.118:0/1635275209 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55ed0d4c61e0 0x55ed0d4c6600 unknown :-1 s=CLOSED pgs=212 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:16:01.226 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:16:01.221+0000 7fbe57e74740 1 -- 172.21.15.118:0/1635275209 >> 172.21.15.118:0/1635275209 conn(0x55ed0d4c1750 msgr2=0x55ed0d4c1b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:16:01.227 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:16:01.222+0000 7fbe57e74740 1 -- 172.21.15.118:0/1635275209 shutdown_connections 2024-03-24T00:16:01.227 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:16:01.222+0000 7fbe57e74740 1 -- 172.21.15.118:0/1635275209 wait complete. 2024-03-24T00:16:01.227 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:16:01.222+0000 7fbe57e74740 1 -- start start 2024-03-24T00:16:01.227 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:16:01.222+0000 7fbe57e74740 1 Processor -- start 2024-03-24T00:16:01.227 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:16:01.223+0000 7fbe57e74740 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55ed0d4c61e0 0x55ed0d4d0420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:16:01.227 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:16:01.223+0000 7fbe57e74740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55ed0d4c6bd0 0x55ed0d4d0960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:16:01.227 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:16:01.223+0000 7fbe57e74740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55ed0d4cbd40 0x55ed0d4d79a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:16:01.227 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:16:01.223+0000 7fbe57e74740 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ed0d3e7a20 con 0x55ed0d4c61e0 2024-03-24T00:16:01.227 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:16:01.223+0000 7fbe57e74740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ed0d4278d0 con 0x55ed0d4c6bd0 2024-03-24T00:16:01.227 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:16:01.223+0000 7fbe57e74740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ed0d4cecf0 con 0x55ed0d4cbd40 2024-03-24T00:16:01.227 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:16:01.223+0000 7fbe57e74740 10 client.0 _ll_register_callbacks cb 0x55ed0d4241c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T00:16:01.227 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:16:01.223+0000 7fbe4de00640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55ed0d4c61e0 0x55ed0d4d0420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:16:01.227 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:16:01.223+0000 7fbe4d400640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55ed0d4c6bd0 0x55ed0d4d0960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:16:01.228 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:16:01.223+0000 7fbe4e800640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55ed0d4cbd40 0x55ed0d4d79a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:16:01.228 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:16:01.223+0000 7fbe4d400640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55ed0d4c6bd0 0x55ed0d4d0960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.118:59324/0 (socket says 172.21.15.118:59324) 2024-03-24T00:16:01.228 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:16:01.223+0000 7fbe4d400640 1 -- 172.21.15.118:0/104789307 learned_addr learned my addr 172.21.15.118:0/104789307 (peer_addr_for_me v2:172.21.15.118:0/0) 2024-03-24T00:16:01.228 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:16:01.224+0000 7fbe4e800640 1 -- 172.21.15.118:0/104789307 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55ed0d4c6bd0 msgr2=0x55ed0d4d0960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:16:01.228 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:16:01.224+0000 7fbe4e800640 1 --2- 172.21.15.118:0/104789307 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55ed0d4c6bd0 0x55ed0d4d0960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:16:01.228 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:16:01.224+0000 7fbe4e800640 1 -- 172.21.15.118:0/104789307 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55ed0d4c61e0 msgr2=0x55ed0d4d0420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:16:01.228 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:16:01.224+0000 7fbe4e800640 1 --2- 172.21.15.118:0/104789307 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55ed0d4c61e0 0x55ed0d4d0420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:16:01.228 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:16:01.224+0000 7fbe4e800640 1 -- 172.21.15.118:0/104789307 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55ed0d140210 con 0x55ed0d4cbd40 2024-03-24T00:16:01.228 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:16:01.224+0000 7fbe4e800640 1 --2- 172.21.15.118:0/104789307 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55ed0d4cbd40 0x55ed0d4d79a0 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7fbe48037040 tx=0x7fbe4802c0f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=83bcdd7e39f1d318 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:16:01.228 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:16:01.224+0000 7fbe2fe00640 10 client.0 ms_handle_connect on v2:172.21.15.193:3300/0 2024-03-24T00:16:01.228 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:16:01.224+0000 7fbe2fe00640 1 -- 172.21.15.118:0/104789307 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbe48039040 con 0x55ed0d4cbd40 2024-03-24T00:16:01.228 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:16:01.224+0000 7fbe2fe00640 1 -- 172.21.15.118:0/104789307 <== mon.1 v2:172.21.15.193:3300/0 2 ==== osd_map(90..90 src has 1..90) v4 ==== 8277+0+0 (secure 0 0 0) 0x7fbe48004230 con 0x55ed0d4cbd40 2024-03-24T00:16:01.229 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:16:01.224+0000 7fbe57e74740 1 -- 172.21.15.118:0/104789307 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fbe4009f6e0 con 0x55ed0d4cbd40 2024-03-24T00:16:01.229 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:16:01.225+0000 7fbe2fe00640 1 -- 172.21.15.118:0/104789307 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) v2 ==== 24+0+0 (secure 0 0 0) 0x7fbe48004430 con 0x55ed0d4cbd40 2024-03-24T00:16:01.229 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:16:01.225+0000 7fbe57e74740 10 client.25429 fetch_fsmap learned FSMap version 28 2024-03-24T00:16:01.229 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:16:01.225+0000 7fbe57e74740 1 -- 172.21.15.118:0/104789307 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) v3 -- 0x55ed0d20c620 con 0x55ed0d4cbd40 2024-03-24T00:16:01.229 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:16:01.225+0000 7fbe2fe00640 1 -- 172.21.15.118:0/104789307 <== mon.1 v2:172.21.15.193:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fbe4009f6e0 con 0x55ed0d4cbd40 2024-03-24T00:16:01.229 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:16:01.225+0000 7fbe2fe00640 1 -- 172.21.15.118:0/104789307 <== mon.1 v2:172.21.15.193:3300/0 5 ==== fsmap(e 28) v1 ==== 7934+0+0 (secure 0 0 0) 0x7fbe4803f070 con 0x55ed0d4cbd40 2024-03-24T00:16:01.229 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:16:01.225+0000 7fbe57e74740 10 client.25429 fetch_fsmap finished waiting for FSMap version 28 2024-03-24T00:16:01.229 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:16:01.225+0000 7fbe57e74740 10 client.25429 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-24T00:16:01.229 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:16:01.225+0000 7fbe57e74740 20 client.25429 populate_metadata read hostname 'smithi118' 2024-03-24T00:16:01.229 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:16:01.225+0000 7fbe57e74740 1 --2- 172.21.15.118:0/104789307 >> [v2:172.21.15.189:6838/4074484168,v1:172.21.15.189:6839/4074484168] conn(0x55ed0d4dc210 0x55ed0d4de600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:16:01.229 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:16:01.225+0000 7fbe57e74740 4 client.25429 mds_command: new command op to 24569 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-24T00:16:01.229 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:16:01.225+0000 7fbe57e74740 1 -- 172.21.15.118:0/104789307 --> [v2:172.21.15.189:6838/4074484168,v1:172.21.15.189:6839/4074484168] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55ed0d438d70 con 0x55ed0d4dc210 2024-03-24T00:16:01.229 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:16:01.225+0000 7fbe4de00640 1 --2- 172.21.15.118:0/104789307 >> [v2:172.21.15.189:6838/4074484168,v1:172.21.15.189:6839/4074484168] conn(0x55ed0d4dc210 0x55ed0d4de600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:16:01.230 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:16:01.226+0000 7fbe4de00640 1 --2- 172.21.15.118:0/104789307 >> [v2:172.21.15.189:6838/4074484168,v1:172.21.15.189:6839/4074484168] conn(0x55ed0d4dc210 0x55ed0d4de600 crc :-1 s=READY pgs=37 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=96b41506b6d30f27 server_cookie=11764f5316caba8e in_seq=0 out_seq=0 2024-03-24T00:16:01.231 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:16:01.226+0000 7fbe2fe00640 10 client.25429 ms_handle_connect on v2:172.21.15.189:6838/4074484168 2024-03-24T00:16:01.231 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:16:01.227+0000 7fbe2fe00640 1 -- 172.21.15.118:0/104789307 <== mds.0 v2:172.21.15.189:6838/4074484168 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x55ed0d438d70 con 0x55ed0d4dc210 2024-03-24T00:16:01.231 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:16:01.227+0000 7fbe2fe00640 10 client.25429 handle_command_reply: tid=0 2024-03-24T00:16:01.231 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:16:01.227+0000 7fbe57e74740 1 client.25429 shutdown 2024-03-24T00:16:01.231 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:16:01.227+0000 7fbe57e74740 1 -- 172.21.15.118:0/104789307 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55ed0d4cbd40 msgr2=0x55ed0d4d79a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:16:01.231 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:16:01.227+0000 7fbe57e74740 1 --2- 172.21.15.118:0/104789307 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55ed0d4cbd40 0x55ed0d4d79a0 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7fbe48037040 tx=0x7fbe4802c0f0 comp rx=0 tx=0).stop 2024-03-24T00:16:01.231 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:16:01.227+0000 7fbe57e74740 1 -- 172.21.15.118:0/104789307 shutdown_connections 2024-03-24T00:16:01.231 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:16:01.227+0000 7fbe57e74740 1 --2- 172.21.15.118:0/104789307 >> [v2:172.21.15.189:6838/4074484168,v1:172.21.15.189:6839/4074484168] conn(0x55ed0d4dc210 0x55ed0d4de600 crc :-1 s=READY pgs=37 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:16:01.231 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:16:01.227+0000 7fbe57e74740 1 --2- 172.21.15.118:0/104789307 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55ed0d4cbd40 0x55ed0d4d79a0 unknown :-1 s=CLOSED pgs=204 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:16:01.231 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:16:01.227+0000 7fbe57e74740 1 --2- 172.21.15.118:0/104789307 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55ed0d4c6bd0 0x55ed0d4d0960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:16:01.231 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:16:01.227+0000 7fbe57e74740 1 --2- 172.21.15.118:0/104789307 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55ed0d4c61e0 0x55ed0d4d0420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:16:01.231 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:16:01.227+0000 7fbe2fe00640 0 client.25429 ms_handle_reset on v2:172.21.15.189:6838/4074484168 2024-03-24T00:16:01.232 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:16:01.227+0000 7fbe57e74740 1 -- 172.21.15.118:0/104789307 >> 172.21.15.118:0/104789307 conn(0x55ed0d4c1750 msgr2=0x55ed0d4c0950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:16:01.232 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:16:01.227+0000 7fbe57e74740 1 -- 172.21.15.118:0/104789307 shutdown_connections 2024-03-24T00:16:01.232 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:16:01.227+0000 7fbe57e74740 1 -- 172.21.15.118:0/104789307 wait complete. 2024-03-24T00:16:01.232 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:16:01.227+0000 7fbe57e74740 20 client.25429 trim_cache size 0 max 16384 2024-03-24T00:16:01.243 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:16:01.242+0000 7fbe57e74740 1 Processor -- start 2024-03-24T00:16:01.243 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:16:01.242+0000 7fbe57e74740 1 -- start start 2024-03-24T00:16:01.243 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:16:01.242+0000 7fbe57e74740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55ed0d4c61e0 0x55ed0d4d0420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:16:01.243 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:16:01.242+0000 7fbe57e74740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55ed0d4c6bd0 0x55ed0d4d7960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:16:01.243 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:16:01.242+0000 7fbe57e74740 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55ed0d4cbd40 0x55ed0d4d0960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:16:01.243 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:16:01.242+0000 7fbe57e74740 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbe501af080 con 0x55ed0d4cbd40 2024-03-24T00:16:01.244 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:16:01.242+0000 7fbe57e74740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbe501b0a00 con 0x55ed0d4c6bd0 2024-03-24T00:16:01.244 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:16:01.242+0000 7fbe57e74740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbe501af6e0 con 0x55ed0d4c61e0 2024-03-24T00:16:01.244 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:16:01.242+0000 7fbe4de00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55ed0d4c61e0 0x55ed0d4d0420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:16:01.244 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:16:01.242+0000 7fbe4de00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55ed0d4c61e0 0x55ed0d4d0420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.118:46922/0 (socket says 172.21.15.118:46922) 2024-03-24T00:16:01.244 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:16:01.242+0000 7fbe4de00640 1 -- 172.21.15.118:0/2696079968 learned_addr learned my addr 172.21.15.118:0/2696079968 (peer_addr_for_me v2:172.21.15.118:0/0) 2024-03-24T00:16:01.244 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:16:01.242+0000 7fbe4d400640 1 --2- 172.21.15.118:0/2696079968 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55ed0d4c6bd0 0x55ed0d4d7960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:16:01.244 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:16:01.242+0000 7fbe4e800640 1 --2- 172.21.15.118:0/2696079968 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55ed0d4cbd40 0x55ed0d4d0960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:16:01.244 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:16:01.243+0000 7fbe4de00640 1 -- 172.21.15.118:0/2696079968 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55ed0d4c6bd0 msgr2=0x55ed0d4d7960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:16:01.244 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:16:01.243+0000 7fbe4de00640 1 --2- 172.21.15.118:0/2696079968 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55ed0d4c6bd0 0x55ed0d4d7960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:16:01.244 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:16:01.243+0000 7fbe4de00640 1 -- 172.21.15.118:0/2696079968 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55ed0d4cbd40 msgr2=0x55ed0d4d0960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:16:01.244 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:16:01.243+0000 7fbe4de00640 1 --2- 172.21.15.118:0/2696079968 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55ed0d4cbd40 0x55ed0d4d0960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:16:01.244 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:16:01.243+0000 7fbe4de00640 1 -- 172.21.15.118:0/2696079968 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55ed0d4de970 con 0x55ed0d4c61e0 2024-03-24T00:16:01.244 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:16:01.243+0000 7fbe4e800640 1 --2- 172.21.15.118:0/2696079968 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55ed0d4cbd40 0x55ed0d4d0960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T00:16:01.246 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:16:01.243+0000 7fbe4de00640 1 --2- 172.21.15.118:0/2696079968 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55ed0d4c61e0 0x55ed0d4d0420 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7fbe4009be40 tx=0x7fbe4000efb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=cd95057e2ee673c9 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:16:01.247 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:16:01.243+0000 7fbe3aa00640 1 -- 172.21.15.118:0/2696079968 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbe400b8070 con 0x55ed0d4c61e0 2024-03-24T00:16:01.247 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:16:01.243+0000 7fbe3aa00640 1 -- 172.21.15.118:0/2696079968 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fbe4009f6e0 con 0x55ed0d4c61e0 2024-03-24T00:16:01.247 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:16:01.243+0000 7fbe3aa00640 1 -- 172.21.15.118:0/2696079968 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbe40010ac0 con 0x55ed0d4c61e0 2024-03-24T00:16:01.247 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:16:01.244+0000 7fbe57e74740 1 -- 172.21.15.118:0/2696079968 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55ed0d4c61e0 msgr2=0x55ed0d4d0420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:16:01.247 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:16:01.244+0000 7fbe57e74740 1 --2- 172.21.15.118:0/2696079968 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55ed0d4c61e0 0x55ed0d4d0420 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7fbe4009be40 tx=0x7fbe4000efb0 comp rx=0 tx=0).stop 2024-03-24T00:16:01.247 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:16:01.244+0000 7fbe57e74740 1 -- 172.21.15.118:0/2696079968 shutdown_connections 2024-03-24T00:16:01.247 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:16:01.244+0000 7fbe57e74740 1 --2- 172.21.15.118:0/2696079968 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55ed0d4cbd40 0x55ed0d4d0960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:16:01.247 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:16:01.244+0000 7fbe57e74740 1 --2- 172.21.15.118:0/2696079968 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55ed0d4c6bd0 0x55ed0d4d7960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:16:01.247 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:16:01.244+0000 7fbe57e74740 1 --2- 172.21.15.118:0/2696079968 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55ed0d4c61e0 0x55ed0d4d0420 unknown :-1 s=CLOSED pgs=205 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:16:01.247 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:16:01.244+0000 7fbe57e74740 1 -- 172.21.15.118:0/2696079968 >> 172.21.15.118:0/2696079968 conn(0x55ed0d4c1750 msgr2=0x55ed0d4c1b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:16:01.247 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:16:01.244+0000 7fbe57e74740 1 -- 172.21.15.118:0/2696079968 shutdown_connections 2024-03-24T00:16:01.247 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:16:01.244+0000 7fbe57e74740 1 -- 172.21.15.118:0/2696079968 wait complete. 2024-03-24T00:16:01.247 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:16:01.245+0000 7fbe57e74740 1 -- start start 2024-03-24T00:16:01.248 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:16:01.245+0000 7fbe57e74740 1 Processor -- start 2024-03-24T00:16:01.248 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:16:01.246+0000 7fbe57e74740 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55ed0d4c61e0 0x55ed0d4d5170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:16:01.248 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:16:01.246+0000 7fbe57e74740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55ed0d4c6bd0 0x55ed0d4d56b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:16:01.248 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:16:01.246+0000 7fbe57e74740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55ed0d4cbd40 0x55ed0d4d66c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:16:01.248 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:16:01.246+0000 7fbe57e74740 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbe501b03a0 con 0x55ed0d4c61e0 2024-03-24T00:16:01.248 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:16:01.246+0000 7fbe57e74740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbe501afd40 con 0x55ed0d4c6bd0 2024-03-24T00:16:01.248 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:16:01.246+0000 7fbe57e74740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbe501ad240 con 0x55ed0d4cbd40 2024-03-24T00:16:01.248 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:16:01.246+0000 7fbe57e74740 10 client.0 _ll_register_callbacks cb 0x7fbe40005e30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T00:16:01.248 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:16:01.246+0000 7fbe4e800640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55ed0d4cbd40 0x55ed0d4d66c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:16:01.248 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:16:01.246+0000 7fbe4de00640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55ed0d4c61e0 0x55ed0d4d5170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:16:01.248 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:16:01.246+0000 7fbe4d400640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55ed0d4c6bd0 0x55ed0d4d56b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:16:01.248 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:16:01.246+0000 7fbe4e800640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55ed0d4cbd40 0x55ed0d4d66c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.118:46932/0 (socket says 172.21.15.118:46932) 2024-03-24T00:16:01.248 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:16:01.246+0000 7fbe4e800640 1 -- 172.21.15.118:0/1181406007 learned_addr learned my addr 172.21.15.118:0/1181406007 (peer_addr_for_me v2:172.21.15.118:0/0) 2024-03-24T00:16:01.249 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:16:01.246+0000 7fbe4e800640 1 -- 172.21.15.118:0/1181406007 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55ed0d4c6bd0 msgr2=0x55ed0d4d56b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:16:01.249 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:16:01.246+0000 7fbe4e800640 1 --2- 172.21.15.118:0/1181406007 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55ed0d4c6bd0 0x55ed0d4d56b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:16:01.249 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:16:01.246+0000 7fbe4e800640 1 -- 172.21.15.118:0/1181406007 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55ed0d4c61e0 msgr2=0x55ed0d4d5170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:16:01.249 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:16:01.246+0000 7fbe4e800640 1 --2- 172.21.15.118:0/1181406007 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55ed0d4c61e0 0x55ed0d4d5170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:16:01.249 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:16:01.246+0000 7fbe4e800640 1 -- 172.21.15.118:0/1181406007 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55ed0d4d6cb0 con 0x55ed0d4cbd40 2024-03-24T00:16:01.250 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:16:01.246+0000 7fbe4e800640 1 --2- 172.21.15.118:0/1181406007 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55ed0d4cbd40 0x55ed0d4d66c0 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7fbe48056af0 tx=0x7fbe480042e0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=587372faa5370ee2 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:16:01.251 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:16:01.246+0000 7fbe2fe00640 10 client.0 ms_handle_connect on v2:172.21.15.193:3300/0 2024-03-24T00:16:01.251 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:16:01.247+0000 7fbe2fe00640 1 -- 172.21.15.118:0/1181406007 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbe48039040 con 0x55ed0d4cbd40 2024-03-24T00:16:01.251 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:16:01.247+0000 7fbe57e74740 1 -- 172.21.15.118:0/1181406007 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fbe4009f6e0 con 0x55ed0d4cbd40 2024-03-24T00:16:01.251 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:16:01.247+0000 7fbe2fe00640 1 -- 172.21.15.118:0/1181406007 <== mon.1 v2:172.21.15.193:3300/0 2 ==== osd_map(90..90 src has 1..90) v4 ==== 8277+0+0 (secure 0 0 0) 0x7fbe48056440 con 0x55ed0d4cbd40 2024-03-24T00:16:01.251 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:16:01.247+0000 7fbe2fe00640 1 -- 172.21.15.118:0/1181406007 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) v2 ==== 24+0+0 (secure 0 0 0) 0x7fbe48066050 con 0x55ed0d4cbd40 2024-03-24T00:16:01.251 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:16:01.247+0000 7fbe57e74740 10 client.25435 fetch_fsmap learned FSMap version 28 2024-03-24T00:16:01.251 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:16:01.247+0000 7fbe57e74740 1 -- 172.21.15.118:0/1181406007 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) v3 -- 0x55ed0d537c10 con 0x55ed0d4cbd40 2024-03-24T00:16:01.251 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:16:01.247+0000 7fbe2fe00640 1 -- 172.21.15.118:0/1181406007 <== mon.1 v2:172.21.15.193:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fbe4009f6e0 con 0x55ed0d4cbd40 2024-03-24T00:16:01.251 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:16:01.247+0000 7fbe2fe00640 1 -- 172.21.15.118:0/1181406007 <== mon.1 v2:172.21.15.193:3300/0 5 ==== fsmap(e 28) v1 ==== 7934+0+0 (secure 0 0 0) 0x7fbe4803f040 con 0x55ed0d4cbd40 2024-03-24T00:16:01.251 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:16:01.247+0000 7fbe57e74740 10 client.25435 fetch_fsmap finished waiting for FSMap version 28 2024-03-24T00:16:01.251 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:16:01.247+0000 7fbe57e74740 10 client.25435 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-24T00:16:01.251 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:16:01.248+0000 7fbe57e74740 20 client.25435 populate_metadata read hostname 'smithi118' 2024-03-24T00:16:01.251 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:16:01.248+0000 7fbe57e74740 1 --2- 172.21.15.118:0/1181406007 >> [v2:172.21.15.189:6838/4074484168,v1:172.21.15.189:6839/4074484168] conn(0x55ed0d538500 0x55ed0d53a920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:16:01.252 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:16:01.248+0000 7fbe57e74740 4 client.25435 mds_command: new command op to 24569 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-03-24T00:16:01.252 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:16:01.248+0000 7fbe57e74740 1 -- 172.21.15.118:0/1181406007 --> [v2:172.21.15.189:6838/4074484168,v1:172.21.15.189:6839/4074484168] -- command(tid 0: {"prefix": "flush journal"}) v1 -- 0x55ed0d4330c0 con 0x55ed0d538500 2024-03-24T00:16:01.252 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:16:01.248+0000 7fbe4de00640 1 --2- 172.21.15.118:0/1181406007 >> [v2:172.21.15.189:6838/4074484168,v1:172.21.15.189:6839/4074484168] conn(0x55ed0d538500 0x55ed0d53a920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:16:01.252 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:16:01.248+0000 7fbe4de00640 1 --2- 172.21.15.118:0/1181406007 >> [v2:172.21.15.189:6838/4074484168,v1:172.21.15.189:6839/4074484168] conn(0x55ed0d538500 0x55ed0d53a920 crc :-1 s=READY pgs=38 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=72cc0f541ad62847 server_cookie=1c77a1ff727af135 in_seq=0 out_seq=0 2024-03-24T00:16:01.252 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:16:01.248+0000 7fbe2fe00640 10 client.25435 ms_handle_connect on v2:172.21.15.189:6838/4074484168 2024-03-24T00:16:01.261 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:16:01.259+0000 7fbe2fe00640 1 -- 172.21.15.118:0/1181406007 <== mds.0 v2:172.21.15.189:6838/4074484168 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+44 (crc 0 0 0) 0x55ed0d4330c0 con 0x55ed0d538500 2024-03-24T00:16:01.261 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:16:01.259+0000 7fbe2fe00640 10 client.25435 handle_command_reply: tid=0 2024-03-24T00:16:01.261 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:16:01.259+0000 7fbe57e74740 1 client.25435 shutdown 2024-03-24T00:16:01.261 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:16:01.260+0000 7fbe57e74740 1 -- 172.21.15.118:0/1181406007 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55ed0d4cbd40 msgr2=0x55ed0d4d66c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:16:01.261 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:16:01.260+0000 7fbe57e74740 1 --2- 172.21.15.118:0/1181406007 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55ed0d4cbd40 0x55ed0d4d66c0 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7fbe48056af0 tx=0x7fbe480042e0 comp rx=0 tx=0).stop 2024-03-24T00:16:01.261 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:16:01.260+0000 7fbe57e74740 1 -- 172.21.15.118:0/1181406007 shutdown_connections 2024-03-24T00:16:01.261 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:16:01.260+0000 7fbe57e74740 1 --2- 172.21.15.118:0/1181406007 >> [v2:172.21.15.189:6838/4074484168,v1:172.21.15.189:6839/4074484168] conn(0x55ed0d538500 0x55ed0d53a920 crc :-1 s=READY pgs=38 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:16:01.261 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:16:01.260+0000 7fbe57e74740 1 --2- 172.21.15.118:0/1181406007 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55ed0d4cbd40 0x55ed0d4d66c0 unknown :-1 s=CLOSED pgs=206 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:16:01.261 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:16:01.260+0000 7fbe57e74740 1 --2- 172.21.15.118:0/1181406007 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55ed0d4c6bd0 0x55ed0d4d56b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:16:01.261 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:16:01.260+0000 7fbe57e74740 1 --2- 172.21.15.118:0/1181406007 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55ed0d4c61e0 0x55ed0d4d5170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:16:01.261 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:16:01.260+0000 7fbe2fe00640 0 client.25435 ms_handle_reset on v2:172.21.15.189:6838/4074484168 2024-03-24T00:16:01.262 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:16:01.260+0000 7fbe57e74740 1 -- 172.21.15.118:0/1181406007 >> 172.21.15.118:0/1181406007 conn(0x55ed0d4c1750 msgr2=0x55ed0d4d03e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:16:01.262 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:16:01.260+0000 7fbe57e74740 1 -- 172.21.15.118:0/1181406007 shutdown_connections 2024-03-24T00:16:01.262 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:16:01.260+0000 7fbe57e74740 1 -- 172.21.15.118:0/1181406007 wait complete. 2024-03-24T00:16:01.262 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:16:01.260+0000 7fbe57e74740 20 client.25435 trim_cache size 0 max 16384 2024-03-24T00:16:01.262 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stdout:{ 2024-03-24T00:16:01.262 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stdout: "message": "", 2024-03-24T00:16:01.262 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stdout: "return_code": 0 2024-03-24T00:16:01.262 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stdout:} 2024-03-24T00:16:01.263 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:16:01.261+0000 7fbe55200640 1 -- 172.21.15.118:0/2694958437 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7fbe340a29c0 msgr2=0x7fbe340a4e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:16:01.263 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:16:01.261+0000 7fbe55200640 1 --2- 172.21.15.118:0/2694958437 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7fbe340a29c0 0x7fbe340a4e80 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7fbe4802c630 tx=0x7fbe4802ce50 comp rx=0 tx=0).stop 2024-03-24T00:16:01.263 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:16:01.261+0000 7fbe55200640 1 -- 172.21.15.118:0/2694958437 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fbe50107d00 msgr2=0x7fbe5019df30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:16:01.263 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:16:01.261+0000 7fbe55200640 1 --2- 172.21.15.118:0/2694958437 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fbe50107d00 0x7fbe5019df30 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7fbe4000e9c0 tx=0x7fbe4000ee80 comp rx=0 tx=0).stop 2024-03-24T00:16:01.263 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:16:01.261+0000 7fbe55200640 1 -- 172.21.15.118:0/2694958437 shutdown_connections 2024-03-24T00:16:01.263 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:16:01.261+0000 7fbe55200640 1 --2- 172.21.15.118:0/2694958437 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7fbe340a29c0 0x7fbe340a4e80 unknown :-1 s=CLOSED pgs=271 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:16:01.264 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:16:01.261+0000 7fbe55200640 1 --2- 172.21.15.118:0/2694958437 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fbe501098b0 0x7fbe501a54e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:16:01.264 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:16:01.261+0000 7fbe55200640 1 --2- 172.21.15.118:0/2694958437 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fbe50108ef0 0x7fbe5019e470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:16:01.264 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:16:01.261+0000 7fbe55200640 1 --2- 172.21.15.118:0/2694958437 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fbe50107d00 0x7fbe5019df30 unknown :-1 s=CLOSED pgs=211 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:16:01.264 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:16:01.261+0000 7fbe55200640 1 -- 172.21.15.118:0/2694958437 >> 172.21.15.118:0/2694958437 conn(0x7fbe50076cb0 msgr2=0x7fbe5010c190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:16:01.264 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:16:01.261+0000 7fbe55200640 1 -- 172.21.15.118:0/2694958437 shutdown_connections 2024-03-24T00:16:01.264 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:16:01.261+0000 7fbe55200640 1 -- 172.21.15.118:0/2694958437 wait complete. 2024-03-24T00:16:02.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:16:02 smithi118 ceph-mon[31535]: pgmap v408: 97 pgs: 97 active+clean; 660 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 341 B/s wr, 9 op/s 2024-03-24T00:16:02.703 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:16:02 smithi189 ceph-mon[36187]: pgmap v408: 97 pgs: 97 active+clean; 660 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 341 B/s wr, 9 op/s 2024-03-24T00:16:02.710 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:16:02 smithi193 ceph-mon[36878]: pgmap v408: 97 pgs: 97 active+clean; 660 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 4.4 KiB/s rd, 341 B/s wr, 9 op/s 2024-03-24T00:16:05.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:16:04 smithi118 ceph-mon[31535]: pgmap v409: 97 pgs: 97 active+clean; 660 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 1023 B/s wr, 13 op/s 2024-03-24T00:16:05.203 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:16:04 smithi189 ceph-mon[36187]: pgmap v409: 97 pgs: 97 active+clean; 660 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 1023 B/s wr, 13 op/s 2024-03-24T00:16:05.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:16:04 smithi193 ceph-mon[36878]: pgmap v409: 97 pgs: 97 active+clean; 660 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 5.2 KiB/s rd, 1023 B/s wr, 13 op/s 2024-03-24T00:16:07.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:16:06 smithi118 ceph-mon[31535]: pgmap v410: 97 pgs: 97 active+clean; 661 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 5.5 KiB/s rd, 1.5 KiB/s wr, 15 op/s 2024-03-24T00:16:07.203 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:16:06 smithi189 ceph-mon[36187]: pgmap v410: 97 pgs: 97 active+clean; 661 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 5.5 KiB/s rd, 1.5 KiB/s wr, 15 op/s 2024-03-24T00:16:07.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:16:06 smithi193 ceph-mon[36878]: pgmap v410: 97 pgs: 97 active+clean; 661 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 5.5 KiB/s rd, 1.5 KiB/s wr, 15 op/s 2024-03-24T00:16:09.144 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:16:08 smithi118 ceph-mon[31535]: pgmap v411: 97 pgs: 97 active+clean; 661 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 1.5 KiB/s wr, 13 op/s 2024-03-24T00:16:09.203 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:16:08 smithi189 ceph-mon[36187]: pgmap v411: 97 pgs: 97 active+clean; 661 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 1.5 KiB/s wr, 13 op/s 2024-03-24T00:16:09.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:16:08 smithi193 ceph-mon[36878]: pgmap v411: 97 pgs: 97 active+clean; 661 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 1.5 KiB/s wr, 13 op/s 2024-03-24T00:16:11.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:16:10 smithi118 ceph-mon[31535]: pgmap v412: 97 pgs: 97 active+clean; 661 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 5.9 KiB/s rd, 1.5 KiB/s wr, 15 op/s 2024-03-24T00:16:11.203 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:16:10 smithi189 ceph-mon[36187]: pgmap v412: 97 pgs: 97 active+clean; 661 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 5.9 KiB/s rd, 1.5 KiB/s wr, 15 op/s 2024-03-24T00:16:11.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:16:10 smithi193 ceph-mon[36878]: pgmap v412: 97 pgs: 97 active+clean; 661 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 5.9 KiB/s rd, 1.5 KiB/s wr, 15 op/s 2024-03-24T00:16:13.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:16:12 smithi118 ceph-mon[31535]: pgmap v413: 97 pgs: 97 active+clean; 661 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 1.5 KiB/s wr, 13 op/s 2024-03-24T00:16:13.203 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:16:12 smithi189 ceph-mon[36187]: pgmap v413: 97 pgs: 97 active+clean; 661 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 1.5 KiB/s wr, 13 op/s 2024-03-24T00:16:13.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:16:12 smithi193 ceph-mon[36878]: pgmap v413: 97 pgs: 97 active+clean; 661 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 1.5 KiB/s wr, 13 op/s 2024-03-24T00:16:14.417 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:16:14.417+0000 7f761ca00640 1 -- 172.21.15.118:0/2523070754 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f76181052d0 msgr2=0x7f761810bb60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:16:14.418 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:16:14.417+0000 7f761ca00640 1 --2- 172.21.15.118:0/2523070754 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f76181052d0 0x7f761810bb60 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7f7610009ca0 tx=0x7f761002c100 comp rx=0 tx=0).stop 2024-03-24T00:16:14.418 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:16:14.417+0000 7f761ca00640 1 -- 172.21.15.118:0/2523070754 shutdown_connections 2024-03-24T00:16:14.418 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:16:14.417+0000 7f761ca00640 1 --2- 172.21.15.118:0/2523070754 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f76181052d0 0x7f761810bb60 unknown :-1 s=CLOSED pgs=207 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:16:14.418 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:16:14.417+0000 7f761ca00640 1 --2- 172.21.15.118:0/2523070754 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7618104910 0x7f7618104d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:16:14.418 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:16:14.417+0000 7f761ca00640 1 --2- 172.21.15.118:0/2523070754 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f7618103720 0x7f7618103b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:16:14.418 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:16:14.417+0000 7f761ca00640 1 -- 172.21.15.118:0/2523070754 >> 172.21.15.118:0/2523070754 conn(0x7f76180fecb0 msgr2=0x7f76181010d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:16:14.418 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:16:14.417+0000 7f761ca00640 1 -- 172.21.15.118:0/2523070754 shutdown_connections 2024-03-24T00:16:14.418 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:16:14.417+0000 7f761ca00640 1 -- 172.21.15.118:0/2523070754 wait complete. 2024-03-24T00:16:14.419 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:16:14.418+0000 7f761ca00640 1 Processor -- start 2024-03-24T00:16:14.419 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:16:14.419+0000 7f761ca00640 1 -- start start 2024-03-24T00:16:14.421 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:16:14.419+0000 7f761ca00640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f7618103720 0x7f76181999b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:16:14.421 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:16:14.419+0000 7f761ca00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7618104910 0x7f7618199ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:16:14.421 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:16:14.419+0000 7f761ca00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f76181052d0 0x7f76181a0f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:16:14.421 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:16:14.419+0000 7f761ca00640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f761810f3a0 con 0x7f7618103720 2024-03-24T00:16:14.421 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:16:14.419+0000 7f761ca00640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f761810f220 con 0x7f7618104910 2024-03-24T00:16:14.421 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:16:14.419+0000 7f761ca00640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f761810f520 con 0x7f76181052d0 2024-03-24T00:16:14.421 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:16:14.419+0000 7f7617400640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f7618103720 0x7f76181999b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:16:14.422 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:16:14.419+0000 7f7617400640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f7618103720 0x7f76181999b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.118:3300/0 says I am v2:172.21.15.118:36990/0 (socket says 172.21.15.118:36990) 2024-03-24T00:16:14.422 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:16:14.419+0000 7f7617400640 1 -- 172.21.15.118:0/4192302220 learned_addr learned my addr 172.21.15.118:0/4192302220 (peer_addr_for_me v2:172.21.15.118:0/0) 2024-03-24T00:16:14.422 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:16:14.419+0000 7f7616a00640 1 --2- 172.21.15.118:0/4192302220 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7618104910 0x7f7618199ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:16:14.422 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:16:14.419+0000 7f7617e00640 1 --2- 172.21.15.118:0/4192302220 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f76181052d0 0x7f76181a0f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:16:14.422 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:16:14.420+0000 7f7617400640 1 -- 172.21.15.118:0/4192302220 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f76181052d0 msgr2=0x7f76181a0f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:16:14.422 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:16:14.420+0000 7f7617400640 1 --2- 172.21.15.118:0/4192302220 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f76181052d0 0x7f76181a0f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:16:14.422 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:16:14.420+0000 7f7617400640 1 -- 172.21.15.118:0/4192302220 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7618104910 msgr2=0x7f7618199ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:16:14.422 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:16:14.420+0000 7f7617400640 1 --2- 172.21.15.118:0/4192302220 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7618104910 0x7f7618199ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:16:14.422 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:16:14.420+0000 7f7617400640 1 -- 172.21.15.118:0/4192302220 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f76181a15d0 con 0x7f7618103720 2024-03-24T00:16:14.422 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:16:14.420+0000 7f7616a00640 1 --2- 172.21.15.118:0/4192302220 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7618104910 0x7f7618199ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T00:16:14.422 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:16:14.420+0000 7f7617400640 1 --2- 172.21.15.118:0/4192302220 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f7618103720 0x7f76181999b0 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7f760800e9c0 tx=0x7f760800ee80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:16:14.422 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:16:14.420+0000 7f760fe00640 1 -- 172.21.15.118:0/4192302220 <== mon.0 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f760800cd80 con 0x7f7618103720 2024-03-24T00:16:14.423 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:16:14.420+0000 7f760fe00640 1 -- 172.21.15.118:0/4192302220 <== mon.0 v2:172.21.15.118:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f76080035f0 con 0x7f7618103720 2024-03-24T00:16:14.423 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:16:14.420+0000 7f761ca00640 1 -- 172.21.15.118:0/4192302220 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f76181a18c0 con 0x7f7618103720 2024-03-24T00:16:14.423 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:16:14.420+0000 7f760fe00640 1 -- 172.21.15.118:0/4192302220 <== mon.0 v2:172.21.15.118:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7608010690 con 0x7f7618103720 2024-03-24T00:16:14.423 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:16:14.420+0000 7f761ca00640 1 -- 172.21.15.118:0/4192302220 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f76181a1e60 con 0x7f7618103720 2024-03-24T00:16:14.424 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:16:14.423+0000 7f7620c28740 1 Processor -- start 2024-03-24T00:16:14.424 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:16:14.423+0000 7f7620c28740 1 -- start start 2024-03-24T00:16:14.426 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:16:14.424+0000 7f7620c28740 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x556caee6d3f0 0x556caee6d810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:16:14.426 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:16:14.424+0000 7f7620c28740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x556caee6dde0 0x556caee728e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:16:14.426 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:16:14.424+0000 7f7620c28740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x556caee72f50 0x556caee753e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:16:14.426 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:16:14.424+0000 7f7620c28740 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x556caee303b0 con 0x556caee6d3f0 2024-03-24T00:16:14.426 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:16:14.424+0000 7f7620c28740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x556caedd2830 con 0x556caee72f50 2024-03-24T00:16:14.426 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:16:14.424+0000 7f7620c28740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x556caed7b5d0 con 0x556caee6dde0 2024-03-24T00:16:14.426 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:16:14.424+0000 7f760fe00640 1 -- 172.21.15.118:0/4192302220 <== mon.0 v2:172.21.15.118:3300/0 4 ==== mgrmap(e 19) v1 ==== 137342+0+0 (secure 0 0 0) 0x7f76080037a0 con 0x7f7618103720 2024-03-24T00:16:14.426 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:16:14.424+0000 7f7616a00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x556caee72f50 0x556caee753e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:16:14.426 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:16:14.424+0000 7f7616a00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x556caee72f50 0x556caee753e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.118:60062/0 (socket says 172.21.15.118:60062) 2024-03-24T00:16:14.427 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:16:14.424+0000 7f7616a00640 1 -- 172.21.15.118:0/3881098289 learned_addr learned my addr 172.21.15.118:0/3881098289 (peer_addr_for_me v2:172.21.15.118:0/0) 2024-03-24T00:16:14.427 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:16:14.424+0000 7f7617e00640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x556caee6d3f0 0x556caee6d810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:16:14.427 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:16:14.424+0000 7f7617400640 1 --2- 172.21.15.118:0/3881098289 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x556caee6dde0 0x556caee728e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:16:14.427 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:16:14.424+0000 7f7616a00640 1 -- 172.21.15.118:0/3881098289 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x556caee6dde0 msgr2=0x556caee728e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:16:14.427 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:16:14.424+0000 7f7616a00640 1 --2- 172.21.15.118:0/3881098289 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x556caee6dde0 0x556caee728e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:16:14.427 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:16:14.424+0000 7f7616a00640 1 -- 172.21.15.118:0/3881098289 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x556caee6d3f0 msgr2=0x556caee6d810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:16:14.427 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:16:14.424+0000 7f7616a00640 1 --2- 172.21.15.118:0/3881098289 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x556caee6d3f0 0x556caee6d810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:16:14.427 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:16:14.425+0000 7f7616a00640 1 -- 172.21.15.118:0/3881098289 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x556caed932c0 con 0x556caee72f50 2024-03-24T00:16:14.427 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:16:14.425+0000 7f7617400640 1 --2- 172.21.15.118:0/3881098289 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x556caee6dde0 0x556caee728e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-24T00:16:14.427 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:16:14.425+0000 7f7617e00640 1 --2- 172.21.15.118:0/3881098289 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x556caee6d3f0 0x556caee6d810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T00:16:14.427 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:16:14.425+0000 7f7616a00640 1 --2- 172.21.15.118:0/3881098289 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x556caee72f50 0x556caee753e0 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7f76040114f0 tx=0x7f76040119b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=8712e75437d93060 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:16:14.427 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:16:14.425+0000 7f7603e00640 1 -- 172.21.15.118:0/3881098289 <== mon.2 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7604004020 con 0x556caee72f50 2024-03-24T00:16:14.427 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:16:14.425+0000 7f7603e00640 1 -- 172.21.15.118:0/3881098289 <== mon.2 v2:172.21.15.189:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f76040041c0 con 0x556caee72f50 2024-03-24T00:16:14.428 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:16:14.425+0000 7f7603e00640 1 -- 172.21.15.118:0/3881098289 <== mon.2 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f760400f950 con 0x556caee72f50 2024-03-24T00:16:14.433 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:16:14.426+0000 7f760fe00640 1 --2- 172.21.15.118:0/4192302220 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7f75fc0a2900 0x7f75fc0a4dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:16:14.433 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:16:14.426+0000 7f760fe00640 1 -- 172.21.15.118:0/4192302220 <== mon.0 v2:172.21.15.118:3300/0 5 ==== osd_map(90..90 src has 1..90) v4 ==== 8277+0+0 (secure 0 0 0) 0x7f76080cefb0 con 0x7f7618103720 2024-03-24T00:16:14.433 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:16:14.426+0000 7f7620c28740 1 -- 172.21.15.118:0/3881098289 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x556caee72f50 msgr2=0x556caee753e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:16:14.433 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:16:14.426+0000 7f7620c28740 1 --2- 172.21.15.118:0/3881098289 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x556caee72f50 0x556caee753e0 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7f76040114f0 tx=0x7f76040119b0 comp rx=0 tx=0).stop 2024-03-24T00:16:14.433 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:16:14.426+0000 7f7617e00640 1 --2- 172.21.15.118:0/4192302220 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7f75fc0a2900 0x7f75fc0a4dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:16:14.433 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:16:14.426+0000 7f7620c28740 1 -- 172.21.15.118:0/3881098289 shutdown_connections 2024-03-24T00:16:14.433 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:16:14.426+0000 7f7620c28740 1 --2- 172.21.15.118:0/3881098289 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x556caee72f50 0x556caee753e0 unknown :-1 s=CLOSED pgs=213 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:16:14.433 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:16:14.426+0000 7f7620c28740 1 --2- 172.21.15.118:0/3881098289 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x556caee6dde0 0x556caee728e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:16:14.433 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:16:14.426+0000 7f7620c28740 1 --2- 172.21.15.118:0/3881098289 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x556caee6d3f0 0x556caee6d810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:16:14.433 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:16:14.426+0000 7f7620c28740 1 -- 172.21.15.118:0/3881098289 >> 172.21.15.118:0/3881098289 conn(0x556caee68960 msgr2=0x556caee68d60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:16:14.433 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:16:14.426+0000 7f7620c28740 1 -- 172.21.15.118:0/3881098289 shutdown_connections 2024-03-24T00:16:14.433 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:16:14.426+0000 7f7620c28740 1 -- 172.21.15.118:0/3881098289 wait complete. 2024-03-24T00:16:14.434 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:16:14.427+0000 7f7620c28740 1 -- start start 2024-03-24T00:16:14.434 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:16:14.427+0000 7f7617e00640 1 --2- 172.21.15.118:0/4192302220 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7f75fc0a2900 0x7f75fc0a4dc0 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7f761002c640 tx=0x7f761002e040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:16:14.434 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:16:14.427+0000 7f7620c28740 1 Processor -- start 2024-03-24T00:16:14.434 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:16:14.427+0000 7f7620c28740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x556caee6d3f0 0x556caee77570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:16:14.434 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:16:14.427+0000 7f7620c28740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x556caee6dde0 0x556caee77ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:16:14.434 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:16:14.427+0000 7f7620c28740 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x556caee72f50 0x556caee7eaf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:16:14.434 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:16:14.428+0000 7f7620c28740 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x556caed8ea20 con 0x556caee72f50 2024-03-24T00:16:14.434 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:16:14.428+0000 7f7620c28740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x556caedce8d0 con 0x556caee6d3f0 2024-03-24T00:16:14.434 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:16:14.428+0000 7f7620c28740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x556caee75f00 con 0x556caee6dde0 2024-03-24T00:16:14.434 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:16:14.428+0000 7f7620c28740 10 client.0 _ll_register_callbacks cb 0x556caedcb1c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T00:16:14.434 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:16:14.428+0000 7f7617400640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x556caee6d3f0 0x556caee77570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:16:14.434 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:16:14.428+0000 7f7617400640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x556caee6d3f0 0x556caee77570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.118:60070/0 (socket says 172.21.15.118:60070) 2024-03-24T00:16:14.434 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:16:14.428+0000 7f7617400640 1 -- 172.21.15.118:0/3857648046 learned_addr learned my addr 172.21.15.118:0/3857648046 (peer_addr_for_me v2:172.21.15.118:0/0) 2024-03-24T00:16:14.435 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:16:14.428+0000 7f7616a00640 1 --2- 172.21.15.118:0/3857648046 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x556caee6dde0 0x556caee77ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:16:14.435 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:16:14.428+0000 7f7617e00640 1 --2- 172.21.15.118:0/3857648046 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x556caee72f50 0x556caee7eaf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:16:14.435 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:16:14.428+0000 7f7617400640 1 -- 172.21.15.118:0/3857648046 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x556caee6dde0 msgr2=0x556caee77ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:16:14.435 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:16:14.428+0000 7f7617400640 1 --2- 172.21.15.118:0/3857648046 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x556caee6dde0 0x556caee77ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:16:14.435 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:16:14.428+0000 7f7617400640 1 -- 172.21.15.118:0/3857648046 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x556caee72f50 msgr2=0x556caee7eaf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:16:14.435 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:16:14.428+0000 7f7617400640 1 --2- 172.21.15.118:0/3857648046 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x556caee72f50 0x556caee7eaf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:16:14.435 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:16:14.428+0000 7f7617400640 1 -- 172.21.15.118:0/3857648046 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x556caeae7210 con 0x556caee6d3f0 2024-03-24T00:16:14.435 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:16:14.428+0000 7f7617400640 1 --2- 172.21.15.118:0/3857648046 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x556caee6d3f0 0x556caee77570 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7f760800c6a0 tx=0x7f760809bf20 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ac379b313c7fad01 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:16:14.435 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:16:14.428+0000 7f7600c00640 10 client.0 ms_handle_connect on v2:172.21.15.189:3300/0 2024-03-24T00:16:14.435 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:16:14.429+0000 7f7600c00640 1 -- 172.21.15.118:0/3857648046 <== mon.2 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f76080a3f00 con 0x556caee6d3f0 2024-03-24T00:16:14.435 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:16:14.429+0000 7f7600c00640 1 -- 172.21.15.118:0/3857648046 <== mon.2 v2:172.21.15.189:3300/0 2 ==== osd_map(90..90 src has 1..90) v4 ==== 8277+0+0 (secure 0 0 0) 0x7f76080ccb20 con 0x556caee6d3f0 2024-03-24T00:16:14.435 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:16:14.429+0000 7f7620c28740 1 -- 172.21.15.118:0/3857648046 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f76040041c0 con 0x556caee6d3f0 2024-03-24T00:16:14.435 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:16:14.429+0000 7f7600c00640 1 -- 172.21.15.118:0/3857648046 <== mon.2 v2:172.21.15.189:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) v2 ==== 24+0+0 (secure 0 0 0) 0x7f76080afb40 con 0x556caee6d3f0 2024-03-24T00:16:14.436 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:16:14.429+0000 7f7620c28740 10 client.25439 fetch_fsmap learned FSMap version 28 2024-03-24T00:16:14.436 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:16:14.429+0000 7f7620c28740 1 -- 172.21.15.118:0/3857648046 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) v3 -- 0x556caebb3620 con 0x556caee6d3f0 2024-03-24T00:16:14.436 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:16:14.429+0000 7f7600c00640 1 -- 172.21.15.118:0/3857648046 <== mon.2 v2:172.21.15.189:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f76040041c0 con 0x556caee6d3f0 2024-03-24T00:16:14.436 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:16:14.429+0000 7f7600c00640 1 -- 172.21.15.118:0/3857648046 <== mon.2 v2:172.21.15.189:3300/0 5 ==== fsmap(e 28) v1 ==== 7934+0+0 (secure 0 0 0) 0x7f7608097640 con 0x556caee6d3f0 2024-03-24T00:16:14.436 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:16:14.429+0000 7f7620c28740 10 client.25439 fetch_fsmap finished waiting for FSMap version 28 2024-03-24T00:16:14.436 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:16:14.429+0000 7f7620c28740 10 client.25439 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-24T00:16:14.436 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:16:14.429+0000 7f7620c28740 20 client.25439 populate_metadata read hostname 'smithi118' 2024-03-24T00:16:14.436 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:16:14.429+0000 7f7620c28740 1 --2- 172.21.15.118:0/3857648046 >> [v2:172.21.15.189:6838/4074484168,v1:172.21.15.189:6839/4074484168] conn(0x556caee833a0 0x556caee85790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:16:14.436 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:16:14.429+0000 7f7620c28740 4 client.25439 mds_command: new command op to 24569 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-24T00:16:14.436 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:16:14.429+0000 7f7620c28740 1 -- 172.21.15.118:0/3857648046 --> [v2:172.21.15.189:6838/4074484168,v1:172.21.15.189:6839/4074484168] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x556caeddfd70 con 0x556caee833a0 2024-03-24T00:16:14.436 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:16:14.430+0000 7f7616a00640 1 --2- 172.21.15.118:0/3857648046 >> [v2:172.21.15.189:6838/4074484168,v1:172.21.15.189:6839/4074484168] conn(0x556caee833a0 0x556caee85790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:16:14.437 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:16:14.430+0000 7f7616a00640 1 --2- 172.21.15.118:0/3857648046 >> [v2:172.21.15.189:6838/4074484168,v1:172.21.15.189:6839/4074484168] conn(0x556caee833a0 0x556caee85790 crc :-1 s=READY pgs=39 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=fb659db1ab157f76 server_cookie=65c2e4f7a58286ab in_seq=0 out_seq=0 2024-03-24T00:16:14.437 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:16:14.430+0000 7f7600c00640 10 client.25439 ms_handle_connect on v2:172.21.15.189:6838/4074484168 2024-03-24T00:16:14.437 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:16:14.431+0000 7f7600c00640 1 -- 172.21.15.118:0/3857648046 <== mds.0 v2:172.21.15.189:6838/4074484168 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x556caeddfd70 con 0x556caee833a0 2024-03-24T00:16:14.437 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:16:14.431+0000 7f7600c00640 10 client.25439 handle_command_reply: tid=0 2024-03-24T00:16:14.437 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:16:14.431+0000 7f7620c28740 1 client.25439 shutdown 2024-03-24T00:16:14.437 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:16:14.431+0000 7f7620c28740 1 -- 172.21.15.118:0/3857648046 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x556caee6d3f0 msgr2=0x556caee77570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:16:14.437 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:16:14.431+0000 7f7620c28740 1 --2- 172.21.15.118:0/3857648046 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x556caee6d3f0 0x556caee77570 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7f760800c6a0 tx=0x7f760809bf20 comp rx=0 tx=0).stop 2024-03-24T00:16:14.437 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:16:14.431+0000 7f7620c28740 1 -- 172.21.15.118:0/3857648046 shutdown_connections 2024-03-24T00:16:14.437 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:16:14.431+0000 7f7620c28740 1 --2- 172.21.15.118:0/3857648046 >> [v2:172.21.15.189:6838/4074484168,v1:172.21.15.189:6839/4074484168] conn(0x556caee833a0 0x556caee85790 crc :-1 s=READY pgs=39 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:16:14.437 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:16:14.431+0000 7f7620c28740 1 --2- 172.21.15.118:0/3857648046 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x556caee72f50 0x556caee7eaf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:16:14.437 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:16:14.431+0000 7f7600c00640 0 client.25439 ms_handle_reset on v2:172.21.15.189:6838/4074484168 2024-03-24T00:16:14.438 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:16:14.431+0000 7f7620c28740 1 --2- 172.21.15.118:0/3857648046 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x556caee6dde0 0x556caee77ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:16:14.438 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:16:14.431+0000 7f7620c28740 1 --2- 172.21.15.118:0/3857648046 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x556caee6d3f0 0x556caee77570 unknown :-1 s=CLOSED pgs=214 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:16:14.438 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:16:14.431+0000 7f7620c28740 1 -- 172.21.15.118:0/3857648046 >> 172.21.15.118:0/3857648046 conn(0x556caee68960 msgr2=0x556caee67b70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:16:14.438 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:16:14.431+0000 7f7620c28740 1 -- 172.21.15.118:0/3857648046 shutdown_connections 2024-03-24T00:16:14.438 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:16:14.431+0000 7f7620c28740 1 -- 172.21.15.118:0/3857648046 wait complete. 2024-03-24T00:16:14.438 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:16:14.431+0000 7f7620c28740 20 client.25439 trim_cache size 0 max 16384 2024-03-24T00:16:14.453 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:16:14.452+0000 7f7620c28740 1 Processor -- start 2024-03-24T00:16:14.453 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:16:14.452+0000 7f7620c28740 1 -- start start 2024-03-24T00:16:14.453 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:16:14.453+0000 7f7620c28740 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x556caee68960 0x556caee75340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:16:14.453 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:16:14.453+0000 7f7620c28740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x556caee6d3f0 0x556caee7eab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:16:14.453 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:16:14.453+0000 7f7620c28740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x556caee6dde0 0x556caee68d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:16:14.454 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:16:14.453+0000 7f7620c28740 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f76181aaa00 con 0x556caee68960 2024-03-24T00:16:14.454 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:16:14.453+0000 7f7620c28740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f76181ac380 con 0x556caee6dde0 2024-03-24T00:16:14.454 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:16:14.453+0000 7f7620c28740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f76181ab060 con 0x556caee6d3f0 2024-03-24T00:16:14.455 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:16:14.453+0000 7f7617e00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x556caee6dde0 0x556caee68d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:16:14.455 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:16:14.453+0000 7f7616a00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x556caee6d3f0 0x556caee7eab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:16:14.455 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:16:14.453+0000 7f7617e00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x556caee6dde0 0x556caee68d40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.118:60082/0 (socket says 172.21.15.118:60082) 2024-03-24T00:16:14.455 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:16:14.453+0000 7f7617e00640 1 -- 172.21.15.118:0/2436096538 learned_addr learned my addr 172.21.15.118:0/2436096538 (peer_addr_for_me v2:172.21.15.118:0/0) 2024-03-24T00:16:14.455 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:16:14.453+0000 7f7617400640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x556caee68960 0x556caee75340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:16:14.455 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:16:14.453+0000 7f7616a00640 1 -- 172.21.15.118:0/2436096538 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x556caee6dde0 msgr2=0x556caee68d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:16:14.455 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:16:14.453+0000 7f7616a00640 1 --2- 172.21.15.118:0/2436096538 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x556caee6dde0 0x556caee68d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:16:14.455 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:16:14.453+0000 7f7616a00640 1 -- 172.21.15.118:0/2436096538 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x556caee68960 msgr2=0x556caee75340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:16:14.455 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:16:14.453+0000 7f7616a00640 1 --2- 172.21.15.118:0/2436096538 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x556caee68960 0x556caee75340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:16:14.455 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:16:14.453+0000 7f7616a00640 1 -- 172.21.15.118:0/2436096538 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x556caee66f80 con 0x556caee6d3f0 2024-03-24T00:16:14.455 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:16:14.454+0000 7f7616a00640 1 --2- 172.21.15.118:0/2436096538 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x556caee6d3f0 0x556caee7eab0 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7f7604011eb0 tx=0x7f7604005db0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=2b0a2b50a51b6637 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:16:14.455 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:16:14.454+0000 7f7603400640 1 -- 172.21.15.118:0/2436096538 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7604005040 con 0x556caee6d3f0 2024-03-24T00:16:14.455 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:16:14.454+0000 7f7603400640 1 -- 172.21.15.118:0/2436096538 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f76040035f0 con 0x556caee6d3f0 2024-03-24T00:16:14.456 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:16:14.454+0000 7f7603400640 1 -- 172.21.15.118:0/2436096538 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7604004660 con 0x556caee6d3f0 2024-03-24T00:16:14.459 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:16:14.455+0000 7f7620c28740 1 -- 172.21.15.118:0/2436096538 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x556caee6d3f0 msgr2=0x556caee7eab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:16:14.459 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:16:14.455+0000 7f7620c28740 1 --2- 172.21.15.118:0/2436096538 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x556caee6d3f0 0x556caee7eab0 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7f7604011eb0 tx=0x7f7604005db0 comp rx=0 tx=0).stop 2024-03-24T00:16:14.459 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:16:14.455+0000 7f7620c28740 1 -- 172.21.15.118:0/2436096538 shutdown_connections 2024-03-24T00:16:14.459 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:16:14.455+0000 7f7620c28740 1 --2- 172.21.15.118:0/2436096538 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x556caee6dde0 0x556caee68d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:16:14.459 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:16:14.455+0000 7f7620c28740 1 --2- 172.21.15.118:0/2436096538 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x556caee6d3f0 0x556caee7eab0 unknown :-1 s=CLOSED pgs=208 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:16:14.459 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:16:14.455+0000 7f7620c28740 1 --2- 172.21.15.118:0/2436096538 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x556caee68960 0x556caee75340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:16:14.459 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:16:14.455+0000 7f7620c28740 1 -- 172.21.15.118:0/2436096538 >> 172.21.15.118:0/2436096538 conn(0x556caee72a20 msgr2=0x556caee6e4d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:16:14.459 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:16:14.455+0000 7f7620c28740 1 -- 172.21.15.118:0/2436096538 shutdown_connections 2024-03-24T00:16:14.460 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:16:14.455+0000 7f7620c28740 1 -- 172.21.15.118:0/2436096538 wait complete. 2024-03-24T00:16:14.460 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:16:14.455+0000 7f7620c28740 1 -- start start 2024-03-24T00:16:14.460 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:16:14.455+0000 7f7620c28740 1 Processor -- start 2024-03-24T00:16:14.460 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:16:14.456+0000 7f7620c28740 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x556caee68960 0x556caee7c530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:16:14.460 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:16:14.456+0000 7f7620c28740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x556caee6d3f0 0x556caee7ca70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:16:14.460 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:16:14.456+0000 7f7620c28740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x556caee6dde0 0x556caee7da80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:16:14.460 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:16:14.456+0000 7f7620c28740 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f76181abd20 con 0x556caee68960 2024-03-24T00:16:14.460 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:16:14.456+0000 7f7620c28740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f76181ab6c0 con 0x556caee6dde0 2024-03-24T00:16:14.460 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:16:14.456+0000 7f7620c28740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f76181a8bc0 con 0x556caee6d3f0 2024-03-24T00:16:14.460 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:16:14.456+0000 7f7620c28740 10 client.0 _ll_register_callbacks cb 0x7f7608003e30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T00:16:14.460 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:16:14.456+0000 7f7617e00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x556caee6dde0 0x556caee7da80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:16:14.460 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:16:14.456+0000 7f7616a00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x556caee6d3f0 0x556caee7ca70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:16:14.460 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:16:14.456+0000 7f7617e00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x556caee6dde0 0x556caee7da80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.118:60092/0 (socket says 172.21.15.118:60092) 2024-03-24T00:16:14.460 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:16:14.456+0000 7f7617e00640 1 -- 172.21.15.118:0/487219027 learned_addr learned my addr 172.21.15.118:0/487219027 (peer_addr_for_me v2:172.21.15.118:0/0) 2024-03-24T00:16:14.461 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:16:14.456+0000 7f7617400640 1 --2- 172.21.15.118:0/487219027 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x556caee68960 0x556caee7c530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:16:14.461 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:16:14.456+0000 7f7617e00640 1 -- 172.21.15.118:0/487219027 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x556caee6d3f0 msgr2=0x556caee7ca70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:16:14.461 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:16:14.456+0000 7f7617e00640 1 --2- 172.21.15.118:0/487219027 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x556caee6d3f0 0x556caee7ca70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:16:14.461 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:16:14.456+0000 7f7617e00640 1 -- 172.21.15.118:0/487219027 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x556caee68960 msgr2=0x556caee7c530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:16:14.461 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:16:14.456+0000 7f7617e00640 1 --2- 172.21.15.118:0/487219027 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x556caee68960 0x556caee7c530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:16:14.461 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:16:14.456+0000 7f7617e00640 1 -- 172.21.15.118:0/487219027 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x556caee7e070 con 0x556caee6dde0 2024-03-24T00:16:14.461 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:16:14.456+0000 7f7616a00640 1 --2- 172.21.15.118:0/487219027 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x556caee6d3f0 0x556caee7ca70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T00:16:14.461 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:16:14.457+0000 7f7617e00640 1 --2- 172.21.15.118:0/487219027 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x556caee6dde0 0x556caee7da80 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7f761002e010 tx=0x7f761002c100 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b0abbbb872b66f3e server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:16:14.461 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:16:14.457+0000 7f7600c00640 10 client.0 ms_handle_connect on v2:172.21.15.189:3300/0 2024-03-24T00:16:14.461 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:16:14.457+0000 7f7600c00640 1 -- 172.21.15.118:0/487219027 <== mon.2 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7610051070 con 0x556caee6dde0 2024-03-24T00:16:14.461 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:16:14.457+0000 7f7620c28740 1 -- 172.21.15.118:0/487219027 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f76040035f0 con 0x556caee6dde0 2024-03-24T00:16:14.461 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:16:14.457+0000 7f7600c00640 1 -- 172.21.15.118:0/487219027 <== mon.2 v2:172.21.15.189:3300/0 2 ==== osd_map(90..90 src has 1..90) v4 ==== 8277+0+0 (secure 0 0 0) 0x7f7610004040 con 0x556caee6dde0 2024-03-24T00:16:14.461 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:16:14.457+0000 7f7600c00640 1 -- 172.21.15.118:0/487219027 <== mon.2 v2:172.21.15.189:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) v2 ==== 24+0+0 (secure 0 0 0) 0x7f761005d050 con 0x556caee6dde0 2024-03-24T00:16:14.462 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:16:14.457+0000 7f7620c28740 10 client.25445 fetch_fsmap learned FSMap version 28 2024-03-24T00:16:14.462 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:16:14.457+0000 7f7620c28740 1 -- 172.21.15.118:0/487219027 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) v3 -- 0x556caeedefb0 con 0x556caee6dde0 2024-03-24T00:16:14.462 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:16:14.457+0000 7f7600c00640 1 -- 172.21.15.118:0/487219027 <== mon.2 v2:172.21.15.189:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f76040035f0 con 0x556caee6dde0 2024-03-24T00:16:14.462 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:16:14.458+0000 7f7600c00640 1 -- 172.21.15.118:0/487219027 <== mon.2 v2:172.21.15.189:3300/0 5 ==== fsmap(e 28) v1 ==== 7934+0+0 (secure 0 0 0) 0x7f7610040b20 con 0x556caee6dde0 2024-03-24T00:16:14.462 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:16:14.458+0000 7f7620c28740 10 client.25445 fetch_fsmap finished waiting for FSMap version 28 2024-03-24T00:16:14.462 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:16:14.458+0000 7f7620c28740 10 client.25445 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-24T00:16:14.462 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:16:14.458+0000 7f7620c28740 20 client.25445 populate_metadata read hostname 'smithi118' 2024-03-24T00:16:14.462 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:16:14.458+0000 7f7620c28740 1 --2- 172.21.15.118:0/487219027 >> [v2:172.21.15.189:6838/4074484168,v1:172.21.15.189:6839/4074484168] conn(0x556caee72f50 0x556caeee19f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:16:14.462 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:16:14.458+0000 7f7620c28740 4 client.25445 mds_command: new command op to 24569 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-03-24T00:16:14.462 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:16:14.458+0000 7f7620c28740 1 -- 172.21.15.118:0/487219027 --> [v2:172.21.15.189:6838/4074484168,v1:172.21.15.189:6839/4074484168] -- command(tid 0: {"prefix": "flush journal"}) v1 -- 0x556caedda0c0 con 0x556caee72f50 2024-03-24T00:16:14.462 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:16:14.458+0000 7f7617400640 1 --2- 172.21.15.118:0/487219027 >> [v2:172.21.15.189:6838/4074484168,v1:172.21.15.189:6839/4074484168] conn(0x556caee72f50 0x556caeee19f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:16:14.462 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:16:14.458+0000 7f7617400640 1 --2- 172.21.15.118:0/487219027 >> [v2:172.21.15.189:6838/4074484168,v1:172.21.15.189:6839/4074484168] conn(0x556caee72f50 0x556caeee19f0 crc :-1 s=READY pgs=40 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=de57e747c0c6275f server_cookie=2fa641b259ae0946 in_seq=0 out_seq=0 2024-03-24T00:16:14.462 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:16:14.458+0000 7f7600c00640 10 client.25445 ms_handle_connect on v2:172.21.15.189:6838/4074484168 2024-03-24T00:16:14.464 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:16:14.461+0000 7f7600c00640 1 -- 172.21.15.118:0/487219027 <== mds.0 v2:172.21.15.189:6838/4074484168 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+44 (crc 0 0 0) 0x556caedda0c0 con 0x556caee72f50 2024-03-24T00:16:14.464 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:16:14.461+0000 7f7600c00640 10 client.25445 handle_command_reply: tid=0 2024-03-24T00:16:14.464 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:16:14.461+0000 7f7620c28740 1 client.25445 shutdown 2024-03-24T00:16:14.464 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:16:14.462+0000 7f7620c28740 1 -- 172.21.15.118:0/487219027 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x556caee6dde0 msgr2=0x556caee7da80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:16:14.464 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:16:14.462+0000 7f7620c28740 1 --2- 172.21.15.118:0/487219027 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x556caee6dde0 0x556caee7da80 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7f761002e010 tx=0x7f761002c100 comp rx=0 tx=0).stop 2024-03-24T00:16:14.464 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:16:14.462+0000 7f7620c28740 1 -- 172.21.15.118:0/487219027 shutdown_connections 2024-03-24T00:16:14.464 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:16:14.462+0000 7f7620c28740 1 --2- 172.21.15.118:0/487219027 >> [v2:172.21.15.189:6838/4074484168,v1:172.21.15.189:6839/4074484168] conn(0x556caee72f50 0x556caeee19f0 crc :-1 s=READY pgs=40 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:16:14.464 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:16:14.462+0000 7f7620c28740 1 --2- 172.21.15.118:0/487219027 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x556caee6dde0 0x556caee7da80 unknown :-1 s=CLOSED pgs=215 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:16:14.464 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:16:14.462+0000 7f7620c28740 1 --2- 172.21.15.118:0/487219027 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x556caee6d3f0 0x556caee7ca70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:16:14.464 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:16:14.462+0000 7f7620c28740 1 --2- 172.21.15.118:0/487219027 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x556caee68960 0x556caee7c530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:16:14.464 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:16:14.462+0000 7f7600c00640 0 client.25445 ms_handle_reset on v2:172.21.15.189:6838/4074484168 2024-03-24T00:16:14.465 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:16:14.462+0000 7f7620c28740 1 -- 172.21.15.118:0/487219027 >> 172.21.15.118:0/487219027 conn(0x556caee72a20 msgr2=0x556caee72280 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:16:14.465 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:16:14.462+0000 7f7620c28740 1 -- 172.21.15.118:0/487219027 shutdown_connections 2024-03-24T00:16:14.465 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:16:14.462+0000 7f7620c28740 1 -- 172.21.15.118:0/487219027 wait complete. 2024-03-24T00:16:14.465 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:16:14.462+0000 7f7620c28740 20 client.25445 trim_cache size 0 max 16384 2024-03-24T00:16:14.465 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stdout:{ 2024-03-24T00:16:14.465 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stdout: "message": "", 2024-03-24T00:16:14.465 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stdout: "return_code": 0 2024-03-24T00:16:14.465 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stdout:} 2024-03-24T00:16:14.466 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:16:14.463+0000 7f761ca00640 1 -- 172.21.15.118:0/4192302220 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7f75fc0a2900 msgr2=0x7f75fc0a4dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:16:14.466 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:16:14.463+0000 7f761ca00640 1 --2- 172.21.15.118:0/4192302220 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7f75fc0a2900 0x7f75fc0a4dc0 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7f761002c640 tx=0x7f761002e040 comp rx=0 tx=0).stop 2024-03-24T00:16:14.466 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:16:14.463+0000 7f761ca00640 1 -- 172.21.15.118:0/4192302220 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f7618103720 msgr2=0x7f76181999b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:16:14.466 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:16:14.463+0000 7f761ca00640 1 --2- 172.21.15.118:0/4192302220 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f7618103720 0x7f76181999b0 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7f760800e9c0 tx=0x7f760800ee80 comp rx=0 tx=0).stop 2024-03-24T00:16:14.466 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:16:14.463+0000 7f761ca00640 1 -- 172.21.15.118:0/4192302220 shutdown_connections 2024-03-24T00:16:14.466 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:16:14.463+0000 7f761ca00640 1 --2- 172.21.15.118:0/4192302220 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7f75fc0a2900 0x7f75fc0a4dc0 unknown :-1 s=CLOSED pgs=272 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:16:14.467 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:16:14.463+0000 7f761ca00640 1 --2- 172.21.15.118:0/4192302220 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f76181052d0 0x7f76181a0f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:16:14.467 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:16:14.463+0000 7f761ca00640 1 --2- 172.21.15.118:0/4192302220 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7618104910 0x7f7618199ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:16:14.467 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:16:14.463+0000 7f761ca00640 1 --2- 172.21.15.118:0/4192302220 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f7618103720 0x7f76181999b0 unknown :-1 s=CLOSED pgs=180 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:16:14.467 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:16:14.463+0000 7f761ca00640 1 -- 172.21.15.118:0/4192302220 >> 172.21.15.118:0/4192302220 conn(0x7f76180fecb0 msgr2=0x7f7618107c10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:16:14.467 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:16:14.463+0000 7f761ca00640 1 -- 172.21.15.118:0/4192302220 shutdown_connections 2024-03-24T00:16:14.467 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:16:14.463+0000 7f761ca00640 1 -- 172.21.15.118:0/4192302220 wait complete. 2024-03-24T00:16:15.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:16:14 smithi118 ceph-mon[31535]: pgmap v414: 97 pgs: 97 active+clean; 661 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 1.5 KiB/s wr, 14 op/s 2024-03-24T00:16:15.203 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:16:14 smithi189 ceph-mon[36187]: pgmap v414: 97 pgs: 97 active+clean; 661 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 1.5 KiB/s wr, 14 op/s 2024-03-24T00:16:15.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:16:14 smithi193 ceph-mon[36878]: pgmap v414: 97 pgs: 97 active+clean; 661 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 1.5 KiB/s wr, 14 op/s 2024-03-24T00:16:17.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:16:16 smithi118 ceph-mon[31535]: pgmap v415: 97 pgs: 97 active+clean; 661 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 5.5 KiB/s rd, 852 B/s wr, 12 op/s 2024-03-24T00:16:17.203 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:16:16 smithi189 ceph-mon[36187]: pgmap v415: 97 pgs: 97 active+clean; 661 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 5.5 KiB/s rd, 852 B/s wr, 12 op/s 2024-03-24T00:16:17.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:16:16 smithi193 ceph-mon[36878]: pgmap v415: 97 pgs: 97 active+clean; 661 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 5.5 KiB/s rd, 852 B/s wr, 12 op/s 2024-03-24T00:16:18.546 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-03-24T00:16:18.547 DEBUG:teuthology.orchestra.run.smithi118:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-24T00:16:18.773 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:18.773+0000 7f2f6ae00640 1 -- 172.21.15.118:0/1926752216 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2f64103270 msgr2=0x7f2f64109b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:16:18.773 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:18.773+0000 7f2f6ae00640 1 --2- 172.21.15.118:0/1926752216 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2f64103270 0x7f2f64109b00 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7f2f5c009ca0 tx=0x7f2f5c02c100 comp rx=0 tx=0).stop 2024-03-24T00:16:18.773 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:18.773+0000 7f2f6ae00640 1 -- 172.21.15.118:0/1926752216 shutdown_connections 2024-03-24T00:16:18.773 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:18.773+0000 7f2f6ae00640 1 --2- 172.21.15.118:0/1926752216 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2f64103270 0x7f2f64109b00 unknown :-1 s=CLOSED pgs=216 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:16:18.774 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:18.773+0000 7f2f6ae00640 1 --2- 172.21.15.118:0/1926752216 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f2f641028b0 0x7f2f64102d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:16:18.774 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:18.773+0000 7f2f6ae00640 1 --2- 172.21.15.118:0/1926752216 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f2f641016c0 0x7f2f64101ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:16:18.774 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:18.773+0000 7f2f6ae00640 1 -- 172.21.15.118:0/1926752216 >> 172.21.15.118:0/1926752216 conn(0x7f2f640fcc30 msgr2=0x7f2f640ff070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:16:18.774 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:18.773+0000 7f2f6ae00640 1 -- 172.21.15.118:0/1926752216 shutdown_connections 2024-03-24T00:16:18.774 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:18.773+0000 7f2f6ae00640 1 -- 172.21.15.118:0/1926752216 wait complete. 2024-03-24T00:16:18.775 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:18.775+0000 7f2f6ae00640 1 Processor -- start 2024-03-24T00:16:18.775 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:18.775+0000 7f2f6ae00640 1 -- start start 2024-03-24T00:16:18.776 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:18.775+0000 7f2f6ae00640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f2f641016c0 0x7f2f641979f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:16:18.776 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:18.775+0000 7f2f6ae00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2f641028b0 0x7f2f64197f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:16:18.776 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:18.775+0000 7f2f6ae00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f2f64103270 0x7f2f6419efa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:16:18.776 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:18.775+0000 7f2f6ae00640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2f6410d340 con 0x7f2f641016c0 2024-03-24T00:16:18.776 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:18.775+0000 7f2f6ae00640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2f6410d1c0 con 0x7f2f641028b0 2024-03-24T00:16:18.776 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:18.775+0000 7f2f6ae00640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2f6410d4c0 con 0x7f2f64103270 2024-03-24T00:16:18.776 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:18.775+0000 7f2f5be00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2f641028b0 0x7f2f64197f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:16:18.776 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:18.775+0000 7f2f5be00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2f641028b0 0x7f2f64197f30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.118:60112/0 (socket says 172.21.15.118:60112) 2024-03-24T00:16:18.776 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:18.776+0000 7f2f5be00640 1 -- 172.21.15.118:0/1441899990 learned_addr learned my addr 172.21.15.118:0/1441899990 (peer_addr_for_me v2:172.21.15.118:0/0) 2024-03-24T00:16:18.776 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:18.776+0000 7f2f68e00640 1 --2- 172.21.15.118:0/1441899990 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f2f64103270 0x7f2f6419efa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:16:18.776 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:18.776+0000 7f2f63e00640 1 --2- 172.21.15.118:0/1441899990 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f2f641016c0 0x7f2f641979f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:16:18.776 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:18.776+0000 7f2f5be00640 1 -- 172.21.15.118:0/1441899990 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f2f64103270 msgr2=0x7f2f6419efa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:16:18.776 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:18.776+0000 7f2f5be00640 1 --2- 172.21.15.118:0/1441899990 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f2f64103270 0x7f2f6419efa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:16:18.777 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:18.776+0000 7f2f5be00640 1 -- 172.21.15.118:0/1441899990 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f2f641016c0 msgr2=0x7f2f641979f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:16:18.777 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:18.776+0000 7f2f5be00640 1 --2- 172.21.15.118:0/1441899990 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f2f641016c0 0x7f2f641979f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:16:18.777 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:18.776+0000 7f2f5be00640 1 -- 172.21.15.118:0/1441899990 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2f6419f610 con 0x7f2f641028b0 2024-03-24T00:16:18.777 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:18.776+0000 7f2f68e00640 1 --2- 172.21.15.118:0/1441899990 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f2f64103270 0x7f2f6419efa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T00:16:18.777 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:18.776+0000 7f2f5be00640 1 --2- 172.21.15.118:0/1441899990 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2f641028b0 0x7f2f64197f30 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7f2f4c00e9f0 tx=0x7f2f4c00edf0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:16:18.777 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:18.776+0000 7f2f61600640 1 -- 172.21.15.118:0/1441899990 <== mon.2 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2f4c00cd80 con 0x7f2f641028b0 2024-03-24T00:16:18.778 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:18.776+0000 7f2f61600640 1 -- 172.21.15.118:0/1441899990 <== mon.2 v2:172.21.15.189:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f2f4c005500 con 0x7f2f641028b0 2024-03-24T00:16:18.778 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:18.777+0000 7f2f61600640 1 -- 172.21.15.118:0/1441899990 <== mon.2 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2f4c010690 con 0x7f2f641028b0 2024-03-24T00:16:18.778 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:18.777+0000 7f2f6ae00640 1 -- 172.21.15.118:0/1441899990 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2f6419f900 con 0x7f2f641028b0 2024-03-24T00:16:18.778 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:18.777+0000 7f2f6ae00640 1 -- 172.21.15.118:0/1441899990 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2f6419fe50 con 0x7f2f641028b0 2024-03-24T00:16:18.779 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:18.779+0000 7f2f61600640 1 -- 172.21.15.118:0/1441899990 <== mon.2 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 19) v1 ==== 137342+0+0 (secure 0 0 0) 0x7f2f4c004f60 con 0x7f2f641028b0 2024-03-24T00:16:18.779 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:18.780+0000 7f2f6dbd7740 1 Processor -- start 2024-03-24T00:16:18.779 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:18.780+0000 7f2f6dbd7740 1 -- start start 2024-03-24T00:16:18.781 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:18.780+0000 7f2f6dbd7740 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55fe4be35290 0x55fe4be356b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:16:18.781 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:18.780+0000 7f2f6dbd7740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55fe4be35c80 0x55fe4be3a780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:16:18.781 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:18.780+0000 7f2f6dbd7740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55fe4be3adf0 0x55fe4be3d280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:16:18.781 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:18.780+0000 7f2f6dbd7740 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x55fe4bd96250 con 0x55fe4be35290 2024-03-24T00:16:18.781 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:18.780+0000 7f2f6dbd7740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x55fe4bd3ea30 con 0x55fe4be35c80 2024-03-24T00:16:18.781 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:18.780+0000 7f2f6dbd7740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x55fe4bd922f0 con 0x55fe4be3adf0 2024-03-24T00:16:18.781 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:18.781+0000 7f2f5be00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55fe4be3adf0 0x55fe4be3d280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:16:18.781 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:18.781+0000 7f2f63e00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55fe4be35c80 0x55fe4be3a780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:16:18.781 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:18.781+0000 7f2f63e00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55fe4be35c80 0x55fe4be3a780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.118:60116/0 (socket says 172.21.15.118:60116) 2024-03-24T00:16:18.782 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:18.781+0000 7f2f63e00640 1 -- 172.21.15.118:0/3058698374 learned_addr learned my addr 172.21.15.118:0/3058698374 (peer_addr_for_me v2:172.21.15.118:0/0) 2024-03-24T00:16:18.782 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:18.781+0000 7f2f5be00640 1 --2- 172.21.15.118:0/3058698374 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55fe4be3adf0 0x55fe4be3d280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.118:45332/0 (socket says 172.21.15.118:45332) 2024-03-24T00:16:18.782 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:18.781+0000 7f2f68e00640 1 --2- 172.21.15.118:0/3058698374 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55fe4be35290 0x55fe4be356b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:16:18.782 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:18.781+0000 7f2f61600640 1 --2- 172.21.15.118:0/1441899990 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7f2f480a2910 0x7f2f480a4dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:16:18.782 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:18.781+0000 7f2f63e00640 1 -- 172.21.15.118:0/3058698374 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55fe4be3adf0 msgr2=0x55fe4be3d280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:16:18.782 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:18.781+0000 7f2f63e00640 1 --2- 172.21.15.118:0/3058698374 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55fe4be3adf0 0x55fe4be3d280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:16:18.782 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:18.781+0000 7f2f63e00640 1 -- 172.21.15.118:0/3058698374 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55fe4be35290 msgr2=0x55fe4be356b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:16:18.782 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:18.781+0000 7f2f63e00640 1 --2- 172.21.15.118:0/3058698374 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55fe4be35290 0x55fe4be356b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:16:18.782 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:18.781+0000 7f2f63e00640 1 -- 172.21.15.118:0/3058698374 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55fe4bd29690 con 0x55fe4be35c80 2024-03-24T00:16:18.782 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:18.781+0000 7f2f61600640 1 -- 172.21.15.118:0/1441899990 <== mon.2 v2:172.21.15.189:3300/0 5 ==== osd_map(90..90 src has 1..90) v4 ==== 8277+0+0 (secure 0 0 0) 0x7f2f4c0cefe0 con 0x7f2f641028b0 2024-03-24T00:16:18.782 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:18.781+0000 7f2f68e00640 1 --2- 172.21.15.118:0/1441899990 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7f2f480a2910 0x7f2f480a4dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:16:18.786 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:18.782+0000 7f2f63e00640 1 --2- 172.21.15.118:0/3058698374 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55fe4be35c80 0x55fe4be3a780 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7f2f54009ca0 tx=0x7f2f54011810 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f32a231cda29c8f2 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:16:18.791 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:18.782+0000 7f2f59600640 1 -- 172.21.15.118:0/3058698374 <== mon.2 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2f54004220 con 0x55fe4be35c80 2024-03-24T00:16:18.791 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:18.782+0000 7f2f59600640 1 -- 172.21.15.118:0/3058698374 <== mon.2 v2:172.21.15.189:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f2f540043c0 con 0x55fe4be35c80 2024-03-24T00:16:18.791 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:18.782+0000 7f2f59600640 1 -- 172.21.15.118:0/3058698374 <== mon.2 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2f54019930 con 0x55fe4be35c80 2024-03-24T00:16:18.791 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:18.783+0000 7f2f6dbd7740 1 -- 172.21.15.118:0/3058698374 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55fe4be35c80 msgr2=0x55fe4be3a780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:16:18.791 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:18.783+0000 7f2f6dbd7740 1 --2- 172.21.15.118:0/3058698374 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55fe4be35c80 0x55fe4be3a780 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7f2f54009ca0 tx=0x7f2f54011810 comp rx=0 tx=0).stop 2024-03-24T00:16:18.791 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:18.783+0000 7f2f68e00640 1 --2- 172.21.15.118:0/1441899990 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7f2f480a2910 0x7f2f480a4dd0 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7f2f5c005e30 tx=0x7f2f5c02e040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:16:18.791 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:18.783+0000 7f2f6dbd7740 1 -- 172.21.15.118:0/3058698374 shutdown_connections 2024-03-24T00:16:18.791 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:18.783+0000 7f2f6dbd7740 1 --2- 172.21.15.118:0/3058698374 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55fe4be3adf0 0x55fe4be3d280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:16:18.791 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:18.783+0000 7f2f6dbd7740 1 --2- 172.21.15.118:0/3058698374 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55fe4be35c80 0x55fe4be3a780 unknown :-1 s=CLOSED pgs=218 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:16:18.791 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:18.783+0000 7f2f6dbd7740 1 --2- 172.21.15.118:0/3058698374 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55fe4be35290 0x55fe4be356b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:16:18.791 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:18.783+0000 7f2f6dbd7740 1 -- 172.21.15.118:0/3058698374 >> 172.21.15.118:0/3058698374 conn(0x55fe4bdf9200 msgr2=0x55fe4be2c7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:16:18.791 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:18.783+0000 7f2f6dbd7740 1 -- 172.21.15.118:0/3058698374 shutdown_connections 2024-03-24T00:16:18.792 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:18.783+0000 7f2f6dbd7740 1 -- 172.21.15.118:0/3058698374 wait complete. 2024-03-24T00:16:18.792 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:18.783+0000 7f2f6dbd7740 1 -- start start 2024-03-24T00:16:18.792 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:18.783+0000 7f2f6dbd7740 1 Processor -- start 2024-03-24T00:16:18.792 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:18.784+0000 7f2f6dbd7740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55fe4be35290 0x55fe4be3f200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:16:18.792 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:18.784+0000 7f2f6dbd7740 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55fe4be35c80 0x55fe4be3f740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:16:18.792 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:18.784+0000 7f2f6dbd7740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55fe4be3adf0 0x55fe4be46780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:16:18.792 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:18.784+0000 7f2f6dbd7740 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x55fe4be3dda0 con 0x55fe4be35c80 2024-03-24T00:16:18.792 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:18.784+0000 7f2f6dbd7740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x55fe4bd51e80 con 0x55fe4be35290 2024-03-24T00:16:18.792 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:18.784+0000 7f2f6dbd7740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x55fe4be3df20 con 0x55fe4be3adf0 2024-03-24T00:16:18.792 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:18.784+0000 7f2f6dbd7740 10 client.0 _ll_register_callbacks cb 0x55fe4bd64130 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T00:16:18.792 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:18.784+0000 7f2f63e00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55fe4be35290 0x55fe4be3f200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:16:18.792 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:18.784+0000 7f2f68e00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55fe4be3adf0 0x55fe4be46780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:16:18.792 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:18.784+0000 7f2f63e00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55fe4be35290 0x55fe4be3f200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.118:60126/0 (socket says 172.21.15.118:60126) 2024-03-24T00:16:18.793 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:18.784+0000 7f2f63e00640 1 -- 172.21.15.118:0/250994201 learned_addr learned my addr 172.21.15.118:0/250994201 (peer_addr_for_me v2:172.21.15.118:0/0) 2024-03-24T00:16:18.793 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:18.784+0000 7f2f5be00640 1 --2- 172.21.15.118:0/250994201 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55fe4be35c80 0x55fe4be3f740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:16:18.793 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:18.784+0000 7f2f63e00640 1 -- 172.21.15.118:0/250994201 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55fe4be3adf0 msgr2=0x55fe4be46780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:16:18.793 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:18.785+0000 7f2f63e00640 1 --2- 172.21.15.118:0/250994201 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55fe4be3adf0 0x55fe4be46780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:16:18.793 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:18.785+0000 7f2f63e00640 1 -- 172.21.15.118:0/250994201 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55fe4be35c80 msgr2=0x55fe4be3f740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:16:18.793 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:18.785+0000 7f2f63e00640 1 --2- 172.21.15.118:0/250994201 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55fe4be35c80 0x55fe4be3f740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:16:18.793 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:18.785+0000 7f2f63e00640 1 -- 172.21.15.118:0/250994201 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55fe4bb747d0 con 0x55fe4be35290 2024-03-24T00:16:18.793 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:18.785+0000 7f2f68e00640 1 --2- 172.21.15.118:0/250994201 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55fe4be3adf0 0x55fe4be46780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T00:16:18.793 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:18.785+0000 7f2f5be00640 1 --2- 172.21.15.118:0/250994201 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55fe4be35c80 0x55fe4be3f740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T00:16:18.793 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:18.785+0000 7f2f63e00640 1 --2- 172.21.15.118:0/250994201 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55fe4be35290 0x55fe4be3f200 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7f2f54011d10 tx=0x7f2f54003820 comp rx=0 tx=0).ready entity=mon.2 client_cookie=68fafeb2c8e4ab06 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:16:18.793 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:18.785+0000 7f2f52000640 10 client.0 ms_handle_connect on v2:172.21.15.189:3300/0 2024-03-24T00:16:18.793 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:18.785+0000 7f2f52000640 1 -- 172.21.15.118:0/250994201 <== mon.2 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2f54028070 con 0x55fe4be35290 2024-03-24T00:16:18.793 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:18.785+0000 7f2f52000640 1 -- 172.21.15.118:0/250994201 <== mon.2 v2:172.21.15.189:3300/0 2 ==== osd_map(90..90 src has 1..90) v4 ==== 8277+0+0 (secure 0 0 0) 0x7f2f54005040 con 0x55fe4be35290 2024-03-24T00:16:18.793 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:18.785+0000 7f2f6dbd7740 1 -- 172.21.15.118:0/250994201 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f2f540043c0 con 0x55fe4be35290 2024-03-24T00:16:18.794 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:18.786+0000 7f2f52000640 1 -- 172.21.15.118:0/250994201 <== mon.2 v2:172.21.15.189:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) v2 ==== 24+0+0 (secure 0 0 0) 0x7f2f54034050 con 0x55fe4be35290 2024-03-24T00:16:18.794 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:18.786+0000 7f2f6dbd7740 10 client.25457 fetch_fsmap learned FSMap version 28 2024-03-24T00:16:18.794 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:18.786+0000 7f2f6dbd7740 1 -- 172.21.15.118:0/250994201 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) v3 -- 0x55fe4baaa3f0 con 0x55fe4be35290 2024-03-24T00:16:18.794 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:18.786+0000 7f2f52000640 1 -- 172.21.15.118:0/250994201 <== mon.2 v2:172.21.15.189:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f2f540043c0 con 0x55fe4be35290 2024-03-24T00:16:18.794 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:18.786+0000 7f2f52000640 1 -- 172.21.15.118:0/250994201 <== mon.2 v2:172.21.15.189:3300/0 5 ==== fsmap(e 28) v1 ==== 7934+0+0 (secure 0 0 0) 0x7f2f54004990 con 0x55fe4be35290 2024-03-24T00:16:18.794 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:18.786+0000 7f2f6dbd7740 10 client.25457 fetch_fsmap finished waiting for FSMap version 28 2024-03-24T00:16:18.794 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:18.786+0000 7f2f6dbd7740 10 client.25457 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-24T00:16:18.794 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:18.786+0000 7f2f6dbd7740 20 client.25457 populate_metadata read hostname 'smithi118' 2024-03-24T00:16:18.794 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:18.786+0000 7f2f6dbd7740 1 --2- 172.21.15.118:0/250994201 >> [v2:172.21.15.189:6838/4074484168,v1:172.21.15.189:6839/4074484168] conn(0x55fe4be4ad40 0x55fe4be4d170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:16:18.794 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:18.786+0000 7f2f6dbd7740 4 client.25457 mds_command: new command op to 24569 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-24T00:16:18.794 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:18.786+0000 7f2f6dbd7740 1 -- 172.21.15.118:0/250994201 --> [v2:172.21.15.189:6838/4074484168,v1:172.21.15.189:6839/4074484168] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55fe4bda3790 con 0x55fe4be4ad40 2024-03-24T00:16:18.794 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:18.786+0000 7f2f5be00640 1 --2- 172.21.15.118:0/250994201 >> [v2:172.21.15.189:6838/4074484168,v1:172.21.15.189:6839/4074484168] conn(0x55fe4be4ad40 0x55fe4be4d170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:16:18.795 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:18.787+0000 7f2f52000640 10 client.25457 ms_handle_connect on v2:172.21.15.189:6838/4074484168 2024-03-24T00:16:18.795 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:18.787+0000 7f2f5be00640 1 --2- 172.21.15.118:0/250994201 >> [v2:172.21.15.189:6838/4074484168,v1:172.21.15.189:6839/4074484168] conn(0x55fe4be4ad40 0x55fe4be4d170 crc :-1 s=READY pgs=41 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=1b92e340a2c051ba server_cookie=c363dc7214af67dd in_seq=0 out_seq=0 2024-03-24T00:16:18.795 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:18.788+0000 7f2f52000640 1 -- 172.21.15.118:0/250994201 <== mds.0 v2:172.21.15.189:6838/4074484168 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x55fe4bda3790 con 0x55fe4be4ad40 2024-03-24T00:16:18.795 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:18.788+0000 7f2f52000640 10 client.25457 handle_command_reply: tid=0 2024-03-24T00:16:18.795 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:18.788+0000 7f2f6dbd7740 1 client.25457 shutdown 2024-03-24T00:16:18.795 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:18.788+0000 7f2f6dbd7740 1 -- 172.21.15.118:0/250994201 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55fe4be35290 msgr2=0x55fe4be3f200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:16:18.795 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:18.788+0000 7f2f6dbd7740 1 --2- 172.21.15.118:0/250994201 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55fe4be35290 0x55fe4be3f200 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7f2f54011d10 tx=0x7f2f54003820 comp rx=0 tx=0).stop 2024-03-24T00:16:18.795 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:18.788+0000 7f2f6dbd7740 1 -- 172.21.15.118:0/250994201 shutdown_connections 2024-03-24T00:16:18.795 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:18.788+0000 7f2f6dbd7740 1 --2- 172.21.15.118:0/250994201 >> [v2:172.21.15.189:6838/4074484168,v1:172.21.15.189:6839/4074484168] conn(0x55fe4be4ad40 0x55fe4be4d170 crc :-1 s=READY pgs=41 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:16:18.795 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:18.788+0000 7f2f6dbd7740 1 --2- 172.21.15.118:0/250994201 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55fe4be3adf0 0x55fe4be46780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:16:18.795 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:18.788+0000 7f2f6dbd7740 1 --2- 172.21.15.118:0/250994201 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55fe4be35c80 0x55fe4be3f740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:16:18.795 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:18.788+0000 7f2f6dbd7740 1 --2- 172.21.15.118:0/250994201 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55fe4be35290 0x55fe4be3f200 unknown :-1 s=CLOSED pgs=219 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:16:18.795 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:18.788+0000 7f2f52000640 0 client.25457 ms_handle_reset on v2:172.21.15.189:6838/4074484168 2024-03-24T00:16:18.795 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:18.788+0000 7f2f6dbd7740 1 -- 172.21.15.118:0/250994201 >> 172.21.15.118:0/250994201 conn(0x55fe4bdf9200 msgr2=0x55fe4be2ba90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:16:18.796 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:18.788+0000 7f2f6dbd7740 1 -- 172.21.15.118:0/250994201 shutdown_connections 2024-03-24T00:16:18.796 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:18.788+0000 7f2f6dbd7740 1 -- 172.21.15.118:0/250994201 wait complete. 2024-03-24T00:16:18.796 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:18.788+0000 7f2f6dbd7740 20 client.25457 trim_cache size 0 max 16384 2024-03-24T00:16:18.805 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:18.805+0000 7f2f6dbd7740 1 Processor -- start 2024-03-24T00:16:18.805 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:18.805+0000 7f2f6dbd7740 1 -- start start 2024-03-24T00:16:18.805 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:18.806+0000 7f2f6dbd7740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55fe4be35290 0x55fe4be2fa30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:16:18.805 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:18.806+0000 7f2f6dbd7740 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55fe4be35c80 0x55fe4be46740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:16:18.805 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:18.806+0000 7f2f6dbd7740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55fe4be3adf0 0x55fe4be2ff70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:16:18.806 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:18.806+0000 7f2f6dbd7740 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2f641a68a0 con 0x55fe4be35c80 2024-03-24T00:16:18.806 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:18.806+0000 7f2f6dbd7740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2f641a8220 con 0x55fe4be35290 2024-03-24T00:16:18.806 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:18.806+0000 7f2f6dbd7740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2f641a6f00 con 0x55fe4be3adf0 2024-03-24T00:16:18.807 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:18.806+0000 7f2f63e00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55fe4be35290 0x55fe4be2fa30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:16:18.807 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:18.806+0000 7f2f5be00640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55fe4be35c80 0x55fe4be46740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:16:18.807 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:18.806+0000 7f2f68e00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55fe4be3adf0 0x55fe4be2ff70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:16:18.808 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:18.806+0000 7f2f5be00640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55fe4be35c80 0x55fe4be46740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.118:3300/0 says I am v2:172.21.15.118:37082/0 (socket says 172.21.15.118:37082) 2024-03-24T00:16:18.808 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:18.806+0000 7f2f5be00640 1 -- 172.21.15.118:0/1319636008 learned_addr learned my addr 172.21.15.118:0/1319636008 (peer_addr_for_me v2:172.21.15.118:0/0) 2024-03-24T00:16:18.808 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:18.806+0000 7f2f63e00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55fe4be35290 0x55fe4be2fa30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.118:60134/0 (socket says 172.21.15.118:60134) 2024-03-24T00:16:18.808 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:18.807+0000 7f2f63e00640 1 -- 172.21.15.118:0/1319636008 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55fe4be3adf0 msgr2=0x55fe4be2ff70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:16:18.808 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:18.807+0000 7f2f63e00640 1 --2- 172.21.15.118:0/1319636008 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55fe4be3adf0 0x55fe4be2ff70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:16:18.808 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:18.807+0000 7f2f63e00640 1 -- 172.21.15.118:0/1319636008 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55fe4be35c80 msgr2=0x55fe4be46740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:16:18.808 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:18.807+0000 7f2f63e00640 1 --2- 172.21.15.118:0/1319636008 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55fe4be35c80 0x55fe4be46740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:16:18.808 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:18.807+0000 7f2f63e00640 1 -- 172.21.15.118:0/1319636008 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55fe4ba6a0f0 con 0x55fe4be35290 2024-03-24T00:16:18.808 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:18.807+0000 7f2f68e00640 1 --2- 172.21.15.118:0/1319636008 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55fe4be3adf0 0x55fe4be2ff70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T00:16:18.808 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:18.807+0000 7f2f63e00640 1 --2- 172.21.15.118:0/1319636008 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55fe4be35290 0x55fe4be2fa30 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7f2f54000c00 tx=0x7f2f540097a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=292a90f4ae7bb1b3 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:16:18.808 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:18.807+0000 7f2f58c00640 1 -- 172.21.15.118:0/1319636008 <== mon.2 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2f54028040 con 0x55fe4be35290 2024-03-24T00:16:18.808 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:18.807+0000 7f2f58c00640 1 -- 172.21.15.118:0/1319636008 <== mon.2 v2:172.21.15.189:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f2f540196f0 con 0x55fe4be35290 2024-03-24T00:16:18.808 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:18.807+0000 7f2f58c00640 1 -- 172.21.15.118:0/1319636008 <== mon.2 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2f540052a0 con 0x55fe4be35290 2024-03-24T00:16:18.808 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:18.808+0000 7f2f6dbd7740 1 -- 172.21.15.118:0/1319636008 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55fe4be35290 msgr2=0x55fe4be2fa30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:16:18.808 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:18.808+0000 7f2f6dbd7740 1 --2- 172.21.15.118:0/1319636008 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55fe4be35290 0x55fe4be2fa30 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7f2f54000c00 tx=0x7f2f540097a0 comp rx=0 tx=0).stop 2024-03-24T00:16:18.809 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:18.808+0000 7f2f6dbd7740 1 -- 172.21.15.118:0/1319636008 shutdown_connections 2024-03-24T00:16:18.809 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:18.808+0000 7f2f6dbd7740 1 --2- 172.21.15.118:0/1319636008 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55fe4be3adf0 0x55fe4be2ff70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:16:18.809 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:18.808+0000 7f2f6dbd7740 1 --2- 172.21.15.118:0/1319636008 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55fe4be35c80 0x55fe4be46740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:16:18.809 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:18.808+0000 7f2f6dbd7740 1 --2- 172.21.15.118:0/1319636008 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55fe4be35290 0x55fe4be2fa30 unknown :-1 s=CLOSED pgs=220 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:16:18.809 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:18.808+0000 7f2f6dbd7740 1 -- 172.21.15.118:0/1319636008 >> 172.21.15.118:0/1319636008 conn(0x55fe4bdf9200 msgr2=0x55fe4be36f80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:16:18.809 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:18.808+0000 7f2f6dbd7740 1 -- 172.21.15.118:0/1319636008 shutdown_connections 2024-03-24T00:16:18.809 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:18.808+0000 7f2f6dbd7740 1 -- 172.21.15.118:0/1319636008 wait complete. 2024-03-24T00:16:18.812 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:18.808+0000 7f2f6dbd7740 1 -- start start 2024-03-24T00:16:18.813 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:18.808+0000 7f2f6dbd7740 1 Processor -- start 2024-03-24T00:16:18.813 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:18.809+0000 7f2f6dbd7740 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55fe4be35290 0x55fe4be43580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:16:18.813 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:18.809+0000 7f2f6dbd7740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55fe4be35c80 0x55fe4be43ac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:16:18.813 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:18.809+0000 7f2f6dbd7740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55fe4be3adf0 0x55fe4bea45f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:16:18.813 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:18.809+0000 7f2f6dbd7740 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2f641a7bc0 con 0x55fe4be35290 2024-03-24T00:16:18.813 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:18.809+0000 7f2f6dbd7740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2f641a7560 con 0x55fe4be35c80 2024-03-24T00:16:18.813 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:18.809+0000 7f2f6dbd7740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2f641a4a60 con 0x55fe4be3adf0 2024-03-24T00:16:18.813 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:18.809+0000 7f2f6dbd7740 10 client.0 _ll_register_callbacks cb 0x7f2f54005e30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T00:16:18.813 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:18.809+0000 7f2f5be00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55fe4be35c80 0x55fe4be43ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:16:18.813 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:18.809+0000 7f2f68e00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55fe4be3adf0 0x55fe4bea45f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:16:18.813 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:18.809+0000 7f2f5be00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55fe4be35c80 0x55fe4be43ac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.118:60142/0 (socket says 172.21.15.118:60142) 2024-03-24T00:16:18.813 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:18.809+0000 7f2f5be00640 1 -- 172.21.15.118:0/1488955437 learned_addr learned my addr 172.21.15.118:0/1488955437 (peer_addr_for_me v2:172.21.15.118:0/0) 2024-03-24T00:16:18.813 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:18.809+0000 7f2f63e00640 1 --2- 172.21.15.118:0/1488955437 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55fe4be35290 0x55fe4be43580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:16:18.814 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:18.809+0000 7f2f5be00640 1 -- 172.21.15.118:0/1488955437 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55fe4be3adf0 msgr2=0x55fe4bea45f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:16:18.814 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:18.809+0000 7f2f5be00640 1 --2- 172.21.15.118:0/1488955437 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55fe4be3adf0 0x55fe4bea45f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:16:18.814 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:18.809+0000 7f2f5be00640 1 -- 172.21.15.118:0/1488955437 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55fe4be35290 msgr2=0x55fe4be43580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:16:18.814 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:18.809+0000 7f2f5be00640 1 --2- 172.21.15.118:0/1488955437 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55fe4be35290 0x55fe4be43580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:16:18.814 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:18.810+0000 7f2f5be00640 1 -- 172.21.15.118:0/1488955437 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55fe4bea4b30 con 0x55fe4be35c80 2024-03-24T00:16:18.814 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:18.810+0000 7f2f5be00640 1 --2- 172.21.15.118:0/1488955437 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55fe4be35c80 0x55fe4be43ac0 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7f2f4c0cc9a0 tx=0x7f2f4c0d14b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d26305f87cb14ba0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:16:18.814 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:18.810+0000 7f2f52000640 10 client.0 ms_handle_connect on v2:172.21.15.189:3300/0 2024-03-24T00:16:18.814 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:18.810+0000 7f2f52000640 1 -- 172.21.15.118:0/1488955437 <== mon.2 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2f4c09b110 con 0x55fe4be35c80 2024-03-24T00:16:18.814 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:18.810+0000 7f2f52000640 1 -- 172.21.15.118:0/1488955437 <== mon.2 v2:172.21.15.189:3300/0 2 ==== osd_map(90..90 src has 1..90) v4 ==== 8277+0+0 (secure 0 0 0) 0x7f2f4c005250 con 0x55fe4be35c80 2024-03-24T00:16:18.814 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:18.810+0000 7f2f6dbd7740 1 -- 172.21.15.118:0/1488955437 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55fe4bd9dae0 con 0x55fe4be35c80 2024-03-24T00:16:18.814 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:18.810+0000 7f2f52000640 1 -- 172.21.15.118:0/1488955437 <== mon.2 v2:172.21.15.189:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) v2 ==== 24+0+0 (secure 0 0 0) 0x7f2f4c0d2940 con 0x55fe4be35c80 2024-03-24T00:16:18.814 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:18.810+0000 7f2f6dbd7740 10 client.25463 fetch_fsmap learned FSMap version 28 2024-03-24T00:16:18.814 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:18.810+0000 7f2f6dbd7740 1 -- 172.21.15.118:0/1488955437 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) v3 -- 0x55fe4bea5db0 con 0x55fe4be35c80 2024-03-24T00:16:18.815 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:18.810+0000 7f2f52000640 1 -- 172.21.15.118:0/1488955437 <== mon.2 v2:172.21.15.189:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x55fe4bd9dae0 con 0x55fe4be35c80 2024-03-24T00:16:18.815 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:18.811+0000 7f2f52000640 1 -- 172.21.15.118:0/1488955437 <== mon.2 v2:172.21.15.189:3300/0 5 ==== fsmap(e 28) v1 ==== 7934+0+0 (secure 0 0 0) 0x7f2f4c0046c0 con 0x55fe4be35c80 2024-03-24T00:16:18.815 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:18.811+0000 7f2f6dbd7740 10 client.25463 fetch_fsmap finished waiting for FSMap version 28 2024-03-24T00:16:18.815 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:18.811+0000 7f2f6dbd7740 10 client.25463 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-24T00:16:18.815 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:18.811+0000 7f2f6dbd7740 20 client.25463 populate_metadata read hostname 'smithi118' 2024-03-24T00:16:18.815 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:18.811+0000 7f2f6dbd7740 1 --2- 172.21.15.118:0/1488955437 >> [v2:172.21.15.189:6838/4074484168,v1:172.21.15.189:6839/4074484168] conn(0x55fe4bea66c0 0x55fe4bea8ae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:16:18.815 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:18.811+0000 7f2f6dbd7740 4 client.25463 mds_command: new command op to 24569 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-24T00:16:18.815 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:18.811+0000 7f2f6dbd7740 1 -- 172.21.15.118:0/1488955437 --> [v2:172.21.15.189:6838/4074484168,v1:172.21.15.189:6839/4074484168] -- command(tid 0: {"prefix": "scrub status"}) v1 -- 0x55fe4bd1b3a0 con 0x55fe4bea66c0 2024-03-24T00:16:18.815 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:18.811+0000 7f2f63e00640 1 --2- 172.21.15.118:0/1488955437 >> [v2:172.21.15.189:6838/4074484168,v1:172.21.15.189:6839/4074484168] conn(0x55fe4bea66c0 0x55fe4bea8ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:16:18.815 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:18.811+0000 7f2f52000640 10 client.25463 ms_handle_connect on v2:172.21.15.189:6838/4074484168 2024-03-24T00:16:18.815 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:18.811+0000 7f2f63e00640 1 --2- 172.21.15.118:0/1488955437 >> [v2:172.21.15.189:6838/4074484168,v1:172.21.15.189:6839/4074484168] conn(0x55fe4bea66c0 0x55fe4bea8ae0 crc :-1 s=READY pgs=42 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=b176d2d81a403da9 server_cookie=847a1ad3c7084f11 in_seq=0 out_seq=0 2024-03-24T00:16:18.815 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:18.812+0000 7f2f52000640 1 -- 172.21.15.118:0/1488955437 <== mds.0 v2:172.21.15.189:6838/4074484168 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+63 (crc 0 0 0) 0x55fe4bd1b3a0 con 0x55fe4bea66c0 2024-03-24T00:16:18.815 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:18.812+0000 7f2f52000640 10 client.25463 handle_command_reply: tid=0 2024-03-24T00:16:18.815 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:18.812+0000 7f2f6dbd7740 1 client.25463 shutdown 2024-03-24T00:16:18.815 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:18.812+0000 7f2f6dbd7740 1 -- 172.21.15.118:0/1488955437 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55fe4be35c80 msgr2=0x55fe4be43ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:16:18.816 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:18.812+0000 7f2f6dbd7740 1 --2- 172.21.15.118:0/1488955437 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55fe4be35c80 0x55fe4be43ac0 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7f2f4c0cc9a0 tx=0x7f2f4c0d14b0 comp rx=0 tx=0).stop 2024-03-24T00:16:18.816 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:18.812+0000 7f2f6dbd7740 1 -- 172.21.15.118:0/1488955437 shutdown_connections 2024-03-24T00:16:18.816 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:18.812+0000 7f2f6dbd7740 1 --2- 172.21.15.118:0/1488955437 >> [v2:172.21.15.189:6838/4074484168,v1:172.21.15.189:6839/4074484168] conn(0x55fe4bea66c0 0x55fe4bea8ae0 crc :-1 s=READY pgs=42 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:16:18.816 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:18.812+0000 7f2f6dbd7740 1 --2- 172.21.15.118:0/1488955437 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55fe4be3adf0 0x55fe4bea45f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:16:18.816 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:18.812+0000 7f2f6dbd7740 1 --2- 172.21.15.118:0/1488955437 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55fe4be35c80 0x55fe4be43ac0 unknown :-1 s=CLOSED pgs=221 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:16:18.816 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:18.812+0000 7f2f52000640 0 client.25463 ms_handle_reset on v2:172.21.15.189:6838/4074484168 2024-03-24T00:16:18.816 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:18.812+0000 7f2f6dbd7740 1 --2- 172.21.15.118:0/1488955437 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55fe4be35290 0x55fe4be43580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:16:18.816 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:18.812+0000 7f2f6dbd7740 1 -- 172.21.15.118:0/1488955437 >> 172.21.15.118:0/1488955437 conn(0x55fe4bdf9200 msgr2=0x55fe4be2fa30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:16:18.816 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:18.812+0000 7f2f6dbd7740 1 -- 172.21.15.118:0/1488955437 shutdown_connections 2024-03-24T00:16:18.816 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:18.812+0000 7f2f6dbd7740 1 -- 172.21.15.118:0/1488955437 wait complete. 2024-03-24T00:16:18.816 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:18.812+0000 7f2f6dbd7740 20 client.25463 trim_cache size 0 max 16384 2024-03-24T00:16:18.817 INFO:teuthology.orchestra.run.smithi118.stdout:{ 2024-03-24T00:16:18.817 INFO:teuthology.orchestra.run.smithi118.stdout: "status": "no active scrubs running", 2024-03-24T00:16:18.817 INFO:teuthology.orchestra.run.smithi118.stdout: "scrubs": {} 2024-03-24T00:16:18.817 INFO:teuthology.orchestra.run.smithi118.stdout:} 2024-03-24T00:16:18.818 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:18.813+0000 7f2f6ae00640 1 -- 172.21.15.118:0/1441899990 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7f2f480a2910 msgr2=0x7f2f480a4dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:16:18.818 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:18.813+0000 7f2f6ae00640 1 --2- 172.21.15.118:0/1441899990 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7f2f480a2910 0x7f2f480a4dd0 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7f2f5c005e30 tx=0x7f2f5c02e040 comp rx=0 tx=0).stop 2024-03-24T00:16:18.818 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:18.813+0000 7f2f6ae00640 1 -- 172.21.15.118:0/1441899990 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2f641028b0 msgr2=0x7f2f64197f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:16:18.818 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:18.813+0000 7f2f6ae00640 1 --2- 172.21.15.118:0/1441899990 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2f641028b0 0x7f2f64197f30 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7f2f4c00e9f0 tx=0x7f2f4c00edf0 comp rx=0 tx=0).stop 2024-03-24T00:16:18.818 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:18.813+0000 7f2f6ae00640 1 -- 172.21.15.118:0/1441899990 shutdown_connections 2024-03-24T00:16:18.818 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:18.813+0000 7f2f6ae00640 1 --2- 172.21.15.118:0/1441899990 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7f2f480a2910 0x7f2f480a4dd0 unknown :-1 s=CLOSED pgs=273 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:16:18.818 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:18.813+0000 7f2f6ae00640 1 --2- 172.21.15.118:0/1441899990 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f2f64103270 0x7f2f6419efa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:16:18.818 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:18.813+0000 7f2f6ae00640 1 --2- 172.21.15.118:0/1441899990 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f2f641028b0 0x7f2f64197f30 unknown :-1 s=CLOSED pgs=217 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:16:18.818 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:18.813+0000 7f2f6ae00640 1 --2- 172.21.15.118:0/1441899990 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f2f641016c0 0x7f2f641979f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:16:18.818 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:18.813+0000 7f2f6ae00640 1 -- 172.21.15.118:0/1441899990 >> 172.21.15.118:0/1441899990 conn(0x7f2f640fcc30 msgr2=0x7f2f64105b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:16:18.818 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:18.813+0000 7f2f6ae00640 1 -- 172.21.15.118:0/1441899990 shutdown_connections 2024-03-24T00:16:18.819 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:18.813+0000 7f2f6ae00640 1 -- 172.21.15.118:0/1441899990 wait complete. 2024-03-24T00:16:18.838 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-03-24T00:16:18.838 DEBUG:teuthology.orchestra.run.smithi118:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-24T00:16:18.907 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:16:18 smithi118 ceph-mon[31535]: pgmap v416: 97 pgs: 97 active+clean; 661 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-03-24T00:16:19.036 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:19.036+0000 7fc63ae00640 1 -- 172.21.15.118:0/2699414000 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc634100030 msgr2=0x7fc634100450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:16:19.036 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:19.036+0000 7fc63ae00640 1 --2- 172.21.15.118:0/2699414000 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc634100030 0x7fc634100450 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7fc624009900 tx=0x7fc62402bcd0 comp rx=0 tx=0).stop 2024-03-24T00:16:19.036 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:19.036+0000 7fc63ae00640 1 -- 172.21.15.118:0/2699414000 shutdown_connections 2024-03-24T00:16:19.036 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:19.036+0000 7fc63ae00640 1 --2- 172.21.15.118:0/2699414000 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fc634101330 0x7fc6341017b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:16:19.036 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:19.036+0000 7fc63ae00640 1 --2- 172.21.15.118:0/2699414000 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fc634100990 0x7fc634100df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:16:19.036 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:19.036+0000 7fc63ae00640 1 --2- 172.21.15.118:0/2699414000 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc634100030 0x7fc634100450 unknown :-1 s=CLOSED pgs=222 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:16:19.036 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:19.036+0000 7fc63ae00640 1 -- 172.21.15.118:0/2699414000 >> 172.21.15.118:0/2699414000 conn(0x7fc6340fbbd0 msgr2=0x7fc6340fdff0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:16:19.036 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:19.036+0000 7fc63ae00640 1 -- 172.21.15.118:0/2699414000 shutdown_connections 2024-03-24T00:16:19.037 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:19.036+0000 7fc63ae00640 1 -- 172.21.15.118:0/2699414000 wait complete. 2024-03-24T00:16:19.037 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:19.038+0000 7fc63ae00640 1 Processor -- start 2024-03-24T00:16:19.037 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:19.038+0000 7fc63ae00640 1 -- start start 2024-03-24T00:16:19.039 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:19.038+0000 7fc63ae00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fc634100030 0x7fc634199c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:16:19.039 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:19.038+0000 7fc63ae00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc634100990 0x7fc63419a150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:16:19.039 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:19.038+0000 7fc63ae00640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fc634101330 0x7fc6341a1170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:16:19.039 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:19.039+0000 7fc63ae00640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc63410f5a0 con 0x7fc634101330 2024-03-24T00:16:19.039 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:19.039+0000 7fc63ae00640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc63410f420 con 0x7fc634100990 2024-03-24T00:16:19.039 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:19.039+0000 7fc63ae00640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc63410f720 con 0x7fc634100030 2024-03-24T00:16:19.039 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:19.039+0000 7fc633400640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc634100990 0x7fc63419a150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:16:19.039 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:19.039+0000 7fc633e00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fc634100030 0x7fc634199c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:16:19.039 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:19.039+0000 7fc633400640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc634100990 0x7fc63419a150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.118:60168/0 (socket says 172.21.15.118:60168) 2024-03-24T00:16:19.039 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:19.039+0000 7fc633400640 1 -- 172.21.15.118:0/3331377977 learned_addr learned my addr 172.21.15.118:0/3331377977 (peer_addr_for_me v2:172.21.15.118:0/0) 2024-03-24T00:16:19.039 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:19.039+0000 7fc638e00640 1 --2- 172.21.15.118:0/3331377977 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fc634101330 0x7fc6341a1170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:16:19.039 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:19.039+0000 7fc633400640 1 -- 172.21.15.118:0/3331377977 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fc634100030 msgr2=0x7fc634199c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:16:19.039 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:19.039+0000 7fc633400640 1 --2- 172.21.15.118:0/3331377977 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fc634100030 0x7fc634199c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:16:19.039 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:19.039+0000 7fc633400640 1 -- 172.21.15.118:0/3331377977 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fc634101330 msgr2=0x7fc6341a1170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:16:19.040 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:19.039+0000 7fc633400640 1 --2- 172.21.15.118:0/3331377977 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fc634101330 0x7fc6341a1170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:16:19.040 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:19.039+0000 7fc633400640 1 -- 172.21.15.118:0/3331377977 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc6341a17e0 con 0x7fc634100990 2024-03-24T00:16:19.040 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:19.039+0000 7fc633400640 1 --2- 172.21.15.118:0/3331377977 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc634100990 0x7fc63419a150 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7fc62000e9f0 tx=0x7fc62000edf0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:16:19.040 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:19.040+0000 7fc630c00640 1 -- 172.21.15.118:0/3331377977 <== mon.2 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc62000cd80 con 0x7fc634100990 2024-03-24T00:16:19.040 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:19.040+0000 7fc630c00640 1 -- 172.21.15.118:0/3331377977 <== mon.2 v2:172.21.15.189:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fc620005500 con 0x7fc634100990 2024-03-24T00:16:19.040 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:19.040+0000 7fc630c00640 1 -- 172.21.15.118:0/3331377977 <== mon.2 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc620010690 con 0x7fc634100990 2024-03-24T00:16:19.041 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:19.040+0000 7fc63ae00640 1 -- 172.21.15.118:0/3331377977 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc6341a1ad0 con 0x7fc634100990 2024-03-24T00:16:19.041 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:19.040+0000 7fc63ae00640 1 -- 172.21.15.118:0/3331377977 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc6341a1fd0 con 0x7fc634100990 2024-03-24T00:16:19.041 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:19.041+0000 7fc63ae00640 1 -- 172.21.15.118:0/3331377977 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc63410ad80 con 0x7fc634100990 2024-03-24T00:16:19.043 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:19.043+0000 7fc630c00640 1 -- 172.21.15.118:0/3331377977 <== mon.2 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 19) v1 ==== 137342+0+0 (secure 0 0 0) 0x7fc6200046c0 con 0x7fc634100990 2024-03-24T00:16:19.044 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:19.045+0000 7fc630c00640 1 --2- 172.21.15.118:0/3331377977 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7fc6180a2890 0x7fc6180a4d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:16:19.044 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:19.045+0000 7fc630c00640 1 -- 172.21.15.118:0/3331377977 <== mon.2 v2:172.21.15.189:3300/0 5 ==== osd_map(90..90 src has 1..90) v4 ==== 8277+0+0 (secure 0 0 0) 0x7fc6200cf060 con 0x7fc634100990 2024-03-24T00:16:19.044 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:19.045+0000 7fc633e00640 1 --2- 172.21.15.118:0/3331377977 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7fc6180a2890 0x7fc6180a4d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:16:19.045 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:19.046+0000 7fc633e00640 1 --2- 172.21.15.118:0/3331377977 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7fc6180a2890 0x7fc6180a4d50 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7fc62402c2e0 tx=0x7fc62402c920 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:16:19.048 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:19.049+0000 7fc630c00640 1 -- 172.21.15.118:0/3331377977 <== mon.2 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7fc62009bf10 con 0x7fc634100990 2024-03-24T00:16:19.203 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:16:18 smithi189 ceph-mon[36187]: pgmap v416: 97 pgs: 97 active+clean; 661 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-03-24T00:16:19.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:16:18 smithi193 ceph-mon[36878]: pgmap v416: 97 pgs: 97 active+clean; 661 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 9 op/s 2024-03-24T00:16:19.288 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:19.289+0000 7fc63ae00640 1 -- 172.21.15.118:0/3331377977 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fc63402d070 con 0x7fc634100990 2024-03-24T00:16:19.290 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:19.290+0000 7fc630c00640 1 -- 172.21.15.118:0/3331377977 <== mon.2 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+11478 (secure 0 0 0) 0x7fc62009b810 con 0x7fc634100990 2024-03-24T00:16:19.290 INFO:teuthology.orchestra.run.smithi118.stdout: 2024-03-24T00:16:19.290 INFO:teuthology.orchestra.run.smithi118.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14385,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.118:6835/3194741373","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6834","nonce":3194741373},{"type":"v1","addr":"172.21.15.118:6835","nonce":3194741373}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":4},{"gid":14397,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.118:6837/1374462956","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6836","nonce":1374462956},{"type":"v1","addr":"172.21.15.118:6837","nonce":1374462956}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":28,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-03-24T00:13:01.782474+0000","modified":"2024-03-24T00:15:38.235450+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24569,"mds_1":24550,"mds_2":24511,"mds_3":24499,"mds_4":24488},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14469":{"gid":14469,"name":"a","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.118:6841/3040946537","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6840","nonce":3040946537},{"type":"v1","addr":"172.21.15.118:6841","nonce":3040946537}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24439":{"gid":24439,"name":"l","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.193:6833/1023428412","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6832","nonce":1023428412},{"type":"v1","addr":"172.21.15.193:6833","nonce":1023428412}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24440":{"gid":24440,"name":"k","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.189:6833/2740024078","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6832","nonce":2740024078},{"type":"v1","addr":"172.21.15.189:6833","nonce":2740024078}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24488":{"gid":24488,"name":"e","rank":4,"incarnation":25,"state":"up:active","state_seq":10,"addr":"172.21.15.189:6835/599473205","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6834","nonce":599473205},{"type":"v1","addr":"172.21.15.189:6835","nonce":599473205}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24499":{"gid":24499,"name":"f","rank":3,"incarnation":23,"state":"up:active","state_seq":9,"addr":"172.21.15.193:6835/2055310541","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6834","nonce":2055310541},{"type":"v1","addr":"172.21.15.193:6835","nonce":2055310541}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24511":{"gid":24511,"name":"c","rank":2,"incarnation":21,"state":"up:active","state_seq":8,"addr":"172.21.15.193:6837/2921520053","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6836","nonce":2921520053},{"type":"v1","addr":"172.21.15.193:6837","nonce":2921520053}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24515":{"gid":24515,"name":"h","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.189:6837/3450187137","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6836","nonce":3450187137},{"type":"v1","addr":"172.21.15.189:6837","nonce":3450187137}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24532":{"gid":24532,"name":"i","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.193:6839/2126842603","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6838","nonce":2126842603},{"type":"v1","addr":"172.21.15.193:6839","nonce":2126842603}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24550":{"gid":24550,"name":"g","rank":1,"incarnation":19,"state":"up:active","state_seq":6,"addr":"172.21.15.118:6839/3285788503","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6838","nonce":3285788503},{"type":"v1","addr":"172.21.15.118:6839","nonce":3285788503}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24569":{"gid":24569,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.189:6839/4074484168","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6838","nonce":4074484168},{"type":"v1","addr":"172.21.15.189:6839","nonce":4074484168}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24569,"qdb_cluster":[24569,24499,24488,24511,24550]},"id":1}]} 2024-03-24T00:16:19.291 INFO:teuthology.orchestra.run.smithi118.stderr:dumped fsmap epoch 28 2024-03-24T00:16:19.292 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:19.293+0000 7fc63ae00640 1 -- 172.21.15.118:0/3331377977 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7fc6180a2890 msgr2=0x7fc6180a4d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:16:19.293 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:19.293+0000 7fc63ae00640 1 --2- 172.21.15.118:0/3331377977 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7fc6180a2890 0x7fc6180a4d50 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7fc62402c2e0 tx=0x7fc62402c920 comp rx=0 tx=0).stop 2024-03-24T00:16:19.293 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:19.293+0000 7fc63ae00640 1 -- 172.21.15.118:0/3331377977 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc634100990 msgr2=0x7fc63419a150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:16:19.293 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:19.293+0000 7fc63ae00640 1 --2- 172.21.15.118:0/3331377977 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc634100990 0x7fc63419a150 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7fc62000e9f0 tx=0x7fc62000edf0 comp rx=0 tx=0).stop 2024-03-24T00:16:19.293 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:19.293+0000 7fc63ae00640 1 -- 172.21.15.118:0/3331377977 shutdown_connections 2024-03-24T00:16:19.293 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:19.293+0000 7fc63ae00640 1 --2- 172.21.15.118:0/3331377977 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7fc6180a2890 0x7fc6180a4d50 unknown :-1 s=CLOSED pgs=274 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:16:19.293 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:19.293+0000 7fc63ae00640 1 --2- 172.21.15.118:0/3331377977 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fc634101330 0x7fc6341a1170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:16:19.293 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:19.293+0000 7fc63ae00640 1 --2- 172.21.15.118:0/3331377977 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc634100990 0x7fc63419a150 unknown :-1 s=CLOSED pgs=223 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:16:19.293 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:19.293+0000 7fc63ae00640 1 --2- 172.21.15.118:0/3331377977 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fc634100030 0x7fc634199c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:16:19.293 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:19.293+0000 7fc63ae00640 1 -- 172.21.15.118:0/3331377977 >> 172.21.15.118:0/3331377977 conn(0x7fc6340fbbd0 msgr2=0x7fc6340fd6d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:16:19.293 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:19.293+0000 7fc63ae00640 1 -- 172.21.15.118:0/3331377977 shutdown_connections 2024-03-24T00:16:19.293 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:19.293+0000 7fc63ae00640 1 -- 172.21.15.118:0/3331377977 wait complete. 2024-03-24T00:16:19.313 INFO:teuthology.orchestra.run:Running command with timeout 120 2024-03-24T00:16:19.313 DEBUG:teuthology.orchestra.run.smithi118:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:4 damage ls 2024-03-24T00:16:19.524 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:19.524+0000 7fc9f3a00640 1 -- 172.21.15.118:0/4254155943 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fc9ec1038e0 msgr2=0x7fc9ec103d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:16:19.524 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:19.524+0000 7fc9f3a00640 1 --2- 172.21.15.118:0/4254155943 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fc9ec1038e0 0x7fc9ec103d00 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7fc9e0009900 tx=0x7fc9e002bc30 comp rx=0 tx=0).stop 2024-03-24T00:16:19.524 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:19.524+0000 7fc9f3a00640 1 -- 172.21.15.118:0/4254155943 shutdown_connections 2024-03-24T00:16:19.524 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:19.524+0000 7fc9f3a00640 1 --2- 172.21.15.118:0/4254155943 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fc9ec105490 0x7fc9ec10bd20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:16:19.524 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:19.524+0000 7fc9f3a00640 1 --2- 172.21.15.118:0/4254155943 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc9ec104ad0 0x7fc9ec104f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:16:19.524 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:19.524+0000 7fc9f3a00640 1 --2- 172.21.15.118:0/4254155943 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fc9ec1038e0 0x7fc9ec103d00 unknown :-1 s=CLOSED pgs=209 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:16:19.524 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:19.524+0000 7fc9f3a00640 1 -- 172.21.15.118:0/4254155943 >> 172.21.15.118:0/4254155943 conn(0x7fc9ec0fee50 msgr2=0x7fc9ec101290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:16:19.525 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:19.524+0000 7fc9f3a00640 1 -- 172.21.15.118:0/4254155943 shutdown_connections 2024-03-24T00:16:19.525 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:19.524+0000 7fc9f3a00640 1 -- 172.21.15.118:0/4254155943 wait complete. 2024-03-24T00:16:19.527 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:19.525+0000 7fc9f3a00640 1 Processor -- start 2024-03-24T00:16:19.527 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:19.525+0000 7fc9f3a00640 1 -- start start 2024-03-24T00:16:19.527 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:19.526+0000 7fc9f3a00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc9ec1038e0 0x7fc9ec199c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:16:19.527 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:19.526+0000 7fc9f3a00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fc9ec104ad0 0x7fc9ec19a180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:16:19.527 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:19.526+0000 7fc9f3a00640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fc9ec105490 0x7fc9ec1a11b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:16:19.527 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:19.526+0000 7fc9f3a00640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc9ec10f580 con 0x7fc9ec105490 2024-03-24T00:16:19.527 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:19.526+0000 7fc9f3a00640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc9ec10f400 con 0x7fc9ec1038e0 2024-03-24T00:16:19.527 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:19.526+0000 7fc9f3a00640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc9ec10f700 con 0x7fc9ec104ad0 2024-03-24T00:16:19.527 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:19.526+0000 7fc9f1000640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc9ec1038e0 0x7fc9ec199c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:16:19.527 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:19.526+0000 7fc9f1000640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc9ec1038e0 0x7fc9ec199c40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.118:60196/0 (socket says 172.21.15.118:60196) 2024-03-24T00:16:19.527 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:19.526+0000 7fc9f1000640 1 -- 172.21.15.118:0/1431881681 learned_addr learned my addr 172.21.15.118:0/1431881681 (peer_addr_for_me v2:172.21.15.118:0/0) 2024-03-24T00:16:19.527 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:19.526+0000 7fc9ebe00640 1 --2- 172.21.15.118:0/1431881681 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fc9ec104ad0 0x7fc9ec19a180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:16:19.527 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:19.526+0000 7fc9f1a00640 1 --2- 172.21.15.118:0/1431881681 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fc9ec105490 0x7fc9ec1a11b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:16:19.527 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:19.526+0000 7fc9f1000640 1 -- 172.21.15.118:0/1431881681 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fc9ec104ad0 msgr2=0x7fc9ec19a180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:16:19.528 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:19.526+0000 7fc9f1000640 1 --2- 172.21.15.118:0/1431881681 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fc9ec104ad0 0x7fc9ec19a180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:16:19.528 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:19.526+0000 7fc9f1000640 1 -- 172.21.15.118:0/1431881681 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fc9ec105490 msgr2=0x7fc9ec1a11b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:16:19.528 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:19.526+0000 7fc9f1000640 1 --2- 172.21.15.118:0/1431881681 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fc9ec105490 0x7fc9ec1a11b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:16:19.528 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:19.526+0000 7fc9f1000640 1 -- 172.21.15.118:0/1431881681 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc9ec1a1850 con 0x7fc9ec1038e0 2024-03-24T00:16:19.528 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:19.526+0000 7fc9f1a00640 1 --2- 172.21.15.118:0/1431881681 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fc9ec105490 0x7fc9ec1a11b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T00:16:19.528 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:19.527+0000 7fc9f1000640 1 --2- 172.21.15.118:0/1431881681 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc9ec1038e0 0x7fc9ec199c40 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7fc9e0000c00 tx=0x7fc9e002c6c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:16:19.528 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:19.527+0000 7fc9e9600640 1 -- 172.21.15.118:0/1431881681 <== mon.2 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc9e0043020 con 0x7fc9ec1038e0 2024-03-24T00:16:19.528 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:19.527+0000 7fc9e9600640 1 -- 172.21.15.118:0/1431881681 <== mon.2 v2:172.21.15.189:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fc9e00054e0 con 0x7fc9ec1038e0 2024-03-24T00:16:19.528 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:19.527+0000 7fc9f3a00640 1 -- 172.21.15.118:0/1431881681 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc9ec1a1ae0 con 0x7fc9ec1038e0 2024-03-24T00:16:19.528 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:19.527+0000 7fc9e9600640 1 -- 172.21.15.118:0/1431881681 <== mon.2 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc9e0003ce0 con 0x7fc9ec1038e0 2024-03-24T00:16:19.528 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:19.527+0000 7fc9f3a00640 1 -- 172.21.15.118:0/1431881681 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc9ec1a2000 con 0x7fc9ec1038e0 2024-03-24T00:16:19.530 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:19.530+0000 7fc9f663c740 1 Processor -- start 2024-03-24T00:16:19.530 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:19.530+0000 7fc9e9600640 1 -- 172.21.15.118:0/1431881681 <== mon.2 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 19) v1 ==== 137342+0+0 (secure 0 0 0) 0x7fc9e0046020 con 0x7fc9ec1038e0 2024-03-24T00:16:19.530 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:19.530+0000 7fc9f663c740 1 -- start start 2024-03-24T00:16:19.531 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:19.531+0000 7fc9f663c740 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55dff7116810 0x55dff7116c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:16:19.531 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:19.531+0000 7fc9f663c740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55dff71179e0 0x55dff7117e60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:16:19.531 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:19.531+0000 7fc9f663c740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55dff71183a0 0x55dff711ec30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:16:19.531 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:19.531+0000 7fc9f663c740 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x55dff707b2c0 con 0x55dff7116810 2024-03-24T00:16:19.531 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:19.531+0000 7fc9f663c740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x55dff7023c90 con 0x55dff71179e0 2024-03-24T00:16:19.531 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:19.531+0000 7fc9f663c740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x55dff7077360 con 0x55dff71183a0 2024-03-24T00:16:19.532 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:19.531+0000 7fc9ebe00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55dff71183a0 0x55dff711ec30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:16:19.532 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:19.531+0000 7fc9f1000640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55dff71179e0 0x55dff7117e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:16:19.532 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:19.531+0000 7fc9f1a00640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55dff7116810 0x55dff7116c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:16:19.532 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:19.531+0000 7fc9ebe00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55dff71183a0 0x55dff711ec30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.118:45408/0 (socket says 172.21.15.118:45408) 2024-03-24T00:16:19.532 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:19.531+0000 7fc9ebe00640 1 -- 172.21.15.118:0/3754467702 learned_addr learned my addr 172.21.15.118:0/3754467702 (peer_addr_for_me v2:172.21.15.118:0/0) 2024-03-24T00:16:19.532 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:19.532+0000 7fc9ebe00640 1 -- 172.21.15.118:0/3754467702 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55dff71179e0 msgr2=0x55dff7117e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:16:19.532 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:19.532+0000 7fc9ebe00640 1 --2- 172.21.15.118:0/3754467702 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55dff71179e0 0x55dff7117e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:16:19.532 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:19.532+0000 7fc9ebe00640 1 -- 172.21.15.118:0/3754467702 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55dff7116810 msgr2=0x55dff7116c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:16:19.532 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:19.532+0000 7fc9ebe00640 1 --2- 172.21.15.118:0/3754467702 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55dff7116810 0x55dff7116c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:16:19.532 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:19.532+0000 7fc9ebe00640 1 -- 172.21.15.118:0/3754467702 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55dff7048080 con 0x55dff71183a0 2024-03-24T00:16:19.532 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:19.532+0000 7fc9f1000640 1 --2- 172.21.15.118:0/3754467702 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55dff71179e0 0x55dff7117e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T00:16:19.532 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:19.532+0000 7fc9ebe00640 1 --2- 172.21.15.118:0/3754467702 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55dff71183a0 0x55dff711ec30 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7fc9d8009cf0 tx=0x7fc9d8011810 comp rx=0 tx=0).ready entity=mon.1 client_cookie=55b838ec8c62d863 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:16:19.534 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:19.532+0000 7fc9de000640 1 -- 172.21.15.118:0/3754467702 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc9d8004230 con 0x55dff71183a0 2024-03-24T00:16:19.534 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:19.532+0000 7fc9e9600640 1 --2- 172.21.15.118:0/1431881681 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7fc9d40a2890 0x7fc9d40a4d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:16:19.534 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:19.532+0000 7fc9de000640 1 -- 172.21.15.118:0/3754467702 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fc9d80043d0 con 0x55dff71183a0 2024-03-24T00:16:19.534 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:19.532+0000 7fc9de000640 1 -- 172.21.15.118:0/3754467702 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc9d8019960 con 0x55dff71183a0 2024-03-24T00:16:19.534 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:19.532+0000 7fc9e9600640 1 -- 172.21.15.118:0/1431881681 <== mon.2 v2:172.21.15.189:3300/0 5 ==== osd_map(90..90 src has 1..90) v4 ==== 8277+0+0 (secure 0 0 0) 0x7fc9e00f0d90 con 0x7fc9ec1038e0 2024-03-24T00:16:19.534 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:19.533+0000 7fc9f1a00640 1 --2- 172.21.15.118:0/1431881681 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7fc9d40a2890 0x7fc9d40a4d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:16:19.534 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:19.533+0000 7fc9f663c740 1 -- 172.21.15.118:0/3754467702 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55dff71183a0 msgr2=0x55dff711ec30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:16:19.534 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:19.533+0000 7fc9f663c740 1 --2- 172.21.15.118:0/3754467702 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55dff71183a0 0x55dff711ec30 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7fc9d8009cf0 tx=0x7fc9d8011810 comp rx=0 tx=0).stop 2024-03-24T00:16:19.534 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:19.533+0000 7fc9f663c740 1 -- 172.21.15.118:0/3754467702 shutdown_connections 2024-03-24T00:16:19.534 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:19.533+0000 7fc9f663c740 1 --2- 172.21.15.118:0/3754467702 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55dff71183a0 0x55dff711ec30 unknown :-1 s=CLOSED pgs=210 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:16:19.534 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:19.533+0000 7fc9f663c740 1 --2- 172.21.15.118:0/3754467702 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55dff71179e0 0x55dff7117e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:16:19.534 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:19.533+0000 7fc9f663c740 1 --2- 172.21.15.118:0/3754467702 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55dff7116810 0x55dff7116c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:16:19.535 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:19.533+0000 7fc9f663c740 1 -- 172.21.15.118:0/3754467702 >> 172.21.15.118:0/3754467702 conn(0x55dff7111c00 msgr2=0x55dff7112000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:16:19.535 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:19.533+0000 7fc9f663c740 1 -- 172.21.15.118:0/3754467702 shutdown_connections 2024-03-24T00:16:19.535 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:19.533+0000 7fc9f663c740 1 -- 172.21.15.118:0/3754467702 wait complete. 2024-03-24T00:16:19.535 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:19.533+0000 7fc9f1a00640 1 --2- 172.21.15.118:0/1431881681 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7fc9d40a2890 0x7fc9d40a4d50 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x55dff71178e0 tx=0x7fc9e40098a0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:16:19.535 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:19.534+0000 7fc9f663c740 1 -- start start 2024-03-24T00:16:19.535 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:19.534+0000 7fc9f663c740 1 Processor -- start 2024-03-24T00:16:19.537 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:19.535+0000 7fc9f663c740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55dff7116810 0x55dff7120ae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:16:19.538 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:19.535+0000 7fc9f663c740 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55dff71179e0 0x55dff7121020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:16:19.538 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:19.535+0000 7fc9f663c740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55dff71183a0 0x55dff7128060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:16:19.538 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:19.535+0000 7fc9f663c740 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x55dff711f3b0 con 0x55dff71179e0 2024-03-24T00:16:19.538 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:19.535+0000 7fc9f663c740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x55dff70370d0 con 0x55dff7116810 2024-03-24T00:16:19.538 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:19.535+0000 7fc9f663c740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x55dff711f530 con 0x55dff71183a0 2024-03-24T00:16:19.538 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:19.535+0000 7fc9f663c740 10 client.0 _ll_register_callbacks cb 0x55dff703be80 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T00:16:19.538 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:19.535+0000 7fc9f1000640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55dff7116810 0x55dff7120ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:16:19.538 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:19.535+0000 7fc9f1000640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55dff7116810 0x55dff7120ae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.118:60216/0 (socket says 172.21.15.118:60216) 2024-03-24T00:16:19.538 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:19.535+0000 7fc9f1000640 1 -- 172.21.15.118:0/2853785561 learned_addr learned my addr 172.21.15.118:0/2853785561 (peer_addr_for_me v2:172.21.15.118:0/0) 2024-03-24T00:16:19.538 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:19.535+0000 7fc9ebe00640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55dff71179e0 0x55dff7121020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:16:19.538 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:19.535+0000 7fc9f1a00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55dff71183a0 0x55dff7128060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:16:19.538 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:19.535+0000 7fc9f1000640 1 -- 172.21.15.118:0/2853785561 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55dff71183a0 msgr2=0x55dff7128060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:16:19.538 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:19.535+0000 7fc9f1000640 1 --2- 172.21.15.118:0/2853785561 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55dff71183a0 0x55dff7128060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:16:19.539 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:19.535+0000 7fc9f1000640 1 -- 172.21.15.118:0/2853785561 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55dff71179e0 msgr2=0x55dff7121020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:16:19.539 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:19.535+0000 7fc9f1000640 1 --2- 172.21.15.118:0/2853785561 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55dff71179e0 0x55dff7121020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:16:19.539 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:19.536+0000 7fc9f1000640 1 -- 172.21.15.118:0/2853785561 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55dff6d8f3f0 con 0x55dff7116810 2024-03-24T00:16:19.539 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:19.536+0000 7fc9f1000640 1 --2- 172.21.15.118:0/2853785561 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55dff7116810 0x55dff7120ae0 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7fc9e002c170 tx=0x7fc9e00b91a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=863053200483bc70 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:16:19.539 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:19.536+0000 7fc9d2a00640 10 client.0 ms_handle_connect on v2:172.21.15.189:3300/0 2024-03-24T00:16:19.539 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:19.536+0000 7fc9d2a00640 1 -- 172.21.15.118:0/2853785561 <== mon.2 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc9e00b9860 con 0x55dff7116810 2024-03-24T00:16:19.539 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:19.536+0000 7fc9d2a00640 1 -- 172.21.15.118:0/2853785561 <== mon.2 v2:172.21.15.189:3300/0 2 ==== osd_map(90..90 src has 1..90) v4 ==== 8277+0+0 (secure 0 0 0) 0x7fc9e00c5960 con 0x55dff7116810 2024-03-24T00:16:19.539 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:19.536+0000 7fc9f663c740 1 -- 172.21.15.118:0/2853785561 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fc9d80043d0 con 0x55dff7116810 2024-03-24T00:16:19.539 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:19.536+0000 7fc9d2a00640 1 -- 172.21.15.118:0/2853785561 <== mon.2 v2:172.21.15.189:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) v2 ==== 24+0+0 (secure 0 0 0) 0x7fc9e00bb790 con 0x55dff7116810 2024-03-24T00:16:19.539 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:19.536+0000 7fc9f663c740 10 client.25481 fetch_fsmap learned FSMap version 28 2024-03-24T00:16:19.539 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:19.536+0000 7fc9f663c740 1 -- 172.21.15.118:0/2853785561 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) v3 -- 0x55dff6e597d0 con 0x55dff7116810 2024-03-24T00:16:19.539 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:19.537+0000 7fc9d2a00640 1 -- 172.21.15.118:0/2853785561 <== mon.2 v2:172.21.15.189:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fc9d80043d0 con 0x55dff7116810 2024-03-24T00:16:19.539 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:19.537+0000 7fc9d2a00640 1 -- 172.21.15.118:0/2853785561 <== mon.2 v2:172.21.15.189:3300/0 5 ==== fsmap(e 28) v1 ==== 7934+0+0 (secure 0 0 0) 0x7fc9e00a6b50 con 0x55dff7116810 2024-03-24T00:16:19.539 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:19.537+0000 7fc9f663c740 10 client.25481 fetch_fsmap finished waiting for FSMap version 28 2024-03-24T00:16:19.539 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:19.537+0000 7fc9f663c740 10 client.25481 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.e 2024-03-24T00:16:19.540 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:19.537+0000 7fc9f663c740 20 client.25481 populate_metadata read hostname 'smithi118' 2024-03-24T00:16:19.540 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:19.537+0000 7fc9f663c740 1 --2- 172.21.15.118:0/2853785561 >> [v2:172.21.15.189:6834/599473205,v1:172.21.15.189:6835/599473205] conn(0x55dff712c550 0x55dff712e9a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:16:19.540 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:19.537+0000 7fc9f663c740 4 client.25481 mds_command: new command op to 24488 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-24T00:16:19.540 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:19.537+0000 7fc9f663c740 1 -- 172.21.15.118:0/2853785561 --> [v2:172.21.15.189:6834/599473205,v1:172.21.15.189:6835/599473205] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55dff7088800 con 0x55dff712c550 2024-03-24T00:16:19.540 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:19.537+0000 7fc9ebe00640 1 --2- 172.21.15.118:0/2853785561 >> [v2:172.21.15.189:6834/599473205,v1:172.21.15.189:6835/599473205] conn(0x55dff712c550 0x55dff712e9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:16:19.540 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:19.538+0000 7fc9ebe00640 1 --2- 172.21.15.118:0/2853785561 >> [v2:172.21.15.189:6834/599473205,v1:172.21.15.189:6835/599473205] conn(0x55dff712c550 0x55dff712e9a0 crc :-1 s=READY pgs=22 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=8fb5f590a7444c39 server_cookie=8f783a51fe9b3069 in_seq=0 out_seq=0 2024-03-24T00:16:19.540 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:19.538+0000 7fc9d2a00640 10 client.25481 ms_handle_connect on v2:172.21.15.189:6834/599473205 2024-03-24T00:16:19.541 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:19.539+0000 7fc9d2a00640 1 -- 172.21.15.118:0/2853785561 <== mds.4 v2:172.21.15.189:6834/599473205 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x55dff7088800 con 0x55dff712c550 2024-03-24T00:16:19.541 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:19.539+0000 7fc9d2a00640 10 client.25481 handle_command_reply: tid=0 2024-03-24T00:16:19.541 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:19.539+0000 7fc9f663c740 1 client.25481 shutdown 2024-03-24T00:16:19.541 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:19.539+0000 7fc9f663c740 1 -- 172.21.15.118:0/2853785561 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55dff7116810 msgr2=0x55dff7120ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:16:19.541 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:19.539+0000 7fc9f663c740 1 --2- 172.21.15.118:0/2853785561 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55dff7116810 0x55dff7120ae0 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7fc9e002c170 tx=0x7fc9e00b91a0 comp rx=0 tx=0).stop 2024-03-24T00:16:19.541 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:19.539+0000 7fc9f663c740 1 -- 172.21.15.118:0/2853785561 shutdown_connections 2024-03-24T00:16:19.541 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:19.539+0000 7fc9f663c740 1 --2- 172.21.15.118:0/2853785561 >> [v2:172.21.15.189:6834/599473205,v1:172.21.15.189:6835/599473205] conn(0x55dff712c550 0x55dff712e9a0 crc :-1 s=READY pgs=22 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:16:19.541 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:19.539+0000 7fc9f663c740 1 --2- 172.21.15.118:0/2853785561 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55dff71183a0 0x55dff7128060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:16:19.541 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:19.539+0000 7fc9f663c740 1 --2- 172.21.15.118:0/2853785561 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55dff71179e0 0x55dff7121020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:16:19.541 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:19.539+0000 7fc9f663c740 1 --2- 172.21.15.118:0/2853785561 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55dff7116810 0x55dff7120ae0 unknown :-1 s=CLOSED pgs=225 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:16:19.542 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:19.539+0000 7fc9f663c740 1 -- 172.21.15.118:0/2853785561 >> 172.21.15.118:0/2853785561 conn(0x55dff7111c00 msgr2=0x55dff7110eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:16:19.542 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:19.539+0000 7fc9d2a00640 0 client.25481 ms_handle_reset on v2:172.21.15.189:6834/599473205 2024-03-24T00:16:19.542 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:19.539+0000 7fc9f663c740 1 -- 172.21.15.118:0/2853785561 shutdown_connections 2024-03-24T00:16:19.542 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:19.539+0000 7fc9f663c740 1 -- 172.21.15.118:0/2853785561 wait complete. 2024-03-24T00:16:19.542 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:19.539+0000 7fc9f663c740 20 client.25481 trim_cache size 0 max 16384 2024-03-24T00:16:19.557 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:19.558+0000 7fc9f663c740 1 Processor -- start 2024-03-24T00:16:19.558 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:19.558+0000 7fc9f663c740 1 -- start start 2024-03-24T00:16:19.559 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:19.558+0000 7fc9f663c740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55dff7116810 0x55dff7117dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:16:19.559 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:19.558+0000 7fc9f663c740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55dff71179e0 0x55dff7128020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:16:19.559 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:19.559+0000 7fc9f663c740 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55dff71183a0 0x55dff711e920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:16:19.559 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:19.559+0000 7fc9f663c740 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc9ec1aac30 con 0x55dff71183a0 2024-03-24T00:16:19.559 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:19.559+0000 7fc9f663c740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc9ec1ac5b0 con 0x55dff7116810 2024-03-24T00:16:19.559 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:19.559+0000 7fc9f663c740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc9ec1ab290 con 0x55dff71179e0 2024-03-24T00:16:19.560 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:19.559+0000 7fc9f1000640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55dff7116810 0x55dff7117dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:16:19.560 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:19.559+0000 7fc9ebe00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55dff71179e0 0x55dff7128020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:16:19.560 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:19.559+0000 7fc9ebe00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55dff71179e0 0x55dff7128020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.118:45432/0 (socket says 172.21.15.118:45432) 2024-03-24T00:16:19.560 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:19.559+0000 7fc9ebe00640 1 -- 172.21.15.118:0/3686078675 learned_addr learned my addr 172.21.15.118:0/3686078675 (peer_addr_for_me v2:172.21.15.118:0/0) 2024-03-24T00:16:19.560 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:19.559+0000 7fc9f1000640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55dff7116810 0x55dff7117dc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.118:60218/0 (socket says 172.21.15.118:60218) 2024-03-24T00:16:19.560 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:19.559+0000 7fc9f1a00640 1 --2- 172.21.15.118:0/3686078675 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55dff71183a0 0x55dff711e920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:16:19.560 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:19.559+0000 7fc9ebe00640 1 -- 172.21.15.118:0/3686078675 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55dff7116810 msgr2=0x55dff7117dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:16:19.560 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:19.559+0000 7fc9ebe00640 1 --2- 172.21.15.118:0/3686078675 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55dff7116810 0x55dff7117dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:16:19.560 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:19.559+0000 7fc9ebe00640 1 -- 172.21.15.118:0/3686078675 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55dff71183a0 msgr2=0x55dff711e920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:16:19.560 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:19.559+0000 7fc9ebe00640 1 --2- 172.21.15.118:0/3686078675 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55dff71183a0 0x55dff711e920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:16:19.560 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:19.559+0000 7fc9ebe00640 1 -- 172.21.15.118:0/3686078675 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55dff710d700 con 0x55dff71179e0 2024-03-24T00:16:19.560 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:19.559+0000 7fc9f1000640 1 --2- 172.21.15.118:0/3686078675 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55dff7116810 0x55dff7117dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T00:16:19.560 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:19.559+0000 7fc9ebe00640 1 --2- 172.21.15.118:0/3686078675 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55dff71179e0 0x55dff7128020 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7fc9d8009c40 tx=0x7fc9d8005850 comp rx=0 tx=0).ready entity=mon.1 client_cookie=ca37637c4bee95c1 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:16:19.560 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:19.560+0000 7fc9dd600640 1 -- 172.21.15.118:0/3686078675 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc9d8031070 con 0x55dff71179e0 2024-03-24T00:16:19.561 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:19.560+0000 7fc9dd600640 1 -- 172.21.15.118:0/3686078675 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fc9d802b2f0 con 0x55dff71179e0 2024-03-24T00:16:19.561 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:19.560+0000 7fc9dd600640 1 -- 172.21.15.118:0/3686078675 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc9d8003b90 con 0x55dff71179e0 2024-03-24T00:16:19.564 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:19.560+0000 7fc9f663c740 1 -- 172.21.15.118:0/3686078675 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55dff71179e0 msgr2=0x55dff7128020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:16:19.564 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:19.560+0000 7fc9f663c740 1 --2- 172.21.15.118:0/3686078675 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55dff71179e0 0x55dff7128020 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7fc9d8009c40 tx=0x7fc9d8005850 comp rx=0 tx=0).stop 2024-03-24T00:16:19.564 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:19.560+0000 7fc9f663c740 1 -- 172.21.15.118:0/3686078675 shutdown_connections 2024-03-24T00:16:19.564 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:19.560+0000 7fc9f663c740 1 --2- 172.21.15.118:0/3686078675 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55dff71183a0 0x55dff711e920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:16:19.564 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:19.561+0000 7fc9f663c740 1 --2- 172.21.15.118:0/3686078675 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55dff71179e0 0x55dff7128020 unknown :-1 s=CLOSED pgs=211 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:16:19.564 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:19.561+0000 7fc9f663c740 1 --2- 172.21.15.118:0/3686078675 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55dff7116810 0x55dff7117dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:16:19.564 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:19.561+0000 7fc9f663c740 1 -- 172.21.15.118:0/3686078675 >> 172.21.15.118:0/3686078675 conn(0x55dff7111c00 msgr2=0x55dff71156f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:16:19.564 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:19.561+0000 7fc9f663c740 1 -- 172.21.15.118:0/3686078675 shutdown_connections 2024-03-24T00:16:19.564 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:19.561+0000 7fc9f663c740 1 -- 172.21.15.118:0/3686078675 wait complete. 2024-03-24T00:16:19.564 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:19.561+0000 7fc9f663c740 1 -- start start 2024-03-24T00:16:19.564 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:19.561+0000 7fc9f663c740 1 Processor -- start 2024-03-24T00:16:19.564 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:19.562+0000 7fc9f663c740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55dff7116810 0x55dff7123d30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:16:19.565 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:19.562+0000 7fc9f663c740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55dff71179e0 0x55dff7124270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:16:19.565 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:19.562+0000 7fc9f663c740 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55dff71183a0 0x55dff7127220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:16:19.565 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:19.562+0000 7fc9f663c740 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc9ec1abf50 con 0x55dff71183a0 2024-03-24T00:16:19.565 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:19.562+0000 7fc9f663c740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc9ec1ab8f0 con 0x55dff7116810 2024-03-24T00:16:19.565 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:19.562+0000 7fc9f663c740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc9ec1a8df0 con 0x55dff71179e0 2024-03-24T00:16:19.565 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:19.562+0000 7fc9f663c740 10 client.0 _ll_register_callbacks cb 0x7fc9e0005b50 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T00:16:19.565 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:19.562+0000 7fc9f1000640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55dff7116810 0x55dff7123d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:16:19.565 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:19.562+0000 7fc9ebe00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55dff71179e0 0x55dff7124270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:16:19.565 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:19.562+0000 7fc9f1000640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55dff7116810 0x55dff7123d30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.118:60222/0 (socket says 172.21.15.118:60222) 2024-03-24T00:16:19.565 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:19.562+0000 7fc9ebe00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55dff71179e0 0x55dff7124270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.118:45442/0 (socket says 172.21.15.118:45442) 2024-03-24T00:16:19.565 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:19.562+0000 7fc9f1000640 1 -- 172.21.15.118:0/2847529476 learned_addr learned my addr 172.21.15.118:0/2847529476 (peer_addr_for_me v2:172.21.15.118:0/0) 2024-03-24T00:16:19.565 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:19.562+0000 7fc9f1a00640 1 --2- 172.21.15.118:0/2847529476 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55dff71183a0 0x55dff7127220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:16:19.565 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:19.563+0000 7fc9f1000640 1 -- 172.21.15.118:0/2847529476 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55dff71179e0 msgr2=0x55dff7124270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:16:19.565 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:19.563+0000 7fc9f1000640 1 --2- 172.21.15.118:0/2847529476 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55dff71179e0 0x55dff7124270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:16:19.566 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:19.563+0000 7fc9f1000640 1 -- 172.21.15.118:0/2847529476 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55dff71183a0 msgr2=0x55dff7127220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:16:19.566 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:19.563+0000 7fc9f1000640 1 --2- 172.21.15.118:0/2847529476 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55dff71183a0 0x55dff7127220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:16:19.566 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:19.563+0000 7fc9f1000640 1 -- 172.21.15.118:0/2847529476 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55dff7127810 con 0x55dff7116810 2024-03-24T00:16:19.566 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:19.563+0000 7fc9ebe00640 1 --2- 172.21.15.118:0/2847529476 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55dff71179e0 0x55dff7124270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T00:16:19.566 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:19.563+0000 7fc9f1000640 1 --2- 172.21.15.118:0/2847529476 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55dff7116810 0x55dff7123d30 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7fc9e0006010 tx=0x7fc9e00eeaa0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=8211a12a9e93a6bf server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:16:19.566 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:19.563+0000 7fc9d2a00640 10 client.0 ms_handle_connect on v2:172.21.15.189:3300/0 2024-03-24T00:16:19.566 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:19.563+0000 7fc9d2a00640 1 -- 172.21.15.118:0/2847529476 <== mon.2 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc9e00bcfb0 con 0x55dff7116810 2024-03-24T00:16:19.566 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:19.563+0000 7fc9d2a00640 1 -- 172.21.15.118:0/2847529476 <== mon.2 v2:172.21.15.189:3300/0 2 ==== osd_map(90..90 src has 1..90) v4 ==== 8277+0+0 (secure 0 0 0) 0x7fc9e00f4310 con 0x55dff7116810 2024-03-24T00:16:19.566 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:19.563+0000 7fc9f663c740 1 -- 172.21.15.118:0/2847529476 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55dff7082b50 con 0x55dff7116810 2024-03-24T00:16:19.566 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:19.563+0000 7fc9d2a00640 1 -- 172.21.15.118:0/2847529476 <== mon.2 v2:172.21.15.189:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) v2 ==== 24+0+0 (secure 0 0 0) 0x7fc9e00bd190 con 0x55dff7116810 2024-03-24T00:16:19.566 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:19.563+0000 7fc9f663c740 10 client.25487 fetch_fsmap learned FSMap version 28 2024-03-24T00:16:19.566 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:19.563+0000 7fc9f663c740 1 -- 172.21.15.118:0/2847529476 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) v3 -- 0x55dff7190c80 con 0x55dff7116810 2024-03-24T00:16:19.566 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:19.564+0000 7fc9d2a00640 1 -- 172.21.15.118:0/2847529476 <== mon.2 v2:172.21.15.189:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x55dff7082b50 con 0x55dff7116810 2024-03-24T00:16:19.566 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:19.564+0000 7fc9d2a00640 1 -- 172.21.15.118:0/2847529476 <== mon.2 v2:172.21.15.189:3300/0 5 ==== fsmap(e 28) v1 ==== 7934+0+0 (secure 0 0 0) 0x7fc9e00d6220 con 0x55dff7116810 2024-03-24T00:16:19.567 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:19.564+0000 7fc9f663c740 10 client.25487 fetch_fsmap finished waiting for FSMap version 28 2024-03-24T00:16:19.567 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:19.564+0000 7fc9f663c740 10 client.25487 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.e 2024-03-24T00:16:19.567 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:19.564+0000 7fc9f663c740 20 client.25487 populate_metadata read hostname 'smithi118' 2024-03-24T00:16:19.567 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:19.564+0000 7fc9f663c740 1 --2- 172.21.15.118:0/2847529476 >> [v2:172.21.15.189:6834/599473205,v1:172.21.15.189:6835/599473205] conn(0x55dff7191390 0x55dff71937b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:16:19.567 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:19.564+0000 7fc9f663c740 4 client.25487 mds_command: new command op to 24488 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-24T00:16:19.567 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:19.564+0000 7fc9f663c740 1 -- 172.21.15.118:0/2847529476 --> [v2:172.21.15.189:6834/599473205,v1:172.21.15.189:6835/599473205] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x55dff7000500 con 0x55dff7191390 2024-03-24T00:16:19.567 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:19.564+0000 7fc9ebe00640 1 --2- 172.21.15.118:0/2847529476 >> [v2:172.21.15.189:6834/599473205,v1:172.21.15.189:6835/599473205] conn(0x55dff7191390 0x55dff71937b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:16:19.567 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:19.564+0000 7fc9ebe00640 1 --2- 172.21.15.118:0/2847529476 >> [v2:172.21.15.189:6834/599473205,v1:172.21.15.189:6835/599473205] conn(0x55dff7191390 0x55dff71937b0 crc :-1 s=READY pgs=23 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=34e88c724b1a391 server_cookie=398dd69d37d3605d in_seq=0 out_seq=0 2024-03-24T00:16:19.567 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:19.564+0000 7fc9d2a00640 10 client.25487 ms_handle_connect on v2:172.21.15.189:6834/599473205 2024-03-24T00:16:19.567 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:19.565+0000 7fc9d2a00640 1 -- 172.21.15.118:0/2847529476 <== mds.4 v2:172.21.15.189:6834/599473205 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x55dff7000500 con 0x55dff7191390 2024-03-24T00:16:19.567 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:19.565+0000 7fc9d2a00640 10 client.25487 handle_command_reply: tid=0 2024-03-24T00:16:19.567 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:19.565+0000 7fc9f663c740 1 client.25487 shutdown 2024-03-24T00:16:19.568 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:19.565+0000 7fc9f663c740 1 -- 172.21.15.118:0/2847529476 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55dff7116810 msgr2=0x55dff7123d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:16:19.568 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:19.565+0000 7fc9f663c740 1 --2- 172.21.15.118:0/2847529476 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55dff7116810 0x55dff7123d30 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7fc9e0006010 tx=0x7fc9e00eeaa0 comp rx=0 tx=0).stop 2024-03-24T00:16:19.568 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:19.565+0000 7fc9f663c740 1 -- 172.21.15.118:0/2847529476 shutdown_connections 2024-03-24T00:16:19.568 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:19.565+0000 7fc9f663c740 1 --2- 172.21.15.118:0/2847529476 >> [v2:172.21.15.189:6834/599473205,v1:172.21.15.189:6835/599473205] conn(0x55dff7191390 0x55dff71937b0 crc :-1 s=READY pgs=23 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:16:19.569 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:19.565+0000 7fc9f663c740 1 --2- 172.21.15.118:0/2847529476 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55dff71183a0 0x55dff7127220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:16:19.569 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:19.565+0000 7fc9f663c740 1 --2- 172.21.15.118:0/2847529476 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55dff71179e0 0x55dff7124270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:16:19.569 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:19.565+0000 7fc9d2a00640 0 client.25487 ms_handle_reset on v2:172.21.15.189:6834/599473205 2024-03-24T00:16:19.569 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:19.565+0000 7fc9f663c740 1 --2- 172.21.15.118:0/2847529476 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55dff7116810 0x55dff7123d30 unknown :-1 s=CLOSED pgs=226 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:16:19.569 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:19.565+0000 7fc9f663c740 1 -- 172.21.15.118:0/2847529476 >> 172.21.15.118:0/2847529476 conn(0x55dff7111c00 msgr2=0x55dff71156f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:16:19.569 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:19.565+0000 7fc9f663c740 1 -- 172.21.15.118:0/2847529476 shutdown_connections 2024-03-24T00:16:19.569 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:19.565+0000 7fc9f663c740 1 -- 172.21.15.118:0/2847529476 wait complete. 2024-03-24T00:16:19.569 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:19.565+0000 7fc9f663c740 20 client.25487 trim_cache size 0 max 16384 2024-03-24T00:16:19.569 INFO:teuthology.orchestra.run.smithi118.stdout:[] 2024-03-24T00:16:19.570 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:19.566+0000 7fc9f3a00640 1 -- 172.21.15.118:0/1431881681 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7fc9d40a2890 msgr2=0x7fc9d40a4d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:16:19.570 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:19.566+0000 7fc9f3a00640 1 --2- 172.21.15.118:0/1431881681 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7fc9d40a2890 0x7fc9d40a4d50 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x55dff71178e0 tx=0x7fc9e40098a0 comp rx=0 tx=0).stop 2024-03-24T00:16:19.570 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:19.566+0000 7fc9f3a00640 1 -- 172.21.15.118:0/1431881681 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc9ec1038e0 msgr2=0x7fc9ec199c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:16:19.570 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:19.566+0000 7fc9f3a00640 1 --2- 172.21.15.118:0/1431881681 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc9ec1038e0 0x7fc9ec199c40 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7fc9e0000c00 tx=0x7fc9e002c6c0 comp rx=0 tx=0).stop 2024-03-24T00:16:19.570 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:19.566+0000 7fc9f3a00640 1 -- 172.21.15.118:0/1431881681 shutdown_connections 2024-03-24T00:16:19.570 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:19.566+0000 7fc9f3a00640 1 --2- 172.21.15.118:0/1431881681 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7fc9d40a2890 0x7fc9d40a4d50 unknown :-1 s=CLOSED pgs=275 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:16:19.571 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:19.566+0000 7fc9f3a00640 1 --2- 172.21.15.118:0/1431881681 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fc9ec105490 0x7fc9ec1a11b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:16:19.571 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:19.566+0000 7fc9f3a00640 1 --2- 172.21.15.118:0/1431881681 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fc9ec104ad0 0x7fc9ec19a180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:16:19.571 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:19.566+0000 7fc9f3a00640 1 --2- 172.21.15.118:0/1431881681 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc9ec1038e0 0x7fc9ec199c40 unknown :-1 s=CLOSED pgs=224 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:16:19.571 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:19.566+0000 7fc9f3a00640 1 -- 172.21.15.118:0/1431881681 >> 172.21.15.118:0/1431881681 conn(0x7fc9ec0fee50 msgr2=0x7fc9ec107dd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:16:19.571 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:19.566+0000 7fc9f3a00640 1 -- 172.21.15.118:0/1431881681 shutdown_connections 2024-03-24T00:16:19.571 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:19.566+0000 7fc9f3a00640 1 -- 172.21.15.118:0/1431881681 wait complete. 2024-03-24T00:16:19.583 INFO:teuthology.orchestra.run:Running command with timeout 120 2024-03-24T00:16:19.583 DEBUG:teuthology.orchestra.run.smithi118:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:3 damage ls 2024-03-24T00:16:19.788 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:19.785+0000 7fbc24400640 1 -- 172.21.15.118:0/1783482562 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fbc1c104ad0 msgr2=0x7fbc1c104f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:16:19.788 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:19.785+0000 7fbc24400640 1 --2- 172.21.15.118:0/1783482562 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fbc1c104ad0 0x7fbc1c104f50 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7fbc08009900 tx=0x7fbc0802bc40 comp rx=0 tx=0).stop 2024-03-24T00:16:19.788 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:19.786+0000 7fbc24400640 1 -- 172.21.15.118:0/1783482562 shutdown_connections 2024-03-24T00:16:19.788 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:19.786+0000 7fbc24400640 1 --2- 172.21.15.118:0/1783482562 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fbc1c105490 0x7fbc1c10bd20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:16:19.788 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:19.786+0000 7fbc24400640 1 --2- 172.21.15.118:0/1783482562 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fbc1c104ad0 0x7fbc1c104f50 unknown :-1 s=CLOSED pgs=212 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:16:19.788 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:19.786+0000 7fbc24400640 1 --2- 172.21.15.118:0/1783482562 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fbc1c1038e0 0x7fbc1c103d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:16:19.788 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:19.786+0000 7fbc24400640 1 -- 172.21.15.118:0/1783482562 >> 172.21.15.118:0/1783482562 conn(0x7fbc1c0fee50 msgr2=0x7fbc1c101290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:16:19.789 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:19.786+0000 7fbc24400640 1 -- 172.21.15.118:0/1783482562 shutdown_connections 2024-03-24T00:16:19.789 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:19.786+0000 7fbc24400640 1 -- 172.21.15.118:0/1783482562 wait complete. 2024-03-24T00:16:19.789 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:19.787+0000 7fbc24400640 1 Processor -- start 2024-03-24T00:16:19.789 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:19.787+0000 7fbc24400640 1 -- start start 2024-03-24T00:16:19.789 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:19.787+0000 7fbc24400640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fbc1c1038e0 0x7fbc1c199bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:16:19.789 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:19.787+0000 7fbc24400640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fbc1c104ad0 0x7fbc1c19a110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:16:19.789 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:19.787+0000 7fbc24400640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fbc1c105490 0x7fbc1c1a1180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:16:19.789 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:19.787+0000 7fbc24400640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbc1c10f5b0 con 0x7fbc1c105490 2024-03-24T00:16:19.789 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:19.787+0000 7fbc24400640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbc1c10f430 con 0x7fbc1c1038e0 2024-03-24T00:16:19.789 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:19.787+0000 7fbc24400640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbc1c10f730 con 0x7fbc1c104ad0 2024-03-24T00:16:19.789 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:19.787+0000 7fbc21a00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fbc1c1038e0 0x7fbc1c199bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:16:19.789 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:19.787+0000 7fbc21a00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fbc1c1038e0 0x7fbc1c199bd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.118:60234/0 (socket says 172.21.15.118:60234) 2024-03-24T00:16:19.789 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:19.787+0000 7fbc21a00640 1 -- 172.21.15.118:0/715498829 learned_addr learned my addr 172.21.15.118:0/715498829 (peer_addr_for_me v2:172.21.15.118:0/0) 2024-03-24T00:16:19.789 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:19.788+0000 7fbc22400640 1 --2- 172.21.15.118:0/715498829 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fbc1c105490 0x7fbc1c1a1180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:16:19.789 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:19.788+0000 7fbc21000640 1 --2- 172.21.15.118:0/715498829 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fbc1c104ad0 0x7fbc1c19a110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:16:19.789 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:19.788+0000 7fbc21a00640 1 -- 172.21.15.118:0/715498829 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fbc1c104ad0 msgr2=0x7fbc1c19a110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:16:19.790 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:19.788+0000 7fbc21a00640 1 --2- 172.21.15.118:0/715498829 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fbc1c104ad0 0x7fbc1c19a110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:16:19.790 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:19.788+0000 7fbc21a00640 1 -- 172.21.15.118:0/715498829 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fbc1c105490 msgr2=0x7fbc1c1a1180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:16:19.790 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:19.788+0000 7fbc21a00640 1 --2- 172.21.15.118:0/715498829 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fbc1c105490 0x7fbc1c1a1180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:16:19.790 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:19.788+0000 7fbc21a00640 1 -- 172.21.15.118:0/715498829 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbc1c1a17f0 con 0x7fbc1c1038e0 2024-03-24T00:16:19.790 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:19.788+0000 7fbc21a00640 1 --2- 172.21.15.118:0/715498829 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fbc1c1038e0 0x7fbc1c199bd0 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7fbc1000d770 tx=0x7fbc1000dc30 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:16:19.790 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:19.788+0000 7fbc1a000640 1 -- 172.21.15.118:0/715498829 <== mon.2 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbc10017070 con 0x7fbc1c1038e0 2024-03-24T00:16:19.790 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:19.788+0000 7fbc1a000640 1 -- 172.21.15.118:0/715498829 <== mon.2 v2:172.21.15.189:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fbc10009c30 con 0x7fbc1c1038e0 2024-03-24T00:16:19.790 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:19.788+0000 7fbc1a000640 1 -- 172.21.15.118:0/715498829 <== mon.2 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbc10003dd0 con 0x7fbc1c1038e0 2024-03-24T00:16:19.790 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:19.789+0000 7fbc24400640 1 -- 172.21.15.118:0/715498829 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbc1c1a1ae0 con 0x7fbc1c1038e0 2024-03-24T00:16:19.790 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:19.789+0000 7fbc24400640 1 -- 172.21.15.118:0/715498829 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbc1c1a2000 con 0x7fbc1c1038e0 2024-03-24T00:16:19.791 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:19.792+0000 7fbc27186740 1 Processor -- start 2024-03-24T00:16:19.791 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:19.792+0000 7fbc27186740 1 -- start start 2024-03-24T00:16:19.792 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:19.792+0000 7fbc27186740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x557c4e162810 0x557c4e162c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:16:19.792 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:19.793+0000 7fbc27186740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x557c4e1639e0 0x557c4e163e60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:16:19.792 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:19.793+0000 7fbc27186740 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x557c4e1643a0 0x557c4e16ac30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:16:19.792 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:19.793+0000 7fbc27186740 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x557c4e0c72c0 con 0x557c4e1643a0 2024-03-24T00:16:19.792 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:19.793+0000 7fbc27186740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x557c4e06fc90 con 0x557c4e162810 2024-03-24T00:16:19.793 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:19.793+0000 7fbc27186740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x557c4e0c3360 con 0x557c4e1639e0 2024-03-24T00:16:19.793 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:19.793+0000 7fbc22400640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x557c4e162810 0x557c4e162c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:16:19.793 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:19.793+0000 7fbc22400640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x557c4e162810 0x557c4e162c30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.118:60244/0 (socket says 172.21.15.118:60244) 2024-03-24T00:16:19.793 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:19.793+0000 7fbc22400640 1 -- 172.21.15.118:0/3480720110 learned_addr learned my addr 172.21.15.118:0/3480720110 (peer_addr_for_me v2:172.21.15.118:0/0) 2024-03-24T00:16:19.793 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:19.793+0000 7fbc21000640 1 --2- 172.21.15.118:0/3480720110 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x557c4e1643a0 0x557c4e16ac30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:16:19.794 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:19.793+0000 7fbc22400640 1 -- 172.21.15.118:0/3480720110 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x557c4e1639e0 msgr2=0x557c4e163e60 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-24T00:16:19.794 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:19.793+0000 7fbc22400640 1 --2- 172.21.15.118:0/3480720110 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x557c4e1639e0 0x557c4e163e60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:16:19.794 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:19.793+0000 7fbc22400640 1 -- 172.21.15.118:0/3480720110 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x557c4e1643a0 msgr2=0x557c4e16ac30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:16:19.795 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:19.793+0000 7fbc22400640 1 --2- 172.21.15.118:0/3480720110 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x557c4e1643a0 0x557c4e16ac30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:16:19.795 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:19.793+0000 7fbc22400640 1 -- 172.21.15.118:0/3480720110 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x557c4e094080 con 0x557c4e162810 2024-03-24T00:16:19.795 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:19.793+0000 7fbc1a000640 1 -- 172.21.15.118:0/715498829 <== mon.2 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 19) v1 ==== 137342+0+0 (secure 0 0 0) 0x7fbc10003fb0 con 0x7fbc1c1038e0 2024-03-24T00:16:19.795 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:19.793+0000 7fbc22400640 1 --2- 172.21.15.118:0/3480720110 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x557c4e162810 0x557c4e162c30 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7fbc14009fd0 tx=0x7fbc140118d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=2003a7559fafc35a server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:16:19.795 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:19.794+0000 7fbc0ea00640 1 -- 172.21.15.118:0/3480720110 <== mon.2 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbc14004d20 con 0x557c4e162810 2024-03-24T00:16:19.795 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:19.794+0000 7fbc0ea00640 1 -- 172.21.15.118:0/3480720110 <== mon.2 v2:172.21.15.189:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fbc14004050 con 0x557c4e162810 2024-03-24T00:16:19.795 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:19.794+0000 7fbc0ea00640 1 -- 172.21.15.118:0/3480720110 <== mon.2 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbc14005260 con 0x557c4e162810 2024-03-24T00:16:19.795 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:19.794+0000 7fbc27186740 1 -- 172.21.15.118:0/3480720110 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x557c4e162810 msgr2=0x557c4e162c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:16:19.795 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:19.794+0000 7fbc27186740 1 --2- 172.21.15.118:0/3480720110 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x557c4e162810 0x557c4e162c30 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7fbc14009fd0 tx=0x7fbc140118d0 comp rx=0 tx=0).stop 2024-03-24T00:16:19.795 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:19.795+0000 7fbc27186740 1 -- 172.21.15.118:0/3480720110 shutdown_connections 2024-03-24T00:16:19.795 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:19.795+0000 7fbc27186740 1 --2- 172.21.15.118:0/3480720110 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x557c4e1643a0 0x557c4e16ac30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:16:19.795 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:19.795+0000 7fbc27186740 1 --2- 172.21.15.118:0/3480720110 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x557c4e1639e0 0x557c4e163e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:16:19.795 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:19.795+0000 7fbc27186740 1 --2- 172.21.15.118:0/3480720110 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x557c4e162810 0x557c4e162c30 unknown :-1 s=CLOSED pgs=228 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:16:19.795 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:19.795+0000 7fbc27186740 1 -- 172.21.15.118:0/3480720110 >> 172.21.15.118:0/3480720110 conn(0x557c4e15dc00 msgr2=0x557c4e15e000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:16:19.795 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:19.795+0000 7fbc1a000640 1 --2- 172.21.15.118:0/715498829 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7fbc040a2910 0x7fbc040a4dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:16:19.796 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:19.795+0000 7fbc27186740 1 -- 172.21.15.118:0/3480720110 shutdown_connections 2024-03-24T00:16:19.796 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:19.795+0000 7fbc1a000640 1 -- 172.21.15.118:0/715498829 <== mon.2 v2:172.21.15.189:3300/0 5 ==== osd_map(90..90 src has 1..90) v4 ==== 8277+0+0 (secure 0 0 0) 0x7fbc100ce490 con 0x7fbc1c1038e0 2024-03-24T00:16:19.796 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:19.795+0000 7fbc27186740 1 -- 172.21.15.118:0/3480720110 wait complete. 2024-03-24T00:16:19.796 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:19.795+0000 7fbc22400640 1 --2- 172.21.15.118:0/715498829 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7fbc040a2910 0x7fbc040a4dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:16:19.798 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:19.795+0000 7fbc27186740 1 -- start start 2024-03-24T00:16:19.798 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:19.796+0000 7fbc27186740 1 Processor -- start 2024-03-24T00:16:19.798 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:19.796+0000 7fbc22400640 1 --2- 172.21.15.118:0/715498829 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7fbc040a2910 0x7fbc040a4dd0 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7fbc14011dd0 tx=0x7fbc14009c80 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:16:19.798 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:19.796+0000 7fbc27186740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x557c4e162810 0x557c4e1752a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:16:19.798 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:19.796+0000 7fbc27186740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x557c4e1639e0 0x557c4e1757e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:16:19.798 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:19.796+0000 7fbc27186740 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x557c4e1643a0 0x557c4e17c820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:16:19.799 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:19.796+0000 7fbc27186740 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x557c4e16b3b0 con 0x557c4e1643a0 2024-03-24T00:16:19.799 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:19.796+0000 7fbc27186740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x557c4e0830d0 con 0x557c4e1639e0 2024-03-24T00:16:19.799 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:19.796+0000 7fbc27186740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x557c4e16b530 con 0x557c4e162810 2024-03-24T00:16:19.799 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:19.796+0000 7fbc27186740 10 client.0 _ll_register_callbacks cb 0x557c4e087e80 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T00:16:19.799 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:19.796+0000 7fbc21a00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x557c4e162810 0x557c4e1752a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:16:19.799 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:19.796+0000 7fbc22400640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x557c4e1643a0 0x557c4e17c820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:16:19.799 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:19.796+0000 7fbc21a00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x557c4e162810 0x557c4e1752a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.118:45468/0 (socket says 172.21.15.118:45468) 2024-03-24T00:16:19.799 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:19.796+0000 7fbc21000640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x557c4e1639e0 0x557c4e1757e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:16:19.799 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:19.796+0000 7fbc21a00640 1 -- 172.21.15.118:0/1880385979 learned_addr learned my addr 172.21.15.118:0/1880385979 (peer_addr_for_me v2:172.21.15.118:0/0) 2024-03-24T00:16:19.799 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:19.797+0000 7fbc21a00640 1 -- 172.21.15.118:0/1880385979 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x557c4e1639e0 msgr2=0x557c4e1757e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:16:19.799 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:19.797+0000 7fbc21a00640 1 --2- 172.21.15.118:0/1880385979 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x557c4e1639e0 0x557c4e1757e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:16:19.799 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:19.797+0000 7fbc21a00640 1 -- 172.21.15.118:0/1880385979 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x557c4e1643a0 msgr2=0x557c4e17c820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:16:19.799 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:19.797+0000 7fbc21a00640 1 --2- 172.21.15.118:0/1880385979 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x557c4e1643a0 0x557c4e17c820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:16:19.799 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:19.797+0000 7fbc21a00640 1 -- 172.21.15.118:0/1880385979 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x557c4dddb3f0 con 0x557c4e162810 2024-03-24T00:16:19.799 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:19.797+0000 7fbc21000640 1 --2- 172.21.15.118:0/1880385979 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x557c4e1639e0 0x557c4e1757e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T00:16:19.800 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:19.797+0000 7fbc21a00640 1 --2- 172.21.15.118:0/1880385979 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x557c4e162810 0x557c4e1752a0 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7fbc10097720 tx=0x7fbc1009a6b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=50131d486d83e931 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:16:19.800 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:19.797+0000 7fbc03400640 10 client.0 ms_handle_connect on v2:172.21.15.193:3300/0 2024-03-24T00:16:19.800 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:19.797+0000 7fbc03400640 1 -- 172.21.15.118:0/1880385979 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbc10020040 con 0x557c4e162810 2024-03-24T00:16:19.800 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:19.797+0000 7fbc03400640 1 -- 172.21.15.118:0/1880385979 <== mon.1 v2:172.21.15.193:3300/0 2 ==== osd_map(90..90 src has 1..90) v4 ==== 8277+0+0 (secure 0 0 0) 0x7fbc100973c0 con 0x557c4e162810 2024-03-24T00:16:19.800 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:19.797+0000 7fbc27186740 1 -- 172.21.15.118:0/1880385979 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fbc14004050 con 0x557c4e162810 2024-03-24T00:16:19.800 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:19.797+0000 7fbc03400640 1 -- 172.21.15.118:0/1880385979 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) v2 ==== 24+0+0 (secure 0 0 0) 0x7fbc1009edf0 con 0x557c4e162810 2024-03-24T00:16:19.800 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:19.797+0000 7fbc27186740 10 client.25501 fetch_fsmap learned FSMap version 28 2024-03-24T00:16:19.800 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:19.797+0000 7fbc27186740 1 -- 172.21.15.118:0/1880385979 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) v3 -- 0x557c4dea57d0 con 0x557c4e162810 2024-03-24T00:16:19.800 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:19.798+0000 7fbc03400640 1 -- 172.21.15.118:0/1880385979 <== mon.1 v2:172.21.15.193:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fbc14004050 con 0x557c4e162810 2024-03-24T00:16:19.800 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:19.798+0000 7fbc03400640 1 -- 172.21.15.118:0/1880385979 <== mon.1 v2:172.21.15.193:3300/0 5 ==== fsmap(e 28) v1 ==== 7934+0+0 (secure 0 0 0) 0x7fbc1009a1a0 con 0x557c4e162810 2024-03-24T00:16:19.800 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:19.798+0000 7fbc27186740 10 client.25501 fetch_fsmap finished waiting for FSMap version 28 2024-03-24T00:16:19.800 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:19.798+0000 7fbc27186740 10 client.25501 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.f 2024-03-24T00:16:19.800 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:19.798+0000 7fbc27186740 20 client.25501 populate_metadata read hostname 'smithi118' 2024-03-24T00:16:19.800 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:19.798+0000 7fbc27186740 1 --2- 172.21.15.118:0/1880385979 >> [v2:172.21.15.193:6834/2055310541,v1:172.21.15.193:6835/2055310541] conn(0x557c4e170750 0x557c4e172b60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:16:19.800 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:19.798+0000 7fbc27186740 4 client.25501 mds_command: new command op to 24499 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-24T00:16:19.800 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:19.798+0000 7fbc27186740 1 -- 172.21.15.118:0/1880385979 --> [v2:172.21.15.193:6834/2055310541,v1:172.21.15.193:6835/2055310541] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x557c4e0d4800 con 0x557c4e170750 2024-03-24T00:16:19.801 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:19.799+0000 7fbc21000640 1 --2- 172.21.15.118:0/1880385979 >> [v2:172.21.15.193:6834/2055310541,v1:172.21.15.193:6835/2055310541] conn(0x557c4e170750 0x557c4e172b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:16:19.801 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:19.799+0000 7fbc21000640 1 --2- 172.21.15.118:0/1880385979 >> [v2:172.21.15.193:6834/2055310541,v1:172.21.15.193:6835/2055310541] conn(0x557c4e170750 0x557c4e172b60 crc :-1 s=READY pgs=24 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=b9e791493c4b67a1 server_cookie=2c6cc0ee7e2a814f in_seq=0 out_seq=0 2024-03-24T00:16:19.802 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:19.799+0000 7fbc03400640 10 client.25501 ms_handle_connect on v2:172.21.15.193:6834/2055310541 2024-03-24T00:16:19.802 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:19.800+0000 7fbc03400640 1 -- 172.21.15.118:0/1880385979 <== mds.3 v2:172.21.15.193:6834/2055310541 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x557c4e0d4800 con 0x557c4e170750 2024-03-24T00:16:19.802 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:19.800+0000 7fbc03400640 10 client.25501 handle_command_reply: tid=0 2024-03-24T00:16:19.802 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:19.800+0000 7fbc27186740 1 client.25501 shutdown 2024-03-24T00:16:19.802 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:19.801+0000 7fbc27186740 1 -- 172.21.15.118:0/1880385979 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x557c4e162810 msgr2=0x557c4e1752a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:16:19.802 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:19.801+0000 7fbc27186740 1 --2- 172.21.15.118:0/1880385979 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x557c4e162810 0x557c4e1752a0 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7fbc10097720 tx=0x7fbc1009a6b0 comp rx=0 tx=0).stop 2024-03-24T00:16:19.802 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:19.801+0000 7fbc27186740 1 -- 172.21.15.118:0/1880385979 shutdown_connections 2024-03-24T00:16:19.802 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:19.801+0000 7fbc27186740 1 --2- 172.21.15.118:0/1880385979 >> [v2:172.21.15.193:6834/2055310541,v1:172.21.15.193:6835/2055310541] conn(0x557c4e170750 0x557c4e172b60 crc :-1 s=READY pgs=24 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:16:19.802 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:19.801+0000 7fbc27186740 1 --2- 172.21.15.118:0/1880385979 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x557c4e1643a0 0x557c4e17c820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:16:19.802 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:19.801+0000 7fbc27186740 1 --2- 172.21.15.118:0/1880385979 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x557c4e1639e0 0x557c4e1757e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:16:19.802 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:19.801+0000 7fbc27186740 1 --2- 172.21.15.118:0/1880385979 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x557c4e162810 0x557c4e1752a0 unknown :-1 s=CLOSED pgs=213 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:16:19.802 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:19.801+0000 7fbc03400640 0 client.25501 ms_handle_reset on v2:172.21.15.193:6834/2055310541 2024-03-24T00:16:19.802 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:19.801+0000 7fbc27186740 1 -- 172.21.15.118:0/1880385979 >> 172.21.15.118:0/1880385979 conn(0x557c4e15dc00 msgr2=0x557c4e15ceb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:16:19.802 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:19.801+0000 7fbc27186740 1 -- 172.21.15.118:0/1880385979 shutdown_connections 2024-03-24T00:16:19.803 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:19.801+0000 7fbc27186740 1 -- 172.21.15.118:0/1880385979 wait complete. 2024-03-24T00:16:19.803 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:19.801+0000 7fbc27186740 20 client.25501 trim_cache size 0 max 16384 2024-03-24T00:16:19.815 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:19.814+0000 7fbc27186740 1 Processor -- start 2024-03-24T00:16:19.815 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:19.815+0000 7fbc27186740 1 -- start start 2024-03-24T00:16:19.816 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:19.815+0000 7fbc27186740 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x557c4e15dc00 0x557c4e166c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:16:19.816 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:19.815+0000 7fbc27186740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x557c4e162810 0x557c4e17c7e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:16:19.816 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:19.815+0000 7fbc27186740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x557c4e1639e0 0x557c4e167150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:16:19.816 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:19.815+0000 7fbc27186740 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbc1c1a6860 con 0x557c4e15dc00 2024-03-24T00:16:19.816 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:19.815+0000 7fbc27186740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbc1c1a81e0 con 0x557c4e1639e0 2024-03-24T00:16:19.816 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:19.815+0000 7fbc27186740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbc1c1a6ec0 con 0x557c4e162810 2024-03-24T00:16:19.816 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:19.815+0000 7fbc21000640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x557c4e162810 0x557c4e17c7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:16:19.816 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:19.815+0000 7fbc21000640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x557c4e162810 0x557c4e17c7e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.118:45470/0 (socket says 172.21.15.118:45470) 2024-03-24T00:16:19.816 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:19.815+0000 7fbc22400640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x557c4e1639e0 0x557c4e167150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:16:19.816 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:19.815+0000 7fbc21000640 1 -- 172.21.15.118:0/1530876858 learned_addr learned my addr 172.21.15.118:0/1530876858 (peer_addr_for_me v2:172.21.15.118:0/0) 2024-03-24T00:16:19.816 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:19.815+0000 7fbc21a00640 1 --2- 172.21.15.118:0/1530876858 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x557c4e15dc00 0x557c4e166c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:16:19.816 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:19.815+0000 7fbc21000640 1 -- 172.21.15.118:0/1530876858 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x557c4e1639e0 msgr2=0x557c4e167150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:16:19.816 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:19.815+0000 7fbc21000640 1 --2- 172.21.15.118:0/1530876858 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x557c4e1639e0 0x557c4e167150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:16:19.816 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:19.815+0000 7fbc21000640 1 -- 172.21.15.118:0/1530876858 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x557c4e15dc00 msgr2=0x557c4e166c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:16:19.816 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:19.815+0000 7fbc21000640 1 --2- 172.21.15.118:0/1530876858 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x557c4e15dc00 0x557c4e166c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:16:19.817 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:19.815+0000 7fbc21000640 1 -- 172.21.15.118:0/1530876858 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x557c4e15d970 con 0x557c4e162810 2024-03-24T00:16:19.817 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:19.816+0000 7fbc21000640 1 --2- 172.21.15.118:0/1530876858 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x557c4e162810 0x557c4e17c7e0 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7fbc0802e010 tx=0x7fbc0802e040 comp rx=0 tx=0).ready entity=mon.1 client_cookie=92d56a53a71bd8cb server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:16:19.817 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:19.816+0000 7fbc0e000640 1 -- 172.21.15.118:0/1530876858 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbc0804b220 con 0x557c4e162810 2024-03-24T00:16:19.817 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:19.816+0000 7fbc0e000640 1 -- 172.21.15.118:0/1530876858 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fbc0804f070 con 0x557c4e162810 2024-03-24T00:16:19.817 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:19.816+0000 7fbc0e000640 1 -- 172.21.15.118:0/1530876858 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbc0804b6f0 con 0x557c4e162810 2024-03-24T00:16:19.821 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:19.816+0000 7fbc27186740 1 -- 172.21.15.118:0/1530876858 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x557c4e162810 msgr2=0x557c4e17c7e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:16:19.821 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:19.816+0000 7fbc27186740 1 --2- 172.21.15.118:0/1530876858 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x557c4e162810 0x557c4e17c7e0 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7fbc0802e010 tx=0x7fbc0802e040 comp rx=0 tx=0).stop 2024-03-24T00:16:19.821 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:19.817+0000 7fbc27186740 1 -- 172.21.15.118:0/1530876858 shutdown_connections 2024-03-24T00:16:19.821 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:19.817+0000 7fbc27186740 1 --2- 172.21.15.118:0/1530876858 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x557c4e1639e0 0x557c4e167150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:16:19.821 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:19.817+0000 7fbc27186740 1 --2- 172.21.15.118:0/1530876858 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x557c4e162810 0x557c4e17c7e0 unknown :-1 s=CLOSED pgs=214 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:16:19.821 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:19.817+0000 7fbc27186740 1 --2- 172.21.15.118:0/1530876858 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x557c4e15dc00 0x557c4e166c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:16:19.821 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:19.817+0000 7fbc27186740 1 -- 172.21.15.118:0/1530876858 >> 172.21.15.118:0/1530876858 conn(0x557c4e16bca0 msgr2=0x557c4e1616f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:16:19.821 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:19.817+0000 7fbc27186740 1 -- 172.21.15.118:0/1530876858 shutdown_connections 2024-03-24T00:16:19.821 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:19.817+0000 7fbc27186740 1 -- 172.21.15.118:0/1530876858 wait complete. 2024-03-24T00:16:19.821 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:19.817+0000 7fbc27186740 1 -- start start 2024-03-24T00:16:19.821 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:19.817+0000 7fbc27186740 1 Processor -- start 2024-03-24T00:16:19.821 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:19.818+0000 7fbc27186740 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x557c4e15dc00 0x557c4e17c7a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:16:19.821 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:19.818+0000 7fbc27186740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x557c4e162810 0x557c4e17ae00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:16:19.821 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:19.818+0000 7fbc27186740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x557c4e1639e0 0x557c4e17b430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:16:19.822 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:19.818+0000 7fbc27186740 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbc1c1a7b80 con 0x557c4e15dc00 2024-03-24T00:16:19.822 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:19.818+0000 7fbc27186740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbc1c1a7520 con 0x557c4e162810 2024-03-24T00:16:19.822 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:19.818+0000 7fbc27186740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbc1c1a4a20 con 0x557c4e1639e0 2024-03-24T00:16:19.822 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:19.818+0000 7fbc27186740 10 client.0 _ll_register_callbacks cb 0x7fbc10003770 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T00:16:19.822 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:19.818+0000 7fbc21000640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x557c4e162810 0x557c4e17ae00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:16:19.822 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:19.818+0000 7fbc22400640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x557c4e1639e0 0x557c4e17b430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:16:19.822 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:19.818+0000 7fbc21a00640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x557c4e15dc00 0x557c4e17c7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:16:19.822 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:19.818+0000 7fbc21000640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x557c4e162810 0x557c4e17ae00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.118:60264/0 (socket says 172.21.15.118:60264) 2024-03-24T00:16:19.822 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:19.818+0000 7fbc22400640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x557c4e1639e0 0x557c4e17b430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.118:45482/0 (socket says 172.21.15.118:45482) 2024-03-24T00:16:19.822 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:19.818+0000 7fbc21000640 1 -- 172.21.15.118:0/3834981058 learned_addr learned my addr 172.21.15.118:0/3834981058 (peer_addr_for_me v2:172.21.15.118:0/0) 2024-03-24T00:16:19.822 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:19.818+0000 7fbc21000640 1 -- 172.21.15.118:0/3834981058 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x557c4e1639e0 msgr2=0x557c4e17b430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:16:19.822 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:19.818+0000 7fbc21000640 1 --2- 172.21.15.118:0/3834981058 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x557c4e1639e0 0x557c4e17b430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:16:19.822 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:19.818+0000 7fbc21000640 1 -- 172.21.15.118:0/3834981058 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x557c4e15dc00 msgr2=0x557c4e17c7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:16:19.822 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:19.818+0000 7fbc21000640 1 --2- 172.21.15.118:0/3834981058 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x557c4e15dc00 0x557c4e17c7a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:16:19.823 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:19.818+0000 7fbc21000640 1 -- 172.21.15.118:0/3834981058 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x557c4e1e4950 con 0x557c4e162810 2024-03-24T00:16:19.823 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:19.818+0000 7fbc22400640 1 --2- 172.21.15.118:0/3834981058 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x557c4e1639e0 0x557c4e17b430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T00:16:19.823 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:19.819+0000 7fbc21000640 1 --2- 172.21.15.118:0/3834981058 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x557c4e162810 0x557c4e17ae00 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7fbc08000c00 tx=0x7fbc08005c90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=edfdb2049d71890e server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:16:19.823 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:19.819+0000 7fbc03400640 10 client.0 ms_handle_connect on v2:172.21.15.189:3300/0 2024-03-24T00:16:19.823 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:19.819+0000 7fbc03400640 1 -- 172.21.15.118:0/3834981058 <== mon.2 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fbc0804b220 con 0x557c4e162810 2024-03-24T00:16:19.823 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:19.819+0000 7fbc27186740 1 -- 172.21.15.118:0/3834981058 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fbc0804f070 con 0x557c4e162810 2024-03-24T00:16:19.823 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:19.819+0000 7fbc03400640 1 -- 172.21.15.118:0/3834981058 <== mon.2 v2:172.21.15.189:3300/0 2 ==== osd_map(90..90 src has 1..90) v4 ==== 8277+0+0 (secure 0 0 0) 0x7fbc0804b6f0 con 0x557c4e162810 2024-03-24T00:16:19.823 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:19.819+0000 7fbc03400640 1 -- 172.21.15.118:0/3834981058 <== mon.2 v2:172.21.15.189:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) v2 ==== 24+0+0 (secure 0 0 0) 0x7fbc0804b910 con 0x557c4e162810 2024-03-24T00:16:19.823 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:19.819+0000 7fbc27186740 10 client.25505 fetch_fsmap learned FSMap version 28 2024-03-24T00:16:19.823 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:19.819+0000 7fbc27186740 1 -- 172.21.15.118:0/3834981058 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) v3 -- 0x557c4e1e5ab0 con 0x557c4e162810 2024-03-24T00:16:19.823 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:19.819+0000 7fbc03400640 1 -- 172.21.15.118:0/3834981058 <== mon.2 v2:172.21.15.189:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fbc0804f070 con 0x557c4e162810 2024-03-24T00:16:19.823 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:19.820+0000 7fbc03400640 1 -- 172.21.15.118:0/3834981058 <== mon.2 v2:172.21.15.189:3300/0 5 ==== fsmap(e 28) v1 ==== 7934+0+0 (secure 0 0 0) 0x7fbc080054f0 con 0x557c4e162810 2024-03-24T00:16:19.823 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:19.820+0000 7fbc27186740 10 client.25505 fetch_fsmap finished waiting for FSMap version 28 2024-03-24T00:16:19.823 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:19.820+0000 7fbc27186740 10 client.25505 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.f 2024-03-24T00:16:19.823 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:19.820+0000 7fbc27186740 20 client.25505 populate_metadata read hostname 'smithi118' 2024-03-24T00:16:19.823 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:19.820+0000 7fbc27186740 1 --2- 172.21.15.118:0/3834981058 >> [v2:172.21.15.193:6834/2055310541,v1:172.21.15.193:6835/2055310541] conn(0x557c4e1643a0 0x557c4e1e83a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:16:19.824 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:19.820+0000 7fbc27186740 4 client.25505 mds_command: new command op to 24499 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-24T00:16:19.824 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:19.820+0000 7fbc27186740 1 -- 172.21.15.118:0/3834981058 --> [v2:172.21.15.193:6834/2055310541,v1:172.21.15.193:6835/2055310541] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x557c4e0ceb50 con 0x557c4e1643a0 2024-03-24T00:16:19.824 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:19.820+0000 7fbc21a00640 1 --2- 172.21.15.118:0/3834981058 >> [v2:172.21.15.193:6834/2055310541,v1:172.21.15.193:6835/2055310541] conn(0x557c4e1643a0 0x557c4e1e83a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:16:19.824 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:19.820+0000 7fbc03400640 10 client.25505 ms_handle_connect on v2:172.21.15.193:6834/2055310541 2024-03-24T00:16:19.824 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:19.820+0000 7fbc21a00640 1 --2- 172.21.15.118:0/3834981058 >> [v2:172.21.15.193:6834/2055310541,v1:172.21.15.193:6835/2055310541] conn(0x557c4e1643a0 0x557c4e1e83a0 crc :-1 s=READY pgs=25 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=427dd1861a976d89 server_cookie=b0a7b5eb6294daf8 in_seq=0 out_seq=0 2024-03-24T00:16:19.824 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:19.821+0000 7fbc03400640 1 -- 172.21.15.118:0/3834981058 <== mds.3 v2:172.21.15.193:6834/2055310541 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x557c4e0ceb50 con 0x557c4e1643a0 2024-03-24T00:16:19.824 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:19.821+0000 7fbc03400640 10 client.25505 handle_command_reply: tid=0 2024-03-24T00:16:19.824 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:19.821+0000 7fbc27186740 1 client.25505 shutdown 2024-03-24T00:16:19.824 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:19.821+0000 7fbc27186740 1 -- 172.21.15.118:0/3834981058 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x557c4e162810 msgr2=0x557c4e17ae00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:16:19.824 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:19.821+0000 7fbc27186740 1 --2- 172.21.15.118:0/3834981058 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x557c4e162810 0x557c4e17ae00 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7fbc08000c00 tx=0x7fbc08005c90 comp rx=0 tx=0).stop 2024-03-24T00:16:19.824 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:19.821+0000 7fbc27186740 1 -- 172.21.15.118:0/3834981058 shutdown_connections 2024-03-24T00:16:19.824 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:19.821+0000 7fbc27186740 1 --2- 172.21.15.118:0/3834981058 >> [v2:172.21.15.193:6834/2055310541,v1:172.21.15.193:6835/2055310541] conn(0x557c4e1643a0 0x557c4e1e83a0 crc :-1 s=READY pgs=25 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:16:19.824 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:19.821+0000 7fbc27186740 1 --2- 172.21.15.118:0/3834981058 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x557c4e1639e0 0x557c4e17b430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:16:19.824 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:19.821+0000 7fbc03400640 0 client.25505 ms_handle_reset on v2:172.21.15.193:6834/2055310541 2024-03-24T00:16:19.824 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:19.821+0000 7fbc27186740 1 --2- 172.21.15.118:0/3834981058 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x557c4e162810 0x557c4e17ae00 unknown :-1 s=CLOSED pgs=229 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:16:19.825 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:19.821+0000 7fbc27186740 1 --2- 172.21.15.118:0/3834981058 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x557c4e15dc00 0x557c4e17c7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:16:19.825 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:19.821+0000 7fbc27186740 1 -- 172.21.15.118:0/3834981058 >> 172.21.15.118:0/3834981058 conn(0x557c4e16bca0 msgr2=0x557c4e15c620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:16:19.825 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:19.821+0000 7fbc27186740 1 -- 172.21.15.118:0/3834981058 shutdown_connections 2024-03-24T00:16:19.825 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:19.821+0000 7fbc27186740 1 -- 172.21.15.118:0/3834981058 wait complete. 2024-03-24T00:16:19.825 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:19.821+0000 7fbc27186740 20 client.25505 trim_cache size 0 max 16384 2024-03-24T00:16:19.825 INFO:teuthology.orchestra.run.smithi118.stdout:[] 2024-03-24T00:16:19.826 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:19.822+0000 7fbc24400640 1 -- 172.21.15.118:0/715498829 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7fbc040a2910 msgr2=0x7fbc040a4dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:16:19.826 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:19.822+0000 7fbc24400640 1 --2- 172.21.15.118:0/715498829 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7fbc040a2910 0x7fbc040a4dd0 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7fbc14011dd0 tx=0x7fbc14009c80 comp rx=0 tx=0).stop 2024-03-24T00:16:19.826 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:19.822+0000 7fbc24400640 1 -- 172.21.15.118:0/715498829 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fbc1c1038e0 msgr2=0x7fbc1c199bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:16:19.826 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:19.822+0000 7fbc24400640 1 --2- 172.21.15.118:0/715498829 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fbc1c1038e0 0x7fbc1c199bd0 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7fbc1000d770 tx=0x7fbc1000dc30 comp rx=0 tx=0).stop 2024-03-24T00:16:19.826 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:19.822+0000 7fbc24400640 1 -- 172.21.15.118:0/715498829 shutdown_connections 2024-03-24T00:16:19.826 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:19.822+0000 7fbc24400640 1 --2- 172.21.15.118:0/715498829 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7fbc040a2910 0x7fbc040a4dd0 unknown :-1 s=CLOSED pgs=276 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:16:19.826 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:19.822+0000 7fbc24400640 1 --2- 172.21.15.118:0/715498829 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fbc1c105490 0x7fbc1c1a1180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:16:19.826 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:19.822+0000 7fbc24400640 1 --2- 172.21.15.118:0/715498829 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fbc1c104ad0 0x7fbc1c19a110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:16:19.826 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:19.822+0000 7fbc24400640 1 --2- 172.21.15.118:0/715498829 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fbc1c1038e0 0x7fbc1c199bd0 unknown :-1 s=CLOSED pgs=227 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:16:19.826 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:19.822+0000 7fbc24400640 1 -- 172.21.15.118:0/715498829 >> 172.21.15.118:0/715498829 conn(0x7fbc1c0fee50 msgr2=0x7fbc1c107d70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:16:19.826 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:19.822+0000 7fbc24400640 1 -- 172.21.15.118:0/715498829 shutdown_connections 2024-03-24T00:16:19.827 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:19.822+0000 7fbc24400640 1 -- 172.21.15.118:0/715498829 wait complete. 2024-03-24T00:16:19.835 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:16:19 smithi118 ceph-mon[31535]: from='client.? 172.21.15.118:0/3331377977' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-24T00:16:19.838 INFO:teuthology.orchestra.run:Running command with timeout 120 2024-03-24T00:16:19.838 DEBUG:teuthology.orchestra.run.smithi118:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:2 damage ls 2024-03-24T00:16:20.049 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:20.049+0000 7f7333600640 1 -- 172.21.15.118:0/2756960588 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f732c102130 msgr2=0x7f732c104520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:16:20.049 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:20.049+0000 7f7333600640 1 --2- 172.21.15.118:0/2756960588 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f732c102130 0x7f732c104520 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7f7314009900 tx=0x7f731402bc50 comp rx=0 tx=0).stop 2024-03-24T00:16:20.049 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:20.049+0000 7f7333600640 1 -- 172.21.15.118:0/2756960588 shutdown_connections 2024-03-24T00:16:20.049 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:20.049+0000 7f7333600640 1 --2- 172.21.15.118:0/2756960588 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f732c107390 0x7f732c1097e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:16:20.049 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:20.049+0000 7f7333600640 1 --2- 172.21.15.118:0/2756960588 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f732c104a60 0x7f732c106e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:16:20.049 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:20.049+0000 7f7333600640 1 --2- 172.21.15.118:0/2756960588 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f732c102130 0x7f732c104520 unknown :-1 s=CLOSED pgs=230 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:16:20.049 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:20.049+0000 7f7333600640 1 -- 172.21.15.118:0/2756960588 >> 172.21.15.118:0/2756960588 conn(0x7f732c0fbb40 msgr2=0x7f732c0fdf60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:16:20.049 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:20.049+0000 7f7333600640 1 -- 172.21.15.118:0/2756960588 shutdown_connections 2024-03-24T00:16:20.049 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:20.049+0000 7f7333600640 1 -- 172.21.15.118:0/2756960588 wait complete. 2024-03-24T00:16:20.050 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:20.051+0000 7f7333600640 1 Processor -- start 2024-03-24T00:16:20.050 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:20.051+0000 7f7333600640 1 -- start start 2024-03-24T00:16:20.052 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:20.051+0000 7f7333600640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f732c102130 0x7f732c197aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:16:20.052 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:20.051+0000 7f7333600640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f732c104a60 0x7f732c197fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:16:20.052 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:20.051+0000 7f7333600640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f732c107390 0x7f732c19f050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:16:20.052 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:20.051+0000 7f7333600640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f732c10d3b0 con 0x7f732c107390 2024-03-24T00:16:20.052 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:20.051+0000 7f7333600640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f732c10d230 con 0x7f732c104a60 2024-03-24T00:16:20.052 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:20.051+0000 7f7333600640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f732c10d530 con 0x7f732c102130 2024-03-24T00:16:20.052 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:20.051+0000 7f732be00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f732c104a60 0x7f732c197fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:16:20.052 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:20.051+0000 7f7330c00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f732c102130 0x7f732c197aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:16:20.053 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:20.051+0000 7f732be00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f732c104a60 0x7f732c197fe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.118:60282/0 (socket says 172.21.15.118:60282) 2024-03-24T00:16:20.053 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:20.051+0000 7f732be00640 1 -- 172.21.15.118:0/282353499 learned_addr learned my addr 172.21.15.118:0/282353499 (peer_addr_for_me v2:172.21.15.118:0/0) 2024-03-24T00:16:20.053 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:20.052+0000 7f7331600640 1 --2- 172.21.15.118:0/282353499 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f732c107390 0x7f732c19f050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:16:20.053 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:20.052+0000 7f7330c00640 1 -- 172.21.15.118:0/282353499 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f732c104a60 msgr2=0x7f732c197fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:16:20.053 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:20.052+0000 7f7330c00640 1 --2- 172.21.15.118:0/282353499 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f732c104a60 0x7f732c197fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:16:20.053 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:20.052+0000 7f7330c00640 1 -- 172.21.15.118:0/282353499 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f732c107390 msgr2=0x7f732c19f050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:16:20.053 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:20.052+0000 7f7330c00640 1 --2- 172.21.15.118:0/282353499 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f732c107390 0x7f732c19f050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:16:20.053 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:20.052+0000 7f7330c00640 1 -- 172.21.15.118:0/282353499 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f732c19f6c0 con 0x7f732c102130 2024-03-24T00:16:20.053 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:20.052+0000 7f7331600640 1 --2- 172.21.15.118:0/282353499 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f732c107390 0x7f732c19f050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T00:16:20.053 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:20.052+0000 7f732be00640 1 --2- 172.21.15.118:0/282353499 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f732c104a60 0x7f732c197fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T00:16:20.053 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:20.052+0000 7f7330c00640 1 --2- 172.21.15.118:0/282353499 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f732c102130 0x7f732c197aa0 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7f7314036040 tx=0x7f7314005dd0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:16:20.053 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:20.052+0000 7f7329600640 1 -- 172.21.15.118:0/282353499 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7314041070 con 0x7f732c102130 2024-03-24T00:16:20.053 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:20.052+0000 7f7329600640 1 -- 172.21.15.118:0/282353499 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f73140054e0 con 0x7f732c102130 2024-03-24T00:16:20.053 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:20.052+0000 7f7333600640 1 -- 172.21.15.118:0/282353499 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f732c19f950 con 0x7f732c102130 2024-03-24T00:16:20.053 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:20.053+0000 7f7329600640 1 -- 172.21.15.118:0/282353499 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f73140346a0 con 0x7f732c102130 2024-03-24T00:16:20.054 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:20.053+0000 7f7333600640 1 -- 172.21.15.118:0/282353499 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f732c19fec0 con 0x7f732c102130 2024-03-24T00:16:20.055 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:20.055+0000 7f7329600640 1 -- 172.21.15.118:0/282353499 <== mon.1 v2:172.21.15.193:3300/0 4 ==== mgrmap(e 19) v1 ==== 137342+0+0 (secure 0 0 0) 0x7f7314044070 con 0x7f732c102130 2024-03-24T00:16:20.056 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:20.057+0000 7f7329600640 1 --2- 172.21.15.118:0/282353499 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7f73100a2910 0x7f73100a4dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:16:20.056 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:20.057+0000 7f7329600640 1 -- 172.21.15.118:0/282353499 <== mon.1 v2:172.21.15.193:3300/0 5 ==== osd_map(90..90 src has 1..90) v4 ==== 8277+0+0 (secure 0 0 0) 0x7f73140f06a0 con 0x7f732c102130 2024-03-24T00:16:20.058 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:20.057+0000 7f732be00640 1 --2- 172.21.15.118:0/282353499 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7f73100a2910 0x7f73100a4dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:16:20.058 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:20.057+0000 7f73363ff740 1 Processor -- start 2024-03-24T00:16:20.058 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:20.057+0000 7f73363ff740 1 -- start start 2024-03-24T00:16:20.058 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:20.058+0000 7f73363ff740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55f9ec039810 0x55f9ec039c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:16:20.058 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:20.058+0000 7f73363ff740 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55f9ec03a9e0 0x55f9ec03ae60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:16:20.058 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:20.058+0000 7f73363ff740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55f9ec03b3a0 0x55f9ec041c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:16:20.058 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:20.058+0000 7f73363ff740 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f9ebf9e2c0 con 0x55f9ec03a9e0 2024-03-24T00:16:20.058 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:20.058+0000 7f73363ff740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f9ebf46c90 con 0x55f9ec03b3a0 2024-03-24T00:16:20.059 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:20.058+0000 7f73363ff740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f9ebf9a360 con 0x55f9ec039810 2024-03-24T00:16:20.059 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:20.058+0000 7f732be00640 1 --2- 172.21.15.118:0/282353499 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7f73100a2910 0x7f73100a4dd0 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7f732c198fb0 tx=0x7f731c0096c0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:16:20.059 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:20.058+0000 7f7330c00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55f9ec039810 0x55f9ec039c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:16:20.059 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:20.058+0000 7f7331600640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55f9ec03b3a0 0x55f9ec041c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:16:20.059 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:20.058+0000 7f7330c00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55f9ec039810 0x55f9ec039c30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.118:45520/0 (socket says 172.21.15.118:45520) 2024-03-24T00:16:20.059 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:20.058+0000 7f7330c00640 1 -- 172.21.15.118:0/2204100167 learned_addr learned my addr 172.21.15.118:0/2204100167 (peer_addr_for_me v2:172.21.15.118:0/0) 2024-03-24T00:16:20.059 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:20.058+0000 7f732be00640 1 --2- 172.21.15.118:0/2204100167 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55f9ec03a9e0 0x55f9ec03ae60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:16:20.059 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:20.058+0000 7f7330c00640 1 -- 172.21.15.118:0/2204100167 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55f9ec03b3a0 msgr2=0x55f9ec041c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:16:20.059 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:20.058+0000 7f7330c00640 1 --2- 172.21.15.118:0/2204100167 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55f9ec03b3a0 0x55f9ec041c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:16:20.059 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:20.058+0000 7f7330c00640 1 -- 172.21.15.118:0/2204100167 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55f9ec03a9e0 msgr2=0x55f9ec03ae60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:16:20.059 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:20.058+0000 7f7330c00640 1 --2- 172.21.15.118:0/2204100167 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55f9ec03a9e0 0x55f9ec03ae60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:16:20.059 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:20.058+0000 7f7330c00640 1 -- 172.21.15.118:0/2204100167 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55f9ebf6b080 con 0x55f9ec039810 2024-03-24T00:16:20.059 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:20.058+0000 7f7331600640 1 --2- 172.21.15.118:0/2204100167 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55f9ec03b3a0 0x55f9ec041c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T00:16:20.059 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:20.059+0000 7f7330c00640 1 --2- 172.21.15.118:0/2204100167 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55f9ec039810 0x55f9ec039c30 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7f73140b9930 tx=0x7f73140bcdf0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=7ce5cd76b29311c2 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:16:20.063 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:20.059+0000 7f7321c00640 1 -- 172.21.15.118:0/2204100167 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f731404b070 con 0x55f9ec039810 2024-03-24T00:16:20.063 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:20.059+0000 7f7321c00640 1 -- 172.21.15.118:0/2204100167 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f73140bd660 con 0x55f9ec039810 2024-03-24T00:16:20.063 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:20.059+0000 7f7321c00640 1 -- 172.21.15.118:0/2204100167 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f73140ee210 con 0x55f9ec039810 2024-03-24T00:16:20.063 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:20.059+0000 7f73363ff740 1 -- 172.21.15.118:0/2204100167 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55f9ec039810 msgr2=0x55f9ec039c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:16:20.063 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:20.059+0000 7f73363ff740 1 --2- 172.21.15.118:0/2204100167 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55f9ec039810 0x55f9ec039c30 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7f73140b9930 tx=0x7f73140bcdf0 comp rx=0 tx=0).stop 2024-03-24T00:16:20.063 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:20.060+0000 7f73363ff740 1 -- 172.21.15.118:0/2204100167 shutdown_connections 2024-03-24T00:16:20.063 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:20.060+0000 7f73363ff740 1 --2- 172.21.15.118:0/2204100167 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55f9ec03b3a0 0x55f9ec041c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:16:20.063 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:20.060+0000 7f73363ff740 1 --2- 172.21.15.118:0/2204100167 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55f9ec03a9e0 0x55f9ec03ae60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:16:20.064 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:20.060+0000 7f73363ff740 1 --2- 172.21.15.118:0/2204100167 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55f9ec039810 0x55f9ec039c30 unknown :-1 s=CLOSED pgs=216 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:16:20.064 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:20.060+0000 7f73363ff740 1 -- 172.21.15.118:0/2204100167 >> 172.21.15.118:0/2204100167 conn(0x55f9ec034c00 msgr2=0x55f9ec035000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:16:20.064 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:20.060+0000 7f73363ff740 1 -- 172.21.15.118:0/2204100167 shutdown_connections 2024-03-24T00:16:20.064 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:20.060+0000 7f73363ff740 1 -- 172.21.15.118:0/2204100167 wait complete. 2024-03-24T00:16:20.064 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:20.060+0000 7f73363ff740 1 -- start start 2024-03-24T00:16:20.064 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:20.060+0000 7f73363ff740 1 Processor -- start 2024-03-24T00:16:20.064 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:20.061+0000 7f73363ff740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55f9ec039810 0x55f9ec043810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:16:20.064 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:20.061+0000 7f73363ff740 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55f9ec03a9e0 0x55f9ec043d50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:16:20.064 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:20.061+0000 7f73363ff740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55f9ec03b3a0 0x55f9ec04ad90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:16:20.064 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:20.061+0000 7f73363ff740 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f9ec042170 con 0x55f9ec03a9e0 2024-03-24T00:16:20.064 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:20.061+0000 7f73363ff740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f9ebf5a0d0 con 0x55f9ec03b3a0 2024-03-24T00:16:20.064 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:20.061+0000 7f73363ff740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f9ec0422f0 con 0x55f9ec039810 2024-03-24T00:16:20.064 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:20.061+0000 7f73363ff740 10 client.0 _ll_register_callbacks cb 0x55f9ebf5ee80 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T00:16:20.064 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:20.061+0000 7f7330c00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55f9ec039810 0x55f9ec043810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:16:20.065 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:20.061+0000 7f7331600640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55f9ec03b3a0 0x55f9ec04ad90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:16:20.065 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:20.061+0000 7f7330c00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55f9ec039810 0x55f9ec043810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.118:45528/0 (socket says 172.21.15.118:45528) 2024-03-24T00:16:20.065 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:20.061+0000 7f7330c00640 1 -- 172.21.15.118:0/1174510969 learned_addr learned my addr 172.21.15.118:0/1174510969 (peer_addr_for_me v2:172.21.15.118:0/0) 2024-03-24T00:16:20.065 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:20.061+0000 7f732be00640 1 --2- 172.21.15.118:0/1174510969 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55f9ec03a9e0 0x55f9ec043d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:16:20.065 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:20.061+0000 7f7330c00640 1 -- 172.21.15.118:0/1174510969 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55f9ec03b3a0 msgr2=0x55f9ec04ad90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:16:20.065 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:20.061+0000 7f7330c00640 1 --2- 172.21.15.118:0/1174510969 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55f9ec03b3a0 0x55f9ec04ad90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:16:20.065 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:20.061+0000 7f7330c00640 1 -- 172.21.15.118:0/1174510969 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55f9ec03a9e0 msgr2=0x55f9ec043d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:16:20.065 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:20.061+0000 7f7330c00640 1 --2- 172.21.15.118:0/1174510969 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55f9ec03a9e0 0x55f9ec043d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:16:20.065 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:20.061+0000 7f7330c00640 1 -- 172.21.15.118:0/1174510969 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55f9ebcb23f0 con 0x55f9ec039810 2024-03-24T00:16:20.065 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:20.061+0000 7f732be00640 1 --2- 172.21.15.118:0/1174510969 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55f9ec03a9e0 0x55f9ec043d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T00:16:20.065 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:20.062+0000 7f7331600640 1 --2- 172.21.15.118:0/1174510969 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55f9ec03b3a0 0x55f9ec04ad90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T00:16:20.065 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:20.062+0000 7f7330c00640 1 --2- 172.21.15.118:0/1174510969 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55f9ec039810 0x55f9ec043810 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7f731402c7d0 tx=0x7f731402bc50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=7ab3f31b8d6d1e8e server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:16:20.065 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:20.062+0000 7f731a000640 10 client.0 ms_handle_connect on v2:172.21.15.193:3300/0 2024-03-24T00:16:20.065 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:20.062+0000 7f731a000640 1 -- 172.21.15.118:0/1174510969 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f731404b070 con 0x55f9ec039810 2024-03-24T00:16:20.065 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:20.062+0000 7f731a000640 1 -- 172.21.15.118:0/1174510969 <== mon.1 v2:172.21.15.193:3300/0 2 ==== osd_map(90..90 src has 1..90) v4 ==== 8277+0+0 (secure 0 0 0) 0x7f7314003810 con 0x55f9ec039810 2024-03-24T00:16:20.066 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:20.062+0000 7f73363ff740 1 -- 172.21.15.118:0/1174510969 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f73140bd660 con 0x55f9ec039810 2024-03-24T00:16:20.066 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:20.062+0000 7f731a000640 1 -- 172.21.15.118:0/1174510969 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) v2 ==== 24+0+0 (secure 0 0 0) 0x7f73140bcf60 con 0x55f9ec039810 2024-03-24T00:16:20.066 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:20.062+0000 7f73363ff740 10 client.25519 fetch_fsmap learned FSMap version 28 2024-03-24T00:16:20.066 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:20.062+0000 7f73363ff740 1 -- 172.21.15.118:0/1174510969 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) v3 -- 0x55f9ebd7c7d0 con 0x55f9ec039810 2024-03-24T00:16:20.066 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:20.062+0000 7f731a000640 1 -- 172.21.15.118:0/1174510969 <== mon.1 v2:172.21.15.193:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f73140bd660 con 0x55f9ec039810 2024-03-24T00:16:20.066 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:20.063+0000 7f731a000640 1 -- 172.21.15.118:0/1174510969 <== mon.1 v2:172.21.15.193:3300/0 5 ==== fsmap(e 28) v1 ==== 7934+0+0 (secure 0 0 0) 0x7f73140c0070 con 0x55f9ec039810 2024-03-24T00:16:20.066 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:20.063+0000 7f73363ff740 10 client.25519 fetch_fsmap finished waiting for FSMap version 28 2024-03-24T00:16:20.066 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:20.063+0000 7f73363ff740 10 client.25519 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.c 2024-03-24T00:16:20.066 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:20.063+0000 7f73363ff740 20 client.25519 populate_metadata read hostname 'smithi118' 2024-03-24T00:16:20.066 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:20.063+0000 7f73363ff740 1 --2- 172.21.15.118:0/1174510969 >> [v2:172.21.15.193:6836/2921520053,v1:172.21.15.193:6837/2921520053] conn(0x55f9ec04efb0 0x55f9ec0513a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:16:20.066 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:20.063+0000 7f73363ff740 4 client.25519 mds_command: new command op to 24511 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-24T00:16:20.066 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:20.063+0000 7f73363ff740 1 -- 172.21.15.118:0/1174510969 --> [v2:172.21.15.193:6836/2921520053,v1:172.21.15.193:6837/2921520053] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55f9ebfab800 con 0x55f9ec04efb0 2024-03-24T00:16:20.066 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:20.063+0000 7f732be00640 1 --2- 172.21.15.118:0/1174510969 >> [v2:172.21.15.193:6836/2921520053,v1:172.21.15.193:6837/2921520053] conn(0x55f9ec04efb0 0x55f9ec0513a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:16:20.067 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:20.063+0000 7f732be00640 1 --2- 172.21.15.118:0/1174510969 >> [v2:172.21.15.193:6836/2921520053,v1:172.21.15.193:6837/2921520053] conn(0x55f9ec04efb0 0x55f9ec0513a0 crc :-1 s=READY pgs=18 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=27a2996ba9830787 server_cookie=b7cbabb8d9c21832 in_seq=0 out_seq=0 2024-03-24T00:16:20.067 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:20.063+0000 7f731a000640 10 client.25519 ms_handle_connect on v2:172.21.15.193:6836/2921520053 2024-03-24T00:16:20.068 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:20.064+0000 7f731a000640 1 -- 172.21.15.118:0/1174510969 <== mds.2 v2:172.21.15.193:6836/2921520053 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x55f9ebfab800 con 0x55f9ec04efb0 2024-03-24T00:16:20.068 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:20.064+0000 7f731a000640 10 client.25519 handle_command_reply: tid=0 2024-03-24T00:16:20.068 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:20.064+0000 7f73363ff740 1 client.25519 shutdown 2024-03-24T00:16:20.068 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:20.064+0000 7f73363ff740 1 -- 172.21.15.118:0/1174510969 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55f9ec039810 msgr2=0x55f9ec043810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:16:20.068 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:20.064+0000 7f73363ff740 1 --2- 172.21.15.118:0/1174510969 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55f9ec039810 0x55f9ec043810 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7f731402c7d0 tx=0x7f731402bc50 comp rx=0 tx=0).stop 2024-03-24T00:16:20.068 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:20.064+0000 7f73363ff740 1 -- 172.21.15.118:0/1174510969 shutdown_connections 2024-03-24T00:16:20.068 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:20.064+0000 7f73363ff740 1 --2- 172.21.15.118:0/1174510969 >> [v2:172.21.15.193:6836/2921520053,v1:172.21.15.193:6837/2921520053] conn(0x55f9ec04efb0 0x55f9ec0513a0 crc :-1 s=READY pgs=18 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:16:20.068 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:20.064+0000 7f73363ff740 1 --2- 172.21.15.118:0/1174510969 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55f9ec03b3a0 0x55f9ec04ad90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:16:20.068 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:20.064+0000 7f73363ff740 1 --2- 172.21.15.118:0/1174510969 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55f9ec03a9e0 0x55f9ec043d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:16:20.068 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:20.064+0000 7f73363ff740 1 --2- 172.21.15.118:0/1174510969 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55f9ec039810 0x55f9ec043810 unknown :-1 s=CLOSED pgs=217 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:16:20.068 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:20.064+0000 7f731a000640 0 client.25519 ms_handle_reset on v2:172.21.15.193:6836/2921520053 2024-03-24T00:16:20.068 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:20.064+0000 7f73363ff740 1 -- 172.21.15.118:0/1174510969 >> 172.21.15.118:0/1174510969 conn(0x55f9ec034c00 msgr2=0x55f9ec033eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:16:20.069 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:20.064+0000 7f73363ff740 1 -- 172.21.15.118:0/1174510969 shutdown_connections 2024-03-24T00:16:20.069 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:20.065+0000 7f73363ff740 1 -- 172.21.15.118:0/1174510969 wait complete. 2024-03-24T00:16:20.069 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:20.065+0000 7f73363ff740 20 client.25519 trim_cache size 0 max 16384 2024-03-24T00:16:20.077 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:20.078+0000 7f73363ff740 1 Processor -- start 2024-03-24T00:16:20.077 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:20.078+0000 7f73363ff740 1 -- start start 2024-03-24T00:16:20.078 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:20.078+0000 7f73363ff740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55f9ec039810 0x55f9ec03da80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:16:20.078 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:20.078+0000 7f73363ff740 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55f9ec03a9e0 0x55f9ec04ad50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:16:20.078 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:20.078+0000 7f73363ff740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55f9ec03b3a0 0x55f9ec03dfc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:16:20.078 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:20.078+0000 7f73363ff740 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f732c1a89e0 con 0x55f9ec03a9e0 2024-03-24T00:16:20.078 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:20.078+0000 7f73363ff740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f732c1aa360 con 0x55f9ec039810 2024-03-24T00:16:20.078 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:20.078+0000 7f73363ff740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f732c1a9040 con 0x55f9ec03b3a0 2024-03-24T00:16:20.080 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:20.078+0000 7f7330c00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55f9ec039810 0x55f9ec03da80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:16:20.080 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:20.078+0000 7f7331600640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55f9ec03b3a0 0x55f9ec03dfc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:16:20.080 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:20.078+0000 7f7330c00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55f9ec039810 0x55f9ec03da80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.118:60306/0 (socket says 172.21.15.118:60306) 2024-03-24T00:16:20.080 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:20.078+0000 7f7330c00640 1 -- 172.21.15.118:0/3477235051 learned_addr learned my addr 172.21.15.118:0/3477235051 (peer_addr_for_me v2:172.21.15.118:0/0) 2024-03-24T00:16:20.080 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:20.079+0000 7f732be00640 1 --2- 172.21.15.118:0/3477235051 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55f9ec03a9e0 0x55f9ec04ad50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:16:20.080 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:20.079+0000 7f7331600640 1 -- 172.21.15.118:0/3477235051 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55f9ec039810 msgr2=0x55f9ec03da80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:16:20.080 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:20.079+0000 7f7331600640 1 --2- 172.21.15.118:0/3477235051 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55f9ec039810 0x55f9ec03da80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:16:20.080 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:20.079+0000 7f7331600640 1 -- 172.21.15.118:0/3477235051 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55f9ec03a9e0 msgr2=0x55f9ec04ad50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:16:20.080 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:20.079+0000 7f7331600640 1 --2- 172.21.15.118:0/3477235051 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55f9ec03a9e0 0x55f9ec04ad50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:16:20.080 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:20.079+0000 7f7331600640 1 -- 172.21.15.118:0/3477235051 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55f9ec0415c0 con 0x55f9ec03b3a0 2024-03-24T00:16:20.080 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:20.079+0000 7f7330c00640 1 --2- 172.21.15.118:0/3477235051 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55f9ec039810 0x55f9ec03da80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T00:16:20.080 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:20.079+0000 7f7331600640 1 --2- 172.21.15.118:0/3477235051 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55f9ec03b3a0 0x55f9ec03dfc0 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7f7324005d00 tx=0x7f7324012820 comp rx=0 tx=0).ready entity=mon.1 client_cookie=bfe61339b45b0ecc server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:16:20.081 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:20.079+0000 7f7321200640 1 -- 172.21.15.118:0/3477235051 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f732400be20 con 0x55f9ec03b3a0 2024-03-24T00:16:20.081 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:20.079+0000 7f7321200640 1 -- 172.21.15.118:0/3477235051 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f73240035f0 con 0x55f9ec03b3a0 2024-03-24T00:16:20.081 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:20.079+0000 7f7321200640 1 -- 172.21.15.118:0/3477235051 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f732401b670 con 0x55f9ec03b3a0 2024-03-24T00:16:20.081 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:20.080+0000 7f73363ff740 1 -- 172.21.15.118:0/3477235051 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55f9ec03b3a0 msgr2=0x55f9ec03dfc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:16:20.081 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:20.080+0000 7f73363ff740 1 --2- 172.21.15.118:0/3477235051 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55f9ec03b3a0 0x55f9ec03dfc0 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7f7324005d00 tx=0x7f7324012820 comp rx=0 tx=0).stop 2024-03-24T00:16:20.081 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:20.080+0000 7f73363ff740 1 -- 172.21.15.118:0/3477235051 shutdown_connections 2024-03-24T00:16:20.081 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:20.080+0000 7f73363ff740 1 --2- 172.21.15.118:0/3477235051 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55f9ec03b3a0 0x55f9ec03dfc0 unknown :-1 s=CLOSED pgs=218 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:16:20.081 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:20.080+0000 7f73363ff740 1 --2- 172.21.15.118:0/3477235051 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55f9ec03a9e0 0x55f9ec04ad50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:16:20.081 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:20.080+0000 7f73363ff740 1 --2- 172.21.15.118:0/3477235051 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55f9ec039810 0x55f9ec03da80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:16:20.081 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:20.080+0000 7f73363ff740 1 -- 172.21.15.118:0/3477235051 >> 172.21.15.118:0/3477235051 conn(0x55f9ec034c00 msgr2=0x55f9ec0386f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:16:20.081 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:20.080+0000 7f73363ff740 1 -- 172.21.15.118:0/3477235051 shutdown_connections 2024-03-24T00:16:20.081 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:20.080+0000 7f73363ff740 1 -- 172.21.15.118:0/3477235051 wait complete. 2024-03-24T00:16:20.081 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:20.080+0000 7f73363ff740 1 -- start start 2024-03-24T00:16:20.085 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:20.081+0000 7f73363ff740 1 Processor -- start 2024-03-24T00:16:20.085 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:20.081+0000 7f73363ff740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55f9ec039810 0x55f9ec04a800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:16:20.085 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:20.081+0000 7f73363ff740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55f9ec03a9e0 0x55f9ec04ad40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:16:20.085 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:20.081+0000 7f73363ff740 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55f9ec03b3a0 0x55f9ec047390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:16:20.085 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:20.081+0000 7f73363ff740 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f732c1a9d00 con 0x55f9ec03b3a0 2024-03-24T00:16:20.085 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:20.081+0000 7f73363ff740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f732c1a96a0 con 0x55f9ec03a9e0 2024-03-24T00:16:20.085 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:20.081+0000 7f73363ff740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f732c1a6ba0 con 0x55f9ec039810 2024-03-24T00:16:20.085 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:20.081+0000 7f73363ff740 10 client.0 _ll_register_callbacks cb 0x7f731c0113e0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T00:16:20.085 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:20.081+0000 7f7330c00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55f9ec039810 0x55f9ec04a800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:16:20.085 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:20.081+0000 7f7330c00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55f9ec039810 0x55f9ec04a800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.118:45546/0 (socket says 172.21.15.118:45546) 2024-03-24T00:16:20.085 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:20.081+0000 7f732be00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55f9ec03a9e0 0x55f9ec04ad40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:16:20.086 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:20.081+0000 7f7330c00640 1 -- 172.21.15.118:0/1700596242 learned_addr learned my addr 172.21.15.118:0/1700596242 (peer_addr_for_me v2:172.21.15.118:0/0) 2024-03-24T00:16:20.086 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:20.081+0000 7f7331600640 1 --2- 172.21.15.118:0/1700596242 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55f9ec03b3a0 0x55f9ec047390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:16:20.086 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:20.082+0000 7f7330c00640 1 -- 172.21.15.118:0/1700596242 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55f9ec03a9e0 msgr2=0x55f9ec04ad40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:16:20.086 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:20.082+0000 7f7330c00640 1 --2- 172.21.15.118:0/1700596242 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55f9ec03a9e0 0x55f9ec04ad40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:16:20.086 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:20.082+0000 7f7330c00640 1 -- 172.21.15.118:0/1700596242 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55f9ec03b3a0 msgr2=0x55f9ec047390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:16:20.086 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:20.082+0000 7f7330c00640 1 --2- 172.21.15.118:0/1700596242 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55f9ec03b3a0 0x55f9ec047390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:16:20.086 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:20.082+0000 7f7330c00640 1 -- 172.21.15.118:0/1700596242 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55f9ec047bc0 con 0x55f9ec039810 2024-03-24T00:16:20.086 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:20.082+0000 7f7331600640 1 --2- 172.21.15.118:0/1700596242 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55f9ec03b3a0 0x55f9ec047390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T00:16:20.086 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:20.082+0000 7f732be00640 1 --2- 172.21.15.118:0/1700596242 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55f9ec03a9e0 0x55f9ec04ad40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T00:16:20.086 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:20.082+0000 7f7330c00640 1 --2- 172.21.15.118:0/1700596242 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55f9ec039810 0x55f9ec04a800 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7f731402c190 tx=0x7f73140dd040 comp rx=0 tx=0).ready entity=mon.1 client_cookie=e55d4e3c8c055065 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:16:20.086 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:20.082+0000 7f731a000640 10 client.0 ms_handle_connect on v2:172.21.15.193:3300/0 2024-03-24T00:16:20.086 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:20.082+0000 7f731a000640 1 -- 172.21.15.118:0/1700596242 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f731404b070 con 0x55f9ec039810 2024-03-24T00:16:20.086 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:20.082+0000 7f731a000640 1 -- 172.21.15.118:0/1700596242 <== mon.1 v2:172.21.15.193:3300/0 2 ==== osd_map(90..90 src has 1..90) v4 ==== 8277+0+0 (secure 0 0 0) 0x7f73140bbf40 con 0x55f9ec039810 2024-03-24T00:16:20.087 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:20.082+0000 7f73363ff740 1 -- 172.21.15.118:0/1700596242 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f73240035f0 con 0x55f9ec039810 2024-03-24T00:16:20.087 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:20.082+0000 7f731a000640 1 -- 172.21.15.118:0/1700596242 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) v2 ==== 24+0+0 (secure 0 0 0) 0x7f73140bc8c0 con 0x55f9ec039810 2024-03-24T00:16:20.087 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:20.082+0000 7f73363ff740 10 client.25525 fetch_fsmap learned FSMap version 28 2024-03-24T00:16:20.087 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:20.082+0000 7f73363ff740 1 -- 172.21.15.118:0/1700596242 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) v3 -- 0x55f9ec047bc0 con 0x55f9ec039810 2024-03-24T00:16:20.087 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:20.083+0000 7f731a000640 1 -- 172.21.15.118:0/1700596242 <== mon.1 v2:172.21.15.193:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f73240035f0 con 0x55f9ec039810 2024-03-24T00:16:20.087 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:20.083+0000 7f731a000640 1 -- 172.21.15.118:0/1700596242 <== mon.1 v2:172.21.15.193:3300/0 5 ==== fsmap(e 28) v1 ==== 7934+0+0 (secure 0 0 0) 0x7f73140c0070 con 0x55f9ec039810 2024-03-24T00:16:20.087 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:20.083+0000 7f73363ff740 10 client.25525 fetch_fsmap finished waiting for FSMap version 28 2024-03-24T00:16:20.087 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:20.083+0000 7f73363ff740 10 client.25525 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.c 2024-03-24T00:16:20.087 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:20.083+0000 7f73363ff740 20 client.25525 populate_metadata read hostname 'smithi118' 2024-03-24T00:16:20.087 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:20.083+0000 7f73363ff740 1 --2- 172.21.15.118:0/1700596242 >> [v2:172.21.15.193:6836/2921520053,v1:172.21.15.193:6837/2921520053] conn(0x55f9ec0abb90 0x55f9ec0adfb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:16:20.087 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:20.083+0000 7f73363ff740 4 client.25525 mds_command: new command op to 24511 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-24T00:16:20.087 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:20.083+0000 7f73363ff740 1 -- 172.21.15.118:0/1700596242 --> [v2:172.21.15.193:6836/2921520053,v1:172.21.15.193:6837/2921520053] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x55f9ebfa5b50 con 0x55f9ec0abb90 2024-03-24T00:16:20.087 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:20.083+0000 7f732be00640 1 --2- 172.21.15.118:0/1700596242 >> [v2:172.21.15.193:6836/2921520053,v1:172.21.15.193:6837/2921520053] conn(0x55f9ec0abb90 0x55f9ec0adfb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:16:20.087 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:20.084+0000 7f732be00640 1 --2- 172.21.15.118:0/1700596242 >> [v2:172.21.15.193:6836/2921520053,v1:172.21.15.193:6837/2921520053] conn(0x55f9ec0abb90 0x55f9ec0adfb0 crc :-1 s=READY pgs=19 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=56a9c2576b865349 server_cookie=92030d5fc4bc5f11 in_seq=0 out_seq=0 2024-03-24T00:16:20.088 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:20.084+0000 7f731a000640 10 client.25525 ms_handle_connect on v2:172.21.15.193:6836/2921520053 2024-03-24T00:16:20.088 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:20.084+0000 7f731a000640 1 -- 172.21.15.118:0/1700596242 <== mds.2 v2:172.21.15.193:6836/2921520053 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x55f9ebfa5b50 con 0x55f9ec0abb90 2024-03-24T00:16:20.088 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:20.084+0000 7f731a000640 10 client.25525 handle_command_reply: tid=0 2024-03-24T00:16:20.088 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:20.084+0000 7f73363ff740 1 client.25525 shutdown 2024-03-24T00:16:20.088 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:20.084+0000 7f73363ff740 1 -- 172.21.15.118:0/1700596242 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55f9ec039810 msgr2=0x55f9ec04a800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:16:20.088 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:20.084+0000 7f73363ff740 1 --2- 172.21.15.118:0/1700596242 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55f9ec039810 0x55f9ec04a800 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7f731402c190 tx=0x7f73140dd040 comp rx=0 tx=0).stop 2024-03-24T00:16:20.088 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:20.084+0000 7f73363ff740 1 -- 172.21.15.118:0/1700596242 shutdown_connections 2024-03-24T00:16:20.088 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:20.084+0000 7f73363ff740 1 --2- 172.21.15.118:0/1700596242 >> [v2:172.21.15.193:6836/2921520053,v1:172.21.15.193:6837/2921520053] conn(0x55f9ec0abb90 0x55f9ec0adfb0 crc :-1 s=READY pgs=19 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:16:20.088 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:20.084+0000 7f73363ff740 1 --2- 172.21.15.118:0/1700596242 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55f9ec03b3a0 0x55f9ec047390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:16:20.088 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:20.084+0000 7f731a000640 0 client.25525 ms_handle_reset on v2:172.21.15.193:6836/2921520053 2024-03-24T00:16:20.088 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:20.084+0000 7f73363ff740 1 --2- 172.21.15.118:0/1700596242 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55f9ec03a9e0 0x55f9ec04ad40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:16:20.088 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:20.084+0000 7f73363ff740 1 --2- 172.21.15.118:0/1700596242 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55f9ec039810 0x55f9ec04a800 unknown :-1 s=CLOSED pgs=219 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:16:20.088 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:20.084+0000 7f73363ff740 1 -- 172.21.15.118:0/1700596242 >> 172.21.15.118:0/1700596242 conn(0x55f9ec034c00 msgr2=0x55f9ec033620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:16:20.088 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:20.084+0000 7f73363ff740 1 -- 172.21.15.118:0/1700596242 shutdown_connections 2024-03-24T00:16:20.089 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:20.084+0000 7f73363ff740 1 -- 172.21.15.118:0/1700596242 wait complete. 2024-03-24T00:16:20.089 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:20.084+0000 7f73363ff740 20 client.25525 trim_cache size 0 max 16384 2024-03-24T00:16:20.089 INFO:teuthology.orchestra.run.smithi118.stdout:[] 2024-03-24T00:16:20.090 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:20.085+0000 7f7333600640 1 -- 172.21.15.118:0/282353499 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7f73100a2910 msgr2=0x7f73100a4dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:16:20.090 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:20.085+0000 7f7333600640 1 --2- 172.21.15.118:0/282353499 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7f73100a2910 0x7f73100a4dd0 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7f732c198fb0 tx=0x7f731c0096c0 comp rx=0 tx=0).stop 2024-03-24T00:16:20.090 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:20.085+0000 7f7333600640 1 -- 172.21.15.118:0/282353499 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f732c102130 msgr2=0x7f732c197aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:16:20.090 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:20.085+0000 7f7333600640 1 --2- 172.21.15.118:0/282353499 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f732c102130 0x7f732c197aa0 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7f7314036040 tx=0x7f7314005dd0 comp rx=0 tx=0).stop 2024-03-24T00:16:20.090 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:20.085+0000 7f7333600640 1 -- 172.21.15.118:0/282353499 shutdown_connections 2024-03-24T00:16:20.090 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:20.085+0000 7f7333600640 1 --2- 172.21.15.118:0/282353499 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7f73100a2910 0x7f73100a4dd0 unknown :-1 s=CLOSED pgs=277 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:16:20.090 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:20.085+0000 7f7333600640 1 --2- 172.21.15.118:0/282353499 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f732c107390 0x7f732c19f050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:16:20.090 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:20.085+0000 7f7333600640 1 --2- 172.21.15.118:0/282353499 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f732c104a60 0x7f732c197fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:16:20.090 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:20.085+0000 7f7333600640 1 --2- 172.21.15.118:0/282353499 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f732c102130 0x7f732c197aa0 unknown :-1 s=CLOSED pgs=215 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:16:20.090 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:20.085+0000 7f7333600640 1 -- 172.21.15.118:0/282353499 >> 172.21.15.118:0/282353499 conn(0x7f732c0fbb40 msgr2=0x7f732c102a90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:16:20.090 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:20.085+0000 7f7333600640 1 -- 172.21.15.118:0/282353499 shutdown_connections 2024-03-24T00:16:20.091 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:20.085+0000 7f7333600640 1 -- 172.21.15.118:0/282353499 wait complete. 2024-03-24T00:16:20.102 INFO:teuthology.orchestra.run:Running command with timeout 120 2024-03-24T00:16:20.102 DEBUG:teuthology.orchestra.run.smithi118:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:1 damage ls 2024-03-24T00:16:20.203 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:16:19 smithi189 ceph-mon[36187]: from='client.? 172.21.15.118:0/3331377977' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-24T00:16:20.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:16:19 smithi193 ceph-mon[36878]: from='client.? 172.21.15.118:0/3331377977' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-24T00:16:20.307 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:20.307+0000 7f34bcc00640 1 -- 172.21.15.118:0/240149269 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f34b81015a0 msgr2=0x7f34b81019c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:16:20.307 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:20.307+0000 7f34bcc00640 1 --2- 172.21.15.118:0/240149269 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f34b81015a0 0x7f34b81019c0 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7f34a8009900 tx=0x7f34a802bc30 comp rx=0 tx=0).stop 2024-03-24T00:16:20.307 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:20.307+0000 7f34bcc00640 1 -- 172.21.15.118:0/240149269 shutdown_connections 2024-03-24T00:16:20.308 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:20.307+0000 7f34bcc00640 1 --2- 172.21.15.118:0/240149269 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f34b8103150 0x7f34b81099e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:16:20.308 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:20.307+0000 7f34bcc00640 1 --2- 172.21.15.118:0/240149269 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f34b8102790 0x7f34b8102c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:16:20.308 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:20.307+0000 7f34bcc00640 1 --2- 172.21.15.118:0/240149269 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f34b81015a0 0x7f34b81019c0 unknown :-1 s=CLOSED pgs=231 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:16:20.308 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:20.307+0000 7f34bcc00640 1 -- 172.21.15.118:0/240149269 >> 172.21.15.118:0/240149269 conn(0x7f34b80fcb10 msgr2=0x7f34b80fef50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:16:20.308 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:20.307+0000 7f34bcc00640 1 -- 172.21.15.118:0/240149269 shutdown_connections 2024-03-24T00:16:20.309 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:20.307+0000 7f34bcc00640 1 -- 172.21.15.118:0/240149269 wait complete. 2024-03-24T00:16:20.311 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:20.309+0000 7f34bcc00640 1 Processor -- start 2024-03-24T00:16:20.311 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:20.309+0000 7f34bcc00640 1 -- start start 2024-03-24T00:16:20.311 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:20.309+0000 7f34bcc00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f34b81015a0 0x7f34b8197910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:16:20.312 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:20.309+0000 7f34bcc00640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f34b8102790 0x7f34b8197e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:16:20.312 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:20.309+0000 7f34bcc00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f34b8103150 0x7f34b819eec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:16:20.312 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:20.309+0000 7f34bcc00640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f34b810d260 con 0x7f34b8102790 2024-03-24T00:16:20.312 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:20.309+0000 7f34bcc00640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f34b810d0e0 con 0x7f34b81015a0 2024-03-24T00:16:20.312 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:20.309+0000 7f34bcc00640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f34b810d3e0 con 0x7f34b8103150 2024-03-24T00:16:20.312 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:20.310+0000 7f34b7400640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f34b81015a0 0x7f34b8197910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:16:20.312 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:20.310+0000 7f34b7400640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f34b81015a0 0x7f34b8197910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.118:60336/0 (socket says 172.21.15.118:60336) 2024-03-24T00:16:20.312 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:20.310+0000 7f34b7400640 1 -- 172.21.15.118:0/524017738 learned_addr learned my addr 172.21.15.118:0/524017738 (peer_addr_for_me v2:172.21.15.118:0/0) 2024-03-24T00:16:20.312 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:20.310+0000 7f34afe00640 1 --2- 172.21.15.118:0/524017738 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f34b8102790 0x7f34b8197e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:16:20.312 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:20.310+0000 7f34b7e00640 1 --2- 172.21.15.118:0/524017738 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f34b8103150 0x7f34b819eec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:16:20.312 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:20.310+0000 7f34b7400640 1 -- 172.21.15.118:0/524017738 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f34b8103150 msgr2=0x7f34b819eec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:16:20.312 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:20.310+0000 7f34b7400640 1 --2- 172.21.15.118:0/524017738 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f34b8103150 0x7f34b819eec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:16:20.312 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:20.310+0000 7f34b7400640 1 -- 172.21.15.118:0/524017738 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f34b8102790 msgr2=0x7f34b8197e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:16:20.312 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:20.310+0000 7f34b7400640 1 --2- 172.21.15.118:0/524017738 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f34b8102790 0x7f34b8197e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:16:20.313 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:20.310+0000 7f34b7400640 1 -- 172.21.15.118:0/524017738 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f34b819f530 con 0x7f34b81015a0 2024-03-24T00:16:20.313 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:20.310+0000 7f34afe00640 1 --2- 172.21.15.118:0/524017738 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f34b8102790 0x7f34b8197e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T00:16:20.313 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:20.310+0000 7f34b7400640 1 --2- 172.21.15.118:0/524017738 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f34b81015a0 0x7f34b8197910 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7f34a8000c00 tx=0x7f34a802c6c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:16:20.313 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:20.310+0000 7f34b4c00640 1 -- 172.21.15.118:0/524017738 <== mon.2 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f34a8043020 con 0x7f34b81015a0 2024-03-24T00:16:20.313 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:20.311+0000 7f34b4c00640 1 -- 172.21.15.118:0/524017738 <== mon.2 v2:172.21.15.189:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f34a80054e0 con 0x7f34b81015a0 2024-03-24T00:16:20.313 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:20.311+0000 7f34bcc00640 1 -- 172.21.15.118:0/524017738 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f34b819f7c0 con 0x7f34b81015a0 2024-03-24T00:16:20.313 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:20.311+0000 7f34b4c00640 1 -- 172.21.15.118:0/524017738 <== mon.2 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f34a8003ce0 con 0x7f34b81015a0 2024-03-24T00:16:20.313 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:20.311+0000 7f34bcc00640 1 -- 172.21.15.118:0/524017738 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f34b819fd30 con 0x7f34b81015a0 2024-03-24T00:16:20.321 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:20.314+0000 7f34b4c00640 1 -- 172.21.15.118:0/524017738 <== mon.2 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 19) v1 ==== 137342+0+0 (secure 0 0 0) 0x7f34a8046020 con 0x7f34b81015a0 2024-03-24T00:16:20.321 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:20.314+0000 7f34c103a740 1 Processor -- start 2024-03-24T00:16:20.321 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:20.314+0000 7f34c103a740 1 -- start start 2024-03-24T00:16:20.321 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:20.315+0000 7f34c103a740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5582d7f449f0 0x5582d7f44e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:16:20.321 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:20.315+0000 7f34c103a740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5582d7f45bc0 0x5582d7f46040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:16:20.322 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:20.315+0000 7f34c103a740 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5582d7f46580 0x5582d7f4ce10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:16:20.322 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:20.315+0000 7f34c103a740 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x5582d7ea92c0 con 0x5582d7f46580 2024-03-24T00:16:20.322 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:20.315+0000 7f34c103a740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x5582d7e51c90 con 0x5582d7f449f0 2024-03-24T00:16:20.322 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:20.315+0000 7f34c103a740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x5582d7ea5360 con 0x5582d7f45bc0 2024-03-24T00:16:20.322 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:20.315+0000 7f34b7400640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5582d7f45bc0 0x5582d7f46040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:16:20.322 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:20.315+0000 7f34b7e00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5582d7f449f0 0x5582d7f44e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:16:20.322 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:20.315+0000 7f34b7e00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5582d7f449f0 0x5582d7f44e10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.118:60350/0 (socket says 172.21.15.118:60350) 2024-03-24T00:16:20.322 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:20.315+0000 7f34b7e00640 1 -- 172.21.15.118:0/189001085 learned_addr learned my addr 172.21.15.118:0/189001085 (peer_addr_for_me v2:172.21.15.118:0/0) 2024-03-24T00:16:20.322 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:20.315+0000 7f34afe00640 1 --2- 172.21.15.118:0/189001085 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5582d7f46580 0x5582d7f4ce10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:16:20.322 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:20.315+0000 7f34b7e00640 1 -- 172.21.15.118:0/189001085 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5582d7f45bc0 msgr2=0x5582d7f46040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:16:20.322 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:20.315+0000 7f34b7e00640 1 --2- 172.21.15.118:0/189001085 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5582d7f45bc0 0x5582d7f46040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:16:20.322 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:20.315+0000 7f34b7e00640 1 -- 172.21.15.118:0/189001085 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5582d7f46580 msgr2=0x5582d7f4ce10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:16:20.322 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:20.315+0000 7f34b7e00640 1 --2- 172.21.15.118:0/189001085 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5582d7f46580 0x5582d7f4ce10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:16:20.323 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:20.315+0000 7f34b7e00640 1 -- 172.21.15.118:0/189001085 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5582d7e76080 con 0x5582d7f449f0 2024-03-24T00:16:20.323 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:20.315+0000 7f34afe00640 1 --2- 172.21.15.118:0/189001085 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5582d7f46580 0x5582d7f4ce10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T00:16:20.323 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:20.315+0000 7f34b4c00640 1 --2- 172.21.15.118:0/524017738 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7f349c0a2910 0x7f349c0a4dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:16:20.323 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:20.316+0000 7f34b7e00640 1 --2- 172.21.15.118:0/189001085 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5582d7f449f0 0x5582d7f44e10 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7f34b00039f0 tx=0x7f34b0003eb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=619844b7a12f6fe5 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:16:20.323 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:20.316+0000 7f34b4c00640 1 -- 172.21.15.118:0/524017738 <== mon.2 v2:172.21.15.189:3300/0 5 ==== osd_map(90..90 src has 1..90) v4 ==== 8277+0+0 (secure 0 0 0) 0x7f34a80f0c70 con 0x7f34b81015a0 2024-03-24T00:16:20.323 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:20.316+0000 7f34b7400640 1 --2- 172.21.15.118:0/524017738 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7f349c0a2910 0x7f349c0a4dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:16:20.323 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:20.316+0000 7f34a7e00640 1 -- 172.21.15.118:0/189001085 <== mon.2 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f34b001a470 con 0x5582d7f449f0 2024-03-24T00:16:20.323 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:20.316+0000 7f34a7e00640 1 -- 172.21.15.118:0/189001085 <== mon.2 v2:172.21.15.189:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f34b0015070 con 0x5582d7f449f0 2024-03-24T00:16:20.323 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:20.316+0000 7f34a7e00640 1 -- 172.21.15.118:0/189001085 <== mon.2 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f34b0010900 con 0x5582d7f449f0 2024-03-24T00:16:20.323 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:20.317+0000 7f34c103a740 1 -- 172.21.15.118:0/189001085 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5582d7f449f0 msgr2=0x5582d7f44e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:16:20.323 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:20.317+0000 7f34b7400640 1 --2- 172.21.15.118:0/524017738 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7f349c0a2910 0x7f349c0a4dd0 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7f34a80c57a0 tx=0x7f34a80f4720 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:16:20.323 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:20.317+0000 7f34c103a740 1 --2- 172.21.15.118:0/189001085 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5582d7f449f0 0x5582d7f44e10 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7f34b00039f0 tx=0x7f34b0003eb0 comp rx=0 tx=0).stop 2024-03-24T00:16:20.323 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:20.317+0000 7f34c103a740 1 -- 172.21.15.118:0/189001085 shutdown_connections 2024-03-24T00:16:20.323 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:20.317+0000 7f34c103a740 1 --2- 172.21.15.118:0/189001085 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5582d7f46580 0x5582d7f4ce10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:16:20.324 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:20.317+0000 7f34c103a740 1 --2- 172.21.15.118:0/189001085 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5582d7f45bc0 0x5582d7f46040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:16:20.324 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:20.317+0000 7f34c103a740 1 --2- 172.21.15.118:0/189001085 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5582d7f449f0 0x5582d7f44e10 unknown :-1 s=CLOSED pgs=233 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:16:20.324 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:20.317+0000 7f34c103a740 1 -- 172.21.15.118:0/189001085 >> 172.21.15.118:0/189001085 conn(0x5582d7f3fde0 msgr2=0x5582d7f401e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:16:20.324 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:20.317+0000 7f34c103a740 1 -- 172.21.15.118:0/189001085 shutdown_connections 2024-03-24T00:16:20.324 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:20.317+0000 7f34c103a740 1 -- 172.21.15.118:0/189001085 wait complete. 2024-03-24T00:16:20.324 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:20.318+0000 7f34c103a740 1 -- start start 2024-03-24T00:16:20.324 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:20.318+0000 7f34c103a740 1 Processor -- start 2024-03-24T00:16:20.324 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:20.319+0000 7f34c103a740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5582d7f449f0 0x5582d7f4ecb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:16:20.324 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:20.319+0000 7f34c103a740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5582d7f45bc0 0x5582d7f4f1f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:16:20.324 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:20.319+0000 7f34c103a740 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5582d7f46580 0x5582d7f56230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:16:20.324 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:20.319+0000 7f34c103a740 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x5582d7f4d590 con 0x5582d7f46580 2024-03-24T00:16:20.324 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:20.319+0000 7f34c103a740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x5582d7e650d0 con 0x5582d7f45bc0 2024-03-24T00:16:20.324 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:20.319+0000 7f34c103a740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x5582d7f4d710 con 0x5582d7f449f0 2024-03-24T00:16:20.324 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:20.319+0000 7f34c103a740 10 client.0 _ll_register_callbacks cb 0x5582d7e69e80 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T00:16:20.324 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:20.319+0000 7f34afe00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5582d7f45bc0 0x5582d7f4f1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:16:20.325 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:20.319+0000 7f34afe00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5582d7f45bc0 0x5582d7f4f1f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.118:60362/0 (socket says 172.21.15.118:60362) 2024-03-24T00:16:20.325 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:20.319+0000 7f34afe00640 1 -- 172.21.15.118:0/2513807091 learned_addr learned my addr 172.21.15.118:0/2513807091 (peer_addr_for_me v2:172.21.15.118:0/0) 2024-03-24T00:16:20.325 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:20.319+0000 7f34b7e00640 1 --2- 172.21.15.118:0/2513807091 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5582d7f449f0 0x5582d7f4ecb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:16:20.325 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:20.319+0000 7f34b7e00640 1 --2- 172.21.15.118:0/2513807091 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5582d7f46580 0x5582d7f56230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:16:20.325 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:20.319+0000 7f34afe00640 1 -- 172.21.15.118:0/2513807091 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5582d7f449f0 msgr2=0x5582d7f4ecb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:16:20.325 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:20.319+0000 7f34afe00640 1 --2- 172.21.15.118:0/2513807091 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5582d7f449f0 0x5582d7f4ecb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:16:20.325 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:20.319+0000 7f34afe00640 1 -- 172.21.15.118:0/2513807091 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5582d7f46580 msgr2=0x5582d7f56230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:16:20.325 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:20.319+0000 7f34afe00640 1 --2- 172.21.15.118:0/2513807091 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5582d7f46580 0x5582d7f56230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:16:20.325 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:20.319+0000 7f34afe00640 1 -- 172.21.15.118:0/2513807091 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5582d7bbd3f0 con 0x5582d7f45bc0 2024-03-24T00:16:20.325 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:20.320+0000 7f34afe00640 1 --2- 172.21.15.118:0/2513807091 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5582d7f45bc0 0x5582d7f4f1f0 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7f34a0013790 tx=0x7f34a0013c50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=1b03e0ed4b38ea84 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:16:20.325 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:20.320+0000 7f34a4c00640 10 client.0 ms_handle_connect on v2:172.21.15.189:3300/0 2024-03-24T00:16:20.325 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:20.320+0000 7f34a4c00640 1 -- 172.21.15.118:0/2513807091 <== mon.2 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f34a0020060 con 0x5582d7f45bc0 2024-03-24T00:16:20.325 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:20.320+0000 7f34a4c00640 1 -- 172.21.15.118:0/2513807091 <== mon.2 v2:172.21.15.189:3300/0 2 ==== osd_map(90..90 src has 1..90) v4 ==== 8277+0+0 (secure 0 0 0) 0x7f34a0011dd0 con 0x5582d7f45bc0 2024-03-24T00:16:20.325 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:20.320+0000 7f34c103a740 1 -- 172.21.15.118:0/2513807091 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f34b0015070 con 0x5582d7f45bc0 2024-03-24T00:16:20.326 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:20.320+0000 7f34a4c00640 1 -- 172.21.15.118:0/2513807091 <== mon.2 v2:172.21.15.189:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) v2 ==== 24+0+0 (secure 0 0 0) 0x7f34a000f450 con 0x5582d7f45bc0 2024-03-24T00:16:20.326 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:20.320+0000 7f34c103a740 10 client.25535 fetch_fsmap learned FSMap version 28 2024-03-24T00:16:20.326 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:20.320+0000 7f34c103a740 1 -- 172.21.15.118:0/2513807091 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) v3 -- 0x5582d7c877d0 con 0x5582d7f45bc0 2024-03-24T00:16:20.326 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:20.321+0000 7f34a4c00640 1 -- 172.21.15.118:0/2513807091 <== mon.2 v2:172.21.15.189:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f34b0015070 con 0x5582d7f45bc0 2024-03-24T00:16:20.326 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:20.321+0000 7f34a4c00640 1 -- 172.21.15.118:0/2513807091 <== mon.2 v2:172.21.15.189:3300/0 5 ==== fsmap(e 28) v1 ==== 7934+0+0 (secure 0 0 0) 0x7f34a001a030 con 0x5582d7f45bc0 2024-03-24T00:16:20.326 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:20.321+0000 7f34c103a740 10 client.25535 fetch_fsmap finished waiting for FSMap version 28 2024-03-24T00:16:20.326 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:20.321+0000 7f34c103a740 10 client.25535 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.g 2024-03-24T00:16:20.326 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:20.321+0000 7f34c103a740 20 client.25535 populate_metadata read hostname 'smithi118' 2024-03-24T00:16:20.326 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:20.321+0000 7f34c103a740 1 --2- 172.21.15.118:0/2513807091 >> [v2:172.21.15.118:6838/3285788503,v1:172.21.15.118:6839/3285788503] conn(0x5582d7f5a820 0x5582d7f5cc40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:16:20.326 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:20.321+0000 7f34c103a740 4 client.25535 mds_command: new command op to 24550 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-24T00:16:20.326 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:20.321+0000 7f34c103a740 1 -- 172.21.15.118:0/2513807091 --> [v2:172.21.15.118:6838/3285788503,v1:172.21.15.118:6839/3285788503] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x5582d7eb6800 con 0x5582d7f5a820 2024-03-24T00:16:20.326 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:20.321+0000 7f34b7e00640 1 --2- 172.21.15.118:0/2513807091 >> [v2:172.21.15.118:6838/3285788503,v1:172.21.15.118:6839/3285788503] conn(0x5582d7f5a820 0x5582d7f5cc40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:16:20.326 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:20.322+0000 7f34b7e00640 1 --2- 172.21.15.118:0/2513807091 >> [v2:172.21.15.118:6838/3285788503,v1:172.21.15.118:6839/3285788503] conn(0x5582d7f5a820 0x5582d7f5cc40 crc :-1 s=READY pgs=22 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=32f378b0d1242dba server_cookie=f3baab8279bd275d in_seq=0 out_seq=0 2024-03-24T00:16:20.326 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:20.322+0000 7f34a4c00640 10 client.25535 ms_handle_connect on v2:172.21.15.118:6838/3285788503 2024-03-24T00:16:20.328 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:20.322+0000 7f34a4c00640 1 -- 172.21.15.118:0/2513807091 <== mds.1 v2:172.21.15.118:6838/3285788503 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x5582d7eb6800 con 0x5582d7f5a820 2024-03-24T00:16:20.328 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:20.322+0000 7f34a4c00640 10 client.25535 handle_command_reply: tid=0 2024-03-24T00:16:20.328 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:20.322+0000 7f34c103a740 1 client.25535 shutdown 2024-03-24T00:16:20.328 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:20.323+0000 7f34c103a740 1 -- 172.21.15.118:0/2513807091 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5582d7f45bc0 msgr2=0x5582d7f4f1f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:16:20.328 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:20.323+0000 7f34c103a740 1 --2- 172.21.15.118:0/2513807091 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5582d7f45bc0 0x5582d7f4f1f0 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7f34a0013790 tx=0x7f34a0013c50 comp rx=0 tx=0).stop 2024-03-24T00:16:20.328 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:20.323+0000 7f34c103a740 1 -- 172.21.15.118:0/2513807091 shutdown_connections 2024-03-24T00:16:20.328 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:20.323+0000 7f34c103a740 1 --2- 172.21.15.118:0/2513807091 >> [v2:172.21.15.118:6838/3285788503,v1:172.21.15.118:6839/3285788503] conn(0x5582d7f5a820 0x5582d7f5cc40 crc :-1 s=READY pgs=22 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:16:20.328 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:20.323+0000 7f34c103a740 1 --2- 172.21.15.118:0/2513807091 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5582d7f46580 0x5582d7f56230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:16:20.328 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:20.323+0000 7f34c103a740 1 --2- 172.21.15.118:0/2513807091 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5582d7f45bc0 0x5582d7f4f1f0 unknown :-1 s=CLOSED pgs=234 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:16:20.328 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:20.323+0000 7f34a4c00640 0 client.25535 ms_handle_reset on v2:172.21.15.118:6838/3285788503 2024-03-24T00:16:20.328 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:20.323+0000 7f34c103a740 1 --2- 172.21.15.118:0/2513807091 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5582d7f449f0 0x5582d7f4ecb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:16:20.328 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:20.323+0000 7f34c103a740 1 -- 172.21.15.118:0/2513807091 >> 172.21.15.118:0/2513807091 conn(0x5582d7f3fde0 msgr2=0x5582d7f3f160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:16:20.328 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:20.323+0000 7f34c103a740 1 -- 172.21.15.118:0/2513807091 shutdown_connections 2024-03-24T00:16:20.328 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:20.323+0000 7f34c103a740 1 -- 172.21.15.118:0/2513807091 wait complete. 2024-03-24T00:16:20.329 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:20.323+0000 7f34c103a740 20 client.25535 trim_cache size 0 max 16384 2024-03-24T00:16:20.336 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:20.336+0000 7f34c103a740 1 Processor -- start 2024-03-24T00:16:20.337 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:20.336+0000 7f34c103a740 1 -- start start 2024-03-24T00:16:20.337 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:20.337+0000 7f34c103a740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5582d7f449f0 0x5582d7f3aff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:16:20.337 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:20.337+0000 7f34c103a740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5582d7f45bc0 0x5582d7f561f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:16:20.337 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:20.337+0000 7f34c103a740 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5582d7f46580 0x5582d7f48c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:16:20.337 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:20.337+0000 7f34c103a740 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f34b81a87d0 con 0x5582d7f46580 2024-03-24T00:16:20.337 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:20.337+0000 7f34c103a740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f34b81aa150 con 0x5582d7f45bc0 2024-03-24T00:16:20.337 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:20.337+0000 7f34c103a740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f34b81a8e30 con 0x5582d7f449f0 2024-03-24T00:16:20.337 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:20.337+0000 7f34afe00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5582d7f45bc0 0x5582d7f561f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:16:20.337 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:20.337+0000 7f34b7e00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5582d7f449f0 0x5582d7f3aff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:16:20.337 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:20.337+0000 7f34afe00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5582d7f45bc0 0x5582d7f561f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.118:60374/0 (socket says 172.21.15.118:60374) 2024-03-24T00:16:20.337 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:20.337+0000 7f34afe00640 1 -- 172.21.15.118:0/1282336328 learned_addr learned my addr 172.21.15.118:0/1282336328 (peer_addr_for_me v2:172.21.15.118:0/0) 2024-03-24T00:16:20.338 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:20.337+0000 7f34b7e00640 1 --2- 172.21.15.118:0/1282336328 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5582d7f46580 0x5582d7f48c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:16:20.340 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:20.337+0000 7f34afe00640 1 -- 172.21.15.118:0/1282336328 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5582d7f449f0 msgr2=0x5582d7f3aff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:16:20.340 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:20.337+0000 7f34afe00640 1 --2- 172.21.15.118:0/1282336328 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5582d7f449f0 0x5582d7f3aff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:16:20.340 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:20.337+0000 7f34afe00640 1 -- 172.21.15.118:0/1282336328 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5582d7f46580 msgr2=0x5582d7f48c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:16:20.340 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:20.337+0000 7f34afe00640 1 --2- 172.21.15.118:0/1282336328 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5582d7f46580 0x5582d7f48c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:16:20.340 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:20.337+0000 7f34afe00640 1 -- 172.21.15.118:0/1282336328 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5582d7f40d50 con 0x5582d7f45bc0 2024-03-24T00:16:20.340 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:20.338+0000 7f34afe00640 1 --2- 172.21.15.118:0/1282336328 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5582d7f45bc0 0x5582d7f561f0 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7f34a0000c00 tx=0x7f34a0026da0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ebfe7c48f12d0c4f server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:16:20.340 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:20.338+0000 7f34a7400640 1 -- 172.21.15.118:0/1282336328 <== mon.2 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f34a000fe60 con 0x5582d7f45bc0 2024-03-24T00:16:20.340 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:20.338+0000 7f34a7400640 1 -- 172.21.15.118:0/1282336328 <== mon.2 v2:172.21.15.189:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f34a0026410 con 0x5582d7f45bc0 2024-03-24T00:16:20.340 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:20.338+0000 7f34a7400640 1 -- 172.21.15.118:0/1282336328 <== mon.2 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f34a0005290 con 0x5582d7f45bc0 2024-03-24T00:16:20.340 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:20.339+0000 7f34c103a740 1 -- 172.21.15.118:0/1282336328 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5582d7f45bc0 msgr2=0x5582d7f561f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:16:20.340 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:20.339+0000 7f34c103a740 1 --2- 172.21.15.118:0/1282336328 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5582d7f45bc0 0x5582d7f561f0 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7f34a0000c00 tx=0x7f34a0026da0 comp rx=0 tx=0).stop 2024-03-24T00:16:20.340 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:20.339+0000 7f34c103a740 1 -- 172.21.15.118:0/1282336328 shutdown_connections 2024-03-24T00:16:20.341 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:20.339+0000 7f34c103a740 1 --2- 172.21.15.118:0/1282336328 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5582d7f46580 0x5582d7f48c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:16:20.341 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:20.339+0000 7f34c103a740 1 --2- 172.21.15.118:0/1282336328 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5582d7f45bc0 0x5582d7f561f0 unknown :-1 s=CLOSED pgs=235 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:16:20.341 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:20.339+0000 7f34c103a740 1 --2- 172.21.15.118:0/1282336328 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5582d7f449f0 0x5582d7f3aff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:16:20.341 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:20.339+0000 7f34c103a740 1 -- 172.21.15.118:0/1282336328 >> 172.21.15.118:0/1282336328 conn(0x5582d7f3fde0 msgr2=0x5582d7f438d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:16:20.341 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:20.339+0000 7f34c103a740 1 -- 172.21.15.118:0/1282336328 shutdown_connections 2024-03-24T00:16:20.341 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:20.339+0000 7f34c103a740 1 -- 172.21.15.118:0/1282336328 wait complete. 2024-03-24T00:16:20.341 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:20.339+0000 7f34c103a740 1 -- start start 2024-03-24T00:16:20.341 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:20.339+0000 7f34c103a740 1 Processor -- start 2024-03-24T00:16:20.341 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:20.340+0000 7f34c103a740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5582d7f449f0 0x5582d7f55be0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:16:20.341 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:20.340+0000 7f34c103a740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5582d7f45bc0 0x5582d7f56120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:16:20.341 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:20.340+0000 7f34c103a740 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5582d7f46580 0x5582d7f526a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:16:20.341 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:20.340+0000 7f34c103a740 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f34b81a9af0 con 0x5582d7f46580 2024-03-24T00:16:20.341 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:20.340+0000 7f34c103a740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f34b81a9490 con 0x5582d7f449f0 2024-03-24T00:16:20.341 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:20.340+0000 7f34c103a740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f34b81a6990 con 0x5582d7f45bc0 2024-03-24T00:16:20.341 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:20.340+0000 7f34c103a740 10 client.0 _ll_register_callbacks cb 0x7f34a0009630 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T00:16:20.341 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:20.340+0000 7f34b7e00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5582d7f449f0 0x5582d7f55be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:16:20.342 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:20.340+0000 7f34b7e00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5582d7f449f0 0x5582d7f55be0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.118:60384/0 (socket says 172.21.15.118:60384) 2024-03-24T00:16:20.342 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:20.340+0000 7f34b7e00640 1 -- 172.21.15.118:0/4180626625 learned_addr learned my addr 172.21.15.118:0/4180626625 (peer_addr_for_me v2:172.21.15.118:0/0) 2024-03-24T00:16:20.342 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:20.340+0000 7f34afe00640 1 --2- 172.21.15.118:0/4180626625 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5582d7f45bc0 0x5582d7f56120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:16:20.342 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:20.340+0000 7f34b7e00640 1 --2- 172.21.15.118:0/4180626625 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5582d7f46580 0x5582d7f526a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:16:20.342 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:20.340+0000 7f34b7e00640 1 -- 172.21.15.118:0/4180626625 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5582d7f45bc0 msgr2=0x5582d7f56120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:16:20.342 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:20.340+0000 7f34b7e00640 1 --2- 172.21.15.118:0/4180626625 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5582d7f45bc0 0x5582d7f56120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:16:20.342 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:20.340+0000 7f34b7e00640 1 -- 172.21.15.118:0/4180626625 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5582d7f46580 msgr2=0x5582d7f526a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:16:20.342 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:20.340+0000 7f34b7e00640 1 --2- 172.21.15.118:0/4180626625 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5582d7f46580 0x5582d7f526a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:16:20.342 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:20.340+0000 7f34b7e00640 1 -- 172.21.15.118:0/4180626625 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5582d7f52ed0 con 0x5582d7f449f0 2024-03-24T00:16:20.344 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:20.341+0000 7f34b7e00640 1 --2- 172.21.15.118:0/4180626625 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5582d7f449f0 0x5582d7f55be0 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7f34b0006010 tx=0x7f34b0003eb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=6fcfc933a6073e52 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:16:20.344 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:20.341+0000 7f34a4c00640 10 client.0 ms_handle_connect on v2:172.21.15.189:3300/0 2024-03-24T00:16:20.344 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:20.341+0000 7f34a4c00640 1 -- 172.21.15.118:0/4180626625 <== mon.2 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f34b0023040 con 0x5582d7f449f0 2024-03-24T00:16:20.344 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:20.341+0000 7f34a4c00640 1 -- 172.21.15.118:0/4180626625 <== mon.2 v2:172.21.15.189:3300/0 2 ==== osd_map(90..90 src has 1..90) v4 ==== 8277+0+0 (secure 0 0 0) 0x7f34b002a9d0 con 0x5582d7f449f0 2024-03-24T00:16:20.344 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:20.341+0000 7f34c103a740 1 -- 172.21.15.118:0/4180626625 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f34a0026410 con 0x5582d7f449f0 2024-03-24T00:16:20.345 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:20.341+0000 7f34a4c00640 1 -- 172.21.15.118:0/4180626625 <== mon.2 v2:172.21.15.189:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) v2 ==== 24+0+0 (secure 0 0 0) 0x7f34b002abd0 con 0x5582d7f449f0 2024-03-24T00:16:20.345 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:20.341+0000 7f34c103a740 10 client.25541 fetch_fsmap learned FSMap version 28 2024-03-24T00:16:20.345 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:20.341+0000 7f34c103a740 1 -- 172.21.15.118:0/4180626625 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) v3 -- 0x5582d7fb6e80 con 0x5582d7f449f0 2024-03-24T00:16:20.345 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:20.341+0000 7f34a4c00640 1 -- 172.21.15.118:0/4180626625 <== mon.2 v2:172.21.15.189:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f34a0026410 con 0x5582d7f449f0 2024-03-24T00:16:20.345 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:20.341+0000 7f34a4c00640 1 -- 172.21.15.118:0/4180626625 <== mon.2 v2:172.21.15.189:3300/0 5 ==== fsmap(e 28) v1 ==== 7934+0+0 (secure 0 0 0) 0x7f34b001a600 con 0x5582d7f449f0 2024-03-24T00:16:20.345 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:20.341+0000 7f34c103a740 10 client.25541 fetch_fsmap finished waiting for FSMap version 28 2024-03-24T00:16:20.345 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:20.341+0000 7f34c103a740 10 client.25541 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.g 2024-03-24T00:16:20.345 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:20.341+0000 7f34c103a740 20 client.25541 populate_metadata read hostname 'smithi118' 2024-03-24T00:16:20.345 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:20.342+0000 7f34c103a740 1 --2- 172.21.15.118:0/4180626625 >> [v2:172.21.15.118:6838/3285788503,v1:172.21.15.118:6839/3285788503] conn(0x5582d7fb7640 0x5582d7fb9a60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:16:20.345 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:20.342+0000 7f34c103a740 4 client.25541 mds_command: new command op to 24550 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-24T00:16:20.345 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:20.342+0000 7f34c103a740 1 -- 172.21.15.118:0/4180626625 --> [v2:172.21.15.118:6838/3285788503,v1:172.21.15.118:6839/3285788503] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x5582d7eb0b50 con 0x5582d7fb7640 2024-03-24T00:16:20.345 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:20.342+0000 7f34afe00640 1 --2- 172.21.15.118:0/4180626625 >> [v2:172.21.15.118:6838/3285788503,v1:172.21.15.118:6839/3285788503] conn(0x5582d7fb7640 0x5582d7fb9a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:16:20.345 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:20.342+0000 7f34afe00640 1 --2- 172.21.15.118:0/4180626625 >> [v2:172.21.15.118:6838/3285788503,v1:172.21.15.118:6839/3285788503] conn(0x5582d7fb7640 0x5582d7fb9a60 crc :-1 s=READY pgs=23 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=882f44cd0de1217e server_cookie=1adc84ef9bc59a95 in_seq=0 out_seq=0 2024-03-24T00:16:20.345 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:20.342+0000 7f34a4c00640 10 client.25541 ms_handle_connect on v2:172.21.15.118:6838/3285788503 2024-03-24T00:16:20.345 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:20.342+0000 7f34a4c00640 1 -- 172.21.15.118:0/4180626625 <== mds.1 v2:172.21.15.118:6838/3285788503 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x5582d7eb0b50 con 0x5582d7fb7640 2024-03-24T00:16:20.346 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:20.342+0000 7f34a4c00640 10 client.25541 handle_command_reply: tid=0 2024-03-24T00:16:20.346 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:20.342+0000 7f34c103a740 1 client.25541 shutdown 2024-03-24T00:16:20.346 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:20.343+0000 7f34c103a740 1 -- 172.21.15.118:0/4180626625 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5582d7f449f0 msgr2=0x5582d7f55be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:16:20.346 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:20.343+0000 7f34c103a740 1 --2- 172.21.15.118:0/4180626625 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5582d7f449f0 0x5582d7f55be0 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7f34b0006010 tx=0x7f34b0003eb0 comp rx=0 tx=0).stop 2024-03-24T00:16:20.346 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:20.343+0000 7f34c103a740 1 -- 172.21.15.118:0/4180626625 shutdown_connections 2024-03-24T00:16:20.346 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:20.343+0000 7f34c103a740 1 --2- 172.21.15.118:0/4180626625 >> [v2:172.21.15.118:6838/3285788503,v1:172.21.15.118:6839/3285788503] conn(0x5582d7fb7640 0x5582d7fb9a60 crc :-1 s=READY pgs=23 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:16:20.346 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:20.343+0000 7f34c103a740 1 --2- 172.21.15.118:0/4180626625 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5582d7f46580 0x5582d7f526a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:16:20.346 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:20.343+0000 7f34c103a740 1 --2- 172.21.15.118:0/4180626625 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5582d7f45bc0 0x5582d7f56120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:16:20.346 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:20.343+0000 7f34a4c00640 0 client.25541 ms_handle_reset on v2:172.21.15.118:6838/3285788503 2024-03-24T00:16:20.346 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:20.343+0000 7f34c103a740 1 --2- 172.21.15.118:0/4180626625 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5582d7f449f0 0x5582d7f55be0 unknown :-1 s=CLOSED pgs=236 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:16:20.346 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:20.343+0000 7f34c103a740 1 -- 172.21.15.118:0/4180626625 >> 172.21.15.118:0/4180626625 conn(0x5582d7f3fde0 msgr2=0x5582d7f3e8d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:16:20.346 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:20.343+0000 7f34c103a740 1 -- 172.21.15.118:0/4180626625 shutdown_connections 2024-03-24T00:16:20.346 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:20.343+0000 7f34c103a740 1 -- 172.21.15.118:0/4180626625 wait complete. 2024-03-24T00:16:20.346 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:20.343+0000 7f34c103a740 20 client.25541 trim_cache size 0 max 16384 2024-03-24T00:16:20.347 INFO:teuthology.orchestra.run.smithi118.stdout:[] 2024-03-24T00:16:20.348 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:20.344+0000 7f34bcc00640 1 -- 172.21.15.118:0/524017738 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7f349c0a2910 msgr2=0x7f349c0a4dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:16:20.348 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:20.344+0000 7f34bcc00640 1 --2- 172.21.15.118:0/524017738 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7f349c0a2910 0x7f349c0a4dd0 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7f34a80c57a0 tx=0x7f34a80f4720 comp rx=0 tx=0).stop 2024-03-24T00:16:20.348 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:20.344+0000 7f34bcc00640 1 -- 172.21.15.118:0/524017738 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f34b81015a0 msgr2=0x7f34b8197910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:16:20.348 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:20.344+0000 7f34bcc00640 1 --2- 172.21.15.118:0/524017738 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f34b81015a0 0x7f34b8197910 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7f34a8000c00 tx=0x7f34a802c6c0 comp rx=0 tx=0).stop 2024-03-24T00:16:20.348 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:20.344+0000 7f34bcc00640 1 -- 172.21.15.118:0/524017738 shutdown_connections 2024-03-24T00:16:20.348 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:20.344+0000 7f34bcc00640 1 --2- 172.21.15.118:0/524017738 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7f349c0a2910 0x7f349c0a4dd0 unknown :-1 s=CLOSED pgs=278 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:16:20.348 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:20.344+0000 7f34bcc00640 1 --2- 172.21.15.118:0/524017738 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f34b8103150 0x7f34b819eec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:16:20.348 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:20.344+0000 7f34bcc00640 1 --2- 172.21.15.118:0/524017738 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f34b8102790 0x7f34b8197e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:16:20.348 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:20.344+0000 7f34bcc00640 1 --2- 172.21.15.118:0/524017738 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f34b81015a0 0x7f34b8197910 unknown :-1 s=CLOSED pgs=232 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:16:20.348 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:20.344+0000 7f34bcc00640 1 -- 172.21.15.118:0/524017738 >> 172.21.15.118:0/524017738 conn(0x7f34b80fcb10 msgr2=0x7f34b8105b00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:16:20.348 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:20.344+0000 7f34bcc00640 1 -- 172.21.15.118:0/524017738 shutdown_connections 2024-03-24T00:16:20.348 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:20.344+0000 7f34bcc00640 1 -- 172.21.15.118:0/524017738 wait complete. 2024-03-24T00:16:20.364 INFO:teuthology.orchestra.run:Running command with timeout 120 2024-03-24T00:16:20.364 DEBUG:teuthology.orchestra.run.smithi118:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2024-03-24T00:16:20.566 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:20.566+0000 7fb661800640 1 -- 172.21.15.118:0/2597840626 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fb65c1038e0 msgr2=0x7fb65c103d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:16:20.566 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:20.566+0000 7fb661800640 1 --2- 172.21.15.118:0/2597840626 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fb65c1038e0 0x7fb65c103d00 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7fb64c009900 tx=0x7fb64c02bc30 comp rx=0 tx=0).stop 2024-03-24T00:16:20.566 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:20.566+0000 7fb661800640 1 -- 172.21.15.118:0/2597840626 shutdown_connections 2024-03-24T00:16:20.566 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:20.566+0000 7fb661800640 1 --2- 172.21.15.118:0/2597840626 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb65c105490 0x7fb65c10bd20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:16:20.566 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:20.566+0000 7fb661800640 1 --2- 172.21.15.118:0/2597840626 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb65c104ad0 0x7fb65c104f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:16:20.567 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:20.566+0000 7fb661800640 1 --2- 172.21.15.118:0/2597840626 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fb65c1038e0 0x7fb65c103d00 unknown :-1 s=CLOSED pgs=220 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:16:20.567 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:20.566+0000 7fb661800640 1 -- 172.21.15.118:0/2597840626 >> 172.21.15.118:0/2597840626 conn(0x7fb65c0fee50 msgr2=0x7fb65c101290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:16:20.567 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:20.566+0000 7fb661800640 1 -- 172.21.15.118:0/2597840626 shutdown_connections 2024-03-24T00:16:20.567 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:20.566+0000 7fb661800640 1 -- 172.21.15.118:0/2597840626 wait complete. 2024-03-24T00:16:20.569 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:20.567+0000 7fb661800640 1 Processor -- start 2024-03-24T00:16:20.569 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:20.568+0000 7fb661800640 1 -- start start 2024-03-24T00:16:20.569 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:20.568+0000 7fb661800640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fb65c1038e0 0x7fb65c199c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:16:20.569 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:20.568+0000 7fb661800640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb65c104ad0 0x7fb65c19a1b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:16:20.569 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:20.568+0000 7fb661800640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb65c105490 0x7fb65c1a11e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:16:20.569 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:20.568+0000 7fb661800640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb65c10f580 con 0x7fb65c105490 2024-03-24T00:16:20.569 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:20.568+0000 7fb661800640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb65c10f400 con 0x7fb65c104ad0 2024-03-24T00:16:20.569 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:20.568+0000 7fb661800640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb65c10f700 con 0x7fb65c1038e0 2024-03-24T00:16:20.569 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:20.568+0000 7fb65aa00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fb65c1038e0 0x7fb65c199c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:16:20.570 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:20.568+0000 7fb65aa00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fb65c1038e0 0x7fb65c199c70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.118:45624/0 (socket says 172.21.15.118:45624) 2024-03-24T00:16:20.570 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:20.568+0000 7fb65aa00640 1 -- 172.21.15.118:0/479850798 learned_addr learned my addr 172.21.15.118:0/479850798 (peer_addr_for_me v2:172.21.15.118:0/0) 2024-03-24T00:16:20.570 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:20.568+0000 7fb653e00640 1 --2- 172.21.15.118:0/479850798 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb65c104ad0 0x7fb65c19a1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:16:20.570 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:20.568+0000 7fb65b400640 1 --2- 172.21.15.118:0/479850798 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb65c105490 0x7fb65c1a11e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:16:20.570 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:20.569+0000 7fb65aa00640 1 -- 172.21.15.118:0/479850798 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb65c104ad0 msgr2=0x7fb65c19a1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:16:20.570 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:20.569+0000 7fb65aa00640 1 --2- 172.21.15.118:0/479850798 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb65c104ad0 0x7fb65c19a1b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:16:20.570 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:20.569+0000 7fb65aa00640 1 -- 172.21.15.118:0/479850798 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb65c105490 msgr2=0x7fb65c1a11e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:16:20.570 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:20.569+0000 7fb65aa00640 1 --2- 172.21.15.118:0/479850798 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb65c105490 0x7fb65c1a11e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:16:20.570 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:20.569+0000 7fb65aa00640 1 -- 172.21.15.118:0/479850798 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb65c1a1880 con 0x7fb65c1038e0 2024-03-24T00:16:20.570 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:20.569+0000 7fb65b400640 1 --2- 172.21.15.118:0/479850798 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb65c105490 0x7fb65c1a11e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T00:16:20.570 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:20.569+0000 7fb653e00640 1 --2- 172.21.15.118:0/479850798 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb65c104ad0 0x7fb65c19a1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T00:16:20.570 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:20.569+0000 7fb65aa00640 1 --2- 172.21.15.118:0/479850798 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fb65c1038e0 0x7fb65c199c70 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7fb64c000c00 tx=0x7fb64c02c650 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:16:20.570 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:20.569+0000 7fb653400640 1 -- 172.21.15.118:0/479850798 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb64c043020 con 0x7fb65c1038e0 2024-03-24T00:16:20.570 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:20.569+0000 7fb653400640 1 -- 172.21.15.118:0/479850798 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fb64c0054e0 con 0x7fb65c1038e0 2024-03-24T00:16:20.571 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:20.569+0000 7fb653400640 1 -- 172.21.15.118:0/479850798 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb64c003c40 con 0x7fb65c1038e0 2024-03-24T00:16:20.571 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:20.569+0000 7fb661800640 1 -- 172.21.15.118:0/479850798 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb65c1a1b10 con 0x7fb65c1038e0 2024-03-24T00:16:20.571 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:20.569+0000 7fb661800640 1 -- 172.21.15.118:0/479850798 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb65c1a2010 con 0x7fb65c1038e0 2024-03-24T00:16:20.572 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:20.572+0000 7fb653400640 1 -- 172.21.15.118:0/479850798 <== mon.1 v2:172.21.15.193:3300/0 4 ==== mgrmap(e 19) v1 ==== 137342+0+0 (secure 0 0 0) 0x7fb64c046020 con 0x7fb65c1038e0 2024-03-24T00:16:20.572 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:20.572+0000 7fb6645c0740 1 Processor -- start 2024-03-24T00:16:20.572 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:20.573+0000 7fb6645c0740 1 -- start start 2024-03-24T00:16:20.574 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:20.573+0000 7fb6645c0740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x562fb545d810 0x562fb545dc30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:16:20.574 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:20.573+0000 7fb6645c0740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x562fb545e9e0 0x562fb545ee60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:16:20.574 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:20.573+0000 7fb6645c0740 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x562fb545f3a0 0x562fb5465c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:16:20.574 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:20.573+0000 7fb6645c0740 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x562fb53c22c0 con 0x562fb545f3a0 2024-03-24T00:16:20.574 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:20.573+0000 7fb6645c0740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x562fb536ac90 con 0x562fb545d810 2024-03-24T00:16:20.574 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:20.573+0000 7fb6645c0740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x562fb53be360 con 0x562fb545e9e0 2024-03-24T00:16:20.574 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:20.573+0000 7fb65b400640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x562fb545d810 0x562fb545dc30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:16:20.574 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:20.573+0000 7fb65b400640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x562fb545d810 0x562fb545dc30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.118:60414/0 (socket says 172.21.15.118:60414) 2024-03-24T00:16:20.574 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:20.573+0000 7fb65aa00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x562fb545e9e0 0x562fb545ee60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:16:20.574 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:20.573+0000 7fb65b400640 1 -- 172.21.15.118:0/2659038011 learned_addr learned my addr 172.21.15.118:0/2659038011 (peer_addr_for_me v2:172.21.15.118:0/0) 2024-03-24T00:16:20.575 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:20.573+0000 7fb653e00640 1 --2- 172.21.15.118:0/2659038011 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x562fb545f3a0 0x562fb5465c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:16:20.575 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:20.574+0000 7fb65b400640 1 -- 172.21.15.118:0/2659038011 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x562fb545e9e0 msgr2=0x562fb545ee60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:16:20.575 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:20.574+0000 7fb65b400640 1 --2- 172.21.15.118:0/2659038011 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x562fb545e9e0 0x562fb545ee60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:16:20.575 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:20.574+0000 7fb65b400640 1 -- 172.21.15.118:0/2659038011 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x562fb545f3a0 msgr2=0x562fb5465c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:16:20.575 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:20.574+0000 7fb65b400640 1 --2- 172.21.15.118:0/2659038011 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x562fb545f3a0 0x562fb5465c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:16:20.575 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:20.574+0000 7fb65b400640 1 -- 172.21.15.118:0/2659038011 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x562fb538f080 con 0x562fb545d810 2024-03-24T00:16:20.575 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:20.574+0000 7fb653400640 1 --2- 172.21.15.118:0/479850798 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7fb6400a2910 0x7fb6400a4dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:16:20.575 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:20.574+0000 7fb65b400640 1 --2- 172.21.15.118:0/2659038011 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x562fb545d810 0x562fb545dc30 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7fb654010450 tx=0x7fb654010910 comp rx=0 tx=0).ready entity=mon.2 client_cookie=7dfb79b499fa96cb server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:16:20.575 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:20.574+0000 7fb653400640 1 -- 172.21.15.118:0/479850798 <== mon.1 v2:172.21.15.193:3300/0 5 ==== osd_map(90..90 src has 1..90) v4 ==== 8277+0+0 (secure 0 0 0) 0x7fb64c0f0c10 con 0x7fb65c1038e0 2024-03-24T00:16:20.575 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:20.574+0000 7fb65aa00640 1 --2- 172.21.15.118:0/479850798 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7fb6400a2910 0x7fb6400a4dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:16:20.575 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:20.574+0000 7fb64be00640 1 -- 172.21.15.118:0/2659038011 <== mon.2 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb654017070 con 0x562fb545d810 2024-03-24T00:16:20.575 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:20.574+0000 7fb64be00640 1 -- 172.21.15.118:0/2659038011 <== mon.2 v2:172.21.15.189:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fb654011c40 con 0x562fb545d810 2024-03-24T00:16:20.575 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:20.574+0000 7fb64be00640 1 -- 172.21.15.118:0/2659038011 <== mon.2 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb65401adb0 con 0x562fb545d810 2024-03-24T00:16:20.579 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:20.575+0000 7fb6645c0740 1 -- 172.21.15.118:0/2659038011 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x562fb545d810 msgr2=0x562fb545dc30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:16:20.579 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:20.575+0000 7fb65aa00640 1 --2- 172.21.15.118:0/479850798 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7fb6400a2910 0x7fb6400a4dd0 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7fb64c0c5740 tx=0x7fb64c0b9040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:16:20.579 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:20.575+0000 7fb6645c0740 1 --2- 172.21.15.118:0/2659038011 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x562fb545d810 0x562fb545dc30 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7fb654010450 tx=0x7fb654010910 comp rx=0 tx=0).stop 2024-03-24T00:16:20.579 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:20.575+0000 7fb6645c0740 1 -- 172.21.15.118:0/2659038011 shutdown_connections 2024-03-24T00:16:20.579 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:20.575+0000 7fb6645c0740 1 --2- 172.21.15.118:0/2659038011 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x562fb545f3a0 0x562fb5465c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:16:20.579 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:20.575+0000 7fb6645c0740 1 --2- 172.21.15.118:0/2659038011 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x562fb545e9e0 0x562fb545ee60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:16:20.579 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:20.575+0000 7fb6645c0740 1 --2- 172.21.15.118:0/2659038011 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x562fb545d810 0x562fb545dc30 unknown :-1 s=CLOSED pgs=237 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:16:20.579 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:20.575+0000 7fb6645c0740 1 -- 172.21.15.118:0/2659038011 >> 172.21.15.118:0/2659038011 conn(0x562fb5458c00 msgr2=0x562fb5459000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:16:20.579 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:20.575+0000 7fb6645c0740 1 -- 172.21.15.118:0/2659038011 shutdown_connections 2024-03-24T00:16:20.580 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:20.575+0000 7fb6645c0740 1 -- 172.21.15.118:0/2659038011 wait complete. 2024-03-24T00:16:20.580 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:20.576+0000 7fb6645c0740 1 -- start start 2024-03-24T00:16:20.580 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:20.576+0000 7fb6645c0740 1 Processor -- start 2024-03-24T00:16:20.580 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:20.577+0000 7fb6645c0740 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x562fb545d810 0x562fb5467ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:16:20.580 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:20.577+0000 7fb6645c0740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x562fb545e9e0 0x562fb5467ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:16:20.580 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:20.577+0000 7fb6645c0740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x562fb545f3a0 0x562fb546f030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:16:20.580 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:20.577+0000 7fb6645c0740 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x562fb54663b0 con 0x562fb545d810 2024-03-24T00:16:20.580 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:20.577+0000 7fb6645c0740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x562fb537e0d0 con 0x562fb545f3a0 2024-03-24T00:16:20.580 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:20.577+0000 7fb6645c0740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x562fb5466530 con 0x562fb545e9e0 2024-03-24T00:16:20.580 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:20.577+0000 7fb6645c0740 10 client.0 _ll_register_callbacks cb 0x562fb5382e80 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T00:16:20.580 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:20.577+0000 7fb653e00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x562fb545e9e0 0x562fb5467ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:16:20.580 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:20.577+0000 7fb65b400640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x562fb545d810 0x562fb5467ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:16:20.580 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:20.577+0000 7fb653e00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x562fb545e9e0 0x562fb5467ff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.118:45646/0 (socket says 172.21.15.118:45646) 2024-03-24T00:16:20.580 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:20.577+0000 7fb653e00640 1 -- 172.21.15.118:0/4216007428 learned_addr learned my addr 172.21.15.118:0/4216007428 (peer_addr_for_me v2:172.21.15.118:0/0) 2024-03-24T00:16:20.581 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:20.577+0000 7fb65b400640 1 --2- 172.21.15.118:0/4216007428 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x562fb545f3a0 0x562fb546f030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:16:20.581 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:20.577+0000 7fb653e00640 1 -- 172.21.15.118:0/4216007428 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x562fb545f3a0 msgr2=0x562fb546f030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:16:20.581 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:20.577+0000 7fb653e00640 1 --2- 172.21.15.118:0/4216007428 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x562fb545f3a0 0x562fb546f030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:16:20.581 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:20.577+0000 7fb653e00640 1 -- 172.21.15.118:0/4216007428 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x562fb545d810 msgr2=0x562fb5467ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:16:20.581 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:20.577+0000 7fb653e00640 1 --2- 172.21.15.118:0/4216007428 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x562fb545d810 0x562fb5467ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:16:20.581 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:20.577+0000 7fb653e00640 1 -- 172.21.15.118:0/4216007428 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x562fb50d63f0 con 0x562fb545e9e0 2024-03-24T00:16:20.581 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:20.577+0000 7fb65b400640 1 --2- 172.21.15.118:0/4216007428 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x562fb545f3a0 0x562fb546f030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T00:16:20.581 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:20.578+0000 7fb653e00640 1 --2- 172.21.15.118:0/4216007428 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x562fb545e9e0 0x562fb5467ff0 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7fb644012d20 tx=0x7fb64400f790 comp rx=0 tx=0).ready entity=mon.1 client_cookie=8074545928cc3c08 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:16:20.581 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:20.578+0000 7fb648c00640 10 client.0 ms_handle_connect on v2:172.21.15.193:3300/0 2024-03-24T00:16:20.581 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:20.578+0000 7fb648c00640 1 -- 172.21.15.118:0/4216007428 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb644012110 con 0x562fb545e9e0 2024-03-24T00:16:20.581 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:20.578+0000 7fb6645c0740 1 -- 172.21.15.118:0/4216007428 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fb654011c40 con 0x562fb545e9e0 2024-03-24T00:16:20.581 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:20.578+0000 7fb648c00640 1 -- 172.21.15.118:0/4216007428 <== mon.1 v2:172.21.15.193:3300/0 2 ==== osd_map(90..90 src has 1..90) v4 ==== 8277+0+0 (secure 0 0 0) 0x7fb644004050 con 0x562fb545e9e0 2024-03-24T00:16:20.581 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:20.578+0000 7fb648c00640 1 -- 172.21.15.118:0/4216007428 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) v2 ==== 24+0+0 (secure 0 0 0) 0x7fb644022030 con 0x562fb545e9e0 2024-03-24T00:16:20.581 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:20.578+0000 7fb6645c0740 10 client.25555 fetch_fsmap learned FSMap version 28 2024-03-24T00:16:20.581 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:20.578+0000 7fb6645c0740 1 -- 172.21.15.118:0/4216007428 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) v3 -- 0x562fb51a07d0 con 0x562fb545e9e0 2024-03-24T00:16:20.581 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:20.579+0000 7fb648c00640 1 -- 172.21.15.118:0/4216007428 <== mon.1 v2:172.21.15.193:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fb654011c40 con 0x562fb545e9e0 2024-03-24T00:16:20.582 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:20.579+0000 7fb648c00640 1 -- 172.21.15.118:0/4216007428 <== mon.1 v2:172.21.15.193:3300/0 5 ==== fsmap(e 28) v1 ==== 7934+0+0 (secure 0 0 0) 0x7fb644027430 con 0x562fb545e9e0 2024-03-24T00:16:20.582 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:20.579+0000 7fb6645c0740 10 client.25555 fetch_fsmap finished waiting for FSMap version 28 2024-03-24T00:16:20.582 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:20.579+0000 7fb6645c0740 10 client.25555 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-24T00:16:20.582 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:20.579+0000 7fb6645c0740 20 client.25555 populate_metadata read hostname 'smithi118' 2024-03-24T00:16:20.582 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:20.579+0000 7fb6645c0740 1 --2- 172.21.15.118:0/4216007428 >> [v2:172.21.15.189:6838/4074484168,v1:172.21.15.189:6839/4074484168] conn(0x562fb5473640 0x562fb5475aa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:16:20.582 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:20.579+0000 7fb6645c0740 4 client.25555 mds_command: new command op to 24569 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-24T00:16:20.582 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:20.579+0000 7fb6645c0740 1 -- 172.21.15.118:0/4216007428 --> [v2:172.21.15.189:6838/4074484168,v1:172.21.15.189:6839/4074484168] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x562fb53cf800 con 0x562fb5473640 2024-03-24T00:16:20.582 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:20.579+0000 7fb65b400640 1 --2- 172.21.15.118:0/4216007428 >> [v2:172.21.15.189:6838/4074484168,v1:172.21.15.189:6839/4074484168] conn(0x562fb5473640 0x562fb5475aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:16:20.582 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:20.580+0000 7fb65b400640 1 --2- 172.21.15.118:0/4216007428 >> [v2:172.21.15.189:6838/4074484168,v1:172.21.15.189:6839/4074484168] conn(0x562fb5473640 0x562fb5475aa0 crc :-1 s=READY pgs=43 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=1a001c1a926a0b7b server_cookie=4953e1d59ed0da07 in_seq=0 out_seq=0 2024-03-24T00:16:20.582 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:20.580+0000 7fb648c00640 10 client.25555 ms_handle_connect on v2:172.21.15.189:6838/4074484168 2024-03-24T00:16:20.583 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:20.580+0000 7fb648c00640 1 -- 172.21.15.118:0/4216007428 <== mds.0 v2:172.21.15.189:6838/4074484168 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x562fb53cf800 con 0x562fb5473640 2024-03-24T00:16:20.583 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:20.580+0000 7fb648c00640 10 client.25555 handle_command_reply: tid=0 2024-03-24T00:16:20.584 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:20.580+0000 7fb6645c0740 1 client.25555 shutdown 2024-03-24T00:16:20.584 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:20.581+0000 7fb6645c0740 1 -- 172.21.15.118:0/4216007428 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x562fb545e9e0 msgr2=0x562fb5467ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:16:20.584 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:20.581+0000 7fb6645c0740 1 --2- 172.21.15.118:0/4216007428 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x562fb545e9e0 0x562fb5467ff0 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7fb644012d20 tx=0x7fb64400f790 comp rx=0 tx=0).stop 2024-03-24T00:16:20.584 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:20.581+0000 7fb6645c0740 1 -- 172.21.15.118:0/4216007428 shutdown_connections 2024-03-24T00:16:20.584 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:20.581+0000 7fb6645c0740 1 --2- 172.21.15.118:0/4216007428 >> [v2:172.21.15.189:6838/4074484168,v1:172.21.15.189:6839/4074484168] conn(0x562fb5473640 0x562fb5475aa0 crc :-1 s=READY pgs=43 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:16:20.584 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:20.581+0000 7fb6645c0740 1 --2- 172.21.15.118:0/4216007428 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x562fb545f3a0 0x562fb546f030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:16:20.584 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:20.581+0000 7fb6645c0740 1 --2- 172.21.15.118:0/4216007428 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x562fb545e9e0 0x562fb5467ff0 unknown :-1 s=CLOSED pgs=222 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:16:20.584 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:20.581+0000 7fb648c00640 0 client.25555 ms_handle_reset on v2:172.21.15.189:6838/4074484168 2024-03-24T00:16:20.584 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:20.581+0000 7fb6645c0740 1 --2- 172.21.15.118:0/4216007428 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x562fb545d810 0x562fb5467ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:16:20.584 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:20.581+0000 7fb6645c0740 1 -- 172.21.15.118:0/4216007428 >> 172.21.15.118:0/4216007428 conn(0x562fb5458c00 msgr2=0x562fb5457eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:16:20.584 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:20.581+0000 7fb6645c0740 1 -- 172.21.15.118:0/4216007428 shutdown_connections 2024-03-24T00:16:20.584 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:20.581+0000 7fb6645c0740 1 -- 172.21.15.118:0/4216007428 wait complete. 2024-03-24T00:16:20.584 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:20.581+0000 7fb6645c0740 20 client.25555 trim_cache size 0 max 16384 2024-03-24T00:16:20.607 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:20.599+0000 7fb6645c0740 1 Processor -- start 2024-03-24T00:16:20.607 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:20.599+0000 7fb6645c0740 1 -- start start 2024-03-24T00:16:20.607 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:20.599+0000 7fb6645c0740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x562fb545d810 0x562fb54619e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:16:20.607 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:20.599+0000 7fb6645c0740 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x562fb545e9e0 0x562fb546eff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:16:20.607 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:20.599+0000 7fb6645c0740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x562fb545f3a0 0x562fb5461f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:16:20.608 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:20.599+0000 7fb6645c0740 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb65c1aac40 con 0x562fb545e9e0 2024-03-24T00:16:20.608 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:20.599+0000 7fb6645c0740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb65c1ac5c0 con 0x562fb545d810 2024-03-24T00:16:20.608 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:20.599+0000 7fb6645c0740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb65c1ab2a0 con 0x562fb545f3a0 2024-03-24T00:16:20.608 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:20.599+0000 7fb65b400640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x562fb545d810 0x562fb54619e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:16:20.608 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:20.600+0000 7fb65b400640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x562fb545d810 0x562fb54619e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.118:60442/0 (socket says 172.21.15.118:60442) 2024-03-24T00:16:20.608 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:20.600+0000 7fb65b400640 1 -- 172.21.15.118:0/3600285373 learned_addr learned my addr 172.21.15.118:0/3600285373 (peer_addr_for_me v2:172.21.15.118:0/0) 2024-03-24T00:16:20.608 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:20.600+0000 7fb653e00640 1 --2- 172.21.15.118:0/3600285373 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x562fb545e9e0 0x562fb546eff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:16:20.608 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:20.600+0000 7fb65b400640 1 --2- 172.21.15.118:0/3600285373 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x562fb545f3a0 0x562fb5461f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:16:20.608 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:20.600+0000 7fb65b400640 1 -- 172.21.15.118:0/3600285373 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x562fb545f3a0 msgr2=0x562fb5461f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:16:20.608 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:20.600+0000 7fb65b400640 1 --2- 172.21.15.118:0/3600285373 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x562fb545f3a0 0x562fb5461f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:16:20.608 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:20.600+0000 7fb65b400640 1 -- 172.21.15.118:0/3600285373 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x562fb545e9e0 msgr2=0x562fb546eff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:16:20.608 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:20.600+0000 7fb65b400640 1 --2- 172.21.15.118:0/3600285373 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x562fb545e9e0 0x562fb546eff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:16:20.608 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:20.600+0000 7fb65b400640 1 -- 172.21.15.118:0/3600285373 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x562fb5459970 con 0x562fb545d810 2024-03-24T00:16:20.609 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:20.600+0000 7fb65b400640 1 --2- 172.21.15.118:0/3600285373 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x562fb545d810 0x562fb54619e0 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7fb654009540 tx=0x7fb654009940 comp rx=0 tx=0).ready entity=mon.2 client_cookie=647ac8bf60397385 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:16:20.609 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:20.600+0000 7fb64b400640 1 -- 172.21.15.118:0/3600285373 <== mon.2 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb654020040 con 0x562fb545d810 2024-03-24T00:16:20.609 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:20.600+0000 7fb64b400640 1 -- 172.21.15.118:0/3600285373 <== mon.2 v2:172.21.15.189:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fb654021040 con 0x562fb545d810 2024-03-24T00:16:20.609 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:20.600+0000 7fb64b400640 1 -- 172.21.15.118:0/3600285373 <== mon.2 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb654017040 con 0x562fb545d810 2024-03-24T00:16:20.609 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:20.601+0000 7fb6645c0740 1 -- 172.21.15.118:0/3600285373 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x562fb545d810 msgr2=0x562fb54619e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:16:20.609 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:20.601+0000 7fb6645c0740 1 --2- 172.21.15.118:0/3600285373 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x562fb545d810 0x562fb54619e0 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7fb654009540 tx=0x7fb654009940 comp rx=0 tx=0).stop 2024-03-24T00:16:20.609 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:20.601+0000 7fb6645c0740 1 -- 172.21.15.118:0/3600285373 shutdown_connections 2024-03-24T00:16:20.609 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:20.601+0000 7fb6645c0740 1 --2- 172.21.15.118:0/3600285373 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x562fb545f3a0 0x562fb5461f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:16:20.609 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:20.601+0000 7fb6645c0740 1 --2- 172.21.15.118:0/3600285373 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x562fb545e9e0 0x562fb546eff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:16:20.609 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:20.601+0000 7fb6645c0740 1 --2- 172.21.15.118:0/3600285373 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x562fb545d810 0x562fb54619e0 unknown :-1 s=CLOSED pgs=238 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:16:20.609 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:20.601+0000 7fb6645c0740 1 -- 172.21.15.118:0/3600285373 >> 172.21.15.118:0/3600285373 conn(0x562fb5458c00 msgr2=0x562fb5457eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:16:20.609 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:20.601+0000 7fb6645c0740 1 -- 172.21.15.118:0/3600285373 shutdown_connections 2024-03-24T00:16:20.609 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:20.601+0000 7fb6645c0740 1 -- 172.21.15.118:0/3600285373 wait complete. 2024-03-24T00:16:20.609 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:20.601+0000 7fb6645c0740 1 -- start start 2024-03-24T00:16:20.610 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:20.602+0000 7fb6645c0740 1 Processor -- start 2024-03-24T00:16:20.610 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:20.602+0000 7fb6645c0740 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x562fb545d810 0x562fb546e9a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:16:20.610 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:20.602+0000 7fb6645c0740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x562fb545e9e0 0x562fb546eee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:16:20.610 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:20.602+0000 7fb6645c0740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x562fb545f3a0 0x562fb546b440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:16:20.610 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:20.602+0000 7fb6645c0740 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb65c1abf60 con 0x562fb545d810 2024-03-24T00:16:20.610 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:20.602+0000 7fb6645c0740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb65c1ab900 con 0x562fb545f3a0 2024-03-24T00:16:20.610 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:20.602+0000 7fb6645c0740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb65c1a8e00 con 0x562fb545e9e0 2024-03-24T00:16:20.610 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:20.602+0000 7fb6645c0740 10 client.0 _ll_register_callbacks cb 0x7fb644009ce0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T00:16:20.610 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:20.602+0000 7fb653e00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x562fb545e9e0 0x562fb546eee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:16:20.610 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:20.602+0000 7fb653e00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x562fb545e9e0 0x562fb546eee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.118:45652/0 (socket says 172.21.15.118:45652) 2024-03-24T00:16:20.610 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:20.602+0000 7fb653e00640 1 -- 172.21.15.118:0/1591239962 learned_addr learned my addr 172.21.15.118:0/1591239962 (peer_addr_for_me v2:172.21.15.118:0/0) 2024-03-24T00:16:20.610 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:20.602+0000 7fb65b400640 1 --2- 172.21.15.118:0/1591239962 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x562fb545d810 0x562fb546e9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:16:20.610 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:20.602+0000 7fb65b400640 1 --2- 172.21.15.118:0/1591239962 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x562fb545f3a0 0x562fb546b440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:16:20.611 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:20.603+0000 7fb653e00640 1 -- 172.21.15.118:0/1591239962 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x562fb545f3a0 msgr2=0x562fb546b440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:16:20.611 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:20.603+0000 7fb653e00640 1 --2- 172.21.15.118:0/1591239962 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x562fb545f3a0 0x562fb546b440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:16:20.611 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:20.603+0000 7fb653e00640 1 -- 172.21.15.118:0/1591239962 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x562fb545d810 msgr2=0x562fb546e9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:16:20.611 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:20.603+0000 7fb653e00640 1 --2- 172.21.15.118:0/1591239962 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x562fb545d810 0x562fb546e9a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:16:20.611 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:20.603+0000 7fb653e00640 1 -- 172.21.15.118:0/1591239962 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x562fb5473420 con 0x562fb545e9e0 2024-03-24T00:16:20.611 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:20.603+0000 7fb653e00640 1 --2- 172.21.15.118:0/1591239962 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x562fb545e9e0 0x562fb546eee0 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7fb644009730 tx=0x7fb644027660 comp rx=0 tx=0).ready entity=mon.1 client_cookie=f7fec4a30e4810a2 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:16:20.611 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:20.603+0000 7fb648c00640 10 client.0 ms_handle_connect on v2:172.21.15.193:3300/0 2024-03-24T00:16:20.611 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:20.603+0000 7fb648c00640 1 -- 172.21.15.118:0/1591239962 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb64402c070 con 0x562fb545e9e0 2024-03-24T00:16:20.611 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:20.603+0000 7fb648c00640 1 -- 172.21.15.118:0/1591239962 <== mon.1 v2:172.21.15.193:3300/0 2 ==== osd_map(90..90 src has 1..90) v4 ==== 8277+0+0 (secure 0 0 0) 0x7fb644023070 con 0x562fb545e9e0 2024-03-24T00:16:20.611 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:20.603+0000 7fb6645c0740 1 -- 172.21.15.118:0/1591239962 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fb654021040 con 0x562fb545e9e0 2024-03-24T00:16:20.611 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:20.604+0000 7fb648c00640 1 -- 172.21.15.118:0/1591239962 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) v2 ==== 24+0+0 (secure 0 0 0) 0x7fb644027ca0 con 0x562fb545e9e0 2024-03-24T00:16:20.611 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:20.604+0000 7fb6645c0740 10 client.25561 fetch_fsmap learned FSMap version 28 2024-03-24T00:16:20.611 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:20.604+0000 7fb6645c0740 1 -- 172.21.15.118:0/1591239962 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) v3 -- 0x562fb54cfb50 con 0x562fb545e9e0 2024-03-24T00:16:20.611 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:20.604+0000 7fb648c00640 1 -- 172.21.15.118:0/1591239962 <== mon.1 v2:172.21.15.193:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fb654021040 con 0x562fb545e9e0 2024-03-24T00:16:20.611 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:20.604+0000 7fb648c00640 1 -- 172.21.15.118:0/1591239962 <== mon.1 v2:172.21.15.193:3300/0 5 ==== fsmap(e 28) v1 ==== 7934+0+0 (secure 0 0 0) 0x7fb644004890 con 0x562fb545e9e0 2024-03-24T00:16:20.612 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:20.604+0000 7fb6645c0740 10 client.25561 fetch_fsmap finished waiting for FSMap version 28 2024-03-24T00:16:20.612 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:20.604+0000 7fb6645c0740 10 client.25561 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-24T00:16:20.612 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:20.604+0000 7fb6645c0740 20 client.25561 populate_metadata read hostname 'smithi118' 2024-03-24T00:16:20.612 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:20.604+0000 7fb6645c0740 1 --2- 172.21.15.118:0/1591239962 >> [v2:172.21.15.189:6838/4074484168,v1:172.21.15.189:6839/4074484168] conn(0x562fb54d0480 0x562fb54d28a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:16:20.612 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:20.604+0000 7fb6645c0740 4 client.25561 mds_command: new command op to 24569 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-24T00:16:20.612 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:20.604+0000 7fb6645c0740 1 -- 172.21.15.118:0/1591239962 --> [v2:172.21.15.189:6838/4074484168,v1:172.21.15.189:6839/4074484168] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x562fb53c9b50 con 0x562fb54d0480 2024-03-24T00:16:20.612 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:20.604+0000 7fb65b400640 1 --2- 172.21.15.118:0/1591239962 >> [v2:172.21.15.189:6838/4074484168,v1:172.21.15.189:6839/4074484168] conn(0x562fb54d0480 0x562fb54d28a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:16:20.612 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:20.605+0000 7fb648c00640 10 client.25561 ms_handle_connect on v2:172.21.15.189:6838/4074484168 2024-03-24T00:16:20.612 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:20.605+0000 7fb65b400640 1 --2- 172.21.15.118:0/1591239962 >> [v2:172.21.15.189:6838/4074484168,v1:172.21.15.189:6839/4074484168] conn(0x562fb54d0480 0x562fb54d28a0 crc :-1 s=READY pgs=44 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=436adb93ab413ee3 server_cookie=7c4bdbfbbf815296 in_seq=0 out_seq=0 2024-03-24T00:16:20.612 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:20.605+0000 7fb648c00640 1 -- 172.21.15.118:0/1591239962 <== mds.0 v2:172.21.15.189:6838/4074484168 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x562fb53c9b50 con 0x562fb54d0480 2024-03-24T00:16:20.612 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:20.605+0000 7fb648c00640 10 client.25561 handle_command_reply: tid=0 2024-03-24T00:16:20.612 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:20.605+0000 7fb6645c0740 1 client.25561 shutdown 2024-03-24T00:16:20.612 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:20.605+0000 7fb6645c0740 1 -- 172.21.15.118:0/1591239962 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x562fb545e9e0 msgr2=0x562fb546eee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:16:20.612 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:20.605+0000 7fb6645c0740 1 --2- 172.21.15.118:0/1591239962 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x562fb545e9e0 0x562fb546eee0 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7fb644009730 tx=0x7fb644027660 comp rx=0 tx=0).stop 2024-03-24T00:16:20.613 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:20.605+0000 7fb6645c0740 1 -- 172.21.15.118:0/1591239962 shutdown_connections 2024-03-24T00:16:20.613 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:20.605+0000 7fb6645c0740 1 --2- 172.21.15.118:0/1591239962 >> [v2:172.21.15.189:6838/4074484168,v1:172.21.15.189:6839/4074484168] conn(0x562fb54d0480 0x562fb54d28a0 crc :-1 s=READY pgs=44 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:16:20.613 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:20.605+0000 7fb6645c0740 1 --2- 172.21.15.118:0/1591239962 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x562fb545f3a0 0x562fb546b440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:16:20.613 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:20.605+0000 7fb6645c0740 1 --2- 172.21.15.118:0/1591239962 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x562fb545e9e0 0x562fb546eee0 unknown :-1 s=CLOSED pgs=223 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:16:20.613 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:20.605+0000 7fb6645c0740 1 --2- 172.21.15.118:0/1591239962 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x562fb545d810 0x562fb546e9a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:16:20.613 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:20.605+0000 7fb648c00640 0 client.25561 ms_handle_reset on v2:172.21.15.189:6838/4074484168 2024-03-24T00:16:20.613 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:20.605+0000 7fb6645c0740 1 -- 172.21.15.118:0/1591239962 >> 172.21.15.118:0/1591239962 conn(0x562fb5458c00 msgr2=0x562fb545be60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:16:20.613 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:20.605+0000 7fb6645c0740 1 -- 172.21.15.118:0/1591239962 shutdown_connections 2024-03-24T00:16:20.613 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:20.606+0000 7fb6645c0740 1 -- 172.21.15.118:0/1591239962 wait complete. 2024-03-24T00:16:20.613 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:20.606+0000 7fb6645c0740 20 client.25561 trim_cache size 0 max 16384 2024-03-24T00:16:20.613 INFO:teuthology.orchestra.run.smithi118.stdout:[] 2024-03-24T00:16:20.614 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:20.606+0000 7fb661800640 1 -- 172.21.15.118:0/479850798 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7fb6400a2910 msgr2=0x7fb6400a4dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:16:20.614 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:20.606+0000 7fb661800640 1 --2- 172.21.15.118:0/479850798 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7fb6400a2910 0x7fb6400a4dd0 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7fb64c0c5740 tx=0x7fb64c0b9040 comp rx=0 tx=0).stop 2024-03-24T00:16:20.614 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:20.606+0000 7fb661800640 1 -- 172.21.15.118:0/479850798 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fb65c1038e0 msgr2=0x7fb65c199c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:16:20.614 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:20.606+0000 7fb661800640 1 --2- 172.21.15.118:0/479850798 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fb65c1038e0 0x7fb65c199c70 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7fb64c000c00 tx=0x7fb64c02c650 comp rx=0 tx=0).stop 2024-03-24T00:16:20.615 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:20.606+0000 7fb661800640 1 -- 172.21.15.118:0/479850798 shutdown_connections 2024-03-24T00:16:20.615 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:20.606+0000 7fb661800640 1 --2- 172.21.15.118:0/479850798 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7fb6400a2910 0x7fb6400a4dd0 unknown :-1 s=CLOSED pgs=279 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:16:20.615 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:20.606+0000 7fb661800640 1 --2- 172.21.15.118:0/479850798 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb65c105490 0x7fb65c1a11e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:16:20.615 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:20.606+0000 7fb661800640 1 --2- 172.21.15.118:0/479850798 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb65c104ad0 0x7fb65c19a1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:16:20.615 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:20.606+0000 7fb661800640 1 --2- 172.21.15.118:0/479850798 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fb65c1038e0 0x7fb65c199c70 unknown :-1 s=CLOSED pgs=221 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:16:20.615 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:20.606+0000 7fb661800640 1 -- 172.21.15.118:0/479850798 >> 172.21.15.118:0/479850798 conn(0x7fb65c0fee50 msgr2=0x7fb65c107df0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:16:20.615 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:20.607+0000 7fb661800640 1 -- 172.21.15.118:0/479850798 shutdown_connections 2024-03-24T00:16:20.615 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:20.607+0000 7fb661800640 1 -- 172.21.15.118:0/479850798 wait complete. 2024-03-24T00:16:20.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:16:20 smithi118 ceph-mon[31535]: pgmap v417: 97 pgs: 97 active+clean; 662 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 5.9 KiB/s rd, 255 B/s wr, 11 op/s 2024-03-24T00:16:21.203 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:16:20 smithi189 ceph-mon[36187]: pgmap v417: 97 pgs: 97 active+clean; 662 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 5.9 KiB/s rd, 255 B/s wr, 11 op/s 2024-03-24T00:16:21.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:16:20 smithi193 ceph-mon[36878]: pgmap v417: 97 pgs: 97 active+clean; 662 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 5.9 KiB/s rd, 255 B/s wr, 11 op/s 2024-03-24T00:16:21.626 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-03-24T00:16:21.626 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T00:16:21.626 DEBUG:teuthology.orchestra.run.smithi118:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2024-03-24T00:16:21.875 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:21.875+0000 7f8d26c00640 1 -- 172.21.15.118:0/2036260474 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f8d20104a60 msgr2=0x7f8d20106e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:16:21.875 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:21.875+0000 7f8d26c00640 1 --2- 172.21.15.118:0/2036260474 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f8d20104a60 0x7f8d20106e50 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7f8d08009900 tx=0x7f8d0802bc30 comp rx=0 tx=0).stop 2024-03-24T00:16:21.875 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:21.875+0000 7f8d26c00640 1 -- 172.21.15.118:0/2036260474 shutdown_connections 2024-03-24T00:16:21.875 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:21.875+0000 7f8d26c00640 1 --2- 172.21.15.118:0/2036260474 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f8d20107390 0x7f8d201097e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:16:21.875 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:21.875+0000 7f8d26c00640 1 --2- 172.21.15.118:0/2036260474 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f8d20104a60 0x7f8d20106e50 unknown :-1 s=CLOSED pgs=224 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:16:21.875 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:21.875+0000 7f8d26c00640 1 --2- 172.21.15.118:0/2036260474 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8d20102130 0x7f8d20104520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:16:21.875 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:21.875+0000 7f8d26c00640 1 -- 172.21.15.118:0/2036260474 >> 172.21.15.118:0/2036260474 conn(0x7f8d200fbb40 msgr2=0x7f8d200fdf60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:16:21.875 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:21.875+0000 7f8d26c00640 1 -- 172.21.15.118:0/2036260474 shutdown_connections 2024-03-24T00:16:21.875 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:21.875+0000 7f8d26c00640 1 -- 172.21.15.118:0/2036260474 wait complete. 2024-03-24T00:16:21.876 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:21.877+0000 7f8d26c00640 1 Processor -- start 2024-03-24T00:16:21.876 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:21.877+0000 7f8d26c00640 1 -- start start 2024-03-24T00:16:21.878 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:21.877+0000 7f8d26c00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8d20102130 0x7f8d20197a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:16:21.878 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:21.877+0000 7f8d26c00640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f8d20104a60 0x7f8d20197f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:16:21.878 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:21.877+0000 7f8d26c00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f8d20107390 0x7f8d2019efe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:16:21.878 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:21.877+0000 7f8d26c00640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8d2010d390 con 0x7f8d20104a60 2024-03-24T00:16:21.878 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:21.877+0000 7f8d26c00640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8d2010d210 con 0x7f8d20102130 2024-03-24T00:16:21.878 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:21.877+0000 7f8d26c00640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8d2010d510 con 0x7f8d20107390 2024-03-24T00:16:21.878 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:21.878+0000 7f8d24c00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f8d20107390 0x7f8d2019efe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:16:21.879 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:21.878+0000 7f8d1fe00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8d20102130 0x7f8d20197a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:16:21.879 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:21.878+0000 7f8d1fe00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8d20102130 0x7f8d20197a30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.118:60482/0 (socket says 172.21.15.118:60482) 2024-03-24T00:16:21.879 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:21.878+0000 7f8d1fe00640 1 -- 172.21.15.118:0/468414580 learned_addr learned my addr 172.21.15.118:0/468414580 (peer_addr_for_me v2:172.21.15.118:0/0) 2024-03-24T00:16:21.879 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:21.878+0000 7f8d17400640 1 --2- 172.21.15.118:0/468414580 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f8d20104a60 0x7f8d20197f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:16:21.879 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:21.878+0000 7f8d24c00640 1 -- 172.21.15.118:0/468414580 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8d20102130 msgr2=0x7f8d20197a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:16:21.879 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:21.878+0000 7f8d24c00640 1 --2- 172.21.15.118:0/468414580 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8d20102130 0x7f8d20197a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:16:21.879 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:21.878+0000 7f8d24c00640 1 -- 172.21.15.118:0/468414580 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f8d20104a60 msgr2=0x7f8d20197f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:16:21.879 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:21.878+0000 7f8d24c00640 1 --2- 172.21.15.118:0/468414580 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f8d20104a60 0x7f8d20197f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:16:21.879 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:21.878+0000 7f8d24c00640 1 -- 172.21.15.118:0/468414580 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8d2019f650 con 0x7f8d20107390 2024-03-24T00:16:21.879 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:21.878+0000 7f8d1fe00640 1 --2- 172.21.15.118:0/468414580 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8d20102130 0x7f8d20197a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T00:16:21.879 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:21.878+0000 7f8d17400640 1 --2- 172.21.15.118:0/468414580 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f8d20104a60 0x7f8d20197f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T00:16:21.879 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:21.878+0000 7f8d24c00640 1 --2- 172.21.15.118:0/468414580 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f8d20107390 0x7f8d2019efe0 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7f8d1800e970 tx=0x7f8d1800ee30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:16:21.879 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:21.878+0000 7f8d1d600640 1 -- 172.21.15.118:0/468414580 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8d1800ccc0 con 0x7f8d20107390 2024-03-24T00:16:21.879 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:21.879+0000 7f8d1d600640 1 -- 172.21.15.118:0/468414580 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f8d18005020 con 0x7f8d20107390 2024-03-24T00:16:21.879 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:21.879+0000 7f8d1d600640 1 -- 172.21.15.118:0/468414580 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8d18004a00 con 0x7f8d20107390 2024-03-24T00:16:21.880 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:21.879+0000 7f8d26c00640 1 -- 172.21.15.118:0/468414580 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8d2019f940 con 0x7f8d20107390 2024-03-24T00:16:21.880 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:21.879+0000 7f8d26c00640 1 -- 172.21.15.118:0/468414580 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8d2019fe10 con 0x7f8d20107390 2024-03-24T00:16:21.881 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:21.882+0000 7f8d1d600640 1 -- 172.21.15.118:0/468414580 <== mon.1 v2:172.21.15.193:3300/0 4 ==== mgrmap(e 19) v1 ==== 137342+0+0 (secure 0 0 0) 0x7f8d180051d0 con 0x7f8d20107390 2024-03-24T00:16:21.883 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:21.883+0000 7f8d1d600640 1 --2- 172.21.15.118:0/468414580 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7f8d040a2890 0x7f8d040a4d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:16:21.883 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:21.883+0000 7f8d1d600640 1 -- 172.21.15.118:0/468414580 <== mon.1 v2:172.21.15.193:3300/0 5 ==== osd_map(90..90 src has 1..90) v4 ==== 8277+0+0 (secure 0 0 0) 0x7f8d180cec90 con 0x7f8d20107390 2024-03-24T00:16:21.883 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:21.883+0000 7f8d299a7740 1 Processor -- start 2024-03-24T00:16:21.883 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:21.883+0000 7f8d1fe00640 1 --2- 172.21.15.118:0/468414580 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7f8d040a2890 0x7f8d040a4d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:16:21.883 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:21.883+0000 7f8d299a7740 1 -- start start 2024-03-24T00:16:21.884 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:21.884+0000 7f8d299a7740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x555b1fc900a0 0x555b1fc904c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:16:21.885 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:21.884+0000 7f8d299a7740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x555b1fc90a90 0x555b1fc95590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:16:21.885 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:21.884+0000 7f8d299a7740 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x555b1fc95c00 0x555b1fc98090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:16:21.885 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:21.884+0000 7f8d299a7740 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x555b1fbf5eb0 con 0x555b1fc95c00 2024-03-24T00:16:21.885 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:21.884+0000 7f8d299a7740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x555b1fb9ea60 con 0x555b1fc90a90 2024-03-24T00:16:21.885 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:21.884+0000 7f8d299a7740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x555b1fbf1f50 con 0x555b1fc900a0 2024-03-24T00:16:21.885 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:21.884+0000 7f8d1fe00640 1 --2- 172.21.15.118:0/468414580 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7f8d040a2890 0x7f8d040a4d50 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7f8d100049d0 tx=0x7f8d10009500 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:16:21.885 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:21.884+0000 7f8d17400640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x555b1fc90a90 0x555b1fc95590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:16:21.885 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:21.884+0000 7f8d1fe00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x555b1fc900a0 0x555b1fc904c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:16:21.885 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:21.884+0000 7f8d17400640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x555b1fc90a90 0x555b1fc95590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.118:60494/0 (socket says 172.21.15.118:60494) 2024-03-24T00:16:21.885 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:21.884+0000 7f8d1fe00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x555b1fc900a0 0x555b1fc904c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.118:45672/0 (socket says 172.21.15.118:45672) 2024-03-24T00:16:21.885 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:21.884+0000 7f8d24c00640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x555b1fc95c00 0x555b1fc98090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:16:21.885 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:21.884+0000 7f8d1fe00640 1 -- 172.21.15.118:0/587544928 learned_addr learned my addr 172.21.15.118:0/587544928 (peer_addr_for_me v2:172.21.15.118:0/0) 2024-03-24T00:16:21.885 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:21.884+0000 7f8d1fe00640 1 -- 172.21.15.118:0/587544928 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x555b1fc90a90 msgr2=0x555b1fc95590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:16:21.885 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:21.884+0000 7f8d1fe00640 1 --2- 172.21.15.118:0/587544928 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x555b1fc90a90 0x555b1fc95590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:16:21.886 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:21.884+0000 7f8d1fe00640 1 -- 172.21.15.118:0/587544928 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x555b1fc95c00 msgr2=0x555b1fc98090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:16:21.886 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:21.884+0000 7f8d1fe00640 1 --2- 172.21.15.118:0/587544928 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x555b1fc95c00 0x555b1fc98090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:16:21.886 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:21.884+0000 7f8d1fe00640 1 -- 172.21.15.118:0/587544928 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x555b1f9d27e0 con 0x555b1fc900a0 2024-03-24T00:16:21.886 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:21.885+0000 7f8d1fe00640 1 --2- 172.21.15.118:0/587544928 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x555b1fc900a0 0x555b1fc904c0 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7f8d10011b10 tx=0x7f8d10014000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d0f516c551830c6b server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:16:21.886 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:21.885+0000 7f8d15200640 1 -- 172.21.15.118:0/587544928 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8d1000e9c0 con 0x555b1fc900a0 2024-03-24T00:16:21.886 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:21.885+0000 7f8d15200640 1 -- 172.21.15.118:0/587544928 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f8d1000eb60 con 0x555b1fc900a0 2024-03-24T00:16:21.886 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:21.885+0000 7f8d15200640 1 -- 172.21.15.118:0/587544928 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8d10026660 con 0x555b1fc900a0 2024-03-24T00:16:21.890 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:21.886+0000 7f8d299a7740 1 -- 172.21.15.118:0/587544928 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x555b1fc900a0 msgr2=0x555b1fc904c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:16:21.890 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:21.886+0000 7f8d299a7740 1 --2- 172.21.15.118:0/587544928 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x555b1fc900a0 0x555b1fc904c0 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7f8d10011b10 tx=0x7f8d10014000 comp rx=0 tx=0).stop 2024-03-24T00:16:21.890 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:21.886+0000 7f8d299a7740 1 -- 172.21.15.118:0/587544928 shutdown_connections 2024-03-24T00:16:21.890 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:21.886+0000 7f8d299a7740 1 --2- 172.21.15.118:0/587544928 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x555b1fc95c00 0x555b1fc98090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:16:21.890 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:21.886+0000 7f8d299a7740 1 --2- 172.21.15.118:0/587544928 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x555b1fc90a90 0x555b1fc95590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:16:21.890 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:21.886+0000 7f8d299a7740 1 --2- 172.21.15.118:0/587544928 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x555b1fc900a0 0x555b1fc904c0 unknown :-1 s=CLOSED pgs=226 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:16:21.890 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:21.886+0000 7f8d299a7740 1 -- 172.21.15.118:0/587544928 >> 172.21.15.118:0/587544928 conn(0x555b1fc8b610 msgr2=0x555b1fc8ba10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:16:21.890 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:21.886+0000 7f8d299a7740 1 -- 172.21.15.118:0/587544928 shutdown_connections 2024-03-24T00:16:21.890 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:21.886+0000 7f8d299a7740 1 -- 172.21.15.118:0/587544928 wait complete. 2024-03-24T00:16:21.890 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:21.887+0000 7f8d299a7740 1 -- start start 2024-03-24T00:16:21.890 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:21.887+0000 7f8d299a7740 1 Processor -- start 2024-03-24T00:16:21.890 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:21.887+0000 7f8d299a7740 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x555b1fc900a0 0x555b1fc9e5b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:16:21.891 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:21.887+0000 7f8d299a7740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x555b1fc90a90 0x555b1fc9eaf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:16:21.891 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:21.887+0000 7f8d299a7740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x555b1fc95c00 0x555b1fca5b30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:16:21.891 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:21.887+0000 7f8d299a7740 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x555b1fc98bb0 con 0x555b1fc900a0 2024-03-24T00:16:21.891 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:21.887+0000 7f8d299a7740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x555b1fbb1eb0 con 0x555b1fc90a90 2024-03-24T00:16:21.891 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:21.887+0000 7f8d299a7740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x555b1fc98d30 con 0x555b1fc95c00 2024-03-24T00:16:21.891 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:21.887+0000 7f8d299a7740 10 client.0 _ll_register_callbacks cb 0x555b1fba6150 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T00:16:21.891 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:21.887+0000 7f8d17400640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x555b1fc90a90 0x555b1fc9eaf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:16:21.891 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:21.887+0000 7f8d17400640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x555b1fc90a90 0x555b1fc9eaf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.118:60498/0 (socket says 172.21.15.118:60498) 2024-03-24T00:16:21.891 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:21.888+0000 7f8d17400640 1 -- 172.21.15.118:0/2773298541 learned_addr learned my addr 172.21.15.118:0/2773298541 (peer_addr_for_me v2:172.21.15.118:0/0) 2024-03-24T00:16:21.891 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:21.888+0000 7f8d24c00640 1 --2- 172.21.15.118:0/2773298541 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x555b1fc95c00 0x555b1fca5b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:16:21.891 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:21.888+0000 7f8d1fe00640 1 --2- 172.21.15.118:0/2773298541 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x555b1fc900a0 0x555b1fc9e5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:16:21.891 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:21.888+0000 7f8d17400640 1 -- 172.21.15.118:0/2773298541 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x555b1fc95c00 msgr2=0x555b1fca5b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:16:21.891 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:21.888+0000 7f8d17400640 1 --2- 172.21.15.118:0/2773298541 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x555b1fc95c00 0x555b1fca5b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:16:21.891 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:21.888+0000 7f8d17400640 1 -- 172.21.15.118:0/2773298541 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x555b1fc900a0 msgr2=0x555b1fc9e5b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:16:21.891 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:21.888+0000 7f8d17400640 1 --2- 172.21.15.118:0/2773298541 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x555b1fc900a0 0x555b1fc9e5b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:16:21.891 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:21.888+0000 7f8d17400640 1 -- 172.21.15.118:0/2773298541 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x555b1fa32f70 con 0x555b1fc90a90 2024-03-24T00:16:21.892 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:21.888+0000 7f8d1fe00640 1 --2- 172.21.15.118:0/2773298541 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x555b1fc900a0 0x555b1fc9e5b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T00:16:21.892 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:21.888+0000 7f8d24c00640 1 --2- 172.21.15.118:0/2773298541 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x555b1fc95c00 0x555b1fca5b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T00:16:21.892 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:21.888+0000 7f8d17400640 1 --2- 172.21.15.118:0/2773298541 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x555b1fc90a90 0x555b1fc9eaf0 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7f8d0802c240 tx=0x7f8d08004270 comp rx=0 tx=0).ready entity=mon.2 client_cookie=6f96fe1d85b047bc server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:16:21.892 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:21.888+0000 7f8d0d600640 10 client.0 ms_handle_connect on v2:172.21.15.189:3300/0 2024-03-24T00:16:21.892 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:21.888+0000 7f8d0d600640 1 -- 172.21.15.118:0/2773298541 <== mon.2 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8d08005d50 con 0x555b1fc90a90 2024-03-24T00:16:21.892 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:21.889+0000 7f8d299a7740 1 -- 172.21.15.118:0/2773298541 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f8d1000eb60 con 0x555b1fc90a90 2024-03-24T00:16:21.892 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:21.889+0000 7f8d0d600640 1 -- 172.21.15.118:0/2773298541 <== mon.2 v2:172.21.15.189:3300/0 2 ==== osd_map(90..90 src has 1..90) v4 ==== 8277+0+0 (secure 0 0 0) 0x7f8d08005220 con 0x555b1fc90a90 2024-03-24T00:16:21.892 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:21.889+0000 7f8d0d600640 1 -- 172.21.15.118:0/2773298541 <== mon.2 v2:172.21.15.189:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) v2 ==== 24+0+0 (secure 0 0 0) 0x7f8d08050050 con 0x555b1fc90a90 2024-03-24T00:16:21.892 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:21.889+0000 7f8d299a7740 10 client.25571 fetch_fsmap learned FSMap version 28 2024-03-24T00:16:21.892 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:21.889+0000 7f8d299a7740 1 -- 172.21.15.118:0/2773298541 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) v3 -- 0x555b1f8ca170 con 0x555b1fc90a90 2024-03-24T00:16:21.892 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:21.889+0000 7f8d0d600640 1 -- 172.21.15.118:0/2773298541 <== mon.2 v2:172.21.15.189:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f8d1000eb60 con 0x555b1fc90a90 2024-03-24T00:16:21.892 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:21.889+0000 7f8d0d600640 1 -- 172.21.15.118:0/2773298541 <== mon.2 v2:172.21.15.189:3300/0 5 ==== fsmap(e 28) v1 ==== 7934+0+0 (secure 0 0 0) 0x7f8d08003950 con 0x555b1fc90a90 2024-03-24T00:16:21.892 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:21.889+0000 7f8d299a7740 10 client.25571 fetch_fsmap finished waiting for FSMap version 28 2024-03-24T00:16:21.892 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:21.889+0000 7f8d299a7740 10 client.25571 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-24T00:16:21.892 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:21.889+0000 7f8d299a7740 20 client.25571 populate_metadata read hostname 'smithi118' 2024-03-24T00:16:21.893 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:21.889+0000 7f8d299a7740 1 --2- 172.21.15.118:0/2773298541 >> [v2:172.21.15.189:6838/4074484168,v1:172.21.15.189:6839/4074484168] conn(0x555b1fcaa3c0 0x555b1fcac7e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:16:21.893 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:21.889+0000 7f8d299a7740 4 client.25571 mds_command: new command op to 24569 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-24T00:16:21.893 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:21.889+0000 7f8d299a7740 1 -- 172.21.15.118:0/2773298541 --> [v2:172.21.15.189:6838/4074484168,v1:172.21.15.189:6839/4074484168] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x555b1fc033f0 con 0x555b1fcaa3c0 2024-03-24T00:16:21.893 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:21.890+0000 7f8d1fe00640 1 --2- 172.21.15.118:0/2773298541 >> [v2:172.21.15.189:6838/4074484168,v1:172.21.15.189:6839/4074484168] conn(0x555b1fcaa3c0 0x555b1fcac7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:16:21.893 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:21.890+0000 7f8d0d600640 10 client.25571 ms_handle_connect on v2:172.21.15.189:6838/4074484168 2024-03-24T00:16:21.893 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:21.890+0000 7f8d1fe00640 1 --2- 172.21.15.118:0/2773298541 >> [v2:172.21.15.189:6838/4074484168,v1:172.21.15.189:6839/4074484168] conn(0x555b1fcaa3c0 0x555b1fcac7e0 crc :-1 s=READY pgs=45 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=63b5894eb7be8ab6 server_cookie=64f53a042b9480f5 in_seq=0 out_seq=0 2024-03-24T00:16:21.894 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:21.891+0000 7f8d0d600640 1 -- 172.21.15.118:0/2773298541 <== mds.0 v2:172.21.15.189:6838/4074484168 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x555b1fc033f0 con 0x555b1fcaa3c0 2024-03-24T00:16:21.894 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:21.891+0000 7f8d0d600640 10 client.25571 handle_command_reply: tid=0 2024-03-24T00:16:21.894 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:21.891+0000 7f8d299a7740 1 client.25571 shutdown 2024-03-24T00:16:21.894 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:21.891+0000 7f8d299a7740 1 -- 172.21.15.118:0/2773298541 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x555b1fc90a90 msgr2=0x555b1fc9eaf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:16:21.894 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:21.891+0000 7f8d299a7740 1 --2- 172.21.15.118:0/2773298541 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x555b1fc90a90 0x555b1fc9eaf0 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7f8d0802c240 tx=0x7f8d08004270 comp rx=0 tx=0).stop 2024-03-24T00:16:21.894 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:21.891+0000 7f8d299a7740 1 -- 172.21.15.118:0/2773298541 shutdown_connections 2024-03-24T00:16:21.894 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:21.891+0000 7f8d299a7740 1 --2- 172.21.15.118:0/2773298541 >> [v2:172.21.15.189:6838/4074484168,v1:172.21.15.189:6839/4074484168] conn(0x555b1fcaa3c0 0x555b1fcac7e0 crc :-1 s=READY pgs=45 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:16:21.894 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:21.891+0000 7f8d299a7740 1 --2- 172.21.15.118:0/2773298541 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x555b1fc95c00 0x555b1fca5b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:16:21.895 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:21.891+0000 7f8d299a7740 1 --2- 172.21.15.118:0/2773298541 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x555b1fc90a90 0x555b1fc9eaf0 unknown :-1 s=CLOSED pgs=239 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:16:21.895 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:21.891+0000 7f8d0d600640 0 client.25571 ms_handle_reset on v2:172.21.15.189:6838/4074484168 2024-03-24T00:16:21.895 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:21.891+0000 7f8d299a7740 1 --2- 172.21.15.118:0/2773298541 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x555b1fc900a0 0x555b1fc9e5b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:16:21.895 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:21.891+0000 7f8d299a7740 1 -- 172.21.15.118:0/2773298541 >> 172.21.15.118:0/2773298541 conn(0x555b1fc8b610 msgr2=0x555b1fc8a830 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:16:21.895 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:21.891+0000 7f8d299a7740 1 -- 172.21.15.118:0/2773298541 shutdown_connections 2024-03-24T00:16:21.895 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:21.892+0000 7f8d299a7740 1 -- 172.21.15.118:0/2773298541 wait complete. 2024-03-24T00:16:21.895 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:21.892+0000 7f8d299a7740 20 client.25571 trim_cache size 0 max 16384 2024-03-24T00:16:21.909 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:21.909+0000 7f8d299a7740 1 Processor -- start 2024-03-24T00:16:21.909 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:21.909+0000 7f8d299a7740 1 -- start start 2024-03-24T00:16:21.909 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:21.909+0000 7f8d299a7740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x555b1fc900a0 0x555b1fc99770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:16:21.909 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:21.909+0000 7f8d299a7740 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x555b1fc90a90 0x555b1fca5af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:16:21.909 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:21.909+0000 7f8d299a7740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x555b1fc95c00 0x555b1fc8ef80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:16:21.909 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:21.909+0000 7f8d299a7740 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8d201a6800 con 0x555b1fc90a90 2024-03-24T00:16:21.909 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:21.909+0000 7f8d299a7740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8d201a8180 con 0x555b1fc900a0 2024-03-24T00:16:21.909 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:21.909+0000 7f8d299a7740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8d201a6e60 con 0x555b1fc95c00 2024-03-24T00:16:21.910 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:21.910+0000 7f8d1fe00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x555b1fc900a0 0x555b1fc99770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:16:21.910 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:21.910+0000 7f8d24c00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x555b1fc95c00 0x555b1fc8ef80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:16:21.910 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:21.910+0000 7f8d1fe00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x555b1fc900a0 0x555b1fc99770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.118:60508/0 (socket says 172.21.15.118:60508) 2024-03-24T00:16:21.910 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:21.910+0000 7f8d1fe00640 1 -- 172.21.15.118:0/2322908666 learned_addr learned my addr 172.21.15.118:0/2322908666 (peer_addr_for_me v2:172.21.15.118:0/0) 2024-03-24T00:16:21.910 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:21.910+0000 7f8d17400640 1 --2- 172.21.15.118:0/2322908666 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x555b1fc90a90 0x555b1fca5af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:16:21.911 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:21.910+0000 7f8d24c00640 1 -- 172.21.15.118:0/2322908666 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x555b1fc900a0 msgr2=0x555b1fc99770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:16:21.911 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:21.910+0000 7f8d24c00640 1 --2- 172.21.15.118:0/2322908666 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x555b1fc900a0 0x555b1fc99770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:16:21.911 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:21.910+0000 7f8d24c00640 1 -- 172.21.15.118:0/2322908666 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x555b1fc90a90 msgr2=0x555b1fca5af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:16:21.911 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:21.910+0000 7f8d24c00640 1 --2- 172.21.15.118:0/2322908666 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x555b1fc90a90 0x555b1fca5af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:16:21.912 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:21.910+0000 7f8d24c00640 1 -- 172.21.15.118:0/2322908666 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x555b1fc86c20 con 0x555b1fc95c00 2024-03-24T00:16:21.912 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:21.910+0000 7f8d1fe00640 1 --2- 172.21.15.118:0/2322908666 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x555b1fc900a0 0x555b1fc99770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T00:16:21.912 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:21.910+0000 7f8d24c00640 1 --2- 172.21.15.118:0/2322908666 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x555b1fc95c00 0x555b1fc8ef80 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7f8d180cc5b0 tx=0x7f8d18097790 comp rx=0 tx=0).ready entity=mon.1 client_cookie=19990bf7b75db999 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:16:21.912 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:21.911+0000 7f8d0fe00640 1 -- 172.21.15.118:0/2322908666 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8d1801c040 con 0x555b1fc95c00 2024-03-24T00:16:21.912 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:21.911+0000 7f8d0fe00640 1 -- 172.21.15.118:0/2322908666 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f8d1809f590 con 0x555b1fc95c00 2024-03-24T00:16:21.912 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:21.911+0000 7f8d0fe00640 1 -- 172.21.15.118:0/2322908666 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8d180ccbc0 con 0x555b1fc95c00 2024-03-24T00:16:21.912 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:21.911+0000 7f8d299a7740 1 -- 172.21.15.118:0/2322908666 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x555b1fc95c00 msgr2=0x555b1fc8ef80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:16:21.912 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:21.911+0000 7f8d299a7740 1 --2- 172.21.15.118:0/2322908666 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x555b1fc95c00 0x555b1fc8ef80 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7f8d180cc5b0 tx=0x7f8d18097790 comp rx=0 tx=0).stop 2024-03-24T00:16:21.912 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:21.911+0000 7f8d299a7740 1 -- 172.21.15.118:0/2322908666 shutdown_connections 2024-03-24T00:16:21.912 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:21.911+0000 7f8d299a7740 1 --2- 172.21.15.118:0/2322908666 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x555b1fc95c00 0x555b1fc8ef80 unknown :-1 s=CLOSED pgs=227 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:16:21.912 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:21.911+0000 7f8d299a7740 1 --2- 172.21.15.118:0/2322908666 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x555b1fc90a90 0x555b1fca5af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:16:21.912 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:21.911+0000 7f8d299a7740 1 --2- 172.21.15.118:0/2322908666 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x555b1fc900a0 0x555b1fc99770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:16:21.912 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:21.911+0000 7f8d299a7740 1 -- 172.21.15.118:0/2322908666 >> 172.21.15.118:0/2322908666 conn(0x555b1fc8b610 msgr2=0x555b1fc8e870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:16:21.912 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:21.911+0000 7f8d299a7740 1 -- 172.21.15.118:0/2322908666 shutdown_connections 2024-03-24T00:16:21.912 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:21.911+0000 7f8d299a7740 1 -- 172.21.15.118:0/2322908666 wait complete. 2024-03-24T00:16:21.913 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:21.912+0000 7f8d299a7740 1 -- start start 2024-03-24T00:16:21.913 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:21.912+0000 7f8d299a7740 1 Processor -- start 2024-03-24T00:16:21.915 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:21.912+0000 7f8d299a7740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x555b1fc900a0 0x555b1fca58a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:16:21.915 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:21.912+0000 7f8d299a7740 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x555b1fc90a90 0x555b1fca3eb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:16:21.915 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:21.912+0000 7f8d299a7740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x555b1fc95c00 0x555b1fca43f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:16:21.915 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:21.912+0000 7f8d299a7740 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8d201a7b20 con 0x555b1fc90a90 2024-03-24T00:16:21.915 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:21.912+0000 7f8d299a7740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8d201a74c0 con 0x555b1fc900a0 2024-03-24T00:16:21.915 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:21.912+0000 7f8d299a7740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8d201a49c0 con 0x555b1fc95c00 2024-03-24T00:16:21.915 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:21.912+0000 7f8d299a7740 10 client.0 _ll_register_callbacks cb 0x7f8d0802cdd0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T00:16:21.915 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:21.913+0000 7f8d1fe00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x555b1fc900a0 0x555b1fca58a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:16:21.916 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:21.913+0000 7f8d1fe00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x555b1fc900a0 0x555b1fca58a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.118:60516/0 (socket says 172.21.15.118:60516) 2024-03-24T00:16:21.916 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:21.913+0000 7f8d1fe00640 1 -- 172.21.15.118:0/1435560591 learned_addr learned my addr 172.21.15.118:0/1435560591 (peer_addr_for_me v2:172.21.15.118:0/0) 2024-03-24T00:16:21.916 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:21.913+0000 7f8d17400640 1 --2- 172.21.15.118:0/1435560591 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x555b1fc90a90 0x555b1fca3eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:16:21.916 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:21.913+0000 7f8d24c00640 1 --2- 172.21.15.118:0/1435560591 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x555b1fc95c00 0x555b1fca43f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:16:21.916 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:21.913+0000 7f8d1fe00640 1 -- 172.21.15.118:0/1435560591 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x555b1fc95c00 msgr2=0x555b1fca43f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:16:21.916 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:21.913+0000 7f8d1fe00640 1 --2- 172.21.15.118:0/1435560591 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x555b1fc95c00 0x555b1fca43f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:16:21.916 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:21.913+0000 7f8d1fe00640 1 -- 172.21.15.118:0/1435560591 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x555b1fc90a90 msgr2=0x555b1fca3eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:16:21.916 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:21.913+0000 7f8d1fe00640 1 --2- 172.21.15.118:0/1435560591 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x555b1fc90a90 0x555b1fca3eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:16:21.916 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:21.913+0000 7f8d1fe00640 1 -- 172.21.15.118:0/1435560591 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x555b1fca4ae0 con 0x555b1fc900a0 2024-03-24T00:16:21.916 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:21.913+0000 7f8d17400640 1 --2- 172.21.15.118:0/1435560591 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x555b1fc90a90 0x555b1fca3eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T00:16:21.916 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:21.913+0000 7f8d1fe00640 1 --2- 172.21.15.118:0/1435560591 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x555b1fc900a0 0x555b1fca58a0 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x555b1fc912f0 tx=0x7f8d10011970 comp rx=0 tx=0).ready entity=mon.2 client_cookie=be5736e3a64f5a57 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:16:21.916 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:21.913+0000 7f8d0d600640 10 client.0 ms_handle_connect on v2:172.21.15.189:3300/0 2024-03-24T00:16:21.916 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:21.913+0000 7f8d0d600640 1 -- 172.21.15.118:0/1435560591 <== mon.2 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8d10038070 con 0x555b1fc900a0 2024-03-24T00:16:21.916 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:21.913+0000 7f8d0d600640 1 -- 172.21.15.118:0/1435560591 <== mon.2 v2:172.21.15.189:3300/0 2 ==== osd_map(90..90 src has 1..90) v4 ==== 8277+0+0 (secure 0 0 0) 0x7f8d10034030 con 0x555b1fc900a0 2024-03-24T00:16:21.917 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:21.913+0000 7f8d299a7740 1 -- 172.21.15.118:0/1435560591 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f8d1809f590 con 0x555b1fc900a0 2024-03-24T00:16:21.917 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:21.914+0000 7f8d0d600640 1 -- 172.21.15.118:0/1435560591 <== mon.2 v2:172.21.15.189:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) v2 ==== 24+0+0 (secure 0 0 0) 0x7f8d10034230 con 0x555b1fc900a0 2024-03-24T00:16:21.917 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:21.914+0000 7f8d299a7740 10 client.25577 fetch_fsmap learned FSMap version 28 2024-03-24T00:16:21.917 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:21.914+0000 7f8d299a7740 1 -- 172.21.15.118:0/1435560591 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) v3 -- 0x555b1fd06c10 con 0x555b1fc900a0 2024-03-24T00:16:21.917 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:21.914+0000 7f8d0d600640 1 -- 172.21.15.118:0/1435560591 <== mon.2 v2:172.21.15.189:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f8d1809f590 con 0x555b1fc900a0 2024-03-24T00:16:21.917 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:21.914+0000 7f8d0d600640 1 -- 172.21.15.118:0/1435560591 <== mon.2 v2:172.21.15.189:3300/0 5 ==== fsmap(e 28) v1 ==== 7934+0+0 (secure 0 0 0) 0x7f8d10005040 con 0x555b1fc900a0 2024-03-24T00:16:21.917 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:21.914+0000 7f8d299a7740 10 client.25577 fetch_fsmap finished waiting for FSMap version 28 2024-03-24T00:16:21.917 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:21.914+0000 7f8d299a7740 10 client.25577 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-24T00:16:21.917 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:21.914+0000 7f8d299a7740 20 client.25577 populate_metadata read hostname 'smithi118' 2024-03-24T00:16:21.917 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:21.914+0000 7f8d299a7740 1 --2- 172.21.15.118:0/1435560591 >> [v2:172.21.15.189:6838/4074484168,v1:172.21.15.189:6839/4074484168] conn(0x555b1fd072f0 0x555b1fd09710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:16:21.917 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:21.914+0000 7f8d299a7740 4 client.25577 mds_command: new command op to 24569 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-03-24T00:16:21.917 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:21.914+0000 7f8d299a7740 1 -- 172.21.15.118:0/1435560591 --> [v2:172.21.15.189:6838/4074484168,v1:172.21.15.189:6839/4074484168] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) v1 -- 0x555b1fbfd740 con 0x555b1fd072f0 2024-03-24T00:16:21.917 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:21.914+0000 7f8d17400640 1 --2- 172.21.15.118:0/1435560591 >> [v2:172.21.15.189:6838/4074484168,v1:172.21.15.189:6839/4074484168] conn(0x555b1fd072f0 0x555b1fd09710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:16:21.917 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:21.915+0000 7f8d17400640 1 --2- 172.21.15.118:0/1435560591 >> [v2:172.21.15.189:6838/4074484168,v1:172.21.15.189:6839/4074484168] conn(0x555b1fd072f0 0x555b1fd09710 crc :-1 s=READY pgs=46 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=a5206351ce3220a0 server_cookie=4dc8d7f342911ba0 in_seq=0 out_seq=0 2024-03-24T00:16:21.917 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:21.915+0000 7f8d0d600640 10 client.25577 ms_handle_connect on v2:172.21.15.189:6838/4074484168 2024-03-24T00:16:21.917 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:21.916+0000 7f8d0d600640 1 -- 172.21.15.118:0/1435560591 <== mds.0 v2:172.21.15.189:6838/4074484168 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+110 (crc 0 0 0) 0x555b1fbfd740 con 0x555b1fd072f0 2024-03-24T00:16:21.918 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:21.916+0000 7f8d0d600640 10 client.25577 handle_command_reply: tid=0 2024-03-24T00:16:21.918 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:21.916+0000 7f8d299a7740 1 client.25577 shutdown 2024-03-24T00:16:21.919 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:21.916+0000 7f8d299a7740 1 -- 172.21.15.118:0/1435560591 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x555b1fc900a0 msgr2=0x555b1fca58a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:16:21.919 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:21.916+0000 7f8d299a7740 1 --2- 172.21.15.118:0/1435560591 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x555b1fc900a0 0x555b1fca58a0 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x555b1fc912f0 tx=0x7f8d10011970 comp rx=0 tx=0).stop 2024-03-24T00:16:21.919 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:21.916+0000 7f8d299a7740 1 -- 172.21.15.118:0/1435560591 shutdown_connections 2024-03-24T00:16:21.919 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:21.916+0000 7f8d299a7740 1 --2- 172.21.15.118:0/1435560591 >> [v2:172.21.15.189:6838/4074484168,v1:172.21.15.189:6839/4074484168] conn(0x555b1fd072f0 0x555b1fd09710 crc :-1 s=READY pgs=46 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:16:21.919 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:21.916+0000 7f8d299a7740 1 --2- 172.21.15.118:0/1435560591 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x555b1fc95c00 0x555b1fca43f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:16:21.919 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:21.916+0000 7f8d299a7740 1 --2- 172.21.15.118:0/1435560591 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x555b1fc90a90 0x555b1fca3eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:16:21.919 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:21.916+0000 7f8d0d600640 0 client.25577 ms_handle_reset on v2:172.21.15.189:6838/4074484168 2024-03-24T00:16:21.919 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:21.916+0000 7f8d299a7740 1 --2- 172.21.15.118:0/1435560591 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x555b1fc900a0 0x555b1fca58a0 unknown :-1 s=CLOSED pgs=240 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:16:21.919 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:21.916+0000 7f8d299a7740 1 -- 172.21.15.118:0/1435560591 >> 172.21.15.118:0/1435560591 conn(0x555b1fc8b610 msgr2=0x555b1fc8e870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:16:21.919 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:21.916+0000 7f8d299a7740 1 -- 172.21.15.118:0/1435560591 shutdown_connections 2024-03-24T00:16:21.919 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:21.916+0000 7f8d299a7740 1 -- 172.21.15.118:0/1435560591 wait complete. 2024-03-24T00:16:21.919 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:21.916+0000 7f8d299a7740 20 client.25577 trim_cache size 0 max 16384 2024-03-24T00:16:21.920 INFO:teuthology.orchestra.run.smithi118.stdout:{ 2024-03-24T00:16:21.920 INFO:teuthology.orchestra.run.smithi118.stdout: "return_code": 0, 2024-03-24T00:16:21.920 INFO:teuthology.orchestra.run.smithi118.stdout: "scrub_tag": "59f621ed-f0ed-4c9b-85bb-5ea67ef1ec18", 2024-03-24T00:16:21.920 INFO:teuthology.orchestra.run.smithi118.stdout: "mode": "asynchronous" 2024-03-24T00:16:21.920 INFO:teuthology.orchestra.run.smithi118.stdout:} 2024-03-24T00:16:21.921 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:21.917+0000 7f8d26c00640 1 -- 172.21.15.118:0/468414580 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7f8d040a2890 msgr2=0x7f8d040a4d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:16:21.921 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:21.917+0000 7f8d26c00640 1 --2- 172.21.15.118:0/468414580 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7f8d040a2890 0x7f8d040a4d50 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7f8d100049d0 tx=0x7f8d10009500 comp rx=0 tx=0).stop 2024-03-24T00:16:21.921 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:21.917+0000 7f8d26c00640 1 -- 172.21.15.118:0/468414580 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f8d20107390 msgr2=0x7f8d2019efe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:16:21.921 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:21.917+0000 7f8d26c00640 1 --2- 172.21.15.118:0/468414580 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f8d20107390 0x7f8d2019efe0 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7f8d1800e970 tx=0x7f8d1800ee30 comp rx=0 tx=0).stop 2024-03-24T00:16:21.921 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:21.917+0000 7f8d26c00640 1 -- 172.21.15.118:0/468414580 shutdown_connections 2024-03-24T00:16:21.921 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:21.917+0000 7f8d26c00640 1 --2- 172.21.15.118:0/468414580 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7f8d040a2890 0x7f8d040a4d50 unknown :-1 s=CLOSED pgs=280 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:16:21.921 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:21.917+0000 7f8d26c00640 1 --2- 172.21.15.118:0/468414580 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f8d20107390 0x7f8d2019efe0 unknown :-1 s=CLOSED pgs=225 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:16:21.921 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:21.917+0000 7f8d26c00640 1 --2- 172.21.15.118:0/468414580 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f8d20104a60 0x7f8d20197f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:16:21.921 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:21.917+0000 7f8d26c00640 1 --2- 172.21.15.118:0/468414580 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8d20102130 0x7f8d20197a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:16:21.921 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:21.917+0000 7f8d26c00640 1 -- 172.21.15.118:0/468414580 >> 172.21.15.118:0/468414580 conn(0x7f8d200fbb40 msgr2=0x7f8d20102bf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:16:21.921 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:21.917+0000 7f8d26c00640 1 -- 172.21.15.118:0/468414580 shutdown_connections 2024-03-24T00:16:21.921 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:21.918+0000 7f8d26c00640 1 -- 172.21.15.118:0/468414580 wait complete. 2024-03-24T00:16:21.934 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-03-24T00:16:21.934 DEBUG:teuthology.orchestra.run.smithi118:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-24T00:16:22.139 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:22.139+0000 7f922ec00640 1 -- 172.21.15.118:0/3046224552 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9228105490 msgr2=0x7f922810bd20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:16:22.139 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:22.139+0000 7f922ec00640 1 --2- 172.21.15.118:0/3046224552 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9228105490 0x7f922810bd20 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7f9220009ca0 tx=0x7f922002c100 comp rx=0 tx=0).stop 2024-03-24T00:16:22.139 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:22.139+0000 7f922ec00640 1 -- 172.21.15.118:0/3046224552 shutdown_connections 2024-03-24T00:16:22.139 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:22.139+0000 7f922ec00640 1 --2- 172.21.15.118:0/3046224552 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9228105490 0x7f922810bd20 unknown :-1 s=CLOSED pgs=241 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:16:22.139 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:22.139+0000 7f922ec00640 1 --2- 172.21.15.118:0/3046224552 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f9228104ad0 0x7f9228104f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:16:22.139 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:22.139+0000 7f922ec00640 1 --2- 172.21.15.118:0/3046224552 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f92281038e0 0x7f9228103d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:16:22.139 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:22.139+0000 7f922ec00640 1 -- 172.21.15.118:0/3046224552 >> 172.21.15.118:0/3046224552 conn(0x7f92280fee50 msgr2=0x7f9228101290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:16:22.139 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:22.139+0000 7f922ec00640 1 -- 172.21.15.118:0/3046224552 shutdown_connections 2024-03-24T00:16:22.139 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:22.139+0000 7f922ec00640 1 -- 172.21.15.118:0/3046224552 wait complete. 2024-03-24T00:16:22.140 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:22.140+0000 7f922ec00640 1 Processor -- start 2024-03-24T00:16:22.140 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:22.140+0000 7f922ec00640 1 -- start start 2024-03-24T00:16:22.141 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:22.141+0000 7f922ec00640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f92281038e0 0x7f9228199bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:16:22.141 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:22.141+0000 7f922ec00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f9228104ad0 0x7f922819a0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:16:22.141 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:22.141+0000 7f922ec00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9228105490 0x7f92281a1160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:16:22.141 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:22.141+0000 7f922ec00640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f922810f560 con 0x7f92281038e0 2024-03-24T00:16:22.141 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:22.141+0000 7f922ec00640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f922810f3e0 con 0x7f9228105490 2024-03-24T00:16:22.141 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:22.141+0000 7f922ec00640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f922810f6e0 con 0x7f9228104ad0 2024-03-24T00:16:22.141 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:22.141+0000 7f9227400640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f9228104ad0 0x7f922819a0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:16:22.141 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:22.141+0000 7f9227e00640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f92281038e0 0x7f9228199bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:16:22.141 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:22.141+0000 7f922cc00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9228105490 0x7f92281a1160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:16:22.142 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:22.141+0000 7f9227400640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f9228104ad0 0x7f922819a0f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.118:45742/0 (socket says 172.21.15.118:45742) 2024-03-24T00:16:22.142 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:22.141+0000 7f9227400640 1 -- 172.21.15.118:0/5788758 learned_addr learned my addr 172.21.15.118:0/5788758 (peer_addr_for_me v2:172.21.15.118:0/0) 2024-03-24T00:16:22.142 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:22.142+0000 7f922cc00640 1 -- 172.21.15.118:0/5788758 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f9228104ad0 msgr2=0x7f922819a0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:16:22.142 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:22.142+0000 7f922cc00640 1 --2- 172.21.15.118:0/5788758 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f9228104ad0 0x7f922819a0f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:16:22.142 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:22.142+0000 7f922cc00640 1 -- 172.21.15.118:0/5788758 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f92281038e0 msgr2=0x7f9228199bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:16:22.142 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:22.142+0000 7f922cc00640 1 --2- 172.21.15.118:0/5788758 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f92281038e0 0x7f9228199bb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:16:22.142 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:22.142+0000 7f922cc00640 1 -- 172.21.15.118:0/5788758 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f92281a17d0 con 0x7f9228105490 2024-03-24T00:16:22.142 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:22.142+0000 7f9227400640 1 --2- 172.21.15.118:0/5788758 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f9228104ad0 0x7f922819a0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T00:16:22.143 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:22.142+0000 7f922cc00640 1 --2- 172.21.15.118:0/5788758 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9228105490 0x7f92281a1160 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7f9220037040 tx=0x7f9220035d10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:16:22.143 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:22.142+0000 7f9224c00640 1 -- 172.21.15.118:0/5788758 <== mon.2 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9220039040 con 0x7f9228105490 2024-03-24T00:16:22.143 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:22.142+0000 7f9224c00640 1 -- 172.21.15.118:0/5788758 <== mon.2 v2:172.21.15.189:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f9220044070 con 0x7f9228105490 2024-03-24T00:16:22.143 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:22.142+0000 7f9224c00640 1 -- 172.21.15.118:0/5788758 <== mon.2 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9220004450 con 0x7f9228105490 2024-03-24T00:16:22.143 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:22.142+0000 7f922ec00640 1 -- 172.21.15.118:0/5788758 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f92281a1a60 con 0x7f9228105490 2024-03-24T00:16:22.143 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:22.142+0000 7f922ec00640 1 -- 172.21.15.118:0/5788758 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f92281a1f70 con 0x7f9228105490 2024-03-24T00:16:22.145 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:22.145+0000 7f92319ab740 1 Processor -- start 2024-03-24T00:16:22.145 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:22.146+0000 7f9224c00640 1 -- 172.21.15.118:0/5788758 <== mon.2 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 19) v1 ==== 137342+0+0 (secure 0 0 0) 0x7f92200045f0 con 0x7f9228105490 2024-03-24T00:16:22.146 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:22.146+0000 7f92319ab740 1 -- start start 2024-03-24T00:16:22.148 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:22.147+0000 7f92319ab740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x558116666e80 0x5581166672a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:16:22.148 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:22.147+0000 7f92319ab740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x558116667870 0x55811666c370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:16:22.148 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:22.147+0000 7f92319ab740 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55811666c9e0 0x55811666ee70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:16:22.148 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:22.147+0000 7f92319ab740 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x5581165cc250 con 0x55811666c9e0 2024-03-24T00:16:22.148 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:22.147+0000 7f92319ab740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x558116574a30 con 0x558116666e80 2024-03-24T00:16:22.148 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:22.147+0000 7f92319ab740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x5581165c82f0 con 0x558116667870 2024-03-24T00:16:22.148 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:22.147+0000 7f9227e00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x558116667870 0x55811666c370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:16:22.148 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:22.147+0000 7f9227e00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x558116667870 0x55811666c370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.118:45748/0 (socket says 172.21.15.118:45748) 2024-03-24T00:16:22.148 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:22.147+0000 7f9227e00640 1 -- 172.21.15.118:0/2728643750 learned_addr learned my addr 172.21.15.118:0/2728643750 (peer_addr_for_me v2:172.21.15.118:0/0) 2024-03-24T00:16:22.148 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:22.147+0000 7f9227400640 1 --2- 172.21.15.118:0/2728643750 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x558116666e80 0x5581166672a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:16:22.149 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:22.147+0000 7f9227e00640 1 -- 172.21.15.118:0/2728643750 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x558116666e80 msgr2=0x5581166672a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:16:22.149 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:22.147+0000 7f9227e00640 1 --2- 172.21.15.118:0/2728643750 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x558116666e80 0x5581166672a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:16:22.149 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:22.147+0000 7f9227400640 1 --2- 172.21.15.118:0/2728643750 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55811666c9e0 0x55811666ee70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:16:22.149 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:22.147+0000 7f9227e00640 1 -- 172.21.15.118:0/2728643750 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55811666c9e0 msgr2=0x55811666ee70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:16:22.149 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:22.147+0000 7f9227e00640 1 --2- 172.21.15.118:0/2728643750 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55811666c9e0 0x55811666ee70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:16:22.149 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:22.147+0000 7f9227e00640 1 -- 172.21.15.118:0/2728643750 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55811655f690 con 0x558116667870 2024-03-24T00:16:22.149 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:22.147+0000 7f9227400640 1 --2- 172.21.15.118:0/2728643750 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55811666c9e0 0x55811666ee70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-24T00:16:22.149 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:22.148+0000 7f9227e00640 1 --2- 172.21.15.118:0/2728643750 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x558116667870 0x55811666c370 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7f9218009ca0 tx=0x7f9218011810 comp rx=0 tx=0).ready entity=mon.1 client_cookie=b70cfd9f8f8cb78f server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:16:22.149 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:22.148+0000 7f9224c00640 1 --2- 172.21.15.118:0/5788758 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7f920c0a2910 0x7f920c0a4dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:16:22.149 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:22.148+0000 7f921d200640 1 -- 172.21.15.118:0/2728643750 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9218004220 con 0x558116667870 2024-03-24T00:16:22.149 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:22.148+0000 7f921d200640 1 -- 172.21.15.118:0/2728643750 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f92180043c0 con 0x558116667870 2024-03-24T00:16:22.149 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:22.148+0000 7f921d200640 1 -- 172.21.15.118:0/2728643750 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9218019970 con 0x558116667870 2024-03-24T00:16:22.149 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:22.148+0000 7f9224c00640 1 -- 172.21.15.118:0/5788758 <== mon.2 v2:172.21.15.189:3300/0 5 ==== osd_map(90..90 src has 1..90) v4 ==== 8277+0+0 (secure 0 0 0) 0x7f92200f1060 con 0x7f9228105490 2024-03-24T00:16:22.149 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:22.148+0000 7f9227400640 1 --2- 172.21.15.118:0/5788758 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7f920c0a2910 0x7f920c0a4dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:16:22.150 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:22.149+0000 7f92319ab740 1 -- 172.21.15.118:0/2728643750 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x558116667870 msgr2=0x55811666c370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:16:22.150 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:22.149+0000 7f92319ab740 1 --2- 172.21.15.118:0/2728643750 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x558116667870 0x55811666c370 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7f9218009ca0 tx=0x7f9218011810 comp rx=0 tx=0).stop 2024-03-24T00:16:22.153 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:22.149+0000 7f92319ab740 1 -- 172.21.15.118:0/2728643750 shutdown_connections 2024-03-24T00:16:22.153 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:22.149+0000 7f92319ab740 1 --2- 172.21.15.118:0/2728643750 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55811666c9e0 0x55811666ee70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:16:22.153 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:22.149+0000 7f92319ab740 1 --2- 172.21.15.118:0/2728643750 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x558116667870 0x55811666c370 unknown :-1 s=CLOSED pgs=228 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:16:22.153 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:22.149+0000 7f92319ab740 1 --2- 172.21.15.118:0/2728643750 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x558116666e80 0x5581166672a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:16:22.153 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:22.149+0000 7f92319ab740 1 -- 172.21.15.118:0/2728643750 >> 172.21.15.118:0/2728643750 conn(0x55811662f200 msgr2=0x5581166627f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:16:22.153 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:22.149+0000 7f92319ab740 1 -- 172.21.15.118:0/2728643750 shutdown_connections 2024-03-24T00:16:22.153 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:22.149+0000 7f92319ab740 1 -- 172.21.15.118:0/2728643750 wait complete. 2024-03-24T00:16:22.153 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:22.149+0000 7f9227400640 1 --2- 172.21.15.118:0/5788758 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7f920c0a2910 0x7f920c0a4dd0 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x5581166680d0 tx=0x7f921400b4a0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:16:22.153 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:22.149+0000 7f92319ab740 1 -- start start 2024-03-24T00:16:22.153 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:22.150+0000 7f92319ab740 1 Processor -- start 2024-03-24T00:16:22.153 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:22.150+0000 7f92319ab740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x558116666e80 0x558116670e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:16:22.154 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:22.150+0000 7f92319ab740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x558116667870 0x558116671390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:16:22.154 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:22.150+0000 7f92319ab740 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55811666c9e0 0x5581166783d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:16:22.154 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:22.150+0000 7f92319ab740 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x55811666f990 con 0x55811666c9e0 2024-03-24T00:16:22.154 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:22.150+0000 7f92319ab740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x558116587e80 con 0x558116667870 2024-03-24T00:16:22.154 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:22.150+0000 7f92319ab740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x55811666fb10 con 0x558116666e80 2024-03-24T00:16:22.154 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:22.150+0000 7f92319ab740 10 client.0 _ll_register_callbacks cb 0x55811659a130 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T00:16:22.154 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:22.150+0000 7f9227e00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x558116666e80 0x558116670e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:16:22.154 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:22.150+0000 7f9227400640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x558116667870 0x558116671390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:16:22.154 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:22.150+0000 7f9227e00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x558116666e80 0x558116670e50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.118:45764/0 (socket says 172.21.15.118:45764) 2024-03-24T00:16:22.154 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:22.150+0000 7f9227e00640 1 -- 172.21.15.118:0/779543132 learned_addr learned my addr 172.21.15.118:0/779543132 (peer_addr_for_me v2:172.21.15.118:0/0) 2024-03-24T00:16:22.154 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:22.150+0000 7f922cc00640 1 --2- 172.21.15.118:0/779543132 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55811666c9e0 0x5581166783d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:16:22.154 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:22.151+0000 7f9227e00640 1 -- 172.21.15.118:0/779543132 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x558116667870 msgr2=0x558116671390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:16:22.154 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:22.151+0000 7f9227e00640 1 --2- 172.21.15.118:0/779543132 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x558116667870 0x558116671390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:16:22.155 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:22.151+0000 7f9227e00640 1 -- 172.21.15.118:0/779543132 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55811666c9e0 msgr2=0x5581166783d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:16:22.155 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:22.151+0000 7f9227e00640 1 --2- 172.21.15.118:0/779543132 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55811666c9e0 0x5581166783d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:16:22.155 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:22.151+0000 7f9227e00640 1 -- 172.21.15.118:0/779543132 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5581163aa7d0 con 0x558116666e80 2024-03-24T00:16:22.155 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:22.151+0000 7f9227e00640 1 --2- 172.21.15.118:0/779543132 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x558116666e80 0x558116670e50 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7f9218009c70 tx=0x7f9218009f80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=8e0d07afbab5647b server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:16:22.155 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:22.151+0000 7f9211600640 10 client.0 ms_handle_connect on v2:172.21.15.193:3300/0 2024-03-24T00:16:22.155 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:22.151+0000 7f9211600640 1 -- 172.21.15.118:0/779543132 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9218028070 con 0x558116666e80 2024-03-24T00:16:22.155 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:22.151+0000 7f9211600640 1 -- 172.21.15.118:0/779543132 <== mon.1 v2:172.21.15.193:3300/0 2 ==== osd_map(90..90 src has 1..90) v4 ==== 8277+0+0 (secure 0 0 0) 0x7f9218003880 con 0x558116666e80 2024-03-24T00:16:22.155 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:22.151+0000 7f92319ab740 1 -- 172.21.15.118:0/779543132 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f92180043c0 con 0x558116666e80 2024-03-24T00:16:22.155 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:22.151+0000 7f9211600640 1 -- 172.21.15.118:0/779543132 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) v2 ==== 24+0+0 (secure 0 0 0) 0x7f9218034050 con 0x558116666e80 2024-03-24T00:16:22.155 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:22.152+0000 7f92319ab740 10 client.25591 fetch_fsmap learned FSMap version 28 2024-03-24T00:16:22.155 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:22.152+0000 7f92319ab740 1 -- 172.21.15.118:0/779543132 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) v3 -- 0x5581162e03f0 con 0x558116666e80 2024-03-24T00:16:22.155 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:22.152+0000 7f9211600640 1 -- 172.21.15.118:0/779543132 <== mon.1 v2:172.21.15.193:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f92180043c0 con 0x558116666e80 2024-03-24T00:16:22.155 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:22.152+0000 7f9211600640 1 -- 172.21.15.118:0/779543132 <== mon.1 v2:172.21.15.193:3300/0 5 ==== fsmap(e 28) v1 ==== 7934+0+0 (secure 0 0 0) 0x7f9218004b60 con 0x558116666e80 2024-03-24T00:16:22.155 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:22.152+0000 7f92319ab740 10 client.25591 fetch_fsmap finished waiting for FSMap version 28 2024-03-24T00:16:22.155 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:22.152+0000 7f92319ab740 10 client.25591 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-24T00:16:22.155 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:22.152+0000 7f92319ab740 20 client.25591 populate_metadata read hostname 'smithi118' 2024-03-24T00:16:22.156 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:22.152+0000 7f92319ab740 1 --2- 172.21.15.118:0/779543132 >> [v2:172.21.15.189:6838/4074484168,v1:172.21.15.189:6839/4074484168] conn(0x55811667cad0 0x55811667eec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:16:22.156 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:22.152+0000 7f92319ab740 4 client.25591 mds_command: new command op to 24569 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-24T00:16:22.156 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:22.152+0000 7f92319ab740 1 -- 172.21.15.118:0/779543132 --> [v2:172.21.15.189:6838/4074484168,v1:172.21.15.189:6839/4074484168] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x5581165d9790 con 0x55811667cad0 2024-03-24T00:16:22.156 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:22.152+0000 7f9227400640 1 --2- 172.21.15.118:0/779543132 >> [v2:172.21.15.189:6838/4074484168,v1:172.21.15.189:6839/4074484168] conn(0x55811667cad0 0x55811667eec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:16:22.156 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:22.153+0000 7f9227400640 1 --2- 172.21.15.118:0/779543132 >> [v2:172.21.15.189:6838/4074484168,v1:172.21.15.189:6839/4074484168] conn(0x55811667cad0 0x55811667eec0 crc :-1 s=READY pgs=47 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=b43f05995a17f0f3 server_cookie=9b40eace98ee37a2 in_seq=0 out_seq=0 2024-03-24T00:16:22.156 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:22.153+0000 7f9211600640 10 client.25591 ms_handle_connect on v2:172.21.15.189:6838/4074484168 2024-03-24T00:16:22.157 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:22.154+0000 7f9211600640 1 -- 172.21.15.118:0/779543132 <== mds.0 v2:172.21.15.189:6838/4074484168 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x5581165d9790 con 0x55811667cad0 2024-03-24T00:16:22.157 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:22.154+0000 7f9211600640 10 client.25591 handle_command_reply: tid=0 2024-03-24T00:16:22.157 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:22.154+0000 7f92319ab740 1 client.25591 shutdown 2024-03-24T00:16:22.157 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:22.154+0000 7f92319ab740 1 -- 172.21.15.118:0/779543132 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x558116666e80 msgr2=0x558116670e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:16:22.157 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:22.154+0000 7f92319ab740 1 --2- 172.21.15.118:0/779543132 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x558116666e80 0x558116670e50 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7f9218009c70 tx=0x7f9218009f80 comp rx=0 tx=0).stop 2024-03-24T00:16:22.157 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:22.154+0000 7f92319ab740 1 -- 172.21.15.118:0/779543132 shutdown_connections 2024-03-24T00:16:22.157 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:22.154+0000 7f92319ab740 1 --2- 172.21.15.118:0/779543132 >> [v2:172.21.15.189:6838/4074484168,v1:172.21.15.189:6839/4074484168] conn(0x55811667cad0 0x55811667eec0 crc :-1 s=READY pgs=47 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:16:22.158 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:22.154+0000 7f92319ab740 1 --2- 172.21.15.118:0/779543132 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55811666c9e0 0x5581166783d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:16:22.158 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:22.154+0000 7f92319ab740 1 --2- 172.21.15.118:0/779543132 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x558116667870 0x558116671390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:16:22.158 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:22.154+0000 7f9211600640 0 client.25591 ms_handle_reset on v2:172.21.15.189:6838/4074484168 2024-03-24T00:16:22.158 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:22.154+0000 7f92319ab740 1 --2- 172.21.15.118:0/779543132 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x558116666e80 0x558116670e50 unknown :-1 s=CLOSED pgs=229 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:16:22.158 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:22.154+0000 7f92319ab740 1 -- 172.21.15.118:0/779543132 >> 172.21.15.118:0/779543132 conn(0x55811662f200 msgr2=0x558116661a90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:16:22.158 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:22.155+0000 7f92319ab740 1 -- 172.21.15.118:0/779543132 shutdown_connections 2024-03-24T00:16:22.158 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:22.155+0000 7f92319ab740 1 -- 172.21.15.118:0/779543132 wait complete. 2024-03-24T00:16:22.158 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:22.155+0000 7f92319ab740 20 client.25591 trim_cache size 0 max 16384 2024-03-24T00:16:22.172 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:22.172+0000 7f92319ab740 1 Processor -- start 2024-03-24T00:16:22.172 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:22.172+0000 7f92319ab740 1 -- start start 2024-03-24T00:16:22.172 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:22.172+0000 7f92319ab740 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x558116666e80 0x558116665a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:16:22.172 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:22.172+0000 7f92319ab740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x558116667870 0x5581166782f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:16:22.172 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:22.172+0000 7f92319ab740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55811666c9e0 0x558116665f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:16:22.172 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:22.172+0000 7f92319ab740 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f92281aab80 con 0x558116666e80 2024-03-24T00:16:22.172 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:22.172+0000 7f92319ab740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f92281ac500 con 0x558116667870 2024-03-24T00:16:22.172 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:22.172+0000 7f92319ab740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f92281ab1e0 con 0x55811666c9e0 2024-03-24T00:16:22.174 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:22.173+0000 7f9227400640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x558116667870 0x5581166782f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:16:22.174 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:22.173+0000 7f9227e00640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x558116666e80 0x558116665a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:16:22.174 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:22.173+0000 7f922cc00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55811666c9e0 0x558116665f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:16:22.174 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:22.173+0000 7f9227400640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x558116667870 0x5581166782f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.118:60572/0 (socket says 172.21.15.118:60572) 2024-03-24T00:16:22.174 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:22.173+0000 7f9227400640 1 -- 172.21.15.118:0/3954184857 learned_addr learned my addr 172.21.15.118:0/3954184857 (peer_addr_for_me v2:172.21.15.118:0/0) 2024-03-24T00:16:22.174 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:22.173+0000 7f9227400640 1 -- 172.21.15.118:0/3954184857 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55811666c9e0 msgr2=0x558116665f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:16:22.175 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:22.173+0000 7f9227400640 1 --2- 172.21.15.118:0/3954184857 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55811666c9e0 0x558116665f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:16:22.175 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:22.173+0000 7f9227400640 1 -- 172.21.15.118:0/3954184857 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x558116666e80 msgr2=0x558116665a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:16:22.175 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:22.173+0000 7f9227400640 1 --2- 172.21.15.118:0/3954184857 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x558116666e80 0x558116665a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:16:22.175 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:22.173+0000 7f9227400640 1 -- 172.21.15.118:0/3954184857 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5581162a00f0 con 0x558116667870 2024-03-24T00:16:22.175 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:22.173+0000 7f922cc00640 1 --2- 172.21.15.118:0/3954184857 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55811666c9e0 0x558116665f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T00:16:22.175 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:22.173+0000 7f9227400640 1 --2- 172.21.15.118:0/3954184857 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x558116667870 0x5581166782f0 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7f9214005d00 tx=0x7f9214014040 comp rx=0 tx=0).ready entity=mon.2 client_cookie=bf2cf24a42fcfa53 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:16:22.175 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:22.174+0000 7f9213e00640 1 -- 172.21.15.118:0/3954184857 <== mon.2 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9214028230 con 0x558116667870 2024-03-24T00:16:22.175 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:22.174+0000 7f9213e00640 1 -- 172.21.15.118:0/3954184857 <== mon.2 v2:172.21.15.189:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f921401c890 con 0x558116667870 2024-03-24T00:16:22.175 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:22.174+0000 7f9213e00640 1 -- 172.21.15.118:0/3954184857 <== mon.2 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9214005260 con 0x558116667870 2024-03-24T00:16:22.175 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:22.174+0000 7f92319ab740 1 -- 172.21.15.118:0/3954184857 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x558116667870 msgr2=0x5581166782f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:16:22.175 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:22.174+0000 7f92319ab740 1 --2- 172.21.15.118:0/3954184857 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x558116667870 0x5581166782f0 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7f9214005d00 tx=0x7f9214014040 comp rx=0 tx=0).stop 2024-03-24T00:16:22.175 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:22.174+0000 7f92319ab740 1 -- 172.21.15.118:0/3954184857 shutdown_connections 2024-03-24T00:16:22.175 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:22.174+0000 7f92319ab740 1 --2- 172.21.15.118:0/3954184857 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55811666c9e0 0x558116665f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:16:22.175 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:22.174+0000 7f92319ab740 1 --2- 172.21.15.118:0/3954184857 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x558116667870 0x5581166782f0 unknown :-1 s=CLOSED pgs=243 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:16:22.175 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:22.174+0000 7f92319ab740 1 --2- 172.21.15.118:0/3954184857 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x558116666e80 0x558116665a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:16:22.175 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:22.174+0000 7f92319ab740 1 -- 172.21.15.118:0/3954184857 >> 172.21.15.118:0/3954184857 conn(0x55811662f200 msgr2=0x558116668b70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:16:22.176 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:22.174+0000 7f92319ab740 1 -- 172.21.15.118:0/3954184857 shutdown_connections 2024-03-24T00:16:22.176 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:22.174+0000 7f92319ab740 1 -- 172.21.15.118:0/3954184857 wait complete. 2024-03-24T00:16:22.176 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:22.175+0000 7f92319ab740 1 -- start start 2024-03-24T00:16:22.176 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:22.175+0000 7f92319ab740 1 Processor -- start 2024-03-24T00:16:22.179 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:22.175+0000 7f92319ab740 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x558116666e80 0x5581166755d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:16:22.179 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:22.175+0000 7f92319ab740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x558116667870 0x558116675b10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:16:22.179 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:22.175+0000 7f92319ab740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55811666c9e0 0x558116676b20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:16:22.179 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:22.175+0000 7f92319ab740 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f92281abea0 con 0x558116666e80 2024-03-24T00:16:22.179 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:22.175+0000 7f92319ab740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f92281ab840 con 0x55811666c9e0 2024-03-24T00:16:22.179 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:22.175+0000 7f92319ab740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f92281a8d40 con 0x558116667870 2024-03-24T00:16:22.179 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:22.175+0000 7f92319ab740 10 client.0 _ll_register_callbacks cb 0x7f9218009af0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T00:16:22.179 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:22.176+0000 7f9227400640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x558116667870 0x558116675b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:16:22.179 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:22.176+0000 7f9227e00640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x558116666e80 0x5581166755d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:16:22.179 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:22.176+0000 7f9227400640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x558116667870 0x558116675b10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.118:45786/0 (socket says 172.21.15.118:45786) 2024-03-24T00:16:22.179 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:22.176+0000 7f9227400640 1 -- 172.21.15.118:0/3919034227 learned_addr learned my addr 172.21.15.118:0/3919034227 (peer_addr_for_me v2:172.21.15.118:0/0) 2024-03-24T00:16:22.179 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:22.176+0000 7f922cc00640 1 --2- 172.21.15.118:0/3919034227 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55811666c9e0 0x558116676b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:16:22.180 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:22.176+0000 7f9227400640 1 -- 172.21.15.118:0/3919034227 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55811666c9e0 msgr2=0x558116676b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:16:22.180 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:22.176+0000 7f9227400640 1 --2- 172.21.15.118:0/3919034227 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55811666c9e0 0x558116676b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:16:22.180 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:22.176+0000 7f9227400640 1 -- 172.21.15.118:0/3919034227 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x558116666e80 msgr2=0x5581166755d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:16:22.180 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:22.176+0000 7f9227400640 1 --2- 172.21.15.118:0/3919034227 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x558116666e80 0x5581166755d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:16:22.180 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:22.176+0000 7f9227400640 1 -- 172.21.15.118:0/3919034227 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x558116677110 con 0x558116667870 2024-03-24T00:16:22.180 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:22.176+0000 7f9227e00640 1 --2- 172.21.15.118:0/3919034227 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x558116666e80 0x5581166755d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T00:16:22.180 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:22.176+0000 7f9227400640 1 --2- 172.21.15.118:0/3919034227 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x558116667870 0x558116675b10 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7f921400be90 tx=0x7f921401c3f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=641df483c4c9a773 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:16:22.180 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:22.176+0000 7f9211600640 10 client.0 ms_handle_connect on v2:172.21.15.193:3300/0 2024-03-24T00:16:22.180 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:22.177+0000 7f9211600640 1 -- 172.21.15.118:0/3919034227 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9214035020 con 0x558116667870 2024-03-24T00:16:22.180 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:22.177+0000 7f9211600640 1 -- 172.21.15.118:0/3919034227 <== mon.1 v2:172.21.15.193:3300/0 2 ==== osd_map(90..90 src has 1..90) v4 ==== 8277+0+0 (secure 0 0 0) 0x7f9214012980 con 0x558116667870 2024-03-24T00:16:22.180 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:22.177+0000 7f92319ab740 1 -- 172.21.15.118:0/3919034227 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f921401c890 con 0x558116667870 2024-03-24T00:16:22.180 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:22.177+0000 7f9211600640 1 -- 172.21.15.118:0/3919034227 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) v2 ==== 24+0+0 (secure 0 0 0) 0x7f9214040050 con 0x558116667870 2024-03-24T00:16:22.180 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:22.177+0000 7f92319ab740 10 client.25597 fetch_fsmap learned FSMap version 28 2024-03-24T00:16:22.180 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:22.177+0000 7f92319ab740 1 -- 172.21.15.118:0/3919034227 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) v3 -- 0x558116678450 con 0x558116667870 2024-03-24T00:16:22.180 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:22.177+0000 7f9211600640 1 -- 172.21.15.118:0/3919034227 <== mon.1 v2:172.21.15.193:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f921401c890 con 0x558116667870 2024-03-24T00:16:22.181 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:22.177+0000 7f9211600640 1 -- 172.21.15.118:0/3919034227 <== mon.1 v2:172.21.15.193:3300/0 5 ==== fsmap(e 28) v1 ==== 7934+0+0 (secure 0 0 0) 0x7f9214030030 con 0x558116667870 2024-03-24T00:16:22.181 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:22.177+0000 7f92319ab740 10 client.25597 fetch_fsmap finished waiting for FSMap version 28 2024-03-24T00:16:22.181 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:22.177+0000 7f92319ab740 10 client.25597 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-24T00:16:22.181 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:22.177+0000 7f92319ab740 20 client.25597 populate_metadata read hostname 'smithi118' 2024-03-24T00:16:22.181 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:22.177+0000 7f92319ab740 1 --2- 172.21.15.118:0/3919034227 >> [v2:172.21.15.189:6838/4074484168,v1:172.21.15.189:6839/4074484168] conn(0x5581166d8660 0x5581166daa80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:16:22.181 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:22.177+0000 7f92319ab740 4 client.25597 mds_command: new command op to 24569 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-24T00:16:22.181 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:22.177+0000 7f92319ab740 1 -- 172.21.15.118:0/3919034227 --> [v2:172.21.15.189:6838/4074484168,v1:172.21.15.189:6839/4074484168] -- command(tid 0: {"prefix": "scrub status"}) v1 -- 0x5581165d3ae0 con 0x5581166d8660 2024-03-24T00:16:22.181 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:22.178+0000 7f9227e00640 1 --2- 172.21.15.118:0/3919034227 >> [v2:172.21.15.189:6838/4074484168,v1:172.21.15.189:6839/4074484168] conn(0x5581166d8660 0x5581166daa80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:16:22.181 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:22.178+0000 7f9227e00640 1 --2- 172.21.15.118:0/3919034227 >> [v2:172.21.15.189:6838/4074484168,v1:172.21.15.189:6839/4074484168] conn(0x5581166d8660 0x5581166daa80 crc :-1 s=READY pgs=48 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=233a8334a00e721c server_cookie=b99572fd9f0e02fa in_seq=0 out_seq=0 2024-03-24T00:16:22.181 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:22.178+0000 7f9211600640 10 client.25597 ms_handle_connect on v2:172.21.15.189:6838/4074484168 2024-03-24T00:16:22.181 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:22.178+0000 7f9211600640 1 -- 172.21.15.118:0/3919034227 <== mds.0 v2:172.21.15.189:6838/4074484168 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+259 (crc 0 0 0) 0x5581165d3ae0 con 0x5581166d8660 2024-03-24T00:16:22.181 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:22.178+0000 7f9211600640 10 client.25597 handle_command_reply: tid=0 2024-03-24T00:16:22.181 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:22.178+0000 7f92319ab740 1 client.25597 shutdown 2024-03-24T00:16:22.181 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:22.178+0000 7f92319ab740 1 -- 172.21.15.118:0/3919034227 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x558116667870 msgr2=0x558116675b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:16:22.181 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:22.178+0000 7f92319ab740 1 --2- 172.21.15.118:0/3919034227 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x558116667870 0x558116675b10 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7f921400be90 tx=0x7f921401c3f0 comp rx=0 tx=0).stop 2024-03-24T00:16:22.182 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:22.178+0000 7f92319ab740 1 -- 172.21.15.118:0/3919034227 shutdown_connections 2024-03-24T00:16:22.182 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:22.178+0000 7f92319ab740 1 --2- 172.21.15.118:0/3919034227 >> [v2:172.21.15.189:6838/4074484168,v1:172.21.15.189:6839/4074484168] conn(0x5581166d8660 0x5581166daa80 crc :-1 s=READY pgs=48 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:16:22.182 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:22.178+0000 7f92319ab740 1 --2- 172.21.15.118:0/3919034227 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55811666c9e0 0x558116676b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:16:22.182 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:22.178+0000 7f92319ab740 1 --2- 172.21.15.118:0/3919034227 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x558116667870 0x558116675b10 unknown :-1 s=CLOSED pgs=230 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:16:22.182 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:22.178+0000 7f92319ab740 1 --2- 172.21.15.118:0/3919034227 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x558116666e80 0x5581166755d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:16:22.182 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:22.178+0000 7f9211600640 0 client.25597 ms_handle_reset on v2:172.21.15.189:6838/4074484168 2024-03-24T00:16:22.182 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:22.178+0000 7f92319ab740 1 -- 172.21.15.118:0/3919034227 >> 172.21.15.118:0/3919034227 conn(0x55811662f200 msgr2=0x558116661200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:16:22.182 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:22.179+0000 7f92319ab740 1 -- 172.21.15.118:0/3919034227 shutdown_connections 2024-03-24T00:16:22.182 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:22.179+0000 7f92319ab740 1 -- 172.21.15.118:0/3919034227 wait complete. 2024-03-24T00:16:22.182 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:22.179+0000 7f92319ab740 20 client.25597 trim_cache size 0 max 16384 2024-03-24T00:16:22.182 INFO:teuthology.orchestra.run.smithi118.stdout:{ 2024-03-24T00:16:22.183 INFO:teuthology.orchestra.run.smithi118.stdout: "status": "IDLE (waiting for more scrubs)", 2024-03-24T00:16:22.183 INFO:teuthology.orchestra.run.smithi118.stdout: "scrubs": { 2024-03-24T00:16:22.183 INFO:teuthology.orchestra.run.smithi118.stdout: "59f621ed-f0ed-4c9b-85bb-5ea67ef1ec18": { 2024-03-24T00:16:22.183 INFO:teuthology.orchestra.run.smithi118.stdout: "path": "/", 2024-03-24T00:16:22.183 INFO:teuthology.orchestra.run.smithi118.stdout: "tag": "59f621ed-f0ed-4c9b-85bb-5ea67ef1ec18", 2024-03-24T00:16:22.183 INFO:teuthology.orchestra.run.smithi118.stdout: "options": "recursive,force" 2024-03-24T00:16:22.183 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-03-24T00:16:22.183 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-03-24T00:16:22.183 INFO:teuthology.orchestra.run.smithi118.stdout:} 2024-03-24T00:16:22.184 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:22.179+0000 7f922ec00640 1 -- 172.21.15.118:0/5788758 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7f920c0a2910 msgr2=0x7f920c0a4dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:16:22.184 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:22.179+0000 7f922ec00640 1 --2- 172.21.15.118:0/5788758 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7f920c0a2910 0x7f920c0a4dd0 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x5581166680d0 tx=0x7f921400b4a0 comp rx=0 tx=0).stop 2024-03-24T00:16:22.184 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:22.179+0000 7f922ec00640 1 -- 172.21.15.118:0/5788758 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9228105490 msgr2=0x7f92281a1160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:16:22.184 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:22.179+0000 7f922ec00640 1 --2- 172.21.15.118:0/5788758 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9228105490 0x7f92281a1160 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7f9220037040 tx=0x7f9220035d10 comp rx=0 tx=0).stop 2024-03-24T00:16:22.184 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:22.179+0000 7f922ec00640 1 -- 172.21.15.118:0/5788758 shutdown_connections 2024-03-24T00:16:22.184 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:22.179+0000 7f922ec00640 1 --2- 172.21.15.118:0/5788758 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7f920c0a2910 0x7f920c0a4dd0 unknown :-1 s=CLOSED pgs=281 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:16:22.184 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:22.179+0000 7f922ec00640 1 --2- 172.21.15.118:0/5788758 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9228105490 0x7f92281a1160 unknown :-1 s=CLOSED pgs=242 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:16:22.184 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:22.179+0000 7f922ec00640 1 --2- 172.21.15.118:0/5788758 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f9228104ad0 0x7f922819a0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:16:22.184 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:22.179+0000 7f922ec00640 1 --2- 172.21.15.118:0/5788758 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f92281038e0 0x7f9228199bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:16:22.184 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:22.179+0000 7f922ec00640 1 -- 172.21.15.118:0/5788758 >> 172.21.15.118:0/5788758 conn(0x7f92280fee50 msgr2=0x7f9228107dd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:16:22.184 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:22.180+0000 7f922ec00640 1 -- 172.21.15.118:0/5788758 shutdown_connections 2024-03-24T00:16:22.185 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:22.180+0000 7f922ec00640 1 -- 172.21.15.118:0/5788758 wait complete. 2024-03-24T00:16:22.199 INFO:tasks.cephfs.filesystem:scrub status for tag:59f621ed-f0ed-4c9b-85bb-5ea67ef1ec18 - {'path': '/', 'tag': '59f621ed-f0ed-4c9b-85bb-5ea67ef1ec18', 'options': 'recursive,force'} 2024-03-24T00:16:23.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:16:22 smithi118 ceph-mon[31535]: pgmap v418: 97 pgs: 97 active+clean; 662 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 255 B/s wr, 9 op/s 2024-03-24T00:16:23.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:16:22 smithi118 ceph-mon[31535]: scrub queued for path: / 2024-03-24T00:16:23.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:16:22 smithi118 ceph-mon[31535]: scrub summary: idle+waiting paths [/] 2024-03-24T00:16:23.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:16:22 smithi118 ceph-mon[31535]: scrub summary: active paths [/] 2024-03-24T00:16:23.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:16:22 smithi118 ceph-mon[31535]: scrub summary: idle+waiting paths [/] 2024-03-24T00:16:23.203 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:16:22 smithi189 ceph-mon[36187]: pgmap v418: 97 pgs: 97 active+clean; 662 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 255 B/s wr, 9 op/s 2024-03-24T00:16:23.203 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:16:22 smithi189 ceph-mon[36187]: scrub queued for path: / 2024-03-24T00:16:23.203 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:16:22 smithi189 ceph-mon[36187]: scrub summary: idle+waiting paths [/] 2024-03-24T00:16:23.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:16:22 smithi189 ceph-mon[36187]: scrub summary: active paths [/] 2024-03-24T00:16:23.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:16:22 smithi189 ceph-mon[36187]: scrub summary: idle+waiting paths [/] 2024-03-24T00:16:23.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:16:22 smithi193 ceph-mon[36878]: pgmap v418: 97 pgs: 97 active+clean; 662 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 255 B/s wr, 9 op/s 2024-03-24T00:16:23.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:16:22 smithi193 ceph-mon[36878]: scrub queued for path: / 2024-03-24T00:16:23.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:16:22 smithi193 ceph-mon[36878]: scrub summary: idle+waiting paths [/] 2024-03-24T00:16:23.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:16:22 smithi193 ceph-mon[36878]: scrub summary: active paths [/] 2024-03-24T00:16:23.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:16:22 smithi193 ceph-mon[36878]: scrub summary: idle+waiting paths [/] 2024-03-24T00:16:25.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:16:24 smithi118 ceph-mon[31535]: pgmap v419: 97 pgs: 97 active+clean; 662 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 5.7 KiB/s rd, 341 B/s wr, 11 op/s 2024-03-24T00:16:25.203 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:16:24 smithi189 ceph-mon[36187]: pgmap v419: 97 pgs: 97 active+clean; 662 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 5.7 KiB/s rd, 341 B/s wr, 11 op/s 2024-03-24T00:16:25.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:16:24 smithi193 ceph-mon[36878]: pgmap v419: 97 pgs: 97 active+clean; 662 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 5.7 KiB/s rd, 341 B/s wr, 11 op/s 2024-03-24T00:16:26.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:16:25 smithi118 ceph-mon[31535]: scrub summary: idle 2024-03-24T00:16:26.203 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:16:25 smithi189 ceph-mon[36187]: scrub summary: idle 2024-03-24T00:16:26.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:16:25 smithi193 ceph-mon[36878]: scrub summary: idle 2024-03-24T00:16:27.097 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:16:26 smithi118 ceph-mon[31535]: pgmap v420: 97 pgs: 97 active+clean; 671 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 7.4 KiB/s rd, 1.2 KiB/s wr, 12 op/s 2024-03-24T00:16:27.203 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:16:26 smithi189 ceph-mon[36187]: pgmap v420: 97 pgs: 97 active+clean; 671 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 7.4 KiB/s rd, 1.2 KiB/s wr, 12 op/s 2024-03-24T00:16:27.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:16:26 smithi193 ceph-mon[36878]: pgmap v420: 97 pgs: 97 active+clean; 671 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 7.4 KiB/s rd, 1.2 KiB/s wr, 12 op/s 2024-03-24T00:16:27.616 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:16:27.616+0000 7fd16b200640 1 -- 172.21.15.118:0/1852651225 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fd16c101fd0 msgr2=0x7fd16c1043c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:16:27.617 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:16:27.616+0000 7fd16b200640 1 --2- 172.21.15.118:0/1852651225 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fd16c101fd0 0x7fd16c1043c0 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7fd15c009900 tx=0x7fd15c02bc40 comp rx=0 tx=0).stop 2024-03-24T00:16:27.617 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:16:27.616+0000 7fd16b200640 1 -- 172.21.15.118:0/1852651225 shutdown_connections 2024-03-24T00:16:27.617 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:16:27.616+0000 7fd16b200640 1 --2- 172.21.15.118:0/1852651225 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fd16c107230 0x7fd16c109620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:16:27.617 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:16:27.616+0000 7fd16b200640 1 --2- 172.21.15.118:0/1852651225 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd16c104900 0x7fd16c106cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:16:27.617 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:16:27.616+0000 7fd16b200640 1 --2- 172.21.15.118:0/1852651225 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fd16c101fd0 0x7fd16c1043c0 unknown :-1 s=CLOSED pgs=231 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:16:27.617 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:16:27.616+0000 7fd16b200640 1 -- 172.21.15.118:0/1852651225 >> 172.21.15.118:0/1852651225 conn(0x7fd16c0fb980 msgr2=0x7fd16c0fdda0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:16:27.617 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:16:27.617+0000 7fd16b200640 1 -- 172.21.15.118:0/1852651225 shutdown_connections 2024-03-24T00:16:27.617 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:16:27.617+0000 7fd16b200640 1 -- 172.21.15.118:0/1852651225 wait complete. 2024-03-24T00:16:27.617 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:16:27.618+0000 7fd16b200640 1 Processor -- start 2024-03-24T00:16:27.618 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:16:27.618+0000 7fd16b200640 1 -- start start 2024-03-24T00:16:27.618 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:16:27.619+0000 7fd16b200640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fd16c101fd0 0x7fd16c199a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:16:27.619 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:16:27.619+0000 7fd16b200640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fd16c104900 0x7fd16c199fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:16:27.619 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:16:27.619+0000 7fd16b200640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd16c107230 0x7fd16c1a1030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:16:27.619 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:16:27.619+0000 7fd16b200640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd16c10d1d0 con 0x7fd16c101fd0 2024-03-24T00:16:27.619 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:16:27.619+0000 7fd16b200640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd16c10d050 con 0x7fd16c107230 2024-03-24T00:16:27.619 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:16:27.619+0000 7fd16b200640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd16c10d350 con 0x7fd16c104900 2024-03-24T00:16:27.619 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:16:27.619+0000 7fd169e00640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fd16c101fd0 0x7fd16c199a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:16:27.619 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:16:27.619+0000 7fd16a800640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd16c107230 0x7fd16c1a1030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:16:27.619 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:16:27.619+0000 7fd169e00640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fd16c101fd0 0x7fd16c199a80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.118:3300/0 says I am v2:172.21.15.118:45762/0 (socket says 172.21.15.118:45762) 2024-03-24T00:16:27.619 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:16:27.619+0000 7fd169400640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fd16c104900 0x7fd16c199fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:16:27.619 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:16:27.619+0000 7fd169e00640 1 -- 172.21.15.118:0/531536702 learned_addr learned my addr 172.21.15.118:0/531536702 (peer_addr_for_me v2:172.21.15.118:0/0) 2024-03-24T00:16:27.620 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:16:27.619+0000 7fd169400640 1 -- 172.21.15.118:0/531536702 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd16c107230 msgr2=0x7fd16c1a1030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:16:27.620 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:16:27.619+0000 7fd169400640 1 --2- 172.21.15.118:0/531536702 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd16c107230 0x7fd16c1a1030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:16:27.621 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:16:27.619+0000 7fd169400640 1 -- 172.21.15.118:0/531536702 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fd16c101fd0 msgr2=0x7fd16c199a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:16:27.621 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:16:27.620+0000 7fd169400640 1 --2- 172.21.15.118:0/531536702 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fd16c101fd0 0x7fd16c199a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:16:27.621 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:16:27.620+0000 7fd169400640 1 -- 172.21.15.118:0/531536702 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd16c1a16a0 con 0x7fd16c104900 2024-03-24T00:16:27.621 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:16:27.620+0000 7fd169e00640 1 --2- 172.21.15.118:0/531536702 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fd16c101fd0 0x7fd16c199a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T00:16:27.621 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:16:27.620+0000 7fd169400640 1 --2- 172.21.15.118:0/531536702 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fd16c104900 0x7fd16c199fc0 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7fd15400e9c0 tx=0x7fd15400ee80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:16:27.621 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:16:27.620+0000 7fd162a00640 1 -- 172.21.15.118:0/531536702 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd15400cd80 con 0x7fd16c104900 2024-03-24T00:16:27.621 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:16:27.620+0000 7fd162a00640 1 -- 172.21.15.118:0/531536702 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fd1540035f0 con 0x7fd16c104900 2024-03-24T00:16:27.621 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:16:27.620+0000 7fd16b200640 1 -- 172.21.15.118:0/531536702 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd16c1a1990 con 0x7fd16c104900 2024-03-24T00:16:27.621 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:16:27.620+0000 7fd162a00640 1 -- 172.21.15.118:0/531536702 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd154010690 con 0x7fd16c104900 2024-03-24T00:16:27.621 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:16:27.620+0000 7fd16b200640 1 -- 172.21.15.118:0/531536702 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd16c1a1ee0 con 0x7fd16c104900 2024-03-24T00:16:27.623 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:16:27.623+0000 7fd173fd8740 1 Processor -- start 2024-03-24T00:16:27.624 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:16:27.624+0000 7fd173fd8740 1 -- start start 2024-03-24T00:16:27.624 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:16:27.624+0000 7fd173fd8740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5559e42ae3f0 0x5559e42ae810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:16:27.624 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:16:27.624+0000 7fd173fd8740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5559e42aede0 0x5559e42b38e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:16:27.624 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:16:27.624+0000 7fd173fd8740 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5559e42b3f50 0x5559e42b63e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:16:27.624 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:16:27.624+0000 7fd173fd8740 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x5559e42713b0 con 0x5559e42b3f50 2024-03-24T00:16:27.624 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:16:27.624+0000 7fd173fd8740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x5559e4213830 con 0x5559e42ae3f0 2024-03-24T00:16:27.624 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:16:27.624+0000 7fd173fd8740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x5559e41bc5d0 con 0x5559e42aede0 2024-03-24T00:16:27.624 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:16:27.624+0000 7fd162a00640 1 -- 172.21.15.118:0/531536702 <== mon.1 v2:172.21.15.193:3300/0 4 ==== mgrmap(e 19) v1 ==== 137342+0+0 (secure 0 0 0) 0x7fd1540037a0 con 0x7fd16c104900 2024-03-24T00:16:27.624 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:16:27.624+0000 7fd16a800640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5559e42ae3f0 0x5559e42ae810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:16:27.624 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:16:27.624+0000 7fd169e00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5559e42aede0 0x5559e42b38e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:16:27.624 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:16:27.624+0000 7fd16a800640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5559e42ae3f0 0x5559e42ae810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.118:35102/0 (socket says 172.21.15.118:35102) 2024-03-24T00:16:27.624 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:16:27.624+0000 7fd16a800640 1 -- 172.21.15.118:0/2958674242 learned_addr learned my addr 172.21.15.118:0/2958674242 (peer_addr_for_me v2:172.21.15.118:0/0) 2024-03-24T00:16:27.626 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:16:27.625+0000 7fd169400640 1 --2- 172.21.15.118:0/2958674242 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5559e42b3f50 0x5559e42b63e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:16:27.627 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:16:27.625+0000 7fd16a800640 1 -- 172.21.15.118:0/2958674242 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5559e42aede0 msgr2=0x5559e42b38e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:16:27.627 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:16:27.625+0000 7fd16a800640 1 --2- 172.21.15.118:0/2958674242 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5559e42aede0 0x5559e42b38e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:16:27.627 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:16:27.625+0000 7fd16a800640 1 -- 172.21.15.118:0/2958674242 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5559e42b3f50 msgr2=0x5559e42b63e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:16:27.627 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:16:27.625+0000 7fd16a800640 1 --2- 172.21.15.118:0/2958674242 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5559e42b3f50 0x5559e42b63e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:16:27.627 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:16:27.625+0000 7fd16a800640 1 -- 172.21.15.118:0/2958674242 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5559e41d42c0 con 0x5559e42ae3f0 2024-03-24T00:16:27.627 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:16:27.625+0000 7fd169e00640 1 --2- 172.21.15.118:0/2958674242 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5559e42aede0 0x5559e42b38e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T00:16:27.627 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:16:27.625+0000 7fd169400640 1 --2- 172.21.15.118:0/2958674242 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5559e42b3f50 0x5559e42b63e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-24T00:16:27.627 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:16:27.625+0000 7fd16a800640 1 --2- 172.21.15.118:0/2958674242 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5559e42ae3f0 0x5559e42ae810 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7fd164010370 tx=0x7fd164010830 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a1e1b12dc8ea32e4 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:16:27.627 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:16:27.625+0000 7fd15b400640 1 -- 172.21.15.118:0/2958674242 <== mon.2 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd164016070 con 0x5559e42ae3f0 2024-03-24T00:16:27.627 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:16:27.625+0000 7fd15b400640 1 -- 172.21.15.118:0/2958674242 <== mon.2 v2:172.21.15.189:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fd164004d20 con 0x5559e42ae3f0 2024-03-24T00:16:27.627 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:16:27.625+0000 7fd15b400640 1 -- 172.21.15.118:0/2958674242 <== mon.2 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd1640053e0 con 0x5559e42ae3f0 2024-03-24T00:16:27.627 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:16:27.626+0000 7fd173fd8740 1 -- 172.21.15.118:0/2958674242 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5559e42ae3f0 msgr2=0x5559e42ae810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:16:27.627 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:16:27.626+0000 7fd173fd8740 1 --2- 172.21.15.118:0/2958674242 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5559e42ae3f0 0x5559e42ae810 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7fd164010370 tx=0x7fd164010830 comp rx=0 tx=0).stop 2024-03-24T00:16:27.628 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:16:27.626+0000 7fd173fd8740 1 -- 172.21.15.118:0/2958674242 shutdown_connections 2024-03-24T00:16:27.628 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:16:27.626+0000 7fd173fd8740 1 --2- 172.21.15.118:0/2958674242 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5559e42b3f50 0x5559e42b63e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:16:27.628 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:16:27.626+0000 7fd173fd8740 1 --2- 172.21.15.118:0/2958674242 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5559e42aede0 0x5559e42b38e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:16:27.628 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:16:27.626+0000 7fd173fd8740 1 --2- 172.21.15.118:0/2958674242 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5559e42ae3f0 0x5559e42ae810 unknown :-1 s=CLOSED pgs=244 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:16:27.628 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:16:27.626+0000 7fd173fd8740 1 -- 172.21.15.118:0/2958674242 >> 172.21.15.118:0/2958674242 conn(0x5559e42a9960 msgr2=0x5559e42a9d60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:16:27.628 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:16:27.626+0000 7fd173fd8740 1 -- 172.21.15.118:0/2958674242 shutdown_connections 2024-03-24T00:16:27.628 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:16:27.626+0000 7fd162a00640 1 --2- 172.21.15.118:0/531536702 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7fd1500a2830 0x7fd1500a4cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:16:27.628 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:16:27.626+0000 7fd173fd8740 1 -- 172.21.15.118:0/2958674242 wait complete. 2024-03-24T00:16:27.628 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:16:27.627+0000 7fd162a00640 1 -- 172.21.15.118:0/531536702 <== mon.1 v2:172.21.15.193:3300/0 5 ==== osd_map(90..90 src has 1..90) v4 ==== 8277+0+0 (secure 0 0 0) 0x7fd1540ceef0 con 0x7fd16c104900 2024-03-24T00:16:27.628 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:16:27.627+0000 7fd169e00640 1 --2- 172.21.15.118:0/531536702 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7fd1500a2830 0x7fd1500a4cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:16:27.628 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:16:27.627+0000 7fd173fd8740 1 -- start start 2024-03-24T00:16:27.628 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:16:27.627+0000 7fd173fd8740 1 Processor -- start 2024-03-24T00:16:27.632 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:16:27.628+0000 7fd169e00640 1 --2- 172.21.15.118:0/531536702 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7fd1500a2830 0x7fd1500a4cf0 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7fd15c006010 tx=0x7fd15c02e040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:16:27.632 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:16:27.628+0000 7fd173fd8740 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5559e42ae3f0 0x5559e42b8620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:16:27.632 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:16:27.628+0000 7fd173fd8740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5559e42aede0 0x5559e42b8b60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:16:27.632 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:16:27.628+0000 7fd173fd8740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5559e42b3f50 0x5559e42bfba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:16:27.632 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:16:27.628+0000 7fd173fd8740 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x5559e41cfa20 con 0x5559e42ae3f0 2024-03-24T00:16:27.632 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:16:27.628+0000 7fd173fd8740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x5559e420f8d0 con 0x5559e42b3f50 2024-03-24T00:16:27.632 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:16:27.628+0000 7fd173fd8740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x5559e42b6f00 con 0x5559e42aede0 2024-03-24T00:16:27.632 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:16:27.628+0000 7fd173fd8740 10 client.0 _ll_register_callbacks cb 0x5559e420c1c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T00:16:27.632 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:16:27.628+0000 7fd169400640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5559e42aede0 0x5559e42b8b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:16:27.632 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:16:27.628+0000 7fd169e00640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5559e42ae3f0 0x5559e42b8620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:16:27.632 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:16:27.628+0000 7fd169400640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5559e42aede0 0x5559e42b8b60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.118:55470/0 (socket says 172.21.15.118:55470) 2024-03-24T00:16:27.632 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:16:27.628+0000 7fd169400640 1 -- 172.21.15.118:0/1207907762 learned_addr learned my addr 172.21.15.118:0/1207907762 (peer_addr_for_me v2:172.21.15.118:0/0) 2024-03-24T00:16:27.633 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:16:27.628+0000 7fd16a800640 1 --2- 172.21.15.118:0/1207907762 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5559e42b3f50 0x5559e42bfba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:16:27.633 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:16:27.628+0000 7fd169400640 1 -- 172.21.15.118:0/1207907762 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5559e42b3f50 msgr2=0x5559e42bfba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:16:27.633 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:16:27.628+0000 7fd169400640 1 --2- 172.21.15.118:0/1207907762 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5559e42b3f50 0x5559e42bfba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:16:27.633 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:16:27.628+0000 7fd169400640 1 -- 172.21.15.118:0/1207907762 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5559e42ae3f0 msgr2=0x5559e42b8620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:16:27.633 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:16:27.628+0000 7fd169400640 1 --2- 172.21.15.118:0/1207907762 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5559e42ae3f0 0x5559e42b8620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:16:27.633 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:16:27.628+0000 7fd169400640 1 -- 172.21.15.118:0/1207907762 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5559e3f28210 con 0x5559e42aede0 2024-03-24T00:16:27.633 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:16:27.629+0000 7fd169400640 1 --2- 172.21.15.118:0/1207907762 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5559e42aede0 0x5559e42b8b60 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7fd1540980d0 tx=0x7fd15409b150 comp rx=0 tx=0).ready entity=mon.1 client_cookie=92a96aee16b314c7 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:16:27.633 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:16:27.629+0000 7fd14fe00640 10 client.0 ms_handle_connect on v2:172.21.15.193:3300/0 2024-03-24T00:16:27.633 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:16:27.629+0000 7fd14fe00640 1 -- 172.21.15.118:0/1207907762 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd1540a3ea0 con 0x5559e42aede0 2024-03-24T00:16:27.633 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:16:27.629+0000 7fd14fe00640 1 -- 172.21.15.118:0/1207907762 <== mon.1 v2:172.21.15.193:3300/0 2 ==== osd_map(90..90 src has 1..90) v4 ==== 8277+0+0 (secure 0 0 0) 0x7fd1540ccac0 con 0x5559e42aede0 2024-03-24T00:16:27.633 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:16:27.629+0000 7fd173fd8740 1 -- 172.21.15.118:0/1207907762 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fd164004d20 con 0x5559e42aede0 2024-03-24T00:16:27.633 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:16:27.629+0000 7fd14fe00640 1 -- 172.21.15.118:0/1207907762 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) v2 ==== 24+0+0 (secure 0 0 0) 0x7fd154003b30 con 0x5559e42aede0 2024-03-24T00:16:27.633 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:16:27.629+0000 7fd173fd8740 10 client.25609 fetch_fsmap learned FSMap version 28 2024-03-24T00:16:27.634 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:16:27.629+0000 7fd173fd8740 1 -- 172.21.15.118:0/1207907762 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) v3 -- 0x5559e3ff4620 con 0x5559e42aede0 2024-03-24T00:16:27.634 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:16:27.629+0000 7fd14fe00640 1 -- 172.21.15.118:0/1207907762 <== mon.1 v2:172.21.15.193:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fd164004d20 con 0x5559e42aede0 2024-03-24T00:16:27.634 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:16:27.629+0000 7fd14fe00640 1 -- 172.21.15.118:0/1207907762 <== mon.1 v2:172.21.15.193:3300/0 5 ==== fsmap(e 28) v1 ==== 7934+0+0 (secure 0 0 0) 0x7fd154097450 con 0x5559e42aede0 2024-03-24T00:16:27.634 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:16:27.629+0000 7fd173fd8740 10 client.25609 fetch_fsmap finished waiting for FSMap version 28 2024-03-24T00:16:27.634 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:16:27.629+0000 7fd173fd8740 10 client.25609 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-24T00:16:27.634 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:16:27.630+0000 7fd173fd8740 20 client.25609 populate_metadata read hostname 'smithi118' 2024-03-24T00:16:27.634 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:16:27.630+0000 7fd173fd8740 1 --2- 172.21.15.118:0/1207907762 >> [v2:172.21.15.189:6838/4074484168,v1:172.21.15.189:6839/4074484168] conn(0x5559e42c4380 0x5559e42c6770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:16:27.634 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:16:27.630+0000 7fd173fd8740 4 client.25609 mds_command: new command op to 24569 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-24T00:16:27.634 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:16:27.630+0000 7fd173fd8740 1 -- 172.21.15.118:0/1207907762 --> [v2:172.21.15.189:6838/4074484168,v1:172.21.15.189:6839/4074484168] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x5559e4220d70 con 0x5559e42c4380 2024-03-24T00:16:27.634 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:16:27.630+0000 7fd169e00640 1 --2- 172.21.15.118:0/1207907762 >> [v2:172.21.15.189:6838/4074484168,v1:172.21.15.189:6839/4074484168] conn(0x5559e42c4380 0x5559e42c6770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:16:27.634 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:16:27.630+0000 7fd169e00640 1 --2- 172.21.15.118:0/1207907762 >> [v2:172.21.15.189:6838/4074484168,v1:172.21.15.189:6839/4074484168] conn(0x5559e42c4380 0x5559e42c6770 crc :-1 s=READY pgs=49 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=3c2b48ca8ac36b0d server_cookie=fb9c3932ac7175f8 in_seq=0 out_seq=0 2024-03-24T00:16:27.634 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:16:27.630+0000 7fd14fe00640 10 client.25609 ms_handle_connect on v2:172.21.15.189:6838/4074484168 2024-03-24T00:16:27.634 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:16:27.631+0000 7fd14fe00640 1 -- 172.21.15.118:0/1207907762 <== mds.0 v2:172.21.15.189:6838/4074484168 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x5559e4220d70 con 0x5559e42c4380 2024-03-24T00:16:27.635 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:16:27.631+0000 7fd14fe00640 10 client.25609 handle_command_reply: tid=0 2024-03-24T00:16:27.635 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:16:27.632+0000 7fd173fd8740 1 client.25609 shutdown 2024-03-24T00:16:27.635 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:16:27.632+0000 7fd173fd8740 1 -- 172.21.15.118:0/1207907762 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5559e42aede0 msgr2=0x5559e42b8b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:16:27.635 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:16:27.632+0000 7fd173fd8740 1 --2- 172.21.15.118:0/1207907762 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5559e42aede0 0x5559e42b8b60 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7fd1540980d0 tx=0x7fd15409b150 comp rx=0 tx=0).stop 2024-03-24T00:16:27.635 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:16:27.632+0000 7fd173fd8740 1 -- 172.21.15.118:0/1207907762 shutdown_connections 2024-03-24T00:16:27.635 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:16:27.632+0000 7fd173fd8740 1 --2- 172.21.15.118:0/1207907762 >> [v2:172.21.15.189:6838/4074484168,v1:172.21.15.189:6839/4074484168] conn(0x5559e42c4380 0x5559e42c6770 crc :-1 s=READY pgs=49 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:16:27.635 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:16:27.632+0000 7fd173fd8740 1 --2- 172.21.15.118:0/1207907762 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5559e42b3f50 0x5559e42bfba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:16:27.635 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:16:27.632+0000 7fd173fd8740 1 --2- 172.21.15.118:0/1207907762 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5559e42aede0 0x5559e42b8b60 unknown :-1 s=CLOSED pgs=233 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:16:27.635 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:16:27.632+0000 7fd14fe00640 0 client.25609 ms_handle_reset on v2:172.21.15.189:6838/4074484168 2024-03-24T00:16:27.635 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:16:27.632+0000 7fd173fd8740 1 --2- 172.21.15.118:0/1207907762 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5559e42ae3f0 0x5559e42b8620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:16:27.635 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:16:27.632+0000 7fd173fd8740 1 -- 172.21.15.118:0/1207907762 >> 172.21.15.118:0/1207907762 conn(0x5559e42a9960 msgr2=0x5559e42a8b70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:16:27.635 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:16:27.632+0000 7fd173fd8740 1 -- 172.21.15.118:0/1207907762 shutdown_connections 2024-03-24T00:16:27.635 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:16:27.632+0000 7fd173fd8740 1 -- 172.21.15.118:0/1207907762 wait complete. 2024-03-24T00:16:27.635 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:16:27.632+0000 7fd173fd8740 20 client.25609 trim_cache size 0 max 16384 2024-03-24T00:16:27.647 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:16:27.647+0000 7fd173fd8740 1 Processor -- start 2024-03-24T00:16:27.647 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:16:27.647+0000 7fd173fd8740 1 -- start start 2024-03-24T00:16:27.647 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:16:27.647+0000 7fd173fd8740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5559e42ae3f0 0x5559e42b3810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:16:27.647 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:16:27.647+0000 7fd173fd8740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5559e42aede0 0x5559e42bfb60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:16:27.647 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:16:27.647+0000 7fd173fd8740 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5559e42b3f50 0x5559e42a82e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:16:27.647 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:16:27.647+0000 7fd173fd8740 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd16c1aaaf0 con 0x5559e42b3f50 2024-03-24T00:16:27.647 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:16:27.647+0000 7fd173fd8740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd16c1ac470 con 0x5559e42aede0 2024-03-24T00:16:27.647 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:16:27.647+0000 7fd173fd8740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd16c1ab150 con 0x5559e42ae3f0 2024-03-24T00:16:27.647 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:16:27.647+0000 7fd169400640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5559e42aede0 0x5559e42bfb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:16:27.647 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:16:27.647+0000 7fd169e00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5559e42ae3f0 0x5559e42b3810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:16:27.648 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:16:27.647+0000 7fd169400640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5559e42aede0 0x5559e42bfb60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.118:35124/0 (socket says 172.21.15.118:35124) 2024-03-24T00:16:27.648 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:16:27.647+0000 7fd169400640 1 -- 172.21.15.118:0/759804806 learned_addr learned my addr 172.21.15.118:0/759804806 (peer_addr_for_me v2:172.21.15.118:0/0) 2024-03-24T00:16:27.648 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:16:27.647+0000 7fd16a800640 1 --2- 172.21.15.118:0/759804806 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5559e42b3f50 0x5559e42a82e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:16:27.649 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:16:27.648+0000 7fd169400640 1 -- 172.21.15.118:0/759804806 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5559e42ae3f0 msgr2=0x5559e42b3810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:16:27.649 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:16:27.648+0000 7fd169400640 1 --2- 172.21.15.118:0/759804806 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5559e42ae3f0 0x5559e42b3810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:16:27.649 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:16:27.648+0000 7fd169400640 1 -- 172.21.15.118:0/759804806 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5559e42b3f50 msgr2=0x5559e42a82e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:16:27.649 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:16:27.648+0000 7fd169400640 1 --2- 172.21.15.118:0/759804806 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5559e42b3f50 0x5559e42a82e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:16:27.650 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:16:27.648+0000 7fd169400640 1 -- 172.21.15.118:0/759804806 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5559e42c6ae0 con 0x5559e42aede0 2024-03-24T00:16:27.650 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:16:27.648+0000 7fd169400640 1 --2- 172.21.15.118:0/759804806 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5559e42aede0 0x5559e42bfb60 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7fd1540999b0 tx=0x7fd15409b5a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=25331daee8f78965 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:16:27.650 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:16:27.648+0000 7fd15aa00640 1 -- 172.21.15.118:0/759804806 <== mon.2 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd1540b8020 con 0x5559e42aede0 2024-03-24T00:16:27.650 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:16:27.648+0000 7fd15aa00640 1 -- 172.21.15.118:0/759804806 <== mon.2 v2:172.21.15.189:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fd15409b700 con 0x5559e42aede0 2024-03-24T00:16:27.650 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:16:27.648+0000 7fd15aa00640 1 -- 172.21.15.118:0/759804806 <== mon.2 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd154097d70 con 0x5559e42aede0 2024-03-24T00:16:27.650 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:16:27.649+0000 7fd173fd8740 1 -- 172.21.15.118:0/759804806 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5559e42aede0 msgr2=0x5559e42bfb60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:16:27.650 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:16:27.649+0000 7fd173fd8740 1 --2- 172.21.15.118:0/759804806 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5559e42aede0 0x5559e42bfb60 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7fd1540999b0 tx=0x7fd15409b5a0 comp rx=0 tx=0).stop 2024-03-24T00:16:27.650 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:16:27.649+0000 7fd173fd8740 1 -- 172.21.15.118:0/759804806 shutdown_connections 2024-03-24T00:16:27.650 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:16:27.649+0000 7fd173fd8740 1 --2- 172.21.15.118:0/759804806 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5559e42b3f50 0x5559e42a82e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:16:27.650 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:16:27.649+0000 7fd173fd8740 1 --2- 172.21.15.118:0/759804806 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5559e42aede0 0x5559e42bfb60 unknown :-1 s=CLOSED pgs=245 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:16:27.650 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:16:27.649+0000 7fd173fd8740 1 --2- 172.21.15.118:0/759804806 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5559e42ae3f0 0x5559e42b3810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:16:27.650 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:16:27.649+0000 7fd173fd8740 1 -- 172.21.15.118:0/759804806 >> 172.21.15.118:0/759804806 conn(0x5559e42a9960 msgr2=0x5559e42a9d40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:16:27.650 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:16:27.649+0000 7fd173fd8740 1 -- 172.21.15.118:0/759804806 shutdown_connections 2024-03-24T00:16:27.651 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:16:27.649+0000 7fd173fd8740 1 -- 172.21.15.118:0/759804806 wait complete. 2024-03-24T00:16:27.651 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:16:27.650+0000 7fd173fd8740 1 -- start start 2024-03-24T00:16:27.651 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:16:27.650+0000 7fd173fd8740 1 Processor -- start 2024-03-24T00:16:27.653 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:16:27.650+0000 7fd173fd8740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5559e42ae3f0 0x5559e42bd710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:16:27.653 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:16:27.650+0000 7fd173fd8740 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5559e42aede0 0x5559e42bdc50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:16:27.653 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:16:27.650+0000 7fd173fd8740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5559e42b3f50 0x5559e42bec60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:16:27.653 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:16:27.650+0000 7fd173fd8740 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd16c1abe10 con 0x5559e42aede0 2024-03-24T00:16:27.653 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:16:27.650+0000 7fd173fd8740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd16c1ab7b0 con 0x5559e42ae3f0 2024-03-24T00:16:27.653 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:16:27.650+0000 7fd173fd8740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd16c1a8cb0 con 0x5559e42b3f50 2024-03-24T00:16:27.653 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:16:27.651+0000 7fd173fd8740 10 client.0 _ll_register_callbacks cb 0x7fd154003e30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T00:16:27.653 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:16:27.651+0000 7fd169400640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5559e42aede0 0x5559e42bdc50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:16:27.654 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:16:27.651+0000 7fd169e00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5559e42ae3f0 0x5559e42bd710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:16:27.654 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:16:27.651+0000 7fd16a800640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5559e42b3f50 0x5559e42bec60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:16:27.654 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:16:27.651+0000 7fd169400640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5559e42aede0 0x5559e42bdc50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.118:3300/0 says I am v2:172.21.15.118:45820/0 (socket says 172.21.15.118:45820) 2024-03-24T00:16:27.654 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:16:27.651+0000 7fd169400640 1 -- 172.21.15.118:0/950297628 learned_addr learned my addr 172.21.15.118:0/950297628 (peer_addr_for_me v2:172.21.15.118:0/0) 2024-03-24T00:16:27.654 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:16:27.651+0000 7fd169e00640 1 --2- 172.21.15.118:0/950297628 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5559e42ae3f0 0x5559e42bd710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.118:35134/0 (socket says 172.21.15.118:35134) 2024-03-24T00:16:27.654 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:16:27.651+0000 7fd169400640 1 -- 172.21.15.118:0/950297628 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5559e42b3f50 msgr2=0x5559e42bec60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:16:27.654 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:16:27.651+0000 7fd169400640 1 --2- 172.21.15.118:0/950297628 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5559e42b3f50 0x5559e42bec60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:16:27.654 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:16:27.651+0000 7fd169400640 1 -- 172.21.15.118:0/950297628 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5559e42ae3f0 msgr2=0x5559e42bd710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:16:27.654 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:16:27.651+0000 7fd169400640 1 --2- 172.21.15.118:0/950297628 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5559e42ae3f0 0x5559e42bd710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:16:27.654 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:16:27.651+0000 7fd169400640 1 -- 172.21.15.118:0/950297628 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5559e42bf250 con 0x5559e42aede0 2024-03-24T00:16:27.654 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:16:27.651+0000 7fd169e00640 1 --2- 172.21.15.118:0/950297628 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5559e42ae3f0 0x5559e42bd710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T00:16:27.654 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:16:27.651+0000 7fd169400640 1 --2- 172.21.15.118:0/950297628 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5559e42aede0 0x5559e42bdc50 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7fd15400c6a0 tx=0x7fd15409b2b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4378d092d3a88da7 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:16:27.654 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:16:27.651+0000 7fd14fe00640 10 client.0 ms_handle_connect on v2:172.21.15.118:3300/0 2024-03-24T00:16:27.654 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:16:27.652+0000 7fd14fe00640 1 -- 172.21.15.118:0/950297628 <== mon.0 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd15407f060 con 0x5559e42aede0 2024-03-24T00:16:27.655 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:16:27.652+0000 7fd14fe00640 1 -- 172.21.15.118:0/950297628 <== mon.0 v2:172.21.15.118:3300/0 2 ==== osd_map(90..90 src has 1..90) v4 ==== 8277+0+0 (secure 0 0 0) 0x7fd15409ab50 con 0x5559e42aede0 2024-03-24T00:16:27.655 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:16:27.652+0000 7fd173fd8740 1 -- 172.21.15.118:0/950297628 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fd15409b700 con 0x5559e42aede0 2024-03-24T00:16:27.655 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:16:27.652+0000 7fd14fe00640 1 -- 172.21.15.118:0/950297628 <== mon.0 v2:172.21.15.118:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) v2 ==== 24+0+0 (secure 0 0 0) 0x7fd1540b3070 con 0x5559e42aede0 2024-03-24T00:16:27.655 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:16:27.652+0000 7fd173fd8740 10 client.15507 fetch_fsmap learned FSMap version 28 2024-03-24T00:16:27.655 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:16:27.652+0000 7fd173fd8740 1 -- 172.21.15.118:0/950297628 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) v3 -- 0x5559e4320040 con 0x5559e42aede0 2024-03-24T00:16:27.655 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:16:27.652+0000 7fd14fe00640 1 -- 172.21.15.118:0/950297628 <== mon.0 v2:172.21.15.118:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fd15409b700 con 0x5559e42aede0 2024-03-24T00:16:27.655 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:16:27.652+0000 7fd14fe00640 1 -- 172.21.15.118:0/950297628 <== mon.0 v2:172.21.15.118:3300/0 5 ==== fsmap(e 28) v1 ==== 7934+0+0 (secure 0 0 0) 0x7fd1540a0770 con 0x5559e42aede0 2024-03-24T00:16:27.655 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:16:27.652+0000 7fd173fd8740 10 client.15507 fetch_fsmap finished waiting for FSMap version 28 2024-03-24T00:16:27.655 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:16:27.652+0000 7fd173fd8740 10 client.15507 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-24T00:16:27.655 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:16:27.652+0000 7fd173fd8740 20 client.15507 populate_metadata read hostname 'smithi118' 2024-03-24T00:16:27.655 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:16:27.652+0000 7fd173fd8740 1 --2- 172.21.15.118:0/950297628 >> [v2:172.21.15.189:6838/4074484168,v1:172.21.15.189:6839/4074484168] conn(0x5559e4320750 0x5559e4322b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:16:27.655 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:16:27.653+0000 7fd173fd8740 4 client.15507 mds_command: new command op to 24569 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-03-24T00:16:27.655 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:16:27.653+0000 7fd173fd8740 1 -- 172.21.15.118:0/950297628 --> [v2:172.21.15.189:6838/4074484168,v1:172.21.15.189:6839/4074484168] -- command(tid 0: {"prefix": "flush journal"}) v1 -- 0x5559e421b0c0 con 0x5559e4320750 2024-03-24T00:16:27.656 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:16:27.653+0000 7fd169e00640 1 --2- 172.21.15.118:0/950297628 >> [v2:172.21.15.189:6838/4074484168,v1:172.21.15.189:6839/4074484168] conn(0x5559e4320750 0x5559e4322b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:16:27.656 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:16:27.653+0000 7fd169e00640 1 --2- 172.21.15.118:0/950297628 >> [v2:172.21.15.189:6838/4074484168,v1:172.21.15.189:6839/4074484168] conn(0x5559e4320750 0x5559e4322b70 crc :-1 s=READY pgs=50 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=90bb4f0aaf999889 server_cookie=13602eedd1d1b053 in_seq=0 out_seq=0 2024-03-24T00:16:27.656 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:16:27.653+0000 7fd14fe00640 10 client.15507 ms_handle_connect on v2:172.21.15.189:6838/4074484168 2024-03-24T00:16:27.660 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:16:27.660+0000 7fd14fe00640 1 -- 172.21.15.118:0/950297628 <== mds.0 v2:172.21.15.189:6838/4074484168 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+44 (crc 0 0 0) 0x5559e421b0c0 con 0x5559e4320750 2024-03-24T00:16:27.660 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:16:27.660+0000 7fd14fe00640 10 client.15507 handle_command_reply: tid=0 2024-03-24T00:16:27.660 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:16:27.660+0000 7fd173fd8740 1 client.15507 shutdown 2024-03-24T00:16:27.660 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:16:27.660+0000 7fd173fd8740 1 -- 172.21.15.118:0/950297628 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5559e42aede0 msgr2=0x5559e42bdc50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:16:27.661 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:16:27.660+0000 7fd173fd8740 1 --2- 172.21.15.118:0/950297628 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5559e42aede0 0x5559e42bdc50 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7fd15400c6a0 tx=0x7fd15409b2b0 comp rx=0 tx=0).stop 2024-03-24T00:16:27.661 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:16:27.661+0000 7fd173fd8740 1 -- 172.21.15.118:0/950297628 shutdown_connections 2024-03-24T00:16:27.661 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:16:27.661+0000 7fd173fd8740 1 --2- 172.21.15.118:0/950297628 >> [v2:172.21.15.189:6838/4074484168,v1:172.21.15.189:6839/4074484168] conn(0x5559e4320750 0x5559e4322b70 crc :-1 s=READY pgs=50 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:16:27.661 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:16:27.661+0000 7fd173fd8740 1 --2- 172.21.15.118:0/950297628 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5559e42b3f50 0x5559e42bec60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:16:27.661 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:16:27.661+0000 7fd173fd8740 1 --2- 172.21.15.118:0/950297628 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5559e42aede0 0x5559e42bdc50 unknown :-1 s=CLOSED pgs=181 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:16:27.661 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:16:27.661+0000 7fd173fd8740 1 --2- 172.21.15.118:0/950297628 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5559e42ae3f0 0x5559e42bd710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:16:27.661 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:16:27.661+0000 7fd14fe00640 0 client.15507 ms_handle_reset on v2:172.21.15.189:6838/4074484168 2024-03-24T00:16:27.661 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:16:27.661+0000 7fd173fd8740 1 -- 172.21.15.118:0/950297628 >> 172.21.15.118:0/950297628 conn(0x5559e42a9960 msgr2=0x5559e42b3680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:16:27.661 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:16:27.661+0000 7fd173fd8740 1 -- 172.21.15.118:0/950297628 shutdown_connections 2024-03-24T00:16:27.661 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:16:27.661+0000 7fd173fd8740 1 -- 172.21.15.118:0/950297628 wait complete. 2024-03-24T00:16:27.661 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:16:27.661+0000 7fd173fd8740 20 client.15507 trim_cache size 0 max 16384 2024-03-24T00:16:27.662 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stdout:{ 2024-03-24T00:16:27.662 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stdout: "message": "", 2024-03-24T00:16:27.662 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stdout: "return_code": 0 2024-03-24T00:16:27.662 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stdout:} 2024-03-24T00:16:27.662 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:16:27.662+0000 7fd16b200640 1 -- 172.21.15.118:0/531536702 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7fd1500a2830 msgr2=0x7fd1500a4cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:16:27.663 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:16:27.662+0000 7fd16b200640 1 --2- 172.21.15.118:0/531536702 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7fd1500a2830 0x7fd1500a4cf0 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7fd15c006010 tx=0x7fd15c02e040 comp rx=0 tx=0).stop 2024-03-24T00:16:27.663 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:16:27.662+0000 7fd16b200640 1 -- 172.21.15.118:0/531536702 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fd16c104900 msgr2=0x7fd16c199fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:16:27.663 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:16:27.662+0000 7fd16b200640 1 --2- 172.21.15.118:0/531536702 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fd16c104900 0x7fd16c199fc0 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7fd15400e9c0 tx=0x7fd15400ee80 comp rx=0 tx=0).stop 2024-03-24T00:16:27.663 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:16:27.662+0000 7fd16b200640 1 -- 172.21.15.118:0/531536702 shutdown_connections 2024-03-24T00:16:27.663 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:16:27.662+0000 7fd16b200640 1 --2- 172.21.15.118:0/531536702 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7fd1500a2830 0x7fd1500a4cf0 unknown :-1 s=CLOSED pgs=282 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:16:27.663 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:16:27.662+0000 7fd16b200640 1 --2- 172.21.15.118:0/531536702 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd16c107230 0x7fd16c1a1030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:16:27.663 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:16:27.662+0000 7fd16b200640 1 --2- 172.21.15.118:0/531536702 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fd16c104900 0x7fd16c199fc0 unknown :-1 s=CLOSED pgs=232 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:16:27.663 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:16:27.662+0000 7fd16b200640 1 --2- 172.21.15.118:0/531536702 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fd16c101fd0 0x7fd16c199a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:16:27.663 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:16:27.662+0000 7fd16b200640 1 -- 172.21.15.118:0/531536702 >> 172.21.15.118:0/531536702 conn(0x7fd16c0fb980 msgr2=0x7fd16c1017c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:16:27.663 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:16:27.662+0000 7fd16b200640 1 -- 172.21.15.118:0/531536702 shutdown_connections 2024-03-24T00:16:27.663 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:16:27.662+0000 7fd16b200640 1 -- 172.21.15.118:0/531536702 wait complete. 2024-03-24T00:16:29.195 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:16:28 smithi118 ceph-mon[31535]: pgmap v421: 97 pgs: 97 active+clean; 671 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 6.6 KiB/s rd, 1.2 KiB/s wr, 11 op/s 2024-03-24T00:16:29.203 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:16:28 smithi189 ceph-mon[36187]: pgmap v421: 97 pgs: 97 active+clean; 671 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 6.6 KiB/s rd, 1.2 KiB/s wr, 11 op/s 2024-03-24T00:16:29.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:16:28 smithi193 ceph-mon[36878]: pgmap v421: 97 pgs: 97 active+clean; 671 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 6.6 KiB/s rd, 1.2 KiB/s wr, 11 op/s 2024-03-24T00:16:30.195 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:16:29 smithi118 ceph-mon[31535]: pgmap v422: 97 pgs: 97 active+clean; 672 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 7.9 KiB/s rd, 1.8 KiB/s wr, 14 op/s 2024-03-24T00:16:30.203 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:16:29 smithi189 ceph-mon[36187]: pgmap v422: 97 pgs: 97 active+clean; 672 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 7.9 KiB/s rd, 1.8 KiB/s wr, 14 op/s 2024-03-24T00:16:30.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:16:29 smithi193 ceph-mon[36878]: pgmap v422: 97 pgs: 97 active+clean; 672 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 7.9 KiB/s rd, 1.8 KiB/s wr, 14 op/s 2024-03-24T00:16:33.195 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:16:32 smithi118 ceph-mon[31535]: pgmap v423: 97 pgs: 97 active+clean; 672 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 1.7 KiB/s wr, 11 op/s 2024-03-24T00:16:33.203 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:16:32 smithi189 ceph-mon[36187]: pgmap v423: 97 pgs: 97 active+clean; 672 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 1.7 KiB/s wr, 11 op/s 2024-03-24T00:16:33.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:16:32 smithi193 ceph-mon[36878]: pgmap v423: 97 pgs: 97 active+clean; 672 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 1.7 KiB/s wr, 11 op/s 2024-03-24T00:16:35.195 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:16:34 smithi118 ceph-mon[31535]: pgmap v424: 97 pgs: 97 active+clean; 672 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 7.0 KiB/s rd, 1.7 KiB/s wr, 13 op/s 2024-03-24T00:16:35.203 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:16:34 smithi189 ceph-mon[36187]: pgmap v424: 97 pgs: 97 active+clean; 672 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 7.0 KiB/s rd, 1.7 KiB/s wr, 13 op/s 2024-03-24T00:16:35.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:16:34 smithi193 ceph-mon[36878]: pgmap v424: 97 pgs: 97 active+clean; 672 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 7.0 KiB/s rd, 1.7 KiB/s wr, 13 op/s 2024-03-24T00:16:37.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:16:36 smithi118 ceph-mon[31535]: pgmap v425: 97 pgs: 97 active+clean; 672 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 6.7 KiB/s rd, 1.6 KiB/s wr, 13 op/s 2024-03-24T00:16:37.203 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:16:36 smithi189 ceph-mon[36187]: pgmap v425: 97 pgs: 97 active+clean; 672 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 6.7 KiB/s rd, 1.6 KiB/s wr, 13 op/s 2024-03-24T00:16:37.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:16:36 smithi193 ceph-mon[36878]: pgmap v425: 97 pgs: 97 active+clean; 672 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 6.7 KiB/s rd, 1.6 KiB/s wr, 13 op/s 2024-03-24T00:16:39.195 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:16:38 smithi118 ceph-mon[31535]: pgmap v426: 97 pgs: 97 active+clean; 672 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 767 B/s wr, 10 op/s 2024-03-24T00:16:39.203 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:16:38 smithi189 ceph-mon[36187]: pgmap v426: 97 pgs: 97 active+clean; 672 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 767 B/s wr, 10 op/s 2024-03-24T00:16:39.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:16:38 smithi193 ceph-mon[36878]: pgmap v426: 97 pgs: 97 active+clean; 672 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 4.7 KiB/s rd, 767 B/s wr, 10 op/s 2024-03-24T00:16:40.841 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:16:40.841+0000 7f921b200640 1 -- 172.21.15.118:0/1792898400 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f921c102700 msgr2=0x7f921c102b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:16:40.841 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:16:40.841+0000 7f921b200640 1 --2- 172.21.15.118:0/1792898400 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f921c102700 0x7f921c102b80 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7f9208009900 tx=0x7f920802bc30 comp rx=0 tx=0).stop 2024-03-24T00:16:40.841 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:16:40.841+0000 7f921b200640 1 -- 172.21.15.118:0/1792898400 shutdown_connections 2024-03-24T00:16:40.842 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:16:40.841+0000 7f921b200640 1 --2- 172.21.15.118:0/1792898400 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f921c1030c0 0x7f921c109950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:16:40.842 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:16:40.841+0000 7f921b200640 1 --2- 172.21.15.118:0/1792898400 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f921c102700 0x7f921c102b80 unknown :-1 s=CLOSED pgs=246 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:16:40.842 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:16:40.841+0000 7f921b200640 1 --2- 172.21.15.118:0/1792898400 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f921c101510 0x7f921c101930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:16:40.842 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:16:40.841+0000 7f921b200640 1 -- 172.21.15.118:0/1792898400 >> 172.21.15.118:0/1792898400 conn(0x7f921c0fca80 msgr2=0x7f921c0feec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:16:40.842 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:16:40.841+0000 7f921b200640 1 -- 172.21.15.118:0/1792898400 shutdown_connections 2024-03-24T00:16:40.842 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:16:40.841+0000 7f921b200640 1 -- 172.21.15.118:0/1792898400 wait complete. 2024-03-24T00:16:40.843 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:16:40.843+0000 7f921b200640 1 Processor -- start 2024-03-24T00:16:40.843 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:16:40.843+0000 7f921b200640 1 -- start start 2024-03-24T00:16:40.843 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:16:40.843+0000 7f921b200640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f921c101510 0x7f921c199a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:16:40.843 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:16:40.843+0000 7f921b200640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f921c102700 0x7f921c199f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:16:40.845 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:16:40.843+0000 7f921b200640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f921c1030c0 0x7f921c1a0fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:16:40.845 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:16:40.843+0000 7f921b200640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f921c10d1f0 con 0x7f921c102700 2024-03-24T00:16:40.845 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:16:40.843+0000 7f921b200640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f921c10d070 con 0x7f921c101510 2024-03-24T00:16:40.845 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:16:40.843+0000 7f921b200640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f921c10d370 con 0x7f921c1030c0 2024-03-24T00:16:40.845 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:16:40.844+0000 7f9219e00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f921c101510 0x7f921c199a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:16:40.845 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:16:40.844+0000 7f9219e00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f921c101510 0x7f921c199a10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.118:51406/0 (socket says 172.21.15.118:51406) 2024-03-24T00:16:40.845 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:16:40.844+0000 7f9219e00640 1 -- 172.21.15.118:0/4044358148 learned_addr learned my addr 172.21.15.118:0/4044358148 (peer_addr_for_me v2:172.21.15.118:0/0) 2024-03-24T00:16:40.845 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:16:40.844+0000 7f921a800640 1 --2- 172.21.15.118:0/4044358148 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f921c1030c0 0x7f921c1a0fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:16:40.845 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:16:40.844+0000 7f9219400640 1 --2- 172.21.15.118:0/4044358148 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f921c102700 0x7f921c199f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:16:40.845 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:16:40.844+0000 7f9219e00640 1 -- 172.21.15.118:0/4044358148 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f921c1030c0 msgr2=0x7f921c1a0fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:16:40.845 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:16:40.844+0000 7f9219e00640 1 --2- 172.21.15.118:0/4044358148 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f921c1030c0 0x7f921c1a0fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:16:40.845 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:16:40.844+0000 7f9219e00640 1 -- 172.21.15.118:0/4044358148 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f921c102700 msgr2=0x7f921c199f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:16:40.845 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:16:40.844+0000 7f9219e00640 1 --2- 172.21.15.118:0/4044358148 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f921c102700 0x7f921c199f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:16:40.846 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:16:40.844+0000 7f9219e00640 1 -- 172.21.15.118:0/4044358148 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f921c1a1630 con 0x7f921c101510 2024-03-24T00:16:40.846 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:16:40.844+0000 7f9219400640 1 --2- 172.21.15.118:0/4044358148 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f921c102700 0x7f921c199f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T00:16:40.846 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:16:40.844+0000 7f9219e00640 1 --2- 172.21.15.118:0/4044358148 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f921c101510 0x7f921c199a10 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7f920c00d950 tx=0x7f920c00de10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:16:40.846 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:16:40.845+0000 7f9212a00640 1 -- 172.21.15.118:0/4044358148 <== mon.2 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f920c017020 con 0x7f921c101510 2024-03-24T00:16:40.846 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:16:40.845+0000 7f9212a00640 1 -- 172.21.15.118:0/4044358148 <== mon.2 v2:172.21.15.189:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f920c009c30 con 0x7f921c101510 2024-03-24T00:16:40.846 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:16:40.845+0000 7f9212a00640 1 -- 172.21.15.118:0/4044358148 <== mon.2 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f920c00f720 con 0x7f921c101510 2024-03-24T00:16:40.846 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:16:40.845+0000 7f921b200640 1 -- 172.21.15.118:0/4044358148 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f921c1a1920 con 0x7f921c101510 2024-03-24T00:16:40.846 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:16:40.845+0000 7f921b200640 1 -- 172.21.15.118:0/4044358148 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f921c1a1df0 con 0x7f921c101510 2024-03-24T00:16:40.848 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:16:40.849+0000 7f9212a00640 1 -- 172.21.15.118:0/4044358148 <== mon.2 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 19) v1 ==== 137342+0+0 (secure 0 0 0) 0x7f920c01a020 con 0x7f921c101510 2024-03-24T00:16:40.849 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:16:40.849+0000 7f9223e6e740 1 Processor -- start 2024-03-24T00:16:40.849 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:16:40.850+0000 7f9223e6e740 1 -- start start 2024-03-24T00:16:40.849 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:16:40.850+0000 7f9223e6e740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55889f7d7c00 0x55889f7d8020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:16:40.850 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:16:40.850+0000 7f9223e6e740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55889f7d85f0 0x55889f7dd0f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:16:40.850 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:16:40.850+0000 7f9223e6e740 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55889f7dd760 0x55889f7dfbf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:16:40.850 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:16:40.850+0000 7f9223e6e740 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x55889f7923b0 con 0x55889f7dd760 2024-03-24T00:16:40.850 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:16:40.850+0000 7f9223e6e740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x55889f734830 con 0x55889f7d85f0 2024-03-24T00:16:40.850 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:16:40.850+0000 7f9223e6e740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x55889f6dd5d0 con 0x55889f7d7c00 2024-03-24T00:16:40.851 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:16:40.850+0000 7f9219e00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55889f7d85f0 0x55889f7dd0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:16:40.851 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:16:40.850+0000 7f9219e00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55889f7d85f0 0x55889f7dd0f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.118:51414/0 (socket says 172.21.15.118:51414) 2024-03-24T00:16:40.851 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:16:40.850+0000 7f9219e00640 1 -- 172.21.15.118:0/2648310092 learned_addr learned my addr 172.21.15.118:0/2648310092 (peer_addr_for_me v2:172.21.15.118:0/0) 2024-03-24T00:16:40.851 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:16:40.851+0000 7f921a800640 1 --2- 172.21.15.118:0/2648310092 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55889f7d7c00 0x55889f7d8020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:16:40.851 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:16:40.851+0000 7f9219400640 1 --2- 172.21.15.118:0/2648310092 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55889f7dd760 0x55889f7dfbf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:16:40.852 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:16:40.851+0000 7f9219e00640 1 -- 172.21.15.118:0/2648310092 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55889f7d7c00 msgr2=0x55889f7d8020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:16:40.852 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:16:40.851+0000 7f9219e00640 1 --2- 172.21.15.118:0/2648310092 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55889f7d7c00 0x55889f7d8020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:16:40.852 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:16:40.851+0000 7f9219e00640 1 -- 172.21.15.118:0/2648310092 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55889f7dd760 msgr2=0x55889f7dfbf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:16:40.852 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:16:40.851+0000 7f9219e00640 1 --2- 172.21.15.118:0/2648310092 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55889f7dd760 0x55889f7dfbf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:16:40.852 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:16:40.851+0000 7f9219e00640 1 -- 172.21.15.118:0/2648310092 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55889f6f52c0 con 0x55889f7d85f0 2024-03-24T00:16:40.852 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:16:40.851+0000 7f9219e00640 1 --2- 172.21.15.118:0/2648310092 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55889f7d85f0 0x55889f7dd0f0 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7f920c0cc340 tx=0x7f920c0d2f80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ac5eb8634bbeb39d server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:16:40.852 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:16:40.851+0000 7f9212a00640 1 --2- 172.21.15.118:0/4044358148 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7f92000a2900 0x7f92000a4dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:16:40.852 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:16:40.851+0000 7f9207400640 1 -- 172.21.15.118:0/2648310092 <== mon.2 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f920c097530 con 0x55889f7d85f0 2024-03-24T00:16:40.852 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:16:40.852+0000 7f9207400640 1 -- 172.21.15.118:0/2648310092 <== mon.2 v2:172.21.15.189:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f920c0e0070 con 0x55889f7d85f0 2024-03-24T00:16:40.852 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:16:40.852+0000 7f9212a00640 1 -- 172.21.15.118:0/4044358148 <== mon.2 v2:172.21.15.189:3300/0 5 ==== osd_map(90..90 src has 1..90) v4 ==== 8277+0+0 (secure 0 0 0) 0x7f920c0cea80 con 0x7f921c101510 2024-03-24T00:16:40.852 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:16:40.852+0000 7f921a800640 1 --2- 172.21.15.118:0/4044358148 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7f92000a2900 0x7f92000a4dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:16:40.853 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:16:40.852+0000 7f9207400640 1 -- 172.21.15.118:0/2648310092 <== mon.2 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f920c09a6e0 con 0x55889f7d85f0 2024-03-24T00:16:40.853 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:16:40.852+0000 7f9223e6e740 1 -- 172.21.15.118:0/2648310092 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55889f7d85f0 msgr2=0x55889f7dd0f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:16:40.853 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:16:40.852+0000 7f9223e6e740 1 --2- 172.21.15.118:0/2648310092 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55889f7d85f0 0x55889f7dd0f0 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7f920c0cc340 tx=0x7f920c0d2f80 comp rx=0 tx=0).stop 2024-03-24T00:16:40.853 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:16:40.853+0000 7f9223e6e740 1 -- 172.21.15.118:0/2648310092 shutdown_connections 2024-03-24T00:16:40.853 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:16:40.853+0000 7f9223e6e740 1 --2- 172.21.15.118:0/2648310092 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55889f7dd760 0x55889f7dfbf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:16:40.853 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:16:40.853+0000 7f9223e6e740 1 --2- 172.21.15.118:0/2648310092 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55889f7d85f0 0x55889f7dd0f0 unknown :-1 s=CLOSED pgs=248 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:16:40.854 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:16:40.853+0000 7f9223e6e740 1 --2- 172.21.15.118:0/2648310092 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55889f7d7c00 0x55889f7d8020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:16:40.854 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:16:40.853+0000 7f9223e6e740 1 -- 172.21.15.118:0/2648310092 >> 172.21.15.118:0/2648310092 conn(0x55889f7d3170 msgr2=0x55889f7d3570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:16:40.854 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:16:40.853+0000 7f9223e6e740 1 -- 172.21.15.118:0/2648310092 shutdown_connections 2024-03-24T00:16:40.854 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:16:40.853+0000 7f9223e6e740 1 -- 172.21.15.118:0/2648310092 wait complete. 2024-03-24T00:16:40.854 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:16:40.853+0000 7f921a800640 1 --2- 172.21.15.118:0/4044358148 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7f92000a2900 0x7f92000a4dc0 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x55889f7d8e50 tx=0x7f9214009500 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:16:40.854 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:16:40.853+0000 7f9223e6e740 1 -- start start 2024-03-24T00:16:40.854 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:16:40.854+0000 7f9223e6e740 1 Processor -- start 2024-03-24T00:16:40.856 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:16:40.854+0000 7f9223e6e740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55889f7d7c00 0x55889f7e1d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:16:40.856 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:16:40.854+0000 7f9223e6e740 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55889f7d85f0 0x55889f7e22c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:16:40.856 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:16:40.854+0000 7f9223e6e740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55889f7dd760 0x55889f7e9300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:16:40.856 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:16:40.854+0000 7f9223e6e740 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x55889f6f0a20 con 0x55889f7d85f0 2024-03-24T00:16:40.856 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:16:40.854+0000 7f9223e6e740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x55889f7308d0 con 0x55889f7dd760 2024-03-24T00:16:40.856 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:16:40.854+0000 7f9223e6e740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x55889f7e0750 con 0x55889f7d7c00 2024-03-24T00:16:40.856 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:16:40.854+0000 7f9223e6e740 10 client.0 _ll_register_callbacks cb 0x55889f72d1c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T00:16:40.857 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:16:40.854+0000 7f9219e00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55889f7d7c00 0x55889f7e1d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:16:40.857 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:16:40.854+0000 7f9219e00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55889f7d7c00 0x55889f7e1d80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.118:53926/0 (socket says 172.21.15.118:53926) 2024-03-24T00:16:40.857 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:16:40.854+0000 7f9219e00640 1 -- 172.21.15.118:0/2300389049 learned_addr learned my addr 172.21.15.118:0/2300389049 (peer_addr_for_me v2:172.21.15.118:0/0) 2024-03-24T00:16:40.857 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:16:40.855+0000 7f9219400640 1 --2- 172.21.15.118:0/2300389049 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55889f7d85f0 0x55889f7e22c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:16:40.857 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:16:40.855+0000 7f921a800640 1 --2- 172.21.15.118:0/2300389049 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55889f7dd760 0x55889f7e9300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:16:40.857 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:16:40.855+0000 7f9219e00640 1 -- 172.21.15.118:0/2300389049 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55889f7dd760 msgr2=0x55889f7e9300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:16:40.857 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:16:40.855+0000 7f9219e00640 1 --2- 172.21.15.118:0/2300389049 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55889f7dd760 0x55889f7e9300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:16:40.857 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:16:40.855+0000 7f9219e00640 1 -- 172.21.15.118:0/2300389049 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55889f7d85f0 msgr2=0x55889f7e22c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:16:40.857 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:16:40.855+0000 7f9219e00640 1 --2- 172.21.15.118:0/2300389049 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55889f7d85f0 0x55889f7e22c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:16:40.857 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:16:40.855+0000 7f9219e00640 1 -- 172.21.15.118:0/2300389049 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55889f449210 con 0x55889f7d7c00 2024-03-24T00:16:40.857 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:16:40.855+0000 7f921a800640 1 --2- 172.21.15.118:0/2300389049 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55889f7dd760 0x55889f7e9300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-24T00:16:40.857 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:16:40.855+0000 7f9219400640 1 --2- 172.21.15.118:0/2300389049 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55889f7d85f0 0x55889f7e22c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T00:16:40.857 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:16:40.856+0000 7f9219e00640 1 --2- 172.21.15.118:0/2300389049 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55889f7d7c00 0x55889f7e1d80 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7f920c004770 tx=0x7f920c0cc370 comp rx=0 tx=0).ready entity=mon.1 client_cookie=f2fb9dd4fe863c2 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:16:40.857 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:16:40.856+0000 7f91fbe00640 10 client.0 ms_handle_connect on v2:172.21.15.193:3300/0 2024-03-24T00:16:40.857 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:16:40.856+0000 7f91fbe00640 1 -- 172.21.15.118:0/2300389049 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f920c09aa10 con 0x55889f7d7c00 2024-03-24T00:16:40.858 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:16:40.856+0000 7f91fbe00640 1 -- 172.21.15.118:0/2300389049 <== mon.1 v2:172.21.15.193:3300/0 2 ==== osd_map(90..90 src has 1..90) v4 ==== 8277+0+0 (secure 0 0 0) 0x7f920c0cc650 con 0x55889f7d7c00 2024-03-24T00:16:40.858 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:16:40.856+0000 7f9223e6e740 1 -- 172.21.15.118:0/2300389049 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f920c0e0070 con 0x55889f7d7c00 2024-03-24T00:16:40.858 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:16:40.856+0000 7f91fbe00640 1 -- 172.21.15.118:0/2300389049 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) v2 ==== 24+0+0 (secure 0 0 0) 0x7f920c0d2020 con 0x55889f7d7c00 2024-03-24T00:16:40.858 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:16:40.856+0000 7f9223e6e740 10 client.25627 fetch_fsmap learned FSMap version 28 2024-03-24T00:16:40.858 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:16:40.856+0000 7f9223e6e740 1 -- 172.21.15.118:0/2300389049 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) v3 -- 0x55889f515620 con 0x55889f7d7c00 2024-03-24T00:16:40.858 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:16:40.856+0000 7f91fbe00640 1 -- 172.21.15.118:0/2300389049 <== mon.1 v2:172.21.15.193:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f920c0e0070 con 0x55889f7d7c00 2024-03-24T00:16:40.858 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:16:40.857+0000 7f91fbe00640 1 -- 172.21.15.118:0/2300389049 <== mon.1 v2:172.21.15.193:3300/0 5 ==== fsmap(e 28) v1 ==== 7934+0+0 (secure 0 0 0) 0x7f920c09a6e0 con 0x55889f7d7c00 2024-03-24T00:16:40.858 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:16:40.857+0000 7f9223e6e740 10 client.25627 fetch_fsmap finished waiting for FSMap version 28 2024-03-24T00:16:40.858 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:16:40.857+0000 7f9223e6e740 10 client.25627 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-24T00:16:40.858 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:16:40.857+0000 7f9223e6e740 20 client.25627 populate_metadata read hostname 'smithi118' 2024-03-24T00:16:40.858 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:16:40.857+0000 7f9223e6e740 1 --2- 172.21.15.118:0/2300389049 >> [v2:172.21.15.189:6838/4074484168,v1:172.21.15.189:6839/4074484168] conn(0x55889f7ed7f0 0x55889f7efbe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:16:40.858 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:16:40.857+0000 7f9223e6e740 4 client.25627 mds_command: new command op to 24569 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-24T00:16:40.859 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:16:40.857+0000 7f9223e6e740 1 -- 172.21.15.118:0/2300389049 --> [v2:172.21.15.189:6838/4074484168,v1:172.21.15.189:6839/4074484168] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55889f741d70 con 0x55889f7ed7f0 2024-03-24T00:16:40.860 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:16:40.857+0000 7f9219400640 1 --2- 172.21.15.118:0/2300389049 >> [v2:172.21.15.189:6838/4074484168,v1:172.21.15.189:6839/4074484168] conn(0x55889f7ed7f0 0x55889f7efbe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:16:40.860 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:16:40.858+0000 7f9219400640 1 --2- 172.21.15.118:0/2300389049 >> [v2:172.21.15.189:6838/4074484168,v1:172.21.15.189:6839/4074484168] conn(0x55889f7ed7f0 0x55889f7efbe0 crc :-1 s=READY pgs=51 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=3beb055095a8bd2d server_cookie=d73177040436e061 in_seq=0 out_seq=0 2024-03-24T00:16:40.860 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:16:40.858+0000 7f91fbe00640 10 client.25627 ms_handle_connect on v2:172.21.15.189:6838/4074484168 2024-03-24T00:16:40.860 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:16:40.858+0000 7f91fbe00640 1 -- 172.21.15.118:0/2300389049 <== mds.0 v2:172.21.15.189:6838/4074484168 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x55889f741d70 con 0x55889f7ed7f0 2024-03-24T00:16:40.860 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:16:40.858+0000 7f91fbe00640 10 client.25627 handle_command_reply: tid=0 2024-03-24T00:16:40.860 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:16:40.859+0000 7f9223e6e740 1 client.25627 shutdown 2024-03-24T00:16:40.860 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:16:40.859+0000 7f9223e6e740 1 -- 172.21.15.118:0/2300389049 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55889f7d7c00 msgr2=0x55889f7e1d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:16:40.860 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:16:40.859+0000 7f9223e6e740 1 --2- 172.21.15.118:0/2300389049 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55889f7d7c00 0x55889f7e1d80 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7f920c004770 tx=0x7f920c0cc370 comp rx=0 tx=0).stop 2024-03-24T00:16:40.860 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:16:40.859+0000 7f9223e6e740 1 -- 172.21.15.118:0/2300389049 shutdown_connections 2024-03-24T00:16:40.860 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:16:40.859+0000 7f9223e6e740 1 --2- 172.21.15.118:0/2300389049 >> [v2:172.21.15.189:6838/4074484168,v1:172.21.15.189:6839/4074484168] conn(0x55889f7ed7f0 0x55889f7efbe0 crc :-1 s=READY pgs=51 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:16:40.861 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:16:40.859+0000 7f9223e6e740 1 --2- 172.21.15.118:0/2300389049 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55889f7dd760 0x55889f7e9300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:16:40.861 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:16:40.859+0000 7f9223e6e740 1 --2- 172.21.15.118:0/2300389049 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55889f7d85f0 0x55889f7e22c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:16:40.861 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:16:40.859+0000 7f9223e6e740 1 --2- 172.21.15.118:0/2300389049 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55889f7d7c00 0x55889f7e1d80 unknown :-1 s=CLOSED pgs=234 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:16:40.861 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:16:40.859+0000 7f9223e6e740 1 -- 172.21.15.118:0/2300389049 >> 172.21.15.118:0/2300389049 conn(0x55889f7d3170 msgr2=0x55889f7c9b70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:16:40.861 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:16:40.859+0000 7f91fbe00640 0 client.25627 ms_handle_reset on v2:172.21.15.189:6838/4074484168 2024-03-24T00:16:40.861 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:16:40.859+0000 7f9223e6e740 1 -- 172.21.15.118:0/2300389049 shutdown_connections 2024-03-24T00:16:40.861 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:16:40.859+0000 7f9223e6e740 1 -- 172.21.15.118:0/2300389049 wait complete. 2024-03-24T00:16:40.861 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:16:40.859+0000 7f9223e6e740 20 client.25627 trim_cache size 0 max 16384 2024-03-24T00:16:40.874 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:16:40.874+0000 7f9223e6e740 1 Processor -- start 2024-03-24T00:16:40.874 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:16:40.874+0000 7f9223e6e740 1 -- start start 2024-03-24T00:16:40.874 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:16:40.874+0000 7f9223e6e740 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55889f7d7c00 0x55889f7e1990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:16:40.874 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:16:40.874+0000 7f9223e6e740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55889f7d85f0 0x55889f7e92c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:16:40.874 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:16:40.874+0000 7f9223e6e740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55889f7dd760 0x55889f7e1ed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:16:40.874 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:16:40.874+0000 7f9223e6e740 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f91fc00e220 con 0x55889f7d7c00 2024-03-24T00:16:40.874 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:16:40.874+0000 7f9223e6e740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f91fc00fba0 con 0x55889f7d85f0 2024-03-24T00:16:40.874 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:16:40.874+0000 7f9223e6e740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f91fc00e880 con 0x55889f7dd760 2024-03-24T00:16:40.875 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:16:40.875+0000 7f9219e00640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55889f7d7c00 0x55889f7e1990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:16:40.875 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:16:40.875+0000 7f921a800640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55889f7dd760 0x55889f7e1ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:16:40.875 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:16:40.875+0000 7f9219e00640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55889f7d7c00 0x55889f7e1990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.118:3300/0 says I am v2:172.21.15.118:49542/0 (socket says 172.21.15.118:49542) 2024-03-24T00:16:40.875 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:16:40.875+0000 7f921a800640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55889f7dd760 0x55889f7e1ed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.118:53932/0 (socket says 172.21.15.118:53932) 2024-03-24T00:16:40.875 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:16:40.875+0000 7f9219e00640 1 -- 172.21.15.118:0/2645512023 learned_addr learned my addr 172.21.15.118:0/2645512023 (peer_addr_for_me v2:172.21.15.118:0/0) 2024-03-24T00:16:40.876 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:16:40.875+0000 7f9219400640 1 --2- 172.21.15.118:0/2645512023 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55889f7d85f0 0x55889f7e92c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:16:40.876 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:16:40.875+0000 7f9219400640 1 -- 172.21.15.118:0/2645512023 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55889f7dd760 msgr2=0x55889f7e1ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:16:40.876 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:16:40.875+0000 7f9219400640 1 --2- 172.21.15.118:0/2645512023 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55889f7dd760 0x55889f7e1ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:16:40.876 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:16:40.875+0000 7f9219400640 1 -- 172.21.15.118:0/2645512023 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55889f7d7c00 msgr2=0x55889f7e1990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:16:40.876 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:16:40.875+0000 7f9219400640 1 --2- 172.21.15.118:0/2645512023 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55889f7d7c00 0x55889f7e1990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:16:40.876 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:16:40.875+0000 7f9219400640 1 -- 172.21.15.118:0/2645512023 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55889f7eff50 con 0x55889f7d85f0 2024-03-24T00:16:40.876 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:16:40.875+0000 7f9219e00640 1 --2- 172.21.15.118:0/2645512023 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55889f7d7c00 0x55889f7e1990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T00:16:40.876 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:16:40.875+0000 7f9219400640 1 --2- 172.21.15.118:0/2645512023 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55889f7d85f0 0x55889f7e92c0 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7f9208036040 tx=0x7f920802bc30 comp rx=0 tx=0).ready entity=mon.2 client_cookie=972e9aa38c9cc461 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:16:40.876 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:16:40.876+0000 7f9206a00640 1 -- 172.21.15.118:0/2645512023 <== mon.2 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f920804a070 con 0x55889f7d85f0 2024-03-24T00:16:40.876 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:16:40.876+0000 7f9206a00640 1 -- 172.21.15.118:0/2645512023 <== mon.2 v2:172.21.15.189:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f9208046250 con 0x55889f7d85f0 2024-03-24T00:16:40.876 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:16:40.876+0000 7f9206a00640 1 -- 172.21.15.118:0/2645512023 <== mon.2 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f92080046c0 con 0x55889f7d85f0 2024-03-24T00:16:40.877 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:16:40.877+0000 7f9223e6e740 1 -- 172.21.15.118:0/2645512023 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55889f7d85f0 msgr2=0x55889f7e92c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:16:40.877 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:16:40.877+0000 7f9223e6e740 1 --2- 172.21.15.118:0/2645512023 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55889f7d85f0 0x55889f7e92c0 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7f9208036040 tx=0x7f920802bc30 comp rx=0 tx=0).stop 2024-03-24T00:16:40.877 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:16:40.877+0000 7f9223e6e740 1 -- 172.21.15.118:0/2645512023 shutdown_connections 2024-03-24T00:16:40.877 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:16:40.877+0000 7f9223e6e740 1 --2- 172.21.15.118:0/2645512023 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55889f7dd760 0x55889f7e1ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:16:40.877 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:16:40.877+0000 7f9223e6e740 1 --2- 172.21.15.118:0/2645512023 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55889f7d85f0 0x55889f7e92c0 unknown :-1 s=CLOSED pgs=249 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:16:40.877 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:16:40.877+0000 7f9223e6e740 1 --2- 172.21.15.118:0/2645512023 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55889f7d7c00 0x55889f7e1990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:16:40.877 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:16:40.877+0000 7f9223e6e740 1 -- 172.21.15.118:0/2645512023 >> 172.21.15.118:0/2645512023 conn(0x55889f7d3170 msgr2=0x55889f7dce10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:16:40.878 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:16:40.877+0000 7f9223e6e740 1 -- 172.21.15.118:0/2645512023 shutdown_connections 2024-03-24T00:16:40.878 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:16:40.877+0000 7f9223e6e740 1 -- 172.21.15.118:0/2645512023 wait complete. 2024-03-24T00:16:40.878 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:16:40.877+0000 7f9223e6e740 1 -- start start 2024-03-24T00:16:40.878 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:16:40.878+0000 7f9223e6e740 1 Processor -- start 2024-03-24T00:16:40.879 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:16:40.878+0000 7f9223e6e740 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55889f7d7c00 0x55889f7e6a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:16:40.879 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:16:40.878+0000 7f9223e6e740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55889f7d85f0 0x55889f7e6f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:16:40.879 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:16:40.878+0000 7f9223e6e740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55889f7dd760 0x55889f7e7ee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:16:40.879 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:16:40.878+0000 7f9223e6e740 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f91fc00f540 con 0x55889f7d7c00 2024-03-24T00:16:40.879 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:16:40.878+0000 7f9223e6e740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f91fc00eee0 con 0x55889f7d85f0 2024-03-24T00:16:40.879 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:16:40.878+0000 7f9223e6e740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f91fc00c3e0 con 0x55889f7dd760 2024-03-24T00:16:40.880 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:16:40.878+0000 7f9223e6e740 10 client.0 _ll_register_callbacks cb 0x7f920c0d2970 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T00:16:40.880 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:16:40.879+0000 7f9219400640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55889f7d85f0 0x55889f7e6f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:16:40.880 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:16:40.879+0000 7f9219400640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55889f7d85f0 0x55889f7e6f70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.118:51434/0 (socket says 172.21.15.118:51434) 2024-03-24T00:16:40.880 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:16:40.879+0000 7f9219400640 1 -- 172.21.15.118:0/4291828770 learned_addr learned my addr 172.21.15.118:0/4291828770 (peer_addr_for_me v2:172.21.15.118:0/0) 2024-03-24T00:16:40.880 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:16:40.879+0000 7f921a800640 1 --2- 172.21.15.118:0/4291828770 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55889f7dd760 0x55889f7e7ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:16:40.880 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:16:40.879+0000 7f9219e00640 1 --2- 172.21.15.118:0/4291828770 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55889f7d7c00 0x55889f7e6a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:16:40.880 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:16:40.879+0000 7f9219400640 1 -- 172.21.15.118:0/4291828770 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55889f7dd760 msgr2=0x55889f7e7ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:16:40.880 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:16:40.879+0000 7f9219400640 1 --2- 172.21.15.118:0/4291828770 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55889f7dd760 0x55889f7e7ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:16:40.880 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:16:40.879+0000 7f9219400640 1 -- 172.21.15.118:0/4291828770 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55889f7d7c00 msgr2=0x55889f7e6a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:16:40.880 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:16:40.879+0000 7f9219400640 1 --2- 172.21.15.118:0/4291828770 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55889f7d7c00 0x55889f7e6a30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:16:40.880 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:16:40.879+0000 7f9219400640 1 -- 172.21.15.118:0/4291828770 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55889f7e84d0 con 0x55889f7d85f0 2024-03-24T00:16:40.880 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:16:40.879+0000 7f9219400640 1 --2- 172.21.15.118:0/4291828770 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55889f7d85f0 0x55889f7e6f70 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7f9208009850 tx=0x7f920802c660 comp rx=0 tx=0).ready entity=mon.2 client_cookie=911f52defa7871dd server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:16:40.880 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:16:40.879+0000 7f91fbe00640 10 client.0 ms_handle_connect on v2:172.21.15.189:3300/0 2024-03-24T00:16:40.880 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:16:40.880+0000 7f91fbe00640 1 -- 172.21.15.118:0/4291828770 <== mon.2 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9208052070 con 0x55889f7d85f0 2024-03-24T00:16:40.880 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:16:40.880+0000 7f91fbe00640 1 -- 172.21.15.118:0/4291828770 <== mon.2 v2:172.21.15.189:3300/0 2 ==== osd_map(90..90 src has 1..90) v4 ==== 8277+0+0 (secure 0 0 0) 0x7f9208004230 con 0x55889f7d85f0 2024-03-24T00:16:40.880 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:16:40.880+0000 7f9223e6e740 1 -- 172.21.15.118:0/4291828770 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f9208046250 con 0x55889f7d85f0 2024-03-24T00:16:40.882 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:16:40.880+0000 7f91fbe00640 1 -- 172.21.15.118:0/4291828770 <== mon.2 v2:172.21.15.189:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) v2 ==== 24+0+0 (secure 0 0 0) 0x7f920805c050 con 0x55889f7d85f0 2024-03-24T00:16:40.882 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:16:40.880+0000 7f9223e6e740 10 client.25628 fetch_fsmap learned FSMap version 28 2024-03-24T00:16:40.882 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:16:40.880+0000 7f9223e6e740 1 -- 172.21.15.118:0/4291828770 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) v3 -- 0x55889f848f80 con 0x55889f7d85f0 2024-03-24T00:16:40.882 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:16:40.881+0000 7f91fbe00640 1 -- 172.21.15.118:0/4291828770 <== mon.2 v2:172.21.15.189:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f9208046250 con 0x55889f7d85f0 2024-03-24T00:16:40.882 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:16:40.881+0000 7f91fbe00640 1 -- 172.21.15.118:0/4291828770 <== mon.2 v2:172.21.15.189:3300/0 5 ==== fsmap(e 28) v1 ==== 7934+0+0 (secure 0 0 0) 0x7f9208034510 con 0x55889f7d85f0 2024-03-24T00:16:40.882 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:16:40.881+0000 7f9223e6e740 10 client.25628 fetch_fsmap finished waiting for FSMap version 28 2024-03-24T00:16:40.882 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:16:40.881+0000 7f9223e6e740 10 client.25628 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-24T00:16:40.882 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:16:40.881+0000 7f9223e6e740 20 client.25628 populate_metadata read hostname 'smithi118' 2024-03-24T00:16:40.882 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:16:40.881+0000 7f9223e6e740 1 --2- 172.21.15.118:0/4291828770 >> [v2:172.21.15.189:6838/4074484168,v1:172.21.15.189:6839/4074484168] conn(0x55889f8498f0 0x55889f84bd10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:16:40.882 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:16:40.881+0000 7f9223e6e740 4 client.25628 mds_command: new command op to 24569 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-03-24T00:16:40.882 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:16:40.881+0000 7f9223e6e740 1 -- 172.21.15.118:0/4291828770 --> [v2:172.21.15.189:6838/4074484168,v1:172.21.15.189:6839/4074484168] -- command(tid 0: {"prefix": "flush journal"}) v1 -- 0x55889f73c0c0 con 0x55889f8498f0 2024-03-24T00:16:40.882 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:16:40.881+0000 7f9219e00640 1 --2- 172.21.15.118:0/4291828770 >> [v2:172.21.15.189:6838/4074484168,v1:172.21.15.189:6839/4074484168] conn(0x55889f8498f0 0x55889f84bd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:16:40.882 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:16:40.882+0000 7f9219e00640 1 --2- 172.21.15.118:0/4291828770 >> [v2:172.21.15.189:6838/4074484168,v1:172.21.15.189:6839/4074484168] conn(0x55889f8498f0 0x55889f84bd10 crc :-1 s=READY pgs=52 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=e81782431e77b092 server_cookie=7c85e1af8131d334 in_seq=0 out_seq=0 2024-03-24T00:16:40.882 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:16:40.882+0000 7f91fbe00640 10 client.25628 ms_handle_connect on v2:172.21.15.189:6838/4074484168 2024-03-24T00:16:40.884 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:16:40.885+0000 7f91fbe00640 1 -- 172.21.15.118:0/4291828770 <== mds.0 v2:172.21.15.189:6838/4074484168 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+44 (crc 0 0 0) 0x55889f73c0c0 con 0x55889f8498f0 2024-03-24T00:16:40.885 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:16:40.885+0000 7f91fbe00640 10 client.25628 handle_command_reply: tid=0 2024-03-24T00:16:40.885 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:16:40.886+0000 7f9223e6e740 1 client.25628 shutdown 2024-03-24T00:16:40.885 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:16:40.886+0000 7f9223e6e740 1 -- 172.21.15.118:0/4291828770 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55889f7d85f0 msgr2=0x55889f7e6f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:16:40.886 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:16:40.886+0000 7f9223e6e740 1 --2- 172.21.15.118:0/4291828770 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55889f7d85f0 0x55889f7e6f70 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7f9208009850 tx=0x7f920802c660 comp rx=0 tx=0).stop 2024-03-24T00:16:40.886 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:16:40.886+0000 7f9223e6e740 1 -- 172.21.15.118:0/4291828770 shutdown_connections 2024-03-24T00:16:40.886 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:16:40.886+0000 7f9223e6e740 1 --2- 172.21.15.118:0/4291828770 >> [v2:172.21.15.189:6838/4074484168,v1:172.21.15.189:6839/4074484168] conn(0x55889f8498f0 0x55889f84bd10 crc :-1 s=READY pgs=52 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:16:40.886 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:16:40.886+0000 7f9223e6e740 1 --2- 172.21.15.118:0/4291828770 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55889f7dd760 0x55889f7e7ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:16:40.886 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:16:40.886+0000 7f9223e6e740 1 --2- 172.21.15.118:0/4291828770 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55889f7d85f0 0x55889f7e6f70 unknown :-1 s=CLOSED pgs=250 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:16:40.886 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:16:40.886+0000 7f9223e6e740 1 --2- 172.21.15.118:0/4291828770 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55889f7d7c00 0x55889f7e6a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:16:40.886 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:16:40.886+0000 7f9223e6e740 1 -- 172.21.15.118:0/4291828770 >> 172.21.15.118:0/4291828770 conn(0x55889f7d3170 msgr2=0x55889f7e1990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:16:40.886 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:16:40.886+0000 7f91fbe00640 0 client.25628 ms_handle_reset on v2:172.21.15.189:6838/4074484168 2024-03-24T00:16:40.887 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:16:40.887+0000 7f9223e6e740 1 -- 172.21.15.118:0/4291828770 shutdown_connections 2024-03-24T00:16:40.887 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:16:40.887+0000 7f9223e6e740 1 -- 172.21.15.118:0/4291828770 wait complete. 2024-03-24T00:16:40.887 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:16:40.887+0000 7f9223e6e740 20 client.25628 trim_cache size 0 max 16384 2024-03-24T00:16:40.887 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stdout:{ 2024-03-24T00:16:40.887 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stdout: "message": "", 2024-03-24T00:16:40.887 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stdout: "return_code": 0 2024-03-24T00:16:40.887 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stdout:} 2024-03-24T00:16:40.888 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:16:40.888+0000 7f921b200640 1 -- 172.21.15.118:0/4044358148 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7f92000a2900 msgr2=0x7f92000a4dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:16:40.888 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:16:40.888+0000 7f921b200640 1 --2- 172.21.15.118:0/4044358148 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7f92000a2900 0x7f92000a4dc0 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x55889f7d8e50 tx=0x7f9214009500 comp rx=0 tx=0).stop 2024-03-24T00:16:40.888 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:16:40.888+0000 7f921b200640 1 -- 172.21.15.118:0/4044358148 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f921c101510 msgr2=0x7f921c199a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:16:40.888 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:16:40.888+0000 7f921b200640 1 --2- 172.21.15.118:0/4044358148 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f921c101510 0x7f921c199a10 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7f920c00d950 tx=0x7f920c00de10 comp rx=0 tx=0).stop 2024-03-24T00:16:40.888 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:16:40.888+0000 7f921b200640 1 -- 172.21.15.118:0/4044358148 shutdown_connections 2024-03-24T00:16:40.888 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:16:40.888+0000 7f921b200640 1 --2- 172.21.15.118:0/4044358148 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7f92000a2900 0x7f92000a4dc0 unknown :-1 s=CLOSED pgs=283 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:16:40.888 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:16:40.888+0000 7f921b200640 1 --2- 172.21.15.118:0/4044358148 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f921c1030c0 0x7f921c1a0fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:16:40.888 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:16:40.888+0000 7f921b200640 1 --2- 172.21.15.118:0/4044358148 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f921c102700 0x7f921c199f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:16:40.888 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:16:40.888+0000 7f921b200640 1 --2- 172.21.15.118:0/4044358148 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f921c101510 0x7f921c199a10 unknown :-1 s=CLOSED pgs=247 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:16:40.888 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:16:40.888+0000 7f921b200640 1 -- 172.21.15.118:0/4044358148 >> 172.21.15.118:0/4044358148 conn(0x7f921c0fca80 msgr2=0x7f921c0738d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:16:40.889 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:16:40.889+0000 7f921b200640 1 -- 172.21.15.118:0/4044358148 shutdown_connections 2024-03-24T00:16:40.889 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:16:40.889+0000 7f921b200640 1 -- 172.21.15.118:0/4044358148 wait complete. 2024-03-24T00:16:41.195 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:16:40 smithi118 ceph-mon[31535]: pgmap v427: 97 pgs: 97 active+clean; 672 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 5.9 KiB/s rd, 682 B/s wr, 12 op/s 2024-03-24T00:16:41.203 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:16:40 smithi189 ceph-mon[36187]: pgmap v427: 97 pgs: 97 active+clean; 672 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 5.9 KiB/s rd, 682 B/s wr, 12 op/s 2024-03-24T00:16:41.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:16:40 smithi193 ceph-mon[36878]: pgmap v427: 97 pgs: 97 active+clean; 672 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 5.9 KiB/s rd, 682 B/s wr, 12 op/s 2024-03-24T00:16:43.195 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:16:42 smithi118 ceph-mon[31535]: pgmap v428: 97 pgs: 97 active+clean; 672 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 85 B/s wr, 8 op/s 2024-03-24T00:16:43.203 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:16:42 smithi189 ceph-mon[36187]: pgmap v428: 97 pgs: 97 active+clean; 672 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 85 B/s wr, 8 op/s 2024-03-24T00:16:43.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:16:42 smithi193 ceph-mon[36878]: pgmap v428: 97 pgs: 97 active+clean; 672 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 85 B/s wr, 8 op/s 2024-03-24T00:16:44.195 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:16:43 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T00:16:44.203 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:16:43 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T00:16:44.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:16:43 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T00:16:45.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:16:44 smithi118 ceph-mon[31535]: pgmap v429: 97 pgs: 97 active+clean; 672 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 0 B/s wr, 10 op/s 2024-03-24T00:16:45.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:16:44 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T00:16:45.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:16:44 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T00:16:45.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:16:44 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T00:16:45.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:16:44 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T00:16:45.197 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:16:44 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T00:16:45.203 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:16:44 smithi189 ceph-mon[36187]: pgmap v429: 97 pgs: 97 active+clean; 672 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 0 B/s wr, 10 op/s 2024-03-24T00:16:45.203 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:16:44 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T00:16:45.203 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:16:44 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T00:16:45.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:16:44 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T00:16:45.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:16:44 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T00:16:45.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:16:44 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T00:16:45.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:16:44 smithi193 ceph-mon[36878]: pgmap v429: 97 pgs: 97 active+clean; 672 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 0 B/s wr, 10 op/s 2024-03-24T00:16:45.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:16:44 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T00:16:45.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:16:44 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T00:16:45.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:16:44 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T00:16:45.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:16:44 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T00:16:45.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:16:44 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T00:16:47.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:16:46 smithi118 ceph-mon[31535]: pgmap v430: 97 pgs: 97 active+clean; 673 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 5.5 KiB/s rd, 255 B/s wr, 11 op/s 2024-03-24T00:16:47.203 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:16:46 smithi189 ceph-mon[36187]: pgmap v430: 97 pgs: 97 active+clean; 673 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 5.5 KiB/s rd, 255 B/s wr, 11 op/s 2024-03-24T00:16:47.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:16:46 smithi193 ceph-mon[36878]: pgmap v430: 97 pgs: 97 active+clean; 673 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 5.5 KiB/s rd, 255 B/s wr, 11 op/s 2024-03-24T00:16:49.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:16:48 smithi118 ceph-mon[31535]: pgmap v431: 97 pgs: 97 active+clean; 673 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 255 B/s wr, 9 op/s 2024-03-24T00:16:49.203 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:16:48 smithi189 ceph-mon[36187]: pgmap v431: 97 pgs: 97 active+clean; 673 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 255 B/s wr, 9 op/s 2024-03-24T00:16:49.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:16:48 smithi193 ceph-mon[36878]: pgmap v431: 97 pgs: 97 active+clean; 673 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 255 B/s wr, 9 op/s 2024-03-24T00:16:51.195 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:16:50 smithi118 ceph-mon[31535]: pgmap v432: 97 pgs: 97 active+clean; 673 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 5.9 KiB/s rd, 255 B/s wr, 11 op/s 2024-03-24T00:16:51.203 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:16:50 smithi189 ceph-mon[36187]: pgmap v432: 97 pgs: 97 active+clean; 673 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 5.9 KiB/s rd, 255 B/s wr, 11 op/s 2024-03-24T00:16:51.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:16:50 smithi193 ceph-mon[36878]: pgmap v432: 97 pgs: 97 active+clean; 673 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 5.9 KiB/s rd, 255 B/s wr, 11 op/s 2024-03-24T00:16:52.199 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-03-24T00:16:52.199 DEBUG:teuthology.orchestra.run.smithi118:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-24T00:16:52.419 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:52.418+0000 7f0b62a00640 1 -- 172.21.15.118:0/413402605 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f0b5c0fffa0 msgr2=0x7f0b5c1003c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:16:52.420 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:52.418+0000 7f0b62a00640 1 --2- 172.21.15.118:0/413402605 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f0b5c0fffa0 0x7f0b5c1003c0 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7f0b4c009900 tx=0x7f0b4c02bcc0 comp rx=0 tx=0).stop 2024-03-24T00:16:52.420 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:52.418+0000 7f0b62a00640 1 -- 172.21.15.118:0/413402605 shutdown_connections 2024-03-24T00:16:52.420 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:52.418+0000 7f0b62a00640 1 --2- 172.21.15.118:0/413402605 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f0b5c1012a0 0x7f0b5c101720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:16:52.420 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:52.418+0000 7f0b62a00640 1 --2- 172.21.15.118:0/413402605 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0b5c100900 0x7f0b5c100d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:16:52.420 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:52.418+0000 7f0b62a00640 1 --2- 172.21.15.118:0/413402605 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f0b5c0fffa0 0x7f0b5c1003c0 unknown :-1 s=CLOSED pgs=235 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:16:52.420 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:52.418+0000 7f0b62a00640 1 -- 172.21.15.118:0/413402605 >> 172.21.15.118:0/413402605 conn(0x7f0b5c0fbb40 msgr2=0x7f0b5c0fdf60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:16:52.420 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:52.419+0000 7f0b62a00640 1 -- 172.21.15.118:0/413402605 shutdown_connections 2024-03-24T00:16:52.420 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:52.419+0000 7f0b62a00640 1 -- 172.21.15.118:0/413402605 wait complete. 2024-03-24T00:16:52.421 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:52.420+0000 7f0b62a00640 1 Processor -- start 2024-03-24T00:16:52.421 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:52.420+0000 7f0b62a00640 1 -- start start 2024-03-24T00:16:52.421 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:52.420+0000 7f0b62a00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0b5c0fffa0 0x7f0b5c105b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:16:52.421 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:52.420+0000 7f0b62a00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f0b5c100900 0x7f0b5c106060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:16:52.421 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:52.421+0000 7f0b62a00640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f0b5c1012a0 0x7f0b5c102690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:16:52.421 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:52.421+0000 7f0b62a00640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0b5c10f550 con 0x7f0b5c1012a0 2024-03-24T00:16:52.422 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:52.421+0000 7f0b62a00640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0b5c10f3d0 con 0x7f0b5c0fffa0 2024-03-24T00:16:52.422 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:52.421+0000 7f0b62a00640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0b5c10f6d0 con 0x7f0b5c100900 2024-03-24T00:16:52.423 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:52.421+0000 7f0b5be00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0b5c0fffa0 0x7f0b5c105b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:16:52.423 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:52.421+0000 7f0b5be00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0b5c0fffa0 0x7f0b5c105b20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.118:53222/0 (socket says 172.21.15.118:53222) 2024-03-24T00:16:52.423 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:52.421+0000 7f0b5be00640 1 -- 172.21.15.118:0/3703167764 learned_addr learned my addr 172.21.15.118:0/3703167764 (peer_addr_for_me v2:172.21.15.118:0/0) 2024-03-24T00:16:52.423 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:52.421+0000 7f0b5b400640 1 --2- 172.21.15.118:0/3703167764 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f0b5c100900 0x7f0b5c106060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:16:52.423 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:52.421+0000 7f0b60a00640 1 --2- 172.21.15.118:0/3703167764 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f0b5c1012a0 0x7f0b5c102690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:16:52.423 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:52.422+0000 7f0b5b400640 1 -- 172.21.15.118:0/3703167764 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0b5c0fffa0 msgr2=0x7f0b5c105b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:16:52.423 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:52.422+0000 7f0b5b400640 1 --2- 172.21.15.118:0/3703167764 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0b5c0fffa0 0x7f0b5c105b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:16:52.423 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:52.422+0000 7f0b5b400640 1 -- 172.21.15.118:0/3703167764 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f0b5c1012a0 msgr2=0x7f0b5c102690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:16:52.423 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:52.422+0000 7f0b5b400640 1 --2- 172.21.15.118:0/3703167764 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f0b5c1012a0 0x7f0b5c102690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:16:52.423 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:52.422+0000 7f0b5b400640 1 -- 172.21.15.118:0/3703167764 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0b5c102f50 con 0x7f0b5c100900 2024-03-24T00:16:52.423 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:52.422+0000 7f0b5b400640 1 --2- 172.21.15.118:0/3703167764 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f0b5c100900 0x7f0b5c106060 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7f0b4800c970 tx=0x7f0b4800ce30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:16:52.424 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:52.422+0000 7f0b58c00640 1 -- 172.21.15.118:0/3703167764 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0b48007bb0 con 0x7f0b5c100900 2024-03-24T00:16:52.424 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:52.422+0000 7f0b58c00640 1 -- 172.21.15.118:0/3703167764 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f0b48007d50 con 0x7f0b5c100900 2024-03-24T00:16:52.424 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:52.423+0000 7f0b58c00640 1 -- 172.21.15.118:0/3703167764 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0b4800f450 con 0x7f0b5c100900 2024-03-24T00:16:52.424 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:52.423+0000 7f0b62a00640 1 -- 172.21.15.118:0/3703167764 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0b5c1997e0 con 0x7f0b5c100900 2024-03-24T00:16:52.424 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:52.423+0000 7f0b62a00640 1 -- 172.21.15.118:0/3703167764 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0b5c199ce0 con 0x7f0b5c100900 2024-03-24T00:16:52.428 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:52.427+0000 7f0b657f7740 1 Processor -- start 2024-03-24T00:16:52.428 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:52.428+0000 7f0b657f7740 1 -- start start 2024-03-24T00:16:52.429 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:52.428+0000 7f0b657f7740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5641bf5dfe80 0x5641bf5e02a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:16:52.429 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:52.428+0000 7f0b657f7740 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5641bf5e0870 0x5641bf5e5370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:16:52.429 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:52.428+0000 7f0b657f7740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5641bf5e59e0 0x5641bf5e7e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:16:52.429 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:52.428+0000 7f0b657f7740 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x5641bf545250 con 0x5641bf5e0870 2024-03-24T00:16:52.429 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:52.428+0000 7f0b657f7740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x5641bf4eda30 con 0x5641bf5e59e0 2024-03-24T00:16:52.429 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:52.428+0000 7f0b657f7740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x5641bf5412f0 con 0x5641bf5dfe80 2024-03-24T00:16:52.430 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:52.429+0000 7f0b5be00640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5641bf5e0870 0x5641bf5e5370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:16:52.430 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:52.429+0000 7f0b58c00640 1 -- 172.21.15.118:0/3703167764 <== mon.1 v2:172.21.15.193:3300/0 4 ==== mgrmap(e 19) v1 ==== 137342+0+0 (secure 0 0 0) 0x7f0b48004f30 con 0x7f0b5c100900 2024-03-24T00:16:52.430 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:52.429+0000 7f0b5be00640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5641bf5e0870 0x5641bf5e5370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.118:3300/0 says I am v2:172.21.15.118:40792/0 (socket says 172.21.15.118:40792) 2024-03-24T00:16:52.430 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:52.429+0000 7f0b5be00640 1 -- 172.21.15.118:0/675084025 learned_addr learned my addr 172.21.15.118:0/675084025 (peer_addr_for_me v2:172.21.15.118:0/0) 2024-03-24T00:16:52.430 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:52.429+0000 7f0b5b400640 1 --2- 172.21.15.118:0/675084025 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5641bf5e59e0 0x5641bf5e7e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:16:52.430 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:52.429+0000 7f0b60a00640 1 --2- 172.21.15.118:0/675084025 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5641bf5dfe80 0x5641bf5e02a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:16:52.431 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:52.430+0000 7f0b60a00640 1 -- 172.21.15.118:0/675084025 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5641bf5e59e0 msgr2=0x5641bf5e7e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:16:52.431 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:52.430+0000 7f0b60a00640 1 --2- 172.21.15.118:0/675084025 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5641bf5e59e0 0x5641bf5e7e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:16:52.431 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:52.430+0000 7f0b60a00640 1 -- 172.21.15.118:0/675084025 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5641bf5e0870 msgr2=0x5641bf5e5370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:16:52.431 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:52.430+0000 7f0b60a00640 1 --2- 172.21.15.118:0/675084025 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5641bf5e0870 0x5641bf5e5370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:16:52.431 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:52.430+0000 7f0b60a00640 1 -- 172.21.15.118:0/675084025 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5641bf4d8690 con 0x5641bf5dfe80 2024-03-24T00:16:52.431 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:52.430+0000 7f0b60a00640 1 --2- 172.21.15.118:0/675084025 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5641bf5dfe80 0x5641bf5e02a0 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7f0b54009e20 tx=0x7f0b54010740 comp rx=0 tx=0).ready entity=mon.1 client_cookie=6612a1c88a2e0e67 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:16:52.431 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:52.430+0000 7f0b51200640 1 -- 172.21.15.118:0/675084025 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0b54016070 con 0x5641bf5dfe80 2024-03-24T00:16:52.431 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:52.431+0000 7f0b51200640 1 -- 172.21.15.118:0/675084025 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f0b54004d20 con 0x5641bf5dfe80 2024-03-24T00:16:52.431 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:52.431+0000 7f0b51200640 1 -- 172.21.15.118:0/675084025 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0b540128b0 con 0x5641bf5dfe80 2024-03-24T00:16:52.432 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:52.431+0000 7f0b58c00640 1 --2- 172.21.15.118:0/3703167764 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7f0b400a2910 0x7f0b400a4dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:16:52.433 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:52.431+0000 7f0b58c00640 1 -- 172.21.15.118:0/3703167764 <== mon.1 v2:172.21.15.193:3300/0 5 ==== osd_map(90..90 src has 1..90) v4 ==== 8277+0+0 (secure 0 0 0) 0x7f0b480cec00 con 0x7f0b5c100900 2024-03-24T00:16:52.433 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:52.432+0000 7f0b657f7740 1 -- 172.21.15.118:0/675084025 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5641bf5dfe80 msgr2=0x5641bf5e02a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:16:52.433 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:52.432+0000 7f0b657f7740 1 --2- 172.21.15.118:0/675084025 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5641bf5dfe80 0x5641bf5e02a0 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7f0b54009e20 tx=0x7f0b54010740 comp rx=0 tx=0).stop 2024-03-24T00:16:52.433 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:52.432+0000 7f0b5be00640 1 --2- 172.21.15.118:0/3703167764 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7f0b400a2910 0x7f0b400a4dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:16:52.433 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:52.432+0000 7f0b657f7740 1 -- 172.21.15.118:0/675084025 shutdown_connections 2024-03-24T00:16:52.433 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:52.432+0000 7f0b657f7740 1 --2- 172.21.15.118:0/675084025 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5641bf5e59e0 0x5641bf5e7e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:16:52.433 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:52.432+0000 7f0b657f7740 1 --2- 172.21.15.118:0/675084025 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5641bf5e0870 0x5641bf5e5370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:16:52.433 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:52.432+0000 7f0b657f7740 1 --2- 172.21.15.118:0/675084025 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5641bf5dfe80 0x5641bf5e02a0 unknown :-1 s=CLOSED pgs=237 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:16:52.433 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:52.432+0000 7f0b657f7740 1 -- 172.21.15.118:0/675084025 >> 172.21.15.118:0/675084025 conn(0x5641bf5a8200 msgr2=0x5641bf5db7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:16:52.433 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:52.432+0000 7f0b657f7740 1 -- 172.21.15.118:0/675084025 shutdown_connections 2024-03-24T00:16:52.433 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:52.432+0000 7f0b657f7740 1 -- 172.21.15.118:0/675084025 wait complete. 2024-03-24T00:16:52.434 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:52.433+0000 7f0b657f7740 1 -- start start 2024-03-24T00:16:52.434 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:52.433+0000 7f0b657f7740 1 Processor -- start 2024-03-24T00:16:52.434 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:52.433+0000 7f0b5be00640 1 --2- 172.21.15.118:0/3703167764 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7f0b400a2910 0x7f0b400a4dd0 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x5641bf5dc740 tx=0x7f0b4c02e040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:16:52.434 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:52.434+0000 7f0b657f7740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5641bf5dfe80 0x5641bf5e9e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:16:52.434 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:52.434+0000 7f0b657f7740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5641bf5e0870 0x5641bf5ea3c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:16:52.435 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:52.434+0000 7f0b657f7740 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5641bf5e59e0 0x5641bf5f1400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:16:52.435 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:52.434+0000 7f0b657f7740 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x5641bf5e8990 con 0x5641bf5e59e0 2024-03-24T00:16:52.435 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:52.434+0000 7f0b657f7740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x5641bf500e80 con 0x5641bf5dfe80 2024-03-24T00:16:52.435 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:52.434+0000 7f0b657f7740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x5641bf5e8b10 con 0x5641bf5e0870 2024-03-24T00:16:52.435 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:52.434+0000 7f0b657f7740 10 client.0 _ll_register_callbacks cb 0x5641bf513130 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T00:16:52.436 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:52.434+0000 7f0b5be00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5641bf5dfe80 0x5641bf5e9e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:16:52.437 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:52.434+0000 7f0b5be00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5641bf5dfe80 0x5641bf5e9e80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.118:53234/0 (socket says 172.21.15.118:53234) 2024-03-24T00:16:52.437 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:52.434+0000 7f0b5be00640 1 -- 172.21.15.118:0/1471483578 learned_addr learned my addr 172.21.15.118:0/1471483578 (peer_addr_for_me v2:172.21.15.118:0/0) 2024-03-24T00:16:52.437 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:52.434+0000 7f0b5b400640 1 --2- 172.21.15.118:0/1471483578 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5641bf5e0870 0x5641bf5ea3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:16:52.437 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:52.434+0000 7f0b60a00640 1 --2- 172.21.15.118:0/1471483578 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5641bf5e59e0 0x5641bf5f1400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:16:52.437 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:52.434+0000 7f0b5b400640 1 -- 172.21.15.118:0/1471483578 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5641bf5dfe80 msgr2=0x5641bf5e9e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:16:52.437 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:52.434+0000 7f0b5b400640 1 --2- 172.21.15.118:0/1471483578 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5641bf5dfe80 0x5641bf5e9e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:16:52.437 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:52.434+0000 7f0b5b400640 1 -- 172.21.15.118:0/1471483578 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5641bf5e59e0 msgr2=0x5641bf5f1400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:16:52.437 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:52.434+0000 7f0b5b400640 1 --2- 172.21.15.118:0/1471483578 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5641bf5e59e0 0x5641bf5f1400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:16:52.437 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:52.435+0000 7f0b5b400640 1 -- 172.21.15.118:0/1471483578 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5641bf3237d0 con 0x5641bf5e0870 2024-03-24T00:16:52.437 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:52.435+0000 7f0b60a00640 1 --2- 172.21.15.118:0/1471483578 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5641bf5e59e0 0x5641bf5f1400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-24T00:16:52.437 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:52.435+0000 7f0b5b400640 1 --2- 172.21.15.118:0/1471483578 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5641bf5e0870 0x5641bf5ea3c0 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7f0b480cc520 tx=0x7f0b4809b250 comp rx=0 tx=0).ready entity=mon.1 client_cookie=df3cc67fa128567a server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:16:52.437 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:52.435+0000 7f0b45600640 10 client.0 ms_handle_connect on v2:172.21.15.193:3300/0 2024-03-24T00:16:52.437 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:52.435+0000 7f0b45600640 1 -- 172.21.15.118:0/1471483578 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0b4801d040 con 0x5641bf5e0870 2024-03-24T00:16:52.438 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:52.435+0000 7f0b45600640 1 -- 172.21.15.118:0/1471483578 <== mon.1 v2:172.21.15.193:3300/0 2 ==== osd_map(90..90 src has 1..90) v4 ==== 8277+0+0 (secure 0 0 0) 0x7f0b480a36c0 con 0x5641bf5e0870 2024-03-24T00:16:52.438 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:52.435+0000 7f0b657f7740 1 -- 172.21.15.118:0/1471483578 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f0b54004d20 con 0x5641bf5e0870 2024-03-24T00:16:52.438 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:52.435+0000 7f0b45600640 1 -- 172.21.15.118:0/1471483578 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) v2 ==== 24+0+0 (secure 0 0 0) 0x7f0b4809bab0 con 0x5641bf5e0870 2024-03-24T00:16:52.438 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:52.435+0000 7f0b657f7740 10 client.25645 fetch_fsmap learned FSMap version 28 2024-03-24T00:16:52.438 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:52.435+0000 7f0b657f7740 1 -- 172.21.15.118:0/1471483578 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) v3 -- 0x5641bf2593f0 con 0x5641bf5e0870 2024-03-24T00:16:52.438 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:52.436+0000 7f0b45600640 1 -- 172.21.15.118:0/1471483578 <== mon.1 v2:172.21.15.193:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f0b54004d20 con 0x5641bf5e0870 2024-03-24T00:16:52.438 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:52.436+0000 7f0b45600640 1 -- 172.21.15.118:0/1471483578 <== mon.1 v2:172.21.15.193:3300/0 5 ==== fsmap(e 28) v1 ==== 7934+0+0 (secure 0 0 0) 0x7f0b480bb070 con 0x5641bf5e0870 2024-03-24T00:16:52.438 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:52.436+0000 7f0b657f7740 10 client.25645 fetch_fsmap finished waiting for FSMap version 28 2024-03-24T00:16:52.438 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:52.436+0000 7f0b657f7740 10 client.25645 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-24T00:16:52.438 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:52.436+0000 7f0b657f7740 20 client.25645 populate_metadata read hostname 'smithi118' 2024-03-24T00:16:52.438 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:52.436+0000 7f0b657f7740 1 --2- 172.21.15.118:0/1471483578 >> [v2:172.21.15.189:6838/4074484168,v1:172.21.15.189:6839/4074484168] conn(0x5641bf5f5c30 0x5641bf5f8060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:16:52.438 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:52.436+0000 7f0b657f7740 4 client.25645 mds_command: new command op to 24569 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-24T00:16:52.438 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:52.436+0000 7f0b657f7740 1 -- 172.21.15.118:0/1471483578 --> [v2:172.21.15.189:6838/4074484168,v1:172.21.15.189:6839/4074484168] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x5641bf552790 con 0x5641bf5f5c30 2024-03-24T00:16:52.440 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:52.436+0000 7f0b5be00640 1 --2- 172.21.15.118:0/1471483578 >> [v2:172.21.15.189:6838/4074484168,v1:172.21.15.189:6839/4074484168] conn(0x5641bf5f5c30 0x5641bf5f8060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:16:52.440 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:52.437+0000 7f0b5be00640 1 --2- 172.21.15.118:0/1471483578 >> [v2:172.21.15.189:6838/4074484168,v1:172.21.15.189:6839/4074484168] conn(0x5641bf5f5c30 0x5641bf5f8060 crc :-1 s=READY pgs=53 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=abe027cae32b1f2c server_cookie=f57500b05330526f in_seq=0 out_seq=0 2024-03-24T00:16:52.440 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:52.437+0000 7f0b45600640 10 client.25645 ms_handle_connect on v2:172.21.15.189:6838/4074484168 2024-03-24T00:16:52.440 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:52.438+0000 7f0b45600640 1 -- 172.21.15.118:0/1471483578 <== mds.0 v2:172.21.15.189:6838/4074484168 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x5641bf552790 con 0x5641bf5f5c30 2024-03-24T00:16:52.440 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:52.438+0000 7f0b45600640 10 client.25645 handle_command_reply: tid=0 2024-03-24T00:16:52.440 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:52.438+0000 7f0b657f7740 1 client.25645 shutdown 2024-03-24T00:16:52.440 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:52.438+0000 7f0b657f7740 1 -- 172.21.15.118:0/1471483578 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5641bf5e0870 msgr2=0x5641bf5ea3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:16:52.440 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:52.438+0000 7f0b657f7740 1 --2- 172.21.15.118:0/1471483578 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5641bf5e0870 0x5641bf5ea3c0 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7f0b480cc520 tx=0x7f0b4809b250 comp rx=0 tx=0).stop 2024-03-24T00:16:52.440 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:52.439+0000 7f0b657f7740 1 -- 172.21.15.118:0/1471483578 shutdown_connections 2024-03-24T00:16:52.440 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:52.439+0000 7f0b657f7740 1 --2- 172.21.15.118:0/1471483578 >> [v2:172.21.15.189:6838/4074484168,v1:172.21.15.189:6839/4074484168] conn(0x5641bf5f5c30 0x5641bf5f8060 crc :-1 s=READY pgs=53 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:16:52.440 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:52.439+0000 7f0b657f7740 1 --2- 172.21.15.118:0/1471483578 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5641bf5e59e0 0x5641bf5f1400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:16:52.440 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:52.439+0000 7f0b657f7740 1 --2- 172.21.15.118:0/1471483578 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5641bf5e0870 0x5641bf5ea3c0 unknown :-1 s=CLOSED pgs=238 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:16:52.441 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:52.439+0000 7f0b45600640 0 client.25645 ms_handle_reset on v2:172.21.15.189:6838/4074484168 2024-03-24T00:16:52.441 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:52.439+0000 7f0b657f7740 1 --2- 172.21.15.118:0/1471483578 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5641bf5dfe80 0x5641bf5e9e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:16:52.441 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:52.439+0000 7f0b657f7740 1 -- 172.21.15.118:0/1471483578 >> 172.21.15.118:0/1471483578 conn(0x5641bf5a8200 msgr2=0x5641bf5daa90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:16:52.441 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:52.439+0000 7f0b657f7740 1 -- 172.21.15.118:0/1471483578 shutdown_connections 2024-03-24T00:16:52.441 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:52.439+0000 7f0b657f7740 1 -- 172.21.15.118:0/1471483578 wait complete. 2024-03-24T00:16:52.441 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:52.439+0000 7f0b657f7740 20 client.25645 trim_cache size 0 max 16384 2024-03-24T00:16:52.472 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:52.472+0000 7f0b657f7740 1 Processor -- start 2024-03-24T00:16:52.472 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:52.472+0000 7f0b657f7740 1 -- start start 2024-03-24T00:16:52.473 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:52.472+0000 7f0b657f7740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5641bf5dfe80 0x5641bf5dea30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:16:52.473 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:52.472+0000 7f0b657f7740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5641bf5e0870 0x5641bf5f13c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:16:52.473 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:52.472+0000 7f0b657f7740 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5641bf5e59e0 0x5641bf5def70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:16:52.473 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:52.472+0000 7f0b657f7740 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0b5c1b79d0 con 0x5641bf5e59e0 2024-03-24T00:16:52.474 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:52.472+0000 7f0b657f7740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0b5c1b9350 con 0x5641bf5e0870 2024-03-24T00:16:52.474 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:52.472+0000 7f0b657f7740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0b5c1b8030 con 0x5641bf5dfe80 2024-03-24T00:16:52.474 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:52.473+0000 7f0b5be00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5641bf5dfe80 0x5641bf5dea30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:16:52.474 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:52.473+0000 7f0b5b400640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5641bf5e0870 0x5641bf5f13c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:16:52.474 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:52.473+0000 7f0b5b400640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5641bf5e0870 0x5641bf5f13c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.118:53250/0 (socket says 172.21.15.118:53250) 2024-03-24T00:16:52.474 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:52.473+0000 7f0b5b400640 1 -- 172.21.15.118:0/1080908172 learned_addr learned my addr 172.21.15.118:0/1080908172 (peer_addr_for_me v2:172.21.15.118:0/0) 2024-03-24T00:16:52.474 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:52.473+0000 7f0b5be00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5641bf5dfe80 0x5641bf5dea30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.118:38484/0 (socket says 172.21.15.118:38484) 2024-03-24T00:16:52.474 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:52.473+0000 7f0b60a00640 1 --2- 172.21.15.118:0/1080908172 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5641bf5e59e0 0x5641bf5def70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:16:52.475 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:52.473+0000 7f0b5b400640 1 -- 172.21.15.118:0/1080908172 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5641bf5dfe80 msgr2=0x5641bf5dea30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:16:52.475 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:52.473+0000 7f0b5b400640 1 --2- 172.21.15.118:0/1080908172 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5641bf5dfe80 0x5641bf5dea30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:16:52.475 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:52.473+0000 7f0b5b400640 1 -- 172.21.15.118:0/1080908172 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5641bf5e59e0 msgr2=0x5641bf5def70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:16:52.475 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:52.473+0000 7f0b5b400640 1 --2- 172.21.15.118:0/1080908172 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5641bf5e59e0 0x5641bf5def70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:16:52.475 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:52.473+0000 7f0b5b400640 1 -- 172.21.15.118:0/1080908172 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5641bf2190f0 con 0x5641bf5e0870 2024-03-24T00:16:52.475 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:52.473+0000 7f0b60a00640 1 --2- 172.21.15.118:0/1080908172 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5641bf5e59e0 0x5641bf5def70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T00:16:52.475 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:52.474+0000 7f0b5b400640 1 --2- 172.21.15.118:0/1080908172 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5641bf5e0870 0x5641bf5f13c0 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7f0b4809b0b0 tx=0x7f0b480998c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=4c8799bc82df3b16 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:16:52.475 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:52.474+0000 7f0b47e00640 1 -- 172.21.15.118:0/1080908172 <== mon.2 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0b4801d030 con 0x5641bf5e0870 2024-03-24T00:16:52.475 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:52.474+0000 7f0b47e00640 1 -- 172.21.15.118:0/1080908172 <== mon.2 v2:172.21.15.189:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f0b480b4240 con 0x5641bf5e0870 2024-03-24T00:16:52.475 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:52.474+0000 7f0b47e00640 1 -- 172.21.15.118:0/1080908172 <== mon.2 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0b480cc770 con 0x5641bf5e0870 2024-03-24T00:16:52.476 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:52.475+0000 7f0b657f7740 1 -- 172.21.15.118:0/1080908172 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5641bf5e0870 msgr2=0x5641bf5f13c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:16:52.476 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:52.475+0000 7f0b657f7740 1 --2- 172.21.15.118:0/1080908172 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5641bf5e0870 0x5641bf5f13c0 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7f0b4809b0b0 tx=0x7f0b480998c0 comp rx=0 tx=0).stop 2024-03-24T00:16:52.476 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:52.475+0000 7f0b657f7740 1 -- 172.21.15.118:0/1080908172 shutdown_connections 2024-03-24T00:16:52.476 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:52.475+0000 7f0b657f7740 1 --2- 172.21.15.118:0/1080908172 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5641bf5e59e0 0x5641bf5def70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:16:52.476 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:52.475+0000 7f0b657f7740 1 --2- 172.21.15.118:0/1080908172 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5641bf5e0870 0x5641bf5f13c0 unknown :-1 s=CLOSED pgs=251 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:16:52.476 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:52.475+0000 7f0b657f7740 1 --2- 172.21.15.118:0/1080908172 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5641bf5dfe80 0x5641bf5dea30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:16:52.476 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:52.475+0000 7f0b657f7740 1 -- 172.21.15.118:0/1080908172 >> 172.21.15.118:0/1080908172 conn(0x5641bf5a8200 msgr2=0x5641bf5e1b70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:16:52.477 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:52.475+0000 7f0b657f7740 1 -- 172.21.15.118:0/1080908172 shutdown_connections 2024-03-24T00:16:52.477 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:52.475+0000 7f0b657f7740 1 -- 172.21.15.118:0/1080908172 wait complete. 2024-03-24T00:16:52.477 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:52.476+0000 7f0b657f7740 1 -- start start 2024-03-24T00:16:52.477 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:52.476+0000 7f0b657f7740 1 Processor -- start 2024-03-24T00:16:52.477 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:52.477+0000 7f0b657f7740 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5641bf5dfe80 0x5641bf5eee90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:16:52.477 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:52.477+0000 7f0b657f7740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5641bf5e0870 0x5641bf5ef3d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:16:52.477 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:52.477+0000 7f0b657f7740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5641bf5e59e0 0x5641bf5f03e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:16:52.478 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:52.477+0000 7f0b657f7740 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0b5c1b8cf0 con 0x5641bf5dfe80 2024-03-24T00:16:52.478 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:52.477+0000 7f0b657f7740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0b5c1b8690 con 0x5641bf5e0870 2024-03-24T00:16:52.478 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:52.477+0000 7f0b657f7740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0b5c1a0b60 con 0x5641bf5e59e0 2024-03-24T00:16:52.478 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:52.477+0000 7f0b657f7740 10 client.0 _ll_register_callbacks cb 0x7f0b480058a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T00:16:52.479 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:52.477+0000 7f0b60a00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5641bf5e59e0 0x5641bf5f03e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:16:52.479 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:52.477+0000 7f0b60a00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5641bf5e59e0 0x5641bf5f03e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.118:38492/0 (socket says 172.21.15.118:38492) 2024-03-24T00:16:52.479 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:52.477+0000 7f0b60a00640 1 -- 172.21.15.118:0/1598662438 learned_addr learned my addr 172.21.15.118:0/1598662438 (peer_addr_for_me v2:172.21.15.118:0/0) 2024-03-24T00:16:52.479 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:52.477+0000 7f0b5b400640 1 --2- 172.21.15.118:0/1598662438 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5641bf5e0870 0x5641bf5ef3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:16:52.480 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:52.477+0000 7f0b5be00640 1 --2- 172.21.15.118:0/1598662438 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5641bf5dfe80 0x5641bf5eee90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:16:52.480 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:52.478+0000 7f0b5b400640 1 -- 172.21.15.118:0/1598662438 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5641bf5e59e0 msgr2=0x5641bf5f03e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:16:52.480 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:52.478+0000 7f0b5b400640 1 --2- 172.21.15.118:0/1598662438 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5641bf5e59e0 0x5641bf5f03e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:16:52.480 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:52.478+0000 7f0b5b400640 1 -- 172.21.15.118:0/1598662438 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5641bf5dfe80 msgr2=0x5641bf5eee90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:16:52.480 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:52.478+0000 7f0b5b400640 1 --2- 172.21.15.118:0/1598662438 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5641bf5dfe80 0x5641bf5eee90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:16:52.480 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:52.478+0000 7f0b5b400640 1 -- 172.21.15.118:0/1598662438 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5641bf5f09d0 con 0x5641bf5e0870 2024-03-24T00:16:52.480 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:52.478+0000 7f0b5be00640 1 --2- 172.21.15.118:0/1598662438 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5641bf5dfe80 0x5641bf5eee90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T00:16:52.480 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:52.478+0000 7f0b5b400640 1 --2- 172.21.15.118:0/1598662438 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5641bf5e0870 0x5641bf5ef3d0 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x5641bf5f2330 tx=0x7f0b480cc520 comp rx=0 tx=0).ready entity=mon.2 client_cookie=2d4ec96069d3492b server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:16:52.480 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:52.478+0000 7f0b45600640 10 client.0 ms_handle_connect on v2:172.21.15.189:3300/0 2024-03-24T00:16:52.480 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:52.478+0000 7f0b45600640 1 -- 172.21.15.118:0/1598662438 <== mon.2 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0b480a3b50 con 0x5641bf5e0870 2024-03-24T00:16:52.480 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:52.478+0000 7f0b657f7740 1 -- 172.21.15.118:0/1598662438 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f0b480b4240 con 0x5641bf5e0870 2024-03-24T00:16:52.480 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:52.478+0000 7f0b45600640 1 -- 172.21.15.118:0/1598662438 <== mon.2 v2:172.21.15.189:3300/0 2 ==== osd_map(90..90 src has 1..90) v4 ==== 8277+0+0 (secure 0 0 0) 0x7f0b4809f0f0 con 0x5641bf5e0870 2024-03-24T00:16:52.480 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:52.478+0000 7f0b45600640 1 -- 172.21.15.118:0/1598662438 <== mon.2 v2:172.21.15.189:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) v2 ==== 24+0+0 (secure 0 0 0) 0x7f0b480ccbb0 con 0x5641bf5e0870 2024-03-24T00:16:52.480 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:52.479+0000 7f0b657f7740 10 client.25643 fetch_fsmap learned FSMap version 28 2024-03-24T00:16:52.480 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:52.479+0000 7f0b657f7740 1 -- 172.21.15.118:0/1598662438 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) v3 -- 0x5641bf5f09d0 con 0x5641bf5e0870 2024-03-24T00:16:52.481 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:52.479+0000 7f0b45600640 1 -- 172.21.15.118:0/1598662438 <== mon.2 v2:172.21.15.189:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f0b480b4240 con 0x5641bf5e0870 2024-03-24T00:16:52.482 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:52.479+0000 7f0b45600640 1 -- 172.21.15.118:0/1598662438 <== mon.2 v2:172.21.15.189:3300/0 5 ==== fsmap(e 28) v1 ==== 7934+0+0 (secure 0 0 0) 0x7f0b4807d080 con 0x5641bf5e0870 2024-03-24T00:16:52.482 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:52.480+0000 7f0b657f7740 10 client.25643 fetch_fsmap finished waiting for FSMap version 28 2024-03-24T00:16:52.482 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:52.480+0000 7f0b657f7740 10 client.25643 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-24T00:16:52.482 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:52.480+0000 7f0b657f7740 20 client.25643 populate_metadata read hostname 'smithi118' 2024-03-24T00:16:52.482 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:52.480+0000 7f0b657f7740 1 --2- 172.21.15.118:0/1598662438 >> [v2:172.21.15.189:6838/4074484168,v1:172.21.15.189:6839/4074484168] conn(0x5641bf651de0 0x5641bf654200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:16:52.483 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:52.480+0000 7f0b657f7740 4 client.25643 mds_command: new command op to 24569 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-24T00:16:52.483 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:52.480+0000 7f0b657f7740 1 -- 172.21.15.118:0/1598662438 --> [v2:172.21.15.189:6838/4074484168,v1:172.21.15.189:6839/4074484168] -- command(tid 0: {"prefix": "scrub status"}) v1 -- 0x5641bf5e9c70 con 0x5641bf651de0 2024-03-24T00:16:52.483 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:52.480+0000 7f0b5be00640 1 --2- 172.21.15.118:0/1598662438 >> [v2:172.21.15.189:6838/4074484168,v1:172.21.15.189:6839/4074484168] conn(0x5641bf651de0 0x5641bf654200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:16:52.483 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:52.480+0000 7f0b45600640 10 client.25643 ms_handle_connect on v2:172.21.15.189:6838/4074484168 2024-03-24T00:16:52.483 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:52.480+0000 7f0b5be00640 1 --2- 172.21.15.118:0/1598662438 >> [v2:172.21.15.189:6838/4074484168,v1:172.21.15.189:6839/4074484168] conn(0x5641bf651de0 0x5641bf654200 crc :-1 s=READY pgs=54 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=4a6ed93c3d36d8f server_cookie=e839db853c1243c9 in_seq=0 out_seq=0 2024-03-24T00:16:52.483 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:52.481+0000 7f0b45600640 1 -- 172.21.15.118:0/1598662438 <== mds.0 v2:172.21.15.189:6838/4074484168 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+63 (crc 0 0 0) 0x5641bf5e9c70 con 0x5641bf651de0 2024-03-24T00:16:52.483 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:52.481+0000 7f0b45600640 10 client.25643 handle_command_reply: tid=0 2024-03-24T00:16:52.483 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:52.481+0000 7f0b657f7740 1 client.25643 shutdown 2024-03-24T00:16:52.483 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:52.481+0000 7f0b657f7740 1 -- 172.21.15.118:0/1598662438 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5641bf5e0870 msgr2=0x5641bf5ef3d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:16:52.483 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:52.481+0000 7f0b657f7740 1 --2- 172.21.15.118:0/1598662438 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5641bf5e0870 0x5641bf5ef3d0 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x5641bf5f2330 tx=0x7f0b480cc520 comp rx=0 tx=0).stop 2024-03-24T00:16:52.483 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:52.481+0000 7f0b657f7740 1 -- 172.21.15.118:0/1598662438 shutdown_connections 2024-03-24T00:16:52.483 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:52.481+0000 7f0b657f7740 1 --2- 172.21.15.118:0/1598662438 >> [v2:172.21.15.189:6838/4074484168,v1:172.21.15.189:6839/4074484168] conn(0x5641bf651de0 0x5641bf654200 crc :-1 s=READY pgs=54 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:16:52.483 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:52.481+0000 7f0b657f7740 1 --2- 172.21.15.118:0/1598662438 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5641bf5e59e0 0x5641bf5f03e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:16:52.483 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:52.481+0000 7f0b657f7740 1 --2- 172.21.15.118:0/1598662438 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5641bf5e0870 0x5641bf5ef3d0 unknown :-1 s=CLOSED pgs=252 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:16:52.484 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:52.481+0000 7f0b657f7740 1 --2- 172.21.15.118:0/1598662438 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5641bf5dfe80 0x5641bf5eee90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:16:52.484 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:52.481+0000 7f0b657f7740 1 -- 172.21.15.118:0/1598662438 >> 172.21.15.118:0/1598662438 conn(0x5641bf5a8200 msgr2=0x5641bf5daa90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:16:52.484 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:52.482+0000 7f0b45600640 0 client.25643 ms_handle_reset on v2:172.21.15.189:6838/4074484168 2024-03-24T00:16:52.484 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:52.482+0000 7f0b657f7740 1 -- 172.21.15.118:0/1598662438 shutdown_connections 2024-03-24T00:16:52.484 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:52.482+0000 7f0b657f7740 1 -- 172.21.15.118:0/1598662438 wait complete. 2024-03-24T00:16:52.484 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:52.482+0000 7f0b657f7740 20 client.25643 trim_cache size 0 max 16384 2024-03-24T00:16:52.484 INFO:teuthology.orchestra.run.smithi118.stdout:{ 2024-03-24T00:16:52.484 INFO:teuthology.orchestra.run.smithi118.stdout: "status": "no active scrubs running", 2024-03-24T00:16:52.484 INFO:teuthology.orchestra.run.smithi118.stdout: "scrubs": {} 2024-03-24T00:16:52.484 INFO:teuthology.orchestra.run.smithi118.stdout:} 2024-03-24T00:16:52.485 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:52.483+0000 7f0b62a00640 1 -- 172.21.15.118:0/3703167764 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7f0b400a2910 msgr2=0x7f0b400a4dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:16:52.485 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:52.483+0000 7f0b62a00640 1 --2- 172.21.15.118:0/3703167764 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7f0b400a2910 0x7f0b400a4dd0 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x5641bf5dc740 tx=0x7f0b4c02e040 comp rx=0 tx=0).stop 2024-03-24T00:16:52.485 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:52.483+0000 7f0b62a00640 1 -- 172.21.15.118:0/3703167764 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f0b5c100900 msgr2=0x7f0b5c106060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:16:52.485 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:52.483+0000 7f0b62a00640 1 --2- 172.21.15.118:0/3703167764 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f0b5c100900 0x7f0b5c106060 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7f0b4800c970 tx=0x7f0b4800ce30 comp rx=0 tx=0).stop 2024-03-24T00:16:52.485 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:52.483+0000 7f0b62a00640 1 -- 172.21.15.118:0/3703167764 shutdown_connections 2024-03-24T00:16:52.485 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:52.483+0000 7f0b62a00640 1 --2- 172.21.15.118:0/3703167764 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7f0b400a2910 0x7f0b400a4dd0 unknown :-1 s=CLOSED pgs=284 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:16:52.486 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:52.483+0000 7f0b62a00640 1 --2- 172.21.15.118:0/3703167764 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f0b5c1012a0 0x7f0b5c102690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:16:52.486 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:52.483+0000 7f0b62a00640 1 --2- 172.21.15.118:0/3703167764 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f0b5c100900 0x7f0b5c106060 unknown :-1 s=CLOSED pgs=236 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:16:52.486 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:52.483+0000 7f0b62a00640 1 --2- 172.21.15.118:0/3703167764 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0b5c0fffa0 0x7f0b5c105b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:16:52.486 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:52.483+0000 7f0b62a00640 1 -- 172.21.15.118:0/3703167764 >> 172.21.15.118:0/3703167764 conn(0x7f0b5c0fbb40 msgr2=0x7f0b5c0fd290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:16:52.486 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:52.483+0000 7f0b62a00640 1 -- 172.21.15.118:0/3703167764 shutdown_connections 2024-03-24T00:16:52.486 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:52.483+0000 7f0b62a00640 1 -- 172.21.15.118:0/3703167764 wait complete. 2024-03-24T00:16:52.506 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-03-24T00:16:52.506 DEBUG:teuthology.orchestra.run.smithi118:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-24T00:16:52.731 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:52.730+0000 7fdc26000640 1 -- 172.21.15.118:0/2370552964 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fdc201006c0 msgr2=0x7fdc20100b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:16:52.731 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:52.730+0000 7fdc26000640 1 --2- 172.21.15.118:0/2370552964 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fdc201006c0 0x7fdc20100b20 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7fdc0c009900 tx=0x7fdc0c02bc80 comp rx=0 tx=0).stop 2024-03-24T00:16:52.732 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:52.730+0000 7fdc26000640 1 -- 172.21.15.118:0/2370552964 shutdown_connections 2024-03-24T00:16:52.732 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:52.730+0000 7fdc26000640 1 --2- 172.21.15.118:0/2370552964 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fdc20101060 0x7fdc201014e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:16:52.732 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:52.730+0000 7fdc26000640 1 --2- 172.21.15.118:0/2370552964 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fdc201006c0 0x7fdc20100b20 unknown :-1 s=CLOSED pgs=182 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:16:52.732 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:52.730+0000 7fdc26000640 1 --2- 172.21.15.118:0/2370552964 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fdc200ffd60 0x7fdc20100180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:16:52.732 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:52.730+0000 7fdc26000640 1 -- 172.21.15.118:0/2370552964 >> 172.21.15.118:0/2370552964 conn(0x7fdc200fb980 msgr2=0x7fdc200fdda0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:16:52.732 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:52.731+0000 7fdc26000640 1 -- 172.21.15.118:0/2370552964 shutdown_connections 2024-03-24T00:16:52.732 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:52.731+0000 7fdc26000640 1 -- 172.21.15.118:0/2370552964 wait complete. 2024-03-24T00:16:52.732 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:52.732+0000 7fdc26000640 1 Processor -- start 2024-03-24T00:16:52.734 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:52.732+0000 7fdc26000640 1 -- start start 2024-03-24T00:16:52.734 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:52.733+0000 7fdc26000640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fdc200ffd60 0x7fdc20199b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:16:52.734 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:52.733+0000 7fdc26000640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fdc20101060 0x7fdc2019a0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:16:52.734 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:52.733+0000 7fdc26000640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fdc2019f130 0x7fdc201a1520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:16:52.734 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:52.733+0000 7fdc26000640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdc2010f330 con 0x7fdc200ffd60 2024-03-24T00:16:52.734 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:52.733+0000 7fdc1fe00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fdc20101060 0x7fdc2019a0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:16:52.734 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:52.733+0000 7fdc26000640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdc2010f1b0 con 0x7fdc20101060 2024-03-24T00:16:52.734 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:52.733+0000 7fdc26000640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdc2010f4b0 con 0x7fdc2019f130 2024-03-24T00:16:52.734 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:52.733+0000 7fdc25600640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fdc2019f130 0x7fdc201a1520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:16:52.734 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:52.733+0000 7fdc25600640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fdc2019f130 0x7fdc201a1520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.118:38510/0 (socket says 172.21.15.118:38510) 2024-03-24T00:16:52.734 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:52.733+0000 7fdc25600640 1 -- 172.21.15.118:0/1738454964 learned_addr learned my addr 172.21.15.118:0/1738454964 (peer_addr_for_me v2:172.21.15.118:0/0) 2024-03-24T00:16:52.735 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:52.733+0000 7fdc24c00640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fdc200ffd60 0x7fdc20199b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:16:52.735 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:52.733+0000 7fdc1fe00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fdc20101060 0x7fdc2019a0d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.118:53274/0 (socket says 172.21.15.118:53274) 2024-03-24T00:16:52.736 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:52.733+0000 7fdc25600640 1 -- 172.21.15.118:0/1738454964 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fdc20101060 msgr2=0x7fdc2019a0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:16:52.736 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:52.733+0000 7fdc25600640 1 --2- 172.21.15.118:0/1738454964 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fdc20101060 0x7fdc2019a0d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:16:52.736 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:52.733+0000 7fdc25600640 1 -- 172.21.15.118:0/1738454964 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fdc200ffd60 msgr2=0x7fdc20199b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:16:52.736 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:52.734+0000 7fdc25600640 1 --2- 172.21.15.118:0/1738454964 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fdc200ffd60 0x7fdc20199b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:16:52.736 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:52.734+0000 7fdc25600640 1 -- 172.21.15.118:0/1738454964 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdc2006b1f0 con 0x7fdc2019f130 2024-03-24T00:16:52.736 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:52.734+0000 7fdc24c00640 1 --2- 172.21.15.118:0/1738454964 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fdc200ffd60 0x7fdc20199b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T00:16:52.736 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:52.734+0000 7fdc25600640 1 --2- 172.21.15.118:0/1738454964 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fdc2019f130 0x7fdc201a1520 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7fdc1800cce0 tx=0x7fdc1800a620 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:16:52.736 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:52.734+0000 7fdc1d600640 1 -- 172.21.15.118:0/1738454964 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdc1800f450 con 0x7fdc2019f130 2024-03-24T00:16:52.736 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:52.734+0000 7fdc1d600640 1 -- 172.21.15.118:0/1738454964 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fdc1800fad0 con 0x7fdc2019f130 2024-03-24T00:16:52.736 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:52.734+0000 7fdc1d600640 1 -- 172.21.15.118:0/1738454964 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdc18018c10 con 0x7fdc2019f130 2024-03-24T00:16:52.736 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:52.734+0000 7fdc26000640 1 -- 172.21.15.118:0/1738454964 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdc2006b4e0 con 0x7fdc2019f130 2024-03-24T00:16:52.736 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:52.734+0000 7fdc26000640 1 -- 172.21.15.118:0/1738454964 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdc2006b9b0 con 0x7fdc2019f130 2024-03-24T00:16:52.737 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:52.735+0000 7fdc26000640 1 -- 172.21.15.118:0/1738454964 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdc200683b0 con 0x7fdc2019f130 2024-03-24T00:16:52.740 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:52.739+0000 7fdc1d600640 1 -- 172.21.15.118:0/1738454964 <== mon.1 v2:172.21.15.193:3300/0 4 ==== mgrmap(e 19) v1 ==== 137342+0+0 (secure 0 0 0) 0x7fdc1800f5f0 con 0x7fdc2019f130 2024-03-24T00:16:52.743 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:52.742+0000 7fdc1d600640 1 --2- 172.21.15.118:0/1738454964 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7fdc080a2990 0x7fdc080a4e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:16:52.743 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:52.742+0000 7fdc1d600640 1 -- 172.21.15.118:0/1738454964 <== mon.1 v2:172.21.15.193:3300/0 5 ==== osd_map(90..90 src has 1..90) v4 ==== 8277+0+0 (secure 0 0 0) 0x7fdc180cf090 con 0x7fdc2019f130 2024-03-24T00:16:52.744 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:52.742+0000 7fdc1d600640 1 -- 172.21.15.118:0/1738454964 <== mon.1 v2:172.21.15.193:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7fdc18014030 con 0x7fdc2019f130 2024-03-24T00:16:52.744 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:52.744+0000 7fdc24c00640 1 --2- 172.21.15.118:0/1738454964 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7fdc080a2990 0x7fdc080a4e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:16:52.746 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:52.745+0000 7fdc24c00640 1 --2- 172.21.15.118:0/1738454964 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7fdc080a2990 0x7fdc080a4e50 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7fdc14005d00 tx=0x7fdc1400b410 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:16:52.945 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:16:52 smithi118 ceph-mon[31535]: pgmap v433: 97 pgs: 97 active+clean; 673 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 255 B/s wr, 9 op/s 2024-03-24T00:16:52.996 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:52.996+0000 7fdc26000640 1 -- 172.21.15.118:0/1738454964 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fdc2010aab0 con 0x7fdc2019f130 2024-03-24T00:16:52.997 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:52.996+0000 7fdc1d600640 1 -- 172.21.15.118:0/1738454964 <== mon.1 v2:172.21.15.193:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+11478 (secure 0 0 0) 0x7fdc2010aab0 con 0x7fdc2019f130 2024-03-24T00:16:52.997 INFO:teuthology.orchestra.run.smithi118.stdout: 2024-03-24T00:16:52.998 INFO:teuthology.orchestra.run.smithi118.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14385,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.118:6835/3194741373","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6834","nonce":3194741373},{"type":"v1","addr":"172.21.15.118:6835","nonce":3194741373}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":4},{"gid":14397,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.118:6837/1374462956","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6836","nonce":1374462956},{"type":"v1","addr":"172.21.15.118:6837","nonce":1374462956}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":28,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-03-24T00:13:01.782474+0000","modified":"2024-03-24T00:15:38.235450+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24569,"mds_1":24550,"mds_2":24511,"mds_3":24499,"mds_4":24488},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14469":{"gid":14469,"name":"a","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.118:6841/3040946537","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6840","nonce":3040946537},{"type":"v1","addr":"172.21.15.118:6841","nonce":3040946537}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24439":{"gid":24439,"name":"l","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.193:6833/1023428412","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6832","nonce":1023428412},{"type":"v1","addr":"172.21.15.193:6833","nonce":1023428412}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24440":{"gid":24440,"name":"k","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.189:6833/2740024078","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6832","nonce":2740024078},{"type":"v1","addr":"172.21.15.189:6833","nonce":2740024078}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24488":{"gid":24488,"name":"e","rank":4,"incarnation":25,"state":"up:active","state_seq":10,"addr":"172.21.15.189:6835/599473205","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6834","nonce":599473205},{"type":"v1","addr":"172.21.15.189:6835","nonce":599473205}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24499":{"gid":24499,"name":"f","rank":3,"incarnation":23,"state":"up:active","state_seq":9,"addr":"172.21.15.193:6835/2055310541","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6834","nonce":2055310541},{"type":"v1","addr":"172.21.15.193:6835","nonce":2055310541}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24511":{"gid":24511,"name":"c","rank":2,"incarnation":21,"state":"up:active","state_seq":8,"addr":"172.21.15.193:6837/2921520053","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6836","nonce":2921520053},{"type":"v1","addr":"172.21.15.193:6837","nonce":2921520053}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24515":{"gid":24515,"name":"h","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.189:6837/3450187137","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6836","nonce":3450187137},{"type":"v1","addr":"172.21.15.189:6837","nonce":3450187137}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24532":{"gid":24532,"name":"i","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.193:6839/2126842603","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6838","nonce":2126842603},{"type":"v1","addr":"172.21.15.193:6839","nonce":2126842603}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24550":{"gid":24550,"name":"g","rank":1,"incarnation":19,"state":"up:active","state_seq":6,"addr":"172.21.15.118:6839/3285788503","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6838","nonce":3285788503},{"type":"v1","addr":"172.21.15.118:6839","nonce":3285788503}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24569":{"gid":24569,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.189:6839/4074484168","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6838","nonce":4074484168},{"type":"v1","addr":"172.21.15.189:6839","nonce":4074484168}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24569,"qdb_cluster":[24569,24499,24488,24511,24550]},"id":1}]} 2024-03-24T00:16:52.998 INFO:teuthology.orchestra.run.smithi118.stderr:dumped fsmap epoch 28 2024-03-24T00:16:53.000 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:52.999+0000 7fdc26000640 1 -- 172.21.15.118:0/1738454964 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7fdc080a2990 msgr2=0x7fdc080a4e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:16:53.000 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:52.999+0000 7fdc26000640 1 --2- 172.21.15.118:0/1738454964 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7fdc080a2990 0x7fdc080a4e50 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7fdc14005d00 tx=0x7fdc1400b410 comp rx=0 tx=0).stop 2024-03-24T00:16:53.000 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:52.999+0000 7fdc26000640 1 -- 172.21.15.118:0/1738454964 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fdc2019f130 msgr2=0x7fdc201a1520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:16:53.000 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:52.999+0000 7fdc26000640 1 --2- 172.21.15.118:0/1738454964 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fdc2019f130 0x7fdc201a1520 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7fdc1800cce0 tx=0x7fdc1800a620 comp rx=0 tx=0).stop 2024-03-24T00:16:53.000 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:52.999+0000 7fdc26000640 1 -- 172.21.15.118:0/1738454964 shutdown_connections 2024-03-24T00:16:53.000 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:52.999+0000 7fdc26000640 1 --2- 172.21.15.118:0/1738454964 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7fdc080a2990 0x7fdc080a4e50 unknown :-1 s=CLOSED pgs=285 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:16:53.000 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:52.999+0000 7fdc26000640 1 --2- 172.21.15.118:0/1738454964 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fdc2019f130 0x7fdc201a1520 unknown :-1 s=CLOSED pgs=239 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:16:53.000 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:52.999+0000 7fdc26000640 1 --2- 172.21.15.118:0/1738454964 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fdc20101060 0x7fdc2019a0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:16:53.000 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:52.999+0000 7fdc26000640 1 --2- 172.21.15.118:0/1738454964 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fdc200ffd60 0x7fdc20199b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:16:53.001 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:52.999+0000 7fdc26000640 1 -- 172.21.15.118:0/1738454964 >> 172.21.15.118:0/1738454964 conn(0x7fdc200fb980 msgr2=0x7fdc200fd480 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:16:53.001 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:52.999+0000 7fdc26000640 1 -- 172.21.15.118:0/1738454964 shutdown_connections 2024-03-24T00:16:53.001 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:52.999+0000 7fdc26000640 1 -- 172.21.15.118:0/1738454964 wait complete. 2024-03-24T00:16:53.020 INFO:teuthology.orchestra.run:Running command with timeout 120 2024-03-24T00:16:53.020 DEBUG:teuthology.orchestra.run.smithi118:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:4 damage ls 2024-03-24T00:16:53.203 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:16:52 smithi189 ceph-mon[36187]: pgmap v433: 97 pgs: 97 active+clean; 673 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 255 B/s wr, 9 op/s 2024-03-24T00:16:53.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:16:52 smithi193 ceph-mon[36878]: pgmap v433: 97 pgs: 97 active+clean; 673 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 255 B/s wr, 9 op/s 2024-03-24T00:16:53.220 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:53.219+0000 7ff98dc00640 1 -- 172.21.15.118:0/3204575797 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7ff988103150 msgr2=0x7ff9881099e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:16:53.220 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:53.219+0000 7ff98dc00640 1 --2- 172.21.15.118:0/3204575797 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7ff988103150 0x7ff9881099e0 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7ff980009ca0 tx=0x7ff98002c100 comp rx=0 tx=0).stop 2024-03-24T00:16:53.220 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:53.219+0000 7ff98dc00640 1 -- 172.21.15.118:0/3204575797 shutdown_connections 2024-03-24T00:16:53.220 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:53.219+0000 7ff98dc00640 1 --2- 172.21.15.118:0/3204575797 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7ff988103150 0x7ff9881099e0 unknown :-1 s=CLOSED pgs=240 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:16:53.220 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:53.219+0000 7ff98dc00640 1 --2- 172.21.15.118:0/3204575797 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7ff988102790 0x7ff988102c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:16:53.220 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:53.219+0000 7ff98dc00640 1 --2- 172.21.15.118:0/3204575797 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff9881015a0 0x7ff9881019c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:16:53.221 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:53.219+0000 7ff98dc00640 1 -- 172.21.15.118:0/3204575797 >> 172.21.15.118:0/3204575797 conn(0x7ff9880fcb10 msgr2=0x7ff9880fef50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:16:53.221 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:53.219+0000 7ff98dc00640 1 -- 172.21.15.118:0/3204575797 shutdown_connections 2024-03-24T00:16:53.221 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:53.219+0000 7ff98dc00640 1 -- 172.21.15.118:0/3204575797 wait complete. 2024-03-24T00:16:53.222 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:53.220+0000 7ff98dc00640 1 Processor -- start 2024-03-24T00:16:53.222 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:53.221+0000 7ff98dc00640 1 -- start start 2024-03-24T00:16:53.222 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:53.221+0000 7ff98dc00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff9881015a0 0x7ff988199a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:16:53.222 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:53.221+0000 7ff98dc00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7ff988102790 0x7ff988199f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:16:53.222 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:53.221+0000 7ff98dc00640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7ff988103150 0x7ff9881a0fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:16:53.222 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:53.221+0000 7ff98dc00640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff98810d230 con 0x7ff988103150 2024-03-24T00:16:53.222 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:53.221+0000 7ff98dc00640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff98810d0b0 con 0x7ff9881015a0 2024-03-24T00:16:53.222 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:53.221+0000 7ff98dc00640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff98810d3b0 con 0x7ff988102790 2024-03-24T00:16:53.222 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:53.221+0000 7ff987e00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff9881015a0 0x7ff988199a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:16:53.222 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:53.221+0000 7ff987400640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7ff988102790 0x7ff988199f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:16:53.222 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:53.221+0000 7ff987e00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff9881015a0 0x7ff988199a20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.118:40322/0 (socket says 172.21.15.118:40322) 2024-03-24T00:16:53.223 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:53.221+0000 7ff987400640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7ff988102790 0x7ff988199f60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.118:36142/0 (socket says 172.21.15.118:36142) 2024-03-24T00:16:53.223 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:53.221+0000 7ff987e00640 1 -- 172.21.15.118:0/3271318818 learned_addr learned my addr 172.21.15.118:0/3271318818 (peer_addr_for_me v2:172.21.15.118:0/0) 2024-03-24T00:16:53.223 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:53.222+0000 7ff987400640 1 -- 172.21.15.118:0/3271318818 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff9881015a0 msgr2=0x7ff988199a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:16:53.224 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:53.222+0000 7ff987400640 1 --2- 172.21.15.118:0/3271318818 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff9881015a0 0x7ff988199a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:16:53.224 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:53.222+0000 7ff987400640 1 -- 172.21.15.118:0/3271318818 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7ff988103150 msgr2=0x7ff9881a0fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:16:53.224 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:53.222+0000 7ff987400640 1 --2- 172.21.15.118:0/3271318818 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7ff988103150 0x7ff9881a0fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:16:53.224 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:53.222+0000 7ff987400640 1 -- 172.21.15.118:0/3271318818 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff9881a1640 con 0x7ff988102790 2024-03-24T00:16:53.224 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:53.222+0000 7ff987e00640 1 --2- 172.21.15.118:0/3271318818 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff9881015a0 0x7ff988199a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T00:16:53.224 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:53.222+0000 7ff987400640 1 --2- 172.21.15.118:0/3271318818 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7ff988102790 0x7ff988199f60 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7ff97400e9f0 tx=0x7ff97400eeb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:16:53.224 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:53.222+0000 7ff984c00640 1 -- 172.21.15.118:0/3271318818 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff97400cd80 con 0x7ff988102790 2024-03-24T00:16:53.224 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:53.222+0000 7ff984c00640 1 -- 172.21.15.118:0/3271318818 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7ff9740035f0 con 0x7ff988102790 2024-03-24T00:16:53.224 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:53.222+0000 7ff984c00640 1 -- 172.21.15.118:0/3271318818 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff974010690 con 0x7ff988102790 2024-03-24T00:16:53.224 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:53.222+0000 7ff98dc00640 1 -- 172.21.15.118:0/3271318818 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff9881a1930 con 0x7ff988102790 2024-03-24T00:16:53.224 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:53.222+0000 7ff98dc00640 1 -- 172.21.15.118:0/3271318818 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff9881a1e30 con 0x7ff988102790 2024-03-24T00:16:53.227 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:53.226+0000 7ff991f9c740 1 Processor -- start 2024-03-24T00:16:53.227 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:53.226+0000 7ff991f9c740 1 -- start start 2024-03-24T00:16:53.227 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:53.227+0000 7ff991f9c740 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5590c76e89f0 0x5590c76e8e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:16:53.227 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:53.227+0000 7ff991f9c740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5590c76e9bc0 0x5590c76ea040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:16:53.227 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:53.227+0000 7ff991f9c740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5590c76ea580 0x5590c76f0e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:16:53.228 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:53.227+0000 7ff991f9c740 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x5590c764d2c0 con 0x5590c76e89f0 2024-03-24T00:16:53.228 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:53.227+0000 7ff991f9c740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x5590c75f5c90 con 0x5590c76e9bc0 2024-03-24T00:16:53.228 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:53.227+0000 7ff991f9c740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x5590c7649360 con 0x5590c76ea580 2024-03-24T00:16:53.228 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:53.227+0000 7ff98d200640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5590c76e89f0 0x5590c76e8e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:16:53.228 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:53.228+0000 7ff98d200640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5590c76e89f0 0x5590c76e8e10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.118:3300/0 says I am v2:172.21.15.118:50356/0 (socket says 172.21.15.118:50356) 2024-03-24T00:16:53.228 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:53.228+0000 7ff98d200640 1 -- 172.21.15.118:0/544219036 learned_addr learned my addr 172.21.15.118:0/544219036 (peer_addr_for_me v2:172.21.15.118:0/0) 2024-03-24T00:16:53.228 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:53.228+0000 7ff987e00640 1 --2- 172.21.15.118:0/544219036 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5590c76e9bc0 0x5590c76ea040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:16:53.229 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:53.228+0000 7ff984c00640 1 -- 172.21.15.118:0/3271318818 <== mon.1 v2:172.21.15.193:3300/0 4 ==== mgrmap(e 19) v1 ==== 137342+0+0 (secure 0 0 0) 0x7ff974003b30 con 0x7ff988102790 2024-03-24T00:16:53.229 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:53.228+0000 7ff987e00640 1 -- 172.21.15.118:0/544219036 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5590c76ea580 msgr2=0x5590c76f0e10 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-03-24T00:16:53.229 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:53.228+0000 7ff987e00640 1 --2- 172.21.15.118:0/544219036 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5590c76ea580 0x5590c76f0e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:16:53.230 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:53.229+0000 7ff987e00640 1 -- 172.21.15.118:0/544219036 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5590c76e89f0 msgr2=0x5590c76e8e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:16:53.230 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:53.229+0000 7ff987e00640 1 --2- 172.21.15.118:0/544219036 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5590c76e89f0 0x5590c76e8e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:16:53.230 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:53.229+0000 7ff987e00640 1 -- 172.21.15.118:0/544219036 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5590c761a080 con 0x5590c76e9bc0 2024-03-24T00:16:53.230 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:53.229+0000 7ff98d200640 1 --2- 172.21.15.118:0/544219036 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5590c76e89f0 0x5590c76e8e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T00:16:53.230 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:53.229+0000 7ff987e00640 1 --2- 172.21.15.118:0/544219036 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5590c76e9bc0 0x5590c76ea040 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7ff97800bfd0 tx=0x7ff978010930 comp rx=0 tx=0).ready entity=mon.2 client_cookie=1ce33d1021ef778e server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:16:53.230 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:53.229+0000 7ff97d600640 1 -- 172.21.15.118:0/544219036 <== mon.2 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff978011760 con 0x5590c76e9bc0 2024-03-24T00:16:53.231 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:53.230+0000 7ff984c00640 1 --2- 172.21.15.118:0/3271318818 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7ff96c0a2910 0x7ff96c0a4dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:16:53.231 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:53.230+0000 7ff97d600640 1 -- 172.21.15.118:0/544219036 <== mon.2 v2:172.21.15.189:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7ff978011de0 con 0x5590c76e9bc0 2024-03-24T00:16:53.231 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:53.230+0000 7ff97d600640 1 -- 172.21.15.118:0/544219036 <== mon.2 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff978011760 con 0x5590c76e9bc0 2024-03-24T00:16:53.231 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:53.230+0000 7ff984c00640 1 -- 172.21.15.118:0/3271318818 <== mon.1 v2:172.21.15.193:3300/0 5 ==== osd_map(90..90 src has 1..90) v4 ==== 8277+0+0 (secure 0 0 0) 0x7ff9740cf250 con 0x7ff988102790 2024-03-24T00:16:53.231 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:53.230+0000 7ff98d200640 1 --2- 172.21.15.118:0/3271318818 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7ff96c0a2910 0x7ff96c0a4dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:16:53.231 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:53.231+0000 7ff991f9c740 1 -- 172.21.15.118:0/544219036 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5590c76e9bc0 msgr2=0x5590c76ea040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:16:53.232 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:53.231+0000 7ff98d200640 1 --2- 172.21.15.118:0/3271318818 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7ff96c0a2910 0x7ff96c0a4dd0 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7ff9800325f0 tx=0x7ff9800093a0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:16:53.232 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:53.231+0000 7ff991f9c740 1 --2- 172.21.15.118:0/544219036 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5590c76e9bc0 0x5590c76ea040 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7ff97800bfd0 tx=0x7ff978010930 comp rx=0 tx=0).stop 2024-03-24T00:16:53.232 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:53.231+0000 7ff991f9c740 1 -- 172.21.15.118:0/544219036 shutdown_connections 2024-03-24T00:16:53.232 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:53.231+0000 7ff991f9c740 1 --2- 172.21.15.118:0/544219036 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5590c76ea580 0x5590c76f0e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:16:53.232 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:53.231+0000 7ff991f9c740 1 --2- 172.21.15.118:0/544219036 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5590c76e9bc0 0x5590c76ea040 unknown :-1 s=CLOSED pgs=253 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:16:53.232 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:53.231+0000 7ff991f9c740 1 --2- 172.21.15.118:0/544219036 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5590c76e89f0 0x5590c76e8e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:16:53.232 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:53.231+0000 7ff991f9c740 1 -- 172.21.15.118:0/544219036 >> 172.21.15.118:0/544219036 conn(0x5590c76e3de0 msgr2=0x5590c76e41e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:16:53.234 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:53.231+0000 7ff991f9c740 1 -- 172.21.15.118:0/544219036 shutdown_connections 2024-03-24T00:16:53.234 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:53.231+0000 7ff991f9c740 1 -- 172.21.15.118:0/544219036 wait complete. 2024-03-24T00:16:53.234 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:53.232+0000 7ff991f9c740 1 -- start start 2024-03-24T00:16:53.235 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:53.232+0000 7ff991f9c740 1 Processor -- start 2024-03-24T00:16:53.235 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:53.232+0000 7ff991f9c740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5590c76e89f0 0x5590c76f2c00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:16:53.235 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:53.232+0000 7ff991f9c740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5590c76e9bc0 0x5590c76f3140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:16:53.235 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:53.232+0000 7ff991f9c740 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5590c76ea580 0x5590c76fa180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:16:53.235 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:53.232+0000 7ff991f9c740 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x5590c76f1590 con 0x5590c76ea580 2024-03-24T00:16:53.235 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:53.232+0000 7ff991f9c740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x5590c76090d0 con 0x5590c76e89f0 2024-03-24T00:16:53.235 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:53.232+0000 7ff991f9c740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x5590c76f1710 con 0x5590c76e9bc0 2024-03-24T00:16:53.235 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:53.233+0000 7ff991f9c740 10 client.0 _ll_register_callbacks cb 0x5590c760de80 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T00:16:53.235 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:53.233+0000 7ff987400640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5590c76e9bc0 0x5590c76f3140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:16:53.235 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:53.233+0000 7ff987400640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5590c76e9bc0 0x5590c76f3140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.118:36162/0 (socket says 172.21.15.118:36162) 2024-03-24T00:16:53.235 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:53.233+0000 7ff987400640 1 -- 172.21.15.118:0/2616883202 learned_addr learned my addr 172.21.15.118:0/2616883202 (peer_addr_for_me v2:172.21.15.118:0/0) 2024-03-24T00:16:53.235 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:53.233+0000 7ff987e00640 1 --2- 172.21.15.118:0/2616883202 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5590c76e89f0 0x5590c76f2c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:16:53.235 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:53.233+0000 7ff98d200640 1 --2- 172.21.15.118:0/2616883202 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5590c76ea580 0x5590c76fa180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:16:53.235 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:53.233+0000 7ff987400640 1 -- 172.21.15.118:0/2616883202 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5590c76e89f0 msgr2=0x5590c76f2c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:16:53.235 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:53.233+0000 7ff987400640 1 --2- 172.21.15.118:0/2616883202 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5590c76e89f0 0x5590c76f2c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:16:53.236 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:53.233+0000 7ff987400640 1 -- 172.21.15.118:0/2616883202 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5590c76ea580 msgr2=0x5590c76fa180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:16:53.236 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:53.233+0000 7ff987400640 1 --2- 172.21.15.118:0/2616883202 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5590c76ea580 0x5590c76fa180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:16:53.236 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:53.233+0000 7ff987400640 1 -- 172.21.15.118:0/2616883202 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5590c73613f0 con 0x5590c76e9bc0 2024-03-24T00:16:53.236 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:53.233+0000 7ff98d200640 1 --2- 172.21.15.118:0/2616883202 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5590c76ea580 0x5590c76fa180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-24T00:16:53.236 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:53.233+0000 7ff987400640 1 --2- 172.21.15.118:0/2616883202 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5590c76e9bc0 0x5590c76f3140 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7ff9740a3c70 tx=0x7ff97409fb50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c4e2e18f0304bac8 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:16:53.236 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:53.233+0000 7ff972000640 10 client.0 ms_handle_connect on v2:172.21.15.193:3300/0 2024-03-24T00:16:53.236 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:53.234+0000 7ff972000640 1 -- 172.21.15.118:0/2616883202 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff9740ccdc0 con 0x5590c76e9bc0 2024-03-24T00:16:53.236 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:53.234+0000 7ff972000640 1 -- 172.21.15.118:0/2616883202 <== mon.1 v2:172.21.15.193:3300/0 2 ==== osd_map(90..90 src has 1..90) v4 ==== 8277+0+0 (secure 0 0 0) 0x7ff97409ae50 con 0x5590c76e9bc0 2024-03-24T00:16:53.236 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:53.234+0000 7ff991f9c740 1 -- 172.21.15.118:0/2616883202 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7ff978011de0 con 0x5590c76e9bc0 2024-03-24T00:16:53.236 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:53.234+0000 7ff972000640 1 -- 172.21.15.118:0/2616883202 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) v2 ==== 24+0+0 (secure 0 0 0) 0x7ff9740ccf60 con 0x5590c76e9bc0 2024-03-24T00:16:53.236 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:53.234+0000 7ff991f9c740 10 client.25663 fetch_fsmap learned FSMap version 28 2024-03-24T00:16:53.236 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:53.234+0000 7ff991f9c740 1 -- 172.21.15.118:0/2616883202 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) v3 -- 0x5590c742b7d0 con 0x5590c76e9bc0 2024-03-24T00:16:53.236 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:53.234+0000 7ff972000640 1 -- 172.21.15.118:0/2616883202 <== mon.1 v2:172.21.15.193:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7ff978011de0 con 0x5590c76e9bc0 2024-03-24T00:16:53.238 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:53.234+0000 7ff972000640 1 -- 172.21.15.118:0/2616883202 <== mon.1 v2:172.21.15.193:3300/0 5 ==== fsmap(e 28) v1 ==== 7934+0+0 (secure 0 0 0) 0x7ff974020070 con 0x5590c76e9bc0 2024-03-24T00:16:53.238 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:53.234+0000 7ff991f9c740 10 client.25663 fetch_fsmap finished waiting for FSMap version 28 2024-03-24T00:16:53.238 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:53.234+0000 7ff991f9c740 10 client.25663 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.e 2024-03-24T00:16:53.238 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:53.234+0000 7ff991f9c740 20 client.25663 populate_metadata read hostname 'smithi118' 2024-03-24T00:16:53.238 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:53.234+0000 7ff991f9c740 1 --2- 172.21.15.118:0/2616883202 >> [v2:172.21.15.189:6834/599473205,v1:172.21.15.189:6835/599473205] conn(0x5590c76fe700 0x5590c7700b20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:16:53.238 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:53.234+0000 7ff991f9c740 4 client.25663 mds_command: new command op to 24488 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-24T00:16:53.238 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:53.234+0000 7ff991f9c740 1 -- 172.21.15.118:0/2616883202 --> [v2:172.21.15.189:6834/599473205,v1:172.21.15.189:6835/599473205] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x5590c765a800 con 0x5590c76fe700 2024-03-24T00:16:53.238 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:53.235+0000 7ff987e00640 1 --2- 172.21.15.118:0/2616883202 >> [v2:172.21.15.189:6834/599473205,v1:172.21.15.189:6835/599473205] conn(0x5590c76fe700 0x5590c7700b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:16:53.238 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:53.235+0000 7ff987e00640 1 --2- 172.21.15.118:0/2616883202 >> [v2:172.21.15.189:6834/599473205,v1:172.21.15.189:6835/599473205] conn(0x5590c76fe700 0x5590c7700b20 crc :-1 s=READY pgs=24 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=a44170a57791f604 server_cookie=dc37e1cbc0b61b59 in_seq=0 out_seq=0 2024-03-24T00:16:53.239 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:53.235+0000 7ff972000640 10 client.25663 ms_handle_connect on v2:172.21.15.189:6834/599473205 2024-03-24T00:16:53.239 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:53.236+0000 7ff972000640 1 -- 172.21.15.118:0/2616883202 <== mds.4 v2:172.21.15.189:6834/599473205 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x5590c765a800 con 0x5590c76fe700 2024-03-24T00:16:53.239 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:53.236+0000 7ff972000640 10 client.25663 handle_command_reply: tid=0 2024-03-24T00:16:53.239 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:53.236+0000 7ff991f9c740 1 client.25663 shutdown 2024-03-24T00:16:53.239 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:53.236+0000 7ff991f9c740 1 -- 172.21.15.118:0/2616883202 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5590c76e9bc0 msgr2=0x5590c76f3140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:16:53.239 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:53.236+0000 7ff991f9c740 1 --2- 172.21.15.118:0/2616883202 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5590c76e9bc0 0x5590c76f3140 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7ff9740a3c70 tx=0x7ff97409fb50 comp rx=0 tx=0).stop 2024-03-24T00:16:53.239 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:53.236+0000 7ff991f9c740 1 -- 172.21.15.118:0/2616883202 shutdown_connections 2024-03-24T00:16:53.239 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:53.236+0000 7ff991f9c740 1 --2- 172.21.15.118:0/2616883202 >> [v2:172.21.15.189:6834/599473205,v1:172.21.15.189:6835/599473205] conn(0x5590c76fe700 0x5590c7700b20 crc :-1 s=READY pgs=24 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:16:53.239 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:53.236+0000 7ff991f9c740 1 --2- 172.21.15.118:0/2616883202 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5590c76ea580 0x5590c76fa180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:16:53.239 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:53.236+0000 7ff991f9c740 1 --2- 172.21.15.118:0/2616883202 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5590c76e9bc0 0x5590c76f3140 unknown :-1 s=CLOSED pgs=242 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:16:53.239 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:53.236+0000 7ff991f9c740 1 --2- 172.21.15.118:0/2616883202 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5590c76e89f0 0x5590c76f2c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:16:53.239 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:53.236+0000 7ff991f9c740 1 -- 172.21.15.118:0/2616883202 >> 172.21.15.118:0/2616883202 conn(0x5590c76e3de0 msgr2=0x5590c76e3160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:16:53.239 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:53.236+0000 7ff972000640 0 client.25663 ms_handle_reset on v2:172.21.15.189:6834/599473205 2024-03-24T00:16:53.239 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:53.237+0000 7ff991f9c740 1 -- 172.21.15.118:0/2616883202 shutdown_connections 2024-03-24T00:16:53.239 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:53.237+0000 7ff991f9c740 1 -- 172.21.15.118:0/2616883202 wait complete. 2024-03-24T00:16:53.240 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:53.237+0000 7ff991f9c740 20 client.25663 trim_cache size 0 max 16384 2024-03-24T00:16:53.251 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:53.250+0000 7ff991f9c740 1 Processor -- start 2024-03-24T00:16:53.251 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:53.250+0000 7ff991f9c740 1 -- start start 2024-03-24T00:16:53.251 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:53.250+0000 7ff991f9c740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5590c76e89f0 0x5590c77053a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:16:53.251 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:53.250+0000 7ff991f9c740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5590c76e9bc0 0x5590c76fa140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:16:53.251 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:53.250+0000 7ff991f9c740 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5590c76ea580 0x5590c77058e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:16:53.251 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:53.250+0000 7ff991f9c740 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff9881aaa80 con 0x5590c76ea580 2024-03-24T00:16:53.251 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:53.250+0000 7ff991f9c740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff9881ac400 con 0x5590c76e9bc0 2024-03-24T00:16:53.251 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:53.250+0000 7ff991f9c740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff9881ab0e0 con 0x5590c76e89f0 2024-03-24T00:16:53.252 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:53.251+0000 7ff987e00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5590c76e89f0 0x5590c77053a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:16:53.252 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:53.251+0000 7ff987e00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5590c76e89f0 0x5590c77053a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.118:36168/0 (socket says 172.21.15.118:36168) 2024-03-24T00:16:53.252 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:53.251+0000 7ff987e00640 1 -- 172.21.15.118:0/3136074071 learned_addr learned my addr 172.21.15.118:0/3136074071 (peer_addr_for_me v2:172.21.15.118:0/0) 2024-03-24T00:16:53.252 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:53.251+0000 7ff987400640 1 --2- 172.21.15.118:0/3136074071 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5590c76e9bc0 0x5590c76fa140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:16:53.253 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:53.251+0000 7ff98d200640 1 --2- 172.21.15.118:0/3136074071 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5590c76ea580 0x5590c77058e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:16:53.253 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:53.251+0000 7ff987e00640 1 -- 172.21.15.118:0/3136074071 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5590c76e9bc0 msgr2=0x5590c76fa140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:16:53.253 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:53.251+0000 7ff987e00640 1 --2- 172.21.15.118:0/3136074071 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5590c76e9bc0 0x5590c76fa140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:16:53.253 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:53.251+0000 7ff987e00640 1 -- 172.21.15.118:0/3136074071 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5590c76ea580 msgr2=0x5590c77058e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:16:53.253 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:53.251+0000 7ff987e00640 1 --2- 172.21.15.118:0/3136074071 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5590c76ea580 0x5590c77058e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:16:53.253 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:53.251+0000 7ff987e00640 1 -- 172.21.15.118:0/3136074071 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5590c76e4850 con 0x5590c76e89f0 2024-03-24T00:16:53.253 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:53.251+0000 7ff987400640 1 --2- 172.21.15.118:0/3136074071 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5590c76e9bc0 0x5590c76fa140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T00:16:53.253 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:53.251+0000 7ff98d200640 1 --2- 172.21.15.118:0/3136074071 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5590c76ea580 0x5590c77058e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T00:16:53.253 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:53.251+0000 7ff987e00640 1 --2- 172.21.15.118:0/3136074071 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5590c76e89f0 0x5590c77053a0 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7ff978011fd0 tx=0x7ff97801ef70 comp rx=0 tx=0).ready entity=mon.1 client_cookie=9f0077ddb37baed0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:16:53.253 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:53.252+0000 7ff97cc00640 1 -- 172.21.15.118:0/3136074071 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff97802a250 con 0x5590c76e89f0 2024-03-24T00:16:53.253 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:53.252+0000 7ff97cc00640 1 -- 172.21.15.118:0/3136074071 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7ff978005550 con 0x5590c76e89f0 2024-03-24T00:16:53.253 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:53.252+0000 7ff97cc00640 1 -- 172.21.15.118:0/3136074071 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff978004660 con 0x5590c76e89f0 2024-03-24T00:16:53.255 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:53.252+0000 7ff991f9c740 1 -- 172.21.15.118:0/3136074071 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5590c76e89f0 msgr2=0x5590c77053a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:16:53.255 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:53.252+0000 7ff991f9c740 1 --2- 172.21.15.118:0/3136074071 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5590c76e89f0 0x5590c77053a0 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7ff978011fd0 tx=0x7ff97801ef70 comp rx=0 tx=0).stop 2024-03-24T00:16:53.255 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:53.252+0000 7ff991f9c740 1 -- 172.21.15.118:0/3136074071 shutdown_connections 2024-03-24T00:16:53.255 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:53.252+0000 7ff991f9c740 1 --2- 172.21.15.118:0/3136074071 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5590c76ea580 0x5590c77058e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:16:53.255 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:53.252+0000 7ff991f9c740 1 --2- 172.21.15.118:0/3136074071 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5590c76e9bc0 0x5590c76fa140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:16:53.255 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:53.252+0000 7ff991f9c740 1 --2- 172.21.15.118:0/3136074071 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5590c76e89f0 0x5590c77053a0 unknown :-1 s=CLOSED pgs=243 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:16:53.255 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:53.252+0000 7ff991f9c740 1 -- 172.21.15.118:0/3136074071 >> 172.21.15.118:0/3136074071 conn(0x5590c76e3de0 msgr2=0x5590c76e78d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:16:53.255 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:53.253+0000 7ff991f9c740 1 -- 172.21.15.118:0/3136074071 shutdown_connections 2024-03-24T00:16:53.255 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:53.253+0000 7ff991f9c740 1 -- 172.21.15.118:0/3136074071 wait complete. 2024-03-24T00:16:53.255 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:53.253+0000 7ff991f9c740 1 -- start start 2024-03-24T00:16:53.255 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:53.253+0000 7ff991f9c740 1 Processor -- start 2024-03-24T00:16:53.256 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:53.254+0000 7ff991f9c740 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5590c76e9bc0 0x5590c76f5b50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:16:53.256 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:53.254+0000 7ff991f9c740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5590c76ea580 0x5590c76f6090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:16:53.256 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:53.254+0000 7ff991f9c740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5590c76f9010 0x5590c76f9490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:16:53.256 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:53.254+0000 7ff991f9c740 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff9881abda0 con 0x5590c76e9bc0 2024-03-24T00:16:53.256 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:53.254+0000 7ff991f9c740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff9881ab740 con 0x5590c76ea580 2024-03-24T00:16:53.256 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:53.254+0000 7ff991f9c740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff9881a8c40 con 0x5590c76f9010 2024-03-24T00:16:53.256 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:53.254+0000 7ff991f9c740 10 client.0 _ll_register_callbacks cb 0x7ff97409c190 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T00:16:53.256 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:53.254+0000 7ff987e00640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5590c76e9bc0 0x5590c76f5b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:16:53.256 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:53.254+0000 7ff987e00640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5590c76e9bc0 0x5590c76f5b50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.118:3300/0 says I am v2:172.21.15.118:50388/0 (socket says 172.21.15.118:50388) 2024-03-24T00:16:53.256 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:53.254+0000 7ff987e00640 1 -- 172.21.15.118:0/2817096061 learned_addr learned my addr 172.21.15.118:0/2817096061 (peer_addr_for_me v2:172.21.15.118:0/0) 2024-03-24T00:16:53.257 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:53.255+0000 7ff98d200640 1 --2- 172.21.15.118:0/2817096061 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5590c76f9010 0x5590c76f9490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:16:53.257 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:53.255+0000 7ff987e00640 1 -- 172.21.15.118:0/2817096061 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5590c76f9010 msgr2=0x5590c76f9490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:16:53.257 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:53.255+0000 7ff987e00640 1 --2- 172.21.15.118:0/2817096061 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5590c76f9010 0x5590c76f9490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:16:53.257 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:53.255+0000 7ff987e00640 1 -- 172.21.15.118:0/2817096061 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5590c76ea580 msgr2=0x5590c76f6090 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-03-24T00:16:53.257 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:53.255+0000 7ff987e00640 1 --2- 172.21.15.118:0/2817096061 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5590c76ea580 0x5590c76f6090 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:16:53.258 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:53.255+0000 7ff987e00640 1 -- 172.21.15.118:0/2817096061 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5590c76f99d0 con 0x5590c76e9bc0 2024-03-24T00:16:53.258 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:53.255+0000 7ff987e00640 1 --2- 172.21.15.118:0/2817096061 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5590c76e9bc0 0x5590c76f5b50 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7ff97802ab80 tx=0x7ff978005c20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3a8d4d2ebb581bf2 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:16:53.258 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:53.255+0000 7ff972000640 10 client.0 ms_handle_connect on v2:172.21.15.118:3300/0 2024-03-24T00:16:53.258 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:53.256+0000 7ff972000640 1 -- 172.21.15.118:0/2817096061 <== mon.0 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff97802a250 con 0x5590c76e9bc0 2024-03-24T00:16:53.258 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:53.256+0000 7ff972000640 1 -- 172.21.15.118:0/2817096061 <== mon.0 v2:172.21.15.118:3300/0 2 ==== osd_map(90..90 src has 1..90) v4 ==== 8277+0+0 (secure 0 0 0) 0x7ff978011560 con 0x5590c76e9bc0 2024-03-24T00:16:53.258 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:53.256+0000 7ff991f9c740 1 -- 172.21.15.118:0/2817096061 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7ff978005550 con 0x5590c76e9bc0 2024-03-24T00:16:53.258 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:53.256+0000 7ff972000640 1 -- 172.21.15.118:0/2817096061 <== mon.0 v2:172.21.15.118:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) v2 ==== 24+0+0 (secure 0 0 0) 0x7ff978011760 con 0x5590c76e9bc0 2024-03-24T00:16:53.258 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:53.256+0000 7ff991f9c740 10 client.15552 fetch_fsmap learned FSMap version 28 2024-03-24T00:16:53.258 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:53.256+0000 7ff991f9c740 1 -- 172.21.15.118:0/2817096061 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) v3 -- 0x5590c7762dd0 con 0x5590c76e9bc0 2024-03-24T00:16:53.258 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:53.256+0000 7ff972000640 1 -- 172.21.15.118:0/2817096061 <== mon.0 v2:172.21.15.118:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7ff978005550 con 0x5590c76e9bc0 2024-03-24T00:16:53.259 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:53.257+0000 7ff972000640 1 -- 172.21.15.118:0/2817096061 <== mon.0 v2:172.21.15.118:3300/0 5 ==== fsmap(e 28) v1 ==== 7934+0+0 (secure 0 0 0) 0x7ff978045070 con 0x5590c76e9bc0 2024-03-24T00:16:53.259 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:53.257+0000 7ff991f9c740 10 client.15552 fetch_fsmap finished waiting for FSMap version 28 2024-03-24T00:16:53.259 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:53.257+0000 7ff991f9c740 10 client.15552 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.e 2024-03-24T00:16:53.259 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:53.257+0000 7ff991f9c740 20 client.15552 populate_metadata read hostname 'smithi118' 2024-03-24T00:16:53.259 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:53.257+0000 7ff991f9c740 1 --2- 172.21.15.118:0/2817096061 >> [v2:172.21.15.189:6834/599473205,v1:172.21.15.189:6835/599473205] conn(0x5590c7763580 0x5590c77659a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:16:53.259 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:53.257+0000 7ff991f9c740 4 client.15552 mds_command: new command op to 24488 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-24T00:16:53.259 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:53.257+0000 7ff991f9c740 1 -- 172.21.15.118:0/2817096061 --> [v2:172.21.15.189:6834/599473205,v1:172.21.15.189:6835/599473205] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x5590c7654b50 con 0x5590c7763580 2024-03-24T00:16:53.259 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:53.258+0000 7ff987400640 1 --2- 172.21.15.118:0/2817096061 >> [v2:172.21.15.189:6834/599473205,v1:172.21.15.189:6835/599473205] conn(0x5590c7763580 0x5590c77659a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:16:53.260 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:53.259+0000 7ff972000640 10 client.15552 ms_handle_connect on v2:172.21.15.189:6834/599473205 2024-03-24T00:16:53.260 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:53.259+0000 7ff987400640 1 --2- 172.21.15.118:0/2817096061 >> [v2:172.21.15.189:6834/599473205,v1:172.21.15.189:6835/599473205] conn(0x5590c7763580 0x5590c77659a0 crc :-1 s=READY pgs=25 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=e97155d9f9b5209b server_cookie=e9e3c0c5277e49fe in_seq=0 out_seq=0 2024-03-24T00:16:53.260 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:53.259+0000 7ff972000640 1 -- 172.21.15.118:0/2817096061 <== mds.4 v2:172.21.15.189:6834/599473205 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x5590c7654b50 con 0x5590c7763580 2024-03-24T00:16:53.260 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:53.259+0000 7ff972000640 10 client.15552 handle_command_reply: tid=0 2024-03-24T00:16:53.261 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:53.259+0000 7ff991f9c740 1 client.15552 shutdown 2024-03-24T00:16:53.261 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:53.259+0000 7ff991f9c740 1 -- 172.21.15.118:0/2817096061 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5590c76e9bc0 msgr2=0x5590c76f5b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:16:53.261 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:53.259+0000 7ff991f9c740 1 --2- 172.21.15.118:0/2817096061 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5590c76e9bc0 0x5590c76f5b50 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7ff97802ab80 tx=0x7ff978005c20 comp rx=0 tx=0).stop 2024-03-24T00:16:53.261 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:53.260+0000 7ff991f9c740 1 -- 172.21.15.118:0/2817096061 shutdown_connections 2024-03-24T00:16:53.261 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:53.260+0000 7ff991f9c740 1 --2- 172.21.15.118:0/2817096061 >> [v2:172.21.15.189:6834/599473205,v1:172.21.15.189:6835/599473205] conn(0x5590c7763580 0x5590c77659a0 crc :-1 s=READY pgs=25 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:16:53.261 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:53.260+0000 7ff991f9c740 1 --2- 172.21.15.118:0/2817096061 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5590c76f9010 0x5590c76f9490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:16:53.261 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:53.260+0000 7ff991f9c740 1 --2- 172.21.15.118:0/2817096061 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5590c76ea580 0x5590c76f6090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:16:53.261 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:53.260+0000 7ff991f9c740 1 --2- 172.21.15.118:0/2817096061 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5590c76e9bc0 0x5590c76f5b50 unknown :-1 s=CLOSED pgs=183 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:16:53.261 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:53.260+0000 7ff991f9c740 1 -- 172.21.15.118:0/2817096061 >> 172.21.15.118:0/2817096061 conn(0x5590c76e3de0 msgr2=0x5590c76ecdf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:16:53.261 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:53.260+0000 7ff972000640 0 client.15552 ms_handle_reset on v2:172.21.15.189:6834/599473205 2024-03-24T00:16:53.261 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:53.260+0000 7ff991f9c740 1 -- 172.21.15.118:0/2817096061 shutdown_connections 2024-03-24T00:16:53.261 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:53.260+0000 7ff991f9c740 1 -- 172.21.15.118:0/2817096061 wait complete. 2024-03-24T00:16:53.261 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:53.260+0000 7ff991f9c740 20 client.15552 trim_cache size 0 max 16384 2024-03-24T00:16:53.262 INFO:teuthology.orchestra.run.smithi118.stdout:[] 2024-03-24T00:16:53.262 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:53.261+0000 7ff98dc00640 1 -- 172.21.15.118:0/3271318818 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7ff96c0a2910 msgr2=0x7ff96c0a4dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:16:53.263 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:53.261+0000 7ff98dc00640 1 --2- 172.21.15.118:0/3271318818 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7ff96c0a2910 0x7ff96c0a4dd0 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7ff9800325f0 tx=0x7ff9800093a0 comp rx=0 tx=0).stop 2024-03-24T00:16:53.263 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:53.261+0000 7ff98dc00640 1 -- 172.21.15.118:0/3271318818 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7ff988102790 msgr2=0x7ff988199f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:16:53.263 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:53.261+0000 7ff98dc00640 1 --2- 172.21.15.118:0/3271318818 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7ff988102790 0x7ff988199f60 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7ff97400e9f0 tx=0x7ff97400eeb0 comp rx=0 tx=0).stop 2024-03-24T00:16:53.263 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:53.261+0000 7ff98dc00640 1 -- 172.21.15.118:0/3271318818 shutdown_connections 2024-03-24T00:16:53.263 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:53.261+0000 7ff98dc00640 1 --2- 172.21.15.118:0/3271318818 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7ff96c0a2910 0x7ff96c0a4dd0 unknown :-1 s=CLOSED pgs=286 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:16:53.263 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:53.261+0000 7ff98dc00640 1 --2- 172.21.15.118:0/3271318818 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7ff988103150 0x7ff9881a0fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:16:53.263 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:53.261+0000 7ff98dc00640 1 --2- 172.21.15.118:0/3271318818 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7ff988102790 0x7ff988199f60 unknown :-1 s=CLOSED pgs=241 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:16:53.263 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:53.261+0000 7ff98dc00640 1 --2- 172.21.15.118:0/3271318818 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff9881015a0 0x7ff988199a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:16:53.263 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:53.261+0000 7ff98dc00640 1 -- 172.21.15.118:0/3271318818 >> 172.21.15.118:0/3271318818 conn(0x7ff9880fcb10 msgr2=0x7ff9880738d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:16:53.263 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:53.261+0000 7ff98dc00640 1 -- 172.21.15.118:0/3271318818 shutdown_connections 2024-03-24T00:16:53.263 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:53.261+0000 7ff98dc00640 1 -- 172.21.15.118:0/3271318818 wait complete. 2024-03-24T00:16:53.278 INFO:teuthology.orchestra.run:Running command with timeout 120 2024-03-24T00:16:53.278 DEBUG:teuthology.orchestra.run.smithi118:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:3 damage ls 2024-03-24T00:16:53.479 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:53.479+0000 7f9cf7e00640 1 -- 172.21.15.118:0/254261693 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9cf81011a0 msgr2=0x7f9cf8101620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:16:53.480 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:53.479+0000 7f9cf7e00640 1 --2- 172.21.15.118:0/254261693 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9cf81011a0 0x7f9cf8101620 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7f9cf0009ca0 tx=0x7f9cf002c1b0 comp rx=0 tx=0).stop 2024-03-24T00:16:53.480 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:53.479+0000 7f9cf7e00640 1 -- 172.21.15.118:0/254261693 shutdown_connections 2024-03-24T00:16:53.480 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:53.479+0000 7f9cf7e00640 1 --2- 172.21.15.118:0/254261693 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9cf81011a0 0x7f9cf8101620 unknown :-1 s=CLOSED pgs=254 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:16:53.480 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:53.479+0000 7f9cf7e00640 1 --2- 172.21.15.118:0/254261693 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f9cf8100800 0x7f9cf8100c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:16:53.480 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:53.479+0000 7f9cf7e00640 1 --2- 172.21.15.118:0/254261693 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f9cf80ffea0 0x7f9cf81002c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:16:53.480 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:53.479+0000 7f9cf7e00640 1 -- 172.21.15.118:0/254261693 >> 172.21.15.118:0/254261693 conn(0x7f9cf80fba20 msgr2=0x7f9cf80fde60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:16:53.480 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:53.479+0000 7f9cf7e00640 1 -- 172.21.15.118:0/254261693 shutdown_connections 2024-03-24T00:16:53.480 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:53.479+0000 7f9cf7e00640 1 -- 172.21.15.118:0/254261693 wait complete. 2024-03-24T00:16:53.481 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:53.480+0000 7f9cf7e00640 1 Processor -- start 2024-03-24T00:16:53.481 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:53.481+0000 7f9cf7e00640 1 -- start start 2024-03-24T00:16:53.482 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:53.481+0000 7f9cf7e00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f9cf80ffea0 0x7f9cf8199a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:16:53.482 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:53.481+0000 7f9cf7e00640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f9cf8100800 0x7f9cf8199f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:16:53.482 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:53.481+0000 7f9cf7e00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9cf81011a0 0x7f9cf81a1000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:16:53.482 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:53.481+0000 7f9cf7e00640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9cf810f430 con 0x7f9cf8100800 2024-03-24T00:16:53.482 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:53.481+0000 7f9cf7e00640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9cf810f2b0 con 0x7f9cf81011a0 2024-03-24T00:16:53.482 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:53.481+0000 7f9cf7e00640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9cf810f5b0 con 0x7f9cf80ffea0 2024-03-24T00:16:53.482 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:53.481+0000 7f9cf6a00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f9cf80ffea0 0x7f9cf8199a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:16:53.482 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:53.481+0000 7f9cf6a00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f9cf80ffea0 0x7f9cf8199a50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.118:36190/0 (socket says 172.21.15.118:36190) 2024-03-24T00:16:53.483 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:53.481+0000 7f9cf6a00640 1 -- 172.21.15.118:0/1466931078 learned_addr learned my addr 172.21.15.118:0/1466931078 (peer_addr_for_me v2:172.21.15.118:0/0) 2024-03-24T00:16:53.483 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:53.481+0000 7f9cf7400640 1 --2- 172.21.15.118:0/1466931078 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9cf81011a0 0x7f9cf81a1000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:16:53.483 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:53.482+0000 7f9cf6000640 1 --2- 172.21.15.118:0/1466931078 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f9cf8100800 0x7f9cf8199f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:16:53.484 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:53.482+0000 7f9cf6a00640 1 -- 172.21.15.118:0/1466931078 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9cf81011a0 msgr2=0x7f9cf81a1000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:16:53.484 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:53.482+0000 7f9cf6a00640 1 --2- 172.21.15.118:0/1466931078 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9cf81011a0 0x7f9cf81a1000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:16:53.484 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:53.482+0000 7f9cf6a00640 1 -- 172.21.15.118:0/1466931078 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f9cf8100800 msgr2=0x7f9cf8199f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:16:53.484 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:53.482+0000 7f9cf6a00640 1 --2- 172.21.15.118:0/1466931078 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f9cf8100800 0x7f9cf8199f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:16:53.484 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:53.482+0000 7f9cf6a00640 1 -- 172.21.15.118:0/1466931078 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9cf81a1670 con 0x7f9cf80ffea0 2024-03-24T00:16:53.484 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:53.482+0000 7f9cf6000640 1 --2- 172.21.15.118:0/1466931078 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f9cf8100800 0x7f9cf8199f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-24T00:16:53.484 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:53.482+0000 7f9cf6a00640 1 --2- 172.21.15.118:0/1466931078 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f9cf80ffea0 0x7f9cf8199a50 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7f9ce800e9f0 tx=0x7f9ce800eeb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:16:53.484 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:53.482+0000 7f9cef400640 1 -- 172.21.15.118:0/1466931078 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9ce800cd80 con 0x7f9cf80ffea0 2024-03-24T00:16:53.484 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:53.482+0000 7f9cef400640 1 -- 172.21.15.118:0/1466931078 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f9ce8010910 con 0x7f9cf80ffea0 2024-03-24T00:16:53.484 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:53.482+0000 7f9cef400640 1 -- 172.21.15.118:0/1466931078 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9ce8018990 con 0x7f9cf80ffea0 2024-03-24T00:16:53.484 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:53.483+0000 7f9cf7e00640 1 -- 172.21.15.118:0/1466931078 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9cf81a1960 con 0x7f9cf80ffea0 2024-03-24T00:16:53.484 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:53.483+0000 7f9cf7e00640 1 -- 172.21.15.118:0/1466931078 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9cf81a1e30 con 0x7f9cf80ffea0 2024-03-24T00:16:53.487 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:53.486+0000 7f9d003db740 1 Processor -- start 2024-03-24T00:16:53.487 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:53.487+0000 7f9d003db740 1 -- start start 2024-03-24T00:16:53.488 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:53.487+0000 7f9d003db740 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55b23dfa69f0 0x55b23dfa6e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:16:53.488 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:53.487+0000 7f9d003db740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55b23dfa7bc0 0x55b23dfa8040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:16:53.488 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:53.487+0000 7f9d003db740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55b23dfa8580 0x55b23dfaee10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:16:53.488 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:53.487+0000 7f9d003db740 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b23df0b2c0 con 0x55b23dfa69f0 2024-03-24T00:16:53.488 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:53.487+0000 7f9d003db740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b23deb3c90 con 0x55b23dfa8580 2024-03-24T00:16:53.488 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:53.487+0000 7f9d003db740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b23df07360 con 0x55b23dfa7bc0 2024-03-24T00:16:53.488 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:53.487+0000 7f9cf7400640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55b23dfa69f0 0x55b23dfa6e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:16:53.489 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:53.487+0000 7f9cf7400640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55b23dfa69f0 0x55b23dfa6e10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.118:3300/0 says I am v2:172.21.15.118:50414/0 (socket says 172.21.15.118:50414) 2024-03-24T00:16:53.489 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:53.488+0000 7f9cf7400640 1 -- 172.21.15.118:0/100058426 learned_addr learned my addr 172.21.15.118:0/100058426 (peer_addr_for_me v2:172.21.15.118:0/0) 2024-03-24T00:16:53.489 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:53.488+0000 7f9cf6000640 1 --2- 172.21.15.118:0/100058426 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55b23dfa8580 0x55b23dfaee10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:16:53.489 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:53.488+0000 7f9cf7400640 1 -- 172.21.15.118:0/100058426 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55b23dfa7bc0 msgr2=0x55b23dfa8040 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-24T00:16:53.489 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:53.488+0000 7f9cf7400640 1 --2- 172.21.15.118:0/100058426 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55b23dfa7bc0 0x55b23dfa8040 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:16:53.489 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:53.488+0000 7f9cf7400640 1 -- 172.21.15.118:0/100058426 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55b23dfa8580 msgr2=0x55b23dfaee10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:16:53.489 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:53.488+0000 7f9cf7400640 1 --2- 172.21.15.118:0/100058426 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55b23dfa8580 0x55b23dfaee10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:16:53.489 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:53.488+0000 7f9cf7400640 1 -- 172.21.15.118:0/100058426 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55b23ded8080 con 0x55b23dfa69f0 2024-03-24T00:16:53.490 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:53.490+0000 7f9cef400640 1 -- 172.21.15.118:0/1466931078 <== mon.1 v2:172.21.15.193:3300/0 4 ==== mgrmap(e 19) v1 ==== 137342+0+0 (secure 0 0 0) 0x7f9ce8010ac0 con 0x7f9cf80ffea0 2024-03-24T00:16:53.491 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:53.490+0000 7f9cf7400640 1 --2- 172.21.15.118:0/100058426 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55b23dfa69f0 0x55b23dfa6e10 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7f9cf002c7c0 tx=0x7f9cf0005f80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1295e06011245077 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:16:53.491 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:53.491+0000 7f9ce3400640 1 -- 172.21.15.118:0/100058426 <== mon.0 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9cf0043070 con 0x55b23dfa69f0 2024-03-24T00:16:53.491 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:53.491+0000 7f9ce3400640 1 -- 172.21.15.118:0/100058426 <== mon.0 v2:172.21.15.118:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f9cf0003e50 con 0x55b23dfa69f0 2024-03-24T00:16:53.492 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:53.491+0000 7f9ce3400640 1 -- 172.21.15.118:0/100058426 <== mon.0 v2:172.21.15.118:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9cf0039040 con 0x55b23dfa69f0 2024-03-24T00:16:53.493 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:53.492+0000 7f9cef400640 1 --2- 172.21.15.118:0/1466931078 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7f9cdc0a2890 0x7f9cdc0a4d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:16:53.493 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:53.492+0000 7f9cef400640 1 -- 172.21.15.118:0/1466931078 <== mon.1 v2:172.21.15.193:3300/0 5 ==== osd_map(90..90 src has 1..90) v4 ==== 8277+0+0 (secure 0 0 0) 0x7f9ce80cf170 con 0x7f9cf80ffea0 2024-03-24T00:16:53.493 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:53.492+0000 7f9d003db740 1 -- 172.21.15.118:0/100058426 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55b23dfa69f0 msgr2=0x55b23dfa6e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:16:53.493 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:53.492+0000 7f9d003db740 1 --2- 172.21.15.118:0/100058426 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55b23dfa69f0 0x55b23dfa6e10 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7f9cf002c7c0 tx=0x7f9cf0005f80 comp rx=0 tx=0).stop 2024-03-24T00:16:53.493 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:53.492+0000 7f9cf6a00640 1 --2- 172.21.15.118:0/1466931078 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7f9cdc0a2890 0x7f9cdc0a4d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:16:53.494 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:53.493+0000 7f9d003db740 1 -- 172.21.15.118:0/100058426 shutdown_connections 2024-03-24T00:16:53.494 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:53.493+0000 7f9d003db740 1 --2- 172.21.15.118:0/100058426 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55b23dfa8580 0x55b23dfaee10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:16:53.494 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:53.493+0000 7f9d003db740 1 --2- 172.21.15.118:0/100058426 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55b23dfa7bc0 0x55b23dfa8040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:16:53.494 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:53.493+0000 7f9d003db740 1 --2- 172.21.15.118:0/100058426 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55b23dfa69f0 0x55b23dfa6e10 unknown :-1 s=CLOSED pgs=184 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:16:53.494 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:53.493+0000 7f9d003db740 1 -- 172.21.15.118:0/100058426 >> 172.21.15.118:0/100058426 conn(0x55b23dfa1de0 msgr2=0x55b23dfa21e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:16:53.494 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:53.493+0000 7f9d003db740 1 -- 172.21.15.118:0/100058426 shutdown_connections 2024-03-24T00:16:53.494 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:53.493+0000 7f9d003db740 1 -- 172.21.15.118:0/100058426 wait complete. 2024-03-24T00:16:53.494 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:53.493+0000 7f9d003db740 1 -- start start 2024-03-24T00:16:53.495 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:53.493+0000 7f9cf6a00640 1 --2- 172.21.15.118:0/1466931078 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7f9cdc0a2890 0x7f9cdc0a4d50 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x7f9ce8098350 tx=0x7f9ce800cf20 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:16:53.495 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:53.494+0000 7f9d003db740 1 Processor -- start 2024-03-24T00:16:53.495 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:53.494+0000 7f9d003db740 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55b23dfa69f0 0x55b23dfb0c70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:16:53.495 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:53.494+0000 7f9d003db740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55b23dfa7bc0 0x55b23dfb11b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:16:53.495 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:53.494+0000 7f9d003db740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55b23dfa8580 0x55b23dfb81f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:16:53.495 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:53.494+0000 7f9d003db740 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b23dfaf530 con 0x55b23dfa69f0 2024-03-24T00:16:53.495 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:53.494+0000 7f9d003db740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b23dec70d0 con 0x55b23dfa7bc0 2024-03-24T00:16:53.495 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:53.494+0000 7f9d003db740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b23dfaf6b0 con 0x55b23dfa8580 2024-03-24T00:16:53.495 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:53.495+0000 7f9d003db740 10 client.0 _ll_register_callbacks cb 0x55b23decbe80 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T00:16:53.497 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:53.495+0000 7f9cf6000640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55b23dfa7bc0 0x55b23dfb11b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:16:53.497 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:53.496+0000 7f9cf7400640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55b23dfa69f0 0x55b23dfb0c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:16:53.497 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:53.496+0000 7f9cf7400640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55b23dfa69f0 0x55b23dfb0c70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.118:3300/0 says I am v2:172.21.15.118:50430/0 (socket says 172.21.15.118:50430) 2024-03-24T00:16:53.497 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:53.496+0000 7f9cf7400640 1 -- 172.21.15.118:0/2991849956 learned_addr learned my addr 172.21.15.118:0/2991849956 (peer_addr_for_me v2:172.21.15.118:0/0) 2024-03-24T00:16:53.498 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:53.496+0000 7f9cf7400640 1 --2- 172.21.15.118:0/2991849956 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55b23dfa8580 0x55b23dfb81f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:16:53.498 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:53.497+0000 7f9cf6000640 1 -- 172.21.15.118:0/2991849956 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55b23dfa8580 msgr2=0x55b23dfb81f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:16:53.498 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:53.497+0000 7f9cf6000640 1 --2- 172.21.15.118:0/2991849956 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55b23dfa8580 0x55b23dfb81f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:16:53.498 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:53.497+0000 7f9cf6000640 1 -- 172.21.15.118:0/2991849956 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55b23dfa69f0 msgr2=0x55b23dfb0c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:16:53.498 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:53.497+0000 7f9cf6000640 1 --2- 172.21.15.118:0/2991849956 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55b23dfa69f0 0x55b23dfb0c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:16:53.498 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:53.497+0000 7f9cf6000640 1 -- 172.21.15.118:0/2991849956 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55b23dc1f3f0 con 0x55b23dfa7bc0 2024-03-24T00:16:53.499 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:53.498+0000 7f9cf7400640 1 --2- 172.21.15.118:0/2991849956 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55b23dfa69f0 0x55b23dfb0c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T00:16:53.499 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:53.498+0000 7f9cf6000640 1 --2- 172.21.15.118:0/2991849956 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55b23dfa7bc0 0x55b23dfb11b0 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7f9cf819af60 tx=0x7f9ce4012b00 comp rx=0 tx=0).ready entity=mon.2 client_cookie=5c00f55a95d5e247 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:16:53.499 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:53.498+0000 7f9cd7e00640 10 client.0 ms_handle_connect on v2:172.21.15.189:3300/0 2024-03-24T00:16:53.499 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:53.498+0000 7f9cd7e00640 1 -- 172.21.15.118:0/2991849956 <== mon.2 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9ce4004460 con 0x55b23dfa7bc0 2024-03-24T00:16:53.500 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:53.499+0000 7f9d003db740 1 -- 172.21.15.118:0/2991849956 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f9cf0003e50 con 0x55b23dfa7bc0 2024-03-24T00:16:53.500 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:53.499+0000 7f9cd7e00640 1 -- 172.21.15.118:0/2991849956 <== mon.2 v2:172.21.15.189:3300/0 2 ==== osd_map(90..90 src has 1..90) v4 ==== 8277+0+0 (secure 0 0 0) 0x7f9ce4005020 con 0x55b23dfa7bc0 2024-03-24T00:16:53.500 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:53.499+0000 7f9cd7e00640 1 -- 172.21.15.118:0/2991849956 <== mon.2 v2:172.21.15.189:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) v2 ==== 24+0+0 (secure 0 0 0) 0x7f9ce4005220 con 0x55b23dfa7bc0 2024-03-24T00:16:53.500 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:53.499+0000 7f9d003db740 10 client.25676 fetch_fsmap learned FSMap version 28 2024-03-24T00:16:53.500 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:53.499+0000 7f9d003db740 1 -- 172.21.15.118:0/2991849956 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) v3 -- 0x55b23dce97d0 con 0x55b23dfa7bc0 2024-03-24T00:16:53.500 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:53.499+0000 7f9cd7e00640 1 -- 172.21.15.118:0/2991849956 <== mon.2 v2:172.21.15.189:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f9cf0003e50 con 0x55b23dfa7bc0 2024-03-24T00:16:53.501 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:53.500+0000 7f9cd7e00640 1 -- 172.21.15.118:0/2991849956 <== mon.2 v2:172.21.15.189:3300/0 5 ==== fsmap(e 28) v1 ==== 7934+0+0 (secure 0 0 0) 0x7f9ce4019030 con 0x55b23dfa7bc0 2024-03-24T00:16:53.501 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:53.500+0000 7f9d003db740 10 client.25676 fetch_fsmap finished waiting for FSMap version 28 2024-03-24T00:16:53.501 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:53.500+0000 7f9d003db740 10 client.25676 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.f 2024-03-24T00:16:53.501 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:53.500+0000 7f9d003db740 20 client.25676 populate_metadata read hostname 'smithi118' 2024-03-24T00:16:53.501 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:53.500+0000 7f9d003db740 1 --2- 172.21.15.118:0/2991849956 >> [v2:172.21.15.193:6834/2055310541,v1:172.21.15.193:6835/2055310541] conn(0x55b23dfbc680 0x55b23dfbeae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:16:53.501 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:53.500+0000 7f9d003db740 4 client.25676 mds_command: new command op to 24499 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-24T00:16:53.501 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:53.500+0000 7f9d003db740 1 -- 172.21.15.118:0/2991849956 --> [v2:172.21.15.193:6834/2055310541,v1:172.21.15.193:6835/2055310541] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55b23df18800 con 0x55b23dfbc680 2024-03-24T00:16:53.501 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:53.500+0000 7f9cf7400640 1 --2- 172.21.15.118:0/2991849956 >> [v2:172.21.15.193:6834/2055310541,v1:172.21.15.193:6835/2055310541] conn(0x55b23dfbc680 0x55b23dfbeae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:16:53.501 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:53.501+0000 7f9cf7400640 1 --2- 172.21.15.118:0/2991849956 >> [v2:172.21.15.193:6834/2055310541,v1:172.21.15.193:6835/2055310541] conn(0x55b23dfbc680 0x55b23dfbeae0 crc :-1 s=READY pgs=26 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=fa7e403c83615a11 server_cookie=d015e8c45c753c3c in_seq=0 out_seq=0 2024-03-24T00:16:53.501 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:53.501+0000 7f9cd7e00640 10 client.25676 ms_handle_connect on v2:172.21.15.193:6834/2055310541 2024-03-24T00:16:53.502 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:53.502+0000 7f9cd7e00640 1 -- 172.21.15.118:0/2991849956 <== mds.3 v2:172.21.15.193:6834/2055310541 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x55b23df18800 con 0x55b23dfbc680 2024-03-24T00:16:53.502 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:53.502+0000 7f9cd7e00640 10 client.25676 handle_command_reply: tid=0 2024-03-24T00:16:53.503 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:53.502+0000 7f9d003db740 1 client.25676 shutdown 2024-03-24T00:16:53.503 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:53.502+0000 7f9d003db740 1 -- 172.21.15.118:0/2991849956 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55b23dfa7bc0 msgr2=0x55b23dfb11b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:16:53.503 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:53.502+0000 7f9d003db740 1 --2- 172.21.15.118:0/2991849956 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55b23dfa7bc0 0x55b23dfb11b0 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7f9cf819af60 tx=0x7f9ce4012b00 comp rx=0 tx=0).stop 2024-03-24T00:16:53.503 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:53.502+0000 7f9d003db740 1 -- 172.21.15.118:0/2991849956 shutdown_connections 2024-03-24T00:16:53.503 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:53.502+0000 7f9d003db740 1 --2- 172.21.15.118:0/2991849956 >> [v2:172.21.15.193:6834/2055310541,v1:172.21.15.193:6835/2055310541] conn(0x55b23dfbc680 0x55b23dfbeae0 crc :-1 s=READY pgs=26 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:16:53.503 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:53.502+0000 7f9d003db740 1 --2- 172.21.15.118:0/2991849956 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55b23dfa8580 0x55b23dfb81f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:16:53.503 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:53.502+0000 7f9d003db740 1 --2- 172.21.15.118:0/2991849956 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55b23dfa7bc0 0x55b23dfb11b0 unknown :-1 s=CLOSED pgs=255 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:16:53.503 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:53.502+0000 7f9d003db740 1 --2- 172.21.15.118:0/2991849956 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55b23dfa69f0 0x55b23dfb0c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:16:53.503 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:53.502+0000 7f9d003db740 1 -- 172.21.15.118:0/2991849956 >> 172.21.15.118:0/2991849956 conn(0x55b23dfa1de0 msgr2=0x55b23dfa1160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:16:53.503 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:53.502+0000 7f9cd7e00640 0 client.25676 ms_handle_reset on v2:172.21.15.193:6834/2055310541 2024-03-24T00:16:53.504 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:53.502+0000 7f9d003db740 1 -- 172.21.15.118:0/2991849956 shutdown_connections 2024-03-24T00:16:53.504 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:53.502+0000 7f9d003db740 1 -- 172.21.15.118:0/2991849956 wait complete. 2024-03-24T00:16:53.504 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:53.503+0000 7f9d003db740 20 client.25676 trim_cache size 0 max 16384 2024-03-24T00:16:53.522 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:53.521+0000 7f9d003db740 1 Processor -- start 2024-03-24T00:16:53.522 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:53.522+0000 7f9d003db740 1 -- start start 2024-03-24T00:16:53.523 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:53.522+0000 7f9d003db740 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55b23dfa69f0 0x55b23df9cff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:16:53.523 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:53.522+0000 7f9d003db740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55b23dfa8580 0x55b23dfb81b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:16:53.523 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:53.522+0000 7f9d003db740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55b23dfa5750 0x55b23dfaac60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:16:53.524 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:53.522+0000 7f9d003db740 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9cf81aaa90 con 0x55b23dfa69f0 2024-03-24T00:16:53.524 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:53.522+0000 7f9d003db740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9cf81ac410 con 0x55b23dfa8580 2024-03-24T00:16:53.524 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:53.522+0000 7f9d003db740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9cf81ab0f0 con 0x55b23dfa5750 2024-03-24T00:16:53.524 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:53.522+0000 7f9cf6000640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55b23dfa8580 0x55b23dfb81b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:16:53.524 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:53.522+0000 7f9cf6000640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55b23dfa8580 0x55b23dfb81b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.118:40390/0 (socket says 172.21.15.118:40390) 2024-03-24T00:16:53.524 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:53.522+0000 7f9cf6000640 1 -- 172.21.15.118:0/1494133262 learned_addr learned my addr 172.21.15.118:0/1494133262 (peer_addr_for_me v2:172.21.15.118:0/0) 2024-03-24T00:16:53.524 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:53.523+0000 7f9cf7400640 1 --2- 172.21.15.118:0/1494133262 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55b23dfa5750 0x55b23dfaac60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:16:53.524 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:53.523+0000 7f9cf7400640 1 --2- 172.21.15.118:0/1494133262 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55b23dfa69f0 0x55b23df9cff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:16:53.524 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:53.523+0000 7f9cf6000640 1 -- 172.21.15.118:0/1494133262 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55b23dfa5750 msgr2=0x55b23dfaac60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:16:53.524 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:53.523+0000 7f9cf6000640 1 --2- 172.21.15.118:0/1494133262 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55b23dfa5750 0x55b23dfaac60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:16:53.524 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:53.523+0000 7f9cf6000640 1 -- 172.21.15.118:0/1494133262 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55b23dfa69f0 msgr2=0x55b23df9cff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:16:53.525 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:53.523+0000 7f9cf6000640 1 --2- 172.21.15.118:0/1494133262 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55b23dfa69f0 0x55b23df9cff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:16:53.525 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:53.523+0000 7f9cf6000640 1 -- 172.21.15.118:0/1494133262 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55b23dfb1da0 con 0x55b23dfa8580 2024-03-24T00:16:53.525 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:53.523+0000 7f9cf7400640 1 --2- 172.21.15.118:0/1494133262 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55b23dfa5750 0x55b23dfaac60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T00:16:53.525 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:53.523+0000 7f9cf6000640 1 --2- 172.21.15.118:0/1494133262 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55b23dfa8580 0x55b23dfb81b0 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7f9ce4003940 tx=0x7f9ce4005db0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=320405a0040b36da server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:16:53.525 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:53.523+0000 7f9ce2a00640 1 -- 172.21.15.118:0/1494133262 <== mon.2 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9ce4029070 con 0x55b23dfa8580 2024-03-24T00:16:53.525 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:53.523+0000 7f9ce2a00640 1 -- 172.21.15.118:0/1494133262 <== mon.2 v2:172.21.15.189:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f9ce4003bb0 con 0x55b23dfa8580 2024-03-24T00:16:53.525 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:53.524+0000 7f9ce2a00640 1 -- 172.21.15.118:0/1494133262 <== mon.2 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9ce4010e00 con 0x55b23dfa8580 2024-03-24T00:16:53.526 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:53.524+0000 7f9d003db740 1 -- 172.21.15.118:0/1494133262 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55b23dfa8580 msgr2=0x55b23dfb81b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:16:53.526 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:53.524+0000 7f9d003db740 1 --2- 172.21.15.118:0/1494133262 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55b23dfa8580 0x55b23dfb81b0 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7f9ce4003940 tx=0x7f9ce4005db0 comp rx=0 tx=0).stop 2024-03-24T00:16:53.526 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:53.524+0000 7f9d003db740 1 -- 172.21.15.118:0/1494133262 shutdown_connections 2024-03-24T00:16:53.526 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:53.524+0000 7f9d003db740 1 --2- 172.21.15.118:0/1494133262 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55b23dfa5750 0x55b23dfaac60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:16:53.526 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:53.524+0000 7f9d003db740 1 --2- 172.21.15.118:0/1494133262 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55b23dfa8580 0x55b23dfb81b0 unknown :-1 s=CLOSED pgs=256 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:16:53.526 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:53.524+0000 7f9d003db740 1 --2- 172.21.15.118:0/1494133262 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55b23dfa69f0 0x55b23df9cff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:16:53.527 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:53.524+0000 7f9d003db740 1 -- 172.21.15.118:0/1494133262 >> 172.21.15.118:0/1494133262 conn(0x55b23dfa1de0 msgr2=0x55b23dfa5040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:16:53.527 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:53.524+0000 7f9d003db740 1 -- 172.21.15.118:0/1494133262 shutdown_connections 2024-03-24T00:16:53.527 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:53.525+0000 7f9d003db740 1 -- 172.21.15.118:0/1494133262 wait complete. 2024-03-24T00:16:53.527 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:53.525+0000 7f9d003db740 1 -- start start 2024-03-24T00:16:53.527 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:53.525+0000 7f9d003db740 1 Processor -- start 2024-03-24T00:16:53.527 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:53.525+0000 7f9d003db740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55b23dfa5750 0x55b23dfb41a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:16:53.527 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:53.526+0000 7f9d003db740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55b23dfa69f0 0x55b23dfb46e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:16:53.527 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:53.526+0000 7f9d003db740 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55b23dfa8580 0x55b23dfb76d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:16:53.527 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:53.526+0000 7f9d003db740 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9cf81abdb0 con 0x55b23dfa8580 2024-03-24T00:16:53.527 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:53.526+0000 7f9d003db740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9cf81ab750 con 0x55b23dfa5750 2024-03-24T00:16:53.527 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:53.526+0000 7f9d003db740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9cf81a8c50 con 0x55b23dfa69f0 2024-03-24T00:16:53.527 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:53.526+0000 7f9d003db740 10 client.0 _ll_register_callbacks cb 0x7f9ce4009550 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T00:16:53.527 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:53.526+0000 7f9cf7400640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55b23dfa5750 0x55b23dfb41a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:16:53.529 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:53.526+0000 7f9cf6000640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55b23dfa69f0 0x55b23dfb46e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:16:53.529 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:53.526+0000 7f9cf6000640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55b23dfa69f0 0x55b23dfb46e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.118:36220/0 (socket says 172.21.15.118:36220) 2024-03-24T00:16:53.529 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:53.526+0000 7f9cf6000640 1 -- 172.21.15.118:0/697106161 learned_addr learned my addr 172.21.15.118:0/697106161 (peer_addr_for_me v2:172.21.15.118:0/0) 2024-03-24T00:16:53.529 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:53.526+0000 7f9cf7400640 1 --2- 172.21.15.118:0/697106161 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55b23dfa8580 0x55b23dfb76d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:16:53.529 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:53.527+0000 7f9cf6000640 1 -- 172.21.15.118:0/697106161 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55b23dfa5750 msgr2=0x55b23dfb41a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:16:53.529 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:53.527+0000 7f9cf6000640 1 --2- 172.21.15.118:0/697106161 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55b23dfa5750 0x55b23dfb41a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:16:53.529 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:53.527+0000 7f9cf6000640 1 -- 172.21.15.118:0/697106161 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55b23dfa8580 msgr2=0x55b23dfb76d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:16:53.529 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:53.527+0000 7f9cf6000640 1 --2- 172.21.15.118:0/697106161 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55b23dfa8580 0x55b23dfb76d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:16:53.530 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:53.527+0000 7f9cf6000640 1 -- 172.21.15.118:0/697106161 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55b23dfb7cc0 con 0x55b23dfa69f0 2024-03-24T00:16:53.530 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:53.527+0000 7f9cf7400640 1 --2- 172.21.15.118:0/697106161 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55b23dfa8580 0x55b23dfb76d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T00:16:53.530 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:53.527+0000 7f9cf6000640 1 --2- 172.21.15.118:0/697106161 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55b23dfa69f0 0x55b23dfb46e0 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7f9ce40097c0 tx=0x7f9ce4003ea0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=4529719db31fbd91 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:16:53.530 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:53.527+0000 7f9cd7e00640 10 client.0 ms_handle_connect on v2:172.21.15.193:3300/0 2024-03-24T00:16:53.530 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:53.528+0000 7f9cd7e00640 1 -- 172.21.15.118:0/697106161 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9ce4029050 con 0x55b23dfa69f0 2024-03-24T00:16:53.530 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:53.528+0000 7f9cd7e00640 1 -- 172.21.15.118:0/697106161 <== mon.1 v2:172.21.15.193:3300/0 2 ==== osd_map(90..90 src has 1..90) v4 ==== 8277+0+0 (secure 0 0 0) 0x7f9ce40104f0 con 0x55b23dfa69f0 2024-03-24T00:16:53.530 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:53.528+0000 7f9d003db740 1 -- 172.21.15.118:0/697106161 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f9ce4003bb0 con 0x55b23dfa69f0 2024-03-24T00:16:53.530 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:53.528+0000 7f9cd7e00640 1 -- 172.21.15.118:0/697106161 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) v2 ==== 24+0+0 (secure 0 0 0) 0x7f9ce40106f0 con 0x55b23dfa69f0 2024-03-24T00:16:53.530 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:53.528+0000 7f9d003db740 10 client.25678 fetch_fsmap learned FSMap version 28 2024-03-24T00:16:53.530 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:53.528+0000 7f9d003db740 1 -- 172.21.15.118:0/697106161 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) v3 -- 0x55b23e020f60 con 0x55b23dfa69f0 2024-03-24T00:16:53.530 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:53.528+0000 7f9cd7e00640 1 -- 172.21.15.118:0/697106161 <== mon.1 v2:172.21.15.193:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f9ce4003bb0 con 0x55b23dfa69f0 2024-03-24T00:16:53.530 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:53.529+0000 7f9cd7e00640 1 -- 172.21.15.118:0/697106161 <== mon.1 v2:172.21.15.193:3300/0 5 ==== fsmap(e 28) v1 ==== 7934+0+0 (secure 0 0 0) 0x7f9ce4019040 con 0x55b23dfa69f0 2024-03-24T00:16:53.530 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:53.529+0000 7f9d003db740 10 client.25678 fetch_fsmap finished waiting for FSMap version 28 2024-03-24T00:16:53.530 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:53.529+0000 7f9d003db740 10 client.25678 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.f 2024-03-24T00:16:53.530 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:53.529+0000 7f9d003db740 20 client.25678 populate_metadata read hostname 'smithi118' 2024-03-24T00:16:53.531 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:53.529+0000 7f9d003db740 1 --2- 172.21.15.118:0/697106161 >> [v2:172.21.15.193:6834/2055310541,v1:172.21.15.193:6835/2055310541] conn(0x55b23e021820 0x55b23e023c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:16:53.531 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:53.529+0000 7f9d003db740 4 client.25678 mds_command: new command op to 24499 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-24T00:16:53.531 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:53.529+0000 7f9d003db740 1 -- 172.21.15.118:0/697106161 --> [v2:172.21.15.193:6834/2055310541,v1:172.21.15.193:6835/2055310541] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x55b23df12b50 con 0x55b23e021820 2024-03-24T00:16:53.532 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:53.529+0000 7f9cf7400640 1 --2- 172.21.15.118:0/697106161 >> [v2:172.21.15.193:6834/2055310541,v1:172.21.15.193:6835/2055310541] conn(0x55b23e021820 0x55b23e023c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:16:53.532 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:53.530+0000 7f9cd7e00640 10 client.25678 ms_handle_connect on v2:172.21.15.193:6834/2055310541 2024-03-24T00:16:53.532 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:53.530+0000 7f9cf7400640 1 --2- 172.21.15.118:0/697106161 >> [v2:172.21.15.193:6834/2055310541,v1:172.21.15.193:6835/2055310541] conn(0x55b23e021820 0x55b23e023c40 crc :-1 s=READY pgs=27 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=65bf6fad5f7d7f9e server_cookie=f03f3baf4dd5d7cd in_seq=0 out_seq=0 2024-03-24T00:16:53.532 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:53.530+0000 7f9cd7e00640 1 -- 172.21.15.118:0/697106161 <== mds.3 v2:172.21.15.193:6834/2055310541 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x55b23df12b50 con 0x55b23e021820 2024-03-24T00:16:53.532 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:53.530+0000 7f9cd7e00640 10 client.25678 handle_command_reply: tid=0 2024-03-24T00:16:53.532 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:53.530+0000 7f9d003db740 1 client.25678 shutdown 2024-03-24T00:16:53.532 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:53.531+0000 7f9d003db740 1 -- 172.21.15.118:0/697106161 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55b23dfa69f0 msgr2=0x55b23dfb46e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:16:53.532 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:53.531+0000 7f9d003db740 1 --2- 172.21.15.118:0/697106161 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55b23dfa69f0 0x55b23dfb46e0 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7f9ce40097c0 tx=0x7f9ce4003ea0 comp rx=0 tx=0).stop 2024-03-24T00:16:53.532 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:53.531+0000 7f9d003db740 1 -- 172.21.15.118:0/697106161 shutdown_connections 2024-03-24T00:16:53.532 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:53.531+0000 7f9d003db740 1 --2- 172.21.15.118:0/697106161 >> [v2:172.21.15.193:6834/2055310541,v1:172.21.15.193:6835/2055310541] conn(0x55b23e021820 0x55b23e023c40 crc :-1 s=READY pgs=27 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:16:53.533 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:53.531+0000 7f9d003db740 1 --2- 172.21.15.118:0/697106161 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55b23dfa8580 0x55b23dfb76d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:16:53.533 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:53.531+0000 7f9d003db740 1 --2- 172.21.15.118:0/697106161 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55b23dfa69f0 0x55b23dfb46e0 unknown :-1 s=CLOSED pgs=245 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:16:53.533 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:53.531+0000 7f9d003db740 1 --2- 172.21.15.118:0/697106161 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55b23dfa5750 0x55b23dfb41a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:16:53.533 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:53.531+0000 7f9d003db740 1 -- 172.21.15.118:0/697106161 >> 172.21.15.118:0/697106161 conn(0x55b23dfa1de0 msgr2=0x55b23dfa08d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:16:53.533 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:53.531+0000 7f9cd7e00640 0 client.25678 ms_handle_reset on v2:172.21.15.193:6834/2055310541 2024-03-24T00:16:53.533 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:53.531+0000 7f9d003db740 1 -- 172.21.15.118:0/697106161 shutdown_connections 2024-03-24T00:16:53.533 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:53.532+0000 7f9d003db740 1 -- 172.21.15.118:0/697106161 wait complete. 2024-03-24T00:16:53.533 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:53.532+0000 7f9d003db740 20 client.25678 trim_cache size 0 max 16384 2024-03-24T00:16:53.534 INFO:teuthology.orchestra.run.smithi118.stdout:[] 2024-03-24T00:16:53.534 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:53.532+0000 7f9cf7e00640 1 -- 172.21.15.118:0/1466931078 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7f9cdc0a2890 msgr2=0x7f9cdc0a4d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:16:53.534 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:53.532+0000 7f9cf7e00640 1 --2- 172.21.15.118:0/1466931078 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7f9cdc0a2890 0x7f9cdc0a4d50 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x7f9ce8098350 tx=0x7f9ce800cf20 comp rx=0 tx=0).stop 2024-03-24T00:16:53.534 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:53.532+0000 7f9cf7e00640 1 -- 172.21.15.118:0/1466931078 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f9cf80ffea0 msgr2=0x7f9cf8199a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:16:53.534 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:53.533+0000 7f9cf7e00640 1 --2- 172.21.15.118:0/1466931078 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f9cf80ffea0 0x7f9cf8199a50 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7f9ce800e9f0 tx=0x7f9ce800eeb0 comp rx=0 tx=0).stop 2024-03-24T00:16:53.534 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:53.533+0000 7f9cf7e00640 1 -- 172.21.15.118:0/1466931078 shutdown_connections 2024-03-24T00:16:53.535 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:53.533+0000 7f9cf7e00640 1 --2- 172.21.15.118:0/1466931078 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7f9cdc0a2890 0x7f9cdc0a4d50 unknown :-1 s=CLOSED pgs=287 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:16:53.535 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:53.533+0000 7f9cf7e00640 1 --2- 172.21.15.118:0/1466931078 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9cf81011a0 0x7f9cf81a1000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:16:53.535 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:53.533+0000 7f9cf7e00640 1 --2- 172.21.15.118:0/1466931078 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f9cf8100800 0x7f9cf8199f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:16:53.535 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:53.533+0000 7f9cf7e00640 1 --2- 172.21.15.118:0/1466931078 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f9cf80ffea0 0x7f9cf8199a50 unknown :-1 s=CLOSED pgs=244 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:16:53.535 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:53.533+0000 7f9cf7e00640 1 -- 172.21.15.118:0/1466931078 >> 172.21.15.118:0/1466931078 conn(0x7f9cf80fba20 msgr2=0x7f9cf80fd100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:16:53.535 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:53.533+0000 7f9cf7e00640 1 -- 172.21.15.118:0/1466931078 shutdown_connections 2024-03-24T00:16:53.535 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:53.533+0000 7f9cf7e00640 1 -- 172.21.15.118:0/1466931078 wait complete. 2024-03-24T00:16:53.554 INFO:teuthology.orchestra.run:Running command with timeout 120 2024-03-24T00:16:53.554 DEBUG:teuthology.orchestra.run.smithi118:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:2 damage ls 2024-03-24T00:16:53.761 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:53.760+0000 7f60e0800640 1 -- 172.21.15.118:0/1264099302 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f60d81005b0 msgr2=0x7f60d81009d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:16:53.762 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:53.760+0000 7f60e0800640 1 --2- 172.21.15.118:0/1264099302 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f60d81005b0 0x7f60d81009d0 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7f60cc009900 tx=0x7f60cc02bc30 comp rx=0 tx=0).stop 2024-03-24T00:16:53.762 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:53.761+0000 7f60e0800640 1 -- 172.21.15.118:0/1264099302 shutdown_connections 2024-03-24T00:16:53.762 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:53.761+0000 7f60e0800640 1 --2- 172.21.15.118:0/1264099302 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f60d8102160 0x7f60d81089f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:16:53.762 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:53.761+0000 7f60e0800640 1 --2- 172.21.15.118:0/1264099302 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f60d81017a0 0x7f60d8101c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:16:53.762 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:53.761+0000 7f60e0800640 1 --2- 172.21.15.118:0/1264099302 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f60d81005b0 0x7f60d81009d0 unknown :-1 s=CLOSED pgs=257 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:16:53.762 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:53.761+0000 7f60e0800640 1 -- 172.21.15.118:0/1264099302 >> 172.21.15.118:0/1264099302 conn(0x7f60d80fbb40 msgr2=0x7f60d80fdf60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:16:53.762 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:53.761+0000 7f60e0800640 1 -- 172.21.15.118:0/1264099302 shutdown_connections 2024-03-24T00:16:53.762 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:53.761+0000 7f60e0800640 1 -- 172.21.15.118:0/1264099302 wait complete. 2024-03-24T00:16:53.763 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:53.762+0000 7f60e0800640 1 Processor -- start 2024-03-24T00:16:53.763 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:53.762+0000 7f60e0800640 1 -- start start 2024-03-24T00:16:53.764 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:53.762+0000 7f60e0800640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f60d81005b0 0x7f60d8104540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:16:53.764 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:53.762+0000 7f60e0800640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f60d81017a0 0x7f60d8102b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:16:53.764 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:53.762+0000 7f60e0800640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f60d8102160 0x7f60d81030c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:16:53.764 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:53.762+0000 7f60e0800640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f60d810c250 con 0x7f60d8102160 2024-03-24T00:16:53.764 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:53.762+0000 7f60e0800640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f60d810c0d0 con 0x7f60d81005b0 2024-03-24T00:16:53.764 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:53.763+0000 7f60e0800640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f60d810c3d0 con 0x7f60d81017a0 2024-03-24T00:16:53.764 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:53.763+0000 7f60dde00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f60d81005b0 0x7f60d8104540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:16:53.764 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:53.763+0000 7f60dde00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f60d81005b0 0x7f60d8104540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.118:40414/0 (socket says 172.21.15.118:40414) 2024-03-24T00:16:53.764 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:53.763+0000 7f60dde00640 1 -- 172.21.15.118:0/4000796329 learned_addr learned my addr 172.21.15.118:0/4000796329 (peer_addr_for_me v2:172.21.15.118:0/0) 2024-03-24T00:16:53.764 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:53.763+0000 7f60de800640 1 --2- 172.21.15.118:0/4000796329 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f60d8102160 0x7f60d81030c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:16:53.764 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:53.763+0000 7f60dd400640 1 --2- 172.21.15.118:0/4000796329 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f60d81017a0 0x7f60d8102b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:16:53.765 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:53.763+0000 7f60dd400640 1 -- 172.21.15.118:0/4000796329 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f60d81005b0 msgr2=0x7f60d8104540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:16:53.765 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:53.763+0000 7f60dd400640 1 --2- 172.21.15.118:0/4000796329 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f60d81005b0 0x7f60d8104540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:16:53.765 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:53.763+0000 7f60dd400640 1 -- 172.21.15.118:0/4000796329 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f60d8102160 msgr2=0x7f60d81030c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:16:53.765 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:53.764+0000 7f60dd400640 1 --2- 172.21.15.118:0/4000796329 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f60d8102160 0x7f60d81030c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:16:53.765 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:53.764+0000 7f60dd400640 1 -- 172.21.15.118:0/4000796329 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f60d8103950 con 0x7f60d81017a0 2024-03-24T00:16:53.765 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:53.764+0000 7f60dde00640 1 --2- 172.21.15.118:0/4000796329 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f60d81005b0 0x7f60d8104540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T00:16:53.765 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:53.764+0000 7f60de800640 1 --2- 172.21.15.118:0/4000796329 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f60d8102160 0x7f60d81030c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T00:16:53.766 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:53.764+0000 7f60dd400640 1 --2- 172.21.15.118:0/4000796329 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f60d81017a0 0x7f60d8102b80 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7f60c8005fd0 tx=0x7f60c800dde0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:16:53.766 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:53.764+0000 7f60d2a00640 1 -- 172.21.15.118:0/4000796329 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f60c8009c30 con 0x7f60d81017a0 2024-03-24T00:16:53.766 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:53.764+0000 7f60d2a00640 1 -- 172.21.15.118:0/4000796329 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f60c8009dd0 con 0x7f60d81017a0 2024-03-24T00:16:53.766 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:53.764+0000 7f60d2a00640 1 -- 172.21.15.118:0/4000796329 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f60c8010460 con 0x7f60d81017a0 2024-03-24T00:16:53.766 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:53.764+0000 7f60e0800640 1 -- 172.21.15.118:0/4000796329 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f60d81a1b60 con 0x7f60d81017a0 2024-03-24T00:16:53.766 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:53.764+0000 7f60e0800640 1 -- 172.21.15.118:0/4000796329 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f60d81a2060 con 0x7f60d81017a0 2024-03-24T00:16:53.768 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:53.767+0000 7f60d2a00640 1 -- 172.21.15.118:0/4000796329 <== mon.1 v2:172.21.15.193:3300/0 4 ==== mgrmap(e 19) v1 ==== 137342+0+0 (secure 0 0 0) 0x7f60c8003f50 con 0x7f60d81017a0 2024-03-24T00:16:53.769 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:53.769+0000 7f60d2a00640 1 --2- 172.21.15.118:0/4000796329 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7f60c00a2910 0x7f60c00a4dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:16:53.769 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:53.769+0000 7f60e3574740 1 Processor -- start 2024-03-24T00:16:53.770 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:53.769+0000 7f60d2a00640 1 -- 172.21.15.118:0/4000796329 <== mon.1 v2:172.21.15.193:3300/0 5 ==== osd_map(90..90 src has 1..90) v4 ==== 8277+0+0 (secure 0 0 0) 0x7f60c80cec00 con 0x7f60d81017a0 2024-03-24T00:16:53.770 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:53.769+0000 7f60e3574740 1 -- start start 2024-03-24T00:16:53.770 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:53.770+0000 7f60e3574740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55c8089c4810 0x55c8089c4c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:16:53.770 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:53.770+0000 7f60e3574740 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55c8089c59e0 0x55c8089c5e60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:16:53.771 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:53.770+0000 7f60e3574740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55c8089c63a0 0x55c8089ccc30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:16:53.771 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:53.770+0000 7f60e3574740 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c8089292c0 con 0x55c8089c59e0 2024-03-24T00:16:53.771 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:53.770+0000 7f60e3574740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c8088d1c90 con 0x55c8089c4810 2024-03-24T00:16:53.771 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:53.770+0000 7f60e3574740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c808925360 con 0x55c8089c63a0 2024-03-24T00:16:53.771 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:53.770+0000 7f60dde00640 1 --2- 172.21.15.118:0/4000796329 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7f60c00a2910 0x7f60c00a4dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:16:53.772 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:53.770+0000 7f60de800640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55c8089c63a0 0x55c8089ccc30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:16:53.772 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:53.770+0000 7f60de800640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55c8089c63a0 0x55c8089ccc30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.118:36234/0 (socket says 172.21.15.118:36234) 2024-03-24T00:16:53.772 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:53.770+0000 7f60de800640 1 -- 172.21.15.118:0/3041288922 learned_addr learned my addr 172.21.15.118:0/3041288922 (peer_addr_for_me v2:172.21.15.118:0/0) 2024-03-24T00:16:53.772 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:53.770+0000 7f60dd400640 1 --2- 172.21.15.118:0/3041288922 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55c8089c59e0 0x55c8089c5e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:16:53.772 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:53.770+0000 7f60dde00640 1 --2- 172.21.15.118:0/3041288922 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55c8089c4810 0x55c8089c4c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:16:53.772 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:53.771+0000 7f60de800640 1 -- 172.21.15.118:0/3041288922 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55c8089c4810 msgr2=0x55c8089c4c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:16:53.772 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:53.771+0000 7f60de800640 1 --2- 172.21.15.118:0/3041288922 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55c8089c4810 0x55c8089c4c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:16:53.772 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:53.771+0000 7f60de800640 1 -- 172.21.15.118:0/3041288922 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55c8089c59e0 msgr2=0x55c8089c5e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:16:53.772 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:53.771+0000 7f60de800640 1 --2- 172.21.15.118:0/3041288922 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55c8089c59e0 0x55c8089c5e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:16:53.772 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:53.771+0000 7f60de800640 1 -- 172.21.15.118:0/3041288922 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55c8088f6080 con 0x55c8089c63a0 2024-03-24T00:16:53.772 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:53.771+0000 7f60dd400640 1 --2- 172.21.15.118:0/3041288922 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55c8089c59e0 0x55c8089c5e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T00:16:53.772 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:53.771+0000 7f60dde00640 1 --2- 172.21.15.118:0/4000796329 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7f60c00a2910 0x7f60c00a4dd0 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 crypto rx=0x7f60cc006010 tx=0x7f60cc005dd0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:16:53.773 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:53.771+0000 7f60de800640 1 --2- 172.21.15.118:0/3041288922 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55c8089c63a0 0x55c8089ccc30 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7f60d4005f40 tx=0x7f60d4010860 comp rx=0 tx=0).ready entity=mon.1 client_cookie=48edefa3d669412a server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:16:53.773 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:53.772+0000 7f60c7000640 1 -- 172.21.15.118:0/3041288922 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f60d4011700 con 0x55c8089c63a0 2024-03-24T00:16:53.774 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:53.772+0000 7f60c7000640 1 -- 172.21.15.118:0/3041288922 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f60d4011d80 con 0x55c8089c63a0 2024-03-24T00:16:53.774 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:53.772+0000 7f60c7000640 1 -- 172.21.15.118:0/3041288922 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f60d401a380 con 0x55c8089c63a0 2024-03-24T00:16:53.774 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:53.773+0000 7f60e3574740 1 -- 172.21.15.118:0/3041288922 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55c8089c63a0 msgr2=0x55c8089ccc30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:16:53.774 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:53.773+0000 7f60e3574740 1 --2- 172.21.15.118:0/3041288922 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55c8089c63a0 0x55c8089ccc30 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7f60d4005f40 tx=0x7f60d4010860 comp rx=0 tx=0).stop 2024-03-24T00:16:53.774 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:53.773+0000 7f60e3574740 1 -- 172.21.15.118:0/3041288922 shutdown_connections 2024-03-24T00:16:53.774 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:53.773+0000 7f60e3574740 1 --2- 172.21.15.118:0/3041288922 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55c8089c63a0 0x55c8089ccc30 unknown :-1 s=CLOSED pgs=247 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:16:53.774 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:53.773+0000 7f60e3574740 1 --2- 172.21.15.118:0/3041288922 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55c8089c59e0 0x55c8089c5e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:16:53.774 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:53.773+0000 7f60e3574740 1 --2- 172.21.15.118:0/3041288922 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55c8089c4810 0x55c8089c4c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:16:53.774 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:53.773+0000 7f60e3574740 1 -- 172.21.15.118:0/3041288922 >> 172.21.15.118:0/3041288922 conn(0x55c8089bfc00 msgr2=0x55c8089c0000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:16:53.774 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:53.773+0000 7f60e3574740 1 -- 172.21.15.118:0/3041288922 shutdown_connections 2024-03-24T00:16:53.774 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:53.773+0000 7f60e3574740 1 -- 172.21.15.118:0/3041288922 wait complete. 2024-03-24T00:16:53.775 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:53.773+0000 7f60e3574740 1 -- start start 2024-03-24T00:16:53.775 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:53.774+0000 7f60e3574740 1 Processor -- start 2024-03-24T00:16:53.775 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:53.774+0000 7f60e3574740 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55c8089c4810 0x55c8089ceb10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:16:53.775 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:53.774+0000 7f60e3574740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55c8089c59e0 0x55c8089cf050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:16:53.775 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:53.774+0000 7f60e3574740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55c8089c63a0 0x55c8089d6020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:16:53.775 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:53.774+0000 7f60e3574740 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c8089cd400 con 0x55c8089c4810 2024-03-24T00:16:53.775 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:53.774+0000 7f60e3574740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c8088e50d0 con 0x55c8089c63a0 2024-03-24T00:16:53.775 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:53.774+0000 7f60e3574740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c8089cd580 con 0x55c8089c59e0 2024-03-24T00:16:53.776 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:53.775+0000 7f60e3574740 10 client.0 _ll_register_callbacks cb 0x55c8088e9e80 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T00:16:53.776 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:53.775+0000 7f60dde00640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55c8089c4810 0x55c8089ceb10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:16:53.776 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:53.775+0000 7f60dde00640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55c8089c4810 0x55c8089ceb10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.118:3300/0 says I am v2:172.21.15.118:50498/0 (socket says 172.21.15.118:50498) 2024-03-24T00:16:53.777 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:53.775+0000 7f60dde00640 1 -- 172.21.15.118:0/2179921932 learned_addr learned my addr 172.21.15.118:0/2179921932 (peer_addr_for_me v2:172.21.15.118:0/0) 2024-03-24T00:16:53.777 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:53.775+0000 7f60de800640 1 --2- 172.21.15.118:0/2179921932 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55c8089c63a0 0x55c8089d6020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:16:53.777 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:53.775+0000 7f60dd400640 1 --2- 172.21.15.118:0/2179921932 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55c8089c59e0 0x55c8089cf050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:16:53.777 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:53.775+0000 7f60de800640 1 -- 172.21.15.118:0/2179921932 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55c8089c59e0 msgr2=0x55c8089cf050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:16:53.777 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:53.775+0000 7f60de800640 1 --2- 172.21.15.118:0/2179921932 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55c8089c59e0 0x55c8089cf050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:16:53.777 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:53.775+0000 7f60de800640 1 -- 172.21.15.118:0/2179921932 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55c8089c4810 msgr2=0x55c8089ceb10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:16:53.777 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:53.775+0000 7f60de800640 1 --2- 172.21.15.118:0/2179921932 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55c8089c4810 0x55c8089ceb10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:16:53.777 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:53.775+0000 7f60de800640 1 -- 172.21.15.118:0/2179921932 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55c80863d3f0 con 0x55c8089c63a0 2024-03-24T00:16:53.777 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:53.775+0000 7f60dd400640 1 --2- 172.21.15.118:0/2179921932 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55c8089c59e0 0x55c8089cf050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-24T00:16:53.777 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:53.775+0000 7f60dde00640 1 --2- 172.21.15.118:0/2179921932 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55c8089c4810 0x55c8089ceb10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T00:16:53.777 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:53.776+0000 7f60de800640 1 --2- 172.21.15.118:0/2179921932 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55c8089c63a0 0x55c8089d6020 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7f60d4010d60 tx=0x7f60d40098b0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=9fcf1c81ad77330c server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:16:53.777 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:53.776+0000 7f60bb400640 10 client.0 ms_handle_connect on v2:172.21.15.189:3300/0 2024-03-24T00:16:53.778 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:53.776+0000 7f60bb400640 1 -- 172.21.15.118:0/2179921932 <== mon.2 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f60d402a060 con 0x55c8089c63a0 2024-03-24T00:16:53.778 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:53.777+0000 7f60bb400640 1 -- 172.21.15.118:0/2179921932 <== mon.2 v2:172.21.15.189:3300/0 2 ==== osd_map(90..90 src has 1..90) v4 ==== 8277+0+0 (secure 0 0 0) 0x7f60d4011500 con 0x55c8089c63a0 2024-03-24T00:16:53.778 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:53.777+0000 7f60e3574740 1 -- 172.21.15.118:0/2179921932 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f60d4011d80 con 0x55c8089c63a0 2024-03-24T00:16:53.779 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:53.777+0000 7f60bb400640 1 -- 172.21.15.118:0/2179921932 <== mon.2 v2:172.21.15.189:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) v2 ==== 24+0+0 (secure 0 0 0) 0x7f60d4009c00 con 0x55c8089c63a0 2024-03-24T00:16:53.779 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:53.777+0000 7f60e3574740 10 client.25694 fetch_fsmap learned FSMap version 28 2024-03-24T00:16:53.779 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:53.777+0000 7f60e3574740 1 -- 172.21.15.118:0/2179921932 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) v3 -- 0x55c8087077d0 con 0x55c8089c63a0 2024-03-24T00:16:53.779 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:53.778+0000 7f60bb400640 1 -- 172.21.15.118:0/2179921932 <== mon.2 v2:172.21.15.189:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f60d4011d80 con 0x55c8089c63a0 2024-03-24T00:16:53.779 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:53.778+0000 7f60bb400640 1 -- 172.21.15.118:0/2179921932 <== mon.2 v2:172.21.15.189:3300/0 5 ==== fsmap(e 28) v1 ==== 7934+0+0 (secure 0 0 0) 0x7f60d40057a0 con 0x55c8089c63a0 2024-03-24T00:16:53.779 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:53.778+0000 7f60e3574740 10 client.25694 fetch_fsmap finished waiting for FSMap version 28 2024-03-24T00:16:53.779 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:53.778+0000 7f60e3574740 10 client.25694 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.c 2024-03-24T00:16:53.779 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:53.778+0000 7f60e3574740 20 client.25694 populate_metadata read hostname 'smithi118' 2024-03-24T00:16:53.779 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:53.778+0000 7f60e3574740 1 --2- 172.21.15.118:0/2179921932 >> [v2:172.21.15.193:6836/2921520053,v1:172.21.15.193:6837/2921520053] conn(0x55c8089da180 0x55c8089dc570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:16:53.780 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:53.778+0000 7f60e3574740 4 client.25694 mds_command: new command op to 24511 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-24T00:16:53.780 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:53.778+0000 7f60e3574740 1 -- 172.21.15.118:0/2179921932 --> [v2:172.21.15.193:6836/2921520053,v1:172.21.15.193:6837/2921520053] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55c808936800 con 0x55c8089da180 2024-03-24T00:16:53.780 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:53.779+0000 7f60dde00640 1 --2- 172.21.15.118:0/2179921932 >> [v2:172.21.15.193:6836/2921520053,v1:172.21.15.193:6837/2921520053] conn(0x55c8089da180 0x55c8089dc570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:16:53.780 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:53.780+0000 7f60dde00640 1 --2- 172.21.15.118:0/2179921932 >> [v2:172.21.15.193:6836/2921520053,v1:172.21.15.193:6837/2921520053] conn(0x55c8089da180 0x55c8089dc570 crc :-1 s=READY pgs=20 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=b6953aea244b8e1d server_cookie=a20982e6b4be7d87 in_seq=0 out_seq=0 2024-03-24T00:16:53.780 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:53.780+0000 7f60bb400640 10 client.25694 ms_handle_connect on v2:172.21.15.193:6836/2921520053 2024-03-24T00:16:53.781 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:53.780+0000 7f60bb400640 1 -- 172.21.15.118:0/2179921932 <== mds.2 v2:172.21.15.193:6836/2921520053 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x55c808936800 con 0x55c8089da180 2024-03-24T00:16:53.781 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:53.780+0000 7f60bb400640 10 client.25694 handle_command_reply: tid=0 2024-03-24T00:16:53.781 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:53.780+0000 7f60e3574740 1 client.25694 shutdown 2024-03-24T00:16:53.782 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:53.781+0000 7f60e3574740 1 -- 172.21.15.118:0/2179921932 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55c8089c63a0 msgr2=0x55c8089d6020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:16:53.782 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:53.781+0000 7f60e3574740 1 --2- 172.21.15.118:0/2179921932 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55c8089c63a0 0x55c8089d6020 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7f60d4010d60 tx=0x7f60d40098b0 comp rx=0 tx=0).stop 2024-03-24T00:16:53.782 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:53.781+0000 7f60e3574740 1 -- 172.21.15.118:0/2179921932 shutdown_connections 2024-03-24T00:16:53.782 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:53.781+0000 7f60e3574740 1 --2- 172.21.15.118:0/2179921932 >> [v2:172.21.15.193:6836/2921520053,v1:172.21.15.193:6837/2921520053] conn(0x55c8089da180 0x55c8089dc570 crc :-1 s=READY pgs=20 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:16:53.782 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:53.781+0000 7f60e3574740 1 --2- 172.21.15.118:0/2179921932 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55c8089c63a0 0x55c8089d6020 unknown :-1 s=CLOSED pgs=258 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:16:53.782 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:53.781+0000 7f60bb400640 0 client.25694 ms_handle_reset on v2:172.21.15.193:6836/2921520053 2024-03-24T00:16:53.782 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:53.781+0000 7f60e3574740 1 --2- 172.21.15.118:0/2179921932 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55c8089c59e0 0x55c8089cf050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:16:53.782 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:53.781+0000 7f60e3574740 1 --2- 172.21.15.118:0/2179921932 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55c8089c4810 0x55c8089ceb10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:16:53.783 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:53.781+0000 7f60e3574740 1 -- 172.21.15.118:0/2179921932 >> 172.21.15.118:0/2179921932 conn(0x55c8089bfc00 msgr2=0x55c8089beeb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:16:53.783 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:53.781+0000 7f60e3574740 1 -- 172.21.15.118:0/2179921932 shutdown_connections 2024-03-24T00:16:53.783 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:53.781+0000 7f60e3574740 1 -- 172.21.15.118:0/2179921932 wait complete. 2024-03-24T00:16:53.783 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:53.781+0000 7f60e3574740 20 client.25694 trim_cache size 0 max 16384 2024-03-24T00:16:53.808 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:53.807+0000 7f60e3574740 1 Processor -- start 2024-03-24T00:16:53.808 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:53.807+0000 7f60e3574740 1 -- start start 2024-03-24T00:16:53.808 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:53.807+0000 7f60e3574740 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55c8089c4810 0x55c8089c8a80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:16:53.808 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:53.807+0000 7f60e3574740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55c8089c59e0 0x55c8089d5fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:16:53.808 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:53.807+0000 7f60e3574740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55c8089c63a0 0x55c8089c8fc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:16:53.808 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:53.807+0000 7f60e3574740 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f60d81aae30 con 0x55c8089c4810 2024-03-24T00:16:53.808 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:53.807+0000 7f60e3574740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f60d81ac7b0 con 0x55c8089c59e0 2024-03-24T00:16:53.809 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:53.807+0000 7f60e3574740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f60d81ab490 con 0x55c8089c63a0 2024-03-24T00:16:53.809 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:53.808+0000 7f60dde00640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55c8089c4810 0x55c8089c8a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:16:53.809 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:53.808+0000 7f60dd400640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55c8089c59e0 0x55c8089d5fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:16:53.809 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:53.808+0000 7f60dd400640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55c8089c59e0 0x55c8089d5fe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.118:40432/0 (socket says 172.21.15.118:40432) 2024-03-24T00:16:53.810 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:53.808+0000 7f60dd400640 1 -- 172.21.15.118:0/2647473886 learned_addr learned my addr 172.21.15.118:0/2647473886 (peer_addr_for_me v2:172.21.15.118:0/0) 2024-03-24T00:16:53.810 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:53.808+0000 7f60de800640 1 --2- 172.21.15.118:0/2647473886 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55c8089c63a0 0x55c8089c8fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:16:53.810 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:53.808+0000 7f60de800640 1 -- 172.21.15.118:0/2647473886 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55c8089c59e0 msgr2=0x55c8089d5fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:16:53.810 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:53.808+0000 7f60de800640 1 --2- 172.21.15.118:0/2647473886 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55c8089c59e0 0x55c8089d5fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:16:53.810 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:53.808+0000 7f60de800640 1 -- 172.21.15.118:0/2647473886 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55c8089c4810 msgr2=0x55c8089c8a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:16:53.810 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:53.808+0000 7f60de800640 1 --2- 172.21.15.118:0/2647473886 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55c8089c4810 0x55c8089c8a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:16:53.810 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:53.808+0000 7f60de800640 1 -- 172.21.15.118:0/2647473886 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55c8089ba2c0 con 0x55c8089c63a0 2024-03-24T00:16:53.810 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:53.809+0000 7f60de800640 1 --2- 172.21.15.118:0/2647473886 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55c8089c63a0 0x55c8089c8fc0 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7f60d4010d60 tx=0x7f60d401cc30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=e0679a1b61d2151f server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:16:53.810 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:53.809+0000 7f60c6600640 1 -- 172.21.15.118:0/2647473886 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f60d402a070 con 0x55c8089c63a0 2024-03-24T00:16:53.810 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:53.809+0000 7f60c6600640 1 -- 172.21.15.118:0/2647473886 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f60d4003e50 con 0x55c8089c63a0 2024-03-24T00:16:53.810 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:53.809+0000 7f60c6600640 1 -- 172.21.15.118:0/2647473886 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f60d4034080 con 0x55c8089c63a0 2024-03-24T00:16:53.811 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:53.810+0000 7f60e3574740 1 -- 172.21.15.118:0/2647473886 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55c8089c63a0 msgr2=0x55c8089c8fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:16:53.811 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:53.810+0000 7f60e3574740 1 --2- 172.21.15.118:0/2647473886 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55c8089c63a0 0x55c8089c8fc0 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7f60d4010d60 tx=0x7f60d401cc30 comp rx=0 tx=0).stop 2024-03-24T00:16:53.811 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:53.810+0000 7f60e3574740 1 -- 172.21.15.118:0/2647473886 shutdown_connections 2024-03-24T00:16:53.811 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:53.810+0000 7f60e3574740 1 --2- 172.21.15.118:0/2647473886 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55c8089c63a0 0x55c8089c8fc0 unknown :-1 s=CLOSED pgs=248 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:16:53.812 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:53.810+0000 7f60e3574740 1 --2- 172.21.15.118:0/2647473886 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55c8089c59e0 0x55c8089d5fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:16:53.812 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:53.810+0000 7f60e3574740 1 --2- 172.21.15.118:0/2647473886 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55c8089c4810 0x55c8089c8a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:16:53.812 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:53.810+0000 7f60e3574740 1 -- 172.21.15.118:0/2647473886 >> 172.21.15.118:0/2647473886 conn(0x55c8089bfc00 msgr2=0x55c8089c36f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:16:53.812 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:53.810+0000 7f60e3574740 1 -- 172.21.15.118:0/2647473886 shutdown_connections 2024-03-24T00:16:53.812 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:53.810+0000 7f60e3574740 1 -- 172.21.15.118:0/2647473886 wait complete. 2024-03-24T00:16:53.812 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:53.811+0000 7f60e3574740 1 -- start start 2024-03-24T00:16:53.812 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:53.811+0000 7f60e3574740 1 Processor -- start 2024-03-24T00:16:53.814 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:53.812+0000 7f60e3574740 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55c8089c4810 0x55c8089d1f10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:16:53.814 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:53.812+0000 7f60e3574740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55c8089c59e0 0x55c8089d2450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:16:53.814 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:53.812+0000 7f60e3574740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55c8089c63a0 0x55c8089d5470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:16:53.814 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:53.812+0000 7f60e3574740 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f60d81ac150 con 0x55c8089c4810 2024-03-24T00:16:53.815 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:53.812+0000 7f60e3574740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f60d81abaf0 con 0x55c8089c63a0 2024-03-24T00:16:53.815 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:53.812+0000 7f60e3574740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f60d81a8ff0 con 0x55c8089c59e0 2024-03-24T00:16:53.815 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:53.812+0000 7f60e3574740 10 client.0 _ll_register_callbacks cb 0x7f60c800df10 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T00:16:53.815 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:53.812+0000 7f60de800640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55c8089c63a0 0x55c8089d5470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:16:53.815 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:53.812+0000 7f60de800640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55c8089c63a0 0x55c8089d5470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.118:40440/0 (socket says 172.21.15.118:40440) 2024-03-24T00:16:53.815 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:53.812+0000 7f60de800640 1 -- 172.21.15.118:0/644002080 learned_addr learned my addr 172.21.15.118:0/644002080 (peer_addr_for_me v2:172.21.15.118:0/0) 2024-03-24T00:16:53.815 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:53.812+0000 7f60dd400640 1 --2- 172.21.15.118:0/644002080 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55c8089c59e0 0x55c8089d2450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:16:53.815 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:53.812+0000 7f60de800640 1 -- 172.21.15.118:0/644002080 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55c8089c59e0 msgr2=0x55c8089d2450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:16:53.815 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:53.812+0000 7f60de800640 1 --2- 172.21.15.118:0/644002080 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55c8089c59e0 0x55c8089d2450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:16:53.815 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:53.812+0000 7f60de800640 1 -- 172.21.15.118:0/644002080 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55c8089c4810 msgr2=0x55c8089d1f10 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-24T00:16:53.816 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:53.813+0000 7f60dde00640 1 --2- 172.21.15.118:0/644002080 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55c8089c4810 0x55c8089d1f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:16:53.816 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:53.813+0000 7f60de800640 1 --2- 172.21.15.118:0/644002080 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55c8089c4810 0x55c8089d1f10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:16:53.816 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:53.813+0000 7f60de800640 1 -- 172.21.15.118:0/644002080 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55c8089d5a60 con 0x55c8089c63a0 2024-03-24T00:16:53.816 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:53.813+0000 7f60dde00640 1 --2- 172.21.15.118:0/644002080 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55c8089c4810 0x55c8089d1f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-24T00:16:53.816 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:53.813+0000 7f60de800640 1 --2- 172.21.15.118:0/644002080 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55c8089c63a0 0x55c8089d5470 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7f60d4006010 tx=0x7f60d4030c80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ad9fb726eae6a58f server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:16:53.816 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:53.813+0000 7f60bb400640 10 client.0 ms_handle_connect on v2:172.21.15.189:3300/0 2024-03-24T00:16:53.816 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:53.813+0000 7f60bb400640 1 -- 172.21.15.118:0/644002080 <== mon.2 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f60d4034070 con 0x55c8089c63a0 2024-03-24T00:16:53.816 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:53.813+0000 7f60bb400640 1 -- 172.21.15.118:0/644002080 <== mon.2 v2:172.21.15.189:3300/0 2 ==== osd_map(90..90 src has 1..90) v4 ==== 8277+0+0 (secure 0 0 0) 0x7f60d40042b0 con 0x55c8089c63a0 2024-03-24T00:16:53.816 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:53.814+0000 7f60e3574740 1 -- 172.21.15.118:0/644002080 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f60d4003e50 con 0x55c8089c63a0 2024-03-24T00:16:53.817 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:53.814+0000 7f60bb400640 1 -- 172.21.15.118:0/644002080 <== mon.2 v2:172.21.15.189:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) v2 ==== 24+0+0 (secure 0 0 0) 0x7f60d402a040 con 0x55c8089c63a0 2024-03-24T00:16:53.817 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:53.814+0000 7f60e3574740 10 client.25700 fetch_fsmap learned FSMap version 28 2024-03-24T00:16:53.817 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:53.814+0000 7f60e3574740 1 -- 172.21.15.118:0/644002080 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) v3 -- 0x55c808a36010 con 0x55c8089c63a0 2024-03-24T00:16:53.817 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:53.816+0000 7f60bb400640 1 -- 172.21.15.118:0/644002080 <== mon.2 v2:172.21.15.189:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f60d4003e50 con 0x55c8089c63a0 2024-03-24T00:16:53.818 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:53.817+0000 7f60bb400640 1 -- 172.21.15.118:0/644002080 <== mon.2 v2:172.21.15.189:3300/0 5 ==== fsmap(e 28) v1 ==== 7934+0+0 (secure 0 0 0) 0x7f60d4011500 con 0x55c8089c63a0 2024-03-24T00:16:53.818 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:53.817+0000 7f60e3574740 10 client.25700 fetch_fsmap finished waiting for FSMap version 28 2024-03-24T00:16:53.818 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:53.817+0000 7f60e3574740 10 client.25700 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.c 2024-03-24T00:16:53.818 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:53.817+0000 7f60e3574740 20 client.25700 populate_metadata read hostname 'smithi118' 2024-03-24T00:16:53.818 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:53.817+0000 7f60e3574740 1 --2- 172.21.15.118:0/644002080 >> [v2:172.21.15.193:6836/2921520053,v1:172.21.15.193:6837/2921520053] conn(0x55c808a367d0 0x55c808a38bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:16:53.818 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:53.817+0000 7f60e3574740 4 client.25700 mds_command: new command op to 24511 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-24T00:16:53.818 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:53.818+0000 7f60e3574740 1 -- 172.21.15.118:0/644002080 --> [v2:172.21.15.193:6836/2921520053,v1:172.21.15.193:6837/2921520053] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x55c808930b50 con 0x55c808a367d0 2024-03-24T00:16:53.819 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:53.818+0000 7f60dde00640 1 --2- 172.21.15.118:0/644002080 >> [v2:172.21.15.193:6836/2921520053,v1:172.21.15.193:6837/2921520053] conn(0x55c808a367d0 0x55c808a38bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:16:53.819 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:53.818+0000 7f60dde00640 1 --2- 172.21.15.118:0/644002080 >> [v2:172.21.15.193:6836/2921520053,v1:172.21.15.193:6837/2921520053] conn(0x55c808a367d0 0x55c808a38bf0 crc :-1 s=READY pgs=21 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=a5321e60660caba server_cookie=be16ad5a0d463bd6 in_seq=0 out_seq=0 2024-03-24T00:16:53.819 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:53.818+0000 7f60bb400640 10 client.25700 ms_handle_connect on v2:172.21.15.193:6836/2921520053 2024-03-24T00:16:53.819 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:53.819+0000 7f60bb400640 1 -- 172.21.15.118:0/644002080 <== mds.2 v2:172.21.15.193:6836/2921520053 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x55c808930b50 con 0x55c808a367d0 2024-03-24T00:16:53.819 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:53.819+0000 7f60bb400640 10 client.25700 handle_command_reply: tid=0 2024-03-24T00:16:53.819 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:53.819+0000 7f60e3574740 1 client.25700 shutdown 2024-03-24T00:16:53.820 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:53.819+0000 7f60e3574740 1 -- 172.21.15.118:0/644002080 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55c8089c63a0 msgr2=0x55c8089d5470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:16:53.820 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:53.819+0000 7f60e3574740 1 --2- 172.21.15.118:0/644002080 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55c8089c63a0 0x55c8089d5470 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7f60d4006010 tx=0x7f60d4030c80 comp rx=0 tx=0).stop 2024-03-24T00:16:53.820 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:53.819+0000 7f60e3574740 1 -- 172.21.15.118:0/644002080 shutdown_connections 2024-03-24T00:16:53.820 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:53.819+0000 7f60e3574740 1 --2- 172.21.15.118:0/644002080 >> [v2:172.21.15.193:6836/2921520053,v1:172.21.15.193:6837/2921520053] conn(0x55c808a367d0 0x55c808a38bf0 crc :-1 s=READY pgs=21 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:16:53.820 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:53.819+0000 7f60e3574740 1 --2- 172.21.15.118:0/644002080 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55c8089c63a0 0x55c8089d5470 unknown :-1 s=CLOSED pgs=259 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:16:53.820 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:53.819+0000 7f60bb400640 0 client.25700 ms_handle_reset on v2:172.21.15.193:6836/2921520053 2024-03-24T00:16:53.821 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:53.819+0000 7f60e3574740 1 --2- 172.21.15.118:0/644002080 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55c8089c59e0 0x55c8089d2450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:16:53.821 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:53.819+0000 7f60e3574740 1 --2- 172.21.15.118:0/644002080 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55c8089c4810 0x55c8089d1f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:16:53.821 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:53.819+0000 7f60e3574740 1 -- 172.21.15.118:0/644002080 >> 172.21.15.118:0/644002080 conn(0x55c8089bfc00 msgr2=0x55c8089c36f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:16:53.821 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:53.820+0000 7f60e3574740 1 -- 172.21.15.118:0/644002080 shutdown_connections 2024-03-24T00:16:53.821 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:53.820+0000 7f60e3574740 1 -- 172.21.15.118:0/644002080 wait complete. 2024-03-24T00:16:53.821 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:53.820+0000 7f60e3574740 20 client.25700 trim_cache size 0 max 16384 2024-03-24T00:16:53.821 INFO:teuthology.orchestra.run.smithi118.stdout:[] 2024-03-24T00:16:53.822 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:53.821+0000 7f60e0800640 1 -- 172.21.15.118:0/4000796329 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7f60c00a2910 msgr2=0x7f60c00a4dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:16:53.822 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:53.821+0000 7f60e0800640 1 --2- 172.21.15.118:0/4000796329 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7f60c00a2910 0x7f60c00a4dd0 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 crypto rx=0x7f60cc006010 tx=0x7f60cc005dd0 comp rx=0 tx=0).stop 2024-03-24T00:16:53.822 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:53.821+0000 7f60e0800640 1 -- 172.21.15.118:0/4000796329 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f60d81017a0 msgr2=0x7f60d8102b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:16:53.822 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:53.821+0000 7f60e0800640 1 --2- 172.21.15.118:0/4000796329 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f60d81017a0 0x7f60d8102b80 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7f60c8005fd0 tx=0x7f60c800dde0 comp rx=0 tx=0).stop 2024-03-24T00:16:53.822 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:53.821+0000 7f60e0800640 1 -- 172.21.15.118:0/4000796329 shutdown_connections 2024-03-24T00:16:53.822 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:53.821+0000 7f60e0800640 1 --2- 172.21.15.118:0/4000796329 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7f60c00a2910 0x7f60c00a4dd0 unknown :-1 s=CLOSED pgs=288 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:16:53.822 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:53.821+0000 7f60e0800640 1 --2- 172.21.15.118:0/4000796329 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f60d8102160 0x7f60d81030c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:16:53.822 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:53.821+0000 7f60e0800640 1 --2- 172.21.15.118:0/4000796329 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f60d81017a0 0x7f60d8102b80 unknown :-1 s=CLOSED pgs=246 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:16:53.822 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:53.821+0000 7f60e0800640 1 --2- 172.21.15.118:0/4000796329 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f60d81005b0 0x7f60d8104540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:16:53.822 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:53.821+0000 7f60e0800640 1 -- 172.21.15.118:0/4000796329 >> 172.21.15.118:0/4000796329 conn(0x7f60d80fbb40 msgr2=0x7f60d80fd610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:16:53.823 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:53.822+0000 7f60e0800640 1 -- 172.21.15.118:0/4000796329 shutdown_connections 2024-03-24T00:16:53.823 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:53.822+0000 7f60e0800640 1 -- 172.21.15.118:0/4000796329 wait complete. 2024-03-24T00:16:53.844 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:16:53 smithi118 ceph-mon[31535]: from='client.? 172.21.15.118:0/1738454964' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-24T00:16:53.847 INFO:teuthology.orchestra.run:Running command with timeout 120 2024-03-24T00:16:53.847 DEBUG:teuthology.orchestra.run.smithi118:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:1 damage ls 2024-03-24T00:16:54.087 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:16:54.085+0000 7fa57da00640 1 -- 172.21.15.118:0/4253982378 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fa5781048a0 msgr2=0x7fa578106c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:16:54.087 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:16:54.085+0000 7fa57da00640 1 --2- 172.21.15.118:0/4253982378 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fa5781048a0 0x7fa578106c90 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7fa564009900 tx=0x7fa56402bc60 comp rx=0 tx=0).stop 2024-03-24T00:16:54.090 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:16:54.086+0000 7fa57da00640 1 -- 172.21.15.118:0/4253982378 shutdown_connections 2024-03-24T00:16:54.090 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:16:54.086+0000 7fa57da00640 1 --2- 172.21.15.118:0/4253982378 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa5781071d0 0x7fa578109620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:16:54.090 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:16:54.086+0000 7fa57da00640 1 --2- 172.21.15.118:0/4253982378 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fa5781048a0 0x7fa578106c90 unknown :-1 s=CLOSED pgs=249 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:16:54.090 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:16:54.086+0000 7fa57da00640 1 --2- 172.21.15.118:0/4253982378 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fa578101f70 0x7fa578104360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:16:54.090 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:16:54.086+0000 7fa57da00640 1 -- 172.21.15.118:0/4253982378 >> 172.21.15.118:0/4253982378 conn(0x7fa5780fb980 msgr2=0x7fa5780fdda0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:16:54.090 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:16:54.086+0000 7fa57da00640 1 -- 172.21.15.118:0/4253982378 shutdown_connections 2024-03-24T00:16:54.091 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:16:54.086+0000 7fa57da00640 1 -- 172.21.15.118:0/4253982378 wait complete. 2024-03-24T00:16:54.091 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:16:54.087+0000 7fa57da00640 1 Processor -- start 2024-03-24T00:16:54.091 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:16:54.087+0000 7fa57da00640 1 -- start start 2024-03-24T00:16:54.091 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:16:54.088+0000 7fa57da00640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fa578101f70 0x7fa578195650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:16:54.091 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:16:54.088+0000 7fa57da00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa5781048a0 0x7fa578195b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:16:54.091 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:16:54.088+0000 7fa57da00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fa5781071d0 0x7fa57819cc00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:16:54.091 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:16:54.088+0000 7fa577400640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa5781048a0 0x7fa578195b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:16:54.091 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:16:54.089+0000 7fa577400640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa5781048a0 0x7fa578195b90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.118:40452/0 (socket says 172.21.15.118:40452) 2024-03-24T00:16:54.091 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:16:54.089+0000 7fa577400640 1 -- 172.21.15.118:0/2161665191 learned_addr learned my addr 172.21.15.118:0/2161665191 (peer_addr_for_me v2:172.21.15.118:0/0) 2024-03-24T00:16:54.091 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:16:54.089+0000 7fa57d000640 1 --2- 172.21.15.118:0/2161665191 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fa5781071d0 0x7fa57819cc00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:16:54.091 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:16:54.089+0000 7fa57da00640 1 -- 172.21.15.118:0/2161665191 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa57810afc0 con 0x7fa578101f70 2024-03-24T00:16:54.091 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:16:54.089+0000 7fa57da00640 1 -- 172.21.15.118:0/2161665191 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa57810ae40 con 0x7fa5781048a0 2024-03-24T00:16:54.091 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:16:54.089+0000 7fa57da00640 1 -- 172.21.15.118:0/2161665191 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa57810b140 con 0x7fa5781071d0 2024-03-24T00:16:54.091 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:16:54.089+0000 7fa577e00640 1 --2- 172.21.15.118:0/2161665191 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fa578101f70 0x7fa578195650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:16:54.091 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:16:54.089+0000 7fa57d000640 1 -- 172.21.15.118:0/2161665191 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa5781048a0 msgr2=0x7fa578195b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:16:54.092 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:16:54.089+0000 7fa57d000640 1 --2- 172.21.15.118:0/2161665191 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa5781048a0 0x7fa578195b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:16:54.092 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:16:54.089+0000 7fa57d000640 1 -- 172.21.15.118:0/2161665191 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fa578101f70 msgr2=0x7fa578195650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:16:54.092 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:16:54.089+0000 7fa57d000640 1 --2- 172.21.15.118:0/2161665191 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fa578101f70 0x7fa578195650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:16:54.092 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:16:54.089+0000 7fa57d000640 1 -- 172.21.15.118:0/2161665191 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa57819d270 con 0x7fa5781071d0 2024-03-24T00:16:54.092 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:16:54.090+0000 7fa57d000640 1 --2- 172.21.15.118:0/2161665191 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fa5781071d0 0x7fa57819cc00 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7fa57000ec50 tx=0x7fa57000c6a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:16:54.092 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:16:54.090+0000 7fa574c00640 1 -- 172.21.15.118:0/2161665191 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa570003b70 con 0x7fa5781071d0 2024-03-24T00:16:54.093 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:16:54.090+0000 7fa574c00640 1 -- 172.21.15.118:0/2161665191 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fa570003d10 con 0x7fa5781071d0 2024-03-24T00:16:54.093 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:16:54.090+0000 7fa574c00640 1 -- 172.21.15.118:0/2161665191 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa570010930 con 0x7fa5781071d0 2024-03-24T00:16:54.093 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:16:54.090+0000 7fa57da00640 1 -- 172.21.15.118:0/2161665191 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa57819d560 con 0x7fa5781071d0 2024-03-24T00:16:54.093 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:16:54.090+0000 7fa57da00640 1 -- 172.21.15.118:0/2161665191 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa57819da60 con 0x7fa5781071d0 2024-03-24T00:16:54.096 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:16:54.095+0000 7fa581dc9740 1 Processor -- start 2024-03-24T00:16:54.096 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:16:54.095+0000 7fa574c00640 1 -- 172.21.15.118:0/2161665191 <== mon.1 v2:172.21.15.193:3300/0 4 ==== mgrmap(e 19) v1 ==== 137342+0+0 (secure 0 0 0) 0x7fa570003fc0 con 0x7fa5781071d0 2024-03-24T00:16:54.096 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:16:54.095+0000 7fa581dc9740 1 -- start start 2024-03-24T00:16:54.097 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:16:54.096+0000 7fa581dc9740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x559aab8463f0 0x559aab846810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:16:54.097 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:16:54.096+0000 7fa581dc9740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x559aab846de0 0x559aab84b8e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:16:54.097 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:16:54.096+0000 7fa581dc9740 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x559aab84bf50 0x559aab84e3e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:16:54.097 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:16:54.096+0000 7fa581dc9740 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x559aab8093b0 con 0x559aab84bf50 2024-03-24T00:16:54.097 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:16:54.096+0000 7fa581dc9740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x559aab7ab830 con 0x559aab846de0 2024-03-24T00:16:54.097 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:16:54.096+0000 7fa581dc9740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x559aab7545d0 con 0x559aab8463f0 2024-03-24T00:16:54.098 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:16:54.097+0000 7fa577e00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x559aab846de0 0x559aab84b8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:16:54.098 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:16:54.097+0000 7fa577e00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x559aab846de0 0x559aab84b8e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.118:40464/0 (socket says 172.21.15.118:40464) 2024-03-24T00:16:54.098 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:16:54.097+0000 7fa577400640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x559aab84bf50 0x559aab84e3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:16:54.098 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:16:54.097+0000 7fa577e00640 1 -- 172.21.15.118:0/972858777 learned_addr learned my addr 172.21.15.118:0/972858777 (peer_addr_for_me v2:172.21.15.118:0/0) 2024-03-24T00:16:54.098 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:16:54.097+0000 7fa577400640 1 -- 172.21.15.118:0/972858777 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x559aab8463f0 msgr2=0x559aab846810 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-03-24T00:16:54.098 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:16:54.097+0000 7fa577400640 1 --2- 172.21.15.118:0/972858777 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x559aab8463f0 0x559aab846810 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:16:54.098 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:16:54.097+0000 7fa577400640 1 -- 172.21.15.118:0/972858777 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x559aab846de0 msgr2=0x559aab84b8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:16:54.098 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:16:54.097+0000 7fa577400640 1 --2- 172.21.15.118:0/972858777 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x559aab846de0 0x559aab84b8e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:16:54.099 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:16:54.097+0000 7fa577400640 1 -- 172.21.15.118:0/972858777 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x559aab76c2c0 con 0x559aab84bf50 2024-03-24T00:16:54.099 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:16:54.098+0000 7fa577400640 1 --2- 172.21.15.118:0/972858777 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x559aab84bf50 0x559aab84e3e0 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7fa564004cf0 tx=0x7fa564004710 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a24a6039f8c0326d server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:16:54.099 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:16:54.099+0000 7fa56d600640 1 -- 172.21.15.118:0/972858777 <== mon.0 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa564004800 con 0x559aab84bf50 2024-03-24T00:16:54.099 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:16:54.099+0000 7fa56d600640 1 -- 172.21.15.118:0/972858777 <== mon.0 v2:172.21.15.118:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fa5640049a0 con 0x559aab84bf50 2024-03-24T00:16:54.099 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:16:54.099+0000 7fa56d600640 1 -- 172.21.15.118:0/972858777 <== mon.0 v2:172.21.15.118:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa564005400 con 0x559aab84bf50 2024-03-24T00:16:54.100 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:16:54.099+0000 7fa581dc9740 1 -- 172.21.15.118:0/972858777 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x559aab84bf50 msgr2=0x559aab84e3e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:16:54.100 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:16:54.100+0000 7fa581dc9740 1 --2- 172.21.15.118:0/972858777 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x559aab84bf50 0x559aab84e3e0 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7fa564004cf0 tx=0x7fa564004710 comp rx=0 tx=0).stop 2024-03-24T00:16:54.100 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:16:54.100+0000 7fa574c00640 1 --2- 172.21.15.118:0/2161665191 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7fa55c0a2aa0 0x7fa55c0a4f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:16:54.101 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:16:54.100+0000 7fa574c00640 1 -- 172.21.15.118:0/2161665191 <== mon.1 v2:172.21.15.193:3300/0 5 ==== osd_map(90..90 src has 1..90) v4 ==== 8277+0+0 (secure 0 0 0) 0x7fa5700cf130 con 0x7fa5781071d0 2024-03-24T00:16:54.101 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:16:54.101+0000 7fa577400640 1 --2- 172.21.15.118:0/2161665191 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7fa55c0a2aa0 0x7fa55c0a4f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:16:54.102 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:16:54.101+0000 7fa581dc9740 1 -- 172.21.15.118:0/972858777 shutdown_connections 2024-03-24T00:16:54.102 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:16:54.101+0000 7fa581dc9740 1 --2- 172.21.15.118:0/972858777 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x559aab84bf50 0x559aab84e3e0 unknown :-1 s=CLOSED pgs=185 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:16:54.102 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:16:54.101+0000 7fa581dc9740 1 --2- 172.21.15.118:0/972858777 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x559aab846de0 0x559aab84b8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:16:54.102 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:16:54.101+0000 7fa581dc9740 1 --2- 172.21.15.118:0/972858777 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x559aab8463f0 0x559aab846810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:16:54.102 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:16:54.101+0000 7fa581dc9740 1 -- 172.21.15.118:0/972858777 >> 172.21.15.118:0/972858777 conn(0x559aab841960 msgr2=0x559aab841d60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:16:54.103 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:16:54.102+0000 7fa577400640 1 --2- 172.21.15.118:0/2161665191 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7fa55c0a2aa0 0x7fa55c0a4f60 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 crypto rx=0x7fa564001eb0 tx=0x7fa564000bc0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:16:54.103 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:16:54.102+0000 7fa581dc9740 1 -- 172.21.15.118:0/972858777 shutdown_connections 2024-03-24T00:16:54.103 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:16:54.102+0000 7fa581dc9740 1 -- 172.21.15.118:0/972858777 wait complete. 2024-03-24T00:16:54.103 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:16:54.102+0000 7fa581dc9740 1 -- start start 2024-03-24T00:16:54.103 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:16:54.103+0000 7fa581dc9740 1 Processor -- start 2024-03-24T00:16:54.104 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:16:54.103+0000 7fa581dc9740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x559aab8463f0 0x559aab8505c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:16:54.105 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:16:54.104+0000 7fa577e00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x559aab8463f0 0x559aab8505c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:16:54.105 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:16:54.104+0000 7fa577e00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x559aab8463f0 0x559aab8505c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.118:36294/0 (socket says 172.21.15.118:36294) 2024-03-24T00:16:54.105 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:16:54.105+0000 7fa577e00640 1 -- 172.21.15.118:0/2435547841 learned_addr learned my addr 172.21.15.118:0/2435547841 (peer_addr_for_me v2:172.21.15.118:0/0) 2024-03-24T00:16:54.106 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:16:54.105+0000 7fa581dc9740 1 --2- 172.21.15.118:0/2435547841 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x559aab846de0 0x559aab850b00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:16:54.107 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:16:54.106+0000 7fa581dc9740 1 --2- 172.21.15.118:0/2435547841 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x559aab84bf50 0x559aab857ad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:16:54.107 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:16:54.106+0000 7fa581dc9740 1 -- 172.21.15.118:0/2435547841 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x559aab767a20 con 0x559aab84bf50 2024-03-24T00:16:54.107 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:16:54.106+0000 7fa581dc9740 1 -- 172.21.15.118:0/2435547841 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x559aab7a78d0 con 0x559aab846de0 2024-03-24T00:16:54.108 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:16:54.106+0000 7fa581dc9740 1 -- 172.21.15.118:0/2435547841 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x559aab84eea0 con 0x559aab8463f0 2024-03-24T00:16:54.108 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:16:54.106+0000 7fa581dc9740 10 client.0 _ll_register_callbacks cb 0x559aab7a41c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T00:16:54.108 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:16:54.106+0000 7fa577400640 1 --2- 172.21.15.118:0/2435547841 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x559aab846de0 0x559aab850b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:16:54.108 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:16:54.106+0000 7fa57d000640 1 --2- 172.21.15.118:0/2435547841 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x559aab84bf50 0x559aab857ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:16:54.108 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:16:54.107+0000 7fa577e00640 1 -- 172.21.15.118:0/2435547841 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x559aab846de0 msgr2=0x559aab850b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:16:54.108 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:16:54.107+0000 7fa577e00640 1 --2- 172.21.15.118:0/2435547841 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x559aab846de0 0x559aab850b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:16:54.108 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:16:54.107+0000 7fa577e00640 1 -- 172.21.15.118:0/2435547841 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x559aab84bf50 msgr2=0x559aab857ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:16:54.108 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:16:54.107+0000 7fa577e00640 1 --2- 172.21.15.118:0/2435547841 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x559aab84bf50 0x559aab857ad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:16:54.108 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:16:54.107+0000 7fa577e00640 1 -- 172.21.15.118:0/2435547841 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x559aab4c0210 con 0x559aab8463f0 2024-03-24T00:16:54.108 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:16:54.107+0000 7fa577400640 1 --2- 172.21.15.118:0/2435547841 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x559aab846de0 0x559aab850b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T00:16:54.108 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:16:54.107+0000 7fa57d000640 1 --2- 172.21.15.118:0/2435547841 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x559aab84bf50 0x559aab857ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T00:16:54.109 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:16:54.107+0000 7fa577e00640 1 --2- 172.21.15.118:0/2435547841 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x559aab8463f0 0x559aab8505c0 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7fa56800bef0 tx=0x7fa568012820 comp rx=0 tx=0).ready entity=mon.1 client_cookie=789fb15795eacdc4 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:16:54.109 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:16:54.107+0000 7fa562000640 10 client.0 ms_handle_connect on v2:172.21.15.193:3300/0 2024-03-24T00:16:54.109 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:16:54.107+0000 7fa562000640 1 -- 172.21.15.118:0/2435547841 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa56801f060 con 0x559aab8463f0 2024-03-24T00:16:54.109 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:16:54.107+0000 7fa562000640 1 -- 172.21.15.118:0/2435547841 <== mon.1 v2:172.21.15.193:3300/0 2 ==== osd_map(90..90 src has 1..90) v4 ==== 8277+0+0 (secure 0 0 0) 0x7fa568010960 con 0x559aab8463f0 2024-03-24T00:16:54.110 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:16:54.108+0000 7fa581dc9740 1 -- 172.21.15.118:0/2435547841 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fa5640049a0 con 0x559aab8463f0 2024-03-24T00:16:54.110 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:54.109+0000 7f765ba00640 1 -- 172.21.15.118:0/3713433094 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f765410d9c0 msgr2=0x7f765410de40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:16:54.111 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:54.109+0000 7f765ba00640 1 --2- 172.21.15.118:0/3713433094 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f765410d9c0 0x7f765410de40 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7f764c009be0 tx=0x7f764c02bef0 comp rx=0 tx=0).stop 2024-03-24T00:16:54.111 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:54.109+0000 7f765ba00640 1 -- 172.21.15.118:0/3713433094 shutdown_connections 2024-03-24T00:16:54.111 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:54.109+0000 7f765ba00640 1 --2- 172.21.15.118:0/3713433094 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f765410d9c0 0x7f765410de40 unknown :-1 s=CLOSED pgs=260 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:16:54.111 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:54.109+0000 7f765ba00640 1 --2- 172.21.15.118:0/3713433094 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f765410b010 0x7f765410d480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:16:54.111 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:54.109+0000 7f765ba00640 1 --2- 172.21.15.118:0/3713433094 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f76541086a0 0x7f765410aad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:16:54.111 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:54.109+0000 7f765ba00640 1 -- 172.21.15.118:0/3713433094 >> 172.21.15.118:0/3713433094 conn(0x7f765406e500 msgr2=0x7f7654070940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:16:54.111 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:16:54.109+0000 7fa562000640 1 -- 172.21.15.118:0/2435547841 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) v2 ==== 24+0+0 (secure 0 0 0) 0x7fa568010b60 con 0x559aab8463f0 2024-03-24T00:16:54.111 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:54.109+0000 7f765ba00640 1 -- 172.21.15.118:0/3713433094 shutdown_connections 2024-03-24T00:16:54.112 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:16:54.109+0000 7fa581dc9740 10 client.25705 fetch_fsmap learned FSMap version 28 2024-03-24T00:16:54.112 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:16:54.109+0000 7fa581dc9740 1 -- 172.21.15.118:0/2435547841 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) v3 -- 0x559aab58c620 con 0x559aab8463f0 2024-03-24T00:16:54.112 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:54.109+0000 7f765ba00640 1 -- 172.21.15.118:0/3713433094 wait complete. 2024-03-24T00:16:54.113 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:16:54.110+0000 7fa562000640 1 -- 172.21.15.118:0/2435547841 <== mon.1 v2:172.21.15.193:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fa5640049a0 con 0x559aab8463f0 2024-03-24T00:16:54.113 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:16:54.110+0000 7fa562000640 1 -- 172.21.15.118:0/2435547841 <== mon.1 v2:172.21.15.193:3300/0 5 ==== fsmap(e 28) v1 ==== 7934+0+0 (secure 0 0 0) 0x7fa568019030 con 0x559aab8463f0 2024-03-24T00:16:54.113 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:16:54.110+0000 7fa581dc9740 10 client.25705 fetch_fsmap finished waiting for FSMap version 28 2024-03-24T00:16:54.113 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:16:54.110+0000 7fa581dc9740 10 client.25705 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-24T00:16:54.113 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:16:54.110+0000 7fa581dc9740 20 client.25705 populate_metadata read hostname 'smithi118' 2024-03-24T00:16:54.113 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:16:54.110+0000 7fa581dc9740 1 --2- 172.21.15.118:0/2435547841 >> [v2:172.21.15.189:6838/4074484168,v1:172.21.15.189:6839/4074484168] conn(0x559aab85c510 0x559aab85e990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:16:54.113 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:16:54.110+0000 7fa581dc9740 4 client.25705 mds_command: new command op to 24569 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-24T00:16:54.113 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:16:54.110+0000 7fa581dc9740 1 -- 172.21.15.118:0/2435547841 --> [v2:172.21.15.189:6838/4074484168,v1:172.21.15.189:6839/4074484168] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x559aab7b8d70 con 0x559aab85c510 2024-03-24T00:16:54.113 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:16:54.111+0000 7fa577400640 1 --2- 172.21.15.118:0/2435547841 >> [v2:172.21.15.189:6838/4074484168,v1:172.21.15.189:6839/4074484168] conn(0x559aab85c510 0x559aab85e990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:16:54.114 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:54.112+0000 7f765ba00640 1 Processor -- start 2024-03-24T00:16:54.114 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:54.112+0000 7f765ba00640 1 -- start start 2024-03-24T00:16:54.114 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:16:54.112+0000 7fa577400640 1 --2- 172.21.15.118:0/2435547841 >> [v2:172.21.15.189:6838/4074484168,v1:172.21.15.189:6839/4074484168] conn(0x559aab85c510 0x559aab85e990 crc :-1 s=READY pgs=55 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=4d0cf25be5c4783a server_cookie=806475e885646b3 in_seq=0 out_seq=0 2024-03-24T00:16:54.114 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:16:54.112+0000 7fa562000640 10 client.25705 ms_handle_connect on v2:172.21.15.189:6838/4074484168 2024-03-24T00:16:54.115 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:16:54.113+0000 7fa562000640 1 -- 172.21.15.118:0/2435547841 <== mds.0 v2:172.21.15.189:6838/4074484168 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x559aab7b8d70 con 0x559aab85c510 2024-03-24T00:16:54.115 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:16:54.113+0000 7fa562000640 10 client.25705 handle_command_reply: tid=0 2024-03-24T00:16:54.115 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:54.113+0000 7f765ba00640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f76541086a0 0x7f7654116d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:16:54.116 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:16:54.113+0000 7fa581dc9740 1 client.25705 shutdown 2024-03-24T00:16:54.116 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:16:54.113+0000 7fa581dc9740 1 -- 172.21.15.118:0/2435547841 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x559aab8463f0 msgr2=0x559aab8505c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:16:54.116 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:16:54.113+0000 7fa581dc9740 1 --2- 172.21.15.118:0/2435547841 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x559aab8463f0 0x559aab8505c0 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7fa56800bef0 tx=0x7fa568012820 comp rx=0 tx=0).stop 2024-03-24T00:16:54.116 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:16:54.113+0000 7fa581dc9740 1 -- 172.21.15.118:0/2435547841 shutdown_connections 2024-03-24T00:16:54.116 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:16:54.113+0000 7fa581dc9740 1 --2- 172.21.15.118:0/2435547841 >> [v2:172.21.15.189:6838/4074484168,v1:172.21.15.189:6839/4074484168] conn(0x559aab85c510 0x559aab85e990 crc :-1 s=READY pgs=55 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:16:54.116 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:54.113+0000 7f765ba00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f765410b010 0x7f7654117290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:16:54.117 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:16:54.113+0000 7fa581dc9740 1 --2- 172.21.15.118:0/2435547841 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x559aab84bf50 0x559aab857ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:16:54.117 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:16:54.113+0000 7fa581dc9740 1 --2- 172.21.15.118:0/2435547841 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x559aab846de0 0x559aab850b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:16:54.117 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:16:54.113+0000 7fa581dc9740 1 --2- 172.21.15.118:0/2435547841 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x559aab8463f0 0x559aab8505c0 unknown :-1 s=CLOSED pgs=251 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:16:54.117 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:16:54.113+0000 7fa581dc9740 1 -- 172.21.15.118:0/2435547841 >> 172.21.15.118:0/2435547841 conn(0x559aab841960 msgr2=0x559aab840b70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:16:54.117 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:16:54.114+0000 7fa562000640 0 client.25705 ms_handle_reset on v2:172.21.15.189:6838/4074484168 2024-03-24T00:16:54.117 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:54.114+0000 7f765ba00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f765410d9c0 0x7f76541b6800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:16:54.118 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:54.114+0000 7f7659000640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f76541086a0 0x7f7654116d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:16:54.118 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:54.114+0000 7f7659000640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f76541086a0 0x7f7654116d50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.118:3300/0 says I am v2:172.21.15.118:50562/0 (socket says 172.21.15.118:50562) 2024-03-24T00:16:54.118 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:54.114+0000 7f7659000640 1 -- 172.21.15.118:0/2951298035 learned_addr learned my addr 172.21.15.118:0/2951298035 (peer_addr_for_me v2:172.21.15.118:0/0) 2024-03-24T00:16:54.118 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:54.114+0000 7f7653e00640 1 --2- 172.21.15.118:0/2951298035 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f765410b010 0x7f7654117290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:16:54.118 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:16:54.114+0000 7fa581dc9740 1 -- 172.21.15.118:0/2435547841 shutdown_connections 2024-03-24T00:16:54.118 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:54.114+0000 7f765ba00640 1 -- 172.21.15.118:0/2951298035 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f765411c150 con 0x7f76541086a0 2024-03-24T00:16:54.119 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:16:54.114+0000 7fa581dc9740 1 -- 172.21.15.118:0/2435547841 wait complete. 2024-03-24T00:16:54.119 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:54.114+0000 7f765ba00640 1 -- 172.21.15.118:0/2951298035 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f765411bfd0 con 0x7f765410b010 2024-03-24T00:16:54.119 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:16:54.114+0000 7fa581dc9740 20 client.25705 trim_cache size 0 max 16384 2024-03-24T00:16:54.120 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:54.114+0000 7f765ba00640 1 -- 172.21.15.118:0/2951298035 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f765411c2d0 con 0x7f765410d9c0 2024-03-24T00:16:54.120 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:54.114+0000 7f7659a00640 1 --2- 172.21.15.118:0/2951298035 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f765410d9c0 0x7f76541b6800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:16:54.120 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:54.115+0000 7f7659a00640 1 -- 172.21.15.118:0/2951298035 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f765410b010 msgr2=0x7f7654117290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:16:54.120 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:54.115+0000 7f7659a00640 1 --2- 172.21.15.118:0/2951298035 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f765410b010 0x7f7654117290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:16:54.121 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:54.115+0000 7f7659a00640 1 -- 172.21.15.118:0/2951298035 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f76541086a0 msgr2=0x7f7654116d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:16:54.121 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:54.115+0000 7f7659a00640 1 --2- 172.21.15.118:0/2951298035 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f76541086a0 0x7f7654116d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:16:54.121 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:54.115+0000 7f7659a00640 1 -- 172.21.15.118:0/2951298035 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f76541b6d40 con 0x7f765410d9c0 2024-03-24T00:16:54.121 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:54.115+0000 7f7659000640 1 --2- 172.21.15.118:0/2951298035 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f76541086a0 0x7f7654116d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T00:16:54.121 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:54.116+0000 7f7659a00640 1 --2- 172.21.15.118:0/2951298035 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f765410d9c0 0x7f76541b6800 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7f764c036040 tx=0x7f764c0337b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:16:54.121 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:54.116+0000 7f7651600640 1 -- 172.21.15.118:0/2951298035 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f764c043020 con 0x7f765410d9c0 2024-03-24T00:16:54.121 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:54.116+0000 7f765ba00640 1 -- 172.21.15.118:0/2951298035 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f76541b6f10 con 0x7f765410d9c0 2024-03-24T00:16:54.121 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:54.117+0000 7f765ba00640 1 -- 172.21.15.118:0/2951298035 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f76541b7310 con 0x7f765410d9c0 2024-03-24T00:16:54.121 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:54.117+0000 7f7651600640 1 -- 172.21.15.118:0/2951298035 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f764c0338a0 con 0x7f765410d9c0 2024-03-24T00:16:54.121 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:54.117+0000 7f7651600640 1 -- 172.21.15.118:0/2951298035 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f764c034690 con 0x7f765410d9c0 2024-03-24T00:16:54.122 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:54.121+0000 7f765e61c740 1 Processor -- start 2024-03-24T00:16:54.122 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:54.122+0000 7f7651600640 1 -- 172.21.15.118:0/2951298035 <== mon.1 v2:172.21.15.193:3300/0 4 ==== mgrmap(e 19) v1 ==== 137342+0+0 (secure 0 0 0) 0x7f764c046020 con 0x7f765410d9c0 2024-03-24T00:16:54.123 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:54.122+0000 7f765e61c740 1 -- start start 2024-03-24T00:16:54.124 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:54.123+0000 7f765e61c740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55f1082d2810 0x55f1082d2c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:16:54.124 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:54.123+0000 7f765e61c740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55f1082d39e0 0x55f1082d3e60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:16:54.124 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:54.123+0000 7f765e61c740 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55f1082d43a0 0x55f1082dac30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:16:54.124 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:54.123+0000 7f765e61c740 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f1082372c0 con 0x55f1082d43a0 2024-03-24T00:16:54.125 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:54.123+0000 7f765e61c740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f1081dfc90 con 0x55f1082d2810 2024-03-24T00:16:54.125 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:54.123+0000 7f765e61c740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f108233360 con 0x55f1082d39e0 2024-03-24T00:16:54.125 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:54.124+0000 7f7651600640 1 --2- 172.21.15.118:0/2951298035 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7f76380a2950 0x7f76380a4e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:16:54.125 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:54.124+0000 7f7651600640 1 -- 172.21.15.118:0/2951298035 <== mon.1 v2:172.21.15.193:3300/0 5 ==== osd_map(90..90 src has 1..90) v4 ==== 8277+0+0 (secure 0 0 0) 0x7f764c0f1000 con 0x7f765410d9c0 2024-03-24T00:16:54.125 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:54.124+0000 7f7653e00640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55f1082d43a0 0x55f1082dac30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:16:54.126 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:54.125+0000 7f7653e00640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55f1082d43a0 0x55f1082dac30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.118:3300/0 says I am v2:172.21.15.118:50572/0 (socket says 172.21.15.118:50572) 2024-03-24T00:16:54.126 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:54.125+0000 7f7653e00640 1 -- 172.21.15.118:0/3014625454 learned_addr learned my addr 172.21.15.118:0/3014625454 (peer_addr_for_me v2:172.21.15.118:0/0) 2024-03-24T00:16:54.126 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:54.125+0000 7f7659a00640 1 --2- 172.21.15.118:0/2951298035 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7f76380a2950 0x7f76380a4e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:16:54.126 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:54.125+0000 7f7659000640 1 --2- 172.21.15.118:0/3014625454 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55f1082d39e0 0x55f1082d3e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:16:54.126 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:54.126+0000 7f7653e00640 1 -- 172.21.15.118:0/3014625454 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55f1082d39e0 msgr2=0x55f1082d3e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:16:54.127 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:54.126+0000 7f7653e00640 1 --2- 172.21.15.118:0/3014625454 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55f1082d39e0 0x55f1082d3e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:16:54.127 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:54.126+0000 7f7653e00640 1 -- 172.21.15.118:0/3014625454 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55f1082d2810 msgr2=0x55f1082d2c30 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-03-24T00:16:54.127 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:54.126+0000 7f7653e00640 1 --2- 172.21.15.118:0/3014625454 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55f1082d2810 0x55f1082d2c30 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:16:54.127 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:54.126+0000 7f7653e00640 1 -- 172.21.15.118:0/3014625454 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55f108204080 con 0x55f1082d43a0 2024-03-24T00:16:54.127 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:54.126+0000 7f7653e00640 1 --2- 172.21.15.118:0/3014625454 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55f1082d43a0 0x55f1082dac30 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7f7640009c50 tx=0x7f7640011810 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2d8b2b56e168be2a server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:16:54.127 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:54.127+0000 7f7659a00640 1 --2- 172.21.15.118:0/2951298035 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7f76380a2950 0x7f76380a4e10 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x7f764c0b9e20 tx=0x7f764c02bef0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:16:54.128 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:54.127+0000 7f764a000640 1 -- 172.21.15.118:0/3014625454 <== mon.0 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f764001e070 con 0x55f1082d43a0 2024-03-24T00:16:54.128 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:54.127+0000 7f764a000640 1 -- 172.21.15.118:0/3014625454 <== mon.0 v2:172.21.15.118:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f76400035f0 con 0x55f1082d43a0 2024-03-24T00:16:54.128 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:54.127+0000 7f764a000640 1 -- 172.21.15.118:0/3014625454 <== mon.0 v2:172.21.15.118:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7640019690 con 0x55f1082d43a0 2024-03-24T00:16:54.128 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:54.128+0000 7f765e61c740 1 -- 172.21.15.118:0/3014625454 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55f1082d43a0 msgr2=0x55f1082dac30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:16:54.128 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:54.128+0000 7f765e61c740 1 --2- 172.21.15.118:0/3014625454 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55f1082d43a0 0x55f1082dac30 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7f7640009c50 tx=0x7f7640011810 comp rx=0 tx=0).stop 2024-03-24T00:16:54.129 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:54.128+0000 7f765e61c740 1 -- 172.21.15.118:0/3014625454 shutdown_connections 2024-03-24T00:16:54.129 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:54.128+0000 7f765e61c740 1 --2- 172.21.15.118:0/3014625454 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55f1082d43a0 0x55f1082dac30 unknown :-1 s=CLOSED pgs=186 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:16:54.129 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:54.128+0000 7f765e61c740 1 --2- 172.21.15.118:0/3014625454 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55f1082d39e0 0x55f1082d3e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:16:54.129 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:54.128+0000 7f765e61c740 1 --2- 172.21.15.118:0/3014625454 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55f1082d2810 0x55f1082d2c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:16:54.129 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:54.128+0000 7f765e61c740 1 -- 172.21.15.118:0/3014625454 >> 172.21.15.118:0/3014625454 conn(0x55f1082cdc00 msgr2=0x55f1082ce000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:16:54.129 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:54.128+0000 7f765e61c740 1 -- 172.21.15.118:0/3014625454 shutdown_connections 2024-03-24T00:16:54.129 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:54.128+0000 7f765e61c740 1 -- 172.21.15.118:0/3014625454 wait complete. 2024-03-24T00:16:54.130 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:54.129+0000 7f765e61c740 1 -- start start 2024-03-24T00:16:54.130 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:54.129+0000 7f765e61c740 1 Processor -- start 2024-03-24T00:16:54.131 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:54.130+0000 7f765e61c740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55f1082d2810 0x55f1082dcb00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:16:54.131 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:54.130+0000 7f765e61c740 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55f1082d39e0 0x55f1082dd040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:16:54.131 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:54.130+0000 7f765e61c740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55f1082d43a0 0x55f1082e4080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:16:54.131 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:54.130+0000 7f765e61c740 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f1082db450 con 0x55f1082d39e0 2024-03-24T00:16:54.131 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:54.130+0000 7f765e61c740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f1081f30d0 con 0x55f1082d2810 2024-03-24T00:16:54.131 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:54.130+0000 7f765e61c740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f1082db5d0 con 0x55f1082d43a0 2024-03-24T00:16:54.131 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:54.130+0000 7f765e61c740 10 client.0 _ll_register_callbacks cb 0x55f1081f7e80 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T00:16:54.132 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:54.131+0000 7f7653e00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55f1082d43a0 0x55f1082e4080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:16:54.132 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:54.131+0000 7f7653e00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55f1082d43a0 0x55f1082e4080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.118:36306/0 (socket says 172.21.15.118:36306) 2024-03-24T00:16:54.132 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:54.131+0000 7f7653e00640 1 -- 172.21.15.118:0/333632579 learned_addr learned my addr 172.21.15.118:0/333632579 (peer_addr_for_me v2:172.21.15.118:0/0) 2024-03-24T00:16:54.132 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:54.131+0000 7f7653e00640 1 --2- 172.21.15.118:0/333632579 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55f1082d2810 0x55f1082dcb00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:16:54.132 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:54.131+0000 7f7659000640 1 --2- 172.21.15.118:0/333632579 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55f1082d39e0 0x55f1082dd040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:16:54.132 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:54.131+0000 7f7653e00640 1 -- 172.21.15.118:0/333632579 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55f1082d2810 msgr2=0x55f1082dcb00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:16:54.132 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:54.131+0000 7f7653e00640 1 --2- 172.21.15.118:0/333632579 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55f1082d2810 0x55f1082dcb00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:16:54.132 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:54.131+0000 7f7653e00640 1 -- 172.21.15.118:0/333632579 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55f1082d39e0 msgr2=0x55f1082dd040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:16:54.133 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:54.131+0000 7f7653e00640 1 --2- 172.21.15.118:0/333632579 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55f1082d39e0 0x55f1082dd040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:16:54.133 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:54.131+0000 7f7653e00640 1 -- 172.21.15.118:0/333632579 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55f107f4b3f0 con 0x55f1082d43a0 2024-03-24T00:16:54.134 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:54.132+0000 7f7653e00640 1 --2- 172.21.15.118:0/333632579 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55f1082d43a0 0x55f1082e4080 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7f7640009690 tx=0x7f7640005fd0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=bd9882c3d902e86 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:16:54.134 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:54.132+0000 7f763ea00640 10 client.0 ms_handle_connect on v2:172.21.15.193:3300/0 2024-03-24T00:16:54.135 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:54.132+0000 7f763ea00640 1 -- 172.21.15.118:0/333632579 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f764001e070 con 0x55f1082d43a0 2024-03-24T00:16:54.135 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:54.132+0000 7f763ea00640 1 -- 172.21.15.118:0/333632579 <== mon.1 v2:172.21.15.193:3300/0 2 ==== osd_map(90..90 src has 1..90) v4 ==== 8277+0+0 (secure 0 0 0) 0x7f7640005250 con 0x55f1082d43a0 2024-03-24T00:16:54.135 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:54.132+0000 7f765e61c740 1 -- 172.21.15.118:0/333632579 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f76400035f0 con 0x55f1082d43a0 2024-03-24T00:16:54.135 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:54.133+0000 7f763ea00640 1 -- 172.21.15.118:0/333632579 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) v2 ==== 24+0+0 (secure 0 0 0) 0x7f7640005490 con 0x55f1082d43a0 2024-03-24T00:16:54.136 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:54.135+0000 7f765e61c740 10 client.25714 fetch_fsmap learned FSMap version 28 2024-03-24T00:16:54.136 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:54.135+0000 7f765e61c740 1 -- 172.21.15.118:0/333632579 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) v3 -- 0x55f1080157d0 con 0x55f1082d43a0 2024-03-24T00:16:54.136 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:54.135+0000 7f763ea00640 1 -- 172.21.15.118:0/333632579 <== mon.1 v2:172.21.15.193:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f76400035f0 con 0x55f1082d43a0 2024-03-24T00:16:54.137 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:54.135+0000 7f763ea00640 1 -- 172.21.15.118:0/333632579 <== mon.1 v2:172.21.15.193:3300/0 5 ==== fsmap(e 28) v1 ==== 7934+0+0 (secure 0 0 0) 0x7f7640033450 con 0x55f1082d43a0 2024-03-24T00:16:54.137 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:54.135+0000 7f765e61c740 10 client.25714 fetch_fsmap finished waiting for FSMap version 28 2024-03-24T00:16:54.137 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:54.136+0000 7f765e61c740 10 client.25714 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.g 2024-03-24T00:16:54.137 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:54.136+0000 7f765e61c740 20 client.25714 populate_metadata read hostname 'smithi118' 2024-03-24T00:16:54.138 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:54.136+0000 7f765e61c740 1 --2- 172.21.15.118:0/333632579 >> [v2:172.21.15.118:6838/3285788503,v1:172.21.15.118:6839/3285788503] conn(0x55f1082e8290 0x55f1082ea680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:16:54.138 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:54.136+0000 7f765e61c740 4 client.25714 mds_command: new command op to 24550 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-24T00:16:54.138 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:54.136+0000 7f765e61c740 1 -- 172.21.15.118:0/333632579 --> [v2:172.21.15.118:6838/3285788503,v1:172.21.15.118:6839/3285788503] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55f108244800 con 0x55f1082e8290 2024-03-24T00:16:54.138 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:54.136+0000 7f7659000640 1 --2- 172.21.15.118:0/333632579 >> [v2:172.21.15.118:6838/3285788503,v1:172.21.15.118:6839/3285788503] conn(0x55f1082e8290 0x55f1082ea680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:16:54.138 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:54.137+0000 7f7659000640 1 --2- 172.21.15.118:0/333632579 >> [v2:172.21.15.118:6838/3285788503,v1:172.21.15.118:6839/3285788503] conn(0x55f1082e8290 0x55f1082ea680 crc :-1 s=READY pgs=24 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=41d27d8bbd17f6f0 server_cookie=fa75b51521b42bc3 in_seq=0 out_seq=0 2024-03-24T00:16:54.138 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:54.137+0000 7f763ea00640 10 client.25714 ms_handle_connect on v2:172.21.15.118:6838/3285788503 2024-03-24T00:16:54.139 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:54.139+0000 7f763ea00640 1 -- 172.21.15.118:0/333632579 <== mds.1 v2:172.21.15.118:6838/3285788503 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x55f108244800 con 0x55f1082e8290 2024-03-24T00:16:54.139 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:54.139+0000 7f763ea00640 10 client.25714 handle_command_reply: tid=0 2024-03-24T00:16:54.139 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:54.139+0000 7f765e61c740 1 client.25714 shutdown 2024-03-24T00:16:54.140 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:54.139+0000 7f765e61c740 1 -- 172.21.15.118:0/333632579 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55f1082d43a0 msgr2=0x55f1082e4080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:16:54.140 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:54.139+0000 7f765e61c740 1 --2- 172.21.15.118:0/333632579 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55f1082d43a0 0x55f1082e4080 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7f7640009690 tx=0x7f7640005fd0 comp rx=0 tx=0).stop 2024-03-24T00:16:54.140 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:54.139+0000 7f765e61c740 1 -- 172.21.15.118:0/333632579 shutdown_connections 2024-03-24T00:16:54.140 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:54.140+0000 7f765e61c740 1 --2- 172.21.15.118:0/333632579 >> [v2:172.21.15.118:6838/3285788503,v1:172.21.15.118:6839/3285788503] conn(0x55f1082e8290 0x55f1082ea680 crc :-1 s=READY pgs=24 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:16:54.140 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:54.140+0000 7f765e61c740 1 --2- 172.21.15.118:0/333632579 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55f1082d43a0 0x55f1082e4080 unknown :-1 s=CLOSED pgs=253 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:16:54.140 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:54.140+0000 7f765e61c740 1 --2- 172.21.15.118:0/333632579 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55f1082d39e0 0x55f1082dd040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:16:54.141 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:54.140+0000 7f765e61c740 1 --2- 172.21.15.118:0/333632579 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55f1082d2810 0x55f1082dcb00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:16:54.141 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:54.140+0000 7f765e61c740 1 -- 172.21.15.118:0/333632579 >> 172.21.15.118:0/333632579 conn(0x55f1082cdc00 msgr2=0x55f1082cd000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:16:54.141 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:54.140+0000 7f763ea00640 0 client.25714 ms_handle_reset on v2:172.21.15.118:6838/3285788503 2024-03-24T00:16:54.141 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:54.140+0000 7f765e61c740 1 -- 172.21.15.118:0/333632579 shutdown_connections 2024-03-24T00:16:54.141 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:54.140+0000 7f765e61c740 1 -- 172.21.15.118:0/333632579 wait complete. 2024-03-24T00:16:54.142 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:54.140+0000 7f765e61c740 20 client.25714 trim_cache size 0 max 16384 2024-03-24T00:16:54.144 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:16:54.143+0000 7fa581dc9740 1 Processor -- start 2024-03-24T00:16:54.144 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:16:54.143+0000 7fa581dc9740 1 -- start start 2024-03-24T00:16:54.145 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:16:54.144+0000 7fa581dc9740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x559aab8463f0 0x559aab841d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:16:54.145 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:16:54.144+0000 7fa581dc9740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x559aab846de0 0x559aab844bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:16:54.145 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:16:54.144+0000 7fa581dc9740 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x559aab84bf50 0x559aab845100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:16:54.145 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:16:54.144+0000 7fa581dc9740 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa5781a6580 con 0x559aab84bf50 2024-03-24T00:16:54.145 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:16:54.144+0000 7fa581dc9740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa5781a7f00 con 0x559aab846de0 2024-03-24T00:16:54.145 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:16:54.144+0000 7fa581dc9740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa5781a6be0 con 0x559aab8463f0 2024-03-24T00:16:54.146 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:16:54.145+0000 7fa577e00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x559aab8463f0 0x559aab841d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:16:54.146 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:16:54.145+0000 7fa577e00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x559aab8463f0 0x559aab841d40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.118:36314/0 (socket says 172.21.15.118:36314) 2024-03-24T00:16:54.147 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:16:54.145+0000 7fa577e00640 1 -- 172.21.15.118:0/3749927435 learned_addr learned my addr 172.21.15.118:0/3749927435 (peer_addr_for_me v2:172.21.15.118:0/0) 2024-03-24T00:16:54.147 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:16:54.145+0000 7fa57d000640 1 --2- 172.21.15.118:0/3749927435 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x559aab84bf50 0x559aab845100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:16:54.147 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:16:54.145+0000 7fa577e00640 1 -- 172.21.15.118:0/3749927435 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x559aab846de0 msgr2=0x559aab844bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:16:54.147 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:16:54.145+0000 7fa577400640 1 --2- 172.21.15.118:0/3749927435 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x559aab846de0 0x559aab844bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:16:54.147 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:16:54.145+0000 7fa577e00640 1 --2- 172.21.15.118:0/3749927435 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x559aab846de0 0x559aab844bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:16:54.147 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:16:54.145+0000 7fa577e00640 1 -- 172.21.15.118:0/3749927435 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x559aab84bf50 msgr2=0x559aab845100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:16:54.147 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:16:54.145+0000 7fa577e00640 1 --2- 172.21.15.118:0/3749927435 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x559aab84bf50 0x559aab845100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:16:54.147 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:16:54.145+0000 7fa577e00640 1 -- 172.21.15.118:0/3749927435 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x559aab8471c0 con 0x559aab8463f0 2024-03-24T00:16:54.147 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:16:54.145+0000 7fa577400640 1 --2- 172.21.15.118:0/3749927435 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x559aab846de0 0x559aab844bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-24T00:16:54.147 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:16:54.145+0000 7fa57d000640 1 --2- 172.21.15.118:0/3749927435 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x559aab84bf50 0x559aab845100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-24T00:16:54.147 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:16:54.146+0000 7fa577e00640 1 --2- 172.21.15.118:0/3749927435 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x559aab8463f0 0x559aab841d40 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7fa568000c00 tx=0x7fa568005f30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=144e96c415de3723 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:16:54.149 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:16:54.146+0000 7fa56cc00640 1 -- 172.21.15.118:0/3749927435 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa568031070 con 0x559aab8463f0 2024-03-24T00:16:54.149 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:16:54.146+0000 7fa56cc00640 1 -- 172.21.15.118:0/3749927435 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fa568005510 con 0x559aab8463f0 2024-03-24T00:16:54.149 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:16:54.146+0000 7fa56cc00640 1 -- 172.21.15.118:0/3749927435 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa568004b40 con 0x559aab8463f0 2024-03-24T00:16:54.149 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:16:54.147+0000 7fa581dc9740 1 -- 172.21.15.118:0/3749927435 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x559aab8463f0 msgr2=0x559aab841d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:16:54.149 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:16:54.147+0000 7fa581dc9740 1 --2- 172.21.15.118:0/3749927435 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x559aab8463f0 0x559aab841d40 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7fa568000c00 tx=0x7fa568005f30 comp rx=0 tx=0).stop 2024-03-24T00:16:54.149 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:16:54.148+0000 7fa581dc9740 1 -- 172.21.15.118:0/3749927435 shutdown_connections 2024-03-24T00:16:54.150 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:16:54.148+0000 7fa581dc9740 1 --2- 172.21.15.118:0/3749927435 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x559aab84bf50 0x559aab845100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:16:54.150 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:16:54.148+0000 7fa581dc9740 1 --2- 172.21.15.118:0/3749927435 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x559aab846de0 0x559aab844bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:16:54.150 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:16:54.148+0000 7fa581dc9740 1 --2- 172.21.15.118:0/3749927435 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x559aab8463f0 0x559aab841d40 unknown :-1 s=CLOSED pgs=254 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:16:54.150 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:16:54.148+0000 7fa581dc9740 1 -- 172.21.15.118:0/3749927435 >> 172.21.15.118:0/3749927435 conn(0x559aab841960 msgr2=0x559aab83c6c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:16:54.150 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:16:54.148+0000 7fa581dc9740 1 -- 172.21.15.118:0/3749927435 shutdown_connections 2024-03-24T00:16:54.150 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:16:54.148+0000 7fa581dc9740 1 -- 172.21.15.118:0/3749927435 wait complete. 2024-03-24T00:16:54.150 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:16:54.148+0000 7fa581dc9740 1 -- start start 2024-03-24T00:16:54.151 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:16:54.149+0000 7fa581dc9740 1 Processor -- start 2024-03-24T00:16:54.151 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:16:54.149+0000 7fa581dc9740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x559aab8463f0 0x559aab854840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:16:54.151 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:16:54.149+0000 7fa581dc9740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x559aab846de0 0x559aab854d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:16:54.151 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:16:54.149+0000 7fa581dc9740 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x559aab84bf50 0x559aab855d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:16:54.151 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:16:54.149+0000 7fa581dc9740 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa5781a78a0 con 0x559aab84bf50 2024-03-24T00:16:54.151 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:16:54.149+0000 7fa581dc9740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa5781a7240 con 0x559aab846de0 2024-03-24T00:16:54.151 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:16:54.149+0000 7fa581dc9740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa5781a4740 con 0x559aab8463f0 2024-03-24T00:16:54.152 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:16:54.149+0000 7fa581dc9740 10 client.0 _ll_register_callbacks cb 0x7fa56800b800 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T00:16:54.152 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:16:54.150+0000 7fa577e00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x559aab8463f0 0x559aab854840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:16:54.152 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:16:54.150+0000 7fa577e00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x559aab8463f0 0x559aab854840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.118:36320/0 (socket says 172.21.15.118:36320) 2024-03-24T00:16:54.152 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:16:54.150+0000 7fa577e00640 1 -- 172.21.15.118:0/3621359542 learned_addr learned my addr 172.21.15.118:0/3621359542 (peer_addr_for_me v2:172.21.15.118:0/0) 2024-03-24T00:16:54.152 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:16:54.150+0000 7fa57d000640 1 --2- 172.21.15.118:0/3621359542 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x559aab84bf50 0x559aab855d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:16:54.152 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:16:54.150+0000 7fa577400640 1 --2- 172.21.15.118:0/3621359542 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x559aab846de0 0x559aab854d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:16:54.152 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:16:54.151+0000 7fa57d000640 1 -- 172.21.15.118:0/3621359542 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x559aab8463f0 msgr2=0x559aab854840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:16:54.153 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:16:54.151+0000 7fa57d000640 1 --2- 172.21.15.118:0/3621359542 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x559aab8463f0 0x559aab854840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:16:54.153 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:16:54.151+0000 7fa57d000640 1 -- 172.21.15.118:0/3621359542 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x559aab846de0 msgr2=0x559aab854d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:16:54.153 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:16:54.151+0000 7fa57d000640 1 --2- 172.21.15.118:0/3621359542 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x559aab846de0 0x559aab854d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:16:54.153 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:16:54.151+0000 7fa57d000640 1 -- 172.21.15.118:0/3621359542 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x559aab856380 con 0x559aab84bf50 2024-03-24T00:16:54.153 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:16:54.152+0000 7fa57d000640 1 --2- 172.21.15.118:0/3621359542 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x559aab84bf50 0x559aab855d90 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7fa5700cca50 tx=0x7fa57009b4e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f0ef9a10c11d0531 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:16:54.153 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:16:54.152+0000 7fa562000640 10 client.0 ms_handle_connect on v2:172.21.15.118:3300/0 2024-03-24T00:16:54.153 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:16:54.152+0000 7fa562000640 1 -- 172.21.15.118:0/3621359542 <== mon.0 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa57009fa30 con 0x559aab84bf50 2024-03-24T00:16:54.153 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:16:54.152+0000 7fa562000640 1 -- 172.21.15.118:0/3621359542 <== mon.0 v2:172.21.15.118:3300/0 2 ==== osd_map(90..90 src has 1..90) v4 ==== 8277+0+0 (secure 0 0 0) 0x7fa57009af40 con 0x559aab84bf50 2024-03-24T00:16:54.153 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:16:54.152+0000 7fa581dc9740 1 -- 172.21.15.118:0/3621359542 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fa568005510 con 0x559aab84bf50 2024-03-24T00:16:54.154 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:16:54.152+0000 7fa562000640 1 -- 172.21.15.118:0/3621359542 <== mon.0 v2:172.21.15.118:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) v2 ==== 24+0+0 (secure 0 0 0) 0x7fa5700a4080 con 0x559aab84bf50 2024-03-24T00:16:54.154 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:16:54.153+0000 7fa581dc9740 10 client.15606 fetch_fsmap learned FSMap version 28 2024-03-24T00:16:54.155 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:16:54.153+0000 7fa581dc9740 1 -- 172.21.15.118:0/3621359542 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) v3 -- 0x559aab8b7e20 con 0x559aab84bf50 2024-03-24T00:16:54.155 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:16:54.153+0000 7fa562000640 1 -- 172.21.15.118:0/3621359542 <== mon.0 v2:172.21.15.118:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fa568005510 con 0x559aab84bf50 2024-03-24T00:16:54.155 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:16:54.153+0000 7fa562000640 1 -- 172.21.15.118:0/3621359542 <== mon.0 v2:172.21.15.118:3300/0 5 ==== fsmap(e 28) v1 ==== 7934+0+0 (secure 0 0 0) 0x7fa570010c30 con 0x559aab84bf50 2024-03-24T00:16:54.155 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:16:54.153+0000 7fa581dc9740 10 client.15606 fetch_fsmap finished waiting for FSMap version 28 2024-03-24T00:16:54.155 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:16:54.153+0000 7fa581dc9740 10 client.15606 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-24T00:16:54.155 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:16:54.153+0000 7fa581dc9740 20 client.15606 populate_metadata read hostname 'smithi118' 2024-03-24T00:16:54.155 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:16:54.153+0000 7fa581dc9740 1 --2- 172.21.15.118:0/3621359542 >> [v2:172.21.15.189:6838/4074484168,v1:172.21.15.189:6839/4074484168] conn(0x559aab8b85e0 0x559aab8baa00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:16:54.155 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:16:54.153+0000 7fa581dc9740 4 client.15606 mds_command: new command op to 24569 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-03-24T00:16:54.155 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:16:54.153+0000 7fa581dc9740 1 -- 172.21.15.118:0/3621359542 --> [v2:172.21.15.189:6838/4074484168,v1:172.21.15.189:6839/4074484168] -- command(tid 0: {"prefix": "flush journal"}) v1 -- 0x559aab7b30c0 con 0x559aab8b85e0 2024-03-24T00:16:54.156 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:16:54.154+0000 7fa577e00640 1 --2- 172.21.15.118:0/3621359542 >> [v2:172.21.15.189:6838/4074484168,v1:172.21.15.189:6839/4074484168] conn(0x559aab8b85e0 0x559aab8baa00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:16:54.156 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:16:54.154+0000 7fa562000640 10 client.15606 ms_handle_connect on v2:172.21.15.189:6838/4074484168 2024-03-24T00:16:54.156 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:16:54.154+0000 7fa577e00640 1 --2- 172.21.15.118:0/3621359542 >> [v2:172.21.15.189:6838/4074484168,v1:172.21.15.189:6839/4074484168] conn(0x559aab8b85e0 0x559aab8baa00 crc :-1 s=READY pgs=56 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=17d622f1f7c84a0e server_cookie=703f8cdaf77483b0 in_seq=0 out_seq=0 2024-03-24T00:16:54.158 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:16:54.157+0000 7fa562000640 1 -- 172.21.15.118:0/3621359542 <== mds.0 v2:172.21.15.189:6838/4074484168 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+44 (crc 0 0 0) 0x559aab7b30c0 con 0x559aab8b85e0 2024-03-24T00:16:54.158 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:16:54.158+0000 7fa562000640 10 client.15606 handle_command_reply: tid=0 2024-03-24T00:16:54.158 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:16:54.158+0000 7fa581dc9740 1 client.15606 shutdown 2024-03-24T00:16:54.159 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:16:54.158+0000 7fa581dc9740 1 -- 172.21.15.118:0/3621359542 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x559aab84bf50 msgr2=0x559aab855d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:16:54.159 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:16:54.158+0000 7fa581dc9740 1 --2- 172.21.15.118:0/3621359542 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x559aab84bf50 0x559aab855d90 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7fa5700cca50 tx=0x7fa57009b4e0 comp rx=0 tx=0).stop 2024-03-24T00:16:54.160 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:16:54.159+0000 7fa581dc9740 1 -- 172.21.15.118:0/3621359542 shutdown_connections 2024-03-24T00:16:54.160 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:16:54.159+0000 7fa581dc9740 1 --2- 172.21.15.118:0/3621359542 >> [v2:172.21.15.189:6838/4074484168,v1:172.21.15.189:6839/4074484168] conn(0x559aab8b85e0 0x559aab8baa00 crc :-1 s=READY pgs=56 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:16:54.160 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:16:54.159+0000 7fa581dc9740 1 --2- 172.21.15.118:0/3621359542 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x559aab84bf50 0x559aab855d90 unknown :-1 s=CLOSED pgs=187 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:16:54.160 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:16:54.159+0000 7fa581dc9740 1 --2- 172.21.15.118:0/3621359542 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x559aab846de0 0x559aab854d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:16:54.161 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:16:54.159+0000 7fa581dc9740 1 --2- 172.21.15.118:0/3621359542 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x559aab8463f0 0x559aab854840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:16:54.161 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:16:54.159+0000 7fa581dc9740 1 -- 172.21.15.118:0/3621359542 >> 172.21.15.118:0/3621359542 conn(0x559aab841960 msgr2=0x559aab841d40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:16:54.161 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:16:54.159+0000 7fa562000640 0 client.15606 ms_handle_reset on v2:172.21.15.189:6838/4074484168 2024-03-24T00:16:54.161 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:16:54.159+0000 7fa581dc9740 1 -- 172.21.15.118:0/3621359542 shutdown_connections 2024-03-24T00:16:54.161 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:16:54.159+0000 7fa581dc9740 1 -- 172.21.15.118:0/3621359542 wait complete. 2024-03-24T00:16:54.161 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:16:54.159+0000 7fa581dc9740 20 client.15606 trim_cache size 0 max 16384 2024-03-24T00:16:54.161 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stdout:{ 2024-03-24T00:16:54.161 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stdout: "message": "", 2024-03-24T00:16:54.161 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stdout: "return_code": 0 2024-03-24T00:16:54.161 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stdout:} 2024-03-24T00:16:54.163 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:16:54.161+0000 7fa57da00640 1 -- 172.21.15.118:0/2161665191 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7fa55c0a2aa0 msgr2=0x7fa55c0a4f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:16:54.163 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:16:54.161+0000 7fa57da00640 1 --2- 172.21.15.118:0/2161665191 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7fa55c0a2aa0 0x7fa55c0a4f60 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 crypto rx=0x7fa564001eb0 tx=0x7fa564000bc0 comp rx=0 tx=0).stop 2024-03-24T00:16:54.163 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:16:54.161+0000 7fa57da00640 1 -- 172.21.15.118:0/2161665191 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fa5781071d0 msgr2=0x7fa57819cc00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:16:54.163 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:16:54.161+0000 7fa57da00640 1 --2- 172.21.15.118:0/2161665191 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fa5781071d0 0x7fa57819cc00 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7fa57000ec50 tx=0x7fa57000c6a0 comp rx=0 tx=0).stop 2024-03-24T00:16:54.163 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:16:54.161+0000 7fa57da00640 1 -- 172.21.15.118:0/2161665191 shutdown_connections 2024-03-24T00:16:54.163 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:16:54.161+0000 7fa57da00640 1 --2- 172.21.15.118:0/2161665191 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7fa55c0a2aa0 0x7fa55c0a4f60 unknown :-1 s=CLOSED pgs=289 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:16:54.163 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:16:54.161+0000 7fa57da00640 1 --2- 172.21.15.118:0/2161665191 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fa5781071d0 0x7fa57819cc00 unknown :-1 s=CLOSED pgs=250 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:16:54.163 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:16:54.161+0000 7fa57da00640 1 --2- 172.21.15.118:0/2161665191 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa5781048a0 0x7fa578195b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:16:54.163 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:16:54.161+0000 7fa57da00640 1 --2- 172.21.15.118:0/2161665191 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fa578101f70 0x7fa578195650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:16:54.163 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:16:54.161+0000 7fa57da00640 1 -- 172.21.15.118:0/2161665191 >> 172.21.15.118:0/2161665191 conn(0x7fa5780fb980 msgr2=0x7fa5781028b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:16:54.163 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:16:54.162+0000 7fa57da00640 1 -- 172.21.15.118:0/2161665191 shutdown_connections 2024-03-24T00:16:54.163 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:16:54.162+0000 7fa57da00640 1 -- 172.21.15.118:0/2161665191 wait complete. 2024-03-24T00:16:54.173 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:54.172+0000 7f765e61c740 1 Processor -- start 2024-03-24T00:16:54.174 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:54.172+0000 7f765e61c740 1 -- start start 2024-03-24T00:16:54.174 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:54.173+0000 7f765e61c740 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55f1082d2810 0x55f1082d1570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:16:54.174 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:54.173+0000 7f765e61c740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55f1082d39e0 0x55f1082e4040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:16:54.174 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:54.173+0000 7f765e61c740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55f1082d43a0 0x55f1082eef00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:16:54.174 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:54.173+0000 7f765e61c740 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f76541bffe0 con 0x55f1082d2810 2024-03-24T00:16:54.174 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:54.173+0000 7f765e61c740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f76541c1960 con 0x55f1082d39e0 2024-03-24T00:16:54.174 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:54.173+0000 7f765e61c740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f76541c0640 con 0x55f1082d43a0 2024-03-24T00:16:54.174 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:54.173+0000 7f7653e00640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55f1082d2810 0x55f1082d1570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:16:54.174 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:54.173+0000 7f7653e00640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55f1082d2810 0x55f1082d1570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.118:3300/0 says I am v2:172.21.15.118:50602/0 (socket says 172.21.15.118:50602) 2024-03-24T00:16:54.174 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:54.173+0000 7f7653e00640 1 -- 172.21.15.118:0/2326237852 learned_addr learned my addr 172.21.15.118:0/2326237852 (peer_addr_for_me v2:172.21.15.118:0/0) 2024-03-24T00:16:54.175 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:54.174+0000 7f7659000640 1 --2- 172.21.15.118:0/2326237852 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55f1082d39e0 0x55f1082e4040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:16:54.175 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:54.174+0000 7f7653e00640 1 --2- 172.21.15.118:0/2326237852 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55f1082d43a0 0x55f1082eef00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:16:54.175 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:54.174+0000 7f7653e00640 1 -- 172.21.15.118:0/2326237852 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55f1082d43a0 msgr2=0x55f1082eef00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:16:54.175 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:54.174+0000 7f7653e00640 1 --2- 172.21.15.118:0/2326237852 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55f1082d43a0 0x55f1082eef00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:16:54.176 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:54.174+0000 7f7653e00640 1 -- 172.21.15.118:0/2326237852 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55f1082d39e0 msgr2=0x55f1082e4040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:16:54.176 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:54.174+0000 7f7653e00640 1 --2- 172.21.15.118:0/2326237852 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55f1082d39e0 0x55f1082e4040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:16:54.176 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:54.174+0000 7f7653e00640 1 -- 172.21.15.118:0/2326237852 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55f1082eacb0 con 0x55f1082d2810 2024-03-24T00:16:54.176 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:54.174+0000 7f7653e00640 1 --2- 172.21.15.118:0/2326237852 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55f1082d2810 0x55f1082d1570 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7f7640009c50 tx=0x7f7640005c50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d18b7ffa74d39557 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:16:54.176 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:54.175+0000 7f7649600640 1 -- 172.21.15.118:0/2326237852 <== mon.0 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f764001e050 con 0x55f1082d2810 2024-03-24T00:16:54.176 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:54.175+0000 7f7649600640 1 -- 172.21.15.118:0/2326237852 <== mon.0 v2:172.21.15.118:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f764003a020 con 0x55f1082d2810 2024-03-24T00:16:54.176 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:54.175+0000 7f7649600640 1 -- 172.21.15.118:0/2326237852 <== mon.0 v2:172.21.15.118:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7640009d40 con 0x55f1082d2810 2024-03-24T00:16:54.177 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:54.176+0000 7f765e61c740 1 -- 172.21.15.118:0/2326237852 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55f1082d2810 msgr2=0x55f1082d1570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:16:54.177 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:54.176+0000 7f765e61c740 1 --2- 172.21.15.118:0/2326237852 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55f1082d2810 0x55f1082d1570 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7f7640009c50 tx=0x7f7640005c50 comp rx=0 tx=0).stop 2024-03-24T00:16:54.177 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:54.176+0000 7f765e61c740 1 -- 172.21.15.118:0/2326237852 shutdown_connections 2024-03-24T00:16:54.177 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:54.176+0000 7f765e61c740 1 --2- 172.21.15.118:0/2326237852 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55f1082d43a0 0x55f1082eef00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:16:54.177 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:54.176+0000 7f765e61c740 1 --2- 172.21.15.118:0/2326237852 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55f1082d39e0 0x55f1082e4040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:16:54.177 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:54.176+0000 7f765e61c740 1 --2- 172.21.15.118:0/2326237852 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55f1082d2810 0x55f1082d1570 unknown :-1 s=CLOSED pgs=188 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:16:54.177 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:54.176+0000 7f765e61c740 1 -- 172.21.15.118:0/2326237852 >> 172.21.15.118:0/2326237852 conn(0x55f1082cdc00 msgr2=0x55f1082d0e60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:16:54.177 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:54.176+0000 7f765e61c740 1 -- 172.21.15.118:0/2326237852 shutdown_connections 2024-03-24T00:16:54.177 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:54.176+0000 7f765e61c740 1 -- 172.21.15.118:0/2326237852 wait complete. 2024-03-24T00:16:54.178 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:54.177+0000 7f765e61c740 1 -- start start 2024-03-24T00:16:54.178 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:54.177+0000 7f765e61c740 1 Processor -- start 2024-03-24T00:16:54.179 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:54.178+0000 7f765e61c740 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55f1082d39e0 0x55f1082e3e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:16:54.179 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:54.178+0000 7f765e61c740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55f1082d43a0 0x55f1082e24e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:16:54.179 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:54.178+0000 7f765e61c740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55f1082e2a20 0x55f1082e2ea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:16:54.179 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:54.178+0000 7f765e61c740 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f76541c1300 con 0x55f1082d39e0 2024-03-24T00:16:54.179 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:54.178+0000 7f765e61c740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f76541c0ca0 con 0x55f1082e2a20 2024-03-24T00:16:54.179 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:54.178+0000 7f765e61c740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f76541be1a0 con 0x55f1082d43a0 2024-03-24T00:16:54.179 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:54.178+0000 7f765e61c740 10 client.0 _ll_register_callbacks cb 0x7f764001c070 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T00:16:54.179 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:54.178+0000 7f7653e00640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55f1082d39e0 0x55f1082e3e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:16:54.179 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:54.178+0000 7f7653e00640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55f1082d39e0 0x55f1082e3e80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.118:3300/0 says I am v2:172.21.15.118:50610/0 (socket says 172.21.15.118:50610) 2024-03-24T00:16:54.179 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:54.178+0000 7f7653e00640 1 -- 172.21.15.118:0/802095376 learned_addr learned my addr 172.21.15.118:0/802095376 (peer_addr_for_me v2:172.21.15.118:0/0) 2024-03-24T00:16:54.179 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:54.178+0000 7f7659000640 1 --2- 172.21.15.118:0/802095376 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55f1082d43a0 0x55f1082e24e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:16:54.180 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:54.179+0000 7f7653e00640 1 --2- 172.21.15.118:0/802095376 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55f1082e2a20 0x55f1082e2ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:16:54.180 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:54.179+0000 7f7653e00640 1 -- 172.21.15.118:0/802095376 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55f1082d43a0 msgr2=0x55f1082e24e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:16:54.180 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:54.179+0000 7f7653e00640 1 --2- 172.21.15.118:0/802095376 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55f1082d43a0 0x55f1082e24e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:16:54.180 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:54.179+0000 7f7653e00640 1 -- 172.21.15.118:0/802095376 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55f1082e2a20 msgr2=0x55f1082e2ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:16:54.181 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:54.179+0000 7f7653e00640 1 --2- 172.21.15.118:0/802095376 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55f1082e2a20 0x55f1082e2ea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:16:54.181 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:54.179+0000 7f7653e00640 1 -- 172.21.15.118:0/802095376 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55f1082db4c0 con 0x55f1082d39e0 2024-03-24T00:16:54.181 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:54.179+0000 7f7653e00640 1 --2- 172.21.15.118:0/802095376 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55f1082d39e0 0x55f1082e3e80 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7f76400059f0 tx=0x7f7640009790 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dfda4f486bb07a90 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:16:54.181 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:54.179+0000 7f763ea00640 10 client.0 ms_handle_connect on v2:172.21.15.118:3300/0 2024-03-24T00:16:54.181 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:54.180+0000 7f763ea00640 1 -- 172.21.15.118:0/802095376 <== mon.0 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f764001e040 con 0x55f1082d39e0 2024-03-24T00:16:54.181 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:54.180+0000 7f763ea00640 1 -- 172.21.15.118:0/802095376 <== mon.0 v2:172.21.15.118:3300/0 2 ==== osd_map(90..90 src has 1..90) v4 ==== 8277+0+0 (secure 0 0 0) 0x7f76400051f0 con 0x55f1082d39e0 2024-03-24T00:16:54.181 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:54.180+0000 7f765e61c740 1 -- 172.21.15.118:0/802095376 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f764003a020 con 0x55f1082d39e0 2024-03-24T00:16:54.181 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:54.181+0000 7f763ea00640 1 -- 172.21.15.118:0/802095376 <== mon.0 v2:172.21.15.118:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) v2 ==== 24+0+0 (secure 0 0 0) 0x7f7640005630 con 0x55f1082d39e0 2024-03-24T00:16:54.182 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:54.181+0000 7f765e61c740 10 client.15612 fetch_fsmap learned FSMap version 28 2024-03-24T00:16:54.182 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:54.181+0000 7f765e61c740 1 -- 172.21.15.118:0/802095376 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) v3 -- 0x55f1082dd3b0 con 0x55f1082d39e0 2024-03-24T00:16:54.182 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:54.181+0000 7f763ea00640 1 -- 172.21.15.118:0/802095376 <== mon.0 v2:172.21.15.118:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f764003a020 con 0x55f1082d39e0 2024-03-24T00:16:54.183 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:54.182+0000 7f763ea00640 1 -- 172.21.15.118:0/802095376 <== mon.0 v2:172.21.15.118:3300/0 5 ==== fsmap(e 28) v1 ==== 7934+0+0 (secure 0 0 0) 0x7f7640049480 con 0x55f1082d39e0 2024-03-24T00:16:54.183 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:54.182+0000 7f765e61c740 10 client.15612 fetch_fsmap finished waiting for FSMap version 28 2024-03-24T00:16:54.184 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:54.182+0000 7f765e61c740 10 client.15612 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.g 2024-03-24T00:16:54.184 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:54.182+0000 7f765e61c740 20 client.15612 populate_metadata read hostname 'smithi118' 2024-03-24T00:16:54.184 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:54.182+0000 7f765e61c740 1 --2- 172.21.15.118:0/802095376 >> [v2:172.21.15.118:6838/3285788503,v1:172.21.15.118:6839/3285788503] conn(0x55f1083451d0 0x55f1083475c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:16:54.184 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:54.182+0000 7f765e61c740 4 client.15612 mds_command: new command op to 24550 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-24T00:16:54.184 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:54.182+0000 7f765e61c740 1 -- 172.21.15.118:0/802095376 --> [v2:172.21.15.118:6838/3285788503,v1:172.21.15.118:6839/3285788503] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x55f10823eb50 con 0x55f1083451d0 2024-03-24T00:16:54.184 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:54.183+0000 7f7659000640 1 --2- 172.21.15.118:0/802095376 >> [v2:172.21.15.118:6838/3285788503,v1:172.21.15.118:6839/3285788503] conn(0x55f1083451d0 0x55f1083475c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:16:54.185 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:54.183+0000 7f7659000640 1 --2- 172.21.15.118:0/802095376 >> [v2:172.21.15.118:6838/3285788503,v1:172.21.15.118:6839/3285788503] conn(0x55f1083451d0 0x55f1083475c0 crc :-1 s=READY pgs=25 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=86dc2b99a5f806b server_cookie=f70156311581ccaa in_seq=0 out_seq=0 2024-03-24T00:16:54.185 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:54.184+0000 7f763ea00640 10 client.15612 ms_handle_connect on v2:172.21.15.118:6838/3285788503 2024-03-24T00:16:54.185 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:54.184+0000 7f763ea00640 1 -- 172.21.15.118:0/802095376 <== mds.1 v2:172.21.15.118:6838/3285788503 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x55f10823eb50 con 0x55f1083451d0 2024-03-24T00:16:54.185 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:54.184+0000 7f763ea00640 10 client.15612 handle_command_reply: tid=0 2024-03-24T00:16:54.185 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:54.184+0000 7f765e61c740 1 client.15612 shutdown 2024-03-24T00:16:54.185 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:54.184+0000 7f765e61c740 1 -- 172.21.15.118:0/802095376 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55f1082d39e0 msgr2=0x55f1082e3e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:16:54.185 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:54.184+0000 7f765e61c740 1 --2- 172.21.15.118:0/802095376 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55f1082d39e0 0x55f1082e3e80 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7f76400059f0 tx=0x7f7640009790 comp rx=0 tx=0).stop 2024-03-24T00:16:54.186 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:54.185+0000 7f765e61c740 1 -- 172.21.15.118:0/802095376 shutdown_connections 2024-03-24T00:16:54.186 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:54.185+0000 7f765e61c740 1 --2- 172.21.15.118:0/802095376 >> [v2:172.21.15.118:6838/3285788503,v1:172.21.15.118:6839/3285788503] conn(0x55f1083451d0 0x55f1083475c0 crc :-1 s=READY pgs=25 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:16:54.186 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:54.185+0000 7f765e61c740 1 --2- 172.21.15.118:0/802095376 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55f1082e2a20 0x55f1082e2ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:16:54.186 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:54.185+0000 7f765e61c740 1 --2- 172.21.15.118:0/802095376 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55f1082d43a0 0x55f1082e24e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:16:54.186 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:54.185+0000 7f765e61c740 1 --2- 172.21.15.118:0/802095376 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55f1082d39e0 0x55f1082e3e80 unknown :-1 s=CLOSED pgs=189 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:16:54.186 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:54.185+0000 7f765e61c740 1 -- 172.21.15.118:0/802095376 >> 172.21.15.118:0/802095376 conn(0x55f1082cdc00 msgr2=0x55f1082d0e60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:16:54.186 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:54.185+0000 7f763ea00640 0 client.15612 ms_handle_reset on v2:172.21.15.118:6838/3285788503 2024-03-24T00:16:54.187 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:54.187+0000 7f765e61c740 1 -- 172.21.15.118:0/802095376 shutdown_connections 2024-03-24T00:16:54.187 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:54.187+0000 7f765e61c740 1 -- 172.21.15.118:0/802095376 wait complete. 2024-03-24T00:16:54.187 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:54.187+0000 7f765e61c740 20 client.15612 trim_cache size 0 max 16384 2024-03-24T00:16:54.188 INFO:teuthology.orchestra.run.smithi118.stdout:[] 2024-03-24T00:16:54.189 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:54.189+0000 7f765ba00640 1 -- 172.21.15.118:0/2951298035 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7f76380a2950 msgr2=0x7f76380a4e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:16:54.189 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:54.189+0000 7f765ba00640 1 --2- 172.21.15.118:0/2951298035 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7f76380a2950 0x7f76380a4e10 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x7f764c0b9e20 tx=0x7f764c02bef0 comp rx=0 tx=0).stop 2024-03-24T00:16:54.189 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:54.189+0000 7f765ba00640 1 -- 172.21.15.118:0/2951298035 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f765410d9c0 msgr2=0x7f76541b6800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:16:54.189 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:54.189+0000 7f765ba00640 1 --2- 172.21.15.118:0/2951298035 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f765410d9c0 0x7f76541b6800 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7f764c036040 tx=0x7f764c0337b0 comp rx=0 tx=0).stop 2024-03-24T00:16:54.190 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:54.189+0000 7f765ba00640 1 -- 172.21.15.118:0/2951298035 shutdown_connections 2024-03-24T00:16:54.190 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:54.189+0000 7f765ba00640 1 --2- 172.21.15.118:0/2951298035 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7f76380a2950 0x7f76380a4e10 unknown :-1 s=CLOSED pgs=290 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:16:54.190 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:54.189+0000 7f765ba00640 1 --2- 172.21.15.118:0/2951298035 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f765410d9c0 0x7f76541b6800 unknown :-1 s=CLOSED pgs=252 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:16:54.190 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:54.189+0000 7f765ba00640 1 --2- 172.21.15.118:0/2951298035 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f765410b010 0x7f7654117290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:16:54.190 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:54.189+0000 7f765ba00640 1 --2- 172.21.15.118:0/2951298035 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f76541086a0 0x7f7654116d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:16:54.191 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:54.189+0000 7f765ba00640 1 -- 172.21.15.118:0/2951298035 >> 172.21.15.118:0/2951298035 conn(0x7f765406e500 msgr2=0x7f765410b840 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:16:54.191 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:54.189+0000 7f765ba00640 1 -- 172.21.15.118:0/2951298035 shutdown_connections 2024-03-24T00:16:54.191 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:54.190+0000 7f765ba00640 1 -- 172.21.15.118:0/2951298035 wait complete. 2024-03-24T00:16:54.203 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:16:53 smithi189 ceph-mon[36187]: from='client.? 172.21.15.118:0/1738454964' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-24T00:16:54.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:16:53 smithi193 ceph-mon[36878]: from='client.? 172.21.15.118:0/1738454964' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-24T00:16:54.218 INFO:teuthology.orchestra.run:Running command with timeout 120 2024-03-24T00:16:54.218 DEBUG:teuthology.orchestra.run.smithi118:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2024-03-24T00:16:54.452 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:54.451+0000 7fd451e00640 1 -- 172.21.15.118:0/850977900 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd44c105110 msgr2=0x7fd44c105590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:16:54.452 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:54.451+0000 7fd451e00640 1 --2- 172.21.15.118:0/850977900 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd44c105110 0x7fd44c105590 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7fd444009ca0 tx=0x7fd44402c100 comp rx=0 tx=0).stop 2024-03-24T00:16:54.452 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:54.451+0000 7fd451e00640 1 -- 172.21.15.118:0/850977900 shutdown_connections 2024-03-24T00:16:54.452 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:54.451+0000 7fd451e00640 1 --2- 172.21.15.118:0/850977900 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd44c105110 0x7fd44c105590 unknown :-1 s=CLOSED pgs=261 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:16:54.452 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:54.451+0000 7fd451e00640 1 --2- 172.21.15.118:0/850977900 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fd44c102760 0x7fd44c104bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:16:54.453 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:54.451+0000 7fd451e00640 1 --2- 172.21.15.118:0/850977900 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fd44c0ffdf0 0x7fd44c102220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:16:54.453 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:54.451+0000 7fd451e00640 1 -- 172.21.15.118:0/850977900 >> 172.21.15.118:0/850977900 conn(0x7fd44c0fba10 msgr2=0x7fd44c0fde30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:16:54.453 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:54.451+0000 7fd451e00640 1 -- 172.21.15.118:0/850977900 shutdown_connections 2024-03-24T00:16:54.453 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:54.451+0000 7fd451e00640 1 -- 172.21.15.118:0/850977900 wait complete. 2024-03-24T00:16:54.454 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:54.452+0000 7fd451e00640 1 Processor -- start 2024-03-24T00:16:54.454 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:54.452+0000 7fd451e00640 1 -- start start 2024-03-24T00:16:54.454 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:54.453+0000 7fd451e00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fd44c0ffdf0 0x7fd44c19dd60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:16:54.454 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:54.453+0000 7fd451e00640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fd44c102760 0x7fd44c19e2a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:16:54.454 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:54.453+0000 7fd451e00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd44c105110 0x7fd44c1a5310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:16:54.455 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:54.453+0000 7fd451e00640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd44c1137d0 con 0x7fd44c102760 2024-03-24T00:16:54.455 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:54.453+0000 7fd451e00640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd44c113650 con 0x7fd44c105110 2024-03-24T00:16:54.455 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:54.453+0000 7fd451e00640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd44c113950 con 0x7fd44c0ffdf0 2024-03-24T00:16:54.455 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:54.453+0000 7fd451400640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd44c105110 0x7fd44c1a5310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:16:54.455 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:54.453+0000 7fd451400640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd44c105110 0x7fd44c1a5310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.118:40552/0 (socket says 172.21.15.118:40552) 2024-03-24T00:16:54.455 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:54.453+0000 7fd451400640 1 -- 172.21.15.118:0/608827690 learned_addr learned my addr 172.21.15.118:0/608827690 (peer_addr_for_me v2:172.21.15.118:0/0) 2024-03-24T00:16:54.455 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:54.453+0000 7fd450a00640 1 --2- 172.21.15.118:0/608827690 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fd44c0ffdf0 0x7fd44c19dd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:16:54.455 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:54.453+0000 7fd451400640 1 -- 172.21.15.118:0/608827690 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fd44c0ffdf0 msgr2=0x7fd44c19dd60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:16:54.455 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:54.453+0000 7fd451400640 1 --2- 172.21.15.118:0/608827690 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fd44c0ffdf0 0x7fd44c19dd60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:16:54.455 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:54.453+0000 7fd451400640 1 -- 172.21.15.118:0/608827690 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fd44c102760 msgr2=0x7fd44c19e2a0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-24T00:16:54.455 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:54.454+0000 7fd44be00640 1 --2- 172.21.15.118:0/608827690 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fd44c102760 0x7fd44c19e2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:16:54.455 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:54.454+0000 7fd451400640 1 --2- 172.21.15.118:0/608827690 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fd44c102760 0x7fd44c19e2a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:16:54.455 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:54.454+0000 7fd451400640 1 -- 172.21.15.118:0/608827690 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd44c1a5980 con 0x7fd44c105110 2024-03-24T00:16:54.455 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:54.454+0000 7fd44be00640 1 --2- 172.21.15.118:0/608827690 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fd44c102760 0x7fd44c19e2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-24T00:16:54.456 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:54.454+0000 7fd451400640 1 --2- 172.21.15.118:0/608827690 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd44c105110 0x7fd44c1a5310 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7fd44402c640 tx=0x7fd444035d10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:16:54.456 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:54.454+0000 7fd449600640 1 -- 172.21.15.118:0/608827690 <== mon.2 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd444039040 con 0x7fd44c105110 2024-03-24T00:16:54.456 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:54.454+0000 7fd449600640 1 -- 172.21.15.118:0/608827690 <== mon.2 v2:172.21.15.189:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fd444044070 con 0x7fd44c105110 2024-03-24T00:16:54.456 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:54.454+0000 7fd449600640 1 -- 172.21.15.118:0/608827690 <== mon.2 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd4440044c0 con 0x7fd44c105110 2024-03-24T00:16:54.456 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:54.454+0000 7fd451e00640 1 -- 172.21.15.118:0/608827690 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd44c1a5c10 con 0x7fd44c105110 2024-03-24T00:16:54.456 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:54.455+0000 7fd451e00640 1 -- 172.21.15.118:0/608827690 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd44c1a60b0 con 0x7fd44c105110 2024-03-24T00:16:54.458 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:54.458+0000 7fd449600640 1 -- 172.21.15.118:0/608827690 <== mon.2 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 19) v1 ==== 137342+0+0 (secure 0 0 0) 0x7fd444032890 con 0x7fd44c105110 2024-03-24T00:16:54.459 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:54.459+0000 7fd449600640 1 --2- 172.21.15.118:0/608827690 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7fd4340a2910 0x7fd4340a4dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:16:54.459 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:54.459+0000 7fd449600640 1 -- 172.21.15.118:0/608827690 <== mon.2 v2:172.21.15.189:3300/0 5 ==== osd_map(90..90 src has 1..90) v4 ==== 8277+0+0 (secure 0 0 0) 0x7fd4440f0d70 con 0x7fd44c105110 2024-03-24T00:16:54.460 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:54.459+0000 7fd450a00640 1 --2- 172.21.15.118:0/608827690 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7fd4340a2910 0x7fd4340a4dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:16:54.462 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:54.462+0000 7fd450a00640 1 --2- 172.21.15.118:0/608827690 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7fd4340a2910 0x7fd4340a4dd0 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x7fd440005e10 tx=0x7fd440005da0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:16:54.464 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:54.464+0000 7fd456002740 1 Processor -- start 2024-03-24T00:16:54.464 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:54.464+0000 7fd456002740 1 -- start start 2024-03-24T00:16:54.464 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:54.464+0000 7fd456002740 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5652c2f129f0 0x5652c2f12e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:16:54.464 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:54.464+0000 7fd456002740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5652c2f13bc0 0x5652c2f14040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:16:54.465 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:54.464+0000 7fd456002740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5652c2f14580 0x5652c2f1ae10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:16:54.465 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:54.464+0000 7fd456002740 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x5652c2e772c0 con 0x5652c2f129f0 2024-03-24T00:16:54.465 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:54.464+0000 7fd456002740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x5652c2e1fc90 con 0x5652c2f13bc0 2024-03-24T00:16:54.465 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:54.464+0000 7fd456002740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x5652c2e73360 con 0x5652c2f14580 2024-03-24T00:16:54.465 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:54.465+0000 7fd450a00640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5652c2f129f0 0x5652c2f12e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:16:54.466 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:54.465+0000 7fd450a00640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5652c2f129f0 0x5652c2f12e10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.118:3300/0 says I am v2:172.21.15.118:50644/0 (socket says 172.21.15.118:50644) 2024-03-24T00:16:54.466 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:54.465+0000 7fd450a00640 1 -- 172.21.15.118:0/1833431255 learned_addr learned my addr 172.21.15.118:0/1833431255 (peer_addr_for_me v2:172.21.15.118:0/0) 2024-03-24T00:16:54.466 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:54.465+0000 7fd44be00640 1 --2- 172.21.15.118:0/1833431255 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5652c2f13bc0 0x5652c2f14040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:16:54.466 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:54.465+0000 7fd451400640 1 --2- 172.21.15.118:0/1833431255 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5652c2f14580 0x5652c2f1ae10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:16:54.466 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:54.465+0000 7fd450a00640 1 -- 172.21.15.118:0/1833431255 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5652c2f14580 msgr2=0x5652c2f1ae10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:16:54.467 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:54.465+0000 7fd450a00640 1 --2- 172.21.15.118:0/1833431255 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5652c2f14580 0x5652c2f1ae10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:16:54.467 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:54.465+0000 7fd450a00640 1 -- 172.21.15.118:0/1833431255 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5652c2f13bc0 msgr2=0x5652c2f14040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:16:54.467 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:54.465+0000 7fd450a00640 1 --2- 172.21.15.118:0/1833431255 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5652c2f13bc0 0x5652c2f14040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:16:54.467 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:54.465+0000 7fd450a00640 1 -- 172.21.15.118:0/1833431255 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5652c2e44080 con 0x5652c2f129f0 2024-03-24T00:16:54.467 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:54.466+0000 7fd450a00640 1 --2- 172.21.15.118:0/1833431255 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5652c2f129f0 0x5652c2f12e10 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7fd44000a320 tx=0x7fd440011000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=aa986fe951575087 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:16:54.467 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:54.466+0000 7fd43dc00640 1 -- 172.21.15.118:0/1833431255 <== mon.0 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd440021070 con 0x5652c2f129f0 2024-03-24T00:16:54.467 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:54.466+0000 7fd43dc00640 1 -- 172.21.15.118:0/1833431255 <== mon.0 v2:172.21.15.118:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fd44001db30 con 0x5652c2f129f0 2024-03-24T00:16:54.467 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:54.466+0000 7fd43dc00640 1 -- 172.21.15.118:0/1833431255 <== mon.0 v2:172.21.15.118:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd44000faf0 con 0x5652c2f129f0 2024-03-24T00:16:54.468 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:54.467+0000 7fd456002740 1 -- 172.21.15.118:0/1833431255 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5652c2f129f0 msgr2=0x5652c2f12e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:16:54.468 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:54.467+0000 7fd456002740 1 --2- 172.21.15.118:0/1833431255 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5652c2f129f0 0x5652c2f12e10 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7fd44000a320 tx=0x7fd440011000 comp rx=0 tx=0).stop 2024-03-24T00:16:54.468 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:54.467+0000 7fd456002740 1 -- 172.21.15.118:0/1833431255 shutdown_connections 2024-03-24T00:16:54.468 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:54.467+0000 7fd456002740 1 --2- 172.21.15.118:0/1833431255 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5652c2f14580 0x5652c2f1ae10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:16:54.468 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:54.467+0000 7fd456002740 1 --2- 172.21.15.118:0/1833431255 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5652c2f13bc0 0x5652c2f14040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:16:54.468 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:54.467+0000 7fd456002740 1 --2- 172.21.15.118:0/1833431255 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5652c2f129f0 0x5652c2f12e10 unknown :-1 s=CLOSED pgs=190 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:16:54.468 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:54.467+0000 7fd456002740 1 -- 172.21.15.118:0/1833431255 >> 172.21.15.118:0/1833431255 conn(0x5652c2f0dde0 msgr2=0x5652c2f0e1e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:16:54.469 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:54.467+0000 7fd456002740 1 -- 172.21.15.118:0/1833431255 shutdown_connections 2024-03-24T00:16:54.469 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:54.468+0000 7fd456002740 1 -- 172.21.15.118:0/1833431255 wait complete. 2024-03-24T00:16:54.469 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:54.468+0000 7fd456002740 1 -- start start 2024-03-24T00:16:54.469 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:54.468+0000 7fd456002740 1 Processor -- start 2024-03-24T00:16:54.470 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:54.469+0000 7fd456002740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5652c2f13bc0 0x5652c2f1cdb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:16:54.470 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:54.470+0000 7fd456002740 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5652c2f14580 0x5652c2f1d2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:16:54.470 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:54.470+0000 7fd450a00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5652c2f13bc0 0x5652c2f1cdb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:16:54.471 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:54.470+0000 7fd450a00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5652c2f13bc0 0x5652c2f1cdb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.118:36366/0 (socket says 172.21.15.118:36366) 2024-03-24T00:16:54.471 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:54.470+0000 7fd450a00640 1 -- 172.21.15.118:0/716937280 learned_addr learned my addr 172.21.15.118:0/716937280 (peer_addr_for_me v2:172.21.15.118:0/0) 2024-03-24T00:16:54.471 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:54.470+0000 7fd456002740 1 --2- 172.21.15.118:0/716937280 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5652c2f22320 0x5652c2f24710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:16:54.472 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:54.470+0000 7fd456002740 1 -- 172.21.15.118:0/716937280 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x5652c2f1b590 con 0x5652c2f14580 2024-03-24T00:16:54.472 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:54.470+0000 7fd456002740 1 -- 172.21.15.118:0/716937280 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x5652c2e330d0 con 0x5652c2f22320 2024-03-24T00:16:54.472 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:54.470+0000 7fd456002740 1 -- 172.21.15.118:0/716937280 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x5652c2f1b710 con 0x5652c2f13bc0 2024-03-24T00:16:54.472 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:54.470+0000 7fd456002740 10 client.0 _ll_register_callbacks cb 0x5652c2e37e80 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T00:16:54.472 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:54.470+0000 7fd44be00640 1 --2- 172.21.15.118:0/716937280 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5652c2f14580 0x5652c2f1d2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:16:54.472 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:54.471+0000 7fd451400640 1 --2- 172.21.15.118:0/716937280 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5652c2f22320 0x5652c2f24710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:16:54.472 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:54.471+0000 7fd450a00640 1 -- 172.21.15.118:0/716937280 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5652c2f22320 msgr2=0x5652c2f24710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:16:54.472 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:54.471+0000 7fd450a00640 1 --2- 172.21.15.118:0/716937280 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5652c2f22320 0x5652c2f24710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:16:54.472 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:54.471+0000 7fd450a00640 1 -- 172.21.15.118:0/716937280 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5652c2f14580 msgr2=0x5652c2f1d2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:16:54.472 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:54.471+0000 7fd450a00640 1 --2- 172.21.15.118:0/716937280 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5652c2f14580 0x5652c2f1d2f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:16:54.472 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:54.471+0000 7fd450a00640 1 -- 172.21.15.118:0/716937280 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5652c2b8b3f0 con 0x5652c2f13bc0 2024-03-24T00:16:54.473 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:54.471+0000 7fd44be00640 1 --2- 172.21.15.118:0/716937280 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5652c2f14580 0x5652c2f1d2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T00:16:54.473 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:54.471+0000 7fd450a00640 1 --2- 172.21.15.118:0/716937280 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5652c2f13bc0 0x5652c2f1cdb0 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7fd44000a2f0 tx=0x7fd440010f00 comp rx=0 tx=0).ready entity=mon.1 client_cookie=474fbe5ceb77eda1 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:16:54.473 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:54.471+0000 7fd432000640 10 client.0 ms_handle_connect on v2:172.21.15.193:3300/0 2024-03-24T00:16:54.473 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:54.472+0000 7fd432000640 1 -- 172.21.15.118:0/716937280 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd440035060 con 0x5652c2f13bc0 2024-03-24T00:16:54.474 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:54.472+0000 7fd432000640 1 -- 172.21.15.118:0/716937280 <== mon.1 v2:172.21.15.193:3300/0 2 ==== osd_map(90..90 src has 1..90) v4 ==== 8277+0+0 (secure 0 0 0) 0x7fd44001d930 con 0x5652c2f13bc0 2024-03-24T00:16:54.474 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:54.472+0000 7fd456002740 1 -- 172.21.15.118:0/716937280 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fd44001db30 con 0x5652c2f13bc0 2024-03-24T00:16:54.474 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:54.472+0000 7fd432000640 1 -- 172.21.15.118:0/716937280 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) v2 ==== 24+0+0 (secure 0 0 0) 0x7fd440021070 con 0x5652c2f13bc0 2024-03-24T00:16:54.474 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:54.472+0000 7fd456002740 10 client.25735 fetch_fsmap learned FSMap version 28 2024-03-24T00:16:54.474 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:54.472+0000 7fd456002740 1 -- 172.21.15.118:0/716937280 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) v3 -- 0x5652c2c557d0 con 0x5652c2f13bc0 2024-03-24T00:16:54.474 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:54.472+0000 7fd432000640 1 -- 172.21.15.118:0/716937280 <== mon.1 v2:172.21.15.193:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fd44000eaf0 con 0x5652c2f13bc0 2024-03-24T00:16:54.474 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:54.473+0000 7fd432000640 1 -- 172.21.15.118:0/716937280 <== mon.1 v2:172.21.15.193:3300/0 5 ==== fsmap(e 28) v1 ==== 7934+0+0 (secure 0 0 0) 0x7fd440004680 con 0x5652c2f13bc0 2024-03-24T00:16:54.474 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:54.473+0000 7fd456002740 10 client.25735 fetch_fsmap finished waiting for FSMap version 28 2024-03-24T00:16:54.474 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:54.473+0000 7fd456002740 10 client.25735 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-24T00:16:54.475 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:54.473+0000 7fd456002740 20 client.25735 populate_metadata read hostname 'smithi118' 2024-03-24T00:16:54.475 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:54.473+0000 7fd456002740 1 --2- 172.21.15.118:0/716937280 >> [v2:172.21.15.189:6838/4074484168,v1:172.21.15.189:6839/4074484168] conn(0x5652c2f28920 0x5652c2f2ad10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:16:54.475 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:54.473+0000 7fd456002740 4 client.25735 mds_command: new command op to 24569 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-24T00:16:54.475 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:54.473+0000 7fd456002740 1 -- 172.21.15.118:0/716937280 --> [v2:172.21.15.189:6838/4074484168,v1:172.21.15.189:6839/4074484168] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x7fd44001db30 con 0x5652c2f28920 2024-03-24T00:16:54.475 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:54.473+0000 7fd44be00640 1 --2- 172.21.15.118:0/716937280 >> [v2:172.21.15.189:6838/4074484168,v1:172.21.15.189:6839/4074484168] conn(0x5652c2f28920 0x5652c2f2ad10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:16:54.475 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:54.474+0000 7fd44be00640 1 --2- 172.21.15.118:0/716937280 >> [v2:172.21.15.189:6838/4074484168,v1:172.21.15.189:6839/4074484168] conn(0x5652c2f28920 0x5652c2f2ad10 crc :-1 s=READY pgs=57 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=2ee60b46c6454aee server_cookie=b62abb0c86143f29 in_seq=0 out_seq=0 2024-03-24T00:16:54.475 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:54.474+0000 7fd432000640 10 client.25735 ms_handle_connect on v2:172.21.15.189:6838/4074484168 2024-03-24T00:16:54.476 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:54.475+0000 7fd432000640 1 -- 172.21.15.118:0/716937280 <== mds.0 v2:172.21.15.189:6838/4074484168 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x7fd44001db30 con 0x5652c2f28920 2024-03-24T00:16:54.476 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:54.475+0000 7fd432000640 10 client.25735 handle_command_reply: tid=0 2024-03-24T00:16:54.476 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:54.475+0000 7fd456002740 1 client.25735 shutdown 2024-03-24T00:16:54.476 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:54.475+0000 7fd456002740 1 -- 172.21.15.118:0/716937280 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5652c2f13bc0 msgr2=0x5652c2f1cdb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:16:54.476 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:54.475+0000 7fd456002740 1 --2- 172.21.15.118:0/716937280 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5652c2f13bc0 0x5652c2f1cdb0 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7fd44000a2f0 tx=0x7fd440010f00 comp rx=0 tx=0).stop 2024-03-24T00:16:54.477 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:54.475+0000 7fd456002740 1 -- 172.21.15.118:0/716937280 shutdown_connections 2024-03-24T00:16:54.477 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:54.475+0000 7fd456002740 1 --2- 172.21.15.118:0/716937280 >> [v2:172.21.15.189:6838/4074484168,v1:172.21.15.189:6839/4074484168] conn(0x5652c2f28920 0x5652c2f2ad10 crc :-1 s=READY pgs=57 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:16:54.477 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:54.475+0000 7fd456002740 1 --2- 172.21.15.118:0/716937280 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5652c2f22320 0x5652c2f24710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:16:54.477 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:54.475+0000 7fd456002740 1 --2- 172.21.15.118:0/716937280 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5652c2f14580 0x5652c2f1d2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:16:54.477 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:54.475+0000 7fd456002740 1 --2- 172.21.15.118:0/716937280 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5652c2f13bc0 0x5652c2f1cdb0 unknown :-1 s=CLOSED pgs=255 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:16:54.477 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:54.475+0000 7fd456002740 1 -- 172.21.15.118:0/716937280 >> 172.21.15.118:0/716937280 conn(0x5652c2f0dde0 msgr2=0x5652c2f0d160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:16:54.477 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:54.475+0000 7fd432000640 0 client.25735 ms_handle_reset on v2:172.21.15.189:6838/4074484168 2024-03-24T00:16:54.477 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:54.475+0000 7fd456002740 1 -- 172.21.15.118:0/716937280 shutdown_connections 2024-03-24T00:16:54.477 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:54.475+0000 7fd456002740 1 -- 172.21.15.118:0/716937280 wait complete. 2024-03-24T00:16:54.477 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:54.476+0000 7fd456002740 20 client.25735 trim_cache size 0 max 16384 2024-03-24T00:16:54.501 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:54.501+0000 7fd456002740 1 Processor -- start 2024-03-24T00:16:54.501 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:54.501+0000 7fd456002740 1 -- start start 2024-03-24T00:16:54.502 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:54.501+0000 7fd456002740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5652c2f13bc0 0x5652c2f1d030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:16:54.502 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:54.501+0000 7fd456002740 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5652c2f14580 0x5652c2f16d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:16:54.502 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:54.501+0000 7fd456002740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5652c2f22320 0x5652c2f17240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:16:54.502 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:54.501+0000 7fd456002740 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd44c1aedd0 con 0x5652c2f14580 2024-03-24T00:16:54.502 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:54.501+0000 7fd456002740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd44c1b0750 con 0x5652c2f22320 2024-03-24T00:16:54.502 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:54.501+0000 7fd456002740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd44c1af430 con 0x5652c2f13bc0 2024-03-24T00:16:54.503 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:54.502+0000 7fd451400640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5652c2f22320 0x5652c2f17240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:16:54.503 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:54.502+0000 7fd451400640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5652c2f22320 0x5652c2f17240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.118:40576/0 (socket says 172.21.15.118:40576) 2024-03-24T00:16:54.503 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:54.502+0000 7fd451400640 1 -- 172.21.15.118:0/2011805862 learned_addr learned my addr 172.21.15.118:0/2011805862 (peer_addr_for_me v2:172.21.15.118:0/0) 2024-03-24T00:16:54.503 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:54.502+0000 7fd44be00640 1 --2- 172.21.15.118:0/2011805862 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5652c2f14580 0x5652c2f16d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:16:54.504 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:54.502+0000 7fd451400640 1 -- 172.21.15.118:0/2011805862 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5652c2f13bc0 msgr2=0x5652c2f1d030 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-03-24T00:16:54.504 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:54.502+0000 7fd451400640 1 --2- 172.21.15.118:0/2011805862 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5652c2f13bc0 0x5652c2f1d030 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:16:54.504 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:54.502+0000 7fd451400640 1 -- 172.21.15.118:0/2011805862 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5652c2f14580 msgr2=0x5652c2f16d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:16:54.504 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:54.502+0000 7fd451400640 1 --2- 172.21.15.118:0/2011805862 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5652c2f14580 0x5652c2f16d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:16:54.504 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:54.502+0000 7fd451400640 1 -- 172.21.15.118:0/2011805862 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5652c2f1be40 con 0x5652c2f22320 2024-03-24T00:16:54.504 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:54.502+0000 7fd451400640 1 --2- 172.21.15.118:0/2011805862 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5652c2f22320 0x5652c2f17240 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7fd444006010 tx=0x7fd4440a8040 comp rx=0 tx=0).ready entity=mon.2 client_cookie=34b8214cf644c0a2 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:16:54.504 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:54.502+0000 7fd43d200640 1 -- 172.21.15.118:0/2011805862 <== mon.2 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd4440bd630 con 0x5652c2f22320 2024-03-24T00:16:54.504 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:54.503+0000 7fd43d200640 1 -- 172.21.15.118:0/2011805862 <== mon.2 v2:172.21.15.189:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fd4440c1780 con 0x5652c2f22320 2024-03-24T00:16:54.504 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:54.503+0000 7fd43d200640 1 -- 172.21.15.118:0/2011805862 <== mon.2 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd444046310 con 0x5652c2f22320 2024-03-24T00:16:54.505 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:54.504+0000 7fd456002740 1 -- 172.21.15.118:0/2011805862 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5652c2f22320 msgr2=0x5652c2f17240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:16:54.505 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:54.504+0000 7fd456002740 1 --2- 172.21.15.118:0/2011805862 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5652c2f22320 0x5652c2f17240 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7fd444006010 tx=0x7fd4440a8040 comp rx=0 tx=0).stop 2024-03-24T00:16:54.505 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:54.504+0000 7fd456002740 1 -- 172.21.15.118:0/2011805862 shutdown_connections 2024-03-24T00:16:54.505 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:54.504+0000 7fd456002740 1 --2- 172.21.15.118:0/2011805862 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5652c2f22320 0x5652c2f17240 unknown :-1 s=CLOSED pgs=263 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:16:54.505 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:54.504+0000 7fd456002740 1 --2- 172.21.15.118:0/2011805862 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5652c2f14580 0x5652c2f16d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:16:54.505 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:54.504+0000 7fd456002740 1 --2- 172.21.15.118:0/2011805862 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5652c2f13bc0 0x5652c2f1d030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:16:54.505 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:54.504+0000 7fd456002740 1 -- 172.21.15.118:0/2011805862 >> 172.21.15.118:0/2011805862 conn(0x5652c2f0dde0 msgr2=0x5652c2f0d160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:16:54.506 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:54.505+0000 7fd456002740 1 -- 172.21.15.118:0/2011805862 shutdown_connections 2024-03-24T00:16:54.506 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:54.505+0000 7fd456002740 1 -- 172.21.15.118:0/2011805862 wait complete. 2024-03-24T00:16:54.506 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:54.505+0000 7fd456002740 1 -- start start 2024-03-24T00:16:54.506 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:54.505+0000 7fd456002740 1 Processor -- start 2024-03-24T00:16:54.509 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:54.506+0000 7fd456002740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5652c2f14580 0x5652c2f37c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:16:54.509 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:54.506+0000 7fd456002740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5652c2f22320 0x5652c2f38180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:16:54.509 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:54.506+0000 7fd456002740 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5652c2f81570 0x5652c2f83960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:16:54.509 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:54.506+0000 7fd456002740 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd44c1b00f0 con 0x5652c2f81570 2024-03-24T00:16:54.509 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:54.506+0000 7fd456002740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd44c1afa90 con 0x5652c2f22320 2024-03-24T00:16:54.509 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:54.506+0000 7fd456002740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd44c1acf90 con 0x5652c2f14580 2024-03-24T00:16:54.509 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:54.506+0000 7fd456002740 10 client.0 _ll_register_callbacks cb 0x7fd438009340 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T00:16:54.509 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:54.506+0000 7fd450a00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5652c2f14580 0x5652c2f37c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:16:54.509 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:54.506+0000 7fd450a00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5652c2f14580 0x5652c2f37c40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.118:36392/0 (socket says 172.21.15.118:36392) 2024-03-24T00:16:54.509 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:54.506+0000 7fd450a00640 1 -- 172.21.15.118:0/84086680 learned_addr learned my addr 172.21.15.118:0/84086680 (peer_addr_for_me v2:172.21.15.118:0/0) 2024-03-24T00:16:54.510 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:54.506+0000 7fd44be00640 1 --2- 172.21.15.118:0/84086680 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5652c2f22320 0x5652c2f38180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:16:54.510 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:54.506+0000 7fd451400640 1 --2- 172.21.15.118:0/84086680 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5652c2f81570 0x5652c2f83960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:16:54.510 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:54.507+0000 7fd450a00640 1 -- 172.21.15.118:0/84086680 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5652c2f22320 msgr2=0x5652c2f38180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:16:54.510 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:54.507+0000 7fd450a00640 1 --2- 172.21.15.118:0/84086680 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5652c2f22320 0x5652c2f38180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:16:54.510 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:54.507+0000 7fd450a00640 1 -- 172.21.15.118:0/84086680 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5652c2f81570 msgr2=0x5652c2f83960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:16:54.510 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:54.507+0000 7fd450a00640 1 --2- 172.21.15.118:0/84086680 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5652c2f81570 0x5652c2f83960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:16:54.510 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:54.507+0000 7fd450a00640 1 -- 172.21.15.118:0/84086680 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5652c2f83ea0 con 0x5652c2f14580 2024-03-24T00:16:54.510 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:54.507+0000 7fd450a00640 1 --2- 172.21.15.118:0/84086680 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5652c2f14580 0x5652c2f37c40 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7fd44000e7c0 tx=0x7fd44001d850 comp rx=0 tx=0).ready entity=mon.1 client_cookie=840e37c972ac19b server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:16:54.510 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:54.507+0000 7fd432000640 10 client.0 ms_handle_connect on v2:172.21.15.193:3300/0 2024-03-24T00:16:54.510 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:54.507+0000 7fd432000640 1 -- 172.21.15.118:0/84086680 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd440014070 con 0x5652c2f14580 2024-03-24T00:16:54.510 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:54.507+0000 7fd432000640 1 -- 172.21.15.118:0/84086680 <== mon.1 v2:172.21.15.193:3300/0 2 ==== osd_map(90..90 src has 1..90) v4 ==== 8277+0+0 (secure 0 0 0) 0x7fd440004f70 con 0x5652c2f14580 2024-03-24T00:16:54.510 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:54.507+0000 7fd456002740 1 -- 172.21.15.118:0/84086680 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fd4440c1780 con 0x5652c2f14580 2024-03-24T00:16:54.510 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:54.508+0000 7fd432000640 1 -- 172.21.15.118:0/84086680 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) v2 ==== 24+0+0 (secure 0 0 0) 0x7fd440035040 con 0x5652c2f14580 2024-03-24T00:16:54.510 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:54.508+0000 7fd456002740 10 client.25738 fetch_fsmap learned FSMap version 28 2024-03-24T00:16:54.511 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:54.508+0000 7fd456002740 1 -- 172.21.15.118:0/84086680 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) v3 -- 0x5652c2f85040 con 0x5652c2f14580 2024-03-24T00:16:54.511 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:54.508+0000 7fd432000640 1 -- 172.21.15.118:0/84086680 <== mon.1 v2:172.21.15.193:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fd4440c1780 con 0x5652c2f14580 2024-03-24T00:16:54.511 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:54.508+0000 7fd432000640 1 -- 172.21.15.118:0/84086680 <== mon.1 v2:172.21.15.193:3300/0 5 ==== fsmap(e 28) v1 ==== 7934+0+0 (secure 0 0 0) 0x7fd440026c30 con 0x5652c2f14580 2024-03-24T00:16:54.511 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:54.508+0000 7fd456002740 10 client.25738 fetch_fsmap finished waiting for FSMap version 28 2024-03-24T00:16:54.511 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:54.508+0000 7fd456002740 10 client.25738 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-24T00:16:54.511 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:54.508+0000 7fd456002740 20 client.25738 populate_metadata read hostname 'smithi118' 2024-03-24T00:16:54.511 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:54.508+0000 7fd456002740 1 --2- 172.21.15.118:0/84086680 >> [v2:172.21.15.189:6838/4074484168,v1:172.21.15.189:6839/4074484168] conn(0x5652c2f858d0 0x5652c2f87cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:16:54.511 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:54.508+0000 7fd456002740 4 client.25738 mds_command: new command op to 24569 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-24T00:16:54.511 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:54.508+0000 7fd456002740 1 -- 172.21.15.118:0/84086680 --> [v2:172.21.15.189:6838/4074484168,v1:172.21.15.189:6839/4074484168] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x5652c2e84800 con 0x5652c2f858d0 2024-03-24T00:16:54.511 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:54.509+0000 7fd44be00640 1 --2- 172.21.15.118:0/84086680 >> [v2:172.21.15.189:6838/4074484168,v1:172.21.15.189:6839/4074484168] conn(0x5652c2f858d0 0x5652c2f87cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:16:54.512 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:54.509+0000 7fd432000640 10 client.25738 ms_handle_connect on v2:172.21.15.189:6838/4074484168 2024-03-24T00:16:54.513 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:54.509+0000 7fd44be00640 1 --2- 172.21.15.118:0/84086680 >> [v2:172.21.15.189:6838/4074484168,v1:172.21.15.189:6839/4074484168] conn(0x5652c2f858d0 0x5652c2f87cf0 crc :-1 s=READY pgs=58 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=45e3343b7768d9fc server_cookie=d2589a59249b90fa in_seq=0 out_seq=0 2024-03-24T00:16:54.513 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:54.509+0000 7fd432000640 1 -- 172.21.15.118:0/84086680 <== mds.0 v2:172.21.15.189:6838/4074484168 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x5652c2e84800 con 0x5652c2f858d0 2024-03-24T00:16:54.513 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:54.509+0000 7fd432000640 10 client.25738 handle_command_reply: tid=0 2024-03-24T00:16:54.513 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:54.509+0000 7fd456002740 1 client.25738 shutdown 2024-03-24T00:16:54.513 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:54.510+0000 7fd456002740 1 -- 172.21.15.118:0/84086680 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5652c2f14580 msgr2=0x5652c2f37c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:16:54.513 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:54.510+0000 7fd456002740 1 --2- 172.21.15.118:0/84086680 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5652c2f14580 0x5652c2f37c40 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7fd44000e7c0 tx=0x7fd44001d850 comp rx=0 tx=0).stop 2024-03-24T00:16:54.513 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:54.510+0000 7fd456002740 1 -- 172.21.15.118:0/84086680 shutdown_connections 2024-03-24T00:16:54.513 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:54.510+0000 7fd456002740 1 --2- 172.21.15.118:0/84086680 >> [v2:172.21.15.189:6838/4074484168,v1:172.21.15.189:6839/4074484168] conn(0x5652c2f858d0 0x5652c2f87cf0 crc :-1 s=READY pgs=58 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:16:54.513 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:54.510+0000 7fd456002740 1 --2- 172.21.15.118:0/84086680 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5652c2f81570 0x5652c2f83960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:16:54.513 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:54.510+0000 7fd456002740 1 --2- 172.21.15.118:0/84086680 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5652c2f22320 0x5652c2f38180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:16:54.513 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:54.510+0000 7fd432000640 0 client.25738 ms_handle_reset on v2:172.21.15.189:6838/4074484168 2024-03-24T00:16:54.513 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:54.510+0000 7fd456002740 1 --2- 172.21.15.118:0/84086680 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5652c2f14580 0x5652c2f37c40 unknown :-1 s=CLOSED pgs=256 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:16:54.513 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:54.510+0000 7fd456002740 1 -- 172.21.15.118:0/84086680 >> 172.21.15.118:0/84086680 conn(0x5652c2f0dde0 msgr2=0x5652c2f118d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:16:54.513 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:54.510+0000 7fd456002740 1 -- 172.21.15.118:0/84086680 shutdown_connections 2024-03-24T00:16:54.513 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:54.510+0000 7fd456002740 1 -- 172.21.15.118:0/84086680 wait complete. 2024-03-24T00:16:54.514 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:54.510+0000 7fd456002740 20 client.25738 trim_cache size 0 max 16384 2024-03-24T00:16:54.514 INFO:teuthology.orchestra.run.smithi118.stdout:[] 2024-03-24T00:16:54.515 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:54.511+0000 7fd451e00640 1 -- 172.21.15.118:0/608827690 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7fd4340a2910 msgr2=0x7fd4340a4dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:16:54.515 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:54.511+0000 7fd451e00640 1 --2- 172.21.15.118:0/608827690 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7fd4340a2910 0x7fd4340a4dd0 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x7fd440005e10 tx=0x7fd440005da0 comp rx=0 tx=0).stop 2024-03-24T00:16:54.515 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:54.511+0000 7fd451e00640 1 -- 172.21.15.118:0/608827690 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd44c105110 msgr2=0x7fd44c1a5310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:16:54.515 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:54.511+0000 7fd451e00640 1 --2- 172.21.15.118:0/608827690 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd44c105110 0x7fd44c1a5310 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7fd44402c640 tx=0x7fd444035d10 comp rx=0 tx=0).stop 2024-03-24T00:16:54.515 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:54.511+0000 7fd451e00640 1 -- 172.21.15.118:0/608827690 shutdown_connections 2024-03-24T00:16:54.515 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:54.511+0000 7fd451e00640 1 --2- 172.21.15.118:0/608827690 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7fd4340a2910 0x7fd4340a4dd0 unknown :-1 s=CLOSED pgs=291 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:16:54.515 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:54.511+0000 7fd451e00640 1 --2- 172.21.15.118:0/608827690 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd44c105110 0x7fd44c1a5310 unknown :-1 s=CLOSED pgs=262 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:16:54.515 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:54.511+0000 7fd451e00640 1 --2- 172.21.15.118:0/608827690 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fd44c102760 0x7fd44c19e2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:16:54.515 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:54.511+0000 7fd451e00640 1 --2- 172.21.15.118:0/608827690 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fd44c0ffdf0 0x7fd44c19dd60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:16:54.515 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:54.511+0000 7fd451e00640 1 -- 172.21.15.118:0/608827690 >> 172.21.15.118:0/608827690 conn(0x7fd44c0fba10 msgr2=0x7fd44c100770 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:16:54.515 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:54.512+0000 7fd451e00640 1 -- 172.21.15.118:0/608827690 shutdown_connections 2024-03-24T00:16:54.515 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:54.513+0000 7fd451e00640 1 -- 172.21.15.118:0/608827690 wait complete. 2024-03-24T00:16:55.195 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:16:54 smithi118 ceph-mon[31535]: pgmap v434: 97 pgs: 97 active+clean; 673 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 255 B/s wr, 10 op/s 2024-03-24T00:16:55.203 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:16:54 smithi189 ceph-mon[36187]: pgmap v434: 97 pgs: 97 active+clean; 673 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 255 B/s wr, 10 op/s 2024-03-24T00:16:55.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:16:54 smithi193 ceph-mon[36878]: pgmap v434: 97 pgs: 97 active+clean; 673 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 255 B/s wr, 10 op/s 2024-03-24T00:16:55.536 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-03-24T00:16:55.537 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T00:16:55.537 DEBUG:teuthology.orchestra.run.smithi118:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2024-03-24T00:16:55.790 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:55.789+0000 7f8df0200640 1 -- 172.21.15.118:0/1370672467 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8de81017a0 msgr2=0x7f8de8101c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:16:55.790 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:55.789+0000 7f8df0200640 1 --2- 172.21.15.118:0/1370672467 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8de81017a0 0x7f8de8101c20 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7f8dd4009900 tx=0x7f8dd402bc40 comp rx=0 tx=0).stop 2024-03-24T00:16:55.790 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:55.789+0000 7f8df0200640 1 -- 172.21.15.118:0/1370672467 shutdown_connections 2024-03-24T00:16:55.790 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:55.789+0000 7f8df0200640 1 --2- 172.21.15.118:0/1370672467 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f8de8102160 0x7f8de81089f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:16:55.790 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:55.789+0000 7f8df0200640 1 --2- 172.21.15.118:0/1370672467 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8de81017a0 0x7f8de8101c20 unknown :-1 s=CLOSED pgs=264 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:16:55.790 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:55.789+0000 7f8df0200640 1 --2- 172.21.15.118:0/1370672467 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f8de81005b0 0x7f8de81009d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:16:55.791 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:55.789+0000 7f8df0200640 1 -- 172.21.15.118:0/1370672467 >> 172.21.15.118:0/1370672467 conn(0x7f8de80fbb40 msgr2=0x7f8de80fdf60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:16:55.791 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:55.789+0000 7f8df0200640 1 -- 172.21.15.118:0/1370672467 shutdown_connections 2024-03-24T00:16:55.791 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:55.789+0000 7f8df0200640 1 -- 172.21.15.118:0/1370672467 wait complete. 2024-03-24T00:16:55.791 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:55.790+0000 7f8df0200640 1 Processor -- start 2024-03-24T00:16:55.791 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:55.790+0000 7f8df0200640 1 -- start start 2024-03-24T00:16:55.792 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:55.791+0000 7f8df0200640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f8de81005b0 0x7f8de8199c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:16:55.792 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:55.791+0000 7f8df0200640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f8de81017a0 0x7f8de819a160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:16:55.792 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:55.791+0000 7f8df0200640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8de8102160 0x7f8de81a1190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:16:55.792 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:55.791+0000 7f8df0200640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8de810d390 con 0x7f8de81017a0 2024-03-24T00:16:55.792 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:55.791+0000 7f8df0200640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8de810d210 con 0x7f8de8102160 2024-03-24T00:16:55.792 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:55.791+0000 7f8df0200640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8de810d510 con 0x7f8de81005b0 2024-03-24T00:16:55.793 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:55.792+0000 7f8ded800640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f8de81005b0 0x7f8de8199c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:16:55.793 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:55.792+0000 7f8ded800640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f8de81005b0 0x7f8de8199c20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.118:36414/0 (socket says 172.21.15.118:36414) 2024-03-24T00:16:55.793 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:55.792+0000 7f8ded800640 1 -- 172.21.15.118:0/390351025 learned_addr learned my addr 172.21.15.118:0/390351025 (peer_addr_for_me v2:172.21.15.118:0/0) 2024-03-24T00:16:55.793 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:55.792+0000 7f8dece00640 1 --2- 172.21.15.118:0/390351025 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f8de81017a0 0x7f8de819a160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:16:55.793 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:55.793+0000 7f8dee200640 1 --2- 172.21.15.118:0/390351025 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8de8102160 0x7f8de81a1190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:16:55.794 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:55.793+0000 7f8dece00640 1 -- 172.21.15.118:0/390351025 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f8de81005b0 msgr2=0x7f8de8199c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:16:55.794 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:55.793+0000 7f8dece00640 1 --2- 172.21.15.118:0/390351025 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f8de81005b0 0x7f8de8199c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:16:55.794 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:55.793+0000 7f8dece00640 1 -- 172.21.15.118:0/390351025 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8de8102160 msgr2=0x7f8de81a1190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:16:55.794 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:55.793+0000 7f8dece00640 1 --2- 172.21.15.118:0/390351025 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8de8102160 0x7f8de81a1190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:16:55.794 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:55.793+0000 7f8dece00640 1 -- 172.21.15.118:0/390351025 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8de81a1830 con 0x7f8de81017a0 2024-03-24T00:16:55.794 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:55.793+0000 7f8dece00640 1 --2- 172.21.15.118:0/390351025 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f8de81017a0 0x7f8de819a160 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7f8dd4036040 tx=0x7f8dd402c660 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:16:55.798 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:55.797+0000 7f8de6000640 1 -- 172.21.15.118:0/390351025 <== mon.0 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8dd4041070 con 0x7f8de81017a0 2024-03-24T00:16:55.798 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:55.797+0000 7f8de6000640 1 -- 172.21.15.118:0/390351025 <== mon.0 v2:172.21.15.118:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f8dd4004450 con 0x7f8de81017a0 2024-03-24T00:16:55.798 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:55.797+0000 7f8df0200640 1 -- 172.21.15.118:0/390351025 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8de81a1ac0 con 0x7f8de81017a0 2024-03-24T00:16:55.799 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:55.797+0000 7f8df0200640 1 -- 172.21.15.118:0/390351025 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8de81a1f80 con 0x7f8de81017a0 2024-03-24T00:16:55.799 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:55.798+0000 7f8de6000640 1 -- 172.21.15.118:0/390351025 <== mon.0 v2:172.21.15.118:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8dd4003e60 con 0x7f8de81017a0 2024-03-24T00:16:55.804 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:55.804+0000 7f8df2f8a740 1 Processor -- start 2024-03-24T00:16:55.804 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:55.804+0000 7f8df2f8a740 1 -- start start 2024-03-24T00:16:55.806 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:55.805+0000 7f8df2f8a740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56181d2ea0a0 0x56181d2ea4c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:16:55.806 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:55.805+0000 7f8df2f8a740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56181d2eaa90 0x56181d2ef590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:16:55.806 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:55.805+0000 7f8df2f8a740 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x56181d2efc00 0x56181d2f2090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:16:55.806 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:55.805+0000 7f8df2f8a740 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x56181d24feb0 con 0x56181d2efc00 2024-03-24T00:16:55.806 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:55.805+0000 7f8df2f8a740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x56181d1f8a60 con 0x56181d2eaa90 2024-03-24T00:16:55.806 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:55.805+0000 7f8df2f8a740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x56181d24bf50 con 0x56181d2ea0a0 2024-03-24T00:16:55.806 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:55.805+0000 7f8dee200640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56181d2ea0a0 0x56181d2ea4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:16:55.806 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:55.805+0000 7f8ded800640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56181d2eaa90 0x56181d2ef590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:16:55.806 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:55.805+0000 7f8dee200640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56181d2ea0a0 0x56181d2ea4c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.118:36424/0 (socket says 172.21.15.118:36424) 2024-03-24T00:16:55.806 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:55.805+0000 7f8ded800640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56181d2eaa90 0x56181d2ef590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.118:40622/0 (socket says 172.21.15.118:40622) 2024-03-24T00:16:55.807 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:55.805+0000 7f8dee200640 1 -- 172.21.15.118:0/192439155 learned_addr learned my addr 172.21.15.118:0/192439155 (peer_addr_for_me v2:172.21.15.118:0/0) 2024-03-24T00:16:55.807 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:55.806+0000 7f8dee200640 1 -- 172.21.15.118:0/192439155 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56181d2eaa90 msgr2=0x56181d2ef590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:16:55.807 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:55.806+0000 7f8dee200640 1 --2- 172.21.15.118:0/192439155 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56181d2eaa90 0x56181d2ef590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:16:55.807 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:55.806+0000 7f8dee200640 1 -- 172.21.15.118:0/192439155 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x56181d2efc00 msgr2=0x56181d2f2090 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-24T00:16:55.808 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:55.806+0000 7f8dee200640 1 --2- 172.21.15.118:0/192439155 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x56181d2efc00 0x56181d2f2090 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:16:55.808 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:55.806+0000 7f8dee200640 1 -- 172.21.15.118:0/192439155 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x56181d02c7e0 con 0x56181d2ea0a0 2024-03-24T00:16:55.808 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:55.806+0000 7f8ded800640 1 --2- 172.21.15.118:0/192439155 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56181d2eaa90 0x56181d2ef590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T00:16:55.808 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:55.806+0000 7f8dee200640 1 --2- 172.21.15.118:0/192439155 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56181d2ea0a0 0x56181d2ea4c0 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7f8de0009fd0 tx=0x7f8de00118d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=73fa9209bf2b141a server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:16:55.808 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:55.806+0000 7f8ddaa00640 1 -- 172.21.15.118:0/192439155 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8de0004d20 con 0x56181d2ea0a0 2024-03-24T00:16:55.808 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:55.807+0000 7f8de6000640 1 -- 172.21.15.118:0/390351025 <== mon.0 v2:172.21.15.118:3300/0 4 ==== mgrmap(e 19) v1 ==== 137342+0+0 (secure 0 0 0) 0x7f8dd4044070 con 0x7f8de81017a0 2024-03-24T00:16:55.808 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:55.807+0000 7f8ddaa00640 1 -- 172.21.15.118:0/192439155 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f8de0004050 con 0x56181d2ea0a0 2024-03-24T00:16:55.808 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:55.807+0000 7f8ddaa00640 1 -- 172.21.15.118:0/192439155 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8de0005280 con 0x56181d2ea0a0 2024-03-24T00:16:55.809 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:55.807+0000 7f8df2f8a740 1 -- 172.21.15.118:0/192439155 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56181d2ea0a0 msgr2=0x56181d2ea4c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:16:55.809 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:55.808+0000 7f8df2f8a740 1 --2- 172.21.15.118:0/192439155 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56181d2ea0a0 0x56181d2ea4c0 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7f8de0009fd0 tx=0x7f8de00118d0 comp rx=0 tx=0).stop 2024-03-24T00:16:55.809 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:55.808+0000 7f8df2f8a740 1 -- 172.21.15.118:0/192439155 shutdown_connections 2024-03-24T00:16:55.809 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:55.808+0000 7f8df2f8a740 1 --2- 172.21.15.118:0/192439155 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x56181d2efc00 0x56181d2f2090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:16:55.809 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:55.808+0000 7f8df2f8a740 1 --2- 172.21.15.118:0/192439155 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56181d2eaa90 0x56181d2ef590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:16:55.809 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:55.808+0000 7f8df2f8a740 1 --2- 172.21.15.118:0/192439155 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56181d2ea0a0 0x56181d2ea4c0 unknown :-1 s=CLOSED pgs=257 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:16:55.809 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:55.808+0000 7f8df2f8a740 1 -- 172.21.15.118:0/192439155 >> 172.21.15.118:0/192439155 conn(0x56181d2e5610 msgr2=0x56181d2e5a10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:16:55.809 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:55.808+0000 7f8df2f8a740 1 -- 172.21.15.118:0/192439155 shutdown_connections 2024-03-24T00:16:55.810 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:55.809+0000 7f8df2f8a740 1 -- 172.21.15.118:0/192439155 wait complete. 2024-03-24T00:16:55.810 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:55.809+0000 7f8de6000640 1 --2- 172.21.15.118:0/390351025 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7f8dd00a28f0 0x7f8dd00a4db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:16:55.810 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:55.809+0000 7f8df2f8a740 1 -- start start 2024-03-24T00:16:55.810 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:55.809+0000 7f8de6000640 1 -- 172.21.15.118:0/390351025 <== mon.0 v2:172.21.15.118:3300/0 5 ==== osd_map(90..90 src has 1..90) v4 ==== 8277+0+0 (secure 0 0 0) 0x7f8dd40f0450 con 0x7f8de81017a0 2024-03-24T00:16:55.810 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:55.809+0000 7f8df2f8a740 1 Processor -- start 2024-03-24T00:16:55.810 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:55.810+0000 7f8dee200640 1 --2- 172.21.15.118:0/390351025 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7f8dd00a28f0 0x7f8dd00a4db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:16:55.811 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:55.810+0000 7f8df2f8a740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56181d2ea0a0 0x56181d2f42f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:16:55.811 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:55.810+0000 7f8df2f8a740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56181d2eaa90 0x56181d2f4830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:16:55.811 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:55.810+0000 7f8df2f8a740 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x56181d2efc00 0x56181d2fb870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:16:55.811 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:55.810+0000 7f8df2f8a740 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x56181d2f2bb0 con 0x56181d2efc00 2024-03-24T00:16:55.811 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:55.810+0000 7f8df2f8a740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x56181d20beb0 con 0x56181d2eaa90 2024-03-24T00:16:55.811 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:55.810+0000 7f8df2f8a740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x56181d2f2d30 con 0x56181d2ea0a0 2024-03-24T00:16:55.811 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:55.810+0000 7f8df2f8a740 10 client.0 _ll_register_callbacks cb 0x56181d200150 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T00:16:55.813 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:55.811+0000 7f8dece00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56181d2eaa90 0x56181d2f4830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:16:55.813 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:55.811+0000 7f8dece00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56181d2eaa90 0x56181d2f4830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.118:40632/0 (socket says 172.21.15.118:40632) 2024-03-24T00:16:55.813 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:55.811+0000 7f8dece00640 1 -- 172.21.15.118:0/393167773 learned_addr learned my addr 172.21.15.118:0/393167773 (peer_addr_for_me v2:172.21.15.118:0/0) 2024-03-24T00:16:55.813 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:55.811+0000 7f8dee200640 1 --2- 172.21.15.118:0/393167773 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x56181d2efc00 0x56181d2fb870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:16:55.813 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:55.811+0000 7f8dece00640 1 -- 172.21.15.118:0/393167773 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56181d2ea0a0 msgr2=0x56181d2f42f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:16:55.813 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:55.811+0000 7f8ded800640 1 --2- 172.21.15.118:0/393167773 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56181d2ea0a0 0x56181d2f42f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:16:55.813 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:55.811+0000 7f8dece00640 1 --2- 172.21.15.118:0/393167773 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56181d2ea0a0 0x56181d2f42f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:16:55.813 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:55.811+0000 7f8dece00640 1 -- 172.21.15.118:0/393167773 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x56181d2efc00 msgr2=0x56181d2fb870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:16:55.813 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:55.811+0000 7f8dece00640 1 --2- 172.21.15.118:0/393167773 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x56181d2efc00 0x56181d2fb870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:16:55.814 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:55.811+0000 7f8dece00640 1 -- 172.21.15.118:0/393167773 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x56181d08cf70 con 0x56181d2eaa90 2024-03-24T00:16:55.814 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:55.811+0000 7f8dee200640 1 --2- 172.21.15.118:0/393167773 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x56181d2efc00 0x56181d2fb870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-24T00:16:55.814 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:55.812+0000 7f8dece00640 1 --2- 172.21.15.118:0/393167773 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56181d2eaa90 0x56181d2f4830 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7f8dd40c4f80 tx=0x7f8dd40bcc50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a3c773b467df4337 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:16:55.814 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:55.812+0000 7f8dcf400640 10 client.0 ms_handle_connect on v2:172.21.15.189:3300/0 2024-03-24T00:16:55.814 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:55.812+0000 7f8dcf400640 1 -- 172.21.15.118:0/393167773 <== mon.2 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8dd404b040 con 0x56181d2eaa90 2024-03-24T00:16:55.814 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:55.812+0000 7f8dee200640 1 --2- 172.21.15.118:0/390351025 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7f8dd00a28f0 0x7f8dd00a4db0 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 crypto rx=0x7f8de0009de0 tx=0x7f8de0005fd0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:16:55.814 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:55.812+0000 7f8dcf400640 1 -- 172.21.15.118:0/393167773 <== mon.2 v2:172.21.15.189:3300/0 2 ==== osd_map(90..90 src has 1..90) v4 ==== 8277+0+0 (secure 0 0 0) 0x7f8dd40c5220 con 0x56181d2eaa90 2024-03-24T00:16:55.814 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:55.812+0000 7f8df2f8a740 1 -- 172.21.15.118:0/393167773 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f8de0004050 con 0x56181d2eaa90 2024-03-24T00:16:55.814 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:55.812+0000 7f8dcf400640 1 -- 172.21.15.118:0/393167773 <== mon.2 v2:172.21.15.189:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) v2 ==== 24+0+0 (secure 0 0 0) 0x7f8dd40bcdc0 con 0x56181d2eaa90 2024-03-24T00:16:55.814 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:55.812+0000 7f8df2f8a740 10 client.25757 fetch_fsmap learned FSMap version 28 2024-03-24T00:16:55.814 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:55.812+0000 7f8df2f8a740 1 -- 172.21.15.118:0/393167773 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) v3 -- 0x56181cf24170 con 0x56181d2eaa90 2024-03-24T00:16:55.814 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:55.812+0000 7f8dcf400640 1 -- 172.21.15.118:0/393167773 <== mon.2 v2:172.21.15.189:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f8de0004050 con 0x56181d2eaa90 2024-03-24T00:16:55.815 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:55.813+0000 7f8dcf400640 1 -- 172.21.15.118:0/393167773 <== mon.2 v2:172.21.15.189:3300/0 5 ==== fsmap(e 28) v1 ==== 7934+0+0 (secure 0 0 0) 0x7f8dd40bbd10 con 0x56181d2eaa90 2024-03-24T00:16:55.815 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:55.813+0000 7f8df2f8a740 10 client.25757 fetch_fsmap finished waiting for FSMap version 28 2024-03-24T00:16:55.816 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:55.813+0000 7f8df2f8a740 10 client.25757 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-24T00:16:55.816 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:55.813+0000 7f8df2f8a740 20 client.25757 populate_metadata read hostname 'smithi118' 2024-03-24T00:16:55.816 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:55.813+0000 7f8df2f8a740 1 --2- 172.21.15.118:0/393167773 >> [v2:172.21.15.189:6838/4074484168,v1:172.21.15.189:6839/4074484168] conn(0x56181d2fffc0 0x56181d302400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:16:55.816 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:55.813+0000 7f8df2f8a740 4 client.25757 mds_command: new command op to 24569 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-24T00:16:55.816 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:55.813+0000 7f8df2f8a740 1 -- 172.21.15.118:0/393167773 --> [v2:172.21.15.189:6838/4074484168,v1:172.21.15.189:6839/4074484168] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x56181d25d3f0 con 0x56181d2fffc0 2024-03-24T00:16:55.816 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:55.814+0000 7f8ded800640 1 --2- 172.21.15.118:0/393167773 >> [v2:172.21.15.189:6838/4074484168,v1:172.21.15.189:6839/4074484168] conn(0x56181d2fffc0 0x56181d302400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:16:55.816 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:55.814+0000 7f8ded800640 1 --2- 172.21.15.118:0/393167773 >> [v2:172.21.15.189:6838/4074484168,v1:172.21.15.189:6839/4074484168] conn(0x56181d2fffc0 0x56181d302400 crc :-1 s=READY pgs=59 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=45fe4c165c728ec0 server_cookie=8cf44c29c1999c9f in_seq=0 out_seq=0 2024-03-24T00:16:55.816 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:55.814+0000 7f8dcf400640 10 client.25757 ms_handle_connect on v2:172.21.15.189:6838/4074484168 2024-03-24T00:16:55.816 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:55.815+0000 7f8dcf400640 1 -- 172.21.15.118:0/393167773 <== mds.0 v2:172.21.15.189:6838/4074484168 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x56181d25d3f0 con 0x56181d2fffc0 2024-03-24T00:16:55.816 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:55.815+0000 7f8dcf400640 10 client.25757 handle_command_reply: tid=0 2024-03-24T00:16:55.816 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:55.815+0000 7f8df2f8a740 1 client.25757 shutdown 2024-03-24T00:16:55.817 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:55.815+0000 7f8df2f8a740 1 -- 172.21.15.118:0/393167773 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56181d2eaa90 msgr2=0x56181d2f4830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:16:55.817 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:55.815+0000 7f8df2f8a740 1 --2- 172.21.15.118:0/393167773 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56181d2eaa90 0x56181d2f4830 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7f8dd40c4f80 tx=0x7f8dd40bcc50 comp rx=0 tx=0).stop 2024-03-24T00:16:55.817 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:55.816+0000 7f8df2f8a740 1 -- 172.21.15.118:0/393167773 shutdown_connections 2024-03-24T00:16:55.818 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:55.816+0000 7f8df2f8a740 1 --2- 172.21.15.118:0/393167773 >> [v2:172.21.15.189:6838/4074484168,v1:172.21.15.189:6839/4074484168] conn(0x56181d2fffc0 0x56181d302400 crc :-1 s=READY pgs=59 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:16:55.818 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:55.816+0000 7f8df2f8a740 1 --2- 172.21.15.118:0/393167773 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x56181d2efc00 0x56181d2fb870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:16:55.818 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:55.816+0000 7f8df2f8a740 1 --2- 172.21.15.118:0/393167773 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56181d2eaa90 0x56181d2f4830 unknown :-1 s=CLOSED pgs=265 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:16:55.818 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:55.816+0000 7f8df2f8a740 1 --2- 172.21.15.118:0/393167773 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56181d2ea0a0 0x56181d2f42f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:16:55.818 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:55.816+0000 7f8df2f8a740 1 -- 172.21.15.118:0/393167773 >> 172.21.15.118:0/393167773 conn(0x56181d2e5610 msgr2=0x56181d2e4830 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:16:55.818 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:55.816+0000 7f8dcf400640 0 client.25757 ms_handle_reset on v2:172.21.15.189:6838/4074484168 2024-03-24T00:16:55.818 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:55.816+0000 7f8df2f8a740 1 -- 172.21.15.118:0/393167773 shutdown_connections 2024-03-24T00:16:55.818 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:55.816+0000 7f8df2f8a740 1 -- 172.21.15.118:0/393167773 wait complete. 2024-03-24T00:16:55.818 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:55.816+0000 7f8df2f8a740 20 client.25757 trim_cache size 0 max 16384 2024-03-24T00:16:55.850 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:55.849+0000 7f8df2f8a740 1 Processor -- start 2024-03-24T00:16:55.888 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:55.849+0000 7f8df2f8a740 1 -- start start 2024-03-24T00:16:55.892 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:55.850+0000 7f8df2f8a740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56181d2ea0a0 0x56181d2eb530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:16:55.892 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:55.850+0000 7f8df2f8a740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56181d2eaa90 0x56181d2fb830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:16:55.892 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:55.850+0000 7f8df2f8a740 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x56181d2efc00 0x56181d2eba70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:16:55.892 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:55.850+0000 7f8df2f8a740 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8dc800e1e0 con 0x56181d2efc00 2024-03-24T00:16:55.893 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:55.850+0000 7f8df2f8a740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8dc800fb60 con 0x56181d2eaa90 2024-03-24T00:16:55.893 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:55.850+0000 7f8df2f8a740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8dc800e840 con 0x56181d2ea0a0 2024-03-24T00:16:55.893 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:55.850+0000 7f8dece00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56181d2eaa90 0x56181d2fb830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:16:55.893 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:55.850+0000 7f8dece00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56181d2eaa90 0x56181d2fb830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.118:40648/0 (socket says 172.21.15.118:40648) 2024-03-24T00:16:55.893 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:55.850+0000 7f8dece00640 1 -- 172.21.15.118:0/1036612141 learned_addr learned my addr 172.21.15.118:0/1036612141 (peer_addr_for_me v2:172.21.15.118:0/0) 2024-03-24T00:16:55.893 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:55.850+0000 7f8dee200640 1 --2- 172.21.15.118:0/1036612141 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x56181d2efc00 0x56181d2eba70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:16:55.893 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:55.850+0000 7f8dece00640 1 -- 172.21.15.118:0/1036612141 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56181d2ea0a0 msgr2=0x56181d2eb530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:16:55.893 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:55.850+0000 7f8dece00640 1 --2- 172.21.15.118:0/1036612141 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56181d2ea0a0 0x56181d2eb530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:16:55.893 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:55.850+0000 7f8dece00640 1 -- 172.21.15.118:0/1036612141 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x56181d2efc00 msgr2=0x56181d2eba70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:16:55.893 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:55.851+0000 7f8dece00640 1 --2- 172.21.15.118:0/1036612141 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x56181d2efc00 0x56181d2eba70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:16:55.893 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:55.851+0000 7f8dece00640 1 -- 172.21.15.118:0/1036612141 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x56181d2ef950 con 0x56181d2eaa90 2024-03-24T00:16:55.893 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:55.851+0000 7f8dee200640 1 --2- 172.21.15.118:0/1036612141 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x56181d2efc00 0x56181d2eba70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-24T00:16:55.893 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:55.851+0000 7f8dece00640 1 --2- 172.21.15.118:0/1036612141 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56181d2eaa90 0x56181d2fb830 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7f8dd40bcac0 tx=0x7f8dd4005fd0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=86897e553f66591d server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:16:55.894 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:55.851+0000 7f8dda000640 1 -- 172.21.15.118:0/1036612141 <== mon.2 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8dd404b040 con 0x56181d2eaa90 2024-03-24T00:16:55.894 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:55.851+0000 7f8dda000640 1 -- 172.21.15.118:0/1036612141 <== mon.2 v2:172.21.15.189:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f8dd40c9070 con 0x56181d2eaa90 2024-03-24T00:16:55.894 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:55.852+0000 7f8dda000640 1 -- 172.21.15.118:0/1036612141 <== mon.2 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8dd40c5020 con 0x56181d2eaa90 2024-03-24T00:16:55.894 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:55.852+0000 7f8df2f8a740 1 -- 172.21.15.118:0/1036612141 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56181d2eaa90 msgr2=0x56181d2fb830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:16:55.894 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:55.852+0000 7f8df2f8a740 1 --2- 172.21.15.118:0/1036612141 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56181d2eaa90 0x56181d2fb830 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7f8dd40bcac0 tx=0x7f8dd4005fd0 comp rx=0 tx=0).stop 2024-03-24T00:16:55.894 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:55.853+0000 7f8df2f8a740 1 -- 172.21.15.118:0/1036612141 shutdown_connections 2024-03-24T00:16:55.894 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:55.853+0000 7f8df2f8a740 1 --2- 172.21.15.118:0/1036612141 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x56181d2efc00 0x56181d2eba70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:16:55.894 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:55.853+0000 7f8df2f8a740 1 --2- 172.21.15.118:0/1036612141 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56181d2eaa90 0x56181d2fb830 unknown :-1 s=CLOSED pgs=266 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:16:55.894 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:55.853+0000 7f8df2f8a740 1 --2- 172.21.15.118:0/1036612141 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56181d2ea0a0 0x56181d2eb530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:16:55.894 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:55.853+0000 7f8df2f8a740 1 -- 172.21.15.118:0/1036612141 >> 172.21.15.118:0/1036612141 conn(0x56181d2e5610 msgr2=0x56181d2f39f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:16:55.894 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:55.853+0000 7f8df2f8a740 1 -- 172.21.15.118:0/1036612141 shutdown_connections 2024-03-24T00:16:55.894 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:55.853+0000 7f8df2f8a740 1 -- 172.21.15.118:0/1036612141 wait complete. 2024-03-24T00:16:55.894 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:55.853+0000 7f8df2f8a740 1 -- start start 2024-03-24T00:16:55.895 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:55.854+0000 7f8df2f8a740 1 Processor -- start 2024-03-24T00:16:55.895 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:55.854+0000 7f8df2f8a740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56181d2ea0a0 0x56181d2f95e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:16:55.895 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:55.854+0000 7f8df2f8a740 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x56181d2eaa90 0x56181d2f9b20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:16:55.895 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:55.854+0000 7f8df2f8a740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56181d2efc00 0x56181d2faae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:16:55.895 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:55.854+0000 7f8df2f8a740 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8dc800f500 con 0x56181d2eaa90 2024-03-24T00:16:55.895 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:55.854+0000 7f8df2f8a740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8dc800eea0 con 0x56181d2efc00 2024-03-24T00:16:55.895 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:55.854+0000 7f8df2f8a740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8dc800c3a0 con 0x56181d2ea0a0 2024-03-24T00:16:55.895 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:55.854+0000 7f8df2f8a740 10 client.0 _ll_register_callbacks cb 0x7f8dd4009880 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T00:16:55.895 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:55.855+0000 7f8dee200640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56181d2efc00 0x56181d2faae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:16:55.895 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:55.855+0000 7f8dee200640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56181d2efc00 0x56181d2faae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.118:40652/0 (socket says 172.21.15.118:40652) 2024-03-24T00:16:55.895 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:55.855+0000 7f8dee200640 1 -- 172.21.15.118:0/2701440555 learned_addr learned my addr 172.21.15.118:0/2701440555 (peer_addr_for_me v2:172.21.15.118:0/0) 2024-03-24T00:16:55.895 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:55.855+0000 7f8ded800640 1 --2- 172.21.15.118:0/2701440555 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56181d2ea0a0 0x56181d2f95e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:16:55.895 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:55.855+0000 7f8dece00640 1 --2- 172.21.15.118:0/2701440555 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x56181d2eaa90 0x56181d2f9b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:16:55.895 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:55.855+0000 7f8dee200640 1 -- 172.21.15.118:0/2701440555 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56181d2ea0a0 msgr2=0x56181d2f95e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:16:55.896 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:55.855+0000 7f8dee200640 1 --2- 172.21.15.118:0/2701440555 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56181d2ea0a0 0x56181d2f95e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:16:55.896 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:55.855+0000 7f8dee200640 1 -- 172.21.15.118:0/2701440555 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x56181d2eaa90 msgr2=0x56181d2f9b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:16:55.896 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:55.855+0000 7f8dee200640 1 --2- 172.21.15.118:0/2701440555 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x56181d2eaa90 0x56181d2f9b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:16:55.896 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:55.855+0000 7f8dee200640 1 -- 172.21.15.118:0/2701440555 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x56181d2fb0d0 con 0x56181d2efc00 2024-03-24T00:16:55.896 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:55.855+0000 7f8ded800640 1 --2- 172.21.15.118:0/2701440555 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56181d2ea0a0 0x56181d2f95e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T00:16:55.896 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:55.855+0000 7f8dece00640 1 --2- 172.21.15.118:0/2701440555 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x56181d2eaa90 0x56181d2f9b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-24T00:16:55.896 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:55.855+0000 7f8dee200640 1 --2- 172.21.15.118:0/2701440555 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56181d2efc00 0x56181d2faae0 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7f8de0009f70 tx=0x7f8de00118d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=1fd0c4b086ffd074 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:16:55.896 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:55.856+0000 7f8dcf400640 10 client.0 ms_handle_connect on v2:172.21.15.189:3300/0 2024-03-24T00:16:55.896 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:55.856+0000 7f8dcf400640 1 -- 172.21.15.118:0/2701440555 <== mon.2 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8de0033070 con 0x56181d2efc00 2024-03-24T00:16:55.896 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:55.856+0000 7f8df2f8a740 1 -- 172.21.15.118:0/2701440555 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f8dd40c9070 con 0x56181d2efc00 2024-03-24T00:16:55.896 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:55.856+0000 7f8dcf400640 1 -- 172.21.15.118:0/2701440555 <== mon.2 v2:172.21.15.189:3300/0 2 ==== osd_map(90..90 src has 1..90) v4 ==== 8277+0+0 (secure 0 0 0) 0x7f8de0005590 con 0x56181d2efc00 2024-03-24T00:16:55.896 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:55.856+0000 7f8dcf400640 1 -- 172.21.15.118:0/2701440555 <== mon.2 v2:172.21.15.189:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) v2 ==== 24+0+0 (secure 0 0 0) 0x7f8de003f050 con 0x56181d2efc00 2024-03-24T00:16:55.896 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:55.856+0000 7f8df2f8a740 10 client.25763 fetch_fsmap learned FSMap version 28 2024-03-24T00:16:55.896 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:55.856+0000 7f8df2f8a740 1 -- 172.21.15.118:0/2701440555 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) v3 -- 0x56181d3647e0 con 0x56181d2efc00 2024-03-24T00:16:55.897 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:55.856+0000 7f8dcf400640 1 -- 172.21.15.118:0/2701440555 <== mon.2 v2:172.21.15.189:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f8dd40c9070 con 0x56181d2efc00 2024-03-24T00:16:55.897 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:55.857+0000 7f8dcf400640 1 -- 172.21.15.118:0/2701440555 <== mon.2 v2:172.21.15.189:3300/0 5 ==== fsmap(e 28) v1 ==== 7934+0+0 (secure 0 0 0) 0x7f8de002e070 con 0x56181d2efc00 2024-03-24T00:16:55.897 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:55.857+0000 7f8df2f8a740 10 client.25763 fetch_fsmap finished waiting for FSMap version 28 2024-03-24T00:16:55.897 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:55.857+0000 7f8df2f8a740 10 client.25763 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-24T00:16:55.897 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:55.857+0000 7f8df2f8a740 20 client.25763 populate_metadata read hostname 'smithi118' 2024-03-24T00:16:55.897 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:55.857+0000 7f8df2f8a740 1 --2- 172.21.15.118:0/2701440555 >> [v2:172.21.15.189:6838/4074484168,v1:172.21.15.189:6839/4074484168] conn(0x56181d364f90 0x56181d3673b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:16:55.897 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:55.857+0000 7f8df2f8a740 4 client.25763 mds_command: new command op to 24569 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-03-24T00:16:55.897 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:55.857+0000 7f8df2f8a740 1 -- 172.21.15.118:0/2701440555 --> [v2:172.21.15.189:6838/4074484168,v1:172.21.15.189:6839/4074484168] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) v1 -- 0x56181d257740 con 0x56181d364f90 2024-03-24T00:16:55.897 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:55.857+0000 7f8ded800640 1 --2- 172.21.15.118:0/2701440555 >> [v2:172.21.15.189:6838/4074484168,v1:172.21.15.189:6839/4074484168] conn(0x56181d364f90 0x56181d3673b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:16:55.897 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:55.859+0000 7f8ded800640 1 --2- 172.21.15.118:0/2701440555 >> [v2:172.21.15.189:6838/4074484168,v1:172.21.15.189:6839/4074484168] conn(0x56181d364f90 0x56181d3673b0 crc :-1 s=READY pgs=60 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=f35b42ec0ba3e7fa server_cookie=9cef4aded07e2dfb in_seq=0 out_seq=0 2024-03-24T00:16:55.897 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:55.859+0000 7f8dcf400640 10 client.25763 ms_handle_connect on v2:172.21.15.189:6838/4074484168 2024-03-24T00:16:55.897 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:55.860+0000 7f8dcf400640 1 -- 172.21.15.118:0/2701440555 <== mds.0 v2:172.21.15.189:6838/4074484168 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+110 (crc 0 0 0) 0x56181d257740 con 0x56181d364f90 2024-03-24T00:16:55.897 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:55.860+0000 7f8dcf400640 10 client.25763 handle_command_reply: tid=0 2024-03-24T00:16:55.897 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:55.860+0000 7f8df2f8a740 1 client.25763 shutdown 2024-03-24T00:16:55.897 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:55.860+0000 7f8df2f8a740 1 -- 172.21.15.118:0/2701440555 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56181d2efc00 msgr2=0x56181d2faae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:16:55.898 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:55.860+0000 7f8df2f8a740 1 --2- 172.21.15.118:0/2701440555 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56181d2efc00 0x56181d2faae0 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7f8de0009f70 tx=0x7f8de00118d0 comp rx=0 tx=0).stop 2024-03-24T00:16:55.898 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:55.860+0000 7f8df2f8a740 1 -- 172.21.15.118:0/2701440555 shutdown_connections 2024-03-24T00:16:55.898 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:55.860+0000 7f8df2f8a740 1 --2- 172.21.15.118:0/2701440555 >> [v2:172.21.15.189:6838/4074484168,v1:172.21.15.189:6839/4074484168] conn(0x56181d364f90 0x56181d3673b0 crc :-1 s=READY pgs=60 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:16:55.898 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:55.860+0000 7f8df2f8a740 1 --2- 172.21.15.118:0/2701440555 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56181d2efc00 0x56181d2faae0 unknown :-1 s=CLOSED pgs=267 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:16:55.898 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:55.860+0000 7f8df2f8a740 1 --2- 172.21.15.118:0/2701440555 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x56181d2eaa90 0x56181d2f9b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:16:55.898 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:55.860+0000 7f8dcf400640 0 client.25763 ms_handle_reset on v2:172.21.15.189:6838/4074484168 2024-03-24T00:16:55.898 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:55.860+0000 7f8df2f8a740 1 --2- 172.21.15.118:0/2701440555 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56181d2ea0a0 0x56181d2f95e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:16:55.898 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:55.860+0000 7f8df2f8a740 1 -- 172.21.15.118:0/2701440555 >> 172.21.15.118:0/2701440555 conn(0x56181d2e5610 msgr2=0x56181d2f3160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:16:55.898 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:55.861+0000 7f8df2f8a740 1 -- 172.21.15.118:0/2701440555 shutdown_connections 2024-03-24T00:16:55.898 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:55.861+0000 7f8df2f8a740 1 -- 172.21.15.118:0/2701440555 wait complete. 2024-03-24T00:16:55.898 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:55.861+0000 7f8df2f8a740 20 client.25763 trim_cache size 0 max 16384 2024-03-24T00:16:55.899 INFO:teuthology.orchestra.run.smithi118.stdout:{ 2024-03-24T00:16:55.899 INFO:teuthology.orchestra.run.smithi118.stdout: "return_code": 0, 2024-03-24T00:16:55.899 INFO:teuthology.orchestra.run.smithi118.stdout: "scrub_tag": "76ef0324-cedd-4952-a156-072938b17479", 2024-03-24T00:16:55.899 INFO:teuthology.orchestra.run.smithi118.stdout: "mode": "asynchronous" 2024-03-24T00:16:55.899 INFO:teuthology.orchestra.run.smithi118.stdout:} 2024-03-24T00:16:55.900 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:55.862+0000 7f8df0200640 1 -- 172.21.15.118:0/390351025 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7f8dd00a28f0 msgr2=0x7f8dd00a4db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:16:55.900 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:55.862+0000 7f8df0200640 1 --2- 172.21.15.118:0/390351025 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7f8dd00a28f0 0x7f8dd00a4db0 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 crypto rx=0x7f8de0009de0 tx=0x7f8de0005fd0 comp rx=0 tx=0).stop 2024-03-24T00:16:55.900 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:55.862+0000 7f8df0200640 1 -- 172.21.15.118:0/390351025 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f8de81017a0 msgr2=0x7f8de819a160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:16:55.900 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:55.862+0000 7f8df0200640 1 --2- 172.21.15.118:0/390351025 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f8de81017a0 0x7f8de819a160 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7f8dd4036040 tx=0x7f8dd402c660 comp rx=0 tx=0).stop 2024-03-24T00:16:55.900 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:55.863+0000 7f8df0200640 1 -- 172.21.15.118:0/390351025 shutdown_connections 2024-03-24T00:16:55.900 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:55.863+0000 7f8df0200640 1 --2- 172.21.15.118:0/390351025 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7f8dd00a28f0 0x7f8dd00a4db0 unknown :-1 s=CLOSED pgs=292 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:16:55.901 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:55.863+0000 7f8df0200640 1 --2- 172.21.15.118:0/390351025 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8de8102160 0x7f8de81a1190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:16:55.901 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:55.863+0000 7f8df0200640 1 --2- 172.21.15.118:0/390351025 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f8de81017a0 0x7f8de819a160 unknown :-1 s=CLOSED pgs=191 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:16:55.901 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:55.863+0000 7f8df0200640 1 --2- 172.21.15.118:0/390351025 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f8de81005b0 0x7f8de8199c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:16:55.901 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:55.863+0000 7f8df0200640 1 -- 172.21.15.118:0/390351025 >> 172.21.15.118:0/390351025 conn(0x7f8de80fbb40 msgr2=0x7f8de8109230 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:16:55.901 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:55.863+0000 7f8df0200640 1 -- 172.21.15.118:0/390351025 shutdown_connections 2024-03-24T00:16:55.901 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:55.863+0000 7f8df0200640 1 -- 172.21.15.118:0/390351025 wait complete. 2024-03-24T00:16:55.901 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-03-24T00:16:55.901 DEBUG:teuthology.orchestra.run.smithi118:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-24T00:16:56.107 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:56.106+0000 7fcc4e200640 1 -- 172.21.15.118:0/2994405323 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fcc48104ad0 msgr2=0x7fcc48104f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:16:56.107 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:56.106+0000 7fcc4e200640 1 --2- 172.21.15.118:0/2994405323 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fcc48104ad0 0x7fcc48104f50 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7fcc34009900 tx=0x7fcc3402bc40 comp rx=0 tx=0).stop 2024-03-24T00:16:56.107 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:56.106+0000 7fcc4e200640 1 -- 172.21.15.118:0/2994405323 shutdown_connections 2024-03-24T00:16:56.107 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:56.106+0000 7fcc4e200640 1 --2- 172.21.15.118:0/2994405323 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fcc48105490 0x7fcc4810bd20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:16:56.107 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:56.106+0000 7fcc4e200640 1 --2- 172.21.15.118:0/2994405323 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fcc48104ad0 0x7fcc48104f50 unknown :-1 s=CLOSED pgs=268 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:16:56.107 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:56.106+0000 7fcc4e200640 1 --2- 172.21.15.118:0/2994405323 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fcc481038e0 0x7fcc48103d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:16:56.107 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:56.106+0000 7fcc4e200640 1 -- 172.21.15.118:0/2994405323 >> 172.21.15.118:0/2994405323 conn(0x7fcc480fee50 msgr2=0x7fcc48101290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:16:56.107 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:56.106+0000 7fcc4e200640 1 -- 172.21.15.118:0/2994405323 shutdown_connections 2024-03-24T00:16:56.108 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:56.106+0000 7fcc4e200640 1 -- 172.21.15.118:0/2994405323 wait complete. 2024-03-24T00:16:56.108 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:56.107+0000 7fcc4e200640 1 Processor -- start 2024-03-24T00:16:56.108 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:56.108+0000 7fcc4e200640 1 -- start start 2024-03-24T00:16:56.109 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:56.108+0000 7fcc4e200640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fcc481038e0 0x7fcc48199bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:16:56.109 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:56.108+0000 7fcc4e200640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fcc48104ad0 0x7fcc4819a110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:16:56.109 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:56.108+0000 7fcc4e200640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fcc48105490 0x7fcc481a1180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:16:56.109 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:56.108+0000 7fcc4e200640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcc4810f590 con 0x7fcc48104ad0 2024-03-24T00:16:56.109 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:56.108+0000 7fcc4e200640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcc4810f410 con 0x7fcc48105490 2024-03-24T00:16:56.109 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:56.108+0000 7fcc4e200640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcc4810f710 con 0x7fcc481038e0 2024-03-24T00:16:56.110 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:56.109+0000 7fcc46a00640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fcc48104ad0 0x7fcc4819a110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:16:56.110 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:56.109+0000 7fcc47400640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fcc481038e0 0x7fcc48199bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:16:56.110 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:56.109+0000 7fcc46a00640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fcc48104ad0 0x7fcc4819a110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.118:3300/0 says I am v2:172.21.15.118:50714/0 (socket says 172.21.15.118:50714) 2024-03-24T00:16:56.110 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:56.109+0000 7fcc46a00640 1 -- 172.21.15.118:0/1146450709 learned_addr learned my addr 172.21.15.118:0/1146450709 (peer_addr_for_me v2:172.21.15.118:0/0) 2024-03-24T00:16:56.110 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:56.109+0000 7fcc47e00640 1 --2- 172.21.15.118:0/1146450709 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fcc48105490 0x7fcc481a1180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:16:56.110 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:56.109+0000 7fcc46a00640 1 -- 172.21.15.118:0/1146450709 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fcc481038e0 msgr2=0x7fcc48199bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:16:56.110 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:56.109+0000 7fcc46a00640 1 --2- 172.21.15.118:0/1146450709 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fcc481038e0 0x7fcc48199bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:16:56.110 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:56.109+0000 7fcc46a00640 1 -- 172.21.15.118:0/1146450709 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fcc48105490 msgr2=0x7fcc481a1180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:16:56.110 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:56.109+0000 7fcc46a00640 1 --2- 172.21.15.118:0/1146450709 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fcc48105490 0x7fcc481a1180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:16:56.110 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:56.109+0000 7fcc46a00640 1 -- 172.21.15.118:0/1146450709 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcc481a17f0 con 0x7fcc48104ad0 2024-03-24T00:16:56.114 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:56.110+0000 7fcc46a00640 1 --2- 172.21.15.118:0/1146450709 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fcc48104ad0 0x7fcc4819a110 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7fcc3402c250 tx=0x7fcc34004e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:16:56.114 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:56.110+0000 7fcc3fe00640 1 -- 172.21.15.118:0/1146450709 <== mon.0 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcc34041070 con 0x7fcc48104ad0 2024-03-24T00:16:56.114 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:56.110+0000 7fcc3fe00640 1 -- 172.21.15.118:0/1146450709 <== mon.0 v2:172.21.15.118:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fcc340054e0 con 0x7fcc48104ad0 2024-03-24T00:16:56.114 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:56.110+0000 7fcc3fe00640 1 -- 172.21.15.118:0/1146450709 <== mon.0 v2:172.21.15.118:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcc34034440 con 0x7fcc48104ad0 2024-03-24T00:16:56.114 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:56.110+0000 7fcc4e200640 1 -- 172.21.15.118:0/1146450709 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcc48075ae0 con 0x7fcc48104ad0 2024-03-24T00:16:56.114 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:56.111+0000 7fcc4e200640 1 -- 172.21.15.118:0/1146450709 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcc48075f80 con 0x7fcc48104ad0 2024-03-24T00:16:56.116 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:56.115+0000 7fcc3fe00640 1 -- 172.21.15.118:0/1146450709 <== mon.0 v2:172.21.15.118:3300/0 4 ==== mgrmap(e 19) v1 ==== 137342+0+0 (secure 0 0 0) 0x7fcc34044070 con 0x7fcc48104ad0 2024-03-24T00:16:56.116 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:56.116+0000 7fcc50fa5740 1 Processor -- start 2024-03-24T00:16:56.117 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:56.116+0000 7fcc50fa5740 1 -- start start 2024-03-24T00:16:56.117 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:56.116+0000 7fcc3fe00640 1 --2- 172.21.15.118:0/1146450709 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7fcc2c0a2950 0x7fcc2c0a4e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:16:56.118 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:56.116+0000 7fcc3fe00640 1 -- 172.21.15.118:0/1146450709 <== mon.0 v2:172.21.15.118:3300/0 5 ==== osd_map(90..90 src has 1..90) v4 ==== 8277+0+0 (secure 0 0 0) 0x7fcc340efef0 con 0x7fcc48104ad0 2024-03-24T00:16:56.118 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:56.117+0000 7fcc50fa5740 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55f665fb6e80 0x55f665fb72a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:16:56.118 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:56.117+0000 7fcc50fa5740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55f665fb7870 0x55f665fbc370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:16:56.118 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:56.117+0000 7fcc50fa5740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55f665fbc9e0 0x55f665fbee70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:16:56.118 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:56.117+0000 7fcc50fa5740 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f665f1c250 con 0x55f665fb6e80 2024-03-24T00:16:56.118 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:56.117+0000 7fcc50fa5740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f665ec4a30 con 0x55f665fbc9e0 2024-03-24T00:16:56.119 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:56.117+0000 7fcc50fa5740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f665f182f0 con 0x55f665fb7870 2024-03-24T00:16:56.119 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:56.118+0000 7fcc46a00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55f665fb7870 0x55f665fbc370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:16:56.119 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:56.118+0000 7fcc46a00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55f665fb7870 0x55f665fbc370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.118:36476/0 (socket says 172.21.15.118:36476) 2024-03-24T00:16:56.119 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:56.118+0000 7fcc46a00640 1 -- 172.21.15.118:0/1020721931 learned_addr learned my addr 172.21.15.118:0/1020721931 (peer_addr_for_me v2:172.21.15.118:0/0) 2024-03-24T00:16:56.119 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:56.118+0000 7fcc47e00640 1 --2- 172.21.15.118:0/1146450709 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7fcc2c0a2950 0x7fcc2c0a4e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:16:56.119 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:56.118+0000 7fcc46a00640 1 -- 172.21.15.118:0/1020721931 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55f665fbc9e0 msgr2=0x55f665fbee70 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-24T00:16:56.119 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:56.118+0000 7fcc46a00640 1 --2- 172.21.15.118:0/1020721931 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55f665fbc9e0 0x55f665fbee70 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:16:56.119 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:56.118+0000 7fcc46a00640 1 -- 172.21.15.118:0/1020721931 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55f665fb6e80 msgr2=0x55f665fb72a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:16:56.120 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:56.118+0000 7fcc46a00640 1 --2- 172.21.15.118:0/1020721931 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55f665fb6e80 0x55f665fb72a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:16:56.120 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:56.118+0000 7fcc46a00640 1 -- 172.21.15.118:0/1020721931 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55f665eaf690 con 0x55f665fb7870 2024-03-24T00:16:56.120 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:56.118+0000 7fcc46a00640 1 --2- 172.21.15.118:0/1020721931 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55f665fb7870 0x55f665fbc370 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7fcc34009850 tx=0x7fcc3402c160 comp rx=0 tx=0).ready entity=mon.1 client_cookie=3969b819b1c543b5 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:16:56.120 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:56.118+0000 7fcc33e00640 1 -- 172.21.15.118:0/1020721931 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcc3404b040 con 0x55f665fb7870 2024-03-24T00:16:56.120 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:56.118+0000 7fcc33e00640 1 -- 172.21.15.118:0/1020721931 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fcc340a5e50 con 0x55f665fb7870 2024-03-24T00:16:56.120 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:56.118+0000 7fcc33e00640 1 -- 172.21.15.118:0/1020721931 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcc340b8f30 con 0x55f665fb7870 2024-03-24T00:16:56.121 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:56.120+0000 7fcc50fa5740 1 -- 172.21.15.118:0/1020721931 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55f665fb7870 msgr2=0x55f665fbc370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:16:56.121 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:56.120+0000 7fcc50fa5740 1 --2- 172.21.15.118:0/1020721931 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55f665fb7870 0x55f665fbc370 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7fcc34009850 tx=0x7fcc3402c160 comp rx=0 tx=0).stop 2024-03-24T00:16:56.121 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:56.120+0000 7fcc50fa5740 1 -- 172.21.15.118:0/1020721931 shutdown_connections 2024-03-24T00:16:56.121 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:56.120+0000 7fcc50fa5740 1 --2- 172.21.15.118:0/1020721931 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55f665fbc9e0 0x55f665fbee70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:16:56.121 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:56.120+0000 7fcc50fa5740 1 --2- 172.21.15.118:0/1020721931 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55f665fb7870 0x55f665fbc370 unknown :-1 s=CLOSED pgs=258 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:16:56.121 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:56.120+0000 7fcc50fa5740 1 --2- 172.21.15.118:0/1020721931 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55f665fb6e80 0x55f665fb72a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:16:56.121 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:56.120+0000 7fcc50fa5740 1 -- 172.21.15.118:0/1020721931 >> 172.21.15.118:0/1020721931 conn(0x55f665f7f200 msgr2=0x55f665fb27f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:16:56.121 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:56.120+0000 7fcc47e00640 1 --2- 172.21.15.118:0/1146450709 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7fcc2c0a2950 0x7fcc2c0a4e10 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x7fcc40005fd0 tx=0x7fcc40009500 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:16:56.122 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:56.121+0000 7fcc50fa5740 1 -- 172.21.15.118:0/1020721931 shutdown_connections 2024-03-24T00:16:56.122 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:56.121+0000 7fcc50fa5740 1 -- 172.21.15.118:0/1020721931 wait complete. 2024-03-24T00:16:56.122 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:56.121+0000 7fcc50fa5740 1 -- start start 2024-03-24T00:16:56.122 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:56.122+0000 7fcc50fa5740 1 Processor -- start 2024-03-24T00:16:56.123 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:56.122+0000 7fcc50fa5740 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55f665fb6e80 0x55f665fd1a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:16:56.123 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:56.122+0000 7fcc50fa5740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55f665fb7870 0x55f665fd1f80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:16:56.123 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:56.122+0000 7fcc50fa5740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55f665fbc9e0 0x55f665fd8fc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:16:56.125 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:56.123+0000 7fcc46a00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55f665fb7870 0x55f665fd1f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:16:56.125 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:56.123+0000 7fcc46a00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55f665fb7870 0x55f665fd1f80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.118:40672/0 (socket says 172.21.15.118:40672) 2024-03-24T00:16:56.125 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:56.123+0000 7fcc47400640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55f665fb6e80 0x55f665fd1a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:16:56.125 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:56.123+0000 7fcc47400640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55f665fb6e80 0x55f665fd1a40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.118:3300/0 says I am v2:172.21.15.118:50726/0 (socket says 172.21.15.118:50726) 2024-03-24T00:16:56.125 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:56.123+0000 7fcc47e00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55f665fbc9e0 0x55f665fd8fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:16:56.125 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:56.123+0000 7fcc47e00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55f665fbc9e0 0x55f665fd8fc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.118:36490/0 (socket says 172.21.15.118:36490) 2024-03-24T00:16:56.125 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:56.123+0000 7fcc46a00640 1 -- 172.21.15.118:0/500649158 learned_addr learned my addr 172.21.15.118:0/500649158 (peer_addr_for_me v2:172.21.15.118:0/0) 2024-03-24T00:16:56.125 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:56.123+0000 7fcc50fa5740 1 -- 172.21.15.118:0/500649158 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f665f182f0 con 0x55f665fb6e80 2024-03-24T00:16:56.125 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:56.123+0000 7fcc50fa5740 1 -- 172.21.15.118:0/500649158 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f665ed7e80 con 0x55f665fb7870 2024-03-24T00:16:56.126 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:56.123+0000 7fcc50fa5740 1 -- 172.21.15.118:0/500649158 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f665fd0760 con 0x55f665fbc9e0 2024-03-24T00:16:56.126 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:56.123+0000 7fcc50fa5740 10 client.0 _ll_register_callbacks cb 0x55f665eea130 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T00:16:56.126 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:56.123+0000 7fcc46a00640 1 -- 172.21.15.118:0/500649158 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55f665fbc9e0 msgr2=0x55f665fd8fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:16:56.126 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:56.123+0000 7fcc46a00640 1 --2- 172.21.15.118:0/500649158 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55f665fbc9e0 0x55f665fd8fc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:16:56.126 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:56.123+0000 7fcc46a00640 1 -- 172.21.15.118:0/500649158 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55f665fb6e80 msgr2=0x55f665fd1a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:16:56.126 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:56.124+0000 7fcc47e00640 1 --2- 172.21.15.118:0/500649158 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55f665fbc9e0 0x55f665fd8fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello state changed while learned_addr, mark_down or replacing must be happened just now 2024-03-24T00:16:56.126 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:56.124+0000 7fcc46a00640 1 --2- 172.21.15.118:0/500649158 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55f665fb6e80 0x55f665fd1a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:16:56.126 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:56.124+0000 7fcc46a00640 1 -- 172.21.15.118:0/500649158 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55f665cfa7d0 con 0x55f665fb7870 2024-03-24T00:16:56.126 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:56.124+0000 7fcc47400640 1 --2- 172.21.15.118:0/500649158 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55f665fb6e80 0x55f665fd1a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T00:16:56.126 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:56.124+0000 7fcc46a00640 1 --2- 172.21.15.118:0/500649158 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55f665fb7870 0x55f665fd1f80 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7fcc340bc5c0 tx=0x7fcc3402bc40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=32930672dacd6d6b server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:16:56.126 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:56.124+0000 7fcc30c00640 10 client.0 ms_handle_connect on v2:172.21.15.189:3300/0 2024-03-24T00:16:56.126 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:56.124+0000 7fcc30c00640 1 -- 172.21.15.118:0/500649158 <== mon.2 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcc3404b070 con 0x55f665fb7870 2024-03-24T00:16:56.126 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:56.124+0000 7fcc30c00640 1 -- 172.21.15.118:0/500649158 <== mon.2 v2:172.21.15.189:3300/0 2 ==== osd_map(90..90 src has 1..90) v4 ==== 8277+0+0 (secure 0 0 0) 0x7fcc340a70c0 con 0x55f665fb7870 2024-03-24T00:16:56.127 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:56.124+0000 7fcc50fa5740 1 -- 172.21.15.118:0/500649158 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fcc340a5e50 con 0x55f665fb7870 2024-03-24T00:16:56.127 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:56.124+0000 7fcc30c00640 1 -- 172.21.15.118:0/500649158 <== mon.2 v2:172.21.15.189:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) v2 ==== 24+0+0 (secure 0 0 0) 0x7fcc34005690 con 0x55f665fb7870 2024-03-24T00:16:56.127 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:56.124+0000 7fcc50fa5740 10 client.25772 fetch_fsmap learned FSMap version 28 2024-03-24T00:16:56.127 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:56.124+0000 7fcc50fa5740 1 -- 172.21.15.118:0/500649158 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) v3 -- 0x55f665c303f0 con 0x55f665fb7870 2024-03-24T00:16:56.127 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:56.125+0000 7fcc30c00640 1 -- 172.21.15.118:0/500649158 <== mon.2 v2:172.21.15.189:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fcc340a5e50 con 0x55f665fb7870 2024-03-24T00:16:56.128 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:56.125+0000 7fcc30c00640 1 -- 172.21.15.118:0/500649158 <== mon.2 v2:172.21.15.189:3300/0 5 ==== fsmap(e 28) v1 ==== 7934+0+0 (secure 0 0 0) 0x7fcc340bdb40 con 0x55f665fb7870 2024-03-24T00:16:56.129 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:56.125+0000 7fcc50fa5740 10 client.25772 fetch_fsmap finished waiting for FSMap version 28 2024-03-24T00:16:56.129 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:56.125+0000 7fcc50fa5740 10 client.25772 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-24T00:16:56.129 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:56.125+0000 7fcc50fa5740 20 client.25772 populate_metadata read hostname 'smithi118' 2024-03-24T00:16:56.129 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:56.125+0000 7fcc50fa5740 1 --2- 172.21.15.118:0/500649158 >> [v2:172.21.15.189:6838/4074484168,v1:172.21.15.189:6839/4074484168] conn(0x55f665fdd610 0x55f665fdfa30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:16:56.129 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:56.125+0000 7fcc50fa5740 4 client.25772 mds_command: new command op to 24569 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-24T00:16:56.129 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:56.125+0000 7fcc50fa5740 1 -- 172.21.15.118:0/500649158 --> [v2:172.21.15.189:6838/4074484168,v1:172.21.15.189:6839/4074484168] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55f665f29790 con 0x55f665fdd610 2024-03-24T00:16:56.129 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:56.125+0000 7fcc47400640 1 --2- 172.21.15.118:0/500649158 >> [v2:172.21.15.189:6838/4074484168,v1:172.21.15.189:6839/4074484168] conn(0x55f665fdd610 0x55f665fdfa30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:16:56.129 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:56.126+0000 7fcc47400640 1 --2- 172.21.15.118:0/500649158 >> [v2:172.21.15.189:6838/4074484168,v1:172.21.15.189:6839/4074484168] conn(0x55f665fdd610 0x55f665fdfa30 crc :-1 s=READY pgs=61 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=e0cf9684360ac6e4 server_cookie=3ba4d23015345400 in_seq=0 out_seq=0 2024-03-24T00:16:56.129 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:56.126+0000 7fcc30c00640 10 client.25772 ms_handle_connect on v2:172.21.15.189:6838/4074484168 2024-03-24T00:16:56.129 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:56.126+0000 7fcc30c00640 1 -- 172.21.15.118:0/500649158 <== mds.0 v2:172.21.15.189:6838/4074484168 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x55f665f29790 con 0x55f665fdd610 2024-03-24T00:16:56.129 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:56.126+0000 7fcc30c00640 10 client.25772 handle_command_reply: tid=0 2024-03-24T00:16:56.129 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:56.127+0000 7fcc50fa5740 1 client.25772 shutdown 2024-03-24T00:16:56.129 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:56.127+0000 7fcc50fa5740 1 -- 172.21.15.118:0/500649158 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55f665fb7870 msgr2=0x55f665fd1f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:16:56.129 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:56.127+0000 7fcc50fa5740 1 --2- 172.21.15.118:0/500649158 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55f665fb7870 0x55f665fd1f80 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7fcc340bc5c0 tx=0x7fcc3402bc40 comp rx=0 tx=0).stop 2024-03-24T00:16:56.130 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:56.127+0000 7fcc50fa5740 1 -- 172.21.15.118:0/500649158 shutdown_connections 2024-03-24T00:16:56.130 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:56.127+0000 7fcc50fa5740 1 --2- 172.21.15.118:0/500649158 >> [v2:172.21.15.189:6838/4074484168,v1:172.21.15.189:6839/4074484168] conn(0x55f665fdd610 0x55f665fdfa30 crc :-1 s=READY pgs=61 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:16:56.130 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:56.127+0000 7fcc50fa5740 1 --2- 172.21.15.118:0/500649158 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55f665fbc9e0 0x55f665fd8fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:16:56.130 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:56.127+0000 7fcc50fa5740 1 --2- 172.21.15.118:0/500649158 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55f665fb7870 0x55f665fd1f80 unknown :-1 s=CLOSED pgs=269 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:16:56.130 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:56.127+0000 7fcc50fa5740 1 --2- 172.21.15.118:0/500649158 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55f665fb6e80 0x55f665fd1a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:16:56.130 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:56.127+0000 7fcc50fa5740 1 -- 172.21.15.118:0/500649158 >> 172.21.15.118:0/500649158 conn(0x55f665f7f200 msgr2=0x55f665fb1a90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:16:56.130 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:56.127+0000 7fcc30c00640 0 client.25772 ms_handle_reset on v2:172.21.15.189:6838/4074484168 2024-03-24T00:16:56.130 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:56.127+0000 7fcc50fa5740 1 -- 172.21.15.118:0/500649158 shutdown_connections 2024-03-24T00:16:56.130 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:56.127+0000 7fcc50fa5740 1 -- 172.21.15.118:0/500649158 wait complete. 2024-03-24T00:16:56.130 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:56.128+0000 7fcc50fa5740 20 client.25772 trim_cache size 0 max 16384 2024-03-24T00:16:56.161 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:56.160+0000 7fcc50fa5740 1 Processor -- start 2024-03-24T00:16:56.162 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:56.161+0000 7fcc50fa5740 1 -- start start 2024-03-24T00:16:56.162 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:56.161+0000 7fcc50fa5740 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55f665fb6e80 0x55f665fd1c70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:16:56.162 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:56.161+0000 7fcc50fa5740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55f665fb7870 0x55f665fd8f80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:16:56.162 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:56.161+0000 7fcc50fa5740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55f665fbc9e0 0x55f665fb1910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:16:56.162 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:56.161+0000 7fcc50fa5740 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcc481a67e0 con 0x55f665fb6e80 2024-03-24T00:16:56.162 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:56.161+0000 7fcc50fa5740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcc481a8160 con 0x55f665fbc9e0 2024-03-24T00:16:56.162 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:56.161+0000 7fcc50fa5740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcc481a6e40 con 0x55f665fb7870 2024-03-24T00:16:56.163 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:56.162+0000 7fcc46a00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55f665fb7870 0x55f665fd8f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:16:56.163 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:56.162+0000 7fcc47400640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55f665fb6e80 0x55f665fd1c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:16:56.163 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:56.162+0000 7fcc47400640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55f665fb6e80 0x55f665fd1c70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.118:3300/0 says I am v2:172.21.15.118:50730/0 (socket says 172.21.15.118:50730) 2024-03-24T00:16:56.163 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:56.162+0000 7fcc47400640 1 -- 172.21.15.118:0/151281927 learned_addr learned my addr 172.21.15.118:0/151281927 (peer_addr_for_me v2:172.21.15.118:0/0) 2024-03-24T00:16:56.163 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:56.162+0000 7fcc47e00640 1 --2- 172.21.15.118:0/151281927 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55f665fbc9e0 0x55f665fb1910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:16:56.163 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:56.162+0000 7fcc46a00640 1 -- 172.21.15.118:0/151281927 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55f665fbc9e0 msgr2=0x55f665fb1910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:16:56.163 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:56.162+0000 7fcc46a00640 1 --2- 172.21.15.118:0/151281927 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55f665fbc9e0 0x55f665fb1910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:16:56.163 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:56.162+0000 7fcc46a00640 1 -- 172.21.15.118:0/151281927 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55f665fb6e80 msgr2=0x55f665fd1c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:16:56.163 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:56.162+0000 7fcc46a00640 1 --2- 172.21.15.118:0/151281927 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55f665fb6e80 0x55f665fd1c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:16:56.163 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:56.162+0000 7fcc46a00640 1 -- 172.21.15.118:0/151281927 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55f665bf00f0 con 0x55f665fb7870 2024-03-24T00:16:56.164 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:56.163+0000 7fcc46a00640 1 --2- 172.21.15.118:0/151281927 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55f665fb7870 0x55f665fd8f80 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7fcc340bab70 tx=0x7fcc340db220 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d826b15847f2fb4a server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:16:56.164 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:56.163+0000 7fcc33400640 1 -- 172.21.15.118:0/151281927 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcc3404b070 con 0x55f665fb7870 2024-03-24T00:16:56.164 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:56.163+0000 7fcc33400640 1 -- 172.21.15.118:0/151281927 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fcc340a5e50 con 0x55f665fb7870 2024-03-24T00:16:56.164 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:56.163+0000 7fcc33400640 1 -- 172.21.15.118:0/151281927 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcc340a7040 con 0x55f665fb7870 2024-03-24T00:16:56.165 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:56.164+0000 7fcc50fa5740 1 -- 172.21.15.118:0/151281927 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55f665fb7870 msgr2=0x55f665fd8f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:16:56.165 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:56.164+0000 7fcc50fa5740 1 --2- 172.21.15.118:0/151281927 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55f665fb7870 0x55f665fd8f80 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7fcc340bab70 tx=0x7fcc340db220 comp rx=0 tx=0).stop 2024-03-24T00:16:56.165 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:56.164+0000 7fcc50fa5740 1 -- 172.21.15.118:0/151281927 shutdown_connections 2024-03-24T00:16:56.165 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:56.164+0000 7fcc50fa5740 1 --2- 172.21.15.118:0/151281927 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55f665fbc9e0 0x55f665fb1910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:16:56.165 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:56.164+0000 7fcc50fa5740 1 --2- 172.21.15.118:0/151281927 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55f665fb7870 0x55f665fd8f80 secure :-1 s=CLOSED pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7fcc340bab70 tx=0x7fcc340db220 comp rx=0 tx=0).stop 2024-03-24T00:16:56.166 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:56.164+0000 7fcc50fa5740 1 --2- 172.21.15.118:0/151281927 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55f665fb6e80 0x55f665fd1c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:16:56.166 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:56.164+0000 7fcc50fa5740 1 -- 172.21.15.118:0/151281927 >> 172.21.15.118:0/151281927 conn(0x55f665f7f200 msgr2=0x55f665fb1200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:16:56.166 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:56.165+0000 7fcc50fa5740 1 -- 172.21.15.118:0/151281927 shutdown_connections 2024-03-24T00:16:56.166 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:56.165+0000 7fcc50fa5740 1 -- 172.21.15.118:0/151281927 wait complete. 2024-03-24T00:16:56.166 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:56.166+0000 7fcc50fa5740 1 -- start start 2024-03-24T00:16:56.166 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:56.166+0000 7fcc50fa5740 1 Processor -- start 2024-03-24T00:16:56.167 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:56.166+0000 7fcc50fa5740 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55f665fb6e80 0x55f665fd66d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:16:56.167 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:56.166+0000 7fcc50fa5740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55f665fbc9e0 0x55f665fd6c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:16:56.167 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:56.166+0000 7fcc50fa5740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55f665fd7c20 0x55f665fd80a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:16:56.167 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:56.166+0000 7fcc50fa5740 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcc481a7b00 con 0x55f665fb6e80 2024-03-24T00:16:56.167 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:56.166+0000 7fcc50fa5740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcc481a74a0 con 0x55f665fbc9e0 2024-03-24T00:16:56.167 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:56.166+0000 7fcc50fa5740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcc481a49a0 con 0x55f665fd7c20 2024-03-24T00:16:56.167 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:56.167+0000 7fcc50fa5740 10 client.0 _ll_register_callbacks cb 0x7fcc340ede80 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T00:16:56.169 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:56.167+0000 7fcc47400640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55f665fb6e80 0x55f665fd66d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:16:56.169 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:56.167+0000 7fcc47400640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55f665fb6e80 0x55f665fd66d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.118:3300/0 says I am v2:172.21.15.118:50734/0 (socket says 172.21.15.118:50734) 2024-03-24T00:16:56.169 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:56.167+0000 7fcc47400640 1 -- 172.21.15.118:0/1319942213 learned_addr learned my addr 172.21.15.118:0/1319942213 (peer_addr_for_me v2:172.21.15.118:0/0) 2024-03-24T00:16:56.169 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:56.167+0000 7fcc46a00640 1 --2- 172.21.15.118:0/1319942213 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55f665fbc9e0 0x55f665fd6c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:16:56.169 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:56.167+0000 7fcc47e00640 1 --2- 172.21.15.118:0/1319942213 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55f665fd7c20 0x55f665fd80a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:16:56.169 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:56.167+0000 7fcc46a00640 1 -- 172.21.15.118:0/1319942213 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55f665fd7c20 msgr2=0x55f665fd80a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:16:56.169 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:56.167+0000 7fcc46a00640 1 --2- 172.21.15.118:0/1319942213 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55f665fd7c20 0x55f665fd80a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:16:56.169 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:56.167+0000 7fcc46a00640 1 -- 172.21.15.118:0/1319942213 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55f665fb6e80 msgr2=0x55f665fd66d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:16:56.169 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:56.167+0000 7fcc46a00640 1 --2- 172.21.15.118:0/1319942213 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55f665fb6e80 0x55f665fd66d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:16:56.169 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:56.167+0000 7fcc46a00640 1 -- 172.21.15.118:0/1319942213 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55f665fd85e0 con 0x55f665fbc9e0 2024-03-24T00:16:56.170 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:56.168+0000 7fcc47400640 1 --2- 172.21.15.118:0/1319942213 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55f665fb6e80 0x55f665fd66d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T00:16:56.170 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:56.168+0000 7fcc47e00640 1 --2- 172.21.15.118:0/1319942213 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55f665fd7c20 0x55f665fd80a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T00:16:56.170 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:56.168+0000 7fcc46a00640 1 --2- 172.21.15.118:0/1319942213 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55f665fbc9e0 0x55f665fd6c10 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7fcc340bab70 tx=0x7fcc34009260 comp rx=0 tx=0).ready entity=mon.2 client_cookie=3ed7c0da920e9411 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:16:56.170 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:56.168+0000 7fcc30c00640 10 client.0 ms_handle_connect on v2:172.21.15.189:3300/0 2024-03-24T00:16:56.170 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:56.168+0000 7fcc30c00640 1 -- 172.21.15.118:0/1319942213 <== mon.2 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fcc3404b050 con 0x55f665fbc9e0 2024-03-24T00:16:56.170 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:56.168+0000 7fcc30c00640 1 -- 172.21.15.118:0/1319942213 <== mon.2 v2:172.21.15.189:3300/0 2 ==== osd_map(90..90 src has 1..90) v4 ==== 8277+0+0 (secure 0 0 0) 0x7fcc340c4b20 con 0x55f665fbc9e0 2024-03-24T00:16:56.170 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:56.169+0000 7fcc50fa5740 1 -- 172.21.15.118:0/1319942213 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fcc340a5e50 con 0x55f665fbc9e0 2024-03-24T00:16:56.171 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:56.169+0000 7fcc30c00640 1 -- 172.21.15.118:0/1319942213 <== mon.2 v2:172.21.15.189:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) v2 ==== 24+0+0 (secure 0 0 0) 0x7fcc340c4d20 con 0x55f665fbc9e0 2024-03-24T00:16:56.171 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:56.169+0000 7fcc50fa5740 10 client.25778 fetch_fsmap learned FSMap version 28 2024-03-24T00:16:56.171 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:56.169+0000 7fcc50fa5740 1 -- 172.21.15.118:0/1319942213 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) v3 -- 0x55f665fd85e0 con 0x55f665fbc9e0 2024-03-24T00:16:56.171 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:56.170+0000 7fcc30c00640 1 -- 172.21.15.118:0/1319942213 <== mon.2 v2:172.21.15.189:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fcc340a5e50 con 0x55f665fbc9e0 2024-03-24T00:16:56.171 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:56.170+0000 7fcc30c00640 1 -- 172.21.15.118:0/1319942213 <== mon.2 v2:172.21.15.189:3300/0 5 ==== fsmap(e 28) v1 ==== 7934+0+0 (secure 0 0 0) 0x7fcc340c1880 con 0x55f665fbc9e0 2024-03-24T00:16:56.171 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:56.171+0000 7fcc50fa5740 10 client.25778 fetch_fsmap finished waiting for FSMap version 28 2024-03-24T00:16:56.171 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:56.171+0000 7fcc50fa5740 10 client.25778 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-24T00:16:56.172 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:56.171+0000 7fcc50fa5740 20 client.25778 populate_metadata read hostname 'smithi118' 2024-03-24T00:16:56.172 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:56.171+0000 7fcc50fa5740 1 --2- 172.21.15.118:0/1319942213 >> [v2:172.21.15.189:6838/4074484168,v1:172.21.15.189:6839/4074484168] conn(0x55f6660397c0 0x55f66603bbe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:16:56.172 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:56.171+0000 7fcc50fa5740 4 client.25778 mds_command: new command op to 24569 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-24T00:16:56.172 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:56.171+0000 7fcc50fa5740 1 -- 172.21.15.118:0/1319942213 --> [v2:172.21.15.189:6838/4074484168,v1:172.21.15.189:6839/4074484168] -- command(tid 0: {"prefix": "scrub status"}) v1 -- 0x55f665f23ae0 con 0x55f6660397c0 2024-03-24T00:16:56.172 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:56.171+0000 7fcc47400640 1 --2- 172.21.15.118:0/1319942213 >> [v2:172.21.15.189:6838/4074484168,v1:172.21.15.189:6839/4074484168] conn(0x55f6660397c0 0x55f66603bbe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:16:56.173 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:56.172+0000 7fcc47400640 1 --2- 172.21.15.118:0/1319942213 >> [v2:172.21.15.189:6838/4074484168,v1:172.21.15.189:6839/4074484168] conn(0x55f6660397c0 0x55f66603bbe0 crc :-1 s=READY pgs=62 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=142b2cf3ec7c30df server_cookie=4a6321c7baf22d11 in_seq=0 out_seq=0 2024-03-24T00:16:56.173 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:56.172+0000 7fcc30c00640 10 client.25778 ms_handle_connect on v2:172.21.15.189:6838/4074484168 2024-03-24T00:16:56.173 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:56.172+0000 7fcc30c00640 1 -- 172.21.15.118:0/1319942213 <== mds.0 v2:172.21.15.189:6838/4074484168 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+259 (crc 0 0 0) 0x55f665f23ae0 con 0x55f6660397c0 2024-03-24T00:16:56.173 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:56.172+0000 7fcc30c00640 10 client.25778 handle_command_reply: tid=0 2024-03-24T00:16:56.173 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:56.172+0000 7fcc50fa5740 1 client.25778 shutdown 2024-03-24T00:16:56.174 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:56.172+0000 7fcc50fa5740 1 -- 172.21.15.118:0/1319942213 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55f665fbc9e0 msgr2=0x55f665fd6c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:16:56.174 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:56.172+0000 7fcc50fa5740 1 --2- 172.21.15.118:0/1319942213 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55f665fbc9e0 0x55f665fd6c10 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7fcc340bab70 tx=0x7fcc34009260 comp rx=0 tx=0).stop 2024-03-24T00:16:56.174 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:56.173+0000 7fcc50fa5740 1 -- 172.21.15.118:0/1319942213 shutdown_connections 2024-03-24T00:16:56.174 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:56.173+0000 7fcc50fa5740 1 --2- 172.21.15.118:0/1319942213 >> [v2:172.21.15.189:6838/4074484168,v1:172.21.15.189:6839/4074484168] conn(0x55f6660397c0 0x55f66603bbe0 crc :-1 s=READY pgs=62 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:16:56.174 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:56.173+0000 7fcc30c00640 0 client.25778 ms_handle_reset on v2:172.21.15.189:6838/4074484168 2024-03-24T00:16:56.174 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:56.173+0000 7fcc50fa5740 1 --2- 172.21.15.118:0/1319942213 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55f665fd7c20 0x55f665fd80a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:16:56.174 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:56.173+0000 7fcc50fa5740 1 --2- 172.21.15.118:0/1319942213 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55f665fbc9e0 0x55f665fd6c10 unknown :-1 s=CLOSED pgs=270 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:16:56.174 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:56.173+0000 7fcc50fa5740 1 --2- 172.21.15.118:0/1319942213 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55f665fb6e80 0x55f665fd66d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:16:56.174 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:56.173+0000 7fcc50fa5740 1 -- 172.21.15.118:0/1319942213 >> 172.21.15.118:0/1319942213 conn(0x55f665f7f200 msgr2=0x55f665fb1a90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:16:56.175 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:56.173+0000 7fcc50fa5740 1 -- 172.21.15.118:0/1319942213 shutdown_connections 2024-03-24T00:16:56.175 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:56.173+0000 7fcc50fa5740 1 -- 172.21.15.118:0/1319942213 wait complete. 2024-03-24T00:16:56.175 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:56.174+0000 7fcc50fa5740 20 client.25778 trim_cache size 0 max 16384 2024-03-24T00:16:56.175 INFO:teuthology.orchestra.run.smithi118.stdout:{ 2024-03-24T00:16:56.175 INFO:teuthology.orchestra.run.smithi118.stdout: "status": "IDLE (waiting for more scrubs)", 2024-03-24T00:16:56.175 INFO:teuthology.orchestra.run.smithi118.stdout: "scrubs": { 2024-03-24T00:16:56.175 INFO:teuthology.orchestra.run.smithi118.stdout: "76ef0324-cedd-4952-a156-072938b17479": { 2024-03-24T00:16:56.176 INFO:teuthology.orchestra.run.smithi118.stdout: "path": "/", 2024-03-24T00:16:56.176 INFO:teuthology.orchestra.run.smithi118.stdout: "tag": "76ef0324-cedd-4952-a156-072938b17479", 2024-03-24T00:16:56.176 INFO:teuthology.orchestra.run.smithi118.stdout: "options": "recursive,force" 2024-03-24T00:16:56.176 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-03-24T00:16:56.176 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-03-24T00:16:56.176 INFO:teuthology.orchestra.run.smithi118.stdout:} 2024-03-24T00:16:56.177 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:56.174+0000 7fcc4e200640 1 -- 172.21.15.118:0/1146450709 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7fcc2c0a2950 msgr2=0x7fcc2c0a4e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:16:56.177 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:56.174+0000 7fcc4e200640 1 --2- 172.21.15.118:0/1146450709 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7fcc2c0a2950 0x7fcc2c0a4e10 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x7fcc40005fd0 tx=0x7fcc40009500 comp rx=0 tx=0).stop 2024-03-24T00:16:56.177 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:56.174+0000 7fcc4e200640 1 -- 172.21.15.118:0/1146450709 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fcc48104ad0 msgr2=0x7fcc4819a110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:16:56.177 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:56.174+0000 7fcc4e200640 1 --2- 172.21.15.118:0/1146450709 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fcc48104ad0 0x7fcc4819a110 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7fcc3402c250 tx=0x7fcc34004e30 comp rx=0 tx=0).stop 2024-03-24T00:16:56.177 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:56.175+0000 7fcc4e200640 1 -- 172.21.15.118:0/1146450709 shutdown_connections 2024-03-24T00:16:56.177 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:56.175+0000 7fcc4e200640 1 --2- 172.21.15.118:0/1146450709 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7fcc2c0a2950 0x7fcc2c0a4e10 unknown :-1 s=CLOSED pgs=293 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:16:56.177 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:56.175+0000 7fcc4e200640 1 --2- 172.21.15.118:0/1146450709 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fcc48105490 0x7fcc481a1180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:16:56.177 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:56.175+0000 7fcc4e200640 1 --2- 172.21.15.118:0/1146450709 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fcc48104ad0 0x7fcc4819a110 unknown :-1 s=CLOSED pgs=192 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:16:56.177 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:56.175+0000 7fcc4e200640 1 --2- 172.21.15.118:0/1146450709 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fcc481038e0 0x7fcc48199bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:16:56.177 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:56.175+0000 7fcc4e200640 1 -- 172.21.15.118:0/1146450709 >> 172.21.15.118:0/1146450709 conn(0x7fcc480fee50 msgr2=0x7fcc48107dd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:16:56.177 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:56.175+0000 7fcc4e200640 1 -- 172.21.15.118:0/1146450709 shutdown_connections 2024-03-24T00:16:56.177 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:16:56.175+0000 7fcc4e200640 1 -- 172.21.15.118:0/1146450709 wait complete. 2024-03-24T00:16:56.199 INFO:tasks.cephfs.filesystem:scrub status for tag:76ef0324-cedd-4952-a156-072938b17479 - {'path': '/', 'tag': '76ef0324-cedd-4952-a156-072938b17479', 'options': 'recursive,force'} 2024-03-24T00:16:57.195 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:16:56 smithi118 ceph-mon[31535]: pgmap v435: 97 pgs: 97 active+clean; 673 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 5.5 KiB/s rd, 255 B/s wr, 11 op/s 2024-03-24T00:16:57.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:16:56 smithi118 ceph-mon[31535]: scrub queued for path: / 2024-03-24T00:16:57.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:16:56 smithi118 ceph-mon[31535]: scrub summary: idle+waiting paths [/] 2024-03-24T00:16:57.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:16:56 smithi118 ceph-mon[31535]: scrub summary: active paths [/] 2024-03-24T00:16:57.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:16:56 smithi118 ceph-mon[31535]: scrub summary: idle+waiting paths [/] 2024-03-24T00:16:57.203 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:16:56 smithi189 ceph-mon[36187]: pgmap v435: 97 pgs: 97 active+clean; 673 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 5.5 KiB/s rd, 255 B/s wr, 11 op/s 2024-03-24T00:16:57.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:16:56 smithi189 ceph-mon[36187]: scrub queued for path: / 2024-03-24T00:16:57.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:16:56 smithi189 ceph-mon[36187]: scrub summary: idle+waiting paths [/] 2024-03-24T00:16:57.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:16:56 smithi189 ceph-mon[36187]: scrub summary: active paths [/] 2024-03-24T00:16:57.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:16:56 smithi189 ceph-mon[36187]: scrub summary: idle+waiting paths [/] 2024-03-24T00:16:57.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:16:56 smithi193 ceph-mon[36878]: pgmap v435: 97 pgs: 97 active+clean; 673 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 5.5 KiB/s rd, 255 B/s wr, 11 op/s 2024-03-24T00:16:57.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:16:56 smithi193 ceph-mon[36878]: scrub queued for path: / 2024-03-24T00:16:57.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:16:56 smithi193 ceph-mon[36878]: scrub summary: idle+waiting paths [/] 2024-03-24T00:16:57.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:16:56 smithi193 ceph-mon[36878]: scrub summary: active paths [/] 2024-03-24T00:16:57.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:16:56 smithi193 ceph-mon[36878]: scrub summary: idle+waiting paths [/] 2024-03-24T00:16:59.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:16:58 smithi118 ceph-mon[31535]: pgmap v436: 97 pgs: 97 active+clean; 673 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 5.5 KiB/s rd, 0 B/s wr, 10 op/s 2024-03-24T00:16:59.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:16:58 smithi189 ceph-mon[36187]: pgmap v436: 97 pgs: 97 active+clean; 673 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 5.5 KiB/s rd, 0 B/s wr, 10 op/s 2024-03-24T00:16:59.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:16:58 smithi193 ceph-mon[36878]: pgmap v436: 97 pgs: 97 active+clean; 673 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 5.5 KiB/s rd, 0 B/s wr, 10 op/s 2024-03-24T00:17:00.195 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:16:59 smithi118 ceph-mon[31535]: pgmap v437: 97 pgs: 97 active+clean; 674 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 255 B/s wr, 12 op/s 2024-03-24T00:17:00.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:16:59 smithi189 ceph-mon[36187]: pgmap v437: 97 pgs: 97 active+clean; 674 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 255 B/s wr, 12 op/s 2024-03-24T00:17:00.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:16:59 smithi193 ceph-mon[36878]: pgmap v437: 97 pgs: 97 active+clean; 674 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 255 B/s wr, 12 op/s 2024-03-24T00:17:01.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:17:00 smithi118 ceph-mon[31535]: scrub summary: idle 2024-03-24T00:17:01.203 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:17:00 smithi189 ceph-mon[36187]: scrub summary: idle 2024-03-24T00:17:01.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:17:00 smithi193 ceph-mon[36878]: scrub summary: idle 2024-03-24T00:17:02.195 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:17:01 smithi118 ceph-mon[31535]: pgmap v438: 97 pgs: 97 active+clean; 674 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 5.3 KiB/s rd, 255 B/s wr, 10 op/s 2024-03-24T00:17:02.203 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:17:01 smithi189 ceph-mon[36187]: pgmap v438: 97 pgs: 97 active+clean; 674 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 5.3 KiB/s rd, 255 B/s wr, 10 op/s 2024-03-24T00:17:02.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:17:01 smithi193 ceph-mon[36878]: pgmap v438: 97 pgs: 97 active+clean; 674 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 5.3 KiB/s rd, 255 B/s wr, 10 op/s 2024-03-24T00:17:05.195 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:17:04 smithi118 ceph-mon[31535]: pgmap v439: 97 pgs: 97 active+clean; 674 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 255 B/s wr, 11 op/s 2024-03-24T00:17:05.203 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:17:04 smithi189 ceph-mon[36187]: pgmap v439: 97 pgs: 97 active+clean; 674 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 255 B/s wr, 11 op/s 2024-03-24T00:17:05.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:17:04 smithi193 ceph-mon[36878]: pgmap v439: 97 pgs: 97 active+clean; 674 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 255 B/s wr, 11 op/s 2024-03-24T00:17:07.195 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:17:06 smithi118 ceph-mon[31535]: pgmap v440: 97 pgs: 97 active+clean; 674 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 6.6 KiB/s rd, 255 B/s wr, 12 op/s 2024-03-24T00:17:07.203 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:17:06 smithi189 ceph-mon[36187]: pgmap v440: 97 pgs: 97 active+clean; 674 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 6.6 KiB/s rd, 255 B/s wr, 12 op/s 2024-03-24T00:17:07.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:17:06 smithi193 ceph-mon[36878]: pgmap v440: 97 pgs: 97 active+clean; 674 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 6.6 KiB/s rd, 255 B/s wr, 12 op/s 2024-03-24T00:17:07.353 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:17:07.352+0000 7f666b200640 1 -- 172.21.15.118:0/1143103520 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f666c1048a0 msgr2=0x7f666c106c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:17:07.353 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:17:07.352+0000 7f666b200640 1 --2- 172.21.15.118:0/1143103520 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f666c1048a0 0x7f666c106c90 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7f6658009900 tx=0x7f665802bc30 comp rx=0 tx=0).stop 2024-03-24T00:17:07.353 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:17:07.353+0000 7f666b200640 1 -- 172.21.15.118:0/1143103520 shutdown_connections 2024-03-24T00:17:07.353 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:17:07.353+0000 7f666b200640 1 --2- 172.21.15.118:0/1143103520 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f666c1071d0 0x7f666c109620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:17:07.353 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:17:07.353+0000 7f666b200640 1 --2- 172.21.15.118:0/1143103520 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f666c1048a0 0x7f666c106c90 unknown :-1 s=CLOSED pgs=260 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:17:07.353 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:17:07.353+0000 7f666b200640 1 --2- 172.21.15.118:0/1143103520 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f666c101f70 0x7f666c104360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:17:07.353 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:17:07.353+0000 7f666b200640 1 -- 172.21.15.118:0/1143103520 >> 172.21.15.118:0/1143103520 conn(0x7f666c0fb980 msgr2=0x7f666c0fdda0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:17:07.353 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:17:07.353+0000 7f666b200640 1 -- 172.21.15.118:0/1143103520 shutdown_connections 2024-03-24T00:17:07.353 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:17:07.353+0000 7f666b200640 1 -- 172.21.15.118:0/1143103520 wait complete. 2024-03-24T00:17:07.354 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:17:07.354+0000 7f666b200640 1 Processor -- start 2024-03-24T00:17:07.354 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:17:07.355+0000 7f666b200640 1 -- start start 2024-03-24T00:17:07.355 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:17:07.355+0000 7f666b200640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f666c101f70 0x7f666c199a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:17:07.355 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:17:07.355+0000 7f666b200640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f666c1048a0 0x7f666c199fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:17:07.356 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:17:07.355+0000 7f666b200640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f666c1071d0 0x7f666c1a1030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:17:07.356 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:17:07.355+0000 7f666b200640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f666c10d1d0 con 0x7f666c1071d0 2024-03-24T00:17:07.356 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:17:07.355+0000 7f666b200640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f666c10d050 con 0x7f666c1048a0 2024-03-24T00:17:07.356 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:17:07.355+0000 7f666b200640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f666c10d350 con 0x7f666c101f70 2024-03-24T00:17:07.356 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:17:07.355+0000 7f6669e00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f666c101f70 0x7f666c199a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:17:07.356 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:17:07.355+0000 7f6669400640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f666c1048a0 0x7f666c199fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:17:07.356 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:17:07.355+0000 7f6669e00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f666c101f70 0x7f666c199a80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.118:34278/0 (socket says 172.21.15.118:34278) 2024-03-24T00:17:07.356 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:17:07.355+0000 7f6669e00640 1 -- 172.21.15.118:0/44379829 learned_addr learned my addr 172.21.15.118:0/44379829 (peer_addr_for_me v2:172.21.15.118:0/0) 2024-03-24T00:17:07.356 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:17:07.355+0000 7f666a800640 1 --2- 172.21.15.118:0/44379829 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f666c1071d0 0x7f666c1a1030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:17:07.356 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:17:07.356+0000 7f6669400640 1 -- 172.21.15.118:0/44379829 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f666c101f70 msgr2=0x7f666c199a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:17:07.356 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:17:07.356+0000 7f6669400640 1 --2- 172.21.15.118:0/44379829 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f666c101f70 0x7f666c199a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:17:07.356 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:17:07.356+0000 7f6669400640 1 -- 172.21.15.118:0/44379829 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f666c1071d0 msgr2=0x7f666c1a1030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:17:07.356 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:17:07.356+0000 7f6669400640 1 --2- 172.21.15.118:0/44379829 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f666c1071d0 0x7f666c1a1030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:17:07.357 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:17:07.356+0000 7f6669400640 1 -- 172.21.15.118:0/44379829 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f666c1a16a0 con 0x7f666c1048a0 2024-03-24T00:17:07.357 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:17:07.356+0000 7f6669e00640 1 --2- 172.21.15.118:0/44379829 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f666c101f70 0x7f666c199a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T00:17:07.357 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:17:07.356+0000 7f6669400640 1 --2- 172.21.15.118:0/44379829 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f666c1048a0 0x7f666c199fc0 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7f6658009230 tx=0x7f665802c6c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:17:07.357 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:17:07.356+0000 7f6662a00640 1 -- 172.21.15.118:0/44379829 <== mon.2 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6658043020 con 0x7f666c1048a0 2024-03-24T00:17:07.357 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:17:07.356+0000 7f666b200640 1 -- 172.21.15.118:0/44379829 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f666c1a1930 con 0x7f666c1048a0 2024-03-24T00:17:07.358 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:17:07.356+0000 7f6662a00640 1 -- 172.21.15.118:0/44379829 <== mon.2 v2:172.21.15.189:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f66580054e0 con 0x7f666c1048a0 2024-03-24T00:17:07.358 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:17:07.356+0000 7f6662a00640 1 -- 172.21.15.118:0/44379829 <== mon.2 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6658003c60 con 0x7f666c1048a0 2024-03-24T00:17:07.358 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:17:07.357+0000 7f666b200640 1 -- 172.21.15.118:0/44379829 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f666c1a1ea0 con 0x7f666c1048a0 2024-03-24T00:17:07.360 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:17:07.361+0000 7f6662a00640 1 -- 172.21.15.118:0/44379829 <== mon.2 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 20) v1 ==== 137291+0+0 (secure 0 0 0) 0x7f6658046020 con 0x7f666c1048a0 2024-03-24T00:17:07.361 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:17:07.361+0000 7f6673be0740 1 Processor -- start 2024-03-24T00:17:07.361 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:17:07.362+0000 7f6673be0740 1 -- start start 2024-03-24T00:17:07.362 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:17:07.363+0000 7f6673be0740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x560155d41c00 0x560155d42020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:17:07.362 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:17:07.363+0000 7f6673be0740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x560155d425f0 0x560155d470f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:17:07.363 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:17:07.363+0000 7f666a800640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x560155d41c00 0x560155d42020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:17:07.363 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:17:07.363+0000 7f6673be0740 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x560155d47760 0x560155d49bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:17:07.363 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:17:07.363+0000 7f666a800640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x560155d41c00 0x560155d42020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.118:47508/0 (socket says 172.21.15.118:47508) 2024-03-24T00:17:07.363 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:17:07.363+0000 7f6673be0740 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x560155cfc3b0 con 0x560155d47760 2024-03-24T00:17:07.363 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:17:07.363+0000 7f666a800640 1 -- 172.21.15.118:0/2756807273 learned_addr learned my addr 172.21.15.118:0/2756807273 (peer_addr_for_me v2:172.21.15.118:0/0) 2024-03-24T00:17:07.363 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:17:07.363+0000 7f6673be0740 1 -- 172.21.15.118:0/2756807273 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x560155c9e830 con 0x560155d41c00 2024-03-24T00:17:07.363 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:17:07.363+0000 7f6673be0740 1 -- 172.21.15.118:0/2756807273 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x560155c475d0 con 0x560155d425f0 2024-03-24T00:17:07.364 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:17:07.363+0000 7f6669e00640 1 --2- 172.21.15.118:0/2756807273 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x560155d425f0 0x560155d470f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:17:07.364 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:17:07.363+0000 7f6662a00640 1 --2- 172.21.15.118:0/44379829 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7f66500a28f0 0x7f66500a4db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:17:07.364 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:17:07.364+0000 7f6662a00640 1 -- 172.21.15.118:0/44379829 <== mon.2 v2:172.21.15.189:3300/0 5 ==== osd_map(90..90 src has 1..90) v4 ==== 8277+0+0 (secure 0 0 0) 0x7f66580f0b80 con 0x7f666c1048a0 2024-03-24T00:17:07.365 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:17:07.364+0000 7f6669e00640 1 -- 172.21.15.118:0/2756807273 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x560155d41c00 msgr2=0x560155d42020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:17:07.365 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:17:07.364+0000 7f6669e00640 1 --2- 172.21.15.118:0/2756807273 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x560155d41c00 0x560155d42020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:17:07.365 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:17:07.364+0000 7f6669e00640 1 -- 172.21.15.118:0/2756807273 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x560155d47760 msgr2=0x560155d49bf0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-24T00:17:07.365 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:17:07.364+0000 7f6669e00640 1 --2- 172.21.15.118:0/2756807273 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x560155d47760 0x560155d49bf0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:17:07.365 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:17:07.364+0000 7f6669e00640 1 -- 172.21.15.118:0/2756807273 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x560155c5f2c0 con 0x560155d425f0 2024-03-24T00:17:07.365 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:17:07.364+0000 7f666a800640 1 --2- 172.21.15.118:0/2756807273 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x560155d41c00 0x560155d42020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-24T00:17:07.365 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:17:07.364+0000 7f6669e00640 1 --2- 172.21.15.118:0/2756807273 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x560155d425f0 0x560155d470f0 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7f665c011510 tx=0x7f665c0119d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=445333a8d37d74e3 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:17:07.365 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:17:07.364+0000 7f666a800640 1 --2- 172.21.15.118:0/44379829 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7f66500a28f0 0x7f66500a4db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:17:07.365 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:17:07.365+0000 7f6657400640 1 -- 172.21.15.118:0/2756807273 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f665c01e070 con 0x560155d425f0 2024-03-24T00:17:07.366 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:17:07.365+0000 7f6657400640 1 -- 172.21.15.118:0/2756807273 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f665c00faa0 con 0x560155d425f0 2024-03-24T00:17:07.366 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:17:07.365+0000 7f6657400640 1 -- 172.21.15.118:0/2756807273 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f665c01e070 con 0x560155d425f0 2024-03-24T00:17:07.366 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:17:07.366+0000 7f666a800640 1 --2- 172.21.15.118:0/44379829 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7f66500a28f0 0x7f66500a4db0 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 crypto rx=0x560155d42e50 tx=0x7f6664004a40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:17:07.367 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:17:07.366+0000 7f6673be0740 1 -- 172.21.15.118:0/2756807273 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x560155d425f0 msgr2=0x560155d470f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:17:07.367 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:17:07.366+0000 7f6673be0740 1 --2- 172.21.15.118:0/2756807273 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x560155d425f0 0x560155d470f0 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7f665c011510 tx=0x7f665c0119d0 comp rx=0 tx=0).stop 2024-03-24T00:17:07.367 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:17:07.366+0000 7f6673be0740 1 -- 172.21.15.118:0/2756807273 shutdown_connections 2024-03-24T00:17:07.367 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:17:07.366+0000 7f6673be0740 1 --2- 172.21.15.118:0/2756807273 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x560155d47760 0x560155d49bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:17:07.367 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:17:07.366+0000 7f6673be0740 1 --2- 172.21.15.118:0/2756807273 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x560155d425f0 0x560155d470f0 unknown :-1 s=CLOSED pgs=261 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:17:07.367 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:17:07.366+0000 7f6673be0740 1 --2- 172.21.15.118:0/2756807273 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x560155d41c00 0x560155d42020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:17:07.367 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:17:07.366+0000 7f6673be0740 1 -- 172.21.15.118:0/2756807273 >> 172.21.15.118:0/2756807273 conn(0x560155d34960 msgr2=0x560155d34d60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:17:07.367 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:17:07.367+0000 7f6673be0740 1 -- 172.21.15.118:0/2756807273 shutdown_connections 2024-03-24T00:17:07.367 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:17:07.367+0000 7f6673be0740 1 -- 172.21.15.118:0/2756807273 wait complete. 2024-03-24T00:17:07.368 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:17:07.367+0000 7f6673be0740 1 -- start start 2024-03-24T00:17:07.368 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:17:07.368+0000 7f6673be0740 1 Processor -- start 2024-03-24T00:17:07.370 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:17:07.368+0000 7f6673be0740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x560155d41c00 0x560155d4be40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:17:07.370 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:17:07.368+0000 7f6673be0740 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x560155d425f0 0x560155d4c380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:17:07.370 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:17:07.368+0000 7f6673be0740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x560155d47760 0x560155d533c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:17:07.370 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:17:07.368+0000 7f6673be0740 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x560155c5aa20 con 0x560155d425f0 2024-03-24T00:17:07.370 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:17:07.368+0000 7f6673be0740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x560155c9a8d0 con 0x560155d47760 2024-03-24T00:17:07.370 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:17:07.368+0000 7f6673be0740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x560155d4a710 con 0x560155d41c00 2024-03-24T00:17:07.370 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:17:07.368+0000 7f6673be0740 10 client.0 _ll_register_callbacks cb 0x560155c971c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T00:17:07.371 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:17:07.369+0000 7f666a800640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x560155d47760 0x560155d533c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:17:07.371 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:17:07.369+0000 7f666a800640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x560155d47760 0x560155d533c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.118:47518/0 (socket says 172.21.15.118:47518) 2024-03-24T00:17:07.371 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:17:07.369+0000 7f666a800640 1 -- 172.21.15.118:0/4128488452 learned_addr learned my addr 172.21.15.118:0/4128488452 (peer_addr_for_me v2:172.21.15.118:0/0) 2024-03-24T00:17:07.371 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:17:07.369+0000 7f6669400640 1 --2- 172.21.15.118:0/4128488452 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x560155d425f0 0x560155d4c380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:17:07.371 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:17:07.369+0000 7f666a800640 1 -- 172.21.15.118:0/4128488452 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x560155d41c00 msgr2=0x560155d4be40 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-03-24T00:17:07.371 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:17:07.369+0000 7f666a800640 1 --2- 172.21.15.118:0/4128488452 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x560155d41c00 0x560155d4be40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:17:07.371 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:17:07.369+0000 7f666a800640 1 -- 172.21.15.118:0/4128488452 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x560155d425f0 msgr2=0x560155d4c380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:17:07.371 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:17:07.369+0000 7f666a800640 1 --2- 172.21.15.118:0/4128488452 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x560155d425f0 0x560155d4c380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:17:07.371 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:17:07.369+0000 7f666a800640 1 -- 172.21.15.118:0/4128488452 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5601559b3210 con 0x560155d47760 2024-03-24T00:17:07.371 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:17:07.369+0000 7f666a800640 1 --2- 172.21.15.118:0/4128488452 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x560155d47760 0x560155d533c0 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7f6664013700 tx=0x7f6664004ab0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c04e5f90b191e704 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:17:07.371 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:17:07.369+0000 7f664be00640 10 client.0 ms_handle_connect on v2:172.21.15.189:3300/0 2024-03-24T00:17:07.371 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:17:07.370+0000 7f664be00640 1 -- 172.21.15.118:0/4128488452 <== mon.2 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f666401c9e0 con 0x560155d47760 2024-03-24T00:17:07.372 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:17:07.370+0000 7f664be00640 1 -- 172.21.15.118:0/4128488452 <== mon.2 v2:172.21.15.189:3300/0 2 ==== osd_map(90..90 src has 1..90) v4 ==== 8277+0+0 (secure 0 0 0) 0x7f666401cb80 con 0x560155d47760 2024-03-24T00:17:07.372 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:17:07.370+0000 7f6673be0740 1 -- 172.21.15.118:0/4128488452 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f665c00faa0 con 0x560155d47760 2024-03-24T00:17:07.372 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:17:07.370+0000 7f664be00640 1 -- 172.21.15.118:0/4128488452 <== mon.2 v2:172.21.15.189:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) v2 ==== 24+0+0 (secure 0 0 0) 0x7f666401cd80 con 0x560155d47760 2024-03-24T00:17:07.372 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:17:07.370+0000 7f6673be0740 10 client.25787 fetch_fsmap learned FSMap version 28 2024-03-24T00:17:07.372 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:17:07.370+0000 7f6673be0740 1 -- 172.21.15.118:0/4128488452 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) v3 -- 0x560155a7f620 con 0x560155d47760 2024-03-24T00:17:07.372 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:17:07.370+0000 7f664be00640 1 -- 172.21.15.118:0/4128488452 <== mon.2 v2:172.21.15.189:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f665c00faa0 con 0x560155d47760 2024-03-24T00:17:07.376 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:17:07.371+0000 7f664be00640 1 -- 172.21.15.118:0/4128488452 <== mon.2 v2:172.21.15.189:3300/0 5 ==== fsmap(e 28) v1 ==== 7934+0+0 (secure 0 0 0) 0x7f6664019030 con 0x560155d47760 2024-03-24T00:17:07.376 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:17:07.371+0000 7f6673be0740 10 client.25787 fetch_fsmap finished waiting for FSMap version 28 2024-03-24T00:17:07.377 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:17:07.371+0000 7f6673be0740 10 client.25787 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-24T00:17:07.377 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:17:07.371+0000 7f6673be0740 20 client.25787 populate_metadata read hostname 'smithi118' 2024-03-24T00:17:07.377 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:17:07.371+0000 7f6673be0740 1 --2- 172.21.15.118:0/4128488452 >> [v2:172.21.15.189:6838/4074484168,v1:172.21.15.189:6839/4074484168] conn(0x560155d57c90 0x560155d5a080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:17:07.377 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:17:07.371+0000 7f6673be0740 4 client.25787 mds_command: new command op to 24569 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-24T00:17:07.377 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:17:07.371+0000 7f6673be0740 1 -- 172.21.15.118:0/4128488452 --> [v2:172.21.15.189:6838/4074484168,v1:172.21.15.189:6839/4074484168] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x560155cabd70 con 0x560155d57c90 2024-03-24T00:17:07.377 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:17:07.371+0000 7f6669e00640 1 --2- 172.21.15.118:0/4128488452 >> [v2:172.21.15.189:6838/4074484168,v1:172.21.15.189:6839/4074484168] conn(0x560155d57c90 0x560155d5a080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:17:07.378 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:17:07.372+0000 7f6669e00640 1 --2- 172.21.15.118:0/4128488452 >> [v2:172.21.15.189:6838/4074484168,v1:172.21.15.189:6839/4074484168] conn(0x560155d57c90 0x560155d5a080 crc :-1 s=READY pgs=63 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=63bf4b7b38ed711d server_cookie=75cc7cfa311210fd in_seq=0 out_seq=0 2024-03-24T00:17:07.378 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:17:07.372+0000 7f664be00640 10 client.25787 ms_handle_connect on v2:172.21.15.189:6838/4074484168 2024-03-24T00:17:07.378 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:17:07.372+0000 7f664be00640 1 -- 172.21.15.118:0/4128488452 <== mds.0 v2:172.21.15.189:6838/4074484168 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x560155cabd70 con 0x560155d57c90 2024-03-24T00:17:07.378 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:17:07.372+0000 7f664be00640 10 client.25787 handle_command_reply: tid=0 2024-03-24T00:17:07.378 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:17:07.372+0000 7f6673be0740 1 client.25787 shutdown 2024-03-24T00:17:07.378 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:17:07.373+0000 7f6673be0740 1 -- 172.21.15.118:0/4128488452 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x560155d47760 msgr2=0x560155d533c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:17:07.378 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:17:07.373+0000 7f6673be0740 1 --2- 172.21.15.118:0/4128488452 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x560155d47760 0x560155d533c0 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7f6664013700 tx=0x7f6664004ab0 comp rx=0 tx=0).stop 2024-03-24T00:17:07.379 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:17:07.373+0000 7f6673be0740 1 -- 172.21.15.118:0/4128488452 shutdown_connections 2024-03-24T00:17:07.379 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:17:07.373+0000 7f6673be0740 1 --2- 172.21.15.118:0/4128488452 >> [v2:172.21.15.189:6838/4074484168,v1:172.21.15.189:6839/4074484168] conn(0x560155d57c90 0x560155d5a080 crc :-1 s=READY pgs=63 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:17:07.379 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:17:07.373+0000 7f6673be0740 1 --2- 172.21.15.118:0/4128488452 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x560155d47760 0x560155d533c0 unknown :-1 s=CLOSED pgs=272 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:17:07.380 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:17:07.373+0000 7f6673be0740 1 --2- 172.21.15.118:0/4128488452 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x560155d425f0 0x560155d4c380 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:17:07.380 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:17:07.373+0000 7f6673be0740 1 --2- 172.21.15.118:0/4128488452 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x560155d41c00 0x560155d4be40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:17:07.380 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:17:07.373+0000 7f664be00640 0 client.25787 ms_handle_reset on v2:172.21.15.189:6838/4074484168 2024-03-24T00:17:07.380 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:17:07.373+0000 7f6673be0740 1 -- 172.21.15.118:0/4128488452 >> 172.21.15.118:0/4128488452 conn(0x560155d34960 msgr2=0x560155d33b70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:17:07.380 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:17:07.374+0000 7f6673be0740 1 -- 172.21.15.118:0/4128488452 shutdown_connections 2024-03-24T00:17:07.380 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:17:07.374+0000 7f6673be0740 1 -- 172.21.15.118:0/4128488452 wait complete. 2024-03-24T00:17:07.380 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:17:07.374+0000 7f6673be0740 20 client.25787 trim_cache size 0 max 16384 2024-03-24T00:17:07.390 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:17:07.389+0000 7f6673be0740 1 Processor -- start 2024-03-24T00:17:07.390 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:17:07.390+0000 7f6673be0740 1 -- start start 2024-03-24T00:17:07.391 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:17:07.390+0000 7f6673be0740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x560155d41c00 0x560155d43950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:17:07.391 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:17:07.390+0000 7f6673be0740 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x560155d425f0 0x560155d43e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:17:07.391 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:17:07.390+0000 7f6673be0740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x560155d47760 0x560155d53380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:17:07.391 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:17:07.390+0000 7f6673be0740 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f666c1a65c0 con 0x560155d425f0 2024-03-24T00:17:07.391 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:17:07.390+0000 7f6673be0740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f666c1a7f40 con 0x560155d41c00 2024-03-24T00:17:07.391 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:17:07.390+0000 7f6673be0740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f666c1a6c20 con 0x560155d47760 2024-03-24T00:17:07.391 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:17:07.390+0000 7f6669e00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x560155d41c00 0x560155d43950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:17:07.391 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:17:07.390+0000 7f6669e00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x560155d41c00 0x560155d43950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.118:47522/0 (socket says 172.21.15.118:47522) 2024-03-24T00:17:07.391 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:17:07.390+0000 7f6669e00640 1 -- 172.21.15.118:0/1423130368 learned_addr learned my addr 172.21.15.118:0/1423130368 (peer_addr_for_me v2:172.21.15.118:0/0) 2024-03-24T00:17:07.391 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:17:07.390+0000 7f6669400640 1 --2- 172.21.15.118:0/1423130368 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x560155d425f0 0x560155d43e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:17:07.392 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:17:07.390+0000 7f666a800640 1 --2- 172.21.15.118:0/1423130368 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x560155d47760 0x560155d53380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:17:07.392 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:17:07.390+0000 7f6669e00640 1 -- 172.21.15.118:0/1423130368 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x560155d47760 msgr2=0x560155d53380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:17:07.392 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:17:07.390+0000 7f6669e00640 1 --2- 172.21.15.118:0/1423130368 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x560155d47760 0x560155d53380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:17:07.392 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:17:07.390+0000 7f6669e00640 1 -- 172.21.15.118:0/1423130368 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x560155d425f0 msgr2=0x560155d43e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:17:07.392 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:17:07.390+0000 7f6669e00640 1 --2- 172.21.15.118:0/1423130368 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x560155d425f0 0x560155d43e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:17:07.392 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:17:07.390+0000 7f6669e00640 1 -- 172.21.15.118:0/1423130368 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x560155d32f80 con 0x560155d41c00 2024-03-24T00:17:07.392 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:17:07.391+0000 7f6669400640 1 --2- 172.21.15.118:0/1423130368 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x560155d425f0 0x560155d43e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-24T00:17:07.392 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:17:07.391+0000 7f6669e00640 1 --2- 172.21.15.118:0/1423130368 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x560155d41c00 0x560155d43950 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7f665c009810 tx=0x7f665c0097a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=4dd9535f367d0ce9 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:17:07.392 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:17:07.391+0000 7f6656a00640 1 -- 172.21.15.118:0/1423130368 <== mon.2 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f665c02b020 con 0x560155d41c00 2024-03-24T00:17:07.392 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:17:07.391+0000 7f6656a00640 1 -- 172.21.15.118:0/1423130368 <== mon.2 v2:172.21.15.189:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f665c01f040 con 0x560155d41c00 2024-03-24T00:17:07.392 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:17:07.391+0000 7f6656a00640 1 -- 172.21.15.118:0/1423130368 <== mon.2 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f665c01e070 con 0x560155d41c00 2024-03-24T00:17:07.395 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:17:07.392+0000 7f6673be0740 1 -- 172.21.15.118:0/1423130368 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x560155d41c00 msgr2=0x560155d43950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:17:07.395 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:17:07.392+0000 7f6673be0740 1 --2- 172.21.15.118:0/1423130368 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x560155d41c00 0x560155d43950 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7f665c009810 tx=0x7f665c0097a0 comp rx=0 tx=0).stop 2024-03-24T00:17:07.396 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:17:07.392+0000 7f6673be0740 1 -- 172.21.15.118:0/1423130368 shutdown_connections 2024-03-24T00:17:07.396 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:17:07.392+0000 7f6673be0740 1 --2- 172.21.15.118:0/1423130368 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x560155d47760 0x560155d53380 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:17:07.396 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:17:07.392+0000 7f6673be0740 1 --2- 172.21.15.118:0/1423130368 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x560155d425f0 0x560155d43e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:17:07.396 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:17:07.392+0000 7f6673be0740 1 --2- 172.21.15.118:0/1423130368 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x560155d41c00 0x560155d43950 unknown :-1 s=CLOSED pgs=273 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:17:07.396 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:17:07.392+0000 7f6673be0740 1 -- 172.21.15.118:0/1423130368 >> 172.21.15.118:0/1423130368 conn(0x560155d34960 msgr2=0x560155d34d40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:17:07.396 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:17:07.392+0000 7f6673be0740 1 -- 172.21.15.118:0/1423130368 shutdown_connections 2024-03-24T00:17:07.396 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:17:07.392+0000 7f6673be0740 1 -- 172.21.15.118:0/1423130368 wait complete. 2024-03-24T00:17:07.396 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:17:07.392+0000 7f6673be0740 1 -- start start 2024-03-24T00:17:07.396 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:17:07.392+0000 7f6673be0740 1 Processor -- start 2024-03-24T00:17:07.396 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:17:07.393+0000 7f6673be0740 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x560155d41c00 0x560155d50df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:17:07.396 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:17:07.393+0000 7f6673be0740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x560155d425f0 0x560155d51330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:17:07.396 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:17:07.393+0000 7f6673be0740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x560155d47760 0x560155d52340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:17:07.396 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:17:07.393+0000 7f6673be0740 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f666c1a78e0 con 0x560155d41c00 2024-03-24T00:17:07.396 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:17:07.393+0000 7f6673be0740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f666c1a7280 con 0x560155d425f0 2024-03-24T00:17:07.397 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:17:07.393+0000 7f6673be0740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f666c1a4780 con 0x560155d47760 2024-03-24T00:17:07.397 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:17:07.393+0000 7f6673be0740 10 client.0 _ll_register_callbacks cb 0x7f6664009bf0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T00:17:07.397 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:17:07.393+0000 7f6669400640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x560155d425f0 0x560155d51330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:17:07.397 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:17:07.393+0000 7f6669400640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x560155d425f0 0x560155d51330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.118:47526/0 (socket says 172.21.15.118:47526) 2024-03-24T00:17:07.397 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:17:07.393+0000 7f6669400640 1 -- 172.21.15.118:0/1464847953 learned_addr learned my addr 172.21.15.118:0/1464847953 (peer_addr_for_me v2:172.21.15.118:0/0) 2024-03-24T00:17:07.397 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:17:07.393+0000 7f666a800640 1 --2- 172.21.15.118:0/1464847953 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x560155d47760 0x560155d52340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:17:07.397 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:17:07.393+0000 7f6669e00640 1 --2- 172.21.15.118:0/1464847953 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x560155d41c00 0x560155d50df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:17:07.397 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:17:07.394+0000 7f666a800640 1 -- 172.21.15.118:0/1464847953 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x560155d425f0 msgr2=0x560155d51330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:17:07.397 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:17:07.394+0000 7f666a800640 1 --2- 172.21.15.118:0/1464847953 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x560155d425f0 0x560155d51330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:17:07.397 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:17:07.394+0000 7f666a800640 1 -- 172.21.15.118:0/1464847953 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x560155d41c00 msgr2=0x560155d50df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:17:07.397 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:17:07.394+0000 7f666a800640 1 --2- 172.21.15.118:0/1464847953 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x560155d41c00 0x560155d50df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:17:07.397 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:17:07.394+0000 7f666a800640 1 -- 172.21.15.118:0/1464847953 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x560155d52930 con 0x560155d47760 2024-03-24T00:17:07.397 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:17:07.394+0000 7f666a800640 1 --2- 172.21.15.118:0/1464847953 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x560155d47760 0x560155d52340 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7f6664011ce0 tx=0x7f66640134f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=7c3c6d0fd5487e4e server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:17:07.397 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:17:07.394+0000 7f664be00640 10 client.0 ms_handle_connect on v2:172.21.15.193:3300/0 2024-03-24T00:17:07.397 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:17:07.394+0000 7f664be00640 1 -- 172.21.15.118:0/1464847953 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f666403d060 con 0x560155d47760 2024-03-24T00:17:07.397 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:17:07.394+0000 7f6673be0740 1 -- 172.21.15.118:0/1464847953 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f665c01f040 con 0x560155d47760 2024-03-24T00:17:07.398 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:17:07.394+0000 7f664be00640 1 -- 172.21.15.118:0/1464847953 <== mon.1 v2:172.21.15.193:3300/0 2 ==== osd_map(90..90 src has 1..90) v4 ==== 8277+0+0 (secure 0 0 0) 0x7f666401dda0 con 0x560155d47760 2024-03-24T00:17:07.398 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:17:07.395+0000 7f664be00640 1 -- 172.21.15.118:0/1464847953 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) v2 ==== 24+0+0 (secure 0 0 0) 0x7f6664025470 con 0x560155d47760 2024-03-24T00:17:07.398 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:17:07.395+0000 7f6673be0740 10 client.25780 fetch_fsmap learned FSMap version 28 2024-03-24T00:17:07.398 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:17:07.395+0000 7f6673be0740 1 -- 172.21.15.118:0/1464847953 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) v3 -- 0x560155db37d0 con 0x560155d47760 2024-03-24T00:17:07.398 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:17:07.395+0000 7f664be00640 1 -- 172.21.15.118:0/1464847953 <== mon.1 v2:172.21.15.193:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f665c01f040 con 0x560155d47760 2024-03-24T00:17:07.398 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:17:07.395+0000 7f664be00640 1 -- 172.21.15.118:0/1464847953 <== mon.1 v2:172.21.15.193:3300/0 5 ==== fsmap(e 28) v1 ==== 7934+0+0 (secure 0 0 0) 0x7f6664019040 con 0x560155d47760 2024-03-24T00:17:07.398 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:17:07.395+0000 7f6673be0740 10 client.25780 fetch_fsmap finished waiting for FSMap version 28 2024-03-24T00:17:07.398 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:17:07.395+0000 7f6673be0740 10 client.25780 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-24T00:17:07.398 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:17:07.396+0000 7f6673be0740 20 client.25780 populate_metadata read hostname 'smithi118' 2024-03-24T00:17:07.398 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:17:07.396+0000 7f6673be0740 1 --2- 172.21.15.118:0/1464847953 >> [v2:172.21.15.189:6838/4074484168,v1:172.21.15.189:6839/4074484168] conn(0x560155db40a0 0x560155db64c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:17:07.398 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:17:07.396+0000 7f6673be0740 4 client.25780 mds_command: new command op to 24569 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-03-24T00:17:07.398 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:17:07.396+0000 7f6673be0740 1 -- 172.21.15.118:0/1464847953 --> [v2:172.21.15.189:6838/4074484168,v1:172.21.15.189:6839/4074484168] -- command(tid 0: {"prefix": "flush journal"}) v1 -- 0x560155ca60c0 con 0x560155db40a0 2024-03-24T00:17:07.398 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:17:07.396+0000 7f6669e00640 1 --2- 172.21.15.118:0/1464847953 >> [v2:172.21.15.189:6838/4074484168,v1:172.21.15.189:6839/4074484168] conn(0x560155db40a0 0x560155db64c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:17:07.399 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:17:07.397+0000 7f664be00640 10 client.25780 ms_handle_connect on v2:172.21.15.189:6838/4074484168 2024-03-24T00:17:07.399 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:17:07.397+0000 7f6669e00640 1 --2- 172.21.15.118:0/1464847953 >> [v2:172.21.15.189:6838/4074484168,v1:172.21.15.189:6839/4074484168] conn(0x560155db40a0 0x560155db64c0 crc :-1 s=READY pgs=64 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=41b7344bbeba52a server_cookie=502d559ddfebf283 in_seq=0 out_seq=0 2024-03-24T00:17:07.402 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:17:07.402+0000 7f664be00640 1 -- 172.21.15.118:0/1464847953 <== mds.0 v2:172.21.15.189:6838/4074484168 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+44 (crc 0 0 0) 0x560155ca60c0 con 0x560155db40a0 2024-03-24T00:17:07.402 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:17:07.402+0000 7f664be00640 10 client.25780 handle_command_reply: tid=0 2024-03-24T00:17:07.402 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:17:07.402+0000 7f6673be0740 1 client.25780 shutdown 2024-03-24T00:17:07.402 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:17:07.403+0000 7f6673be0740 1 -- 172.21.15.118:0/1464847953 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x560155d47760 msgr2=0x560155d52340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:17:07.402 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:17:07.403+0000 7f6673be0740 1 --2- 172.21.15.118:0/1464847953 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x560155d47760 0x560155d52340 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7f6664011ce0 tx=0x7f66640134f0 comp rx=0 tx=0).stop 2024-03-24T00:17:07.404 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:17:07.403+0000 7f6673be0740 1 -- 172.21.15.118:0/1464847953 shutdown_connections 2024-03-24T00:17:07.404 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:17:07.403+0000 7f6673be0740 1 --2- 172.21.15.118:0/1464847953 >> [v2:172.21.15.189:6838/4074484168,v1:172.21.15.189:6839/4074484168] conn(0x560155db40a0 0x560155db64c0 crc :-1 s=READY pgs=64 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:17:07.404 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:17:07.403+0000 7f6673be0740 1 --2- 172.21.15.118:0/1464847953 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x560155d47760 0x560155d52340 unknown :-1 s=CLOSED pgs=262 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:17:07.405 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:17:07.403+0000 7f6673be0740 1 --2- 172.21.15.118:0/1464847953 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x560155d425f0 0x560155d51330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:17:07.405 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:17:07.403+0000 7f6673be0740 1 --2- 172.21.15.118:0/1464847953 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x560155d41c00 0x560155d50df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:17:07.405 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:17:07.403+0000 7f6673be0740 1 -- 172.21.15.118:0/1464847953 >> 172.21.15.118:0/1464847953 conn(0x560155d34960 msgr2=0x560155d34d40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:17:07.405 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:17:07.403+0000 7f664be00640 0 client.25780 ms_handle_reset on v2:172.21.15.189:6838/4074484168 2024-03-24T00:17:07.405 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:17:07.403+0000 7f6673be0740 1 -- 172.21.15.118:0/1464847953 shutdown_connections 2024-03-24T00:17:07.405 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:17:07.403+0000 7f6673be0740 1 -- 172.21.15.118:0/1464847953 wait complete. 2024-03-24T00:17:07.405 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:17:07.405+0000 7f6673be0740 20 client.25780 trim_cache size 0 max 16384 2024-03-24T00:17:07.405 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stdout:{ 2024-03-24T00:17:07.405 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stdout: "message": "", 2024-03-24T00:17:07.405 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stdout: "return_code": 0 2024-03-24T00:17:07.406 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stdout:} 2024-03-24T00:17:07.406 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:17:07.405+0000 7f666b200640 1 -- 172.21.15.118:0/44379829 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7f66500a28f0 msgr2=0x7f66500a4db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:17:07.406 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:17:07.405+0000 7f666b200640 1 --2- 172.21.15.118:0/44379829 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7f66500a28f0 0x7f66500a4db0 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 crypto rx=0x560155d42e50 tx=0x7f6664004a40 comp rx=0 tx=0).stop 2024-03-24T00:17:07.406 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:17:07.405+0000 7f666b200640 1 -- 172.21.15.118:0/44379829 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f666c1048a0 msgr2=0x7f666c199fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:17:07.406 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:17:07.405+0000 7f666b200640 1 --2- 172.21.15.118:0/44379829 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f666c1048a0 0x7f666c199fc0 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7f6658009230 tx=0x7f665802c6c0 comp rx=0 tx=0).stop 2024-03-24T00:17:07.407 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:17:07.405+0000 7f666b200640 1 -- 172.21.15.118:0/44379829 shutdown_connections 2024-03-24T00:17:07.407 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:17:07.405+0000 7f666b200640 1 --2- 172.21.15.118:0/44379829 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7f66500a28f0 0x7f66500a4db0 unknown :-1 s=CLOSED pgs=294 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:17:07.407 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:17:07.405+0000 7f666b200640 1 --2- 172.21.15.118:0/44379829 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f666c1071d0 0x7f666c1a1030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:17:07.407 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:17:07.406+0000 7f666b200640 1 --2- 172.21.15.118:0/44379829 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f666c1048a0 0x7f666c199fc0 unknown :-1 s=CLOSED pgs=271 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:17:07.407 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:17:07.406+0000 7f666b200640 1 --2- 172.21.15.118:0/44379829 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f666c101f70 0x7f666c199a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:17:07.407 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:17:07.406+0000 7f666b200640 1 -- 172.21.15.118:0/44379829 >> 172.21.15.118:0/44379829 conn(0x7f666c0fb980 msgr2=0x7f666c101760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:17:07.407 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:17:07.406+0000 7f666b200640 1 -- 172.21.15.118:0/44379829 shutdown_connections 2024-03-24T00:17:07.407 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:17:07.406+0000 7f666b200640 1 -- 172.21.15.118:0/44379829 wait complete. 2024-03-24T00:17:08.195 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:17:07 smithi118 ceph-mon[31535]: mgrmap e20: x(active, since 12m), standbys: z, y 2024-03-24T00:17:08.203 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:17:07 smithi189 ceph-mon[36187]: mgrmap e20: x(active, since 12m), standbys: z, y 2024-03-24T00:17:08.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:17:07 smithi193 ceph-mon[36878]: mgrmap e20: x(active, since 12m), standbys: z, y 2024-03-24T00:17:09.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:17:08 smithi118 ceph-mon[31535]: pgmap v441: 97 pgs: 97 active+clean; 674 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 511 B/s wr, 11 op/s 2024-03-24T00:17:09.203 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:17:08 smithi189 ceph-mon[36187]: pgmap v441: 97 pgs: 97 active+clean; 674 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 511 B/s wr, 11 op/s 2024-03-24T00:17:09.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:17:08 smithi193 ceph-mon[36878]: pgmap v441: 97 pgs: 97 active+clean; 674 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 KiB/s rd, 511 B/s wr, 11 op/s 2024-03-24T00:17:11.195 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:17:10 smithi118 ceph-mon[31535]: pgmap v442: 97 pgs: 97 active+clean; 675 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 1023 B/s wr, 13 op/s 2024-03-24T00:17:11.203 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:17:10 smithi189 ceph-mon[36187]: pgmap v442: 97 pgs: 97 active+clean; 675 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 1023 B/s wr, 13 op/s 2024-03-24T00:17:11.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:17:10 smithi193 ceph-mon[36878]: pgmap v442: 97 pgs: 97 active+clean; 675 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 6.3 KiB/s rd, 1023 B/s wr, 13 op/s 2024-03-24T00:17:13.195 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:17:12 smithi118 ceph-mon[31535]: pgmap v443: 97 pgs: 97 active+clean; 675 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 853 B/s wr, 10 op/s 2024-03-24T00:17:13.203 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:17:12 smithi189 ceph-mon[36187]: pgmap v443: 97 pgs: 97 active+clean; 675 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 853 B/s wr, 10 op/s 2024-03-24T00:17:13.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:17:12 smithi193 ceph-mon[36878]: pgmap v443: 97 pgs: 97 active+clean; 675 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 853 B/s wr, 10 op/s 2024-03-24T00:17:15.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:17:14 smithi118 ceph-mon[31535]: pgmap v444: 97 pgs: 97 active+clean; 675 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 5.3 KiB/s rd, 853 B/s wr, 11 op/s 2024-03-24T00:17:15.203 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:17:14 smithi189 ceph-mon[36187]: pgmap v444: 97 pgs: 97 active+clean; 675 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 5.3 KiB/s rd, 853 B/s wr, 11 op/s 2024-03-24T00:17:15.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:17:14 smithi193 ceph-mon[36878]: pgmap v444: 97 pgs: 97 active+clean; 675 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 5.3 KiB/s rd, 853 B/s wr, 11 op/s 2024-03-24T00:17:17.024 INFO:tasks.workunit.client.0.smithi118.stderr:Updating files: 57% (7057/12271) Updating files: 58% (7118/12271) Updating files: 59% (7240/12271) Updating files: 60% (7363/12271) Updating files: 61% (7486/12271) Updating files: 62% (7609/12271) Updating files: 63% (7731/12271) Updating files: 64% (7854/12271) Updating files: 65% (7977/12271) Updating files: 66% (8099/12271) Updating files: 67% (8222/12271) Updating files: 68% (8345/12271) Updating files: 69% (8467/12271) Updating files: 70% (8590/12271) Updating files: 71% (8713/12271) Updating files: 72% (8836/12271) Updating files: 73% (8958/12271) Updating files: 74% (9081/12271) Updating files: 75% (9204/12271) Updating files: 76% (9326/12271) Updating files: 77% (9449/12271) Updating files: 78% (9572/12271) Updating files: 79% (9695/12271) Updating files: 80% (9817/12271) Updating files: 81% (9940/12271) Updating files: 82% (10063/12271) Updating files: 83% (10185/12271) Updating files: 84% (10308/12271) Updating files: 85% (10431/12271) Updating files: 86% (10554/12271) Updating files: 87% (10676/12271) Updating files: 88% (10799/12271) Updating files: 89% (10922/12271) Updating files: 90% (11044/12271) Updating files: 91% (11167/12271) Updating files: 92% (11290/12271) Updating files: 93% (11413/12271) Updating files: 94% (11535/12271) Updating files: 95% (11658/12271) Updating files: 96% (11781/12271) Updating files: 97% (11903/12271) Updating files: 97% (11983/12271) Updating files: 98% (12026/12271) Updating files: 99% (12149/12271) Updating files: 100% (12271/12271) Updating files: 100% (12271/12271), done. 2024-03-24T00:17:17.195 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:17:16 smithi118 ceph-mon[31535]: pgmap v445: 97 pgs: 97 active+clean; 675 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 5.7 KiB/s rd, 853 B/s wr, 12 op/s 2024-03-24T00:17:17.203 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:17:16 smithi189 ceph-mon[36187]: pgmap v445: 97 pgs: 97 active+clean; 675 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 5.7 KiB/s rd, 853 B/s wr, 12 op/s 2024-03-24T00:17:17.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:17:16 smithi193 ceph-mon[36878]: pgmap v445: 97 pgs: 97 active+clean; 675 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 5.7 KiB/s rd, 853 B/s wr, 12 op/s 2024-03-24T00:17:17.366 INFO:tasks.workunit.client.0.smithi118.stderr:Note: switching to '3be9a5c9bc793e11e2800a8c0c696e8b46742033'. 2024-03-24T00:17:17.367 INFO:tasks.workunit.client.0.smithi118.stderr: 2024-03-24T00:17:17.367 INFO:tasks.workunit.client.0.smithi118.stderr:You are in 'detached HEAD' state. You can look around, make experimental 2024-03-24T00:17:17.367 INFO:tasks.workunit.client.0.smithi118.stderr:changes and commit them, and you can discard any commits you make in this 2024-03-24T00:17:17.367 INFO:tasks.workunit.client.0.smithi118.stderr:state without impacting any branches by switching back to a branch. 2024-03-24T00:17:17.367 INFO:tasks.workunit.client.0.smithi118.stderr: 2024-03-24T00:17:17.367 INFO:tasks.workunit.client.0.smithi118.stderr:If you want to create a new branch to retain commits you create, you may 2024-03-24T00:17:17.367 INFO:tasks.workunit.client.0.smithi118.stderr:do so (now or later) by using -c with the switch command. Example: 2024-03-24T00:17:17.367 INFO:tasks.workunit.client.0.smithi118.stderr: 2024-03-24T00:17:17.367 INFO:tasks.workunit.client.0.smithi118.stderr: git switch -c 2024-03-24T00:17:17.367 INFO:tasks.workunit.client.0.smithi118.stderr: 2024-03-24T00:17:17.367 INFO:tasks.workunit.client.0.smithi118.stderr:Or undo this operation with: 2024-03-24T00:17:17.367 INFO:tasks.workunit.client.0.smithi118.stderr: 2024-03-24T00:17:17.367 INFO:tasks.workunit.client.0.smithi118.stderr: git switch - 2024-03-24T00:17:17.367 INFO:tasks.workunit.client.0.smithi118.stderr: 2024-03-24T00:17:17.368 INFO:tasks.workunit.client.0.smithi118.stderr:Turn off this advice by setting config variable advice.detachedHead to false 2024-03-24T00:17:17.368 INFO:tasks.workunit.client.0.smithi118.stderr: 2024-03-24T00:17:17.368 INFO:tasks.workunit.client.0.smithi118.stderr:HEAD is now at 3be9a5c9bc7 dummy commit #1 2024-03-24T00:17:17.375 DEBUG:teuthology.orchestra.run.smithi118:> cd -- /home/ubuntu/cephtest/clone.client.0/qa/workunits && if test -e Makefile ; then make ; fi && find -executable -type f -printf '%P\0' >/home/ubuntu/cephtest/workunits.list.client.0 2024-03-24T00:17:17.430 INFO:tasks.workunit.client.0.smithi118.stdout:for d in direct_io fs ; do ( cd $d ; make all ) ; done 2024-03-24T00:17:17.438 INFO:tasks.workunit.client.0.smithi118.stdout:make[1]: Entering directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/direct_io' 2024-03-24T00:17:17.438 INFO:tasks.workunit.client.0.smithi118.stdout:cc -Wall -Wextra -D_GNU_SOURCE direct_io_test.c -o direct_io_test 2024-03-24T00:17:18.374 INFO:tasks.workunit.client.0.smithi118.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_sync_io.c -o test_sync_io 2024-03-24T00:17:18.513 INFO:tasks.workunit.client.0.smithi118.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_short_dio_read.c -o test_short_dio_read 2024-03-24T00:17:18.601 INFO:tasks.workunit.client.0.smithi118.stdout:make[1]: Leaving directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/direct_io' 2024-03-24T00:17:18.607 INFO:tasks.workunit.client.0.smithi118.stdout:make[1]: Entering directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/fs' 2024-03-24T00:17:18.607 INFO:tasks.workunit.client.0.smithi118.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_o_trunc.c -o test_o_trunc 2024-03-24T00:17:18.681 INFO:tasks.workunit.client.0.smithi118.stdout:make[1]: Leaving directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/fs' 2024-03-24T00:17:18.691 DEBUG:teuthology.orchestra.run.smithi118:> set -ex 2024-03-24T00:17:18.691 DEBUG:teuthology.orchestra.run.smithi118:> dd if=/home/ubuntu/cephtest/workunits.list.client.0 of=/dev/stdout 2024-03-24T00:17:18.742 INFO:tasks.workunit:Running workunits matching suites/blogbench.sh on client.0... 2024-03-24T00:17:18.743 INFO:tasks.workunit:Running workunit suites/blogbench.sh... 2024-03-24T00:17:18.743 DEBUG:teuthology.orchestra.run.smithi118:workunit test suites/blogbench.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=3be9a5c9bc793e11e2800a8c0c696e8b46742033 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/suites/blogbench.sh 2024-03-24T00:17:18.812 INFO:tasks.workunit.client.0.smithi118.stdout:getting blogbench 2024-03-24T00:17:18.812 INFO:tasks.workunit.client.0.smithi118.stderr:+ echo 'getting blogbench' 2024-03-24T00:17:18.812 INFO:tasks.workunit.client.0.smithi118.stderr:+ wget http://download.ceph.com/qa/blogbench-1.0.tar.bz2 2024-03-24T00:17:18.872 INFO:tasks.workunit.client.0.smithi118.stderr:--2024-03-24 00:17:18-- http://download.ceph.com/qa/blogbench-1.0.tar.bz2 2024-03-24T00:17:18.873 INFO:tasks.workunit.client.0.smithi118.stderr:Resolving download.ceph.com (download.ceph.com)... 158.69.68.124, 2607:5300:201:2000::3:58a1 2024-03-24T00:17:18.925 INFO:tasks.workunit.client.0.smithi118.stderr:Connecting to download.ceph.com (download.ceph.com)|158.69.68.124|:80... connected. 2024-03-24T00:17:18.977 INFO:tasks.workunit.client.0.smithi118.stderr:HTTP request sent, awaiting response... 200 OK 2024-03-24T00:17:18.977 INFO:tasks.workunit.client.0.smithi118.stderr:Length: 103355 (101K) [application/octet-stream] 2024-03-24T00:17:18.980 INFO:tasks.workunit.client.0.smithi118.stderr:Saving to: ‘blogbench-1.0.tar.bz2’ 2024-03-24T00:17:18.986 INFO:tasks.workunit.client.0.smithi118.stderr: 2024-03-24T00:17:19.088 INFO:tasks.workunit.client.0.smithi118.stderr: 0K .......... .......... .......... .......... .......... 49% 463K 0s 2024-03-24T00:17:19.147 INFO:tasks.workunit.client.0.smithi118.stderr: 50K .......... .......... .......... .......... .......... 99% 855K 0s 2024-03-24T00:17:19.147 INFO:tasks.workunit.client.0.smithi118.stderr: 100K 100% 1.74T=0.2s 2024-03-24T00:17:19.147 INFO:tasks.workunit.client.0.smithi118.stderr: 2024-03-24T00:17:19.147 INFO:tasks.workunit.client.0.smithi118.stderr:2024-03-24 00:17:19 (606 KB/s) - ‘blogbench-1.0.tar.bz2’ saved [103355/103355] 2024-03-24T00:17:19.147 INFO:tasks.workunit.client.0.smithi118.stderr: 2024-03-24T00:17:19.149 INFO:tasks.workunit.client.0.smithi118.stderr:+ tar -xvf blogbench-1.0.tar.bz2 2024-03-24T00:17:19.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:17:18 smithi118 ceph-mon[31535]: pgmap v446: 97 pgs: 97 active+clean; 675 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 5.3 KiB/s rd, 853 B/s wr, 11 op/s 2024-03-24T00:17:19.203 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:17:18 smithi189 ceph-mon[36187]: pgmap v446: 97 pgs: 97 active+clean; 675 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 5.3 KiB/s rd, 853 B/s wr, 11 op/s 2024-03-24T00:17:19.209 INFO:tasks.workunit.client.0.smithi118.stdout:blogbench-1.0/ 2024-03-24T00:17:19.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:17:18 smithi193 ceph-mon[36878]: pgmap v446: 97 pgs: 97 active+clean; 675 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 5.3 KiB/s rd, 853 B/s wr, 11 op/s 2024-03-24T00:17:19.214 INFO:tasks.workunit.client.0.smithi118.stdout:blogbench-1.0/m4/ 2024-03-24T00:17:19.216 INFO:tasks.workunit.client.0.smithi118.stdout:blogbench-1.0/m4/acx_pthread.m4 2024-03-24T00:17:19.218 INFO:tasks.workunit.client.0.smithi118.stdout:blogbench-1.0/man/ 2024-03-24T00:17:19.226 INFO:tasks.workunit.client.0.smithi118.stdout:blogbench-1.0/man/blogbench.8 2024-03-24T00:17:19.228 INFO:tasks.workunit.client.0.smithi118.stdout:blogbench-1.0/man/Makefile.am 2024-03-24T00:17:19.233 INFO:tasks.workunit.client.0.smithi118.stdout:blogbench-1.0/man/Makefile.in 2024-03-24T00:17:19.238 INFO:tasks.workunit.client.0.smithi118.stdout:blogbench-1.0/src/ 2024-03-24T00:17:19.244 INFO:tasks.workunit.client.0.smithi118.stdout:blogbench-1.0/src/mysnprintf.c 2024-03-24T00:17:19.246 INFO:tasks.workunit.client.0.smithi118.stdout:blogbench-1.0/src/mysnprintf.h 2024-03-24T00:17:19.251 INFO:tasks.workunit.client.0.smithi118.stdout:blogbench-1.0/src/blogbench.c 2024-03-24T00:17:19.254 INFO:tasks.workunit.client.0.smithi118.stdout:blogbench-1.0/src/blogbench.h 2024-03-24T00:17:19.258 INFO:tasks.workunit.client.0.smithi118.stdout:blogbench-1.0/src/fakesnprintf.c 2024-03-24T00:17:19.261 INFO:tasks.workunit.client.0.smithi118.stdout:blogbench-1.0/src/fakesnprintf.h 2024-03-24T00:17:19.264 INFO:tasks.workunit.client.0.smithi118.stdout:blogbench-1.0/src/blogbench_p.h 2024-03-24T00:17:19.268 INFO:tasks.workunit.client.0.smithi118.stdout:blogbench-1.0/src/bsd-getopt_long.c 2024-03-24T00:17:19.272 INFO:tasks.workunit.client.0.smithi118.stdout:blogbench-1.0/src/bsd-getopt_long.h 2024-03-24T00:17:19.275 INFO:tasks.workunit.client.0.smithi118.stdout:blogbench-1.0/src/Makefile.am 2024-03-24T00:17:19.278 INFO:tasks.workunit.client.0.smithi118.stdout:blogbench-1.0/src/Makefile.in 2024-03-24T00:17:19.281 INFO:tasks.workunit.client.0.smithi118.stdout:blogbench-1.0/src/process.c 2024-03-24T00:17:19.284 INFO:tasks.workunit.client.0.smithi118.stdout:blogbench-1.0/src/globals.h 2024-03-24T00:17:19.288 INFO:tasks.workunit.client.0.smithi118.stdout:blogbench-1.0/src/commenter.c 2024-03-24T00:17:19.291 INFO:tasks.workunit.client.0.smithi118.stdout:blogbench-1.0/src/rewriter.c 2024-03-24T00:17:19.294 INFO:tasks.workunit.client.0.smithi118.stdout:blogbench-1.0/src/reader.c 2024-03-24T00:17:19.298 INFO:tasks.workunit.client.0.smithi118.stdout:blogbench-1.0/src/writer.c 2024-03-24T00:17:19.301 INFO:tasks.workunit.client.0.smithi118.stdout:blogbench-1.0/src/helpers.c 2024-03-24T00:17:19.304 INFO:tasks.workunit.client.0.smithi118.stdout:blogbench-1.0/NEWS 2024-03-24T00:17:19.309 INFO:tasks.workunit.client.0.smithi118.stdout:blogbench-1.0/compile 2024-03-24T00:17:19.312 INFO:tasks.workunit.client.0.smithi118.stdout:blogbench-1.0/depcomp 2024-03-24T00:17:19.315 INFO:tasks.workunit.client.0.smithi118.stdout:blogbench-1.0/aclocal.m4 2024-03-24T00:17:19.319 INFO:tasks.workunit.client.0.smithi118.stdout:blogbench-1.0/README 2024-03-24T00:17:19.323 INFO:tasks.workunit.client.0.smithi118.stdout:blogbench-1.0/configure 2024-03-24T00:17:19.334 INFO:tasks.workunit.client.0.smithi118.stdout:blogbench-1.0/configure.ac 2024-03-24T00:17:19.338 INFO:tasks.workunit.client.0.smithi118.stdout:blogbench-1.0/config.guess 2024-03-24T00:17:19.342 INFO:tasks.workunit.client.0.smithi118.stdout:blogbench-1.0/install-sh 2024-03-24T00:17:19.346 INFO:tasks.workunit.client.0.smithi118.stdout:blogbench-1.0/config.sub 2024-03-24T00:17:19.350 INFO:tasks.workunit.client.0.smithi118.stdout:blogbench-1.0/missing 2024-03-24T00:17:19.354 INFO:tasks.workunit.client.0.smithi118.stdout:blogbench-1.0/mkinstalldirs 2024-03-24T00:17:19.358 INFO:tasks.workunit.client.0.smithi118.stdout:blogbench-1.0/Makefile.am 2024-03-24T00:17:19.362 INFO:tasks.workunit.client.0.smithi118.stdout:blogbench-1.0/Makefile.in 2024-03-24T00:17:19.365 INFO:tasks.workunit.client.0.smithi118.stdout:blogbench-1.0/config.h.in 2024-03-24T00:17:19.369 INFO:tasks.workunit.client.0.smithi118.stdout:blogbench-1.0/AUTHORS 2024-03-24T00:17:19.372 INFO:tasks.workunit.client.0.smithi118.stdout:blogbench-1.0/INSTALL 2024-03-24T00:17:19.375 INFO:tasks.workunit.client.0.smithi118.stdout:blogbench-1.0/ChangeLog 2024-03-24T00:17:19.379 INFO:tasks.workunit.client.0.smithi118.stdout:blogbench-1.0/COPYING 2024-03-24T00:17:19.386 INFO:tasks.workunit.client.0.smithi118.stderr:+ cd blogbench-1.0/ 2024-03-24T00:17:19.387 INFO:tasks.workunit.client.0.smithi118.stdout:making blogbench 2024-03-24T00:17:19.387 INFO:tasks.workunit.client.0.smithi118.stderr:+ echo 'making blogbench' 2024-03-24T00:17:19.387 INFO:tasks.workunit.client.0.smithi118.stderr:+ ./configure 2024-03-24T00:17:19.730 INFO:tasks.workunit.client.0.smithi118.stdout:checking for a BSD-compatible install... /usr/bin/install -c 2024-03-24T00:17:20.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:17:19 smithi118 ceph-mon[31535]: pgmap v447: 97 pgs: 97 active+clean; 675 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 5.7 KiB/s rd, 597 B/s wr, 11 op/s 2024-03-24T00:17:20.203 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:17:19 smithi189 ceph-mon[36187]: pgmap v447: 97 pgs: 97 active+clean; 675 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 5.7 KiB/s rd, 597 B/s wr, 11 op/s 2024-03-24T00:17:20.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:17:19 smithi193 ceph-mon[36878]: pgmap v447: 97 pgs: 97 active+clean; 675 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 5.7 KiB/s rd, 597 B/s wr, 11 op/s 2024-03-24T00:17:20.548 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:17:20.548+0000 7ffac8400640 1 -- 172.21.15.118:0/550535038 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ffac01008e0 msgr2=0x7ffac0100d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:17:20.548 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:17:20.548+0000 7ffac8400640 1 --2- 172.21.15.118:0/550535038 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ffac01008e0 0x7ffac0100d40 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7ffaac009900 tx=0x7ffaac02bc50 comp rx=0 tx=0).stop 2024-03-24T00:17:20.548 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:17:20.548+0000 7ffac8400640 1 -- 172.21.15.118:0/550535038 shutdown_connections 2024-03-24T00:17:20.549 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:17:20.548+0000 7ffac8400640 1 --2- 172.21.15.118:0/550535038 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7ffac0101280 0x7ffac0101700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:17:20.549 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:17:20.548+0000 7ffac8400640 1 --2- 172.21.15.118:0/550535038 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ffac01008e0 0x7ffac0100d40 unknown :-1 s=CLOSED pgs=274 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:17:20.549 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:17:20.548+0000 7ffac8400640 1 --2- 172.21.15.118:0/550535038 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7ffac00fff80 0x7ffac01003a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:17:20.549 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:17:20.548+0000 7ffac8400640 1 -- 172.21.15.118:0/550535038 >> 172.21.15.118:0/550535038 conn(0x7ffac00fbb80 msgr2=0x7ffac00fdfc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:17:20.549 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:17:20.548+0000 7ffac8400640 1 -- 172.21.15.118:0/550535038 shutdown_connections 2024-03-24T00:17:20.549 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:17:20.548+0000 7ffac8400640 1 -- 172.21.15.118:0/550535038 wait complete. 2024-03-24T00:17:20.549 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:17:20.550+0000 7ffac8400640 1 Processor -- start 2024-03-24T00:17:20.549 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:17:20.550+0000 7ffac8400640 1 -- start start 2024-03-24T00:17:20.551 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:17:20.550+0000 7ffac8400640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7ffac00fff80 0x7ffac0199af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:17:20.551 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:17:20.551+0000 7ffac8400640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ffac01008e0 0x7ffac019a030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:17:20.551 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:17:20.551+0000 7ffac8400640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7ffac0101280 0x7ffac01a10a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:17:20.551 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:17:20.551+0000 7ffac8400640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffac010f5a0 con 0x7ffac00fff80 2024-03-24T00:17:20.551 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:17:20.551+0000 7ffac8400640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffac010f420 con 0x7ffac01008e0 2024-03-24T00:17:20.551 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:17:20.551+0000 7ffac8400640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffac010f720 con 0x7ffac0101280 2024-03-24T00:17:20.551 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:17:20.551+0000 7ffac5a00640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7ffac00fff80 0x7ffac0199af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:17:20.551 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:17:20.551+0000 7ffac5000640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ffac01008e0 0x7ffac019a030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:17:20.551 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:17:20.551+0000 7ffac5a00640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7ffac00fff80 0x7ffac0199af0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.118:3300/0 says I am v2:172.21.15.118:37594/0 (socket says 172.21.15.118:37594) 2024-03-24T00:17:20.551 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:17:20.551+0000 7ffac5a00640 1 -- 172.21.15.118:0/2028890322 learned_addr learned my addr 172.21.15.118:0/2028890322 (peer_addr_for_me v2:172.21.15.118:0/0) 2024-03-24T00:17:20.551 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:17:20.551+0000 7ffac6400640 1 --2- 172.21.15.118:0/2028890322 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7ffac0101280 0x7ffac01a10a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:17:20.551 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:17:20.551+0000 7ffac5a00640 1 -- 172.21.15.118:0/2028890322 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7ffac0101280 msgr2=0x7ffac01a10a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:17:20.552 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:17:20.551+0000 7ffac5a00640 1 --2- 172.21.15.118:0/2028890322 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7ffac0101280 0x7ffac01a10a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:17:20.552 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:17:20.551+0000 7ffac5a00640 1 -- 172.21.15.118:0/2028890322 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ffac01008e0 msgr2=0x7ffac019a030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:17:20.552 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:17:20.551+0000 7ffac5a00640 1 --2- 172.21.15.118:0/2028890322 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ffac01008e0 0x7ffac019a030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:17:20.552 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:17:20.551+0000 7ffac5a00640 1 -- 172.21.15.118:0/2028890322 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffac01a1710 con 0x7ffac00fff80 2024-03-24T00:17:20.552 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:17:20.552+0000 7ffac5a00640 1 --2- 172.21.15.118:0/2028890322 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7ffac00fff80 0x7ffac0199af0 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7ffab400eab0 tx=0x7ffab400ef70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:17:20.553 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:17:20.552+0000 7ffabe000640 1 -- 172.21.15.118:0/2028890322 <== mon.0 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffab400ce40 con 0x7ffac00fff80 2024-03-24T00:17:20.553 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:17:20.552+0000 7ffabe000640 1 -- 172.21.15.118:0/2028890322 <== mon.0 v2:172.21.15.118:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7ffab4011950 con 0x7ffac00fff80 2024-03-24T00:17:20.553 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:17:20.552+0000 7ffac8400640 1 -- 172.21.15.118:0/2028890322 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ffac01a1a00 con 0x7ffac00fff80 2024-03-24T00:17:20.553 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:17:20.552+0000 7ffabe000640 1 -- 172.21.15.118:0/2028890322 <== mon.0 v2:172.21.15.118:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffab4010ae0 con 0x7ffac00fff80 2024-03-24T00:17:20.553 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:17:20.552+0000 7ffac8400640 1 -- 172.21.15.118:0/2028890322 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ffac01a1f20 con 0x7ffac00fff80 2024-03-24T00:17:20.556 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:17:20.556+0000 7ffacb1ca740 1 Processor -- start 2024-03-24T00:17:20.557 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:17:20.557+0000 7ffacb1ca740 1 -- start start 2024-03-24T00:17:20.557 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:17:20.557+0000 7ffabe000640 1 -- 172.21.15.118:0/2028890322 <== mon.0 v2:172.21.15.118:3300/0 4 ==== mgrmap(e 20) v1 ==== 137291+0+0 (secure 0 0 0) 0x7ffab4011b00 con 0x7ffac00fff80 2024-03-24T00:17:20.557 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:17:20.557+0000 7ffacb1ca740 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55955b8c11e0 0x55955b8c1600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:17:20.557 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:17:20.557+0000 7ffacb1ca740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55955b8c1bd0 0x55955b8c66d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:17:20.557 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:17:20.557+0000 7ffacb1ca740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55955b8c6d40 0x55955b8c91d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:17:20.557 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:17:20.557+0000 7ffacb1ca740 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x55955b8843b0 con 0x55955b8c11e0 2024-03-24T00:17:20.557 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:17:20.557+0000 7ffacb1ca740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x55955b826830 con 0x55955b8c6d40 2024-03-24T00:17:20.558 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:17:20.557+0000 7ffacb1ca740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x55955b7cf5d0 con 0x55955b8c1bd0 2024-03-24T00:17:20.558 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:17:20.557+0000 7ffac5a00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55955b8c1bd0 0x55955b8c66d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:17:20.558 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:17:20.557+0000 7ffac6400640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55955b8c11e0 0x55955b8c1600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:17:20.558 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:17:20.557+0000 7ffac5a00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55955b8c1bd0 0x55955b8c66d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.118:42436/0 (socket says 172.21.15.118:42436) 2024-03-24T00:17:20.558 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:17:20.557+0000 7ffac5000640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55955b8c6d40 0x55955b8c91d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:17:20.558 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:17:20.557+0000 7ffac5a00640 1 -- 172.21.15.118:0/3257233280 learned_addr learned my addr 172.21.15.118:0/3257233280 (peer_addr_for_me v2:172.21.15.118:0/0) 2024-03-24T00:17:20.558 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:17:20.558+0000 7ffac5a00640 1 -- 172.21.15.118:0/3257233280 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55955b8c6d40 msgr2=0x55955b8c91d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:17:20.558 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:17:20.558+0000 7ffac5a00640 1 --2- 172.21.15.118:0/3257233280 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55955b8c6d40 0x55955b8c91d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:17:20.558 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:17:20.558+0000 7ffac5a00640 1 -- 172.21.15.118:0/3257233280 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55955b8c11e0 msgr2=0x55955b8c1600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:17:20.558 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:17:20.558+0000 7ffac5a00640 1 --2- 172.21.15.118:0/3257233280 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55955b8c11e0 0x55955b8c1600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:17:20.558 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:17:20.558+0000 7ffac5a00640 1 -- 172.21.15.118:0/3257233280 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55955b7e72c0 con 0x55955b8c1bd0 2024-03-24T00:17:20.561 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:17:20.558+0000 7ffac5a00640 1 --2- 172.21.15.118:0/3257233280 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55955b8c1bd0 0x55955b8c66d0 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7ffab40d72a0 tx=0x7ffab40d7390 comp rx=0 tx=0).ready entity=mon.1 client_cookie=dd8ba51affddfe8b server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:17:20.561 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:17:20.558+0000 7ffabe000640 1 --2- 172.21.15.118:0/2028890322 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7ffaa80a28a0 0x7ffaa80a4d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:17:20.561 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:17:20.558+0000 7ffab2a00640 1 -- 172.21.15.118:0/3257233280 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffab40e2070 con 0x55955b8c1bd0 2024-03-24T00:17:20.561 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:17:20.558+0000 7ffab2a00640 1 -- 172.21.15.118:0/3257233280 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7ffab40c0d80 con 0x55955b8c1bd0 2024-03-24T00:17:20.561 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:17:20.558+0000 7ffab2a00640 1 -- 172.21.15.118:0/3257233280 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffab40c3680 con 0x55955b8c1bd0 2024-03-24T00:17:20.561 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:17:20.559+0000 7ffabe000640 1 -- 172.21.15.118:0/2028890322 <== mon.0 v2:172.21.15.118:3300/0 5 ==== osd_map(90..90 src has 1..90) v4 ==== 8277+0+0 (secure 0 0 0) 0x7ffab40cf020 con 0x7ffac00fff80 2024-03-24T00:17:20.561 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:17:20.559+0000 7ffac6400640 1 --2- 172.21.15.118:0/2028890322 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7ffaa80a28a0 0x7ffaa80a4d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:17:20.561 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:17:20.559+0000 7ffacb1ca740 1 -- 172.21.15.118:0/3257233280 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55955b8c1bd0 msgr2=0x55955b8c66d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:17:20.561 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:17:20.559+0000 7ffacb1ca740 1 --2- 172.21.15.118:0/3257233280 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55955b8c1bd0 0x55955b8c66d0 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7ffab40d72a0 tx=0x7ffab40d7390 comp rx=0 tx=0).stop 2024-03-24T00:17:20.561 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:17:20.559+0000 7ffac6400640 1 --2- 172.21.15.118:0/2028890322 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7ffaa80a28a0 0x7ffaa80a4d60 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 crypto rx=0x55955b8c2430 tx=0x7ffab8005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:17:20.562 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:17:20.559+0000 7ffacb1ca740 1 -- 172.21.15.118:0/3257233280 shutdown_connections 2024-03-24T00:17:20.562 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:17:20.559+0000 7ffacb1ca740 1 --2- 172.21.15.118:0/3257233280 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55955b8c6d40 0x55955b8c91d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:17:20.562 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:17:20.559+0000 7ffacb1ca740 1 --2- 172.21.15.118:0/3257233280 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55955b8c1bd0 0x55955b8c66d0 unknown :-1 s=CLOSED pgs=263 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:17:20.562 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:17:20.559+0000 7ffacb1ca740 1 --2- 172.21.15.118:0/3257233280 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55955b8c11e0 0x55955b8c1600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:17:20.562 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:17:20.559+0000 7ffacb1ca740 1 -- 172.21.15.118:0/3257233280 >> 172.21.15.118:0/3257233280 conn(0x55955b8bc750 msgr2=0x55955b8bcb50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:17:20.562 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:17:20.560+0000 7ffacb1ca740 1 -- 172.21.15.118:0/3257233280 shutdown_connections 2024-03-24T00:17:20.562 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:17:20.560+0000 7ffacb1ca740 1 -- 172.21.15.118:0/3257233280 wait complete. 2024-03-24T00:17:20.562 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:17:20.560+0000 7ffacb1ca740 1 -- start start 2024-03-24T00:17:20.562 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:17:20.560+0000 7ffacb1ca740 1 Processor -- start 2024-03-24T00:17:20.562 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:17:20.561+0000 7ffacb1ca740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55955b8c11e0 0x55955b8cb300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:17:20.562 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:17:20.561+0000 7ffacb1ca740 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55955b8c1bd0 0x55955b8cb840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:17:20.562 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:17:20.561+0000 7ffacb1ca740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55955b8c6d40 0x55955b8d2880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:17:20.562 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:17:20.561+0000 7ffacb1ca740 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x55955b7e2a20 con 0x55955b8c1bd0 2024-03-24T00:17:20.562 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:17:20.561+0000 7ffacb1ca740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x55955b8228d0 con 0x55955b8c11e0 2024-03-24T00:17:20.563 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:17:20.561+0000 7ffacb1ca740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x55955b8c9cf0 con 0x55955b8c6d40 2024-03-24T00:17:20.563 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:17:20.561+0000 7ffacb1ca740 10 client.0 _ll_register_callbacks cb 0x55955b81f1c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T00:17:20.563 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:17:20.561+0000 7ffac5a00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55955b8c11e0 0x55955b8cb300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:17:20.563 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:17:20.561+0000 7ffac6400640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55955b8c6d40 0x55955b8d2880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:17:20.563 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:17:20.561+0000 7ffac6400640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55955b8c6d40 0x55955b8d2880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.118:42446/0 (socket says 172.21.15.118:42446) 2024-03-24T00:17:20.563 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:17:20.561+0000 7ffac5a00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55955b8c11e0 0x55955b8cb300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.118:39782/0 (socket says 172.21.15.118:39782) 2024-03-24T00:17:20.563 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:17:20.561+0000 7ffac6400640 1 -- 172.21.15.118:0/3098242641 learned_addr learned my addr 172.21.15.118:0/3098242641 (peer_addr_for_me v2:172.21.15.118:0/0) 2024-03-24T00:17:20.563 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:17:20.561+0000 7ffac5000640 1 --2- 172.21.15.118:0/3098242641 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55955b8c1bd0 0x55955b8cb840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:17:20.563 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:17:20.561+0000 7ffac5a00640 1 -- 172.21.15.118:0/3098242641 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55955b8c6d40 msgr2=0x55955b8d2880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:17:20.563 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:17:20.561+0000 7ffac5a00640 1 --2- 172.21.15.118:0/3098242641 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55955b8c6d40 0x55955b8d2880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:17:20.563 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:17:20.561+0000 7ffac5a00640 1 -- 172.21.15.118:0/3098242641 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55955b8c1bd0 msgr2=0x55955b8cb840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:17:20.563 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:17:20.561+0000 7ffac5a00640 1 --2- 172.21.15.118:0/3098242641 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55955b8c1bd0 0x55955b8cb840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:17:20.563 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:17:20.561+0000 7ffac5a00640 1 -- 172.21.15.118:0/3098242641 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55955b53b210 con 0x55955b8c11e0 2024-03-24T00:17:20.563 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:17:20.561+0000 7ffac6400640 1 --2- 172.21.15.118:0/3098242641 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55955b8c6d40 0x55955b8d2880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T00:17:20.566 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:17:20.562+0000 7ffac5a00640 1 --2- 172.21.15.118:0/3098242641 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55955b8c11e0 0x55955b8cb300 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7ffab40df000 tx=0x7ffab40cf400 comp rx=0 tx=0).ready entity=mon.2 client_cookie=8ef8aa52930bd18b server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:17:20.566 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:17:20.562+0000 7ffaa7400640 10 client.0 ms_handle_connect on v2:172.21.15.189:3300/0 2024-03-24T00:17:20.566 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:17:20.562+0000 7ffaa7400640 1 -- 172.21.15.118:0/3098242641 <== mon.2 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffab40c64f0 con 0x55955b8c11e0 2024-03-24T00:17:20.566 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:17:20.562+0000 7ffaa7400640 1 -- 172.21.15.118:0/3098242641 <== mon.2 v2:172.21.15.189:3300/0 2 ==== osd_map(90..90 src has 1..90) v4 ==== 8277+0+0 (secure 0 0 0) 0x7ffab4066370 con 0x55955b8c11e0 2024-03-24T00:17:20.566 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:17:20.562+0000 7ffacb1ca740 1 -- 172.21.15.118:0/3098242641 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7ffab40c0d80 con 0x55955b8c11e0 2024-03-24T00:17:20.566 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:17:20.562+0000 7ffaa7400640 1 -- 172.21.15.118:0/3098242641 <== mon.2 v2:172.21.15.189:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) v2 ==== 24+0+0 (secure 0 0 0) 0x7ffab40c66e0 con 0x55955b8c11e0 2024-03-24T00:17:20.566 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:17:20.562+0000 7ffacb1ca740 10 client.25805 fetch_fsmap learned FSMap version 28 2024-03-24T00:17:20.566 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:17:20.562+0000 7ffacb1ca740 1 -- 172.21.15.118:0/3098242641 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) v3 -- 0x55955b607620 con 0x55955b8c11e0 2024-03-24T00:17:20.567 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:17:20.562+0000 7ffaa7400640 1 -- 172.21.15.118:0/3098242641 <== mon.2 v2:172.21.15.189:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7ffab40c0d80 con 0x55955b8c11e0 2024-03-24T00:17:20.567 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:17:20.562+0000 7ffaa7400640 1 -- 172.21.15.118:0/3098242641 <== mon.2 v2:172.21.15.189:3300/0 5 ==== fsmap(e 28) v1 ==== 7934+0+0 (secure 0 0 0) 0x7ffab400f040 con 0x55955b8c11e0 2024-03-24T00:17:20.567 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:17:20.563+0000 7ffacb1ca740 10 client.25805 fetch_fsmap finished waiting for FSMap version 28 2024-03-24T00:17:20.567 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:17:20.563+0000 7ffacb1ca740 10 client.25805 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-24T00:17:20.567 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:17:20.563+0000 7ffacb1ca740 20 client.25805 populate_metadata read hostname 'smithi118' 2024-03-24T00:17:20.567 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:17:20.563+0000 7ffacb1ca740 1 --2- 172.21.15.118:0/3098242641 >> [v2:172.21.15.189:6838/4074484168,v1:172.21.15.189:6839/4074484168] conn(0x55955b8d6d70 0x55955b8d9160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:17:20.567 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:17:20.563+0000 7ffacb1ca740 4 client.25805 mds_command: new command op to 24569 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-24T00:17:20.567 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:17:20.563+0000 7ffacb1ca740 1 -- 172.21.15.118:0/3098242641 --> [v2:172.21.15.189:6838/4074484168,v1:172.21.15.189:6839/4074484168] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55955b833d70 con 0x55955b8d6d70 2024-03-24T00:17:20.567 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:17:20.563+0000 7ffac5000640 1 --2- 172.21.15.118:0/3098242641 >> [v2:172.21.15.189:6838/4074484168,v1:172.21.15.189:6839/4074484168] conn(0x55955b8d6d70 0x55955b8d9160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:17:20.567 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:17:20.564+0000 7ffac5000640 1 --2- 172.21.15.118:0/3098242641 >> [v2:172.21.15.189:6838/4074484168,v1:172.21.15.189:6839/4074484168] conn(0x55955b8d6d70 0x55955b8d9160 crc :-1 s=READY pgs=65 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=ead9af7247b7cffc server_cookie=9dea21e8f50a75e0 in_seq=0 out_seq=0 2024-03-24T00:17:20.567 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:17:20.564+0000 7ffaa7400640 10 client.25805 ms_handle_connect on v2:172.21.15.189:6838/4074484168 2024-03-24T00:17:20.567 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:17:20.564+0000 7ffaa7400640 1 -- 172.21.15.118:0/3098242641 <== mds.0 v2:172.21.15.189:6838/4074484168 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x55955b833d70 con 0x55955b8d6d70 2024-03-24T00:17:20.567 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:17:20.564+0000 7ffaa7400640 10 client.25805 handle_command_reply: tid=0 2024-03-24T00:17:20.568 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:17:20.565+0000 7ffacb1ca740 1 client.25805 shutdown 2024-03-24T00:17:20.568 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:17:20.565+0000 7ffacb1ca740 1 -- 172.21.15.118:0/3098242641 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55955b8c11e0 msgr2=0x55955b8cb300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:17:20.568 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:17:20.565+0000 7ffacb1ca740 1 --2- 172.21.15.118:0/3098242641 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55955b8c11e0 0x55955b8cb300 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7ffab40df000 tx=0x7ffab40cf400 comp rx=0 tx=0).stop 2024-03-24T00:17:20.568 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:17:20.565+0000 7ffacb1ca740 1 -- 172.21.15.118:0/3098242641 shutdown_connections 2024-03-24T00:17:20.568 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:17:20.565+0000 7ffacb1ca740 1 --2- 172.21.15.118:0/3098242641 >> [v2:172.21.15.189:6838/4074484168,v1:172.21.15.189:6839/4074484168] conn(0x55955b8d6d70 0x55955b8d9160 crc :-1 s=READY pgs=65 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:17:20.568 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:17:20.565+0000 7ffacb1ca740 1 --2- 172.21.15.118:0/3098242641 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55955b8c6d40 0x55955b8d2880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:17:20.568 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:17:20.565+0000 7ffacb1ca740 1 --2- 172.21.15.118:0/3098242641 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55955b8c1bd0 0x55955b8cb840 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:17:20.568 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:17:20.565+0000 7ffacb1ca740 1 --2- 172.21.15.118:0/3098242641 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55955b8c11e0 0x55955b8cb300 unknown :-1 s=CLOSED pgs=275 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:17:20.568 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:17:20.565+0000 7ffacb1ca740 1 -- 172.21.15.118:0/3098242641 >> 172.21.15.118:0/3098242641 conn(0x55955b8bc750 msgr2=0x55955b8bb950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:17:20.568 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:17:20.565+0000 7ffaa7400640 0 client.25805 ms_handle_reset on v2:172.21.15.189:6838/4074484168 2024-03-24T00:17:20.568 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:17:20.565+0000 7ffacb1ca740 1 -- 172.21.15.118:0/3098242641 shutdown_connections 2024-03-24T00:17:20.568 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:17:20.565+0000 7ffacb1ca740 1 -- 172.21.15.118:0/3098242641 wait complete. 2024-03-24T00:17:20.568 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:17:20.565+0000 7ffacb1ca740 20 client.25805 trim_cache size 0 max 16384 2024-03-24T00:17:20.580 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:17:20.580+0000 7ffacb1ca740 1 Processor -- start 2024-03-24T00:17:20.580 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:17:20.580+0000 7ffacb1ca740 1 -- start start 2024-03-24T00:17:20.580 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:17:20.580+0000 7ffacb1ca740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55955b8c11e0 0x55955b8caf10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:17:20.580 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:17:20.580+0000 7ffacb1ca740 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55955b8c1bd0 0x55955b8d2810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:17:20.580 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:17:20.580+0000 7ffacb1ca740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55955b8c6d40 0x55955b8cb450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:17:20.580 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:17:20.580+0000 7ffacb1ca740 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffac01aa9e0 con 0x55955b8c1bd0 2024-03-24T00:17:20.580 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:17:20.580+0000 7ffacb1ca740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffac01ac360 con 0x55955b8c11e0 2024-03-24T00:17:20.581 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:17:20.580+0000 7ffacb1ca740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffac01ab040 con 0x55955b8c6d40 2024-03-24T00:17:20.581 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:17:20.580+0000 7ffac5000640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55955b8c1bd0 0x55955b8d2810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:17:20.581 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:17:20.581+0000 7ffac5000640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55955b8c1bd0 0x55955b8d2810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.118:3300/0 says I am v2:172.21.15.118:37620/0 (socket says 172.21.15.118:37620) 2024-03-24T00:17:20.581 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:17:20.581+0000 7ffac6400640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55955b8c6d40 0x55955b8cb450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:17:20.581 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:17:20.581+0000 7ffac5000640 1 -- 172.21.15.118:0/1030759979 learned_addr learned my addr 172.21.15.118:0/1030759979 (peer_addr_for_me v2:172.21.15.118:0/0) 2024-03-24T00:17:20.581 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:17:20.581+0000 7ffac5a00640 1 --2- 172.21.15.118:0/1030759979 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55955b8c11e0 0x55955b8caf10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:17:20.584 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:17:20.581+0000 7ffac5000640 1 -- 172.21.15.118:0/1030759979 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55955b8c6d40 msgr2=0x55955b8cb450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:17:20.585 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:17:20.581+0000 7ffac5000640 1 --2- 172.21.15.118:0/1030759979 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55955b8c6d40 0x55955b8cb450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:17:20.585 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:17:20.581+0000 7ffac5000640 1 -- 172.21.15.118:0/1030759979 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55955b8c11e0 msgr2=0x55955b8caf10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:17:20.585 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:17:20.581+0000 7ffac5000640 1 --2- 172.21.15.118:0/1030759979 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55955b8c11e0 0x55955b8caf10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:17:20.585 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:17:20.581+0000 7ffac5000640 1 -- 172.21.15.118:0/1030759979 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55955b8b6920 con 0x55955b8c1bd0 2024-03-24T00:17:20.585 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:17:20.581+0000 7ffac5000640 1 --2- 172.21.15.118:0/1030759979 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55955b8c1bd0 0x55955b8d2810 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7ffaac009900 tx=0x7ffaac000bc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fbb60a54e9ec4fe6 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:17:20.585 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:17:20.582+0000 7ffab2000640 1 -- 172.21.15.118:0/1030759979 <== mon.0 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffaac050070 con 0x55955b8c1bd0 2024-03-24T00:17:20.585 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:17:20.582+0000 7ffab2000640 1 -- 172.21.15.118:0/1030759979 <== mon.0 v2:172.21.15.118:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7ffaac04a0d0 con 0x55955b8c1bd0 2024-03-24T00:17:20.585 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:17:20.582+0000 7ffab2000640 1 -- 172.21.15.118:0/1030759979 <== mon.0 v2:172.21.15.118:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffaac004c90 con 0x55955b8c1bd0 2024-03-24T00:17:20.585 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:17:20.582+0000 7ffacb1ca740 1 -- 172.21.15.118:0/1030759979 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55955b8c1bd0 msgr2=0x55955b8d2810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:17:20.585 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:17:20.582+0000 7ffacb1ca740 1 --2- 172.21.15.118:0/1030759979 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55955b8c1bd0 0x55955b8d2810 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7ffaac009900 tx=0x7ffaac000bc0 comp rx=0 tx=0).stop 2024-03-24T00:17:20.585 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:17:20.582+0000 7ffacb1ca740 1 -- 172.21.15.118:0/1030759979 shutdown_connections 2024-03-24T00:17:20.585 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:17:20.582+0000 7ffacb1ca740 1 --2- 172.21.15.118:0/1030759979 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55955b8c6d40 0x55955b8cb450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:17:20.585 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:17:20.582+0000 7ffacb1ca740 1 --2- 172.21.15.118:0/1030759979 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55955b8c1bd0 0x55955b8d2810 unknown :-1 s=CLOSED pgs=194 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:17:20.585 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:17:20.582+0000 7ffacb1ca740 1 --2- 172.21.15.118:0/1030759979 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55955b8c11e0 0x55955b8caf10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:17:20.586 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:17:20.582+0000 7ffacb1ca740 1 -- 172.21.15.118:0/1030759979 >> 172.21.15.118:0/1030759979 conn(0x55955b8bc750 msgr2=0x55955b8bf9b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:17:20.586 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:17:20.582+0000 7ffacb1ca740 1 -- 172.21.15.118:0/1030759979 shutdown_connections 2024-03-24T00:17:20.586 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:17:20.583+0000 7ffacb1ca740 1 -- 172.21.15.118:0/1030759979 wait complete. 2024-03-24T00:17:20.586 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:17:20.583+0000 7ffacb1ca740 1 -- start start 2024-03-24T00:17:20.586 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:17:20.583+0000 7ffacb1ca740 1 Processor -- start 2024-03-24T00:17:20.586 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:17:20.583+0000 7ffacb1ca740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55955b8c11e0 0x55955b8cffb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:17:20.586 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:17:20.583+0000 7ffacb1ca740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55955b8c1bd0 0x55955b8d04f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:17:20.586 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:17:20.583+0000 7ffacb1ca740 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55955b8c6d40 0x55955b8d1500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:17:20.586 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:17:20.583+0000 7ffacb1ca740 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffac01abd00 con 0x55955b8c6d40 2024-03-24T00:17:20.586 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:17:20.583+0000 7ffacb1ca740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffac01ab6a0 con 0x55955b8c1bd0 2024-03-24T00:17:20.586 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:17:20.583+0000 7ffacb1ca740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffac01a8ba0 con 0x55955b8c11e0 2024-03-24T00:17:20.586 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:17:20.583+0000 7ffacb1ca740 10 client.0 _ll_register_callbacks cb 0x7ffab40e0070 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T00:17:20.586 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:17:20.584+0000 7ffac5a00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55955b8c11e0 0x55955b8cffb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:17:20.587 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:17:20.584+0000 7ffac5000640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55955b8c1bd0 0x55955b8d04f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:17:20.587 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:17:20.584+0000 7ffac5a00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55955b8c11e0 0x55955b8cffb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.118:42466/0 (socket says 172.21.15.118:42466) 2024-03-24T00:17:20.587 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:17:20.584+0000 7ffac5a00640 1 -- 172.21.15.118:0/2754008972 learned_addr learned my addr 172.21.15.118:0/2754008972 (peer_addr_for_me v2:172.21.15.118:0/0) 2024-03-24T00:17:20.587 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:17:20.584+0000 7ffac6400640 1 --2- 172.21.15.118:0/2754008972 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55955b8c6d40 0x55955b8d1500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:17:20.587 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:17:20.584+0000 7ffac5a00640 1 -- 172.21.15.118:0/2754008972 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55955b8c1bd0 msgr2=0x55955b8d04f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:17:20.587 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:17:20.584+0000 7ffac5a00640 1 --2- 172.21.15.118:0/2754008972 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55955b8c1bd0 0x55955b8d04f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:17:20.587 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:17:20.584+0000 7ffac5a00640 1 -- 172.21.15.118:0/2754008972 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55955b8c6d40 msgr2=0x55955b8d1500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:17:20.587 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:17:20.584+0000 7ffac5a00640 1 --2- 172.21.15.118:0/2754008972 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55955b8c6d40 0x55955b8d1500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:17:20.587 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:17:20.584+0000 7ffac5a00640 1 -- 172.21.15.118:0/2754008972 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55955b8d1af0 con 0x55955b8c11e0 2024-03-24T00:17:20.587 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:17:20.584+0000 7ffac6400640 1 --2- 172.21.15.118:0/2754008972 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55955b8c6d40 0x55955b8d1500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T00:17:20.587 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:17:20.584+0000 7ffac5a00640 1 --2- 172.21.15.118:0/2754008972 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55955b8c11e0 0x55955b8cffb0 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7ffab4010580 tx=0x7ffab4011ea0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=1900a1730cccdf8a server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:17:20.587 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:17:20.584+0000 7ffaa7400640 10 client.0 ms_handle_connect on v2:172.21.15.193:3300/0 2024-03-24T00:17:20.587 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:17:20.584+0000 7ffaa7400640 1 -- 172.21.15.118:0/2754008972 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffab40c7110 con 0x55955b8c11e0 2024-03-24T00:17:20.587 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:17:20.584+0000 7ffaa7400640 1 -- 172.21.15.118:0/2754008972 <== mon.1 v2:172.21.15.193:3300/0 2 ==== osd_map(90..90 src has 1..90) v4 ==== 8277+0+0 (secure 0 0 0) 0x7ffab40d29e0 con 0x55955b8c11e0 2024-03-24T00:17:20.588 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:17:20.584+0000 7ffacb1ca740 1 -- 172.21.15.118:0/2754008972 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7ffaac04a0d0 con 0x55955b8c11e0 2024-03-24T00:17:20.588 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:17:20.585+0000 7ffaa7400640 1 -- 172.21.15.118:0/2754008972 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) v2 ==== 24+0+0 (secure 0 0 0) 0x7ffab40c7690 con 0x55955b8c11e0 2024-03-24T00:17:20.588 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:17:20.585+0000 7ffacb1ca740 10 client.25798 fetch_fsmap learned FSMap version 28 2024-03-24T00:17:20.588 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:17:20.585+0000 7ffacb1ca740 1 -- 172.21.15.118:0/2754008972 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) v3 -- 0x55955b9324c0 con 0x55955b8c11e0 2024-03-24T00:17:20.588 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:17:20.585+0000 7ffaa7400640 1 -- 172.21.15.118:0/2754008972 <== mon.1 v2:172.21.15.193:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7ffaac04a0d0 con 0x55955b8c11e0 2024-03-24T00:17:20.588 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:17:20.585+0000 7ffaa7400640 1 -- 172.21.15.118:0/2754008972 <== mon.1 v2:172.21.15.193:3300/0 5 ==== fsmap(e 28) v1 ==== 7934+0+0 (secure 0 0 0) 0x7ffab40c3120 con 0x55955b8c11e0 2024-03-24T00:17:20.588 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:17:20.585+0000 7ffacb1ca740 10 client.25798 fetch_fsmap finished waiting for FSMap version 28 2024-03-24T00:17:20.588 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:17:20.585+0000 7ffacb1ca740 10 client.25798 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-24T00:17:20.588 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:17:20.585+0000 7ffacb1ca740 20 client.25798 populate_metadata read hostname 'smithi118' 2024-03-24T00:17:20.588 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:17:20.585+0000 7ffacb1ca740 1 --2- 172.21.15.118:0/2754008972 >> [v2:172.21.15.189:6838/4074484168,v1:172.21.15.189:6839/4074484168] conn(0x55955b932bd0 0x55955b934ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:17:20.588 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:17:20.585+0000 7ffacb1ca740 4 client.25798 mds_command: new command op to 24569 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-03-24T00:17:20.588 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:17:20.585+0000 7ffacb1ca740 1 -- 172.21.15.118:0/2754008972 --> [v2:172.21.15.189:6838/4074484168,v1:172.21.15.189:6839/4074484168] -- command(tid 0: {"prefix": "flush journal"}) v1 -- 0x55955b82e0c0 con 0x55955b932bd0 2024-03-24T00:17:20.589 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:17:20.585+0000 7ffac5000640 1 --2- 172.21.15.118:0/2754008972 >> [v2:172.21.15.189:6838/4074484168,v1:172.21.15.189:6839/4074484168] conn(0x55955b932bd0 0x55955b934ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:17:20.589 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:17:20.586+0000 7ffac5000640 1 --2- 172.21.15.118:0/2754008972 >> [v2:172.21.15.189:6838/4074484168,v1:172.21.15.189:6839/4074484168] conn(0x55955b932bd0 0x55955b934ff0 crc :-1 s=READY pgs=66 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=ecaa4e858f6cc7fc server_cookie=d1a35adf30bbc8fc in_seq=0 out_seq=0 2024-03-24T00:17:20.589 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:17:20.586+0000 7ffaa7400640 10 client.25798 ms_handle_connect on v2:172.21.15.189:6838/4074484168 2024-03-24T00:17:20.610 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:17:20.610+0000 7ffaa7400640 1 -- 172.21.15.118:0/2754008972 <== mds.0 v2:172.21.15.189:6838/4074484168 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+44 (crc 0 0 0) 0x55955b82e0c0 con 0x55955b932bd0 2024-03-24T00:17:20.610 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:17:20.610+0000 7ffaa7400640 10 client.25798 handle_command_reply: tid=0 2024-03-24T00:17:20.610 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:17:20.610+0000 7ffacb1ca740 1 client.25798 shutdown 2024-03-24T00:17:20.610 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:17:20.610+0000 7ffacb1ca740 1 -- 172.21.15.118:0/2754008972 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55955b8c11e0 msgr2=0x55955b8cffb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:17:20.611 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:17:20.610+0000 7ffacb1ca740 1 --2- 172.21.15.118:0/2754008972 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55955b8c11e0 0x55955b8cffb0 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7ffab4010580 tx=0x7ffab4011ea0 comp rx=0 tx=0).stop 2024-03-24T00:17:20.611 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:17:20.610+0000 7ffacb1ca740 1 -- 172.21.15.118:0/2754008972 shutdown_connections 2024-03-24T00:17:20.611 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:17:20.610+0000 7ffacb1ca740 1 --2- 172.21.15.118:0/2754008972 >> [v2:172.21.15.189:6838/4074484168,v1:172.21.15.189:6839/4074484168] conn(0x55955b932bd0 0x55955b934ff0 crc :-1 s=READY pgs=66 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:17:20.611 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:17:20.610+0000 7ffacb1ca740 1 --2- 172.21.15.118:0/2754008972 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55955b8c6d40 0x55955b8d1500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:17:20.611 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:17:20.610+0000 7ffacb1ca740 1 --2- 172.21.15.118:0/2754008972 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55955b8c1bd0 0x55955b8d04f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:17:20.611 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:17:20.610+0000 7ffacb1ca740 1 --2- 172.21.15.118:0/2754008972 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55955b8c11e0 0x55955b8cffb0 unknown :-1 s=CLOSED pgs=264 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:17:20.611 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:17:20.610+0000 7ffaa7400640 0 client.25798 ms_handle_reset on v2:172.21.15.189:6838/4074484168 2024-03-24T00:17:20.611 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:17:20.610+0000 7ffacb1ca740 1 -- 172.21.15.118:0/2754008972 >> 172.21.15.118:0/2754008972 conn(0x55955b8bc750 msgr2=0x55955b8caef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:17:20.611 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:17:20.610+0000 7ffacb1ca740 1 -- 172.21.15.118:0/2754008972 shutdown_connections 2024-03-24T00:17:20.611 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:17:20.610+0000 7ffacb1ca740 1 -- 172.21.15.118:0/2754008972 wait complete. 2024-03-24T00:17:20.611 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:17:20.611+0000 7ffacb1ca740 20 client.25798 trim_cache size 0 max 16384 2024-03-24T00:17:20.612 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stdout:{ 2024-03-24T00:17:20.612 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stdout: "message": "", 2024-03-24T00:17:20.612 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stdout: "return_code": 0 2024-03-24T00:17:20.612 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stdout:} 2024-03-24T00:17:20.613 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:17:20.611+0000 7ffac8400640 1 -- 172.21.15.118:0/2028890322 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7ffaa80a28a0 msgr2=0x7ffaa80a4d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:17:20.613 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:17:20.611+0000 7ffac8400640 1 --2- 172.21.15.118:0/2028890322 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7ffaa80a28a0 0x7ffaa80a4d60 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 crypto rx=0x55955b8c2430 tx=0x7ffab8005f90 comp rx=0 tx=0).stop 2024-03-24T00:17:20.613 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:17:20.611+0000 7ffac8400640 1 -- 172.21.15.118:0/2028890322 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7ffac00fff80 msgr2=0x7ffac0199af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:17:20.613 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:17:20.611+0000 7ffac8400640 1 --2- 172.21.15.118:0/2028890322 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7ffac00fff80 0x7ffac0199af0 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7ffab400eab0 tx=0x7ffab400ef70 comp rx=0 tx=0).stop 2024-03-24T00:17:20.613 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:17:20.612+0000 7ffac8400640 1 -- 172.21.15.118:0/2028890322 shutdown_connections 2024-03-24T00:17:20.613 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:17:20.612+0000 7ffac8400640 1 --2- 172.21.15.118:0/2028890322 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7ffaa80a28a0 0x7ffaa80a4d60 unknown :-1 s=CLOSED pgs=295 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:17:20.613 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:17:20.612+0000 7ffac8400640 1 --2- 172.21.15.118:0/2028890322 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7ffac0101280 0x7ffac01a10a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:17:20.613 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:17:20.612+0000 7ffac8400640 1 --2- 172.21.15.118:0/2028890322 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ffac01008e0 0x7ffac019a030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:17:20.613 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:17:20.612+0000 7ffac8400640 1 --2- 172.21.15.118:0/2028890322 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7ffac00fff80 0x7ffac0199af0 unknown :-1 s=CLOSED pgs=193 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:17:20.613 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:17:20.612+0000 7ffac8400640 1 -- 172.21.15.118:0/2028890322 >> 172.21.15.118:0/2028890322 conn(0x7ffac00fbb80 msgr2=0x7ffac00fd680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:17:20.613 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:17:20.612+0000 7ffac8400640 1 -- 172.21.15.118:0/2028890322 shutdown_connections 2024-03-24T00:17:20.614 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:17:20.612+0000 7ffac8400640 1 -- 172.21.15.118:0/2028890322 wait complete. 2024-03-24T00:17:20.750 INFO:tasks.workunit.client.0.smithi118.stdout:checking whether build environment is sane... yes 2024-03-24T00:17:20.780 INFO:tasks.workunit.client.0.smithi118.stdout:checking for gawk... gawk 2024-03-24T00:17:20.809 INFO:tasks.workunit.client.0.smithi118.stdout:checking whether make sets $(MAKE)... yes 2024-03-24T00:17:20.858 INFO:tasks.workunit.client.0.smithi118.stdout:checking for gcc... gcc 2024-03-24T00:17:20.983 INFO:tasks.workunit.client.0.smithi118.stdout:checking for C compiler default output file name... a.out 2024-03-24T00:17:20.990 INFO:tasks.workunit.client.0.smithi118.stdout:checking whether the C compiler works... yes 2024-03-24T00:17:21.000 INFO:tasks.workunit.client.0.smithi118.stdout:checking whether we are cross compiling... no 2024-03-24T00:17:21.077 INFO:tasks.workunit.client.0.smithi118.stdout:checking for suffix of executables... 2024-03-24T00:17:21.168 INFO:tasks.workunit.client.0.smithi118.stdout:checking for suffix of object files... o 2024-03-24T00:17:21.277 INFO:tasks.workunit.client.0.smithi118.stdout:checking whether we are using the GNU C compiler... yes 2024-03-24T00:17:21.394 INFO:tasks.workunit.client.0.smithi118.stdout:checking whether gcc accepts -g... yes 2024-03-24T00:17:21.574 INFO:tasks.workunit.client.0.smithi118.stdout:checking for gcc option to accept ANSI C... none needed 2024-03-24T00:17:21.665 INFO:tasks.workunit.client.0.smithi118.stdout:checking for style of include used by make... GNU 2024-03-24T00:17:21.929 INFO:tasks.workunit.client.0.smithi118.stdout:checking dependency style of gcc... gcc3 2024-03-24T00:17:21.929 INFO:tasks.workunit.client.0.smithi118.stdout:checking for a BSD-compatible install... /usr/bin/install -c 2024-03-24T00:17:22.113 INFO:tasks.workunit.client.0.smithi118.stdout:checking how to run the C preprocessor... gcc -E 2024-03-24T00:17:22.273 INFO:tasks.workunit.client.0.smithi118.stdout:checking for egrep... grep -E 2024-03-24T00:17:22.305 INFO:tasks.workunit.client.0.smithi118.stdout:checking for AIX... no 2024-03-24T00:17:22.458 INFO:tasks.workunit.client.0.smithi118.stdout:checking for library containing strerror... none required 2024-03-24T00:17:22.875 INFO:tasks.workunit.client.0.smithi118.stdout:checking for ANSI C header files... yes 2024-03-24T00:17:23.005 INFO:tasks.workunit.client.0.smithi118.stdout:checking whether time.h and sys/time.h may both be included... yes 2024-03-24T00:17:23.139 INFO:tasks.workunit.client.0.smithi118.stdout:checking for sys/types.h... yes 2024-03-24T00:17:23.195 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:17:22 smithi118 ceph-mon[31535]: pgmap v448: 97 pgs: 97 active+clean; 675 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 1.5 KiB/s wr, 12 op/s 2024-03-24T00:17:23.203 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:17:22 smithi189 ceph-mon[36187]: pgmap v448: 97 pgs: 97 active+clean; 675 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 1.5 KiB/s wr, 12 op/s 2024-03-24T00:17:23.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:17:22 smithi193 ceph-mon[36878]: pgmap v448: 97 pgs: 97 active+clean; 675 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 4.3 KiB/s rd, 1.5 KiB/s wr, 12 op/s 2024-03-24T00:17:23.278 INFO:tasks.workunit.client.0.smithi118.stdout:checking for sys/stat.h... yes 2024-03-24T00:17:23.429 INFO:tasks.workunit.client.0.smithi118.stdout:checking for stdlib.h... yes 2024-03-24T00:17:23.567 INFO:tasks.workunit.client.0.smithi118.stdout:checking for string.h... yes 2024-03-24T00:17:23.721 INFO:tasks.workunit.client.0.smithi118.stdout:checking for memory.h... yes 2024-03-24T00:17:23.871 INFO:tasks.workunit.client.0.smithi118.stdout:checking for strings.h... yes 2024-03-24T00:17:24.025 INFO:tasks.workunit.client.0.smithi118.stdout:checking for inttypes.h... yes 2024-03-24T00:17:24.174 INFO:tasks.workunit.client.0.smithi118.stdout:checking for stdint.h... yes 2024-03-24T00:17:24.312 INFO:tasks.workunit.client.0.smithi118.stdout:checking for unistd.h... yes 2024-03-24T00:17:24.335 INFO:tasks.workunit.client.0.smithi118.stdout:checking for memory.h... (cached) yes 2024-03-24T00:17:24.352 INFO:tasks.workunit.client.0.smithi118.stdout:checking for strings.h... (cached) yes 2024-03-24T00:17:24.368 INFO:tasks.workunit.client.0.smithi118.stdout:checking for unistd.h... (cached) yes 2024-03-24T00:17:24.542 INFO:tasks.workunit.client.0.smithi118.stdout:checking locale.h usability... yes 2024-03-24T00:17:24.640 INFO:tasks.workunit.client.0.smithi118.stdout:checking locale.h presence... yes 2024-03-24T00:17:24.642 INFO:tasks.workunit.client.0.smithi118.stdout:checking for locale.h... yes 2024-03-24T00:17:24.659 INFO:tasks.workunit.client.0.smithi118.stdout:checking for stdlib.h... (cached) yes 2024-03-24T00:17:24.673 INFO:tasks.workunit.client.0.smithi118.stdout:checking for string.h... (cached) yes 2024-03-24T00:17:24.836 INFO:tasks.workunit.client.0.smithi118.stdout:checking sys/socket.h usability... yes 2024-03-24T00:17:24.914 INFO:tasks.workunit.client.0.smithi118.stdout:checking sys/socket.h presence... yes 2024-03-24T00:17:24.917 INFO:tasks.workunit.client.0.smithi118.stdout:checking for sys/socket.h... yes 2024-03-24T00:17:25.012 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:17:24 smithi193 ceph-mon[36878]: pgmap v449: 97 pgs: 97 active+clean; 675 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 3.2 KiB/s wr, 20 op/s 2024-03-24T00:17:25.059 INFO:tasks.workunit.client.0.smithi118.stdout:checking sys/time.h usability... yes 2024-03-24T00:17:25.137 INFO:tasks.workunit.client.0.smithi118.stdout:checking sys/time.h presence... yes 2024-03-24T00:17:25.142 INFO:tasks.workunit.client.0.smithi118.stdout:checking for sys/time.h... yes 2024-03-24T00:17:25.195 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:17:24 smithi118 ceph-mon[31535]: pgmap v449: 97 pgs: 97 active+clean; 675 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 3.2 KiB/s wr, 20 op/s 2024-03-24T00:17:25.203 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:17:24 smithi189 ceph-mon[36187]: pgmap v449: 97 pgs: 97 active+clean; 675 KiB data, 349 MiB used, 1.0 TiB / 1.0 TiB avail; 5.1 KiB/s rd, 3.2 KiB/s wr, 20 op/s 2024-03-24T00:17:25.283 INFO:tasks.workunit.client.0.smithi118.stdout:checking fcntl.h usability... yes 2024-03-24T00:17:25.361 INFO:tasks.workunit.client.0.smithi118.stdout:checking fcntl.h presence... yes 2024-03-24T00:17:25.365 INFO:tasks.workunit.client.0.smithi118.stdout:checking for fcntl.h... yes 2024-03-24T00:17:25.543 INFO:tasks.workunit.client.0.smithi118.stdout:checking sys/fcntl.h usability... yes 2024-03-24T00:17:25.611 INFO:tasks.workunit.client.0.smithi118.stdout:checking sys/fcntl.h presence... yes 2024-03-24T00:17:25.615 INFO:tasks.workunit.client.0.smithi118.stdout:checking for sys/fcntl.h... yes 2024-03-24T00:17:25.809 INFO:tasks.workunit.client.0.smithi118.stdout:checking sys/param.h usability... yes 2024-03-24T00:17:25.893 INFO:tasks.workunit.client.0.smithi118.stdout:checking sys/param.h presence... yes 2024-03-24T00:17:25.897 INFO:tasks.workunit.client.0.smithi118.stdout:checking for sys/param.h... yes 2024-03-24T00:17:26.023 INFO:tasks.workunit.client.0.smithi118.stdout:checking ioctl.h usability... no 2024-03-24T00:17:26.094 INFO:tasks.workunit.client.0.smithi118.stdout:checking ioctl.h presence... no 2024-03-24T00:17:26.098 INFO:tasks.workunit.client.0.smithi118.stdout:checking for ioctl.h... no 2024-03-24T00:17:26.200 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-03-24T00:17:26.200 DEBUG:teuthology.orchestra.run.smithi118:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-24T00:17:26.237 INFO:tasks.workunit.client.0.smithi118.stdout:checking sys/ioctl.h usability... yes 2024-03-24T00:17:26.301 INFO:tasks.workunit.client.0.smithi118.stdout:checking sys/ioctl.h presence... yes 2024-03-24T00:17:26.303 INFO:tasks.workunit.client.0.smithi118.stdout:checking for sys/ioctl.h... yes 2024-03-24T00:17:26.414 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:26.413+0000 7f8e7d600640 1 -- 172.21.15.118:0/4100388103 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f8e78100910 msgr2=0x7f8e78100d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:17:26.414 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:26.413+0000 7f8e7d600640 1 --2- 172.21.15.118:0/4100388103 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f8e78100910 0x7f8e78100d70 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7f8e64009900 tx=0x7f8e6402bcd0 comp rx=0 tx=0).stop 2024-03-24T00:17:26.414 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:26.414+0000 7f8e7d600640 1 -- 172.21.15.118:0/4100388103 shutdown_connections 2024-03-24T00:17:26.414 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:26.414+0000 7f8e7d600640 1 --2- 172.21.15.118:0/4100388103 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8e781012b0 0x7f8e78101730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:17:26.414 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:26.414+0000 7f8e7d600640 1 --2- 172.21.15.118:0/4100388103 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f8e78100910 0x7f8e78100d70 unknown :-1 s=CLOSED pgs=265 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:17:26.414 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:26.414+0000 7f8e7d600640 1 --2- 172.21.15.118:0/4100388103 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f8e780fffb0 0x7f8e781003d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:17:26.415 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:26.414+0000 7f8e7d600640 1 -- 172.21.15.118:0/4100388103 >> 172.21.15.118:0/4100388103 conn(0x7f8e780fbb50 msgr2=0x7f8e780fdf70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:17:26.415 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:26.414+0000 7f8e7d600640 1 -- 172.21.15.118:0/4100388103 shutdown_connections 2024-03-24T00:17:26.415 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:26.414+0000 7f8e7d600640 1 -- 172.21.15.118:0/4100388103 wait complete. 2024-03-24T00:17:26.417 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:26.415+0000 7f8e7d600640 1 Processor -- start 2024-03-24T00:17:26.417 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:26.415+0000 7f8e7d600640 1 -- start start 2024-03-24T00:17:26.417 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:26.416+0000 7f8e7d600640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8e780fffb0 0x7f8e78199bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:17:26.417 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:26.416+0000 7f8e7d600640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f8e78100910 0x7f8e7819a110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:17:26.417 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:26.416+0000 7f8e7d600640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f8e781012b0 0x7f8e781a1180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:17:26.418 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:26.416+0000 7f8e7d600640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8e7810f590 con 0x7f8e78100910 2024-03-24T00:17:26.418 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:26.416+0000 7f8e7d600640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8e7810f410 con 0x7f8e780fffb0 2024-03-24T00:17:26.418 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:26.416+0000 7f8e7d600640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8e7810f710 con 0x7f8e781012b0 2024-03-24T00:17:26.418 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:26.416+0000 7f8e76a00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8e780fffb0 0x7f8e78199bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:17:26.418 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:26.416+0000 7f8e76a00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8e780fffb0 0x7f8e78199bd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.118:45232/0 (socket says 172.21.15.118:45232) 2024-03-24T00:17:26.418 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:26.416+0000 7f8e76a00640 1 -- 172.21.15.118:0/761784334 learned_addr learned my addr 172.21.15.118:0/761784334 (peer_addr_for_me v2:172.21.15.118:0/0) 2024-03-24T00:17:26.418 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:26.416+0000 7f8e77400640 1 --2- 172.21.15.118:0/761784334 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f8e781012b0 0x7f8e781a1180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:17:26.418 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:26.416+0000 7f8e76000640 1 --2- 172.21.15.118:0/761784334 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f8e78100910 0x7f8e7819a110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:17:26.418 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:26.416+0000 7f8e77400640 1 -- 172.21.15.118:0/761784334 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8e780fffb0 msgr2=0x7f8e78199bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:17:26.418 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:26.416+0000 7f8e77400640 1 --2- 172.21.15.118:0/761784334 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8e780fffb0 0x7f8e78199bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:17:26.419 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:26.416+0000 7f8e77400640 1 -- 172.21.15.118:0/761784334 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f8e78100910 msgr2=0x7f8e7819a110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:17:26.419 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:26.416+0000 7f8e77400640 1 --2- 172.21.15.118:0/761784334 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f8e78100910 0x7f8e7819a110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:17:26.419 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:26.416+0000 7f8e77400640 1 -- 172.21.15.118:0/761784334 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8e781a17f0 con 0x7f8e781012b0 2024-03-24T00:17:26.419 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:26.416+0000 7f8e76000640 1 --2- 172.21.15.118:0/761784334 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f8e78100910 0x7f8e7819a110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T00:17:26.419 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:26.417+0000 7f8e77400640 1 --2- 172.21.15.118:0/761784334 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f8e781012b0 0x7f8e781a1180 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7f8e70009fd0 tx=0x7f8e7000ef10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:17:26.419 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:26.417+0000 7f8e6f400640 1 -- 172.21.15.118:0/761784334 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8e7000ce20 con 0x7f8e781012b0 2024-03-24T00:17:26.419 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:26.417+0000 7f8e6f400640 1 -- 172.21.15.118:0/761784334 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f8e70005020 con 0x7f8e781012b0 2024-03-24T00:17:26.419 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:26.417+0000 7f8e7d600640 1 -- 172.21.15.118:0/761784334 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8e781a1ae0 con 0x7f8e781012b0 2024-03-24T00:17:26.419 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:26.417+0000 7f8e6f400640 1 -- 172.21.15.118:0/761784334 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8e70010720 con 0x7f8e781012b0 2024-03-24T00:17:26.419 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:26.417+0000 7f8e7d600640 1 -- 172.21.15.118:0/761784334 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8e781a2000 con 0x7f8e781012b0 2024-03-24T00:17:26.420 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:26.421+0000 7f8e6f400640 1 -- 172.21.15.118:0/761784334 <== mon.1 v2:172.21.15.193:3300/0 4 ==== mgrmap(e 20) v1 ==== 137291+0+0 (secure 0 0 0) 0x7f8e70005540 con 0x7f8e781012b0 2024-03-24T00:17:26.424 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:26.422+0000 7f8e6f400640 1 --2- 172.21.15.118:0/761784334 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7f8e5c0a2800 0x7f8e5c0a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:17:26.424 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:26.422+0000 7f8e803cd740 1 Processor -- start 2024-03-24T00:17:26.424 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:26.422+0000 7f8e6f400640 1 -- 172.21.15.118:0/761784334 <== mon.1 v2:172.21.15.193:3300/0 5 ==== osd_map(90..90 src has 1..90) v4 ==== 8277+0+0 (secure 0 0 0) 0x7f8e700cf1d0 con 0x7f8e781012b0 2024-03-24T00:17:26.424 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:26.422+0000 7f8e803cd740 1 -- start start 2024-03-24T00:17:26.424 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:26.422+0000 7f8e76a00640 1 --2- 172.21.15.118:0/761784334 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7f8e5c0a2800 0x7f8e5c0a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:17:26.424 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:26.423+0000 7f8e803cd740 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x56051a881e80 0x56051a8822a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:17:26.424 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:26.423+0000 7f8e803cd740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56051a882870 0x56051a887370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:17:26.424 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:26.423+0000 7f8e803cd740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56051a8879e0 0x56051a889e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:17:26.425 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:26.423+0000 7f8e803cd740 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x56051a7e7250 con 0x56051a881e80 2024-03-24T00:17:26.425 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:26.423+0000 7f8e803cd740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x56051a78fa30 con 0x56051a882870 2024-03-24T00:17:26.425 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:26.423+0000 7f8e803cd740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x56051a7e32f0 con 0x56051a8879e0 2024-03-24T00:17:26.425 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:26.423+0000 7f8e76000640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56051a882870 0x56051a887370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:17:26.425 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:26.423+0000 7f8e76000640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56051a882870 0x56051a887370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.118:45238/0 (socket says 172.21.15.118:45238) 2024-03-24T00:17:26.425 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:26.423+0000 7f8e76000640 1 -- 172.21.15.118:0/2929294492 learned_addr learned my addr 172.21.15.118:0/2929294492 (peer_addr_for_me v2:172.21.15.118:0/0) 2024-03-24T00:17:26.425 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:26.423+0000 7f8e77400640 1 --2- 172.21.15.118:0/2929294492 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56051a8879e0 0x56051a889e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:17:26.425 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:26.423+0000 7f8e76a00640 1 --2- 172.21.15.118:0/2929294492 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x56051a881e80 0x56051a8822a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:17:26.425 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:26.423+0000 7f8e76000640 1 -- 172.21.15.118:0/2929294492 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56051a8879e0 msgr2=0x56051a889e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:17:26.426 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:26.423+0000 7f8e76000640 1 --2- 172.21.15.118:0/2929294492 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56051a8879e0 0x56051a889e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:17:26.426 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:26.423+0000 7f8e76000640 1 -- 172.21.15.118:0/2929294492 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x56051a881e80 msgr2=0x56051a8822a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:17:26.426 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:26.423+0000 7f8e76000640 1 --2- 172.21.15.118:0/2929294492 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x56051a881e80 0x56051a8822a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:17:26.426 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:26.423+0000 7f8e76000640 1 -- 172.21.15.118:0/2929294492 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x56051a77a690 con 0x56051a882870 2024-03-24T00:17:26.426 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:26.423+0000 7f8e76a00640 1 --2- 172.21.15.118:0/2929294492 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x56051a881e80 0x56051a8822a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T00:17:26.426 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:26.424+0000 7f8e76a00640 1 --2- 172.21.15.118:0/761784334 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7f8e5c0a2800 0x7f8e5c0a4cc0 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x7f8e68009ce0 tx=0x7f8e68011810 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:17:26.426 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:26.424+0000 7f8e76000640 1 --2- 172.21.15.118:0/2929294492 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56051a882870 0x56051a887370 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7f8e64000c00 tx=0x7f8e64005d50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e1f95d076cb332f3 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:17:26.426 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:26.424+0000 7f8e63400640 1 -- 172.21.15.118:0/2929294492 <== mon.2 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8e6402dcb0 con 0x56051a882870 2024-03-24T00:17:26.426 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:26.424+0000 7f8e63400640 1 -- 172.21.15.118:0/2929294492 <== mon.2 v2:172.21.15.189:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f8e6402de50 con 0x56051a882870 2024-03-24T00:17:26.426 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:26.424+0000 7f8e63400640 1 -- 172.21.15.118:0/2929294492 <== mon.2 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8e640048e0 con 0x56051a882870 2024-03-24T00:17:26.431 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:26.425+0000 7f8e803cd740 1 -- 172.21.15.118:0/2929294492 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56051a882870 msgr2=0x56051a887370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:17:26.431 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:26.425+0000 7f8e803cd740 1 --2- 172.21.15.118:0/2929294492 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56051a882870 0x56051a887370 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7f8e64000c00 tx=0x7f8e64005d50 comp rx=0 tx=0).stop 2024-03-24T00:17:26.431 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:26.425+0000 7f8e803cd740 1 -- 172.21.15.118:0/2929294492 shutdown_connections 2024-03-24T00:17:26.432 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:26.425+0000 7f8e803cd740 1 --2- 172.21.15.118:0/2929294492 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56051a8879e0 0x56051a889e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:17:26.432 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:26.425+0000 7f8e803cd740 1 --2- 172.21.15.118:0/2929294492 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56051a882870 0x56051a887370 unknown :-1 s=CLOSED pgs=276 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:17:26.432 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:26.425+0000 7f8e803cd740 1 --2- 172.21.15.118:0/2929294492 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x56051a881e80 0x56051a8822a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:17:26.432 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:26.425+0000 7f8e803cd740 1 -- 172.21.15.118:0/2929294492 >> 172.21.15.118:0/2929294492 conn(0x56051a84a200 msgr2=0x56051a87d7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:17:26.432 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:26.425+0000 7f8e803cd740 1 -- 172.21.15.118:0/2929294492 shutdown_connections 2024-03-24T00:17:26.432 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:26.425+0000 7f8e803cd740 1 -- 172.21.15.118:0/2929294492 wait complete. 2024-03-24T00:17:26.432 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:26.426+0000 7f8e803cd740 1 -- start start 2024-03-24T00:17:26.432 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:26.426+0000 7f8e803cd740 1 Processor -- start 2024-03-24T00:17:26.432 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:26.426+0000 7f8e803cd740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56051a881e80 0x56051a88be00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:17:26.432 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:26.426+0000 7f8e803cd740 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x56051a882870 0x56051a88c340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:17:26.432 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:26.426+0000 7f8e803cd740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56051a8879e0 0x56051a893380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:17:26.432 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:26.426+0000 7f8e803cd740 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x56051a88a930 con 0x56051a882870 2024-03-24T00:17:26.433 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:26.426+0000 7f8e803cd740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x56051a7a2e80 con 0x56051a8879e0 2024-03-24T00:17:26.433 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:26.426+0000 7f8e803cd740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x56051a88aab0 con 0x56051a881e80 2024-03-24T00:17:26.433 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:26.426+0000 7f8e803cd740 10 client.0 _ll_register_callbacks cb 0x56051a7b5130 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T00:17:26.433 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:26.427+0000 7f8e76a00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56051a881e80 0x56051a88be00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:17:26.433 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:26.427+0000 7f8e77400640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56051a8879e0 0x56051a893380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:17:26.433 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:26.427+0000 7f8e76a00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56051a881e80 0x56051a88be00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.118:37628/0 (socket says 172.21.15.118:37628) 2024-03-24T00:17:26.433 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:26.427+0000 7f8e77400640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56051a8879e0 0x56051a893380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.118:45250/0 (socket says 172.21.15.118:45250) 2024-03-24T00:17:26.433 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:26.427+0000 7f8e76a00640 1 -- 172.21.15.118:0/257993623 learned_addr learned my addr 172.21.15.118:0/257993623 (peer_addr_for_me v2:172.21.15.118:0/0) 2024-03-24T00:17:26.433 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:26.427+0000 7f8e76000640 1 --2- 172.21.15.118:0/257993623 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x56051a882870 0x56051a88c340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:17:26.433 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:26.427+0000 7f8e77400640 1 -- 172.21.15.118:0/257993623 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56051a881e80 msgr2=0x56051a88be00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:17:26.433 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:26.427+0000 7f8e77400640 1 --2- 172.21.15.118:0/257993623 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56051a881e80 0x56051a88be00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:17:26.434 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:26.427+0000 7f8e77400640 1 -- 172.21.15.118:0/257993623 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x56051a882870 msgr2=0x56051a88c340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:17:26.434 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:26.427+0000 7f8e77400640 1 --2- 172.21.15.118:0/257993623 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x56051a882870 0x56051a88c340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:17:26.434 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:26.427+0000 7f8e77400640 1 -- 172.21.15.118:0/257993623 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x56051a5c57d0 con 0x56051a8879e0 2024-03-24T00:17:26.434 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:26.427+0000 7f8e76a00640 1 --2- 172.21.15.118:0/257993623 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56051a881e80 0x56051a88be00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T00:17:26.434 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:26.427+0000 7f8e76000640 1 --2- 172.21.15.118:0/257993623 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x56051a882870 0x56051a88c340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T00:17:26.434 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:26.427+0000 7f8e77400640 1 --2- 172.21.15.118:0/257993623 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56051a8879e0 0x56051a893380 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7f8e7000c760 tx=0x7f8e7009bba0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=cd0234db63b2fcba server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:17:26.434 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:26.427+0000 7f8e57e00640 10 client.0 ms_handle_connect on v2:172.21.15.189:3300/0 2024-03-24T00:17:26.434 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:26.427+0000 7f8e57e00640 1 -- 172.21.15.118:0/257993623 <== mon.2 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8e700a3df0 con 0x56051a8879e0 2024-03-24T00:17:26.434 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:26.427+0000 7f8e57e00640 1 -- 172.21.15.118:0/257993623 <== mon.2 v2:172.21.15.189:3300/0 2 ==== osd_map(90..90 src has 1..90) v4 ==== 8277+0+0 (secure 0 0 0) 0x7f8e700a3f90 con 0x56051a8879e0 2024-03-24T00:17:26.434 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:26.427+0000 7f8e803cd740 1 -- 172.21.15.118:0/257993623 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f8e6402de50 con 0x56051a8879e0 2024-03-24T00:17:26.434 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:26.428+0000 7f8e57e00640 1 -- 172.21.15.118:0/257993623 <== mon.2 v2:172.21.15.189:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) v2 ==== 24+0+0 (secure 0 0 0) 0x7f8e700b4ba0 con 0x56051a8879e0 2024-03-24T00:17:26.435 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:26.428+0000 7f8e803cd740 10 client.25823 fetch_fsmap learned FSMap version 28 2024-03-24T00:17:26.435 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:26.428+0000 7f8e803cd740 1 -- 172.21.15.118:0/257993623 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) v3 -- 0x56051a4fb3f0 con 0x56051a8879e0 2024-03-24T00:17:26.435 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:26.428+0000 7f8e57e00640 1 -- 172.21.15.118:0/257993623 <== mon.2 v2:172.21.15.189:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f8e6402de50 con 0x56051a8879e0 2024-03-24T00:17:26.435 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:26.428+0000 7f8e57e00640 1 -- 172.21.15.118:0/257993623 <== mon.2 v2:172.21.15.189:3300/0 5 ==== fsmap(e 28) v1 ==== 7934+0+0 (secure 0 0 0) 0x7f8e7009aba0 con 0x56051a8879e0 2024-03-24T00:17:26.435 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:26.428+0000 7f8e803cd740 10 client.25823 fetch_fsmap finished waiting for FSMap version 28 2024-03-24T00:17:26.435 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:26.428+0000 7f8e803cd740 10 client.25823 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-24T00:17:26.435 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:26.428+0000 7f8e803cd740 20 client.25823 populate_metadata read hostname 'smithi118' 2024-03-24T00:17:26.435 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:26.428+0000 7f8e803cd740 1 --2- 172.21.15.118:0/257993623 >> [v2:172.21.15.189:6838/4074484168,v1:172.21.15.189:6839/4074484168] conn(0x56051a897b80 0x56051a89a000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:17:26.435 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:26.428+0000 7f8e803cd740 4 client.25823 mds_command: new command op to 24569 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-24T00:17:26.435 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:26.428+0000 7f8e803cd740 1 -- 172.21.15.118:0/257993623 --> [v2:172.21.15.189:6838/4074484168,v1:172.21.15.189:6839/4074484168] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x56051a7f4790 con 0x56051a897b80 2024-03-24T00:17:26.435 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:26.429+0000 7f8e76a00640 1 --2- 172.21.15.118:0/257993623 >> [v2:172.21.15.189:6838/4074484168,v1:172.21.15.189:6839/4074484168] conn(0x56051a897b80 0x56051a89a000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:17:26.435 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:26.429+0000 7f8e76a00640 1 --2- 172.21.15.118:0/257993623 >> [v2:172.21.15.189:6838/4074484168,v1:172.21.15.189:6839/4074484168] conn(0x56051a897b80 0x56051a89a000 crc :-1 s=READY pgs=67 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=a3a14cd2d6f2a161 server_cookie=604159244382db62 in_seq=0 out_seq=0 2024-03-24T00:17:26.435 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:26.429+0000 7f8e57e00640 10 client.25823 ms_handle_connect on v2:172.21.15.189:6838/4074484168 2024-03-24T00:17:26.436 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:26.430+0000 7f8e57e00640 1 -- 172.21.15.118:0/257993623 <== mds.0 v2:172.21.15.189:6838/4074484168 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x56051a7f4790 con 0x56051a897b80 2024-03-24T00:17:26.436 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:26.430+0000 7f8e57e00640 10 client.25823 handle_command_reply: tid=0 2024-03-24T00:17:26.436 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:26.430+0000 7f8e803cd740 1 client.25823 shutdown 2024-03-24T00:17:26.436 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:26.431+0000 7f8e803cd740 1 -- 172.21.15.118:0/257993623 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56051a8879e0 msgr2=0x56051a893380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:17:26.436 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:26.431+0000 7f8e803cd740 1 --2- 172.21.15.118:0/257993623 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56051a8879e0 0x56051a893380 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7f8e7000c760 tx=0x7f8e7009bba0 comp rx=0 tx=0).stop 2024-03-24T00:17:26.436 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:26.431+0000 7f8e803cd740 1 -- 172.21.15.118:0/257993623 shutdown_connections 2024-03-24T00:17:26.436 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:26.431+0000 7f8e803cd740 1 --2- 172.21.15.118:0/257993623 >> [v2:172.21.15.189:6838/4074484168,v1:172.21.15.189:6839/4074484168] conn(0x56051a897b80 0x56051a89a000 crc :-1 s=READY pgs=67 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:17:26.436 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:26.431+0000 7f8e803cd740 1 --2- 172.21.15.118:0/257993623 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56051a8879e0 0x56051a893380 unknown :-1 s=CLOSED pgs=277 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:17:26.436 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:26.431+0000 7f8e803cd740 1 --2- 172.21.15.118:0/257993623 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x56051a882870 0x56051a88c340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:17:26.436 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:26.431+0000 7f8e803cd740 1 --2- 172.21.15.118:0/257993623 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56051a881e80 0x56051a88be00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:17:26.437 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:26.431+0000 7f8e803cd740 1 -- 172.21.15.118:0/257993623 >> 172.21.15.118:0/257993623 conn(0x56051a84a200 msgr2=0x56051a87ca90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:17:26.437 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:26.431+0000 7f8e57e00640 0 client.25823 ms_handle_reset on v2:172.21.15.189:6838/4074484168 2024-03-24T00:17:26.437 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:26.431+0000 7f8e803cd740 1 -- 172.21.15.118:0/257993623 shutdown_connections 2024-03-24T00:17:26.437 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:26.431+0000 7f8e803cd740 1 -- 172.21.15.118:0/257993623 wait complete. 2024-03-24T00:17:26.437 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:26.431+0000 7f8e803cd740 20 client.25823 trim_cache size 0 max 16384 2024-03-24T00:17:26.437 INFO:tasks.workunit.client.0.smithi118.stdout:checking return type of signal handlers... void 2024-03-24T00:17:26.451 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:26.451+0000 7f8e803cd740 1 Processor -- start 2024-03-24T00:17:26.452 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:26.451+0000 7f8e803cd740 1 -- start start 2024-03-24T00:17:26.452 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:26.452+0000 7f8e803cd740 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x56051a881e80 0x56051a883310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:17:26.452 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:26.452+0000 7f8e803cd740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56051a882870 0x56051a893340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:17:26.452 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:26.452+0000 7f8e803cd740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56051a8879e0 0x56051a883850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:17:26.452 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:26.452+0000 7f8e803cd740 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8e781aac00 con 0x56051a881e80 2024-03-24T00:17:26.452 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:26.452+0000 7f8e803cd740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8e781ac580 con 0x56051a8879e0 2024-03-24T00:17:26.452 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:26.452+0000 7f8e803cd740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8e781ab260 con 0x56051a882870 2024-03-24T00:17:26.452 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:26.452+0000 7f8e76a00640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x56051a881e80 0x56051a883310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:17:26.452 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:26.452+0000 7f8e76a00640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x56051a881e80 0x56051a883310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.118:3300/0 says I am v2:172.21.15.118:38234/0 (socket says 172.21.15.118:38234) 2024-03-24T00:17:26.452 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:26.452+0000 7f8e77400640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56051a8879e0 0x56051a883850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:17:26.452 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:26.452+0000 7f8e76a00640 1 -- 172.21.15.118:0/2308814902 learned_addr learned my addr 172.21.15.118:0/2308814902 (peer_addr_for_me v2:172.21.15.118:0/0) 2024-03-24T00:17:26.452 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:26.452+0000 7f8e76000640 1 --2- 172.21.15.118:0/2308814902 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56051a882870 0x56051a893340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:17:26.456 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:26.452+0000 7f8e77400640 1 -- 172.21.15.118:0/2308814902 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56051a882870 msgr2=0x56051a893340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:17:26.457 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:26.452+0000 7f8e77400640 1 --2- 172.21.15.118:0/2308814902 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56051a882870 0x56051a893340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:17:26.457 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:26.452+0000 7f8e77400640 1 -- 172.21.15.118:0/2308814902 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x56051a881e80 msgr2=0x56051a883310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:17:26.457 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:26.452+0000 7f8e77400640 1 --2- 172.21.15.118:0/2308814902 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x56051a881e80 0x56051a883310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:17:26.457 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:26.452+0000 7f8e77400640 1 -- 172.21.15.118:0/2308814902 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x56051a4bb0f0 con 0x56051a8879e0 2024-03-24T00:17:26.457 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:26.452+0000 7f8e76a00640 1 --2- 172.21.15.118:0/2308814902 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x56051a881e80 0x56051a883310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T00:17:26.457 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:26.452+0000 7f8e76000640 1 --2- 172.21.15.118:0/2308814902 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56051a882870 0x56051a893340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T00:17:26.457 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:26.453+0000 7f8e77400640 1 --2- 172.21.15.118:0/2308814902 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56051a8879e0 0x56051a883850 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7f8e700a3d50 tx=0x7f8e70099ff0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=82e6f27384d581b4 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:17:26.457 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:26.453+0000 7f8e62a00640 1 -- 172.21.15.118:0/2308814902 <== mon.2 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8e700ba250 con 0x56051a8879e0 2024-03-24T00:17:26.457 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:26.453+0000 7f8e62a00640 1 -- 172.21.15.118:0/2308814902 <== mon.2 v2:172.21.15.189:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f8e700a3df0 con 0x56051a8879e0 2024-03-24T00:17:26.457 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:26.453+0000 7f8e62a00640 1 -- 172.21.15.118:0/2308814902 <== mon.2 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8e700a0bf0 con 0x56051a8879e0 2024-03-24T00:17:26.458 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:26.454+0000 7f8e803cd740 1 -- 172.21.15.118:0/2308814902 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56051a8879e0 msgr2=0x56051a883850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:17:26.458 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:26.454+0000 7f8e803cd740 1 --2- 172.21.15.118:0/2308814902 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56051a8879e0 0x56051a883850 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7f8e700a3d50 tx=0x7f8e70099ff0 comp rx=0 tx=0).stop 2024-03-24T00:17:26.458 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:26.454+0000 7f8e803cd740 1 -- 172.21.15.118:0/2308814902 shutdown_connections 2024-03-24T00:17:26.458 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:26.454+0000 7f8e803cd740 1 --2- 172.21.15.118:0/2308814902 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56051a8879e0 0x56051a883850 unknown :-1 s=CLOSED pgs=278 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:17:26.458 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:26.454+0000 7f8e803cd740 1 --2- 172.21.15.118:0/2308814902 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56051a882870 0x56051a893340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:17:26.458 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:26.454+0000 7f8e803cd740 1 --2- 172.21.15.118:0/2308814902 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x56051a881e80 0x56051a883310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:17:26.458 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:26.454+0000 7f8e803cd740 1 -- 172.21.15.118:0/2308814902 >> 172.21.15.118:0/2308814902 conn(0x56051a84a200 msgr2=0x56051a88aee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:17:26.458 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:26.454+0000 7f8e803cd740 1 -- 172.21.15.118:0/2308814902 shutdown_connections 2024-03-24T00:17:26.458 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:26.454+0000 7f8e803cd740 1 -- 172.21.15.118:0/2308814902 wait complete. 2024-03-24T00:17:26.458 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:26.454+0000 7f8e803cd740 1 -- start start 2024-03-24T00:17:26.458 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:26.455+0000 7f8e803cd740 1 Processor -- start 2024-03-24T00:17:26.458 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:26.455+0000 7f8e803cd740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56051a881e80 0x56051a890980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:17:26.459 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:26.455+0000 7f8e803cd740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56051a882870 0x56051a890ec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:17:26.459 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:26.455+0000 7f8e803cd740 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x56051a8879e0 0x56051a891e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:17:26.459 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:26.455+0000 7f8e803cd740 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8e781abf20 con 0x56051a8879e0 2024-03-24T00:17:26.459 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:26.455+0000 7f8e803cd740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8e781ab8c0 con 0x56051a882870 2024-03-24T00:17:26.459 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:26.455+0000 7f8e803cd740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8e781a8dc0 con 0x56051a881e80 2024-03-24T00:17:26.459 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:26.455+0000 7f8e803cd740 10 client.0 _ll_register_callbacks cb 0x7f8e70005b20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T00:17:26.459 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:26.455+0000 7f8e76a00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56051a881e80 0x56051a890980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:17:26.459 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:26.455+0000 7f8e76000640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56051a882870 0x56051a890ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:17:26.459 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:26.455+0000 7f8e76a00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56051a881e80 0x56051a890980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.118:37640/0 (socket says 172.21.15.118:37640) 2024-03-24T00:17:26.459 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:26.455+0000 7f8e76a00640 1 -- 172.21.15.118:0/92512849 learned_addr learned my addr 172.21.15.118:0/92512849 (peer_addr_for_me v2:172.21.15.118:0/0) 2024-03-24T00:17:26.459 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:26.455+0000 7f8e77400640 1 --2- 172.21.15.118:0/92512849 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x56051a8879e0 0x56051a891e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:17:26.459 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:26.456+0000 7f8e76a00640 1 -- 172.21.15.118:0/92512849 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56051a882870 msgr2=0x56051a890ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:17:26.460 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:26.456+0000 7f8e76a00640 1 --2- 172.21.15.118:0/92512849 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56051a882870 0x56051a890ec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:17:26.460 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:26.456+0000 7f8e76a00640 1 -- 172.21.15.118:0/92512849 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x56051a8879e0 msgr2=0x56051a891e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:17:26.460 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:26.456+0000 7f8e76a00640 1 --2- 172.21.15.118:0/92512849 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x56051a8879e0 0x56051a891e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:17:26.460 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:26.456+0000 7f8e76a00640 1 -- 172.21.15.118:0/92512849 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x56051a892470 con 0x56051a881e80 2024-03-24T00:17:26.460 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:26.456+0000 7f8e76000640 1 --2- 172.21.15.118:0/92512849 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56051a882870 0x56051a890ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T00:17:26.460 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:26.456+0000 7f8e76a00640 1 --2- 172.21.15.118:0/92512849 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56051a881e80 0x56051a890980 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7f8e68009fd0 tx=0x7f8e68004730 comp rx=0 tx=0).ready entity=mon.1 client_cookie=15149d9124971630 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:17:26.460 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:26.456+0000 7f8e57e00640 10 client.0 ms_handle_connect on v2:172.21.15.193:3300/0 2024-03-24T00:17:26.460 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:26.456+0000 7f8e57e00640 1 -- 172.21.15.118:0/92512849 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f8e68017040 con 0x56051a881e80 2024-03-24T00:17:26.460 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:26.456+0000 7f8e803cd740 1 -- 172.21.15.118:0/92512849 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f8e700a3df0 con 0x56051a881e80 2024-03-24T00:17:26.460 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:26.456+0000 7f8e57e00640 1 -- 172.21.15.118:0/92512849 <== mon.1 v2:172.21.15.193:3300/0 2 ==== osd_map(90..90 src has 1..90) v4 ==== 8277+0+0 (secure 0 0 0) 0x7f8e68009040 con 0x56051a881e80 2024-03-24T00:17:26.460 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:26.456+0000 7f8e57e00640 1 -- 172.21.15.118:0/92512849 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) v2 ==== 24+0+0 (secure 0 0 0) 0x7f8e6802c110 con 0x56051a881e80 2024-03-24T00:17:26.460 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:26.456+0000 7f8e803cd740 10 client.25816 fetch_fsmap learned FSMap version 28 2024-03-24T00:17:26.461 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:26.456+0000 7f8e803cd740 1 -- 172.21.15.118:0/92512849 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) v3 -- 0x56051a8f3490 con 0x56051a881e80 2024-03-24T00:17:26.461 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:26.457+0000 7f8e57e00640 1 -- 172.21.15.118:0/92512849 <== mon.1 v2:172.21.15.193:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f8e700a3df0 con 0x56051a881e80 2024-03-24T00:17:26.461 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:26.457+0000 7f8e57e00640 1 -- 172.21.15.118:0/92512849 <== mon.1 v2:172.21.15.193:3300/0 5 ==== fsmap(e 28) v1 ==== 7934+0+0 (secure 0 0 0) 0x7f8e68004aa0 con 0x56051a881e80 2024-03-24T00:17:26.461 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:26.457+0000 7f8e803cd740 10 client.25816 fetch_fsmap finished waiting for FSMap version 28 2024-03-24T00:17:26.461 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:26.457+0000 7f8e803cd740 10 client.25816 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-24T00:17:26.461 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:26.457+0000 7f8e803cd740 20 client.25816 populate_metadata read hostname 'smithi118' 2024-03-24T00:17:26.461 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:26.457+0000 7f8e803cd740 1 --2- 172.21.15.118:0/92512849 >> [v2:172.21.15.189:6838/4074484168,v1:172.21.15.189:6839/4074484168] conn(0x56051a8f3df0 0x56051a8f6210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:17:26.461 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:26.457+0000 7f8e803cd740 4 client.25816 mds_command: new command op to 24569 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-24T00:17:26.461 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:26.457+0000 7f8e803cd740 1 -- 172.21.15.118:0/92512849 --> [v2:172.21.15.189:6838/4074484168,v1:172.21.15.189:6839/4074484168] -- command(tid 0: {"prefix": "scrub status"}) v1 -- 0x56051a7eeae0 con 0x56051a8f3df0 2024-03-24T00:17:26.461 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:26.457+0000 7f8e76000640 1 --2- 172.21.15.118:0/92512849 >> [v2:172.21.15.189:6838/4074484168,v1:172.21.15.189:6839/4074484168] conn(0x56051a8f3df0 0x56051a8f6210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:17:26.463 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:26.458+0000 7f8e57e00640 10 client.25816 ms_handle_connect on v2:172.21.15.189:6838/4074484168 2024-03-24T00:17:26.463 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:26.458+0000 7f8e76000640 1 --2- 172.21.15.118:0/92512849 >> [v2:172.21.15.189:6838/4074484168,v1:172.21.15.189:6839/4074484168] conn(0x56051a8f3df0 0x56051a8f6210 crc :-1 s=READY pgs=68 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=bb7032109811835 server_cookie=f28b11e0c65d18f7 in_seq=0 out_seq=0 2024-03-24T00:17:26.463 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:26.458+0000 7f8e57e00640 1 -- 172.21.15.118:0/92512849 <== mds.0 v2:172.21.15.189:6838/4074484168 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+63 (crc 0 0 0) 0x56051a7eeae0 con 0x56051a8f3df0 2024-03-24T00:17:26.463 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:26.458+0000 7f8e57e00640 10 client.25816 handle_command_reply: tid=0 2024-03-24T00:17:26.463 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:26.458+0000 7f8e803cd740 1 client.25816 shutdown 2024-03-24T00:17:26.463 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:26.459+0000 7f8e803cd740 1 -- 172.21.15.118:0/92512849 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56051a881e80 msgr2=0x56051a890980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:17:26.463 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:26.459+0000 7f8e803cd740 1 --2- 172.21.15.118:0/92512849 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56051a881e80 0x56051a890980 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7f8e68009fd0 tx=0x7f8e68004730 comp rx=0 tx=0).stop 2024-03-24T00:17:26.463 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:26.459+0000 7f8e803cd740 1 -- 172.21.15.118:0/92512849 shutdown_connections 2024-03-24T00:17:26.463 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:26.459+0000 7f8e803cd740 1 --2- 172.21.15.118:0/92512849 >> [v2:172.21.15.189:6838/4074484168,v1:172.21.15.189:6839/4074484168] conn(0x56051a8f3df0 0x56051a8f6210 crc :-1 s=READY pgs=68 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:17:26.464 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:26.459+0000 7f8e803cd740 1 --2- 172.21.15.118:0/92512849 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x56051a8879e0 0x56051a891e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:17:26.464 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:26.459+0000 7f8e803cd740 1 --2- 172.21.15.118:0/92512849 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56051a882870 0x56051a890ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:17:26.464 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:26.459+0000 7f8e803cd740 1 --2- 172.21.15.118:0/92512849 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56051a881e80 0x56051a890980 unknown :-1 s=CLOSED pgs=267 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:17:26.464 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:26.459+0000 7f8e57e00640 0 client.25816 ms_handle_reset on v2:172.21.15.189:6838/4074484168 2024-03-24T00:17:26.464 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:26.459+0000 7f8e803cd740 1 -- 172.21.15.118:0/92512849 >> 172.21.15.118:0/92512849 conn(0x56051a84a200 msgr2=0x56051a883310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:17:26.464 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:26.459+0000 7f8e803cd740 1 -- 172.21.15.118:0/92512849 shutdown_connections 2024-03-24T00:17:26.464 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:26.459+0000 7f8e803cd740 1 -- 172.21.15.118:0/92512849 wait complete. 2024-03-24T00:17:26.464 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:26.459+0000 7f8e803cd740 20 client.25816 trim_cache size 0 max 16384 2024-03-24T00:17:26.464 INFO:teuthology.orchestra.run.smithi118.stdout:{ 2024-03-24T00:17:26.465 INFO:teuthology.orchestra.run.smithi118.stdout: "status": "no active scrubs running", 2024-03-24T00:17:26.465 INFO:teuthology.orchestra.run.smithi118.stdout: "scrubs": {} 2024-03-24T00:17:26.465 INFO:teuthology.orchestra.run.smithi118.stdout:} 2024-03-24T00:17:26.465 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:26.460+0000 7f8e7d600640 1 -- 172.21.15.118:0/761784334 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7f8e5c0a2800 msgr2=0x7f8e5c0a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:17:26.466 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:26.460+0000 7f8e7d600640 1 --2- 172.21.15.118:0/761784334 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7f8e5c0a2800 0x7f8e5c0a4cc0 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x7f8e68009ce0 tx=0x7f8e68011810 comp rx=0 tx=0).stop 2024-03-24T00:17:26.466 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:26.460+0000 7f8e7d600640 1 -- 172.21.15.118:0/761784334 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f8e781012b0 msgr2=0x7f8e781a1180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:17:26.466 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:26.460+0000 7f8e7d600640 1 --2- 172.21.15.118:0/761784334 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f8e781012b0 0x7f8e781a1180 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7f8e70009fd0 tx=0x7f8e7000ef10 comp rx=0 tx=0).stop 2024-03-24T00:17:26.466 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:26.460+0000 7f8e7d600640 1 -- 172.21.15.118:0/761784334 shutdown_connections 2024-03-24T00:17:26.466 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:26.460+0000 7f8e7d600640 1 --2- 172.21.15.118:0/761784334 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7f8e5c0a2800 0x7f8e5c0a4cc0 unknown :-1 s=CLOSED pgs=296 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:17:26.466 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:26.460+0000 7f8e7d600640 1 --2- 172.21.15.118:0/761784334 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f8e781012b0 0x7f8e781a1180 unknown :-1 s=CLOSED pgs=266 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:17:26.466 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:26.460+0000 7f8e7d600640 1 --2- 172.21.15.118:0/761784334 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f8e78100910 0x7f8e7819a110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:17:26.466 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:26.460+0000 7f8e7d600640 1 --2- 172.21.15.118:0/761784334 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f8e780fffb0 0x7f8e78199bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:17:26.466 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:26.460+0000 7f8e7d600640 1 -- 172.21.15.118:0/761784334 >> 172.21.15.118:0/761784334 conn(0x7f8e780fbb50 msgr2=0x7f8e780fd650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:17:26.466 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:26.460+0000 7f8e7d600640 1 -- 172.21.15.118:0/761784334 shutdown_connections 2024-03-24T00:17:26.466 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:26.460+0000 7f8e7d600640 1 -- 172.21.15.118:0/761784334 wait complete. 2024-03-24T00:17:26.479 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-03-24T00:17:26.479 DEBUG:teuthology.orchestra.run.smithi118:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-24T00:17:26.566 INFO:tasks.workunit.client.0.smithi118.stdout:checking for size_t... yes 2024-03-24T00:17:26.688 INFO:tasks.workunit.client.0.smithi118.stdout:checking for dev_t... yes 2024-03-24T00:17:26.723 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:26.723+0000 7f2422200640 1 -- 172.21.15.118:0/3894525246 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f241c107ce0 msgr2=0x7f241c108100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:17:26.723 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:26.723+0000 7f2422200640 1 --2- 172.21.15.118:0/3894525246 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f241c107ce0 0x7f241c108100 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7f240c009900 tx=0x7f240c02bc40 comp rx=0 tx=0).stop 2024-03-24T00:17:26.723 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:26.723+0000 7f2422200640 1 -- 172.21.15.118:0/3894525246 shutdown_connections 2024-03-24T00:17:26.723 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:26.723+0000 7f2422200640 1 --2- 172.21.15.118:0/3894525246 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f241c109890 0x7f241c110120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:17:26.723 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:26.723+0000 7f2422200640 1 --2- 172.21.15.118:0/3894525246 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f241c108ed0 0x7f241c109350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:17:26.723 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:26.723+0000 7f2422200640 1 --2- 172.21.15.118:0/3894525246 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f241c107ce0 0x7f241c108100 unknown :-1 s=CLOSED pgs=268 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:17:26.723 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:26.723+0000 7f2422200640 1 -- 172.21.15.118:0/3894525246 >> 172.21.15.118:0/3894525246 conn(0x7f241c076cb0 msgr2=0x7f241c0790d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:17:26.723 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:26.723+0000 7f2422200640 1 -- 172.21.15.118:0/3894525246 shutdown_connections 2024-03-24T00:17:26.723 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:26.723+0000 7f2422200640 1 -- 172.21.15.118:0/3894525246 wait complete. 2024-03-24T00:17:26.726 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:26.724+0000 7f2422200640 1 Processor -- start 2024-03-24T00:17:26.726 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:26.724+0000 7f2422200640 1 -- start start 2024-03-24T00:17:26.726 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:26.724+0000 7f2422200640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f241c107ce0 0x7f241c19df60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:17:26.726 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:26.724+0000 7f2422200640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f241c108ed0 0x7f241c19e4a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:17:26.726 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:26.725+0000 7f2422200640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f241c109890 0x7f241c1a5510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:17:26.726 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:26.725+0000 7f2422200640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f241c113980 con 0x7f241c109890 2024-03-24T00:17:26.726 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:26.725+0000 7f2422200640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f241c113800 con 0x7f241c108ed0 2024-03-24T00:17:26.726 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:26.725+0000 7f2422200640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f241c113b00 con 0x7f241c107ce0 2024-03-24T00:17:26.726 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:26.725+0000 7f241aa00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f241c108ed0 0x7f241c19e4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:17:26.726 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:26.725+0000 7f241b400640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f241c107ce0 0x7f241c19df60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:17:26.726 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:26.725+0000 7f241aa00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f241c108ed0 0x7f241c19e4a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.118:45286/0 (socket says 172.21.15.118:45286) 2024-03-24T00:17:26.726 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:26.725+0000 7f241aa00640 1 -- 172.21.15.118:0/2171715402 learned_addr learned my addr 172.21.15.118:0/2171715402 (peer_addr_for_me v2:172.21.15.118:0/0) 2024-03-24T00:17:26.727 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:26.725+0000 7f241be00640 1 --2- 172.21.15.118:0/2171715402 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f241c109890 0x7f241c1a5510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:17:26.727 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:26.725+0000 7f241aa00640 1 -- 172.21.15.118:0/2171715402 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f241c107ce0 msgr2=0x7f241c19df60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:17:26.727 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:26.725+0000 7f241aa00640 1 --2- 172.21.15.118:0/2171715402 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f241c107ce0 0x7f241c19df60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:17:26.727 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:26.725+0000 7f241aa00640 1 -- 172.21.15.118:0/2171715402 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f241c109890 msgr2=0x7f241c1a5510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:17:26.727 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:26.725+0000 7f241aa00640 1 --2- 172.21.15.118:0/2171715402 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f241c109890 0x7f241c1a5510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:17:26.727 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:26.725+0000 7f241aa00640 1 -- 172.21.15.118:0/2171715402 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f241c1a5b80 con 0x7f241c108ed0 2024-03-24T00:17:26.727 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:26.725+0000 7f241b400640 1 --2- 172.21.15.118:0/2171715402 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f241c107ce0 0x7f241c19df60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T00:17:26.727 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:26.726+0000 7f241aa00640 1 --2- 172.21.15.118:0/2171715402 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f241c108ed0 0x7f241c19e4a0 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7f240800e9f0 tx=0x7f240800eeb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:17:26.727 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:26.726+0000 7f2413e00640 1 -- 172.21.15.118:0/2171715402 <== mon.2 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f240800cd80 con 0x7f241c108ed0 2024-03-24T00:17:26.727 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:26.726+0000 7f2413e00640 1 -- 172.21.15.118:0/2171715402 <== mon.2 v2:172.21.15.189:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f2408010910 con 0x7f241c108ed0 2024-03-24T00:17:26.727 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:26.726+0000 7f2422200640 1 -- 172.21.15.118:0/2171715402 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f241c1a5e70 con 0x7f241c108ed0 2024-03-24T00:17:26.727 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:26.726+0000 7f2413e00640 1 -- 172.21.15.118:0/2171715402 <== mon.2 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f2408018990 con 0x7f241c108ed0 2024-03-24T00:17:26.727 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:26.726+0000 7f2422200640 1 -- 172.21.15.118:0/2171715402 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f241c1a6380 con 0x7f241c108ed0 2024-03-24T00:17:26.728 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:26.727+0000 7f2422200640 1 -- 172.21.15.118:0/2171715402 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f241c10efa0 con 0x7f241c108ed0 2024-03-24T00:17:26.731 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:26.732+0000 7f2413e00640 1 -- 172.21.15.118:0/2171715402 <== mon.2 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 20) v1 ==== 137291+0+0 (secure 0 0 0) 0x7f2408010ac0 con 0x7f241c108ed0 2024-03-24T00:17:26.733 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:26.733+0000 7f2413e00640 1 --2- 172.21.15.118:0/2171715402 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7f24000a2880 0x7f24000a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:17:26.733 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:26.734+0000 7f2413e00640 1 -- 172.21.15.118:0/2171715402 <== mon.2 v2:172.21.15.189:3300/0 5 ==== osd_map(90..90 src has 1..90) v4 ==== 8277+0+0 (secure 0 0 0) 0x7f24080ced10 con 0x7f241c108ed0 2024-03-24T00:17:26.734 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:26.734+0000 7f241b400640 1 --2- 172.21.15.118:0/2171715402 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7f24000a2880 0x7f24000a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:17:26.734 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:26.735+0000 7f241b400640 1 --2- 172.21.15.118:0/2171715402 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7f24000a2880 0x7f24000a4d40 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0x7f240c02c180 tx=0x7f240c02cf90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:17:26.737 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:26.738+0000 7f2413e00640 1 -- 172.21.15.118:0/2171715402 <== mon.2 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f240809b730 con 0x7f241c108ed0 2024-03-24T00:17:26.811 INFO:tasks.workunit.client.0.smithi118.stdout:checking for short... yes 2024-03-24T00:17:26.945 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:17:26 smithi118 ceph-mon[31535]: pgmap v450: 97 pgs: 97 active+clean; 1.5 MiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 59 KiB/s rd, 96 KiB/s wr, 46 op/s 2024-03-24T00:17:26.954 INFO:tasks.workunit.client.0.smithi118.stdout:checking size of short... 2 2024-03-24T00:17:27.009 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:27.009+0000 7f2422200640 1 -- 172.21.15.118:0/2171715402 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f241c02d070 con 0x7f241c108ed0 2024-03-24T00:17:27.010 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:27.010+0000 7f2413e00640 1 -- 172.21.15.118:0/2171715402 <== mon.2 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+11478 (secure 0 0 0) 0x7f240809f7d0 con 0x7f241c108ed0 2024-03-24T00:17:27.010 INFO:teuthology.orchestra.run.smithi118.stdout: 2024-03-24T00:17:27.011 INFO:teuthology.orchestra.run.smithi118.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14385,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.118:6835/3194741373","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6834","nonce":3194741373},{"type":"v1","addr":"172.21.15.118:6835","nonce":3194741373}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":4},{"gid":14397,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.118:6837/1374462956","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6836","nonce":1374462956},{"type":"v1","addr":"172.21.15.118:6837","nonce":1374462956}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":28,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-03-24T00:13:01.782474+0000","modified":"2024-03-24T00:15:38.235450+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24569,"mds_1":24550,"mds_2":24511,"mds_3":24499,"mds_4":24488},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14469":{"gid":14469,"name":"a","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.118:6841/3040946537","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6840","nonce":3040946537},{"type":"v1","addr":"172.21.15.118:6841","nonce":3040946537}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24439":{"gid":24439,"name":"l","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.193:6833/1023428412","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6832","nonce":1023428412},{"type":"v1","addr":"172.21.15.193:6833","nonce":1023428412}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24440":{"gid":24440,"name":"k","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.189:6833/2740024078","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6832","nonce":2740024078},{"type":"v1","addr":"172.21.15.189:6833","nonce":2740024078}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24488":{"gid":24488,"name":"e","rank":4,"incarnation":25,"state":"up:active","state_seq":10,"addr":"172.21.15.189:6835/599473205","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6834","nonce":599473205},{"type":"v1","addr":"172.21.15.189:6835","nonce":599473205}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24499":{"gid":24499,"name":"f","rank":3,"incarnation":23,"state":"up:active","state_seq":9,"addr":"172.21.15.193:6835/2055310541","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6834","nonce":2055310541},{"type":"v1","addr":"172.21.15.193:6835","nonce":2055310541}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24511":{"gid":24511,"name":"c","rank":2,"incarnation":21,"state":"up:active","state_seq":8,"addr":"172.21.15.193:6837/2921520053","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6836","nonce":2921520053},{"type":"v1","addr":"172.21.15.193:6837","nonce":2921520053}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24515":{"gid":24515,"name":"h","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.189:6837/3450187137","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6836","nonce":3450187137},{"type":"v1","addr":"172.21.15.189:6837","nonce":3450187137}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24532":{"gid":24532,"name":"i","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.193:6839/2126842603","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6838","nonce":2126842603},{"type":"v1","addr":"172.21.15.193:6839","nonce":2126842603}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24550":{"gid":24550,"name":"g","rank":1,"incarnation":19,"state":"up:active","state_seq":6,"addr":"172.21.15.118:6839/3285788503","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6838","nonce":3285788503},{"type":"v1","addr":"172.21.15.118:6839","nonce":3285788503}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24569":{"gid":24569,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.189:6839/4074484168","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6838","nonce":4074484168},{"type":"v1","addr":"172.21.15.189:6839","nonce":4074484168}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24569,"qdb_cluster":[24569,24499,24488,24511,24550]},"id":1}]} 2024-03-24T00:17:27.011 INFO:teuthology.orchestra.run.smithi118.stderr:dumped fsmap epoch 28 2024-03-24T00:17:27.013 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:27.013+0000 7f2422200640 1 -- 172.21.15.118:0/2171715402 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7f24000a2880 msgr2=0x7f24000a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:17:27.013 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:27.013+0000 7f2422200640 1 --2- 172.21.15.118:0/2171715402 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7f24000a2880 0x7f24000a4d40 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0x7f240c02c180 tx=0x7f240c02cf90 comp rx=0 tx=0).stop 2024-03-24T00:17:27.013 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:27.013+0000 7f2422200640 1 -- 172.21.15.118:0/2171715402 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f241c108ed0 msgr2=0x7f241c19e4a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:17:27.013 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:27.013+0000 7f2422200640 1 --2- 172.21.15.118:0/2171715402 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f241c108ed0 0x7f241c19e4a0 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7f240800e9f0 tx=0x7f240800eeb0 comp rx=0 tx=0).stop 2024-03-24T00:17:27.013 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:27.013+0000 7f2422200640 1 -- 172.21.15.118:0/2171715402 shutdown_connections 2024-03-24T00:17:27.013 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:27.013+0000 7f2422200640 1 --2- 172.21.15.118:0/2171715402 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7f24000a2880 0x7f24000a4d40 unknown :-1 s=CLOSED pgs=297 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:17:27.014 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:27.013+0000 7f2422200640 1 --2- 172.21.15.118:0/2171715402 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f241c109890 0x7f241c1a5510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:17:27.014 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:27.013+0000 7f2422200640 1 --2- 172.21.15.118:0/2171715402 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f241c108ed0 0x7f241c19e4a0 unknown :-1 s=CLOSED pgs=279 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:17:27.014 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:27.013+0000 7f2422200640 1 --2- 172.21.15.118:0/2171715402 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f241c107ce0 0x7f241c19df60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:17:27.014 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:27.013+0000 7f2422200640 1 -- 172.21.15.118:0/2171715402 >> 172.21.15.118:0/2171715402 conn(0x7f241c076cb0 msgr2=0x7f241c10c170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:17:27.014 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:27.014+0000 7f2422200640 1 -- 172.21.15.118:0/2171715402 shutdown_connections 2024-03-24T00:17:27.014 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:27.014+0000 7f2422200640 1 -- 172.21.15.118:0/2171715402 wait complete. 2024-03-24T00:17:27.031 INFO:teuthology.orchestra.run:Running command with timeout 120 2024-03-24T00:17:27.031 DEBUG:teuthology.orchestra.run.smithi118:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:4 damage ls 2024-03-24T00:17:27.080 INFO:tasks.workunit.client.0.smithi118.stdout:checking for int... yes 2024-03-24T00:17:27.203 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:17:26 smithi189 ceph-mon[36187]: pgmap v450: 97 pgs: 97 active+clean; 1.5 MiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 59 KiB/s rd, 96 KiB/s wr, 46 op/s 2024-03-24T00:17:27.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:17:26 smithi193 ceph-mon[36878]: pgmap v450: 97 pgs: 97 active+clean; 1.5 MiB data, 351 MiB used, 1.0 TiB / 1.0 TiB avail; 59 KiB/s rd, 96 KiB/s wr, 46 op/s 2024-03-24T00:17:27.227 INFO:tasks.workunit.client.0.smithi118.stdout:checking size of int... 4 2024-03-24T00:17:27.268 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:27.267+0000 7f673e800640 1 -- 172.21.15.118:0/2933017155 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6738074100 msgr2=0x7f6738074580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:17:27.268 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:27.267+0000 7f673e800640 1 --2- 172.21.15.118:0/2933017155 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6738074100 0x7f6738074580 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7f6724009900 tx=0x7f672402bc30 comp rx=0 tx=0).stop 2024-03-24T00:17:27.268 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:27.267+0000 7f673e800640 1 -- 172.21.15.118:0/2933017155 shutdown_connections 2024-03-24T00:17:27.269 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:27.267+0000 7f673e800640 1 --2- 172.21.15.118:0/2933017155 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f6738109460 0x7f673810b8f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:17:27.269 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:27.267+0000 7f673e800640 1 --2- 172.21.15.118:0/2933017155 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6738074100 0x7f6738074580 unknown :-1 s=CLOSED pgs=280 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:17:27.269 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:27.267+0000 7f673e800640 1 --2- 172.21.15.118:0/2933017155 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f67380756e0 0x7f6738073b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:17:27.269 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:27.267+0000 7f673e800640 1 -- 172.21.15.118:0/2933017155 >> 172.21.15.118:0/2933017155 conn(0x7f67380fcb10 msgr2=0x7f67380fef50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:17:27.269 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:27.267+0000 7f673e800640 1 -- 172.21.15.118:0/2933017155 shutdown_connections 2024-03-24T00:17:27.269 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:27.267+0000 7f673e800640 1 -- 172.21.15.118:0/2933017155 wait complete. 2024-03-24T00:17:27.271 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:27.268+0000 7f673e800640 1 Processor -- start 2024-03-24T00:17:27.271 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:27.268+0000 7f673e800640 1 -- start start 2024-03-24T00:17:27.271 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:27.269+0000 7f673e800640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f6738074100 0x7f6738199a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:17:27.271 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:27.269+0000 7f673e800640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f67380756e0 0x7f6738199fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:17:27.271 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:27.269+0000 7f673e800640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f6738109460 0x7f67381a1010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:17:27.271 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:27.269+0000 7f673e800640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f673810f450 con 0x7f6738109460 2024-03-24T00:17:27.271 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:27.269+0000 7f673e800640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f673810f2d0 con 0x7f67380756e0 2024-03-24T00:17:27.271 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:27.269+0000 7f673e800640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f673810f5d0 con 0x7f6738074100 2024-03-24T00:17:27.271 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:27.269+0000 7f673d400640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f6738074100 0x7f6738199a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:17:27.271 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:27.269+0000 7f673ca00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f67380756e0 0x7f6738199fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:17:27.272 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:27.269+0000 7f673d400640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f6738074100 0x7f6738199a60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.118:37682/0 (socket says 172.21.15.118:37682) 2024-03-24T00:17:27.272 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:27.269+0000 7f673d400640 1 -- 172.21.15.118:0/2304797307 learned_addr learned my addr 172.21.15.118:0/2304797307 (peer_addr_for_me v2:172.21.15.118:0/0) 2024-03-24T00:17:27.272 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:27.269+0000 7f673de00640 1 --2- 172.21.15.118:0/2304797307 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f6738109460 0x7f67381a1010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:17:27.272 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:27.269+0000 7f673ca00640 1 -- 172.21.15.118:0/2304797307 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f6738074100 msgr2=0x7f6738199a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:17:27.272 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:27.269+0000 7f673ca00640 1 --2- 172.21.15.118:0/2304797307 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f6738074100 0x7f6738199a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:17:27.272 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:27.269+0000 7f673ca00640 1 -- 172.21.15.118:0/2304797307 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f6738109460 msgr2=0x7f67381a1010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:17:27.272 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:27.269+0000 7f673ca00640 1 --2- 172.21.15.118:0/2304797307 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f6738109460 0x7f67381a1010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:17:27.272 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:27.269+0000 7f673ca00640 1 -- 172.21.15.118:0/2304797307 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f67381a1680 con 0x7f67380756e0 2024-03-24T00:17:27.272 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:27.270+0000 7f673ca00640 1 --2- 172.21.15.118:0/2304797307 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f67380756e0 0x7f6738199fa0 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7f6724036040 tx=0x7f672402c6c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:17:27.272 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:27.270+0000 7f6736000640 1 -- 172.21.15.118:0/2304797307 <== mon.2 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6724043020 con 0x7f67380756e0 2024-03-24T00:17:27.272 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:27.270+0000 7f6736000640 1 -- 172.21.15.118:0/2304797307 <== mon.2 v2:172.21.15.189:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f67240054e0 con 0x7f67380756e0 2024-03-24T00:17:27.272 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:27.270+0000 7f673e800640 1 -- 172.21.15.118:0/2304797307 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f67381a1910 con 0x7f67380756e0 2024-03-24T00:17:27.272 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:27.270+0000 7f6736000640 1 -- 172.21.15.118:0/2304797307 <== mon.2 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6724003ca0 con 0x7f67380756e0 2024-03-24T00:17:27.272 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:27.270+0000 7f673e800640 1 -- 172.21.15.118:0/2304797307 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f67381a1e80 con 0x7f67380756e0 2024-03-24T00:17:27.274 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:27.273+0000 7f6742bd5740 1 Processor -- start 2024-03-24T00:17:27.274 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:27.273+0000 7f6742bd5740 1 -- start start 2024-03-24T00:17:27.275 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:27.274+0000 7f6742bd5740 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5606205959f0 0x560620595e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:17:27.275 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:27.274+0000 7f6742bd5740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x560620596bc0 0x560620597040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:17:27.275 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:27.274+0000 7f6742bd5740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x560620597580 0x56062059de10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:17:27.276 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:27.274+0000 7f6742bd5740 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x5606204fa2c0 con 0x5606205959f0 2024-03-24T00:17:27.276 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:27.274+0000 7f6742bd5740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x5606204a2c90 con 0x560620596bc0 2024-03-24T00:17:27.276 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:27.274+0000 7f6742bd5740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x5606204f6360 con 0x560620597580 2024-03-24T00:17:27.276 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:27.274+0000 7f673d400640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x560620596bc0 0x560620597040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:17:27.276 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:27.274+0000 7f673d400640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x560620596bc0 0x560620597040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.118:45312/0 (socket says 172.21.15.118:45312) 2024-03-24T00:17:27.276 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:27.274+0000 7f673d400640 1 -- 172.21.15.118:0/3976368417 learned_addr learned my addr 172.21.15.118:0/3976368417 (peer_addr_for_me v2:172.21.15.118:0/0) 2024-03-24T00:17:27.276 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:27.274+0000 7f673de00640 1 --2- 172.21.15.118:0/3976368417 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5606205959f0 0x560620595e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:17:27.276 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:27.274+0000 7f673d400640 1 -- 172.21.15.118:0/3976368417 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x560620597580 msgr2=0x56062059de10 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-24T00:17:27.276 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:27.274+0000 7f673d400640 1 --2- 172.21.15.118:0/3976368417 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x560620597580 0x56062059de10 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:17:27.276 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:27.274+0000 7f673d400640 1 -- 172.21.15.118:0/3976368417 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5606205959f0 msgr2=0x560620595e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:17:27.276 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:27.274+0000 7f673d400640 1 --2- 172.21.15.118:0/3976368417 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5606205959f0 0x560620595e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:17:27.276 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:27.274+0000 7f673d400640 1 -- 172.21.15.118:0/3976368417 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5606204c7080 con 0x560620596bc0 2024-03-24T00:17:27.276 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:27.274+0000 7f673de00640 1 --2- 172.21.15.118:0/3976368417 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5606205959f0 0x560620595e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T00:17:27.276 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:27.275+0000 7f673d400640 1 --2- 172.21.15.118:0/3976368417 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x560620596bc0 0x560620597040 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7f672c00bda0 tx=0x7f672c0107a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e4b18244f0fcb3aa server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:17:27.276 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:27.275+0000 7f6736000640 1 -- 172.21.15.118:0/2304797307 <== mon.2 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 20) v1 ==== 137291+0+0 (secure 0 0 0) 0x7f6724046020 con 0x7f67380756e0 2024-03-24T00:17:27.277 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:27.275+0000 7f672a600640 1 -- 172.21.15.118:0/3976368417 <== mon.2 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f672c01c680 con 0x560620596bc0 2024-03-24T00:17:27.277 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:27.275+0000 7f672a600640 1 -- 172.21.15.118:0/3976368417 <== mon.2 v2:172.21.15.189:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f672c011920 con 0x560620596bc0 2024-03-24T00:17:27.277 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:27.275+0000 7f672a600640 1 -- 172.21.15.118:0/3976368417 <== mon.2 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f672c01a930 con 0x560620596bc0 2024-03-24T00:17:27.280 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:27.275+0000 7f6742bd5740 1 -- 172.21.15.118:0/3976368417 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x560620596bc0 msgr2=0x560620597040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:17:27.280 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:27.276+0000 7f6742bd5740 1 --2- 172.21.15.118:0/3976368417 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x560620596bc0 0x560620597040 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7f672c00bda0 tx=0x7f672c0107a0 comp rx=0 tx=0).stop 2024-03-24T00:17:27.280 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:27.276+0000 7f6742bd5740 1 -- 172.21.15.118:0/3976368417 shutdown_connections 2024-03-24T00:17:27.280 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:27.276+0000 7f6742bd5740 1 --2- 172.21.15.118:0/3976368417 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x560620597580 0x56062059de10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:17:27.281 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:27.276+0000 7f6742bd5740 1 --2- 172.21.15.118:0/3976368417 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x560620596bc0 0x560620597040 unknown :-1 s=CLOSED pgs=282 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:17:27.281 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:27.276+0000 7f6742bd5740 1 --2- 172.21.15.118:0/3976368417 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5606205959f0 0x560620595e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:17:27.281 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:27.276+0000 7f6742bd5740 1 -- 172.21.15.118:0/3976368417 >> 172.21.15.118:0/3976368417 conn(0x560620590de0 msgr2=0x5606205911e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:17:27.281 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:27.276+0000 7f6742bd5740 1 -- 172.21.15.118:0/3976368417 shutdown_connections 2024-03-24T00:17:27.281 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:27.276+0000 7f6742bd5740 1 -- 172.21.15.118:0/3976368417 wait complete. 2024-03-24T00:17:27.281 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:27.276+0000 7f6742bd5740 1 -- start start 2024-03-24T00:17:27.281 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:27.276+0000 7f6742bd5740 1 Processor -- start 2024-03-24T00:17:27.281 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:27.277+0000 7f6742bd5740 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5606205959f0 0x56062059fcc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:17:27.281 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:27.277+0000 7f6742bd5740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x560620596bc0 0x5606205a0200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:17:27.281 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:27.277+0000 7f6742bd5740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x560620597580 0x5606205a7240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:17:27.281 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:27.277+0000 7f6742bd5740 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x56062059e590 con 0x5606205959f0 2024-03-24T00:17:27.281 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:27.277+0000 7f6742bd5740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x5606204b60d0 con 0x560620596bc0 2024-03-24T00:17:27.281 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:27.277+0000 7f6742bd5740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x56062059e710 con 0x560620597580 2024-03-24T00:17:27.281 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:27.277+0000 7f6742bd5740 10 client.0 _ll_register_callbacks cb 0x5606204bae80 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T00:17:27.282 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:27.277+0000 7f6736000640 1 --2- 172.21.15.118:0/2304797307 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7f67200a2790 0x7f67200a4c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:17:27.282 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:27.277+0000 7f673de00640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5606205959f0 0x56062059fcc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:17:27.282 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:27.277+0000 7f673ca00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x560620597580 0x5606205a7240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:17:27.282 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:27.277+0000 7f673ca00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x560620597580 0x5606205a7240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.118:37692/0 (socket says 172.21.15.118:37692) 2024-03-24T00:17:27.282 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:27.277+0000 7f673ca00640 1 -- 172.21.15.118:0/3873532401 learned_addr learned my addr 172.21.15.118:0/3873532401 (peer_addr_for_me v2:172.21.15.118:0/0) 2024-03-24T00:17:27.282 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:27.277+0000 7f673d400640 1 --2- 172.21.15.118:0/3873532401 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x560620596bc0 0x5606205a0200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:17:27.282 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:27.277+0000 7f6736000640 1 -- 172.21.15.118:0/2304797307 <== mon.2 v2:172.21.15.189:3300/0 5 ==== osd_map(90..90 src has 1..90) v4 ==== 8277+0+0 (secure 0 0 0) 0x7f67240f0a60 con 0x7f67380756e0 2024-03-24T00:17:27.282 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:27.278+0000 7f673de00640 1 --2- 172.21.15.118:0/2304797307 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7f67200a2790 0x7f67200a4c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:17:27.282 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:27.278+0000 7f673d400640 1 -- 172.21.15.118:0/3873532401 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x560620597580 msgr2=0x5606205a7240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:17:27.282 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:27.278+0000 7f673d400640 1 --2- 172.21.15.118:0/3873532401 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x560620597580 0x5606205a7240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:17:27.282 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:27.278+0000 7f673d400640 1 -- 172.21.15.118:0/3873532401 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5606205959f0 msgr2=0x56062059fcc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:17:27.282 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:27.278+0000 7f673d400640 1 --2- 172.21.15.118:0/3873532401 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5606205959f0 0x56062059fcc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:17:27.282 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:27.278+0000 7f673d400640 1 -- 172.21.15.118:0/3873532401 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x56062020e3f0 con 0x560620596bc0 2024-03-24T00:17:27.283 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:27.278+0000 7f673d400640 1 --2- 172.21.15.118:0/3873532401 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x560620596bc0 0x5606205a0200 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7f672c011010 tx=0x7f672c011ce0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=1498df62ae809266 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:17:27.283 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:27.278+0000 7f671ea00640 10 client.0 ms_handle_connect on v2:172.21.15.189:3300/0 2024-03-24T00:17:27.283 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:27.279+0000 7f671ea00640 1 -- 172.21.15.118:0/3873532401 <== mon.2 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f672c004040 con 0x560620596bc0 2024-03-24T00:17:27.283 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:27.279+0000 7f671ea00640 1 -- 172.21.15.118:0/3873532401 <== mon.2 v2:172.21.15.189:3300/0 2 ==== osd_map(90..90 src has 1..90) v4 ==== 8277+0+0 (secure 0 0 0) 0x7f672c0041e0 con 0x560620596bc0 2024-03-24T00:17:27.283 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:27.279+0000 7f6742bd5740 1 -- 172.21.15.118:0/3873532401 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f672c011920 con 0x560620596bc0 2024-03-24T00:17:27.283 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:27.279+0000 7f671ea00640 1 -- 172.21.15.118:0/3873532401 <== mon.2 v2:172.21.15.189:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) v2 ==== 24+0+0 (secure 0 0 0) 0x7f672c0043e0 con 0x560620596bc0 2024-03-24T00:17:27.283 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:27.279+0000 7f6742bd5740 10 client.25847 fetch_fsmap learned FSMap version 28 2024-03-24T00:17:27.283 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:27.279+0000 7f6742bd5740 1 -- 172.21.15.118:0/3873532401 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) v3 -- 0x5606202d87d0 con 0x560620596bc0 2024-03-24T00:17:27.283 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:27.279+0000 7f673de00640 1 --2- 172.21.15.118:0/2304797307 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7f67200a2790 0x7f67200a4c50 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 crypto rx=0x7f6730009fd0 tx=0x7f6730009500 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:17:27.283 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:27.279+0000 7f671ea00640 1 -- 172.21.15.118:0/3873532401 <== mon.2 v2:172.21.15.189:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f672c011920 con 0x560620596bc0 2024-03-24T00:17:27.283 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:27.279+0000 7f671ea00640 1 -- 172.21.15.118:0/3873532401 <== mon.2 v2:172.21.15.189:3300/0 5 ==== fsmap(e 28) v1 ==== 7934+0+0 (secure 0 0 0) 0x7f672c024070 con 0x560620596bc0 2024-03-24T00:17:27.283 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:27.280+0000 7f6742bd5740 10 client.25847 fetch_fsmap finished waiting for FSMap version 28 2024-03-24T00:17:27.283 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:27.280+0000 7f6742bd5740 10 client.25847 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.e 2024-03-24T00:17:27.283 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:27.280+0000 7f6742bd5740 20 client.25847 populate_metadata read hostname 'smithi118' 2024-03-24T00:17:27.284 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:27.280+0000 7f6742bd5740 1 --2- 172.21.15.118:0/3873532401 >> [v2:172.21.15.189:6834/599473205,v1:172.21.15.189:6835/599473205] conn(0x5606205ab3a0 0x5606205ad790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:17:27.284 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:27.280+0000 7f6742bd5740 4 client.25847 mds_command: new command op to 24488 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-24T00:17:27.284 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:27.280+0000 7f6742bd5740 1 -- 172.21.15.118:0/3873532401 --> [v2:172.21.15.189:6834/599473205,v1:172.21.15.189:6835/599473205] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x560620507800 con 0x5606205ab3a0 2024-03-24T00:17:27.284 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:27.280+0000 7f673ca00640 1 --2- 172.21.15.118:0/3873532401 >> [v2:172.21.15.189:6834/599473205,v1:172.21.15.189:6835/599473205] conn(0x5606205ab3a0 0x5606205ad790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:17:27.285 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:27.280+0000 7f673ca00640 1 --2- 172.21.15.118:0/3873532401 >> [v2:172.21.15.189:6834/599473205,v1:172.21.15.189:6835/599473205] conn(0x5606205ab3a0 0x5606205ad790 crc :-1 s=READY pgs=26 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=8fa510bf7d4b34be server_cookie=c1d6ff3c0cc4d2aa in_seq=0 out_seq=0 2024-03-24T00:17:27.285 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:27.280+0000 7f671ea00640 10 client.25847 ms_handle_connect on v2:172.21.15.189:6834/599473205 2024-03-24T00:17:27.285 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:27.281+0000 7f671ea00640 1 -- 172.21.15.118:0/3873532401 <== mds.4 v2:172.21.15.189:6834/599473205 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x560620507800 con 0x5606205ab3a0 2024-03-24T00:17:27.285 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:27.281+0000 7f671ea00640 10 client.25847 handle_command_reply: tid=0 2024-03-24T00:17:27.285 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:27.281+0000 7f6742bd5740 1 client.25847 shutdown 2024-03-24T00:17:27.285 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:27.282+0000 7f6742bd5740 1 -- 172.21.15.118:0/3873532401 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x560620596bc0 msgr2=0x5606205a0200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:17:27.285 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:27.282+0000 7f6742bd5740 1 --2- 172.21.15.118:0/3873532401 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x560620596bc0 0x5606205a0200 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7f672c011010 tx=0x7f672c011ce0 comp rx=0 tx=0).stop 2024-03-24T00:17:27.285 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:27.282+0000 7f6742bd5740 1 -- 172.21.15.118:0/3873532401 shutdown_connections 2024-03-24T00:17:27.286 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:27.282+0000 7f6742bd5740 1 --2- 172.21.15.118:0/3873532401 >> [v2:172.21.15.189:6834/599473205,v1:172.21.15.189:6835/599473205] conn(0x5606205ab3a0 0x5606205ad790 crc :-1 s=READY pgs=26 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:17:27.286 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:27.282+0000 7f6742bd5740 1 --2- 172.21.15.118:0/3873532401 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x560620597580 0x5606205a7240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:17:27.286 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:27.282+0000 7f671ea00640 0 client.25847 ms_handle_reset on v2:172.21.15.189:6834/599473205 2024-03-24T00:17:27.286 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:27.282+0000 7f6742bd5740 1 --2- 172.21.15.118:0/3873532401 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x560620596bc0 0x5606205a0200 unknown :-1 s=CLOSED pgs=283 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:17:27.286 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:27.282+0000 7f6742bd5740 1 --2- 172.21.15.118:0/3873532401 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5606205959f0 0x56062059fcc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:17:27.286 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:27.282+0000 7f6742bd5740 1 -- 172.21.15.118:0/3873532401 >> 172.21.15.118:0/3873532401 conn(0x560620590de0 msgr2=0x560620590160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:17:27.286 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:27.282+0000 7f6742bd5740 1 -- 172.21.15.118:0/3873532401 shutdown_connections 2024-03-24T00:17:27.286 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:27.282+0000 7f6742bd5740 1 -- 172.21.15.118:0/3873532401 wait complete. 2024-03-24T00:17:27.286 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:27.282+0000 7f6742bd5740 20 client.25847 trim_cache size 0 max 16384 2024-03-24T00:17:27.297 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:27.296+0000 7f6742bd5740 1 Processor -- start 2024-03-24T00:17:27.297 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:27.296+0000 7f6742bd5740 1 -- start start 2024-03-24T00:17:27.297 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:27.296+0000 7f6742bd5740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5606205959f0 0x560620599c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:17:27.297 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:27.296+0000 7f6742bd5740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x560620596bc0 0x5606205a7200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:17:27.297 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:27.296+0000 7f6742bd5740 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x560620597580 0x56062059a1a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:17:27.298 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:27.296+0000 7f6742bd5740 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f67381aa9f0 con 0x560620597580 2024-03-24T00:17:27.298 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:27.296+0000 7f6742bd5740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f67381ac370 con 0x560620596bc0 2024-03-24T00:17:27.298 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:27.296+0000 7f6742bd5740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f67381ab050 con 0x5606205959f0 2024-03-24T00:17:27.298 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:27.297+0000 7f673ca00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x560620596bc0 0x5606205a7200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:17:27.298 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:27.297+0000 7f673d400640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5606205959f0 0x560620599c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:17:27.298 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:27.297+0000 7f673de00640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x560620597580 0x56062059a1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:17:27.298 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:27.297+0000 7f673ca00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x560620596bc0 0x5606205a7200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.118:45336/0 (socket says 172.21.15.118:45336) 2024-03-24T00:17:27.298 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:27.297+0000 7f673ca00640 1 -- 172.21.15.118:0/1339363395 learned_addr learned my addr 172.21.15.118:0/1339363395 (peer_addr_for_me v2:172.21.15.118:0/0) 2024-03-24T00:17:27.299 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:27.297+0000 7f673ca00640 1 -- 172.21.15.118:0/1339363395 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5606205959f0 msgr2=0x560620599c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:17:27.300 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:27.297+0000 7f673ca00640 1 --2- 172.21.15.118:0/1339363395 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5606205959f0 0x560620599c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:17:27.300 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:27.297+0000 7f673ca00640 1 -- 172.21.15.118:0/1339363395 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x560620597580 msgr2=0x56062059a1a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:17:27.300 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:27.297+0000 7f673ca00640 1 --2- 172.21.15.118:0/1339363395 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x560620597580 0x56062059a1a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:17:27.300 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:27.297+0000 7f673ca00640 1 -- 172.21.15.118:0/1339363395 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x56062058c8b0 con 0x560620596bc0 2024-03-24T00:17:27.300 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:27.297+0000 7f673d400640 1 --2- 172.21.15.118:0/1339363395 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5606205959f0 0x560620599c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T00:17:27.300 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:27.297+0000 7f673ca00640 1 --2- 172.21.15.118:0/1339363395 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x560620596bc0 0x5606205a7200 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7f672402c170 tx=0x7f672403ac10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f5506c3d10a64e66 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:17:27.300 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:27.298+0000 7f6729c00640 1 -- 172.21.15.118:0/1339363395 <== mon.2 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6724003e40 con 0x560620596bc0 2024-03-24T00:17:27.300 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:27.298+0000 7f6729c00640 1 -- 172.21.15.118:0/1339363395 <== mon.2 v2:172.21.15.189:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f6724004480 con 0x560620596bc0 2024-03-24T00:17:27.300 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:27.298+0000 7f6729c00640 1 -- 172.21.15.118:0/1339363395 <== mon.2 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f67240bb6d0 con 0x560620596bc0 2024-03-24T00:17:27.300 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:27.299+0000 7f6742bd5740 1 -- 172.21.15.118:0/1339363395 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x560620596bc0 msgr2=0x5606205a7200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:17:27.300 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:27.299+0000 7f6742bd5740 1 --2- 172.21.15.118:0/1339363395 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x560620596bc0 0x5606205a7200 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7f672402c170 tx=0x7f672403ac10 comp rx=0 tx=0).stop 2024-03-24T00:17:27.300 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:27.299+0000 7f6742bd5740 1 -- 172.21.15.118:0/1339363395 shutdown_connections 2024-03-24T00:17:27.300 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:27.299+0000 7f6742bd5740 1 --2- 172.21.15.118:0/1339363395 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x560620597580 0x56062059a1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:17:27.300 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:27.299+0000 7f6742bd5740 1 --2- 172.21.15.118:0/1339363395 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x560620596bc0 0x5606205a7200 unknown :-1 s=CLOSED pgs=284 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:17:27.301 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:27.299+0000 7f6742bd5740 1 --2- 172.21.15.118:0/1339363395 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5606205959f0 0x560620599c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:17:27.301 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:27.299+0000 7f6742bd5740 1 -- 172.21.15.118:0/1339363395 >> 172.21.15.118:0/1339363395 conn(0x560620590de0 msgr2=0x5606205948d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:17:27.301 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:27.299+0000 7f6742bd5740 1 -- 172.21.15.118:0/1339363395 shutdown_connections 2024-03-24T00:17:27.301 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:27.299+0000 7f6742bd5740 1 -- 172.21.15.118:0/1339363395 wait complete. 2024-03-24T00:17:27.303 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:27.299+0000 7f6742bd5740 1 -- start start 2024-03-24T00:17:27.303 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:27.300+0000 7f6742bd5740 1 Processor -- start 2024-03-24T00:17:27.303 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:27.300+0000 7f6742bd5740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5606205959f0 0x5606205a30f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:17:27.303 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:27.300+0000 7f6742bd5740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x560620596bc0 0x5606205a3630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:17:27.304 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:27.300+0000 7f6742bd5740 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x560620597580 0x5606205a6600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:17:27.304 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:27.300+0000 7f6742bd5740 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f67381abd10 con 0x560620597580 2024-03-24T00:17:27.304 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:27.300+0000 7f6742bd5740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f67381ab6b0 con 0x5606205959f0 2024-03-24T00:17:27.304 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:27.300+0000 7f6742bd5740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f67381a8bb0 con 0x560620596bc0 2024-03-24T00:17:27.304 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:27.300+0000 7f6742bd5740 10 client.0 _ll_register_callbacks cb 0x7f672c00bbf0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T00:17:27.304 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:27.301+0000 7f673ca00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x560620596bc0 0x5606205a3630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:17:27.304 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:27.301+0000 7f673d400640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5606205959f0 0x5606205a30f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:17:27.304 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:27.301+0000 7f673ca00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x560620596bc0 0x5606205a3630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.118:37698/0 (socket says 172.21.15.118:37698) 2024-03-24T00:17:27.304 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:27.301+0000 7f673ca00640 1 -- 172.21.15.118:0/1392708780 learned_addr learned my addr 172.21.15.118:0/1392708780 (peer_addr_for_me v2:172.21.15.118:0/0) 2024-03-24T00:17:27.304 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:27.301+0000 7f673de00640 1 --2- 172.21.15.118:0/1392708780 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x560620597580 0x5606205a6600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:17:27.304 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:27.301+0000 7f673ca00640 1 -- 172.21.15.118:0/1392708780 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5606205959f0 msgr2=0x5606205a30f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:17:27.304 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:27.301+0000 7f673ca00640 1 --2- 172.21.15.118:0/1392708780 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5606205959f0 0x5606205a30f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:17:27.304 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:27.301+0000 7f673ca00640 1 -- 172.21.15.118:0/1392708780 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x560620597580 msgr2=0x5606205a6600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:17:27.304 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:27.301+0000 7f673ca00640 1 --2- 172.21.15.118:0/1392708780 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x560620597580 0x5606205a6600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:17:27.305 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:27.301+0000 7f673ca00640 1 -- 172.21.15.118:0/1392708780 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5606205a6bf0 con 0x560620596bc0 2024-03-24T00:17:27.305 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:27.301+0000 7f673de00640 1 --2- 172.21.15.118:0/1392708780 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x560620597580 0x5606205a6600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T00:17:27.305 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:27.301+0000 7f673ca00640 1 --2- 172.21.15.118:0/1392708780 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x560620596bc0 0x5606205a3630 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7f672402c7b0 tx=0x7f67240c1590 comp rx=0 tx=0).ready entity=mon.1 client_cookie=4e9f3e3eb6e350cf server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:17:27.305 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:27.301+0000 7f671ea00640 10 client.0 ms_handle_connect on v2:172.21.15.193:3300/0 2024-03-24T00:17:27.305 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:27.301+0000 7f671ea00640 1 -- 172.21.15.118:0/1392708780 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f67240bd3d0 con 0x560620596bc0 2024-03-24T00:17:27.305 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:27.302+0000 7f6742bd5740 1 -- 172.21.15.118:0/1392708780 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x560620501b50 con 0x560620596bc0 2024-03-24T00:17:27.305 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:27.302+0000 7f671ea00640 1 -- 172.21.15.118:0/1392708780 <== mon.1 v2:172.21.15.193:3300/0 2 ==== osd_map(90..90 src has 1..90) v4 ==== 8277+0+0 (secure 0 0 0) 0x7f67240bb6d0 con 0x560620596bc0 2024-03-24T00:17:27.305 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:27.302+0000 7f671ea00640 1 -- 172.21.15.118:0/1392708780 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) v2 ==== 24+0+0 (secure 0 0 0) 0x7f67240d6260 con 0x560620596bc0 2024-03-24T00:17:27.305 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:27.302+0000 7f6742bd5740 10 client.25837 fetch_fsmap learned FSMap version 28 2024-03-24T00:17:27.305 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:27.302+0000 7f6742bd5740 1 -- 172.21.15.118:0/1392708780 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) v3 -- 0x560620607280 con 0x560620596bc0 2024-03-24T00:17:27.305 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:27.302+0000 7f671ea00640 1 -- 172.21.15.118:0/1392708780 <== mon.1 v2:172.21.15.193:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x560620501b50 con 0x560620596bc0 2024-03-24T00:17:27.305 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:27.302+0000 7f671ea00640 1 -- 172.21.15.118:0/1392708780 <== mon.1 v2:172.21.15.193:3300/0 5 ==== fsmap(e 28) v1 ==== 7934+0+0 (secure 0 0 0) 0x7f67240c1150 con 0x560620596bc0 2024-03-24T00:17:27.305 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:27.302+0000 7f6742bd5740 10 client.25837 fetch_fsmap finished waiting for FSMap version 28 2024-03-24T00:17:27.305 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:27.302+0000 7f6742bd5740 10 client.25837 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.e 2024-03-24T00:17:27.306 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:27.302+0000 7f6742bd5740 20 client.25837 populate_metadata read hostname 'smithi118' 2024-03-24T00:17:27.306 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:27.302+0000 7f6742bd5740 1 --2- 172.21.15.118:0/1392708780 >> [v2:172.21.15.189:6834/599473205,v1:172.21.15.189:6835/599473205] conn(0x5606205ab3a0 0x560620609ae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:17:27.306 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:27.302+0000 7f6742bd5740 4 client.25837 mds_command: new command op to 24488 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-24T00:17:27.306 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:27.302+0000 7f6742bd5740 1 -- 172.21.15.118:0/1392708780 --> [v2:172.21.15.189:6834/599473205,v1:172.21.15.189:6835/599473205] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x56062047f500 con 0x5606205ab3a0 2024-03-24T00:17:27.306 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:27.302+0000 7f673d400640 1 --2- 172.21.15.118:0/1392708780 >> [v2:172.21.15.189:6834/599473205,v1:172.21.15.189:6835/599473205] conn(0x5606205ab3a0 0x560620609ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:17:27.306 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:27.303+0000 7f671ea00640 10 client.25837 ms_handle_connect on v2:172.21.15.189:6834/599473205 2024-03-24T00:17:27.306 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:27.303+0000 7f673d400640 1 --2- 172.21.15.118:0/1392708780 >> [v2:172.21.15.189:6834/599473205,v1:172.21.15.189:6835/599473205] conn(0x5606205ab3a0 0x560620609ae0 crc :-1 s=READY pgs=27 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=55adf6c0c99205d server_cookie=e65b97c9f529fcd1 in_seq=0 out_seq=0 2024-03-24T00:17:27.307 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:27.303+0000 7f671ea00640 1 -- 172.21.15.118:0/1392708780 <== mds.4 v2:172.21.15.189:6834/599473205 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x56062047f500 con 0x5606205ab3a0 2024-03-24T00:17:27.307 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:27.303+0000 7f671ea00640 10 client.25837 handle_command_reply: tid=0 2024-03-24T00:17:27.307 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:27.303+0000 7f6742bd5740 1 client.25837 shutdown 2024-03-24T00:17:27.307 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:27.303+0000 7f6742bd5740 1 -- 172.21.15.118:0/1392708780 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x560620596bc0 msgr2=0x5606205a3630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:17:27.307 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:27.303+0000 7f6742bd5740 1 --2- 172.21.15.118:0/1392708780 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x560620596bc0 0x5606205a3630 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7f672402c7b0 tx=0x7f67240c1590 comp rx=0 tx=0).stop 2024-03-24T00:17:27.308 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:27.303+0000 7f6742bd5740 1 -- 172.21.15.118:0/1392708780 shutdown_connections 2024-03-24T00:17:27.308 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:27.303+0000 7f6742bd5740 1 --2- 172.21.15.118:0/1392708780 >> [v2:172.21.15.189:6834/599473205,v1:172.21.15.189:6835/599473205] conn(0x5606205ab3a0 0x560620609ae0 crc :-1 s=READY pgs=27 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:17:27.308 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:27.303+0000 7f6742bd5740 1 --2- 172.21.15.118:0/1392708780 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x560620597580 0x5606205a6600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:17:27.308 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:27.303+0000 7f6742bd5740 1 --2- 172.21.15.118:0/1392708780 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x560620596bc0 0x5606205a3630 unknown :-1 s=CLOSED pgs=269 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:17:27.308 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:27.303+0000 7f671ea00640 0 client.25837 ms_handle_reset on v2:172.21.15.189:6834/599473205 2024-03-24T00:17:27.308 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:27.303+0000 7f6742bd5740 1 --2- 172.21.15.118:0/1392708780 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5606205959f0 0x5606205a30f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:17:27.308 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:27.303+0000 7f6742bd5740 1 -- 172.21.15.118:0/1392708780 >> 172.21.15.118:0/1392708780 conn(0x560620590de0 msgr2=0x56062058f8d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:17:27.308 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:27.304+0000 7f6742bd5740 1 -- 172.21.15.118:0/1392708780 shutdown_connections 2024-03-24T00:17:27.308 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:27.304+0000 7f6742bd5740 1 -- 172.21.15.118:0/1392708780 wait complete. 2024-03-24T00:17:27.308 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:27.304+0000 7f6742bd5740 20 client.25837 trim_cache size 0 max 16384 2024-03-24T00:17:27.308 INFO:teuthology.orchestra.run.smithi118.stdout:[] 2024-03-24T00:17:27.309 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:27.304+0000 7f673e800640 1 -- 172.21.15.118:0/2304797307 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7f67200a2790 msgr2=0x7f67200a4c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:17:27.309 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:27.304+0000 7f673e800640 1 --2- 172.21.15.118:0/2304797307 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7f67200a2790 0x7f67200a4c50 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 crypto rx=0x7f6730009fd0 tx=0x7f6730009500 comp rx=0 tx=0).stop 2024-03-24T00:17:27.309 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:27.304+0000 7f673e800640 1 -- 172.21.15.118:0/2304797307 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f67380756e0 msgr2=0x7f6738199fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:17:27.309 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:27.304+0000 7f673e800640 1 --2- 172.21.15.118:0/2304797307 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f67380756e0 0x7f6738199fa0 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7f6724036040 tx=0x7f672402c6c0 comp rx=0 tx=0).stop 2024-03-24T00:17:27.309 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:27.305+0000 7f673e800640 1 -- 172.21.15.118:0/2304797307 shutdown_connections 2024-03-24T00:17:27.309 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:27.305+0000 7f673e800640 1 --2- 172.21.15.118:0/2304797307 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7f67200a2790 0x7f67200a4c50 unknown :-1 s=CLOSED pgs=298 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:17:27.310 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:27.305+0000 7f673e800640 1 --2- 172.21.15.118:0/2304797307 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f6738109460 0x7f67381a1010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:17:27.310 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:27.305+0000 7f673e800640 1 --2- 172.21.15.118:0/2304797307 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f67380756e0 0x7f6738199fa0 unknown :-1 s=CLOSED pgs=281 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:17:27.310 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:27.305+0000 7f673e800640 1 --2- 172.21.15.118:0/2304797307 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f6738074100 0x7f6738199a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:17:27.310 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:27.305+0000 7f673e800640 1 -- 172.21.15.118:0/2304797307 >> 172.21.15.118:0/2304797307 conn(0x7f67380fcb10 msgr2=0x7f6738105360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:17:27.310 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:27.305+0000 7f673e800640 1 -- 172.21.15.118:0/2304797307 shutdown_connections 2024-03-24T00:17:27.310 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:27.305+0000 7f673e800640 1 -- 172.21.15.118:0/2304797307 wait complete. 2024-03-24T00:17:27.325 INFO:teuthology.orchestra.run:Running command with timeout 120 2024-03-24T00:17:27.325 DEBUG:teuthology.orchestra.run.smithi118:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:3 damage ls 2024-03-24T00:17:27.352 INFO:tasks.workunit.client.0.smithi118.stdout:checking for long... yes 2024-03-24T00:17:27.489 INFO:tasks.workunit.client.0.smithi118.stdout:checking size of long... 8 2024-03-24T00:17:27.547 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:27.546+0000 7f7e69c00640 1 -- 172.21.15.118:0/4268979796 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7e641028a0 msgr2=0x7f7e64104d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:17:27.547 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:27.546+0000 7f7e69c00640 1 --2- 172.21.15.118:0/4268979796 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7e641028a0 0x7f7e64104d10 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7f7e4c009900 tx=0x7f7e4c02bc30 comp rx=0 tx=0).stop 2024-03-24T00:17:27.547 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:27.546+0000 7f7e69c00640 1 -- 172.21.15.118:0/4268979796 shutdown_connections 2024-03-24T00:17:27.547 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:27.546+0000 7f7e69c00640 1 --2- 172.21.15.118:0/4268979796 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f7e64105250 0x7f7e641056d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:17:27.548 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:27.546+0000 7f7e69c00640 1 --2- 172.21.15.118:0/4268979796 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7e641028a0 0x7f7e64104d10 unknown :-1 s=CLOSED pgs=285 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:17:27.548 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:27.546+0000 7f7e69c00640 1 --2- 172.21.15.118:0/4268979796 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f7e640fff30 0x7f7e64102360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:17:27.548 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:27.546+0000 7f7e69c00640 1 -- 172.21.15.118:0/4268979796 >> 172.21.15.118:0/4268979796 conn(0x7f7e640fbb50 msgr2=0x7f7e640fdf70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:17:27.548 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:27.546+0000 7f7e69c00640 1 -- 172.21.15.118:0/4268979796 shutdown_connections 2024-03-24T00:17:27.548 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:27.546+0000 7f7e69c00640 1 -- 172.21.15.118:0/4268979796 wait complete. 2024-03-24T00:17:27.550 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:27.547+0000 7f7e69c00640 1 Processor -- start 2024-03-24T00:17:27.550 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:27.548+0000 7f7e69c00640 1 -- start start 2024-03-24T00:17:27.550 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:27.548+0000 7f7e69c00640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f7e640fff30 0x7f7e6419dfc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:17:27.550 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:27.548+0000 7f7e69c00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f7e641028a0 0x7f7e6419e500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:17:27.550 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:27.548+0000 7f7e69c00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7e64105250 0x7f7e641a5570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:17:27.550 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:27.548+0000 7f7e69c00640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7e64113980 con 0x7f7e640fff30 2024-03-24T00:17:27.550 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:27.548+0000 7f7e69c00640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7e64113800 con 0x7f7e64105250 2024-03-24T00:17:27.550 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:27.548+0000 7f7e69c00640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7e64113b00 con 0x7f7e641028a0 2024-03-24T00:17:27.550 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:27.548+0000 7f7e5be00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f7e641028a0 0x7f7e6419e500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:17:27.550 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:27.548+0000 7f7e63400640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7e64105250 0x7f7e641a5570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:17:27.550 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:27.548+0000 7f7e62a00640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f7e640fff30 0x7f7e6419dfc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:17:27.551 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:27.548+0000 7f7e5be00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f7e641028a0 0x7f7e6419e500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.118:37716/0 (socket says 172.21.15.118:37716) 2024-03-24T00:17:27.551 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:27.548+0000 7f7e5be00640 1 -- 172.21.15.118:0/2482114526 learned_addr learned my addr 172.21.15.118:0/2482114526 (peer_addr_for_me v2:172.21.15.118:0/0) 2024-03-24T00:17:27.551 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:27.549+0000 7f7e63400640 1 -- 172.21.15.118:0/2482114526 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f7e641028a0 msgr2=0x7f7e6419e500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:17:27.551 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:27.549+0000 7f7e63400640 1 --2- 172.21.15.118:0/2482114526 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f7e641028a0 0x7f7e6419e500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:17:27.551 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:27.549+0000 7f7e63400640 1 -- 172.21.15.118:0/2482114526 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f7e640fff30 msgr2=0x7f7e6419dfc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:17:27.551 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:27.549+0000 7f7e63400640 1 --2- 172.21.15.118:0/2482114526 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f7e640fff30 0x7f7e6419dfc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:17:27.551 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:27.549+0000 7f7e63400640 1 -- 172.21.15.118:0/2482114526 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7e641a5be0 con 0x7f7e64105250 2024-03-24T00:17:27.551 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:27.549+0000 7f7e5be00640 1 --2- 172.21.15.118:0/2482114526 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f7e641028a0 0x7f7e6419e500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T00:17:27.551 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:27.549+0000 7f7e63400640 1 --2- 172.21.15.118:0/2482114526 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7e64105250 0x7f7e641a5570 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7f7e5c009fd0 tx=0x7f7e5c00ef10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:17:27.551 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:27.549+0000 7f7e5b400640 1 -- 172.21.15.118:0/2482114526 <== mon.2 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7e5c00ce20 con 0x7f7e64105250 2024-03-24T00:17:27.551 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:27.549+0000 7f7e5b400640 1 -- 172.21.15.118:0/2482114526 <== mon.2 v2:172.21.15.189:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f7e5c005020 con 0x7f7e64105250 2024-03-24T00:17:27.551 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:27.549+0000 7f7e5b400640 1 -- 172.21.15.118:0/2482114526 <== mon.2 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7e5c010720 con 0x7f7e64105250 2024-03-24T00:17:27.551 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:27.549+0000 7f7e69c00640 1 -- 172.21.15.118:0/2482114526 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7e641a5ed0 con 0x7f7e64105250 2024-03-24T00:17:27.551 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:27.549+0000 7f7e69c00640 1 -- 172.21.15.118:0/2482114526 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7e641a63e0 con 0x7f7e64105250 2024-03-24T00:17:27.553 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:27.552+0000 7f7e5b400640 1 -- 172.21.15.118:0/2482114526 <== mon.2 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 20) v1 ==== 137291+0+0 (secure 0 0 0) 0x7f7e5c005540 con 0x7f7e64105250 2024-03-24T00:17:27.554 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:27.554+0000 7f7e6ca07740 1 Processor -- start 2024-03-24T00:17:27.555 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:27.554+0000 7f7e5b400640 1 --2- 172.21.15.118:0/2482114526 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7f7e480a2880 0x7f7e480a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:17:27.555 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:27.554+0000 7f7e6ca07740 1 -- start start 2024-03-24T00:17:27.555 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:27.554+0000 7f7e5b400640 1 -- 172.21.15.118:0/2482114526 <== mon.2 v2:172.21.15.189:3300/0 5 ==== osd_map(90..90 src has 1..90) v4 ==== 8277+0+0 (secure 0 0 0) 0x7f7e5c0cf070 con 0x7f7e64105250 2024-03-24T00:17:27.556 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:27.554+0000 7f7e6ca07740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5621eb049810 0x5621eb049c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:17:27.557 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:27.554+0000 7f7e5be00640 1 --2- 172.21.15.118:0/2482114526 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7f7e480a2880 0x7f7e480a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:17:27.557 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:27.554+0000 7f7e6ca07740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5621eb04a9e0 0x5621eb04ae60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:17:27.557 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:27.554+0000 7f7e6ca07740 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5621eb04b3a0 0x5621eb051c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:17:27.557 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:27.554+0000 7f7e6ca07740 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x5621eafae2c0 con 0x5621eb04b3a0 2024-03-24T00:17:27.557 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:27.554+0000 7f7e6ca07740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x5621eaf56c90 con 0x5621eb049810 2024-03-24T00:17:27.557 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:27.554+0000 7f7e6ca07740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x5621eafaa360 con 0x5621eb04a9e0 2024-03-24T00:17:27.557 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:27.555+0000 7f7e62a00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5621eb049810 0x5621eb049c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:17:27.557 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:27.555+0000 7f7e62a00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5621eb049810 0x5621eb049c30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.118:45356/0 (socket says 172.21.15.118:45356) 2024-03-24T00:17:27.557 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:27.555+0000 7f7e62a00640 1 -- 172.21.15.118:0/2196232279 learned_addr learned my addr 172.21.15.118:0/2196232279 (peer_addr_for_me v2:172.21.15.118:0/0) 2024-03-24T00:17:27.557 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:27.555+0000 7f7e63400640 1 --2- 172.21.15.118:0/2196232279 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5621eb04b3a0 0x5621eb051c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:17:27.557 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:27.555+0000 7f7e5be00640 1 --2- 172.21.15.118:0/2196232279 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5621eb04a9e0 0x5621eb04ae60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:17:27.557 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:27.555+0000 7f7e62a00640 1 -- 172.21.15.118:0/2196232279 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5621eb04a9e0 msgr2=0x5621eb04ae60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:17:27.557 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:27.555+0000 7f7e62a00640 1 --2- 172.21.15.118:0/2196232279 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5621eb04a9e0 0x5621eb04ae60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:17:27.558 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:27.555+0000 7f7e62a00640 1 -- 172.21.15.118:0/2196232279 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5621eb04b3a0 msgr2=0x5621eb051c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:17:27.558 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:27.555+0000 7f7e62a00640 1 --2- 172.21.15.118:0/2196232279 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5621eb04b3a0 0x5621eb051c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:17:27.558 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:27.555+0000 7f7e62a00640 1 -- 172.21.15.118:0/2196232279 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5621eaf7b080 con 0x5621eb049810 2024-03-24T00:17:27.558 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:27.556+0000 7f7e62a00640 1 --2- 172.21.15.118:0/2196232279 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5621eb049810 0x5621eb049c30 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x7f7e54009cf0 tx=0x7f7e54011810 comp rx=0 tx=0).ready entity=mon.2 client_cookie=95796628548bbc server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:17:27.558 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:27.556+0000 7f7e5be00640 1 --2- 172.21.15.118:0/2482114526 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7f7e480a2880 0x7f7e480a4d40 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 crypto rx=0x7f7e4c02c8e0 tx=0x7f7e4c0041a0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:17:27.558 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:27.556+0000 7f7e53e00640 1 -- 172.21.15.118:0/2196232279 <== mon.2 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7e54004220 con 0x5621eb049810 2024-03-24T00:17:27.558 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:27.556+0000 7f7e53e00640 1 -- 172.21.15.118:0/2196232279 <== mon.2 v2:172.21.15.189:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f7e540043c0 con 0x5621eb049810 2024-03-24T00:17:27.558 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:27.556+0000 7f7e53e00640 1 -- 172.21.15.118:0/2196232279 <== mon.2 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7e540188b0 con 0x5621eb049810 2024-03-24T00:17:27.563 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:27.557+0000 7f7e6ca07740 1 -- 172.21.15.118:0/2196232279 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5621eb049810 msgr2=0x5621eb049c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:17:27.563 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:27.557+0000 7f7e6ca07740 1 --2- 172.21.15.118:0/2196232279 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5621eb049810 0x5621eb049c30 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x7f7e54009cf0 tx=0x7f7e54011810 comp rx=0 tx=0).stop 2024-03-24T00:17:27.563 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:27.557+0000 7f7e6ca07740 1 -- 172.21.15.118:0/2196232279 shutdown_connections 2024-03-24T00:17:27.563 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:27.557+0000 7f7e6ca07740 1 --2- 172.21.15.118:0/2196232279 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5621eb04b3a0 0x5621eb051c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:17:27.563 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:27.557+0000 7f7e6ca07740 1 --2- 172.21.15.118:0/2196232279 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5621eb04a9e0 0x5621eb04ae60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:17:27.563 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:27.557+0000 7f7e6ca07740 1 --2- 172.21.15.118:0/2196232279 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5621eb049810 0x5621eb049c30 unknown :-1 s=CLOSED pgs=287 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:17:27.563 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:27.557+0000 7f7e6ca07740 1 -- 172.21.15.118:0/2196232279 >> 172.21.15.118:0/2196232279 conn(0x5621eb044c00 msgr2=0x5621eb045000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:17:27.563 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:27.557+0000 7f7e6ca07740 1 -- 172.21.15.118:0/2196232279 shutdown_connections 2024-03-24T00:17:27.563 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:27.557+0000 7f7e6ca07740 1 -- 172.21.15.118:0/2196232279 wait complete. 2024-03-24T00:17:27.563 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:27.557+0000 7f7e6ca07740 1 -- start start 2024-03-24T00:17:27.563 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:27.557+0000 7f7e6ca07740 1 Processor -- start 2024-03-24T00:17:27.564 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:27.558+0000 7f7e6ca07740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5621eb049810 0x5621eb0539d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:17:27.564 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:27.558+0000 7f7e6ca07740 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5621eb04a9e0 0x5621eb053f10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:17:27.564 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:27.558+0000 7f7e6ca07740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5621eb04b3a0 0x5621eb05af50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:17:27.564 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:27.558+0000 7f7e6ca07740 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x5621eb0523b0 con 0x5621eb04a9e0 2024-03-24T00:17:27.564 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:27.558+0000 7f7e6ca07740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x5621eaf6a0d0 con 0x5621eb049810 2024-03-24T00:17:27.564 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:27.558+0000 7f7e6ca07740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x5621eb052530 con 0x5621eb04b3a0 2024-03-24T00:17:27.564 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:27.558+0000 7f7e6ca07740 10 client.0 _ll_register_callbacks cb 0x5621eaf6ee80 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T00:17:27.564 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:27.558+0000 7f7e62a00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5621eb049810 0x5621eb0539d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:17:27.564 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:27.558+0000 7f7e63400640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5621eb04b3a0 0x5621eb05af50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:17:27.564 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:27.558+0000 7f7e5be00640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5621eb04a9e0 0x5621eb053f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:17:27.564 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:27.558+0000 7f7e62a00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5621eb049810 0x5621eb0539d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.118:45366/0 (socket says 172.21.15.118:45366) 2024-03-24T00:17:27.564 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:27.558+0000 7f7e63400640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5621eb04b3a0 0x5621eb05af50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.118:37740/0 (socket says 172.21.15.118:37740) 2024-03-24T00:17:27.564 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:27.558+0000 7f7e62a00640 1 -- 172.21.15.118:0/3220559435 learned_addr learned my addr 172.21.15.118:0/3220559435 (peer_addr_for_me v2:172.21.15.118:0/0) 2024-03-24T00:17:27.564 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:27.559+0000 7f7e63400640 1 -- 172.21.15.118:0/3220559435 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5621eb049810 msgr2=0x5621eb0539d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:17:27.565 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:27.559+0000 7f7e63400640 1 --2- 172.21.15.118:0/3220559435 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5621eb049810 0x5621eb0539d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:17:27.565 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:27.559+0000 7f7e63400640 1 -- 172.21.15.118:0/3220559435 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5621eb04a9e0 msgr2=0x5621eb053f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:17:27.565 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:27.559+0000 7f7e63400640 1 --2- 172.21.15.118:0/3220559435 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5621eb04a9e0 0x5621eb053f10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:17:27.565 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:27.559+0000 7f7e63400640 1 -- 172.21.15.118:0/3220559435 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5621eacc23f0 con 0x5621eb04b3a0 2024-03-24T00:17:27.565 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:27.559+0000 7f7e62a00640 1 --2- 172.21.15.118:0/3220559435 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5621eb049810 0x5621eb0539d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T00:17:27.565 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:27.559+0000 7f7e5be00640 1 --2- 172.21.15.118:0/3220559435 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5621eb04a9e0 0x5621eb053f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T00:17:27.565 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:27.559+0000 7f7e63400640 1 --2- 172.21.15.118:0/3220559435 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5621eb04b3a0 0x5621eb05af50 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7f7e5c00c760 tx=0x7f7e5c09ba40 comp rx=0 tx=0).ready entity=mon.1 client_cookie=458d3bb6e062f431 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:17:27.565 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:27.559+0000 7f7e50c00640 10 client.0 ms_handle_connect on v2:172.21.15.193:3300/0 2024-03-24T00:17:27.565 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:27.559+0000 7f7e50c00640 1 -- 172.21.15.118:0/3220559435 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7e5c0b3070 con 0x5621eb04b3a0 2024-03-24T00:17:27.565 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:27.559+0000 7f7e50c00640 1 -- 172.21.15.118:0/3220559435 <== mon.1 v2:172.21.15.193:3300/0 2 ==== osd_map(90..90 src has 1..90) v4 ==== 8277+0+0 (secure 0 0 0) 0x7f7e5c0a3c90 con 0x5621eb04b3a0 2024-03-24T00:17:27.565 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:27.559+0000 7f7e6ca07740 1 -- 172.21.15.118:0/3220559435 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f7e540043c0 con 0x5621eb04b3a0 2024-03-24T00:17:27.565 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:27.559+0000 7f7e50c00640 1 -- 172.21.15.118:0/3220559435 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) v2 ==== 24+0+0 (secure 0 0 0) 0x7f7e5c0a3e90 con 0x5621eb04b3a0 2024-03-24T00:17:27.565 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:27.559+0000 7f7e6ca07740 10 client.25849 fetch_fsmap learned FSMap version 28 2024-03-24T00:17:27.565 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:27.559+0000 7f7e6ca07740 1 -- 172.21.15.118:0/3220559435 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) v3 -- 0x5621ead8c7d0 con 0x5621eb04b3a0 2024-03-24T00:17:27.566 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:27.560+0000 7f7e50c00640 1 -- 172.21.15.118:0/3220559435 <== mon.1 v2:172.21.15.193:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f7e540043c0 con 0x5621eb04b3a0 2024-03-24T00:17:27.566 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:27.560+0000 7f7e50c00640 1 -- 172.21.15.118:0/3220559435 <== mon.1 v2:172.21.15.193:3300/0 5 ==== fsmap(e 28) v1 ==== 7934+0+0 (secure 0 0 0) 0x7f7e5c09add0 con 0x5621eb04b3a0 2024-03-24T00:17:27.566 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:27.560+0000 7f7e6ca07740 10 client.25849 fetch_fsmap finished waiting for FSMap version 28 2024-03-24T00:17:27.566 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:27.560+0000 7f7e6ca07740 10 client.25849 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.f 2024-03-24T00:17:27.566 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:27.560+0000 7f7e6ca07740 20 client.25849 populate_metadata read hostname 'smithi118' 2024-03-24T00:17:27.566 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:27.560+0000 7f7e6ca07740 1 --2- 172.21.15.118:0/3220559435 >> [v2:172.21.15.193:6834/2055310541,v1:172.21.15.193:6835/2055310541] conn(0x5621eb05f4a0 0x5621eb061890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:17:27.566 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:27.560+0000 7f7e6ca07740 4 client.25849 mds_command: new command op to 24499 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-24T00:17:27.566 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:27.560+0000 7f7e6ca07740 1 -- 172.21.15.118:0/3220559435 --> [v2:172.21.15.193:6834/2055310541,v1:172.21.15.193:6835/2055310541] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x5621eafbb800 con 0x5621eb05f4a0 2024-03-24T00:17:27.566 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:27.560+0000 7f7e62a00640 1 --2- 172.21.15.118:0/3220559435 >> [v2:172.21.15.193:6834/2055310541,v1:172.21.15.193:6835/2055310541] conn(0x5621eb05f4a0 0x5621eb061890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:17:27.566 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:27.561+0000 7f7e50c00640 10 client.25849 ms_handle_connect on v2:172.21.15.193:6834/2055310541 2024-03-24T00:17:27.566 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:27.561+0000 7f7e62a00640 1 --2- 172.21.15.118:0/3220559435 >> [v2:172.21.15.193:6834/2055310541,v1:172.21.15.193:6835/2055310541] conn(0x5621eb05f4a0 0x5621eb061890 crc :-1 s=READY pgs=29 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=8cafdb9f283a34d0 server_cookie=83766c079985630f in_seq=0 out_seq=0 2024-03-24T00:17:27.566 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:27.562+0000 7f7e50c00640 1 -- 172.21.15.118:0/3220559435 <== mds.3 v2:172.21.15.193:6834/2055310541 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x5621eafbb800 con 0x5621eb05f4a0 2024-03-24T00:17:27.566 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:27.562+0000 7f7e50c00640 10 client.25849 handle_command_reply: tid=0 2024-03-24T00:17:27.566 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:27.562+0000 7f7e6ca07740 1 client.25849 shutdown 2024-03-24T00:17:27.567 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:27.562+0000 7f7e6ca07740 1 -- 172.21.15.118:0/3220559435 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5621eb04b3a0 msgr2=0x5621eb05af50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:17:27.567 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:27.562+0000 7f7e6ca07740 1 --2- 172.21.15.118:0/3220559435 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5621eb04b3a0 0x5621eb05af50 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7f7e5c00c760 tx=0x7f7e5c09ba40 comp rx=0 tx=0).stop 2024-03-24T00:17:27.567 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:27.562+0000 7f7e6ca07740 1 -- 172.21.15.118:0/3220559435 shutdown_connections 2024-03-24T00:17:27.567 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:27.562+0000 7f7e6ca07740 1 --2- 172.21.15.118:0/3220559435 >> [v2:172.21.15.193:6834/2055310541,v1:172.21.15.193:6835/2055310541] conn(0x5621eb05f4a0 0x5621eb061890 crc :-1 s=READY pgs=29 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:17:27.567 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:27.562+0000 7f7e6ca07740 1 --2- 172.21.15.118:0/3220559435 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5621eb04b3a0 0x5621eb05af50 unknown :-1 s=CLOSED pgs=270 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:17:27.567 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:27.562+0000 7f7e50c00640 0 client.25849 ms_handle_reset on v2:172.21.15.193:6834/2055310541 2024-03-24T00:17:27.567 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:27.562+0000 7f7e6ca07740 1 --2- 172.21.15.118:0/3220559435 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5621eb04a9e0 0x5621eb053f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:17:27.567 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:27.562+0000 7f7e6ca07740 1 --2- 172.21.15.118:0/3220559435 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5621eb049810 0x5621eb0539d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:17:27.567 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:27.562+0000 7f7e6ca07740 1 -- 172.21.15.118:0/3220559435 >> 172.21.15.118:0/3220559435 conn(0x5621eb044c00 msgr2=0x5621eb043eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:17:27.567 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:27.562+0000 7f7e6ca07740 1 -- 172.21.15.118:0/3220559435 shutdown_connections 2024-03-24T00:17:27.567 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:27.562+0000 7f7e6ca07740 1 -- 172.21.15.118:0/3220559435 wait complete. 2024-03-24T00:17:27.567 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:27.562+0000 7f7e6ca07740 20 client.25849 trim_cache size 0 max 16384 2024-03-24T00:17:27.577 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:27.576+0000 7f7e6ca07740 1 Processor -- start 2024-03-24T00:17:27.577 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:27.576+0000 7f7e6ca07740 1 -- start start 2024-03-24T00:17:27.577 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:27.576+0000 7f7e6ca07740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5621eb049810 0x5621eb052ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:17:27.577 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:27.576+0000 7f7e6ca07740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5621eb04a9e0 0x5621eb05af10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:17:27.577 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:27.576+0000 7f7e6ca07740 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5621eb04b3a0 0x5621eb043620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:17:27.577 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:27.576+0000 7f7e6ca07740 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7e641aeff0 con 0x5621eb04b3a0 2024-03-24T00:17:27.577 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:27.576+0000 7f7e6ca07740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7e641b0970 con 0x5621eb049810 2024-03-24T00:17:27.577 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:27.576+0000 7f7e6ca07740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7e641af650 con 0x5621eb04a9e0 2024-03-24T00:17:27.579 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:27.577+0000 7f7e5be00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5621eb04a9e0 0x5621eb05af10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:17:27.579 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:27.577+0000 7f7e63400640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5621eb04b3a0 0x5621eb043620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:17:27.579 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:27.577+0000 7f7e62a00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5621eb049810 0x5621eb052ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:17:27.579 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:27.577+0000 7f7e5be00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5621eb04a9e0 0x5621eb05af10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.118:37748/0 (socket says 172.21.15.118:37748) 2024-03-24T00:17:27.579 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:27.577+0000 7f7e5be00640 1 -- 172.21.15.118:0/1091662265 learned_addr learned my addr 172.21.15.118:0/1091662265 (peer_addr_for_me v2:172.21.15.118:0/0) 2024-03-24T00:17:27.579 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:27.577+0000 7f7e5be00640 1 -- 172.21.15.118:0/1091662265 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5621eb049810 msgr2=0x5621eb052ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:17:27.579 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:27.577+0000 7f7e5be00640 1 --2- 172.21.15.118:0/1091662265 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5621eb049810 0x5621eb052ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:17:27.579 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:27.577+0000 7f7e5be00640 1 -- 172.21.15.118:0/1091662265 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5621eb04b3a0 msgr2=0x5621eb043620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:17:27.579 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:27.577+0000 7f7e5be00640 1 --2- 172.21.15.118:0/1091662265 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5621eb04b3a0 0x5621eb043620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:17:27.579 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:27.577+0000 7f7e5be00640 1 -- 172.21.15.118:0/1091662265 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5621eb0537c0 con 0x5621eb04a9e0 2024-03-24T00:17:27.579 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:27.577+0000 7f7e62a00640 1 --2- 172.21.15.118:0/1091662265 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5621eb049810 0x5621eb052ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T00:17:27.579 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:27.578+0000 7f7e5be00640 1 --2- 172.21.15.118:0/1091662265 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5621eb04a9e0 0x5621eb05af10 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7f7e4c036040 tx=0x7f7e4c046000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=ee3bb6546b93b86e server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:17:27.579 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:27.578+0000 7f7e53400640 1 -- 172.21.15.118:0/1091662265 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7e4c053070 con 0x5621eb04a9e0 2024-03-24T00:17:27.580 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:27.578+0000 7f7e53400640 1 -- 172.21.15.118:0/1091662265 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f7e4c004b50 con 0x5621eb04a9e0 2024-03-24T00:17:27.580 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:27.578+0000 7f7e53400640 1 -- 172.21.15.118:0/1091662265 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7e4c003a50 con 0x5621eb04a9e0 2024-03-24T00:17:27.583 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:27.578+0000 7f7e6ca07740 1 -- 172.21.15.118:0/1091662265 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5621eb04a9e0 msgr2=0x5621eb05af10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:17:27.583 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:27.579+0000 7f7e6ca07740 1 --2- 172.21.15.118:0/1091662265 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5621eb04a9e0 0x5621eb05af10 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7f7e4c036040 tx=0x7f7e4c046000 comp rx=0 tx=0).stop 2024-03-24T00:17:27.583 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:27.579+0000 7f7e6ca07740 1 -- 172.21.15.118:0/1091662265 shutdown_connections 2024-03-24T00:17:27.583 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:27.579+0000 7f7e6ca07740 1 --2- 172.21.15.118:0/1091662265 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5621eb04b3a0 0x5621eb043620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:17:27.583 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:27.579+0000 7f7e6ca07740 1 --2- 172.21.15.118:0/1091662265 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5621eb04a9e0 0x5621eb05af10 unknown :-1 s=CLOSED pgs=271 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:17:27.583 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:27.579+0000 7f7e6ca07740 1 --2- 172.21.15.118:0/1091662265 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5621eb049810 0x5621eb052ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:17:27.583 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:27.579+0000 7f7e6ca07740 1 -- 172.21.15.118:0/1091662265 >> 172.21.15.118:0/1091662265 conn(0x5621eb044c00 msgr2=0x5621eb0486f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:17:27.583 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:27.579+0000 7f7e6ca07740 1 -- 172.21.15.118:0/1091662265 shutdown_connections 2024-03-24T00:17:27.583 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:27.579+0000 7f7e6ca07740 1 -- 172.21.15.118:0/1091662265 wait complete. 2024-03-24T00:17:27.584 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:27.579+0000 7f7e6ca07740 1 -- start start 2024-03-24T00:17:27.584 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:27.579+0000 7f7e6ca07740 1 Processor -- start 2024-03-24T00:17:27.584 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:27.580+0000 7f7e6ca07740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5621eb049810 0x5621eb05a820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:17:27.584 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:27.580+0000 7f7e6ca07740 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5621eb04a9e0 0x5621eb05ad60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:17:27.584 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:27.580+0000 7f7e6ca07740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5621eb04b3a0 0x5621eb058e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:17:27.584 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:27.580+0000 7f7e6ca07740 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7e641b0310 con 0x5621eb04a9e0 2024-03-24T00:17:27.584 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:27.580+0000 7f7e6ca07740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7e641afcb0 con 0x5621eb04b3a0 2024-03-24T00:17:27.584 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:27.580+0000 7f7e6ca07740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7e641ad1b0 con 0x5621eb049810 2024-03-24T00:17:27.584 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:27.580+0000 7f7e6ca07740 10 client.0 _ll_register_callbacks cb 0x7f7e5c005b20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T00:17:27.584 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:27.580+0000 7f7e63400640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5621eb04b3a0 0x5621eb058e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:17:27.584 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:27.580+0000 7f7e62a00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5621eb049810 0x5621eb05a820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:17:27.584 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:27.580+0000 7f7e5be00640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5621eb04a9e0 0x5621eb05ad60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:17:27.584 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:27.580+0000 7f7e63400640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5621eb04b3a0 0x5621eb058e30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.118:45380/0 (socket says 172.21.15.118:45380) 2024-03-24T00:17:27.585 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:27.580+0000 7f7e62a00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5621eb049810 0x5621eb05a820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.118:37756/0 (socket says 172.21.15.118:37756) 2024-03-24T00:17:27.585 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:27.580+0000 7f7e63400640 1 -- 172.21.15.118:0/4213951548 learned_addr learned my addr 172.21.15.118:0/4213951548 (peer_addr_for_me v2:172.21.15.118:0/0) 2024-03-24T00:17:27.585 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:27.580+0000 7f7e63400640 1 -- 172.21.15.118:0/4213951548 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5621eb049810 msgr2=0x5621eb05a820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:17:27.585 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:27.580+0000 7f7e63400640 1 --2- 172.21.15.118:0/4213951548 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5621eb049810 0x5621eb05a820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:17:27.585 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:27.580+0000 7f7e63400640 1 -- 172.21.15.118:0/4213951548 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5621eb04a9e0 msgr2=0x5621eb05ad60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:17:27.585 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:27.580+0000 7f7e63400640 1 --2- 172.21.15.118:0/4213951548 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5621eb04a9e0 0x5621eb05ad60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:17:27.585 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:27.580+0000 7f7e63400640 1 -- 172.21.15.118:0/4213951548 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5621eb059520 con 0x5621eb04b3a0 2024-03-24T00:17:27.585 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:27.580+0000 7f7e5be00640 1 --2- 172.21.15.118:0/4213951548 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5621eb04a9e0 0x5621eb05ad60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T00:17:27.585 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:27.580+0000 7f7e62a00640 1 --2- 172.21.15.118:0/4213951548 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5621eb049810 0x5621eb05a820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T00:17:27.585 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:27.581+0000 7f7e63400640 1 --2- 172.21.15.118:0/4213951548 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5621eb04b3a0 0x5621eb058e30 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 crypto rx=0x7f7e5c07c040 tx=0x7f7e5c09b880 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f4585991d76366fa server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:17:27.585 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:27.581+0000 7f7e50c00640 10 client.0 ms_handle_connect on v2:172.21.15.189:3300/0 2024-03-24T00:17:27.585 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:27.581+0000 7f7e50c00640 1 -- 172.21.15.118:0/4213951548 <== mon.2 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7e5c0b3070 con 0x5621eb04b3a0 2024-03-24T00:17:27.585 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:27.581+0000 7f7e50c00640 1 -- 172.21.15.118:0/4213951548 <== mon.2 v2:172.21.15.189:3300/0 2 ==== osd_map(90..90 src has 1..90) v4 ==== 8277+0+0 (secure 0 0 0) 0x7f7e5c0860b0 con 0x5621eb04b3a0 2024-03-24T00:17:27.585 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:27.581+0000 7f7e6ca07740 1 -- 172.21.15.118:0/4213951548 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f7e4c004b50 con 0x5621eb04b3a0 2024-03-24T00:17:27.585 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:27.581+0000 7f7e50c00640 1 -- 172.21.15.118:0/4213951548 <== mon.2 v2:172.21.15.189:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) v2 ==== 24+0+0 (secure 0 0 0) 0x7f7e5c09c080 con 0x5621eb04b3a0 2024-03-24T00:17:27.586 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:27.581+0000 7f7e6ca07740 10 client.25871 fetch_fsmap learned FSMap version 28 2024-03-24T00:17:27.586 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:27.581+0000 7f7e6ca07740 1 -- 172.21.15.118:0/4213951548 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) v3 -- 0x5621eb0bb700 con 0x5621eb04b3a0 2024-03-24T00:17:27.586 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:27.581+0000 7f7e50c00640 1 -- 172.21.15.118:0/4213951548 <== mon.2 v2:172.21.15.189:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f7e4c004b50 con 0x5621eb04b3a0 2024-03-24T00:17:27.586 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:27.582+0000 7f7e50c00640 1 -- 172.21.15.118:0/4213951548 <== mon.2 v2:172.21.15.189:3300/0 5 ==== fsmap(e 28) v1 ==== 7934+0+0 (secure 0 0 0) 0x7f7e5c0a0bb0 con 0x5621eb04b3a0 2024-03-24T00:17:27.586 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:27.582+0000 7f7e6ca07740 10 client.25871 fetch_fsmap finished waiting for FSMap version 28 2024-03-24T00:17:27.586 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:27.582+0000 7f7e6ca07740 10 client.25871 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.f 2024-03-24T00:17:27.586 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:27.582+0000 7f7e6ca07740 20 client.25871 populate_metadata read hostname 'smithi118' 2024-03-24T00:17:27.586 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:27.582+0000 7f7e6ca07740 1 --2- 172.21.15.118:0/4213951548 >> [v2:172.21.15.193:6834/2055310541,v1:172.21.15.193:6835/2055310541] conn(0x5621eb0bc0b0 0x5621eb0be4d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:17:27.586 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:27.582+0000 7f7e6ca07740 4 client.25871 mds_command: new command op to 24499 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-24T00:17:27.586 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:27.582+0000 7f7e6ca07740 1 -- 172.21.15.118:0/4213951548 --> [v2:172.21.15.193:6834/2055310541,v1:172.21.15.193:6835/2055310541] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x5621eafb5b50 con 0x5621eb0bc0b0 2024-03-24T00:17:27.586 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:27.582+0000 7f7e62a00640 1 --2- 172.21.15.118:0/4213951548 >> [v2:172.21.15.193:6834/2055310541,v1:172.21.15.193:6835/2055310541] conn(0x5621eb0bc0b0 0x5621eb0be4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:17:27.586 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:27.582+0000 7f7e62a00640 1 --2- 172.21.15.118:0/4213951548 >> [v2:172.21.15.193:6834/2055310541,v1:172.21.15.193:6835/2055310541] conn(0x5621eb0bc0b0 0x5621eb0be4d0 crc :-1 s=READY pgs=30 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=633e0c7f705647c0 server_cookie=ff2b6f28fc73f4ba in_seq=0 out_seq=0 2024-03-24T00:17:27.586 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:27.582+0000 7f7e50c00640 10 client.25871 ms_handle_connect on v2:172.21.15.193:6834/2055310541 2024-03-24T00:17:27.587 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:27.583+0000 7f7e50c00640 1 -- 172.21.15.118:0/4213951548 <== mds.3 v2:172.21.15.193:6834/2055310541 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x5621eafb5b50 con 0x5621eb0bc0b0 2024-03-24T00:17:27.587 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:27.583+0000 7f7e50c00640 10 client.25871 handle_command_reply: tid=0 2024-03-24T00:17:27.587 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:27.583+0000 7f7e6ca07740 1 client.25871 shutdown 2024-03-24T00:17:27.588 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:27.583+0000 7f7e6ca07740 1 -- 172.21.15.118:0/4213951548 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5621eb04b3a0 msgr2=0x5621eb058e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:17:27.588 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:27.583+0000 7f7e6ca07740 1 --2- 172.21.15.118:0/4213951548 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5621eb04b3a0 0x5621eb058e30 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 crypto rx=0x7f7e5c07c040 tx=0x7f7e5c09b880 comp rx=0 tx=0).stop 2024-03-24T00:17:27.588 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:27.583+0000 7f7e6ca07740 1 -- 172.21.15.118:0/4213951548 shutdown_connections 2024-03-24T00:17:27.588 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:27.583+0000 7f7e6ca07740 1 --2- 172.21.15.118:0/4213951548 >> [v2:172.21.15.193:6834/2055310541,v1:172.21.15.193:6835/2055310541] conn(0x5621eb0bc0b0 0x5621eb0be4d0 crc :-1 s=READY pgs=30 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:17:27.588 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:27.583+0000 7f7e6ca07740 1 --2- 172.21.15.118:0/4213951548 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5621eb04b3a0 0x5621eb058e30 unknown :-1 s=CLOSED pgs=288 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:17:27.588 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:27.583+0000 7f7e6ca07740 1 --2- 172.21.15.118:0/4213951548 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5621eb04a9e0 0x5621eb05ad60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:17:27.588 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:27.583+0000 7f7e6ca07740 1 --2- 172.21.15.118:0/4213951548 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5621eb049810 0x5621eb05a820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:17:27.588 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:27.583+0000 7f7e50c00640 0 client.25871 ms_handle_reset on v2:172.21.15.193:6834/2055310541 2024-03-24T00:17:27.588 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:27.583+0000 7f7e6ca07740 1 -- 172.21.15.118:0/4213951548 >> 172.21.15.118:0/4213951548 conn(0x5621eb044c00 msgr2=0x5621eb0486f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:17:27.588 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:27.583+0000 7f7e6ca07740 1 -- 172.21.15.118:0/4213951548 shutdown_connections 2024-03-24T00:17:27.588 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:27.583+0000 7f7e6ca07740 1 -- 172.21.15.118:0/4213951548 wait complete. 2024-03-24T00:17:27.589 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:27.583+0000 7f7e6ca07740 20 client.25871 trim_cache size 0 max 16384 2024-03-24T00:17:27.589 INFO:teuthology.orchestra.run.smithi118.stdout:[] 2024-03-24T00:17:27.590 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:27.584+0000 7f7e69c00640 1 -- 172.21.15.118:0/2482114526 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7f7e480a2880 msgr2=0x7f7e480a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:17:27.590 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:27.584+0000 7f7e69c00640 1 --2- 172.21.15.118:0/2482114526 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7f7e480a2880 0x7f7e480a4d40 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 crypto rx=0x7f7e4c02c8e0 tx=0x7f7e4c0041a0 comp rx=0 tx=0).stop 2024-03-24T00:17:27.590 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:27.584+0000 7f7e69c00640 1 -- 172.21.15.118:0/2482114526 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7e64105250 msgr2=0x7f7e641a5570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:17:27.590 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:27.584+0000 7f7e69c00640 1 --2- 172.21.15.118:0/2482114526 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7e64105250 0x7f7e641a5570 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7f7e5c009fd0 tx=0x7f7e5c00ef10 comp rx=0 tx=0).stop 2024-03-24T00:17:27.590 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:27.584+0000 7f7e69c00640 1 -- 172.21.15.118:0/2482114526 shutdown_connections 2024-03-24T00:17:27.590 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:27.584+0000 7f7e69c00640 1 --2- 172.21.15.118:0/2482114526 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7f7e480a2880 0x7f7e480a4d40 unknown :-1 s=CLOSED pgs=299 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:17:27.590 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:27.584+0000 7f7e69c00640 1 --2- 172.21.15.118:0/2482114526 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7e64105250 0x7f7e641a5570 unknown :-1 s=CLOSED pgs=286 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:17:27.590 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:27.584+0000 7f7e69c00640 1 --2- 172.21.15.118:0/2482114526 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f7e641028a0 0x7f7e6419e500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:17:27.590 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:27.584+0000 7f7e69c00640 1 --2- 172.21.15.118:0/2482114526 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f7e640fff30 0x7f7e6419dfc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:17:27.590 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:27.584+0000 7f7e69c00640 1 -- 172.21.15.118:0/2482114526 >> 172.21.15.118:0/2482114526 conn(0x7f7e640fbb50 msgr2=0x7f7e64100c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:17:27.590 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:27.584+0000 7f7e69c00640 1 -- 172.21.15.118:0/2482114526 shutdown_connections 2024-03-24T00:17:27.591 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:27.584+0000 7f7e69c00640 1 -- 172.21.15.118:0/2482114526 wait complete. 2024-03-24T00:17:27.602 INFO:teuthology.orchestra.run:Running command with timeout 120 2024-03-24T00:17:27.602 DEBUG:teuthology.orchestra.run.smithi118:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:2 damage ls 2024-03-24T00:17:27.622 INFO:tasks.workunit.client.0.smithi118.stdout:checking for long long... yes 2024-03-24T00:17:27.766 INFO:tasks.workunit.client.0.smithi118.stdout:checking size of long long... 8 2024-03-24T00:17:27.801 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:27.799+0000 7f3c72400640 1 -- 172.21.15.118:0/2372849977 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f3c6c102790 msgr2=0x7f3c6c102c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:17:27.801 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:27.799+0000 7f3c72400640 1 --2- 172.21.15.118:0/2372849977 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f3c6c102790 0x7f3c6c102c10 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7f3c58009860 tx=0x7f3c5802bb70 comp rx=0 tx=0).stop 2024-03-24T00:17:27.801 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:27.799+0000 7f3c72400640 1 -- 172.21.15.118:0/2372849977 shutdown_connections 2024-03-24T00:17:27.801 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:27.799+0000 7f3c72400640 1 --2- 172.21.15.118:0/2372849977 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f3c6c103150 0x7f3c6c1099e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:17:27.801 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:27.799+0000 7f3c72400640 1 --2- 172.21.15.118:0/2372849977 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f3c6c102790 0x7f3c6c102c10 unknown :-1 s=CLOSED pgs=272 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:17:27.802 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:27.799+0000 7f3c72400640 1 --2- 172.21.15.118:0/2372849977 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3c6c1015a0 0x7f3c6c1019c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:17:27.802 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:27.799+0000 7f3c72400640 1 -- 172.21.15.118:0/2372849977 >> 172.21.15.118:0/2372849977 conn(0x7f3c6c0fcb10 msgr2=0x7f3c6c0fef50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:17:27.802 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:27.800+0000 7f3c72400640 1 -- 172.21.15.118:0/2372849977 shutdown_connections 2024-03-24T00:17:27.802 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:27.800+0000 7f3c72400640 1 -- 172.21.15.118:0/2372849977 wait complete. 2024-03-24T00:17:27.806 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:27.801+0000 7f3c72400640 1 Processor -- start 2024-03-24T00:17:27.806 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:27.801+0000 7f3c72400640 1 -- start start 2024-03-24T00:17:27.806 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:27.802+0000 7f3c72400640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f3c6c1015a0 0x7f3c6c075450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:17:27.806 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:27.802+0000 7f3c72400640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3c6c102790 0x7f3c6c073a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:17:27.806 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:27.802+0000 7f3c72400640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f3c6c103150 0x7f3c6c073fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:17:27.806 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:27.802+0000 7f3c72400640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3c6c10d230 con 0x7f3c6c1015a0 2024-03-24T00:17:27.806 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:27.802+0000 7f3c72400640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3c6c10d0b0 con 0x7f3c6c102790 2024-03-24T00:17:27.807 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:27.802+0000 7f3c72400640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3c6c10d3b0 con 0x7f3c6c103150 2024-03-24T00:17:27.807 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:27.802+0000 7f3c71000640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f3c6c1015a0 0x7f3c6c075450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:17:27.807 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:27.802+0000 7f3c6be00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3c6c102790 0x7f3c6c073a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:17:27.807 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:27.802+0000 7f3c71000640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f3c6c1015a0 0x7f3c6c075450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.118:3300/0 says I am v2:172.21.15.118:38412/0 (socket says 172.21.15.118:38412) 2024-03-24T00:17:27.807 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:27.802+0000 7f3c71000640 1 -- 172.21.15.118:0/2319075406 learned_addr learned my addr 172.21.15.118:0/2319075406 (peer_addr_for_me v2:172.21.15.118:0/0) 2024-03-24T00:17:27.807 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:27.802+0000 7f3c71a00640 1 --2- 172.21.15.118:0/2319075406 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f3c6c103150 0x7f3c6c073fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:17:27.807 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:27.803+0000 7f3c6be00640 1 -- 172.21.15.118:0/2319075406 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f3c6c103150 msgr2=0x7f3c6c073fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:17:27.807 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:27.803+0000 7f3c6be00640 1 --2- 172.21.15.118:0/2319075406 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f3c6c103150 0x7f3c6c073fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:17:27.808 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:27.803+0000 7f3c6be00640 1 -- 172.21.15.118:0/2319075406 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f3c6c1015a0 msgr2=0x7f3c6c075450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:17:27.808 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:27.803+0000 7f3c6be00640 1 --2- 172.21.15.118:0/2319075406 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f3c6c1015a0 0x7f3c6c075450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:17:27.808 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:27.803+0000 7f3c6be00640 1 -- 172.21.15.118:0/2319075406 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3c6c074860 con 0x7f3c6c102790 2024-03-24T00:17:27.808 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:27.803+0000 7f3c71a00640 1 --2- 172.21.15.118:0/2319075406 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f3c6c103150 0x7f3c6c073fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T00:17:27.808 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:27.803+0000 7f3c71000640 1 --2- 172.21.15.118:0/2319075406 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f3c6c1015a0 0x7f3c6c075450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T00:17:27.808 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:27.803+0000 7f3c6be00640 1 --2- 172.21.15.118:0/2319075406 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3c6c102790 0x7f3c6c073a90 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 crypto rx=0x7f3c5802c0b0 tx=0x7f3c58032430 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:17:27.808 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:27.803+0000 7f3c69600640 1 -- 172.21.15.118:0/2319075406 <== mon.2 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3c58041020 con 0x7f3c6c102790 2024-03-24T00:17:27.808 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:27.803+0000 7f3c72400640 1 -- 172.21.15.118:0/2319075406 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3c6c1a18f0 con 0x7f3c6c102790 2024-03-24T00:17:27.809 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:27.804+0000 7f3c69600640 1 -- 172.21.15.118:0/2319075406 <== mon.2 v2:172.21.15.189:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f3c5802c9f0 con 0x7f3c6c102790 2024-03-24T00:17:27.809 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:27.804+0000 7f3c69600640 1 -- 172.21.15.118:0/2319075406 <== mon.2 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3c58039040 con 0x7f3c6c102790 2024-03-24T00:17:27.809 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:27.804+0000 7f3c72400640 1 -- 172.21.15.118:0/2319075406 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3c6c1a1e60 con 0x7f3c6c102790 2024-03-24T00:17:27.819 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:27.807+0000 7f3c69600640 1 -- 172.21.15.118:0/2319075406 <== mon.2 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 20) v1 ==== 137291+0+0 (secure 0 0 0) 0x7f3c58044020 con 0x7f3c6c102790 2024-03-24T00:17:27.819 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:27.808+0000 7f3c76620740 1 Processor -- start 2024-03-24T00:17:27.819 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:27.808+0000 7f3c76620740 1 -- start start 2024-03-24T00:17:27.819 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:27.808+0000 7f3c76620740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5588c70a49f0 0x5588c70a4e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:17:27.819 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:27.808+0000 7f3c76620740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5588c70a5bc0 0x5588c70a6040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:17:27.819 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:27.808+0000 7f3c76620740 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5588c70a6580 0x5588c70ace10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:17:27.819 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:27.808+0000 7f3c76620740 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x5588c70092c0 con 0x5588c70a6580 2024-03-24T00:17:27.820 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:27.808+0000 7f3c76620740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x5588c6fb1c90 con 0x5588c70a49f0 2024-03-24T00:17:27.820 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:27.808+0000 7f3c76620740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x5588c7005360 con 0x5588c70a5bc0 2024-03-24T00:17:27.820 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:27.809+0000 7f3c71a00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5588c70a49f0 0x5588c70a4e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:17:27.820 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:27.809+0000 7f3c71a00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5588c70a49f0 0x5588c70a4e10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.118:45412/0 (socket says 172.21.15.118:45412) 2024-03-24T00:17:27.820 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:27.809+0000 7f3c71a00640 1 -- 172.21.15.118:0/2197017222 learned_addr learned my addr 172.21.15.118:0/2197017222 (peer_addr_for_me v2:172.21.15.118:0/0) 2024-03-24T00:17:27.820 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:27.809+0000 7f3c6be00640 1 --2- 172.21.15.118:0/2197017222 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5588c70a6580 0x5588c70ace10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:17:27.820 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:27.809+0000 7f3c71000640 1 --2- 172.21.15.118:0/2197017222 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5588c70a5bc0 0x5588c70a6040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:17:27.820 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:27.809+0000 7f3c71a00640 1 -- 172.21.15.118:0/2197017222 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5588c70a5bc0 msgr2=0x5588c70a6040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:17:27.821 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:27.809+0000 7f3c71a00640 1 --2- 172.21.15.118:0/2197017222 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5588c70a5bc0 0x5588c70a6040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:17:27.821 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:27.809+0000 7f3c71a00640 1 -- 172.21.15.118:0/2197017222 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5588c70a6580 msgr2=0x5588c70ace10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:17:27.821 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:27.809+0000 7f3c71a00640 1 --2- 172.21.15.118:0/2197017222 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5588c70a6580 0x5588c70ace10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:17:27.821 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:27.809+0000 7f3c71a00640 1 -- 172.21.15.118:0/2197017222 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5588c6fd6080 con 0x5588c70a49f0 2024-03-24T00:17:27.821 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:27.809+0000 7f3c69600640 1 --2- 172.21.15.118:0/2319075406 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7f3c540a2880 0x7f3c540a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:17:27.821 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:27.809+0000 7f3c71a00640 1 --2- 172.21.15.118:0/2197017222 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5588c70a49f0 0x5588c70a4e10 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x7f3c640117e0 tx=0x7f3c64011ca0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=55d65a258018dfb server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:17:27.821 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:27.809+0000 7f3c69600640 1 -- 172.21.15.118:0/2319075406 <== mon.2 v2:172.21.15.189:3300/0 5 ==== osd_map(90..90 src has 1..90) v4 ==== 8277+0+0 (secure 0 0 0) 0x7f3c580f0210 con 0x7f3c6c102790 2024-03-24T00:17:27.821 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:27.810+0000 7f3c71000640 1 --2- 172.21.15.118:0/2319075406 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7f3c540a2880 0x7f3c540a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:17:27.822 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:27.810+0000 7f3c5e000640 1 -- 172.21.15.118:0/2197017222 <== mon.2 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3c64011310 con 0x5588c70a49f0 2024-03-24T00:17:27.822 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:27.810+0000 7f3c5e000640 1 -- 172.21.15.118:0/2197017222 <== mon.2 v2:172.21.15.189:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f3c64003e50 con 0x5588c70a49f0 2024-03-24T00:17:27.822 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:27.810+0000 7f3c5e000640 1 -- 172.21.15.118:0/2197017222 <== mon.2 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3c64004620 con 0x5588c70a49f0 2024-03-24T00:17:27.822 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:27.811+0000 7f3c76620740 1 -- 172.21.15.118:0/2197017222 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5588c70a49f0 msgr2=0x5588c70a4e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:17:27.822 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:27.811+0000 7f3c71000640 1 --2- 172.21.15.118:0/2319075406 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7f3c540a2880 0x7f3c540a4d40 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 crypto rx=0x7f3c60009fd0 tx=0x7f3c60009260 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:17:27.822 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:27.811+0000 7f3c76620740 1 --2- 172.21.15.118:0/2197017222 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5588c70a49f0 0x5588c70a4e10 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x7f3c640117e0 tx=0x7f3c64011ca0 comp rx=0 tx=0).stop 2024-03-24T00:17:27.822 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:27.811+0000 7f3c76620740 1 -- 172.21.15.118:0/2197017222 shutdown_connections 2024-03-24T00:17:27.822 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:27.811+0000 7f3c76620740 1 --2- 172.21.15.118:0/2197017222 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5588c70a6580 0x5588c70ace10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:17:27.822 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:27.811+0000 7f3c76620740 1 --2- 172.21.15.118:0/2197017222 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5588c70a5bc0 0x5588c70a6040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:17:27.823 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:27.811+0000 7f3c76620740 1 --2- 172.21.15.118:0/2197017222 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5588c70a49f0 0x5588c70a4e10 unknown :-1 s=CLOSED pgs=290 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:17:27.823 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:27.811+0000 7f3c76620740 1 -- 172.21.15.118:0/2197017222 >> 172.21.15.118:0/2197017222 conn(0x5588c709fde0 msgr2=0x5588c70a01e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:17:27.823 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:27.811+0000 7f3c76620740 1 -- 172.21.15.118:0/2197017222 shutdown_connections 2024-03-24T00:17:27.823 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:27.811+0000 7f3c76620740 1 -- 172.21.15.118:0/2197017222 wait complete. 2024-03-24T00:17:27.823 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:27.811+0000 7f3c76620740 1 -- start start 2024-03-24T00:17:27.823 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:27.812+0000 7f3c76620740 1 Processor -- start 2024-03-24T00:17:27.823 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:27.812+0000 7f3c76620740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5588c70a49f0 0x5588c70aec90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:17:27.823 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:27.812+0000 7f3c76620740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5588c70a5bc0 0x5588c70af1d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:17:27.824 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:27.812+0000 7f3c76620740 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5588c70a6580 0x5588c70b6210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:17:27.824 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:27.812+0000 7f3c76620740 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x5588c70ad590 con 0x5588c70a6580 2024-03-24T00:17:27.824 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:27.812+0000 7f3c76620740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x5588c6fc50d0 con 0x5588c70a5bc0 2024-03-24T00:17:27.824 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:27.812+0000 7f3c76620740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x5588c70ad710 con 0x5588c70a49f0 2024-03-24T00:17:27.824 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:27.813+0000 7f3c71000640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5588c70a49f0 0x5588c70aec90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:17:27.824 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:27.813+0000 7f3c76620740 10 client.0 _ll_register_callbacks cb 0x5588c6fc9e80 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T00:17:27.824 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:27.813+0000 7f3c71000640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5588c70a49f0 0x5588c70aec90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.118:37802/0 (socket says 172.21.15.118:37802) 2024-03-24T00:17:27.824 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:27.813+0000 7f3c6be00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5588c70a5bc0 0x5588c70af1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:17:27.824 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:27.813+0000 7f3c71000640 1 -- 172.21.15.118:0/3357090347 learned_addr learned my addr 172.21.15.118:0/3357090347 (peer_addr_for_me v2:172.21.15.118:0/0) 2024-03-24T00:17:27.825 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:27.813+0000 7f3c71a00640 1 --2- 172.21.15.118:0/3357090347 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5588c70a6580 0x5588c70b6210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:17:27.825 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:27.813+0000 7f3c71000640 1 -- 172.21.15.118:0/3357090347 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5588c70a5bc0 msgr2=0x5588c70af1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:17:27.825 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:27.813+0000 7f3c71000640 1 --2- 172.21.15.118:0/3357090347 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5588c70a5bc0 0x5588c70af1d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:17:27.825 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:27.813+0000 7f3c71000640 1 -- 172.21.15.118:0/3357090347 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5588c70a6580 msgr2=0x5588c70b6210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:17:27.825 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:27.813+0000 7f3c71000640 1 --2- 172.21.15.118:0/3357090347 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5588c70a6580 0x5588c70b6210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:17:27.825 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:27.813+0000 7f3c71000640 1 -- 172.21.15.118:0/3357090347 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5588c6d1d3f0 con 0x5588c70a49f0 2024-03-24T00:17:27.825 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:27.813+0000 7f3c6be00640 1 --2- 172.21.15.118:0/3357090347 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5588c70a5bc0 0x5588c70af1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T00:17:27.825 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:27.814+0000 7f3c71000640 1 --2- 172.21.15.118:0/3357090347 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5588c70a49f0 0x5588c70aec90 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7f3c60003640 tx=0x7f3c60004730 comp rx=0 tx=0).ready entity=mon.1 client_cookie=69c8cc5bd49f99b3 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:17:27.825 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:27.814+0000 7f3c52a00640 10 client.0 ms_handle_connect on v2:172.21.15.193:3300/0 2024-03-24T00:17:27.826 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:27.814+0000 7f3c52a00640 1 -- 172.21.15.118:0/3357090347 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3c6002c060 con 0x5588c70a49f0 2024-03-24T00:17:27.826 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:27.814+0000 7f3c52a00640 1 -- 172.21.15.118:0/3357090347 <== mon.1 v2:172.21.15.193:3300/0 2 ==== osd_map(90..90 src has 1..90) v4 ==== 8277+0+0 (secure 0 0 0) 0x7f3c6000f890 con 0x5588c70a49f0 2024-03-24T00:17:27.826 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:27.814+0000 7f3c76620740 1 -- 172.21.15.118:0/3357090347 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f3c64003e50 con 0x5588c70a49f0 2024-03-24T00:17:27.826 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:27.814+0000 7f3c52a00640 1 -- 172.21.15.118:0/3357090347 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) v2 ==== 24+0+0 (secure 0 0 0) 0x7f3c6000fa90 con 0x5588c70a49f0 2024-03-24T00:17:27.827 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:27.814+0000 7f3c76620740 10 client.25867 fetch_fsmap learned FSMap version 28 2024-03-24T00:17:27.827 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:27.814+0000 7f3c76620740 1 -- 172.21.15.118:0/3357090347 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) v3 -- 0x5588c6de77d0 con 0x5588c70a49f0 2024-03-24T00:17:27.827 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:27.814+0000 7f3c52a00640 1 -- 172.21.15.118:0/3357090347 <== mon.1 v2:172.21.15.193:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f3c64003e50 con 0x5588c70a49f0 2024-03-24T00:17:27.827 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:27.815+0000 7f3c52a00640 1 -- 172.21.15.118:0/3357090347 <== mon.1 v2:172.21.15.193:3300/0 5 ==== fsmap(e 28) v1 ==== 7934+0+0 (secure 0 0 0) 0x7f3c60026030 con 0x5588c70a49f0 2024-03-24T00:17:27.827 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:27.815+0000 7f3c76620740 10 client.25867 fetch_fsmap finished waiting for FSMap version 28 2024-03-24T00:17:27.827 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:27.815+0000 7f3c76620740 10 client.25867 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.c 2024-03-24T00:17:27.827 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:27.815+0000 7f3c76620740 20 client.25867 populate_metadata read hostname 'smithi118' 2024-03-24T00:17:27.827 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:27.815+0000 7f3c76620740 1 --2- 172.21.15.118:0/3357090347 >> [v2:172.21.15.193:6836/2921520053,v1:172.21.15.193:6837/2921520053] conn(0x5588c70ba780 0x5588c70bcbc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:17:27.827 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:27.815+0000 7f3c76620740 4 client.25867 mds_command: new command op to 24511 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-24T00:17:27.828 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:27.815+0000 7f3c76620740 1 -- 172.21.15.118:0/3357090347 --> [v2:172.21.15.193:6836/2921520053,v1:172.21.15.193:6837/2921520053] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x5588c7016800 con 0x5588c70ba780 2024-03-24T00:17:27.828 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:27.815+0000 7f3c6be00640 1 --2- 172.21.15.118:0/3357090347 >> [v2:172.21.15.193:6836/2921520053,v1:172.21.15.193:6837/2921520053] conn(0x5588c70ba780 0x5588c70bcbc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:17:27.828 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:27.816+0000 7f3c6be00640 1 --2- 172.21.15.118:0/3357090347 >> [v2:172.21.15.193:6836/2921520053,v1:172.21.15.193:6837/2921520053] conn(0x5588c70ba780 0x5588c70bcbc0 crc :-1 s=READY pgs=23 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=83e399bc1777f72c server_cookie=e8ab40c063a6f9f1 in_seq=0 out_seq=0 2024-03-24T00:17:27.828 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:27.816+0000 7f3c52a00640 10 client.25867 ms_handle_connect on v2:172.21.15.193:6836/2921520053 2024-03-24T00:17:27.828 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:27.816+0000 7f3c52a00640 1 -- 172.21.15.118:0/3357090347 <== mds.2 v2:172.21.15.193:6836/2921520053 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x5588c7016800 con 0x5588c70ba780 2024-03-24T00:17:27.828 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:27.816+0000 7f3c52a00640 10 client.25867 handle_command_reply: tid=0 2024-03-24T00:17:27.828 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:27.816+0000 7f3c76620740 1 client.25867 shutdown 2024-03-24T00:17:27.828 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:27.817+0000 7f3c76620740 1 -- 172.21.15.118:0/3357090347 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5588c70a49f0 msgr2=0x5588c70aec90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:17:27.829 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:27.817+0000 7f3c76620740 1 --2- 172.21.15.118:0/3357090347 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5588c70a49f0 0x5588c70aec90 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7f3c60003640 tx=0x7f3c60004730 comp rx=0 tx=0).stop 2024-03-24T00:17:27.829 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:27.817+0000 7f3c76620740 1 -- 172.21.15.118:0/3357090347 shutdown_connections 2024-03-24T00:17:27.829 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:27.817+0000 7f3c76620740 1 --2- 172.21.15.118:0/3357090347 >> [v2:172.21.15.193:6836/2921520053,v1:172.21.15.193:6837/2921520053] conn(0x5588c70ba780 0x5588c70bcbc0 crc :-1 s=READY pgs=23 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:17:27.829 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:27.817+0000 7f3c76620740 1 --2- 172.21.15.118:0/3357090347 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5588c70a6580 0x5588c70b6210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:17:27.829 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:27.817+0000 7f3c76620740 1 --2- 172.21.15.118:0/3357090347 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5588c70a5bc0 0x5588c70af1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:17:27.829 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:27.817+0000 7f3c76620740 1 --2- 172.21.15.118:0/3357090347 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5588c70a49f0 0x5588c70aec90 unknown :-1 s=CLOSED pgs=273 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:17:27.829 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:27.817+0000 7f3c76620740 1 -- 172.21.15.118:0/3357090347 >> 172.21.15.118:0/3357090347 conn(0x5588c709fde0 msgr2=0x5588c709f160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:17:27.829 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:27.817+0000 7f3c52a00640 0 client.25867 ms_handle_reset on v2:172.21.15.193:6836/2921520053 2024-03-24T00:17:27.830 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:27.817+0000 7f3c76620740 1 -- 172.21.15.118:0/3357090347 shutdown_connections 2024-03-24T00:17:27.830 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:27.817+0000 7f3c76620740 1 -- 172.21.15.118:0/3357090347 wait complete. 2024-03-24T00:17:27.830 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:27.817+0000 7f3c76620740 20 client.25867 trim_cache size 0 max 16384 2024-03-24T00:17:27.833 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:27.831+0000 7f3c76620740 1 Processor -- start 2024-03-24T00:17:27.833 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:27.831+0000 7f3c76620740 1 -- start start 2024-03-24T00:17:27.833 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:27.831+0000 7f3c76620740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5588c70a49f0 0x5588c709e8d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:17:27.833 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:27.831+0000 7f3c76620740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5588c70a5bc0 0x5588c70b61d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:17:27.833 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:27.831+0000 7f3c76620740 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5588c70a6580 0x5588c709ee10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:17:27.833 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:27.831+0000 7f3c76620740 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3c6c1aa9e0 con 0x5588c70a6580 2024-03-24T00:17:27.833 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:27.831+0000 7f3c76620740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3c6c1ac360 con 0x5588c70a5bc0 2024-03-24T00:17:27.834 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:27.831+0000 7f3c76620740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3c6c1ab040 con 0x5588c70a49f0 2024-03-24T00:17:27.834 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:27.831+0000 7f3c71000640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5588c70a49f0 0x5588c709e8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:17:27.834 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:27.831+0000 7f3c71a00640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5588c70a6580 0x5588c709ee10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:17:27.834 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:27.831+0000 7f3c6be00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5588c70a5bc0 0x5588c70b61d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:17:27.834 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:27.831+0000 7f3c71000640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5588c70a49f0 0x5588c709e8d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.118:37808/0 (socket says 172.21.15.118:37808) 2024-03-24T00:17:27.834 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:27.831+0000 7f3c71000640 1 -- 172.21.15.118:0/2544510195 learned_addr learned my addr 172.21.15.118:0/2544510195 (peer_addr_for_me v2:172.21.15.118:0/0) 2024-03-24T00:17:27.834 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:27.832+0000 7f3c6be00640 1 -- 172.21.15.118:0/2544510195 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5588c70a49f0 msgr2=0x5588c709e8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:17:27.834 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:27.832+0000 7f3c6be00640 1 --2- 172.21.15.118:0/2544510195 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5588c70a49f0 0x5588c709e8d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:17:27.835 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:27.832+0000 7f3c6be00640 1 -- 172.21.15.118:0/2544510195 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5588c70a6580 msgr2=0x5588c709ee10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:17:27.835 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:27.832+0000 7f3c6be00640 1 --2- 172.21.15.118:0/2544510195 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5588c70a6580 0x5588c709ee10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:17:27.835 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:27.832+0000 7f3c6be00640 1 -- 172.21.15.118:0/2544510195 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5588c70a0d50 con 0x5588c70a5bc0 2024-03-24T00:17:27.835 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:27.832+0000 7f3c71a00640 1 --2- 172.21.15.118:0/2544510195 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5588c70a6580 0x5588c709ee10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T00:17:27.835 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:27.832+0000 7f3c6be00640 1 --2- 172.21.15.118:0/2544510195 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5588c70a5bc0 0x5588c70b61d0 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x7f3c580b94f0 tx=0x7f3c580edb50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=1310b24a8e0ffe3e server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:17:27.835 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:27.832+0000 7f3c5d600640 1 -- 172.21.15.118:0/2544510195 <== mon.2 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3c580bd220 con 0x5588c70a5bc0 2024-03-24T00:17:27.835 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:27.832+0000 7f3c5d600640 1 -- 172.21.15.118:0/2544510195 <== mon.2 v2:172.21.15.189:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f3c580bde50 con 0x5588c70a5bc0 2024-03-24T00:17:27.835 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:27.832+0000 7f3c5d600640 1 -- 172.21.15.118:0/2544510195 <== mon.2 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3c580042c0 con 0x5588c70a5bc0 2024-03-24T00:17:27.841 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:27.833+0000 7f3c76620740 1 -- 172.21.15.118:0/2544510195 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5588c70a5bc0 msgr2=0x5588c70b61d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:17:27.842 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:27.833+0000 7f3c76620740 1 --2- 172.21.15.118:0/2544510195 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5588c70a5bc0 0x5588c70b61d0 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x7f3c580b94f0 tx=0x7f3c580edb50 comp rx=0 tx=0).stop 2024-03-24T00:17:27.842 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:27.833+0000 7f3c76620740 1 -- 172.21.15.118:0/2544510195 shutdown_connections 2024-03-24T00:17:27.842 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:27.833+0000 7f3c76620740 1 --2- 172.21.15.118:0/2544510195 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5588c70a6580 0x5588c709ee10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:17:27.842 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:27.833+0000 7f3c76620740 1 --2- 172.21.15.118:0/2544510195 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5588c70a5bc0 0x5588c70b61d0 unknown :-1 s=CLOSED pgs=291 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:17:27.842 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:27.833+0000 7f3c76620740 1 --2- 172.21.15.118:0/2544510195 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5588c70a49f0 0x5588c709e8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:17:27.842 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:27.833+0000 7f3c76620740 1 -- 172.21.15.118:0/2544510195 >> 172.21.15.118:0/2544510195 conn(0x5588c709fde0 msgr2=0x5588c70a01c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:17:27.842 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:27.833+0000 7f3c76620740 1 -- 172.21.15.118:0/2544510195 shutdown_connections 2024-03-24T00:17:27.843 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:27.833+0000 7f3c76620740 1 -- 172.21.15.118:0/2544510195 wait complete. 2024-03-24T00:17:27.843 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:27.834+0000 7f3c76620740 1 -- start start 2024-03-24T00:17:27.843 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:27.834+0000 7f3c76620740 1 Processor -- start 2024-03-24T00:17:27.843 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:27.834+0000 7f3c76620740 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5588c70a49f0 0x5588c70b20e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:17:27.843 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:27.834+0000 7f3c76620740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5588c70a5bc0 0x5588c70b2620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:17:27.843 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:27.834+0000 7f3c76620740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5588c70a6580 0x5588c70b5640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:17:27.843 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:27.834+0000 7f3c76620740 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3c6c1abd00 con 0x5588c70a49f0 2024-03-24T00:17:27.843 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:27.834+0000 7f3c76620740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3c6c1ab6a0 con 0x5588c70a5bc0 2024-03-24T00:17:27.843 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:27.834+0000 7f3c76620740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3c6c1a8ba0 con 0x5588c70a6580 2024-03-24T00:17:27.844 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:27.834+0000 7f3c76620740 10 client.0 _ll_register_callbacks cb 0x7f3c60009eb0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T00:17:27.844 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:27.834+0000 7f3c6be00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5588c70a5bc0 0x5588c70b2620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:17:27.844 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:27.834+0000 7f3c71000640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5588c70a49f0 0x5588c70b20e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:17:27.844 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:27.834+0000 7f3c6be00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5588c70a5bc0 0x5588c70b2620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.118:45432/0 (socket says 172.21.15.118:45432) 2024-03-24T00:17:27.844 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:27.834+0000 7f3c6be00640 1 -- 172.21.15.118:0/355341124 learned_addr learned my addr 172.21.15.118:0/355341124 (peer_addr_for_me v2:172.21.15.118:0/0) 2024-03-24T00:17:27.844 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:27.835+0000 7f3c71a00640 1 --2- 172.21.15.118:0/355341124 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5588c70a6580 0x5588c70b5640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:17:27.844 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:27.835+0000 7f3c6be00640 1 -- 172.21.15.118:0/355341124 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5588c70a6580 msgr2=0x5588c70b5640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:17:27.844 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:27.835+0000 7f3c6be00640 1 --2- 172.21.15.118:0/355341124 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5588c70a6580 0x5588c70b5640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:17:27.845 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:27.835+0000 7f3c6be00640 1 -- 172.21.15.118:0/355341124 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5588c70a49f0 msgr2=0x5588c70b20e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:17:27.845 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:27.835+0000 7f3c6be00640 1 --2- 172.21.15.118:0/355341124 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5588c70a49f0 0x5588c70b20e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:17:27.845 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:27.835+0000 7f3c6be00640 1 -- 172.21.15.118:0/355341124 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5588c70b5c30 con 0x5588c70a5bc0 2024-03-24T00:17:27.845 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:27.835+0000 7f3c6be00640 1 --2- 172.21.15.118:0/355341124 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5588c70a5bc0 0x5588c70b2620 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 crypto rx=0x7f3c58009860 tx=0x7f3c5802bb70 comp rx=0 tx=0).ready entity=mon.2 client_cookie=25acc0584a5d4b61 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:17:27.845 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:27.835+0000 7f3c52a00640 10 client.0 ms_handle_connect on v2:172.21.15.189:3300/0 2024-03-24T00:17:27.845 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:27.835+0000 7f3c52a00640 1 -- 172.21.15.118:0/355341124 <== mon.2 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3c580bcb20 con 0x5588c70a5bc0 2024-03-24T00:17:27.845 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:27.835+0000 7f3c52a00640 1 -- 172.21.15.118:0/355341124 <== mon.2 v2:172.21.15.189:3300/0 2 ==== osd_map(90..90 src has 1..90) v4 ==== 8277+0+0 (secure 0 0 0) 0x7f3c580b9190 con 0x5588c70a5bc0 2024-03-24T00:17:27.845 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:27.835+0000 7f3c76620740 1 -- 172.21.15.118:0/355341124 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f3c580bde50 con 0x5588c70a5bc0 2024-03-24T00:17:27.845 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:27.836+0000 7f3c52a00640 1 -- 172.21.15.118:0/355341124 <== mon.2 v2:172.21.15.189:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) v2 ==== 24+0+0 (secure 0 0 0) 0x7f3c580c0c70 con 0x5588c70a5bc0 2024-03-24T00:17:27.845 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:27.836+0000 7f3c76620740 10 client.25889 fetch_fsmap learned FSMap version 28 2024-03-24T00:17:27.846 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:27.836+0000 7f3c76620740 1 -- 172.21.15.118:0/355341124 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) v3 -- 0x5588c7116860 con 0x5588c70a5bc0 2024-03-24T00:17:27.846 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:27.836+0000 7f3c52a00640 1 -- 172.21.15.118:0/355341124 <== mon.2 v2:172.21.15.189:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f3c580bde50 con 0x5588c70a5bc0 2024-03-24T00:17:27.846 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:27.836+0000 7f3c52a00640 1 -- 172.21.15.118:0/355341124 <== mon.2 v2:172.21.15.189:3300/0 5 ==== fsmap(e 28) v1 ==== 7934+0+0 (secure 0 0 0) 0x7f3c5803c030 con 0x5588c70a5bc0 2024-03-24T00:17:27.846 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:27.836+0000 7f3c76620740 10 client.25889 fetch_fsmap finished waiting for FSMap version 28 2024-03-24T00:17:27.846 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:27.836+0000 7f3c76620740 10 client.25889 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.c 2024-03-24T00:17:27.846 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:27.836+0000 7f3c76620740 20 client.25889 populate_metadata read hostname 'smithi118' 2024-03-24T00:17:27.846 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:27.836+0000 7f3c76620740 1 --2- 172.21.15.118:0/355341124 >> [v2:172.21.15.193:6836/2921520053,v1:172.21.15.193:6837/2921520053] conn(0x5588c71171b0 0x5588c71195d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:17:27.847 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:27.836+0000 7f3c76620740 4 client.25889 mds_command: new command op to 24511 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-24T00:17:27.847 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:27.836+0000 7f3c76620740 1 -- 172.21.15.118:0/355341124 --> [v2:172.21.15.193:6836/2921520053,v1:172.21.15.193:6837/2921520053] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x5588c7010b50 con 0x5588c71171b0 2024-03-24T00:17:27.847 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:27.836+0000 7f3c71000640 1 --2- 172.21.15.118:0/355341124 >> [v2:172.21.15.193:6836/2921520053,v1:172.21.15.193:6837/2921520053] conn(0x5588c71171b0 0x5588c71195d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:17:27.847 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:27.837+0000 7f3c71000640 1 --2- 172.21.15.118:0/355341124 >> [v2:172.21.15.193:6836/2921520053,v1:172.21.15.193:6837/2921520053] conn(0x5588c71171b0 0x5588c71195d0 crc :-1 s=READY pgs=24 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=e628a225d2bbcd8f server_cookie=c1fc815c49324cd in_seq=0 out_seq=0 2024-03-24T00:17:27.847 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:27.837+0000 7f3c52a00640 10 client.25889 ms_handle_connect on v2:172.21.15.193:6836/2921520053 2024-03-24T00:17:27.847 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:27.837+0000 7f3c52a00640 1 -- 172.21.15.118:0/355341124 <== mds.2 v2:172.21.15.193:6836/2921520053 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x5588c7010b50 con 0x5588c71171b0 2024-03-24T00:17:27.847 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:27.837+0000 7f3c52a00640 10 client.25889 handle_command_reply: tid=0 2024-03-24T00:17:27.847 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:27.837+0000 7f3c76620740 1 client.25889 shutdown 2024-03-24T00:17:27.848 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:27.837+0000 7f3c76620740 1 -- 172.21.15.118:0/355341124 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5588c70a5bc0 msgr2=0x5588c70b2620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:17:27.848 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:27.837+0000 7f3c76620740 1 --2- 172.21.15.118:0/355341124 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5588c70a5bc0 0x5588c70b2620 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 crypto rx=0x7f3c58009860 tx=0x7f3c5802bb70 comp rx=0 tx=0).stop 2024-03-24T00:17:27.848 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:27.837+0000 7f3c76620740 1 -- 172.21.15.118:0/355341124 shutdown_connections 2024-03-24T00:17:27.848 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:27.837+0000 7f3c76620740 1 --2- 172.21.15.118:0/355341124 >> [v2:172.21.15.193:6836/2921520053,v1:172.21.15.193:6837/2921520053] conn(0x5588c71171b0 0x5588c71195d0 crc :-1 s=READY pgs=24 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:17:27.848 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:27.837+0000 7f3c76620740 1 --2- 172.21.15.118:0/355341124 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5588c70a6580 0x5588c70b5640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:17:27.848 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:27.837+0000 7f3c76620740 1 --2- 172.21.15.118:0/355341124 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5588c70a5bc0 0x5588c70b2620 unknown :-1 s=CLOSED pgs=292 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:17:27.848 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:27.837+0000 7f3c76620740 1 --2- 172.21.15.118:0/355341124 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5588c70a49f0 0x5588c70b20e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:17:27.848 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:27.837+0000 7f3c52a00640 0 client.25889 ms_handle_reset on v2:172.21.15.193:6836/2921520053 2024-03-24T00:17:27.848 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:27.837+0000 7f3c76620740 1 -- 172.21.15.118:0/355341124 >> 172.21.15.118:0/355341124 conn(0x5588c709fde0 msgr2=0x5588c70a3040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:17:27.849 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:27.838+0000 7f3c76620740 1 -- 172.21.15.118:0/355341124 shutdown_connections 2024-03-24T00:17:27.849 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:27.838+0000 7f3c76620740 1 -- 172.21.15.118:0/355341124 wait complete. 2024-03-24T00:17:27.849 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:27.838+0000 7f3c76620740 20 client.25889 trim_cache size 0 max 16384 2024-03-24T00:17:27.849 INFO:teuthology.orchestra.run.smithi118.stdout:[] 2024-03-24T00:17:27.850 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:27.838+0000 7f3c72400640 1 -- 172.21.15.118:0/2319075406 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7f3c540a2880 msgr2=0x7f3c540a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:17:27.851 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:27.838+0000 7f3c72400640 1 --2- 172.21.15.118:0/2319075406 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7f3c540a2880 0x7f3c540a4d40 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 crypto rx=0x7f3c60009fd0 tx=0x7f3c60009260 comp rx=0 tx=0).stop 2024-03-24T00:17:27.851 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:27.838+0000 7f3c72400640 1 -- 172.21.15.118:0/2319075406 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3c6c102790 msgr2=0x7f3c6c073a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:17:27.851 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:27.838+0000 7f3c72400640 1 --2- 172.21.15.118:0/2319075406 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3c6c102790 0x7f3c6c073a90 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 crypto rx=0x7f3c5802c0b0 tx=0x7f3c58032430 comp rx=0 tx=0).stop 2024-03-24T00:17:27.851 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:27.839+0000 7f3c72400640 1 -- 172.21.15.118:0/2319075406 shutdown_connections 2024-03-24T00:17:27.851 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:27.839+0000 7f3c72400640 1 --2- 172.21.15.118:0/2319075406 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7f3c540a2880 0x7f3c540a4d40 unknown :-1 s=CLOSED pgs=300 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:17:27.851 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:27.839+0000 7f3c72400640 1 --2- 172.21.15.118:0/2319075406 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f3c6c103150 0x7f3c6c073fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:17:27.851 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:27.839+0000 7f3c72400640 1 --2- 172.21.15.118:0/2319075406 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3c6c102790 0x7f3c6c073a90 unknown :-1 s=CLOSED pgs=289 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:17:27.851 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:27.839+0000 7f3c72400640 1 --2- 172.21.15.118:0/2319075406 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f3c6c1015a0 0x7f3c6c075450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:17:27.851 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:27.839+0000 7f3c72400640 1 -- 172.21.15.118:0/2319075406 >> 172.21.15.118:0/2319075406 conn(0x7f3c6c0fcb10 msgr2=0x7f3c6c105a30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:17:27.852 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:27.839+0000 7f3c72400640 1 -- 172.21.15.118:0/2319075406 shutdown_connections 2024-03-24T00:17:27.852 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:27.839+0000 7f3c72400640 1 -- 172.21.15.118:0/2319075406 wait complete. 2024-03-24T00:17:27.855 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:17:27 smithi118 ceph-mon[31535]: from='client.? 172.21.15.118:0/2171715402' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-24T00:17:27.858 INFO:teuthology.orchestra.run:Running command with timeout 120 2024-03-24T00:17:27.858 DEBUG:teuthology.orchestra.run.smithi118:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:1 damage ls 2024-03-24T00:17:27.896 INFO:tasks.workunit.client.0.smithi118.stdout:checking for ssize_t... yes 2024-03-24T00:17:28.005 INFO:tasks.workunit.client.0.smithi118.stdout:checking for sig_atomic_t... yes 2024-03-24T00:17:28.091 INFO:tasks.workunit.client.0.smithi118.stdout:checking whether gcc needs -traditional... no 2024-03-24T00:17:28.131 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:28.130+0000 7f0af6600640 1 -- 172.21.15.118:0/3015804912 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f0af01038e0 msgr2=0x7f0af0103d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:17:28.132 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:28.130+0000 7f0af6600640 1 --2- 172.21.15.118:0/3015804912 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f0af01038e0 0x7f0af0103d00 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7f0ae0009900 tx=0x7f0ae002bc30 comp rx=0 tx=0).stop 2024-03-24T00:17:28.132 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:28.130+0000 7f0af6600640 1 -- 172.21.15.118:0/3015804912 shutdown_connections 2024-03-24T00:17:28.132 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:28.130+0000 7f0af6600640 1 --2- 172.21.15.118:0/3015804912 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0af0105490 0x7f0af010bd20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:17:28.132 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:28.130+0000 7f0af6600640 1 --2- 172.21.15.118:0/3015804912 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f0af0104ad0 0x7f0af0104f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:17:28.132 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:28.130+0000 7f0af6600640 1 --2- 172.21.15.118:0/3015804912 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f0af01038e0 0x7f0af0103d00 unknown :-1 s=CLOSED pgs=274 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:17:28.132 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:28.130+0000 7f0af6600640 1 -- 172.21.15.118:0/3015804912 >> 172.21.15.118:0/3015804912 conn(0x7f0af00fee50 msgr2=0x7f0af0101290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:17:28.132 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:28.131+0000 7f0af6600640 1 -- 172.21.15.118:0/3015804912 shutdown_connections 2024-03-24T00:17:28.132 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:28.131+0000 7f0af6600640 1 -- 172.21.15.118:0/3015804912 wait complete. 2024-03-24T00:17:28.136 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:28.131+0000 7f0af6600640 1 Processor -- start 2024-03-24T00:17:28.136 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:28.132+0000 7f0af6600640 1 -- start start 2024-03-24T00:17:28.136 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:28.132+0000 7f0af6600640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f0af01038e0 0x7f0af0199bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:17:28.136 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:28.132+0000 7f0af6600640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f0af0104ad0 0x7f0af019a130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:17:28.136 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:28.132+0000 7f0af6600640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0af0105490 0x7f0af01a11a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:17:28.136 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:28.132+0000 7f0af6600640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0af010f5b0 con 0x7f0af01038e0 2024-03-24T00:17:28.137 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:28.132+0000 7f0af6600640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0af010f430 con 0x7f0af0105490 2024-03-24T00:17:28.137 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:28.132+0000 7f0af6600640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0af010f730 con 0x7f0af0104ad0 2024-03-24T00:17:28.137 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:28.132+0000 7f0aeea00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f0af0104ad0 0x7f0af019a130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:17:28.137 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:28.132+0000 7f0aefe00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0af0105490 0x7f0af01a11a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:17:28.137 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:28.132+0000 7f0aeea00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f0af0104ad0 0x7f0af019a130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.118:37832/0 (socket says 172.21.15.118:37832) 2024-03-24T00:17:28.137 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:28.132+0000 7f0aefe00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0af0105490 0x7f0af01a11a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.118:45454/0 (socket says 172.21.15.118:45454) 2024-03-24T00:17:28.137 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:28.132+0000 7f0aeea00640 1 -- 172.21.15.118:0/2023590726 learned_addr learned my addr 172.21.15.118:0/2023590726 (peer_addr_for_me v2:172.21.15.118:0/0) 2024-03-24T00:17:28.138 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:28.132+0000 7f0aef400640 1 --2- 172.21.15.118:0/2023590726 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f0af01038e0 0x7f0af0199bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:17:28.138 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:28.133+0000 7f0aeea00640 1 -- 172.21.15.118:0/2023590726 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0af0105490 msgr2=0x7f0af01a11a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:17:28.138 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:28.133+0000 7f0aeea00640 1 --2- 172.21.15.118:0/2023590726 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0af0105490 0x7f0af01a11a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:17:28.138 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:28.133+0000 7f0aeea00640 1 -- 172.21.15.118:0/2023590726 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f0af01038e0 msgr2=0x7f0af0199bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:17:28.138 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:28.133+0000 7f0aeea00640 1 --2- 172.21.15.118:0/2023590726 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f0af01038e0 0x7f0af0199bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:17:28.138 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:28.133+0000 7f0aeea00640 1 -- 172.21.15.118:0/2023590726 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0af01a1810 con 0x7f0af0104ad0 2024-03-24T00:17:28.138 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:28.133+0000 7f0aefe00640 1 --2- 172.21.15.118:0/2023590726 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0af0105490 0x7f0af01a11a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T00:17:28.138 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:28.133+0000 7f0aef400640 1 --2- 172.21.15.118:0/2023590726 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f0af01038e0 0x7f0af0199bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T00:17:28.138 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:28.133+0000 7f0aeea00640 1 --2- 172.21.15.118:0/2023590726 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f0af0104ad0 0x7f0af019a130 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7f0adc00e9f0 tx=0x7f0adc00eeb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:17:28.139 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:28.133+0000 7f0ae7e00640 1 -- 172.21.15.118:0/2023590726 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0adc00cd80 con 0x7f0af0104ad0 2024-03-24T00:17:28.139 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:28.133+0000 7f0ae7e00640 1 -- 172.21.15.118:0/2023590726 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f0adc010910 con 0x7f0af0104ad0 2024-03-24T00:17:28.139 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:28.133+0000 7f0af6600640 1 -- 172.21.15.118:0/2023590726 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0af01a1b00 con 0x7f0af0104ad0 2024-03-24T00:17:28.139 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:28.133+0000 7f0ae7e00640 1 -- 172.21.15.118:0/2023590726 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0adc018990 con 0x7f0af0104ad0 2024-03-24T00:17:28.139 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:28.133+0000 7f0af6600640 1 -- 172.21.15.118:0/2023590726 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0af01a20a0 con 0x7f0af0104ad0 2024-03-24T00:17:28.149 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:28.138+0000 7f0af93b0740 1 Processor -- start 2024-03-24T00:17:28.149 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:28.138+0000 7f0af93b0740 1 -- start start 2024-03-24T00:17:28.149 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:28.138+0000 7f0af93b0740 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x56029fecc810 0x56029feccc30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:17:28.149 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:28.138+0000 7f0af93b0740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56029fecd9e0 0x56029fecde60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:17:28.149 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:28.138+0000 7f0af93b0740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56029fece3a0 0x56029fed4c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:17:28.149 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:28.139+0000 7f0af93b0740 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x56029fe312c0 con 0x56029fecc810 2024-03-24T00:17:28.150 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:28.139+0000 7f0af93b0740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x56029fdd9c90 con 0x56029fece3a0 2024-03-24T00:17:28.150 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:28.139+0000 7f0af93b0740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x56029fe2d360 con 0x56029fecd9e0 2024-03-24T00:17:28.150 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:28.139+0000 7f0aefe00640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x56029fecc810 0x56029feccc30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:17:28.150 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:28.139+0000 7f0aef400640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56029fecd9e0 0x56029fecde60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:17:28.150 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:28.139+0000 7f0aef400640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56029fecd9e0 0x56029fecde60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.118:37836/0 (socket says 172.21.15.118:37836) 2024-03-24T00:17:28.150 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:28.139+0000 7f0aefe00640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x56029fecc810 0x56029feccc30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.118:3300/0 says I am v2:172.21.15.118:38450/0 (socket says 172.21.15.118:38450) 2024-03-24T00:17:28.150 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:28.139+0000 7f0aef400640 1 -- 172.21.15.118:0/2858428482 learned_addr learned my addr 172.21.15.118:0/2858428482 (peer_addr_for_me v2:172.21.15.118:0/0) 2024-03-24T00:17:28.150 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:28.139+0000 7f0ae7e00640 1 -- 172.21.15.118:0/2023590726 <== mon.1 v2:172.21.15.193:3300/0 4 ==== mgrmap(e 20) v1 ==== 137291+0+0 (secure 0 0 0) 0x7f0adc010ac0 con 0x7f0af0104ad0 2024-03-24T00:17:28.151 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:28.139+0000 7f0aef400640 1 -- 172.21.15.118:0/2858428482 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56029fece3a0 msgr2=0x56029fed4c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:17:28.151 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:28.139+0000 7f0aef400640 1 --2- 172.21.15.118:0/2858428482 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56029fece3a0 0x56029fed4c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:17:28.151 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:28.139+0000 7f0aef400640 1 -- 172.21.15.118:0/2858428482 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x56029fecc810 msgr2=0x56029feccc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:17:28.151 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:28.139+0000 7f0aef400640 1 --2- 172.21.15.118:0/2858428482 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x56029fecc810 0x56029feccc30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:17:28.151 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:28.139+0000 7f0aef400640 1 -- 172.21.15.118:0/2858428482 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x56029fdfe080 con 0x56029fecd9e0 2024-03-24T00:17:28.151 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:28.140+0000 7f0aef400640 1 --2- 172.21.15.118:0/2858428482 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56029fecd9e0 0x56029fecde60 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7f0ae0000c00 tx=0x7f0ae00095e0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=1c7155a63797d7d server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:17:28.151 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:28.140+0000 7f0ae4c00640 1 -- 172.21.15.118:0/2858428482 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0ae0044020 con 0x56029fecd9e0 2024-03-24T00:17:28.152 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:28.140+0000 7f0ae4c00640 1 -- 172.21.15.118:0/2858428482 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f0ae002cc40 con 0x56029fecd9e0 2024-03-24T00:17:28.152 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:28.140+0000 7f0ae4c00640 1 -- 172.21.15.118:0/2858428482 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0ae0004920 con 0x56029fecd9e0 2024-03-24T00:17:28.152 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:28.141+0000 7f0ae7e00640 1 --2- 172.21.15.118:0/2023590726 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7f0ad40a2880 0x7f0ad40a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:17:28.152 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:28.141+0000 7f0ae7e00640 1 -- 172.21.15.118:0/2023590726 <== mon.1 v2:172.21.15.193:3300/0 5 ==== osd_map(90..90 src has 1..90) v4 ==== 8277+0+0 (secure 0 0 0) 0x7f0adc0cee30 con 0x7f0af0104ad0 2024-03-24T00:17:28.152 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:28.141+0000 7f0aefe00640 1 --2- 172.21.15.118:0/2023590726 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7f0ad40a2880 0x7f0ad40a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:17:28.152 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:28.141+0000 7f0af93b0740 1 -- 172.21.15.118:0/2858428482 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56029fecd9e0 msgr2=0x56029fecde60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:17:28.153 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:28.141+0000 7f0af93b0740 1 --2- 172.21.15.118:0/2858428482 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56029fecd9e0 0x56029fecde60 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7f0ae0000c00 tx=0x7f0ae00095e0 comp rx=0 tx=0).stop 2024-03-24T00:17:28.153 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:28.141+0000 7f0af93b0740 1 -- 172.21.15.118:0/2858428482 shutdown_connections 2024-03-24T00:17:28.153 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:28.141+0000 7f0af93b0740 1 --2- 172.21.15.118:0/2858428482 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56029fece3a0 0x56029fed4c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:17:28.153 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:28.141+0000 7f0af93b0740 1 --2- 172.21.15.118:0/2858428482 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56029fecd9e0 0x56029fecde60 unknown :-1 s=CLOSED pgs=276 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:17:28.153 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:28.141+0000 7f0af93b0740 1 --2- 172.21.15.118:0/2858428482 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x56029fecc810 0x56029feccc30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:17:28.153 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:28.141+0000 7f0af93b0740 1 -- 172.21.15.118:0/2858428482 >> 172.21.15.118:0/2858428482 conn(0x56029fec7c00 msgr2=0x56029fec8000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:17:28.153 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:28.141+0000 7f0af93b0740 1 -- 172.21.15.118:0/2858428482 shutdown_connections 2024-03-24T00:17:28.153 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:28.142+0000 7f0af93b0740 1 -- 172.21.15.118:0/2858428482 wait complete. 2024-03-24T00:17:28.153 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:28.142+0000 7f0aefe00640 1 --2- 172.21.15.118:0/2023590726 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7f0ad40a2880 0x7f0ad40a4d40 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 crypto rx=0x56029fecd8e0 tx=0x7f0ae8009500 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:17:28.154 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:28.142+0000 7f0af93b0740 1 -- start start 2024-03-24T00:17:28.154 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:28.142+0000 7f0af93b0740 1 Processor -- start 2024-03-24T00:17:28.154 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:28.143+0000 7f0af93b0740 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x56029fecc810 0x56029fed6ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:17:28.154 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:28.143+0000 7f0af93b0740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56029fecd9e0 0x56029fed6ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:17:28.154 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:28.143+0000 7f0af93b0740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56029fece3a0 0x56029fede030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:17:28.154 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:28.143+0000 7f0af93b0740 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x56029fed5350 con 0x56029fecc810 2024-03-24T00:17:28.154 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:28.143+0000 7f0af93b0740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x56029fded0d0 con 0x56029fece3a0 2024-03-24T00:17:28.154 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:28.143+0000 7f0af93b0740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x56029fed54d0 con 0x56029fecd9e0 2024-03-24T00:17:28.155 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:28.143+0000 7f0af93b0740 10 client.0 _ll_register_callbacks cb 0x56029fdf1e80 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T00:17:28.155 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:28.143+0000 7f0aeea00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56029fecd9e0 0x56029fed6ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:17:28.155 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:28.143+0000 7f0aefe00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56029fece3a0 0x56029fede030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:17:28.155 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:28.143+0000 7f0aeea00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56029fecd9e0 0x56029fed6ff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.118:37852/0 (socket says 172.21.15.118:37852) 2024-03-24T00:17:28.155 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:28.143+0000 7f0aefe00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56029fece3a0 0x56029fede030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.118:45478/0 (socket says 172.21.15.118:45478) 2024-03-24T00:17:28.155 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:28.143+0000 7f0aeea00640 1 -- 172.21.15.118:0/98367357 learned_addr learned my addr 172.21.15.118:0/98367357 (peer_addr_for_me v2:172.21.15.118:0/0) 2024-03-24T00:17:28.155 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:28.143+0000 7f0aef400640 1 --2- 172.21.15.118:0/98367357 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x56029fecc810 0x56029fed6ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:17:28.155 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:28.143+0000 7f0aeea00640 1 -- 172.21.15.118:0/98367357 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56029fece3a0 msgr2=0x56029fede030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:17:28.155 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:28.143+0000 7f0aeea00640 1 --2- 172.21.15.118:0/98367357 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56029fece3a0 0x56029fede030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:17:28.156 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:28.143+0000 7f0aeea00640 1 -- 172.21.15.118:0/98367357 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x56029fecc810 msgr2=0x56029fed6ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:17:28.156 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:28.143+0000 7f0aeea00640 1 --2- 172.21.15.118:0/98367357 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x56029fecc810 0x56029fed6ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:17:28.156 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:28.143+0000 7f0aeea00640 1 -- 172.21.15.118:0/98367357 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x56029fb453f0 con 0x56029fecd9e0 2024-03-24T00:17:28.156 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:28.144+0000 7f0aeea00640 1 --2- 172.21.15.118:0/98367357 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56029fecd9e0 0x56029fed6ff0 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7f0adc097ba0 tx=0x7f0adc09be40 comp rx=0 tx=0).ready entity=mon.1 client_cookie=af8accebc4fec179 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:17:28.156 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:28.144+0000 7f0ad9600640 10 client.0 ms_handle_connect on v2:172.21.15.193:3300/0 2024-03-24T00:17:28.156 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:28.144+0000 7f0ad9600640 1 -- 172.21.15.118:0/98367357 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0adc0a3ee0 con 0x56029fecd9e0 2024-03-24T00:17:28.156 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:28.144+0000 7f0ad9600640 1 -- 172.21.15.118:0/98367357 <== mon.1 v2:172.21.15.193:3300/0 2 ==== osd_map(90..90 src has 1..90) v4 ==== 8277+0+0 (secure 0 0 0) 0x7f0adc018b30 con 0x56029fecd9e0 2024-03-24T00:17:28.156 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:28.144+0000 7f0af93b0740 1 -- 172.21.15.118:0/98367357 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f0ae002cc40 con 0x56029fecd9e0 2024-03-24T00:17:28.157 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:28.144+0000 7f0ad9600640 1 -- 172.21.15.118:0/98367357 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) v2 ==== 24+0+0 (secure 0 0 0) 0x7f0adc018d30 con 0x56029fecd9e0 2024-03-24T00:17:28.157 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:28.144+0000 7f0af93b0740 10 client.25885 fetch_fsmap learned FSMap version 28 2024-03-24T00:17:28.157 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:28.144+0000 7f0af93b0740 1 -- 172.21.15.118:0/98367357 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) v3 -- 0x56029fc0f7d0 con 0x56029fecd9e0 2024-03-24T00:17:28.157 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:28.144+0000 7f0ad9600640 1 -- 172.21.15.118:0/98367357 <== mon.1 v2:172.21.15.193:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f0ae002cc40 con 0x56029fecd9e0 2024-03-24T00:17:28.157 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:28.145+0000 7f0ad9600640 1 -- 172.21.15.118:0/98367357 <== mon.1 v2:172.21.15.193:3300/0 5 ==== fsmap(e 28) v1 ==== 7934+0+0 (secure 0 0 0) 0x7f0adc097690 con 0x56029fecd9e0 2024-03-24T00:17:28.157 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:28.145+0000 7f0af93b0740 10 client.25885 fetch_fsmap finished waiting for FSMap version 28 2024-03-24T00:17:28.157 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:28.145+0000 7f0af93b0740 10 client.25885 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.g 2024-03-24T00:17:28.157 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:28.145+0000 7f0af93b0740 20 client.25885 populate_metadata read hostname 'smithi118' 2024-03-24T00:17:28.157 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:28.145+0000 7f0af93b0740 1 --2- 172.21.15.118:0/98367357 >> [v2:172.21.15.118:6838/3285788503,v1:172.21.15.118:6839/3285788503] conn(0x56029fee2430 0x56029fee4820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:17:28.158 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:28.145+0000 7f0af93b0740 4 client.25885 mds_command: new command op to 24550 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-24T00:17:28.158 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:28.145+0000 7f0af93b0740 1 -- 172.21.15.118:0/98367357 --> [v2:172.21.15.118:6838/3285788503,v1:172.21.15.118:6839/3285788503] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x56029fe3e800 con 0x56029fee2430 2024-03-24T00:17:28.158 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:28.145+0000 7f0aef400640 1 --2- 172.21.15.118:0/98367357 >> [v2:172.21.15.118:6838/3285788503,v1:172.21.15.118:6839/3285788503] conn(0x56029fee2430 0x56029fee4820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:17:28.158 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:28.146+0000 7f0aef400640 1 --2- 172.21.15.118:0/98367357 >> [v2:172.21.15.118:6838/3285788503,v1:172.21.15.118:6839/3285788503] conn(0x56029fee2430 0x56029fee4820 crc :-1 s=READY pgs=26 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=392344c01060dd68 server_cookie=d1a7a44be38bc983 in_seq=0 out_seq=0 2024-03-24T00:17:28.158 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:28.146+0000 7f0ad9600640 10 client.25885 ms_handle_connect on v2:172.21.15.118:6838/3285788503 2024-03-24T00:17:28.158 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:28.146+0000 7f0ad9600640 1 -- 172.21.15.118:0/98367357 <== mds.1 v2:172.21.15.118:6838/3285788503 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x56029fe3e800 con 0x56029fee2430 2024-03-24T00:17:28.158 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:28.146+0000 7f0ad9600640 10 client.25885 handle_command_reply: tid=0 2024-03-24T00:17:28.158 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:28.146+0000 7f0af93b0740 1 client.25885 shutdown 2024-03-24T00:17:28.159 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:28.147+0000 7f0af93b0740 1 -- 172.21.15.118:0/98367357 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56029fecd9e0 msgr2=0x56029fed6ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:17:28.159 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:28.147+0000 7f0af93b0740 1 --2- 172.21.15.118:0/98367357 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56029fecd9e0 0x56029fed6ff0 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7f0adc097ba0 tx=0x7f0adc09be40 comp rx=0 tx=0).stop 2024-03-24T00:17:28.159 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:28.147+0000 7f0af93b0740 1 -- 172.21.15.118:0/98367357 shutdown_connections 2024-03-24T00:17:28.159 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:28.147+0000 7f0af93b0740 1 --2- 172.21.15.118:0/98367357 >> [v2:172.21.15.118:6838/3285788503,v1:172.21.15.118:6839/3285788503] conn(0x56029fee2430 0x56029fee4820 crc :-1 s=READY pgs=26 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:17:28.159 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:28.147+0000 7f0af93b0740 1 --2- 172.21.15.118:0/98367357 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56029fece3a0 0x56029fede030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:17:28.159 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:28.147+0000 7f0ad9600640 0 client.25885 ms_handle_reset on v2:172.21.15.118:6838/3285788503 2024-03-24T00:17:28.159 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:28.147+0000 7f0af93b0740 1 --2- 172.21.15.118:0/98367357 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56029fecd9e0 0x56029fed6ff0 unknown :-1 s=CLOSED pgs=277 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:17:28.159 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:28.147+0000 7f0af93b0740 1 --2- 172.21.15.118:0/98367357 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x56029fecc810 0x56029fed6ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:17:28.159 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:28.147+0000 7f0af93b0740 1 -- 172.21.15.118:0/98367357 >> 172.21.15.118:0/98367357 conn(0x56029fec7c00 msgr2=0x56029fec6eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:17:28.160 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:28.147+0000 7f0af93b0740 1 -- 172.21.15.118:0/98367357 shutdown_connections 2024-03-24T00:17:28.160 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:28.147+0000 7f0af93b0740 1 -- 172.21.15.118:0/98367357 wait complete. 2024-03-24T00:17:28.160 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:28.147+0000 7f0af93b0740 20 client.25885 trim_cache size 0 max 16384 2024-03-24T00:17:28.169 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:28.161+0000 7f0af93b0740 1 Processor -- start 2024-03-24T00:17:28.169 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:28.161+0000 7f0af93b0740 1 -- start start 2024-03-24T00:17:28.169 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:28.161+0000 7f0af93b0740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56029fecc810 0x56029fed60d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:17:28.170 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:28.161+0000 7f0af93b0740 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x56029fecd9e0 0x56029feddff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:17:28.170 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:28.161+0000 7f0af93b0740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56029fece3a0 0x56029fed0ac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:17:28.170 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:28.161+0000 7f0af93b0740 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0af01aac60 con 0x56029fecd9e0 2024-03-24T00:17:28.170 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:28.161+0000 7f0af93b0740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0af01ac5e0 con 0x56029fecc810 2024-03-24T00:17:28.170 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:28.161+0000 7f0af93b0740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0af01ab2c0 con 0x56029fece3a0 2024-03-24T00:17:28.170 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:28.161+0000 7f0aef400640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56029fecc810 0x56029fed60d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:17:28.170 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:28.161+0000 7f0aefe00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56029fece3a0 0x56029fed0ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:17:28.170 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:28.161+0000 7f0aef400640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56029fecc810 0x56029fed60d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.118:45486/0 (socket says 172.21.15.118:45486) 2024-03-24T00:17:28.171 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:28.161+0000 7f0aef400640 1 -- 172.21.15.118:0/2079297210 learned_addr learned my addr 172.21.15.118:0/2079297210 (peer_addr_for_me v2:172.21.15.118:0/0) 2024-03-24T00:17:28.171 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:28.161+0000 7f0aeea00640 1 --2- 172.21.15.118:0/2079297210 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x56029fecd9e0 0x56029feddff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:17:28.171 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:28.161+0000 7f0aef400640 1 -- 172.21.15.118:0/2079297210 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56029fece3a0 msgr2=0x56029fed0ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:17:28.171 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:28.161+0000 7f0aef400640 1 --2- 172.21.15.118:0/2079297210 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56029fece3a0 0x56029fed0ac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:17:28.171 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:28.161+0000 7f0aef400640 1 -- 172.21.15.118:0/2079297210 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x56029fecd9e0 msgr2=0x56029feddff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:17:28.171 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:28.161+0000 7f0aef400640 1 --2- 172.21.15.118:0/2079297210 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x56029fecd9e0 0x56029feddff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:17:28.171 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:28.161+0000 7f0aef400640 1 -- 172.21.15.118:0/2079297210 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x56029fed68a0 con 0x56029fecc810 2024-03-24T00:17:28.171 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:28.162+0000 7f0aefe00640 1 --2- 172.21.15.118:0/2079297210 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56029fece3a0 0x56029fed0ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T00:17:28.172 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:28.162+0000 7f0aef400640 1 --2- 172.21.15.118:0/2079297210 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56029fecc810 0x56029fed60d0 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x7f0ae002c640 tx=0x7f0ae0005fd0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=3122d721882909e2 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:17:28.172 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:28.162+0000 7f0adbe00640 1 -- 172.21.15.118:0/2079297210 <== mon.2 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0ae0044040 con 0x56029fecc810 2024-03-24T00:17:28.172 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:28.162+0000 7f0adbe00640 1 -- 172.21.15.118:0/2079297210 <== mon.2 v2:172.21.15.189:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f0ae002cc20 con 0x56029fecc810 2024-03-24T00:17:28.172 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:28.162+0000 7f0adbe00640 1 -- 172.21.15.118:0/2079297210 <== mon.2 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0ae00406d0 con 0x56029fecc810 2024-03-24T00:17:28.172 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:28.163+0000 7f0af93b0740 1 -- 172.21.15.118:0/2079297210 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56029fecc810 msgr2=0x56029fed60d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:17:28.172 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:28.163+0000 7f0af93b0740 1 --2- 172.21.15.118:0/2079297210 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56029fecc810 0x56029fed60d0 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x7f0ae002c640 tx=0x7f0ae0005fd0 comp rx=0 tx=0).stop 2024-03-24T00:17:28.172 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:28.163+0000 7f0af93b0740 1 -- 172.21.15.118:0/2079297210 shutdown_connections 2024-03-24T00:17:28.172 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:28.163+0000 7f0af93b0740 1 --2- 172.21.15.118:0/2079297210 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56029fece3a0 0x56029fed0ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:17:28.172 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:28.163+0000 7f0af93b0740 1 --2- 172.21.15.118:0/2079297210 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x56029fecd9e0 0x56029feddff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:17:28.173 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:28.163+0000 7f0af93b0740 1 --2- 172.21.15.118:0/2079297210 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56029fecc810 0x56029fed60d0 unknown :-1 s=CLOSED pgs=293 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:17:28.173 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:28.163+0000 7f0af93b0740 1 -- 172.21.15.118:0/2079297210 >> 172.21.15.118:0/2079297210 conn(0x56029fec7c00 msgr2=0x56029fecb6f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:17:28.173 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:28.163+0000 7f0af93b0740 1 -- 172.21.15.118:0/2079297210 shutdown_connections 2024-03-24T00:17:28.173 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:28.163+0000 7f0af93b0740 1 -- 172.21.15.118:0/2079297210 wait complete. 2024-03-24T00:17:28.173 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:28.163+0000 7f0af93b0740 1 -- start start 2024-03-24T00:17:28.173 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:28.164+0000 7f0af93b0740 1 Processor -- start 2024-03-24T00:17:28.173 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:28.164+0000 7f0af93b0740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56029fecc810 0x56029feda150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:17:28.173 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:28.164+0000 7f0af93b0740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56029fecd9e0 0x56029feda690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:17:28.174 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:28.164+0000 7f0af93b0740 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x56029fece3a0 0x56029fedd6b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:17:28.174 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:28.164+0000 7f0af93b0740 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0af01abf80 con 0x56029fece3a0 2024-03-24T00:17:28.174 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:28.164+0000 7f0af93b0740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0af01ab920 con 0x56029fecd9e0 2024-03-24T00:17:28.174 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:28.164+0000 7f0af93b0740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0af01a8e20 con 0x56029fecc810 2024-03-24T00:17:28.174 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:28.164+0000 7f0af93b0740 10 client.0 _ll_register_callbacks cb 0x7f0adc00cf20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T00:17:28.174 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:28.164+0000 7f0aeea00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56029fecd9e0 0x56029feda690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:17:28.174 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:28.164+0000 7f0aef400640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56029fecc810 0x56029feda150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:17:28.174 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:28.164+0000 7f0aeea00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56029fecd9e0 0x56029feda690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.118:45498/0 (socket says 172.21.15.118:45498) 2024-03-24T00:17:28.175 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:28.164+0000 7f0aef400640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56029fecc810 0x56029feda150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.118:37870/0 (socket says 172.21.15.118:37870) 2024-03-24T00:17:28.175 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:28.164+0000 7f0aeea00640 1 -- 172.21.15.118:0/2173353117 learned_addr learned my addr 172.21.15.118:0/2173353117 (peer_addr_for_me v2:172.21.15.118:0/0) 2024-03-24T00:17:28.175 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:28.164+0000 7f0aefe00640 1 --2- 172.21.15.118:0/2173353117 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x56029fece3a0 0x56029fedd6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:17:28.175 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:28.165+0000 7f0aef400640 1 -- 172.21.15.118:0/2173353117 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56029fecd9e0 msgr2=0x56029feda690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:17:28.175 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:28.165+0000 7f0aef400640 1 --2- 172.21.15.118:0/2173353117 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56029fecd9e0 0x56029feda690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:17:28.175 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:28.165+0000 7f0aef400640 1 -- 172.21.15.118:0/2173353117 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x56029fece3a0 msgr2=0x56029fedd6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:17:28.175 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:28.165+0000 7f0aef400640 1 --2- 172.21.15.118:0/2173353117 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x56029fece3a0 0x56029fedd6b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:17:28.175 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:28.165+0000 7f0aef400640 1 -- 172.21.15.118:0/2173353117 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x56029feddca0 con 0x56029fecc810 2024-03-24T00:17:28.175 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:28.165+0000 7f0aeea00640 1 --2- 172.21.15.118:0/2173353117 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56029fecd9e0 0x56029feda690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T00:17:28.176 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:28.165+0000 7f0aefe00640 1 --2- 172.21.15.118:0/2173353117 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x56029fece3a0 0x56029fedd6b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T00:17:28.176 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:28.165+0000 7f0aef400640 1 --2- 172.21.15.118:0/2173353117 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56029fecc810 0x56029feda150 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7f0ae002c170 tx=0x7f0ae002cdd0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=69fcb49e737b0647 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:17:28.176 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:28.165+0000 7f0ad9600640 10 client.0 ms_handle_connect on v2:172.21.15.193:3300/0 2024-03-24T00:17:28.176 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:28.165+0000 7f0ad9600640 1 -- 172.21.15.118:0/2173353117 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0ae0044040 con 0x56029fecc810 2024-03-24T00:17:28.176 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:28.165+0000 7f0ad9600640 1 -- 172.21.15.118:0/2173353117 <== mon.1 v2:172.21.15.193:3300/0 2 ==== osd_map(90..90 src has 1..90) v4 ==== 8277+0+0 (secure 0 0 0) 0x7f0ae0003800 con 0x56029fecc810 2024-03-24T00:17:28.176 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:28.165+0000 7f0af93b0740 1 -- 172.21.15.118:0/2173353117 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f0ae002cc20 con 0x56029fecc810 2024-03-24T00:17:28.176 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:28.166+0000 7f0ad9600640 1 -- 172.21.15.118:0/2173353117 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) v2 ==== 24+0+0 (secure 0 0 0) 0x7f0ae0003a00 con 0x56029fecc810 2024-03-24T00:17:28.176 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:28.166+0000 7f0af93b0740 10 client.25891 fetch_fsmap learned FSMap version 28 2024-03-24T00:17:28.177 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:28.166+0000 7f0af93b0740 1 -- 172.21.15.118:0/2173353117 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) v3 -- 0x56029ff3e730 con 0x56029fecc810 2024-03-24T00:17:28.177 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:28.166+0000 7f0ad9600640 1 -- 172.21.15.118:0/2173353117 <== mon.1 v2:172.21.15.193:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f0ae002cc20 con 0x56029fecc810 2024-03-24T00:17:28.177 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:28.166+0000 7f0ad9600640 1 -- 172.21.15.118:0/2173353117 <== mon.1 v2:172.21.15.193:3300/0 5 ==== fsmap(e 28) v1 ==== 7934+0+0 (secure 0 0 0) 0x7f0ae005e080 con 0x56029fecc810 2024-03-24T00:17:28.177 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:28.166+0000 7f0af93b0740 10 client.25891 fetch_fsmap finished waiting for FSMap version 28 2024-03-24T00:17:28.177 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:28.166+0000 7f0af93b0740 10 client.25891 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.g 2024-03-24T00:17:28.177 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:28.166+0000 7f0af93b0740 20 client.25891 populate_metadata read hostname 'smithi118' 2024-03-24T00:17:28.177 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:28.166+0000 7f0af93b0740 1 --2- 172.21.15.118:0/2173353117 >> [v2:172.21.15.118:6838/3285788503,v1:172.21.15.118:6839/3285788503] conn(0x56029fee2430 0x56029ff40c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:17:28.177 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:28.166+0000 7f0af93b0740 4 client.25891 mds_command: new command op to 24550 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-24T00:17:28.177 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:28.166+0000 7f0af93b0740 1 -- 172.21.15.118:0/2173353117 --> [v2:172.21.15.118:6838/3285788503,v1:172.21.15.118:6839/3285788503] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x56029fe38b50 con 0x56029fee2430 2024-03-24T00:17:28.178 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:28.166+0000 7f0aeea00640 1 --2- 172.21.15.118:0/2173353117 >> [v2:172.21.15.118:6838/3285788503,v1:172.21.15.118:6839/3285788503] conn(0x56029fee2430 0x56029ff40c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:17:28.178 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:28.167+0000 7f0aeea00640 1 --2- 172.21.15.118:0/2173353117 >> [v2:172.21.15.118:6838/3285788503,v1:172.21.15.118:6839/3285788503] conn(0x56029fee2430 0x56029ff40c90 crc :-1 s=READY pgs=27 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=675f893d44d994e9 server_cookie=8dd248ac19fb62bc in_seq=0 out_seq=0 2024-03-24T00:17:28.178 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:28.167+0000 7f0ad9600640 10 client.25891 ms_handle_connect on v2:172.21.15.118:6838/3285788503 2024-03-24T00:17:28.178 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:28.167+0000 7f0ad9600640 1 -- 172.21.15.118:0/2173353117 <== mds.1 v2:172.21.15.118:6838/3285788503 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x56029fe38b50 con 0x56029fee2430 2024-03-24T00:17:28.178 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:28.167+0000 7f0ad9600640 10 client.25891 handle_command_reply: tid=0 2024-03-24T00:17:28.178 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:28.167+0000 7f0af93b0740 1 client.25891 shutdown 2024-03-24T00:17:28.178 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:28.168+0000 7f0af93b0740 1 -- 172.21.15.118:0/2173353117 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56029fecc810 msgr2=0x56029feda150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:17:28.178 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:28.168+0000 7f0af93b0740 1 --2- 172.21.15.118:0/2173353117 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56029fecc810 0x56029feda150 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7f0ae002c170 tx=0x7f0ae002cdd0 comp rx=0 tx=0).stop 2024-03-24T00:17:28.179 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:28.168+0000 7f0af93b0740 1 -- 172.21.15.118:0/2173353117 shutdown_connections 2024-03-24T00:17:28.179 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:28.168+0000 7f0af93b0740 1 --2- 172.21.15.118:0/2173353117 >> [v2:172.21.15.118:6838/3285788503,v1:172.21.15.118:6839/3285788503] conn(0x56029fee2430 0x56029ff40c90 crc :-1 s=READY pgs=27 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:17:28.179 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:28.168+0000 7f0af93b0740 1 --2- 172.21.15.118:0/2173353117 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x56029fece3a0 0x56029fedd6b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:17:28.179 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:28.168+0000 7f0af93b0740 1 --2- 172.21.15.118:0/2173353117 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56029fecd9e0 0x56029feda690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:17:28.179 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:28.168+0000 7f0af93b0740 1 --2- 172.21.15.118:0/2173353117 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56029fecc810 0x56029feda150 unknown :-1 s=CLOSED pgs=278 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:17:28.179 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:28.168+0000 7f0af93b0740 1 -- 172.21.15.118:0/2173353117 >> 172.21.15.118:0/2173353117 conn(0x56029fec7c00 msgr2=0x56029fecb6f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:17:28.179 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:28.168+0000 7f0ad9600640 0 client.25891 ms_handle_reset on v2:172.21.15.118:6838/3285788503 2024-03-24T00:17:28.179 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:28.168+0000 7f0af93b0740 1 -- 172.21.15.118:0/2173353117 shutdown_connections 2024-03-24T00:17:28.180 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:28.168+0000 7f0af93b0740 1 -- 172.21.15.118:0/2173353117 wait complete. 2024-03-24T00:17:28.180 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:28.168+0000 7f0af93b0740 20 client.25891 trim_cache size 0 max 16384 2024-03-24T00:17:28.180 INFO:teuthology.orchestra.run.smithi118.stdout:[] 2024-03-24T00:17:28.181 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:28.169+0000 7f0af6600640 1 -- 172.21.15.118:0/2023590726 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7f0ad40a2880 msgr2=0x7f0ad40a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:17:28.181 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:28.169+0000 7f0af6600640 1 --2- 172.21.15.118:0/2023590726 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7f0ad40a2880 0x7f0ad40a4d40 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 crypto rx=0x56029fecd8e0 tx=0x7f0ae8009500 comp rx=0 tx=0).stop 2024-03-24T00:17:28.181 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:28.169+0000 7f0af6600640 1 -- 172.21.15.118:0/2023590726 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f0af0104ad0 msgr2=0x7f0af019a130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:17:28.181 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:28.169+0000 7f0af6600640 1 --2- 172.21.15.118:0/2023590726 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f0af0104ad0 0x7f0af019a130 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7f0adc00e9f0 tx=0x7f0adc00eeb0 comp rx=0 tx=0).stop 2024-03-24T00:17:28.182 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:28.169+0000 7f0af6600640 1 -- 172.21.15.118:0/2023590726 shutdown_connections 2024-03-24T00:17:28.182 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:28.169+0000 7f0af6600640 1 --2- 172.21.15.118:0/2023590726 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7f0ad40a2880 0x7f0ad40a4d40 unknown :-1 s=CLOSED pgs=301 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:17:28.182 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:28.169+0000 7f0af6600640 1 --2- 172.21.15.118:0/2023590726 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0af0105490 0x7f0af01a11a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:17:28.182 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:28.169+0000 7f0af6600640 1 --2- 172.21.15.118:0/2023590726 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f0af0104ad0 0x7f0af019a130 unknown :-1 s=CLOSED pgs=275 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:17:28.182 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:28.169+0000 7f0af6600640 1 --2- 172.21.15.118:0/2023590726 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f0af01038e0 0x7f0af0199bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:17:28.182 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:28.169+0000 7f0af6600640 1 -- 172.21.15.118:0/2023590726 >> 172.21.15.118:0/2023590726 conn(0x7f0af00fee50 msgr2=0x7f0af0107d70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:17:28.182 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:28.169+0000 7f0af6600640 1 -- 172.21.15.118:0/2023590726 shutdown_connections 2024-03-24T00:17:28.182 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:28.169+0000 7f0af6600640 1 -- 172.21.15.118:0/2023590726 wait complete. 2024-03-24T00:17:28.187 INFO:teuthology.orchestra.run:Running command with timeout 120 2024-03-24T00:17:28.188 DEBUG:teuthology.orchestra.run.smithi118:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2024-03-24T00:17:28.197 INFO:tasks.workunit.client.0.smithi118.stdout:checking for an ANSI C-conforming const... yes 2024-03-24T00:17:28.203 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:17:27 smithi189 ceph-mon[36187]: from='client.? 172.21.15.118:0/2171715402' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-24T00:17:28.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:17:27 smithi193 ceph-mon[36878]: from='client.? 172.21.15.118:0/2171715402' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-24T00:17:28.278 INFO:tasks.workunit.client.0.smithi118.stdout:checking for inline... inline 2024-03-24T00:17:28.382 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:28.380+0000 7f80f6600640 1 -- 172.21.15.118:0/2174036555 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f80f01016b0 msgr2=0x7f80f0101ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:17:28.382 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:28.380+0000 7f80f6600640 1 --2- 172.21.15.118:0/2174036555 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f80f01016b0 0x7f80f0101ad0 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 crypto rx=0x7f80e0009900 tx=0x7f80e002bc30 comp rx=0 tx=0).stop 2024-03-24T00:17:28.382 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:28.381+0000 7f80f6600640 1 -- 172.21.15.118:0/2174036555 shutdown_connections 2024-03-24T00:17:28.382 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:28.381+0000 7f80f6600640 1 --2- 172.21.15.118:0/2174036555 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f80f0103260 0x7f80f0109af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:17:28.382 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:28.381+0000 7f80f6600640 1 --2- 172.21.15.118:0/2174036555 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f80f01028a0 0x7f80f0102d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:17:28.382 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:28.381+0000 7f80f6600640 1 --2- 172.21.15.118:0/2174036555 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f80f01016b0 0x7f80f0101ad0 unknown :-1 s=CLOSED pgs=294 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:17:28.382 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:28.381+0000 7f80f6600640 1 -- 172.21.15.118:0/2174036555 >> 172.21.15.118:0/2174036555 conn(0x7f80f00fcc40 msgr2=0x7f80f00ff060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:17:28.382 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:28.381+0000 7f80f6600640 1 -- 172.21.15.118:0/2174036555 shutdown_connections 2024-03-24T00:17:28.382 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:28.381+0000 7f80f6600640 1 -- 172.21.15.118:0/2174036555 wait complete. 2024-03-24T00:17:28.384 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:28.382+0000 7f80f6600640 1 Processor -- start 2024-03-24T00:17:28.384 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:28.382+0000 7f80f6600640 1 -- start start 2024-03-24T00:17:28.384 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:28.382+0000 7f80f6600640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f80f01016b0 0x7f80f0199c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:17:28.384 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:28.382+0000 7f80f6600640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f80f01028a0 0x7f80f019a190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:17:28.384 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:28.383+0000 7f80f6600640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f80f0103260 0x7f80f01a11c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:17:28.384 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:28.383+0000 7f80f6600640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f80f010d350 con 0x7f80f0103260 2024-03-24T00:17:28.384 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:28.383+0000 7f80f6600640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f80f010d1d0 con 0x7f80f01016b0 2024-03-24T00:17:28.385 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:28.383+0000 7f80f6600640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f80f010d4d0 con 0x7f80f01028a0 2024-03-24T00:17:28.385 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:28.383+0000 7f80ef400640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f80f01016b0 0x7f80f0199c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:17:28.385 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:28.383+0000 7f80ef400640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f80f01016b0 0x7f80f0199c50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.118:45520/0 (socket says 172.21.15.118:45520) 2024-03-24T00:17:28.385 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:28.383+0000 7f80ef400640 1 -- 172.21.15.118:0/1526965738 learned_addr learned my addr 172.21.15.118:0/1526965738 (peer_addr_for_me v2:172.21.15.118:0/0) 2024-03-24T00:17:28.385 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:28.383+0000 7f80eea00640 1 --2- 172.21.15.118:0/1526965738 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f80f01028a0 0x7f80f019a190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:17:28.385 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:28.383+0000 7f80efe00640 1 --2- 172.21.15.118:0/1526965738 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f80f0103260 0x7f80f01a11c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:17:28.385 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:28.383+0000 7f80ef400640 1 -- 172.21.15.118:0/1526965738 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f80f01028a0 msgr2=0x7f80f019a190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:17:28.385 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:28.383+0000 7f80ef400640 1 --2- 172.21.15.118:0/1526965738 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f80f01028a0 0x7f80f019a190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:17:28.385 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:28.383+0000 7f80ef400640 1 -- 172.21.15.118:0/1526965738 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f80f0103260 msgr2=0x7f80f01a11c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:17:28.385 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:28.383+0000 7f80ef400640 1 --2- 172.21.15.118:0/1526965738 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f80f0103260 0x7f80f01a11c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:17:28.385 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:28.383+0000 7f80ef400640 1 -- 172.21.15.118:0/1526965738 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f80f01a1860 con 0x7f80f01016b0 2024-03-24T00:17:28.385 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:28.383+0000 7f80eea00640 1 --2- 172.21.15.118:0/1526965738 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f80f01028a0 0x7f80f019a190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T00:17:28.385 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:28.384+0000 7f80ef400640 1 --2- 172.21.15.118:0/1526965738 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f80f01016b0 0x7f80f0199c50 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 crypto rx=0x7f80e0000c00 tx=0x7f80e002c650 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:17:28.386 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:28.384+0000 7f80e7e00640 1 -- 172.21.15.118:0/1526965738 <== mon.2 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f80e0043020 con 0x7f80f01016b0 2024-03-24T00:17:28.386 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:28.384+0000 7f80e7e00640 1 -- 172.21.15.118:0/1526965738 <== mon.2 v2:172.21.15.189:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f80e00054e0 con 0x7f80f01016b0 2024-03-24T00:17:28.386 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:28.384+0000 7f80e7e00640 1 -- 172.21.15.118:0/1526965738 <== mon.2 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f80e0003c40 con 0x7f80f01016b0 2024-03-24T00:17:28.386 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:28.384+0000 7f80f6600640 1 -- 172.21.15.118:0/1526965738 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f80f01a1af0 con 0x7f80f01016b0 2024-03-24T00:17:28.390 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:28.384+0000 7f80f6600640 1 -- 172.21.15.118:0/1526965738 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f80f01a1ff0 con 0x7f80f01016b0 2024-03-24T00:17:28.392 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:28.388+0000 7f80e7e00640 1 -- 172.21.15.118:0/1526965738 <== mon.2 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 20) v1 ==== 137291+0+0 (secure 0 0 0) 0x7f80e0046020 con 0x7f80f01016b0 2024-03-24T00:17:28.392 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:28.388+0000 7f80f924a740 1 Processor -- start 2024-03-24T00:17:28.393 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:28.388+0000 7f80f924a740 1 -- start start 2024-03-24T00:17:28.393 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:28.388+0000 7f80f924a740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5650b8eb7810 0x5650b8eb7c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:17:28.393 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:28.388+0000 7f80f924a740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5650b8eb89e0 0x5650b8eb8e60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:17:28.393 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:28.388+0000 7f80f924a740 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5650b8eb93a0 0x5650b8ebfc30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:17:28.393 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:28.388+0000 7f80f924a740 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x5650b8e1c2c0 con 0x5650b8eb93a0 2024-03-24T00:17:28.393 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:28.388+0000 7f80f924a740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x5650b8dc4c90 con 0x5650b8eb89e0 2024-03-24T00:17:28.393 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:28.388+0000 7f80f924a740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x5650b8e18360 con 0x5650b8eb7810 2024-03-24T00:17:28.393 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:28.389+0000 7f80efe00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5650b8eb7810 0x5650b8eb7c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:17:28.393 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:28.389+0000 7f80efe00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5650b8eb7810 0x5650b8eb7c30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.118:37896/0 (socket says 172.21.15.118:37896) 2024-03-24T00:17:28.393 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:28.389+0000 7f80efe00640 1 -- 172.21.15.118:0/1668730581 learned_addr learned my addr 172.21.15.118:0/1668730581 (peer_addr_for_me v2:172.21.15.118:0/0) 2024-03-24T00:17:28.393 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:28.389+0000 7f80ef400640 1 --2- 172.21.15.118:0/1668730581 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5650b8eb89e0 0x5650b8eb8e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:17:28.393 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:28.389+0000 7f80eea00640 1 --2- 172.21.15.118:0/1668730581 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5650b8eb93a0 0x5650b8ebfc30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:17:28.393 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:28.389+0000 7f80efe00640 1 -- 172.21.15.118:0/1668730581 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5650b8eb89e0 msgr2=0x5650b8eb8e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:17:28.394 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:28.389+0000 7f80efe00640 1 --2- 172.21.15.118:0/1668730581 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5650b8eb89e0 0x5650b8eb8e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:17:28.394 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:28.389+0000 7f80efe00640 1 -- 172.21.15.118:0/1668730581 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5650b8eb93a0 msgr2=0x5650b8ebfc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:17:28.394 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:28.389+0000 7f80efe00640 1 --2- 172.21.15.118:0/1668730581 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5650b8eb93a0 0x5650b8ebfc30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:17:28.394 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:28.389+0000 7f80efe00640 1 -- 172.21.15.118:0/1668730581 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5650b8de9080 con 0x5650b8eb7810 2024-03-24T00:17:28.394 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:28.389+0000 7f80ef400640 1 --2- 172.21.15.118:0/1668730581 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5650b8eb89e0 0x5650b8eb8e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T00:17:28.394 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:28.389+0000 7f80eea00640 1 --2- 172.21.15.118:0/1668730581 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5650b8eb93a0 0x5650b8ebfc30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-24T00:17:28.394 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:28.389+0000 7f80efe00640 1 --2- 172.21.15.118:0/1668730581 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5650b8eb7810 0x5650b8eb7c30 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7f80e8009fd0 tx=0x7f80e80118d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=b7f332161ef0863c server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:17:28.394 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:28.389+0000 7f80e7e00640 1 --2- 172.21.15.118:0/1526965738 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7f80d40a2880 0x7f80d40a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:17:28.394 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:28.390+0000 7f80e4c00640 1 -- 172.21.15.118:0/1668730581 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f80e8004d20 con 0x5650b8eb7810 2024-03-24T00:17:28.394 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:28.390+0000 7f80e4c00640 1 -- 172.21.15.118:0/1668730581 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f80e8004050 con 0x5650b8eb7810 2024-03-24T00:17:28.394 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:28.390+0000 7f80e4c00640 1 -- 172.21.15.118:0/1668730581 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f80e8005260 con 0x5650b8eb7810 2024-03-24T00:17:28.394 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:28.390+0000 7f80e7e00640 1 -- 172.21.15.118:0/1526965738 <== mon.2 v2:172.21.15.189:3300/0 5 ==== osd_map(90..90 src has 1..90) v4 ==== 8277+0+0 (secure 0 0 0) 0x7f80e00f0b10 con 0x7f80f01016b0 2024-03-24T00:17:28.394 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:28.390+0000 7f80ef400640 1 --2- 172.21.15.118:0/1526965738 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7f80d40a2880 0x7f80d40a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:17:28.394 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:28.391+0000 7f80f924a740 1 -- 172.21.15.118:0/1668730581 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5650b8eb7810 msgr2=0x5650b8eb7c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:17:28.395 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:28.391+0000 7f80ef400640 1 --2- 172.21.15.118:0/1526965738 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7f80d40a2880 0x7f80d40a4d40 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 crypto rx=0x7f80e00ee580 tx=0x7f80e00dd040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:17:28.395 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:28.391+0000 7f80f924a740 1 --2- 172.21.15.118:0/1668730581 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5650b8eb7810 0x5650b8eb7c30 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7f80e8009fd0 tx=0x7f80e80118d0 comp rx=0 tx=0).stop 2024-03-24T00:17:28.395 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:28.391+0000 7f80f924a740 1 -- 172.21.15.118:0/1668730581 shutdown_connections 2024-03-24T00:17:28.395 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:28.391+0000 7f80f924a740 1 --2- 172.21.15.118:0/1668730581 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5650b8eb93a0 0x5650b8ebfc30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:17:28.395 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:28.391+0000 7f80f924a740 1 --2- 172.21.15.118:0/1668730581 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5650b8eb89e0 0x5650b8eb8e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:17:28.395 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:28.391+0000 7f80f924a740 1 --2- 172.21.15.118:0/1668730581 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5650b8eb7810 0x5650b8eb7c30 unknown :-1 s=CLOSED pgs=279 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:17:28.395 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:28.391+0000 7f80f924a740 1 -- 172.21.15.118:0/1668730581 >> 172.21.15.118:0/1668730581 conn(0x5650b8eb2c00 msgr2=0x5650b8eb3000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:17:28.395 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:28.391+0000 7f80f924a740 1 -- 172.21.15.118:0/1668730581 shutdown_connections 2024-03-24T00:17:28.395 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:28.391+0000 7f80f924a740 1 -- 172.21.15.118:0/1668730581 wait complete. 2024-03-24T00:17:28.395 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:28.391+0000 7f80f924a740 1 -- start start 2024-03-24T00:17:28.396 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:28.392+0000 7f80f924a740 1 Processor -- start 2024-03-24T00:17:28.400 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:28.392+0000 7f80f924a740 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5650b8eb7810 0x5650b8ec1ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:17:28.400 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:28.392+0000 7f80f924a740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5650b8eb89e0 0x5650b8ec1ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:17:28.400 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:28.393+0000 7f80f924a740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5650b8eb93a0 0x5650b8ec9030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:17:28.400 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:28.393+0000 7f80f924a740 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x5650b8ec03b0 con 0x5650b8eb7810 2024-03-24T00:17:28.400 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:28.393+0000 7f80f924a740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x5650b8dd80d0 con 0x5650b8eb93a0 2024-03-24T00:17:28.400 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:28.393+0000 7f80f924a740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x5650b8ec0530 con 0x5650b8eb89e0 2024-03-24T00:17:28.401 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:28.393+0000 7f80f924a740 10 client.0 _ll_register_callbacks cb 0x5650b8ddce80 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T00:17:28.401 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:28.393+0000 7f80efe00640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5650b8eb7810 0x5650b8ec1ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:17:28.401 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:28.393+0000 7f80efe00640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5650b8eb7810 0x5650b8ec1ab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.118:3300/0 says I am v2:172.21.15.118:38534/0 (socket says 172.21.15.118:38534) 2024-03-24T00:17:28.401 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:28.393+0000 7f80efe00640 1 -- 172.21.15.118:0/2198964719 learned_addr learned my addr 172.21.15.118:0/2198964719 (peer_addr_for_me v2:172.21.15.118:0/0) 2024-03-24T00:17:28.401 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:28.393+0000 7f80eea00640 1 --2- 172.21.15.118:0/2198964719 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5650b8eb89e0 0x5650b8ec1ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:17:28.401 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:28.393+0000 7f80efe00640 1 --2- 172.21.15.118:0/2198964719 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5650b8eb93a0 0x5650b8ec9030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:17:28.401 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:28.393+0000 7f80eea00640 1 -- 172.21.15.118:0/2198964719 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5650b8eb93a0 msgr2=0x5650b8ec9030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:17:28.401 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:28.393+0000 7f80eea00640 1 --2- 172.21.15.118:0/2198964719 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5650b8eb93a0 0x5650b8ec9030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:17:28.401 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:28.393+0000 7f80eea00640 1 -- 172.21.15.118:0/2198964719 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5650b8eb7810 msgr2=0x5650b8ec1ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:17:28.401 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:28.393+0000 7f80eea00640 1 --2- 172.21.15.118:0/2198964719 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5650b8eb7810 0x5650b8ec1ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:17:28.401 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:28.393+0000 7f80eea00640 1 -- 172.21.15.118:0/2198964719 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5650b8b303f0 con 0x5650b8eb89e0 2024-03-24T00:17:28.401 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:28.393+0000 7f80efe00640 1 --2- 172.21.15.118:0/2198964719 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5650b8eb7810 0x5650b8ec1ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T00:17:28.401 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:28.394+0000 7f80eea00640 1 --2- 172.21.15.118:0/2198964719 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5650b8eb89e0 0x5650b8ec1ff0 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7f80dc013980 tx=0x7f80dc013e40 comp rx=0 tx=0).ready entity=mon.1 client_cookie=8ce419bef31e8a46 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:17:28.401 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:28.394+0000 7f80d9600640 10 client.0 ms_handle_connect on v2:172.21.15.193:3300/0 2024-03-24T00:17:28.402 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:28.394+0000 7f80d9600640 1 -- 172.21.15.118:0/2198964719 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f80dc011cf0 con 0x5650b8eb89e0 2024-03-24T00:17:28.402 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:28.394+0000 7f80f924a740 1 -- 172.21.15.118:0/2198964719 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f80e8004050 con 0x5650b8eb89e0 2024-03-24T00:17:28.402 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:28.394+0000 7f80d9600640 1 -- 172.21.15.118:0/2198964719 <== mon.1 v2:172.21.15.193:3300/0 2 ==== osd_map(90..90 src has 1..90) v4 ==== 8277+0+0 (secure 0 0 0) 0x7f80dc005020 con 0x5650b8eb89e0 2024-03-24T00:17:28.402 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:28.394+0000 7f80d9600640 1 -- 172.21.15.118:0/2198964719 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) v2 ==== 24+0+0 (secure 0 0 0) 0x7f80dc011e90 con 0x5650b8eb89e0 2024-03-24T00:17:28.402 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:28.394+0000 7f80f924a740 10 client.25903 fetch_fsmap learned FSMap version 28 2024-03-24T00:17:28.402 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:28.394+0000 7f80f924a740 1 -- 172.21.15.118:0/2198964719 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) v3 -- 0x5650b8bfa7d0 con 0x5650b8eb89e0 2024-03-24T00:17:28.402 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:28.395+0000 7f80d9600640 1 -- 172.21.15.118:0/2198964719 <== mon.1 v2:172.21.15.193:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f80e8004050 con 0x5650b8eb89e0 2024-03-24T00:17:28.402 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:28.395+0000 7f80d9600640 1 -- 172.21.15.118:0/2198964719 <== mon.1 v2:172.21.15.193:3300/0 5 ==== fsmap(e 28) v1 ==== 7934+0+0 (secure 0 0 0) 0x7f80dc01b410 con 0x5650b8eb89e0 2024-03-24T00:17:28.402 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:28.395+0000 7f80f924a740 10 client.25903 fetch_fsmap finished waiting for FSMap version 28 2024-03-24T00:17:28.402 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:28.395+0000 7f80f924a740 10 client.25903 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-24T00:17:28.402 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:28.395+0000 7f80f924a740 20 client.25903 populate_metadata read hostname 'smithi118' 2024-03-24T00:17:28.402 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:28.395+0000 7f80f924a740 1 --2- 172.21.15.118:0/2198964719 >> [v2:172.21.15.189:6838/4074484168,v1:172.21.15.189:6839/4074484168] conn(0x5650b8ecd4f0 0x5650b8ecf930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:17:28.402 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:28.395+0000 7f80f924a740 4 client.25903 mds_command: new command op to 24569 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-24T00:17:28.403 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:28.395+0000 7f80f924a740 1 -- 172.21.15.118:0/2198964719 --> [v2:172.21.15.189:6838/4074484168,v1:172.21.15.189:6839/4074484168] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x5650b8e29800 con 0x5650b8ecd4f0 2024-03-24T00:17:28.403 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:28.395+0000 7f80efe00640 1 --2- 172.21.15.118:0/2198964719 >> [v2:172.21.15.189:6838/4074484168,v1:172.21.15.189:6839/4074484168] conn(0x5650b8ecd4f0 0x5650b8ecf930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:17:28.403 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:28.396+0000 7f80efe00640 1 --2- 172.21.15.118:0/2198964719 >> [v2:172.21.15.189:6838/4074484168,v1:172.21.15.189:6839/4074484168] conn(0x5650b8ecd4f0 0x5650b8ecf930 crc :-1 s=READY pgs=69 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=927bc410ad1d8011 server_cookie=7af2e226d3f11aa2 in_seq=0 out_seq=0 2024-03-24T00:17:28.403 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:28.396+0000 7f80d9600640 10 client.25903 ms_handle_connect on v2:172.21.15.189:6838/4074484168 2024-03-24T00:17:28.403 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:28.396+0000 7f80d9600640 1 -- 172.21.15.118:0/2198964719 <== mds.0 v2:172.21.15.189:6838/4074484168 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x5650b8e29800 con 0x5650b8ecd4f0 2024-03-24T00:17:28.403 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:28.396+0000 7f80d9600640 10 client.25903 handle_command_reply: tid=0 2024-03-24T00:17:28.403 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:28.396+0000 7f80f924a740 1 client.25903 shutdown 2024-03-24T00:17:28.403 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:28.397+0000 7f80f924a740 1 -- 172.21.15.118:0/2198964719 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5650b8eb89e0 msgr2=0x5650b8ec1ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:17:28.403 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:28.397+0000 7f80f924a740 1 --2- 172.21.15.118:0/2198964719 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5650b8eb89e0 0x5650b8ec1ff0 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7f80dc013980 tx=0x7f80dc013e40 comp rx=0 tx=0).stop 2024-03-24T00:17:28.403 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:28.397+0000 7f80f924a740 1 -- 172.21.15.118:0/2198964719 shutdown_connections 2024-03-24T00:17:28.403 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:28.397+0000 7f80f924a740 1 --2- 172.21.15.118:0/2198964719 >> [v2:172.21.15.189:6838/4074484168,v1:172.21.15.189:6839/4074484168] conn(0x5650b8ecd4f0 0x5650b8ecf930 crc :-1 s=READY pgs=69 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:17:28.403 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:28.397+0000 7f80f924a740 1 --2- 172.21.15.118:0/2198964719 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5650b8eb93a0 0x5650b8ec9030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:17:28.403 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:28.397+0000 7f80f924a740 1 --2- 172.21.15.118:0/2198964719 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5650b8eb89e0 0x5650b8ec1ff0 unknown :-1 s=CLOSED pgs=280 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:17:28.403 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:28.397+0000 7f80d9600640 0 client.25903 ms_handle_reset on v2:172.21.15.189:6838/4074484168 2024-03-24T00:17:28.403 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:28.397+0000 7f80f924a740 1 --2- 172.21.15.118:0/2198964719 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5650b8eb7810 0x5650b8ec1ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:17:28.404 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:28.397+0000 7f80f924a740 1 -- 172.21.15.118:0/2198964719 >> 172.21.15.118:0/2198964719 conn(0x5650b8eb2c00 msgr2=0x5650b8eb1eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:17:28.404 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:28.397+0000 7f80f924a740 1 -- 172.21.15.118:0/2198964719 shutdown_connections 2024-03-24T00:17:28.404 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:28.397+0000 7f80f924a740 1 -- 172.21.15.118:0/2198964719 wait complete. 2024-03-24T00:17:28.404 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:28.397+0000 7f80f924a740 20 client.25903 trim_cache size 0 max 16384 2024-03-24T00:17:28.414 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:28.412+0000 7f80f924a740 1 Processor -- start 2024-03-24T00:17:28.414 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:28.412+0000 7f80f924a740 1 -- start start 2024-03-24T00:17:28.414 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:28.412+0000 7f80f924a740 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5650b8eb7810 0x5650b8eb1620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:17:28.414 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:28.412+0000 7f80f924a740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5650b8eb89e0 0x5650b8ec8ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:17:28.414 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:28.412+0000 7f80f924a740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5650b8eb93a0 0x5650b8eb1b60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:17:28.414 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:28.412+0000 7f80f924a740 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f80f01aac90 con 0x5650b8eb7810 2024-03-24T00:17:28.414 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:28.412+0000 7f80f924a740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f80f01ac610 con 0x5650b8eb93a0 2024-03-24T00:17:28.414 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:28.412+0000 7f80f924a740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f80f01ab2f0 con 0x5650b8eb89e0 2024-03-24T00:17:28.414 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:28.412+0000 7f80eea00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5650b8eb89e0 0x5650b8ec8ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:17:28.414 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:28.412+0000 7f80eea00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5650b8eb89e0 0x5650b8ec8ff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.118:37918/0 (socket says 172.21.15.118:37918) 2024-03-24T00:17:28.415 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:28.412+0000 7f80eea00640 1 -- 172.21.15.118:0/1468550598 learned_addr learned my addr 172.21.15.118:0/1468550598 (peer_addr_for_me v2:172.21.15.118:0/0) 2024-03-24T00:17:28.415 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:28.412+0000 7f80efe00640 1 --2- 172.21.15.118:0/1468550598 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5650b8eb7810 0x5650b8eb1620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:17:28.415 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:28.413+0000 7f80efe00640 1 --2- 172.21.15.118:0/1468550598 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5650b8eb93a0 0x5650b8eb1b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:17:28.415 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:28.413+0000 7f80eea00640 1 -- 172.21.15.118:0/1468550598 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5650b8eb93a0 msgr2=0x5650b8eb1b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:17:28.415 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:28.413+0000 7f80eea00640 1 --2- 172.21.15.118:0/1468550598 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5650b8eb93a0 0x5650b8eb1b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:17:28.415 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:28.413+0000 7f80eea00640 1 -- 172.21.15.118:0/1468550598 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5650b8eb7810 msgr2=0x5650b8eb1620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:17:28.415 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:28.413+0000 7f80eea00640 1 --2- 172.21.15.118:0/1468550598 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5650b8eb7810 0x5650b8eb1620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:17:28.415 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:28.413+0000 7f80eea00640 1 -- 172.21.15.118:0/1468550598 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5650b8eb3970 con 0x5650b8eb89e0 2024-03-24T00:17:28.415 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:28.413+0000 7f80efe00640 1 --2- 172.21.15.118:0/1468550598 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5650b8eb7810 0x5650b8eb1620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T00:17:28.415 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:28.413+0000 7f80eea00640 1 --2- 172.21.15.118:0/1468550598 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5650b8eb89e0 0x5650b8ec8ff0 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7f80dc01b8d0 tx=0x7f80dc0134f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=decd626a80be6750 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:17:28.415 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:28.413+0000 7f80dbe00640 1 -- 172.21.15.118:0/1468550598 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f80dc029030 con 0x5650b8eb89e0 2024-03-24T00:17:28.415 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:28.413+0000 7f80dbe00640 1 -- 172.21.15.118:0/1468550598 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f80dc0117b0 con 0x5650b8eb89e0 2024-03-24T00:17:28.415 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:28.413+0000 7f80dbe00640 1 -- 172.21.15.118:0/1468550598 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f80dc0035f0 con 0x5650b8eb89e0 2024-03-24T00:17:28.416 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:28.414+0000 7f80f924a740 1 -- 172.21.15.118:0/1468550598 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5650b8eb89e0 msgr2=0x5650b8ec8ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:17:28.416 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:28.414+0000 7f80f924a740 1 --2- 172.21.15.118:0/1468550598 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5650b8eb89e0 0x5650b8ec8ff0 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7f80dc01b8d0 tx=0x7f80dc0134f0 comp rx=0 tx=0).stop 2024-03-24T00:17:28.416 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:28.414+0000 7f80f924a740 1 -- 172.21.15.118:0/1468550598 shutdown_connections 2024-03-24T00:17:28.416 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:28.414+0000 7f80f924a740 1 --2- 172.21.15.118:0/1468550598 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5650b8eb93a0 0x5650b8eb1b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:17:28.417 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:28.414+0000 7f80f924a740 1 --2- 172.21.15.118:0/1468550598 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5650b8eb89e0 0x5650b8ec8ff0 unknown :-1 s=CLOSED pgs=281 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:17:28.417 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:28.414+0000 7f80f924a740 1 --2- 172.21.15.118:0/1468550598 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5650b8eb7810 0x5650b8eb1620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:17:28.417 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:28.414+0000 7f80f924a740 1 -- 172.21.15.118:0/1468550598 >> 172.21.15.118:0/1468550598 conn(0x5650b8eb2c00 msgr2=0x5650b8eb2fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:17:28.417 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:28.414+0000 7f80f924a740 1 -- 172.21.15.118:0/1468550598 shutdown_connections 2024-03-24T00:17:28.417 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:28.414+0000 7f80f924a740 1 -- 172.21.15.118:0/1468550598 wait complete. 2024-03-24T00:17:28.417 INFO:tasks.workunit.client.0.smithi118.stdout:checking for working alloca.h... yes 2024-03-24T00:17:28.419 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:28.415+0000 7f80f924a740 1 -- start start 2024-03-24T00:17:28.419 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:28.415+0000 7f80f924a740 1 Processor -- start 2024-03-24T00:17:28.419 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:28.416+0000 7f80f924a740 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5650b8eb7810 0x5650b8ec46e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:17:28.419 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:28.416+0000 7f80f924a740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5650b8eb89e0 0x5650b8ec4c20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:17:28.420 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:28.416+0000 7f80f924a740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5650b8eb93a0 0x5650b8ec7c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:17:28.420 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:28.416+0000 7f80f924a740 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f80f01abfb0 con 0x5650b8eb7810 2024-03-24T00:17:28.420 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:28.416+0000 7f80f924a740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f80f01ab950 con 0x5650b8eb89e0 2024-03-24T00:17:28.420 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:28.416+0000 7f80f924a740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f80f01a8e50 con 0x5650b8eb93a0 2024-03-24T00:17:28.420 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:28.416+0000 7f80f924a740 10 client.0 _ll_register_callbacks cb 0x7f80dc009570 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T00:17:28.420 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:28.416+0000 7f80efe00640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5650b8eb7810 0x5650b8ec46e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:17:28.420 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:28.416+0000 7f80eea00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5650b8eb89e0 0x5650b8ec4c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:17:28.420 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:28.416+0000 7f80eea00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5650b8eb89e0 0x5650b8ec4c20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.118:45554/0 (socket says 172.21.15.118:45554) 2024-03-24T00:17:28.420 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:28.416+0000 7f80eea00640 1 -- 172.21.15.118:0/4079887487 learned_addr learned my addr 172.21.15.118:0/4079887487 (peer_addr_for_me v2:172.21.15.118:0/0) 2024-03-24T00:17:28.420 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:28.416+0000 7f80efe00640 1 --2- 172.21.15.118:0/4079887487 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5650b8eb93a0 0x5650b8ec7c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:17:28.420 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:28.416+0000 7f80eea00640 1 -- 172.21.15.118:0/4079887487 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5650b8eb93a0 msgr2=0x5650b8ec7c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:17:28.420 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:28.416+0000 7f80eea00640 1 --2- 172.21.15.118:0/4079887487 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5650b8eb93a0 0x5650b8ec7c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:17:28.420 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:28.416+0000 7f80eea00640 1 -- 172.21.15.118:0/4079887487 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5650b8eb7810 msgr2=0x5650b8ec46e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:17:28.420 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:28.416+0000 7f80eea00640 1 --2- 172.21.15.118:0/4079887487 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5650b8eb7810 0x5650b8ec46e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:17:28.421 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:28.416+0000 7f80eea00640 1 -- 172.21.15.118:0/4079887487 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5650b8ec8230 con 0x5650b8eb89e0 2024-03-24T00:17:28.421 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:28.417+0000 7f80eea00640 1 --2- 172.21.15.118:0/4079887487 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5650b8eb89e0 0x5650b8ec4c20 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x7f80dc000c00 tx=0x7f80dc005620 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a7adb6125c649063 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:17:28.421 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:28.417+0000 7f80d9600640 10 client.0 ms_handle_connect on v2:172.21.15.189:3300/0 2024-03-24T00:17:28.421 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:28.417+0000 7f80d9600640 1 -- 172.21.15.118:0/4079887487 <== mon.2 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f80dc029070 con 0x5650b8eb89e0 2024-03-24T00:17:28.421 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:28.417+0000 7f80d9600640 1 -- 172.21.15.118:0/4079887487 <== mon.2 v2:172.21.15.189:3300/0 2 ==== osd_map(90..90 src has 1..90) v4 ==== 8277+0+0 (secure 0 0 0) 0x7f80dc003a00 con 0x5650b8eb89e0 2024-03-24T00:17:28.421 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:28.417+0000 7f80f924a740 1 -- 172.21.15.118:0/4079887487 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f80dc0117b0 con 0x5650b8eb89e0 2024-03-24T00:17:28.421 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:28.417+0000 7f80d9600640 1 -- 172.21.15.118:0/4079887487 <== mon.2 v2:172.21.15.189:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) v2 ==== 24+0+0 (secure 0 0 0) 0x7f80dc003c00 con 0x5650b8eb89e0 2024-03-24T00:17:28.421 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:28.417+0000 7f80f924a740 10 client.25922 fetch_fsmap learned FSMap version 28 2024-03-24T00:17:28.421 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:28.417+0000 7f80f924a740 1 -- 172.21.15.118:0/4079887487 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) v3 -- 0x5650b8f28dc0 con 0x5650b8eb89e0 2024-03-24T00:17:28.421 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:28.418+0000 7f80d9600640 1 -- 172.21.15.118:0/4079887487 <== mon.2 v2:172.21.15.189:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f80dc0117b0 con 0x5650b8eb89e0 2024-03-24T00:17:28.423 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:28.418+0000 7f80d9600640 1 -- 172.21.15.118:0/4079887487 <== mon.2 v2:172.21.15.189:3300/0 5 ==== fsmap(e 28) v1 ==== 7934+0+0 (secure 0 0 0) 0x7f80dc03fb90 con 0x5650b8eb89e0 2024-03-24T00:17:28.423 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:28.418+0000 7f80f924a740 10 client.25922 fetch_fsmap finished waiting for FSMap version 28 2024-03-24T00:17:28.423 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:28.418+0000 7f80f924a740 10 client.25922 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-24T00:17:28.423 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:28.418+0000 7f80f924a740 20 client.25922 populate_metadata read hostname 'smithi118' 2024-03-24T00:17:28.423 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:28.418+0000 7f80f924a740 1 --2- 172.21.15.118:0/4079887487 >> [v2:172.21.15.189:6838/4074484168,v1:172.21.15.189:6839/4074484168] conn(0x5650b8f295e0 0x5650b8f2ba00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:17:28.423 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:28.418+0000 7f80f924a740 4 client.25922 mds_command: new command op to 24569 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-24T00:17:28.423 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:28.418+0000 7f80f924a740 1 -- 172.21.15.118:0/4079887487 --> [v2:172.21.15.189:6838/4074484168,v1:172.21.15.189:6839/4074484168] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x5650b8e23b50 con 0x5650b8f295e0 2024-03-24T00:17:28.424 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:28.418+0000 7f80efe00640 1 --2- 172.21.15.118:0/4079887487 >> [v2:172.21.15.189:6838/4074484168,v1:172.21.15.189:6839/4074484168] conn(0x5650b8f295e0 0x5650b8f2ba00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:17:28.424 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:28.419+0000 7f80efe00640 1 --2- 172.21.15.118:0/4079887487 >> [v2:172.21.15.189:6838/4074484168,v1:172.21.15.189:6839/4074484168] conn(0x5650b8f295e0 0x5650b8f2ba00 crc :-1 s=READY pgs=70 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=c9e1720038845391 server_cookie=6074ccf526a64901 in_seq=0 out_seq=0 2024-03-24T00:17:28.424 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:28.419+0000 7f80d9600640 10 client.25922 ms_handle_connect on v2:172.21.15.189:6838/4074484168 2024-03-24T00:17:28.424 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:28.420+0000 7f80d9600640 1 -- 172.21.15.118:0/4079887487 <== mds.0 v2:172.21.15.189:6838/4074484168 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x5650b8e23b50 con 0x5650b8f295e0 2024-03-24T00:17:28.424 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:28.420+0000 7f80d9600640 10 client.25922 handle_command_reply: tid=0 2024-03-24T00:17:28.424 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:28.420+0000 7f80f924a740 1 client.25922 shutdown 2024-03-24T00:17:28.424 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:28.420+0000 7f80f924a740 1 -- 172.21.15.118:0/4079887487 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5650b8eb89e0 msgr2=0x5650b8ec4c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:17:28.424 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:28.420+0000 7f80f924a740 1 --2- 172.21.15.118:0/4079887487 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5650b8eb89e0 0x5650b8ec4c20 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x7f80dc000c00 tx=0x7f80dc005620 comp rx=0 tx=0).stop 2024-03-24T00:17:28.424 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:28.420+0000 7f80f924a740 1 -- 172.21.15.118:0/4079887487 shutdown_connections 2024-03-24T00:17:28.424 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:28.420+0000 7f80f924a740 1 --2- 172.21.15.118:0/4079887487 >> [v2:172.21.15.189:6838/4074484168,v1:172.21.15.189:6839/4074484168] conn(0x5650b8f295e0 0x5650b8f2ba00 crc :-1 s=READY pgs=70 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:17:28.424 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:28.420+0000 7f80f924a740 1 --2- 172.21.15.118:0/4079887487 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5650b8eb93a0 0x5650b8ec7c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:17:28.424 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:28.420+0000 7f80f924a740 1 --2- 172.21.15.118:0/4079887487 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5650b8eb89e0 0x5650b8ec4c20 unknown :-1 s=CLOSED pgs=296 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:17:28.424 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:28.420+0000 7f80d9600640 0 client.25922 ms_handle_reset on v2:172.21.15.189:6838/4074484168 2024-03-24T00:17:28.424 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:28.420+0000 7f80f924a740 1 --2- 172.21.15.118:0/4079887487 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5650b8eb7810 0x5650b8ec46e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:17:28.425 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:28.420+0000 7f80f924a740 1 -- 172.21.15.118:0/4079887487 >> 172.21.15.118:0/4079887487 conn(0x5650b8eb2c00 msgr2=0x5650b8eb5e60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:17:28.425 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:28.420+0000 7f80f924a740 1 -- 172.21.15.118:0/4079887487 shutdown_connections 2024-03-24T00:17:28.425 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:28.420+0000 7f80f924a740 1 -- 172.21.15.118:0/4079887487 wait complete. 2024-03-24T00:17:28.425 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:28.420+0000 7f80f924a740 20 client.25922 trim_cache size 0 max 16384 2024-03-24T00:17:28.425 INFO:teuthology.orchestra.run.smithi118.stdout:[] 2024-03-24T00:17:28.426 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:28.421+0000 7f80f6600640 1 -- 172.21.15.118:0/1526965738 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7f80d40a2880 msgr2=0x7f80d40a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:17:28.426 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:28.421+0000 7f80f6600640 1 --2- 172.21.15.118:0/1526965738 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7f80d40a2880 0x7f80d40a4d40 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 crypto rx=0x7f80e00ee580 tx=0x7f80e00dd040 comp rx=0 tx=0).stop 2024-03-24T00:17:28.426 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:28.421+0000 7f80f6600640 1 -- 172.21.15.118:0/1526965738 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f80f01016b0 msgr2=0x7f80f0199c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:17:28.426 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:28.421+0000 7f80f6600640 1 --2- 172.21.15.118:0/1526965738 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f80f01016b0 0x7f80f0199c50 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 crypto rx=0x7f80e0000c00 tx=0x7f80e002c650 comp rx=0 tx=0).stop 2024-03-24T00:17:28.426 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:28.421+0000 7f80f6600640 1 -- 172.21.15.118:0/1526965738 shutdown_connections 2024-03-24T00:17:28.426 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:28.421+0000 7f80f6600640 1 --2- 172.21.15.118:0/1526965738 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7f80d40a2880 0x7f80d40a4d40 unknown :-1 s=CLOSED pgs=302 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:17:28.426 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:28.421+0000 7f80f6600640 1 --2- 172.21.15.118:0/1526965738 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f80f0103260 0x7f80f01a11c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:17:28.426 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:28.421+0000 7f80f6600640 1 --2- 172.21.15.118:0/1526965738 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f80f01028a0 0x7f80f019a190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:17:28.426 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:28.421+0000 7f80f6600640 1 --2- 172.21.15.118:0/1526965738 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f80f01016b0 0x7f80f0199c50 unknown :-1 s=CLOSED pgs=295 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:17:28.426 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:28.421+0000 7f80f6600640 1 -- 172.21.15.118:0/1526965738 >> 172.21.15.118:0/1526965738 conn(0x7f80f00fcc40 msgr2=0x7f80f00738d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:17:28.426 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:28.421+0000 7f80f6600640 1 -- 172.21.15.118:0/1526965738 shutdown_connections 2024-03-24T00:17:28.426 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:28.421+0000 7f80f6600640 1 -- 172.21.15.118:0/1526965738 wait complete. 2024-03-24T00:17:28.551 INFO:tasks.workunit.client.0.smithi118.stdout:checking for alloca... yes 2024-03-24T00:17:28.694 INFO:tasks.workunit.client.0.smithi118.stdout:checking for working memcmp... yes 2024-03-24T00:17:28.860 INFO:tasks.workunit.client.0.smithi118.stdout:checking for strftime... yes 2024-03-24T00:17:29.075 INFO:tasks.workunit.client.0.smithi118.stdout:checking whether lstat dereferences a symlink specified with a trailing slash... yes 2024-03-24T00:17:29.104 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:17:28 smithi189 ceph-mon[36187]: pgmap v451: 97 pgs: 97 active+clean; 2.1 MiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 60 KiB/s rd, 159 KiB/s wr, 55 op/s 2024-03-24T00:17:29.195 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:17:28 smithi118 ceph-mon[31535]: pgmap v451: 97 pgs: 97 active+clean; 2.1 MiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 60 KiB/s rd, 159 KiB/s wr, 55 op/s 2024-03-24T00:17:29.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:17:28 smithi193 ceph-mon[36878]: pgmap v451: 97 pgs: 97 active+clean; 2.1 MiB data, 357 MiB used, 1.0 TiB / 1.0 TiB avail; 60 KiB/s rd, 159 KiB/s wr, 55 op/s 2024-03-24T00:17:29.221 INFO:tasks.workunit.client.0.smithi118.stdout:checking whether stat accepts an empty string... no 2024-03-24T00:17:29.390 INFO:tasks.workunit.client.0.smithi118.stdout:checking for vprintf... yes 2024-03-24T00:17:29.444 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-03-24T00:17:29.444 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T00:17:29.445 DEBUG:teuthology.orchestra.run.smithi118:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2024-03-24T00:17:29.536 INFO:tasks.workunit.client.0.smithi118.stdout:checking for _doprnt... no 2024-03-24T00:17:29.597 INFO:tasks.workunit.client.0.smithi118.stdout:checking build system type... x86_64-unknown-linux-gnu 2024-03-24T00:17:29.649 INFO:tasks.workunit.client.0.smithi118.stdout:checking host system type... x86_64-unknown-linux-gnu 2024-03-24T00:17:29.666 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:29.665+0000 7f0027200640 1 -- 172.21.15.118:0/4078466409 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0020104990 msgr2=0x7f0020104e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:17:29.666 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:29.665+0000 7f0027200640 1 --2- 172.21.15.118:0/4078466409 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0020104990 0x7f0020104e10 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0x7f0010009900 tx=0x7f001002bc30 comp rx=0 tx=0).stop 2024-03-24T00:17:29.666 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:29.665+0000 7f0027200640 1 -- 172.21.15.118:0/4078466409 shutdown_connections 2024-03-24T00:17:29.666 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:29.665+0000 7f0027200640 1 --2- 172.21.15.118:0/4078466409 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f0020105350 0x7f002010bbe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:17:29.666 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:29.665+0000 7f0027200640 1 --2- 172.21.15.118:0/4078466409 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0020104990 0x7f0020104e10 unknown :-1 s=CLOSED pgs=297 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:17:29.666 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:29.665+0000 7f0027200640 1 --2- 172.21.15.118:0/4078466409 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f00201037a0 0x7f0020103bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:17:29.666 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:29.665+0000 7f0027200640 1 -- 172.21.15.118:0/4078466409 >> 172.21.15.118:0/4078466409 conn(0x7f00200fed10 msgr2=0x7f0020101150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:17:29.666 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:29.665+0000 7f0027200640 1 -- 172.21.15.118:0/4078466409 shutdown_connections 2024-03-24T00:17:29.666 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:29.665+0000 7f0027200640 1 -- 172.21.15.118:0/4078466409 wait complete. 2024-03-24T00:17:29.668 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:29.666+0000 7f0027200640 1 Processor -- start 2024-03-24T00:17:29.668 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:29.666+0000 7f0027200640 1 -- start start 2024-03-24T00:17:29.668 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:29.666+0000 7f0027200640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f00201037a0 0x7f0020079450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:17:29.668 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:29.666+0000 7f0027200640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0020104990 0x7f0020079990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:17:29.669 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:29.666+0000 7f0027200640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f0020105350 0x7f0020075fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:17:29.669 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:29.666+0000 7f0027200640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f002010f450 con 0x7f00201037a0 2024-03-24T00:17:29.669 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:29.666+0000 7f0027200640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f002010f2d0 con 0x7f0020104990 2024-03-24T00:17:29.669 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:29.666+0000 7f0027200640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f002010f5d0 con 0x7f0020105350 2024-03-24T00:17:29.669 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:29.666+0000 7f0025400640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0020104990 0x7f0020079990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:17:29.669 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:29.667+0000 7f0025400640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0020104990 0x7f0020079990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.118:45574/0 (socket says 172.21.15.118:45574) 2024-03-24T00:17:29.669 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:29.667+0000 7f0025400640 1 -- 172.21.15.118:0/4175524288 learned_addr learned my addr 172.21.15.118:0/4175524288 (peer_addr_for_me v2:172.21.15.118:0/0) 2024-03-24T00:17:29.669 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:29.667+0000 7f0025e00640 1 --2- 172.21.15.118:0/4175524288 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f00201037a0 0x7f0020079450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:17:29.669 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:29.667+0000 7f0026800640 1 --2- 172.21.15.118:0/4175524288 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f0020105350 0x7f0020075fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:17:29.669 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:29.667+0000 7f0025400640 1 -- 172.21.15.118:0/4175524288 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f0020105350 msgr2=0x7f0020075fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:17:29.669 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:29.667+0000 7f0025400640 1 --2- 172.21.15.118:0/4175524288 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f0020105350 0x7f0020075fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:17:29.669 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:29.667+0000 7f0025400640 1 -- 172.21.15.118:0/4175524288 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f00201037a0 msgr2=0x7f0020079450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:17:29.669 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:29.667+0000 7f0025400640 1 --2- 172.21.15.118:0/4175524288 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f00201037a0 0x7f0020079450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:17:29.670 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:29.667+0000 7f0025400640 1 -- 172.21.15.118:0/4175524288 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0020076880 con 0x7f0020104990 2024-03-24T00:17:29.670 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:29.667+0000 7f0025e00640 1 --2- 172.21.15.118:0/4175524288 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f00201037a0 0x7f0020079450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T00:17:29.670 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:29.667+0000 7f0025400640 1 --2- 172.21.15.118:0/4175524288 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0020104990 0x7f0020079990 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 crypto rx=0x7f0010005ab0 tx=0x7f001002c6c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:17:29.670 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:29.668+0000 7f001aa00640 1 -- 172.21.15.118:0/4175524288 <== mon.2 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0010043020 con 0x7f0020104990 2024-03-24T00:17:29.670 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:29.668+0000 7f001aa00640 1 -- 172.21.15.118:0/4175524288 <== mon.2 v2:172.21.15.189:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f00100054e0 con 0x7f0020104990 2024-03-24T00:17:29.670 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:29.668+0000 7f0027200640 1 -- 172.21.15.118:0/4175524288 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f00201a5ee0 con 0x7f0020104990 2024-03-24T00:17:29.670 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:29.668+0000 7f001aa00640 1 -- 172.21.15.118:0/4175524288 <== mon.2 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0010003ce0 con 0x7f0020104990 2024-03-24T00:17:29.670 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:29.668+0000 7f0027200640 1 -- 172.21.15.118:0/4175524288 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f00201a6290 con 0x7f0020104990 2024-03-24T00:17:29.672 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:29.671+0000 7f002b60f740 1 Processor -- start 2024-03-24T00:17:29.672 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:29.671+0000 7f002b60f740 1 -- start start 2024-03-24T00:17:29.672 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:29.671+0000 7f001aa00640 1 -- 172.21.15.118:0/4175524288 <== mon.2 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 20) v1 ==== 137291+0+0 (secure 0 0 0) 0x7f0010046020 con 0x7f0020104990 2024-03-24T00:17:29.672 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:29.672+0000 7f002b60f740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55c5e11742b0 0x55c5e11746d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:17:29.673 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:29.672+0000 7f002b60f740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55c5e1174ca0 0x55c5e11797a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:17:29.673 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:29.672+0000 7f002b60f740 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55c5e1179e10 0x55c5e117c2a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:17:29.673 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:29.672+0000 7f002b60f740 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c5e10d9eb0 con 0x55c5e1179e10 2024-03-24T00:17:29.673 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:29.672+0000 7f002b60f740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c5e1082a60 con 0x55c5e1174ca0 2024-03-24T00:17:29.673 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:29.672+0000 7f002b60f740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c5e10d5f50 con 0x55c5e11742b0 2024-03-24T00:17:29.673 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:29.672+0000 7f0025e00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55c5e1174ca0 0x55c5e11797a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:17:29.673 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:29.672+0000 7f0026800640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55c5e11742b0 0x55c5e11746d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:17:29.675 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:29.672+0000 7f0026800640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55c5e11742b0 0x55c5e11746d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.118:37972/0 (socket says 172.21.15.118:37972) 2024-03-24T00:17:29.675 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:29.672+0000 7f0025e00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55c5e1174ca0 0x55c5e11797a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.118:45590/0 (socket says 172.21.15.118:45590) 2024-03-24T00:17:29.675 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:29.672+0000 7f0026800640 1 -- 172.21.15.118:0/1235992073 learned_addr learned my addr 172.21.15.118:0/1235992073 (peer_addr_for_me v2:172.21.15.118:0/0) 2024-03-24T00:17:29.675 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:29.672+0000 7f0025400640 1 --2- 172.21.15.118:0/1235992073 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55c5e1179e10 0x55c5e117c2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:17:29.675 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:29.672+0000 7f0026800640 1 -- 172.21.15.118:0/1235992073 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55c5e1174ca0 msgr2=0x55c5e11797a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:17:29.675 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:29.672+0000 7f0026800640 1 --2- 172.21.15.118:0/1235992073 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55c5e1174ca0 0x55c5e11797a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:17:29.675 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:29.672+0000 7f0026800640 1 -- 172.21.15.118:0/1235992073 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55c5e1179e10 msgr2=0x55c5e117c2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:17:29.675 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:29.672+0000 7f0026800640 1 --2- 172.21.15.118:0/1235992073 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55c5e1179e10 0x55c5e117c2a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:17:29.675 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:29.672+0000 7f0026800640 1 -- 172.21.15.118:0/1235992073 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55c5e0eb67e0 con 0x55c5e11742b0 2024-03-24T00:17:29.675 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:29.673+0000 7f0026800640 1 --2- 172.21.15.118:0/1235992073 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55c5e11742b0 0x55c5e11746d0 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7f001c010370 tx=0x7f001c010830 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d4a07a5214febc4b server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:17:29.675 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:29.673+0000 7f000f000640 1 -- 172.21.15.118:0/1235992073 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f001c016070 con 0x55c5e11742b0 2024-03-24T00:17:29.676 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:29.673+0000 7f000f000640 1 -- 172.21.15.118:0/1235992073 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f001c004d20 con 0x55c5e11742b0 2024-03-24T00:17:29.676 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:29.673+0000 7f000f000640 1 -- 172.21.15.118:0/1235992073 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f001c005420 con 0x55c5e11742b0 2024-03-24T00:17:29.676 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:29.673+0000 7f001aa00640 1 --2- 172.21.15.118:0/4175524288 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7f00080a2800 0x7f00080a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:17:29.676 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:29.673+0000 7f001aa00640 1 -- 172.21.15.118:0/4175524288 <== mon.2 v2:172.21.15.189:3300/0 5 ==== osd_map(90..90 src has 1..90) v4 ==== 8277+0+0 (secure 0 0 0) 0x7f00100f0c20 con 0x7f0020104990 2024-03-24T00:17:29.676 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:29.673+0000 7f0025e00640 1 --2- 172.21.15.118:0/4175524288 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7f00080a2800 0x7f00080a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:17:29.676 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:29.674+0000 7f002b60f740 1 -- 172.21.15.118:0/1235992073 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55c5e11742b0 msgr2=0x55c5e11746d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:17:29.676 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:29.674+0000 7f002b60f740 1 --2- 172.21.15.118:0/1235992073 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55c5e11742b0 0x55c5e11746d0 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7f001c010370 tx=0x7f001c010830 comp rx=0 tx=0).stop 2024-03-24T00:17:29.676 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:29.674+0000 7f002b60f740 1 -- 172.21.15.118:0/1235992073 shutdown_connections 2024-03-24T00:17:29.676 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:29.674+0000 7f002b60f740 1 --2- 172.21.15.118:0/1235992073 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55c5e1179e10 0x55c5e117c2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:17:29.676 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:29.674+0000 7f002b60f740 1 --2- 172.21.15.118:0/1235992073 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55c5e1174ca0 0x55c5e11797a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:17:29.676 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:29.674+0000 7f002b60f740 1 --2- 172.21.15.118:0/1235992073 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55c5e11742b0 0x55c5e11746d0 unknown :-1 s=CLOSED pgs=282 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:17:29.676 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:29.674+0000 7f002b60f740 1 -- 172.21.15.118:0/1235992073 >> 172.21.15.118:0/1235992073 conn(0x55c5e116f820 msgr2=0x55c5e116fc20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:17:29.676 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:29.674+0000 7f002b60f740 1 -- 172.21.15.118:0/1235992073 shutdown_connections 2024-03-24T00:17:29.676 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:29.674+0000 7f002b60f740 1 -- 172.21.15.118:0/1235992073 wait complete. 2024-03-24T00:17:29.677 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:29.674+0000 7f0025e00640 1 --2- 172.21.15.118:0/4175524288 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7f00080a2800 0x7f00080a4cc0 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 crypto rx=0x55c5e1170790 tx=0x7f0014005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:17:29.677 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:29.674+0000 7f002b60f740 1 -- start start 2024-03-24T00:17:29.680 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:29.675+0000 7f002b60f740 1 Processor -- start 2024-03-24T00:17:29.680 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:29.675+0000 7f002b60f740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55c5e11742b0 0x55c5e117e500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:17:29.680 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:29.675+0000 7f002b60f740 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55c5e1174ca0 0x55c5e117ea40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:17:29.680 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:29.675+0000 7f002b60f740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55c5e1179e10 0x55c5e1185a80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:17:29.680 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:29.675+0000 7f002b60f740 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c5e117cdc0 con 0x55c5e1174ca0 2024-03-24T00:17:29.680 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:29.675+0000 7f002b60f740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c5e1095eb0 con 0x55c5e1179e10 2024-03-24T00:17:29.680 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:29.675+0000 7f002b60f740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c5e117cf40 con 0x55c5e11742b0 2024-03-24T00:17:29.680 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:29.675+0000 7f002b60f740 10 client.0 _ll_register_callbacks cb 0x55c5e108a150 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T00:17:29.680 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:29.675+0000 7f0025e00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55c5e11742b0 0x55c5e117e500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:17:29.681 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:29.675+0000 7f0025e00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55c5e11742b0 0x55c5e117e500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.118:37988/0 (socket says 172.21.15.118:37988) 2024-03-24T00:17:29.681 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:29.675+0000 7f0025e00640 1 -- 172.21.15.118:0/2230078843 learned_addr learned my addr 172.21.15.118:0/2230078843 (peer_addr_for_me v2:172.21.15.118:0/0) 2024-03-24T00:17:29.681 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:29.676+0000 7f0026800640 1 --2- 172.21.15.118:0/2230078843 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55c5e1179e10 0x55c5e1185a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:17:29.681 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:29.676+0000 7f0025400640 1 --2- 172.21.15.118:0/2230078843 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55c5e1174ca0 0x55c5e117ea40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:17:29.681 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:29.676+0000 7f0025e00640 1 -- 172.21.15.118:0/2230078843 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55c5e1179e10 msgr2=0x55c5e1185a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:17:29.681 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:29.676+0000 7f0025e00640 1 --2- 172.21.15.118:0/2230078843 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55c5e1179e10 0x55c5e1185a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:17:29.681 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:29.676+0000 7f0025e00640 1 -- 172.21.15.118:0/2230078843 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55c5e1174ca0 msgr2=0x55c5e117ea40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:17:29.681 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:29.676+0000 7f0025e00640 1 --2- 172.21.15.118:0/2230078843 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55c5e1174ca0 0x55c5e117ea40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:17:29.681 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:29.676+0000 7f0025e00640 1 -- 172.21.15.118:0/2230078843 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55c5e0f16f70 con 0x55c5e11742b0 2024-03-24T00:17:29.681 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:29.676+0000 7f0025e00640 1 --2- 172.21.15.118:0/2230078843 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55c5e11742b0 0x55c5e117e500 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7f0014009fd0 tx=0x7f0014004730 comp rx=0 tx=0).ready entity=mon.1 client_cookie=42220a230908218d server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:17:29.681 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:29.676+0000 7f0003400640 10 client.0 ms_handle_connect on v2:172.21.15.193:3300/0 2024-03-24T00:17:29.681 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:29.677+0000 7f0003400640 1 -- 172.21.15.118:0/2230078843 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0014012e90 con 0x55c5e11742b0 2024-03-24T00:17:29.681 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:29.677+0000 7f0003400640 1 -- 172.21.15.118:0/2230078843 <== mon.1 v2:172.21.15.193:3300/0 2 ==== osd_map(90..90 src has 1..90) v4 ==== 8277+0+0 (secure 0 0 0) 0x7f0014003d80 con 0x55c5e11742b0 2024-03-24T00:17:29.681 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:29.677+0000 7f002b60f740 1 -- 172.21.15.118:0/2230078843 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f001c004d20 con 0x55c5e11742b0 2024-03-24T00:17:29.682 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:29.677+0000 7f0003400640 1 -- 172.21.15.118:0/2230078843 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) v2 ==== 24+0+0 (secure 0 0 0) 0x7f001402a420 con 0x55c5e11742b0 2024-03-24T00:17:29.682 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:29.677+0000 7f002b60f740 10 client.25921 fetch_fsmap learned FSMap version 28 2024-03-24T00:17:29.682 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:29.677+0000 7f002b60f740 1 -- 172.21.15.118:0/2230078843 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) v3 -- 0x55c5e0dae170 con 0x55c5e11742b0 2024-03-24T00:17:29.682 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:29.677+0000 7f0003400640 1 -- 172.21.15.118:0/2230078843 <== mon.1 v2:172.21.15.193:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f001c004d20 con 0x55c5e11742b0 2024-03-24T00:17:29.682 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:29.677+0000 7f0003400640 1 -- 172.21.15.118:0/2230078843 <== mon.1 v2:172.21.15.193:3300/0 5 ==== fsmap(e 28) v1 ==== 7934+0+0 (secure 0 0 0) 0x7f0014033070 con 0x55c5e11742b0 2024-03-24T00:17:29.682 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:29.677+0000 7f002b60f740 10 client.25921 fetch_fsmap finished waiting for FSMap version 28 2024-03-24T00:17:29.682 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:29.677+0000 7f002b60f740 10 client.25921 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-24T00:17:29.682 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:29.677+0000 7f002b60f740 20 client.25921 populate_metadata read hostname 'smithi118' 2024-03-24T00:17:29.682 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:29.677+0000 7f002b60f740 1 --2- 172.21.15.118:0/2230078843 >> [v2:172.21.15.189:6838/4074484168,v1:172.21.15.189:6839/4074484168] conn(0x55c5e118a3f0 0x55c5e118c870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:17:29.682 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:29.677+0000 7f002b60f740 4 client.25921 mds_command: new command op to 24569 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-24T00:17:29.682 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:29.677+0000 7f002b60f740 1 -- 172.21.15.118:0/2230078843 --> [v2:172.21.15.189:6838/4074484168,v1:172.21.15.189:6839/4074484168] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55c5e10e73f0 con 0x55c5e118a3f0 2024-03-24T00:17:29.682 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:29.678+0000 7f0025400640 1 --2- 172.21.15.118:0/2230078843 >> [v2:172.21.15.189:6838/4074484168,v1:172.21.15.189:6839/4074484168] conn(0x55c5e118a3f0 0x55c5e118c870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:17:29.682 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:29.678+0000 7f0025400640 1 --2- 172.21.15.118:0/2230078843 >> [v2:172.21.15.189:6838/4074484168,v1:172.21.15.189:6839/4074484168] conn(0x55c5e118a3f0 0x55c5e118c870 crc :-1 s=READY pgs=71 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=e027861e85121861 server_cookie=9a868abe83e6009d in_seq=0 out_seq=0 2024-03-24T00:17:29.683 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:29.678+0000 7f0003400640 10 client.25921 ms_handle_connect on v2:172.21.15.189:6838/4074484168 2024-03-24T00:17:29.683 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:29.679+0000 7f0003400640 1 -- 172.21.15.118:0/2230078843 <== mds.0 v2:172.21.15.189:6838/4074484168 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x55c5e10e73f0 con 0x55c5e118a3f0 2024-03-24T00:17:29.683 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:29.679+0000 7f0003400640 10 client.25921 handle_command_reply: tid=0 2024-03-24T00:17:29.683 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:29.679+0000 7f002b60f740 1 client.25921 shutdown 2024-03-24T00:17:29.683 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:29.679+0000 7f002b60f740 1 -- 172.21.15.118:0/2230078843 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55c5e11742b0 msgr2=0x55c5e117e500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:17:29.683 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:29.679+0000 7f002b60f740 1 --2- 172.21.15.118:0/2230078843 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55c5e11742b0 0x55c5e117e500 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7f0014009fd0 tx=0x7f0014004730 comp rx=0 tx=0).stop 2024-03-24T00:17:29.683 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:29.679+0000 7f002b60f740 1 -- 172.21.15.118:0/2230078843 shutdown_connections 2024-03-24T00:17:29.683 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:29.679+0000 7f002b60f740 1 --2- 172.21.15.118:0/2230078843 >> [v2:172.21.15.189:6838/4074484168,v1:172.21.15.189:6839/4074484168] conn(0x55c5e118a3f0 0x55c5e118c870 crc :-1 s=READY pgs=71 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:17:29.683 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:29.679+0000 7f002b60f740 1 --2- 172.21.15.118:0/2230078843 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55c5e1179e10 0x55c5e1185a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:17:29.683 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:29.679+0000 7f002b60f740 1 --2- 172.21.15.118:0/2230078843 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55c5e1174ca0 0x55c5e117ea40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:17:29.683 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:29.679+0000 7f002b60f740 1 --2- 172.21.15.118:0/2230078843 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55c5e11742b0 0x55c5e117e500 unknown :-1 s=CLOSED pgs=283 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:17:29.683 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:29.679+0000 7f0003400640 0 client.25921 ms_handle_reset on v2:172.21.15.189:6838/4074484168 2024-03-24T00:17:29.683 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:29.679+0000 7f002b60f740 1 -- 172.21.15.118:0/2230078843 >> 172.21.15.118:0/2230078843 conn(0x55c5e116f820 msgr2=0x55c5e116ea80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:17:29.683 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:29.679+0000 7f002b60f740 1 -- 172.21.15.118:0/2230078843 shutdown_connections 2024-03-24T00:17:29.684 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:29.679+0000 7f002b60f740 1 -- 172.21.15.118:0/2230078843 wait complete. 2024-03-24T00:17:29.684 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:29.680+0000 7f002b60f740 20 client.25921 trim_cache size 0 max 16384 2024-03-24T00:17:29.699 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:29.698+0000 7f002b60f740 1 Processor -- start 2024-03-24T00:17:29.699 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:29.698+0000 7f002b60f740 1 -- start start 2024-03-24T00:17:29.699 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:29.698+0000 7f002b60f740 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55c5e11742b0 0x55c5e116e900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:17:29.700 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:29.698+0000 7f002b60f740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55c5e1174ca0 0x55c5e1185a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:17:29.700 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:29.698+0000 7f002b60f740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55c5e1179e10 0x55c5e1191130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:17:29.700 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:29.698+0000 7f002b60f740 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f00201aee00 con 0x55c5e11742b0 2024-03-24T00:17:29.700 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:29.698+0000 7f002b60f740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f00201b0780 con 0x55c5e1179e10 2024-03-24T00:17:29.700 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:29.698+0000 7f002b60f740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f00201af460 con 0x55c5e1174ca0 2024-03-24T00:17:29.700 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:29.698+0000 7f0025400640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55c5e1174ca0 0x55c5e1185a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:17:29.700 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:29.698+0000 7f0025400640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55c5e1174ca0 0x55c5e1185a40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.118:37994/0 (socket says 172.21.15.118:37994) 2024-03-24T00:17:29.700 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:29.698+0000 7f0025400640 1 -- 172.21.15.118:0/373944080 learned_addr learned my addr 172.21.15.118:0/373944080 (peer_addr_for_me v2:172.21.15.118:0/0) 2024-03-24T00:17:29.700 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:29.698+0000 7f0026800640 1 --2- 172.21.15.118:0/373944080 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55c5e1179e10 0x55c5e1191130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:17:29.700 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:29.698+0000 7f0025e00640 1 --2- 172.21.15.118:0/373944080 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55c5e11742b0 0x55c5e116e900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:17:29.700 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:29.699+0000 7f0025400640 1 -- 172.21.15.118:0/373944080 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55c5e1179e10 msgr2=0x55c5e1191130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:17:29.700 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:29.699+0000 7f0025400640 1 --2- 172.21.15.118:0/373944080 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55c5e1179e10 0x55c5e1191130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:17:29.700 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:29.699+0000 7f0025400640 1 -- 172.21.15.118:0/373944080 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55c5e11742b0 msgr2=0x55c5e116e900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:17:29.701 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:29.699+0000 7f0025400640 1 --2- 172.21.15.118:0/373944080 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55c5e11742b0 0x55c5e116e900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:17:29.701 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:29.699+0000 7f0025400640 1 -- 172.21.15.118:0/373944080 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55c5e118a1b0 con 0x55c5e1174ca0 2024-03-24T00:17:29.701 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:29.699+0000 7f0025400640 1 --2- 172.21.15.118:0/373944080 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55c5e1174ca0 0x55c5e1185a40 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7f00100b9f00 tx=0x7f0010041040 comp rx=0 tx=0).ready entity=mon.1 client_cookie=794e020b7b9a3e7f server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:17:29.704 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:29.699+0000 7f000e600640 1 -- 172.21.15.118:0/373944080 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f00100d5020 con 0x55c5e1174ca0 2024-03-24T00:17:29.704 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:29.699+0000 7f000e600640 1 -- 172.21.15.118:0/373944080 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f0010004c80 con 0x55c5e1174ca0 2024-03-24T00:17:29.704 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:29.699+0000 7f000e600640 1 -- 172.21.15.118:0/373944080 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f00100bd530 con 0x55c5e1174ca0 2024-03-24T00:17:29.704 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:29.700+0000 7f002b60f740 1 -- 172.21.15.118:0/373944080 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55c5e1174ca0 msgr2=0x55c5e1185a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:17:29.705 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:29.700+0000 7f002b60f740 1 --2- 172.21.15.118:0/373944080 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55c5e1174ca0 0x55c5e1185a40 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7f00100b9f00 tx=0x7f0010041040 comp rx=0 tx=0).stop 2024-03-24T00:17:29.705 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:29.700+0000 7f002b60f740 1 -- 172.21.15.118:0/373944080 shutdown_connections 2024-03-24T00:17:29.705 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:29.700+0000 7f002b60f740 1 --2- 172.21.15.118:0/373944080 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55c5e1179e10 0x55c5e1191130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:17:29.705 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:29.700+0000 7f002b60f740 1 --2- 172.21.15.118:0/373944080 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55c5e1174ca0 0x55c5e1185a40 unknown :-1 s=CLOSED pgs=284 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:17:29.705 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:29.700+0000 7f002b60f740 1 --2- 172.21.15.118:0/373944080 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55c5e11742b0 0x55c5e116e900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:17:29.705 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:29.700+0000 7f002b60f740 1 -- 172.21.15.118:0/373944080 >> 172.21.15.118:0/373944080 conn(0x55c5e116f820 msgr2=0x55c5e116e1f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:17:29.705 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:29.700+0000 7f002b60f740 1 -- 172.21.15.118:0/373944080 shutdown_connections 2024-03-24T00:17:29.705 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:29.700+0000 7f002b60f740 1 -- 172.21.15.118:0/373944080 wait complete. 2024-03-24T00:17:29.705 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:29.701+0000 7f002b60f740 1 -- start start 2024-03-24T00:17:29.705 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:29.701+0000 7f002b60f740 1 Processor -- start 2024-03-24T00:17:29.705 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:29.701+0000 7f002b60f740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55c5e11742b0 0x55c5e1184930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:17:29.705 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:29.701+0000 7f002b60f740 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55c5e1174ca0 0x55c5e1184e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:17:29.705 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:29.701+0000 7f002b60f740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55c5e1179e10 0x55c5e11e5d00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:17:29.705 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:29.701+0000 7f002b60f740 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f00201b0120 con 0x55c5e1174ca0 2024-03-24T00:17:29.706 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:29.701+0000 7f002b60f740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f00201afac0 con 0x55c5e11742b0 2024-03-24T00:17:29.706 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:29.701+0000 7f002b60f740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f00201acfc0 con 0x55c5e1179e10 2024-03-24T00:17:29.706 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:29.701+0000 7f002b60f740 10 client.0 _ll_register_callbacks cb 0x7f00140044a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T00:17:29.706 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:29.701+0000 7f0025e00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55c5e11742b0 0x55c5e1184930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:17:29.706 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:29.701+0000 7f0025e00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55c5e11742b0 0x55c5e1184930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.118:45600/0 (socket says 172.21.15.118:45600) 2024-03-24T00:17:29.706 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:29.701+0000 7f0025e00640 1 -- 172.21.15.118:0/3162922949 learned_addr learned my addr 172.21.15.118:0/3162922949 (peer_addr_for_me v2:172.21.15.118:0/0) 2024-03-24T00:17:29.706 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:29.701+0000 7f0025400640 1 --2- 172.21.15.118:0/3162922949 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55c5e1174ca0 0x55c5e1184e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:17:29.706 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:29.701+0000 7f0026800640 1 --2- 172.21.15.118:0/3162922949 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55c5e1179e10 0x55c5e11e5d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:17:29.706 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:29.702+0000 7f0025e00640 1 -- 172.21.15.118:0/3162922949 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55c5e1179e10 msgr2=0x55c5e11e5d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:17:29.706 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:29.702+0000 7f0025e00640 1 --2- 172.21.15.118:0/3162922949 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55c5e1179e10 0x55c5e11e5d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:17:29.706 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:29.702+0000 7f0025e00640 1 -- 172.21.15.118:0/3162922949 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55c5e1174ca0 msgr2=0x55c5e1184e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:17:29.706 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:29.702+0000 7f0025e00640 1 --2- 172.21.15.118:0/3162922949 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55c5e1174ca0 0x55c5e1184e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:17:29.706 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:29.702+0000 7f0025e00640 1 -- 172.21.15.118:0/3162922949 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55c5e11e6240 con 0x55c5e11742b0 2024-03-24T00:17:29.706 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:29.702+0000 7f0025e00640 1 --2- 172.21.15.118:0/3162922949 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55c5e11742b0 0x55c5e1184930 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 crypto rx=0x7f00140209c0 tx=0x7f0014020f10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=87c6b2866cccc532 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:17:29.707 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:29.702+0000 7f0003400640 10 client.0 ms_handle_connect on v2:172.21.15.189:3300/0 2024-03-24T00:17:29.707 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:29.702+0000 7f0003400640 1 -- 172.21.15.118:0/3162922949 <== mon.2 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0014003e40 con 0x55c5e11742b0 2024-03-24T00:17:29.707 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:29.702+0000 7f0003400640 1 -- 172.21.15.118:0/3162922949 <== mon.2 v2:172.21.15.189:3300/0 2 ==== osd_map(90..90 src has 1..90) v4 ==== 8277+0+0 (secure 0 0 0) 0x7f0014020410 con 0x55c5e11742b0 2024-03-24T00:17:29.707 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:29.702+0000 7f002b60f740 1 -- 172.21.15.118:0/3162922949 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f0010004c80 con 0x55c5e11742b0 2024-03-24T00:17:29.707 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:29.703+0000 7f0003400640 1 -- 172.21.15.118:0/3162922949 <== mon.2 v2:172.21.15.189:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) v2 ==== 24+0+0 (secure 0 0 0) 0x7f0014020610 con 0x55c5e11742b0 2024-03-24T00:17:29.707 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:29.703+0000 7f002b60f740 10 client.25940 fetch_fsmap learned FSMap version 28 2024-03-24T00:17:29.707 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:29.703+0000 7f002b60f740 1 -- 172.21.15.118:0/3162922949 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) v3 -- 0x55c5e11e7500 con 0x55c5e11742b0 2024-03-24T00:17:29.707 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:29.703+0000 7f0003400640 1 -- 172.21.15.118:0/3162922949 <== mon.2 v2:172.21.15.189:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f0010004c80 con 0x55c5e11742b0 2024-03-24T00:17:29.707 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:29.703+0000 7f0003400640 1 -- 172.21.15.118:0/3162922949 <== mon.2 v2:172.21.15.189:3300/0 5 ==== fsmap(e 28) v1 ==== 7934+0+0 (secure 0 0 0) 0x7f0014033040 con 0x55c5e11742b0 2024-03-24T00:17:29.707 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:29.703+0000 7f002b60f740 10 client.25940 fetch_fsmap finished waiting for FSMap version 28 2024-03-24T00:17:29.707 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:29.703+0000 7f002b60f740 10 client.25940 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-24T00:17:29.707 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:29.703+0000 7f002b60f740 20 client.25940 populate_metadata read hostname 'smithi118' 2024-03-24T00:17:29.707 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:29.703+0000 7f002b60f740 1 --2- 172.21.15.118:0/3162922949 >> [v2:172.21.15.189:6838/4074484168,v1:172.21.15.189:6839/4074484168] conn(0x55c5e11e7e30 0x55c5e11ea250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:17:29.708 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:29.703+0000 7f002b60f740 4 client.25940 mds_command: new command op to 24569 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-03-24T00:17:29.708 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:29.703+0000 7f002b60f740 1 -- 172.21.15.118:0/3162922949 --> [v2:172.21.15.189:6838/4074484168,v1:172.21.15.189:6839/4074484168] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) v1 -- 0x55c5e10e1740 con 0x55c5e11e7e30 2024-03-24T00:17:29.708 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:29.703+0000 7f0025400640 1 --2- 172.21.15.118:0/3162922949 >> [v2:172.21.15.189:6838/4074484168,v1:172.21.15.189:6839/4074484168] conn(0x55c5e11e7e30 0x55c5e11ea250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:17:29.708 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:29.704+0000 7f0025400640 1 --2- 172.21.15.118:0/3162922949 >> [v2:172.21.15.189:6838/4074484168,v1:172.21.15.189:6839/4074484168] conn(0x55c5e11e7e30 0x55c5e11ea250 crc :-1 s=READY pgs=72 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=3658004c3d97f6a8 server_cookie=c6b69b836260592f in_seq=0 out_seq=0 2024-03-24T00:17:29.708 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:29.704+0000 7f0003400640 10 client.25940 ms_handle_connect on v2:172.21.15.189:6838/4074484168 2024-03-24T00:17:29.709 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:29.705+0000 7f0003400640 1 -- 172.21.15.118:0/3162922949 <== mds.0 v2:172.21.15.189:6838/4074484168 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+110 (crc 0 0 0) 0x55c5e10e1740 con 0x55c5e11e7e30 2024-03-24T00:17:29.709 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:29.705+0000 7f0003400640 10 client.25940 handle_command_reply: tid=0 2024-03-24T00:17:29.709 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:29.705+0000 7f002b60f740 1 client.25940 shutdown 2024-03-24T00:17:29.709 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:29.705+0000 7f002b60f740 1 -- 172.21.15.118:0/3162922949 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55c5e11742b0 msgr2=0x55c5e1184930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:17:29.709 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:29.705+0000 7f002b60f740 1 --2- 172.21.15.118:0/3162922949 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55c5e11742b0 0x55c5e1184930 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 crypto rx=0x7f00140209c0 tx=0x7f0014020f10 comp rx=0 tx=0).stop 2024-03-24T00:17:29.709 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:29.705+0000 7f002b60f740 1 -- 172.21.15.118:0/3162922949 shutdown_connections 2024-03-24T00:17:29.709 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:29.705+0000 7f002b60f740 1 --2- 172.21.15.118:0/3162922949 >> [v2:172.21.15.189:6838/4074484168,v1:172.21.15.189:6839/4074484168] conn(0x55c5e11e7e30 0x55c5e11ea250 crc :-1 s=READY pgs=72 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:17:29.709 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:29.705+0000 7f002b60f740 1 --2- 172.21.15.118:0/3162922949 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55c5e1179e10 0x55c5e11e5d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:17:29.709 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:29.705+0000 7f002b60f740 1 --2- 172.21.15.118:0/3162922949 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55c5e1174ca0 0x55c5e1184e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:17:29.710 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:29.705+0000 7f002b60f740 1 --2- 172.21.15.118:0/3162922949 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55c5e11742b0 0x55c5e1184930 unknown :-1 s=CLOSED pgs=299 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:17:29.710 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:29.705+0000 7f002b60f740 1 -- 172.21.15.118:0/3162922949 >> 172.21.15.118:0/3162922949 conn(0x55c5e116f820 msgr2=0x55c5e116ea80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:17:29.710 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:29.705+0000 7f0003400640 0 client.25940 ms_handle_reset on v2:172.21.15.189:6838/4074484168 2024-03-24T00:17:29.710 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:29.705+0000 7f002b60f740 1 -- 172.21.15.118:0/3162922949 shutdown_connections 2024-03-24T00:17:29.710 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:29.705+0000 7f002b60f740 1 -- 172.21.15.118:0/3162922949 wait complete. 2024-03-24T00:17:29.710 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:29.705+0000 7f002b60f740 20 client.25940 trim_cache size 0 max 16384 2024-03-24T00:17:29.710 INFO:teuthology.orchestra.run.smithi118.stdout:{ 2024-03-24T00:17:29.710 INFO:teuthology.orchestra.run.smithi118.stdout: "return_code": 0, 2024-03-24T00:17:29.710 INFO:teuthology.orchestra.run.smithi118.stdout: "scrub_tag": "961ace21-39d6-4b4e-a091-b12c4f1cfeaa", 2024-03-24T00:17:29.711 INFO:teuthology.orchestra.run.smithi118.stdout: "mode": "asynchronous" 2024-03-24T00:17:29.711 INFO:teuthology.orchestra.run.smithi118.stdout:} 2024-03-24T00:17:29.711 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:29.706+0000 7f0027200640 1 -- 172.21.15.118:0/4175524288 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7f00080a2800 msgr2=0x7f00080a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:17:29.711 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:29.706+0000 7f0027200640 1 --2- 172.21.15.118:0/4175524288 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7f00080a2800 0x7f00080a4cc0 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 crypto rx=0x55c5e1170790 tx=0x7f0014005f90 comp rx=0 tx=0).stop 2024-03-24T00:17:29.711 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:29.706+0000 7f0027200640 1 -- 172.21.15.118:0/4175524288 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0020104990 msgr2=0x7f0020079990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:17:29.711 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:29.706+0000 7f0027200640 1 --2- 172.21.15.118:0/4175524288 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0020104990 0x7f0020079990 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 crypto rx=0x7f0010005ab0 tx=0x7f001002c6c0 comp rx=0 tx=0).stop 2024-03-24T00:17:29.712 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:29.706+0000 7f0027200640 1 -- 172.21.15.118:0/4175524288 shutdown_connections 2024-03-24T00:17:29.712 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:29.706+0000 7f0027200640 1 --2- 172.21.15.118:0/4175524288 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7f00080a2800 0x7f00080a4cc0 unknown :-1 s=CLOSED pgs=303 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:17:29.712 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:29.706+0000 7f0027200640 1 --2- 172.21.15.118:0/4175524288 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f0020105350 0x7f0020075fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:17:29.712 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:29.706+0000 7f0027200640 1 --2- 172.21.15.118:0/4175524288 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0020104990 0x7f0020079990 unknown :-1 s=CLOSED pgs=298 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:17:29.712 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:29.706+0000 7f0027200640 1 --2- 172.21.15.118:0/4175524288 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f00201037a0 0x7f0020079450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:17:29.712 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:29.706+0000 7f0027200640 1 -- 172.21.15.118:0/4175524288 >> 172.21.15.118:0/4175524288 conn(0x7f00200fed10 msgr2=0x7f0020107c90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:17:29.712 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:29.707+0000 7f0027200640 1 -- 172.21.15.118:0/4175524288 shutdown_connections 2024-03-24T00:17:29.712 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:29.707+0000 7f0027200640 1 -- 172.21.15.118:0/4175524288 wait complete. 2024-03-24T00:17:29.725 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-03-24T00:17:29.725 DEBUG:teuthology.orchestra.run.smithi118:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-24T00:17:29.803 INFO:tasks.workunit.client.0.smithi118.stdout:checking for the pthreads library -lpthreads... no 2024-03-24T00:17:29.916 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:29.915+0000 7fdc19600640 1 -- 172.21.15.118:0/1466009594 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fdc14105490 msgr2=0x7fdc1410bd20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:17:29.916 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:29.915+0000 7fdc19600640 1 --2- 172.21.15.118:0/1466009594 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fdc14105490 0x7fdc1410bd20 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 crypto rx=0x7fdc0c009ca0 tx=0x7fdc0c02c100 comp rx=0 tx=0).stop 2024-03-24T00:17:29.916 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:29.915+0000 7fdc19600640 1 -- 172.21.15.118:0/1466009594 shutdown_connections 2024-03-24T00:17:29.916 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:29.915+0000 7fdc19600640 1 --2- 172.21.15.118:0/1466009594 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fdc14105490 0x7fdc1410bd20 unknown :-1 s=CLOSED pgs=300 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:17:29.917 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:29.915+0000 7fdc19600640 1 --2- 172.21.15.118:0/1466009594 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fdc14104ad0 0x7fdc14104f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:17:29.917 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:29.915+0000 7fdc19600640 1 --2- 172.21.15.118:0/1466009594 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fdc141038e0 0x7fdc14103d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:17:29.917 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:29.915+0000 7fdc19600640 1 -- 172.21.15.118:0/1466009594 >> 172.21.15.118:0/1466009594 conn(0x7fdc140fee50 msgr2=0x7fdc14101290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:17:29.917 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:29.915+0000 7fdc19600640 1 -- 172.21.15.118:0/1466009594 shutdown_connections 2024-03-24T00:17:29.917 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:29.915+0000 7fdc19600640 1 -- 172.21.15.118:0/1466009594 wait complete. 2024-03-24T00:17:29.919 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:29.916+0000 7fdc19600640 1 Processor -- start 2024-03-24T00:17:29.919 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:29.916+0000 7fdc19600640 1 -- start start 2024-03-24T00:17:29.919 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:29.917+0000 7fdc19600640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fdc141038e0 0x7fdc14199b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:17:29.919 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:29.917+0000 7fdc19600640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fdc14104ad0 0x7fdc1419a090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:17:29.919 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:29.917+0000 7fdc19600640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fdc14105490 0x7fdc141a1100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:17:29.920 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:29.917+0000 7fdc19600640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdc1410f540 con 0x7fdc14105490 2024-03-24T00:17:29.920 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:29.917+0000 7fdc19600640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdc1410f3c0 con 0x7fdc141038e0 2024-03-24T00:17:29.920 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:29.917+0000 7fdc19600640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdc1410f6c0 con 0x7fdc14104ad0 2024-03-24T00:17:29.920 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:29.917+0000 7fdc12a00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fdc141038e0 0x7fdc14199b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:17:29.920 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:29.917+0000 7fdc12000640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fdc14104ad0 0x7fdc1419a090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:17:29.920 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:29.917+0000 7fdc12a00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fdc141038e0 0x7fdc14199b50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.118:45614/0 (socket says 172.21.15.118:45614) 2024-03-24T00:17:29.920 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:29.917+0000 7fdc12a00640 1 -- 172.21.15.118:0/2226737263 learned_addr learned my addr 172.21.15.118:0/2226737263 (peer_addr_for_me v2:172.21.15.118:0/0) 2024-03-24T00:17:29.920 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:29.917+0000 7fdc12000640 1 --2- 172.21.15.118:0/2226737263 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fdc14104ad0 0x7fdc1419a090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.118:38024/0 (socket says 172.21.15.118:38024) 2024-03-24T00:17:29.920 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:29.917+0000 7fdc13400640 1 --2- 172.21.15.118:0/2226737263 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fdc14105490 0x7fdc141a1100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:17:29.920 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:29.917+0000 7fdc12a00640 1 -- 172.21.15.118:0/2226737263 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fdc14104ad0 msgr2=0x7fdc1419a090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:17:29.920 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:29.917+0000 7fdc12a00640 1 --2- 172.21.15.118:0/2226737263 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fdc14104ad0 0x7fdc1419a090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:17:29.920 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:29.917+0000 7fdc12a00640 1 -- 172.21.15.118:0/2226737263 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fdc14105490 msgr2=0x7fdc141a1100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:17:29.920 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:29.917+0000 7fdc12a00640 1 --2- 172.21.15.118:0/2226737263 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fdc14105490 0x7fdc141a1100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:17:29.921 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:29.917+0000 7fdc12a00640 1 -- 172.21.15.118:0/2226737263 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdc141a1770 con 0x7fdc141038e0 2024-03-24T00:17:29.921 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:29.917+0000 7fdc13400640 1 --2- 172.21.15.118:0/2226737263 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fdc14105490 0x7fdc141a1100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T00:17:29.921 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:29.918+0000 7fdc12a00640 1 --2- 172.21.15.118:0/2226737263 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fdc141038e0 0x7fdc14199b50 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 crypto rx=0x7fdc0400e9c0 tx=0x7fdc0400ee80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:17:29.921 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:29.918+0000 7fdc0b400640 1 -- 172.21.15.118:0/2226737263 <== mon.2 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdc0400cde0 con 0x7fdc141038e0 2024-03-24T00:17:29.921 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:29.918+0000 7fdc0b400640 1 -- 172.21.15.118:0/2226737263 <== mon.2 v2:172.21.15.189:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fdc04010910 con 0x7fdc141038e0 2024-03-24T00:17:29.921 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:29.918+0000 7fdc19600640 1 -- 172.21.15.118:0/2226737263 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdc141a1a60 con 0x7fdc141038e0 2024-03-24T00:17:29.921 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:29.918+0000 7fdc0b400640 1 -- 172.21.15.118:0/2226737263 <== mon.2 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdc04018a00 con 0x7fdc141038e0 2024-03-24T00:17:29.921 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:29.918+0000 7fdc19600640 1 -- 172.21.15.118:0/2226737263 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdc141a2000 con 0x7fdc141038e0 2024-03-24T00:17:29.921 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:29.921+0000 7fdc0b400640 1 -- 172.21.15.118:0/2226737263 <== mon.2 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 20) v1 ==== 137291+0+0 (secure 0 0 0) 0x7fdc0401f020 con 0x7fdc141038e0 2024-03-24T00:17:29.922 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:29.922+0000 7fdc1bffd740 1 Processor -- start 2024-03-24T00:17:29.923 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:29.922+0000 7fdc1bffd740 1 -- start start 2024-03-24T00:17:29.923 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:29.922+0000 7fdc1bffd740 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55730e7f4e80 0x55730e7f52a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:17:29.924 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:29.922+0000 7fdc1bffd740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55730e7f5870 0x55730e7fa370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:17:29.924 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:29.922+0000 7fdc1bffd740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55730e7fa9e0 0x55730e7fce70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:17:29.924 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:29.922+0000 7fdc1bffd740 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x55730e75a250 con 0x55730e7f4e80 2024-03-24T00:17:29.924 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:29.922+0000 7fdc1bffd740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x55730e702a30 con 0x55730e7f5870 2024-03-24T00:17:29.924 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:29.922+0000 7fdc1bffd740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x55730e7562f0 con 0x55730e7fa9e0 2024-03-24T00:17:29.924 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:29.923+0000 7fdc12a00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55730e7f5870 0x55730e7fa370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:17:29.924 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:29.923+0000 7fdc13400640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55730e7f4e80 0x55730e7f52a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:17:29.924 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:29.923+0000 7fdc12a00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55730e7f5870 0x55730e7fa370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.118:45624/0 (socket says 172.21.15.118:45624) 2024-03-24T00:17:29.924 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:29.923+0000 7fdc12a00640 1 -- 172.21.15.118:0/2611088161 learned_addr learned my addr 172.21.15.118:0/2611088161 (peer_addr_for_me v2:172.21.15.118:0/0) 2024-03-24T00:17:29.924 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:29.923+0000 7fdc12000640 1 --2- 172.21.15.118:0/2611088161 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55730e7fa9e0 0x55730e7fce70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:17:29.924 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:29.923+0000 7fdc0b400640 1 --2- 172.21.15.118:0/2226737263 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7fdbf80a2880 0x7fdbf80a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:17:29.924 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:29.923+0000 7fdc12a00640 1 -- 172.21.15.118:0/2611088161 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55730e7fa9e0 msgr2=0x55730e7fce70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:17:29.926 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:29.923+0000 7fdc0b400640 1 -- 172.21.15.118:0/2226737263 <== mon.2 v2:172.21.15.189:3300/0 5 ==== osd_map(90..90 src has 1..90) v4 ==== 8277+0+0 (secure 0 0 0) 0x7fdc040ced00 con 0x7fdc141038e0 2024-03-24T00:17:29.926 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:29.923+0000 7fdc12a00640 1 --2- 172.21.15.118:0/2611088161 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55730e7fa9e0 0x55730e7fce70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:17:29.926 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:29.923+0000 7fdc12a00640 1 -- 172.21.15.118:0/2611088161 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55730e7f4e80 msgr2=0x55730e7f52a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:17:29.926 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:29.924+0000 7fdc12a00640 1 --2- 172.21.15.118:0/2611088161 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55730e7f4e80 0x55730e7f52a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:17:29.926 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:29.924+0000 7fdc12a00640 1 -- 172.21.15.118:0/2611088161 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55730e6ed690 con 0x55730e7f5870 2024-03-24T00:17:29.927 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:29.924+0000 7fdc12000640 1 --2- 172.21.15.118:0/2611088161 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55730e7fa9e0 0x55730e7fce70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T00:17:29.927 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:29.924+0000 7fdc13400640 1 --2- 172.21.15.118:0/2611088161 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55730e7f4e80 0x55730e7f52a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T00:17:29.927 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:29.924+0000 7fdc13400640 1 --2- 172.21.15.118:0/2226737263 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7fdbf80a2880 0x7fdbf80a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:17:29.927 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:29.924+0000 7fdc12a00640 1 --2- 172.21.15.118:0/2611088161 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55730e7f5870 0x55730e7fa370 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 crypto rx=0x7fdc04097a70 tx=0x7fdc0400efb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ae78b54bb3acca7a server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:17:29.927 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:29.924+0000 7fdbff400640 1 -- 172.21.15.118:0/2611088161 <== mon.2 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdc0409b3e0 con 0x55730e7f5870 2024-03-24T00:17:29.927 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:29.924+0000 7fdbff400640 1 -- 172.21.15.118:0/2611088161 <== mon.2 v2:172.21.15.189:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fdc04097040 con 0x55730e7f5870 2024-03-24T00:17:29.927 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:29.924+0000 7fdbff400640 1 -- 172.21.15.118:0/2611088161 <== mon.2 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdc0409af70 con 0x55730e7f5870 2024-03-24T00:17:29.927 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:29.925+0000 7fdc1bffd740 1 -- 172.21.15.118:0/2611088161 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55730e7f5870 msgr2=0x55730e7fa370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:17:29.927 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:29.925+0000 7fdc1bffd740 1 --2- 172.21.15.118:0/2611088161 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55730e7f5870 0x55730e7fa370 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 crypto rx=0x7fdc04097a70 tx=0x7fdc0400efb0 comp rx=0 tx=0).stop 2024-03-24T00:17:29.927 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:29.925+0000 7fdc13400640 1 --2- 172.21.15.118:0/2226737263 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7fdbf80a2880 0x7fdbf80a4d40 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 crypto rx=0x7fdc0c02c640 tx=0x7fdc0c02e040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:17:29.927 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:29.925+0000 7fdc1bffd740 1 -- 172.21.15.118:0/2611088161 shutdown_connections 2024-03-24T00:17:29.927 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:29.925+0000 7fdc1bffd740 1 --2- 172.21.15.118:0/2611088161 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55730e7fa9e0 0x55730e7fce70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:17:29.927 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:29.925+0000 7fdc1bffd740 1 --2- 172.21.15.118:0/2611088161 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55730e7f5870 0x55730e7fa370 unknown :-1 s=CLOSED pgs=302 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:17:29.927 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:29.925+0000 7fdc1bffd740 1 --2- 172.21.15.118:0/2611088161 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55730e7f4e80 0x55730e7f52a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:17:29.927 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:29.925+0000 7fdc1bffd740 1 -- 172.21.15.118:0/2611088161 >> 172.21.15.118:0/2611088161 conn(0x55730e7bd200 msgr2=0x55730e7f07f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:17:29.928 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:29.925+0000 7fdc1bffd740 1 -- 172.21.15.118:0/2611088161 shutdown_connections 2024-03-24T00:17:29.928 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:29.925+0000 7fdc1bffd740 1 -- 172.21.15.118:0/2611088161 wait complete. 2024-03-24T00:17:29.928 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:29.926+0000 7fdc1bffd740 1 -- start start 2024-03-24T00:17:29.928 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:29.926+0000 7fdc1bffd740 1 Processor -- start 2024-03-24T00:17:29.931 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:29.926+0000 7fdc1bffd740 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55730e7f4e80 0x55730e7fed90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:17:29.932 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:29.926+0000 7fdc1bffd740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55730e7f5870 0x55730e7ff2d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:17:29.932 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:29.926+0000 7fdc1bffd740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55730e7fa9e0 0x55730e806310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:17:29.932 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:29.926+0000 7fdc1bffd740 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x55730e7fd930 con 0x55730e7f4e80 2024-03-24T00:17:29.932 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:29.926+0000 7fdc1bffd740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x55730e715e80 con 0x55730e7f5870 2024-03-24T00:17:29.932 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:29.926+0000 7fdc1bffd740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x55730e7fdab0 con 0x55730e7fa9e0 2024-03-24T00:17:29.932 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:29.926+0000 7fdc1bffd740 10 client.0 _ll_register_callbacks cb 0x55730e728130 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T00:17:29.932 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:29.927+0000 7fdc12a00640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55730e7f4e80 0x55730e7fed90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:17:29.932 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:29.927+0000 7fdc13400640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55730e7fa9e0 0x55730e806310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:17:29.932 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:29.927+0000 7fdc13400640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55730e7fa9e0 0x55730e806310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.118:38046/0 (socket says 172.21.15.118:38046) 2024-03-24T00:17:29.932 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:29.927+0000 7fdc12a00640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55730e7f4e80 0x55730e7fed90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.118:3300/0 says I am v2:172.21.15.118:38632/0 (socket says 172.21.15.118:38632) 2024-03-24T00:17:29.932 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:29.927+0000 7fdc13400640 1 -- 172.21.15.118:0/3422159689 learned_addr learned my addr 172.21.15.118:0/3422159689 (peer_addr_for_me v2:172.21.15.118:0/0) 2024-03-24T00:17:29.932 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:29.927+0000 7fdc12000640 1 --2- 172.21.15.118:0/3422159689 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55730e7f5870 0x55730e7ff2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:17:29.933 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:29.927+0000 7fdc13400640 1 -- 172.21.15.118:0/3422159689 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55730e7f5870 msgr2=0x55730e7ff2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:17:29.933 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:29.927+0000 7fdc13400640 1 --2- 172.21.15.118:0/3422159689 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55730e7f5870 0x55730e7ff2d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:17:29.933 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:29.927+0000 7fdc13400640 1 -- 172.21.15.118:0/3422159689 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55730e7f4e80 msgr2=0x55730e7fed90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:17:29.933 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:29.927+0000 7fdc13400640 1 --2- 172.21.15.118:0/3422159689 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55730e7f4e80 0x55730e7fed90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:17:29.933 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:29.927+0000 7fdc13400640 1 -- 172.21.15.118:0/3422159689 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55730e5387d0 con 0x55730e7fa9e0 2024-03-24T00:17:29.933 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:29.927+0000 7fdc13400640 1 --2- 172.21.15.118:0/3422159689 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55730e7fa9e0 0x55730e806310 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7fdc0c009ca0 tx=0x7fdc0c005ef0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=1245cfbc817bd662 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:17:29.933 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:29.927+0000 7fdbf3e00640 10 client.0 ms_handle_connect on v2:172.21.15.193:3300/0 2024-03-24T00:17:29.933 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:29.928+0000 7fdbf3e00640 1 -- 172.21.15.118:0/3422159689 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdc0c051060 con 0x55730e7fa9e0 2024-03-24T00:17:29.933 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:29.928+0000 7fdbf3e00640 1 -- 172.21.15.118:0/3422159689 <== mon.1 v2:172.21.15.193:3300/0 2 ==== osd_map(90..90 src has 1..90) v4 ==== 8277+0+0 (secure 0 0 0) 0x7fdc0c034d20 con 0x55730e7fa9e0 2024-03-24T00:17:29.933 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:29.928+0000 7fdc1bffd740 1 -- 172.21.15.118:0/3422159689 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fdc04097040 con 0x55730e7fa9e0 2024-03-24T00:17:29.933 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:29.928+0000 7fdbf3e00640 1 -- 172.21.15.118:0/3422159689 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) v2 ==== 24+0+0 (secure 0 0 0) 0x7fdc0c045070 con 0x55730e7fa9e0 2024-03-24T00:17:29.933 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:29.928+0000 7fdc1bffd740 10 client.25939 fetch_fsmap learned FSMap version 28 2024-03-24T00:17:29.933 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:29.928+0000 7fdc1bffd740 1 -- 172.21.15.118:0/3422159689 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) v3 -- 0x55730e46e3f0 con 0x55730e7fa9e0 2024-03-24T00:17:29.933 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:29.928+0000 7fdbf3e00640 1 -- 172.21.15.118:0/3422159689 <== mon.1 v2:172.21.15.193:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fdc04097040 con 0x55730e7fa9e0 2024-03-24T00:17:29.934 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:29.928+0000 7fdbf3e00640 1 -- 172.21.15.118:0/3422159689 <== mon.1 v2:172.21.15.193:3300/0 5 ==== fsmap(e 28) v1 ==== 7934+0+0 (secure 0 0 0) 0x7fdc0c03f040 con 0x55730e7fa9e0 2024-03-24T00:17:29.934 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:29.929+0000 7fdc1bffd740 10 client.25939 fetch_fsmap finished waiting for FSMap version 28 2024-03-24T00:17:29.934 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:29.929+0000 7fdc1bffd740 10 client.25939 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-24T00:17:29.934 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:29.929+0000 7fdc1bffd740 20 client.25939 populate_metadata read hostname 'smithi118' 2024-03-24T00:17:29.934 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:29.929+0000 7fdc1bffd740 1 --2- 172.21.15.118:0/3422159689 >> [v2:172.21.15.189:6838/4074484168,v1:172.21.15.189:6839/4074484168] conn(0x55730e80abd0 0x55730e80d030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:17:29.934 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:29.929+0000 7fdc1bffd740 4 client.25939 mds_command: new command op to 24569 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-24T00:17:29.934 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:29.929+0000 7fdc1bffd740 1 -- 172.21.15.118:0/3422159689 --> [v2:172.21.15.189:6838/4074484168,v1:172.21.15.189:6839/4074484168] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55730e767790 con 0x55730e80abd0 2024-03-24T00:17:29.934 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:29.929+0000 7fdc12a00640 1 --2- 172.21.15.118:0/3422159689 >> [v2:172.21.15.189:6838/4074484168,v1:172.21.15.189:6839/4074484168] conn(0x55730e80abd0 0x55730e80d030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:17:29.934 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:29.929+0000 7fdc12a00640 1 --2- 172.21.15.118:0/3422159689 >> [v2:172.21.15.189:6838/4074484168,v1:172.21.15.189:6839/4074484168] conn(0x55730e80abd0 0x55730e80d030 crc :-1 s=READY pgs=73 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=b5b27bc0daee505b server_cookie=65a75054807a2671 in_seq=0 out_seq=0 2024-03-24T00:17:29.934 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:29.929+0000 7fdbf3e00640 10 client.25939 ms_handle_connect on v2:172.21.15.189:6838/4074484168 2024-03-24T00:17:29.934 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:29.930+0000 7fdbf3e00640 1 -- 172.21.15.118:0/3422159689 <== mds.0 v2:172.21.15.189:6838/4074484168 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x55730e767790 con 0x55730e80abd0 2024-03-24T00:17:29.934 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:29.930+0000 7fdbf3e00640 10 client.25939 handle_command_reply: tid=0 2024-03-24T00:17:29.934 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:29.930+0000 7fdc1bffd740 1 client.25939 shutdown 2024-03-24T00:17:29.934 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:29.930+0000 7fdc1bffd740 1 -- 172.21.15.118:0/3422159689 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55730e7fa9e0 msgr2=0x55730e806310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:17:29.935 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:29.930+0000 7fdc1bffd740 1 --2- 172.21.15.118:0/3422159689 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55730e7fa9e0 0x55730e806310 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7fdc0c009ca0 tx=0x7fdc0c005ef0 comp rx=0 tx=0).stop 2024-03-24T00:17:29.935 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:29.930+0000 7fdc1bffd740 1 -- 172.21.15.118:0/3422159689 shutdown_connections 2024-03-24T00:17:29.935 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:29.930+0000 7fdc1bffd740 1 --2- 172.21.15.118:0/3422159689 >> [v2:172.21.15.189:6838/4074484168,v1:172.21.15.189:6839/4074484168] conn(0x55730e80abd0 0x55730e80d030 crc :-1 s=READY pgs=73 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:17:29.935 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:29.930+0000 7fdbf3e00640 0 client.25939 ms_handle_reset on v2:172.21.15.189:6838/4074484168 2024-03-24T00:17:29.935 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:29.930+0000 7fdc1bffd740 1 --2- 172.21.15.118:0/3422159689 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55730e7fa9e0 0x55730e806310 unknown :-1 s=CLOSED pgs=285 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:17:29.935 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:29.930+0000 7fdc1bffd740 1 --2- 172.21.15.118:0/3422159689 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55730e7f5870 0x55730e7ff2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:17:29.935 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:29.930+0000 7fdc1bffd740 1 --2- 172.21.15.118:0/3422159689 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55730e7f4e80 0x55730e7fed90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:17:29.935 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:29.930+0000 7fdc1bffd740 1 -- 172.21.15.118:0/3422159689 >> 172.21.15.118:0/3422159689 conn(0x55730e7bd200 msgr2=0x55730e7efa90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:17:29.935 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:29.931+0000 7fdc1bffd740 1 -- 172.21.15.118:0/3422159689 shutdown_connections 2024-03-24T00:17:29.935 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:29.931+0000 7fdc1bffd740 1 -- 172.21.15.118:0/3422159689 wait complete. 2024-03-24T00:17:29.935 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:29.931+0000 7fdc1bffd740 20 client.25939 trim_cache size 0 max 16384 2024-03-24T00:17:29.951 INFO:tasks.workunit.client.0.smithi118.stdout:checking whether pthreads work without any flags... yes 2024-03-24T00:17:29.965 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:29.965+0000 7fdc1bffd740 1 Processor -- start 2024-03-24T00:17:29.966 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:29.965+0000 7fdc1bffd740 1 -- start start 2024-03-24T00:17:29.967 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:29.965+0000 7fdc1bffd740 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55730e7f4e80 0x55730e7f0a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:17:29.967 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:29.965+0000 7fdc1bffd740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55730e7f5870 0x55730e8062d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:17:29.967 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:29.965+0000 7fdc1bffd740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55730e7fa9e0 0x55730e7f6310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:17:29.967 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:29.965+0000 7fdc1bffd740 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdc141aac20 con 0x55730e7f4e80 2024-03-24T00:17:29.968 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:29.965+0000 7fdc1bffd740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdc141ac5a0 con 0x55730e7fa9e0 2024-03-24T00:17:29.968 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:29.965+0000 7fdc1bffd740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdc141ab280 con 0x55730e7f5870 2024-03-24T00:17:29.968 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:29.966+0000 7fdc12000640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55730e7f5870 0x55730e8062d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:17:29.968 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:29.966+0000 7fdc13400640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55730e7fa9e0 0x55730e7f6310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:17:29.968 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:29.966+0000 7fdc12000640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55730e7f5870 0x55730e8062d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.118:38062/0 (socket says 172.21.15.118:38062) 2024-03-24T00:17:29.968 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:29.966+0000 7fdc12000640 1 -- 172.21.15.118:0/2358025450 learned_addr learned my addr 172.21.15.118:0/2358025450 (peer_addr_for_me v2:172.21.15.118:0/0) 2024-03-24T00:17:29.968 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:29.966+0000 7fdc12a00640 1 --2- 172.21.15.118:0/2358025450 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55730e7f4e80 0x55730e7f0a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:17:29.968 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:29.966+0000 7fdc12000640 1 -- 172.21.15.118:0/2358025450 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55730e7fa9e0 msgr2=0x55730e7f6310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:17:29.968 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:29.966+0000 7fdc12000640 1 --2- 172.21.15.118:0/2358025450 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55730e7fa9e0 0x55730e7f6310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:17:29.968 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:29.966+0000 7fdc12000640 1 -- 172.21.15.118:0/2358025450 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55730e7f4e80 msgr2=0x55730e7f0a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:17:29.968 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:29.966+0000 7fdc12000640 1 --2- 172.21.15.118:0/2358025450 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55730e7f4e80 0x55730e7f0a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:17:29.968 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:29.966+0000 7fdc12000640 1 -- 172.21.15.118:0/2358025450 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55730e42e0f0 con 0x55730e7f5870 2024-03-24T00:17:29.968 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:29.966+0000 7fdc12000640 1 --2- 172.21.15.118:0/2358025450 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55730e7f5870 0x55730e8062d0 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7fdc00009df0 tx=0x7fdc00005f90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=839415fe5011a7ee server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:17:29.968 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:29.967+0000 7fdbfea00640 1 -- 172.21.15.118:0/2358025450 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdc00012e40 con 0x55730e7f5870 2024-03-24T00:17:29.969 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:29.967+0000 7fdbfea00640 1 -- 172.21.15.118:0/2358025450 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fdc00021070 con 0x55730e7f5870 2024-03-24T00:17:29.969 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:29.967+0000 7fdbfea00640 1 -- 172.21.15.118:0/2358025450 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdc0001c860 con 0x55730e7f5870 2024-03-24T00:17:29.972 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:29.968+0000 7fdc1bffd740 1 -- 172.21.15.118:0/2358025450 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55730e7f5870 msgr2=0x55730e8062d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:17:29.972 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:29.968+0000 7fdc1bffd740 1 --2- 172.21.15.118:0/2358025450 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55730e7f5870 0x55730e8062d0 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7fdc00009df0 tx=0x7fdc00005f90 comp rx=0 tx=0).stop 2024-03-24T00:17:29.972 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:29.968+0000 7fdc1bffd740 1 -- 172.21.15.118:0/2358025450 shutdown_connections 2024-03-24T00:17:29.972 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:29.968+0000 7fdc1bffd740 1 --2- 172.21.15.118:0/2358025450 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55730e7fa9e0 0x55730e7f6310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:17:29.972 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:29.968+0000 7fdc1bffd740 1 --2- 172.21.15.118:0/2358025450 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55730e7f5870 0x55730e8062d0 unknown :-1 s=CLOSED pgs=286 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:17:29.972 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:29.968+0000 7fdc1bffd740 1 --2- 172.21.15.118:0/2358025450 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55730e7f4e80 0x55730e7f0a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:17:29.972 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:29.968+0000 7fdc1bffd740 1 -- 172.21.15.118:0/2358025450 >> 172.21.15.118:0/2358025450 conn(0x55730e7bd200 msgr2=0x55730e7f0310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:17:29.972 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:29.968+0000 7fdc1bffd740 1 -- 172.21.15.118:0/2358025450 shutdown_connections 2024-03-24T00:17:29.972 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:29.968+0000 7fdc1bffd740 1 -- 172.21.15.118:0/2358025450 wait complete. 2024-03-24T00:17:29.973 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:29.968+0000 7fdc1bffd740 1 -- start start 2024-03-24T00:17:29.973 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:29.969+0000 7fdc1bffd740 1 Processor -- start 2024-03-24T00:17:29.973 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:29.969+0000 7fdc1bffd740 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55730e7f4e80 0x55730e803870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:17:29.973 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:29.969+0000 7fdc1bffd740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55730e7f5870 0x55730e803db0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:17:29.973 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:29.969+0000 7fdc1bffd740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55730e7fa9e0 0x55730e804d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:17:29.973 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:29.969+0000 7fdc1bffd740 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdc141abf40 con 0x55730e7f4e80 2024-03-24T00:17:29.973 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:29.969+0000 7fdc1bffd740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdc141ab8e0 con 0x55730e7fa9e0 2024-03-24T00:17:29.973 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:29.969+0000 7fdc1bffd740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdc141a8de0 con 0x55730e7f5870 2024-03-24T00:17:29.973 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:29.969+0000 7fdc1bffd740 10 client.0 _ll_register_callbacks cb 0x7fdc04005e30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T00:17:29.973 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:29.969+0000 7fdc12000640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55730e7f5870 0x55730e803db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:17:29.974 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:29.969+0000 7fdc12a00640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55730e7f4e80 0x55730e803870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:17:29.974 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:29.969+0000 7fdc13400640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55730e7fa9e0 0x55730e804d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:17:29.974 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:29.969+0000 7fdc12000640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55730e7f5870 0x55730e803db0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.118:38074/0 (socket says 172.21.15.118:38074) 2024-03-24T00:17:29.974 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:29.969+0000 7fdc12000640 1 -- 172.21.15.118:0/3392336263 learned_addr learned my addr 172.21.15.118:0/3392336263 (peer_addr_for_me v2:172.21.15.118:0/0) 2024-03-24T00:17:29.974 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:29.970+0000 7fdc13400640 1 -- 172.21.15.118:0/3392336263 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55730e7f5870 msgr2=0x55730e803db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:17:29.974 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:29.970+0000 7fdc13400640 1 --2- 172.21.15.118:0/3392336263 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55730e7f5870 0x55730e803db0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:17:29.974 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:29.970+0000 7fdc13400640 1 -- 172.21.15.118:0/3392336263 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55730e7f4e80 msgr2=0x55730e803870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:17:29.974 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:29.970+0000 7fdc13400640 1 --2- 172.21.15.118:0/3392336263 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55730e7f4e80 0x55730e803870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:17:29.974 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:29.970+0000 7fdc13400640 1 -- 172.21.15.118:0/3392336263 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55730e805360 con 0x55730e7fa9e0 2024-03-24T00:17:29.974 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:29.970+0000 7fdc13400640 1 --2- 172.21.15.118:0/3392336263 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55730e7fa9e0 0x55730e804d70 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 crypto rx=0x7fdc0c009bf0 tx=0x7fdc0c058970 comp rx=0 tx=0).ready entity=mon.2 client_cookie=5b0b8383b5a1f425 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:17:29.974 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:29.970+0000 7fdbf3e00640 10 client.0 ms_handle_connect on v2:172.21.15.189:3300/0 2024-03-24T00:17:29.974 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:29.970+0000 7fdbf3e00640 1 -- 172.21.15.118:0/3392336263 <== mon.2 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdc0c051040 con 0x55730e7fa9e0 2024-03-24T00:17:29.975 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:29.970+0000 7fdc1bffd740 1 -- 172.21.15.118:0/3392336263 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fdc00021070 con 0x55730e7fa9e0 2024-03-24T00:17:29.975 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:29.970+0000 7fdbf3e00640 1 -- 172.21.15.118:0/3392336263 <== mon.2 v2:172.21.15.189:3300/0 2 ==== osd_map(90..90 src has 1..90) v4 ==== 8277+0+0 (secure 0 0 0) 0x7fdc0c04c470 con 0x55730e7fa9e0 2024-03-24T00:17:29.975 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:29.971+0000 7fdbf3e00640 1 -- 172.21.15.118:0/3392336263 <== mon.2 v2:172.21.15.189:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) v2 ==== 24+0+0 (secure 0 0 0) 0x7fdc0c039040 con 0x55730e7fa9e0 2024-03-24T00:17:29.975 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:29.971+0000 7fdc1bffd740 10 client.25958 fetch_fsmap learned FSMap version 28 2024-03-24T00:17:29.975 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:29.971+0000 7fdc1bffd740 1 -- 172.21.15.118:0/3392336263 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) v3 -- 0x55730e866500 con 0x55730e7fa9e0 2024-03-24T00:17:29.975 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:29.971+0000 7fdbf3e00640 1 -- 172.21.15.118:0/3392336263 <== mon.2 v2:172.21.15.189:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fdc00021070 con 0x55730e7fa9e0 2024-03-24T00:17:29.975 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:29.971+0000 7fdbf3e00640 1 -- 172.21.15.118:0/3392336263 <== mon.2 v2:172.21.15.189:3300/0 5 ==== fsmap(e 28) v1 ==== 7934+0+0 (secure 0 0 0) 0x7fdc0c068080 con 0x55730e7fa9e0 2024-03-24T00:17:29.975 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:29.971+0000 7fdc1bffd740 10 client.25958 fetch_fsmap finished waiting for FSMap version 28 2024-03-24T00:17:29.975 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:29.971+0000 7fdc1bffd740 10 client.25958 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-24T00:17:29.975 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:29.971+0000 7fdc1bffd740 20 client.25958 populate_metadata read hostname 'smithi118' 2024-03-24T00:17:29.975 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:29.971+0000 7fdc1bffd740 1 --2- 172.21.15.118:0/3392336263 >> [v2:172.21.15.189:6838/4074484168,v1:172.21.15.189:6839/4074484168] conn(0x55730e80abd0 0x55730e868cb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:17:29.975 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:29.971+0000 7fdc1bffd740 4 client.25958 mds_command: new command op to 24569 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-24T00:17:29.975 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:29.971+0000 7fdc1bffd740 1 -- 172.21.15.118:0/3392336263 --> [v2:172.21.15.189:6838/4074484168,v1:172.21.15.189:6839/4074484168] -- command(tid 0: {"prefix": "scrub status"}) v1 -- 0x55730e761ae0 con 0x55730e80abd0 2024-03-24T00:17:29.975 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:29.971+0000 7fdc12a00640 1 --2- 172.21.15.118:0/3392336263 >> [v2:172.21.15.189:6838/4074484168,v1:172.21.15.189:6839/4074484168] conn(0x55730e80abd0 0x55730e868cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:17:29.977 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:29.972+0000 7fdc12a00640 1 --2- 172.21.15.118:0/3392336263 >> [v2:172.21.15.189:6838/4074484168,v1:172.21.15.189:6839/4074484168] conn(0x55730e80abd0 0x55730e868cb0 crc :-1 s=READY pgs=74 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=f7dbae9e8fdf65b1 server_cookie=2fcde879af7c13f4 in_seq=0 out_seq=0 2024-03-24T00:17:29.977 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:29.972+0000 7fdbf3e00640 10 client.25958 ms_handle_connect on v2:172.21.15.189:6838/4074484168 2024-03-24T00:17:29.977 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:29.973+0000 7fdbf3e00640 1 -- 172.21.15.118:0/3392336263 <== mds.0 v2:172.21.15.189:6838/4074484168 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+259 (crc 0 0 0) 0x55730e761ae0 con 0x55730e80abd0 2024-03-24T00:17:29.977 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:29.973+0000 7fdbf3e00640 10 client.25958 handle_command_reply: tid=0 2024-03-24T00:17:29.977 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:29.973+0000 7fdc1bffd740 1 client.25958 shutdown 2024-03-24T00:17:29.977 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:29.973+0000 7fdc1bffd740 1 -- 172.21.15.118:0/3392336263 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55730e7fa9e0 msgr2=0x55730e804d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:17:29.977 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:29.973+0000 7fdc1bffd740 1 --2- 172.21.15.118:0/3392336263 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55730e7fa9e0 0x55730e804d70 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 crypto rx=0x7fdc0c009bf0 tx=0x7fdc0c058970 comp rx=0 tx=0).stop 2024-03-24T00:17:29.977 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:29.973+0000 7fdc1bffd740 1 -- 172.21.15.118:0/3392336263 shutdown_connections 2024-03-24T00:17:29.977 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:29.973+0000 7fdc1bffd740 1 --2- 172.21.15.118:0/3392336263 >> [v2:172.21.15.189:6838/4074484168,v1:172.21.15.189:6839/4074484168] conn(0x55730e80abd0 0x55730e868cb0 crc :-1 s=READY pgs=74 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:17:29.977 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:29.973+0000 7fdc1bffd740 1 --2- 172.21.15.118:0/3392336263 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55730e7fa9e0 0x55730e804d70 unknown :-1 s=CLOSED pgs=303 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:17:29.977 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:29.973+0000 7fdc1bffd740 1 --2- 172.21.15.118:0/3392336263 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55730e7f5870 0x55730e803db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:17:29.978 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:29.973+0000 7fdbf3e00640 0 client.25958 ms_handle_reset on v2:172.21.15.189:6838/4074484168 2024-03-24T00:17:29.978 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:29.973+0000 7fdc1bffd740 1 --2- 172.21.15.118:0/3392336263 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55730e7f4e80 0x55730e803870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:17:29.978 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:29.973+0000 7fdc1bffd740 1 -- 172.21.15.118:0/3392336263 >> 172.21.15.118:0/3392336263 conn(0x55730e7bd200 msgr2=0x55730e7f6310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:17:29.978 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:29.973+0000 7fdc1bffd740 1 -- 172.21.15.118:0/3392336263 shutdown_connections 2024-03-24T00:17:29.978 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:29.973+0000 7fdc1bffd740 1 -- 172.21.15.118:0/3392336263 wait complete. 2024-03-24T00:17:29.978 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:29.973+0000 7fdc1bffd740 20 client.25958 trim_cache size 0 max 16384 2024-03-24T00:17:29.978 INFO:teuthology.orchestra.run.smithi118.stdout:{ 2024-03-24T00:17:29.978 INFO:teuthology.orchestra.run.smithi118.stdout: "status": "IDLE (waiting for more scrubs)", 2024-03-24T00:17:29.978 INFO:teuthology.orchestra.run.smithi118.stdout: "scrubs": { 2024-03-24T00:17:29.978 INFO:teuthology.orchestra.run.smithi118.stdout: "961ace21-39d6-4b4e-a091-b12c4f1cfeaa": { 2024-03-24T00:17:29.979 INFO:teuthology.orchestra.run.smithi118.stdout: "path": "/", 2024-03-24T00:17:29.979 INFO:teuthology.orchestra.run.smithi118.stdout: "tag": "961ace21-39d6-4b4e-a091-b12c4f1cfeaa", 2024-03-24T00:17:29.979 INFO:teuthology.orchestra.run.smithi118.stdout: "options": "recursive,force" 2024-03-24T00:17:29.979 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-03-24T00:17:29.979 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-03-24T00:17:29.979 INFO:teuthology.orchestra.run.smithi118.stdout:} 2024-03-24T00:17:29.980 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:29.974+0000 7fdc19600640 1 -- 172.21.15.118:0/2226737263 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7fdbf80a2880 msgr2=0x7fdbf80a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:17:29.980 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:29.974+0000 7fdc19600640 1 --2- 172.21.15.118:0/2226737263 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7fdbf80a2880 0x7fdbf80a4d40 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 crypto rx=0x7fdc0c02c640 tx=0x7fdc0c02e040 comp rx=0 tx=0).stop 2024-03-24T00:17:29.980 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:29.974+0000 7fdc19600640 1 -- 172.21.15.118:0/2226737263 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fdc141038e0 msgr2=0x7fdc14199b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:17:29.980 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:29.974+0000 7fdc19600640 1 --2- 172.21.15.118:0/2226737263 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fdc141038e0 0x7fdc14199b50 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 crypto rx=0x7fdc0400e9c0 tx=0x7fdc0400ee80 comp rx=0 tx=0).stop 2024-03-24T00:17:29.980 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:29.974+0000 7fdc19600640 1 -- 172.21.15.118:0/2226737263 shutdown_connections 2024-03-24T00:17:29.980 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:29.974+0000 7fdc19600640 1 --2- 172.21.15.118:0/2226737263 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7fdbf80a2880 0x7fdbf80a4d40 unknown :-1 s=CLOSED pgs=304 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:17:29.980 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:29.974+0000 7fdc19600640 1 --2- 172.21.15.118:0/2226737263 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fdc14105490 0x7fdc141a1100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:17:29.980 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:29.974+0000 7fdc19600640 1 --2- 172.21.15.118:0/2226737263 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fdc14104ad0 0x7fdc1419a090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:17:29.980 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:29.974+0000 7fdc19600640 1 --2- 172.21.15.118:0/2226737263 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fdc141038e0 0x7fdc14199b50 unknown :-1 s=CLOSED pgs=301 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:17:29.980 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:29.974+0000 7fdc19600640 1 -- 172.21.15.118:0/2226737263 >> 172.21.15.118:0/2226737263 conn(0x7fdc140fee50 msgr2=0x7fdc14107d70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:17:29.980 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:29.974+0000 7fdc19600640 1 -- 172.21.15.118:0/2226737263 shutdown_connections 2024-03-24T00:17:29.981 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:17:29.974+0000 7fdc19600640 1 -- 172.21.15.118:0/2226737263 wait complete. 2024-03-24T00:17:29.993 INFO:tasks.cephfs.filesystem:scrub status for tag:961ace21-39d6-4b4e-a091-b12c4f1cfeaa - {'path': '/', 'tag': '961ace21-39d6-4b4e-a091-b12c4f1cfeaa', 'options': 'recursive,force'} 2024-03-24T00:17:30.078 INFO:tasks.workunit.client.0.smithi118.stdout:checking for joinable pthread attribute... PTHREAD_CREATE_JOINABLE 2024-03-24T00:17:30.079 INFO:tasks.workunit.client.0.smithi118.stdout:checking if more special flags are required for pthreads... no 2024-03-24T00:17:30.079 INFO:tasks.workunit.client.0.smithi118.stdout:checking for cc_r... gcc 2024-03-24T00:17:30.250 INFO:tasks.workunit.client.0.smithi118.stdout:checking for setrlimit... yes 2024-03-24T00:17:30.422 INFO:tasks.workunit.client.0.smithi118.stdout:checking for setproctitle... no 2024-03-24T00:17:30.572 INFO:tasks.workunit.client.0.smithi118.stdout:checking for getopt_long... yes 2024-03-24T00:17:30.738 INFO:tasks.workunit.client.0.smithi118.stdout:checking for strtoull... yes 2024-03-24T00:17:30.904 INFO:tasks.workunit.client.0.smithi118.stdout:checking for strtoq... yes 2024-03-24T00:17:31.091 INFO:tasks.workunit.client.0.smithi118.stdout:checking for snprintf... yes 2024-03-24T00:17:31.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:17:30 smithi118 ceph-mon[31535]: scrub queued for path: / 2024-03-24T00:17:31.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:17:30 smithi118 ceph-mon[31535]: scrub summary: idle+waiting paths [/] 2024-03-24T00:17:31.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:17:30 smithi118 ceph-mon[31535]: scrub summary: active paths [/] 2024-03-24T00:17:31.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:17:30 smithi118 ceph-mon[31535]: pgmap v452: 97 pgs: 97 active+clean; 3.2 MiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 152 KiB/s rd, 278 KiB/s wr, 84 op/s 2024-03-24T00:17:31.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:17:30 smithi118 ceph-mon[31535]: scrub summary: idle+waiting paths [/] 2024-03-24T00:17:31.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:17:30 smithi118 ceph-mon[31535]: scrub summary: idle 2024-03-24T00:17:31.203 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:17:30 smithi189 ceph-mon[36187]: scrub queued for path: / 2024-03-24T00:17:31.203 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:17:30 smithi189 ceph-mon[36187]: scrub summary: idle+waiting paths [/] 2024-03-24T00:17:31.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:17:30 smithi189 ceph-mon[36187]: scrub summary: active paths [/] 2024-03-24T00:17:31.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:17:30 smithi189 ceph-mon[36187]: pgmap v452: 97 pgs: 97 active+clean; 3.2 MiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 152 KiB/s rd, 278 KiB/s wr, 84 op/s 2024-03-24T00:17:31.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:17:30 smithi189 ceph-mon[36187]: scrub summary: idle+waiting paths [/] 2024-03-24T00:17:31.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:17:30 smithi189 ceph-mon[36187]: scrub summary: idle 2024-03-24T00:17:31.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:17:30 smithi193 ceph-mon[36878]: scrub queued for path: / 2024-03-24T00:17:31.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:17:30 smithi193 ceph-mon[36878]: scrub summary: idle+waiting paths [/] 2024-03-24T00:17:31.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:17:30 smithi193 ceph-mon[36878]: scrub summary: active paths [/] 2024-03-24T00:17:31.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:17:30 smithi193 ceph-mon[36878]: pgmap v452: 97 pgs: 97 active+clean; 3.2 MiB data, 367 MiB used, 1.0 TiB / 1.0 TiB avail; 152 KiB/s rd, 278 KiB/s wr, 84 op/s 2024-03-24T00:17:31.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:17:30 smithi193 ceph-mon[36878]: scrub summary: idle+waiting paths [/] 2024-03-24T00:17:31.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:17:30 smithi193 ceph-mon[36878]: scrub summary: idle 2024-03-24T00:17:31.254 INFO:tasks.workunit.client.0.smithi118.stdout:checking for vsnprintf... yes 2024-03-24T00:17:31.431 INFO:tasks.workunit.client.0.smithi118.stdout:checking for vfprintf... yes 2024-03-24T00:17:31.603 INFO:tasks.workunit.client.0.smithi118.stdout:checking for gethostname... yes 2024-03-24T00:17:31.766 INFO:tasks.workunit.client.0.smithi118.stdout:checking for setlocale... yes 2024-03-24T00:17:31.922 INFO:tasks.workunit.client.0.smithi118.stdout:checking for sync... yes 2024-03-24T00:17:32.067 INFO:tasks.workunit.client.0.smithi118.stdout:checking whether snprintf is C99 conformant... done 2024-03-24T00:17:32.101 INFO:tasks.workunit.client.0.smithi118.stdout:configure: creating ./config.status 2024-03-24T00:17:32.577 INFO:tasks.workunit.client.0.smithi118.stdout:config.status: creating Makefile 2024-03-24T00:17:32.629 INFO:tasks.workunit.client.0.smithi118.stdout:config.status: creating src/Makefile 2024-03-24T00:17:32.685 INFO:tasks.workunit.client.0.smithi118.stdout:config.status: creating man/Makefile 2024-03-24T00:17:32.712 INFO:tasks.workunit.client.0.smithi118.stdout:config.status: creating config.h 2024-03-24T00:17:32.873 INFO:tasks.workunit.client.0.smithi118.stdout:config.status: executing depfiles commands 2024-03-24T00:17:33.195 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:17:32 smithi118 ceph-mon[31535]: pgmap v453: 97 pgs: 97 active+clean; 3.3 MiB data, 368 MiB used, 1.0 TiB / 1.0 TiB avail; 152 KiB/s rd, 291 KiB/s wr, 86 op/s 2024-03-24T00:17:33.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:17:32 smithi189 ceph-mon[36187]: pgmap v453: 97 pgs: 97 active+clean; 3.3 MiB data, 368 MiB used, 1.0 TiB / 1.0 TiB avail; 152 KiB/s rd, 291 KiB/s wr, 86 op/s 2024-03-24T00:17:33.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:17:32 smithi193 ceph-mon[36878]: pgmap v453: 97 pgs: 97 active+clean; 3.3 MiB data, 368 MiB used, 1.0 TiB / 1.0 TiB avail; 152 KiB/s rd, 291 KiB/s wr, 86 op/s 2024-03-24T00:17:33.313 INFO:tasks.workunit.client.0.smithi118.stderr:+ make 2024-03-24T00:17:33.328 INFO:tasks.workunit.client.0.smithi118.stdout:make all-recursive 2024-03-24T00:17:33.333 INFO:tasks.workunit.client.0.smithi118.stdout:make[1]: Entering directory '/home/ubuntu/cephtest/mnt.0/client.0/tmp/blogbench-1.0' 2024-03-24T00:17:33.341 INFO:tasks.workunit.client.0.smithi118.stdout:Making all in src 2024-03-24T00:17:33.354 INFO:tasks.workunit.client.0.smithi118.stdout:make[2]: Entering directory '/home/ubuntu/cephtest/mnt.0/client.0/tmp/blogbench-1.0/src' 2024-03-24T00:17:33.354 INFO:tasks.workunit.client.0.smithi118.stdout:if gcc -DHAVE_CONFIG_H -I. -I. -I.. -g -O2 -MT blogbench.o -MD -MP -MF ".deps/blogbench.Tpo" -c -o blogbench.o blogbench.c; \ 2024-03-24T00:17:33.354 INFO:tasks.workunit.client.0.smithi118.stdout:then mv -f ".deps/blogbench.Tpo" ".deps/blogbench.Po"; else rm -f ".deps/blogbench.Tpo"; exit 1; fi 2024-03-24T00:17:33.524 INFO:tasks.workunit.client.0.smithi118.stdout:if gcc -DHAVE_CONFIG_H -I. -I. -I.. -g -O2 -MT process.o -MD -MP -MF ".deps/process.Tpo" -c -o process.o process.c; \ 2024-03-24T00:17:33.524 INFO:tasks.workunit.client.0.smithi118.stdout:then mv -f ".deps/process.Tpo" ".deps/process.Po"; else rm -f ".deps/process.Tpo"; exit 1; fi 2024-03-24T00:17:33.667 INFO:tasks.workunit.client.0.smithi118.stdout:if gcc -DHAVE_CONFIG_H -I. -I. -I.. -g -O2 -MT writer.o -MD -MP -MF ".deps/writer.Tpo" -c -o writer.o writer.c; \ 2024-03-24T00:17:33.667 INFO:tasks.workunit.client.0.smithi118.stdout:then mv -f ".deps/writer.Tpo" ".deps/writer.Po"; else rm -f ".deps/writer.Tpo"; exit 1; fi 2024-03-24T00:17:33.757 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:17:33.756+0000 7fb65e000640 1 -- 172.21.15.118:0/2150258042 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb658074100 msgr2=0x7fb658074580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:17:33.757 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:17:33.756+0000 7fb65e000640 1 --2- 172.21.15.118:0/2150258042 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb658074100 0x7fb658074580 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 crypto rx=0x7fb644009930 tx=0x7fb64402bc30 comp rx=0 tx=0).stop 2024-03-24T00:17:33.757 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:17:33.756+0000 7fb65e000640 1 -- 172.21.15.118:0/2150258042 shutdown_connections 2024-03-24T00:17:33.757 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:17:33.756+0000 7fb65e000640 1 --2- 172.21.15.118:0/2150258042 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb65810dae0 0x7fb65810fed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:17:33.757 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:17:33.756+0000 7fb65e000640 1 --2- 172.21.15.118:0/2150258042 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb658074100 0x7fb658074580 unknown :-1 s=CLOSED pgs=304 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:17:33.757 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:17:33.756+0000 7fb65e000640 1 --2- 172.21.15.118:0/2150258042 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fb6580756e0 0x7fb658073b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:17:33.757 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:17:33.756+0000 7fb65e000640 1 -- 172.21.15.118:0/2150258042 >> 172.21.15.118:0/2150258042 conn(0x7fb6580fca80 msgr2=0x7fb6580feec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:17:33.757 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:17:33.756+0000 7fb65e000640 1 -- 172.21.15.118:0/2150258042 shutdown_connections 2024-03-24T00:17:33.757 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:17:33.756+0000 7fb65e000640 1 -- 172.21.15.118:0/2150258042 wait complete. 2024-03-24T00:17:33.759 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:17:33.757+0000 7fb65e000640 1 Processor -- start 2024-03-24T00:17:33.760 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:17:33.757+0000 7fb65e000640 1 -- start start 2024-03-24T00:17:33.760 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:17:33.757+0000 7fb65e000640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb658074100 0x7fb65819de20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:17:33.760 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:17:33.757+0000 7fb65e000640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fb6580756e0 0x7fb65819e360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:17:33.760 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:17:33.758+0000 7fb65e000640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb65810dae0 0x7fb6581a53d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:17:33.760 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:17:33.758+0000 7fb65e000640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb6581137d0 con 0x7fb658074100 2024-03-24T00:17:33.760 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:17:33.758+0000 7fb65e000640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb658113650 con 0x7fb65810dae0 2024-03-24T00:17:33.760 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:17:33.758+0000 7fb65e000640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb658113950 con 0x7fb6580756e0 2024-03-24T00:17:33.760 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:17:33.758+0000 7fb657e00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fb6580756e0 0x7fb65819e360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:17:33.760 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:17:33.758+0000 7fb65cc00640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb658074100 0x7fb65819de20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:17:33.760 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:17:33.758+0000 7fb65d600640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb65810dae0 0x7fb6581a53d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:17:33.760 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:17:33.758+0000 7fb657e00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fb6580756e0 0x7fb65819e360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.118:42348/0 (socket says 172.21.15.118:42348) 2024-03-24T00:17:33.760 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:17:33.758+0000 7fb657e00640 1 -- 172.21.15.118:0/2349861125 learned_addr learned my addr 172.21.15.118:0/2349861125 (peer_addr_for_me v2:172.21.15.118:0/0) 2024-03-24T00:17:33.760 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:17:33.758+0000 7fb657e00640 1 -- 172.21.15.118:0/2349861125 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb65810dae0 msgr2=0x7fb6581a53d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:17:33.760 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:17:33.758+0000 7fb657e00640 1 --2- 172.21.15.118:0/2349861125 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb65810dae0 0x7fb6581a53d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:17:33.761 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:17:33.758+0000 7fb657e00640 1 -- 172.21.15.118:0/2349861125 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb658074100 msgr2=0x7fb65819de20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:17:33.761 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:17:33.758+0000 7fb657e00640 1 --2- 172.21.15.118:0/2349861125 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb658074100 0x7fb65819de20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:17:33.761 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:17:33.758+0000 7fb657e00640 1 -- 172.21.15.118:0/2349861125 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb6581a5a40 con 0x7fb6580756e0 2024-03-24T00:17:33.761 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:17:33.759+0000 7fb657e00640 1 --2- 172.21.15.118:0/2349861125 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fb6580756e0 0x7fb65819e360 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x7fb644000c00 tx=0x7fb64402c6c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:17:33.761 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:17:33.759+0000 7fb655600640 1 -- 172.21.15.118:0/2349861125 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb644043020 con 0x7fb6580756e0 2024-03-24T00:17:33.761 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:17:33.759+0000 7fb655600640 1 -- 172.21.15.118:0/2349861125 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fb6440054e0 con 0x7fb6580756e0 2024-03-24T00:17:33.761 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:17:33.759+0000 7fb65e000640 1 -- 172.21.15.118:0/2349861125 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb6581a5cd0 con 0x7fb6580756e0 2024-03-24T00:17:33.761 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:17:33.759+0000 7fb655600640 1 -- 172.21.15.118:0/2349861125 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb644003ce0 con 0x7fb6580756e0 2024-03-24T00:17:33.761 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:17:33.759+0000 7fb65e000640 1 -- 172.21.15.118:0/2349861125 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb6581a61e0 con 0x7fb6580756e0 2024-03-24T00:17:33.762 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:17:33.762+0000 7fb655600640 1 -- 172.21.15.118:0/2349861125 <== mon.1 v2:172.21.15.193:3300/0 4 ==== mgrmap(e 20) v1 ==== 137291+0+0 (secure 0 0 0) 0x7fb644046020 con 0x7fb6580756e0 2024-03-24T00:17:33.763 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:17:33.762+0000 7fb662363740 1 Processor -- start 2024-03-24T00:17:33.763 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:17:33.762+0000 7fb662363740 1 -- start start 2024-03-24T00:17:33.764 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:17:33.763+0000 7fb662363740 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x564b5b2613f0 0x564b5b261810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:17:33.764 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:17:33.763+0000 7fb662363740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x564b5b261de0 0x564b5b2668e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:17:33.764 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:17:33.763+0000 7fb662363740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x564b5b266f50 0x564b5b2693e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:17:33.764 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:17:33.763+0000 7fb662363740 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x564b5b2243b0 con 0x564b5b2613f0 2024-03-24T00:17:33.764 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:17:33.763+0000 7fb662363740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x564b5b1c6830 con 0x564b5b261de0 2024-03-24T00:17:33.764 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:17:33.763+0000 7fb662363740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x564b5b16f5d0 con 0x564b5b266f50 2024-03-24T00:17:33.764 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:17:33.763+0000 7fb65cc00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x564b5b261de0 0x564b5b2668e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:17:33.765 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:17:33.763+0000 7fb65cc00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x564b5b261de0 0x564b5b2668e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.118:54214/0 (socket says 172.21.15.118:54214) 2024-03-24T00:17:33.765 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:17:33.763+0000 7fb65cc00640 1 -- 172.21.15.118:0/523658553 learned_addr learned my addr 172.21.15.118:0/523658553 (peer_addr_for_me v2:172.21.15.118:0/0) 2024-03-24T00:17:33.765 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:17:33.763+0000 7fb65d600640 1 --2- 172.21.15.118:0/523658553 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x564b5b2613f0 0x564b5b261810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:17:33.765 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:17:33.763+0000 7fb657e00640 1 --2- 172.21.15.118:0/523658553 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x564b5b266f50 0x564b5b2693e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:17:33.765 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:17:33.764+0000 7fb65d600640 1 -- 172.21.15.118:0/523658553 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x564b5b266f50 msgr2=0x564b5b2693e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:17:33.765 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:17:33.764+0000 7fb65d600640 1 --2- 172.21.15.118:0/523658553 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x564b5b266f50 0x564b5b2693e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:17:33.765 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:17:33.764+0000 7fb65d600640 1 -- 172.21.15.118:0/523658553 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x564b5b261de0 msgr2=0x564b5b2668e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:17:33.766 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:17:33.764+0000 7fb65d600640 1 --2- 172.21.15.118:0/523658553 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x564b5b261de0 0x564b5b2668e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:17:33.766 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:17:33.764+0000 7fb65d600640 1 -- 172.21.15.118:0/523658553 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x564b5b1872c0 con 0x564b5b2613f0 2024-03-24T00:17:33.767 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:17:33.764+0000 7fb655600640 1 --2- 172.21.15.118:0/2349861125 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7fb6400a2870 0x7fb6400a4d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:17:33.767 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:17:33.764+0000 7fb65cc00640 1 --2- 172.21.15.118:0/523658553 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x564b5b261de0 0x564b5b2668e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T00:17:33.767 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:17:33.764+0000 7fb655600640 1 -- 172.21.15.118:0/2349861125 <== mon.1 v2:172.21.15.193:3300/0 5 ==== osd_map(90..90 src has 1..90) v4 ==== 8277+0+0 (secure 0 0 0) 0x7fb6440f0b40 con 0x7fb6580756e0 2024-03-24T00:17:33.767 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:17:33.764+0000 7fb65d600640 1 --2- 172.21.15.118:0/523658553 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x564b5b2613f0 0x564b5b261810 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7fb650009f30 tx=0x7fb650010800 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2c45c229fe864eda server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:17:33.767 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:17:33.764+0000 7fb65cc00640 1 --2- 172.21.15.118:0/2349861125 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7fb6400a2870 0x7fb6400a4d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:17:33.767 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:17:33.765+0000 7fb649c00640 1 -- 172.21.15.118:0/523658553 <== mon.0 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb650016070 con 0x564b5b2613f0 2024-03-24T00:17:33.767 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:17:33.765+0000 7fb649c00640 1 -- 172.21.15.118:0/523658553 <== mon.0 v2:172.21.15.118:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fb650004d20 con 0x564b5b2613f0 2024-03-24T00:17:33.767 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:17:33.765+0000 7fb649c00640 1 -- 172.21.15.118:0/523658553 <== mon.0 v2:172.21.15.118:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb650012870 con 0x564b5b2613f0 2024-03-24T00:17:33.767 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:17:33.766+0000 7fb662363740 1 -- 172.21.15.118:0/523658553 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x564b5b2613f0 msgr2=0x564b5b261810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:17:33.767 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:17:33.766+0000 7fb65cc00640 1 --2- 172.21.15.118:0/2349861125 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7fb6400a2870 0x7fb6400a4d30 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 crypto rx=0x564b5b2621e0 tx=0x7fb64c005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:17:33.767 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:17:33.766+0000 7fb662363740 1 --2- 172.21.15.118:0/523658553 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x564b5b2613f0 0x564b5b261810 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7fb650009f30 tx=0x7fb650010800 comp rx=0 tx=0).stop 2024-03-24T00:17:33.767 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:17:33.766+0000 7fb662363740 1 -- 172.21.15.118:0/523658553 shutdown_connections 2024-03-24T00:17:33.767 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:17:33.766+0000 7fb662363740 1 --2- 172.21.15.118:0/523658553 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x564b5b266f50 0x564b5b2693e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:17:33.768 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:17:33.766+0000 7fb662363740 1 --2- 172.21.15.118:0/523658553 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x564b5b261de0 0x564b5b2668e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:17:33.768 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:17:33.766+0000 7fb662363740 1 --2- 172.21.15.118:0/523658553 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x564b5b2613f0 0x564b5b261810 unknown :-1 s=CLOSED pgs=195 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:17:33.768 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:17:33.766+0000 7fb662363740 1 -- 172.21.15.118:0/523658553 >> 172.21.15.118:0/523658553 conn(0x564b5b25c960 msgr2=0x564b5b25cd60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:17:33.768 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:17:33.766+0000 7fb662363740 1 -- 172.21.15.118:0/523658553 shutdown_connections 2024-03-24T00:17:33.768 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:17:33.766+0000 7fb662363740 1 -- 172.21.15.118:0/523658553 wait complete. 2024-03-24T00:17:33.772 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:17:33.766+0000 7fb662363740 1 -- start start 2024-03-24T00:17:33.772 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:17:33.767+0000 7fb662363740 1 Processor -- start 2024-03-24T00:17:33.772 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:17:33.767+0000 7fb662363740 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x564b5b2613f0 0x564b5b26b5e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:17:33.772 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:17:33.767+0000 7fb662363740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x564b5b261de0 0x564b5b26bb20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:17:33.772 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:17:33.767+0000 7fb662363740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x564b5b266f50 0x564b5b272b60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:17:33.772 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:17:33.767+0000 7fb662363740 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x564b5b182a20 con 0x564b5b2613f0 2024-03-24T00:17:33.772 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:17:33.767+0000 7fb662363740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x564b5b1c28d0 con 0x564b5b261de0 2024-03-24T00:17:33.772 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:17:33.767+0000 7fb662363740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x564b5b269f00 con 0x564b5b266f50 2024-03-24T00:17:33.772 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:17:33.767+0000 7fb662363740 10 client.0 _ll_register_callbacks cb 0x564b5b1bf1c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T00:17:33.772 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:17:33.767+0000 7fb65d600640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x564b5b266f50 0x564b5b272b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:17:33.772 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:17:33.767+0000 7fb657e00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x564b5b261de0 0x564b5b26bb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:17:33.773 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:17:33.767+0000 7fb65d600640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x564b5b266f50 0x564b5b272b60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.118:42360/0 (socket says 172.21.15.118:42360) 2024-03-24T00:17:33.773 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:17:33.767+0000 7fb65d600640 1 -- 172.21.15.118:0/928702014 learned_addr learned my addr 172.21.15.118:0/928702014 (peer_addr_for_me v2:172.21.15.118:0/0) 2024-03-24T00:17:33.773 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:17:33.767+0000 7fb65cc00640 1 --2- 172.21.15.118:0/928702014 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x564b5b2613f0 0x564b5b26b5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:17:33.773 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:17:33.768+0000 7fb657e00640 1 -- 172.21.15.118:0/928702014 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x564b5b266f50 msgr2=0x564b5b272b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:17:33.773 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:17:33.768+0000 7fb657e00640 1 --2- 172.21.15.118:0/928702014 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x564b5b266f50 0x564b5b272b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:17:33.773 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:17:33.768+0000 7fb657e00640 1 -- 172.21.15.118:0/928702014 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x564b5b2613f0 msgr2=0x564b5b26b5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:17:33.773 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:17:33.768+0000 7fb657e00640 1 --2- 172.21.15.118:0/928702014 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x564b5b2613f0 0x564b5b26b5e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:17:33.773 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:17:33.768+0000 7fb657e00640 1 -- 172.21.15.118:0/928702014 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x564b5aedb210 con 0x564b5b261de0 2024-03-24T00:17:33.773 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:17:33.768+0000 7fb65d600640 1 --2- 172.21.15.118:0/928702014 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x564b5b266f50 0x564b5b272b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T00:17:33.773 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:17:33.768+0000 7fb657e00640 1 --2- 172.21.15.118:0/928702014 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x564b5b261de0 0x564b5b26bb20 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 crypto rx=0x7fb6440ee5b0 tx=0x7fb64402c4a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=2dfc513d68836bc server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:17:33.773 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:17:33.768+0000 7fb63e000640 10 client.0 ms_handle_connect on v2:172.21.15.189:3300/0 2024-03-24T00:17:33.773 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:17:33.768+0000 7fb63e000640 1 -- 172.21.15.118:0/928702014 <== mon.2 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb6440b96f0 con 0x564b5b261de0 2024-03-24T00:17:33.773 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:17:33.768+0000 7fb63e000640 1 -- 172.21.15.118:0/928702014 <== mon.2 v2:172.21.15.189:3300/0 2 ==== osd_map(90..90 src has 1..90) v4 ==== 8277+0+0 (secure 0 0 0) 0x7fb6440ee6c0 con 0x564b5b261de0 2024-03-24T00:17:33.773 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:17:33.768+0000 7fb662363740 1 -- 172.21.15.118:0/928702014 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fb650004d20 con 0x564b5b261de0 2024-03-24T00:17:33.774 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:17:33.769+0000 7fb63e000640 1 -- 172.21.15.118:0/928702014 <== mon.2 v2:172.21.15.189:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) v2 ==== 24+0+0 (secure 0 0 0) 0x7fb6440bcdb0 con 0x564b5b261de0 2024-03-24T00:17:33.774 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:17:33.769+0000 7fb662363740 10 client.25970 fetch_fsmap learned FSMap version 28 2024-03-24T00:17:33.774 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:17:33.769+0000 7fb662363740 1 -- 172.21.15.118:0/928702014 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) v3 -- 0x564b5afa7620 con 0x564b5b261de0 2024-03-24T00:17:33.774 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:17:33.769+0000 7fb63e000640 1 -- 172.21.15.118:0/928702014 <== mon.2 v2:172.21.15.189:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fb650004d20 con 0x564b5b261de0 2024-03-24T00:17:33.774 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:17:33.769+0000 7fb63e000640 1 -- 172.21.15.118:0/928702014 <== mon.2 v2:172.21.15.189:3300/0 5 ==== fsmap(e 28) v1 ==== 7934+0+0 (secure 0 0 0) 0x7fb6440c1150 con 0x564b5b261de0 2024-03-24T00:17:33.774 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:17:33.769+0000 7fb662363740 10 client.25970 fetch_fsmap finished waiting for FSMap version 28 2024-03-24T00:17:33.774 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:17:33.769+0000 7fb662363740 10 client.25970 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-24T00:17:33.774 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:17:33.769+0000 7fb662363740 20 client.25970 populate_metadata read hostname 'smithi118' 2024-03-24T00:17:33.774 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:17:33.769+0000 7fb662363740 1 --2- 172.21.15.118:0/928702014 >> [v2:172.21.15.189:6838/4074484168,v1:172.21.15.189:6839/4074484168] conn(0x564b5b277480 0x564b5b279870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:17:33.774 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:17:33.769+0000 7fb662363740 4 client.25970 mds_command: new command op to 24569 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-24T00:17:33.774 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:17:33.769+0000 7fb662363740 1 -- 172.21.15.118:0/928702014 --> [v2:172.21.15.189:6838/4074484168,v1:172.21.15.189:6839/4074484168] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x564b5b1d3d70 con 0x564b5b277480 2024-03-24T00:17:33.774 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:17:33.770+0000 7fb65cc00640 1 --2- 172.21.15.118:0/928702014 >> [v2:172.21.15.189:6838/4074484168,v1:172.21.15.189:6839/4074484168] conn(0x564b5b277480 0x564b5b279870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:17:33.774 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:17:33.770+0000 7fb65cc00640 1 --2- 172.21.15.118:0/928702014 >> [v2:172.21.15.189:6838/4074484168,v1:172.21.15.189:6839/4074484168] conn(0x564b5b277480 0x564b5b279870 crc :-1 s=READY pgs=75 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=1417cad7fbad80b3 server_cookie=e16ce51be2bb05d0 in_seq=0 out_seq=0 2024-03-24T00:17:33.775 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:17:33.770+0000 7fb63e000640 10 client.25970 ms_handle_connect on v2:172.21.15.189:6838/4074484168 2024-03-24T00:17:33.775 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:17:33.771+0000 7fb63e000640 1 -- 172.21.15.118:0/928702014 <== mds.0 v2:172.21.15.189:6838/4074484168 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x564b5b1d3d70 con 0x564b5b277480 2024-03-24T00:17:33.775 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:17:33.771+0000 7fb63e000640 10 client.25970 handle_command_reply: tid=0 2024-03-24T00:17:33.775 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:17:33.771+0000 7fb662363740 1 client.25970 shutdown 2024-03-24T00:17:33.775 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:17:33.771+0000 7fb662363740 1 -- 172.21.15.118:0/928702014 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x564b5b261de0 msgr2=0x564b5b26bb20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:17:33.775 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:17:33.771+0000 7fb662363740 1 --2- 172.21.15.118:0/928702014 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x564b5b261de0 0x564b5b26bb20 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 crypto rx=0x7fb6440ee5b0 tx=0x7fb64402c4a0 comp rx=0 tx=0).stop 2024-03-24T00:17:33.775 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:17:33.771+0000 7fb662363740 1 -- 172.21.15.118:0/928702014 shutdown_connections 2024-03-24T00:17:33.775 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:17:33.771+0000 7fb662363740 1 --2- 172.21.15.118:0/928702014 >> [v2:172.21.15.189:6838/4074484168,v1:172.21.15.189:6839/4074484168] conn(0x564b5b277480 0x564b5b279870 crc :-1 s=READY pgs=75 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:17:33.775 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:17:33.771+0000 7fb662363740 1 --2- 172.21.15.118:0/928702014 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x564b5b266f50 0x564b5b272b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:17:33.775 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:17:33.771+0000 7fb662363740 1 --2- 172.21.15.118:0/928702014 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x564b5b261de0 0x564b5b26bb20 unknown :-1 s=CLOSED pgs=305 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:17:33.775 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:17:33.771+0000 7fb63e000640 0 client.25970 ms_handle_reset on v2:172.21.15.189:6838/4074484168 2024-03-24T00:17:33.775 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:17:33.771+0000 7fb662363740 1 --2- 172.21.15.118:0/928702014 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x564b5b2613f0 0x564b5b26b5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:17:33.775 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:17:33.771+0000 7fb662363740 1 -- 172.21.15.118:0/928702014 >> 172.21.15.118:0/928702014 conn(0x564b5b25c960 msgr2=0x564b5b25bb70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:17:33.775 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:17:33.771+0000 7fb662363740 1 -- 172.21.15.118:0/928702014 shutdown_connections 2024-03-24T00:17:33.776 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:17:33.771+0000 7fb662363740 1 -- 172.21.15.118:0/928702014 wait complete. 2024-03-24T00:17:33.776 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:17:33.771+0000 7fb662363740 20 client.25970 trim_cache size 0 max 16384 2024-03-24T00:17:33.789 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:17:33.789+0000 7fb662363740 1 Processor -- start 2024-03-24T00:17:33.789 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:17:33.789+0000 7fb662363740 1 -- start start 2024-03-24T00:17:33.790 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:17:33.789+0000 7fb662363740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x564b5b2613f0 0x564b5b25b2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:17:33.790 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:17:33.789+0000 7fb662363740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x564b5b261de0 0x564b5b272b20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:17:33.790 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:17:33.789+0000 7fb662363740 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x564b5b266f50 0x564b5b25b820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:17:33.790 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:17:33.789+0000 7fb662363740 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb6581aede0 con 0x564b5b266f50 2024-03-24T00:17:33.790 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:17:33.789+0000 7fb662363740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb6581b0760 con 0x564b5b2613f0 2024-03-24T00:17:33.790 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:17:33.789+0000 7fb662363740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb6581af440 con 0x564b5b261de0 2024-03-24T00:17:33.791 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:17:33.789+0000 7fb65d600640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x564b5b266f50 0x564b5b25b820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:17:33.791 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:17:33.789+0000 7fb65d600640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x564b5b266f50 0x564b5b25b820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.118:3300/0 says I am v2:172.21.15.118:39590/0 (socket says 172.21.15.118:39590) 2024-03-24T00:17:33.791 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:17:33.789+0000 7fb65d600640 1 -- 172.21.15.118:0/2673252706 learned_addr learned my addr 172.21.15.118:0/2673252706 (peer_addr_for_me v2:172.21.15.118:0/0) 2024-03-24T00:17:33.791 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:17:33.790+0000 7fb65cc00640 1 --2- 172.21.15.118:0/2673252706 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x564b5b2613f0 0x564b5b25b2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:17:33.791 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:17:33.790+0000 7fb657e00640 1 --2- 172.21.15.118:0/2673252706 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x564b5b261de0 0x564b5b272b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:17:33.791 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:17:33.790+0000 7fb65cc00640 1 -- 172.21.15.118:0/2673252706 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x564b5b261de0 msgr2=0x564b5b272b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:17:33.792 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:17:33.790+0000 7fb65cc00640 1 --2- 172.21.15.118:0/2673252706 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x564b5b261de0 0x564b5b272b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:17:33.792 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:17:33.790+0000 7fb65cc00640 1 -- 172.21.15.118:0/2673252706 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x564b5b266f50 msgr2=0x564b5b25b820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:17:33.792 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:17:33.790+0000 7fb65cc00640 1 --2- 172.21.15.118:0/2673252706 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x564b5b266f50 0x564b5b25b820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:17:33.792 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:17:33.790+0000 7fb65cc00640 1 -- 172.21.15.118:0/2673252706 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x564b5b2617d0 con 0x564b5b2613f0 2024-03-24T00:17:33.792 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:17:33.790+0000 7fb65cc00640 1 --2- 172.21.15.118:0/2673252706 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x564b5b2613f0 0x564b5b25b2e0 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 crypto rx=0x564b5b273120 tx=0x7fb64c019000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=68cc1df5be56fd34 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:17:33.792 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:17:33.791+0000 7fb649200640 1 -- 172.21.15.118:0/2673252706 <== mon.2 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb64c02b030 con 0x564b5b2613f0 2024-03-24T00:17:33.792 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:17:33.791+0000 7fb649200640 1 -- 172.21.15.118:0/2673252706 <== mon.2 v2:172.21.15.189:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fb64c02b1d0 con 0x564b5b2613f0 2024-03-24T00:17:33.792 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:17:33.791+0000 7fb649200640 1 -- 172.21.15.118:0/2673252706 <== mon.2 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb64c004a00 con 0x564b5b2613f0 2024-03-24T00:17:33.793 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:17:33.792+0000 7fb662363740 1 -- 172.21.15.118:0/2673252706 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x564b5b2613f0 msgr2=0x564b5b25b2e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:17:33.793 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:17:33.792+0000 7fb662363740 1 --2- 172.21.15.118:0/2673252706 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x564b5b2613f0 0x564b5b25b2e0 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 crypto rx=0x564b5b273120 tx=0x7fb64c019000 comp rx=0 tx=0).stop 2024-03-24T00:17:33.793 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:17:33.792+0000 7fb662363740 1 -- 172.21.15.118:0/2673252706 shutdown_connections 2024-03-24T00:17:33.793 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:17:33.792+0000 7fb662363740 1 --2- 172.21.15.118:0/2673252706 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x564b5b266f50 0x564b5b25b820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:17:33.793 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:17:33.792+0000 7fb662363740 1 --2- 172.21.15.118:0/2673252706 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x564b5b261de0 0x564b5b272b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:17:33.793 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:17:33.792+0000 7fb662363740 1 --2- 172.21.15.118:0/2673252706 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x564b5b2613f0 0x564b5b25b2e0 unknown :-1 s=CLOSED pgs=306 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:17:33.793 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:17:33.792+0000 7fb662363740 1 -- 172.21.15.118:0/2673252706 >> 172.21.15.118:0/2673252706 conn(0x564b5b25c960 msgr2=0x564b5b25cd40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:17:33.793 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:17:33.792+0000 7fb662363740 1 -- 172.21.15.118:0/2673252706 shutdown_connections 2024-03-24T00:17:33.794 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:17:33.792+0000 7fb662363740 1 -- 172.21.15.118:0/2673252706 wait complete. 2024-03-24T00:17:33.794 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:17:33.792+0000 7fb662363740 1 -- start start 2024-03-24T00:17:33.794 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:17:33.793+0000 7fb662363740 1 Processor -- start 2024-03-24T00:17:33.796 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:17:33.793+0000 7fb662363740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x564b5b2613f0 0x564b5b2723b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:17:33.796 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:17:33.793+0000 7fb662363740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x564b5b261de0 0x564b5b2728f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:17:33.797 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:17:33.793+0000 7fb662363740 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x564b5b266f50 0x564b5b2709c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:17:33.797 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:17:33.793+0000 7fb662363740 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb6581b0100 con 0x564b5b266f50 2024-03-24T00:17:33.797 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:17:33.793+0000 7fb662363740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb6581afaa0 con 0x564b5b2613f0 2024-03-24T00:17:33.797 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:17:33.793+0000 7fb662363740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb6581acfa0 con 0x564b5b261de0 2024-03-24T00:17:33.797 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:17:33.793+0000 7fb662363740 10 client.0 _ll_register_callbacks cb 0x7fb644009880 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T00:17:33.797 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:17:33.793+0000 7fb65cc00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x564b5b2613f0 0x564b5b2723b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:17:33.797 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:17:33.793+0000 7fb65cc00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x564b5b2613f0 0x564b5b2723b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.118:54238/0 (socket says 172.21.15.118:54238) 2024-03-24T00:17:33.797 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:17:33.793+0000 7fb65cc00640 1 -- 172.21.15.118:0/3585913751 learned_addr learned my addr 172.21.15.118:0/3585913751 (peer_addr_for_me v2:172.21.15.118:0/0) 2024-03-24T00:17:33.797 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:17:33.793+0000 7fb657e00640 1 --2- 172.21.15.118:0/3585913751 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x564b5b261de0 0x564b5b2728f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:17:33.797 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:17:33.794+0000 7fb65d600640 1 --2- 172.21.15.118:0/3585913751 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x564b5b266f50 0x564b5b2709c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:17:33.797 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:17:33.794+0000 7fb65cc00640 1 -- 172.21.15.118:0/3585913751 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x564b5b261de0 msgr2=0x564b5b2728f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:17:33.797 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:17:33.794+0000 7fb65cc00640 1 --2- 172.21.15.118:0/3585913751 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x564b5b261de0 0x564b5b2728f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:17:33.797 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:17:33.794+0000 7fb65cc00640 1 -- 172.21.15.118:0/3585913751 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x564b5b266f50 msgr2=0x564b5b2709c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:17:33.798 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:17:33.794+0000 7fb65cc00640 1 --2- 172.21.15.118:0/3585913751 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x564b5b266f50 0x564b5b2709c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:17:33.798 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:17:33.794+0000 7fb65cc00640 1 -- 172.21.15.118:0/3585913751 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x564b5b2710b0 con 0x564b5b2613f0 2024-03-24T00:17:33.798 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:17:33.794+0000 7fb657e00640 1 --2- 172.21.15.118:0/3585913751 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x564b5b261de0 0x564b5b2728f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T00:17:33.798 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:17:33.794+0000 7fb65cc00640 1 --2- 172.21.15.118:0/3585913751 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x564b5b2613f0 0x564b5b2723b0 secure :-1 s=READY pgs=307 cs=0 l=1 rev1=1 crypto rx=0x7fb64c00f4a0 tx=0x7fb64c02df40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=4d901d8caac8c645 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:17:33.798 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:17:33.794+0000 7fb63e000640 10 client.0 ms_handle_connect on v2:172.21.15.189:3300/0 2024-03-24T00:17:33.798 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:17:33.794+0000 7fb63e000640 1 -- 172.21.15.118:0/3585913751 <== mon.2 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb64c030040 con 0x564b5b2613f0 2024-03-24T00:17:33.798 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:17:33.795+0000 7fb662363740 1 -- 172.21.15.118:0/3585913751 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fb64c02b1d0 con 0x564b5b2613f0 2024-03-24T00:17:33.798 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:17:33.795+0000 7fb63e000640 1 -- 172.21.15.118:0/3585913751 <== mon.2 v2:172.21.15.189:3300/0 2 ==== osd_map(90..90 src has 1..90) v4 ==== 8277+0+0 (secure 0 0 0) 0x7fb64c00f750 con 0x564b5b2613f0 2024-03-24T00:17:33.798 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:17:33.795+0000 7fb63e000640 1 -- 172.21.15.118:0/3585913751 <== mon.2 v2:172.21.15.189:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) v2 ==== 24+0+0 (secure 0 0 0) 0x7fb64c005600 con 0x564b5b2613f0 2024-03-24T00:17:33.798 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:17:33.795+0000 7fb662363740 10 client.25976 fetch_fsmap learned FSMap version 28 2024-03-24T00:17:33.798 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:17:33.795+0000 7fb662363740 1 -- 172.21.15.118:0/3585913751 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) v3 -- 0x564b5b2d3740 con 0x564b5b2613f0 2024-03-24T00:17:33.798 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:17:33.795+0000 7fb63e000640 1 -- 172.21.15.118:0/3585913751 <== mon.2 v2:172.21.15.189:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fb64c02b1d0 con 0x564b5b2613f0 2024-03-24T00:17:33.798 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:17:33.795+0000 7fb63e000640 1 -- 172.21.15.118:0/3585913751 <== mon.2 v2:172.21.15.189:3300/0 5 ==== fsmap(e 28) v1 ==== 7934+0+0 (secure 0 0 0) 0x7fb64c02aa10 con 0x564b5b2613f0 2024-03-24T00:17:33.799 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:17:33.795+0000 7fb662363740 10 client.25976 fetch_fsmap finished waiting for FSMap version 28 2024-03-24T00:17:33.799 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:17:33.795+0000 7fb662363740 10 client.25976 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-24T00:17:33.799 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:17:33.795+0000 7fb662363740 20 client.25976 populate_metadata read hostname 'smithi118' 2024-03-24T00:17:33.799 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:17:33.795+0000 7fb662363740 1 --2- 172.21.15.118:0/3585913751 >> [v2:172.21.15.189:6838/4074484168,v1:172.21.15.189:6839/4074484168] conn(0x564b5b2d40a0 0x564b5b2d64c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:17:33.799 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:17:33.795+0000 7fb662363740 4 client.25976 mds_command: new command op to 24569 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-03-24T00:17:33.799 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:17:33.795+0000 7fb662363740 1 -- 172.21.15.118:0/3585913751 --> [v2:172.21.15.189:6838/4074484168,v1:172.21.15.189:6839/4074484168] -- command(tid 0: {"prefix": "flush journal"}) v1 -- 0x564b5b1ce0c0 con 0x564b5b2d40a0 2024-03-24T00:17:33.799 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:17:33.796+0000 7fb657e00640 1 --2- 172.21.15.118:0/3585913751 >> [v2:172.21.15.189:6838/4074484168,v1:172.21.15.189:6839/4074484168] conn(0x564b5b2d40a0 0x564b5b2d64c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:17:33.799 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:17:33.796+0000 7fb657e00640 1 --2- 172.21.15.118:0/3585913751 >> [v2:172.21.15.189:6838/4074484168,v1:172.21.15.189:6839/4074484168] conn(0x564b5b2d40a0 0x564b5b2d64c0 crc :-1 s=READY pgs=76 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=d6b30f2db08a5a04 server_cookie=942c8ae69ff67da9 in_seq=0 out_seq=0 2024-03-24T00:17:33.799 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:17:33.796+0000 7fb63e000640 10 client.25976 ms_handle_connect on v2:172.21.15.189:6838/4074484168 2024-03-24T00:17:33.805 INFO:tasks.workunit.client.0.smithi118.stdout:if gcc -DHAVE_CONFIG_H -I. -I. -I.. -g -O2 -MT rewriter.o -MD -MP -MF ".deps/rewriter.Tpo" -c -o rewriter.o rewriter.c; \ 2024-03-24T00:17:33.805 INFO:tasks.workunit.client.0.smithi118.stdout:then mv -f ".deps/rewriter.Tpo" ".deps/rewriter.Po"; else rm -f ".deps/rewriter.Tpo"; exit 1; fi 2024-03-24T00:17:33.937 INFO:tasks.workunit.client.0.smithi118.stdout:if gcc -DHAVE_CONFIG_H -I. -I. -I.. -g -O2 -MT reader.o -MD -MP -MF ".deps/reader.Tpo" -c -o reader.o reader.c; \ 2024-03-24T00:17:33.938 INFO:tasks.workunit.client.0.smithi118.stdout:then mv -f ".deps/reader.Tpo" ".deps/reader.Po"; else rm -f ".deps/reader.Tpo"; exit 1; fi 2024-03-24T00:17:33.939 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:17:33.938+0000 7fb63e000640 1 -- 172.21.15.118:0/3585913751 <== mds.0 v2:172.21.15.189:6838/4074484168 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+44 (crc 0 0 0) 0x564b5b1ce0c0 con 0x564b5b2d40a0 2024-03-24T00:17:33.939 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:17:33.938+0000 7fb63e000640 10 client.25976 handle_command_reply: tid=0 2024-03-24T00:17:33.939 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:17:33.938+0000 7fb662363740 1 client.25976 shutdown 2024-03-24T00:17:33.939 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:17:33.938+0000 7fb662363740 1 -- 172.21.15.118:0/3585913751 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x564b5b2613f0 msgr2=0x564b5b2723b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:17:33.939 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:17:33.938+0000 7fb662363740 1 --2- 172.21.15.118:0/3585913751 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x564b5b2613f0 0x564b5b2723b0 secure :-1 s=READY pgs=307 cs=0 l=1 rev1=1 crypto rx=0x7fb64c00f4a0 tx=0x7fb64c02df40 comp rx=0 tx=0).stop 2024-03-24T00:17:33.939 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:17:33.938+0000 7fb662363740 1 -- 172.21.15.118:0/3585913751 shutdown_connections 2024-03-24T00:17:33.940 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:17:33.938+0000 7fb662363740 1 --2- 172.21.15.118:0/3585913751 >> [v2:172.21.15.189:6838/4074484168,v1:172.21.15.189:6839/4074484168] conn(0x564b5b2d40a0 0x564b5b2d64c0 crc :-1 s=READY pgs=76 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:17:33.940 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:17:33.938+0000 7fb662363740 1 --2- 172.21.15.118:0/3585913751 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x564b5b266f50 0x564b5b2709c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:17:33.940 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:17:33.938+0000 7fb662363740 1 --2- 172.21.15.118:0/3585913751 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x564b5b261de0 0x564b5b2728f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:17:33.940 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:17:33.938+0000 7fb63e000640 0 client.25976 ms_handle_reset on v2:172.21.15.189:6838/4074484168 2024-03-24T00:17:33.940 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:17:33.938+0000 7fb662363740 1 --2- 172.21.15.118:0/3585913751 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x564b5b2613f0 0x564b5b2723b0 unknown :-1 s=CLOSED pgs=307 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:17:33.940 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:17:33.938+0000 7fb662363740 1 -- 172.21.15.118:0/3585913751 >> 172.21.15.118:0/3585913751 conn(0x564b5b25c960 msgr2=0x564b5b25fbc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:17:33.940 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:17:33.938+0000 7fb662363740 1 -- 172.21.15.118:0/3585913751 shutdown_connections 2024-03-24T00:17:33.940 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:17:33.938+0000 7fb662363740 1 -- 172.21.15.118:0/3585913751 wait complete. 2024-03-24T00:17:33.940 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:17:33.939+0000 7fb662363740 20 client.25976 trim_cache size 0 max 16384 2024-03-24T00:17:33.940 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stdout:{ 2024-03-24T00:17:33.941 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stdout: "message": "", 2024-03-24T00:17:33.941 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stdout: "return_code": 0 2024-03-24T00:17:33.941 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stdout:} 2024-03-24T00:17:33.941 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:17:33.939+0000 7fb65e000640 1 -- 172.21.15.118:0/2349861125 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7fb6400a2870 msgr2=0x7fb6400a4d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:17:33.941 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:17:33.939+0000 7fb65e000640 1 --2- 172.21.15.118:0/2349861125 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7fb6400a2870 0x7fb6400a4d30 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 crypto rx=0x564b5b2621e0 tx=0x7fb64c005f90 comp rx=0 tx=0).stop 2024-03-24T00:17:33.942 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:17:33.939+0000 7fb65e000640 1 -- 172.21.15.118:0/2349861125 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fb6580756e0 msgr2=0x7fb65819e360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:17:33.942 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:17:33.939+0000 7fb65e000640 1 --2- 172.21.15.118:0/2349861125 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fb6580756e0 0x7fb65819e360 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x7fb644000c00 tx=0x7fb64402c6c0 comp rx=0 tx=0).stop 2024-03-24T00:17:33.942 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:17:33.939+0000 7fb65e000640 1 -- 172.21.15.118:0/2349861125 shutdown_connections 2024-03-24T00:17:33.942 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:17:33.939+0000 7fb65e000640 1 --2- 172.21.15.118:0/2349861125 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7fb6400a2870 0x7fb6400a4d30 unknown :-1 s=CLOSED pgs=305 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:17:33.942 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:17:33.939+0000 7fb65e000640 1 --2- 172.21.15.118:0/2349861125 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb65810dae0 0x7fb6581a53d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:17:33.942 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:17:33.939+0000 7fb65e000640 1 --2- 172.21.15.118:0/2349861125 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fb6580756e0 0x7fb65819e360 unknown :-1 s=CLOSED pgs=287 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:17:33.942 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:17:33.939+0000 7fb65e000640 1 --2- 172.21.15.118:0/2349861125 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb658074100 0x7fb65819de20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:17:33.942 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:17:33.939+0000 7fb65e000640 1 -- 172.21.15.118:0/2349861125 >> 172.21.15.118:0/2349861125 conn(0x7fb6580fca80 msgr2=0x7fb658100d70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:17:33.942 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:17:33.940+0000 7fb65e000640 1 -- 172.21.15.118:0/2349861125 shutdown_connections 2024-03-24T00:17:33.942 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:17:33.940+0000 7fb65e000640 1 -- 172.21.15.118:0/2349861125 wait complete. 2024-03-24T00:17:34.063 INFO:tasks.workunit.client.0.smithi118.stdout:if gcc -DHAVE_CONFIG_H -I. -I. -I.. -g -O2 -MT commenter.o -MD -MP -MF ".deps/commenter.Tpo" -c -o commenter.o commenter.c; \ 2024-03-24T00:17:34.063 INFO:tasks.workunit.client.0.smithi118.stdout:then mv -f ".deps/commenter.Tpo" ".deps/commenter.Po"; else rm -f ".deps/commenter.Tpo"; exit 1; fi 2024-03-24T00:17:34.214 INFO:tasks.workunit.client.0.smithi118.stdout:if gcc -DHAVE_CONFIG_H -I. -I. -I.. -g -O2 -MT helpers.o -MD -MP -MF ".deps/helpers.Tpo" -c -o helpers.o helpers.c; \ 2024-03-24T00:17:34.214 INFO:tasks.workunit.client.0.smithi118.stdout:then mv -f ".deps/helpers.Tpo" ".deps/helpers.Po"; else rm -f ".deps/helpers.Tpo"; exit 1; fi 2024-03-24T00:17:34.373 INFO:tasks.workunit.client.0.smithi118.stdout:if gcc -DHAVE_CONFIG_H -I. -I. -I.. -g -O2 -MT bsd-getopt_long.o -MD -MP -MF ".deps/bsd-getopt_long.Tpo" -c -o bsd-getopt_long.o bsd-getopt_long.c; \ 2024-03-24T00:17:34.373 INFO:tasks.workunit.client.0.smithi118.stdout:then mv -f ".deps/bsd-getopt_long.Tpo" ".deps/bsd-getopt_long.Po"; else rm -f ".deps/bsd-getopt_long.Tpo"; exit 1; fi 2024-03-24T00:17:34.481 INFO:tasks.workunit.client.0.smithi118.stdout:if gcc -DHAVE_CONFIG_H -I. -I. -I.. -g -O2 -MT fakesnprintf.o -MD -MP -MF ".deps/fakesnprintf.Tpo" -c -o fakesnprintf.o fakesnprintf.c; \ 2024-03-24T00:17:34.481 INFO:tasks.workunit.client.0.smithi118.stdout:then mv -f ".deps/fakesnprintf.Tpo" ".deps/fakesnprintf.Po"; else rm -f ".deps/fakesnprintf.Tpo"; exit 1; fi 2024-03-24T00:17:34.593 INFO:tasks.workunit.client.0.smithi118.stdout:if gcc -DHAVE_CONFIG_H -I. -I. -I.. -g -O2 -MT mysnprintf.o -MD -MP -MF ".deps/mysnprintf.Tpo" -c -o mysnprintf.o mysnprintf.c; \ 2024-03-24T00:17:34.593 INFO:tasks.workunit.client.0.smithi118.stdout:then mv -f ".deps/mysnprintf.Tpo" ".deps/mysnprintf.Po"; else rm -f ".deps/mysnprintf.Tpo"; exit 1; fi 2024-03-24T00:17:34.705 INFO:tasks.workunit.client.0.smithi118.stdout:gcc -g -O2 -o blogbench blogbench.o process.o writer.o rewriter.o reader.o commenter.o helpers.o bsd-getopt_long.o fakesnprintf.o mysnprintf.o 2024-03-24T00:17:34.759 INFO:tasks.workunit.client.0.smithi118.stdout:make[2]: Leaving directory '/home/ubuntu/cephtest/mnt.0/client.0/tmp/blogbench-1.0/src' 2024-03-24T00:17:34.759 INFO:tasks.workunit.client.0.smithi118.stdout:Making all in man 2024-03-24T00:17:34.766 INFO:tasks.workunit.client.0.smithi118.stdout:make[2]: Entering directory '/home/ubuntu/cephtest/mnt.0/client.0/tmp/blogbench-1.0/man' 2024-03-24T00:17:34.766 INFO:tasks.workunit.client.0.smithi118.stdout:make[2]: Nothing to be done for 'all'. 2024-03-24T00:17:34.766 INFO:tasks.workunit.client.0.smithi118.stdout:make[2]: Leaving directory '/home/ubuntu/cephtest/mnt.0/client.0/tmp/blogbench-1.0/man' 2024-03-24T00:17:34.773 INFO:tasks.workunit.client.0.smithi118.stdout:make[2]: Entering directory '/home/ubuntu/cephtest/mnt.0/client.0/tmp/blogbench-1.0' 2024-03-24T00:17:34.776 INFO:tasks.workunit.client.0.smithi118.stdout:make[2]: Leaving directory '/home/ubuntu/cephtest/mnt.0/client.0/tmp/blogbench-1.0' 2024-03-24T00:17:34.777 INFO:tasks.workunit.client.0.smithi118.stdout:make[1]: Leaving directory '/home/ubuntu/cephtest/mnt.0/client.0/tmp/blogbench-1.0' 2024-03-24T00:17:34.778 INFO:tasks.workunit.client.0.smithi118.stderr:+ cd src 2024-03-24T00:17:34.779 INFO:tasks.workunit.client.0.smithi118.stderr:+ mkdir blogtest_in 2024-03-24T00:17:34.789 INFO:tasks.workunit.client.0.smithi118.stderr:+ echo 'running blogbench' 2024-03-24T00:17:34.789 INFO:tasks.workunit.client.0.smithi118.stderr:+ ./blogbench -d blogtest_in 2024-03-24T00:17:34.790 INFO:tasks.workunit.client.0.smithi118.stdout:running blogbench 2024-03-24T00:17:34.792 INFO:tasks.workunit.client.0.smithi118.stdout: 2024-03-24T00:17:34.792 INFO:tasks.workunit.client.0.smithi118.stdout:Frequency = 10 secs 2024-03-24T00:17:34.792 INFO:tasks.workunit.client.0.smithi118.stdout:Scratch dir = [blogtest_in] 2024-03-24T00:17:35.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:17:34 smithi118 ceph-mon[31535]: pgmap v454: 97 pgs: 97 active+clean; 3.4 MiB data, 372 MiB used, 1.0 TiB / 1.0 TiB avail; 157 KiB/s rd, 314 KiB/s wr, 94 op/s 2024-03-24T00:17:35.203 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:17:34 smithi189 ceph-mon[36187]: pgmap v454: 97 pgs: 97 active+clean; 3.4 MiB data, 372 MiB used, 1.0 TiB / 1.0 TiB avail; 157 KiB/s rd, 314 KiB/s wr, 94 op/s 2024-03-24T00:17:35.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:17:34 smithi193 ceph-mon[36878]: pgmap v454: 97 pgs: 97 active+clean; 3.4 MiB data, 372 MiB used, 1.0 TiB / 1.0 TiB avail; 157 KiB/s rd, 314 KiB/s wr, 94 op/s 2024-03-24T00:17:36.826 INFO:tasks.workunit.client.0.smithi118.stdout:Spawning 3 writers... 2024-03-24T00:17:36.827 INFO:tasks.workunit.client.0.smithi118.stdout:Spawning 1 rewriters... 2024-03-24T00:17:36.827 INFO:tasks.workunit.client.0.smithi118.stdout:Spawning 5 commenters... 2024-03-24T00:17:36.827 INFO:tasks.workunit.client.0.smithi118.stdout:Spawning 100 readers... 2024-03-24T00:17:36.834 INFO:tasks.workunit.client.0.smithi118.stdout:Benchmarking for 30 iterations. 2024-03-24T00:17:36.834 INFO:tasks.workunit.client.0.smithi118.stdout:The test will run during 5 minutes. 2024-03-24T00:17:36.834 INFO:tasks.workunit.client.0.smithi118.stdout: 2024-03-24T00:17:36.834 INFO:tasks.workunit.client.0.smithi118.stdout: Nb blogs R articles W articles R pictures W pictures R comments W comments 2024-03-24T00:17:37.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:17:36 smithi118 ceph-mon[31535]: pgmap v455: 97 pgs: 97 active+clean; 4.2 MiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 240 KiB/s rd, 425 KiB/s wr, 119 op/s 2024-03-24T00:17:37.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:17:36 smithi189 ceph-mon[36187]: pgmap v455: 97 pgs: 97 active+clean; 4.2 MiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 240 KiB/s rd, 425 KiB/s wr, 119 op/s 2024-03-24T00:17:37.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:17:36 smithi193 ceph-mon[36878]: pgmap v455: 97 pgs: 97 active+clean; 4.2 MiB data, 374 MiB used, 1.0 TiB / 1.0 TiB avail; 240 KiB/s rd, 425 KiB/s wr, 119 op/s 2024-03-24T00:17:39.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:17:38 smithi118 ceph-mon[31535]: pgmap v456: 97 pgs: 97 active+clean; 4.3 MiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 192 KiB/s rd, 358 KiB/s wr, 111 op/s 2024-03-24T00:17:39.203 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:17:38 smithi189 ceph-mon[36187]: pgmap v456: 97 pgs: 97 active+clean; 4.3 MiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 192 KiB/s rd, 358 KiB/s wr, 111 op/s 2024-03-24T00:17:39.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:17:38 smithi193 ceph-mon[36878]: pgmap v456: 97 pgs: 97 active+clean; 4.3 MiB data, 375 MiB used, 1.0 TiB / 1.0 TiB avail; 192 KiB/s rd, 358 KiB/s wr, 111 op/s 2024-03-24T00:17:41.195 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:17:40 smithi118 ceph-mon[31535]: pgmap v457: 97 pgs: 97 active+clean; 5.8 MiB data, 378 MiB used, 1.0 TiB / 1.0 TiB avail; 301 KiB/s rd, 444 KiB/s wr, 134 op/s 2024-03-24T00:17:41.203 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:17:40 smithi189 ceph-mon[36187]: pgmap v457: 97 pgs: 97 active+clean; 5.8 MiB data, 378 MiB used, 1.0 TiB / 1.0 TiB avail; 301 KiB/s rd, 444 KiB/s wr, 134 op/s 2024-03-24T00:17:41.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:17:40 smithi193 ceph-mon[36878]: pgmap v457: 97 pgs: 97 active+clean; 5.8 MiB data, 378 MiB used, 1.0 TiB / 1.0 TiB avail; 301 KiB/s rd, 444 KiB/s wr, 134 op/s 2024-03-24T00:17:43.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:17:42 smithi118 ceph-mon[31535]: pgmap v458: 97 pgs: 97 active+clean; 5.7 MiB data, 383 MiB used, 1.0 TiB / 1.0 TiB avail; 208 KiB/s rd, 325 KiB/s wr, 104 op/s 2024-03-24T00:17:43.203 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:17:42 smithi189 ceph-mon[36187]: pgmap v458: 97 pgs: 97 active+clean; 5.7 MiB data, 383 MiB used, 1.0 TiB / 1.0 TiB avail; 208 KiB/s rd, 325 KiB/s wr, 104 op/s 2024-03-24T00:17:43.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:17:42 smithi193 ceph-mon[36878]: pgmap v458: 97 pgs: 97 active+clean; 5.7 MiB data, 383 MiB used, 1.0 TiB / 1.0 TiB avail; 208 KiB/s rd, 325 KiB/s wr, 104 op/s 2024-03-24T00:17:45.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:17:44 smithi118 ceph-mon[31535]: pgmap v459: 97 pgs: 97 active+clean; 15 MiB data, 401 MiB used, 1.0 TiB / 1.0 TiB avail; 441 KiB/s rd, 1.2 MiB/s wr, 170 op/s 2024-03-24T00:17:45.203 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:17:44 smithi189 ceph-mon[36187]: pgmap v459: 97 pgs: 97 active+clean; 15 MiB data, 401 MiB used, 1.0 TiB / 1.0 TiB avail; 441 KiB/s rd, 1.2 MiB/s wr, 170 op/s 2024-03-24T00:17:45.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:17:44 smithi193 ceph-mon[36878]: pgmap v459: 97 pgs: 97 active+clean; 15 MiB data, 401 MiB used, 1.0 TiB / 1.0 TiB avail; 441 KiB/s rd, 1.2 MiB/s wr, 170 op/s 2024-03-24T00:17:46.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:17:45 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T00:17:46.203 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:17:45 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T00:17:46.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:17:45 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T00:17:46.834 INFO:tasks.workunit.client.0.smithi118.stdout: 2 22114 199 7933 266 3862 263 2024-03-24T00:17:47.089 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:17:47.088+0000 7fc33fc00640 1 -- 172.21.15.118:0/2946872410 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fc338101290 msgr2=0x7fc338101710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:17:47.392 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:17:47.088+0000 7fc33fc00640 1 --2- 172.21.15.118:0/2946872410 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fc338101290 0x7fc338101710 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 crypto rx=0x7fc330009ca0 tx=0x7fc33002c0f0 comp rx=0 tx=0).stop 2024-03-24T00:17:47.392 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:17:47.088+0000 7fc33fc00640 1 -- 172.21.15.118:0/2946872410 shutdown_connections 2024-03-24T00:17:47.392 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:17:47.088+0000 7fc33fc00640 1 --2- 172.21.15.118:0/2946872410 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fc338101290 0x7fc338101710 unknown :-1 s=CLOSED pgs=288 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:17:47.392 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:17:47.088+0000 7fc33fc00640 1 --2- 172.21.15.118:0/2946872410 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fc3381008f0 0x7fc338100d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:17:47.392 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:17:47.088+0000 7fc33fc00640 1 --2- 172.21.15.118:0/2946872410 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc3380fff90 0x7fc3381003b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:17:47.392 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:17:47.088+0000 7fc33fc00640 1 -- 172.21.15.118:0/2946872410 >> 172.21.15.118:0/2946872410 conn(0x7fc3380fbb30 msgr2=0x7fc3380fdf70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:17:47.393 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:17:47.089+0000 7fc33fc00640 1 -- 172.21.15.118:0/2946872410 shutdown_connections 2024-03-24T00:17:47.393 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:17:47.089+0000 7fc33fc00640 1 -- 172.21.15.118:0/2946872410 wait complete. 2024-03-24T00:17:47.393 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:17:47.090+0000 7fc33fc00640 1 Processor -- start 2024-03-24T00:17:47.394 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:17:46 smithi193 ceph-mon[36878]: pgmap v460: 97 pgs: 97 active+clean; 20 MiB data, 408 MiB used, 1.0 TiB / 1.0 TiB avail; 438 KiB/s rd, 1.6 MiB/s wr, 188 op/s 2024-03-24T00:17:47.394 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:17:46 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T00:17:47.394 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:17:46 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T00:17:47.394 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:17:46 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T00:17:47.394 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:17:46 smithi189 ceph-mon[36187]: pgmap v460: 97 pgs: 97 active+clean; 20 MiB data, 408 MiB used, 1.0 TiB / 1.0 TiB avail; 438 KiB/s rd, 1.6 MiB/s wr, 188 op/s 2024-03-24T00:17:47.394 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:17:46 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T00:17:47.394 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:17:46 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T00:17:47.394 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:17:46 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T00:17:47.405 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:17:47.090+0000 7fc33fc00640 1 -- start start 2024-03-24T00:17:47.406 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:17:47.090+0000 7fc33fc00640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fc3380fff90 0x7fc338199ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:17:47.406 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:17:47.090+0000 7fc33fc00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc3381008f0 0x7fc33819a020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:17:47.406 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:17:47.090+0000 7fc33fc00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fc338101290 0x7fc3381a1090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:17:47.406 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:17:47.090+0000 7fc33fc00640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc33810f530 con 0x7fc3380fff90 2024-03-24T00:17:47.406 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:17:47.090+0000 7fc33fc00640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc33810f3b0 con 0x7fc3381008f0 2024-03-24T00:17:47.406 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:17:47.090+0000 7fc33fc00640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc33810f6b0 con 0x7fc338101290 2024-03-24T00:17:47.406 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:17:47.090+0000 7fc33d200640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fc3380fff90 0x7fc338199ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:17:47.406 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:17:47.090+0000 7fc337e00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc3381008f0 0x7fc33819a020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:17:47.406 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:17:47.090+0000 7fc33dc00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fc338101290 0x7fc3381a1090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:17:47.406 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:17:47.090+0000 7fc33d200640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fc3380fff90 0x7fc338199ae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.118:3300/0 says I am v2:172.21.15.118:44326/0 (socket says 172.21.15.118:44326) 2024-03-24T00:17:47.406 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:17:47.090+0000 7fc33d200640 1 -- 172.21.15.118:0/3347686696 learned_addr learned my addr 172.21.15.118:0/3347686696 (peer_addr_for_me v2:172.21.15.118:0/0) 2024-03-24T00:17:47.407 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:17:47.090+0000 7fc337e00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc3381008f0 0x7fc33819a020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.118:44944/0 (socket says 172.21.15.118:44944) 2024-03-24T00:17:47.407 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:17:47.091+0000 7fc33dc00640 1 -- 172.21.15.118:0/3347686696 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc3381008f0 msgr2=0x7fc33819a020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:17:47.407 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:17:47.091+0000 7fc33dc00640 1 --2- 172.21.15.118:0/3347686696 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc3381008f0 0x7fc33819a020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:17:47.407 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:17:47.091+0000 7fc33dc00640 1 -- 172.21.15.118:0/3347686696 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fc3380fff90 msgr2=0x7fc338199ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:17:47.407 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:17:47.091+0000 7fc33dc00640 1 --2- 172.21.15.118:0/3347686696 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fc3380fff90 0x7fc338199ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:17:47.407 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:17:47.091+0000 7fc33dc00640 1 -- 172.21.15.118:0/3347686696 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc3381a1700 con 0x7fc338101290 2024-03-24T00:17:47.407 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:17:47.091+0000 7fc33d200640 1 --2- 172.21.15.118:0/3347686696 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fc3380fff90 0x7fc338199ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T00:17:47.407 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:17:47.091+0000 7fc33dc00640 1 --2- 172.21.15.118:0/3347686696 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fc338101290 0x7fc3381a1090 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 crypto rx=0x7fc330037040 tx=0x7fc330032ea0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:17:47.407 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:17:47.091+0000 7fc335600640 1 -- 172.21.15.118:0/3347686696 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc330041020 con 0x7fc338101290 2024-03-24T00:17:47.407 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:17:47.092+0000 7fc335600640 1 -- 172.21.15.118:0/3347686696 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fc330034cd0 con 0x7fc338101290 2024-03-24T00:17:47.407 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:17:47.092+0000 7fc33fc00640 1 -- 172.21.15.118:0/3347686696 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc3381a1990 con 0x7fc338101290 2024-03-24T00:17:47.407 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:17:47.092+0000 7fc335600640 1 -- 172.21.15.118:0/3347686696 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc330039040 con 0x7fc338101290 2024-03-24T00:17:47.407 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:17:47.092+0000 7fc33fc00640 1 -- 172.21.15.118:0/3347686696 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc3381a1ea0 con 0x7fc338101290 2024-03-24T00:17:47.407 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:17:47.095+0000 7fc3429ff740 1 Processor -- start 2024-03-24T00:17:47.408 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:17:47.095+0000 7fc335600640 1 -- 172.21.15.118:0/3347686696 <== mon.1 v2:172.21.15.193:3300/0 4 ==== mgrmap(e 20) v1 ==== 137291+0+0 (secure 0 0 0) 0x7fc330044020 con 0x7fc338101290 2024-03-24T00:17:47.408 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:17:47.096+0000 7fc3429ff740 1 -- start start 2024-03-24T00:17:47.408 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:17:47.096+0000 7fc3429ff740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x560104a831e0 0x560104a83600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:17:47.408 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:17:47.096+0000 7fc3429ff740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x560104a83bd0 0x560104a886d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:17:47.408 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:17:47.096+0000 7fc3429ff740 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x560104a88d40 0x560104a8b1d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:17:47.408 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:17:47.096+0000 7fc3429ff740 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x560104a463b0 con 0x560104a88d40 2024-03-24T00:17:47.408 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:17:47.096+0000 7fc3429ff740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x5601049e8830 con 0x560104a83bd0 2024-03-24T00:17:47.408 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:17:47.096+0000 7fc3429ff740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x5601049915d0 con 0x560104a831e0 2024-03-24T00:17:47.408 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:17:47.096+0000 7fc337e00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x560104a831e0 0x560104a83600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:17:47.408 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:17:47.096+0000 7fc33d200640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x560104a83bd0 0x560104a886d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:17:47.408 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:17:47.096+0000 7fc33d200640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x560104a83bd0 0x560104a886d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.118:44954/0 (socket says 172.21.15.118:44954) 2024-03-24T00:17:47.408 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:17:47.096+0000 7fc33d200640 1 -- 172.21.15.118:0/2724575450 learned_addr learned my addr 172.21.15.118:0/2724575450 (peer_addr_for_me v2:172.21.15.118:0/0) 2024-03-24T00:17:47.409 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:17:47.096+0000 7fc337e00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x560104a831e0 0x560104a83600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.118:46350/0 (socket says 172.21.15.118:46350) 2024-03-24T00:17:47.409 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:17:47.096+0000 7fc337e00640 1 --2- 172.21.15.118:0/2724575450 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x560104a88d40 0x560104a8b1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:17:47.409 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:17:47.097+0000 7fc33d200640 1 -- 172.21.15.118:0/2724575450 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x560104a831e0 msgr2=0x560104a83600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:17:47.409 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:17:47.097+0000 7fc33d200640 1 --2- 172.21.15.118:0/2724575450 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x560104a831e0 0x560104a83600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:17:47.409 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:17:47.097+0000 7fc33d200640 1 -- 172.21.15.118:0/2724575450 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x560104a88d40 msgr2=0x560104a8b1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:17:47.409 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:17:47.097+0000 7fc33d200640 1 --2- 172.21.15.118:0/2724575450 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x560104a88d40 0x560104a8b1d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:17:47.409 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:17:47.097+0000 7fc33d200640 1 -- 172.21.15.118:0/2724575450 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5601049a92c0 con 0x560104a83bd0 2024-03-24T00:17:47.409 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:17:47.097+0000 7fc337e00640 1 --2- 172.21.15.118:0/2724575450 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x560104a831e0 0x560104a83600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T00:17:47.409 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:17:47.097+0000 7fc335600640 1 --2- 172.21.15.118:0/3347686696 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7fc3200a2920 0x7fc3200a4de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:17:47.409 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:17:47.097+0000 7fc33d200640 1 --2- 172.21.15.118:0/2724575450 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x560104a83bd0 0x560104a886d0 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 crypto rx=0x7fc32c011630 tx=0x7fc32c011af0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c33af8199c041c9e server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:17:47.409 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:17:47.097+0000 7fc335600640 1 -- 172.21.15.118:0/3347686696 <== mon.1 v2:172.21.15.193:3300/0 5 ==== osd_map(90..90 src has 1..90) v4 ==== 8277+0+0 (secure 0 0 0) 0x7fc3300f0790 con 0x7fc338101290 2024-03-24T00:17:47.409 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:17:47.097+0000 7fc32a000640 1 -- 172.21.15.118:0/2724575450 <== mon.2 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc32c01e070 con 0x560104a83bd0 2024-03-24T00:17:47.409 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:17:47.098+0000 7fc32a000640 1 -- 172.21.15.118:0/2724575450 <== mon.2 v2:172.21.15.189:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fc32c00fbd0 con 0x560104a83bd0 2024-03-24T00:17:47.410 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:17:47.098+0000 7fc337e00640 1 --2- 172.21.15.118:0/3347686696 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7fc3200a2920 0x7fc3200a4de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:17:47.410 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:17:47.098+0000 7fc32a000640 1 -- 172.21.15.118:0/2724575450 <== mon.2 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc32c019d00 con 0x560104a83bd0 2024-03-24T00:17:47.410 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:17:47.099+0000 7fc337e00640 1 --2- 172.21.15.118:0/3347686696 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7fc3200a2920 0x7fc3200a4de0 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 crypto rx=0x560104a84430 tx=0x7fc324013040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:17:47.410 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:17:47.099+0000 7fc3429ff740 1 -- 172.21.15.118:0/2724575450 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x560104a83bd0 msgr2=0x560104a886d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:17:47.410 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:17:47.099+0000 7fc3429ff740 1 --2- 172.21.15.118:0/2724575450 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x560104a83bd0 0x560104a886d0 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 crypto rx=0x7fc32c011630 tx=0x7fc32c011af0 comp rx=0 tx=0).stop 2024-03-24T00:17:47.410 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:17:47.099+0000 7fc3429ff740 1 -- 172.21.15.118:0/2724575450 shutdown_connections 2024-03-24T00:17:47.410 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:17:47.099+0000 7fc3429ff740 1 --2- 172.21.15.118:0/2724575450 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x560104a88d40 0x560104a8b1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:17:47.410 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:17:47.099+0000 7fc3429ff740 1 --2- 172.21.15.118:0/2724575450 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x560104a83bd0 0x560104a886d0 unknown :-1 s=CLOSED pgs=308 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:17:47.410 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:17:47.099+0000 7fc3429ff740 1 --2- 172.21.15.118:0/2724575450 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x560104a831e0 0x560104a83600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:17:47.410 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:17:47.099+0000 7fc3429ff740 1 -- 172.21.15.118:0/2724575450 >> 172.21.15.118:0/2724575450 conn(0x560104a7e750 msgr2=0x560104a7eb50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:17:47.410 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:17:47.099+0000 7fc3429ff740 1 -- 172.21.15.118:0/2724575450 shutdown_connections 2024-03-24T00:17:47.410 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:17:47.099+0000 7fc3429ff740 1 -- 172.21.15.118:0/2724575450 wait complete. 2024-03-24T00:17:47.410 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:17:47.100+0000 7fc3429ff740 1 -- start start 2024-03-24T00:17:47.411 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:17:47.100+0000 7fc3429ff740 1 Processor -- start 2024-03-24T00:17:47.411 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:17:47.100+0000 7fc3429ff740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x560104a831e0 0x560104a8d420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:17:47.411 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:17:47.100+0000 7fc3429ff740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x560104a83bd0 0x560104a8d960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:17:47.411 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:17:47.100+0000 7fc3429ff740 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x560104a88d40 0x560104a94930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:17:47.411 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:17:47.100+0000 7fc3429ff740 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x5601049a4a20 con 0x560104a88d40 2024-03-24T00:17:47.411 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:17:47.100+0000 7fc3429ff740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x5601049e48d0 con 0x560104a831e0 2024-03-24T00:17:47.411 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:17:47.100+0000 7fc3429ff740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x560104a8bcf0 con 0x560104a83bd0 2024-03-24T00:17:47.411 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:17:47.100+0000 7fc3429ff740 10 client.0 _ll_register_callbacks cb 0x5601049e11c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T00:17:47.411 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:17:47.101+0000 7fc337e00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x560104a83bd0 0x560104a8d960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:17:47.411 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:17:47.101+0000 7fc33d200640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x560104a831e0 0x560104a8d420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:17:47.411 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:17:47.101+0000 7fc337e00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x560104a83bd0 0x560104a8d960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.118:46366/0 (socket says 172.21.15.118:46366) 2024-03-24T00:17:47.411 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:17:47.101+0000 7fc337e00640 1 -- 172.21.15.118:0/3167778209 learned_addr learned my addr 172.21.15.118:0/3167778209 (peer_addr_for_me v2:172.21.15.118:0/0) 2024-03-24T00:17:47.411 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:17:47.101+0000 7fc33dc00640 1 --2- 172.21.15.118:0/3167778209 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x560104a88d40 0x560104a94930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:17:47.412 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:17:47.101+0000 7fc337e00640 1 -- 172.21.15.118:0/3167778209 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x560104a831e0 msgr2=0x560104a8d420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:17:47.412 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:17:47.101+0000 7fc337e00640 1 --2- 172.21.15.118:0/3167778209 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x560104a831e0 0x560104a8d420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:17:47.412 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:17:47.101+0000 7fc337e00640 1 -- 172.21.15.118:0/3167778209 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x560104a88d40 msgr2=0x560104a94930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:17:47.412 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:17:47.101+0000 7fc337e00640 1 --2- 172.21.15.118:0/3167778209 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x560104a88d40 0x560104a94930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:17:47.412 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:17:47.101+0000 7fc337e00640 1 -- 172.21.15.118:0/3167778209 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5601046fd210 con 0x560104a83bd0 2024-03-24T00:17:47.412 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:17:47.101+0000 7fc33d200640 1 --2- 172.21.15.118:0/3167778209 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x560104a831e0 0x560104a8d420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T00:17:47.412 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:17:47.101+0000 7fc337e00640 1 --2- 172.21.15.118:0/3167778209 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x560104a83bd0 0x560104a8d960 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x7fc324009560 tx=0x7fc324004730 comp rx=0 tx=0).ready entity=mon.1 client_cookie=72b9493c7a0adee9 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:17:47.412 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:17:47.101+0000 7fc31ea00640 10 client.0 ms_handle_connect on v2:172.21.15.193:3300/0 2024-03-24T00:17:47.412 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:17:47.102+0000 7fc31ea00640 1 -- 172.21.15.118:0/3167778209 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc32402d060 con 0x560104a83bd0 2024-03-24T00:17:47.412 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:17:47.102+0000 7fc3429ff740 1 -- 172.21.15.118:0/3167778209 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fc32c00fbd0 con 0x560104a83bd0 2024-03-24T00:17:47.412 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:17:47.102+0000 7fc31ea00640 1 -- 172.21.15.118:0/3167778209 <== mon.1 v2:172.21.15.193:3300/0 2 ==== osd_map(90..90 src has 1..90) v4 ==== 8277+0+0 (secure 0 0 0) 0x7fc32401a070 con 0x560104a83bd0 2024-03-24T00:17:47.412 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:17:47.102+0000 7fc31ea00640 1 -- 172.21.15.118:0/3167778209 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) v2 ==== 24+0+0 (secure 0 0 0) 0x7fc324020e10 con 0x560104a83bd0 2024-03-24T00:17:47.413 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:17:47.102+0000 7fc3429ff740 10 client.25975 fetch_fsmap learned FSMap version 28 2024-03-24T00:17:47.413 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:17:47.102+0000 7fc3429ff740 1 -- 172.21.15.118:0/3167778209 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) v3 -- 0x5601047c9620 con 0x560104a83bd0 2024-03-24T00:17:47.413 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:17:47.102+0000 7fc31ea00640 1 -- 172.21.15.118:0/3167778209 <== mon.1 v2:172.21.15.193:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fc32c00fbd0 con 0x560104a83bd0 2024-03-24T00:17:47.413 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:17:47.102+0000 7fc31ea00640 1 -- 172.21.15.118:0/3167778209 <== mon.1 v2:172.21.15.193:3300/0 5 ==== fsmap(e 28) v1 ==== 7934+0+0 (secure 0 0 0) 0x7fc324027030 con 0x560104a83bd0 2024-03-24T00:17:47.413 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:17:47.103+0000 7fc3429ff740 10 client.25975 fetch_fsmap finished waiting for FSMap version 28 2024-03-24T00:17:47.413 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:17:47.103+0000 7fc3429ff740 10 client.25975 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-24T00:17:47.413 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:17:47.103+0000 7fc3429ff740 20 client.25975 populate_metadata read hostname 'smithi118' 2024-03-24T00:17:47.413 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:17:47.103+0000 7fc3429ff740 1 --2- 172.21.15.118:0/3167778209 >> [v2:172.21.15.189:6838/4074484168,v1:172.21.15.189:6839/4074484168] conn(0x560104a99380 0x560104a9b7e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:17:47.413 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:17:47.103+0000 7fc3429ff740 4 client.25975 mds_command: new command op to 24569 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-24T00:17:47.413 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:17:47.103+0000 7fc3429ff740 1 -- 172.21.15.118:0/3167778209 --> [v2:172.21.15.189:6838/4074484168,v1:172.21.15.189:6839/4074484168] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x5601049f5d70 con 0x560104a99380 2024-03-24T00:17:47.413 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:17:47.103+0000 7fc33d200640 1 --2- 172.21.15.118:0/3167778209 >> [v2:172.21.15.189:6838/4074484168,v1:172.21.15.189:6839/4074484168] conn(0x560104a99380 0x560104a9b7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:17:47.413 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:17:47.103+0000 7fc33d200640 1 --2- 172.21.15.118:0/3167778209 >> [v2:172.21.15.189:6838/4074484168,v1:172.21.15.189:6839/4074484168] conn(0x560104a99380 0x560104a9b7e0 crc :-1 s=READY pgs=77 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=ed22b8f550b56d6 server_cookie=c4530049b9270b9d in_seq=0 out_seq=0 2024-03-24T00:17:47.413 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:17:47.103+0000 7fc31ea00640 10 client.25975 ms_handle_connect on v2:172.21.15.189:6838/4074484168 2024-03-24T00:17:47.413 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:17:47.105+0000 7fc31ea00640 1 -- 172.21.15.118:0/3167778209 <== mds.0 v2:172.21.15.189:6838/4074484168 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x5601049f5d70 con 0x560104a99380 2024-03-24T00:17:47.414 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:17:47.105+0000 7fc31ea00640 10 client.25975 handle_command_reply: tid=0 2024-03-24T00:17:47.414 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:17:47.105+0000 7fc3429ff740 1 client.25975 shutdown 2024-03-24T00:17:47.414 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:17:47.105+0000 7fc3429ff740 1 -- 172.21.15.118:0/3167778209 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x560104a83bd0 msgr2=0x560104a8d960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:17:47.414 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:17:47.105+0000 7fc3429ff740 1 --2- 172.21.15.118:0/3167778209 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x560104a83bd0 0x560104a8d960 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x7fc324009560 tx=0x7fc324004730 comp rx=0 tx=0).stop 2024-03-24T00:17:47.414 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:17:47.105+0000 7fc3429ff740 1 -- 172.21.15.118:0/3167778209 shutdown_connections 2024-03-24T00:17:47.414 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:17:47.105+0000 7fc3429ff740 1 --2- 172.21.15.118:0/3167778209 >> [v2:172.21.15.189:6838/4074484168,v1:172.21.15.189:6839/4074484168] conn(0x560104a99380 0x560104a9b7e0 crc :-1 s=READY pgs=77 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:17:47.414 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:17:47.105+0000 7fc3429ff740 1 --2- 172.21.15.118:0/3167778209 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x560104a88d40 0x560104a94930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:17:47.414 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:17:47.105+0000 7fc3429ff740 1 --2- 172.21.15.118:0/3167778209 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x560104a83bd0 0x560104a8d960 unknown :-1 s=CLOSED pgs=290 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:17:47.414 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:17:47.105+0000 7fc3429ff740 1 --2- 172.21.15.118:0/3167778209 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x560104a831e0 0x560104a8d420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:17:47.414 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:17:47.105+0000 7fc31ea00640 0 client.25975 ms_handle_reset on v2:172.21.15.189:6838/4074484168 2024-03-24T00:17:47.414 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:17:47.105+0000 7fc3429ff740 1 -- 172.21.15.118:0/3167778209 >> 172.21.15.118:0/3167778209 conn(0x560104a7e750 msgr2=0x560104a7d950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:17:47.414 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:17:47.105+0000 7fc3429ff740 1 -- 172.21.15.118:0/3167778209 shutdown_connections 2024-03-24T00:17:47.414 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:17:47.105+0000 7fc3429ff740 1 -- 172.21.15.118:0/3167778209 wait complete. 2024-03-24T00:17:47.415 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:17:47.105+0000 7fc3429ff740 20 client.25975 trim_cache size 0 max 16384 2024-03-24T00:17:47.415 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:17:47.125+0000 7fc3429ff740 1 Processor -- start 2024-03-24T00:17:47.415 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:17:47.125+0000 7fc3429ff740 1 -- start start 2024-03-24T00:17:47.415 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:17:47.126+0000 7fc3429ff740 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x560104a831e0 0x560104a7d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:17:47.415 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:17:47.126+0000 7fc3429ff740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x560104a83bd0 0x560104a948f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:17:47.415 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:17:47.126+0000 7fc3429ff740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x560104a88d40 0x560104a7d600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:17:47.415 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:17:47.126+0000 7fc3429ff740 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc3381aaa90 con 0x560104a831e0 2024-03-24T00:17:47.415 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:17:47.126+0000 7fc3429ff740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc3381ac410 con 0x560104a88d40 2024-03-24T00:17:47.415 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:17:47.126+0000 7fc3429ff740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc3381ab0f0 con 0x560104a83bd0 2024-03-24T00:17:47.415 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:17:47.126+0000 7fc337e00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x560104a83bd0 0x560104a948f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:17:47.415 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:17:47.126+0000 7fc337e00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x560104a83bd0 0x560104a948f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.118:46380/0 (socket says 172.21.15.118:46380) 2024-03-24T00:17:47.415 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:17:47.126+0000 7fc337e00640 1 -- 172.21.15.118:0/4218974405 learned_addr learned my addr 172.21.15.118:0/4218974405 (peer_addr_for_me v2:172.21.15.118:0/0) 2024-03-24T00:17:47.416 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:17:47.126+0000 7fc33d200640 1 --2- 172.21.15.118:0/4218974405 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x560104a831e0 0x560104a7d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:17:47.416 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:17:47.126+0000 7fc33dc00640 1 --2- 172.21.15.118:0/4218974405 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x560104a88d40 0x560104a7d600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:17:47.416 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:17:47.126+0000 7fc337e00640 1 -- 172.21.15.118:0/4218974405 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x560104a88d40 msgr2=0x560104a7d600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:17:47.416 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:17:47.126+0000 7fc337e00640 1 --2- 172.21.15.118:0/4218974405 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x560104a88d40 0x560104a7d600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:17:47.416 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:17:47.126+0000 7fc337e00640 1 -- 172.21.15.118:0/4218974405 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x560104a831e0 msgr2=0x560104a7d0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:17:47.416 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:17:47.126+0000 7fc337e00640 1 --2- 172.21.15.118:0/4218974405 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x560104a831e0 0x560104a7d0c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:17:47.416 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:17:47.126+0000 7fc337e00640 1 -- 172.21.15.118:0/4218974405 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x560104a793a0 con 0x560104a83bd0 2024-03-24T00:17:47.416 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:17:47.127+0000 7fc337e00640 1 --2- 172.21.15.118:0/4218974405 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x560104a83bd0 0x560104a948f0 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x7fc324003e60 tx=0x7fc324020a00 comp rx=0 tx=0).ready entity=mon.1 client_cookie=14a6a929c0bdbafa server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:17:47.416 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:17:47.127+0000 7fc329600640 1 -- 172.21.15.118:0/4218974405 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc324004200 con 0x560104a83bd0 2024-03-24T00:17:47.416 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:17:47.127+0000 7fc329600640 1 -- 172.21.15.118:0/4218974405 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fc324003ad0 con 0x560104a83bd0 2024-03-24T00:17:47.416 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:17:47.127+0000 7fc329600640 1 -- 172.21.15.118:0/4218974405 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc324033be0 con 0x560104a83bd0 2024-03-24T00:17:47.416 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:17:47.128+0000 7fc3429ff740 1 -- 172.21.15.118:0/4218974405 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x560104a83bd0 msgr2=0x560104a948f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:17:47.416 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:17:47.128+0000 7fc3429ff740 1 --2- 172.21.15.118:0/4218974405 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x560104a83bd0 0x560104a948f0 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x7fc324003e60 tx=0x7fc324020a00 comp rx=0 tx=0).stop 2024-03-24T00:17:47.417 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:17:47.128+0000 7fc3429ff740 1 -- 172.21.15.118:0/4218974405 shutdown_connections 2024-03-24T00:17:47.417 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:17:47.128+0000 7fc3429ff740 1 --2- 172.21.15.118:0/4218974405 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x560104a88d40 0x560104a7d600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:17:47.417 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:17:47.128+0000 7fc3429ff740 1 --2- 172.21.15.118:0/4218974405 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x560104a83bd0 0x560104a948f0 unknown :-1 s=CLOSED pgs=291 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:17:47.417 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:17:47.128+0000 7fc3429ff740 1 --2- 172.21.15.118:0/4218974405 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x560104a831e0 0x560104a7d0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:17:47.417 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:17:47.128+0000 7fc3429ff740 1 -- 172.21.15.118:0/4218974405 >> 172.21.15.118:0/4218974405 conn(0x560104a7e750 msgr2=0x560104a95380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:17:47.417 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:17:47.129+0000 7fc3429ff740 1 -- 172.21.15.118:0/4218974405 shutdown_connections 2024-03-24T00:17:47.417 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:17:47.129+0000 7fc3429ff740 1 -- 172.21.15.118:0/4218974405 wait complete. 2024-03-24T00:17:47.417 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:17:47.129+0000 7fc3429ff740 1 -- start start 2024-03-24T00:17:47.417 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:17:47.129+0000 7fc3429ff740 1 Processor -- start 2024-03-24T00:17:47.417 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:17:47.129+0000 7fc3429ff740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x560104a831e0 0x560104a91b90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:17:47.417 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:17:47.129+0000 7fc3429ff740 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x560104a83bd0 0x560104a920d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:17:47.417 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:17:47.130+0000 7fc3429ff740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x560104a88d40 0x560104a93070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:17:47.417 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:17:47.130+0000 7fc3429ff740 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc3381abdb0 con 0x560104a83bd0 2024-03-24T00:17:47.417 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:17:47.130+0000 7fc3429ff740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc3381ab750 con 0x560104a88d40 2024-03-24T00:17:47.418 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:17:47.130+0000 7fc3429ff740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc3381a8c50 con 0x560104a831e0 2024-03-24T00:17:47.418 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:17:47.130+0000 7fc3429ff740 10 client.0 _ll_register_callbacks cb 0x7fc324009ce0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T00:17:47.418 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:17:47.130+0000 7fc33d200640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x560104a831e0 0x560104a91b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:17:47.418 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:17:47.130+0000 7fc33d200640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x560104a831e0 0x560104a91b90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.118:46396/0 (socket says 172.21.15.118:46396) 2024-03-24T00:17:47.418 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:17:47.130+0000 7fc33d200640 1 -- 172.21.15.118:0/3388258968 learned_addr learned my addr 172.21.15.118:0/3388258968 (peer_addr_for_me v2:172.21.15.118:0/0) 2024-03-24T00:17:47.418 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:17:47.130+0000 7fc33dc00640 1 --2- 172.21.15.118:0/3388258968 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x560104a88d40 0x560104a93070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:17:47.418 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:17:47.130+0000 7fc337e00640 1 --2- 172.21.15.118:0/3388258968 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x560104a83bd0 0x560104a920d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:17:47.418 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:17:47.130+0000 7fc33d200640 1 -- 172.21.15.118:0/3388258968 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x560104a88d40 msgr2=0x560104a93070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:17:47.418 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:17:47.130+0000 7fc33d200640 1 --2- 172.21.15.118:0/3388258968 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x560104a88d40 0x560104a93070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:17:47.418 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:17:47.130+0000 7fc33d200640 1 -- 172.21.15.118:0/3388258968 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x560104a83bd0 msgr2=0x560104a920d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:17:47.418 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:17:47.130+0000 7fc33d200640 1 --2- 172.21.15.118:0/3388258968 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x560104a83bd0 0x560104a920d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:17:47.418 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:17:47.130+0000 7fc33d200640 1 -- 172.21.15.118:0/3388258968 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x560104a93660 con 0x560104a831e0 2024-03-24T00:17:47.418 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:17:47.131+0000 7fc33d200640 1 --2- 172.21.15.118:0/3388258968 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x560104a831e0 0x560104a91b90 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 crypto rx=0x7fc32c01c040 tx=0x7fc32c009600 comp rx=0 tx=0).ready entity=mon.1 client_cookie=ef70f7d5857dabf5 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:17:47.419 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:17:47.131+0000 7fc31ea00640 10 client.0 ms_handle_connect on v2:172.21.15.193:3300/0 2024-03-24T00:17:47.419 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:17:47.131+0000 7fc31ea00640 1 -- 172.21.15.118:0/3388258968 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc32c01e040 con 0x560104a831e0 2024-03-24T00:17:47.419 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:17:47.131+0000 7fc31ea00640 1 -- 172.21.15.118:0/3388258968 <== mon.1 v2:172.21.15.193:3300/0 2 ==== osd_map(90..90 src has 1..90) v4 ==== 8277+0+0 (secure 0 0 0) 0x7fc32c02a260 con 0x560104a831e0 2024-03-24T00:17:47.419 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:17:47.131+0000 7fc3429ff740 1 -- 172.21.15.118:0/3388258968 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fc324003ad0 con 0x560104a831e0 2024-03-24T00:17:47.419 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:17:47.131+0000 7fc31ea00640 1 -- 172.21.15.118:0/3388258968 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) v2 ==== 24+0+0 (secure 0 0 0) 0x7fc32c00fa00 con 0x560104a831e0 2024-03-24T00:17:47.419 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:17:47.131+0000 7fc3429ff740 10 client.25981 fetch_fsmap learned FSMap version 28 2024-03-24T00:17:47.419 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:17:47.131+0000 7fc3429ff740 1 -- 172.21.15.118:0/3388258968 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) v3 -- 0x560104a94970 con 0x560104a831e0 2024-03-24T00:17:47.419 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:17:47.132+0000 7fc31ea00640 1 -- 172.21.15.118:0/3388258968 <== mon.1 v2:172.21.15.193:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fc324003ad0 con 0x560104a831e0 2024-03-24T00:17:47.419 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:17:47.132+0000 7fc31ea00640 1 -- 172.21.15.118:0/3388258968 <== mon.1 v2:172.21.15.193:3300/0 5 ==== fsmap(e 28) v1 ==== 7934+0+0 (secure 0 0 0) 0x7fc32c005040 con 0x560104a831e0 2024-03-24T00:17:47.419 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:17:47.132+0000 7fc3429ff740 10 client.25981 fetch_fsmap finished waiting for FSMap version 28 2024-03-24T00:17:47.419 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:17:47.132+0000 7fc3429ff740 10 client.25981 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-24T00:17:47.419 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:17:47.132+0000 7fc3429ff740 20 client.25981 populate_metadata read hostname 'smithi118' 2024-03-24T00:17:47.419 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:17:47.132+0000 7fc3429ff740 1 --2- 172.21.15.118:0/3388258968 >> [v2:172.21.15.189:6838/4074484168,v1:172.21.15.189:6839/4074484168] conn(0x560104af4f30 0x560104af7320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:17:47.419 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:17:47.132+0000 7fc3429ff740 4 client.25981 mds_command: new command op to 24569 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-03-24T00:17:47.420 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:17:47.132+0000 7fc3429ff740 1 -- 172.21.15.118:0/3388258968 --> [v2:172.21.15.189:6838/4074484168,v1:172.21.15.189:6839/4074484168] -- command(tid 0: {"prefix": "flush journal"}) v1 -- 0x5601049f00c0 con 0x560104af4f30 2024-03-24T00:17:47.420 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:17:47.132+0000 7fc337e00640 1 --2- 172.21.15.118:0/3388258968 >> [v2:172.21.15.189:6838/4074484168,v1:172.21.15.189:6839/4074484168] conn(0x560104af4f30 0x560104af7320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:17:47.420 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:17:47.133+0000 7fc31ea00640 10 client.25981 ms_handle_connect on v2:172.21.15.189:6838/4074484168 2024-03-24T00:17:47.420 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:17:47.133+0000 7fc337e00640 1 --2- 172.21.15.118:0/3388258968 >> [v2:172.21.15.189:6838/4074484168,v1:172.21.15.189:6839/4074484168] conn(0x560104af4f30 0x560104af7320 crc :-1 s=READY pgs=78 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=6fc6ea4e19675f72 server_cookie=ab0578883c452bed in_seq=0 out_seq=0 2024-03-24T00:17:47.420 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:17:46 smithi118 ceph-mon[31535]: pgmap v460: 97 pgs: 97 active+clean; 20 MiB data, 408 MiB used, 1.0 TiB / 1.0 TiB avail; 438 KiB/s rd, 1.6 MiB/s wr, 188 op/s 2024-03-24T00:17:47.420 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:17:46 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T00:17:47.420 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:17:46 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T00:17:47.420 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:17:46 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T00:17:47.421 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:17:47.336+0000 7fc31ea00640 1 -- 172.21.15.118:0/3388258968 <== mds.0 v2:172.21.15.189:6838/4074484168 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+44 (crc 0 0 0) 0x5601049f00c0 con 0x560104af4f30 2024-03-24T00:17:47.421 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:17:47.336+0000 7fc31ea00640 10 client.25981 handle_command_reply: tid=0 2024-03-24T00:17:47.421 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:17:47.336+0000 7fc3429ff740 1 client.25981 shutdown 2024-03-24T00:17:47.422 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:17:47.336+0000 7fc3429ff740 1 -- 172.21.15.118:0/3388258968 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x560104a831e0 msgr2=0x560104a91b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:17:47.422 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:17:47.336+0000 7fc3429ff740 1 --2- 172.21.15.118:0/3388258968 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x560104a831e0 0x560104a91b90 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 crypto rx=0x7fc32c01c040 tx=0x7fc32c009600 comp rx=0 tx=0).stop 2024-03-24T00:17:47.422 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:17:47.336+0000 7fc3429ff740 1 -- 172.21.15.118:0/3388258968 shutdown_connections 2024-03-24T00:17:47.422 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:17:47.336+0000 7fc3429ff740 1 --2- 172.21.15.118:0/3388258968 >> [v2:172.21.15.189:6838/4074484168,v1:172.21.15.189:6839/4074484168] conn(0x560104af4f30 0x560104af7320 crc :-1 s=READY pgs=78 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:17:47.422 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:17:47.336+0000 7fc3429ff740 1 --2- 172.21.15.118:0/3388258968 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x560104a88d40 0x560104a93070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:17:47.422 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:17:47.336+0000 7fc31ea00640 0 client.25981 ms_handle_reset on v2:172.21.15.189:6838/4074484168 2024-03-24T00:17:47.422 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:17:47.336+0000 7fc3429ff740 1 --2- 172.21.15.118:0/3388258968 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x560104a83bd0 0x560104a920d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:17:47.422 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:17:47.336+0000 7fc3429ff740 1 --2- 172.21.15.118:0/3388258968 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x560104a831e0 0x560104a91b90 unknown :-1 s=CLOSED pgs=292 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:17:47.422 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:17:47.336+0000 7fc3429ff740 1 -- 172.21.15.118:0/3388258968 >> 172.21.15.118:0/3388258968 conn(0x560104a7e750 msgr2=0x560104a95380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:17:47.422 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:17:47.336+0000 7fc3429ff740 1 -- 172.21.15.118:0/3388258968 shutdown_connections 2024-03-24T00:17:47.422 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:17:47.336+0000 7fc3429ff740 1 -- 172.21.15.118:0/3388258968 wait complete. 2024-03-24T00:17:47.423 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:17:47.337+0000 7fc3429ff740 20 client.25981 trim_cache size 0 max 16384 2024-03-24T00:17:47.423 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stdout:{ 2024-03-24T00:17:47.423 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stdout: "message": "", 2024-03-24T00:17:47.423 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stdout: "return_code": 0 2024-03-24T00:17:47.423 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stdout:} 2024-03-24T00:17:47.424 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:17:47.337+0000 7fc33fc00640 1 -- 172.21.15.118:0/3347686696 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7fc3200a2920 msgr2=0x7fc3200a4de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:17:47.424 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:17:47.337+0000 7fc33fc00640 1 --2- 172.21.15.118:0/3347686696 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7fc3200a2920 0x7fc3200a4de0 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 crypto rx=0x560104a84430 tx=0x7fc324013040 comp rx=0 tx=0).stop 2024-03-24T00:17:47.424 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:17:47.337+0000 7fc33fc00640 1 -- 172.21.15.118:0/3347686696 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fc338101290 msgr2=0x7fc3381a1090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:17:47.424 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:17:47.337+0000 7fc33fc00640 1 --2- 172.21.15.118:0/3347686696 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fc338101290 0x7fc3381a1090 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 crypto rx=0x7fc330037040 tx=0x7fc330032ea0 comp rx=0 tx=0).stop 2024-03-24T00:17:47.424 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:17:47.337+0000 7fc33fc00640 1 -- 172.21.15.118:0/3347686696 shutdown_connections 2024-03-24T00:17:47.424 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:17:47.337+0000 7fc33fc00640 1 --2- 172.21.15.118:0/3347686696 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7fc3200a2920 0x7fc3200a4de0 unknown :-1 s=CLOSED pgs=306 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:17:47.424 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:17:47.337+0000 7fc33fc00640 1 --2- 172.21.15.118:0/3347686696 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fc338101290 0x7fc3381a1090 unknown :-1 s=CLOSED pgs=289 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:17:47.424 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:17:47.337+0000 7fc33fc00640 1 --2- 172.21.15.118:0/3347686696 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc3381008f0 0x7fc33819a020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:17:47.425 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:17:47.337+0000 7fc33fc00640 1 --2- 172.21.15.118:0/3347686696 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fc3380fff90 0x7fc338199ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:17:47.425 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:17:47.337+0000 7fc33fc00640 1 -- 172.21.15.118:0/3347686696 >> 172.21.15.118:0/3347686696 conn(0x7fc3380fbb30 msgr2=0x7fc3380fd630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:17:47.425 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:17:47.337+0000 7fc33fc00640 1 -- 172.21.15.118:0/3347686696 shutdown_connections 2024-03-24T00:17:47.425 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:17:47.338+0000 7fc33fc00640 1 -- 172.21.15.118:0/3347686696 wait complete. 2024-03-24T00:17:48.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:17:47 smithi118 ceph-mon[31535]: pgmap v461: 97 pgs: 97 active+clean; 18 MiB data, 408 MiB used, 1.0 TiB / 1.0 TiB avail; 574 KiB/s rd, 1.7 MiB/s wr, 230 op/s 2024-03-24T00:17:48.203 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:17:47 smithi189 ceph-mon[36187]: pgmap v461: 97 pgs: 97 active+clean; 18 MiB data, 408 MiB used, 1.0 TiB / 1.0 TiB avail; 574 KiB/s rd, 1.7 MiB/s wr, 230 op/s 2024-03-24T00:17:48.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:17:47 smithi193 ceph-mon[36878]: pgmap v461: 97 pgs: 97 active+clean; 18 MiB data, 408 MiB used, 1.0 TiB / 1.0 TiB avail; 574 KiB/s rd, 1.7 MiB/s wr, 230 op/s 2024-03-24T00:17:51.195 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:17:50 smithi118 ceph-mon[31535]: pgmap v462: 97 pgs: 97 active+clean; 14 MiB data, 406 MiB used, 1.0 TiB / 1.0 TiB avail; 576 KiB/s rd, 1.7 MiB/s wr, 251 op/s 2024-03-24T00:17:51.203 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:17:50 smithi189 ceph-mon[36187]: pgmap v462: 97 pgs: 97 active+clean; 14 MiB data, 406 MiB used, 1.0 TiB / 1.0 TiB avail; 576 KiB/s rd, 1.7 MiB/s wr, 251 op/s 2024-03-24T00:17:51.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:17:50 smithi193 ceph-mon[36878]: pgmap v462: 97 pgs: 97 active+clean; 14 MiB data, 406 MiB used, 1.0 TiB / 1.0 TiB avail; 576 KiB/s rd, 1.7 MiB/s wr, 251 op/s 2024-03-24T00:17:53.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:17:52 smithi118 ceph-mon[31535]: pgmap v463: 97 pgs: 97 active+clean; 14 MiB data, 405 MiB used, 1.0 TiB / 1.0 TiB avail; 473 KiB/s rd, 1.6 MiB/s wr, 254 op/s 2024-03-24T00:17:53.203 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:17:52 smithi189 ceph-mon[36187]: pgmap v463: 97 pgs: 97 active+clean; 14 MiB data, 405 MiB used, 1.0 TiB / 1.0 TiB avail; 473 KiB/s rd, 1.6 MiB/s wr, 254 op/s 2024-03-24T00:17:53.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:17:52 smithi193 ceph-mon[36878]: pgmap v463: 97 pgs: 97 active+clean; 14 MiB data, 405 MiB used, 1.0 TiB / 1.0 TiB avail; 473 KiB/s rd, 1.6 MiB/s wr, 254 op/s 2024-03-24T00:17:55.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:17:54 smithi118 ceph-mon[31535]: pgmap v464: 97 pgs: 97 active+clean; 16 MiB data, 406 MiB used, 1.0 TiB / 1.0 TiB avail; 638 KiB/s rd, 1.7 MiB/s wr, 277 op/s 2024-03-24T00:17:55.203 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:17:54 smithi189 ceph-mon[36187]: pgmap v464: 97 pgs: 97 active+clean; 16 MiB data, 406 MiB used, 1.0 TiB / 1.0 TiB avail; 638 KiB/s rd, 1.7 MiB/s wr, 277 op/s 2024-03-24T00:17:55.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:17:54 smithi193 ceph-mon[36878]: pgmap v464: 97 pgs: 97 active+clean; 16 MiB data, 406 MiB used, 1.0 TiB / 1.0 TiB avail; 638 KiB/s rd, 1.7 MiB/s wr, 277 op/s 2024-03-24T00:17:56.834 INFO:tasks.workunit.client.0.smithi118.stdout: 5 27983 70 15037 149 5987 276 2024-03-24T00:17:57.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:17:56 smithi118 ceph-mon[31535]: pgmap v465: 97 pgs: 97 active+clean; 15 MiB data, 404 MiB used, 1.0 TiB / 1.0 TiB avail; 422 KiB/s rd, 909 KiB/s wr, 211 op/s 2024-03-24T00:17:57.203 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:17:56 smithi189 ceph-mon[36187]: pgmap v465: 97 pgs: 97 active+clean; 15 MiB data, 404 MiB used, 1.0 TiB / 1.0 TiB avail; 422 KiB/s rd, 909 KiB/s wr, 211 op/s 2024-03-24T00:17:57.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:17:56 smithi193 ceph-mon[36878]: pgmap v465: 97 pgs: 97 active+clean; 15 MiB data, 404 MiB used, 1.0 TiB / 1.0 TiB avail; 422 KiB/s rd, 909 KiB/s wr, 211 op/s 2024-03-24T00:17:59.195 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:17:58 smithi118 ceph-mon[31535]: pgmap v466: 97 pgs: 97 active+clean; 18 MiB data, 409 MiB used, 1.0 TiB / 1.0 TiB avail; 541 KiB/s rd, 655 KiB/s wr, 205 op/s 2024-03-24T00:17:59.203 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:17:58 smithi189 ceph-mon[36187]: pgmap v466: 97 pgs: 97 active+clean; 18 MiB data, 409 MiB used, 1.0 TiB / 1.0 TiB avail; 541 KiB/s rd, 655 KiB/s wr, 205 op/s 2024-03-24T00:17:59.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:17:58 smithi193 ceph-mon[36878]: pgmap v466: 97 pgs: 97 active+clean; 18 MiB data, 409 MiB used, 1.0 TiB / 1.0 TiB avail; 541 KiB/s rd, 655 KiB/s wr, 205 op/s 2024-03-24T00:17:59.994 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-03-24T00:17:59.994 DEBUG:teuthology.orchestra.run.smithi118:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-24T00:18:00.237 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:00.237+0000 7fdc30c00640 1 -- 172.21.15.118:0/2411934784 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fdc2c1015a0 msgr2=0x7fdc2c1019c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:18:00.237 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:00.237+0000 7fdc30c00640 1 --2- 172.21.15.118:0/2411934784 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fdc2c1015a0 0x7fdc2c1019c0 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x7fdc1c009900 tx=0x7fdc1c02bc30 comp rx=0 tx=0).stop 2024-03-24T00:18:00.237 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:00.237+0000 7fdc30c00640 1 -- 172.21.15.118:0/2411934784 shutdown_connections 2024-03-24T00:18:00.237 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:00.237+0000 7fdc30c00640 1 --2- 172.21.15.118:0/2411934784 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fdc2c103150 0x7fdc2c1099e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:18:00.237 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:00.237+0000 7fdc30c00640 1 --2- 172.21.15.118:0/2411934784 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fdc2c102790 0x7fdc2c102c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:18:00.237 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:00.237+0000 7fdc30c00640 1 --2- 172.21.15.118:0/2411934784 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fdc2c1015a0 0x7fdc2c1019c0 unknown :-1 s=CLOSED pgs=293 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:18:00.237 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:00.237+0000 7fdc30c00640 1 -- 172.21.15.118:0/2411934784 >> 172.21.15.118:0/2411934784 conn(0x7fdc2c0fcb10 msgr2=0x7fdc2c0fef50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:18:00.237 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:00.237+0000 7fdc30c00640 1 -- 172.21.15.118:0/2411934784 shutdown_connections 2024-03-24T00:18:00.237 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:00.237+0000 7fdc30c00640 1 -- 172.21.15.118:0/2411934784 wait complete. 2024-03-24T00:18:00.238 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:00.238+0000 7fdc30c00640 1 Processor -- start 2024-03-24T00:18:00.238 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:00.239+0000 7fdc30c00640 1 -- start start 2024-03-24T00:18:00.239 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:00.239+0000 7fdc30c00640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fdc2c1015a0 0x7fdc2c199b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:18:00.239 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:00.239+0000 7fdc30c00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fdc2c102790 0x7fdc2c19a050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:18:00.239 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:00.239+0000 7fdc30c00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fdc2c103150 0x7fdc2c1a10c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:18:00.239 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:00.239+0000 7fdc30c00640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdc2c10d260 con 0x7fdc2c1015a0 2024-03-24T00:18:00.240 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:00.239+0000 7fdc30c00640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdc2c10d0e0 con 0x7fdc2c102790 2024-03-24T00:18:00.240 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:00.239+0000 7fdc30c00640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdc2c10d3e0 con 0x7fdc2c103150 2024-03-24T00:18:00.240 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:00.239+0000 7fdc2aa00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fdc2c102790 0x7fdc2c19a050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:18:00.240 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:00.239+0000 7fdc2aa00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fdc2c102790 0x7fdc2c19a050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.118:37922/0 (socket says 172.21.15.118:37922) 2024-03-24T00:18:00.240 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:00.239+0000 7fdc2aa00640 1 -- 172.21.15.118:0/2290328322 learned_addr learned my addr 172.21.15.118:0/2290328322 (peer_addr_for_me v2:172.21.15.118:0/0) 2024-03-24T00:18:00.240 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:00.239+0000 7fdc2be00640 1 --2- 172.21.15.118:0/2290328322 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fdc2c103150 0x7fdc2c1a10c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:18:00.240 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:00.239+0000 7fdc2b400640 1 --2- 172.21.15.118:0/2290328322 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fdc2c1015a0 0x7fdc2c199b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:18:00.240 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:00.240+0000 7fdc2be00640 1 -- 172.21.15.118:0/2290328322 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fdc2c102790 msgr2=0x7fdc2c19a050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:18:00.240 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:00.240+0000 7fdc2be00640 1 --2- 172.21.15.118:0/2290328322 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fdc2c102790 0x7fdc2c19a050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:18:00.240 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:00.240+0000 7fdc2be00640 1 -- 172.21.15.118:0/2290328322 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fdc2c1015a0 msgr2=0x7fdc2c199b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:18:00.240 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:00.240+0000 7fdc2be00640 1 --2- 172.21.15.118:0/2290328322 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fdc2c1015a0 0x7fdc2c199b10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:18:00.240 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:00.240+0000 7fdc2be00640 1 -- 172.21.15.118:0/2290328322 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdc2c1a1730 con 0x7fdc2c103150 2024-03-24T00:18:00.240 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:00.240+0000 7fdc2aa00640 1 --2- 172.21.15.118:0/2290328322 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fdc2c102790 0x7fdc2c19a050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T00:18:00.241 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:00.240+0000 7fdc2be00640 1 --2- 172.21.15.118:0/2290328322 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fdc2c103150 0x7fdc2c1a10c0 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 crypto rx=0x7fdc24009fd0 tx=0x7fdc2400ef10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:18:00.241 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:00.240+0000 7fdc23e00640 1 -- 172.21.15.118:0/2290328322 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdc2400ce20 con 0x7fdc2c103150 2024-03-24T00:18:00.241 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:00.240+0000 7fdc23e00640 1 -- 172.21.15.118:0/2290328322 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fdc24005020 con 0x7fdc2c103150 2024-03-24T00:18:00.241 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:00.240+0000 7fdc30c00640 1 -- 172.21.15.118:0/2290328322 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdc2c1a1a20 con 0x7fdc2c103150 2024-03-24T00:18:00.241 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:00.240+0000 7fdc23e00640 1 -- 172.21.15.118:0/2290328322 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdc24010720 con 0x7fdc2c103150 2024-03-24T00:18:00.242 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:00.241+0000 7fdc30c00640 1 -- 172.21.15.118:0/2290328322 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdc2c1a1f70 con 0x7fdc2c103150 2024-03-24T00:18:00.244 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:00.244+0000 7fdc34e4d740 1 Processor -- start 2024-03-24T00:18:00.245 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:00.245+0000 7fdc23e00640 1 -- 172.21.15.118:0/2290328322 <== mon.1 v2:172.21.15.193:3300/0 4 ==== mgrmap(e 20) v1 ==== 137291+0+0 (secure 0 0 0) 0x7fdc24005540 con 0x7fdc2c103150 2024-03-24T00:18:00.245 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:00.246+0000 7fdc34e4d740 1 -- start start 2024-03-24T00:18:00.245 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:00.246+0000 7fdc34e4d740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55577e56e860 0x55577e56ec80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:18:00.246 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:00.246+0000 7fdc34e4d740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55577e56f250 0x55577e573d50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:18:00.246 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:00.246+0000 7fdc34e4d740 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55577e5743c0 0x55577e576850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:18:00.246 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:00.246+0000 7fdc34e4d740 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x55577e4cb250 con 0x55577e5743c0 2024-03-24T00:18:00.246 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:00.246+0000 7fdc34e4d740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x55577e473a30 con 0x55577e56e860 2024-03-24T00:18:00.246 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:00.246+0000 7fdc34e4d740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x55577e4c72f0 con 0x55577e56f250 2024-03-24T00:18:00.247 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:00.247+0000 7fdc2aa00640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55577e5743c0 0x55577e576850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:18:00.247 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:00.247+0000 7fdc2aa00640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55577e5743c0 0x55577e576850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.118:3300/0 says I am v2:172.21.15.118:48488/0 (socket says 172.21.15.118:48488) 2024-03-24T00:18:00.247 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:00.247+0000 7fdc2aa00640 1 -- 172.21.15.118:0/3606015341 learned_addr learned my addr 172.21.15.118:0/3606015341 (peer_addr_for_me v2:172.21.15.118:0/0) 2024-03-24T00:18:00.247 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:00.247+0000 7fdc2aa00640 1 --2- 172.21.15.118:0/3606015341 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55577e56e860 0x55577e56ec80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:18:00.247 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:00.247+0000 7fdc2b400640 1 --2- 172.21.15.118:0/3606015341 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55577e56f250 0x55577e573d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:18:00.248 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:00.247+0000 7fdc2aa00640 1 -- 172.21.15.118:0/3606015341 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55577e56f250 msgr2=0x55577e573d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:18:00.248 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:00.247+0000 7fdc2aa00640 1 --2- 172.21.15.118:0/3606015341 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55577e56f250 0x55577e573d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:18:00.248 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:00.247+0000 7fdc2aa00640 1 -- 172.21.15.118:0/3606015341 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55577e56e860 msgr2=0x55577e56ec80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:18:00.248 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:00.247+0000 7fdc2aa00640 1 --2- 172.21.15.118:0/3606015341 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55577e56e860 0x55577e56ec80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:18:00.248 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:00.247+0000 7fdc2aa00640 1 -- 172.21.15.118:0/3606015341 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55577e45e690 con 0x55577e5743c0 2024-03-24T00:18:00.248 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:00.247+0000 7fdc23e00640 1 --2- 172.21.15.118:0/2290328322 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7fdc100a2880 0x7fdc100a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:18:00.248 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:00.248+0000 7fdc23e00640 1 -- 172.21.15.118:0/2290328322 <== mon.1 v2:172.21.15.193:3300/0 5 ==== osd_map(90..90 src has 1..90) v4 ==== 8277+0+0 (secure 0 0 0) 0x7fdc240cf0d0 con 0x7fdc2c103150 2024-03-24T00:18:00.248 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:00.248+0000 7fdc2aa00640 1 --2- 172.21.15.118:0/3606015341 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55577e5743c0 0x55577e576850 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7fdc18013af0 tx=0x7fdc18005f90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=37c1c912c7e994fc server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:18:00.249 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:00.248+0000 7fdc2b400640 1 --2- 172.21.15.118:0/2290328322 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7fdc100a2880 0x7fdc100a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:18:00.249 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:00.248+0000 7fdc17e00640 1 -- 172.21.15.118:0/3606015341 <== mon.0 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdc1800f9c0 con 0x55577e5743c0 2024-03-24T00:18:00.249 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:00.248+0000 7fdc17e00640 1 -- 172.21.15.118:0/3606015341 <== mon.0 v2:172.21.15.118:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fdc18010430 con 0x55577e5743c0 2024-03-24T00:18:00.249 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:00.248+0000 7fdc17e00640 1 -- 172.21.15.118:0/3606015341 <== mon.0 v2:172.21.15.118:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdc1801e600 con 0x55577e5743c0 2024-03-24T00:18:00.250 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:00.249+0000 7fdc34e4d740 1 -- 172.21.15.118:0/3606015341 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55577e5743c0 msgr2=0x55577e576850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:18:00.250 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:00.249+0000 7fdc2b400640 1 --2- 172.21.15.118:0/2290328322 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7fdc100a2880 0x7fdc100a4d40 secure :-1 s=READY pgs=307 cs=0 l=1 rev1=1 crypto rx=0x7fdc1c02c170 tx=0x7fdc1c02df90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:18:00.250 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:00.249+0000 7fdc34e4d740 1 --2- 172.21.15.118:0/3606015341 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55577e5743c0 0x55577e576850 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7fdc18013af0 tx=0x7fdc18005f90 comp rx=0 tx=0).stop 2024-03-24T00:18:00.250 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:00.249+0000 7fdc34e4d740 1 -- 172.21.15.118:0/3606015341 shutdown_connections 2024-03-24T00:18:00.250 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:00.249+0000 7fdc34e4d740 1 --2- 172.21.15.118:0/3606015341 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55577e5743c0 0x55577e576850 unknown :-1 s=CLOSED pgs=196 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:18:00.250 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:00.249+0000 7fdc34e4d740 1 --2- 172.21.15.118:0/3606015341 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55577e56f250 0x55577e573d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:18:00.250 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:00.249+0000 7fdc34e4d740 1 --2- 172.21.15.118:0/3606015341 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55577e56e860 0x55577e56ec80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:18:00.250 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:00.249+0000 7fdc34e4d740 1 -- 172.21.15.118:0/3606015341 >> 172.21.15.118:0/3606015341 conn(0x55577e52e200 msgr2=0x55577e56a1d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:18:00.250 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:00.250+0000 7fdc34e4d740 1 -- 172.21.15.118:0/3606015341 shutdown_connections 2024-03-24T00:18:00.250 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:00.250+0000 7fdc34e4d740 1 -- 172.21.15.118:0/3606015341 wait complete. 2024-03-24T00:18:00.250 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:00.250+0000 7fdc34e4d740 1 -- start start 2024-03-24T00:18:00.253 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:00.251+0000 7fdc34e4d740 1 Processor -- start 2024-03-24T00:18:00.253 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:00.251+0000 7fdc34e4d740 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55577e56e860 0x55577e578810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:18:00.253 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:00.251+0000 7fdc34e4d740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55577e56f250 0x55577e578d50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:18:00.253 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:00.251+0000 7fdc34e4d740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55577e5743c0 0x55577e57fd90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:18:00.253 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:00.251+0000 7fdc34e4d740 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x55577e577370 con 0x55577e56e860 2024-03-24T00:18:00.254 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:00.251+0000 7fdc34e4d740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x55577e486e80 con 0x55577e5743c0 2024-03-24T00:18:00.254 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:00.251+0000 7fdc34e4d740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x55577e5774f0 con 0x55577e56f250 2024-03-24T00:18:00.254 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:00.251+0000 7fdc34e4d740 10 client.0 _ll_register_callbacks cb 0x55577e499130 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T00:18:00.254 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:00.251+0000 7fdc2be00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55577e5743c0 0x55577e57fd90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:18:00.254 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:00.251+0000 7fdc2aa00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55577e56f250 0x55577e578d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:18:00.254 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:00.251+0000 7fdc2aa00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55577e56f250 0x55577e578d50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.118:52884/0 (socket says 172.21.15.118:52884) 2024-03-24T00:18:00.254 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:00.251+0000 7fdc2aa00640 1 -- 172.21.15.118:0/1093657046 learned_addr learned my addr 172.21.15.118:0/1093657046 (peer_addr_for_me v2:172.21.15.118:0/0) 2024-03-24T00:18:00.254 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:00.251+0000 7fdc2be00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55577e5743c0 0x55577e57fd90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.118:37952/0 (socket says 172.21.15.118:37952) 2024-03-24T00:18:00.254 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:00.251+0000 7fdc2b400640 1 --2- 172.21.15.118:0/1093657046 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55577e56e860 0x55577e578810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:18:00.254 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:00.252+0000 7fdc2aa00640 1 -- 172.21.15.118:0/1093657046 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55577e5743c0 msgr2=0x55577e57fd90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:18:00.254 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:00.252+0000 7fdc2aa00640 1 --2- 172.21.15.118:0/1093657046 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55577e5743c0 0x55577e57fd90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:18:00.254 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:00.252+0000 7fdc2aa00640 1 -- 172.21.15.118:0/1093657046 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55577e56e860 msgr2=0x55577e578810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:18:00.254 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:00.252+0000 7fdc2aa00640 1 --2- 172.21.15.118:0/1093657046 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55577e56e860 0x55577e578810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:18:00.255 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:00.252+0000 7fdc2aa00640 1 -- 172.21.15.118:0/1093657046 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55577e2a97d0 con 0x55577e56f250 2024-03-24T00:18:00.255 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:00.252+0000 7fdc2be00640 1 --2- 172.21.15.118:0/1093657046 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55577e5743c0 0x55577e57fd90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T00:18:00.255 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:00.252+0000 7fdc2aa00640 1 --2- 172.21.15.118:0/1093657046 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55577e56f250 0x55577e578d50 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 crypto rx=0x7fdc1800ff80 tx=0x7fdc1800ffb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=74d7ffec4e68435 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:18:00.255 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:00.252+0000 7fdc14c00640 10 client.0 ms_handle_connect on v2:172.21.15.193:3300/0 2024-03-24T00:18:00.255 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:00.252+0000 7fdc14c00640 1 -- 172.21.15.118:0/1093657046 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdc180109f0 con 0x55577e56f250 2024-03-24T00:18:00.255 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:00.252+0000 7fdc14c00640 1 -- 172.21.15.118:0/1093657046 <== mon.1 v2:172.21.15.193:3300/0 2 ==== osd_map(90..90 src has 1..90) v4 ==== 8277+0+0 (secure 0 0 0) 0x7fdc18005050 con 0x55577e56f250 2024-03-24T00:18:00.255 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:00.252+0000 7fdc34e4d740 1 -- 172.21.15.118:0/1093657046 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fdc18010430 con 0x55577e56f250 2024-03-24T00:18:00.255 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:00.253+0000 7fdc14c00640 1 -- 172.21.15.118:0/1093657046 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) v2 ==== 24+0+0 (secure 0 0 0) 0x7fdc18010b90 con 0x55577e56f250 2024-03-24T00:18:00.255 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:00.253+0000 7fdc34e4d740 10 client.25993 fetch_fsmap learned FSMap version 28 2024-03-24T00:18:00.255 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:00.253+0000 7fdc34e4d740 1 -- 172.21.15.118:0/1093657046 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) v3 -- 0x55577e1df3f0 con 0x55577e56f250 2024-03-24T00:18:00.255 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:00.253+0000 7fdc14c00640 1 -- 172.21.15.118:0/1093657046 <== mon.1 v2:172.21.15.193:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fdc18010430 con 0x55577e56f250 2024-03-24T00:18:00.255 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:00.253+0000 7fdc14c00640 1 -- 172.21.15.118:0/1093657046 <== mon.1 v2:172.21.15.193:3300/0 5 ==== fsmap(e 28) v1 ==== 7934+0+0 (secure 0 0 0) 0x7fdc18031070 con 0x55577e56f250 2024-03-24T00:18:00.255 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:00.253+0000 7fdc34e4d740 10 client.25993 fetch_fsmap finished waiting for FSMap version 28 2024-03-24T00:18:00.255 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:00.253+0000 7fdc34e4d740 10 client.25993 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-24T00:18:00.256 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:00.253+0000 7fdc34e4d740 20 client.25993 populate_metadata read hostname 'smithi118' 2024-03-24T00:18:00.256 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:00.253+0000 7fdc34e4d740 1 --2- 172.21.15.118:0/1093657046 >> [v2:172.21.15.189:6838/4074484168,v1:172.21.15.189:6839/4074484168] conn(0x55577e584430 0x55577e586820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:18:00.256 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:00.253+0000 7fdc34e4d740 4 client.25993 mds_command: new command op to 24569 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-24T00:18:00.256 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:00.253+0000 7fdc34e4d740 1 -- 172.21.15.118:0/1093657046 --> [v2:172.21.15.189:6838/4074484168,v1:172.21.15.189:6839/4074484168] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55577e4d8790 con 0x55577e584430 2024-03-24T00:18:00.256 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:00.254+0000 7fdc2b400640 1 --2- 172.21.15.118:0/1093657046 >> [v2:172.21.15.189:6838/4074484168,v1:172.21.15.189:6839/4074484168] conn(0x55577e584430 0x55577e586820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:18:00.257 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:00.254+0000 7fdc14c00640 10 client.25993 ms_handle_connect on v2:172.21.15.189:6838/4074484168 2024-03-24T00:18:00.257 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:00.254+0000 7fdc2b400640 1 --2- 172.21.15.118:0/1093657046 >> [v2:172.21.15.189:6838/4074484168,v1:172.21.15.189:6839/4074484168] conn(0x55577e584430 0x55577e586820 crc :-1 s=READY pgs=79 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=250266a7ed3a4dd3 server_cookie=51eec1d6dc3667ca in_seq=0 out_seq=0 2024-03-24T00:18:00.257 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:00.255+0000 7fdc14c00640 1 -- 172.21.15.118:0/1093657046 <== mds.0 v2:172.21.15.189:6838/4074484168 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x55577e4d8790 con 0x55577e584430 2024-03-24T00:18:00.257 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:00.255+0000 7fdc14c00640 10 client.25993 handle_command_reply: tid=0 2024-03-24T00:18:00.257 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:00.255+0000 7fdc34e4d740 1 client.25993 shutdown 2024-03-24T00:18:00.257 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:00.255+0000 7fdc34e4d740 1 -- 172.21.15.118:0/1093657046 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55577e56f250 msgr2=0x55577e578d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:18:00.257 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:00.255+0000 7fdc34e4d740 1 --2- 172.21.15.118:0/1093657046 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55577e56f250 0x55577e578d50 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 crypto rx=0x7fdc1800ff80 tx=0x7fdc1800ffb0 comp rx=0 tx=0).stop 2024-03-24T00:18:00.257 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:00.256+0000 7fdc34e4d740 1 -- 172.21.15.118:0/1093657046 shutdown_connections 2024-03-24T00:18:00.258 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:00.256+0000 7fdc34e4d740 1 --2- 172.21.15.118:0/1093657046 >> [v2:172.21.15.189:6838/4074484168,v1:172.21.15.189:6839/4074484168] conn(0x55577e584430 0x55577e586820 crc :-1 s=READY pgs=79 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:18:00.258 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:00.256+0000 7fdc34e4d740 1 --2- 172.21.15.118:0/1093657046 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55577e5743c0 0x55577e57fd90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:18:00.258 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:00.256+0000 7fdc14c00640 0 client.25993 ms_handle_reset on v2:172.21.15.189:6838/4074484168 2024-03-24T00:18:00.258 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:00.256+0000 7fdc34e4d740 1 --2- 172.21.15.118:0/1093657046 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55577e56f250 0x55577e578d50 unknown :-1 s=CLOSED pgs=295 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:18:00.258 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:00.256+0000 7fdc34e4d740 1 --2- 172.21.15.118:0/1093657046 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55577e56e860 0x55577e578810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:18:00.258 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:00.256+0000 7fdc34e4d740 1 -- 172.21.15.118:0/1093657046 >> 172.21.15.118:0/1093657046 conn(0x55577e52e200 msgr2=0x55577e560bc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:18:00.258 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:00.256+0000 7fdc34e4d740 1 -- 172.21.15.118:0/1093657046 shutdown_connections 2024-03-24T00:18:00.258 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:00.256+0000 7fdc34e4d740 1 -- 172.21.15.118:0/1093657046 wait complete. 2024-03-24T00:18:00.258 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:00.256+0000 7fdc34e4d740 20 client.25993 trim_cache size 0 max 16384 2024-03-24T00:18:00.276 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:00.276+0000 7fdc34e4d740 1 Processor -- start 2024-03-24T00:18:00.276 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:00.277+0000 7fdc34e4d740 1 -- start start 2024-03-24T00:18:00.276 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:00.277+0000 7fdc34e4d740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55577e56e860 0x55577e578210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:18:00.276 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:00.277+0000 7fdc34e4d740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55577e56f250 0x55577e57fd50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:18:00.277 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:00.277+0000 7fdc34e4d740 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55577e5743c0 0x55577e56d410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:18:00.277 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:00.277+0000 7fdc34e4d740 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdc2c1a66e0 con 0x55577e5743c0 2024-03-24T00:18:00.277 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:00.277+0000 7fdc34e4d740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdc2c1a8060 con 0x55577e56e860 2024-03-24T00:18:00.277 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:00.277+0000 7fdc34e4d740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdc2c1a6d40 con 0x55577e56f250 2024-03-24T00:18:00.278 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:00.277+0000 7fdc2aa00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55577e56f250 0x55577e57fd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:18:00.278 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:00.277+0000 7fdc2b400640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55577e56e860 0x55577e578210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:18:00.278 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:00.277+0000 7fdc2aa00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55577e56f250 0x55577e57fd50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.118:52890/0 (socket says 172.21.15.118:52890) 2024-03-24T00:18:00.278 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:00.277+0000 7fdc2b400640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55577e56e860 0x55577e578210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.118:37954/0 (socket says 172.21.15.118:37954) 2024-03-24T00:18:00.278 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:00.277+0000 7fdc2aa00640 1 -- 172.21.15.118:0/696774735 learned_addr learned my addr 172.21.15.118:0/696774735 (peer_addr_for_me v2:172.21.15.118:0/0) 2024-03-24T00:18:00.278 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:00.277+0000 7fdc2be00640 1 --2- 172.21.15.118:0/696774735 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55577e5743c0 0x55577e56d410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:18:00.278 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:00.277+0000 7fdc2aa00640 1 -- 172.21.15.118:0/696774735 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55577e56e860 msgr2=0x55577e578210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:18:00.278 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:00.277+0000 7fdc2aa00640 1 --2- 172.21.15.118:0/696774735 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55577e56e860 0x55577e578210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:18:00.278 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:00.277+0000 7fdc2aa00640 1 -- 172.21.15.118:0/696774735 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55577e5743c0 msgr2=0x55577e56d410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:18:00.278 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:00.277+0000 7fdc2aa00640 1 --2- 172.21.15.118:0/696774735 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55577e5743c0 0x55577e56d410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:18:00.278 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:00.277+0000 7fdc2aa00640 1 -- 172.21.15.118:0/696774735 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55577e19f0f0 con 0x55577e56f250 2024-03-24T00:18:00.279 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:00.278+0000 7fdc2aa00640 1 --2- 172.21.15.118:0/696774735 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55577e56f250 0x55577e57fd50 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x7fdc18009c50 tx=0x7fdc18005970 comp rx=0 tx=0).ready entity=mon.1 client_cookie=991e7d505cf2d360 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:18:00.279 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:00.278+0000 7fdc17400640 1 -- 172.21.15.118:0/696774735 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdc18038070 con 0x55577e56f250 2024-03-24T00:18:00.279 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:00.278+0000 7fdc17400640 1 -- 172.21.15.118:0/696774735 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fdc1802e070 con 0x55577e56f250 2024-03-24T00:18:00.279 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:00.278+0000 7fdc17400640 1 -- 172.21.15.118:0/696774735 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdc18028820 con 0x55577e56f250 2024-03-24T00:18:00.282 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:00.279+0000 7fdc34e4d740 1 -- 172.21.15.118:0/696774735 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55577e56f250 msgr2=0x55577e57fd50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:18:00.282 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:00.279+0000 7fdc34e4d740 1 --2- 172.21.15.118:0/696774735 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55577e56f250 0x55577e57fd50 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x7fdc18009c50 tx=0x7fdc18005970 comp rx=0 tx=0).stop 2024-03-24T00:18:00.282 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:00.279+0000 7fdc34e4d740 1 -- 172.21.15.118:0/696774735 shutdown_connections 2024-03-24T00:18:00.282 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:00.279+0000 7fdc34e4d740 1 --2- 172.21.15.118:0/696774735 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55577e5743c0 0x55577e56d410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:18:00.282 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:00.279+0000 7fdc34e4d740 1 --2- 172.21.15.118:0/696774735 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55577e56f250 0x55577e57fd50 unknown :-1 s=CLOSED pgs=296 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:18:00.282 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:00.279+0000 7fdc34e4d740 1 --2- 172.21.15.118:0/696774735 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55577e56e860 0x55577e578210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:18:00.282 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:00.279+0000 7fdc34e4d740 1 -- 172.21.15.118:0/696774735 >> 172.21.15.118:0/696774735 conn(0x55577e52e200 msgr2=0x55577e577b00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:18:00.282 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:00.279+0000 7fdc34e4d740 1 -- 172.21.15.118:0/696774735 shutdown_connections 2024-03-24T00:18:00.282 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:00.279+0000 7fdc34e4d740 1 -- 172.21.15.118:0/696774735 wait complete. 2024-03-24T00:18:00.282 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:00.279+0000 7fdc34e4d740 1 -- start start 2024-03-24T00:18:00.282 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:00.280+0000 7fdc34e4d740 1 Processor -- start 2024-03-24T00:18:00.282 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:00.280+0000 7fdc34e4d740 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55577e56e860 0x55577e57cd50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:18:00.283 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:00.280+0000 7fdc34e4d740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55577e56f250 0x55577e57d290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:18:00.283 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:00.280+0000 7fdc34e4d740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55577e5743c0 0x55577e5ddcc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:18:00.283 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:00.280+0000 7fdc34e4d740 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdc2c1a7a00 con 0x55577e56e860 2024-03-24T00:18:00.283 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:00.280+0000 7fdc34e4d740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdc2c1a73a0 con 0x55577e56f250 2024-03-24T00:18:00.283 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:00.280+0000 7fdc34e4d740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdc2c1a48e0 con 0x55577e5743c0 2024-03-24T00:18:00.283 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:00.280+0000 7fdc34e4d740 10 client.0 _ll_register_callbacks cb 0x7fdc180139d0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T00:18:00.283 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:00.280+0000 7fdc2aa00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55577e56f250 0x55577e57d290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:18:00.283 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:00.280+0000 7fdc2aa00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55577e56f250 0x55577e57d290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.118:37968/0 (socket says 172.21.15.118:37968) 2024-03-24T00:18:00.283 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:00.280+0000 7fdc2aa00640 1 -- 172.21.15.118:0/2966595273 learned_addr learned my addr 172.21.15.118:0/2966595273 (peer_addr_for_me v2:172.21.15.118:0/0) 2024-03-24T00:18:00.283 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:00.280+0000 7fdc2be00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55577e5743c0 0x55577e5ddcc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:18:00.283 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:00.280+0000 7fdc2b400640 1 --2- 172.21.15.118:0/2966595273 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55577e56e860 0x55577e57cd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:18:00.283 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:00.281+0000 7fdc2aa00640 1 -- 172.21.15.118:0/2966595273 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55577e5743c0 msgr2=0x55577e5ddcc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:18:00.283 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:00.281+0000 7fdc2aa00640 1 --2- 172.21.15.118:0/2966595273 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55577e5743c0 0x55577e5ddcc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:18:00.283 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:00.281+0000 7fdc2aa00640 1 -- 172.21.15.118:0/2966595273 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55577e56e860 msgr2=0x55577e57cd50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:18:00.284 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:00.281+0000 7fdc2aa00640 1 --2- 172.21.15.118:0/2966595273 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55577e56e860 0x55577e57cd50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:18:00.284 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:00.281+0000 7fdc2aa00640 1 -- 172.21.15.118:0/2966595273 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55577e5de280 con 0x55577e56f250 2024-03-24T00:18:00.284 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:00.281+0000 7fdc2be00640 1 --2- 172.21.15.118:0/2966595273 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55577e5743c0 0x55577e5ddcc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T00:18:00.284 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:00.281+0000 7fdc2aa00640 1 --2- 172.21.15.118:0/2966595273 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55577e56f250 0x55577e57d290 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 crypto rx=0x7fdc18013af0 tx=0x7fdc18001e50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d80a5028ba0394f2 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:18:00.284 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:00.281+0000 7fdc14c00640 10 client.0 ms_handle_connect on v2:172.21.15.189:3300/0 2024-03-24T00:18:00.284 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:00.281+0000 7fdc14c00640 1 -- 172.21.15.118:0/2966595273 <== mon.2 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdc18038070 con 0x55577e56f250 2024-03-24T00:18:00.284 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:00.281+0000 7fdc34e4d740 1 -- 172.21.15.118:0/2966595273 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fdc1802e070 con 0x55577e56f250 2024-03-24T00:18:00.284 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:00.281+0000 7fdc14c00640 1 -- 172.21.15.118:0/2966595273 <== mon.2 v2:172.21.15.189:3300/0 2 ==== osd_map(90..90 src has 1..90) v4 ==== 8277+0+0 (secure 0 0 0) 0x7fdc18004890 con 0x55577e56f250 2024-03-24T00:18:00.284 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:00.282+0000 7fdc14c00640 1 -- 172.21.15.118:0/2966595273 <== mon.2 v2:172.21.15.189:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) v2 ==== 24+0+0 (secure 0 0 0) 0x7fdc18010430 con 0x55577e56f250 2024-03-24T00:18:00.284 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:00.282+0000 7fdc34e4d740 10 client.26012 fetch_fsmap learned FSMap version 28 2024-03-24T00:18:00.284 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:00.282+0000 7fdc34e4d740 1 -- 172.21.15.118:0/2966595273 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) v3 -- 0x55577e5df5c0 con 0x55577e56f250 2024-03-24T00:18:00.284 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:00.282+0000 7fdc14c00640 1 -- 172.21.15.118:0/2966595273 <== mon.2 v2:172.21.15.189:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fdc1802e070 con 0x55577e56f250 2024-03-24T00:18:00.284 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:00.282+0000 7fdc14c00640 1 -- 172.21.15.118:0/2966595273 <== mon.2 v2:172.21.15.189:3300/0 5 ==== fsmap(e 28) v1 ==== 7934+0+0 (secure 0 0 0) 0x7fdc18031070 con 0x55577e56f250 2024-03-24T00:18:00.284 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:00.282+0000 7fdc34e4d740 10 client.26012 fetch_fsmap finished waiting for FSMap version 28 2024-03-24T00:18:00.284 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:00.282+0000 7fdc34e4d740 10 client.26012 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-24T00:18:00.285 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:00.282+0000 7fdc34e4d740 20 client.26012 populate_metadata read hostname 'smithi118' 2024-03-24T00:18:00.285 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:00.282+0000 7fdc34e4d740 1 --2- 172.21.15.118:0/2966595273 >> [v2:172.21.15.189:6838/4074484168,v1:172.21.15.189:6839/4074484168] conn(0x55577e5dfec0 0x55577e5e22e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:18:00.285 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:00.282+0000 7fdc34e4d740 4 client.26012 mds_command: new command op to 24569 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-24T00:18:00.285 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:00.282+0000 7fdc34e4d740 1 -- 172.21.15.118:0/2966595273 --> [v2:172.21.15.189:6838/4074484168,v1:172.21.15.189:6839/4074484168] -- command(tid 0: {"prefix": "scrub status"}) v1 -- 0x55577e4d2ae0 con 0x55577e5dfec0 2024-03-24T00:18:00.285 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:00.283+0000 7fdc2b400640 1 --2- 172.21.15.118:0/2966595273 >> [v2:172.21.15.189:6838/4074484168,v1:172.21.15.189:6839/4074484168] conn(0x55577e5dfec0 0x55577e5e22e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:18:00.285 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:00.283+0000 7fdc2b400640 1 --2- 172.21.15.118:0/2966595273 >> [v2:172.21.15.189:6838/4074484168,v1:172.21.15.189:6839/4074484168] conn(0x55577e5dfec0 0x55577e5e22e0 crc :-1 s=READY pgs=80 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=3208d26c7a3b0f7c server_cookie=8da7c04c49e71876 in_seq=0 out_seq=0 2024-03-24T00:18:00.285 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:00.283+0000 7fdc14c00640 10 client.26012 ms_handle_connect on v2:172.21.15.189:6838/4074484168 2024-03-24T00:18:00.290 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:00.290+0000 7fdc14c00640 1 -- 172.21.15.118:0/2966595273 <== mds.0 v2:172.21.15.189:6838/4074484168 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+63 (crc 0 0 0) 0x55577e4d2ae0 con 0x55577e5dfec0 2024-03-24T00:18:00.290 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:00.290+0000 7fdc14c00640 10 client.26012 handle_command_reply: tid=0 2024-03-24T00:18:00.290 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:00.290+0000 7fdc34e4d740 1 client.26012 shutdown 2024-03-24T00:18:00.290 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:00.290+0000 7fdc34e4d740 1 -- 172.21.15.118:0/2966595273 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55577e56f250 msgr2=0x55577e57d290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:18:00.290 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:00.290+0000 7fdc34e4d740 1 --2- 172.21.15.118:0/2966595273 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55577e56f250 0x55577e57d290 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 crypto rx=0x7fdc18013af0 tx=0x7fdc18001e50 comp rx=0 tx=0).stop 2024-03-24T00:18:00.290 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:00.290+0000 7fdc34e4d740 1 -- 172.21.15.118:0/2966595273 shutdown_connections 2024-03-24T00:18:00.290 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:00.290+0000 7fdc34e4d740 1 --2- 172.21.15.118:0/2966595273 >> [v2:172.21.15.189:6838/4074484168,v1:172.21.15.189:6839/4074484168] conn(0x55577e5dfec0 0x55577e5e22e0 crc :-1 s=READY pgs=80 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:18:00.291 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:00.290+0000 7fdc34e4d740 1 --2- 172.21.15.118:0/2966595273 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55577e5743c0 0x55577e5ddcc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:18:00.291 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:00.290+0000 7fdc34e4d740 1 --2- 172.21.15.118:0/2966595273 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55577e56f250 0x55577e57d290 unknown :-1 s=CLOSED pgs=309 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:18:00.291 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:00.290+0000 7fdc14c00640 0 client.26012 ms_handle_reset on v2:172.21.15.189:6838/4074484168 2024-03-24T00:18:00.291 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:00.290+0000 7fdc34e4d740 1 --2- 172.21.15.118:0/2966595273 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55577e56e860 0x55577e57cd50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:18:00.291 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:00.290+0000 7fdc34e4d740 1 -- 172.21.15.118:0/2966595273 >> 172.21.15.118:0/2966595273 conn(0x55577e52e200 msgr2=0x55577e560330 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:18:00.291 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:00.290+0000 7fdc34e4d740 1 -- 172.21.15.118:0/2966595273 shutdown_connections 2024-03-24T00:18:00.291 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:00.290+0000 7fdc34e4d740 1 -- 172.21.15.118:0/2966595273 wait complete. 2024-03-24T00:18:00.291 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:00.290+0000 7fdc34e4d740 20 client.26012 trim_cache size 0 max 16384 2024-03-24T00:18:00.291 INFO:teuthology.orchestra.run.smithi118.stdout:{ 2024-03-24T00:18:00.291 INFO:teuthology.orchestra.run.smithi118.stdout: "status": "no active scrubs running", 2024-03-24T00:18:00.291 INFO:teuthology.orchestra.run.smithi118.stdout: "scrubs": {} 2024-03-24T00:18:00.292 INFO:teuthology.orchestra.run.smithi118.stdout:} 2024-03-24T00:18:00.292 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:00.291+0000 7fdc30c00640 1 -- 172.21.15.118:0/2290328322 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7fdc100a2880 msgr2=0x7fdc100a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:18:00.292 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:00.291+0000 7fdc30c00640 1 --2- 172.21.15.118:0/2290328322 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7fdc100a2880 0x7fdc100a4d40 secure :-1 s=READY pgs=307 cs=0 l=1 rev1=1 crypto rx=0x7fdc1c02c170 tx=0x7fdc1c02df90 comp rx=0 tx=0).stop 2024-03-24T00:18:00.292 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:00.291+0000 7fdc30c00640 1 -- 172.21.15.118:0/2290328322 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fdc2c103150 msgr2=0x7fdc2c1a10c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:18:00.293 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:00.291+0000 7fdc30c00640 1 --2- 172.21.15.118:0/2290328322 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fdc2c103150 0x7fdc2c1a10c0 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 crypto rx=0x7fdc24009fd0 tx=0x7fdc2400ef10 comp rx=0 tx=0).stop 2024-03-24T00:18:00.293 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:00.291+0000 7fdc30c00640 1 -- 172.21.15.118:0/2290328322 shutdown_connections 2024-03-24T00:18:00.293 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:00.291+0000 7fdc30c00640 1 --2- 172.21.15.118:0/2290328322 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7fdc100a2880 0x7fdc100a4d40 unknown :-1 s=CLOSED pgs=307 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:18:00.293 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:00.291+0000 7fdc30c00640 1 --2- 172.21.15.118:0/2290328322 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fdc2c103150 0x7fdc2c1a10c0 unknown :-1 s=CLOSED pgs=294 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:18:00.293 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:00.291+0000 7fdc30c00640 1 --2- 172.21.15.118:0/2290328322 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fdc2c102790 0x7fdc2c19a050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:18:00.293 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:00.291+0000 7fdc30c00640 1 --2- 172.21.15.118:0/2290328322 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fdc2c1015a0 0x7fdc2c199b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:18:00.293 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:00.291+0000 7fdc30c00640 1 -- 172.21.15.118:0/2290328322 >> 172.21.15.118:0/2290328322 conn(0x7fdc2c0fcb10 msgr2=0x7fdc2c0738d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:18:00.293 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:00.291+0000 7fdc30c00640 1 -- 172.21.15.118:0/2290328322 shutdown_connections 2024-03-24T00:18:00.293 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:00.291+0000 7fdc30c00640 1 -- 172.21.15.118:0/2290328322 wait complete. 2024-03-24T00:18:00.310 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-03-24T00:18:00.310 DEBUG:teuthology.orchestra.run.smithi118:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-24T00:18:00.486 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:18:00.486+0000 7fe7ee600640 1 -- 172.21.15.118:0/894926656 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fe7e81032a0 msgr2=0x7fe7e8109b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:18:00.487 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:18:00.486+0000 7fe7ee600640 1 --2- 172.21.15.118:0/894926656 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fe7e81032a0 0x7fe7e8109b30 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0x7fe7e0009ca0 tx=0x7fe7e002c100 comp rx=0 tx=0).stop 2024-03-24T00:18:00.487 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:18:00.486+0000 7fe7ee600640 1 -- 172.21.15.118:0/894926656 shutdown_connections 2024-03-24T00:18:00.487 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:18:00.486+0000 7fe7ee600640 1 --2- 172.21.15.118:0/894926656 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fe7e81032a0 0x7fe7e8109b30 unknown :-1 s=CLOSED pgs=297 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:18:00.487 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:18:00.487+0000 7fe7ee600640 1 --2- 172.21.15.118:0/894926656 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe7e81028e0 0x7fe7e8102d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:18:00.487 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:18:00.487+0000 7fe7ee600640 1 --2- 172.21.15.118:0/894926656 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fe7e81016f0 0x7fe7e8101b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:18:00.487 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:18:00.487+0000 7fe7ee600640 1 -- 172.21.15.118:0/894926656 >> 172.21.15.118:0/894926656 conn(0x7fe7e80fcc60 msgr2=0x7fe7e80ff0a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:18:00.487 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:18:00.487+0000 7fe7ee600640 1 -- 172.21.15.118:0/894926656 shutdown_connections 2024-03-24T00:18:00.487 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:18:00.487+0000 7fe7ee600640 1 -- 172.21.15.118:0/894926656 wait complete. 2024-03-24T00:18:00.489 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:18:00.488+0000 7fe7ee600640 1 Processor -- start 2024-03-24T00:18:00.489 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:18:00.488+0000 7fe7ee600640 1 -- start start 2024-03-24T00:18:00.489 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:18:00.488+0000 7fe7ee600640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fe7e81016f0 0x7fe7e8197930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:18:00.489 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:18:00.488+0000 7fe7ee600640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe7e81028e0 0x7fe7e8197e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:18:00.489 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:18:00.488+0000 7fe7ee600640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fe7e81032a0 0x7fe7e819eee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:18:00.489 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:18:00.488+0000 7fe7ee600640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe7e810d350 con 0x7fe7e81016f0 2024-03-24T00:18:00.489 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:18:00.488+0000 7fe7ee600640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe7e810d1d0 con 0x7fe7e81028e0 2024-03-24T00:18:00.489 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:18:00.488+0000 7fe7ee600640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe7e810d4d0 con 0x7fe7e81032a0 2024-03-24T00:18:00.490 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:18:00.488+0000 7fe7e6a00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe7e81028e0 0x7fe7e8197e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:18:00.490 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:18:00.488+0000 7fe7e6a00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe7e81028e0 0x7fe7e8197e70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.118:37988/0 (socket says 172.21.15.118:37988) 2024-03-24T00:18:00.490 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:18:00.488+0000 7fe7e6a00640 1 -- 172.21.15.118:0/2200287762 learned_addr learned my addr 172.21.15.118:0/2200287762 (peer_addr_for_me v2:172.21.15.118:0/0) 2024-03-24T00:18:00.490 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:18:00.489+0000 7fe7e7e00640 1 --2- 172.21.15.118:0/2200287762 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fe7e81032a0 0x7fe7e819eee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:18:00.490 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:18:00.489+0000 7fe7e7400640 1 --2- 172.21.15.118:0/2200287762 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fe7e81016f0 0x7fe7e8197930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:18:00.490 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:18:00.489+0000 7fe7e6a00640 1 -- 172.21.15.118:0/2200287762 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fe7e81032a0 msgr2=0x7fe7e819eee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:18:00.490 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:18:00.489+0000 7fe7e6a00640 1 --2- 172.21.15.118:0/2200287762 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fe7e81032a0 0x7fe7e819eee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:18:00.490 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:18:00.489+0000 7fe7e6a00640 1 -- 172.21.15.118:0/2200287762 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fe7e81016f0 msgr2=0x7fe7e8197930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:18:00.490 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:18:00.489+0000 7fe7e6a00640 1 --2- 172.21.15.118:0/2200287762 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fe7e81016f0 0x7fe7e8197930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:18:00.490 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:18:00.489+0000 7fe7e6a00640 1 -- 172.21.15.118:0/2200287762 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe7e819f550 con 0x7fe7e81028e0 2024-03-24T00:18:00.490 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:18:00.489+0000 7fe7e7400640 1 --2- 172.21.15.118:0/2200287762 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fe7e81016f0 0x7fe7e8197930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T00:18:00.490 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:18:00.489+0000 7fe7e6a00640 1 --2- 172.21.15.118:0/2200287762 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe7e81028e0 0x7fe7e8197e70 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 crypto rx=0x7fe7d400e9f0 tx=0x7fe7d400eeb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:18:00.490 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:18:00.490+0000 7fe7dfe00640 1 -- 172.21.15.118:0/2200287762 <== mon.2 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe7d400cd80 con 0x7fe7e81028e0 2024-03-24T00:18:00.490 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:18:00.490+0000 7fe7ee600640 1 -- 172.21.15.118:0/2200287762 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe7e819f840 con 0x7fe7e81028e0 2024-03-24T00:18:00.491 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:18:00.490+0000 7fe7dfe00640 1 -- 172.21.15.118:0/2200287762 <== mon.2 v2:172.21.15.189:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fe7d4010910 con 0x7fe7e81028e0 2024-03-24T00:18:00.491 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:18:00.490+0000 7fe7dfe00640 1 -- 172.21.15.118:0/2200287762 <== mon.2 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe7d40188f0 con 0x7fe7e81028e0 2024-03-24T00:18:00.491 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:18:00.490+0000 7fe7ee600640 1 -- 172.21.15.118:0/2200287762 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe7e819fde0 con 0x7fe7e81028e0 2024-03-24T00:18:00.496 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:18:00.493+0000 7fe7f13ed740 1 Processor -- start 2024-03-24T00:18:00.496 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:18:00.494+0000 7fe7f13ed740 1 -- start start 2024-03-24T00:18:00.496 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:18:00.494+0000 7fe7f13ed740 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5628140ce4e0 0x5628140ce900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:18:00.496 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:18:00.494+0000 7fe7f13ed740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5628140ceed0 0x5628140d19c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:18:00.496 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:18:00.494+0000 7fe7f13ed740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5628140d2030 0x5628140e3b00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:18:00.496 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:18:00.494+0000 7fe7f13ed740 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x5628140963b0 con 0x5628140ce4e0 2024-03-24T00:18:00.496 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:18:00.494+0000 7fe7f13ed740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x562814038830 con 0x5628140d2030 2024-03-24T00:18:00.496 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:18:00.494+0000 7fe7f13ed740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x562813fe15d0 con 0x5628140ceed0 2024-03-24T00:18:00.496 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:18:00.494+0000 7fe7e7400640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5628140ceed0 0x5628140d19c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:18:00.496 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:18:00.494+0000 7fe7e7400640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5628140ceed0 0x5628140d19c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.118:52930/0 (socket says 172.21.15.118:52930) 2024-03-24T00:18:00.496 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:18:00.494+0000 7fe7e7400640 1 -- 172.21.15.118:0/3632150488 learned_addr learned my addr 172.21.15.118:0/3632150488 (peer_addr_for_me v2:172.21.15.118:0/0) 2024-03-24T00:18:00.496 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:18:00.494+0000 7fe7e7e00640 1 --2- 172.21.15.118:0/3632150488 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5628140ce4e0 0x5628140ce900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:18:00.496 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:18:00.495+0000 7fe7e7400640 1 -- 172.21.15.118:0/3632150488 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5628140d2030 msgr2=0x5628140e3b00 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-24T00:18:00.497 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:18:00.495+0000 7fe7e7400640 1 --2- 172.21.15.118:0/3632150488 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5628140d2030 0x5628140e3b00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:18:00.497 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:18:00.495+0000 7fe7e7400640 1 -- 172.21.15.118:0/3632150488 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5628140ce4e0 msgr2=0x5628140ce900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:18:00.497 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:18:00.495+0000 7fe7e7400640 1 --2- 172.21.15.118:0/3632150488 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5628140ce4e0 0x5628140ce900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:18:00.497 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:18:00.495+0000 7fe7e7400640 1 -- 172.21.15.118:0/3632150488 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x562813ff92c0 con 0x5628140ceed0 2024-03-24T00:18:00.497 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:18:00.495+0000 7fe7e7400640 1 --2- 172.21.15.118:0/3632150488 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5628140ceed0 0x5628140d19c0 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 crypto rx=0x7fe7d8009cd0 tx=0x7fe7d80117b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=88a565ed22fa1537 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:18:00.497 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:18:00.495+0000 7fe7dcc00640 1 -- 172.21.15.118:0/3632150488 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe7d8005020 con 0x5628140ceed0 2024-03-24T00:18:00.497 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:18:00.495+0000 7fe7dfe00640 1 -- 172.21.15.118:0/2200287762 <== mon.2 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 20) v1 ==== 137291+0+0 (secure 0 0 0) 0x7fe7d4010ac0 con 0x7fe7e81028e0 2024-03-24T00:18:00.497 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:18:00.496+0000 7fe7dcc00640 1 -- 172.21.15.118:0/3632150488 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fe7d80047a0 con 0x5628140ceed0 2024-03-24T00:18:00.497 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:18:00.496+0000 7fe7dcc00640 1 -- 172.21.15.118:0/3632150488 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe7d8012850 con 0x5628140ceed0 2024-03-24T00:18:00.498 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:18:00.497+0000 7fe7f13ed740 1 -- 172.21.15.118:0/3632150488 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5628140ceed0 msgr2=0x5628140d19c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:18:00.498 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:18:00.497+0000 7fe7f13ed740 1 --2- 172.21.15.118:0/3632150488 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5628140ceed0 0x5628140d19c0 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 crypto rx=0x7fe7d8009cd0 tx=0x7fe7d80117b0 comp rx=0 tx=0).stop 2024-03-24T00:18:00.498 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:18:00.497+0000 7fe7dfe00640 1 --2- 172.21.15.118:0/2200287762 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7fe7cc0a2920 0x7fe7cc0a4de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:18:00.498 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:18:00.497+0000 7fe7dfe00640 1 -- 172.21.15.118:0/2200287762 <== mon.2 v2:172.21.15.189:3300/0 5 ==== osd_map(90..90 src has 1..90) v4 ==== 8277+0+0 (secure 0 0 0) 0x7fe7d40ceea0 con 0x7fe7e81028e0 2024-03-24T00:18:00.498 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:18:00.498+0000 7fe7e7e00640 1 --2- 172.21.15.118:0/2200287762 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7fe7cc0a2920 0x7fe7cc0a4de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:18:00.498 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:18:00.498+0000 7fe7f13ed740 1 -- 172.21.15.118:0/3632150488 shutdown_connections 2024-03-24T00:18:00.498 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:18:00.498+0000 7fe7f13ed740 1 --2- 172.21.15.118:0/3632150488 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5628140d2030 0x5628140e3b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:18:00.498 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:18:00.498+0000 7fe7f13ed740 1 --2- 172.21.15.118:0/3632150488 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5628140ceed0 0x5628140d19c0 unknown :-1 s=CLOSED pgs=298 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:18:00.498 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:18:00.498+0000 7fe7f13ed740 1 --2- 172.21.15.118:0/3632150488 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5628140ce4e0 0x5628140ce900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:18:00.499 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:18:00.498+0000 7fe7f13ed740 1 -- 172.21.15.118:0/3632150488 >> 172.21.15.118:0/3632150488 conn(0x5628140d2b60 msgr2=0x5628140d2f60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:18:00.499 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:18:00.499+0000 7fe7f13ed740 1 -- 172.21.15.118:0/3632150488 shutdown_connections 2024-03-24T00:18:00.499 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:18:00.499+0000 7fe7f13ed740 1 -- 172.21.15.118:0/3632150488 wait complete. 2024-03-24T00:18:00.499 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:18:00.499+0000 7fe7e7e00640 1 --2- 172.21.15.118:0/2200287762 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7fe7cc0a2920 0x7fe7cc0a4de0 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 crypto rx=0x7fe7e002c640 tx=0x7fe7e002e040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:18:00.499 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:18:00.500+0000 7fe7f13ed740 1 -- start start 2024-03-24T00:18:00.500 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:18:00.500+0000 7fe7f13ed740 1 Processor -- start 2024-03-24T00:18:00.501 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:18:00.501+0000 7fe7f13ed740 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5628140ce4e0 0x5628140dd2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:18:00.501 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:18:00.501+0000 7fe7f13ed740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5628140ceed0 0x5628140dd820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:18:00.501 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:18:00.501+0000 7fe7f13ed740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5628140d2030 0x5628140e2850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:18:00.502 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:18:00.501+0000 7fe7f13ed740 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x562813ff4a20 con 0x5628140ce4e0 2024-03-24T00:18:00.502 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:18:00.501+0000 7fe7f13ed740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x5628140348d0 con 0x5628140d2030 2024-03-24T00:18:00.502 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:18:00.501+0000 7fe7f13ed740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x5628140d76d0 con 0x5628140ceed0 2024-03-24T00:18:00.502 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:18:00.501+0000 7fe7f13ed740 10 client.0 _ll_register_callbacks cb 0x5628140311c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T00:18:00.502 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:18:00.501+0000 7fe7e6a00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5628140ceed0 0x5628140dd820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:18:00.502 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:18:00.501+0000 7fe7e6a00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5628140ceed0 0x5628140dd820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.118:52944/0 (socket says 172.21.15.118:52944) 2024-03-24T00:18:00.502 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:18:00.501+0000 7fe7e6a00640 1 -- 172.21.15.118:0/884779018 learned_addr learned my addr 172.21.15.118:0/884779018 (peer_addr_for_me v2:172.21.15.118:0/0) 2024-03-24T00:18:00.502 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:18:00.501+0000 7fe7e7e00640 1 --2- 172.21.15.118:0/884779018 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5628140d2030 0x5628140e2850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:18:00.502 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:18:00.501+0000 7fe7e7400640 1 --2- 172.21.15.118:0/884779018 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5628140ce4e0 0x5628140dd2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:18:00.502 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:18:00.502+0000 7fe7e6a00640 1 -- 172.21.15.118:0/884779018 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5628140d2030 msgr2=0x5628140e2850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:18:00.502 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:18:00.502+0000 7fe7e6a00640 1 --2- 172.21.15.118:0/884779018 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5628140d2030 0x5628140e2850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:18:00.502 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:18:00.502+0000 7fe7e6a00640 1 -- 172.21.15.118:0/884779018 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5628140ce4e0 msgr2=0x5628140dd2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:18:00.502 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:18:00.502+0000 7fe7e6a00640 1 --2- 172.21.15.118:0/884779018 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5628140ce4e0 0x5628140dd2e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:18:00.502 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:18:00.502+0000 7fe7e6a00640 1 -- 172.21.15.118:0/884779018 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x562813d4d210 con 0x5628140ceed0 2024-03-24T00:18:00.503 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:18:00.502+0000 7fe7e7e00640 1 --2- 172.21.15.118:0/884779018 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5628140d2030 0x5628140e2850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T00:18:00.503 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:18:00.502+0000 7fe7e7400640 1 --2- 172.21.15.118:0/884779018 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5628140ce4e0 0x5628140dd2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T00:18:00.503 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:18:00.502+0000 7fe7e6a00640 1 --2- 172.21.15.118:0/884779018 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5628140ceed0 0x5628140dd820 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 crypto rx=0x7fe7d400cfd0 tx=0x7fe7d409bf10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=2e98dc1230cd864c server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:18:00.504 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:18:00.502+0000 7fe7d1600640 10 client.0 ms_handle_connect on v2:172.21.15.193:3300/0 2024-03-24T00:18:00.504 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:18:00.502+0000 7fe7d1600640 1 -- 172.21.15.118:0/884779018 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe7d40a3fb0 con 0x5628140ceed0 2024-03-24T00:18:00.504 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:18:00.502+0000 7fe7f13ed740 1 -- 172.21.15.118:0/884779018 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fe7d80047a0 con 0x5628140ceed0 2024-03-24T00:18:00.504 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:18:00.503+0000 7fe7d1600640 1 -- 172.21.15.118:0/884779018 <== mon.1 v2:172.21.15.193:3300/0 2 ==== osd_map(90..90 src has 1..90) v4 ==== 8277+0+0 (secure 0 0 0) 0x7fe7d4018b50 con 0x5628140ceed0 2024-03-24T00:18:00.504 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:18:00.504+0000 7fe7d1600640 1 -- 172.21.15.118:0/884779018 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) v2 ==== 24+0+0 (secure 0 0 0) 0x7fe7d4018d50 con 0x5628140ceed0 2024-03-24T00:18:00.504 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:18:00.504+0000 7fe7f13ed740 10 client.26011 fetch_fsmap learned FSMap version 28 2024-03-24T00:18:00.504 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:18:00.504+0000 7fe7f13ed740 1 -- 172.21.15.118:0/884779018 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) v3 -- 0x562813e19620 con 0x5628140ceed0 2024-03-24T00:18:00.504 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:18:00.504+0000 7fe7d1600640 1 -- 172.21.15.118:0/884779018 <== mon.1 v2:172.21.15.193:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fe7d80047a0 con 0x5628140ceed0 2024-03-24T00:18:00.505 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:18:00.505+0000 7fe7d1600640 1 -- 172.21.15.118:0/884779018 <== mon.1 v2:172.21.15.193:3300/0 5 ==== fsmap(e 28) v1 ==== 7934+0+0 (secure 0 0 0) 0x7fe7d409ad80 con 0x5628140ceed0 2024-03-24T00:18:00.505 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:18:00.505+0000 7fe7f13ed740 10 client.26011 fetch_fsmap finished waiting for FSMap version 28 2024-03-24T00:18:00.505 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:18:00.505+0000 7fe7f13ed740 10 client.26011 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-24T00:18:00.505 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:18:00.505+0000 7fe7f13ed740 20 client.26011 populate_metadata read hostname 'smithi118' 2024-03-24T00:18:00.505 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:18:00.505+0000 7fe7f13ed740 1 --2- 172.21.15.118:0/884779018 >> [v2:172.21.15.189:6838/4074484168,v1:172.21.15.189:6839/4074484168] conn(0x5628140e9150 0x5628140eb540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:18:00.505 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:18:00.505+0000 7fe7f13ed740 4 client.26011 mds_command: new command op to 24569 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-24T00:18:00.506 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:18:00.505+0000 7fe7f13ed740 1 -- 172.21.15.118:0/884779018 --> [v2:172.21.15.189:6838/4074484168,v1:172.21.15.189:6839/4074484168] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x562814045d70 con 0x5628140e9150 2024-03-24T00:18:00.506 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:18:00.505+0000 7fe7e7400640 1 --2- 172.21.15.118:0/884779018 >> [v2:172.21.15.189:6838/4074484168,v1:172.21.15.189:6839/4074484168] conn(0x5628140e9150 0x5628140eb540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:18:00.506 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:18:00.506+0000 7fe7e7400640 1 --2- 172.21.15.118:0/884779018 >> [v2:172.21.15.189:6838/4074484168,v1:172.21.15.189:6839/4074484168] conn(0x5628140e9150 0x5628140eb540 crc :-1 s=READY pgs=81 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=eadca04849ceefdb server_cookie=eca0d924359ff543 in_seq=0 out_seq=0 2024-03-24T00:18:00.506 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:18:00.506+0000 7fe7d1600640 10 client.26011 ms_handle_connect on v2:172.21.15.189:6838/4074484168 2024-03-24T00:18:00.507 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:18:00.507+0000 7fe7d1600640 1 -- 172.21.15.118:0/884779018 <== mds.0 v2:172.21.15.189:6838/4074484168 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x562814045d70 con 0x5628140e9150 2024-03-24T00:18:00.509 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:18:00.507+0000 7fe7d1600640 10 client.26011 handle_command_reply: tid=0 2024-03-24T00:18:00.509 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:18:00.507+0000 7fe7f13ed740 1 client.26011 shutdown 2024-03-24T00:18:00.509 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:18:00.508+0000 7fe7f13ed740 1 -- 172.21.15.118:0/884779018 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5628140ceed0 msgr2=0x5628140dd820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:18:00.509 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:18:00.508+0000 7fe7f13ed740 1 --2- 172.21.15.118:0/884779018 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5628140ceed0 0x5628140dd820 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 crypto rx=0x7fe7d400cfd0 tx=0x7fe7d409bf10 comp rx=0 tx=0).stop 2024-03-24T00:18:00.510 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:18:00.508+0000 7fe7f13ed740 1 -- 172.21.15.118:0/884779018 shutdown_connections 2024-03-24T00:18:00.510 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:18:00.508+0000 7fe7f13ed740 1 --2- 172.21.15.118:0/884779018 >> [v2:172.21.15.189:6838/4074484168,v1:172.21.15.189:6839/4074484168] conn(0x5628140e9150 0x5628140eb540 crc :-1 s=READY pgs=81 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:18:00.510 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:18:00.508+0000 7fe7f13ed740 1 --2- 172.21.15.118:0/884779018 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5628140d2030 0x5628140e2850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:18:00.510 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:18:00.508+0000 7fe7f13ed740 1 --2- 172.21.15.118:0/884779018 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5628140ceed0 0x5628140dd820 unknown :-1 s=CLOSED pgs=299 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:18:00.510 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:18:00.508+0000 7fe7d1600640 0 client.26011 ms_handle_reset on v2:172.21.15.189:6838/4074484168 2024-03-24T00:18:00.510 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:18:00.508+0000 7fe7f13ed740 1 --2- 172.21.15.118:0/884779018 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5628140ce4e0 0x5628140dd2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:18:00.511 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:18:00.508+0000 7fe7f13ed740 1 -- 172.21.15.118:0/884779018 >> 172.21.15.118:0/884779018 conn(0x5628140d2b60 msgr2=0x5628140cd950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:18:00.511 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:18:00.508+0000 7fe7f13ed740 1 -- 172.21.15.118:0/884779018 shutdown_connections 2024-03-24T00:18:00.511 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:18:00.508+0000 7fe7f13ed740 1 -- 172.21.15.118:0/884779018 wait complete. 2024-03-24T00:18:00.511 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:18:00.508+0000 7fe7f13ed740 20 client.26011 trim_cache size 0 max 16384 2024-03-24T00:18:00.516 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:00.516+0000 7f3727200640 1 -- 172.21.15.118:0/2982510808 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3728101f70 msgr2=0x7f3728104360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:18:00.517 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:00.516+0000 7f3727200640 1 --2- 172.21.15.118:0/2982510808 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3728101f70 0x7f3728104360 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 crypto rx=0x7f3718009900 tx=0x7f371802bc30 comp rx=0 tx=0).stop 2024-03-24T00:18:00.517 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:00.517+0000 7f3724a00640 1 -- 172.21.15.118:0/2982510808 <== mon.2 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f371802cc80 con 0x7f3728101f70 2024-03-24T00:18:00.517 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:00.517+0000 7f3727200640 1 -- 172.21.15.118:0/2982510808 shutdown_connections 2024-03-24T00:18:00.517 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:00.517+0000 7f3727200640 1 --2- 172.21.15.118:0/2982510808 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f37281071d0 0x7f3728109620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:18:00.517 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:00.517+0000 7f3727200640 1 --2- 172.21.15.118:0/2982510808 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f37281048a0 0x7f3728106c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:18:00.517 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:00.517+0000 7f3727200640 1 --2- 172.21.15.118:0/2982510808 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3728101f70 0x7f3728104360 unknown :-1 s=CLOSED pgs=311 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:18:00.517 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:00.517+0000 7f3727200640 1 -- 172.21.15.118:0/2982510808 >> 172.21.15.118:0/2982510808 conn(0x7f37280fb980 msgr2=0x7f37280fdda0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:18:00.517 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:00.517+0000 7f3727200640 1 -- 172.21.15.118:0/2982510808 shutdown_connections 2024-03-24T00:18:00.517 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:00.517+0000 7f3727200640 1 -- 172.21.15.118:0/2982510808 wait complete. 2024-03-24T00:18:00.518 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:00.518+0000 7f3727200640 1 Processor -- start 2024-03-24T00:18:00.518 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:00.518+0000 7f3727200640 1 -- start start 2024-03-24T00:18:00.518 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:00.519+0000 7f3727200640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f3728101f70 0x7f37281977f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:18:00.519 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:00.519+0000 7f3727200640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f37281048a0 0x7f3728197d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:18:00.519 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:00.519+0000 7f3727200640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f37281071d0 0x7f372819eda0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:18:00.519 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:00.519+0000 7f3725e00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f3728101f70 0x7f37281977f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:18:00.520 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:00.520+0000 7f3725e00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f3728101f70 0x7f37281977f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.118:52948/0 (socket says 172.21.15.118:52948) 2024-03-24T00:18:00.520 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:00.520+0000 7f3725e00640 1 -- 172.21.15.118:0/3784280489 learned_addr learned my addr 172.21.15.118:0/3784280489 (peer_addr_for_me v2:172.21.15.118:0/0) 2024-03-24T00:18:00.520 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:00.519+0000 7f3725400640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f37281048a0 0x7f3728197d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:18:00.520 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:00.520+0000 7f3727200640 1 -- 172.21.15.118:0/3784280489 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f372810d1d0 con 0x7f37281071d0 2024-03-24T00:18:00.520 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:00.520+0000 7f3727200640 1 -- 172.21.15.118:0/3784280489 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f372810d050 con 0x7f37281048a0 2024-03-24T00:18:00.520 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:00.520+0000 7f3727200640 1 -- 172.21.15.118:0/3784280489 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f372810d350 con 0x7f3728101f70 2024-03-24T00:18:00.523 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:00.520+0000 7f3725e00640 1 -- 172.21.15.118:0/3784280489 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f37281048a0 msgr2=0x7f3728197d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:18:00.523 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:00.520+0000 7f3725e00640 1 --2- 172.21.15.118:0/3784280489 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f37281048a0 0x7f3728197d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:18:00.523 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:00.520+0000 7f3725e00640 1 -- 172.21.15.118:0/3784280489 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f37281071d0 msgr2=0x7f372819eda0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-03-24T00:18:00.523 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:00.520+0000 7f3725e00640 1 --2- 172.21.15.118:0/3784280489 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f37281071d0 0x7f372819eda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:18:00.523 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:00.520+0000 7f3725e00640 1 -- 172.21.15.118:0/3784280489 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f372819f410 con 0x7f3728101f70 2024-03-24T00:18:00.523 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:00.520+0000 7f3725400640 1 --2- 172.21.15.118:0/3784280489 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f37281048a0 0x7f3728197d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T00:18:00.524 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:00.521+0000 7f3725e00640 1 --2- 172.21.15.118:0/3784280489 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f3728101f70 0x7f37281977f0 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 crypto rx=0x7f371802c8e0 tx=0x7f371802c650 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:18:00.524 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:00.521+0000 7f371ea00640 1 -- 172.21.15.118:0/3784280489 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3718041070 con 0x7f3728101f70 2024-03-24T00:18:00.524 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:00.521+0000 7f371ea00640 1 -- 172.21.15.118:0/3784280489 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f3718004450 con 0x7f3728101f70 2024-03-24T00:18:00.524 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:00.521+0000 7f3727200640 1 -- 172.21.15.118:0/3784280489 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f372819f6a0 con 0x7f3728101f70 2024-03-24T00:18:00.524 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:00.521+0000 7f3727200640 1 -- 172.21.15.118:0/3784280489 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f372819fbe0 con 0x7f3728101f70 2024-03-24T00:18:00.524 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:00.521+0000 7f371ea00640 1 -- 172.21.15.118:0/3784280489 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3718034440 con 0x7f3728101f70 2024-03-24T00:18:00.524 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:00.523+0000 7f3727200640 1 -- 172.21.15.118:0/3784280489 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f37280fd470 con 0x7f3728101f70 2024-03-24T00:18:00.525 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:00.524+0000 7f371ea00640 1 -- 172.21.15.118:0/3784280489 <== mon.1 v2:172.21.15.193:3300/0 4 ==== mgrmap(e 20) v1 ==== 137291+0+0 (secure 0 0 0) 0x7f3718044070 con 0x7f3728101f70 2024-03-24T00:18:00.526 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:00.526+0000 7f371ea00640 1 --2- 172.21.15.118:0/3784280489 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7f370c0a28c0 0x7f370c0a4d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:18:00.526 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:00.526+0000 7f371ea00640 1 -- 172.21.15.118:0/3784280489 <== mon.1 v2:172.21.15.193:3300/0 5 ==== osd_map(90..90 src has 1..90) v4 ==== 8277+0+0 (secure 0 0 0) 0x7f37180f0660 con 0x7f3728101f70 2024-03-24T00:18:00.526 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:00.526+0000 7f3725400640 1 --2- 172.21.15.118:0/3784280489 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7f370c0a28c0 0x7f370c0a4d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:18:00.527 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:00.527+0000 7f3725400640 1 --2- 172.21.15.118:0/3784280489 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7f370c0a28c0 0x7f370c0a4d80 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 crypto rx=0x7f3728198d00 tx=0x7f3714005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:18:00.529 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:00.530+0000 7f371ea00640 1 -- 172.21.15.118:0/3784280489 <== mon.1 v2:172.21.15.193:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f37180bd670 con 0x7f3728101f70 2024-03-24T00:18:00.536 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:18:00.528+0000 7fe7f13ed740 1 Processor -- start 2024-03-24T00:18:00.536 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:18:00.528+0000 7fe7f13ed740 1 -- start start 2024-03-24T00:18:00.536 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:18:00.528+0000 7fe7f13ed740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5628140ce4e0 0x5628140d5dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:18:00.536 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:18:00.528+0000 7fe7f13ed740 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5628140ceed0 0x5628140d6300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:18:00.536 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:18:00.528+0000 7fe7f13ed740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5628140d2030 0x5628140db940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:18:00.536 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:18:00.528+0000 7fe7f13ed740 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe7e81c8930 con 0x5628140ceed0 2024-03-24T00:18:00.536 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:18:00.529+0000 7fe7f13ed740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe7e81ca2b0 con 0x5628140ce4e0 2024-03-24T00:18:00.536 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:18:00.529+0000 7fe7f13ed740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe7e81c8f90 con 0x5628140d2030 2024-03-24T00:18:00.536 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:18:00.529+0000 7fe7e7400640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5628140ce4e0 0x5628140d5dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:18:00.537 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:18:00.529+0000 7fe7e7e00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5628140d2030 0x5628140db940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:18:00.537 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:18:00.529+0000 7fe7e7e00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5628140d2030 0x5628140db940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.118:52950/0 (socket says 172.21.15.118:52950) 2024-03-24T00:18:00.537 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:18:00.529+0000 7fe7e7e00640 1 -- 172.21.15.118:0/2553934646 learned_addr learned my addr 172.21.15.118:0/2553934646 (peer_addr_for_me v2:172.21.15.118:0/0) 2024-03-24T00:18:00.537 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:18:00.529+0000 7fe7e7400640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5628140ce4e0 0x5628140d5dc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.118:38028/0 (socket says 172.21.15.118:38028) 2024-03-24T00:18:00.537 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:18:00.529+0000 7fe7e6a00640 1 --2- 172.21.15.118:0/2553934646 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5628140ceed0 0x5628140d6300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:18:00.537 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:18:00.529+0000 7fe7e7e00640 1 -- 172.21.15.118:0/2553934646 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5628140ce4e0 msgr2=0x5628140d5dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:18:00.537 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:18:00.529+0000 7fe7e7e00640 1 --2- 172.21.15.118:0/2553934646 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5628140ce4e0 0x5628140d5dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:18:00.537 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:18:00.529+0000 7fe7e7e00640 1 -- 172.21.15.118:0/2553934646 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5628140ceed0 msgr2=0x5628140d6300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:18:00.537 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:18:00.529+0000 7fe7e7e00640 1 --2- 172.21.15.118:0/2553934646 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5628140ceed0 0x5628140d6300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:18:00.537 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:18:00.529+0000 7fe7e7e00640 1 -- 172.21.15.118:0/2553934646 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5628140d38d0 con 0x5628140d2030 2024-03-24T00:18:00.537 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:18:00.529+0000 7fe7e7e00640 1 --2- 172.21.15.118:0/2553934646 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5628140d2030 0x5628140db940 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 crypto rx=0x7fe7e002c710 tx=0x7fe7e0040000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=1f83f9e6a507d3c1 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:18:00.537 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:18:00.530+0000 7fe7e7400640 1 --2- 172.21.15.118:0/2553934646 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5628140ce4e0 0x5628140d5dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-24T00:18:00.537 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:18:00.530+0000 7fe7d3e00640 1 -- 172.21.15.118:0/2553934646 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe7e0039040 con 0x5628140d2030 2024-03-24T00:18:00.537 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:18:00.530+0000 7fe7d3e00640 1 -- 172.21.15.118:0/2553934646 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fe7e0053070 con 0x5628140d2030 2024-03-24T00:18:00.538 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:18:00.530+0000 7fe7d3e00640 1 -- 172.21.15.118:0/2553934646 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe7e00406c0 con 0x5628140d2030 2024-03-24T00:18:00.538 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:18:00.531+0000 7fe7f13ed740 1 -- 172.21.15.118:0/2553934646 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5628140d2030 msgr2=0x5628140db940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:18:00.538 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:18:00.531+0000 7fe7f13ed740 1 --2- 172.21.15.118:0/2553934646 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5628140d2030 0x5628140db940 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 crypto rx=0x7fe7e002c710 tx=0x7fe7e0040000 comp rx=0 tx=0).stop 2024-03-24T00:18:00.538 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:18:00.531+0000 7fe7f13ed740 1 -- 172.21.15.118:0/2553934646 shutdown_connections 2024-03-24T00:18:00.538 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:18:00.531+0000 7fe7f13ed740 1 --2- 172.21.15.118:0/2553934646 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5628140d2030 0x5628140db940 unknown :-1 s=CLOSED pgs=301 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:18:00.538 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:18:00.531+0000 7fe7f13ed740 1 --2- 172.21.15.118:0/2553934646 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5628140ceed0 0x5628140d6300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:18:00.538 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:18:00.531+0000 7fe7f13ed740 1 --2- 172.21.15.118:0/2553934646 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5628140ce4e0 0x5628140d5dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:18:00.538 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:18:00.531+0000 7fe7f13ed740 1 -- 172.21.15.118:0/2553934646 >> 172.21.15.118:0/2553934646 conn(0x5628140d2b60 msgr2=0x5628140d82f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:18:00.538 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:18:00.534+0000 7fe7f13ed740 1 -- 172.21.15.118:0/2553934646 shutdown_connections 2024-03-24T00:18:00.538 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:18:00.534+0000 7fe7f13ed740 1 -- 172.21.15.118:0/2553934646 wait complete. 2024-03-24T00:18:00.538 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:18:00.535+0000 7fe7f13ed740 1 -- start start 2024-03-24T00:18:00.538 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:18:00.535+0000 7fe7f13ed740 1 Processor -- start 2024-03-24T00:18:00.538 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:18:00.536+0000 7fe7f13ed740 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5628140ce4e0 0x5628140fb4f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:18:00.538 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:18:00.536+0000 7fe7f13ed740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5628140ceed0 0x5628140f9b00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:18:00.539 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:18:00.536+0000 7fe7f13ed740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5628140d2030 0x5628140fa040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:18:00.539 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:18:00.536+0000 7fe7f13ed740 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe7e81c9c50 con 0x5628140ce4e0 2024-03-24T00:18:00.539 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:18:00.536+0000 7fe7f13ed740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe7e81c95f0 con 0x5628140ceed0 2024-03-24T00:18:00.539 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:18:00.536+0000 7fe7f13ed740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe7e81c6af0 con 0x5628140d2030 2024-03-24T00:18:00.539 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:18:00.536+0000 7fe7f13ed740 10 client.0 _ll_register_callbacks cb 0x7fe7d400cf20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T00:18:00.539 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:18:00.536+0000 7fe7e6a00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5628140ceed0 0x5628140f9b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:18:00.539 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:18:00.536+0000 7fe7e6a00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5628140ceed0 0x5628140f9b00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.118:38044/0 (socket says 172.21.15.118:38044) 2024-03-24T00:18:00.539 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:18:00.536+0000 7fe7e6a00640 1 -- 172.21.15.118:0/1646370223 learned_addr learned my addr 172.21.15.118:0/1646370223 (peer_addr_for_me v2:172.21.15.118:0/0) 2024-03-24T00:18:00.539 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:18:00.536+0000 7fe7e7e00640 1 --2- 172.21.15.118:0/1646370223 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5628140d2030 0x5628140fa040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:18:00.541 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:18:00.537+0000 7fe7e6a00640 1 -- 172.21.15.118:0/1646370223 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5628140d2030 msgr2=0x5628140fa040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:18:00.541 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:18:00.537+0000 7fe7e6a00640 1 --2- 172.21.15.118:0/1646370223 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5628140d2030 0x5628140fa040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:18:00.541 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:18:00.537+0000 7fe7e6a00640 1 -- 172.21.15.118:0/1646370223 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5628140ce4e0 msgr2=0x5628140fb4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:18:00.541 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:18:00.537+0000 7fe7e7400640 1 --2- 172.21.15.118:0/1646370223 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5628140ce4e0 0x5628140fb4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:18:00.541 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:18:00.539+0000 7fe7e6a00640 1 --2- 172.21.15.118:0/1646370223 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5628140ce4e0 0x5628140fb4f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:18:00.541 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:18:00.539+0000 7fe7e6a00640 1 -- 172.21.15.118:0/1646370223 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5628140fa730 con 0x5628140ceed0 2024-03-24T00:18:00.541 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:18:00.539+0000 7fe7e7400640 1 --2- 172.21.15.118:0/1646370223 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5628140ce4e0 0x5628140fb4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-24T00:18:00.542 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:18:00.539+0000 7fe7e6a00640 1 --2- 172.21.15.118:0/1646370223 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5628140ceed0 0x5628140f9b00 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 crypto rx=0x7fe7d400e9c0 tx=0x7fe7d4086340 comp rx=0 tx=0).ready entity=mon.2 client_cookie=ba081ace6c123734 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:18:00.542 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:18:00.539+0000 7fe7d1600640 10 client.0 ms_handle_connect on v2:172.21.15.189:3300/0 2024-03-24T00:18:00.542 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:18:00.540+0000 7fe7d1600640 1 -- 172.21.15.118:0/1646370223 <== mon.2 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe7d409ae70 con 0x5628140ceed0 2024-03-24T00:18:00.542 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:18:00.540+0000 7fe7d1600640 1 -- 172.21.15.118:0/1646370223 <== mon.2 v2:172.21.15.189:3300/0 2 ==== osd_map(90..90 src has 1..90) v4 ==== 8277+0+0 (secure 0 0 0) 0x7fe7d4010430 con 0x5628140ceed0 2024-03-24T00:18:00.542 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:18:00.540+0000 7fe7f13ed740 1 -- 172.21.15.118:0/1646370223 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fe7e0053070 con 0x5628140ceed0 2024-03-24T00:18:00.542 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:18:00.540+0000 7fe7d1600640 1 -- 172.21.15.118:0/1646370223 <== mon.2 v2:172.21.15.189:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) v2 ==== 24+0+0 (secure 0 0 0) 0x7fe7d40af050 con 0x5628140ceed0 2024-03-24T00:18:00.542 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:18:00.540+0000 7fe7f13ed740 10 client.26030 fetch_fsmap learned FSMap version 28 2024-03-24T00:18:00.542 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:18:00.540+0000 7fe7f13ed740 1 -- 172.21.15.118:0/1646370223 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) v3 -- 0x562814144d40 con 0x5628140ceed0 2024-03-24T00:18:00.542 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:18:00.540+0000 7fe7d1600640 1 -- 172.21.15.118:0/1646370223 <== mon.2 v2:172.21.15.189:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fe7e0053070 con 0x5628140ceed0 2024-03-24T00:18:00.542 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:18:00.540+0000 7fe7d1600640 1 -- 172.21.15.118:0/1646370223 <== mon.2 v2:172.21.15.189:3300/0 5 ==== fsmap(e 28) v1 ==== 7934+0+0 (secure 0 0 0) 0x7fe7d40a0880 con 0x5628140ceed0 2024-03-24T00:18:00.542 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:18:00.540+0000 7fe7f13ed740 10 client.26030 fetch_fsmap finished waiting for FSMap version 28 2024-03-24T00:18:00.542 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:18:00.540+0000 7fe7f13ed740 10 client.26030 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-24T00:18:00.542 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:18:00.540+0000 7fe7f13ed740 20 client.26030 populate_metadata read hostname 'smithi118' 2024-03-24T00:18:00.542 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:18:00.540+0000 7fe7f13ed740 1 --2- 172.21.15.118:0/1646370223 >> [v2:172.21.15.189:6838/4074484168,v1:172.21.15.189:6839/4074484168] conn(0x562814145360 0x5628141477c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:18:00.543 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:18:00.540+0000 7fe7f13ed740 4 client.26030 mds_command: new command op to 24569 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-03-24T00:18:00.543 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:18:00.540+0000 7fe7f13ed740 1 -- 172.21.15.118:0/1646370223 --> [v2:172.21.15.189:6838/4074484168,v1:172.21.15.189:6839/4074484168] -- command(tid 0: {"prefix": "flush journal"}) v1 -- 0x5628140400c0 con 0x562814145360 2024-03-24T00:18:00.543 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:18:00.541+0000 7fe7e7400640 1 --2- 172.21.15.118:0/1646370223 >> [v2:172.21.15.189:6838/4074484168,v1:172.21.15.189:6839/4074484168] conn(0x562814145360 0x5628141477c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:18:00.543 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:18:00.541+0000 7fe7e7400640 1 --2- 172.21.15.118:0/1646370223 >> [v2:172.21.15.189:6838/4074484168,v1:172.21.15.189:6839/4074484168] conn(0x562814145360 0x5628141477c0 crc :-1 s=READY pgs=82 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=6994f07dbc1f7152 server_cookie=e21d9084fd838f85 in_seq=0 out_seq=0 2024-03-24T00:18:00.543 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:18:00.541+0000 7fe7d1600640 10 client.26030 ms_handle_connect on v2:172.21.15.189:6838/4074484168 2024-03-24T00:18:00.877 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:18:00.877+0000 7fe7d1600640 1 -- 172.21.15.118:0/1646370223 <== mds.0 v2:172.21.15.189:6838/4074484168 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+44 (crc 0 0 0) 0x5628140400c0 con 0x562814145360 2024-03-24T00:18:00.878 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:18:00.877+0000 7fe7d1600640 10 client.26030 handle_command_reply: tid=0 2024-03-24T00:18:00.878 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:18:00.877+0000 7fe7f13ed740 1 client.26030 shutdown 2024-03-24T00:18:00.878 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:18:00.877+0000 7fe7f13ed740 1 -- 172.21.15.118:0/1646370223 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5628140ceed0 msgr2=0x5628140f9b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:18:00.878 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:18:00.877+0000 7fe7f13ed740 1 --2- 172.21.15.118:0/1646370223 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5628140ceed0 0x5628140f9b00 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 crypto rx=0x7fe7d400e9c0 tx=0x7fe7d4086340 comp rx=0 tx=0).stop 2024-03-24T00:18:00.878 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:18:00.877+0000 7fe7f13ed740 1 -- 172.21.15.118:0/1646370223 shutdown_connections 2024-03-24T00:18:00.878 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:18:00.877+0000 7fe7f13ed740 1 --2- 172.21.15.118:0/1646370223 >> [v2:172.21.15.189:6838/4074484168,v1:172.21.15.189:6839/4074484168] conn(0x562814145360 0x5628141477c0 crc :-1 s=READY pgs=82 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:18:00.878 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:18:00.878+0000 7fe7f13ed740 1 --2- 172.21.15.118:0/1646370223 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5628140d2030 0x5628140fa040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:18:00.878 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:18:00.878+0000 7fe7f13ed740 1 --2- 172.21.15.118:0/1646370223 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5628140ceed0 0x5628140f9b00 unknown :-1 s=CLOSED pgs=312 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:18:00.878 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:18:00.878+0000 7fe7f13ed740 1 --2- 172.21.15.118:0/1646370223 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5628140ce4e0 0x5628140fb4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:18:00.878 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:18:00.878+0000 7fe7d1600640 0 client.26030 ms_handle_reset on v2:172.21.15.189:6838/4074484168 2024-03-24T00:18:00.878 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:18:00.878+0000 7fe7f13ed740 1 -- 172.21.15.118:0/1646370223 >> 172.21.15.118:0/1646370223 conn(0x5628140d2b60 msgr2=0x5628140d82f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:18:00.878 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:18:00.878+0000 7fe7f13ed740 1 -- 172.21.15.118:0/1646370223 shutdown_connections 2024-03-24T00:18:00.878 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:18:00.878+0000 7fe7f13ed740 1 -- 172.21.15.118:0/1646370223 wait complete. 2024-03-24T00:18:00.879 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:18:00.878+0000 7fe7f13ed740 20 client.26030 trim_cache size 0 max 16384 2024-03-24T00:18:00.879 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stdout:{ 2024-03-24T00:18:00.879 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stdout: "message": "", 2024-03-24T00:18:00.879 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stdout: "return_code": 0 2024-03-24T00:18:00.879 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stdout:} 2024-03-24T00:18:00.880 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:18:00.879+0000 7fe7ee600640 1 -- 172.21.15.118:0/2200287762 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7fe7cc0a2920 msgr2=0x7fe7cc0a4de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:18:00.880 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:18:00.879+0000 7fe7ee600640 1 --2- 172.21.15.118:0/2200287762 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7fe7cc0a2920 0x7fe7cc0a4de0 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 crypto rx=0x7fe7e002c640 tx=0x7fe7e002e040 comp rx=0 tx=0).stop 2024-03-24T00:18:00.880 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:18:00.879+0000 7fe7ee600640 1 -- 172.21.15.118:0/2200287762 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe7e81028e0 msgr2=0x7fe7e8197e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:18:00.880 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:18:00.879+0000 7fe7ee600640 1 --2- 172.21.15.118:0/2200287762 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe7e81028e0 0x7fe7e8197e70 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 crypto rx=0x7fe7d400e9f0 tx=0x7fe7d400eeb0 comp rx=0 tx=0).stop 2024-03-24T00:18:00.880 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:18:00.879+0000 7fe7ee600640 1 -- 172.21.15.118:0/2200287762 shutdown_connections 2024-03-24T00:18:00.880 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:18:00.879+0000 7fe7ee600640 1 --2- 172.21.15.118:0/2200287762 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7fe7cc0a2920 0x7fe7cc0a4de0 unknown :-1 s=CLOSED pgs=308 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:18:00.880 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:18:00.879+0000 7fe7ee600640 1 --2- 172.21.15.118:0/2200287762 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fe7e81032a0 0x7fe7e819eee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:18:00.880 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:18:00.879+0000 7fe7ee600640 1 --2- 172.21.15.118:0/2200287762 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe7e81028e0 0x7fe7e8197e70 unknown :-1 s=CLOSED pgs=310 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:18:00.880 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:18:00.879+0000 7fe7ee600640 1 --2- 172.21.15.118:0/2200287762 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fe7e81016f0 0x7fe7e8197930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:18:00.880 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:18:00.879+0000 7fe7ee600640 1 -- 172.21.15.118:0/2200287762 >> 172.21.15.118:0/2200287762 conn(0x7fe7e80fcc60 msgr2=0x7fe7e8105b80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:18:00.881 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:18:00.879+0000 7fe7ee600640 1 -- 172.21.15.118:0/2200287762 shutdown_connections 2024-03-24T00:18:00.881 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:18:00.879+0000 7fe7ee600640 1 -- 172.21.15.118:0/2200287762 wait complete. 2024-03-24T00:18:00.912 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:00.913+0000 7f3727200640 1 -- 172.21.15.118:0/3784280489 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f372802d070 con 0x7f3728101f70 2024-03-24T00:18:00.913 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:00.914+0000 7f371ea00640 1 -- 172.21.15.118:0/3784280489 <== mon.1 v2:172.21.15.193:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+11478 (secure 0 0 0) 0x7f37180bcf70 con 0x7f3728101f70 2024-03-24T00:18:00.914 INFO:teuthology.orchestra.run.smithi118.stdout: 2024-03-24T00:18:00.914 INFO:teuthology.orchestra.run.smithi118.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14385,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.118:6835/3194741373","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6834","nonce":3194741373},{"type":"v1","addr":"172.21.15.118:6835","nonce":3194741373}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":4},{"gid":14397,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.118:6837/1374462956","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6836","nonce":1374462956},{"type":"v1","addr":"172.21.15.118:6837","nonce":1374462956}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":28,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-03-24T00:13:01.782474+0000","modified":"2024-03-24T00:15:38.235450+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24569,"mds_1":24550,"mds_2":24511,"mds_3":24499,"mds_4":24488},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14469":{"gid":14469,"name":"a","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.118:6841/3040946537","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6840","nonce":3040946537},{"type":"v1","addr":"172.21.15.118:6841","nonce":3040946537}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24439":{"gid":24439,"name":"l","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.193:6833/1023428412","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6832","nonce":1023428412},{"type":"v1","addr":"172.21.15.193:6833","nonce":1023428412}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24440":{"gid":24440,"name":"k","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.189:6833/2740024078","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6832","nonce":2740024078},{"type":"v1","addr":"172.21.15.189:6833","nonce":2740024078}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24488":{"gid":24488,"name":"e","rank":4,"incarnation":25,"state":"up:active","state_seq":10,"addr":"172.21.15.189:6835/599473205","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6834","nonce":599473205},{"type":"v1","addr":"172.21.15.189:6835","nonce":599473205}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24499":{"gid":24499,"name":"f","rank":3,"incarnation":23,"state":"up:active","state_seq":9,"addr":"172.21.15.193:6835/2055310541","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6834","nonce":2055310541},{"type":"v1","addr":"172.21.15.193:6835","nonce":2055310541}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24511":{"gid":24511,"name":"c","rank":2,"incarnation":21,"state":"up:active","state_seq":8,"addr":"172.21.15.193:6837/2921520053","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6836","nonce":2921520053},{"type":"v1","addr":"172.21.15.193:6837","nonce":2921520053}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24515":{"gid":24515,"name":"h","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.189:6837/3450187137","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6836","nonce":3450187137},{"type":"v1","addr":"172.21.15.189:6837","nonce":3450187137}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24532":{"gid":24532,"name":"i","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.193:6839/2126842603","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6838","nonce":2126842603},{"type":"v1","addr":"172.21.15.193:6839","nonce":2126842603}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24550":{"gid":24550,"name":"g","rank":1,"incarnation":19,"state":"up:active","state_seq":6,"addr":"172.21.15.118:6839/3285788503","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6838","nonce":3285788503},{"type":"v1","addr":"172.21.15.118:6839","nonce":3285788503}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24569":{"gid":24569,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.189:6839/4074484168","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6838","nonce":4074484168},{"type":"v1","addr":"172.21.15.189:6839","nonce":4074484168}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24569,"qdb_cluster":[24569,24499,24488,24511,24550]},"id":1}]} 2024-03-24T00:18:00.914 INFO:teuthology.orchestra.run.smithi118.stderr:dumped fsmap epoch 28 2024-03-24T00:18:00.917 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:00.917+0000 7f3727200640 1 -- 172.21.15.118:0/3784280489 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7f370c0a28c0 msgr2=0x7f370c0a4d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:18:00.917 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:00.917+0000 7f3727200640 1 --2- 172.21.15.118:0/3784280489 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7f370c0a28c0 0x7f370c0a4d80 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 crypto rx=0x7f3728198d00 tx=0x7f3714005f90 comp rx=0 tx=0).stop 2024-03-24T00:18:00.917 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:00.917+0000 7f3727200640 1 -- 172.21.15.118:0/3784280489 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f3728101f70 msgr2=0x7f37281977f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:18:00.917 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:00.917+0000 7f3727200640 1 --2- 172.21.15.118:0/3784280489 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f3728101f70 0x7f37281977f0 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 crypto rx=0x7f371802c8e0 tx=0x7f371802c650 comp rx=0 tx=0).stop 2024-03-24T00:18:00.918 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:00.917+0000 7f3727200640 1 -- 172.21.15.118:0/3784280489 shutdown_connections 2024-03-24T00:18:00.918 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:00.917+0000 7f3727200640 1 --2- 172.21.15.118:0/3784280489 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7f370c0a28c0 0x7f370c0a4d80 unknown :-1 s=CLOSED pgs=309 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:18:00.918 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:00.917+0000 7f3727200640 1 --2- 172.21.15.118:0/3784280489 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f37281071d0 0x7f372819eda0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:18:00.918 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:00.917+0000 7f3727200640 1 --2- 172.21.15.118:0/3784280489 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f37281048a0 0x7f3728197d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:18:00.918 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:00.917+0000 7f3727200640 1 --2- 172.21.15.118:0/3784280489 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f3728101f70 0x7f37281977f0 unknown :-1 s=CLOSED pgs=300 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:18:00.918 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:00.917+0000 7f3727200640 1 -- 172.21.15.118:0/3784280489 >> 172.21.15.118:0/3784280489 conn(0x7f37280fb980 msgr2=0x7f3728102a10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:18:00.918 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:00.917+0000 7f3727200640 1 -- 172.21.15.118:0/3784280489 shutdown_connections 2024-03-24T00:18:00.918 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:00.917+0000 7f3727200640 1 -- 172.21.15.118:0/3784280489 wait complete. 2024-03-24T00:18:00.935 INFO:teuthology.orchestra.run:Running command with timeout 120 2024-03-24T00:18:00.935 DEBUG:teuthology.orchestra.run.smithi118:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:4 damage ls 2024-03-24T00:18:01.184 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:01.184+0000 7fb4e0000640 1 -- 172.21.15.118:0/106588020 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fb4d8102150 msgr2=0x7fb4d81089e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:18:01.185 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:01.184+0000 7fb4e0000640 1 --2- 172.21.15.118:0/106588020 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fb4d8102150 0x7fb4d81089e0 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 crypto rx=0x7fb4d0009900 tx=0x7fb4d002bc50 comp rx=0 tx=0).stop 2024-03-24T00:18:01.185 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:01.185+0000 7fb4e0000640 1 -- 172.21.15.118:0/106588020 shutdown_connections 2024-03-24T00:18:01.185 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:01.185+0000 7fb4e0000640 1 --2- 172.21.15.118:0/106588020 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fb4d8102150 0x7fb4d81089e0 unknown :-1 s=CLOSED pgs=302 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:18:01.185 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:01.185+0000 7fb4e0000640 1 --2- 172.21.15.118:0/106588020 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb4d8101790 0x7fb4d8101c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:18:01.185 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:01.185+0000 7fb4e0000640 1 --2- 172.21.15.118:0/106588020 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb4d81005a0 0x7fb4d81009c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:18:01.185 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:01.185+0000 7fb4e0000640 1 -- 172.21.15.118:0/106588020 >> 172.21.15.118:0/106588020 conn(0x7fb4d80fbb30 msgr2=0x7fb4d80fdf50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:18:01.185 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:01.185+0000 7fb4e0000640 1 -- 172.21.15.118:0/106588020 shutdown_connections 2024-03-24T00:18:01.185 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:01.185+0000 7fb4e0000640 1 -- 172.21.15.118:0/106588020 wait complete. 2024-03-24T00:18:01.187 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:01.186+0000 7fb4e0000640 1 Processor -- start 2024-03-24T00:18:01.187 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:01.186+0000 7fb4e0000640 1 -- start start 2024-03-24T00:18:01.187 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:01.186+0000 7fb4e0000640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fb4d81005a0 0x7fb4d8199bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:18:01.188 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:01.186+0000 7fb4e0000640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb4d8101790 0x7fb4d819a130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:18:01.188 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:01.186+0000 7fb4e0000640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb4d8102150 0x7fb4d81a1130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:18:01.188 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:01.186+0000 7fb4e0000640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb4d810ac90 con 0x7fb4d8101790 2024-03-24T00:18:01.188 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:01.186+0000 7fb4e0000640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb4d810ab10 con 0x7fb4d8102150 2024-03-24T00:18:01.188 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:01.186+0000 7fb4e0000640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb4d810ae10 con 0x7fb4d81005a0 2024-03-24T00:18:01.188 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:01.187+0000 7fb4dd600640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fb4d81005a0 0x7fb4d8199bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:18:01.188 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:01.187+0000 7fb4de000640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb4d8102150 0x7fb4d81a1130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:18:01.188 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:01.187+0000 7fb4dd600640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fb4d81005a0 0x7fb4d8199bf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.118:52980/0 (socket says 172.21.15.118:52980) 2024-03-24T00:18:01.188 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:01.187+0000 7fb4dd600640 1 -- 172.21.15.118:0/1466774292 learned_addr learned my addr 172.21.15.118:0/1466774292 (peer_addr_for_me v2:172.21.15.118:0/0) 2024-03-24T00:18:01.188 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:01.187+0000 7fb4dcc00640 1 --2- 172.21.15.118:0/1466774292 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb4d8101790 0x7fb4d819a130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:18:01.188 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:01.187+0000 7fb4dd600640 1 -- 172.21.15.118:0/1466774292 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb4d8102150 msgr2=0x7fb4d81a1130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:18:01.188 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:01.187+0000 7fb4dd600640 1 --2- 172.21.15.118:0/1466774292 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb4d8102150 0x7fb4d81a1130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:18:01.188 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:01.187+0000 7fb4dd600640 1 -- 172.21.15.118:0/1466774292 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb4d8101790 msgr2=0x7fb4d819a130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:18:01.188 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:01.187+0000 7fb4dd600640 1 --2- 172.21.15.118:0/1466774292 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb4d8101790 0x7fb4d819a130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:18:01.189 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:01.187+0000 7fb4dd600640 1 -- 172.21.15.118:0/1466774292 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb4d81a17a0 con 0x7fb4d81005a0 2024-03-24T00:18:01.189 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:01.187+0000 7fb4dcc00640 1 --2- 172.21.15.118:0/1466774292 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb4d8101790 0x7fb4d819a130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T00:18:01.189 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:01.187+0000 7fb4de000640 1 --2- 172.21.15.118:0/1466774292 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb4d8102150 0x7fb4d81a1130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T00:18:01.189 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:01.187+0000 7fb4dd600640 1 --2- 172.21.15.118:0/1466774292 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fb4d81005a0 0x7fb4d8199bf0 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 crypto rx=0x7fb4cc00ebc0 tx=0x7fb4cc00c6a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:18:01.189 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:01.188+0000 7fb4d6000640 1 -- 172.21.15.118:0/1466774292 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb4cc003b70 con 0x7fb4d81005a0 2024-03-24T00:18:01.189 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:01.188+0000 7fb4d6000640 1 -- 172.21.15.118:0/1466774292 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fb4cc003d10 con 0x7fb4d81005a0 2024-03-24T00:18:01.189 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:01.188+0000 7fb4e0000640 1 -- 172.21.15.118:0/1466774292 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb4d81a1a90 con 0x7fb4d81005a0 2024-03-24T00:18:01.189 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:01.188+0000 7fb4d6000640 1 -- 172.21.15.118:0/1466774292 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb4cc019980 con 0x7fb4d81005a0 2024-03-24T00:18:01.189 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:01.188+0000 7fb4e0000640 1 -- 172.21.15.118:0/1466774292 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb4d81a1fe0 con 0x7fb4d81005a0 2024-03-24T00:18:01.191 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:01.191+0000 7fb4e2dd9740 1 Processor -- start 2024-03-24T00:18:01.191 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:01.191+0000 7fb4e2dd9740 1 -- start start 2024-03-24T00:18:01.192 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:01.191+0000 7fb4e2dd9740 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55744869f810 0x55744869fc30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:18:01.192 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:01.192+0000 7fb4e2dd9740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5574486a09e0 0x5574486a0e60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:18:01.193 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:01.192+0000 7fb4e2dd9740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5574486a13a0 0x5574486a7c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:18:01.193 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:01.192+0000 7fb4e2dd9740 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x5574486042c0 con 0x55744869f810 2024-03-24T00:18:01.193 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:01.192+0000 7fb4e2dd9740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x5574485acc90 con 0x5574486a09e0 2024-03-24T00:18:01.193 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:01.192+0000 7fb4e2dd9740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x557448600360 con 0x5574486a13a0 2024-03-24T00:18:01.193 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:01.192+0000 7fb4d6000640 1 -- 172.21.15.118:0/1466774292 <== mon.1 v2:172.21.15.193:3300/0 4 ==== mgrmap(e 20) v1 ==== 137291+0+0 (secure 0 0 0) 0x7fb4cc019b20 con 0x7fb4d81005a0 2024-03-24T00:18:01.193 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:01.192+0000 7fb4de000640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55744869f810 0x55744869fc30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:18:01.193 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:01.192+0000 7fb4de000640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55744869f810 0x55744869fc30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.118:3300/0 says I am v2:172.21.15.118:48582/0 (socket says 172.21.15.118:48582) 2024-03-24T00:18:01.193 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:01.192+0000 7fb4de000640 1 -- 172.21.15.118:0/1405623430 learned_addr learned my addr 172.21.15.118:0/1405623430 (peer_addr_for_me v2:172.21.15.118:0/0) 2024-03-24T00:18:01.193 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:01.192+0000 7fb4dcc00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5574486a13a0 0x5574486a7c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:18:01.193 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:01.192+0000 7fb4dd600640 1 --2- 172.21.15.118:0/1405623430 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5574486a09e0 0x5574486a0e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:18:01.193 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:01.192+0000 7fb4dcc00640 1 -- 172.21.15.118:0/1405623430 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5574486a09e0 msgr2=0x5574486a0e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:18:01.193 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:01.192+0000 7fb4dcc00640 1 --2- 172.21.15.118:0/1405623430 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5574486a09e0 0x5574486a0e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:18:01.193 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:01.192+0000 7fb4dcc00640 1 -- 172.21.15.118:0/1405623430 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55744869f810 msgr2=0x55744869fc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:18:01.193 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:01.192+0000 7fb4dcc00640 1 --2- 172.21.15.118:0/1405623430 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55744869f810 0x55744869fc30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:18:01.194 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:01.192+0000 7fb4dcc00640 1 -- 172.21.15.118:0/1405623430 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5574485d1080 con 0x5574486a13a0 2024-03-24T00:18:01.194 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:01.192+0000 7fb4dd600640 1 --2- 172.21.15.118:0/1405623430 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5574486a09e0 0x5574486a0e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T00:18:01.194 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:01.193+0000 7fb4dcc00640 1 --2- 172.21.15.118:0/1405623430 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5574486a13a0 0x5574486a7c30 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 crypto rx=0x7fb4c4009c00 tx=0x7fb4c40117b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=30a612918b8287c2 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:18:01.194 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:01.193+0000 7fb4ca600640 1 -- 172.21.15.118:0/1405623430 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb4c4005020 con 0x5574486a13a0 2024-03-24T00:18:01.194 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:01.193+0000 7fb4ca600640 1 -- 172.21.15.118:0/1405623430 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fb4c40047a0 con 0x5574486a13a0 2024-03-24T00:18:01.194 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:01.193+0000 7fb4ca600640 1 -- 172.21.15.118:0/1405623430 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb4c4012780 con 0x5574486a13a0 2024-03-24T00:18:01.196 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:01.194+0000 7fb4d6000640 1 --2- 172.21.15.118:0/1466774292 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7fb4c00a2800 0x7fb4c00a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:18:01.196 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:01.194+0000 7fb4e2dd9740 1 -- 172.21.15.118:0/1405623430 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5574486a13a0 msgr2=0x5574486a7c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:18:01.196 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:01.194+0000 7fb4e2dd9740 1 --2- 172.21.15.118:0/1405623430 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5574486a13a0 0x5574486a7c30 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 crypto rx=0x7fb4c4009c00 tx=0x7fb4c40117b0 comp rx=0 tx=0).stop 2024-03-24T00:18:01.196 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:01.194+0000 7fb4d6000640 1 -- 172.21.15.118:0/1466774292 <== mon.1 v2:172.21.15.193:3300/0 5 ==== osd_map(90..90 src has 1..90) v4 ==== 8277+0+0 (secure 0 0 0) 0x7fb4cc0cf390 con 0x7fb4d81005a0 2024-03-24T00:18:01.196 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:01.194+0000 7fb4de000640 1 --2- 172.21.15.118:0/1466774292 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7fb4c00a2800 0x7fb4c00a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:18:01.196 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:01.194+0000 7fb4e2dd9740 1 -- 172.21.15.118:0/1405623430 shutdown_connections 2024-03-24T00:18:01.196 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:01.194+0000 7fb4e2dd9740 1 --2- 172.21.15.118:0/1405623430 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5574486a13a0 0x5574486a7c30 unknown :-1 s=CLOSED pgs=304 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:18:01.196 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:01.194+0000 7fb4e2dd9740 1 --2- 172.21.15.118:0/1405623430 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5574486a09e0 0x5574486a0e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:18:01.197 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:01.194+0000 7fb4e2dd9740 1 --2- 172.21.15.118:0/1405623430 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55744869f810 0x55744869fc30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:18:01.197 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:01.194+0000 7fb4e2dd9740 1 -- 172.21.15.118:0/1405623430 >> 172.21.15.118:0/1405623430 conn(0x55744869ac00 msgr2=0x55744869b000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:18:01.197 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:01.194+0000 7fb4e2dd9740 1 -- 172.21.15.118:0/1405623430 shutdown_connections 2024-03-24T00:18:01.197 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:01.194+0000 7fb4e2dd9740 1 -- 172.21.15.118:0/1405623430 wait complete. 2024-03-24T00:18:01.197 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:01.195+0000 7fb4e2dd9740 1 -- start start 2024-03-24T00:18:01.197 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:01.195+0000 7fb4de000640 1 --2- 172.21.15.118:0/1466774292 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7fb4c00a2800 0x7fb4c00a4cc0 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 crypto rx=0x7fb4d00096c0 tx=0x7fb4d002c8a0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:18:01.197 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:01.195+0000 7fb4e2dd9740 1 Processor -- start 2024-03-24T00:18:01.197 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:01.196+0000 7fb4e2dd9740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55744869f810 0x5574486a9a50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:18:01.197 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:01.196+0000 7fb4e2dd9740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5574486a09e0 0x5574486a9f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:18:01.197 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:01.196+0000 7fb4e2dd9740 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5574486a13a0 0x5574486b0fd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:18:01.197 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:01.196+0000 7fb4e2dd9740 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x5574486a83b0 con 0x5574486a13a0 2024-03-24T00:18:01.197 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:01.196+0000 7fb4e2dd9740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x5574485c00d0 con 0x5574486a09e0 2024-03-24T00:18:01.197 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:01.196+0000 7fb4e2dd9740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x5574486a8530 con 0x55744869f810 2024-03-24T00:18:01.197 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:01.196+0000 7fb4e2dd9740 10 client.0 _ll_register_callbacks cb 0x5574485c4e80 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T00:18:01.198 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:18:00 smithi118 ceph-mon[31535]: pgmap v467: 97 pgs: 97 active+clean; 18 MiB data, 410 MiB used, 1.0 TiB / 1.0 TiB avail; 323 KiB/s rd, 372 KiB/s wr, 131 op/s 2024-03-24T00:18:01.201 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:01.196+0000 7fb4dcc00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5574486a09e0 0x5574486a9f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:18:01.201 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:01.196+0000 7fb4dcc00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5574486a09e0 0x5574486a9f90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.118:38072/0 (socket says 172.21.15.118:38072) 2024-03-24T00:18:01.201 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:01.196+0000 7fb4dcc00640 1 -- 172.21.15.118:0/855512797 learned_addr learned my addr 172.21.15.118:0/855512797 (peer_addr_for_me v2:172.21.15.118:0/0) 2024-03-24T00:18:01.201 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:01.196+0000 7fb4dd600640 1 --2- 172.21.15.118:0/855512797 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55744869f810 0x5574486a9a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:18:01.201 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:01.196+0000 7fb4de000640 1 --2- 172.21.15.118:0/855512797 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5574486a13a0 0x5574486b0fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:18:01.201 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:01.196+0000 7fb4dcc00640 1 -- 172.21.15.118:0/855512797 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55744869f810 msgr2=0x5574486a9a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:18:01.201 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:01.196+0000 7fb4dcc00640 1 --2- 172.21.15.118:0/855512797 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55744869f810 0x5574486a9a50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:18:01.201 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:01.196+0000 7fb4dcc00640 1 -- 172.21.15.118:0/855512797 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5574486a13a0 msgr2=0x5574486b0fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:18:01.201 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:01.196+0000 7fb4dcc00640 1 --2- 172.21.15.118:0/855512797 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5574486a13a0 0x5574486b0fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:18:01.201 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:01.196+0000 7fb4dcc00640 1 -- 172.21.15.118:0/855512797 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5574483183f0 con 0x5574486a09e0 2024-03-24T00:18:01.201 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:01.196+0000 7fb4dd600640 1 --2- 172.21.15.118:0/855512797 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55744869f810 0x5574486a9a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T00:18:01.201 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:01.197+0000 7fb4dcc00640 1 --2- 172.21.15.118:0/855512797 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5574486a09e0 0x5574486a9f90 secure :-1 s=READY pgs=313 cs=0 l=1 rev1=1 crypto rx=0x7fb4c4004e70 tx=0x7fb4c4004ea0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e7109bb84a7fea56 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:18:01.202 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:01.197+0000 7fb4bea00640 10 client.0 ms_handle_connect on v2:172.21.15.189:3300/0 2024-03-24T00:18:01.202 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:01.197+0000 7fb4bea00640 1 -- 172.21.15.118:0/855512797 <== mon.2 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb4c4028020 con 0x5574486a09e0 2024-03-24T00:18:01.202 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:01.197+0000 7fb4bea00640 1 -- 172.21.15.118:0/855512797 <== mon.2 v2:172.21.15.189:3300/0 2 ==== osd_map(90..90 src has 1..90) v4 ==== 8277+0+0 (secure 0 0 0) 0x7fb4c40051b0 con 0x5574486a09e0 2024-03-24T00:18:01.202 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:01.197+0000 7fb4e2dd9740 1 -- 172.21.15.118:0/855512797 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fb4c40047a0 con 0x5574486a09e0 2024-03-24T00:18:01.202 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:01.197+0000 7fb4bea00640 1 -- 172.21.15.118:0/855512797 <== mon.2 v2:172.21.15.189:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) v2 ==== 24+0+0 (secure 0 0 0) 0x7fb4c4033050 con 0x5574486a09e0 2024-03-24T00:18:01.202 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:01.197+0000 7fb4e2dd9740 10 client.26042 fetch_fsmap learned FSMap version 28 2024-03-24T00:18:01.202 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:01.197+0000 7fb4e2dd9740 1 -- 172.21.15.118:0/855512797 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) v3 -- 0x5574483e27d0 con 0x5574486a09e0 2024-03-24T00:18:01.202 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:01.198+0000 7fb4bea00640 1 -- 172.21.15.118:0/855512797 <== mon.2 v2:172.21.15.189:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fb4c40047a0 con 0x5574486a09e0 2024-03-24T00:18:01.202 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:01.198+0000 7fb4bea00640 1 -- 172.21.15.118:0/855512797 <== mon.2 v2:172.21.15.189:3300/0 5 ==== fsmap(e 28) v1 ==== 7934+0+0 (secure 0 0 0) 0x7fb4c4024460 con 0x5574486a09e0 2024-03-24T00:18:01.202 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:01.198+0000 7fb4e2dd9740 10 client.26042 fetch_fsmap finished waiting for FSMap version 28 2024-03-24T00:18:01.202 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:01.198+0000 7fb4e2dd9740 10 client.26042 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.e 2024-03-24T00:18:01.202 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:01.198+0000 7fb4e2dd9740 20 client.26042 populate_metadata read hostname 'smithi118' 2024-03-24T00:18:01.203 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:01.198+0000 7fb4e2dd9740 1 --2- 172.21.15.118:0/855512797 >> [v2:172.21.15.189:6834/599473205,v1:172.21.15.189:6835/599473205] conn(0x5574486b5230 0x5574486b7620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:18:01.203 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:01.198+0000 7fb4e2dd9740 4 client.26042 mds_command: new command op to 24488 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-24T00:18:01.203 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:01.198+0000 7fb4e2dd9740 1 -- 172.21.15.118:0/855512797 --> [v2:172.21.15.189:6834/599473205,v1:172.21.15.189:6835/599473205] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x557448611800 con 0x5574486b5230 2024-03-24T00:18:01.203 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:01.198+0000 7fb4dd600640 1 --2- 172.21.15.118:0/855512797 >> [v2:172.21.15.189:6834/599473205,v1:172.21.15.189:6835/599473205] conn(0x5574486b5230 0x5574486b7620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:18:01.203 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:01.199+0000 7fb4dd600640 1 --2- 172.21.15.118:0/855512797 >> [v2:172.21.15.189:6834/599473205,v1:172.21.15.189:6835/599473205] conn(0x5574486b5230 0x5574486b7620 crc :-1 s=READY pgs=28 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=7d6c2a691afa07e5 server_cookie=9106d8c06ce78973 in_seq=0 out_seq=0 2024-03-24T00:18:01.203 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:01.199+0000 7fb4bea00640 10 client.26042 ms_handle_connect on v2:172.21.15.189:6834/599473205 2024-03-24T00:18:01.203 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:01.200+0000 7fb4bea00640 1 -- 172.21.15.118:0/855512797 <== mds.4 v2:172.21.15.189:6834/599473205 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x557448611800 con 0x5574486b5230 2024-03-24T00:18:01.203 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:01.200+0000 7fb4bea00640 10 client.26042 handle_command_reply: tid=0 2024-03-24T00:18:01.203 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:01.200+0000 7fb4e2dd9740 1 client.26042 shutdown 2024-03-24T00:18:01.203 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:01.200+0000 7fb4e2dd9740 1 -- 172.21.15.118:0/855512797 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5574486a09e0 msgr2=0x5574486a9f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:18:01.203 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:01.200+0000 7fb4e2dd9740 1 --2- 172.21.15.118:0/855512797 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5574486a09e0 0x5574486a9f90 secure :-1 s=READY pgs=313 cs=0 l=1 rev1=1 crypto rx=0x7fb4c4004e70 tx=0x7fb4c4004ea0 comp rx=0 tx=0).stop 2024-03-24T00:18:01.203 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:01.201+0000 7fb4e2dd9740 1 -- 172.21.15.118:0/855512797 shutdown_connections 2024-03-24T00:18:01.203 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:01.201+0000 7fb4e2dd9740 1 --2- 172.21.15.118:0/855512797 >> [v2:172.21.15.189:6834/599473205,v1:172.21.15.189:6835/599473205] conn(0x5574486b5230 0x5574486b7620 crc :-1 s=READY pgs=28 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:18:01.204 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:01.201+0000 7fb4e2dd9740 1 --2- 172.21.15.118:0/855512797 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5574486a13a0 0x5574486b0fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:18:01.204 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:01.201+0000 7fb4e2dd9740 1 --2- 172.21.15.118:0/855512797 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5574486a09e0 0x5574486a9f90 unknown :-1 s=CLOSED pgs=313 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:18:01.204 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:01.201+0000 7fb4e2dd9740 1 --2- 172.21.15.118:0/855512797 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55744869f810 0x5574486a9a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:18:01.204 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:01.201+0000 7fb4bea00640 0 client.26042 ms_handle_reset on v2:172.21.15.189:6834/599473205 2024-03-24T00:18:01.204 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:01.201+0000 7fb4e2dd9740 1 -- 172.21.15.118:0/855512797 >> 172.21.15.118:0/855512797 conn(0x55744869ac00 msgr2=0x557448699eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:18:01.204 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:01.201+0000 7fb4e2dd9740 1 -- 172.21.15.118:0/855512797 shutdown_connections 2024-03-24T00:18:01.204 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:01.201+0000 7fb4e2dd9740 1 -- 172.21.15.118:0/855512797 wait complete. 2024-03-24T00:18:01.204 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:01.201+0000 7fb4e2dd9740 20 client.26042 trim_cache size 0 max 16384 2024-03-24T00:18:01.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:18:00 smithi189 ceph-mon[36187]: pgmap v467: 97 pgs: 97 active+clean; 18 MiB data, 410 MiB used, 1.0 TiB / 1.0 TiB avail; 323 KiB/s rd, 372 KiB/s wr, 131 op/s 2024-03-24T00:18:01.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:18:00 smithi193 ceph-mon[36878]: pgmap v467: 97 pgs: 97 active+clean; 18 MiB data, 410 MiB used, 1.0 TiB / 1.0 TiB avail; 323 KiB/s rd, 372 KiB/s wr, 131 op/s 2024-03-24T00:18:01.216 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:01.215+0000 7fb4e2dd9740 1 Processor -- start 2024-03-24T00:18:01.217 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:01.216+0000 7fb4e2dd9740 1 -- start start 2024-03-24T00:18:01.217 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:01.216+0000 7fb4e2dd9740 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55744869f810 0x5574486a3c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:18:01.217 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:01.216+0000 7fb4e2dd9740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5574486a09e0 0x5574486b0f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:18:01.217 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:01.216+0000 7fb4e2dd9740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5574486a13a0 0x5574486a4150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:18:01.217 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:01.216+0000 7fb4e2dd9740 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb4d81aab30 con 0x55744869f810 2024-03-24T00:18:01.217 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:01.216+0000 7fb4e2dd9740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb4d81ac4b0 con 0x5574486a13a0 2024-03-24T00:18:01.217 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:01.216+0000 7fb4e2dd9740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb4d81ab190 con 0x5574486a09e0 2024-03-24T00:18:01.217 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:01.216+0000 7fb4dcc00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5574486a09e0 0x5574486b0f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:18:01.217 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:01.216+0000 7fb4dd600640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55744869f810 0x5574486a3c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:18:01.217 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:01.216+0000 7fb4dcc00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5574486a09e0 0x5574486b0f90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.118:52994/0 (socket says 172.21.15.118:52994) 2024-03-24T00:18:01.217 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:01.216+0000 7fb4dcc00640 1 -- 172.21.15.118:0/231817458 learned_addr learned my addr 172.21.15.118:0/231817458 (peer_addr_for_me v2:172.21.15.118:0/0) 2024-03-24T00:18:01.217 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:01.216+0000 7fb4de000640 1 --2- 172.21.15.118:0/231817458 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5574486a13a0 0x5574486a4150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:18:01.218 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:01.216+0000 7fb4dcc00640 1 -- 172.21.15.118:0/231817458 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5574486a13a0 msgr2=0x5574486a4150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:18:01.218 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:01.216+0000 7fb4dcc00640 1 --2- 172.21.15.118:0/231817458 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5574486a13a0 0x5574486a4150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:18:01.218 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:01.216+0000 7fb4dcc00640 1 -- 172.21.15.118:0/231817458 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55744869f810 msgr2=0x5574486a3c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:18:01.218 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:01.216+0000 7fb4dcc00640 1 --2- 172.21.15.118:0/231817458 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55744869f810 0x5574486a3c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:18:01.218 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:01.216+0000 7fb4dcc00640 1 -- 172.21.15.118:0/231817458 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5574486a0460 con 0x5574486a09e0 2024-03-24T00:18:01.218 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:01.217+0000 7fb4dd600640 1 --2- 172.21.15.118:0/231817458 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55744869f810 0x5574486a3c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T00:18:01.218 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:01.217+0000 7fb4dcc00640 1 --2- 172.21.15.118:0/231817458 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5574486a09e0 0x5574486b0f90 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 crypto rx=0x7fb4c40117f0 tx=0x7fb4c4004d70 comp rx=0 tx=0).ready entity=mon.1 client_cookie=7ca20cd2f3697614 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:18:01.218 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:01.217+0000 7fb4c9c00640 1 -- 172.21.15.118:0/231817458 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb4c403c070 con 0x5574486a09e0 2024-03-24T00:18:01.218 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:01.217+0000 7fb4c9c00640 1 -- 172.21.15.118:0/231817458 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fb4c4004660 con 0x5574486a09e0 2024-03-24T00:18:01.218 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:01.217+0000 7fb4c9c00640 1 -- 172.21.15.118:0/231817458 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb4c4028040 con 0x5574486a09e0 2024-03-24T00:18:01.223 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:01.218+0000 7fb4e2dd9740 1 -- 172.21.15.118:0/231817458 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5574486a09e0 msgr2=0x5574486b0f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:18:01.223 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:01.218+0000 7fb4e2dd9740 1 --2- 172.21.15.118:0/231817458 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5574486a09e0 0x5574486b0f90 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 crypto rx=0x7fb4c40117f0 tx=0x7fb4c4004d70 comp rx=0 tx=0).stop 2024-03-24T00:18:01.223 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:01.218+0000 7fb4e2dd9740 1 -- 172.21.15.118:0/231817458 shutdown_connections 2024-03-24T00:18:01.223 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:01.218+0000 7fb4e2dd9740 1 --2- 172.21.15.118:0/231817458 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5574486a13a0 0x5574486a4150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:18:01.223 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:01.218+0000 7fb4e2dd9740 1 --2- 172.21.15.118:0/231817458 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5574486a09e0 0x5574486b0f90 unknown :-1 s=CLOSED pgs=305 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:18:01.223 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:01.218+0000 7fb4e2dd9740 1 --2- 172.21.15.118:0/231817458 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55744869f810 0x5574486a3c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:18:01.223 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:01.218+0000 7fb4e2dd9740 1 -- 172.21.15.118:0/231817458 >> 172.21.15.118:0/231817458 conn(0x55744869ac00 msgr2=0x55744869de60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:18:01.223 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:01.218+0000 7fb4e2dd9740 1 -- 172.21.15.118:0/231817458 shutdown_connections 2024-03-24T00:18:01.224 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:01.218+0000 7fb4e2dd9740 1 -- 172.21.15.118:0/231817458 wait complete. 2024-03-24T00:18:01.224 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:01.218+0000 7fb4e2dd9740 1 -- start start 2024-03-24T00:18:01.224 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:01.218+0000 7fb4e2dd9740 1 Processor -- start 2024-03-24T00:18:01.224 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:01.219+0000 7fb4e2dd9740 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55744869f810 0x5574486aef10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:18:01.224 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:01.219+0000 7fb4e2dd9740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5574486a09e0 0x5574486af450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:18:01.224 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:01.219+0000 7fb4e2dd9740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5574486a13a0 0x5574486b0460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:18:01.224 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:01.219+0000 7fb4e2dd9740 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb4d81abe50 con 0x55744869f810 2024-03-24T00:18:01.224 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:01.219+0000 7fb4e2dd9740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb4d81ab7f0 con 0x5574486a09e0 2024-03-24T00:18:01.224 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:01.219+0000 7fb4e2dd9740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb4d81a8cf0 con 0x5574486a13a0 2024-03-24T00:18:01.224 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:01.219+0000 7fb4e2dd9740 10 client.0 _ll_register_callbacks cb 0x7fb4c4005e30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T00:18:01.224 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:01.219+0000 7fb4dcc00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5574486a09e0 0x5574486af450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:18:01.224 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:01.219+0000 7fb4dcc00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5574486a09e0 0x5574486af450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.118:38080/0 (socket says 172.21.15.118:38080) 2024-03-24T00:18:01.224 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:01.219+0000 7fb4dcc00640 1 -- 172.21.15.118:0/1044687147 learned_addr learned my addr 172.21.15.118:0/1044687147 (peer_addr_for_me v2:172.21.15.118:0/0) 2024-03-24T00:18:01.224 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:01.219+0000 7fb4dd600640 1 --2- 172.21.15.118:0/1044687147 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55744869f810 0x5574486aef10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:18:01.224 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:01.219+0000 7fb4de000640 1 --2- 172.21.15.118:0/1044687147 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5574486a13a0 0x5574486b0460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:18:01.225 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:01.220+0000 7fb4dcc00640 1 -- 172.21.15.118:0/1044687147 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5574486a13a0 msgr2=0x5574486b0460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:18:01.225 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:01.220+0000 7fb4dcc00640 1 --2- 172.21.15.118:0/1044687147 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5574486a13a0 0x5574486b0460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:18:01.225 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:01.220+0000 7fb4dcc00640 1 -- 172.21.15.118:0/1044687147 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55744869f810 msgr2=0x5574486aef10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:18:01.225 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:01.220+0000 7fb4dcc00640 1 --2- 172.21.15.118:0/1044687147 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55744869f810 0x5574486aef10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:18:01.225 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:01.220+0000 7fb4dcc00640 1 -- 172.21.15.118:0/1044687147 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5574486b0a50 con 0x5574486a09e0 2024-03-24T00:18:01.225 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:01.220+0000 7fb4dcc00640 1 --2- 172.21.15.118:0/1044687147 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5574486a09e0 0x5574486af450 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 crypto rx=0x7fb4c4009c00 tx=0x7fb4c40048c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=5fd2f5372677b16f server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:18:01.225 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:01.220+0000 7fb4bea00640 10 client.0 ms_handle_connect on v2:172.21.15.189:3300/0 2024-03-24T00:18:01.225 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:01.220+0000 7fb4bea00640 1 -- 172.21.15.118:0/1044687147 <== mon.2 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb4c4044060 con 0x5574486a09e0 2024-03-24T00:18:01.225 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:01.220+0000 7fb4e2dd9740 1 -- 172.21.15.118:0/1044687147 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fb4c4004660 con 0x5574486a09e0 2024-03-24T00:18:01.225 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:01.220+0000 7fb4bea00640 1 -- 172.21.15.118:0/1044687147 <== mon.2 v2:172.21.15.189:3300/0 2 ==== osd_map(90..90 src has 1..90) v4 ==== 8277+0+0 (secure 0 0 0) 0x7fb4c40051b0 con 0x5574486a09e0 2024-03-24T00:18:01.225 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:01.220+0000 7fb4bea00640 1 -- 172.21.15.118:0/1044687147 <== mon.2 v2:172.21.15.189:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) v2 ==== 24+0+0 (secure 0 0 0) 0x7fb4c403c070 con 0x5574486a09e0 2024-03-24T00:18:01.225 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:01.220+0000 7fb4e2dd9740 10 client.26048 fetch_fsmap learned FSMap version 28 2024-03-24T00:18:01.225 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:01.220+0000 7fb4e2dd9740 1 -- 172.21.15.118:0/1044687147 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) v3 -- 0x5574487111f0 con 0x5574486a09e0 2024-03-24T00:18:01.225 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:01.221+0000 7fb4bea00640 1 -- 172.21.15.118:0/1044687147 <== mon.2 v2:172.21.15.189:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fb4c4004660 con 0x5574486a09e0 2024-03-24T00:18:01.226 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:01.221+0000 7fb4bea00640 1 -- 172.21.15.118:0/1044687147 <== mon.2 v2:172.21.15.189:3300/0 5 ==== fsmap(e 28) v1 ==== 7934+0+0 (secure 0 0 0) 0x7fb4c4012c30 con 0x5574486a09e0 2024-03-24T00:18:01.226 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:01.221+0000 7fb4e2dd9740 10 client.26048 fetch_fsmap finished waiting for FSMap version 28 2024-03-24T00:18:01.226 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:01.221+0000 7fb4e2dd9740 10 client.26048 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.e 2024-03-24T00:18:01.226 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:01.221+0000 7fb4e2dd9740 20 client.26048 populate_metadata read hostname 'smithi118' 2024-03-24T00:18:01.226 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:01.221+0000 7fb4e2dd9740 1 --2- 172.21.15.118:0/1044687147 >> [v2:172.21.15.189:6834/599473205,v1:172.21.15.189:6835/599473205] conn(0x5574487119d0 0x557448713df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:18:01.226 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:01.221+0000 7fb4e2dd9740 4 client.26048 mds_command: new command op to 24488 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-24T00:18:01.226 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:01.221+0000 7fb4e2dd9740 1 -- 172.21.15.118:0/1044687147 --> [v2:172.21.15.189:6834/599473205,v1:172.21.15.189:6835/599473205] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x55744860bb50 con 0x5574487119d0 2024-03-24T00:18:01.226 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:01.221+0000 7fb4dd600640 1 --2- 172.21.15.118:0/1044687147 >> [v2:172.21.15.189:6834/599473205,v1:172.21.15.189:6835/599473205] conn(0x5574487119d0 0x557448713df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:18:01.226 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:01.222+0000 7fb4dd600640 1 --2- 172.21.15.118:0/1044687147 >> [v2:172.21.15.189:6834/599473205,v1:172.21.15.189:6835/599473205] conn(0x5574487119d0 0x557448713df0 crc :-1 s=READY pgs=29 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=50cfe46a3dfbdfc7 server_cookie=aa22ac536c374658 in_seq=0 out_seq=0 2024-03-24T00:18:01.226 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:01.222+0000 7fb4bea00640 10 client.26048 ms_handle_connect on v2:172.21.15.189:6834/599473205 2024-03-24T00:18:01.226 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:01.222+0000 7fb4bea00640 1 -- 172.21.15.118:0/1044687147 <== mds.4 v2:172.21.15.189:6834/599473205 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x55744860bb50 con 0x5574487119d0 2024-03-24T00:18:01.226 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:01.222+0000 7fb4bea00640 10 client.26048 handle_command_reply: tid=0 2024-03-24T00:18:01.226 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:01.222+0000 7fb4e2dd9740 1 client.26048 shutdown 2024-03-24T00:18:01.226 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:01.222+0000 7fb4e2dd9740 1 -- 172.21.15.118:0/1044687147 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5574486a09e0 msgr2=0x5574486af450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:18:01.227 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:01.222+0000 7fb4e2dd9740 1 --2- 172.21.15.118:0/1044687147 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5574486a09e0 0x5574486af450 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 crypto rx=0x7fb4c4009c00 tx=0x7fb4c40048c0 comp rx=0 tx=0).stop 2024-03-24T00:18:01.227 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:01.222+0000 7fb4e2dd9740 1 -- 172.21.15.118:0/1044687147 shutdown_connections 2024-03-24T00:18:01.227 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:01.222+0000 7fb4e2dd9740 1 --2- 172.21.15.118:0/1044687147 >> [v2:172.21.15.189:6834/599473205,v1:172.21.15.189:6835/599473205] conn(0x5574487119d0 0x557448713df0 crc :-1 s=READY pgs=29 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:18:01.227 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:01.222+0000 7fb4e2dd9740 1 --2- 172.21.15.118:0/1044687147 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5574486a13a0 0x5574486b0460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:18:01.227 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:01.222+0000 7fb4e2dd9740 1 --2- 172.21.15.118:0/1044687147 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5574486a09e0 0x5574486af450 unknown :-1 s=CLOSED pgs=314 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:18:01.227 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:01.222+0000 7fb4e2dd9740 1 --2- 172.21.15.118:0/1044687147 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55744869f810 0x5574486aef10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:18:01.227 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:01.222+0000 7fb4bea00640 0 client.26048 ms_handle_reset on v2:172.21.15.189:6834/599473205 2024-03-24T00:18:01.227 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:01.222+0000 7fb4e2dd9740 1 -- 172.21.15.118:0/1044687147 >> 172.21.15.118:0/1044687147 conn(0x55744869ac00 msgr2=0x557448699620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:18:01.227 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:01.222+0000 7fb4e2dd9740 1 -- 172.21.15.118:0/1044687147 shutdown_connections 2024-03-24T00:18:01.227 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:01.223+0000 7fb4e2dd9740 1 -- 172.21.15.118:0/1044687147 wait complete. 2024-03-24T00:18:01.227 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:01.223+0000 7fb4e2dd9740 20 client.26048 trim_cache size 0 max 16384 2024-03-24T00:18:01.228 INFO:teuthology.orchestra.run.smithi118.stdout:[] 2024-03-24T00:18:01.228 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:01.223+0000 7fb4e0000640 1 -- 172.21.15.118:0/1466774292 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7fb4c00a2800 msgr2=0x7fb4c00a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:18:01.228 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:01.223+0000 7fb4e0000640 1 --2- 172.21.15.118:0/1466774292 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7fb4c00a2800 0x7fb4c00a4cc0 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 crypto rx=0x7fb4d00096c0 tx=0x7fb4d002c8a0 comp rx=0 tx=0).stop 2024-03-24T00:18:01.228 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:01.223+0000 7fb4e0000640 1 -- 172.21.15.118:0/1466774292 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fb4d81005a0 msgr2=0x7fb4d8199bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:18:01.229 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:01.223+0000 7fb4e0000640 1 --2- 172.21.15.118:0/1466774292 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fb4d81005a0 0x7fb4d8199bf0 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 crypto rx=0x7fb4cc00ebc0 tx=0x7fb4cc00c6a0 comp rx=0 tx=0).stop 2024-03-24T00:18:01.229 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:01.223+0000 7fb4e0000640 1 -- 172.21.15.118:0/1466774292 shutdown_connections 2024-03-24T00:18:01.229 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:01.223+0000 7fb4e0000640 1 --2- 172.21.15.118:0/1466774292 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7fb4c00a2800 0x7fb4c00a4cc0 unknown :-1 s=CLOSED pgs=310 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:18:01.229 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:01.223+0000 7fb4e0000640 1 --2- 172.21.15.118:0/1466774292 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb4d8102150 0x7fb4d81a1130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:18:01.229 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:01.223+0000 7fb4e0000640 1 --2- 172.21.15.118:0/1466774292 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb4d8101790 0x7fb4d819a130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:18:01.229 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:01.223+0000 7fb4e0000640 1 --2- 172.21.15.118:0/1466774292 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fb4d81005a0 0x7fb4d8199bf0 unknown :-1 s=CLOSED pgs=303 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:18:01.229 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:01.223+0000 7fb4e0000640 1 -- 172.21.15.118:0/1466774292 >> 172.21.15.118:0/1466774292 conn(0x7fb4d80fbb30 msgr2=0x7fb4d80fd5e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:18:01.229 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:01.223+0000 7fb4e0000640 1 -- 172.21.15.118:0/1466774292 shutdown_connections 2024-03-24T00:18:01.229 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:01.224+0000 7fb4e0000640 1 -- 172.21.15.118:0/1466774292 wait complete. 2024-03-24T00:18:01.240 INFO:teuthology.orchestra.run:Running command with timeout 120 2024-03-24T00:18:01.240 DEBUG:teuthology.orchestra.run.smithi118:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:3 damage ls 2024-03-24T00:18:01.450 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:01.450+0000 7f101ee00640 1 -- 172.21.15.118:0/3435572364 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f10180fff10 msgr2=0x7f1018102340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:18:01.451 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:01.450+0000 7f101ee00640 1 --2- 172.21.15.118:0/3435572364 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f10180fff10 0x7f1018102340 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 crypto rx=0x7f1008009900 tx=0x7f100802bc30 comp rx=0 tx=0).stop 2024-03-24T00:18:01.451 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:01.451+0000 7f101ee00640 1 -- 172.21.15.118:0/3435572364 shutdown_connections 2024-03-24T00:18:01.451 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:01.451+0000 7f101ee00640 1 --2- 172.21.15.118:0/3435572364 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f1018105230 0x7f10181056b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:18:01.451 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:01.451+0000 7f101ee00640 1 --2- 172.21.15.118:0/3435572364 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f1018102880 0x7f1018104cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:18:01.451 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:01.451+0000 7f101ee00640 1 --2- 172.21.15.118:0/3435572364 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f10180fff10 0x7f1018102340 unknown :-1 s=CLOSED pgs=315 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:18:01.451 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:01.451+0000 7f101ee00640 1 -- 172.21.15.118:0/3435572364 >> 172.21.15.118:0/3435572364 conn(0x7f10180fbb10 msgr2=0x7f10180fdf50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:18:01.451 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:01.451+0000 7f101ee00640 1 -- 172.21.15.118:0/3435572364 shutdown_connections 2024-03-24T00:18:01.451 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:01.451+0000 7f101ee00640 1 -- 172.21.15.118:0/3435572364 wait complete. 2024-03-24T00:18:01.453 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:01.452+0000 7f101ee00640 1 Processor -- start 2024-03-24T00:18:01.453 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:01.452+0000 7f101ee00640 1 -- start start 2024-03-24T00:18:01.453 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:01.452+0000 7f101ee00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f10180fff10 0x7f101819e050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:18:01.454 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:01.452+0000 7f101ee00640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f1018102880 0x7f101819e590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:18:01.454 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:01.452+0000 7f101ee00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f1018105230 0x7f10181a55b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:18:01.454 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:01.452+0000 7f101ee00640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1018113940 con 0x7f1018102880 2024-03-24T00:18:01.454 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:01.452+0000 7f101ee00640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f10181137c0 con 0x7f10180fff10 2024-03-24T00:18:01.454 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:01.452+0000 7f101ee00640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1018113ac0 con 0x7f1018105230 2024-03-24T00:18:01.454 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:01.452+0000 7f1017e00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f10180fff10 0x7f101819e050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:18:01.454 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:01.452+0000 7f1017400640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f1018102880 0x7f101819e590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:18:01.454 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:01.452+0000 7f1017e00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f10180fff10 0x7f101819e050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.118:38104/0 (socket says 172.21.15.118:38104) 2024-03-24T00:18:01.454 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:01.452+0000 7f1017e00640 1 -- 172.21.15.118:0/1530969945 learned_addr learned my addr 172.21.15.118:0/1530969945 (peer_addr_for_me v2:172.21.15.118:0/0) 2024-03-24T00:18:01.454 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:01.452+0000 7f101ce00640 1 --2- 172.21.15.118:0/1530969945 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f1018105230 0x7f10181a55b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:18:01.454 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:01.453+0000 7f1017e00640 1 -- 172.21.15.118:0/1530969945 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f1018105230 msgr2=0x7f10181a55b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:18:01.454 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:01.453+0000 7f1017e00640 1 --2- 172.21.15.118:0/1530969945 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f1018105230 0x7f10181a55b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:18:01.454 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:01.453+0000 7f1017e00640 1 -- 172.21.15.118:0/1530969945 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f1018102880 msgr2=0x7f101819e590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:18:01.455 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:01.453+0000 7f1017e00640 1 --2- 172.21.15.118:0/1530969945 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f1018102880 0x7f101819e590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:18:01.455 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:01.453+0000 7f1017e00640 1 -- 172.21.15.118:0/1530969945 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f10181a5c20 con 0x7f10180fff10 2024-03-24T00:18:01.455 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:01.453+0000 7f1017e00640 1 --2- 172.21.15.118:0/1530969945 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f10180fff10 0x7f101819e050 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 crypto rx=0x7f1008000c00 tx=0x7f100802c650 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:18:01.455 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:01.454+0000 7f1014c00640 1 -- 172.21.15.118:0/1530969945 <== mon.2 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1008043020 con 0x7f10180fff10 2024-03-24T00:18:01.455 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:01.454+0000 7f1014c00640 1 -- 172.21.15.118:0/1530969945 <== mon.2 v2:172.21.15.189:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f10080054e0 con 0x7f10180fff10 2024-03-24T00:18:01.455 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:01.454+0000 7f101ee00640 1 -- 172.21.15.118:0/1530969945 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f10181a5eb0 con 0x7f10180fff10 2024-03-24T00:18:01.455 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:01.454+0000 7f1014c00640 1 -- 172.21.15.118:0/1530969945 <== mon.2 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1008003bd0 con 0x7f10180fff10 2024-03-24T00:18:01.455 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:01.454+0000 7f101ee00640 1 -- 172.21.15.118:0/1530969945 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f10181a6370 con 0x7f10180fff10 2024-03-24T00:18:01.456 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:01.457+0000 7f1014c00640 1 -- 172.21.15.118:0/1530969945 <== mon.2 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 20) v1 ==== 137291+0+0 (secure 0 0 0) 0x7f1008046020 con 0x7f10180fff10 2024-03-24T00:18:01.457 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:01.458+0000 7f1021bfc740 1 Processor -- start 2024-03-24T00:18:01.459 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:01.458+0000 7f1021bfc740 1 -- start start 2024-03-24T00:18:01.460 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:01.458+0000 7f1021bfc740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55c3b0b7f810 0x55c3b0b7fc30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:18:01.460 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:01.458+0000 7f1021bfc740 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55c3b0b809e0 0x55c3b0b80e60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:18:01.460 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:01.459+0000 7f1021bfc740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55c3b0b813a0 0x55c3b0b87c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:18:01.460 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:01.459+0000 7f1021bfc740 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c3b0ae42c0 con 0x55c3b0b809e0 2024-03-24T00:18:01.460 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:01.459+0000 7f1021bfc740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c3b0a8cc90 con 0x55c3b0b7f810 2024-03-24T00:18:01.460 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:01.459+0000 7f1021bfc740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c3b0ae0360 con 0x55c3b0b813a0 2024-03-24T00:18:01.460 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:01.459+0000 7f101ce00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55c3b0b7f810 0x55c3b0b7fc30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:18:01.460 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:01.459+0000 7f101ce00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55c3b0b7f810 0x55c3b0b7fc30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.118:38106/0 (socket says 172.21.15.118:38106) 2024-03-24T00:18:01.460 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:01.459+0000 7f101ce00640 1 -- 172.21.15.118:0/327917250 learned_addr learned my addr 172.21.15.118:0/327917250 (peer_addr_for_me v2:172.21.15.118:0/0) 2024-03-24T00:18:01.460 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:01.459+0000 7f1017400640 1 --2- 172.21.15.118:0/327917250 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55c3b0b813a0 0x55c3b0b87c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:18:01.460 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:01.459+0000 7f1017e00640 1 --2- 172.21.15.118:0/327917250 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55c3b0b809e0 0x55c3b0b80e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:18:01.460 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:01.459+0000 7f1014c00640 1 --2- 172.21.15.118:0/1530969945 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7f0ffc0a2880 0x7f0ffc0a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:18:01.460 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:01.459+0000 7f1017400640 1 -- 172.21.15.118:0/327917250 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55c3b0b7f810 msgr2=0x55c3b0b7fc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:18:01.461 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:01.459+0000 7f1017400640 1 --2- 172.21.15.118:0/327917250 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55c3b0b7f810 0x55c3b0b7fc30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:18:01.461 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:01.459+0000 7f1017400640 1 -- 172.21.15.118:0/327917250 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55c3b0b809e0 msgr2=0x55c3b0b80e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:18:01.461 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:01.459+0000 7f1017400640 1 --2- 172.21.15.118:0/327917250 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55c3b0b809e0 0x55c3b0b80e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:18:01.461 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:01.459+0000 7f1017400640 1 -- 172.21.15.118:0/327917250 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55c3b0ab1080 con 0x55c3b0b813a0 2024-03-24T00:18:01.461 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:01.459+0000 7f1017e00640 1 --2- 172.21.15.118:0/327917250 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55c3b0b809e0 0x55c3b0b80e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T00:18:01.461 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:01.459+0000 7f1014c00640 1 -- 172.21.15.118:0/1530969945 <== mon.2 v2:172.21.15.189:3300/0 5 ==== osd_map(90..90 src has 1..90) v4 ==== 8277+0+0 (secure 0 0 0) 0x7f10080f0ae0 con 0x7f10180fff10 2024-03-24T00:18:01.461 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:01.460+0000 7f101ce00640 1 --2- 172.21.15.118:0/1530969945 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7f0ffc0a2880 0x7f0ffc0a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:18:01.461 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:01.460+0000 7f1017400640 1 --2- 172.21.15.118:0/327917250 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55c3b0b813a0 0x55c3b0b87c30 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 crypto rx=0x7f1004009bf0 tx=0x7f1004011810 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c5e124e168f76926 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:18:01.461 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:01.460+0000 7f100d600640 1 -- 172.21.15.118:0/327917250 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1004009da0 con 0x55c3b0b813a0 2024-03-24T00:18:01.461 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:01.460+0000 7f100d600640 1 -- 172.21.15.118:0/327917250 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f1004005020 con 0x55c3b0b813a0 2024-03-24T00:18:01.461 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:01.460+0000 7f100d600640 1 -- 172.21.15.118:0/327917250 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1004004ae0 con 0x55c3b0b813a0 2024-03-24T00:18:01.464 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:01.461+0000 7f1021bfc740 1 -- 172.21.15.118:0/327917250 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55c3b0b813a0 msgr2=0x55c3b0b87c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:18:01.464 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:01.461+0000 7f101ce00640 1 --2- 172.21.15.118:0/1530969945 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7f0ffc0a2880 0x7f0ffc0a4d40 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 crypto rx=0x55c3b0b808e0 tx=0x7f1010004a40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:18:01.464 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:01.461+0000 7f1021bfc740 1 --2- 172.21.15.118:0/327917250 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55c3b0b813a0 0x55c3b0b87c30 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 crypto rx=0x7f1004009bf0 tx=0x7f1004011810 comp rx=0 tx=0).stop 2024-03-24T00:18:01.464 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:01.461+0000 7f1021bfc740 1 -- 172.21.15.118:0/327917250 shutdown_connections 2024-03-24T00:18:01.465 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:01.461+0000 7f1021bfc740 1 --2- 172.21.15.118:0/327917250 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55c3b0b813a0 0x55c3b0b87c30 unknown :-1 s=CLOSED pgs=306 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:18:01.465 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:01.461+0000 7f1021bfc740 1 --2- 172.21.15.118:0/327917250 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55c3b0b809e0 0x55c3b0b80e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:18:01.465 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:01.461+0000 7f1021bfc740 1 --2- 172.21.15.118:0/327917250 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55c3b0b7f810 0x55c3b0b7fc30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:18:01.465 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:01.461+0000 7f1021bfc740 1 -- 172.21.15.118:0/327917250 >> 172.21.15.118:0/327917250 conn(0x55c3b0b7ac00 msgr2=0x55c3b0b7b000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:18:01.465 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:01.461+0000 7f1021bfc740 1 -- 172.21.15.118:0/327917250 shutdown_connections 2024-03-24T00:18:01.465 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:01.461+0000 7f1021bfc740 1 -- 172.21.15.118:0/327917250 wait complete. 2024-03-24T00:18:01.465 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:01.462+0000 7f1021bfc740 1 -- start start 2024-03-24T00:18:01.465 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:01.462+0000 7f1021bfc740 1 Processor -- start 2024-03-24T00:18:01.465 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:01.462+0000 7f1021bfc740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55c3b0b7f810 0x55c3b0b89a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:18:01.465 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:01.462+0000 7f1021bfc740 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55c3b0b809e0 0x55c3b0b89f60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:18:01.465 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:01.462+0000 7f1021bfc740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55c3b0b813a0 0x55c3b0b90fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:18:01.465 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:01.462+0000 7f1021bfc740 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c3b0b883b0 con 0x55c3b0b809e0 2024-03-24T00:18:01.465 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:01.462+0000 7f1021bfc740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c3b0aa00d0 con 0x55c3b0b7f810 2024-03-24T00:18:01.466 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:01.462+0000 7f1021bfc740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c3b0b88530 con 0x55c3b0b813a0 2024-03-24T00:18:01.466 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:01.462+0000 7f1021bfc740 10 client.0 _ll_register_callbacks cb 0x55c3b0aa4e80 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T00:18:01.466 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:01.462+0000 7f1017e00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55c3b0b7f810 0x55c3b0b89a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:18:01.466 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:01.462+0000 7f1017e00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55c3b0b7f810 0x55c3b0b89a20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.118:38108/0 (socket says 172.21.15.118:38108) 2024-03-24T00:18:01.466 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:01.462+0000 7f1017e00640 1 -- 172.21.15.118:0/3926348764 learned_addr learned my addr 172.21.15.118:0/3926348764 (peer_addr_for_me v2:172.21.15.118:0/0) 2024-03-24T00:18:01.466 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:01.463+0000 7f101ce00640 1 --2- 172.21.15.118:0/3926348764 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55c3b0b813a0 0x55c3b0b90fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:18:01.466 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:01.463+0000 7f1017400640 1 --2- 172.21.15.118:0/3926348764 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55c3b0b809e0 0x55c3b0b89f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:18:01.466 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:01.463+0000 7f1017e00640 1 -- 172.21.15.118:0/3926348764 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55c3b0b813a0 msgr2=0x55c3b0b90fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:18:01.466 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:01.463+0000 7f1017e00640 1 --2- 172.21.15.118:0/3926348764 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55c3b0b813a0 0x55c3b0b90fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:18:01.466 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:01.463+0000 7f1017e00640 1 -- 172.21.15.118:0/3926348764 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55c3b0b809e0 msgr2=0x55c3b0b89f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:18:01.466 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:01.463+0000 7f1017e00640 1 --2- 172.21.15.118:0/3926348764 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55c3b0b809e0 0x55c3b0b89f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:18:01.466 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:01.463+0000 7f1017e00640 1 -- 172.21.15.118:0/3926348764 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55c3b07f83f0 con 0x55c3b0b7f810 2024-03-24T00:18:01.466 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:01.463+0000 7f1017e00640 1 --2- 172.21.15.118:0/3926348764 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55c3b0b7f810 0x55c3b0b89a20 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 crypto rx=0x7f10080bd380 tx=0x7f10080bd1c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=4f6ffca8c4aadbef server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:18:01.466 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:01.463+0000 7f1002000640 10 client.0 ms_handle_connect on v2:172.21.15.189:3300/0 2024-03-24T00:18:01.467 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:01.463+0000 7f1002000640 1 -- 172.21.15.118:0/3926348764 <== mon.2 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1008005690 con 0x55c3b0b7f810 2024-03-24T00:18:01.467 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:01.463+0000 7f1002000640 1 -- 172.21.15.118:0/3926348764 <== mon.2 v2:172.21.15.189:3300/0 2 ==== osd_map(90..90 src has 1..90) v4 ==== 8277+0+0 (secure 0 0 0) 0x7f10080ee660 con 0x55c3b0b7f810 2024-03-24T00:18:01.467 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:01.463+0000 7f1021bfc740 1 -- 172.21.15.118:0/3926348764 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f1004005020 con 0x55c3b0b7f810 2024-03-24T00:18:01.467 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:01.464+0000 7f1002000640 1 -- 172.21.15.118:0/3926348764 <== mon.2 v2:172.21.15.189:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) v2 ==== 24+0+0 (secure 0 0 0) 0x7f10080bdaf0 con 0x55c3b0b7f810 2024-03-24T00:18:01.467 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:01.464+0000 7f1021bfc740 10 client.26060 fetch_fsmap learned FSMap version 28 2024-03-24T00:18:01.467 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:01.464+0000 7f1021bfc740 1 -- 172.21.15.118:0/3926348764 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) v3 -- 0x55c3b08c27d0 con 0x55c3b0b7f810 2024-03-24T00:18:01.467 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:01.464+0000 7f1002000640 1 -- 172.21.15.118:0/3926348764 <== mon.2 v2:172.21.15.189:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f1004005020 con 0x55c3b0b7f810 2024-03-24T00:18:01.467 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:01.464+0000 7f1002000640 1 -- 172.21.15.118:0/3926348764 <== mon.2 v2:172.21.15.189:3300/0 5 ==== fsmap(e 28) v1 ==== 7934+0+0 (secure 0 0 0) 0x7f10080c1110 con 0x55c3b0b7f810 2024-03-24T00:18:01.467 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:01.464+0000 7f1021bfc740 10 client.26060 fetch_fsmap finished waiting for FSMap version 28 2024-03-24T00:18:01.467 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:01.464+0000 7f1021bfc740 10 client.26060 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.f 2024-03-24T00:18:01.467 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:01.464+0000 7f1021bfc740 20 client.26060 populate_metadata read hostname 'smithi118' 2024-03-24T00:18:01.467 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:01.464+0000 7f1021bfc740 1 --2- 172.21.15.118:0/3926348764 >> [v2:172.21.15.193:6834/2055310541,v1:172.21.15.193:6835/2055310541] conn(0x55c3b0b954e0 0x55c3b0b97960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:18:01.467 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:01.464+0000 7f1021bfc740 4 client.26060 mds_command: new command op to 24499 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-24T00:18:01.467 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:01.464+0000 7f1021bfc740 1 -- 172.21.15.118:0/3926348764 --> [v2:172.21.15.193:6834/2055310541,v1:172.21.15.193:6835/2055310541] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55c3b0af1800 con 0x55c3b0b954e0 2024-03-24T00:18:01.468 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:01.465+0000 7f1017400640 1 --2- 172.21.15.118:0/3926348764 >> [v2:172.21.15.193:6834/2055310541,v1:172.21.15.193:6835/2055310541] conn(0x55c3b0b954e0 0x55c3b0b97960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:18:01.469 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:01.466+0000 7f1017400640 1 --2- 172.21.15.118:0/3926348764 >> [v2:172.21.15.193:6834/2055310541,v1:172.21.15.193:6835/2055310541] conn(0x55c3b0b954e0 0x55c3b0b97960 crc :-1 s=READY pgs=31 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=a6ed74244314b6d2 server_cookie=2590b8796a3c2974 in_seq=0 out_seq=0 2024-03-24T00:18:01.469 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:01.466+0000 7f1002000640 10 client.26060 ms_handle_connect on v2:172.21.15.193:6834/2055310541 2024-03-24T00:18:01.469 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:01.467+0000 7f1002000640 1 -- 172.21.15.118:0/3926348764 <== mds.3 v2:172.21.15.193:6834/2055310541 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x55c3b0af1800 con 0x55c3b0b954e0 2024-03-24T00:18:01.469 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:01.467+0000 7f1002000640 10 client.26060 handle_command_reply: tid=0 2024-03-24T00:18:01.469 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:01.467+0000 7f1021bfc740 1 client.26060 shutdown 2024-03-24T00:18:01.469 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:01.468+0000 7f1021bfc740 1 -- 172.21.15.118:0/3926348764 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55c3b0b7f810 msgr2=0x55c3b0b89a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:18:01.469 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:01.468+0000 7f1021bfc740 1 --2- 172.21.15.118:0/3926348764 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55c3b0b7f810 0x55c3b0b89a20 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 crypto rx=0x7f10080bd380 tx=0x7f10080bd1c0 comp rx=0 tx=0).stop 2024-03-24T00:18:01.469 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:01.468+0000 7f1021bfc740 1 -- 172.21.15.118:0/3926348764 shutdown_connections 2024-03-24T00:18:01.469 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:01.468+0000 7f1021bfc740 1 --2- 172.21.15.118:0/3926348764 >> [v2:172.21.15.193:6834/2055310541,v1:172.21.15.193:6835/2055310541] conn(0x55c3b0b954e0 0x55c3b0b97960 crc :-1 s=READY pgs=31 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:18:01.469 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:01.468+0000 7f1021bfc740 1 --2- 172.21.15.118:0/3926348764 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55c3b0b813a0 0x55c3b0b90fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:18:01.469 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:01.468+0000 7f1021bfc740 1 --2- 172.21.15.118:0/3926348764 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55c3b0b809e0 0x55c3b0b89f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:18:01.470 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:01.468+0000 7f1021bfc740 1 --2- 172.21.15.118:0/3926348764 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55c3b0b7f810 0x55c3b0b89a20 unknown :-1 s=CLOSED pgs=317 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:18:01.470 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:01.468+0000 7f1021bfc740 1 -- 172.21.15.118:0/3926348764 >> 172.21.15.118:0/3926348764 conn(0x55c3b0b7ac00 msgr2=0x55c3b0b79eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:18:01.470 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:01.468+0000 7f1002000640 0 client.26060 ms_handle_reset on v2:172.21.15.193:6834/2055310541 2024-03-24T00:18:01.470 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:01.468+0000 7f1021bfc740 1 -- 172.21.15.118:0/3926348764 shutdown_connections 2024-03-24T00:18:01.470 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:01.468+0000 7f1021bfc740 1 -- 172.21.15.118:0/3926348764 wait complete. 2024-03-24T00:18:01.470 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:01.468+0000 7f1021bfc740 20 client.26060 trim_cache size 0 max 16384 2024-03-24T00:18:01.482 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:01.482+0000 7f1021bfc740 1 Processor -- start 2024-03-24T00:18:01.482 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:01.482+0000 7f1021bfc740 1 -- start start 2024-03-24T00:18:01.482 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:01.482+0000 7f1021bfc740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55c3b0b7f810 0x55c3b0b83c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:18:01.482 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:01.482+0000 7f1021bfc740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55c3b0b809e0 0x55c3b0b90f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:18:01.482 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:01.482+0000 7f1021bfc740 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55c3b0b813a0 0x55c3b0b84150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:18:01.482 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:01.482+0000 7f1021bfc740 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f10181af020 con 0x55c3b0b813a0 2024-03-24T00:18:01.482 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:01.482+0000 7f1021bfc740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f10181b09a0 con 0x55c3b0b7f810 2024-03-24T00:18:01.482 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:01.482+0000 7f1021bfc740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f10181af680 con 0x55c3b0b809e0 2024-03-24T00:18:01.482 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:01.482+0000 7f1017e00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55c3b0b7f810 0x55c3b0b83c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:18:01.482 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:01.482+0000 7f1017e00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55c3b0b7f810 0x55c3b0b83c10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.118:38114/0 (socket says 172.21.15.118:38114) 2024-03-24T00:18:01.482 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:01.482+0000 7f1017e00640 1 -- 172.21.15.118:0/3876265140 learned_addr learned my addr 172.21.15.118:0/3876265140 (peer_addr_for_me v2:172.21.15.118:0/0) 2024-03-24T00:18:01.482 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:01.482+0000 7f101ce00640 1 --2- 172.21.15.118:0/3876265140 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55c3b0b813a0 0x55c3b0b84150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:18:01.483 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:01.482+0000 7f1017400640 1 --2- 172.21.15.118:0/3876265140 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55c3b0b809e0 0x55c3b0b90f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:18:01.488 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:01.482+0000 7f1017e00640 1 -- 172.21.15.118:0/3876265140 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55c3b0b809e0 msgr2=0x55c3b0b90f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:18:01.488 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:01.483+0000 7f1017e00640 1 --2- 172.21.15.118:0/3876265140 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55c3b0b809e0 0x55c3b0b90f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:18:01.488 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:01.483+0000 7f1017e00640 1 -- 172.21.15.118:0/3876265140 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55c3b0b813a0 msgr2=0x55c3b0b84150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:18:01.488 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:01.483+0000 7f1017e00640 1 --2- 172.21.15.118:0/3876265140 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55c3b0b813a0 0x55c3b0b84150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:18:01.488 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:01.483+0000 7f1017e00640 1 -- 172.21.15.118:0/3876265140 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55c3b0b80460 con 0x55c3b0b7f810 2024-03-24T00:18:01.488 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:01.483+0000 7f1017e00640 1 --2- 172.21.15.118:0/3876265140 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55c3b0b7f810 0x55c3b0b83c10 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 crypto rx=0x7f10080b9dc0 tx=0x7f10080bb900 comp rx=0 tx=0).ready entity=mon.2 client_cookie=5e0dc3e13b386ac2 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:18:01.488 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:01.483+0000 7f100cc00640 1 -- 172.21.15.118:0/3876265140 <== mon.2 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f10080043c0 con 0x55c3b0b7f810 2024-03-24T00:18:01.488 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:01.483+0000 7f100cc00640 1 -- 172.21.15.118:0/3876265140 <== mon.2 v2:172.21.15.189:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f10080bd3f0 con 0x55c3b0b7f810 2024-03-24T00:18:01.488 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:01.483+0000 7f100cc00640 1 -- 172.21.15.118:0/3876265140 <== mon.2 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f10080c5700 con 0x55c3b0b7f810 2024-03-24T00:18:01.488 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:01.484+0000 7f1021bfc740 1 -- 172.21.15.118:0/3876265140 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55c3b0b7f810 msgr2=0x55c3b0b83c10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:18:01.488 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:01.484+0000 7f1021bfc740 1 --2- 172.21.15.118:0/3876265140 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55c3b0b7f810 0x55c3b0b83c10 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 crypto rx=0x7f10080b9dc0 tx=0x7f10080bb900 comp rx=0 tx=0).stop 2024-03-24T00:18:01.488 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:01.484+0000 7f1021bfc740 1 -- 172.21.15.118:0/3876265140 shutdown_connections 2024-03-24T00:18:01.488 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:01.484+0000 7f1021bfc740 1 --2- 172.21.15.118:0/3876265140 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55c3b0b813a0 0x55c3b0b84150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:18:01.488 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:01.484+0000 7f1021bfc740 1 --2- 172.21.15.118:0/3876265140 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55c3b0b809e0 0x55c3b0b90f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:18:01.489 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:01.484+0000 7f1021bfc740 1 --2- 172.21.15.118:0/3876265140 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55c3b0b7f810 0x55c3b0b83c10 unknown :-1 s=CLOSED pgs=318 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:18:01.489 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:01.484+0000 7f1021bfc740 1 -- 172.21.15.118:0/3876265140 >> 172.21.15.118:0/3876265140 conn(0x55c3b0b7ac00 msgr2=0x55c3b0b7e6f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:18:01.489 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:01.484+0000 7f1021bfc740 1 -- 172.21.15.118:0/3876265140 shutdown_connections 2024-03-24T00:18:01.489 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:01.484+0000 7f1021bfc740 1 -- 172.21.15.118:0/3876265140 wait complete. 2024-03-24T00:18:01.489 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:01.484+0000 7f1021bfc740 1 -- start start 2024-03-24T00:18:01.489 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:01.484+0000 7f1021bfc740 1 Processor -- start 2024-03-24T00:18:01.489 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:01.485+0000 7f1021bfc740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55c3b0b7f810 0x55c3b0b90da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:18:01.489 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:01.485+0000 7f1021bfc740 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55c3b0b809e0 0x55c3b0b8f420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:18:01.489 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:01.485+0000 7f1021bfc740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55c3b0b813a0 0x55c3b0b8f960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:18:01.489 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:01.485+0000 7f1021bfc740 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f10181b0340 con 0x55c3b0b809e0 2024-03-24T00:18:01.489 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:01.485+0000 7f1021bfc740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f10181afce0 con 0x55c3b0b7f810 2024-03-24T00:18:01.489 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:01.485+0000 7f1021bfc740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f10181ad1e0 con 0x55c3b0b813a0 2024-03-24T00:18:01.489 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:01.485+0000 7f1021bfc740 10 client.0 _ll_register_callbacks cb 0x7f10080035f0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T00:18:01.489 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:01.485+0000 7f1017e00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55c3b0b7f810 0x55c3b0b90da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:18:01.490 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:01.485+0000 7f1017e00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55c3b0b7f810 0x55c3b0b90da0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.118:38120/0 (socket says 172.21.15.118:38120) 2024-03-24T00:18:01.490 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:01.485+0000 7f1017e00640 1 -- 172.21.15.118:0/2100739381 learned_addr learned my addr 172.21.15.118:0/2100739381 (peer_addr_for_me v2:172.21.15.118:0/0) 2024-03-24T00:18:01.490 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:01.485+0000 7f101ce00640 1 --2- 172.21.15.118:0/2100739381 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55c3b0b813a0 0x55c3b0b8f960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:18:01.490 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:01.485+0000 7f1017400640 1 --2- 172.21.15.118:0/2100739381 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55c3b0b809e0 0x55c3b0b8f420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:18:01.490 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:01.485+0000 7f1017e00640 1 -- 172.21.15.118:0/2100739381 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55c3b0b813a0 msgr2=0x55c3b0b8f960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:18:01.490 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:01.485+0000 7f1017e00640 1 --2- 172.21.15.118:0/2100739381 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55c3b0b813a0 0x55c3b0b8f960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:18:01.490 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:01.485+0000 7f1017e00640 1 -- 172.21.15.118:0/2100739381 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55c3b0b809e0 msgr2=0x55c3b0b8f420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:18:01.490 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:01.485+0000 7f1017e00640 1 --2- 172.21.15.118:0/2100739381 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55c3b0b809e0 0x55c3b0b8f420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:18:01.490 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:01.485+0000 7f1017e00640 1 -- 172.21.15.118:0/2100739381 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55c3b0b900f0 con 0x55c3b0b7f810 2024-03-24T00:18:01.490 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:01.485+0000 7f1017400640 1 --2- 172.21.15.118:0/2100739381 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55c3b0b809e0 0x55c3b0b8f420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T00:18:01.490 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:01.486+0000 7f1017e00640 1 --2- 172.21.15.118:0/2100739381 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55c3b0b7f810 0x55c3b0b90da0 secure :-1 s=READY pgs=319 cs=0 l=1 rev1=1 crypto rx=0x7f1008004ec0 tx=0x7f10080bd2f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f4e666c6b13557de server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:18:01.490 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:01.486+0000 7f1002000640 10 client.0 ms_handle_connect on v2:172.21.15.189:3300/0 2024-03-24T00:18:01.491 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:01.486+0000 7f1002000640 1 -- 172.21.15.118:0/2100739381 <== mon.2 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f10080c1310 con 0x55c3b0b7f810 2024-03-24T00:18:01.491 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:01.486+0000 7f1002000640 1 -- 172.21.15.118:0/2100739381 <== mon.2 v2:172.21.15.189:3300/0 2 ==== osd_map(90..90 src has 1..90) v4 ==== 8277+0+0 (secure 0 0 0) 0x7f10080dd0b0 con 0x55c3b0b7f810 2024-03-24T00:18:01.491 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:01.486+0000 7f1021bfc740 1 -- 172.21.15.118:0/2100739381 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f10080bd3f0 con 0x55c3b0b7f810 2024-03-24T00:18:01.491 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:01.486+0000 7f1002000640 1 -- 172.21.15.118:0/2100739381 <== mon.2 v2:172.21.15.189:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) v2 ==== 24+0+0 (secure 0 0 0) 0x7f10080c1540 con 0x55c3b0b7f810 2024-03-24T00:18:01.491 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:01.486+0000 7f1021bfc740 10 client.26066 fetch_fsmap learned FSMap version 28 2024-03-24T00:18:01.491 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:01.486+0000 7f1021bfc740 1 -- 172.21.15.118:0/2100739381 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) v3 -- 0x55c3b0bf1e00 con 0x55c3b0b7f810 2024-03-24T00:18:01.491 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:01.486+0000 7f1002000640 1 -- 172.21.15.118:0/2100739381 <== mon.2 v2:172.21.15.189:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f10080bd3f0 con 0x55c3b0b7f810 2024-03-24T00:18:01.491 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:01.486+0000 7f1002000640 1 -- 172.21.15.118:0/2100739381 <== mon.2 v2:172.21.15.189:3300/0 5 ==== fsmap(e 28) v1 ==== 7934+0+0 (secure 0 0 0) 0x7f10080bc2b0 con 0x55c3b0b7f810 2024-03-24T00:18:01.491 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:01.487+0000 7f1021bfc740 10 client.26066 fetch_fsmap finished waiting for FSMap version 28 2024-03-24T00:18:01.491 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:01.487+0000 7f1021bfc740 10 client.26066 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.f 2024-03-24T00:18:01.491 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:01.487+0000 7f1021bfc740 20 client.26066 populate_metadata read hostname 'smithi118' 2024-03-24T00:18:01.491 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:01.487+0000 7f1021bfc740 1 --2- 172.21.15.118:0/2100739381 >> [v2:172.21.15.193:6834/2055310541,v1:172.21.15.193:6835/2055310541] conn(0x55c3b0bf2540 0x55c3b0bf4960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:18:01.491 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:01.487+0000 7f1021bfc740 4 client.26066 mds_command: new command op to 24499 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-24T00:18:01.492 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:01.487+0000 7f1021bfc740 1 -- 172.21.15.118:0/2100739381 --> [v2:172.21.15.193:6834/2055310541,v1:172.21.15.193:6835/2055310541] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x55c3b0aebb50 con 0x55c3b0bf2540 2024-03-24T00:18:01.492 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:01.487+0000 7f1017400640 1 --2- 172.21.15.118:0/2100739381 >> [v2:172.21.15.193:6834/2055310541,v1:172.21.15.193:6835/2055310541] conn(0x55c3b0bf2540 0x55c3b0bf4960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:18:01.492 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:01.487+0000 7f1017400640 1 --2- 172.21.15.118:0/2100739381 >> [v2:172.21.15.193:6834/2055310541,v1:172.21.15.193:6835/2055310541] conn(0x55c3b0bf2540 0x55c3b0bf4960 crc :-1 s=READY pgs=32 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=1c8e39550874b8d server_cookie=3f929fd2e82bf080 in_seq=0 out_seq=0 2024-03-24T00:18:01.492 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:01.487+0000 7f1002000640 10 client.26066 ms_handle_connect on v2:172.21.15.193:6834/2055310541 2024-03-24T00:18:01.492 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:01.488+0000 7f1002000640 1 -- 172.21.15.118:0/2100739381 <== mds.3 v2:172.21.15.193:6834/2055310541 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x55c3b0aebb50 con 0x55c3b0bf2540 2024-03-24T00:18:01.492 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:01.488+0000 7f1002000640 10 client.26066 handle_command_reply: tid=0 2024-03-24T00:18:01.492 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:01.488+0000 7f1021bfc740 1 client.26066 shutdown 2024-03-24T00:18:01.492 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:01.488+0000 7f1021bfc740 1 -- 172.21.15.118:0/2100739381 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55c3b0b7f810 msgr2=0x55c3b0b90da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:18:01.492 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:01.488+0000 7f1021bfc740 1 --2- 172.21.15.118:0/2100739381 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55c3b0b7f810 0x55c3b0b90da0 secure :-1 s=READY pgs=319 cs=0 l=1 rev1=1 crypto rx=0x7f1008004ec0 tx=0x7f10080bd2f0 comp rx=0 tx=0).stop 2024-03-24T00:18:01.492 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:01.488+0000 7f1021bfc740 1 -- 172.21.15.118:0/2100739381 shutdown_connections 2024-03-24T00:18:01.492 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:01.488+0000 7f1021bfc740 1 --2- 172.21.15.118:0/2100739381 >> [v2:172.21.15.193:6834/2055310541,v1:172.21.15.193:6835/2055310541] conn(0x55c3b0bf2540 0x55c3b0bf4960 crc :-1 s=READY pgs=32 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:18:01.492 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:01.488+0000 7f1021bfc740 1 --2- 172.21.15.118:0/2100739381 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55c3b0b813a0 0x55c3b0b8f960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:18:01.492 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:01.488+0000 7f1021bfc740 1 --2- 172.21.15.118:0/2100739381 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55c3b0b809e0 0x55c3b0b8f420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:18:01.492 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:01.488+0000 7f1002000640 0 client.26066 ms_handle_reset on v2:172.21.15.193:6834/2055310541 2024-03-24T00:18:01.493 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:01.488+0000 7f1021bfc740 1 --2- 172.21.15.118:0/2100739381 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55c3b0b7f810 0x55c3b0b90da0 unknown :-1 s=CLOSED pgs=319 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:18:01.493 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:01.488+0000 7f1021bfc740 1 -- 172.21.15.118:0/2100739381 >> 172.21.15.118:0/2100739381 conn(0x55c3b0b7ac00 msgr2=0x55c3b0b83c10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:18:01.493 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:01.488+0000 7f1021bfc740 1 -- 172.21.15.118:0/2100739381 shutdown_connections 2024-03-24T00:18:01.493 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:01.488+0000 7f1021bfc740 1 -- 172.21.15.118:0/2100739381 wait complete. 2024-03-24T00:18:01.493 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:01.488+0000 7f1021bfc740 20 client.26066 trim_cache size 0 max 16384 2024-03-24T00:18:01.493 INFO:teuthology.orchestra.run.smithi118.stdout:[] 2024-03-24T00:18:01.494 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:01.489+0000 7f101ee00640 1 -- 172.21.15.118:0/1530969945 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7f0ffc0a2880 msgr2=0x7f0ffc0a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:18:01.494 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:01.489+0000 7f101ee00640 1 --2- 172.21.15.118:0/1530969945 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7f0ffc0a2880 0x7f0ffc0a4d40 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 crypto rx=0x55c3b0b808e0 tx=0x7f1010004a40 comp rx=0 tx=0).stop 2024-03-24T00:18:01.494 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:01.489+0000 7f101ee00640 1 -- 172.21.15.118:0/1530969945 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f10180fff10 msgr2=0x7f101819e050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:18:01.494 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:01.489+0000 7f101ee00640 1 --2- 172.21.15.118:0/1530969945 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f10180fff10 0x7f101819e050 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 crypto rx=0x7f1008000c00 tx=0x7f100802c650 comp rx=0 tx=0).stop 2024-03-24T00:18:01.494 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:01.489+0000 7f101ee00640 1 -- 172.21.15.118:0/1530969945 shutdown_connections 2024-03-24T00:18:01.494 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:01.489+0000 7f101ee00640 1 --2- 172.21.15.118:0/1530969945 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7f0ffc0a2880 0x7f0ffc0a4d40 unknown :-1 s=CLOSED pgs=311 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:18:01.495 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:01.489+0000 7f101ee00640 1 --2- 172.21.15.118:0/1530969945 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f1018105230 0x7f10181a55b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:18:01.495 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:01.489+0000 7f101ee00640 1 --2- 172.21.15.118:0/1530969945 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f1018102880 0x7f101819e590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:18:01.495 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:01.489+0000 7f101ee00640 1 --2- 172.21.15.118:0/1530969945 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f10180fff10 0x7f101819e050 unknown :-1 s=CLOSED pgs=316 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:18:01.495 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:01.489+0000 7f101ee00640 1 -- 172.21.15.118:0/1530969945 >> 172.21.15.118:0/1530969945 conn(0x7f10180fbb10 msgr2=0x7f10181032b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:18:01.495 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:01.489+0000 7f101ee00640 1 -- 172.21.15.118:0/1530969945 shutdown_connections 2024-03-24T00:18:01.495 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:01.489+0000 7f101ee00640 1 -- 172.21.15.118:0/1530969945 wait complete. 2024-03-24T00:18:01.508 INFO:teuthology.orchestra.run:Running command with timeout 120 2024-03-24T00:18:01.508 DEBUG:teuthology.orchestra.run.smithi118:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:2 damage ls 2024-03-24T00:18:01.729 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:01.729+0000 7fe771400640 1 -- 172.21.15.118:0/10303639 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fe76c0772a0 msgr2=0x7fe76c077700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:18:01.730 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:01.730+0000 7fe771400640 1 --2- 172.21.15.118:0/10303639 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fe76c0772a0 0x7fe76c077700 secure :-1 s=READY pgs=307 cs=0 l=1 rev1=1 crypto rx=0x7fe758009900 tx=0x7fe75802bc40 comp rx=0 tx=0).stop 2024-03-24T00:18:01.730 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:01.730+0000 7fe771400640 1 -- 172.21.15.118:0/10303639 shutdown_connections 2024-03-24T00:18:01.730 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:01.730+0000 7fe771400640 1 --2- 172.21.15.118:0/10303639 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fe76c077c40 0x7fe76c0784e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:18:01.730 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:01.730+0000 7fe771400640 1 --2- 172.21.15.118:0/10303639 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fe76c0772a0 0x7fe76c077700 unknown :-1 s=CLOSED pgs=307 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:18:01.730 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:01.730+0000 7fe771400640 1 --2- 172.21.15.118:0/10303639 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe76c076060 0x7fe76c076480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:18:01.730 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:01.730+0000 7fe771400640 1 -- 172.21.15.118:0/10303639 >> 172.21.15.118:0/10303639 conn(0x7fe76c0fee50 msgr2=0x7fe76c101290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:18:01.730 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:01.730+0000 7fe771400640 1 -- 172.21.15.118:0/10303639 shutdown_connections 2024-03-24T00:18:01.730 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:01.730+0000 7fe771400640 1 -- 172.21.15.118:0/10303639 wait complete. 2024-03-24T00:18:01.732 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:01.731+0000 7fe771400640 1 Processor -- start 2024-03-24T00:18:01.733 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:01.731+0000 7fe771400640 1 -- start start 2024-03-24T00:18:01.733 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:01.731+0000 7fe771400640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe76c076060 0x7fe76c19e000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:18:01.733 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:01.731+0000 7fe771400640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fe76c0772a0 0x7fe76c19e540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:18:01.733 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:01.731+0000 7fe771400640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fe76c077c40 0x7fe76c1a55b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:18:01.733 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:01.731+0000 7fe771400640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe76c1139b0 con 0x7fe76c077c40 2024-03-24T00:18:01.733 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:01.731+0000 7fe771400640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe76c113830 con 0x7fe76c076060 2024-03-24T00:18:01.733 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:01.731+0000 7fe771400640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe76c113b30 con 0x7fe76c0772a0 2024-03-24T00:18:01.733 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:01.731+0000 7fe76a800640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe76c076060 0x7fe76c19e000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:18:01.733 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:01.731+0000 7fe76a800640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe76c076060 0x7fe76c19e000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.118:38142/0 (socket says 172.21.15.118:38142) 2024-03-24T00:18:01.733 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:01.731+0000 7fe76a800640 1 -- 172.21.15.118:0/1954903951 learned_addr learned my addr 172.21.15.118:0/1954903951 (peer_addr_for_me v2:172.21.15.118:0/0) 2024-03-24T00:18:01.733 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:01.731+0000 7fe769e00640 1 --2- 172.21.15.118:0/1954903951 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fe76c0772a0 0x7fe76c19e540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:18:01.733 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:01.731+0000 7fe76b200640 1 --2- 172.21.15.118:0/1954903951 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fe76c077c40 0x7fe76c1a55b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:18:01.734 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:01.732+0000 7fe76a800640 1 -- 172.21.15.118:0/1954903951 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fe76c0772a0 msgr2=0x7fe76c19e540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:18:01.734 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:01.732+0000 7fe76a800640 1 --2- 172.21.15.118:0/1954903951 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fe76c0772a0 0x7fe76c19e540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:18:01.734 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:01.732+0000 7fe76a800640 1 -- 172.21.15.118:0/1954903951 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fe76c077c40 msgr2=0x7fe76c1a55b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:18:01.734 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:01.732+0000 7fe76a800640 1 --2- 172.21.15.118:0/1954903951 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fe76c077c40 0x7fe76c1a55b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:18:01.734 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:01.732+0000 7fe76a800640 1 -- 172.21.15.118:0/1954903951 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe76c1a5c20 con 0x7fe76c076060 2024-03-24T00:18:01.734 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:01.732+0000 7fe76a800640 1 --2- 172.21.15.118:0/1954903951 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe76c076060 0x7fe76c19e000 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 crypto rx=0x7fe75c00e9f0 tx=0x7fe75c00edf0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:18:01.734 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:01.732+0000 7fe763400640 1 -- 172.21.15.118:0/1954903951 <== mon.2 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe75c00cd80 con 0x7fe76c076060 2024-03-24T00:18:01.734 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:01.732+0000 7fe771400640 1 -- 172.21.15.118:0/1954903951 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe76c1a5f10 con 0x7fe76c076060 2024-03-24T00:18:01.734 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:01.732+0000 7fe763400640 1 -- 172.21.15.118:0/1954903951 <== mon.2 v2:172.21.15.189:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fe75c005500 con 0x7fe76c076060 2024-03-24T00:18:01.734 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:01.732+0000 7fe763400640 1 -- 172.21.15.118:0/1954903951 <== mon.2 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe75c010690 con 0x7fe76c076060 2024-03-24T00:18:01.734 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:01.732+0000 7fe771400640 1 -- 172.21.15.118:0/1954903951 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe76c1a6320 con 0x7fe76c076060 2024-03-24T00:18:01.739 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:01.736+0000 7fe774043740 1 Processor -- start 2024-03-24T00:18:01.740 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:01.736+0000 7fe774043740 1 -- start start 2024-03-24T00:18:01.740 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:01.736+0000 7fe763400640 1 -- 172.21.15.118:0/1954903951 <== mon.2 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 20) v1 ==== 137291+0+0 (secure 0 0 0) 0x7fe75c0046c0 con 0x7fe76c076060 2024-03-24T00:18:01.740 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:01.736+0000 7fe774043740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55e554e54810 0x55e554e54c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:18:01.740 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:01.736+0000 7fe774043740 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55e554e559e0 0x55e554e55e60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:18:01.740 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:01.736+0000 7fe774043740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55e554e563a0 0x55e554e5cc30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:18:01.740 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:01.736+0000 7fe774043740 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e554db92c0 con 0x55e554e559e0 2024-03-24T00:18:01.740 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:01.736+0000 7fe774043740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e554d61c90 con 0x55e554e563a0 2024-03-24T00:18:01.740 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:01.736+0000 7fe774043740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e554db5360 con 0x55e554e54810 2024-03-24T00:18:01.740 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:01.737+0000 7fe76b200640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55e554e54810 0x55e554e54c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:18:01.740 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:01.737+0000 7fe76a800640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55e554e559e0 0x55e554e55e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:18:01.740 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:01.737+0000 7fe769e00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55e554e563a0 0x55e554e5cc30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:18:01.740 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:01.737+0000 7fe76b200640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55e554e54810 0x55e554e54c30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.118:53080/0 (socket says 172.21.15.118:53080) 2024-03-24T00:18:01.740 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:01.737+0000 7fe76b200640 1 -- 172.21.15.118:0/2931721531 learned_addr learned my addr 172.21.15.118:0/2931721531 (peer_addr_for_me v2:172.21.15.118:0/0) 2024-03-24T00:18:01.741 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:01.737+0000 7fe76a800640 1 -- 172.21.15.118:0/2931721531 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55e554e54810 msgr2=0x55e554e54c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:18:01.741 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:01.737+0000 7fe76a800640 1 --2- 172.21.15.118:0/2931721531 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55e554e54810 0x55e554e54c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:18:01.741 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:01.737+0000 7fe76a800640 1 -- 172.21.15.118:0/2931721531 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55e554e563a0 msgr2=0x55e554e5cc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:18:01.741 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:01.737+0000 7fe76a800640 1 --2- 172.21.15.118:0/2931721531 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55e554e563a0 0x55e554e5cc30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:18:01.741 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:01.737+0000 7fe76a800640 1 -- 172.21.15.118:0/2931721531 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55e554d86080 con 0x55e554e559e0 2024-03-24T00:18:01.741 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:01.737+0000 7fe76b200640 1 --2- 172.21.15.118:0/2931721531 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55e554e54810 0x55e554e54c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T00:18:01.741 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:01.737+0000 7fe769e00640 1 --2- 172.21.15.118:0/2931721531 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55e554e563a0 0x55e554e5cc30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T00:18:01.741 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:01.738+0000 7fe76a800640 1 --2- 172.21.15.118:0/2931721531 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55e554e559e0 0x55e554e55e60 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7fe75c0cc990 tx=0x7fe75c0d2f20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=8659cb07c8dbdc05 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:18:01.741 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:01.738+0000 7fe763400640 1 --2- 172.21.15.118:0/1954903951 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7fe7500a2800 0x7fe7500a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:18:01.741 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:01.738+0000 7fe757400640 1 -- 172.21.15.118:0/2931721531 <== mon.0 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe75c0e2070 con 0x55e554e559e0 2024-03-24T00:18:01.741 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:01.738+0000 7fe763400640 1 -- 172.21.15.118:0/1954903951 <== mon.2 v2:172.21.15.189:3300/0 5 ==== osd_map(90..90 src has 1..90) v4 ==== 8277+0+0 (secure 0 0 0) 0x7fe75c0cef80 con 0x7fe76c076060 2024-03-24T00:18:01.741 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:01.738+0000 7fe76b200640 1 --2- 172.21.15.118:0/1954903951 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7fe7500a2800 0x7fe7500a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:18:01.741 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:01.738+0000 7fe757400640 1 -- 172.21.15.118:0/2931721531 <== mon.0 v2:172.21.15.118:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fe75c09bf90 con 0x55e554e559e0 2024-03-24T00:18:01.741 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:01.739+0000 7fe757400640 1 -- 172.21.15.118:0/2931721531 <== mon.0 v2:172.21.15.118:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe75c068310 con 0x55e554e559e0 2024-03-24T00:18:01.742 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:01.739+0000 7fe774043740 1 -- 172.21.15.118:0/2931721531 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55e554e559e0 msgr2=0x55e554e55e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:18:01.742 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:01.739+0000 7fe76b200640 1 --2- 172.21.15.118:0/1954903951 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7fe7500a2800 0x7fe7500a4cc0 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 crypto rx=0x55e554e558e0 tx=0x7fe7640098a0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:18:01.742 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:01.739+0000 7fe774043740 1 --2- 172.21.15.118:0/2931721531 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55e554e559e0 0x55e554e55e60 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7fe75c0cc990 tx=0x7fe75c0d2f20 comp rx=0 tx=0).stop 2024-03-24T00:18:01.742 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:01.739+0000 7fe774043740 1 -- 172.21.15.118:0/2931721531 shutdown_connections 2024-03-24T00:18:01.742 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:01.740+0000 7fe774043740 1 --2- 172.21.15.118:0/2931721531 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55e554e563a0 0x55e554e5cc30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:18:01.742 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:01.740+0000 7fe774043740 1 --2- 172.21.15.118:0/2931721531 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55e554e559e0 0x55e554e55e60 unknown :-1 s=CLOSED pgs=197 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:18:01.742 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:01.740+0000 7fe774043740 1 --2- 172.21.15.118:0/2931721531 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55e554e54810 0x55e554e54c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:18:01.742 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:01.740+0000 7fe774043740 1 -- 172.21.15.118:0/2931721531 >> 172.21.15.118:0/2931721531 conn(0x55e554e4fc00 msgr2=0x55e554e50000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:18:01.742 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:01.740+0000 7fe774043740 1 -- 172.21.15.118:0/2931721531 shutdown_connections 2024-03-24T00:18:01.742 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:01.740+0000 7fe774043740 1 -- 172.21.15.118:0/2931721531 wait complete. 2024-03-24T00:18:01.742 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:01.740+0000 7fe774043740 1 -- start start 2024-03-24T00:18:01.746 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:01.741+0000 7fe774043740 1 Processor -- start 2024-03-24T00:18:01.746 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:01.741+0000 7fe774043740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55e554e54810 0x55e554e5ea30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:18:01.746 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:01.741+0000 7fe774043740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55e554e559e0 0x55e554e5ef70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:18:01.746 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:01.741+0000 7fe774043740 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55e554e563a0 0x55e554e65fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:18:01.746 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:01.741+0000 7fe774043740 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e554e5d3b0 con 0x55e554e563a0 2024-03-24T00:18:01.746 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:01.741+0000 7fe774043740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e554d750d0 con 0x55e554e54810 2024-03-24T00:18:01.746 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:01.741+0000 7fe774043740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e554e5d530 con 0x55e554e559e0 2024-03-24T00:18:01.746 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:01.741+0000 7fe774043740 10 client.0 _ll_register_callbacks cb 0x55e554d79e80 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T00:18:01.746 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:01.741+0000 7fe76a800640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55e554e54810 0x55e554e5ea30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:18:01.746 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:01.741+0000 7fe769e00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55e554e559e0 0x55e554e5ef70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:18:01.747 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:01.741+0000 7fe76a800640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55e554e54810 0x55e554e5ea30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.118:38166/0 (socket says 172.21.15.118:38166) 2024-03-24T00:18:01.747 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:01.741+0000 7fe76a800640 1 -- 172.21.15.118:0/1086971302 learned_addr learned my addr 172.21.15.118:0/1086971302 (peer_addr_for_me v2:172.21.15.118:0/0) 2024-03-24T00:18:01.747 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:01.741+0000 7fe76b200640 1 --2- 172.21.15.118:0/1086971302 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55e554e563a0 0x55e554e65fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:18:01.747 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:01.742+0000 7fe769e00640 1 -- 172.21.15.118:0/1086971302 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55e554e54810 msgr2=0x55e554e5ea30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:18:01.747 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:01.742+0000 7fe769e00640 1 --2- 172.21.15.118:0/1086971302 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55e554e54810 0x55e554e5ea30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:18:01.747 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:01.742+0000 7fe769e00640 1 -- 172.21.15.118:0/1086971302 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55e554e563a0 msgr2=0x55e554e65fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:18:01.747 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:01.742+0000 7fe769e00640 1 --2- 172.21.15.118:0/1086971302 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55e554e563a0 0x55e554e65fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:18:01.747 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:01.742+0000 7fe769e00640 1 -- 172.21.15.118:0/1086971302 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55e554acd3f0 con 0x55e554e559e0 2024-03-24T00:18:01.747 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:01.742+0000 7fe76b200640 1 --2- 172.21.15.118:0/1086971302 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55e554e563a0 0x55e554e65fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T00:18:01.747 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:01.742+0000 7fe76a800640 1 --2- 172.21.15.118:0/1086971302 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55e554e54810 0x55e554e5ea30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T00:18:01.747 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:01.742+0000 7fe769e00640 1 --2- 172.21.15.118:0/1086971302 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55e554e559e0 0x55e554e5ef70 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 crypto rx=0x7fe758036040 tx=0x7fe758004270 comp rx=0 tx=0).ready entity=mon.1 client_cookie=b2af1d5e42a89d4b server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:18:01.747 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:01.742+0000 7fe74be00640 10 client.0 ms_handle_connect on v2:172.21.15.193:3300/0 2024-03-24T00:18:01.747 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:01.742+0000 7fe74be00640 1 -- 172.21.15.118:0/1086971302 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe758005d50 con 0x55e554e559e0 2024-03-24T00:18:01.747 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:01.742+0000 7fe74be00640 1 -- 172.21.15.118:0/1086971302 <== mon.1 v2:172.21.15.193:3300/0 2 ==== osd_map(90..90 src has 1..90) v4 ==== 8277+0+0 (secure 0 0 0) 0x7fe7580046c0 con 0x55e554e559e0 2024-03-24T00:18:01.748 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:01.742+0000 7fe774043740 1 -- 172.21.15.118:0/1086971302 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fe75c09bf90 con 0x55e554e559e0 2024-03-24T00:18:01.748 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:01.743+0000 7fe74be00640 1 -- 172.21.15.118:0/1086971302 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) v2 ==== 24+0+0 (secure 0 0 0) 0x7fe758004900 con 0x55e554e559e0 2024-03-24T00:18:01.748 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:01.743+0000 7fe774043740 10 client.26071 fetch_fsmap learned FSMap version 28 2024-03-24T00:18:01.748 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:01.743+0000 7fe774043740 1 -- 172.21.15.118:0/1086971302 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) v3 -- 0x55e554b977d0 con 0x55e554e559e0 2024-03-24T00:18:01.748 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:01.743+0000 7fe74be00640 1 -- 172.21.15.118:0/1086971302 <== mon.1 v2:172.21.15.193:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fe75c09bf90 con 0x55e554e559e0 2024-03-24T00:18:01.748 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:01.743+0000 7fe74be00640 1 -- 172.21.15.118:0/1086971302 <== mon.1 v2:172.21.15.193:3300/0 5 ==== fsmap(e 28) v1 ==== 7934+0+0 (secure 0 0 0) 0x7fe7580035f0 con 0x55e554e559e0 2024-03-24T00:18:01.748 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:01.743+0000 7fe774043740 10 client.26071 fetch_fsmap finished waiting for FSMap version 28 2024-03-24T00:18:01.748 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:01.743+0000 7fe774043740 10 client.26071 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.c 2024-03-24T00:18:01.748 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:01.743+0000 7fe774043740 20 client.26071 populate_metadata read hostname 'smithi118' 2024-03-24T00:18:01.748 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:01.743+0000 7fe774043740 1 --2- 172.21.15.118:0/1086971302 >> [v2:172.21.15.193:6836/2921520053,v1:172.21.15.193:6837/2921520053] conn(0x55e554e6a4f0 0x55e554e6c900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:18:01.748 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:01.743+0000 7fe774043740 4 client.26071 mds_command: new command op to 24511 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-24T00:18:01.748 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:01.743+0000 7fe774043740 1 -- 172.21.15.118:0/1086971302 --> [v2:172.21.15.193:6836/2921520053,v1:172.21.15.193:6837/2921520053] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55e554dc6800 con 0x55e554e6a4f0 2024-03-24T00:18:01.748 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:01.744+0000 7fe76a800640 1 --2- 172.21.15.118:0/1086971302 >> [v2:172.21.15.193:6836/2921520053,v1:172.21.15.193:6837/2921520053] conn(0x55e554e6a4f0 0x55e554e6c900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:18:01.748 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:01.744+0000 7fe76a800640 1 --2- 172.21.15.118:0/1086971302 >> [v2:172.21.15.193:6836/2921520053,v1:172.21.15.193:6837/2921520053] conn(0x55e554e6a4f0 0x55e554e6c900 crc :-1 s=READY pgs=25 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=e34693714fdd2ced server_cookie=32429e849e6056a7 in_seq=0 out_seq=0 2024-03-24T00:18:01.749 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:01.744+0000 7fe74be00640 10 client.26071 ms_handle_connect on v2:172.21.15.193:6836/2921520053 2024-03-24T00:18:01.749 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:01.745+0000 7fe74be00640 1 -- 172.21.15.118:0/1086971302 <== mds.2 v2:172.21.15.193:6836/2921520053 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x55e554dc6800 con 0x55e554e6a4f0 2024-03-24T00:18:01.749 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:01.745+0000 7fe74be00640 10 client.26071 handle_command_reply: tid=0 2024-03-24T00:18:01.749 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:01.745+0000 7fe774043740 1 client.26071 shutdown 2024-03-24T00:18:01.749 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:01.745+0000 7fe774043740 1 -- 172.21.15.118:0/1086971302 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55e554e559e0 msgr2=0x55e554e5ef70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:18:01.749 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:01.745+0000 7fe774043740 1 --2- 172.21.15.118:0/1086971302 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55e554e559e0 0x55e554e5ef70 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 crypto rx=0x7fe758036040 tx=0x7fe758004270 comp rx=0 tx=0).stop 2024-03-24T00:18:01.749 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:01.745+0000 7fe774043740 1 -- 172.21.15.118:0/1086971302 shutdown_connections 2024-03-24T00:18:01.749 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:01.745+0000 7fe774043740 1 --2- 172.21.15.118:0/1086971302 >> [v2:172.21.15.193:6836/2921520053,v1:172.21.15.193:6837/2921520053] conn(0x55e554e6a4f0 0x55e554e6c900 crc :-1 s=READY pgs=25 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:18:01.749 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:01.746+0000 7fe774043740 1 --2- 172.21.15.118:0/1086971302 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55e554e563a0 0x55e554e65fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:18:01.749 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:01.746+0000 7fe774043740 1 --2- 172.21.15.118:0/1086971302 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55e554e559e0 0x55e554e5ef70 unknown :-1 s=CLOSED pgs=308 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:18:01.749 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:01.746+0000 7fe774043740 1 --2- 172.21.15.118:0/1086971302 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55e554e54810 0x55e554e5ea30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:18:01.749 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:01.746+0000 7fe774043740 1 -- 172.21.15.118:0/1086971302 >> 172.21.15.118:0/1086971302 conn(0x55e554e4fc00 msgr2=0x55e554e4eeb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:18:01.749 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:01.746+0000 7fe74be00640 0 client.26071 ms_handle_reset on v2:172.21.15.193:6836/2921520053 2024-03-24T00:18:01.749 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:01.746+0000 7fe774043740 1 -- 172.21.15.118:0/1086971302 shutdown_connections 2024-03-24T00:18:01.750 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:01.746+0000 7fe774043740 1 -- 172.21.15.118:0/1086971302 wait complete. 2024-03-24T00:18:01.750 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:01.746+0000 7fe774043740 20 client.26071 trim_cache size 0 max 16384 2024-03-24T00:18:01.760 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:01.760+0000 7fe774043740 1 Processor -- start 2024-03-24T00:18:01.760 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:01.761+0000 7fe774043740 1 -- start start 2024-03-24T00:18:01.761 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:01.761+0000 7fe774043740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55e554e54810 0x55e554e589e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:18:01.761 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:01.761+0000 7fe774043740 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55e554e559e0 0x55e554e65f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:18:01.761 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:01.761+0000 7fe774043740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55e554e563a0 0x55e554e58f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:18:01.761 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:01.761+0000 7fe774043740 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe76c1aef80 con 0x55e554e559e0 2024-03-24T00:18:01.761 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:01.761+0000 7fe774043740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe76c1b0900 con 0x55e554e563a0 2024-03-24T00:18:01.761 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:01.761+0000 7fe774043740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe76c1af5e0 con 0x55e554e54810 2024-03-24T00:18:01.762 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:01.761+0000 7fe76a800640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55e554e54810 0x55e554e589e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:18:01.762 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:01.761+0000 7fe76a800640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55e554e54810 0x55e554e589e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.118:53098/0 (socket says 172.21.15.118:53098) 2024-03-24T00:18:01.762 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:01.761+0000 7fe76a800640 1 -- 172.21.15.118:0/1138992971 learned_addr learned my addr 172.21.15.118:0/1138992971 (peer_addr_for_me v2:172.21.15.118:0/0) 2024-03-24T00:18:01.762 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:01.761+0000 7fe76b200640 1 --2- 172.21.15.118:0/1138992971 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55e554e563a0 0x55e554e58f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:18:01.762 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:01.761+0000 7fe769e00640 1 --2- 172.21.15.118:0/1138992971 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55e554e559e0 0x55e554e65f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:18:01.762 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:01.762+0000 7fe76a800640 1 -- 172.21.15.118:0/1138992971 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55e554e563a0 msgr2=0x55e554e58f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:18:01.762 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:01.762+0000 7fe76a800640 1 --2- 172.21.15.118:0/1138992971 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55e554e563a0 0x55e554e58f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:18:01.762 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:01.762+0000 7fe76a800640 1 -- 172.21.15.118:0/1138992971 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55e554e559e0 msgr2=0x55e554e65f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:18:01.763 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:01.762+0000 7fe76a800640 1 --2- 172.21.15.118:0/1138992971 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55e554e559e0 0x55e554e65f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:18:01.763 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:01.762+0000 7fe76a800640 1 -- 172.21.15.118:0/1138992971 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55e554e4b510 con 0x55e554e54810 2024-03-24T00:18:01.763 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:01.762+0000 7fe76a800640 1 --2- 172.21.15.118:0/1138992971 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55e554e54810 0x55e554e589e0 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 crypto rx=0x7fe75c0049d0 tx=0x7fe75c0d2f20 comp rx=0 tx=0).ready entity=mon.1 client_cookie=8d801324f0d3d462 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:18:01.763 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:01.762+0000 7fe756a00640 1 -- 172.21.15.118:0/1138992971 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe75c067280 con 0x55e554e54810 2024-03-24T00:18:01.763 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:01.762+0000 7fe756a00640 1 -- 172.21.15.118:0/1138992971 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fe75c09bf90 con 0x55e554e54810 2024-03-24T00:18:01.763 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:01.762+0000 7fe756a00640 1 -- 172.21.15.118:0/1138992971 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe75c0675a0 con 0x55e554e54810 2024-03-24T00:18:01.766 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:01.763+0000 7fe774043740 1 -- 172.21.15.118:0/1138992971 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55e554e54810 msgr2=0x55e554e589e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:18:01.766 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:01.763+0000 7fe774043740 1 --2- 172.21.15.118:0/1138992971 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55e554e54810 0x55e554e589e0 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 crypto rx=0x7fe75c0049d0 tx=0x7fe75c0d2f20 comp rx=0 tx=0).stop 2024-03-24T00:18:01.788 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:01.787+0000 7fe774043740 1 -- 172.21.15.118:0/1138992971 shutdown_connections 2024-03-24T00:18:01.788 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:01.787+0000 7fe774043740 1 --2- 172.21.15.118:0/1138992971 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55e554e563a0 0x55e554e58f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:18:01.788 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:01.787+0000 7fe774043740 1 --2- 172.21.15.118:0/1138992971 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55e554e559e0 0x55e554e65f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:18:01.788 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:01.787+0000 7fe774043740 1 --2- 172.21.15.118:0/1138992971 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55e554e54810 0x55e554e589e0 unknown :-1 s=CLOSED pgs=309 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:18:01.788 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:01.787+0000 7fe774043740 1 -- 172.21.15.118:0/1138992971 >> 172.21.15.118:0/1138992971 conn(0x55e554e4fc00 msgr2=0x55e554e536f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:18:01.798 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:01.798+0000 7fe774043740 1 -- 172.21.15.118:0/1138992971 shutdown_connections 2024-03-24T00:18:01.799 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:01.798+0000 7fe774043740 1 -- 172.21.15.118:0/1138992971 wait complete. 2024-03-24T00:18:01.799 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:01.799+0000 7fe774043740 1 -- start start 2024-03-24T00:18:01.799 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:01.799+0000 7fe774043740 1 Processor -- start 2024-03-24T00:18:01.799 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:01.800+0000 7fe774043740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55e554e54810 0x55e554e61f60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:18:01.805 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:01.800+0000 7fe774043740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55e554e559e0 0x55e554e624a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:18:01.805 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:01.800+0000 7fe76a800640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55e554e54810 0x55e554e61f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:18:01.806 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:01.800+0000 7fe76a800640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55e554e54810 0x55e554e61f60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.118:38178/0 (socket says 172.21.15.118:38178) 2024-03-24T00:18:01.806 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:01.800+0000 7fe769e00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55e554e559e0 0x55e554e624a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:18:01.806 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:01.801+0000 7fe769e00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55e554e559e0 0x55e554e624a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.118:53104/0 (socket says 172.21.15.118:53104) 2024-03-24T00:18:01.806 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:01.801+0000 7fe76a800640 1 -- 172.21.15.118:0/1210143911 learned_addr learned my addr 172.21.15.118:0/1210143911 (peer_addr_for_me v2:172.21.15.118:0/0) 2024-03-24T00:18:01.806 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:01.801+0000 7fe774043740 1 --2- 172.21.15.118:0/1210143911 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55e554e563a0 0x55e554e654c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:18:01.806 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:01.801+0000 7fe76b200640 1 --2- 172.21.15.118:0/1210143911 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55e554e563a0 0x55e554e654c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:18:01.806 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:01.801+0000 7fe774043740 1 -- 172.21.15.118:0/1210143911 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe76c1b02a0 con 0x55e554e563a0 2024-03-24T00:18:01.806 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:01.801+0000 7fe774043740 1 -- 172.21.15.118:0/1210143911 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe76c1afc40 con 0x55e554e54810 2024-03-24T00:18:01.806 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:01.801+0000 7fe774043740 1 -- 172.21.15.118:0/1210143911 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe76c1ad140 con 0x55e554e559e0 2024-03-24T00:18:01.806 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:01.802+0000 7fe769e00640 1 -- 172.21.15.118:0/1210143911 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55e554e54810 msgr2=0x55e554e61f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:18:01.806 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:01.802+0000 7fe769e00640 1 --2- 172.21.15.118:0/1210143911 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55e554e54810 0x55e554e61f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:18:01.806 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:01.802+0000 7fe769e00640 1 -- 172.21.15.118:0/1210143911 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55e554e563a0 msgr2=0x55e554e654c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:18:01.807 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:01.802+0000 7fe769e00640 1 --2- 172.21.15.118:0/1210143911 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55e554e563a0 0x55e554e654c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:18:01.807 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:01.802+0000 7fe769e00640 1 -- 172.21.15.118:0/1210143911 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55e554e65ab0 con 0x55e554e559e0 2024-03-24T00:18:01.807 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:01.805+0000 7fe76b200640 1 --2- 172.21.15.118:0/1210143911 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55e554e563a0 0x55e554e654c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T00:18:01.807 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:01.806+0000 7fe774043740 10 client.0 _ll_register_callbacks cb 0x7fe75802c5e0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T00:18:01.807 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:01.806+0000 7fe774043740 1 -- 172.21.15.118:0/1210143911 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fe75c09bf90 con 0x55e554e559e0 2024-03-24T00:18:01.823 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:18:01 smithi118 ceph-mon[31535]: from='client.? 172.21.15.118:0/3784280489' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-24T00:18:01.825 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:01.823+0000 7fe769e00640 1 --2- 172.21.15.118:0/1210143911 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55e554e559e0 0x55e554e624a0 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 crypto rx=0x7fe75802c7c0 tx=0x7fe758005d50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=9218704f3c8b86e2 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:18:01.825 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:01.826+0000 7fe74be00640 10 client.26077 ms_handle_connect on v2:172.21.15.193:3300/0 2024-03-24T00:18:01.826 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:01.826+0000 7fe74be00640 1 -- 172.21.15.118:0/1210143911 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe758072060 con 0x55e554e559e0 2024-03-24T00:18:01.826 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:01.826+0000 7fe74be00640 1 -- 172.21.15.118:0/1210143911 <== mon.1 v2:172.21.15.193:3300/0 2 ==== osd_map(90..90 src has 1..90) v4 ==== 8277+0+0 (secure 0 0 0) 0x7fe7580042e0 con 0x55e554e559e0 2024-03-24T00:18:01.826 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:01.826+0000 7fe74be00640 1 -- 172.21.15.118:0/1210143911 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) v2 ==== 24+0+0 (secure 0 0 0) 0x7fe7580037f0 con 0x55e554e559e0 2024-03-24T00:18:01.879 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:01.860+0000 7fe774043740 10 client.26077 fetch_fsmap learned FSMap version 28 2024-03-24T00:18:01.879 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:01.860+0000 7fe774043740 1 -- 172.21.15.118:0/1210143911 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({fsmap=28}) v3 -- 0x55e554ec6410 con 0x55e554e559e0 2024-03-24T00:18:01.880 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:01.881+0000 7fe74be00640 1 -- 172.21.15.118:0/1210143911 <== mon.1 v2:172.21.15.193:3300/0 4 ==== fsmap(e 28) v1 ==== 7934+0+0 (secure 0 0 0) 0x7fe758004aa0 con 0x55e554e559e0 2024-03-24T00:18:01.897 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:01.897+0000 7fe774043740 10 client.26077 fetch_fsmap finished waiting for FSMap version 28 2024-03-24T00:18:01.897 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:01.897+0000 7fe774043740 10 client.26077 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.c 2024-03-24T00:18:01.897 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:01.897+0000 7fe774043740 20 client.26077 populate_metadata read hostname 'smithi118' 2024-03-24T00:18:01.897 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:01.898+0000 7fe774043740 1 --2- 172.21.15.118:0/1210143911 >> [v2:172.21.15.193:6836/2921520053,v1:172.21.15.193:6837/2921520053] conn(0x55e554ec6c20 0x55e554ec90e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:18:01.897 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:01.898+0000 7fe774043740 4 client.26077 mds_command: new command op to 24511 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-24T00:18:01.897 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:01.898+0000 7fe774043740 1 -- 172.21.15.118:0/1210143911 --> [v2:172.21.15.193:6836/2921520053,v1:172.21.15.193:6837/2921520053] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x55e554e5e820 con 0x55e554ec6c20 2024-03-24T00:18:01.911 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:01.912+0000 7fe76a800640 1 --2- 172.21.15.118:0/1210143911 >> [v2:172.21.15.193:6836/2921520053,v1:172.21.15.193:6837/2921520053] conn(0x55e554ec6c20 0x55e554ec90e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:18:01.912 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:01.913+0000 7fe76a800640 1 --2- 172.21.15.118:0/1210143911 >> [v2:172.21.15.193:6836/2921520053,v1:172.21.15.193:6837/2921520053] conn(0x55e554ec6c20 0x55e554ec90e0 crc :-1 s=READY pgs=26 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=4b4e0de0b5ab383c server_cookie=2462a72a000f78c3 in_seq=0 out_seq=0 2024-03-24T00:18:01.912 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:01.913+0000 7fe74be00640 10 client.26077 ms_handle_connect on v2:172.21.15.193:6836/2921520053 2024-03-24T00:18:01.913 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:01.913+0000 7fe74be00640 1 -- 172.21.15.118:0/1210143911 <== mds.2 v2:172.21.15.193:6836/2921520053 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x55e554e5e820 con 0x55e554ec6c20 2024-03-24T00:18:01.913 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:01.913+0000 7fe74be00640 10 client.26077 handle_command_reply: tid=0 2024-03-24T00:18:01.913 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:01.913+0000 7fe774043740 1 client.26077 shutdown 2024-03-24T00:18:01.913 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:01.913+0000 7fe774043740 1 -- 172.21.15.118:0/1210143911 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55e554e559e0 msgr2=0x55e554e624a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:18:01.913 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:01.913+0000 7fe774043740 1 --2- 172.21.15.118:0/1210143911 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55e554e559e0 0x55e554e624a0 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 crypto rx=0x7fe75802c7c0 tx=0x7fe758005d50 comp rx=0 tx=0).stop 2024-03-24T00:18:01.914 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:01.914+0000 7fe774043740 1 -- 172.21.15.118:0/1210143911 shutdown_connections 2024-03-24T00:18:01.914 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:01.914+0000 7fe774043740 1 --2- 172.21.15.118:0/1210143911 >> [v2:172.21.15.193:6836/2921520053,v1:172.21.15.193:6837/2921520053] conn(0x55e554ec6c20 0x55e554ec90e0 crc :-1 s=READY pgs=26 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:18:01.915 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:01.914+0000 7fe774043740 1 --2- 172.21.15.118:0/1210143911 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55e554e563a0 0x55e554e654c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:18:01.915 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:01.914+0000 7fe774043740 1 --2- 172.21.15.118:0/1210143911 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55e554e559e0 0x55e554e624a0 unknown :-1 s=CLOSED pgs=310 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:18:01.915 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:01.914+0000 7fe774043740 1 --2- 172.21.15.118:0/1210143911 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55e554e54810 0x55e554e61f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:18:01.915 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:01.914+0000 7fe774043740 1 -- 172.21.15.118:0/1210143911 >> 172.21.15.118:0/1210143911 conn(0x55e554e4fc00 msgr2=0x55e554e4e620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:18:01.915 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:01.914+0000 7fe74be00640 0 client.26077 ms_handle_reset on v2:172.21.15.193:6836/2921520053 2024-03-24T00:18:01.915 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:01.915+0000 7fe774043740 1 -- 172.21.15.118:0/1210143911 shutdown_connections 2024-03-24T00:18:01.915 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:01.915+0000 7fe774043740 1 -- 172.21.15.118:0/1210143911 wait complete. 2024-03-24T00:18:01.915 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:01.915+0000 7fe774043740 20 client.26077 trim_cache size 0 max 16384 2024-03-24T00:18:01.915 INFO:teuthology.orchestra.run.smithi118.stdout:[] 2024-03-24T00:18:01.954 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:01.951+0000 7fe771400640 1 -- 172.21.15.118:0/1954903951 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7fe7500a2800 msgr2=0x7fe7500a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:18:01.954 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:01.951+0000 7fe771400640 1 --2- 172.21.15.118:0/1954903951 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7fe7500a2800 0x7fe7500a4cc0 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 crypto rx=0x55e554e558e0 tx=0x7fe7640098a0 comp rx=0 tx=0).stop 2024-03-24T00:18:01.954 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:01.951+0000 7fe771400640 1 -- 172.21.15.118:0/1954903951 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe76c076060 msgr2=0x7fe76c19e000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:18:01.954 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:01.951+0000 7fe771400640 1 --2- 172.21.15.118:0/1954903951 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe76c076060 0x7fe76c19e000 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 crypto rx=0x7fe75c00e9f0 tx=0x7fe75c00edf0 comp rx=0 tx=0).stop 2024-03-24T00:18:01.954 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:01.952+0000 7fe771400640 1 -- 172.21.15.118:0/1954903951 shutdown_connections 2024-03-24T00:18:01.954 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:01.952+0000 7fe771400640 1 --2- 172.21.15.118:0/1954903951 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7fe7500a2800 0x7fe7500a4cc0 unknown :-1 s=CLOSED pgs=312 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:18:01.955 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:01.952+0000 7fe771400640 1 --2- 172.21.15.118:0/1954903951 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fe76c077c40 0x7fe76c1a55b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:18:01.955 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:01.952+0000 7fe771400640 1 --2- 172.21.15.118:0/1954903951 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fe76c0772a0 0x7fe76c19e540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:18:01.955 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:01.952+0000 7fe771400640 1 --2- 172.21.15.118:0/1954903951 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe76c076060 0x7fe76c19e000 unknown :-1 s=CLOSED pgs=320 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:18:01.955 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:01.952+0000 7fe771400640 1 -- 172.21.15.118:0/1954903951 >> 172.21.15.118:0/1954903951 conn(0x7fe76c0fee50 msgr2=0x7fe76c100950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:18:01.969 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:01.968+0000 7fe771400640 1 -- 172.21.15.118:0/1954903951 shutdown_connections 2024-03-24T00:18:01.969 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:01.968+0000 7fe771400640 1 -- 172.21.15.118:0/1954903951 wait complete. 2024-03-24T00:18:02.062 INFO:teuthology.orchestra.run:Running command with timeout 120 2024-03-24T00:18:02.062 DEBUG:teuthology.orchestra.run.smithi118:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:1 damage ls 2024-03-24T00:18:02.203 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:18:01 smithi189 ceph-mon[36187]: from='client.? 172.21.15.118:0/3784280489' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-24T00:18:02.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:18:01 smithi193 ceph-mon[36878]: from='client.? 172.21.15.118:0/3784280489' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-24T00:18:02.312 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:02.312+0000 7fc4dfe00640 1 -- 172.21.15.118:0/4182023628 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc4d81038e0 msgr2=0x7fc4d8103d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:18:02.312 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:02.312+0000 7fc4dfe00640 1 --2- 172.21.15.118:0/4182023628 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc4d81038e0 0x7fc4d8103d00 secure :-1 s=READY pgs=321 cs=0 l=1 rev1=1 crypto rx=0x7fc4cc009900 tx=0x7fc4cc02bc50 comp rx=0 tx=0).stop 2024-03-24T00:18:02.312 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:02.312+0000 7fc4dfe00640 1 -- 172.21.15.118:0/4182023628 shutdown_connections 2024-03-24T00:18:02.312 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:02.312+0000 7fc4dfe00640 1 --2- 172.21.15.118:0/4182023628 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fc4d8105490 0x7fc4d810bd20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:18:02.312 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:02.312+0000 7fc4dfe00640 1 --2- 172.21.15.118:0/4182023628 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fc4d8104ad0 0x7fc4d8104f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:18:02.312 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:02.312+0000 7fc4dfe00640 1 --2- 172.21.15.118:0/4182023628 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc4d81038e0 0x7fc4d8103d00 unknown :-1 s=CLOSED pgs=321 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:18:02.312 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:02.312+0000 7fc4dfe00640 1 -- 172.21.15.118:0/4182023628 >> 172.21.15.118:0/4182023628 conn(0x7fc4d80fee50 msgr2=0x7fc4d8101290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:18:02.312 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:02.312+0000 7fc4dfe00640 1 -- 172.21.15.118:0/4182023628 shutdown_connections 2024-03-24T00:18:02.312 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:02.312+0000 7fc4dfe00640 1 -- 172.21.15.118:0/4182023628 wait complete. 2024-03-24T00:18:02.313 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:02.314+0000 7fc4dfe00640 1 Processor -- start 2024-03-24T00:18:02.313 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:02.314+0000 7fc4dfe00640 1 -- start start 2024-03-24T00:18:02.314 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:02.314+0000 7fc4dfe00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fc4d81038e0 0x7fc4d8199c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:18:02.314 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:02.314+0000 7fc4dfe00640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fc4d8104ad0 0x7fc4d819a1c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:18:02.314 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:02.314+0000 7fc4dfe00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc4d8105490 0x7fc4d81a1220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:18:02.314 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:02.314+0000 7fc4dfe00640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc4d810f5a0 con 0x7fc4d8104ad0 2024-03-24T00:18:02.315 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:02.314+0000 7fc4dfe00640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc4d810f420 con 0x7fc4d8105490 2024-03-24T00:18:02.315 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:02.314+0000 7fc4dfe00640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc4d810f720 con 0x7fc4d81038e0 2024-03-24T00:18:02.315 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:02.314+0000 7fc4dd400640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fc4d81038e0 0x7fc4d8199c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:18:02.315 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:02.314+0000 7fc4dd400640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fc4d81038e0 0x7fc4d8199c80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.118:53134/0 (socket says 172.21.15.118:53134) 2024-03-24T00:18:02.315 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:02.314+0000 7fc4dd400640 1 -- 172.21.15.118:0/1760591617 learned_addr learned my addr 172.21.15.118:0/1760591617 (peer_addr_for_me v2:172.21.15.118:0/0) 2024-03-24T00:18:02.315 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:02.315+0000 7fc4dca00640 1 --2- 172.21.15.118:0/1760591617 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fc4d8104ad0 0x7fc4d819a1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:18:02.315 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:02.315+0000 7fc4dde00640 1 --2- 172.21.15.118:0/1760591617 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc4d8105490 0x7fc4d81a1220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:18:02.315 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:02.315+0000 7fc4dd400640 1 -- 172.21.15.118:0/1760591617 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc4d8105490 msgr2=0x7fc4d81a1220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:18:02.315 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:02.315+0000 7fc4dd400640 1 --2- 172.21.15.118:0/1760591617 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc4d8105490 0x7fc4d81a1220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:18:02.315 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:02.315+0000 7fc4dd400640 1 -- 172.21.15.118:0/1760591617 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fc4d8104ad0 msgr2=0x7fc4d819a1c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:18:02.315 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:02.315+0000 7fc4dd400640 1 --2- 172.21.15.118:0/1760591617 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fc4d8104ad0 0x7fc4d819a1c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:18:02.315 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:02.315+0000 7fc4dd400640 1 -- 172.21.15.118:0/1760591617 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc4d81a18c0 con 0x7fc4d81038e0 2024-03-24T00:18:02.316 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:02.315+0000 7fc4dd400640 1 --2- 172.21.15.118:0/1760591617 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fc4d81038e0 0x7fc4d8199c80 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 crypto rx=0x7fc4cc005ab0 tx=0x7fc4cc02c6e0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:18:02.316 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:02.316+0000 7fc4d6000640 1 -- 172.21.15.118:0/1760591617 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc4cc041070 con 0x7fc4d81038e0 2024-03-24T00:18:02.316 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:02.316+0000 7fc4d6000640 1 -- 172.21.15.118:0/1760591617 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fc4cc004450 con 0x7fc4d81038e0 2024-03-24T00:18:02.316 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:02.316+0000 7fc4d6000640 1 -- 172.21.15.118:0/1760591617 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc4cc034440 con 0x7fc4d81038e0 2024-03-24T00:18:02.316 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:02.316+0000 7fc4dfe00640 1 -- 172.21.15.118:0/1760591617 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc4d81a1b50 con 0x7fc4d81038e0 2024-03-24T00:18:02.317 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:02.316+0000 7fc4dfe00640 1 -- 172.21.15.118:0/1760591617 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc4d81a2050 con 0x7fc4d81038e0 2024-03-24T00:18:02.319 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:02.319+0000 7fc4e2a01740 1 Processor -- start 2024-03-24T00:18:02.319 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:02.320+0000 7fc4e2a01740 1 -- start start 2024-03-24T00:18:02.321 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:02.320+0000 7fc4e2a01740 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x560eabf63810 0x560eabf63c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:18:02.321 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:02.320+0000 7fc4e2a01740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x560eabf649e0 0x560eabf64e60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:18:02.321 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:02.320+0000 7fc4e2a01740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x560eabf653a0 0x560eabf6bc30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:18:02.321 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:02.320+0000 7fc4e2a01740 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x560eabec82c0 con 0x560eabf63810 2024-03-24T00:18:02.321 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:02.320+0000 7fc4e2a01740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x560eabe70c90 con 0x560eabf649e0 2024-03-24T00:18:02.321 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:02.320+0000 7fc4e2a01740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x560eabec4360 con 0x560eabf653a0 2024-03-24T00:18:02.321 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:02.320+0000 7fc4d6000640 1 -- 172.21.15.118:0/1760591617 <== mon.1 v2:172.21.15.193:3300/0 4 ==== mgrmap(e 20) v1 ==== 137291+0+0 (secure 0 0 0) 0x7fc4cc044070 con 0x7fc4d81038e0 2024-03-24T00:18:02.321 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:02.320+0000 7fc4dca00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x560eabf653a0 0x560eabf6bc30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:18:02.321 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:02.320+0000 7fc4dca00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x560eabf653a0 0x560eabf6bc30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.118:53148/0 (socket says 172.21.15.118:53148) 2024-03-24T00:18:02.321 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:02.320+0000 7fc4dca00640 1 -- 172.21.15.118:0/283861187 learned_addr learned my addr 172.21.15.118:0/283861187 (peer_addr_for_me v2:172.21.15.118:0/0) 2024-03-24T00:18:02.321 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:02.321+0000 7fc4dd400640 1 --2- 172.21.15.118:0/283861187 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x560eabf649e0 0x560eabf64e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:18:02.321 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:02.321+0000 7fc4dca00640 1 -- 172.21.15.118:0/283861187 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x560eabf649e0 msgr2=0x560eabf64e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:18:02.322 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:02.321+0000 7fc4dca00640 1 --2- 172.21.15.118:0/283861187 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x560eabf649e0 0x560eabf64e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:18:02.322 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:02.321+0000 7fc4dca00640 1 -- 172.21.15.118:0/283861187 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x560eabf63810 msgr2=0x560eabf63c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:18:02.322 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:02.321+0000 7fc4dca00640 1 --2- 172.21.15.118:0/283861187 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x560eabf63810 0x560eabf63c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:18:02.322 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:02.321+0000 7fc4dca00640 1 -- 172.21.15.118:0/283861187 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x560eabe95080 con 0x560eabf653a0 2024-03-24T00:18:02.322 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:02.321+0000 7fc4dd400640 1 --2- 172.21.15.118:0/283861187 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x560eabf649e0 0x560eabf64e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-24T00:18:02.322 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:02.321+0000 7fc4dca00640 1 --2- 172.21.15.118:0/283861187 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x560eabf653a0 0x560eabf6bc30 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 crypto rx=0x7fc4c4009ca0 tx=0x7fc4c4011810 comp rx=0 tx=0).ready entity=mon.1 client_cookie=3adbd3c880eb9121 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:18:02.322 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:02.321+0000 7fc4ca600640 1 -- 172.21.15.118:0/283861187 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc4c401e070 con 0x560eabf653a0 2024-03-24T00:18:02.322 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:02.321+0000 7fc4ca600640 1 -- 172.21.15.118:0/283861187 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fc4c4019b10 con 0x560eabf653a0 2024-03-24T00:18:02.322 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:02.321+0000 7fc4ca600640 1 -- 172.21.15.118:0/283861187 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc4c400ea20 con 0x560eabf653a0 2024-03-24T00:18:02.325 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:02.322+0000 7fc4d6000640 1 --2- 172.21.15.118:0/1760591617 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7fc4c00a2880 0x7fc4c00a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:18:02.325 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:02.322+0000 7fc4d6000640 1 -- 172.21.15.118:0/1760591617 <== mon.1 v2:172.21.15.193:3300/0 5 ==== osd_map(90..90 src has 1..90) v4 ==== 8277+0+0 (secure 0 0 0) 0x7fc4cc0f0140 con 0x7fc4d81038e0 2024-03-24T00:18:02.325 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:02.322+0000 7fc4dde00640 1 --2- 172.21.15.118:0/1760591617 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7fc4c00a2880 0x7fc4c00a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:18:02.325 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:02.322+0000 7fc4e2a01740 1 -- 172.21.15.118:0/283861187 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x560eabf653a0 msgr2=0x560eabf6bc30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:18:02.325 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:02.322+0000 7fc4e2a01740 1 --2- 172.21.15.118:0/283861187 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x560eabf653a0 0x560eabf6bc30 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 crypto rx=0x7fc4c4009ca0 tx=0x7fc4c4011810 comp rx=0 tx=0).stop 2024-03-24T00:18:02.325 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:02.322+0000 7fc4e2a01740 1 -- 172.21.15.118:0/283861187 shutdown_connections 2024-03-24T00:18:02.325 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:02.322+0000 7fc4e2a01740 1 --2- 172.21.15.118:0/283861187 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x560eabf653a0 0x560eabf6bc30 unknown :-1 s=CLOSED pgs=312 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:18:02.326 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:02.322+0000 7fc4e2a01740 1 --2- 172.21.15.118:0/283861187 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x560eabf649e0 0x560eabf64e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:18:02.326 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:02.322+0000 7fc4e2a01740 1 --2- 172.21.15.118:0/283861187 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x560eabf63810 0x560eabf63c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:18:02.326 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:02.322+0000 7fc4e2a01740 1 -- 172.21.15.118:0/283861187 >> 172.21.15.118:0/283861187 conn(0x560eabf5ec00 msgr2=0x560eabf5f000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:18:02.326 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:02.322+0000 7fc4e2a01740 1 -- 172.21.15.118:0/283861187 shutdown_connections 2024-03-24T00:18:02.326 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:02.323+0000 7fc4e2a01740 1 -- 172.21.15.118:0/283861187 wait complete. 2024-03-24T00:18:02.326 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:02.323+0000 7fc4dde00640 1 --2- 172.21.15.118:0/1760591617 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7fc4c00a2880 0x7fc4c00a4d40 secure :-1 s=READY pgs=313 cs=0 l=1 rev1=1 crypto rx=0x560eabf648e0 tx=0x7fc4d0009b00 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:18:02.326 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:02.323+0000 7fc4e2a01740 1 -- start start 2024-03-24T00:18:02.326 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:02.323+0000 7fc4e2a01740 1 Processor -- start 2024-03-24T00:18:02.326 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:02.324+0000 7fc4e2a01740 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x560eabf63810 0x560eabf6da60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:18:02.326 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:02.324+0000 7fc4e2a01740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x560eabf649e0 0x560eabf6dfa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:18:02.326 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:02.324+0000 7fc4e2a01740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x560eabf653a0 0x560eabf74fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:18:02.326 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:02.324+0000 7fc4e2a01740 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x560eabf6c3b0 con 0x560eabf63810 2024-03-24T00:18:02.326 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:02.324+0000 7fc4e2a01740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x560eabe840d0 con 0x560eabf649e0 2024-03-24T00:18:02.327 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:02.324+0000 7fc4e2a01740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x560eabf6c530 con 0x560eabf653a0 2024-03-24T00:18:02.327 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:02.324+0000 7fc4e2a01740 10 client.0 _ll_register_callbacks cb 0x560eabe88e80 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T00:18:02.327 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:02.324+0000 7fc4dd400640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x560eabf63810 0x560eabf6da60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:18:02.327 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:02.324+0000 7fc4dca00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x560eabf649e0 0x560eabf6dfa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:18:02.327 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:02.324+0000 7fc4dca00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x560eabf649e0 0x560eabf6dfa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.118:38214/0 (socket says 172.21.15.118:38214) 2024-03-24T00:18:02.327 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:02.324+0000 7fc4dca00640 1 -- 172.21.15.118:0/3179467830 learned_addr learned my addr 172.21.15.118:0/3179467830 (peer_addr_for_me v2:172.21.15.118:0/0) 2024-03-24T00:18:02.327 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:02.324+0000 7fc4dd400640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x560eabf63810 0x560eabf6da60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.118:3300/0 says I am v2:172.21.15.118:48762/0 (socket says 172.21.15.118:48762) 2024-03-24T00:18:02.327 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:02.324+0000 7fc4dde00640 1 --2- 172.21.15.118:0/3179467830 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x560eabf653a0 0x560eabf74fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:18:02.327 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:02.324+0000 7fc4dca00640 1 -- 172.21.15.118:0/3179467830 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x560eabf653a0 msgr2=0x560eabf74fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:18:02.327 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:02.324+0000 7fc4dca00640 1 --2- 172.21.15.118:0/3179467830 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x560eabf653a0 0x560eabf74fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:18:02.327 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:02.324+0000 7fc4dca00640 1 -- 172.21.15.118:0/3179467830 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x560eabf63810 msgr2=0x560eabf6da60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:18:02.327 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:02.324+0000 7fc4dca00640 1 --2- 172.21.15.118:0/3179467830 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x560eabf63810 0x560eabf6da60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:18:02.327 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:02.324+0000 7fc4dca00640 1 -- 172.21.15.118:0/3179467830 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x560eabbdc3f0 con 0x560eabf649e0 2024-03-24T00:18:02.327 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:02.324+0000 7fc4dd400640 1 --2- 172.21.15.118:0/3179467830 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x560eabf63810 0x560eabf6da60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T00:18:02.328 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:02.325+0000 7fc4dca00640 1 --2- 172.21.15.118:0/3179467830 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x560eabf649e0 0x560eabf6dfa0 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 crypto rx=0x7fc4c4009710 tx=0x7fc4c40196e0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b207a41d5f334cd5 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:18:02.328 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:02.325+0000 7fc4bea00640 10 client.0 ms_handle_connect on v2:172.21.15.189:3300/0 2024-03-24T00:18:02.328 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:02.325+0000 7fc4bea00640 1 -- 172.21.15.118:0/3179467830 <== mon.2 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc4c4028020 con 0x560eabf649e0 2024-03-24T00:18:02.328 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:02.325+0000 7fc4e2a01740 1 -- 172.21.15.118:0/3179467830 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fc4c4019b10 con 0x560eabf649e0 2024-03-24T00:18:02.328 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:02.325+0000 7fc4bea00640 1 -- 172.21.15.118:0/3179467830 <== mon.2 v2:172.21.15.189:3300/0 2 ==== osd_map(90..90 src has 1..90) v4 ==== 8277+0+0 (secure 0 0 0) 0x7fc4c4005050 con 0x560eabf649e0 2024-03-24T00:18:02.328 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:02.325+0000 7fc4bea00640 1 -- 172.21.15.118:0/3179467830 <== mon.2 v2:172.21.15.189:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) v2 ==== 24+0+0 (secure 0 0 0) 0x7fc4c4033050 con 0x560eabf649e0 2024-03-24T00:18:02.328 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:02.325+0000 7fc4e2a01740 10 client.26093 fetch_fsmap learned FSMap version 28 2024-03-24T00:18:02.328 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:02.325+0000 7fc4e2a01740 1 -- 172.21.15.118:0/3179467830 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) v3 -- 0x560eabca67d0 con 0x560eabf649e0 2024-03-24T00:18:02.328 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:02.325+0000 7fc4bea00640 1 -- 172.21.15.118:0/3179467830 <== mon.2 v2:172.21.15.189:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fc4c4019b10 con 0x560eabf649e0 2024-03-24T00:18:02.330 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:02.326+0000 7fc4bea00640 1 -- 172.21.15.118:0/3179467830 <== mon.2 v2:172.21.15.189:3300/0 5 ==== fsmap(e 28) v1 ==== 7934+0+0 (secure 0 0 0) 0x7fc4c40046c0 con 0x560eabf649e0 2024-03-24T00:18:02.330 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:02.326+0000 7fc4e2a01740 10 client.26093 fetch_fsmap finished waiting for FSMap version 28 2024-03-24T00:18:02.330 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:02.326+0000 7fc4e2a01740 10 client.26093 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.g 2024-03-24T00:18:02.330 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:02.326+0000 7fc4e2a01740 20 client.26093 populate_metadata read hostname 'smithi118' 2024-03-24T00:18:02.330 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:02.326+0000 7fc4e2a01740 1 --2- 172.21.15.118:0/3179467830 >> [v2:172.21.15.118:6838/3285788503,v1:172.21.15.118:6839/3285788503] conn(0x560eabf791b0 0x560eabf7b5c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:18:02.330 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:02.326+0000 7fc4e2a01740 4 client.26093 mds_command: new command op to 24550 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-24T00:18:02.330 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:02.326+0000 7fc4e2a01740 1 -- 172.21.15.118:0/3179467830 --> [v2:172.21.15.118:6838/3285788503,v1:172.21.15.118:6839/3285788503] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x560eabed5800 con 0x560eabf791b0 2024-03-24T00:18:02.330 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:02.327+0000 7fc4dd400640 1 --2- 172.21.15.118:0/3179467830 >> [v2:172.21.15.118:6838/3285788503,v1:172.21.15.118:6839/3285788503] conn(0x560eabf791b0 0x560eabf7b5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:18:02.330 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:02.327+0000 7fc4dd400640 1 --2- 172.21.15.118:0/3179467830 >> [v2:172.21.15.118:6838/3285788503,v1:172.21.15.118:6839/3285788503] conn(0x560eabf791b0 0x560eabf7b5c0 crc :-1 s=READY pgs=28 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=831629d33bd161ff server_cookie=29e4aa0e1f1f0051 in_seq=0 out_seq=0 2024-03-24T00:18:02.330 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:02.327+0000 7fc4bea00640 10 client.26093 ms_handle_connect on v2:172.21.15.118:6838/3285788503 2024-03-24T00:18:02.331 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:02.328+0000 7fc4bea00640 1 -- 172.21.15.118:0/3179467830 <== mds.1 v2:172.21.15.118:6838/3285788503 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x560eabed5800 con 0x560eabf791b0 2024-03-24T00:18:02.331 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:02.328+0000 7fc4bea00640 10 client.26093 handle_command_reply: tid=0 2024-03-24T00:18:02.331 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:02.328+0000 7fc4e2a01740 1 client.26093 shutdown 2024-03-24T00:18:02.331 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:02.328+0000 7fc4e2a01740 1 -- 172.21.15.118:0/3179467830 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x560eabf649e0 msgr2=0x560eabf6dfa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:18:02.331 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:02.328+0000 7fc4e2a01740 1 --2- 172.21.15.118:0/3179467830 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x560eabf649e0 0x560eabf6dfa0 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 crypto rx=0x7fc4c4009710 tx=0x7fc4c40196e0 comp rx=0 tx=0).stop 2024-03-24T00:18:02.331 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:02.328+0000 7fc4e2a01740 1 -- 172.21.15.118:0/3179467830 shutdown_connections 2024-03-24T00:18:02.331 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:02.328+0000 7fc4e2a01740 1 --2- 172.21.15.118:0/3179467830 >> [v2:172.21.15.118:6838/3285788503,v1:172.21.15.118:6839/3285788503] conn(0x560eabf791b0 0x560eabf7b5c0 crc :-1 s=READY pgs=28 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:18:02.331 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:02.328+0000 7fc4e2a01740 1 --2- 172.21.15.118:0/3179467830 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x560eabf653a0 0x560eabf74fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:18:02.331 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:02.328+0000 7fc4e2a01740 1 --2- 172.21.15.118:0/3179467830 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x560eabf649e0 0x560eabf6dfa0 unknown :-1 s=CLOSED pgs=322 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:18:02.331 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:02.328+0000 7fc4bea00640 0 client.26093 ms_handle_reset on v2:172.21.15.118:6838/3285788503 2024-03-24T00:18:02.331 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:02.328+0000 7fc4e2a01740 1 --2- 172.21.15.118:0/3179467830 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x560eabf63810 0x560eabf6da60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:18:02.331 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:02.328+0000 7fc4e2a01740 1 -- 172.21.15.118:0/3179467830 >> 172.21.15.118:0/3179467830 conn(0x560eabf5ec00 msgr2=0x560eabf5deb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:18:02.331 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:02.329+0000 7fc4e2a01740 1 -- 172.21.15.118:0/3179467830 shutdown_connections 2024-03-24T00:18:02.331 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:02.329+0000 7fc4e2a01740 1 -- 172.21.15.118:0/3179467830 wait complete. 2024-03-24T00:18:02.332 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:02.329+0000 7fc4e2a01740 20 client.26093 trim_cache size 0 max 16384 2024-03-24T00:18:02.343 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:02.343+0000 7fc4e2a01740 1 Processor -- start 2024-03-24T00:18:02.343 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:02.343+0000 7fc4e2a01740 1 -- start start 2024-03-24T00:18:02.343 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:02.343+0000 7fc4e2a01740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x560eabf63810 0x560eabf67c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:18:02.343 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:02.343+0000 7fc4e2a01740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x560eabf649e0 0x560eabf74fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:18:02.343 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:02.343+0000 7fc4e2a01740 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x560eabf653a0 0x560eabf68150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:18:02.343 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:02.343+0000 7fc4e2a01740 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc4d81aac20 con 0x560eabf653a0 2024-03-24T00:18:02.343 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:02.343+0000 7fc4e2a01740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc4d81ac5a0 con 0x560eabf63810 2024-03-24T00:18:02.343 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:02.343+0000 7fc4e2a01740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc4d81ab280 con 0x560eabf649e0 2024-03-24T00:18:02.343 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:02.343+0000 7fc4dca00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x560eabf649e0 0x560eabf74fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:18:02.343 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:02.343+0000 7fc4dca00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x560eabf649e0 0x560eabf74fa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.118:53170/0 (socket says 172.21.15.118:53170) 2024-03-24T00:18:02.343 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:02.343+0000 7fc4dca00640 1 -- 172.21.15.118:0/1111148033 learned_addr learned my addr 172.21.15.118:0/1111148033 (peer_addr_for_me v2:172.21.15.118:0/0) 2024-03-24T00:18:02.344 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:02.343+0000 7fc4dd400640 1 --2- 172.21.15.118:0/1111148033 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x560eabf63810 0x560eabf67c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:18:02.344 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:02.343+0000 7fc4dde00640 1 --2- 172.21.15.118:0/1111148033 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x560eabf653a0 0x560eabf68150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:18:02.345 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:02.344+0000 7fc4dca00640 1 -- 172.21.15.118:0/1111148033 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x560eabf63810 msgr2=0x560eabf67c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:18:02.345 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:02.344+0000 7fc4dca00640 1 --2- 172.21.15.118:0/1111148033 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x560eabf63810 0x560eabf67c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:18:02.345 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:02.344+0000 7fc4dca00640 1 -- 172.21.15.118:0/1111148033 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x560eabf653a0 msgr2=0x560eabf68150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:18:02.345 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:02.344+0000 7fc4dca00640 1 --2- 172.21.15.118:0/1111148033 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x560eabf653a0 0x560eabf68150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:18:02.345 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:02.344+0000 7fc4dca00640 1 -- 172.21.15.118:0/1111148033 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x560eabf64460 con 0x560eabf649e0 2024-03-24T00:18:02.345 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:02.344+0000 7fc4dca00640 1 --2- 172.21.15.118:0/1111148033 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x560eabf649e0 0x560eabf74fa0 secure :-1 s=READY pgs=313 cs=0 l=1 rev1=1 crypto rx=0x7fc4c4004a30 tx=0x7fc4c4011810 comp rx=0 tx=0).ready entity=mon.1 client_cookie=93a1e70e860eac8 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:18:02.345 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:02.344+0000 7fc4c9c00640 1 -- 172.21.15.118:0/1111148033 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc4c4028070 con 0x560eabf649e0 2024-03-24T00:18:02.345 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:02.344+0000 7fc4c9c00640 1 -- 172.21.15.118:0/1111148033 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fc4c4004020 con 0x560eabf649e0 2024-03-24T00:18:02.346 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:02.344+0000 7fc4c9c00640 1 -- 172.21.15.118:0/1111148033 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc4c401e040 con 0x560eabf649e0 2024-03-24T00:18:02.346 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:02.345+0000 7fc4e2a01740 1 -- 172.21.15.118:0/1111148033 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x560eabf649e0 msgr2=0x560eabf74fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:18:02.346 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:02.345+0000 7fc4e2a01740 1 --2- 172.21.15.118:0/1111148033 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x560eabf649e0 0x560eabf74fa0 secure :-1 s=READY pgs=313 cs=0 l=1 rev1=1 crypto rx=0x7fc4c4004a30 tx=0x7fc4c4011810 comp rx=0 tx=0).stop 2024-03-24T00:18:02.346 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:02.345+0000 7fc4e2a01740 1 -- 172.21.15.118:0/1111148033 shutdown_connections 2024-03-24T00:18:02.346 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:02.345+0000 7fc4e2a01740 1 --2- 172.21.15.118:0/1111148033 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x560eabf653a0 0x560eabf68150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:18:02.346 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:02.345+0000 7fc4e2a01740 1 --2- 172.21.15.118:0/1111148033 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x560eabf649e0 0x560eabf74fa0 unknown :-1 s=CLOSED pgs=313 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:18:02.346 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:02.345+0000 7fc4e2a01740 1 --2- 172.21.15.118:0/1111148033 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x560eabf63810 0x560eabf67c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:18:02.346 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:02.345+0000 7fc4e2a01740 1 -- 172.21.15.118:0/1111148033 >> 172.21.15.118:0/1111148033 conn(0x560eabf5ec00 msgr2=0x560eabf626f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:18:02.346 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:02.345+0000 7fc4e2a01740 1 -- 172.21.15.118:0/1111148033 shutdown_connections 2024-03-24T00:18:02.346 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:02.345+0000 7fc4e2a01740 1 -- 172.21.15.118:0/1111148033 wait complete. 2024-03-24T00:18:02.346 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:02.345+0000 7fc4e2a01740 1 -- start start 2024-03-24T00:18:02.346 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:02.346+0000 7fc4e2a01740 1 Processor -- start 2024-03-24T00:18:02.351 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:02.346+0000 7fc4e2a01740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x560eabf63810 0x560eabf72a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:18:02.351 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:02.346+0000 7fc4e2a01740 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x560eabf649e0 0x560eabf72f50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:18:02.351 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:02.346+0000 7fc4e2a01740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x560eabf653a0 0x560eabf73f60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:18:02.351 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:02.346+0000 7fc4e2a01740 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc4d81abf40 con 0x560eabf649e0 2024-03-24T00:18:02.351 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:02.346+0000 7fc4e2a01740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc4d81ab8e0 con 0x560eabf63810 2024-03-24T00:18:02.351 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:02.346+0000 7fc4e2a01740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc4d81a8de0 con 0x560eabf653a0 2024-03-24T00:18:02.351 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:02.346+0000 7fc4e2a01740 10 client.0 _ll_register_callbacks cb 0x7fc4c401c070 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T00:18:02.351 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:02.346+0000 7fc4dca00640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x560eabf649e0 0x560eabf72f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:18:02.351 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:02.346+0000 7fc4dde00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x560eabf653a0 0x560eabf73f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:18:02.351 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:02.346+0000 7fc4dde00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x560eabf653a0 0x560eabf73f60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.118:53176/0 (socket says 172.21.15.118:53176) 2024-03-24T00:18:02.352 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:02.346+0000 7fc4dca00640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x560eabf649e0 0x560eabf72f50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.118:3300/0 says I am v2:172.21.15.118:48778/0 (socket says 172.21.15.118:48778) 2024-03-24T00:18:02.352 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:02.346+0000 7fc4dde00640 1 -- 172.21.15.118:0/4182676411 learned_addr learned my addr 172.21.15.118:0/4182676411 (peer_addr_for_me v2:172.21.15.118:0/0) 2024-03-24T00:18:02.352 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:02.347+0000 7fc4dd400640 1 --2- 172.21.15.118:0/4182676411 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x560eabf63810 0x560eabf72a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:18:02.352 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:02.347+0000 7fc4dde00640 1 -- 172.21.15.118:0/4182676411 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x560eabf63810 msgr2=0x560eabf72a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:18:02.352 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:02.347+0000 7fc4dde00640 1 --2- 172.21.15.118:0/4182676411 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x560eabf63810 0x560eabf72a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:18:02.352 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:02.347+0000 7fc4dde00640 1 -- 172.21.15.118:0/4182676411 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x560eabf649e0 msgr2=0x560eabf72f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:18:02.352 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:02.347+0000 7fc4dde00640 1 --2- 172.21.15.118:0/4182676411 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x560eabf649e0 0x560eabf72f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:18:02.352 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:02.347+0000 7fc4dde00640 1 -- 172.21.15.118:0/4182676411 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x560eabf74550 con 0x560eabf653a0 2024-03-24T00:18:02.352 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:02.347+0000 7fc4dca00640 1 --2- 172.21.15.118:0/4182676411 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x560eabf649e0 0x560eabf72f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T00:18:02.352 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:02.347+0000 7fc4dd400640 1 --2- 172.21.15.118:0/4182676411 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x560eabf63810 0x560eabf72a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-24T00:18:02.352 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:02.347+0000 7fc4dde00640 1 --2- 172.21.15.118:0/4182676411 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x560eabf653a0 0x560eabf73f60 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 crypto rx=0x7fc4d00117e0 tx=0x7fc4d00096a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=bb8f97ad32226e3c server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:18:02.352 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:02.347+0000 7fc4bea00640 10 client.0 ms_handle_connect on v2:172.21.15.193:3300/0 2024-03-24T00:18:02.352 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:02.347+0000 7fc4bea00640 1 -- 172.21.15.118:0/4182676411 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc4d001f930 con 0x560eabf653a0 2024-03-24T00:18:02.352 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:02.347+0000 7fc4bea00640 1 -- 172.21.15.118:0/4182676411 <== mon.1 v2:172.21.15.193:3300/0 2 ==== osd_map(90..90 src has 1..90) v4 ==== 8277+0+0 (secure 0 0 0) 0x7fc4d001fad0 con 0x560eabf653a0 2024-03-24T00:18:02.353 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:02.348+0000 7fc4e2a01740 1 -- 172.21.15.118:0/4182676411 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fc4c4004020 con 0x560eabf653a0 2024-03-24T00:18:02.353 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:02.348+0000 7fc4bea00640 1 -- 172.21.15.118:0/4182676411 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) v2 ==== 24+0+0 (secure 0 0 0) 0x7fc4d001fcd0 con 0x560eabf653a0 2024-03-24T00:18:02.353 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:02.348+0000 7fc4e2a01740 10 client.26095 fetch_fsmap learned FSMap version 28 2024-03-24T00:18:02.353 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:02.348+0000 7fc4e2a01740 1 -- 172.21.15.118:0/4182676411 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) v3 -- 0x560eabfe5ef0 con 0x560eabf653a0 2024-03-24T00:18:02.353 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:02.348+0000 7fc4bea00640 1 -- 172.21.15.118:0/4182676411 <== mon.1 v2:172.21.15.193:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fc4c4004020 con 0x560eabf653a0 2024-03-24T00:18:02.353 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:02.349+0000 7fc4bea00640 1 -- 172.21.15.118:0/4182676411 <== mon.1 v2:172.21.15.193:3300/0 5 ==== fsmap(e 28) v1 ==== 7934+0+0 (secure 0 0 0) 0x7fc4d0033470 con 0x560eabf653a0 2024-03-24T00:18:02.353 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:02.349+0000 7fc4e2a01740 10 client.26095 fetch_fsmap finished waiting for FSMap version 28 2024-03-24T00:18:02.353 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:02.349+0000 7fc4e2a01740 10 client.26095 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.g 2024-03-24T00:18:02.353 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:02.349+0000 7fc4e2a01740 20 client.26095 populate_metadata read hostname 'smithi118' 2024-03-24T00:18:02.353 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:02.349+0000 7fc4e2a01740 1 --2- 172.21.15.118:0/4182676411 >> [v2:172.21.15.118:6838/3285788503,v1:172.21.15.118:6839/3285788503] conn(0x560eabfe6850 0x560eabfe8c70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:18:02.353 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:02.349+0000 7fc4e2a01740 4 client.26095 mds_command: new command op to 24550 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-24T00:18:02.353 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:02.349+0000 7fc4e2a01740 1 -- 172.21.15.118:0/4182676411 --> [v2:172.21.15.118:6838/3285788503,v1:172.21.15.118:6839/3285788503] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x560eabecfb50 con 0x560eabfe6850 2024-03-24T00:18:02.353 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:02.349+0000 7fc4dd400640 1 --2- 172.21.15.118:0/4182676411 >> [v2:172.21.15.118:6838/3285788503,v1:172.21.15.118:6839/3285788503] conn(0x560eabfe6850 0x560eabfe8c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:18:02.354 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:02.350+0000 7fc4dd400640 1 --2- 172.21.15.118:0/4182676411 >> [v2:172.21.15.118:6838/3285788503,v1:172.21.15.118:6839/3285788503] conn(0x560eabfe6850 0x560eabfe8c70 crc :-1 s=READY pgs=29 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=d6ac36ccadd62af3 server_cookie=b576d7ba44c192bd in_seq=0 out_seq=0 2024-03-24T00:18:02.354 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:02.350+0000 7fc4bea00640 10 client.26095 ms_handle_connect on v2:172.21.15.118:6838/3285788503 2024-03-24T00:18:02.354 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:02.350+0000 7fc4bea00640 1 -- 172.21.15.118:0/4182676411 <== mds.1 v2:172.21.15.118:6838/3285788503 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x560eabecfb50 con 0x560eabfe6850 2024-03-24T00:18:02.354 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:02.350+0000 7fc4bea00640 10 client.26095 handle_command_reply: tid=0 2024-03-24T00:18:02.354 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:02.350+0000 7fc4e2a01740 1 client.26095 shutdown 2024-03-24T00:18:02.354 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:02.350+0000 7fc4e2a01740 1 -- 172.21.15.118:0/4182676411 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x560eabf653a0 msgr2=0x560eabf73f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:18:02.354 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:02.350+0000 7fc4e2a01740 1 --2- 172.21.15.118:0/4182676411 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x560eabf653a0 0x560eabf73f60 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 crypto rx=0x7fc4d00117e0 tx=0x7fc4d00096a0 comp rx=0 tx=0).stop 2024-03-24T00:18:02.354 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:02.350+0000 7fc4e2a01740 1 -- 172.21.15.118:0/4182676411 shutdown_connections 2024-03-24T00:18:02.354 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:02.350+0000 7fc4e2a01740 1 --2- 172.21.15.118:0/4182676411 >> [v2:172.21.15.118:6838/3285788503,v1:172.21.15.118:6839/3285788503] conn(0x560eabfe6850 0x560eabfe8c70 crc :-1 s=READY pgs=29 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:18:02.354 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:02.350+0000 7fc4e2a01740 1 --2- 172.21.15.118:0/4182676411 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x560eabf653a0 0x560eabf73f60 unknown :-1 s=CLOSED pgs=314 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:18:02.354 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:02.350+0000 7fc4e2a01740 1 --2- 172.21.15.118:0/4182676411 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x560eabf649e0 0x560eabf72f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:18:02.354 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:02.350+0000 7fc4e2a01740 1 --2- 172.21.15.118:0/4182676411 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x560eabf63810 0x560eabf72a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:18:02.354 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:02.350+0000 7fc4bea00640 0 client.26095 ms_handle_reset on v2:172.21.15.118:6838/3285788503 2024-03-24T00:18:02.354 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:02.350+0000 7fc4e2a01740 1 -- 172.21.15.118:0/4182676411 >> 172.21.15.118:0/4182676411 conn(0x560eabf5ec00 msgr2=0x560eabf5d620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:18:02.354 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:02.350+0000 7fc4e2a01740 1 -- 172.21.15.118:0/4182676411 shutdown_connections 2024-03-24T00:18:02.355 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:02.350+0000 7fc4e2a01740 1 -- 172.21.15.118:0/4182676411 wait complete. 2024-03-24T00:18:02.355 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:02.350+0000 7fc4e2a01740 20 client.26095 trim_cache size 0 max 16384 2024-03-24T00:18:02.355 INFO:teuthology.orchestra.run.smithi118.stdout:[] 2024-03-24T00:18:02.356 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:02.351+0000 7fc4dfe00640 1 -- 172.21.15.118:0/1760591617 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7fc4c00a2880 msgr2=0x7fc4c00a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:18:02.356 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:02.351+0000 7fc4dfe00640 1 --2- 172.21.15.118:0/1760591617 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7fc4c00a2880 0x7fc4c00a4d40 secure :-1 s=READY pgs=313 cs=0 l=1 rev1=1 crypto rx=0x560eabf648e0 tx=0x7fc4d0009b00 comp rx=0 tx=0).stop 2024-03-24T00:18:02.356 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:02.351+0000 7fc4dfe00640 1 -- 172.21.15.118:0/1760591617 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fc4d81038e0 msgr2=0x7fc4d8199c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:18:02.356 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:02.351+0000 7fc4dfe00640 1 --2- 172.21.15.118:0/1760591617 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fc4d81038e0 0x7fc4d8199c80 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 crypto rx=0x7fc4cc005ab0 tx=0x7fc4cc02c6e0 comp rx=0 tx=0).stop 2024-03-24T00:18:02.356 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:02.351+0000 7fc4dfe00640 1 -- 172.21.15.118:0/1760591617 shutdown_connections 2024-03-24T00:18:02.356 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:02.351+0000 7fc4dfe00640 1 --2- 172.21.15.118:0/1760591617 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7fc4c00a2880 0x7fc4c00a4d40 unknown :-1 s=CLOSED pgs=313 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:18:02.356 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:02.351+0000 7fc4dfe00640 1 --2- 172.21.15.118:0/1760591617 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc4d8105490 0x7fc4d81a1220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:18:02.356 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:02.351+0000 7fc4dfe00640 1 --2- 172.21.15.118:0/1760591617 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fc4d8104ad0 0x7fc4d819a1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:18:02.356 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:02.351+0000 7fc4dfe00640 1 --2- 172.21.15.118:0/1760591617 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fc4d81038e0 0x7fc4d8199c80 unknown :-1 s=CLOSED pgs=311 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:18:02.356 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:02.351+0000 7fc4dfe00640 1 -- 172.21.15.118:0/1760591617 >> 172.21.15.118:0/1760591617 conn(0x7fc4d80fee50 msgr2=0x7fc4d8107e20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:18:02.356 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:02.351+0000 7fc4dfe00640 1 -- 172.21.15.118:0/1760591617 shutdown_connections 2024-03-24T00:18:02.356 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:02.351+0000 7fc4dfe00640 1 -- 172.21.15.118:0/1760591617 wait complete. 2024-03-24T00:18:02.367 INFO:teuthology.orchestra.run:Running command with timeout 120 2024-03-24T00:18:02.368 DEBUG:teuthology.orchestra.run.smithi118:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2024-03-24T00:18:02.682 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:02.682+0000 7f03a1600640 1 -- 172.21.15.118:0/1030946976 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f039c0ffd90 msgr2=0x7f039c106d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:18:02.682 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:02.682+0000 7f03a1600640 1 --2- 172.21.15.118:0/1030946976 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f039c0ffd90 0x7f039c106d90 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7f0388009860 tx=0x7f038802bb90 comp rx=0 tx=0).stop 2024-03-24T00:18:02.683 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:02.684+0000 7f03a1600640 1 -- 172.21.15.118:0/1030946976 shutdown_connections 2024-03-24T00:18:02.684 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:02.684+0000 7f03a1600640 1 --2- 172.21.15.118:0/1030946976 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f039c107320 0x7f039c1097e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:18:02.684 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:02.684+0000 7f03a1600640 1 --2- 172.21.15.118:0/1030946976 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f039c0ffd90 0x7f039c106d90 unknown :-1 s=CLOSED pgs=198 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:18:02.684 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:02.684+0000 7f03a1600640 1 --2- 172.21.15.118:0/1030946976 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f039c0ff430 0x7f039c0ff850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:18:02.684 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:02.684+0000 7f03a1600640 1 -- 172.21.15.118:0/1030946976 >> 172.21.15.118:0/1030946976 conn(0x7f039c0fae60 msgr2=0x7f039c0fd2a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:18:02.684 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:02.684+0000 7f03a1600640 1 -- 172.21.15.118:0/1030946976 shutdown_connections 2024-03-24T00:18:02.684 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:02.684+0000 7f03a1600640 1 -- 172.21.15.118:0/1030946976 wait complete. 2024-03-24T00:18:02.687 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:02.686+0000 7f03a1600640 1 Processor -- start 2024-03-24T00:18:02.688 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:02.688+0000 7f03a1600640 1 -- start start 2024-03-24T00:18:02.689 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:02.690+0000 7f03a1600640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f039c0ff430 0x7f039c0700d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:18:02.689 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:02.690+0000 7f03a1600640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f039c0ffd90 0x7f039c070610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:18:02.689 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:02.690+0000 7f03a1600640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f039c107320 0x7f039c1a9c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:18:02.690 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:02.690+0000 7f03a1600640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f039c10f540 con 0x7f039c107320 2024-03-24T00:18:02.690 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:02.690+0000 7f03a1600640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f039c10f3c0 con 0x7f039c0ff430 2024-03-24T00:18:02.690 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:02.690+0000 7f03a1600640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f039c10f6c0 con 0x7f039c0ffd90 2024-03-24T00:18:02.690 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:02.690+0000 7f039a000640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f039c0ffd90 0x7f039c070610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:18:02.690 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:02.690+0000 7f039a000640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f039c0ffd90 0x7f039c070610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.118:53200/0 (socket says 172.21.15.118:53200) 2024-03-24T00:18:02.690 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:02.690+0000 7f039a000640 1 -- 172.21.15.118:0/4282678437 learned_addr learned my addr 172.21.15.118:0/4282678437 (peer_addr_for_me v2:172.21.15.118:0/0) 2024-03-24T00:18:02.691 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:02.690+0000 7f039aa00640 1 --2- 172.21.15.118:0/4282678437 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f039c0ff430 0x7f039c0700d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:18:02.691 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:02.691+0000 7f039a000640 1 -- 172.21.15.118:0/4282678437 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f039c0ff430 msgr2=0x7f039c0700d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:18:02.691 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:02.691+0000 7f039a000640 1 --2- 172.21.15.118:0/4282678437 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f039c0ff430 0x7f039c0700d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:18:02.691 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:02.691+0000 7f039a000640 1 -- 172.21.15.118:0/4282678437 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f039c107320 msgr2=0x7f039c1a9c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:18:02.691 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:02.691+0000 7f039a000640 1 --2- 172.21.15.118:0/4282678437 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f039c107320 0x7f039c1a9c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:18:02.691 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:02.691+0000 7f039a000640 1 -- 172.21.15.118:0/4282678437 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f039c1aa150 con 0x7f039c0ffd90 2024-03-24T00:18:02.692 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:02.693+0000 7f039a000640 1 --2- 172.21.15.118:0/4282678437 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f039c0ffd90 0x7f039c070610 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 crypto rx=0x7f038802c0d0 tx=0x7f03880042f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:18:02.693 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:02.693+0000 7f0393400640 1 -- 172.21.15.118:0/4282678437 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0388041020 con 0x7f039c0ffd90 2024-03-24T00:18:02.693 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:02.693+0000 7f0393400640 1 -- 172.21.15.118:0/4282678437 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f03880347b0 con 0x7f039c0ffd90 2024-03-24T00:18:02.693 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:02.693+0000 7f0393400640 1 -- 172.21.15.118:0/4282678437 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0388039040 con 0x7f039c0ffd90 2024-03-24T00:18:02.694 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:02.693+0000 7f03a1600640 1 -- 172.21.15.118:0/4282678437 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f039c1aa320 con 0x7f039c0ffd90 2024-03-24T00:18:02.694 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:02.694+0000 7f03a1600640 1 -- 172.21.15.118:0/4282678437 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f039c1aa790 con 0x7f039c0ffd90 2024-03-24T00:18:02.702 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:02.703+0000 7f03a43ce740 1 Processor -- start 2024-03-24T00:18:02.703 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:02.703+0000 7f03a43ce740 1 -- start start 2024-03-24T00:18:02.705 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:02.704+0000 7f0393400640 1 -- 172.21.15.118:0/4282678437 <== mon.1 v2:172.21.15.193:3300/0 4 ==== mgrmap(e 20) v1 ==== 137291+0+0 (secure 0 0 0) 0x7f0388044020 con 0x7f039c0ffd90 2024-03-24T00:18:02.705 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:02.706+0000 7f03a43ce740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x560227205020 0x560227205440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:18:02.705 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:02.706+0000 7f03a43ce740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5602272061f0 0x560227206670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:18:02.706 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:02.706+0000 7f03a43ce740 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x560227206bb0 0x56022720d440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:18:02.706 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:02.706+0000 7f03a43ce740 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x5602271612c0 con 0x560227206bb0 2024-03-24T00:18:02.706 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:02.706+0000 7f03a43ce740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x560227109c90 con 0x5602272061f0 2024-03-24T00:18:02.706 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:02.706+0000 7f03a43ce740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x56022715d360 con 0x560227205020 2024-03-24T00:18:02.706 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:02.706+0000 7f039aa00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5602272061f0 0x560227206670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:18:02.706 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:02.706+0000 7f039aa00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5602272061f0 0x560227206670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.118:38268/0 (socket says 172.21.15.118:38268) 2024-03-24T00:18:02.706 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:02.706+0000 7f039aa00640 1 -- 172.21.15.118:0/3591934014 learned_addr learned my addr 172.21.15.118:0/3591934014 (peer_addr_for_me v2:172.21.15.118:0/0) 2024-03-24T00:18:02.706 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:02.706+0000 7f039b400640 1 --2- 172.21.15.118:0/3591934014 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x560227205020 0x560227205440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:18:02.709 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:02.707+0000 7f039a000640 1 --2- 172.21.15.118:0/3591934014 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x560227206bb0 0x56022720d440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:18:02.709 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:02.707+0000 7f039aa00640 1 -- 172.21.15.118:0/3591934014 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x560227205020 msgr2=0x560227205440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:18:02.709 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:02.707+0000 7f039aa00640 1 --2- 172.21.15.118:0/3591934014 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x560227205020 0x560227205440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:18:02.709 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:02.707+0000 7f039aa00640 1 -- 172.21.15.118:0/3591934014 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x560227206bb0 msgr2=0x56022720d440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:18:02.709 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:02.707+0000 7f039aa00640 1 --2- 172.21.15.118:0/3591934014 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x560227206bb0 0x56022720d440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:18:02.709 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:02.707+0000 7f039aa00640 1 -- 172.21.15.118:0/3591934014 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x56022712e080 con 0x5602272061f0 2024-03-24T00:18:02.710 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:02.708+0000 7f039aa00640 1 --2- 172.21.15.118:0/3591934014 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5602272061f0 0x560227206670 secure :-1 s=READY pgs=323 cs=0 l=1 rev1=1 crypto rx=0x7f038c00bcb0 tx=0x7f038c00f730 comp rx=0 tx=0).ready entity=mon.2 client_cookie=6786b2acda5f9dcc server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:18:02.710 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:02.708+0000 7f0387400640 1 -- 172.21.15.118:0/3591934014 <== mon.2 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f038c015070 con 0x5602272061f0 2024-03-24T00:18:02.710 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:02.708+0000 7f0387400640 1 -- 172.21.15.118:0/3591934014 <== mon.2 v2:172.21.15.189:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f038c01a940 con 0x5602272061f0 2024-03-24T00:18:02.710 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:02.708+0000 7f0387400640 1 -- 172.21.15.118:0/3591934014 <== mon.2 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f038c011980 con 0x5602272061f0 2024-03-24T00:18:02.710 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:02.708+0000 7f0393400640 1 --2- 172.21.15.118:0/4282678437 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7f03800a2900 0x7f03800a4dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:18:02.711 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:02.709+0000 7f0393400640 1 -- 172.21.15.118:0/4282678437 <== mon.1 v2:172.21.15.193:3300/0 5 ==== osd_map(90..90 src has 1..90) v4 ==== 8277+0+0 (secure 0 0 0) 0x7f03880eff00 con 0x7f039c0ffd90 2024-03-24T00:18:02.711 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:02.710+0000 7f039b400640 1 --2- 172.21.15.118:0/4282678437 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7f03800a2900 0x7f03800a4dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:18:02.711 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:02.711+0000 7f03a43ce740 1 -- 172.21.15.118:0/3591934014 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5602272061f0 msgr2=0x560227206670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:18:02.711 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:02.711+0000 7f03a43ce740 1 --2- 172.21.15.118:0/3591934014 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5602272061f0 0x560227206670 secure :-1 s=READY pgs=323 cs=0 l=1 rev1=1 crypto rx=0x7f038c00bcb0 tx=0x7f038c00f730 comp rx=0 tx=0).stop 2024-03-24T00:18:02.711 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:02.712+0000 7f039b400640 1 --2- 172.21.15.118:0/4282678437 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7f03800a2900 0x7f03800a4dc0 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 crypto rx=0x5602272060f0 tx=0x7f03940092c0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:18:02.712 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:02.712+0000 7f03a43ce740 1 -- 172.21.15.118:0/3591934014 shutdown_connections 2024-03-24T00:18:02.712 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:02.712+0000 7f03a43ce740 1 --2- 172.21.15.118:0/3591934014 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x560227206bb0 0x56022720d440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:18:02.712 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:02.712+0000 7f03a43ce740 1 --2- 172.21.15.118:0/3591934014 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5602272061f0 0x560227206670 unknown :-1 s=CLOSED pgs=323 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:18:02.712 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:02.712+0000 7f03a43ce740 1 --2- 172.21.15.118:0/3591934014 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x560227205020 0x560227205440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:18:02.712 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:02.712+0000 7f03a43ce740 1 -- 172.21.15.118:0/3591934014 >> 172.21.15.118:0/3591934014 conn(0x560227200410 msgr2=0x560227200810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:18:02.713 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:02.714+0000 7f03a43ce740 1 -- 172.21.15.118:0/3591934014 shutdown_connections 2024-03-24T00:18:02.713 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:02.714+0000 7f03a43ce740 1 -- 172.21.15.118:0/3591934014 wait complete. 2024-03-24T00:18:02.715 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:02.716+0000 7f03a43ce740 1 -- start start 2024-03-24T00:18:02.716 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:02.716+0000 7f03a43ce740 1 Processor -- start 2024-03-24T00:18:02.718 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:02.717+0000 7f03a43ce740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x560227205020 0x56022720f2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:18:02.718 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:02.717+0000 7f03a43ce740 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5602272061f0 0x56022720f830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:18:02.718 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:02.717+0000 7f03a43ce740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x560227206bb0 0x560227216800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:18:02.718 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:02.717+0000 7f03a43ce740 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x56022720dbc0 con 0x5602272061f0 2024-03-24T00:18:02.718 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:02.717+0000 7f03a43ce740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x56022711d0d0 con 0x560227205020 2024-03-24T00:18:02.718 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:02.717+0000 7f03a43ce740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x56022720dd40 con 0x560227206bb0 2024-03-24T00:18:02.718 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:02.717+0000 7f03a43ce740 10 client.0 _ll_register_callbacks cb 0x560227121e80 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T00:18:02.718 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:02.717+0000 7f039aa00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x560227205020 0x56022720f2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:18:02.718 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:02.717+0000 7f039aa00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x560227205020 0x56022720f2f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.118:38274/0 (socket says 172.21.15.118:38274) 2024-03-24T00:18:02.719 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:02.717+0000 7f039aa00640 1 -- 172.21.15.118:0/1326477370 learned_addr learned my addr 172.21.15.118:0/1326477370 (peer_addr_for_me v2:172.21.15.118:0/0) 2024-03-24T00:18:02.719 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:02.718+0000 7f039b400640 1 --2- 172.21.15.118:0/1326477370 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x560227206bb0 0x560227216800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:18:02.719 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:02.718+0000 7f039aa00640 1 -- 172.21.15.118:0/1326477370 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x560227206bb0 msgr2=0x560227216800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:18:02.719 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:02.718+0000 7f039aa00640 1 --2- 172.21.15.118:0/1326477370 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x560227206bb0 0x560227216800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:18:02.719 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:02.718+0000 7f039aa00640 1 -- 172.21.15.118:0/1326477370 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5602272061f0 msgr2=0x56022720f830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:18:02.719 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:02.718+0000 7f039a000640 1 --2- 172.21.15.118:0/1326477370 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5602272061f0 0x56022720f830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:18:02.719 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:02.718+0000 7f039aa00640 1 --2- 172.21.15.118:0/1326477370 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5602272061f0 0x56022720f830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:18:02.719 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:02.718+0000 7f039aa00640 1 -- 172.21.15.118:0/1326477370 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x560226e753f0 con 0x560227205020 2024-03-24T00:18:02.719 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:02.718+0000 7f039b400640 1 --2- 172.21.15.118:0/1326477370 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x560227206bb0 0x560227216800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T00:18:02.719 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:02.718+0000 7f039a000640 1 --2- 172.21.15.118:0/1326477370 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5602272061f0 0x56022720f830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-24T00:18:02.719 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:02.718+0000 7f039aa00640 1 --2- 172.21.15.118:0/1326477370 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x560227205020 0x56022720f2f0 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 crypto rx=0x7f038c000c00 tx=0x7f038c00bf80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=29a111d974c560 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:18:02.719 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:02.718+0000 7f037be00640 10 client.0 ms_handle_connect on v2:172.21.15.189:3300/0 2024-03-24T00:18:02.720 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:02.719+0000 7f037be00640 1 -- 172.21.15.118:0/1326477370 <== mon.2 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f038c028060 con 0x560227205020 2024-03-24T00:18:02.720 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:02.719+0000 7f037be00640 1 -- 172.21.15.118:0/1326477370 <== mon.2 v2:172.21.15.189:3300/0 2 ==== osd_map(90..90 src has 1..90) v4 ==== 8277+0+0 (secure 0 0 0) 0x7f038c01acf0 con 0x560227205020 2024-03-24T00:18:02.720 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:02.719+0000 7f03a43ce740 1 -- 172.21.15.118:0/1326477370 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f038c01a940 con 0x560227205020 2024-03-24T00:18:02.720 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:02.720+0000 7f037be00640 1 -- 172.21.15.118:0/1326477370 <== mon.2 v2:172.21.15.189:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) v2 ==== 24+0+0 (secure 0 0 0) 0x7f038c015040 con 0x560227205020 2024-03-24T00:18:02.721 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:02.720+0000 7f03a43ce740 10 client.26108 fetch_fsmap learned FSMap version 28 2024-03-24T00:18:02.721 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:02.720+0000 7f03a43ce740 1 -- 172.21.15.118:0/1326477370 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) v3 -- 0x560226f3f7d0 con 0x560227205020 2024-03-24T00:18:02.721 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:02.720+0000 7f037be00640 1 -- 172.21.15.118:0/1326477370 <== mon.2 v2:172.21.15.189:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f038c01a940 con 0x560227205020 2024-03-24T00:18:02.721 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:02.720+0000 7f037be00640 1 -- 172.21.15.118:0/1326477370 <== mon.2 v2:172.21.15.189:3300/0 5 ==== fsmap(e 28) v1 ==== 7934+0+0 (secure 0 0 0) 0x7f038c022030 con 0x560227205020 2024-03-24T00:18:02.721 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:02.720+0000 7f03a43ce740 10 client.26108 fetch_fsmap finished waiting for FSMap version 28 2024-03-24T00:18:02.721 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:02.720+0000 7f03a43ce740 10 client.26108 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-24T00:18:02.721 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:02.720+0000 7f03a43ce740 20 client.26108 populate_metadata read hostname 'smithi118' 2024-03-24T00:18:02.721 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:02.720+0000 7f03a43ce740 1 --2- 172.21.15.118:0/1326477370 >> [v2:172.21.15.189:6838/4074484168,v1:172.21.15.189:6839/4074484168] conn(0x56022721a960 0x56022721cd50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:18:02.721 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:02.720+0000 7f03a43ce740 4 client.26108 mds_command: new command op to 24569 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-24T00:18:02.721 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:02.720+0000 7f03a43ce740 1 -- 172.21.15.118:0/1326477370 --> [v2:172.21.15.189:6838/4074484168,v1:172.21.15.189:6839/4074484168] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x56022716e800 con 0x56022721a960 2024-03-24T00:18:02.721 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:02.721+0000 7f039a000640 1 --2- 172.21.15.118:0/1326477370 >> [v2:172.21.15.189:6838/4074484168,v1:172.21.15.189:6839/4074484168] conn(0x56022721a960 0x56022721cd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:18:02.722 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:02.721+0000 7f039a000640 1 --2- 172.21.15.118:0/1326477370 >> [v2:172.21.15.189:6838/4074484168,v1:172.21.15.189:6839/4074484168] conn(0x56022721a960 0x56022721cd50 crc :-1 s=READY pgs=83 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=bb29b1dcf0144e0e server_cookie=f608b7641460fd69 in_seq=0 out_seq=0 2024-03-24T00:18:02.722 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:02.721+0000 7f037be00640 10 client.26108 ms_handle_connect on v2:172.21.15.189:6838/4074484168 2024-03-24T00:18:02.724 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:02.725+0000 7f037be00640 1 -- 172.21.15.118:0/1326477370 <== mds.0 v2:172.21.15.189:6838/4074484168 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x56022716e800 con 0x56022721a960 2024-03-24T00:18:02.724 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:02.725+0000 7f037be00640 10 client.26108 handle_command_reply: tid=0 2024-03-24T00:18:02.724 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:02.725+0000 7f03a43ce740 1 client.26108 shutdown 2024-03-24T00:18:02.724 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:02.725+0000 7f03a43ce740 1 -- 172.21.15.118:0/1326477370 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x560227205020 msgr2=0x56022720f2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:18:02.725 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:02.725+0000 7f03a43ce740 1 --2- 172.21.15.118:0/1326477370 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x560227205020 0x56022720f2f0 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 crypto rx=0x7f038c000c00 tx=0x7f038c00bf80 comp rx=0 tx=0).stop 2024-03-24T00:18:02.725 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:02.726+0000 7f03a43ce740 1 -- 172.21.15.118:0/1326477370 shutdown_connections 2024-03-24T00:18:02.725 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:02.726+0000 7f03a43ce740 1 --2- 172.21.15.118:0/1326477370 >> [v2:172.21.15.189:6838/4074484168,v1:172.21.15.189:6839/4074484168] conn(0x56022721a960 0x56022721cd50 crc :-1 s=READY pgs=83 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:18:02.725 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:02.726+0000 7f03a43ce740 1 --2- 172.21.15.118:0/1326477370 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x560227206bb0 0x560227216800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:18:02.726 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:02.726+0000 7f03a43ce740 1 --2- 172.21.15.118:0/1326477370 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5602272061f0 0x56022720f830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:18:02.726 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:02.726+0000 7f037be00640 0 client.26108 ms_handle_reset on v2:172.21.15.189:6838/4074484168 2024-03-24T00:18:02.726 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:02.726+0000 7f03a43ce740 1 --2- 172.21.15.118:0/1326477370 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x560227205020 0x56022720f2f0 unknown :-1 s=CLOSED pgs=324 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:18:02.726 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:02.726+0000 7f03a43ce740 1 -- 172.21.15.118:0/1326477370 >> 172.21.15.118:0/1326477370 conn(0x560227200410 msgr2=0x5602271f6eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:18:02.726 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:02.726+0000 7f03a43ce740 1 -- 172.21.15.118:0/1326477370 shutdown_connections 2024-03-24T00:18:02.726 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:02.726+0000 7f03a43ce740 1 -- 172.21.15.118:0/1326477370 wait complete. 2024-03-24T00:18:02.726 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:02.726+0000 7f03a43ce740 20 client.26108 trim_cache size 0 max 16384 2024-03-24T00:18:02.740 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:02.741+0000 7f03a43ce740 1 Processor -- start 2024-03-24T00:18:02.740 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:02.741+0000 7f03a43ce740 1 -- start start 2024-03-24T00:18:02.741 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:02.741+0000 7f03a43ce740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x560227205020 0x5602271f6d30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:18:02.742 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:02.741+0000 7f03a43ce740 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5602272061f0 0x5602272215a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:18:02.742 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:02.741+0000 7f03a43ce740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x560227206bb0 0x560227221ae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:18:02.742 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:02.741+0000 7f03a43ce740 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f039c1af0d0 con 0x5602272061f0 2024-03-24T00:18:02.742 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:02.741+0000 7f03a43ce740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f039c1b0a50 con 0x560227205020 2024-03-24T00:18:02.742 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:02.741+0000 7f03a43ce740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f039c1af730 con 0x560227206bb0 2024-03-24T00:18:02.742 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:02.741+0000 7f039aa00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x560227205020 0x5602271f6d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:18:02.742 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:02.741+0000 7f039b400640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x560227206bb0 0x560227221ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:18:02.742 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:02.741+0000 7f039aa00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x560227205020 0x5602271f6d30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.118:38290/0 (socket says 172.21.15.118:38290) 2024-03-24T00:18:02.742 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:02.741+0000 7f039a000640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5602272061f0 0x5602272215a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:18:02.742 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:02.742+0000 7f039aa00640 1 -- 172.21.15.118:0/1643348326 learned_addr learned my addr 172.21.15.118:0/1643348326 (peer_addr_for_me v2:172.21.15.118:0/0) 2024-03-24T00:18:02.742 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:02.742+0000 7f039aa00640 1 -- 172.21.15.118:0/1643348326 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x560227206bb0 msgr2=0x560227221ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:18:02.742 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:02.742+0000 7f039aa00640 1 --2- 172.21.15.118:0/1643348326 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x560227206bb0 0x560227221ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:18:02.742 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:02.742+0000 7f039aa00640 1 -- 172.21.15.118:0/1643348326 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5602272061f0 msgr2=0x5602272215a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:18:02.742 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:02.742+0000 7f039aa00640 1 --2- 172.21.15.118:0/1643348326 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5602272061f0 0x5602272215a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:18:02.743 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:02.742+0000 7f039aa00640 1 -- 172.21.15.118:0/1643348326 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x56022721d380 con 0x560227205020 2024-03-24T00:18:02.743 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:02.742+0000 7f039a000640 1 --2- 172.21.15.118:0/1643348326 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5602272061f0 0x5602272215a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T00:18:02.743 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:02.742+0000 7f039b400640 1 --2- 172.21.15.118:0/1643348326 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x560227206bb0 0x560227221ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T00:18:02.743 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:02.742+0000 7f039aa00640 1 --2- 172.21.15.118:0/1643348326 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x560227205020 0x5602271f6d30 secure :-1 s=READY pgs=325 cs=0 l=1 rev1=1 crypto rx=0x7f038c011bd0 tx=0x7f038c005fd0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=12db4913a9ca7c6 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:18:02.745 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:02.742+0000 7f0386a00640 1 -- 172.21.15.118:0/1643348326 <== mon.2 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f038c028050 con 0x560227205020 2024-03-24T00:18:02.745 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:02.743+0000 7f0386a00640 1 -- 172.21.15.118:0/1643348326 <== mon.2 v2:172.21.15.189:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f038c00b330 con 0x560227205020 2024-03-24T00:18:02.745 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:02.743+0000 7f0386a00640 1 -- 172.21.15.118:0/1643348326 <== mon.2 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f038c032080 con 0x560227205020 2024-03-24T00:18:02.745 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:02.743+0000 7f03a43ce740 1 -- 172.21.15.118:0/1643348326 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x560227205020 msgr2=0x5602271f6d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:18:02.746 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:02.743+0000 7f03a43ce740 1 --2- 172.21.15.118:0/1643348326 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x560227205020 0x5602271f6d30 secure :-1 s=READY pgs=325 cs=0 l=1 rev1=1 crypto rx=0x7f038c011bd0 tx=0x7f038c005fd0 comp rx=0 tx=0).stop 2024-03-24T00:18:02.746 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:02.743+0000 7f03a43ce740 1 -- 172.21.15.118:0/1643348326 shutdown_connections 2024-03-24T00:18:02.746 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:02.743+0000 7f03a43ce740 1 --2- 172.21.15.118:0/1643348326 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x560227206bb0 0x560227221ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:18:02.746 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:02.743+0000 7f03a43ce740 1 --2- 172.21.15.118:0/1643348326 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5602272061f0 0x5602272215a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:18:02.746 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:02.743+0000 7f03a43ce740 1 --2- 172.21.15.118:0/1643348326 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x560227205020 0x5602271f6d30 unknown :-1 s=CLOSED pgs=325 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:18:02.746 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:02.743+0000 7f03a43ce740 1 -- 172.21.15.118:0/1643348326 >> 172.21.15.118:0/1643348326 conn(0x560227200410 msgr2=0x5602271f6620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:18:02.746 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:02.743+0000 7f03a43ce740 1 -- 172.21.15.118:0/1643348326 shutdown_connections 2024-03-24T00:18:02.746 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:02.743+0000 7f03a43ce740 1 -- 172.21.15.118:0/1643348326 wait complete. 2024-03-24T00:18:02.746 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:02.744+0000 7f03a43ce740 1 -- start start 2024-03-24T00:18:02.746 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:02.744+0000 7f03a43ce740 1 Processor -- start 2024-03-24T00:18:02.746 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:02.744+0000 7f03a43ce740 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x560227205020 0x560227211d20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:18:02.746 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:02.744+0000 7f03a43ce740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5602272061f0 0x560227212260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:18:02.746 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:02.744+0000 7f03a43ce740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x560227206bb0 0x560227215280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:18:02.747 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:02.744+0000 7f03a43ce740 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f039c1b03f0 con 0x560227205020 2024-03-24T00:18:02.747 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:02.744+0000 7f03a43ce740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f039c1afd90 con 0x560227206bb0 2024-03-24T00:18:02.747 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:02.744+0000 7f03a43ce740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f039c1ad2b0 con 0x5602272061f0 2024-03-24T00:18:02.747 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:02.744+0000 7f03a43ce740 10 client.0 _ll_register_callbacks cb 0x7f038c00bb00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T00:18:02.747 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:02.745+0000 7f039b400640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x560227206bb0 0x560227215280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:18:02.747 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:02.745+0000 7f039aa00640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x560227205020 0x560227211d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:18:02.747 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:02.745+0000 7f039b400640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x560227206bb0 0x560227215280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.118:38296/0 (socket says 172.21.15.118:38296) 2024-03-24T00:18:02.747 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:02.745+0000 7f039b400640 1 -- 172.21.15.118:0/1123196899 learned_addr learned my addr 172.21.15.118:0/1123196899 (peer_addr_for_me v2:172.21.15.118:0/0) 2024-03-24T00:18:02.747 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:02.745+0000 7f039a000640 1 --2- 172.21.15.118:0/1123196899 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5602272061f0 0x560227212260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:18:02.747 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:02.745+0000 7f039b400640 1 -- 172.21.15.118:0/1123196899 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5602272061f0 msgr2=0x560227212260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:18:02.747 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:02.745+0000 7f039b400640 1 --2- 172.21.15.118:0/1123196899 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5602272061f0 0x560227212260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:18:02.747 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:02.745+0000 7f039b400640 1 -- 172.21.15.118:0/1123196899 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x560227205020 msgr2=0x560227211d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:18:02.747 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:02.745+0000 7f039b400640 1 --2- 172.21.15.118:0/1123196899 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x560227205020 0x560227211d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:18:02.747 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:02.745+0000 7f039b400640 1 -- 172.21.15.118:0/1123196899 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x560227215870 con 0x560227206bb0 2024-03-24T00:18:02.747 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:02.745+0000 7f039a000640 1 --2- 172.21.15.118:0/1123196899 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5602272061f0 0x560227212260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T00:18:02.748 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:02.745+0000 7f039b400640 1 --2- 172.21.15.118:0/1123196899 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x560227206bb0 0x560227215280 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 crypto rx=0x7f039401cc50 tx=0x7f03940095c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=201c572648ca4738 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:18:02.748 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:02.745+0000 7f037be00640 10 client.0 ms_handle_connect on v2:172.21.15.189:3300/0 2024-03-24T00:18:02.748 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:02.746+0000 7f037be00640 1 -- 172.21.15.118:0/1123196899 <== mon.2 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f039401cc80 con 0x560227206bb0 2024-03-24T00:18:02.748 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:02.746+0000 7f03a43ce740 1 -- 172.21.15.118:0/1123196899 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x560227168b50 con 0x560227206bb0 2024-03-24T00:18:02.748 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:02.746+0000 7f037be00640 1 -- 172.21.15.118:0/1123196899 <== mon.2 v2:172.21.15.189:3300/0 2 ==== osd_map(90..90 src has 1..90) v4 ==== 8277+0+0 (secure 0 0 0) 0x7f03940273f0 con 0x560227206bb0 2024-03-24T00:18:02.748 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:02.746+0000 7f037be00640 1 -- 172.21.15.118:0/1123196899 <== mon.2 v2:172.21.15.189:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) v2 ==== 24+0+0 (secure 0 0 0) 0x7f039401ce20 con 0x560227206bb0 2024-03-24T00:18:02.748 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:02.746+0000 7f03a43ce740 10 client.26114 fetch_fsmap learned FSMap version 28 2024-03-24T00:18:02.748 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:02.746+0000 7f03a43ce740 1 -- 172.21.15.118:0/1123196899 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) v3 -- 0x560227276880 con 0x560227206bb0 2024-03-24T00:18:02.750 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:02.746+0000 7f037be00640 1 -- 172.21.15.118:0/1123196899 <== mon.2 v2:172.21.15.189:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x560227168b50 con 0x560227206bb0 2024-03-24T00:18:02.750 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:02.746+0000 7f037be00640 1 -- 172.21.15.118:0/1123196899 <== mon.2 v2:172.21.15.189:3300/0 5 ==== fsmap(e 28) v1 ==== 7934+0+0 (secure 0 0 0) 0x7f03940308e0 con 0x560227206bb0 2024-03-24T00:18:02.750 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:02.747+0000 7f03a43ce740 10 client.26114 fetch_fsmap finished waiting for FSMap version 28 2024-03-24T00:18:02.750 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:02.747+0000 7f03a43ce740 10 client.26114 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-24T00:18:02.750 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:02.747+0000 7f03a43ce740 20 client.26114 populate_metadata read hostname 'smithi118' 2024-03-24T00:18:02.750 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:02.747+0000 7f03a43ce740 1 --2- 172.21.15.118:0/1123196899 >> [v2:172.21.15.189:6838/4074484168,v1:172.21.15.189:6839/4074484168] conn(0x560227277200 0x560227279620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:18:02.750 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:02.747+0000 7f03a43ce740 4 client.26114 mds_command: new command op to 24569 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-24T00:18:02.750 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:02.747+0000 7f03a43ce740 1 -- 172.21.15.118:0/1123196899 --> [v2:172.21.15.189:6838/4074484168,v1:172.21.15.189:6839/4074484168] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x5602270e6500 con 0x560227277200 2024-03-24T00:18:02.750 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:02.747+0000 7f039aa00640 1 --2- 172.21.15.118:0/1123196899 >> [v2:172.21.15.189:6838/4074484168,v1:172.21.15.189:6839/4074484168] conn(0x560227277200 0x560227279620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:18:02.750 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:02.747+0000 7f039aa00640 1 --2- 172.21.15.118:0/1123196899 >> [v2:172.21.15.189:6838/4074484168,v1:172.21.15.189:6839/4074484168] conn(0x560227277200 0x560227279620 crc :-1 s=READY pgs=84 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=2ea6970a3b727a0a server_cookie=8ef070ac5b2b3587 in_seq=0 out_seq=0 2024-03-24T00:18:02.750 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:02.747+0000 7f037be00640 10 client.26114 ms_handle_connect on v2:172.21.15.189:6838/4074484168 2024-03-24T00:18:02.750 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:02.749+0000 7f037be00640 1 -- 172.21.15.118:0/1123196899 <== mds.0 v2:172.21.15.189:6838/4074484168 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x5602270e6500 con 0x560227277200 2024-03-24T00:18:02.751 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:02.749+0000 7f037be00640 10 client.26114 handle_command_reply: tid=0 2024-03-24T00:18:02.751 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:02.749+0000 7f03a43ce740 1 client.26114 shutdown 2024-03-24T00:18:02.751 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:02.749+0000 7f03a43ce740 1 -- 172.21.15.118:0/1123196899 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x560227206bb0 msgr2=0x560227215280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:18:02.751 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:02.749+0000 7f03a43ce740 1 --2- 172.21.15.118:0/1123196899 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x560227206bb0 0x560227215280 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 crypto rx=0x7f039401cc50 tx=0x7f03940095c0 comp rx=0 tx=0).stop 2024-03-24T00:18:02.751 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:02.749+0000 7f03a43ce740 1 -- 172.21.15.118:0/1123196899 shutdown_connections 2024-03-24T00:18:02.751 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:02.749+0000 7f03a43ce740 1 --2- 172.21.15.118:0/1123196899 >> [v2:172.21.15.189:6838/4074484168,v1:172.21.15.189:6839/4074484168] conn(0x560227277200 0x560227279620 crc :-1 s=READY pgs=84 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:18:02.751 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:02.749+0000 7f03a43ce740 1 --2- 172.21.15.118:0/1123196899 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x560227206bb0 0x560227215280 unknown :-1 s=CLOSED pgs=326 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:18:02.751 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:02.749+0000 7f03a43ce740 1 --2- 172.21.15.118:0/1123196899 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5602272061f0 0x560227212260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:18:02.751 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:02.749+0000 7f03a43ce740 1 --2- 172.21.15.118:0/1123196899 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x560227205020 0x560227211d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:18:02.751 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:02.749+0000 7f03a43ce740 1 -- 172.21.15.118:0/1123196899 >> 172.21.15.118:0/1123196899 conn(0x560227200410 msgr2=0x5602271f6eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:18:02.751 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:02.749+0000 7f037be00640 0 client.26114 ms_handle_reset on v2:172.21.15.189:6838/4074484168 2024-03-24T00:18:02.751 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:02.750+0000 7f03a43ce740 1 -- 172.21.15.118:0/1123196899 shutdown_connections 2024-03-24T00:18:02.751 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:02.750+0000 7f03a43ce740 1 -- 172.21.15.118:0/1123196899 wait complete. 2024-03-24T00:18:02.751 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:02.750+0000 7f03a43ce740 20 client.26114 trim_cache size 0 max 16384 2024-03-24T00:18:02.752 INFO:teuthology.orchestra.run.smithi118.stdout:[] 2024-03-24T00:18:02.752 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:02.750+0000 7f03a1600640 1 -- 172.21.15.118:0/4282678437 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7f03800a2900 msgr2=0x7f03800a4dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:18:02.752 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:02.750+0000 7f03a1600640 1 --2- 172.21.15.118:0/4282678437 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7f03800a2900 0x7f03800a4dc0 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 crypto rx=0x5602272060f0 tx=0x7f03940092c0 comp rx=0 tx=0).stop 2024-03-24T00:18:02.752 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:02.750+0000 7f03a1600640 1 -- 172.21.15.118:0/4282678437 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f039c0ffd90 msgr2=0x7f039c070610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:18:02.753 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:02.750+0000 7f03a1600640 1 --2- 172.21.15.118:0/4282678437 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f039c0ffd90 0x7f039c070610 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 crypto rx=0x7f038802c0d0 tx=0x7f03880042f0 comp rx=0 tx=0).stop 2024-03-24T00:18:02.753 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:02.751+0000 7f03a1600640 1 -- 172.21.15.118:0/4282678437 shutdown_connections 2024-03-24T00:18:02.753 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:02.751+0000 7f03a1600640 1 --2- 172.21.15.118:0/4282678437 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7f03800a2900 0x7f03800a4dc0 unknown :-1 s=CLOSED pgs=314 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:18:02.753 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:02.751+0000 7f03a1600640 1 --2- 172.21.15.118:0/4282678437 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f039c107320 0x7f039c1a9c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:18:02.753 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:02.751+0000 7f03a1600640 1 --2- 172.21.15.118:0/4282678437 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f039c0ffd90 0x7f039c070610 unknown :-1 s=CLOSED pgs=315 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:18:02.753 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:02.751+0000 7f03a1600640 1 --2- 172.21.15.118:0/4282678437 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f039c0ff430 0x7f039c0700d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:18:02.753 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:02.751+0000 7f03a1600640 1 -- 172.21.15.118:0/4282678437 >> 172.21.15.118:0/4282678437 conn(0x7f039c0fae60 msgr2=0x7f039c102cd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:18:02.753 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:02.751+0000 7f03a1600640 1 -- 172.21.15.118:0/4282678437 shutdown_connections 2024-03-24T00:18:02.753 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:02.751+0000 7f03a1600640 1 -- 172.21.15.118:0/4282678437 wait complete. 2024-03-24T00:18:03.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:18:02 smithi118 ceph-mon[31535]: pgmap v468: 97 pgs: 97 active+clean; 17 MiB data, 413 MiB used, 1.0 TiB / 1.0 TiB avail; 316 KiB/s rd, 361 KiB/s wr, 106 op/s 2024-03-24T00:18:03.203 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:18:02 smithi189 ceph-mon[36187]: pgmap v468: 97 pgs: 97 active+clean; 17 MiB data, 413 MiB used, 1.0 TiB / 1.0 TiB avail; 316 KiB/s rd, 361 KiB/s wr, 106 op/s 2024-03-24T00:18:03.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:18:02 smithi193 ceph-mon[36878]: pgmap v468: 97 pgs: 97 active+clean; 17 MiB data, 413 MiB used, 1.0 TiB / 1.0 TiB avail; 316 KiB/s rd, 361 KiB/s wr, 106 op/s 2024-03-24T00:18:03.767 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-03-24T00:18:03.767 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T00:18:03.768 DEBUG:teuthology.orchestra.run.smithi118:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2024-03-24T00:18:03.993 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:03.993+0000 7fa1e5c00640 1 -- 172.21.15.118:0/3181413591 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fa1e00fff10 msgr2=0x7fa1e0102340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:18:03.993 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:03.993+0000 7fa1e5c00640 1 --2- 172.21.15.118:0/3181413591 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fa1e00fff10 0x7fa1e0102340 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 crypto rx=0x7fa1d0009900 tx=0x7fa1d002bc40 comp rx=0 tx=0).stop 2024-03-24T00:18:03.993 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:03.993+0000 7fa1e5c00640 1 -- 172.21.15.118:0/3181413591 shutdown_connections 2024-03-24T00:18:03.993 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:03.993+0000 7fa1e5c00640 1 --2- 172.21.15.118:0/3181413591 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa1e0105230 0x7fa1e01056b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:18:03.994 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:03.993+0000 7fa1e5c00640 1 --2- 172.21.15.118:0/3181413591 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fa1e0102880 0x7fa1e0104cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:18:03.994 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:03.993+0000 7fa1e5c00640 1 --2- 172.21.15.118:0/3181413591 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fa1e00fff10 0x7fa1e0102340 unknown :-1 s=CLOSED pgs=316 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:18:03.994 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:03.993+0000 7fa1e5c00640 1 -- 172.21.15.118:0/3181413591 >> 172.21.15.118:0/3181413591 conn(0x7fa1e00fbb10 msgr2=0x7fa1e00fdf50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:18:03.994 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:03.993+0000 7fa1e5c00640 1 -- 172.21.15.118:0/3181413591 shutdown_connections 2024-03-24T00:18:03.994 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:03.993+0000 7fa1e5c00640 1 -- 172.21.15.118:0/3181413591 wait complete. 2024-03-24T00:18:03.994 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:03.995+0000 7fa1e5c00640 1 Processor -- start 2024-03-24T00:18:03.994 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:03.995+0000 7fa1e5c00640 1 -- start start 2024-03-24T00:18:03.996 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:03.995+0000 7fa1e5c00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fa1e00fff10 0x7fa1e019dfc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:18:03.996 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:03.995+0000 7fa1e5c00640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fa1e0102880 0x7fa1e019e500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:18:03.996 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:03.995+0000 7fa1e5c00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa1e0105230 0x7fa1e01a5570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:18:03.996 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:03.995+0000 7fa1e5c00640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa1e0113940 con 0x7fa1e0102880 2024-03-24T00:18:03.996 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:03.995+0000 7fa1e5c00640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa1e01137c0 con 0x7fa1e0105230 2024-03-24T00:18:03.996 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:03.995+0000 7fa1e5c00640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa1e0113ac0 con 0x7fa1e00fff10 2024-03-24T00:18:03.996 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:03.996+0000 7fa1dea00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fa1e00fff10 0x7fa1e019dfc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:18:03.996 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:03.996+0000 7fa1dea00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fa1e00fff10 0x7fa1e019dfc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.118:44418/0 (socket says 172.21.15.118:44418) 2024-03-24T00:18:03.996 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:03.996+0000 7fa1dea00640 1 -- 172.21.15.118:0/2424794299 learned_addr learned my addr 172.21.15.118:0/2424794299 (peer_addr_for_me v2:172.21.15.118:0/0) 2024-03-24T00:18:03.996 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:03.996+0000 7fa1df400640 1 --2- 172.21.15.118:0/2424794299 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa1e0105230 0x7fa1e01a5570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:18:03.996 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:03.996+0000 7fa1d6000640 1 --2- 172.21.15.118:0/2424794299 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fa1e0102880 0x7fa1e019e500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:18:03.996 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:03.996+0000 7fa1dea00640 1 -- 172.21.15.118:0/2424794299 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa1e0105230 msgr2=0x7fa1e01a5570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:18:03.996 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:03.996+0000 7fa1dea00640 1 --2- 172.21.15.118:0/2424794299 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa1e0105230 0x7fa1e01a5570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:18:03.996 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:03.996+0000 7fa1dea00640 1 -- 172.21.15.118:0/2424794299 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fa1e0102880 msgr2=0x7fa1e019e500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:18:03.997 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:03.996+0000 7fa1dea00640 1 --2- 172.21.15.118:0/2424794299 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fa1e0102880 0x7fa1e019e500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:18:03.997 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:03.996+0000 7fa1dea00640 1 -- 172.21.15.118:0/2424794299 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa1e01a5be0 con 0x7fa1e00fff10 2024-03-24T00:18:03.997 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:03.996+0000 7fa1dea00640 1 --2- 172.21.15.118:0/2424794299 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fa1e00fff10 0x7fa1e019dfc0 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 crypto rx=0x7fa1d002c8f0 tx=0x7fa1d002c660 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:18:03.997 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:03.997+0000 7fa1d7e00640 1 -- 172.21.15.118:0/2424794299 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa1d0041070 con 0x7fa1e00fff10 2024-03-24T00:18:03.997 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:03.997+0000 7fa1d7e00640 1 -- 172.21.15.118:0/2424794299 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fa1d0004450 con 0x7fa1e00fff10 2024-03-24T00:18:03.998 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:03.997+0000 7fa1e5c00640 1 -- 172.21.15.118:0/2424794299 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa1e01a5e70 con 0x7fa1e00fff10 2024-03-24T00:18:03.998 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:03.997+0000 7fa1d7e00640 1 -- 172.21.15.118:0/2424794299 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa1d0003df0 con 0x7fa1e00fff10 2024-03-24T00:18:03.998 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:03.997+0000 7fa1e5c00640 1 -- 172.21.15.118:0/2424794299 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa1e01a62d0 con 0x7fa1e00fff10 2024-03-24T00:18:03.999 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:04.000+0000 7fa1d7e00640 1 -- 172.21.15.118:0/2424794299 <== mon.1 v2:172.21.15.193:3300/0 4 ==== mgrmap(e 20) v1 ==== 137291+0+0 (secure 0 0 0) 0x7fa1d0044070 con 0x7fa1e00fff10 2024-03-24T00:18:04.001 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:04.001+0000 7fa1d7e00640 1 --2- 172.21.15.118:0/2424794299 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7fa1c40a2880 0x7fa1c40a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:18:04.001 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:04.001+0000 7fa1e8841740 1 Processor -- start 2024-03-24T00:18:04.001 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:04.001+0000 7fa1e8841740 1 -- start start 2024-03-24T00:18:04.001 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:04.002+0000 7fa1d7e00640 1 -- 172.21.15.118:0/2424794299 <== mon.1 v2:172.21.15.193:3300/0 5 ==== osd_map(90..90 src has 1..90) v4 ==== 8277+0+0 (secure 0 0 0) 0x7fa1d00f0620 con 0x7fa1e00fff10 2024-03-24T00:18:04.002 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:04.002+0000 7fa1e8841740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5607e8d5c0a0 0x5607e8d5c4c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:18:04.002 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:04.002+0000 7fa1d6000640 1 --2- 172.21.15.118:0/2424794299 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7fa1c40a2880 0x7fa1c40a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:18:04.002 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:04.002+0000 7fa1e8841740 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5607e8d5ca90 0x5607e8d61590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:18:04.002 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:04.002+0000 7fa1e8841740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5607e8d61c00 0x5607e8d64090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:18:04.002 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:04.002+0000 7fa1e8841740 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x5607e8cc1eb0 con 0x5607e8d5ca90 2024-03-24T00:18:04.002 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:04.002+0000 7fa1e8841740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x5607e8c6aa60 con 0x5607e8d61c00 2024-03-24T00:18:04.003 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:04.002+0000 7fa1e8841740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x5607e8cbdf50 con 0x5607e8d5c0a0 2024-03-24T00:18:04.003 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:04.002+0000 7fa1dea00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5607e8d5c0a0 0x5607e8d5c4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:18:04.003 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:04.002+0000 7fa1dea00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5607e8d5c0a0 0x5607e8d5c4c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.118:44426/0 (socket says 172.21.15.118:44426) 2024-03-24T00:18:04.003 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:04.002+0000 7fa1dea00640 1 -- 172.21.15.118:0/3799579396 learned_addr learned my addr 172.21.15.118:0/3799579396 (peer_addr_for_me v2:172.21.15.118:0/0) 2024-03-24T00:18:04.003 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:04.002+0000 7fa1d6000640 1 --2- 172.21.15.118:0/3799579396 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5607e8d5ca90 0x5607e8d61590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:18:04.003 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:04.002+0000 7fa1df400640 1 --2- 172.21.15.118:0/3799579396 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5607e8d61c00 0x5607e8d64090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:18:04.003 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:04.002+0000 7fa1dea00640 1 -- 172.21.15.118:0/3799579396 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5607e8d61c00 msgr2=0x5607e8d64090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:18:04.003 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:04.002+0000 7fa1dea00640 1 --2- 172.21.15.118:0/3799579396 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5607e8d61c00 0x5607e8d64090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:18:04.003 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:04.002+0000 7fa1dea00640 1 -- 172.21.15.118:0/3799579396 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5607e8d5ca90 msgr2=0x5607e8d61590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:18:04.003 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:04.002+0000 7fa1dea00640 1 --2- 172.21.15.118:0/3799579396 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5607e8d5ca90 0x5607e8d61590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:18:04.003 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:04.002+0000 7fa1dea00640 1 -- 172.21.15.118:0/3799579396 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5607e8a9e7e0 con 0x5607e8d5c0a0 2024-03-24T00:18:04.003 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:04.003+0000 7fa1df400640 1 --2- 172.21.15.118:0/3799579396 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5607e8d61c00 0x5607e8d64090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T00:18:04.003 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:04.003+0000 7fa1dea00640 1 --2- 172.21.15.118:0/3799579396 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5607e8d5c0a0 0x5607e8d5c4c0 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 crypto rx=0x7fa1d00bcc30 tx=0x7fa1d00038b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c1a2e1b6529b484 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:18:04.007 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:04.003+0000 7fa1cfe00640 1 -- 172.21.15.118:0/3799579396 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa1d004b040 con 0x5607e8d5c0a0 2024-03-24T00:18:04.007 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:04.003+0000 7fa1cfe00640 1 -- 172.21.15.118:0/3799579396 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fa1d00bcf30 con 0x5607e8d5c0a0 2024-03-24T00:18:04.007 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:04.003+0000 7fa1cfe00640 1 -- 172.21.15.118:0/3799579396 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa1d00b95a0 con 0x5607e8d5c0a0 2024-03-24T00:18:04.007 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:04.004+0000 7fa1e8841740 1 -- 172.21.15.118:0/3799579396 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5607e8d5c0a0 msgr2=0x5607e8d5c4c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:18:04.007 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:04.004+0000 7fa1e8841740 1 --2- 172.21.15.118:0/3799579396 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5607e8d5c0a0 0x5607e8d5c4c0 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 crypto rx=0x7fa1d00bcc30 tx=0x7fa1d00038b0 comp rx=0 tx=0).stop 2024-03-24T00:18:04.007 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:04.004+0000 7fa1d6000640 1 --2- 172.21.15.118:0/2424794299 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7fa1c40a2880 0x7fa1c40a4d40 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 crypto rx=0x5607e8d5ce90 tx=0x7fa1c8009570 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:18:04.007 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:04.004+0000 7fa1e8841740 1 -- 172.21.15.118:0/3799579396 shutdown_connections 2024-03-24T00:18:04.007 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:04.004+0000 7fa1e8841740 1 --2- 172.21.15.118:0/3799579396 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5607e8d61c00 0x5607e8d64090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:18:04.007 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:04.004+0000 7fa1e8841740 1 --2- 172.21.15.118:0/3799579396 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5607e8d5ca90 0x5607e8d61590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:18:04.008 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:04.004+0000 7fa1e8841740 1 --2- 172.21.15.118:0/3799579396 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5607e8d5c0a0 0x5607e8d5c4c0 unknown :-1 s=CLOSED pgs=318 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:18:04.008 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:04.004+0000 7fa1e8841740 1 -- 172.21.15.118:0/3799579396 >> 172.21.15.118:0/3799579396 conn(0x5607e8d57610 msgr2=0x5607e8d57a10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:18:04.008 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:04.004+0000 7fa1e8841740 1 -- 172.21.15.118:0/3799579396 shutdown_connections 2024-03-24T00:18:04.008 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:04.004+0000 7fa1e8841740 1 -- 172.21.15.118:0/3799579396 wait complete. 2024-03-24T00:18:04.008 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:04.004+0000 7fa1e8841740 1 -- start start 2024-03-24T00:18:04.008 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:04.005+0000 7fa1e8841740 1 Processor -- start 2024-03-24T00:18:04.008 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:04.005+0000 7fa1e8841740 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5607e8d5c0a0 0x5607e8d65f80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:18:04.008 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:04.005+0000 7fa1e8841740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5607e8d5ca90 0x5607e8d664c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:18:04.008 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:04.005+0000 7fa1e8841740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5607e8d61c00 0x5607e8d6d500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:18:04.008 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:04.005+0000 7fa1e8841740 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x5607e8d64970 con 0x5607e8d5c0a0 2024-03-24T00:18:04.008 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:04.005+0000 7fa1e8841740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x5607e8c7deb0 con 0x5607e8d5ca90 2024-03-24T00:18:04.008 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:04.005+0000 7fa1e8841740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x5607e8d64af0 con 0x5607e8d61c00 2024-03-24T00:18:04.008 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:04.005+0000 7fa1e8841740 10 client.0 _ll_register_callbacks cb 0x5607e8c72150 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T00:18:04.008 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:04.005+0000 7fa1dea00640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5607e8d5c0a0 0x5607e8d65f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:18:04.008 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:04.005+0000 7fa1d6000640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5607e8d5ca90 0x5607e8d664c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:18:04.009 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:04.005+0000 7fa1d6000640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5607e8d5ca90 0x5607e8d664c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.118:33582/0 (socket says 172.21.15.118:33582) 2024-03-24T00:18:04.009 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:04.005+0000 7fa1dea00640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5607e8d5c0a0 0x5607e8d65f80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.118:3300/0 says I am v2:172.21.15.118:55470/0 (socket says 172.21.15.118:55470) 2024-03-24T00:18:04.009 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:04.005+0000 7fa1d6000640 1 -- 172.21.15.118:0/308400066 learned_addr learned my addr 172.21.15.118:0/308400066 (peer_addr_for_me v2:172.21.15.118:0/0) 2024-03-24T00:18:04.009 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:04.005+0000 7fa1df400640 1 --2- 172.21.15.118:0/308400066 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5607e8d61c00 0x5607e8d6d500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:18:04.009 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:04.006+0000 7fa1d6000640 1 -- 172.21.15.118:0/308400066 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5607e8d61c00 msgr2=0x5607e8d6d500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:18:04.009 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:04.006+0000 7fa1d6000640 1 --2- 172.21.15.118:0/308400066 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5607e8d61c00 0x5607e8d6d500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:18:04.009 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:04.006+0000 7fa1d6000640 1 -- 172.21.15.118:0/308400066 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5607e8d5c0a0 msgr2=0x5607e8d65f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:18:04.009 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:04.006+0000 7fa1d6000640 1 --2- 172.21.15.118:0/308400066 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5607e8d5c0a0 0x5607e8d65f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:18:04.009 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:04.006+0000 7fa1d6000640 1 -- 172.21.15.118:0/308400066 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5607e8afef70 con 0x5607e8d5ca90 2024-03-24T00:18:04.009 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:04.006+0000 7fa1dea00640 1 --2- 172.21.15.118:0/308400066 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5607e8d5c0a0 0x5607e8d65f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T00:18:04.009 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:04.006+0000 7fa1d6000640 1 --2- 172.21.15.118:0/308400066 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5607e8d5ca90 0x5607e8d664c0 secure :-1 s=READY pgs=327 cs=0 l=1 rev1=1 crypto rx=0x7fa1c8011f50 tx=0x7fa1c8005f90 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d0f3bc359fc72ec9 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:18:04.009 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:04.006+0000 7fa1ccc00640 10 client.0 ms_handle_connect on v2:172.21.15.189:3300/0 2024-03-24T00:18:04.009 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:04.006+0000 7fa1ccc00640 1 -- 172.21.15.118:0/308400066 <== mon.2 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa1c801fcd0 con 0x5607e8d5ca90 2024-03-24T00:18:04.009 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:04.006+0000 7fa1e8841740 1 -- 172.21.15.118:0/308400066 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fa1d00bcf30 con 0x5607e8d5ca90 2024-03-24T00:18:04.009 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:04.007+0000 7fa1ccc00640 1 -- 172.21.15.118:0/308400066 <== mon.2 v2:172.21.15.189:3300/0 2 ==== osd_map(90..90 src has 1..90) v4 ==== 8277+0+0 (secure 0 0 0) 0x7fa1c801c050 con 0x5607e8d5ca90 2024-03-24T00:18:04.009 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:04.007+0000 7fa1ccc00640 1 -- 172.21.15.118:0/308400066 <== mon.2 v2:172.21.15.189:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) v2 ==== 24+0+0 (secure 0 0 0) 0x7fa1c802a410 con 0x5607e8d5ca90 2024-03-24T00:18:04.010 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:04.007+0000 7fa1e8841740 10 client.26126 fetch_fsmap learned FSMap version 28 2024-03-24T00:18:04.010 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:04.007+0000 7fa1e8841740 1 -- 172.21.15.118:0/308400066 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) v3 -- 0x5607e8996170 con 0x5607e8d5ca90 2024-03-24T00:18:04.010 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:04.007+0000 7fa1ccc00640 1 -- 172.21.15.118:0/308400066 <== mon.2 v2:172.21.15.189:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fa1d00bcf30 con 0x5607e8d5ca90 2024-03-24T00:18:04.010 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:04.007+0000 7fa1ccc00640 1 -- 172.21.15.118:0/308400066 <== mon.2 v2:172.21.15.189:3300/0 5 ==== fsmap(e 28) v1 ==== 7934+0+0 (secure 0 0 0) 0x7fa1c801e070 con 0x5607e8d5ca90 2024-03-24T00:18:04.010 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:04.007+0000 7fa1e8841740 10 client.26126 fetch_fsmap finished waiting for FSMap version 28 2024-03-24T00:18:04.010 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:04.007+0000 7fa1e8841740 10 client.26126 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-24T00:18:04.010 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:04.007+0000 7fa1e8841740 20 client.26126 populate_metadata read hostname 'smithi118' 2024-03-24T00:18:04.010 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:04.007+0000 7fa1e8841740 1 --2- 172.21.15.118:0/308400066 >> [v2:172.21.15.189:6838/4074484168,v1:172.21.15.189:6839/4074484168] conn(0x5607e8d71e20 0x5607e8d74260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:18:04.010 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:04.007+0000 7fa1e8841740 4 client.26126 mds_command: new command op to 24569 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-24T00:18:04.010 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:04.007+0000 7fa1e8841740 1 -- 172.21.15.118:0/308400066 --> [v2:172.21.15.189:6838/4074484168,v1:172.21.15.189:6839/4074484168] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x5607e8ccf3f0 con 0x5607e8d71e20 2024-03-24T00:18:04.010 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:04.008+0000 7fa1dea00640 1 --2- 172.21.15.118:0/308400066 >> [v2:172.21.15.189:6838/4074484168,v1:172.21.15.189:6839/4074484168] conn(0x5607e8d71e20 0x5607e8d74260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:18:04.011 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:04.008+0000 7fa1dea00640 1 --2- 172.21.15.118:0/308400066 >> [v2:172.21.15.189:6838/4074484168,v1:172.21.15.189:6839/4074484168] conn(0x5607e8d71e20 0x5607e8d74260 crc :-1 s=READY pgs=85 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=a7063dda7977692d server_cookie=6f573677d016e124 in_seq=0 out_seq=0 2024-03-24T00:18:04.011 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:04.008+0000 7fa1ccc00640 10 client.26126 ms_handle_connect on v2:172.21.15.189:6838/4074484168 2024-03-24T00:18:04.012 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:04.010+0000 7fa1ccc00640 1 -- 172.21.15.118:0/308400066 <== mds.0 v2:172.21.15.189:6838/4074484168 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x5607e8ccf3f0 con 0x5607e8d71e20 2024-03-24T00:18:04.012 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:04.010+0000 7fa1ccc00640 10 client.26126 handle_command_reply: tid=0 2024-03-24T00:18:04.012 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:04.010+0000 7fa1e8841740 1 client.26126 shutdown 2024-03-24T00:18:04.012 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:04.010+0000 7fa1e8841740 1 -- 172.21.15.118:0/308400066 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5607e8d5ca90 msgr2=0x5607e8d664c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:18:04.012 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:04.010+0000 7fa1e8841740 1 --2- 172.21.15.118:0/308400066 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5607e8d5ca90 0x5607e8d664c0 secure :-1 s=READY pgs=327 cs=0 l=1 rev1=1 crypto rx=0x7fa1c8011f50 tx=0x7fa1c8005f90 comp rx=0 tx=0).stop 2024-03-24T00:18:04.012 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:04.010+0000 7fa1e8841740 1 -- 172.21.15.118:0/308400066 shutdown_connections 2024-03-24T00:18:04.012 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:04.010+0000 7fa1e8841740 1 --2- 172.21.15.118:0/308400066 >> [v2:172.21.15.189:6838/4074484168,v1:172.21.15.189:6839/4074484168] conn(0x5607e8d71e20 0x5607e8d74260 crc :-1 s=READY pgs=85 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:18:04.012 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:04.010+0000 7fa1e8841740 1 --2- 172.21.15.118:0/308400066 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5607e8d61c00 0x5607e8d6d500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:18:04.012 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:04.010+0000 7fa1e8841740 1 --2- 172.21.15.118:0/308400066 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5607e8d5ca90 0x5607e8d664c0 unknown :-1 s=CLOSED pgs=327 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:18:04.012 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:04.010+0000 7fa1e8841740 1 --2- 172.21.15.118:0/308400066 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5607e8d5c0a0 0x5607e8d65f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:18:04.012 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:04.010+0000 7fa1ccc00640 0 client.26126 ms_handle_reset on v2:172.21.15.189:6838/4074484168 2024-03-24T00:18:04.012 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:04.010+0000 7fa1e8841740 1 -- 172.21.15.118:0/308400066 >> 172.21.15.118:0/308400066 conn(0x5607e8d57610 msgr2=0x5607e8d56830 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:18:04.012 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:04.011+0000 7fa1e8841740 1 -- 172.21.15.118:0/308400066 shutdown_connections 2024-03-24T00:18:04.012 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:04.011+0000 7fa1e8841740 1 -- 172.21.15.118:0/308400066 wait complete. 2024-03-24T00:18:04.013 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:04.011+0000 7fa1e8841740 20 client.26126 trim_cache size 0 max 16384 2024-03-24T00:18:04.041 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:04.042+0000 7fa1e8841740 1 Processor -- start 2024-03-24T00:18:04.043 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:04.043+0000 7fa1e8841740 1 -- start start 2024-03-24T00:18:04.044 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:04.044+0000 7fa1e8841740 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5607e8d5c0a0 0x5607e8d5aee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:18:04.044 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:04.044+0000 7fa1e8841740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5607e8d5ca90 0x5607e8d6d4c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:18:04.044 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:04.045+0000 7fa1e8841740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5607e8d61c00 0x5607e8d566b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:18:04.045 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:04.045+0000 7fa1df400640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5607e8d61c00 0x5607e8d566b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:18:04.045 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:04.045+0000 7fa1df400640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5607e8d61c00 0x5607e8d566b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.118:33586/0 (socket says 172.21.15.118:33586) 2024-03-24T00:18:04.045 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:04.045+0000 7fa1df400640 1 -- 172.21.15.118:0/3653444317 learned_addr learned my addr 172.21.15.118:0/3653444317 (peer_addr_for_me v2:172.21.15.118:0/0) 2024-03-24T00:18:04.045 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:04.046+0000 7fa1e8841740 1 -- 172.21.15.118:0/3653444317 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa1e01aef40 con 0x5607e8d5c0a0 2024-03-24T00:18:04.045 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:04.046+0000 7fa1e8841740 1 -- 172.21.15.118:0/3653444317 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa1e01b08c0 con 0x5607e8d61c00 2024-03-24T00:18:04.046 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:04.046+0000 7fa1e8841740 1 -- 172.21.15.118:0/3653444317 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa1e01af5a0 con 0x5607e8d5ca90 2024-03-24T00:18:04.046 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:04.046+0000 7fa1d6000640 1 --2- 172.21.15.118:0/3653444317 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5607e8d5ca90 0x5607e8d6d4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:18:04.046 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:04.046+0000 7fa1d6000640 1 -- 172.21.15.118:0/3653444317 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5607e8d61c00 msgr2=0x5607e8d566b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:18:04.046 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:04.046+0000 7fa1d6000640 1 --2- 172.21.15.118:0/3653444317 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5607e8d61c00 0x5607e8d566b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:18:04.046 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:04.046+0000 7fa1d6000640 1 -- 172.21.15.118:0/3653444317 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5607e8d5c0a0 msgr2=0x5607e8d5aee0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-24T00:18:04.046 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:04.046+0000 7fa1d6000640 1 --2- 172.21.15.118:0/3653444317 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5607e8d5c0a0 0x5607e8d5aee0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:18:04.046 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:04.046+0000 7fa1d6000640 1 -- 172.21.15.118:0/3653444317 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5607e8d52c20 con 0x5607e8d5ca90 2024-03-24T00:18:04.049 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:04.046+0000 7fa1d6000640 1 --2- 172.21.15.118:0/3653444317 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5607e8d5ca90 0x5607e8d6d4c0 secure :-1 s=READY pgs=319 cs=0 l=1 rev1=1 crypto rx=0x7fa1c8034530 tx=0x7fa1c8009d60 comp rx=0 tx=0).ready entity=mon.1 client_cookie=49d4607104bf05fe server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:18:04.050 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:04.050+0000 7fa1cf400640 1 -- 172.21.15.118:0/3653444317 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa1c8039070 con 0x5607e8d5ca90 2024-03-24T00:18:04.050 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:04.050+0000 7fa1cf400640 1 -- 172.21.15.118:0/3653444317 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fa1c8013660 con 0x5607e8d5ca90 2024-03-24T00:18:04.050 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:04.050+0000 7fa1cf400640 1 -- 172.21.15.118:0/3653444317 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa1c8034de0 con 0x5607e8d5ca90 2024-03-24T00:18:04.051 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:04.051+0000 7fa1e8841740 1 -- 172.21.15.118:0/3653444317 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5607e8d5ca90 msgr2=0x5607e8d6d4c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:18:04.051 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:04.051+0000 7fa1e8841740 1 --2- 172.21.15.118:0/3653444317 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5607e8d5ca90 0x5607e8d6d4c0 secure :-1 s=READY pgs=319 cs=0 l=1 rev1=1 crypto rx=0x7fa1c8034530 tx=0x7fa1c8009d60 comp rx=0 tx=0).stop 2024-03-24T00:18:04.051 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:04.051+0000 7fa1e8841740 1 -- 172.21.15.118:0/3653444317 shutdown_connections 2024-03-24T00:18:04.051 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:04.051+0000 7fa1e8841740 1 --2- 172.21.15.118:0/3653444317 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5607e8d61c00 0x5607e8d566b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:18:04.051 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:04.051+0000 7fa1e8841740 1 --2- 172.21.15.118:0/3653444317 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5607e8d5ca90 0x5607e8d6d4c0 unknown :-1 s=CLOSED pgs=319 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:18:04.051 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:04.051+0000 7fa1e8841740 1 --2- 172.21.15.118:0/3653444317 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5607e8d5c0a0 0x5607e8d5aee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:18:04.051 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:04.051+0000 7fa1e8841740 1 -- 172.21.15.118:0/3653444317 >> 172.21.15.118:0/3653444317 conn(0x5607e8d57610 msgr2=0x5607e8d55fa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:18:04.051 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:04.051+0000 7fa1e8841740 1 -- 172.21.15.118:0/3653444317 shutdown_connections 2024-03-24T00:18:04.051 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:04.052+0000 7fa1e8841740 1 -- 172.21.15.118:0/3653444317 wait complete. 2024-03-24T00:18:04.052 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:04.052+0000 7fa1e8841740 1 -- start start 2024-03-24T00:18:04.052 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:04.053+0000 7fa1e8841740 1 Processor -- start 2024-03-24T00:18:04.052 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:04.053+0000 7fa1e8841740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5607e8d5c0a0 0x5607e8d6d280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:18:04.053 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:04.054+0000 7fa1dea00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5607e8d5c0a0 0x5607e8d6d280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:18:04.053 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:04.054+0000 7fa1dea00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5607e8d5c0a0 0x5607e8d6d280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.118:44448/0 (socket says 172.21.15.118:44448) 2024-03-24T00:18:04.053 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:04.054+0000 7fa1dea00640 1 -- 172.21.15.118:0/431932511 learned_addr learned my addr 172.21.15.118:0/431932511 (peer_addr_for_me v2:172.21.15.118:0/0) 2024-03-24T00:18:04.053 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:04.054+0000 7fa1e8841740 1 --2- 172.21.15.118:0/431932511 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5607e8d5ca90 0x5607e8d6b900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:18:04.054 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:04.054+0000 7fa1e8841740 1 --2- 172.21.15.118:0/431932511 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5607e8d61c00 0x5607e8d6be40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:18:04.054 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:04.054+0000 7fa1e8841740 1 -- 172.21.15.118:0/431932511 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa1e01b0260 con 0x5607e8d5ca90 2024-03-24T00:18:04.054 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:04.054+0000 7fa1e8841740 1 -- 172.21.15.118:0/431932511 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa1e01afc00 con 0x5607e8d61c00 2024-03-24T00:18:04.054 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:04.054+0000 7fa1e8841740 1 -- 172.21.15.118:0/431932511 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa1e01ad100 con 0x5607e8d5c0a0 2024-03-24T00:18:04.054 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:04.054+0000 7fa1e8841740 10 client.0 _ll_register_callbacks cb 0x7fa1c8009700 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T00:18:04.055 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:04.054+0000 7fa1d6000640 1 --2- 172.21.15.118:0/431932511 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5607e8d5ca90 0x5607e8d6b900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:18:04.055 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:04.054+0000 7fa1dea00640 1 -- 172.21.15.118:0/431932511 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5607e8d61c00 msgr2=0x5607e8d6be40 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-03-24T00:18:04.055 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:04.054+0000 7fa1dea00640 1 --2- 172.21.15.118:0/431932511 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5607e8d61c00 0x5607e8d6be40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:18:04.055 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:04.054+0000 7fa1dea00640 1 -- 172.21.15.118:0/431932511 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5607e8d5ca90 msgr2=0x5607e8d6b900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:18:04.055 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:04.054+0000 7fa1dea00640 1 --2- 172.21.15.118:0/431932511 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5607e8d5ca90 0x5607e8d6b900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:18:04.055 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:04.054+0000 7fa1dea00640 1 -- 172.21.15.118:0/431932511 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5607e8d6c530 con 0x5607e8d5c0a0 2024-03-24T00:18:04.055 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:04.055+0000 7fa1dea00640 1 --2- 172.21.15.118:0/431932511 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5607e8d5c0a0 0x5607e8d6d280 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 crypto rx=0x7fa1d00b9340 tx=0x7fa1d002c4a0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=7b591e1fc22edc05 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:18:04.055 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:04.055+0000 7fa1ccc00640 10 client.0 ms_handle_connect on v2:172.21.15.193:3300/0 2024-03-24T00:18:04.055 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:04.055+0000 7fa1ccc00640 1 -- 172.21.15.118:0/431932511 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa1d004b040 con 0x5607e8d5c0a0 2024-03-24T00:18:04.055 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:04.055+0000 7fa1ccc00640 1 -- 172.21.15.118:0/431932511 <== mon.1 v2:172.21.15.193:3300/0 2 ==== osd_map(90..90 src has 1..90) v4 ==== 8277+0+0 (secure 0 0 0) 0x7fa1d00035f0 con 0x5607e8d5c0a0 2024-03-24T00:18:04.055 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:04.055+0000 7fa1e8841740 1 -- 172.21.15.118:0/431932511 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x5607e8cc9740 con 0x5607e8d5c0a0 2024-03-24T00:18:04.055 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:04.055+0000 7fa1ccc00640 1 -- 172.21.15.118:0/431932511 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) v2 ==== 24+0+0 (secure 0 0 0) 0x7fa1d00bcf30 con 0x5607e8d5c0a0 2024-03-24T00:18:04.056 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:04.056+0000 7fa1e8841740 10 client.26131 fetch_fsmap learned FSMap version 28 2024-03-24T00:18:04.056 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:04.056+0000 7fa1e8841740 1 -- 172.21.15.118:0/431932511 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) v3 -- 0x5607e8dce690 con 0x5607e8d5c0a0 2024-03-24T00:18:04.056 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:04.056+0000 7fa1ccc00640 1 -- 172.21.15.118:0/431932511 <== mon.1 v2:172.21.15.193:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x5607e8cc9740 con 0x5607e8d5c0a0 2024-03-24T00:18:04.057 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:04.057+0000 7fa1ccc00640 1 -- 172.21.15.118:0/431932511 <== mon.1 v2:172.21.15.193:3300/0 5 ==== fsmap(e 28) v1 ==== 7934+0+0 (secure 0 0 0) 0x7fa1d00f00f0 con 0x5607e8d5c0a0 2024-03-24T00:18:04.057 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:04.057+0000 7fa1e8841740 10 client.26131 fetch_fsmap finished waiting for FSMap version 28 2024-03-24T00:18:04.057 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:04.057+0000 7fa1e8841740 10 client.26131 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-24T00:18:04.057 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:04.057+0000 7fa1e8841740 20 client.26131 populate_metadata read hostname 'smithi118' 2024-03-24T00:18:04.057 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:04.057+0000 7fa1e8841740 1 --2- 172.21.15.118:0/431932511 >> [v2:172.21.15.189:6838/4074484168,v1:172.21.15.189:6839/4074484168] conn(0x5607e8dcee60 0x5607e8dd1280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:18:04.057 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:04.057+0000 7fa1e8841740 4 client.26131 mds_command: new command op to 24569 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-03-24T00:18:04.057 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:04.057+0000 7fa1e8841740 1 -- 172.21.15.118:0/431932511 --> [v2:172.21.15.189:6838/4074484168,v1:172.21.15.189:6839/4074484168] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) v1 -- 0x5607e8c47a40 con 0x5607e8dcee60 2024-03-24T00:18:04.057 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:04.057+0000 7fa1d6000640 1 --2- 172.21.15.118:0/431932511 >> [v2:172.21.15.189:6838/4074484168,v1:172.21.15.189:6839/4074484168] conn(0x5607e8dcee60 0x5607e8dd1280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:18:04.057 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:04.057+0000 7fa1d6000640 1 --2- 172.21.15.118:0/431932511 >> [v2:172.21.15.189:6838/4074484168,v1:172.21.15.189:6839/4074484168] conn(0x5607e8dcee60 0x5607e8dd1280 crc :-1 s=READY pgs=86 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=f823afc17d44a319 server_cookie=6f3a29b96a2bb675 in_seq=0 out_seq=0 2024-03-24T00:18:04.058 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:04.057+0000 7fa1ccc00640 10 client.26131 ms_handle_connect on v2:172.21.15.189:6838/4074484168 2024-03-24T00:18:04.060 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:04.060+0000 7fa1ccc00640 1 -- 172.21.15.118:0/431932511 <== mds.0 v2:172.21.15.189:6838/4074484168 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+110 (crc 0 0 0) 0x5607e8c47a40 con 0x5607e8dcee60 2024-03-24T00:18:04.060 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:04.060+0000 7fa1ccc00640 10 client.26131 handle_command_reply: tid=0 2024-03-24T00:18:04.060 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:04.060+0000 7fa1e8841740 1 client.26131 shutdown 2024-03-24T00:18:04.060 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:04.061+0000 7fa1e8841740 1 -- 172.21.15.118:0/431932511 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5607e8d5c0a0 msgr2=0x5607e8d6d280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:18:04.061 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:04.061+0000 7fa1e8841740 1 --2- 172.21.15.118:0/431932511 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5607e8d5c0a0 0x5607e8d6d280 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 crypto rx=0x7fa1d00b9340 tx=0x7fa1d002c4a0 comp rx=0 tx=0).stop 2024-03-24T00:18:04.061 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:04.061+0000 7fa1e8841740 1 -- 172.21.15.118:0/431932511 shutdown_connections 2024-03-24T00:18:04.061 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:04.061+0000 7fa1e8841740 1 --2- 172.21.15.118:0/431932511 >> [v2:172.21.15.189:6838/4074484168,v1:172.21.15.189:6839/4074484168] conn(0x5607e8dcee60 0x5607e8dd1280 crc :-1 s=READY pgs=86 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:18:04.061 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:04.061+0000 7fa1e8841740 1 --2- 172.21.15.118:0/431932511 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5607e8d61c00 0x5607e8d6be40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:18:04.061 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:04.061+0000 7fa1e8841740 1 --2- 172.21.15.118:0/431932511 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5607e8d5ca90 0x5607e8d6b900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:18:04.061 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:04.061+0000 7fa1e8841740 1 --2- 172.21.15.118:0/431932511 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5607e8d5c0a0 0x5607e8d6d280 unknown :-1 s=CLOSED pgs=320 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:18:04.061 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:04.061+0000 7fa1e8841740 1 -- 172.21.15.118:0/431932511 >> 172.21.15.118:0/431932511 conn(0x5607e8d57610 msgr2=0x5607e8d56830 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:18:04.061 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:04.061+0000 7fa1ccc00640 0 client.26131 ms_handle_reset on v2:172.21.15.189:6838/4074484168 2024-03-24T00:18:04.062 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:04.061+0000 7fa1e8841740 1 -- 172.21.15.118:0/431932511 shutdown_connections 2024-03-24T00:18:04.062 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:04.061+0000 7fa1e8841740 1 -- 172.21.15.118:0/431932511 wait complete. 2024-03-24T00:18:04.062 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:04.061+0000 7fa1e8841740 20 client.26131 trim_cache size 0 max 16384 2024-03-24T00:18:04.062 INFO:teuthology.orchestra.run.smithi118.stdout:{ 2024-03-24T00:18:04.062 INFO:teuthology.orchestra.run.smithi118.stdout: "return_code": 0, 2024-03-24T00:18:04.062 INFO:teuthology.orchestra.run.smithi118.stdout: "scrub_tag": "990e5fc3-bf40-49c5-bd80-e6ac9038901c", 2024-03-24T00:18:04.062 INFO:teuthology.orchestra.run.smithi118.stdout: "mode": "asynchronous" 2024-03-24T00:18:04.062 INFO:teuthology.orchestra.run.smithi118.stdout:} 2024-03-24T00:18:04.063 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:04.062+0000 7fa1e5c00640 1 -- 172.21.15.118:0/2424794299 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7fa1c40a2880 msgr2=0x7fa1c40a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:18:04.063 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:04.062+0000 7fa1e5c00640 1 --2- 172.21.15.118:0/2424794299 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7fa1c40a2880 0x7fa1c40a4d40 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 crypto rx=0x5607e8d5ce90 tx=0x7fa1c8009570 comp rx=0 tx=0).stop 2024-03-24T00:18:04.063 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:04.062+0000 7fa1e5c00640 1 -- 172.21.15.118:0/2424794299 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fa1e00fff10 msgr2=0x7fa1e019dfc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:18:04.063 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:04.062+0000 7fa1e5c00640 1 --2- 172.21.15.118:0/2424794299 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fa1e00fff10 0x7fa1e019dfc0 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 crypto rx=0x7fa1d002c8f0 tx=0x7fa1d002c660 comp rx=0 tx=0).stop 2024-03-24T00:18:04.063 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:04.062+0000 7fa1e5c00640 1 -- 172.21.15.118:0/2424794299 shutdown_connections 2024-03-24T00:18:04.063 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:04.062+0000 7fa1e5c00640 1 --2- 172.21.15.118:0/2424794299 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7fa1c40a2880 0x7fa1c40a4d40 unknown :-1 s=CLOSED pgs=315 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:18:04.063 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:04.062+0000 7fa1e5c00640 1 --2- 172.21.15.118:0/2424794299 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa1e0105230 0x7fa1e01a5570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:18:04.064 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:04.062+0000 7fa1e5c00640 1 --2- 172.21.15.118:0/2424794299 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fa1e0102880 0x7fa1e019e500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:18:04.064 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:04.062+0000 7fa1e5c00640 1 --2- 172.21.15.118:0/2424794299 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fa1e00fff10 0x7fa1e019dfc0 unknown :-1 s=CLOSED pgs=317 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:18:04.064 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:04.062+0000 7fa1e5c00640 1 -- 172.21.15.118:0/2424794299 >> 172.21.15.118:0/2424794299 conn(0x7fa1e00fbb10 msgr2=0x7fa1e0103210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:18:04.064 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:04.063+0000 7fa1e5c00640 1 -- 172.21.15.118:0/2424794299 shutdown_connections 2024-03-24T00:18:04.064 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:04.063+0000 7fa1e5c00640 1 -- 172.21.15.118:0/2424794299 wait complete. 2024-03-24T00:18:04.083 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-03-24T00:18:04.083 DEBUG:teuthology.orchestra.run.smithi118:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-24T00:18:04.392 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:04.393+0000 7f3a1be00640 1 -- 172.21.15.118:0/24402358 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f3a1c104930 msgr2=0x7f3a1c106d20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:18:04.392 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:04.393+0000 7f3a1be00640 1 --2- 172.21.15.118:0/24402358 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f3a1c104930 0x7f3a1c106d20 secure :-1 s=READY pgs=321 cs=0 l=1 rev1=1 crypto rx=0x7f3a08009900 tx=0x7f3a0802bc30 comp rx=0 tx=0).stop 2024-03-24T00:18:04.393 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:04.393+0000 7f3a1be00640 1 -- 172.21.15.118:0/24402358 shutdown_connections 2024-03-24T00:18:04.393 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:04.393+0000 7f3a1be00640 1 --2- 172.21.15.118:0/24402358 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f3a1c107260 0x7f3a1c1096b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:18:04.393 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:04.393+0000 7f3a1be00640 1 --2- 172.21.15.118:0/24402358 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f3a1c104930 0x7f3a1c106d20 unknown :-1 s=CLOSED pgs=321 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:18:04.393 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:04.393+0000 7f3a1be00640 1 --2- 172.21.15.118:0/24402358 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3a1c102000 0x7f3a1c1043f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:18:04.394 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:04.393+0000 7f3a1be00640 1 -- 172.21.15.118:0/24402358 >> 172.21.15.118:0/24402358 conn(0x7f3a1c0fba10 msgr2=0x7f3a1c0fde30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:18:04.394 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:04.394+0000 7f3a1be00640 1 -- 172.21.15.118:0/24402358 shutdown_connections 2024-03-24T00:18:04.394 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:04.394+0000 7f3a1be00640 1 -- 172.21.15.118:0/24402358 wait complete. 2024-03-24T00:18:04.394 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:04.395+0000 7f3a1be00640 1 Processor -- start 2024-03-24T00:18:04.395 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:04.395+0000 7f3a1be00640 1 -- start start 2024-03-24T00:18:04.395 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:04.396+0000 7f3a1be00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3a1c102000 0x7f3a1c101970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:18:04.395 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:04.396+0000 7f3a1be00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f3a1c104930 0x7f3a1c0fffb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:18:04.395 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:04.396+0000 7f3a1be00640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f3a1c107260 0x7f3a1c1004f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:18:04.395 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:04.396+0000 7f3a1be00640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3a1c10d260 con 0x7f3a1c107260 2024-03-24T00:18:04.395 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:04.396+0000 7f3a1be00640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3a1c10d0e0 con 0x7f3a1c102000 2024-03-24T00:18:04.395 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:04.396+0000 7f3a1be00640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3a1c10d3e0 con 0x7f3a1c104930 2024-03-24T00:18:04.396 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:04.396+0000 7f3a1a000640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f3a1c104930 0x7f3a1c0fffb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:18:04.397 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:04.396+0000 7f3a1a000640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f3a1c104930 0x7f3a1c0fffb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.118:44460/0 (socket says 172.21.15.118:44460) 2024-03-24T00:18:04.397 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:04.396+0000 7f3a1a000640 1 -- 172.21.15.118:0/1138875809 learned_addr learned my addr 172.21.15.118:0/1138875809 (peer_addr_for_me v2:172.21.15.118:0/0) 2024-03-24T00:18:04.397 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:04.396+0000 7f3a1b400640 1 --2- 172.21.15.118:0/1138875809 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f3a1c107260 0x7f3a1c1004f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:18:04.397 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:04.397+0000 7f3a1a000640 1 -- 172.21.15.118:0/1138875809 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3a1c102000 msgr2=0x7f3a1c101970 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-03-24T00:18:04.397 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:04.397+0000 7f3a1a000640 1 --2- 172.21.15.118:0/1138875809 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3a1c102000 0x7f3a1c101970 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:18:04.397 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:04.397+0000 7f3a1a000640 1 -- 172.21.15.118:0/1138875809 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f3a1c107260 msgr2=0x7f3a1c1004f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:18:04.397 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:04.397+0000 7f3a1a000640 1 --2- 172.21.15.118:0/1138875809 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f3a1c107260 0x7f3a1c1004f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:18:04.397 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:04.397+0000 7f3a1a000640 1 -- 172.21.15.118:0/1138875809 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3a1c100d80 con 0x7f3a1c104930 2024-03-24T00:18:04.397 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:04.397+0000 7f3a1a000640 1 --2- 172.21.15.118:0/1138875809 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f3a1c104930 0x7f3a1c0fffb0 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 crypto rx=0x7f3a08000c00 tx=0x7f3a0802c6c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:18:04.398 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:04.397+0000 7f3a13400640 1 -- 172.21.15.118:0/1138875809 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3a08043020 con 0x7f3a1c104930 2024-03-24T00:18:04.398 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:04.397+0000 7f3a13400640 1 -- 172.21.15.118:0/1138875809 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f3a080054e0 con 0x7f3a1c104930 2024-03-24T00:18:04.398 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:04.398+0000 7f3a1be00640 1 -- 172.21.15.118:0/1138875809 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3a1c1a19e0 con 0x7f3a1c104930 2024-03-24T00:18:04.398 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:04.398+0000 7f3a13400640 1 -- 172.21.15.118:0/1138875809 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3a08003ce0 con 0x7f3a1c104930 2024-03-24T00:18:04.398 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:04.398+0000 7f3a1be00640 1 -- 172.21.15.118:0/1138875809 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3a1c1a1f00 con 0x7f3a1c104930 2024-03-24T00:18:04.401 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:04.402+0000 7f3a245f0740 1 Processor -- start 2024-03-24T00:18:04.402 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:04.402+0000 7f3a245f0740 1 -- start start 2024-03-24T00:18:04.403 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:04.403+0000 7f3a245f0740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x564d253ef050 0x564d253ef470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:18:04.403 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:04.403+0000 7f3a245f0740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x564d253efa40 0x564d253f4540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:18:04.403 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:04.403+0000 7f3a245f0740 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x564d253f4bb0 0x564d253f7040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:18:04.403 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:04.403+0000 7f3a245f0740 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x564d25354250 con 0x564d253f4bb0 2024-03-24T00:18:04.403 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:04.403+0000 7f3a245f0740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x564d252fca30 con 0x564d253ef050 2024-03-24T00:18:04.403 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:04.403+0000 7f3a245f0740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x564d253502f0 con 0x564d253efa40 2024-03-24T00:18:04.403 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:04.403+0000 7f3a1b400640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x564d253ef050 0x564d253ef470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:18:04.403 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:04.403+0000 7f3a1b400640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x564d253ef050 0x564d253ef470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.118:33636/0 (socket says 172.21.15.118:33636) 2024-03-24T00:18:04.403 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:04.403+0000 7f3a1b400640 1 -- 172.21.15.118:0/2880471329 learned_addr learned my addr 172.21.15.118:0/2880471329 (peer_addr_for_me v2:172.21.15.118:0/0) 2024-03-24T00:18:04.403 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:04.403+0000 7f3a1aa00640 1 --2- 172.21.15.118:0/2880471329 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x564d253efa40 0x564d253f4540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:18:04.404 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:04.403+0000 7f3a1b400640 1 -- 172.21.15.118:0/2880471329 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x564d253efa40 msgr2=0x564d253f4540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:18:04.404 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:04.403+0000 7f3a1b400640 1 --2- 172.21.15.118:0/2880471329 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x564d253efa40 0x564d253f4540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:18:04.404 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:04.403+0000 7f3a1b400640 1 -- 172.21.15.118:0/2880471329 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x564d253f4bb0 msgr2=0x564d253f7040 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-03-24T00:18:04.404 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:04.404+0000 7f3a1b400640 1 --2- 172.21.15.118:0/2880471329 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x564d253f4bb0 0x564d253f7040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:18:04.404 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:04.404+0000 7f3a1b400640 1 -- 172.21.15.118:0/2880471329 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x564d252e7690 con 0x564d253ef050 2024-03-24T00:18:04.404 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:04.404+0000 7f3a13400640 1 -- 172.21.15.118:0/1138875809 <== mon.1 v2:172.21.15.193:3300/0 4 ==== mgrmap(e 20) v1 ==== 137291+0+0 (secure 0 0 0) 0x7f3a08046020 con 0x7f3a1c104930 2024-03-24T00:18:04.404 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:04.404+0000 7f3a1b400640 1 --2- 172.21.15.118:0/2880471329 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x564d253ef050 0x564d253ef470 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 crypto rx=0x7f3a14009eb0 tx=0x7f3a1400f730 comp rx=0 tx=0).ready entity=mon.2 client_cookie=28105ed9f545a569 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:18:04.404 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:04.404+0000 7f3a07e00640 1 -- 172.21.15.118:0/2880471329 <== mon.2 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3a14011640 con 0x564d253ef050 2024-03-24T00:18:04.404 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:04.404+0000 7f3a07e00640 1 -- 172.21.15.118:0/2880471329 <== mon.2 v2:172.21.15.189:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f3a14011cc0 con 0x564d253ef050 2024-03-24T00:18:04.404 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:04.404+0000 7f3a07e00640 1 -- 172.21.15.118:0/2880471329 <== mon.2 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3a1401bd40 con 0x564d253ef050 2024-03-24T00:18:04.405 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:04.405+0000 7f3a245f0740 1 -- 172.21.15.118:0/2880471329 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x564d253ef050 msgr2=0x564d253ef470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:18:04.405 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:04.405+0000 7f3a245f0740 1 --2- 172.21.15.118:0/2880471329 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x564d253ef050 0x564d253ef470 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 crypto rx=0x7f3a14009eb0 tx=0x7f3a1400f730 comp rx=0 tx=0).stop 2024-03-24T00:18:04.405 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:04.405+0000 7f3a245f0740 1 -- 172.21.15.118:0/2880471329 shutdown_connections 2024-03-24T00:18:04.405 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:04.405+0000 7f3a245f0740 1 --2- 172.21.15.118:0/2880471329 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x564d253f4bb0 0x564d253f7040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:18:04.405 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:04.405+0000 7f3a245f0740 1 --2- 172.21.15.118:0/2880471329 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x564d253efa40 0x564d253f4540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:18:04.406 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:04.405+0000 7f3a245f0740 1 --2- 172.21.15.118:0/2880471329 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x564d253ef050 0x564d253ef470 unknown :-1 s=CLOSED pgs=328 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:18:04.406 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:04.405+0000 7f3a245f0740 1 -- 172.21.15.118:0/2880471329 >> 172.21.15.118:0/2880471329 conn(0x564d253b7200 msgr2=0x564d253ea9c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:18:04.406 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:04.406+0000 7f3a245f0740 1 -- 172.21.15.118:0/2880471329 shutdown_connections 2024-03-24T00:18:04.406 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:04.406+0000 7f3a245f0740 1 -- 172.21.15.118:0/2880471329 wait complete. 2024-03-24T00:18:04.406 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:04.406+0000 7f3a13400640 1 --2- 172.21.15.118:0/1138875809 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7f3a000a28e0 0x7f3a000a4da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:18:04.406 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:04.406+0000 7f3a13400640 1 -- 172.21.15.118:0/1138875809 <== mon.1 v2:172.21.15.193:3300/0 5 ==== osd_map(90..90 src has 1..90) v4 ==== 8277+0+0 (secure 0 0 0) 0x7f3a080f0c00 con 0x7f3a1c104930 2024-03-24T00:18:04.406 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:04.406+0000 7f3a1aa00640 1 --2- 172.21.15.118:0/1138875809 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7f3a000a28e0 0x7f3a000a4da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:18:04.407 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:04.406+0000 7f3a245f0740 1 -- start start 2024-03-24T00:18:04.407 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:04.406+0000 7f3a245f0740 1 Processor -- start 2024-03-24T00:18:04.407 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:04.407+0000 7f3a1aa00640 1 --2- 172.21.15.118:0/1138875809 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7f3a000a28e0 0x7f3a000a4da0 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 crypto rx=0x564d253eb910 tx=0x7f3a0c00b2e0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:18:04.407 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:04.407+0000 7f3a245f0740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x564d253ef050 0x564d25401a70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:18:04.407 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:04.407+0000 7f3a245f0740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x564d253efa40 0x564d25401fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:18:04.408 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:04.408+0000 7f3a245f0740 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x564d25406f70 0x564d25409360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:18:04.408 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:04.408+0000 7f3a1aa00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x564d253ef050 0x564d25401a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:18:04.408 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:04.408+0000 7f3a1aa00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x564d253ef050 0x564d25401a70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.118:44478/0 (socket says 172.21.15.118:44478) 2024-03-24T00:18:04.408 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:04.408+0000 7f3a1aa00640 1 -- 172.21.15.118:0/2911525244 learned_addr learned my addr 172.21.15.118:0/2911525244 (peer_addr_for_me v2:172.21.15.118:0/0) 2024-03-24T00:18:04.409 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:04.408+0000 7f3a245f0740 1 -- 172.21.15.118:0/2911525244 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x564d25400370 con 0x564d25406f70 2024-03-24T00:18:04.409 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:04.408+0000 7f3a245f0740 1 -- 172.21.15.118:0/2911525244 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x564d2530fe80 con 0x564d253efa40 2024-03-24T00:18:04.409 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:04.408+0000 7f3a245f0740 1 -- 172.21.15.118:0/2911525244 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x564d254004f0 con 0x564d253ef050 2024-03-24T00:18:04.409 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:04.408+0000 7f3a245f0740 10 client.0 _ll_register_callbacks cb 0x564d25322130 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T00:18:04.409 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:04.408+0000 7f3a1b400640 1 --2- 172.21.15.118:0/2911525244 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x564d25406f70 0x564d25409360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:18:04.409 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:04.408+0000 7f3a1aa00640 1 -- 172.21.15.118:0/2911525244 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x564d253efa40 msgr2=0x564d25401fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:18:04.409 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:04.408+0000 7f3a1a000640 1 --2- 172.21.15.118:0/2911525244 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x564d253efa40 0x564d25401fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:18:04.409 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:04.408+0000 7f3a1aa00640 1 --2- 172.21.15.118:0/2911525244 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x564d253efa40 0x564d25401fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:18:04.409 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:04.408+0000 7f3a1aa00640 1 -- 172.21.15.118:0/2911525244 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x564d25406f70 msgr2=0x564d25409360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:18:04.409 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:04.408+0000 7f3a1aa00640 1 --2- 172.21.15.118:0/2911525244 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x564d25406f70 0x564d25409360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:18:04.409 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:04.409+0000 7f3a1aa00640 1 -- 172.21.15.118:0/2911525244 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x564d251327d0 con 0x564d253ef050 2024-03-24T00:18:04.409 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:04.409+0000 7f3a1b400640 1 --2- 172.21.15.118:0/2911525244 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x564d25406f70 0x564d25409360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-24T00:18:04.409 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:04.409+0000 7f3a1a000640 1 --2- 172.21.15.118:0/2911525244 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x564d253efa40 0x564d25401fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-24T00:18:04.409 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:04.409+0000 7f3a245f0740 1 -- 172.21.15.118:0/2911525244 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f3a14011cc0 con 0x564d253ef050 2024-03-24T00:18:04.411 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:04.409+0000 7f3a1aa00640 1 --2- 172.21.15.118:0/2911525244 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x564d253ef050 0x564d25401a70 secure :-1 s=READY pgs=323 cs=0 l=1 rev1=1 crypto rx=0x7f3a0c0041b0 tx=0x7f3a0c004730 comp rx=0 tx=0).ready entity=mon.1 client_cookie=bf3c4d82a31b4238 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:18:04.411 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:04.409+0000 7f3a04c00640 10 client.26143 ms_handle_connect on v2:172.21.15.193:3300/0 2024-03-24T00:18:04.411 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:04.410+0000 7f3a04c00640 1 -- 172.21.15.118:0/2911525244 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3a0c01e420 con 0x564d253ef050 2024-03-24T00:18:04.411 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:04.410+0000 7f3a04c00640 1 -- 172.21.15.118:0/2911525244 <== mon.1 v2:172.21.15.193:3300/0 2 ==== osd_map(90..90 src has 1..90) v4 ==== 8277+0+0 (secure 0 0 0) 0x7f3a0c01e5c0 con 0x564d253ef050 2024-03-24T00:18:04.411 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:04.410+0000 7f3a04c00640 1 -- 172.21.15.118:0/2911525244 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) v2 ==== 24+0+0 (secure 0 0 0) 0x7f3a0c01e7c0 con 0x564d253ef050 2024-03-24T00:18:04.411 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:04.410+0000 7f3a245f0740 10 client.26143 fetch_fsmap learned FSMap version 28 2024-03-24T00:18:04.411 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:04.410+0000 7f3a245f0740 1 -- 172.21.15.118:0/2911525244 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({fsmap=28}) v3 -- 0x564d250683f0 con 0x564d253ef050 2024-03-24T00:18:04.411 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:04.410+0000 7f3a04c00640 1 -- 172.21.15.118:0/2911525244 <== mon.1 v2:172.21.15.193:3300/0 4 ==== fsmap(e 28) v1 ==== 7934+0+0 (secure 0 0 0) 0x7f3a0c01e960 con 0x564d253ef050 2024-03-24T00:18:04.411 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:04.410+0000 7f3a245f0740 10 client.26143 fetch_fsmap finished waiting for FSMap version 28 2024-03-24T00:18:04.411 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:04.410+0000 7f3a245f0740 10 client.26143 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-24T00:18:04.412 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:04.410+0000 7f3a245f0740 20 client.26143 populate_metadata read hostname 'smithi118' 2024-03-24T00:18:04.412 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:04.410+0000 7f3a245f0740 1 --2- 172.21.15.118:0/2911525244 >> [v2:172.21.15.189:6838/4074484168,v1:172.21.15.189:6839/4074484168] conn(0x564d2540dad0 0x564d2540fee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:18:04.412 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:04.410+0000 7f3a245f0740 4 client.26143 mds_command: new command op to 24569 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-24T00:18:04.412 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:04.410+0000 7f3a245f0740 1 -- 172.21.15.118:0/2911525244 --> [v2:172.21.15.189:6838/4074484168,v1:172.21.15.189:6839/4074484168] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x564d25361790 con 0x564d2540dad0 2024-03-24T00:18:04.412 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:04.411+0000 7f3a1a000640 1 --2- 172.21.15.118:0/2911525244 >> [v2:172.21.15.189:6838/4074484168,v1:172.21.15.189:6839/4074484168] conn(0x564d2540dad0 0x564d2540fee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:18:04.412 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:04.411+0000 7f3a1a000640 1 --2- 172.21.15.118:0/2911525244 >> [v2:172.21.15.189:6838/4074484168,v1:172.21.15.189:6839/4074484168] conn(0x564d2540dad0 0x564d2540fee0 crc :-1 s=READY pgs=87 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=8e79bc85faab6e6c server_cookie=45c6eb0175c3011b in_seq=0 out_seq=0 2024-03-24T00:18:04.412 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:04.411+0000 7f3a04c00640 10 client.26143 ms_handle_connect on v2:172.21.15.189:6838/4074484168 2024-03-24T00:18:04.415 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:04.415+0000 7f3a04c00640 1 -- 172.21.15.118:0/2911525244 <== mds.0 v2:172.21.15.189:6838/4074484168 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x564d25361790 con 0x564d2540dad0 2024-03-24T00:18:04.415 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:04.415+0000 7f3a04c00640 10 client.26143 handle_command_reply: tid=0 2024-03-24T00:18:04.415 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:04.415+0000 7f3a245f0740 1 client.26143 shutdown 2024-03-24T00:18:04.415 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:04.415+0000 7f3a245f0740 1 -- 172.21.15.118:0/2911525244 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x564d253ef050 msgr2=0x564d25401a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:18:04.415 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:04.415+0000 7f3a245f0740 1 --2- 172.21.15.118:0/2911525244 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x564d253ef050 0x564d25401a70 secure :-1 s=READY pgs=323 cs=0 l=1 rev1=1 crypto rx=0x7f3a0c0041b0 tx=0x7f3a0c004730 comp rx=0 tx=0).stop 2024-03-24T00:18:04.415 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:04.415+0000 7f3a245f0740 1 -- 172.21.15.118:0/2911525244 shutdown_connections 2024-03-24T00:18:04.415 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:04.415+0000 7f3a245f0740 1 --2- 172.21.15.118:0/2911525244 >> [v2:172.21.15.189:6838/4074484168,v1:172.21.15.189:6839/4074484168] conn(0x564d2540dad0 0x564d2540fee0 crc :-1 s=READY pgs=87 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:18:04.415 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:04.415+0000 7f3a245f0740 1 --2- 172.21.15.118:0/2911525244 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x564d25406f70 0x564d25409360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:18:04.415 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:04.415+0000 7f3a245f0740 1 --2- 172.21.15.118:0/2911525244 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x564d253efa40 0x564d25401fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:18:04.415 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:04.415+0000 7f3a245f0740 1 --2- 172.21.15.118:0/2911525244 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x564d253ef050 0x564d25401a70 unknown :-1 s=CLOSED pgs=323 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:18:04.415 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:04.415+0000 7f3a245f0740 1 -- 172.21.15.118:0/2911525244 >> 172.21.15.118:0/2911525244 conn(0x564d253b7200 msgr2=0x564d253e9bc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:18:04.415 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:04.415+0000 7f3a04c00640 0 client.26143 ms_handle_reset on v2:172.21.15.189:6838/4074484168 2024-03-24T00:18:04.416 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:04.415+0000 7f3a245f0740 1 -- 172.21.15.118:0/2911525244 shutdown_connections 2024-03-24T00:18:04.416 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:04.415+0000 7f3a245f0740 1 -- 172.21.15.118:0/2911525244 wait complete. 2024-03-24T00:18:04.416 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:04.416+0000 7f3a245f0740 20 client.26143 trim_cache size 0 max 16384 2024-03-24T00:18:04.436 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:04.437+0000 7f3a245f0740 1 Processor -- start 2024-03-24T00:18:04.436 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:04.437+0000 7f3a245f0740 1 -- start start 2024-03-24T00:18:04.438 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:04.437+0000 7f3a245f0740 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x564d253ef050 0x564d25401f10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:18:04.438 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:04.437+0000 7f3a245f0740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x564d253efa40 0x564d253ee310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:18:04.438 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:04.437+0000 7f3a245f0740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x564d25406f70 0x564d253ee850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:18:04.438 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:04.437+0000 7f3a245f0740 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3a1c1a65b0 con 0x564d253ef050 2024-03-24T00:18:04.438 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:04.437+0000 7f3a245f0740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3a1c1a7f30 con 0x564d25406f70 2024-03-24T00:18:04.438 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:04.437+0000 7f3a245f0740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3a1c1a6c10 con 0x564d253efa40 2024-03-24T00:18:04.438 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:04.437+0000 7f3a1a000640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x564d253efa40 0x564d253ee310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:18:04.438 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:04.438+0000 7f3a1aa00640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x564d253ef050 0x564d25401f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:18:04.439 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:04.438+0000 7f3a1b400640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x564d25406f70 0x564d253ee850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:18:04.439 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:04.438+0000 7f3a1a000640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x564d253efa40 0x564d253ee310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.118:44492/0 (socket says 172.21.15.118:44492) 2024-03-24T00:18:04.439 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:04.438+0000 7f3a1a000640 1 -- 172.21.15.118:0/1680095607 learned_addr learned my addr 172.21.15.118:0/1680095607 (peer_addr_for_me v2:172.21.15.118:0/0) 2024-03-24T00:18:04.439 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:04.438+0000 7f3a1a000640 1 -- 172.21.15.118:0/1680095607 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x564d25406f70 msgr2=0x564d253ee850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:18:04.439 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:04.438+0000 7f3a1a000640 1 --2- 172.21.15.118:0/1680095607 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x564d25406f70 0x564d253ee850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:18:04.439 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:04.438+0000 7f3a1a000640 1 -- 172.21.15.118:0/1680095607 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x564d253ef050 msgr2=0x564d25401f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:18:04.439 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:04.438+0000 7f3a1a000640 1 --2- 172.21.15.118:0/1680095607 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x564d253ef050 0x564d25401f10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:18:04.439 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:04.438+0000 7f3a1a000640 1 -- 172.21.15.118:0/1680095607 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x564d250280f0 con 0x564d253efa40 2024-03-24T00:18:04.439 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:04.438+0000 7f3a1b400640 1 --2- 172.21.15.118:0/1680095607 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x564d25406f70 0x564d253ee850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T00:18:04.439 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:04.438+0000 7f3a1aa00640 1 --2- 172.21.15.118:0/1680095607 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x564d253ef050 0x564d25401f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T00:18:04.439 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:04.438+0000 7f3a1a000640 1 --2- 172.21.15.118:0/1680095607 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x564d253efa40 0x564d253ee310 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 crypto rx=0x7f3a080b9ee0 tx=0x7f3a0802bc30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=dd22217de3049bbd server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:18:04.439 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:04.438+0000 7f3a07400640 1 -- 172.21.15.118:0/1680095607 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3a080d5020 con 0x564d253efa40 2024-03-24T00:18:04.439 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:04.438+0000 7f3a07400640 1 -- 172.21.15.118:0/1680095607 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f3a080044a0 con 0x564d253efa40 2024-03-24T00:18:04.439 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:04.438+0000 7f3a07400640 1 -- 172.21.15.118:0/1680095607 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3a080bb810 con 0x564d253efa40 2024-03-24T00:18:04.443 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:04.439+0000 7f3a245f0740 1 -- 172.21.15.118:0/1680095607 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x564d253efa40 msgr2=0x564d253ee310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:18:04.443 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:04.439+0000 7f3a245f0740 1 --2- 172.21.15.118:0/1680095607 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x564d253efa40 0x564d253ee310 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 crypto rx=0x7f3a080b9ee0 tx=0x7f3a0802bc30 comp rx=0 tx=0).stop 2024-03-24T00:18:04.443 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:04.439+0000 7f3a245f0740 1 -- 172.21.15.118:0/1680095607 shutdown_connections 2024-03-24T00:18:04.443 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:04.439+0000 7f3a245f0740 1 --2- 172.21.15.118:0/1680095607 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x564d25406f70 0x564d253ee850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:18:04.443 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:04.439+0000 7f3a245f0740 1 --2- 172.21.15.118:0/1680095607 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x564d253efa40 0x564d253ee310 unknown :-1 s=CLOSED pgs=324 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:18:04.443 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:04.439+0000 7f3a245f0740 1 --2- 172.21.15.118:0/1680095607 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x564d253ef050 0x564d25401f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:18:04.443 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:04.439+0000 7f3a245f0740 1 -- 172.21.15.118:0/1680095607 >> 172.21.15.118:0/1680095607 conn(0x564d253b7200 msgr2=0x564d253edc00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:18:04.444 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:04.439+0000 7f3a245f0740 1 -- 172.21.15.118:0/1680095607 shutdown_connections 2024-03-24T00:18:04.444 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:04.439+0000 7f3a245f0740 1 -- 172.21.15.118:0/1680095607 wait complete. 2024-03-24T00:18:04.444 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:04.440+0000 7f3a245f0740 1 -- start start 2024-03-24T00:18:04.444 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:04.440+0000 7f3a245f0740 1 Processor -- start 2024-03-24T00:18:04.444 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:04.440+0000 7f3a245f0740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x564d253ef050 0x564d2541fab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:18:04.444 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:04.440+0000 7f3a245f0740 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x564d253efa40 0x564d2541fff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:18:04.444 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:04.440+0000 7f3a245f0740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x564d25406f70 0x564d2541e140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:18:04.444 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:04.440+0000 7f3a245f0740 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3a1c1a78d0 con 0x564d253efa40 2024-03-24T00:18:04.444 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:04.440+0000 7f3a245f0740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3a1c1a7270 con 0x564d253ef050 2024-03-24T00:18:04.444 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:04.440+0000 7f3a245f0740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3a1c1a4770 con 0x564d25406f70 2024-03-24T00:18:04.444 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:04.441+0000 7f3a245f0740 10 client.0 _ll_register_callbacks cb 0x7f3a0c004300 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T00:18:04.444 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:04.441+0000 7f3a1aa00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x564d253ef050 0x564d2541fab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:18:04.444 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:04.441+0000 7f3a1aa00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x564d253ef050 0x564d2541fab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.118:33664/0 (socket says 172.21.15.118:33664) 2024-03-24T00:18:04.444 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:04.441+0000 7f3a1aa00640 1 -- 172.21.15.118:0/1862341662 learned_addr learned my addr 172.21.15.118:0/1862341662 (peer_addr_for_me v2:172.21.15.118:0/0) 2024-03-24T00:18:04.444 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:04.441+0000 7f3a1b400640 1 --2- 172.21.15.118:0/1862341662 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x564d25406f70 0x564d2541e140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:18:04.445 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:04.441+0000 7f3a1a000640 1 --2- 172.21.15.118:0/1862341662 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x564d253efa40 0x564d2541fff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:18:04.445 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:04.441+0000 7f3a1aa00640 1 -- 172.21.15.118:0/1862341662 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x564d25406f70 msgr2=0x564d2541e140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:18:04.445 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:04.441+0000 7f3a1aa00640 1 --2- 172.21.15.118:0/1862341662 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x564d25406f70 0x564d2541e140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:18:04.445 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:04.441+0000 7f3a1aa00640 1 -- 172.21.15.118:0/1862341662 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x564d253efa40 msgr2=0x564d2541fff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:18:04.445 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:04.441+0000 7f3a1aa00640 1 --2- 172.21.15.118:0/1862341662 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x564d253efa40 0x564d2541fff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:18:04.445 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:04.441+0000 7f3a1aa00640 1 -- 172.21.15.118:0/1862341662 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x564d2541e970 con 0x564d253ef050 2024-03-24T00:18:04.445 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:04.441+0000 7f3a1aa00640 1 --2- 172.21.15.118:0/1862341662 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x564d253ef050 0x564d2541fab0 secure :-1 s=READY pgs=329 cs=0 l=1 rev1=1 crypto rx=0x7f3a0c0041b0 tx=0x7f3a0c017000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=5210ea8160e568b6 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:18:04.445 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:04.441+0000 7f3a04c00640 10 client.0 ms_handle_connect on v2:172.21.15.189:3300/0 2024-03-24T00:18:04.445 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:04.442+0000 7f3a04c00640 1 -- 172.21.15.118:0/1862341662 <== mon.2 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3a0c03c430 con 0x564d253ef050 2024-03-24T00:18:04.445 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:04.442+0000 7f3a245f0740 1 -- 172.21.15.118:0/1862341662 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x564d2535bae0 con 0x564d253ef050 2024-03-24T00:18:04.445 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:04.442+0000 7f3a04c00640 1 -- 172.21.15.118:0/1862341662 <== mon.2 v2:172.21.15.189:3300/0 2 ==== osd_map(90..90 src has 1..90) v4 ==== 8277+0+0 (secure 0 0 0) 0x7f3a0c030420 con 0x564d253ef050 2024-03-24T00:18:04.445 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:04.442+0000 7f3a04c00640 1 -- 172.21.15.118:0/1862341662 <== mon.2 v2:172.21.15.189:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) v2 ==== 24+0+0 (secure 0 0 0) 0x7f3a0c030660 con 0x564d253ef050 2024-03-24T00:18:04.445 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:04.442+0000 7f3a245f0740 10 client.26141 fetch_fsmap learned FSMap version 28 2024-03-24T00:18:04.445 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:04.442+0000 7f3a245f0740 1 -- 172.21.15.118:0/1862341662 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) v3 -- 0x564d254693a0 con 0x564d253ef050 2024-03-24T00:18:04.446 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:04.442+0000 7f3a04c00640 1 -- 172.21.15.118:0/1862341662 <== mon.2 v2:172.21.15.189:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x564d2535bae0 con 0x564d253ef050 2024-03-24T00:18:04.446 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:04.443+0000 7f3a04c00640 1 -- 172.21.15.118:0/1862341662 <== mon.2 v2:172.21.15.189:3300/0 5 ==== fsmap(e 28) v1 ==== 7934+0+0 (secure 0 0 0) 0x7f3a0c03cb20 con 0x564d253ef050 2024-03-24T00:18:04.446 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:04.443+0000 7f3a245f0740 10 client.26141 fetch_fsmap finished waiting for FSMap version 28 2024-03-24T00:18:04.446 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:04.443+0000 7f3a245f0740 10 client.26141 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-24T00:18:04.446 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:04.443+0000 7f3a245f0740 20 client.26141 populate_metadata read hostname 'smithi118' 2024-03-24T00:18:04.446 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:04.443+0000 7f3a245f0740 1 --2- 172.21.15.118:0/1862341662 >> [v2:172.21.15.189:6838/4074484168,v1:172.21.15.189:6839/4074484168] conn(0x564d25469c40 0x564d2546c060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:18:04.446 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:04.443+0000 7f3a245f0740 4 client.26141 mds_command: new command op to 24569 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-24T00:18:04.446 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:04.443+0000 7f3a245f0740 1 -- 172.21.15.118:0/1862341662 --> [v2:172.21.15.189:6838/4074484168,v1:172.21.15.189:6839/4074484168] -- command(tid 0: {"prefix": "scrub status"}) v1 -- 0x564d252d93a0 con 0x564d25469c40 2024-03-24T00:18:04.446 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:04.443+0000 7f3a1a000640 1 --2- 172.21.15.118:0/1862341662 >> [v2:172.21.15.189:6838/4074484168,v1:172.21.15.189:6839/4074484168] conn(0x564d25469c40 0x564d2546c060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:18:04.446 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:04.443+0000 7f3a1a000640 1 --2- 172.21.15.118:0/1862341662 >> [v2:172.21.15.189:6838/4074484168,v1:172.21.15.189:6839/4074484168] conn(0x564d25469c40 0x564d2546c060 crc :-1 s=READY pgs=88 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=2d5879d0e1e9db3a server_cookie=afdb1f4af65d2e41 in_seq=0 out_seq=0 2024-03-24T00:18:04.446 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:04.443+0000 7f3a04c00640 10 client.26141 ms_handle_connect on v2:172.21.15.189:6838/4074484168 2024-03-24T00:18:04.447 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:04.446+0000 7f3a04c00640 1 -- 172.21.15.118:0/1862341662 <== mds.0 v2:172.21.15.189:6838/4074484168 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+268 (crc 0 0 0) 0x564d252d93a0 con 0x564d25469c40 2024-03-24T00:18:04.447 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:04.446+0000 7f3a04c00640 10 client.26141 handle_command_reply: tid=0 2024-03-24T00:18:04.448 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:04.446+0000 7f3a245f0740 1 client.26141 shutdown 2024-03-24T00:18:04.448 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:04.446+0000 7f3a245f0740 1 -- 172.21.15.118:0/1862341662 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x564d253ef050 msgr2=0x564d2541fab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:18:04.448 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:04.446+0000 7f3a245f0740 1 --2- 172.21.15.118:0/1862341662 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x564d253ef050 0x564d2541fab0 secure :-1 s=READY pgs=329 cs=0 l=1 rev1=1 crypto rx=0x7f3a0c0041b0 tx=0x7f3a0c017000 comp rx=0 tx=0).stop 2024-03-24T00:18:04.448 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:04.446+0000 7f3a245f0740 1 -- 172.21.15.118:0/1862341662 shutdown_connections 2024-03-24T00:18:04.448 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:04.446+0000 7f3a245f0740 1 --2- 172.21.15.118:0/1862341662 >> [v2:172.21.15.189:6838/4074484168,v1:172.21.15.189:6839/4074484168] conn(0x564d25469c40 0x564d2546c060 crc :-1 s=READY pgs=88 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:18:04.448 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:04.446+0000 7f3a245f0740 1 --2- 172.21.15.118:0/1862341662 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x564d25406f70 0x564d2541e140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:18:04.448 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:04.446+0000 7f3a245f0740 1 --2- 172.21.15.118:0/1862341662 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x564d253efa40 0x564d2541fff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:18:04.448 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:04.446+0000 7f3a245f0740 1 --2- 172.21.15.118:0/1862341662 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x564d253ef050 0x564d2541fab0 unknown :-1 s=CLOSED pgs=329 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:18:04.448 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:04.446+0000 7f3a04c00640 0 client.26141 ms_handle_reset on v2:172.21.15.189:6838/4074484168 2024-03-24T00:18:04.448 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:04.446+0000 7f3a245f0740 1 -- 172.21.15.118:0/1862341662 >> 172.21.15.118:0/1862341662 conn(0x564d253b7200 msgr2=0x564d253ee490 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:18:04.448 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:04.446+0000 7f3a245f0740 1 -- 172.21.15.118:0/1862341662 shutdown_connections 2024-03-24T00:18:04.448 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:04.446+0000 7f3a245f0740 1 -- 172.21.15.118:0/1862341662 wait complete. 2024-03-24T00:18:04.448 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:04.446+0000 7f3a245f0740 20 client.26141 trim_cache size 0 max 16384 2024-03-24T00:18:04.449 INFO:teuthology.orchestra.run.smithi118.stdout:{ 2024-03-24T00:18:04.449 INFO:teuthology.orchestra.run.smithi118.stdout: "status": "scrub active (1052 inodes in the stack)", 2024-03-24T00:18:04.449 INFO:teuthology.orchestra.run.smithi118.stdout: "scrubs": { 2024-03-24T00:18:04.449 INFO:teuthology.orchestra.run.smithi118.stdout: "990e5fc3-bf40-49c5-bd80-e6ac9038901c": { 2024-03-24T00:18:04.449 INFO:teuthology.orchestra.run.smithi118.stdout: "path": "/", 2024-03-24T00:18:04.449 INFO:teuthology.orchestra.run.smithi118.stdout: "tag": "990e5fc3-bf40-49c5-bd80-e6ac9038901c", 2024-03-24T00:18:04.449 INFO:teuthology.orchestra.run.smithi118.stdout: "options": "recursive,force" 2024-03-24T00:18:04.449 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-03-24T00:18:04.449 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-03-24T00:18:04.449 INFO:teuthology.orchestra.run.smithi118.stdout:} 2024-03-24T00:18:04.450 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:04.447+0000 7f3a1be00640 1 -- 172.21.15.118:0/1138875809 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7f3a000a28e0 msgr2=0x7f3a000a4da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:18:04.450 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:04.447+0000 7f3a1be00640 1 --2- 172.21.15.118:0/1138875809 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7f3a000a28e0 0x7f3a000a4da0 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 crypto rx=0x564d253eb910 tx=0x7f3a0c00b2e0 comp rx=0 tx=0).stop 2024-03-24T00:18:04.450 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:04.447+0000 7f3a1be00640 1 -- 172.21.15.118:0/1138875809 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f3a1c104930 msgr2=0x7f3a1c0fffb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:18:04.450 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:04.447+0000 7f3a1be00640 1 --2- 172.21.15.118:0/1138875809 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f3a1c104930 0x7f3a1c0fffb0 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 crypto rx=0x7f3a08000c00 tx=0x7f3a0802c6c0 comp rx=0 tx=0).stop 2024-03-24T00:18:04.450 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:04.447+0000 7f3a1be00640 1 -- 172.21.15.118:0/1138875809 shutdown_connections 2024-03-24T00:18:04.450 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:04.447+0000 7f3a1be00640 1 --2- 172.21.15.118:0/1138875809 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7f3a000a28e0 0x7f3a000a4da0 unknown :-1 s=CLOSED pgs=316 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:18:04.450 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:04.447+0000 7f3a1be00640 1 --2- 172.21.15.118:0/1138875809 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f3a1c107260 0x7f3a1c1004f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:18:04.450 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:04.447+0000 7f3a1be00640 1 --2- 172.21.15.118:0/1138875809 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f3a1c104930 0x7f3a1c0fffb0 unknown :-1 s=CLOSED pgs=322 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:18:04.451 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:04.447+0000 7f3a1be00640 1 --2- 172.21.15.118:0/1138875809 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3a1c102000 0x7f3a1c101970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:18:04.451 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:04.447+0000 7f3a1be00640 1 -- 172.21.15.118:0/1138875809 >> 172.21.15.118:0/1138875809 conn(0x7f3a1c0fba10 msgr2=0x7f3a1c102a60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:18:04.451 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:04.448+0000 7f3a1be00640 1 -- 172.21.15.118:0/1138875809 shutdown_connections 2024-03-24T00:18:04.451 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:04.448+0000 7f3a1be00640 1 -- 172.21.15.118:0/1138875809 wait complete. 2024-03-24T00:18:04.464 INFO:tasks.cephfs.filesystem:scrub status for tag:990e5fc3-bf40-49c5-bd80-e6ac9038901c - {'path': '/', 'tag': '990e5fc3-bf40-49c5-bd80-e6ac9038901c', 'options': 'recursive,force'} 2024-03-24T00:18:05.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:18:04 smithi118 ceph-mon[31535]: pgmap v469: 97 pgs: 97 active+clean; 21 MiB data, 419 MiB used, 1.0 TiB / 1.0 TiB avail; 575 KiB/s rd, 1.0 MiB/s wr, 220 op/s 2024-03-24T00:18:05.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:18:04 smithi118 ceph-mon[31535]: scrub queued for path: / 2024-03-24T00:18:05.197 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:18:04 smithi118 ceph-mon[31535]: scrub summary: idle+waiting paths [/] 2024-03-24T00:18:05.197 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:18:04 smithi118 ceph-mon[31535]: scrub summary: active paths [/] 2024-03-24T00:18:05.203 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:18:04 smithi189 ceph-mon[36187]: pgmap v469: 97 pgs: 97 active+clean; 21 MiB data, 419 MiB used, 1.0 TiB / 1.0 TiB avail; 575 KiB/s rd, 1.0 MiB/s wr, 220 op/s 2024-03-24T00:18:05.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:18:04 smithi189 ceph-mon[36187]: scrub queued for path: / 2024-03-24T00:18:05.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:18:04 smithi189 ceph-mon[36187]: scrub summary: idle+waiting paths [/] 2024-03-24T00:18:05.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:18:04 smithi189 ceph-mon[36187]: scrub summary: active paths [/] 2024-03-24T00:18:05.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:18:04 smithi193 ceph-mon[36878]: pgmap v469: 97 pgs: 97 active+clean; 21 MiB data, 419 MiB used, 1.0 TiB / 1.0 TiB avail; 575 KiB/s rd, 1.0 MiB/s wr, 220 op/s 2024-03-24T00:18:05.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:18:04 smithi193 ceph-mon[36878]: scrub queued for path: / 2024-03-24T00:18:05.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:18:04 smithi193 ceph-mon[36878]: scrub summary: idle+waiting paths [/] 2024-03-24T00:18:05.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:18:04 smithi193 ceph-mon[36878]: scrub summary: active paths [/] 2024-03-24T00:18:06.834 INFO:tasks.workunit.client.0.smithi118.stdout: 7 82213 154 45483 163 40153 355 2024-03-24T00:18:07.195 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:18:06 smithi118 ceph-mon[31535]: scrub summary: idle+waiting paths [/] 2024-03-24T00:18:07.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:18:06 smithi118 ceph-mon[31535]: pgmap v470: 97 pgs: 97 active+clean; 31 MiB data, 439 MiB used, 1.0 TiB / 1.0 TiB avail; 646 KiB/s rd, 1.8 MiB/s wr, 331 op/s 2024-03-24T00:18:07.203 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:18:06 smithi189 ceph-mon[36187]: scrub summary: idle+waiting paths [/] 2024-03-24T00:18:07.203 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:18:06 smithi189 ceph-mon[36187]: pgmap v470: 97 pgs: 97 active+clean; 31 MiB data, 439 MiB used, 1.0 TiB / 1.0 TiB avail; 646 KiB/s rd, 1.8 MiB/s wr, 331 op/s 2024-03-24T00:18:07.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:18:06 smithi193 ceph-mon[36878]: scrub summary: idle+waiting paths [/] 2024-03-24T00:18:07.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:18:06 smithi193 ceph-mon[36878]: pgmap v470: 97 pgs: 97 active+clean; 31 MiB data, 439 MiB used, 1.0 TiB / 1.0 TiB avail; 646 KiB/s rd, 1.8 MiB/s wr, 331 op/s 2024-03-24T00:18:08.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:18:07 smithi118 ceph-mon[31535]: pgmap v471: 97 pgs: 97 active+clean; 36 MiB data, 454 MiB used, 1.0 TiB / 1.0 TiB avail; 751 KiB/s rd, 2.3 MiB/s wr, 429 op/s 2024-03-24T00:18:08.203 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:18:07 smithi189 ceph-mon[36187]: pgmap v471: 97 pgs: 97 active+clean; 36 MiB data, 454 MiB used, 1.0 TiB / 1.0 TiB avail; 751 KiB/s rd, 2.3 MiB/s wr, 429 op/s 2024-03-24T00:18:08.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:18:07 smithi193 ceph-mon[36878]: pgmap v471: 97 pgs: 97 active+clean; 36 MiB data, 454 MiB used, 1.0 TiB / 1.0 TiB avail; 751 KiB/s rd, 2.3 MiB/s wr, 429 op/s 2024-03-24T00:18:11.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:18:10 smithi118 ceph-mon[31535]: pgmap v472: 97 pgs: 97 active+clean; 36 MiB data, 457 MiB used, 1.0 TiB / 1.0 TiB avail; 746 KiB/s rd, 2.2 MiB/s wr, 454 op/s 2024-03-24T00:18:11.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:18:10 smithi118 ceph-mon[31535]: scrub summary: idle 2024-03-24T00:18:11.203 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:18:10 smithi189 ceph-mon[36187]: pgmap v472: 97 pgs: 97 active+clean; 36 MiB data, 457 MiB used, 1.0 TiB / 1.0 TiB avail; 746 KiB/s rd, 2.2 MiB/s wr, 454 op/s 2024-03-24T00:18:11.203 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:18:10 smithi189 ceph-mon[36187]: scrub summary: idle 2024-03-24T00:18:11.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:18:10 smithi193 ceph-mon[36878]: pgmap v472: 97 pgs: 97 active+clean; 36 MiB data, 457 MiB used, 1.0 TiB / 1.0 TiB avail; 746 KiB/s rd, 2.2 MiB/s wr, 454 op/s 2024-03-24T00:18:11.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:18:10 smithi193 ceph-mon[36878]: scrub summary: idle 2024-03-24T00:18:13.195 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:18:12 smithi118 ceph-mon[31535]: pgmap v473: 97 pgs: 97 active+clean; 36 MiB data, 457 MiB used, 1.0 TiB / 1.0 TiB avail; 748 KiB/s rd, 2.2 MiB/s wr, 454 op/s 2024-03-24T00:18:13.203 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:18:12 smithi189 ceph-mon[36187]: pgmap v473: 97 pgs: 97 active+clean; 36 MiB data, 457 MiB used, 1.0 TiB / 1.0 TiB avail; 748 KiB/s rd, 2.2 MiB/s wr, 454 op/s 2024-03-24T00:18:13.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:18:12 smithi193 ceph-mon[36878]: pgmap v473: 97 pgs: 97 active+clean; 36 MiB data, 457 MiB used, 1.0 TiB / 1.0 TiB avail; 748 KiB/s rd, 2.2 MiB/s wr, 454 op/s 2024-03-24T00:18:14.037 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:18:14.036+0000 7f741ce00640 1 -- 172.21.15.118:0/371484845 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f74181016d0 msgr2=0x7f7418101af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:18:14.037 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:18:14.036+0000 7f741ce00640 1 --2- 172.21.15.118:0/371484845 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f74181016d0 0x7f7418101af0 secure :-1 s=READY pgs=325 cs=0 l=1 rev1=1 crypto rx=0x7f7408009900 tx=0x7f740802bc30 comp rx=0 tx=0).stop 2024-03-24T00:18:14.037 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:18:14.036+0000 7f741ce00640 1 -- 172.21.15.118:0/371484845 shutdown_connections 2024-03-24T00:18:14.037 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:18:14.036+0000 7f741ce00640 1 --2- 172.21.15.118:0/371484845 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f7418103280 0x7f7418109b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:18:14.037 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:18:14.036+0000 7f741ce00640 1 --2- 172.21.15.118:0/371484845 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f74181028c0 0x7f7418102d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:18:14.037 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:18:14.036+0000 7f741ce00640 1 --2- 172.21.15.118:0/371484845 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f74181016d0 0x7f7418101af0 unknown :-1 s=CLOSED pgs=325 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:18:14.037 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:18:14.036+0000 7f741ce00640 1 -- 172.21.15.118:0/371484845 >> 172.21.15.118:0/371484845 conn(0x7f74180fcc60 msgr2=0x7f74180ff080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:18:14.037 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:18:14.036+0000 7f741ce00640 1 -- 172.21.15.118:0/371484845 shutdown_connections 2024-03-24T00:18:14.038 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:18:14.036+0000 7f741ce00640 1 -- 172.21.15.118:0/371484845 wait complete. 2024-03-24T00:18:14.040 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:18:14.037+0000 7f741ce00640 1 Processor -- start 2024-03-24T00:18:14.040 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:18:14.037+0000 7f741ce00640 1 -- start start 2024-03-24T00:18:14.040 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:18:14.037+0000 7f741ce00640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f74181016d0 0x7f7418199b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:18:14.040 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:18:14.037+0000 7f741ce00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f74181028c0 0x7f741819a060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:18:14.040 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:18:14.038+0000 7f741ce00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7418103280 0x7f74181a10d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:18:14.040 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:18:14.038+0000 7f741ce00640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f741810d370 con 0x7f74181016d0 2024-03-24T00:18:14.040 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:18:14.038+0000 7f741ce00640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f741810d1f0 con 0x7f7418103280 2024-03-24T00:18:14.040 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:18:14.038+0000 7f741ce00640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f741810d4f0 con 0x7f74181028c0 2024-03-24T00:18:14.040 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:18:14.038+0000 7f7415400640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f74181028c0 0x7f741819a060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:18:14.040 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:18:14.038+0000 7f7415e00640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f74181016d0 0x7f7418199b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:18:14.040 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:18:14.038+0000 7f7415400640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f74181028c0 0x7f741819a060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.118:42468/0 (socket says 172.21.15.118:42468) 2024-03-24T00:18:14.040 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:18:14.038+0000 7f7415400640 1 -- 172.21.15.118:0/608737549 learned_addr learned my addr 172.21.15.118:0/608737549 (peer_addr_for_me v2:172.21.15.118:0/0) 2024-03-24T00:18:14.041 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:18:14.038+0000 7f7416800640 1 --2- 172.21.15.118:0/608737549 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7418103280 0x7f74181a10d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:18:14.041 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:18:14.038+0000 7f7415400640 1 -- 172.21.15.118:0/608737549 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7418103280 msgr2=0x7f74181a10d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:18:14.041 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:18:14.038+0000 7f7415400640 1 --2- 172.21.15.118:0/608737549 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7418103280 0x7f74181a10d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:18:14.041 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:18:14.038+0000 7f7415400640 1 -- 172.21.15.118:0/608737549 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f74181016d0 msgr2=0x7f7418199b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:18:14.041 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:18:14.038+0000 7f7415400640 1 --2- 172.21.15.118:0/608737549 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f74181016d0 0x7f7418199b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:18:14.041 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:18:14.038+0000 7f7415400640 1 -- 172.21.15.118:0/608737549 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f74181a1740 con 0x7f74181028c0 2024-03-24T00:18:14.041 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:18:14.039+0000 7f7415400640 1 --2- 172.21.15.118:0/608737549 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f74181028c0 0x7f741819a060 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 crypto rx=0x7f740400e9f0 tx=0x7f740400eeb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:18:14.041 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:18:14.039+0000 7f740ea00640 1 -- 172.21.15.118:0/608737549 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f740400cd80 con 0x7f74181028c0 2024-03-24T00:18:14.041 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:18:14.039+0000 7f740ea00640 1 -- 172.21.15.118:0/608737549 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f7404010910 con 0x7f74181028c0 2024-03-24T00:18:14.041 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:18:14.039+0000 7f741ce00640 1 -- 172.21.15.118:0/608737549 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f74181a1a30 con 0x7f74181028c0 2024-03-24T00:18:14.041 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:18:14.039+0000 7f740ea00640 1 -- 172.21.15.118:0/608737549 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7404018980 con 0x7f74181028c0 2024-03-24T00:18:14.041 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:18:14.039+0000 7f741ce00640 1 -- 172.21.15.118:0/608737549 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f74181a1fd0 con 0x7f74181028c0 2024-03-24T00:18:14.044 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:18:14.043+0000 7f740ea00640 1 -- 172.21.15.118:0/608737549 <== mon.1 v2:172.21.15.193:3300/0 4 ==== mgrmap(e 20) v1 ==== 137291+0+0 (secure 0 0 0) 0x7f7404010ac0 con 0x7f74181028c0 2024-03-24T00:18:14.044 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:18:14.043+0000 7f741fc09740 1 Processor -- start 2024-03-24T00:18:14.044 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:18:14.043+0000 7f741fc09740 1 -- start start 2024-03-24T00:18:14.046 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:18:14.044+0000 7f741fc09740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5632b25521e0 0x5632b2552600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:18:14.046 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:18:14.044+0000 7f741fc09740 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5632b2552bd0 0x5632b25576d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:18:14.046 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:18:14.044+0000 7f741fc09740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5632b2557d40 0x5632b255a1d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:18:14.046 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:18:14.044+0000 7f741fc09740 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x5632b25153b0 con 0x5632b2552bd0 2024-03-24T00:18:14.046 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:18:14.044+0000 7f741fc09740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x5632b24b7830 con 0x5632b25521e0 2024-03-24T00:18:14.046 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:18:14.044+0000 7f741fc09740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x5632b24605d0 con 0x5632b2557d40 2024-03-24T00:18:14.046 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:18:14.044+0000 7f7416800640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5632b25521e0 0x5632b2552600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:18:14.046 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:18:14.044+0000 7f7416800640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5632b25521e0 0x5632b2552600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.118:38722/0 (socket says 172.21.15.118:38722) 2024-03-24T00:18:14.047 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:18:14.044+0000 7f7415400640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5632b2557d40 0x5632b255a1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:18:14.047 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:18:14.044+0000 7f7415e00640 1 --2- 172.21.15.118:0/1377964367 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5632b2552bd0 0x5632b25576d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:18:14.047 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:18:14.044+0000 7f7416800640 1 -- 172.21.15.118:0/1377964367 learned_addr learned my addr 172.21.15.118:0/1377964367 (peer_addr_for_me v2:172.21.15.118:0/0) 2024-03-24T00:18:14.047 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:18:14.044+0000 7f740ea00640 1 --2- 172.21.15.118:0/608737549 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7f73fc0a2920 0x7f73fc0a4de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:18:14.047 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:18:14.045+0000 7f7415400640 1 -- 172.21.15.118:0/1377964367 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5632b25521e0 msgr2=0x5632b2552600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:18:14.047 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:18:14.045+0000 7f7415400640 1 --2- 172.21.15.118:0/1377964367 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5632b25521e0 0x5632b2552600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:18:14.047 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:18:14.045+0000 7f7415400640 1 -- 172.21.15.118:0/1377964367 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5632b2552bd0 msgr2=0x5632b25576d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:18:14.047 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:18:14.045+0000 7f7415400640 1 --2- 172.21.15.118:0/1377964367 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5632b2552bd0 0x5632b25576d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:18:14.047 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:18:14.045+0000 7f7415400640 1 -- 172.21.15.118:0/1377964367 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5632b24782c0 con 0x5632b2557d40 2024-03-24T00:18:14.047 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:18:14.045+0000 7f7416800640 1 --2- 172.21.15.118:0/1377964367 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5632b25521e0 0x5632b2552600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T00:18:14.047 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:18:14.045+0000 7f7415e00640 1 --2- 172.21.15.118:0/1377964367 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5632b2552bd0 0x5632b25576d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T00:18:14.047 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:18:14.045+0000 7f740ea00640 1 -- 172.21.15.118:0/608737549 <== mon.1 v2:172.21.15.193:3300/0 5 ==== osd_map(90..90 src has 1..90) v4 ==== 8277+0+0 (secure 0 0 0) 0x7f74040ced70 con 0x7f74181028c0 2024-03-24T00:18:14.047 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:18:14.045+0000 7f7416800640 1 --2- 172.21.15.118:0/608737549 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7f73fc0a2920 0x7f73fc0a4de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:18:14.048 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:18:14.045+0000 7f7415400640 1 --2- 172.21.15.118:0/1377964367 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5632b2557d40 0x5632b255a1d0 secure :-1 s=READY pgs=327 cs=0 l=1 rev1=1 crypto rx=0x7f7404097b90 tx=0x7f740400cf20 comp rx=0 tx=0).ready entity=mon.1 client_cookie=94290cd65571dd06 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:18:14.048 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:18:14.045+0000 7f7416800640 1 --2- 172.21.15.118:0/608737549 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7f73fc0a2920 0x7f73fc0a4de0 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 crypto rx=0x5632b2553430 tx=0x7f7410004a40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:18:14.048 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:18:14.046+0000 7f7403400640 1 -- 172.21.15.118:0/1377964367 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f740400e130 con 0x5632b2557d40 2024-03-24T00:18:14.048 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:18:14.046+0000 7f7403400640 1 -- 172.21.15.118:0/1377964367 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f74040bb030 con 0x5632b2557d40 2024-03-24T00:18:14.048 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:18:14.046+0000 7f7403400640 1 -- 172.21.15.118:0/1377964367 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7404097e30 con 0x5632b2557d40 2024-03-24T00:18:14.051 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:18:14.046+0000 7f741fc09740 1 -- 172.21.15.118:0/1377964367 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5632b2557d40 msgr2=0x5632b255a1d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:18:14.051 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:18:14.046+0000 7f741fc09740 1 --2- 172.21.15.118:0/1377964367 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5632b2557d40 0x5632b255a1d0 secure :-1 s=READY pgs=327 cs=0 l=1 rev1=1 crypto rx=0x7f7404097b90 tx=0x7f740400cf20 comp rx=0 tx=0).stop 2024-03-24T00:18:14.051 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:18:14.046+0000 7f741fc09740 1 -- 172.21.15.118:0/1377964367 shutdown_connections 2024-03-24T00:18:14.051 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:18:14.046+0000 7f741fc09740 1 --2- 172.21.15.118:0/1377964367 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5632b2557d40 0x5632b255a1d0 unknown :-1 s=CLOSED pgs=327 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:18:14.051 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:18:14.046+0000 7f741fc09740 1 --2- 172.21.15.118:0/1377964367 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5632b2552bd0 0x5632b25576d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:18:14.051 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:18:14.046+0000 7f741fc09740 1 --2- 172.21.15.118:0/1377964367 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5632b25521e0 0x5632b2552600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:18:14.051 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:18:14.046+0000 7f741fc09740 1 -- 172.21.15.118:0/1377964367 >> 172.21.15.118:0/1377964367 conn(0x5632b254d750 msgr2=0x5632b254db50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:18:14.051 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:18:14.046+0000 7f741fc09740 1 -- 172.21.15.118:0/1377964367 shutdown_connections 2024-03-24T00:18:14.051 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:18:14.047+0000 7f741fc09740 1 -- 172.21.15.118:0/1377964367 wait complete. 2024-03-24T00:18:14.052 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:18:14.047+0000 7f741fc09740 1 -- start start 2024-03-24T00:18:14.052 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:18:14.047+0000 7f741fc09740 1 Processor -- start 2024-03-24T00:18:14.052 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:18:14.048+0000 7f741fc09740 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5632b25521e0 0x5632b255c300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:18:14.052 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:18:14.048+0000 7f741fc09740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5632b2552bd0 0x5632b255c840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:18:14.052 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:18:14.048+0000 7f741fc09740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5632b2557d40 0x5632b2563880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:18:14.052 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:18:14.048+0000 7f741fc09740 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x5632b2473a20 con 0x5632b25521e0 2024-03-24T00:18:14.052 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:18:14.048+0000 7f741fc09740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x5632b24b38d0 con 0x5632b2557d40 2024-03-24T00:18:14.052 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:18:14.048+0000 7f741fc09740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x5632b255ac90 con 0x5632b2552bd0 2024-03-24T00:18:14.052 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:18:14.048+0000 7f741fc09740 10 client.0 _ll_register_callbacks cb 0x5632b24b01c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T00:18:14.052 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:18:14.048+0000 7f7415400640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5632b2552bd0 0x5632b255c840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:18:14.052 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:18:14.048+0000 7f7415e00640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5632b25521e0 0x5632b255c300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:18:14.053 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:18:14.048+0000 7f7415400640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5632b2552bd0 0x5632b255c840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.118:42484/0 (socket says 172.21.15.118:42484) 2024-03-24T00:18:14.053 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:18:14.048+0000 7f7415400640 1 -- 172.21.15.118:0/501424645 learned_addr learned my addr 172.21.15.118:0/501424645 (peer_addr_for_me v2:172.21.15.118:0/0) 2024-03-24T00:18:14.053 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:18:14.048+0000 7f7416800640 1 --2- 172.21.15.118:0/501424645 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5632b2557d40 0x5632b2563880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:18:14.053 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:18:14.048+0000 7f7415400640 1 -- 172.21.15.118:0/501424645 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5632b2557d40 msgr2=0x5632b2563880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:18:14.053 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:18:14.048+0000 7f7415400640 1 --2- 172.21.15.118:0/501424645 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5632b2557d40 0x5632b2563880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:18:14.053 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:18:14.048+0000 7f7415400640 1 -- 172.21.15.118:0/501424645 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5632b25521e0 msgr2=0x5632b255c300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:18:14.053 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:18:14.048+0000 7f7415400640 1 --2- 172.21.15.118:0/501424645 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5632b25521e0 0x5632b255c300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:18:14.053 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:18:14.048+0000 7f7415400640 1 -- 172.21.15.118:0/501424645 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5632b21cc210 con 0x5632b2552bd0 2024-03-24T00:18:14.053 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:18:14.049+0000 7f7415400640 1 --2- 172.21.15.118:0/501424645 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5632b2552bd0 0x5632b255c840 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 crypto rx=0x7f740400c6a0 tx=0x7f74040d2560 comp rx=0 tx=0).ready entity=mon.1 client_cookie=8b3154987a8ed7e7 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:18:14.053 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:18:14.049+0000 7f73f7e00640 10 client.0 ms_handle_connect on v2:172.21.15.193:3300/0 2024-03-24T00:18:14.053 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:18:14.049+0000 7f73f7e00640 1 -- 172.21.15.118:0/501424645 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f740409f7d0 con 0x5632b2552bd0 2024-03-24T00:18:14.054 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:18:14.049+0000 7f73f7e00640 1 -- 172.21.15.118:0/501424645 <== mon.1 v2:172.21.15.193:3300/0 2 ==== osd_map(90..90 src has 1..90) v4 ==== 8277+0+0 (secure 0 0 0) 0x7f74040cc8f0 con 0x5632b2552bd0 2024-03-24T00:18:14.054 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:18:14.049+0000 7f741fc09740 1 -- 172.21.15.118:0/501424645 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f74040bb030 con 0x5632b2552bd0 2024-03-24T00:18:14.054 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:18:14.049+0000 7f73f7e00640 1 -- 172.21.15.118:0/501424645 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) v2 ==== 24+0+0 (secure 0 0 0) 0x7f7404097e30 con 0x5632b2552bd0 2024-03-24T00:18:14.054 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:18:14.049+0000 7f741fc09740 10 client.26161 fetch_fsmap learned FSMap version 28 2024-03-24T00:18:14.054 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:18:14.049+0000 7f741fc09740 1 -- 172.21.15.118:0/501424645 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) v3 -- 0x5632b2298620 con 0x5632b2552bd0 2024-03-24T00:18:14.054 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:18:14.050+0000 7f73f7e00640 1 -- 172.21.15.118:0/501424645 <== mon.1 v2:172.21.15.193:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f74040bb030 con 0x5632b2552bd0 2024-03-24T00:18:14.054 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:18:14.050+0000 7f73f7e00640 1 -- 172.21.15.118:0/501424645 <== mon.1 v2:172.21.15.193:3300/0 5 ==== fsmap(e 28) v1 ==== 7934+0+0 (secure 0 0 0) 0x7f7404010430 con 0x5632b2552bd0 2024-03-24T00:18:14.054 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:18:14.050+0000 7f741fc09740 10 client.26161 fetch_fsmap finished waiting for FSMap version 28 2024-03-24T00:18:14.054 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:18:14.050+0000 7f741fc09740 10 client.26161 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-24T00:18:14.054 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:18:14.050+0000 7f741fc09740 20 client.26161 populate_metadata read hostname 'smithi118' 2024-03-24T00:18:14.054 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:18:14.050+0000 7f741fc09740 1 --2- 172.21.15.118:0/501424645 >> [v2:172.21.15.189:6838/4074484168,v1:172.21.15.189:6839/4074484168] conn(0x5632b2567d70 0x5632b256a160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:18:14.055 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:18:14.050+0000 7f741fc09740 4 client.26161 mds_command: new command op to 24569 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-24T00:18:14.055 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:18:14.050+0000 7f741fc09740 1 -- 172.21.15.118:0/501424645 --> [v2:172.21.15.189:6838/4074484168,v1:172.21.15.189:6839/4074484168] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x5632b24c4d70 con 0x5632b2567d70 2024-03-24T00:18:14.055 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:18:14.050+0000 7f7415e00640 1 --2- 172.21.15.118:0/501424645 >> [v2:172.21.15.189:6838/4074484168,v1:172.21.15.189:6839/4074484168] conn(0x5632b2567d70 0x5632b256a160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:18:14.056 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:18:14.051+0000 7f7415e00640 1 --2- 172.21.15.118:0/501424645 >> [v2:172.21.15.189:6838/4074484168,v1:172.21.15.189:6839/4074484168] conn(0x5632b2567d70 0x5632b256a160 crc :-1 s=READY pgs=89 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=43ea3072787de4dc server_cookie=90204e2e315a82ad in_seq=0 out_seq=0 2024-03-24T00:18:14.056 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:18:14.051+0000 7f73f7e00640 10 client.26161 ms_handle_connect on v2:172.21.15.189:6838/4074484168 2024-03-24T00:18:14.056 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:18:14.052+0000 7f73f7e00640 1 -- 172.21.15.118:0/501424645 <== mds.0 v2:172.21.15.189:6838/4074484168 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x5632b24c4d70 con 0x5632b2567d70 2024-03-24T00:18:14.056 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:18:14.052+0000 7f73f7e00640 10 client.26161 handle_command_reply: tid=0 2024-03-24T00:18:14.056 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:18:14.052+0000 7f741fc09740 1 client.26161 shutdown 2024-03-24T00:18:14.056 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:18:14.052+0000 7f741fc09740 1 -- 172.21.15.118:0/501424645 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5632b2552bd0 msgr2=0x5632b255c840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:18:14.056 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:18:14.052+0000 7f741fc09740 1 --2- 172.21.15.118:0/501424645 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5632b2552bd0 0x5632b255c840 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 crypto rx=0x7f740400c6a0 tx=0x7f74040d2560 comp rx=0 tx=0).stop 2024-03-24T00:18:14.056 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:18:14.053+0000 7f741fc09740 1 -- 172.21.15.118:0/501424645 shutdown_connections 2024-03-24T00:18:14.056 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:18:14.053+0000 7f741fc09740 1 --2- 172.21.15.118:0/501424645 >> [v2:172.21.15.189:6838/4074484168,v1:172.21.15.189:6839/4074484168] conn(0x5632b2567d70 0x5632b256a160 crc :-1 s=READY pgs=89 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:18:14.057 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:18:14.053+0000 7f741fc09740 1 --2- 172.21.15.118:0/501424645 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5632b2557d40 0x5632b2563880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:18:14.057 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:18:14.053+0000 7f741fc09740 1 --2- 172.21.15.118:0/501424645 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5632b2552bd0 0x5632b255c840 unknown :-1 s=CLOSED pgs=328 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:18:14.057 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:18:14.053+0000 7f741fc09740 1 --2- 172.21.15.118:0/501424645 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5632b25521e0 0x5632b255c300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:18:14.057 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:18:14.053+0000 7f73f7e00640 0 client.26161 ms_handle_reset on v2:172.21.15.189:6838/4074484168 2024-03-24T00:18:14.057 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:18:14.053+0000 7f741fc09740 1 -- 172.21.15.118:0/501424645 >> 172.21.15.118:0/501424645 conn(0x5632b254d750 msgr2=0x5632b254c950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:18:14.057 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:18:14.053+0000 7f741fc09740 1 -- 172.21.15.118:0/501424645 shutdown_connections 2024-03-24T00:18:14.057 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:18:14.054+0000 7f741fc09740 1 -- 172.21.15.118:0/501424645 wait complete. 2024-03-24T00:18:14.057 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:18:14.054+0000 7f741fc09740 20 client.26161 trim_cache size 0 max 16384 2024-03-24T00:18:14.076 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:18:14.075+0000 7f741fc09740 1 Processor -- start 2024-03-24T00:18:14.076 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:18:14.075+0000 7f741fc09740 1 -- start start 2024-03-24T00:18:14.076 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:18:14.075+0000 7f741fc09740 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5632b25521e0 0x5632b2553e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:18:14.077 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:18:14.076+0000 7f741fc09740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5632b2552bd0 0x5632b25543d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:18:14.077 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:18:14.076+0000 7f741fc09740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5632b2557d40 0x5632b2563840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:18:14.077 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:18:14.076+0000 7f741fc09740 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f74181aabd0 con 0x5632b25521e0 2024-03-24T00:18:14.077 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:18:14.076+0000 7f741fc09740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f74181ac550 con 0x5632b2552bd0 2024-03-24T00:18:14.077 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:18:14.076+0000 7f741fc09740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f74181ab230 con 0x5632b2557d40 2024-03-24T00:18:14.077 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:18:14.076+0000 7f7416800640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5632b2557d40 0x5632b2563840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:18:14.077 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:18:14.076+0000 7f7415e00640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5632b25521e0 0x5632b2553e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:18:14.077 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:18:14.076+0000 7f7416800640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5632b2557d40 0x5632b2563840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.118:42498/0 (socket says 172.21.15.118:42498) 2024-03-24T00:18:14.077 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:18:14.076+0000 7f7416800640 1 -- 172.21.15.118:0/1998329126 learned_addr learned my addr 172.21.15.118:0/1998329126 (peer_addr_for_me v2:172.21.15.118:0/0) 2024-03-24T00:18:14.078 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:18:14.076+0000 7f7416800640 1 -- 172.21.15.118:0/1998329126 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5632b2552bd0 msgr2=0x5632b25543d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:18:14.078 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:18:14.076+0000 7f7416800640 1 --2- 172.21.15.118:0/1998329126 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5632b2552bd0 0x5632b25543d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:18:14.078 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:18:14.076+0000 7f7416800640 1 -- 172.21.15.118:0/1998329126 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5632b25521e0 msgr2=0x5632b2553e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:18:14.078 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:18:14.076+0000 7f7416800640 1 --2- 172.21.15.118:0/1998329126 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5632b25521e0 0x5632b2553e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:18:14.078 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:18:14.076+0000 7f7416800640 1 -- 172.21.15.118:0/1998329126 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5632b2552fb0 con 0x5632b2557d40 2024-03-24T00:18:14.078 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:18:14.077+0000 7f7416800640 1 --2- 172.21.15.118:0/1998329126 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5632b2557d40 0x5632b2563840 secure :-1 s=READY pgs=329 cs=0 l=1 rev1=1 crypto rx=0x7f7410005f40 tx=0x7f7410017000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=a6145b79ecb496d0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:18:14.078 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:18:14.077+0000 7f7402a00640 1 -- 172.21.15.118:0/1998329126 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f741001fbd0 con 0x5632b2557d40 2024-03-24T00:18:14.078 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:18:14.077+0000 7f7402a00640 1 -- 172.21.15.118:0/1998329126 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f741001fd70 con 0x5632b2557d40 2024-03-24T00:18:14.078 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:18:14.077+0000 7f7402a00640 1 -- 172.21.15.118:0/1998329126 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f741001c7d0 con 0x5632b2557d40 2024-03-24T00:18:14.079 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:18:14.078+0000 7f741fc09740 1 -- 172.21.15.118:0/1998329126 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5632b2557d40 msgr2=0x5632b2563840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:18:14.080 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:18:14.078+0000 7f741fc09740 1 --2- 172.21.15.118:0/1998329126 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5632b2557d40 0x5632b2563840 secure :-1 s=READY pgs=329 cs=0 l=1 rev1=1 crypto rx=0x7f7410005f40 tx=0x7f7410017000 comp rx=0 tx=0).stop 2024-03-24T00:18:14.080 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:18:14.078+0000 7f741fc09740 1 -- 172.21.15.118:0/1998329126 shutdown_connections 2024-03-24T00:18:14.080 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:18:14.078+0000 7f741fc09740 1 --2- 172.21.15.118:0/1998329126 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5632b2557d40 0x5632b2563840 unknown :-1 s=CLOSED pgs=329 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:18:14.080 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:18:14.078+0000 7f741fc09740 1 --2- 172.21.15.118:0/1998329126 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5632b2552bd0 0x5632b25543d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:18:14.080 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:18:14.078+0000 7f741fc09740 1 --2- 172.21.15.118:0/1998329126 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5632b25521e0 0x5632b2553e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:18:14.080 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:18:14.078+0000 7f741fc09740 1 -- 172.21.15.118:0/1998329126 >> 172.21.15.118:0/1998329126 conn(0x5632b254d750 msgr2=0x5632b255c300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:18:14.080 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:18:14.078+0000 7f741fc09740 1 -- 172.21.15.118:0/1998329126 shutdown_connections 2024-03-24T00:18:14.080 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:18:14.079+0000 7f741fc09740 1 -- 172.21.15.118:0/1998329126 wait complete. 2024-03-24T00:18:14.080 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:18:14.079+0000 7f741fc09740 1 -- start start 2024-03-24T00:18:14.080 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:18:14.079+0000 7f741fc09740 1 Processor -- start 2024-03-24T00:18:14.083 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:18:14.079+0000 7f741fc09740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5632b25521e0 0x5632b2561370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:18:14.083 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:18:14.079+0000 7f741fc09740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5632b2552bd0 0x5632b25618b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:18:14.083 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:18:14.079+0000 7f741fc09740 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5632b2557d40 0x5632b25628c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:18:14.083 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:18:14.079+0000 7f741fc09740 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f74181abef0 con 0x5632b2557d40 2024-03-24T00:18:14.083 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:18:14.079+0000 7f741fc09740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f74181ab890 con 0x5632b25521e0 2024-03-24T00:18:14.083 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:18:14.079+0000 7f741fc09740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f74181a8d90 con 0x5632b2552bd0 2024-03-24T00:18:14.083 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:18:14.080+0000 7f741fc09740 10 client.0 _ll_register_callbacks cb 0x7f7408009780 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T00:18:14.083 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:18:14.080+0000 7f7415400640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5632b2552bd0 0x5632b25618b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:18:14.083 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:18:14.080+0000 7f7415400640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5632b2552bd0 0x5632b25618b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.118:42504/0 (socket says 172.21.15.118:42504) 2024-03-24T00:18:14.083 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:18:14.080+0000 7f7415e00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5632b25521e0 0x5632b2561370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:18:14.083 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:18:14.080+0000 7f7415400640 1 -- 172.21.15.118:0/1795582352 learned_addr learned my addr 172.21.15.118:0/1795582352 (peer_addr_for_me v2:172.21.15.118:0/0) 2024-03-24T00:18:14.083 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:18:14.080+0000 7f7416800640 1 --2- 172.21.15.118:0/1795582352 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5632b2557d40 0x5632b25628c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:18:14.084 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:18:14.080+0000 7f7415400640 1 -- 172.21.15.118:0/1795582352 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5632b25521e0 msgr2=0x5632b2561370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:18:14.084 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:18:14.080+0000 7f7415400640 1 --2- 172.21.15.118:0/1795582352 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5632b25521e0 0x5632b2561370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:18:14.084 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:18:14.080+0000 7f7415400640 1 -- 172.21.15.118:0/1795582352 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5632b2557d40 msgr2=0x5632b25628c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:18:14.084 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:18:14.080+0000 7f7415400640 1 --2- 172.21.15.118:0/1795582352 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5632b2557d40 0x5632b25628c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:18:14.084 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:18:14.080+0000 7f7415400640 1 -- 172.21.15.118:0/1795582352 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5632b2562eb0 con 0x5632b2552bd0 2024-03-24T00:18:14.084 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:18:14.080+0000 7f7415e00640 1 --2- 172.21.15.118:0/1795582352 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5632b25521e0 0x5632b2561370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T00:18:14.084 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:18:14.080+0000 7f7415400640 1 --2- 172.21.15.118:0/1795582352 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5632b2552bd0 0x5632b25618b0 secure :-1 s=READY pgs=330 cs=0 l=1 rev1=1 crypto rx=0x7f740409b430 tx=0x7f7404005ab0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=aab930969d67cb50 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:18:14.084 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:18:14.080+0000 7f73f7e00640 10 client.0 ms_handle_connect on v2:172.21.15.193:3300/0 2024-03-24T00:18:14.084 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:18:14.081+0000 7f73f7e00640 1 -- 172.21.15.118:0/1795582352 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f740409ad80 con 0x5632b2552bd0 2024-03-24T00:18:14.084 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:18:14.081+0000 7f73f7e00640 1 -- 172.21.15.118:0/1795582352 <== mon.1 v2:172.21.15.193:3300/0 2 ==== osd_map(90..90 src has 1..90) v4 ==== 8277+0+0 (secure 0 0 0) 0x7f7404010430 con 0x5632b2552bd0 2024-03-24T00:18:14.084 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:18:14.081+0000 7f741fc09740 1 -- 172.21.15.118:0/1795582352 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f741001fd70 con 0x5632b2552bd0 2024-03-24T00:18:14.084 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:18:14.081+0000 7f73f7e00640 1 -- 172.21.15.118:0/1795582352 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) v2 ==== 24+0+0 (secure 0 0 0) 0x7f7404081050 con 0x5632b2552bd0 2024-03-24T00:18:14.084 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:18:14.081+0000 7f741fc09740 10 client.26167 fetch_fsmap learned FSMap version 28 2024-03-24T00:18:14.084 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:18:14.081+0000 7f741fc09740 1 -- 172.21.15.118:0/1795582352 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) v3 -- 0x5632b2562eb0 con 0x5632b2552bd0 2024-03-24T00:18:14.085 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:18:14.081+0000 7f73f7e00640 1 -- 172.21.15.118:0/1795582352 <== mon.1 v2:172.21.15.193:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f741001fd70 con 0x5632b2552bd0 2024-03-24T00:18:14.085 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:18:14.081+0000 7f73f7e00640 1 -- 172.21.15.118:0/1795582352 <== mon.1 v2:172.21.15.193:3300/0 5 ==== fsmap(e 28) v1 ==== 7934+0+0 (secure 0 0 0) 0x7f7404097480 con 0x5632b2552bd0 2024-03-24T00:18:14.085 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:18:14.081+0000 7f741fc09740 10 client.26167 fetch_fsmap finished waiting for FSMap version 28 2024-03-24T00:18:14.085 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:18:14.081+0000 7f741fc09740 10 client.26167 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-24T00:18:14.085 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:18:14.081+0000 7f741fc09740 20 client.26167 populate_metadata read hostname 'smithi118' 2024-03-24T00:18:14.085 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:18:14.081+0000 7f741fc09740 1 --2- 172.21.15.118:0/1795582352 >> [v2:172.21.15.189:6838/4074484168,v1:172.21.15.189:6839/4074484168] conn(0x5632b25c3fb0 0x5632b25c63a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:18:14.085 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:18:14.081+0000 7f741fc09740 4 client.26167 mds_command: new command op to 24569 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-03-24T00:18:14.085 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:18:14.081+0000 7f741fc09740 1 -- 172.21.15.118:0/1795582352 --> [v2:172.21.15.189:6838/4074484168,v1:172.21.15.189:6839/4074484168] -- command(tid 0: {"prefix": "flush journal"}) v1 -- 0x5632b24bf0c0 con 0x5632b25c3fb0 2024-03-24T00:18:14.085 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:18:14.082+0000 7f7415e00640 1 --2- 172.21.15.118:0/1795582352 >> [v2:172.21.15.189:6838/4074484168,v1:172.21.15.189:6839/4074484168] conn(0x5632b25c3fb0 0x5632b25c63a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:18:14.085 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:18:14.082+0000 7f7415e00640 1 --2- 172.21.15.118:0/1795582352 >> [v2:172.21.15.189:6838/4074484168,v1:172.21.15.189:6839/4074484168] conn(0x5632b25c3fb0 0x5632b25c63a0 crc :-1 s=READY pgs=90 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=14e7f236bbfe7fff server_cookie=235156acd4df433 in_seq=0 out_seq=0 2024-03-24T00:18:14.085 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:18:14.082+0000 7f73f7e00640 10 client.26167 ms_handle_connect on v2:172.21.15.189:6838/4074484168 2024-03-24T00:18:14.382 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:18:14.381+0000 7f73f7e00640 1 -- 172.21.15.118:0/1795582352 <== mds.0 v2:172.21.15.189:6838/4074484168 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+44 (crc 0 0 0) 0x5632b24bf0c0 con 0x5632b25c3fb0 2024-03-24T00:18:14.382 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:18:14.381+0000 7f73f7e00640 10 client.26167 handle_command_reply: tid=0 2024-03-24T00:18:14.382 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:18:14.381+0000 7f741fc09740 1 client.26167 shutdown 2024-03-24T00:18:14.382 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:18:14.381+0000 7f741fc09740 1 -- 172.21.15.118:0/1795582352 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5632b2552bd0 msgr2=0x5632b25618b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:18:14.382 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:18:14.381+0000 7f741fc09740 1 --2- 172.21.15.118:0/1795582352 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5632b2552bd0 0x5632b25618b0 secure :-1 s=READY pgs=330 cs=0 l=1 rev1=1 crypto rx=0x7f740409b430 tx=0x7f7404005ab0 comp rx=0 tx=0).stop 2024-03-24T00:18:14.383 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:18:14.381+0000 7f741fc09740 1 -- 172.21.15.118:0/1795582352 shutdown_connections 2024-03-24T00:18:14.383 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:18:14.382+0000 7f741fc09740 1 --2- 172.21.15.118:0/1795582352 >> [v2:172.21.15.189:6838/4074484168,v1:172.21.15.189:6839/4074484168] conn(0x5632b25c3fb0 0x5632b25c63a0 crc :-1 s=READY pgs=90 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:18:14.383 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:18:14.382+0000 7f741fc09740 1 --2- 172.21.15.118:0/1795582352 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5632b2557d40 0x5632b25628c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:18:14.383 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:18:14.382+0000 7f741fc09740 1 --2- 172.21.15.118:0/1795582352 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5632b2552bd0 0x5632b25618b0 unknown :-1 s=CLOSED pgs=330 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:18:14.383 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:18:14.382+0000 7f741fc09740 1 --2- 172.21.15.118:0/1795582352 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5632b25521e0 0x5632b2561370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:18:14.383 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:18:14.382+0000 7f741fc09740 1 -- 172.21.15.118:0/1795582352 >> 172.21.15.118:0/1795582352 conn(0x5632b254d750 msgr2=0x5632b2553e90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:18:14.383 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:18:14.382+0000 7f73f7e00640 0 client.26167 ms_handle_reset on v2:172.21.15.189:6838/4074484168 2024-03-24T00:18:14.384 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:18:14.383+0000 7f741fc09740 1 -- 172.21.15.118:0/1795582352 shutdown_connections 2024-03-24T00:18:14.384 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:18:14.384+0000 7f741fc09740 1 -- 172.21.15.118:0/1795582352 wait complete. 2024-03-24T00:18:14.384 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:18:14.384+0000 7f741fc09740 20 client.26167 trim_cache size 0 max 16384 2024-03-24T00:18:14.385 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stdout:{ 2024-03-24T00:18:14.385 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stdout: "message": "", 2024-03-24T00:18:14.386 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stdout: "return_code": 0 2024-03-24T00:18:14.386 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stdout:} 2024-03-24T00:18:14.387 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:18:14.386+0000 7f741ce00640 1 -- 172.21.15.118:0/608737549 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7f73fc0a2920 msgr2=0x7f73fc0a4de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:18:14.387 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:18:14.386+0000 7f741ce00640 1 --2- 172.21.15.118:0/608737549 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7f73fc0a2920 0x7f73fc0a4de0 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 crypto rx=0x5632b2553430 tx=0x7f7410004a40 comp rx=0 tx=0).stop 2024-03-24T00:18:14.387 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:18:14.386+0000 7f741ce00640 1 -- 172.21.15.118:0/608737549 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f74181028c0 msgr2=0x7f741819a060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:18:14.387 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:18:14.386+0000 7f741ce00640 1 --2- 172.21.15.118:0/608737549 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f74181028c0 0x7f741819a060 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 crypto rx=0x7f740400e9f0 tx=0x7f740400eeb0 comp rx=0 tx=0).stop 2024-03-24T00:18:14.387 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:18:14.386+0000 7f741ce00640 1 -- 172.21.15.118:0/608737549 shutdown_connections 2024-03-24T00:18:14.387 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:18:14.386+0000 7f741ce00640 1 --2- 172.21.15.118:0/608737549 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7f73fc0a2920 0x7f73fc0a4de0 unknown :-1 s=CLOSED pgs=317 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:18:14.387 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:18:14.386+0000 7f741ce00640 1 --2- 172.21.15.118:0/608737549 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7418103280 0x7f74181a10d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:18:14.387 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:18:14.386+0000 7f741ce00640 1 --2- 172.21.15.118:0/608737549 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f74181028c0 0x7f741819a060 secure :-1 s=CLOSED pgs=326 cs=0 l=1 rev1=1 crypto rx=0x7f740400e9f0 tx=0x7f740400eeb0 comp rx=0 tx=0).stop 2024-03-24T00:18:14.387 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:18:14.386+0000 7f741ce00640 1 --2- 172.21.15.118:0/608737549 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f74181016d0 0x7f7418199b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:18:14.387 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:18:14.386+0000 7f741ce00640 1 -- 172.21.15.118:0/608737549 >> 172.21.15.118:0/608737549 conn(0x7f74180fcc60 msgr2=0x7f74180738d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:18:14.387 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:18:14.386+0000 7f741ce00640 1 -- 172.21.15.118:0/608737549 shutdown_connections 2024-03-24T00:18:14.388 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:18:14.387+0000 7f741ce00640 1 -- 172.21.15.118:0/608737549 wait complete. 2024-03-24T00:18:15.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:18:14 smithi118 ceph-mon[31535]: pgmap v474: 97 pgs: 97 active+clean; 37 MiB data, 462 MiB used, 1.0 TiB / 1.0 TiB avail; 783 KiB/s rd, 2.3 MiB/s wr, 450 op/s 2024-03-24T00:18:15.203 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:18:14 smithi189 ceph-mon[36187]: pgmap v474: 97 pgs: 97 active+clean; 37 MiB data, 462 MiB used, 1.0 TiB / 1.0 TiB avail; 783 KiB/s rd, 2.3 MiB/s wr, 450 op/s 2024-03-24T00:18:15.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:18:14 smithi193 ceph-mon[36878]: pgmap v474: 97 pgs: 97 active+clean; 37 MiB data, 462 MiB used, 1.0 TiB / 1.0 TiB avail; 783 KiB/s rd, 2.3 MiB/s wr, 450 op/s 2024-03-24T00:18:16.835 INFO:tasks.workunit.client.0.smithi118.stdout: 10 52714 178 24547 61 25458 294 2024-03-24T00:18:17.199 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:18:16 smithi118 ceph-mon[31535]: pgmap v475: 97 pgs: 97 active+clean; 37 MiB data, 463 MiB used, 1.0 TiB / 1.0 TiB avail; 598 KiB/s rd, 1.6 MiB/s wr, 331 op/s 2024-03-24T00:18:17.203 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:18:16 smithi189 ceph-mon[36187]: pgmap v475: 97 pgs: 97 active+clean; 37 MiB data, 463 MiB used, 1.0 TiB / 1.0 TiB avail; 598 KiB/s rd, 1.6 MiB/s wr, 331 op/s 2024-03-24T00:18:17.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:18:16 smithi193 ceph-mon[36878]: pgmap v475: 97 pgs: 97 active+clean; 37 MiB data, 463 MiB used, 1.0 TiB / 1.0 TiB avail; 598 KiB/s rd, 1.6 MiB/s wr, 331 op/s 2024-03-24T00:18:19.195 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:18:18 smithi118 ceph-mon[31535]: pgmap v476: 97 pgs: 97 active+clean; 35 MiB data, 464 MiB used, 1.0 TiB / 1.0 TiB avail; 597 KiB/s rd, 1.0 MiB/s wr, 298 op/s 2024-03-24T00:18:19.203 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:18:18 smithi189 ceph-mon[36187]: pgmap v476: 97 pgs: 97 active+clean; 35 MiB data, 464 MiB used, 1.0 TiB / 1.0 TiB avail; 597 KiB/s rd, 1.0 MiB/s wr, 298 op/s 2024-03-24T00:18:19.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:18:18 smithi193 ceph-mon[36878]: pgmap v476: 97 pgs: 97 active+clean; 35 MiB data, 464 MiB used, 1.0 TiB / 1.0 TiB avail; 597 KiB/s rd, 1.0 MiB/s wr, 298 op/s 2024-03-24T00:18:21.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:18:20 smithi118 ceph-mon[31535]: pgmap v477: 97 pgs: 97 active+clean; 32 MiB data, 462 MiB used, 1.0 TiB / 1.0 TiB avail; 507 KiB/s rd, 511 KiB/s wr, 242 op/s 2024-03-24T00:18:21.203 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:18:20 smithi189 ceph-mon[36187]: pgmap v477: 97 pgs: 97 active+clean; 32 MiB data, 462 MiB used, 1.0 TiB / 1.0 TiB avail; 507 KiB/s rd, 511 KiB/s wr, 242 op/s 2024-03-24T00:18:21.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:18:20 smithi193 ceph-mon[36878]: pgmap v477: 97 pgs: 97 active+clean; 32 MiB data, 462 MiB used, 1.0 TiB / 1.0 TiB avail; 507 KiB/s rd, 511 KiB/s wr, 242 op/s 2024-03-24T00:18:23.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:18:22 smithi118 ceph-mon[31535]: pgmap v478: 97 pgs: 97 active+clean; 31 MiB data, 460 MiB used, 1.0 TiB / 1.0 TiB avail; 391 KiB/s rd, 418 KiB/s wr, 197 op/s 2024-03-24T00:18:23.203 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:18:22 smithi189 ceph-mon[36187]: pgmap v478: 97 pgs: 97 active+clean; 31 MiB data, 460 MiB used, 1.0 TiB / 1.0 TiB avail; 391 KiB/s rd, 418 KiB/s wr, 197 op/s 2024-03-24T00:18:23.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:18:22 smithi193 ceph-mon[36878]: pgmap v478: 97 pgs: 97 active+clean; 31 MiB data, 460 MiB used, 1.0 TiB / 1.0 TiB avail; 391 KiB/s rd, 418 KiB/s wr, 197 op/s 2024-03-24T00:18:25.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:18:24 smithi118 ceph-mon[31535]: pgmap v479: 97 pgs: 97 active+clean; 31 MiB data, 458 MiB used, 1.0 TiB / 1.0 TiB avail; 510 KiB/s rd, 491 KiB/s wr, 214 op/s 2024-03-24T00:18:25.203 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:18:24 smithi189 ceph-mon[36187]: pgmap v479: 97 pgs: 97 active+clean; 31 MiB data, 458 MiB used, 1.0 TiB / 1.0 TiB avail; 510 KiB/s rd, 491 KiB/s wr, 214 op/s 2024-03-24T00:18:25.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:18:24 smithi193 ceph-mon[36878]: pgmap v479: 97 pgs: 97 active+clean; 31 MiB data, 458 MiB used, 1.0 TiB / 1.0 TiB avail; 510 KiB/s rd, 491 KiB/s wr, 214 op/s 2024-03-24T00:18:26.835 INFO:tasks.workunit.client.0.smithi118.stdout: 11 83796 109 43453 133 47708 241 2024-03-24T00:18:27.195 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:18:26 smithi118 ceph-mon[31535]: pgmap v480: 97 pgs: 97 active+clean; 33 MiB data, 459 MiB used, 1.0 TiB / 1.0 TiB avail; 574 KiB/s rd, 546 KiB/s wr, 217 op/s 2024-03-24T00:18:27.203 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:18:26 smithi189 ceph-mon[36187]: pgmap v480: 97 pgs: 97 active+clean; 33 MiB data, 459 MiB used, 1.0 TiB / 1.0 TiB avail; 574 KiB/s rd, 546 KiB/s wr, 217 op/s 2024-03-24T00:18:27.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:18:26 smithi193 ceph-mon[36878]: pgmap v480: 97 pgs: 97 active+clean; 33 MiB data, 459 MiB used, 1.0 TiB / 1.0 TiB avail; 574 KiB/s rd, 546 KiB/s wr, 217 op/s 2024-03-24T00:18:27.528 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:18:27.528+0000 7fafc0c00640 1 -- 172.21.15.118:0/2589993577 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fafbc074100 msgr2=0x7fafbc074580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:18:27.528 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:18:27.528+0000 7fafc0c00640 1 --2- 172.21.15.118:0/2589993577 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fafbc074100 0x7fafbc074580 secure :-1 s=READY pgs=331 cs=0 l=1 rev1=1 crypto rx=0x7fafa8009860 tx=0x7fafa802bb90 comp rx=0 tx=0).stop 2024-03-24T00:18:27.528 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:18:27.528+0000 7fafc0c00640 1 -- 172.21.15.118:0/2589993577 shutdown_connections 2024-03-24T00:18:27.528 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:18:27.528+0000 7fafc0c00640 1 --2- 172.21.15.118:0/2589993577 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fafbc1093d0 0x7fafbc10b860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:18:27.528 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:18:27.528+0000 7fafc0c00640 1 --2- 172.21.15.118:0/2589993577 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fafbc074100 0x7fafbc074580 unknown :-1 s=CLOSED pgs=331 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:18:27.528 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:18:27.528+0000 7fafc0c00640 1 --2- 172.21.15.118:0/2589993577 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fafbc0756e0 0x7fafbc073b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:18:27.529 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:18:27.528+0000 7fafc0c00640 1 -- 172.21.15.118:0/2589993577 >> 172.21.15.118:0/2589993577 conn(0x7fafbc0fca80 msgr2=0x7fafbc0feec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:18:27.529 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:18:27.528+0000 7fafc0c00640 1 -- 172.21.15.118:0/2589993577 shutdown_connections 2024-03-24T00:18:27.529 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:18:27.528+0000 7fafc0c00640 1 -- 172.21.15.118:0/2589993577 wait complete. 2024-03-24T00:18:27.531 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:18:27.529+0000 7fafc0c00640 1 Processor -- start 2024-03-24T00:18:27.531 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:18:27.530+0000 7fafc0c00640 1 -- start start 2024-03-24T00:18:27.531 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:18:27.530+0000 7fafc0c00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fafbc074100 0x7fafbc199a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:18:27.532 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:18:27.530+0000 7fafc0c00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fafbc0756e0 0x7fafbc199f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:18:27.532 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:18:27.530+0000 7fafc0c00640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fafbc1093d0 0x7fafbc1a0fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:18:27.532 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:18:27.530+0000 7fafc0c00640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fafbc10f350 con 0x7fafbc1093d0 2024-03-24T00:18:27.532 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:18:27.530+0000 7fafc0c00640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fafbc10f1d0 con 0x7fafbc0756e0 2024-03-24T00:18:27.532 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:18:27.530+0000 7fafc0c00640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fafbc10f4d0 con 0x7fafbc074100 2024-03-24T00:18:27.532 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:18:27.530+0000 7fafbb400640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fafbc074100 0x7fafbc199a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:18:27.532 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:18:27.530+0000 7fafbaa00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fafbc0756e0 0x7fafbc199f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:18:27.532 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:18:27.530+0000 7fafbb400640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fafbc074100 0x7fafbc199a20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.118:48774/0 (socket says 172.21.15.118:48774) 2024-03-24T00:18:27.532 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:18:27.530+0000 7fafbaa00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fafbc0756e0 0x7fafbc199f60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.118:59354/0 (socket says 172.21.15.118:59354) 2024-03-24T00:18:27.532 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:18:27.530+0000 7fafbb400640 1 -- 172.21.15.118:0/3599393872 learned_addr learned my addr 172.21.15.118:0/3599393872 (peer_addr_for_me v2:172.21.15.118:0/0) 2024-03-24T00:18:27.532 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:18:27.530+0000 7fafbbe00640 1 --2- 172.21.15.118:0/3599393872 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fafbc1093d0 0x7fafbc1a0fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:18:27.532 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:18:27.531+0000 7fafbb400640 1 -- 172.21.15.118:0/3599393872 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fafbc0756e0 msgr2=0x7fafbc199f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:18:27.532 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:18:27.531+0000 7fafbb400640 1 --2- 172.21.15.118:0/3599393872 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fafbc0756e0 0x7fafbc199f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:18:27.533 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:18:27.531+0000 7fafbb400640 1 -- 172.21.15.118:0/3599393872 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fafbc1093d0 msgr2=0x7fafbc1a0fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:18:27.533 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:18:27.531+0000 7fafbb400640 1 --2- 172.21.15.118:0/3599393872 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fafbc1093d0 0x7fafbc1a0fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:18:27.533 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:18:27.531+0000 7fafbb400640 1 -- 172.21.15.118:0/3599393872 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fafbc1a1640 con 0x7fafbc074100 2024-03-24T00:18:27.533 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:18:27.531+0000 7fafbb400640 1 --2- 172.21.15.118:0/3599393872 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fafbc074100 0x7fafbc199a20 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 crypto rx=0x7fafac00e9f0 tx=0x7fafac00eeb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:18:27.533 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:18:27.531+0000 7fafb3e00640 1 -- 172.21.15.118:0/3599393872 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fafac00cd80 con 0x7fafbc074100 2024-03-24T00:18:27.533 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:18:27.531+0000 7fafb3e00640 1 -- 172.21.15.118:0/3599393872 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fafac010910 con 0x7fafbc074100 2024-03-24T00:18:27.533 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:18:27.531+0000 7fafb3e00640 1 -- 172.21.15.118:0/3599393872 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fafac018990 con 0x7fafbc074100 2024-03-24T00:18:27.533 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:18:27.531+0000 7fafc0c00640 1 -- 172.21.15.118:0/3599393872 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fafbc1a1930 con 0x7fafbc074100 2024-03-24T00:18:27.533 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:18:27.532+0000 7fafc0c00640 1 -- 172.21.15.118:0/3599393872 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fafbc1a1e00 con 0x7fafbc074100 2024-03-24T00:18:27.534 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:18:27.535+0000 7fafc4e36740 1 Processor -- start 2024-03-24T00:18:27.534 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:18:27.535+0000 7fafb3e00640 1 -- 172.21.15.118:0/3599393872 <== mon.1 v2:172.21.15.193:3300/0 4 ==== mgrmap(e 20) v1 ==== 137291+0+0 (secure 0 0 0) 0x7fafac010ac0 con 0x7fafbc074100 2024-03-24T00:18:27.535 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:18:27.535+0000 7fafc4e36740 1 -- start start 2024-03-24T00:18:27.537 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:18:27.535+0000 7fafc4e36740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x557c9e0823f0 0x557c9e082810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:18:27.537 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:18:27.535+0000 7fafc4e36740 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x557c9e082de0 0x557c9e0878e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:18:27.537 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:18:27.535+0000 7fafc4e36740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x557c9e087f50 0x557c9e08a3e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:18:27.537 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:18:27.535+0000 7fafc4e36740 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x557c9e0453b0 con 0x557c9e082de0 2024-03-24T00:18:27.537 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:18:27.535+0000 7fafc4e36740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x557c9dfe7830 con 0x557c9e0823f0 2024-03-24T00:18:27.537 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:18:27.535+0000 7fafc4e36740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x557c9df905d0 con 0x557c9e087f50 2024-03-24T00:18:27.537 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:18:27.536+0000 7fafbbe00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x557c9e0823f0 0x557c9e082810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:18:27.537 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:18:27.536+0000 7fafbbe00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x557c9e0823f0 0x557c9e082810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.118:59360/0 (socket says 172.21.15.118:59360) 2024-03-24T00:18:27.537 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:18:27.536+0000 7fafbbe00640 1 -- 172.21.15.118:0/2431055003 learned_addr learned my addr 172.21.15.118:0/2431055003 (peer_addr_for_me v2:172.21.15.118:0/0) 2024-03-24T00:18:27.537 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:18:27.536+0000 7fafbb400640 1 --2- 172.21.15.118:0/2431055003 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x557c9e082de0 0x557c9e0878e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:18:27.538 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:18:27.536+0000 7fafbaa00640 1 --2- 172.21.15.118:0/2431055003 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x557c9e087f50 0x557c9e08a3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:18:27.538 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:18:27.536+0000 7fafbbe00640 1 -- 172.21.15.118:0/2431055003 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x557c9e087f50 msgr2=0x557c9e08a3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:18:27.538 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:18:27.536+0000 7fafbbe00640 1 --2- 172.21.15.118:0/2431055003 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x557c9e087f50 0x557c9e08a3e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:18:27.538 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:18:27.536+0000 7fafbbe00640 1 -- 172.21.15.118:0/2431055003 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x557c9e082de0 msgr2=0x557c9e0878e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:18:27.538 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:18:27.536+0000 7fafbbe00640 1 --2- 172.21.15.118:0/2431055003 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x557c9e082de0 0x557c9e0878e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:18:27.538 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:18:27.536+0000 7fafbbe00640 1 -- 172.21.15.118:0/2431055003 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x557c9dfa82c0 con 0x557c9e0823f0 2024-03-24T00:18:27.538 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:18:27.536+0000 7fafbbe00640 1 --2- 172.21.15.118:0/2431055003 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x557c9e0823f0 0x557c9e082810 secure :-1 s=READY pgs=330 cs=0 l=1 rev1=1 crypto rx=0x7fafb400e450 tx=0x7fafb40095c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=81e2fa9031d66e66 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:18:27.538 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:18:27.537+0000 7fafa7e00640 1 -- 172.21.15.118:0/2431055003 <== mon.2 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fafb400d910 con 0x557c9e0823f0 2024-03-24T00:18:27.538 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:18:27.537+0000 7fafa7e00640 1 -- 172.21.15.118:0/2431055003 <== mon.2 v2:172.21.15.189:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fafb401c070 con 0x557c9e0823f0 2024-03-24T00:18:27.538 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:18:27.537+0000 7fafa7e00640 1 -- 172.21.15.118:0/2431055003 <== mon.2 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fafb4016380 con 0x557c9e0823f0 2024-03-24T00:18:27.538 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:18:27.537+0000 7fafb3e00640 1 --2- 172.21.15.118:0/3599393872 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7fafa00a2870 0x7fafa00a4d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:18:27.538 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:18:27.537+0000 7fafb3e00640 1 -- 172.21.15.118:0/3599393872 <== mon.1 v2:172.21.15.193:3300/0 5 ==== osd_map(90..90 src has 1..90) v4 ==== 8277+0+0 (secure 0 0 0) 0x7fafac0ced40 con 0x7fafbc074100 2024-03-24T00:18:27.539 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:18:27.537+0000 7fafbb400640 1 --2- 172.21.15.118:0/3599393872 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7fafa00a2870 0x7fafa00a4d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:18:27.540 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:18:27.538+0000 7fafc4e36740 1 -- 172.21.15.118:0/2431055003 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x557c9e0823f0 msgr2=0x557c9e082810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:18:27.540 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:18:27.538+0000 7fafbb400640 1 --2- 172.21.15.118:0/3599393872 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7fafa00a2870 0x7fafa00a4d30 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 crypto rx=0x557c9e0831e0 tx=0x7fafac0d2810 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:18:27.540 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:18:27.538+0000 7fafc4e36740 1 --2- 172.21.15.118:0/2431055003 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x557c9e0823f0 0x557c9e082810 secure :-1 s=READY pgs=330 cs=0 l=1 rev1=1 crypto rx=0x7fafb400e450 tx=0x7fafb40095c0 comp rx=0 tx=0).stop 2024-03-24T00:18:27.540 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:18:27.538+0000 7fafc4e36740 1 -- 172.21.15.118:0/2431055003 shutdown_connections 2024-03-24T00:18:27.540 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:18:27.538+0000 7fafc4e36740 1 --2- 172.21.15.118:0/2431055003 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x557c9e087f50 0x557c9e08a3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:18:27.540 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:18:27.538+0000 7fafc4e36740 1 --2- 172.21.15.118:0/2431055003 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x557c9e082de0 0x557c9e0878e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:18:27.540 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:18:27.538+0000 7fafc4e36740 1 --2- 172.21.15.118:0/2431055003 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x557c9e0823f0 0x557c9e082810 unknown :-1 s=CLOSED pgs=330 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:18:27.541 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:18:27.538+0000 7fafc4e36740 1 -- 172.21.15.118:0/2431055003 >> 172.21.15.118:0/2431055003 conn(0x557c9e07d960 msgr2=0x557c9e07dd60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:18:27.541 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:18:27.538+0000 7fafc4e36740 1 -- 172.21.15.118:0/2431055003 shutdown_connections 2024-03-24T00:18:27.541 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:18:27.538+0000 7fafc4e36740 1 -- 172.21.15.118:0/2431055003 wait complete. 2024-03-24T00:18:27.541 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:18:27.539+0000 7fafc4e36740 1 -- start start 2024-03-24T00:18:27.542 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:18:27.539+0000 7fafc4e36740 1 Processor -- start 2024-03-24T00:18:27.545 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:18:27.540+0000 7fafc4e36740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x557c9e0823f0 0x557c9e08c520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:18:27.545 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:18:27.540+0000 7fafc4e36740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x557c9e082de0 0x557c9e08ca60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:18:27.545 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:18:27.540+0000 7fafc4e36740 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x557c9e087f50 0x557c9e093aa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:18:27.545 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:18:27.540+0000 7fafc4e36740 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x557c9dfa3a20 con 0x557c9e087f50 2024-03-24T00:18:27.545 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:18:27.540+0000 7fafc4e36740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x557c9dfe38d0 con 0x557c9e0823f0 2024-03-24T00:18:27.545 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:18:27.540+0000 7fafc4e36740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x557c9e08af00 con 0x557c9e082de0 2024-03-24T00:18:27.545 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:18:27.540+0000 7fafbbe00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x557c9e0823f0 0x557c9e08c520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:18:27.545 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:18:27.541+0000 7fafbaa00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x557c9e082de0 0x557c9e08ca60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:18:27.546 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:18:27.541+0000 7fafc4e36740 10 client.0 _ll_register_callbacks cb 0x557c9dfe01c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T00:18:27.546 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:18:27.541+0000 7fafbbe00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x557c9e0823f0 0x557c9e08c520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.118:59368/0 (socket says 172.21.15.118:59368) 2024-03-24T00:18:27.546 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:18:27.541+0000 7fafbbe00640 1 -- 172.21.15.118:0/3229823837 learned_addr learned my addr 172.21.15.118:0/3229823837 (peer_addr_for_me v2:172.21.15.118:0/0) 2024-03-24T00:18:27.546 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:18:27.541+0000 7fafbbe00640 1 --2- 172.21.15.118:0/3229823837 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x557c9e087f50 0x557c9e093aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:18:27.546 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:18:27.541+0000 7fafbaa00640 1 -- 172.21.15.118:0/3229823837 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x557c9e0823f0 msgr2=0x557c9e08c520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:18:27.546 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:18:27.541+0000 7fafbaa00640 1 --2- 172.21.15.118:0/3229823837 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x557c9e0823f0 0x557c9e08c520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:18:27.546 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:18:27.541+0000 7fafbaa00640 1 -- 172.21.15.118:0/3229823837 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x557c9e087f50 msgr2=0x557c9e093aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:18:27.546 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:18:27.541+0000 7fafbaa00640 1 --2- 172.21.15.118:0/3229823837 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x557c9e087f50 0x557c9e093aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:18:27.546 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:18:27.541+0000 7fafbaa00640 1 -- 172.21.15.118:0/3229823837 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x557c9dcfc210 con 0x557c9e082de0 2024-03-24T00:18:27.546 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:18:27.541+0000 7fafbbe00640 1 --2- 172.21.15.118:0/3229823837 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x557c9e0823f0 0x557c9e08c520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T00:18:27.547 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:18:27.541+0000 7fafbaa00640 1 --2- 172.21.15.118:0/3229823837 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x557c9e082de0 0x557c9e08ca60 secure :-1 s=READY pgs=333 cs=0 l=1 rev1=1 crypto rx=0x7fafa802c0d0 tx=0x7fafa8005d90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=9bed354574d66213 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:18:27.547 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:18:27.541+0000 7fafa4c00640 10 client.0 ms_handle_connect on v2:172.21.15.193:3300/0 2024-03-24T00:18:27.547 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:18:27.542+0000 7fafa4c00640 1 -- 172.21.15.118:0/3229823837 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fafa8043070 con 0x557c9e082de0 2024-03-24T00:18:27.547 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:18:27.542+0000 7fafa4c00640 1 -- 172.21.15.118:0/3229823837 <== mon.1 v2:172.21.15.193:3300/0 2 ==== osd_map(90..90 src has 1..90) v4 ==== 8277+0+0 (secure 0 0 0) 0x7fafa8004040 con 0x557c9e082de0 2024-03-24T00:18:27.547 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:18:27.542+0000 7fafc4e36740 1 -- 172.21.15.118:0/3229823837 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fafb401c070 con 0x557c9e082de0 2024-03-24T00:18:27.547 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:18:27.542+0000 7fafa4c00640 1 -- 172.21.15.118:0/3229823837 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) v2 ==== 24+0+0 (secure 0 0 0) 0x7fafa8039040 con 0x557c9e082de0 2024-03-24T00:18:27.547 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:18:27.542+0000 7fafc4e36740 10 client.26179 fetch_fsmap learned FSMap version 28 2024-03-24T00:18:27.547 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:18:27.542+0000 7fafc4e36740 1 -- 172.21.15.118:0/3229823837 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) v3 -- 0x557c9ddc8620 con 0x557c9e082de0 2024-03-24T00:18:27.547 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:18:27.542+0000 7fafa4c00640 1 -- 172.21.15.118:0/3229823837 <== mon.1 v2:172.21.15.193:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fafb401c070 con 0x557c9e082de0 2024-03-24T00:18:27.547 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:18:27.542+0000 7fafa4c00640 1 -- 172.21.15.118:0/3229823837 <== mon.1 v2:172.21.15.193:3300/0 5 ==== fsmap(e 28) v1 ==== 7934+0+0 (secure 0 0 0) 0x7fafa8032640 con 0x557c9e082de0 2024-03-24T00:18:27.547 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:18:27.542+0000 7fafc4e36740 10 client.26179 fetch_fsmap finished waiting for FSMap version 28 2024-03-24T00:18:27.547 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:18:27.542+0000 7fafc4e36740 10 client.26179 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-24T00:18:27.547 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:18:27.543+0000 7fafc4e36740 20 client.26179 populate_metadata read hostname 'smithi118' 2024-03-24T00:18:27.547 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:18:27.543+0000 7fafc4e36740 1 --2- 172.21.15.118:0/3229823837 >> [v2:172.21.15.189:6838/4074484168,v1:172.21.15.189:6839/4074484168] conn(0x557c9e0a9450 0x557c9e0ab840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:18:27.547 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:18:27.543+0000 7fafc4e36740 4 client.26179 mds_command: new command op to 24569 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-24T00:18:27.548 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:18:27.543+0000 7fafc4e36740 1 -- 172.21.15.118:0/3229823837 --> [v2:172.21.15.189:6838/4074484168,v1:172.21.15.189:6839/4074484168] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x557c9dff4d70 con 0x557c9e0a9450 2024-03-24T00:18:27.548 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:18:27.543+0000 7fafbbe00640 1 --2- 172.21.15.118:0/3229823837 >> [v2:172.21.15.189:6838/4074484168,v1:172.21.15.189:6839/4074484168] conn(0x557c9e0a9450 0x557c9e0ab840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:18:27.548 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:18:27.543+0000 7fafa4c00640 10 client.26179 ms_handle_connect on v2:172.21.15.189:6838/4074484168 2024-03-24T00:18:27.548 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:18:27.544+0000 7fafbbe00640 1 --2- 172.21.15.118:0/3229823837 >> [v2:172.21.15.189:6838/4074484168,v1:172.21.15.189:6839/4074484168] conn(0x557c9e0a9450 0x557c9e0ab840 crc :-1 s=READY pgs=91 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=d9892d5513dd0932 server_cookie=fa8b069b90954e94 in_seq=0 out_seq=0 2024-03-24T00:18:27.549 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:18:27.548+0000 7fafa4c00640 1 -- 172.21.15.118:0/3229823837 <== mds.0 v2:172.21.15.189:6838/4074484168 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x557c9dff4d70 con 0x557c9e0a9450 2024-03-24T00:18:27.549 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:18:27.548+0000 7fafa4c00640 10 client.26179 handle_command_reply: tid=0 2024-03-24T00:18:27.549 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:18:27.548+0000 7fafc4e36740 1 client.26179 shutdown 2024-03-24T00:18:27.549 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:18:27.548+0000 7fafc4e36740 1 -- 172.21.15.118:0/3229823837 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x557c9e082de0 msgr2=0x557c9e08ca60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:18:27.549 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:18:27.548+0000 7fafc4e36740 1 --2- 172.21.15.118:0/3229823837 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x557c9e082de0 0x557c9e08ca60 secure :-1 s=READY pgs=333 cs=0 l=1 rev1=1 crypto rx=0x7fafa802c0d0 tx=0x7fafa8005d90 comp rx=0 tx=0).stop 2024-03-24T00:18:27.549 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:18:27.549+0000 7fafc4e36740 1 -- 172.21.15.118:0/3229823837 shutdown_connections 2024-03-24T00:18:27.549 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:18:27.549+0000 7fafc4e36740 1 --2- 172.21.15.118:0/3229823837 >> [v2:172.21.15.189:6838/4074484168,v1:172.21.15.189:6839/4074484168] conn(0x557c9e0a9450 0x557c9e0ab840 crc :-1 s=READY pgs=91 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:18:27.549 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:18:27.549+0000 7fafc4e36740 1 --2- 172.21.15.118:0/3229823837 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x557c9e087f50 0x557c9e093aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:18:27.549 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:18:27.549+0000 7fafc4e36740 1 --2- 172.21.15.118:0/3229823837 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x557c9e082de0 0x557c9e08ca60 unknown :-1 s=CLOSED pgs=333 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:18:27.549 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:18:27.549+0000 7fafc4e36740 1 --2- 172.21.15.118:0/3229823837 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x557c9e0823f0 0x557c9e08c520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:18:27.549 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:18:27.549+0000 7fafc4e36740 1 -- 172.21.15.118:0/3229823837 >> 172.21.15.118:0/3229823837 conn(0x557c9e07d960 msgr2=0x557c9e07cb70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:18:27.549 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:18:27.549+0000 7fafa4c00640 0 client.26179 ms_handle_reset on v2:172.21.15.189:6838/4074484168 2024-03-24T00:18:27.550 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:18:27.550+0000 7fafc4e36740 1 -- 172.21.15.118:0/3229823837 shutdown_connections 2024-03-24T00:18:27.550 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:18:27.550+0000 7fafc4e36740 1 -- 172.21.15.118:0/3229823837 wait complete. 2024-03-24T00:18:27.550 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:18:27.551+0000 7fafc4e36740 20 client.26179 trim_cache size 0 max 16384 2024-03-24T00:18:27.566 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:18:27.566+0000 7fafc4e36740 1 Processor -- start 2024-03-24T00:18:27.566 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:18:27.567+0000 7fafc4e36740 1 -- start start 2024-03-24T00:18:27.568 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:18:27.567+0000 7fafc4e36740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x557c9e0823f0 0x557c9e07dd40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:18:27.568 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:18:27.567+0000 7fafc4e36740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x557c9e082de0 0x557c9e093a60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:18:27.568 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:18:27.567+0000 7fafc4e36740 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x557c9e087f50 0x557c9e08c520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:18:27.568 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:18:27.567+0000 7fafc4e36740 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fafbc1a2260 con 0x557c9e087f50 2024-03-24T00:18:27.568 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:18:27.567+0000 7fafc4e36740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fafbc1a3be0 con 0x557c9e0823f0 2024-03-24T00:18:27.568 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:18:27.567+0000 7fafc4e36740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fafbc1a28c0 con 0x557c9e082de0 2024-03-24T00:18:27.568 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:18:27.567+0000 7fafbaa00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x557c9e082de0 0x557c9e093a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:18:27.568 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:18:27.567+0000 7fafbaa00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x557c9e082de0 0x557c9e093a60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.118:48798/0 (socket says 172.21.15.118:48798) 2024-03-24T00:18:27.568 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:18:27.567+0000 7fafbaa00640 1 -- 172.21.15.118:0/2076964992 learned_addr learned my addr 172.21.15.118:0/2076964992 (peer_addr_for_me v2:172.21.15.118:0/0) 2024-03-24T00:18:27.568 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:18:27.567+0000 7fafbbe00640 1 --2- 172.21.15.118:0/2076964992 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x557c9e0823f0 0x557c9e07dd40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:18:27.568 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:18:27.567+0000 7fafbbe00640 1 --2- 172.21.15.118:0/2076964992 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x557c9e087f50 0x557c9e08c520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:18:27.569 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:18:27.568+0000 7fafbaa00640 1 -- 172.21.15.118:0/2076964992 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x557c9e0823f0 msgr2=0x557c9e07dd40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:18:27.569 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:18:27.568+0000 7fafbaa00640 1 --2- 172.21.15.118:0/2076964992 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x557c9e0823f0 0x557c9e07dd40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:18:27.569 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:18:27.568+0000 7fafbaa00640 1 -- 172.21.15.118:0/2076964992 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x557c9e087f50 msgr2=0x557c9e08c520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:18:27.569 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:18:27.568+0000 7fafbaa00640 1 --2- 172.21.15.118:0/2076964992 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x557c9e087f50 0x557c9e08c520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:18:27.569 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:18:27.568+0000 7fafbaa00640 1 -- 172.21.15.118:0/2076964992 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x557c9e08b040 con 0x557c9e082de0 2024-03-24T00:18:27.569 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:18:27.568+0000 7fafbaa00640 1 --2- 172.21.15.118:0/2076964992 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x557c9e082de0 0x557c9e093a60 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 crypto rx=0x7fafa80095f0 tx=0x7fafa8005fb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=e661d1c226a0a687 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:18:27.569 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:18:27.568+0000 7fafa7400640 1 -- 172.21.15.118:0/2076964992 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fafa8043040 con 0x557c9e082de0 2024-03-24T00:18:27.569 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:18:27.568+0000 7fafa7400640 1 -- 172.21.15.118:0/2076964992 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fafa8005520 con 0x557c9e082de0 2024-03-24T00:18:27.569 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:18:27.568+0000 7fafa7400640 1 -- 172.21.15.118:0/2076964992 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fafa802c530 con 0x557c9e082de0 2024-03-24T00:18:27.572 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:18:27.569+0000 7fafc4e36740 1 -- 172.21.15.118:0/2076964992 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x557c9e082de0 msgr2=0x557c9e093a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:18:27.572 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:18:27.569+0000 7fafc4e36740 1 --2- 172.21.15.118:0/2076964992 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x557c9e082de0 0x557c9e093a60 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 crypto rx=0x7fafa80095f0 tx=0x7fafa8005fb0 comp rx=0 tx=0).stop 2024-03-24T00:18:27.572 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:18:27.569+0000 7fafc4e36740 1 -- 172.21.15.118:0/2076964992 shutdown_connections 2024-03-24T00:18:27.572 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:18:27.569+0000 7fafc4e36740 1 --2- 172.21.15.118:0/2076964992 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x557c9e087f50 0x557c9e08c520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:18:27.572 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:18:27.569+0000 7fafc4e36740 1 --2- 172.21.15.118:0/2076964992 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x557c9e082de0 0x557c9e093a60 unknown :-1 s=CLOSED pgs=334 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:18:27.572 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:18:27.569+0000 7fafc4e36740 1 --2- 172.21.15.118:0/2076964992 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x557c9e0823f0 0x557c9e07dd40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:18:27.572 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:18:27.569+0000 7fafc4e36740 1 -- 172.21.15.118:0/2076964992 >> 172.21.15.118:0/2076964992 conn(0x557c9e07d960 msgr2=0x557c9e0876d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:18:27.572 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:18:27.569+0000 7fafc4e36740 1 -- 172.21.15.118:0/2076964992 shutdown_connections 2024-03-24T00:18:27.572 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:18:27.569+0000 7fafc4e36740 1 -- 172.21.15.118:0/2076964992 wait complete. 2024-03-24T00:18:27.572 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:18:27.570+0000 7fafc4e36740 1 -- start start 2024-03-24T00:18:27.573 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:18:27.570+0000 7fafc4e36740 1 Processor -- start 2024-03-24T00:18:27.573 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:18:27.570+0000 7fafc4e36740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x557c9e0823f0 0x557c9e090f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:18:27.573 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:18:27.571+0000 7fafc4e36740 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x557c9e087f50 0x557c9e091460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:18:27.573 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:18:27.571+0000 7fafc4e36740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x557c9e092470 0x557c9e0928f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:18:27.573 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:18:27.571+0000 7fafc4e36740 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fafbc1a3580 con 0x557c9e087f50 2024-03-24T00:18:27.573 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:18:27.571+0000 7fafc4e36740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fafbc1a2f20 con 0x557c9e092470 2024-03-24T00:18:27.573 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:18:27.571+0000 7fafc4e36740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fafbc071cb0 con 0x557c9e0823f0 2024-03-24T00:18:27.573 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:18:27.571+0000 7fafc4e36740 10 client.0 _ll_register_callbacks cb 0x7fafb4009460 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T00:18:27.573 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:18:27.571+0000 7fafbbe00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x557c9e0823f0 0x557c9e090f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:18:27.573 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:18:27.571+0000 7fafbaa00640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x557c9e087f50 0x557c9e091460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:18:27.573 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:18:27.571+0000 7fafbbe00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x557c9e0823f0 0x557c9e090f20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.118:48812/0 (socket says 172.21.15.118:48812) 2024-03-24T00:18:27.573 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:18:27.571+0000 7fafbbe00640 1 -- 172.21.15.118:0/980871684 learned_addr learned my addr 172.21.15.118:0/980871684 (peer_addr_for_me v2:172.21.15.118:0/0) 2024-03-24T00:18:27.573 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:18:27.571+0000 7fafbbe00640 1 --2- 172.21.15.118:0/980871684 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x557c9e092470 0x557c9e0928f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:18:27.574 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:18:27.571+0000 7fafbbe00640 1 -- 172.21.15.118:0/980871684 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x557c9e092470 msgr2=0x557c9e0928f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:18:27.574 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:18:27.571+0000 7fafbbe00640 1 --2- 172.21.15.118:0/980871684 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x557c9e092470 0x557c9e0928f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:18:27.574 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:18:27.571+0000 7fafbbe00640 1 -- 172.21.15.118:0/980871684 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x557c9e087f50 msgr2=0x557c9e091460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:18:27.574 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:18:27.571+0000 7fafbbe00640 1 --2- 172.21.15.118:0/980871684 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x557c9e087f50 0x557c9e091460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:18:27.574 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:18:27.571+0000 7fafbbe00640 1 -- 172.21.15.118:0/980871684 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x557c9e092e30 con 0x557c9e0823f0 2024-03-24T00:18:27.574 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:18:27.571+0000 7fafbaa00640 1 --2- 172.21.15.118:0/980871684 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x557c9e087f50 0x557c9e091460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T00:18:27.574 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:18:27.572+0000 7fafbbe00640 1 --2- 172.21.15.118:0/980871684 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x557c9e0823f0 0x557c9e090f20 secure :-1 s=READY pgs=335 cs=0 l=1 rev1=1 crypto rx=0x7fafb4009e20 tx=0x7fafb4027040 comp rx=0 tx=0).ready entity=mon.1 client_cookie=a3fbded849d9a7d5 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:18:27.574 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:18:27.572+0000 7fafa4c00640 10 client.0 ms_handle_connect on v2:172.21.15.193:3300/0 2024-03-24T00:18:27.574 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:18:27.572+0000 7fafa4c00640 1 -- 172.21.15.118:0/980871684 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fafb4017a80 con 0x557c9e0823f0 2024-03-24T00:18:27.574 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:18:27.572+0000 7fafc4e36740 1 -- 172.21.15.118:0/980871684 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fafa8005520 con 0x557c9e0823f0 2024-03-24T00:18:27.574 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:18:27.572+0000 7fafa4c00640 1 -- 172.21.15.118:0/980871684 <== mon.1 v2:172.21.15.193:3300/0 2 ==== osd_map(90..90 src has 1..90) v4 ==== 8277+0+0 (secure 0 0 0) 0x7fafb40324b0 con 0x557c9e0823f0 2024-03-24T00:18:27.574 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:18:27.572+0000 7fafa4c00640 1 -- 172.21.15.118:0/980871684 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) v2 ==== 24+0+0 (secure 0 0 0) 0x7fafb4017630 con 0x557c9e0823f0 2024-03-24T00:18:27.574 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:18:27.572+0000 7fafc4e36740 10 client.26185 fetch_fsmap learned FSMap version 28 2024-03-24T00:18:27.574 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:18:27.572+0000 7fafc4e36740 1 -- 172.21.15.118:0/980871684 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) v3 -- 0x557c9e104f40 con 0x557c9e0823f0 2024-03-24T00:18:27.575 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:18:27.572+0000 7fafa4c00640 1 -- 172.21.15.118:0/980871684 <== mon.1 v2:172.21.15.193:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fafa8005520 con 0x557c9e0823f0 2024-03-24T00:18:27.575 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:18:27.572+0000 7fafa4c00640 1 -- 172.21.15.118:0/980871684 <== mon.1 v2:172.21.15.193:3300/0 5 ==== fsmap(e 28) v1 ==== 7934+0+0 (secure 0 0 0) 0x7fafb400b040 con 0x557c9e0823f0 2024-03-24T00:18:27.575 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:18:27.573+0000 7fafc4e36740 10 client.26185 fetch_fsmap finished waiting for FSMap version 28 2024-03-24T00:18:27.576 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:18:27.573+0000 7fafc4e36740 10 client.26185 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-24T00:18:27.576 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:18:27.573+0000 7fafc4e36740 20 client.26185 populate_metadata read hostname 'smithi118' 2024-03-24T00:18:27.576 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:18:27.573+0000 7fafc4e36740 1 --2- 172.21.15.118:0/980871684 >> [v2:172.21.15.189:6838/4074484168,v1:172.21.15.189:6839/4074484168] conn(0x557c9e105830 0x557c9e107c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:18:27.576 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:18:27.573+0000 7fafc4e36740 4 client.26185 mds_command: new command op to 24569 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-03-24T00:18:27.576 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:18:27.573+0000 7fafc4e36740 1 -- 172.21.15.118:0/980871684 --> [v2:172.21.15.189:6838/4074484168,v1:172.21.15.189:6839/4074484168] -- command(tid 0: {"prefix": "flush journal"}) v1 -- 0x557c9dfef0c0 con 0x557c9e105830 2024-03-24T00:18:27.576 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:18:27.573+0000 7fafbaa00640 1 --2- 172.21.15.118:0/980871684 >> [v2:172.21.15.189:6838/4074484168,v1:172.21.15.189:6839/4074484168] conn(0x557c9e105830 0x557c9e107c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:18:27.576 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:18:27.574+0000 7fafbaa00640 1 --2- 172.21.15.118:0/980871684 >> [v2:172.21.15.189:6838/4074484168,v1:172.21.15.189:6839/4074484168] conn(0x557c9e105830 0x557c9e107c50 crc :-1 s=READY pgs=92 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=a575d1cb1e6cbff server_cookie=62b394876cebb182 in_seq=0 out_seq=0 2024-03-24T00:18:27.576 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:18:27.574+0000 7fafa4c00640 10 client.26185 ms_handle_connect on v2:172.21.15.189:6838/4074484168 2024-03-24T00:18:27.892 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:18:27.891+0000 7fafa4c00640 1 -- 172.21.15.118:0/980871684 <== mds.0 v2:172.21.15.189:6838/4074484168 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+44 (crc 0 0 0) 0x557c9dfef0c0 con 0x557c9e105830 2024-03-24T00:18:27.892 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:18:27.891+0000 7fafa4c00640 10 client.26185 handle_command_reply: tid=0 2024-03-24T00:18:27.892 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:18:27.891+0000 7fafc4e36740 1 client.26185 shutdown 2024-03-24T00:18:27.892 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:18:27.891+0000 7fafc4e36740 1 -- 172.21.15.118:0/980871684 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x557c9e0823f0 msgr2=0x557c9e090f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:18:27.892 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:18:27.891+0000 7fafc4e36740 1 --2- 172.21.15.118:0/980871684 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x557c9e0823f0 0x557c9e090f20 secure :-1 s=READY pgs=335 cs=0 l=1 rev1=1 crypto rx=0x7fafb4009e20 tx=0x7fafb4027040 comp rx=0 tx=0).stop 2024-03-24T00:18:27.892 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:18:27.892+0000 7fafc4e36740 1 -- 172.21.15.118:0/980871684 shutdown_connections 2024-03-24T00:18:27.892 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:18:27.892+0000 7fafc4e36740 1 --2- 172.21.15.118:0/980871684 >> [v2:172.21.15.189:6838/4074484168,v1:172.21.15.189:6839/4074484168] conn(0x557c9e105830 0x557c9e107c50 crc :-1 s=READY pgs=92 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:18:27.893 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:18:27.892+0000 7fafc4e36740 1 --2- 172.21.15.118:0/980871684 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x557c9e092470 0x557c9e0928f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:18:27.893 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:18:27.892+0000 7fafc4e36740 1 --2- 172.21.15.118:0/980871684 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x557c9e087f50 0x557c9e091460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:18:27.893 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:18:27.892+0000 7fafc4e36740 1 --2- 172.21.15.118:0/980871684 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x557c9e0823f0 0x557c9e090f20 unknown :-1 s=CLOSED pgs=335 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:18:27.893 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:18:27.892+0000 7fafc4e36740 1 -- 172.21.15.118:0/980871684 >> 172.21.15.118:0/980871684 conn(0x557c9e07d960 msgr2=0x557c9e0872a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:18:27.893 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:18:27.892+0000 7fafa4c00640 0 client.26185 ms_handle_reset on v2:172.21.15.189:6838/4074484168 2024-03-24T00:18:27.893 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:18:27.892+0000 7fafc4e36740 1 -- 172.21.15.118:0/980871684 shutdown_connections 2024-03-24T00:18:27.893 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:18:27.892+0000 7fafc4e36740 1 -- 172.21.15.118:0/980871684 wait complete. 2024-03-24T00:18:27.893 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:18:27.892+0000 7fafc4e36740 20 client.26185 trim_cache size 0 max 16384 2024-03-24T00:18:27.894 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stdout:{ 2024-03-24T00:18:27.894 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stdout: "message": "", 2024-03-24T00:18:27.894 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stdout: "return_code": 0 2024-03-24T00:18:27.894 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stdout:} 2024-03-24T00:18:27.895 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:18:27.893+0000 7fafc0c00640 1 -- 172.21.15.118:0/3599393872 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7fafa00a2870 msgr2=0x7fafa00a4d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:18:27.895 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:18:27.893+0000 7fafc0c00640 1 --2- 172.21.15.118:0/3599393872 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7fafa00a2870 0x7fafa00a4d30 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 crypto rx=0x557c9e0831e0 tx=0x7fafac0d2810 comp rx=0 tx=0).stop 2024-03-24T00:18:27.895 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:18:27.893+0000 7fafc0c00640 1 -- 172.21.15.118:0/3599393872 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fafbc074100 msgr2=0x7fafbc199a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:18:27.895 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:18:27.893+0000 7fafc0c00640 1 --2- 172.21.15.118:0/3599393872 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fafbc074100 0x7fafbc199a20 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 crypto rx=0x7fafac00e9f0 tx=0x7fafac00eeb0 comp rx=0 tx=0).stop 2024-03-24T00:18:27.895 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:18:27.893+0000 7fafc0c00640 1 -- 172.21.15.118:0/3599393872 shutdown_connections 2024-03-24T00:18:27.895 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:18:27.893+0000 7fafc0c00640 1 --2- 172.21.15.118:0/3599393872 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7fafa00a2870 0x7fafa00a4d30 unknown :-1 s=CLOSED pgs=318 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:18:27.895 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:18:27.893+0000 7fafc0c00640 1 --2- 172.21.15.118:0/3599393872 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fafbc1093d0 0x7fafbc1a0fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:18:27.895 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:18:27.893+0000 7fafc0c00640 1 --2- 172.21.15.118:0/3599393872 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fafbc0756e0 0x7fafbc199f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:18:27.895 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:18:27.893+0000 7fafc0c00640 1 --2- 172.21.15.118:0/3599393872 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fafbc074100 0x7fafbc199a20 unknown :-1 s=CLOSED pgs=332 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:18:27.895 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:18:27.893+0000 7fafc0c00640 1 -- 172.21.15.118:0/3599393872 >> 172.21.15.118:0/3599393872 conn(0x7fafbc0fca80 msgr2=0x7fafbc1052d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:18:27.895 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:18:27.894+0000 7fafc0c00640 1 -- 172.21.15.118:0/3599393872 shutdown_connections 2024-03-24T00:18:27.895 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:18:27.894+0000 7fafc0c00640 1 -- 172.21.15.118:0/3599393872 wait complete. 2024-03-24T00:18:28.197 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:18:27 smithi118 ceph-mon[31535]: pgmap v481: 97 pgs: 97 active+clean; 32 MiB data, 458 MiB used, 1.0 TiB / 1.0 TiB avail; 494 KiB/s rd, 527 KiB/s wr, 198 op/s 2024-03-24T00:18:28.203 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:18:27 smithi189 ceph-mon[36187]: pgmap v481: 97 pgs: 97 active+clean; 32 MiB data, 458 MiB used, 1.0 TiB / 1.0 TiB avail; 494 KiB/s rd, 527 KiB/s wr, 198 op/s 2024-03-24T00:18:28.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:18:27 smithi193 ceph-mon[36878]: pgmap v481: 97 pgs: 97 active+clean; 32 MiB data, 458 MiB used, 1.0 TiB / 1.0 TiB avail; 494 KiB/s rd, 527 KiB/s wr, 198 op/s 2024-03-24T00:18:31.195 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:18:30 smithi118 ceph-mon[31535]: pgmap v482: 97 pgs: 97 active+clean; 39 MiB data, 465 MiB used, 1.0 TiB / 1.0 TiB avail; 491 KiB/s rd, 813 KiB/s wr, 190 op/s 2024-03-24T00:18:31.203 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:18:30 smithi189 ceph-mon[36187]: pgmap v482: 97 pgs: 97 active+clean; 39 MiB data, 465 MiB used, 1.0 TiB / 1.0 TiB avail; 491 KiB/s rd, 813 KiB/s wr, 190 op/s 2024-03-24T00:18:31.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:18:30 smithi193 ceph-mon[36878]: pgmap v482: 97 pgs: 97 active+clean; 39 MiB data, 465 MiB used, 1.0 TiB / 1.0 TiB avail; 491 KiB/s rd, 813 KiB/s wr, 190 op/s 2024-03-24T00:18:33.195 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:18:32 smithi118 ceph-mon[31535]: pgmap v483: 97 pgs: 97 active+clean; 43 MiB data, 473 MiB used, 1.0 TiB / 1.0 TiB avail; 464 KiB/s rd, 1.2 MiB/s wr, 224 op/s 2024-03-24T00:18:33.203 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:18:32 smithi189 ceph-mon[36187]: pgmap v483: 97 pgs: 97 active+clean; 43 MiB data, 473 MiB used, 1.0 TiB / 1.0 TiB avail; 464 KiB/s rd, 1.2 MiB/s wr, 224 op/s 2024-03-24T00:18:33.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:18:32 smithi193 ceph-mon[36878]: pgmap v483: 97 pgs: 97 active+clean; 43 MiB data, 473 MiB used, 1.0 TiB / 1.0 TiB avail; 464 KiB/s rd, 1.2 MiB/s wr, 224 op/s 2024-03-24T00:18:34.465 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-03-24T00:18:34.466 DEBUG:teuthology.orchestra.run.smithi118:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-24T00:18:34.677 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:34.677+0000 7f3e50a00640 1 -- 172.21.15.118:0/845909115 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f3e4c1016b0 msgr2=0x7f3e4c101ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:18:34.677 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:34.677+0000 7f3e50a00640 1 --2- 172.21.15.118:0/845909115 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f3e4c1016b0 0x7f3e4c101ad0 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7f3e3c009860 tx=0x7f3e3c02bb70 comp rx=0 tx=0).stop 2024-03-24T00:18:34.677 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:34.677+0000 7f3e50a00640 1 -- 172.21.15.118:0/845909115 shutdown_connections 2024-03-24T00:18:34.678 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:34.677+0000 7f3e50a00640 1 --2- 172.21.15.118:0/845909115 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f3e4c103260 0x7f3e4c109af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:18:34.678 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:34.677+0000 7f3e50a00640 1 --2- 172.21.15.118:0/845909115 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3e4c1028a0 0x7f3e4c102d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:18:34.678 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:34.677+0000 7f3e50a00640 1 --2- 172.21.15.118:0/845909115 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f3e4c1016b0 0x7f3e4c101ad0 unknown :-1 s=CLOSED pgs=199 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:18:34.678 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:34.677+0000 7f3e50a00640 1 -- 172.21.15.118:0/845909115 >> 172.21.15.118:0/845909115 conn(0x7f3e4c0fcc40 msgr2=0x7f3e4c0ff060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:18:34.678 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:34.677+0000 7f3e50a00640 1 -- 172.21.15.118:0/845909115 shutdown_connections 2024-03-24T00:18:34.678 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:34.678+0000 7f3e50a00640 1 -- 172.21.15.118:0/845909115 wait complete. 2024-03-24T00:18:34.678 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:34.679+0000 7f3e50a00640 1 Processor -- start 2024-03-24T00:18:34.679 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:34.679+0000 7f3e50a00640 1 -- start start 2024-03-24T00:18:34.680 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:34.679+0000 7f3e50a00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f3e4c1016b0 0x7f3e4c1957f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:18:34.680 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:34.679+0000 7f3e50a00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3e4c1028a0 0x7f3e4c195d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:18:34.680 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:34.680+0000 7f3e50a00640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f3e4c103260 0x7f3e4c19cda0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:18:34.680 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:34.680+0000 7f3e50a00640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3e4c10b150 con 0x7f3e4c103260 2024-03-24T00:18:34.680 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:34.680+0000 7f3e50a00640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3e4c10afd0 con 0x7f3e4c1028a0 2024-03-24T00:18:34.680 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:34.680+0000 7f3e50a00640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3e4c10b2d0 con 0x7f3e4c1016b0 2024-03-24T00:18:34.680 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:34.680+0000 7f3e49400640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3e4c1028a0 0x7f3e4c195d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:18:34.681 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:34.680+0000 7f3e49400640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3e4c1028a0 0x7f3e4c195d30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.118:60880/0 (socket says 172.21.15.118:60880) 2024-03-24T00:18:34.681 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:34.680+0000 7f3e49e00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f3e4c1016b0 0x7f3e4c1957f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:18:34.681 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:34.680+0000 7f3e49400640 1 -- 172.21.15.118:0/2370219655 learned_addr learned my addr 172.21.15.118:0/2370219655 (peer_addr_for_me v2:172.21.15.118:0/0) 2024-03-24T00:18:34.681 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:34.680+0000 7f3e4a800640 1 --2- 172.21.15.118:0/2370219655 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f3e4c103260 0x7f3e4c19cda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:18:34.681 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:34.680+0000 7f3e49400640 1 -- 172.21.15.118:0/2370219655 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f3e4c1016b0 msgr2=0x7f3e4c1957f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:18:34.681 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:34.680+0000 7f3e49400640 1 --2- 172.21.15.118:0/2370219655 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f3e4c1016b0 0x7f3e4c1957f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:18:34.681 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:34.680+0000 7f3e49400640 1 -- 172.21.15.118:0/2370219655 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f3e4c103260 msgr2=0x7f3e4c19cda0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:18:34.681 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:34.680+0000 7f3e49400640 1 --2- 172.21.15.118:0/2370219655 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f3e4c103260 0x7f3e4c19cda0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:18:34.681 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:34.680+0000 7f3e49400640 1 -- 172.21.15.118:0/2370219655 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3e4c19d410 con 0x7f3e4c1028a0 2024-03-24T00:18:34.681 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:34.681+0000 7f3e49400640 1 --2- 172.21.15.118:0/2370219655 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3e4c1028a0 0x7f3e4c195d30 secure :-1 s=READY pgs=331 cs=0 l=1 rev1=1 crypto rx=0x7f3e3800e9c0 tx=0x7f3e3800ee80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:18:34.682 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:34.681+0000 7f3e42a00640 1 -- 172.21.15.118:0/2370219655 <== mon.2 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3e3800cde0 con 0x7f3e4c1028a0 2024-03-24T00:18:34.682 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:34.681+0000 7f3e42a00640 1 -- 172.21.15.118:0/2370219655 <== mon.2 v2:172.21.15.189:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f3e380035f0 con 0x7f3e4c1028a0 2024-03-24T00:18:34.682 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:34.681+0000 7f3e50a00640 1 -- 172.21.15.118:0/2370219655 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3e4c19d700 con 0x7f3e4c1028a0 2024-03-24T00:18:34.682 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:34.681+0000 7f3e42a00640 1 -- 172.21.15.118:0/2370219655 <== mon.2 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3e38010690 con 0x7f3e4c1028a0 2024-03-24T00:18:34.682 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:34.681+0000 7f3e50a00640 1 -- 172.21.15.118:0/2370219655 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3e4c19dc50 con 0x7f3e4c1028a0 2024-03-24T00:18:34.684 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:34.684+0000 7f3e537b6740 1 Processor -- start 2024-03-24T00:18:34.684 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:34.684+0000 7f3e42a00640 1 -- 172.21.15.118:0/2370219655 <== mon.2 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 20) v1 ==== 137291+0+0 (secure 0 0 0) 0x7f3e380037a0 con 0x7f3e4c1028a0 2024-03-24T00:18:34.684 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:34.684+0000 7f3e537b6740 1 -- start start 2024-03-24T00:18:34.686 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:34.685+0000 7f3e537b6740 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5610ec16fe80 0x5610ec1702a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:18:34.686 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:34.685+0000 7f3e537b6740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5610ec170870 0x5610ec175370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:18:34.686 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:34.685+0000 7f3e537b6740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5610ec1759e0 0x5610ec177e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:18:34.686 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:34.685+0000 7f3e537b6740 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x5610ec0d5250 con 0x5610ec16fe80 2024-03-24T00:18:34.686 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:34.685+0000 7f3e537b6740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x5610ec07da30 con 0x5610ec170870 2024-03-24T00:18:34.686 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:34.685+0000 7f3e537b6740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x5610ec0d12f0 con 0x5610ec1759e0 2024-03-24T00:18:34.686 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:34.685+0000 7f3e49400640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5610ec1759e0 0x5610ec177e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:18:34.686 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:34.685+0000 7f3e49e00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5610ec170870 0x5610ec175370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:18:34.686 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:34.685+0000 7f3e4a800640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5610ec16fe80 0x5610ec1702a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:18:34.686 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:34.685+0000 7f3e49e00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5610ec170870 0x5610ec175370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.118:60896/0 (socket says 172.21.15.118:60896) 2024-03-24T00:18:34.686 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:34.685+0000 7f3e49e00640 1 -- 172.21.15.118:0/2198892493 learned_addr learned my addr 172.21.15.118:0/2198892493 (peer_addr_for_me v2:172.21.15.118:0/0) 2024-03-24T00:18:34.687 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:34.686+0000 7f3e49e00640 1 -- 172.21.15.118:0/2198892493 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5610ec1759e0 msgr2=0x5610ec177e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:18:34.687 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:34.686+0000 7f3e49e00640 1 --2- 172.21.15.118:0/2198892493 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5610ec1759e0 0x5610ec177e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:18:34.687 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:34.686+0000 7f3e49e00640 1 -- 172.21.15.118:0/2198892493 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5610ec16fe80 msgr2=0x5610ec1702a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:18:34.687 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:34.686+0000 7f3e49e00640 1 --2- 172.21.15.118:0/2198892493 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5610ec16fe80 0x5610ec1702a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:18:34.687 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:34.686+0000 7f3e49e00640 1 -- 172.21.15.118:0/2198892493 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5610ec068690 con 0x5610ec170870 2024-03-24T00:18:34.687 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:34.686+0000 7f3e49e00640 1 --2- 172.21.15.118:0/2198892493 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5610ec170870 0x5610ec175370 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 crypto rx=0x7f3e3c0057c0 tx=0x7f3e3c009490 comp rx=0 tx=0).ready entity=mon.2 client_cookie=bfaabe3bc6b68323 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:18:34.687 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:34.686+0000 7f3e37000640 1 -- 172.21.15.118:0/2198892493 <== mon.2 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3e3c043020 con 0x5610ec170870 2024-03-24T00:18:34.687 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:34.686+0000 7f3e37000640 1 -- 172.21.15.118:0/2198892493 <== mon.2 v2:172.21.15.189:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f3e3c0323c0 con 0x5610ec170870 2024-03-24T00:18:34.687 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:34.686+0000 7f3e37000640 1 -- 172.21.15.118:0/2198892493 <== mon.2 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3e3c039040 con 0x5610ec170870 2024-03-24T00:18:34.691 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:34.687+0000 7f3e42a00640 1 --2- 172.21.15.118:0/2370219655 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7f3e300a2800 0x7f3e300a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:18:34.691 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:34.687+0000 7f3e537b6740 1 -- 172.21.15.118:0/2198892493 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5610ec170870 msgr2=0x5610ec175370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:18:34.691 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:34.687+0000 7f3e537b6740 1 --2- 172.21.15.118:0/2198892493 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5610ec170870 0x5610ec175370 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 crypto rx=0x7f3e3c0057c0 tx=0x7f3e3c009490 comp rx=0 tx=0).stop 2024-03-24T00:18:34.691 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:34.687+0000 7f3e42a00640 1 -- 172.21.15.118:0/2370219655 <== mon.2 v2:172.21.15.189:3300/0 5 ==== osd_map(90..90 src has 1..90) v4 ==== 8277+0+0 (secure 0 0 0) 0x7f3e380cee20 con 0x7f3e4c1028a0 2024-03-24T00:18:34.691 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:34.687+0000 7f3e4a800640 1 --2- 172.21.15.118:0/2370219655 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7f3e300a2800 0x7f3e300a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:18:34.691 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:34.687+0000 7f3e537b6740 1 -- 172.21.15.118:0/2198892493 shutdown_connections 2024-03-24T00:18:34.691 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:34.687+0000 7f3e537b6740 1 --2- 172.21.15.118:0/2198892493 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5610ec1759e0 0x5610ec177e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:18:34.692 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:34.687+0000 7f3e537b6740 1 --2- 172.21.15.118:0/2198892493 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5610ec170870 0x5610ec175370 unknown :-1 s=CLOSED pgs=332 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:18:34.692 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:34.687+0000 7f3e537b6740 1 --2- 172.21.15.118:0/2198892493 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5610ec16fe80 0x5610ec1702a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:18:34.692 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:34.687+0000 7f3e537b6740 1 -- 172.21.15.118:0/2198892493 >> 172.21.15.118:0/2198892493 conn(0x5610ec138200 msgr2=0x5610ec16b7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:18:34.692 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:34.687+0000 7f3e537b6740 1 -- 172.21.15.118:0/2198892493 shutdown_connections 2024-03-24T00:18:34.692 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:34.687+0000 7f3e537b6740 1 -- 172.21.15.118:0/2198892493 wait complete. 2024-03-24T00:18:34.692 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:34.688+0000 7f3e4a800640 1 --2- 172.21.15.118:0/2370219655 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7f3e300a2800 0x7f3e300a4cc0 secure :-1 s=READY pgs=319 cs=0 l=1 rev1=1 crypto rx=0x7f3e4400d620 tx=0x7f3e4400f040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:18:34.692 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:34.688+0000 7f3e537b6740 1 -- start start 2024-03-24T00:18:34.692 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:34.688+0000 7f3e537b6740 1 Processor -- start 2024-03-24T00:18:34.692 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:34.689+0000 7f3e537b6740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5610ec16fe80 0x5610ec179e00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:18:34.692 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:34.689+0000 7f3e537b6740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5610ec170870 0x5610ec17a340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:18:34.692 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:34.689+0000 7f3e537b6740 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5610ec1759e0 0x5610ec181310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:18:34.692 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:34.689+0000 7f3e537b6740 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x5610ec178930 con 0x5610ec1759e0 2024-03-24T00:18:34.692 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:34.689+0000 7f3e537b6740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x5610ec090e80 con 0x5610ec170870 2024-03-24T00:18:34.693 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:34.689+0000 7f3e537b6740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x5610ec178ab0 con 0x5610ec16fe80 2024-03-24T00:18:34.693 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:34.689+0000 7f3e537b6740 10 client.0 _ll_register_callbacks cb 0x5610ec0a3130 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T00:18:34.693 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:34.689+0000 7f3e49400640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5610ec170870 0x5610ec17a340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:18:34.693 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:34.689+0000 7f3e49e00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5610ec16fe80 0x5610ec179e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:18:34.693 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:34.689+0000 7f3e49400640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5610ec170870 0x5610ec17a340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.118:60904/0 (socket says 172.21.15.118:60904) 2024-03-24T00:18:34.693 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:34.689+0000 7f3e49e00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5610ec16fe80 0x5610ec179e00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.118:52850/0 (socket says 172.21.15.118:52850) 2024-03-24T00:18:34.693 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:34.689+0000 7f3e49400640 1 -- 172.21.15.118:0/2864413122 learned_addr learned my addr 172.21.15.118:0/2864413122 (peer_addr_for_me v2:172.21.15.118:0/0) 2024-03-24T00:18:34.693 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:34.689+0000 7f3e4a800640 1 --2- 172.21.15.118:0/2864413122 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5610ec1759e0 0x5610ec181310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:18:34.693 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:34.689+0000 7f3e49e00640 1 -- 172.21.15.118:0/2864413122 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5610ec170870 msgr2=0x5610ec17a340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:18:34.693 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:34.689+0000 7f3e49e00640 1 --2- 172.21.15.118:0/2864413122 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5610ec170870 0x5610ec17a340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:18:34.693 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:34.689+0000 7f3e49e00640 1 -- 172.21.15.118:0/2864413122 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5610ec1759e0 msgr2=0x5610ec181310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:18:34.693 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:34.690+0000 7f3e49e00640 1 --2- 172.21.15.118:0/2864413122 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5610ec1759e0 0x5610ec181310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:18:34.694 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:34.690+0000 7f3e49e00640 1 -- 172.21.15.118:0/2864413122 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5610ebeb37d0 con 0x5610ec16fe80 2024-03-24T00:18:34.694 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:34.690+0000 7f3e49400640 1 --2- 172.21.15.118:0/2864413122 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5610ec170870 0x5610ec17a340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T00:18:34.694 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:34.690+0000 7f3e4a800640 1 --2- 172.21.15.118:0/2864413122 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5610ec1759e0 0x5610ec181310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T00:18:34.694 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:34.690+0000 7f3e49e00640 1 --2- 172.21.15.118:0/2864413122 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5610ec16fe80 0x5610ec179e00 secure :-1 s=READY pgs=336 cs=0 l=1 rev1=1 crypto rx=0x5610ec16c740 tx=0x7f3e3c005f30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c2debf33503db61e server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:18:34.694 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:34.690+0000 7f3e2b400640 10 client.0 ms_handle_connect on v2:172.21.15.193:3300/0 2024-03-24T00:18:34.694 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:34.690+0000 7f3e2b400640 1 -- 172.21.15.118:0/2864413122 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3e3c043030 con 0x5610ec16fe80 2024-03-24T00:18:34.694 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:34.690+0000 7f3e2b400640 1 -- 172.21.15.118:0/2864413122 <== mon.1 v2:172.21.15.193:3300/0 2 ==== osd_map(90..90 src has 1..90) v4 ==== 8277+0+0 (secure 0 0 0) 0x7f3e3c034440 con 0x5610ec16fe80 2024-03-24T00:18:34.694 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:34.690+0000 7f3e537b6740 1 -- 172.21.15.118:0/2864413122 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f3e3c0323c0 con 0x5610ec16fe80 2024-03-24T00:18:34.694 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:34.690+0000 7f3e2b400640 1 -- 172.21.15.118:0/2864413122 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) v2 ==== 24+0+0 (secure 0 0 0) 0x7f3e3c05a050 con 0x5610ec16fe80 2024-03-24T00:18:34.694 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:34.691+0000 7f3e537b6740 10 client.26197 fetch_fsmap learned FSMap version 28 2024-03-24T00:18:34.694 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:34.691+0000 7f3e537b6740 1 -- 172.21.15.118:0/2864413122 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) v3 -- 0x5610ebde93f0 con 0x5610ec16fe80 2024-03-24T00:18:34.695 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:34.691+0000 7f3e2b400640 1 -- 172.21.15.118:0/2864413122 <== mon.1 v2:172.21.15.193:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f3e3c0323c0 con 0x5610ec16fe80 2024-03-24T00:18:34.695 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:34.691+0000 7f3e2b400640 1 -- 172.21.15.118:0/2864413122 <== mon.1 v2:172.21.15.193:3300/0 5 ==== fsmap(e 28) v1 ==== 7934+0+0 (secure 0 0 0) 0x7f3e3c032ab0 con 0x5610ec16fe80 2024-03-24T00:18:34.695 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:34.691+0000 7f3e537b6740 10 client.26197 fetch_fsmap finished waiting for FSMap version 28 2024-03-24T00:18:34.695 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:34.691+0000 7f3e537b6740 10 client.26197 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-24T00:18:34.695 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:34.691+0000 7f3e537b6740 20 client.26197 populate_metadata read hostname 'smithi118' 2024-03-24T00:18:34.695 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:34.691+0000 7f3e537b6740 1 --2- 172.21.15.118:0/2864413122 >> [v2:172.21.15.189:6838/4074484168,v1:172.21.15.189:6839/4074484168] conn(0x5610ec185800 0x5610ec187bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:18:34.695 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:34.691+0000 7f3e537b6740 4 client.26197 mds_command: new command op to 24569 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-24T00:18:34.695 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:34.691+0000 7f3e537b6740 1 -- 172.21.15.118:0/2864413122 --> [v2:172.21.15.189:6838/4074484168,v1:172.21.15.189:6839/4074484168] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x5610ec0e2790 con 0x5610ec185800 2024-03-24T00:18:34.695 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:34.691+0000 7f3e49400640 1 --2- 172.21.15.118:0/2864413122 >> [v2:172.21.15.189:6838/4074484168,v1:172.21.15.189:6839/4074484168] conn(0x5610ec185800 0x5610ec187bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:18:34.696 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:34.692+0000 7f3e49400640 1 --2- 172.21.15.118:0/2864413122 >> [v2:172.21.15.189:6838/4074484168,v1:172.21.15.189:6839/4074484168] conn(0x5610ec185800 0x5610ec187bf0 crc :-1 s=READY pgs=93 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=c4c390de73ad1623 server_cookie=dae41bf58042267b in_seq=0 out_seq=0 2024-03-24T00:18:34.696 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:34.692+0000 7f3e2b400640 10 client.26197 ms_handle_connect on v2:172.21.15.189:6838/4074484168 2024-03-24T00:18:34.697 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:34.694+0000 7f3e2b400640 1 -- 172.21.15.118:0/2864413122 <== mds.0 v2:172.21.15.189:6838/4074484168 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x5610ec0e2790 con 0x5610ec185800 2024-03-24T00:18:34.697 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:34.694+0000 7f3e2b400640 10 client.26197 handle_command_reply: tid=0 2024-03-24T00:18:34.697 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:34.694+0000 7f3e537b6740 1 client.26197 shutdown 2024-03-24T00:18:34.697 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:34.694+0000 7f3e537b6740 1 -- 172.21.15.118:0/2864413122 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5610ec16fe80 msgr2=0x5610ec179e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:18:34.697 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:34.694+0000 7f3e537b6740 1 --2- 172.21.15.118:0/2864413122 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5610ec16fe80 0x5610ec179e00 secure :-1 s=READY pgs=336 cs=0 l=1 rev1=1 crypto rx=0x5610ec16c740 tx=0x7f3e3c005f30 comp rx=0 tx=0).stop 2024-03-24T00:18:34.697 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:34.694+0000 7f3e537b6740 1 -- 172.21.15.118:0/2864413122 shutdown_connections 2024-03-24T00:18:34.697 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:34.694+0000 7f3e537b6740 1 --2- 172.21.15.118:0/2864413122 >> [v2:172.21.15.189:6838/4074484168,v1:172.21.15.189:6839/4074484168] conn(0x5610ec185800 0x5610ec187bf0 crc :-1 s=READY pgs=93 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:18:34.697 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:34.694+0000 7f3e537b6740 1 --2- 172.21.15.118:0/2864413122 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5610ec1759e0 0x5610ec181310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:18:34.697 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:34.694+0000 7f3e537b6740 1 --2- 172.21.15.118:0/2864413122 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5610ec170870 0x5610ec17a340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:18:34.697 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:34.694+0000 7f3e2b400640 0 client.26197 ms_handle_reset on v2:172.21.15.189:6838/4074484168 2024-03-24T00:18:34.697 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:34.694+0000 7f3e537b6740 1 --2- 172.21.15.118:0/2864413122 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5610ec16fe80 0x5610ec179e00 unknown :-1 s=CLOSED pgs=336 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:18:34.697 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:34.694+0000 7f3e537b6740 1 -- 172.21.15.118:0/2864413122 >> 172.21.15.118:0/2864413122 conn(0x5610ec138200 msgr2=0x5610ec16aa90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:18:34.697 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:34.695+0000 7f3e537b6740 1 -- 172.21.15.118:0/2864413122 shutdown_connections 2024-03-24T00:18:34.697 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:34.695+0000 7f3e537b6740 1 -- 172.21.15.118:0/2864413122 wait complete. 2024-03-24T00:18:34.698 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:34.695+0000 7f3e537b6740 20 client.26197 trim_cache size 0 max 16384 2024-03-24T00:18:34.713 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:34.712+0000 7f3e537b6740 1 Processor -- start 2024-03-24T00:18:34.713 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:34.712+0000 7f3e537b6740 1 -- start start 2024-03-24T00:18:34.713 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:34.713+0000 7f3e537b6740 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5610ec16fe80 0x5610ec171db0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:18:34.713 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:34.713+0000 7f3e537b6740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5610ec170870 0x5610ec1812d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:18:34.713 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:34.713+0000 7f3e537b6740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5610ec1759e0 0x5610ec18c3a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:18:34.714 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:34.713+0000 7f3e537b6740 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3e4c1a67f0 con 0x5610ec16fe80 2024-03-24T00:18:34.714 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:34.713+0000 7f3e537b6740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3e4c1a8170 con 0x5610ec1759e0 2024-03-24T00:18:34.714 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:34.713+0000 7f3e537b6740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3e4c1a6e50 con 0x5610ec170870 2024-03-24T00:18:34.714 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:34.713+0000 7f3e49400640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5610ec170870 0x5610ec1812d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:18:34.714 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:34.713+0000 7f3e49e00640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5610ec16fe80 0x5610ec171db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:18:34.714 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:34.713+0000 7f3e49400640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5610ec170870 0x5610ec1812d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.118:52858/0 (socket says 172.21.15.118:52858) 2024-03-24T00:18:34.714 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:34.713+0000 7f3e49400640 1 -- 172.21.15.118:0/3611369685 learned_addr learned my addr 172.21.15.118:0/3611369685 (peer_addr_for_me v2:172.21.15.118:0/0) 2024-03-24T00:18:34.714 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:34.713+0000 7f3e4a800640 1 --2- 172.21.15.118:0/3611369685 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5610ec1759e0 0x5610ec18c3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:18:34.714 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:34.713+0000 7f3e49400640 1 -- 172.21.15.118:0/3611369685 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5610ec1759e0 msgr2=0x5610ec18c3a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:18:34.714 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:34.713+0000 7f3e49400640 1 --2- 172.21.15.118:0/3611369685 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5610ec1759e0 0x5610ec18c3a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:18:34.714 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:34.713+0000 7f3e49400640 1 -- 172.21.15.118:0/3611369685 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5610ec16fe80 msgr2=0x5610ec171db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:18:34.714 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:34.713+0000 7f3e49400640 1 --2- 172.21.15.118:0/3611369685 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5610ec16fe80 0x5610ec171db0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:18:34.714 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:34.713+0000 7f3e49400640 1 -- 172.21.15.118:0/3611369685 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5610ebda90f0 con 0x5610ec170870 2024-03-24T00:18:34.714 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:34.713+0000 7f3e49e00640 1 --2- 172.21.15.118:0/3611369685 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5610ec16fe80 0x5610ec171db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T00:18:34.715 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:34.714+0000 7f3e49400640 1 --2- 172.21.15.118:0/3611369685 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5610ec170870 0x5610ec1812d0 secure :-1 s=READY pgs=337 cs=0 l=1 rev1=1 crypto rx=0x7f3e38009230 tx=0x7f3e380d1270 comp rx=0 tx=0).ready entity=mon.1 client_cookie=e836bbc921ede36e server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:18:34.715 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:34.714+0000 7f3e36600640 1 -- 172.21.15.118:0/3611369685 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3e3809b500 con 0x5610ec170870 2024-03-24T00:18:34.715 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:34.714+0000 7f3e36600640 1 -- 172.21.15.118:0/3611369685 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f3e3809be30 con 0x5610ec170870 2024-03-24T00:18:34.715 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:34.714+0000 7f3e36600640 1 -- 172.21.15.118:0/3611369685 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3e3809b090 con 0x5610ec170870 2024-03-24T00:18:34.718 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:34.715+0000 7f3e537b6740 1 -- 172.21.15.118:0/3611369685 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5610ec170870 msgr2=0x5610ec1812d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:18:34.718 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:34.715+0000 7f3e537b6740 1 --2- 172.21.15.118:0/3611369685 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5610ec170870 0x5610ec1812d0 secure :-1 s=READY pgs=337 cs=0 l=1 rev1=1 crypto rx=0x7f3e38009230 tx=0x7f3e380d1270 comp rx=0 tx=0).stop 2024-03-24T00:18:34.718 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:34.715+0000 7f3e537b6740 1 -- 172.21.15.118:0/3611369685 shutdown_connections 2024-03-24T00:18:34.718 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:34.715+0000 7f3e537b6740 1 --2- 172.21.15.118:0/3611369685 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5610ec1759e0 0x5610ec18c3a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:18:34.718 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:34.715+0000 7f3e537b6740 1 --2- 172.21.15.118:0/3611369685 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5610ec170870 0x5610ec1812d0 unknown :-1 s=CLOSED pgs=337 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:18:34.718 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:34.715+0000 7f3e537b6740 1 --2- 172.21.15.118:0/3611369685 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5610ec16fe80 0x5610ec171db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:18:34.718 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:34.715+0000 7f3e537b6740 1 -- 172.21.15.118:0/3611369685 >> 172.21.15.118:0/3611369685 conn(0x5610ec138200 msgr2=0x5610ec16aa90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:18:34.719 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:34.715+0000 7f3e537b6740 1 -- 172.21.15.118:0/3611369685 shutdown_connections 2024-03-24T00:18:34.719 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:34.715+0000 7f3e537b6740 1 -- 172.21.15.118:0/3611369685 wait complete. 2024-03-24T00:18:34.719 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:34.715+0000 7f3e537b6740 1 -- start start 2024-03-24T00:18:34.719 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:34.715+0000 7f3e537b6740 1 Processor -- start 2024-03-24T00:18:34.719 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:34.716+0000 7f3e537b6740 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5610ec16fe80 0x5610ec17eba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:18:34.719 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:34.716+0000 7f3e537b6740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5610ec170870 0x5610ec17f0e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:18:34.719 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:34.716+0000 7f3e537b6740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5610ec1759e0 0x5610ec1800f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:18:34.719 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:34.716+0000 7f3e537b6740 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3e4c1a7b10 con 0x5610ec16fe80 2024-03-24T00:18:34.719 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:34.716+0000 7f3e537b6740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3e4c1a74b0 con 0x5610ec1759e0 2024-03-24T00:18:34.719 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:34.716+0000 7f3e537b6740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3e4c1a49b0 con 0x5610ec170870 2024-03-24T00:18:34.719 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:34.716+0000 7f3e537b6740 10 client.0 _ll_register_callbacks cb 0x7f3e380d2730 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T00:18:34.719 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:34.716+0000 7f3e49400640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5610ec170870 0x5610ec17f0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:18:34.719 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:34.716+0000 7f3e4a800640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5610ec1759e0 0x5610ec1800f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:18:34.719 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:34.716+0000 7f3e49e00640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5610ec16fe80 0x5610ec17eba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:18:34.720 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:34.716+0000 7f3e49400640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5610ec170870 0x5610ec17f0e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.118:52868/0 (socket says 172.21.15.118:52868) 2024-03-24T00:18:34.720 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:34.716+0000 7f3e4a800640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5610ec1759e0 0x5610ec1800f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.118:60934/0 (socket says 172.21.15.118:60934) 2024-03-24T00:18:34.720 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:34.716+0000 7f3e49400640 1 -- 172.21.15.118:0/3959417471 learned_addr learned my addr 172.21.15.118:0/3959417471 (peer_addr_for_me v2:172.21.15.118:0/0) 2024-03-24T00:18:34.720 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:34.717+0000 7f3e49400640 1 -- 172.21.15.118:0/3959417471 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5610ec1759e0 msgr2=0x5610ec1800f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:18:34.720 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:34.717+0000 7f3e49400640 1 --2- 172.21.15.118:0/3959417471 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5610ec1759e0 0x5610ec1800f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:18:34.720 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:34.717+0000 7f3e49400640 1 -- 172.21.15.118:0/3959417471 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5610ec16fe80 msgr2=0x5610ec17eba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:18:34.720 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:34.717+0000 7f3e49400640 1 --2- 172.21.15.118:0/3959417471 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5610ec16fe80 0x5610ec17eba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:18:34.720 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:34.717+0000 7f3e49400640 1 -- 172.21.15.118:0/3959417471 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5610ec1806e0 con 0x5610ec170870 2024-03-24T00:18:34.720 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:34.717+0000 7f3e49400640 1 --2- 172.21.15.118:0/3959417471 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5610ec170870 0x5610ec17f0e0 secure :-1 s=READY pgs=338 cs=0 l=1 rev1=1 crypto rx=0x7f3e380a39a0 tx=0x7f3e38097a90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=765ffaded5206321 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:18:34.720 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:34.717+0000 7f3e2b400640 10 client.0 ms_handle_connect on v2:172.21.15.193:3300/0 2024-03-24T00:18:34.720 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:34.717+0000 7f3e2b400640 1 -- 172.21.15.118:0/3959417471 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3e380d7070 con 0x5610ec170870 2024-03-24T00:18:34.720 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:34.717+0000 7f3e2b400640 1 -- 172.21.15.118:0/3959417471 <== mon.1 v2:172.21.15.193:3300/0 2 ==== osd_map(90..90 src has 1..90) v4 ==== 8277+0+0 (secure 0 0 0) 0x7f3e38003b10 con 0x5610ec170870 2024-03-24T00:18:34.720 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:34.717+0000 7f3e537b6740 1 -- 172.21.15.118:0/3959417471 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f3e3809be30 con 0x5610ec170870 2024-03-24T00:18:34.721 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:34.717+0000 7f3e2b400640 1 -- 172.21.15.118:0/3959417471 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) v2 ==== 24+0+0 (secure 0 0 0) 0x7f3e380b3070 con 0x5610ec170870 2024-03-24T00:18:34.721 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:34.717+0000 7f3e537b6740 10 client.26203 fetch_fsmap learned FSMap version 28 2024-03-24T00:18:34.721 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:34.717+0000 7f3e537b6740 1 -- 172.21.15.118:0/3959417471 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) v3 -- 0x5610ec1e1090 con 0x5610ec170870 2024-03-24T00:18:34.721 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:34.718+0000 7f3e2b400640 1 -- 172.21.15.118:0/3959417471 <== mon.1 v2:172.21.15.193:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f3e3809be30 con 0x5610ec170870 2024-03-24T00:18:34.721 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:34.718+0000 7f3e2b400640 1 -- 172.21.15.118:0/3959417471 <== mon.1 v2:172.21.15.193:3300/0 5 ==== fsmap(e 28) v1 ==== 7934+0+0 (secure 0 0 0) 0x7f3e3809ac10 con 0x5610ec170870 2024-03-24T00:18:34.721 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:34.718+0000 7f3e537b6740 10 client.26203 fetch_fsmap finished waiting for FSMap version 28 2024-03-24T00:18:34.721 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:34.718+0000 7f3e537b6740 10 client.26203 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-24T00:18:34.721 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:34.718+0000 7f3e537b6740 20 client.26203 populate_metadata read hostname 'smithi118' 2024-03-24T00:18:34.721 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:34.718+0000 7f3e537b6740 1 --2- 172.21.15.118:0/3959417471 >> [v2:172.21.15.189:6838/4074484168,v1:172.21.15.189:6839/4074484168] conn(0x5610ec1e1a20 0x5610ec1e3e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:18:34.721 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:34.718+0000 7f3e537b6740 4 client.26203 mds_command: new command op to 24569 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-24T00:18:34.721 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:34.718+0000 7f3e537b6740 1 -- 172.21.15.118:0/3959417471 --> [v2:172.21.15.189:6838/4074484168,v1:172.21.15.189:6839/4074484168] -- command(tid 0: {"prefix": "scrub status"}) v1 -- 0x5610ec0dcae0 con 0x5610ec1e1a20 2024-03-24T00:18:34.721 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:34.718+0000 7f3e49e00640 1 --2- 172.21.15.118:0/3959417471 >> [v2:172.21.15.189:6838/4074484168,v1:172.21.15.189:6839/4074484168] conn(0x5610ec1e1a20 0x5610ec1e3e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:18:34.721 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:34.718+0000 7f3e49e00640 1 --2- 172.21.15.118:0/3959417471 >> [v2:172.21.15.189:6838/4074484168,v1:172.21.15.189:6839/4074484168] conn(0x5610ec1e1a20 0x5610ec1e3e40 crc :-1 s=READY pgs=94 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=7bef36f1bc457f53 server_cookie=e39eb4020200ca32 in_seq=0 out_seq=0 2024-03-24T00:18:34.721 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:34.719+0000 7f3e2b400640 10 client.26203 ms_handle_connect on v2:172.21.15.189:6838/4074484168 2024-03-24T00:18:34.723 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:34.720+0000 7f3e2b400640 1 -- 172.21.15.118:0/3959417471 <== mds.0 v2:172.21.15.189:6838/4074484168 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+63 (crc 0 0 0) 0x5610ec0dcae0 con 0x5610ec1e1a20 2024-03-24T00:18:34.723 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:34.720+0000 7f3e2b400640 10 client.26203 handle_command_reply: tid=0 2024-03-24T00:18:34.723 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:34.720+0000 7f3e537b6740 1 client.26203 shutdown 2024-03-24T00:18:34.723 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:34.720+0000 7f3e537b6740 1 -- 172.21.15.118:0/3959417471 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5610ec170870 msgr2=0x5610ec17f0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:18:34.723 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:34.720+0000 7f3e537b6740 1 --2- 172.21.15.118:0/3959417471 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5610ec170870 0x5610ec17f0e0 secure :-1 s=READY pgs=338 cs=0 l=1 rev1=1 crypto rx=0x7f3e380a39a0 tx=0x7f3e38097a90 comp rx=0 tx=0).stop 2024-03-24T00:18:34.723 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:34.720+0000 7f3e537b6740 1 -- 172.21.15.118:0/3959417471 shutdown_connections 2024-03-24T00:18:34.723 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:34.720+0000 7f3e537b6740 1 --2- 172.21.15.118:0/3959417471 >> [v2:172.21.15.189:6838/4074484168,v1:172.21.15.189:6839/4074484168] conn(0x5610ec1e1a20 0x5610ec1e3e40 crc :-1 s=READY pgs=94 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:18:34.723 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:34.720+0000 7f3e537b6740 1 --2- 172.21.15.118:0/3959417471 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5610ec1759e0 0x5610ec1800f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:18:34.724 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:34.720+0000 7f3e537b6740 1 --2- 172.21.15.118:0/3959417471 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5610ec170870 0x5610ec17f0e0 unknown :-1 s=CLOSED pgs=338 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:18:34.724 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:34.720+0000 7f3e537b6740 1 --2- 172.21.15.118:0/3959417471 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5610ec16fe80 0x5610ec17eba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:18:34.724 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:34.720+0000 7f3e2b400640 0 client.26203 ms_handle_reset on v2:172.21.15.189:6838/4074484168 2024-03-24T00:18:34.724 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:34.720+0000 7f3e537b6740 1 -- 172.21.15.118:0/3959417471 >> 172.21.15.118:0/3959417471 conn(0x5610ec138200 msgr2=0x5610ec16ea30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:18:34.724 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:34.721+0000 7f3e537b6740 1 -- 172.21.15.118:0/3959417471 shutdown_connections 2024-03-24T00:18:34.724 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:34.721+0000 7f3e537b6740 1 -- 172.21.15.118:0/3959417471 wait complete. 2024-03-24T00:18:34.724 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:34.721+0000 7f3e537b6740 20 client.26203 trim_cache size 0 max 16384 2024-03-24T00:18:34.724 INFO:teuthology.orchestra.run.smithi118.stdout:{ 2024-03-24T00:18:34.724 INFO:teuthology.orchestra.run.smithi118.stdout: "status": "no active scrubs running", 2024-03-24T00:18:34.724 INFO:teuthology.orchestra.run.smithi118.stdout: "scrubs": {} 2024-03-24T00:18:34.725 INFO:teuthology.orchestra.run.smithi118.stdout:} 2024-03-24T00:18:34.725 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:34.722+0000 7f3e50a00640 1 -- 172.21.15.118:0/2370219655 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7f3e300a2800 msgr2=0x7f3e300a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:18:34.725 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:34.722+0000 7f3e50a00640 1 --2- 172.21.15.118:0/2370219655 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7f3e300a2800 0x7f3e300a4cc0 secure :-1 s=READY pgs=319 cs=0 l=1 rev1=1 crypto rx=0x7f3e4400d620 tx=0x7f3e4400f040 comp rx=0 tx=0).stop 2024-03-24T00:18:34.725 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:34.722+0000 7f3e50a00640 1 -- 172.21.15.118:0/2370219655 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3e4c1028a0 msgr2=0x7f3e4c195d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:18:34.725 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:34.722+0000 7f3e50a00640 1 --2- 172.21.15.118:0/2370219655 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3e4c1028a0 0x7f3e4c195d30 secure :-1 s=READY pgs=331 cs=0 l=1 rev1=1 crypto rx=0x7f3e3800e9c0 tx=0x7f3e3800ee80 comp rx=0 tx=0).stop 2024-03-24T00:18:34.725 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:34.722+0000 7f3e50a00640 1 -- 172.21.15.118:0/2370219655 shutdown_connections 2024-03-24T00:18:34.726 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:34.722+0000 7f3e50a00640 1 --2- 172.21.15.118:0/2370219655 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7f3e300a2800 0x7f3e300a4cc0 unknown :-1 s=CLOSED pgs=319 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:18:34.726 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:34.722+0000 7f3e50a00640 1 --2- 172.21.15.118:0/2370219655 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f3e4c103260 0x7f3e4c19cda0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:18:34.726 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:34.722+0000 7f3e50a00640 1 --2- 172.21.15.118:0/2370219655 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3e4c1028a0 0x7f3e4c195d30 unknown :-1 s=CLOSED pgs=331 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:18:34.726 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:34.722+0000 7f3e50a00640 1 --2- 172.21.15.118:0/2370219655 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f3e4c1016b0 0x7f3e4c1957f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:18:34.726 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:34.722+0000 7f3e50a00640 1 -- 172.21.15.118:0/2370219655 >> 172.21.15.118:0/2370219655 conn(0x7f3e4c0fcc40 msgr2=0x7f3e4c105c20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:18:34.726 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:34.722+0000 7f3e50a00640 1 -- 172.21.15.118:0/2370219655 shutdown_connections 2024-03-24T00:18:34.726 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:34.722+0000 7f3e50a00640 1 -- 172.21.15.118:0/2370219655 wait complete. 2024-03-24T00:18:34.739 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-03-24T00:18:34.739 DEBUG:teuthology.orchestra.run.smithi118:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-24T00:18:34.822 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:18:34 smithi118 ceph-mon[31535]: pgmap v484: 97 pgs: 97 active+clean; 45 MiB data, 483 MiB used, 1.0 TiB / 1.0 TiB avail; 524 KiB/s rd, 1.8 MiB/s wr, 358 op/s 2024-03-24T00:18:34.957 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:34.957+0000 7f09ea400640 1 -- 172.21.15.118:0/3750253338 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f09e41028b0 msgr2=0x7f09e4102d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:18:34.957 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:34.957+0000 7f09ea400640 1 --2- 172.21.15.118:0/3750253338 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f09e41028b0 0x7f09e4102d30 secure :-1 s=READY pgs=333 cs=0 l=1 rev1=1 crypto rx=0x7f09d0009900 tx=0x7f09d002bc30 comp rx=0 tx=0).stop 2024-03-24T00:18:34.957 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:34.957+0000 7f09ea400640 1 -- 172.21.15.118:0/3750253338 shutdown_connections 2024-03-24T00:18:34.957 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:34.957+0000 7f09ea400640 1 --2- 172.21.15.118:0/3750253338 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f09e4103270 0x7f09e4109b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:18:34.957 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:34.957+0000 7f09ea400640 1 --2- 172.21.15.118:0/3750253338 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f09e41028b0 0x7f09e4102d30 unknown :-1 s=CLOSED pgs=333 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:18:34.957 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:34.957+0000 7f09ea400640 1 --2- 172.21.15.118:0/3750253338 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f09e41016c0 0x7f09e4101ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:18:34.957 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:34.957+0000 7f09ea400640 1 -- 172.21.15.118:0/3750253338 >> 172.21.15.118:0/3750253338 conn(0x7f09e40fcc50 msgr2=0x7f09e40ff070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:18:34.957 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:34.957+0000 7f09ea400640 1 -- 172.21.15.118:0/3750253338 shutdown_connections 2024-03-24T00:18:34.958 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:34.957+0000 7f09ea400640 1 -- 172.21.15.118:0/3750253338 wait complete. 2024-03-24T00:18:34.959 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:34.958+0000 7f09ea400640 1 Processor -- start 2024-03-24T00:18:34.959 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:34.958+0000 7f09ea400640 1 -- start start 2024-03-24T00:18:34.959 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:34.959+0000 7f09ea400640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f09e41016c0 0x7f09e4075450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:18:34.959 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:34.959+0000 7f09ea400640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f09e41028b0 0x7f09e4073a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:18:34.959 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:34.959+0000 7f09ea400640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f09e4103270 0x7f09e4073fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:18:34.959 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:34.959+0000 7f09ea400640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f09e410d340 con 0x7f09e4103270 2024-03-24T00:18:34.959 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:34.959+0000 7f09ea400640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f09e410d1c0 con 0x7f09e41016c0 2024-03-24T00:18:34.959 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:34.959+0000 7f09ea400640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f09e410d4c0 con 0x7f09e41028b0 2024-03-24T00:18:34.959 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:34.959+0000 7f09e3400640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f09e41016c0 0x7f09e4075450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:18:34.959 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:34.959+0000 7f09e3400640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f09e41016c0 0x7f09e4075450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.118:60952/0 (socket says 172.21.15.118:60952) 2024-03-24T00:18:34.960 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:34.959+0000 7f09e3400640 1 -- 172.21.15.118:0/1085924255 learned_addr learned my addr 172.21.15.118:0/1085924255 (peer_addr_for_me v2:172.21.15.118:0/0) 2024-03-24T00:18:34.960 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:34.959+0000 7f09e2a00640 1 --2- 172.21.15.118:0/1085924255 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f09e41028b0 0x7f09e4073a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:18:34.960 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:34.959+0000 7f09e3e00640 1 --2- 172.21.15.118:0/1085924255 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f09e4103270 0x7f09e4073fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:18:34.961 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:34.960+0000 7f09e2a00640 1 -- 172.21.15.118:0/1085924255 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f09e41016c0 msgr2=0x7f09e4075450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:18:34.961 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:34.960+0000 7f09e2a00640 1 --2- 172.21.15.118:0/1085924255 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f09e41016c0 0x7f09e4075450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:18:34.961 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:34.960+0000 7f09e2a00640 1 -- 172.21.15.118:0/1085924255 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f09e4103270 msgr2=0x7f09e4073fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:18:34.961 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:34.960+0000 7f09e2a00640 1 --2- 172.21.15.118:0/1085924255 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f09e4103270 0x7f09e4073fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:18:34.961 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:34.960+0000 7f09e2a00640 1 -- 172.21.15.118:0/1085924255 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f09e4074860 con 0x7f09e41028b0 2024-03-24T00:18:34.962 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:34.960+0000 7f09e2a00640 1 --2- 172.21.15.118:0/1085924255 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f09e41028b0 0x7f09e4073a90 secure :-1 s=READY pgs=339 cs=0 l=1 rev1=1 crypto rx=0x7f09d0009230 tx=0x7f09d002c650 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:18:34.962 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:34.960+0000 7f09dbe00640 1 -- 172.21.15.118:0/1085924255 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f09d0041070 con 0x7f09e41028b0 2024-03-24T00:18:34.962 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:34.961+0000 7f09dbe00640 1 -- 172.21.15.118:0/1085924255 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f09d0004450 con 0x7f09e41028b0 2024-03-24T00:18:34.962 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:34.961+0000 7f09ea400640 1 -- 172.21.15.118:0/1085924255 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f09e41a1a50 con 0x7f09e41028b0 2024-03-24T00:18:34.962 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:34.961+0000 7f09dbe00640 1 -- 172.21.15.118:0/1085924255 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f09d0003df0 con 0x7f09e41028b0 2024-03-24T00:18:34.962 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:34.961+0000 7f09ea400640 1 -- 172.21.15.118:0/1085924255 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f09e41a1fa0 con 0x7f09e41028b0 2024-03-24T00:18:34.962 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:34.962+0000 7f09ea400640 1 -- 172.21.15.118:0/1085924255 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f09e41089e0 con 0x7f09e41028b0 2024-03-24T00:18:34.964 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:34.965+0000 7f09dbe00640 1 -- 172.21.15.118:0/1085924255 <== mon.1 v2:172.21.15.193:3300/0 4 ==== mgrmap(e 20) v1 ==== 137291+0+0 (secure 0 0 0) 0x7f09d0044070 con 0x7f09e41028b0 2024-03-24T00:18:34.966 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:34.966+0000 7f09dbe00640 1 --2- 172.21.15.118:0/1085924255 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7f09c80a2800 0x7f09c80a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:18:34.966 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:34.966+0000 7f09dbe00640 1 -- 172.21.15.118:0/1085924255 <== mon.1 v2:172.21.15.193:3300/0 5 ==== osd_map(90..90 src has 1..90) v4 ==== 8277+0+0 (secure 0 0 0) 0x7f09d00f05d0 con 0x7f09e41028b0 2024-03-24T00:18:34.966 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:34.966+0000 7f09e3400640 1 --2- 172.21.15.118:0/1085924255 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7f09c80a2800 0x7f09c80a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:18:34.967 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:34.967+0000 7f09e3400640 1 --2- 172.21.15.118:0/1085924255 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7f09c80a2800 0x7f09c80a4cc0 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 crypto rx=0x7f09d40058c0 tx=0x7f09d4005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:18:34.971 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:34.972+0000 7f09dbe00640 1 -- 172.21.15.118:0/1085924255 <== mon.1 v2:172.21.15.193:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f09d00bd5e0 con 0x7f09e41028b0 2024-03-24T00:18:35.203 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:18:34 smithi189 ceph-mon[36187]: pgmap v484: 97 pgs: 97 active+clean; 45 MiB data, 483 MiB used, 1.0 TiB / 1.0 TiB avail; 524 KiB/s rd, 1.8 MiB/s wr, 358 op/s 2024-03-24T00:18:35.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:18:34 smithi193 ceph-mon[36878]: pgmap v484: 97 pgs: 97 active+clean; 45 MiB data, 483 MiB used, 1.0 TiB / 1.0 TiB avail; 524 KiB/s rd, 1.8 MiB/s wr, 358 op/s 2024-03-24T00:18:35.246 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:35.247+0000 7f09ea400640 1 -- 172.21.15.118:0/1085924255 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f09e40622a0 con 0x7f09e41028b0 2024-03-24T00:18:35.247 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:35.247+0000 7f09dbe00640 1 -- 172.21.15.118:0/1085924255 <== mon.1 v2:172.21.15.193:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+11478 (secure 0 0 0) 0x7f09d00bcee0 con 0x7f09e41028b0 2024-03-24T00:18:35.247 INFO:teuthology.orchestra.run.smithi118.stdout: 2024-03-24T00:18:35.247 INFO:teuthology.orchestra.run.smithi118.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14385,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.118:6835/3194741373","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6834","nonce":3194741373},{"type":"v1","addr":"172.21.15.118:6835","nonce":3194741373}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":4},{"gid":14397,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.118:6837/1374462956","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6836","nonce":1374462956},{"type":"v1","addr":"172.21.15.118:6837","nonce":1374462956}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":28,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-03-24T00:13:01.782474+0000","modified":"2024-03-24T00:15:38.235450+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24569,"mds_1":24550,"mds_2":24511,"mds_3":24499,"mds_4":24488},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14469":{"gid":14469,"name":"a","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.118:6841/3040946537","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6840","nonce":3040946537},{"type":"v1","addr":"172.21.15.118:6841","nonce":3040946537}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24439":{"gid":24439,"name":"l","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.193:6833/1023428412","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6832","nonce":1023428412},{"type":"v1","addr":"172.21.15.193:6833","nonce":1023428412}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24440":{"gid":24440,"name":"k","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.189:6833/2740024078","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6832","nonce":2740024078},{"type":"v1","addr":"172.21.15.189:6833","nonce":2740024078}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24488":{"gid":24488,"name":"e","rank":4,"incarnation":25,"state":"up:active","state_seq":10,"addr":"172.21.15.189:6835/599473205","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6834","nonce":599473205},{"type":"v1","addr":"172.21.15.189:6835","nonce":599473205}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24499":{"gid":24499,"name":"f","rank":3,"incarnation":23,"state":"up:active","state_seq":9,"addr":"172.21.15.193:6835/2055310541","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6834","nonce":2055310541},{"type":"v1","addr":"172.21.15.193:6835","nonce":2055310541}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24511":{"gid":24511,"name":"c","rank":2,"incarnation":21,"state":"up:active","state_seq":8,"addr":"172.21.15.193:6837/2921520053","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6836","nonce":2921520053},{"type":"v1","addr":"172.21.15.193:6837","nonce":2921520053}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24515":{"gid":24515,"name":"h","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.189:6837/3450187137","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6836","nonce":3450187137},{"type":"v1","addr":"172.21.15.189:6837","nonce":3450187137}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24532":{"gid":24532,"name":"i","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.193:6839/2126842603","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6838","nonce":2126842603},{"type":"v1","addr":"172.21.15.193:6839","nonce":2126842603}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24550":{"gid":24550,"name":"g","rank":1,"incarnation":19,"state":"up:active","state_seq":6,"addr":"172.21.15.118:6839/3285788503","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6838","nonce":3285788503},{"type":"v1","addr":"172.21.15.118:6839","nonce":3285788503}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24569":{"gid":24569,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.189:6839/4074484168","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6838","nonce":4074484168},{"type":"v1","addr":"172.21.15.189:6839","nonce":4074484168}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24569,"qdb_cluster":[24569,24499,24488,24511,24550]},"id":1}]} 2024-03-24T00:18:35.248 INFO:teuthology.orchestra.run.smithi118.stderr:dumped fsmap epoch 28 2024-03-24T00:18:35.250 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:35.250+0000 7f09ea400640 1 -- 172.21.15.118:0/1085924255 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7f09c80a2800 msgr2=0x7f09c80a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:18:35.250 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:35.250+0000 7f09ea400640 1 --2- 172.21.15.118:0/1085924255 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7f09c80a2800 0x7f09c80a4cc0 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 crypto rx=0x7f09d40058c0 tx=0x7f09d4005f90 comp rx=0 tx=0).stop 2024-03-24T00:18:35.250 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:35.250+0000 7f09ea400640 1 -- 172.21.15.118:0/1085924255 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f09e41028b0 msgr2=0x7f09e4073a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:18:35.250 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:35.250+0000 7f09ea400640 1 --2- 172.21.15.118:0/1085924255 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f09e41028b0 0x7f09e4073a90 secure :-1 s=READY pgs=339 cs=0 l=1 rev1=1 crypto rx=0x7f09d0009230 tx=0x7f09d002c650 comp rx=0 tx=0).stop 2024-03-24T00:18:35.250 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:35.250+0000 7f09ea400640 1 -- 172.21.15.118:0/1085924255 shutdown_connections 2024-03-24T00:18:35.250 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:35.250+0000 7f09ea400640 1 --2- 172.21.15.118:0/1085924255 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7f09c80a2800 0x7f09c80a4cc0 unknown :-1 s=CLOSED pgs=320 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:18:35.250 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:35.250+0000 7f09ea400640 1 --2- 172.21.15.118:0/1085924255 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f09e4103270 0x7f09e4073fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:18:35.250 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:35.250+0000 7f09ea400640 1 --2- 172.21.15.118:0/1085924255 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f09e41028b0 0x7f09e4073a90 unknown :-1 s=CLOSED pgs=339 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:18:35.250 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:35.250+0000 7f09ea400640 1 --2- 172.21.15.118:0/1085924255 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f09e41016c0 0x7f09e4075450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:18:35.250 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:35.250+0000 7f09ea400640 1 -- 172.21.15.118:0/1085924255 >> 172.21.15.118:0/1085924255 conn(0x7f09e40fcc50 msgr2=0x7f09e4105bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:18:35.251 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:35.250+0000 7f09ea400640 1 -- 172.21.15.118:0/1085924255 shutdown_connections 2024-03-24T00:18:35.251 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:35.250+0000 7f09ea400640 1 -- 172.21.15.118:0/1085924255 wait complete. 2024-03-24T00:18:35.273 INFO:teuthology.orchestra.run:Running command with timeout 120 2024-03-24T00:18:35.273 DEBUG:teuthology.orchestra.run.smithi118:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:4 damage ls 2024-03-24T00:18:35.499 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:35.499+0000 7f6556a00640 1 -- 172.21.15.118:0/395190245 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6550105490 msgr2=0x7f655010bd20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:18:35.500 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:35.499+0000 7f6556a00640 1 --2- 172.21.15.118:0/395190245 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6550105490 0x7f655010bd20 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 crypto rx=0x7f6548009ca0 tx=0x7f654802c100 comp rx=0 tx=0).stop 2024-03-24T00:18:35.500 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:35.499+0000 7f6556a00640 1 -- 172.21.15.118:0/395190245 shutdown_connections 2024-03-24T00:18:35.500 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:35.499+0000 7f6556a00640 1 --2- 172.21.15.118:0/395190245 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6550105490 0x7f655010bd20 unknown :-1 s=CLOSED pgs=334 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:18:35.500 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:35.500+0000 7f6556a00640 1 --2- 172.21.15.118:0/395190245 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f6550104ad0 0x7f6550104f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:18:35.500 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:35.500+0000 7f6556a00640 1 --2- 172.21.15.118:0/395190245 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f65501038e0 0x7f6550103d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:18:35.500 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:35.500+0000 7f6556a00640 1 -- 172.21.15.118:0/395190245 >> 172.21.15.118:0/395190245 conn(0x7f65500fee50 msgr2=0x7f6550101290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:18:35.500 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:35.500+0000 7f6556a00640 1 -- 172.21.15.118:0/395190245 shutdown_connections 2024-03-24T00:18:35.500 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:35.500+0000 7f6556a00640 1 -- 172.21.15.118:0/395190245 wait complete. 2024-03-24T00:18:35.502 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:35.501+0000 7f6556a00640 1 Processor -- start 2024-03-24T00:18:35.502 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:35.501+0000 7f6556a00640 1 -- start start 2024-03-24T00:18:35.502 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:35.501+0000 7f6556a00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f65501038e0 0x7f6550199b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:18:35.502 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:35.501+0000 7f6556a00640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f6550104ad0 0x7f655019a0b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:18:35.503 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:35.501+0000 7f6556a00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6550105490 0x7f65501a1120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:18:35.503 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:35.501+0000 7f6556a00640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f655010f560 con 0x7f6550104ad0 2024-03-24T00:18:35.503 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:35.501+0000 7f6556a00640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f655010f3e0 con 0x7f6550105490 2024-03-24T00:18:35.503 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:35.501+0000 7f6556a00640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f655010f6e0 con 0x7f65501038e0 2024-03-24T00:18:35.503 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:35.501+0000 7f654fe00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f65501038e0 0x7f6550199b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:18:35.503 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:35.501+0000 7f654f400640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f6550104ad0 0x7f655019a0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:18:35.503 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:35.501+0000 7f654fe00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f65501038e0 0x7f6550199b70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.118:52902/0 (socket says 172.21.15.118:52902) 2024-03-24T00:18:35.503 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:35.501+0000 7f654fe00640 1 -- 172.21.15.118:0/4011343092 learned_addr learned my addr 172.21.15.118:0/4011343092 (peer_addr_for_me v2:172.21.15.118:0/0) 2024-03-24T00:18:35.503 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:35.501+0000 7f6554a00640 1 --2- 172.21.15.118:0/4011343092 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6550105490 0x7f65501a1120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:18:35.503 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:35.502+0000 7f654fe00640 1 -- 172.21.15.118:0/4011343092 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6550105490 msgr2=0x7f65501a1120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:18:35.503 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:35.502+0000 7f654fe00640 1 --2- 172.21.15.118:0/4011343092 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6550105490 0x7f65501a1120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:18:35.503 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:35.502+0000 7f654fe00640 1 -- 172.21.15.118:0/4011343092 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f6550104ad0 msgr2=0x7f655019a0b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:18:35.504 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:35.502+0000 7f654fe00640 1 --2- 172.21.15.118:0/4011343092 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f6550104ad0 0x7f655019a0b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:18:35.504 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:35.502+0000 7f654fe00640 1 -- 172.21.15.118:0/4011343092 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f65501a1790 con 0x7f65501038e0 2024-03-24T00:18:35.504 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:35.502+0000 7f6554a00640 1 --2- 172.21.15.118:0/4011343092 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6550105490 0x7f65501a1120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T00:18:35.504 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:35.502+0000 7f654fe00640 1 --2- 172.21.15.118:0/4011343092 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f65501038e0 0x7f6550199b70 secure :-1 s=READY pgs=340 cs=0 l=1 rev1=1 crypto rx=0x7f654000e9c0 tx=0x7f654000ee80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:18:35.504 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:35.502+0000 7f654cc00640 1 -- 172.21.15.118:0/4011343092 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f654000cd80 con 0x7f65501038e0 2024-03-24T00:18:35.504 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:35.502+0000 7f654cc00640 1 -- 172.21.15.118:0/4011343092 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f65400035f0 con 0x7f65501038e0 2024-03-24T00:18:35.504 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:35.502+0000 7f654cc00640 1 -- 172.21.15.118:0/4011343092 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6540010690 con 0x7f65501038e0 2024-03-24T00:18:35.504 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:35.502+0000 7f6556a00640 1 -- 172.21.15.118:0/4011343092 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f65501a1a80 con 0x7f65501038e0 2024-03-24T00:18:35.504 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:35.503+0000 7f6556a00640 1 -- 172.21.15.118:0/4011343092 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f65501a1f80 con 0x7f65501038e0 2024-03-24T00:18:35.505 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:35.506+0000 7f654cc00640 1 -- 172.21.15.118:0/4011343092 <== mon.1 v2:172.21.15.193:3300/0 4 ==== mgrmap(e 20) v1 ==== 137291+0+0 (secure 0 0 0) 0x7f65400037a0 con 0x7f65501038e0 2024-03-24T00:18:35.514 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:35.507+0000 7f65597ac740 1 Processor -- start 2024-03-24T00:18:35.514 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:35.507+0000 7f65597ac740 1 -- start start 2024-03-24T00:18:35.514 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:35.507+0000 7f654cc00640 1 --2- 172.21.15.118:0/4011343092 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7f65340a2880 0x7f65340a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:18:35.514 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:35.507+0000 7f65597ac740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x560c6d4ff810 0x560c6d4ffc30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:18:35.514 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:35.507+0000 7f65597ac740 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x560c6d5009e0 0x560c6d500e60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:18:35.514 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:35.507+0000 7f65597ac740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x560c6d5013a0 0x560c6d507c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:18:35.514 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:35.507+0000 7f65597ac740 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x560c6d4642c0 con 0x560c6d5009e0 2024-03-24T00:18:35.514 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:35.507+0000 7f65597ac740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x560c6d40cc90 con 0x560c6d4ff810 2024-03-24T00:18:35.515 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:35.507+0000 7f65597ac740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x560c6d460360 con 0x560c6d5013a0 2024-03-24T00:18:35.515 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:35.508+0000 7f654cc00640 1 -- 172.21.15.118:0/4011343092 <== mon.1 v2:172.21.15.193:3300/0 5 ==== osd_map(90..90 src has 1..90) v4 ==== 8277+0+0 (secure 0 0 0) 0x7f65400cedc0 con 0x7f65501038e0 2024-03-24T00:18:35.515 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:35.508+0000 7f654fe00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x560c6d4ff810 0x560c6d4ffc30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:18:35.515 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:35.508+0000 7f654fe00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x560c6d4ff810 0x560c6d4ffc30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.118:60976/0 (socket says 172.21.15.118:60976) 2024-03-24T00:18:35.515 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:35.508+0000 7f654fe00640 1 -- 172.21.15.118:0/170780325 learned_addr learned my addr 172.21.15.118:0/170780325 (peer_addr_for_me v2:172.21.15.118:0/0) 2024-03-24T00:18:35.515 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:35.508+0000 7f6554a00640 1 --2- 172.21.15.118:0/4011343092 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7f65340a2880 0x7f65340a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:18:35.515 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:35.508+0000 7f654f400640 1 --2- 172.21.15.118:0/170780325 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x560c6d5009e0 0x560c6d500e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:18:35.515 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:35.508+0000 7f6554a00640 1 --2- 172.21.15.118:0/170780325 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x560c6d5013a0 0x560c6d507c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:18:35.515 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:35.508+0000 7f654fe00640 1 -- 172.21.15.118:0/170780325 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x560c6d5013a0 msgr2=0x560c6d507c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:18:35.515 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:35.508+0000 7f654fe00640 1 --2- 172.21.15.118:0/170780325 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x560c6d5013a0 0x560c6d507c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:18:35.515 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:35.508+0000 7f654fe00640 1 -- 172.21.15.118:0/170780325 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x560c6d5009e0 msgr2=0x560c6d500e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:18:35.515 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:35.508+0000 7f654fe00640 1 --2- 172.21.15.118:0/170780325 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x560c6d5009e0 0x560c6d500e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:18:35.515 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:35.508+0000 7f654fe00640 1 -- 172.21.15.118:0/170780325 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x560c6d431080 con 0x560c6d4ff810 2024-03-24T00:18:35.516 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:35.508+0000 7f654fe00640 1 --2- 172.21.15.118:0/170780325 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x560c6d4ff810 0x560c6d4ffc30 secure :-1 s=READY pgs=335 cs=0 l=1 rev1=1 crypto rx=0x7f6540097b30 tx=0x7f6540097aa0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=bd7dc1f5a09dc2ac server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:18:35.516 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:35.509+0000 7f6545200640 1 -- 172.21.15.118:0/170780325 <== mon.2 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f654009f820 con 0x560c6d4ff810 2024-03-24T00:18:35.516 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:35.509+0000 7f6545200640 1 -- 172.21.15.118:0/170780325 <== mon.2 v2:172.21.15.189:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f65400a3e70 con 0x560c6d4ff810 2024-03-24T00:18:35.516 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:35.509+0000 7f6545200640 1 -- 172.21.15.118:0/170780325 <== mon.2 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f65400875a0 con 0x560c6d4ff810 2024-03-24T00:18:35.516 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:35.509+0000 7f65597ac740 1 -- 172.21.15.118:0/170780325 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x560c6d4ff810 msgr2=0x560c6d4ffc30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:18:35.516 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:35.509+0000 7f65597ac740 1 --2- 172.21.15.118:0/170780325 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x560c6d4ff810 0x560c6d4ffc30 secure :-1 s=READY pgs=335 cs=0 l=1 rev1=1 crypto rx=0x7f6540097b30 tx=0x7f6540097aa0 comp rx=0 tx=0).stop 2024-03-24T00:18:35.516 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:35.509+0000 7f6554a00640 1 --2- 172.21.15.118:0/4011343092 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7f65340a2880 0x7f65340a4d40 secure :-1 s=READY pgs=321 cs=0 l=1 rev1=1 crypto rx=0x7f654802c640 tx=0x7f654802e040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:18:35.516 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:35.509+0000 7f65597ac740 1 -- 172.21.15.118:0/170780325 shutdown_connections 2024-03-24T00:18:35.516 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:35.509+0000 7f65597ac740 1 --2- 172.21.15.118:0/170780325 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x560c6d5013a0 0x560c6d507c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:18:35.516 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:35.509+0000 7f65597ac740 1 --2- 172.21.15.118:0/170780325 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x560c6d5009e0 0x560c6d500e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:18:35.516 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:35.509+0000 7f65597ac740 1 --2- 172.21.15.118:0/170780325 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x560c6d4ff810 0x560c6d4ffc30 unknown :-1 s=CLOSED pgs=335 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:18:35.516 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:35.509+0000 7f65597ac740 1 -- 172.21.15.118:0/170780325 >> 172.21.15.118:0/170780325 conn(0x560c6d4fac00 msgr2=0x560c6d4fb000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:18:35.516 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:35.510+0000 7f65597ac740 1 -- 172.21.15.118:0/170780325 shutdown_connections 2024-03-24T00:18:35.516 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:35.510+0000 7f65597ac740 1 -- 172.21.15.118:0/170780325 wait complete. 2024-03-24T00:18:35.517 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:35.510+0000 7f65597ac740 1 -- start start 2024-03-24T00:18:35.517 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:35.510+0000 7f65597ac740 1 Processor -- start 2024-03-24T00:18:35.517 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:35.511+0000 7f65597ac740 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x560c6d4ff810 0x560c6d509a50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:18:35.517 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:35.511+0000 7f65597ac740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x560c6d5009e0 0x560c6d509f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:18:35.517 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:35.511+0000 7f65597ac740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x560c6d5013a0 0x560c6d510f60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:18:35.517 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:35.511+0000 7f65597ac740 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x560c6d5082f0 con 0x560c6d4ff810 2024-03-24T00:18:35.517 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:35.511+0000 7f65597ac740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x560c6d4200d0 con 0x560c6d5009e0 2024-03-24T00:18:35.517 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:35.511+0000 7f65597ac740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x560c6d508470 con 0x560c6d5013a0 2024-03-24T00:18:35.517 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:35.511+0000 7f65597ac740 10 client.0 _ll_register_callbacks cb 0x560c6d424e80 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T00:18:35.517 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:35.511+0000 7f654f400640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x560c6d5009e0 0x560c6d509f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:18:35.517 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:35.511+0000 7f654f400640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x560c6d5009e0 0x560c6d509f90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.118:60988/0 (socket says 172.21.15.118:60988) 2024-03-24T00:18:35.517 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:35.511+0000 7f654f400640 1 -- 172.21.15.118:0/4214577446 learned_addr learned my addr 172.21.15.118:0/4214577446 (peer_addr_for_me v2:172.21.15.118:0/0) 2024-03-24T00:18:35.517 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:35.511+0000 7f6554a00640 1 --2- 172.21.15.118:0/4214577446 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x560c6d5013a0 0x560c6d510f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:18:35.517 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:35.511+0000 7f654fe00640 1 --2- 172.21.15.118:0/4214577446 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x560c6d4ff810 0x560c6d509a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:18:35.518 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:35.511+0000 7f654f400640 1 -- 172.21.15.118:0/4214577446 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x560c6d5013a0 msgr2=0x560c6d510f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:18:35.518 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:35.511+0000 7f654f400640 1 --2- 172.21.15.118:0/4214577446 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x560c6d5013a0 0x560c6d510f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:18:35.518 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:35.511+0000 7f654f400640 1 -- 172.21.15.118:0/4214577446 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x560c6d4ff810 msgr2=0x560c6d509a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:18:35.518 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:35.511+0000 7f654f400640 1 --2- 172.21.15.118:0/4214577446 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x560c6d4ff810 0x560c6d509a50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:18:35.518 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:35.511+0000 7f654f400640 1 -- 172.21.15.118:0/4214577446 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x560c6d1783f0 con 0x560c6d5009e0 2024-03-24T00:18:35.518 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:35.512+0000 7f654f400640 1 --2- 172.21.15.118:0/4214577446 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x560c6d5009e0 0x560c6d509f90 secure :-1 s=READY pgs=336 cs=0 l=1 rev1=1 crypto rx=0x7f653c00bdf0 tx=0x7f653c012820 comp rx=0 tx=0).ready entity=mon.2 client_cookie=2acb4374e2470e04 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:18:35.518 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:35.512+0000 7f6539600640 10 client.0 ms_handle_connect on v2:172.21.15.189:3300/0 2024-03-24T00:18:35.518 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:35.512+0000 7f6539600640 1 -- 172.21.15.118:0/4214577446 <== mon.2 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f653c005220 con 0x560c6d5009e0 2024-03-24T00:18:35.518 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:35.512+0000 7f65597ac740 1 -- 172.21.15.118:0/4214577446 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f65400a3e70 con 0x560c6d5009e0 2024-03-24T00:18:35.518 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:35.512+0000 7f6539600640 1 -- 172.21.15.118:0/4214577446 <== mon.2 v2:172.21.15.189:3300/0 2 ==== osd_map(90..90 src has 1..90) v4 ==== 8277+0+0 (secure 0 0 0) 0x7f653c0053c0 con 0x560c6d5009e0 2024-03-24T00:18:35.518 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:35.512+0000 7f6539600640 1 -- 172.21.15.118:0/4214577446 <== mon.2 v2:172.21.15.189:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) v2 ==== 24+0+0 (secure 0 0 0) 0x7f653c0055c0 con 0x560c6d5009e0 2024-03-24T00:18:35.518 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:35.512+0000 7f65597ac740 10 client.26210 fetch_fsmap learned FSMap version 28 2024-03-24T00:18:35.519 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:35.512+0000 7f65597ac740 1 -- 172.21.15.118:0/4214577446 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) v3 -- 0x560c6d2427d0 con 0x560c6d5009e0 2024-03-24T00:18:35.519 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:35.512+0000 7f6539600640 1 -- 172.21.15.118:0/4214577446 <== mon.2 v2:172.21.15.189:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f65400a3e70 con 0x560c6d5009e0 2024-03-24T00:18:35.519 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:35.513+0000 7f6539600640 1 -- 172.21.15.118:0/4214577446 <== mon.2 v2:172.21.15.189:3300/0 5 ==== fsmap(e 28) v1 ==== 7934+0+0 (secure 0 0 0) 0x7f653c025070 con 0x560c6d5009e0 2024-03-24T00:18:35.519 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:35.513+0000 7f65597ac740 10 client.26210 fetch_fsmap finished waiting for FSMap version 28 2024-03-24T00:18:35.519 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:35.513+0000 7f65597ac740 10 client.26210 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.e 2024-03-24T00:18:35.519 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:35.513+0000 7f65597ac740 20 client.26210 populate_metadata read hostname 'smithi118' 2024-03-24T00:18:35.519 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:35.513+0000 7f65597ac740 1 --2- 172.21.15.118:0/4214577446 >> [v2:172.21.15.189:6834/599473205,v1:172.21.15.189:6835/599473205] conn(0x560c6d515400 0x560c6d517880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:18:35.519 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:35.513+0000 7f65597ac740 4 client.26210 mds_command: new command op to 24488 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-24T00:18:35.519 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:35.513+0000 7f65597ac740 1 -- 172.21.15.118:0/4214577446 --> [v2:172.21.15.189:6834/599473205,v1:172.21.15.189:6835/599473205] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x560c6d471800 con 0x560c6d515400 2024-03-24T00:18:35.519 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:35.513+0000 7f654fe00640 1 --2- 172.21.15.118:0/4214577446 >> [v2:172.21.15.189:6834/599473205,v1:172.21.15.189:6835/599473205] conn(0x560c6d515400 0x560c6d517880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:18:35.519 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:35.514+0000 7f654fe00640 1 --2- 172.21.15.118:0/4214577446 >> [v2:172.21.15.189:6834/599473205,v1:172.21.15.189:6835/599473205] conn(0x560c6d515400 0x560c6d517880 crc :-1 s=READY pgs=30 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=6ec81782e9e08ebf server_cookie=864984baf02e6458 in_seq=0 out_seq=0 2024-03-24T00:18:35.519 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:35.514+0000 7f6539600640 10 client.26210 ms_handle_connect on v2:172.21.15.189:6834/599473205 2024-03-24T00:18:35.520 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:35.515+0000 7f6539600640 1 -- 172.21.15.118:0/4214577446 <== mds.4 v2:172.21.15.189:6834/599473205 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x560c6d471800 con 0x560c6d515400 2024-03-24T00:18:35.521 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:35.515+0000 7f6539600640 10 client.26210 handle_command_reply: tid=0 2024-03-24T00:18:35.521 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:35.515+0000 7f65597ac740 1 client.26210 shutdown 2024-03-24T00:18:35.521 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:35.515+0000 7f65597ac740 1 -- 172.21.15.118:0/4214577446 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x560c6d5009e0 msgr2=0x560c6d509f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:18:35.521 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:35.515+0000 7f65597ac740 1 --2- 172.21.15.118:0/4214577446 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x560c6d5009e0 0x560c6d509f90 secure :-1 s=READY pgs=336 cs=0 l=1 rev1=1 crypto rx=0x7f653c00bdf0 tx=0x7f653c012820 comp rx=0 tx=0).stop 2024-03-24T00:18:35.521 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:35.515+0000 7f65597ac740 1 -- 172.21.15.118:0/4214577446 shutdown_connections 2024-03-24T00:18:35.521 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:35.515+0000 7f65597ac740 1 --2- 172.21.15.118:0/4214577446 >> [v2:172.21.15.189:6834/599473205,v1:172.21.15.189:6835/599473205] conn(0x560c6d515400 0x560c6d517880 crc :-1 s=READY pgs=30 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:18:35.521 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:35.515+0000 7f65597ac740 1 --2- 172.21.15.118:0/4214577446 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x560c6d5013a0 0x560c6d510f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:18:35.521 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:35.515+0000 7f65597ac740 1 --2- 172.21.15.118:0/4214577446 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x560c6d5009e0 0x560c6d509f90 unknown :-1 s=CLOSED pgs=336 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:18:35.521 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:35.515+0000 7f65597ac740 1 --2- 172.21.15.118:0/4214577446 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x560c6d4ff810 0x560c6d509a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:18:35.521 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:35.515+0000 7f6539600640 0 client.26210 ms_handle_reset on v2:172.21.15.189:6834/599473205 2024-03-24T00:18:35.521 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:35.515+0000 7f65597ac740 1 -- 172.21.15.118:0/4214577446 >> 172.21.15.118:0/4214577446 conn(0x560c6d4fac00 msgr2=0x560c6d4f9eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:18:35.521 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:35.515+0000 7f65597ac740 1 -- 172.21.15.118:0/4214577446 shutdown_connections 2024-03-24T00:18:35.521 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:35.515+0000 7f65597ac740 1 -- 172.21.15.118:0/4214577446 wait complete. 2024-03-24T00:18:35.521 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:35.515+0000 7f65597ac740 20 client.26210 trim_cache size 0 max 16384 2024-03-24T00:18:35.530 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:35.530+0000 7f65597ac740 1 Processor -- start 2024-03-24T00:18:35.530 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:35.530+0000 7f65597ac740 1 -- start start 2024-03-24T00:18:35.530 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:35.530+0000 7f65597ac740 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x560c6d4ff810 0x560c6d4fe570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:18:35.530 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:35.530+0000 7f65597ac740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x560c6d5009e0 0x560c6d510f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:18:35.531 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:35.530+0000 7f65597ac740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x560c6d5013a0 0x560c6d503b20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:18:35.531 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:35.530+0000 7f65597ac740 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f65501aac40 con 0x560c6d4ff810 2024-03-24T00:18:35.531 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:35.530+0000 7f65597ac740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f65501ac5c0 con 0x560c6d5009e0 2024-03-24T00:18:35.531 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:35.530+0000 7f65597ac740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f65501ab2a0 con 0x560c6d5013a0 2024-03-24T00:18:35.531 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:35.530+0000 7f654f400640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x560c6d5009e0 0x560c6d510f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:18:35.531 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:35.530+0000 7f654f400640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x560c6d5009e0 0x560c6d510f20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.118:60992/0 (socket says 172.21.15.118:60992) 2024-03-24T00:18:35.531 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:35.530+0000 7f654f400640 1 -- 172.21.15.118:0/3627296836 learned_addr learned my addr 172.21.15.118:0/3627296836 (peer_addr_for_me v2:172.21.15.118:0/0) 2024-03-24T00:18:35.531 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:35.530+0000 7f6554a00640 1 --2- 172.21.15.118:0/3627296836 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x560c6d5013a0 0x560c6d503b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:18:35.531 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:35.530+0000 7f654fe00640 1 --2- 172.21.15.118:0/3627296836 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x560c6d4ff810 0x560c6d4fe570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:18:35.531 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:35.531+0000 7f654f400640 1 -- 172.21.15.118:0/3627296836 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x560c6d5013a0 msgr2=0x560c6d503b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:18:35.531 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:35.531+0000 7f654f400640 1 --2- 172.21.15.118:0/3627296836 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x560c6d5013a0 0x560c6d503b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:18:35.531 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:35.531+0000 7f654f400640 1 -- 172.21.15.118:0/3627296836 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x560c6d4ff810 msgr2=0x560c6d4fe570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:18:35.532 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:35.531+0000 7f654f400640 1 --2- 172.21.15.118:0/3627296836 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x560c6d4ff810 0x560c6d4fe570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:18:35.532 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:35.531+0000 7f654f400640 1 -- 172.21.15.118:0/3627296836 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x560c6d4fb650 con 0x560c6d5009e0 2024-03-24T00:18:35.532 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:35.531+0000 7f654fe00640 1 --2- 172.21.15.118:0/3627296836 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x560c6d4ff810 0x560c6d4fe570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T00:18:35.532 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:35.531+0000 7f654f400640 1 --2- 172.21.15.118:0/3627296836 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x560c6d5009e0 0x560c6d510f20 secure :-1 s=READY pgs=337 cs=0 l=1 rev1=1 crypto rx=0x7f653c0057c0 tx=0x7f653c00b8c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=8c61f199b2f3be69 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:18:35.536 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:35.531+0000 7f653be00640 1 -- 172.21.15.118:0/3627296836 <== mon.2 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f653c010910 con 0x560c6d5009e0 2024-03-24T00:18:35.536 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:35.531+0000 7f653be00640 1 -- 172.21.15.118:0/3627296836 <== mon.2 v2:172.21.15.189:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f653c02b070 con 0x560c6d5009e0 2024-03-24T00:18:35.536 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:35.531+0000 7f653be00640 1 -- 172.21.15.118:0/3627296836 <== mon.2 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f653c01c660 con 0x560c6d5009e0 2024-03-24T00:18:35.536 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:35.532+0000 7f65597ac740 1 -- 172.21.15.118:0/3627296836 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x560c6d5009e0 msgr2=0x560c6d510f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:18:35.536 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:35.532+0000 7f65597ac740 1 --2- 172.21.15.118:0/3627296836 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x560c6d5009e0 0x560c6d510f20 secure :-1 s=READY pgs=337 cs=0 l=1 rev1=1 crypto rx=0x7f653c0057c0 tx=0x7f653c00b8c0 comp rx=0 tx=0).stop 2024-03-24T00:18:35.537 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:35.532+0000 7f65597ac740 1 -- 172.21.15.118:0/3627296836 shutdown_connections 2024-03-24T00:18:35.537 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:35.532+0000 7f65597ac740 1 --2- 172.21.15.118:0/3627296836 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x560c6d5013a0 0x560c6d503b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:18:35.537 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:35.532+0000 7f65597ac740 1 --2- 172.21.15.118:0/3627296836 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x560c6d5009e0 0x560c6d510f20 unknown :-1 s=CLOSED pgs=337 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:18:35.537 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:35.532+0000 7f65597ac740 1 --2- 172.21.15.118:0/3627296836 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x560c6d4ff810 0x560c6d4fe570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:18:35.537 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:35.532+0000 7f65597ac740 1 -- 172.21.15.118:0/3627296836 >> 172.21.15.118:0/3627296836 conn(0x560c6d4fac00 msgr2=0x560c6d4fde60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:18:35.537 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:35.532+0000 7f65597ac740 1 -- 172.21.15.118:0/3627296836 shutdown_connections 2024-03-24T00:18:35.537 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:35.532+0000 7f65597ac740 1 -- 172.21.15.118:0/3627296836 wait complete. 2024-03-24T00:18:35.537 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:35.532+0000 7f65597ac740 1 -- start start 2024-03-24T00:18:35.537 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:35.533+0000 7f65597ac740 1 Processor -- start 2024-03-24T00:18:35.537 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:35.533+0000 7f65597ac740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x560c6d4ff810 0x560c6d50c790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:18:35.537 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:35.533+0000 7f65597ac740 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x560c6d5009e0 0x560c6d50ccd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:18:35.537 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:35.533+0000 7f65597ac740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x560c6d5013a0 0x560c6d50fca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:18:35.537 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:35.533+0000 7f65597ac740 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f65501abf60 con 0x560c6d5009e0 2024-03-24T00:18:35.538 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:35.533+0000 7f65597ac740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f65501ab900 con 0x560c6d4ff810 2024-03-24T00:18:35.538 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:35.533+0000 7f65597ac740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f65501a8e00 con 0x560c6d5013a0 2024-03-24T00:18:35.538 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:35.533+0000 7f65597ac740 10 client.0 _ll_register_callbacks cb 0x7f653c00ba00 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T00:18:35.538 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:35.533+0000 7f654fe00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x560c6d4ff810 0x560c6d50c790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:18:35.538 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:35.533+0000 7f654fe00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x560c6d4ff810 0x560c6d50c790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.118:32776/0 (socket says 172.21.15.118:32776) 2024-03-24T00:18:35.538 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:35.533+0000 7f654fe00640 1 -- 172.21.15.118:0/1899084416 learned_addr learned my addr 172.21.15.118:0/1899084416 (peer_addr_for_me v2:172.21.15.118:0/0) 2024-03-24T00:18:35.538 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:35.533+0000 7f6554a00640 1 --2- 172.21.15.118:0/1899084416 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x560c6d5013a0 0x560c6d50fca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:18:35.538 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:35.533+0000 7f654f400640 1 --2- 172.21.15.118:0/1899084416 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x560c6d5009e0 0x560c6d50ccd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:18:35.538 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:35.534+0000 7f654fe00640 1 -- 172.21.15.118:0/1899084416 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x560c6d5013a0 msgr2=0x560c6d50fca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:18:35.538 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:35.534+0000 7f654fe00640 1 --2- 172.21.15.118:0/1899084416 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x560c6d5013a0 0x560c6d50fca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:18:35.538 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:35.534+0000 7f654fe00640 1 -- 172.21.15.118:0/1899084416 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x560c6d5009e0 msgr2=0x560c6d50ccd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:18:35.538 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:35.534+0000 7f654fe00640 1 --2- 172.21.15.118:0/1899084416 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x560c6d5009e0 0x560c6d50ccd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:18:35.538 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:35.534+0000 7f654fe00640 1 -- 172.21.15.118:0/1899084416 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x560c6d510290 con 0x560c6d4ff810 2024-03-24T00:18:35.538 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:35.534+0000 7f654fe00640 1 --2- 172.21.15.118:0/1899084416 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x560c6d4ff810 0x560c6d50c790 secure :-1 s=READY pgs=338 cs=0 l=1 rev1=1 crypto rx=0x7f654009b3d0 tx=0x7f6540097340 comp rx=0 tx=0).ready entity=mon.2 client_cookie=61fb32605d655c7f server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:18:35.539 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:35.534+0000 7f6539600640 10 client.0 ms_handle_connect on v2:172.21.15.189:3300/0 2024-03-24T00:18:35.539 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:35.534+0000 7f6539600640 1 -- 172.21.15.118:0/1899084416 <== mon.2 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f65400a3e70 con 0x560c6d4ff810 2024-03-24T00:18:35.539 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:35.534+0000 7f6539600640 1 -- 172.21.15.118:0/1899084416 <== mon.2 v2:172.21.15.189:3300/0 2 ==== osd_map(90..90 src has 1..90) v4 ==== 8277+0+0 (secure 0 0 0) 0x7f6540003b10 con 0x560c6d4ff810 2024-03-24T00:18:35.539 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:35.534+0000 7f65597ac740 1 -- 172.21.15.118:0/1899084416 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f653c02b070 con 0x560c6d4ff810 2024-03-24T00:18:35.539 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:35.534+0000 7f6539600640 1 -- 172.21.15.118:0/1899084416 <== mon.2 v2:172.21.15.189:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) v2 ==== 24+0+0 (secure 0 0 0) 0x7f654009f820 con 0x560c6d4ff810 2024-03-24T00:18:35.539 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:35.534+0000 7f65597ac740 10 client.26216 fetch_fsmap learned FSMap version 28 2024-03-24T00:18:35.539 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:35.534+0000 7f65597ac740 1 -- 172.21.15.118:0/1899084416 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) v3 -- 0x560c6d570e00 con 0x560c6d4ff810 2024-03-24T00:18:35.539 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:35.535+0000 7f6539600640 1 -- 172.21.15.118:0/1899084416 <== mon.2 v2:172.21.15.189:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f653c02b070 con 0x560c6d4ff810 2024-03-24T00:18:35.539 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:35.535+0000 7f6539600640 1 -- 172.21.15.118:0/1899084416 <== mon.2 v2:172.21.15.189:3300/0 5 ==== fsmap(e 28) v1 ==== 7934+0+0 (secure 0 0 0) 0x7f654008ad00 con 0x560c6d4ff810 2024-03-24T00:18:35.539 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:35.535+0000 7f65597ac740 10 client.26216 fetch_fsmap finished waiting for FSMap version 28 2024-03-24T00:18:35.539 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:35.535+0000 7f65597ac740 10 client.26216 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.e 2024-03-24T00:18:35.539 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:35.535+0000 7f65597ac740 20 client.26216 populate_metadata read hostname 'smithi118' 2024-03-24T00:18:35.539 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:35.535+0000 7f65597ac740 1 --2- 172.21.15.118:0/1899084416 >> [v2:172.21.15.189:6834/599473205,v1:172.21.15.189:6835/599473205] conn(0x560c6d5712d0 0x560c6d5736c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:18:35.539 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:35.535+0000 7f65597ac740 4 client.26216 mds_command: new command op to 24488 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-24T00:18:35.540 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:35.535+0000 7f65597ac740 1 -- 172.21.15.118:0/1899084416 --> [v2:172.21.15.189:6834/599473205,v1:172.21.15.189:6835/599473205] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x560c6d46bb50 con 0x560c6d5712d0 2024-03-24T00:18:35.540 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:35.535+0000 7f654f400640 1 --2- 172.21.15.118:0/1899084416 >> [v2:172.21.15.189:6834/599473205,v1:172.21.15.189:6835/599473205] conn(0x560c6d5712d0 0x560c6d5736c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:18:35.540 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:35.536+0000 7f654f400640 1 --2- 172.21.15.118:0/1899084416 >> [v2:172.21.15.189:6834/599473205,v1:172.21.15.189:6835/599473205] conn(0x560c6d5712d0 0x560c6d5736c0 crc :-1 s=READY pgs=31 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=c80711086fc5a3a0 server_cookie=3326d97b2a858652 in_seq=0 out_seq=0 2024-03-24T00:18:35.540 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:35.536+0000 7f6539600640 10 client.26216 ms_handle_connect on v2:172.21.15.189:6834/599473205 2024-03-24T00:18:35.540 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:35.536+0000 7f6539600640 1 -- 172.21.15.118:0/1899084416 <== mds.4 v2:172.21.15.189:6834/599473205 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x560c6d46bb50 con 0x560c6d5712d0 2024-03-24T00:18:35.540 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:35.536+0000 7f6539600640 10 client.26216 handle_command_reply: tid=0 2024-03-24T00:18:35.540 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:35.536+0000 7f65597ac740 1 client.26216 shutdown 2024-03-24T00:18:35.540 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:35.536+0000 7f65597ac740 1 -- 172.21.15.118:0/1899084416 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x560c6d4ff810 msgr2=0x560c6d50c790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:18:35.540 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:35.536+0000 7f65597ac740 1 --2- 172.21.15.118:0/1899084416 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x560c6d4ff810 0x560c6d50c790 secure :-1 s=READY pgs=338 cs=0 l=1 rev1=1 crypto rx=0x7f654009b3d0 tx=0x7f6540097340 comp rx=0 tx=0).stop 2024-03-24T00:18:35.540 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:35.536+0000 7f65597ac740 1 -- 172.21.15.118:0/1899084416 shutdown_connections 2024-03-24T00:18:35.540 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:35.536+0000 7f65597ac740 1 --2- 172.21.15.118:0/1899084416 >> [v2:172.21.15.189:6834/599473205,v1:172.21.15.189:6835/599473205] conn(0x560c6d5712d0 0x560c6d5736c0 crc :-1 s=READY pgs=31 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:18:35.540 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:35.536+0000 7f65597ac740 1 --2- 172.21.15.118:0/1899084416 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x560c6d5013a0 0x560c6d50fca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:18:35.540 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:35.536+0000 7f65597ac740 1 --2- 172.21.15.118:0/1899084416 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x560c6d5009e0 0x560c6d50ccd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:18:35.541 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:35.536+0000 7f6539600640 0 client.26216 ms_handle_reset on v2:172.21.15.189:6834/599473205 2024-03-24T00:18:35.541 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:35.536+0000 7f65597ac740 1 --2- 172.21.15.118:0/1899084416 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x560c6d4ff810 0x560c6d50c790 unknown :-1 s=CLOSED pgs=338 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:18:35.541 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:35.536+0000 7f65597ac740 1 -- 172.21.15.118:0/1899084416 >> 172.21.15.118:0/1899084416 conn(0x560c6d4fac00 msgr2=0x560c6d4fe6f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:18:35.541 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:35.536+0000 7f65597ac740 1 -- 172.21.15.118:0/1899084416 shutdown_connections 2024-03-24T00:18:35.541 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:35.536+0000 7f65597ac740 1 -- 172.21.15.118:0/1899084416 wait complete. 2024-03-24T00:18:35.541 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:35.536+0000 7f65597ac740 20 client.26216 trim_cache size 0 max 16384 2024-03-24T00:18:35.541 INFO:teuthology.orchestra.run.smithi118.stdout:[] 2024-03-24T00:18:35.542 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:35.537+0000 7f6556a00640 1 -- 172.21.15.118:0/4011343092 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7f65340a2880 msgr2=0x7f65340a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:18:35.542 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:35.537+0000 7f6556a00640 1 --2- 172.21.15.118:0/4011343092 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7f65340a2880 0x7f65340a4d40 secure :-1 s=READY pgs=321 cs=0 l=1 rev1=1 crypto rx=0x7f654802c640 tx=0x7f654802e040 comp rx=0 tx=0).stop 2024-03-24T00:18:35.542 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:35.537+0000 7f6556a00640 1 -- 172.21.15.118:0/4011343092 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f65501038e0 msgr2=0x7f6550199b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:18:35.542 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:35.537+0000 7f6556a00640 1 --2- 172.21.15.118:0/4011343092 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f65501038e0 0x7f6550199b70 secure :-1 s=READY pgs=340 cs=0 l=1 rev1=1 crypto rx=0x7f654000e9c0 tx=0x7f654000ee80 comp rx=0 tx=0).stop 2024-03-24T00:18:35.542 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:35.537+0000 7f6556a00640 1 -- 172.21.15.118:0/4011343092 shutdown_connections 2024-03-24T00:18:35.542 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:35.537+0000 7f6556a00640 1 --2- 172.21.15.118:0/4011343092 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7f65340a2880 0x7f65340a4d40 unknown :-1 s=CLOSED pgs=321 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:18:35.542 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:35.537+0000 7f6556a00640 1 --2- 172.21.15.118:0/4011343092 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6550105490 0x7f65501a1120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:18:35.543 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:35.537+0000 7f6556a00640 1 --2- 172.21.15.118:0/4011343092 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f6550104ad0 0x7f655019a0b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:18:35.543 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:35.537+0000 7f6556a00640 1 --2- 172.21.15.118:0/4011343092 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f65501038e0 0x7f6550199b70 unknown :-1 s=CLOSED pgs=340 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:18:35.543 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:35.537+0000 7f6556a00640 1 -- 172.21.15.118:0/4011343092 >> 172.21.15.118:0/4011343092 conn(0x7f65500fee50 msgr2=0x7f6550107d70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:18:35.543 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:35.537+0000 7f6556a00640 1 -- 172.21.15.118:0/4011343092 shutdown_connections 2024-03-24T00:18:35.543 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:35.538+0000 7f6556a00640 1 -- 172.21.15.118:0/4011343092 wait complete. 2024-03-24T00:18:35.558 INFO:teuthology.orchestra.run:Running command with timeout 120 2024-03-24T00:18:35.559 DEBUG:teuthology.orchestra.run.smithi118:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:3 damage ls 2024-03-24T00:18:35.770 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:35.770+0000 7f6843600640 1 -- 172.21.15.118:0/408179427 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f683c1038e0 msgr2=0x7f683c103d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:18:35.770 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:35.770+0000 7f6843600640 1 --2- 172.21.15.118:0/408179427 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f683c1038e0 0x7f683c103d00 secure :-1 s=READY pgs=339 cs=0 l=1 rev1=1 crypto rx=0x7f6830009900 tx=0x7f683002bc30 comp rx=0 tx=0).stop 2024-03-24T00:18:35.770 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:35.770+0000 7f6843600640 1 -- 172.21.15.118:0/408179427 shutdown_connections 2024-03-24T00:18:35.770 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:35.770+0000 7f6843600640 1 --2- 172.21.15.118:0/408179427 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f683c105490 0x7f683c10bd20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:18:35.770 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:35.770+0000 7f6843600640 1 --2- 172.21.15.118:0/408179427 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f683c104ad0 0x7f683c104f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:18:35.770 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:35.770+0000 7f6843600640 1 --2- 172.21.15.118:0/408179427 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f683c1038e0 0x7f683c103d00 unknown :-1 s=CLOSED pgs=339 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:18:35.770 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:35.770+0000 7f6843600640 1 -- 172.21.15.118:0/408179427 >> 172.21.15.118:0/408179427 conn(0x7f683c0fee50 msgr2=0x7f683c101290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:18:35.770 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:35.770+0000 7f6843600640 1 -- 172.21.15.118:0/408179427 shutdown_connections 2024-03-24T00:18:35.771 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:35.770+0000 7f6843600640 1 -- 172.21.15.118:0/408179427 wait complete. 2024-03-24T00:18:35.772 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:35.771+0000 7f6843600640 1 Processor -- start 2024-03-24T00:18:35.773 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:35.771+0000 7f6843600640 1 -- start start 2024-03-24T00:18:35.773 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:35.771+0000 7f6843600640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f683c1038e0 0x7f683c199c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:18:35.773 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:35.771+0000 7f6843600640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f683c104ad0 0x7f683c19a180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:18:35.773 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:35.771+0000 7f6843600640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f683c105490 0x7f683c1a11b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:18:35.773 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:35.771+0000 7f6843600640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f683c10f580 con 0x7f683c1038e0 2024-03-24T00:18:35.773 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:35.771+0000 7f6843600640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f683c10f400 con 0x7f683c104ad0 2024-03-24T00:18:35.773 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:35.771+0000 7f6843600640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f683c10f700 con 0x7f683c105490 2024-03-24T00:18:35.773 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:35.772+0000 7f683be00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f683c104ad0 0x7f683c19a180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:18:35.773 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:35.772+0000 7f683be00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f683c104ad0 0x7f683c19a180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.118:32786/0 (socket says 172.21.15.118:32786) 2024-03-24T00:18:35.773 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:35.772+0000 7f683be00640 1 -- 172.21.15.118:0/1494892837 learned_addr learned my addr 172.21.15.118:0/1494892837 (peer_addr_for_me v2:172.21.15.118:0/0) 2024-03-24T00:18:35.773 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:35.772+0000 7f6841600640 1 --2- 172.21.15.118:0/1494892837 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f683c105490 0x7f683c1a11b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:18:35.774 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:35.772+0000 7f6840c00640 1 --2- 172.21.15.118:0/1494892837 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f683c1038e0 0x7f683c199c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:18:35.774 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:35.772+0000 7f683be00640 1 -- 172.21.15.118:0/1494892837 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f683c105490 msgr2=0x7f683c1a11b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:18:35.774 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:35.772+0000 7f683be00640 1 --2- 172.21.15.118:0/1494892837 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f683c105490 0x7f683c1a11b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:18:35.774 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:35.772+0000 7f683be00640 1 -- 172.21.15.118:0/1494892837 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f683c1038e0 msgr2=0x7f683c199c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:18:35.774 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:35.772+0000 7f683be00640 1 --2- 172.21.15.118:0/1494892837 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f683c1038e0 0x7f683c199c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:18:35.774 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:35.772+0000 7f683be00640 1 -- 172.21.15.118:0/1494892837 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f683c1a1850 con 0x7f683c104ad0 2024-03-24T00:18:35.774 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:35.772+0000 7f6841600640 1 --2- 172.21.15.118:0/1494892837 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f683c105490 0x7f683c1a11b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T00:18:35.774 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:35.772+0000 7f6840c00640 1 --2- 172.21.15.118:0/1494892837 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f683c1038e0 0x7f683c199c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T00:18:35.774 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:35.772+0000 7f683be00640 1 --2- 172.21.15.118:0/1494892837 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f683c104ad0 0x7f683c19a180 secure :-1 s=READY pgs=340 cs=0 l=1 rev1=1 crypto rx=0x7f682800e9f0 tx=0x7f682800eeb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:18:35.774 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:35.773+0000 7f6839600640 1 -- 172.21.15.118:0/1494892837 <== mon.2 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f682800cd80 con 0x7f683c104ad0 2024-03-24T00:18:35.774 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:35.773+0000 7f6839600640 1 -- 172.21.15.118:0/1494892837 <== mon.2 v2:172.21.15.189:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f6828010910 con 0x7f683c104ad0 2024-03-24T00:18:35.774 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:35.773+0000 7f6843600640 1 -- 172.21.15.118:0/1494892837 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f683c1a1b40 con 0x7f683c104ad0 2024-03-24T00:18:35.775 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:35.773+0000 7f6839600640 1 -- 172.21.15.118:0/1494892837 <== mon.2 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6828018990 con 0x7f683c104ad0 2024-03-24T00:18:35.775 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:35.773+0000 7f6843600640 1 -- 172.21.15.118:0/1494892837 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f683c1a2060 con 0x7f683c104ad0 2024-03-24T00:18:35.776 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:35.777+0000 7f684624f740 1 Processor -- start 2024-03-24T00:18:35.777 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:35.777+0000 7f684624f740 1 -- start start 2024-03-24T00:18:35.778 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:35.777+0000 7f684624f740 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5601bd249810 0x5601bd249c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:18:35.779 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:35.777+0000 7f684624f740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5601bd24a9e0 0x5601bd24ae60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:18:35.779 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:35.777+0000 7f684624f740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5601bd24b3a0 0x5601bd251c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:18:35.779 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:35.777+0000 7f684624f740 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x5601bd1ae2c0 con 0x5601bd249810 2024-03-24T00:18:35.779 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:35.777+0000 7f684624f740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x5601bd156c90 con 0x5601bd24a9e0 2024-03-24T00:18:35.779 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:35.777+0000 7f684624f740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x5601bd1aa360 con 0x5601bd24b3a0 2024-03-24T00:18:35.779 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:35.778+0000 7f6840c00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5601bd24a9e0 0x5601bd24ae60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:18:35.779 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:35.778+0000 7f6840c00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5601bd24a9e0 0x5601bd24ae60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.118:32796/0 (socket says 172.21.15.118:32796) 2024-03-24T00:18:35.779 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:35.778+0000 7f6840c00640 1 -- 172.21.15.118:0/1844259454 learned_addr learned my addr 172.21.15.118:0/1844259454 (peer_addr_for_me v2:172.21.15.118:0/0) 2024-03-24T00:18:35.779 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:35.778+0000 7f6841600640 1 --2- 172.21.15.118:0/1844259454 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5601bd249810 0x5601bd249c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:18:35.779 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:35.778+0000 7f6839600640 1 -- 172.21.15.118:0/1494892837 <== mon.2 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 20) v1 ==== 137291+0+0 (secure 0 0 0) 0x7f6828010ac0 con 0x7f683c104ad0 2024-03-24T00:18:35.779 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:35.778+0000 7f683be00640 1 --2- 172.21.15.118:0/1844259454 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5601bd24b3a0 0x5601bd251c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:18:35.779 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:35.778+0000 7f6840c00640 1 -- 172.21.15.118:0/1844259454 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5601bd24b3a0 msgr2=0x5601bd251c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:18:35.779 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:35.778+0000 7f6840c00640 1 --2- 172.21.15.118:0/1844259454 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5601bd24b3a0 0x5601bd251c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:18:35.780 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:35.778+0000 7f6840c00640 1 -- 172.21.15.118:0/1844259454 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5601bd249810 msgr2=0x5601bd249c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:18:35.780 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:35.778+0000 7f6840c00640 1 --2- 172.21.15.118:0/1844259454 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5601bd249810 0x5601bd249c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:18:35.780 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:35.778+0000 7f6840c00640 1 -- 172.21.15.118:0/1844259454 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5601bd17b080 con 0x5601bd24a9e0 2024-03-24T00:18:35.780 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:35.778+0000 7f6841600640 1 --2- 172.21.15.118:0/1844259454 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5601bd249810 0x5601bd249c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T00:18:35.780 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:35.779+0000 7f6840c00640 1 --2- 172.21.15.118:0/1844259454 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5601bd24a9e0 0x5601bd24ae60 secure :-1 s=READY pgs=341 cs=0 l=1 rev1=1 crypto rx=0x7f6830009900 tx=0x7f68300095e0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=86d47ba534b3828a server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:18:35.780 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:35.779+0000 7f682dc00640 1 -- 172.21.15.118:0/1844259454 <== mon.2 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6830044020 con 0x5601bd24a9e0 2024-03-24T00:18:35.780 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:35.779+0000 7f682dc00640 1 -- 172.21.15.118:0/1844259454 <== mon.2 v2:172.21.15.189:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f683002cc40 con 0x5601bd24a9e0 2024-03-24T00:18:35.780 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:35.779+0000 7f682dc00640 1 -- 172.21.15.118:0/1844259454 <== mon.2 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6830004920 con 0x5601bd24a9e0 2024-03-24T00:18:35.783 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:35.780+0000 7f684624f740 1 -- 172.21.15.118:0/1844259454 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5601bd24a9e0 msgr2=0x5601bd24ae60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:18:35.783 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:35.780+0000 7f684624f740 1 --2- 172.21.15.118:0/1844259454 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5601bd24a9e0 0x5601bd24ae60 secure :-1 s=READY pgs=341 cs=0 l=1 rev1=1 crypto rx=0x7f6830009900 tx=0x7f68300095e0 comp rx=0 tx=0).stop 2024-03-24T00:18:35.783 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:35.780+0000 7f684624f740 1 -- 172.21.15.118:0/1844259454 shutdown_connections 2024-03-24T00:18:35.783 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:35.780+0000 7f684624f740 1 --2- 172.21.15.118:0/1844259454 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5601bd24b3a0 0x5601bd251c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:18:35.783 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:35.780+0000 7f684624f740 1 --2- 172.21.15.118:0/1844259454 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5601bd24a9e0 0x5601bd24ae60 unknown :-1 s=CLOSED pgs=341 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:18:35.783 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:35.780+0000 7f684624f740 1 --2- 172.21.15.118:0/1844259454 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5601bd249810 0x5601bd249c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:18:35.783 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:35.780+0000 7f684624f740 1 -- 172.21.15.118:0/1844259454 >> 172.21.15.118:0/1844259454 conn(0x5601bd244c00 msgr2=0x5601bd245000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:18:35.783 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:35.780+0000 7f6839600640 1 --2- 172.21.15.118:0/1494892837 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7f68240a2800 0x7f68240a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:18:35.783 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:35.780+0000 7f6839600640 1 -- 172.21.15.118:0/1494892837 <== mon.2 v2:172.21.15.189:3300/0 5 ==== osd_map(90..90 src has 1..90) v4 ==== 8277+0+0 (secure 0 0 0) 0x7f68280cef20 con 0x7f683c104ad0 2024-03-24T00:18:35.784 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:35.780+0000 7f6841600640 1 --2- 172.21.15.118:0/1494892837 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7f68240a2800 0x7f68240a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:18:35.784 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:35.781+0000 7f684624f740 1 -- 172.21.15.118:0/1844259454 shutdown_connections 2024-03-24T00:18:35.784 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:35.781+0000 7f684624f740 1 -- 172.21.15.118:0/1844259454 wait complete. 2024-03-24T00:18:35.784 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:35.781+0000 7f684624f740 1 -- start start 2024-03-24T00:18:35.784 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:35.781+0000 7f6841600640 1 --2- 172.21.15.118:0/1494892837 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7f68240a2800 0x7f68240a4cc0 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 crypto rx=0x5601bd24a8e0 tx=0x7f6834009500 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:18:35.784 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:35.781+0000 7f684624f740 1 Processor -- start 2024-03-24T00:18:35.784 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:35.782+0000 7f684624f740 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5601bd249810 0x5601bd253b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:18:35.784 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:35.782+0000 7f684624f740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5601bd24a9e0 0x5601bd2540c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:18:35.784 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:35.782+0000 7f684624f740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5601bd24b3a0 0x5601bd25b020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:18:35.784 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:35.782+0000 7f684624f740 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x5601bd252350 con 0x5601bd249810 2024-03-24T00:18:35.784 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:35.782+0000 7f684624f740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x5601bd16a0d0 con 0x5601bd24a9e0 2024-03-24T00:18:35.784 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:35.782+0000 7f684624f740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x5601bd2524d0 con 0x5601bd24b3a0 2024-03-24T00:18:35.784 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:35.782+0000 7f684624f740 10 client.0 _ll_register_callbacks cb 0x5601bd16ee80 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T00:18:35.784 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:35.782+0000 7f6840c00640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5601bd249810 0x5601bd253b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:18:35.785 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:35.782+0000 7f6841600640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5601bd24b3a0 0x5601bd25b020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:18:35.785 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:35.782+0000 7f683be00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5601bd24a9e0 0x5601bd2540c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:18:35.785 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:35.782+0000 7f6840c00640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5601bd249810 0x5601bd253b80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.118:3300/0 says I am v2:172.21.15.118:38022/0 (socket says 172.21.15.118:38022) 2024-03-24T00:18:35.785 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:35.782+0000 7f6840c00640 1 -- 172.21.15.118:0/2995423677 learned_addr learned my addr 172.21.15.118:0/2995423677 (peer_addr_for_me v2:172.21.15.118:0/0) 2024-03-24T00:18:35.785 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:35.782+0000 7f6841600640 1 -- 172.21.15.118:0/2995423677 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5601bd24a9e0 msgr2=0x5601bd2540c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:18:35.785 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:35.782+0000 7f6841600640 1 --2- 172.21.15.118:0/2995423677 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5601bd24a9e0 0x5601bd2540c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:18:35.785 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:35.782+0000 7f6841600640 1 -- 172.21.15.118:0/2995423677 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5601bd249810 msgr2=0x5601bd253b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:18:35.785 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:35.782+0000 7f6841600640 1 --2- 172.21.15.118:0/2995423677 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5601bd249810 0x5601bd253b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:18:35.785 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:35.783+0000 7f6841600640 1 -- 172.21.15.118:0/2995423677 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5601bcec23f0 con 0x5601bd24b3a0 2024-03-24T00:18:35.785 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:35.783+0000 7f6840c00640 1 --2- 172.21.15.118:0/2995423677 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5601bd249810 0x5601bd253b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T00:18:35.785 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:35.783+0000 7f6841600640 1 --2- 172.21.15.118:0/2995423677 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5601bd24b3a0 0x5601bd25b020 secure :-1 s=READY pgs=341 cs=0 l=1 rev1=1 crypto rx=0x7f6834009fd0 tx=0x7f6834004ab0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=4bb352dc253ddbd7 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:18:35.785 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:35.783+0000 7f6822000640 10 client.0 ms_handle_connect on v2:172.21.15.193:3300/0 2024-03-24T00:18:35.786 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:35.783+0000 7f6822000640 1 -- 172.21.15.118:0/2995423677 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6834003e50 con 0x5601bd24b3a0 2024-03-24T00:18:35.786 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:35.783+0000 7f684624f740 1 -- 172.21.15.118:0/2995423677 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f683002cc40 con 0x5601bd24b3a0 2024-03-24T00:18:35.786 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:35.783+0000 7f6822000640 1 -- 172.21.15.118:0/2995423677 <== mon.1 v2:172.21.15.193:3300/0 2 ==== osd_map(90..90 src has 1..90) v4 ==== 8277+0+0 (secure 0 0 0) 0x7f6834004030 con 0x5601bd24b3a0 2024-03-24T00:18:35.786 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:35.783+0000 7f6822000640 1 -- 172.21.15.118:0/2995423677 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) v2 ==== 24+0+0 (secure 0 0 0) 0x7f6834004230 con 0x5601bd24b3a0 2024-03-24T00:18:35.786 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:35.783+0000 7f684624f740 10 client.26239 fetch_fsmap learned FSMap version 28 2024-03-24T00:18:35.786 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:35.783+0000 7f684624f740 1 -- 172.21.15.118:0/2995423677 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) v3 -- 0x5601bcf8c7d0 con 0x5601bd24b3a0 2024-03-24T00:18:35.788 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:35.784+0000 7f6822000640 1 -- 172.21.15.118:0/2995423677 <== mon.1 v2:172.21.15.193:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f683002cc40 con 0x5601bd24b3a0 2024-03-24T00:18:35.788 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:35.784+0000 7f6822000640 1 -- 172.21.15.118:0/2995423677 <== mon.1 v2:172.21.15.193:3300/0 5 ==== fsmap(e 28) v1 ==== 7934+0+0 (secure 0 0 0) 0x7f6834033070 con 0x5601bd24b3a0 2024-03-24T00:18:35.788 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:35.784+0000 7f684624f740 10 client.26239 fetch_fsmap finished waiting for FSMap version 28 2024-03-24T00:18:35.788 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:35.784+0000 7f684624f740 10 client.26239 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.f 2024-03-24T00:18:35.788 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:35.784+0000 7f684624f740 20 client.26239 populate_metadata read hostname 'smithi118' 2024-03-24T00:18:35.788 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:35.784+0000 7f684624f740 1 --2- 172.21.15.118:0/2995423677 >> [v2:172.21.15.193:6834/2055310541,v1:172.21.15.193:6835/2055310541] conn(0x5601bd25f4e0 0x5601bd261980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:18:35.788 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:35.784+0000 7f684624f740 4 client.26239 mds_command: new command op to 24499 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-24T00:18:35.788 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:35.784+0000 7f684624f740 1 -- 172.21.15.118:0/2995423677 --> [v2:172.21.15.193:6834/2055310541,v1:172.21.15.193:6835/2055310541] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x5601bd1bb800 con 0x5601bd25f4e0 2024-03-24T00:18:35.788 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:35.784+0000 7f6840c00640 1 --2- 172.21.15.118:0/2995423677 >> [v2:172.21.15.193:6834/2055310541,v1:172.21.15.193:6835/2055310541] conn(0x5601bd25f4e0 0x5601bd261980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:18:35.788 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:35.785+0000 7f6840c00640 1 --2- 172.21.15.118:0/2995423677 >> [v2:172.21.15.193:6834/2055310541,v1:172.21.15.193:6835/2055310541] conn(0x5601bd25f4e0 0x5601bd261980 crc :-1 s=READY pgs=33 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=78c6cc17a3e90b12 server_cookie=1280150963ca45e0 in_seq=0 out_seq=0 2024-03-24T00:18:35.788 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:35.785+0000 7f6822000640 10 client.26239 ms_handle_connect on v2:172.21.15.193:6834/2055310541 2024-03-24T00:18:35.788 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:35.786+0000 7f6822000640 1 -- 172.21.15.118:0/2995423677 <== mds.3 v2:172.21.15.193:6834/2055310541 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x5601bd1bb800 con 0x5601bd25f4e0 2024-03-24T00:18:35.789 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:35.786+0000 7f6822000640 10 client.26239 handle_command_reply: tid=0 2024-03-24T00:18:35.789 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:35.786+0000 7f684624f740 1 client.26239 shutdown 2024-03-24T00:18:35.789 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:35.786+0000 7f684624f740 1 -- 172.21.15.118:0/2995423677 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5601bd24b3a0 msgr2=0x5601bd25b020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:18:35.789 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:35.786+0000 7f684624f740 1 --2- 172.21.15.118:0/2995423677 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5601bd24b3a0 0x5601bd25b020 secure :-1 s=READY pgs=341 cs=0 l=1 rev1=1 crypto rx=0x7f6834009fd0 tx=0x7f6834004ab0 comp rx=0 tx=0).stop 2024-03-24T00:18:35.789 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:35.786+0000 7f684624f740 1 -- 172.21.15.118:0/2995423677 shutdown_connections 2024-03-24T00:18:35.789 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:35.786+0000 7f684624f740 1 --2- 172.21.15.118:0/2995423677 >> [v2:172.21.15.193:6834/2055310541,v1:172.21.15.193:6835/2055310541] conn(0x5601bd25f4e0 0x5601bd261980 crc :-1 s=READY pgs=33 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:18:35.789 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:35.786+0000 7f684624f740 1 --2- 172.21.15.118:0/2995423677 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5601bd24b3a0 0x5601bd25b020 unknown :-1 s=CLOSED pgs=341 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:18:35.789 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:35.786+0000 7f684624f740 1 --2- 172.21.15.118:0/2995423677 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5601bd24a9e0 0x5601bd2540c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:18:35.789 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:35.786+0000 7f684624f740 1 --2- 172.21.15.118:0/2995423677 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5601bd249810 0x5601bd253b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:18:35.789 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:35.786+0000 7f684624f740 1 -- 172.21.15.118:0/2995423677 >> 172.21.15.118:0/2995423677 conn(0x5601bd244c00 msgr2=0x5601bd243eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:18:35.789 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:35.786+0000 7f6822000640 0 client.26239 ms_handle_reset on v2:172.21.15.193:6834/2055310541 2024-03-24T00:18:35.789 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:35.786+0000 7f684624f740 1 -- 172.21.15.118:0/2995423677 shutdown_connections 2024-03-24T00:18:35.789 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:35.786+0000 7f684624f740 1 -- 172.21.15.118:0/2995423677 wait complete. 2024-03-24T00:18:35.789 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:35.786+0000 7f684624f740 20 client.26239 trim_cache size 0 max 16384 2024-03-24T00:18:35.807 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:35.807+0000 7f684624f740 1 Processor -- start 2024-03-24T00:18:35.807 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:35.807+0000 7f684624f740 1 -- start start 2024-03-24T00:18:35.807 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:35.807+0000 7f684624f740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5601bd249810 0x5601bd2614f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:18:35.807 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:35.807+0000 7f684624f740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5601bd24a9e0 0x5601bd25afe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:18:35.807 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:35.807+0000 7f684624f740 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5601bd24b3a0 0x5601bd24db60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:18:35.807 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:35.807+0000 7f684624f740 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f683c1aacd0 con 0x5601bd24b3a0 2024-03-24T00:18:35.807 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:35.807+0000 7f684624f740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f683c1ac650 con 0x5601bd249810 2024-03-24T00:18:35.807 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:35.807+0000 7f684624f740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f683c1ab330 con 0x5601bd24a9e0 2024-03-24T00:18:35.808 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:35.808+0000 7f6840c00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5601bd249810 0x5601bd2614f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:18:35.808 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:35.808+0000 7f683be00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5601bd24a9e0 0x5601bd25afe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:18:35.809 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:35.808+0000 7f6840c00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5601bd249810 0x5601bd2614f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.118:32824/0 (socket says 172.21.15.118:32824) 2024-03-24T00:18:35.809 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:35.808+0000 7f6840c00640 1 -- 172.21.15.118:0/2654582513 learned_addr learned my addr 172.21.15.118:0/2654582513 (peer_addr_for_me v2:172.21.15.118:0/0) 2024-03-24T00:18:35.809 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:35.808+0000 7f6841600640 1 --2- 172.21.15.118:0/2654582513 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5601bd24b3a0 0x5601bd24db60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:18:35.809 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:35.808+0000 7f683be00640 1 -- 172.21.15.118:0/2654582513 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5601bd249810 msgr2=0x5601bd2614f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:18:35.809 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:35.808+0000 7f683be00640 1 --2- 172.21.15.118:0/2654582513 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5601bd249810 0x5601bd2614f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:18:35.809 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:35.808+0000 7f683be00640 1 -- 172.21.15.118:0/2654582513 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5601bd24b3a0 msgr2=0x5601bd24db60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:18:35.809 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:35.808+0000 7f683be00640 1 --2- 172.21.15.118:0/2654582513 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5601bd24b3a0 0x5601bd24db60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:18:35.809 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:35.808+0000 7f683be00640 1 -- 172.21.15.118:0/2654582513 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5601bd245580 con 0x5601bd24a9e0 2024-03-24T00:18:35.809 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:35.808+0000 7f683be00640 1 --2- 172.21.15.118:0/2654582513 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5601bd24a9e0 0x5601bd25afe0 secure :-1 s=READY pgs=342 cs=0 l=1 rev1=1 crypto rx=0x7f68280cca40 tx=0x7f6828018420 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c35c9b005c1926e1 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:18:35.809 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:35.809+0000 7f682d200640 1 -- 172.21.15.118:0/2654582513 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f682809b1f0 con 0x5601bd24a9e0 2024-03-24T00:18:35.809 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:35.809+0000 7f682d200640 1 -- 172.21.15.118:0/2654582513 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f682809bf90 con 0x5601bd24a9e0 2024-03-24T00:18:35.809 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:35.809+0000 7f682d200640 1 -- 172.21.15.118:0/2654582513 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f68280999d0 con 0x5601bd24a9e0 2024-03-24T00:18:35.810 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:35.810+0000 7f684624f740 1 -- 172.21.15.118:0/2654582513 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5601bd24a9e0 msgr2=0x5601bd25afe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:18:35.810 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:35.810+0000 7f684624f740 1 --2- 172.21.15.118:0/2654582513 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5601bd24a9e0 0x5601bd25afe0 secure :-1 s=READY pgs=342 cs=0 l=1 rev1=1 crypto rx=0x7f68280cca40 tx=0x7f6828018420 comp rx=0 tx=0).stop 2024-03-24T00:18:35.810 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:35.810+0000 7f684624f740 1 -- 172.21.15.118:0/2654582513 shutdown_connections 2024-03-24T00:18:35.811 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:35.810+0000 7f684624f740 1 --2- 172.21.15.118:0/2654582513 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5601bd24b3a0 0x5601bd24db60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:18:35.811 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:35.810+0000 7f684624f740 1 --2- 172.21.15.118:0/2654582513 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5601bd24a9e0 0x5601bd25afe0 unknown :-1 s=CLOSED pgs=342 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:18:35.811 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:35.810+0000 7f684624f740 1 --2- 172.21.15.118:0/2654582513 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5601bd249810 0x5601bd2614f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:18:35.811 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:35.810+0000 7f684624f740 1 -- 172.21.15.118:0/2654582513 >> 172.21.15.118:0/2654582513 conn(0x5601bd244c00 msgr2=0x5601bd243eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:18:35.811 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:35.810+0000 7f684624f740 1 -- 172.21.15.118:0/2654582513 shutdown_connections 2024-03-24T00:18:35.811 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:35.810+0000 7f684624f740 1 -- 172.21.15.118:0/2654582513 wait complete. 2024-03-24T00:18:35.811 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:35.810+0000 7f684624f740 1 -- start start 2024-03-24T00:18:35.811 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:35.811+0000 7f684624f740 1 Processor -- start 2024-03-24T00:18:35.813 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:35.811+0000 7f684624f740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5601bd249810 0x5601bd257060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:18:35.813 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:35.811+0000 7f684624f740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5601bd24a9e0 0x5601bd2575a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:18:35.813 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:35.811+0000 7f684624f740 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5601bd24b3a0 0x5601bd25a570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:18:35.813 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:35.811+0000 7f684624f740 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f683c1abff0 con 0x5601bd24b3a0 2024-03-24T00:18:35.814 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:35.811+0000 7f684624f740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f683c1ab990 con 0x5601bd249810 2024-03-24T00:18:35.814 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:35.811+0000 7f684624f740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f683c1a8e90 con 0x5601bd24a9e0 2024-03-24T00:18:35.814 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:35.811+0000 7f684624f740 10 client.0 _ll_register_callbacks cb 0x7f6834012630 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T00:18:35.814 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:35.811+0000 7f6840c00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5601bd249810 0x5601bd257060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:18:35.814 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:35.812+0000 7f6840c00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5601bd249810 0x5601bd257060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.118:32840/0 (socket says 172.21.15.118:32840) 2024-03-24T00:18:35.814 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:35.812+0000 7f6840c00640 1 -- 172.21.15.118:0/3735394745 learned_addr learned my addr 172.21.15.118:0/3735394745 (peer_addr_for_me v2:172.21.15.118:0/0) 2024-03-24T00:18:35.814 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:35.812+0000 7f683be00640 1 --2- 172.21.15.118:0/3735394745 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5601bd24a9e0 0x5601bd2575a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:18:35.814 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:35.812+0000 7f6841600640 1 --2- 172.21.15.118:0/3735394745 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5601bd24b3a0 0x5601bd25a570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:18:35.814 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:35.812+0000 7f6840c00640 1 -- 172.21.15.118:0/3735394745 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5601bd24a9e0 msgr2=0x5601bd2575a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:18:35.814 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:35.812+0000 7f6840c00640 1 --2- 172.21.15.118:0/3735394745 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5601bd24a9e0 0x5601bd2575a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:18:35.814 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:35.812+0000 7f6840c00640 1 -- 172.21.15.118:0/3735394745 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5601bd24b3a0 msgr2=0x5601bd25a570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:18:35.814 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:35.812+0000 7f6840c00640 1 --2- 172.21.15.118:0/3735394745 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5601bd24b3a0 0x5601bd25a570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:18:35.814 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:35.812+0000 7f6840c00640 1 -- 172.21.15.118:0/3735394745 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5601bd25ab60 con 0x5601bd249810 2024-03-24T00:18:35.815 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:35.812+0000 7f6840c00640 1 --2- 172.21.15.118:0/3735394745 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5601bd249810 0x5601bd257060 secure :-1 s=READY pgs=342 cs=0 l=1 rev1=1 crypto rx=0x7f683002c7b0 tx=0x7f683002cdf0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=91b2cf8d1de786c0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:18:35.815 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:35.812+0000 7f6822000640 10 client.0 ms_handle_connect on v2:172.21.15.189:3300/0 2024-03-24T00:18:35.815 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:35.813+0000 7f6822000640 1 -- 172.21.15.118:0/3735394745 <== mon.2 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6830057060 con 0x5601bd249810 2024-03-24T00:18:35.815 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:35.813+0000 7f6822000640 1 -- 172.21.15.118:0/3735394745 <== mon.2 v2:172.21.15.189:3300/0 2 ==== osd_map(90..90 src has 1..90) v4 ==== 8277+0+0 (secure 0 0 0) 0x7f683002c980 con 0x5601bd249810 2024-03-24T00:18:35.815 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:35.813+0000 7f684624f740 1 -- 172.21.15.118:0/3735394745 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f682809bf90 con 0x5601bd249810 2024-03-24T00:18:35.815 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:35.813+0000 7f6822000640 1 -- 172.21.15.118:0/3735394745 <== mon.2 v2:172.21.15.189:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) v2 ==== 24+0+0 (secure 0 0 0) 0x7f6830044040 con 0x5601bd249810 2024-03-24T00:18:35.815 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:35.813+0000 7f684624f740 10 client.26234 fetch_fsmap learned FSMap version 28 2024-03-24T00:18:35.815 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:35.813+0000 7f684624f740 1 -- 172.21.15.118:0/3735394745 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) v3 -- 0x5601bd2bb6e0 con 0x5601bd249810 2024-03-24T00:18:35.815 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:35.813+0000 7f6822000640 1 -- 172.21.15.118:0/3735394745 <== mon.2 v2:172.21.15.189:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f682809bf90 con 0x5601bd249810 2024-03-24T00:18:35.815 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:35.813+0000 7f6822000640 1 -- 172.21.15.118:0/3735394745 <== mon.2 v2:172.21.15.189:3300/0 5 ==== fsmap(e 28) v1 ==== 7934+0+0 (secure 0 0 0) 0x7f68300405e0 con 0x5601bd249810 2024-03-24T00:18:35.815 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:35.813+0000 7f684624f740 10 client.26234 fetch_fsmap finished waiting for FSMap version 28 2024-03-24T00:18:35.815 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:35.813+0000 7f684624f740 10 client.26234 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.f 2024-03-24T00:18:35.815 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:35.813+0000 7f684624f740 20 client.26234 populate_metadata read hostname 'smithi118' 2024-03-24T00:18:35.815 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:35.813+0000 7f684624f740 1 --2- 172.21.15.118:0/3735394745 >> [v2:172.21.15.193:6834/2055310541,v1:172.21.15.193:6835/2055310541] conn(0x5601bd2bbc40 0x5601bd2be030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:18:35.815 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:35.813+0000 7f684624f740 4 client.26234 mds_command: new command op to 24499 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-24T00:18:35.816 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:35.813+0000 7f684624f740 1 -- 172.21.15.118:0/3735394745 --> [v2:172.21.15.193:6834/2055310541,v1:172.21.15.193:6835/2055310541] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x5601bd1b5b50 con 0x5601bd2bbc40 2024-03-24T00:18:35.816 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:35.814+0000 7f683be00640 1 --2- 172.21.15.118:0/3735394745 >> [v2:172.21.15.193:6834/2055310541,v1:172.21.15.193:6835/2055310541] conn(0x5601bd2bbc40 0x5601bd2be030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:18:35.817 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:35.814+0000 7f683be00640 1 --2- 172.21.15.118:0/3735394745 >> [v2:172.21.15.193:6834/2055310541,v1:172.21.15.193:6835/2055310541] conn(0x5601bd2bbc40 0x5601bd2be030 crc :-1 s=READY pgs=34 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=55401610efd271f5 server_cookie=d4d5295499c79aba in_seq=0 out_seq=0 2024-03-24T00:18:35.817 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:35.814+0000 7f6822000640 10 client.26234 ms_handle_connect on v2:172.21.15.193:6834/2055310541 2024-03-24T00:18:35.817 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:35.814+0000 7f6822000640 1 -- 172.21.15.118:0/3735394745 <== mds.3 v2:172.21.15.193:6834/2055310541 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x5601bd1b5b50 con 0x5601bd2bbc40 2024-03-24T00:18:35.817 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:35.814+0000 7f6822000640 10 client.26234 handle_command_reply: tid=0 2024-03-24T00:18:35.817 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:35.814+0000 7f684624f740 1 client.26234 shutdown 2024-03-24T00:18:35.817 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:35.815+0000 7f684624f740 1 -- 172.21.15.118:0/3735394745 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5601bd249810 msgr2=0x5601bd257060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:18:35.817 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:35.815+0000 7f684624f740 1 --2- 172.21.15.118:0/3735394745 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5601bd249810 0x5601bd257060 secure :-1 s=READY pgs=342 cs=0 l=1 rev1=1 crypto rx=0x7f683002c7b0 tx=0x7f683002cdf0 comp rx=0 tx=0).stop 2024-03-24T00:18:35.818 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:35.815+0000 7f684624f740 1 -- 172.21.15.118:0/3735394745 shutdown_connections 2024-03-24T00:18:35.818 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:35.815+0000 7f684624f740 1 --2- 172.21.15.118:0/3735394745 >> [v2:172.21.15.193:6834/2055310541,v1:172.21.15.193:6835/2055310541] conn(0x5601bd2bbc40 0x5601bd2be030 crc :-1 s=READY pgs=34 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:18:35.818 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:35.815+0000 7f684624f740 1 --2- 172.21.15.118:0/3735394745 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5601bd24b3a0 0x5601bd25a570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:18:35.818 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:35.815+0000 7f684624f740 1 --2- 172.21.15.118:0/3735394745 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5601bd24a9e0 0x5601bd2575a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:18:35.818 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:35.815+0000 7f6822000640 0 client.26234 ms_handle_reset on v2:172.21.15.193:6834/2055310541 2024-03-24T00:18:35.818 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:35.815+0000 7f684624f740 1 --2- 172.21.15.118:0/3735394745 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5601bd249810 0x5601bd257060 unknown :-1 s=CLOSED pgs=342 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:18:35.818 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:35.815+0000 7f684624f740 1 -- 172.21.15.118:0/3735394745 >> 172.21.15.118:0/3735394745 conn(0x5601bd244c00 msgr2=0x5601bd247e60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:18:35.818 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:35.815+0000 7f684624f740 1 -- 172.21.15.118:0/3735394745 shutdown_connections 2024-03-24T00:18:35.818 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:35.815+0000 7f684624f740 1 -- 172.21.15.118:0/3735394745 wait complete. 2024-03-24T00:18:35.818 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:35.815+0000 7f684624f740 20 client.26234 trim_cache size 0 max 16384 2024-03-24T00:18:35.819 INFO:teuthology.orchestra.run.smithi118.stdout:[] 2024-03-24T00:18:35.819 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:35.816+0000 7f6843600640 1 -- 172.21.15.118:0/1494892837 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7f68240a2800 msgr2=0x7f68240a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:18:35.819 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:35.816+0000 7f6843600640 1 --2- 172.21.15.118:0/1494892837 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7f68240a2800 0x7f68240a4cc0 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 crypto rx=0x5601bd24a8e0 tx=0x7f6834009500 comp rx=0 tx=0).stop 2024-03-24T00:18:35.819 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:35.816+0000 7f6843600640 1 -- 172.21.15.118:0/1494892837 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f683c104ad0 msgr2=0x7f683c19a180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:18:35.820 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:35.816+0000 7f6843600640 1 --2- 172.21.15.118:0/1494892837 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f683c104ad0 0x7f683c19a180 secure :-1 s=READY pgs=340 cs=0 l=1 rev1=1 crypto rx=0x7f682800e9f0 tx=0x7f682800eeb0 comp rx=0 tx=0).stop 2024-03-24T00:18:35.820 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:35.816+0000 7f6843600640 1 -- 172.21.15.118:0/1494892837 shutdown_connections 2024-03-24T00:18:35.820 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:35.816+0000 7f6843600640 1 --2- 172.21.15.118:0/1494892837 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7f68240a2800 0x7f68240a4cc0 unknown :-1 s=CLOSED pgs=322 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:18:35.820 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:35.816+0000 7f6843600640 1 --2- 172.21.15.118:0/1494892837 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f683c105490 0x7f683c1a11b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:18:35.820 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:35.816+0000 7f6843600640 1 --2- 172.21.15.118:0/1494892837 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f683c104ad0 0x7f683c19a180 unknown :-1 s=CLOSED pgs=340 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:18:35.820 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:35.816+0000 7f6843600640 1 --2- 172.21.15.118:0/1494892837 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f683c1038e0 0x7f683c199c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:18:35.820 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:35.816+0000 7f6843600640 1 -- 172.21.15.118:0/1494892837 >> 172.21.15.118:0/1494892837 conn(0x7f683c0fee50 msgr2=0x7f683c107e20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:18:35.820 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:35.816+0000 7f6843600640 1 -- 172.21.15.118:0/1494892837 shutdown_connections 2024-03-24T00:18:35.820 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:35.816+0000 7f6843600640 1 -- 172.21.15.118:0/1494892837 wait complete. 2024-03-24T00:18:35.831 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:18:35 smithi118 ceph-mon[31535]: from='client.? 172.21.15.118:0/1085924255' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-24T00:18:35.834 INFO:teuthology.orchestra.run:Running command with timeout 120 2024-03-24T00:18:35.834 DEBUG:teuthology.orchestra.run.smithi118:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:2 damage ls 2024-03-24T00:18:36.042 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:36.042+0000 7f5123600640 1 -- 172.21.15.118:0/2869773010 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f511c1028a0 msgr2=0x7f511c102d20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:18:36.042 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:36.042+0000 7f5123600640 1 --2- 172.21.15.118:0/2869773010 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f511c1028a0 0x7f511c102d20 secure :-1 s=READY pgs=343 cs=0 l=1 rev1=1 crypto rx=0x7f5104009900 tx=0x7f510402bc30 comp rx=0 tx=0).stop 2024-03-24T00:18:36.042 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:36.042+0000 7f5123600640 1 -- 172.21.15.118:0/2869773010 shutdown_connections 2024-03-24T00:18:36.042 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:36.042+0000 7f5123600640 1 --2- 172.21.15.118:0/2869773010 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f511c103260 0x7f511c109af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:18:36.042 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:36.042+0000 7f5123600640 1 --2- 172.21.15.118:0/2869773010 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f511c1028a0 0x7f511c102d20 unknown :-1 s=CLOSED pgs=343 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:18:36.042 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:36.042+0000 7f5123600640 1 --2- 172.21.15.118:0/2869773010 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f511c1016b0 0x7f511c101ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:18:36.043 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:36.042+0000 7f5123600640 1 -- 172.21.15.118:0/2869773010 >> 172.21.15.118:0/2869773010 conn(0x7f511c0fcc40 msgr2=0x7f511c0ff060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:18:36.043 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:36.042+0000 7f5123600640 1 -- 172.21.15.118:0/2869773010 shutdown_connections 2024-03-24T00:18:36.043 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:36.042+0000 7f5123600640 1 -- 172.21.15.118:0/2869773010 wait complete. 2024-03-24T00:18:36.044 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:36.043+0000 7f5123600640 1 Processor -- start 2024-03-24T00:18:36.044 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:36.043+0000 7f5123600640 1 -- start start 2024-03-24T00:18:36.044 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:36.044+0000 7f5123600640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f511c1016b0 0x7f511c199bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:18:36.044 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:36.044+0000 7f5123600640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f511c1028a0 0x7f511c19a110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:18:36.044 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:36.044+0000 7f5123600640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f511c103260 0x7f511c1a1180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:18:36.045 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:36.044+0000 7f5123600640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f511c073c60 con 0x7f511c1016b0 2024-03-24T00:18:36.045 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:36.044+0000 7f5123600640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f511c073ae0 con 0x7f511c103260 2024-03-24T00:18:36.045 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:36.044+0000 7f5123600640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f511c073de0 con 0x7f511c1028a0 2024-03-24T00:18:36.045 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:36.044+0000 7f5120c00640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f511c1016b0 0x7f511c199bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:18:36.045 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:36.044+0000 7f5113e00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f511c1028a0 0x7f511c19a110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:18:36.045 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:36.044+0000 7f5120c00640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f511c1016b0 0x7f511c199bd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.118:3300/0 says I am v2:172.21.15.118:38074/0 (socket says 172.21.15.118:38074) 2024-03-24T00:18:36.045 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:36.044+0000 7f5113e00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f511c1028a0 0x7f511c19a110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.118:53008/0 (socket says 172.21.15.118:53008) 2024-03-24T00:18:36.045 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:36.044+0000 7f5120c00640 1 -- 172.21.15.118:0/3871521605 learned_addr learned my addr 172.21.15.118:0/3871521605 (peer_addr_for_me v2:172.21.15.118:0/0) 2024-03-24T00:18:36.045 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:36.044+0000 7f5121600640 1 --2- 172.21.15.118:0/3871521605 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f511c103260 0x7f511c1a1180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:18:36.045 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:36.045+0000 7f5113e00640 1 -- 172.21.15.118:0/3871521605 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f511c103260 msgr2=0x7f511c1a1180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:18:36.045 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:36.045+0000 7f5113e00640 1 --2- 172.21.15.118:0/3871521605 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f511c103260 0x7f511c1a1180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:18:36.045 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:36.045+0000 7f5113e00640 1 -- 172.21.15.118:0/3871521605 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f511c1016b0 msgr2=0x7f511c199bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:18:36.045 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:36.045+0000 7f5113e00640 1 --2- 172.21.15.118:0/3871521605 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f511c1016b0 0x7f511c199bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:18:36.046 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:36.045+0000 7f5113e00640 1 -- 172.21.15.118:0/3871521605 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f511c1a17f0 con 0x7f511c1028a0 2024-03-24T00:18:36.046 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:36.045+0000 7f5120c00640 1 --2- 172.21.15.118:0/3871521605 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f511c1016b0 0x7f511c199bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T00:18:36.046 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:36.045+0000 7f5121600640 1 --2- 172.21.15.118:0/3871521605 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f511c103260 0x7f511c1a1180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T00:18:36.046 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:36.045+0000 7f5113e00640 1 --2- 172.21.15.118:0/3871521605 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f511c1028a0 0x7f511c19a110 secure :-1 s=READY pgs=343 cs=0 l=1 rev1=1 crypto rx=0x7f5104036040 tx=0x7f510402c650 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:18:36.046 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:36.045+0000 7f511a000640 1 -- 172.21.15.118:0/3871521605 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5104043020 con 0x7f511c1028a0 2024-03-24T00:18:36.047 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:36.045+0000 7f5123600640 1 -- 172.21.15.118:0/3871521605 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f511c1a1a80 con 0x7f511c1028a0 2024-03-24T00:18:36.047 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:36.045+0000 7f511a000640 1 -- 172.21.15.118:0/3871521605 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f51040054e0 con 0x7f511c1028a0 2024-03-24T00:18:36.047 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:36.045+0000 7f511a000640 1 -- 172.21.15.118:0/3871521605 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5104003c30 con 0x7f511c1028a0 2024-03-24T00:18:36.047 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:36.046+0000 7f5123600640 1 -- 172.21.15.118:0/3871521605 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f511c1a1ff0 con 0x7f511c1028a0 2024-03-24T00:18:36.048 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:36.049+0000 7f511a000640 1 -- 172.21.15.118:0/3871521605 <== mon.1 v2:172.21.15.193:3300/0 4 ==== mgrmap(e 20) v1 ==== 137291+0+0 (secure 0 0 0) 0x7f5104046020 con 0x7f511c1028a0 2024-03-24T00:18:36.048 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:36.049+0000 7f51263ae740 1 Processor -- start 2024-03-24T00:18:36.050 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:36.049+0000 7f51263ae740 1 -- start start 2024-03-24T00:18:36.050 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:36.049+0000 7f51263ae740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56289ec9d810 0x56289ec9dc30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:18:36.050 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:36.049+0000 7f51263ae740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56289ec9e9e0 0x56289ec9ee60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:18:36.050 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:36.049+0000 7f51263ae740 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x56289ec9f3a0 0x56289eca5c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:18:36.050 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:36.050+0000 7f51263ae740 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x56289ec022c0 con 0x56289ec9f3a0 2024-03-24T00:18:36.050 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:36.050+0000 7f51263ae740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x56289ebaac90 con 0x56289ec9d810 2024-03-24T00:18:36.050 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:36.050+0000 7f51263ae740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x56289ebfe360 con 0x56289ec9e9e0 2024-03-24T00:18:36.050 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:36.050+0000 7f5120c00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56289ec9e9e0 0x56289ec9ee60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:18:36.050 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:36.050+0000 7f5121600640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56289ec9d810 0x56289ec9dc30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:18:36.050 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:36.050+0000 7f5121600640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56289ec9d810 0x56289ec9dc30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.118:32866/0 (socket says 172.21.15.118:32866) 2024-03-24T00:18:36.050 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:36.050+0000 7f5121600640 1 -- 172.21.15.118:0/2262759657 learned_addr learned my addr 172.21.15.118:0/2262759657 (peer_addr_for_me v2:172.21.15.118:0/0) 2024-03-24T00:18:36.051 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:36.050+0000 7f5120c00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56289ec9e9e0 0x56289ec9ee60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.118:53024/0 (socket says 172.21.15.118:53024) 2024-03-24T00:18:36.051 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:36.050+0000 7f5113e00640 1 --2- 172.21.15.118:0/2262759657 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x56289ec9f3a0 0x56289eca5c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:18:36.051 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:36.050+0000 7f5121600640 1 -- 172.21.15.118:0/2262759657 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56289ec9e9e0 msgr2=0x56289ec9ee60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:18:36.051 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:36.050+0000 7f5121600640 1 --2- 172.21.15.118:0/2262759657 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56289ec9e9e0 0x56289ec9ee60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:18:36.051 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:36.050+0000 7f5121600640 1 -- 172.21.15.118:0/2262759657 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x56289ec9f3a0 msgr2=0x56289eca5c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:18:36.051 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:36.050+0000 7f5121600640 1 --2- 172.21.15.118:0/2262759657 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x56289ec9f3a0 0x56289eca5c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:18:36.051 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:36.050+0000 7f5121600640 1 -- 172.21.15.118:0/2262759657 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x56289ebcf080 con 0x56289ec9d810 2024-03-24T00:18:36.051 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:36.050+0000 7f5113e00640 1 --2- 172.21.15.118:0/2262759657 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x56289ec9f3a0 0x56289eca5c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T00:18:36.052 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:36.051+0000 7f5121600640 1 --2- 172.21.15.118:0/2262759657 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56289ec9d810 0x56289ec9dc30 secure :-1 s=READY pgs=344 cs=0 l=1 rev1=1 crypto rx=0x7f51140117e0 tx=0x7f5114011ca0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=1eaace0444cc3be9 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:18:36.053 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:36.051+0000 7f511a000640 1 --2- 172.21.15.118:0/3871521605 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7f51000a2880 0x7f51000a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:18:36.053 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:36.051+0000 7f5111c00640 1 -- 172.21.15.118:0/2262759657 <== mon.2 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5114011310 con 0x56289ec9d810 2024-03-24T00:18:36.053 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:36.051+0000 7f511a000640 1 -- 172.21.15.118:0/3871521605 <== mon.1 v2:172.21.15.193:3300/0 5 ==== osd_map(90..90 src has 1..90) v4 ==== 8277+0+0 (secure 0 0 0) 0x7f51040f09e0 con 0x7f511c1028a0 2024-03-24T00:18:36.053 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:36.051+0000 7f5111c00640 1 -- 172.21.15.118:0/2262759657 <== mon.2 v2:172.21.15.189:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f5114003e50 con 0x56289ec9d810 2024-03-24T00:18:36.053 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:36.051+0000 7f5111c00640 1 -- 172.21.15.118:0/2262759657 <== mon.2 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5114004620 con 0x56289ec9d810 2024-03-24T00:18:36.053 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:36.051+0000 7f5120c00640 1 --2- 172.21.15.118:0/3871521605 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7f51000a2880 0x7f51000a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:18:36.053 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:36.051+0000 7f51263ae740 1 -- 172.21.15.118:0/2262759657 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56289ec9d810 msgr2=0x56289ec9dc30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:18:36.053 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:36.051+0000 7f51263ae740 1 --2- 172.21.15.118:0/2262759657 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56289ec9d810 0x56289ec9dc30 secure :-1 s=READY pgs=344 cs=0 l=1 rev1=1 crypto rx=0x7f51140117e0 tx=0x7f5114011ca0 comp rx=0 tx=0).stop 2024-03-24T00:18:36.053 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:36.052+0000 7f51263ae740 1 -- 172.21.15.118:0/2262759657 shutdown_connections 2024-03-24T00:18:36.053 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:36.052+0000 7f51263ae740 1 --2- 172.21.15.118:0/2262759657 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x56289ec9f3a0 0x56289eca5c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:18:36.053 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:36.052+0000 7f51263ae740 1 --2- 172.21.15.118:0/2262759657 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56289ec9e9e0 0x56289ec9ee60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:18:36.053 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:36.052+0000 7f51263ae740 1 --2- 172.21.15.118:0/2262759657 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56289ec9d810 0x56289ec9dc30 unknown :-1 s=CLOSED pgs=344 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:18:36.053 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:36.052+0000 7f51263ae740 1 -- 172.21.15.118:0/2262759657 >> 172.21.15.118:0/2262759657 conn(0x56289ec98c00 msgr2=0x56289ec99000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:18:36.054 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:36.052+0000 7f51263ae740 1 -- 172.21.15.118:0/2262759657 shutdown_connections 2024-03-24T00:18:36.054 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:36.052+0000 7f51263ae740 1 -- 172.21.15.118:0/2262759657 wait complete. 2024-03-24T00:18:36.054 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:36.052+0000 7f5120c00640 1 --2- 172.21.15.118:0/3871521605 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7f51000a2880 0x7f51000a4d40 secure :-1 s=READY pgs=323 cs=0 l=1 rev1=1 crypto rx=0x7f510c0058c0 tx=0x7f510c005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:18:36.054 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:36.052+0000 7f51263ae740 1 -- start start 2024-03-24T00:18:36.054 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:36.053+0000 7f51263ae740 1 Processor -- start 2024-03-24T00:18:36.054 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:36.053+0000 7f51263ae740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56289ec9d810 0x56289ecb02c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:18:36.054 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:36.053+0000 7f51263ae740 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x56289ec9e9e0 0x56289ecb0800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:18:36.055 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:36.053+0000 7f51263ae740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56289ec9f3a0 0x56289ecb7840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:18:36.055 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:36.053+0000 7f51263ae740 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x56289eca63b0 con 0x56289ec9e9e0 2024-03-24T00:18:36.055 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:36.053+0000 7f51263ae740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x56289ebbe0d0 con 0x56289ec9d810 2024-03-24T00:18:36.055 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:36.053+0000 7f51263ae740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x56289eca6530 con 0x56289ec9f3a0 2024-03-24T00:18:36.055 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:36.053+0000 7f51263ae740 10 client.0 _ll_register_callbacks cb 0x56289ebc2e80 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T00:18:36.055 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:36.053+0000 7f5120c00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56289ec9d810 0x56289ecb02c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:18:36.055 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:36.053+0000 7f5120c00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56289ec9d810 0x56289ecb02c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.118:32872/0 (socket says 172.21.15.118:32872) 2024-03-24T00:18:36.055 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:36.053+0000 7f5120c00640 1 -- 172.21.15.118:0/2981073715 learned_addr learned my addr 172.21.15.118:0/2981073715 (peer_addr_for_me v2:172.21.15.118:0/0) 2024-03-24T00:18:36.055 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:36.053+0000 7f5121600640 1 --2- 172.21.15.118:0/2981073715 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56289ec9f3a0 0x56289ecb7840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:18:36.056 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:36.054+0000 7f5121600640 1 -- 172.21.15.118:0/2981073715 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56289ec9d810 msgr2=0x56289ecb02c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:18:36.056 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:36.054+0000 7f5121600640 1 --2- 172.21.15.118:0/2981073715 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56289ec9d810 0x56289ecb02c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:18:36.057 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:36.054+0000 7f5121600640 1 -- 172.21.15.118:0/2981073715 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x56289ec9e9e0 msgr2=0x56289ecb0800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:18:36.057 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:36.054+0000 7f5113e00640 1 --2- 172.21.15.118:0/2981073715 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x56289ec9e9e0 0x56289ecb0800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:18:36.057 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:36.054+0000 7f5121600640 1 --2- 172.21.15.118:0/2981073715 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x56289ec9e9e0 0x56289ecb0800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:18:36.057 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:36.054+0000 7f5121600640 1 -- 172.21.15.118:0/2981073715 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x56289e9163f0 con 0x56289ec9f3a0 2024-03-24T00:18:36.057 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:36.054+0000 7f5113e00640 1 --2- 172.21.15.118:0/2981073715 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x56289ec9e9e0 0x56289ecb0800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-24T00:18:36.057 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:36.054+0000 7f5121600640 1 --2- 172.21.15.118:0/2981073715 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56289ec9f3a0 0x56289ecb7840 secure :-1 s=READY pgs=344 cs=0 l=1 rev1=1 crypto rx=0x7f511400f790 tx=0x7f5114011340 comp rx=0 tx=0).ready entity=mon.1 client_cookie=3c0138597830d738 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:18:36.057 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:36.054+0000 7f510a000640 10 client.0 ms_handle_connect on v2:172.21.15.193:3300/0 2024-03-24T00:18:36.057 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:36.055+0000 7f510a000640 1 -- 172.21.15.118:0/2981073715 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f511402a060 con 0x56289ec9f3a0 2024-03-24T00:18:36.057 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:36.055+0000 7f510a000640 1 -- 172.21.15.118:0/2981073715 <== mon.1 v2:172.21.15.193:3300/0 2 ==== osd_map(90..90 src has 1..90) v4 ==== 8277+0+0 (secure 0 0 0) 0x7f5114004a40 con 0x56289ec9f3a0 2024-03-24T00:18:36.057 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:36.055+0000 7f51263ae740 1 -- 172.21.15.118:0/2981073715 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f5114003e50 con 0x56289ec9f3a0 2024-03-24T00:18:36.057 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:36.055+0000 7f510a000640 1 -- 172.21.15.118:0/2981073715 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) v2 ==== 24+0+0 (secure 0 0 0) 0x7f5114004c40 con 0x56289ec9f3a0 2024-03-24T00:18:36.057 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:36.055+0000 7f51263ae740 10 client.26257 fetch_fsmap learned FSMap version 28 2024-03-24T00:18:36.057 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:36.055+0000 7f51263ae740 1 -- 172.21.15.118:0/2981073715 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) v3 -- 0x56289e9e07d0 con 0x56289ec9f3a0 2024-03-24T00:18:36.057 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:36.055+0000 7f510a000640 1 -- 172.21.15.118:0/2981073715 <== mon.1 v2:172.21.15.193:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f5114003e50 con 0x56289ec9f3a0 2024-03-24T00:18:36.058 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:36.055+0000 7f510a000640 1 -- 172.21.15.118:0/2981073715 <== mon.1 v2:172.21.15.193:3300/0 5 ==== fsmap(e 28) v1 ==== 7934+0+0 (secure 0 0 0) 0x7f5114018020 con 0x56289ec9f3a0 2024-03-24T00:18:36.058 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:36.055+0000 7f51263ae740 10 client.26257 fetch_fsmap finished waiting for FSMap version 28 2024-03-24T00:18:36.058 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:36.055+0000 7f51263ae740 10 client.26257 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.c 2024-03-24T00:18:36.058 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:36.055+0000 7f51263ae740 20 client.26257 populate_metadata read hostname 'smithi118' 2024-03-24T00:18:36.058 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:36.055+0000 7f51263ae740 1 --2- 172.21.15.118:0/2981073715 >> [v2:172.21.15.193:6836/2921520053,v1:172.21.15.193:6837/2921520053] conn(0x56289ecaa750 0x56289ecacb60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:18:36.058 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:36.055+0000 7f51263ae740 4 client.26257 mds_command: new command op to 24511 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-24T00:18:36.058 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:36.055+0000 7f51263ae740 1 -- 172.21.15.118:0/2981073715 --> [v2:172.21.15.193:6836/2921520053,v1:172.21.15.193:6837/2921520053] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x56289ec0f800 con 0x56289ecaa750 2024-03-24T00:18:36.058 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:36.056+0000 7f5120c00640 1 --2- 172.21.15.118:0/2981073715 >> [v2:172.21.15.193:6836/2921520053,v1:172.21.15.193:6837/2921520053] conn(0x56289ecaa750 0x56289ecacb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:18:36.058 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:36.056+0000 7f5120c00640 1 --2- 172.21.15.118:0/2981073715 >> [v2:172.21.15.193:6836/2921520053,v1:172.21.15.193:6837/2921520053] conn(0x56289ecaa750 0x56289ecacb60 crc :-1 s=READY pgs=27 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=3cdc2681bf58694e server_cookie=5b2bcafdc38a0b8a in_seq=0 out_seq=0 2024-03-24T00:18:36.058 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:36.056+0000 7f510a000640 10 client.26257 ms_handle_connect on v2:172.21.15.193:6836/2921520053 2024-03-24T00:18:36.058 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:36.057+0000 7f510a000640 1 -- 172.21.15.118:0/2981073715 <== mds.2 v2:172.21.15.193:6836/2921520053 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x56289ec0f800 con 0x56289ecaa750 2024-03-24T00:18:36.058 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:36.057+0000 7f510a000640 10 client.26257 handle_command_reply: tid=0 2024-03-24T00:18:36.058 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:36.057+0000 7f51263ae740 1 client.26257 shutdown 2024-03-24T00:18:36.059 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:36.058+0000 7f51263ae740 1 -- 172.21.15.118:0/2981073715 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56289ec9f3a0 msgr2=0x56289ecb7840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:18:36.059 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:36.058+0000 7f51263ae740 1 --2- 172.21.15.118:0/2981073715 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56289ec9f3a0 0x56289ecb7840 secure :-1 s=READY pgs=344 cs=0 l=1 rev1=1 crypto rx=0x7f511400f790 tx=0x7f5114011340 comp rx=0 tx=0).stop 2024-03-24T00:18:36.060 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:36.058+0000 7f51263ae740 1 -- 172.21.15.118:0/2981073715 shutdown_connections 2024-03-24T00:18:36.060 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:36.058+0000 7f51263ae740 1 --2- 172.21.15.118:0/2981073715 >> [v2:172.21.15.193:6836/2921520053,v1:172.21.15.193:6837/2921520053] conn(0x56289ecaa750 0x56289ecacb60 crc :-1 s=READY pgs=27 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:18:36.060 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:36.058+0000 7f51263ae740 1 --2- 172.21.15.118:0/2981073715 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56289ec9f3a0 0x56289ecb7840 unknown :-1 s=CLOSED pgs=344 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:18:36.060 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:36.058+0000 7f51263ae740 1 --2- 172.21.15.118:0/2981073715 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x56289ec9e9e0 0x56289ecb0800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:18:36.060 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:36.058+0000 7f51263ae740 1 --2- 172.21.15.118:0/2981073715 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56289ec9d810 0x56289ecb02c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:18:36.060 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:36.058+0000 7f51263ae740 1 -- 172.21.15.118:0/2981073715 >> 172.21.15.118:0/2981073715 conn(0x56289ec98c00 msgr2=0x56289ec97eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:18:36.060 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:36.058+0000 7f510a000640 0 client.26257 ms_handle_reset on v2:172.21.15.193:6836/2921520053 2024-03-24T00:18:36.060 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:36.058+0000 7f51263ae740 1 -- 172.21.15.118:0/2981073715 shutdown_connections 2024-03-24T00:18:36.060 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:36.058+0000 7f51263ae740 1 -- 172.21.15.118:0/2981073715 wait complete. 2024-03-24T00:18:36.060 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:36.058+0000 7f51263ae740 20 client.26257 trim_cache size 0 max 16384 2024-03-24T00:18:36.073 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:36.073+0000 7f51263ae740 1 Processor -- start 2024-03-24T00:18:36.073 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:36.073+0000 7f51263ae740 1 -- start start 2024-03-24T00:18:36.073 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:36.074+0000 7f51263ae740 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x56289ec9d810 0x56289ec9c570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:18:36.073 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:36.074+0000 7f51263ae740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56289ec9e9e0 0x56289ecb7800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:18:36.073 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:36.074+0000 7f51263ae740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56289ec9f3a0 0x56289eca1a80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:18:36.073 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:36.074+0000 7f51263ae740 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f511c1a67b0 con 0x56289ec9d810 2024-03-24T00:18:36.074 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:36.074+0000 7f51263ae740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f511c1a8130 con 0x56289ec9e9e0 2024-03-24T00:18:36.074 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:36.074+0000 7f51263ae740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f511c1a6e10 con 0x56289ec9f3a0 2024-03-24T00:18:36.075 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:36.074+0000 7f5121600640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56289ec9f3a0 0x56289eca1a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:18:36.075 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:36.074+0000 7f5113e00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56289ec9e9e0 0x56289ecb7800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:18:36.075 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:36.074+0000 7f5121600640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56289ec9f3a0 0x56289eca1a80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.118:53048/0 (socket says 172.21.15.118:53048) 2024-03-24T00:18:36.075 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:36.074+0000 7f5121600640 1 -- 172.21.15.118:0/3908494342 learned_addr learned my addr 172.21.15.118:0/3908494342 (peer_addr_for_me v2:172.21.15.118:0/0) 2024-03-24T00:18:36.075 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:36.074+0000 7f5120c00640 1 --2- 172.21.15.118:0/3908494342 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x56289ec9d810 0x56289ec9c570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:18:36.075 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:36.074+0000 7f5121600640 1 -- 172.21.15.118:0/3908494342 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56289ec9e9e0 msgr2=0x56289ecb7800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:18:36.075 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:36.074+0000 7f5121600640 1 --2- 172.21.15.118:0/3908494342 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56289ec9e9e0 0x56289ecb7800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:18:36.075 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:36.074+0000 7f5121600640 1 -- 172.21.15.118:0/3908494342 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x56289ec9d810 msgr2=0x56289ec9c570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:18:36.075 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:36.074+0000 7f5121600640 1 --2- 172.21.15.118:0/3908494342 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x56289ec9d810 0x56289ec9c570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:18:36.075 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:36.074+0000 7f5121600640 1 -- 172.21.15.118:0/3908494342 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x56289ecb0b50 con 0x56289ec9f3a0 2024-03-24T00:18:36.076 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:36.074+0000 7f5120c00640 1 --2- 172.21.15.118:0/3908494342 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x56289ec9d810 0x56289ec9c570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T00:18:36.076 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:36.075+0000 7f5121600640 1 --2- 172.21.15.118:0/3908494342 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56289ec9f3a0 0x56289eca1a80 secure :-1 s=READY pgs=345 cs=0 l=1 rev1=1 crypto rx=0x7f5114009350 tx=0x7f5114011430 comp rx=0 tx=0).ready entity=mon.1 client_cookie=303392a4d2f38770 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:18:36.076 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:36.075+0000 7f5111200640 1 -- 172.21.15.118:0/3908494342 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f511402a070 con 0x56289ec9f3a0 2024-03-24T00:18:36.076 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:36.075+0000 7f5111200640 1 -- 172.21.15.118:0/3908494342 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f511400fbb0 con 0x56289ec9f3a0 2024-03-24T00:18:36.076 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:36.075+0000 7f5111200640 1 -- 172.21.15.118:0/3908494342 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f511401a920 con 0x56289ec9f3a0 2024-03-24T00:18:36.079 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:36.076+0000 7f51263ae740 1 -- 172.21.15.118:0/3908494342 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56289ec9f3a0 msgr2=0x56289eca1a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:18:36.079 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:36.076+0000 7f51263ae740 1 --2- 172.21.15.118:0/3908494342 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56289ec9f3a0 0x56289eca1a80 secure :-1 s=READY pgs=345 cs=0 l=1 rev1=1 crypto rx=0x7f5114009350 tx=0x7f5114011430 comp rx=0 tx=0).stop 2024-03-24T00:18:36.079 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:36.076+0000 7f51263ae740 1 -- 172.21.15.118:0/3908494342 shutdown_connections 2024-03-24T00:18:36.079 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:36.076+0000 7f51263ae740 1 --2- 172.21.15.118:0/3908494342 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56289ec9f3a0 0x56289eca1a80 unknown :-1 s=CLOSED pgs=345 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:18:36.079 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:36.076+0000 7f51263ae740 1 --2- 172.21.15.118:0/3908494342 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56289ec9e9e0 0x56289ecb7800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:18:36.079 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:36.076+0000 7f51263ae740 1 --2- 172.21.15.118:0/3908494342 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x56289ec9d810 0x56289ec9c570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:18:36.079 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:36.076+0000 7f51263ae740 1 -- 172.21.15.118:0/3908494342 >> 172.21.15.118:0/3908494342 conn(0x56289ec98c00 msgr2=0x56289ec9be60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:18:36.079 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:36.076+0000 7f51263ae740 1 -- 172.21.15.118:0/3908494342 shutdown_connections 2024-03-24T00:18:36.079 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:36.076+0000 7f51263ae740 1 -- 172.21.15.118:0/3908494342 wait complete. 2024-03-24T00:18:36.079 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:36.076+0000 7f51263ae740 1 -- start start 2024-03-24T00:18:36.079 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:36.077+0000 7f51263ae740 1 Processor -- start 2024-03-24T00:18:36.079 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:36.077+0000 7f51263ae740 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x56289ec9d810 0x56289ecb34b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:18:36.079 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:36.077+0000 7f51263ae740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56289ec9e9e0 0x56289ecb39f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:18:36.080 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:36.077+0000 7f51263ae740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56289ec9f3a0 0x56289ecb6a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:18:36.080 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:36.077+0000 7f51263ae740 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f511c1a7ad0 con 0x56289ec9d810 2024-03-24T00:18:36.080 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:36.077+0000 7f51263ae740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f511c1a7470 con 0x56289ec9e9e0 2024-03-24T00:18:36.080 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:36.077+0000 7f51263ae740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f511c1a4970 con 0x56289ec9f3a0 2024-03-24T00:18:36.080 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:36.077+0000 7f51263ae740 10 client.0 _ll_register_callbacks cb 0x7f510c009550 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T00:18:36.080 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:36.077+0000 7f5113e00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56289ec9e9e0 0x56289ecb39f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:18:36.080 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:36.077+0000 7f5113e00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56289ec9e9e0 0x56289ecb39f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.118:32882/0 (socket says 172.21.15.118:32882) 2024-03-24T00:18:36.080 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:36.077+0000 7f5113e00640 1 -- 172.21.15.118:0/1110486405 learned_addr learned my addr 172.21.15.118:0/1110486405 (peer_addr_for_me v2:172.21.15.118:0/0) 2024-03-24T00:18:36.080 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:36.077+0000 7f5121600640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56289ec9f3a0 0x56289ecb6a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:18:36.080 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:36.077+0000 7f5120c00640 1 --2- 172.21.15.118:0/1110486405 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x56289ec9d810 0x56289ecb34b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:18:36.080 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:36.078+0000 7f5113e00640 1 -- 172.21.15.118:0/1110486405 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56289ec9f3a0 msgr2=0x56289ecb6a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:18:36.080 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:36.078+0000 7f5113e00640 1 --2- 172.21.15.118:0/1110486405 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56289ec9f3a0 0x56289ecb6a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:18:36.080 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:36.078+0000 7f5113e00640 1 -- 172.21.15.118:0/1110486405 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x56289ec9d810 msgr2=0x56289ecb34b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:18:36.080 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:36.078+0000 7f5113e00640 1 --2- 172.21.15.118:0/1110486405 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x56289ec9d810 0x56289ecb34b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:18:36.080 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:36.078+0000 7f5113e00640 1 -- 172.21.15.118:0/1110486405 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x56289ecb7000 con 0x56289ec9e9e0 2024-03-24T00:18:36.081 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:36.078+0000 7f5113e00640 1 --2- 172.21.15.118:0/1110486405 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56289ec9e9e0 0x56289ecb39f0 secure :-1 s=READY pgs=345 cs=0 l=1 rev1=1 crypto rx=0x7f51040b9cc0 tx=0x7f5104000bc0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=23595c1e71f8a23a server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:18:36.081 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:36.078+0000 7f510a000640 10 client.0 ms_handle_connect on v2:172.21.15.189:3300/0 2024-03-24T00:18:36.081 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:36.078+0000 7f510a000640 1 -- 172.21.15.118:0/1110486405 <== mon.2 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f51040b7e60 con 0x56289ec9e9e0 2024-03-24T00:18:36.081 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:36.078+0000 7f510a000640 1 -- 172.21.15.118:0/1110486405 <== mon.2 v2:172.21.15.189:3300/0 2 ==== osd_map(90..90 src has 1..90) v4 ==== 8277+0+0 (secure 0 0 0) 0x7f5104004410 con 0x56289ec9e9e0 2024-03-24T00:18:36.081 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:36.078+0000 7f51263ae740 1 -- 172.21.15.118:0/1110486405 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f511400fbb0 con 0x56289ec9e9e0 2024-03-24T00:18:36.081 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:36.078+0000 7f510a000640 1 -- 172.21.15.118:0/1110486405 <== mon.2 v2:172.21.15.189:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) v2 ==== 24+0+0 (secure 0 0 0) 0x7f51040bcc50 con 0x56289ec9e9e0 2024-03-24T00:18:36.081 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:36.078+0000 7f51263ae740 10 client.26252 fetch_fsmap learned FSMap version 28 2024-03-24T00:18:36.081 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:36.078+0000 7f51263ae740 1 -- 172.21.15.118:0/1110486405 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) v3 -- 0x56289ed1ffa0 con 0x56289ec9e9e0 2024-03-24T00:18:36.081 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:36.079+0000 7f510a000640 1 -- 172.21.15.118:0/1110486405 <== mon.2 v2:172.21.15.189:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f511400fbb0 con 0x56289ec9e9e0 2024-03-24T00:18:36.081 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:36.079+0000 7f510a000640 1 -- 172.21.15.118:0/1110486405 <== mon.2 v2:172.21.15.189:3300/0 5 ==== fsmap(e 28) v1 ==== 7934+0+0 (secure 0 0 0) 0x7f51040dd0b0 con 0x56289ec9e9e0 2024-03-24T00:18:36.081 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:36.079+0000 7f51263ae740 10 client.26252 fetch_fsmap finished waiting for FSMap version 28 2024-03-24T00:18:36.081 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:36.079+0000 7f51263ae740 10 client.26252 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.c 2024-03-24T00:18:36.081 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:36.079+0000 7f51263ae740 20 client.26252 populate_metadata read hostname 'smithi118' 2024-03-24T00:18:36.081 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:36.079+0000 7f51263ae740 1 --2- 172.21.15.118:0/1110486405 >> [v2:172.21.15.193:6836/2921520053,v1:172.21.15.193:6837/2921520053] conn(0x56289ed20940 0x56289ed22d60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:18:36.081 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:36.079+0000 7f51263ae740 4 client.26252 mds_command: new command op to 24511 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-24T00:18:36.082 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:36.079+0000 7f51263ae740 1 -- 172.21.15.118:0/1110486405 --> [v2:172.21.15.193:6836/2921520053,v1:172.21.15.193:6837/2921520053] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x56289ec09b50 con 0x56289ed20940 2024-03-24T00:18:36.082 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:36.079+0000 7f5120c00640 1 --2- 172.21.15.118:0/1110486405 >> [v2:172.21.15.193:6836/2921520053,v1:172.21.15.193:6837/2921520053] conn(0x56289ed20940 0x56289ed22d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:18:36.083 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:36.080+0000 7f5120c00640 1 --2- 172.21.15.118:0/1110486405 >> [v2:172.21.15.193:6836/2921520053,v1:172.21.15.193:6837/2921520053] conn(0x56289ed20940 0x56289ed22d60 crc :-1 s=READY pgs=28 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=4d2f0f19e4560149 server_cookie=b2d74af2dd98c7e1 in_seq=0 out_seq=0 2024-03-24T00:18:36.083 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:36.080+0000 7f510a000640 10 client.26252 ms_handle_connect on v2:172.21.15.193:6836/2921520053 2024-03-24T00:18:36.083 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:36.080+0000 7f510a000640 1 -- 172.21.15.118:0/1110486405 <== mds.2 v2:172.21.15.193:6836/2921520053 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x56289ec09b50 con 0x56289ed20940 2024-03-24T00:18:36.083 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:36.080+0000 7f510a000640 10 client.26252 handle_command_reply: tid=0 2024-03-24T00:18:36.083 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:36.080+0000 7f51263ae740 1 client.26252 shutdown 2024-03-24T00:18:36.083 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:36.080+0000 7f51263ae740 1 -- 172.21.15.118:0/1110486405 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56289ec9e9e0 msgr2=0x56289ecb39f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:18:36.083 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:36.080+0000 7f51263ae740 1 --2- 172.21.15.118:0/1110486405 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56289ec9e9e0 0x56289ecb39f0 secure :-1 s=READY pgs=345 cs=0 l=1 rev1=1 crypto rx=0x7f51040b9cc0 tx=0x7f5104000bc0 comp rx=0 tx=0).stop 2024-03-24T00:18:36.083 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:36.080+0000 7f51263ae740 1 -- 172.21.15.118:0/1110486405 shutdown_connections 2024-03-24T00:18:36.083 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:36.080+0000 7f51263ae740 1 --2- 172.21.15.118:0/1110486405 >> [v2:172.21.15.193:6836/2921520053,v1:172.21.15.193:6837/2921520053] conn(0x56289ed20940 0x56289ed22d60 crc :-1 s=READY pgs=28 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:18:36.083 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:36.080+0000 7f51263ae740 1 --2- 172.21.15.118:0/1110486405 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56289ec9f3a0 0x56289ecb6a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:18:36.083 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:36.080+0000 7f51263ae740 1 --2- 172.21.15.118:0/1110486405 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56289ec9e9e0 0x56289ecb39f0 unknown :-1 s=CLOSED pgs=345 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:18:36.084 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:36.080+0000 7f51263ae740 1 --2- 172.21.15.118:0/1110486405 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x56289ec9d810 0x56289ecb34b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:18:36.084 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:36.080+0000 7f510a000640 0 client.26252 ms_handle_reset on v2:172.21.15.193:6836/2921520053 2024-03-24T00:18:36.084 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:36.080+0000 7f51263ae740 1 -- 172.21.15.118:0/1110486405 >> 172.21.15.118:0/1110486405 conn(0x56289ec98c00 msgr2=0x56289ec9c6f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:18:36.084 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:36.080+0000 7f51263ae740 1 -- 172.21.15.118:0/1110486405 shutdown_connections 2024-03-24T00:18:36.084 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:36.080+0000 7f51263ae740 1 -- 172.21.15.118:0/1110486405 wait complete. 2024-03-24T00:18:36.084 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:36.080+0000 7f51263ae740 20 client.26252 trim_cache size 0 max 16384 2024-03-24T00:18:36.084 INFO:teuthology.orchestra.run.smithi118.stdout:[] 2024-03-24T00:18:36.085 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:36.081+0000 7f5123600640 1 -- 172.21.15.118:0/3871521605 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7f51000a2880 msgr2=0x7f51000a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:18:36.085 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:36.081+0000 7f5123600640 1 --2- 172.21.15.118:0/3871521605 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7f51000a2880 0x7f51000a4d40 secure :-1 s=READY pgs=323 cs=0 l=1 rev1=1 crypto rx=0x7f510c0058c0 tx=0x7f510c005f90 comp rx=0 tx=0).stop 2024-03-24T00:18:36.085 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:36.081+0000 7f5123600640 1 -- 172.21.15.118:0/3871521605 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f511c1028a0 msgr2=0x7f511c19a110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:18:36.085 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:36.081+0000 7f5123600640 1 --2- 172.21.15.118:0/3871521605 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f511c1028a0 0x7f511c19a110 secure :-1 s=READY pgs=343 cs=0 l=1 rev1=1 crypto rx=0x7f5104036040 tx=0x7f510402c650 comp rx=0 tx=0).stop 2024-03-24T00:18:36.085 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:36.081+0000 7f5123600640 1 -- 172.21.15.118:0/3871521605 shutdown_connections 2024-03-24T00:18:36.085 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:36.081+0000 7f5123600640 1 --2- 172.21.15.118:0/3871521605 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7f51000a2880 0x7f51000a4d40 unknown :-1 s=CLOSED pgs=323 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:18:36.085 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:36.081+0000 7f5123600640 1 --2- 172.21.15.118:0/3871521605 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f511c103260 0x7f511c1a1180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:18:36.085 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:36.081+0000 7f5123600640 1 --2- 172.21.15.118:0/3871521605 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f511c1028a0 0x7f511c19a110 unknown :-1 s=CLOSED pgs=343 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:18:36.085 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:36.081+0000 7f5123600640 1 --2- 172.21.15.118:0/3871521605 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f511c1016b0 0x7f511c199bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:18:36.085 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:36.081+0000 7f5123600640 1 -- 172.21.15.118:0/3871521605 >> 172.21.15.118:0/3871521605 conn(0x7f511c0fcc40 msgr2=0x7f511c0fd7e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:18:36.085 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:36.081+0000 7f5123600640 1 -- 172.21.15.118:0/3871521605 shutdown_connections 2024-03-24T00:18:36.086 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:36.081+0000 7f5123600640 1 -- 172.21.15.118:0/3871521605 wait complete. 2024-03-24T00:18:36.098 INFO:teuthology.orchestra.run:Running command with timeout 120 2024-03-24T00:18:36.099 DEBUG:teuthology.orchestra.run.smithi118:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:1 damage ls 2024-03-24T00:18:36.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:18:35 smithi189 ceph-mon[36187]: from='client.? 172.21.15.118:0/1085924255' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-24T00:18:36.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:18:35 smithi193 ceph-mon[36878]: from='client.? 172.21.15.118:0/1085924255' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-24T00:18:36.301 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:36.301+0000 7f3353200640 1 -- 172.21.15.118:0/1176016414 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f334c101100 msgr2=0x7f334c101580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:18:36.301 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:36.301+0000 7f3353200640 1 --2- 172.21.15.118:0/1176016414 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f334c101100 0x7f334c101580 secure :-1 s=READY pgs=346 cs=0 l=1 rev1=1 crypto rx=0x7f3348009be0 tx=0x7f334802bef0 comp rx=0 tx=0).stop 2024-03-24T00:18:36.302 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:36.301+0000 7f3353200640 1 -- 172.21.15.118:0/1176016414 shutdown_connections 2024-03-24T00:18:36.302 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:36.301+0000 7f3353200640 1 --2- 172.21.15.118:0/1176016414 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f334c101100 0x7f334c101580 unknown :-1 s=CLOSED pgs=346 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:18:36.302 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:36.301+0000 7f3353200640 1 --2- 172.21.15.118:0/1176016414 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f334c100760 0x7f334c100bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:18:36.302 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:36.301+0000 7f3353200640 1 --2- 172.21.15.118:0/1176016414 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f334c0ffe00 0x7f334c100220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:18:36.302 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:36.301+0000 7f3353200640 1 -- 172.21.15.118:0/1176016414 >> 172.21.15.118:0/1176016414 conn(0x7f334c0fba20 msgr2=0x7f334c0fde40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:18:36.302 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:36.301+0000 7f3353200640 1 -- 172.21.15.118:0/1176016414 shutdown_connections 2024-03-24T00:18:36.302 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:36.302+0000 7f3353200640 1 -- 172.21.15.118:0/1176016414 wait complete. 2024-03-24T00:18:36.303 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:36.303+0000 7f3353200640 1 Processor -- start 2024-03-24T00:18:36.303 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:36.303+0000 7f3353200640 1 -- start start 2024-03-24T00:18:36.304 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:36.303+0000 7f3353200640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f334c0ffe00 0x7f334c199a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:18:36.304 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:36.303+0000 7f3353200640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f334c100760 0x7f334c199f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:18:36.304 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:36.303+0000 7f3353200640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f334c101100 0x7f334c1a0fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:18:36.304 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:36.303+0000 7f3353200640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f334c10f430 con 0x7f334c101100 2024-03-24T00:18:36.304 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:36.303+0000 7f3353200640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f334c10f2b0 con 0x7f334c100760 2024-03-24T00:18:36.304 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:36.303+0000 7f3353200640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f334c10f5b0 con 0x7f334c0ffe00 2024-03-24T00:18:36.304 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:36.304+0000 7f3351e00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f334c0ffe00 0x7f334c199a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:18:36.304 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:36.304+0000 7f3351e00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f334c0ffe00 0x7f334c199a10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.118:53078/0 (socket says 172.21.15.118:53078) 2024-03-24T00:18:36.304 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:36.304+0000 7f3351e00640 1 -- 172.21.15.118:0/1827646816 learned_addr learned my addr 172.21.15.118:0/1827646816 (peer_addr_for_me v2:172.21.15.118:0/0) 2024-03-24T00:18:36.304 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:36.304+0000 7f3351400640 1 --2- 172.21.15.118:0/1827646816 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f334c100760 0x7f334c199f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:18:36.304 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:36.304+0000 7f3352800640 1 --2- 172.21.15.118:0/1827646816 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f334c101100 0x7f334c1a0fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:18:36.305 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:36.304+0000 7f3351400640 1 -- 172.21.15.118:0/1827646816 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f334c0ffe00 msgr2=0x7f334c199a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:18:36.305 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:36.304+0000 7f3351400640 1 --2- 172.21.15.118:0/1827646816 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f334c0ffe00 0x7f334c199a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:18:36.306 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:36.304+0000 7f3351400640 1 -- 172.21.15.118:0/1827646816 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f334c101100 msgr2=0x7f334c1a0fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:18:36.306 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:36.304+0000 7f3351400640 1 --2- 172.21.15.118:0/1827646816 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f334c101100 0x7f334c1a0fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:18:36.306 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:36.304+0000 7f3351400640 1 -- 172.21.15.118:0/1827646816 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f334c1a1630 con 0x7f334c100760 2024-03-24T00:18:36.306 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:36.305+0000 7f3351400640 1 --2- 172.21.15.118:0/1827646816 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f334c100760 0x7f334c199f50 secure :-1 s=READY pgs=346 cs=0 l=1 rev1=1 crypto rx=0x7f333800e9f0 tx=0x7f333800eeb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:18:36.306 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:36.305+0000 7f3346a00640 1 -- 172.21.15.118:0/1827646816 <== mon.2 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f333800ccc0 con 0x7f334c100760 2024-03-24T00:18:36.306 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:36.305+0000 7f3346a00640 1 -- 172.21.15.118:0/1827646816 <== mon.2 v2:172.21.15.189:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f3338010910 con 0x7f334c100760 2024-03-24T00:18:36.306 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:36.305+0000 7f3346a00640 1 -- 172.21.15.118:0/1827646816 <== mon.2 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3338018960 con 0x7f334c100760 2024-03-24T00:18:36.306 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:36.305+0000 7f3353200640 1 -- 172.21.15.118:0/1827646816 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f334c1a1920 con 0x7f334c100760 2024-03-24T00:18:36.306 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:36.305+0000 7f3353200640 1 -- 172.21.15.118:0/1827646816 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f334c1a1e20 con 0x7f334c100760 2024-03-24T00:18:36.309 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:36.309+0000 7f33575fc740 1 Processor -- start 2024-03-24T00:18:36.309 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:36.309+0000 7f33575fc740 1 -- start start 2024-03-24T00:18:36.309 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:36.309+0000 7f33575fc740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55de180989f0 0x55de18098e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:18:36.309 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:36.309+0000 7f33575fc740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55de18099bc0 0x55de1809a040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:18:36.310 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:36.309+0000 7f3346a00640 1 -- 172.21.15.118:0/1827646816 <== mon.2 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 20) v1 ==== 137291+0+0 (secure 0 0 0) 0x7f3338010ac0 con 0x7f334c100760 2024-03-24T00:18:36.310 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:36.309+0000 7f33575fc740 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55de1809a580 0x55de180a0e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:18:36.310 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:36.309+0000 7f33575fc740 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x55de17ffd2c0 con 0x55de1809a580 2024-03-24T00:18:36.310 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:36.309+0000 7f33575fc740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x55de17fa5c90 con 0x55de180989f0 2024-03-24T00:18:36.310 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:36.309+0000 7f33575fc740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x55de17ff9360 con 0x55de18099bc0 2024-03-24T00:18:36.310 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:36.309+0000 7f3351e00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55de18099bc0 0x55de1809a040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:18:36.310 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:36.309+0000 7f3352800640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55de180989f0 0x55de18098e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:18:36.310 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:36.309+0000 7f3351e00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55de18099bc0 0x55de1809a040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.118:53082/0 (socket says 172.21.15.118:53082) 2024-03-24T00:18:36.310 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:36.309+0000 7f3351e00640 1 -- 172.21.15.118:0/2601928691 learned_addr learned my addr 172.21.15.118:0/2601928691 (peer_addr_for_me v2:172.21.15.118:0/0) 2024-03-24T00:18:36.310 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:36.310+0000 7f3351400640 1 --2- 172.21.15.118:0/2601928691 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55de1809a580 0x55de180a0e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:18:36.310 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:36.310+0000 7f3351e00640 1 -- 172.21.15.118:0/2601928691 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55de180989f0 msgr2=0x55de18098e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:18:36.310 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:36.310+0000 7f3351e00640 1 --2- 172.21.15.118:0/2601928691 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55de180989f0 0x55de18098e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:18:36.310 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:36.310+0000 7f3351e00640 1 -- 172.21.15.118:0/2601928691 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55de1809a580 msgr2=0x55de180a0e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:18:36.310 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:36.310+0000 7f3351e00640 1 --2- 172.21.15.118:0/2601928691 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55de1809a580 0x55de180a0e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:18:36.311 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:36.310+0000 7f3351e00640 1 -- 172.21.15.118:0/2601928691 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55de17fca080 con 0x55de18099bc0 2024-03-24T00:18:36.313 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:36.310+0000 7f3351e00640 1 --2- 172.21.15.118:0/2601928691 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55de18099bc0 0x55de1809a040 secure :-1 s=READY pgs=347 cs=0 l=1 rev1=1 crypto rx=0x7f3340009d80 tx=0x7f3340011810 comp rx=0 tx=0).ready entity=mon.1 client_cookie=a1feda643d6f005 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:18:36.313 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:36.310+0000 7f333f000640 1 -- 172.21.15.118:0/2601928691 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f334001e070 con 0x55de18099bc0 2024-03-24T00:18:36.313 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:36.310+0000 7f333f000640 1 -- 172.21.15.118:0/2601928691 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f3340019b10 con 0x55de18099bc0 2024-03-24T00:18:36.313 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:36.310+0000 7f333f000640 1 -- 172.21.15.118:0/2601928691 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f334000eb80 con 0x55de18099bc0 2024-03-24T00:18:36.313 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:36.311+0000 7f3346a00640 1 --2- 172.21.15.118:0/1827646816 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7f33340a2800 0x7f33340a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:18:36.313 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:36.311+0000 7f3346a00640 1 -- 172.21.15.118:0/1827646816 <== mon.2 v2:172.21.15.189:3300/0 5 ==== osd_map(90..90 src has 1..90) v4 ==== 8277+0+0 (secure 0 0 0) 0x7f33380ceea0 con 0x7f334c100760 2024-03-24T00:18:36.314 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:36.311+0000 7f33575fc740 1 -- 172.21.15.118:0/2601928691 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55de18099bc0 msgr2=0x55de1809a040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:18:36.314 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:36.311+0000 7f33575fc740 1 --2- 172.21.15.118:0/2601928691 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55de18099bc0 0x55de1809a040 secure :-1 s=READY pgs=347 cs=0 l=1 rev1=1 crypto rx=0x7f3340009d80 tx=0x7f3340011810 comp rx=0 tx=0).stop 2024-03-24T00:18:36.314 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:36.311+0000 7f3352800640 1 --2- 172.21.15.118:0/1827646816 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7f33340a2800 0x7f33340a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:18:36.314 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:36.312+0000 7f33575fc740 1 -- 172.21.15.118:0/2601928691 shutdown_connections 2024-03-24T00:18:36.314 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:36.312+0000 7f33575fc740 1 --2- 172.21.15.118:0/2601928691 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55de1809a580 0x55de180a0e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:18:36.314 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:36.312+0000 7f33575fc740 1 --2- 172.21.15.118:0/2601928691 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55de18099bc0 0x55de1809a040 unknown :-1 s=CLOSED pgs=347 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:18:36.314 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:36.312+0000 7f33575fc740 1 --2- 172.21.15.118:0/2601928691 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55de180989f0 0x55de18098e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:18:36.314 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:36.312+0000 7f33575fc740 1 -- 172.21.15.118:0/2601928691 >> 172.21.15.118:0/2601928691 conn(0x55de18093de0 msgr2=0x55de180941e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:18:36.314 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:36.312+0000 7f33575fc740 1 -- 172.21.15.118:0/2601928691 shutdown_connections 2024-03-24T00:18:36.314 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:36.312+0000 7f33575fc740 1 -- 172.21.15.118:0/2601928691 wait complete. 2024-03-24T00:18:36.314 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:36.312+0000 7f3352800640 1 --2- 172.21.15.118:0/1827646816 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7f33340a2800 0x7f33340a4cc0 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 crypto rx=0x7f3348006010 tx=0x7f3348004a40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:18:36.314 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:36.312+0000 7f33575fc740 1 -- start start 2024-03-24T00:18:36.314 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:36.313+0000 7f33575fc740 1 Processor -- start 2024-03-24T00:18:36.315 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:36.313+0000 7f33575fc740 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55de180989f0 0x55de180a2c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:18:36.315 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:36.313+0000 7f33575fc740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55de18099bc0 0x55de180a3150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:18:36.315 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:36.313+0000 7f33575fc740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55de1809a580 0x55de180aa190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:18:36.315 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:36.313+0000 7f33575fc740 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x55de180a1590 con 0x55de180989f0 2024-03-24T00:18:36.315 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:36.313+0000 7f33575fc740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x55de17fb90d0 con 0x55de18099bc0 2024-03-24T00:18:36.315 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:36.313+0000 7f33575fc740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x55de180a1710 con 0x55de1809a580 2024-03-24T00:18:36.315 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:36.313+0000 7f33575fc740 10 client.0 _ll_register_callbacks cb 0x55de17fbde80 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T00:18:36.315 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:36.313+0000 7f3351400640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55de18099bc0 0x55de180a3150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:18:36.315 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:36.313+0000 7f3351e00640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55de180989f0 0x55de180a2c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:18:36.315 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:36.313+0000 7f3351400640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55de18099bc0 0x55de180a3150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.118:32922/0 (socket says 172.21.15.118:32922) 2024-03-24T00:18:36.315 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:36.313+0000 7f3351400640 1 -- 172.21.15.118:0/2253204216 learned_addr learned my addr 172.21.15.118:0/2253204216 (peer_addr_for_me v2:172.21.15.118:0/0) 2024-03-24T00:18:36.315 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:36.313+0000 7f3352800640 1 --2- 172.21.15.118:0/2253204216 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55de1809a580 0x55de180aa190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:18:36.315 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:36.314+0000 7f3351400640 1 -- 172.21.15.118:0/2253204216 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55de1809a580 msgr2=0x55de180aa190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:18:36.315 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:36.314+0000 7f3351400640 1 --2- 172.21.15.118:0/2253204216 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55de1809a580 0x55de180aa190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:18:36.315 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:36.314+0000 7f3351400640 1 -- 172.21.15.118:0/2253204216 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55de180989f0 msgr2=0x55de180a2c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:18:36.316 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:36.314+0000 7f3351400640 1 --2- 172.21.15.118:0/2253204216 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55de180989f0 0x55de180a2c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:18:36.316 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:36.314+0000 7f3351400640 1 -- 172.21.15.118:0/2253204216 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55de17d113f0 con 0x55de18099bc0 2024-03-24T00:18:36.318 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:36.314+0000 7f3351400640 1 --2- 172.21.15.118:0/2253204216 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55de18099bc0 0x55de180a3150 secure :-1 s=READY pgs=347 cs=0 l=1 rev1=1 crypto rx=0x7f33380a3a20 tx=0x7f333809b3a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=eae2c15aee813860 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:18:36.318 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:36.314+0000 7f3333400640 10 client.0 ms_handle_connect on v2:172.21.15.189:3300/0 2024-03-24T00:18:36.318 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:36.314+0000 7f3333400640 1 -- 172.21.15.118:0/2253204216 <== mon.2 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f333809f900 con 0x55de18099bc0 2024-03-24T00:18:36.318 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:36.314+0000 7f3333400640 1 -- 172.21.15.118:0/2253204216 <== mon.2 v2:172.21.15.189:3300/0 2 ==== osd_map(90..90 src has 1..90) v4 ==== 8277+0+0 (secure 0 0 0) 0x7f33380860e0 con 0x55de18099bc0 2024-03-24T00:18:36.318 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:36.314+0000 7f33575fc740 1 -- 172.21.15.118:0/2253204216 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f3340019b10 con 0x55de18099bc0 2024-03-24T00:18:36.318 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:36.315+0000 7f3333400640 1 -- 172.21.15.118:0/2253204216 <== mon.2 v2:172.21.15.189:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) v2 ==== 24+0+0 (secure 0 0 0) 0x7f33380a3f50 con 0x55de18099bc0 2024-03-24T00:18:36.318 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:36.315+0000 7f33575fc740 10 client.26264 fetch_fsmap learned FSMap version 28 2024-03-24T00:18:36.318 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:36.315+0000 7f33575fc740 1 -- 172.21.15.118:0/2253204216 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) v3 -- 0x55de17ddb7d0 con 0x55de18099bc0 2024-03-24T00:18:36.318 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:36.315+0000 7f3333400640 1 -- 172.21.15.118:0/2253204216 <== mon.2 v2:172.21.15.189:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f3340019b10 con 0x55de18099bc0 2024-03-24T00:18:36.318 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:36.315+0000 7f3333400640 1 -- 172.21.15.118:0/2253204216 <== mon.2 v2:172.21.15.189:3300/0 5 ==== fsmap(e 28) v1 ==== 7934+0+0 (secure 0 0 0) 0x7f3338018b90 con 0x55de18099bc0 2024-03-24T00:18:36.319 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:36.315+0000 7f33575fc740 10 client.26264 fetch_fsmap finished waiting for FSMap version 28 2024-03-24T00:18:36.319 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:36.315+0000 7f33575fc740 10 client.26264 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.g 2024-03-24T00:18:36.319 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:36.315+0000 7f33575fc740 20 client.26264 populate_metadata read hostname 'smithi118' 2024-03-24T00:18:36.319 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:36.315+0000 7f33575fc740 1 --2- 172.21.15.118:0/2253204216 >> [v2:172.21.15.118:6838/3285788503,v1:172.21.15.118:6839/3285788503] conn(0x55de180ae590 0x55de180b09f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:18:36.319 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:36.315+0000 7f33575fc740 4 client.26264 mds_command: new command op to 24550 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-24T00:18:36.319 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:36.315+0000 7f33575fc740 1 -- 172.21.15.118:0/2253204216 --> [v2:172.21.15.118:6838/3285788503,v1:172.21.15.118:6839/3285788503] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55de1800a800 con 0x55de180ae590 2024-03-24T00:18:36.319 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:36.315+0000 7f3351e00640 1 --2- 172.21.15.118:0/2253204216 >> [v2:172.21.15.118:6838/3285788503,v1:172.21.15.118:6839/3285788503] conn(0x55de180ae590 0x55de180b09f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:18:36.319 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:36.316+0000 7f3351e00640 1 --2- 172.21.15.118:0/2253204216 >> [v2:172.21.15.118:6838/3285788503,v1:172.21.15.118:6839/3285788503] conn(0x55de180ae590 0x55de180b09f0 crc :-1 s=READY pgs=30 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=36d8b746c06f0b15 server_cookie=9cfbe539471f1bb1 in_seq=0 out_seq=0 2024-03-24T00:18:36.319 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:36.316+0000 7f3333400640 10 client.26264 ms_handle_connect on v2:172.21.15.118:6838/3285788503 2024-03-24T00:18:36.319 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:36.317+0000 7f3333400640 1 -- 172.21.15.118:0/2253204216 <== mds.1 v2:172.21.15.118:6838/3285788503 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x55de1800a800 con 0x55de180ae590 2024-03-24T00:18:36.319 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:36.317+0000 7f3333400640 10 client.26264 handle_command_reply: tid=0 2024-03-24T00:18:36.319 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:36.317+0000 7f33575fc740 1 client.26264 shutdown 2024-03-24T00:18:36.319 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:36.317+0000 7f33575fc740 1 -- 172.21.15.118:0/2253204216 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55de18099bc0 msgr2=0x55de180a3150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:18:36.319 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:36.317+0000 7f33575fc740 1 --2- 172.21.15.118:0/2253204216 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55de18099bc0 0x55de180a3150 secure :-1 s=READY pgs=347 cs=0 l=1 rev1=1 crypto rx=0x7f33380a3a20 tx=0x7f333809b3a0 comp rx=0 tx=0).stop 2024-03-24T00:18:36.319 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:36.317+0000 7f33575fc740 1 -- 172.21.15.118:0/2253204216 shutdown_connections 2024-03-24T00:18:36.320 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:36.317+0000 7f33575fc740 1 --2- 172.21.15.118:0/2253204216 >> [v2:172.21.15.118:6838/3285788503,v1:172.21.15.118:6839/3285788503] conn(0x55de180ae590 0x55de180b09f0 crc :-1 s=READY pgs=30 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:18:36.320 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:36.317+0000 7f3333400640 0 client.26264 ms_handle_reset on v2:172.21.15.118:6838/3285788503 2024-03-24T00:18:36.320 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:36.317+0000 7f33575fc740 1 --2- 172.21.15.118:0/2253204216 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55de1809a580 0x55de180aa190 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:18:36.320 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:36.317+0000 7f33575fc740 1 --2- 172.21.15.118:0/2253204216 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55de18099bc0 0x55de180a3150 unknown :-1 s=CLOSED pgs=347 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:18:36.320 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:36.317+0000 7f33575fc740 1 --2- 172.21.15.118:0/2253204216 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55de180989f0 0x55de180a2c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:18:36.320 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:36.317+0000 7f33575fc740 1 -- 172.21.15.118:0/2253204216 >> 172.21.15.118:0/2253204216 conn(0x55de18093de0 msgr2=0x55de18093160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:18:36.320 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:36.317+0000 7f33575fc740 1 -- 172.21.15.118:0/2253204216 shutdown_connections 2024-03-24T00:18:36.320 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:36.317+0000 7f33575fc740 1 -- 172.21.15.118:0/2253204216 wait complete. 2024-03-24T00:18:36.320 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:36.317+0000 7f33575fc740 20 client.26264 trim_cache size 0 max 16384 2024-03-24T00:18:36.333 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:36.333+0000 7f33575fc740 1 Processor -- start 2024-03-24T00:18:36.333 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:36.333+0000 7f33575fc740 1 -- start start 2024-03-24T00:18:36.333 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:36.333+0000 7f33575fc740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55de18093de0 0x55de18092fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:18:36.333 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:36.333+0000 7f33575fc740 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55de180989f0 0x55de180aa150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:18:36.333 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:36.333+0000 7f33575fc740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55de18099bc0 0x55de180b52b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:18:36.333 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:36.333+0000 7f33575fc740 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f334c1a6660 con 0x55de180989f0 2024-03-24T00:18:36.333 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:36.333+0000 7f33575fc740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f334c1a7fe0 con 0x55de18093de0 2024-03-24T00:18:36.333 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:36.333+0000 7f33575fc740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f334c1a6cc0 con 0x55de18099bc0 2024-03-24T00:18:36.334 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:36.333+0000 7f3351e00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55de18093de0 0x55de18092fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:18:36.334 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:36.333+0000 7f3352800640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55de18099bc0 0x55de180b52b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:18:36.334 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:36.334+0000 7f3351400640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55de180989f0 0x55de180aa150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:18:36.334 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:36.334+0000 7f3351e00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55de18093de0 0x55de18092fe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.118:32934/0 (socket says 172.21.15.118:32934) 2024-03-24T00:18:36.334 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:36.334+0000 7f3351e00640 1 -- 172.21.15.118:0/2342210168 learned_addr learned my addr 172.21.15.118:0/2342210168 (peer_addr_for_me v2:172.21.15.118:0/0) 2024-03-24T00:18:36.335 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:36.334+0000 7f3352800640 1 -- 172.21.15.118:0/2342210168 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55de18093de0 msgr2=0x55de18092fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:18:36.335 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:36.334+0000 7f3352800640 1 --2- 172.21.15.118:0/2342210168 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55de18093de0 0x55de18092fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:18:36.335 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:36.334+0000 7f3352800640 1 -- 172.21.15.118:0/2342210168 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55de180989f0 msgr2=0x55de180aa150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:18:36.335 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:36.334+0000 7f3352800640 1 --2- 172.21.15.118:0/2342210168 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55de180989f0 0x55de180aa150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:18:36.335 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:36.334+0000 7f3352800640 1 -- 172.21.15.118:0/2342210168 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55de18099640 con 0x55de18099bc0 2024-03-24T00:18:36.335 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:36.334+0000 7f3352800640 1 --2- 172.21.15.118:0/2342210168 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55de18099bc0 0x55de180b52b0 secure :-1 s=READY pgs=348 cs=0 l=1 rev1=1 crypto rx=0x7f334802c430 tx=0x7f3348042000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=5ebdcbb3821b242a server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:18:36.335 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:36.334+0000 7f333e600640 1 -- 172.21.15.118:0/2342210168 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f334802c950 con 0x55de18099bc0 2024-03-24T00:18:36.335 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:36.334+0000 7f333e600640 1 -- 172.21.15.118:0/2342210168 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f3348004040 con 0x55de18099bc0 2024-03-24T00:18:36.335 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:36.334+0000 7f333e600640 1 -- 172.21.15.118:0/2342210168 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3348034690 con 0x55de18099bc0 2024-03-24T00:18:36.338 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:36.335+0000 7f33575fc740 1 -- 172.21.15.118:0/2342210168 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55de18099bc0 msgr2=0x55de180b52b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:18:36.338 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:36.335+0000 7f33575fc740 1 --2- 172.21.15.118:0/2342210168 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55de18099bc0 0x55de180b52b0 secure :-1 s=READY pgs=348 cs=0 l=1 rev1=1 crypto rx=0x7f334802c430 tx=0x7f3348042000 comp rx=0 tx=0).stop 2024-03-24T00:18:36.338 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:36.335+0000 7f33575fc740 1 -- 172.21.15.118:0/2342210168 shutdown_connections 2024-03-24T00:18:36.338 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:36.335+0000 7f33575fc740 1 --2- 172.21.15.118:0/2342210168 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55de18099bc0 0x55de180b52b0 unknown :-1 s=CLOSED pgs=348 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:18:36.338 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:36.335+0000 7f33575fc740 1 --2- 172.21.15.118:0/2342210168 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55de180989f0 0x55de180aa150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:18:36.339 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:36.335+0000 7f33575fc740 1 --2- 172.21.15.118:0/2342210168 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55de18093de0 0x55de18092fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:18:36.339 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:36.335+0000 7f33575fc740 1 -- 172.21.15.118:0/2342210168 >> 172.21.15.118:0/2342210168 conn(0x55de180a1e20 msgr2=0x55de180928d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:18:36.339 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:36.335+0000 7f33575fc740 1 -- 172.21.15.118:0/2342210168 shutdown_connections 2024-03-24T00:18:36.339 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:36.335+0000 7f33575fc740 1 -- 172.21.15.118:0/2342210168 wait complete. 2024-03-24T00:18:36.339 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:36.336+0000 7f33575fc740 1 -- start start 2024-03-24T00:18:36.339 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:36.336+0000 7f33575fc740 1 Processor -- start 2024-03-24T00:18:36.339 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:36.336+0000 7f33575fc740 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55de18093de0 0x55de180a6de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:18:36.339 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:36.336+0000 7f33575fc740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55de180989f0 0x55de180a7320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:18:36.339 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:36.336+0000 7f33575fc740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55de18099bc0 0x55de18109e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:18:36.339 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:36.336+0000 7f33575fc740 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f334c1a7980 con 0x55de18093de0 2024-03-24T00:18:36.339 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:36.336+0000 7f33575fc740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f334c1a7320 con 0x55de180989f0 2024-03-24T00:18:36.339 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:36.336+0000 7f33575fc740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f334c1a4840 con 0x55de18099bc0 2024-03-24T00:18:36.339 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:36.336+0000 7f33575fc740 10 client.0 _ll_register_callbacks cb 0x7f333800ce60 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T00:18:36.339 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:36.337+0000 7f3351400640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55de180989f0 0x55de180a7320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:18:36.340 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:36.337+0000 7f3351400640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55de180989f0 0x55de180a7320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.118:32936/0 (socket says 172.21.15.118:32936) 2024-03-24T00:18:36.340 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:36.337+0000 7f3351400640 1 -- 172.21.15.118:0/705371843 learned_addr learned my addr 172.21.15.118:0/705371843 (peer_addr_for_me v2:172.21.15.118:0/0) 2024-03-24T00:18:36.340 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:36.337+0000 7f3352800640 1 --2- 172.21.15.118:0/705371843 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55de18099bc0 0x55de18109e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:18:36.340 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:36.337+0000 7f3351e00640 1 --2- 172.21.15.118:0/705371843 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55de18093de0 0x55de180a6de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:18:36.340 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:36.337+0000 7f3351400640 1 -- 172.21.15.118:0/705371843 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55de18099bc0 msgr2=0x55de18109e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:18:36.340 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:36.337+0000 7f3351400640 1 --2- 172.21.15.118:0/705371843 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55de18099bc0 0x55de18109e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:18:36.340 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:36.337+0000 7f3351400640 1 -- 172.21.15.118:0/705371843 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55de18093de0 msgr2=0x55de180a6de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:18:36.340 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:36.337+0000 7f3351400640 1 --2- 172.21.15.118:0/705371843 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55de18093de0 0x55de180a6de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:18:36.340 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:36.337+0000 7f3351400640 1 -- 172.21.15.118:0/705371843 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55de180aa3f0 con 0x55de180989f0 2024-03-24T00:18:36.340 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:36.337+0000 7f3352800640 1 --2- 172.21.15.118:0/705371843 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55de18099bc0 0x55de18109e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T00:18:36.340 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:36.337+0000 7f3351400640 1 --2- 172.21.15.118:0/705371843 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55de180989f0 0x55de180a7320 secure :-1 s=READY pgs=348 cs=0 l=1 rev1=1 crypto rx=0x7f333809b560 tx=0x7f33380863e0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=34016ee3442930e8 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:18:36.340 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:36.337+0000 7f3333400640 10 client.0 ms_handle_connect on v2:172.21.15.189:3300/0 2024-03-24T00:18:36.340 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:36.338+0000 7f3333400640 1 -- 172.21.15.118:0/705371843 <== mon.2 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3338097b00 con 0x55de180989f0 2024-03-24T00:18:36.340 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:36.338+0000 7f3333400640 1 -- 172.21.15.118:0/705371843 <== mon.2 v2:172.21.15.189:3300/0 2 ==== osd_map(90..90 src has 1..90) v4 ==== 8277+0+0 (secure 0 0 0) 0x7f33380afd90 con 0x55de180989f0 2024-03-24T00:18:36.341 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:36.338+0000 7f33575fc740 1 -- 172.21.15.118:0/705371843 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f3348004040 con 0x55de180989f0 2024-03-24T00:18:36.341 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:36.338+0000 7f3333400640 1 -- 172.21.15.118:0/705371843 <== mon.2 v2:172.21.15.189:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) v2 ==== 24+0+0 (secure 0 0 0) 0x7f333809b1c0 con 0x55de180989f0 2024-03-24T00:18:36.341 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:36.338+0000 7f33575fc740 10 client.26270 fetch_fsmap learned FSMap version 28 2024-03-24T00:18:36.341 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:36.338+0000 7f33575fc740 1 -- 172.21.15.118:0/705371843 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) v3 -- 0x55de1810b3a0 con 0x55de180989f0 2024-03-24T00:18:36.341 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:36.338+0000 7f3333400640 1 -- 172.21.15.118:0/705371843 <== mon.2 v2:172.21.15.189:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f3348004040 con 0x55de180989f0 2024-03-24T00:18:36.341 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:36.338+0000 7f3333400640 1 -- 172.21.15.118:0/705371843 <== mon.2 v2:172.21.15.189:3300/0 5 ==== fsmap(e 28) v1 ==== 7934+0+0 (secure 0 0 0) 0x7f3338097740 con 0x55de180989f0 2024-03-24T00:18:36.341 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:36.338+0000 7f33575fc740 10 client.26270 fetch_fsmap finished waiting for FSMap version 28 2024-03-24T00:18:36.341 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:36.338+0000 7f33575fc740 10 client.26270 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.g 2024-03-24T00:18:36.341 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:36.338+0000 7f33575fc740 20 client.26270 populate_metadata read hostname 'smithi118' 2024-03-24T00:18:36.341 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:36.338+0000 7f33575fc740 1 --2- 172.21.15.118:0/705371843 >> [v2:172.21.15.118:6838/3285788503,v1:172.21.15.118:6839/3285788503] conn(0x55de1809a580 0x55de1810daf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:18:36.341 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:36.338+0000 7f33575fc740 4 client.26270 mds_command: new command op to 24550 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-24T00:18:36.341 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:36.338+0000 7f33575fc740 1 -- 172.21.15.118:0/705371843 --> [v2:172.21.15.118:6838/3285788503,v1:172.21.15.118:6839/3285788503] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x55de18004b50 con 0x55de1809a580 2024-03-24T00:18:36.341 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:36.339+0000 7f3351e00640 1 --2- 172.21.15.118:0/705371843 >> [v2:172.21.15.118:6838/3285788503,v1:172.21.15.118:6839/3285788503] conn(0x55de1809a580 0x55de1810daf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:18:36.343 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:36.339+0000 7f3351e00640 1 --2- 172.21.15.118:0/705371843 >> [v2:172.21.15.118:6838/3285788503,v1:172.21.15.118:6839/3285788503] conn(0x55de1809a580 0x55de1810daf0 crc :-1 s=READY pgs=31 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=31b66b15b787ab01 server_cookie=84c37cbde9595b00 in_seq=0 out_seq=0 2024-03-24T00:18:36.343 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:36.339+0000 7f3333400640 10 client.26270 ms_handle_connect on v2:172.21.15.118:6838/3285788503 2024-03-24T00:18:36.343 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:36.340+0000 7f3333400640 1 -- 172.21.15.118:0/705371843 <== mds.1 v2:172.21.15.118:6838/3285788503 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x55de18004b50 con 0x55de1809a580 2024-03-24T00:18:36.343 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:36.340+0000 7f3333400640 10 client.26270 handle_command_reply: tid=0 2024-03-24T00:18:36.343 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:36.340+0000 7f33575fc740 1 client.26270 shutdown 2024-03-24T00:18:36.343 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:36.340+0000 7f33575fc740 1 -- 172.21.15.118:0/705371843 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55de180989f0 msgr2=0x55de180a7320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:18:36.343 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:36.340+0000 7f33575fc740 1 --2- 172.21.15.118:0/705371843 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55de180989f0 0x55de180a7320 secure :-1 s=READY pgs=348 cs=0 l=1 rev1=1 crypto rx=0x7f333809b560 tx=0x7f33380863e0 comp rx=0 tx=0).stop 2024-03-24T00:18:36.343 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:36.340+0000 7f33575fc740 1 -- 172.21.15.118:0/705371843 shutdown_connections 2024-03-24T00:18:36.343 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:36.340+0000 7f33575fc740 1 --2- 172.21.15.118:0/705371843 >> [v2:172.21.15.118:6838/3285788503,v1:172.21.15.118:6839/3285788503] conn(0x55de1809a580 0x55de1810daf0 crc :-1 s=READY pgs=31 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:18:36.343 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:36.340+0000 7f33575fc740 1 --2- 172.21.15.118:0/705371843 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55de18099bc0 0x55de18109e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:18:36.343 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:36.340+0000 7f33575fc740 1 --2- 172.21.15.118:0/705371843 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55de180989f0 0x55de180a7320 unknown :-1 s=CLOSED pgs=348 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:18:36.343 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:36.340+0000 7f33575fc740 1 --2- 172.21.15.118:0/705371843 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55de18093de0 0x55de180a6de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:18:36.344 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:36.340+0000 7f33575fc740 1 -- 172.21.15.118:0/705371843 >> 172.21.15.118:0/705371843 conn(0x55de180a1e20 msgr2=0x55de180928d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:18:36.344 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:36.340+0000 7f3333400640 0 client.26270 ms_handle_reset on v2:172.21.15.118:6838/3285788503 2024-03-24T00:18:36.344 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:36.340+0000 7f33575fc740 1 -- 172.21.15.118:0/705371843 shutdown_connections 2024-03-24T00:18:36.344 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:36.340+0000 7f33575fc740 1 -- 172.21.15.118:0/705371843 wait complete. 2024-03-24T00:18:36.344 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:36.340+0000 7f33575fc740 20 client.26270 trim_cache size 0 max 16384 2024-03-24T00:18:36.344 INFO:teuthology.orchestra.run.smithi118.stdout:[] 2024-03-24T00:18:36.345 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:36.341+0000 7f3353200640 1 -- 172.21.15.118:0/1827646816 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7f33340a2800 msgr2=0x7f33340a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:18:36.345 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:36.341+0000 7f3353200640 1 --2- 172.21.15.118:0/1827646816 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7f33340a2800 0x7f33340a4cc0 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 crypto rx=0x7f3348006010 tx=0x7f3348004a40 comp rx=0 tx=0).stop 2024-03-24T00:18:36.345 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:36.341+0000 7f3353200640 1 -- 172.21.15.118:0/1827646816 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f334c100760 msgr2=0x7f334c199f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:18:36.345 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:36.341+0000 7f3353200640 1 --2- 172.21.15.118:0/1827646816 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f334c100760 0x7f334c199f50 secure :-1 s=READY pgs=346 cs=0 l=1 rev1=1 crypto rx=0x7f333800e9f0 tx=0x7f333800eeb0 comp rx=0 tx=0).stop 2024-03-24T00:18:36.345 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:36.341+0000 7f3353200640 1 -- 172.21.15.118:0/1827646816 shutdown_connections 2024-03-24T00:18:36.345 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:36.341+0000 7f3353200640 1 --2- 172.21.15.118:0/1827646816 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7f33340a2800 0x7f33340a4cc0 unknown :-1 s=CLOSED pgs=324 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:18:36.345 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:36.341+0000 7f3353200640 1 --2- 172.21.15.118:0/1827646816 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f334c101100 0x7f334c1a0fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:18:36.345 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:36.341+0000 7f3353200640 1 --2- 172.21.15.118:0/1827646816 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f334c100760 0x7f334c199f50 unknown :-1 s=CLOSED pgs=346 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:18:36.346 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:36.341+0000 7f3353200640 1 --2- 172.21.15.118:0/1827646816 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f334c0ffe00 0x7f334c199a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:18:36.346 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:36.341+0000 7f3353200640 1 -- 172.21.15.118:0/1827646816 >> 172.21.15.118:0/1827646816 conn(0x7f334c0fba20 msgr2=0x7f334c0fd050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:18:36.346 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:36.341+0000 7f3353200640 1 -- 172.21.15.118:0/1827646816 shutdown_connections 2024-03-24T00:18:36.346 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:36.341+0000 7f3353200640 1 -- 172.21.15.118:0/1827646816 wait complete. 2024-03-24T00:18:36.358 INFO:teuthology.orchestra.run:Running command with timeout 120 2024-03-24T00:18:36.359 DEBUG:teuthology.orchestra.run.smithi118:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2024-03-24T00:18:36.563 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:36.563+0000 7fb3f1000640 1 -- 172.21.15.118:0/3972714366 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fb3ec103260 msgr2=0x7fb3ec109af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:18:36.564 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:36.563+0000 7fb3f1000640 1 --2- 172.21.15.118:0/3972714366 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fb3ec103260 0x7fb3ec109af0 secure :-1 s=READY pgs=349 cs=0 l=1 rev1=1 crypto rx=0x7fb3dc009900 tx=0x7fb3dc02bc30 comp rx=0 tx=0).stop 2024-03-24T00:18:36.564 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:36.563+0000 7fb3f1000640 1 -- 172.21.15.118:0/3972714366 shutdown_connections 2024-03-24T00:18:36.564 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:36.563+0000 7fb3f1000640 1 --2- 172.21.15.118:0/3972714366 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fb3ec103260 0x7fb3ec109af0 unknown :-1 s=CLOSED pgs=349 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:18:36.564 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:36.563+0000 7fb3f1000640 1 --2- 172.21.15.118:0/3972714366 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb3ec1028a0 0x7fb3ec102d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:18:36.564 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:36.563+0000 7fb3f1000640 1 --2- 172.21.15.118:0/3972714366 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb3ec1016b0 0x7fb3ec101ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:18:36.564 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:36.563+0000 7fb3f1000640 1 -- 172.21.15.118:0/3972714366 >> 172.21.15.118:0/3972714366 conn(0x7fb3ec0fcc40 msgr2=0x7fb3ec0ff060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:18:36.564 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:36.563+0000 7fb3f1000640 1 -- 172.21.15.118:0/3972714366 shutdown_connections 2024-03-24T00:18:36.564 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:36.563+0000 7fb3f1000640 1 -- 172.21.15.118:0/3972714366 wait complete. 2024-03-24T00:18:36.564 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:36.564+0000 7fb3f1000640 1 Processor -- start 2024-03-24T00:18:36.564 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:36.564+0000 7fb3f1000640 1 -- start start 2024-03-24T00:18:36.566 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:36.564+0000 7fb3f1000640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fb3ec1016b0 0x7fb3ec197970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:18:36.567 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:36.564+0000 7fb3f1000640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb3ec1028a0 0x7fb3ec197eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:18:36.567 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:36.564+0000 7fb3f1000640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb3ec103260 0x7fb3ec19ef20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:18:36.567 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:36.564+0000 7fb3f1000640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb3ec10d380 con 0x7fb3ec1028a0 2024-03-24T00:18:36.567 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:36.564+0000 7fb3f1000640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb3ec10d200 con 0x7fb3ec103260 2024-03-24T00:18:36.567 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:36.564+0000 7fb3f1000640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb3ec10d500 con 0x7fb3ec1016b0 2024-03-24T00:18:36.567 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:36.565+0000 7fb3ea800640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb3ec103260 0x7fb3ec19ef20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:18:36.567 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:36.565+0000 7fb3ea800640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb3ec103260 0x7fb3ec19ef20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.118:32954/0 (socket says 172.21.15.118:32954) 2024-03-24T00:18:36.567 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:36.565+0000 7fb3ea800640 1 -- 172.21.15.118:0/1047641023 learned_addr learned my addr 172.21.15.118:0/1047641023 (peer_addr_for_me v2:172.21.15.118:0/0) 2024-03-24T00:18:36.567 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:36.565+0000 7fb3e9400640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb3ec1028a0 0x7fb3ec197eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:18:36.567 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:36.565+0000 7fb3e9e00640 1 --2- 172.21.15.118:0/1047641023 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fb3ec1016b0 0x7fb3ec197970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:18:36.567 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:36.565+0000 7fb3ea800640 1 -- 172.21.15.118:0/1047641023 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fb3ec1016b0 msgr2=0x7fb3ec197970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:18:36.567 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:36.565+0000 7fb3ea800640 1 --2- 172.21.15.118:0/1047641023 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fb3ec1016b0 0x7fb3ec197970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:18:36.567 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:36.565+0000 7fb3ea800640 1 -- 172.21.15.118:0/1047641023 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb3ec1028a0 msgr2=0x7fb3ec197eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:18:36.568 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:36.565+0000 7fb3ea800640 1 --2- 172.21.15.118:0/1047641023 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb3ec1028a0 0x7fb3ec197eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:18:36.568 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:36.565+0000 7fb3ea800640 1 -- 172.21.15.118:0/1047641023 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb3ec19f590 con 0x7fb3ec103260 2024-03-24T00:18:36.568 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:36.565+0000 7fb3ea800640 1 --2- 172.21.15.118:0/1047641023 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb3ec103260 0x7fb3ec19ef20 secure :-1 s=READY pgs=349 cs=0 l=1 rev1=1 crypto rx=0x7fb3dc02c8e0 tx=0x7fb3dc02c640 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:18:36.568 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:36.566+0000 7fb3e2a00640 1 -- 172.21.15.118:0/1047641023 <== mon.2 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb3dc043020 con 0x7fb3ec103260 2024-03-24T00:18:36.568 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:36.566+0000 7fb3f1000640 1 -- 172.21.15.118:0/1047641023 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb3ec19f820 con 0x7fb3ec103260 2024-03-24T00:18:36.568 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:36.566+0000 7fb3e2a00640 1 -- 172.21.15.118:0/1047641023 <== mon.2 v2:172.21.15.189:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fb3dc0054e0 con 0x7fb3ec103260 2024-03-24T00:18:36.568 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:36.566+0000 7fb3e2a00640 1 -- 172.21.15.118:0/1047641023 <== mon.2 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb3dc003d00 con 0x7fb3ec103260 2024-03-24T00:18:36.568 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:36.566+0000 7fb3f1000640 1 -- 172.21.15.118:0/1047641023 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb3ec19fd70 con 0x7fb3ec103260 2024-03-24T00:18:36.569 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:36.569+0000 7fb3f3d96740 1 Processor -- start 2024-03-24T00:18:36.569 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:36.569+0000 7fb3e2a00640 1 -- 172.21.15.118:0/1047641023 <== mon.2 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 20) v1 ==== 137291+0+0 (secure 0 0 0) 0x7fb3dc046020 con 0x7fb3ec103260 2024-03-24T00:18:36.569 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:36.569+0000 7fb3f3d96740 1 -- start start 2024-03-24T00:18:36.571 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:36.570+0000 7fb3f3d96740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x561eb216b810 0x561eb216bc30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:18:36.571 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:36.570+0000 7fb3f3d96740 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x561eb216c9e0 0x561eb216ce60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:18:36.571 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:36.570+0000 7fb3f3d96740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x561eb216d3a0 0x561eb2173c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:18:36.572 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:36.570+0000 7fb3f3d96740 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x561eb20d02c0 con 0x561eb216c9e0 2024-03-24T00:18:36.572 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:36.570+0000 7fb3f3d96740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x561eb2078c90 con 0x561eb216d3a0 2024-03-24T00:18:36.572 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:36.570+0000 7fb3f3d96740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x561eb20cc360 con 0x561eb216b810 2024-03-24T00:18:36.572 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:36.570+0000 7fb3e9e00640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x561eb216c9e0 0x561eb216ce60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:18:36.572 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:36.570+0000 7fb3e9400640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x561eb216b810 0x561eb216bc30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:18:36.572 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:36.570+0000 7fb3e9e00640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x561eb216c9e0 0x561eb216ce60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.118:3300/0 says I am v2:172.21.15.118:38176/0 (socket says 172.21.15.118:38176) 2024-03-24T00:18:36.572 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:36.570+0000 7fb3e9e00640 1 -- 172.21.15.118:0/2203756829 learned_addr learned my addr 172.21.15.118:0/2203756829 (peer_addr_for_me v2:172.21.15.118:0/0) 2024-03-24T00:18:36.572 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:36.570+0000 7fb3e9400640 1 --2- 172.21.15.118:0/2203756829 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x561eb216d3a0 0x561eb2173c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:18:36.572 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:36.571+0000 7fb3e9400640 1 -- 172.21.15.118:0/2203756829 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x561eb216d3a0 msgr2=0x561eb2173c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:18:36.572 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:36.571+0000 7fb3e9400640 1 --2- 172.21.15.118:0/2203756829 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x561eb216d3a0 0x561eb2173c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:18:36.572 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:36.571+0000 7fb3e9400640 1 -- 172.21.15.118:0/2203756829 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x561eb216c9e0 msgr2=0x561eb216ce60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:18:36.572 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:36.571+0000 7fb3e9400640 1 --2- 172.21.15.118:0/2203756829 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x561eb216c9e0 0x561eb216ce60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:18:36.573 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:36.571+0000 7fb3e9400640 1 -- 172.21.15.118:0/2203756829 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x561eb209d080 con 0x561eb216b810 2024-03-24T00:18:36.573 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:36.571+0000 7fb3e9e00640 1 --2- 172.21.15.118:0/2203756829 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x561eb216c9e0 0x561eb216ce60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T00:18:36.573 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:36.571+0000 7fb3e9400640 1 --2- 172.21.15.118:0/2203756829 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x561eb216b810 0x561eb216bc30 secure :-1 s=READY pgs=350 cs=0 l=1 rev1=1 crypto rx=0x7fb3ec198e80 tx=0x7fb3d8004730 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c06baf4fb9d9b158 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:18:36.573 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:36.571+0000 7fb3e2a00640 1 --2- 172.21.15.118:0/1047641023 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7fb3d00a2800 0x7fb3d00a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:18:36.573 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:36.571+0000 7fb3d7400640 1 -- 172.21.15.118:0/2203756829 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb3d8004050 con 0x561eb216b810 2024-03-24T00:18:36.573 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:36.571+0000 7fb3d7400640 1 -- 172.21.15.118:0/2203756829 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fb3d800fca0 con 0x561eb216b810 2024-03-24T00:18:36.573 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:36.571+0000 7fb3d7400640 1 -- 172.21.15.118:0/2203756829 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb3d80039b0 con 0x561eb216b810 2024-03-24T00:18:36.573 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:36.571+0000 7fb3e2a00640 1 -- 172.21.15.118:0/1047641023 <== mon.2 v2:172.21.15.189:3300/0 5 ==== osd_map(90..90 src has 1..90) v4 ==== 8277+0+0 (secure 0 0 0) 0x7fb3dc0f0b60 con 0x7fb3ec103260 2024-03-24T00:18:36.573 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:36.571+0000 7fb3e9e00640 1 --2- 172.21.15.118:0/1047641023 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7fb3d00a2800 0x7fb3d00a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:18:36.573 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:36.572+0000 7fb3f3d96740 1 -- 172.21.15.118:0/2203756829 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x561eb216b810 msgr2=0x561eb216bc30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:18:36.573 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:36.572+0000 7fb3f3d96740 1 --2- 172.21.15.118:0/2203756829 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x561eb216b810 0x561eb216bc30 secure :-1 s=READY pgs=350 cs=0 l=1 rev1=1 crypto rx=0x7fb3ec198e80 tx=0x7fb3d8004730 comp rx=0 tx=0).stop 2024-03-24T00:18:36.577 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:36.572+0000 7fb3f3d96740 1 -- 172.21.15.118:0/2203756829 shutdown_connections 2024-03-24T00:18:36.577 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:36.572+0000 7fb3f3d96740 1 --2- 172.21.15.118:0/2203756829 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x561eb216d3a0 0x561eb2173c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:18:36.577 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:36.572+0000 7fb3f3d96740 1 --2- 172.21.15.118:0/2203756829 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x561eb216c9e0 0x561eb216ce60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:18:36.577 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:36.572+0000 7fb3f3d96740 1 --2- 172.21.15.118:0/2203756829 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x561eb216b810 0x561eb216bc30 unknown :-1 s=CLOSED pgs=350 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:18:36.577 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:36.572+0000 7fb3f3d96740 1 -- 172.21.15.118:0/2203756829 >> 172.21.15.118:0/2203756829 conn(0x561eb2166c00 msgr2=0x561eb2167000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:18:36.577 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:36.572+0000 7fb3e9e00640 1 --2- 172.21.15.118:0/1047641023 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7fb3d00a2800 0x7fb3d00a4cc0 secure :-1 s=READY pgs=325 cs=0 l=1 rev1=1 crypto rx=0x7fb3e4009fd0 tx=0x7fb3e4009500 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:18:36.577 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:36.572+0000 7fb3f3d96740 1 -- 172.21.15.118:0/2203756829 shutdown_connections 2024-03-24T00:18:36.577 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:36.572+0000 7fb3f3d96740 1 -- 172.21.15.118:0/2203756829 wait complete. 2024-03-24T00:18:36.577 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:36.573+0000 7fb3f3d96740 1 -- start start 2024-03-24T00:18:36.578 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:36.573+0000 7fb3f3d96740 1 Processor -- start 2024-03-24T00:18:36.578 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:36.574+0000 7fb3f3d96740 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x561eb216b810 0x561eb2175ad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:18:36.578 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:36.574+0000 7fb3f3d96740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x561eb216c9e0 0x561eb2176010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:18:36.578 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:36.574+0000 7fb3f3d96740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x561eb216d3a0 0x561eb217d050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:18:36.578 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:36.574+0000 7fb3f3d96740 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x561eb2174400 con 0x561eb216b810 2024-03-24T00:18:36.578 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:36.574+0000 7fb3f3d96740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x561eb208c0d0 con 0x561eb216c9e0 2024-03-24T00:18:36.578 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:36.574+0000 7fb3f3d96740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x561eb2174580 con 0x561eb216d3a0 2024-03-24T00:18:36.578 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:36.574+0000 7fb3f3d96740 10 client.0 _ll_register_callbacks cb 0x561eb2090e80 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T00:18:36.578 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:36.574+0000 7fb3e9400640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x561eb216c9e0 0x561eb2176010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:18:36.578 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:36.574+0000 7fb3e9400640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x561eb216c9e0 0x561eb2176010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.118:32974/0 (socket says 172.21.15.118:32974) 2024-03-24T00:18:36.578 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:36.574+0000 7fb3e9400640 1 -- 172.21.15.118:0/3326954214 learned_addr learned my addr 172.21.15.118:0/3326954214 (peer_addr_for_me v2:172.21.15.118:0/0) 2024-03-24T00:18:36.578 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:36.574+0000 7fb3ea800640 1 --2- 172.21.15.118:0/3326954214 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x561eb216d3a0 0x561eb217d050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:18:36.578 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:36.574+0000 7fb3e9e00640 1 --2- 172.21.15.118:0/3326954214 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x561eb216b810 0x561eb2175ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:18:36.578 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:36.574+0000 7fb3e9400640 1 -- 172.21.15.118:0/3326954214 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x561eb216d3a0 msgr2=0x561eb217d050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:18:36.579 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:36.574+0000 7fb3e9400640 1 --2- 172.21.15.118:0/3326954214 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x561eb216d3a0 0x561eb217d050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:18:36.579 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:36.574+0000 7fb3e9400640 1 -- 172.21.15.118:0/3326954214 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x561eb216b810 msgr2=0x561eb2175ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:18:36.579 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:36.574+0000 7fb3e9400640 1 --2- 172.21.15.118:0/3326954214 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x561eb216b810 0x561eb2175ad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:18:36.579 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:36.574+0000 7fb3e9400640 1 -- 172.21.15.118:0/3326954214 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x561eb1de43f0 con 0x561eb216c9e0 2024-03-24T00:18:36.579 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:36.575+0000 7fb3e9400640 1 --2- 172.21.15.118:0/3326954214 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x561eb216c9e0 0x561eb2176010 secure :-1 s=READY pgs=350 cs=0 l=1 rev1=1 crypto rx=0x7fb3d80097a0 tx=0x7fb3d8009bf0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=303917d0a7b14888 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:18:36.579 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:36.575+0000 7fb3cbe00640 10 client.0 ms_handle_connect on v2:172.21.15.189:3300/0 2024-03-24T00:18:36.579 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:36.575+0000 7fb3cbe00640 1 -- 172.21.15.118:0/3326954214 <== mon.2 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb3d801f850 con 0x561eb216c9e0 2024-03-24T00:18:36.579 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:36.575+0000 7fb3cbe00640 1 -- 172.21.15.118:0/3326954214 <== mon.2 v2:172.21.15.189:3300/0 2 ==== osd_map(90..90 src has 1..90) v4 ==== 8277+0+0 (secure 0 0 0) 0x7fb3d801f9f0 con 0x561eb216c9e0 2024-03-24T00:18:36.579 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:36.575+0000 7fb3f3d96740 1 -- 172.21.15.118:0/3326954214 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fb3d800fca0 con 0x561eb216c9e0 2024-03-24T00:18:36.579 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:36.575+0000 7fb3cbe00640 1 -- 172.21.15.118:0/3326954214 <== mon.2 v2:172.21.15.189:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) v2 ==== 24+0+0 (secure 0 0 0) 0x7fb3d801fbf0 con 0x561eb216c9e0 2024-03-24T00:18:36.579 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:36.575+0000 7fb3f3d96740 10 client.26282 fetch_fsmap learned FSMap version 28 2024-03-24T00:18:36.579 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:36.575+0000 7fb3f3d96740 1 -- 172.21.15.118:0/3326954214 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) v3 -- 0x561eb1eae7d0 con 0x561eb216c9e0 2024-03-24T00:18:36.579 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:36.576+0000 7fb3cbe00640 1 -- 172.21.15.118:0/3326954214 <== mon.2 v2:172.21.15.189:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fb3d800fca0 con 0x561eb216c9e0 2024-03-24T00:18:36.579 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:36.576+0000 7fb3cbe00640 1 -- 172.21.15.118:0/3326954214 <== mon.2 v2:172.21.15.189:3300/0 5 ==== fsmap(e 28) v1 ==== 7934+0+0 (secure 0 0 0) 0x7fb3d8027070 con 0x561eb216c9e0 2024-03-24T00:18:36.580 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:36.576+0000 7fb3f3d96740 10 client.26282 fetch_fsmap finished waiting for FSMap version 28 2024-03-24T00:18:36.580 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:36.576+0000 7fb3f3d96740 10 client.26282 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-24T00:18:36.580 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:36.576+0000 7fb3f3d96740 20 client.26282 populate_metadata read hostname 'smithi118' 2024-03-24T00:18:36.580 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:36.576+0000 7fb3f3d96740 1 --2- 172.21.15.118:0/3326954214 >> [v2:172.21.15.189:6838/4074484168,v1:172.21.15.189:6839/4074484168] conn(0x561eb21812a0 0x561eb2183690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:18:36.580 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:36.576+0000 7fb3f3d96740 4 client.26282 mds_command: new command op to 24569 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-24T00:18:36.580 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:36.576+0000 7fb3f3d96740 1 -- 172.21.15.118:0/3326954214 --> [v2:172.21.15.189:6838/4074484168,v1:172.21.15.189:6839/4074484168] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x561eb20dd800 con 0x561eb21812a0 2024-03-24T00:18:36.580 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:36.576+0000 7fb3e9e00640 1 --2- 172.21.15.118:0/3326954214 >> [v2:172.21.15.189:6838/4074484168,v1:172.21.15.189:6839/4074484168] conn(0x561eb21812a0 0x561eb2183690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:18:36.580 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:36.577+0000 7fb3e9e00640 1 --2- 172.21.15.118:0/3326954214 >> [v2:172.21.15.189:6838/4074484168,v1:172.21.15.189:6839/4074484168] conn(0x561eb21812a0 0x561eb2183690 crc :-1 s=READY pgs=95 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=ffa32e7765f66d66 server_cookie=7fd5cba35b035322 in_seq=0 out_seq=0 2024-03-24T00:18:36.580 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:36.577+0000 7fb3cbe00640 10 client.26282 ms_handle_connect on v2:172.21.15.189:6838/4074484168 2024-03-24T00:18:36.584 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:36.584+0000 7fb3cbe00640 1 -- 172.21.15.118:0/3326954214 <== mds.0 v2:172.21.15.189:6838/4074484168 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x561eb20dd800 con 0x561eb21812a0 2024-03-24T00:18:36.584 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:36.584+0000 7fb3cbe00640 10 client.26282 handle_command_reply: tid=0 2024-03-24T00:18:36.584 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:36.584+0000 7fb3f3d96740 1 client.26282 shutdown 2024-03-24T00:18:36.584 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:36.584+0000 7fb3f3d96740 1 -- 172.21.15.118:0/3326954214 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x561eb216c9e0 msgr2=0x561eb2176010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:18:36.584 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:36.584+0000 7fb3f3d96740 1 --2- 172.21.15.118:0/3326954214 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x561eb216c9e0 0x561eb2176010 secure :-1 s=READY pgs=350 cs=0 l=1 rev1=1 crypto rx=0x7fb3d80097a0 tx=0x7fb3d8009bf0 comp rx=0 tx=0).stop 2024-03-24T00:18:36.584 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:36.584+0000 7fb3f3d96740 1 -- 172.21.15.118:0/3326954214 shutdown_connections 2024-03-24T00:18:36.584 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:36.584+0000 7fb3f3d96740 1 --2- 172.21.15.118:0/3326954214 >> [v2:172.21.15.189:6838/4074484168,v1:172.21.15.189:6839/4074484168] conn(0x561eb21812a0 0x561eb2183690 crc :-1 s=READY pgs=95 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:18:36.584 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:36.584+0000 7fb3f3d96740 1 --2- 172.21.15.118:0/3326954214 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x561eb216d3a0 0x561eb217d050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:18:36.584 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:36.584+0000 7fb3f3d96740 1 --2- 172.21.15.118:0/3326954214 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x561eb216c9e0 0x561eb2176010 unknown :-1 s=CLOSED pgs=350 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:18:36.584 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:36.584+0000 7fb3f3d96740 1 --2- 172.21.15.118:0/3326954214 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x561eb216b810 0x561eb2175ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:18:36.585 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:36.584+0000 7fb3cbe00640 0 client.26282 ms_handle_reset on v2:172.21.15.189:6838/4074484168 2024-03-24T00:18:36.585 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:36.584+0000 7fb3f3d96740 1 -- 172.21.15.118:0/3326954214 >> 172.21.15.118:0/3326954214 conn(0x561eb2166c00 msgr2=0x561eb2165eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:18:36.585 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:36.584+0000 7fb3f3d96740 1 -- 172.21.15.118:0/3326954214 shutdown_connections 2024-03-24T00:18:36.585 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:36.584+0000 7fb3f3d96740 1 -- 172.21.15.118:0/3326954214 wait complete. 2024-03-24T00:18:36.585 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:36.584+0000 7fb3f3d96740 20 client.26282 trim_cache size 0 max 16384 2024-03-24T00:18:36.598 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:36.599+0000 7fb3f3d96740 1 Processor -- start 2024-03-24T00:18:36.599 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:36.599+0000 7fb3f3d96740 1 -- start start 2024-03-24T00:18:36.599 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:36.599+0000 7fb3f3d96740 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x561eb216b810 0x561eb2165d30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:18:36.599 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:36.600+0000 7fb3f3d96740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x561eb216c9e0 0x561eb217d010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:18:36.599 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:36.600+0000 7fb3f3d96740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x561eb216d3a0 0x561eb2187f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:18:36.599 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:36.600+0000 7fb3f3d96740 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb3ec1a89d0 con 0x561eb216b810 2024-03-24T00:18:36.599 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:36.600+0000 7fb3f3d96740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb3ec1aa350 con 0x561eb216c9e0 2024-03-24T00:18:36.599 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:36.600+0000 7fb3f3d96740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb3ec1a9030 con 0x561eb216d3a0 2024-03-24T00:18:36.600 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:36.600+0000 7fb3ea800640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x561eb216d3a0 0x561eb2187f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:18:36.601 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:36.600+0000 7fb3ea800640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x561eb216d3a0 0x561eb2187f30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.118:53150/0 (socket says 172.21.15.118:53150) 2024-03-24T00:18:36.601 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:36.600+0000 7fb3ea800640 1 -- 172.21.15.118:0/3191021080 learned_addr learned my addr 172.21.15.118:0/3191021080 (peer_addr_for_me v2:172.21.15.118:0/0) 2024-03-24T00:18:36.601 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:36.600+0000 7fb3e9e00640 1 --2- 172.21.15.118:0/3191021080 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x561eb216b810 0x561eb2165d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:18:36.601 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:36.600+0000 7fb3e9400640 1 --2- 172.21.15.118:0/3191021080 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x561eb216c9e0 0x561eb217d010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:18:36.601 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:36.600+0000 7fb3ea800640 1 -- 172.21.15.118:0/3191021080 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x561eb216c9e0 msgr2=0x561eb217d010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:18:36.601 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:36.600+0000 7fb3ea800640 1 --2- 172.21.15.118:0/3191021080 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x561eb216c9e0 0x561eb217d010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:18:36.601 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:36.600+0000 7fb3ea800640 1 -- 172.21.15.118:0/3191021080 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x561eb216b810 msgr2=0x561eb2165d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:18:36.601 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:36.600+0000 7fb3ea800640 1 --2- 172.21.15.118:0/3191021080 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x561eb216b810 0x561eb2165d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:18:36.601 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:36.600+0000 7fb3ea800640 1 -- 172.21.15.118:0/3191021080 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x561eb2167970 con 0x561eb216d3a0 2024-03-24T00:18:36.601 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:36.600+0000 7fb3e9400640 1 --2- 172.21.15.118:0/3191021080 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x561eb216c9e0 0x561eb217d010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T00:18:36.602 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:36.601+0000 7fb3ea800640 1 --2- 172.21.15.118:0/3191021080 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x561eb216d3a0 0x561eb2187f30 secure :-1 s=READY pgs=351 cs=0 l=1 rev1=1 crypto rx=0x7fb3dc0ee5d0 tx=0x7fb3dc0037d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=3e6660806cfdbf7 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:18:36.602 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:36.601+0000 7fb3d6a00640 1 -- 172.21.15.118:0/3191021080 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb3dc0bce60 con 0x561eb216d3a0 2024-03-24T00:18:36.602 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:36.601+0000 7fb3d6a00640 1 -- 172.21.15.118:0/3191021080 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fb3dc004c80 con 0x561eb216d3a0 2024-03-24T00:18:36.602 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:36.601+0000 7fb3d6a00640 1 -- 172.21.15.118:0/3191021080 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb3dc0c5c10 con 0x561eb216d3a0 2024-03-24T00:18:36.604 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:36.601+0000 7fb3f3d96740 1 -- 172.21.15.118:0/3191021080 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x561eb216d3a0 msgr2=0x561eb2187f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:18:36.605 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:36.601+0000 7fb3f3d96740 1 --2- 172.21.15.118:0/3191021080 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x561eb216d3a0 0x561eb2187f30 secure :-1 s=READY pgs=351 cs=0 l=1 rev1=1 crypto rx=0x7fb3dc0ee5d0 tx=0x7fb3dc0037d0 comp rx=0 tx=0).stop 2024-03-24T00:18:36.605 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:36.602+0000 7fb3f3d96740 1 -- 172.21.15.118:0/3191021080 shutdown_connections 2024-03-24T00:18:36.605 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:36.602+0000 7fb3f3d96740 1 --2- 172.21.15.118:0/3191021080 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x561eb216d3a0 0x561eb2187f30 unknown :-1 s=CLOSED pgs=351 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:18:36.605 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:36.602+0000 7fb3f3d96740 1 --2- 172.21.15.118:0/3191021080 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x561eb216c9e0 0x561eb217d010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:18:36.605 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:36.602+0000 7fb3f3d96740 1 --2- 172.21.15.118:0/3191021080 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x561eb216b810 0x561eb2165d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:18:36.605 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:36.602+0000 7fb3f3d96740 1 -- 172.21.15.118:0/3191021080 >> 172.21.15.118:0/3191021080 conn(0x561eb2166c00 msgr2=0x561eb2165620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:18:36.605 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:36.602+0000 7fb3f3d96740 1 -- 172.21.15.118:0/3191021080 shutdown_connections 2024-03-24T00:18:36.605 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:36.602+0000 7fb3f3d96740 1 -- 172.21.15.118:0/3191021080 wait complete. 2024-03-24T00:18:36.605 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:36.602+0000 7fb3f3d96740 1 -- start start 2024-03-24T00:18:36.605 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:36.602+0000 7fb3f3d96740 1 Processor -- start 2024-03-24T00:18:36.605 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:36.603+0000 7fb3f3d96740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x561eb216b810 0x561eb217caa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:18:36.605 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:36.603+0000 7fb3f3d96740 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x561eb216c9e0 0x561eb217cfe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:18:36.606 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:36.603+0000 7fb3f3d96740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x561eb216d3a0 0x561eb217b180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:18:36.606 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:36.603+0000 7fb3f3d96740 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb3ec1a9cf0 con 0x561eb216c9e0 2024-03-24T00:18:36.606 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:36.603+0000 7fb3f3d96740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb3ec1a9690 con 0x561eb216b810 2024-03-24T00:18:36.606 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:36.603+0000 7fb3f3d96740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb3ec1a6b90 con 0x561eb216d3a0 2024-03-24T00:18:36.606 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:36.603+0000 7fb3e9e00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x561eb216b810 0x561eb217caa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:18:36.606 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:36.603+0000 7fb3e9e00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x561eb216b810 0x561eb217caa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.118:33002/0 (socket says 172.21.15.118:33002) 2024-03-24T00:18:36.606 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:36.603+0000 7fb3e9e00640 1 -- 172.21.15.118:0/514752531 learned_addr learned my addr 172.21.15.118:0/514752531 (peer_addr_for_me v2:172.21.15.118:0/0) 2024-03-24T00:18:36.606 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:36.603+0000 7fb3f3d96740 10 client.0 _ll_register_callbacks cb 0x7fb3e4009900 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T00:18:36.606 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:36.603+0000 7fb3ea800640 1 --2- 172.21.15.118:0/514752531 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x561eb216d3a0 0x561eb217b180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:18:36.606 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:36.603+0000 7fb3e9e00640 1 -- 172.21.15.118:0/514752531 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x561eb216d3a0 msgr2=0x561eb217b180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:18:36.606 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:36.603+0000 7fb3e9e00640 1 --2- 172.21.15.118:0/514752531 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x561eb216d3a0 0x561eb217b180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:18:36.606 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:36.603+0000 7fb3e9e00640 1 -- 172.21.15.118:0/514752531 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x561eb216c9e0 msgr2=0x561eb217cfe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:18:36.606 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:36.603+0000 7fb3e9e00640 1 --2- 172.21.15.118:0/514752531 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x561eb216c9e0 0x561eb217cfe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:18:36.607 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:36.603+0000 7fb3e9e00640 1 -- 172.21.15.118:0/514752531 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x561eb2167670 con 0x561eb216b810 2024-03-24T00:18:36.607 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:36.604+0000 7fb3e9e00640 1 --2- 172.21.15.118:0/514752531 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x561eb216b810 0x561eb217caa0 secure :-1 s=READY pgs=351 cs=0 l=1 rev1=1 crypto rx=0x7fb3e4009fa0 tx=0x7fb3e4004a40 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a98c6f1cdf76b07d server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:18:36.607 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:36.604+0000 7fb3cbe00640 10 client.0 ms_handle_connect on v2:172.21.15.189:3300/0 2024-03-24T00:18:36.607 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:36.604+0000 7fb3cbe00640 1 -- 172.21.15.118:0/514752531 <== mon.2 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb3e4004b20 con 0x561eb216b810 2024-03-24T00:18:36.607 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:36.604+0000 7fb3cbe00640 1 -- 172.21.15.118:0/514752531 <== mon.2 v2:172.21.15.189:3300/0 2 ==== osd_map(90..90 src has 1..90) v4 ==== 8277+0+0 (secure 0 0 0) 0x7fb3e4004cc0 con 0x561eb216b810 2024-03-24T00:18:36.607 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:36.604+0000 7fb3f3d96740 1 -- 172.21.15.118:0/514752531 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fb3dc004c80 con 0x561eb216b810 2024-03-24T00:18:36.607 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:36.604+0000 7fb3cbe00640 1 -- 172.21.15.118:0/514752531 <== mon.2 v2:172.21.15.189:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) v2 ==== 24+0+0 (secure 0 0 0) 0x7fb3e4039050 con 0x561eb216b810 2024-03-24T00:18:36.607 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:36.604+0000 7fb3f3d96740 10 client.26288 fetch_fsmap learned FSMap version 28 2024-03-24T00:18:36.607 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:36.604+0000 7fb3f3d96740 1 -- 172.21.15.118:0/514752531 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) v3 -- 0x561eb216c200 con 0x561eb216b810 2024-03-24T00:18:36.607 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:36.605+0000 7fb3cbe00640 1 -- 172.21.15.118:0/514752531 <== mon.2 v2:172.21.15.189:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fb3dc004c80 con 0x561eb216b810 2024-03-24T00:18:36.607 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:36.605+0000 7fb3cbe00640 1 -- 172.21.15.118:0/514752531 <== mon.2 v2:172.21.15.189:3300/0 5 ==== fsmap(e 28) v1 ==== 7934+0+0 (secure 0 0 0) 0x7fb3e4005580 con 0x561eb216b810 2024-03-24T00:18:36.607 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:36.605+0000 7fb3f3d96740 10 client.26288 fetch_fsmap finished waiting for FSMap version 28 2024-03-24T00:18:36.607 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:36.605+0000 7fb3f3d96740 10 client.26288 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-24T00:18:36.608 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:36.605+0000 7fb3f3d96740 20 client.26288 populate_metadata read hostname 'smithi118' 2024-03-24T00:18:36.608 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:36.605+0000 7fb3f3d96740 1 --2- 172.21.15.118:0/514752531 >> [v2:172.21.15.189:6838/4074484168,v1:172.21.15.189:6839/4074484168] conn(0x561eb21dbd40 0x561eb21de160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:18:36.608 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:36.605+0000 7fb3f3d96740 4 client.26288 mds_command: new command op to 24569 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-24T00:18:36.608 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:36.605+0000 7fb3f3d96740 1 -- 172.21.15.118:0/514752531 --> [v2:172.21.15.189:6838/4074484168,v1:172.21.15.189:6839/4074484168] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x561eb20d7b50 con 0x561eb21dbd40 2024-03-24T00:18:36.608 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:36.605+0000 7fb3e9400640 1 --2- 172.21.15.118:0/514752531 >> [v2:172.21.15.189:6838/4074484168,v1:172.21.15.189:6839/4074484168] conn(0x561eb21dbd40 0x561eb21de160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:18:36.608 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:36.606+0000 7fb3e9400640 1 --2- 172.21.15.118:0/514752531 >> [v2:172.21.15.189:6838/4074484168,v1:172.21.15.189:6839/4074484168] conn(0x561eb21dbd40 0x561eb21de160 crc :-1 s=READY pgs=96 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=b798772c1710c03b server_cookie=56064f97988ee721 in_seq=0 out_seq=0 2024-03-24T00:18:36.608 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:36.606+0000 7fb3cbe00640 10 client.26288 ms_handle_connect on v2:172.21.15.189:6838/4074484168 2024-03-24T00:18:36.611 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:36.611+0000 7fb3cbe00640 1 -- 172.21.15.118:0/514752531 <== mds.0 v2:172.21.15.189:6838/4074484168 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x561eb20d7b50 con 0x561eb21dbd40 2024-03-24T00:18:36.611 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:36.611+0000 7fb3cbe00640 10 client.26288 handle_command_reply: tid=0 2024-03-24T00:18:36.611 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:36.611+0000 7fb3f3d96740 1 client.26288 shutdown 2024-03-24T00:18:36.611 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:36.611+0000 7fb3f3d96740 1 -- 172.21.15.118:0/514752531 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x561eb216b810 msgr2=0x561eb217caa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:18:36.611 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:36.611+0000 7fb3f3d96740 1 --2- 172.21.15.118:0/514752531 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x561eb216b810 0x561eb217caa0 secure :-1 s=READY pgs=351 cs=0 l=1 rev1=1 crypto rx=0x7fb3e4009fa0 tx=0x7fb3e4004a40 comp rx=0 tx=0).stop 2024-03-24T00:18:36.611 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:36.611+0000 7fb3f3d96740 1 -- 172.21.15.118:0/514752531 shutdown_connections 2024-03-24T00:18:36.611 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:36.611+0000 7fb3f3d96740 1 --2- 172.21.15.118:0/514752531 >> [v2:172.21.15.189:6838/4074484168,v1:172.21.15.189:6839/4074484168] conn(0x561eb21dbd40 0x561eb21de160 crc :-1 s=READY pgs=96 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:18:36.611 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:36.611+0000 7fb3f3d96740 1 --2- 172.21.15.118:0/514752531 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x561eb216d3a0 0x561eb217b180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:18:36.611 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:36.611+0000 7fb3f3d96740 1 --2- 172.21.15.118:0/514752531 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x561eb216c9e0 0x561eb217cfe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:18:36.611 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:36.611+0000 7fb3cbe00640 0 client.26288 ms_handle_reset on v2:172.21.15.189:6838/4074484168 2024-03-24T00:18:36.611 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:36.611+0000 7fb3f3d96740 1 --2- 172.21.15.118:0/514752531 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x561eb216b810 0x561eb217caa0 unknown :-1 s=CLOSED pgs=351 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:18:36.612 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:36.611+0000 7fb3f3d96740 1 -- 172.21.15.118:0/514752531 >> 172.21.15.118:0/514752531 conn(0x561eb2166c00 msgr2=0x561eb2165eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:18:36.612 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:36.611+0000 7fb3f3d96740 1 -- 172.21.15.118:0/514752531 shutdown_connections 2024-03-24T00:18:36.612 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:36.611+0000 7fb3f3d96740 1 -- 172.21.15.118:0/514752531 wait complete. 2024-03-24T00:18:36.612 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:36.611+0000 7fb3f3d96740 20 client.26288 trim_cache size 0 max 16384 2024-03-24T00:18:36.612 INFO:teuthology.orchestra.run.smithi118.stdout:[] 2024-03-24T00:18:36.613 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:36.612+0000 7fb3f1000640 1 -- 172.21.15.118:0/1047641023 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7fb3d00a2800 msgr2=0x7fb3d00a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:18:36.613 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:36.612+0000 7fb3f1000640 1 --2- 172.21.15.118:0/1047641023 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7fb3d00a2800 0x7fb3d00a4cc0 secure :-1 s=READY pgs=325 cs=0 l=1 rev1=1 crypto rx=0x7fb3e4009fd0 tx=0x7fb3e4009500 comp rx=0 tx=0).stop 2024-03-24T00:18:36.613 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:36.612+0000 7fb3f1000640 1 -- 172.21.15.118:0/1047641023 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb3ec103260 msgr2=0x7fb3ec19ef20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:18:36.613 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:36.612+0000 7fb3f1000640 1 --2- 172.21.15.118:0/1047641023 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb3ec103260 0x7fb3ec19ef20 secure :-1 s=READY pgs=349 cs=0 l=1 rev1=1 crypto rx=0x7fb3dc02c8e0 tx=0x7fb3dc02c640 comp rx=0 tx=0).stop 2024-03-24T00:18:36.613 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:36.612+0000 7fb3f1000640 1 -- 172.21.15.118:0/1047641023 shutdown_connections 2024-03-24T00:18:36.613 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:36.612+0000 7fb3f1000640 1 --2- 172.21.15.118:0/1047641023 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7fb3d00a2800 0x7fb3d00a4cc0 unknown :-1 s=CLOSED pgs=325 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:18:36.614 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:36.612+0000 7fb3f1000640 1 --2- 172.21.15.118:0/1047641023 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb3ec103260 0x7fb3ec19ef20 unknown :-1 s=CLOSED pgs=349 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:18:36.614 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:36.612+0000 7fb3f1000640 1 --2- 172.21.15.118:0/1047641023 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb3ec1028a0 0x7fb3ec197eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:18:36.614 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:36.612+0000 7fb3f1000640 1 --2- 172.21.15.118:0/1047641023 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fb3ec1016b0 0x7fb3ec197970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:18:36.614 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:36.612+0000 7fb3f1000640 1 -- 172.21.15.118:0/1047641023 >> 172.21.15.118:0/1047641023 conn(0x7fb3ec0fcc40 msgr2=0x7fb3ec105ba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:18:36.614 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:36.612+0000 7fb3f1000640 1 -- 172.21.15.118:0/1047641023 shutdown_connections 2024-03-24T00:18:36.614 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:36.612+0000 7fb3f1000640 1 -- 172.21.15.118:0/1047641023 wait complete. 2024-03-24T00:18:36.835 INFO:tasks.workunit.client.0.smithi118.stdout: 14 31250 125 18954 170 10564 327 2024-03-24T00:18:36.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:18:36 smithi118 ceph-mon[31535]: pgmap v485: 97 pgs: 97 active+clean; 49 MiB data, 491 MiB used, 1.0 TiB / 1.0 TiB avail; 703 KiB/s rd, 2.1 MiB/s wr, 370 op/s 2024-03-24T00:18:37.203 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:18:36 smithi189 ceph-mon[36187]: pgmap v485: 97 pgs: 97 active+clean; 49 MiB data, 491 MiB used, 1.0 TiB / 1.0 TiB avail; 703 KiB/s rd, 2.1 MiB/s wr, 370 op/s 2024-03-24T00:18:37.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:18:36 smithi193 ceph-mon[36878]: pgmap v485: 97 pgs: 97 active+clean; 49 MiB data, 491 MiB used, 1.0 TiB / 1.0 TiB avail; 703 KiB/s rd, 2.1 MiB/s wr, 370 op/s 2024-03-24T00:18:37.629 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-03-24T00:18:37.629 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T00:18:37.629 DEBUG:teuthology.orchestra.run.smithi118:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2024-03-24T00:18:37.868 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:37.867+0000 7f37f9000640 1 -- 172.21.15.118:0/2410984001 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f37f40fff70 msgr2=0x7f37f4100390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:18:37.868 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:37.867+0000 7f37f9000640 1 --2- 172.21.15.118:0/2410984001 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f37f40fff70 0x7f37f4100390 secure :-1 s=READY pgs=352 cs=0 l=1 rev1=1 crypto rx=0x7f37e4009900 tx=0x7f37e402bc40 comp rx=0 tx=0).stop 2024-03-24T00:18:37.868 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:37.868+0000 7f37f9000640 1 -- 172.21.15.118:0/2410984001 shutdown_connections 2024-03-24T00:18:37.868 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:37.868+0000 7f37f9000640 1 --2- 172.21.15.118:0/2410984001 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f37f4101270 0x7f37f41016f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:18:37.868 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:37.868+0000 7f37f9000640 1 --2- 172.21.15.118:0/2410984001 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f37f41008d0 0x7f37f4100d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:18:37.868 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:37.868+0000 7f37f9000640 1 --2- 172.21.15.118:0/2410984001 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f37f40fff70 0x7f37f4100390 unknown :-1 s=CLOSED pgs=352 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:18:37.868 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:37.868+0000 7f37f9000640 1 -- 172.21.15.118:0/2410984001 >> 172.21.15.118:0/2410984001 conn(0x7f37f40fbb30 msgr2=0x7f37f40fdf50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:18:37.868 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:37.868+0000 7f37f9000640 1 -- 172.21.15.118:0/2410984001 shutdown_connections 2024-03-24T00:18:37.868 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:37.868+0000 7f37f9000640 1 -- 172.21.15.118:0/2410984001 wait complete. 2024-03-24T00:18:37.870 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:37.869+0000 7f37f9000640 1 Processor -- start 2024-03-24T00:18:37.870 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:37.869+0000 7f37f9000640 1 -- start start 2024-03-24T00:18:37.870 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:37.869+0000 7f37f9000640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f37f40fff70 0x7f37f4199ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:18:37.871 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:37.869+0000 7f37f9000640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f37f41008d0 0x7f37f419a0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:18:37.871 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:37.869+0000 7f37f9000640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f37f4101270 0x7f37f41a1150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:18:37.871 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:37.869+0000 7f37f9000640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f37f410f530 con 0x7f37f41008d0 2024-03-24T00:18:37.871 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:37.869+0000 7f37f9000640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f37f410f3b0 con 0x7f37f40fff70 2024-03-24T00:18:37.871 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:37.869+0000 7f37f9000640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f37f410f6b0 con 0x7f37f4101270 2024-03-24T00:18:37.871 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:37.869+0000 7f37f1e00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f37f40fff70 0x7f37f4199ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:18:37.871 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:37.869+0000 7f37f1e00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f37f40fff70 0x7f37f4199ba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.118:33008/0 (socket says 172.21.15.118:33008) 2024-03-24T00:18:37.871 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:37.869+0000 7f37f1e00640 1 -- 172.21.15.118:0/2800749896 learned_addr learned my addr 172.21.15.118:0/2800749896 (peer_addr_for_me v2:172.21.15.118:0/0) 2024-03-24T00:18:37.871 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:37.869+0000 7f37f2800640 1 --2- 172.21.15.118:0/2800749896 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f37f4101270 0x7f37f41a1150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:18:37.871 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:37.869+0000 7f37f1400640 1 --2- 172.21.15.118:0/2800749896 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f37f41008d0 0x7f37f419a0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:18:37.871 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:37.870+0000 7f37f1e00640 1 -- 172.21.15.118:0/2800749896 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f37f4101270 msgr2=0x7f37f41a1150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:18:37.871 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:37.870+0000 7f37f1e00640 1 --2- 172.21.15.118:0/2800749896 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f37f4101270 0x7f37f41a1150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:18:37.872 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:37.870+0000 7f37f1e00640 1 -- 172.21.15.118:0/2800749896 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f37f41008d0 msgr2=0x7f37f419a0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:18:37.872 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:37.870+0000 7f37f1e00640 1 --2- 172.21.15.118:0/2800749896 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f37f41008d0 0x7f37f419a0e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:18:37.872 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:37.870+0000 7f37f1e00640 1 -- 172.21.15.118:0/2800749896 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f37f41a17c0 con 0x7f37f40fff70 2024-03-24T00:18:37.872 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:37.870+0000 7f37f1400640 1 --2- 172.21.15.118:0/2800749896 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f37f41008d0 0x7f37f419a0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T00:18:37.872 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:37.870+0000 7f37f1e00640 1 --2- 172.21.15.118:0/2800749896 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f37f40fff70 0x7f37f4199ba0 secure :-1 s=READY pgs=353 cs=0 l=1 rev1=1 crypto rx=0x7f37e402c8f0 tx=0x7f37e402c660 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:18:37.872 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:37.870+0000 7f37eaa00640 1 -- 172.21.15.118:0/2800749896 <== mon.2 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f37e4041070 con 0x7f37f40fff70 2024-03-24T00:18:37.872 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:37.870+0000 7f37f9000640 1 -- 172.21.15.118:0/2800749896 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f37f41a1a50 con 0x7f37f40fff70 2024-03-24T00:18:37.872 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:37.870+0000 7f37eaa00640 1 -- 172.21.15.118:0/2800749896 <== mon.2 v2:172.21.15.189:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f37e4004450 con 0x7f37f40fff70 2024-03-24T00:18:37.872 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:37.870+0000 7f37eaa00640 1 -- 172.21.15.118:0/2800749896 <== mon.2 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f37e4034440 con 0x7f37f40fff70 2024-03-24T00:18:37.872 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:37.871+0000 7f37f9000640 1 -- 172.21.15.118:0/2800749896 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f37f41a1f40 con 0x7f37f40fff70 2024-03-24T00:18:37.873 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:37.874+0000 7f37fbddc740 1 Processor -- start 2024-03-24T00:18:37.873 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:37.874+0000 7f37fbddc740 1 -- start start 2024-03-24T00:18:37.875 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:37.874+0000 7f37fbddc740 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x56134ca4f0a0 0x56134ca4f4c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:18:37.875 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:37.874+0000 7f37fbddc740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56134ca4fa90 0x56134ca54590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:18:37.875 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:37.874+0000 7f37fbddc740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56134ca54c00 0x56134ca57090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:18:37.875 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:37.874+0000 7f37fbddc740 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x56134c9b4eb0 con 0x56134ca4f0a0 2024-03-24T00:18:37.875 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:37.874+0000 7f37fbddc740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x56134c95da60 con 0x56134ca4fa90 2024-03-24T00:18:37.875 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:37.874+0000 7f37fbddc740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x56134c9b0f50 con 0x56134ca54c00 2024-03-24T00:18:37.875 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:37.874+0000 7f37eaa00640 1 -- 172.21.15.118:0/2800749896 <== mon.2 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 20) v1 ==== 137291+0+0 (secure 0 0 0) 0x7f37e4044070 con 0x7f37f40fff70 2024-03-24T00:18:37.876 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:37.875+0000 7f37f2800640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x56134ca4f0a0 0x56134ca4f4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:18:37.876 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:37.875+0000 7f37f1400640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56134ca54c00 0x56134ca57090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:18:37.876 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:37.875+0000 7f37f1400640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56134ca54c00 0x56134ca57090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.118:53202/0 (socket says 172.21.15.118:53202) 2024-03-24T00:18:37.876 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:37.875+0000 7f37f1400640 1 -- 172.21.15.118:0/3187259473 learned_addr learned my addr 172.21.15.118:0/3187259473 (peer_addr_for_me v2:172.21.15.118:0/0) 2024-03-24T00:18:37.876 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:37.875+0000 7f37f1e00640 1 --2- 172.21.15.118:0/3187259473 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56134ca4fa90 0x56134ca54590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:18:37.876 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:37.875+0000 7f37f2800640 1 -- 172.21.15.118:0/3187259473 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56134ca54c00 msgr2=0x56134ca57090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:18:37.876 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:37.875+0000 7f37f2800640 1 --2- 172.21.15.118:0/3187259473 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56134ca54c00 0x56134ca57090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:18:37.876 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:37.875+0000 7f37f2800640 1 -- 172.21.15.118:0/3187259473 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56134ca4fa90 msgr2=0x56134ca54590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:18:37.876 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:37.875+0000 7f37f2800640 1 --2- 172.21.15.118:0/3187259473 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56134ca4fa90 0x56134ca54590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:18:37.876 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:37.875+0000 7f37f2800640 1 -- 172.21.15.118:0/3187259473 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x56134c7917e0 con 0x56134ca4f0a0 2024-03-24T00:18:37.876 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:37.875+0000 7f37f2800640 1 --2- 172.21.15.118:0/3187259473 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x56134ca4f0a0 0x56134ca4f4c0 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7f37ec009f00 tx=0x7f37ec011810 comp rx=0 tx=0).ready entity=mon.0 client_cookie=349e56943c13bdb3 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:18:37.877 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:37.876+0000 7f37df400640 1 -- 172.21.15.118:0/3187259473 <== mon.0 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f37ec004d20 con 0x56134ca4f0a0 2024-03-24T00:18:37.877 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:37.876+0000 7f37df400640 1 -- 172.21.15.118:0/3187259473 <== mon.0 v2:172.21.15.118:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f37ec004030 con 0x56134ca4f0a0 2024-03-24T00:18:37.877 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:37.876+0000 7f37df400640 1 -- 172.21.15.118:0/3187259473 <== mon.0 v2:172.21.15.118:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f37ec005280 con 0x56134ca4f0a0 2024-03-24T00:18:37.881 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:37.876+0000 7f37eaa00640 1 --2- 172.21.15.118:0/2800749896 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7f37d80a2800 0x7f37d80a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:18:37.881 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:37.876+0000 7f37eaa00640 1 -- 172.21.15.118:0/2800749896 <== mon.2 v2:172.21.15.189:3300/0 5 ==== osd_map(90..90 src has 1..90) v4 ==== 8277+0+0 (secure 0 0 0) 0x7f37e40f0710 con 0x7f37f40fff70 2024-03-24T00:18:37.881 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:37.876+0000 7f37f1e00640 1 --2- 172.21.15.118:0/2800749896 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7f37d80a2800 0x7f37d80a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:18:37.881 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:37.876+0000 7f37fbddc740 1 -- 172.21.15.118:0/3187259473 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x56134ca4f0a0 msgr2=0x56134ca4f4c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:18:37.881 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:37.876+0000 7f37fbddc740 1 --2- 172.21.15.118:0/3187259473 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x56134ca4f0a0 0x56134ca4f4c0 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7f37ec009f00 tx=0x7f37ec011810 comp rx=0 tx=0).stop 2024-03-24T00:18:37.881 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:37.877+0000 7f37fbddc740 1 -- 172.21.15.118:0/3187259473 shutdown_connections 2024-03-24T00:18:37.881 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:37.877+0000 7f37fbddc740 1 --2- 172.21.15.118:0/3187259473 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56134ca54c00 0x56134ca57090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:18:37.881 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:37.877+0000 7f37fbddc740 1 --2- 172.21.15.118:0/3187259473 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56134ca4fa90 0x56134ca54590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:18:37.882 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:37.877+0000 7f37fbddc740 1 --2- 172.21.15.118:0/3187259473 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x56134ca4f0a0 0x56134ca4f4c0 unknown :-1 s=CLOSED pgs=200 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:18:37.882 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:37.877+0000 7f37fbddc740 1 -- 172.21.15.118:0/3187259473 >> 172.21.15.118:0/3187259473 conn(0x56134ca4a610 msgr2=0x56134ca4aa10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:18:37.882 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:37.877+0000 7f37fbddc740 1 -- 172.21.15.118:0/3187259473 shutdown_connections 2024-03-24T00:18:37.882 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:37.877+0000 7f37fbddc740 1 -- 172.21.15.118:0/3187259473 wait complete. 2024-03-24T00:18:37.882 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:37.877+0000 7f37f1e00640 1 --2- 172.21.15.118:0/2800749896 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7f37d80a2800 0x7f37d80a4cc0 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 crypto rx=0x7f37e40b9480 tx=0x7f37e40dd040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:18:37.882 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:37.877+0000 7f37fbddc740 1 -- start start 2024-03-24T00:18:37.882 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:37.878+0000 7f37fbddc740 1 Processor -- start 2024-03-24T00:18:37.882 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:37.878+0000 7f37fbddc740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56134ca4f0a0 0x56134ca59220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:18:37.882 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:37.878+0000 7f37fbddc740 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x56134ca4fa90 0x56134ca59760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:18:37.882 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:37.878+0000 7f37fbddc740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56134ca54c00 0x56134ca607a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:18:37.882 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:37.878+0000 7f37fbddc740 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x56134ca57bb0 con 0x56134ca4fa90 2024-03-24T00:18:37.883 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:37.878+0000 7f37fbddc740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x56134c970eb0 con 0x56134ca4f0a0 2024-03-24T00:18:37.883 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:37.878+0000 7f37fbddc740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x56134ca57d30 con 0x56134ca54c00 2024-03-24T00:18:37.883 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:37.878+0000 7f37fbddc740 10 client.0 _ll_register_callbacks cb 0x56134c965150 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T00:18:37.883 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:37.878+0000 7f37f2800640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56134ca4f0a0 0x56134ca59220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:18:37.883 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:37.878+0000 7f37f2800640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56134ca54c00 0x56134ca607a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:18:37.883 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:37.878+0000 7f37f1400640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x56134ca4fa90 0x56134ca59760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:18:37.883 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:37.878+0000 7f37f1400640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x56134ca4fa90 0x56134ca59760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.118:3300/0 says I am v2:172.21.15.118:38240/0 (socket says 172.21.15.118:38240) 2024-03-24T00:18:37.883 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:37.878+0000 7f37f2800640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56134ca4f0a0 0x56134ca59220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.118:33026/0 (socket says 172.21.15.118:33026) 2024-03-24T00:18:37.883 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:37.878+0000 7f37f1400640 1 -- 172.21.15.118:0/4279753497 learned_addr learned my addr 172.21.15.118:0/4279753497 (peer_addr_for_me v2:172.21.15.118:0/0) 2024-03-24T00:18:37.883 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:37.879+0000 7f37f1400640 1 -- 172.21.15.118:0/4279753497 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56134ca54c00 msgr2=0x56134ca607a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:18:37.883 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:37.879+0000 7f37f1400640 1 --2- 172.21.15.118:0/4279753497 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56134ca54c00 0x56134ca607a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:18:37.884 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:37.879+0000 7f37f1400640 1 -- 172.21.15.118:0/4279753497 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56134ca4f0a0 msgr2=0x56134ca59220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:18:37.884 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:37.879+0000 7f37f1400640 1 --2- 172.21.15.118:0/4279753497 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56134ca4f0a0 0x56134ca59220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:18:37.884 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:37.879+0000 7f37f1400640 1 -- 172.21.15.118:0/4279753497 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x56134c7f1f70 con 0x56134ca4fa90 2024-03-24T00:18:37.884 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:37.879+0000 7f37f2800640 1 --2- 172.21.15.118:0/4279753497 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56134ca4f0a0 0x56134ca59220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T00:18:37.884 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:37.879+0000 7f37f1400640 1 --2- 172.21.15.118:0/4279753497 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x56134ca4fa90 0x56134ca59760 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7f37e0012700 tx=0x7f37e0012bc0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1a2c5980429efd25 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:18:37.884 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:37.879+0000 7f37d3e00640 10 client.0 ms_handle_connect on v2:172.21.15.118:3300/0 2024-03-24T00:18:37.884 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:37.879+0000 7f37d3e00640 1 -- 172.21.15.118:0/4279753497 <== mon.0 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f37e0004260 con 0x56134ca4fa90 2024-03-24T00:18:37.884 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:37.879+0000 7f37d3e00640 1 -- 172.21.15.118:0/4279753497 <== mon.0 v2:172.21.15.118:3300/0 2 ==== osd_map(90..90 src has 1..90) v4 ==== 8277+0+0 (secure 0 0 0) 0x7f37e0004400 con 0x56134ca4fa90 2024-03-24T00:18:37.884 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:37.879+0000 7f37fbddc740 1 -- 172.21.15.118:0/4279753497 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f37ec004030 con 0x56134ca4fa90 2024-03-24T00:18:37.884 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:37.880+0000 7f37d3e00640 1 -- 172.21.15.118:0/4279753497 <== mon.0 v2:172.21.15.118:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) v2 ==== 24+0+0 (secure 0 0 0) 0x7f37e0011940 con 0x56134ca4fa90 2024-03-24T00:18:37.884 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:37.880+0000 7f37fbddc740 10 client.16149 fetch_fsmap learned FSMap version 28 2024-03-24T00:18:37.885 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:37.880+0000 7f37fbddc740 1 -- 172.21.15.118:0/4279753497 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) v3 -- 0x56134c689170 con 0x56134ca4fa90 2024-03-24T00:18:37.885 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:37.880+0000 7f37d3e00640 1 -- 172.21.15.118:0/4279753497 <== mon.0 v2:172.21.15.118:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f37ec004030 con 0x56134ca4fa90 2024-03-24T00:18:37.885 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:37.880+0000 7f37d3e00640 1 -- 172.21.15.118:0/4279753497 <== mon.0 v2:172.21.15.118:3300/0 5 ==== fsmap(e 28) v1 ==== 7934+0+0 (secure 0 0 0) 0x7f37e0025070 con 0x56134ca4fa90 2024-03-24T00:18:37.885 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:37.880+0000 7f37fbddc740 10 client.16149 fetch_fsmap finished waiting for FSMap version 28 2024-03-24T00:18:37.885 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:37.880+0000 7f37fbddc740 10 client.16149 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-24T00:18:37.885 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:37.880+0000 7f37fbddc740 20 client.16149 populate_metadata read hostname 'smithi118' 2024-03-24T00:18:37.885 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:37.880+0000 7f37fbddc740 1 --2- 172.21.15.118:0/4279753497 >> [v2:172.21.15.189:6838/4074484168,v1:172.21.15.189:6839/4074484168] conn(0x56134ca650f0 0x56134ca67570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:18:37.885 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:37.880+0000 7f37fbddc740 4 client.16149 mds_command: new command op to 24569 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-24T00:18:37.885 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:37.880+0000 7f37fbddc740 1 -- 172.21.15.118:0/4279753497 --> [v2:172.21.15.189:6838/4074484168,v1:172.21.15.189:6839/4074484168] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x56134c9c23f0 con 0x56134ca650f0 2024-03-24T00:18:37.885 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:37.881+0000 7f37f2800640 1 --2- 172.21.15.118:0/4279753497 >> [v2:172.21.15.189:6838/4074484168,v1:172.21.15.189:6839/4074484168] conn(0x56134ca650f0 0x56134ca67570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:18:37.885 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:37.881+0000 7f37f2800640 1 --2- 172.21.15.118:0/4279753497 >> [v2:172.21.15.189:6838/4074484168,v1:172.21.15.189:6839/4074484168] conn(0x56134ca650f0 0x56134ca67570 crc :-1 s=READY pgs=97 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=63fcf5d058ea2cee server_cookie=608f47fe3b4374b6 in_seq=0 out_seq=0 2024-03-24T00:18:37.885 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:37.881+0000 7f37d3e00640 10 client.16149 ms_handle_connect on v2:172.21.15.189:6838/4074484168 2024-03-24T00:18:37.887 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:37.885+0000 7f37d3e00640 1 -- 172.21.15.118:0/4279753497 <== mds.0 v2:172.21.15.189:6838/4074484168 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x56134c9c23f0 con 0x56134ca650f0 2024-03-24T00:18:37.887 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:37.885+0000 7f37d3e00640 10 client.16149 handle_command_reply: tid=0 2024-03-24T00:18:37.887 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:37.885+0000 7f37fbddc740 1 client.16149 shutdown 2024-03-24T00:18:37.887 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:37.885+0000 7f37fbddc740 1 -- 172.21.15.118:0/4279753497 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x56134ca4fa90 msgr2=0x56134ca59760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:18:37.887 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:37.885+0000 7f37fbddc740 1 --2- 172.21.15.118:0/4279753497 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x56134ca4fa90 0x56134ca59760 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7f37e0012700 tx=0x7f37e0012bc0 comp rx=0 tx=0).stop 2024-03-24T00:18:37.887 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:37.885+0000 7f37fbddc740 1 -- 172.21.15.118:0/4279753497 shutdown_connections 2024-03-24T00:18:37.887 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:37.885+0000 7f37fbddc740 1 --2- 172.21.15.118:0/4279753497 >> [v2:172.21.15.189:6838/4074484168,v1:172.21.15.189:6839/4074484168] conn(0x56134ca650f0 0x56134ca67570 crc :-1 s=READY pgs=97 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:18:37.887 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:37.885+0000 7f37fbddc740 1 --2- 172.21.15.118:0/4279753497 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56134ca54c00 0x56134ca607a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:18:37.887 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:37.885+0000 7f37fbddc740 1 --2- 172.21.15.118:0/4279753497 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x56134ca4fa90 0x56134ca59760 unknown :-1 s=CLOSED pgs=201 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:18:37.887 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:37.885+0000 7f37fbddc740 1 --2- 172.21.15.118:0/4279753497 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56134ca4f0a0 0x56134ca59220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:18:37.888 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:37.885+0000 7f37d3e00640 0 client.16149 ms_handle_reset on v2:172.21.15.189:6838/4074484168 2024-03-24T00:18:37.888 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:37.885+0000 7f37fbddc740 1 -- 172.21.15.118:0/4279753497 >> 172.21.15.118:0/4279753497 conn(0x56134ca4a610 msgr2=0x56134ca49830 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:18:37.888 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:37.885+0000 7f37fbddc740 1 -- 172.21.15.118:0/4279753497 shutdown_connections 2024-03-24T00:18:37.888 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:37.886+0000 7f37fbddc740 1 -- 172.21.15.118:0/4279753497 wait complete. 2024-03-24T00:18:37.888 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:37.886+0000 7f37fbddc740 20 client.16149 trim_cache size 0 max 16384 2024-03-24T00:18:37.905 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:37.904+0000 7f37fbddc740 1 Processor -- start 2024-03-24T00:18:37.905 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:37.904+0000 7f37fbddc740 1 -- start start 2024-03-24T00:18:37.905 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:37.904+0000 7f37fbddc740 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x56134ca4f0a0 0x56134ca50530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:18:37.905 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:37.905+0000 7f37fbddc740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56134ca4fa90 0x56134ca50a70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:18:37.905 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:37.905+0000 7f37fbddc740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56134ca54c00 0x56134ca50fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:18:37.905 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:37.905+0000 7f37fbddc740 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f37f41aac30 con 0x56134ca4f0a0 2024-03-24T00:18:37.906 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:37.905+0000 7f37fbddc740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f37f41ac5b0 con 0x56134ca54c00 2024-03-24T00:18:37.906 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:37.905+0000 7f37fbddc740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f37f41ab290 con 0x56134ca4fa90 2024-03-24T00:18:37.906 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:37.905+0000 7f37f1400640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56134ca4fa90 0x56134ca50a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:18:37.906 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:37.905+0000 7f37f2800640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x56134ca4f0a0 0x56134ca50530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:18:37.906 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:37.905+0000 7f37f1400640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56134ca4fa90 0x56134ca50a70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.118:53228/0 (socket says 172.21.15.118:53228) 2024-03-24T00:18:37.906 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:37.905+0000 7f37f1400640 1 -- 172.21.15.118:0/4184890090 learned_addr learned my addr 172.21.15.118:0/4184890090 (peer_addr_for_me v2:172.21.15.118:0/0) 2024-03-24T00:18:37.906 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:37.905+0000 7f37f2800640 1 --2- 172.21.15.118:0/4184890090 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56134ca54c00 0x56134ca50fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:18:37.906 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:37.905+0000 7f37f1400640 1 -- 172.21.15.118:0/4184890090 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56134ca54c00 msgr2=0x56134ca50fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:18:37.906 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:37.905+0000 7f37f1400640 1 --2- 172.21.15.118:0/4184890090 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56134ca54c00 0x56134ca50fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:18:37.906 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:37.905+0000 7f37f1400640 1 -- 172.21.15.118:0/4184890090 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x56134ca4f0a0 msgr2=0x56134ca50530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:18:37.906 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:37.905+0000 7f37f1400640 1 --2- 172.21.15.118:0/4184890090 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x56134ca4f0a0 0x56134ca50530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:18:37.907 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:37.905+0000 7f37f1400640 1 -- 172.21.15.118:0/4184890090 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x56134ca64ed0 con 0x56134ca4fa90 2024-03-24T00:18:37.907 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:37.905+0000 7f37f2800640 1 --2- 172.21.15.118:0/4184890090 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x56134ca4f0a0 0x56134ca50530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T00:18:37.907 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:37.906+0000 7f37f1400640 1 --2- 172.21.15.118:0/4184890090 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56134ca4fa90 0x56134ca50a70 secure :-1 s=READY pgs=352 cs=0 l=1 rev1=1 crypto rx=0x7f37e00092e0 tx=0x7f37e0009760 comp rx=0 tx=0).ready entity=mon.1 client_cookie=f19ee125a5270f51 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:18:37.910 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:37.906+0000 7f37dea00640 1 -- 172.21.15.118:0/4184890090 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f37e00052d0 con 0x56134ca4fa90 2024-03-24T00:18:37.910 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:37.906+0000 7f37dea00640 1 -- 172.21.15.118:0/4184890090 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f37e0022070 con 0x56134ca4fa90 2024-03-24T00:18:37.910 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:37.906+0000 7f37dea00640 1 -- 172.21.15.118:0/4184890090 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f37e000f6b0 con 0x56134ca4fa90 2024-03-24T00:18:37.910 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:37.907+0000 7f37fbddc740 1 -- 172.21.15.118:0/4184890090 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56134ca4fa90 msgr2=0x56134ca50a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:18:37.911 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:37.907+0000 7f37fbddc740 1 --2- 172.21.15.118:0/4184890090 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56134ca4fa90 0x56134ca50a70 secure :-1 s=READY pgs=352 cs=0 l=1 rev1=1 crypto rx=0x7f37e00092e0 tx=0x7f37e0009760 comp rx=0 tx=0).stop 2024-03-24T00:18:37.911 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:37.907+0000 7f37fbddc740 1 -- 172.21.15.118:0/4184890090 shutdown_connections 2024-03-24T00:18:37.911 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:37.907+0000 7f37fbddc740 1 --2- 172.21.15.118:0/4184890090 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56134ca54c00 0x56134ca50fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:18:37.911 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:37.907+0000 7f37fbddc740 1 --2- 172.21.15.118:0/4184890090 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56134ca4fa90 0x56134ca50a70 unknown :-1 s=CLOSED pgs=352 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:18:37.911 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:37.907+0000 7f37fbddc740 1 --2- 172.21.15.118:0/4184890090 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x56134ca4f0a0 0x56134ca50530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:18:37.911 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:37.907+0000 7f37fbddc740 1 -- 172.21.15.118:0/4184890090 >> 172.21.15.118:0/4184890090 conn(0x56134ca4a610 msgr2=0x56134ca4e100 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:18:37.911 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:37.907+0000 7f37fbddc740 1 -- 172.21.15.118:0/4184890090 shutdown_connections 2024-03-24T00:18:37.911 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:37.907+0000 7f37fbddc740 1 -- 172.21.15.118:0/4184890090 wait complete. 2024-03-24T00:18:37.911 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:37.907+0000 7f37fbddc740 1 -- start start 2024-03-24T00:18:37.911 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:37.908+0000 7f37fbddc740 1 Processor -- start 2024-03-24T00:18:37.911 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:37.908+0000 7f37fbddc740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56134ca4f0a0 0x56134ca5e7e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:18:37.911 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:37.908+0000 7f37fbddc740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56134ca4fa90 0x56134ca5ed20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:18:37.912 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:37.908+0000 7f37fbddc740 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x56134ca54c00 0x56134ca5f260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:18:37.912 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:37.908+0000 7f37fbddc740 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f37f41abf50 con 0x56134ca54c00 2024-03-24T00:18:37.912 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:37.908+0000 7f37fbddc740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f37f41ab8f0 con 0x56134ca4f0a0 2024-03-24T00:18:37.912 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:37.908+0000 7f37fbddc740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f37f41a8df0 con 0x56134ca4fa90 2024-03-24T00:18:37.912 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:37.908+0000 7f37fbddc740 10 client.0 _ll_register_callbacks cb 0x7f37e0009840 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T00:18:37.912 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:37.908+0000 7f37f1400640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56134ca4fa90 0x56134ca5ed20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:18:37.912 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:37.908+0000 7f37f2800640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56134ca4f0a0 0x56134ca5e7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:18:37.912 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:37.909+0000 7f37f1400640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56134ca4fa90 0x56134ca5ed20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.118:53244/0 (socket says 172.21.15.118:53244) 2024-03-24T00:18:37.912 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:37.909+0000 7f37f1400640 1 -- 172.21.15.118:0/1632965947 learned_addr learned my addr 172.21.15.118:0/1632965947 (peer_addr_for_me v2:172.21.15.118:0/0) 2024-03-24T00:18:37.912 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:37.909+0000 7f37f2800640 1 --2- 172.21.15.118:0/1632965947 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x56134ca54c00 0x56134ca5f260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:18:37.912 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:37.909+0000 7f37f1400640 1 -- 172.21.15.118:0/1632965947 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56134ca4f0a0 msgr2=0x56134ca5e7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:18:37.913 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:37.909+0000 7f37f1400640 1 --2- 172.21.15.118:0/1632965947 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56134ca4f0a0 0x56134ca5e7e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:18:37.913 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:37.909+0000 7f37f1400640 1 -- 172.21.15.118:0/1632965947 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x56134ca54c00 msgr2=0x56134ca5f260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:18:37.913 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:37.909+0000 7f37f1400640 1 --2- 172.21.15.118:0/1632965947 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x56134ca54c00 0x56134ca5f260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:18:37.913 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:37.909+0000 7f37f1400640 1 -- 172.21.15.118:0/1632965947 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x56134ca45c20 con 0x56134ca4fa90 2024-03-24T00:18:37.913 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:37.909+0000 7f37f1400640 1 --2- 172.21.15.118:0/1632965947 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56134ca4fa90 0x56134ca5ed20 secure :-1 s=READY pgs=353 cs=0 l=1 rev1=1 crypto rx=0x7f37e000f440 tx=0x7f37e00095b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=eb610516afa9bf94 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:18:37.913 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:37.909+0000 7f37d3e00640 10 client.0 ms_handle_connect on v2:172.21.15.193:3300/0 2024-03-24T00:18:37.913 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:37.909+0000 7f37d3e00640 1 -- 172.21.15.118:0/1632965947 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f37e00035f0 con 0x56134ca4fa90 2024-03-24T00:18:37.913 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:37.910+0000 7f37fbddc740 1 -- 172.21.15.118:0/1632965947 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f37e0022070 con 0x56134ca4fa90 2024-03-24T00:18:37.913 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:37.910+0000 7f37d3e00640 1 -- 172.21.15.118:0/1632965947 <== mon.1 v2:172.21.15.193:3300/0 2 ==== osd_map(90..90 src has 1..90) v4 ==== 8277+0+0 (secure 0 0 0) 0x7f37e0011ae0 con 0x56134ca4fa90 2024-03-24T00:18:37.913 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:37.910+0000 7f37d3e00640 1 -- 172.21.15.118:0/1632965947 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) v2 ==== 24+0+0 (secure 0 0 0) 0x7f37e0041050 con 0x56134ca4fa90 2024-03-24T00:18:37.913 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:37.910+0000 7f37fbddc740 10 client.26317 fetch_fsmap learned FSMap version 28 2024-03-24T00:18:37.913 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:37.910+0000 7f37fbddc740 1 -- 172.21.15.118:0/1632965947 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) v3 -- 0x56134cac1bc0 con 0x56134ca4fa90 2024-03-24T00:18:37.913 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:37.910+0000 7f37d3e00640 1 -- 172.21.15.118:0/1632965947 <== mon.1 v2:172.21.15.193:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f37e0022070 con 0x56134ca4fa90 2024-03-24T00:18:37.914 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:37.910+0000 7f37d3e00640 1 -- 172.21.15.118:0/1632965947 <== mon.1 v2:172.21.15.193:3300/0 5 ==== fsmap(e 28) v1 ==== 7934+0+0 (secure 0 0 0) 0x7f37e0030070 con 0x56134ca4fa90 2024-03-24T00:18:37.914 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:37.910+0000 7f37fbddc740 10 client.26317 fetch_fsmap finished waiting for FSMap version 28 2024-03-24T00:18:37.914 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:37.910+0000 7f37fbddc740 10 client.26317 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-24T00:18:37.914 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:37.910+0000 7f37fbddc740 20 client.26317 populate_metadata read hostname 'smithi118' 2024-03-24T00:18:37.914 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:37.910+0000 7f37fbddc740 1 --2- 172.21.15.118:0/1632965947 >> [v2:172.21.15.189:6838/4074484168,v1:172.21.15.189:6839/4074484168] conn(0x56134cac2490 0x56134cac48b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:18:37.914 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:37.910+0000 7f37fbddc740 4 client.26317 mds_command: new command op to 24569 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-03-24T00:18:37.914 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:37.910+0000 7f37fbddc740 1 -- 172.21.15.118:0/1632965947 --> [v2:172.21.15.189:6838/4074484168,v1:172.21.15.189:6839/4074484168] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) v1 -- 0x56134c9bc740 con 0x56134cac2490 2024-03-24T00:18:37.914 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:37.911+0000 7f37f2800640 1 --2- 172.21.15.118:0/1632965947 >> [v2:172.21.15.189:6838/4074484168,v1:172.21.15.189:6839/4074484168] conn(0x56134cac2490 0x56134cac48b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:18:37.916 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:37.911+0000 7f37f2800640 1 --2- 172.21.15.118:0/1632965947 >> [v2:172.21.15.189:6838/4074484168,v1:172.21.15.189:6839/4074484168] conn(0x56134cac2490 0x56134cac48b0 crc :-1 s=READY pgs=98 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=bce712e7d97afe05 server_cookie=8f25e3a5c4741e5d in_seq=0 out_seq=0 2024-03-24T00:18:37.916 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:37.911+0000 7f37d3e00640 10 client.26317 ms_handle_connect on v2:172.21.15.189:6838/4074484168 2024-03-24T00:18:37.916 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:37.915+0000 7f37d3e00640 1 -- 172.21.15.118:0/1632965947 <== mds.0 v2:172.21.15.189:6838/4074484168 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+110 (crc 0 0 0) 0x56134c9bc740 con 0x56134cac2490 2024-03-24T00:18:37.916 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:37.915+0000 7f37d3e00640 10 client.26317 handle_command_reply: tid=0 2024-03-24T00:18:37.916 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:37.915+0000 7f37fbddc740 1 client.26317 shutdown 2024-03-24T00:18:37.916 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:37.916+0000 7f37fbddc740 1 -- 172.21.15.118:0/1632965947 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56134ca4fa90 msgr2=0x56134ca5ed20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:18:37.916 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:37.916+0000 7f37fbddc740 1 --2- 172.21.15.118:0/1632965947 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56134ca4fa90 0x56134ca5ed20 secure :-1 s=READY pgs=353 cs=0 l=1 rev1=1 crypto rx=0x7f37e000f440 tx=0x7f37e00095b0 comp rx=0 tx=0).stop 2024-03-24T00:18:37.916 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:37.916+0000 7f37fbddc740 1 -- 172.21.15.118:0/1632965947 shutdown_connections 2024-03-24T00:18:37.916 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:37.916+0000 7f37fbddc740 1 --2- 172.21.15.118:0/1632965947 >> [v2:172.21.15.189:6838/4074484168,v1:172.21.15.189:6839/4074484168] conn(0x56134cac2490 0x56134cac48b0 crc :-1 s=READY pgs=98 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:18:37.916 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:37.916+0000 7f37fbddc740 1 --2- 172.21.15.118:0/1632965947 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x56134ca54c00 0x56134ca5f260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:18:37.917 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:37.916+0000 7f37fbddc740 1 --2- 172.21.15.118:0/1632965947 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56134ca4fa90 0x56134ca5ed20 unknown :-1 s=CLOSED pgs=353 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:18:37.917 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:37.916+0000 7f37d3e00640 0 client.26317 ms_handle_reset on v2:172.21.15.189:6838/4074484168 2024-03-24T00:18:37.917 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:37.916+0000 7f37fbddc740 1 --2- 172.21.15.118:0/1632965947 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56134ca4f0a0 0x56134ca5e7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:18:37.917 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:37.916+0000 7f37fbddc740 1 -- 172.21.15.118:0/1632965947 >> 172.21.15.118:0/1632965947 conn(0x56134ca4a610 msgr2=0x56134ca48fa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:18:37.917 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:37.916+0000 7f37fbddc740 1 -- 172.21.15.118:0/1632965947 shutdown_connections 2024-03-24T00:18:37.917 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:37.916+0000 7f37fbddc740 1 -- 172.21.15.118:0/1632965947 wait complete. 2024-03-24T00:18:37.917 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:37.916+0000 7f37fbddc740 20 client.26317 trim_cache size 0 max 16384 2024-03-24T00:18:37.917 INFO:teuthology.orchestra.run.smithi118.stdout:{ 2024-03-24T00:18:37.917 INFO:teuthology.orchestra.run.smithi118.stdout: "return_code": 0, 2024-03-24T00:18:37.918 INFO:teuthology.orchestra.run.smithi118.stdout: "scrub_tag": "93b12463-083c-4dae-8952-bcaa9cbcedf9", 2024-03-24T00:18:37.918 INFO:teuthology.orchestra.run.smithi118.stdout: "mode": "asynchronous" 2024-03-24T00:18:37.918 INFO:teuthology.orchestra.run.smithi118.stdout:} 2024-03-24T00:18:37.918 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:37.917+0000 7f37f9000640 1 -- 172.21.15.118:0/2800749896 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7f37d80a2800 msgr2=0x7f37d80a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:18:37.918 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:37.917+0000 7f37f9000640 1 --2- 172.21.15.118:0/2800749896 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7f37d80a2800 0x7f37d80a4cc0 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 crypto rx=0x7f37e40b9480 tx=0x7f37e40dd040 comp rx=0 tx=0).stop 2024-03-24T00:18:37.919 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:37.917+0000 7f37f9000640 1 -- 172.21.15.118:0/2800749896 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f37f40fff70 msgr2=0x7f37f4199ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:18:37.919 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:37.917+0000 7f37f9000640 1 --2- 172.21.15.118:0/2800749896 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f37f40fff70 0x7f37f4199ba0 secure :-1 s=READY pgs=353 cs=0 l=1 rev1=1 crypto rx=0x7f37e402c8f0 tx=0x7f37e402c660 comp rx=0 tx=0).stop 2024-03-24T00:18:37.919 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:37.917+0000 7f37f9000640 1 -- 172.21.15.118:0/2800749896 shutdown_connections 2024-03-24T00:18:37.919 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:37.917+0000 7f37f9000640 1 --2- 172.21.15.118:0/2800749896 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7f37d80a2800 0x7f37d80a4cc0 unknown :-1 s=CLOSED pgs=326 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:18:37.919 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:37.917+0000 7f37f9000640 1 --2- 172.21.15.118:0/2800749896 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f37f4101270 0x7f37f41a1150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:18:37.919 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:37.917+0000 7f37f9000640 1 --2- 172.21.15.118:0/2800749896 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f37f41008d0 0x7f37f419a0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:18:37.919 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:37.917+0000 7f37f9000640 1 --2- 172.21.15.118:0/2800749896 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f37f40fff70 0x7f37f4199ba0 unknown :-1 s=CLOSED pgs=353 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:18:37.919 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:37.917+0000 7f37f9000640 1 -- 172.21.15.118:0/2800749896 >> 172.21.15.118:0/2800749896 conn(0x7f37f40fbb30 msgr2=0x7f37f40fd260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:18:37.919 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:37.917+0000 7f37f9000640 1 -- 172.21.15.118:0/2800749896 shutdown_connections 2024-03-24T00:18:37.919 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:37.917+0000 7f37f9000640 1 -- 172.21.15.118:0/2800749896 wait complete. 2024-03-24T00:18:37.940 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-03-24T00:18:37.940 DEBUG:teuthology.orchestra.run.smithi118:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-24T00:18:38.142 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:38.142+0000 7f1deb200640 1 -- 172.21.15.118:0/3016833738 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1dec1015a0 msgr2=0x7f1dec1019c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:18:38.142 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:38.142+0000 7f1deb200640 1 --2- 172.21.15.118:0/3016833738 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1dec1015a0 0x7f1dec1019c0 secure :-1 s=READY pgs=354 cs=0 l=1 rev1=1 crypto rx=0x7f1ddc009900 tx=0x7f1ddc02bc50 comp rx=0 tx=0).stop 2024-03-24T00:18:38.142 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:38.142+0000 7f1deb200640 1 -- 172.21.15.118:0/3016833738 shutdown_connections 2024-03-24T00:18:38.142 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:38.142+0000 7f1deb200640 1 --2- 172.21.15.118:0/3016833738 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f1dec103150 0x7f1dec1099e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:18:38.142 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:38.142+0000 7f1deb200640 1 --2- 172.21.15.118:0/3016833738 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f1dec102790 0x7f1dec102c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:18:38.142 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:38.142+0000 7f1deb200640 1 --2- 172.21.15.118:0/3016833738 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1dec1015a0 0x7f1dec1019c0 unknown :-1 s=CLOSED pgs=354 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:18:38.143 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:38.142+0000 7f1deb200640 1 -- 172.21.15.118:0/3016833738 >> 172.21.15.118:0/3016833738 conn(0x7f1dec0fcb10 msgr2=0x7f1dec0fef50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:18:38.143 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:38.142+0000 7f1deb200640 1 -- 172.21.15.118:0/3016833738 shutdown_connections 2024-03-24T00:18:38.143 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:38.142+0000 7f1deb200640 1 -- 172.21.15.118:0/3016833738 wait complete. 2024-03-24T00:18:38.145 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:38.143+0000 7f1deb200640 1 Processor -- start 2024-03-24T00:18:38.145 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:38.143+0000 7f1deb200640 1 -- start start 2024-03-24T00:18:38.145 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:38.143+0000 7f1deb200640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f1dec1015a0 0x7f1dec199b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:18:38.145 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:38.143+0000 7f1deb200640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f1dec102790 0x7f1dec19a0b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:18:38.145 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:38.143+0000 7f1deb200640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1dec103150 0x7f1dec1a1120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:18:38.145 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:38.143+0000 7f1deb200640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1dec10d230 con 0x7f1dec102790 2024-03-24T00:18:38.146 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:38.143+0000 7f1deb200640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1dec10d0b0 con 0x7f1dec103150 2024-03-24T00:18:38.146 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:38.143+0000 7f1deb200640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1dec10d3b0 con 0x7f1dec1015a0 2024-03-24T00:18:38.146 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:38.144+0000 7f1de9e00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f1dec1015a0 0x7f1dec199b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:18:38.146 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:38.144+0000 7f1dea800640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1dec103150 0x7f1dec1a1120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:18:38.146 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:38.144+0000 7f1de9e00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f1dec1015a0 0x7f1dec199b70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.118:53258/0 (socket says 172.21.15.118:53258) 2024-03-24T00:18:38.146 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:38.144+0000 7f1de9e00640 1 -- 172.21.15.118:0/2885188292 learned_addr learned my addr 172.21.15.118:0/2885188292 (peer_addr_for_me v2:172.21.15.118:0/0) 2024-03-24T00:18:38.146 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:38.144+0000 7f1de9400640 1 --2- 172.21.15.118:0/2885188292 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f1dec102790 0x7f1dec19a0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:18:38.146 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:38.144+0000 7f1dea800640 1 -- 172.21.15.118:0/2885188292 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f1dec1015a0 msgr2=0x7f1dec199b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:18:38.146 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:38.144+0000 7f1dea800640 1 --2- 172.21.15.118:0/2885188292 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f1dec1015a0 0x7f1dec199b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:18:38.146 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:38.144+0000 7f1dea800640 1 -- 172.21.15.118:0/2885188292 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f1dec102790 msgr2=0x7f1dec19a0b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:18:38.146 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:38.144+0000 7f1dea800640 1 --2- 172.21.15.118:0/2885188292 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f1dec102790 0x7f1dec19a0b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:18:38.146 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:38.144+0000 7f1dea800640 1 -- 172.21.15.118:0/2885188292 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1dec1a1790 con 0x7f1dec103150 2024-03-24T00:18:38.147 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:38.144+0000 7f1de9e00640 1 --2- 172.21.15.118:0/2885188292 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f1dec1015a0 0x7f1dec199b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T00:18:38.147 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:38.144+0000 7f1dea800640 1 --2- 172.21.15.118:0/2885188292 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1dec103150 0x7f1dec1a1120 secure :-1 s=READY pgs=355 cs=0 l=1 rev1=1 crypto rx=0x7f1de400ec50 tx=0x7f1de400c6a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:18:38.147 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:38.145+0000 7f1de2a00640 1 -- 172.21.15.118:0/2885188292 <== mon.2 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1de4003b70 con 0x7f1dec103150 2024-03-24T00:18:38.147 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:38.145+0000 7f1de2a00640 1 -- 172.21.15.118:0/2885188292 <== mon.2 v2:172.21.15.189:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f1de4003d10 con 0x7f1dec103150 2024-03-24T00:18:38.147 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:38.145+0000 7f1deb200640 1 -- 172.21.15.118:0/2885188292 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1dec1a1a80 con 0x7f1dec103150 2024-03-24T00:18:38.147 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:38.145+0000 7f1de2a00640 1 -- 172.21.15.118:0/2885188292 <== mon.2 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1de4010890 con 0x7f1dec103150 2024-03-24T00:18:38.147 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:38.145+0000 7f1deb200640 1 -- 172.21.15.118:0/2885188292 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1dec1a1fd0 con 0x7f1dec103150 2024-03-24T00:18:38.150 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:38.148+0000 7f1de2a00640 1 -- 172.21.15.118:0/2885188292 <== mon.2 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 20) v1 ==== 137291+0+0 (secure 0 0 0) 0x7f1de4003fc0 con 0x7f1dec103150 2024-03-24T00:18:38.150 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:38.148+0000 7f1df3dd6740 1 Processor -- start 2024-03-24T00:18:38.150 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:38.149+0000 7f1df3dd6740 1 -- start start 2024-03-24T00:18:38.150 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:38.149+0000 7f1df3dd6740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5600bfe1e050 0x5600bfe1e470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:18:38.150 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:38.149+0000 7f1df3dd6740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5600bfe1ea40 0x5600bfe23540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:18:38.150 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:38.149+0000 7f1df3dd6740 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5600bfe23bb0 0x5600bfe26040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:18:38.151 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:38.149+0000 7f1df3dd6740 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x5600bfd83250 con 0x5600bfe23bb0 2024-03-24T00:18:38.151 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:38.149+0000 7f1df3dd6740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x5600bfd2ba30 con 0x5600bfe1e050 2024-03-24T00:18:38.151 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:38.149+0000 7f1df3dd6740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x5600bfd7f2f0 con 0x5600bfe1ea40 2024-03-24T00:18:38.151 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:38.149+0000 7f1de9400640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5600bfe1e050 0x5600bfe1e470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:18:38.151 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:38.149+0000 7f1de9400640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5600bfe1e050 0x5600bfe1e470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.118:33060/0 (socket says 172.21.15.118:33060) 2024-03-24T00:18:38.151 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:38.149+0000 7f1de9400640 1 -- 172.21.15.118:0/3741370376 learned_addr learned my addr 172.21.15.118:0/3741370376 (peer_addr_for_me v2:172.21.15.118:0/0) 2024-03-24T00:18:38.151 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:38.149+0000 7f1de9e00640 1 --2- 172.21.15.118:0/3741370376 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5600bfe1ea40 0x5600bfe23540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:18:38.151 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:38.149+0000 7f1de9400640 1 --2- 172.21.15.118:0/3741370376 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5600bfe23bb0 0x5600bfe26040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:18:38.152 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:38.150+0000 7f1de9e00640 1 -- 172.21.15.118:0/3741370376 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5600bfe1e050 msgr2=0x5600bfe1e470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:18:38.152 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:38.150+0000 7f1de9e00640 1 --2- 172.21.15.118:0/3741370376 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5600bfe1e050 0x5600bfe1e470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:18:38.152 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:38.150+0000 7f1de9e00640 1 -- 172.21.15.118:0/3741370376 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5600bfe23bb0 msgr2=0x5600bfe26040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:18:38.152 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:38.150+0000 7f1de9e00640 1 --2- 172.21.15.118:0/3741370376 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5600bfe23bb0 0x5600bfe26040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:18:38.152 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:38.150+0000 7f1de9e00640 1 -- 172.21.15.118:0/3741370376 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5600bfd16690 con 0x5600bfe1ea40 2024-03-24T00:18:38.152 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:38.150+0000 7f1de9e00640 1 --2- 172.21.15.118:0/3741370376 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5600bfe1ea40 0x5600bfe23540 secure :-1 s=READY pgs=354 cs=0 l=1 rev1=1 crypto rx=0x7f1ddc02c7d0 tx=0x7f1ddc02cd50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=f9dddc9a4208a699 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:18:38.152 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:38.150+0000 7f1de2a00640 1 --2- 172.21.15.118:0/2885188292 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7f1dd00a2880 0x7f1dd00a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:18:38.152 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:38.150+0000 7f1dd7400640 1 -- 172.21.15.118:0/3741370376 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1ddc004030 con 0x5600bfe1ea40 2024-03-24T00:18:38.152 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:38.150+0000 7f1dd7400640 1 -- 172.21.15.118:0/3741370376 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f1ddc004ba0 con 0x5600bfe1ea40 2024-03-24T00:18:38.153 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:38.150+0000 7f1de2a00640 1 -- 172.21.15.118:0/2885188292 <== mon.2 v2:172.21.15.189:3300/0 5 ==== osd_map(90..90 src has 1..90) v4 ==== 8277+0+0 (secure 0 0 0) 0x7f1de40ceff0 con 0x7f1dec103150 2024-03-24T00:18:38.153 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:38.150+0000 7f1dd7400640 1 -- 172.21.15.118:0/3741370376 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1ddc0052f0 con 0x5600bfe1ea40 2024-03-24T00:18:38.153 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:38.151+0000 7f1de9400640 1 --2- 172.21.15.118:0/2885188292 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7f1dd00a2880 0x7f1dd00a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:18:38.157 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:38.151+0000 7f1df3dd6740 1 -- 172.21.15.118:0/3741370376 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5600bfe1ea40 msgr2=0x5600bfe23540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:18:38.157 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:38.151+0000 7f1df3dd6740 1 --2- 172.21.15.118:0/3741370376 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5600bfe1ea40 0x5600bfe23540 secure :-1 s=READY pgs=354 cs=0 l=1 rev1=1 crypto rx=0x7f1ddc02c7d0 tx=0x7f1ddc02cd50 comp rx=0 tx=0).stop 2024-03-24T00:18:38.157 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:38.151+0000 7f1df3dd6740 1 -- 172.21.15.118:0/3741370376 shutdown_connections 2024-03-24T00:18:38.158 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:38.151+0000 7f1df3dd6740 1 --2- 172.21.15.118:0/3741370376 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5600bfe23bb0 0x5600bfe26040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:18:38.158 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:38.151+0000 7f1df3dd6740 1 --2- 172.21.15.118:0/3741370376 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5600bfe1ea40 0x5600bfe23540 unknown :-1 s=CLOSED pgs=354 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:18:38.158 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:38.151+0000 7f1df3dd6740 1 --2- 172.21.15.118:0/3741370376 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5600bfe1e050 0x5600bfe1e470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:18:38.158 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:38.151+0000 7f1df3dd6740 1 -- 172.21.15.118:0/3741370376 >> 172.21.15.118:0/3741370376 conn(0x5600bfde6200 msgr2=0x5600bfe199c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:18:38.158 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:38.151+0000 7f1de9400640 1 --2- 172.21.15.118:0/2885188292 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7f1dd00a2880 0x7f1dd00a4d40 secure :-1 s=READY pgs=327 cs=0 l=1 rev1=1 crypto rx=0x5600bfe1f2a0 tx=0x7f1dd8005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:18:38.158 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:38.152+0000 7f1df3dd6740 1 -- 172.21.15.118:0/3741370376 shutdown_connections 2024-03-24T00:18:38.158 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:38.152+0000 7f1df3dd6740 1 -- 172.21.15.118:0/3741370376 wait complete. 2024-03-24T00:18:38.158 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:38.152+0000 7f1df3dd6740 1 -- start start 2024-03-24T00:18:38.158 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:38.152+0000 7f1df3dd6740 1 Processor -- start 2024-03-24T00:18:38.158 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:38.152+0000 7f1df3dd6740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5600bfe1e050 0x5600bfe28050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:18:38.159 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:38.153+0000 7f1df3dd6740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5600bfe1ea40 0x5600bfe28590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:18:38.159 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:38.153+0000 7f1df3dd6740 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5600bfe23bb0 0x5600bfe2f560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:18:38.159 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:38.153+0000 7f1df3dd6740 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x5600bfe26b00 con 0x5600bfe23bb0 2024-03-24T00:18:38.159 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:38.153+0000 7f1df3dd6740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x5600bfd3ee80 con 0x5600bfe1e050 2024-03-24T00:18:38.159 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:38.153+0000 7f1df3dd6740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x5600bfe26c80 con 0x5600bfe1ea40 2024-03-24T00:18:38.159 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:38.153+0000 7f1de9400640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5600bfe1ea40 0x5600bfe28590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:18:38.159 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:38.153+0000 7f1df3dd6740 10 client.0 _ll_register_callbacks cb 0x5600bfd51130 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T00:18:38.159 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:38.153+0000 7f1de9e00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5600bfe1e050 0x5600bfe28050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:18:38.159 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:38.153+0000 7f1de9400640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5600bfe1ea40 0x5600bfe28590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.118:53266/0 (socket says 172.21.15.118:53266) 2024-03-24T00:18:38.159 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:38.153+0000 7f1de9400640 1 -- 172.21.15.118:0/2928287308 learned_addr learned my addr 172.21.15.118:0/2928287308 (peer_addr_for_me v2:172.21.15.118:0/0) 2024-03-24T00:18:38.159 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:38.153+0000 7f1dea800640 1 --2- 172.21.15.118:0/2928287308 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5600bfe23bb0 0x5600bfe2f560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:18:38.160 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:38.153+0000 7f1de9e00640 1 -- 172.21.15.118:0/2928287308 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5600bfe1ea40 msgr2=0x5600bfe28590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:18:38.160 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:38.153+0000 7f1de9e00640 1 --2- 172.21.15.118:0/2928287308 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5600bfe1ea40 0x5600bfe28590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:18:38.160 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:38.153+0000 7f1de9e00640 1 -- 172.21.15.118:0/2928287308 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5600bfe23bb0 msgr2=0x5600bfe2f560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:18:38.160 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:38.153+0000 7f1de9e00640 1 --2- 172.21.15.118:0/2928287308 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5600bfe23bb0 0x5600bfe2f560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:18:38.160 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:38.153+0000 7f1de9e00640 1 -- 172.21.15.118:0/2928287308 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5600bfb617d0 con 0x5600bfe1e050 2024-03-24T00:18:38.160 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:38.154+0000 7f1de9e00640 1 --2- 172.21.15.118:0/2928287308 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5600bfe1e050 0x5600bfe28050 secure :-1 s=READY pgs=356 cs=0 l=1 rev1=1 crypto rx=0x5600bfe1a910 tx=0x7f1ddc005280 comp rx=0 tx=0).ready entity=mon.2 client_cookie=15a8ab96928ff4df server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:18:38.160 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:38.154+0000 7f1dcbe00640 10 client.0 ms_handle_connect on v2:172.21.15.189:3300/0 2024-03-24T00:18:38.160 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:38.154+0000 7f1dcbe00640 1 -- 172.21.15.118:0/2928287308 <== mon.2 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1ddc034440 con 0x5600bfe1e050 2024-03-24T00:18:38.160 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:38.154+0000 7f1df3dd6740 1 -- 172.21.15.118:0/2928287308 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f1ddc004ba0 con 0x5600bfe1e050 2024-03-24T00:18:38.160 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:38.154+0000 7f1dcbe00640 1 -- 172.21.15.118:0/2928287308 <== mon.2 v2:172.21.15.189:3300/0 2 ==== osd_map(90..90 src has 1..90) v4 ==== 8277+0+0 (secure 0 0 0) 0x7f1ddc0345e0 con 0x5600bfe1e050 2024-03-24T00:18:38.160 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:38.154+0000 7f1dcbe00640 1 -- 172.21.15.118:0/2928287308 <== mon.2 v2:172.21.15.189:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) v2 ==== 24+0+0 (secure 0 0 0) 0x7f1ddc0347e0 con 0x5600bfe1e050 2024-03-24T00:18:38.161 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:38.154+0000 7f1df3dd6740 10 client.26318 fetch_fsmap learned FSMap version 28 2024-03-24T00:18:38.161 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:38.154+0000 7f1df3dd6740 1 -- 172.21.15.118:0/2928287308 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) v3 -- 0x5600bfa973f0 con 0x5600bfe1e050 2024-03-24T00:18:38.161 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:38.154+0000 7f1dcbe00640 1 -- 172.21.15.118:0/2928287308 <== mon.2 v2:172.21.15.189:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f1ddc004ba0 con 0x5600bfe1e050 2024-03-24T00:18:38.161 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:38.155+0000 7f1dcbe00640 1 -- 172.21.15.118:0/2928287308 <== mon.2 v2:172.21.15.189:3300/0 5 ==== fsmap(e 28) v1 ==== 7934+0+0 (secure 0 0 0) 0x7f1ddc05a070 con 0x5600bfe1e050 2024-03-24T00:18:38.161 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:38.155+0000 7f1df3dd6740 10 client.26318 fetch_fsmap finished waiting for FSMap version 28 2024-03-24T00:18:38.161 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:38.155+0000 7f1df3dd6740 10 client.26318 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-24T00:18:38.161 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:38.155+0000 7f1df3dd6740 20 client.26318 populate_metadata read hostname 'smithi118' 2024-03-24T00:18:38.161 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:38.155+0000 7f1df3dd6740 1 --2- 172.21.15.118:0/2928287308 >> [v2:172.21.15.189:6838/4074484168,v1:172.21.15.189:6839/4074484168] conn(0x5600bfe33a50 0x5600bfe35e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:18:38.161 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:38.155+0000 7f1df3dd6740 4 client.26318 mds_command: new command op to 24569 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-24T00:18:38.161 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:38.155+0000 7f1df3dd6740 1 -- 172.21.15.118:0/2928287308 --> [v2:172.21.15.189:6838/4074484168,v1:172.21.15.189:6839/4074484168] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x5600bfd90790 con 0x5600bfe33a50 2024-03-24T00:18:38.161 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:38.155+0000 7f1de9400640 1 --2- 172.21.15.118:0/2928287308 >> [v2:172.21.15.189:6838/4074484168,v1:172.21.15.189:6839/4074484168] conn(0x5600bfe33a50 0x5600bfe35e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:18:38.161 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:38.155+0000 7f1de9400640 1 --2- 172.21.15.118:0/2928287308 >> [v2:172.21.15.189:6838/4074484168,v1:172.21.15.189:6839/4074484168] conn(0x5600bfe33a50 0x5600bfe35e40 crc :-1 s=READY pgs=99 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=9495bdb3935e1457 server_cookie=466830e42b241cb9 in_seq=0 out_seq=0 2024-03-24T00:18:38.162 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:38.155+0000 7f1dcbe00640 10 client.26318 ms_handle_connect on v2:172.21.15.189:6838/4074484168 2024-03-24T00:18:38.163 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:38.159+0000 7f1dcbe00640 1 -- 172.21.15.118:0/2928287308 <== mds.0 v2:172.21.15.189:6838/4074484168 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x5600bfd90790 con 0x5600bfe33a50 2024-03-24T00:18:38.163 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:38.159+0000 7f1dcbe00640 10 client.26318 handle_command_reply: tid=0 2024-03-24T00:18:38.163 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:38.159+0000 7f1df3dd6740 1 client.26318 shutdown 2024-03-24T00:18:38.163 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:38.159+0000 7f1df3dd6740 1 -- 172.21.15.118:0/2928287308 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5600bfe1e050 msgr2=0x5600bfe28050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:18:38.163 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:38.159+0000 7f1df3dd6740 1 --2- 172.21.15.118:0/2928287308 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5600bfe1e050 0x5600bfe28050 secure :-1 s=READY pgs=356 cs=0 l=1 rev1=1 crypto rx=0x5600bfe1a910 tx=0x7f1ddc005280 comp rx=0 tx=0).stop 2024-03-24T00:18:38.163 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:38.159+0000 7f1df3dd6740 1 -- 172.21.15.118:0/2928287308 shutdown_connections 2024-03-24T00:18:38.163 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:38.159+0000 7f1df3dd6740 1 --2- 172.21.15.118:0/2928287308 >> [v2:172.21.15.189:6838/4074484168,v1:172.21.15.189:6839/4074484168] conn(0x5600bfe33a50 0x5600bfe35e40 crc :-1 s=READY pgs=99 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:18:38.163 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:38.159+0000 7f1df3dd6740 1 --2- 172.21.15.118:0/2928287308 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5600bfe23bb0 0x5600bfe2f560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:18:38.163 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:38.159+0000 7f1df3dd6740 1 --2- 172.21.15.118:0/2928287308 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5600bfe1ea40 0x5600bfe28590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:18:38.163 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:38.159+0000 7f1df3dd6740 1 --2- 172.21.15.118:0/2928287308 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5600bfe1e050 0x5600bfe28050 unknown :-1 s=CLOSED pgs=356 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:18:38.164 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:38.159+0000 7f1dcbe00640 0 client.26318 ms_handle_reset on v2:172.21.15.189:6838/4074484168 2024-03-24T00:18:38.164 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:38.159+0000 7f1df3dd6740 1 -- 172.21.15.118:0/2928287308 >> 172.21.15.118:0/2928287308 conn(0x5600bfde6200 msgr2=0x5600bfe18bc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:18:38.164 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:38.159+0000 7f1df3dd6740 1 -- 172.21.15.118:0/2928287308 shutdown_connections 2024-03-24T00:18:38.164 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:38.159+0000 7f1df3dd6740 1 -- 172.21.15.118:0/2928287308 wait complete. 2024-03-24T00:18:38.164 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:38.159+0000 7f1df3dd6740 20 client.26318 trim_cache size 0 max 16384 2024-03-24T00:18:38.177 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:38.178+0000 7f1df3dd6740 1 Processor -- start 2024-03-24T00:18:38.178 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:38.178+0000 7f1df3dd6740 1 -- start start 2024-03-24T00:18:38.179 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:38.178+0000 7f1df3dd6740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5600bfe1e050 0x5600bfe18bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:18:38.179 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:38.178+0000 7f1df3dd6740 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5600bfe1ea40 0x5600bfe2f4f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:18:38.179 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:38.178+0000 7f1df3dd6740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5600bfe23bb0 0x5600bfe1d310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:18:38.179 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:38.178+0000 7f1df3dd6740 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1dec1aa9d0 con 0x5600bfe1ea40 2024-03-24T00:18:38.179 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:38.178+0000 7f1df3dd6740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1dec1ac350 con 0x5600bfe1e050 2024-03-24T00:18:38.180 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:38.178+0000 7f1df3dd6740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1dec1ab030 con 0x5600bfe23bb0 2024-03-24T00:18:38.180 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:38.179+0000 7f1de9e00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5600bfe1e050 0x5600bfe18bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:18:38.180 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:38.179+0000 7f1de9400640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5600bfe1ea40 0x5600bfe2f4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:18:38.180 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:38.179+0000 7f1de9400640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5600bfe1ea40 0x5600bfe2f4f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.118:3300/0 says I am v2:172.21.15.118:38294/0 (socket says 172.21.15.118:38294) 2024-03-24T00:18:38.180 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:38.179+0000 7f1de9400640 1 -- 172.21.15.118:0/1136680193 learned_addr learned my addr 172.21.15.118:0/1136680193 (peer_addr_for_me v2:172.21.15.118:0/0) 2024-03-24T00:18:38.180 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:38.179+0000 7f1dea800640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5600bfe23bb0 0x5600bfe1d310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:18:38.180 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:38.179+0000 7f1de9e00640 1 -- 172.21.15.118:0/1136680193 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5600bfe23bb0 msgr2=0x5600bfe1d310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:18:38.180 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:38.179+0000 7f1de9e00640 1 --2- 172.21.15.118:0/1136680193 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5600bfe23bb0 0x5600bfe1d310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:18:38.180 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:38.179+0000 7f1de9e00640 1 -- 172.21.15.118:0/1136680193 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5600bfe1ea40 msgr2=0x5600bfe2f4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:18:38.180 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:38.179+0000 7f1de9e00640 1 --2- 172.21.15.118:0/1136680193 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5600bfe1ea40 0x5600bfe2f4f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:18:38.180 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:38.179+0000 7f1de9e00640 1 -- 172.21.15.118:0/1136680193 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5600bfa570f0 con 0x5600bfe1e050 2024-03-24T00:18:38.181 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:38.179+0000 7f1de9e00640 1 --2- 172.21.15.118:0/1136680193 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5600bfe1e050 0x5600bfe18bc0 secure :-1 s=READY pgs=357 cs=0 l=1 rev1=1 crypto rx=0x7f1ddc03f8f0 tx=0x7f1ddc000bc0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=186488c6ff0db932 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:18:38.181 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:38.180+0000 7f1dd6a00640 1 -- 172.21.15.118:0/1136680193 <== mon.2 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1ddc005d50 con 0x5600bfe1e050 2024-03-24T00:18:38.181 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:38.180+0000 7f1dd6a00640 1 -- 172.21.15.118:0/1136680193 <== mon.2 v2:172.21.15.189:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f1ddc057070 con 0x5600bfe1e050 2024-03-24T00:18:38.181 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:38.180+0000 7f1dd6a00640 1 -- 172.21.15.118:0/1136680193 <== mon.2 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1ddc03f410 con 0x5600bfe1e050 2024-03-24T00:18:38.184 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:38.180+0000 7f1df3dd6740 1 -- 172.21.15.118:0/1136680193 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5600bfe1e050 msgr2=0x5600bfe18bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:18:38.184 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:38.180+0000 7f1df3dd6740 1 --2- 172.21.15.118:0/1136680193 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5600bfe1e050 0x5600bfe18bc0 secure :-1 s=READY pgs=357 cs=0 l=1 rev1=1 crypto rx=0x7f1ddc03f8f0 tx=0x7f1ddc000bc0 comp rx=0 tx=0).stop 2024-03-24T00:18:38.184 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:38.181+0000 7f1df3dd6740 1 -- 172.21.15.118:0/1136680193 shutdown_connections 2024-03-24T00:18:38.185 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:38.181+0000 7f1df3dd6740 1 --2- 172.21.15.118:0/1136680193 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5600bfe23bb0 0x5600bfe1d310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:18:38.185 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:38.181+0000 7f1df3dd6740 1 --2- 172.21.15.118:0/1136680193 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5600bfe1ea40 0x5600bfe2f4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:18:38.185 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:38.181+0000 7f1df3dd6740 1 --2- 172.21.15.118:0/1136680193 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5600bfe1e050 0x5600bfe18bc0 unknown :-1 s=CLOSED pgs=357 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:18:38.185 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:38.181+0000 7f1df3dd6740 1 -- 172.21.15.118:0/1136680193 >> 172.21.15.118:0/1136680193 conn(0x5600bfde6200 msgr2=0x5600bfe1cc00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:18:38.185 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:38.181+0000 7f1df3dd6740 1 -- 172.21.15.118:0/1136680193 shutdown_connections 2024-03-24T00:18:38.185 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:38.181+0000 7f1df3dd6740 1 -- 172.21.15.118:0/1136680193 wait complete. 2024-03-24T00:18:38.185 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:38.181+0000 7f1df3dd6740 1 -- start start 2024-03-24T00:18:38.185 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:38.181+0000 7f1df3dd6740 1 Processor -- start 2024-03-24T00:18:38.186 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:38.182+0000 7f1df3dd6740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5600bfe1e050 0x5600bfe2cf40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:18:38.186 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:38.182+0000 7f1df3dd6740 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5600bfe1ea40 0x5600bfe2d480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:18:38.186 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:38.182+0000 7f1df3dd6740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5600bfe23bb0 0x5600bfe2e490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:18:38.186 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:38.182+0000 7f1df3dd6740 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1dec1abcf0 con 0x5600bfe1ea40 2024-03-24T00:18:38.186 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:38.182+0000 7f1df3dd6740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1dec1ab690 con 0x5600bfe23bb0 2024-03-24T00:18:38.186 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:38.182+0000 7f1df3dd6740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1dec1a8b90 con 0x5600bfe1e050 2024-03-24T00:18:38.186 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:38.182+0000 7f1df3dd6740 10 client.0 _ll_register_callbacks cb 0x7f1dd8009ea0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T00:18:38.186 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:38.182+0000 7f1de9e00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5600bfe1e050 0x5600bfe2cf40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:18:38.186 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:38.182+0000 7f1de9e00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5600bfe1e050 0x5600bfe2cf40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.118:53292/0 (socket says 172.21.15.118:53292) 2024-03-24T00:18:38.186 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:38.182+0000 7f1de9e00640 1 -- 172.21.15.118:0/1902677053 learned_addr learned my addr 172.21.15.118:0/1902677053 (peer_addr_for_me v2:172.21.15.118:0/0) 2024-03-24T00:18:38.187 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:38.182+0000 7f1dea800640 1 --2- 172.21.15.118:0/1902677053 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5600bfe23bb0 0x5600bfe2e490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:18:38.187 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:38.182+0000 7f1de9400640 1 --2- 172.21.15.118:0/1902677053 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5600bfe1ea40 0x5600bfe2d480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:18:38.187 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:38.182+0000 7f1de9e00640 1 -- 172.21.15.118:0/1902677053 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5600bfe23bb0 msgr2=0x5600bfe2e490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:18:38.187 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:38.182+0000 7f1de9e00640 1 --2- 172.21.15.118:0/1902677053 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5600bfe23bb0 0x5600bfe2e490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:18:38.187 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:38.182+0000 7f1de9e00640 1 -- 172.21.15.118:0/1902677053 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5600bfe1ea40 msgr2=0x5600bfe2d480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:18:38.187 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:38.182+0000 7f1de9e00640 1 --2- 172.21.15.118:0/1902677053 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5600bfe1ea40 0x5600bfe2d480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:18:38.187 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:38.183+0000 7f1de9e00640 1 -- 172.21.15.118:0/1902677053 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5600bfe2ea80 con 0x5600bfe1e050 2024-03-24T00:18:38.187 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:38.183+0000 7f1de9e00640 1 --2- 172.21.15.118:0/1902677053 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5600bfe1e050 0x5600bfe2cf40 secure :-1 s=READY pgs=355 cs=0 l=1 rev1=1 crypto rx=0x7f1ddc004650 tx=0x7f1ddc005970 comp rx=0 tx=0).ready entity=mon.1 client_cookie=ec936b13d26bf496 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:18:38.187 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:38.183+0000 7f1dcbe00640 10 client.0 ms_handle_connect on v2:172.21.15.193:3300/0 2024-03-24T00:18:38.187 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:38.183+0000 7f1dcbe00640 1 -- 172.21.15.118:0/1902677053 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1ddc005260 con 0x5600bfe1e050 2024-03-24T00:18:38.187 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:38.183+0000 7f1df3dd6740 1 -- 172.21.15.118:0/1902677053 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f1ddc057070 con 0x5600bfe1e050 2024-03-24T00:18:38.188 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:38.183+0000 7f1dcbe00640 1 -- 172.21.15.118:0/1902677053 <== mon.1 v2:172.21.15.193:3300/0 2 ==== osd_map(90..90 src has 1..90) v4 ==== 8277+0+0 (secure 0 0 0) 0x7f1ddc03f410 con 0x5600bfe1e050 2024-03-24T00:18:38.188 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:38.183+0000 7f1dcbe00640 1 -- 172.21.15.118:0/1902677053 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) v2 ==== 24+0+0 (secure 0 0 0) 0x7f1ddc06e050 con 0x5600bfe1e050 2024-03-24T00:18:38.188 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:38.183+0000 7f1df3dd6740 10 client.26335 fetch_fsmap learned FSMap version 28 2024-03-24T00:18:38.188 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:38.183+0000 7f1df3dd6740 1 -- 172.21.15.118:0/1902677053 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) v3 -- 0x5600bfe8f3b0 con 0x5600bfe1e050 2024-03-24T00:18:38.188 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:38.184+0000 7f1dcbe00640 1 -- 172.21.15.118:0/1902677053 <== mon.1 v2:172.21.15.193:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f1ddc057070 con 0x5600bfe1e050 2024-03-24T00:18:38.188 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:38.184+0000 7f1dcbe00640 1 -- 172.21.15.118:0/1902677053 <== mon.1 v2:172.21.15.193:3300/0 5 ==== fsmap(e 28) v1 ==== 7934+0+0 (secure 0 0 0) 0x7f1ddc05a040 con 0x5600bfe1e050 2024-03-24T00:18:38.188 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:38.184+0000 7f1df3dd6740 10 client.26335 fetch_fsmap finished waiting for FSMap version 28 2024-03-24T00:18:38.188 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:38.184+0000 7f1df3dd6740 10 client.26335 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-24T00:18:38.188 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:38.184+0000 7f1df3dd6740 20 client.26335 populate_metadata read hostname 'smithi118' 2024-03-24T00:18:38.188 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:38.184+0000 7f1df3dd6740 1 --2- 172.21.15.118:0/1902677053 >> [v2:172.21.15.189:6838/4074484168,v1:172.21.15.189:6839/4074484168] conn(0x5600bfe8fc80 0x5600bfe920a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:18:38.188 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:38.184+0000 7f1df3dd6740 4 client.26335 mds_command: new command op to 24569 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-24T00:18:38.188 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:38.184+0000 7f1df3dd6740 1 -- 172.21.15.118:0/1902677053 --> [v2:172.21.15.189:6838/4074484168,v1:172.21.15.189:6839/4074484168] -- command(tid 0: {"prefix": "scrub status"}) v1 -- 0x5600bfd8aae0 con 0x5600bfe8fc80 2024-03-24T00:18:38.189 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:38.184+0000 7f1de9400640 1 --2- 172.21.15.118:0/1902677053 >> [v2:172.21.15.189:6838/4074484168,v1:172.21.15.189:6839/4074484168] conn(0x5600bfe8fc80 0x5600bfe920a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:18:38.189 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:38.185+0000 7f1de9400640 1 --2- 172.21.15.118:0/1902677053 >> [v2:172.21.15.189:6838/4074484168,v1:172.21.15.189:6839/4074484168] conn(0x5600bfe8fc80 0x5600bfe920a0 crc :-1 s=READY pgs=100 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=c3cf25821a15b6eb server_cookie=d8c927a599729532 in_seq=0 out_seq=0 2024-03-24T00:18:38.189 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:38.185+0000 7f1dcbe00640 10 client.26335 ms_handle_connect on v2:172.21.15.189:6838/4074484168 2024-03-24T00:18:38.190 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:38.190+0000 7f1dcbe00640 1 -- 172.21.15.118:0/1902677053 <== mds.0 v2:172.21.15.189:6838/4074484168 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+268 (crc 0 0 0) 0x5600bfd8aae0 con 0x5600bfe8fc80 2024-03-24T00:18:38.190 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:38.190+0000 7f1dcbe00640 10 client.26335 handle_command_reply: tid=0 2024-03-24T00:18:38.190 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:38.190+0000 7f1df3dd6740 1 client.26335 shutdown 2024-03-24T00:18:38.190 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:38.190+0000 7f1df3dd6740 1 -- 172.21.15.118:0/1902677053 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5600bfe1e050 msgr2=0x5600bfe2cf40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:18:38.190 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:38.190+0000 7f1df3dd6740 1 --2- 172.21.15.118:0/1902677053 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5600bfe1e050 0x5600bfe2cf40 secure :-1 s=READY pgs=355 cs=0 l=1 rev1=1 crypto rx=0x7f1ddc004650 tx=0x7f1ddc005970 comp rx=0 tx=0).stop 2024-03-24T00:18:38.190 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:38.190+0000 7f1df3dd6740 1 -- 172.21.15.118:0/1902677053 shutdown_connections 2024-03-24T00:18:38.191 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:38.190+0000 7f1df3dd6740 1 --2- 172.21.15.118:0/1902677053 >> [v2:172.21.15.189:6838/4074484168,v1:172.21.15.189:6839/4074484168] conn(0x5600bfe8fc80 0x5600bfe920a0 crc :-1 s=READY pgs=100 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:18:38.191 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:38.190+0000 7f1df3dd6740 1 --2- 172.21.15.118:0/1902677053 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5600bfe23bb0 0x5600bfe2e490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:18:38.191 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:38.190+0000 7f1df3dd6740 1 --2- 172.21.15.118:0/1902677053 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5600bfe1ea40 0x5600bfe2d480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:18:38.191 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:38.190+0000 7f1dcbe00640 0 client.26335 ms_handle_reset on v2:172.21.15.189:6838/4074484168 2024-03-24T00:18:38.191 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:38.190+0000 7f1df3dd6740 1 --2- 172.21.15.118:0/1902677053 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5600bfe1e050 0x5600bfe2cf40 unknown :-1 s=CLOSED pgs=355 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:18:38.191 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:38.190+0000 7f1df3dd6740 1 -- 172.21.15.118:0/1902677053 >> 172.21.15.118:0/1902677053 conn(0x5600bfde6200 msgr2=0x5600bfe18a40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:18:38.191 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:38.190+0000 7f1df3dd6740 1 -- 172.21.15.118:0/1902677053 shutdown_connections 2024-03-24T00:18:38.191 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:38.190+0000 7f1df3dd6740 1 -- 172.21.15.118:0/1902677053 wait complete. 2024-03-24T00:18:38.191 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:38.190+0000 7f1df3dd6740 20 client.26335 trim_cache size 0 max 16384 2024-03-24T00:18:38.192 INFO:teuthology.orchestra.run.smithi118.stdout:{ 2024-03-24T00:18:38.192 INFO:teuthology.orchestra.run.smithi118.stdout: "status": "scrub active (2354 inodes in the stack)", 2024-03-24T00:18:38.192 INFO:teuthology.orchestra.run.smithi118.stdout: "scrubs": { 2024-03-24T00:18:38.192 INFO:teuthology.orchestra.run.smithi118.stdout: "93b12463-083c-4dae-8952-bcaa9cbcedf9": { 2024-03-24T00:18:38.192 INFO:teuthology.orchestra.run.smithi118.stdout: "path": "/", 2024-03-24T00:18:38.192 INFO:teuthology.orchestra.run.smithi118.stdout: "tag": "93b12463-083c-4dae-8952-bcaa9cbcedf9", 2024-03-24T00:18:38.192 INFO:teuthology.orchestra.run.smithi118.stdout: "options": "recursive,force" 2024-03-24T00:18:38.192 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-03-24T00:18:38.192 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-03-24T00:18:38.192 INFO:teuthology.orchestra.run.smithi118.stdout:} 2024-03-24T00:18:38.193 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:38.191+0000 7f1deb200640 1 -- 172.21.15.118:0/2885188292 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7f1dd00a2880 msgr2=0x7f1dd00a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:18:38.193 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:38.191+0000 7f1deb200640 1 --2- 172.21.15.118:0/2885188292 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7f1dd00a2880 0x7f1dd00a4d40 secure :-1 s=READY pgs=327 cs=0 l=1 rev1=1 crypto rx=0x5600bfe1f2a0 tx=0x7f1dd8005f90 comp rx=0 tx=0).stop 2024-03-24T00:18:38.193 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:38.191+0000 7f1deb200640 1 -- 172.21.15.118:0/2885188292 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1dec103150 msgr2=0x7f1dec1a1120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:18:38.193 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:38.191+0000 7f1deb200640 1 --2- 172.21.15.118:0/2885188292 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1dec103150 0x7f1dec1a1120 secure :-1 s=READY pgs=355 cs=0 l=1 rev1=1 crypto rx=0x7f1de400ec50 tx=0x7f1de400c6a0 comp rx=0 tx=0).stop 2024-03-24T00:18:38.193 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:38.191+0000 7f1deb200640 1 -- 172.21.15.118:0/2885188292 shutdown_connections 2024-03-24T00:18:38.193 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:38.191+0000 7f1deb200640 1 --2- 172.21.15.118:0/2885188292 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7f1dd00a2880 0x7f1dd00a4d40 unknown :-1 s=CLOSED pgs=327 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:18:38.194 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:38.191+0000 7f1deb200640 1 --2- 172.21.15.118:0/2885188292 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1dec103150 0x7f1dec1a1120 unknown :-1 s=CLOSED pgs=355 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:18:38.194 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:38.191+0000 7f1deb200640 1 --2- 172.21.15.118:0/2885188292 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f1dec102790 0x7f1dec19a0b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:18:38.194 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:38.191+0000 7f1deb200640 1 --2- 172.21.15.118:0/2885188292 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f1dec1015a0 0x7f1dec199b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:18:38.194 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:38.191+0000 7f1deb200640 1 -- 172.21.15.118:0/2885188292 >> 172.21.15.118:0/2885188292 conn(0x7f1dec0fcb10 msgr2=0x7f1dec0738d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:18:38.194 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:38.191+0000 7f1deb200640 1 -- 172.21.15.118:0/2885188292 shutdown_connections 2024-03-24T00:18:38.194 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:18:38.191+0000 7f1deb200640 1 -- 172.21.15.118:0/2885188292 wait complete. 2024-03-24T00:18:38.209 INFO:tasks.cephfs.filesystem:scrub status for tag:93b12463-083c-4dae-8952-bcaa9cbcedf9 - {'path': '/', 'tag': '93b12463-083c-4dae-8952-bcaa9cbcedf9', 'options': 'recursive,force'} 2024-03-24T00:18:39.197 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:18:38 smithi118 ceph-mon[31535]: pgmap v486: 97 pgs: 97 active+clean; 49 MiB data, 493 MiB used, 1.0 TiB / 1.0 TiB avail; 644 KiB/s rd, 2.0 MiB/s wr, 370 op/s 2024-03-24T00:18:39.197 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:18:38 smithi118 ceph-mon[31535]: scrub queued for path: / 2024-03-24T00:18:39.197 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:18:38 smithi118 ceph-mon[31535]: scrub summary: idle+waiting paths [/] 2024-03-24T00:18:39.197 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:18:38 smithi118 ceph-mon[31535]: scrub summary: active paths [/] 2024-03-24T00:18:39.203 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:18:38 smithi189 ceph-mon[36187]: pgmap v486: 97 pgs: 97 active+clean; 49 MiB data, 493 MiB used, 1.0 TiB / 1.0 TiB avail; 644 KiB/s rd, 2.0 MiB/s wr, 370 op/s 2024-03-24T00:18:39.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:18:38 smithi189 ceph-mon[36187]: scrub queued for path: / 2024-03-24T00:18:39.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:18:38 smithi189 ceph-mon[36187]: scrub summary: idle+waiting paths [/] 2024-03-24T00:18:39.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:18:38 smithi189 ceph-mon[36187]: scrub summary: active paths [/] 2024-03-24T00:18:39.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:18:38 smithi193 ceph-mon[36878]: pgmap v486: 97 pgs: 97 active+clean; 49 MiB data, 493 MiB used, 1.0 TiB / 1.0 TiB avail; 644 KiB/s rd, 2.0 MiB/s wr, 370 op/s 2024-03-24T00:18:39.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:18:38 smithi193 ceph-mon[36878]: scrub queued for path: / 2024-03-24T00:18:39.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:18:38 smithi193 ceph-mon[36878]: scrub summary: idle+waiting paths [/] 2024-03-24T00:18:39.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:18:38 smithi193 ceph-mon[36878]: scrub summary: active paths [/] 2024-03-24T00:18:41.039 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:18:41.038+0000 7fc965c00640 1 -- 172.21.15.118:0/165844907 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fc9601028c0 msgr2=0x7fc960102d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:18:41.039 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:18:41.038+0000 7fc965c00640 1 --2- 172.21.15.118:0/165844907 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fc9601028c0 0x7fc960102d40 secure :-1 s=READY pgs=356 cs=0 l=1 rev1=1 crypto rx=0x7fc948009900 tx=0x7fc94802bc30 comp rx=0 tx=0).stop 2024-03-24T00:18:41.039 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:18:41.039+0000 7fc965c00640 1 -- 172.21.15.118:0/165844907 shutdown_connections 2024-03-24T00:18:41.039 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:18:41.039+0000 7fc965c00640 1 --2- 172.21.15.118:0/165844907 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc960103280 0x7fc960109b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:18:41.039 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:18:41.039+0000 7fc965c00640 1 --2- 172.21.15.118:0/165844907 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fc9601028c0 0x7fc960102d40 unknown :-1 s=CLOSED pgs=356 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:18:41.039 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:18:41.039+0000 7fc965c00640 1 --2- 172.21.15.118:0/165844907 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fc9601016d0 0x7fc960101af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:18:41.039 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:18:41.039+0000 7fc965c00640 1 -- 172.21.15.118:0/165844907 >> 172.21.15.118:0/165844907 conn(0x7fc9600fcc60 msgr2=0x7fc9600ff080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:18:41.039 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:18:41.039+0000 7fc965c00640 1 -- 172.21.15.118:0/165844907 shutdown_connections 2024-03-24T00:18:41.039 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:18:41.039+0000 7fc965c00640 1 -- 172.21.15.118:0/165844907 wait complete. 2024-03-24T00:18:41.040 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:18:41.040+0000 7fc965c00640 1 Processor -- start 2024-03-24T00:18:41.041 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:18:41.040+0000 7fc965c00640 1 -- start start 2024-03-24T00:18:41.041 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:18:41.040+0000 7fc965c00640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fc9601016d0 0x7fc9601978d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:18:41.041 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:18:41.040+0000 7fc965c00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc9601028c0 0x7fc960197e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:18:41.041 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:18:41.040+0000 7fc965c00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fc960103280 0x7fc96019ee80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:18:41.041 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:18:41.040+0000 7fc965c00640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc96010d380 con 0x7fc9601016d0 2024-03-24T00:18:41.041 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:18:41.040+0000 7fc965c00640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc96010d200 con 0x7fc9601028c0 2024-03-24T00:18:41.041 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:18:41.040+0000 7fc965c00640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc96010d500 con 0x7fc960103280 2024-03-24T00:18:41.041 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:18:41.041+0000 7fc95ea00640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fc9601016d0 0x7fc9601978d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:18:41.041 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:18:41.041+0000 7fc956000640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc9601028c0 0x7fc960197e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:18:41.041 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:18:41.041+0000 7fc95f400640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fc960103280 0x7fc96019ee80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:18:41.041 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:18:41.041+0000 7fc95ea00640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fc9601016d0 0x7fc9601978d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.118:3300/0 says I am v2:172.21.15.118:38318/0 (socket says 172.21.15.118:38318) 2024-03-24T00:18:41.041 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:18:41.041+0000 7fc95ea00640 1 -- 172.21.15.118:0/1310209229 learned_addr learned my addr 172.21.15.118:0/1310209229 (peer_addr_for_me v2:172.21.15.118:0/0) 2024-03-24T00:18:41.041 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:18:41.041+0000 7fc956000640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc9601028c0 0x7fc960197e10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.118:33102/0 (socket says 172.21.15.118:33102) 2024-03-24T00:18:41.042 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:18:41.041+0000 7fc956000640 1 -- 172.21.15.118:0/1310209229 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fc960103280 msgr2=0x7fc96019ee80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:18:41.042 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:18:41.041+0000 7fc956000640 1 --2- 172.21.15.118:0/1310209229 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fc960103280 0x7fc96019ee80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:18:41.042 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:18:41.041+0000 7fc956000640 1 -- 172.21.15.118:0/1310209229 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fc9601016d0 msgr2=0x7fc9601978d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:18:41.042 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:18:41.041+0000 7fc956000640 1 --2- 172.21.15.118:0/1310209229 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fc9601016d0 0x7fc9601978d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:18:41.042 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:18:41.041+0000 7fc956000640 1 -- 172.21.15.118:0/1310209229 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc960073a00 con 0x7fc9601028c0 2024-03-24T00:18:41.042 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:18:41.041+0000 7fc95ea00640 1 --2- 172.21.15.118:0/1310209229 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fc9601016d0 0x7fc9601978d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T00:18:41.043 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:18:41.041+0000 7fc956000640 1 --2- 172.21.15.118:0/1310209229 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc9601028c0 0x7fc960197e10 secure :-1 s=READY pgs=358 cs=0 l=1 rev1=1 crypto rx=0x7fc948036040 tx=0x7fc94802c650 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:18:41.043 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:18:41.042+0000 7fc957e00640 1 -- 172.21.15.118:0/1310209229 <== mon.2 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc948043020 con 0x7fc9601028c0 2024-03-24T00:18:41.043 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:18:41.042+0000 7fc957e00640 1 -- 172.21.15.118:0/1310209229 <== mon.2 v2:172.21.15.189:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fc9480054e0 con 0x7fc9601028c0 2024-03-24T00:18:41.043 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:18:41.042+0000 7fc965c00640 1 -- 172.21.15.118:0/1310209229 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc960073c90 con 0x7fc9601028c0 2024-03-24T00:18:41.043 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:18:41.042+0000 7fc957e00640 1 -- 172.21.15.118:0/1310209229 <== mon.2 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc948003c30 con 0x7fc9601028c0 2024-03-24T00:18:41.043 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:18:41.042+0000 7fc965c00640 1 -- 172.21.15.118:0/1310209229 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc960074200 con 0x7fc9601028c0 2024-03-24T00:18:41.045 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:18:41.045+0000 7fc957e00640 1 -- 172.21.15.118:0/1310209229 <== mon.2 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 20) v1 ==== 137291+0+0 (secure 0 0 0) 0x7fc948046020 con 0x7fc9601028c0 2024-03-24T00:18:41.045 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:18:41.046+0000 7fc9689d5740 1 Processor -- start 2024-03-24T00:18:41.045 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:18:41.046+0000 7fc9689d5740 1 -- start start 2024-03-24T00:18:41.046 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:18:41.046+0000 7fc9689d5740 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55f3c8dbf1e0 0x55f3c8dbf600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:18:41.046 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:18:41.046+0000 7fc9689d5740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55f3c8dbfbd0 0x55f3c8dc46d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:18:41.046 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:18:41.046+0000 7fc9689d5740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55f3c8dc4d40 0x55f3c8dc71d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:18:41.046 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:18:41.046+0000 7fc9689d5740 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f3c8d823b0 con 0x55f3c8dbf1e0 2024-03-24T00:18:41.047 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:18:41.046+0000 7fc9689d5740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f3c8d24830 con 0x55f3c8dbfbd0 2024-03-24T00:18:41.047 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:18:41.046+0000 7fc9689d5740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f3c8ccd5d0 con 0x55f3c8dc4d40 2024-03-24T00:18:41.047 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:18:41.047+0000 7fc95f400640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55f3c8dbf1e0 0x55f3c8dbf600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:18:41.047 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:18:41.047+0000 7fc956000640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55f3c8dc4d40 0x55f3c8dc71d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:18:41.047 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:18:41.047+0000 7fc95f400640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55f3c8dbf1e0 0x55f3c8dbf600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.118:3300/0 says I am v2:172.21.15.118:38332/0 (socket says 172.21.15.118:38332) 2024-03-24T00:18:41.047 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:18:41.047+0000 7fc95f400640 1 -- 172.21.15.118:0/3334834901 learned_addr learned my addr 172.21.15.118:0/3334834901 (peer_addr_for_me v2:172.21.15.118:0/0) 2024-03-24T00:18:41.049 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:18:41.047+0000 7fc95f400640 1 -- 172.21.15.118:0/3334834901 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55f3c8dc4d40 msgr2=0x55f3c8dc71d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:18:41.049 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:18:41.047+0000 7fc95f400640 1 --2- 172.21.15.118:0/3334834901 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55f3c8dc4d40 0x55f3c8dc71d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:18:41.049 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:18:41.047+0000 7fc95f400640 1 -- 172.21.15.118:0/3334834901 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55f3c8dbfbd0 msgr2=0x55f3c8dc46d0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-24T00:18:41.049 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:18:41.047+0000 7fc95f400640 1 --2- 172.21.15.118:0/3334834901 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55f3c8dbfbd0 0x55f3c8dc46d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:18:41.049 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:18:41.047+0000 7fc95f400640 1 -- 172.21.15.118:0/3334834901 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55f3c8ce52c0 con 0x55f3c8dbf1e0 2024-03-24T00:18:41.049 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:18:41.048+0000 7fc95f400640 1 --2- 172.21.15.118:0/3334834901 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55f3c8dbf1e0 0x55f3c8dbf600 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7fc958009fd0 tx=0x7fc9580118d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=16d29f56606bab91 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:18:41.049 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:18:41.048+0000 7fc94fe00640 1 -- 172.21.15.118:0/3334834901 <== mon.0 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc958004d20 con 0x55f3c8dbf1e0 2024-03-24T00:18:41.049 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:18:41.048+0000 7fc94fe00640 1 -- 172.21.15.118:0/3334834901 <== mon.0 v2:172.21.15.118:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fc958004050 con 0x55f3c8dbf1e0 2024-03-24T00:18:41.050 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:18:41.048+0000 7fc94fe00640 1 -- 172.21.15.118:0/3334834901 <== mon.0 v2:172.21.15.118:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc958005220 con 0x55f3c8dbf1e0 2024-03-24T00:18:41.050 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:18:41.048+0000 7fc957e00640 1 --2- 172.21.15.118:0/1310209229 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7fc9440a2920 0x7fc9440a4de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:18:41.050 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:18:41.048+0000 7fc957e00640 1 -- 172.21.15.118:0/1310209229 <== mon.2 v2:172.21.15.189:3300/0 5 ==== osd_map(90..90 src has 1..90) v4 ==== 8277+0+0 (secure 0 0 0) 0x7fc9480f09e0 con 0x7fc9601028c0 2024-03-24T00:18:41.050 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:18:41.048+0000 7fc95ea00640 1 --2- 172.21.15.118:0/1310209229 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7fc9440a2920 0x7fc9440a4de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:18:41.050 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:18:41.049+0000 7fc9689d5740 1 -- 172.21.15.118:0/3334834901 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55f3c8dbf1e0 msgr2=0x55f3c8dbf600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:18:41.050 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:18:41.049+0000 7fc9689d5740 1 --2- 172.21.15.118:0/3334834901 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55f3c8dbf1e0 0x55f3c8dbf600 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7fc958009fd0 tx=0x7fc9580118d0 comp rx=0 tx=0).stop 2024-03-24T00:18:41.050 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:18:41.049+0000 7fc9689d5740 1 -- 172.21.15.118:0/3334834901 shutdown_connections 2024-03-24T00:18:41.050 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:18:41.049+0000 7fc9689d5740 1 --2- 172.21.15.118:0/3334834901 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55f3c8dc4d40 0x55f3c8dc71d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:18:41.050 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:18:41.049+0000 7fc9689d5740 1 --2- 172.21.15.118:0/3334834901 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55f3c8dbfbd0 0x55f3c8dc46d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:18:41.050 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:18:41.049+0000 7fc9689d5740 1 --2- 172.21.15.118:0/3334834901 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55f3c8dbf1e0 0x55f3c8dbf600 unknown :-1 s=CLOSED pgs=202 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:18:41.050 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:18:41.049+0000 7fc9689d5740 1 -- 172.21.15.118:0/3334834901 >> 172.21.15.118:0/3334834901 conn(0x55f3c8dba750 msgr2=0x55f3c8dbab50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:18:41.050 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:18:41.049+0000 7fc9689d5740 1 -- 172.21.15.118:0/3334834901 shutdown_connections 2024-03-24T00:18:41.050 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:18:41.049+0000 7fc9689d5740 1 -- 172.21.15.118:0/3334834901 wait complete. 2024-03-24T00:18:41.051 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:18:41.049+0000 7fc95ea00640 1 --2- 172.21.15.118:0/1310209229 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7fc9440a2920 0x7fc9440a4de0 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 crypto rx=0x55f3c8dbffd0 tx=0x7fc950005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:18:41.051 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:18:41.050+0000 7fc9689d5740 1 -- start start 2024-03-24T00:18:41.053 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:18:41.050+0000 7fc9689d5740 1 Processor -- start 2024-03-24T00:18:41.054 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:18:41.050+0000 7fc9689d5740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55f3c8dbf1e0 0x55f3c8dc9350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:18:41.054 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:18:41.050+0000 7fc9689d5740 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55f3c8dbfbd0 0x55f3c8dc9890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:18:41.054 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:18:41.050+0000 7fc9689d5740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55f3c8dc4d40 0x55f3c8dd08d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:18:41.054 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:18:41.050+0000 7fc9689d5740 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f3c8ce0a20 con 0x55f3c8dbfbd0 2024-03-24T00:18:41.054 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:18:41.050+0000 7fc9689d5740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f3c8d208d0 con 0x55f3c8dbf1e0 2024-03-24T00:18:41.054 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:18:41.050+0000 7fc9689d5740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f3c8dc7cf0 con 0x55f3c8dc4d40 2024-03-24T00:18:41.054 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:18:41.051+0000 7fc95ea00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55f3c8dbf1e0 0x55f3c8dc9350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:18:41.054 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:18:41.051+0000 7fc9689d5740 10 client.0 _ll_register_callbacks cb 0x55f3c8d1d1c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T00:18:41.054 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:18:41.051+0000 7fc95ea00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55f3c8dbf1e0 0x55f3c8dc9350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.118:33104/0 (socket says 172.21.15.118:33104) 2024-03-24T00:18:41.054 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:18:41.051+0000 7fc95ea00640 1 -- 172.21.15.118:0/1701889729 learned_addr learned my addr 172.21.15.118:0/1701889729 (peer_addr_for_me v2:172.21.15.118:0/0) 2024-03-24T00:18:41.054 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:18:41.051+0000 7fc956000640 1 --2- 172.21.15.118:0/1701889729 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55f3c8dbfbd0 0x55f3c8dc9890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:18:41.055 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:18:41.051+0000 7fc95f400640 1 --2- 172.21.15.118:0/1701889729 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55f3c8dc4d40 0x55f3c8dd08d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:18:41.055 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:18:41.051+0000 7fc95ea00640 1 -- 172.21.15.118:0/1701889729 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55f3c8dc4d40 msgr2=0x55f3c8dd08d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:18:41.055 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:18:41.051+0000 7fc95ea00640 1 --2- 172.21.15.118:0/1701889729 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55f3c8dc4d40 0x55f3c8dd08d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:18:41.055 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:18:41.051+0000 7fc95ea00640 1 -- 172.21.15.118:0/1701889729 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55f3c8dbfbd0 msgr2=0x55f3c8dc9890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:18:41.055 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:18:41.051+0000 7fc95ea00640 1 --2- 172.21.15.118:0/1701889729 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55f3c8dbfbd0 0x55f3c8dc9890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:18:41.055 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:18:41.051+0000 7fc95ea00640 1 -- 172.21.15.118:0/1701889729 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55f3c8a39210 con 0x55f3c8dbf1e0 2024-03-24T00:18:41.055 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:18:41.051+0000 7fc95ea00640 1 --2- 172.21.15.118:0/1701889729 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55f3c8dbf1e0 0x55f3c8dc9350 secure :-1 s=READY pgs=359 cs=0 l=1 rev1=1 crypto rx=0x7fc950009fd0 tx=0x7fc950004730 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c599553314ef14eb server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:18:41.055 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:18:41.051+0000 7fc94cc00640 10 client.0 ms_handle_connect on v2:172.21.15.189:3300/0 2024-03-24T00:18:41.055 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:18:41.052+0000 7fc94cc00640 1 -- 172.21.15.118:0/1701889729 <== mon.2 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc95002b060 con 0x55f3c8dbf1e0 2024-03-24T00:18:41.055 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:18:41.052+0000 7fc9689d5740 1 -- 172.21.15.118:0/1701889729 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fc958004050 con 0x55f3c8dbf1e0 2024-03-24T00:18:41.055 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:18:41.052+0000 7fc94cc00640 1 -- 172.21.15.118:0/1701889729 <== mon.2 v2:172.21.15.189:3300/0 2 ==== osd_map(90..90 src has 1..90) v4 ==== 8277+0+0 (secure 0 0 0) 0x7fc95001cb90 con 0x55f3c8dbf1e0 2024-03-24T00:18:41.055 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:18:41.052+0000 7fc94cc00640 1 -- 172.21.15.118:0/1701889729 <== mon.2 v2:172.21.15.189:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) v2 ==== 24+0+0 (secure 0 0 0) 0x7fc95001cd90 con 0x55f3c8dbf1e0 2024-03-24T00:18:41.055 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:18:41.052+0000 7fc9689d5740 10 client.26333 fetch_fsmap learned FSMap version 28 2024-03-24T00:18:41.055 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:18:41.052+0000 7fc9689d5740 1 -- 172.21.15.118:0/1701889729 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) v3 -- 0x55f3c8b05620 con 0x55f3c8dbf1e0 2024-03-24T00:18:41.055 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:18:41.052+0000 7fc94cc00640 1 -- 172.21.15.118:0/1701889729 <== mon.2 v2:172.21.15.189:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fc958004050 con 0x55f3c8dbf1e0 2024-03-24T00:18:41.056 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:18:41.052+0000 7fc94cc00640 1 -- 172.21.15.118:0/1701889729 <== mon.2 v2:172.21.15.189:3300/0 5 ==== fsmap(e 28) v1 ==== 7934+0+0 (secure 0 0 0) 0x7fc95000e570 con 0x55f3c8dbf1e0 2024-03-24T00:18:41.056 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:18:41.052+0000 7fc9689d5740 10 client.26333 fetch_fsmap finished waiting for FSMap version 28 2024-03-24T00:18:41.056 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:18:41.052+0000 7fc9689d5740 10 client.26333 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-24T00:18:41.056 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:18:41.053+0000 7fc9689d5740 20 client.26333 populate_metadata read hostname 'smithi118' 2024-03-24T00:18:41.056 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:18:41.053+0000 7fc9689d5740 1 --2- 172.21.15.118:0/1701889729 >> [v2:172.21.15.189:6838/4074484168,v1:172.21.15.189:6839/4074484168] conn(0x55f3c8dd51b0 0x55f3c8dd7630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:18:41.056 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:18:41.053+0000 7fc9689d5740 4 client.26333 mds_command: new command op to 24569 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-24T00:18:41.056 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:18:41.053+0000 7fc9689d5740 1 -- 172.21.15.118:0/1701889729 --> [v2:172.21.15.189:6838/4074484168,v1:172.21.15.189:6839/4074484168] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55f3c8d31d70 con 0x55f3c8dd51b0 2024-03-24T00:18:41.056 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:18:41.053+0000 7fc956000640 1 --2- 172.21.15.118:0/1701889729 >> [v2:172.21.15.189:6838/4074484168,v1:172.21.15.189:6839/4074484168] conn(0x55f3c8dd51b0 0x55f3c8dd7630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:18:41.056 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:18:41.053+0000 7fc956000640 1 --2- 172.21.15.118:0/1701889729 >> [v2:172.21.15.189:6838/4074484168,v1:172.21.15.189:6839/4074484168] conn(0x55f3c8dd51b0 0x55f3c8dd7630 crc :-1 s=READY pgs=101 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=c46782153d7fb305 server_cookie=57d89f993e4b0f94 in_seq=0 out_seq=0 2024-03-24T00:18:41.056 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:18:41.053+0000 7fc94cc00640 10 client.26333 ms_handle_connect on v2:172.21.15.189:6838/4074484168 2024-03-24T00:18:41.057 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:18:41.055+0000 7fc94cc00640 1 -- 172.21.15.118:0/1701889729 <== mds.0 v2:172.21.15.189:6838/4074484168 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x55f3c8d31d70 con 0x55f3c8dd51b0 2024-03-24T00:18:41.057 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:18:41.055+0000 7fc94cc00640 10 client.26333 handle_command_reply: tid=0 2024-03-24T00:18:41.057 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:18:41.055+0000 7fc9689d5740 1 client.26333 shutdown 2024-03-24T00:18:41.058 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:18:41.055+0000 7fc9689d5740 1 -- 172.21.15.118:0/1701889729 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55f3c8dbf1e0 msgr2=0x55f3c8dc9350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:18:41.058 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:18:41.055+0000 7fc9689d5740 1 --2- 172.21.15.118:0/1701889729 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55f3c8dbf1e0 0x55f3c8dc9350 secure :-1 s=READY pgs=359 cs=0 l=1 rev1=1 crypto rx=0x7fc950009fd0 tx=0x7fc950004730 comp rx=0 tx=0).stop 2024-03-24T00:18:41.058 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:18:41.055+0000 7fc9689d5740 1 -- 172.21.15.118:0/1701889729 shutdown_connections 2024-03-24T00:18:41.058 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:18:41.055+0000 7fc9689d5740 1 --2- 172.21.15.118:0/1701889729 >> [v2:172.21.15.189:6838/4074484168,v1:172.21.15.189:6839/4074484168] conn(0x55f3c8dd51b0 0x55f3c8dd7630 crc :-1 s=READY pgs=101 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:18:41.058 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:18:41.056+0000 7fc9689d5740 1 --2- 172.21.15.118:0/1701889729 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55f3c8dc4d40 0x55f3c8dd08d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:18:41.058 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:18:41.056+0000 7fc9689d5740 1 --2- 172.21.15.118:0/1701889729 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55f3c8dbfbd0 0x55f3c8dc9890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:18:41.058 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:18:41.056+0000 7fc9689d5740 1 --2- 172.21.15.118:0/1701889729 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55f3c8dbf1e0 0x55f3c8dc9350 unknown :-1 s=CLOSED pgs=359 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:18:41.058 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:18:41.056+0000 7fc94cc00640 0 client.26333 ms_handle_reset on v2:172.21.15.189:6838/4074484168 2024-03-24T00:18:41.058 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:18:41.056+0000 7fc9689d5740 1 -- 172.21.15.118:0/1701889729 >> 172.21.15.118:0/1701889729 conn(0x55f3c8dba750 msgr2=0x55f3c8db9950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:18:41.058 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:18:41.056+0000 7fc9689d5740 1 -- 172.21.15.118:0/1701889729 shutdown_connections 2024-03-24T00:18:41.058 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:18:41.056+0000 7fc9689d5740 1 -- 172.21.15.118:0/1701889729 wait complete. 2024-03-24T00:18:41.058 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:18:41.056+0000 7fc9689d5740 20 client.26333 trim_cache size 0 max 16384 2024-03-24T00:18:41.073 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:18:41.072+0000 7fc9689d5740 1 Processor -- start 2024-03-24T00:18:41.073 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:18:41.072+0000 7fc9689d5740 1 -- start start 2024-03-24T00:18:41.073 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:18:41.072+0000 7fc9689d5740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55f3c8dbf1e0 0x55f3c8dc8f60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:18:41.073 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:18:41.072+0000 7fc9689d5740 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55f3c8dbfbd0 0x55f3c8dd0890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:18:41.073 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:18:41.072+0000 7fc9689d5740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55f3c8dc4d40 0x55f3c8dc94a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:18:41.073 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:18:41.072+0000 7fc9689d5740 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc9601aab40 con 0x55f3c8dbfbd0 2024-03-24T00:18:41.073 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:18:41.072+0000 7fc9689d5740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc9601ac4c0 con 0x55f3c8dc4d40 2024-03-24T00:18:41.073 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:18:41.072+0000 7fc9689d5740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc9601ab1a0 con 0x55f3c8dbf1e0 2024-03-24T00:18:41.073 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:18:41.072+0000 7fc95ea00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55f3c8dbf1e0 0x55f3c8dc8f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:18:41.073 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:18:41.072+0000 7fc956000640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55f3c8dbfbd0 0x55f3c8dd0890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:18:41.074 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:18:41.073+0000 7fc95ea00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55f3c8dbf1e0 0x55f3c8dc8f60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.118:53344/0 (socket says 172.21.15.118:53344) 2024-03-24T00:18:41.074 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:18:41.073+0000 7fc95ea00640 1 -- 172.21.15.118:0/3671776648 learned_addr learned my addr 172.21.15.118:0/3671776648 (peer_addr_for_me v2:172.21.15.118:0/0) 2024-03-24T00:18:41.074 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:18:41.073+0000 7fc95f400640 1 --2- 172.21.15.118:0/3671776648 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55f3c8dc4d40 0x55f3c8dc94a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:18:41.074 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:18:41.073+0000 7fc95ea00640 1 -- 172.21.15.118:0/3671776648 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55f3c8dc4d40 msgr2=0x55f3c8dc94a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:18:41.074 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:18:41.073+0000 7fc95ea00640 1 --2- 172.21.15.118:0/3671776648 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55f3c8dc4d40 0x55f3c8dc94a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:18:41.074 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:18:41.073+0000 7fc95ea00640 1 -- 172.21.15.118:0/3671776648 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55f3c8dbfbd0 msgr2=0x55f3c8dd0890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:18:41.074 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:18:41.073+0000 7fc95ea00640 1 --2- 172.21.15.118:0/3671776648 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55f3c8dbfbd0 0x55f3c8dd0890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:18:41.074 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:18:41.073+0000 7fc95ea00640 1 -- 172.21.15.118:0/3671776648 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55f3c8dbf5c0 con 0x55f3c8dbf1e0 2024-03-24T00:18:41.074 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:18:41.073+0000 7fc956000640 1 --2- 172.21.15.118:0/3671776648 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55f3c8dbfbd0 0x55f3c8dd0890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T00:18:41.074 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:18:41.073+0000 7fc95ea00640 1 --2- 172.21.15.118:0/3671776648 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55f3c8dbf1e0 0x55f3c8dc8f60 secure :-1 s=READY pgs=357 cs=0 l=1 rev1=1 crypto rx=0x7fc95000e960 tx=0x7fc950005cf0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=3cf06350a04fea39 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:18:41.074 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:18:41.073+0000 7fc94f400640 1 -- 172.21.15.118:0/3671776648 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc95000ebd0 con 0x55f3c8dbf1e0 2024-03-24T00:18:41.074 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:18:41.073+0000 7fc94f400640 1 -- 172.21.15.118:0/3671776648 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fc95000ed70 con 0x55f3c8dbf1e0 2024-03-24T00:18:41.075 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:18:41.073+0000 7fc94f400640 1 -- 172.21.15.118:0/3671776648 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc950031a50 con 0x55f3c8dbf1e0 2024-03-24T00:18:41.078 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:18:41.074+0000 7fc9689d5740 1 -- 172.21.15.118:0/3671776648 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55f3c8dbf1e0 msgr2=0x55f3c8dc8f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:18:41.078 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:18:41.074+0000 7fc9689d5740 1 --2- 172.21.15.118:0/3671776648 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55f3c8dbf1e0 0x55f3c8dc8f60 secure :-1 s=READY pgs=357 cs=0 l=1 rev1=1 crypto rx=0x7fc95000e960 tx=0x7fc950005cf0 comp rx=0 tx=0).stop 2024-03-24T00:18:41.078 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:18:41.074+0000 7fc9689d5740 1 -- 172.21.15.118:0/3671776648 shutdown_connections 2024-03-24T00:18:41.078 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:18:41.074+0000 7fc9689d5740 1 --2- 172.21.15.118:0/3671776648 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55f3c8dc4d40 0x55f3c8dc94a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:18:41.078 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:18:41.074+0000 7fc9689d5740 1 --2- 172.21.15.118:0/3671776648 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55f3c8dbfbd0 0x55f3c8dd0890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:18:41.078 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:18:41.074+0000 7fc9689d5740 1 --2- 172.21.15.118:0/3671776648 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55f3c8dbf1e0 0x55f3c8dc8f60 unknown :-1 s=CLOSED pgs=357 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:18:41.078 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:18:41.074+0000 7fc9689d5740 1 -- 172.21.15.118:0/3671776648 >> 172.21.15.118:0/3671776648 conn(0x55f3c8dba750 msgr2=0x55f3c8dbab30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:18:41.078 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:18:41.074+0000 7fc9689d5740 1 -- 172.21.15.118:0/3671776648 shutdown_connections 2024-03-24T00:18:41.078 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:18:41.074+0000 7fc9689d5740 1 -- 172.21.15.118:0/3671776648 wait complete. 2024-03-24T00:18:41.078 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:18:41.075+0000 7fc9689d5740 1 -- start start 2024-03-24T00:18:41.078 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:18:41.075+0000 7fc9689d5740 1 Processor -- start 2024-03-24T00:18:41.079 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:18:41.075+0000 7fc9689d5740 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55f3c8dbf1e0 0x55f3c8dd0030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:18:41.079 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:18:41.075+0000 7fc9689d5740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55f3c8dbfbd0 0x55f3c8dd0570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:18:41.079 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:18:41.075+0000 7fc9689d5740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55f3c8dc4d40 0x55f3c8dce640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:18:41.079 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:18:41.075+0000 7fc9689d5740 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc9601abe60 con 0x55f3c8dbf1e0 2024-03-24T00:18:41.079 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:18:41.075+0000 7fc9689d5740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc9601ab800 con 0x55f3c8dc4d40 2024-03-24T00:18:41.079 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:18:41.075+0000 7fc9689d5740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc9601a8d00 con 0x55f3c8dbfbd0 2024-03-24T00:18:41.079 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:18:41.076+0000 7fc9689d5740 10 client.0 _ll_register_callbacks cb 0x7fc950009e50 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T00:18:41.079 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:18:41.076+0000 7fc956000640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55f3c8dbfbd0 0x55f3c8dd0570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:18:41.079 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:18:41.076+0000 7fc95f400640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55f3c8dc4d40 0x55f3c8dce640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:18:41.079 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:18:41.076+0000 7fc956000640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55f3c8dbfbd0 0x55f3c8dd0570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.118:53346/0 (socket says 172.21.15.118:53346) 2024-03-24T00:18:41.079 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:18:41.076+0000 7fc956000640 1 -- 172.21.15.118:0/1875661910 learned_addr learned my addr 172.21.15.118:0/1875661910 (peer_addr_for_me v2:172.21.15.118:0/0) 2024-03-24T00:18:41.079 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:18:41.076+0000 7fc95ea00640 1 --2- 172.21.15.118:0/1875661910 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55f3c8dbf1e0 0x55f3c8dd0030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:18:41.079 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:18:41.076+0000 7fc956000640 1 -- 172.21.15.118:0/1875661910 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55f3c8dc4d40 msgr2=0x55f3c8dce640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:18:41.079 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:18:41.076+0000 7fc956000640 1 --2- 172.21.15.118:0/1875661910 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55f3c8dc4d40 0x55f3c8dce640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:18:41.080 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:18:41.076+0000 7fc956000640 1 -- 172.21.15.118:0/1875661910 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55f3c8dbf1e0 msgr2=0x55f3c8dd0030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:18:41.080 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:18:41.076+0000 7fc956000640 1 --2- 172.21.15.118:0/1875661910 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55f3c8dbf1e0 0x55f3c8dd0030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:18:41.080 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:18:41.076+0000 7fc956000640 1 -- 172.21.15.118:0/1875661910 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55f3c8dcec80 con 0x55f3c8dbfbd0 2024-03-24T00:18:41.080 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:18:41.076+0000 7fc95f400640 1 --2- 172.21.15.118:0/1875661910 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55f3c8dc4d40 0x55f3c8dce640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T00:18:41.080 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:18:41.076+0000 7fc956000640 1 --2- 172.21.15.118:0/1875661910 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55f3c8dbfbd0 0x55f3c8dd0570 secure :-1 s=READY pgs=358 cs=0 l=1 rev1=1 crypto rx=0x7fc9480bd280 tx=0x7fc9480dd040 comp rx=0 tx=0).ready entity=mon.1 client_cookie=a81283390e82d311 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:18:41.080 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:18:41.076+0000 7fc94cc00640 10 client.0 ms_handle_connect on v2:172.21.15.193:3300/0 2024-03-24T00:18:41.080 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:18:41.077+0000 7fc94cc00640 1 -- 172.21.15.118:0/1875661910 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc9480b7e60 con 0x55f3c8dbfbd0 2024-03-24T00:18:41.080 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:18:41.077+0000 7fc94cc00640 1 -- 172.21.15.118:0/1875661910 <== mon.1 v2:172.21.15.193:3300/0 2 ==== osd_map(90..90 src has 1..90) v4 ==== 8277+0+0 (secure 0 0 0) 0x7fc948003dd0 con 0x55f3c8dbfbd0 2024-03-24T00:18:41.080 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:18:41.077+0000 7fc9689d5740 1 -- 172.21.15.118:0/1875661910 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fc95000ed70 con 0x55f3c8dbfbd0 2024-03-24T00:18:41.080 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:18:41.077+0000 7fc94cc00640 1 -- 172.21.15.118:0/1875661910 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) v2 ==== 24+0+0 (secure 0 0 0) 0x7fc948005690 con 0x55f3c8dbfbd0 2024-03-24T00:18:41.080 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:18:41.077+0000 7fc9689d5740 10 client.26353 fetch_fsmap learned FSMap version 28 2024-03-24T00:18:41.080 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:18:41.077+0000 7fc9689d5740 1 -- 172.21.15.118:0/1875661910 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) v3 -- 0x55f3c8e314a0 con 0x55f3c8dbfbd0 2024-03-24T00:18:41.080 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:18:41.077+0000 7fc94cc00640 1 -- 172.21.15.118:0/1875661910 <== mon.1 v2:172.21.15.193:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fc95000ed70 con 0x55f3c8dbfbd0 2024-03-24T00:18:41.081 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:18:41.077+0000 7fc94cc00640 1 -- 172.21.15.118:0/1875661910 <== mon.1 v2:172.21.15.193:3300/0 5 ==== fsmap(e 28) v1 ==== 7934+0+0 (secure 0 0 0) 0x7fc9480b9960 con 0x55f3c8dbfbd0 2024-03-24T00:18:41.081 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:18:41.077+0000 7fc9689d5740 10 client.26353 fetch_fsmap finished waiting for FSMap version 28 2024-03-24T00:18:41.081 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:18:41.077+0000 7fc9689d5740 10 client.26353 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-24T00:18:41.081 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:18:41.077+0000 7fc9689d5740 20 client.26353 populate_metadata read hostname 'smithi118' 2024-03-24T00:18:41.081 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:18:41.077+0000 7fc9689d5740 1 --2- 172.21.15.118:0/1875661910 >> [v2:172.21.15.189:6838/4074484168,v1:172.21.15.189:6839/4074484168] conn(0x55f3c8e31e00 0x55f3c8e34220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:18:41.081 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:18:41.077+0000 7fc9689d5740 4 client.26353 mds_command: new command op to 24569 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-03-24T00:18:41.081 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:18:41.077+0000 7fc9689d5740 1 -- 172.21.15.118:0/1875661910 --> [v2:172.21.15.189:6838/4074484168,v1:172.21.15.189:6839/4074484168] -- command(tid 0: {"prefix": "flush journal"}) v1 -- 0x55f3c8d2c0c0 con 0x55f3c8e31e00 2024-03-24T00:18:41.081 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:18:41.078+0000 7fc95ea00640 1 --2- 172.21.15.118:0/1875661910 >> [v2:172.21.15.189:6838/4074484168,v1:172.21.15.189:6839/4074484168] conn(0x55f3c8e31e00 0x55f3c8e34220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:18:41.081 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:18:41.078+0000 7fc95ea00640 1 --2- 172.21.15.118:0/1875661910 >> [v2:172.21.15.189:6838/4074484168,v1:172.21.15.189:6839/4074484168] conn(0x55f3c8e31e00 0x55f3c8e34220 crc :-1 s=READY pgs=102 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=b7b8fae543dec07c server_cookie=7fa9e91c5968c2ef in_seq=0 out_seq=0 2024-03-24T00:18:41.081 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:18:41.078+0000 7fc94cc00640 10 client.26353 ms_handle_connect on v2:172.21.15.189:6838/4074484168 2024-03-24T00:18:41.197 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:18:40 smithi118 ceph-mon[31535]: pgmap v487: 97 pgs: 97 active+clean; 52 MiB data, 499 MiB used, 1.0 TiB / 1.0 TiB avail; 919 KiB/s rd, 2.2 MiB/s wr, 396 op/s 2024-03-24T00:18:41.203 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:18:40 smithi189 ceph-mon[36187]: pgmap v487: 97 pgs: 97 active+clean; 52 MiB data, 499 MiB used, 1.0 TiB / 1.0 TiB avail; 919 KiB/s rd, 2.2 MiB/s wr, 396 op/s 2024-03-24T00:18:41.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:18:40 smithi193 ceph-mon[36878]: pgmap v487: 97 pgs: 97 active+clean; 52 MiB data, 499 MiB used, 1.0 TiB / 1.0 TiB avail; 919 KiB/s rd, 2.2 MiB/s wr, 396 op/s 2024-03-24T00:18:41.395 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:18:41.395+0000 7fc94cc00640 1 -- 172.21.15.118:0/1875661910 <== mds.0 v2:172.21.15.189:6838/4074484168 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+44 (crc 0 0 0) 0x55f3c8d2c0c0 con 0x55f3c8e31e00 2024-03-24T00:18:41.395 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:18:41.395+0000 7fc94cc00640 10 client.26353 handle_command_reply: tid=0 2024-03-24T00:18:41.396 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:18:41.395+0000 7fc9689d5740 1 client.26353 shutdown 2024-03-24T00:18:41.396 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:18:41.395+0000 7fc9689d5740 1 -- 172.21.15.118:0/1875661910 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55f3c8dbfbd0 msgr2=0x55f3c8dd0570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:18:41.396 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:18:41.395+0000 7fc9689d5740 1 --2- 172.21.15.118:0/1875661910 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55f3c8dbfbd0 0x55f3c8dd0570 secure :-1 s=READY pgs=358 cs=0 l=1 rev1=1 crypto rx=0x7fc9480bd280 tx=0x7fc9480dd040 comp rx=0 tx=0).stop 2024-03-24T00:18:41.396 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:18:41.395+0000 7fc9689d5740 1 -- 172.21.15.118:0/1875661910 shutdown_connections 2024-03-24T00:18:41.396 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:18:41.395+0000 7fc9689d5740 1 --2- 172.21.15.118:0/1875661910 >> [v2:172.21.15.189:6838/4074484168,v1:172.21.15.189:6839/4074484168] conn(0x55f3c8e31e00 0x55f3c8e34220 crc :-1 s=READY pgs=102 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:18:41.396 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:18:41.396+0000 7fc9689d5740 1 --2- 172.21.15.118:0/1875661910 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55f3c8dc4d40 0x55f3c8dce640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:18:41.396 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:18:41.396+0000 7fc9689d5740 1 --2- 172.21.15.118:0/1875661910 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55f3c8dbfbd0 0x55f3c8dd0570 unknown :-1 s=CLOSED pgs=358 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:18:41.396 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:18:41.396+0000 7fc9689d5740 1 --2- 172.21.15.118:0/1875661910 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55f3c8dbf1e0 0x55f3c8dd0030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:18:41.396 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:18:41.396+0000 7fc94cc00640 0 client.26353 ms_handle_reset on v2:172.21.15.189:6838/4074484168 2024-03-24T00:18:41.396 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:18:41.396+0000 7fc9689d5740 1 -- 172.21.15.118:0/1875661910 >> 172.21.15.118:0/1875661910 conn(0x55f3c8dba750 msgr2=0x55f3c8dc8f40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:18:41.396 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:18:41.396+0000 7fc9689d5740 1 -- 172.21.15.118:0/1875661910 shutdown_connections 2024-03-24T00:18:41.396 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:18:41.396+0000 7fc9689d5740 1 -- 172.21.15.118:0/1875661910 wait complete. 2024-03-24T00:18:41.396 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:18:41.396+0000 7fc9689d5740 20 client.26353 trim_cache size 0 max 16384 2024-03-24T00:18:41.397 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stdout:{ 2024-03-24T00:18:41.397 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stdout: "message": "", 2024-03-24T00:18:41.397 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stdout: "return_code": 0 2024-03-24T00:18:41.397 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stdout:} 2024-03-24T00:18:41.398 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:18:41.397+0000 7fc965c00640 1 -- 172.21.15.118:0/1310209229 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7fc9440a2920 msgr2=0x7fc9440a4de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:18:41.398 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:18:41.397+0000 7fc965c00640 1 --2- 172.21.15.118:0/1310209229 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7fc9440a2920 0x7fc9440a4de0 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 crypto rx=0x55f3c8dbffd0 tx=0x7fc950005f90 comp rx=0 tx=0).stop 2024-03-24T00:18:41.398 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:18:41.397+0000 7fc965c00640 1 -- 172.21.15.118:0/1310209229 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc9601028c0 msgr2=0x7fc960197e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:18:41.398 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:18:41.397+0000 7fc965c00640 1 --2- 172.21.15.118:0/1310209229 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc9601028c0 0x7fc960197e10 secure :-1 s=READY pgs=358 cs=0 l=1 rev1=1 crypto rx=0x7fc948036040 tx=0x7fc94802c650 comp rx=0 tx=0).stop 2024-03-24T00:18:41.398 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:18:41.397+0000 7fc965c00640 1 -- 172.21.15.118:0/1310209229 shutdown_connections 2024-03-24T00:18:41.398 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:18:41.397+0000 7fc965c00640 1 --2- 172.21.15.118:0/1310209229 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7fc9440a2920 0x7fc9440a4de0 unknown :-1 s=CLOSED pgs=328 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:18:41.398 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:18:41.397+0000 7fc965c00640 1 --2- 172.21.15.118:0/1310209229 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fc960103280 0x7fc96019ee80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:18:41.398 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:18:41.397+0000 7fc965c00640 1 --2- 172.21.15.118:0/1310209229 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc9601028c0 0x7fc960197e10 unknown :-1 s=CLOSED pgs=358 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:18:41.398 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:18:41.397+0000 7fc965c00640 1 --2- 172.21.15.118:0/1310209229 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fc9601016d0 0x7fc9601978d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:18:41.399 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:18:41.397+0000 7fc965c00640 1 -- 172.21.15.118:0/1310209229 >> 172.21.15.118:0/1310209229 conn(0x7fc9600fcc60 msgr2=0x7fc960105af0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:18:41.399 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:18:41.397+0000 7fc965c00640 1 -- 172.21.15.118:0/1310209229 shutdown_connections 2024-03-24T00:18:41.399 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:18:41.397+0000 7fc965c00640 1 -- 172.21.15.118:0/1310209229 wait complete. 2024-03-24T00:18:42.195 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:18:41 smithi118 ceph-mon[31535]: scrub summary: idle+waiting paths [/] 2024-03-24T00:18:42.203 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:18:41 smithi189 ceph-mon[36187]: scrub summary: idle+waiting paths [/] 2024-03-24T00:18:42.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:18:41 smithi193 ceph-mon[36878]: scrub summary: idle+waiting paths [/] 2024-03-24T00:18:43.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:18:42 smithi118 ceph-mon[31535]: pgmap v488: 97 pgs: 97 active+clean; 52 MiB data, 499 MiB used, 1.0 TiB / 1.0 TiB avail; 741 KiB/s rd, 1.7 MiB/s wr, 364 op/s 2024-03-24T00:18:43.203 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:18:42 smithi189 ceph-mon[36187]: pgmap v488: 97 pgs: 97 active+clean; 52 MiB data, 499 MiB used, 1.0 TiB / 1.0 TiB avail; 741 KiB/s rd, 1.7 MiB/s wr, 364 op/s 2024-03-24T00:18:43.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:18:42 smithi193 ceph-mon[36878]: pgmap v488: 97 pgs: 97 active+clean; 52 MiB data, 499 MiB used, 1.0 TiB / 1.0 TiB avail; 741 KiB/s rd, 1.7 MiB/s wr, 364 op/s 2024-03-24T00:18:44.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:18:43 smithi118 ceph-mon[31535]: pgmap v489: 97 pgs: 97 active+clean; 53 MiB data, 524 MiB used, 1.0 TiB / 1.0 TiB avail; 990 KiB/s rd, 1.5 MiB/s wr, 489 op/s 2024-03-24T00:18:44.203 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:18:43 smithi189 ceph-mon[36187]: pgmap v489: 97 pgs: 97 active+clean; 53 MiB data, 524 MiB used, 1.0 TiB / 1.0 TiB avail; 990 KiB/s rd, 1.5 MiB/s wr, 489 op/s 2024-03-24T00:18:44.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:18:43 smithi193 ceph-mon[36878]: pgmap v489: 97 pgs: 97 active+clean; 53 MiB data, 524 MiB used, 1.0 TiB / 1.0 TiB avail; 990 KiB/s rd, 1.5 MiB/s wr, 489 op/s 2024-03-24T00:18:45.699 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:18:45 smithi118 ceph-mon[31535]: scrub summary: idle 2024-03-24T00:18:45.703 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:18:45 smithi189 ceph-mon[36187]: scrub summary: idle 2024-03-24T00:18:45.710 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:18:45 smithi193 ceph-mon[36878]: scrub summary: idle 2024-03-24T00:18:46.695 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:18:46 smithi118 ceph-mon[31535]: pgmap v490: 97 pgs: 97 active+clean; 45 MiB data, 518 MiB used, 1.0 TiB / 1.0 TiB avail; 971 KiB/s rd, 901 KiB/s wr, 459 op/s 2024-03-24T00:18:46.703 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:18:46 smithi189 ceph-mon[36187]: pgmap v490: 97 pgs: 97 active+clean; 45 MiB data, 518 MiB used, 1.0 TiB / 1.0 TiB avail; 971 KiB/s rd, 901 KiB/s wr, 459 op/s 2024-03-24T00:18:46.710 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:18:46 smithi193 ceph-mon[36878]: pgmap v490: 97 pgs: 97 active+clean; 45 MiB data, 518 MiB used, 1.0 TiB / 1.0 TiB avail; 971 KiB/s rd, 901 KiB/s wr, 459 op/s 2024-03-24T00:18:46.842 INFO:tasks.workunit.client.0.smithi118.stdout: 17 94174 109 62165 160 47285 327 2024-03-24T00:18:47.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:18:47 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T00:18:47.703 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:18:47 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T00:18:47.710 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:18:47 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T00:18:48.697 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:18:48 smithi118 ceph-mon[31535]: pgmap v491: 97 pgs: 97 active+clean; 45 MiB data, 519 MiB used, 1.0 TiB / 1.0 TiB avail; 681 KiB/s rd, 580 KiB/s wr, 434 op/s 2024-03-24T00:18:48.704 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:18:48 smithi189 ceph-mon[36187]: pgmap v491: 97 pgs: 97 active+clean; 45 MiB data, 519 MiB used, 1.0 TiB / 1.0 TiB avail; 681 KiB/s rd, 580 KiB/s wr, 434 op/s 2024-03-24T00:18:48.710 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:18:48 smithi193 ceph-mon[36878]: pgmap v491: 97 pgs: 97 active+clean; 45 MiB data, 519 MiB used, 1.0 TiB / 1.0 TiB avail; 681 KiB/s rd, 580 KiB/s wr, 434 op/s 2024-03-24T00:18:49.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:18:49 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T00:18:49.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:18:49 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T00:18:49.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:18:49 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T00:18:49.703 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:18:49 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T00:18:49.703 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:18:49 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T00:18:49.704 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:18:49 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T00:18:49.710 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:18:49 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T00:18:49.710 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:18:49 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T00:18:49.710 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:18:49 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T00:18:50.698 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:18:50 smithi118 ceph-mon[31535]: pgmap v492: 97 pgs: 97 active+clean; 49 MiB data, 527 MiB used, 1.0 TiB / 1.0 TiB avail; 703 KiB/s rd, 948 KiB/s wr, 451 op/s 2024-03-24T00:18:50.703 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:18:50 smithi189 ceph-mon[36187]: pgmap v492: 97 pgs: 97 active+clean; 49 MiB data, 527 MiB used, 1.0 TiB / 1.0 TiB avail; 703 KiB/s rd, 948 KiB/s wr, 451 op/s 2024-03-24T00:18:50.710 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:18:50 smithi193 ceph-mon[36878]: pgmap v492: 97 pgs: 97 active+clean; 49 MiB data, 527 MiB used, 1.0 TiB / 1.0 TiB avail; 703 KiB/s rd, 948 KiB/s wr, 451 op/s 2024-03-24T00:18:53.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:18:52 smithi118 ceph-mon[31535]: pgmap v493: 97 pgs: 97 active+clean; 58 MiB data, 535 MiB used, 1.0 TiB / 1.0 TiB avail; 754 KiB/s rd, 1.5 MiB/s wr, 464 op/s 2024-03-24T00:18:53.203 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:18:52 smithi189 ceph-mon[36187]: pgmap v493: 97 pgs: 97 active+clean; 58 MiB data, 535 MiB used, 1.0 TiB / 1.0 TiB avail; 754 KiB/s rd, 1.5 MiB/s wr, 464 op/s 2024-03-24T00:18:53.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:18:52 smithi193 ceph-mon[36878]: pgmap v493: 97 pgs: 97 active+clean; 58 MiB data, 535 MiB used, 1.0 TiB / 1.0 TiB avail; 754 KiB/s rd, 1.5 MiB/s wr, 464 op/s 2024-03-24T00:18:54.563 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:18:54.561+0000 7fe3b9800640 1 -- 172.21.15.118:0/2379602977 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe3b41054b0 msgr2=0x7fe3b410bd40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:18:54.563 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:18:54.561+0000 7fe3b9800640 1 --2- 172.21.15.118:0/2379602977 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe3b41054b0 0x7fe3b410bd40 secure :-1 s=READY pgs=360 cs=0 l=1 rev1=1 crypto rx=0x7fe3ac009ca0 tx=0x7fe3ac02c100 comp rx=0 tx=0).stop 2024-03-24T00:18:54.563 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:18:54.562+0000 7fe3b9800640 1 -- 172.21.15.118:0/2379602977 shutdown_connections 2024-03-24T00:18:54.563 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:18:54.562+0000 7fe3b9800640 1 --2- 172.21.15.118:0/2379602977 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe3b41054b0 0x7fe3b410bd40 unknown :-1 s=CLOSED pgs=360 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:18:54.563 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:18:54.562+0000 7fe3b9800640 1 --2- 172.21.15.118:0/2379602977 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fe3b4104af0 0x7fe3b4104f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:18:54.563 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:18:54.562+0000 7fe3b9800640 1 --2- 172.21.15.118:0/2379602977 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fe3b4103900 0x7fe3b4103d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:18:54.563 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:18:54.562+0000 7fe3b9800640 1 -- 172.21.15.118:0/2379602977 >> 172.21.15.118:0/2379602977 conn(0x7fe3b40fee90 msgr2=0x7fe3b41012b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:18:54.563 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:18:54.562+0000 7fe3b9800640 1 -- 172.21.15.118:0/2379602977 shutdown_connections 2024-03-24T00:18:54.563 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:18:54.562+0000 7fe3b9800640 1 -- 172.21.15.118:0/2379602977 wait complete. 2024-03-24T00:18:54.564 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:18:54.563+0000 7fe3b9800640 1 Processor -- start 2024-03-24T00:18:54.564 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:18:54.563+0000 7fe3b9800640 1 -- start start 2024-03-24T00:18:54.565 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:18:54.564+0000 7fe3b9800640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fe3b4103900 0x7fe3b4199ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:18:54.565 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:18:54.564+0000 7fe3b9800640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fe3b4104af0 0x7fe3b4199ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:18:54.565 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:18:54.564+0000 7fe3b9800640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe3b41054b0 0x7fe3b41a1060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:18:54.565 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:18:54.564+0000 7fe3b9800640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe3b410f560 con 0x7fe3b4104af0 2024-03-24T00:18:54.565 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:18:54.564+0000 7fe3b9800640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe3b410f3e0 con 0x7fe3b41054b0 2024-03-24T00:18:54.565 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:18:54.564+0000 7fe3b9800640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe3b410f6e0 con 0x7fe3b4103900 2024-03-24T00:18:54.565 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:18:54.564+0000 7fe3b2a00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fe3b4103900 0x7fe3b4199ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:18:54.566 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:18:54.564+0000 7fe3b3400640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe3b41054b0 0x7fe3b41a1060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:18:54.566 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:18:54.564+0000 7fe3b2000640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fe3b4104af0 0x7fe3b4199ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:18:54.566 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:18:54.564+0000 7fe3b3400640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe3b41054b0 0x7fe3b41a1060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.118:47794/0 (socket says 172.21.15.118:47794) 2024-03-24T00:18:54.566 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:18:54.564+0000 7fe3b2a00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fe3b4103900 0x7fe3b4199ab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.118:34620/0 (socket says 172.21.15.118:34620) 2024-03-24T00:18:54.566 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:18:54.564+0000 7fe3b3400640 1 -- 172.21.15.118:0/4006193620 learned_addr learned my addr 172.21.15.118:0/4006193620 (peer_addr_for_me v2:172.21.15.118:0/0) 2024-03-24T00:18:54.566 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:18:54.564+0000 7fe3b3400640 1 -- 172.21.15.118:0/4006193620 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fe3b4103900 msgr2=0x7fe3b4199ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:18:54.566 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:18:54.564+0000 7fe3b3400640 1 --2- 172.21.15.118:0/4006193620 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fe3b4103900 0x7fe3b4199ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:18:54.566 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:18:54.564+0000 7fe3b3400640 1 -- 172.21.15.118:0/4006193620 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fe3b4104af0 msgr2=0x7fe3b4199ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:18:54.566 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:18:54.564+0000 7fe3b3400640 1 --2- 172.21.15.118:0/4006193620 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fe3b4104af0 0x7fe3b4199ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:18:54.566 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:18:54.564+0000 7fe3b3400640 1 -- 172.21.15.118:0/4006193620 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe3b41a16d0 con 0x7fe3b41054b0 2024-03-24T00:18:54.566 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:18:54.564+0000 7fe3b2a00640 1 --2- 172.21.15.118:0/4006193620 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fe3b4103900 0x7fe3b4199ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T00:18:54.567 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:18:54.565+0000 7fe3b3400640 1 --2- 172.21.15.118:0/4006193620 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe3b41054b0 0x7fe3b41a1060 secure :-1 s=READY pgs=361 cs=0 l=1 rev1=1 crypto rx=0x7fe3ac02c640 tx=0x7fe3ac035d10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:18:54.567 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:18:54.565+0000 7fe3ab400640 1 -- 172.21.15.118:0/4006193620 <== mon.2 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe3ac039040 con 0x7fe3b41054b0 2024-03-24T00:18:54.567 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:18:54.565+0000 7fe3ab400640 1 -- 172.21.15.118:0/4006193620 <== mon.2 v2:172.21.15.189:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fe3ac044070 con 0x7fe3b41054b0 2024-03-24T00:18:54.567 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:18:54.565+0000 7fe3b9800640 1 -- 172.21.15.118:0/4006193620 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe3b41a1960 con 0x7fe3b41054b0 2024-03-24T00:18:54.567 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:18:54.565+0000 7fe3ab400640 1 -- 172.21.15.118:0/4006193620 <== mon.2 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe3ac0043c0 con 0x7fe3b41054b0 2024-03-24T00:18:54.567 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:18:54.565+0000 7fe3b9800640 1 -- 172.21.15.118:0/4006193620 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe3b41a1e50 con 0x7fe3b41054b0 2024-03-24T00:18:54.569 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:18:54.568+0000 7fe3bc5ee740 1 Processor -- start 2024-03-24T00:18:54.570 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:18:54.570+0000 7fe3ab400640 1 -- 172.21.15.118:0/4006193620 <== mon.2 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 20) v1 ==== 137291+0+0 (secure 0 0 0) 0x7fe3ac004560 con 0x7fe3b41054b0 2024-03-24T00:18:54.573 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:18:54.570+0000 7fe3bc5ee740 1 -- start start 2024-03-24T00:18:54.573 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:18:54.571+0000 7fe3bc5ee740 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5628fad521e0 0x5628fad52600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:18:54.573 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:18:54.571+0000 7fe3bc5ee740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5628fad52bd0 0x5628fad576d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:18:54.573 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:18:54.571+0000 7fe3bc5ee740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5628fad57d40 0x5628fad5a1d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:18:54.573 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:18:54.571+0000 7fe3bc5ee740 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x5628fad153b0 con 0x5628fad521e0 2024-03-24T00:18:54.573 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:18:54.571+0000 7fe3bc5ee740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x5628facb7830 con 0x5628fad57d40 2024-03-24T00:18:54.573 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:18:54.571+0000 7fe3bc5ee740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x5628fac605d0 con 0x5628fad52bd0 2024-03-24T00:18:54.573 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:18:54.571+0000 7fe3b2000640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5628fad521e0 0x5628fad52600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:18:54.573 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:18:54.571+0000 7fe3b2000640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5628fad521e0 0x5628fad52600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.118:3300/0 says I am v2:172.21.15.118:33022/0 (socket says 172.21.15.118:33022) 2024-03-24T00:18:54.573 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:18:54.571+0000 7fe3b2000640 1 -- 172.21.15.118:0/3214860039 learned_addr learned my addr 172.21.15.118:0/3214860039 (peer_addr_for_me v2:172.21.15.118:0/0) 2024-03-24T00:18:54.573 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:18:54.571+0000 7fe3b2000640 1 --2- 172.21.15.118:0/3214860039 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5628fad57d40 0x5628fad5a1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:18:54.573 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:18:54.572+0000 7fe3ab400640 1 --2- 172.21.15.118:0/4006193620 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7fe3980a28a0 0x7fe3980a4d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:18:54.574 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:18:54.572+0000 7fe3b2000640 1 -- 172.21.15.118:0/3214860039 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5628fad52bd0 msgr2=0x5628fad576d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:18:54.574 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:18:54.572+0000 7fe3b2000640 1 --2- 172.21.15.118:0/3214860039 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5628fad52bd0 0x5628fad576d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:18:54.574 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:18:54.572+0000 7fe3b2000640 1 -- 172.21.15.118:0/3214860039 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5628fad57d40 msgr2=0x5628fad5a1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:18:54.574 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:18:54.572+0000 7fe3b2000640 1 --2- 172.21.15.118:0/3214860039 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5628fad57d40 0x5628fad5a1d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:18:54.574 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:18:54.572+0000 7fe3b2000640 1 -- 172.21.15.118:0/3214860039 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5628fac782c0 con 0x5628fad521e0 2024-03-24T00:18:54.574 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:18:54.572+0000 7fe3ab400640 1 -- 172.21.15.118:0/4006193620 <== mon.2 v2:172.21.15.189:3300/0 5 ==== osd_map(90..90 src has 1..90) v4 ==== 8277+0+0 (secure 0 0 0) 0x7fe3ac0f1040 con 0x7fe3b41054b0 2024-03-24T00:18:54.574 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:18:54.572+0000 7fe3b2000640 1 --2- 172.21.15.118:0/3214860039 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5628fad521e0 0x5628fad52600 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7fe3a0005bc0 tx=0x7fe3a0005f90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=e4f89c1267be1034 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:18:54.574 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:18:54.572+0000 7fe3b3400640 1 --2- 172.21.15.118:0/4006193620 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7fe3980a28a0 0x7fe3980a4d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:18:54.578 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:18:54.573+0000 7fe39fe00640 1 -- 172.21.15.118:0/3214860039 <== mon.0 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe3a0013db0 con 0x5628fad521e0 2024-03-24T00:18:54.578 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:18:54.573+0000 7fe39fe00640 1 -- 172.21.15.118:0/3214860039 <== mon.0 v2:172.21.15.118:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fe3a0013170 con 0x5628fad521e0 2024-03-24T00:18:54.578 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:18:54.573+0000 7fe39fe00640 1 -- 172.21.15.118:0/3214860039 <== mon.0 v2:172.21.15.118:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe3a0010990 con 0x5628fad521e0 2024-03-24T00:18:54.578 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:18:54.573+0000 7fe3bc5ee740 1 -- 172.21.15.118:0/3214860039 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5628fad521e0 msgr2=0x5628fad52600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:18:54.578 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:18:54.573+0000 7fe3bc5ee740 1 --2- 172.21.15.118:0/3214860039 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5628fad521e0 0x5628fad52600 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7fe3a0005bc0 tx=0x7fe3a0005f90 comp rx=0 tx=0).stop 2024-03-24T00:18:54.578 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:18:54.573+0000 7fe3b3400640 1 --2- 172.21.15.118:0/4006193620 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7fe3980a28a0 0x7fe3980a4d60 secure :-1 s=READY pgs=329 cs=0 l=1 rev1=1 crypto rx=0x7fe3ac0b9e10 tx=0x7fe3ac02c100 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:18:54.578 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:18:54.573+0000 7fe3bc5ee740 1 -- 172.21.15.118:0/3214860039 shutdown_connections 2024-03-24T00:18:54.578 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:18:54.573+0000 7fe3bc5ee740 1 --2- 172.21.15.118:0/3214860039 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5628fad57d40 0x5628fad5a1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:18:54.578 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:18:54.573+0000 7fe3bc5ee740 1 --2- 172.21.15.118:0/3214860039 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5628fad52bd0 0x5628fad576d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:18:54.578 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:18:54.573+0000 7fe3bc5ee740 1 --2- 172.21.15.118:0/3214860039 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5628fad521e0 0x5628fad52600 unknown :-1 s=CLOSED pgs=203 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:18:54.578 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:18:54.573+0000 7fe3bc5ee740 1 -- 172.21.15.118:0/3214860039 >> 172.21.15.118:0/3214860039 conn(0x5628fad4d750 msgr2=0x5628fad4db50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:18:54.579 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:18:54.574+0000 7fe3bc5ee740 1 -- 172.21.15.118:0/3214860039 shutdown_connections 2024-03-24T00:18:54.579 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:18:54.574+0000 7fe3bc5ee740 1 -- 172.21.15.118:0/3214860039 wait complete. 2024-03-24T00:18:54.579 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:18:54.574+0000 7fe3bc5ee740 1 -- start start 2024-03-24T00:18:54.579 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:18:54.574+0000 7fe3bc5ee740 1 Processor -- start 2024-03-24T00:18:54.579 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:18:54.575+0000 7fe3bc5ee740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5628fad521e0 0x5628fad5c3c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:18:54.579 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:18:54.575+0000 7fe3bc5ee740 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5628fad52bd0 0x5628fad5c900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:18:54.579 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:18:54.575+0000 7fe3bc5ee740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5628fad57d40 0x5628fad638d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:18:54.579 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:18:54.575+0000 7fe3bc5ee740 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x5628fac73a20 con 0x5628fad52bd0 2024-03-24T00:18:54.579 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:18:54.575+0000 7fe3bc5ee740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x5628facb38d0 con 0x5628fad521e0 2024-03-24T00:18:54.579 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:18:54.575+0000 7fe3bc5ee740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x5628fad5ace0 con 0x5628fad57d40 2024-03-24T00:18:54.579 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:18:54.575+0000 7fe3bc5ee740 10 client.0 _ll_register_callbacks cb 0x5628facb01c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T00:18:54.579 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:18:54.575+0000 7fe3b2000640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5628fad521e0 0x5628fad5c3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:18:54.579 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:18:54.575+0000 7fe3b2a00640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5628fad52bd0 0x5628fad5c900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:18:54.579 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:18:54.575+0000 7fe3b2000640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5628fad521e0 0x5628fad5c3c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.118:47826/0 (socket says 172.21.15.118:47826) 2024-03-24T00:18:54.580 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:18:54.575+0000 7fe3b2000640 1 -- 172.21.15.118:0/3346017461 learned_addr learned my addr 172.21.15.118:0/3346017461 (peer_addr_for_me v2:172.21.15.118:0/0) 2024-03-24T00:18:54.580 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:18:54.575+0000 7fe3b2000640 1 --2- 172.21.15.118:0/3346017461 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5628fad57d40 0x5628fad638d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:18:54.580 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:18:54.575+0000 7fe3b2a00640 1 -- 172.21.15.118:0/3346017461 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5628fad57d40 msgr2=0x5628fad638d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:18:54.580 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:18:54.575+0000 7fe3b2a00640 1 --2- 172.21.15.118:0/3346017461 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5628fad57d40 0x5628fad638d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:18:54.580 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:18:54.575+0000 7fe3b2a00640 1 -- 172.21.15.118:0/3346017461 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5628fad521e0 msgr2=0x5628fad5c3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:18:54.580 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:18:54.575+0000 7fe3b2a00640 1 --2- 172.21.15.118:0/3346017461 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5628fad521e0 0x5628fad5c3c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:18:54.580 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:18:54.575+0000 7fe3b2a00640 1 -- 172.21.15.118:0/3346017461 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5628fa9cc210 con 0x5628fad52bd0 2024-03-24T00:18:54.580 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:18:54.575+0000 7fe3b2000640 1 --2- 172.21.15.118:0/3346017461 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5628fad57d40 0x5628fad638d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T00:18:54.580 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:18:54.576+0000 7fe3b2a00640 1 --2- 172.21.15.118:0/3346017461 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5628fad52bd0 0x5628fad5c900 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x5628fad52fd0 tx=0x7fe3a4011e20 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a204a6333666d71f server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:18:54.580 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:18:54.576+0000 7fe39cc00640 10 client.0 ms_handle_connect on v2:172.21.15.118:3300/0 2024-03-24T00:18:54.580 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:18:54.576+0000 7fe39cc00640 1 -- 172.21.15.118:0/3346017461 <== mon.0 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe3a401e060 con 0x5628fad52bd0 2024-03-24T00:18:54.580 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:18:54.576+0000 7fe39cc00640 1 -- 172.21.15.118:0/3346017461 <== mon.0 v2:172.21.15.118:3300/0 2 ==== osd_map(90..90 src has 1..90) v4 ==== 8277+0+0 (secure 0 0 0) 0x7fe3a400fda0 con 0x5628fad52bd0 2024-03-24T00:18:54.580 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:18:54.576+0000 7fe3bc5ee740 1 -- 172.21.15.118:0/3346017461 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fe3a0013170 con 0x5628fad52bd0 2024-03-24T00:18:54.580 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:18:54.576+0000 7fe39cc00640 1 -- 172.21.15.118:0/3346017461 <== mon.0 v2:172.21.15.118:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) v2 ==== 24+0+0 (secure 0 0 0) 0x7fe3a4019430 con 0x5628fad52bd0 2024-03-24T00:18:54.580 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:18:54.576+0000 7fe3bc5ee740 10 client.16203 fetch_fsmap learned FSMap version 28 2024-03-24T00:18:54.581 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:18:54.576+0000 7fe3bc5ee740 1 -- 172.21.15.118:0/3346017461 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) v3 -- 0x5628faa98620 con 0x5628fad52bd0 2024-03-24T00:18:54.581 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:18:54.577+0000 7fe39cc00640 1 -- 172.21.15.118:0/3346017461 <== mon.0 v2:172.21.15.118:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fe3a0013170 con 0x5628fad52bd0 2024-03-24T00:18:54.581 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:18:54.577+0000 7fe39cc00640 1 -- 172.21.15.118:0/3346017461 <== mon.0 v2:172.21.15.118:3300/0 5 ==== fsmap(e 28) v1 ==== 7934+0+0 (secure 0 0 0) 0x7fe3a4023530 con 0x5628fad52bd0 2024-03-24T00:18:54.581 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:18:54.577+0000 7fe3bc5ee740 10 client.16203 fetch_fsmap finished waiting for FSMap version 28 2024-03-24T00:18:54.581 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:18:54.577+0000 7fe3bc5ee740 10 client.16203 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-24T00:18:54.581 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:18:54.577+0000 7fe3bc5ee740 20 client.16203 populate_metadata read hostname 'smithi118' 2024-03-24T00:18:54.581 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:18:54.577+0000 7fe3bc5ee740 1 --2- 172.21.15.118:0/3346017461 >> [v2:172.21.15.189:6838/4074484168,v1:172.21.15.189:6839/4074484168] conn(0x5628fad681c0 0x5628fad6a5d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:18:54.581 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:18:54.577+0000 7fe3bc5ee740 4 client.16203 mds_command: new command op to 24569 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-24T00:18:54.581 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:18:54.577+0000 7fe3bc5ee740 1 -- 172.21.15.118:0/3346017461 --> [v2:172.21.15.189:6838/4074484168,v1:172.21.15.189:6839/4074484168] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x5628facc4d70 con 0x5628fad681c0 2024-03-24T00:18:54.581 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:18:54.577+0000 7fe3b2000640 1 --2- 172.21.15.118:0/3346017461 >> [v2:172.21.15.189:6838/4074484168,v1:172.21.15.189:6839/4074484168] conn(0x5628fad681c0 0x5628fad6a5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:18:54.582 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:18:54.578+0000 7fe3b2000640 1 --2- 172.21.15.118:0/3346017461 >> [v2:172.21.15.189:6838/4074484168,v1:172.21.15.189:6839/4074484168] conn(0x5628fad681c0 0x5628fad6a5d0 crc :-1 s=READY pgs=103 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=43d59a83e9d140b0 server_cookie=45470f6ce3704885 in_seq=0 out_seq=0 2024-03-24T00:18:54.583 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:18:54.578+0000 7fe39cc00640 10 client.16203 ms_handle_connect on v2:172.21.15.189:6838/4074484168 2024-03-24T00:18:54.583 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:18:54.579+0000 7fe39cc00640 1 -- 172.21.15.118:0/3346017461 <== mds.0 v2:172.21.15.189:6838/4074484168 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x5628facc4d70 con 0x5628fad681c0 2024-03-24T00:18:54.583 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:18:54.579+0000 7fe39cc00640 10 client.16203 handle_command_reply: tid=0 2024-03-24T00:18:54.583 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:18:54.579+0000 7fe3bc5ee740 1 client.16203 shutdown 2024-03-24T00:18:54.583 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:18:54.579+0000 7fe3bc5ee740 1 -- 172.21.15.118:0/3346017461 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5628fad52bd0 msgr2=0x5628fad5c900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:18:54.583 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:18:54.579+0000 7fe3bc5ee740 1 --2- 172.21.15.118:0/3346017461 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5628fad52bd0 0x5628fad5c900 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x5628fad52fd0 tx=0x7fe3a4011e20 comp rx=0 tx=0).stop 2024-03-24T00:18:54.583 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:18:54.579+0000 7fe3bc5ee740 1 -- 172.21.15.118:0/3346017461 shutdown_connections 2024-03-24T00:18:54.583 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:18:54.579+0000 7fe3bc5ee740 1 --2- 172.21.15.118:0/3346017461 >> [v2:172.21.15.189:6838/4074484168,v1:172.21.15.189:6839/4074484168] conn(0x5628fad681c0 0x5628fad6a5d0 crc :-1 s=READY pgs=103 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:18:54.583 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:18:54.579+0000 7fe3bc5ee740 1 --2- 172.21.15.118:0/3346017461 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5628fad57d40 0x5628fad638d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:18:54.583 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:18:54.579+0000 7fe3bc5ee740 1 --2- 172.21.15.118:0/3346017461 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5628fad52bd0 0x5628fad5c900 unknown :-1 s=CLOSED pgs=204 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:18:54.583 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:18:54.579+0000 7fe39cc00640 0 client.16203 ms_handle_reset on v2:172.21.15.189:6838/4074484168 2024-03-24T00:18:54.583 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:18:54.579+0000 7fe3bc5ee740 1 --2- 172.21.15.118:0/3346017461 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5628fad521e0 0x5628fad5c3c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:18:54.583 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:18:54.579+0000 7fe3bc5ee740 1 -- 172.21.15.118:0/3346017461 >> 172.21.15.118:0/3346017461 conn(0x5628fad4d750 msgr2=0x5628fad4c950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:18:54.584 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:18:54.579+0000 7fe3bc5ee740 1 -- 172.21.15.118:0/3346017461 shutdown_connections 2024-03-24T00:18:54.584 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:18:54.579+0000 7fe3bc5ee740 1 -- 172.21.15.118:0/3346017461 wait complete. 2024-03-24T00:18:54.584 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:18:54.580+0000 7fe3bc5ee740 20 client.16203 trim_cache size 0 max 16384 2024-03-24T00:18:54.595 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:18:54.594+0000 7fe3bc5ee740 1 Processor -- start 2024-03-24T00:18:54.595 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:18:54.595+0000 7fe3bc5ee740 1 -- start start 2024-03-24T00:18:54.595 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:18:54.595+0000 7fe3bc5ee740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5628fad521e0 0x5628fad485d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:18:54.595 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:18:54.595+0000 7fe3bc5ee740 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5628fad52bd0 0x5628fad63890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:18:54.597 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:18:54.595+0000 7fe3bc5ee740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5628fad57d40 0x5628fad510c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:18:54.597 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:18:54.595+0000 7fe3bc5ee740 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe3b41aab50 con 0x5628fad52bd0 2024-03-24T00:18:54.597 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:18:54.595+0000 7fe3bc5ee740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe3b41ac4d0 con 0x5628fad57d40 2024-03-24T00:18:54.597 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:18:54.595+0000 7fe3bc5ee740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe3b41ab1b0 con 0x5628fad521e0 2024-03-24T00:18:54.597 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:18:54.595+0000 7fe3b2000640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5628fad521e0 0x5628fad485d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:18:54.597 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:18:54.595+0000 7fe3b2000640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5628fad521e0 0x5628fad485d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.118:34630/0 (socket says 172.21.15.118:34630) 2024-03-24T00:18:54.597 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:18:54.595+0000 7fe3b2000640 1 -- 172.21.15.118:0/906873891 learned_addr learned my addr 172.21.15.118:0/906873891 (peer_addr_for_me v2:172.21.15.118:0/0) 2024-03-24T00:18:54.597 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:18:54.595+0000 7fe3b2a00640 1 --2- 172.21.15.118:0/906873891 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5628fad52bd0 0x5628fad63890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:18:54.597 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:18:54.595+0000 7fe3b2000640 1 --2- 172.21.15.118:0/906873891 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5628fad57d40 0x5628fad510c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:18:54.597 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:18:54.595+0000 7fe3b2000640 1 -- 172.21.15.118:0/906873891 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5628fad57d40 msgr2=0x5628fad510c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:18:54.597 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:18:54.595+0000 7fe3b2000640 1 --2- 172.21.15.118:0/906873891 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5628fad57d40 0x5628fad510c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:18:54.597 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:18:54.595+0000 7fe3b2000640 1 -- 172.21.15.118:0/906873891 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5628fad52bd0 msgr2=0x5628fad63890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:18:54.597 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:18:54.595+0000 7fe3b2000640 1 --2- 172.21.15.118:0/906873891 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5628fad52bd0 0x5628fad63890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:18:54.598 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:18:54.595+0000 7fe3b2000640 1 -- 172.21.15.118:0/906873891 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5628fad6ac50 con 0x5628fad521e0 2024-03-24T00:18:54.598 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:18:54.596+0000 7fe3b2000640 1 --2- 172.21.15.118:0/906873891 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5628fad521e0 0x5628fad485d0 secure :-1 s=READY pgs=359 cs=0 l=1 rev1=1 crypto rx=0x5628fad6ae50 tx=0x7fe3a000ffa0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=6c12d7ce61e2c44b server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:18:54.598 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:18:54.596+0000 7fe39f400640 1 -- 172.21.15.118:0/906873891 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe3a002e2e0 con 0x5628fad521e0 2024-03-24T00:18:54.598 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:18:54.596+0000 7fe39f400640 1 -- 172.21.15.118:0/906873891 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fe3a0009af0 con 0x5628fad521e0 2024-03-24T00:18:54.598 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:18:54.596+0000 7fe39f400640 1 -- 172.21.15.118:0/906873891 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe3a002e890 con 0x5628fad521e0 2024-03-24T00:18:54.601 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:18:54.597+0000 7fe3bc5ee740 1 -- 172.21.15.118:0/906873891 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5628fad521e0 msgr2=0x5628fad485d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:18:54.601 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:18:54.597+0000 7fe3bc5ee740 1 --2- 172.21.15.118:0/906873891 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5628fad521e0 0x5628fad485d0 secure :-1 s=READY pgs=359 cs=0 l=1 rev1=1 crypto rx=0x5628fad6ae50 tx=0x7fe3a000ffa0 comp rx=0 tx=0).stop 2024-03-24T00:18:54.601 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:18:54.597+0000 7fe3bc5ee740 1 -- 172.21.15.118:0/906873891 shutdown_connections 2024-03-24T00:18:54.601 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:18:54.597+0000 7fe3bc5ee740 1 --2- 172.21.15.118:0/906873891 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5628fad57d40 0x5628fad510c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:18:54.601 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:18:54.597+0000 7fe3bc5ee740 1 --2- 172.21.15.118:0/906873891 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5628fad52bd0 0x5628fad63890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:18:54.601 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:18:54.597+0000 7fe3bc5ee740 1 --2- 172.21.15.118:0/906873891 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5628fad521e0 0x5628fad485d0 unknown :-1 s=CLOSED pgs=359 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:18:54.602 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:18:54.597+0000 7fe3bc5ee740 1 -- 172.21.15.118:0/906873891 >> 172.21.15.118:0/906873891 conn(0x5628fad4d750 msgr2=0x5628fad509b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:18:54.602 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:18:54.597+0000 7fe3bc5ee740 1 -- 172.21.15.118:0/906873891 shutdown_connections 2024-03-24T00:18:54.602 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:18:54.597+0000 7fe3bc5ee740 1 -- 172.21.15.118:0/906873891 wait complete. 2024-03-24T00:18:54.602 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:18:54.597+0000 7fe3bc5ee740 1 -- start start 2024-03-24T00:18:54.602 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:18:54.598+0000 7fe3bc5ee740 1 Processor -- start 2024-03-24T00:18:54.602 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:18:54.598+0000 7fe3bc5ee740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5628fad521e0 0x5628fad63570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:18:54.602 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:18:54.598+0000 7fe3bc5ee740 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5628fad52bd0 0x5628fad61c40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:18:54.602 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:18:54.598+0000 7fe3bc5ee740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5628fad57d40 0x5628fad62180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:18:54.602 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:18:54.598+0000 7fe3bc5ee740 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe3b41abe70 con 0x5628fad52bd0 2024-03-24T00:18:54.602 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:18:54.598+0000 7fe3bc5ee740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe3b41ab810 con 0x5628fad521e0 2024-03-24T00:18:54.602 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:18:54.598+0000 7fe3bc5ee740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe3b41a8d10 con 0x5628fad57d40 2024-03-24T00:18:54.602 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:18:54.598+0000 7fe3bc5ee740 10 client.0 _ll_register_callbacks cb 0x7fe3a4009b50 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T00:18:54.602 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:18:54.598+0000 7fe3b2000640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5628fad521e0 0x5628fad63570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:18:54.603 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:18:54.598+0000 7fe3b2000640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5628fad521e0 0x5628fad63570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.118:47848/0 (socket says 172.21.15.118:47848) 2024-03-24T00:18:54.603 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:18:54.598+0000 7fe3b2000640 1 -- 172.21.15.118:0/2240395006 learned_addr learned my addr 172.21.15.118:0/2240395006 (peer_addr_for_me v2:172.21.15.118:0/0) 2024-03-24T00:18:54.603 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:18:54.598+0000 7fe3b2a00640 1 --2- 172.21.15.118:0/2240395006 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5628fad52bd0 0x5628fad61c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:18:54.603 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:18:54.599+0000 7fe3b2000640 1 --2- 172.21.15.118:0/2240395006 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5628fad57d40 0x5628fad62180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:18:54.603 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:18:54.599+0000 7fe3b2000640 1 -- 172.21.15.118:0/2240395006 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5628fad57d40 msgr2=0x5628fad62180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:18:54.603 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:18:54.599+0000 7fe3b2000640 1 --2- 172.21.15.118:0/2240395006 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5628fad57d40 0x5628fad62180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:18:54.603 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:18:54.599+0000 7fe3b2000640 1 -- 172.21.15.118:0/2240395006 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5628fad52bd0 msgr2=0x5628fad61c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:18:54.603 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:18:54.599+0000 7fe3b2000640 1 --2- 172.21.15.118:0/2240395006 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5628fad52bd0 0x5628fad61c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:18:54.603 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:18:54.599+0000 7fe3b2000640 1 -- 172.21.15.118:0/2240395006 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5628fad626c0 con 0x5628fad521e0 2024-03-24T00:18:54.603 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:18:54.599+0000 7fe3b2000640 1 --2- 172.21.15.118:0/2240395006 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5628fad521e0 0x5628fad63570 secure :-1 s=READY pgs=362 cs=0 l=1 rev1=1 crypto rx=0x7fe3a0009d40 tx=0x7fe3a0009cd0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d95ad75a5078ff1a server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:18:54.603 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:18:54.599+0000 7fe39cc00640 10 client.0 ms_handle_connect on v2:172.21.15.189:3300/0 2024-03-24T00:18:54.603 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:18:54.599+0000 7fe39cc00640 1 -- 172.21.15.118:0/2240395006 <== mon.2 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe3a00050d0 con 0x5628fad521e0 2024-03-24T00:18:54.603 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:18:54.599+0000 7fe3bc5ee740 1 -- 172.21.15.118:0/2240395006 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fe3a0009af0 con 0x5628fad521e0 2024-03-24T00:18:54.603 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:18:54.599+0000 7fe39cc00640 1 -- 172.21.15.118:0/2240395006 <== mon.2 v2:172.21.15.189:3300/0 2 ==== osd_map(90..90 src has 1..90) v4 ==== 8277+0+0 (secure 0 0 0) 0x7fe3a0005270 con 0x5628fad521e0 2024-03-24T00:18:54.604 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:18:54.600+0000 7fe39cc00640 1 -- 172.21.15.118:0/2240395006 <== mon.2 v2:172.21.15.189:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) v2 ==== 24+0+0 (secure 0 0 0) 0x7fe3a0005470 con 0x5628fad521e0 2024-03-24T00:18:54.604 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:18:54.600+0000 7fe3bc5ee740 10 client.26357 fetch_fsmap learned FSMap version 28 2024-03-24T00:18:54.604 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:18:54.600+0000 7fe3bc5ee740 1 -- 172.21.15.118:0/2240395006 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) v3 -- 0x5628fadc47f0 con 0x5628fad521e0 2024-03-24T00:18:54.604 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:18:54.600+0000 7fe39cc00640 1 -- 172.21.15.118:0/2240395006 <== mon.2 v2:172.21.15.189:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fe3a0009af0 con 0x5628fad521e0 2024-03-24T00:18:54.604 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:18:54.600+0000 7fe39cc00640 1 -- 172.21.15.118:0/2240395006 <== mon.2 v2:172.21.15.189:3300/0 5 ==== fsmap(e 28) v1 ==== 7934+0+0 (secure 0 0 0) 0x7fe3a0010530 con 0x5628fad521e0 2024-03-24T00:18:54.604 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:18:54.600+0000 7fe3bc5ee740 10 client.26357 fetch_fsmap finished waiting for FSMap version 28 2024-03-24T00:18:54.604 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:18:54.600+0000 7fe3bc5ee740 10 client.26357 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-24T00:18:54.604 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:18:54.600+0000 7fe3bc5ee740 20 client.26357 populate_metadata read hostname 'smithi118' 2024-03-24T00:18:54.604 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:18:54.600+0000 7fe3bc5ee740 1 --2- 172.21.15.118:0/2240395006 >> [v2:172.21.15.189:6838/4074484168,v1:172.21.15.189:6839/4074484168] conn(0x5628fadc5010 0x5628fadc7430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:18:54.604 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:18:54.600+0000 7fe3bc5ee740 4 client.26357 mds_command: new command op to 24569 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-03-24T00:18:54.604 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:18:54.600+0000 7fe3bc5ee740 1 -- 172.21.15.118:0/2240395006 --> [v2:172.21.15.189:6838/4074484168,v1:172.21.15.189:6839/4074484168] -- command(tid 0: {"prefix": "flush journal"}) v1 -- 0x5628facbf0c0 con 0x5628fadc5010 2024-03-24T00:18:54.604 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:18:54.600+0000 7fe3b2a00640 1 --2- 172.21.15.118:0/2240395006 >> [v2:172.21.15.189:6838/4074484168,v1:172.21.15.189:6839/4074484168] conn(0x5628fadc5010 0x5628fadc7430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:18:54.605 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:18:54.601+0000 7fe3b2a00640 1 --2- 172.21.15.118:0/2240395006 >> [v2:172.21.15.189:6838/4074484168,v1:172.21.15.189:6839/4074484168] conn(0x5628fadc5010 0x5628fadc7430 crc :-1 s=READY pgs=104 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=2fff89b7a3576a13 server_cookie=e3f0875b78ceaee5 in_seq=0 out_seq=0 2024-03-24T00:18:54.605 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:18:54.601+0000 7fe39cc00640 10 client.26357 ms_handle_connect on v2:172.21.15.189:6838/4074484168 2024-03-24T00:18:54.863 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:18:54.862+0000 7fe39cc00640 1 -- 172.21.15.118:0/2240395006 <== mds.0 v2:172.21.15.189:6838/4074484168 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+44 (crc 0 0 0) 0x5628facbf0c0 con 0x5628fadc5010 2024-03-24T00:18:54.863 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:18:54.862+0000 7fe39cc00640 10 client.26357 handle_command_reply: tid=0 2024-03-24T00:18:54.863 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:18:54.862+0000 7fe3bc5ee740 1 client.26357 shutdown 2024-03-24T00:18:54.864 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:18:54.862+0000 7fe3bc5ee740 1 -- 172.21.15.118:0/2240395006 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5628fad521e0 msgr2=0x5628fad63570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:18:54.864 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:18:54.862+0000 7fe3bc5ee740 1 --2- 172.21.15.118:0/2240395006 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5628fad521e0 0x5628fad63570 secure :-1 s=READY pgs=362 cs=0 l=1 rev1=1 crypto rx=0x7fe3a0009d40 tx=0x7fe3a0009cd0 comp rx=0 tx=0).stop 2024-03-24T00:18:54.864 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:18:54.862+0000 7fe3bc5ee740 1 -- 172.21.15.118:0/2240395006 shutdown_connections 2024-03-24T00:18:54.864 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:18:54.862+0000 7fe3bc5ee740 1 --2- 172.21.15.118:0/2240395006 >> [v2:172.21.15.189:6838/4074484168,v1:172.21.15.189:6839/4074484168] conn(0x5628fadc5010 0x5628fadc7430 crc :-1 s=READY pgs=104 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:18:54.864 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:18:54.862+0000 7fe3bc5ee740 1 --2- 172.21.15.118:0/2240395006 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5628fad57d40 0x5628fad62180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:18:54.864 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:18:54.862+0000 7fe3bc5ee740 1 --2- 172.21.15.118:0/2240395006 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5628fad52bd0 0x5628fad61c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:18:54.864 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:18:54.862+0000 7fe3bc5ee740 1 --2- 172.21.15.118:0/2240395006 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5628fad521e0 0x5628fad63570 unknown :-1 s=CLOSED pgs=362 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:18:54.864 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:18:54.862+0000 7fe3bc5ee740 1 -- 172.21.15.118:0/2240395006 >> 172.21.15.118:0/2240395006 conn(0x5628fad4d750 msgr2=0x5628fad51240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:18:54.864 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:18:54.862+0000 7fe39cc00640 0 client.26357 ms_handle_reset on v2:172.21.15.189:6838/4074484168 2024-03-24T00:18:54.864 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:18:54.863+0000 7fe3bc5ee740 1 -- 172.21.15.118:0/2240395006 shutdown_connections 2024-03-24T00:18:54.864 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:18:54.863+0000 7fe3bc5ee740 1 -- 172.21.15.118:0/2240395006 wait complete. 2024-03-24T00:18:54.864 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:18:54.863+0000 7fe3bc5ee740 20 client.26357 trim_cache size 0 max 16384 2024-03-24T00:18:54.865 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stdout:{ 2024-03-24T00:18:54.865 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stdout: "message": "", 2024-03-24T00:18:54.865 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stdout: "return_code": 0 2024-03-24T00:18:54.865 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stdout:} 2024-03-24T00:18:54.865 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:18:54.863+0000 7fe3b9800640 1 -- 172.21.15.118:0/4006193620 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7fe3980a28a0 msgr2=0x7fe3980a4d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:18:54.866 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:18:54.863+0000 7fe3b9800640 1 --2- 172.21.15.118:0/4006193620 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7fe3980a28a0 0x7fe3980a4d60 secure :-1 s=READY pgs=329 cs=0 l=1 rev1=1 crypto rx=0x7fe3ac0b9e10 tx=0x7fe3ac02c100 comp rx=0 tx=0).stop 2024-03-24T00:18:54.866 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:18:54.863+0000 7fe3b9800640 1 -- 172.21.15.118:0/4006193620 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe3b41054b0 msgr2=0x7fe3b41a1060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:18:54.866 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:18:54.863+0000 7fe3b9800640 1 --2- 172.21.15.118:0/4006193620 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe3b41054b0 0x7fe3b41a1060 secure :-1 s=READY pgs=361 cs=0 l=1 rev1=1 crypto rx=0x7fe3ac02c640 tx=0x7fe3ac035d10 comp rx=0 tx=0).stop 2024-03-24T00:18:54.866 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:18:54.864+0000 7fe3b9800640 1 -- 172.21.15.118:0/4006193620 shutdown_connections 2024-03-24T00:18:54.866 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:18:54.864+0000 7fe3b9800640 1 --2- 172.21.15.118:0/4006193620 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7fe3980a28a0 0x7fe3980a4d60 unknown :-1 s=CLOSED pgs=329 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:18:54.866 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:18:54.864+0000 7fe3b9800640 1 --2- 172.21.15.118:0/4006193620 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe3b41054b0 0x7fe3b41a1060 unknown :-1 s=CLOSED pgs=361 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:18:54.866 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:18:54.864+0000 7fe3b9800640 1 --2- 172.21.15.118:0/4006193620 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fe3b4104af0 0x7fe3b4199ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:18:54.866 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:18:54.864+0000 7fe3b9800640 1 --2- 172.21.15.118:0/4006193620 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fe3b4103900 0x7fe3b4199ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:18:54.866 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:18:54.864+0000 7fe3b9800640 1 -- 172.21.15.118:0/4006193620 >> 172.21.15.118:0/4006193620 conn(0x7fe3b40fee90 msgr2=0x7fe3b4107d20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:18:54.866 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:18:54.864+0000 7fe3b9800640 1 -- 172.21.15.118:0/4006193620 shutdown_connections 2024-03-24T00:18:54.866 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:18:54.864+0000 7fe3b9800640 1 -- 172.21.15.118:0/4006193620 wait complete. 2024-03-24T00:18:55.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:18:54 smithi118 ceph-mon[31535]: pgmap v494: 97 pgs: 97 active+clean; 67 MiB data, 535 MiB used, 1.0 TiB / 1.0 TiB avail; 808 KiB/s rd, 2.4 MiB/s wr, 479 op/s 2024-03-24T00:18:55.203 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:18:54 smithi189 ceph-mon[36187]: pgmap v494: 97 pgs: 97 active+clean; 67 MiB data, 535 MiB used, 1.0 TiB / 1.0 TiB avail; 808 KiB/s rd, 2.4 MiB/s wr, 479 op/s 2024-03-24T00:18:55.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:18:54 smithi193 ceph-mon[36878]: pgmap v494: 97 pgs: 97 active+clean; 67 MiB data, 535 MiB used, 1.0 TiB / 1.0 TiB avail; 808 KiB/s rd, 2.4 MiB/s wr, 479 op/s 2024-03-24T00:18:56.843 INFO:tasks.workunit.client.0.smithi118.stdout: 19 203605 174 141258 75 108693 237 2024-03-24T00:18:57.195 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:18:56 smithi118 ceph-mon[31535]: pgmap v495: 97 pgs: 97 active+clean; 63 MiB data, 542 MiB used, 1.0 TiB / 1.0 TiB avail; 569 KiB/s rd, 2.2 MiB/s wr, 306 op/s 2024-03-24T00:18:57.203 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:18:56 smithi189 ceph-mon[36187]: pgmap v495: 97 pgs: 97 active+clean; 63 MiB data, 542 MiB used, 1.0 TiB / 1.0 TiB avail; 569 KiB/s rd, 2.2 MiB/s wr, 306 op/s 2024-03-24T00:18:57.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:18:56 smithi193 ceph-mon[36878]: pgmap v495: 97 pgs: 97 active+clean; 63 MiB data, 542 MiB used, 1.0 TiB / 1.0 TiB avail; 569 KiB/s rd, 2.2 MiB/s wr, 306 op/s 2024-03-24T00:18:59.195 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:18:58 smithi118 ceph-mon[31535]: pgmap v496: 97 pgs: 97 active+clean; 63 MiB data, 537 MiB used, 1.0 TiB / 1.0 TiB avail; 536 KiB/s rd, 2.2 MiB/s wr, 275 op/s 2024-03-24T00:18:59.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:18:58 smithi189 ceph-mon[36187]: pgmap v496: 97 pgs: 97 active+clean; 63 MiB data, 537 MiB used, 1.0 TiB / 1.0 TiB avail; 536 KiB/s rd, 2.2 MiB/s wr, 275 op/s 2024-03-24T00:18:59.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:18:58 smithi193 ceph-mon[36878]: pgmap v496: 97 pgs: 97 active+clean; 63 MiB data, 537 MiB used, 1.0 TiB / 1.0 TiB avail; 536 KiB/s rd, 2.2 MiB/s wr, 275 op/s 2024-03-24T00:19:01.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:19:00 smithi118 ceph-mon[31535]: pgmap v497: 97 pgs: 97 active+clean; 61 MiB data, 526 MiB used, 1.0 TiB / 1.0 TiB avail; 697 KiB/s rd, 2.4 MiB/s wr, 343 op/s 2024-03-24T00:19:01.203 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:19:00 smithi189 ceph-mon[36187]: pgmap v497: 97 pgs: 97 active+clean; 61 MiB data, 526 MiB used, 1.0 TiB / 1.0 TiB avail; 697 KiB/s rd, 2.4 MiB/s wr, 343 op/s 2024-03-24T00:19:01.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:19:00 smithi193 ceph-mon[36878]: pgmap v497: 97 pgs: 97 active+clean; 61 MiB data, 526 MiB used, 1.0 TiB / 1.0 TiB avail; 697 KiB/s rd, 2.4 MiB/s wr, 343 op/s 2024-03-24T00:19:03.195 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:19:02 smithi118 ceph-mon[31535]: pgmap v498: 97 pgs: 97 active+clean; 61 MiB data, 543 MiB used, 1.0 TiB / 1.0 TiB avail; 648 KiB/s rd, 2.0 MiB/s wr, 316 op/s 2024-03-24T00:19:03.203 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:19:02 smithi189 ceph-mon[36187]: pgmap v498: 97 pgs: 97 active+clean; 61 MiB data, 543 MiB used, 1.0 TiB / 1.0 TiB avail; 648 KiB/s rd, 2.0 MiB/s wr, 316 op/s 2024-03-24T00:19:03.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:19:02 smithi193 ceph-mon[36878]: pgmap v498: 97 pgs: 97 active+clean; 61 MiB data, 543 MiB used, 1.0 TiB / 1.0 TiB avail; 648 KiB/s rd, 2.0 MiB/s wr, 316 op/s 2024-03-24T00:19:05.195 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:19:04 smithi118 ceph-mon[31535]: pgmap v499: 97 pgs: 97 active+clean; 63 MiB data, 548 MiB used, 1.0 TiB / 1.0 TiB avail; 611 KiB/s rd, 1.5 MiB/s wr, 300 op/s 2024-03-24T00:19:05.203 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:19:04 smithi189 ceph-mon[36187]: pgmap v499: 97 pgs: 97 active+clean; 63 MiB data, 548 MiB used, 1.0 TiB / 1.0 TiB avail; 611 KiB/s rd, 1.5 MiB/s wr, 300 op/s 2024-03-24T00:19:05.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:19:04 smithi193 ceph-mon[36878]: pgmap v499: 97 pgs: 97 active+clean; 63 MiB data, 548 MiB used, 1.0 TiB / 1.0 TiB avail; 611 KiB/s rd, 1.5 MiB/s wr, 300 op/s 2024-03-24T00:19:06.843 INFO:tasks.workunit.client.0.smithi118.stdout: 21 31353 133 21152 198 12866 444 2024-03-24T00:19:07.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:19:06 smithi118 ceph-mon[31535]: pgmap v500: 97 pgs: 97 active+clean; 63 MiB data, 547 MiB used, 1.0 TiB / 1.0 TiB avail; 526 KiB/s rd, 601 KiB/s wr, 231 op/s 2024-03-24T00:19:07.203 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:19:06 smithi189 ceph-mon[36187]: pgmap v500: 97 pgs: 97 active+clean; 63 MiB data, 547 MiB used, 1.0 TiB / 1.0 TiB avail; 526 KiB/s rd, 601 KiB/s wr, 231 op/s 2024-03-24T00:19:07.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:19:06 smithi193 ceph-mon[36878]: pgmap v500: 97 pgs: 97 active+clean; 63 MiB data, 547 MiB used, 1.0 TiB / 1.0 TiB avail; 526 KiB/s rd, 601 KiB/s wr, 231 op/s 2024-03-24T00:19:08.023 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:19:08.022+0000 7f35dec00640 1 -- 172.21.15.118:0/3555914747 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f35d80756e0 msgr2=0x7f35d8073b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:19:08.023 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:19:08.022+0000 7f35dec00640 1 --2- 172.21.15.118:0/3555914747 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f35d80756e0 0x7f35d8073b30 secure :-1 s=READY pgs=360 cs=0 l=1 rev1=1 crypto rx=0x7f35cc009900 tx=0x7f35cc02bc30 comp rx=0 tx=0).stop 2024-03-24T00:19:08.023 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:19:08.022+0000 7f35dec00640 1 -- 172.21.15.118:0/3555914747 shutdown_connections 2024-03-24T00:19:08.023 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:19:08.022+0000 7f35dec00640 1 --2- 172.21.15.118:0/3555914747 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f35d810dae0 0x7f35d810fed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:19:08.023 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:19:08.022+0000 7f35dec00640 1 --2- 172.21.15.118:0/3555914747 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f35d8074100 0x7f35d8074580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:19:08.023 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:19:08.022+0000 7f35dec00640 1 --2- 172.21.15.118:0/3555914747 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f35d80756e0 0x7f35d8073b30 unknown :-1 s=CLOSED pgs=360 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:19:08.023 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:19:08.022+0000 7f35dec00640 1 -- 172.21.15.118:0/3555914747 >> 172.21.15.118:0/3555914747 conn(0x7f35d80fca80 msgr2=0x7f35d80feec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:19:08.024 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:19:08.022+0000 7f35dec00640 1 -- 172.21.15.118:0/3555914747 shutdown_connections 2024-03-24T00:19:08.024 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:19:08.022+0000 7f35dec00640 1 -- 172.21.15.118:0/3555914747 wait complete. 2024-03-24T00:19:08.025 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:19:08.023+0000 7f35dec00640 1 Processor -- start 2024-03-24T00:19:08.025 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:19:08.023+0000 7f35dec00640 1 -- start start 2024-03-24T00:19:08.025 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:19:08.024+0000 7f35dec00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f35d8074100 0x7f35d819de20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:19:08.025 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:19:08.024+0000 7f35dec00640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f35d80756e0 0x7f35d819e360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:19:08.025 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:19:08.024+0000 7f35dec00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f35d810dae0 0x7f35d81a53d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:19:08.025 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:19:08.024+0000 7f35dec00640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f35d81137d0 con 0x7f35d80756e0 2024-03-24T00:19:08.025 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:19:08.024+0000 7f35dec00640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f35d8113650 con 0x7f35d810dae0 2024-03-24T00:19:08.025 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:19:08.024+0000 7f35dec00640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f35d8113950 con 0x7f35d8074100 2024-03-24T00:19:08.026 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:19:08.024+0000 7f35dd800640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f35d8074100 0x7f35d819de20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:19:08.026 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:19:08.024+0000 7f35dd800640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f35d8074100 0x7f35d819de20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.118:47076/0 (socket says 172.21.15.118:47076) 2024-03-24T00:19:08.026 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:19:08.024+0000 7f35dd800640 1 -- 172.21.15.118:0/1130715760 learned_addr learned my addr 172.21.15.118:0/1130715760 (peer_addr_for_me v2:172.21.15.118:0/0) 2024-03-24T00:19:08.026 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:19:08.024+0000 7f35dce00640 1 --2- 172.21.15.118:0/1130715760 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f35d80756e0 0x7f35d819e360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:19:08.026 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:19:08.024+0000 7f35de200640 1 --2- 172.21.15.118:0/1130715760 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f35d810dae0 0x7f35d81a53d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:19:08.026 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:19:08.025+0000 7f35dd800640 1 -- 172.21.15.118:0/1130715760 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f35d810dae0 msgr2=0x7f35d81a53d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:19:08.026 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:19:08.025+0000 7f35dd800640 1 --2- 172.21.15.118:0/1130715760 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f35d810dae0 0x7f35d81a53d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:19:08.027 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:19:08.025+0000 7f35dd800640 1 -- 172.21.15.118:0/1130715760 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f35d80756e0 msgr2=0x7f35d819e360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:19:08.027 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:19:08.025+0000 7f35dd800640 1 --2- 172.21.15.118:0/1130715760 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f35d80756e0 0x7f35d819e360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:19:08.027 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:19:08.025+0000 7f35dd800640 1 -- 172.21.15.118:0/1130715760 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f35d81a5a40 con 0x7f35d8074100 2024-03-24T00:19:08.027 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:19:08.025+0000 7f35de200640 1 --2- 172.21.15.118:0/1130715760 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f35d810dae0 0x7f35d81a53d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T00:19:08.028 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:19:08.026+0000 7f35dd800640 1 --2- 172.21.15.118:0/1130715760 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f35d8074100 0x7f35d819de20 secure :-1 s=READY pgs=361 cs=0 l=1 rev1=1 crypto rx=0x7f35cc005ab0 tx=0x7f35cc005dd0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:19:08.028 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:19:08.026+0000 7f35d6000640 1 -- 172.21.15.118:0/1130715760 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f35cc041070 con 0x7f35d8074100 2024-03-24T00:19:08.028 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:19:08.026+0000 7f35dec00640 1 -- 172.21.15.118:0/1130715760 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f35d81a5cd0 con 0x7f35d8074100 2024-03-24T00:19:08.028 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:19:08.026+0000 7f35d6000640 1 -- 172.21.15.118:0/1130715760 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f35cc0054e0 con 0x7f35d8074100 2024-03-24T00:19:08.028 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:19:08.026+0000 7f35d6000640 1 -- 172.21.15.118:0/1130715760 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f35cc034440 con 0x7f35d8074100 2024-03-24T00:19:08.028 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:19:08.027+0000 7f35dec00640 1 -- 172.21.15.118:0/1130715760 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f35d81a61c0 con 0x7f35d8074100 2024-03-24T00:19:08.033 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:19:08.031+0000 7f35d6000640 1 -- 172.21.15.118:0/1130715760 <== mon.1 v2:172.21.15.193:3300/0 4 ==== mgrmap(e 20) v1 ==== 137291+0+0 (secure 0 0 0) 0x7f35cc044070 con 0x7f35d8074100 2024-03-24T00:19:08.033 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:19:08.032+0000 7f35e2fb6740 1 Processor -- start 2024-03-24T00:19:08.033 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:19:08.032+0000 7f35e2fb6740 1 -- start start 2024-03-24T00:19:08.034 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:19:08.032+0000 7f35e2fb6740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5654340ed3f0 0x5654340ed810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:19:08.034 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:19:08.032+0000 7f35e2fb6740 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5654340edde0 0x5654340f28e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:19:08.034 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:19:08.032+0000 7f35e2fb6740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5654340f2f50 0x5654340f53e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:19:08.034 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:19:08.032+0000 7f35e2fb6740 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x5654340b03b0 con 0x5654340edde0 2024-03-24T00:19:08.035 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:19:08.032+0000 7f35e2fb6740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x565434052830 con 0x5654340f2f50 2024-03-24T00:19:08.035 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:19:08.033+0000 7f35e2fb6740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x565433ffb5d0 con 0x5654340ed3f0 2024-03-24T00:19:08.035 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:19:08.033+0000 7f35de200640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5654340ed3f0 0x5654340ed810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:19:08.035 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:19:08.033+0000 7f35de200640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5654340ed3f0 0x5654340ed810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.118:47092/0 (socket says 172.21.15.118:47092) 2024-03-24T00:19:08.035 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:19:08.033+0000 7f35de200640 1 -- 172.21.15.118:0/2501527033 learned_addr learned my addr 172.21.15.118:0/2501527033 (peer_addr_for_me v2:172.21.15.118:0/0) 2024-03-24T00:19:08.035 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:19:08.033+0000 7f35de200640 1 -- 172.21.15.118:0/2501527033 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5654340f2f50 msgr2=0x5654340f53e0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-24T00:19:08.036 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:19:08.034+0000 7f35de200640 1 --2- 172.21.15.118:0/2501527033 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5654340f2f50 0x5654340f53e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:19:08.036 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:19:08.034+0000 7f35de200640 1 -- 172.21.15.118:0/2501527033 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5654340edde0 msgr2=0x5654340f28e0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-24T00:19:08.036 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:19:08.034+0000 7f35de200640 1 --2- 172.21.15.118:0/2501527033 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5654340edde0 0x5654340f28e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:19:08.036 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:19:08.034+0000 7f35de200640 1 -- 172.21.15.118:0/2501527033 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5654340132c0 con 0x5654340ed3f0 2024-03-24T00:19:08.036 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:19:08.034+0000 7f35d6000640 1 --2- 172.21.15.118:0/1130715760 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7f35c00a28f0 0x7f35c00a4db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:19:08.036 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:19:08.034+0000 7f35de200640 1 --2- 172.21.15.118:0/2501527033 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5654340ed3f0 0x5654340ed810 secure :-1 s=READY pgs=362 cs=0 l=1 rev1=1 crypto rx=0x7f35d00107e0 tx=0x7f35d0010ca0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=ac38c1a0d3801963 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:19:08.036 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:19:08.034+0000 7f35d6000640 1 -- 172.21.15.118:0/1130715760 <== mon.1 v2:172.21.15.193:3300/0 5 ==== osd_map(90..90 src has 1..90) v4 ==== 8277+0+0 (secure 0 0 0) 0x7f35cc005000 con 0x7f35d8074100 2024-03-24T00:19:08.036 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:19:08.035+0000 7f35caa00640 1 -- 172.21.15.118:0/2501527033 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f35d0003b70 con 0x5654340ed3f0 2024-03-24T00:19:08.036 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:19:08.035+0000 7f35caa00640 1 -- 172.21.15.118:0/2501527033 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f35d000ee10 con 0x5654340ed3f0 2024-03-24T00:19:08.036 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:19:08.035+0000 7f35caa00640 1 -- 172.21.15.118:0/2501527033 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f35d0011440 con 0x5654340ed3f0 2024-03-24T00:19:08.037 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:19:08.035+0000 7f35dd800640 1 --2- 172.21.15.118:0/1130715760 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7f35c00a28f0 0x7f35c00a4db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:19:08.037 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:19:08.036+0000 7f35e2fb6740 1 -- 172.21.15.118:0/2501527033 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5654340ed3f0 msgr2=0x5654340ed810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:19:08.037 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:19:08.036+0000 7f35e2fb6740 1 --2- 172.21.15.118:0/2501527033 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5654340ed3f0 0x5654340ed810 secure :-1 s=READY pgs=362 cs=0 l=1 rev1=1 crypto rx=0x7f35d00107e0 tx=0x7f35d0010ca0 comp rx=0 tx=0).stop 2024-03-24T00:19:08.037 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:19:08.036+0000 7f35dd800640 1 --2- 172.21.15.118:0/1130715760 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7f35c00a28f0 0x7f35c00a4db0 secure :-1 s=READY pgs=330 cs=0 l=1 rev1=1 crypto rx=0x7f35cc005ec0 tx=0x7f35cc02c8b0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:19:08.038 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:19:08.037+0000 7f35e2fb6740 1 -- 172.21.15.118:0/2501527033 shutdown_connections 2024-03-24T00:19:08.038 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:19:08.037+0000 7f35e2fb6740 1 --2- 172.21.15.118:0/2501527033 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5654340f2f50 0x5654340f53e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:19:08.038 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:19:08.037+0000 7f35e2fb6740 1 --2- 172.21.15.118:0/2501527033 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5654340edde0 0x5654340f28e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:19:08.038 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:19:08.037+0000 7f35e2fb6740 1 --2- 172.21.15.118:0/2501527033 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5654340ed3f0 0x5654340ed810 unknown :-1 s=CLOSED pgs=362 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:19:08.038 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:19:08.037+0000 7f35e2fb6740 1 -- 172.21.15.118:0/2501527033 >> 172.21.15.118:0/2501527033 conn(0x5654340e8960 msgr2=0x5654340e8d60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:19:08.038 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:19:08.037+0000 7f35e2fb6740 1 -- 172.21.15.118:0/2501527033 shutdown_connections 2024-03-24T00:19:08.039 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:19:08.037+0000 7f35e2fb6740 1 -- 172.21.15.118:0/2501527033 wait complete. 2024-03-24T00:19:08.039 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:19:08.038+0000 7f35e2fb6740 1 -- start start 2024-03-24T00:19:08.039 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:19:08.038+0000 7f35e2fb6740 1 Processor -- start 2024-03-24T00:19:08.040 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:19:08.039+0000 7f35e2fb6740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5654340ed3f0 0x5654341086a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:19:08.040 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:19:08.039+0000 7f35e2fb6740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5654340edde0 0x565434108be0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:19:08.040 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:19:08.039+0000 7f35e2fb6740 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5654340f2f50 0x56543410fbb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:19:08.040 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:19:08.039+0000 7f35e2fb6740 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x56543400ea20 con 0x5654340f2f50 2024-03-24T00:19:08.040 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:19:08.039+0000 7f35e2fb6740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x56543404e8d0 con 0x5654340ed3f0 2024-03-24T00:19:08.040 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:19:08.039+0000 7f35e2fb6740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x565434106f10 con 0x5654340edde0 2024-03-24T00:19:08.040 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:19:08.039+0000 7f35e2fb6740 10 client.0 _ll_register_callbacks cb 0x56543404b1c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T00:19:08.040 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:19:08.039+0000 7f35dce00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5654340edde0 0x565434108be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:19:08.041 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:19:08.039+0000 7f35dce00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5654340edde0 0x565434108be0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.118:47098/0 (socket says 172.21.15.118:47098) 2024-03-24T00:19:08.041 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:19:08.039+0000 7f35dce00640 1 -- 172.21.15.118:0/1036664566 learned_addr learned my addr 172.21.15.118:0/1036664566 (peer_addr_for_me v2:172.21.15.118:0/0) 2024-03-24T00:19:08.041 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:19:08.039+0000 7f35de200640 1 --2- 172.21.15.118:0/1036664566 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5654340ed3f0 0x5654341086a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:19:08.041 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:19:08.039+0000 7f35de200640 1 --2- 172.21.15.118:0/1036664566 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5654340f2f50 0x56543410fbb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:19:08.041 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:19:08.039+0000 7f35dce00640 1 -- 172.21.15.118:0/1036664566 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5654340ed3f0 msgr2=0x5654341086a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:19:08.041 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:19:08.039+0000 7f35dce00640 1 --2- 172.21.15.118:0/1036664566 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5654340ed3f0 0x5654341086a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:19:08.041 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:19:08.039+0000 7f35dce00640 1 -- 172.21.15.118:0/1036664566 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5654340f2f50 msgr2=0x56543410fbb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:19:08.041 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:19:08.040+0000 7f35dce00640 1 --2- 172.21.15.118:0/1036664566 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5654340f2f50 0x56543410fbb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:19:08.041 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:19:08.040+0000 7f35dce00640 1 -- 172.21.15.118:0/1036664566 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x565433d67210 con 0x5654340edde0 2024-03-24T00:19:08.042 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:19:08.040+0000 7f35dce00640 1 --2- 172.21.15.118:0/1036664566 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5654340edde0 0x565434108be0 secure :-1 s=READY pgs=363 cs=0 l=1 rev1=1 crypto rx=0x7f35c40116f0 tx=0x7f35c4011bb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c97fce32a578d146 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:19:08.042 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:19:08.040+0000 7f35bf400640 10 client.0 ms_handle_connect on v2:172.21.15.193:3300/0 2024-03-24T00:19:08.042 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:19:08.040+0000 7f35bf400640 1 -- 172.21.15.118:0/1036664566 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f35c401e060 con 0x5654340edde0 2024-03-24T00:19:08.043 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:19:08.040+0000 7f35bf400640 1 -- 172.21.15.118:0/1036664566 <== mon.1 v2:172.21.15.193:3300/0 2 ==== osd_map(90..90 src has 1..90) v4 ==== 8277+0+0 (secure 0 0 0) 0x7f35c400fd10 con 0x5654340edde0 2024-03-24T00:19:08.043 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:19:08.041+0000 7f35e2fb6740 1 -- 172.21.15.118:0/1036664566 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f35d000ee10 con 0x5654340edde0 2024-03-24T00:19:08.043 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:19:08.041+0000 7f35bf400640 1 -- 172.21.15.118:0/1036664566 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) v2 ==== 24+0+0 (secure 0 0 0) 0x7f35c4019430 con 0x5654340edde0 2024-03-24T00:19:08.043 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:19:08.041+0000 7f35e2fb6740 10 client.26380 fetch_fsmap learned FSMap version 28 2024-03-24T00:19:08.043 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:19:08.041+0000 7f35e2fb6740 1 -- 172.21.15.118:0/1036664566 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) v3 -- 0x565433e33620 con 0x5654340edde0 2024-03-24T00:19:08.043 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:19:08.042+0000 7f35bf400640 1 -- 172.21.15.118:0/1036664566 <== mon.1 v2:172.21.15.193:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f35d000ee10 con 0x5654340edde0 2024-03-24T00:19:08.043 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:19:08.042+0000 7f35bf400640 1 -- 172.21.15.118:0/1036664566 <== mon.1 v2:172.21.15.193:3300/0 5 ==== fsmap(e 28) v1 ==== 7934+0+0 (secure 0 0 0) 0x7f35c4011ce0 con 0x5654340edde0 2024-03-24T00:19:08.044 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:19:08.043+0000 7f35e2fb6740 10 client.26380 fetch_fsmap finished waiting for FSMap version 28 2024-03-24T00:19:08.044 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:19:08.043+0000 7f35e2fb6740 10 client.26380 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-24T00:19:08.044 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:19:08.043+0000 7f35e2fb6740 20 client.26380 populate_metadata read hostname 'smithi118' 2024-03-24T00:19:08.044 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:19:08.043+0000 7f35e2fb6740 1 --2- 172.21.15.118:0/1036664566 >> [v2:172.21.15.189:6838/4074484168,v1:172.21.15.189:6839/4074484168] conn(0x5654341145a0 0x5654341169c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:19:08.044 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:19:08.043+0000 7f35e2fb6740 4 client.26380 mds_command: new command op to 24569 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-24T00:19:08.044 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:19:08.043+0000 7f35e2fb6740 1 -- 172.21.15.118:0/1036664566 --> [v2:172.21.15.189:6838/4074484168,v1:172.21.15.189:6839/4074484168] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x56543405fd70 con 0x5654341145a0 2024-03-24T00:19:08.044 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:19:08.044+0000 7f35de200640 1 --2- 172.21.15.118:0/1036664566 >> [v2:172.21.15.189:6838/4074484168,v1:172.21.15.189:6839/4074484168] conn(0x5654341145a0 0x5654341169c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:19:08.045 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:19:08.044+0000 7f35de200640 1 --2- 172.21.15.118:0/1036664566 >> [v2:172.21.15.189:6838/4074484168,v1:172.21.15.189:6839/4074484168] conn(0x5654341145a0 0x5654341169c0 crc :-1 s=READY pgs=105 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=2d4bb3cd54e4d3aa server_cookie=e46a686afa34ac5d in_seq=0 out_seq=0 2024-03-24T00:19:08.045 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:19:08.044+0000 7f35bf400640 10 client.26380 ms_handle_connect on v2:172.21.15.189:6838/4074484168 2024-03-24T00:19:08.047 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:19:08.046+0000 7f35bf400640 1 -- 172.21.15.118:0/1036664566 <== mds.0 v2:172.21.15.189:6838/4074484168 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x56543405fd70 con 0x5654341145a0 2024-03-24T00:19:08.047 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:19:08.046+0000 7f35bf400640 10 client.26380 handle_command_reply: tid=0 2024-03-24T00:19:08.047 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:19:08.046+0000 7f35e2fb6740 1 client.26380 shutdown 2024-03-24T00:19:08.048 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:19:08.047+0000 7f35e2fb6740 1 -- 172.21.15.118:0/1036664566 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5654340edde0 msgr2=0x565434108be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:19:08.048 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:19:08.047+0000 7f35e2fb6740 1 --2- 172.21.15.118:0/1036664566 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5654340edde0 0x565434108be0 secure :-1 s=READY pgs=363 cs=0 l=1 rev1=1 crypto rx=0x7f35c40116f0 tx=0x7f35c4011bb0 comp rx=0 tx=0).stop 2024-03-24T00:19:08.048 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:19:08.047+0000 7f35e2fb6740 1 -- 172.21.15.118:0/1036664566 shutdown_connections 2024-03-24T00:19:08.048 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:19:08.047+0000 7f35e2fb6740 1 --2- 172.21.15.118:0/1036664566 >> [v2:172.21.15.189:6838/4074484168,v1:172.21.15.189:6839/4074484168] conn(0x5654341145a0 0x5654341169c0 crc :-1 s=READY pgs=105 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:19:08.048 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:19:08.047+0000 7f35e2fb6740 1 --2- 172.21.15.118:0/1036664566 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5654340f2f50 0x56543410fbb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:19:08.048 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:19:08.047+0000 7f35e2fb6740 1 --2- 172.21.15.118:0/1036664566 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5654340edde0 0x565434108be0 unknown :-1 s=CLOSED pgs=363 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:19:08.048 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:19:08.047+0000 7f35e2fb6740 1 --2- 172.21.15.118:0/1036664566 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5654340ed3f0 0x5654341086a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:19:08.048 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:19:08.047+0000 7f35e2fb6740 1 -- 172.21.15.118:0/1036664566 >> 172.21.15.118:0/1036664566 conn(0x5654340e8960 msgr2=0x5654340e7b70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:19:08.048 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:19:08.047+0000 7f35bf400640 0 client.26380 ms_handle_reset on v2:172.21.15.189:6838/4074484168 2024-03-24T00:19:08.049 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:19:08.047+0000 7f35e2fb6740 1 -- 172.21.15.118:0/1036664566 shutdown_connections 2024-03-24T00:19:08.049 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:19:08.047+0000 7f35e2fb6740 1 -- 172.21.15.118:0/1036664566 wait complete. 2024-03-24T00:19:08.049 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:19:08.047+0000 7f35e2fb6740 20 client.26380 trim_cache size 0 max 16384 2024-03-24T00:19:08.069 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:19:08.070+0000 7f35e2fb6740 1 Processor -- start 2024-03-24T00:19:08.069 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:19:08.070+0000 7f35e2fb6740 1 -- start start 2024-03-24T00:19:08.070 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:19:08.070+0000 7f35e2fb6740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5654340ed3f0 0x5654340e72e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:19:08.070 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:19:08.070+0000 7f35e2fb6740 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5654340edde0 0x56543410fb70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:19:08.070 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:19:08.070+0000 7f35e2fb6740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5654340f2f50 0x5654340e7820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:19:08.070 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:19:08.070+0000 7f35e2fb6740 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f35b800e300 con 0x5654340edde0 2024-03-24T00:19:08.070 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:19:08.070+0000 7f35e2fb6740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f35b800fc80 con 0x5654340ed3f0 2024-03-24T00:19:08.070 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:19:08.070+0000 7f35e2fb6740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f35b800e960 con 0x5654340f2f50 2024-03-24T00:19:08.070 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:19:08.070+0000 7f35de200640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5654340ed3f0 0x5654340e72e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:19:08.071 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:19:08.070+0000 7f35de200640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5654340ed3f0 0x5654340e72e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.118:53982/0 (socket says 172.21.15.118:53982) 2024-03-24T00:19:08.071 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:19:08.070+0000 7f35dce00640 1 --2- 172.21.15.118:0/142860180 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5654340edde0 0x56543410fb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:19:08.071 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:19:08.070+0000 7f35de200640 1 -- 172.21.15.118:0/142860180 learned_addr learned my addr 172.21.15.118:0/142860180 (peer_addr_for_me v2:172.21.15.118:0/0) 2024-03-24T00:19:08.071 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:19:08.070+0000 7f35de200640 1 --2- 172.21.15.118:0/142860180 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5654340f2f50 0x5654340e7820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:19:08.072 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:19:08.070+0000 7f35de200640 1 -- 172.21.15.118:0/142860180 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5654340f2f50 msgr2=0x5654340e7820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:19:08.072 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:19:08.070+0000 7f35de200640 1 --2- 172.21.15.118:0/142860180 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5654340f2f50 0x5654340e7820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:19:08.072 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:19:08.070+0000 7f35de200640 1 -- 172.21.15.118:0/142860180 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5654340edde0 msgr2=0x56543410fb70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:19:08.072 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:19:08.070+0000 7f35de200640 1 --2- 172.21.15.118:0/142860180 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5654340edde0 0x56543410fb70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:19:08.072 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:19:08.070+0000 7f35de200640 1 -- 172.21.15.118:0/142860180 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5654340e6f80 con 0x5654340ed3f0 2024-03-24T00:19:08.072 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:19:08.070+0000 7f35dce00640 1 --2- 172.21.15.118:0/142860180 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5654340edde0 0x56543410fb70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T00:19:08.072 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:19:08.070+0000 7f35de200640 1 --2- 172.21.15.118:0/142860180 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5654340ed3f0 0x5654340e72e0 secure :-1 s=READY pgs=363 cs=0 l=1 rev1=1 crypto rx=0x7f35d00107e0 tx=0x7f35d0005fd0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e353586cdc5ad02d server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:19:08.072 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:19:08.070+0000 7f35ca000640 1 -- 172.21.15.118:0/142860180 <== mon.2 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f35d0025040 con 0x5654340ed3f0 2024-03-24T00:19:08.073 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:19:08.071+0000 7f35ca000640 1 -- 172.21.15.118:0/142860180 <== mon.2 v2:172.21.15.189:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f35d000e930 con 0x5654340ed3f0 2024-03-24T00:19:08.073 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:19:08.071+0000 7f35ca000640 1 -- 172.21.15.118:0/142860180 <== mon.2 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f35d002b9b0 con 0x5654340ed3f0 2024-03-24T00:19:08.073 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:19:08.071+0000 7f35e2fb6740 1 -- 172.21.15.118:0/142860180 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5654340ed3f0 msgr2=0x5654340e72e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:19:08.073 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:19:08.071+0000 7f35e2fb6740 1 --2- 172.21.15.118:0/142860180 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5654340ed3f0 0x5654340e72e0 secure :-1 s=READY pgs=363 cs=0 l=1 rev1=1 crypto rx=0x7f35d00107e0 tx=0x7f35d0005fd0 comp rx=0 tx=0).stop 2024-03-24T00:19:08.073 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:19:08.071+0000 7f35e2fb6740 1 -- 172.21.15.118:0/142860180 shutdown_connections 2024-03-24T00:19:08.073 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:19:08.071+0000 7f35e2fb6740 1 --2- 172.21.15.118:0/142860180 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5654340f2f50 0x5654340e7820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:19:08.073 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:19:08.071+0000 7f35e2fb6740 1 --2- 172.21.15.118:0/142860180 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5654340edde0 0x56543410fb70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:19:08.073 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:19:08.071+0000 7f35e2fb6740 1 --2- 172.21.15.118:0/142860180 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5654340ed3f0 0x5654340e72e0 unknown :-1 s=CLOSED pgs=363 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:19:08.073 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:19:08.071+0000 7f35e2fb6740 1 -- 172.21.15.118:0/142860180 >> 172.21.15.118:0/142860180 conn(0x5654340e8960 msgr2=0x5654340ef140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:19:08.073 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:19:08.071+0000 7f35e2fb6740 1 -- 172.21.15.118:0/142860180 shutdown_connections 2024-03-24T00:19:08.073 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:19:08.071+0000 7f35e2fb6740 1 -- 172.21.15.118:0/142860180 wait complete. 2024-03-24T00:19:08.076 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:19:08.072+0000 7f35e2fb6740 1 -- start start 2024-03-24T00:19:08.076 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:19:08.072+0000 7f35e2fb6740 1 Processor -- start 2024-03-24T00:19:08.076 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:19:08.072+0000 7f35e2fb6740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5654340ed3f0 0x56543410c7d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:19:08.076 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:19:08.072+0000 7f35e2fb6740 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5654340edde0 0x56543410cd10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:19:08.076 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:19:08.072+0000 7f35e2fb6740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5654340f2f50 0x56543416de00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:19:08.076 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:19:08.072+0000 7f35e2fb6740 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f35b800f620 con 0x5654340edde0 2024-03-24T00:19:08.076 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:19:08.072+0000 7f35e2fb6740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f35b800efc0 con 0x5654340f2f50 2024-03-24T00:19:08.076 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:19:08.072+0000 7f35e2fb6740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f35b800c4c0 con 0x5654340ed3f0 2024-03-24T00:19:08.076 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:19:08.072+0000 7f35e2fb6740 10 client.0 _ll_register_callbacks cb 0x7f35c4009d50 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T00:19:08.076 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:19:08.073+0000 7f35de200640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5654340ed3f0 0x56543410c7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:19:08.077 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:19:08.073+0000 7f35de200640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5654340ed3f0 0x56543410c7d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.118:47108/0 (socket says 172.21.15.118:47108) 2024-03-24T00:19:08.077 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:19:08.073+0000 7f35de200640 1 -- 172.21.15.118:0/3397609304 learned_addr learned my addr 172.21.15.118:0/3397609304 (peer_addr_for_me v2:172.21.15.118:0/0) 2024-03-24T00:19:08.077 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:19:08.073+0000 7f35dce00640 1 --2- 172.21.15.118:0/3397609304 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5654340edde0 0x56543410cd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:19:08.077 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:19:08.073+0000 7f35de200640 1 --2- 172.21.15.118:0/3397609304 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5654340f2f50 0x56543416de00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:19:08.077 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:19:08.073+0000 7f35de200640 1 -- 172.21.15.118:0/3397609304 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5654340f2f50 msgr2=0x56543416de00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:19:08.077 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:19:08.073+0000 7f35de200640 1 --2- 172.21.15.118:0/3397609304 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5654340f2f50 0x56543416de00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:19:08.077 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:19:08.073+0000 7f35de200640 1 -- 172.21.15.118:0/3397609304 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5654340edde0 msgr2=0x56543410cd10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:19:08.077 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:19:08.073+0000 7f35de200640 1 --2- 172.21.15.118:0/3397609304 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5654340edde0 0x56543410cd10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:19:08.077 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:19:08.073+0000 7f35de200640 1 -- 172.21.15.118:0/3397609304 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x56543410fde0 con 0x5654340ed3f0 2024-03-24T00:19:08.077 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:19:08.073+0000 7f35de200640 1 --2- 172.21.15.118:0/3397609304 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5654340ed3f0 0x56543410c7d0 secure :-1 s=READY pgs=364 cs=0 l=1 rev1=1 crypto rx=0x7f35d0004060 tx=0x7f35d002b380 comp rx=0 tx=0).ready entity=mon.1 client_cookie=e8f628eb4e1bd606 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:19:08.077 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:19:08.073+0000 7f35bf400640 10 client.0 ms_handle_connect on v2:172.21.15.193:3300/0 2024-03-24T00:19:08.077 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:19:08.073+0000 7f35bf400640 1 -- 172.21.15.118:0/3397609304 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f35d0003970 con 0x5654340ed3f0 2024-03-24T00:19:08.077 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:19:08.074+0000 7f35bf400640 1 -- 172.21.15.118:0/3397609304 <== mon.1 v2:172.21.15.193:3300/0 2 ==== osd_map(90..90 src has 1..90) v4 ==== 8277+0+0 (secure 0 0 0) 0x7f35d0003b10 con 0x5654340ed3f0 2024-03-24T00:19:08.078 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:19:08.074+0000 7f35e2fb6740 1 -- 172.21.15.118:0/3397609304 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f35d000e930 con 0x5654340ed3f0 2024-03-24T00:19:08.078 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:19:08.074+0000 7f35bf400640 1 -- 172.21.15.118:0/3397609304 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) v2 ==== 24+0+0 (secure 0 0 0) 0x7f35d000eae0 con 0x5654340ed3f0 2024-03-24T00:19:08.078 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:19:08.074+0000 7f35e2fb6740 10 client.26386 fetch_fsmap learned FSMap version 28 2024-03-24T00:19:08.078 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:19:08.074+0000 7f35e2fb6740 1 -- 172.21.15.118:0/3397609304 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) v3 -- 0x56543416f360 con 0x5654340ed3f0 2024-03-24T00:19:08.078 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:19:08.074+0000 7f35bf400640 1 -- 172.21.15.118:0/3397609304 <== mon.1 v2:172.21.15.193:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f35d000e930 con 0x5654340ed3f0 2024-03-24T00:19:08.078 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:19:08.074+0000 7f35bf400640 1 -- 172.21.15.118:0/3397609304 <== mon.1 v2:172.21.15.193:3300/0 5 ==== fsmap(e 28) v1 ==== 7934+0+0 (secure 0 0 0) 0x7f35d002b730 con 0x5654340ed3f0 2024-03-24T00:19:08.078 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:19:08.074+0000 7f35e2fb6740 10 client.26386 fetch_fsmap finished waiting for FSMap version 28 2024-03-24T00:19:08.078 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:19:08.074+0000 7f35e2fb6740 10 client.26386 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-24T00:19:08.078 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:19:08.074+0000 7f35e2fb6740 20 client.26386 populate_metadata read hostname 'smithi118' 2024-03-24T00:19:08.078 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:19:08.074+0000 7f35e2fb6740 1 --2- 172.21.15.118:0/3397609304 >> [v2:172.21.15.189:6838/4074484168,v1:172.21.15.189:6839/4074484168] conn(0x56543416fab0 0x565434171ed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:19:08.078 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:19:08.074+0000 7f35e2fb6740 4 client.26386 mds_command: new command op to 24569 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-03-24T00:19:08.078 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:19:08.074+0000 7f35e2fb6740 1 -- 172.21.15.118:0/3397609304 --> [v2:172.21.15.189:6838/4074484168,v1:172.21.15.189:6839/4074484168] -- command(tid 0: {"prefix": "flush journal"}) v1 -- 0x56543405a0c0 con 0x56543416fab0 2024-03-24T00:19:08.078 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:19:08.075+0000 7f35dce00640 1 --2- 172.21.15.118:0/3397609304 >> [v2:172.21.15.189:6838/4074484168,v1:172.21.15.189:6839/4074484168] conn(0x56543416fab0 0x565434171ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:19:08.079 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:19:08.075+0000 7f35dce00640 1 --2- 172.21.15.118:0/3397609304 >> [v2:172.21.15.189:6838/4074484168,v1:172.21.15.189:6839/4074484168] conn(0x56543416fab0 0x565434171ed0 crc :-1 s=READY pgs=106 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=5fde5a99fea46773 server_cookie=61613042e1cba6da in_seq=0 out_seq=0 2024-03-24T00:19:08.079 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:19:08.075+0000 7f35bf400640 10 client.26386 ms_handle_connect on v2:172.21.15.189:6838/4074484168 2024-03-24T00:19:08.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:19:07 smithi118 ceph-mon[31535]: pgmap v501: 97 pgs: 97 active+clean; 63 MiB data, 571 MiB used, 1.0 TiB / 1.0 TiB avail; 524 KiB/s rd, 590 KiB/s wr, 208 op/s 2024-03-24T00:19:08.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:19:07 smithi189 ceph-mon[36187]: pgmap v501: 97 pgs: 97 active+clean; 63 MiB data, 571 MiB used, 1.0 TiB / 1.0 TiB avail; 524 KiB/s rd, 590 KiB/s wr, 208 op/s 2024-03-24T00:19:08.210 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-03-24T00:19:08.210 DEBUG:teuthology.orchestra.run.smithi118:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-24T00:19:08.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:19:07 smithi193 ceph-mon[36878]: pgmap v501: 97 pgs: 97 active+clean; 63 MiB data, 571 MiB used, 1.0 TiB / 1.0 TiB avail; 524 KiB/s rd, 590 KiB/s wr, 208 op/s 2024-03-24T00:19:08.498 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:08.497+0000 7ffb7e600640 1 -- 172.21.15.118:0/1997685523 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ffb781028a0 msgr2=0x7ffb78102d20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:19:08.499 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:08.497+0000 7ffb7e600640 1 --2- 172.21.15.118:0/1997685523 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ffb781028a0 0x7ffb78102d20 secure :-1 s=READY pgs=364 cs=0 l=1 rev1=1 crypto rx=0x7ffb64009900 tx=0x7ffb6402bc50 comp rx=0 tx=0).stop 2024-03-24T00:19:08.499 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:08.497+0000 7ffb7e600640 1 -- 172.21.15.118:0/1997685523 shutdown_connections 2024-03-24T00:19:08.499 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:08.497+0000 7ffb7e600640 1 --2- 172.21.15.118:0/1997685523 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7ffb78103260 0x7ffb78109af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:19:08.499 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:08.497+0000 7ffb7e600640 1 --2- 172.21.15.118:0/1997685523 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ffb781028a0 0x7ffb78102d20 unknown :-1 s=CLOSED pgs=364 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:19:08.499 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:08.497+0000 7ffb7e600640 1 --2- 172.21.15.118:0/1997685523 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7ffb781016b0 0x7ffb78101ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:19:08.499 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:08.497+0000 7ffb7e600640 1 -- 172.21.15.118:0/1997685523 >> 172.21.15.118:0/1997685523 conn(0x7ffb780fcc40 msgr2=0x7ffb780ff060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:19:08.499 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:08.498+0000 7ffb7e600640 1 -- 172.21.15.118:0/1997685523 shutdown_connections 2024-03-24T00:19:08.499 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:08.498+0000 7ffb7e600640 1 -- 172.21.15.118:0/1997685523 wait complete. 2024-03-24T00:19:08.500 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:08.498+0000 7ffb7e600640 1 Processor -- start 2024-03-24T00:19:08.500 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:08.499+0000 7ffb7e600640 1 -- start start 2024-03-24T00:19:08.500 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:19:08.499+0000 7f35bf400640 1 -- 172.21.15.118:0/3397609304 <== mds.0 v2:172.21.15.189:6838/4074484168 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+44 (crc 0 0 0) 0x56543405a0c0 con 0x56543416fab0 2024-03-24T00:19:08.500 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:19:08.499+0000 7f35bf400640 10 client.26386 handle_command_reply: tid=0 2024-03-24T00:19:08.500 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:19:08.499+0000 7f35e2fb6740 1 client.26386 shutdown 2024-03-24T00:19:08.500 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:19:08.499+0000 7f35e2fb6740 1 -- 172.21.15.118:0/3397609304 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5654340ed3f0 msgr2=0x56543410c7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:19:08.500 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:19:08.499+0000 7f35e2fb6740 1 --2- 172.21.15.118:0/3397609304 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5654340ed3f0 0x56543410c7d0 secure :-1 s=READY pgs=364 cs=0 l=1 rev1=1 crypto rx=0x7f35d0004060 tx=0x7f35d002b380 comp rx=0 tx=0).stop 2024-03-24T00:19:08.501 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:08.499+0000 7ffb7e600640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7ffb781016b0 0x7ffb78075450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:19:08.501 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:19:08.500+0000 7f35e2fb6740 1 -- 172.21.15.118:0/3397609304 shutdown_connections 2024-03-24T00:19:08.501 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:19:08.500+0000 7f35e2fb6740 1 --2- 172.21.15.118:0/3397609304 >> [v2:172.21.15.189:6838/4074484168,v1:172.21.15.189:6839/4074484168] conn(0x56543416fab0 0x565434171ed0 crc :-1 s=READY pgs=106 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:19:08.501 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:19:08.500+0000 7f35bf400640 0 client.26386 ms_handle_reset on v2:172.21.15.189:6838/4074484168 2024-03-24T00:19:08.501 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:19:08.500+0000 7f35e2fb6740 1 --2- 172.21.15.118:0/3397609304 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5654340f2f50 0x56543416de00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:19:08.501 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:19:08.500+0000 7f35e2fb6740 1 --2- 172.21.15.118:0/3397609304 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5654340edde0 0x56543410cd10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:19:08.501 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:19:08.500+0000 7f35e2fb6740 1 --2- 172.21.15.118:0/3397609304 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5654340ed3f0 0x56543410c7d0 unknown :-1 s=CLOSED pgs=364 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:19:08.502 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:19:08.500+0000 7f35e2fb6740 1 -- 172.21.15.118:0/3397609304 >> 172.21.15.118:0/3397609304 conn(0x5654340e8960 msgr2=0x5654340ec450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:19:08.502 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:08.499+0000 7ffb7e600640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ffb781028a0 0x7ffb78073a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:19:08.502 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:08.499+0000 7ffb7e600640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7ffb78103260 0x7ffb78073fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:19:08.502 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:08.499+0000 7ffb7e600640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffb7810d380 con 0x7ffb781016b0 2024-03-24T00:19:08.502 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:08.499+0000 7ffb7e600640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffb7810d200 con 0x7ffb781028a0 2024-03-24T00:19:08.502 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:08.499+0000 7ffb7e600640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffb7810d500 con 0x7ffb78103260 2024-03-24T00:19:08.502 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:08.499+0000 7ffb76a00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ffb781028a0 0x7ffb78073a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:19:08.502 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:08.499+0000 7ffb76a00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ffb781028a0 0x7ffb78073a90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.118:54004/0 (socket says 172.21.15.118:54004) 2024-03-24T00:19:08.502 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:08.499+0000 7ffb76a00640 1 -- 172.21.15.118:0/1588188322 learned_addr learned my addr 172.21.15.118:0/1588188322 (peer_addr_for_me v2:172.21.15.118:0/0) 2024-03-24T00:19:08.502 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:08.500+0000 7ffb77e00640 1 --2- 172.21.15.118:0/1588188322 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7ffb78103260 0x7ffb78073fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:19:08.503 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:08.500+0000 7ffb77400640 1 --2- 172.21.15.118:0/1588188322 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7ffb781016b0 0x7ffb78075450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:19:08.503 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:08.500+0000 7ffb76a00640 1 -- 172.21.15.118:0/1588188322 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7ffb78103260 msgr2=0x7ffb78073fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:19:08.503 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:08.500+0000 7ffb76a00640 1 --2- 172.21.15.118:0/1588188322 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7ffb78103260 0x7ffb78073fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:19:08.503 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:08.500+0000 7ffb76a00640 1 -- 172.21.15.118:0/1588188322 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7ffb781016b0 msgr2=0x7ffb78075450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:19:08.503 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:08.500+0000 7ffb76a00640 1 --2- 172.21.15.118:0/1588188322 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7ffb781016b0 0x7ffb78075450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:19:08.503 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:08.500+0000 7ffb76a00640 1 -- 172.21.15.118:0/1588188322 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffb78074860 con 0x7ffb781028a0 2024-03-24T00:19:08.503 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:08.500+0000 7ffb77400640 1 --2- 172.21.15.118:0/1588188322 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7ffb781016b0 0x7ffb78075450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-24T00:19:08.503 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:19:08.500+0000 7f35e2fb6740 1 -- 172.21.15.118:0/3397609304 shutdown_connections 2024-03-24T00:19:08.504 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:19:08.500+0000 7f35e2fb6740 1 -- 172.21.15.118:0/3397609304 wait complete. 2024-03-24T00:19:08.504 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:19:08.500+0000 7f35e2fb6740 20 client.26386 trim_cache size 0 max 16384 2024-03-24T00:19:08.504 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:08.500+0000 7ffb76a00640 1 --2- 172.21.15.118:0/1588188322 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ffb781028a0 0x7ffb78073a90 secure :-1 s=READY pgs=365 cs=0 l=1 rev1=1 crypto rx=0x7ffb6402c260 tx=0x7ffb6402c670 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:19:08.504 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:08.501+0000 7ffb6fe00640 1 -- 172.21.15.118:0/1588188322 <== mon.2 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffb64041070 con 0x7ffb781028a0 2024-03-24T00:19:08.504 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:08.501+0000 7ffb7e600640 1 -- 172.21.15.118:0/1588188322 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ffb781a1ae0 con 0x7ffb781028a0 2024-03-24T00:19:08.504 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:08.501+0000 7ffb6fe00640 1 -- 172.21.15.118:0/1588188322 <== mon.2 v2:172.21.15.189:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7ffb64004450 con 0x7ffb781028a0 2024-03-24T00:19:08.504 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:08.501+0000 7ffb7e600640 1 -- 172.21.15.118:0/1588188322 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ffb781a2080 con 0x7ffb781028a0 2024-03-24T00:19:08.505 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stdout:{ 2024-03-24T00:19:08.505 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stdout: "message": "", 2024-03-24T00:19:08.505 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stdout: "return_code": 0 2024-03-24T00:19:08.505 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stdout:} 2024-03-24T00:19:08.505 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:08.502+0000 7ffb6fe00640 1 -- 172.21.15.118:0/1588188322 <== mon.2 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffb6404b060 con 0x7ffb781028a0 2024-03-24T00:19:08.506 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:19:08.502+0000 7f35dec00640 1 -- 172.21.15.118:0/1130715760 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7f35c00a28f0 msgr2=0x7f35c00a4db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:19:08.506 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:19:08.502+0000 7f35dec00640 1 --2- 172.21.15.118:0/1130715760 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7f35c00a28f0 0x7f35c00a4db0 secure :-1 s=READY pgs=330 cs=0 l=1 rev1=1 crypto rx=0x7f35cc005ec0 tx=0x7f35cc02c8b0 comp rx=0 tx=0).stop 2024-03-24T00:19:08.506 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:19:08.502+0000 7f35dec00640 1 -- 172.21.15.118:0/1130715760 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f35d8074100 msgr2=0x7f35d819de20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:19:08.506 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:19:08.502+0000 7f35dec00640 1 --2- 172.21.15.118:0/1130715760 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f35d8074100 0x7f35d819de20 secure :-1 s=READY pgs=361 cs=0 l=1 rev1=1 crypto rx=0x7f35cc005ab0 tx=0x7f35cc005dd0 comp rx=0 tx=0).stop 2024-03-24T00:19:08.506 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:19:08.503+0000 7f35dec00640 1 -- 172.21.15.118:0/1130715760 shutdown_connections 2024-03-24T00:19:08.506 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:19:08.503+0000 7f35dec00640 1 --2- 172.21.15.118:0/1130715760 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7f35c00a28f0 0x7f35c00a4db0 unknown :-1 s=CLOSED pgs=330 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:19:08.506 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:19:08.503+0000 7f35dec00640 1 --2- 172.21.15.118:0/1130715760 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f35d810dae0 0x7f35d81a53d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:19:08.506 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:19:08.503+0000 7f35dec00640 1 --2- 172.21.15.118:0/1130715760 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f35d80756e0 0x7f35d819e360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:19:08.506 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:19:08.503+0000 7f35dec00640 1 --2- 172.21.15.118:0/1130715760 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f35d8074100 0x7f35d819de20 unknown :-1 s=CLOSED pgs=361 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:19:08.506 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:19:08.503+0000 7f35dec00640 1 -- 172.21.15.118:0/1130715760 >> 172.21.15.118:0/1130715760 conn(0x7f35d80fca80 msgr2=0x7f35d8100e80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:19:08.506 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:19:08.503+0000 7f35dec00640 1 -- 172.21.15.118:0/1130715760 shutdown_connections 2024-03-24T00:19:08.506 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:19:08.503+0000 7f35dec00640 1 -- 172.21.15.118:0/1130715760 wait complete. 2024-03-24T00:19:08.508 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:08.505+0000 7ffb6fe00640 1 -- 172.21.15.118:0/1588188322 <== mon.2 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 20) v1 ==== 137291+0+0 (secure 0 0 0) 0x7ffb64044070 con 0x7ffb781028a0 2024-03-24T00:19:08.508 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:08.505+0000 7ffb81404740 1 Processor -- start 2024-03-24T00:19:08.508 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:08.506+0000 7ffb81404740 1 -- start start 2024-03-24T00:19:08.508 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:08.506+0000 7ffb81404740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x561663440690 0x561663440ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:19:08.509 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:08.506+0000 7ffb81404740 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x561663441080 0x561663445b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:19:08.509 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:08.506+0000 7ffb81404740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5616634461f0 0x561663448680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:19:08.509 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:08.506+0000 7ffb81404740 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x56166339d250 con 0x561663441080 2024-03-24T00:19:08.509 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:08.506+0000 7ffb81404740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x561663345a30 con 0x561663440690 2024-03-24T00:19:08.509 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:08.506+0000 7ffb81404740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x5616633992f0 con 0x5616634461f0 2024-03-24T00:19:08.509 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:08.507+0000 7ffb77e00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x561663440690 0x561663440ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:19:08.509 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:08.507+0000 7ffb77e00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x561663440690 0x561663440ab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.118:54018/0 (socket says 172.21.15.118:54018) 2024-03-24T00:19:08.509 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:08.507+0000 7ffb77e00640 1 -- 172.21.15.118:0/3771252996 learned_addr learned my addr 172.21.15.118:0/3771252996 (peer_addr_for_me v2:172.21.15.118:0/0) 2024-03-24T00:19:08.509 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:08.507+0000 7ffb77400640 1 --2- 172.21.15.118:0/3771252996 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x561663441080 0x561663445b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:19:08.509 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:08.507+0000 7ffb76a00640 1 --2- 172.21.15.118:0/3771252996 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5616634461f0 0x561663448680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:19:08.509 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:08.507+0000 7ffb6fe00640 1 --2- 172.21.15.118:0/1588188322 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7ffb5c0a2800 0x7ffb5c0a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:19:08.509 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:08.507+0000 7ffb76a00640 1 -- 172.21.15.118:0/3771252996 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x561663440690 msgr2=0x561663440ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:19:08.509 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:08.507+0000 7ffb76a00640 1 --2- 172.21.15.118:0/3771252996 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x561663440690 0x561663440ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:19:08.509 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:08.507+0000 7ffb76a00640 1 -- 172.21.15.118:0/3771252996 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x561663441080 msgr2=0x561663445b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:19:08.509 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:08.507+0000 7ffb6fe00640 1 -- 172.21.15.118:0/1588188322 <== mon.2 v2:172.21.15.189:3300/0 5 ==== osd_map(90..90 src has 1..90) v4 ==== 8277+0+0 (secure 0 0 0) 0x7ffb640eff30 con 0x7ffb781028a0 2024-03-24T00:19:08.510 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:08.508+0000 7ffb76a00640 1 --2- 172.21.15.118:0/3771252996 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x561663441080 0x561663445b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:19:08.510 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:08.508+0000 7ffb76a00640 1 -- 172.21.15.118:0/3771252996 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x561663330690 con 0x5616634461f0 2024-03-24T00:19:08.510 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:08.508+0000 7ffb77400640 1 --2- 172.21.15.118:0/3771252996 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x561663441080 0x561663445b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T00:19:08.510 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:08.508+0000 7ffb76a00640 1 --2- 172.21.15.118:0/3771252996 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5616634461f0 0x561663448680 secure :-1 s=READY pgs=365 cs=0 l=1 rev1=1 crypto rx=0x7ffb640eda50 tx=0x7ffb64009260 comp rx=0 tx=0).ready entity=mon.1 client_cookie=ff022ca4d37d2e68 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:19:08.512 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:08.508+0000 7ffb6cc00640 1 -- 172.21.15.118:0/3771252996 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffb640bc3e0 con 0x5616634461f0 2024-03-24T00:19:08.512 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:08.508+0000 7ffb6cc00640 1 -- 172.21.15.118:0/3771252996 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7ffb64004c60 con 0x5616634461f0 2024-03-24T00:19:08.512 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:08.508+0000 7ffb6cc00640 1 -- 172.21.15.118:0/3771252996 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffb6400a250 con 0x5616634461f0 2024-03-24T00:19:08.512 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:08.508+0000 7ffb77e00640 1 --2- 172.21.15.118:0/1588188322 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7ffb5c0a2800 0x7ffb5c0a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:19:08.512 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:08.509+0000 7ffb81404740 1 -- 172.21.15.118:0/3771252996 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5616634461f0 msgr2=0x561663448680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:19:08.512 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:08.509+0000 7ffb81404740 1 --2- 172.21.15.118:0/3771252996 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5616634461f0 0x561663448680 secure :-1 s=READY pgs=365 cs=0 l=1 rev1=1 crypto rx=0x7ffb640eda50 tx=0x7ffb64009260 comp rx=0 tx=0).stop 2024-03-24T00:19:08.513 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:08.509+0000 7ffb77e00640 1 --2- 172.21.15.118:0/1588188322 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7ffb5c0a2800 0x7ffb5c0a4cc0 secure :-1 s=READY pgs=331 cs=0 l=1 rev1=1 crypto rx=0x7ffb70005c40 tx=0x7ffb70014040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:19:08.513 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:08.509+0000 7ffb81404740 1 -- 172.21.15.118:0/3771252996 shutdown_connections 2024-03-24T00:19:08.513 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:08.509+0000 7ffb81404740 1 --2- 172.21.15.118:0/3771252996 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5616634461f0 0x561663448680 unknown :-1 s=CLOSED pgs=365 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:19:08.513 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:08.509+0000 7ffb81404740 1 --2- 172.21.15.118:0/3771252996 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x561663441080 0x561663445b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:19:08.513 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:08.509+0000 7ffb81404740 1 --2- 172.21.15.118:0/3771252996 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x561663440690 0x561663440ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:19:08.513 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:08.509+0000 7ffb81404740 1 -- 172.21.15.118:0/3771252996 >> 172.21.15.118:0/3771252996 conn(0x561663400200 msgr2=0x56166343c000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:19:08.513 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:08.510+0000 7ffb81404740 1 -- 172.21.15.118:0/3771252996 shutdown_connections 2024-03-24T00:19:08.513 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:08.510+0000 7ffb81404740 1 -- 172.21.15.118:0/3771252996 wait complete. 2024-03-24T00:19:08.513 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:08.510+0000 7ffb81404740 1 -- start start 2024-03-24T00:19:08.513 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:08.510+0000 7ffb81404740 1 Processor -- start 2024-03-24T00:19:08.513 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:08.511+0000 7ffb81404740 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x561663440690 0x56166344a0a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:19:08.513 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:08.511+0000 7ffb81404740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x561663441080 0x56166344a5e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:19:08.513 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:08.511+0000 7ffb81404740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5616634461f0 0x561663451620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:19:08.514 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:08.511+0000 7ffb81404740 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x561663448f60 con 0x561663440690 2024-03-24T00:19:08.514 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:08.511+0000 7ffb81404740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x561663358e80 con 0x561663441080 2024-03-24T00:19:08.514 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:08.511+0000 7ffb81404740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x5616634490e0 con 0x5616634461f0 2024-03-24T00:19:08.514 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:08.511+0000 7ffb81404740 10 client.0 _ll_register_callbacks cb 0x56166336b130 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T00:19:08.514 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:08.511+0000 7ffb77e00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5616634461f0 0x561663451620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:19:08.514 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:08.511+0000 7ffb77e00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5616634461f0 0x561663451620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.118:47134/0 (socket says 172.21.15.118:47134) 2024-03-24T00:19:08.514 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:08.511+0000 7ffb77e00640 1 -- 172.21.15.118:0/1787408847 learned_addr learned my addr 172.21.15.118:0/1787408847 (peer_addr_for_me v2:172.21.15.118:0/0) 2024-03-24T00:19:08.514 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:08.512+0000 7ffb76a00640 1 --2- 172.21.15.118:0/1787408847 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x561663441080 0x56166344a5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:19:08.514 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:08.512+0000 7ffb77400640 1 --2- 172.21.15.118:0/1787408847 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x561663440690 0x56166344a0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:19:08.516 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:08.512+0000 7ffb77400640 1 -- 172.21.15.118:0/1787408847 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5616634461f0 msgr2=0x561663451620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:19:08.516 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:08.512+0000 7ffb77400640 1 --2- 172.21.15.118:0/1787408847 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5616634461f0 0x561663451620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:19:08.517 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:08.512+0000 7ffb77400640 1 -- 172.21.15.118:0/1787408847 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x561663441080 msgr2=0x56166344a5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:19:08.517 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:08.512+0000 7ffb77400640 1 --2- 172.21.15.118:0/1787408847 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x561663441080 0x56166344a5e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:19:08.517 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:08.512+0000 7ffb77400640 1 -- 172.21.15.118:0/1787408847 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x56166317b7d0 con 0x561663440690 2024-03-24T00:19:08.517 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:08.513+0000 7ffb77400640 1 --2- 172.21.15.118:0/1787408847 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x561663440690 0x56166344a0a0 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x56166343cf50 tx=0x7ffb68013b70 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d6783669029ae1d5 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:19:08.517 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:08.513+0000 7ffb61600640 10 client.0 ms_handle_connect on v2:172.21.15.118:3300/0 2024-03-24T00:19:08.517 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:08.513+0000 7ffb61600640 1 -- 172.21.15.118:0/1787408847 <== mon.0 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffb68011d40 con 0x561663440690 2024-03-24T00:19:08.517 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:08.513+0000 7ffb61600640 1 -- 172.21.15.118:0/1787408847 <== mon.0 v2:172.21.15.118:3300/0 2 ==== osd_map(90..90 src has 1..90) v4 ==== 8277+0+0 (secure 0 0 0) 0x7ffb68011b40 con 0x561663440690 2024-03-24T00:19:08.517 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:08.514+0000 7ffb81404740 1 -- 172.21.15.118:0/1787408847 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7ffb64004c60 con 0x561663440690 2024-03-24T00:19:08.517 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:08.514+0000 7ffb61600640 1 -- 172.21.15.118:0/1787408847 <== mon.0 v2:172.21.15.118:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) v2 ==== 24+0+0 (secure 0 0 0) 0x7ffb68011d40 con 0x561663440690 2024-03-24T00:19:08.517 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:08.514+0000 7ffb81404740 10 client.16233 fetch_fsmap learned FSMap version 28 2024-03-24T00:19:08.517 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:08.514+0000 7ffb81404740 1 -- 172.21.15.118:0/1787408847 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) v3 -- 0x5616630b13f0 con 0x561663440690 2024-03-24T00:19:08.517 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:08.514+0000 7ffb61600640 1 -- 172.21.15.118:0/1787408847 <== mon.0 v2:172.21.15.118:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7ffb68026950 con 0x561663440690 2024-03-24T00:19:08.517 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:08.514+0000 7ffb61600640 1 -- 172.21.15.118:0/1787408847 <== mon.0 v2:172.21.15.118:3300/0 5 ==== fsmap(e 28) v1 ==== 7934+0+0 (secure 0 0 0) 0x7ffb6801a070 con 0x561663440690 2024-03-24T00:19:08.518 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:08.515+0000 7ffb81404740 10 client.16233 fetch_fsmap finished waiting for FSMap version 28 2024-03-24T00:19:08.518 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:08.515+0000 7ffb81404740 10 client.16233 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-24T00:19:08.518 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:08.515+0000 7ffb81404740 20 client.16233 populate_metadata read hostname 'smithi118' 2024-03-24T00:19:08.518 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:08.515+0000 7ffb81404740 1 --2- 172.21.15.118:0/1787408847 >> [v2:172.21.15.189:6838/4074484168,v1:172.21.15.189:6839/4074484168] conn(0x561663455f00 0x561663458340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:19:08.518 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:08.515+0000 7ffb81404740 4 client.16233 mds_command: new command op to 24569 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-24T00:19:08.518 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:08.515+0000 7ffb81404740 1 -- 172.21.15.118:0/1787408847 --> [v2:172.21.15.189:6838/4074484168,v1:172.21.15.189:6839/4074484168] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x7ffb64004c60 con 0x561663455f00 2024-03-24T00:19:08.518 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:08.515+0000 7ffb76a00640 1 --2- 172.21.15.118:0/1787408847 >> [v2:172.21.15.189:6838/4074484168,v1:172.21.15.189:6839/4074484168] conn(0x561663455f00 0x561663458340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:19:08.518 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:08.515+0000 7ffb76a00640 1 --2- 172.21.15.118:0/1787408847 >> [v2:172.21.15.189:6838/4074484168,v1:172.21.15.189:6839/4074484168] conn(0x561663455f00 0x561663458340 crc :-1 s=READY pgs=107 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=fd9137ea1b9aff85 server_cookie=c2b7e71de4703f3f in_seq=0 out_seq=0 2024-03-24T00:19:08.518 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:08.515+0000 7ffb61600640 10 client.16233 ms_handle_connect on v2:172.21.15.189:6838/4074484168 2024-03-24T00:19:08.519 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:08.517+0000 7ffb61600640 1 -- 172.21.15.118:0/1787408847 <== mds.0 v2:172.21.15.189:6838/4074484168 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x7ffb64004c60 con 0x561663455f00 2024-03-24T00:19:08.519 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:08.517+0000 7ffb61600640 10 client.16233 handle_command_reply: tid=0 2024-03-24T00:19:08.519 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:08.517+0000 7ffb81404740 1 client.16233 shutdown 2024-03-24T00:19:08.519 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:08.517+0000 7ffb81404740 1 -- 172.21.15.118:0/1787408847 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x561663440690 msgr2=0x56166344a0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:19:08.520 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:08.517+0000 7ffb81404740 1 --2- 172.21.15.118:0/1787408847 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x561663440690 0x56166344a0a0 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x56166343cf50 tx=0x7ffb68013b70 comp rx=0 tx=0).stop 2024-03-24T00:19:08.520 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:08.517+0000 7ffb81404740 1 -- 172.21.15.118:0/1787408847 shutdown_connections 2024-03-24T00:19:08.520 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:08.517+0000 7ffb81404740 1 --2- 172.21.15.118:0/1787408847 >> [v2:172.21.15.189:6838/4074484168,v1:172.21.15.189:6839/4074484168] conn(0x561663455f00 0x561663458340 crc :-1 s=READY pgs=107 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:19:08.520 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:08.517+0000 7ffb81404740 1 --2- 172.21.15.118:0/1787408847 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5616634461f0 0x561663451620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:19:08.520 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:08.517+0000 7ffb81404740 1 --2- 172.21.15.118:0/1787408847 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x561663441080 0x56166344a5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:19:08.520 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:08.517+0000 7ffb81404740 1 --2- 172.21.15.118:0/1787408847 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x561663440690 0x56166344a0a0 unknown :-1 s=CLOSED pgs=205 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:19:08.520 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:08.517+0000 7ffb81404740 1 -- 172.21.15.118:0/1787408847 >> 172.21.15.118:0/1787408847 conn(0x561663400200 msgr2=0x561663432a90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:19:08.520 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:08.517+0000 7ffb61600640 0 client.16233 ms_handle_reset on v2:172.21.15.189:6838/4074484168 2024-03-24T00:19:08.520 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:08.518+0000 7ffb81404740 1 -- 172.21.15.118:0/1787408847 shutdown_connections 2024-03-24T00:19:08.520 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:08.518+0000 7ffb81404740 1 -- 172.21.15.118:0/1787408847 wait complete. 2024-03-24T00:19:08.520 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:08.518+0000 7ffb81404740 20 client.16233 trim_cache size 0 max 16384 2024-03-24T00:19:08.542 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:08.542+0000 7ffb81404740 1 Processor -- start 2024-03-24T00:19:08.542 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:08.542+0000 7ffb81404740 1 -- start start 2024-03-24T00:19:08.544 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:08.542+0000 7ffb81404740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x561663440690 0x561663442000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:19:08.544 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:08.542+0000 7ffb81404740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x561663441080 0x5616634515e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:19:08.544 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:08.542+0000 7ffb81404740 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5616634461f0 0x5616634496c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:19:08.544 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:08.542+0000 7ffb81404740 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffb781a6a30 con 0x5616634461f0 2024-03-24T00:19:08.544 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:08.542+0000 7ffb81404740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffb781a83b0 con 0x561663441080 2024-03-24T00:19:08.544 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:08.542+0000 7ffb81404740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffb781a7090 con 0x561663440690 2024-03-24T00:19:08.544 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:08.542+0000 7ffb76a00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x561663441080 0x5616634515e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:19:08.544 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:08.542+0000 7ffb77400640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x561663440690 0x561663442000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:19:08.544 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:08.542+0000 7ffb77400640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x561663440690 0x561663442000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.118:47142/0 (socket says 172.21.15.118:47142) 2024-03-24T00:19:08.545 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:08.542+0000 7ffb76a00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x561663441080 0x5616634515e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.118:54038/0 (socket says 172.21.15.118:54038) 2024-03-24T00:19:08.545 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:08.542+0000 7ffb77400640 1 -- 172.21.15.118:0/1951673228 learned_addr learned my addr 172.21.15.118:0/1951673228 (peer_addr_for_me v2:172.21.15.118:0/0) 2024-03-24T00:19:08.545 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:08.543+0000 7ffb77e00640 1 --2- 172.21.15.118:0/1951673228 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5616634461f0 0x5616634496c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:19:08.545 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:08.543+0000 7ffb77400640 1 -- 172.21.15.118:0/1951673228 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x561663441080 msgr2=0x5616634515e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:19:08.545 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:08.543+0000 7ffb77400640 1 --2- 172.21.15.118:0/1951673228 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x561663441080 0x5616634515e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:19:08.545 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:08.543+0000 7ffb77400640 1 -- 172.21.15.118:0/1951673228 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5616634461f0 msgr2=0x5616634496c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:19:08.545 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:08.543+0000 7ffb77400640 1 --2- 172.21.15.118:0/1951673228 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5616634461f0 0x5616634496c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:19:08.545 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:08.543+0000 7ffb77400640 1 -- 172.21.15.118:0/1951673228 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5616630710f0 con 0x561663440690 2024-03-24T00:19:08.545 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:08.543+0000 7ffb77e00640 1 --2- 172.21.15.118:0/1951673228 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5616634461f0 0x5616634496c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T00:19:08.545 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:08.543+0000 7ffb77400640 1 --2- 172.21.15.118:0/1951673228 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x561663440690 0x561663442000 secure :-1 s=READY pgs=366 cs=0 l=1 rev1=1 crypto rx=0x7ffb68024410 tx=0x7ffb6801af60 comp rx=0 tx=0).ready entity=mon.1 client_cookie=e9cdfa077c7fdea1 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:19:08.545 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:08.543+0000 7ffb63e00640 1 -- 172.21.15.118:0/1951673228 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffb68011d40 con 0x561663440690 2024-03-24T00:19:08.545 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:08.543+0000 7ffb63e00640 1 -- 172.21.15.118:0/1951673228 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7ffb6802b070 con 0x561663440690 2024-03-24T00:19:08.545 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:08.544+0000 7ffb63e00640 1 -- 172.21.15.118:0/1951673228 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffb6800f8b0 con 0x561663440690 2024-03-24T00:19:08.548 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:08.544+0000 7ffb81404740 1 -- 172.21.15.118:0/1951673228 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x561663440690 msgr2=0x561663442000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:19:08.548 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:08.544+0000 7ffb81404740 1 --2- 172.21.15.118:0/1951673228 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x561663440690 0x561663442000 secure :-1 s=READY pgs=366 cs=0 l=1 rev1=1 crypto rx=0x7ffb68024410 tx=0x7ffb6801af60 comp rx=0 tx=0).stop 2024-03-24T00:19:08.548 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:08.544+0000 7ffb81404740 1 -- 172.21.15.118:0/1951673228 shutdown_connections 2024-03-24T00:19:08.548 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:08.544+0000 7ffb81404740 1 --2- 172.21.15.118:0/1951673228 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5616634461f0 0x5616634496c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:19:08.548 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:08.544+0000 7ffb81404740 1 --2- 172.21.15.118:0/1951673228 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x561663441080 0x5616634515e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:19:08.548 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:08.544+0000 7ffb81404740 1 --2- 172.21.15.118:0/1951673228 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x561663440690 0x561663442000 unknown :-1 s=CLOSED pgs=366 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:19:08.548 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:08.544+0000 7ffb81404740 1 -- 172.21.15.118:0/1951673228 >> 172.21.15.118:0/1951673228 conn(0x561663400200 msgr2=0x5616634329d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:19:08.548 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:08.545+0000 7ffb81404740 1 -- 172.21.15.118:0/1951673228 shutdown_connections 2024-03-24T00:19:08.548 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:08.545+0000 7ffb81404740 1 -- 172.21.15.118:0/1951673228 wait complete. 2024-03-24T00:19:08.549 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:08.545+0000 7ffb81404740 1 -- start start 2024-03-24T00:19:08.549 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:08.545+0000 7ffb81404740 1 Processor -- start 2024-03-24T00:19:08.549 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:08.546+0000 7ffb81404740 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x561663440690 0x56166344f100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:19:08.549 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:08.546+0000 7ffb81404740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x561663441080 0x56166344f640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:19:08.549 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:08.546+0000 7ffb81404740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5616634461f0 0x561663450650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:19:08.549 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:08.546+0000 7ffb81404740 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffb781a7d50 con 0x561663440690 2024-03-24T00:19:08.549 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:08.546+0000 7ffb81404740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffb781a76f0 con 0x561663441080 2024-03-24T00:19:08.549 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:08.546+0000 7ffb81404740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffb781a4bb0 con 0x5616634461f0 2024-03-24T00:19:08.549 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:08.546+0000 7ffb81404740 10 client.0 _ll_register_callbacks cb 0x7ffb68009530 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T00:19:08.549 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:08.546+0000 7ffb76a00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x561663441080 0x56166344f640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:19:08.549 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:08.546+0000 7ffb77400640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x561663440690 0x56166344f100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:19:08.549 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:08.546+0000 7ffb76a00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x561663441080 0x56166344f640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.118:54050/0 (socket says 172.21.15.118:54050) 2024-03-24T00:19:08.549 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:08.546+0000 7ffb76a00640 1 -- 172.21.15.118:0/4140951298 learned_addr learned my addr 172.21.15.118:0/4140951298 (peer_addr_for_me v2:172.21.15.118:0/0) 2024-03-24T00:19:08.550 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:08.546+0000 7ffb77e00640 1 --2- 172.21.15.118:0/4140951298 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5616634461f0 0x561663450650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:19:08.550 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:08.546+0000 7ffb77400640 1 -- 172.21.15.118:0/4140951298 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5616634461f0 msgr2=0x561663450650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:19:08.550 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:08.546+0000 7ffb77400640 1 --2- 172.21.15.118:0/4140951298 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5616634461f0 0x561663450650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:19:08.550 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:08.546+0000 7ffb77400640 1 -- 172.21.15.118:0/4140951298 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x561663441080 msgr2=0x56166344f640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:19:08.550 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:08.546+0000 7ffb77400640 1 --2- 172.21.15.118:0/4140951298 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x561663441080 0x56166344f640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:19:08.550 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:08.546+0000 7ffb77400640 1 -- 172.21.15.118:0/4140951298 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x561663450c40 con 0x561663440690 2024-03-24T00:19:08.550 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:08.546+0000 7ffb76a00640 1 --2- 172.21.15.118:0/4140951298 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x561663441080 0x56166344f640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T00:19:08.550 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:08.546+0000 7ffb77e00640 1 --2- 172.21.15.118:0/4140951298 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5616634461f0 0x561663450650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T00:19:08.550 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:08.547+0000 7ffb77400640 1 --2- 172.21.15.118:0/4140951298 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x561663440690 0x56166344f100 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7ffb6801aff0 tx=0x7ffb68005bd0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=22824ea8a7eb9ce7 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:19:08.550 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:08.547+0000 7ffb61600640 10 client.0 ms_handle_connect on v2:172.21.15.118:3300/0 2024-03-24T00:19:08.550 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:08.547+0000 7ffb61600640 1 -- 172.21.15.118:0/4140951298 <== mon.0 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ffb68033020 con 0x561663440690 2024-03-24T00:19:08.550 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:08.547+0000 7ffb61600640 1 -- 172.21.15.118:0/4140951298 <== mon.0 v2:172.21.15.118:3300/0 2 ==== osd_map(90..90 src has 1..90) v4 ==== 8277+0+0 (secure 0 0 0) 0x7ffb6801aad0 con 0x561663440690 2024-03-24T00:19:08.550 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:08.547+0000 7ffb81404740 1 -- 172.21.15.118:0/4140951298 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7ffb6802b070 con 0x561663440690 2024-03-24T00:19:08.550 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:08.547+0000 7ffb61600640 1 -- 172.21.15.118:0/4140951298 <== mon.0 v2:172.21.15.118:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) v2 ==== 24+0+0 (secure 0 0 0) 0x7ffb6803e050 con 0x561663440690 2024-03-24T00:19:08.551 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:08.547+0000 7ffb81404740 10 client.16239 fetch_fsmap learned FSMap version 28 2024-03-24T00:19:08.551 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:08.547+0000 7ffb81404740 1 -- 172.21.15.118:0/4140951298 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) v3 -- 0x5616634b1c40 con 0x561663440690 2024-03-24T00:19:08.552 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:08.548+0000 7ffb61600640 1 -- 172.21.15.118:0/4140951298 <== mon.0 v2:172.21.15.118:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7ffb6802b070 con 0x561663440690 2024-03-24T00:19:08.552 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:08.548+0000 7ffb61600640 1 -- 172.21.15.118:0/4140951298 <== mon.0 v2:172.21.15.118:3300/0 5 ==== fsmap(e 28) v1 ==== 7934+0+0 (secure 0 0 0) 0x7ffb6801a070 con 0x561663440690 2024-03-24T00:19:08.552 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:08.548+0000 7ffb81404740 10 client.16239 fetch_fsmap finished waiting for FSMap version 28 2024-03-24T00:19:08.552 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:08.548+0000 7ffb81404740 10 client.16239 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-24T00:19:08.553 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:08.548+0000 7ffb81404740 20 client.16239 populate_metadata read hostname 'smithi118' 2024-03-24T00:19:08.553 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:08.548+0000 7ffb81404740 1 --2- 172.21.15.118:0/4140951298 >> [v2:172.21.15.189:6838/4074484168,v1:172.21.15.189:6839/4074484168] conn(0x5616634b24d0 0x5616634b48c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:19:08.553 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:08.548+0000 7ffb81404740 4 client.16239 mds_command: new command op to 24569 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-24T00:19:08.553 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:08.548+0000 7ffb81404740 1 -- 172.21.15.118:0/4140951298 --> [v2:172.21.15.189:6838/4074484168,v1:172.21.15.189:6839/4074484168] -- command(tid 0: {"prefix": "scrub status"}) v1 -- 0x5616633aa790 con 0x5616634b24d0 2024-03-24T00:19:08.553 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:08.548+0000 7ffb76a00640 1 --2- 172.21.15.118:0/4140951298 >> [v2:172.21.15.189:6838/4074484168,v1:172.21.15.189:6839/4074484168] conn(0x5616634b24d0 0x5616634b48c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:19:08.553 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:08.549+0000 7ffb76a00640 1 --2- 172.21.15.118:0/4140951298 >> [v2:172.21.15.189:6838/4074484168,v1:172.21.15.189:6839/4074484168] conn(0x5616634b24d0 0x5616634b48c0 crc :-1 s=READY pgs=108 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=2b5604b58b46ecb7 server_cookie=d9f8f1706585cc18 in_seq=0 out_seq=0 2024-03-24T00:19:08.553 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:08.549+0000 7ffb61600640 10 client.16239 ms_handle_connect on v2:172.21.15.189:6838/4074484168 2024-03-24T00:19:08.553 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:08.550+0000 7ffb61600640 1 -- 172.21.15.118:0/4140951298 <== mds.0 v2:172.21.15.189:6838/4074484168 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+63 (crc 0 0 0) 0x5616633aa790 con 0x5616634b24d0 2024-03-24T00:19:08.553 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:08.550+0000 7ffb61600640 10 client.16239 handle_command_reply: tid=0 2024-03-24T00:19:08.553 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:08.550+0000 7ffb81404740 1 client.16239 shutdown 2024-03-24T00:19:08.553 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:08.550+0000 7ffb81404740 1 -- 172.21.15.118:0/4140951298 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x561663440690 msgr2=0x56166344f100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:19:08.553 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:08.550+0000 7ffb81404740 1 --2- 172.21.15.118:0/4140951298 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x561663440690 0x56166344f100 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7ffb6801aff0 tx=0x7ffb68005bd0 comp rx=0 tx=0).stop 2024-03-24T00:19:08.553 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:08.550+0000 7ffb81404740 1 -- 172.21.15.118:0/4140951298 shutdown_connections 2024-03-24T00:19:08.553 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:08.550+0000 7ffb81404740 1 --2- 172.21.15.118:0/4140951298 >> [v2:172.21.15.189:6838/4074484168,v1:172.21.15.189:6839/4074484168] conn(0x5616634b24d0 0x5616634b48c0 crc :-1 s=READY pgs=108 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:19:08.554 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:08.550+0000 7ffb81404740 1 --2- 172.21.15.118:0/4140951298 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5616634461f0 0x561663450650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:19:08.554 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:08.550+0000 7ffb81404740 1 --2- 172.21.15.118:0/4140951298 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x561663441080 0x56166344f640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:19:08.554 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:08.550+0000 7ffb61600640 0 client.16239 ms_handle_reset on v2:172.21.15.189:6838/4074484168 2024-03-24T00:19:08.554 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:08.550+0000 7ffb81404740 1 --2- 172.21.15.118:0/4140951298 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x561663440690 0x56166344f100 unknown :-1 s=CLOSED pgs=206 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:19:08.554 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:08.550+0000 7ffb81404740 1 -- 172.21.15.118:0/4140951298 >> 172.21.15.118:0/4140951298 conn(0x561663400200 msgr2=0x561663442000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:19:08.554 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:08.550+0000 7ffb81404740 1 -- 172.21.15.118:0/4140951298 shutdown_connections 2024-03-24T00:19:08.554 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:08.550+0000 7ffb81404740 1 -- 172.21.15.118:0/4140951298 wait complete. 2024-03-24T00:19:08.554 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:08.551+0000 7ffb81404740 20 client.16239 trim_cache size 0 max 16384 2024-03-24T00:19:08.554 INFO:teuthology.orchestra.run.smithi118.stdout:{ 2024-03-24T00:19:08.554 INFO:teuthology.orchestra.run.smithi118.stdout: "status": "no active scrubs running", 2024-03-24T00:19:08.555 INFO:teuthology.orchestra.run.smithi118.stdout: "scrubs": {} 2024-03-24T00:19:08.555 INFO:teuthology.orchestra.run.smithi118.stdout:} 2024-03-24T00:19:08.555 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:08.551+0000 7ffb7e600640 1 -- 172.21.15.118:0/1588188322 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7ffb5c0a2800 msgr2=0x7ffb5c0a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:19:08.556 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:08.551+0000 7ffb7e600640 1 --2- 172.21.15.118:0/1588188322 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7ffb5c0a2800 0x7ffb5c0a4cc0 secure :-1 s=READY pgs=331 cs=0 l=1 rev1=1 crypto rx=0x7ffb70005c40 tx=0x7ffb70014040 comp rx=0 tx=0).stop 2024-03-24T00:19:08.556 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:08.551+0000 7ffb7e600640 1 -- 172.21.15.118:0/1588188322 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ffb781028a0 msgr2=0x7ffb78073a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:19:08.556 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:08.551+0000 7ffb7e600640 1 --2- 172.21.15.118:0/1588188322 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ffb781028a0 0x7ffb78073a90 secure :-1 s=READY pgs=365 cs=0 l=1 rev1=1 crypto rx=0x7ffb6402c260 tx=0x7ffb6402c670 comp rx=0 tx=0).stop 2024-03-24T00:19:08.556 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:08.551+0000 7ffb7e600640 1 -- 172.21.15.118:0/1588188322 shutdown_connections 2024-03-24T00:19:08.556 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:08.551+0000 7ffb7e600640 1 --2- 172.21.15.118:0/1588188322 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7ffb5c0a2800 0x7ffb5c0a4cc0 unknown :-1 s=CLOSED pgs=331 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:19:08.556 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:08.551+0000 7ffb7e600640 1 --2- 172.21.15.118:0/1588188322 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7ffb78103260 0x7ffb78073fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:19:08.556 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:08.551+0000 7ffb7e600640 1 --2- 172.21.15.118:0/1588188322 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ffb781028a0 0x7ffb78073a90 unknown :-1 s=CLOSED pgs=365 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:19:08.556 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:08.551+0000 7ffb7e600640 1 --2- 172.21.15.118:0/1588188322 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7ffb781016b0 0x7ffb78075450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:19:08.556 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:08.551+0000 7ffb7e600640 1 -- 172.21.15.118:0/1588188322 >> 172.21.15.118:0/1588188322 conn(0x7ffb780fcc40 msgr2=0x7ffb78105c20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:19:08.556 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:08.552+0000 7ffb7e600640 1 -- 172.21.15.118:0/1588188322 shutdown_connections 2024-03-24T00:19:08.556 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:08.552+0000 7ffb7e600640 1 -- 172.21.15.118:0/1588188322 wait complete. 2024-03-24T00:19:08.570 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-03-24T00:19:08.570 DEBUG:teuthology.orchestra.run.smithi118:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-24T00:19:08.814 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:08.813+0000 7f06f8a00640 1 -- 172.21.15.118:0/2116935569 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f06f4101f70 msgr2=0x7f06f4104360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:19:08.814 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:08.813+0000 7f06f8a00640 1 --2- 172.21.15.118:0/2116935569 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f06f4101f70 0x7f06f4104360 secure :-1 s=READY pgs=366 cs=0 l=1 rev1=1 crypto rx=0x7f06e4009900 tx=0x7f06e402bc30 comp rx=0 tx=0).stop 2024-03-24T00:19:08.814 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:08.813+0000 7f06f8a00640 1 -- 172.21.15.118:0/2116935569 shutdown_connections 2024-03-24T00:19:08.814 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:08.813+0000 7f06f8a00640 1 --2- 172.21.15.118:0/2116935569 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f06f41071d0 0x7f06f4109620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:19:08.815 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:08.813+0000 7f06f8a00640 1 --2- 172.21.15.118:0/2116935569 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f06f41048a0 0x7f06f4106c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:19:08.815 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:08.813+0000 7f06f8a00640 1 --2- 172.21.15.118:0/2116935569 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f06f4101f70 0x7f06f4104360 unknown :-1 s=CLOSED pgs=366 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:19:08.815 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:08.813+0000 7f06f8a00640 1 -- 172.21.15.118:0/2116935569 >> 172.21.15.118:0/2116935569 conn(0x7f06f40fb980 msgr2=0x7f06f40fdda0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:19:08.815 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:08.813+0000 7f06f8a00640 1 -- 172.21.15.118:0/2116935569 shutdown_connections 2024-03-24T00:19:08.815 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:08.813+0000 7f06f8a00640 1 -- 172.21.15.118:0/2116935569 wait complete. 2024-03-24T00:19:08.817 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:08.814+0000 7f06f8a00640 1 Processor -- start 2024-03-24T00:19:08.817 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:08.814+0000 7f06f8a00640 1 -- start start 2024-03-24T00:19:08.817 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:08.815+0000 7f06f8a00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f06f4101f70 0x7f06f4199a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:19:08.817 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:08.815+0000 7f06f8a00640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f06f41048a0 0x7f06f4199f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:19:08.817 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:08.815+0000 7f06f8a00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f06f41071d0 0x7f06f41a1000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:19:08.817 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:08.815+0000 7f06f8a00640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f06f410d180 con 0x7f06f41048a0 2024-03-24T00:19:08.817 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:08.815+0000 7f06f8a00640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f06f410d000 con 0x7f06f41071d0 2024-03-24T00:19:08.817 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:08.815+0000 7f06f8a00640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f06f410d300 con 0x7f06f4101f70 2024-03-24T00:19:08.818 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:08.815+0000 7f06f3400640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f06f4101f70 0x7f06f4199a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:19:08.818 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:08.815+0000 7f06f3400640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f06f4101f70 0x7f06f4199a50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.118:47166/0 (socket says 172.21.15.118:47166) 2024-03-24T00:19:08.818 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:08.815+0000 7f06f3400640 1 -- 172.21.15.118:0/266970391 learned_addr learned my addr 172.21.15.118:0/266970391 (peer_addr_for_me v2:172.21.15.118:0/0) 2024-03-24T00:19:08.818 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:08.815+0000 7f06f3e00640 1 --2- 172.21.15.118:0/266970391 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f06f41071d0 0x7f06f41a1000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:19:08.818 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:08.815+0000 7f06f2a00640 1 --2- 172.21.15.118:0/266970391 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f06f41048a0 0x7f06f4199f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:19:08.818 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:08.815+0000 7f06f3400640 1 -- 172.21.15.118:0/266970391 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f06f41071d0 msgr2=0x7f06f41a1000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:19:08.818 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:08.815+0000 7f06f3400640 1 --2- 172.21.15.118:0/266970391 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f06f41071d0 0x7f06f41a1000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:19:08.818 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:08.815+0000 7f06f3400640 1 -- 172.21.15.118:0/266970391 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f06f41048a0 msgr2=0x7f06f4199f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:19:08.818 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:08.815+0000 7f06f3400640 1 --2- 172.21.15.118:0/266970391 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f06f41048a0 0x7f06f4199f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:19:08.818 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:08.815+0000 7f06f3400640 1 -- 172.21.15.118:0/266970391 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f06f41a1670 con 0x7f06f4101f70 2024-03-24T00:19:08.818 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:08.815+0000 7f06f2a00640 1 --2- 172.21.15.118:0/266970391 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f06f41048a0 0x7f06f4199f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T00:19:08.818 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:08.816+0000 7f06f3400640 1 --2- 172.21.15.118:0/266970391 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f06f4101f70 0x7f06f4199a50 secure :-1 s=READY pgs=367 cs=0 l=1 rev1=1 crypto rx=0x7f06e4009230 tx=0x7f06e402c650 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:19:08.818 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:08.816+0000 7f06ebe00640 1 -- 172.21.15.118:0/266970391 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f06e4041070 con 0x7f06f4101f70 2024-03-24T00:19:08.818 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:08.816+0000 7f06f8a00640 1 -- 172.21.15.118:0/266970391 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f06f41a1900 con 0x7f06f4101f70 2024-03-24T00:19:08.819 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:08.816+0000 7f06ebe00640 1 -- 172.21.15.118:0/266970391 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f06e4004450 con 0x7f06f4101f70 2024-03-24T00:19:08.819 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:08.816+0000 7f06ebe00640 1 -- 172.21.15.118:0/266970391 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f06e4003db0 con 0x7f06f4101f70 2024-03-24T00:19:08.819 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:08.816+0000 7f06f8a00640 1 -- 172.21.15.118:0/266970391 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f06f41a1df0 con 0x7f06f4101f70 2024-03-24T00:19:08.819 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:08.817+0000 7f06f8a00640 1 -- 172.21.15.118:0/266970391 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f06f4108d20 con 0x7f06f4101f70 2024-03-24T00:19:08.820 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:08.819+0000 7f06ebe00640 1 -- 172.21.15.118:0/266970391 <== mon.1 v2:172.21.15.193:3300/0 4 ==== mgrmap(e 20) v1 ==== 137291+0+0 (secure 0 0 0) 0x7f06e4044070 con 0x7f06f4101f70 2024-03-24T00:19:08.821 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:08.821+0000 7f06ebe00640 1 --2- 172.21.15.118:0/266970391 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7f06d80a2800 0x7f06d80a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:19:08.821 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:08.821+0000 7f06ebe00640 1 -- 172.21.15.118:0/266970391 <== mon.1 v2:172.21.15.193:3300/0 5 ==== osd_map(90..90 src has 1..90) v4 ==== 8277+0+0 (secure 0 0 0) 0x7f06e40f06f0 con 0x7f06f4101f70 2024-03-24T00:19:08.821 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:08.821+0000 7f06f2a00640 1 --2- 172.21.15.118:0/266970391 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7f06d80a2800 0x7f06d80a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:19:08.822 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:08.823+0000 7f06f2a00640 1 --2- 172.21.15.118:0/266970391 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7f06d80a2800 0x7f06d80a4cc0 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 crypto rx=0x7f06f419af60 tx=0x7f06e0005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:19:08.825 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:08.825+0000 7f06ebe00640 1 -- 172.21.15.118:0/266970391 <== mon.1 v2:172.21.15.193:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f06e40bd760 con 0x7f06f4101f70 2024-03-24T00:19:09.303 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:09.303+0000 7f06f8a00640 1 -- 172.21.15.118:0/266970391 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f06f40622a0 con 0x7f06f4101f70 2024-03-24T00:19:09.303 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:09.304+0000 7f06ebe00640 1 -- 172.21.15.118:0/266970391 <== mon.1 v2:172.21.15.193:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 28 v28) v1 ==== 94+0+11478 (secure 0 0 0) 0x7f06e40bd060 con 0x7f06f4101f70 2024-03-24T00:19:09.304 INFO:teuthology.orchestra.run.smithi118.stdout: 2024-03-24T00:19:09.304 INFO:teuthology.orchestra.run.smithi118.stdout:{"epoch":28,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14385,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.118:6835/3194741373","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6834","nonce":3194741373},{"type":"v1","addr":"172.21.15.118:6835","nonce":3194741373}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":4},{"gid":14397,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.118:6837/1374462956","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6836","nonce":1374462956},{"type":"v1","addr":"172.21.15.118:6837","nonce":1374462956}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":5}],"filesystems":[{"mdsmap":{"epoch":28,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-03-24T00:13:01.782474+0000","modified":"2024-03-24T00:15:38.235450+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24569,"mds_1":24550,"mds_2":24511,"mds_3":24499,"mds_4":24488},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14469":{"gid":14469,"name":"a","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.118:6841/3040946537","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6840","nonce":3040946537},{"type":"v1","addr":"172.21.15.118:6841","nonce":3040946537}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24439":{"gid":24439,"name":"l","rank":3,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.193:6833/1023428412","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6832","nonce":1023428412},{"type":"v1","addr":"172.21.15.193:6833","nonce":1023428412}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24440":{"gid":24440,"name":"k","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.189:6833/2740024078","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6832","nonce":2740024078},{"type":"v1","addr":"172.21.15.189:6833","nonce":2740024078}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24488":{"gid":24488,"name":"e","rank":4,"incarnation":25,"state":"up:active","state_seq":10,"addr":"172.21.15.189:6835/599473205","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6834","nonce":599473205},{"type":"v1","addr":"172.21.15.189:6835","nonce":599473205}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24499":{"gid":24499,"name":"f","rank":3,"incarnation":23,"state":"up:active","state_seq":9,"addr":"172.21.15.193:6835/2055310541","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6834","nonce":2055310541},{"type":"v1","addr":"172.21.15.193:6835","nonce":2055310541}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24511":{"gid":24511,"name":"c","rank":2,"incarnation":21,"state":"up:active","state_seq":8,"addr":"172.21.15.193:6837/2921520053","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6836","nonce":2921520053},{"type":"v1","addr":"172.21.15.193:6837","nonce":2921520053}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24515":{"gid":24515,"name":"h","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.189:6837/3450187137","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6836","nonce":3450187137},{"type":"v1","addr":"172.21.15.189:6837","nonce":3450187137}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24532":{"gid":24532,"name":"i","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.193:6839/2126842603","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6838","nonce":2126842603},{"type":"v1","addr":"172.21.15.193:6839","nonce":2126842603}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24550":{"gid":24550,"name":"g","rank":1,"incarnation":19,"state":"up:active","state_seq":6,"addr":"172.21.15.118:6839/3285788503","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6838","nonce":3285788503},{"type":"v1","addr":"172.21.15.118:6839","nonce":3285788503}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24569":{"gid":24569,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.189:6839/4074484168","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6838","nonce":4074484168},{"type":"v1","addr":"172.21.15.189:6839","nonce":4074484168}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24569,"qdb_cluster":[24569,24499,24488,24511,24550]},"id":1}]} 2024-03-24T00:19:09.304 INFO:teuthology.orchestra.run.smithi118.stderr:dumped fsmap epoch 28 2024-03-24T00:19:09.306 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:09.306+0000 7f06f8a00640 1 -- 172.21.15.118:0/266970391 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7f06d80a2800 msgr2=0x7f06d80a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:19:09.306 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:09.306+0000 7f06f8a00640 1 --2- 172.21.15.118:0/266970391 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7f06d80a2800 0x7f06d80a4cc0 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 crypto rx=0x7f06f419af60 tx=0x7f06e0005f90 comp rx=0 tx=0).stop 2024-03-24T00:19:09.306 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:09.306+0000 7f06f8a00640 1 -- 172.21.15.118:0/266970391 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f06f4101f70 msgr2=0x7f06f4199a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:19:09.306 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:09.306+0000 7f06f8a00640 1 --2- 172.21.15.118:0/266970391 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f06f4101f70 0x7f06f4199a50 secure :-1 s=READY pgs=367 cs=0 l=1 rev1=1 crypto rx=0x7f06e4009230 tx=0x7f06e402c650 comp rx=0 tx=0).stop 2024-03-24T00:19:09.306 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:09.306+0000 7f06f8a00640 1 -- 172.21.15.118:0/266970391 shutdown_connections 2024-03-24T00:19:09.306 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:09.306+0000 7f06f8a00640 1 --2- 172.21.15.118:0/266970391 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7f06d80a2800 0x7f06d80a4cc0 unknown :-1 s=CLOSED pgs=332 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:19:09.307 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:09.306+0000 7f06f8a00640 1 --2- 172.21.15.118:0/266970391 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f06f41071d0 0x7f06f41a1000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:19:09.307 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:09.306+0000 7f06f8a00640 1 --2- 172.21.15.118:0/266970391 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f06f41048a0 0x7f06f4199f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:19:09.307 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:09.306+0000 7f06f8a00640 1 --2- 172.21.15.118:0/266970391 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f06f4101f70 0x7f06f4199a50 unknown :-1 s=CLOSED pgs=367 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:19:09.307 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:09.306+0000 7f06f8a00640 1 -- 172.21.15.118:0/266970391 >> 172.21.15.118:0/266970391 conn(0x7f06f40fb980 msgr2=0x7f06f4101690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:19:09.307 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:09.306+0000 7f06f8a00640 1 -- 172.21.15.118:0/266970391 shutdown_connections 2024-03-24T00:19:09.307 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:09.306+0000 7f06f8a00640 1 -- 172.21.15.118:0/266970391 wait complete. 2024-03-24T00:19:09.324 INFO:teuthology.orchestra.run:Running command with timeout 120 2024-03-24T00:19:09.324 DEBUG:teuthology.orchestra.run.smithi118:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:4 damage ls 2024-03-24T00:19:09.514 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:09.511+0000 7fe4a7e00640 1 -- 172.21.15.118:0/1084455514 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe4a81015a0 msgr2=0x7fe4a81019c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:19:09.514 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:09.511+0000 7fe4a7e00640 1 --2- 172.21.15.118:0/1084455514 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe4a81015a0 0x7fe4a81019c0 secure :-1 s=READY pgs=367 cs=0 l=1 rev1=1 crypto rx=0x7fe498009900 tx=0x7fe49802bc30 comp rx=0 tx=0).stop 2024-03-24T00:19:09.514 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:09.511+0000 7fe4a7e00640 1 -- 172.21.15.118:0/1084455514 shutdown_connections 2024-03-24T00:19:09.514 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:09.511+0000 7fe4a7e00640 1 --2- 172.21.15.118:0/1084455514 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fe4a8103150 0x7fe4a81099e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:19:09.515 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:09.511+0000 7fe4a7e00640 1 --2- 172.21.15.118:0/1084455514 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fe4a8102790 0x7fe4a8102c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:19:09.515 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:09.511+0000 7fe4a7e00640 1 --2- 172.21.15.118:0/1084455514 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe4a81015a0 0x7fe4a81019c0 unknown :-1 s=CLOSED pgs=367 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:19:09.515 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:09.511+0000 7fe4a7e00640 1 -- 172.21.15.118:0/1084455514 >> 172.21.15.118:0/1084455514 conn(0x7fe4a80fcb10 msgr2=0x7fe4a80fef50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:19:09.515 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:09.511+0000 7fe4a7e00640 1 -- 172.21.15.118:0/1084455514 shutdown_connections 2024-03-24T00:19:09.515 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:09.511+0000 7fe4a7e00640 1 -- 172.21.15.118:0/1084455514 wait complete. 2024-03-24T00:19:09.515 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:09.512+0000 7fe4a7e00640 1 Processor -- start 2024-03-24T00:19:09.515 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:09.512+0000 7fe4a7e00640 1 -- start start 2024-03-24T00:19:09.515 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:09.512+0000 7fe4a7e00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fe4a81015a0 0x7fe4a8199b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:19:09.515 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:09.512+0000 7fe4a7e00640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fe4a8102790 0x7fe4a819a040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:19:09.515 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:09.512+0000 7fe4a7e00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe4a8103150 0x7fe4a81a10b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:19:09.515 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:09.512+0000 7fe4a7e00640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe4a810d260 con 0x7fe4a8102790 2024-03-24T00:19:09.515 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:09.512+0000 7fe4a7e00640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe4a810d0e0 con 0x7fe4a8103150 2024-03-24T00:19:09.515 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:09.512+0000 7fe4a7e00640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe4a810d3e0 con 0x7fe4a81015a0 2024-03-24T00:19:09.515 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:09.513+0000 7fe4a6a00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fe4a81015a0 0x7fe4a8199b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:19:09.516 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:09.513+0000 7fe4a7400640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe4a8103150 0x7fe4a81a10b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:19:09.516 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:09.513+0000 7fe4a7400640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe4a8103150 0x7fe4a81a10b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.118:54068/0 (socket says 172.21.15.118:54068) 2024-03-24T00:19:09.516 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:09.513+0000 7fe4a6a00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fe4a81015a0 0x7fe4a8199b00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.118:47180/0 (socket says 172.21.15.118:47180) 2024-03-24T00:19:09.516 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:09.513+0000 7fe4a7400640 1 -- 172.21.15.118:0/2805784131 learned_addr learned my addr 172.21.15.118:0/2805784131 (peer_addr_for_me v2:172.21.15.118:0/0) 2024-03-24T00:19:09.516 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:09.513+0000 7fe49fe00640 1 --2- 172.21.15.118:0/2805784131 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fe4a8102790 0x7fe4a819a040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:19:09.516 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:09.513+0000 7fe4a7400640 1 -- 172.21.15.118:0/2805784131 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fe4a81015a0 msgr2=0x7fe4a8199b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:19:09.516 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:09.513+0000 7fe4a7400640 1 --2- 172.21.15.118:0/2805784131 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fe4a81015a0 0x7fe4a8199b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:19:09.516 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:09.513+0000 7fe4a7400640 1 -- 172.21.15.118:0/2805784131 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fe4a8102790 msgr2=0x7fe4a819a040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:19:09.516 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:09.513+0000 7fe4a7400640 1 --2- 172.21.15.118:0/2805784131 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fe4a8102790 0x7fe4a819a040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:19:09.516 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:09.513+0000 7fe4a7400640 1 -- 172.21.15.118:0/2805784131 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe4a81a1720 con 0x7fe4a8103150 2024-03-24T00:19:09.516 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:09.513+0000 7fe4a7400640 1 --2- 172.21.15.118:0/2805784131 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe4a8103150 0x7fe4a81a10b0 secure :-1 s=READY pgs=368 cs=0 l=1 rev1=1 crypto rx=0x7fe4a000eaa0 tx=0x7fe4a000ef60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:19:09.516 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:09.513+0000 7fe49f400640 1 -- 172.21.15.118:0/2805784131 <== mon.2 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe4a000cdb0 con 0x7fe4a8103150 2024-03-24T00:19:09.516 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:09.514+0000 7fe49f400640 1 -- 172.21.15.118:0/2805784131 <== mon.2 v2:172.21.15.189:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fe4a0005020 con 0x7fe4a8103150 2024-03-24T00:19:09.516 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:09.514+0000 7fe49f400640 1 -- 172.21.15.118:0/2805784131 <== mon.2 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe4a00116b0 con 0x7fe4a8103150 2024-03-24T00:19:09.516 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:09.514+0000 7fe4a7e00640 1 -- 172.21.15.118:0/2805784131 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe4a81a1a10 con 0x7fe4a8103150 2024-03-24T00:19:09.517 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:09.514+0000 7fe4a7e00640 1 -- 172.21.15.118:0/2805784131 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe4a81a1f10 con 0x7fe4a8103150 2024-03-24T00:19:09.519 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:09.517+0000 7fe49f400640 1 -- 172.21.15.118:0/2805784131 <== mon.2 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 20) v1 ==== 137291+0+0 (secure 0 0 0) 0x7fe4a0005560 con 0x7fe4a8103150 2024-03-24T00:19:09.519 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:09.517+0000 7fe4b0777740 1 Processor -- start 2024-03-24T00:19:09.519 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:09.517+0000 7fe4b0777740 1 -- start start 2024-03-24T00:19:09.519 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:09.517+0000 7fe4b0777740 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55e4f566e9f0 0x55e4f566ee10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:19:09.519 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:09.517+0000 7fe4b0777740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55e4f566fbc0 0x55e4f5670040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:19:09.520 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:09.517+0000 7fe4b0777740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55e4f5670580 0x55e4f5676e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:19:09.520 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:09.517+0000 7fe4b0777740 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e4f55d32c0 con 0x55e4f566e9f0 2024-03-24T00:19:09.520 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:09.517+0000 7fe4b0777740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e4f557bc90 con 0x55e4f566fbc0 2024-03-24T00:19:09.520 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:09.517+0000 7fe4b0777740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e4f55cf360 con 0x55e4f5670580 2024-03-24T00:19:09.520 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:09.518+0000 7fe4a6a00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55e4f566fbc0 0x55e4f5670040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:19:09.520 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:09.518+0000 7fe4a6a00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55e4f566fbc0 0x55e4f5670040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.118:54078/0 (socket says 172.21.15.118:54078) 2024-03-24T00:19:09.520 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:09.518+0000 7fe4a6a00640 1 -- 172.21.15.118:0/2950680267 learned_addr learned my addr 172.21.15.118:0/2950680267 (peer_addr_for_me v2:172.21.15.118:0/0) 2024-03-24T00:19:09.520 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:09.518+0000 7fe49fe00640 1 --2- 172.21.15.118:0/2950680267 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55e4f566e9f0 0x55e4f566ee10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:19:09.520 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:09.518+0000 7fe49fe00640 1 --2- 172.21.15.118:0/2950680267 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55e4f5670580 0x55e4f5676e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:19:09.520 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:09.518+0000 7fe4a6a00640 1 -- 172.21.15.118:0/2950680267 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55e4f5670580 msgr2=0x55e4f5676e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:19:09.520 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:09.518+0000 7fe4a6a00640 1 --2- 172.21.15.118:0/2950680267 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55e4f5670580 0x55e4f5676e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:19:09.520 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:09.518+0000 7fe4a6a00640 1 -- 172.21.15.118:0/2950680267 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55e4f566e9f0 msgr2=0x55e4f566ee10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:19:09.520 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:09.518+0000 7fe4a6a00640 1 --2- 172.21.15.118:0/2950680267 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55e4f566e9f0 0x55e4f566ee10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:19:09.520 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:09.518+0000 7fe4a6a00640 1 -- 172.21.15.118:0/2950680267 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55e4f55a0080 con 0x55e4f566fbc0 2024-03-24T00:19:09.521 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:09.518+0000 7fe49f400640 1 --2- 172.21.15.118:0/2805784131 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7fe48c0a2880 0x7fe48c0a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:19:09.521 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:09.518+0000 7fe4a6a00640 1 --2- 172.21.15.118:0/2950680267 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55e4f566fbc0 0x55e4f5670040 secure :-1 s=READY pgs=369 cs=0 l=1 rev1=1 crypto rx=0x7fe498000c00 tx=0x7fe4980095e0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=222134b55b9281a0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:19:09.521 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:09.519+0000 7fe49f400640 1 -- 172.21.15.118:0/2805784131 <== mon.2 v2:172.21.15.189:3300/0 5 ==== osd_map(90..90 src has 1..90) v4 ==== 8277+0+0 (secure 0 0 0) 0x7fe4a00cec30 con 0x7fe4a8103150 2024-03-24T00:19:09.521 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:09.519+0000 7fe49fe00640 1 --2- 172.21.15.118:0/2805784131 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7fe48c0a2880 0x7fe48c0a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:19:09.521 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:09.519+0000 7fe497e00640 1 -- 172.21.15.118:0/2950680267 <== mon.2 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe498045070 con 0x55e4f566fbc0 2024-03-24T00:19:09.521 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:09.519+0000 7fe497e00640 1 -- 172.21.15.118:0/2950680267 <== mon.2 v2:172.21.15.189:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fe498004040 con 0x55e4f566fbc0 2024-03-24T00:19:09.521 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:09.519+0000 7fe497e00640 1 -- 172.21.15.118:0/2950680267 <== mon.2 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe498004c70 con 0x55e4f566fbc0 2024-03-24T00:19:09.521 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:09.519+0000 7fe49fe00640 1 --2- 172.21.15.118:0/2805784131 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7fe48c0a2880 0x7fe48c0a4d40 secure :-1 s=READY pgs=333 cs=0 l=1 rev1=1 crypto rx=0x7fe490009710 tx=0x7fe490005850 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:19:09.521 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:09.519+0000 7fe4b0777740 1 -- 172.21.15.118:0/2950680267 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55e4f566fbc0 msgr2=0x55e4f5670040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:19:09.521 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:09.519+0000 7fe4b0777740 1 --2- 172.21.15.118:0/2950680267 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55e4f566fbc0 0x55e4f5670040 secure :-1 s=READY pgs=369 cs=0 l=1 rev1=1 crypto rx=0x7fe498000c00 tx=0x7fe4980095e0 comp rx=0 tx=0).stop 2024-03-24T00:19:09.521 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:09.520+0000 7fe4b0777740 1 -- 172.21.15.118:0/2950680267 shutdown_connections 2024-03-24T00:19:09.521 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:09.520+0000 7fe4b0777740 1 --2- 172.21.15.118:0/2950680267 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55e4f5670580 0x55e4f5676e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:19:09.521 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:09.520+0000 7fe4b0777740 1 --2- 172.21.15.118:0/2950680267 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55e4f566fbc0 0x55e4f5670040 unknown :-1 s=CLOSED pgs=369 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:19:09.521 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:09.520+0000 7fe4b0777740 1 --2- 172.21.15.118:0/2950680267 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55e4f566e9f0 0x55e4f566ee10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:19:09.521 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:09.520+0000 7fe4b0777740 1 -- 172.21.15.118:0/2950680267 >> 172.21.15.118:0/2950680267 conn(0x55e4f5669de0 msgr2=0x55e4f566a1e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:19:09.522 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:09.520+0000 7fe4b0777740 1 -- 172.21.15.118:0/2950680267 shutdown_connections 2024-03-24T00:19:09.522 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:09.520+0000 7fe4b0777740 1 -- 172.21.15.118:0/2950680267 wait complete. 2024-03-24T00:19:09.525 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:09.520+0000 7fe4b0777740 1 -- start start 2024-03-24T00:19:09.526 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:09.521+0000 7fe4b0777740 1 Processor -- start 2024-03-24T00:19:09.526 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:09.521+0000 7fe4b0777740 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55e4f566e9f0 0x55e4f5678c00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:19:09.526 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:09.521+0000 7fe4b0777740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55e4f566fbc0 0x55e4f5679140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:19:09.526 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:09.521+0000 7fe4b0777740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55e4f5670580 0x55e4f5680180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:19:09.526 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:09.521+0000 7fe4b0777740 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e4f5677530 con 0x55e4f566e9f0 2024-03-24T00:19:09.526 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:09.521+0000 7fe4b0777740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e4f558f0d0 con 0x55e4f5670580 2024-03-24T00:19:09.526 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:09.521+0000 7fe4b0777740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e4f56776b0 con 0x55e4f566fbc0 2024-03-24T00:19:09.526 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:09.521+0000 7fe4b0777740 10 client.0 _ll_register_callbacks cb 0x55e4f5593e80 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T00:19:09.526 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:09.522+0000 7fe4a7400640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55e4f5670580 0x55e4f5680180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:19:09.527 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:09.522+0000 7fe4a6a00640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55e4f566e9f0 0x55e4f5678c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:19:09.527 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:09.522+0000 7fe4a7400640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55e4f5670580 0x55e4f5680180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.118:54082/0 (socket says 172.21.15.118:54082) 2024-03-24T00:19:09.527 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:09.522+0000 7fe4a7400640 1 -- 172.21.15.118:0/1136884558 learned_addr learned my addr 172.21.15.118:0/1136884558 (peer_addr_for_me v2:172.21.15.118:0/0) 2024-03-24T00:19:09.527 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:09.522+0000 7fe49fe00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55e4f566fbc0 0x55e4f5679140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:19:09.527 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:09.522+0000 7fe4a7400640 1 -- 172.21.15.118:0/1136884558 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55e4f566fbc0 msgr2=0x55e4f5679140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:19:09.527 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:09.522+0000 7fe4a7400640 1 --2- 172.21.15.118:0/1136884558 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55e4f566fbc0 0x55e4f5679140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:19:09.527 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:09.522+0000 7fe4a7400640 1 -- 172.21.15.118:0/1136884558 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55e4f566e9f0 msgr2=0x55e4f5678c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:19:09.527 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:09.522+0000 7fe4a7400640 1 --2- 172.21.15.118:0/1136884558 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55e4f566e9f0 0x55e4f5678c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:19:09.527 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:09.522+0000 7fe4a7400640 1 -- 172.21.15.118:0/1136884558 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55e4f52e73f0 con 0x55e4f5670580 2024-03-24T00:19:09.527 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:09.522+0000 7fe4a6a00640 1 --2- 172.21.15.118:0/1136884558 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55e4f566e9f0 0x55e4f5678c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T00:19:09.527 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:09.522+0000 7fe4a7400640 1 --2- 172.21.15.118:0/1136884558 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55e4f5670580 0x55e4f5680180 secure :-1 s=READY pgs=370 cs=0 l=1 rev1=1 crypto rx=0x7fe4a0097a50 tx=0x7fe4a009b730 comp rx=0 tx=0).ready entity=mon.2 client_cookie=9068db34667d9a59 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:19:09.527 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:09.522+0000 7fe494c00640 10 client.0 ms_handle_connect on v2:172.21.15.189:3300/0 2024-03-24T00:19:09.527 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:09.522+0000 7fe494c00640 1 -- 172.21.15.118:0/1136884558 <== mon.2 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe4a00b4060 con 0x55e4f5670580 2024-03-24T00:19:09.527 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:09.522+0000 7fe4b0777740 1 -- 172.21.15.118:0/1136884558 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fe498004040 con 0x55e4f5670580 2024-03-24T00:19:09.528 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:09.522+0000 7fe494c00640 1 -- 172.21.15.118:0/1136884558 <== mon.2 v2:172.21.15.189:3300/0 2 ==== osd_map(90..90 src has 1..90) v4 ==== 8277+0+0 (secure 0 0 0) 0x7fe4a00b4200 con 0x55e4f5670580 2024-03-24T00:19:09.528 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:09.523+0000 7fe494c00640 1 -- 172.21.15.118:0/1136884558 <== mon.2 v2:172.21.15.189:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) v2 ==== 24+0+0 (secure 0 0 0) 0x7fe4a001f040 con 0x55e4f5670580 2024-03-24T00:19:09.528 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:09.523+0000 7fe4b0777740 10 client.26408 fetch_fsmap learned FSMap version 28 2024-03-24T00:19:09.528 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:09.523+0000 7fe4b0777740 1 -- 172.21.15.118:0/1136884558 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) v3 -- 0x55e4f53b17d0 con 0x55e4f5670580 2024-03-24T00:19:09.528 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:09.523+0000 7fe494c00640 1 -- 172.21.15.118:0/1136884558 <== mon.2 v2:172.21.15.189:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fe498004040 con 0x55e4f5670580 2024-03-24T00:19:09.528 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:09.523+0000 7fe494c00640 1 -- 172.21.15.118:0/1136884558 <== mon.2 v2:172.21.15.189:3300/0 5 ==== fsmap(e 28) v1 ==== 7934+0+0 (secure 0 0 0) 0x7fe4a000f040 con 0x55e4f5670580 2024-03-24T00:19:09.528 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:09.523+0000 7fe4b0777740 10 client.26408 fetch_fsmap finished waiting for FSMap version 28 2024-03-24T00:19:09.528 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:09.523+0000 7fe4b0777740 10 client.26408 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.e 2024-03-24T00:19:09.528 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:09.523+0000 7fe4b0777740 20 client.26408 populate_metadata read hostname 'smithi118' 2024-03-24T00:19:09.528 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:09.523+0000 7fe4b0777740 1 --2- 172.21.15.118:0/1136884558 >> [v2:172.21.15.189:6834/599473205,v1:172.21.15.189:6835/599473205] conn(0x55e4f56845f0 0x55e4f5686a60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:19:09.528 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:09.523+0000 7fe4b0777740 4 client.26408 mds_command: new command op to 24488 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-24T00:19:09.528 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:09.523+0000 7fe4b0777740 1 -- 172.21.15.118:0/1136884558 --> [v2:172.21.15.189:6834/599473205,v1:172.21.15.189:6835/599473205] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55e4f55e0800 con 0x55e4f56845f0 2024-03-24T00:19:09.528 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:09.523+0000 7fe4a6a00640 1 --2- 172.21.15.118:0/1136884558 >> [v2:172.21.15.189:6834/599473205,v1:172.21.15.189:6835/599473205] conn(0x55e4f56845f0 0x55e4f5686a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:19:09.528 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:09.524+0000 7fe4a6a00640 1 --2- 172.21.15.118:0/1136884558 >> [v2:172.21.15.189:6834/599473205,v1:172.21.15.189:6835/599473205] conn(0x55e4f56845f0 0x55e4f5686a60 crc :-1 s=READY pgs=32 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=5c914199168fe86c server_cookie=145833d684061606 in_seq=0 out_seq=0 2024-03-24T00:19:09.528 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:09.524+0000 7fe494c00640 10 client.26408 ms_handle_connect on v2:172.21.15.189:6834/599473205 2024-03-24T00:19:09.529 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:09.525+0000 7fe494c00640 1 -- 172.21.15.118:0/1136884558 <== mds.4 v2:172.21.15.189:6834/599473205 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x55e4f55e0800 con 0x55e4f56845f0 2024-03-24T00:19:09.529 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:09.525+0000 7fe494c00640 10 client.26408 handle_command_reply: tid=0 2024-03-24T00:19:09.529 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:09.525+0000 7fe4b0777740 1 client.26408 shutdown 2024-03-24T00:19:09.529 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:09.525+0000 7fe4b0777740 1 -- 172.21.15.118:0/1136884558 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55e4f5670580 msgr2=0x55e4f5680180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:19:09.529 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:09.525+0000 7fe4b0777740 1 --2- 172.21.15.118:0/1136884558 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55e4f5670580 0x55e4f5680180 secure :-1 s=READY pgs=370 cs=0 l=1 rev1=1 crypto rx=0x7fe4a0097a50 tx=0x7fe4a009b730 comp rx=0 tx=0).stop 2024-03-24T00:19:09.529 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:09.525+0000 7fe4b0777740 1 -- 172.21.15.118:0/1136884558 shutdown_connections 2024-03-24T00:19:09.529 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:09.525+0000 7fe4b0777740 1 --2- 172.21.15.118:0/1136884558 >> [v2:172.21.15.189:6834/599473205,v1:172.21.15.189:6835/599473205] conn(0x55e4f56845f0 0x55e4f5686a60 crc :-1 s=READY pgs=32 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:19:09.529 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:09.525+0000 7fe4b0777740 1 --2- 172.21.15.118:0/1136884558 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55e4f5670580 0x55e4f5680180 unknown :-1 s=CLOSED pgs=370 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:19:09.529 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:09.525+0000 7fe4b0777740 1 --2- 172.21.15.118:0/1136884558 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55e4f566fbc0 0x55e4f5679140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:19:09.529 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:09.525+0000 7fe4b0777740 1 --2- 172.21.15.118:0/1136884558 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55e4f566e9f0 0x55e4f5678c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:19:09.529 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:09.525+0000 7fe494c00640 0 client.26408 ms_handle_reset on v2:172.21.15.189:6834/599473205 2024-03-24T00:19:09.529 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:09.525+0000 7fe4b0777740 1 -- 172.21.15.118:0/1136884558 >> 172.21.15.118:0/1136884558 conn(0x55e4f5669de0 msgr2=0x55e4f5669160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:19:09.530 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:09.525+0000 7fe4b0777740 1 -- 172.21.15.118:0/1136884558 shutdown_connections 2024-03-24T00:19:09.530 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:09.525+0000 7fe4b0777740 1 -- 172.21.15.118:0/1136884558 wait complete. 2024-03-24T00:19:09.530 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:09.525+0000 7fe4b0777740 20 client.26408 trim_cache size 0 max 16384 2024-03-24T00:19:09.539 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:09.538+0000 7fe4b0777740 1 Processor -- start 2024-03-24T00:19:09.539 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:09.538+0000 7fe4b0777740 1 -- start start 2024-03-24T00:19:09.539 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:09.538+0000 7fe4b0777740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55e4f566e9f0 0x55e4f5664ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:19:09.539 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:09.538+0000 7fe4b0777740 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55e4f566fbc0 0x55e4f5680140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:19:09.539 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:09.538+0000 7fe4b0777740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55e4f5670580 0x55e4f5672c60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:19:09.539 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:09.538+0000 7fe4b0777740 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe4a81aaa10 con 0x55e4f566fbc0 2024-03-24T00:19:09.539 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:09.538+0000 7fe4b0777740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe4a81ac390 con 0x55e4f5670580 2024-03-24T00:19:09.539 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:09.538+0000 7fe4b0777740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe4a81ab070 con 0x55e4f566e9f0 2024-03-24T00:19:09.539 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:09.538+0000 7fe4a6a00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55e4f566e9f0 0x55e4f5664ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:19:09.539 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:09.538+0000 7fe4a7400640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55e4f5670580 0x55e4f5672c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:19:09.540 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:09.538+0000 7fe4a6a00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55e4f566e9f0 0x55e4f5664ff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.118:47212/0 (socket says 172.21.15.118:47212) 2024-03-24T00:19:09.540 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:09.539+0000 7fe4a6a00640 1 -- 172.21.15.118:0/1199109525 learned_addr learned my addr 172.21.15.118:0/1199109525 (peer_addr_for_me v2:172.21.15.118:0/0) 2024-03-24T00:19:09.540 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:09.539+0000 7fe49fe00640 1 --2- 172.21.15.118:0/1199109525 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55e4f566fbc0 0x55e4f5680140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:19:09.540 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:09.539+0000 7fe4a6a00640 1 -- 172.21.15.118:0/1199109525 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55e4f5670580 msgr2=0x55e4f5672c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:19:09.540 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:09.539+0000 7fe4a6a00640 1 --2- 172.21.15.118:0/1199109525 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55e4f5670580 0x55e4f5672c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:19:09.540 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:09.539+0000 7fe4a6a00640 1 -- 172.21.15.118:0/1199109525 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55e4f566fbc0 msgr2=0x55e4f5680140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:19:09.540 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:09.539+0000 7fe4a6a00640 1 --2- 172.21.15.118:0/1199109525 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55e4f566fbc0 0x55e4f5680140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:19:09.540 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:09.539+0000 7fe4a6a00640 1 -- 172.21.15.118:0/1199109525 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55e4f566f640 con 0x55e4f566e9f0 2024-03-24T00:19:09.540 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:09.539+0000 7fe4a6a00640 1 --2- 172.21.15.118:0/1199109525 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55e4f566e9f0 0x55e4f5664ff0 secure :-1 s=READY pgs=368 cs=0 l=1 rev1=1 crypto rx=0x7fe498005ec0 tx=0x7fe498005e50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=a34367d82fa24118 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:19:09.540 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:09.539+0000 7fe497400640 1 -- 172.21.15.118:0/1199109525 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe498056070 con 0x55e4f566e9f0 2024-03-24T00:19:09.540 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:09.539+0000 7fe497400640 1 -- 172.21.15.118:0/1199109525 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fe49802cc80 con 0x55e4f566e9f0 2024-03-24T00:19:09.540 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:09.539+0000 7fe497400640 1 -- 172.21.15.118:0/1199109525 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe498045040 con 0x55e4f566e9f0 2024-03-24T00:19:09.544 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:09.540+0000 7fe4b0777740 1 -- 172.21.15.118:0/1199109525 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55e4f566e9f0 msgr2=0x55e4f5664ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:19:09.544 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:09.540+0000 7fe4b0777740 1 --2- 172.21.15.118:0/1199109525 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55e4f566e9f0 0x55e4f5664ff0 secure :-1 s=READY pgs=368 cs=0 l=1 rev1=1 crypto rx=0x7fe498005ec0 tx=0x7fe498005e50 comp rx=0 tx=0).stop 2024-03-24T00:19:09.544 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:09.540+0000 7fe4b0777740 1 -- 172.21.15.118:0/1199109525 shutdown_connections 2024-03-24T00:19:09.544 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:09.540+0000 7fe4b0777740 1 --2- 172.21.15.118:0/1199109525 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55e4f5670580 0x55e4f5672c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:19:09.544 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:09.540+0000 7fe4b0777740 1 --2- 172.21.15.118:0/1199109525 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55e4f566fbc0 0x55e4f5680140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:19:09.545 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:09.540+0000 7fe4b0777740 1 --2- 172.21.15.118:0/1199109525 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55e4f566e9f0 0x55e4f5664ff0 unknown :-1 s=CLOSED pgs=368 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:19:09.545 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:09.540+0000 7fe4b0777740 1 -- 172.21.15.118:0/1199109525 >> 172.21.15.118:0/1199109525 conn(0x55e4f5669de0 msgr2=0x55e4f566d8d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:19:09.545 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:09.540+0000 7fe4b0777740 1 -- 172.21.15.118:0/1199109525 shutdown_connections 2024-03-24T00:19:09.545 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:09.540+0000 7fe4b0777740 1 -- 172.21.15.118:0/1199109525 wait complete. 2024-03-24T00:19:09.545 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:09.541+0000 7fe4b0777740 1 -- start start 2024-03-24T00:19:09.545 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:09.541+0000 7fe4b0777740 1 Processor -- start 2024-03-24T00:19:09.545 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:09.541+0000 7fe4b0777740 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55e4f566e9f0 0x55e4f567f950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:19:09.545 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:09.541+0000 7fe4b0777740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55e4f566fbc0 0x55e4f567fe90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:19:09.545 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:09.541+0000 7fe4b0777740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55e4f5670580 0x55e4f567c3f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:19:09.545 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:09.541+0000 7fe4b0777740 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe4a81abd30 con 0x55e4f566e9f0 2024-03-24T00:19:09.546 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:09.541+0000 7fe4b0777740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe4a81ab6d0 con 0x55e4f566fbc0 2024-03-24T00:19:09.546 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:09.541+0000 7fe4b0777740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe4a81a8bd0 con 0x55e4f5670580 2024-03-24T00:19:09.546 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:09.541+0000 7fe4b0777740 10 client.0 _ll_register_callbacks cb 0x7fe4a0097890 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T00:19:09.546 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:09.541+0000 7fe4a7400640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55e4f5670580 0x55e4f567c3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:19:09.546 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:09.541+0000 7fe49fe00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55e4f566fbc0 0x55e4f567fe90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:19:09.546 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:09.541+0000 7fe4a7400640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55e4f5670580 0x55e4f567c3f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.118:47220/0 (socket says 172.21.15.118:47220) 2024-03-24T00:19:09.546 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:09.541+0000 7fe49fe00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55e4f566fbc0 0x55e4f567fe90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.118:54086/0 (socket says 172.21.15.118:54086) 2024-03-24T00:19:09.546 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:09.541+0000 7fe4a7400640 1 -- 172.21.15.118:0/395803471 learned_addr learned my addr 172.21.15.118:0/395803471 (peer_addr_for_me v2:172.21.15.118:0/0) 2024-03-24T00:19:09.546 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:09.541+0000 7fe4a6a00640 1 --2- 172.21.15.118:0/395803471 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55e4f566e9f0 0x55e4f567f950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:19:09.546 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:09.542+0000 7fe4a7400640 1 -- 172.21.15.118:0/395803471 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55e4f566fbc0 msgr2=0x55e4f567fe90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:19:09.546 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:09.542+0000 7fe4a7400640 1 --2- 172.21.15.118:0/395803471 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55e4f566fbc0 0x55e4f567fe90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:19:09.546 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:09.542+0000 7fe4a7400640 1 -- 172.21.15.118:0/395803471 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55e4f566e9f0 msgr2=0x55e4f567f950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:19:09.546 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:09.542+0000 7fe4a7400640 1 --2- 172.21.15.118:0/395803471 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55e4f566e9f0 0x55e4f567f950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:19:09.547 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:09.542+0000 7fe4a7400640 1 -- 172.21.15.118:0/395803471 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55e4f567ca60 con 0x55e4f5670580 2024-03-24T00:19:09.547 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:09.542+0000 7fe4a7400640 1 --2- 172.21.15.118:0/395803471 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55e4f5670580 0x55e4f567c3f0 secure :-1 s=READY pgs=369 cs=0 l=1 rev1=1 crypto rx=0x7fe4a00972d0 tx=0x7fe4a00b4530 comp rx=0 tx=0).ready entity=mon.1 client_cookie=13f17753d3dcd064 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:19:09.547 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:09.542+0000 7fe494c00640 10 client.0 ms_handle_connect on v2:172.21.15.193:3300/0 2024-03-24T00:19:09.547 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:09.542+0000 7fe494c00640 1 -- 172.21.15.118:0/395803471 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe4a00b4040 con 0x55e4f5670580 2024-03-24T00:19:09.547 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:09.542+0000 7fe494c00640 1 -- 172.21.15.118:0/395803471 <== mon.1 v2:172.21.15.193:3300/0 2 ==== osd_map(90..90 src has 1..90) v4 ==== 8277+0+0 (secure 0 0 0) 0x7fe4a0084d80 con 0x55e4f5670580 2024-03-24T00:19:09.547 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:09.542+0000 7fe4b0777740 1 -- 172.21.15.118:0/395803471 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fe49802cc80 con 0x55e4f5670580 2024-03-24T00:19:09.547 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:09.542+0000 7fe494c00640 1 -- 172.21.15.118:0/395803471 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) v2 ==== 24+0+0 (secure 0 0 0) 0x7fe4a009af50 con 0x55e4f5670580 2024-03-24T00:19:09.547 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:09.542+0000 7fe4b0777740 10 client.26428 fetch_fsmap learned FSMap version 28 2024-03-24T00:19:09.547 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:09.542+0000 7fe4b0777740 1 -- 172.21.15.118:0/395803471 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) v3 -- 0x55e4f56e0b50 con 0x55e4f5670580 2024-03-24T00:19:09.547 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:09.543+0000 7fe494c00640 1 -- 172.21.15.118:0/395803471 <== mon.1 v2:172.21.15.193:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fe49802cc80 con 0x55e4f5670580 2024-03-24T00:19:09.547 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:09.543+0000 7fe494c00640 1 -- 172.21.15.118:0/395803471 <== mon.1 v2:172.21.15.193:3300/0 5 ==== fsmap(e 28) v1 ==== 7934+0+0 (secure 0 0 0) 0x7fe4a0083020 con 0x55e4f5670580 2024-03-24T00:19:09.547 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:09.543+0000 7fe4b0777740 10 client.26428 fetch_fsmap finished waiting for FSMap version 28 2024-03-24T00:19:09.548 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:09.543+0000 7fe4b0777740 10 client.26428 resolve_mds: resolved 1:4 to role '1:4' aka daemon mds.e 2024-03-24T00:19:09.548 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:09.543+0000 7fe4b0777740 20 client.26428 populate_metadata read hostname 'smithi118' 2024-03-24T00:19:09.548 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:09.543+0000 7fe4b0777740 1 --2- 172.21.15.118:0/395803471 >> [v2:172.21.15.189:6834/599473205,v1:172.21.15.189:6835/599473205] conn(0x55e4f56e12c0 0x55e4f56e36e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:19:09.548 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:09.543+0000 7fe4b0777740 4 client.26428 mds_command: new command op to 24488 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-24T00:19:09.548 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:09.543+0000 7fe4b0777740 1 -- 172.21.15.118:0/395803471 --> [v2:172.21.15.189:6834/599473205,v1:172.21.15.189:6835/599473205] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x55e4f55dab50 con 0x55e4f56e12c0 2024-03-24T00:19:09.548 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:09.543+0000 7fe4a6a00640 1 --2- 172.21.15.118:0/395803471 >> [v2:172.21.15.189:6834/599473205,v1:172.21.15.189:6835/599473205] conn(0x55e4f56e12c0 0x55e4f56e36e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:19:09.548 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:09.544+0000 7fe4a6a00640 1 --2- 172.21.15.118:0/395803471 >> [v2:172.21.15.189:6834/599473205,v1:172.21.15.189:6835/599473205] conn(0x55e4f56e12c0 0x55e4f56e36e0 crc :-1 s=READY pgs=33 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.4 client_cookie=9a293562263d3fa6 server_cookie=ffa8bdabbfd9b019 in_seq=0 out_seq=0 2024-03-24T00:19:09.548 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:09.544+0000 7fe494c00640 10 client.26428 ms_handle_connect on v2:172.21.15.189:6834/599473205 2024-03-24T00:19:09.548 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:09.544+0000 7fe494c00640 1 -- 172.21.15.118:0/395803471 <== mds.4 v2:172.21.15.189:6834/599473205 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x55e4f55dab50 con 0x55e4f56e12c0 2024-03-24T00:19:09.548 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:09.544+0000 7fe494c00640 10 client.26428 handle_command_reply: tid=0 2024-03-24T00:19:09.548 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:09.544+0000 7fe4b0777740 1 client.26428 shutdown 2024-03-24T00:19:09.548 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:09.544+0000 7fe4b0777740 1 -- 172.21.15.118:0/395803471 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55e4f5670580 msgr2=0x55e4f567c3f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:19:09.549 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:09.544+0000 7fe4b0777740 1 --2- 172.21.15.118:0/395803471 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55e4f5670580 0x55e4f567c3f0 secure :-1 s=READY pgs=369 cs=0 l=1 rev1=1 crypto rx=0x7fe4a00972d0 tx=0x7fe4a00b4530 comp rx=0 tx=0).stop 2024-03-24T00:19:09.549 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:09.544+0000 7fe4b0777740 1 -- 172.21.15.118:0/395803471 shutdown_connections 2024-03-24T00:19:09.549 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:09.544+0000 7fe4b0777740 1 --2- 172.21.15.118:0/395803471 >> [v2:172.21.15.189:6834/599473205,v1:172.21.15.189:6835/599473205] conn(0x55e4f56e12c0 0x55e4f56e36e0 crc :-1 s=READY pgs=33 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:19:09.549 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:09.544+0000 7fe4b0777740 1 --2- 172.21.15.118:0/395803471 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55e4f5670580 0x55e4f567c3f0 unknown :-1 s=CLOSED pgs=369 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:19:09.549 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:09.544+0000 7fe4b0777740 1 --2- 172.21.15.118:0/395803471 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55e4f566fbc0 0x55e4f567fe90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:19:09.549 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:09.544+0000 7fe494c00640 0 client.26428 ms_handle_reset on v2:172.21.15.189:6834/599473205 2024-03-24T00:19:09.549 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:09.544+0000 7fe4b0777740 1 --2- 172.21.15.118:0/395803471 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55e4f566e9f0 0x55e4f567f950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:19:09.549 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:09.544+0000 7fe4b0777740 1 -- 172.21.15.118:0/395803471 >> 172.21.15.118:0/395803471 conn(0x55e4f5669de0 msgr2=0x55e4f56688d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:19:09.549 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:09.544+0000 7fe4b0777740 1 -- 172.21.15.118:0/395803471 shutdown_connections 2024-03-24T00:19:09.549 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:09.544+0000 7fe4b0777740 1 -- 172.21.15.118:0/395803471 wait complete. 2024-03-24T00:19:09.549 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:09.544+0000 7fe4b0777740 20 client.26428 trim_cache size 0 max 16384 2024-03-24T00:19:09.550 INFO:teuthology.orchestra.run.smithi118.stdout:[] 2024-03-24T00:19:09.550 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:09.545+0000 7fe4a7e00640 1 -- 172.21.15.118:0/2805784131 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7fe48c0a2880 msgr2=0x7fe48c0a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:19:09.550 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:09.545+0000 7fe4a7e00640 1 --2- 172.21.15.118:0/2805784131 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7fe48c0a2880 0x7fe48c0a4d40 secure :-1 s=READY pgs=333 cs=0 l=1 rev1=1 crypto rx=0x7fe490009710 tx=0x7fe490005850 comp rx=0 tx=0).stop 2024-03-24T00:19:09.551 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:09.545+0000 7fe4a7e00640 1 -- 172.21.15.118:0/2805784131 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe4a8103150 msgr2=0x7fe4a81a10b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:19:09.551 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:09.545+0000 7fe4a7e00640 1 --2- 172.21.15.118:0/2805784131 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe4a8103150 0x7fe4a81a10b0 secure :-1 s=READY pgs=368 cs=0 l=1 rev1=1 crypto rx=0x7fe4a000eaa0 tx=0x7fe4a000ef60 comp rx=0 tx=0).stop 2024-03-24T00:19:09.551 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:09.545+0000 7fe4a7e00640 1 -- 172.21.15.118:0/2805784131 shutdown_connections 2024-03-24T00:19:09.551 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:09.545+0000 7fe4a7e00640 1 --2- 172.21.15.118:0/2805784131 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7fe48c0a2880 0x7fe48c0a4d40 unknown :-1 s=CLOSED pgs=333 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:19:09.551 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:09.545+0000 7fe4a7e00640 1 --2- 172.21.15.118:0/2805784131 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe4a8103150 0x7fe4a81a10b0 unknown :-1 s=CLOSED pgs=368 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:19:09.551 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:09.545+0000 7fe4a7e00640 1 --2- 172.21.15.118:0/2805784131 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fe4a8102790 0x7fe4a819a040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:19:09.551 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:09.545+0000 7fe4a7e00640 1 --2- 172.21.15.118:0/2805784131 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fe4a81015a0 0x7fe4a8199b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:19:09.551 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:09.545+0000 7fe4a7e00640 1 -- 172.21.15.118:0/2805784131 >> 172.21.15.118:0/2805784131 conn(0x7fe4a80fcb10 msgr2=0x7fe4a80738d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:19:09.551 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:09.545+0000 7fe4a7e00640 1 -- 172.21.15.118:0/2805784131 shutdown_connections 2024-03-24T00:19:09.551 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:09.545+0000 7fe4a7e00640 1 -- 172.21.15.118:0/2805784131 wait complete. 2024-03-24T00:19:09.560 INFO:teuthology.orchestra.run:Running command with timeout 120 2024-03-24T00:19:09.561 DEBUG:teuthology.orchestra.run.smithi118:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:3 damage ls 2024-03-24T00:19:09.632 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:19:09 smithi118 ceph-mon[31535]: from='client.? 172.21.15.118:0/266970391' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-24T00:19:09.704 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:19:09 smithi189 ceph-mon[36187]: from='client.? 172.21.15.118:0/266970391' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-24T00:19:09.710 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:19:09 smithi193 ceph-mon[36878]: from='client.? 172.21.15.118:0/266970391' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-24T00:19:09.768 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:09.768+0000 7f82fa000640 1 -- 172.21.15.118:0/3451067619 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f82f4104990 msgr2=0x7f82f4104e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:19:09.768 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:09.768+0000 7f82fa000640 1 --2- 172.21.15.118:0/3451067619 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f82f4104990 0x7f82f4104e10 secure :-1 s=READY pgs=370 cs=0 l=1 rev1=1 crypto rx=0x7f82e0009900 tx=0x7f82e002bc30 comp rx=0 tx=0).stop 2024-03-24T00:19:09.768 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:09.768+0000 7f82fa000640 1 -- 172.21.15.118:0/3451067619 shutdown_connections 2024-03-24T00:19:09.768 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:09.768+0000 7f82fa000640 1 --2- 172.21.15.118:0/3451067619 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f82f4105350 0x7f82f410bbe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:19:09.769 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:09.768+0000 7f82fa000640 1 --2- 172.21.15.118:0/3451067619 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f82f4104990 0x7f82f4104e10 unknown :-1 s=CLOSED pgs=370 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:19:09.769 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:09.768+0000 7f82fa000640 1 --2- 172.21.15.118:0/3451067619 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f82f41037a0 0x7f82f4103bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:19:09.769 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:09.768+0000 7f82fa000640 1 -- 172.21.15.118:0/3451067619 >> 172.21.15.118:0/3451067619 conn(0x7f82f40fed10 msgr2=0x7f82f4101150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:19:09.769 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:09.768+0000 7f82fa000640 1 -- 172.21.15.118:0/3451067619 shutdown_connections 2024-03-24T00:19:09.769 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:09.768+0000 7f82fa000640 1 -- 172.21.15.118:0/3451067619 wait complete. 2024-03-24T00:19:09.771 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:09.769+0000 7f82fa000640 1 Processor -- start 2024-03-24T00:19:09.771 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:09.769+0000 7f82fa000640 1 -- start start 2024-03-24T00:19:09.771 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:09.769+0000 7f82fa000640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f82f41037a0 0x7f82f4199a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:19:09.772 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:09.769+0000 7f82fa000640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f82f4104990 0x7f82f4199f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:19:09.772 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:09.769+0000 7f82fa000640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f82f4105350 0x7f82f41a0ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:19:09.772 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:09.769+0000 7f82fa000640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f82f410f420 con 0x7f82f41037a0 2024-03-24T00:19:09.772 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:09.769+0000 7f82fa000640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f82f410f2a0 con 0x7f82f4104990 2024-03-24T00:19:09.772 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:09.769+0000 7f82fa000640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f82f410f5a0 con 0x7f82f4105350 2024-03-24T00:19:09.772 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:09.770+0000 7f82f3e00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f82f4104990 0x7f82f4199f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:19:09.772 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:09.770+0000 7f82f8c00640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f82f41037a0 0x7f82f4199a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:19:09.772 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:09.770+0000 7f82f3e00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f82f4104990 0x7f82f4199f80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.118:54110/0 (socket says 172.21.15.118:54110) 2024-03-24T00:19:09.772 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:09.770+0000 7f82f3e00640 1 -- 172.21.15.118:0/3945488699 learned_addr learned my addr 172.21.15.118:0/3945488699 (peer_addr_for_me v2:172.21.15.118:0/0) 2024-03-24T00:19:09.772 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:09.770+0000 7f82f9600640 1 --2- 172.21.15.118:0/3945488699 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f82f4105350 0x7f82f41a0ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:19:09.772 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:09.770+0000 7f82f3e00640 1 -- 172.21.15.118:0/3945488699 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f82f4105350 msgr2=0x7f82f41a0ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:19:09.773 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:09.770+0000 7f82f3e00640 1 --2- 172.21.15.118:0/3945488699 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f82f4105350 0x7f82f41a0ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:19:09.773 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:09.770+0000 7f82f3e00640 1 -- 172.21.15.118:0/3945488699 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f82f41037a0 msgr2=0x7f82f4199a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:19:09.773 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:09.770+0000 7f82f3e00640 1 --2- 172.21.15.118:0/3945488699 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f82f41037a0 0x7f82f4199a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:19:09.773 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:09.770+0000 7f82f3e00640 1 -- 172.21.15.118:0/3945488699 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f82f41a1660 con 0x7f82f4104990 2024-03-24T00:19:09.773 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:09.770+0000 7f82f3e00640 1 --2- 172.21.15.118:0/3945488699 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f82f4104990 0x7f82f4199f80 secure :-1 s=READY pgs=371 cs=0 l=1 rev1=1 crypto rx=0x7f82e0009230 tx=0x7f82e002c650 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:19:09.773 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:09.771+0000 7f82f1600640 1 -- 172.21.15.118:0/3945488699 <== mon.2 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f82e0041070 con 0x7f82f4104990 2024-03-24T00:19:09.773 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:09.771+0000 7f82fa000640 1 -- 172.21.15.118:0/3945488699 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f82f41a18f0 con 0x7f82f4104990 2024-03-24T00:19:09.773 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:09.771+0000 7f82f1600640 1 -- 172.21.15.118:0/3945488699 <== mon.2 v2:172.21.15.189:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f82e0004450 con 0x7f82f4104990 2024-03-24T00:19:09.773 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:09.771+0000 7f82f1600640 1 -- 172.21.15.118:0/3945488699 <== mon.2 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f82e0003e20 con 0x7f82f4104990 2024-03-24T00:19:09.773 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:09.771+0000 7f82fa000640 1 -- 172.21.15.118:0/3945488699 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f82f41a1de0 con 0x7f82f4104990 2024-03-24T00:19:09.774 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:09.774+0000 7f82fe218740 1 Processor -- start 2024-03-24T00:19:09.774 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:09.775+0000 7f82fe218740 1 -- start start 2024-03-24T00:19:09.777 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:09.775+0000 7f82fe218740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x562f54d8d9f0 0x562f54d8de10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:19:09.777 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:09.775+0000 7f82fe218740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x562f54d8ebc0 0x562f54d8f040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:19:09.778 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:09.775+0000 7f82fe218740 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x562f54d8f580 0x562f54d95e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:19:09.778 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:09.775+0000 7f82fe218740 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x562f54cf22c0 con 0x562f54d8f580 2024-03-24T00:19:09.778 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:09.775+0000 7f82fe218740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x562f54c9ac90 con 0x562f54d8d9f0 2024-03-24T00:19:09.778 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:09.775+0000 7f82fe218740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x562f54cee360 con 0x562f54d8ebc0 2024-03-24T00:19:09.778 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:09.775+0000 7f82f1600640 1 -- 172.21.15.118:0/3945488699 <== mon.2 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 20) v1 ==== 137291+0+0 (secure 0 0 0) 0x7f82e0044070 con 0x7f82f4104990 2024-03-24T00:19:09.778 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:09.775+0000 7f82f9600640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x562f54d8d9f0 0x562f54d8de10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:19:09.778 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:09.775+0000 7f82f9600640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x562f54d8d9f0 0x562f54d8de10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.118:54126/0 (socket says 172.21.15.118:54126) 2024-03-24T00:19:09.778 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:09.775+0000 7f82f9600640 1 -- 172.21.15.118:0/3467384142 learned_addr learned my addr 172.21.15.118:0/3467384142 (peer_addr_for_me v2:172.21.15.118:0/0) 2024-03-24T00:19:09.778 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:09.775+0000 7f82f8c00640 1 --2- 172.21.15.118:0/3467384142 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x562f54d8ebc0 0x562f54d8f040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:19:09.778 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:09.776+0000 7f82f3e00640 1 --2- 172.21.15.118:0/3467384142 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x562f54d8f580 0x562f54d95e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:19:09.779 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:09.776+0000 7f82f9600640 1 -- 172.21.15.118:0/3467384142 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x562f54d8ebc0 msgr2=0x562f54d8f040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:19:09.779 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:09.776+0000 7f82f9600640 1 --2- 172.21.15.118:0/3467384142 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x562f54d8ebc0 0x562f54d8f040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:19:09.779 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:09.776+0000 7f82f9600640 1 -- 172.21.15.118:0/3467384142 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x562f54d8f580 msgr2=0x562f54d95e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:19:09.779 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:09.776+0000 7f82f9600640 1 --2- 172.21.15.118:0/3467384142 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x562f54d8f580 0x562f54d95e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:19:09.779 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:09.776+0000 7f82f9600640 1 -- 172.21.15.118:0/3467384142 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x562f54cbf080 con 0x562f54d8d9f0 2024-03-24T00:19:09.779 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:09.776+0000 7f82f3e00640 1 --2- 172.21.15.118:0/3467384142 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x562f54d8f580 0x562f54d95e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-24T00:19:09.779 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:09.776+0000 7f82f8c00640 1 --2- 172.21.15.118:0/3467384142 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x562f54d8ebc0 0x562f54d8f040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T00:19:09.779 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:09.776+0000 7f82f9600640 1 --2- 172.21.15.118:0/3467384142 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x562f54d8d9f0 0x562f54d8de10 secure :-1 s=READY pgs=372 cs=0 l=1 rev1=1 crypto rx=0x7f82ec009fd0 tx=0x7f82ec0118d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d121214a67a0260a server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:19:09.779 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:09.776+0000 7f82e5c00640 1 -- 172.21.15.118:0/3467384142 <== mon.2 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f82ec004d20 con 0x562f54d8d9f0 2024-03-24T00:19:09.779 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:09.776+0000 7f82e5c00640 1 -- 172.21.15.118:0/3467384142 <== mon.2 v2:172.21.15.189:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f82ec004050 con 0x562f54d8d9f0 2024-03-24T00:19:09.779 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:09.776+0000 7f82e5c00640 1 -- 172.21.15.118:0/3467384142 <== mon.2 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f82ec005280 con 0x562f54d8d9f0 2024-03-24T00:19:09.779 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:09.777+0000 7f82f1600640 1 --2- 172.21.15.118:0/3945488699 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7f82dc0a2800 0x7f82dc0a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:19:09.780 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:09.777+0000 7f82fe218740 1 -- 172.21.15.118:0/3467384142 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x562f54d8d9f0 msgr2=0x562f54d8de10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:19:09.780 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:09.777+0000 7f82fe218740 1 --2- 172.21.15.118:0/3467384142 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x562f54d8d9f0 0x562f54d8de10 secure :-1 s=READY pgs=372 cs=0 l=1 rev1=1 crypto rx=0x7f82ec009fd0 tx=0x7f82ec0118d0 comp rx=0 tx=0).stop 2024-03-24T00:19:09.780 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:09.777+0000 7f82fe218740 1 -- 172.21.15.118:0/3467384142 shutdown_connections 2024-03-24T00:19:09.780 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:09.777+0000 7f82fe218740 1 --2- 172.21.15.118:0/3467384142 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x562f54d8f580 0x562f54d95e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:19:09.780 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:09.777+0000 7f82fe218740 1 --2- 172.21.15.118:0/3467384142 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x562f54d8ebc0 0x562f54d8f040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:19:09.780 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:09.777+0000 7f82fe218740 1 --2- 172.21.15.118:0/3467384142 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x562f54d8d9f0 0x562f54d8de10 unknown :-1 s=CLOSED pgs=372 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:19:09.780 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:09.777+0000 7f82fe218740 1 -- 172.21.15.118:0/3467384142 >> 172.21.15.118:0/3467384142 conn(0x562f54d88de0 msgr2=0x562f54d891e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:19:09.780 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:09.777+0000 7f82f1600640 1 -- 172.21.15.118:0/3945488699 <== mon.2 v2:172.21.15.189:3300/0 5 ==== osd_map(90..90 src has 1..90) v4 ==== 8277+0+0 (secure 0 0 0) 0x7f82e00f0630 con 0x7f82f4104990 2024-03-24T00:19:09.780 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:09.777+0000 7f82f8c00640 1 --2- 172.21.15.118:0/3945488699 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7f82dc0a2800 0x7f82dc0a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:19:09.780 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:09.777+0000 7f82fe218740 1 -- 172.21.15.118:0/3467384142 shutdown_connections 2024-03-24T00:19:09.780 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:09.777+0000 7f82fe218740 1 -- 172.21.15.118:0/3467384142 wait complete. 2024-03-24T00:19:09.781 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:09.778+0000 7f82fe218740 1 -- start start 2024-03-24T00:19:09.781 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:09.778+0000 7f82fe218740 1 Processor -- start 2024-03-24T00:19:09.781 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:09.778+0000 7f82f8c00640 1 --2- 172.21.15.118:0/3945488699 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7f82dc0a2800 0x7f82dc0a4cc0 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 crypto rx=0x562f54d89d50 tx=0x7f82e8005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:19:09.785 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:09.778+0000 7f82fe218740 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x562f54d8d9f0 0x562f54d97c20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:19:09.785 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:09.778+0000 7f82fe218740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x562f54d8ebc0 0x562f54d98160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:19:09.785 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:09.778+0000 7f82fe218740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x562f54d8f580 0x562f54d9f1a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:19:09.785 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:09.778+0000 7f82fe218740 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x562f54d96590 con 0x562f54d8d9f0 2024-03-24T00:19:09.785 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:09.778+0000 7f82fe218740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x562f54cae0d0 con 0x562f54d8f580 2024-03-24T00:19:09.785 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:09.778+0000 7f82fe218740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x562f54d96710 con 0x562f54d8ebc0 2024-03-24T00:19:09.785 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:09.778+0000 7f82fe218740 10 client.0 _ll_register_callbacks cb 0x562f54cb2e80 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T00:19:09.785 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:09.779+0000 7f82f3e00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x562f54d8ebc0 0x562f54d98160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:19:09.786 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:09.779+0000 7f82f3e00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x562f54d8ebc0 0x562f54d98160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.118:47278/0 (socket says 172.21.15.118:47278) 2024-03-24T00:19:09.786 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:09.779+0000 7f82f3e00640 1 -- 172.21.15.118:0/1026927331 learned_addr learned my addr 172.21.15.118:0/1026927331 (peer_addr_for_me v2:172.21.15.118:0/0) 2024-03-24T00:19:09.786 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:09.779+0000 7f82f8c00640 1 --2- 172.21.15.118:0/1026927331 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x562f54d8d9f0 0x562f54d97c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:19:09.786 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:09.779+0000 7f82f9600640 1 --2- 172.21.15.118:0/1026927331 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x562f54d8f580 0x562f54d9f1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:19:09.786 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:09.779+0000 7f82f3e00640 1 -- 172.21.15.118:0/1026927331 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x562f54d8f580 msgr2=0x562f54d9f1a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:19:09.786 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:09.779+0000 7f82f3e00640 1 --2- 172.21.15.118:0/1026927331 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x562f54d8f580 0x562f54d9f1a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:19:09.786 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:09.779+0000 7f82f3e00640 1 -- 172.21.15.118:0/1026927331 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x562f54d8d9f0 msgr2=0x562f54d97c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:19:09.786 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:09.779+0000 7f82f3e00640 1 --2- 172.21.15.118:0/1026927331 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x562f54d8d9f0 0x562f54d97c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:19:09.786 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:09.779+0000 7f82f3e00640 1 -- 172.21.15.118:0/1026927331 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x562f54a063f0 con 0x562f54d8ebc0 2024-03-24T00:19:09.786 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:09.779+0000 7f82f9600640 1 --2- 172.21.15.118:0/1026927331 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x562f54d8f580 0x562f54d9f1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T00:19:09.786 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:09.779+0000 7f82f3e00640 1 --2- 172.21.15.118:0/1026927331 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x562f54d8ebc0 0x562f54d98160 secure :-1 s=READY pgs=371 cs=0 l=1 rev1=1 crypto rx=0x7f82e002cfd0 tx=0x7f82e00b93e0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=4f776be794bdff87 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:19:09.787 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:09.779+0000 7f82da000640 10 client.0 ms_handle_connect on v2:172.21.15.193:3300/0 2024-03-24T00:19:09.787 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:09.779+0000 7f82da000640 1 -- 172.21.15.118:0/1026927331 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f82e004b070 con 0x562f54d8ebc0 2024-03-24T00:19:09.787 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:09.780+0000 7f82fe218740 1 -- 172.21.15.118:0/1026927331 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f82ec004050 con 0x562f54d8ebc0 2024-03-24T00:19:09.787 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:09.780+0000 7f82da000640 1 -- 172.21.15.118:0/1026927331 <== mon.1 v2:172.21.15.193:3300/0 2 ==== osd_map(90..90 src has 1..90) v4 ==== 8277+0+0 (secure 0 0 0) 0x7f82e00c52b0 con 0x562f54d8ebc0 2024-03-24T00:19:09.787 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:09.780+0000 7f82da000640 1 -- 172.21.15.118:0/1026927331 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) v2 ==== 24+0+0 (secure 0 0 0) 0x7f82e0004c60 con 0x562f54d8ebc0 2024-03-24T00:19:09.787 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:09.780+0000 7f82fe218740 10 client.26440 fetch_fsmap learned FSMap version 28 2024-03-24T00:19:09.787 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:09.780+0000 7f82fe218740 1 -- 172.21.15.118:0/1026927331 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) v3 -- 0x562f54ad07d0 con 0x562f54d8ebc0 2024-03-24T00:19:09.787 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:09.780+0000 7f82da000640 1 -- 172.21.15.118:0/1026927331 <== mon.1 v2:172.21.15.193:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f82ec004050 con 0x562f54d8ebc0 2024-03-24T00:19:09.787 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:09.780+0000 7f82da000640 1 -- 172.21.15.118:0/1026927331 <== mon.1 v2:172.21.15.193:3300/0 5 ==== fsmap(e 28) v1 ==== 7934+0+0 (secure 0 0 0) 0x7f82e00b7bc0 con 0x562f54d8ebc0 2024-03-24T00:19:09.787 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:09.780+0000 7f82fe218740 10 client.26440 fetch_fsmap finished waiting for FSMap version 28 2024-03-24T00:19:09.787 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:09.780+0000 7f82fe218740 10 client.26440 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.f 2024-03-24T00:19:09.787 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:09.780+0000 7f82fe218740 20 client.26440 populate_metadata read hostname 'smithi118' 2024-03-24T00:19:09.787 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:09.780+0000 7f82fe218740 1 --2- 172.21.15.118:0/1026927331 >> [v2:172.21.15.193:6834/2055310541,v1:172.21.15.193:6835/2055310541] conn(0x562f54da3650 0x562f54da5a60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:19:09.788 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:09.780+0000 7f82fe218740 4 client.26440 mds_command: new command op to 24499 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-24T00:19:09.788 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:09.780+0000 7f82fe218740 1 -- 172.21.15.118:0/1026927331 --> [v2:172.21.15.193:6834/2055310541,v1:172.21.15.193:6835/2055310541] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x562f54cff800 con 0x562f54da3650 2024-03-24T00:19:09.788 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:09.781+0000 7f82f8c00640 1 --2- 172.21.15.118:0/1026927331 >> [v2:172.21.15.193:6834/2055310541,v1:172.21.15.193:6835/2055310541] conn(0x562f54da3650 0x562f54da5a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:19:09.788 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:09.781+0000 7f82f8c00640 1 --2- 172.21.15.118:0/1026927331 >> [v2:172.21.15.193:6834/2055310541,v1:172.21.15.193:6835/2055310541] conn(0x562f54da3650 0x562f54da5a60 crc :-1 s=READY pgs=35 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=b68c0e656fdce247 server_cookie=e043f6031a8c5e0b in_seq=0 out_seq=0 2024-03-24T00:19:09.788 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:09.781+0000 7f82da000640 10 client.26440 ms_handle_connect on v2:172.21.15.193:6834/2055310541 2024-03-24T00:19:09.788 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:09.782+0000 7f82da000640 1 -- 172.21.15.118:0/1026927331 <== mds.3 v2:172.21.15.193:6834/2055310541 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x562f54cff800 con 0x562f54da3650 2024-03-24T00:19:09.788 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:09.782+0000 7f82da000640 10 client.26440 handle_command_reply: tid=0 2024-03-24T00:19:09.788 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:09.782+0000 7f82fe218740 1 client.26440 shutdown 2024-03-24T00:19:09.788 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:09.782+0000 7f82fe218740 1 -- 172.21.15.118:0/1026927331 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x562f54d8ebc0 msgr2=0x562f54d98160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:19:09.788 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:09.782+0000 7f82fe218740 1 --2- 172.21.15.118:0/1026927331 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x562f54d8ebc0 0x562f54d98160 secure :-1 s=READY pgs=371 cs=0 l=1 rev1=1 crypto rx=0x7f82e002cfd0 tx=0x7f82e00b93e0 comp rx=0 tx=0).stop 2024-03-24T00:19:09.788 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:09.782+0000 7f82fe218740 1 -- 172.21.15.118:0/1026927331 shutdown_connections 2024-03-24T00:19:09.788 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:09.782+0000 7f82fe218740 1 --2- 172.21.15.118:0/1026927331 >> [v2:172.21.15.193:6834/2055310541,v1:172.21.15.193:6835/2055310541] conn(0x562f54da3650 0x562f54da5a60 crc :-1 s=READY pgs=35 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:19:09.789 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:09.782+0000 7f82fe218740 1 --2- 172.21.15.118:0/1026927331 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x562f54d8f580 0x562f54d9f1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:19:09.789 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:09.782+0000 7f82fe218740 1 --2- 172.21.15.118:0/1026927331 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x562f54d8ebc0 0x562f54d98160 unknown :-1 s=CLOSED pgs=371 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:19:09.789 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:09.782+0000 7f82da000640 0 client.26440 ms_handle_reset on v2:172.21.15.193:6834/2055310541 2024-03-24T00:19:09.789 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:09.782+0000 7f82fe218740 1 --2- 172.21.15.118:0/1026927331 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x562f54d8d9f0 0x562f54d97c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:19:09.789 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:09.782+0000 7f82fe218740 1 -- 172.21.15.118:0/1026927331 >> 172.21.15.118:0/1026927331 conn(0x562f54d88de0 msgr2=0x562f54d88160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:19:09.789 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:09.782+0000 7f82fe218740 1 -- 172.21.15.118:0/1026927331 shutdown_connections 2024-03-24T00:19:09.789 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:09.783+0000 7f82fe218740 1 -- 172.21.15.118:0/1026927331 wait complete. 2024-03-24T00:19:09.789 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:09.783+0000 7f82fe218740 20 client.26440 trim_cache size 0 max 16384 2024-03-24T00:19:09.797 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:09.796+0000 7f82fe218740 1 Processor -- start 2024-03-24T00:19:09.797 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:09.796+0000 7f82fe218740 1 -- start start 2024-03-24T00:19:09.797 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:09.796+0000 7f82fe218740 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x562f54d8d9f0 0x562f54d92500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:19:09.797 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:09.796+0000 7f82fe218740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x562f54d8ebc0 0x562f54d9f160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:19:09.797 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:09.796+0000 7f82fe218740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x562f54d8f580 0x562f54daa320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:19:09.797 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:09.796+0000 7f82fe218740 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f82f41aa9b0 con 0x562f54d8d9f0 2024-03-24T00:19:09.797 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:09.796+0000 7f82fe218740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f82f41ac330 con 0x562f54d8f580 2024-03-24T00:19:09.797 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:09.796+0000 7f82fe218740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f82f41ab010 con 0x562f54d8ebc0 2024-03-24T00:19:09.797 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:09.797+0000 7f82f3e00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x562f54d8ebc0 0x562f54d9f160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:19:09.798 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:09.797+0000 7f82f3e00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x562f54d8ebc0 0x562f54d9f160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.118:47290/0 (socket says 172.21.15.118:47290) 2024-03-24T00:19:09.798 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:09.797+0000 7f82f8c00640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x562f54d8d9f0 0x562f54d92500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:19:09.798 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:09.797+0000 7f82f3e00640 1 -- 172.21.15.118:0/22962262 learned_addr learned my addr 172.21.15.118:0/22962262 (peer_addr_for_me v2:172.21.15.118:0/0) 2024-03-24T00:19:09.798 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:09.797+0000 7f82f9600640 1 --2- 172.21.15.118:0/22962262 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x562f54d8f580 0x562f54daa320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:19:09.798 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:09.797+0000 7f82f3e00640 1 -- 172.21.15.118:0/22962262 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x562f54d8f580 msgr2=0x562f54daa320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:19:09.798 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:09.797+0000 7f82f3e00640 1 --2- 172.21.15.118:0/22962262 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x562f54d8f580 0x562f54daa320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:19:09.798 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:09.797+0000 7f82f3e00640 1 -- 172.21.15.118:0/22962262 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x562f54d8d9f0 msgr2=0x562f54d92500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:19:09.798 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:09.797+0000 7f82f3e00640 1 --2- 172.21.15.118:0/22962262 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x562f54d8d9f0 0x562f54d92500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:19:09.798 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:09.797+0000 7f82f3e00640 1 -- 172.21.15.118:0/22962262 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x562f54d956e0 con 0x562f54d8ebc0 2024-03-24T00:19:09.798 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:09.797+0000 7f82f3e00640 1 --2- 172.21.15.118:0/22962262 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x562f54d8ebc0 0x562f54d9f160 secure :-1 s=READY pgs=372 cs=0 l=1 rev1=1 crypto rx=0x7f82e00b9a20 tx=0x7f82e0003920 comp rx=0 tx=0).ready entity=mon.1 client_cookie=7af7f5396e2b1580 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:19:09.798 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:09.797+0000 7f82e5200640 1 -- 172.21.15.118:0/22962262 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f82e004b070 con 0x562f54d8ebc0 2024-03-24T00:19:09.799 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:09.797+0000 7f82e5200640 1 -- 172.21.15.118:0/22962262 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f82e00c5740 con 0x562f54d8ebc0 2024-03-24T00:19:09.799 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:09.797+0000 7f82e5200640 1 -- 172.21.15.118:0/22962262 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f82e00bc700 con 0x562f54d8ebc0 2024-03-24T00:19:09.803 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:09.798+0000 7f82fe218740 1 -- 172.21.15.118:0/22962262 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x562f54d8ebc0 msgr2=0x562f54d9f160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:19:09.804 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:09.798+0000 7f82fe218740 1 --2- 172.21.15.118:0/22962262 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x562f54d8ebc0 0x562f54d9f160 secure :-1 s=READY pgs=372 cs=0 l=1 rev1=1 crypto rx=0x7f82e00b9a20 tx=0x7f82e0003920 comp rx=0 tx=0).stop 2024-03-24T00:19:09.804 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:09.798+0000 7f82fe218740 1 -- 172.21.15.118:0/22962262 shutdown_connections 2024-03-24T00:19:09.804 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:09.798+0000 7f82fe218740 1 --2- 172.21.15.118:0/22962262 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x562f54d8f580 0x562f54daa320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:19:09.804 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:09.798+0000 7f82fe218740 1 --2- 172.21.15.118:0/22962262 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x562f54d8ebc0 0x562f54d9f160 unknown :-1 s=CLOSED pgs=372 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:19:09.804 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:09.798+0000 7f82fe218740 1 --2- 172.21.15.118:0/22962262 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x562f54d8d9f0 0x562f54d92500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:19:09.804 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:09.798+0000 7f82fe218740 1 -- 172.21.15.118:0/22962262 >> 172.21.15.118:0/22962262 conn(0x562f54d88de0 msgr2=0x562f54d91df0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:19:09.804 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:09.798+0000 7f82fe218740 1 -- 172.21.15.118:0/22962262 shutdown_connections 2024-03-24T00:19:09.804 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:09.798+0000 7f82fe218740 1 -- 172.21.15.118:0/22962262 wait complete. 2024-03-24T00:19:09.804 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:09.799+0000 7f82fe218740 1 -- start start 2024-03-24T00:19:09.804 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:09.799+0000 7f82fe218740 1 Processor -- start 2024-03-24T00:19:09.804 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:09.799+0000 7f82fe218740 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x562f54d8d9f0 0x562f54d9eab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:19:09.805 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:09.799+0000 7f82fe218740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x562f54d8ebc0 0x562f54d9eff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:19:09.805 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:09.799+0000 7f82fe218740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x562f54d8f580 0x562f54d9d110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:19:09.805 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:09.799+0000 7f82fe218740 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f82f41abcd0 con 0x562f54d8d9f0 2024-03-24T00:19:09.805 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:09.799+0000 7f82fe218740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f82f41ab670 con 0x562f54d8f580 2024-03-24T00:19:09.805 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:09.799+0000 7f82fe218740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f82f41a8b70 con 0x562f54d8ebc0 2024-03-24T00:19:09.805 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:09.800+0000 7f82fe218740 10 client.0 _ll_register_callbacks cb 0x7f82e0009560 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T00:19:09.805 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:09.800+0000 7f82f8c00640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x562f54d8d9f0 0x562f54d9eab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:19:09.805 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:09.800+0000 7f82f3e00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x562f54d8ebc0 0x562f54d9eff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:19:09.805 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:09.800+0000 7f82f8c00640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x562f54d8d9f0 0x562f54d9eab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.118:3300/0 says I am v2:172.21.15.118:52704/0 (socket says 172.21.15.118:52704) 2024-03-24T00:19:09.805 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:09.800+0000 7f82f8c00640 1 -- 172.21.15.118:0/3958252252 learned_addr learned my addr 172.21.15.118:0/3958252252 (peer_addr_for_me v2:172.21.15.118:0/0) 2024-03-24T00:19:09.805 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:09.800+0000 7f82f3e00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x562f54d8ebc0 0x562f54d9eff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.118:47298/0 (socket says 172.21.15.118:47298) 2024-03-24T00:19:09.806 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:09.800+0000 7f82f9600640 1 --2- 172.21.15.118:0/3958252252 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x562f54d8f580 0x562f54d9d110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:19:09.806 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:09.800+0000 7f82f3e00640 1 -- 172.21.15.118:0/3958252252 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x562f54d8f580 msgr2=0x562f54d9d110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:19:09.806 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:09.800+0000 7f82f3e00640 1 --2- 172.21.15.118:0/3958252252 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x562f54d8f580 0x562f54d9d110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:19:09.806 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:09.800+0000 7f82f3e00640 1 -- 172.21.15.118:0/3958252252 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x562f54d8d9f0 msgr2=0x562f54d9eab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:19:09.806 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:09.800+0000 7f82f3e00640 1 --2- 172.21.15.118:0/3958252252 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x562f54d8d9f0 0x562f54d9eab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:19:09.806 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:09.800+0000 7f82f3e00640 1 -- 172.21.15.118:0/3958252252 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x562f54d9d940 con 0x562f54d8ebc0 2024-03-24T00:19:09.806 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:09.800+0000 7f82f3e00640 1 --2- 172.21.15.118:0/3958252252 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x562f54d8ebc0 0x562f54d9eff0 secure :-1 s=READY pgs=373 cs=0 l=1 rev1=1 crypto rx=0x7f82e0004e00 tx=0x7f82e00b92f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=62e1824ffae048cc server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:19:09.806 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:09.800+0000 7f82da000640 10 client.0 ms_handle_connect on v2:172.21.15.193:3300/0 2024-03-24T00:19:09.806 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:09.801+0000 7f82da000640 1 -- 172.21.15.118:0/3958252252 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f82e00a0060 con 0x562f54d8ebc0 2024-03-24T00:19:09.807 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:09.801+0000 7f82da000640 1 -- 172.21.15.118:0/3958252252 <== mon.1 v2:172.21.15.193:3300/0 2 ==== osd_map(90..90 src has 1..90) v4 ==== 8277+0+0 (secure 0 0 0) 0x7f82e00b8da0 con 0x562f54d8ebc0 2024-03-24T00:19:09.807 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:09.801+0000 7f82fe218740 1 -- 172.21.15.118:0/3958252252 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f82e00c5740 con 0x562f54d8ebc0 2024-03-24T00:19:09.807 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:09.801+0000 7f82da000640 1 -- 172.21.15.118:0/3958252252 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) v2 ==== 24+0+0 (secure 0 0 0) 0x7f82e004b030 con 0x562f54d8ebc0 2024-03-24T00:19:09.807 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:09.801+0000 7f82fe218740 10 client.26446 fetch_fsmap learned FSMap version 28 2024-03-24T00:19:09.807 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:09.801+0000 7f82fe218740 1 -- 172.21.15.118:0/3958252252 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) v3 -- 0x562f54d9d940 con 0x562f54d8ebc0 2024-03-24T00:19:09.807 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:09.801+0000 7f82da000640 1 -- 172.21.15.118:0/3958252252 <== mon.1 v2:172.21.15.193:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f82e00c5740 con 0x562f54d8ebc0 2024-03-24T00:19:09.807 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:09.801+0000 7f82da000640 1 -- 172.21.15.118:0/3958252252 <== mon.1 v2:172.21.15.193:3300/0 5 ==== fsmap(e 28) v1 ==== 7934+0+0 (secure 0 0 0) 0x7f82e00bc030 con 0x562f54d8ebc0 2024-03-24T00:19:09.807 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:09.801+0000 7f82fe218740 10 client.26446 fetch_fsmap finished waiting for FSMap version 28 2024-03-24T00:19:09.807 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:09.801+0000 7f82fe218740 10 client.26446 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.f 2024-03-24T00:19:09.807 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:09.801+0000 7f82fe218740 20 client.26446 populate_metadata read hostname 'smithi118' 2024-03-24T00:19:09.807 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:09.801+0000 7f82fe218740 1 --2- 172.21.15.118:0/3958252252 >> [v2:172.21.15.193:6834/2055310541,v1:172.21.15.193:6835/2055310541] conn(0x562f54e086c0 0x562f54e0aae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:19:09.808 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:09.801+0000 7f82fe218740 4 client.26446 mds_command: new command op to 24499 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-24T00:19:09.808 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:09.801+0000 7f82fe218740 1 -- 172.21.15.118:0/3958252252 --> [v2:172.21.15.193:6834/2055310541,v1:172.21.15.193:6835/2055310541] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x562f54d97a10 con 0x562f54e086c0 2024-03-24T00:19:09.808 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:09.802+0000 7f82f8c00640 1 --2- 172.21.15.118:0/3958252252 >> [v2:172.21.15.193:6834/2055310541,v1:172.21.15.193:6835/2055310541] conn(0x562f54e086c0 0x562f54e0aae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:19:09.808 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:09.802+0000 7f82f8c00640 1 --2- 172.21.15.118:0/3958252252 >> [v2:172.21.15.193:6834/2055310541,v1:172.21.15.193:6835/2055310541] conn(0x562f54e086c0 0x562f54e0aae0 crc :-1 s=READY pgs=36 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.3 client_cookie=9b7538606bbe1866 server_cookie=7c718bb9935459a9 in_seq=0 out_seq=0 2024-03-24T00:19:09.808 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:09.802+0000 7f82da000640 10 client.26446 ms_handle_connect on v2:172.21.15.193:6834/2055310541 2024-03-24T00:19:09.808 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:09.802+0000 7f82da000640 1 -- 172.21.15.118:0/3958252252 <== mds.3 v2:172.21.15.193:6834/2055310541 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x562f54d97a10 con 0x562f54e086c0 2024-03-24T00:19:09.808 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:09.802+0000 7f82da000640 10 client.26446 handle_command_reply: tid=0 2024-03-24T00:19:09.808 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:09.802+0000 7f82fe218740 1 client.26446 shutdown 2024-03-24T00:19:09.808 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:09.802+0000 7f82fe218740 1 -- 172.21.15.118:0/3958252252 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x562f54d8ebc0 msgr2=0x562f54d9eff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:19:09.808 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:09.802+0000 7f82fe218740 1 --2- 172.21.15.118:0/3958252252 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x562f54d8ebc0 0x562f54d9eff0 secure :-1 s=READY pgs=373 cs=0 l=1 rev1=1 crypto rx=0x7f82e0004e00 tx=0x7f82e00b92f0 comp rx=0 tx=0).stop 2024-03-24T00:19:09.808 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:09.802+0000 7f82fe218740 1 -- 172.21.15.118:0/3958252252 shutdown_connections 2024-03-24T00:19:09.808 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:09.802+0000 7f82fe218740 1 --2- 172.21.15.118:0/3958252252 >> [v2:172.21.15.193:6834/2055310541,v1:172.21.15.193:6835/2055310541] conn(0x562f54e086c0 0x562f54e0aae0 crc :-1 s=READY pgs=36 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:19:09.809 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:09.803+0000 7f82fe218740 1 --2- 172.21.15.118:0/3958252252 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x562f54d8f580 0x562f54d9d110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:19:09.809 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:09.803+0000 7f82fe218740 1 --2- 172.21.15.118:0/3958252252 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x562f54d8ebc0 0x562f54d9eff0 unknown :-1 s=CLOSED pgs=373 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:19:09.809 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:09.803+0000 7f82da000640 0 client.26446 ms_handle_reset on v2:172.21.15.193:6834/2055310541 2024-03-24T00:19:09.809 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:09.803+0000 7f82fe218740 1 --2- 172.21.15.118:0/3958252252 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x562f54d8d9f0 0x562f54d9eab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:19:09.809 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:09.803+0000 7f82fe218740 1 -- 172.21.15.118:0/3958252252 >> 172.21.15.118:0/3958252252 conn(0x562f54d88de0 msgr2=0x562f54d92680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:19:09.809 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:09.803+0000 7f82fe218740 1 -- 172.21.15.118:0/3958252252 shutdown_connections 2024-03-24T00:19:09.809 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:09.803+0000 7f82fe218740 1 -- 172.21.15.118:0/3958252252 wait complete. 2024-03-24T00:19:09.809 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:09.803+0000 7f82fe218740 20 client.26446 trim_cache size 0 max 16384 2024-03-24T00:19:09.809 INFO:teuthology.orchestra.run.smithi118.stdout:[] 2024-03-24T00:19:09.810 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:09.804+0000 7f82fa000640 1 -- 172.21.15.118:0/3945488699 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7f82dc0a2800 msgr2=0x7f82dc0a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:19:09.810 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:09.804+0000 7f82fa000640 1 --2- 172.21.15.118:0/3945488699 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7f82dc0a2800 0x7f82dc0a4cc0 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 crypto rx=0x562f54d89d50 tx=0x7f82e8005f90 comp rx=0 tx=0).stop 2024-03-24T00:19:09.810 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:09.804+0000 7f82fa000640 1 -- 172.21.15.118:0/3945488699 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f82f4104990 msgr2=0x7f82f4199f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:19:09.811 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:09.804+0000 7f82fa000640 1 --2- 172.21.15.118:0/3945488699 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f82f4104990 0x7f82f4199f80 secure :-1 s=READY pgs=371 cs=0 l=1 rev1=1 crypto rx=0x7f82e0009230 tx=0x7f82e002c650 comp rx=0 tx=0).stop 2024-03-24T00:19:09.811 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:09.804+0000 7f82fa000640 1 -- 172.21.15.118:0/3945488699 shutdown_connections 2024-03-24T00:19:09.811 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:09.804+0000 7f82fa000640 1 --2- 172.21.15.118:0/3945488699 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7f82dc0a2800 0x7f82dc0a4cc0 unknown :-1 s=CLOSED pgs=334 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:19:09.811 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:09.804+0000 7f82fa000640 1 --2- 172.21.15.118:0/3945488699 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f82f4105350 0x7f82f41a0ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:19:09.811 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:09.804+0000 7f82fa000640 1 --2- 172.21.15.118:0/3945488699 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f82f4104990 0x7f82f4199f80 unknown :-1 s=CLOSED pgs=371 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:19:09.811 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:09.804+0000 7f82fa000640 1 --2- 172.21.15.118:0/3945488699 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f82f41037a0 0x7f82f4199a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:19:09.811 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:09.804+0000 7f82fa000640 1 -- 172.21.15.118:0/3945488699 >> 172.21.15.118:0/3945488699 conn(0x7f82f40fed10 msgr2=0x7f82f4107c90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:19:09.811 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:09.804+0000 7f82fa000640 1 -- 172.21.15.118:0/3945488699 shutdown_connections 2024-03-24T00:19:09.811 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:09.804+0000 7f82fa000640 1 -- 172.21.15.118:0/3945488699 wait complete. 2024-03-24T00:19:09.821 INFO:teuthology.orchestra.run:Running command with timeout 120 2024-03-24T00:19:09.821 DEBUG:teuthology.orchestra.run.smithi118:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:2 damage ls 2024-03-24T00:19:10.065 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:10.065+0000 7f4751600640 1 -- 172.21.15.118:0/914203619 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f474c1020a0 msgr2=0x7f474c104490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:19:10.065 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:10.065+0000 7f4751600640 1 --2- 172.21.15.118:0/914203619 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f474c1020a0 0x7f474c104490 secure :-1 s=READY pgs=373 cs=0 l=1 rev1=1 crypto rx=0x7f473c009900 tx=0x7f473c02bc40 comp rx=0 tx=0).stop 2024-03-24T00:19:10.065 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:10.065+0000 7f4751600640 1 -- 172.21.15.118:0/914203619 shutdown_connections 2024-03-24T00:19:10.065 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:10.065+0000 7f4751600640 1 --2- 172.21.15.118:0/914203619 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f474c107300 0x7f474c1096f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:19:10.066 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:10.065+0000 7f4751600640 1 --2- 172.21.15.118:0/914203619 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f474c1049d0 0x7f474c106dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:19:10.066 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:10.065+0000 7f4751600640 1 --2- 172.21.15.118:0/914203619 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f474c1020a0 0x7f474c104490 unknown :-1 s=CLOSED pgs=373 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:19:10.066 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:10.065+0000 7f4751600640 1 -- 172.21.15.118:0/914203619 >> 172.21.15.118:0/914203619 conn(0x7f474c0fba10 msgr2=0x7f474c0fde30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:19:10.066 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:10.065+0000 7f4751600640 1 -- 172.21.15.118:0/914203619 shutdown_connections 2024-03-24T00:19:10.066 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:10.065+0000 7f4751600640 1 -- 172.21.15.118:0/914203619 wait complete. 2024-03-24T00:19:10.067 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:10.066+0000 7f4751600640 1 Processor -- start 2024-03-24T00:19:10.067 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:10.066+0000 7f4751600640 1 -- start start 2024-03-24T00:19:10.067 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:10.067+0000 7f4751600640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f474c1020a0 0x7f474c199ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:19:10.067 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:10.067+0000 7f4751600640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f474c1049d0 0x7f474c19a010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:19:10.067 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:10.067+0000 7f4751600640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f474c107300 0x7f474c1a1010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:19:10.067 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:10.067+0000 7f4751600640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f474c10d260 con 0x7f474c1020a0 2024-03-24T00:19:10.067 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:10.067+0000 7f4751600640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f474c10d0e0 con 0x7f474c107300 2024-03-24T00:19:10.068 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:10.067+0000 7f4751600640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f474c10d3e0 con 0x7f474c1049d0 2024-03-24T00:19:10.068 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:10.067+0000 7f474b400640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f474c1049d0 0x7f474c19a010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:19:10.068 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:10.067+0000 7f4750c00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f474c107300 0x7f474c1a1010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:19:10.068 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:10.067+0000 7f474be00640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f474c1020a0 0x7f474c199ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:19:10.068 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:10.067+0000 7f474b400640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f474c1049d0 0x7f474c19a010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.118:47318/0 (socket says 172.21.15.118:47318) 2024-03-24T00:19:10.068 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:10.067+0000 7f474b400640 1 -- 172.21.15.118:0/1235124710 learned_addr learned my addr 172.21.15.118:0/1235124710 (peer_addr_for_me v2:172.21.15.118:0/0) 2024-03-24T00:19:10.068 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:10.067+0000 7f4750c00640 1 -- 172.21.15.118:0/1235124710 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f474c1049d0 msgr2=0x7f474c19a010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:19:10.068 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:10.067+0000 7f4750c00640 1 --2- 172.21.15.118:0/1235124710 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f474c1049d0 0x7f474c19a010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:19:10.068 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:10.068+0000 7f4750c00640 1 -- 172.21.15.118:0/1235124710 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f474c1020a0 msgr2=0x7f474c199ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:19:10.068 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:10.068+0000 7f4750c00640 1 --2- 172.21.15.118:0/1235124710 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f474c1020a0 0x7f474c199ad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:19:10.068 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:10.068+0000 7f4750c00640 1 -- 172.21.15.118:0/1235124710 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f474c1a1680 con 0x7f474c107300 2024-03-24T00:19:10.069 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:10.068+0000 7f4750c00640 1 --2- 172.21.15.118:0/1235124710 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f474c107300 0x7f474c1a1010 secure :-1 s=READY pgs=374 cs=0 l=1 rev1=1 crypto rx=0x7f4744009fd0 tx=0x7f474400ef10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:19:10.069 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:10.068+0000 7f4748c00640 1 -- 172.21.15.118:0/1235124710 <== mon.2 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f474400ce20 con 0x7f474c107300 2024-03-24T00:19:10.069 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:10.068+0000 7f4748c00640 1 -- 172.21.15.118:0/1235124710 <== mon.2 v2:172.21.15.189:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f4744005020 con 0x7f474c107300 2024-03-24T00:19:10.069 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:10.068+0000 7f4748c00640 1 -- 172.21.15.118:0/1235124710 <== mon.2 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4744010720 con 0x7f474c107300 2024-03-24T00:19:10.069 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:10.068+0000 7f4751600640 1 -- 172.21.15.118:0/1235124710 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f474c1a1970 con 0x7f474c107300 2024-03-24T00:19:10.069 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:10.068+0000 7f4751600640 1 -- 172.21.15.118:0/1235124710 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f474c1a1e70 con 0x7f474c107300 2024-03-24T00:19:10.071 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:10.071+0000 7f4748c00640 1 -- 172.21.15.118:0/1235124710 <== mon.2 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 20) v1 ==== 137291+0+0 (secure 0 0 0) 0x7f4744005540 con 0x7f474c107300 2024-03-24T00:19:10.071 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:10.072+0000 7f4755a15740 1 Processor -- start 2024-03-24T00:19:10.072 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:10.072+0000 7f4755a15740 1 -- start start 2024-03-24T00:19:10.073 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:10.072+0000 7f4755a15740 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5627d21549f0 0x5627d2154e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:19:10.074 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:10.072+0000 7f4755a15740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5627d2155bc0 0x5627d2156040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:19:10.074 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:10.072+0000 7f4755a15740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5627d2156580 0x5627d215ce10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:19:10.074 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:10.073+0000 7f4755a15740 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x5627d20b92c0 con 0x5627d21549f0 2024-03-24T00:19:10.074 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:10.073+0000 7f4755a15740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x5627d2061c90 con 0x5627d2156580 2024-03-24T00:19:10.074 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:10.073+0000 7f4755a15740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x5627d20b5360 con 0x5627d2155bc0 2024-03-24T00:19:10.074 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:10.073+0000 7f474b400640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5627d21549f0 0x5627d2154e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:19:10.074 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:10.073+0000 7f474b400640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5627d21549f0 0x5627d2154e10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.118:3300/0 says I am v2:172.21.15.118:52728/0 (socket says 172.21.15.118:52728) 2024-03-24T00:19:10.074 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:10.073+0000 7f474b400640 1 -- 172.21.15.118:0/3162141288 learned_addr learned my addr 172.21.15.118:0/3162141288 (peer_addr_for_me v2:172.21.15.118:0/0) 2024-03-24T00:19:10.074 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:10.073+0000 7f474b400640 1 --2- 172.21.15.118:0/3162141288 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5627d2156580 0x5627d215ce10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:19:10.074 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:10.073+0000 7f474be00640 1 --2- 172.21.15.118:0/3162141288 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5627d2155bc0 0x5627d2156040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:19:10.074 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:10.073+0000 7f4748c00640 1 --2- 172.21.15.118:0/1235124710 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7f47300a2880 0x7f47300a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:19:10.074 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:10.073+0000 7f474b400640 1 -- 172.21.15.118:0/3162141288 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5627d2155bc0 msgr2=0x5627d2156040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:19:10.074 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:10.073+0000 7f474b400640 1 --2- 172.21.15.118:0/3162141288 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5627d2155bc0 0x5627d2156040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:19:10.074 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:10.073+0000 7f474b400640 1 -- 172.21.15.118:0/3162141288 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5627d21549f0 msgr2=0x5627d2154e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:19:10.075 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:10.073+0000 7f474b400640 1 --2- 172.21.15.118:0/3162141288 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5627d21549f0 0x5627d2154e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:19:10.075 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:10.073+0000 7f474b400640 1 -- 172.21.15.118:0/3162141288 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5627d2086080 con 0x5627d2156580 2024-03-24T00:19:10.075 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:10.073+0000 7f4748c00640 1 -- 172.21.15.118:0/1235124710 <== mon.2 v2:172.21.15.189:3300/0 5 ==== osd_map(90..90 src has 1..90) v4 ==== 8277+0+0 (secure 0 0 0) 0x7f47440cf080 con 0x7f474c107300 2024-03-24T00:19:10.075 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:10.074+0000 7f474b400640 1 --2- 172.21.15.118:0/3162141288 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5627d2156580 0x5627d215ce10 secure :-1 s=READY pgs=375 cs=0 l=1 rev1=1 crypto rx=0x7f473800bfd0 tx=0x7f4738004730 comp rx=0 tx=0).ready entity=mon.2 client_cookie=31edb26811018093 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:19:10.075 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:10.074+0000 7f4741200640 1 -- 172.21.15.118:0/3162141288 <== mon.2 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f4738016030 con 0x5627d2156580 2024-03-24T00:19:10.075 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:10.074+0000 7f4741200640 1 -- 172.21.15.118:0/3162141288 <== mon.2 v2:172.21.15.189:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f473800dc20 con 0x5627d2156580 2024-03-24T00:19:10.075 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:10.074+0000 7f4741200640 1 -- 172.21.15.118:0/3162141288 <== mon.2 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f473800f940 con 0x5627d2156580 2024-03-24T00:19:10.076 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:10.075+0000 7f4755a15740 1 -- 172.21.15.118:0/3162141288 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5627d2156580 msgr2=0x5627d215ce10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:19:10.076 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:10.075+0000 7f4755a15740 1 --2- 172.21.15.118:0/3162141288 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5627d2156580 0x5627d215ce10 secure :-1 s=READY pgs=375 cs=0 l=1 rev1=1 crypto rx=0x7f473800bfd0 tx=0x7f4738004730 comp rx=0 tx=0).stop 2024-03-24T00:19:10.076 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:10.075+0000 7f4755a15740 1 -- 172.21.15.118:0/3162141288 shutdown_connections 2024-03-24T00:19:10.076 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:10.075+0000 7f4755a15740 1 --2- 172.21.15.118:0/3162141288 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5627d2156580 0x5627d215ce10 unknown :-1 s=CLOSED pgs=375 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:19:10.076 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:10.075+0000 7f4755a15740 1 --2- 172.21.15.118:0/3162141288 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5627d2155bc0 0x5627d2156040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:19:10.076 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:10.075+0000 7f4755a15740 1 --2- 172.21.15.118:0/3162141288 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5627d21549f0 0x5627d2154e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:19:10.076 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:10.075+0000 7f4755a15740 1 -- 172.21.15.118:0/3162141288 >> 172.21.15.118:0/3162141288 conn(0x5627d214fde0 msgr2=0x5627d21501e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:19:10.076 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:10.075+0000 7f4755a15740 1 -- 172.21.15.118:0/3162141288 shutdown_connections 2024-03-24T00:19:10.077 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:10.075+0000 7f4755a15740 1 -- 172.21.15.118:0/3162141288 wait complete. 2024-03-24T00:19:10.077 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:10.076+0000 7f4755a15740 1 -- start start 2024-03-24T00:19:10.077 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:10.076+0000 7f4755a15740 1 Processor -- start 2024-03-24T00:19:10.077 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:10.076+0000 7f4750c00640 1 --2- 172.21.15.118:0/1235124710 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7f47300a2880 0x7f47300a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:19:10.079 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:10.079+0000 7f4750c00640 1 --2- 172.21.15.118:0/1235124710 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7f47300a2880 0x7f47300a4d40 secure :-1 s=READY pgs=335 cs=0 l=1 rev1=1 crypto rx=0x7f474409b690 tx=0x7f4744005f70 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:19:10.080 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:10.080+0000 7f4755a15740 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5627d21549f0 0x5627d215ebd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:19:10.080 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:10.080+0000 7f4755a15740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5627d2155bc0 0x5627d215f110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:19:10.080 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:10.080+0000 7f4755a15740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5627d2156580 0x5627d2166150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:19:10.080 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:10.080+0000 7f4755a15740 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x5627d215d530 con 0x5627d21549f0 2024-03-24T00:19:10.080 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:10.080+0000 7f4755a15740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x5627d20750d0 con 0x5627d2156580 2024-03-24T00:19:10.080 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:10.080+0000 7f4755a15740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x5627d215d6b0 con 0x5627d2155bc0 2024-03-24T00:19:10.080 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:10.080+0000 7f4755a15740 10 client.0 _ll_register_callbacks cb 0x5627d2079e80 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T00:19:10.080 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:10.080+0000 7f474b400640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5627d21549f0 0x5627d215ebd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:19:10.080 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:10.080+0000 7f474b400640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5627d21549f0 0x5627d215ebd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.118:3300/0 says I am v2:172.21.15.118:52740/0 (socket says 172.21.15.118:52740) 2024-03-24T00:19:10.081 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:10.080+0000 7f474b400640 1 -- 172.21.15.118:0/1978931346 learned_addr learned my addr 172.21.15.118:0/1978931346 (peer_addr_for_me v2:172.21.15.118:0/0) 2024-03-24T00:19:10.081 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:10.080+0000 7f474be00640 1 --2- 172.21.15.118:0/1978931346 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5627d2155bc0 0x5627d215f110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:19:10.081 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:10.081+0000 7f474b400640 1 --2- 172.21.15.118:0/1978931346 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5627d2156580 0x5627d2166150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:19:10.082 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:10.081+0000 7f474be00640 1 -- 172.21.15.118:0/1978931346 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5627d2156580 msgr2=0x5627d2166150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:19:10.082 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:10.081+0000 7f474be00640 1 --2- 172.21.15.118:0/1978931346 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5627d2156580 0x5627d2166150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:19:10.082 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:10.081+0000 7f474be00640 1 -- 172.21.15.118:0/1978931346 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5627d21549f0 msgr2=0x5627d215ebd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:19:10.082 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:10.081+0000 7f474be00640 1 --2- 172.21.15.118:0/1978931346 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5627d21549f0 0x5627d215ebd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:19:10.083 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:10.081+0000 7f474be00640 1 -- 172.21.15.118:0/1978931346 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5627d1dcd3f0 con 0x5627d2155bc0 2024-03-24T00:19:10.083 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:10.081+0000 7f474b400640 1 --2- 172.21.15.118:0/1978931346 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5627d2156580 0x5627d2166150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T00:19:10.083 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:10.081+0000 7f474be00640 1 --2- 172.21.15.118:0/1978931346 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5627d2155bc0 0x5627d215f110 secure :-1 s=READY pgs=374 cs=0 l=1 rev1=1 crypto rx=0x7f473c036040 tx=0x7f473c004270 comp rx=0 tx=0).ready entity=mon.1 client_cookie=6b8f47796901997 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:19:10.083 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:10.081+0000 7f4735600640 10 client.0 ms_handle_connect on v2:172.21.15.193:3300/0 2024-03-24T00:19:10.083 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:10.081+0000 7f4735600640 1 -- 172.21.15.118:0/1978931346 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f473c005d50 con 0x5627d2155bc0 2024-03-24T00:19:10.083 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:10.082+0000 7f4735600640 1 -- 172.21.15.118:0/1978931346 <== mon.1 v2:172.21.15.193:3300/0 2 ==== osd_map(90..90 src has 1..90) v4 ==== 8277+0+0 (secure 0 0 0) 0x7f473c0046c0 con 0x5627d2155bc0 2024-03-24T00:19:10.083 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:10.082+0000 7f4755a15740 1 -- 172.21.15.118:0/1978931346 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f473800dc20 con 0x5627d2155bc0 2024-03-24T00:19:10.083 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:10.082+0000 7f4735600640 1 -- 172.21.15.118:0/1978931346 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) v2 ==== 24+0+0 (secure 0 0 0) 0x7f473c0048c0 con 0x5627d2155bc0 2024-03-24T00:19:10.083 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:10.082+0000 7f4755a15740 10 client.26458 fetch_fsmap learned FSMap version 28 2024-03-24T00:19:10.084 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:10.082+0000 7f4755a15740 1 -- 172.21.15.118:0/1978931346 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) v3 -- 0x5627d1e977d0 con 0x5627d2155bc0 2024-03-24T00:19:10.084 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:10.082+0000 7f4735600640 1 -- 172.21.15.118:0/1978931346 <== mon.1 v2:172.21.15.193:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f473800dc20 con 0x5627d2155bc0 2024-03-24T00:19:10.084 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:10.082+0000 7f4735600640 1 -- 172.21.15.118:0/1978931346 <== mon.1 v2:172.21.15.193:3300/0 5 ==== fsmap(e 28) v1 ==== 7934+0+0 (secure 0 0 0) 0x7f473c0035f0 con 0x5627d2155bc0 2024-03-24T00:19:10.084 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:10.083+0000 7f4755a15740 10 client.26458 fetch_fsmap finished waiting for FSMap version 28 2024-03-24T00:19:10.084 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:10.083+0000 7f4755a15740 10 client.26458 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.c 2024-03-24T00:19:10.084 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:10.083+0000 7f4755a15740 20 client.26458 populate_metadata read hostname 'smithi118' 2024-03-24T00:19:10.084 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:10.083+0000 7f4755a15740 1 --2- 172.21.15.118:0/1978931346 >> [v2:172.21.15.193:6836/2921520053,v1:172.21.15.193:6837/2921520053] conn(0x5627d216a5a0 0x5627d216c9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:19:10.084 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:10.083+0000 7f4755a15740 4 client.26458 mds_command: new command op to 24511 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-24T00:19:10.084 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:10.083+0000 7f4755a15740 1 -- 172.21.15.118:0/1978931346 --> [v2:172.21.15.193:6836/2921520053,v1:172.21.15.193:6837/2921520053] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x5627d20c6800 con 0x5627d216a5a0 2024-03-24T00:19:10.084 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:10.083+0000 7f474b400640 1 --2- 172.21.15.118:0/1978931346 >> [v2:172.21.15.193:6836/2921520053,v1:172.21.15.193:6837/2921520053] conn(0x5627d216a5a0 0x5627d216c9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:19:10.085 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:10.083+0000 7f474b400640 1 --2- 172.21.15.118:0/1978931346 >> [v2:172.21.15.193:6836/2921520053,v1:172.21.15.193:6837/2921520053] conn(0x5627d216a5a0 0x5627d216c9d0 crc :-1 s=READY pgs=29 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=e105778357c04dd1 server_cookie=7c8f98e37cc06446 in_seq=0 out_seq=0 2024-03-24T00:19:10.086 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:10.083+0000 7f4735600640 10 client.26458 ms_handle_connect on v2:172.21.15.193:6836/2921520053 2024-03-24T00:19:10.086 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:10.084+0000 7f4735600640 1 -- 172.21.15.118:0/1978931346 <== mds.2 v2:172.21.15.193:6836/2921520053 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x5627d20c6800 con 0x5627d216a5a0 2024-03-24T00:19:10.086 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:10.084+0000 7f4735600640 10 client.26458 handle_command_reply: tid=0 2024-03-24T00:19:10.086 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:10.084+0000 7f4755a15740 1 client.26458 shutdown 2024-03-24T00:19:10.086 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:10.084+0000 7f4755a15740 1 -- 172.21.15.118:0/1978931346 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5627d2155bc0 msgr2=0x5627d215f110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:19:10.086 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:10.084+0000 7f4755a15740 1 --2- 172.21.15.118:0/1978931346 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5627d2155bc0 0x5627d215f110 secure :-1 s=READY pgs=374 cs=0 l=1 rev1=1 crypto rx=0x7f473c036040 tx=0x7f473c004270 comp rx=0 tx=0).stop 2024-03-24T00:19:10.086 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:10.085+0000 7f4755a15740 1 -- 172.21.15.118:0/1978931346 shutdown_connections 2024-03-24T00:19:10.086 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:10.085+0000 7f4755a15740 1 --2- 172.21.15.118:0/1978931346 >> [v2:172.21.15.193:6836/2921520053,v1:172.21.15.193:6837/2921520053] conn(0x5627d216a5a0 0x5627d216c9d0 crc :-1 s=READY pgs=29 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:19:10.086 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:10.085+0000 7f4755a15740 1 --2- 172.21.15.118:0/1978931346 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5627d2156580 0x5627d2166150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:19:10.086 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:10.085+0000 7f4755a15740 1 --2- 172.21.15.118:0/1978931346 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5627d2155bc0 0x5627d215f110 unknown :-1 s=CLOSED pgs=374 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:19:10.086 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:10.085+0000 7f4755a15740 1 --2- 172.21.15.118:0/1978931346 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5627d21549f0 0x5627d215ebd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:19:10.086 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:10.085+0000 7f4735600640 0 client.26458 ms_handle_reset on v2:172.21.15.193:6836/2921520053 2024-03-24T00:19:10.086 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:10.085+0000 7f4755a15740 1 -- 172.21.15.118:0/1978931346 >> 172.21.15.118:0/1978931346 conn(0x5627d214fde0 msgr2=0x5627d214f160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:19:10.087 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:10.085+0000 7f4755a15740 1 -- 172.21.15.118:0/1978931346 shutdown_connections 2024-03-24T00:19:10.087 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:10.085+0000 7f4755a15740 1 -- 172.21.15.118:0/1978931346 wait complete. 2024-03-24T00:19:10.087 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:10.085+0000 7f4755a15740 20 client.26458 trim_cache size 0 max 16384 2024-03-24T00:19:10.099 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:10.099+0000 7f4755a15740 1 Processor -- start 2024-03-24T00:19:10.100 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:10.100+0000 7f4755a15740 1 -- start start 2024-03-24T00:19:10.100 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:10.100+0000 7f4755a15740 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5627d21549f0 0x5627d2158df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:19:10.100 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:10.100+0000 7f4755a15740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5627d2155bc0 0x5627d2166110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:19:10.100 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:10.100+0000 7f4755a15740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5627d2156580 0x5627d2159330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:19:10.100 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:10.100+0000 7f4755a15740 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f474c1aaa40 con 0x5627d21549f0 2024-03-24T00:19:10.100 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:10.100+0000 7f4755a15740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f474c1ac3c0 con 0x5627d2155bc0 2024-03-24T00:19:10.100 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:10.100+0000 7f4755a15740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f474c1ab0a0 con 0x5627d2156580 2024-03-24T00:19:10.102 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:10.100+0000 7f474be00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5627d2155bc0 0x5627d2166110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:19:10.103 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:10.100+0000 7f474be00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5627d2155bc0 0x5627d2166110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.118:54212/0 (socket says 172.21.15.118:54212) 2024-03-24T00:19:10.103 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:10.100+0000 7f474be00640 1 -- 172.21.15.118:0/3320552191 learned_addr learned my addr 172.21.15.118:0/3320552191 (peer_addr_for_me v2:172.21.15.118:0/0) 2024-03-24T00:19:10.103 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:10.100+0000 7f474b400640 1 --2- 172.21.15.118:0/3320552191 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5627d21549f0 0x5627d2158df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:19:10.103 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:10.101+0000 7f474be00640 1 -- 172.21.15.118:0/3320552191 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5627d2156580 msgr2=0x5627d2159330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:19:10.103 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:10.101+0000 7f474be00640 1 --2- 172.21.15.118:0/3320552191 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5627d2156580 0x5627d2159330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:19:10.103 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:10.101+0000 7f474be00640 1 -- 172.21.15.118:0/3320552191 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5627d21549f0 msgr2=0x5627d2158df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:19:10.103 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:10.101+0000 7f474be00640 1 --2- 172.21.15.118:0/3320552191 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5627d21549f0 0x5627d2158df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:19:10.103 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:10.101+0000 7f474be00640 1 -- 172.21.15.118:0/3320552191 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5627d2155640 con 0x5627d2155bc0 2024-03-24T00:19:10.103 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:10.101+0000 7f474be00640 1 --2- 172.21.15.118:0/3320552191 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5627d2155bc0 0x5627d2166110 secure :-1 s=READY pgs=376 cs=0 l=1 rev1=1 crypto rx=0x7f473c02c7c0 tx=0x7f473c001e70 comp rx=0 tx=0).ready entity=mon.2 client_cookie=5cf5ff6040cb9959 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:19:10.103 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:10.101+0000 7f4737e00640 1 -- 172.21.15.118:0/3320552191 <== mon.2 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f473c005d50 con 0x5627d2155bc0 2024-03-24T00:19:10.103 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:10.101+0000 7f4737e00640 1 -- 172.21.15.118:0/3320552191 <== mon.2 v2:172.21.15.189:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f473c02cd30 con 0x5627d2155bc0 2024-03-24T00:19:10.103 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:10.101+0000 7f4737e00640 1 -- 172.21.15.118:0/3320552191 <== mon.2 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f473c004a60 con 0x5627d2155bc0 2024-03-24T00:19:10.104 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:10.102+0000 7f4755a15740 1 -- 172.21.15.118:0/3320552191 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5627d2155bc0 msgr2=0x5627d2166110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:19:10.104 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:10.102+0000 7f4755a15740 1 --2- 172.21.15.118:0/3320552191 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5627d2155bc0 0x5627d2166110 secure :-1 s=READY pgs=376 cs=0 l=1 rev1=1 crypto rx=0x7f473c02c7c0 tx=0x7f473c001e70 comp rx=0 tx=0).stop 2024-03-24T00:19:10.104 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:10.102+0000 7f4755a15740 1 -- 172.21.15.118:0/3320552191 shutdown_connections 2024-03-24T00:19:10.104 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:10.102+0000 7f4755a15740 1 --2- 172.21.15.118:0/3320552191 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5627d2156580 0x5627d2159330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:19:10.104 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:10.102+0000 7f4755a15740 1 --2- 172.21.15.118:0/3320552191 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5627d2155bc0 0x5627d2166110 unknown :-1 s=CLOSED pgs=376 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:19:10.104 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:10.102+0000 7f4755a15740 1 --2- 172.21.15.118:0/3320552191 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5627d21549f0 0x5627d2158df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:19:10.105 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:10.102+0000 7f4755a15740 1 -- 172.21.15.118:0/3320552191 >> 172.21.15.118:0/3320552191 conn(0x5627d214fde0 msgr2=0x5627d21538d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:19:10.105 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:10.102+0000 7f4755a15740 1 -- 172.21.15.118:0/3320552191 shutdown_connections 2024-03-24T00:19:10.105 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:10.102+0000 7f4755a15740 1 -- 172.21.15.118:0/3320552191 wait complete. 2024-03-24T00:19:10.105 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:10.102+0000 7f4755a15740 1 -- start start 2024-03-24T00:19:10.105 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:10.103+0000 7f4755a15740 1 Processor -- start 2024-03-24T00:19:10.108 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:10.103+0000 7f4755a15740 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5627d21549f0 0x5627d2165c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:19:10.108 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:10.103+0000 7f4755a15740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5627d2155bc0 0x5627d2166150 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:19:10.108 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:10.103+0000 7f4755a15740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5627d2156580 0x5627d2164340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:19:10.108 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:10.103+0000 7f4755a15740 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f474c1abd60 con 0x5627d21549f0 2024-03-24T00:19:10.109 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:10.103+0000 7f4755a15740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f474c1ab700 con 0x5627d2155bc0 2024-03-24T00:19:10.109 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:10.103+0000 7f4755a15740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f474c1a8c00 con 0x5627d2156580 2024-03-24T00:19:10.109 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:10.103+0000 7f4755a15740 10 client.0 _ll_register_callbacks cb 0x7f473c009780 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T00:19:10.109 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:10.104+0000 7f474b400640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5627d21549f0 0x5627d2165c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:19:10.109 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:10.104+0000 7f474b400640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5627d21549f0 0x5627d2165c10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.118:3300/0 says I am v2:172.21.15.118:52764/0 (socket says 172.21.15.118:52764) 2024-03-24T00:19:10.109 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:10.104+0000 7f474b400640 1 -- 172.21.15.118:0/3987321497 learned_addr learned my addr 172.21.15.118:0/3987321497 (peer_addr_for_me v2:172.21.15.118:0/0) 2024-03-24T00:19:10.109 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:10.104+0000 7f474b400640 1 --2- 172.21.15.118:0/3987321497 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5627d2156580 0x5627d2164340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:19:10.109 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:10.104+0000 7f474be00640 1 --2- 172.21.15.118:0/3987321497 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5627d2155bc0 0x5627d2166150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:19:10.109 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:10.104+0000 7f474b400640 1 -- 172.21.15.118:0/3987321497 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5627d2155bc0 msgr2=0x5627d2166150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:19:10.109 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:10.104+0000 7f474b400640 1 --2- 172.21.15.118:0/3987321497 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5627d2155bc0 0x5627d2166150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:19:10.109 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:10.105+0000 7f474b400640 1 -- 172.21.15.118:0/3987321497 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5627d21549f0 msgr2=0x5627d2165c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:19:10.109 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:10.105+0000 7f474b400640 1 --2- 172.21.15.118:0/3987321497 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5627d21549f0 0x5627d2165c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:19:10.109 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:10.105+0000 7f474b400640 1 -- 172.21.15.118:0/3987321497 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5627d2164b70 con 0x5627d2156580 2024-03-24T00:19:10.109 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:10.105+0000 7f474be00640 1 --2- 172.21.15.118:0/3987321497 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5627d2155bc0 0x5627d2166150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T00:19:10.110 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:10.105+0000 7f474b400640 1 --2- 172.21.15.118:0/3987321497 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5627d2156580 0x5627d2164340 secure :-1 s=READY pgs=375 cs=0 l=1 rev1=1 crypto rx=0x7f4738012800 tx=0x7f473c02cf90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=4fcd07fe7c5ce385 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:19:10.110 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:10.105+0000 7f4735600640 10 client.0 ms_handle_connect on v2:172.21.15.193:3300/0 2024-03-24T00:19:10.110 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:10.105+0000 7f4735600640 1 -- 172.21.15.118:0/3987321497 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f473802b040 con 0x5627d2156580 2024-03-24T00:19:10.110 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:10.105+0000 7f4755a15740 1 -- 172.21.15.118:0/3987321497 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x5627d20c0b50 con 0x5627d2156580 2024-03-24T00:19:10.110 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:10.105+0000 7f4735600640 1 -- 172.21.15.118:0/3987321497 <== mon.1 v2:172.21.15.193:3300/0 2 ==== osd_map(90..90 src has 1..90) v4 ==== 8277+0+0 (secure 0 0 0) 0x7f473800d710 con 0x5627d2156580 2024-03-24T00:19:10.110 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:10.105+0000 7f4735600640 1 -- 172.21.15.118:0/3987321497 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) v2 ==== 24+0+0 (secure 0 0 0) 0x7f4738016040 con 0x5627d2156580 2024-03-24T00:19:10.110 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:10.105+0000 7f4755a15740 10 client.26461 fetch_fsmap learned FSMap version 28 2024-03-24T00:19:10.110 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:10.105+0000 7f4755a15740 1 -- 172.21.15.118:0/3987321497 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) v3 -- 0x5627d21cf2d0 con 0x5627d2156580 2024-03-24T00:19:10.110 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:10.106+0000 7f4735600640 1 -- 172.21.15.118:0/3987321497 <== mon.1 v2:172.21.15.193:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x5627d20c0b50 con 0x5627d2156580 2024-03-24T00:19:10.110 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:10.106+0000 7f4735600640 1 -- 172.21.15.118:0/3987321497 <== mon.1 v2:172.21.15.193:3300/0 5 ==== fsmap(e 28) v1 ==== 7934+0+0 (secure 0 0 0) 0x7f473800fc20 con 0x5627d2156580 2024-03-24T00:19:10.111 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:10.106+0000 7f4755a15740 10 client.26461 fetch_fsmap finished waiting for FSMap version 28 2024-03-24T00:19:10.111 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:10.106+0000 7f4755a15740 10 client.26461 resolve_mds: resolved 1:2 to role '1:2' aka daemon mds.c 2024-03-24T00:19:10.111 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:10.106+0000 7f4755a15740 20 client.26461 populate_metadata read hostname 'smithi118' 2024-03-24T00:19:10.111 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:10.106+0000 7f4755a15740 1 --2- 172.21.15.118:0/3987321497 >> [v2:172.21.15.193:6836/2921520053,v1:172.21.15.193:6837/2921520053] conn(0x5627d21cfa20 0x5627d21d1e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:19:10.111 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:10.106+0000 7f4755a15740 4 client.26461 mds_command: new command op to 24511 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-24T00:19:10.111 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:10.106+0000 7f4755a15740 1 -- 172.21.15.118:0/3987321497 --> [v2:172.21.15.193:6836/2921520053,v1:172.21.15.193:6837/2921520053] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x5627d203e500 con 0x5627d21cfa20 2024-03-24T00:19:10.111 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:10.106+0000 7f474be00640 1 --2- 172.21.15.118:0/3987321497 >> [v2:172.21.15.193:6836/2921520053,v1:172.21.15.193:6837/2921520053] conn(0x5627d21cfa20 0x5627d21d1e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:19:10.111 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:10.107+0000 7f474be00640 1 --2- 172.21.15.118:0/3987321497 >> [v2:172.21.15.193:6836/2921520053,v1:172.21.15.193:6837/2921520053] conn(0x5627d21cfa20 0x5627d21d1e40 crc :-1 s=READY pgs=30 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.2 client_cookie=3253303c9dbb9de8 server_cookie=3afb93224903ef46 in_seq=0 out_seq=0 2024-03-24T00:19:10.111 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:10.107+0000 7f4735600640 10 client.26461 ms_handle_connect on v2:172.21.15.193:6836/2921520053 2024-03-24T00:19:10.111 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:10.107+0000 7f4735600640 1 -- 172.21.15.118:0/3987321497 <== mds.2 v2:172.21.15.193:6836/2921520053 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x5627d203e500 con 0x5627d21cfa20 2024-03-24T00:19:10.111 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:10.107+0000 7f4735600640 10 client.26461 handle_command_reply: tid=0 2024-03-24T00:19:10.111 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:10.107+0000 7f4755a15740 1 client.26461 shutdown 2024-03-24T00:19:10.111 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:10.107+0000 7f4755a15740 1 -- 172.21.15.118:0/3987321497 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5627d2156580 msgr2=0x5627d2164340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:19:10.111 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:10.107+0000 7f4755a15740 1 --2- 172.21.15.118:0/3987321497 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5627d2156580 0x5627d2164340 secure :-1 s=READY pgs=375 cs=0 l=1 rev1=1 crypto rx=0x7f4738012800 tx=0x7f473c02cf90 comp rx=0 tx=0).stop 2024-03-24T00:19:10.112 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:10.107+0000 7f4755a15740 1 -- 172.21.15.118:0/3987321497 shutdown_connections 2024-03-24T00:19:10.112 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:10.107+0000 7f4755a15740 1 --2- 172.21.15.118:0/3987321497 >> [v2:172.21.15.193:6836/2921520053,v1:172.21.15.193:6837/2921520053] conn(0x5627d21cfa20 0x5627d21d1e40 crc :-1 s=READY pgs=30 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:19:10.112 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:10.107+0000 7f4755a15740 1 --2- 172.21.15.118:0/3987321497 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5627d2156580 0x5627d2164340 unknown :-1 s=CLOSED pgs=375 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:19:10.112 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:10.107+0000 7f4735600640 0 client.26461 ms_handle_reset on v2:172.21.15.193:6836/2921520053 2024-03-24T00:19:10.112 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:10.107+0000 7f4755a15740 1 --2- 172.21.15.118:0/3987321497 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5627d2155bc0 0x5627d2166150 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:19:10.112 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:10.107+0000 7f4755a15740 1 --2- 172.21.15.118:0/3987321497 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5627d21549f0 0x5627d2165c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:19:10.112 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:10.107+0000 7f4755a15740 1 -- 172.21.15.118:0/3987321497 >> 172.21.15.118:0/3987321497 conn(0x5627d214fde0 msgr2=0x5627d214e8d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:19:10.112 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:10.107+0000 7f4755a15740 1 -- 172.21.15.118:0/3987321497 shutdown_connections 2024-03-24T00:19:10.112 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:10.108+0000 7f4755a15740 1 -- 172.21.15.118:0/3987321497 wait complete. 2024-03-24T00:19:10.112 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:10.108+0000 7f4755a15740 20 client.26461 trim_cache size 0 max 16384 2024-03-24T00:19:10.112 INFO:teuthology.orchestra.run.smithi118.stdout:[] 2024-03-24T00:19:10.113 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:10.109+0000 7f4751600640 1 -- 172.21.15.118:0/1235124710 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7f47300a2880 msgr2=0x7f47300a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:19:10.113 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:10.109+0000 7f4751600640 1 --2- 172.21.15.118:0/1235124710 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7f47300a2880 0x7f47300a4d40 secure :-1 s=READY pgs=335 cs=0 l=1 rev1=1 crypto rx=0x7f474409b690 tx=0x7f4744005f70 comp rx=0 tx=0).stop 2024-03-24T00:19:10.113 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:10.109+0000 7f4751600640 1 -- 172.21.15.118:0/1235124710 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f474c107300 msgr2=0x7f474c1a1010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:19:10.113 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:10.109+0000 7f4751600640 1 --2- 172.21.15.118:0/1235124710 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f474c107300 0x7f474c1a1010 secure :-1 s=READY pgs=374 cs=0 l=1 rev1=1 crypto rx=0x7f4744009fd0 tx=0x7f474400ef10 comp rx=0 tx=0).stop 2024-03-24T00:19:10.113 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:10.109+0000 7f4751600640 1 -- 172.21.15.118:0/1235124710 shutdown_connections 2024-03-24T00:19:10.113 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:10.109+0000 7f4751600640 1 --2- 172.21.15.118:0/1235124710 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7f47300a2880 0x7f47300a4d40 unknown :-1 s=CLOSED pgs=335 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:19:10.114 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:10.109+0000 7f4751600640 1 --2- 172.21.15.118:0/1235124710 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f474c107300 0x7f474c1a1010 unknown :-1 s=CLOSED pgs=374 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:19:10.114 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:10.109+0000 7f4751600640 1 --2- 172.21.15.118:0/1235124710 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f474c1049d0 0x7f474c19a010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:19:10.114 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:10.109+0000 7f4751600640 1 --2- 172.21.15.118:0/1235124710 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f474c1020a0 0x7f474c199ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:19:10.114 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:10.109+0000 7f4751600640 1 -- 172.21.15.118:0/1235124710 >> 172.21.15.118:0/1235124710 conn(0x7f474c0fba10 msgr2=0x7f474c101790 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:19:10.114 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:10.109+0000 7f4751600640 1 -- 172.21.15.118:0/1235124710 shutdown_connections 2024-03-24T00:19:10.114 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:10.109+0000 7f4751600640 1 -- 172.21.15.118:0/1235124710 wait complete. 2024-03-24T00:19:10.136 INFO:teuthology.orchestra.run:Running command with timeout 120 2024-03-24T00:19:10.136 DEBUG:teuthology.orchestra.run.smithi118:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:1 damage ls 2024-03-24T00:19:10.324 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:10.324+0000 7fb685200640 1 -- 172.21.15.118:0/1504713090 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb6801028a0 msgr2=0x7fb680102d20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:19:10.325 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:10.324+0000 7fb685200640 1 --2- 172.21.15.118:0/1504713090 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb6801028a0 0x7fb680102d20 secure :-1 s=READY pgs=377 cs=0 l=1 rev1=1 crypto rx=0x7fb66c009900 tx=0x7fb66c02bc30 comp rx=0 tx=0).stop 2024-03-24T00:19:10.325 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:10.324+0000 7fb685200640 1 -- 172.21.15.118:0/1504713090 shutdown_connections 2024-03-24T00:19:10.325 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:10.324+0000 7fb685200640 1 --2- 172.21.15.118:0/1504713090 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb680103260 0x7fb680109af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:19:10.325 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:10.324+0000 7fb685200640 1 --2- 172.21.15.118:0/1504713090 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb6801028a0 0x7fb680102d20 unknown :-1 s=CLOSED pgs=377 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:19:10.325 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:10.324+0000 7fb685200640 1 --2- 172.21.15.118:0/1504713090 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fb6801016b0 0x7fb680101ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:19:10.325 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:10.325+0000 7fb685200640 1 -- 172.21.15.118:0/1504713090 >> 172.21.15.118:0/1504713090 conn(0x7fb6800fcc40 msgr2=0x7fb6800ff060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:19:10.325 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:10.325+0000 7fb685200640 1 -- 172.21.15.118:0/1504713090 shutdown_connections 2024-03-24T00:19:10.325 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:10.325+0000 7fb685200640 1 -- 172.21.15.118:0/1504713090 wait complete. 2024-03-24T00:19:10.327 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:10.325+0000 7fb685200640 1 Processor -- start 2024-03-24T00:19:10.327 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:10.326+0000 7fb685200640 1 -- start start 2024-03-24T00:19:10.327 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:10.326+0000 7fb685200640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fb6801016b0 0x7fb680197a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:19:10.327 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:10.326+0000 7fb685200640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb6801028a0 0x7fb680197f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:19:10.327 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:10.326+0000 7fb685200640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb680103260 0x7fb68019efb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:19:10.327 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:10.326+0000 7fb685200640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb68010d380 con 0x7fb6801028a0 2024-03-24T00:19:10.327 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:10.326+0000 7fb685200640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb68010d200 con 0x7fb680103260 2024-03-24T00:19:10.328 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:10.326+0000 7fb685200640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb68010d500 con 0x7fb6801016b0 2024-03-24T00:19:10.328 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:10.326+0000 7fb67de00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fb6801016b0 0x7fb680197a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:19:10.328 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:10.326+0000 7fb67de00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fb6801016b0 0x7fb680197a00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.118:47372/0 (socket says 172.21.15.118:47372) 2024-03-24T00:19:10.328 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:10.326+0000 7fb67de00640 1 -- 172.21.15.118:0/3488545981 learned_addr learned my addr 172.21.15.118:0/3488545981 (peer_addr_for_me v2:172.21.15.118:0/0) 2024-03-24T00:19:10.328 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:10.326+0000 7fb67d400640 1 --2- 172.21.15.118:0/3488545981 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb6801028a0 0x7fb680197f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:19:10.328 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:10.326+0000 7fb67e800640 1 --2- 172.21.15.118:0/3488545981 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb680103260 0x7fb68019efb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:19:10.328 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:10.326+0000 7fb67de00640 1 -- 172.21.15.118:0/3488545981 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb680103260 msgr2=0x7fb68019efb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:19:10.328 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:10.326+0000 7fb67de00640 1 --2- 172.21.15.118:0/3488545981 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb680103260 0x7fb68019efb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:19:10.328 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:10.326+0000 7fb67de00640 1 -- 172.21.15.118:0/3488545981 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb6801028a0 msgr2=0x7fb680197f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:19:10.328 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:10.326+0000 7fb67de00640 1 --2- 172.21.15.118:0/3488545981 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb6801028a0 0x7fb680197f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:19:10.328 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:10.326+0000 7fb67de00640 1 -- 172.21.15.118:0/3488545981 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb68019f620 con 0x7fb6801016b0 2024-03-24T00:19:10.328 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:10.326+0000 7fb67e800640 1 --2- 172.21.15.118:0/3488545981 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb680103260 0x7fb68019efb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T00:19:10.328 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:10.327+0000 7fb67de00640 1 --2- 172.21.15.118:0/3488545981 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fb6801016b0 0x7fb680197a00 secure :-1 s=READY pgs=376 cs=0 l=1 rev1=1 crypto rx=0x7fb67000e9c0 tx=0x7fb67000ee80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:19:10.328 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:10.327+0000 7fb676a00640 1 -- 172.21.15.118:0/3488545981 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb67000cd80 con 0x7fb6801016b0 2024-03-24T00:19:10.329 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:10.327+0000 7fb676a00640 1 -- 172.21.15.118:0/3488545981 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fb6700035f0 con 0x7fb6801016b0 2024-03-24T00:19:10.329 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:10.327+0000 7fb685200640 1 -- 172.21.15.118:0/3488545981 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb68019f910 con 0x7fb6801016b0 2024-03-24T00:19:10.329 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:10.327+0000 7fb676a00640 1 -- 172.21.15.118:0/3488545981 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb670010690 con 0x7fb6801016b0 2024-03-24T00:19:10.329 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:10.327+0000 7fb685200640 1 -- 172.21.15.118:0/3488545981 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb68019fde0 con 0x7fb6801016b0 2024-03-24T00:19:10.330 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:10.330+0000 7fb676a00640 1 -- 172.21.15.118:0/3488545981 <== mon.1 v2:172.21.15.193:3300/0 4 ==== mgrmap(e 20) v1 ==== 137291+0+0 (secure 0 0 0) 0x7fb6700037a0 con 0x7fb6801016b0 2024-03-24T00:19:10.330 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:10.330+0000 7fb687fe8740 1 Processor -- start 2024-03-24T00:19:10.331 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:10.331+0000 7fb687fe8740 1 -- start start 2024-03-24T00:19:10.331 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:10.331+0000 7fb687fe8740 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x556ea26a5c20 0x556ea26a6040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:19:10.331 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:10.331+0000 7fb687fe8740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x556ea26a6df0 0x556ea26a7270 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:19:10.331 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:10.331+0000 7fb687fe8740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x556ea26a77b0 0x556ea26ae040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:19:10.331 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:10.331+0000 7fb687fe8740 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x556ea26062c0 con 0x556ea26a5c20 2024-03-24T00:19:10.331 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:10.331+0000 7fb687fe8740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x556ea25aec90 con 0x556ea26a6df0 2024-03-24T00:19:10.331 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:10.331+0000 7fb687fe8740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x556ea2602360 con 0x556ea26a77b0 2024-03-24T00:19:10.332 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:10.331+0000 7fb67d400640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x556ea26a77b0 0x556ea26ae040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:19:10.332 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:10.331+0000 7fb67e800640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x556ea26a5c20 0x556ea26a6040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:19:10.332 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:10.331+0000 7fb67d400640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x556ea26a77b0 0x556ea26ae040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.118:47388/0 (socket says 172.21.15.118:47388) 2024-03-24T00:19:10.332 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:10.331+0000 7fb67d400640 1 -- 172.21.15.118:0/3191848295 learned_addr learned my addr 172.21.15.118:0/3191848295 (peer_addr_for_me v2:172.21.15.118:0/0) 2024-03-24T00:19:10.332 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:10.332+0000 7fb67d400640 1 -- 172.21.15.118:0/3191848295 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x556ea26a6df0 msgr2=0x556ea26a7270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:19:10.332 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:10.332+0000 7fb67de00640 1 --2- 172.21.15.118:0/3191848295 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x556ea26a6df0 0x556ea26a7270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:19:10.332 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:10.332+0000 7fb67d400640 1 --2- 172.21.15.118:0/3191848295 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x556ea26a6df0 0x556ea26a7270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:19:10.332 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:10.332+0000 7fb67d400640 1 -- 172.21.15.118:0/3191848295 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x556ea26a5c20 msgr2=0x556ea26a6040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:19:10.332 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:10.332+0000 7fb67d400640 1 --2- 172.21.15.118:0/3191848295 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x556ea26a5c20 0x556ea26a6040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:19:10.333 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:10.332+0000 7fb67d400640 1 -- 172.21.15.118:0/3191848295 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x556ea25d3080 con 0x556ea26a77b0 2024-03-24T00:19:10.333 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:10.332+0000 7fb67e800640 1 --2- 172.21.15.118:0/3191848295 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x556ea26a5c20 0x556ea26a6040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T00:19:10.333 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:10.332+0000 7fb67de00640 1 --2- 172.21.15.118:0/3191848295 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x556ea26a6df0 0x556ea26a7270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-24T00:19:10.333 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:10.333+0000 7fb67d400640 1 --2- 172.21.15.118:0/3191848295 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x556ea26a77b0 0x556ea26ae040 secure :-1 s=READY pgs=377 cs=0 l=1 rev1=1 crypto rx=0x7fb66c000c00 tx=0x7fb66c0095e0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=a2c03b3a69e280e5 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:19:10.333 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:10.333+0000 7fb676a00640 1 --2- 172.21.15.118:0/3488545981 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7fb6640a2800 0x7fb6640a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:19:10.333 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:10.333+0000 7fb676a00640 1 -- 172.21.15.118:0/3488545981 <== mon.1 v2:172.21.15.193:3300/0 5 ==== osd_map(90..90 src has 1..90) v4 ==== 8277+0+0 (secure 0 0 0) 0x7fb6700ceea0 con 0x7fb6801016b0 2024-03-24T00:19:10.333 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:10.333+0000 7fb67e800640 1 --2- 172.21.15.118:0/3488545981 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7fb6640a2800 0x7fb6640a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:19:10.333 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:10.333+0000 7fb66b400640 1 -- 172.21.15.118:0/3191848295 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb66c045070 con 0x556ea26a77b0 2024-03-24T00:19:10.333 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:10.333+0000 7fb66b400640 1 -- 172.21.15.118:0/3191848295 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fb66c004040 con 0x556ea26a77b0 2024-03-24T00:19:10.333 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:10.333+0000 7fb66b400640 1 -- 172.21.15.118:0/3191848295 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb66c004eb0 con 0x556ea26a77b0 2024-03-24T00:19:10.334 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:10.334+0000 7fb687fe8740 1 -- 172.21.15.118:0/3191848295 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x556ea26a77b0 msgr2=0x556ea26ae040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:19:10.335 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:10.334+0000 7fb687fe8740 1 --2- 172.21.15.118:0/3191848295 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x556ea26a77b0 0x556ea26ae040 secure :-1 s=READY pgs=377 cs=0 l=1 rev1=1 crypto rx=0x7fb66c000c00 tx=0x7fb66c0095e0 comp rx=0 tx=0).stop 2024-03-24T00:19:10.335 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:10.334+0000 7fb67e800640 1 --2- 172.21.15.118:0/3488545981 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7fb6640a2800 0x7fb6640a4cc0 secure :-1 s=READY pgs=336 cs=0 l=1 rev1=1 crypto rx=0x7fb678009fd0 tx=0x7fb6780098a0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:19:10.335 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:10.334+0000 7fb687fe8740 1 -- 172.21.15.118:0/3191848295 shutdown_connections 2024-03-24T00:19:10.335 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:10.334+0000 7fb687fe8740 1 --2- 172.21.15.118:0/3191848295 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x556ea26a77b0 0x556ea26ae040 unknown :-1 s=CLOSED pgs=377 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:19:10.335 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:10.334+0000 7fb687fe8740 1 --2- 172.21.15.118:0/3191848295 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x556ea26a6df0 0x556ea26a7270 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:19:10.335 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:10.334+0000 7fb687fe8740 1 --2- 172.21.15.118:0/3191848295 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x556ea26a5c20 0x556ea26a6040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:19:10.335 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:10.334+0000 7fb687fe8740 1 -- 172.21.15.118:0/3191848295 >> 172.21.15.118:0/3191848295 conn(0x556ea269cc00 msgr2=0x556ea269d000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:19:10.335 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:10.334+0000 7fb687fe8740 1 -- 172.21.15.118:0/3191848295 shutdown_connections 2024-03-24T00:19:10.335 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:10.334+0000 7fb687fe8740 1 -- 172.21.15.118:0/3191848295 wait complete. 2024-03-24T00:19:10.335 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:10.335+0000 7fb687fe8740 1 -- start start 2024-03-24T00:19:10.335 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:10.335+0000 7fb687fe8740 1 Processor -- start 2024-03-24T00:19:10.336 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:10.336+0000 7fb687fe8740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x556ea26a5c20 0x556ea26aba70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:19:10.336 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:10.336+0000 7fb687fe8740 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x556ea26a6df0 0x556ea26abfb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:19:10.336 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:10.336+0000 7fb687fe8740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x556ea26a77b0 0x556ea26b2ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:19:10.336 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:10.336+0000 7fb687fe8740 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x556ea26a1550 con 0x556ea26a6df0 2024-03-24T00:19:10.336 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:10.336+0000 7fb687fe8740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x556ea25c20d0 con 0x556ea26a5c20 2024-03-24T00:19:10.336 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:10.336+0000 7fb687fe8740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x556ea26a16d0 con 0x556ea26a77b0 2024-03-24T00:19:10.337 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:10.336+0000 7fb687fe8740 10 client.0 _ll_register_callbacks cb 0x556ea25c6e80 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T00:19:10.337 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:10.336+0000 7fb67de00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x556ea26a5c20 0x556ea26aba70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:19:10.337 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:10.336+0000 7fb67de00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x556ea26a5c20 0x556ea26aba70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.118:54274/0 (socket says 172.21.15.118:54274) 2024-03-24T00:19:10.337 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:10.336+0000 7fb67de00640 1 -- 172.21.15.118:0/2831932165 learned_addr learned my addr 172.21.15.118:0/2831932165 (peer_addr_for_me v2:172.21.15.118:0/0) 2024-03-24T00:19:10.337 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:10.337+0000 7fb67e800640 1 --2- 172.21.15.118:0/2831932165 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x556ea26a77b0 0x556ea26b2ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:19:10.337 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:10.337+0000 7fb67d400640 1 --2- 172.21.15.118:0/2831932165 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x556ea26a6df0 0x556ea26abfb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:19:10.337 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:10.338+0000 7fb67de00640 1 -- 172.21.15.118:0/2831932165 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x556ea26a77b0 msgr2=0x556ea26b2ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:19:10.338 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:10.339+0000 7fb67de00640 1 --2- 172.21.15.118:0/2831932165 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x556ea26a77b0 0x556ea26b2ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:19:10.338 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:10.339+0000 7fb67de00640 1 -- 172.21.15.118:0/2831932165 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x556ea26a6df0 msgr2=0x556ea26abfb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:19:10.338 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:10.339+0000 7fb67de00640 1 --2- 172.21.15.118:0/2831932165 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x556ea26a6df0 0x556ea26abfb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:19:10.338 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:10.339+0000 7fb67de00640 1 -- 172.21.15.118:0/2831932165 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x556ea231a3f0 con 0x556ea26a5c20 2024-03-24T00:19:10.340 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:10.339+0000 7fb67e800640 1 --2- 172.21.15.118:0/2831932165 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x556ea26a77b0 0x556ea26b2ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T00:19:10.340 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:10.339+0000 7fb67d400640 1 --2- 172.21.15.118:0/2831932165 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x556ea26a6df0 0x556ea26abfb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T00:19:10.340 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:10.339+0000 7fb67de00640 1 --2- 172.21.15.118:0/2831932165 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x556ea26a5c20 0x556ea26aba70 secure :-1 s=READY pgs=378 cs=0 l=1 rev1=1 crypto rx=0x7fb6700981e0 tx=0x7fb67009b3d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=49da03a468e38d67 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:19:10.340 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:10.339+0000 7fb65fe00640 10 client.0 ms_handle_connect on v2:172.21.15.189:3300/0 2024-03-24T00:19:10.340 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:10.339+0000 7fb65fe00640 1 -- 172.21.15.118:0/2831932165 <== mon.2 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb67009f960 con 0x556ea26a5c20 2024-03-24T00:19:10.340 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:10.340+0000 7fb687fe8740 1 -- 172.21.15.118:0/2831932165 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fb66c004040 con 0x556ea26a5c20 2024-03-24T00:19:10.340 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:10.340+0000 7fb65fe00640 1 -- 172.21.15.118:0/2831932165 <== mon.2 v2:172.21.15.189:3300/0 2 ==== osd_map(90..90 src has 1..90) v4 ==== 8277+0+0 (secure 0 0 0) 0x7fb6700860d0 con 0x556ea26a5c20 2024-03-24T00:19:10.340 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:10.340+0000 7fb65fe00640 1 -- 172.21.15.118:0/2831932165 <== mon.2 v2:172.21.15.189:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) v2 ==== 24+0+0 (secure 0 0 0) 0x7fb6700b9050 con 0x556ea26a5c20 2024-03-24T00:19:10.340 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:10.340+0000 7fb687fe8740 10 client.26459 fetch_fsmap learned FSMap version 28 2024-03-24T00:19:10.340 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:10.340+0000 7fb687fe8740 1 -- 172.21.15.118:0/2831932165 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) v3 -- 0x556ea23e47d0 con 0x556ea26a5c20 2024-03-24T00:19:10.340 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:10.340+0000 7fb65fe00640 1 -- 172.21.15.118:0/2831932165 <== mon.2 v2:172.21.15.189:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fb66c004040 con 0x556ea26a5c20 2024-03-24T00:19:10.340 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:10.340+0000 7fb65fe00640 1 -- 172.21.15.118:0/2831932165 <== mon.2 v2:172.21.15.189:3300/0 5 ==== fsmap(e 28) v1 ==== 7934+0+0 (secure 0 0 0) 0x7fb67009cb40 con 0x556ea26a5c20 2024-03-24T00:19:10.341 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:10.340+0000 7fb687fe8740 10 client.26459 fetch_fsmap finished waiting for FSMap version 28 2024-03-24T00:19:10.341 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:10.340+0000 7fb687fe8740 10 client.26459 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.g 2024-03-24T00:19:10.341 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:10.340+0000 7fb687fe8740 20 client.26459 populate_metadata read hostname 'smithi118' 2024-03-24T00:19:10.341 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:10.340+0000 7fb687fe8740 1 --2- 172.21.15.118:0/2831932165 >> [v2:172.21.15.118:6838/3285788503,v1:172.21.15.118:6839/3285788503] conn(0x556ea26b7410 0x556ea26b9820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:19:10.341 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:10.341+0000 7fb687fe8740 4 client.26459 mds_command: new command op to 24550 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-24T00:19:10.342 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:10.341+0000 7fb687fe8740 1 -- 172.21.15.118:0/2831932165 --> [v2:172.21.15.118:6838/3285788503,v1:172.21.15.118:6839/3285788503] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x556ea2613800 con 0x556ea26b7410 2024-03-24T00:19:10.342 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:10.341+0000 7fb67d400640 1 --2- 172.21.15.118:0/2831932165 >> [v2:172.21.15.118:6838/3285788503,v1:172.21.15.118:6839/3285788503] conn(0x556ea26b7410 0x556ea26b9820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:19:10.342 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:10.341+0000 7fb67d400640 1 --2- 172.21.15.118:0/2831932165 >> [v2:172.21.15.118:6838/3285788503,v1:172.21.15.118:6839/3285788503] conn(0x556ea26b7410 0x556ea26b9820 crc :-1 s=READY pgs=32 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=7dc255ed4c272865 server_cookie=d3ce8be7eb4f92d0 in_seq=0 out_seq=0 2024-03-24T00:19:10.342 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:10.341+0000 7fb65fe00640 10 client.26459 ms_handle_connect on v2:172.21.15.118:6838/3285788503 2024-03-24T00:19:10.343 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:10.342+0000 7fb65fe00640 1 -- 172.21.15.118:0/2831932165 <== mds.1 v2:172.21.15.118:6838/3285788503 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x556ea2613800 con 0x556ea26b7410 2024-03-24T00:19:10.343 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:10.342+0000 7fb65fe00640 10 client.26459 handle_command_reply: tid=0 2024-03-24T00:19:10.343 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:10.342+0000 7fb687fe8740 1 client.26459 shutdown 2024-03-24T00:19:10.343 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:10.343+0000 7fb687fe8740 1 -- 172.21.15.118:0/2831932165 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x556ea26a5c20 msgr2=0x556ea26aba70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:19:10.343 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:10.343+0000 7fb687fe8740 1 --2- 172.21.15.118:0/2831932165 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x556ea26a5c20 0x556ea26aba70 secure :-1 s=READY pgs=378 cs=0 l=1 rev1=1 crypto rx=0x7fb6700981e0 tx=0x7fb67009b3d0 comp rx=0 tx=0).stop 2024-03-24T00:19:10.343 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:10.343+0000 7fb687fe8740 1 -- 172.21.15.118:0/2831932165 shutdown_connections 2024-03-24T00:19:10.343 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:10.343+0000 7fb687fe8740 1 --2- 172.21.15.118:0/2831932165 >> [v2:172.21.15.118:6838/3285788503,v1:172.21.15.118:6839/3285788503] conn(0x556ea26b7410 0x556ea26b9820 crc :-1 s=READY pgs=32 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:19:10.343 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:10.343+0000 7fb687fe8740 1 --2- 172.21.15.118:0/2831932165 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x556ea26a77b0 0x556ea26b2ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:19:10.343 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:10.343+0000 7fb687fe8740 1 --2- 172.21.15.118:0/2831932165 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x556ea26a6df0 0x556ea26abfb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:19:10.343 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:10.343+0000 7fb687fe8740 1 --2- 172.21.15.118:0/2831932165 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x556ea26a5c20 0x556ea26aba70 unknown :-1 s=CLOSED pgs=378 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:19:10.344 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:10.343+0000 7fb687fe8740 1 -- 172.21.15.118:0/2831932165 >> 172.21.15.118:0/2831932165 conn(0x556ea269cc00 msgr2=0x556ea269beb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:19:10.344 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:10.343+0000 7fb65fe00640 0 client.26459 ms_handle_reset on v2:172.21.15.118:6838/3285788503 2024-03-24T00:19:10.344 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:10.343+0000 7fb687fe8740 1 -- 172.21.15.118:0/2831932165 shutdown_connections 2024-03-24T00:19:10.344 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:10.343+0000 7fb687fe8740 1 -- 172.21.15.118:0/2831932165 wait complete. 2024-03-24T00:19:10.344 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:10.343+0000 7fb687fe8740 20 client.26459 trim_cache size 0 max 16384 2024-03-24T00:19:10.360 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:10.360+0000 7fb687fe8740 1 Processor -- start 2024-03-24T00:19:10.360 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:10.360+0000 7fb687fe8740 1 -- start start 2024-03-24T00:19:10.360 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:10.360+0000 7fb687fe8740 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x556ea26a5c20 0x556ea26abf10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:19:10.360 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:10.361+0000 7fb687fe8740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x556ea26a6df0 0x556ea26b2fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:19:10.360 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:10.361+0000 7fb687fe8740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x556ea26a77b0 0x556ea26a0570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:19:10.360 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:10.361+0000 7fb687fe8740 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb6801aef00 con 0x556ea26a5c20 2024-03-24T00:19:10.360 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:10.361+0000 7fb687fe8740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb6801b0880 con 0x556ea26a77b0 2024-03-24T00:19:10.360 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:10.361+0000 7fb687fe8740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb6801af560 con 0x556ea26a6df0 2024-03-24T00:19:10.361 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:10.361+0000 7fb67d400640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x556ea26a6df0 0x556ea26b2fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:19:10.361 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:10.361+0000 7fb67e800640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x556ea26a77b0 0x556ea26a0570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:19:10.361 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:10.361+0000 7fb67de00640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x556ea26a5c20 0x556ea26abf10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:19:10.361 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:10.361+0000 7fb67d400640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x556ea26a6df0 0x556ea26b2fb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.118:47408/0 (socket says 172.21.15.118:47408) 2024-03-24T00:19:10.361 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:10.361+0000 7fb67d400640 1 -- 172.21.15.118:0/3607913719 learned_addr learned my addr 172.21.15.118:0/3607913719 (peer_addr_for_me v2:172.21.15.118:0/0) 2024-03-24T00:19:10.361 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:10.361+0000 7fb67d400640 1 -- 172.21.15.118:0/3607913719 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x556ea26a77b0 msgr2=0x556ea26a0570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:19:10.361 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:10.361+0000 7fb67d400640 1 --2- 172.21.15.118:0/3607913719 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x556ea26a77b0 0x556ea26a0570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:19:10.362 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:10.361+0000 7fb67d400640 1 -- 172.21.15.118:0/3607913719 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x556ea26a5c20 msgr2=0x556ea26abf10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:19:10.362 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:10.361+0000 7fb67d400640 1 --2- 172.21.15.118:0/3607913719 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x556ea26a5c20 0x556ea26abf10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:19:10.362 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:10.361+0000 7fb67d400640 1 -- 172.21.15.118:0/3607913719 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x556ea2698510 con 0x556ea26a6df0 2024-03-24T00:19:10.362 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:10.362+0000 7fb67d400640 1 --2- 172.21.15.118:0/3607913719 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x556ea26a6df0 0x556ea26b2fb0 secure :-1 s=READY pgs=378 cs=0 l=1 rev1=1 crypto rx=0x7fb66c009900 tx=0x7fb66c02df90 comp rx=0 tx=0).ready entity=mon.1 client_cookie=b0176a48622a9446 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:19:10.363 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:10.362+0000 7fb66aa00640 1 -- 172.21.15.118:0/3607913719 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb66c045040 con 0x556ea26a6df0 2024-03-24T00:19:10.363 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:10.362+0000 7fb66aa00640 1 -- 172.21.15.118:0/3607913719 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fb66c0042d0 con 0x556ea26a6df0 2024-03-24T00:19:10.363 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:10.362+0000 7fb66aa00640 1 -- 172.21.15.118:0/3607913719 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb66c005420 con 0x556ea26a6df0 2024-03-24T00:19:10.363 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:10.363+0000 7fb687fe8740 1 -- 172.21.15.118:0/3607913719 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x556ea26a6df0 msgr2=0x556ea26b2fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:19:10.363 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:10.363+0000 7fb687fe8740 1 --2- 172.21.15.118:0/3607913719 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x556ea26a6df0 0x556ea26b2fb0 secure :-1 s=READY pgs=378 cs=0 l=1 rev1=1 crypto rx=0x7fb66c009900 tx=0x7fb66c02df90 comp rx=0 tx=0).stop 2024-03-24T00:19:10.363 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:10.363+0000 7fb687fe8740 1 -- 172.21.15.118:0/3607913719 shutdown_connections 2024-03-24T00:19:10.363 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:10.363+0000 7fb687fe8740 1 --2- 172.21.15.118:0/3607913719 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x556ea26a77b0 0x556ea26a0570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:19:10.363 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:10.363+0000 7fb687fe8740 1 --2- 172.21.15.118:0/3607913719 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x556ea26a6df0 0x556ea26b2fb0 unknown :-1 s=CLOSED pgs=378 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:19:10.363 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:10.363+0000 7fb687fe8740 1 --2- 172.21.15.118:0/3607913719 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x556ea26a5c20 0x556ea26abf10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:19:10.363 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:10.363+0000 7fb687fe8740 1 -- 172.21.15.118:0/3607913719 >> 172.21.15.118:0/3607913719 conn(0x556ea269cc00 msgr2=0x556ea269fe60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:19:10.363 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:10.363+0000 7fb687fe8740 1 -- 172.21.15.118:0/3607913719 shutdown_connections 2024-03-24T00:19:10.363 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:10.363+0000 7fb687fe8740 1 -- 172.21.15.118:0/3607913719 wait complete. 2024-03-24T00:19:10.364 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:10.363+0000 7fb687fe8740 1 -- start start 2024-03-24T00:19:10.365 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:10.364+0000 7fb687fe8740 1 Processor -- start 2024-03-24T00:19:10.365 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:10.364+0000 7fb687fe8740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x556ea26a5c20 0x556ea26adf80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:19:10.365 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:10.364+0000 7fb687fe8740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x556ea26a6df0 0x556ea26ae4c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:19:10.365 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:10.364+0000 7fb687fe8740 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x556ea26a77b0 0x556ea2710c70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:19:10.365 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:10.364+0000 7fb687fe8740 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb6801b0220 con 0x556ea26a77b0 2024-03-24T00:19:10.365 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:10.364+0000 7fb687fe8740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb6801afbc0 con 0x556ea26a6df0 2024-03-24T00:19:10.365 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:10.364+0000 7fb687fe8740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb6801ad0c0 con 0x556ea26a5c20 2024-03-24T00:19:10.365 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:10.364+0000 7fb687fe8740 10 client.0 _ll_register_callbacks cb 0x7fb670003e10 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T00:19:10.365 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:10.364+0000 7fb67de00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x556ea26a5c20 0x556ea26adf80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:19:10.365 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:10.364+0000 7fb67de00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x556ea26a5c20 0x556ea26adf80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.118:47412/0 (socket says 172.21.15.118:47412) 2024-03-24T00:19:10.366 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:10.364+0000 7fb67e800640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x556ea26a77b0 0x556ea2710c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:19:10.366 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:10.364+0000 7fb67de00640 1 -- 172.21.15.118:0/3502454553 learned_addr learned my addr 172.21.15.118:0/3502454553 (peer_addr_for_me v2:172.21.15.118:0/0) 2024-03-24T00:19:10.366 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:10.364+0000 7fb67d400640 1 --2- 172.21.15.118:0/3502454553 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x556ea26a6df0 0x556ea26ae4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:19:10.366 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:10.365+0000 7fb67de00640 1 -- 172.21.15.118:0/3502454553 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x556ea26a6df0 msgr2=0x556ea26ae4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:19:10.366 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:10.365+0000 7fb67de00640 1 --2- 172.21.15.118:0/3502454553 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x556ea26a6df0 0x556ea26ae4c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:19:10.366 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:10.365+0000 7fb67de00640 1 -- 172.21.15.118:0/3502454553 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x556ea26a77b0 msgr2=0x556ea2710c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:19:10.366 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:10.365+0000 7fb67de00640 1 --2- 172.21.15.118:0/3502454553 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x556ea26a77b0 0x556ea2710c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:19:10.366 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:10.365+0000 7fb67de00640 1 -- 172.21.15.118:0/3502454553 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x556ea2711260 con 0x556ea26a5c20 2024-03-24T00:19:10.366 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:10.365+0000 7fb67de00640 1 --2- 172.21.15.118:0/3502454553 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x556ea26a5c20 0x556ea26adf80 secure :-1 s=READY pgs=379 cs=0 l=1 rev1=1 crypto rx=0x7fb670098190 tx=0x7fb670005b20 comp rx=0 tx=0).ready entity=mon.1 client_cookie=72f0cfabf3e2c9dd server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:19:10.366 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:10.365+0000 7fb65fe00640 10 client.0 ms_handle_connect on v2:172.21.15.193:3300/0 2024-03-24T00:19:10.366 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:10.365+0000 7fb65fe00640 1 -- 172.21.15.118:0/3502454553 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fb67009af90 con 0x556ea26a5c20 2024-03-24T00:19:10.366 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:10.365+0000 7fb65fe00640 1 -- 172.21.15.118:0/3502454553 <== mon.1 v2:172.21.15.193:3300/0 2 ==== osd_map(90..90 src has 1..90) v4 ==== 8277+0+0 (secure 0 0 0) 0x7fb6700d20e0 con 0x556ea26a5c20 2024-03-24T00:19:10.366 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:10.365+0000 7fb687fe8740 1 -- 172.21.15.118:0/3502454553 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fb66c0042d0 con 0x556ea26a5c20 2024-03-24T00:19:10.368 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:10.366+0000 7fb65fe00640 1 -- 172.21.15.118:0/3502454553 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) v2 ==== 24+0+0 (secure 0 0 0) 0x7fb67009b5e0 con 0x556ea26a5c20 2024-03-24T00:19:10.369 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:10.366+0000 7fb687fe8740 10 client.26479 fetch_fsmap learned FSMap version 28 2024-03-24T00:19:10.369 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:10.366+0000 7fb687fe8740 1 -- 172.21.15.118:0/3502454553 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) v3 -- 0x556ea27125d0 con 0x556ea26a5c20 2024-03-24T00:19:10.369 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:10.366+0000 7fb65fe00640 1 -- 172.21.15.118:0/3502454553 <== mon.1 v2:172.21.15.193:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fb66c0042d0 con 0x556ea26a5c20 2024-03-24T00:19:10.369 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:10.367+0000 7fb65fe00640 1 -- 172.21.15.118:0/3502454553 <== mon.1 v2:172.21.15.193:3300/0 5 ==== fsmap(e 28) v1 ==== 7934+0+0 (secure 0 0 0) 0x7fb670084020 con 0x556ea26a5c20 2024-03-24T00:19:10.369 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:10.367+0000 7fb687fe8740 10 client.26479 fetch_fsmap finished waiting for FSMap version 28 2024-03-24T00:19:10.369 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:10.367+0000 7fb687fe8740 10 client.26479 resolve_mds: resolved 1:1 to role '1:1' aka daemon mds.g 2024-03-24T00:19:10.369 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:10.367+0000 7fb687fe8740 20 client.26479 populate_metadata read hostname 'smithi118' 2024-03-24T00:19:10.369 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:10.367+0000 7fb687fe8740 1 --2- 172.21.15.118:0/3502454553 >> [v2:172.21.15.118:6838/3285788503,v1:172.21.15.118:6839/3285788503] conn(0x556ea2712d00 0x556ea27150f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:19:10.369 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:10.367+0000 7fb687fe8740 4 client.26479 mds_command: new command op to 24550 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-24T00:19:10.369 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:10.367+0000 7fb687fe8740 1 -- 172.21.15.118:0/3502454553 --> [v2:172.21.15.118:6838/3285788503,v1:172.21.15.118:6839/3285788503] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x556ea260db50 con 0x556ea2712d00 2024-03-24T00:19:10.369 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:10.367+0000 7fb67d400640 1 --2- 172.21.15.118:0/3502454553 >> [v2:172.21.15.118:6838/3285788503,v1:172.21.15.118:6839/3285788503] conn(0x556ea2712d00 0x556ea27150f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:19:10.369 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:10.367+0000 7fb67d400640 1 --2- 172.21.15.118:0/3502454553 >> [v2:172.21.15.118:6838/3285788503,v1:172.21.15.118:6839/3285788503] conn(0x556ea2712d00 0x556ea27150f0 crc :-1 s=READY pgs=33 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.1 client_cookie=c529be12da57c393 server_cookie=568db0260619bc5d in_seq=0 out_seq=0 2024-03-24T00:19:10.369 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:10.367+0000 7fb65fe00640 10 client.26479 ms_handle_connect on v2:172.21.15.118:6838/3285788503 2024-03-24T00:19:10.369 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:10.368+0000 7fb65fe00640 1 -- 172.21.15.118:0/3502454553 <== mds.1 v2:172.21.15.118:6838/3285788503 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x556ea260db50 con 0x556ea2712d00 2024-03-24T00:19:10.369 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:10.368+0000 7fb65fe00640 10 client.26479 handle_command_reply: tid=0 2024-03-24T00:19:10.370 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:10.368+0000 7fb687fe8740 1 client.26479 shutdown 2024-03-24T00:19:10.370 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:10.368+0000 7fb687fe8740 1 -- 172.21.15.118:0/3502454553 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x556ea26a5c20 msgr2=0x556ea26adf80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:19:10.370 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:10.368+0000 7fb687fe8740 1 --2- 172.21.15.118:0/3502454553 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x556ea26a5c20 0x556ea26adf80 secure :-1 s=READY pgs=379 cs=0 l=1 rev1=1 crypto rx=0x7fb670098190 tx=0x7fb670005b20 comp rx=0 tx=0).stop 2024-03-24T00:19:10.370 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:10.368+0000 7fb687fe8740 1 -- 172.21.15.118:0/3502454553 shutdown_connections 2024-03-24T00:19:10.370 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:10.368+0000 7fb687fe8740 1 --2- 172.21.15.118:0/3502454553 >> [v2:172.21.15.118:6838/3285788503,v1:172.21.15.118:6839/3285788503] conn(0x556ea2712d00 0x556ea27150f0 crc :-1 s=READY pgs=33 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:19:10.370 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:10.368+0000 7fb687fe8740 1 --2- 172.21.15.118:0/3502454553 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x556ea26a77b0 0x556ea2710c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:19:10.370 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:10.368+0000 7fb687fe8740 1 --2- 172.21.15.118:0/3502454553 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x556ea26a6df0 0x556ea26ae4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:19:10.370 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:10.368+0000 7fb687fe8740 1 --2- 172.21.15.118:0/3502454553 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x556ea26a5c20 0x556ea26adf80 unknown :-1 s=CLOSED pgs=379 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:19:10.370 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:10.368+0000 7fb687fe8740 1 -- 172.21.15.118:0/3502454553 >> 172.21.15.118:0/3502454553 conn(0x556ea269cc00 msgr2=0x556ea26a06f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:19:10.370 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:10.368+0000 7fb65fe00640 0 client.26479 ms_handle_reset on v2:172.21.15.118:6838/3285788503 2024-03-24T00:19:10.370 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:10.368+0000 7fb687fe8740 1 -- 172.21.15.118:0/3502454553 shutdown_connections 2024-03-24T00:19:10.370 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:10.368+0000 7fb687fe8740 1 -- 172.21.15.118:0/3502454553 wait complete. 2024-03-24T00:19:10.370 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:10.369+0000 7fb687fe8740 20 client.26479 trim_cache size 0 max 16384 2024-03-24T00:19:10.371 INFO:teuthology.orchestra.run.smithi118.stdout:[] 2024-03-24T00:19:10.372 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:10.369+0000 7fb685200640 1 -- 172.21.15.118:0/3488545981 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7fb6640a2800 msgr2=0x7fb6640a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:19:10.372 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:10.369+0000 7fb685200640 1 --2- 172.21.15.118:0/3488545981 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7fb6640a2800 0x7fb6640a4cc0 secure :-1 s=READY pgs=336 cs=0 l=1 rev1=1 crypto rx=0x7fb678009fd0 tx=0x7fb6780098a0 comp rx=0 tx=0).stop 2024-03-24T00:19:10.372 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:10.369+0000 7fb685200640 1 -- 172.21.15.118:0/3488545981 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fb6801016b0 msgr2=0x7fb680197a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:19:10.372 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:10.369+0000 7fb685200640 1 --2- 172.21.15.118:0/3488545981 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fb6801016b0 0x7fb680197a00 secure :-1 s=READY pgs=376 cs=0 l=1 rev1=1 crypto rx=0x7fb67000e9c0 tx=0x7fb67000ee80 comp rx=0 tx=0).stop 2024-03-24T00:19:10.372 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:10.369+0000 7fb685200640 1 -- 172.21.15.118:0/3488545981 shutdown_connections 2024-03-24T00:19:10.372 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:10.369+0000 7fb685200640 1 --2- 172.21.15.118:0/3488545981 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7fb6640a2800 0x7fb6640a4cc0 unknown :-1 s=CLOSED pgs=336 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:19:10.372 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:10.369+0000 7fb685200640 1 --2- 172.21.15.118:0/3488545981 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fb680103260 0x7fb68019efb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:19:10.372 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:10.369+0000 7fb685200640 1 --2- 172.21.15.118:0/3488545981 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fb6801028a0 0x7fb680197f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:19:10.372 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:10.369+0000 7fb685200640 1 --2- 172.21.15.118:0/3488545981 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fb6801016b0 0x7fb680197a00 unknown :-1 s=CLOSED pgs=376 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:19:10.373 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:10.369+0000 7fb685200640 1 -- 172.21.15.118:0/3488545981 >> 172.21.15.118:0/3488545981 conn(0x7fb6800fcc40 msgr2=0x7fb680105c20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:19:10.373 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:10.369+0000 7fb685200640 1 -- 172.21.15.118:0/3488545981 shutdown_connections 2024-03-24T00:19:10.373 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:10.370+0000 7fb685200640 1 -- 172.21.15.118:0/3488545981 wait complete. 2024-03-24T00:19:10.387 INFO:teuthology.orchestra.run:Running command with timeout 120 2024-03-24T00:19:10.387 DEBUG:teuthology.orchestra.run.smithi118:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:0 damage ls 2024-03-24T00:19:10.450 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:19:10 smithi118 ceph-mon[31535]: pgmap v502: 97 pgs: 97 active+clean; 64 MiB data, 572 MiB used, 1.0 TiB / 1.0 TiB avail; 683 KiB/s rd, 705 KiB/s wr, 194 op/s 2024-03-24T00:19:10.582 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:10.579+0000 7fd894c00640 1 -- 172.21.15.118:0/3741971535 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fd8901016b0 msgr2=0x7fd890101ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:19:10.583 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:10.579+0000 7fd894c00640 1 --2- 172.21.15.118:0/3741971535 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fd8901016b0 0x7fd890101ad0 secure :-1 s=READY pgs=380 cs=0 l=1 rev1=1 crypto rx=0x7fd880009860 tx=0x7fd88002bb70 comp rx=0 tx=0).stop 2024-03-24T00:19:10.583 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:10.580+0000 7fd894c00640 1 -- 172.21.15.118:0/3741971535 shutdown_connections 2024-03-24T00:19:10.583 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:10.580+0000 7fd894c00640 1 --2- 172.21.15.118:0/3741971535 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd890103260 0x7fd890109af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:19:10.583 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:10.580+0000 7fd894c00640 1 --2- 172.21.15.118:0/3741971535 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fd8901028a0 0x7fd890102d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:19:10.583 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:10.580+0000 7fd894c00640 1 --2- 172.21.15.118:0/3741971535 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fd8901016b0 0x7fd890101ad0 unknown :-1 s=CLOSED pgs=380 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:19:10.583 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:10.580+0000 7fd894c00640 1 -- 172.21.15.118:0/3741971535 >> 172.21.15.118:0/3741971535 conn(0x7fd8900fcc40 msgr2=0x7fd8900ff060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:19:10.583 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:10.580+0000 7fd894c00640 1 -- 172.21.15.118:0/3741971535 shutdown_connections 2024-03-24T00:19:10.583 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:10.580+0000 7fd894c00640 1 -- 172.21.15.118:0/3741971535 wait complete. 2024-03-24T00:19:10.583 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:10.581+0000 7fd894c00640 1 Processor -- start 2024-03-24T00:19:10.583 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:10.581+0000 7fd894c00640 1 -- start start 2024-03-24T00:19:10.583 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:10.581+0000 7fd894c00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fd8901016b0 0x7fd890199c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:19:10.583 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:10.581+0000 7fd894c00640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fd8901028a0 0x7fd89019a170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:19:10.584 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:10.581+0000 7fd894c00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd890103260 0x7fd8901a11a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:19:10.584 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:10.581+0000 7fd894c00640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd89010d300 con 0x7fd8901028a0 2024-03-24T00:19:10.584 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:10.581+0000 7fd894c00640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd89010d180 con 0x7fd890103260 2024-03-24T00:19:10.584 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:10.581+0000 7fd894c00640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd89010d480 con 0x7fd8901016b0 2024-03-24T00:19:10.584 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:10.581+0000 7fd88de00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fd8901016b0 0x7fd890199c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:19:10.584 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:10.581+0000 7fd88de00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fd8901016b0 0x7fd890199c30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.118:47428/0 (socket says 172.21.15.118:47428) 2024-03-24T00:19:10.584 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:10.581+0000 7fd88de00640 1 -- 172.21.15.118:0/2746976000 learned_addr learned my addr 172.21.15.118:0/2746976000 (peer_addr_for_me v2:172.21.15.118:0/0) 2024-03-24T00:19:10.584 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:10.581+0000 7fd88e800640 1 --2- 172.21.15.118:0/2746976000 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd890103260 0x7fd8901a11a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:19:10.584 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:10.582+0000 7fd88d400640 1 --2- 172.21.15.118:0/2746976000 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fd8901028a0 0x7fd89019a170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:19:10.584 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:10.582+0000 7fd88de00640 1 -- 172.21.15.118:0/2746976000 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd890103260 msgr2=0x7fd8901a11a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:19:10.584 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:10.582+0000 7fd88de00640 1 --2- 172.21.15.118:0/2746976000 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd890103260 0x7fd8901a11a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:19:10.584 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:10.582+0000 7fd88de00640 1 -- 172.21.15.118:0/2746976000 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fd8901028a0 msgr2=0x7fd89019a170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:19:10.584 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:10.582+0000 7fd88de00640 1 --2- 172.21.15.118:0/2746976000 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fd8901028a0 0x7fd89019a170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:19:10.584 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:10.582+0000 7fd88de00640 1 -- 172.21.15.118:0/2746976000 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd8901a1840 con 0x7fd8901016b0 2024-03-24T00:19:10.585 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:10.582+0000 7fd88d400640 1 --2- 172.21.15.118:0/2746976000 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fd8901028a0 0x7fd89019a170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T00:19:10.585 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:10.582+0000 7fd88de00640 1 --2- 172.21.15.118:0/2746976000 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fd8901016b0 0x7fd890199c30 secure :-1 s=READY pgs=381 cs=0 l=1 rev1=1 crypto rx=0x7fd8800349e0 tx=0x7fd880032430 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:19:10.585 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:10.582+0000 7fd886a00640 1 -- 172.21.15.118:0/2746976000 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd880041020 con 0x7fd8901016b0 2024-03-24T00:19:10.585 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:10.582+0000 7fd886a00640 1 -- 172.21.15.118:0/2746976000 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fd88002c9f0 con 0x7fd8901016b0 2024-03-24T00:19:10.585 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:10.582+0000 7fd894c00640 1 -- 172.21.15.118:0/2746976000 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd8901a1ad0 con 0x7fd8901016b0 2024-03-24T00:19:10.585 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:10.583+0000 7fd886a00640 1 -- 172.21.15.118:0/2746976000 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd880039040 con 0x7fd8901016b0 2024-03-24T00:19:10.585 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:10.583+0000 7fd894c00640 1 -- 172.21.15.118:0/2746976000 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd8901a1fc0 con 0x7fd8901016b0 2024-03-24T00:19:10.586 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:10.586+0000 7fd8979a5740 1 Processor -- start 2024-03-24T00:19:10.588 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:10.587+0000 7fd8979a5740 1 -- start start 2024-03-24T00:19:10.588 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:10.587+0000 7fd8979a5740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5594786ff810 0x5594786ffc30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:19:10.588 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:10.587+0000 7fd8979a5740 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5594787009e0 0x559478700e60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:19:10.588 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:10.587+0000 7fd8979a5740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5594787013a0 0x559478707c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:19:10.588 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:10.587+0000 7fd8979a5740 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x5594786642c0 con 0x5594787009e0 2024-03-24T00:19:10.588 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:10.587+0000 7fd8979a5740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x55947860cc90 con 0x5594786ff810 2024-03-24T00:19:10.588 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:10.587+0000 7fd8979a5740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x559478660360 con 0x5594787013a0 2024-03-24T00:19:10.588 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:10.587+0000 7fd886a00640 1 -- 172.21.15.118:0/2746976000 <== mon.1 v2:172.21.15.193:3300/0 4 ==== mgrmap(e 20) v1 ==== 137291+0+0 (secure 0 0 0) 0x7fd880044020 con 0x7fd8901016b0 2024-03-24T00:19:10.588 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:10.587+0000 7fd88e800640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5594786ff810 0x5594786ffc30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:19:10.588 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:10.587+0000 7fd88d400640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5594787013a0 0x559478707c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:19:10.588 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:10.587+0000 7fd88e800640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5594786ff810 0x5594786ffc30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.118:54324/0 (socket says 172.21.15.118:54324) 2024-03-24T00:19:10.588 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:10.587+0000 7fd88e800640 1 -- 172.21.15.118:0/2011461085 learned_addr learned my addr 172.21.15.118:0/2011461085 (peer_addr_for_me v2:172.21.15.118:0/0) 2024-03-24T00:19:10.589 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:10.588+0000 7fd88e800640 1 -- 172.21.15.118:0/2011461085 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5594787013a0 msgr2=0x559478707c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:19:10.589 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:10.588+0000 7fd88e800640 1 --2- 172.21.15.118:0/2011461085 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5594787013a0 0x559478707c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:19:10.589 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:10.588+0000 7fd88e800640 1 -- 172.21.15.118:0/2011461085 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5594787009e0 msgr2=0x559478700e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:19:10.589 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:10.588+0000 7fd88de00640 1 --2- 172.21.15.118:0/2011461085 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5594787009e0 0x559478700e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:19:10.589 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:10.588+0000 7fd88e800640 1 --2- 172.21.15.118:0/2011461085 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5594787009e0 0x559478700e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:19:10.589 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:10.588+0000 7fd88e800640 1 -- 172.21.15.118:0/2011461085 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x559478631080 con 0x5594786ff810 2024-03-24T00:19:10.589 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:10.588+0000 7fd88d400640 1 --2- 172.21.15.118:0/2011461085 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5594787013a0 0x559478707c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T00:19:10.589 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:10.588+0000 7fd88de00640 1 --2- 172.21.15.118:0/2011461085 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5594787009e0 0x559478700e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-24T00:19:10.589 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:10.588+0000 7fd88e800640 1 --2- 172.21.15.118:0/2011461085 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5594786ff810 0x5594786ffc30 secure :-1 s=READY pgs=379 cs=0 l=1 rev1=1 crypto rx=0x7fd88800e450 tx=0x7fd8880095c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=8e6ebdeb5734c638 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:19:10.592 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:10.589+0000 7fd87b000640 1 -- 172.21.15.118:0/2011461085 <== mon.2 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd88800d910 con 0x5594786ff810 2024-03-24T00:19:10.592 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:10.589+0000 7fd87b000640 1 -- 172.21.15.118:0/2011461085 <== mon.2 v2:172.21.15.189:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fd88801c070 con 0x5594786ff810 2024-03-24T00:19:10.593 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:10.589+0000 7fd87b000640 1 -- 172.21.15.118:0/2011461085 <== mon.2 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd888016380 con 0x5594786ff810 2024-03-24T00:19:10.593 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:10.589+0000 7fd886a00640 1 --2- 172.21.15.118:0/2746976000 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7fd8740a2800 0x7fd8740a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:19:10.593 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:10.589+0000 7fd886a00640 1 -- 172.21.15.118:0/2746976000 <== mon.1 v2:172.21.15.193:3300/0 5 ==== osd_map(90..90 src has 1..90) v4 ==== 8277+0+0 (secure 0 0 0) 0x7fd8800f0430 con 0x7fd8901016b0 2024-03-24T00:19:10.593 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:10.589+0000 7fd88de00640 1 --2- 172.21.15.118:0/2746976000 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7fd8740a2800 0x7fd8740a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:19:10.593 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:10.590+0000 7fd8979a5740 1 -- 172.21.15.118:0/2011461085 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5594786ff810 msgr2=0x5594786ffc30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:19:10.593 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:10.590+0000 7fd88de00640 1 --2- 172.21.15.118:0/2746976000 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7fd8740a2800 0x7fd8740a4cc0 secure :-1 s=READY pgs=337 cs=0 l=1 rev1=1 crypto rx=0x7fd88003d040 tx=0x7fd880005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:19:10.593 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:10.590+0000 7fd8979a5740 1 --2- 172.21.15.118:0/2011461085 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5594786ff810 0x5594786ffc30 secure :-1 s=READY pgs=379 cs=0 l=1 rev1=1 crypto rx=0x7fd88800e450 tx=0x7fd8880095c0 comp rx=0 tx=0).stop 2024-03-24T00:19:10.593 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:10.590+0000 7fd8979a5740 1 -- 172.21.15.118:0/2011461085 shutdown_connections 2024-03-24T00:19:10.593 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:10.590+0000 7fd8979a5740 1 --2- 172.21.15.118:0/2011461085 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5594787013a0 0x559478707c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:19:10.593 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:10.590+0000 7fd8979a5740 1 --2- 172.21.15.118:0/2011461085 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5594787009e0 0x559478700e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:19:10.593 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:10.590+0000 7fd8979a5740 1 --2- 172.21.15.118:0/2011461085 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5594786ff810 0x5594786ffc30 unknown :-1 s=CLOSED pgs=379 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:19:10.593 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:10.590+0000 7fd8979a5740 1 -- 172.21.15.118:0/2011461085 >> 172.21.15.118:0/2011461085 conn(0x5594786fac00 msgr2=0x5594786fb000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:19:10.593 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:10.590+0000 7fd8979a5740 1 -- 172.21.15.118:0/2011461085 shutdown_connections 2024-03-24T00:19:10.593 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:10.590+0000 7fd8979a5740 1 -- 172.21.15.118:0/2011461085 wait complete. 2024-03-24T00:19:10.594 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:10.591+0000 7fd8979a5740 1 -- start start 2024-03-24T00:19:10.594 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:10.591+0000 7fd8979a5740 1 Processor -- start 2024-03-24T00:19:10.594 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:10.591+0000 7fd8979a5740 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5594786ff810 0x559478709a60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:19:10.594 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:10.591+0000 7fd8979a5740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5594787009e0 0x559478709fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:19:10.594 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:10.591+0000 7fd8979a5740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5594787013a0 0x559478710fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:19:10.594 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:10.591+0000 7fd8979a5740 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x5594787083b0 con 0x5594786ff810 2024-03-24T00:19:10.594 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:10.591+0000 7fd8979a5740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x5594786200d0 con 0x5594787009e0 2024-03-24T00:19:10.594 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:10.591+0000 7fd8979a5740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x559478708530 con 0x5594787013a0 2024-03-24T00:19:10.594 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:10.591+0000 7fd8979a5740 10 client.0 _ll_register_callbacks cb 0x559478624e80 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T00:19:10.594 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:10.592+0000 7fd88d400640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5594787009e0 0x559478709fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:19:10.594 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:10.592+0000 7fd88d400640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5594787009e0 0x559478709fa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.118:54330/0 (socket says 172.21.15.118:54330) 2024-03-24T00:19:10.594 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:10.592+0000 7fd88d400640 1 -- 172.21.15.118:0/1413351117 learned_addr learned my addr 172.21.15.118:0/1413351117 (peer_addr_for_me v2:172.21.15.118:0/0) 2024-03-24T00:19:10.594 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:10.592+0000 7fd88e800640 1 --2- 172.21.15.118:0/1413351117 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5594786ff810 0x559478709a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:19:10.595 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:10.592+0000 7fd88d400640 1 -- 172.21.15.118:0/1413351117 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5594787013a0 msgr2=0x559478710fe0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-24T00:19:10.595 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:10.592+0000 7fd88d400640 1 --2- 172.21.15.118:0/1413351117 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5594787013a0 0x559478710fe0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:19:10.595 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:10.592+0000 7fd88d400640 1 -- 172.21.15.118:0/1413351117 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5594786ff810 msgr2=0x559478709a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:19:10.595 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:10.592+0000 7fd88d400640 1 --2- 172.21.15.118:0/1413351117 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5594786ff810 0x559478709a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:19:10.595 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:10.592+0000 7fd88d400640 1 -- 172.21.15.118:0/1413351117 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5594783783f0 con 0x5594787009e0 2024-03-24T00:19:10.595 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:10.592+0000 7fd88e800640 1 --2- 172.21.15.118:0/1413351117 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5594786ff810 0x559478709a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-24T00:19:10.595 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:10.592+0000 7fd88d400640 1 --2- 172.21.15.118:0/1413351117 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5594787009e0 0x559478709fa0 secure :-1 s=READY pgs=380 cs=0 l=1 rev1=1 crypto rx=0x7fd87c012700 tx=0x7fd87c012bc0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=490573486592bdf5 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:19:10.595 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:10.592+0000 7fd86f400640 10 client.0 ms_handle_connect on v2:172.21.15.189:3300/0 2024-03-24T00:19:10.595 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:10.592+0000 7fd86f400640 1 -- 172.21.15.118:0/1413351117 <== mon.2 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd87c011bb0 con 0x5594787009e0 2024-03-24T00:19:10.595 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:10.593+0000 7fd8979a5740 1 -- 172.21.15.118:0/1413351117 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fd88801c070 con 0x5594787009e0 2024-03-24T00:19:10.595 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:10.593+0000 7fd86f400640 1 -- 172.21.15.118:0/1413351117 <== mon.2 v2:172.21.15.189:3300/0 2 ==== osd_map(90..90 src has 1..90) v4 ==== 8277+0+0 (secure 0 0 0) 0x7fd87c011d50 con 0x5594787009e0 2024-03-24T00:19:10.595 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:10.593+0000 7fd86f400640 1 -- 172.21.15.118:0/1413351117 <== mon.2 v2:172.21.15.189:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) v2 ==== 24+0+0 (secure 0 0 0) 0x7fd87c004030 con 0x5594787009e0 2024-03-24T00:19:10.595 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:10.593+0000 7fd8979a5740 10 client.26477 fetch_fsmap learned FSMap version 28 2024-03-24T00:19:10.595 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:10.593+0000 7fd8979a5740 1 -- 172.21.15.118:0/1413351117 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) v3 -- 0x5594784427d0 con 0x5594787009e0 2024-03-24T00:19:10.596 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:10.593+0000 7fd86f400640 1 -- 172.21.15.118:0/1413351117 <== mon.2 v2:172.21.15.189:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fd88801c070 con 0x5594787009e0 2024-03-24T00:19:10.597 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:10.593+0000 7fd86f400640 1 -- 172.21.15.118:0/1413351117 <== mon.2 v2:172.21.15.189:3300/0 5 ==== fsmap(e 28) v1 ==== 7934+0+0 (secure 0 0 0) 0x7fd87c025070 con 0x5594787009e0 2024-03-24T00:19:10.598 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:10.593+0000 7fd8979a5740 10 client.26477 fetch_fsmap finished waiting for FSMap version 28 2024-03-24T00:19:10.598 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:10.593+0000 7fd8979a5740 10 client.26477 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-24T00:19:10.598 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:10.593+0000 7fd8979a5740 20 client.26477 populate_metadata read hostname 'smithi118' 2024-03-24T00:19:10.598 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:10.593+0000 7fd8979a5740 1 --2- 172.21.15.118:0/1413351117 >> [v2:172.21.15.189:6838/4074484168,v1:172.21.15.189:6839/4074484168] conn(0x559478726410 0x559478728870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:19:10.598 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:10.593+0000 7fd8979a5740 4 client.26477 mds_command: new command op to 24569 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-24T00:19:10.598 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:10.593+0000 7fd8979a5740 1 -- 172.21.15.118:0/1413351117 --> [v2:172.21.15.189:6838/4074484168,v1:172.21.15.189:6839/4074484168] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x559478671800 con 0x559478726410 2024-03-24T00:19:10.598 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:10.594+0000 7fd88e800640 1 --2- 172.21.15.118:0/1413351117 >> [v2:172.21.15.189:6838/4074484168,v1:172.21.15.189:6839/4074484168] conn(0x559478726410 0x559478728870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:19:10.598 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:10.594+0000 7fd88e800640 1 --2- 172.21.15.118:0/1413351117 >> [v2:172.21.15.189:6838/4074484168,v1:172.21.15.189:6839/4074484168] conn(0x559478726410 0x559478728870 crc :-1 s=READY pgs=109 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=20a373dc1389b65f server_cookie=d5f19bed7aa50718 in_seq=0 out_seq=0 2024-03-24T00:19:10.598 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:10.594+0000 7fd86f400640 10 client.26477 ms_handle_connect on v2:172.21.15.189:6838/4074484168 2024-03-24T00:19:10.598 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:10.597+0000 7fd86f400640 1 -- 172.21.15.118:0/1413351117 <== mds.0 v2:172.21.15.189:6838/4074484168 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x559478671800 con 0x559478726410 2024-03-24T00:19:10.598 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:10.597+0000 7fd86f400640 10 client.26477 handle_command_reply: tid=0 2024-03-24T00:19:10.598 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:10.597+0000 7fd8979a5740 1 client.26477 shutdown 2024-03-24T00:19:10.598 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:10.597+0000 7fd8979a5740 1 -- 172.21.15.118:0/1413351117 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5594787009e0 msgr2=0x559478709fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:19:10.599 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:10.597+0000 7fd8979a5740 1 --2- 172.21.15.118:0/1413351117 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5594787009e0 0x559478709fa0 secure :-1 s=READY pgs=380 cs=0 l=1 rev1=1 crypto rx=0x7fd87c012700 tx=0x7fd87c012bc0 comp rx=0 tx=0).stop 2024-03-24T00:19:10.599 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:10.597+0000 7fd8979a5740 1 -- 172.21.15.118:0/1413351117 shutdown_connections 2024-03-24T00:19:10.599 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:10.597+0000 7fd8979a5740 1 --2- 172.21.15.118:0/1413351117 >> [v2:172.21.15.189:6838/4074484168,v1:172.21.15.189:6839/4074484168] conn(0x559478726410 0x559478728870 crc :-1 s=READY pgs=109 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:19:10.599 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:10.597+0000 7fd8979a5740 1 --2- 172.21.15.118:0/1413351117 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5594787013a0 0x559478710fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:19:10.599 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:10.597+0000 7fd8979a5740 1 --2- 172.21.15.118:0/1413351117 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5594787009e0 0x559478709fa0 unknown :-1 s=CLOSED pgs=380 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:19:10.599 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:10.597+0000 7fd8979a5740 1 --2- 172.21.15.118:0/1413351117 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5594786ff810 0x559478709a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:19:10.599 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:10.597+0000 7fd8979a5740 1 -- 172.21.15.118:0/1413351117 >> 172.21.15.118:0/1413351117 conn(0x5594786fac00 msgr2=0x5594786f9eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:19:10.599 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:10.597+0000 7fd86f400640 0 client.26477 ms_handle_reset on v2:172.21.15.189:6838/4074484168 2024-03-24T00:19:10.599 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:10.597+0000 7fd8979a5740 1 -- 172.21.15.118:0/1413351117 shutdown_connections 2024-03-24T00:19:10.599 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:10.597+0000 7fd8979a5740 1 -- 172.21.15.118:0/1413351117 wait complete. 2024-03-24T00:19:10.599 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:10.597+0000 7fd8979a5740 20 client.26477 trim_cache size 0 max 16384 2024-03-24T00:19:10.614 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:10.614+0000 7fd8979a5740 1 Processor -- start 2024-03-24T00:19:10.614 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:10.615+0000 7fd8979a5740 1 -- start start 2024-03-24T00:19:10.615 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:10.615+0000 7fd8979a5740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5594786ff810 0x5594786fe570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:19:10.615 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:10.615+0000 7fd8979a5740 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5594787009e0 0x559478710fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:19:10.615 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:10.615+0000 7fd8979a5740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5594787013a0 0x559478703c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:19:10.615 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:10.615+0000 7fd8979a5740 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd8901a2430 con 0x5594787009e0 2024-03-24T00:19:10.615 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:10.615+0000 7fd8979a5740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd8901a3db0 con 0x5594787013a0 2024-03-24T00:19:10.615 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:10.615+0000 7fd8979a5740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd8901a2a90 con 0x5594786ff810 2024-03-24T00:19:10.616 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:10.615+0000 7fd88e800640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5594786ff810 0x5594786fe570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:19:10.616 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:10.615+0000 7fd88d400640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5594787009e0 0x559478710fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:19:10.616 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:10.615+0000 7fd88e800640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5594786ff810 0x5594786fe570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.118:47464/0 (socket says 172.21.15.118:47464) 2024-03-24T00:19:10.616 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:10.615+0000 7fd88e800640 1 -- 172.21.15.118:0/2515818921 learned_addr learned my addr 172.21.15.118:0/2515818921 (peer_addr_for_me v2:172.21.15.118:0/0) 2024-03-24T00:19:10.616 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:10.615+0000 7fd88e800640 1 --2- 172.21.15.118:0/2515818921 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5594787013a0 0x559478703c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:19:10.616 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:10.616+0000 7fd88e800640 1 -- 172.21.15.118:0/2515818921 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5594787013a0 msgr2=0x559478703c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:19:10.616 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:10.616+0000 7fd88e800640 1 --2- 172.21.15.118:0/2515818921 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5594787013a0 0x559478703c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:19:10.617 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:10.616+0000 7fd88e800640 1 -- 172.21.15.118:0/2515818921 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5594787009e0 msgr2=0x559478710fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:19:10.617 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:10.616+0000 7fd88e800640 1 --2- 172.21.15.118:0/2515818921 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5594787009e0 0x559478710fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:19:10.617 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:10.616+0000 7fd88e800640 1 -- 172.21.15.118:0/2515818921 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5594786fb670 con 0x5594786ff810 2024-03-24T00:19:10.617 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:10.616+0000 7fd88d400640 1 --2- 172.21.15.118:0/2515818921 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5594787009e0 0x559478710fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T00:19:10.617 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:10.616+0000 7fd88e800640 1 --2- 172.21.15.118:0/2515818921 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5594786ff810 0x5594786fe570 secure :-1 s=READY pgs=382 cs=0 l=1 rev1=1 crypto rx=0x7fd888009c50 tx=0x7fd888005fd0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=9bffba05f41f434b server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:19:10.617 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:10.616+0000 7fd87a600640 1 -- 172.21.15.118:0/2515818921 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd888004d00 con 0x5594786ff810 2024-03-24T00:19:10.617 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:10.616+0000 7fd87a600640 1 -- 172.21.15.118:0/2515818921 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fd88801c040 con 0x5594786ff810 2024-03-24T00:19:10.617 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:10.616+0000 7fd87a600640 1 -- 172.21.15.118:0/2515818921 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd88800d710 con 0x5594786ff810 2024-03-24T00:19:10.620 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:10.617+0000 7fd8979a5740 1 -- 172.21.15.118:0/2515818921 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5594786ff810 msgr2=0x5594786fe570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:19:10.620 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:10.617+0000 7fd8979a5740 1 --2- 172.21.15.118:0/2515818921 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5594786ff810 0x5594786fe570 secure :-1 s=READY pgs=382 cs=0 l=1 rev1=1 crypto rx=0x7fd888009c50 tx=0x7fd888005fd0 comp rx=0 tx=0).stop 2024-03-24T00:19:10.620 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:10.617+0000 7fd8979a5740 1 -- 172.21.15.118:0/2515818921 shutdown_connections 2024-03-24T00:19:10.620 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:10.617+0000 7fd8979a5740 1 --2- 172.21.15.118:0/2515818921 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5594787013a0 0x559478703c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:19:10.620 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:10.617+0000 7fd8979a5740 1 --2- 172.21.15.118:0/2515818921 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5594787009e0 0x559478710fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:19:10.620 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:10.617+0000 7fd8979a5740 1 --2- 172.21.15.118:0/2515818921 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5594786ff810 0x5594786fe570 unknown :-1 s=CLOSED pgs=382 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:19:10.620 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:10.617+0000 7fd8979a5740 1 -- 172.21.15.118:0/2515818921 >> 172.21.15.118:0/2515818921 conn(0x5594786fac00 msgr2=0x5594786fde60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:19:10.620 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:10.617+0000 7fd8979a5740 1 -- 172.21.15.118:0/2515818921 shutdown_connections 2024-03-24T00:19:10.620 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:10.617+0000 7fd8979a5740 1 -- 172.21.15.118:0/2515818921 wait complete. 2024-03-24T00:19:10.620 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:10.618+0000 7fd8979a5740 1 -- start start 2024-03-24T00:19:10.620 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:10.618+0000 7fd8979a5740 1 Processor -- start 2024-03-24T00:19:10.620 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:10.619+0000 7fd8979a5740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5594786ff810 0x55947870ea70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:19:10.621 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:10.619+0000 7fd8979a5740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5594787009e0 0x55947870efb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:19:10.621 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:10.619+0000 7fd8979a5740 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5594787013a0 0x55947870ffc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:19:10.621 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:10.619+0000 7fd8979a5740 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd8901a3750 con 0x5594787013a0 2024-03-24T00:19:10.621 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:10.619+0000 7fd8979a5740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd8901a30f0 con 0x5594787009e0 2024-03-24T00:19:10.621 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:10.619+0000 7fd8979a5740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd890071d20 con 0x5594786ff810 2024-03-24T00:19:10.621 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:10.619+0000 7fd8979a5740 10 client.0 _ll_register_callbacks cb 0x7fd87c009840 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T00:19:10.621 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:10.619+0000 7fd88d400640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5594787009e0 0x55947870efb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:19:10.621 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:10.619+0000 7fd88e800640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5594786ff810 0x55947870ea70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:19:10.621 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:10.619+0000 7fd88d400640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5594787009e0 0x55947870efb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.118:54354/0 (socket says 172.21.15.118:54354) 2024-03-24T00:19:10.621 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:10.619+0000 7fd88d400640 1 -- 172.21.15.118:0/112712859 learned_addr learned my addr 172.21.15.118:0/112712859 (peer_addr_for_me v2:172.21.15.118:0/0) 2024-03-24T00:19:10.621 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:10.619+0000 7fd88e800640 1 --2- 172.21.15.118:0/112712859 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5594787013a0 0x55947870ffc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:19:10.621 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:10.619+0000 7fd88d400640 1 -- 172.21.15.118:0/112712859 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5594786ff810 msgr2=0x55947870ea70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:19:10.621 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:10.619+0000 7fd88d400640 1 --2- 172.21.15.118:0/112712859 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5594786ff810 0x55947870ea70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:19:10.622 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:10.619+0000 7fd88d400640 1 -- 172.21.15.118:0/112712859 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5594787013a0 msgr2=0x55947870ffc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:19:10.622 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:10.619+0000 7fd88d400640 1 --2- 172.21.15.118:0/112712859 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5594787013a0 0x55947870ffc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:19:10.622 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:10.619+0000 7fd88d400640 1 -- 172.21.15.118:0/112712859 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5594787105b0 con 0x5594787009e0 2024-03-24T00:19:10.622 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:10.619+0000 7fd88e800640 1 --2- 172.21.15.118:0/112712859 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5594786ff810 0x55947870ea70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T00:19:10.622 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:10.620+0000 7fd88d400640 1 --2- 172.21.15.118:0/112712859 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5594787009e0 0x55947870efb0 secure :-1 s=READY pgs=381 cs=0 l=1 rev1=1 crypto rx=0x7fd87c0095a0 tx=0x7fd87c0054e0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=5e98aa58f74e305b server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:19:10.622 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:10.620+0000 7fd86f400640 10 client.0 ms_handle_connect on v2:172.21.15.189:3300/0 2024-03-24T00:19:10.622 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:10.620+0000 7fd86f400640 1 -- 172.21.15.118:0/112712859 <== mon.2 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd87c000e60 con 0x5594787009e0 2024-03-24T00:19:10.622 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:10.620+0000 7fd8979a5740 1 -- 172.21.15.118:0/112712859 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fd88801c040 con 0x5594787009e0 2024-03-24T00:19:10.622 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:10.620+0000 7fd86f400640 1 -- 172.21.15.118:0/112712859 <== mon.2 v2:172.21.15.189:3300/0 2 ==== osd_map(90..90 src has 1..90) v4 ==== 8277+0+0 (secure 0 0 0) 0x7fd87c018070 con 0x5594787009e0 2024-03-24T00:19:10.622 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:10.620+0000 7fd86f400640 1 -- 172.21.15.118:0/112712859 <== mon.2 v2:172.21.15.189:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) v2 ==== 24+0+0 (secure 0 0 0) 0x7fd87c011dc0 con 0x5594787009e0 2024-03-24T00:19:10.622 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:10.620+0000 7fd8979a5740 10 client.26483 fetch_fsmap learned FSMap version 28 2024-03-24T00:19:10.622 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:10.620+0000 7fd8979a5740 1 -- 172.21.15.118:0/112712859 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) v3 -- 0x559478782150 con 0x5594787009e0 2024-03-24T00:19:10.624 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:10.621+0000 7fd86f400640 1 -- 172.21.15.118:0/112712859 <== mon.2 v2:172.21.15.189:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fd88801c040 con 0x5594787009e0 2024-03-24T00:19:10.624 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:10.621+0000 7fd86f400640 1 -- 172.21.15.118:0/112712859 <== mon.2 v2:172.21.15.189:3300/0 5 ==== fsmap(e 28) v1 ==== 7934+0+0 (secure 0 0 0) 0x7fd87c03b070 con 0x5594787009e0 2024-03-24T00:19:10.624 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:10.621+0000 7fd8979a5740 10 client.26483 fetch_fsmap finished waiting for FSMap version 28 2024-03-24T00:19:10.624 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:10.621+0000 7fd8979a5740 10 client.26483 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-24T00:19:10.624 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:10.621+0000 7fd8979a5740 20 client.26483 populate_metadata read hostname 'smithi118' 2024-03-24T00:19:10.624 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:10.621+0000 7fd8979a5740 1 --2- 172.21.15.118:0/112712859 >> [v2:172.21.15.189:6838/4074484168,v1:172.21.15.189:6839/4074484168] conn(0x559478782a20 0x559478784e40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:19:10.624 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:10.621+0000 7fd8979a5740 4 client.26483 mds_command: new command op to 24569 tid=0 multi_id=0 [{"prefix": "damage ls"}] 2024-03-24T00:19:10.625 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:10.621+0000 7fd8979a5740 1 -- 172.21.15.118:0/112712859 --> [v2:172.21.15.189:6838/4074484168,v1:172.21.15.189:6839/4074484168] -- command(tid 0: {"prefix": "damage ls"}) v1 -- 0x55947866bb50 con 0x559478782a20 2024-03-24T00:19:10.625 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:10.621+0000 7fd88e800640 1 --2- 172.21.15.118:0/112712859 >> [v2:172.21.15.189:6838/4074484168,v1:172.21.15.189:6839/4074484168] conn(0x559478782a20 0x559478784e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:19:10.625 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:10.622+0000 7fd88e800640 1 --2- 172.21.15.118:0/112712859 >> [v2:172.21.15.189:6838/4074484168,v1:172.21.15.189:6839/4074484168] conn(0x559478782a20 0x559478784e40 crc :-1 s=READY pgs=110 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=407088c6b6866593 server_cookie=3a95d3d81ab7c57a in_seq=0 out_seq=0 2024-03-24T00:19:10.625 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:10.622+0000 7fd86f400640 10 client.26483 ms_handle_connect on v2:172.21.15.189:6838/4074484168 2024-03-24T00:19:10.625 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:10.623+0000 7fd86f400640 1 -- 172.21.15.118:0/112712859 <== mds.0 v2:172.21.15.189:6838/4074484168 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+3 (crc 0 0 0) 0x55947866bb50 con 0x559478782a20 2024-03-24T00:19:10.625 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:10.623+0000 7fd86f400640 10 client.26483 handle_command_reply: tid=0 2024-03-24T00:19:10.625 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:10.623+0000 7fd8979a5740 1 client.26483 shutdown 2024-03-24T00:19:10.625 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:10.623+0000 7fd8979a5740 1 -- 172.21.15.118:0/112712859 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5594787009e0 msgr2=0x55947870efb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:19:10.625 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:10.623+0000 7fd8979a5740 1 --2- 172.21.15.118:0/112712859 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5594787009e0 0x55947870efb0 secure :-1 s=READY pgs=381 cs=0 l=1 rev1=1 crypto rx=0x7fd87c0095a0 tx=0x7fd87c0054e0 comp rx=0 tx=0).stop 2024-03-24T00:19:10.625 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:10.623+0000 7fd8979a5740 1 -- 172.21.15.118:0/112712859 shutdown_connections 2024-03-24T00:19:10.625 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:10.623+0000 7fd8979a5740 1 --2- 172.21.15.118:0/112712859 >> [v2:172.21.15.189:6838/4074484168,v1:172.21.15.189:6839/4074484168] conn(0x559478782a20 0x559478784e40 crc :-1 s=READY pgs=110 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:19:10.625 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:10.623+0000 7fd8979a5740 1 --2- 172.21.15.118:0/112712859 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5594787013a0 0x55947870ffc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:19:10.625 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:10.623+0000 7fd8979a5740 1 --2- 172.21.15.118:0/112712859 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5594787009e0 0x55947870efb0 unknown :-1 s=CLOSED pgs=381 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:19:10.625 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:10.623+0000 7fd8979a5740 1 --2- 172.21.15.118:0/112712859 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5594786ff810 0x55947870ea70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:19:10.626 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:10.623+0000 7fd86f400640 0 client.26483 ms_handle_reset on v2:172.21.15.189:6838/4074484168 2024-03-24T00:19:10.626 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:10.623+0000 7fd8979a5740 1 -- 172.21.15.118:0/112712859 >> 172.21.15.118:0/112712859 conn(0x5594786fac00 msgr2=0x5594786fe6f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:19:10.626 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:10.624+0000 7fd8979a5740 1 -- 172.21.15.118:0/112712859 shutdown_connections 2024-03-24T00:19:10.626 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:10.624+0000 7fd8979a5740 1 -- 172.21.15.118:0/112712859 wait complete. 2024-03-24T00:19:10.626 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:10.624+0000 7fd8979a5740 20 client.26483 trim_cache size 0 max 16384 2024-03-24T00:19:10.626 INFO:teuthology.orchestra.run.smithi118.stdout:[] 2024-03-24T00:19:10.627 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:10.624+0000 7fd894c00640 1 -- 172.21.15.118:0/2746976000 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7fd8740a2800 msgr2=0x7fd8740a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:19:10.627 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:10.624+0000 7fd894c00640 1 --2- 172.21.15.118:0/2746976000 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7fd8740a2800 0x7fd8740a4cc0 secure :-1 s=READY pgs=337 cs=0 l=1 rev1=1 crypto rx=0x7fd88003d040 tx=0x7fd880005f90 comp rx=0 tx=0).stop 2024-03-24T00:19:10.627 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:10.624+0000 7fd894c00640 1 -- 172.21.15.118:0/2746976000 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fd8901016b0 msgr2=0x7fd890199c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:19:10.627 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:10.624+0000 7fd894c00640 1 --2- 172.21.15.118:0/2746976000 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fd8901016b0 0x7fd890199c30 secure :-1 s=READY pgs=381 cs=0 l=1 rev1=1 crypto rx=0x7fd8800349e0 tx=0x7fd880032430 comp rx=0 tx=0).stop 2024-03-24T00:19:10.627 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:10.624+0000 7fd894c00640 1 -- 172.21.15.118:0/2746976000 shutdown_connections 2024-03-24T00:19:10.627 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:10.624+0000 7fd894c00640 1 --2- 172.21.15.118:0/2746976000 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7fd8740a2800 0x7fd8740a4cc0 unknown :-1 s=CLOSED pgs=337 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:19:10.627 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:10.624+0000 7fd894c00640 1 --2- 172.21.15.118:0/2746976000 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd890103260 0x7fd8901a11a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:19:10.627 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:10.624+0000 7fd894c00640 1 --2- 172.21.15.118:0/2746976000 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fd8901028a0 0x7fd89019a170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:19:10.628 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:10.624+0000 7fd894c00640 1 --2- 172.21.15.118:0/2746976000 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fd8901016b0 0x7fd890199c30 unknown :-1 s=CLOSED pgs=381 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:19:10.628 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:10.624+0000 7fd894c00640 1 -- 172.21.15.118:0/2746976000 >> 172.21.15.118:0/2746976000 conn(0x7fd8900fcc40 msgr2=0x7fd8901080c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:19:10.628 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:10.625+0000 7fd894c00640 1 -- 172.21.15.118:0/2746976000 shutdown_connections 2024-03-24T00:19:10.628 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:10.625+0000 7fd894c00640 1 -- 172.21.15.118:0/2746976000 wait complete. 2024-03-24T00:19:10.703 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:19:10 smithi189 ceph-mon[36187]: pgmap v502: 97 pgs: 97 active+clean; 64 MiB data, 572 MiB used, 1.0 TiB / 1.0 TiB avail; 683 KiB/s rd, 705 KiB/s wr, 194 op/s 2024-03-24T00:19:10.710 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:19:10 smithi193 ceph-mon[36878]: pgmap v502: 97 pgs: 97 active+clean; 64 MiB data, 572 MiB used, 1.0 TiB / 1.0 TiB avail; 683 KiB/s rd, 705 KiB/s wr, 194 op/s 2024-03-24T00:19:11.643 INFO:tasks.fwd_scrub.fs.[cephfs]:scrubbing fs: cephfs 2024-03-24T00:19:11.643 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T00:19:11.643 DEBUG:teuthology.orchestra.run.smithi118:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 300 ceph --cluster ceph tell mds.1:0 scrub start / force,recursive 2024-03-24T00:19:11.861 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:11.860+0000 7fc8e1800640 1 -- 172.21.15.118:0/906364589 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc8dc1037a0 msgr2=0x7fc8dc103bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:19:11.861 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:11.860+0000 7fc8e1800640 1 --2- 172.21.15.118:0/906364589 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc8dc1037a0 0x7fc8dc103bc0 secure :-1 s=READY pgs=382 cs=0 l=1 rev1=1 crypto rx=0x7fc8cc009900 tx=0x7fc8cc02bc40 comp rx=0 tx=0).stop 2024-03-24T00:19:11.861 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:11.861+0000 7fc8e1800640 1 -- 172.21.15.118:0/906364589 shutdown_connections 2024-03-24T00:19:11.861 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:11.861+0000 7fc8e1800640 1 --2- 172.21.15.118:0/906364589 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fc8dc105350 0x7fc8dc10bbe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:19:11.861 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:11.861+0000 7fc8e1800640 1 --2- 172.21.15.118:0/906364589 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fc8dc104990 0x7fc8dc104e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:19:11.861 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:11.861+0000 7fc8e1800640 1 --2- 172.21.15.118:0/906364589 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc8dc1037a0 0x7fc8dc103bc0 unknown :-1 s=CLOSED pgs=382 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:19:11.861 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:11.861+0000 7fc8e1800640 1 -- 172.21.15.118:0/906364589 >> 172.21.15.118:0/906364589 conn(0x7fc8dc0fed10 msgr2=0x7fc8dc101150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:19:11.861 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:11.861+0000 7fc8e1800640 1 -- 172.21.15.118:0/906364589 shutdown_connections 2024-03-24T00:19:11.861 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:11.861+0000 7fc8e1800640 1 -- 172.21.15.118:0/906364589 wait complete. 2024-03-24T00:19:11.863 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:11.862+0000 7fc8e1800640 1 Processor -- start 2024-03-24T00:19:11.863 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:11.862+0000 7fc8e1800640 1 -- start start 2024-03-24T00:19:11.863 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:11.862+0000 7fc8e1800640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc8dc1037a0 0x7fc8dc199ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:19:11.863 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:11.862+0000 7fc8e1800640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fc8dc104990 0x7fc8dc19a020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:19:11.863 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:11.862+0000 7fc8e1800640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fc8dc105350 0x7fc8dc1a1090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:19:11.863 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:11.862+0000 7fc8e1800640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc8dc10f470 con 0x7fc8dc105350 2024-03-24T00:19:11.863 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:11.862+0000 7fc8e1800640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc8dc10f2f0 con 0x7fc8dc1037a0 2024-03-24T00:19:11.863 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:11.862+0000 7fc8e1800640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc8dc10f5f0 con 0x7fc8dc104990 2024-03-24T00:19:11.863 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:11.862+0000 7fc8dbe00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc8dc1037a0 0x7fc8dc199ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:19:11.863 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:11.863+0000 7fc8db400640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fc8dc104990 0x7fc8dc19a020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:19:11.863 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:11.863+0000 7fc8dbe00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc8dc1037a0 0x7fc8dc199ae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.118:54378/0 (socket says 172.21.15.118:54378) 2024-03-24T00:19:11.864 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:11.863+0000 7fc8db400640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fc8dc104990 0x7fc8dc19a020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.118:47484/0 (socket says 172.21.15.118:47484) 2024-03-24T00:19:11.864 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:11.863+0000 7fc8dbe00640 1 -- 172.21.15.118:0/1067441042 learned_addr learned my addr 172.21.15.118:0/1067441042 (peer_addr_for_me v2:172.21.15.118:0/0) 2024-03-24T00:19:11.865 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:11.863+0000 7fc8e0e00640 1 --2- 172.21.15.118:0/1067441042 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fc8dc105350 0x7fc8dc1a1090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:19:11.865 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:11.863+0000 7fc8dbe00640 1 -- 172.21.15.118:0/1067441042 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fc8dc104990 msgr2=0x7fc8dc19a020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:19:11.865 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:11.863+0000 7fc8dbe00640 1 --2- 172.21.15.118:0/1067441042 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fc8dc104990 0x7fc8dc19a020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:19:11.865 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:11.863+0000 7fc8dbe00640 1 -- 172.21.15.118:0/1067441042 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fc8dc105350 msgr2=0x7fc8dc1a1090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:19:11.865 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:11.863+0000 7fc8dbe00640 1 --2- 172.21.15.118:0/1067441042 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fc8dc105350 0x7fc8dc1a1090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:19:11.865 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:11.863+0000 7fc8dbe00640 1 -- 172.21.15.118:0/1067441042 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc8dc1a1700 con 0x7fc8dc1037a0 2024-03-24T00:19:11.867 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:11.863+0000 7fc8db400640 1 --2- 172.21.15.118:0/1067441042 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fc8dc104990 0x7fc8dc19a020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T00:19:11.867 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:11.863+0000 7fc8e0e00640 1 --2- 172.21.15.118:0/1067441042 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fc8dc105350 0x7fc8dc1a1090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T00:19:11.867 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:11.863+0000 7fc8dbe00640 1 --2- 172.21.15.118:0/1067441042 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc8dc1037a0 0x7fc8dc199ae0 secure :-1 s=READY pgs=383 cs=0 l=1 rev1=1 crypto rx=0x7fc8cc02efd0 tx=0x7fc8cc02c660 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:19:11.867 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:11.864+0000 7fc8d8c00640 1 -- 172.21.15.118:0/1067441042 <== mon.2 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc8cc041070 con 0x7fc8dc1037a0 2024-03-24T00:19:11.867 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:11.864+0000 7fc8e1800640 1 -- 172.21.15.118:0/1067441042 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc8dc1a1990 con 0x7fc8dc1037a0 2024-03-24T00:19:11.867 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:11.864+0000 7fc8d8c00640 1 -- 172.21.15.118:0/1067441042 <== mon.2 v2:172.21.15.189:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fc8cc004450 con 0x7fc8dc1037a0 2024-03-24T00:19:11.867 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:11.864+0000 7fc8d8c00640 1 -- 172.21.15.118:0/1067441042 <== mon.2 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc8cc003e50 con 0x7fc8dc1037a0 2024-03-24T00:19:11.867 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:11.864+0000 7fc8e1800640 1 -- 172.21.15.118:0/1067441042 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc8dc1a1ea0 con 0x7fc8dc1037a0 2024-03-24T00:19:11.867 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:11.867+0000 7fc8d8c00640 1 -- 172.21.15.118:0/1067441042 <== mon.2 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 20) v1 ==== 137291+0+0 (secure 0 0 0) 0x7fc8cc044070 con 0x7fc8dc1037a0 2024-03-24T00:19:11.869 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:11.868+0000 7fc8e5b88740 1 Processor -- start 2024-03-24T00:19:11.870 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:11.868+0000 7fc8e5b88740 1 -- start start 2024-03-24T00:19:11.870 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:11.868+0000 7fc8e5b88740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56322d94d2b0 0x56322d94d6d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:19:11.870 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:11.869+0000 7fc8e5b88740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56322d94dca0 0x56322d9527a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:19:11.870 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:11.869+0000 7fc8e5b88740 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x56322d952e10 0x56322d9552a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:19:11.870 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:11.869+0000 7fc8d8c00640 1 --2- 172.21.15.118:0/1067441042 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7fc8c00a2a20 0x7fc8c00a4ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:19:11.870 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:11.869+0000 7fc8e5b88740 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x56322d8b2eb0 con 0x56322d952e10 2024-03-24T00:19:11.870 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:11.869+0000 7fc8e5b88740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x56322d85ba60 con 0x56322d94d2b0 2024-03-24T00:19:11.870 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:11.869+0000 7fc8e5b88740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x56322d8aef50 con 0x56322d94dca0 2024-03-24T00:19:11.870 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:11.869+0000 7fc8dbe00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56322d94dca0 0x56322d9527a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:19:11.870 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:11.869+0000 7fc8d8c00640 1 -- 172.21.15.118:0/1067441042 <== mon.2 v2:172.21.15.189:3300/0 5 ==== osd_map(90..90 src has 1..90) v4 ==== 8277+0+0 (secure 0 0 0) 0x7fc8cc0f05b0 con 0x7fc8dc1037a0 2024-03-24T00:19:11.870 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:11.869+0000 7fc8e0e00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56322d94d2b0 0x56322d94d6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:19:11.870 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:11.869+0000 7fc8dbe00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56322d94dca0 0x56322d9527a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.118:47492/0 (socket says 172.21.15.118:47492) 2024-03-24T00:19:11.870 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:11.869+0000 7fc8dbe00640 1 -- 172.21.15.118:0/58472854 learned_addr learned my addr 172.21.15.118:0/58472854 (peer_addr_for_me v2:172.21.15.118:0/0) 2024-03-24T00:19:11.871 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:11.869+0000 7fc8e0e00640 1 --2- 172.21.15.118:0/58472854 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x56322d952e10 0x56322d9552a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:19:11.871 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:11.869+0000 7fc8db400640 1 --2- 172.21.15.118:0/1067441042 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7fc8c00a2a20 0x7fc8c00a4ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:19:11.871 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:11.869+0000 7fc8dbe00640 1 -- 172.21.15.118:0/58472854 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56322d94d2b0 msgr2=0x56322d94d6d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:19:11.871 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:11.869+0000 7fc8dbe00640 1 --2- 172.21.15.118:0/58472854 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56322d94d2b0 0x56322d94d6d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:19:11.871 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:11.869+0000 7fc8dbe00640 1 -- 172.21.15.118:0/58472854 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x56322d952e10 msgr2=0x56322d9552a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:19:11.871 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:11.869+0000 7fc8dbe00640 1 --2- 172.21.15.118:0/58472854 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x56322d952e10 0x56322d9552a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:19:11.871 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:11.869+0000 7fc8dbe00640 1 -- 172.21.15.118:0/58472854 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x56322d68f7e0 con 0x56322d94dca0 2024-03-24T00:19:11.871 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:11.869+0000 7fc8e0e00640 1 --2- 172.21.15.118:0/58472854 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x56322d952e10 0x56322d9552a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T00:19:11.871 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:11.870+0000 7fc8dbe00640 1 --2- 172.21.15.118:0/58472854 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56322d94dca0 0x56322d9527a0 secure :-1 s=READY pgs=383 cs=0 l=1 rev1=1 crypto rx=0x7fc8cc006010 tx=0x7fc8cc005e50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=7be8e1c8fcdce672 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:19:11.871 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:11.870+0000 7fc8db400640 1 --2- 172.21.15.118:0/1067441042 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7fc8c00a2a20 0x7fc8c00a4ee0 secure :-1 s=READY pgs=338 cs=0 l=1 rev1=1 crypto rx=0x7fc8c80095f0 tx=0x7fc8c8005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:19:11.871 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:11.870+0000 7fc8d1600640 1 -- 172.21.15.118:0/58472854 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc8cc04b040 con 0x56322d94dca0 2024-03-24T00:19:11.871 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:11.870+0000 7fc8d1600640 1 -- 172.21.15.118:0/58472854 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fc8cc004c60 con 0x56322d94dca0 2024-03-24T00:19:11.871 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:11.870+0000 7fc8d1600640 1 -- 172.21.15.118:0/58472854 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc8cc0035f0 con 0x56322d94dca0 2024-03-24T00:19:11.875 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:11.871+0000 7fc8e5b88740 1 -- 172.21.15.118:0/58472854 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56322d94dca0 msgr2=0x56322d9527a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:19:11.875 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:11.871+0000 7fc8e5b88740 1 --2- 172.21.15.118:0/58472854 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56322d94dca0 0x56322d9527a0 secure :-1 s=READY pgs=383 cs=0 l=1 rev1=1 crypto rx=0x7fc8cc006010 tx=0x7fc8cc005e50 comp rx=0 tx=0).stop 2024-03-24T00:19:11.875 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:11.871+0000 7fc8e5b88740 1 -- 172.21.15.118:0/58472854 shutdown_connections 2024-03-24T00:19:11.875 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:11.871+0000 7fc8e5b88740 1 --2- 172.21.15.118:0/58472854 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x56322d952e10 0x56322d9552a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:19:11.875 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:11.871+0000 7fc8e5b88740 1 --2- 172.21.15.118:0/58472854 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56322d94dca0 0x56322d9527a0 unknown :-1 s=CLOSED pgs=383 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:19:11.875 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:11.871+0000 7fc8e5b88740 1 --2- 172.21.15.118:0/58472854 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56322d94d2b0 0x56322d94d6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:19:11.875 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:11.871+0000 7fc8e5b88740 1 -- 172.21.15.118:0/58472854 >> 172.21.15.118:0/58472854 conn(0x56322d948820 msgr2=0x56322d948c20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:19:11.875 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:11.871+0000 7fc8e5b88740 1 -- 172.21.15.118:0/58472854 shutdown_connections 2024-03-24T00:19:11.875 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:11.871+0000 7fc8e5b88740 1 -- 172.21.15.118:0/58472854 wait complete. 2024-03-24T00:19:11.875 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:11.871+0000 7fc8e5b88740 1 -- start start 2024-03-24T00:19:11.876 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:11.872+0000 7fc8e5b88740 1 Processor -- start 2024-03-24T00:19:11.876 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:11.872+0000 7fc8e5b88740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56322d94d2b0 0x56322d957160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:19:11.876 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:11.872+0000 7fc8e5b88740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56322d94dca0 0x56322d9576a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:19:11.876 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:11.872+0000 7fc8e5b88740 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x56322d952e10 0x56322d95e6e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:19:11.876 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:11.872+0000 7fc8e5b88740 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x56322d955b80 con 0x56322d952e10 2024-03-24T00:19:11.876 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:11.872+0000 7fc8e5b88740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x56322d86eeb0 con 0x56322d94dca0 2024-03-24T00:19:11.876 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:11.872+0000 7fc8e5b88740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x56322d955d00 con 0x56322d94d2b0 2024-03-24T00:19:11.876 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:11.872+0000 7fc8e5b88740 10 client.0 _ll_register_callbacks cb 0x56322d863150 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T00:19:11.876 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:11.872+0000 7fc8db400640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56322d94dca0 0x56322d9576a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:19:11.876 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:11.872+0000 7fc8db400640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56322d94dca0 0x56322d9576a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.118:54382/0 (socket says 172.21.15.118:54382) 2024-03-24T00:19:11.876 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:11.872+0000 7fc8db400640 1 -- 172.21.15.118:0/1998722774 learned_addr learned my addr 172.21.15.118:0/1998722774 (peer_addr_for_me v2:172.21.15.118:0/0) 2024-03-24T00:19:11.876 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:11.872+0000 7fc8e0e00640 1 --2- 172.21.15.118:0/1998722774 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x56322d952e10 0x56322d95e6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:19:11.876 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:11.872+0000 7fc8dbe00640 1 --2- 172.21.15.118:0/1998722774 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56322d94d2b0 0x56322d957160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:19:11.876 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:11.873+0000 7fc8db400640 1 -- 172.21.15.118:0/1998722774 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56322d94d2b0 msgr2=0x56322d957160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:19:11.877 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:11.873+0000 7fc8db400640 1 --2- 172.21.15.118:0/1998722774 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56322d94d2b0 0x56322d957160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:19:11.877 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:11.873+0000 7fc8db400640 1 -- 172.21.15.118:0/1998722774 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x56322d952e10 msgr2=0x56322d95e6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:19:11.877 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:11.873+0000 7fc8db400640 1 --2- 172.21.15.118:0/1998722774 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x56322d952e10 0x56322d95e6e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:19:11.877 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:11.873+0000 7fc8db400640 1 -- 172.21.15.118:0/1998722774 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x56322d6eff70 con 0x56322d94dca0 2024-03-24T00:19:11.877 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:11.873+0000 7fc8db400640 1 --2- 172.21.15.118:0/1998722774 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56322d94dca0 0x56322d9576a0 secure :-1 s=READY pgs=384 cs=0 l=1 rev1=1 crypto rx=0x7fc8c800fcf0 tx=0x7fc8c8004730 comp rx=0 tx=0).ready entity=mon.2 client_cookie=15fda72627db1b31 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:19:11.877 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:11.873+0000 7fc8c6000640 10 client.0 ms_handle_connect on v2:172.21.15.189:3300/0 2024-03-24T00:19:11.877 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:11.873+0000 7fc8c6000640 1 -- 172.21.15.118:0/1998722774 <== mon.2 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc8c800fea0 con 0x56322d94dca0 2024-03-24T00:19:11.877 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:11.873+0000 7fc8e5b88740 1 -- 172.21.15.118:0/1998722774 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fc8cc004c60 con 0x56322d94dca0 2024-03-24T00:19:11.877 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:11.873+0000 7fc8c6000640 1 -- 172.21.15.118:0/1998722774 <== mon.2 v2:172.21.15.189:3300/0 2 ==== osd_map(90..90 src has 1..90) v4 ==== 8277+0+0 (secure 0 0 0) 0x7fc8c800eab0 con 0x56322d94dca0 2024-03-24T00:19:11.877 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:11.874+0000 7fc8c6000640 1 -- 172.21.15.118:0/1998722774 <== mon.2 v2:172.21.15.189:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) v2 ==== 24+0+0 (secure 0 0 0) 0x7fc8c800ecb0 con 0x56322d94dca0 2024-03-24T00:19:11.877 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:11.874+0000 7fc8e5b88740 10 client.26495 fetch_fsmap learned FSMap version 28 2024-03-24T00:19:11.877 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:11.874+0000 7fc8e5b88740 1 -- 172.21.15.118:0/1998722774 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) v3 -- 0x56322d587170 con 0x56322d94dca0 2024-03-24T00:19:11.877 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:11.874+0000 7fc8c6000640 1 -- 172.21.15.118:0/1998722774 <== mon.2 v2:172.21.15.189:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fc8cc004c60 con 0x56322d94dca0 2024-03-24T00:19:11.878 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:11.874+0000 7fc8c6000640 1 -- 172.21.15.118:0/1998722774 <== mon.2 v2:172.21.15.189:3300/0 5 ==== fsmap(e 28) v1 ==== 7934+0+0 (secure 0 0 0) 0x7fc8c80265c0 con 0x56322d94dca0 2024-03-24T00:19:11.878 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:11.874+0000 7fc8e5b88740 10 client.26495 fetch_fsmap finished waiting for FSMap version 28 2024-03-24T00:19:11.878 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:11.874+0000 7fc8e5b88740 10 client.26495 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-24T00:19:11.878 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:11.874+0000 7fc8e5b88740 20 client.26495 populate_metadata read hostname 'smithi118' 2024-03-24T00:19:11.878 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:11.874+0000 7fc8e5b88740 1 --2- 172.21.15.118:0/1998722774 >> [v2:172.21.15.189:6838/4074484168,v1:172.21.15.189:6839/4074484168] conn(0x56322d962fb0 0x56322d9653c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:19:11.878 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:11.874+0000 7fc8e5b88740 4 client.26495 mds_command: new command op to 24569 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-24T00:19:11.878 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:11.874+0000 7fc8e5b88740 1 -- 172.21.15.118:0/1998722774 --> [v2:172.21.15.189:6838/4074484168,v1:172.21.15.189:6839/4074484168] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x56322d8c03f0 con 0x56322d962fb0 2024-03-24T00:19:11.878 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:11.874+0000 7fc8dbe00640 1 --2- 172.21.15.118:0/1998722774 >> [v2:172.21.15.189:6838/4074484168,v1:172.21.15.189:6839/4074484168] conn(0x56322d962fb0 0x56322d9653c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:19:11.878 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:11.875+0000 7fc8c6000640 10 client.26495 ms_handle_connect on v2:172.21.15.189:6838/4074484168 2024-03-24T00:19:11.879 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:11.875+0000 7fc8dbe00640 1 --2- 172.21.15.118:0/1998722774 >> [v2:172.21.15.189:6838/4074484168,v1:172.21.15.189:6839/4074484168] conn(0x56322d962fb0 0x56322d9653c0 crc :-1 s=READY pgs=111 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=c61e460f429d490b server_cookie=4015a59019fbd3f5 in_seq=0 out_seq=0 2024-03-24T00:19:11.880 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:11.877+0000 7fc8c6000640 1 -- 172.21.15.118:0/1998722774 <== mds.0 v2:172.21.15.189:6838/4074484168 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x56322d8c03f0 con 0x56322d962fb0 2024-03-24T00:19:11.880 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:11.877+0000 7fc8c6000640 10 client.26495 handle_command_reply: tid=0 2024-03-24T00:19:11.880 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:11.877+0000 7fc8e5b88740 1 client.26495 shutdown 2024-03-24T00:19:11.880 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:11.877+0000 7fc8e5b88740 1 -- 172.21.15.118:0/1998722774 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56322d94dca0 msgr2=0x56322d9576a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:19:11.880 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:11.877+0000 7fc8e5b88740 1 --2- 172.21.15.118:0/1998722774 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56322d94dca0 0x56322d9576a0 secure :-1 s=READY pgs=384 cs=0 l=1 rev1=1 crypto rx=0x7fc8c800fcf0 tx=0x7fc8c8004730 comp rx=0 tx=0).stop 2024-03-24T00:19:11.880 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:11.877+0000 7fc8e5b88740 1 -- 172.21.15.118:0/1998722774 shutdown_connections 2024-03-24T00:19:11.880 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:11.877+0000 7fc8e5b88740 1 --2- 172.21.15.118:0/1998722774 >> [v2:172.21.15.189:6838/4074484168,v1:172.21.15.189:6839/4074484168] conn(0x56322d962fb0 0x56322d9653c0 crc :-1 s=READY pgs=111 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:19:11.880 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:11.877+0000 7fc8e5b88740 1 --2- 172.21.15.118:0/1998722774 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x56322d952e10 0x56322d95e6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:19:11.880 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:11.877+0000 7fc8e5b88740 1 --2- 172.21.15.118:0/1998722774 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56322d94dca0 0x56322d9576a0 unknown :-1 s=CLOSED pgs=384 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:19:11.880 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:11.877+0000 7fc8c6000640 0 client.26495 ms_handle_reset on v2:172.21.15.189:6838/4074484168 2024-03-24T00:19:11.880 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:11.877+0000 7fc8e5b88740 1 --2- 172.21.15.118:0/1998722774 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56322d94d2b0 0x56322d957160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:19:11.881 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:11.877+0000 7fc8e5b88740 1 -- 172.21.15.118:0/1998722774 >> 172.21.15.118:0/1998722774 conn(0x56322d948820 msgr2=0x56322d947a80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:19:11.881 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:11.878+0000 7fc8e5b88740 1 -- 172.21.15.118:0/1998722774 shutdown_connections 2024-03-24T00:19:11.881 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:11.878+0000 7fc8e5b88740 1 -- 172.21.15.118:0/1998722774 wait complete. 2024-03-24T00:19:11.881 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:11.878+0000 7fc8e5b88740 20 client.26495 trim_cache size 0 max 16384 2024-03-24T00:19:11.899 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:11.899+0000 7fc8e5b88740 1 Processor -- start 2024-03-24T00:19:11.899 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:11.899+0000 7fc8e5b88740 1 -- start start 2024-03-24T00:19:11.899 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:11.900+0000 7fc8e5b88740 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x56322d94d2b0 0x56322d947900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:19:11.899 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:11.900+0000 7fc8e5b88740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56322d94dca0 0x56322d95e6a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:19:11.899 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:11.900+0000 7fc8e5b88740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56322d952e10 0x56322d969c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:19:11.899 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:11.900+0000 7fc8e5b88740 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc8dc1aaa60 con 0x56322d94d2b0 2024-03-24T00:19:11.900 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:11.900+0000 7fc8e5b88740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc8dc1ac3e0 con 0x56322d952e10 2024-03-24T00:19:11.900 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:11.900+0000 7fc8e5b88740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc8dc1ab0c0 con 0x56322d94dca0 2024-03-24T00:19:11.901 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:11.900+0000 7fc8dbe00640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x56322d94d2b0 0x56322d947900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:19:11.901 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:11.900+0000 7fc8db400640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56322d94dca0 0x56322d95e6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:19:11.901 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:11.900+0000 7fc8db400640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56322d94dca0 0x56322d95e6a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.118:47510/0 (socket says 172.21.15.118:47510) 2024-03-24T00:19:11.901 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:11.900+0000 7fc8db400640 1 -- 172.21.15.118:0/676805322 learned_addr learned my addr 172.21.15.118:0/676805322 (peer_addr_for_me v2:172.21.15.118:0/0) 2024-03-24T00:19:11.901 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:11.900+0000 7fc8e0e00640 1 --2- 172.21.15.118:0/676805322 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56322d952e10 0x56322d969c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:19:11.901 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:11.900+0000 7fc8db400640 1 -- 172.21.15.118:0/676805322 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56322d952e10 msgr2=0x56322d969c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:19:11.901 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:11.900+0000 7fc8db400640 1 --2- 172.21.15.118:0/676805322 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56322d952e10 0x56322d969c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:19:11.901 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:11.900+0000 7fc8db400640 1 -- 172.21.15.118:0/676805322 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x56322d94d2b0 msgr2=0x56322d947900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:19:11.901 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:11.900+0000 7fc8db400640 1 --2- 172.21.15.118:0/676805322 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x56322d94d2b0 0x56322d947900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:19:11.901 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:11.901+0000 7fc8db400640 1 -- 172.21.15.118:0/676805322 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x56322d94e530 con 0x56322d94dca0 2024-03-24T00:19:11.902 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:11.901+0000 7fc8db400640 1 --2- 172.21.15.118:0/676805322 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56322d94dca0 0x56322d95e6a0 secure :-1 s=READY pgs=384 cs=0 l=1 rev1=1 crypto rx=0x7fc8c8016040 tx=0x7fc8c8000bc0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=1439091e08275311 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:19:11.902 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:11.901+0000 7fc8d0c00640 1 -- 172.21.15.118:0/676805322 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc8c800fea0 con 0x56322d94dca0 2024-03-24T00:19:11.902 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:11.901+0000 7fc8d0c00640 1 -- 172.21.15.118:0/676805322 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fc8c800ee50 con 0x56322d94dca0 2024-03-24T00:19:11.902 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:11.901+0000 7fc8d0c00640 1 -- 172.21.15.118:0/676805322 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc8c8030430 con 0x56322d94dca0 2024-03-24T00:19:11.903 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:11.902+0000 7fc8e5b88740 1 -- 172.21.15.118:0/676805322 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56322d94dca0 msgr2=0x56322d95e6a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:19:11.903 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:11.902+0000 7fc8e5b88740 1 --2- 172.21.15.118:0/676805322 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56322d94dca0 0x56322d95e6a0 secure :-1 s=READY pgs=384 cs=0 l=1 rev1=1 crypto rx=0x7fc8c8016040 tx=0x7fc8c8000bc0 comp rx=0 tx=0).stop 2024-03-24T00:19:11.903 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:11.902+0000 7fc8e5b88740 1 -- 172.21.15.118:0/676805322 shutdown_connections 2024-03-24T00:19:11.903 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:11.902+0000 7fc8e5b88740 1 --2- 172.21.15.118:0/676805322 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56322d952e10 0x56322d969c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:19:11.903 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:11.902+0000 7fc8e5b88740 1 --2- 172.21.15.118:0/676805322 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56322d94dca0 0x56322d95e6a0 unknown :-1 s=CLOSED pgs=384 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:19:11.903 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:11.902+0000 7fc8e5b88740 1 --2- 172.21.15.118:0/676805322 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x56322d94d2b0 0x56322d947900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:19:11.903 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:11.902+0000 7fc8e5b88740 1 -- 172.21.15.118:0/676805322 >> 172.21.15.118:0/676805322 conn(0x56322d948820 msgr2=0x56322d9471f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:19:11.903 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:11.903+0000 7fc8e5b88740 1 -- 172.21.15.118:0/676805322 shutdown_connections 2024-03-24T00:19:11.903 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:11.903+0000 7fc8e5b88740 1 -- 172.21.15.118:0/676805322 wait complete. 2024-03-24T00:19:11.903 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:11.903+0000 7fc8e5b88740 1 -- start start 2024-03-24T00:19:11.904 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:11.903+0000 7fc8e5b88740 1 Processor -- start 2024-03-24T00:19:11.905 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:11.904+0000 7fc8e5b88740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56322d94d2b0 0x56322d95cde0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:19:11.905 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:11.904+0000 7fc8e5b88740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56322d94dca0 0x56322d95d320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:19:11.906 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:11.904+0000 7fc8e5b88740 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x56322d952e10 0x56322d95d860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:19:11.906 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:11.904+0000 7fc8e5b88740 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc8dc1abd80 con 0x56322d952e10 2024-03-24T00:19:11.906 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:11.904+0000 7fc8e5b88740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc8dc1ab720 con 0x56322d94dca0 2024-03-24T00:19:11.906 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:11.904+0000 7fc8e5b88740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc8dc1a8c20 con 0x56322d94d2b0 2024-03-24T00:19:11.906 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:11.904+0000 7fc8e5b88740 10 client.0 _ll_register_callbacks cb 0x7fc8c800f970 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T00:19:11.906 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:11.904+0000 7fc8dbe00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56322d94d2b0 0x56322d95cde0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:19:11.906 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:11.904+0000 7fc8db400640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56322d94dca0 0x56322d95d320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:19:11.906 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:11.904+0000 7fc8dbe00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56322d94d2b0 0x56322d95cde0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.118:47526/0 (socket says 172.21.15.118:47526) 2024-03-24T00:19:11.906 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:11.904+0000 7fc8dbe00640 1 -- 172.21.15.118:0/1893892824 learned_addr learned my addr 172.21.15.118:0/1893892824 (peer_addr_for_me v2:172.21.15.118:0/0) 2024-03-24T00:19:11.906 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:11.905+0000 7fc8e0e00640 1 --2- 172.21.15.118:0/1893892824 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x56322d952e10 0x56322d95d860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:19:11.906 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:11.905+0000 7fc8dbe00640 1 -- 172.21.15.118:0/1893892824 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56322d94dca0 msgr2=0x56322d95d320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:19:11.906 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:11.905+0000 7fc8dbe00640 1 --2- 172.21.15.118:0/1893892824 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56322d94dca0 0x56322d95d320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:19:11.906 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:11.905+0000 7fc8dbe00640 1 -- 172.21.15.118:0/1893892824 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x56322d952e10 msgr2=0x56322d95d860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:19:11.907 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:11.905+0000 7fc8dbe00640 1 --2- 172.21.15.118:0/1893892824 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x56322d952e10 0x56322d95d860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:19:11.907 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:11.905+0000 7fc8dbe00640 1 -- 172.21.15.118:0/1893892824 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x56322d9be850 con 0x56322d94d2b0 2024-03-24T00:19:11.907 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:11.905+0000 7fc8dbe00640 1 --2- 172.21.15.118:0/1893892824 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56322d94d2b0 0x56322d95cde0 secure :-1 s=READY pgs=385 cs=0 l=1 rev1=1 crypto rx=0x7fc8cc02cfd0 tx=0x7fc8cc005dd0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=6c5ac0ef42f7c979 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:19:11.907 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:11.905+0000 7fc8c6000640 10 client.0 ms_handle_connect on v2:172.21.15.193:3300/0 2024-03-24T00:19:11.907 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:11.905+0000 7fc8c6000640 1 -- 172.21.15.118:0/1893892824 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc8cc04b040 con 0x56322d94d2b0 2024-03-24T00:19:11.907 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:11.905+0000 7fc8c6000640 1 -- 172.21.15.118:0/1893892824 <== mon.1 v2:172.21.15.193:3300/0 2 ==== osd_map(90..90 src has 1..90) v4 ==== 8277+0+0 (secure 0 0 0) 0x7fc8cc0b95e0 con 0x56322d94d2b0 2024-03-24T00:19:11.907 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:11.905+0000 7fc8e5b88740 1 -- 172.21.15.118:0/1893892824 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fc8c800ee50 con 0x56322d94d2b0 2024-03-24T00:19:11.907 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:11.906+0000 7fc8c6000640 1 -- 172.21.15.118:0/1893892824 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) v2 ==== 24+0+0 (secure 0 0 0) 0x7fc8cc004c60 con 0x56322d94d2b0 2024-03-24T00:19:11.907 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:11.906+0000 7fc8e5b88740 10 client.26512 fetch_fsmap learned FSMap version 28 2024-03-24T00:19:11.907 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:11.906+0000 7fc8e5b88740 1 -- 172.21.15.118:0/1893892824 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) v3 -- 0x56322d9bfa70 con 0x56322d94d2b0 2024-03-24T00:19:11.907 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:11.906+0000 7fc8c6000640 1 -- 172.21.15.118:0/1893892824 <== mon.1 v2:172.21.15.193:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fc8c800ee50 con 0x56322d94d2b0 2024-03-24T00:19:11.908 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:11.907+0000 7fc8c6000640 1 -- 172.21.15.118:0/1893892824 <== mon.1 v2:172.21.15.193:3300/0 5 ==== fsmap(e 28) v1 ==== 7934+0+0 (secure 0 0 0) 0x7fc8cc0c51d0 con 0x56322d94d2b0 2024-03-24T00:19:11.908 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:11.907+0000 7fc8e5b88740 10 client.26512 fetch_fsmap finished waiting for FSMap version 28 2024-03-24T00:19:11.908 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:11.907+0000 7fc8e5b88740 10 client.26512 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-24T00:19:11.908 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:11.907+0000 7fc8e5b88740 20 client.26512 populate_metadata read hostname 'smithi118' 2024-03-24T00:19:11.908 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:11.907+0000 7fc8e5b88740 1 --2- 172.21.15.118:0/1893892824 >> [v2:172.21.15.189:6838/4074484168,v1:172.21.15.189:6839/4074484168] conn(0x56322d9c0170 0x56322d9c2560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:19:11.909 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:11.907+0000 7fc8e5b88740 4 client.26512 mds_command: new command op to 24569 tid=0 multi_id=0 [{"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}] 2024-03-24T00:19:11.909 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:11.907+0000 7fc8e5b88740 1 -- 172.21.15.118:0/1893892824 --> [v2:172.21.15.189:6838/4074484168,v1:172.21.15.189:6839/4074484168] -- command(tid 0: {"prefix": "scrub start", "path": "/", "scrubops": ["force", "recursive"]}) v1 -- 0x56322d8ba740 con 0x56322d9c0170 2024-03-24T00:19:11.909 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:11.907+0000 7fc8db400640 1 --2- 172.21.15.118:0/1893892824 >> [v2:172.21.15.189:6838/4074484168,v1:172.21.15.189:6839/4074484168] conn(0x56322d9c0170 0x56322d9c2560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:19:11.909 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:11.908+0000 7fc8db400640 1 --2- 172.21.15.118:0/1893892824 >> [v2:172.21.15.189:6838/4074484168,v1:172.21.15.189:6839/4074484168] conn(0x56322d9c0170 0x56322d9c2560 crc :-1 s=READY pgs=112 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=46ec207fd4e3f2b3 server_cookie=2961a26aec5661ad in_seq=0 out_seq=0 2024-03-24T00:19:11.909 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:11.908+0000 7fc8c6000640 10 client.26512 ms_handle_connect on v2:172.21.15.189:6838/4074484168 2024-03-24T00:19:11.910 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:11.910+0000 7fc8c6000640 1 -- 172.21.15.118:0/1893892824 <== mds.0 v2:172.21.15.189:6838/4074484168 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+110 (crc 0 0 0) 0x56322d8ba740 con 0x56322d9c0170 2024-03-24T00:19:11.911 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:11.910+0000 7fc8c6000640 10 client.26512 handle_command_reply: tid=0 2024-03-24T00:19:11.911 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:11.910+0000 7fc8e5b88740 1 client.26512 shutdown 2024-03-24T00:19:11.911 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:11.910+0000 7fc8e5b88740 1 -- 172.21.15.118:0/1893892824 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56322d94d2b0 msgr2=0x56322d95cde0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:19:11.911 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:11.910+0000 7fc8e5b88740 1 --2- 172.21.15.118:0/1893892824 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56322d94d2b0 0x56322d95cde0 secure :-1 s=READY pgs=385 cs=0 l=1 rev1=1 crypto rx=0x7fc8cc02cfd0 tx=0x7fc8cc005dd0 comp rx=0 tx=0).stop 2024-03-24T00:19:11.911 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:11.911+0000 7fc8e5b88740 1 -- 172.21.15.118:0/1893892824 shutdown_connections 2024-03-24T00:19:11.911 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:11.911+0000 7fc8e5b88740 1 --2- 172.21.15.118:0/1893892824 >> [v2:172.21.15.189:6838/4074484168,v1:172.21.15.189:6839/4074484168] conn(0x56322d9c0170 0x56322d9c2560 crc :-1 s=READY pgs=112 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:19:11.911 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:11.911+0000 7fc8e5b88740 1 --2- 172.21.15.118:0/1893892824 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x56322d952e10 0x56322d95d860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:19:11.911 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:11.911+0000 7fc8e5b88740 1 --2- 172.21.15.118:0/1893892824 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56322d94dca0 0x56322d95d320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:19:11.911 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:11.911+0000 7fc8e5b88740 1 --2- 172.21.15.118:0/1893892824 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56322d94d2b0 0x56322d95cde0 unknown :-1 s=CLOSED pgs=385 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:19:11.911 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:11.911+0000 7fc8e5b88740 1 -- 172.21.15.118:0/1893892824 >> 172.21.15.118:0/1893892824 conn(0x56322d948820 msgr2=0x56322d947a80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:19:11.911 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:11.911+0000 7fc8c6000640 0 client.26512 ms_handle_reset on v2:172.21.15.189:6838/4074484168 2024-03-24T00:19:11.911 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:11.911+0000 7fc8e5b88740 1 -- 172.21.15.118:0/1893892824 shutdown_connections 2024-03-24T00:19:11.911 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:11.911+0000 7fc8e5b88740 1 -- 172.21.15.118:0/1893892824 wait complete. 2024-03-24T00:19:11.911 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:11.911+0000 7fc8e5b88740 20 client.26512 trim_cache size 0 max 16384 2024-03-24T00:19:11.912 INFO:teuthology.orchestra.run.smithi118.stdout:{ 2024-03-24T00:19:11.912 INFO:teuthology.orchestra.run.smithi118.stdout: "return_code": 0, 2024-03-24T00:19:11.912 INFO:teuthology.orchestra.run.smithi118.stdout: "scrub_tag": "01d80384-ff69-4b87-9df0-e818e48b6822", 2024-03-24T00:19:11.912 INFO:teuthology.orchestra.run.smithi118.stdout: "mode": "asynchronous" 2024-03-24T00:19:11.912 INFO:teuthology.orchestra.run.smithi118.stdout:} 2024-03-24T00:19:11.913 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:11.912+0000 7fc8e1800640 1 -- 172.21.15.118:0/1067441042 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7fc8c00a2a20 msgr2=0x7fc8c00a4ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:19:11.913 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:11.912+0000 7fc8e1800640 1 --2- 172.21.15.118:0/1067441042 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7fc8c00a2a20 0x7fc8c00a4ee0 secure :-1 s=READY pgs=338 cs=0 l=1 rev1=1 crypto rx=0x7fc8c80095f0 tx=0x7fc8c8005f90 comp rx=0 tx=0).stop 2024-03-24T00:19:11.913 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:11.912+0000 7fc8e1800640 1 -- 172.21.15.118:0/1067441042 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc8dc1037a0 msgr2=0x7fc8dc199ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:19:11.913 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:11.912+0000 7fc8e1800640 1 --2- 172.21.15.118:0/1067441042 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc8dc1037a0 0x7fc8dc199ae0 secure :-1 s=READY pgs=383 cs=0 l=1 rev1=1 crypto rx=0x7fc8cc02efd0 tx=0x7fc8cc02c660 comp rx=0 tx=0).stop 2024-03-24T00:19:11.913 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:11.912+0000 7fc8e1800640 1 -- 172.21.15.118:0/1067441042 shutdown_connections 2024-03-24T00:19:11.914 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:11.912+0000 7fc8e1800640 1 --2- 172.21.15.118:0/1067441042 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7fc8c00a2a20 0x7fc8c00a4ee0 unknown :-1 s=CLOSED pgs=338 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:19:11.914 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:11.912+0000 7fc8e1800640 1 --2- 172.21.15.118:0/1067441042 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fc8dc105350 0x7fc8dc1a1090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:19:11.914 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:11.912+0000 7fc8e1800640 1 --2- 172.21.15.118:0/1067441042 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fc8dc104990 0x7fc8dc19a020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:19:11.914 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:11.912+0000 7fc8e1800640 1 --2- 172.21.15.118:0/1067441042 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc8dc1037a0 0x7fc8dc199ae0 unknown :-1 s=CLOSED pgs=383 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:19:11.914 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:11.912+0000 7fc8e1800640 1 -- 172.21.15.118:0/1067441042 >> 172.21.15.118:0/1067441042 conn(0x7fc8dc0fed10 msgr2=0x7fc8dc107c90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:19:11.914 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:11.912+0000 7fc8e1800640 1 -- 172.21.15.118:0/1067441042 shutdown_connections 2024-03-24T00:19:11.914 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:11.912+0000 7fc8e1800640 1 -- 172.21.15.118:0/1067441042 wait complete. 2024-03-24T00:19:11.933 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-03-24T00:19:11.933 DEBUG:teuthology.orchestra.run.smithi118:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-24T00:19:12.136 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:12.136+0000 7f0559200640 1 -- 172.21.15.118:0/308279626 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f0554102940 msgr2=0x7f0554102dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:19:12.137 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:12.136+0000 7f0559200640 1 --2- 172.21.15.118:0/308279626 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f0554102940 0x7f0554102dc0 secure :-1 s=READY pgs=386 cs=0 l=1 rev1=1 crypto rx=0x7f0544009930 tx=0x7f054402bc20 comp rx=0 tx=0).stop 2024-03-24T00:19:12.137 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:12.136+0000 7f0559200640 1 -- 172.21.15.118:0/308279626 shutdown_connections 2024-03-24T00:19:12.137 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:12.136+0000 7f0559200640 1 --2- 172.21.15.118:0/308279626 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0554103300 0x7f0554109b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:19:12.137 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:12.137+0000 7f0559200640 1 --2- 172.21.15.118:0/308279626 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f0554102940 0x7f0554102dc0 unknown :-1 s=CLOSED pgs=386 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:19:12.137 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:12.137+0000 7f0559200640 1 --2- 172.21.15.118:0/308279626 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f05540ff970 0x7f05540ffd90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:19:12.137 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:12.137+0000 7f0559200640 1 -- 172.21.15.118:0/308279626 >> 172.21.15.118:0/308279626 conn(0x7f05540fb2c0 msgr2=0x7f05540fd6e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:19:12.137 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:12.137+0000 7f0559200640 1 -- 172.21.15.118:0/308279626 shutdown_connections 2024-03-24T00:19:12.137 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:12.137+0000 7f0559200640 1 -- 172.21.15.118:0/308279626 wait complete. 2024-03-24T00:19:12.139 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:12.137+0000 7f0559200640 1 Processor -- start 2024-03-24T00:19:12.140 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:12.138+0000 7f0559200640 1 -- start start 2024-03-24T00:19:12.140 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:12.138+0000 7f0559200640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f05540ff970 0x7f0554199c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:19:12.140 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:12.138+0000 7f0559200640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0554102940 0x7f055419a180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:19:12.140 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:12.138+0000 7f0559200640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f0554103300 0x7f05541a11b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:19:12.140 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:12.138+0000 7f0559200640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f055410d3a0 con 0x7f0554103300 2024-03-24T00:19:12.140 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:12.138+0000 7f0559200640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f055410d220 con 0x7f0554102940 2024-03-24T00:19:12.140 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:12.138+0000 7f0559200640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f055410d520 con 0x7f05540ff970 2024-03-24T00:19:12.140 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:12.138+0000 7f0551e00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f05540ff970 0x7f0554199c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:19:12.140 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:12.138+0000 7f0551400640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0554102940 0x7f055419a180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:19:12.140 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:12.138+0000 7f0551e00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f05540ff970 0x7f0554199c40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.118:47548/0 (socket says 172.21.15.118:47548) 2024-03-24T00:19:12.140 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:12.138+0000 7f0551e00640 1 -- 172.21.15.118:0/3476096783 learned_addr learned my addr 172.21.15.118:0/3476096783 (peer_addr_for_me v2:172.21.15.118:0/0) 2024-03-24T00:19:12.140 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:12.138+0000 7f0552800640 1 --2- 172.21.15.118:0/3476096783 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f0554103300 0x7f05541a11b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:19:12.140 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:12.139+0000 7f0551400640 1 -- 172.21.15.118:0/3476096783 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f05540ff970 msgr2=0x7f0554199c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:19:12.141 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:12.139+0000 7f0551400640 1 --2- 172.21.15.118:0/3476096783 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f05540ff970 0x7f0554199c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:19:12.141 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:12.139+0000 7f0551400640 1 -- 172.21.15.118:0/3476096783 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f0554103300 msgr2=0x7f05541a11b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:19:12.141 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:12.139+0000 7f0551400640 1 --2- 172.21.15.118:0/3476096783 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f0554103300 0x7f05541a11b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:19:12.141 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:12.139+0000 7f0551400640 1 -- 172.21.15.118:0/3476096783 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f05541a1850 con 0x7f0554102940 2024-03-24T00:19:12.141 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:12.139+0000 7f0551e00640 1 --2- 172.21.15.118:0/3476096783 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f05540ff970 0x7f0554199c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T00:19:12.141 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:12.139+0000 7f0551400640 1 --2- 172.21.15.118:0/3476096783 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0554102940 0x7f055419a180 secure :-1 s=READY pgs=385 cs=0 l=1 rev1=1 crypto rx=0x7f054402c160 tx=0x7f054402ce50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:19:12.141 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:12.139+0000 7f054aa00640 1 -- 172.21.15.118:0/3476096783 <== mon.2 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0544043020 con 0x7f0554102940 2024-03-24T00:19:12.141 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:12.139+0000 7f0559200640 1 -- 172.21.15.118:0/3476096783 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f05541a1ae0 con 0x7f0554102940 2024-03-24T00:19:12.141 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:12.139+0000 7f054aa00640 1 -- 172.21.15.118:0/3476096783 <== mon.2 v2:172.21.15.189:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f05440054e0 con 0x7f0554102940 2024-03-24T00:19:12.141 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:12.139+0000 7f054aa00640 1 -- 172.21.15.118:0/3476096783 <== mon.2 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0544003bf0 con 0x7f0554102940 2024-03-24T00:19:12.141 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:12.139+0000 7f0559200640 1 -- 172.21.15.118:0/3476096783 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f05541a2000 con 0x7f0554102940 2024-03-24T00:19:12.143 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:12.143+0000 7f054aa00640 1 -- 172.21.15.118:0/3476096783 <== mon.2 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 20) v1 ==== 137291+0+0 (secure 0 0 0) 0x7f0544046020 con 0x7f0554102940 2024-03-24T00:19:12.143 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:12.143+0000 7f055be57740 1 Processor -- start 2024-03-24T00:19:12.143 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:12.143+0000 7f055be57740 1 -- start start 2024-03-24T00:19:12.145 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:12.144+0000 7f055be57740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x562971d9ce80 0x562971d9d2a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:19:12.145 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:12.144+0000 7f055be57740 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x562971d9d870 0x562971da2370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:19:12.145 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:12.144+0000 7f055be57740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x562971da29e0 0x562971da4e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:19:12.145 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:12.144+0000 7f055be57740 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x562971d02250 con 0x562971d9d870 2024-03-24T00:19:12.145 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:12.144+0000 7f055be57740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x562971caaa30 con 0x562971d9ce80 2024-03-24T00:19:12.145 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:12.144+0000 7f055be57740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x562971cfe2f0 con 0x562971da29e0 2024-03-24T00:19:12.145 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:12.144+0000 7f0552800640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x562971d9ce80 0x562971d9d2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:19:12.145 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:12.144+0000 7f0552800640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x562971d9ce80 0x562971d9d2a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.118:54430/0 (socket says 172.21.15.118:54430) 2024-03-24T00:19:12.145 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:12.144+0000 7f0552800640 1 -- 172.21.15.118:0/4013630799 learned_addr learned my addr 172.21.15.118:0/4013630799 (peer_addr_for_me v2:172.21.15.118:0/0) 2024-03-24T00:19:12.146 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:12.144+0000 7f0551e00640 1 --2- 172.21.15.118:0/4013630799 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x562971d9d870 0x562971da2370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:19:12.146 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:12.144+0000 7f0551400640 1 --2- 172.21.15.118:0/4013630799 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x562971da29e0 0x562971da4e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:19:12.146 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:12.144+0000 7f0552800640 1 -- 172.21.15.118:0/4013630799 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x562971da29e0 msgr2=0x562971da4e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:19:12.146 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:12.144+0000 7f0552800640 1 --2- 172.21.15.118:0/4013630799 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x562971da29e0 0x562971da4e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:19:12.146 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:12.144+0000 7f0552800640 1 -- 172.21.15.118:0/4013630799 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x562971d9d870 msgr2=0x562971da2370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:19:12.146 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:12.144+0000 7f0552800640 1 --2- 172.21.15.118:0/4013630799 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x562971d9d870 0x562971da2370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:19:12.146 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:12.144+0000 7f0552800640 1 -- 172.21.15.118:0/4013630799 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x562971c95690 con 0x562971d9ce80 2024-03-24T00:19:12.146 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:12.144+0000 7f0551e00640 1 --2- 172.21.15.118:0/4013630799 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x562971d9d870 0x562971da2370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T00:19:12.146 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:12.145+0000 7f0552800640 1 --2- 172.21.15.118:0/4013630799 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x562971d9ce80 0x562971d9d2a0 secure :-1 s=READY pgs=386 cs=0 l=1 rev1=1 crypto rx=0x7f054c009fc0 tx=0x7f054c010860 comp rx=0 tx=0).ready entity=mon.2 client_cookie=e0b04921a86500cb server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:19:12.146 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:12.145+0000 7f053f400640 1 -- 172.21.15.118:0/4013630799 <== mon.2 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f054c003b70 con 0x562971d9ce80 2024-03-24T00:19:12.146 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:12.145+0000 7f054aa00640 1 --2- 172.21.15.118:0/3476096783 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7f05380a2800 0x7f05380a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:19:12.146 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:12.145+0000 7f053f400640 1 -- 172.21.15.118:0/4013630799 <== mon.2 v2:172.21.15.189:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f054c003d10 con 0x562971d9ce80 2024-03-24T00:19:12.146 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:12.145+0000 7f053f400640 1 -- 172.21.15.118:0/4013630799 <== mon.2 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f054c011aa0 con 0x562971d9ce80 2024-03-24T00:19:12.147 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:12.145+0000 7f054aa00640 1 -- 172.21.15.118:0/3476096783 <== mon.2 v2:172.21.15.189:3300/0 5 ==== osd_map(90..90 src has 1..90) v4 ==== 8277+0+0 (secure 0 0 0) 0x7f05440f0c00 con 0x7f0554102940 2024-03-24T00:19:12.147 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:12.145+0000 7f0551e00640 1 --2- 172.21.15.118:0/3476096783 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7f05380a2800 0x7f05380a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:19:12.150 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:12.146+0000 7f055be57740 1 -- 172.21.15.118:0/4013630799 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x562971d9ce80 msgr2=0x562971d9d2a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:19:12.151 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:12.146+0000 7f055be57740 1 --2- 172.21.15.118:0/4013630799 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x562971d9ce80 0x562971d9d2a0 secure :-1 s=READY pgs=386 cs=0 l=1 rev1=1 crypto rx=0x7f054c009fc0 tx=0x7f054c010860 comp rx=0 tx=0).stop 2024-03-24T00:19:12.151 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:12.146+0000 7f055be57740 1 -- 172.21.15.118:0/4013630799 shutdown_connections 2024-03-24T00:19:12.151 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:12.146+0000 7f055be57740 1 --2- 172.21.15.118:0/4013630799 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x562971da29e0 0x562971da4e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:19:12.151 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:12.146+0000 7f055be57740 1 --2- 172.21.15.118:0/4013630799 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x562971d9d870 0x562971da2370 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:19:12.151 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:12.146+0000 7f055be57740 1 --2- 172.21.15.118:0/4013630799 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x562971d9ce80 0x562971d9d2a0 unknown :-1 s=CLOSED pgs=386 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:19:12.151 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:12.146+0000 7f055be57740 1 -- 172.21.15.118:0/4013630799 >> 172.21.15.118:0/4013630799 conn(0x562971d65200 msgr2=0x562971d987f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:19:12.151 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:12.146+0000 7f055be57740 1 -- 172.21.15.118:0/4013630799 shutdown_connections 2024-03-24T00:19:12.151 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:12.146+0000 7f0551e00640 1 --2- 172.21.15.118:0/3476096783 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7f05380a2800 0x7f05380a4cc0 secure :-1 s=READY pgs=339 cs=0 l=1 rev1=1 crypto rx=0x562971d9dc70 tx=0x7f054000b410 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:19:12.151 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:12.146+0000 7f055be57740 1 -- 172.21.15.118:0/4013630799 wait complete. 2024-03-24T00:19:12.151 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:12.147+0000 7f055be57740 1 -- start start 2024-03-24T00:19:12.151 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:12.147+0000 7f055be57740 1 Processor -- start 2024-03-24T00:19:12.151 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:12.148+0000 7f055be57740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x562971d9ce80 0x562971da6ee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:19:12.151 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:12.148+0000 7f055be57740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x562971d9d870 0x562971da7420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:19:12.151 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:12.148+0000 7f055be57740 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x562971da29e0 0x562971dae3f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:19:12.152 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:12.148+0000 7f055be57740 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x562971da5990 con 0x562971da29e0 2024-03-24T00:19:12.152 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:12.148+0000 7f055be57740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x562971cbde80 con 0x562971d9d870 2024-03-24T00:19:12.152 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:12.148+0000 7f055be57740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x562971da5b10 con 0x562971d9ce80 2024-03-24T00:19:12.152 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:12.148+0000 7f055be57740 10 client.0 _ll_register_callbacks cb 0x562971cd0130 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T00:19:12.152 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:12.148+0000 7f0551e00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x562971d9ce80 0x562971da6ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:19:12.152 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:12.148+0000 7f0551400640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x562971d9d870 0x562971da7420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:19:12.152 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:12.148+0000 7f0551e00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x562971d9ce80 0x562971da6ee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.118:47566/0 (socket says 172.21.15.118:47566) 2024-03-24T00:19:12.152 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:12.148+0000 7f0551e00640 1 -- 172.21.15.118:0/1682526981 learned_addr learned my addr 172.21.15.118:0/1682526981 (peer_addr_for_me v2:172.21.15.118:0/0) 2024-03-24T00:19:12.152 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:12.148+0000 7f0552800640 1 --2- 172.21.15.118:0/1682526981 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x562971da29e0 0x562971dae3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:19:12.152 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:12.149+0000 7f0551400640 1 -- 172.21.15.118:0/1682526981 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x562971d9ce80 msgr2=0x562971da6ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:19:12.152 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:12.149+0000 7f0551400640 1 --2- 172.21.15.118:0/1682526981 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x562971d9ce80 0x562971da6ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:19:12.152 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:12.149+0000 7f0551400640 1 -- 172.21.15.118:0/1682526981 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x562971da29e0 msgr2=0x562971dae3f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:19:12.152 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:12.149+0000 7f0551400640 1 --2- 172.21.15.118:0/1682526981 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x562971da29e0 0x562971dae3f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:19:12.152 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:12.149+0000 7f0551400640 1 -- 172.21.15.118:0/1682526981 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x562971ae07d0 con 0x562971d9d870 2024-03-24T00:19:12.153 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:12.149+0000 7f0551e00640 1 --2- 172.21.15.118:0/1682526981 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x562971d9ce80 0x562971da6ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T00:19:12.153 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:12.149+0000 7f0552800640 1 --2- 172.21.15.118:0/1682526981 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x562971da29e0 0x562971dae3f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T00:19:12.153 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:12.149+0000 7f0551400640 1 --2- 172.21.15.118:0/1682526981 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x562971d9d870 0x562971da7420 secure :-1 s=READY pgs=387 cs=0 l=1 rev1=1 crypto rx=0x7f05440ee720 tx=0x7f05440f42d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=187edb16fec8746d server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:19:12.153 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:12.149+0000 7f0533e00640 10 client.0 ms_handle_connect on v2:172.21.15.189:3300/0 2024-03-24T00:19:12.153 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:12.149+0000 7f0533e00640 1 -- 172.21.15.118:0/1682526981 <== mon.2 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f05440bd340 con 0x562971d9d870 2024-03-24T00:19:12.153 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:12.149+0000 7f0533e00640 1 -- 172.21.15.118:0/1682526981 <== mon.2 v2:172.21.15.189:3300/0 2 ==== osd_map(90..90 src has 1..90) v4 ==== 8277+0+0 (secure 0 0 0) 0x7f05440bd9d0 con 0x562971d9d870 2024-03-24T00:19:12.153 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:12.149+0000 7f055be57740 1 -- 172.21.15.118:0/1682526981 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f054c003d10 con 0x562971d9d870 2024-03-24T00:19:12.153 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:12.149+0000 7f0533e00640 1 -- 172.21.15.118:0/1682526981 <== mon.2 v2:172.21.15.189:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) v2 ==== 24+0+0 (secure 0 0 0) 0x7f05440bb6b0 con 0x562971d9d870 2024-03-24T00:19:12.153 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:12.149+0000 7f055be57740 10 client.26513 fetch_fsmap learned FSMap version 28 2024-03-24T00:19:12.153 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:12.149+0000 7f055be57740 1 -- 172.21.15.118:0/1682526981 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) v3 -- 0x562971a163f0 con 0x562971d9d870 2024-03-24T00:19:12.153 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:12.150+0000 7f0533e00640 1 -- 172.21.15.118:0/1682526981 <== mon.2 v2:172.21.15.189:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f054c003d10 con 0x562971d9d870 2024-03-24T00:19:12.153 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:12.150+0000 7f0533e00640 1 -- 172.21.15.118:0/1682526981 <== mon.2 v2:172.21.15.189:3300/0 5 ==== fsmap(e 28) v1 ==== 7934+0+0 (secure 0 0 0) 0x7f05440f9070 con 0x562971d9d870 2024-03-24T00:19:12.153 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:12.150+0000 7f055be57740 10 client.26513 fetch_fsmap finished waiting for FSMap version 28 2024-03-24T00:19:12.154 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:12.150+0000 7f055be57740 10 client.26513 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-24T00:19:12.154 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:12.150+0000 7f055be57740 20 client.26513 populate_metadata read hostname 'smithi118' 2024-03-24T00:19:12.154 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:12.150+0000 7f055be57740 1 --2- 172.21.15.118:0/1682526981 >> [v2:172.21.15.189:6838/4074484168,v1:172.21.15.189:6839/4074484168] conn(0x562971db2d60 0x562971db5180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:19:12.154 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:12.150+0000 7f055be57740 4 client.26513 mds_command: new command op to 24569 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-24T00:19:12.154 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:12.150+0000 7f055be57740 1 -- 172.21.15.118:0/1682526981 --> [v2:172.21.15.189:6838/4074484168,v1:172.21.15.189:6839/4074484168] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x562971d0f790 con 0x562971db2d60 2024-03-24T00:19:12.154 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:12.151+0000 7f0551e00640 1 --2- 172.21.15.118:0/1682526981 >> [v2:172.21.15.189:6838/4074484168,v1:172.21.15.189:6839/4074484168] conn(0x562971db2d60 0x562971db5180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:19:12.154 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:12.151+0000 7f0551e00640 1 --2- 172.21.15.118:0/1682526981 >> [v2:172.21.15.189:6838/4074484168,v1:172.21.15.189:6839/4074484168] conn(0x562971db2d60 0x562971db5180 crc :-1 s=READY pgs=113 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=319ba8d6fe658955 server_cookie=a75bd8cfe0819356 in_seq=0 out_seq=0 2024-03-24T00:19:12.154 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:12.151+0000 7f0533e00640 10 client.26513 ms_handle_connect on v2:172.21.15.189:6838/4074484168 2024-03-24T00:19:12.155 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:12.153+0000 7f0533e00640 1 -- 172.21.15.118:0/1682526981 <== mds.0 v2:172.21.15.189:6838/4074484168 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x562971d0f790 con 0x562971db2d60 2024-03-24T00:19:12.155 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:12.153+0000 7f0533e00640 10 client.26513 handle_command_reply: tid=0 2024-03-24T00:19:12.155 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:12.153+0000 7f055be57740 1 client.26513 shutdown 2024-03-24T00:19:12.155 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:12.154+0000 7f055be57740 1 -- 172.21.15.118:0/1682526981 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x562971d9d870 msgr2=0x562971da7420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:19:12.155 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:12.154+0000 7f055be57740 1 --2- 172.21.15.118:0/1682526981 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x562971d9d870 0x562971da7420 secure :-1 s=READY pgs=387 cs=0 l=1 rev1=1 crypto rx=0x7f05440ee720 tx=0x7f05440f42d0 comp rx=0 tx=0).stop 2024-03-24T00:19:12.156 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:12.154+0000 7f055be57740 1 -- 172.21.15.118:0/1682526981 shutdown_connections 2024-03-24T00:19:12.156 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:12.154+0000 7f055be57740 1 --2- 172.21.15.118:0/1682526981 >> [v2:172.21.15.189:6838/4074484168,v1:172.21.15.189:6839/4074484168] conn(0x562971db2d60 0x562971db5180 crc :-1 s=READY pgs=113 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:19:12.156 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:12.154+0000 7f055be57740 1 --2- 172.21.15.118:0/1682526981 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x562971da29e0 0x562971dae3f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:19:12.156 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:12.154+0000 7f055be57740 1 --2- 172.21.15.118:0/1682526981 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x562971d9d870 0x562971da7420 unknown :-1 s=CLOSED pgs=387 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:19:12.156 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:12.154+0000 7f0533e00640 0 client.26513 ms_handle_reset on v2:172.21.15.189:6838/4074484168 2024-03-24T00:19:12.156 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:12.154+0000 7f055be57740 1 --2- 172.21.15.118:0/1682526981 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x562971d9ce80 0x562971da6ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:19:12.156 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:12.154+0000 7f055be57740 1 -- 172.21.15.118:0/1682526981 >> 172.21.15.118:0/1682526981 conn(0x562971d65200 msgr2=0x562971d97a90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:19:12.156 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:12.154+0000 7f055be57740 1 -- 172.21.15.118:0/1682526981 shutdown_connections 2024-03-24T00:19:12.156 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:12.154+0000 7f055be57740 1 -- 172.21.15.118:0/1682526981 wait complete. 2024-03-24T00:19:12.156 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:12.154+0000 7f055be57740 20 client.26513 trim_cache size 0 max 16384 2024-03-24T00:19:12.174 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:12.173+0000 7f055be57740 1 Processor -- start 2024-03-24T00:19:12.174 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:12.173+0000 7f055be57740 1 -- start start 2024-03-24T00:19:12.174 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:12.174+0000 7f055be57740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x562971d65200 0x562971da1ee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:19:12.174 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:12.174+0000 7f055be57740 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x562971d9ce80 0x562971dae3b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:19:12.174 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:12.174+0000 7f055be57740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x562971d9d870 0x562971d9e100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:19:12.174 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:12.174+0000 7f055be57740 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f05541aabd0 con 0x562971d9ce80 2024-03-24T00:19:12.174 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:12.174+0000 7f055be57740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f05541ac550 con 0x562971d65200 2024-03-24T00:19:12.174 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:12.174+0000 7f055be57740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f05541ab230 con 0x562971d9d870 2024-03-24T00:19:12.175 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:12.174+0000 7f0551400640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x562971d9ce80 0x562971dae3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:19:12.175 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:12.174+0000 7f0551400640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x562971d9ce80 0x562971dae3b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.118:3300/0 says I am v2:172.21.15.118:53018/0 (socket says 172.21.15.118:53018) 2024-03-24T00:19:12.175 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:12.174+0000 7f0552800640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x562971d9d870 0x562971d9e100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:19:12.175 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:12.174+0000 7f0551400640 1 -- 172.21.15.118:0/673291068 learned_addr learned my addr 172.21.15.118:0/673291068 (peer_addr_for_me v2:172.21.15.118:0/0) 2024-03-24T00:19:12.175 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:12.174+0000 7f0551e00640 1 --2- 172.21.15.118:0/673291068 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x562971d65200 0x562971da1ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:19:12.175 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:12.174+0000 7f0552800640 1 -- 172.21.15.118:0/673291068 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x562971d65200 msgr2=0x562971da1ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:19:12.175 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:12.174+0000 7f0552800640 1 --2- 172.21.15.118:0/673291068 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x562971d65200 0x562971da1ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:19:12.175 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:12.174+0000 7f0552800640 1 -- 172.21.15.118:0/673291068 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x562971d9ce80 msgr2=0x562971dae3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:19:12.175 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:12.174+0000 7f0552800640 1 --2- 172.21.15.118:0/673291068 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x562971d9ce80 0x562971dae3b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:19:12.175 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:12.174+0000 7f0552800640 1 -- 172.21.15.118:0/673291068 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5629719d60f0 con 0x562971d9d870 2024-03-24T00:19:12.175 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:12.174+0000 7f0551400640 1 --2- 172.21.15.118:0/673291068 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x562971d9ce80 0x562971dae3b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T00:19:12.178 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:12.175+0000 7f0552800640 1 --2- 172.21.15.118:0/673291068 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x562971d9d870 0x562971d9e100 secure :-1 s=READY pgs=387 cs=0 l=1 rev1=1 crypto rx=0x7f054c009f90 tx=0x7f054c009bd0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d420f4c12ec8ecde server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:19:12.178 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:12.175+0000 7f053ea00640 1 -- 172.21.15.118:0/673291068 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f054c029070 con 0x562971d9d870 2024-03-24T00:19:12.179 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:12.175+0000 7f053ea00640 1 -- 172.21.15.118:0/673291068 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f054c003b70 con 0x562971d9d870 2024-03-24T00:19:12.179 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:12.175+0000 7f053ea00640 1 -- 172.21.15.118:0/673291068 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f054c0052a0 con 0x562971d9d870 2024-03-24T00:19:12.179 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:12.176+0000 7f055be57740 1 -- 172.21.15.118:0/673291068 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x562971d9d870 msgr2=0x562971d9e100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:19:12.179 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:12.176+0000 7f055be57740 1 --2- 172.21.15.118:0/673291068 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x562971d9d870 0x562971d9e100 secure :-1 s=READY pgs=387 cs=0 l=1 rev1=1 crypto rx=0x7f054c009f90 tx=0x7f054c009bd0 comp rx=0 tx=0).stop 2024-03-24T00:19:12.179 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:12.176+0000 7f055be57740 1 -- 172.21.15.118:0/673291068 shutdown_connections 2024-03-24T00:19:12.179 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:12.176+0000 7f055be57740 1 --2- 172.21.15.118:0/673291068 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x562971d9d870 0x562971d9e100 unknown :-1 s=CLOSED pgs=387 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:19:12.179 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:12.176+0000 7f055be57740 1 --2- 172.21.15.118:0/673291068 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x562971d9ce80 0x562971dae3b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:19:12.179 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:12.176+0000 7f055be57740 1 --2- 172.21.15.118:0/673291068 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x562971d65200 0x562971da1ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:19:12.179 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:12.176+0000 7f055be57740 1 -- 172.21.15.118:0/673291068 >> 172.21.15.118:0/673291068 conn(0x562971da6aa0 msgr2=0x562971d97a90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:19:12.179 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:12.176+0000 7f055be57740 1 -- 172.21.15.118:0/673291068 shutdown_connections 2024-03-24T00:19:12.179 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:12.176+0000 7f055be57740 1 -- 172.21.15.118:0/673291068 wait complete. 2024-03-24T00:19:12.179 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:12.177+0000 7f055be57740 1 -- start start 2024-03-24T00:19:12.179 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:12.177+0000 7f055be57740 1 Processor -- start 2024-03-24T00:19:12.179 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:12.177+0000 7f055be57740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x562971d65200 0x562971dac160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:19:12.180 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:12.177+0000 7f055be57740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x562971d9ce80 0x562971dac6a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:19:12.180 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:12.177+0000 7f055be57740 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x562971d9d870 0x562971dad6b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:19:12.180 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:12.177+0000 7f055be57740 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f05541abef0 con 0x562971d9d870 2024-03-24T00:19:12.180 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:12.177+0000 7f055be57740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f05541ab890 con 0x562971d65200 2024-03-24T00:19:12.180 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:12.177+0000 7f055be57740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f05541a8d90 con 0x562971d9ce80 2024-03-24T00:19:12.180 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:12.177+0000 7f055be57740 10 client.0 _ll_register_callbacks cb 0x7f05440035f0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T00:19:12.180 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:12.177+0000 7f0551400640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x562971d9ce80 0x562971dac6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:19:12.180 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:12.177+0000 7f0551e00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x562971d65200 0x562971dac160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:19:12.180 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:12.177+0000 7f0551400640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x562971d9ce80 0x562971dac6a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.118:47590/0 (socket says 172.21.15.118:47590) 2024-03-24T00:19:12.180 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:12.177+0000 7f0551400640 1 -- 172.21.15.118:0/1166681290 learned_addr learned my addr 172.21.15.118:0/1166681290 (peer_addr_for_me v2:172.21.15.118:0/0) 2024-03-24T00:19:12.180 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:12.177+0000 7f0552800640 1 --2- 172.21.15.118:0/1166681290 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x562971d9d870 0x562971dad6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:19:12.180 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:12.178+0000 7f0551400640 1 -- 172.21.15.118:0/1166681290 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x562971d65200 msgr2=0x562971dac160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:19:12.180 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:12.178+0000 7f0551400640 1 --2- 172.21.15.118:0/1166681290 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x562971d65200 0x562971dac160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:19:12.180 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:12.178+0000 7f0551400640 1 -- 172.21.15.118:0/1166681290 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x562971d9d870 msgr2=0x562971dad6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:19:12.181 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:12.178+0000 7f0551400640 1 --2- 172.21.15.118:0/1166681290 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x562971d9d870 0x562971dad6b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:19:12.181 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:12.178+0000 7f0551400640 1 -- 172.21.15.118:0/1166681290 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x562971dadca0 con 0x562971d9ce80 2024-03-24T00:19:12.181 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:12.178+0000 7f0551400640 1 --2- 172.21.15.118:0/1166681290 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x562971d9ce80 0x562971dac6a0 secure :-1 s=READY pgs=388 cs=0 l=1 rev1=1 crypto rx=0x7f054402c8b0 tx=0x7f05440bba80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=77abd4fe05ab4e6b server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:19:12.181 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:12.178+0000 7f0533e00640 10 client.0 ms_handle_connect on v2:172.21.15.193:3300/0 2024-03-24T00:19:12.181 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:12.178+0000 7f0533e00640 1 -- 172.21.15.118:0/1166681290 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f05440bced0 con 0x562971d9ce80 2024-03-24T00:19:12.181 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:12.178+0000 7f0533e00640 1 -- 172.21.15.118:0/1166681290 <== mon.1 v2:172.21.15.193:3300/0 2 ==== osd_map(90..90 src has 1..90) v4 ==== 8277+0+0 (secure 0 0 0) 0x7f05440d6370 con 0x562971d9ce80 2024-03-24T00:19:12.181 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:12.178+0000 7f055be57740 1 -- 172.21.15.118:0/1166681290 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f054c003b70 con 0x562971d9ce80 2024-03-24T00:19:12.181 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:12.179+0000 7f0533e00640 1 -- 172.21.15.118:0/1166681290 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) v2 ==== 24+0+0 (secure 0 0 0) 0x7f05440bd0b0 con 0x562971d9ce80 2024-03-24T00:19:12.181 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:12.179+0000 7f055be57740 10 client.26530 fetch_fsmap learned FSMap version 28 2024-03-24T00:19:12.181 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:12.179+0000 7f055be57740 1 -- 172.21.15.118:0/1166681290 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) v3 -- 0x562971e0ed70 con 0x562971d9ce80 2024-03-24T00:19:12.181 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:12.179+0000 7f0533e00640 1 -- 172.21.15.118:0/1166681290 <== mon.1 v2:172.21.15.193:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f054c003b70 con 0x562971d9ce80 2024-03-24T00:19:12.181 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:12.179+0000 7f0533e00640 1 -- 172.21.15.118:0/1166681290 <== mon.1 v2:172.21.15.193:3300/0 5 ==== fsmap(e 28) v1 ==== 7934+0+0 (secure 0 0 0) 0x7f05440f9040 con 0x562971d9ce80 2024-03-24T00:19:12.181 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:12.179+0000 7f055be57740 10 client.26530 fetch_fsmap finished waiting for FSMap version 28 2024-03-24T00:19:12.181 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:12.179+0000 7f055be57740 10 client.26530 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-24T00:19:12.181 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:12.179+0000 7f055be57740 20 client.26530 populate_metadata read hostname 'smithi118' 2024-03-24T00:19:12.182 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:12.179+0000 7f055be57740 1 --2- 172.21.15.118:0/1166681290 >> [v2:172.21.15.189:6838/4074484168,v1:172.21.15.189:6839/4074484168] conn(0x562971da29e0 0x562971e11510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:19:12.182 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:12.179+0000 7f055be57740 4 client.26530 mds_command: new command op to 24569 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-24T00:19:12.182 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:12.179+0000 7f055be57740 1 -- 172.21.15.118:0/1166681290 --> [v2:172.21.15.189:6838/4074484168,v1:172.21.15.189:6839/4074484168] -- command(tid 0: {"prefix": "scrub status"}) v1 -- 0x562971d09ae0 con 0x562971da29e0 2024-03-24T00:19:12.183 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:12.179+0000 7f0551e00640 1 --2- 172.21.15.118:0/1166681290 >> [v2:172.21.15.189:6838/4074484168,v1:172.21.15.189:6839/4074484168] conn(0x562971da29e0 0x562971e11510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:19:12.183 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:12.180+0000 7f0533e00640 10 client.26530 ms_handle_connect on v2:172.21.15.189:6838/4074484168 2024-03-24T00:19:12.183 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:12.180+0000 7f0551e00640 1 --2- 172.21.15.118:0/1166681290 >> [v2:172.21.15.189:6838/4074484168,v1:172.21.15.189:6839/4074484168] conn(0x562971da29e0 0x562971e11510 crc :-1 s=READY pgs=114 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=c677adc69c882af2 server_cookie=e02e6f2f1054e8ad in_seq=0 out_seq=0 2024-03-24T00:19:12.183 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:12.182+0000 7f0533e00640 1 -- 172.21.15.118:0/1166681290 <== mds.0 v2:172.21.15.189:6838/4074484168 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+268 (crc 0 0 0) 0x562971d09ae0 con 0x562971da29e0 2024-03-24T00:19:12.183 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:12.182+0000 7f0533e00640 10 client.26530 handle_command_reply: tid=0 2024-03-24T00:19:12.184 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:12.182+0000 7f055be57740 1 client.26530 shutdown 2024-03-24T00:19:12.184 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:12.182+0000 7f055be57740 1 -- 172.21.15.118:0/1166681290 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x562971d9ce80 msgr2=0x562971dac6a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:19:12.184 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:12.182+0000 7f055be57740 1 --2- 172.21.15.118:0/1166681290 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x562971d9ce80 0x562971dac6a0 secure :-1 s=READY pgs=388 cs=0 l=1 rev1=1 crypto rx=0x7f054402c8b0 tx=0x7f05440bba80 comp rx=0 tx=0).stop 2024-03-24T00:19:12.184 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:12.182+0000 7f055be57740 1 -- 172.21.15.118:0/1166681290 shutdown_connections 2024-03-24T00:19:12.184 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:12.182+0000 7f055be57740 1 --2- 172.21.15.118:0/1166681290 >> [v2:172.21.15.189:6838/4074484168,v1:172.21.15.189:6839/4074484168] conn(0x562971da29e0 0x562971e11510 crc :-1 s=READY pgs=114 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:19:12.184 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:12.182+0000 7f055be57740 1 --2- 172.21.15.118:0/1166681290 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x562971d9d870 0x562971dad6b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:19:12.184 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:12.182+0000 7f055be57740 1 --2- 172.21.15.118:0/1166681290 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x562971d9ce80 0x562971dac6a0 unknown :-1 s=CLOSED pgs=388 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:19:12.184 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:12.182+0000 7f0533e00640 0 client.26530 ms_handle_reset on v2:172.21.15.189:6838/4074484168 2024-03-24T00:19:12.184 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:12.182+0000 7f055be57740 1 --2- 172.21.15.118:0/1166681290 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x562971d65200 0x562971dac160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:19:12.184 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:12.182+0000 7f055be57740 1 -- 172.21.15.118:0/1166681290 >> 172.21.15.118:0/1166681290 conn(0x562971da6aa0 msgr2=0x562971d9ba30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:19:12.184 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:12.182+0000 7f055be57740 1 -- 172.21.15.118:0/1166681290 shutdown_connections 2024-03-24T00:19:12.184 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:12.182+0000 7f055be57740 1 -- 172.21.15.118:0/1166681290 wait complete. 2024-03-24T00:19:12.184 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:12.182+0000 7f055be57740 20 client.26530 trim_cache size 0 max 16384 2024-03-24T00:19:12.185 INFO:teuthology.orchestra.run.smithi118.stdout:{ 2024-03-24T00:19:12.185 INFO:teuthology.orchestra.run.smithi118.stdout: "status": "scrub active (3470 inodes in the stack)", 2024-03-24T00:19:12.185 INFO:teuthology.orchestra.run.smithi118.stdout: "scrubs": { 2024-03-24T00:19:12.185 INFO:teuthology.orchestra.run.smithi118.stdout: "01d80384-ff69-4b87-9df0-e818e48b6822": { 2024-03-24T00:19:12.185 INFO:teuthology.orchestra.run.smithi118.stdout: "path": "/", 2024-03-24T00:19:12.185 INFO:teuthology.orchestra.run.smithi118.stdout: "tag": "01d80384-ff69-4b87-9df0-e818e48b6822", 2024-03-24T00:19:12.185 INFO:teuthology.orchestra.run.smithi118.stdout: "options": "recursive,force" 2024-03-24T00:19:12.185 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-03-24T00:19:12.185 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-03-24T00:19:12.185 INFO:teuthology.orchestra.run.smithi118.stdout:} 2024-03-24T00:19:12.186 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:12.183+0000 7f0559200640 1 -- 172.21.15.118:0/3476096783 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7f05380a2800 msgr2=0x7f05380a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:19:12.186 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:12.183+0000 7f0559200640 1 --2- 172.21.15.118:0/3476096783 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7f05380a2800 0x7f05380a4cc0 secure :-1 s=READY pgs=339 cs=0 l=1 rev1=1 crypto rx=0x562971d9dc70 tx=0x7f054000b410 comp rx=0 tx=0).stop 2024-03-24T00:19:12.186 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:12.183+0000 7f0559200640 1 -- 172.21.15.118:0/3476096783 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0554102940 msgr2=0x7f055419a180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:19:12.186 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:12.183+0000 7f0559200640 1 --2- 172.21.15.118:0/3476096783 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0554102940 0x7f055419a180 secure :-1 s=READY pgs=385 cs=0 l=1 rev1=1 crypto rx=0x7f054402c160 tx=0x7f054402ce50 comp rx=0 tx=0).stop 2024-03-24T00:19:12.186 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:12.183+0000 7f0559200640 1 -- 172.21.15.118:0/3476096783 shutdown_connections 2024-03-24T00:19:12.186 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:12.183+0000 7f0559200640 1 --2- 172.21.15.118:0/3476096783 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7f05380a2800 0x7f05380a4cc0 unknown :-1 s=CLOSED pgs=339 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:19:12.186 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:12.183+0000 7f0559200640 1 --2- 172.21.15.118:0/3476096783 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f0554103300 0x7f05541a11b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:19:12.186 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:12.183+0000 7f0559200640 1 --2- 172.21.15.118:0/3476096783 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0554102940 0x7f055419a180 unknown :-1 s=CLOSED pgs=385 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:19:12.186 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:12.183+0000 7f0559200640 1 --2- 172.21.15.118:0/3476096783 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f05540ff970 0x7f0554199c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:19:12.187 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:12.183+0000 7f0559200640 1 -- 172.21.15.118:0/3476096783 >> 172.21.15.118:0/3476096783 conn(0x7f05540fb2c0 msgr2=0x7f0554100730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:19:12.187 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:12.183+0000 7f0559200640 1 -- 172.21.15.118:0/3476096783 shutdown_connections 2024-03-24T00:19:12.187 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:12.183+0000 7f0559200640 1 -- 172.21.15.118:0/3476096783 wait complete. 2024-03-24T00:19:12.207 INFO:tasks.cephfs.filesystem:scrub status for tag:01d80384-ff69-4b87-9df0-e818e48b6822 - {'path': '/', 'tag': '01d80384-ff69-4b87-9df0-e818e48b6822', 'options': 'recursive,force'} 2024-03-24T00:19:13.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:19:12 smithi118 ceph-mon[31535]: pgmap v503: 97 pgs: 97 active+clean; 72 MiB data, 597 MiB used, 1.0 TiB / 1.0 TiB avail; 771 KiB/s rd, 1.1 MiB/s wr, 206 op/s 2024-03-24T00:19:13.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:19:12 smithi118 ceph-mon[31535]: scrub queued for path: / 2024-03-24T00:19:13.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:19:12 smithi118 ceph-mon[31535]: scrub summary: idle+waiting paths [/] 2024-03-24T00:19:13.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:19:12 smithi118 ceph-mon[31535]: scrub summary: active paths [/] 2024-03-24T00:19:13.203 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:19:12 smithi189 ceph-mon[36187]: pgmap v503: 97 pgs: 97 active+clean; 72 MiB data, 597 MiB used, 1.0 TiB / 1.0 TiB avail; 771 KiB/s rd, 1.1 MiB/s wr, 206 op/s 2024-03-24T00:19:13.203 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:19:12 smithi189 ceph-mon[36187]: scrub queued for path: / 2024-03-24T00:19:13.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:19:12 smithi189 ceph-mon[36187]: scrub summary: idle+waiting paths [/] 2024-03-24T00:19:13.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:19:12 smithi189 ceph-mon[36187]: scrub summary: active paths [/] 2024-03-24T00:19:13.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:19:12 smithi193 ceph-mon[36878]: pgmap v503: 97 pgs: 97 active+clean; 72 MiB data, 597 MiB used, 1.0 TiB / 1.0 TiB avail; 771 KiB/s rd, 1.1 MiB/s wr, 206 op/s 2024-03-24T00:19:13.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:19:12 smithi193 ceph-mon[36878]: scrub queued for path: / 2024-03-24T00:19:13.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:19:12 smithi193 ceph-mon[36878]: scrub summary: idle+waiting paths [/] 2024-03-24T00:19:13.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:19:12 smithi193 ceph-mon[36878]: scrub summary: active paths [/] 2024-03-24T00:19:15.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:19:14 smithi118 ceph-mon[31535]: pgmap v504: 97 pgs: 97 active+clean; 72 MiB data, 628 MiB used, 1.0 TiB / 1.0 TiB avail; 840 KiB/s rd, 2.0 MiB/s wr, 427 op/s 2024-03-24T00:19:15.203 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:19:14 smithi189 ceph-mon[36187]: pgmap v504: 97 pgs: 97 active+clean; 72 MiB data, 628 MiB used, 1.0 TiB / 1.0 TiB avail; 840 KiB/s rd, 2.0 MiB/s wr, 427 op/s 2024-03-24T00:19:15.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:19:14 smithi193 ceph-mon[36878]: pgmap v504: 97 pgs: 97 active+clean; 72 MiB data, 628 MiB used, 1.0 TiB / 1.0 TiB avail; 840 KiB/s rd, 2.0 MiB/s wr, 427 op/s 2024-03-24T00:19:16.843 INFO:tasks.workunit.client.0.smithi118.stdout: 23 56369 57 38679 129 29053 275 2024-03-24T00:19:17.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:19:16 smithi118 ceph-mon[31535]: scrub summary: idle+waiting paths [/] 2024-03-24T00:19:17.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:19:16 smithi118 ceph-mon[31535]: pgmap v505: 97 pgs: 97 active+clean; 76 MiB data, 623 MiB used, 1.0 TiB / 1.0 TiB avail; 782 KiB/s rd, 2.1 MiB/s wr, 521 op/s 2024-03-24T00:19:17.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:19:16 smithi189 ceph-mon[36187]: scrub summary: idle+waiting paths [/] 2024-03-24T00:19:17.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:19:16 smithi189 ceph-mon[36187]: pgmap v505: 97 pgs: 97 active+clean; 76 MiB data, 623 MiB used, 1.0 TiB / 1.0 TiB avail; 782 KiB/s rd, 2.1 MiB/s wr, 521 op/s 2024-03-24T00:19:17.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:19:16 smithi193 ceph-mon[36878]: scrub summary: idle+waiting paths [/] 2024-03-24T00:19:17.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:19:16 smithi193 ceph-mon[36878]: pgmap v505: 97 pgs: 97 active+clean; 76 MiB data, 623 MiB used, 1.0 TiB / 1.0 TiB avail; 782 KiB/s rd, 2.1 MiB/s wr, 521 op/s 2024-03-24T00:19:19.195 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:19:18 smithi118 ceph-mon[31535]: pgmap v506: 97 pgs: 97 active+clean; 77 MiB data, 592 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 2.2 MiB/s wr, 665 op/s 2024-03-24T00:19:19.203 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:19:18 smithi189 ceph-mon[36187]: pgmap v506: 97 pgs: 97 active+clean; 77 MiB data, 592 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 2.2 MiB/s wr, 665 op/s 2024-03-24T00:19:19.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:19:18 smithi193 ceph-mon[36878]: pgmap v506: 97 pgs: 97 active+clean; 77 MiB data, 592 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 2.2 MiB/s wr, 665 op/s 2024-03-24T00:19:21.195 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:19:20 smithi118 ceph-mon[31535]: pgmap v507: 97 pgs: 97 active+clean; 77 MiB data, 592 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 2.2 MiB/s wr, 715 op/s 2024-03-24T00:19:21.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:19:20 smithi118 ceph-mon[31535]: scrub summary: idle 2024-03-24T00:19:21.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:19:20 smithi189 ceph-mon[36187]: pgmap v507: 97 pgs: 97 active+clean; 77 MiB data, 592 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 2.2 MiB/s wr, 715 op/s 2024-03-24T00:19:21.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:19:20 smithi189 ceph-mon[36187]: scrub summary: idle 2024-03-24T00:19:21.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:19:20 smithi193 ceph-mon[36878]: pgmap v507: 97 pgs: 97 active+clean; 77 MiB data, 592 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 2.2 MiB/s wr, 715 op/s 2024-03-24T00:19:21.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:19:20 smithi193 ceph-mon[36878]: scrub summary: idle 2024-03-24T00:19:21.651 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:19:21.651+0000 7ff194200640 1 -- 172.21.15.118:0/326888654 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff18c1054a0 msgr2=0x7ff18c10bd30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:19:21.651 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:19:21.651+0000 7ff194200640 1 --2- 172.21.15.118:0/326888654 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff18c1054a0 0x7ff18c10bd30 secure :-1 s=READY pgs=388 cs=0 l=1 rev1=1 crypto rx=0x7ff184009ca0 tx=0x7ff18402c100 comp rx=0 tx=0).stop 2024-03-24T00:19:21.651 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:19:21.651+0000 7ff194200640 1 -- 172.21.15.118:0/326888654 shutdown_connections 2024-03-24T00:19:21.651 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:19:21.651+0000 7ff194200640 1 --2- 172.21.15.118:0/326888654 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff18c1054a0 0x7ff18c10bd30 unknown :-1 s=CLOSED pgs=388 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:19:21.651 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:19:21.651+0000 7ff194200640 1 --2- 172.21.15.118:0/326888654 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7ff18c104ae0 0x7ff18c104f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:19:21.651 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:19:21.651+0000 7ff194200640 1 --2- 172.21.15.118:0/326888654 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7ff18c1038f0 0x7ff18c103d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:19:21.652 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:19:21.651+0000 7ff194200640 1 -- 172.21.15.118:0/326888654 >> 172.21.15.118:0/326888654 conn(0x7ff18c0fee60 msgr2=0x7ff18c1012a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:19:21.652 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:19:21.651+0000 7ff194200640 1 -- 172.21.15.118:0/326888654 shutdown_connections 2024-03-24T00:19:21.652 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:19:21.651+0000 7ff194200640 1 -- 172.21.15.118:0/326888654 wait complete. 2024-03-24T00:19:21.654 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:19:21.652+0000 7ff194200640 1 Processor -- start 2024-03-24T00:19:21.655 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:19:21.653+0000 7ff194200640 1 -- start start 2024-03-24T00:19:21.655 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:19:21.653+0000 7ff194200640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff18c1038f0 0x7ff18c199ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:19:21.655 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:19:21.653+0000 7ff194200640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7ff18c104ae0 0x7ff18c19a000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:19:21.655 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:19:21.653+0000 7ff194200640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7ff18c1054a0 0x7ff18c1a1070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:19:21.655 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:19:21.653+0000 7ff194200640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff18c10f570 con 0x7ff18c104ae0 2024-03-24T00:19:21.655 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:19:21.653+0000 7ff194200640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff18c10f3f0 con 0x7ff18c1038f0 2024-03-24T00:19:21.655 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:19:21.653+0000 7ff194200640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff18c10f6f0 con 0x7ff18c1054a0 2024-03-24T00:19:21.655 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:19:21.653+0000 7ff191800640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff18c1038f0 0x7ff18c199ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:19:21.655 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:19:21.653+0000 7ff191800640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff18c1038f0 0x7ff18c199ac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.118:38382/0 (socket says 172.21.15.118:38382) 2024-03-24T00:19:21.655 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:19:21.653+0000 7ff191800640 1 -- 172.21.15.118:0/320181713 learned_addr learned my addr 172.21.15.118:0/320181713 (peer_addr_for_me v2:172.21.15.118:0/0) 2024-03-24T00:19:21.655 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:19:21.653+0000 7ff192200640 1 --2- 172.21.15.118:0/320181713 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7ff18c1054a0 0x7ff18c1a1070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:19:21.655 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:19:21.653+0000 7ff190e00640 1 --2- 172.21.15.118:0/320181713 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7ff18c104ae0 0x7ff18c19a000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:19:21.655 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:19:21.653+0000 7ff191800640 1 -- 172.21.15.118:0/320181713 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7ff18c1054a0 msgr2=0x7ff18c1a1070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:19:21.656 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:19:21.653+0000 7ff191800640 1 --2- 172.21.15.118:0/320181713 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7ff18c1054a0 0x7ff18c1a1070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:19:21.656 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:19:21.654+0000 7ff191800640 1 -- 172.21.15.118:0/320181713 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7ff18c104ae0 msgr2=0x7ff18c19a000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:19:21.656 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:19:21.654+0000 7ff191800640 1 --2- 172.21.15.118:0/320181713 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7ff18c104ae0 0x7ff18c19a000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:19:21.656 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:19:21.654+0000 7ff191800640 1 -- 172.21.15.118:0/320181713 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff18c1a16e0 con 0x7ff18c1038f0 2024-03-24T00:19:21.656 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:19:21.654+0000 7ff192200640 1 --2- 172.21.15.118:0/320181713 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7ff18c1054a0 0x7ff18c1a1070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T00:19:21.656 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:19:21.654+0000 7ff190e00640 1 --2- 172.21.15.118:0/320181713 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7ff18c104ae0 0x7ff18c19a000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T00:19:21.656 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:19:21.654+0000 7ff191800640 1 --2- 172.21.15.118:0/320181713 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff18c1038f0 0x7ff18c199ac0 secure :-1 s=READY pgs=389 cs=0 l=1 rev1=1 crypto rx=0x7ff18000e9c0 tx=0x7ff18000ee80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:19:21.656 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:19:21.654+0000 7ff18a000640 1 -- 172.21.15.118:0/320181713 <== mon.2 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff18000cd80 con 0x7ff18c1038f0 2024-03-24T00:19:21.656 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:19:21.654+0000 7ff194200640 1 -- 172.21.15.118:0/320181713 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff18c1a19d0 con 0x7ff18c1038f0 2024-03-24T00:19:21.656 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:19:21.654+0000 7ff18a000640 1 -- 172.21.15.118:0/320181713 <== mon.2 v2:172.21.15.189:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7ff1800035f0 con 0x7ff18c1038f0 2024-03-24T00:19:21.656 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:19:21.654+0000 7ff18a000640 1 -- 172.21.15.118:0/320181713 <== mon.2 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff180010690 con 0x7ff18c1038f0 2024-03-24T00:19:21.656 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:19:21.654+0000 7ff194200640 1 -- 172.21.15.118:0/320181713 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff18c1a1f70 con 0x7ff18c1038f0 2024-03-24T00:19:21.658 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:19:21.658+0000 7ff197036740 1 Processor -- start 2024-03-24T00:19:21.659 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:19:21.659+0000 7ff197036740 1 -- start start 2024-03-24T00:19:21.659 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:19:21.659+0000 7ff18a000640 1 -- 172.21.15.118:0/320181713 <== mon.2 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 20) v1 ==== 137291+0+0 (secure 0 0 0) 0x7ff1800037a0 con 0x7ff18c1038f0 2024-03-24T00:19:21.659 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:19:21.659+0000 7ff197036740 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x560cb2a791e0 0x560cb2a79600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:19:21.659 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:19:21.659+0000 7ff197036740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x560cb2a79bd0 0x560cb2a7e6d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:19:21.659 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:19:21.659+0000 7ff197036740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x560cb2a7ed40 0x560cb2a811d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:19:21.659 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:19:21.659+0000 7ff197036740 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x560cb2a3c3b0 con 0x560cb2a791e0 2024-03-24T00:19:21.659 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:19:21.659+0000 7ff197036740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x560cb29de830 con 0x560cb2a7ed40 2024-03-24T00:19:21.659 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:19:21.659+0000 7ff197036740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x560cb29875d0 con 0x560cb2a79bd0 2024-03-24T00:19:21.659 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:19:21.659+0000 7ff191800640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x560cb2a79bd0 0x560cb2a7e6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:19:21.660 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:19:21.659+0000 7ff190e00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x560cb2a7ed40 0x560cb2a811d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:19:21.660 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:19:21.659+0000 7ff191800640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x560cb2a79bd0 0x560cb2a7e6d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.118:50902/0 (socket says 172.21.15.118:50902) 2024-03-24T00:19:21.660 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:19:21.659+0000 7ff191800640 1 -- 172.21.15.118:0/3899854141 learned_addr learned my addr 172.21.15.118:0/3899854141 (peer_addr_for_me v2:172.21.15.118:0/0) 2024-03-24T00:19:21.660 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:19:21.659+0000 7ff192200640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x560cb2a791e0 0x560cb2a79600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:19:21.661 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:19:21.660+0000 7ff191800640 1 -- 172.21.15.118:0/3899854141 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x560cb2a7ed40 msgr2=0x560cb2a811d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:19:21.661 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:19:21.660+0000 7ff191800640 1 --2- 172.21.15.118:0/3899854141 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x560cb2a7ed40 0x560cb2a811d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:19:21.661 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:19:21.660+0000 7ff191800640 1 -- 172.21.15.118:0/3899854141 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x560cb2a791e0 msgr2=0x560cb2a79600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:19:21.661 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:19:21.660+0000 7ff191800640 1 --2- 172.21.15.118:0/3899854141 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x560cb2a791e0 0x560cb2a79600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:19:21.661 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:19:21.660+0000 7ff191800640 1 -- 172.21.15.118:0/3899854141 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x560cb299f2c0 con 0x560cb2a79bd0 2024-03-24T00:19:21.661 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:19:21.660+0000 7ff190e00640 1 --2- 172.21.15.118:0/3899854141 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x560cb2a7ed40 0x560cb2a811d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T00:19:21.661 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:19:21.660+0000 7ff191800640 1 --2- 172.21.15.118:0/3899854141 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x560cb2a79bd0 0x560cb2a7e6d0 secure :-1 s=READY pgs=389 cs=0 l=1 rev1=1 crypto rx=0x7ff1800d2e70 tx=0x7ff1800d74b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=dbdbc0c9813048c1 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:19:21.661 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:19:21.660+0000 7ff17ea00640 1 -- 172.21.15.118:0/3899854141 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff1800e2070 con 0x560cb2a79bd0 2024-03-24T00:19:21.661 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:19:21.661+0000 7ff17ea00640 1 -- 172.21.15.118:0/3899854141 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7ff1800ddae0 con 0x560cb2a79bd0 2024-03-24T00:19:21.661 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:19:21.661+0000 7ff17ea00640 1 -- 172.21.15.118:0/3899854141 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff1800dcaf0 con 0x560cb2a79bd0 2024-03-24T00:19:21.662 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:19:21.661+0000 7ff18a000640 1 --2- 172.21.15.118:0/320181713 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7ff1740a28a0 0x7ff1740a4d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:19:21.662 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:19:21.661+0000 7ff18a000640 1 -- 172.21.15.118:0/320181713 <== mon.2 v2:172.21.15.189:3300/0 5 ==== osd_map(90..90 src has 1..90) v4 ==== 8277+0+0 (secure 0 0 0) 0x7ff1800ceef0 con 0x7ff18c1038f0 2024-03-24T00:19:21.662 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:19:21.661+0000 7ff197036740 1 -- 172.21.15.118:0/3899854141 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x560cb2a79bd0 msgr2=0x560cb2a7e6d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:19:21.662 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:19:21.661+0000 7ff197036740 1 --2- 172.21.15.118:0/3899854141 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x560cb2a79bd0 0x560cb2a7e6d0 secure :-1 s=READY pgs=389 cs=0 l=1 rev1=1 crypto rx=0x7ff1800d2e70 tx=0x7ff1800d74b0 comp rx=0 tx=0).stop 2024-03-24T00:19:21.663 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:19:21.662+0000 7ff192200640 1 --2- 172.21.15.118:0/320181713 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7ff1740a28a0 0x7ff1740a4d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:19:21.663 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:19:21.663+0000 7ff197036740 1 -- 172.21.15.118:0/3899854141 shutdown_connections 2024-03-24T00:19:21.663 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:19:21.663+0000 7ff197036740 1 --2- 172.21.15.118:0/3899854141 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x560cb2a7ed40 0x560cb2a811d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:19:21.663 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:19:21.663+0000 7ff197036740 1 --2- 172.21.15.118:0/3899854141 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x560cb2a79bd0 0x560cb2a7e6d0 unknown :-1 s=CLOSED pgs=389 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:19:21.663 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:19:21.663+0000 7ff197036740 1 --2- 172.21.15.118:0/3899854141 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x560cb2a791e0 0x560cb2a79600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:19:21.663 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:19:21.663+0000 7ff197036740 1 -- 172.21.15.118:0/3899854141 >> 172.21.15.118:0/3899854141 conn(0x560cb2a74750 msgr2=0x560cb2a74b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:19:21.663 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:19:21.663+0000 7ff197036740 1 -- 172.21.15.118:0/3899854141 shutdown_connections 2024-03-24T00:19:21.663 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:19:21.663+0000 7ff192200640 1 --2- 172.21.15.118:0/320181713 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7ff1740a28a0 0x7ff1740a4d60 secure :-1 s=READY pgs=340 cs=0 l=1 rev1=1 crypto rx=0x7ff18402c640 tx=0x7ff184032590 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:19:21.663 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:19:21.663+0000 7ff197036740 1 -- 172.21.15.118:0/3899854141 wait complete. 2024-03-24T00:19:21.664 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:19:21.664+0000 7ff197036740 1 -- start start 2024-03-24T00:19:21.664 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:19:21.664+0000 7ff197036740 1 Processor -- start 2024-03-24T00:19:21.664 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:19:21.664+0000 7ff197036740 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x560cb2a791e0 0x560cb2a833c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:19:21.664 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:19:21.664+0000 7ff197036740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x560cb2a79bd0 0x560cb2a83900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:19:21.664 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:19:21.664+0000 7ff197036740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x560cb2a7ed40 0x560cb2a8a940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:19:21.664 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:19:21.664+0000 7ff197036740 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x560cb299aa20 con 0x560cb2a791e0 2024-03-24T00:19:21.666 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:19:21.665+0000 7ff197036740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x560cb29da8d0 con 0x560cb2a79bd0 2024-03-24T00:19:21.666 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:19:21.665+0000 7ff197036740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x560cb2a81cf0 con 0x560cb2a7ed40 2024-03-24T00:19:21.666 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:19:21.665+0000 7ff197036740 10 client.0 _ll_register_callbacks cb 0x560cb29d71c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T00:19:21.666 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:19:21.665+0000 7ff192200640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x560cb2a7ed40 0x560cb2a8a940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:19:21.666 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:19:21.665+0000 7ff191800640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x560cb2a791e0 0x560cb2a833c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:19:21.666 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:19:21.665+0000 7ff192200640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x560cb2a7ed40 0x560cb2a8a940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.118:50912/0 (socket says 172.21.15.118:50912) 2024-03-24T00:19:21.666 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:19:21.665+0000 7ff192200640 1 -- 172.21.15.118:0/3509534575 learned_addr learned my addr 172.21.15.118:0/3509534575 (peer_addr_for_me v2:172.21.15.118:0/0) 2024-03-24T00:19:21.666 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:19:21.665+0000 7ff191800640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x560cb2a791e0 0x560cb2a833c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.118:3300/0 says I am v2:172.21.15.118:39804/0 (socket says 172.21.15.118:39804) 2024-03-24T00:19:21.666 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:19:21.665+0000 7ff190e00640 1 --2- 172.21.15.118:0/3509534575 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x560cb2a79bd0 0x560cb2a83900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:19:21.666 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:19:21.665+0000 7ff190e00640 1 -- 172.21.15.118:0/3509534575 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x560cb2a7ed40 msgr2=0x560cb2a8a940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:19:21.667 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:19:21.665+0000 7ff190e00640 1 --2- 172.21.15.118:0/3509534575 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x560cb2a7ed40 0x560cb2a8a940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:19:21.667 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:19:21.665+0000 7ff190e00640 1 -- 172.21.15.118:0/3509534575 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x560cb2a791e0 msgr2=0x560cb2a833c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:19:21.667 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:19:21.665+0000 7ff190e00640 1 --2- 172.21.15.118:0/3509534575 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x560cb2a791e0 0x560cb2a833c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:19:21.667 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:19:21.665+0000 7ff190e00640 1 -- 172.21.15.118:0/3509534575 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x560cb26f3210 con 0x560cb2a79bd0 2024-03-24T00:19:21.667 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:19:21.666+0000 7ff191800640 1 --2- 172.21.15.118:0/3509534575 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x560cb2a791e0 0x560cb2a833c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T00:19:21.667 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:19:21.666+0000 7ff190e00640 1 --2- 172.21.15.118:0/3509534575 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x560cb2a79bd0 0x560cb2a83900 secure :-1 s=READY pgs=390 cs=0 l=1 rev1=1 crypto rx=0x7ff18c19afd0 tx=0x7ff178012b10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=eca606435d361ee4 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:19:21.667 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:19:21.666+0000 7ff173400640 10 client.0 ms_handle_connect on v2:172.21.15.189:3300/0 2024-03-24T00:19:21.667 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:19:21.666+0000 7ff173400640 1 -- 172.21.15.118:0/3509534575 <== mon.2 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff178011c70 con 0x560cb2a79bd0 2024-03-24T00:19:21.667 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:19:21.666+0000 7ff173400640 1 -- 172.21.15.118:0/3509534575 <== mon.2 v2:172.21.15.189:3300/0 2 ==== osd_map(90..90 src has 1..90) v4 ==== 8277+0+0 (secure 0 0 0) 0x7ff178011e10 con 0x560cb2a79bd0 2024-03-24T00:19:21.667 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:19:21.666+0000 7ff197036740 1 -- 172.21.15.118:0/3509534575 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7ff1800ddae0 con 0x560cb2a79bd0 2024-03-24T00:19:21.667 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:19:21.666+0000 7ff173400640 1 -- 172.21.15.118:0/3509534575 <== mon.2 v2:172.21.15.189:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) v2 ==== 24+0+0 (secure 0 0 0) 0x7ff178022030 con 0x560cb2a79bd0 2024-03-24T00:19:21.667 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:19:21.666+0000 7ff197036740 10 client.26531 fetch_fsmap learned FSMap version 28 2024-03-24T00:19:21.667 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:19:21.666+0000 7ff197036740 1 -- 172.21.15.118:0/3509534575 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) v3 -- 0x560cb27bf620 con 0x560cb2a79bd0 2024-03-24T00:19:21.668 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:19:21.667+0000 7ff173400640 1 -- 172.21.15.118:0/3509534575 <== mon.2 v2:172.21.15.189:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7ff1800ddae0 con 0x560cb2a79bd0 2024-03-24T00:19:21.668 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:19:21.667+0000 7ff173400640 1 -- 172.21.15.118:0/3509534575 <== mon.2 v2:172.21.15.189:3300/0 5 ==== fsmap(e 28) v1 ==== 7934+0+0 (secure 0 0 0) 0x7ff178010660 con 0x560cb2a79bd0 2024-03-24T00:19:21.669 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:19:21.667+0000 7ff197036740 10 client.26531 fetch_fsmap finished waiting for FSMap version 28 2024-03-24T00:19:21.669 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:19:21.667+0000 7ff197036740 10 client.26531 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-24T00:19:21.669 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:19:21.667+0000 7ff197036740 20 client.26531 populate_metadata read hostname 'smithi118' 2024-03-24T00:19:21.669 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:19:21.667+0000 7ff197036740 1 --2- 172.21.15.118:0/3509534575 >> [v2:172.21.15.189:6838/4074484168,v1:172.21.15.189:6839/4074484168] conn(0x560cb2a8f240 0x560cb2a91660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:19:21.669 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:19:21.667+0000 7ff197036740 4 client.26531 mds_command: new command op to 24569 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-24T00:19:21.669 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:19:21.667+0000 7ff197036740 1 -- 172.21.15.118:0/3509534575 --> [v2:172.21.15.189:6838/4074484168,v1:172.21.15.189:6839/4074484168] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x560cb29ebd70 con 0x560cb2a8f240 2024-03-24T00:19:21.669 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:19:21.667+0000 7ff191800640 1 --2- 172.21.15.118:0/3509534575 >> [v2:172.21.15.189:6838/4074484168,v1:172.21.15.189:6839/4074484168] conn(0x560cb2a8f240 0x560cb2a91660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:19:21.669 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:19:21.668+0000 7ff191800640 1 --2- 172.21.15.118:0/3509534575 >> [v2:172.21.15.189:6838/4074484168,v1:172.21.15.189:6839/4074484168] conn(0x560cb2a8f240 0x560cb2a91660 crc :-1 s=READY pgs=115 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=aca1d3f0b63fdee server_cookie=d1ff8dfff98c73e5 in_seq=0 out_seq=0 2024-03-24T00:19:21.669 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:19:21.668+0000 7ff173400640 10 client.26531 ms_handle_connect on v2:172.21.15.189:6838/4074484168 2024-03-24T00:19:21.673 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:19:21.673+0000 7ff173400640 1 -- 172.21.15.118:0/3509534575 <== mds.0 v2:172.21.15.189:6838/4074484168 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x560cb29ebd70 con 0x560cb2a8f240 2024-03-24T00:19:21.673 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:19:21.673+0000 7ff173400640 10 client.26531 handle_command_reply: tid=0 2024-03-24T00:19:21.673 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:19:21.673+0000 7ff197036740 1 client.26531 shutdown 2024-03-24T00:19:21.673 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:19:21.673+0000 7ff197036740 1 -- 172.21.15.118:0/3509534575 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x560cb2a79bd0 msgr2=0x560cb2a83900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:19:21.673 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:19:21.673+0000 7ff197036740 1 --2- 172.21.15.118:0/3509534575 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x560cb2a79bd0 0x560cb2a83900 secure :-1 s=READY pgs=390 cs=0 l=1 rev1=1 crypto rx=0x7ff18c19afd0 tx=0x7ff178012b10 comp rx=0 tx=0).stop 2024-03-24T00:19:21.673 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:19:21.673+0000 7ff197036740 1 -- 172.21.15.118:0/3509534575 shutdown_connections 2024-03-24T00:19:21.674 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:19:21.673+0000 7ff197036740 1 --2- 172.21.15.118:0/3509534575 >> [v2:172.21.15.189:6838/4074484168,v1:172.21.15.189:6839/4074484168] conn(0x560cb2a8f240 0x560cb2a91660 crc :-1 s=READY pgs=115 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:19:21.674 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:19:21.673+0000 7ff197036740 1 --2- 172.21.15.118:0/3509534575 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x560cb2a7ed40 0x560cb2a8a940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:19:21.674 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:19:21.673+0000 7ff197036740 1 --2- 172.21.15.118:0/3509534575 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x560cb2a79bd0 0x560cb2a83900 unknown :-1 s=CLOSED pgs=390 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:19:21.674 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:19:21.673+0000 7ff197036740 1 --2- 172.21.15.118:0/3509534575 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x560cb2a791e0 0x560cb2a833c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:19:21.674 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:19:21.673+0000 7ff173400640 0 client.26531 ms_handle_reset on v2:172.21.15.189:6838/4074484168 2024-03-24T00:19:21.674 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:19:21.673+0000 7ff197036740 1 -- 172.21.15.118:0/3509534575 >> 172.21.15.118:0/3509534575 conn(0x560cb2a74750 msgr2=0x560cb2a73950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:19:21.674 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:19:21.673+0000 7ff197036740 1 -- 172.21.15.118:0/3509534575 shutdown_connections 2024-03-24T00:19:21.674 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:19:21.673+0000 7ff197036740 1 -- 172.21.15.118:0/3509534575 wait complete. 2024-03-24T00:19:21.674 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:19:21.674+0000 7ff197036740 20 client.26531 trim_cache size 0 max 16384 2024-03-24T00:19:21.689 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:19:21.688+0000 7ff197036740 1 Processor -- start 2024-03-24T00:19:21.690 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:19:21.689+0000 7ff197036740 1 -- start start 2024-03-24T00:19:21.690 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:19:21.689+0000 7ff197036740 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x560cb2a791e0 0x560cb2a7af30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:19:21.690 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:19:21.689+0000 7ff197036740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x560cb2a79bd0 0x560cb2a7b470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:19:21.690 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:19:21.689+0000 7ff197036740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x560cb2a7ed40 0x560cb2a8a900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:19:21.690 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:19:21.689+0000 7ff197036740 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff18c1aabb0 con 0x560cb2a791e0 2024-03-24T00:19:21.690 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:19:21.689+0000 7ff197036740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff18c1ac530 con 0x560cb2a79bd0 2024-03-24T00:19:21.690 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:19:21.689+0000 7ff197036740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff18c1ab210 con 0x560cb2a7ed40 2024-03-24T00:19:21.690 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:19:21.689+0000 7ff191800640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x560cb2a791e0 0x560cb2a7af30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:19:21.690 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:19:21.689+0000 7ff190e00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x560cb2a79bd0 0x560cb2a7b470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:19:21.690 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:19:21.689+0000 7ff192200640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x560cb2a7ed40 0x560cb2a8a900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:19:21.690 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:19:21.689+0000 7ff190e00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x560cb2a79bd0 0x560cb2a7b470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.118:38410/0 (socket says 172.21.15.118:38410) 2024-03-24T00:19:21.690 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:19:21.689+0000 7ff192200640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x560cb2a7ed40 0x560cb2a8a900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.118:50920/0 (socket says 172.21.15.118:50920) 2024-03-24T00:19:21.690 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:19:21.689+0000 7ff191800640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x560cb2a791e0 0x560cb2a7af30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.118:3300/0 says I am v2:172.21.15.118:39806/0 (socket says 172.21.15.118:39806) 2024-03-24T00:19:21.691 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:19:21.689+0000 7ff190e00640 1 -- 172.21.15.118:0/17709760 learned_addr learned my addr 172.21.15.118:0/17709760 (peer_addr_for_me v2:172.21.15.118:0/0) 2024-03-24T00:19:21.691 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:19:21.689+0000 7ff192200640 1 -- 172.21.15.118:0/17709760 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x560cb2a79bd0 msgr2=0x560cb2a7b470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:19:21.691 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:19:21.689+0000 7ff192200640 1 --2- 172.21.15.118:0/17709760 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x560cb2a79bd0 0x560cb2a7b470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:19:21.691 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:19:21.689+0000 7ff192200640 1 -- 172.21.15.118:0/17709760 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x560cb2a791e0 msgr2=0x560cb2a7af30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:19:21.691 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:19:21.689+0000 7ff192200640 1 --2- 172.21.15.118:0/17709760 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x560cb2a791e0 0x560cb2a7af30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:19:21.691 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:19:21.690+0000 7ff192200640 1 -- 172.21.15.118:0/17709760 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x560cb2a81e30 con 0x560cb2a7ed40 2024-03-24T00:19:21.691 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:19:21.690+0000 7ff191800640 1 --2- 172.21.15.118:0/17709760 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x560cb2a791e0 0x560cb2a7af30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T00:19:21.691 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:19:21.690+0000 7ff192200640 1 --2- 172.21.15.118:0/17709760 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x560cb2a7ed40 0x560cb2a8a900 secure :-1 s=READY pgs=390 cs=0 l=1 rev1=1 crypto rx=0x7ff184037040 tx=0x7ff184041000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=9df5668a3c6bf7c server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:19:21.694 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:19:21.690+0000 7ff17e000640 1 -- 172.21.15.118:0/17709760 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff184039040 con 0x560cb2a7ed40 2024-03-24T00:19:21.694 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:19:21.690+0000 7ff17e000640 1 -- 172.21.15.118:0/17709760 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7ff1840347f0 con 0x560cb2a7ed40 2024-03-24T00:19:21.694 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:19:21.690+0000 7ff17e000640 1 -- 172.21.15.118:0/17709760 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff184004460 con 0x560cb2a7ed40 2024-03-24T00:19:21.694 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:19:21.691+0000 7ff197036740 1 -- 172.21.15.118:0/17709760 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x560cb2a7ed40 msgr2=0x560cb2a8a900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:19:21.695 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:19:21.691+0000 7ff197036740 1 --2- 172.21.15.118:0/17709760 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x560cb2a7ed40 0x560cb2a8a900 secure :-1 s=READY pgs=390 cs=0 l=1 rev1=1 crypto rx=0x7ff184037040 tx=0x7ff184041000 comp rx=0 tx=0).stop 2024-03-24T00:19:21.695 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:19:21.691+0000 7ff197036740 1 -- 172.21.15.118:0/17709760 shutdown_connections 2024-03-24T00:19:21.695 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:19:21.691+0000 7ff197036740 1 --2- 172.21.15.118:0/17709760 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x560cb2a7ed40 0x560cb2a8a900 unknown :-1 s=CLOSED pgs=390 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:19:21.695 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:19:21.691+0000 7ff197036740 1 --2- 172.21.15.118:0/17709760 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x560cb2a79bd0 0x560cb2a7b470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:19:21.695 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:19:21.691+0000 7ff197036740 1 --2- 172.21.15.118:0/17709760 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x560cb2a791e0 0x560cb2a7af30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:19:21.695 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:19:21.691+0000 7ff197036740 1 -- 172.21.15.118:0/17709760 >> 172.21.15.118:0/17709760 conn(0x560cb2a74750 msgr2=0x560cb2a74b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:19:21.695 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:19:21.691+0000 7ff197036740 1 -- 172.21.15.118:0/17709760 shutdown_connections 2024-03-24T00:19:21.695 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:19:21.691+0000 7ff197036740 1 -- 172.21.15.118:0/17709760 wait complete. 2024-03-24T00:19:21.695 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:19:21.691+0000 7ff197036740 1 -- start start 2024-03-24T00:19:21.695 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:19:21.692+0000 7ff197036740 1 Processor -- start 2024-03-24T00:19:21.695 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:19:21.692+0000 7ff197036740 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x560cb2a791e0 0x560cb2a87c20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:19:21.695 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:19:21.692+0000 7ff197036740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x560cb2a79bd0 0x560cb2a88160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:19:21.695 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:19:21.692+0000 7ff197036740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x560cb2a7ed40 0x560cb2a89100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:19:21.695 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:19:21.692+0000 7ff197036740 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff18c1abed0 con 0x560cb2a791e0 2024-03-24T00:19:21.696 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:19:21.692+0000 7ff197036740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff18c1ab870 con 0x560cb2a79bd0 2024-03-24T00:19:21.696 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:19:21.692+0000 7ff197036740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff18c1a8d70 con 0x560cb2a7ed40 2024-03-24T00:19:21.696 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:19:21.692+0000 7ff197036740 10 client.0 _ll_register_callbacks cb 0x7ff178009840 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T00:19:21.696 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:19:21.692+0000 7ff191800640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x560cb2a791e0 0x560cb2a87c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:19:21.696 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:19:21.692+0000 7ff190e00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x560cb2a79bd0 0x560cb2a88160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:19:21.696 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:19:21.692+0000 7ff190e00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x560cb2a79bd0 0x560cb2a88160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.118:38420/0 (socket says 172.21.15.118:38420) 2024-03-24T00:19:21.696 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:19:21.692+0000 7ff191800640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x560cb2a791e0 0x560cb2a87c20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.118:3300/0 says I am v2:172.21.15.118:39822/0 (socket says 172.21.15.118:39822) 2024-03-24T00:19:21.696 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:19:21.692+0000 7ff190e00640 1 -- 172.21.15.118:0/3005588200 learned_addr learned my addr 172.21.15.118:0/3005588200 (peer_addr_for_me v2:172.21.15.118:0/0) 2024-03-24T00:19:21.696 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:19:21.692+0000 7ff192200640 1 --2- 172.21.15.118:0/3005588200 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x560cb2a7ed40 0x560cb2a89100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:19:21.696 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:19:21.693+0000 7ff191800640 1 -- 172.21.15.118:0/3005588200 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x560cb2a7ed40 msgr2=0x560cb2a89100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:19:21.696 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:19:21.693+0000 7ff191800640 1 --2- 172.21.15.118:0/3005588200 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x560cb2a7ed40 0x560cb2a89100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:19:21.696 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:19:21.693+0000 7ff191800640 1 -- 172.21.15.118:0/3005588200 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x560cb2a79bd0 msgr2=0x560cb2a88160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:19:21.696 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:19:21.693+0000 7ff191800640 1 --2- 172.21.15.118:0/3005588200 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x560cb2a79bd0 0x560cb2a88160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:19:21.696 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:19:21.693+0000 7ff191800640 1 -- 172.21.15.118:0/3005588200 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x560cb2a896f0 con 0x560cb2a791e0 2024-03-24T00:19:21.697 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:19:21.693+0000 7ff190e00640 1 --2- 172.21.15.118:0/3005588200 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x560cb2a79bd0 0x560cb2a88160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T00:19:21.697 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:19:21.693+0000 7ff191800640 1 --2- 172.21.15.118:0/3005588200 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x560cb2a791e0 0x560cb2a87c20 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7ff18009b560 tx=0x7ff1800d7ae0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4ddf1b722d8a9b6d server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:19:21.697 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:19:21.693+0000 7ff173400640 10 client.0 ms_handle_connect on v2:172.21.15.118:3300/0 2024-03-24T00:19:21.697 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:19:21.693+0000 7ff173400640 1 -- 172.21.15.118:0/3005588200 <== mon.0 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff1800b3020 con 0x560cb2a791e0 2024-03-24T00:19:21.697 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:19:21.693+0000 7ff173400640 1 -- 172.21.15.118:0/3005588200 <== mon.0 v2:172.21.15.118:3300/0 2 ==== osd_map(90..90 src has 1..90) v4 ==== 8277+0+0 (secure 0 0 0) 0x7ff180099b60 con 0x560cb2a791e0 2024-03-24T00:19:21.697 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:19:21.693+0000 7ff197036740 1 -- 172.21.15.118:0/3005588200 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7ff1840347f0 con 0x560cb2a791e0 2024-03-24T00:19:21.697 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:19:21.694+0000 7ff173400640 1 -- 172.21.15.118:0/3005588200 <== mon.0 v2:172.21.15.118:3300/0 3 ==== mon_get_version_reply(handle=1 version=28) v2 ==== 24+0+0 (secure 0 0 0) 0x7ff180097b00 con 0x560cb2a791e0 2024-03-24T00:19:21.697 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:19:21.694+0000 7ff197036740 10 client.16380 fetch_fsmap learned FSMap version 28 2024-03-24T00:19:21.697 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:19:21.694+0000 7ff197036740 1 -- 172.21.15.118:0/3005588200 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({config=0+,fsmap=28,monmap=4+}) v3 -- 0x560cb2a8aa00 con 0x560cb2a791e0 2024-03-24T00:19:21.697 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:19:21.694+0000 7ff173400640 1 -- 172.21.15.118:0/3005588200 <== mon.0 v2:172.21.15.118:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7ff1840347f0 con 0x560cb2a791e0 2024-03-24T00:19:21.697 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:19:21.694+0000 7ff173400640 1 -- 172.21.15.118:0/3005588200 <== mon.0 v2:172.21.15.118:3300/0 5 ==== fsmap(e 28) v1 ==== 7934+0+0 (secure 0 0 0) 0x7ff1800860b0 con 0x560cb2a791e0 2024-03-24T00:19:21.697 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:19:21.694+0000 7ff197036740 10 client.16380 fetch_fsmap finished waiting for FSMap version 28 2024-03-24T00:19:21.697 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:19:21.694+0000 7ff197036740 10 client.16380 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-24T00:19:21.697 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:19:21.694+0000 7ff197036740 20 client.16380 populate_metadata read hostname 'smithi118' 2024-03-24T00:19:21.698 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:19:21.694+0000 7ff197036740 1 --2- 172.21.15.118:0/3005588200 >> [v2:172.21.15.189:6838/4074484168,v1:172.21.15.189:6839/4074484168] conn(0x560cb2aeabb0 0x560cb2aecfd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:19:21.698 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:19:21.694+0000 7ff197036740 4 client.16380 mds_command: new command op to 24569 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-03-24T00:19:21.698 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:19:21.694+0000 7ff197036740 1 -- 172.21.15.118:0/3005588200 --> [v2:172.21.15.189:6838/4074484168,v1:172.21.15.189:6839/4074484168] -- command(tid 0: {"prefix": "flush journal"}) v1 -- 0x560cb29e60c0 con 0x560cb2aeabb0 2024-03-24T00:19:21.698 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:19:21.694+0000 7ff190e00640 1 --2- 172.21.15.118:0/3005588200 >> [v2:172.21.15.189:6838/4074484168,v1:172.21.15.189:6839/4074484168] conn(0x560cb2aeabb0 0x560cb2aecfd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:19:21.698 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:19:21.695+0000 7ff190e00640 1 --2- 172.21.15.118:0/3005588200 >> [v2:172.21.15.189:6838/4074484168,v1:172.21.15.189:6839/4074484168] conn(0x560cb2aeabb0 0x560cb2aecfd0 crc :-1 s=READY pgs=116 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=c8a732df8f60716a server_cookie=852205de64d8a046 in_seq=0 out_seq=0 2024-03-24T00:19:21.698 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:19:21.695+0000 7ff173400640 10 client.16380 ms_handle_connect on v2:172.21.15.189:6838/4074484168 2024-03-24T00:19:22.046 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:19:22.046+0000 7ff173400640 1 -- 172.21.15.118:0/3005588200 <== mds.0 v2:172.21.15.189:6838/4074484168 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+44 (crc 0 0 0) 0x560cb29e60c0 con 0x560cb2aeabb0 2024-03-24T00:19:22.047 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:19:22.046+0000 7ff173400640 10 client.16380 handle_command_reply: tid=0 2024-03-24T00:19:22.047 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:19:22.046+0000 7ff197036740 1 client.16380 shutdown 2024-03-24T00:19:22.047 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:19:22.046+0000 7ff197036740 1 -- 172.21.15.118:0/3005588200 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x560cb2a791e0 msgr2=0x560cb2a87c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:19:22.047 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:19:22.046+0000 7ff197036740 1 --2- 172.21.15.118:0/3005588200 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x560cb2a791e0 0x560cb2a87c20 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7ff18009b560 tx=0x7ff1800d7ae0 comp rx=0 tx=0).stop 2024-03-24T00:19:22.047 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:19:22.046+0000 7ff197036740 1 -- 172.21.15.118:0/3005588200 shutdown_connections 2024-03-24T00:19:22.047 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:19:22.046+0000 7ff197036740 1 --2- 172.21.15.118:0/3005588200 >> [v2:172.21.15.189:6838/4074484168,v1:172.21.15.189:6839/4074484168] conn(0x560cb2aeabb0 0x560cb2aecfd0 crc :-1 s=READY pgs=116 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:19:22.047 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:19:22.046+0000 7ff197036740 1 --2- 172.21.15.118:0/3005588200 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x560cb2a7ed40 0x560cb2a89100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:19:22.047 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:19:22.046+0000 7ff197036740 1 --2- 172.21.15.118:0/3005588200 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x560cb2a79bd0 0x560cb2a88160 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:19:22.047 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:19:22.046+0000 7ff197036740 1 --2- 172.21.15.118:0/3005588200 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x560cb2a791e0 0x560cb2a87c20 unknown :-1 s=CLOSED pgs=207 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:19:22.047 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:19:22.046+0000 7ff197036740 1 -- 172.21.15.118:0/3005588200 >> 172.21.15.118:0/3005588200 conn(0x560cb2a74750 msgr2=0x560cb2a7af30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:19:22.047 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:19:22.046+0000 7ff173400640 0 client.16380 ms_handle_reset on v2:172.21.15.189:6838/4074484168 2024-03-24T00:19:22.047 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:19:22.046+0000 7ff197036740 1 -- 172.21.15.118:0/3005588200 shutdown_connections 2024-03-24T00:19:22.047 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:19:22.047+0000 7ff197036740 1 -- 172.21.15.118:0/3005588200 wait complete. 2024-03-24T00:19:22.048 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:19:22.047+0000 7ff197036740 20 client.16380 trim_cache size 0 max 16384 2024-03-24T00:19:22.048 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stdout:{ 2024-03-24T00:19:22.048 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stdout: "message": "", 2024-03-24T00:19:22.048 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stdout: "return_code": 0 2024-03-24T00:19:22.048 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stdout:} 2024-03-24T00:19:22.049 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:19:22.048+0000 7ff194200640 1 -- 172.21.15.118:0/320181713 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7ff1740a28a0 msgr2=0x7ff1740a4d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:19:22.049 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:19:22.048+0000 7ff194200640 1 --2- 172.21.15.118:0/320181713 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7ff1740a28a0 0x7ff1740a4d60 secure :-1 s=READY pgs=340 cs=0 l=1 rev1=1 crypto rx=0x7ff18402c640 tx=0x7ff184032590 comp rx=0 tx=0).stop 2024-03-24T00:19:22.049 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:19:22.048+0000 7ff194200640 1 -- 172.21.15.118:0/320181713 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff18c1038f0 msgr2=0x7ff18c199ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:19:22.049 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:19:22.048+0000 7ff194200640 1 --2- 172.21.15.118:0/320181713 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff18c1038f0 0x7ff18c199ac0 secure :-1 s=READY pgs=389 cs=0 l=1 rev1=1 crypto rx=0x7ff18000e9c0 tx=0x7ff18000ee80 comp rx=0 tx=0).stop 2024-03-24T00:19:22.049 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:19:22.048+0000 7ff194200640 1 -- 172.21.15.118:0/320181713 shutdown_connections 2024-03-24T00:19:22.049 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:19:22.048+0000 7ff194200640 1 --2- 172.21.15.118:0/320181713 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7ff1740a28a0 0x7ff1740a4d60 unknown :-1 s=CLOSED pgs=340 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:19:22.049 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:19:22.048+0000 7ff194200640 1 --2- 172.21.15.118:0/320181713 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7ff18c1054a0 0x7ff18c1a1070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:19:22.049 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:19:22.048+0000 7ff194200640 1 --2- 172.21.15.118:0/320181713 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7ff18c104ae0 0x7ff18c19a000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:19:22.049 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:19:22.048+0000 7ff194200640 1 --2- 172.21.15.118:0/320181713 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff18c1038f0 0x7ff18c199ac0 unknown :-1 s=CLOSED pgs=389 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:19:22.049 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:19:22.048+0000 7ff194200640 1 -- 172.21.15.118:0/320181713 >> 172.21.15.118:0/320181713 conn(0x7ff18c0fee60 msgr2=0x7ff18c107d10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:19:22.050 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:19:22.048+0000 7ff194200640 1 -- 172.21.15.118:0/320181713 shutdown_connections 2024-03-24T00:19:22.050 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:19:22.048+0000 7ff194200640 1 -- 172.21.15.118:0/320181713 wait complete. 2024-03-24T00:19:23.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:19:22 smithi189 ceph-mon[36187]: pgmap v508: 97 pgs: 97 active+clean; 77 MiB data, 577 MiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 2.0 MiB/s wr, 660 op/s 2024-03-24T00:19:23.207 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:19:22 smithi118 ceph-mon[31535]: pgmap v508: 97 pgs: 97 active+clean; 77 MiB data, 577 MiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 2.0 MiB/s wr, 660 op/s 2024-03-24T00:19:23.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:19:22 smithi193 ceph-mon[36878]: pgmap v508: 97 pgs: 97 active+clean; 77 MiB data, 577 MiB used, 1.0 TiB / 1.0 TiB avail; 1.0 MiB/s rd, 2.0 MiB/s wr, 660 op/s 2024-03-24T00:19:24.195 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:19:23 smithi118 ceph-mon[31535]: pgmap v509: 97 pgs: 97 active+clean; 78 MiB data, 578 MiB used, 1.0 TiB / 1.0 TiB avail; 1018 KiB/s rd, 1.6 MiB/s wr, 695 op/s 2024-03-24T00:19:24.203 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:19:23 smithi189 ceph-mon[36187]: pgmap v509: 97 pgs: 97 active+clean; 78 MiB data, 578 MiB used, 1.0 TiB / 1.0 TiB avail; 1018 KiB/s rd, 1.6 MiB/s wr, 695 op/s 2024-03-24T00:19:24.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:19:23 smithi193 ceph-mon[36878]: pgmap v509: 97 pgs: 97 active+clean; 78 MiB data, 578 MiB used, 1.0 TiB / 1.0 TiB avail; 1018 KiB/s rd, 1.6 MiB/s wr, 695 op/s 2024-03-24T00:19:26.843 INFO:tasks.workunit.client.0.smithi118.stdout: 26 140546 59 105150 191 66768 284 2024-03-24T00:19:27.195 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:19:26 smithi118 ceph-mon[31535]: pgmap v510: 97 pgs: 97 active+clean; 74 MiB data, 576 MiB used, 1.0 TiB / 1.0 TiB avail; 964 KiB/s rd, 828 KiB/s wr, 541 op/s 2024-03-24T00:19:27.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:19:26 smithi189 ceph-mon[36187]: pgmap v510: 97 pgs: 97 active+clean; 74 MiB data, 576 MiB used, 1.0 TiB / 1.0 TiB avail; 964 KiB/s rd, 828 KiB/s wr, 541 op/s 2024-03-24T00:19:27.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:19:26 smithi193 ceph-mon[36878]: pgmap v510: 97 pgs: 97 active+clean; 74 MiB data, 576 MiB used, 1.0 TiB / 1.0 TiB avail; 964 KiB/s rd, 828 KiB/s wr, 541 op/s 2024-03-24T00:19:29.198 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:19:28 smithi118 ceph-mon[31535]: pgmap v511: 97 pgs: 97 active+clean; 75 MiB data, 559 MiB used, 1.0 TiB / 1.0 TiB avail; 868 KiB/s rd, 601 KiB/s wr, 448 op/s 2024-03-24T00:19:29.203 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:19:28 smithi189 ceph-mon[36187]: pgmap v511: 97 pgs: 97 active+clean; 75 MiB data, 559 MiB used, 1.0 TiB / 1.0 TiB avail; 868 KiB/s rd, 601 KiB/s wr, 448 op/s 2024-03-24T00:19:29.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:19:28 smithi193 ceph-mon[36878]: pgmap v511: 97 pgs: 97 active+clean; 75 MiB data, 559 MiB used, 1.0 TiB / 1.0 TiB avail; 868 KiB/s rd, 601 KiB/s wr, 448 op/s 2024-03-24T00:19:31.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:19:30 smithi118 ceph-mon[31535]: pgmap v512: 97 pgs: 97 active+clean; 75 MiB data, 557 MiB used, 1.0 TiB / 1.0 TiB avail; 549 KiB/s rd, 410 KiB/s wr, 300 op/s 2024-03-24T00:19:31.203 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:19:30 smithi189 ceph-mon[36187]: pgmap v512: 97 pgs: 97 active+clean; 75 MiB data, 557 MiB used, 1.0 TiB / 1.0 TiB avail; 549 KiB/s rd, 410 KiB/s wr, 300 op/s 2024-03-24T00:19:31.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:19:30 smithi193 ceph-mon[36878]: pgmap v512: 97 pgs: 97 active+clean; 75 MiB data, 557 MiB used, 1.0 TiB / 1.0 TiB avail; 549 KiB/s rd, 410 KiB/s wr, 300 op/s 2024-03-24T00:19:33.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:19:32 smithi118 ceph-mon[31535]: pgmap v513: 97 pgs: 97 active+clean; 82 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 681 KiB/s rd, 1.0 MiB/s wr, 276 op/s 2024-03-24T00:19:33.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:19:32 smithi189 ceph-mon[36187]: pgmap v513: 97 pgs: 97 active+clean; 82 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 681 KiB/s rd, 1.0 MiB/s wr, 276 op/s 2024-03-24T00:19:33.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:19:32 smithi193 ceph-mon[36878]: pgmap v513: 97 pgs: 97 active+clean; 82 MiB data, 568 MiB used, 1.0 TiB / 1.0 TiB avail; 681 KiB/s rd, 1.0 MiB/s wr, 276 op/s 2024-03-24T00:19:34.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:19:33 smithi118 ceph-mon[31535]: pgmap v514: 97 pgs: 97 active+clean; 92 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 741 KiB/s rd, 2.0 MiB/s wr, 331 op/s 2024-03-24T00:19:34.203 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:19:33 smithi189 ceph-mon[36187]: pgmap v514: 97 pgs: 97 active+clean; 92 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 741 KiB/s rd, 2.0 MiB/s wr, 331 op/s 2024-03-24T00:19:34.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:19:33 smithi193 ceph-mon[36878]: pgmap v514: 97 pgs: 97 active+clean; 92 MiB data, 596 MiB used, 1.0 TiB / 1.0 TiB avail; 741 KiB/s rd, 2.0 MiB/s wr, 331 op/s 2024-03-24T00:19:35.285 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:19:35.284+0000 7f0ae0a00640 1 -- 172.21.15.118:0/3179989702 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f0adc109390 msgr2=0x7f0adc10e000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:19:35.285 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:19:35.284+0000 7f0ae0a00640 1 --2- 172.21.15.118:0/3179989702 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f0adc109390 0x7f0adc10e000 secure :-1 s=READY pgs=391 cs=0 l=1 rev1=1 crypto rx=0x7f0ad4009be0 tx=0x7f0ad402bf30 comp rx=0 tx=0).stop 2024-03-24T00:19:35.287 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:19:35.287+0000 7f0ae0a00640 1 -- 172.21.15.118:0/3179989702 shutdown_connections 2024-03-24T00:19:35.287 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:19:35.287+0000 7f0ae0a00640 1 --2- 172.21.15.118:0/3179989702 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f0adc109390 0x7f0adc10e000 unknown :-1 s=CLOSED pgs=391 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:19:35.287 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:19:35.287+0000 7f0ae0a00640 1 --2- 172.21.15.118:0/3179989702 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f0adc0732b0 0x7f0adc108cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:19:35.287 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:19:35.287+0000 7f0ae0a00640 1 --2- 172.21.15.118:0/3179989702 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0adc072950 0x7f0adc072d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:19:35.287 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:19:35.287+0000 7f0ae0a00640 1 -- 172.21.15.118:0/3179989702 >> 172.21.15.118:0/3179989702 conn(0x7f0adc06e510 msgr2=0x7f0adc070950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:19:35.288 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:19:35.288+0000 7f0ae0a00640 1 -- 172.21.15.118:0/3179989702 shutdown_connections 2024-03-24T00:19:35.289 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:19:35.288+0000 7f0ae0a00640 1 -- 172.21.15.118:0/3179989702 wait complete. 2024-03-24T00:19:35.289 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:19:35.289+0000 7f0ae0a00640 1 Processor -- start 2024-03-24T00:19:35.293 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:19:35.292+0000 7f0ae0a00640 1 -- start start 2024-03-24T00:19:35.293 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:19:35.293+0000 7f0ae0a00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f0adc072950 0x7f0adc1a6530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:19:35.293 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:19:35.293+0000 7f0ae0a00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0adc0732b0 0x7f0adc1a6a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:19:35.294 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:19:35.293+0000 7f0ae0a00640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f0adc109390 0x7f0adc1adae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:19:35.294 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:19:35.293+0000 7f0adaa00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0adc0732b0 0x7f0adc1a6a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:19:35.294 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:19:35.293+0000 7f0adaa00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0adc0732b0 0x7f0adc1a6a70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.118:34258/0 (socket says 172.21.15.118:34258) 2024-03-24T00:19:35.294 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:19:35.293+0000 7f0adbe00640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f0adc109390 0x7f0adc1adae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:19:35.294 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:19:35.293+0000 7f0adbe00640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f0adc109390 0x7f0adc1adae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.118:3300/0 says I am v2:172.21.15.118:38470/0 (socket says 172.21.15.118:38470) 2024-03-24T00:19:35.294 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:19:35.293+0000 7f0ae0a00640 1 -- 172.21.15.118:0/2340288462 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0adc11bfd0 con 0x7f0adc109390 2024-03-24T00:19:35.294 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:19:35.293+0000 7f0ae0a00640 1 -- 172.21.15.118:0/2340288462 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0adc11be50 con 0x7f0adc0732b0 2024-03-24T00:19:35.294 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:19:35.293+0000 7f0ae0a00640 1 -- 172.21.15.118:0/2340288462 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0adc11c150 con 0x7f0adc072950 2024-03-24T00:19:35.294 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:19:35.294+0000 7f0adb400640 1 --2- 172.21.15.118:0/2340288462 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f0adc072950 0x7f0adc1a6530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:19:35.294 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:19:35.293+0000 7f0adaa00640 1 -- 172.21.15.118:0/2340288462 learned_addr learned my addr 172.21.15.118:0/2340288462 (peer_addr_for_me v2:172.21.15.118:0/0) 2024-03-24T00:19:35.295 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:19:35.295+0000 7f0adaa00640 1 -- 172.21.15.118:0/2340288462 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f0adc072950 msgr2=0x7f0adc1a6530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:19:35.295 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:19:35.295+0000 7f0adaa00640 1 --2- 172.21.15.118:0/2340288462 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f0adc072950 0x7f0adc1a6530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:19:35.295 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:19:35.295+0000 7f0adaa00640 1 -- 172.21.15.118:0/2340288462 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f0adc109390 msgr2=0x7f0adc1adae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:19:35.296 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:19:35.295+0000 7f0adaa00640 1 --2- 172.21.15.118:0/2340288462 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f0adc109390 0x7f0adc1adae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:19:35.296 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:19:35.295+0000 7f0adaa00640 1 -- 172.21.15.118:0/2340288462 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0adc10ff00 con 0x7f0adc0732b0 2024-03-24T00:19:35.296 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:19:35.295+0000 7f0adbe00640 1 --2- 172.21.15.118:0/2340288462 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f0adc109390 0x7f0adc1adae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T00:19:35.296 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:19:35.297+0000 7f0adaa00640 1 --2- 172.21.15.118:0/2340288462 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0adc0732b0 0x7f0adc1a6a70 secure :-1 s=READY pgs=391 cs=0 l=1 rev1=1 crypto rx=0x7f0ac800e9a0 tx=0x7f0ac800ee60 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:19:35.297 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:19:35.297+0000 7f0ad3e00640 1 -- 172.21.15.118:0/2340288462 <== mon.2 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0ac800cd80 con 0x7f0adc0732b0 2024-03-24T00:19:35.308 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:19:35.297+0000 7f0ae0a00640 1 -- 172.21.15.118:0/2340288462 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0adc1101f0 con 0x7f0adc0732b0 2024-03-24T00:19:35.308 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:19:35.297+0000 7f0ad3e00640 1 -- 172.21.15.118:0/2340288462 <== mon.2 v2:172.21.15.189:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f0ac80035f0 con 0x7f0adc0732b0 2024-03-24T00:19:35.308 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:19:35.297+0000 7f0ad3e00640 1 -- 172.21.15.118:0/2340288462 <== mon.2 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0ac8010670 con 0x7f0adc0732b0 2024-03-24T00:19:35.308 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:19:35.297+0000 7f0ae0a00640 1 -- 172.21.15.118:0/2340288462 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0adc110790 con 0x7f0adc0732b0 2024-03-24T00:19:35.313 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:19:35.311+0000 7f0ad3e00640 1 -- 172.21.15.118:0/2340288462 <== mon.2 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 20) v1 ==== 137291+0+0 (secure 0 0 0) 0x7f0ac80037a0 con 0x7f0adc0732b0 2024-03-24T00:19:35.313 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:19:35.313+0000 7f0ad3e00640 1 --2- 172.21.15.118:0/2340288462 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7f0ac00a28f0 0x7f0ac00a4db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:19:35.313 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:19:35.313+0000 7f0ad3e00640 1 -- 172.21.15.118:0/2340288462 <== mon.2 v2:172.21.15.189:3300/0 5 ==== osd_map(90..90 src has 1..90) v4 ==== 8277+0+0 (secure 0 0 0) 0x7f0ac80ce730 con 0x7f0adc0732b0 2024-03-24T00:19:35.314 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:19:35.314+0000 7f0adb400640 1 --2- 172.21.15.118:0/2340288462 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7f0ac00a28f0 0x7f0ac00a4db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:19:35.314 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:19:35.314+0000 7f0ae4d7b740 1 Processor -- start 2024-03-24T00:19:35.314 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:19:35.314+0000 7f0ae4d7b740 1 -- start start 2024-03-24T00:19:35.319 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:19:35.319+0000 7f0ae4d7b740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5645df30c3b0 0x5645df30c7d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:19:35.319 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:19:35.319+0000 7f0ae4d7b740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5645df30cda0 0x5645df3118a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:19:35.319 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:19:35.319+0000 7f0ae4d7b740 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5645df311f10 0x5645df3143a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:19:35.319 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:19:35.319+0000 7f0ae4d7b740 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x5645df2cf3b0 con 0x5645df311f10 2024-03-24T00:19:35.319 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:19:35.319+0000 7f0ae4d7b740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x5645df271830 con 0x5645df30cda0 2024-03-24T00:19:35.319 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:19:35.319+0000 7f0ae4d7b740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x5645df21a5d0 con 0x5645df30c3b0 2024-03-24T00:19:35.319 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:19:35.317+0000 7f0adb400640 1 --2- 172.21.15.118:0/2340288462 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7f0ac00a28f0 0x7f0ac00a4db0 secure :-1 s=READY pgs=341 cs=0 l=1 rev1=1 crypto rx=0x7f0acc005fd0 tx=0x7f0acc00a560 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:19:35.320 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:19:35.320+0000 7f0adaa00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5645df30cda0 0x5645df3118a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:19:35.320 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:19:35.320+0000 7f0adaa00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5645df30cda0 0x5645df3118a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.118:34262/0 (socket says 172.21.15.118:34262) 2024-03-24T00:19:35.320 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:19:35.320+0000 7f0adaa00640 1 -- 172.21.15.118:0/815837149 learned_addr learned my addr 172.21.15.118:0/815837149 (peer_addr_for_me v2:172.21.15.118:0/0) 2024-03-24T00:19:35.320 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:19:35.320+0000 7f0adb400640 1 --2- 172.21.15.118:0/815837149 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5645df30c3b0 0x5645df30c7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:19:35.320 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:19:35.320+0000 7f0adaa00640 1 -- 172.21.15.118:0/815837149 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5645df30c3b0 msgr2=0x5645df30c7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:19:35.320 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:19:35.320+0000 7f0adaa00640 1 --2- 172.21.15.118:0/815837149 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5645df30c3b0 0x5645df30c7d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:19:35.320 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:19:35.320+0000 7f0adaa00640 1 -- 172.21.15.118:0/815837149 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5645df311f10 msgr2=0x5645df3143a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:19:35.321 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:19:35.321+0000 7f0adbe00640 1 --2- 172.21.15.118:0/815837149 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5645df311f10 0x5645df3143a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:19:35.321 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:19:35.321+0000 7f0adaa00640 1 --2- 172.21.15.118:0/815837149 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5645df311f10 0x5645df3143a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:19:35.321 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:19:35.321+0000 7f0adaa00640 1 -- 172.21.15.118:0/815837149 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5645df2322c0 con 0x5645df30cda0 2024-03-24T00:19:35.321 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:19:35.321+0000 7f0adbe00640 1 --2- 172.21.15.118:0/815837149 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5645df311f10 0x5645df3143a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-24T00:19:35.321 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:19:35.322+0000 7f0adaa00640 1 --2- 172.21.15.118:0/815837149 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5645df30cda0 0x5645df3118a0 secure :-1 s=READY pgs=392 cs=0 l=1 rev1=1 crypto rx=0x7f0ac80974a0 tx=0x7f0ac8097410 comp rx=0 tx=0).ready entity=mon.2 client_cookie=14e6d8182b52eed server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:19:35.322 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:19:35.322+0000 7f0ac7e00640 1 -- 172.21.15.118:0/815837149 <== mon.2 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0ac809b040 con 0x5645df30cda0 2024-03-24T00:19:35.322 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:19:35.322+0000 7f0ac7e00640 1 -- 172.21.15.118:0/815837149 <== mon.2 v2:172.21.15.189:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f0ac809f190 con 0x5645df30cda0 2024-03-24T00:19:35.323 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:19:35.323+0000 7f0ae4d7b740 1 -- 172.21.15.118:0/815837149 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5645df30cda0 msgr2=0x5645df3118a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:19:35.323 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:19:35.323+0000 7f0ae4d7b740 1 --2- 172.21.15.118:0/815837149 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5645df30cda0 0x5645df3118a0 secure :-1 s=READY pgs=392 cs=0 l=1 rev1=1 crypto rx=0x7f0ac80974a0 tx=0x7f0ac8097410 comp rx=0 tx=0).stop 2024-03-24T00:19:35.323 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:19:35.324+0000 7f0ae4d7b740 1 -- 172.21.15.118:0/815837149 shutdown_connections 2024-03-24T00:19:35.323 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:19:35.324+0000 7f0ae4d7b740 1 --2- 172.21.15.118:0/815837149 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5645df311f10 0x5645df3143a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:19:35.324 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:19:35.324+0000 7f0ae4d7b740 1 --2- 172.21.15.118:0/815837149 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5645df30cda0 0x5645df3118a0 unknown :-1 s=CLOSED pgs=392 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:19:35.324 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:19:35.324+0000 7f0ae4d7b740 1 --2- 172.21.15.118:0/815837149 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5645df30c3b0 0x5645df30c7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:19:35.324 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:19:35.324+0000 7f0ae4d7b740 1 -- 172.21.15.118:0/815837149 >> 172.21.15.118:0/815837149 conn(0x5645df307920 msgr2=0x5645df307d20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:19:35.324 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:19:35.324+0000 7f0ae4d7b740 1 -- 172.21.15.118:0/815837149 shutdown_connections 2024-03-24T00:19:35.325 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:19:35.324+0000 7f0ae4d7b740 1 -- 172.21.15.118:0/815837149 wait complete. 2024-03-24T00:19:35.325 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:19:35.324+0000 7f0ae4d7b740 1 -- start start 2024-03-24T00:19:35.325 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:19:35.325+0000 7f0ae4d7b740 1 Processor -- start 2024-03-24T00:19:35.325 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:19:35.325+0000 7f0ae4d7b740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5645df30c3b0 0x5645df316500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:19:35.326 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:19:35.325+0000 7f0ae4d7b740 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5645df30cda0 0x5645df316a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:19:35.326 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:19:35.326+0000 7f0adb400640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5645df30c3b0 0x5645df316500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:19:35.326 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:19:35.326+0000 7f0adb400640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5645df30c3b0 0x5645df316500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.118:34264/0 (socket says 172.21.15.118:34264) 2024-03-24T00:19:35.327 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:19:35.328+0000 7f0adaa00640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5645df30cda0 0x5645df316a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:19:35.328 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:19:35.328+0000 7f0adaa00640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5645df30cda0 0x5645df316a40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.118:3300/0 says I am v2:172.21.15.118:38490/0 (socket says 172.21.15.118:38490) 2024-03-24T00:19:35.329 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:19:35.328+0000 7f0adb400640 1 -- 172.21.15.118:0/682065906 learned_addr learned my addr 172.21.15.118:0/682065906 (peer_addr_for_me v2:172.21.15.118:0/0) 2024-03-24T00:19:35.329 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:19:35.328+0000 7f0ae4d7b740 1 --2- 172.21.15.118:0/682065906 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5645df311f10 0x5645df31da80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:19:35.329 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:19:35.328+0000 7f0ae4d7b740 1 -- 172.21.15.118:0/682065906 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x5645df22da20 con 0x5645df30cda0 2024-03-24T00:19:35.329 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:19:35.328+0000 7f0ae4d7b740 1 -- 172.21.15.118:0/682065906 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x5645df26d8d0 con 0x5645df30c3b0 2024-03-24T00:19:35.329 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:19:35.328+0000 7f0ae4d7b740 1 -- 172.21.15.118:0/682065906 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x5645df314ea0 con 0x5645df311f10 2024-03-24T00:19:35.329 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:19:35.328+0000 7f0ae4d7b740 10 client.0 _ll_register_callbacks cb 0x5645df26a1c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T00:19:35.329 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:19:35.329+0000 7f0adbe00640 1 --2- 172.21.15.118:0/682065906 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5645df311f10 0x5645df31da80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:19:35.329 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:19:35.329+0000 7f0adbe00640 1 -- 172.21.15.118:0/682065906 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5645df30c3b0 msgr2=0x5645df316500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:19:35.329 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:19:35.329+0000 7f0adbe00640 1 --2- 172.21.15.118:0/682065906 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5645df30c3b0 0x5645df316500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:19:35.329 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:19:35.329+0000 7f0adbe00640 1 -- 172.21.15.118:0/682065906 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5645df30cda0 msgr2=0x5645df316a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:19:35.330 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:19:35.330+0000 7f0adbe00640 1 --2- 172.21.15.118:0/682065906 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5645df30cda0 0x5645df316a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:19:35.330 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:19:35.330+0000 7f0adbe00640 1 -- 172.21.15.118:0/682065906 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5645def86210 con 0x5645df311f10 2024-03-24T00:19:35.330 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:19:35.330+0000 7f0adaa00640 1 --2- 172.21.15.118:0/682065906 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5645df30cda0 0x5645df316a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T00:19:35.330 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:19:35.330+0000 7f0adbe00640 1 --2- 172.21.15.118:0/682065906 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5645df311f10 0x5645df31da80 secure :-1 s=READY pgs=392 cs=0 l=1 rev1=1 crypto rx=0x7f0ad4006010 tx=0x7f0ad402c8d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=ecefcca2b30c13a5 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:19:35.330 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:19:35.330+0000 7f0ac4c00640 10 client.0 ms_handle_connect on v2:172.21.15.193:3300/0 2024-03-24T00:19:35.340 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:19:35.331+0000 7f0ac4c00640 1 -- 172.21.15.118:0/682065906 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0ad4055070 con 0x5645df311f10 2024-03-24T00:19:35.341 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:19:35.331+0000 7f0ac4c00640 1 -- 172.21.15.118:0/682065906 <== mon.1 v2:172.21.15.193:3300/0 2 ==== osd_map(90..90 src has 1..90) v4 ==== 8277+0+0 (secure 0 0 0) 0x7f0ad40337a0 con 0x5645df311f10 2024-03-24T00:19:35.341 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:19:35.331+0000 7f0ae4d7b740 1 -- 172.21.15.118:0/682065906 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f0ac809f190 con 0x5645df311f10 2024-03-24T00:19:35.341 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:19:35.331+0000 7f0ac4c00640 1 -- 172.21.15.118:0/682065906 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) v2 ==== 24+0+0 (secure 0 0 0) 0x7f0ad4061050 con 0x5645df311f10 2024-03-24T00:19:35.341 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:19:35.332+0000 7f0ae4d7b740 10 client.26560 fetch_fsmap learned FSMap version 29 2024-03-24T00:19:35.341 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:19:35.332+0000 7f0ae4d7b740 1 -- 172.21.15.118:0/682065906 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) v3 -- 0x5645df052620 con 0x5645df311f10 2024-03-24T00:19:35.341 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:19:35.332+0000 7f0ac4c00640 1 -- 172.21.15.118:0/682065906 <== mon.1 v2:172.21.15.193:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f0ac809f190 con 0x5645df311f10 2024-03-24T00:19:35.341 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:19:35.333+0000 7f0ac4c00640 1 -- 172.21.15.118:0/682065906 <== mon.1 v2:172.21.15.193:3300/0 5 ==== fsmap(e 29) v1 ==== 7938+0+0 (secure 0 0 0) 0x7f0ad40339a0 con 0x5645df311f10 2024-03-24T00:19:35.341 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:19:35.334+0000 7f0ae4d7b740 10 client.26560 fetch_fsmap finished waiting for FSMap version 29 2024-03-24T00:19:35.341 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:19:35.334+0000 7f0ae4d7b740 10 client.26560 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-24T00:19:35.341 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:19:35.334+0000 7f0ae4d7b740 20 client.26560 populate_metadata read hostname 'smithi118' 2024-03-24T00:19:35.341 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:19:35.334+0000 7f0ae4d7b740 1 --2- 172.21.15.118:0/682065906 >> [v2:172.21.15.189:6838/4074484168,v1:172.21.15.189:6839/4074484168] conn(0x5645df322700 0x5645df324b10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:19:35.341 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:19:35.334+0000 7f0ae4d7b740 4 client.26560 mds_command: new command op to 24569 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-24T00:19:35.341 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:19:35.334+0000 7f0ae4d7b740 1 -- 172.21.15.118:0/682065906 --> [v2:172.21.15.189:6838/4074484168,v1:172.21.15.189:6839/4074484168] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x5645df27ed70 con 0x5645df322700 2024-03-24T00:19:35.342 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:19:35.334+0000 7f0adb400640 1 --2- 172.21.15.118:0/682065906 >> [v2:172.21.15.189:6838/4074484168,v1:172.21.15.189:6839/4074484168] conn(0x5645df322700 0x5645df324b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:19:35.342 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:19:35.335+0000 7f0ac4c00640 10 client.26560 ms_handle_connect on v2:172.21.15.189:6838/4074484168 2024-03-24T00:19:35.342 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:19:35.335+0000 7f0adb400640 1 --2- 172.21.15.118:0/682065906 >> [v2:172.21.15.189:6838/4074484168,v1:172.21.15.189:6839/4074484168] conn(0x5645df322700 0x5645df324b10 crc :-1 s=READY pgs=117 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=1b747f3c8cc91acd server_cookie=9e958cf65e5f50f7 in_seq=0 out_seq=0 2024-03-24T00:19:35.342 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:19:35.339+0000 7f0ac4c00640 1 -- 172.21.15.118:0/682065906 <== mds.0 v2:172.21.15.189:6838/4074484168 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x5645df27ed70 con 0x5645df322700 2024-03-24T00:19:35.342 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:19:35.339+0000 7f0ac4c00640 10 client.26560 handle_command_reply: tid=0 2024-03-24T00:19:35.342 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:19:35.339+0000 7f0ae4d7b740 1 client.26560 shutdown 2024-03-24T00:19:35.342 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:19:35.339+0000 7f0ae4d7b740 1 -- 172.21.15.118:0/682065906 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5645df311f10 msgr2=0x5645df31da80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:19:35.342 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:19:35.339+0000 7f0ae4d7b740 1 --2- 172.21.15.118:0/682065906 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5645df311f10 0x5645df31da80 secure :-1 s=READY pgs=392 cs=0 l=1 rev1=1 crypto rx=0x7f0ad4006010 tx=0x7f0ad402c8d0 comp rx=0 tx=0).stop 2024-03-24T00:19:35.342 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:19:35.340+0000 7f0ae4d7b740 1 -- 172.21.15.118:0/682065906 shutdown_connections 2024-03-24T00:19:35.342 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:19:35.340+0000 7f0ae4d7b740 1 --2- 172.21.15.118:0/682065906 >> [v2:172.21.15.189:6838/4074484168,v1:172.21.15.189:6839/4074484168] conn(0x5645df322700 0x5645df324b10 crc :-1 s=READY pgs=117 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:19:35.343 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:19:35.340+0000 7f0ae4d7b740 1 --2- 172.21.15.118:0/682065906 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5645df311f10 0x5645df31da80 unknown :-1 s=CLOSED pgs=392 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:19:35.343 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:19:35.340+0000 7f0ac4c00640 0 client.26560 ms_handle_reset on v2:172.21.15.189:6838/4074484168 2024-03-24T00:19:35.343 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:19:35.340+0000 7f0ae4d7b740 1 --2- 172.21.15.118:0/682065906 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5645df30cda0 0x5645df316a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:19:35.343 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:19:35.340+0000 7f0ae4d7b740 1 --2- 172.21.15.118:0/682065906 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5645df30c3b0 0x5645df316500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:19:35.343 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:19:35.340+0000 7f0ae4d7b740 1 -- 172.21.15.118:0/682065906 >> 172.21.15.118:0/682065906 conn(0x5645df307920 msgr2=0x5645df306b40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:19:35.343 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:19:35.340+0000 7f0ae4d7b740 1 -- 172.21.15.118:0/682065906 shutdown_connections 2024-03-24T00:19:35.343 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:19:35.340+0000 7f0ae4d7b740 1 -- 172.21.15.118:0/682065906 wait complete. 2024-03-24T00:19:35.343 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:19:35.341+0000 7f0ae4d7b740 20 client.26560 trim_cache size 0 max 16384 2024-03-24T00:19:35.391 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:19:35.391+0000 7f0ae4d7b740 1 Processor -- start 2024-03-24T00:19:35.391 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:19:35.392+0000 7f0ae4d7b740 1 -- start start 2024-03-24T00:19:35.394 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:19:35.394+0000 7f0ae4d7b740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5645df30c3b0 0x5645df324af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:19:35.394 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:19:35.395+0000 7f0ae4d7b740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5645df30cda0 0x5645df31da40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:19:35.395 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:19:35.395+0000 7f0ae4d7b740 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5645df311f10 0x5645df315b20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:19:35.395 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:19:35.395+0000 7f0adaa00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5645df30cda0 0x5645df31da40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:19:35.395 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:19:35.395+0000 7f0adaa00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5645df30cda0 0x5645df31da40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.118:34266/0 (socket says 172.21.15.118:34266) 2024-03-24T00:19:35.396 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:19:35.396+0000 7f0adaa00640 1 -- 172.21.15.118:0/1736910782 learned_addr learned my addr 172.21.15.118:0/1736910782 (peer_addr_for_me v2:172.21.15.118:0/0) 2024-03-24T00:19:35.396 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:19:35.396+0000 7f0ae4d7b740 1 -- 172.21.15.118:0/1736910782 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0adc1aee50 con 0x5645df311f10 2024-03-24T00:19:35.396 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:19:35.397+0000 7f0adbe00640 1 --2- 172.21.15.118:0/1736910782 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5645df311f10 0x5645df315b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:19:35.397 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:19:35.397+0000 7f0ae4d7b740 1 -- 172.21.15.118:0/1736910782 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0adc1b0800 con 0x5645df30cda0 2024-03-24T00:19:35.397 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:19:35.397+0000 7f0ae4d7b740 1 -- 172.21.15.118:0/1736910782 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0adc1af4b0 con 0x5645df30c3b0 2024-03-24T00:19:35.397 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:19:35.398+0000 7f0adb400640 1 --2- 172.21.15.118:0/1736910782 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5645df30c3b0 0x5645df324af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:19:35.399 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:19:35.399+0000 7f0adbe00640 1 -- 172.21.15.118:0/1736910782 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5645df30c3b0 msgr2=0x5645df324af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:19:35.399 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:19:35.399+0000 7f0adbe00640 1 --2- 172.21.15.118:0/1736910782 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5645df30c3b0 0x5645df324af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:19:35.399 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:19:35.399+0000 7f0adbe00640 1 -- 172.21.15.118:0/1736910782 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5645df30cda0 msgr2=0x5645df31da40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:19:35.399 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:19:35.399+0000 7f0adbe00640 1 --2- 172.21.15.118:0/1736910782 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5645df30cda0 0x5645df31da40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:19:35.399 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:19:35.400+0000 7f0adbe00640 1 -- 172.21.15.118:0/1736910782 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5645df325190 con 0x5645df311f10 2024-03-24T00:19:35.399 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:19:35.400+0000 7f0adb400640 1 --2- 172.21.15.118:0/1736910782 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5645df30c3b0 0x5645df324af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T00:19:35.400 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:19:35.400+0000 7f0adaa00640 1 --2- 172.21.15.118:0/1736910782 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5645df30cda0 0x5645df31da40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T00:19:35.400 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:19:35.401+0000 7f0adbe00640 1 --2- 172.21.15.118:0/1736910782 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5645df311f10 0x5645df315b20 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7f0ad40045e0 tx=0x7f0ad4034910 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2f70183c12ce694d server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:19:35.401 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:19:35.402+0000 7f0ac7400640 1 -- 172.21.15.118:0/1736910782 <== mon.0 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0ad4061050 con 0x5645df311f10 2024-03-24T00:19:35.402 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:19:35.402+0000 7f0ac7400640 1 -- 172.21.15.118:0/1736910782 <== mon.0 v2:172.21.15.118:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f0ac809f190 con 0x5645df311f10 2024-03-24T00:19:35.402 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:19:35.402+0000 7f0ac7400640 1 -- 172.21.15.118:0/1736910782 <== mon.0 v2:172.21.15.118:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0ad4033cc0 con 0x5645df311f10 2024-03-24T00:19:35.406 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:19:35.404+0000 7f0ae4d7b740 1 -- 172.21.15.118:0/1736910782 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5645df311f10 msgr2=0x5645df315b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:19:35.406 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:19:35.404+0000 7f0ae4d7b740 1 --2- 172.21.15.118:0/1736910782 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5645df311f10 0x5645df315b20 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7f0ad40045e0 tx=0x7f0ad4034910 comp rx=0 tx=0).stop 2024-03-24T00:19:35.406 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:19:35.406+0000 7f0ae4d7b740 1 -- 172.21.15.118:0/1736910782 shutdown_connections 2024-03-24T00:19:35.407 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:19:35.406+0000 7f0ae4d7b740 1 --2- 172.21.15.118:0/1736910782 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5645df311f10 0x5645df315b20 unknown :-1 s=CLOSED pgs=208 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:19:35.407 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:19:35.406+0000 7f0ae4d7b740 1 --2- 172.21.15.118:0/1736910782 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5645df30cda0 0x5645df31da40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:19:35.407 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:19:35.406+0000 7f0ae4d7b740 1 --2- 172.21.15.118:0/1736910782 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5645df30c3b0 0x5645df324af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:19:35.407 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:19:35.406+0000 7f0ae4d7b740 1 -- 172.21.15.118:0/1736910782 >> 172.21.15.118:0/1736910782 conn(0x5645df307920 msgr2=0x5645df311680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:19:35.407 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:19:35.407+0000 7f0ae4d7b740 1 -- 172.21.15.118:0/1736910782 shutdown_connections 2024-03-24T00:19:35.407 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:19:35.407+0000 7f0ae4d7b740 1 -- 172.21.15.118:0/1736910782 wait complete. 2024-03-24T00:19:35.407 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:19:35.408+0000 7f0ae4d7b740 1 -- start start 2024-03-24T00:19:35.407 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:19:35.408+0000 7f0ae4d7b740 1 Processor -- start 2024-03-24T00:19:35.411 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:19:35.410+0000 7f0ae4d7b740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5645df30c3b0 0x5645df31af20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:19:35.411 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:19:35.411+0000 7f0ae4d7b740 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5645df30cda0 0x5645df31b460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:19:35.411 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:19:35.411+0000 7f0adb400640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5645df30c3b0 0x5645df31af20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:19:35.411 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:19:35.411+0000 7f0adb400640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5645df30c3b0 0x5645df31af20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.118:34270/0 (socket says 172.21.15.118:34270) 2024-03-24T00:19:35.411 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:19:35.411+0000 7f0adaa00640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5645df30cda0 0x5645df31b460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:19:35.411 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:19:35.411+0000 7f0adb400640 1 -- 172.21.15.118:0/1649052402 learned_addr learned my addr 172.21.15.118:0/1649052402 (peer_addr_for_me v2:172.21.15.118:0/0) 2024-03-24T00:19:35.411 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:19:35.411+0000 7f0ae4d7b740 1 --2- 172.21.15.118:0/1649052402 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5645df311f10 0x5645df31c470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:19:35.412 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:19:35.412+0000 7f0ae4d7b740 1 -- 172.21.15.118:0/1649052402 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0adc1b0170 con 0x5645df30cda0 2024-03-24T00:19:35.412 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:19:35.412+0000 7f0ae4d7b740 1 -- 172.21.15.118:0/1649052402 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0adc1afb10 con 0x5645df30c3b0 2024-03-24T00:19:35.412 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:19:35.412+0000 7f0ae4d7b740 1 -- 172.21.15.118:0/1649052402 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0adc1175e0 con 0x5645df311f10 2024-03-24T00:19:35.413 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:19:35.412+0000 7f0adbe00640 1 --2- 172.21.15.118:0/1649052402 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5645df311f10 0x5645df31c470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:19:35.413 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:19:35.412+0000 7f0ae4d7b740 10 client.0 _ll_register_callbacks cb 0x7f0ad4009a60 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T00:19:35.413 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:19:35.413+0000 7f0adaa00640 1 -- 172.21.15.118:0/1649052402 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5645df311f10 msgr2=0x5645df31c470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:19:35.413 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:19:35.413+0000 7f0adaa00640 1 --2- 172.21.15.118:0/1649052402 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5645df311f10 0x5645df31c470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:19:35.413 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:19:35.413+0000 7f0adaa00640 1 -- 172.21.15.118:0/1649052402 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5645df30c3b0 msgr2=0x5645df31af20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:19:35.413 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:19:35.413+0000 7f0adaa00640 1 --2- 172.21.15.118:0/1649052402 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5645df30c3b0 0x5645df31af20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:19:35.414 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:19:35.413+0000 7f0adaa00640 1 -- 172.21.15.118:0/1649052402 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5645df316db0 con 0x5645df30cda0 2024-03-24T00:19:35.414 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:19:35.414+0000 7f0adb400640 1 --2- 172.21.15.118:0/1649052402 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5645df30c3b0 0x5645df31af20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T00:19:35.415 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:19:35.414+0000 7f0adaa00640 1 --2- 172.21.15.118:0/1649052402 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5645df30cda0 0x5645df31b460 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7f0ac809ad40 tx=0x7f0ac8009310 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ae127286c1f2ae1f server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:19:35.415 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:19:35.414+0000 7f0ac4c00640 10 client.0 ms_handle_connect on v2:172.21.15.118:3300/0 2024-03-24T00:19:35.417 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:19:35.417+0000 7f0ac4c00640 1 -- 172.21.15.118:0/1649052402 <== mon.0 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0ac809a690 con 0x5645df30cda0 2024-03-24T00:19:35.417 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:19:35.417+0000 7f0ac4c00640 1 -- 172.21.15.118:0/1649052402 <== mon.0 v2:172.21.15.118:3300/0 2 ==== osd_map(90..90 src has 1..90) v4 ==== 8277+0+0 (secure 0 0 0) 0x7f0ac8096d90 con 0x5645df30cda0 2024-03-24T00:19:35.417 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:19:35.417+0000 7f0ae4d7b740 1 -- 172.21.15.118:0/1649052402 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f0ac809f190 con 0x5645df30cda0 2024-03-24T00:19:35.418 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:19:35.419+0000 7f0ac4c00640 1 -- 172.21.15.118:0/1649052402 <== mon.0 v2:172.21.15.118:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) v2 ==== 24+0+0 (secure 0 0 0) 0x7f0ac80a37e0 con 0x5645df30cda0 2024-03-24T00:19:35.419 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:19:35.419+0000 7f0ae4d7b740 10 client.16395 fetch_fsmap learned FSMap version 29 2024-03-24T00:19:35.419 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:19:35.419+0000 7f0ae4d7b740 1 -- 172.21.15.118:0/1649052402 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) v3 -- 0x5645df302ec0 con 0x5645df30cda0 2024-03-24T00:19:35.420 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:19:35.420+0000 7f0ac4c00640 1 -- 172.21.15.118:0/1649052402 <== mon.0 v2:172.21.15.118:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f0ac809f190 con 0x5645df30cda0 2024-03-24T00:19:35.423 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:19:35.422+0000 7f0ac4c00640 1 -- 172.21.15.118:0/1649052402 <== mon.0 v2:172.21.15.118:3300/0 5 ==== fsmap(e 29) v1 ==== 7938+0+0 (secure 0 0 0) 0x7f0ac80a00b0 con 0x5645df30cda0 2024-03-24T00:19:35.423 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:19:35.422+0000 7f0ae4d7b740 10 client.16395 fetch_fsmap finished waiting for FSMap version 29 2024-03-24T00:19:35.423 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:19:35.422+0000 7f0ae4d7b740 10 client.16395 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-24T00:19:35.423 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:19:35.422+0000 7f0ae4d7b740 20 client.16395 populate_metadata read hostname 'smithi118' 2024-03-24T00:19:35.423 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:19:35.422+0000 7f0ae4d7b740 1 --2- 172.21.15.118:0/1649052402 >> [v2:172.21.15.189:6838/4074484168,v1:172.21.15.189:6839/4074484168] conn(0x5645df37dfe0 0x5645df3803d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:19:35.423 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:19:35.423+0000 7f0ae4d7b740 4 client.16395 mds_command: new command op to 24569 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-03-24T00:19:35.423 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:19:35.423+0000 7f0ae4d7b740 1 -- 172.21.15.118:0/1649052402 --> [v2:172.21.15.189:6838/4074484168,v1:172.21.15.189:6839/4074484168] -- command(tid 0: {"prefix": "flush journal"}) v1 -- 0x5645df2790c0 con 0x5645df37dfe0 2024-03-24T00:19:35.423 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:19:35.423+0000 7f0adb400640 1 --2- 172.21.15.118:0/1649052402 >> [v2:172.21.15.189:6838/4074484168,v1:172.21.15.189:6839/4074484168] conn(0x5645df37dfe0 0x5645df3803d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:19:35.424 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:19:35.424+0000 7f0adb400640 1 --2- 172.21.15.118:0/1649052402 >> [v2:172.21.15.189:6838/4074484168,v1:172.21.15.189:6839/4074484168] conn(0x5645df37dfe0 0x5645df3803d0 crc :-1 s=READY pgs=118 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=f4969bbc453e4703 server_cookie=a80f586eeeb316aa in_seq=0 out_seq=0 2024-03-24T00:19:35.424 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:19:35.424+0000 7f0ac4c00640 10 client.16395 ms_handle_connect on v2:172.21.15.189:6838/4074484168 2024-03-24T00:19:35.711 INFO:journalctl@ceph.mds.f.smithi193.stdout:Mar 24 00:19:35 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-f[67465]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: In function 'void Locker::set_xlocks_done(MutationImpl*, bool)' thread 7f753193c700 time 2024-03-24T00:19:35.353808+0000 2024-03-24T00:19:35.711 INFO:journalctl@ceph.mds.f.smithi193.stdout:Mar 24 00:19:35 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-f[67465]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: 626: FAILED ceph_assert(obj->is_auth()) 2024-03-24T00:19:35.711 INFO:journalctl@ceph.mds.f.smithi193.stdout:Mar 24 00:19:35 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-f[67465]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-24T00:19:35.712 INFO:journalctl@ceph.mds.f.smithi193.stdout:Mar 24 00:19:35 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-f[67465]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x135) [0x7f753bf664a1] 2024-03-24T00:19:35.712 INFO:journalctl@ceph.mds.f.smithi193.stdout:Mar 24 00:19:35 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-f[67465]: 2: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7f753bf66667] 2024-03-24T00:19:35.712 INFO:journalctl@ceph.mds.f.smithi193.stdout:Mar 24 00:19:35 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-f[67465]: 3: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x555b06046223] 2024-03-24T00:19:35.712 INFO:journalctl@ceph.mds.f.smithi193.stdout:Mar 24 00:19:35 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-f[67465]: 4: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x555b05ec05d6] 2024-03-24T00:19:35.712 INFO:journalctl@ceph.mds.f.smithi193.stdout:Mar 24 00:19:35 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-f[67465]: 5: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x555b05ec0da1] 2024-03-24T00:19:35.712 INFO:journalctl@ceph.mds.f.smithi193.stdout:Mar 24 00:19:35 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-f[67465]: 6: (Server::handle_client_rename(boost::intrusive_ptr const&)+0x259d) [0x555b05efd97d] 2024-03-24T00:19:35.712 INFO:journalctl@ceph.mds.f.smithi193.stdout:Mar 24 00:19:35 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-f[67465]: 7: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x555b05f13233] 2024-03-24T00:19:35.712 INFO:journalctl@ceph.mds.f.smithi193.stdout:Mar 24 00:19:35 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-f[67465]: 8: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x555b05f15197] 2024-03-24T00:19:35.712 INFO:journalctl@ceph.mds.f.smithi193.stdout:Mar 24 00:19:35 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-f[67465]: 9: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x555b05e1411b] 2024-03-24T00:19:35.712 INFO:journalctl@ceph.mds.f.smithi193.stdout:Mar 24 00:19:35 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-f[67465]: 10: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x555b05e146bc] 2024-03-24T00:19:35.712 INFO:journalctl@ceph.mds.f.smithi193.stdout:Mar 24 00:19:35 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-f[67465]: 11: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x555b05dfe1ff] 2024-03-24T00:19:35.712 INFO:journalctl@ceph.mds.f.smithi193.stdout:Mar 24 00:19:35 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-f[67465]: 12: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7f753c1f8488] 2024-03-24T00:19:35.712 INFO:journalctl@ceph.mds.f.smithi193.stdout:Mar 24 00:19:35 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-f[67465]: 13: (DispatchQueue::entry()+0x51f) [0x7f753c1f560f] 2024-03-24T00:19:35.712 INFO:journalctl@ceph.mds.f.smithi193.stdout:Mar 24 00:19:35 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-f[67465]: 14: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f753c2c8e81] 2024-03-24T00:19:35.712 INFO:journalctl@ceph.mds.f.smithi193.stdout:Mar 24 00:19:35 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-f[67465]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7f753ace31ca] 2024-03-24T00:19:35.713 INFO:journalctl@ceph.mds.f.smithi193.stdout:Mar 24 00:19:35 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-f[67465]: 16: clone() 2024-03-24T00:19:35.713 INFO:journalctl@ceph.mds.f.smithi193.stdout:Mar 24 00:19:35 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-f[67465]: 2024-03-24T00:19:35.354+0000 7f753193c700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: In function 'void Locker::set_xlocks_done(MutationImpl*, bool)' thread 7f753193c700 time 2024-03-24T00:19:35.353808+0000 2024-03-24T00:19:35.714 INFO:journalctl@ceph.mds.f.smithi193.stdout:Mar 24 00:19:35 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-f[67465]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: 626: FAILED ceph_assert(obj->is_auth()) 2024-03-24T00:19:35.714 INFO:journalctl@ceph.mds.f.smithi193.stdout:Mar 24 00:19:35 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-f[67465]: 2024-03-24T00:19:35.714 INFO:journalctl@ceph.mds.f.smithi193.stdout:Mar 24 00:19:35 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-f[67465]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-24T00:19:35.714 INFO:journalctl@ceph.mds.f.smithi193.stdout:Mar 24 00:19:35 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-f[67465]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x135) [0x7f753bf664a1] 2024-03-24T00:19:35.714 INFO:journalctl@ceph.mds.f.smithi193.stdout:Mar 24 00:19:35 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-f[67465]: 2: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7f753bf66667] 2024-03-24T00:19:35.714 INFO:journalctl@ceph.mds.f.smithi193.stdout:Mar 24 00:19:35 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-f[67465]: 3: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x555b06046223] 2024-03-24T00:19:35.714 INFO:journalctl@ceph.mds.f.smithi193.stdout:Mar 24 00:19:35 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-f[67465]: 4: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x555b05ec05d6] 2024-03-24T00:19:35.714 INFO:journalctl@ceph.mds.f.smithi193.stdout:Mar 24 00:19:35 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-f[67465]: 5: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x555b05ec0da1] 2024-03-24T00:19:35.714 INFO:journalctl@ceph.mds.f.smithi193.stdout:Mar 24 00:19:35 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-f[67465]: 6: (Server::handle_client_rename(boost::intrusive_ptr const&)+0x259d) [0x555b05efd97d] 2024-03-24T00:19:35.714 INFO:journalctl@ceph.mds.f.smithi193.stdout:Mar 24 00:19:35 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-f[67465]: 7: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x555b05f13233] 2024-03-24T00:19:35.714 INFO:journalctl@ceph.mds.f.smithi193.stdout:Mar 24 00:19:35 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-f[67465]: 8: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x555b05f15197] 2024-03-24T00:19:35.714 INFO:journalctl@ceph.mds.f.smithi193.stdout:Mar 24 00:19:35 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-f[67465]: 9: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x555b05e1411b] 2024-03-24T00:19:35.715 INFO:journalctl@ceph.mds.f.smithi193.stdout:Mar 24 00:19:35 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-f[67465]: 10: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x555b05e146bc] 2024-03-24T00:19:35.715 INFO:journalctl@ceph.mds.f.smithi193.stdout:Mar 24 00:19:35 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-f[67465]: 11: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x555b05dfe1ff] 2024-03-24T00:19:35.715 INFO:journalctl@ceph.mds.f.smithi193.stdout:Mar 24 00:19:35 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-f[67465]: 12: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7f753c1f8488] 2024-03-24T00:19:35.715 INFO:journalctl@ceph.mds.f.smithi193.stdout:Mar 24 00:19:35 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-f[67465]: 13: (DispatchQueue::entry()+0x51f) [0x7f753c1f560f] 2024-03-24T00:19:35.715 INFO:journalctl@ceph.mds.f.smithi193.stdout:Mar 24 00:19:35 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-f[67465]: 14: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f753c2c8e81] 2024-03-24T00:19:35.715 INFO:journalctl@ceph.mds.f.smithi193.stdout:Mar 24 00:19:35 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-f[67465]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7f753ace31ca] 2024-03-24T00:19:35.715 INFO:journalctl@ceph.mds.f.smithi193.stdout:Mar 24 00:19:35 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-f[67465]: 16: clone() 2024-03-24T00:19:35.715 INFO:journalctl@ceph.mds.f.smithi193.stdout:Mar 24 00:19:35 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-f[67465]: 2024-03-24T00:19:35.715 INFO:journalctl@ceph.mds.f.smithi193.stdout:Mar 24 00:19:35 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-f[67465]: *** Caught signal (Aborted) ** 2024-03-24T00:19:35.715 INFO:journalctl@ceph.mds.f.smithi193.stdout:Mar 24 00:19:35 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-f[67465]: in thread 7f753193c700 thread_name:ms_dispatch 2024-03-24T00:19:35.716 INFO:journalctl@ceph.mds.f.smithi193.stdout:Mar 24 00:19:35 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-f[67465]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-24T00:19:35.716 INFO:journalctl@ceph.mds.f.smithi193.stdout:Mar 24 00:19:35 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-f[67465]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f753acedd20] 2024-03-24T00:19:35.716 INFO:journalctl@ceph.mds.f.smithi193.stdout:Mar 24 00:19:35 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-f[67465]: 2: gsignal() 2024-03-24T00:19:35.716 INFO:journalctl@ceph.mds.f.smithi193.stdout:Mar 24 00:19:35 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-f[67465]: 3: abort() 2024-03-24T00:19:35.716 INFO:journalctl@ceph.mds.f.smithi193.stdout:Mar 24 00:19:35 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-f[67465]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x18f) [0x7f753bf664fb] 2024-03-24T00:19:35.716 INFO:journalctl@ceph.mds.f.smithi193.stdout:Mar 24 00:19:35 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-f[67465]: 5: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7f753bf66667] 2024-03-24T00:19:35.716 INFO:journalctl@ceph.mds.f.smithi193.stdout:Mar 24 00:19:35 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-f[67465]: 6: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x555b06046223] 2024-03-24T00:19:35.716 INFO:journalctl@ceph.mds.f.smithi193.stdout:Mar 24 00:19:35 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-f[67465]: 7: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x555b05ec05d6] 2024-03-24T00:19:35.716 INFO:journalctl@ceph.mds.f.smithi193.stdout:Mar 24 00:19:35 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-f[67465]: 8: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x555b05ec0da1] 2024-03-24T00:19:35.717 INFO:journalctl@ceph.mds.f.smithi193.stdout:Mar 24 00:19:35 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-f[67465]: 9: (Server::handle_client_rename(boost::intrusive_ptr const&)+0x259d) [0x555b05efd97d] 2024-03-24T00:19:35.717 INFO:journalctl@ceph.mds.f.smithi193.stdout:Mar 24 00:19:35 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-f[67465]: 10: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x555b05f13233] 2024-03-24T00:19:35.717 INFO:journalctl@ceph.mds.f.smithi193.stdout:Mar 24 00:19:35 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-f[67465]: 11: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x555b05f15197] 2024-03-24T00:19:35.717 INFO:journalctl@ceph.mds.f.smithi193.stdout:Mar 24 00:19:35 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-f[67465]: 12: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x555b05e1411b] 2024-03-24T00:19:35.717 INFO:journalctl@ceph.mds.f.smithi193.stdout:Mar 24 00:19:35 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-f[67465]: 13: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x555b05e146bc] 2024-03-24T00:19:35.717 INFO:journalctl@ceph.mds.f.smithi193.stdout:Mar 24 00:19:35 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-f[67465]: 14: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x555b05dfe1ff] 2024-03-24T00:19:35.717 INFO:journalctl@ceph.mds.f.smithi193.stdout:Mar 24 00:19:35 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-f[67465]: 15: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7f753c1f8488] 2024-03-24T00:19:35.717 INFO:journalctl@ceph.mds.f.smithi193.stdout:Mar 24 00:19:35 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-f[67465]: 16: (DispatchQueue::entry()+0x51f) [0x7f753c1f560f] 2024-03-24T00:19:35.717 INFO:journalctl@ceph.mds.f.smithi193.stdout:Mar 24 00:19:35 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-f[67465]: 17: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f753c2c8e81] 2024-03-24T00:19:35.717 INFO:journalctl@ceph.mds.f.smithi193.stdout:Mar 24 00:19:35 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-f[67465]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7f753ace31ca] 2024-03-24T00:19:35.718 INFO:journalctl@ceph.mds.f.smithi193.stdout:Mar 24 00:19:35 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-f[67465]: 19: clone() 2024-03-24T00:19:35.718 INFO:journalctl@ceph.mds.f.smithi193.stdout:Mar 24 00:19:35 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-f[67465]: 2024-03-24T00:19:35.356+0000 7f753193c700 -1 *** Caught signal (Aborted) ** 2024-03-24T00:19:35.718 INFO:journalctl@ceph.mds.f.smithi193.stdout:Mar 24 00:19:35 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-f[67465]: in thread 7f753193c700 thread_name:ms_dispatch 2024-03-24T00:19:35.718 INFO:journalctl@ceph.mds.f.smithi193.stdout:Mar 24 00:19:35 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-f[67465]: 2024-03-24T00:19:35.718 INFO:journalctl@ceph.mds.f.smithi193.stdout:Mar 24 00:19:35 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-f[67465]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-24T00:19:35.718 INFO:journalctl@ceph.mds.f.smithi193.stdout:Mar 24 00:19:35 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-f[67465]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f753acedd20] 2024-03-24T00:19:35.718 INFO:journalctl@ceph.mds.f.smithi193.stdout:Mar 24 00:19:35 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-f[67465]: 2: gsignal() 2024-03-24T00:19:35.718 INFO:journalctl@ceph.mds.f.smithi193.stdout:Mar 24 00:19:35 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-f[67465]: 3: abort() 2024-03-24T00:19:35.718 INFO:journalctl@ceph.mds.f.smithi193.stdout:Mar 24 00:19:35 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-f[67465]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x18f) [0x7f753bf664fb] 2024-03-24T00:19:35.719 INFO:journalctl@ceph.mds.f.smithi193.stdout:Mar 24 00:19:35 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-f[67465]: 5: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7f753bf66667] 2024-03-24T00:19:35.719 INFO:journalctl@ceph.mds.f.smithi193.stdout:Mar 24 00:19:35 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-f[67465]: 6: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x555b06046223] 2024-03-24T00:19:35.719 INFO:journalctl@ceph.mds.f.smithi193.stdout:Mar 24 00:19:35 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-f[67465]: 7: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x555b05ec05d6] 2024-03-24T00:19:35.720 INFO:journalctl@ceph.mds.f.smithi193.stdout:Mar 24 00:19:35 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-f[67465]: 8: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x555b05ec0da1] 2024-03-24T00:19:35.720 INFO:journalctl@ceph.mds.f.smithi193.stdout:Mar 24 00:19:35 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-f[67465]: 9: (Server::handle_client_rename(boost::intrusive_ptr const&)+0x259d) [0x555b05efd97d] 2024-03-24T00:19:35.720 INFO:journalctl@ceph.mds.f.smithi193.stdout:Mar 24 00:19:35 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-f[67465]: 10: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x555b05f13233] 2024-03-24T00:19:35.721 INFO:journalctl@ceph.mds.f.smithi193.stdout:Mar 24 00:19:35 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-f[67465]: 11: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x555b05f15197] 2024-03-24T00:19:35.721 INFO:journalctl@ceph.mds.f.smithi193.stdout:Mar 24 00:19:35 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-f[67465]: 12: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x555b05e1411b] 2024-03-24T00:19:35.721 INFO:journalctl@ceph.mds.f.smithi193.stdout:Mar 24 00:19:35 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-f[67465]: 13: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x555b05e146bc] 2024-03-24T00:19:35.721 INFO:journalctl@ceph.mds.f.smithi193.stdout:Mar 24 00:19:35 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-f[67465]: 14: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x555b05dfe1ff] 2024-03-24T00:19:35.721 INFO:journalctl@ceph.mds.f.smithi193.stdout:Mar 24 00:19:35 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-f[67465]: 15: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7f753c1f8488] 2024-03-24T00:19:35.721 INFO:journalctl@ceph.mds.f.smithi193.stdout:Mar 24 00:19:35 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-f[67465]: 16: (DispatchQueue::entry()+0x51f) [0x7f753c1f560f] 2024-03-24T00:19:35.721 INFO:journalctl@ceph.mds.f.smithi193.stdout:Mar 24 00:19:35 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-f[67465]: 17: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f753c2c8e81] 2024-03-24T00:19:35.721 INFO:journalctl@ceph.mds.f.smithi193.stdout:Mar 24 00:19:35 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-f[67465]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7f753ace31ca] 2024-03-24T00:19:35.721 INFO:journalctl@ceph.mds.f.smithi193.stdout:Mar 24 00:19:35 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-f[67465]: 19: clone() 2024-03-24T00:19:35.721 INFO:journalctl@ceph.mds.f.smithi193.stdout:Mar 24 00:19:35 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-f[67465]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-03-24T00:19:35.721 INFO:journalctl@ceph.mds.f.smithi193.stdout:Mar 24 00:19:35 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-f[67465]: 2024-03-24T00:19:35.721 INFO:journalctl@ceph.mds.f.smithi193.stdout:Mar 24 00:19:35 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-f[67465]: -1> 2024-03-24T00:19:35.354+0000 7f753193c700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: In function 'void Locker::set_xlocks_done(MutationImpl*, bool)' thread 7f753193c700 time 2024-03-24T00:19:35.353808+0000 2024-03-24T00:19:35.721 INFO:journalctl@ceph.mds.f.smithi193.stdout:Mar 24 00:19:35 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-f[67465]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: 626: FAILED ceph_assert(obj->is_auth()) 2024-03-24T00:19:35.721 INFO:journalctl@ceph.mds.f.smithi193.stdout:Mar 24 00:19:35 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-f[67465]: 2024-03-24T00:19:35.722 INFO:journalctl@ceph.mds.f.smithi193.stdout:Mar 24 00:19:35 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-f[67465]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-24T00:19:35.722 INFO:journalctl@ceph.mds.f.smithi193.stdout:Mar 24 00:19:35 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-f[67465]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x135) [0x7f753bf664a1] 2024-03-24T00:19:35.722 INFO:journalctl@ceph.mds.f.smithi193.stdout:Mar 24 00:19:35 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-f[67465]: 2: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7f753bf66667] 2024-03-24T00:19:35.722 INFO:journalctl@ceph.mds.f.smithi193.stdout:Mar 24 00:19:35 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-f[67465]: 3: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x555b06046223] 2024-03-24T00:19:35.722 INFO:journalctl@ceph.mds.f.smithi193.stdout:Mar 24 00:19:35 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-f[67465]: 4: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x555b05ec05d6] 2024-03-24T00:19:35.722 INFO:journalctl@ceph.mds.f.smithi193.stdout:Mar 24 00:19:35 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-f[67465]: 5: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x555b05ec0da1] 2024-03-24T00:19:35.722 INFO:journalctl@ceph.mds.f.smithi193.stdout:Mar 24 00:19:35 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-f[67465]: 6: (Server::handle_client_rename(boost::intrusive_ptr const&)+0x259d) [0x555b05efd97d] 2024-03-24T00:19:35.722 INFO:journalctl@ceph.mds.f.smithi193.stdout:Mar 24 00:19:35 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-f[67465]: 7: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x555b05f13233] 2024-03-24T00:19:35.722 INFO:journalctl@ceph.mds.f.smithi193.stdout:Mar 24 00:19:35 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-f[67465]: 8: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x555b05f15197] 2024-03-24T00:19:35.722 INFO:journalctl@ceph.mds.f.smithi193.stdout:Mar 24 00:19:35 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-f[67465]: 9: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x555b05e1411b] 2024-03-24T00:19:35.722 INFO:journalctl@ceph.mds.f.smithi193.stdout:Mar 24 00:19:35 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-f[67465]: 10: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x555b05e146bc] 2024-03-24T00:19:35.722 INFO:journalctl@ceph.mds.f.smithi193.stdout:Mar 24 00:19:35 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-f[67465]: 11: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x555b05dfe1ff] 2024-03-24T00:19:35.722 INFO:journalctl@ceph.mds.f.smithi193.stdout:Mar 24 00:19:35 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-f[67465]: 12: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7f753c1f8488] 2024-03-24T00:19:35.722 INFO:journalctl@ceph.mds.f.smithi193.stdout:Mar 24 00:19:35 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-f[67465]: 13: (DispatchQueue::entry()+0x51f) [0x7f753c1f560f] 2024-03-24T00:19:35.722 INFO:journalctl@ceph.mds.f.smithi193.stdout:Mar 24 00:19:35 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-f[67465]: 14: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f753c2c8e81] 2024-03-24T00:19:35.723 INFO:journalctl@ceph.mds.f.smithi193.stdout:Mar 24 00:19:35 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-f[67465]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7f753ace31ca] 2024-03-24T00:19:35.723 INFO:journalctl@ceph.mds.f.smithi193.stdout:Mar 24 00:19:35 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-f[67465]: 16: clone() 2024-03-24T00:19:35.723 INFO:journalctl@ceph.mds.f.smithi193.stdout:Mar 24 00:19:35 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-f[67465]: 2024-03-24T00:19:35.723 INFO:journalctl@ceph.mds.f.smithi193.stdout:Mar 24 00:19:35 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-f[67465]: 0> 2024-03-24T00:19:35.356+0000 7f753193c700 -1 *** Caught signal (Aborted) ** 2024-03-24T00:19:35.723 INFO:journalctl@ceph.mds.f.smithi193.stdout:Mar 24 00:19:35 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-f[67465]: in thread 7f753193c700 thread_name:ms_dispatch 2024-03-24T00:19:35.723 INFO:journalctl@ceph.mds.f.smithi193.stdout:Mar 24 00:19:35 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-f[67465]: 2024-03-24T00:19:35.723 INFO:journalctl@ceph.mds.f.smithi193.stdout:Mar 24 00:19:35 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-f[67465]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-24T00:19:35.723 INFO:journalctl@ceph.mds.f.smithi193.stdout:Mar 24 00:19:35 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-f[67465]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f753acedd20] 2024-03-24T00:19:35.723 INFO:journalctl@ceph.mds.f.smithi193.stdout:Mar 24 00:19:35 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-f[67465]: 2: gsignal() 2024-03-24T00:19:35.723 INFO:journalctl@ceph.mds.f.smithi193.stdout:Mar 24 00:19:35 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-f[67465]: 3: abort() 2024-03-24T00:19:35.723 INFO:journalctl@ceph.mds.f.smithi193.stdout:Mar 24 00:19:35 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-f[67465]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x18f) [0x7f753bf664fb] 2024-03-24T00:19:35.723 INFO:journalctl@ceph.mds.f.smithi193.stdout:Mar 24 00:19:35 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-f[67465]: 5: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7f753bf66667] 2024-03-24T00:19:35.723 INFO:journalctl@ceph.mds.f.smithi193.stdout:Mar 24 00:19:35 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-f[67465]: 6: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x555b06046223] 2024-03-24T00:19:35.724 INFO:journalctl@ceph.mds.f.smithi193.stdout:Mar 24 00:19:35 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-f[67465]: 7: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x555b05ec05d6] 2024-03-24T00:19:35.724 INFO:journalctl@ceph.mds.f.smithi193.stdout:Mar 24 00:19:35 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-f[67465]: 8: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x555b05ec0da1] 2024-03-24T00:19:35.724 INFO:journalctl@ceph.mds.f.smithi193.stdout:Mar 24 00:19:35 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-f[67465]: 9: (Server::handle_client_rename(boost::intrusive_ptr const&)+0x259d) [0x555b05efd97d] 2024-03-24T00:19:35.724 INFO:journalctl@ceph.mds.f.smithi193.stdout:Mar 24 00:19:35 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-f[67465]: 10: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x555b05f13233] 2024-03-24T00:19:35.724 INFO:journalctl@ceph.mds.f.smithi193.stdout:Mar 24 00:19:35 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-f[67465]: 11: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x555b05f15197] 2024-03-24T00:19:35.724 INFO:journalctl@ceph.mds.f.smithi193.stdout:Mar 24 00:19:35 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-f[67465]: 12: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x555b05e1411b] 2024-03-24T00:19:35.724 INFO:journalctl@ceph.mds.f.smithi193.stdout:Mar 24 00:19:35 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-f[67465]: 13: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x555b05e146bc] 2024-03-24T00:19:35.724 INFO:journalctl@ceph.mds.f.smithi193.stdout:Mar 24 00:19:35 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-f[67465]: 14: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x555b05dfe1ff] 2024-03-24T00:19:35.724 INFO:journalctl@ceph.mds.f.smithi193.stdout:Mar 24 00:19:35 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-f[67465]: 15: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7f753c1f8488] 2024-03-24T00:19:35.724 INFO:journalctl@ceph.mds.f.smithi193.stdout:Mar 24 00:19:35 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-f[67465]: 16: (DispatchQueue::entry()+0x51f) [0x7f753c1f560f] 2024-03-24T00:19:35.724 INFO:journalctl@ceph.mds.f.smithi193.stdout:Mar 24 00:19:35 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-f[67465]: 17: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f753c2c8e81] 2024-03-24T00:19:35.724 INFO:journalctl@ceph.mds.f.smithi193.stdout:Mar 24 00:19:35 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-f[67465]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7f753ace31ca] 2024-03-24T00:19:35.724 INFO:journalctl@ceph.mds.f.smithi193.stdout:Mar 24 00:19:35 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-f[67465]: 19: clone() 2024-03-24T00:19:35.724 INFO:journalctl@ceph.mds.f.smithi193.stdout:Mar 24 00:19:35 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-f[67465]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-03-24T00:19:35.725 INFO:journalctl@ceph.mds.f.smithi193.stdout:Mar 24 00:19:35 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-f[67465]: 2024-03-24T00:19:35.725 INFO:journalctl@ceph.mds.f.smithi193.stdout:Mar 24 00:19:35 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-f[67465]: -1> 2024-03-24T00:19:35.354+0000 7f753193c700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: In function 'void Locker::set_xlocks_done(MutationImpl*, bool)' thread 7f753193c700 time 2024-03-24T00:19:35.353808+0000 2024-03-24T00:19:35.725 INFO:journalctl@ceph.mds.f.smithi193.stdout:Mar 24 00:19:35 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-f[67465]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: 626: FAILED ceph_assert(obj->is_auth()) 2024-03-24T00:19:35.725 INFO:journalctl@ceph.mds.f.smithi193.stdout:Mar 24 00:19:35 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-f[67465]: 2024-03-24T00:19:35.725 INFO:journalctl@ceph.mds.f.smithi193.stdout:Mar 24 00:19:35 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-f[67465]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-24T00:19:35.725 INFO:journalctl@ceph.mds.f.smithi193.stdout:Mar 24 00:19:35 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-f[67465]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x135) [0x7f753bf664a1] 2024-03-24T00:19:35.725 INFO:journalctl@ceph.mds.f.smithi193.stdout:Mar 24 00:19:35 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-f[67465]: 2: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7f753bf66667] 2024-03-24T00:19:35.725 INFO:journalctl@ceph.mds.f.smithi193.stdout:Mar 24 00:19:35 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-f[67465]: 3: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x555b06046223] 2024-03-24T00:19:35.725 INFO:journalctl@ceph.mds.f.smithi193.stdout:Mar 24 00:19:35 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-f[67465]: 4: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x555b05ec05d6] 2024-03-24T00:19:35.726 INFO:journalctl@ceph.mds.f.smithi193.stdout:Mar 24 00:19:35 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-f[67465]: 5: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x555b05ec0da1] 2024-03-24T00:19:35.726 INFO:journalctl@ceph.mds.f.smithi193.stdout:Mar 24 00:19:35 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-f[67465]: 6: (Server::handle_client_rename(boost::intrusive_ptr const&)+0x259d) [0x555b05efd97d] 2024-03-24T00:19:35.726 INFO:journalctl@ceph.mds.f.smithi193.stdout:Mar 24 00:19:35 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-f[67465]: 7: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x555b05f13233] 2024-03-24T00:19:35.726 INFO:journalctl@ceph.mds.f.smithi193.stdout:Mar 24 00:19:35 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-f[67465]: 8: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x555b05f15197] 2024-03-24T00:19:35.726 INFO:journalctl@ceph.mds.f.smithi193.stdout:Mar 24 00:19:35 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-f[67465]: 9: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x555b05e1411b] 2024-03-24T00:19:35.726 INFO:journalctl@ceph.mds.f.smithi193.stdout:Mar 24 00:19:35 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-f[67465]: 10: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x555b05e146bc] 2024-03-24T00:19:35.726 INFO:journalctl@ceph.mds.f.smithi193.stdout:Mar 24 00:19:35 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-f[67465]: 11: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x555b05dfe1ff] 2024-03-24T00:19:35.726 INFO:journalctl@ceph.mds.f.smithi193.stdout:Mar 24 00:19:35 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-f[67465]: 12: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7f753c1f8488] 2024-03-24T00:19:35.726 INFO:journalctl@ceph.mds.f.smithi193.stdout:Mar 24 00:19:35 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-f[67465]: 13: (DispatchQueue::entry()+0x51f) [0x7f753c1f560f] 2024-03-24T00:19:35.726 INFO:journalctl@ceph.mds.f.smithi193.stdout:Mar 24 00:19:35 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-f[67465]: 14: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f753c2c8e81] 2024-03-24T00:19:35.727 INFO:journalctl@ceph.mds.f.smithi193.stdout:Mar 24 00:19:35 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-f[67465]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7f753ace31ca] 2024-03-24T00:19:35.727 INFO:journalctl@ceph.mds.f.smithi193.stdout:Mar 24 00:19:35 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-f[67465]: 16: clone() 2024-03-24T00:19:35.727 INFO:journalctl@ceph.mds.f.smithi193.stdout:Mar 24 00:19:35 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-f[67465]: 2024-03-24T00:19:35.727 INFO:journalctl@ceph.mds.f.smithi193.stdout:Mar 24 00:19:35 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-f[67465]: 0> 2024-03-24T00:19:35.356+0000 7f753193c700 -1 *** Caught signal (Aborted) ** 2024-03-24T00:19:35.727 INFO:journalctl@ceph.mds.f.smithi193.stdout:Mar 24 00:19:35 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-f[67465]: in thread 7f753193c700 thread_name:ms_dispatch 2024-03-24T00:19:35.727 INFO:journalctl@ceph.mds.f.smithi193.stdout:Mar 24 00:19:35 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-f[67465]: 2024-03-24T00:19:35.727 INFO:journalctl@ceph.mds.f.smithi193.stdout:Mar 24 00:19:35 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-f[67465]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-24T00:19:35.727 INFO:journalctl@ceph.mds.f.smithi193.stdout:Mar 24 00:19:35 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-f[67465]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f753acedd20] 2024-03-24T00:19:35.727 INFO:journalctl@ceph.mds.f.smithi193.stdout:Mar 24 00:19:35 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-f[67465]: 2: gsignal() 2024-03-24T00:19:35.727 INFO:journalctl@ceph.mds.f.smithi193.stdout:Mar 24 00:19:35 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-f[67465]: 3: abort() 2024-03-24T00:19:35.727 INFO:journalctl@ceph.mds.f.smithi193.stdout:Mar 24 00:19:35 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-f[67465]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x18f) [0x7f753bf664fb] 2024-03-24T00:19:35.727 INFO:journalctl@ceph.mds.f.smithi193.stdout:Mar 24 00:19:35 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-f[67465]: 5: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7f753bf66667] 2024-03-24T00:19:35.727 INFO:journalctl@ceph.mds.f.smithi193.stdout:Mar 24 00:19:35 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-f[67465]: 6: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x555b06046223] 2024-03-24T00:19:35.727 INFO:journalctl@ceph.mds.f.smithi193.stdout:Mar 24 00:19:35 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-f[67465]: 7: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x555b05ec05d6] 2024-03-24T00:19:35.728 INFO:journalctl@ceph.mds.f.smithi193.stdout:Mar 24 00:19:35 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-f[67465]: 8: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x555b05ec0da1] 2024-03-24T00:19:35.728 INFO:journalctl@ceph.mds.f.smithi193.stdout:Mar 24 00:19:35 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-f[67465]: 9: (Server::handle_client_rename(boost::intrusive_ptr const&)+0x259d) [0x555b05efd97d] 2024-03-24T00:19:35.728 INFO:journalctl@ceph.mds.f.smithi193.stdout:Mar 24 00:19:35 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-f[67465]: 10: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x555b05f13233] 2024-03-24T00:19:35.728 INFO:journalctl@ceph.mds.f.smithi193.stdout:Mar 24 00:19:35 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-f[67465]: 11: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x555b05f15197] 2024-03-24T00:19:35.728 INFO:journalctl@ceph.mds.f.smithi193.stdout:Mar 24 00:19:35 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-f[67465]: 12: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x555b05e1411b] 2024-03-24T00:19:35.728 INFO:journalctl@ceph.mds.f.smithi193.stdout:Mar 24 00:19:35 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-f[67465]: 13: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x555b05e146bc] 2024-03-24T00:19:35.728 INFO:journalctl@ceph.mds.f.smithi193.stdout:Mar 24 00:19:35 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-f[67465]: 14: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x555b05dfe1ff] 2024-03-24T00:19:35.728 INFO:journalctl@ceph.mds.f.smithi193.stdout:Mar 24 00:19:35 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-f[67465]: 15: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7f753c1f8488] 2024-03-24T00:19:35.728 INFO:journalctl@ceph.mds.f.smithi193.stdout:Mar 24 00:19:35 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-f[67465]: 16: (DispatchQueue::entry()+0x51f) [0x7f753c1f560f] 2024-03-24T00:19:35.728 INFO:journalctl@ceph.mds.f.smithi193.stdout:Mar 24 00:19:35 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-f[67465]: 17: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f753c2c8e81] 2024-03-24T00:19:35.728 INFO:journalctl@ceph.mds.f.smithi193.stdout:Mar 24 00:19:35 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-f[67465]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7f753ace31ca] 2024-03-24T00:19:35.728 INFO:journalctl@ceph.mds.f.smithi193.stdout:Mar 24 00:19:35 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-f[67465]: 19: clone() 2024-03-24T00:19:35.728 INFO:journalctl@ceph.mds.f.smithi193.stdout:Mar 24 00:19:35 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-f[67465]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-03-24T00:19:35.728 INFO:journalctl@ceph.mds.f.smithi193.stdout:Mar 24 00:19:35 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-f[67465]: 2024-03-24T00:19:35.728 INFO:journalctl@ceph.mds.f.smithi193.stdout:Mar 24 00:19:35 smithi193 podman[70529]: 2024-03-24 00:19:35.434536099 +0000 UTC m=+0.022841973 container died b0ac59f98847bdf62c8c530bbedfda67e984890ecfc89cae5f599bd350b72480 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-f, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, io.buildah.version=1.33.5, org.label-schema.build-date=20240319, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, ceph=True, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux ) 2024-03-24T00:19:36.477 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:19:36 smithi193 ceph-mon[36878]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-03-24T00:19:36.477 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:19:36 smithi193 ceph-mon[36878]: pgmap v515: 97 pgs: 97 active+clean; 96 MiB data, 625 MiB used, 1.0 TiB / 1.0 TiB avail; 500 KiB/s rd, 2.0 MiB/s wr, 250 op/s 2024-03-24T00:19:36.477 INFO:journalctl@ceph.mds.f.smithi193.stdout:Mar 24 00:19:36 smithi193 podman[70529]: 2024-03-24 00:19:36.242002378 +0000 UTC m=+0.830308253 container remove b0ac59f98847bdf62c8c530bbedfda67e984890ecfc89cae5f599bd350b72480 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-f, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240319, org.label-schema.schema-version=1.0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True) 2024-03-24T00:19:36.477 INFO:journalctl@ceph.mds.f.smithi193.stdout:Mar 24 00:19:36 smithi193 systemd[1]: ceph-d79a5420-e971-11ee-95cd-87774f69a715@mds.f.service: Main process exited, code=exited, status=134/n/a 2024-03-24T00:19:36.490 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:19:36 smithi118 ceph-mon[31535]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-03-24T00:19:36.490 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:19:36 smithi118 ceph-mon[31535]: pgmap v515: 97 pgs: 97 active+clean; 96 MiB data, 625 MiB used, 1.0 TiB / 1.0 TiB avail; 500 KiB/s rd, 2.0 MiB/s wr, 250 op/s 2024-03-24T00:19:36.703 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:19:36 smithi189 ceph-mon[36187]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-03-24T00:19:36.704 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:19:36 smithi189 ceph-mon[36187]: pgmap v515: 97 pgs: 97 active+clean; 96 MiB data, 625 MiB used, 1.0 TiB / 1.0 TiB avail; 500 KiB/s rd, 2.0 MiB/s wr, 250 op/s 2024-03-24T00:19:36.843 INFO:tasks.workunit.client.0.smithi118.stdout: 27 323063 97 244604 94 186152 246 2024-03-24T00:19:36.960 INFO:journalctl@ceph.mds.f.smithi193.stdout:Mar 24 00:19:36 smithi193 systemd[1]: ceph-d79a5420-e971-11ee-95cd-87774f69a715@mds.f.service: Failed with result 'exit-code'. 2024-03-24T00:19:36.961 INFO:journalctl@ceph.mds.f.smithi193.stdout:Mar 24 00:19:36 smithi193 systemd[1]: ceph-d79a5420-e971-11ee-95cd-87774f69a715@mds.f.service: Consumed 2min 2.716s CPU time. 2024-03-24T00:19:39.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:19:38 smithi118 ceph-mon[31535]: pgmap v516: 97 pgs: 97 active+clean; 97 MiB data, 630 MiB used, 1.0 TiB / 1.0 TiB avail; 622 KiB/s rd, 2.2 MiB/s wr, 278 op/s 2024-03-24T00:19:39.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:19:38 smithi189 ceph-mon[36187]: pgmap v516: 97 pgs: 97 active+clean; 97 MiB data, 630 MiB used, 1.0 TiB / 1.0 TiB avail; 622 KiB/s rd, 2.2 MiB/s wr, 278 op/s 2024-03-24T00:19:39.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:19:38 smithi193 ceph-mon[36878]: pgmap v516: 97 pgs: 97 active+clean; 97 MiB data, 630 MiB used, 1.0 TiB / 1.0 TiB avail; 622 KiB/s rd, 2.2 MiB/s wr, 278 op/s 2024-03-24T00:19:41.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:19:40 smithi118 ceph-mon[31535]: pgmap v517: 97 pgs: 97 active+clean; 97 MiB data, 631 MiB used, 1.0 TiB / 1.0 TiB avail; 488 KiB/s rd, 2.1 MiB/s wr, 297 op/s 2024-03-24T00:19:41.203 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:19:40 smithi189 ceph-mon[36187]: pgmap v517: 97 pgs: 97 active+clean; 97 MiB data, 631 MiB used, 1.0 TiB / 1.0 TiB avail; 488 KiB/s rd, 2.1 MiB/s wr, 297 op/s 2024-03-24T00:19:41.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:19:40 smithi193 ceph-mon[36878]: pgmap v517: 97 pgs: 97 active+clean; 97 MiB data, 631 MiB used, 1.0 TiB / 1.0 TiB avail; 488 KiB/s rd, 2.1 MiB/s wr, 297 op/s 2024-03-24T00:19:42.208 INFO:teuthology.orchestra.run:Running command with timeout 900 2024-03-24T00:19:42.208 DEBUG:teuthology.orchestra.run.smithi118:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph tell mds.1:0 scrub status 2024-03-24T00:19:42.422 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:42.421+0000 7f80f5a00640 1 -- 172.21.15.118:0/2197594310 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f80f0103260 msgr2=0x7f80f0109af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:19:42.422 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:42.421+0000 7f80f5a00640 1 --2- 172.21.15.118:0/2197594310 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f80f0103260 0x7f80f0109af0 secure :-1 s=READY pgs=393 cs=0 l=1 rev1=1 crypto rx=0x7f80e8009ca0 tx=0x7f80e802c100 comp rx=0 tx=0).stop 2024-03-24T00:19:42.422 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:42.421+0000 7f80f5a00640 1 -- 172.21.15.118:0/2197594310 shutdown_connections 2024-03-24T00:19:42.422 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:42.421+0000 7f80f5a00640 1 --2- 172.21.15.118:0/2197594310 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f80f0103260 0x7f80f0109af0 unknown :-1 s=CLOSED pgs=393 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:19:42.422 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:42.421+0000 7f80f5a00640 1 --2- 172.21.15.118:0/2197594310 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f80f01028a0 0x7f80f0102d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:19:42.423 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:42.421+0000 7f80f5a00640 1 --2- 172.21.15.118:0/2197594310 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f80f01016b0 0x7f80f0101ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:19:42.423 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:42.421+0000 7f80f5a00640 1 -- 172.21.15.118:0/2197594310 >> 172.21.15.118:0/2197594310 conn(0x7f80f00fcc40 msgr2=0x7f80f00ff060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:19:42.423 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:42.421+0000 7f80f5a00640 1 -- 172.21.15.118:0/2197594310 shutdown_connections 2024-03-24T00:19:42.423 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:42.421+0000 7f80f5a00640 1 -- 172.21.15.118:0/2197594310 wait complete. 2024-03-24T00:19:42.425 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:42.422+0000 7f80f5a00640 1 Processor -- start 2024-03-24T00:19:42.425 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:42.422+0000 7f80f5a00640 1 -- start start 2024-03-24T00:19:42.425 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:42.423+0000 7f80f5a00640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f80f01016b0 0x7f80f0075450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:19:42.425 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:42.423+0000 7f80f5a00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f80f01028a0 0x7f80f0073a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:19:42.425 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:42.423+0000 7f80f5a00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f80f0103260 0x7f80f0073fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:19:42.425 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:42.423+0000 7f80f5a00640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f80f010d310 con 0x7f80f01016b0 2024-03-24T00:19:42.426 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:42.423+0000 7f80f5a00640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f80f010d190 con 0x7f80f0103260 2024-03-24T00:19:42.426 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:42.423+0000 7f80f5a00640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f80f010d490 con 0x7f80f01028a0 2024-03-24T00:19:42.426 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:42.423+0000 7f80eea00640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f80f01016b0 0x7f80f0075450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:19:42.426 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:42.423+0000 7f80ef400640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f80f0103260 0x7f80f0073fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:19:42.426 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:42.423+0000 7f80eea00640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f80f01016b0 0x7f80f0075450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.118:3300/0 says I am v2:172.21.15.118:38518/0 (socket says 172.21.15.118:38518) 2024-03-24T00:19:42.426 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:42.423+0000 7f80e6000640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f80f01028a0 0x7f80f0073a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:19:42.426 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:42.423+0000 7f80eea00640 1 -- 172.21.15.118:0/2325826204 learned_addr learned my addr 172.21.15.118:0/2325826204 (peer_addr_for_me v2:172.21.15.118:0/0) 2024-03-24T00:19:42.426 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:42.423+0000 7f80ef400640 1 -- 172.21.15.118:0/2325826204 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f80f01028a0 msgr2=0x7f80f0073a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:19:42.426 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:42.423+0000 7f80ef400640 1 --2- 172.21.15.118:0/2325826204 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f80f01028a0 0x7f80f0073a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:19:42.426 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:42.423+0000 7f80ef400640 1 -- 172.21.15.118:0/2325826204 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f80f01016b0 msgr2=0x7f80f0075450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:19:42.426 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:42.423+0000 7f80ef400640 1 --2- 172.21.15.118:0/2325826204 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f80f01016b0 0x7f80f0075450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:19:42.426 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:42.423+0000 7f80ef400640 1 -- 172.21.15.118:0/2325826204 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f80f0074860 con 0x7f80f0103260 2024-03-24T00:19:42.427 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:42.423+0000 7f80eea00640 1 --2- 172.21.15.118:0/2325826204 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f80f01016b0 0x7f80f0075450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T00:19:42.427 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:42.423+0000 7f80e6000640 1 --2- 172.21.15.118:0/2325826204 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f80f01028a0 0x7f80f0073a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T00:19:42.427 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:42.424+0000 7f80ef400640 1 --2- 172.21.15.118:0/2325826204 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f80f0103260 0x7f80f0073fd0 secure :-1 s=READY pgs=394 cs=0 l=1 rev1=1 crypto rx=0x7f80e8037040 tx=0x7f80e8035d10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:19:42.427 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:42.424+0000 7f80e7e00640 1 -- 172.21.15.118:0/2325826204 <== mon.2 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f80e8039040 con 0x7f80f0103260 2024-03-24T00:19:42.427 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:42.424+0000 7f80e7e00640 1 -- 172.21.15.118:0/2325826204 <== mon.2 v2:172.21.15.189:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f80e8044070 con 0x7f80f0103260 2024-03-24T00:19:42.427 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:42.424+0000 7f80e7e00640 1 -- 172.21.15.118:0/2325826204 <== mon.2 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f80e8004450 con 0x7f80f0103260 2024-03-24T00:19:42.427 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:42.424+0000 7f80f5a00640 1 -- 172.21.15.118:0/2325826204 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f80f01a1a10 con 0x7f80f0103260 2024-03-24T00:19:42.427 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:42.424+0000 7f80f5a00640 1 -- 172.21.15.118:0/2325826204 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f80f01a1f80 con 0x7f80f0103260 2024-03-24T00:19:42.427 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:42.427+0000 7f80e7e00640 1 -- 172.21.15.118:0/2325826204 <== mon.2 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 20) v1 ==== 137291+0+0 (secure 0 0 0) 0x7f80e80045f0 con 0x7f80f0103260 2024-03-24T00:19:42.428 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:42.428+0000 7f80e7e00640 1 --2- 172.21.15.118:0/2325826204 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7f80d40a2880 0x7f80d40a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:19:42.429 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:42.428+0000 7f80e7e00640 1 -- 172.21.15.118:0/2325826204 <== mon.2 v2:172.21.15.189:3300/0 5 ==== osd_map(90..90 src has 1..90) v4 ==== 8277+0+0 (secure 0 0 0) 0x7f80e80f0fe0 con 0x7f80f0103260 2024-03-24T00:19:42.429 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:42.429+0000 7f80eea00640 1 --2- 172.21.15.118:0/2325826204 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7f80d40a2880 0x7f80d40a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:19:42.429 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:42.429+0000 7f80f87de740 1 Processor -- start 2024-03-24T00:19:42.429 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:42.429+0000 7f80f87de740 1 -- start start 2024-03-24T00:19:42.431 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:42.429+0000 7f80f87de740 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55c05a4eee80 0x55c05a4ef2a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:19:42.431 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:42.429+0000 7f80f87de740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55c05a4ef870 0x55c05a4f4370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:19:42.431 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:42.429+0000 7f80f87de740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55c05a4f49e0 0x55c05a4f6e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:19:42.431 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:42.429+0000 7f80f87de740 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c05a454250 con 0x55c05a4eee80 2024-03-24T00:19:42.431 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:42.429+0000 7f80f87de740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c05a3fca30 con 0x55c05a4ef870 2024-03-24T00:19:42.431 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:42.429+0000 7f80f87de740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c05a4502f0 con 0x55c05a4f49e0 2024-03-24T00:19:42.431 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:42.429+0000 7f80eea00640 1 --2- 172.21.15.118:0/2325826204 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7f80d40a2880 0x7f80d40a4d40 secure :-1 s=READY pgs=342 cs=0 l=1 rev1=1 crypto rx=0x7f80e00104e0 tx=0x7f80e000b410 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:19:42.431 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:42.429+0000 7f80eea00640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55c05a4eee80 0x55c05a4ef2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:19:42.432 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:42.430+0000 7f80e6000640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55c05a4ef870 0x55c05a4f4370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:19:42.432 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:42.430+0000 7f80eea00640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55c05a4eee80 0x55c05a4ef2a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.118:3300/0 says I am v2:172.21.15.118:38520/0 (socket says 172.21.15.118:38520) 2024-03-24T00:19:42.432 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:42.430+0000 7f80eea00640 1 -- 172.21.15.118:0/4135845067 learned_addr learned my addr 172.21.15.118:0/4135845067 (peer_addr_for_me v2:172.21.15.118:0/0) 2024-03-24T00:19:42.432 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:42.430+0000 7f80ef400640 1 --2- 172.21.15.118:0/4135845067 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55c05a4f49e0 0x55c05a4f6e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:19:42.432 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:42.430+0000 7f80e6000640 1 -- 172.21.15.118:0/4135845067 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55c05a4f49e0 msgr2=0x55c05a4f6e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:19:42.432 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:42.430+0000 7f80e6000640 1 --2- 172.21.15.118:0/4135845067 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55c05a4f49e0 0x55c05a4f6e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:19:42.432 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:42.430+0000 7f80e6000640 1 -- 172.21.15.118:0/4135845067 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55c05a4eee80 msgr2=0x55c05a4ef2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:19:42.432 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:42.430+0000 7f80e6000640 1 --2- 172.21.15.118:0/4135845067 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55c05a4eee80 0x55c05a4ef2a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:19:42.432 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:42.430+0000 7f80e6000640 1 -- 172.21.15.118:0/4135845067 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55c05a3e7690 con 0x55c05a4ef870 2024-03-24T00:19:42.432 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:42.430+0000 7f80e6000640 1 --2- 172.21.15.118:0/4135845067 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55c05a4ef870 0x55c05a4f4370 secure :-1 s=READY pgs=395 cs=0 l=1 rev1=1 crypto rx=0x7f80d8009fa0 tx=0x7f80d80118d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=c129867115d0064a server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:19:42.432 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:42.430+0000 7f80dfe00640 1 -- 172.21.15.118:0/4135845067 <== mon.2 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f80d8004020 con 0x55c05a4ef870 2024-03-24T00:19:42.432 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:42.430+0000 7f80dfe00640 1 -- 172.21.15.118:0/4135845067 <== mon.2 v2:172.21.15.189:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f80d8005500 con 0x55c05a4ef870 2024-03-24T00:19:42.432 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:42.430+0000 7f80dfe00640 1 -- 172.21.15.118:0/4135845067 <== mon.2 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f80d8004d40 con 0x55c05a4ef870 2024-03-24T00:19:42.435 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:42.431+0000 7f80f87de740 1 -- 172.21.15.118:0/4135845067 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55c05a4ef870 msgr2=0x55c05a4f4370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:19:42.436 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:42.431+0000 7f80f87de740 1 --2- 172.21.15.118:0/4135845067 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55c05a4ef870 0x55c05a4f4370 secure :-1 s=READY pgs=395 cs=0 l=1 rev1=1 crypto rx=0x7f80d8009fa0 tx=0x7f80d80118d0 comp rx=0 tx=0).stop 2024-03-24T00:19:42.436 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:42.431+0000 7f80f87de740 1 -- 172.21.15.118:0/4135845067 shutdown_connections 2024-03-24T00:19:42.436 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:42.431+0000 7f80f87de740 1 --2- 172.21.15.118:0/4135845067 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55c05a4f49e0 0x55c05a4f6e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:19:42.436 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:42.431+0000 7f80f87de740 1 --2- 172.21.15.118:0/4135845067 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55c05a4ef870 0x55c05a4f4370 unknown :-1 s=CLOSED pgs=395 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:19:42.436 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:42.431+0000 7f80f87de740 1 --2- 172.21.15.118:0/4135845067 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55c05a4eee80 0x55c05a4ef2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:19:42.436 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:42.431+0000 7f80f87de740 1 -- 172.21.15.118:0/4135845067 >> 172.21.15.118:0/4135845067 conn(0x55c05a4b7200 msgr2=0x55c05a4ea7f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:19:42.436 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:42.431+0000 7f80f87de740 1 -- 172.21.15.118:0/4135845067 shutdown_connections 2024-03-24T00:19:42.436 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:42.431+0000 7f80f87de740 1 -- 172.21.15.118:0/4135845067 wait complete. 2024-03-24T00:19:42.436 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:42.432+0000 7f80f87de740 1 -- start start 2024-03-24T00:19:42.436 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:42.432+0000 7f80f87de740 1 Processor -- start 2024-03-24T00:19:42.436 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:42.432+0000 7f80f87de740 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55c05a4eee80 0x55c05a4f8df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:19:42.436 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:42.432+0000 7f80f87de740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55c05a4ef870 0x55c05a4f9330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:19:42.437 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:42.432+0000 7f80f87de740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55c05a4f49e0 0x55c05a500370 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:19:42.437 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:42.432+0000 7f80f87de740 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c05a4f7990 con 0x55c05a4eee80 2024-03-24T00:19:42.437 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:42.432+0000 7f80f87de740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c05a40fe80 con 0x55c05a4f49e0 2024-03-24T00:19:42.437 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:42.432+0000 7f80f87de740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c05a4f7b10 con 0x55c05a4ef870 2024-03-24T00:19:42.437 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:42.432+0000 7f80f87de740 10 client.0 _ll_register_callbacks cb 0x55c05a422130 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T00:19:42.437 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:42.433+0000 7f80eea00640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55c05a4eee80 0x55c05a4f8df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:19:42.437 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:42.433+0000 7f80e6000640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55c05a4ef870 0x55c05a4f9330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:19:42.437 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:42.433+0000 7f80ef400640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55c05a4f49e0 0x55c05a500370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:19:42.437 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:42.433+0000 7f80ef400640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55c05a4f49e0 0x55c05a500370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.118:34316/0 (socket says 172.21.15.118:34316) 2024-03-24T00:19:42.437 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:42.433+0000 7f80ef400640 1 -- 172.21.15.118:0/2335635957 learned_addr learned my addr 172.21.15.118:0/2335635957 (peer_addr_for_me v2:172.21.15.118:0/0) 2024-03-24T00:19:42.437 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:42.433+0000 7f80e6000640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55c05a4ef870 0x55c05a4f9330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.118:50634/0 (socket says 172.21.15.118:50634) 2024-03-24T00:19:42.437 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:42.433+0000 7f80ef400640 1 -- 172.21.15.118:0/2335635957 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55c05a4ef870 msgr2=0x55c05a4f9330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:19:42.438 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:42.433+0000 7f80ef400640 1 --2- 172.21.15.118:0/2335635957 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55c05a4ef870 0x55c05a4f9330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:19:42.438 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:42.433+0000 7f80ef400640 1 -- 172.21.15.118:0/2335635957 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55c05a4eee80 msgr2=0x55c05a4f8df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:19:42.438 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:42.433+0000 7f80ef400640 1 --2- 172.21.15.118:0/2335635957 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55c05a4eee80 0x55c05a4f8df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:19:42.438 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:42.433+0000 7f80ef400640 1 -- 172.21.15.118:0/2335635957 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55c05a2327d0 con 0x55c05a4f49e0 2024-03-24T00:19:42.438 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:42.433+0000 7f80ef400640 1 --2- 172.21.15.118:0/2335635957 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55c05a4f49e0 0x55c05a500370 secure :-1 s=READY pgs=396 cs=0 l=1 rev1=1 crypto rx=0x7f80e8005e30 tx=0x7f80e80bd310 comp rx=0 tx=0).ready entity=mon.2 client_cookie=69388aabd94f8c3f server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:19:42.438 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:42.433+0000 7f80dcc00640 10 client.0 ms_handle_connect on v2:172.21.15.189:3300/0 2024-03-24T00:19:42.438 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:42.433+0000 7f80dcc00640 1 -- 172.21.15.118:0/2335635957 <== mon.2 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f80e80d6060 con 0x55c05a4f49e0 2024-03-24T00:19:42.438 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:42.433+0000 7f80f87de740 1 -- 172.21.15.118:0/2335635957 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f80d8005500 con 0x55c05a4f49e0 2024-03-24T00:19:42.438 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:42.433+0000 7f80dcc00640 1 -- 172.21.15.118:0/2335635957 <== mon.2 v2:172.21.15.189:3300/0 2 ==== osd_map(90..90 src has 1..90) v4 ==== 8277+0+0 (secure 0 0 0) 0x7f80e80bdb50 con 0x55c05a4f49e0 2024-03-24T00:19:42.438 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:42.434+0000 7f80dcc00640 1 -- 172.21.15.118:0/2335635957 <== mon.2 v2:172.21.15.189:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) v2 ==== 24+0+0 (secure 0 0 0) 0x7f80e80dc1a0 con 0x55c05a4f49e0 2024-03-24T00:19:42.438 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:42.434+0000 7f80f87de740 10 client.26564 fetch_fsmap learned FSMap version 29 2024-03-24T00:19:42.438 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:42.434+0000 7f80f87de740 1 -- 172.21.15.118:0/2335635957 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=4+}) v3 -- 0x55c05a1683f0 con 0x55c05a4f49e0 2024-03-24T00:19:42.438 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:42.434+0000 7f80dcc00640 1 -- 172.21.15.118:0/2335635957 <== mon.2 v2:172.21.15.189:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f80d8005500 con 0x55c05a4f49e0 2024-03-24T00:19:42.439 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:42.434+0000 7f80dcc00640 1 -- 172.21.15.118:0/2335635957 <== mon.2 v2:172.21.15.189:3300/0 5 ==== fsmap(e 29) v1 ==== 7938+0+0 (secure 0 0 0) 0x7f80e803c030 con 0x55c05a4f49e0 2024-03-24T00:19:42.439 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:42.434+0000 7f80f87de740 10 client.26564 fetch_fsmap finished waiting for FSMap version 29 2024-03-24T00:19:42.439 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:42.434+0000 7f80f87de740 10 client.26564 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-24T00:19:42.439 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:42.434+0000 7f80f87de740 20 client.26564 populate_metadata read hostname 'smithi118' 2024-03-24T00:19:42.439 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:42.434+0000 7f80f87de740 1 --2- 172.21.15.118:0/2335635957 >> [v2:172.21.15.189:6838/4074484168,v1:172.21.15.189:6839/4074484168] conn(0x55c05a504b80 0x55c05a506f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:19:42.439 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:42.434+0000 7f80f87de740 4 client.26564 mds_command: new command op to 24569 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-24T00:19:42.439 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:42.434+0000 7f80f87de740 1 -- 172.21.15.118:0/2335635957 --> [v2:172.21.15.189:6838/4074484168,v1:172.21.15.189:6839/4074484168] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55c05a461790 con 0x55c05a504b80 2024-03-24T00:19:42.439 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:42.434+0000 7f80eea00640 1 --2- 172.21.15.118:0/2335635957 >> [v2:172.21.15.189:6838/4074484168,v1:172.21.15.189:6839/4074484168] conn(0x55c05a504b80 0x55c05a506f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:19:42.439 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:42.435+0000 7f80eea00640 1 --2- 172.21.15.118:0/2335635957 >> [v2:172.21.15.189:6838/4074484168,v1:172.21.15.189:6839/4074484168] conn(0x55c05a504b80 0x55c05a506f70 crc :-1 s=READY pgs=125 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=257a1b1cd12c157a server_cookie=97fa9d4dacf2c9fc in_seq=0 out_seq=0 2024-03-24T00:19:42.439 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:42.435+0000 7f80dcc00640 10 client.26564 ms_handle_connect on v2:172.21.15.189:6838/4074484168 2024-03-24T00:19:43.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:19:42 smithi118 ceph-mon[31535]: pgmap v518: 97 pgs: 97 active+clean; 97 MiB data, 631 MiB used, 1.0 TiB / 1.0 TiB avail; 481 KiB/s rd, 2.1 MiB/s wr, 302 op/s 2024-03-24T00:19:43.203 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:19:42 smithi189 ceph-mon[36187]: pgmap v518: 97 pgs: 97 active+clean; 97 MiB data, 631 MiB used, 1.0 TiB / 1.0 TiB avail; 481 KiB/s rd, 2.1 MiB/s wr, 302 op/s 2024-03-24T00:19:43.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:19:42 smithi193 ceph-mon[36878]: pgmap v518: 97 pgs: 97 active+clean; 97 MiB data, 631 MiB used, 1.0 TiB / 1.0 TiB avail; 481 KiB/s rd, 2.1 MiB/s wr, 302 op/s 2024-03-24T00:19:45.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:19:44 smithi118 ceph-mon[31535]: pgmap v519: 97 pgs: 97 active+clean; 97 MiB data, 630 MiB used, 1.0 TiB / 1.0 TiB avail; 267 KiB/s rd, 1.4 MiB/s wr, 270 op/s 2024-03-24T00:19:45.203 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:19:44 smithi189 ceph-mon[36187]: pgmap v519: 97 pgs: 97 active+clean; 97 MiB data, 630 MiB used, 1.0 TiB / 1.0 TiB avail; 267 KiB/s rd, 1.4 MiB/s wr, 270 op/s 2024-03-24T00:19:45.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:19:44 smithi193 ceph-mon[36878]: pgmap v519: 97 pgs: 97 active+clean; 97 MiB data, 630 MiB used, 1.0 TiB / 1.0 TiB avail; 267 KiB/s rd, 1.4 MiB/s wr, 270 op/s 2024-03-24T00:19:46.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:19:45 smithi118 ceph-mon[31535]: pgmap v520: 97 pgs: 97 active+clean; 97 MiB data, 614 MiB used, 1.0 TiB / 1.0 TiB avail; 173 KiB/s rd, 547 KiB/s wr, 210 op/s 2024-03-24T00:19:46.203 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:19:45 smithi189 ceph-mon[36187]: pgmap v520: 97 pgs: 97 active+clean; 97 MiB data, 614 MiB used, 1.0 TiB / 1.0 TiB avail; 173 KiB/s rd, 547 KiB/s wr, 210 op/s 2024-03-24T00:19:46.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:19:45 smithi193 ceph-mon[36878]: pgmap v520: 97 pgs: 97 active+clean; 97 MiB data, 614 MiB used, 1.0 TiB / 1.0 TiB avail; 173 KiB/s rd, 547 KiB/s wr, 210 op/s 2024-03-24T00:19:46.843 INFO:tasks.workunit.client.0.smithi118.stdout: 27 17494 0 13159 0 8665 0 2024-03-24T00:19:46.983 INFO:journalctl@ceph.mds.f.smithi193.stdout:Mar 24 00:19:46 smithi193 systemd[1]: ceph-d79a5420-e971-11ee-95cd-87774f69a715@mds.f.service: Scheduled restart job, restart counter is at 1. 2024-03-24T00:19:46.983 INFO:journalctl@ceph.mds.f.smithi193.stdout:Mar 24 00:19:46 smithi193 systemd[1]: Stopped Ceph mds.f for d79a5420-e971-11ee-95cd-87774f69a715. 2024-03-24T00:19:46.984 INFO:journalctl@ceph.mds.f.smithi193.stdout:Mar 24 00:19:46 smithi193 systemd[1]: ceph-d79a5420-e971-11ee-95cd-87774f69a715@mds.f.service: Consumed 2min 2.716s CPU time. 2024-03-24T00:19:46.984 INFO:journalctl@ceph.mds.f.smithi193.stdout:Mar 24 00:19:46 smithi193 systemd[1]: Starting Ceph mds.f for d79a5420-e971-11ee-95cd-87774f69a715... 2024-03-24T00:19:46.984 INFO:journalctl@ceph.mds.f.smithi193.stdout:Mar 24 00:19:46 smithi193 podman[70630]: 2024-03-24 00:19:46.85007367 +0000 UTC m=+0.017299065 image pull c664afe56bb01db4f9afa9532c20598e201e664f2e9ba2be50db0d52674a26d2 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc 2024-03-24T00:19:46.984 INFO:journalctl@ceph.mds.f.smithi193.stdout:Mar 24 00:19:46 smithi193 podman[70630]: 2024-03-24 00:19:46.984327662 +0000 UTC m=+0.151553049 container create 43046138110a3fada8001f090558acf465631fd5ade04440a53e6a4f9c03cad7 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-f, org.label-schema.vendor=CentOS, ceph=True, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240319, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_CLEAN=True) 2024-03-24T00:19:47.510 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:19:47 smithi193 ceph-mon[36878]: daemon mds.f restarted 2024-03-24T00:19:47.511 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:19:47 smithi193 ceph-mon[36878]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-03-24T00:19:47.511 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:19:47 smithi193 ceph-mon[36878]: osdmap e91: 12 total, 12 up, 12 in 2024-03-24T00:19:47.511 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:19:47 smithi193 ceph-mon[36878]: mds.? [v2:172.21.15.193:6834/3744499911,v1:172.21.15.193:6835/3744499911] up:boot 2024-03-24T00:19:47.511 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:19:47 smithi193 ceph-mon[36878]: Standby daemon mds.l assigned to filesystem cephfs as rank 3 2024-03-24T00:19:47.511 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:19:47 smithi193 ceph-mon[36878]: fsmap cephfs:4/5 {0=b=up:active,1=g=up:active,2=c=up:active,4=e=up:active} 5 up:standby-replay 3 up:standby, 1 failed 2024-03-24T00:19:47.511 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:19:47 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-03-24T00:19:47.511 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:19:47 smithi193 ceph-mon[36878]: fsmap cephfs:5/5 4 up:standby-replay 3 up:standby 1 up:replay 4 up:active 2024-03-24T00:19:47.511 INFO:journalctl@ceph.mds.f.smithi193.stdout:Mar 24 00:19:47 smithi193 podman[70630]: 2024-03-24 00:19:47.237118777 +0000 UTC m=+0.404344186 container init 43046138110a3fada8001f090558acf465631fd5ade04440a53e6a4f9c03cad7 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-f, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, GIT_CLEAN=True, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240319, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.vendor=CentOS) 2024-03-24T00:19:47.511 INFO:journalctl@ceph.mds.f.smithi193.stdout:Mar 24 00:19:47 smithi193 podman[70630]: 2024-03-24 00:19:47.240626476 +0000 UTC m=+0.407851871 container start 43046138110a3fada8001f090558acf465631fd5ade04440a53e6a4f9c03cad7 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-f, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, ceph=True, org.label-schema.vendor=CentOS, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.build-date=20240319, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.schema-version=1.0, GIT_CLEAN=True) 2024-03-24T00:19:47.512 INFO:journalctl@ceph.mds.f.smithi193.stdout:Mar 24 00:19:47 smithi193 ceph-mds[70648]: -- 172.21.15.193:0/4117572012 <== mon.1 v2:172.21.15.193:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x55d22bcefa40 con 0x55d22bcc3000 2024-03-24T00:19:47.512 INFO:journalctl@ceph.mds.f.smithi193.stdout:Mar 24 00:19:47 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-f[70644]: starting mds.f at 2024-03-24T00:19:47.512 INFO:journalctl@ceph.mds.f.smithi193.stdout:Mar 24 00:19:47 smithi193 bash[70630]: 43046138110a3fada8001f090558acf465631fd5ade04440a53e6a4f9c03cad7 2024-03-24T00:19:47.512 INFO:journalctl@ceph.mds.f.smithi193.stdout:Mar 24 00:19:47 smithi193 systemd[1]: Started Ceph mds.f for d79a5420-e971-11ee-95cd-87774f69a715. 2024-03-24T00:19:47.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:19:47 smithi118 ceph-mon[31535]: daemon mds.f restarted 2024-03-24T00:19:47.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:19:47 smithi118 ceph-mon[31535]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-03-24T00:19:47.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:19:47 smithi118 ceph-mon[31535]: osdmap e91: 12 total, 12 up, 12 in 2024-03-24T00:19:47.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:19:47 smithi118 ceph-mon[31535]: mds.? [v2:172.21.15.193:6834/3744499911,v1:172.21.15.193:6835/3744499911] up:boot 2024-03-24T00:19:47.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:19:47 smithi118 ceph-mon[31535]: Standby daemon mds.l assigned to filesystem cephfs as rank 3 2024-03-24T00:19:47.697 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:19:47 smithi118 ceph-mon[31535]: fsmap cephfs:4/5 {0=b=up:active,1=g=up:active,2=c=up:active,4=e=up:active} 5 up:standby-replay 3 up:standby, 1 failed 2024-03-24T00:19:47.697 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:19:47 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-03-24T00:19:47.697 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:19:47 smithi118 ceph-mon[31535]: fsmap cephfs:5/5 4 up:standby-replay 3 up:standby 1 up:replay 4 up:active 2024-03-24T00:19:47.704 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:19:47 smithi189 ceph-mon[36187]: daemon mds.f restarted 2024-03-24T00:19:47.704 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:19:47 smithi189 ceph-mon[36187]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-03-24T00:19:47.704 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:19:47 smithi189 ceph-mon[36187]: osdmap e91: 12 total, 12 up, 12 in 2024-03-24T00:19:47.704 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:19:47 smithi189 ceph-mon[36187]: mds.? [v2:172.21.15.193:6834/3744499911,v1:172.21.15.193:6835/3744499911] up:boot 2024-03-24T00:19:47.704 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:19:47 smithi189 ceph-mon[36187]: Standby daemon mds.l assigned to filesystem cephfs as rank 3 2024-03-24T00:19:47.704 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:19:47 smithi189 ceph-mon[36187]: fsmap cephfs:4/5 {0=b=up:active,1=g=up:active,2=c=up:active,4=e=up:active} 5 up:standby-replay 3 up:standby, 1 failed 2024-03-24T00:19:47.704 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:19:47 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-03-24T00:19:47.704 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:19:47 smithi189 ceph-mon[36187]: fsmap cephfs:5/5 4 up:standby-replay 3 up:standby 1 up:replay 4 up:active 2024-03-24T00:19:48.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:19:48 smithi118 ceph-mon[31535]: pgmap v522: 97 pgs: 97 active+clean; 97 MiB data, 614 MiB used, 1.0 TiB / 1.0 TiB avail; 10 KiB/s rd, 58 KiB/s wr, 88 op/s 2024-03-24T00:19:48.703 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:19:48 smithi189 ceph-mon[36187]: pgmap v522: 97 pgs: 97 active+clean; 97 MiB data, 614 MiB used, 1.0 TiB / 1.0 TiB avail; 10 KiB/s rd, 58 KiB/s wr, 88 op/s 2024-03-24T00:19:48.710 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:19:48 smithi193 ceph-mon[36878]: pgmap v522: 97 pgs: 97 active+clean; 97 MiB data, 614 MiB used, 1.0 TiB / 1.0 TiB avail; 10 KiB/s rd, 58 KiB/s wr, 88 op/s 2024-03-24T00:19:49.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:19:49 smithi118 ceph-mon[31535]: mds.? [v2:172.21.15.193:6832/1023428412,v1:172.21.15.193:6833/1023428412] up:resolve 2024-03-24T00:19:49.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:19:49 smithi118 ceph-mon[31535]: fsmap cephfs:5/5 4 up:standby-replay 3 up:standby 1 up:resolve 4 up:active 2024-03-24T00:19:49.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:19:49 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T00:19:49.703 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:19:49 smithi189 ceph-mon[36187]: mds.? [v2:172.21.15.193:6832/1023428412,v1:172.21.15.193:6833/1023428412] up:resolve 2024-03-24T00:19:49.704 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:19:49 smithi189 ceph-mon[36187]: fsmap cephfs:5/5 4 up:standby-replay 3 up:standby 1 up:resolve 4 up:active 2024-03-24T00:19:49.704 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:19:49 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T00:19:49.710 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:19:49 smithi193 ceph-mon[36878]: mds.? [v2:172.21.15.193:6832/1023428412,v1:172.21.15.193:6833/1023428412] up:resolve 2024-03-24T00:19:49.710 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:19:49 smithi193 ceph-mon[36878]: fsmap cephfs:5/5 4 up:standby-replay 3 up:standby 1 up:resolve 4 up:active 2024-03-24T00:19:49.710 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:19:49 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T00:19:50.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:19:50 smithi118 ceph-mon[31535]: mds.? [v2:172.21.15.193:6832/1023428412,v1:172.21.15.193:6833/1023428412] up:reconnect 2024-03-24T00:19:50.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:19:50 smithi118 ceph-mon[31535]: fsmap cephfs:5/5 4 up:standby-replay 3 up:standby 1 up:reconnect 4 up:active 2024-03-24T00:19:50.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:19:50 smithi118 ceph-mon[31535]: reconnect by client.15159 192.168.0.1:0/3463739313 after 0.0979984 2024-03-24T00:19:50.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:19:50 smithi118 ceph-mon[31535]: pgmap v523: 97 pgs: 97 active+clean; 97 MiB data, 614 MiB used, 1.0 TiB / 1.0 TiB avail; 10 KiB/s rd, 13 KiB/s wr, 25 op/s 2024-03-24T00:19:50.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:19:50 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T00:19:50.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:19:50 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T00:19:50.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:19:50 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T00:19:50.704 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:19:50 smithi189 ceph-mon[36187]: mds.? [v2:172.21.15.193:6832/1023428412,v1:172.21.15.193:6833/1023428412] up:reconnect 2024-03-24T00:19:50.704 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:19:50 smithi189 ceph-mon[36187]: fsmap cephfs:5/5 4 up:standby-replay 3 up:standby 1 up:reconnect 4 up:active 2024-03-24T00:19:50.704 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:19:50 smithi189 ceph-mon[36187]: reconnect by client.15159 192.168.0.1:0/3463739313 after 0.0979984 2024-03-24T00:19:50.704 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:19:50 smithi189 ceph-mon[36187]: pgmap v523: 97 pgs: 97 active+clean; 97 MiB data, 614 MiB used, 1.0 TiB / 1.0 TiB avail; 10 KiB/s rd, 13 KiB/s wr, 25 op/s 2024-03-24T00:19:50.704 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:19:50 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T00:19:50.704 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:19:50 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T00:19:50.704 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:19:50 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T00:19:50.710 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:19:50 smithi193 ceph-mon[36878]: mds.? [v2:172.21.15.193:6832/1023428412,v1:172.21.15.193:6833/1023428412] up:reconnect 2024-03-24T00:19:50.710 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:19:50 smithi193 ceph-mon[36878]: fsmap cephfs:5/5 4 up:standby-replay 3 up:standby 1 up:reconnect 4 up:active 2024-03-24T00:19:50.710 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:19:50 smithi193 ceph-mon[36878]: reconnect by client.15159 192.168.0.1:0/3463739313 after 0.0979984 2024-03-24T00:19:50.711 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:19:50 smithi193 ceph-mon[36878]: pgmap v523: 97 pgs: 97 active+clean; 97 MiB data, 614 MiB used, 1.0 TiB / 1.0 TiB avail; 10 KiB/s rd, 13 KiB/s wr, 25 op/s 2024-03-24T00:19:50.711 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:19:50 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T00:19:50.711 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:19:50 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T00:19:50.711 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:19:50 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T00:19:50.800 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:19:50.799+0000 7f0ac4c00640 1 -- 172.21.15.118:0/1649052402 <== mds.0 v2:172.21.15.189:6838/4074484168 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+44 (crc 0 0 0) 0x5645df2790c0 con 0x5645df37dfe0 2024-03-24T00:19:50.800 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:19:50.799+0000 7f0ac4c00640 10 client.16395 handle_command_reply: tid=0 2024-03-24T00:19:50.800 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:19:50.799+0000 7f0ae4d7b740 1 client.16395 shutdown 2024-03-24T00:19:50.800 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:50.799+0000 7f80dcc00640 1 -- 172.21.15.118:0/2335635957 <== mds.0 v2:172.21.15.189:6838/4074484168 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x55c05a461790 con 0x55c05a504b80 2024-03-24T00:19:50.800 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:50.799+0000 7f80dcc00640 10 client.26564 handle_command_reply: tid=0 2024-03-24T00:19:50.801 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:19:50.800+0000 7f0ae4d7b740 1 -- 172.21.15.118:0/1649052402 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5645df30cda0 msgr2=0x5645df31b460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:19:50.801 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:19:50.800+0000 7f0ae4d7b740 1 --2- 172.21.15.118:0/1649052402 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5645df30cda0 0x5645df31b460 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7f0ac809ad40 tx=0x7f0ac8009310 comp rx=0 tx=0).stop 2024-03-24T00:19:50.801 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:50.800+0000 7f80f87de740 1 client.26564 shutdown 2024-03-24T00:19:50.801 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:19:50.800+0000 7f0ae4d7b740 1 -- 172.21.15.118:0/1649052402 shutdown_connections 2024-03-24T00:19:50.802 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:19:50.800+0000 7f0ae4d7b740 1 --2- 172.21.15.118:0/1649052402 >> [v2:172.21.15.189:6838/4074484168,v1:172.21.15.189:6839/4074484168] conn(0x5645df37dfe0 0x5645df3803d0 crc :-1 s=READY pgs=118 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:19:50.802 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:19:50.800+0000 7f0ae4d7b740 1 --2- 172.21.15.118:0/1649052402 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5645df311f10 0x5645df31c470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:19:50.802 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:19:50.800+0000 7f0ae4d7b740 1 --2- 172.21.15.118:0/1649052402 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5645df30cda0 0x5645df31b460 unknown :-1 s=CLOSED pgs=209 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:19:50.802 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:19:50.800+0000 7f0ae4d7b740 1 --2- 172.21.15.118:0/1649052402 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5645df30c3b0 0x5645df31af20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:19:50.802 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:19:50.800+0000 7f0ac4c00640 0 client.16395 ms_handle_reset on v2:172.21.15.189:6838/4074484168 2024-03-24T00:19:50.802 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:19:50.800+0000 7f0ae4d7b740 1 -- 172.21.15.118:0/1649052402 >> 172.21.15.118:0/1649052402 conn(0x5645df307920 msgr2=0x5645df30e020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:19:50.802 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:19:50.800+0000 7f0ae4d7b740 1 -- 172.21.15.118:0/1649052402 shutdown_connections 2024-03-24T00:19:50.802 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:50.800+0000 7f80f87de740 1 -- 172.21.15.118:0/2335635957 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55c05a4f49e0 msgr2=0x55c05a500370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:19:50.802 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:50.800+0000 7f80f87de740 1 --2- 172.21.15.118:0/2335635957 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55c05a4f49e0 0x55c05a500370 secure :-1 s=READY pgs=396 cs=0 l=1 rev1=1 crypto rx=0x7f80e8005e30 tx=0x7f80e80bd310 comp rx=0 tx=0).stop 2024-03-24T00:19:50.803 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:19:50.800+0000 7f0ae4d7b740 1 -- 172.21.15.118:0/1649052402 wait complete. 2024-03-24T00:19:50.803 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:50.800+0000 7f80f87de740 1 -- 172.21.15.118:0/2335635957 shutdown_connections 2024-03-24T00:19:50.803 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:50.800+0000 7f80f87de740 1 --2- 172.21.15.118:0/2335635957 >> [v2:172.21.15.189:6838/4074484168,v1:172.21.15.189:6839/4074484168] conn(0x55c05a504b80 0x55c05a506f70 crc :-1 s=READY pgs=125 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:19:50.803 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:50.800+0000 7f80f87de740 1 --2- 172.21.15.118:0/2335635957 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55c05a4f49e0 0x55c05a500370 unknown :-1 s=CLOSED pgs=396 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:19:50.804 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:50.800+0000 7f80f87de740 1 --2- 172.21.15.118:0/2335635957 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55c05a4ef870 0x55c05a4f9330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:19:50.804 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:50.800+0000 7f80f87de740 1 --2- 172.21.15.118:0/2335635957 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55c05a4eee80 0x55c05a4f8df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:19:50.804 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:50.800+0000 7f80f87de740 1 -- 172.21.15.118:0/2335635957 >> 172.21.15.118:0/2335635957 conn(0x55c05a4b7200 msgr2=0x55c05a4e9a90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:19:50.804 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:50.800+0000 7f80dcc00640 0 client.26564 ms_handle_reset on v2:172.21.15.189:6838/4074484168 2024-03-24T00:19:50.805 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:19:50.801+0000 7f0ae4d7b740 20 client.16395 trim_cache size 0 max 16384 2024-03-24T00:19:50.806 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:50.801+0000 7f80f87de740 1 -- 172.21.15.118:0/2335635957 shutdown_connections 2024-03-24T00:19:50.806 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:50.801+0000 7f80f87de740 1 -- 172.21.15.118:0/2335635957 wait complete. 2024-03-24T00:19:50.806 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stdout:{ 2024-03-24T00:19:50.806 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stdout: "message": "", 2024-03-24T00:19:50.806 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stdout: "return_code": 0 2024-03-24T00:19:50.806 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stdout:} 2024-03-24T00:19:50.807 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:50.801+0000 7f80f87de740 20 client.26564 trim_cache size 0 max 16384 2024-03-24T00:19:50.807 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:19:50.802+0000 7f0ad1200640 1 -- 172.21.15.118:0/2340288462 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7f0ac00a28f0 msgr2=0x7f0ac00a4db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:19:50.807 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:19:50.802+0000 7f0ad1200640 1 --2- 172.21.15.118:0/2340288462 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7f0ac00a28f0 0x7f0ac00a4db0 secure :-1 s=READY pgs=341 cs=0 l=1 rev1=1 crypto rx=0x7f0acc005fd0 tx=0x7f0acc00a560 comp rx=0 tx=0).stop 2024-03-24T00:19:50.808 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:19:50.802+0000 7f0ad1200640 1 -- 172.21.15.118:0/2340288462 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0adc0732b0 msgr2=0x7f0adc1a6a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:19:50.808 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:19:50.802+0000 7f0ad1200640 1 --2- 172.21.15.118:0/2340288462 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0adc0732b0 0x7f0adc1a6a70 secure :-1 s=READY pgs=391 cs=0 l=1 rev1=1 crypto rx=0x7f0ac800e9a0 tx=0x7f0ac800ee60 comp rx=0 tx=0).stop 2024-03-24T00:19:50.808 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:19:50.802+0000 7f0ad1200640 1 -- 172.21.15.118:0/2340288462 shutdown_connections 2024-03-24T00:19:50.808 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:19:50.802+0000 7f0ad1200640 1 --2- 172.21.15.118:0/2340288462 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7f0ac00a28f0 0x7f0ac00a4db0 unknown :-1 s=CLOSED pgs=341 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:19:50.808 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:19:50.802+0000 7f0ad1200640 1 --2- 172.21.15.118:0/2340288462 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f0adc109390 0x7f0adc1adae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:19:50.808 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:19:50.802+0000 7f0ad1200640 1 --2- 172.21.15.118:0/2340288462 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0adc0732b0 0x7f0adc1a6a70 unknown :-1 s=CLOSED pgs=391 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:19:50.808 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:19:50.802+0000 7f0ad1200640 1 --2- 172.21.15.118:0/2340288462 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f0adc072950 0x7f0adc1a6530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:19:50.808 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:19:50.802+0000 7f0ad1200640 1 -- 172.21.15.118:0/2340288462 >> 172.21.15.118:0/2340288462 conn(0x7f0adc06e510 msgr2=0x7f0adc0700e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:19:50.808 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:19:50.802+0000 7f0ad1200640 1 -- 172.21.15.118:0/2340288462 shutdown_connections 2024-03-24T00:19:50.808 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:19:50.802+0000 7f0ad1200640 1 -- 172.21.15.118:0/2340288462 wait complete. 2024-03-24T00:19:50.832 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:50.831+0000 7f80f87de740 1 Processor -- start 2024-03-24T00:19:50.832 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:50.831+0000 7f80f87de740 1 -- start start 2024-03-24T00:19:50.832 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:50.831+0000 7f80f87de740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55c05a4eee80 0x55c05a4f0a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:19:50.832 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:50.831+0000 7f80f87de740 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55c05a4ef870 0x55c05a500330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:19:50.832 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:50.831+0000 7f80f87de740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55c05a4f49e0 0x55c05a4eda30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:19:50.832 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:50.831+0000 7f80f87de740 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f80f01aabe0 con 0x55c05a4ef870 2024-03-24T00:19:50.832 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:50.831+0000 7f80f87de740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f80f01ac590 con 0x55c05a4eee80 2024-03-24T00:19:50.832 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:50.831+0000 7f80f87de740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f80f01ab240 con 0x55c05a4f49e0 2024-03-24T00:19:50.833 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:50.832+0000 7f80eea00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55c05a4eee80 0x55c05a4f0a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:19:50.834 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:50.832+0000 7f80eea00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55c05a4eee80 0x55c05a4f0a20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.118:43524/0 (socket says 172.21.15.118:43524) 2024-03-24T00:19:50.834 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:50.832+0000 7f80eea00640 1 -- 172.21.15.118:0/517146837 learned_addr learned my addr 172.21.15.118:0/517146837 (peer_addr_for_me v2:172.21.15.118:0/0) 2024-03-24T00:19:50.834 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:50.832+0000 7f80ef400640 1 --2- 172.21.15.118:0/517146837 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55c05a4f49e0 0x55c05a4eda30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:19:50.834 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:50.832+0000 7f80e6000640 1 --2- 172.21.15.118:0/517146837 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55c05a4ef870 0x55c05a500330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:19:50.834 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:50.832+0000 7f80eea00640 1 -- 172.21.15.118:0/517146837 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55c05a4f49e0 msgr2=0x55c05a4eda30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:19:50.834 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:50.832+0000 7f80eea00640 1 --2- 172.21.15.118:0/517146837 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55c05a4f49e0 0x55c05a4eda30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:19:50.834 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:50.832+0000 7f80eea00640 1 -- 172.21.15.118:0/517146837 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55c05a4ef870 msgr2=0x55c05a500330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:19:50.834 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:50.832+0000 7f80eea00640 1 --2- 172.21.15.118:0/517146837 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55c05a4ef870 0x55c05a500330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:19:50.834 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:50.832+0000 7f80eea00640 1 -- 172.21.15.118:0/517146837 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55c05a1280f0 con 0x55c05a4eee80 2024-03-24T00:19:50.834 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:50.832+0000 7f80ef400640 1 --2- 172.21.15.118:0/517146837 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55c05a4f49e0 0x55c05a4eda30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T00:19:50.834 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:50.832+0000 7f80eea00640 1 --2- 172.21.15.118:0/517146837 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55c05a4eee80 0x55c05a4f0a20 secure :-1 s=READY pgs=398 cs=0 l=1 rev1=1 crypto rx=0x7f80e000be50 tx=0x7f80e0013000 comp rx=0 tx=0).ready entity=mon.2 client_cookie=7def8a5afb0c58ba server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:19:50.834 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:50.833+0000 7f80df400640 1 -- 172.21.15.118:0/517146837 <== mon.2 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f80e002adf0 con 0x55c05a4eee80 2024-03-24T00:19:50.835 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:50.833+0000 7f80df400640 1 -- 172.21.15.118:0/517146837 <== mon.2 v2:172.21.15.189:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f80e00035f0 con 0x55c05a4eee80 2024-03-24T00:19:50.835 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:50.833+0000 7f80df400640 1 -- 172.21.15.118:0/517146837 <== mon.2 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f80e001f690 con 0x55c05a4eee80 2024-03-24T00:19:50.837 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:50.833+0000 7f80f87de740 1 -- 172.21.15.118:0/517146837 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55c05a4eee80 msgr2=0x55c05a4f0a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:19:50.837 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:50.833+0000 7f80f87de740 1 --2- 172.21.15.118:0/517146837 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55c05a4eee80 0x55c05a4f0a20 secure :-1 s=READY pgs=398 cs=0 l=1 rev1=1 crypto rx=0x7f80e000be50 tx=0x7f80e0013000 comp rx=0 tx=0).stop 2024-03-24T00:19:50.837 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:50.834+0000 7f80f87de740 1 -- 172.21.15.118:0/517146837 shutdown_connections 2024-03-24T00:19:50.837 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:50.834+0000 7f80f87de740 1 --2- 172.21.15.118:0/517146837 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55c05a4f49e0 0x55c05a4eda30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:19:50.837 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:50.834+0000 7f80f87de740 1 --2- 172.21.15.118:0/517146837 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55c05a4ef870 0x55c05a500330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:19:50.838 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:50.834+0000 7f80f87de740 1 --2- 172.21.15.118:0/517146837 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55c05a4eee80 0x55c05a4f0a20 unknown :-1 s=CLOSED pgs=398 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:19:50.838 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:50.834+0000 7f80f87de740 1 -- 172.21.15.118:0/517146837 >> 172.21.15.118:0/517146837 conn(0x55c05a4b7200 msgr2=0x55c05a4f0310 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:19:50.838 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:50.834+0000 7f80f87de740 1 -- 172.21.15.118:0/517146837 shutdown_connections 2024-03-24T00:19:50.838 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:50.834+0000 7f80f87de740 1 -- 172.21.15.118:0/517146837 wait complete. 2024-03-24T00:19:50.838 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:50.834+0000 7f80f87de740 1 -- start start 2024-03-24T00:19:50.838 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:50.834+0000 7f80f87de740 1 Processor -- start 2024-03-24T00:19:50.838 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:50.835+0000 7f80f87de740 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55c05a4eee80 0x55c05a4fdbf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:19:50.838 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:50.835+0000 7f80f87de740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55c05a4ef870 0x55c05a4fe130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:19:50.838 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:50.835+0000 7f80f87de740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55c05a4f49e0 0x55c05a4ff140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:19:50.838 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:50.835+0000 7f80f87de740 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f80f01abf00 con 0x55c05a4eee80 2024-03-24T00:19:50.838 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:50.835+0000 7f80f87de740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f80f01ab8a0 con 0x55c05a4f49e0 2024-03-24T00:19:50.838 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:50.835+0000 7f80f87de740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f80f01a8da0 con 0x55c05a4ef870 2024-03-24T00:19:50.838 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:50.835+0000 7f80f87de740 10 client.0 _ll_register_callbacks cb 0x7f80e80b9c20 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T00:19:50.838 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:50.835+0000 7f80e6000640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55c05a4ef870 0x55c05a4fe130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:19:50.839 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:50.835+0000 7f80eea00640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55c05a4eee80 0x55c05a4fdbf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:19:50.839 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:50.835+0000 7f80e6000640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55c05a4ef870 0x55c05a4fe130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.118:37198/0 (socket says 172.21.15.118:37198) 2024-03-24T00:19:50.839 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:50.835+0000 7f80e6000640 1 -- 172.21.15.118:0/3517410735 learned_addr learned my addr 172.21.15.118:0/3517410735 (peer_addr_for_me v2:172.21.15.118:0/0) 2024-03-24T00:19:50.839 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:50.835+0000 7f80ef400640 1 --2- 172.21.15.118:0/3517410735 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55c05a4f49e0 0x55c05a4ff140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:19:50.839 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:50.836+0000 7f80e6000640 1 -- 172.21.15.118:0/3517410735 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55c05a4f49e0 msgr2=0x55c05a4ff140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:19:50.839 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:50.836+0000 7f80e6000640 1 --2- 172.21.15.118:0/3517410735 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55c05a4f49e0 0x55c05a4ff140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:19:50.839 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:50.836+0000 7f80e6000640 1 -- 172.21.15.118:0/3517410735 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55c05a4eee80 msgr2=0x55c05a4fdbf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:19:50.839 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:50.836+0000 7f80e6000640 1 --2- 172.21.15.118:0/3517410735 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55c05a4eee80 0x55c05a4fdbf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:19:50.839 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:50.836+0000 7f80e6000640 1 -- 172.21.15.118:0/3517410735 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55c05a4ff730 con 0x55c05a4ef870 2024-03-24T00:19:50.839 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:50.836+0000 7f80ef400640 1 --2- 172.21.15.118:0/3517410735 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55c05a4f49e0 0x55c05a4ff140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T00:19:50.839 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:50.836+0000 7f80e6000640 1 --2- 172.21.15.118:0/3517410735 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55c05a4ef870 0x55c05a4fe130 secure :-1 s=READY pgs=394 cs=0 l=1 rev1=1 crypto rx=0x7f80d80097e0 tx=0x7f80d8009c00 comp rx=0 tx=0).ready entity=mon.1 client_cookie=36c6bd60885bdae1 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:19:50.839 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:50.836+0000 7f80dcc00640 10 client.0 ms_handle_connect on v2:172.21.15.193:3300/0 2024-03-24T00:19:50.839 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:50.836+0000 7f80dcc00640 1 -- 172.21.15.118:0/3517410735 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f80d801d040 con 0x55c05a4ef870 2024-03-24T00:19:50.839 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:50.836+0000 7f80f87de740 1 -- 172.21.15.118:0/3517410735 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f80e00035f0 con 0x55c05a4ef870 2024-03-24T00:19:50.840 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:50.836+0000 7f80dcc00640 1 -- 172.21.15.118:0/3517410735 <== mon.1 v2:172.21.15.193:3300/0 2 ==== osd_map(91..91 src has 1..91) v4 ==== 8363+0+0 (secure 0 0 0) 0x7f80d8005310 con 0x55c05a4ef870 2024-03-24T00:19:50.840 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:50.837+0000 7f80dcc00640 1 -- 172.21.15.118:0/3517410735 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_get_version_reply(handle=1 version=34) v2 ==== 24+0+0 (secure 0 0 0) 0x7f80d8031020 con 0x55c05a4ef870 2024-03-24T00:19:50.840 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:50.837+0000 7f80f87de740 10 client.26590 fetch_fsmap learned FSMap version 34 2024-03-24T00:19:50.840 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:50.837+0000 7f80f87de740 1 -- 172.21.15.118:0/3517410735 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,fsmap=34,monmap=4+}) v3 -- 0x55c05a5606b0 con 0x55c05a4ef870 2024-03-24T00:19:50.840 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:50.837+0000 7f80dcc00640 1 -- 172.21.15.118:0/3517410735 <== mon.1 v2:172.21.15.193:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f80e00035f0 con 0x55c05a4ef870 2024-03-24T00:19:50.841 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:50.837+0000 7f80dcc00640 1 -- 172.21.15.118:0/3517410735 <== mon.1 v2:172.21.15.193:3300/0 5 ==== fsmap(e 34) v1 ==== 7942+0+0 (secure 0 0 0) 0x7f80d800f4e0 con 0x55c05a4ef870 2024-03-24T00:19:50.841 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:50.837+0000 7f80f87de740 10 client.26590 fetch_fsmap finished waiting for FSMap version 34 2024-03-24T00:19:50.842 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:50.837+0000 7f80f87de740 10 client.26590 resolve_mds: resolved 1:0 to role '1:0' aka daemon mds.b 2024-03-24T00:19:50.842 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:50.837+0000 7f80f87de740 20 client.26590 populate_metadata read hostname 'smithi118' 2024-03-24T00:19:50.842 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:50.837+0000 7f80f87de740 1 --2- 172.21.15.118:0/3517410735 >> [v2:172.21.15.189:6838/4074484168,v1:172.21.15.189:6839/4074484168] conn(0x55c05a560e40 0x55c05a563260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:19:50.842 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:50.837+0000 7f80f87de740 4 client.26590 mds_command: new command op to 24569 tid=0 multi_id=0 [{"prefix": "scrub status"}] 2024-03-24T00:19:50.842 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:50.837+0000 7f80f87de740 1 -- 172.21.15.118:0/3517410735 --> [v2:172.21.15.189:6838/4074484168,v1:172.21.15.189:6839/4074484168] -- command(tid 0: {"prefix": "scrub status"}) v1 -- 0x55c05a45bae0 con 0x55c05a560e40 2024-03-24T00:19:50.842 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:50.837+0000 7f80eea00640 1 --2- 172.21.15.118:0/3517410735 >> [v2:172.21.15.189:6838/4074484168,v1:172.21.15.189:6839/4074484168] conn(0x55c05a560e40 0x55c05a563260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:19:50.842 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:50.838+0000 7f80eea00640 1 --2- 172.21.15.118:0/3517410735 >> [v2:172.21.15.189:6838/4074484168,v1:172.21.15.189:6839/4074484168] conn(0x55c05a560e40 0x55c05a563260 crc :-1 s=READY pgs=128 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=32bbdcca4c4a2960 server_cookie=dda0e8ad67977e7a in_seq=0 out_seq=0 2024-03-24T00:19:50.842 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:50.838+0000 7f80dcc00640 10 client.26590 ms_handle_connect on v2:172.21.15.189:6838/4074484168 2024-03-24T00:19:50.842 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:50.838+0000 7f80dcc00640 1 -- 172.21.15.118:0/3517410735 <== mds.0 v2:172.21.15.189:6838/4074484168 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+63 (crc 0 0 0) 0x55c05a45bae0 con 0x55c05a560e40 2024-03-24T00:19:50.842 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:50.838+0000 7f80dcc00640 10 client.26590 handle_command_reply: tid=0 2024-03-24T00:19:50.842 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:50.838+0000 7f80f87de740 1 client.26590 shutdown 2024-03-24T00:19:50.842 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:50.838+0000 7f80f87de740 1 -- 172.21.15.118:0/3517410735 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55c05a4ef870 msgr2=0x55c05a4fe130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:19:50.842 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:50.838+0000 7f80f87de740 1 --2- 172.21.15.118:0/3517410735 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55c05a4ef870 0x55c05a4fe130 secure :-1 s=READY pgs=394 cs=0 l=1 rev1=1 crypto rx=0x7f80d80097e0 tx=0x7f80d8009c00 comp rx=0 tx=0).stop 2024-03-24T00:19:50.842 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:50.839+0000 7f80f87de740 1 -- 172.21.15.118:0/3517410735 shutdown_connections 2024-03-24T00:19:50.843 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:50.839+0000 7f80f87de740 1 --2- 172.21.15.118:0/3517410735 >> [v2:172.21.15.189:6838/4074484168,v1:172.21.15.189:6839/4074484168] conn(0x55c05a560e40 0x55c05a563260 crc :-1 s=READY pgs=128 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:19:50.843 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:50.839+0000 7f80f87de740 1 --2- 172.21.15.118:0/3517410735 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55c05a4f49e0 0x55c05a4ff140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:19:50.843 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:50.839+0000 7f80f87de740 1 --2- 172.21.15.118:0/3517410735 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55c05a4ef870 0x55c05a4fe130 unknown :-1 s=CLOSED pgs=394 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:19:50.843 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:50.839+0000 7f80f87de740 1 --2- 172.21.15.118:0/3517410735 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55c05a4eee80 0x55c05a4fdbf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:19:50.843 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:50.839+0000 7f80dcc00640 0 client.26590 ms_handle_reset on v2:172.21.15.189:6838/4074484168 2024-03-24T00:19:50.843 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:50.839+0000 7f80f87de740 1 -- 172.21.15.118:0/3517410735 >> 172.21.15.118:0/3517410735 conn(0x55c05a4b7200 msgr2=0x55c05a4e9200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:19:50.843 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:50.839+0000 7f80f87de740 1 -- 172.21.15.118:0/3517410735 shutdown_connections 2024-03-24T00:19:50.843 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:50.839+0000 7f80f87de740 1 -- 172.21.15.118:0/3517410735 wait complete. 2024-03-24T00:19:50.843 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:50.839+0000 7f80f87de740 20 client.26590 trim_cache size 0 max 16384 2024-03-24T00:19:50.843 INFO:teuthology.orchestra.run.smithi118.stdout:{ 2024-03-24T00:19:50.844 INFO:teuthology.orchestra.run.smithi118.stdout: "status": "no active scrubs running", 2024-03-24T00:19:50.844 INFO:teuthology.orchestra.run.smithi118.stdout: "scrubs": {} 2024-03-24T00:19:50.844 INFO:teuthology.orchestra.run.smithi118.stdout:} 2024-03-24T00:19:50.844 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:50.840+0000 7f80f5a00640 1 -- 172.21.15.118:0/2325826204 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7f80d40a2880 msgr2=0x7f80d40a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:19:50.844 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:50.840+0000 7f80f5a00640 1 --2- 172.21.15.118:0/2325826204 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7f80d40a2880 0x7f80d40a4d40 secure :-1 s=READY pgs=342 cs=0 l=1 rev1=1 crypto rx=0x7f80e00104e0 tx=0x7f80e000b410 comp rx=0 tx=0).stop 2024-03-24T00:19:50.844 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:50.840+0000 7f80f5a00640 1 -- 172.21.15.118:0/2325826204 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f80f0103260 msgr2=0x7f80f0073fd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:19:50.845 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:50.840+0000 7f80f5a00640 1 --2- 172.21.15.118:0/2325826204 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f80f0103260 0x7f80f0073fd0 secure :-1 s=READY pgs=394 cs=0 l=1 rev1=1 crypto rx=0x7f80e8037040 tx=0x7f80e8035d10 comp rx=0 tx=0).stop 2024-03-24T00:19:50.845 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:50.840+0000 7f80f5a00640 1 -- 172.21.15.118:0/2325826204 shutdown_connections 2024-03-24T00:19:50.845 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:50.840+0000 7f80f5a00640 1 --2- 172.21.15.118:0/2325826204 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7f80d40a2880 0x7f80d40a4d40 unknown :-1 s=CLOSED pgs=342 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:19:50.845 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:50.840+0000 7f80f5a00640 1 --2- 172.21.15.118:0/2325826204 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f80f0103260 0x7f80f0073fd0 unknown :-1 s=CLOSED pgs=394 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:19:50.845 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:50.840+0000 7f80f5a00640 1 --2- 172.21.15.118:0/2325826204 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f80f01028a0 0x7f80f0073a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:19:50.845 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:50.840+0000 7f80f5a00640 1 --2- 172.21.15.118:0/2325826204 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f80f01016b0 0x7f80f0075450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:19:50.845 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:50.840+0000 7f80f5a00640 1 -- 172.21.15.118:0/2325826204 >> 172.21.15.118:0/2325826204 conn(0x7f80f00fcc40 msgr2=0x7f80f0105ba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:19:50.845 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:50.840+0000 7f80f5a00640 1 -- 172.21.15.118:0/2325826204 shutdown_connections 2024-03-24T00:19:50.845 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:50.840+0000 7f80f5a00640 1 -- 172.21.15.118:0/2325826204 wait complete. 2024-03-24T00:19:50.861 INFO:tasks.cephfs.filesystem:all active scrubs completed 2024-03-24T00:19:50.861 DEBUG:teuthology.orchestra.run.smithi118:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-24T00:19:51.066 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:51.065+0000 7fe12e800640 1 -- 172.21.15.118:0/2328845453 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fe1280fff50 msgr2=0x7fe128102380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:19:51.066 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:51.065+0000 7fe12e800640 1 --2- 172.21.15.118:0/2328845453 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fe1280fff50 0x7fe128102380 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7fe118009860 tx=0x7fe11802bb70 comp rx=0 tx=0).stop 2024-03-24T00:19:51.066 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:51.065+0000 7fe12e800640 1 -- 172.21.15.118:0/2328845453 shutdown_connections 2024-03-24T00:19:51.066 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:51.065+0000 7fe12e800640 1 --2- 172.21.15.118:0/2328845453 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe128105270 0x7fe1281056f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:19:51.067 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:51.065+0000 7fe12e800640 1 --2- 172.21.15.118:0/2328845453 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fe1281028c0 0x7fe128104d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:19:51.067 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:51.065+0000 7fe12e800640 1 --2- 172.21.15.118:0/2328845453 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fe1280fff50 0x7fe128102380 unknown :-1 s=CLOSED pgs=210 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:19:51.067 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:51.065+0000 7fe12e800640 1 -- 172.21.15.118:0/2328845453 >> 172.21.15.118:0/2328845453 conn(0x7fe1280fbb50 msgr2=0x7fe1280fdf90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:19:51.067 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:51.065+0000 7fe12e800640 1 -- 172.21.15.118:0/2328845453 shutdown_connections 2024-03-24T00:19:51.067 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:51.065+0000 7fe12e800640 1 -- 172.21.15.118:0/2328845453 wait complete. 2024-03-24T00:19:51.069 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:51.066+0000 7fe12e800640 1 Processor -- start 2024-03-24T00:19:51.069 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:51.066+0000 7fe12e800640 1 -- start start 2024-03-24T00:19:51.069 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:51.067+0000 7fe12e800640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fe1280fff50 0x7fe12819e000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:19:51.069 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:51.067+0000 7fe12e800640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fe1281028c0 0x7fe12819e540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:19:51.069 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:51.067+0000 7fe12e800640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe128105270 0x7fe1281a5560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:19:51.069 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:51.067+0000 7fe12e800640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe128113930 con 0x7fe1280fff50 2024-03-24T00:19:51.069 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:51.067+0000 7fe12e800640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe1281137b0 con 0x7fe128105270 2024-03-24T00:19:51.069 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:51.067+0000 7fe12e800640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe128113ab0 con 0x7fe1281028c0 2024-03-24T00:19:51.069 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:51.067+0000 7fe127400640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fe1280fff50 0x7fe12819e000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:19:51.069 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:51.067+0000 7fe11ea00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fe1281028c0 0x7fe12819e540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:19:51.069 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:51.067+0000 7fe127400640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fe1280fff50 0x7fe12819e000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.118:3300/0 says I am v2:172.21.15.118:60838/0 (socket says 172.21.15.118:60838) 2024-03-24T00:19:51.069 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:51.067+0000 7fe11ea00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fe1281028c0 0x7fe12819e540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.118:37228/0 (socket says 172.21.15.118:37228) 2024-03-24T00:19:51.070 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:51.067+0000 7fe127400640 1 -- 172.21.15.118:0/4071544082 learned_addr learned my addr 172.21.15.118:0/4071544082 (peer_addr_for_me v2:172.21.15.118:0/0) 2024-03-24T00:19:51.070 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:51.067+0000 7fe127e00640 1 --2- 172.21.15.118:0/4071544082 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe128105270 0x7fe1281a5560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:19:51.070 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:51.067+0000 7fe11ea00640 1 -- 172.21.15.118:0/4071544082 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe128105270 msgr2=0x7fe1281a5560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:19:51.070 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:51.067+0000 7fe11ea00640 1 --2- 172.21.15.118:0/4071544082 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe128105270 0x7fe1281a5560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:19:51.070 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:51.067+0000 7fe11ea00640 1 -- 172.21.15.118:0/4071544082 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fe1280fff50 msgr2=0x7fe12819e000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:19:51.070 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:51.067+0000 7fe11ea00640 1 --2- 172.21.15.118:0/4071544082 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fe1280fff50 0x7fe12819e000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:19:51.070 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:51.067+0000 7fe11ea00640 1 -- 172.21.15.118:0/4071544082 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe1281a5bd0 con 0x7fe1281028c0 2024-03-24T00:19:51.070 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:51.067+0000 7fe127400640 1 --2- 172.21.15.118:0/4071544082 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fe1280fff50 0x7fe12819e000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T00:19:51.070 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:51.068+0000 7fe11ea00640 1 --2- 172.21.15.118:0/4071544082 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fe1281028c0 0x7fe12819e540 secure :-1 s=READY pgs=395 cs=0 l=1 rev1=1 crypto rx=0x7fe11000e790 tx=0x7fe11000ec50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:19:51.070 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:51.068+0000 7fe124c00640 1 -- 172.21.15.118:0/4071544082 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe11000eea0 con 0x7fe1281028c0 2024-03-24T00:19:51.070 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:51.068+0000 7fe124c00640 1 -- 172.21.15.118:0/4071544082 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fe11000ccc0 con 0x7fe1281028c0 2024-03-24T00:19:51.070 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:51.068+0000 7fe12e800640 1 -- 172.21.15.118:0/4071544082 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe1281a5ec0 con 0x7fe1281028c0 2024-03-24T00:19:51.070 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:51.068+0000 7fe124c00640 1 -- 172.21.15.118:0/4071544082 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fe110018760 con 0x7fe1281028c0 2024-03-24T00:19:51.070 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:51.068+0000 7fe12e800640 1 -- 172.21.15.118:0/4071544082 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe1281a6380 con 0x7fe1281028c0 2024-03-24T00:19:51.071 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:51.070+0000 7fe12e800640 1 -- 172.21.15.118:0/4071544082 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe1280fd640 con 0x7fe1281028c0 2024-03-24T00:19:51.072 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:51.071+0000 7fe124c00640 1 -- 172.21.15.118:0/4071544082 <== mon.1 v2:172.21.15.193:3300/0 4 ==== mgrmap(e 20) v1 ==== 137291+0+0 (secure 0 0 0) 0x7fe110010430 con 0x7fe1281028c0 2024-03-24T00:19:51.073 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:51.072+0000 7fe124c00640 1 --2- 172.21.15.118:0/4071544082 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7fe10c0a2800 0x7fe10c0a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:19:51.073 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:51.073+0000 7fe124c00640 1 -- 172.21.15.118:0/4071544082 <== mon.1 v2:172.21.15.193:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 8363+0+0 (secure 0 0 0) 0x7fe1100ceca0 con 0x7fe1281028c0 2024-03-24T00:19:51.073 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:51.073+0000 7fe127400640 1 --2- 172.21.15.118:0/4071544082 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7fe10c0a2800 0x7fe10c0a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:19:51.074 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:51.073+0000 7fe127400640 1 --2- 172.21.15.118:0/4071544082 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7fe10c0a2800 0x7fe10c0a4cc0 secure :-1 s=READY pgs=344 cs=0 l=1 rev1=1 crypto rx=0x7fe118035fd0 tx=0x7fe11802e040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:19:51.081 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:51.082+0000 7fe124c00640 1 -- 172.21.15.118:0/4071544082 <== mon.1 v2:172.21.15.193:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7fe11009b610 con 0x7fe1281028c0 2024-03-24T00:19:51.347 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:51.346+0000 7fe12e800640 1 -- 172.21.15.118:0/4071544082 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fe12802d070 con 0x7fe1281028c0 2024-03-24T00:19:51.348 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:51.347+0000 7fe124c00640 1 -- 172.21.15.118:0/4071544082 <== mon.1 v2:172.21.15.193:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 34 v34) v1 ==== 94+0+11469 (secure 0 0 0) 0x7fe11009f6b0 con 0x7fe1281028c0 2024-03-24T00:19:51.348 INFO:teuthology.orchestra.run.smithi118.stdout: 2024-03-24T00:19:51.348 INFO:teuthology.orchestra.run.smithi118.stdout:{"epoch":34,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":14385,"name":"j","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.118:6835/3194741373","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6834","nonce":3194741373},{"type":"v1","addr":"172.21.15.118:6835","nonce":3194741373}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":4},{"gid":14397,"name":"d","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.118:6837/1374462956","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6836","nonce":1374462956},{"type":"v1","addr":"172.21.15.118:6837","nonce":1374462956}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":5},{"gid":26570,"name":"f","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.193:6835/3744499911","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6834","nonce":3744499911},{"type":"v1","addr":"172.21.15.193:6835","nonce":3744499911}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"epoch":30}],"filesystems":[{"mdsmap":{"epoch":34,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-03-24T00:13:01.782474+0000","modified":"2024-03-24T00:19:49.449513+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":91,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24569,"mds_1":24550,"mds_2":24511,"mds_3":24439,"mds_4":24488},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14469":{"gid":14469,"name":"a","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.118:6841/3040946537","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6840","nonce":3040946537},{"type":"v1","addr":"172.21.15.118:6841","nonce":3040946537}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24439":{"gid":24439,"name":"l","rank":3,"incarnation":31,"state":"up:rejoin","state_seq":116,"addr":"172.21.15.193:6833/1023428412","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6832","nonce":1023428412},{"type":"v1","addr":"172.21.15.193:6833","nonce":1023428412}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24440":{"gid":24440,"name":"k","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.189:6833/2740024078","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6832","nonce":2740024078},{"type":"v1","addr":"172.21.15.189:6833","nonce":2740024078}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24488":{"gid":24488,"name":"e","rank":4,"incarnation":25,"state":"up:active","state_seq":10,"addr":"172.21.15.189:6835/599473205","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6834","nonce":599473205},{"type":"v1","addr":"172.21.15.189:6835","nonce":599473205}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24511":{"gid":24511,"name":"c","rank":2,"incarnation":21,"state":"up:active","state_seq":8,"addr":"172.21.15.193:6837/2921520053","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6836","nonce":2921520053},{"type":"v1","addr":"172.21.15.193:6837","nonce":2921520053}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24515":{"gid":24515,"name":"h","rank":1,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.189:6837/3450187137","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6836","nonce":3450187137},{"type":"v1","addr":"172.21.15.189:6837","nonce":3450187137}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24532":{"gid":24532,"name":"i","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.193:6839/2126842603","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6838","nonce":2126842603},{"type":"v1","addr":"172.21.15.193:6839","nonce":2126842603}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24550":{"gid":24550,"name":"g","rank":1,"incarnation":19,"state":"up:active","state_seq":6,"addr":"172.21.15.118:6839/3285788503","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6838","nonce":3285788503},{"type":"v1","addr":"172.21.15.118:6839","nonce":3285788503}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24569":{"gid":24569,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":3,"addr":"172.21.15.189:6839/4074484168","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6838","nonce":4074484168},{"type":"v1","addr":"172.21.15.189:6839","nonce":4074484168}]},"join_fscid":-1,"export_targets":[3],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24569,"qdb_cluster":[24511,24488,24569,24550]},"id":1}]} 2024-03-24T00:19:51.349 INFO:teuthology.orchestra.run.smithi118.stderr:dumped fsmap epoch 34 2024-03-24T00:19:51.351 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:51.350+0000 7fe12e800640 1 -- 172.21.15.118:0/4071544082 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7fe10c0a2800 msgr2=0x7fe10c0a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:19:51.351 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:51.350+0000 7fe12e800640 1 --2- 172.21.15.118:0/4071544082 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7fe10c0a2800 0x7fe10c0a4cc0 secure :-1 s=READY pgs=344 cs=0 l=1 rev1=1 crypto rx=0x7fe118035fd0 tx=0x7fe11802e040 comp rx=0 tx=0).stop 2024-03-24T00:19:51.351 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:51.350+0000 7fe12e800640 1 -- 172.21.15.118:0/4071544082 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fe1281028c0 msgr2=0x7fe12819e540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:19:51.351 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:51.350+0000 7fe12e800640 1 --2- 172.21.15.118:0/4071544082 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fe1281028c0 0x7fe12819e540 secure :-1 s=READY pgs=395 cs=0 l=1 rev1=1 crypto rx=0x7fe11000e790 tx=0x7fe11000ec50 comp rx=0 tx=0).stop 2024-03-24T00:19:51.351 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:51.350+0000 7fe12e800640 1 -- 172.21.15.118:0/4071544082 shutdown_connections 2024-03-24T00:19:51.351 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:51.350+0000 7fe12e800640 1 --2- 172.21.15.118:0/4071544082 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7fe10c0a2800 0x7fe10c0a4cc0 unknown :-1 s=CLOSED pgs=344 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:19:51.351 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:51.350+0000 7fe12e800640 1 --2- 172.21.15.118:0/4071544082 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fe128105270 0x7fe1281a5560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:19:51.351 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:51.350+0000 7fe12e800640 1 --2- 172.21.15.118:0/4071544082 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fe1281028c0 0x7fe12819e540 unknown :-1 s=CLOSED pgs=395 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:19:51.351 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:51.350+0000 7fe12e800640 1 --2- 172.21.15.118:0/4071544082 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fe1280fff50 0x7fe12819e000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:19:51.351 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:51.350+0000 7fe12e800640 1 -- 172.21.15.118:0/4071544082 >> 172.21.15.118:0/4071544082 conn(0x7fe1280fbb50 msgr2=0x7fe128103270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:19:51.351 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:51.350+0000 7fe12e800640 1 -- 172.21.15.118:0/4071544082 shutdown_connections 2024-03-24T00:19:51.351 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:51.350+0000 7fe12e800640 1 -- 172.21.15.118:0/4071544082 wait complete. 2024-03-24T00:19:51.370 INFO:teuthology.orchestra.run:Running command with timeout 120 2024-03-24T00:19:51.371 DEBUG:teuthology.orchestra.run.smithi118:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.1:3 damage ls 2024-03-24T00:19:51.578 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:51.577+0000 7fd634a00640 1 -- 172.21.15.118:0/1895236957 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd6300756e0 msgr2=0x7fd630073b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:19:51.578 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:51.577+0000 7fd634a00640 1 --2- 172.21.15.118:0/1895236957 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd6300756e0 0x7fd630073b30 secure :-1 s=READY pgs=399 cs=0 l=1 rev1=1 crypto rx=0x7fd620009900 tx=0x7fd62002bc40 comp rx=0 tx=0).stop 2024-03-24T00:19:51.578 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:51.577+0000 7fd634a00640 1 -- 172.21.15.118:0/1895236957 shutdown_connections 2024-03-24T00:19:51.578 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:51.577+0000 7fd634a00640 1 --2- 172.21.15.118:0/1895236957 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fd630109460 0x7fd63010b8f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:19:51.578 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:51.577+0000 7fd634a00640 1 --2- 172.21.15.118:0/1895236957 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fd630074100 0x7fd630074580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:19:51.578 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:51.577+0000 7fd634a00640 1 --2- 172.21.15.118:0/1895236957 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd6300756e0 0x7fd630073b30 unknown :-1 s=CLOSED pgs=399 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:19:51.578 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:51.577+0000 7fd634a00640 1 -- 172.21.15.118:0/1895236957 >> 172.21.15.118:0/1895236957 conn(0x7fd6300fcb10 msgr2=0x7fd6300fef50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:19:51.578 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:51.577+0000 7fd634a00640 1 -- 172.21.15.118:0/1895236957 shutdown_connections 2024-03-24T00:19:51.578 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:51.577+0000 7fd634a00640 1 -- 172.21.15.118:0/1895236957 wait complete. 2024-03-24T00:19:51.581 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:51.578+0000 7fd634a00640 1 Processor -- start 2024-03-24T00:19:51.581 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:51.578+0000 7fd634a00640 1 -- start start 2024-03-24T00:19:51.581 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:51.579+0000 7fd634a00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd630074100 0x7fd630199a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:19:51.581 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:51.579+0000 7fd634a00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fd6300756e0 0x7fd630199fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:19:51.581 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:51.579+0000 7fd634a00640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fd630109460 0x7fd6301a1020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:19:51.581 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:51.579+0000 7fd634a00640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd63010f400 con 0x7fd630109460 2024-03-24T00:19:51.581 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:51.579+0000 7fd634a00640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd63010f280 con 0x7fd630074100 2024-03-24T00:19:51.581 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:51.579+0000 7fd634a00640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd63010f580 con 0x7fd6300756e0 2024-03-24T00:19:51.581 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:51.579+0000 7fd62e800640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd630074100 0x7fd630199a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:19:51.581 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:51.579+0000 7fd62e800640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd630074100 0x7fd630199a70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.118:43568/0 (socket says 172.21.15.118:43568) 2024-03-24T00:19:51.581 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:51.579+0000 7fd62e800640 1 -- 172.21.15.118:0/2091361560 learned_addr learned my addr 172.21.15.118:0/2091361560 (peer_addr_for_me v2:172.21.15.118:0/0) 2024-03-24T00:19:51.582 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:51.579+0000 7fd627e00640 1 --2- 172.21.15.118:0/2091361560 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fd6300756e0 0x7fd630199fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:19:51.582 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:51.579+0000 7fd62f200640 1 --2- 172.21.15.118:0/2091361560 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fd630109460 0x7fd6301a1020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:19:51.582 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:51.579+0000 7fd62e800640 1 -- 172.21.15.118:0/2091361560 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fd6300756e0 msgr2=0x7fd630199fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:19:51.582 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:51.579+0000 7fd62e800640 1 --2- 172.21.15.118:0/2091361560 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fd6300756e0 0x7fd630199fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:19:51.582 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:51.579+0000 7fd62e800640 1 -- 172.21.15.118:0/2091361560 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fd630109460 msgr2=0x7fd6301a1020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:19:51.582 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:51.579+0000 7fd62e800640 1 --2- 172.21.15.118:0/2091361560 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fd630109460 0x7fd6301a1020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:19:51.582 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:51.579+0000 7fd62e800640 1 -- 172.21.15.118:0/2091361560 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd6301a1690 con 0x7fd630074100 2024-03-24T00:19:51.582 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:51.579+0000 7fd62f200640 1 --2- 172.21.15.118:0/2091361560 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fd630109460 0x7fd6301a1020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T00:19:51.582 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:51.579+0000 7fd627e00640 1 --2- 172.21.15.118:0/2091361560 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fd6300756e0 0x7fd630199fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T00:19:51.582 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:51.580+0000 7fd62e800640 1 --2- 172.21.15.118:0/2091361560 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd630074100 0x7fd630199a70 secure :-1 s=READY pgs=400 cs=0 l=1 rev1=1 crypto rx=0x7fd620009230 tx=0x7fd62002c660 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:19:51.582 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:51.580+0000 7fd627400640 1 -- 172.21.15.118:0/2091361560 <== mon.2 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd620041070 con 0x7fd630074100 2024-03-24T00:19:51.582 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:51.580+0000 7fd627400640 1 -- 172.21.15.118:0/2091361560 <== mon.2 v2:172.21.15.189:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fd620004450 con 0x7fd630074100 2024-03-24T00:19:51.582 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:51.580+0000 7fd634a00640 1 -- 172.21.15.118:0/2091361560 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd6301a1920 con 0x7fd630074100 2024-03-24T00:19:51.583 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:51.580+0000 7fd627400640 1 -- 172.21.15.118:0/2091361560 <== mon.2 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd620003e00 con 0x7fd630074100 2024-03-24T00:19:51.583 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:51.580+0000 7fd634a00640 1 -- 172.21.15.118:0/2091361560 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd6301a1e10 con 0x7fd630074100 2024-03-24T00:19:51.584 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:51.583+0000 7fd638029740 1 Processor -- start 2024-03-24T00:19:51.584 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:51.584+0000 7fd638029740 1 -- start start 2024-03-24T00:19:51.586 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:51.584+0000 7fd638029740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55c5538f49f0 0x55c5538f4e10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:19:51.586 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:51.584+0000 7fd638029740 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55c5538f5bc0 0x55c5538f6040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:19:51.586 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:51.584+0000 7fd638029740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55c5538f6580 0x55c5538fce10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:19:51.587 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:51.584+0000 7fd638029740 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c5538592c0 con 0x55c5538f5bc0 2024-03-24T00:19:51.587 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:51.584+0000 7fd638029740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c553801c90 con 0x55c5538f6580 2024-03-24T00:19:51.587 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:51.584+0000 7fd638029740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c553855360 con 0x55c5538f49f0 2024-03-24T00:19:51.587 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:51.584+0000 7fd62f200640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55c5538f49f0 0x55c5538f4e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:19:51.587 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:51.584+0000 7fd627e00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55c5538f6580 0x55c5538fce10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:19:51.587 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:51.584+0000 7fd62f200640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55c5538f49f0 0x55c5538f4e10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.118:37268/0 (socket says 172.21.15.118:37268) 2024-03-24T00:19:51.587 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:51.584+0000 7fd62f200640 1 -- 172.21.15.118:0/3896008140 learned_addr learned my addr 172.21.15.118:0/3896008140 (peer_addr_for_me v2:172.21.15.118:0/0) 2024-03-24T00:19:51.587 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:51.584+0000 7fd62f200640 1 -- 172.21.15.118:0/3896008140 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55c5538f6580 msgr2=0x55c5538fce10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:19:51.587 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:51.584+0000 7fd62f200640 1 --2- 172.21.15.118:0/3896008140 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55c5538f6580 0x55c5538fce10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:19:51.587 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:51.584+0000 7fd62f200640 1 -- 172.21.15.118:0/3896008140 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55c5538f5bc0 msgr2=0x55c5538f6040 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-24T00:19:51.587 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:51.584+0000 7fd62f200640 1 --2- 172.21.15.118:0/3896008140 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55c5538f5bc0 0x55c5538f6040 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:19:51.587 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:51.584+0000 7fd62f200640 1 -- 172.21.15.118:0/3896008140 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55c553826080 con 0x55c5538f49f0 2024-03-24T00:19:51.587 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:51.585+0000 7fd62f200640 1 --2- 172.21.15.118:0/3896008140 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55c5538f49f0 0x55c5538f4e10 secure :-1 s=READY pgs=396 cs=0 l=1 rev1=1 crypto rx=0x7fd628009fd0 tx=0x7fd6280108e0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=5ed4e5b4db176d31 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:19:51.588 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:51.585+0000 7fd61f400640 1 -- 172.21.15.118:0/3896008140 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd628011760 con 0x55c5538f49f0 2024-03-24T00:19:51.588 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:51.585+0000 7fd61f400640 1 -- 172.21.15.118:0/3896008140 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fd628011de0 con 0x55c5538f49f0 2024-03-24T00:19:51.588 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:51.585+0000 7fd61f400640 1 -- 172.21.15.118:0/3896008140 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd62800e3e0 con 0x55c5538f49f0 2024-03-24T00:19:51.588 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:51.585+0000 7fd627400640 1 -- 172.21.15.118:0/2091361560 <== mon.2 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 20) v1 ==== 137291+0+0 (secure 0 0 0) 0x7fd620044070 con 0x7fd630074100 2024-03-24T00:19:51.588 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:51.586+0000 7fd638029740 1 -- 172.21.15.118:0/3896008140 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55c5538f49f0 msgr2=0x55c5538f4e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:19:51.588 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:51.586+0000 7fd638029740 1 --2- 172.21.15.118:0/3896008140 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55c5538f49f0 0x55c5538f4e10 secure :-1 s=READY pgs=396 cs=0 l=1 rev1=1 crypto rx=0x7fd628009fd0 tx=0x7fd6280108e0 comp rx=0 tx=0).stop 2024-03-24T00:19:51.588 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:51.586+0000 7fd638029740 1 -- 172.21.15.118:0/3896008140 shutdown_connections 2024-03-24T00:19:51.588 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:51.586+0000 7fd638029740 1 --2- 172.21.15.118:0/3896008140 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55c5538f6580 0x55c5538fce10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:19:51.588 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:51.586+0000 7fd638029740 1 --2- 172.21.15.118:0/3896008140 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55c5538f5bc0 0x55c5538f6040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:19:51.588 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:51.586+0000 7fd638029740 1 --2- 172.21.15.118:0/3896008140 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55c5538f49f0 0x55c5538f4e10 unknown :-1 s=CLOSED pgs=396 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:19:51.588 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:51.586+0000 7fd638029740 1 -- 172.21.15.118:0/3896008140 >> 172.21.15.118:0/3896008140 conn(0x55c5538efde0 msgr2=0x55c5538f01e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:19:51.588 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:51.586+0000 7fd638029740 1 -- 172.21.15.118:0/3896008140 shutdown_connections 2024-03-24T00:19:51.588 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:51.586+0000 7fd638029740 1 -- 172.21.15.118:0/3896008140 wait complete. 2024-03-24T00:19:51.591 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:51.586+0000 7fd638029740 1 -- start start 2024-03-24T00:19:51.591 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:51.587+0000 7fd638029740 1 Processor -- start 2024-03-24T00:19:51.591 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:51.587+0000 7fd638029740 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55c5538f49f0 0x55c5538fecd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:19:51.591 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:51.587+0000 7fd638029740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55c5538f5bc0 0x55c5538ff210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:19:51.591 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:51.587+0000 7fd638029740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55c5538f6580 0x55c553906250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:19:51.591 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:51.587+0000 7fd638029740 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c5538fd590 con 0x55c5538f49f0 2024-03-24T00:19:51.592 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:51.587+0000 7fd638029740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c5538150d0 con 0x55c5538f5bc0 2024-03-24T00:19:51.592 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:51.587+0000 7fd638029740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c5538fd710 con 0x55c5538f6580 2024-03-24T00:19:51.592 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:51.587+0000 7fd638029740 10 client.0 _ll_register_callbacks cb 0x55c553819e80 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T00:19:51.592 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:51.587+0000 7fd627400640 1 --2- 172.21.15.118:0/2091361560 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7fd6140a2720 0x7fd6140a4be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:19:51.592 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:51.587+0000 7fd62e800640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55c5538f5bc0 0x55c5538ff210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:19:51.592 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:51.587+0000 7fd62e800640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55c5538f5bc0 0x55c5538ff210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.118:43584/0 (socket says 172.21.15.118:43584) 2024-03-24T00:19:51.592 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:51.587+0000 7fd62e800640 1 -- 172.21.15.118:0/736842230 learned_addr learned my addr 172.21.15.118:0/736842230 (peer_addr_for_me v2:172.21.15.118:0/0) 2024-03-24T00:19:51.592 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:51.587+0000 7fd627e00640 1 --2- 172.21.15.118:0/736842230 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55c5538f6580 0x55c553906250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:19:51.592 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:51.588+0000 7fd62f200640 1 --2- 172.21.15.118:0/736842230 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55c5538f49f0 0x55c5538fecd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:19:51.592 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:51.588+0000 7fd62f200640 1 --2- 172.21.15.118:0/2091361560 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7fd6140a2720 0x7fd6140a4be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:19:51.592 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:51.588+0000 7fd627400640 1 -- 172.21.15.118:0/2091361560 <== mon.2 v2:172.21.15.189:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 8363+0+0 (secure 0 0 0) 0x7fd6200f0780 con 0x7fd630074100 2024-03-24T00:19:51.592 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:51.588+0000 7fd62e800640 1 -- 172.21.15.118:0/736842230 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55c5538f6580 msgr2=0x55c553906250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:19:51.592 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:51.588+0000 7fd62e800640 1 --2- 172.21.15.118:0/736842230 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55c5538f6580 0x55c553906250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:19:51.592 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:51.588+0000 7fd62e800640 1 -- 172.21.15.118:0/736842230 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55c5538f49f0 msgr2=0x55c5538fecd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:19:51.593 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:51.588+0000 7fd62e800640 1 --2- 172.21.15.118:0/736842230 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55c5538f49f0 0x55c5538fecd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:19:51.593 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:51.588+0000 7fd62e800640 1 -- 172.21.15.118:0/736842230 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55c55356d3f0 con 0x55c5538f5bc0 2024-03-24T00:19:51.593 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:51.588+0000 7fd62e800640 1 --2- 172.21.15.118:0/736842230 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55c5538f5bc0 0x55c5538ff210 secure :-1 s=READY pgs=401 cs=0 l=1 rev1=1 crypto rx=0x7fd6200b31e0 tx=0x7fd6200057e0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=8142b8a981ca6da1 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:19:51.593 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:51.588+0000 7fd613e00640 10 client.0 ms_handle_connect on v2:172.21.15.189:3300/0 2024-03-24T00:19:51.593 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:51.588+0000 7fd613e00640 1 -- 172.21.15.118:0/736842230 <== mon.2 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd62004b070 con 0x55c5538f5bc0 2024-03-24T00:19:51.593 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:51.588+0000 7fd613e00640 1 -- 172.21.15.118:0/736842230 <== mon.2 v2:172.21.15.189:3300/0 2 ==== osd_map(91..91 src has 1..91) v4 ==== 8363+0+0 (secure 0 0 0) 0x7fd6200bd300 con 0x55c5538f5bc0 2024-03-24T00:19:51.593 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:51.588+0000 7fd638029740 1 -- 172.21.15.118:0/736842230 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fd628011de0 con 0x55c5538f5bc0 2024-03-24T00:19:51.593 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:51.589+0000 7fd613e00640 1 -- 172.21.15.118:0/736842230 <== mon.2 v2:172.21.15.189:3300/0 3 ==== mon_get_version_reply(handle=1 version=36) v2 ==== 24+0+0 (secure 0 0 0) 0x7fd6200bd7a0 con 0x55c5538f5bc0 2024-03-24T00:19:51.593 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:51.589+0000 7fd638029740 10 client.26594 fetch_fsmap learned FSMap version 36 2024-03-24T00:19:51.593 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:51.589+0000 7fd62f200640 1 --2- 172.21.15.118:0/2091361560 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7fd6140a2720 0x7fd6140a4be0 secure :-1 s=READY pgs=345 cs=0 l=1 rev1=1 crypto rx=0x7fd628010de0 tx=0x7fd628005fd0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:19:51.593 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:51.589+0000 7fd638029740 1 -- 172.21.15.118:0/736842230 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,fsmap=36,monmap=4+}) v3 -- 0x55c5536377d0 con 0x55c5538f5bc0 2024-03-24T00:19:51.593 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:51.589+0000 7fd613e00640 1 -- 172.21.15.118:0/736842230 <== mon.2 v2:172.21.15.189:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fd628011de0 con 0x55c5538f5bc0 2024-03-24T00:19:51.593 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:51.589+0000 7fd613e00640 1 -- 172.21.15.118:0/736842230 <== mon.2 v2:172.21.15.189:3300/0 5 ==== fsmap(e 36) v1 ==== 7938+0+0 (secure 0 0 0) 0x7fd6200c53b0 con 0x55c5538f5bc0 2024-03-24T00:19:51.593 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:51.589+0000 7fd638029740 10 client.26594 fetch_fsmap finished waiting for FSMap version 36 2024-03-24T00:19:51.593 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:51.589+0000 7fd638029740 10 client.26594 resolve_mds: resolved 1:3 to role '1:3' aka daemon mds.l 2024-03-24T00:19:51.594 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:51.589+0000 7fd638029740 20 client.26594 populate_metadata read hostname 'smithi118' 2024-03-24T00:19:51.594 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:51.589+0000 7fd638029740 1 --2- 172.21.15.118:0/736842230 >> [v2:172.21.15.193:6832/1023428412,v1:172.21.15.193:6833/1023428412] conn(0x55c55390a6a0 0x55c55390cac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:19:51.594 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:51.589+0000 7fd638029740 4 client.26594 mds_command: new command op to 24439 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-24T00:19:51.594 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:51.589+0000 7fd638029740 1 -- 172.21.15.118:0/736842230 --> [v2:172.21.15.193:6832/1023428412,v1:172.21.15.193:6833/1023428412] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55c553866800 con 0x55c55390a6a0 2024-03-24T00:19:51.594 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:51.589+0000 7fd627e00640 1 -- 172.21.15.118:0/736842230 >> [v2:172.21.15.193:6832/1023428412,v1:172.21.15.193:6833/1023428412] conn(0x55c55390a6a0 msgr2=0x55c55390cac0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.193:6832/1023428412 2024-03-24T00:19:51.594 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:51.589+0000 7fd613e00640 1 client.26594 ms_handle_refused on v2:172.21.15.193:6832/1023428412 2024-03-24T00:19:51.594 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:51.589+0000 7fd627e00640 1 --2- 172.21.15.118:0/736842230 >> [v2:172.21.15.193:6832/1023428412,v1:172.21.15.193:6833/1023428412] conn(0x55c55390a6a0 0x55c55390cac0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-24T00:19:51.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:19:51 smithi118 ceph-mon[31535]: mds.? [v2:172.21.15.193:6832/1023428412,v1:172.21.15.193:6833/1023428412] up:rejoin 2024-03-24T00:19:51.697 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:19:51 smithi118 ceph-mon[31535]: fsmap cephfs:5/5 4 up:standby-replay 3 up:standby 1 up:rejoin 4 up:active 2024-03-24T00:19:51.697 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:19:51 smithi118 ceph-mon[31535]: daemon mds.l is now active in filesystem cephfs as rank 3 2024-03-24T00:19:51.697 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:19:51 smithi118 ceph-mon[31535]: from='client.? 172.21.15.118:0/4071544082' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-24T00:19:51.703 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:19:51 smithi189 ceph-mon[36187]: mds.? [v2:172.21.15.193:6832/1023428412,v1:172.21.15.193:6833/1023428412] up:rejoin 2024-03-24T00:19:51.704 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:19:51 smithi189 ceph-mon[36187]: fsmap cephfs:5/5 4 up:standby-replay 3 up:standby 1 up:rejoin 4 up:active 2024-03-24T00:19:51.704 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:19:51 smithi189 ceph-mon[36187]: daemon mds.l is now active in filesystem cephfs as rank 3 2024-03-24T00:19:51.704 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:19:51 smithi189 ceph-mon[36187]: from='client.? 172.21.15.118:0/4071544082' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-24T00:19:51.710 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:19:51 smithi193 ceph-mon[36878]: mds.? [v2:172.21.15.193:6832/1023428412,v1:172.21.15.193:6833/1023428412] up:rejoin 2024-03-24T00:19:51.710 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:19:51 smithi193 ceph-mon[36878]: fsmap cephfs:5/5 4 up:standby-replay 3 up:standby 1 up:rejoin 4 up:active 2024-03-24T00:19:51.710 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:19:51 smithi193 ceph-mon[36878]: daemon mds.l is now active in filesystem cephfs as rank 3 2024-03-24T00:19:51.710 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:19:51 smithi193 ceph-mon[36878]: from='client.? 172.21.15.118:0/4071544082' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-24T00:19:51.711 INFO:journalctl@ceph.mds.l.smithi193.stdout:Mar 24 00:19:51 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-l[67161]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: In function 'void Locker::set_xlocks_done(MutationImpl*, bool)' thread 7fa281ba4700 time 2024-03-24T00:19:51.396131+0000 2024-03-24T00:19:51.711 INFO:journalctl@ceph.mds.l.smithi193.stdout:Mar 24 00:19:51 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-l[67161]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: 626: FAILED ceph_assert(obj->is_auth()) 2024-03-24T00:19:51.711 INFO:journalctl@ceph.mds.l.smithi193.stdout:Mar 24 00:19:51 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-l[67161]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-24T00:19:51.711 INFO:journalctl@ceph.mds.l.smithi193.stdout:Mar 24 00:19:51 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-l[67161]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x135) [0x7fa28c1ce4a1] 2024-03-24T00:19:51.711 INFO:journalctl@ceph.mds.l.smithi193.stdout:Mar 24 00:19:51 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-l[67161]: 2: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7fa28c1ce667] 2024-03-24T00:19:51.711 INFO:journalctl@ceph.mds.l.smithi193.stdout:Mar 24 00:19:51 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-l[67161]: 3: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x5557d8896223] 2024-03-24T00:19:51.711 INFO:journalctl@ceph.mds.l.smithi193.stdout:Mar 24 00:19:51 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-l[67161]: 4: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x5557d87105d6] 2024-03-24T00:19:51.712 INFO:journalctl@ceph.mds.l.smithi193.stdout:Mar 24 00:19:51 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-l[67161]: 5: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x5557d8710da1] 2024-03-24T00:19:51.712 INFO:journalctl@ceph.mds.l.smithi193.stdout:Mar 24 00:19:51 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-l[67161]: 6: (Server::handle_client_rename(boost::intrusive_ptr const&)+0x259d) [0x5557d874d97d] 2024-03-24T00:19:51.712 INFO:journalctl@ceph.mds.l.smithi193.stdout:Mar 24 00:19:51 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-l[67161]: 7: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x5557d8763233] 2024-03-24T00:19:51.712 INFO:journalctl@ceph.mds.l.smithi193.stdout:Mar 24 00:19:51 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-l[67161]: 8: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x5557d8765197] 2024-03-24T00:19:51.712 INFO:journalctl@ceph.mds.l.smithi193.stdout:Mar 24 00:19:51 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-l[67161]: 9: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x5557d866411b] 2024-03-24T00:19:51.712 INFO:journalctl@ceph.mds.l.smithi193.stdout:Mar 24 00:19:51 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-l[67161]: 10: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x5557d86646bc] 2024-03-24T00:19:51.712 INFO:journalctl@ceph.mds.l.smithi193.stdout:Mar 24 00:19:51 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-l[67161]: 11: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x5557d864e1ff] 2024-03-24T00:19:51.712 INFO:journalctl@ceph.mds.l.smithi193.stdout:Mar 24 00:19:51 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-l[67161]: 12: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7fa28c460488] 2024-03-24T00:19:51.712 INFO:journalctl@ceph.mds.l.smithi193.stdout:Mar 24 00:19:51 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-l[67161]: 13: (DispatchQueue::entry()+0x51f) [0x7fa28c45d60f] 2024-03-24T00:19:51.712 INFO:journalctl@ceph.mds.l.smithi193.stdout:Mar 24 00:19:51 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-l[67161]: 14: (DispatchQueue::DispatchThread::entry()+0x11) [0x7fa28c530e81] 2024-03-24T00:19:51.712 INFO:journalctl@ceph.mds.l.smithi193.stdout:Mar 24 00:19:51 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-l[67161]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7fa28af4b1ca] 2024-03-24T00:19:51.712 INFO:journalctl@ceph.mds.l.smithi193.stdout:Mar 24 00:19:51 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-l[67161]: 16: clone() 2024-03-24T00:19:51.713 INFO:journalctl@ceph.mds.l.smithi193.stdout:Mar 24 00:19:51 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-l[67161]: 2024-03-24T00:19:51.396+0000 7fa281ba4700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: In function 'void Locker::set_xlocks_done(MutationImpl*, bool)' thread 7fa281ba4700 time 2024-03-24T00:19:51.396131+0000 2024-03-24T00:19:51.713 INFO:journalctl@ceph.mds.l.smithi193.stdout:Mar 24 00:19:51 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-l[67161]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: 626: FAILED ceph_assert(obj->is_auth()) 2024-03-24T00:19:51.713 INFO:journalctl@ceph.mds.l.smithi193.stdout:Mar 24 00:19:51 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-l[67161]: 2024-03-24T00:19:51.714 INFO:journalctl@ceph.mds.l.smithi193.stdout:Mar 24 00:19:51 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-l[67161]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-24T00:19:51.714 INFO:journalctl@ceph.mds.l.smithi193.stdout:Mar 24 00:19:51 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-l[67161]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x135) [0x7fa28c1ce4a1] 2024-03-24T00:19:51.714 INFO:journalctl@ceph.mds.l.smithi193.stdout:Mar 24 00:19:51 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-l[67161]: 2: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7fa28c1ce667] 2024-03-24T00:19:51.714 INFO:journalctl@ceph.mds.l.smithi193.stdout:Mar 24 00:19:51 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-l[67161]: 3: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x5557d8896223] 2024-03-24T00:19:51.714 INFO:journalctl@ceph.mds.l.smithi193.stdout:Mar 24 00:19:51 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-l[67161]: 4: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x5557d87105d6] 2024-03-24T00:19:51.714 INFO:journalctl@ceph.mds.l.smithi193.stdout:Mar 24 00:19:51 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-l[67161]: 5: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x5557d8710da1] 2024-03-24T00:19:51.714 INFO:journalctl@ceph.mds.l.smithi193.stdout:Mar 24 00:19:51 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-l[67161]: 6: (Server::handle_client_rename(boost::intrusive_ptr const&)+0x259d) [0x5557d874d97d] 2024-03-24T00:19:51.714 INFO:journalctl@ceph.mds.l.smithi193.stdout:Mar 24 00:19:51 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-l[67161]: 7: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x5557d8763233] 2024-03-24T00:19:51.714 INFO:journalctl@ceph.mds.l.smithi193.stdout:Mar 24 00:19:51 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-l[67161]: 8: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x5557d8765197] 2024-03-24T00:19:51.714 INFO:journalctl@ceph.mds.l.smithi193.stdout:Mar 24 00:19:51 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-l[67161]: 9: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x5557d866411b] 2024-03-24T00:19:51.714 INFO:journalctl@ceph.mds.l.smithi193.stdout:Mar 24 00:19:51 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-l[67161]: 10: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x5557d86646bc] 2024-03-24T00:19:51.714 INFO:journalctl@ceph.mds.l.smithi193.stdout:Mar 24 00:19:51 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-l[67161]: 11: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x5557d864e1ff] 2024-03-24T00:19:51.715 INFO:journalctl@ceph.mds.l.smithi193.stdout:Mar 24 00:19:51 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-l[67161]: 12: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7fa28c460488] 2024-03-24T00:19:51.715 INFO:journalctl@ceph.mds.l.smithi193.stdout:Mar 24 00:19:51 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-l[67161]: 13: (DispatchQueue::entry()+0x51f) [0x7fa28c45d60f] 2024-03-24T00:19:51.715 INFO:journalctl@ceph.mds.l.smithi193.stdout:Mar 24 00:19:51 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-l[67161]: 14: (DispatchQueue::DispatchThread::entry()+0x11) [0x7fa28c530e81] 2024-03-24T00:19:51.715 INFO:journalctl@ceph.mds.l.smithi193.stdout:Mar 24 00:19:51 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-l[67161]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7fa28af4b1ca] 2024-03-24T00:19:51.715 INFO:journalctl@ceph.mds.l.smithi193.stdout:Mar 24 00:19:51 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-l[67161]: 16: clone() 2024-03-24T00:19:51.715 INFO:journalctl@ceph.mds.l.smithi193.stdout:Mar 24 00:19:51 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-l[67161]: 2024-03-24T00:19:51.715 INFO:journalctl@ceph.mds.l.smithi193.stdout:Mar 24 00:19:51 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-l[67161]: *** Caught signal (Aborted) ** 2024-03-24T00:19:51.715 INFO:journalctl@ceph.mds.l.smithi193.stdout:Mar 24 00:19:51 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-l[67161]: in thread 7fa281ba4700 thread_name:ms_dispatch 2024-03-24T00:19:51.715 INFO:journalctl@ceph.mds.l.smithi193.stdout:Mar 24 00:19:51 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-l[67161]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-24T00:19:51.715 INFO:journalctl@ceph.mds.l.smithi193.stdout:Mar 24 00:19:51 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-l[67161]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7fa28af55d20] 2024-03-24T00:19:51.715 INFO:journalctl@ceph.mds.l.smithi193.stdout:Mar 24 00:19:51 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-l[67161]: 2: gsignal() 2024-03-24T00:19:51.715 INFO:journalctl@ceph.mds.l.smithi193.stdout:Mar 24 00:19:51 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-l[67161]: 3: abort() 2024-03-24T00:19:51.715 INFO:journalctl@ceph.mds.l.smithi193.stdout:Mar 24 00:19:51 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-l[67161]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x18f) [0x7fa28c1ce4fb] 2024-03-24T00:19:51.715 INFO:journalctl@ceph.mds.l.smithi193.stdout:Mar 24 00:19:51 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-l[67161]: 5: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7fa28c1ce667] 2024-03-24T00:19:51.715 INFO:journalctl@ceph.mds.l.smithi193.stdout:Mar 24 00:19:51 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-l[67161]: 6: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x5557d8896223] 2024-03-24T00:19:51.716 INFO:journalctl@ceph.mds.l.smithi193.stdout:Mar 24 00:19:51 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-l[67161]: 7: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x5557d87105d6] 2024-03-24T00:19:51.716 INFO:journalctl@ceph.mds.l.smithi193.stdout:Mar 24 00:19:51 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-l[67161]: 8: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x5557d8710da1] 2024-03-24T00:19:51.716 INFO:journalctl@ceph.mds.l.smithi193.stdout:Mar 24 00:19:51 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-l[67161]: 9: (Server::handle_client_rename(boost::intrusive_ptr const&)+0x259d) [0x5557d874d97d] 2024-03-24T00:19:51.716 INFO:journalctl@ceph.mds.l.smithi193.stdout:Mar 24 00:19:51 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-l[67161]: 10: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x5557d8763233] 2024-03-24T00:19:51.716 INFO:journalctl@ceph.mds.l.smithi193.stdout:Mar 24 00:19:51 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-l[67161]: 11: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x5557d8765197] 2024-03-24T00:19:51.716 INFO:journalctl@ceph.mds.l.smithi193.stdout:Mar 24 00:19:51 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-l[67161]: 12: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x5557d866411b] 2024-03-24T00:19:51.716 INFO:journalctl@ceph.mds.l.smithi193.stdout:Mar 24 00:19:51 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-l[67161]: 13: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x5557d86646bc] 2024-03-24T00:19:51.716 INFO:journalctl@ceph.mds.l.smithi193.stdout:Mar 24 00:19:51 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-l[67161]: 14: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x5557d864e1ff] 2024-03-24T00:19:51.716 INFO:journalctl@ceph.mds.l.smithi193.stdout:Mar 24 00:19:51 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-l[67161]: 15: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7fa28c460488] 2024-03-24T00:19:51.716 INFO:journalctl@ceph.mds.l.smithi193.stdout:Mar 24 00:19:51 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-l[67161]: 16: (DispatchQueue::entry()+0x51f) [0x7fa28c45d60f] 2024-03-24T00:19:51.716 INFO:journalctl@ceph.mds.l.smithi193.stdout:Mar 24 00:19:51 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-l[67161]: 17: (DispatchQueue::DispatchThread::entry()+0x11) [0x7fa28c530e81] 2024-03-24T00:19:51.716 INFO:journalctl@ceph.mds.l.smithi193.stdout:Mar 24 00:19:51 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-l[67161]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7fa28af4b1ca] 2024-03-24T00:19:51.716 INFO:journalctl@ceph.mds.l.smithi193.stdout:Mar 24 00:19:51 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-l[67161]: 19: clone() 2024-03-24T00:19:51.716 INFO:journalctl@ceph.mds.l.smithi193.stdout:Mar 24 00:19:51 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-l[67161]: 2024-03-24T00:19:51.398+0000 7fa281ba4700 -1 *** Caught signal (Aborted) ** 2024-03-24T00:19:51.716 INFO:journalctl@ceph.mds.l.smithi193.stdout:Mar 24 00:19:51 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-l[67161]: in thread 7fa281ba4700 thread_name:ms_dispatch 2024-03-24T00:19:51.717 INFO:journalctl@ceph.mds.l.smithi193.stdout:Mar 24 00:19:51 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-l[67161]: 2024-03-24T00:19:51.717 INFO:journalctl@ceph.mds.l.smithi193.stdout:Mar 24 00:19:51 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-l[67161]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-24T00:19:51.717 INFO:journalctl@ceph.mds.l.smithi193.stdout:Mar 24 00:19:51 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-l[67161]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7fa28af55d20] 2024-03-24T00:19:51.717 INFO:journalctl@ceph.mds.l.smithi193.stdout:Mar 24 00:19:51 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-l[67161]: 2: gsignal() 2024-03-24T00:19:51.717 INFO:journalctl@ceph.mds.l.smithi193.stdout:Mar 24 00:19:51 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-l[67161]: 3: abort() 2024-03-24T00:19:51.717 INFO:journalctl@ceph.mds.l.smithi193.stdout:Mar 24 00:19:51 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-l[67161]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x18f) [0x7fa28c1ce4fb] 2024-03-24T00:19:51.717 INFO:journalctl@ceph.mds.l.smithi193.stdout:Mar 24 00:19:51 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-l[67161]: 5: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7fa28c1ce667] 2024-03-24T00:19:51.717 INFO:journalctl@ceph.mds.l.smithi193.stdout:Mar 24 00:19:51 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-l[67161]: 6: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x5557d8896223] 2024-03-24T00:19:51.717 INFO:journalctl@ceph.mds.l.smithi193.stdout:Mar 24 00:19:51 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-l[67161]: 7: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x5557d87105d6] 2024-03-24T00:19:51.717 INFO:journalctl@ceph.mds.l.smithi193.stdout:Mar 24 00:19:51 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-l[67161]: 8: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x5557d8710da1] 2024-03-24T00:19:51.717 INFO:journalctl@ceph.mds.l.smithi193.stdout:Mar 24 00:19:51 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-l[67161]: 9: (Server::handle_client_rename(boost::intrusive_ptr const&)+0x259d) [0x5557d874d97d] 2024-03-24T00:19:51.717 INFO:journalctl@ceph.mds.l.smithi193.stdout:Mar 24 00:19:51 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-l[67161]: 10: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x5557d8763233] 2024-03-24T00:19:51.717 INFO:journalctl@ceph.mds.l.smithi193.stdout:Mar 24 00:19:51 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-l[67161]: 11: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x5557d8765197] 2024-03-24T00:19:51.717 INFO:journalctl@ceph.mds.l.smithi193.stdout:Mar 24 00:19:51 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-l[67161]: 12: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x5557d866411b] 2024-03-24T00:19:51.718 INFO:journalctl@ceph.mds.l.smithi193.stdout:Mar 24 00:19:51 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-l[67161]: 13: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x5557d86646bc] 2024-03-24T00:19:51.718 INFO:journalctl@ceph.mds.l.smithi193.stdout:Mar 24 00:19:51 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-l[67161]: 14: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x5557d864e1ff] 2024-03-24T00:19:51.718 INFO:journalctl@ceph.mds.l.smithi193.stdout:Mar 24 00:19:51 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-l[67161]: 15: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7fa28c460488] 2024-03-24T00:19:51.718 INFO:journalctl@ceph.mds.l.smithi193.stdout:Mar 24 00:19:51 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-l[67161]: 16: (DispatchQueue::entry()+0x51f) [0x7fa28c45d60f] 2024-03-24T00:19:51.718 INFO:journalctl@ceph.mds.l.smithi193.stdout:Mar 24 00:19:51 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-l[67161]: 17: (DispatchQueue::DispatchThread::entry()+0x11) [0x7fa28c530e81] 2024-03-24T00:19:51.718 INFO:journalctl@ceph.mds.l.smithi193.stdout:Mar 24 00:19:51 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-l[67161]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7fa28af4b1ca] 2024-03-24T00:19:51.718 INFO:journalctl@ceph.mds.l.smithi193.stdout:Mar 24 00:19:51 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-l[67161]: 19: clone() 2024-03-24T00:19:51.718 INFO:journalctl@ceph.mds.l.smithi193.stdout:Mar 24 00:19:51 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-l[67161]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-03-24T00:19:51.718 INFO:journalctl@ceph.mds.l.smithi193.stdout:Mar 24 00:19:51 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-l[67161]: 2024-03-24T00:19:51.718 INFO:journalctl@ceph.mds.l.smithi193.stdout:Mar 24 00:19:51 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-l[67161]: -1> 2024-03-24T00:19:51.396+0000 7fa281ba4700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: In function 'void Locker::set_xlocks_done(MutationImpl*, bool)' thread 7fa281ba4700 time 2024-03-24T00:19:51.396131+0000 2024-03-24T00:19:51.718 INFO:journalctl@ceph.mds.l.smithi193.stdout:Mar 24 00:19:51 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-l[67161]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: 626: FAILED ceph_assert(obj->is_auth()) 2024-03-24T00:19:51.718 INFO:journalctl@ceph.mds.l.smithi193.stdout:Mar 24 00:19:51 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-l[67161]: 2024-03-24T00:19:51.718 INFO:journalctl@ceph.mds.l.smithi193.stdout:Mar 24 00:19:51 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-l[67161]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-24T00:19:51.718 INFO:journalctl@ceph.mds.l.smithi193.stdout:Mar 24 00:19:51 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-l[67161]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x135) [0x7fa28c1ce4a1] 2024-03-24T00:19:51.719 INFO:journalctl@ceph.mds.l.smithi193.stdout:Mar 24 00:19:51 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-l[67161]: 2: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7fa28c1ce667] 2024-03-24T00:19:51.719 INFO:journalctl@ceph.mds.l.smithi193.stdout:Mar 24 00:19:51 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-l[67161]: 3: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x5557d8896223] 2024-03-24T00:19:51.719 INFO:journalctl@ceph.mds.l.smithi193.stdout:Mar 24 00:19:51 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-l[67161]: 4: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x5557d87105d6] 2024-03-24T00:19:51.719 INFO:journalctl@ceph.mds.l.smithi193.stdout:Mar 24 00:19:51 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-l[67161]: 5: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x5557d8710da1] 2024-03-24T00:19:51.719 INFO:journalctl@ceph.mds.l.smithi193.stdout:Mar 24 00:19:51 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-l[67161]: 6: (Server::handle_client_rename(boost::intrusive_ptr const&)+0x259d) [0x5557d874d97d] 2024-03-24T00:19:51.719 INFO:journalctl@ceph.mds.l.smithi193.stdout:Mar 24 00:19:51 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-l[67161]: 7: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x5557d8763233] 2024-03-24T00:19:51.719 INFO:journalctl@ceph.mds.l.smithi193.stdout:Mar 24 00:19:51 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-l[67161]: 8: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x5557d8765197] 2024-03-24T00:19:51.719 INFO:journalctl@ceph.mds.l.smithi193.stdout:Mar 24 00:19:51 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-l[67161]: 9: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x5557d866411b] 2024-03-24T00:19:51.719 INFO:journalctl@ceph.mds.l.smithi193.stdout:Mar 24 00:19:51 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-l[67161]: 10: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x5557d86646bc] 2024-03-24T00:19:51.719 INFO:journalctl@ceph.mds.l.smithi193.stdout:Mar 24 00:19:51 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-l[67161]: 11: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x5557d864e1ff] 2024-03-24T00:19:51.719 INFO:journalctl@ceph.mds.l.smithi193.stdout:Mar 24 00:19:51 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-l[67161]: 12: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7fa28c460488] 2024-03-24T00:19:51.719 INFO:journalctl@ceph.mds.l.smithi193.stdout:Mar 24 00:19:51 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-l[67161]: 13: (DispatchQueue::entry()+0x51f) [0x7fa28c45d60f] 2024-03-24T00:19:51.719 INFO:journalctl@ceph.mds.l.smithi193.stdout:Mar 24 00:19:51 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-l[67161]: 14: (DispatchQueue::DispatchThread::entry()+0x11) [0x7fa28c530e81] 2024-03-24T00:19:51.719 INFO:journalctl@ceph.mds.l.smithi193.stdout:Mar 24 00:19:51 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-l[67161]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7fa28af4b1ca] 2024-03-24T00:19:51.719 INFO:journalctl@ceph.mds.l.smithi193.stdout:Mar 24 00:19:51 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-l[67161]: 16: clone() 2024-03-24T00:19:51.720 INFO:journalctl@ceph.mds.l.smithi193.stdout:Mar 24 00:19:51 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-l[67161]: 2024-03-24T00:19:51.720 INFO:journalctl@ceph.mds.l.smithi193.stdout:Mar 24 00:19:51 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-l[67161]: 0> 2024-03-24T00:19:51.398+0000 7fa281ba4700 -1 *** Caught signal (Aborted) ** 2024-03-24T00:19:51.720 INFO:journalctl@ceph.mds.l.smithi193.stdout:Mar 24 00:19:51 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-l[67161]: in thread 7fa281ba4700 thread_name:ms_dispatch 2024-03-24T00:19:51.720 INFO:journalctl@ceph.mds.l.smithi193.stdout:Mar 24 00:19:51 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-l[67161]: 2024-03-24T00:19:51.720 INFO:journalctl@ceph.mds.l.smithi193.stdout:Mar 24 00:19:51 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-l[67161]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-24T00:19:51.720 INFO:journalctl@ceph.mds.l.smithi193.stdout:Mar 24 00:19:51 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-l[67161]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7fa28af55d20] 2024-03-24T00:19:51.720 INFO:journalctl@ceph.mds.l.smithi193.stdout:Mar 24 00:19:51 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-l[67161]: 2: gsignal() 2024-03-24T00:19:51.720 INFO:journalctl@ceph.mds.l.smithi193.stdout:Mar 24 00:19:51 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-l[67161]: 3: abort() 2024-03-24T00:19:51.720 INFO:journalctl@ceph.mds.l.smithi193.stdout:Mar 24 00:19:51 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-l[67161]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x18f) [0x7fa28c1ce4fb] 2024-03-24T00:19:51.720 INFO:journalctl@ceph.mds.l.smithi193.stdout:Mar 24 00:19:51 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-l[67161]: 5: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7fa28c1ce667] 2024-03-24T00:19:51.720 INFO:journalctl@ceph.mds.l.smithi193.stdout:Mar 24 00:19:51 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-l[67161]: 6: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x5557d8896223] 2024-03-24T00:19:51.720 INFO:journalctl@ceph.mds.l.smithi193.stdout:Mar 24 00:19:51 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-l[67161]: 7: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x5557d87105d6] 2024-03-24T00:19:51.720 INFO:journalctl@ceph.mds.l.smithi193.stdout:Mar 24 00:19:51 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-l[67161]: 8: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x5557d8710da1] 2024-03-24T00:19:51.720 INFO:journalctl@ceph.mds.l.smithi193.stdout:Mar 24 00:19:51 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-l[67161]: 9: (Server::handle_client_rename(boost::intrusive_ptr const&)+0x259d) [0x5557d874d97d] 2024-03-24T00:19:51.721 INFO:journalctl@ceph.mds.l.smithi193.stdout:Mar 24 00:19:51 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-l[67161]: 10: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x5557d8763233] 2024-03-24T00:19:51.721 INFO:journalctl@ceph.mds.l.smithi193.stdout:Mar 24 00:19:51 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-l[67161]: 11: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x5557d8765197] 2024-03-24T00:19:51.721 INFO:journalctl@ceph.mds.l.smithi193.stdout:Mar 24 00:19:51 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-l[67161]: 12: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x5557d866411b] 2024-03-24T00:19:51.721 INFO:journalctl@ceph.mds.l.smithi193.stdout:Mar 24 00:19:51 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-l[67161]: 13: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x5557d86646bc] 2024-03-24T00:19:51.721 INFO:journalctl@ceph.mds.l.smithi193.stdout:Mar 24 00:19:51 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-l[67161]: 14: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x5557d864e1ff] 2024-03-24T00:19:51.721 INFO:journalctl@ceph.mds.l.smithi193.stdout:Mar 24 00:19:51 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-l[67161]: 15: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7fa28c460488] 2024-03-24T00:19:51.721 INFO:journalctl@ceph.mds.l.smithi193.stdout:Mar 24 00:19:51 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-l[67161]: 16: (DispatchQueue::entry()+0x51f) [0x7fa28c45d60f] 2024-03-24T00:19:51.721 INFO:journalctl@ceph.mds.l.smithi193.stdout:Mar 24 00:19:51 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-l[67161]: 17: (DispatchQueue::DispatchThread::entry()+0x11) [0x7fa28c530e81] 2024-03-24T00:19:51.721 INFO:journalctl@ceph.mds.l.smithi193.stdout:Mar 24 00:19:51 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-l[67161]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7fa28af4b1ca] 2024-03-24T00:19:51.721 INFO:journalctl@ceph.mds.l.smithi193.stdout:Mar 24 00:19:51 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-l[67161]: 19: clone() 2024-03-24T00:19:51.721 INFO:journalctl@ceph.mds.l.smithi193.stdout:Mar 24 00:19:51 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-l[67161]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-03-24T00:19:51.721 INFO:journalctl@ceph.mds.l.smithi193.stdout:Mar 24 00:19:51 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-l[67161]: 2024-03-24T00:19:51.721 INFO:journalctl@ceph.mds.l.smithi193.stdout:Mar 24 00:19:51 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-l[67161]: -1> 2024-03-24T00:19:51.396+0000 7fa281ba4700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: In function 'void Locker::set_xlocks_done(MutationImpl*, bool)' thread 7fa281ba4700 time 2024-03-24T00:19:51.396131+0000 2024-03-24T00:19:51.721 INFO:journalctl@ceph.mds.l.smithi193.stdout:Mar 24 00:19:51 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-l[67161]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: 626: FAILED ceph_assert(obj->is_auth()) 2024-03-24T00:19:51.721 INFO:journalctl@ceph.mds.l.smithi193.stdout:Mar 24 00:19:51 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-l[67161]: 2024-03-24T00:19:51.722 INFO:journalctl@ceph.mds.l.smithi193.stdout:Mar 24 00:19:51 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-l[67161]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-24T00:19:51.722 INFO:journalctl@ceph.mds.l.smithi193.stdout:Mar 24 00:19:51 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-l[67161]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x135) [0x7fa28c1ce4a1] 2024-03-24T00:19:51.722 INFO:journalctl@ceph.mds.l.smithi193.stdout:Mar 24 00:19:51 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-l[67161]: 2: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7fa28c1ce667] 2024-03-24T00:19:51.722 INFO:journalctl@ceph.mds.l.smithi193.stdout:Mar 24 00:19:51 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-l[67161]: 3: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x5557d8896223] 2024-03-24T00:19:51.722 INFO:journalctl@ceph.mds.l.smithi193.stdout:Mar 24 00:19:51 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-l[67161]: 4: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x5557d87105d6] 2024-03-24T00:19:51.722 INFO:journalctl@ceph.mds.l.smithi193.stdout:Mar 24 00:19:51 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-l[67161]: 5: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x5557d8710da1] 2024-03-24T00:19:51.722 INFO:journalctl@ceph.mds.l.smithi193.stdout:Mar 24 00:19:51 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-l[67161]: 6: (Server::handle_client_rename(boost::intrusive_ptr const&)+0x259d) [0x5557d874d97d] 2024-03-24T00:19:51.722 INFO:journalctl@ceph.mds.l.smithi193.stdout:Mar 24 00:19:51 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-l[67161]: 7: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x5557d8763233] 2024-03-24T00:19:51.723 INFO:journalctl@ceph.mds.l.smithi193.stdout:Mar 24 00:19:51 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-l[67161]: 8: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x5557d8765197] 2024-03-24T00:19:51.723 INFO:journalctl@ceph.mds.l.smithi193.stdout:Mar 24 00:19:51 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-l[67161]: 9: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x5557d866411b] 2024-03-24T00:19:51.723 INFO:journalctl@ceph.mds.l.smithi193.stdout:Mar 24 00:19:51 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-l[67161]: 10: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x5557d86646bc] 2024-03-24T00:19:51.723 INFO:journalctl@ceph.mds.l.smithi193.stdout:Mar 24 00:19:51 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-l[67161]: 11: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x5557d864e1ff] 2024-03-24T00:19:51.723 INFO:journalctl@ceph.mds.l.smithi193.stdout:Mar 24 00:19:51 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-l[67161]: 12: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7fa28c460488] 2024-03-24T00:19:51.723 INFO:journalctl@ceph.mds.l.smithi193.stdout:Mar 24 00:19:51 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-l[67161]: 13: (DispatchQueue::entry()+0x51f) [0x7fa28c45d60f] 2024-03-24T00:19:51.723 INFO:journalctl@ceph.mds.l.smithi193.stdout:Mar 24 00:19:51 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-l[67161]: 14: (DispatchQueue::DispatchThread::entry()+0x11) [0x7fa28c530e81] 2024-03-24T00:19:51.723 INFO:journalctl@ceph.mds.l.smithi193.stdout:Mar 24 00:19:51 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-l[67161]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7fa28af4b1ca] 2024-03-24T00:19:51.723 INFO:journalctl@ceph.mds.l.smithi193.stdout:Mar 24 00:19:51 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-l[67161]: 16: clone() 2024-03-24T00:19:51.723 INFO:journalctl@ceph.mds.l.smithi193.stdout:Mar 24 00:19:51 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-l[67161]: 2024-03-24T00:19:51.723 INFO:journalctl@ceph.mds.l.smithi193.stdout:Mar 24 00:19:51 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-l[67161]: 0> 2024-03-24T00:19:51.398+0000 7fa281ba4700 -1 *** Caught signal (Aborted) ** 2024-03-24T00:19:51.723 INFO:journalctl@ceph.mds.l.smithi193.stdout:Mar 24 00:19:51 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-l[67161]: in thread 7fa281ba4700 thread_name:ms_dispatch 2024-03-24T00:19:51.723 INFO:journalctl@ceph.mds.l.smithi193.stdout:Mar 24 00:19:51 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-l[67161]: 2024-03-24T00:19:51.723 INFO:journalctl@ceph.mds.l.smithi193.stdout:Mar 24 00:19:51 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-l[67161]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-24T00:19:51.724 INFO:journalctl@ceph.mds.l.smithi193.stdout:Mar 24 00:19:51 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-l[67161]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7fa28af55d20] 2024-03-24T00:19:51.724 INFO:journalctl@ceph.mds.l.smithi193.stdout:Mar 24 00:19:51 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-l[67161]: 2: gsignal() 2024-03-24T00:19:51.724 INFO:journalctl@ceph.mds.l.smithi193.stdout:Mar 24 00:19:51 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-l[67161]: 3: abort() 2024-03-24T00:19:51.724 INFO:journalctl@ceph.mds.l.smithi193.stdout:Mar 24 00:19:51 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-l[67161]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x18f) [0x7fa28c1ce4fb] 2024-03-24T00:19:51.724 INFO:journalctl@ceph.mds.l.smithi193.stdout:Mar 24 00:19:51 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-l[67161]: 5: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7fa28c1ce667] 2024-03-24T00:19:51.724 INFO:journalctl@ceph.mds.l.smithi193.stdout:Mar 24 00:19:51 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-l[67161]: 6: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x5557d8896223] 2024-03-24T00:19:51.724 INFO:journalctl@ceph.mds.l.smithi193.stdout:Mar 24 00:19:51 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-l[67161]: 7: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x5557d87105d6] 2024-03-24T00:19:51.724 INFO:journalctl@ceph.mds.l.smithi193.stdout:Mar 24 00:19:51 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-l[67161]: 8: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x5557d8710da1] 2024-03-24T00:19:51.724 INFO:journalctl@ceph.mds.l.smithi193.stdout:Mar 24 00:19:51 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-l[67161]: 9: (Server::handle_client_rename(boost::intrusive_ptr const&)+0x259d) [0x5557d874d97d] 2024-03-24T00:19:51.724 INFO:journalctl@ceph.mds.l.smithi193.stdout:Mar 24 00:19:51 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-l[67161]: 10: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x5557d8763233] 2024-03-24T00:19:51.724 INFO:journalctl@ceph.mds.l.smithi193.stdout:Mar 24 00:19:51 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-l[67161]: 11: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x5557d8765197] 2024-03-24T00:19:51.724 INFO:journalctl@ceph.mds.l.smithi193.stdout:Mar 24 00:19:51 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-l[67161]: 12: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x5557d866411b] 2024-03-24T00:19:51.724 INFO:journalctl@ceph.mds.l.smithi193.stdout:Mar 24 00:19:51 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-l[67161]: 13: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x5557d86646bc] 2024-03-24T00:19:51.724 INFO:journalctl@ceph.mds.l.smithi193.stdout:Mar 24 00:19:51 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-l[67161]: 14: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x5557d864e1ff] 2024-03-24T00:19:51.725 INFO:journalctl@ceph.mds.l.smithi193.stdout:Mar 24 00:19:51 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-l[67161]: 15: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7fa28c460488] 2024-03-24T00:19:51.725 INFO:journalctl@ceph.mds.l.smithi193.stdout:Mar 24 00:19:51 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-l[67161]: 16: (DispatchQueue::entry()+0x51f) [0x7fa28c45d60f] 2024-03-24T00:19:51.725 INFO:journalctl@ceph.mds.l.smithi193.stdout:Mar 24 00:19:51 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-l[67161]: 17: (DispatchQueue::DispatchThread::entry()+0x11) [0x7fa28c530e81] 2024-03-24T00:19:51.725 INFO:journalctl@ceph.mds.l.smithi193.stdout:Mar 24 00:19:51 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-l[67161]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7fa28af4b1ca] 2024-03-24T00:19:51.725 INFO:journalctl@ceph.mds.l.smithi193.stdout:Mar 24 00:19:51 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-l[67161]: 19: clone() 2024-03-24T00:19:51.725 INFO:journalctl@ceph.mds.l.smithi193.stdout:Mar 24 00:19:51 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-l[67161]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-03-24T00:19:51.725 INFO:journalctl@ceph.mds.l.smithi193.stdout:Mar 24 00:19:51 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-l[67161]: 2024-03-24T00:19:51.725 INFO:journalctl@ceph.mds.l.smithi193.stdout:Mar 24 00:19:51 smithi193 podman[70826]: 2024-03-24 00:19:51.455131771 +0000 UTC m=+0.019942208 container died af712abd88afa4d7323b2ec8d40f9e391e6b5a25c411254b91eb86f79dd3c586 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-l, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, io.buildah.version=1.33.5, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, org.label-schema.build-date=20240319, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, GIT_CLEAN=True) 2024-03-24T00:19:51.791 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:51.790+0000 7fd627e00640 1 -- 172.21.15.118:0/736842230 >> [v2:172.21.15.193:6832/1023428412,v1:172.21.15.193:6833/1023428412] conn(0x55c55390a6a0 msgr2=0x55c55390cac0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.193:6832/1023428412 2024-03-24T00:19:51.791 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:51.790+0000 7fd627e00640 1 --2- 172.21.15.118:0/736842230 >> [v2:172.21.15.193:6832/1023428412,v1:172.21.15.193:6833/1023428412] conn(0x55c55390a6a0 0x55c55390cac0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-24T00:19:51.791 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:51.790+0000 7fd613e00640 1 client.26594 ms_handle_refused on v2:172.21.15.193:6832/1023428412 2024-03-24T00:19:52.192 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:52.192+0000 7fd627e00640 1 -- 172.21.15.118:0/736842230 >> [v2:172.21.15.193:6832/1023428412,v1:172.21.15.193:6833/1023428412] conn(0x55c55390a6a0 msgr2=0x55c55390cac0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.193:6832/1023428412 2024-03-24T00:19:52.192 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:52.192+0000 7fd613e00640 1 client.26594 ms_handle_refused on v2:172.21.15.193:6832/1023428412 2024-03-24T00:19:52.192 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:52.192+0000 7fd627e00640 1 --2- 172.21.15.118:0/736842230 >> [v2:172.21.15.193:6832/1023428412,v1:172.21.15.193:6833/1023428412] conn(0x55c55390a6a0 0x55c55390cac0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-03-24T00:19:52.386 INFO:journalctl@ceph.mds.l.smithi193.stdout:Mar 24 00:19:52 smithi193 podman[70826]: 2024-03-24 00:19:52.117699883 +0000 UTC m=+0.682510326 container remove af712abd88afa4d7323b2ec8d40f9e391e6b5a25c411254b91eb86f79dd3c586 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-l, io.buildah.version=1.33.5, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, org.label-schema.build-date=20240319, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux ) 2024-03-24T00:19:52.386 INFO:journalctl@ceph.mds.l.smithi193.stdout:Mar 24 00:19:52 smithi193 systemd[1]: ceph-d79a5420-e971-11ee-95cd-87774f69a715@mds.l.service: Main process exited, code=exited, status=134/n/a 2024-03-24T00:19:52.386 INFO:journalctl@ceph.mds.l.smithi193.stdout:Mar 24 00:19:52 smithi193 systemd[1]: ceph-d79a5420-e971-11ee-95cd-87774f69a715@mds.l.service: Failed with result 'exit-code'. 2024-03-24T00:19:52.386 INFO:journalctl@ceph.mds.l.smithi193.stdout:Mar 24 00:19:52 smithi193 systemd[1]: ceph-d79a5420-e971-11ee-95cd-87774f69a715@mds.l.service: Consumed 1.082s CPU time. 2024-03-24T00:19:52.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:19:52 smithi118 ceph-mon[31535]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-03-24T00:19:52.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:19:52 smithi118 ceph-mon[31535]: Cluster is now healthy 2024-03-24T00:19:52.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:19:52 smithi118 ceph-mon[31535]: mds.? [v2:172.21.15.193:6832/1023428412,v1:172.21.15.193:6833/1023428412] up:active 2024-03-24T00:19:52.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:19:52 smithi118 ceph-mon[31535]: fsmap cephfs:5 4 up:standby-replay 3 up:standby 5 up:active 2024-03-24T00:19:52.697 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:19:52 smithi118 ceph-mon[31535]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-03-24T00:19:52.697 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:19:52 smithi118 ceph-mon[31535]: pgmap v524: 97 pgs: 97 active+clean; 93 MiB data, 613 MiB used, 1.0 TiB / 1.0 TiB avail; 8.1 KiB/s rd, 3.0 KiB/s wr, 14 op/s 2024-03-24T00:19:52.703 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:19:52 smithi189 ceph-mon[36187]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-03-24T00:19:52.704 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:19:52 smithi189 ceph-mon[36187]: Cluster is now healthy 2024-03-24T00:19:52.704 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:19:52 smithi189 ceph-mon[36187]: mds.? [v2:172.21.15.193:6832/1023428412,v1:172.21.15.193:6833/1023428412] up:active 2024-03-24T00:19:52.704 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:19:52 smithi189 ceph-mon[36187]: fsmap cephfs:5 4 up:standby-replay 3 up:standby 5 up:active 2024-03-24T00:19:52.704 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:19:52 smithi189 ceph-mon[36187]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-03-24T00:19:52.704 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:19:52 smithi189 ceph-mon[36187]: pgmap v524: 97 pgs: 97 active+clean; 93 MiB data, 613 MiB used, 1.0 TiB / 1.0 TiB avail; 8.1 KiB/s rd, 3.0 KiB/s wr, 14 op/s 2024-03-24T00:19:52.710 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:19:52 smithi193 ceph-mon[36878]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-03-24T00:19:52.710 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:19:52 smithi193 ceph-mon[36878]: Cluster is now healthy 2024-03-24T00:19:52.711 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:19:52 smithi193 ceph-mon[36878]: mds.? [v2:172.21.15.193:6832/1023428412,v1:172.21.15.193:6833/1023428412] up:active 2024-03-24T00:19:52.711 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:19:52 smithi193 ceph-mon[36878]: fsmap cephfs:5 4 up:standby-replay 3 up:standby 5 up:active 2024-03-24T00:19:52.711 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:19:52 smithi193 ceph-mon[36878]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-03-24T00:19:52.711 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:19:52 smithi193 ceph-mon[36878]: pgmap v524: 97 pgs: 97 active+clean; 93 MiB data, 613 MiB used, 1.0 TiB / 1.0 TiB avail; 8.1 KiB/s rd, 3.0 KiB/s wr, 14 op/s 2024-03-24T00:19:52.993 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:52.993+0000 7fd627e00640 1 -- 172.21.15.118:0/736842230 >> [v2:172.21.15.193:6832/1023428412,v1:172.21.15.193:6833/1023428412] conn(0x55c55390a6a0 msgr2=0x55c55390cac0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.193:6832/1023428412 2024-03-24T00:19:52.993 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:52.993+0000 7fd613e00640 1 client.26594 ms_handle_refused on v2:172.21.15.193:6832/1023428412 2024-03-24T00:19:52.993 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:52.993+0000 7fd627e00640 1 --2- 172.21.15.118:0/736842230 >> [v2:172.21.15.193:6832/1023428412,v1:172.21.15.193:6833/1023428412] conn(0x55c55390a6a0 0x55c55390cac0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-03-24T00:19:54.595 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:54.595+0000 7fd627e00640 1 -- 172.21.15.118:0/736842230 >> [v2:172.21.15.193:6832/1023428412,v1:172.21.15.193:6833/1023428412] conn(0x55c55390a6a0 msgr2=0x55c55390cac0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.193:6832/1023428412 2024-03-24T00:19:54.595 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:54.595+0000 7fd613e00640 1 client.26594 ms_handle_refused on v2:172.21.15.193:6832/1023428412 2024-03-24T00:19:54.595 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:54.595+0000 7fd627e00640 1 --2- 172.21.15.118:0/736842230 >> [v2:172.21.15.193:6832/1023428412,v1:172.21.15.193:6833/1023428412] conn(0x55c55390a6a0 0x55c55390cac0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 3.200000 2024-03-24T00:19:55.195 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:19:54 smithi118 ceph-mon[31535]: pgmap v525: 97 pgs: 97 active+clean; 95 MiB data, 615 MiB used, 1.0 TiB / 1.0 TiB avail; 641 KiB/s rd, 632 KiB/s wr, 18 op/s 2024-03-24T00:19:55.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:19:54 smithi189 ceph-mon[36187]: pgmap v525: 97 pgs: 97 active+clean; 95 MiB data, 615 MiB used, 1.0 TiB / 1.0 TiB avail; 641 KiB/s rd, 632 KiB/s wr, 18 op/s 2024-03-24T00:19:55.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:19:54 smithi193 ceph-mon[36878]: pgmap v525: 97 pgs: 97 active+clean; 95 MiB data, 615 MiB used, 1.0 TiB / 1.0 TiB avail; 641 KiB/s rd, 632 KiB/s wr, 18 op/s 2024-03-24T00:19:56.843 INFO:tasks.workunit.client.0.smithi118.stdout: 27 0 0 0 0 0 0 2024-03-24T00:19:57.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:19:56 smithi118 ceph-mon[31535]: pgmap v526: 97 pgs: 97 active+clean; 95 MiB data, 611 MiB used, 1.0 TiB / 1.0 TiB avail; 646 KiB/s rd, 695 KiB/s wr, 25 op/s 2024-03-24T00:19:57.203 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:19:56 smithi189 ceph-mon[36187]: pgmap v526: 97 pgs: 97 active+clean; 95 MiB data, 611 MiB used, 1.0 TiB / 1.0 TiB avail; 646 KiB/s rd, 695 KiB/s wr, 25 op/s 2024-03-24T00:19:57.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:19:56 smithi193 ceph-mon[36878]: pgmap v526: 97 pgs: 97 active+clean; 95 MiB data, 611 MiB used, 1.0 TiB / 1.0 TiB avail; 646 KiB/s rd, 695 KiB/s wr, 25 op/s 2024-03-24T00:19:57.797 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:57.797+0000 7fd627e00640 1 -- 172.21.15.118:0/736842230 >> [v2:172.21.15.193:6832/1023428412,v1:172.21.15.193:6833/1023428412] conn(0x55c55390a6a0 msgr2=0x55c55390cac0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.193:6832/1023428412 2024-03-24T00:19:57.798 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:57.797+0000 7fd627e00640 1 --2- 172.21.15.118:0/736842230 >> [v2:172.21.15.193:6832/1023428412,v1:172.21.15.193:6833/1023428412] conn(0x55c55390a6a0 0x55c55390cac0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 6.400000 2024-03-24T00:19:57.798 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:19:57.798+0000 7fd613e00640 1 client.26594 ms_handle_refused on v2:172.21.15.193:6832/1023428412 2024-03-24T00:19:59.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:19:58 smithi118 ceph-mon[31535]: pgmap v527: 97 pgs: 97 active+clean; 98 MiB data, 610 MiB used, 1.0 TiB / 1.0 TiB avail; 671 KiB/s rd, 885 KiB/s wr, 40 op/s 2024-03-24T00:19:59.203 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:19:58 smithi189 ceph-mon[36187]: pgmap v527: 97 pgs: 97 active+clean; 98 MiB data, 610 MiB used, 1.0 TiB / 1.0 TiB avail; 671 KiB/s rd, 885 KiB/s wr, 40 op/s 2024-03-24T00:19:59.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:19:58 smithi193 ceph-mon[36878]: pgmap v527: 97 pgs: 97 active+clean; 98 MiB data, 610 MiB used, 1.0 TiB / 1.0 TiB avail; 671 KiB/s rd, 885 KiB/s wr, 40 op/s 2024-03-24T00:20:00.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:19:59 smithi118 ceph-mon[31535]: pgmap v528: 97 pgs: 97 active+clean; 99 MiB data, 614 MiB used, 1.0 TiB / 1.0 TiB avail; 585 KiB/s rd, 886 KiB/s wr, 43 op/s 2024-03-24T00:20:00.203 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:19:59 smithi189 ceph-mon[36187]: pgmap v528: 97 pgs: 97 active+clean; 99 MiB data, 614 MiB used, 1.0 TiB / 1.0 TiB avail; 585 KiB/s rd, 886 KiB/s wr, 43 op/s 2024-03-24T00:20:00.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:19:59 smithi193 ceph-mon[36878]: pgmap v528: 97 pgs: 97 active+clean; 99 MiB data, 614 MiB used, 1.0 TiB / 1.0 TiB avail; 585 KiB/s rd, 886 KiB/s wr, 43 op/s 2024-03-24T00:20:01.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:20:00 smithi118 ceph-mon[31535]: overall HEALTH_OK 2024-03-24T00:20:01.203 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:20:00 smithi189 ceph-mon[36187]: overall HEALTH_OK 2024-03-24T00:20:01.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:20:00 smithi193 ceph-mon[36878]: overall HEALTH_OK 2024-03-24T00:20:02.718 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:20:02 smithi193 ceph-mon[36878]: pgmap v529: 97 pgs: 97 active+clean; 99 MiB data, 615 MiB used, 1.0 TiB / 1.0 TiB avail; 583 KiB/s rd, 911 KiB/s wr, 43 op/s 2024-03-24T00:20:02.718 INFO:journalctl@ceph.mds.l.smithi193.stdout:Mar 24 00:20:02 smithi193 systemd[1]: ceph-d79a5420-e971-11ee-95cd-87774f69a715@mds.l.service: Scheduled restart job, restart counter is at 1. 2024-03-24T00:20:02.718 INFO:journalctl@ceph.mds.l.smithi193.stdout:Mar 24 00:20:02 smithi193 systemd[1]: Stopped Ceph mds.l for d79a5420-e971-11ee-95cd-87774f69a715. 2024-03-24T00:20:02.718 INFO:journalctl@ceph.mds.l.smithi193.stdout:Mar 24 00:20:02 smithi193 systemd[1]: ceph-d79a5420-e971-11ee-95cd-87774f69a715@mds.l.service: Consumed 1.082s CPU time. 2024-03-24T00:20:02.718 INFO:journalctl@ceph.mds.l.smithi193.stdout:Mar 24 00:20:02 smithi193 systemd[1]: Starting Ceph mds.l for d79a5420-e971-11ee-95cd-87774f69a715... 2024-03-24T00:20:02.719 INFO:journalctl@ceph.mds.l.smithi193.stdout:Mar 24 00:20:02 smithi193 podman[70933]: 2024-03-24 00:20:02.595182715 +0000 UTC m=+0.018494181 image pull c664afe56bb01db4f9afa9532c20598e201e664f2e9ba2be50db0d52674a26d2 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc 2024-03-24T00:20:02.719 INFO:journalctl@ceph.mds.l.smithi193.stdout:Mar 24 00:20:02 smithi193 podman[70933]: 2024-03-24 00:20:02.718464222 +0000 UTC m=+0.141775666 container create 85d07539cdbb4c91a033882ae0632ed4070ba7d9cd6e7785198c28d300fe8a17 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-l, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, org.label-schema.build-date=20240319, GIT_CLEAN=True, org.label-schema.schema-version=1.0, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, ceph=True) 2024-03-24T00:20:02.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:20:02 smithi118 ceph-mon[31535]: pgmap v529: 97 pgs: 97 active+clean; 99 MiB data, 615 MiB used, 1.0 TiB / 1.0 TiB avail; 583 KiB/s rd, 911 KiB/s wr, 43 op/s 2024-03-24T00:20:02.953 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:20:02 smithi189 ceph-mon[36187]: pgmap v529: 97 pgs: 97 active+clean; 99 MiB data, 615 MiB used, 1.0 TiB / 1.0 TiB avail; 583 KiB/s rd, 911 KiB/s wr, 43 op/s 2024-03-24T00:20:02.989 INFO:journalctl@ceph.mds.l.smithi193.stdout:Mar 24 00:20:02 smithi193 podman[70933]: 2024-03-24 00:20:02.962951178 +0000 UTC m=+0.386262639 container init 85d07539cdbb4c91a033882ae0632ed4070ba7d9cd6e7785198c28d300fe8a17 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-l, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, io.buildah.version=1.33.5, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_CLEAN=True, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240319, org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-24T00:20:02.989 INFO:journalctl@ceph.mds.l.smithi193.stdout:Mar 24 00:20:02 smithi193 podman[70933]: 2024-03-24 00:20:02.965518011 +0000 UTC m=+0.388829467 container start 85d07539cdbb4c91a033882ae0632ed4070ba7d9cd6e7785198c28d300fe8a17 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-l, GIT_CLEAN=True, io.buildah.version=1.33.5, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240319, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, ceph=True, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS) 2024-03-24T00:20:03.461 INFO:journalctl@ceph.mds.l.smithi193.stdout:Mar 24 00:20:02 smithi193 ceph-mds[70951]: -- 172.21.15.193:0/3586878159 <== mon.1 v2:172.21.15.193:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x55ecb9f17a40 con 0x55ecb9eeac00 2024-03-24T00:20:03.461 INFO:journalctl@ceph.mds.l.smithi193.stdout:Mar 24 00:20:02 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-l[70947]: starting mds.l at 2024-03-24T00:20:03.461 INFO:journalctl@ceph.mds.l.smithi193.stdout:Mar 24 00:20:03 smithi193 bash[70933]: 85d07539cdbb4c91a033882ae0632ed4070ba7d9cd6e7785198c28d300fe8a17 2024-03-24T00:20:03.461 INFO:journalctl@ceph.mds.l.smithi193.stdout:Mar 24 00:20:03 smithi193 systemd[1]: Started Ceph mds.l for d79a5420-e971-11ee-95cd-87774f69a715. 2024-03-24T00:20:03.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:20:03 smithi118 ceph-mon[31535]: daemon mds.l restarted 2024-03-24T00:20:03.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:20:03 smithi118 ceph-mon[31535]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-03-24T00:20:03.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:20:03 smithi118 ceph-mon[31535]: osdmap e92: 12 total, 12 up, 12 in 2024-03-24T00:20:03.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:20:03 smithi118 ceph-mon[31535]: mds.? [v2:172.21.15.193:6832/3097400619,v1:172.21.15.193:6833/3097400619] up:boot 2024-03-24T00:20:03.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:20:03 smithi118 ceph-mon[31535]: Standby daemon mds.f assigned to filesystem cephfs as rank 3 2024-03-24T00:20:03.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:20:03 smithi118 ceph-mon[31535]: fsmap cephfs:4/5 {0=b=up:active,1=g=up:active,2=c=up:active,4=e=up:active} 5 up:standby-replay 3 up:standby, 1 failed 2024-03-24T00:20:03.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:20:03 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "l"}]: dispatch 2024-03-24T00:20:03.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:20:03 smithi118 ceph-mon[31535]: fsmap cephfs:5/5 4 up:standby-replay 3 up:standby 1 up:replay 4 up:active 2024-03-24T00:20:03.947 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:20:03.948+0000 7f6436800640 1 -- 172.21.15.118:0/3889879746 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f6430102700 msgr2=0x7f6430102b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:20:03.947 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:20:03.948+0000 7f6436800640 1 --2- 172.21.15.118:0/3889879746 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f6430102700 0x7f6430102b80 secure :-1 s=READY pgs=399 cs=0 l=1 rev1=1 crypto rx=0x7f6420009900 tx=0x7f642002bc30 comp rx=0 tx=0).stop 2024-03-24T00:20:03.948 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:20:03.948+0000 7f6436800640 1 -- 172.21.15.118:0/3889879746 shutdown_connections 2024-03-24T00:20:03.948 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:20:03.948+0000 7f6436800640 1 --2- 172.21.15.118:0/3889879746 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f64301030c0 0x7f6430109950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:20:03.948 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:20:03.948+0000 7f6436800640 1 --2- 172.21.15.118:0/3889879746 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f6430102700 0x7f6430102b80 unknown :-1 s=CLOSED pgs=399 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:20:03.948 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:20:03.948+0000 7f6436800640 1 --2- 172.21.15.118:0/3889879746 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6430101510 0x7f6430101930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:20:03.948 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:20:03.948+0000 7f6436800640 1 -- 172.21.15.118:0/3889879746 >> 172.21.15.118:0/3889879746 conn(0x7f64300fca80 msgr2=0x7f64300feec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:20:03.948 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:20:03.948+0000 7f6436800640 1 -- 172.21.15.118:0/3889879746 shutdown_connections 2024-03-24T00:20:03.948 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:20:03.948+0000 7f6436800640 1 -- 172.21.15.118:0/3889879746 wait complete. 2024-03-24T00:20:03.949 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:20:03.949+0000 7f6436800640 1 Processor -- start 2024-03-24T00:20:03.949 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:20:03.949+0000 7f6436800640 1 -- start start 2024-03-24T00:20:03.949 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:20:03.949+0000 7f6436800640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6430101510 0x7f6430199a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:20:03.949 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:20:03.949+0000 7f6436800640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f6430102700 0x7f6430199f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:20:03.949 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:20:03.949+0000 7f6436800640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f64301030c0 0x7f64301a0f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:20:03.949 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:20:03.949+0000 7f6436800640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f643010d1f0 con 0x7f6430102700 2024-03-24T00:20:03.950 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:20:03.949+0000 7f6436800640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f643010d070 con 0x7f6430101510 2024-03-24T00:20:03.950 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:20:03.949+0000 7f6436800640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f643010d370 con 0x7f64301030c0 2024-03-24T00:20:03.950 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:20:03.949+0000 7f6435400640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6430101510 0x7f6430199a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:20:03.950 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:20:03.950+0000 7f6435e00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f64301030c0 0x7f64301a0f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:20:03.950 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:20:03.950+0000 7f6434a00640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f6430102700 0x7f6430199f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:20:03.950 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:20:03.950+0000 7f6435400640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6430101510 0x7f6430199a30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.118:52454/0 (socket says 172.21.15.118:52454) 2024-03-24T00:20:03.950 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:20:03.950+0000 7f6435400640 1 -- 172.21.15.118:0/569197783 learned_addr learned my addr 172.21.15.118:0/569197783 (peer_addr_for_me v2:172.21.15.118:0/0) 2024-03-24T00:20:03.951 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:20:03.950+0000 7f6435e00640 1 -- 172.21.15.118:0/569197783 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6430101510 msgr2=0x7f6430199a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:20:03.951 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:20:03.950+0000 7f6435e00640 1 --2- 172.21.15.118:0/569197783 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6430101510 0x7f6430199a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:20:03.951 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:20:03.950+0000 7f6435e00640 1 -- 172.21.15.118:0/569197783 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f6430102700 msgr2=0x7f6430199f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:20:03.951 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:20:03.950+0000 7f6435e00640 1 --2- 172.21.15.118:0/569197783 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f6430102700 0x7f6430199f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:20:03.951 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:20:03.950+0000 7f6435e00640 1 -- 172.21.15.118:0/569197783 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f64301a1600 con 0x7f64301030c0 2024-03-24T00:20:03.951 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:20:03.950+0000 7f6435400640 1 --2- 172.21.15.118:0/569197783 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6430101510 0x7f6430199a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T00:20:03.952 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:20:03.950+0000 7f6435e00640 1 --2- 172.21.15.118:0/569197783 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f64301030c0 0x7f64301a0f90 secure :-1 s=READY pgs=400 cs=0 l=1 rev1=1 crypto rx=0x7f6428009fd0 tx=0x7f642800ef10 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:20:03.952 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:20:03.950+0000 7f642e000640 1 -- 172.21.15.118:0/569197783 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f642800ce20 con 0x7f64301030c0 2024-03-24T00:20:03.952 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:20:03.951+0000 7f6436800640 1 -- 172.21.15.118:0/569197783 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f64301a18f0 con 0x7f64301030c0 2024-03-24T00:20:03.952 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:20:03.951+0000 7f642e000640 1 -- 172.21.15.118:0/569197783 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f6428005020 con 0x7f64301030c0 2024-03-24T00:20:03.952 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:20:03.951+0000 7f642e000640 1 -- 172.21.15.118:0/569197783 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6428010720 con 0x7f64301030c0 2024-03-24T00:20:03.952 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:20:03.951+0000 7f6436800640 1 -- 172.21.15.118:0/569197783 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f64301a1e10 con 0x7f64301030c0 2024-03-24T00:20:03.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:20:03 smithi189 ceph-mon[36187]: daemon mds.l restarted 2024-03-24T00:20:03.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:20:03 smithi189 ceph-mon[36187]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-03-24T00:20:03.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:20:03 smithi189 ceph-mon[36187]: osdmap e92: 12 total, 12 up, 12 in 2024-03-24T00:20:03.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:20:03 smithi189 ceph-mon[36187]: mds.? [v2:172.21.15.193:6832/3097400619,v1:172.21.15.193:6833/3097400619] up:boot 2024-03-24T00:20:03.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:20:03 smithi189 ceph-mon[36187]: Standby daemon mds.f assigned to filesystem cephfs as rank 3 2024-03-24T00:20:03.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:20:03 smithi189 ceph-mon[36187]: fsmap cephfs:4/5 {0=b=up:active,1=g=up:active,2=c=up:active,4=e=up:active} 5 up:standby-replay 3 up:standby, 1 failed 2024-03-24T00:20:03.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:20:03 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "l"}]: dispatch 2024-03-24T00:20:03.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:20:03 smithi189 ceph-mon[36187]: fsmap cephfs:5/5 4 up:standby-replay 3 up:standby 1 up:replay 4 up:active 2024-03-24T00:20:03.956 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:20:03.955+0000 7f643aa37740 1 Processor -- start 2024-03-24T00:20:03.956 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:20:03.955+0000 7f642e000640 1 -- 172.21.15.118:0/569197783 <== mon.1 v2:172.21.15.193:3300/0 4 ==== mgrmap(e 20) v1 ==== 137291+0+0 (secure 0 0 0) 0x7f6428005540 con 0x7f64301030c0 2024-03-24T00:20:03.957 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:20:03.955+0000 7f643aa37740 1 -- start start 2024-03-24T00:20:03.957 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:20:03.956+0000 7f643aa37740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x555f4b31c3f0 0x555f4b31c810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:20:03.957 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:20:03.956+0000 7f643aa37740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x555f4b31cde0 0x555f4b3218e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:20:03.957 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:20:03.956+0000 7f643aa37740 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x555f4b321f50 0x555f4b3243e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:20:03.957 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:20:03.956+0000 7f643aa37740 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x555f4b2df3b0 con 0x555f4b321f50 2024-03-24T00:20:03.957 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:20:03.956+0000 7f643aa37740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x555f4b281830 con 0x555f4b31c3f0 2024-03-24T00:20:03.957 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:20:03.956+0000 7f643aa37740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x555f4b22a5d0 con 0x555f4b31cde0 2024-03-24T00:20:03.957 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:20:03.956+0000 7f6435400640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x555f4b31cde0 0x555f4b3218e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:20:03.957 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:20:03.956+0000 7f6434a00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x555f4b31c3f0 0x555f4b31c810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:20:03.957 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:20:03.956+0000 7f6435400640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x555f4b31cde0 0x555f4b3218e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.118:46162/0 (socket says 172.21.15.118:46162) 2024-03-24T00:20:03.957 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:20:03.956+0000 7f6435400640 1 -- 172.21.15.118:0/738310695 learned_addr learned my addr 172.21.15.118:0/738310695 (peer_addr_for_me v2:172.21.15.118:0/0) 2024-03-24T00:20:03.957 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:20:03.956+0000 7f6434a00640 1 --2- 172.21.15.118:0/738310695 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x555f4b321f50 0x555f4b3243e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:20:03.958 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:20:03.956+0000 7f6435400640 1 -- 172.21.15.118:0/738310695 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x555f4b31c3f0 msgr2=0x555f4b31c810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:20:03.958 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:20:03.956+0000 7f6435400640 1 --2- 172.21.15.118:0/738310695 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x555f4b31c3f0 0x555f4b31c810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:20:03.958 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:20:03.956+0000 7f6435400640 1 -- 172.21.15.118:0/738310695 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x555f4b321f50 msgr2=0x555f4b3243e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:20:03.958 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:20:03.956+0000 7f6435400640 1 --2- 172.21.15.118:0/738310695 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x555f4b321f50 0x555f4b3243e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:20:03.958 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:20:03.956+0000 7f6435400640 1 -- 172.21.15.118:0/738310695 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x555f4b2422c0 con 0x555f4b31cde0 2024-03-24T00:20:03.958 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:20:03.956+0000 7f6434a00640 1 --2- 172.21.15.118:0/738310695 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x555f4b321f50 0x555f4b3243e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T00:20:03.958 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:20:03.957+0000 7f6435400640 1 --2- 172.21.15.118:0/738310695 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x555f4b31cde0 0x555f4b3218e0 secure :-1 s=READY pgs=401 cs=0 l=1 rev1=1 crypto rx=0x7f64240115c0 tx=0x7f6424011a80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=1f119c66fe208e0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:20:03.958 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:20:03.957+0000 7f641e600640 1 -- 172.21.15.118:0/738310695 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6424005020 con 0x555f4b31cde0 2024-03-24T00:20:03.958 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:20:03.957+0000 7f641e600640 1 -- 172.21.15.118:0/738310695 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f64240047a0 con 0x555f4b31cde0 2024-03-24T00:20:03.958 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:20:03.957+0000 7f641e600640 1 -- 172.21.15.118:0/738310695 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f642400fa00 con 0x555f4b31cde0 2024-03-24T00:20:03.958 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:20:03.957+0000 7f642e000640 1 --2- 172.21.15.118:0/569197783 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7f64180a27f0 0x7f64180a4cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:20:03.962 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:20:03.957+0000 7f642e000640 1 -- 172.21.15.118:0/569197783 <== mon.1 v2:172.21.15.193:3300/0 5 ==== osd_map(92..92 src has 1..92) v4 ==== 8449+0+0 (secure 0 0 0) 0x7f64280cf220 con 0x7f64301030c0 2024-03-24T00:20:03.962 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:20:03.957+0000 7f6434a00640 1 --2- 172.21.15.118:0/569197783 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7f64180a27f0 0x7f64180a4cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:20:03.962 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:20:03.958+0000 7f643aa37740 1 -- 172.21.15.118:0/738310695 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x555f4b31cde0 msgr2=0x555f4b3218e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:20:03.962 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:20:03.958+0000 7f643aa37740 1 --2- 172.21.15.118:0/738310695 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x555f4b31cde0 0x555f4b3218e0 secure :-1 s=READY pgs=401 cs=0 l=1 rev1=1 crypto rx=0x7f64240115c0 tx=0x7f6424011a80 comp rx=0 tx=0).stop 2024-03-24T00:20:03.962 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:20:03.958+0000 7f643aa37740 1 -- 172.21.15.118:0/738310695 shutdown_connections 2024-03-24T00:20:03.962 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:20:03.958+0000 7f643aa37740 1 --2- 172.21.15.118:0/738310695 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x555f4b321f50 0x555f4b3243e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:20:03.962 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:20:03.958+0000 7f643aa37740 1 --2- 172.21.15.118:0/738310695 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x555f4b31cde0 0x555f4b3218e0 unknown :-1 s=CLOSED pgs=401 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:20:03.962 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:20:03.958+0000 7f643aa37740 1 --2- 172.21.15.118:0/738310695 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x555f4b31c3f0 0x555f4b31c810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:20:03.963 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:20:03.958+0000 7f643aa37740 1 -- 172.21.15.118:0/738310695 >> 172.21.15.118:0/738310695 conn(0x555f4b317960 msgr2=0x555f4b317d60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:20:03.963 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:20:03.958+0000 7f643aa37740 1 -- 172.21.15.118:0/738310695 shutdown_connections 2024-03-24T00:20:03.963 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:20:03.958+0000 7f6434a00640 1 --2- 172.21.15.118:0/569197783 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7f64180a27f0 0x7f64180a4cb0 secure :-1 s=READY pgs=347 cs=0 l=1 rev1=1 crypto rx=0x7f642002c240 tx=0x7f642002df90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:20:03.963 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:20:03.958+0000 7f643aa37740 1 -- 172.21.15.118:0/738310695 wait complete. 2024-03-24T00:20:03.963 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:20:03.959+0000 7f643aa37740 1 -- start start 2024-03-24T00:20:03.963 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:20:03.959+0000 7f643aa37740 1 Processor -- start 2024-03-24T00:20:03.963 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:20:03.960+0000 7f643aa37740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x555f4b31c3f0 0x555f4b326610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:20:03.963 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:20:03.960+0000 7f643aa37740 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x555f4b31cde0 0x555f4b326b50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:20:03.963 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:20:03.960+0000 7f643aa37740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x555f4b321f50 0x555f4b32db20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:20:03.963 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:20:03.960+0000 7f643aa37740 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x555f4b23da20 con 0x555f4b31cde0 2024-03-24T00:20:03.963 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:20:03.960+0000 7f643aa37740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x555f4b27d8d0 con 0x555f4b31c3f0 2024-03-24T00:20:03.963 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:20:03.960+0000 7f643aa37740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x555f4b324f00 con 0x555f4b321f50 2024-03-24T00:20:03.963 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:20:03.960+0000 7f643aa37740 10 client.0 _ll_register_callbacks cb 0x555f4b27a1c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T00:20:03.963 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:20:03.960+0000 7f6435400640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x555f4b31c3f0 0x555f4b326610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:20:03.964 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:20:03.960+0000 7f6435400640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x555f4b31c3f0 0x555f4b326610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.118:52462/0 (socket says 172.21.15.118:52462) 2024-03-24T00:20:03.964 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:20:03.960+0000 7f6435400640 1 -- 172.21.15.118:0/3860074578 learned_addr learned my addr 172.21.15.118:0/3860074578 (peer_addr_for_me v2:172.21.15.118:0/0) 2024-03-24T00:20:03.964 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:20:03.960+0000 7f6434a00640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x555f4b31cde0 0x555f4b326b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:20:03.964 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:20:03.960+0000 7f6435e00640 1 --2- 172.21.15.118:0/3860074578 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x555f4b321f50 0x555f4b32db20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:20:03.964 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:20:03.960+0000 7f6435400640 1 -- 172.21.15.118:0/3860074578 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x555f4b321f50 msgr2=0x555f4b32db20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:20:03.964 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:20:03.960+0000 7f6435400640 1 --2- 172.21.15.118:0/3860074578 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x555f4b321f50 0x555f4b32db20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:20:03.964 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:20:03.960+0000 7f6435400640 1 -- 172.21.15.118:0/3860074578 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x555f4b31cde0 msgr2=0x555f4b326b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:20:03.964 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:20:03.960+0000 7f6435400640 1 --2- 172.21.15.118:0/3860074578 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x555f4b31cde0 0x555f4b326b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:20:03.964 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:20:03.960+0000 7f6435400640 1 -- 172.21.15.118:0/3860074578 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x555f4af96210 con 0x555f4b31c3f0 2024-03-24T00:20:03.964 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:20:03.961+0000 7f6435400640 1 --2- 172.21.15.118:0/3860074578 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x555f4b31c3f0 0x555f4b326610 secure :-1 s=READY pgs=402 cs=0 l=1 rev1=1 crypto rx=0x7f642401b040 tx=0x7f6424005230 comp rx=0 tx=0).ready entity=mon.2 client_cookie=26018442431eab5c server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:20:03.964 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:20:03.961+0000 7f6412a00640 10 client.0 ms_handle_connect on v2:172.21.15.189:3300/0 2024-03-24T00:20:03.964 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:20:03.961+0000 7f6412a00640 1 -- 172.21.15.118:0/3860074578 <== mon.2 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6424028020 con 0x555f4b31c3f0 2024-03-24T00:20:03.964 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:20:03.961+0000 7f6412a00640 1 -- 172.21.15.118:0/3860074578 <== mon.2 v2:172.21.15.189:3300/0 2 ==== osd_map(92..92 src has 1..92) v4 ==== 8449+0+0 (secure 0 0 0) 0x7f6424005420 con 0x555f4b31c3f0 2024-03-24T00:20:03.964 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:20:03.961+0000 7f643aa37740 1 -- 172.21.15.118:0/3860074578 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f64240047a0 con 0x555f4b31c3f0 2024-03-24T00:20:03.965 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:20:03.961+0000 7f6412a00640 1 -- 172.21.15.118:0/3860074578 <== mon.2 v2:172.21.15.189:3300/0 3 ==== mon_get_version_reply(handle=1 version=38) v2 ==== 24+0+0 (secure 0 0 0) 0x7f6424033050 con 0x555f4b31c3f0 2024-03-24T00:20:03.965 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:20:03.961+0000 7f643aa37740 10 client.26612 fetch_fsmap learned FSMap version 38 2024-03-24T00:20:03.965 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:20:03.961+0000 7f643aa37740 1 -- 172.21.15.118:0/3860074578 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,fsmap=38,monmap=4+}) v3 -- 0x555f4b062620 con 0x555f4b31c3f0 2024-03-24T00:20:03.965 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:20:03.961+0000 7f6412a00640 1 -- 172.21.15.118:0/3860074578 <== mon.2 v2:172.21.15.189:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f64240047a0 con 0x555f4b31c3f0 2024-03-24T00:20:03.965 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:20:03.962+0000 7f6412a00640 1 -- 172.21.15.118:0/3860074578 <== mon.2 v2:172.21.15.189:3300/0 5 ==== fsmap(e 38) v1 ==== 7942+0+0 (secure 0 0 0) 0x7f6424023600 con 0x555f4b31c3f0 2024-03-24T00:20:03.965 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:20:03.962+0000 7f643aa37740 10 client.26612 fetch_fsmap finished waiting for FSMap version 38 2024-03-24T00:20:03.965 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:20:03.962+0000 7f643aa37740 10 client.26612 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-24T00:20:03.965 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:20:03.962+0000 7f643aa37740 20 client.26612 populate_metadata read hostname 'smithi118' 2024-03-24T00:20:03.965 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:20:03.962+0000 7f643aa37740 1 --2- 172.21.15.118:0/3860074578 >> [v2:172.21.15.189:6838/4074484168,v1:172.21.15.189:6839/4074484168] conn(0x555f4b3321c0 0x555f4b3345d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:20:03.965 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:20:03.962+0000 7f643aa37740 4 client.26612 mds_command: new command op to 24569 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-24T00:20:03.965 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:20:03.962+0000 7f643aa37740 1 -- 172.21.15.118:0/3860074578 --> [v2:172.21.15.189:6838/4074484168,v1:172.21.15.189:6839/4074484168] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x555f4b28ed70 con 0x555f4b3321c0 2024-03-24T00:20:03.965 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:20:03.962+0000 7f6434a00640 1 --2- 172.21.15.118:0/3860074578 >> [v2:172.21.15.189:6838/4074484168,v1:172.21.15.189:6839/4074484168] conn(0x555f4b3321c0 0x555f4b3345d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:20:03.965 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:20:03.963+0000 7f6434a00640 1 --2- 172.21.15.118:0/3860074578 >> [v2:172.21.15.189:6838/4074484168,v1:172.21.15.189:6839/4074484168] conn(0x555f4b3321c0 0x555f4b3345d0 crc :-1 s=READY pgs=135 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=ad19974421d04467 server_cookie=a8cd6324e51d5209 in_seq=0 out_seq=0 2024-03-24T00:20:03.965 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:20:03.963+0000 7f6412a00640 10 client.26612 ms_handle_connect on v2:172.21.15.189:6838/4074484168 2024-03-24T00:20:03.966 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:20:03 smithi193 ceph-mon[36878]: daemon mds.l restarted 2024-03-24T00:20:03.966 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:20:03 smithi193 ceph-mon[36878]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-03-24T00:20:03.966 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:20:03 smithi193 ceph-mon[36878]: osdmap e92: 12 total, 12 up, 12 in 2024-03-24T00:20:03.966 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:20:03 smithi193 ceph-mon[36878]: mds.? [v2:172.21.15.193:6832/3097400619,v1:172.21.15.193:6833/3097400619] up:boot 2024-03-24T00:20:03.966 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:20:03 smithi193 ceph-mon[36878]: Standby daemon mds.f assigned to filesystem cephfs as rank 3 2024-03-24T00:20:03.966 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:20:03 smithi193 ceph-mon[36878]: fsmap cephfs:4/5 {0=b=up:active,1=g=up:active,2=c=up:active,4=e=up:active} 5 up:standby-replay 3 up:standby, 1 failed 2024-03-24T00:20:03.967 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:20:03 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "l"}]: dispatch 2024-03-24T00:20:03.967 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:20:03 smithi193 ceph-mon[36878]: fsmap cephfs:5/5 4 up:standby-replay 3 up:standby 1 up:replay 4 up:active 2024-03-24T00:20:03.967 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:20:03.963+0000 7f6412a00640 1 -- 172.21.15.118:0/3860074578 <== mds.0 v2:172.21.15.189:6838/4074484168 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x555f4b28ed70 con 0x555f4b3321c0 2024-03-24T00:20:03.968 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:20:03.963+0000 7f6412a00640 10 client.26612 handle_command_reply: tid=0 2024-03-24T00:20:03.968 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:20:03.963+0000 7f643aa37740 1 client.26612 shutdown 2024-03-24T00:20:03.968 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:20:03.964+0000 7f643aa37740 1 -- 172.21.15.118:0/3860074578 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x555f4b31c3f0 msgr2=0x555f4b326610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:20:03.968 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:20:03.964+0000 7f643aa37740 1 --2- 172.21.15.118:0/3860074578 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x555f4b31c3f0 0x555f4b326610 secure :-1 s=READY pgs=402 cs=0 l=1 rev1=1 crypto rx=0x7f642401b040 tx=0x7f6424005230 comp rx=0 tx=0).stop 2024-03-24T00:20:03.968 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:20:03.964+0000 7f643aa37740 1 -- 172.21.15.118:0/3860074578 shutdown_connections 2024-03-24T00:20:03.968 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:20:03.964+0000 7f643aa37740 1 --2- 172.21.15.118:0/3860074578 >> [v2:172.21.15.189:6838/4074484168,v1:172.21.15.189:6839/4074484168] conn(0x555f4b3321c0 0x555f4b3345d0 crc :-1 s=READY pgs=135 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:20:03.968 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:20:03.964+0000 7f643aa37740 1 --2- 172.21.15.118:0/3860074578 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x555f4b321f50 0x555f4b32db20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:20:03.968 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:20:03.964+0000 7f643aa37740 1 --2- 172.21.15.118:0/3860074578 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x555f4b31cde0 0x555f4b326b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:20:03.968 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:20:03.964+0000 7f643aa37740 1 --2- 172.21.15.118:0/3860074578 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x555f4b31c3f0 0x555f4b326610 unknown :-1 s=CLOSED pgs=402 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:20:03.968 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:20:03.964+0000 7f6412a00640 0 client.26612 ms_handle_reset on v2:172.21.15.189:6838/4074484168 2024-03-24T00:20:03.968 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:20:03.964+0000 7f643aa37740 1 -- 172.21.15.118:0/3860074578 >> 172.21.15.118:0/3860074578 conn(0x555f4b317960 msgr2=0x555f4b316b70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:20:03.968 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:20:03.964+0000 7f643aa37740 1 -- 172.21.15.118:0/3860074578 shutdown_connections 2024-03-24T00:20:03.968 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:20:03.964+0000 7f643aa37740 1 -- 172.21.15.118:0/3860074578 wait complete. 2024-03-24T00:20:03.968 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:20:03.964+0000 7f643aa37740 20 client.26612 trim_cache size 0 max 16384 2024-03-24T00:20:03.987 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:20:03.987+0000 7f643aa37740 1 Processor -- start 2024-03-24T00:20:03.987 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:20:03.987+0000 7f643aa37740 1 -- start start 2024-03-24T00:20:03.987 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:20:03.987+0000 7f643aa37740 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x555f4b31c3f0 0x555f4b317d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:20:03.987 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:20:03.987+0000 7f643aa37740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x555f4b31cde0 0x555f4b32dae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:20:03.987 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:20:03.987+0000 7f643aa37740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x555f4b321f50 0x555f4b338ed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:20:03.987 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:20:03.987+0000 7f643aa37740 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f64301ab330 con 0x555f4b31c3f0 2024-03-24T00:20:03.988 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:20:03.987+0000 7f643aa37740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f64301ac680 con 0x555f4b321f50 2024-03-24T00:20:03.988 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:20:03.987+0000 7f643aa37740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f64301ab990 con 0x555f4b31cde0 2024-03-24T00:20:03.988 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:20:03.987+0000 7f6435e00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x555f4b321f50 0x555f4b338ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:20:03.988 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:20:03.987+0000 7f6435400640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x555f4b31c3f0 0x555f4b317d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:20:03.988 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:20:03.987+0000 7f6435e00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x555f4b321f50 0x555f4b338ed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.118:52466/0 (socket says 172.21.15.118:52466) 2024-03-24T00:20:03.988 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:20:03.987+0000 7f6435400640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x555f4b31c3f0 0x555f4b317d40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.118:3300/0 says I am v2:172.21.15.118:40708/0 (socket says 172.21.15.118:40708) 2024-03-24T00:20:03.988 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:20:03.987+0000 7f6435e00640 1 -- 172.21.15.118:0/4165540502 learned_addr learned my addr 172.21.15.118:0/4165540502 (peer_addr_for_me v2:172.21.15.118:0/0) 2024-03-24T00:20:03.988 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:20:03.987+0000 7f6434a00640 1 --2- 172.21.15.118:0/4165540502 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x555f4b31cde0 0x555f4b32dae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:20:03.988 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:20:03.988+0000 7f6435e00640 1 -- 172.21.15.118:0/4165540502 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x555f4b31cde0 msgr2=0x555f4b32dae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:20:03.988 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:20:03.988+0000 7f6435e00640 1 --2- 172.21.15.118:0/4165540502 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x555f4b31cde0 0x555f4b32dae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:20:03.988 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:20:03.988+0000 7f6435e00640 1 -- 172.21.15.118:0/4165540502 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x555f4b31c3f0 msgr2=0x555f4b317d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:20:03.988 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:20:03.988+0000 7f6435e00640 1 --2- 172.21.15.118:0/4165540502 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x555f4b31c3f0 0x555f4b317d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:20:03.988 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:20:03.988+0000 7f6435e00640 1 -- 172.21.15.118:0/4165540502 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x555f4b315f80 con 0x555f4b321f50 2024-03-24T00:20:03.988 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:20:03.988+0000 7f6435400640 1 --2- 172.21.15.118:0/4165540502 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x555f4b31c3f0 0x555f4b317d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T00:20:03.992 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:20:03.988+0000 7f6435e00640 1 --2- 172.21.15.118:0/4165540502 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x555f4b321f50 0x555f4b338ed0 secure :-1 s=READY pgs=403 cs=0 l=1 rev1=1 crypto rx=0x7f642809ba20 tx=0x7f64280ccb10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=b25dbe1e87655477 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:20:03.992 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:20:03.988+0000 7f641dc00640 1 -- 172.21.15.118:0/4165540502 <== mon.2 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f64280b9070 con 0x555f4b321f50 2024-03-24T00:20:03.992 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:20:03.988+0000 7f641dc00640 1 -- 172.21.15.118:0/4165540502 <== mon.2 v2:172.21.15.189:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f6428099b80 con 0x555f4b321f50 2024-03-24T00:20:03.992 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:20:03.988+0000 7f641dc00640 1 -- 172.21.15.118:0/4165540502 <== mon.2 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f64280b3030 con 0x555f4b321f50 2024-03-24T00:20:03.992 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:20:03.989+0000 7f643aa37740 1 -- 172.21.15.118:0/4165540502 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x555f4b321f50 msgr2=0x555f4b338ed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:20:03.992 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:20:03.989+0000 7f643aa37740 1 --2- 172.21.15.118:0/4165540502 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x555f4b321f50 0x555f4b338ed0 secure :-1 s=READY pgs=403 cs=0 l=1 rev1=1 crypto rx=0x7f642809ba20 tx=0x7f64280ccb10 comp rx=0 tx=0).stop 2024-03-24T00:20:03.992 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:20:03.989+0000 7f643aa37740 1 -- 172.21.15.118:0/4165540502 shutdown_connections 2024-03-24T00:20:03.992 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:20:03.989+0000 7f643aa37740 1 --2- 172.21.15.118:0/4165540502 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x555f4b321f50 0x555f4b338ed0 unknown :-1 s=CLOSED pgs=403 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:20:03.992 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:20:03.989+0000 7f643aa37740 1 --2- 172.21.15.118:0/4165540502 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x555f4b31cde0 0x555f4b32dae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:20:03.992 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:20:03.989+0000 7f643aa37740 1 --2- 172.21.15.118:0/4165540502 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x555f4b31c3f0 0x555f4b317d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:20:03.992 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:20:03.989+0000 7f643aa37740 1 -- 172.21.15.118:0/4165540502 >> 172.21.15.118:0/4165540502 conn(0x555f4b317960 msgr2=0x555f4b31b450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:20:03.992 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:20:03.989+0000 7f643aa37740 1 -- 172.21.15.118:0/4165540502 shutdown_connections 2024-03-24T00:20:03.993 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:20:03.989+0000 7f643aa37740 1 -- 172.21.15.118:0/4165540502 wait complete. 2024-03-24T00:20:03.993 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:20:03.989+0000 7f643aa37740 1 -- start start 2024-03-24T00:20:03.993 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:20:03.989+0000 7f643aa37740 1 Processor -- start 2024-03-24T00:20:03.993 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:20:03.990+0000 7f643aa37740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x555f4b31c3f0 0x555f4b32b5a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:20:03.993 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:20:03.990+0000 7f643aa37740 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x555f4b31cde0 0x555f4b32bae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:20:03.993 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:20:03.990+0000 7f643aa37740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x555f4b321f50 0x555f4b32caf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:20:03.993 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:20:03.990+0000 7f643aa37740 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f64301a9ab0 con 0x555f4b31cde0 2024-03-24T00:20:03.993 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:20:03.990+0000 7f643aa37740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f64301ac020 con 0x555f4b321f50 2024-03-24T00:20:03.993 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:20:03.990+0000 7f643aa37740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f64301a8e70 con 0x555f4b31c3f0 2024-03-24T00:20:03.993 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:20:03.990+0000 7f643aa37740 10 client.0 _ll_register_callbacks cb 0x7f6424005e30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T00:20:03.993 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:20:03.990+0000 7f6435400640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x555f4b31c3f0 0x555f4b32b5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:20:03.993 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:20:03.990+0000 7f6434a00640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x555f4b31cde0 0x555f4b32bae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:20:03.993 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:20:03.990+0000 7f6435400640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x555f4b31c3f0 0x555f4b32b5a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.118:46184/0 (socket says 172.21.15.118:46184) 2024-03-24T00:20:03.994 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:20:03.990+0000 7f6435400640 1 -- 172.21.15.118:0/231389577 learned_addr learned my addr 172.21.15.118:0/231389577 (peer_addr_for_me v2:172.21.15.118:0/0) 2024-03-24T00:20:03.994 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:20:03.990+0000 7f6435e00640 1 --2- 172.21.15.118:0/231389577 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x555f4b321f50 0x555f4b32caf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:20:03.994 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:20:03.990+0000 7f6435400640 1 -- 172.21.15.118:0/231389577 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x555f4b321f50 msgr2=0x555f4b32caf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:20:03.994 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:20:03.990+0000 7f6435400640 1 --2- 172.21.15.118:0/231389577 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x555f4b321f50 0x555f4b32caf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:20:03.994 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:20:03.990+0000 7f6435400640 1 -- 172.21.15.118:0/231389577 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x555f4b31cde0 msgr2=0x555f4b32bae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:20:03.994 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:20:03.990+0000 7f6435400640 1 --2- 172.21.15.118:0/231389577 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x555f4b31cde0 0x555f4b32bae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:20:03.994 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:20:03.990+0000 7f6435400640 1 -- 172.21.15.118:0/231389577 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x555f4b32d0e0 con 0x555f4b31c3f0 2024-03-24T00:20:03.994 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:20:03.990+0000 7f6435e00640 1 --2- 172.21.15.118:0/231389577 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x555f4b321f50 0x555f4b32caf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T00:20:03.994 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:20:03.990+0000 7f6434a00640 1 --2- 172.21.15.118:0/231389577 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x555f4b31cde0 0x555f4b32bae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T00:20:03.994 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:20:03.991+0000 7f6435400640 1 --2- 172.21.15.118:0/231389577 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x555f4b31c3f0 0x555f4b32b5a0 secure :-1 s=READY pgs=402 cs=0 l=1 rev1=1 crypto rx=0x7f6424001eb0 tx=0x7f6424009e40 comp rx=0 tx=0).ready entity=mon.1 client_cookie=e1182448eb8f4f7b server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:20:03.994 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:20:03.991+0000 7f6412a00640 10 client.0 ms_handle_connect on v2:172.21.15.193:3300/0 2024-03-24T00:20:03.994 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:20:03.991+0000 7f6412a00640 1 -- 172.21.15.118:0/231389577 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6424028070 con 0x555f4b31c3f0 2024-03-24T00:20:03.994 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:20:03.991+0000 7f643aa37740 1 -- 172.21.15.118:0/231389577 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f6428099b80 con 0x555f4b31c3f0 2024-03-24T00:20:03.995 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:20:03.991+0000 7f6412a00640 1 -- 172.21.15.118:0/231389577 <== mon.1 v2:172.21.15.193:3300/0 2 ==== osd_map(92..92 src has 1..92) v4 ==== 8449+0+0 (secure 0 0 0) 0x7f6424011070 con 0x555f4b31c3f0 2024-03-24T00:20:03.995 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:20:03.991+0000 7f6412a00640 1 -- 172.21.15.118:0/231389577 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_get_version_reply(handle=1 version=38) v2 ==== 24+0+0 (secure 0 0 0) 0x7f642401d040 con 0x555f4b31c3f0 2024-03-24T00:20:03.995 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:20:03.991+0000 7f643aa37740 10 client.26632 fetch_fsmap learned FSMap version 38 2024-03-24T00:20:03.995 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:20:03.991+0000 7f643aa37740 1 -- 172.21.15.118:0/231389577 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,fsmap=38,monmap=4+}) v3 -- 0x555f4b396680 con 0x555f4b31c3f0 2024-03-24T00:20:03.995 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:20:03.992+0000 7f6412a00640 1 -- 172.21.15.118:0/231389577 <== mon.1 v2:172.21.15.193:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f6428099b80 con 0x555f4b31c3f0 2024-03-24T00:20:03.995 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:20:03.992+0000 7f6412a00640 1 -- 172.21.15.118:0/231389577 <== mon.1 v2:172.21.15.193:3300/0 5 ==== fsmap(e 38) v1 ==== 7942+0+0 (secure 0 0 0) 0x7f6424004040 con 0x555f4b31c3f0 2024-03-24T00:20:03.995 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:20:03.992+0000 7f643aa37740 10 client.26632 fetch_fsmap finished waiting for FSMap version 38 2024-03-24T00:20:03.995 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:20:03.992+0000 7f643aa37740 10 client.26632 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-24T00:20:03.995 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:20:03.992+0000 7f643aa37740 20 client.26632 populate_metadata read hostname 'smithi118' 2024-03-24T00:20:03.995 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:20:03.992+0000 7f643aa37740 1 --2- 172.21.15.118:0/231389577 >> [v2:172.21.15.189:6838/4074484168,v1:172.21.15.189:6839/4074484168] conn(0x555f4b396f70 0x555f4b399390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:20:03.995 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:20:03.992+0000 7f643aa37740 4 client.26632 mds_command: new command op to 24569 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-03-24T00:20:03.995 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:20:03.992+0000 7f643aa37740 1 -- 172.21.15.118:0/231389577 --> [v2:172.21.15.189:6838/4074484168,v1:172.21.15.189:6839/4074484168] -- command(tid 0: {"prefix": "flush journal"}) v1 -- 0x555f4b2890c0 con 0x555f4b396f70 2024-03-24T00:20:03.995 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:20:03.992+0000 7f6434a00640 1 --2- 172.21.15.118:0/231389577 >> [v2:172.21.15.189:6838/4074484168,v1:172.21.15.189:6839/4074484168] conn(0x555f4b396f70 0x555f4b399390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:20:03.996 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:20:03.993+0000 7f6412a00640 10 client.26632 ms_handle_connect on v2:172.21.15.189:6838/4074484168 2024-03-24T00:20:03.996 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:20:03.993+0000 7f6434a00640 1 --2- 172.21.15.118:0/231389577 >> [v2:172.21.15.189:6838/4074484168,v1:172.21.15.189:6839/4074484168] conn(0x555f4b396f70 0x555f4b399390 crc :-1 s=READY pgs=136 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=bf66dff6f85af3eb server_cookie=2f6572169cb573a in_seq=0 out_seq=0 2024-03-24T00:20:04.204 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:20:04.204+0000 7fd627e00640 1 --2- 172.21.15.118:0/736842230 >> [v2:172.21.15.193:6832/1023428412,v1:172.21.15.193:6833/1023428412] conn(0x55c55390a6a0 0x55c55390cac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:20:04.205 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:20:04.205+0000 7fd627e00640 1 -- 172.21.15.118:0/736842230 >> [v2:172.21.15.193:6832/1023428412,v1:172.21.15.193:6833/1023428412] conn(0x55c55390a6a0 msgr2=0x55c55390cac0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-03-24T00:20:04.205 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:20:04.205+0000 7fd627e00640 1 -- 172.21.15.118:0/736842230 >> [v2:172.21.15.193:6832/1023428412,v1:172.21.15.193:6833/1023428412] conn(0x55c55390a6a0 msgr2=0x55c55390cac0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-03-24T00:20:04.205 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:20:04.205+0000 7fd627e00640 1 --2- 172.21.15.118:0/736842230 >> [v2:172.21.15.193:6832/1023428412,v1:172.21.15.193:6833/1023428412] conn(0x55c55390a6a0 0x55c55390cac0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-03-24T00:20:04.205 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:20:04.205+0000 7fd627e00640 1 --2- 172.21.15.118:0/736842230 >> [v2:172.21.15.193:6832/1023428412,v1:172.21.15.193:6833/1023428412] conn(0x55c55390a6a0 0x55c55390cac0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 12.800000 2024-03-24T00:20:04.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:20:04 smithi118 ceph-mon[31535]: pgmap v531: 97 pgs: 97 active+clean; 99 MiB data, 615 MiB used, 1.0 TiB / 1.0 TiB avail; 66 KiB/s rd, 463 KiB/s wr, 45 op/s 2024-03-24T00:20:04.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:20:04 smithi118 ceph-mon[31535]: mds.? [v2:172.21.15.193:6834/3744499911,v1:172.21.15.193:6835/3744499911] up:resolve 2024-03-24T00:20:04.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:20:04 smithi118 ceph-mon[31535]: fsmap cephfs:5/5 4 up:standby-replay 3 up:standby 1 up:resolve 4 up:active 2024-03-24T00:20:04.953 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:20:04 smithi189 ceph-mon[36187]: pgmap v531: 97 pgs: 97 active+clean; 99 MiB data, 615 MiB used, 1.0 TiB / 1.0 TiB avail; 66 KiB/s rd, 463 KiB/s wr, 45 op/s 2024-03-24T00:20:04.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:20:04 smithi189 ceph-mon[36187]: mds.? [v2:172.21.15.193:6834/3744499911,v1:172.21.15.193:6835/3744499911] up:resolve 2024-03-24T00:20:04.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:20:04 smithi189 ceph-mon[36187]: fsmap cephfs:5/5 4 up:standby-replay 3 up:standby 1 up:resolve 4 up:active 2024-03-24T00:20:04.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:20:04 smithi193 ceph-mon[36878]: pgmap v531: 97 pgs: 97 active+clean; 99 MiB data, 615 MiB used, 1.0 TiB / 1.0 TiB avail; 66 KiB/s rd, 463 KiB/s wr, 45 op/s 2024-03-24T00:20:04.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:20:04 smithi193 ceph-mon[36878]: mds.? [v2:172.21.15.193:6834/3744499911,v1:172.21.15.193:6835/3744499911] up:resolve 2024-03-24T00:20:04.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:20:04 smithi193 ceph-mon[36878]: fsmap cephfs:5/5 4 up:standby-replay 3 up:standby 1 up:resolve 4 up:active 2024-03-24T00:20:06.089 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:20:06.089+0000 7f6412a00640 1 -- 172.21.15.118:0/231389577 <== mds.0 v2:172.21.15.189:6838/4074484168 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+44 (crc 0 0 0) 0x555f4b2890c0 con 0x555f4b396f70 2024-03-24T00:20:06.089 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:20:06.089+0000 7f6412a00640 10 client.26632 handle_command_reply: tid=0 2024-03-24T00:20:06.089 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:20:06.089+0000 7f643aa37740 1 client.26632 shutdown 2024-03-24T00:20:06.090 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:20:06.090+0000 7f643aa37740 1 -- 172.21.15.118:0/231389577 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x555f4b31c3f0 msgr2=0x555f4b32b5a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:20:06.091 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:20:06.090+0000 7f643aa37740 1 --2- 172.21.15.118:0/231389577 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x555f4b31c3f0 0x555f4b32b5a0 secure :-1 s=READY pgs=402 cs=0 l=1 rev1=1 crypto rx=0x7f6424001eb0 tx=0x7f6424009e40 comp rx=0 tx=0).stop 2024-03-24T00:20:06.091 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:20:06.090+0000 7f643aa37740 1 -- 172.21.15.118:0/231389577 shutdown_connections 2024-03-24T00:20:06.091 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:20:06.090+0000 7f643aa37740 1 --2- 172.21.15.118:0/231389577 >> [v2:172.21.15.189:6838/4074484168,v1:172.21.15.189:6839/4074484168] conn(0x555f4b396f70 0x555f4b399390 crc :-1 s=READY pgs=136 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:20:06.091 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:20:06.090+0000 7f643aa37740 1 --2- 172.21.15.118:0/231389577 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x555f4b321f50 0x555f4b32caf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:20:06.091 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:20:06.090+0000 7f643aa37740 1 --2- 172.21.15.118:0/231389577 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x555f4b31cde0 0x555f4b32bae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:20:06.091 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:20:06.090+0000 7f643aa37740 1 --2- 172.21.15.118:0/231389577 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x555f4b31c3f0 0x555f4b32b5a0 unknown :-1 s=CLOSED pgs=402 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:20:06.091 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:20:06.090+0000 7f643aa37740 1 -- 172.21.15.118:0/231389577 >> 172.21.15.118:0/231389577 conn(0x555f4b317960 msgr2=0x555f4b31b450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:20:06.091 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:20:06.090+0000 7f6412a00640 0 client.26632 ms_handle_reset on v2:172.21.15.189:6838/4074484168 2024-03-24T00:20:06.091 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:20:06.090+0000 7f643aa37740 1 -- 172.21.15.118:0/231389577 shutdown_connections 2024-03-24T00:20:06.091 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:20:06.090+0000 7f643aa37740 1 -- 172.21.15.118:0/231389577 wait complete. 2024-03-24T00:20:06.091 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:20:06.090+0000 7f643aa37740 20 client.26632 trim_cache size 0 max 16384 2024-03-24T00:20:06.092 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stdout:{ 2024-03-24T00:20:06.092 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stdout: "message": "", 2024-03-24T00:20:06.092 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stdout: "return_code": 0 2024-03-24T00:20:06.092 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stdout:} 2024-03-24T00:20:06.093 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:20:06.092+0000 7f6436800640 1 -- 172.21.15.118:0/569197783 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7f64180a27f0 msgr2=0x7f64180a4cb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:20:06.093 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:20:06.092+0000 7f6436800640 1 --2- 172.21.15.118:0/569197783 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7f64180a27f0 0x7f64180a4cb0 secure :-1 s=READY pgs=347 cs=0 l=1 rev1=1 crypto rx=0x7f642002c240 tx=0x7f642002df90 comp rx=0 tx=0).stop 2024-03-24T00:20:06.093 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:20:06.092+0000 7f6436800640 1 -- 172.21.15.118:0/569197783 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f64301030c0 msgr2=0x7f64301a0f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:20:06.093 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:20:06.092+0000 7f6436800640 1 --2- 172.21.15.118:0/569197783 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f64301030c0 0x7f64301a0f90 secure :-1 s=READY pgs=400 cs=0 l=1 rev1=1 crypto rx=0x7f6428009fd0 tx=0x7f642800ef10 comp rx=0 tx=0).stop 2024-03-24T00:20:06.093 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:20:06.092+0000 7f6436800640 1 -- 172.21.15.118:0/569197783 shutdown_connections 2024-03-24T00:20:06.093 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:20:06.092+0000 7f6436800640 1 --2- 172.21.15.118:0/569197783 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7f64180a27f0 0x7f64180a4cb0 unknown :-1 s=CLOSED pgs=347 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:20:06.094 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:20:06.092+0000 7f6436800640 1 --2- 172.21.15.118:0/569197783 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f64301030c0 0x7f64301a0f90 unknown :-1 s=CLOSED pgs=400 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:20:06.094 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:20:06.092+0000 7f6436800640 1 --2- 172.21.15.118:0/569197783 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f6430102700 0x7f6430199f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:20:06.094 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:20:06.092+0000 7f6436800640 1 --2- 172.21.15.118:0/569197783 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f6430101510 0x7f6430199a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:20:06.094 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:20:06.092+0000 7f6436800640 1 -- 172.21.15.118:0/569197783 >> 172.21.15.118:0/569197783 conn(0x7f64300fca80 msgr2=0x7f64300738d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:20:06.094 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:20:06.092+0000 7f6436800640 1 -- 172.21.15.118:0/569197783 shutdown_connections 2024-03-24T00:20:06.094 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:20:06.092+0000 7f6436800640 1 -- 172.21.15.118:0/569197783 wait complete. 2024-03-24T00:20:06.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:20:06 smithi118 ceph-mon[31535]: mds.? [v2:172.21.15.193:6834/3744499911,v1:172.21.15.193:6835/3744499911] up:reconnect 2024-03-24T00:20:06.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:20:06 smithi118 ceph-mon[31535]: fsmap cephfs:5/5 4 up:standby-replay 3 up:standby 1 up:reconnect 4 up:active 2024-03-24T00:20:06.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:20:06 smithi118 ceph-mon[31535]: reconnect by client.15159 192.168.0.1:0/3463739313 after 0.0109998 2024-03-24T00:20:06.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:20:06 smithi118 ceph-mon[31535]: pgmap v532: 97 pgs: 97 active+clean; 99 MiB data, 615 MiB used, 1.0 TiB / 1.0 TiB avail; 61 KiB/s rd, 411 KiB/s wr, 42 op/s 2024-03-24T00:20:06.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:20:06 smithi189 ceph-mon[36187]: mds.? [v2:172.21.15.193:6834/3744499911,v1:172.21.15.193:6835/3744499911] up:reconnect 2024-03-24T00:20:06.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:20:06 smithi189 ceph-mon[36187]: fsmap cephfs:5/5 4 up:standby-replay 3 up:standby 1 up:reconnect 4 up:active 2024-03-24T00:20:06.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:20:06 smithi189 ceph-mon[36187]: reconnect by client.15159 192.168.0.1:0/3463739313 after 0.0109998 2024-03-24T00:20:06.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:20:06 smithi189 ceph-mon[36187]: pgmap v532: 97 pgs: 97 active+clean; 99 MiB data, 615 MiB used, 1.0 TiB / 1.0 TiB avail; 61 KiB/s rd, 411 KiB/s wr, 42 op/s 2024-03-24T00:20:06.460 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:20:06 smithi193 ceph-mon[36878]: mds.? [v2:172.21.15.193:6834/3744499911,v1:172.21.15.193:6835/3744499911] up:reconnect 2024-03-24T00:20:06.460 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:20:06 smithi193 ceph-mon[36878]: fsmap cephfs:5/5 4 up:standby-replay 3 up:standby 1 up:reconnect 4 up:active 2024-03-24T00:20:06.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:20:06 smithi193 ceph-mon[36878]: reconnect by client.15159 192.168.0.1:0/3463739313 after 0.0109998 2024-03-24T00:20:06.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:20:06 smithi193 ceph-mon[36878]: pgmap v532: 97 pgs: 97 active+clean; 99 MiB data, 615 MiB used, 1.0 TiB / 1.0 TiB avail; 61 KiB/s rd, 411 KiB/s wr, 42 op/s 2024-03-24T00:20:06.844 INFO:tasks.workunit.client.0.smithi118.stdout: 28 0 42 0 51 22 41 2024-03-24T00:20:06.892 INFO:journalctl@ceph.mds.f.smithi193.stdout:Mar 24 00:20:06 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-f[70644]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: In function 'void Locker::set_xlocks_done(MutationImpl*, bool)' thread 7f14ca6d6700 time 2024-03-24T00:20:06.530963+0000 2024-03-24T00:20:06.892 INFO:journalctl@ceph.mds.f.smithi193.stdout:Mar 24 00:20:06 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-f[70644]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: 626: FAILED ceph_assert(obj->is_auth()) 2024-03-24T00:20:06.892 INFO:journalctl@ceph.mds.f.smithi193.stdout:Mar 24 00:20:06 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-f[70644]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-24T00:20:06.892 INFO:journalctl@ceph.mds.f.smithi193.stdout:Mar 24 00:20:06 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-f[70644]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x135) [0x7f14d4d004a1] 2024-03-24T00:20:06.892 INFO:journalctl@ceph.mds.f.smithi193.stdout:Mar 24 00:20:06 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-f[70644]: 2: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7f14d4d00667] 2024-03-24T00:20:06.892 INFO:journalctl@ceph.mds.f.smithi193.stdout:Mar 24 00:20:06 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-f[70644]: 3: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x55d22a98c223] 2024-03-24T00:20:06.892 INFO:journalctl@ceph.mds.f.smithi193.stdout:Mar 24 00:20:06 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-f[70644]: 4: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x55d22a8065d6] 2024-03-24T00:20:06.892 INFO:journalctl@ceph.mds.f.smithi193.stdout:Mar 24 00:20:06 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-f[70644]: 5: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x55d22a806da1] 2024-03-24T00:20:06.893 INFO:journalctl@ceph.mds.f.smithi193.stdout:Mar 24 00:20:06 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-f[70644]: 6: (Server::handle_client_rename(boost::intrusive_ptr const&)+0x259d) [0x55d22a84397d] 2024-03-24T00:20:06.893 INFO:journalctl@ceph.mds.f.smithi193.stdout:Mar 24 00:20:06 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-f[70644]: 7: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x55d22a859233] 2024-03-24T00:20:06.893 INFO:journalctl@ceph.mds.f.smithi193.stdout:Mar 24 00:20:06 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-f[70644]: 8: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x55d22a85b197] 2024-03-24T00:20:06.893 INFO:journalctl@ceph.mds.f.smithi193.stdout:Mar 24 00:20:06 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-f[70644]: 9: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x55d22a75a11b] 2024-03-24T00:20:06.893 INFO:journalctl@ceph.mds.f.smithi193.stdout:Mar 24 00:20:06 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-f[70644]: 10: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x55d22a75a6bc] 2024-03-24T00:20:06.893 INFO:journalctl@ceph.mds.f.smithi193.stdout:Mar 24 00:20:06 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-f[70644]: 11: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x55d22a7441ff] 2024-03-24T00:20:06.893 INFO:journalctl@ceph.mds.f.smithi193.stdout:Mar 24 00:20:06 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-f[70644]: 12: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7f14d4f92488] 2024-03-24T00:20:06.893 INFO:journalctl@ceph.mds.f.smithi193.stdout:Mar 24 00:20:06 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-f[70644]: 13: (DispatchQueue::entry()+0x51f) [0x7f14d4f8f60f] 2024-03-24T00:20:06.893 INFO:journalctl@ceph.mds.f.smithi193.stdout:Mar 24 00:20:06 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-f[70644]: 14: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f14d5062e81] 2024-03-24T00:20:06.893 INFO:journalctl@ceph.mds.f.smithi193.stdout:Mar 24 00:20:06 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-f[70644]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7f14d3a7d1ca] 2024-03-24T00:20:06.893 INFO:journalctl@ceph.mds.f.smithi193.stdout:Mar 24 00:20:06 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-f[70644]: 16: clone() 2024-03-24T00:20:06.894 INFO:journalctl@ceph.mds.f.smithi193.stdout:Mar 24 00:20:06 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-f[70644]: 2024-03-24T00:20:06.531+0000 7f14ca6d6700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: In function 'void Locker::set_xlocks_done(MutationImpl*, bool)' thread 7f14ca6d6700 time 2024-03-24T00:20:06.530963+0000 2024-03-24T00:20:06.894 INFO:journalctl@ceph.mds.f.smithi193.stdout:Mar 24 00:20:06 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-f[70644]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: 626: FAILED ceph_assert(obj->is_auth()) 2024-03-24T00:20:06.894 INFO:journalctl@ceph.mds.f.smithi193.stdout:Mar 24 00:20:06 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-f[70644]: 2024-03-24T00:20:06.895 INFO:journalctl@ceph.mds.f.smithi193.stdout:Mar 24 00:20:06 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-f[70644]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-24T00:20:06.895 INFO:journalctl@ceph.mds.f.smithi193.stdout:Mar 24 00:20:06 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-f[70644]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x135) [0x7f14d4d004a1] 2024-03-24T00:20:06.895 INFO:journalctl@ceph.mds.f.smithi193.stdout:Mar 24 00:20:06 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-f[70644]: 2: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7f14d4d00667] 2024-03-24T00:20:06.895 INFO:journalctl@ceph.mds.f.smithi193.stdout:Mar 24 00:20:06 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-f[70644]: 3: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x55d22a98c223] 2024-03-24T00:20:06.895 INFO:journalctl@ceph.mds.f.smithi193.stdout:Mar 24 00:20:06 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-f[70644]: 4: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x55d22a8065d6] 2024-03-24T00:20:06.895 INFO:journalctl@ceph.mds.f.smithi193.stdout:Mar 24 00:20:06 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-f[70644]: 5: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x55d22a806da1] 2024-03-24T00:20:06.895 INFO:journalctl@ceph.mds.f.smithi193.stdout:Mar 24 00:20:06 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-f[70644]: 6: (Server::handle_client_rename(boost::intrusive_ptr const&)+0x259d) [0x55d22a84397d] 2024-03-24T00:20:06.895 INFO:journalctl@ceph.mds.f.smithi193.stdout:Mar 24 00:20:06 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-f[70644]: 7: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x55d22a859233] 2024-03-24T00:20:06.895 INFO:journalctl@ceph.mds.f.smithi193.stdout:Mar 24 00:20:06 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-f[70644]: 8: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x55d22a85b197] 2024-03-24T00:20:06.895 INFO:journalctl@ceph.mds.f.smithi193.stdout:Mar 24 00:20:06 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-f[70644]: 9: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x55d22a75a11b] 2024-03-24T00:20:06.895 INFO:journalctl@ceph.mds.f.smithi193.stdout:Mar 24 00:20:06 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-f[70644]: 10: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x55d22a75a6bc] 2024-03-24T00:20:06.895 INFO:journalctl@ceph.mds.f.smithi193.stdout:Mar 24 00:20:06 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-f[70644]: 11: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x55d22a7441ff] 2024-03-24T00:20:06.895 INFO:journalctl@ceph.mds.f.smithi193.stdout:Mar 24 00:20:06 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-f[70644]: 12: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7f14d4f92488] 2024-03-24T00:20:06.895 INFO:journalctl@ceph.mds.f.smithi193.stdout:Mar 24 00:20:06 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-f[70644]: 13: (DispatchQueue::entry()+0x51f) [0x7f14d4f8f60f] 2024-03-24T00:20:06.896 INFO:journalctl@ceph.mds.f.smithi193.stdout:Mar 24 00:20:06 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-f[70644]: 14: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f14d5062e81] 2024-03-24T00:20:06.896 INFO:journalctl@ceph.mds.f.smithi193.stdout:Mar 24 00:20:06 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-f[70644]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7f14d3a7d1ca] 2024-03-24T00:20:06.896 INFO:journalctl@ceph.mds.f.smithi193.stdout:Mar 24 00:20:06 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-f[70644]: 16: clone() 2024-03-24T00:20:06.896 INFO:journalctl@ceph.mds.f.smithi193.stdout:Mar 24 00:20:06 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-f[70644]: 2024-03-24T00:20:06.896 INFO:journalctl@ceph.mds.f.smithi193.stdout:Mar 24 00:20:06 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-f[70644]: *** Caught signal (Aborted) ** 2024-03-24T00:20:06.896 INFO:journalctl@ceph.mds.f.smithi193.stdout:Mar 24 00:20:06 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-f[70644]: in thread 7f14ca6d6700 thread_name:ms_dispatch 2024-03-24T00:20:06.896 INFO:journalctl@ceph.mds.f.smithi193.stdout:Mar 24 00:20:06 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-f[70644]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-24T00:20:06.896 INFO:journalctl@ceph.mds.f.smithi193.stdout:Mar 24 00:20:06 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-f[70644]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f14d3a87d20] 2024-03-24T00:20:06.896 INFO:journalctl@ceph.mds.f.smithi193.stdout:Mar 24 00:20:06 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-f[70644]: 2: gsignal() 2024-03-24T00:20:06.896 INFO:journalctl@ceph.mds.f.smithi193.stdout:Mar 24 00:20:06 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-f[70644]: 3: abort() 2024-03-24T00:20:06.896 INFO:journalctl@ceph.mds.f.smithi193.stdout:Mar 24 00:20:06 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-f[70644]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x18f) [0x7f14d4d004fb] 2024-03-24T00:20:06.896 INFO:journalctl@ceph.mds.f.smithi193.stdout:Mar 24 00:20:06 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-f[70644]: 5: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7f14d4d00667] 2024-03-24T00:20:06.896 INFO:journalctl@ceph.mds.f.smithi193.stdout:Mar 24 00:20:06 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-f[70644]: 6: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x55d22a98c223] 2024-03-24T00:20:06.896 INFO:journalctl@ceph.mds.f.smithi193.stdout:Mar 24 00:20:06 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-f[70644]: 7: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x55d22a8065d6] 2024-03-24T00:20:06.897 INFO:journalctl@ceph.mds.f.smithi193.stdout:Mar 24 00:20:06 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-f[70644]: 8: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x55d22a806da1] 2024-03-24T00:20:06.897 INFO:journalctl@ceph.mds.f.smithi193.stdout:Mar 24 00:20:06 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-f[70644]: 9: (Server::handle_client_rename(boost::intrusive_ptr const&)+0x259d) [0x55d22a84397d] 2024-03-24T00:20:06.897 INFO:journalctl@ceph.mds.f.smithi193.stdout:Mar 24 00:20:06 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-f[70644]: 10: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x55d22a859233] 2024-03-24T00:20:06.897 INFO:journalctl@ceph.mds.f.smithi193.stdout:Mar 24 00:20:06 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-f[70644]: 11: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x55d22a85b197] 2024-03-24T00:20:06.897 INFO:journalctl@ceph.mds.f.smithi193.stdout:Mar 24 00:20:06 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-f[70644]: 12: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x55d22a75a11b] 2024-03-24T00:20:06.897 INFO:journalctl@ceph.mds.f.smithi193.stdout:Mar 24 00:20:06 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-f[70644]: 13: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x55d22a75a6bc] 2024-03-24T00:20:06.897 INFO:journalctl@ceph.mds.f.smithi193.stdout:Mar 24 00:20:06 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-f[70644]: 14: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x55d22a7441ff] 2024-03-24T00:20:06.897 INFO:journalctl@ceph.mds.f.smithi193.stdout:Mar 24 00:20:06 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-f[70644]: 15: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7f14d4f92488] 2024-03-24T00:20:06.897 INFO:journalctl@ceph.mds.f.smithi193.stdout:Mar 24 00:20:06 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-f[70644]: 16: (DispatchQueue::entry()+0x51f) [0x7f14d4f8f60f] 2024-03-24T00:20:06.897 INFO:journalctl@ceph.mds.f.smithi193.stdout:Mar 24 00:20:06 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-f[70644]: 17: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f14d5062e81] 2024-03-24T00:20:06.897 INFO:journalctl@ceph.mds.f.smithi193.stdout:Mar 24 00:20:06 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-f[70644]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7f14d3a7d1ca] 2024-03-24T00:20:06.897 INFO:journalctl@ceph.mds.f.smithi193.stdout:Mar 24 00:20:06 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-f[70644]: 19: clone() 2024-03-24T00:20:06.897 INFO:journalctl@ceph.mds.f.smithi193.stdout:Mar 24 00:20:06 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-f[70644]: 2024-03-24T00:20:06.533+0000 7f14ca6d6700 -1 *** Caught signal (Aborted) ** 2024-03-24T00:20:06.897 INFO:journalctl@ceph.mds.f.smithi193.stdout:Mar 24 00:20:06 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-f[70644]: in thread 7f14ca6d6700 thread_name:ms_dispatch 2024-03-24T00:20:06.897 INFO:journalctl@ceph.mds.f.smithi193.stdout:Mar 24 00:20:06 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-f[70644]: 2024-03-24T00:20:06.898 INFO:journalctl@ceph.mds.f.smithi193.stdout:Mar 24 00:20:06 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-f[70644]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-24T00:20:06.898 INFO:journalctl@ceph.mds.f.smithi193.stdout:Mar 24 00:20:06 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-f[70644]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f14d3a87d20] 2024-03-24T00:20:06.898 INFO:journalctl@ceph.mds.f.smithi193.stdout:Mar 24 00:20:06 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-f[70644]: 2: gsignal() 2024-03-24T00:20:06.898 INFO:journalctl@ceph.mds.f.smithi193.stdout:Mar 24 00:20:06 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-f[70644]: 3: abort() 2024-03-24T00:20:06.898 INFO:journalctl@ceph.mds.f.smithi193.stdout:Mar 24 00:20:06 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-f[70644]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x18f) [0x7f14d4d004fb] 2024-03-24T00:20:06.898 INFO:journalctl@ceph.mds.f.smithi193.stdout:Mar 24 00:20:06 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-f[70644]: 5: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7f14d4d00667] 2024-03-24T00:20:06.898 INFO:journalctl@ceph.mds.f.smithi193.stdout:Mar 24 00:20:06 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-f[70644]: 6: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x55d22a98c223] 2024-03-24T00:20:06.898 INFO:journalctl@ceph.mds.f.smithi193.stdout:Mar 24 00:20:06 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-f[70644]: 7: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x55d22a8065d6] 2024-03-24T00:20:06.898 INFO:journalctl@ceph.mds.f.smithi193.stdout:Mar 24 00:20:06 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-f[70644]: 8: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x55d22a806da1] 2024-03-24T00:20:06.898 INFO:journalctl@ceph.mds.f.smithi193.stdout:Mar 24 00:20:06 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-f[70644]: 9: (Server::handle_client_rename(boost::intrusive_ptr const&)+0x259d) [0x55d22a84397d] 2024-03-24T00:20:06.898 INFO:journalctl@ceph.mds.f.smithi193.stdout:Mar 24 00:20:06 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-f[70644]: 10: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x55d22a859233] 2024-03-24T00:20:06.898 INFO:journalctl@ceph.mds.f.smithi193.stdout:Mar 24 00:20:06 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-f[70644]: 11: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x55d22a85b197] 2024-03-24T00:20:06.898 INFO:journalctl@ceph.mds.f.smithi193.stdout:Mar 24 00:20:06 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-f[70644]: 12: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x55d22a75a11b] 2024-03-24T00:20:06.898 INFO:journalctl@ceph.mds.f.smithi193.stdout:Mar 24 00:20:06 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-f[70644]: 13: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x55d22a75a6bc] 2024-03-24T00:20:06.899 INFO:journalctl@ceph.mds.f.smithi193.stdout:Mar 24 00:20:06 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-f[70644]: 14: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x55d22a7441ff] 2024-03-24T00:20:06.899 INFO:journalctl@ceph.mds.f.smithi193.stdout:Mar 24 00:20:06 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-f[70644]: 15: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7f14d4f92488] 2024-03-24T00:20:06.899 INFO:journalctl@ceph.mds.f.smithi193.stdout:Mar 24 00:20:06 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-f[70644]: 16: (DispatchQueue::entry()+0x51f) [0x7f14d4f8f60f] 2024-03-24T00:20:06.899 INFO:journalctl@ceph.mds.f.smithi193.stdout:Mar 24 00:20:06 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-f[70644]: 17: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f14d5062e81] 2024-03-24T00:20:06.899 INFO:journalctl@ceph.mds.f.smithi193.stdout:Mar 24 00:20:06 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-f[70644]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7f14d3a7d1ca] 2024-03-24T00:20:06.899 INFO:journalctl@ceph.mds.f.smithi193.stdout:Mar 24 00:20:06 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-f[70644]: 19: clone() 2024-03-24T00:20:06.899 INFO:journalctl@ceph.mds.f.smithi193.stdout:Mar 24 00:20:06 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-f[70644]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-03-24T00:20:06.899 INFO:journalctl@ceph.mds.f.smithi193.stdout:Mar 24 00:20:06 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-f[70644]: 2024-03-24T00:20:06.899 INFO:journalctl@ceph.mds.f.smithi193.stdout:Mar 24 00:20:06 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-f[70644]: -1> 2024-03-24T00:20:06.531+0000 7f14ca6d6700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: In function 'void Locker::set_xlocks_done(MutationImpl*, bool)' thread 7f14ca6d6700 time 2024-03-24T00:20:06.530963+0000 2024-03-24T00:20:06.899 INFO:journalctl@ceph.mds.f.smithi193.stdout:Mar 24 00:20:06 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-f[70644]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: 626: FAILED ceph_assert(obj->is_auth()) 2024-03-24T00:20:06.899 INFO:journalctl@ceph.mds.f.smithi193.stdout:Mar 24 00:20:06 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-f[70644]: 2024-03-24T00:20:06.899 INFO:journalctl@ceph.mds.f.smithi193.stdout:Mar 24 00:20:06 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-f[70644]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-24T00:20:06.899 INFO:journalctl@ceph.mds.f.smithi193.stdout:Mar 24 00:20:06 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-f[70644]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x135) [0x7f14d4d004a1] 2024-03-24T00:20:06.899 INFO:journalctl@ceph.mds.f.smithi193.stdout:Mar 24 00:20:06 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-f[70644]: 2: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7f14d4d00667] 2024-03-24T00:20:06.900 INFO:journalctl@ceph.mds.f.smithi193.stdout:Mar 24 00:20:06 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-f[70644]: 3: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x55d22a98c223] 2024-03-24T00:20:06.900 INFO:journalctl@ceph.mds.f.smithi193.stdout:Mar 24 00:20:06 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-f[70644]: 4: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x55d22a8065d6] 2024-03-24T00:20:06.900 INFO:journalctl@ceph.mds.f.smithi193.stdout:Mar 24 00:20:06 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-f[70644]: 5: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x55d22a806da1] 2024-03-24T00:20:06.900 INFO:journalctl@ceph.mds.f.smithi193.stdout:Mar 24 00:20:06 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-f[70644]: 6: (Server::handle_client_rename(boost::intrusive_ptr const&)+0x259d) [0x55d22a84397d] 2024-03-24T00:20:06.900 INFO:journalctl@ceph.mds.f.smithi193.stdout:Mar 24 00:20:06 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-f[70644]: 7: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x55d22a859233] 2024-03-24T00:20:06.900 INFO:journalctl@ceph.mds.f.smithi193.stdout:Mar 24 00:20:06 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-f[70644]: 8: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x55d22a85b197] 2024-03-24T00:20:06.900 INFO:journalctl@ceph.mds.f.smithi193.stdout:Mar 24 00:20:06 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-f[70644]: 9: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x55d22a75a11b] 2024-03-24T00:20:06.900 INFO:journalctl@ceph.mds.f.smithi193.stdout:Mar 24 00:20:06 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-f[70644]: 10: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x55d22a75a6bc] 2024-03-24T00:20:06.900 INFO:journalctl@ceph.mds.f.smithi193.stdout:Mar 24 00:20:06 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-f[70644]: 11: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x55d22a7441ff] 2024-03-24T00:20:06.900 INFO:journalctl@ceph.mds.f.smithi193.stdout:Mar 24 00:20:06 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-f[70644]: 12: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7f14d4f92488] 2024-03-24T00:20:06.900 INFO:journalctl@ceph.mds.f.smithi193.stdout:Mar 24 00:20:06 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-f[70644]: 13: (DispatchQueue::entry()+0x51f) [0x7f14d4f8f60f] 2024-03-24T00:20:06.900 INFO:journalctl@ceph.mds.f.smithi193.stdout:Mar 24 00:20:06 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-f[70644]: 14: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f14d5062e81] 2024-03-24T00:20:06.900 INFO:journalctl@ceph.mds.f.smithi193.stdout:Mar 24 00:20:06 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-f[70644]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7f14d3a7d1ca] 2024-03-24T00:20:06.900 INFO:journalctl@ceph.mds.f.smithi193.stdout:Mar 24 00:20:06 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-f[70644]: 16: clone() 2024-03-24T00:20:06.900 INFO:journalctl@ceph.mds.f.smithi193.stdout:Mar 24 00:20:06 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-f[70644]: 2024-03-24T00:20:06.901 INFO:journalctl@ceph.mds.f.smithi193.stdout:Mar 24 00:20:06 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-f[70644]: 0> 2024-03-24T00:20:06.533+0000 7f14ca6d6700 -1 *** Caught signal (Aborted) ** 2024-03-24T00:20:06.901 INFO:journalctl@ceph.mds.f.smithi193.stdout:Mar 24 00:20:06 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-f[70644]: in thread 7f14ca6d6700 thread_name:ms_dispatch 2024-03-24T00:20:06.901 INFO:journalctl@ceph.mds.f.smithi193.stdout:Mar 24 00:20:06 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-f[70644]: 2024-03-24T00:20:06.901 INFO:journalctl@ceph.mds.f.smithi193.stdout:Mar 24 00:20:06 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-f[70644]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-24T00:20:06.901 INFO:journalctl@ceph.mds.f.smithi193.stdout:Mar 24 00:20:06 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-f[70644]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f14d3a87d20] 2024-03-24T00:20:06.901 INFO:journalctl@ceph.mds.f.smithi193.stdout:Mar 24 00:20:06 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-f[70644]: 2: gsignal() 2024-03-24T00:20:06.901 INFO:journalctl@ceph.mds.f.smithi193.stdout:Mar 24 00:20:06 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-f[70644]: 3: abort() 2024-03-24T00:20:06.901 INFO:journalctl@ceph.mds.f.smithi193.stdout:Mar 24 00:20:06 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-f[70644]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x18f) [0x7f14d4d004fb] 2024-03-24T00:20:06.901 INFO:journalctl@ceph.mds.f.smithi193.stdout:Mar 24 00:20:06 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-f[70644]: 5: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7f14d4d00667] 2024-03-24T00:20:06.901 INFO:journalctl@ceph.mds.f.smithi193.stdout:Mar 24 00:20:06 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-f[70644]: 6: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x55d22a98c223] 2024-03-24T00:20:06.901 INFO:journalctl@ceph.mds.f.smithi193.stdout:Mar 24 00:20:06 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-f[70644]: 7: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x55d22a8065d6] 2024-03-24T00:20:06.901 INFO:journalctl@ceph.mds.f.smithi193.stdout:Mar 24 00:20:06 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-f[70644]: 8: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x55d22a806da1] 2024-03-24T00:20:06.901 INFO:journalctl@ceph.mds.f.smithi193.stdout:Mar 24 00:20:06 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-f[70644]: 9: (Server::handle_client_rename(boost::intrusive_ptr const&)+0x259d) [0x55d22a84397d] 2024-03-24T00:20:06.901 INFO:journalctl@ceph.mds.f.smithi193.stdout:Mar 24 00:20:06 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-f[70644]: 10: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x55d22a859233] 2024-03-24T00:20:06.901 INFO:journalctl@ceph.mds.f.smithi193.stdout:Mar 24 00:20:06 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-f[70644]: 11: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x55d22a85b197] 2024-03-24T00:20:06.902 INFO:journalctl@ceph.mds.f.smithi193.stdout:Mar 24 00:20:06 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-f[70644]: 12: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x55d22a75a11b] 2024-03-24T00:20:06.902 INFO:journalctl@ceph.mds.f.smithi193.stdout:Mar 24 00:20:06 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-f[70644]: 13: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x55d22a75a6bc] 2024-03-24T00:20:06.902 INFO:journalctl@ceph.mds.f.smithi193.stdout:Mar 24 00:20:06 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-f[70644]: 14: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x55d22a7441ff] 2024-03-24T00:20:06.902 INFO:journalctl@ceph.mds.f.smithi193.stdout:Mar 24 00:20:06 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-f[70644]: 15: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7f14d4f92488] 2024-03-24T00:20:06.902 INFO:journalctl@ceph.mds.f.smithi193.stdout:Mar 24 00:20:06 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-f[70644]: 16: (DispatchQueue::entry()+0x51f) [0x7f14d4f8f60f] 2024-03-24T00:20:06.902 INFO:journalctl@ceph.mds.f.smithi193.stdout:Mar 24 00:20:06 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-f[70644]: 17: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f14d5062e81] 2024-03-24T00:20:06.902 INFO:journalctl@ceph.mds.f.smithi193.stdout:Mar 24 00:20:06 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-f[70644]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7f14d3a7d1ca] 2024-03-24T00:20:06.902 INFO:journalctl@ceph.mds.f.smithi193.stdout:Mar 24 00:20:06 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-f[70644]: 19: clone() 2024-03-24T00:20:06.902 INFO:journalctl@ceph.mds.f.smithi193.stdout:Mar 24 00:20:06 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-f[70644]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-03-24T00:20:06.902 INFO:journalctl@ceph.mds.f.smithi193.stdout:Mar 24 00:20:06 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-f[70644]: 2024-03-24T00:20:06.902 INFO:journalctl@ceph.mds.f.smithi193.stdout:Mar 24 00:20:06 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-f[70644]: -1> 2024-03-24T00:20:06.531+0000 7f14ca6d6700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: In function 'void Locker::set_xlocks_done(MutationImpl*, bool)' thread 7f14ca6d6700 time 2024-03-24T00:20:06.530963+0000 2024-03-24T00:20:06.902 INFO:journalctl@ceph.mds.f.smithi193.stdout:Mar 24 00:20:06 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-f[70644]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: 626: FAILED ceph_assert(obj->is_auth()) 2024-03-24T00:20:06.902 INFO:journalctl@ceph.mds.f.smithi193.stdout:Mar 24 00:20:06 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-f[70644]: 2024-03-24T00:20:06.903 INFO:journalctl@ceph.mds.f.smithi193.stdout:Mar 24 00:20:06 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-f[70644]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-24T00:20:06.903 INFO:journalctl@ceph.mds.f.smithi193.stdout:Mar 24 00:20:06 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-f[70644]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x135) [0x7f14d4d004a1] 2024-03-24T00:20:06.903 INFO:journalctl@ceph.mds.f.smithi193.stdout:Mar 24 00:20:06 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-f[70644]: 2: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7f14d4d00667] 2024-03-24T00:20:06.903 INFO:journalctl@ceph.mds.f.smithi193.stdout:Mar 24 00:20:06 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-f[70644]: 3: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x55d22a98c223] 2024-03-24T00:20:06.903 INFO:journalctl@ceph.mds.f.smithi193.stdout:Mar 24 00:20:06 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-f[70644]: 4: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x55d22a8065d6] 2024-03-24T00:20:06.903 INFO:journalctl@ceph.mds.f.smithi193.stdout:Mar 24 00:20:06 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-f[70644]: 5: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x55d22a806da1] 2024-03-24T00:20:06.903 INFO:journalctl@ceph.mds.f.smithi193.stdout:Mar 24 00:20:06 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-f[70644]: 6: (Server::handle_client_rename(boost::intrusive_ptr const&)+0x259d) [0x55d22a84397d] 2024-03-24T00:20:06.904 INFO:journalctl@ceph.mds.f.smithi193.stdout:Mar 24 00:20:06 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-f[70644]: 7: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x55d22a859233] 2024-03-24T00:20:06.904 INFO:journalctl@ceph.mds.f.smithi193.stdout:Mar 24 00:20:06 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-f[70644]: 8: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x55d22a85b197] 2024-03-24T00:20:06.904 INFO:journalctl@ceph.mds.f.smithi193.stdout:Mar 24 00:20:06 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-f[70644]: 9: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x55d22a75a11b] 2024-03-24T00:20:06.904 INFO:journalctl@ceph.mds.f.smithi193.stdout:Mar 24 00:20:06 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-f[70644]: 10: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x55d22a75a6bc] 2024-03-24T00:20:06.904 INFO:journalctl@ceph.mds.f.smithi193.stdout:Mar 24 00:20:06 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-f[70644]: 11: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x55d22a7441ff] 2024-03-24T00:20:06.904 INFO:journalctl@ceph.mds.f.smithi193.stdout:Mar 24 00:20:06 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-f[70644]: 12: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7f14d4f92488] 2024-03-24T00:20:06.904 INFO:journalctl@ceph.mds.f.smithi193.stdout:Mar 24 00:20:06 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-f[70644]: 13: (DispatchQueue::entry()+0x51f) [0x7f14d4f8f60f] 2024-03-24T00:20:06.904 INFO:journalctl@ceph.mds.f.smithi193.stdout:Mar 24 00:20:06 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-f[70644]: 14: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f14d5062e81] 2024-03-24T00:20:06.904 INFO:journalctl@ceph.mds.f.smithi193.stdout:Mar 24 00:20:06 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-f[70644]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7f14d3a7d1ca] 2024-03-24T00:20:06.904 INFO:journalctl@ceph.mds.f.smithi193.stdout:Mar 24 00:20:06 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-f[70644]: 16: clone() 2024-03-24T00:20:06.904 INFO:journalctl@ceph.mds.f.smithi193.stdout:Mar 24 00:20:06 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-f[70644]: 2024-03-24T00:20:06.904 INFO:journalctl@ceph.mds.f.smithi193.stdout:Mar 24 00:20:06 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-f[70644]: 0> 2024-03-24T00:20:06.533+0000 7f14ca6d6700 -1 *** Caught signal (Aborted) ** 2024-03-24T00:20:06.904 INFO:journalctl@ceph.mds.f.smithi193.stdout:Mar 24 00:20:06 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-f[70644]: in thread 7f14ca6d6700 thread_name:ms_dispatch 2024-03-24T00:20:06.904 INFO:journalctl@ceph.mds.f.smithi193.stdout:Mar 24 00:20:06 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-f[70644]: 2024-03-24T00:20:06.904 INFO:journalctl@ceph.mds.f.smithi193.stdout:Mar 24 00:20:06 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-f[70644]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-24T00:20:06.905 INFO:journalctl@ceph.mds.f.smithi193.stdout:Mar 24 00:20:06 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-f[70644]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f14d3a87d20] 2024-03-24T00:20:06.905 INFO:journalctl@ceph.mds.f.smithi193.stdout:Mar 24 00:20:06 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-f[70644]: 2: gsignal() 2024-03-24T00:20:06.905 INFO:journalctl@ceph.mds.f.smithi193.stdout:Mar 24 00:20:06 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-f[70644]: 3: abort() 2024-03-24T00:20:06.905 INFO:journalctl@ceph.mds.f.smithi193.stdout:Mar 24 00:20:06 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-f[70644]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x18f) [0x7f14d4d004fb] 2024-03-24T00:20:06.905 INFO:journalctl@ceph.mds.f.smithi193.stdout:Mar 24 00:20:06 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-f[70644]: 5: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7f14d4d00667] 2024-03-24T00:20:06.905 INFO:journalctl@ceph.mds.f.smithi193.stdout:Mar 24 00:20:06 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-f[70644]: 6: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x55d22a98c223] 2024-03-24T00:20:06.905 INFO:journalctl@ceph.mds.f.smithi193.stdout:Mar 24 00:20:06 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-f[70644]: 7: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x55d22a8065d6] 2024-03-24T00:20:06.905 INFO:journalctl@ceph.mds.f.smithi193.stdout:Mar 24 00:20:06 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-f[70644]: 8: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x55d22a806da1] 2024-03-24T00:20:06.905 INFO:journalctl@ceph.mds.f.smithi193.stdout:Mar 24 00:20:06 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-f[70644]: 9: (Server::handle_client_rename(boost::intrusive_ptr const&)+0x259d) [0x55d22a84397d] 2024-03-24T00:20:06.905 INFO:journalctl@ceph.mds.f.smithi193.stdout:Mar 24 00:20:06 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-f[70644]: 10: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x55d22a859233] 2024-03-24T00:20:06.905 INFO:journalctl@ceph.mds.f.smithi193.stdout:Mar 24 00:20:06 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-f[70644]: 11: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x55d22a85b197] 2024-03-24T00:20:06.905 INFO:journalctl@ceph.mds.f.smithi193.stdout:Mar 24 00:20:06 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-f[70644]: 12: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x55d22a75a11b] 2024-03-24T00:20:06.905 INFO:journalctl@ceph.mds.f.smithi193.stdout:Mar 24 00:20:06 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-f[70644]: 13: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x55d22a75a6bc] 2024-03-24T00:20:06.905 INFO:journalctl@ceph.mds.f.smithi193.stdout:Mar 24 00:20:06 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-f[70644]: 14: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x55d22a7441ff] 2024-03-24T00:20:06.906 INFO:journalctl@ceph.mds.f.smithi193.stdout:Mar 24 00:20:06 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-f[70644]: 15: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7f14d4f92488] 2024-03-24T00:20:06.906 INFO:journalctl@ceph.mds.f.smithi193.stdout:Mar 24 00:20:06 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-f[70644]: 16: (DispatchQueue::entry()+0x51f) [0x7f14d4f8f60f] 2024-03-24T00:20:06.906 INFO:journalctl@ceph.mds.f.smithi193.stdout:Mar 24 00:20:06 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-f[70644]: 17: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f14d5062e81] 2024-03-24T00:20:06.906 INFO:journalctl@ceph.mds.f.smithi193.stdout:Mar 24 00:20:06 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-f[70644]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7f14d3a7d1ca] 2024-03-24T00:20:06.906 INFO:journalctl@ceph.mds.f.smithi193.stdout:Mar 24 00:20:06 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-f[70644]: 19: clone() 2024-03-24T00:20:06.906 INFO:journalctl@ceph.mds.f.smithi193.stdout:Mar 24 00:20:06 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-f[70644]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-03-24T00:20:06.906 INFO:journalctl@ceph.mds.f.smithi193.stdout:Mar 24 00:20:06 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-f[70644]: 2024-03-24T00:20:06.906 INFO:journalctl@ceph.mds.f.smithi193.stdout:Mar 24 00:20:06 smithi193 podman[70987]: 2024-03-24 00:20:06.592272452 +0000 UTC m=+0.019226929 container died 43046138110a3fada8001f090558acf465631fd5ade04440a53e6a4f9c03cad7 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-f, org.label-schema.vendor=CentOS, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240319, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, GIT_CLEAN=True, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, ceph=True, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-24T00:20:07.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:20:07 smithi193 ceph-mon[36878]: mds.? [v2:172.21.15.193:6834/3744499911,v1:172.21.15.193:6835/3744499911] up:rejoin 2024-03-24T00:20:07.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:20:07 smithi193 ceph-mon[36878]: fsmap cephfs:5/5 4 up:standby-replay 3 up:standby 1 up:rejoin 4 up:active 2024-03-24T00:20:07.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:20:07 smithi193 ceph-mon[36878]: daemon mds.f is now active in filesystem cephfs as rank 3 2024-03-24T00:20:07.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:20:07 smithi193 ceph-mon[36878]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-03-24T00:20:07.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:20:07 smithi193 ceph-mon[36878]: Cluster is now healthy 2024-03-24T00:20:07.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:20:07 smithi193 ceph-mon[36878]: mds.? [v2:172.21.15.193:6834/3744499911,v1:172.21.15.193:6835/3744499911] up:active 2024-03-24T00:20:07.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:20:07 smithi193 ceph-mon[36878]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-03-24T00:20:07.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:20:07 smithi118 ceph-mon[31535]: mds.? [v2:172.21.15.193:6834/3744499911,v1:172.21.15.193:6835/3744499911] up:rejoin 2024-03-24T00:20:07.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:20:07 smithi118 ceph-mon[31535]: fsmap cephfs:5/5 4 up:standby-replay 3 up:standby 1 up:rejoin 4 up:active 2024-03-24T00:20:07.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:20:07 smithi118 ceph-mon[31535]: daemon mds.f is now active in filesystem cephfs as rank 3 2024-03-24T00:20:07.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:20:07 smithi118 ceph-mon[31535]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-03-24T00:20:07.447 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:20:07 smithi118 ceph-mon[31535]: Cluster is now healthy 2024-03-24T00:20:07.447 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:20:07 smithi118 ceph-mon[31535]: mds.? [v2:172.21.15.193:6834/3744499911,v1:172.21.15.193:6835/3744499911] up:active 2024-03-24T00:20:07.447 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:20:07 smithi118 ceph-mon[31535]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-03-24T00:20:07.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:20:07 smithi189 ceph-mon[36187]: mds.? [v2:172.21.15.193:6834/3744499911,v1:172.21.15.193:6835/3744499911] up:rejoin 2024-03-24T00:20:07.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:20:07 smithi189 ceph-mon[36187]: fsmap cephfs:5/5 4 up:standby-replay 3 up:standby 1 up:rejoin 4 up:active 2024-03-24T00:20:07.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:20:07 smithi189 ceph-mon[36187]: daemon mds.f is now active in filesystem cephfs as rank 3 2024-03-24T00:20:07.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:20:07 smithi189 ceph-mon[36187]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-03-24T00:20:07.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:20:07 smithi189 ceph-mon[36187]: Cluster is now healthy 2024-03-24T00:20:07.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:20:07 smithi189 ceph-mon[36187]: mds.? [v2:172.21.15.193:6834/3744499911,v1:172.21.15.193:6835/3744499911] up:active 2024-03-24T00:20:07.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:20:07 smithi189 ceph-mon[36187]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-03-24T00:20:07.711 INFO:journalctl@ceph.mds.f.smithi193.stdout:Mar 24 00:20:07 smithi193 podman[70987]: 2024-03-24 00:20:07.277866192 +0000 UTC m=+0.704820667 container remove 43046138110a3fada8001f090558acf465631fd5ade04440a53e6a4f9c03cad7 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-f, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240319, org.label-schema.schema-version=1.0, ceph=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_CLEAN=True, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD) 2024-03-24T00:20:07.711 INFO:journalctl@ceph.mds.f.smithi193.stdout:Mar 24 00:20:07 smithi193 systemd[1]: ceph-d79a5420-e971-11ee-95cd-87774f69a715@mds.f.service: Main process exited, code=exited, status=134/n/a 2024-03-24T00:20:07.711 INFO:journalctl@ceph.mds.f.smithi193.stdout:Mar 24 00:20:07 smithi193 systemd[1]: ceph-d79a5420-e971-11ee-95cd-87774f69a715@mds.f.service: Failed with result 'exit-code'. 2024-03-24T00:20:08.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:20:08 smithi118 ceph-mon[31535]: pgmap v533: 97 pgs: 97 active+clean; 96 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 116 KiB/s rd, 436 KiB/s wr, 92 op/s 2024-03-24T00:20:08.453 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:20:08 smithi189 ceph-mon[36187]: pgmap v533: 97 pgs: 97 active+clean; 96 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 116 KiB/s rd, 436 KiB/s wr, 92 op/s 2024-03-24T00:20:08.460 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:20:08 smithi193 ceph-mon[36878]: pgmap v533: 97 pgs: 97 active+clean; 96 MiB data, 612 MiB used, 1.0 TiB / 1.0 TiB avail; 116 KiB/s rd, 436 KiB/s wr, 92 op/s 2024-03-24T00:20:11.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:20:10 smithi118 ceph-mon[31535]: pgmap v534: 97 pgs: 97 active+clean; 92 MiB data, 633 MiB used, 1.0 TiB / 1.0 TiB avail; 121 KiB/s rd, 354 KiB/s wr, 106 op/s 2024-03-24T00:20:11.203 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:20:10 smithi189 ceph-mon[36187]: pgmap v534: 97 pgs: 97 active+clean; 92 MiB data, 633 MiB used, 1.0 TiB / 1.0 TiB avail; 121 KiB/s rd, 354 KiB/s wr, 106 op/s 2024-03-24T00:20:11.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:20:10 smithi193 ceph-mon[36878]: pgmap v534: 97 pgs: 97 active+clean; 92 MiB data, 633 MiB used, 1.0 TiB / 1.0 TiB avail; 121 KiB/s rd, 354 KiB/s wr, 106 op/s 2024-03-24T00:20:13.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:20:12 smithi118 ceph-mon[31535]: pgmap v535: 97 pgs: 97 active+clean; 92 MiB data, 634 MiB used, 1.0 TiB / 1.0 TiB avail; 127 KiB/s rd, 327 KiB/s wr, 107 op/s 2024-03-24T00:20:13.203 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:20:12 smithi189 ceph-mon[36187]: pgmap v535: 97 pgs: 97 active+clean; 92 MiB data, 634 MiB used, 1.0 TiB / 1.0 TiB avail; 127 KiB/s rd, 327 KiB/s wr, 107 op/s 2024-03-24T00:20:13.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:20:12 smithi193 ceph-mon[36878]: pgmap v535: 97 pgs: 97 active+clean; 92 MiB data, 634 MiB used, 1.0 TiB / 1.0 TiB avail; 127 KiB/s rd, 327 KiB/s wr, 107 op/s 2024-03-24T00:20:14.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:20:13 smithi118 ceph-mon[31535]: pgmap v536: 97 pgs: 97 active+clean; 92 MiB data, 634 MiB used, 1.0 TiB / 1.0 TiB avail; 152 KiB/s rd, 320 KiB/s wr, 107 op/s 2024-03-24T00:20:14.203 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:20:13 smithi189 ceph-mon[36187]: pgmap v536: 97 pgs: 97 active+clean; 92 MiB data, 634 MiB used, 1.0 TiB / 1.0 TiB avail; 152 KiB/s rd, 320 KiB/s wr, 107 op/s 2024-03-24T00:20:14.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:20:13 smithi193 ceph-mon[36878]: pgmap v536: 97 pgs: 97 active+clean; 92 MiB data, 634 MiB used, 1.0 TiB / 1.0 TiB avail; 152 KiB/s rd, 320 KiB/s wr, 107 op/s 2024-03-24T00:20:16.847 INFO:tasks.workunit.client.0.smithi118.stdout: 30 0 12 0 106 0 0 2024-03-24T00:20:17.018 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:20:17.019+0000 7fd627e00640 1 --2- 172.21.15.118:0/736842230 >> [v2:172.21.15.193:6832/1023428412,v1:172.21.15.193:6833/1023428412] conn(0x55c55390a6a0 0x55c55390cac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:20:17.019 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:20:17.019+0000 7fd627e00640 1 -- 172.21.15.118:0/736842230 >> [v2:172.21.15.193:6832/1023428412,v1:172.21.15.193:6833/1023428412] conn(0x55c55390a6a0 msgr2=0x55c55390cac0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-03-24T00:20:17.019 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:20:17.019+0000 7fd627e00640 1 -- 172.21.15.118:0/736842230 >> [v2:172.21.15.193:6832/1023428412,v1:172.21.15.193:6833/1023428412] conn(0x55c55390a6a0 msgr2=0x55c55390cac0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-03-24T00:20:17.019 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:20:17.019+0000 7fd627e00640 1 --2- 172.21.15.118:0/736842230 >> [v2:172.21.15.193:6832/1023428412,v1:172.21.15.193:6833/1023428412] conn(0x55c55390a6a0 0x55c55390cac0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-03-24T00:20:17.019 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:20:17.020+0000 7fd627e00640 1 --2- 172.21.15.118:0/736842230 >> [v2:172.21.15.193:6832/1023428412,v1:172.21.15.193:6833/1023428412] conn(0x55c55390a6a0 0x55c55390cac0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-03-24T00:20:17.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:20:16 smithi118 ceph-mon[31535]: pgmap v537: 97 pgs: 97 active+clean; 92 MiB data, 634 MiB used, 1.0 TiB / 1.0 TiB avail; 137 KiB/s rd, 291 KiB/s wr, 98 op/s 2024-03-24T00:20:17.203 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:20:16 smithi189 ceph-mon[36187]: pgmap v537: 97 pgs: 97 active+clean; 92 MiB data, 634 MiB used, 1.0 TiB / 1.0 TiB avail; 137 KiB/s rd, 291 KiB/s wr, 98 op/s 2024-03-24T00:20:17.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:20:16 smithi193 ceph-mon[36878]: pgmap v537: 97 pgs: 97 active+clean; 92 MiB data, 634 MiB used, 1.0 TiB / 1.0 TiB avail; 137 KiB/s rd, 291 KiB/s wr, 98 op/s 2024-03-24T00:20:18.065 INFO:journalctl@ceph.mds.f.smithi193.stdout:Mar 24 00:20:17 smithi193 systemd[1]: ceph-d79a5420-e971-11ee-95cd-87774f69a715@mds.f.service: Scheduled restart job, restart counter is at 2. 2024-03-24T00:20:18.065 INFO:journalctl@ceph.mds.f.smithi193.stdout:Mar 24 00:20:17 smithi193 systemd[1]: Stopped Ceph mds.f for d79a5420-e971-11ee-95cd-87774f69a715. 2024-03-24T00:20:18.065 INFO:journalctl@ceph.mds.f.smithi193.stdout:Mar 24 00:20:17 smithi193 systemd[1]: Starting Ceph mds.f for d79a5420-e971-11ee-95cd-87774f69a715... 2024-03-24T00:20:18.065 INFO:journalctl@ceph.mds.f.smithi193.stdout:Mar 24 00:20:17 smithi193 podman[71087]: 2024-03-24 00:20:17.840285857 +0000 UTC m=+0.021248667 image pull c664afe56bb01db4f9afa9532c20598e201e664f2e9ba2be50db0d52674a26d2 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc 2024-03-24T00:20:18.065 INFO:journalctl@ceph.mds.f.smithi193.stdout:Mar 24 00:20:17 smithi193 podman[71087]: 2024-03-24 00:20:17.945213851 +0000 UTC m=+0.126176660 container create 4b67230e0d857dd5235101666cd773f385288cea0ef4623a9d3c3deca41e9070 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-f, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, org.label-schema.build-date=20240319, maintainer=Guillaume Abrioux , ceph=True, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.vendor=CentOS, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, io.buildah.version=1.33.5) 2024-03-24T00:20:18.316 INFO:journalctl@ceph.mds.f.smithi193.stdout:Mar 24 00:20:18 smithi193 podman[71087]: 2024-03-24 00:20:18.197742666 +0000 UTC m=+0.378705478 container init 4b67230e0d857dd5235101666cd773f385288cea0ef4623a9d3c3deca41e9070 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-f, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, ceph=True, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, org.label-schema.build-date=20240319, org.label-schema.license=GPLv2) 2024-03-24T00:20:18.316 INFO:journalctl@ceph.mds.f.smithi193.stdout:Mar 24 00:20:18 smithi193 podman[71087]: 2024-03-24 00:20:18.200341757 +0000 UTC m=+0.381304569 container start 4b67230e0d857dd5235101666cd773f385288cea0ef4623a9d3c3deca41e9070 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-f, org.label-schema.schema-version=1.0, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, CEPH_POINT_RELEASE=, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240319, maintainer=Guillaume Abrioux ) 2024-03-24T00:20:18.316 INFO:journalctl@ceph.mds.f.smithi193.stdout:Mar 24 00:20:18 smithi193 ceph-mds[71106]: -- 172.21.15.193:0/2242309165 <== mon.1 v2:172.21.15.193:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x55c8b3ca9a40 con 0x55c8b3c7c800 2024-03-24T00:20:18.316 INFO:journalctl@ceph.mds.f.smithi193.stdout:Mar 24 00:20:18 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-f[71102]: starting mds.f at 2024-03-24T00:20:18.316 INFO:journalctl@ceph.mds.f.smithi193.stdout:Mar 24 00:20:18 smithi193 bash[71087]: 4b67230e0d857dd5235101666cd773f385288cea0ef4623a9d3c3deca41e9070 2024-03-24T00:20:18.710 INFO:journalctl@ceph.mds.f.smithi193.stdout:Mar 24 00:20:18 smithi193 systemd[1]: Started Ceph mds.f for d79a5420-e971-11ee-95cd-87774f69a715. 2024-03-24T00:20:19.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:20:18 smithi118 ceph-mon[31535]: pgmap v538: 97 pgs: 97 active+clean; 97 MiB data, 636 MiB used, 1.0 TiB / 1.0 TiB avail; 179 KiB/s rd, 657 KiB/s wr, 105 op/s 2024-03-24T00:20:19.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:20:18 smithi118 ceph-mon[31535]: daemon mds.f restarted 2024-03-24T00:20:19.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:20:18 smithi118 ceph-mon[31535]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-03-24T00:20:19.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:20:18 smithi118 ceph-mon[31535]: osdmap e93: 12 total, 12 up, 12 in 2024-03-24T00:20:19.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:20:18 smithi118 ceph-mon[31535]: mds.? [v2:172.21.15.193:6834/1699655293,v1:172.21.15.193:6835/1699655293] up:boot 2024-03-24T00:20:19.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:20:18 smithi118 ceph-mon[31535]: Standby daemon mds.l assigned to filesystem cephfs as rank 3 2024-03-24T00:20:19.197 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:20:18 smithi118 ceph-mon[31535]: fsmap cephfs:4/5 {0=b=up:active,1=g=up:active,2=c=up:active,4=e=up:active} 5 up:standby-replay 3 up:standby, 1 failed 2024-03-24T00:20:19.197 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:20:18 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-03-24T00:20:19.197 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:20:18 smithi118 ceph-mon[31535]: fsmap cephfs:5/5 4 up:standby-replay 3 up:standby 1 up:replay 4 up:active 2024-03-24T00:20:19.203 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:20:18 smithi189 ceph-mon[36187]: pgmap v538: 97 pgs: 97 active+clean; 97 MiB data, 636 MiB used, 1.0 TiB / 1.0 TiB avail; 179 KiB/s rd, 657 KiB/s wr, 105 op/s 2024-03-24T00:20:19.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:20:18 smithi189 ceph-mon[36187]: daemon mds.f restarted 2024-03-24T00:20:19.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:20:18 smithi189 ceph-mon[36187]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-03-24T00:20:19.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:20:18 smithi189 ceph-mon[36187]: osdmap e93: 12 total, 12 up, 12 in 2024-03-24T00:20:19.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:20:18 smithi189 ceph-mon[36187]: mds.? [v2:172.21.15.193:6834/1699655293,v1:172.21.15.193:6835/1699655293] up:boot 2024-03-24T00:20:19.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:20:18 smithi189 ceph-mon[36187]: Standby daemon mds.l assigned to filesystem cephfs as rank 3 2024-03-24T00:20:19.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:20:18 smithi189 ceph-mon[36187]: fsmap cephfs:4/5 {0=b=up:active,1=g=up:active,2=c=up:active,4=e=up:active} 5 up:standby-replay 3 up:standby, 1 failed 2024-03-24T00:20:19.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:20:18 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-03-24T00:20:19.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:20:18 smithi189 ceph-mon[36187]: fsmap cephfs:5/5 4 up:standby-replay 3 up:standby 1 up:replay 4 up:active 2024-03-24T00:20:19.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:20:18 smithi193 ceph-mon[36878]: pgmap v538: 97 pgs: 97 active+clean; 97 MiB data, 636 MiB used, 1.0 TiB / 1.0 TiB avail; 179 KiB/s rd, 657 KiB/s wr, 105 op/s 2024-03-24T00:20:19.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:20:18 smithi193 ceph-mon[36878]: daemon mds.f restarted 2024-03-24T00:20:19.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:20:18 smithi193 ceph-mon[36878]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-03-24T00:20:19.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:20:18 smithi193 ceph-mon[36878]: osdmap e93: 12 total, 12 up, 12 in 2024-03-24T00:20:19.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:20:18 smithi193 ceph-mon[36878]: mds.? [v2:172.21.15.193:6834/1699655293,v1:172.21.15.193:6835/1699655293] up:boot 2024-03-24T00:20:19.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:20:18 smithi193 ceph-mon[36878]: Standby daemon mds.l assigned to filesystem cephfs as rank 3 2024-03-24T00:20:19.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:20:18 smithi193 ceph-mon[36878]: fsmap cephfs:4/5 {0=b=up:active,1=g=up:active,2=c=up:active,4=e=up:active} 5 up:standby-replay 3 up:standby, 1 failed 2024-03-24T00:20:19.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:20:18 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-03-24T00:20:19.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:20:18 smithi193 ceph-mon[36878]: fsmap cephfs:5/5 4 up:standby-replay 3 up:standby 1 up:replay 4 up:active 2024-03-24T00:20:19.247 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:20:19.246+0000 7f9245e00640 1 -- 172.21.15.118:0/1482161250 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f9240104ac0 msgr2=0x7f9240104f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:20:19.247 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:20:19.246+0000 7f9245e00640 1 --2- 172.21.15.118:0/1482161250 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f9240104ac0 0x7f9240104f40 secure :-1 s=READY pgs=405 cs=0 l=1 rev1=1 crypto rx=0x7f922c009900 tx=0x7f922c02bc30 comp rx=0 tx=0).stop 2024-03-24T00:20:19.248 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:20:19.246+0000 7f9245e00640 1 -- 172.21.15.118:0/1482161250 shutdown_connections 2024-03-24T00:20:19.248 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:20:19.247+0000 7f9245e00640 1 --2- 172.21.15.118:0/1482161250 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f9240105480 0x7f924010bd10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:20:19.248 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:20:19.247+0000 7f9245e00640 1 --2- 172.21.15.118:0/1482161250 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f9240104ac0 0x7f9240104f40 unknown :-1 s=CLOSED pgs=405 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:20:19.248 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:20:19.247+0000 7f9245e00640 1 --2- 172.21.15.118:0/1482161250 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f92401038d0 0x7f9240103cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:20:19.248 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:20:19.247+0000 7f9245e00640 1 -- 172.21.15.118:0/1482161250 >> 172.21.15.118:0/1482161250 conn(0x7f92400fee60 msgr2=0x7f9240101280 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:20:19.248 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:20:19.247+0000 7f9245e00640 1 -- 172.21.15.118:0/1482161250 shutdown_connections 2024-03-24T00:20:19.248 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:20:19.247+0000 7f9245e00640 1 -- 172.21.15.118:0/1482161250 wait complete. 2024-03-24T00:20:19.250 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:20:19.247+0000 7f9245e00640 1 Processor -- start 2024-03-24T00:20:19.250 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:20:19.248+0000 7f9245e00640 1 -- start start 2024-03-24T00:20:19.250 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:20:19.248+0000 7f9245e00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f92401038d0 0x7f9240199b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:20:19.250 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:20:19.248+0000 7f9245e00640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f9240104ac0 0x7f924019a040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:20:19.250 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:20:19.248+0000 7f9245e00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f9240105480 0x7f92401a10b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:20:19.250 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:20:19.248+0000 7f9245e00640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f924010f580 con 0x7f9240104ac0 2024-03-24T00:20:19.251 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:20:19.248+0000 7f9245e00640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f924010f400 con 0x7f92401038d0 2024-03-24T00:20:19.251 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:20:19.248+0000 7f9245e00640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f924010f700 con 0x7f9240105480 2024-03-24T00:20:19.251 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:20:19.248+0000 7f923ea00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f92401038d0 0x7f9240199b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:20:19.251 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:20:19.248+0000 7f923f400640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f9240105480 0x7f92401a10b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:20:19.251 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:20:19.248+0000 7f923ea00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f92401038d0 0x7f9240199b00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.118:36898/0 (socket says 172.21.15.118:36898) 2024-03-24T00:20:19.251 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:20:19.248+0000 7f923ea00640 1 -- 172.21.15.118:0/2209975981 learned_addr learned my addr 172.21.15.118:0/2209975981 (peer_addr_for_me v2:172.21.15.118:0/0) 2024-03-24T00:20:19.251 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:20:19.248+0000 7f923e000640 1 --2- 172.21.15.118:0/2209975981 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f9240104ac0 0x7f924019a040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:20:19.251 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:20:19.249+0000 7f923ea00640 1 -- 172.21.15.118:0/2209975981 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f9240105480 msgr2=0x7f92401a10b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:20:19.251 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:20:19.249+0000 7f923ea00640 1 --2- 172.21.15.118:0/2209975981 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f9240105480 0x7f92401a10b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:20:19.251 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:20:19.249+0000 7f923ea00640 1 -- 172.21.15.118:0/2209975981 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f9240104ac0 msgr2=0x7f924019a040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:20:19.251 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:20:19.249+0000 7f923ea00640 1 --2- 172.21.15.118:0/2209975981 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f9240104ac0 0x7f924019a040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:20:19.251 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:20:19.249+0000 7f923ea00640 1 -- 172.21.15.118:0/2209975981 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f92401a1720 con 0x7f92401038d0 2024-03-24T00:20:19.251 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:20:19.249+0000 7f923f400640 1 --2- 172.21.15.118:0/2209975981 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f9240105480 0x7f92401a10b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T00:20:19.251 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:20:19.249+0000 7f923ea00640 1 --2- 172.21.15.118:0/2209975981 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f92401038d0 0x7f9240199b00 secure :-1 s=READY pgs=404 cs=0 l=1 rev1=1 crypto rx=0x7f923000e9c0 tx=0x7f923000ee80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:20:19.252 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:20:19.249+0000 7f9244a00640 1 -- 172.21.15.118:0/2209975981 <== mon.2 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f923000cd80 con 0x7f92401038d0 2024-03-24T00:20:19.252 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:20:19.249+0000 7f9244a00640 1 -- 172.21.15.118:0/2209975981 <== mon.2 v2:172.21.15.189:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f92300035f0 con 0x7f92401038d0 2024-03-24T00:20:19.252 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:20:19.249+0000 7f9245e00640 1 -- 172.21.15.118:0/2209975981 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f92401a1a10 con 0x7f92401038d0 2024-03-24T00:20:19.252 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:20:19.249+0000 7f9244a00640 1 -- 172.21.15.118:0/2209975981 <== mon.2 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9230010690 con 0x7f92401038d0 2024-03-24T00:20:19.252 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:20:19.249+0000 7f9245e00640 1 -- 172.21.15.118:0/2209975981 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f92401a1f30 con 0x7f92401038d0 2024-03-24T00:20:19.253 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:20:19.252+0000 7f9244a00640 1 -- 172.21.15.118:0/2209975981 <== mon.2 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 20) v1 ==== 137291+0+0 (secure 0 0 0) 0x7f92300037a0 con 0x7f92401038d0 2024-03-24T00:20:19.253 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:20:19.252+0000 7f9248c10740 1 Processor -- start 2024-03-24T00:20:19.253 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:20:19.253+0000 7f9248c10740 1 -- start start 2024-03-24T00:20:19.255 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:20:19.253+0000 7f9248c10740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55f183d191e0 0x55f183d19600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:20:19.255 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:20:19.253+0000 7f9248c10740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55f183d19bd0 0x55f183d1e6d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:20:19.255 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:20:19.253+0000 7f9248c10740 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55f183d1ed40 0x55f183d211d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:20:19.255 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:20:19.253+0000 7f9248c10740 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f183cdc3b0 con 0x55f183d1ed40 2024-03-24T00:20:19.255 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:20:19.253+0000 7f9248c10740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f183c7e830 con 0x55f183d19bd0 2024-03-24T00:20:19.255 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:20:19.253+0000 7f9248c10740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f183c275d0 con 0x55f183d191e0 2024-03-24T00:20:19.255 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:20:19.253+0000 7f923f400640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55f183d191e0 0x55f183d19600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:20:19.255 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:20:19.253+0000 7f923f400640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55f183d191e0 0x55f183d19600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.118:43890/0 (socket says 172.21.15.118:43890) 2024-03-24T00:20:19.255 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:20:19.253+0000 7f923f400640 1 -- 172.21.15.118:0/1070175218 learned_addr learned my addr 172.21.15.118:0/1070175218 (peer_addr_for_me v2:172.21.15.118:0/0) 2024-03-24T00:20:19.256 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:20:19.253+0000 7f923ea00640 1 --2- 172.21.15.118:0/1070175218 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55f183d19bd0 0x55f183d1e6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:20:19.256 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:20:19.253+0000 7f923e000640 1 --2- 172.21.15.118:0/1070175218 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55f183d1ed40 0x55f183d211d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:20:19.256 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:20:19.254+0000 7f923f400640 1 -- 172.21.15.118:0/1070175218 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55f183d19bd0 msgr2=0x55f183d1e6d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:20:19.256 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:20:19.254+0000 7f923f400640 1 --2- 172.21.15.118:0/1070175218 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55f183d19bd0 0x55f183d1e6d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:20:19.256 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:20:19.254+0000 7f923f400640 1 -- 172.21.15.118:0/1070175218 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55f183d1ed40 msgr2=0x55f183d211d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:20:19.256 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:20:19.254+0000 7f923f400640 1 --2- 172.21.15.118:0/1070175218 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55f183d1ed40 0x55f183d211d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:20:19.256 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:20:19.254+0000 7f923f400640 1 -- 172.21.15.118:0/1070175218 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55f183c3f2c0 con 0x55f183d191e0 2024-03-24T00:20:19.256 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:20:19.254+0000 7f923e000640 1 --2- 172.21.15.118:0/1070175218 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55f183d1ed40 0x55f183d211d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T00:20:19.256 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:20:19.254+0000 7f923f400640 1 --2- 172.21.15.118:0/1070175218 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55f183d191e0 0x55f183d19600 secure :-1 s=READY pgs=406 cs=0 l=1 rev1=1 crypto rx=0x7f92380117e0 tx=0x7f9238011ca0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=f12571a7ddff9c0e server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:20:19.256 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:20:19.254+0000 7f9244a00640 1 --2- 172.21.15.118:0/2209975981 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7f92240a28a0 0x7f92240a4d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:20:19.256 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:20:19.254+0000 7f922be00640 1 -- 172.21.15.118:0/1070175218 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9238011310 con 0x55f183d191e0 2024-03-24T00:20:19.256 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:20:19.254+0000 7f922be00640 1 -- 172.21.15.118:0/1070175218 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f9238003e50 con 0x55f183d191e0 2024-03-24T00:20:19.256 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:20:19.254+0000 7f922be00640 1 -- 172.21.15.118:0/1070175218 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9238004620 con 0x55f183d191e0 2024-03-24T00:20:19.256 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:20:19.254+0000 7f9244a00640 1 -- 172.21.15.118:0/2209975981 <== mon.2 v2:172.21.15.189:3300/0 5 ==== osd_map(93..93 src has 1..93) v4 ==== 8535+0+0 (secure 0 0 0) 0x7f92300cefa0 con 0x7f92401038d0 2024-03-24T00:20:19.257 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:20:19.254+0000 7f923ea00640 1 --2- 172.21.15.118:0/2209975981 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7f92240a28a0 0x7f92240a4d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:20:19.259 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:20:19.255+0000 7f923ea00640 1 --2- 172.21.15.118:0/2209975981 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7f92240a28a0 0x7f92240a4d60 secure :-1 s=READY pgs=352 cs=0 l=1 rev1=1 crypto rx=0x7f9230098180 tx=0x7f92300d29e0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:20:19.259 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:20:19.255+0000 7f9248c10740 1 -- 172.21.15.118:0/1070175218 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55f183d191e0 msgr2=0x55f183d19600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:20:19.259 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:20:19.255+0000 7f9248c10740 1 --2- 172.21.15.118:0/1070175218 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55f183d191e0 0x55f183d19600 secure :-1 s=READY pgs=406 cs=0 l=1 rev1=1 crypto rx=0x7f92380117e0 tx=0x7f9238011ca0 comp rx=0 tx=0).stop 2024-03-24T00:20:19.259 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:20:19.255+0000 7f9248c10740 1 -- 172.21.15.118:0/1070175218 shutdown_connections 2024-03-24T00:20:19.260 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:20:19.255+0000 7f9248c10740 1 --2- 172.21.15.118:0/1070175218 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55f183d1ed40 0x55f183d211d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:20:19.260 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:20:19.255+0000 7f9248c10740 1 --2- 172.21.15.118:0/1070175218 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55f183d19bd0 0x55f183d1e6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:20:19.260 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:20:19.255+0000 7f9248c10740 1 --2- 172.21.15.118:0/1070175218 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55f183d191e0 0x55f183d19600 unknown :-1 s=CLOSED pgs=406 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:20:19.260 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:20:19.255+0000 7f9248c10740 1 -- 172.21.15.118:0/1070175218 >> 172.21.15.118:0/1070175218 conn(0x55f183d14750 msgr2=0x55f183d14b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:20:19.260 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:20:19.255+0000 7f9248c10740 1 -- 172.21.15.118:0/1070175218 shutdown_connections 2024-03-24T00:20:19.260 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:20:19.255+0000 7f9248c10740 1 -- 172.21.15.118:0/1070175218 wait complete. 2024-03-24T00:20:19.260 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:20:19.256+0000 7f9248c10740 1 -- start start 2024-03-24T00:20:19.260 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:20:19.256+0000 7f9248c10740 1 Processor -- start 2024-03-24T00:20:19.260 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:20:19.257+0000 7f9248c10740 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55f183d191e0 0x55f183d23420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:20:19.260 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:20:19.257+0000 7f9248c10740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55f183d19bd0 0x55f183d23960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:20:19.260 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:20:19.257+0000 7f9248c10740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55f183d1ed40 0x55f183d2a930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:20:19.260 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:20:19.257+0000 7f9248c10740 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f183c3aa20 con 0x55f183d191e0 2024-03-24T00:20:19.260 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:20:19.257+0000 7f9248c10740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f183c7a8d0 con 0x55f183d19bd0 2024-03-24T00:20:19.261 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:20:19.257+0000 7f9248c10740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f183d21cf0 con 0x55f183d1ed40 2024-03-24T00:20:19.261 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:20:19.257+0000 7f9248c10740 10 client.0 _ll_register_callbacks cb 0x55f183c771c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T00:20:19.261 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:20:19.257+0000 7f923e000640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55f183d19bd0 0x55f183d23960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:20:19.261 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:20:19.257+0000 7f923e000640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55f183d19bd0 0x55f183d23960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.118:36922/0 (socket says 172.21.15.118:36922) 2024-03-24T00:20:19.261 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:20:19.257+0000 7f923e000640 1 -- 172.21.15.118:0/242220422 learned_addr learned my addr 172.21.15.118:0/242220422 (peer_addr_for_me v2:172.21.15.118:0/0) 2024-03-24T00:20:19.261 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:20:19.257+0000 7f923f400640 1 --2- 172.21.15.118:0/242220422 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55f183d1ed40 0x55f183d2a930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:20:19.261 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:20:19.257+0000 7f923f400640 1 --2- 172.21.15.118:0/242220422 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55f183d191e0 0x55f183d23420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:20:19.261 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:20:19.258+0000 7f923e000640 1 -- 172.21.15.118:0/242220422 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55f183d1ed40 msgr2=0x55f183d2a930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:20:19.261 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:20:19.258+0000 7f923e000640 1 --2- 172.21.15.118:0/242220422 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55f183d1ed40 0x55f183d2a930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:20:19.261 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:20:19.258+0000 7f923e000640 1 -- 172.21.15.118:0/242220422 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55f183d191e0 msgr2=0x55f183d23420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:20:19.261 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:20:19.258+0000 7f923e000640 1 --2- 172.21.15.118:0/242220422 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55f183d191e0 0x55f183d23420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:20:19.261 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:20:19.258+0000 7f923e000640 1 -- 172.21.15.118:0/242220422 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55f183993210 con 0x55f183d19bd0 2024-03-24T00:20:19.261 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:20:19.258+0000 7f923f400640 1 --2- 172.21.15.118:0/242220422 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55f183d191e0 0x55f183d23420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T00:20:19.262 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:20:19.258+0000 7f923e000640 1 --2- 172.21.15.118:0/242220422 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55f183d19bd0 0x55f183d23960 secure :-1 s=READY pgs=405 cs=0 l=1 rev1=1 crypto rx=0x7f922c005ab0 tx=0x7f922c0042f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=9f8bc204a2b113a5 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:20:19.262 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:20:19.258+0000 7f9228c00640 10 client.0 ms_handle_connect on v2:172.21.15.189:3300/0 2024-03-24T00:20:19.263 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:20:19.261+0000 7f9228c00640 1 -- 172.21.15.118:0/242220422 <== mon.2 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f922c005d50 con 0x55f183d19bd0 2024-03-24T00:20:19.263 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:20:19.261+0000 7f9228c00640 1 -- 172.21.15.118:0/242220422 <== mon.2 v2:172.21.15.189:3300/0 2 ==== osd_map(93..93 src has 1..93) v4 ==== 8535+0+0 (secure 0 0 0) 0x7f922c0048d0 con 0x55f183d19bd0 2024-03-24T00:20:19.263 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:20:19.261+0000 7f9248c10740 1 -- 172.21.15.118:0/242220422 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f9238003e50 con 0x55f183d19bd0 2024-03-24T00:20:19.263 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:20:19.261+0000 7f9228c00640 1 -- 172.21.15.118:0/242220422 <== mon.2 v2:172.21.15.189:3300/0 3 ==== mon_get_version_reply(handle=1 version=45) v2 ==== 24+0+0 (secure 0 0 0) 0x7f922c0044e0 con 0x55f183d19bd0 2024-03-24T00:20:19.263 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:20:19.261+0000 7f9248c10740 10 client.26636 fetch_fsmap learned FSMap version 45 2024-03-24T00:20:19.264 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:20:19.262+0000 7f9248c10740 1 -- 172.21.15.118:0/242220422 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,fsmap=45,monmap=4+}) v3 -- 0x55f183a5f620 con 0x55f183d19bd0 2024-03-24T00:20:19.264 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:20:19.262+0000 7f9228c00640 1 -- 172.21.15.118:0/242220422 <== mon.2 v2:172.21.15.189:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f9238003e50 con 0x55f183d19bd0 2024-03-24T00:20:19.264 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:20:19.262+0000 7f9228c00640 1 -- 172.21.15.118:0/242220422 <== mon.2 v2:172.21.15.189:3300/0 5 ==== fsmap(e 45) v1 ==== 7942+0+0 (secure 0 0 0) 0x7f922c003680 con 0x55f183d19bd0 2024-03-24T00:20:19.264 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:20:19.262+0000 7f9248c10740 10 client.26636 fetch_fsmap finished waiting for FSMap version 45 2024-03-24T00:20:19.264 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:20:19.262+0000 7f9248c10740 10 client.26636 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-24T00:20:19.264 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:20:19.262+0000 7f9248c10740 20 client.26636 populate_metadata read hostname 'smithi118' 2024-03-24T00:20:19.264 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:20:19.262+0000 7f9248c10740 1 --2- 172.21.15.118:0/242220422 >> [v2:172.21.15.189:6838/4074484168,v1:172.21.15.189:6839/4074484168] conn(0x55f183d2f160 0x55f183d31570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:20:19.264 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:20:19.262+0000 7f9248c10740 4 client.26636 mds_command: new command op to 24569 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-24T00:20:19.264 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:20:19.262+0000 7f9248c10740 1 -- 172.21.15.118:0/242220422 --> [v2:172.21.15.189:6838/4074484168,v1:172.21.15.189:6839/4074484168] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55f183c8bd70 con 0x55f183d2f160 2024-03-24T00:20:19.264 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:20:19.262+0000 7f923f400640 1 --2- 172.21.15.118:0/242220422 >> [v2:172.21.15.189:6838/4074484168,v1:172.21.15.189:6839/4074484168] conn(0x55f183d2f160 0x55f183d31570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:20:19.264 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:20:19.263+0000 7f923f400640 1 --2- 172.21.15.118:0/242220422 >> [v2:172.21.15.189:6838/4074484168,v1:172.21.15.189:6839/4074484168] conn(0x55f183d2f160 0x55f183d31570 crc :-1 s=READY pgs=147 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=710e0bd3de1ea37e server_cookie=d0faad102f089d79 in_seq=0 out_seq=0 2024-03-24T00:20:19.264 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:20:19.263+0000 7f9228c00640 10 client.26636 ms_handle_connect on v2:172.21.15.189:6838/4074484168 2024-03-24T00:20:19.266 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:20:19.263+0000 7f9228c00640 1 -- 172.21.15.118:0/242220422 <== mds.0 v2:172.21.15.189:6838/4074484168 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x55f183c8bd70 con 0x55f183d2f160 2024-03-24T00:20:19.266 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:20:19.263+0000 7f9228c00640 10 client.26636 handle_command_reply: tid=0 2024-03-24T00:20:19.266 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:20:19.264+0000 7f9248c10740 1 client.26636 shutdown 2024-03-24T00:20:19.266 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:20:19.264+0000 7f9248c10740 1 -- 172.21.15.118:0/242220422 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55f183d19bd0 msgr2=0x55f183d23960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:20:19.266 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:20:19.264+0000 7f9248c10740 1 --2- 172.21.15.118:0/242220422 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55f183d19bd0 0x55f183d23960 secure :-1 s=READY pgs=405 cs=0 l=1 rev1=1 crypto rx=0x7f922c005ab0 tx=0x7f922c0042f0 comp rx=0 tx=0).stop 2024-03-24T00:20:19.266 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:20:19.264+0000 7f9248c10740 1 -- 172.21.15.118:0/242220422 shutdown_connections 2024-03-24T00:20:19.266 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:20:19.264+0000 7f9248c10740 1 --2- 172.21.15.118:0/242220422 >> [v2:172.21.15.189:6838/4074484168,v1:172.21.15.189:6839/4074484168] conn(0x55f183d2f160 0x55f183d31570 crc :-1 s=READY pgs=147 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:20:19.266 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:20:19.264+0000 7f9248c10740 1 --2- 172.21.15.118:0/242220422 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55f183d1ed40 0x55f183d2a930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:20:19.266 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:20:19.264+0000 7f9248c10740 1 --2- 172.21.15.118:0/242220422 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55f183d19bd0 0x55f183d23960 unknown :-1 s=CLOSED pgs=405 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:20:19.266 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:20:19.264+0000 7f9248c10740 1 --2- 172.21.15.118:0/242220422 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55f183d191e0 0x55f183d23420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:20:19.266 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:20:19.264+0000 7f9228c00640 0 client.26636 ms_handle_reset on v2:172.21.15.189:6838/4074484168 2024-03-24T00:20:19.266 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:20:19.264+0000 7f9248c10740 1 -- 172.21.15.118:0/242220422 >> 172.21.15.118:0/242220422 conn(0x55f183d14750 msgr2=0x55f183d13950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:20:19.267 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:20:19.264+0000 7f9248c10740 1 -- 172.21.15.118:0/242220422 shutdown_connections 2024-03-24T00:20:19.267 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:20:19.264+0000 7f9248c10740 1 -- 172.21.15.118:0/242220422 wait complete. 2024-03-24T00:20:19.267 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:20:19.264+0000 7f9248c10740 20 client.26636 trim_cache size 0 max 16384 2024-03-24T00:20:19.281 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:20:19.279+0000 7f9248c10740 1 Processor -- start 2024-03-24T00:20:19.281 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:20:19.279+0000 7f9248c10740 1 -- start start 2024-03-24T00:20:19.281 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:20:19.279+0000 7f9248c10740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55f183d191e0 0x55f183d180c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:20:19.281 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:20:19.279+0000 7f9248c10740 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55f183d19bd0 0x55f183d2a8f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:20:19.281 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:20:19.279+0000 7f9248c10740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55f183d1ed40 0x55f183d35e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:20:19.281 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:20:19.279+0000 7f9248c10740 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f92401ab500 con 0x55f183d19bd0 2024-03-24T00:20:19.281 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:20:19.279+0000 7f9248c10740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f92401ac850 con 0x55f183d1ed40 2024-03-24T00:20:19.281 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:20:19.279+0000 7f9248c10740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f92401abb60 con 0x55f183d191e0 2024-03-24T00:20:19.281 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:20:19.280+0000 7f923f400640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55f183d191e0 0x55f183d180c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:20:19.281 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:20:19.280+0000 7f923f400640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55f183d191e0 0x55f183d180c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.118:43916/0 (socket says 172.21.15.118:43916) 2024-03-24T00:20:19.281 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:20:19.280+0000 7f923f400640 1 -- 172.21.15.118:0/921383032 learned_addr learned my addr 172.21.15.118:0/921383032 (peer_addr_for_me v2:172.21.15.118:0/0) 2024-03-24T00:20:19.281 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:20:19.280+0000 7f923e000640 1 --2- 172.21.15.118:0/921383032 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55f183d19bd0 0x55f183d2a8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:20:19.282 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:20:19.280+0000 7f923f400640 1 --2- 172.21.15.118:0/921383032 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55f183d1ed40 0x55f183d35e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:20:19.282 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:20:19.280+0000 7f923f400640 1 -- 172.21.15.118:0/921383032 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55f183d1ed40 msgr2=0x55f183d35e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:20:19.282 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:20:19.280+0000 7f923f400640 1 --2- 172.21.15.118:0/921383032 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55f183d1ed40 0x55f183d35e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:20:19.282 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:20:19.280+0000 7f923f400640 1 -- 172.21.15.118:0/921383032 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55f183d19bd0 msgr2=0x55f183d2a8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:20:19.282 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:20:19.280+0000 7f923f400640 1 --2- 172.21.15.118:0/921383032 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55f183d19bd0 0x55f183d2a8f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:20:19.282 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:20:19.280+0000 7f923f400640 1 -- 172.21.15.118:0/921383032 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55f183d24410 con 0x55f183d191e0 2024-03-24T00:20:19.282 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:20:19.280+0000 7f923f400640 1 --2- 172.21.15.118:0/921383032 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55f183d191e0 0x55f183d180c0 secure :-1 s=READY pgs=407 cs=0 l=1 rev1=1 crypto rx=0x7f923800f790 tx=0x7f9238009500 comp rx=0 tx=0).ready entity=mon.1 client_cookie=2dfed651335618e4 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:20:19.290 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:20:19.281+0000 7f922b400640 1 -- 172.21.15.118:0/921383032 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f923802e0b0 con 0x55f183d191e0 2024-03-24T00:20:19.290 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:20:19.281+0000 7f922b400640 1 -- 172.21.15.118:0/921383032 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f923802e250 con 0x55f183d191e0 2024-03-24T00:20:19.290 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:20:19.281+0000 7f922b400640 1 -- 172.21.15.118:0/921383032 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f923802eaa0 con 0x55f183d191e0 2024-03-24T00:20:19.290 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:20:19.281+0000 7f9248c10740 1 -- 172.21.15.118:0/921383032 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55f183d191e0 msgr2=0x55f183d180c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:20:19.290 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:20:19.281+0000 7f9248c10740 1 --2- 172.21.15.118:0/921383032 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55f183d191e0 0x55f183d180c0 secure :-1 s=READY pgs=407 cs=0 l=1 rev1=1 crypto rx=0x7f923800f790 tx=0x7f9238009500 comp rx=0 tx=0).stop 2024-03-24T00:20:19.290 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:20:19.281+0000 7f9248c10740 1 -- 172.21.15.118:0/921383032 shutdown_connections 2024-03-24T00:20:19.290 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:20:19.281+0000 7f9248c10740 1 --2- 172.21.15.118:0/921383032 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55f183d1ed40 0x55f183d35e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:20:19.290 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:20:19.281+0000 7f9248c10740 1 --2- 172.21.15.118:0/921383032 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55f183d19bd0 0x55f183d2a8f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:20:19.290 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:20:19.281+0000 7f9248c10740 1 --2- 172.21.15.118:0/921383032 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55f183d191e0 0x55f183d180c0 unknown :-1 s=CLOSED pgs=407 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:20:19.290 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:20:19.281+0000 7f9248c10740 1 -- 172.21.15.118:0/921383032 >> 172.21.15.118:0/921383032 conn(0x55f183d14750 msgr2=0x55f183d179b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:20:19.291 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:20:19.282+0000 7f9248c10740 1 -- 172.21.15.118:0/921383032 shutdown_connections 2024-03-24T00:20:19.291 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:20:19.282+0000 7f9248c10740 1 -- 172.21.15.118:0/921383032 wait complete. 2024-03-24T00:20:19.291 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:20:19.282+0000 7f9248c10740 1 -- start start 2024-03-24T00:20:19.291 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:20:19.282+0000 7f9248c10740 1 Processor -- start 2024-03-24T00:20:19.291 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:20:19.283+0000 7f9248c10740 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55f183d191e0 0x55f183d2a380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:20:19.291 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:20:19.283+0000 7f9248c10740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55f183d19bd0 0x55f183d2a8c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:20:19.291 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:20:19.283+0000 7f9248c10740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55f183d1ed40 0x55f183d28a60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:20:19.291 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:20:19.283+0000 7f9248c10740 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f92401a9c80 con 0x55f183d191e0 2024-03-24T00:20:19.291 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:20:19.283+0000 7f9248c10740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f92401ac1f0 con 0x55f183d1ed40 2024-03-24T00:20:19.291 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:20:19.283+0000 7f9248c10740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f92401a9040 con 0x55f183d19bd0 2024-03-24T00:20:19.291 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:20:19.283+0000 7f9248c10740 10 client.0 _ll_register_callbacks cb 0x7f9230005e30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T00:20:19.291 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:20:19.283+0000 7f923e000640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55f183d19bd0 0x55f183d2a8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:20:19.292 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:20:19.283+0000 7f923e000640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55f183d19bd0 0x55f183d2a8c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.118:43928/0 (socket says 172.21.15.118:43928) 2024-03-24T00:20:19.292 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:20:19.283+0000 7f923e000640 1 -- 172.21.15.118:0/961510185 learned_addr learned my addr 172.21.15.118:0/961510185 (peer_addr_for_me v2:172.21.15.118:0/0) 2024-03-24T00:20:19.292 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:20:19.283+0000 7f923f400640 1 --2- 172.21.15.118:0/961510185 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55f183d191e0 0x55f183d2a380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:20:19.292 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:20:19.283+0000 7f923f400640 1 --2- 172.21.15.118:0/961510185 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55f183d1ed40 0x55f183d28a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:20:19.292 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:20:19.283+0000 7f923e000640 1 -- 172.21.15.118:0/961510185 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55f183d1ed40 msgr2=0x55f183d28a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:20:19.292 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:20:19.283+0000 7f923e000640 1 --2- 172.21.15.118:0/961510185 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55f183d1ed40 0x55f183d28a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:20:19.292 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:20:19.283+0000 7f923e000640 1 -- 172.21.15.118:0/961510185 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55f183d191e0 msgr2=0x55f183d2a380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:20:19.292 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:20:19.283+0000 7f923e000640 1 --2- 172.21.15.118:0/961510185 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55f183d191e0 0x55f183d2a380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:20:19.292 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:20:19.283+0000 7f923e000640 1 -- 172.21.15.118:0/961510185 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55f183d29290 con 0x55f183d19bd0 2024-03-24T00:20:19.292 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:20:19.283+0000 7f923f400640 1 --2- 172.21.15.118:0/961510185 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55f183d1ed40 0x55f183d28a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-24T00:20:19.292 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:20:19.283+0000 7f923e000640 1 --2- 172.21.15.118:0/961510185 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55f183d19bd0 0x55f183d2a8c0 secure :-1 s=READY pgs=408 cs=0 l=1 rev1=1 crypto rx=0x7f922c02c7b0 tx=0x7f922c009850 comp rx=0 tx=0).ready entity=mon.1 client_cookie=5fa5dc5005b0497b server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:20:19.292 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:20:19.283+0000 7f9228c00640 10 client.0 ms_handle_connect on v2:172.21.15.193:3300/0 2024-03-24T00:20:19.292 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:20:19.284+0000 7f9228c00640 1 -- 172.21.15.118:0/961510185 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f922c056070 con 0x55f183d19bd0 2024-03-24T00:20:19.292 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:20:19.284+0000 7f9248c10740 1 -- 172.21.15.118:0/961510185 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f923802e250 con 0x55f183d19bd0 2024-03-24T00:20:19.292 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:20:19.284+0000 7f9228c00640 1 -- 172.21.15.118:0/961510185 <== mon.1 v2:172.21.15.193:3300/0 2 ==== osd_map(93..93 src has 1..93) v4 ==== 8535+0+0 (secure 0 0 0) 0x7f922c004eb0 con 0x55f183d19bd0 2024-03-24T00:20:19.293 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:20:19.284+0000 7f9228c00640 1 -- 172.21.15.118:0/961510185 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_get_version_reply(handle=1 version=45) v2 ==== 24+0+0 (secure 0 0 0) 0x7f922c000e60 con 0x55f183d19bd0 2024-03-24T00:20:19.293 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:20:19.284+0000 7f9248c10740 10 client.26656 fetch_fsmap learned FSMap version 45 2024-03-24T00:20:19.293 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:20:19.284+0000 7f9248c10740 1 -- 172.21.15.118:0/961510185 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,fsmap=45,monmap=4+}) v3 -- 0x55f183d93ed0 con 0x55f183d19bd0 2024-03-24T00:20:19.293 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:20:19.284+0000 7f9228c00640 1 -- 172.21.15.118:0/961510185 <== mon.1 v2:172.21.15.193:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f923802e250 con 0x55f183d19bd0 2024-03-24T00:20:19.293 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:20:19.284+0000 7f9228c00640 1 -- 172.21.15.118:0/961510185 <== mon.1 v2:172.21.15.193:3300/0 5 ==== fsmap(e 45) v1 ==== 7942+0+0 (secure 0 0 0) 0x7f922c04a480 con 0x55f183d19bd0 2024-03-24T00:20:19.293 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:20:19.284+0000 7f9248c10740 10 client.26656 fetch_fsmap finished waiting for FSMap version 45 2024-03-24T00:20:19.293 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:20:19.284+0000 7f9248c10740 10 client.26656 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-24T00:20:19.293 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:20:19.284+0000 7f9248c10740 20 client.26656 populate_metadata read hostname 'smithi118' 2024-03-24T00:20:19.293 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:20:19.284+0000 7f9248c10740 1 --2- 172.21.15.118:0/961510185 >> [v2:172.21.15.189:6838/4074484168,v1:172.21.15.189:6839/4074484168] conn(0x55f183d94770 0x55f183d96b90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:20:19.293 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:20:19.284+0000 7f9248c10740 4 client.26656 mds_command: new command op to 24569 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-03-24T00:20:19.293 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:20:19.284+0000 7f9248c10740 1 -- 172.21.15.118:0/961510185 --> [v2:172.21.15.189:6838/4074484168,v1:172.21.15.189:6839/4074484168] -- command(tid 0: {"prefix": "flush journal"}) v1 -- 0x55f183c860c0 con 0x55f183d94770 2024-03-24T00:20:19.293 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:20:19.285+0000 7f923f400640 1 --2- 172.21.15.118:0/961510185 >> [v2:172.21.15.189:6838/4074484168,v1:172.21.15.189:6839/4074484168] conn(0x55f183d94770 0x55f183d96b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:20:19.293 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:20:19.285+0000 7f9228c00640 10 client.26656 ms_handle_connect on v2:172.21.15.189:6838/4074484168 2024-03-24T00:20:19.293 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:20:19.285+0000 7f923f400640 1 --2- 172.21.15.118:0/961510185 >> [v2:172.21.15.189:6838/4074484168,v1:172.21.15.189:6839/4074484168] conn(0x55f183d94770 0x55f183d96b90 crc :-1 s=READY pgs=148 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=4c61e4b02c537fd2 server_cookie=88b1b52ff31847a2 in_seq=0 out_seq=0 2024-03-24T00:20:20.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:20:20 smithi118 ceph-mon[31535]: mds.? [v2:172.21.15.193:6832/3097400619,v1:172.21.15.193:6833/3097400619] up:resolve 2024-03-24T00:20:20.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:20:20 smithi118 ceph-mon[31535]: fsmap cephfs:5/5 4 up:standby-replay 3 up:standby 1 up:resolve 4 up:active 2024-03-24T00:20:20.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:20:20 smithi118 ceph-mon[31535]: pgmap v540: 97 pgs: 97 active+clean; 99 MiB data, 637 MiB used, 1.0 TiB / 1.0 TiB avail; 98 KiB/s rd, 673 KiB/s wr, 43 op/s 2024-03-24T00:20:20.705 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:20:20 smithi189 ceph-mon[36187]: mds.? [v2:172.21.15.193:6832/3097400619,v1:172.21.15.193:6833/3097400619] up:resolve 2024-03-24T00:20:20.705 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:20:20 smithi189 ceph-mon[36187]: fsmap cephfs:5/5 4 up:standby-replay 3 up:standby 1 up:resolve 4 up:active 2024-03-24T00:20:20.705 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:20:20 smithi189 ceph-mon[36187]: pgmap v540: 97 pgs: 97 active+clean; 99 MiB data, 637 MiB used, 1.0 TiB / 1.0 TiB avail; 98 KiB/s rd, 673 KiB/s wr, 43 op/s 2024-03-24T00:20:20.710 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:20:20 smithi193 ceph-mon[36878]: mds.? [v2:172.21.15.193:6832/3097400619,v1:172.21.15.193:6833/3097400619] up:resolve 2024-03-24T00:20:20.711 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:20:20 smithi193 ceph-mon[36878]: fsmap cephfs:5/5 4 up:standby-replay 3 up:standby 1 up:resolve 4 up:active 2024-03-24T00:20:20.711 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:20:20 smithi193 ceph-mon[36878]: pgmap v540: 97 pgs: 97 active+clean; 99 MiB data, 637 MiB used, 1.0 TiB / 1.0 TiB avail; 98 KiB/s rd, 673 KiB/s wr, 43 op/s 2024-03-24T00:20:21.303 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:20:21.302+0000 7f9228c00640 1 -- 172.21.15.118:0/961510185 <== mds.0 v2:172.21.15.189:6838/4074484168 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+44 (crc 0 0 0) 0x55f183c860c0 con 0x55f183d94770 2024-03-24T00:20:21.303 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:20:21.302+0000 7f9228c00640 10 client.26656 handle_command_reply: tid=0 2024-03-24T00:20:21.303 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:20:21.302+0000 7f9248c10740 1 client.26656 shutdown 2024-03-24T00:20:21.304 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:20:21.303+0000 7f9248c10740 1 -- 172.21.15.118:0/961510185 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55f183d19bd0 msgr2=0x55f183d2a8c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:20:21.304 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:20:21.303+0000 7f9248c10740 1 --2- 172.21.15.118:0/961510185 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55f183d19bd0 0x55f183d2a8c0 secure :-1 s=READY pgs=408 cs=0 l=1 rev1=1 crypto rx=0x7f922c02c7b0 tx=0x7f922c009850 comp rx=0 tx=0).stop 2024-03-24T00:20:21.304 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:20:21.303+0000 7f9248c10740 1 -- 172.21.15.118:0/961510185 shutdown_connections 2024-03-24T00:20:21.304 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:20:21.303+0000 7f9248c10740 1 --2- 172.21.15.118:0/961510185 >> [v2:172.21.15.189:6838/4074484168,v1:172.21.15.189:6839/4074484168] conn(0x55f183d94770 0x55f183d96b90 crc :-1 s=READY pgs=148 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:20:21.305 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:20:21.303+0000 7f9248c10740 1 --2- 172.21.15.118:0/961510185 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55f183d1ed40 0x55f183d28a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:20:21.305 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:20:21.303+0000 7f9248c10740 1 --2- 172.21.15.118:0/961510185 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55f183d19bd0 0x55f183d2a8c0 unknown :-1 s=CLOSED pgs=408 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:20:21.305 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:20:21.303+0000 7f9248c10740 1 --2- 172.21.15.118:0/961510185 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55f183d191e0 0x55f183d2a380 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:20:21.305 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:20:21.303+0000 7f9248c10740 1 -- 172.21.15.118:0/961510185 >> 172.21.15.118:0/961510185 conn(0x55f183d14750 msgr2=0x55f183d18240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:20:21.305 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:20:21.303+0000 7f9228c00640 0 client.26656 ms_handle_reset on v2:172.21.15.189:6838/4074484168 2024-03-24T00:20:21.305 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:20:21.303+0000 7f9248c10740 1 -- 172.21.15.118:0/961510185 shutdown_connections 2024-03-24T00:20:21.305 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:20:21.303+0000 7f9248c10740 1 -- 172.21.15.118:0/961510185 wait complete. 2024-03-24T00:20:21.305 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:20:21.304+0000 7f9248c10740 20 client.26656 trim_cache size 0 max 16384 2024-03-24T00:20:21.305 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stdout:{ 2024-03-24T00:20:21.306 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stdout: "message": "", 2024-03-24T00:20:21.306 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stdout: "return_code": 0 2024-03-24T00:20:21.306 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stdout:} 2024-03-24T00:20:21.306 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:20:21.305+0000 7f9245e00640 1 -- 172.21.15.118:0/2209975981 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7f92240a28a0 msgr2=0x7f92240a4d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:20:21.306 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:20:21.305+0000 7f9245e00640 1 --2- 172.21.15.118:0/2209975981 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7f92240a28a0 0x7f92240a4d60 secure :-1 s=READY pgs=352 cs=0 l=1 rev1=1 crypto rx=0x7f9230098180 tx=0x7f92300d29e0 comp rx=0 tx=0).stop 2024-03-24T00:20:21.307 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:20:21.305+0000 7f9245e00640 1 -- 172.21.15.118:0/2209975981 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f92401038d0 msgr2=0x7f9240199b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:20:21.307 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:20:21.305+0000 7f9245e00640 1 --2- 172.21.15.118:0/2209975981 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f92401038d0 0x7f9240199b00 secure :-1 s=READY pgs=404 cs=0 l=1 rev1=1 crypto rx=0x7f923000e9c0 tx=0x7f923000ee80 comp rx=0 tx=0).stop 2024-03-24T00:20:21.307 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:20:21.305+0000 7f9245e00640 1 -- 172.21.15.118:0/2209975981 shutdown_connections 2024-03-24T00:20:21.307 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:20:21.305+0000 7f9245e00640 1 --2- 172.21.15.118:0/2209975981 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7f92240a28a0 0x7f92240a4d60 unknown :-1 s=CLOSED pgs=352 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:20:21.307 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:20:21.305+0000 7f9245e00640 1 --2- 172.21.15.118:0/2209975981 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f9240105480 0x7f92401a10b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:20:21.307 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:20:21.305+0000 7f9245e00640 1 --2- 172.21.15.118:0/2209975981 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f9240104ac0 0x7f924019a040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:20:21.307 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:20:21.305+0000 7f9245e00640 1 --2- 172.21.15.118:0/2209975981 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f92401038d0 0x7f9240199b00 unknown :-1 s=CLOSED pgs=404 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:20:21.307 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:20:21.305+0000 7f9245e00640 1 -- 172.21.15.118:0/2209975981 >> 172.21.15.118:0/2209975981 conn(0x7f92400fee60 msgr2=0x7f9240107cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:20:21.307 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:20:21.306+0000 7f9245e00640 1 -- 172.21.15.118:0/2209975981 shutdown_connections 2024-03-24T00:20:21.307 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:20:21.306+0000 7f9245e00640 1 -- 172.21.15.118:0/2209975981 wait complete. 2024-03-24T00:20:21.531 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:20:21 smithi193 ceph-mon[36878]: mds.? [v2:172.21.15.193:6832/3097400619,v1:172.21.15.193:6833/3097400619] up:reconnect 2024-03-24T00:20:21.532 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:20:21 smithi193 ceph-mon[36878]: fsmap cephfs:5/5 4 up:standby-replay 3 up:standby 1 up:reconnect 4 up:active 2024-03-24T00:20:21.532 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:20:21 smithi193 ceph-mon[36878]: reconnect by client.15159 192.168.0.1:0/3463739313 after 0.0149998 2024-03-24T00:20:21.588 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:20:21 smithi118 ceph-mon[31535]: mds.? [v2:172.21.15.193:6832/3097400619,v1:172.21.15.193:6833/3097400619] up:reconnect 2024-03-24T00:20:21.588 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:20:21 smithi118 ceph-mon[31535]: fsmap cephfs:5/5 4 up:standby-replay 3 up:standby 1 up:reconnect 4 up:active 2024-03-24T00:20:21.588 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:20:21 smithi118 ceph-mon[31535]: reconnect by client.15159 192.168.0.1:0/3463739313 after 0.0149998 2024-03-24T00:20:21.704 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:20:21 smithi189 ceph-mon[36187]: mds.? [v2:172.21.15.193:6832/3097400619,v1:172.21.15.193:6833/3097400619] up:reconnect 2024-03-24T00:20:21.704 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:20:21 smithi189 ceph-mon[36187]: fsmap cephfs:5/5 4 up:standby-replay 3 up:standby 1 up:reconnect 4 up:active 2024-03-24T00:20:21.704 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:20:21 smithi189 ceph-mon[36187]: reconnect by client.15159 192.168.0.1:0/3463739313 after 0.0149998 2024-03-24T00:20:21.868 INFO:journalctl@ceph.mds.l.smithi193.stdout:Mar 24 00:20:21 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-l[70947]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: In function 'void Locker::set_xlocks_done(MutationImpl*, bool)' thread 7f8e3811b700 time 2024-03-24T00:20:21.532675+0000 2024-03-24T00:20:21.868 INFO:journalctl@ceph.mds.l.smithi193.stdout:Mar 24 00:20:21 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-l[70947]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: 626: FAILED ceph_assert(obj->is_auth()) 2024-03-24T00:20:21.868 INFO:journalctl@ceph.mds.l.smithi193.stdout:Mar 24 00:20:21 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-l[70947]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-24T00:20:21.868 INFO:journalctl@ceph.mds.l.smithi193.stdout:Mar 24 00:20:21 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-l[70947]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x135) [0x7f8e427454a1] 2024-03-24T00:20:21.869 INFO:journalctl@ceph.mds.l.smithi193.stdout:Mar 24 00:20:21 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-l[70947]: 2: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7f8e42745667] 2024-03-24T00:20:21.869 INFO:journalctl@ceph.mds.l.smithi193.stdout:Mar 24 00:20:21 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-l[70947]: 3: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x55ecb88bd223] 2024-03-24T00:20:21.869 INFO:journalctl@ceph.mds.l.smithi193.stdout:Mar 24 00:20:21 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-l[70947]: 4: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x55ecb87375d6] 2024-03-24T00:20:21.869 INFO:journalctl@ceph.mds.l.smithi193.stdout:Mar 24 00:20:21 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-l[70947]: 5: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x55ecb8737da1] 2024-03-24T00:20:21.869 INFO:journalctl@ceph.mds.l.smithi193.stdout:Mar 24 00:20:21 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-l[70947]: 6: (Server::handle_client_rename(boost::intrusive_ptr const&)+0x259d) [0x55ecb877497d] 2024-03-24T00:20:21.869 INFO:journalctl@ceph.mds.l.smithi193.stdout:Mar 24 00:20:21 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-l[70947]: 7: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x55ecb878a233] 2024-03-24T00:20:21.869 INFO:journalctl@ceph.mds.l.smithi193.stdout:Mar 24 00:20:21 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-l[70947]: 8: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x55ecb878c197] 2024-03-24T00:20:21.869 INFO:journalctl@ceph.mds.l.smithi193.stdout:Mar 24 00:20:21 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-l[70947]: 9: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x55ecb868b11b] 2024-03-24T00:20:21.869 INFO:journalctl@ceph.mds.l.smithi193.stdout:Mar 24 00:20:21 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-l[70947]: 10: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x55ecb868b6bc] 2024-03-24T00:20:21.869 INFO:journalctl@ceph.mds.l.smithi193.stdout:Mar 24 00:20:21 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-l[70947]: 11: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x55ecb86751ff] 2024-03-24T00:20:21.869 INFO:journalctl@ceph.mds.l.smithi193.stdout:Mar 24 00:20:21 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-l[70947]: 12: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7f8e429d7488] 2024-03-24T00:20:21.869 INFO:journalctl@ceph.mds.l.smithi193.stdout:Mar 24 00:20:21 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-l[70947]: 13: (DispatchQueue::entry()+0x51f) [0x7f8e429d460f] 2024-03-24T00:20:21.869 INFO:journalctl@ceph.mds.l.smithi193.stdout:Mar 24 00:20:21 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-l[70947]: 14: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f8e42aa7e81] 2024-03-24T00:20:21.870 INFO:journalctl@ceph.mds.l.smithi193.stdout:Mar 24 00:20:21 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-l[70947]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7f8e414c21ca] 2024-03-24T00:20:21.870 INFO:journalctl@ceph.mds.l.smithi193.stdout:Mar 24 00:20:21 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-l[70947]: 16: clone() 2024-03-24T00:20:21.870 INFO:journalctl@ceph.mds.l.smithi193.stdout:Mar 24 00:20:21 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-l[70947]: 2024-03-24T00:20:21.533+0000 7f8e3811b700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: In function 'void Locker::set_xlocks_done(MutationImpl*, bool)' thread 7f8e3811b700 time 2024-03-24T00:20:21.532675+0000 2024-03-24T00:20:21.870 INFO:journalctl@ceph.mds.l.smithi193.stdout:Mar 24 00:20:21 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-l[70947]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: 626: FAILED ceph_assert(obj->is_auth()) 2024-03-24T00:20:21.870 INFO:journalctl@ceph.mds.l.smithi193.stdout:Mar 24 00:20:21 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-l[70947]: 2024-03-24T00:20:21.870 INFO:journalctl@ceph.mds.l.smithi193.stdout:Mar 24 00:20:21 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-l[70947]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-24T00:20:21.870 INFO:journalctl@ceph.mds.l.smithi193.stdout:Mar 24 00:20:21 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-l[70947]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x135) [0x7f8e427454a1] 2024-03-24T00:20:21.870 INFO:journalctl@ceph.mds.l.smithi193.stdout:Mar 24 00:20:21 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-l[70947]: 2: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7f8e42745667] 2024-03-24T00:20:21.870 INFO:journalctl@ceph.mds.l.smithi193.stdout:Mar 24 00:20:21 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-l[70947]: 3: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x55ecb88bd223] 2024-03-24T00:20:21.870 INFO:journalctl@ceph.mds.l.smithi193.stdout:Mar 24 00:20:21 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-l[70947]: 4: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x55ecb87375d6] 2024-03-24T00:20:21.870 INFO:journalctl@ceph.mds.l.smithi193.stdout:Mar 24 00:20:21 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-l[70947]: 5: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x55ecb8737da1] 2024-03-24T00:20:21.870 INFO:journalctl@ceph.mds.l.smithi193.stdout:Mar 24 00:20:21 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-l[70947]: 6: (Server::handle_client_rename(boost::intrusive_ptr const&)+0x259d) [0x55ecb877497d] 2024-03-24T00:20:21.871 INFO:journalctl@ceph.mds.l.smithi193.stdout:Mar 24 00:20:21 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-l[70947]: 7: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x55ecb878a233] 2024-03-24T00:20:21.871 INFO:journalctl@ceph.mds.l.smithi193.stdout:Mar 24 00:20:21 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-l[70947]: 8: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x55ecb878c197] 2024-03-24T00:20:21.871 INFO:journalctl@ceph.mds.l.smithi193.stdout:Mar 24 00:20:21 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-l[70947]: 9: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x55ecb868b11b] 2024-03-24T00:20:21.871 INFO:journalctl@ceph.mds.l.smithi193.stdout:Mar 24 00:20:21 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-l[70947]: 10: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x55ecb868b6bc] 2024-03-24T00:20:21.871 INFO:journalctl@ceph.mds.l.smithi193.stdout:Mar 24 00:20:21 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-l[70947]: 11: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x55ecb86751ff] 2024-03-24T00:20:21.871 INFO:journalctl@ceph.mds.l.smithi193.stdout:Mar 24 00:20:21 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-l[70947]: 12: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7f8e429d7488] 2024-03-24T00:20:21.871 INFO:journalctl@ceph.mds.l.smithi193.stdout:Mar 24 00:20:21 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-l[70947]: 13: (DispatchQueue::entry()+0x51f) [0x7f8e429d460f] 2024-03-24T00:20:21.871 INFO:journalctl@ceph.mds.l.smithi193.stdout:Mar 24 00:20:21 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-l[70947]: 14: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f8e42aa7e81] 2024-03-24T00:20:21.871 INFO:journalctl@ceph.mds.l.smithi193.stdout:Mar 24 00:20:21 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-l[70947]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7f8e414c21ca] 2024-03-24T00:20:21.871 INFO:journalctl@ceph.mds.l.smithi193.stdout:Mar 24 00:20:21 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-l[70947]: 16: clone() 2024-03-24T00:20:21.871 INFO:journalctl@ceph.mds.l.smithi193.stdout:Mar 24 00:20:21 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-l[70947]: 2024-03-24T00:20:21.871 INFO:journalctl@ceph.mds.l.smithi193.stdout:Mar 24 00:20:21 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-l[70947]: *** Caught signal (Aborted) ** 2024-03-24T00:20:21.871 INFO:journalctl@ceph.mds.l.smithi193.stdout:Mar 24 00:20:21 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-l[70947]: in thread 7f8e3811b700 thread_name:ms_dispatch 2024-03-24T00:20:21.871 INFO:journalctl@ceph.mds.l.smithi193.stdout:Mar 24 00:20:21 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-l[70947]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-24T00:20:21.872 INFO:journalctl@ceph.mds.l.smithi193.stdout:Mar 24 00:20:21 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-l[70947]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f8e414ccd20] 2024-03-24T00:20:21.873 INFO:journalctl@ceph.mds.l.smithi193.stdout:Mar 24 00:20:21 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-l[70947]: 2: gsignal() 2024-03-24T00:20:21.873 INFO:journalctl@ceph.mds.l.smithi193.stdout:Mar 24 00:20:21 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-l[70947]: 3: abort() 2024-03-24T00:20:21.873 INFO:journalctl@ceph.mds.l.smithi193.stdout:Mar 24 00:20:21 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-l[70947]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x18f) [0x7f8e427454fb] 2024-03-24T00:20:21.873 INFO:journalctl@ceph.mds.l.smithi193.stdout:Mar 24 00:20:21 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-l[70947]: 5: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7f8e42745667] 2024-03-24T00:20:21.873 INFO:journalctl@ceph.mds.l.smithi193.stdout:Mar 24 00:20:21 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-l[70947]: 6: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x55ecb88bd223] 2024-03-24T00:20:21.873 INFO:journalctl@ceph.mds.l.smithi193.stdout:Mar 24 00:20:21 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-l[70947]: 7: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x55ecb87375d6] 2024-03-24T00:20:21.873 INFO:journalctl@ceph.mds.l.smithi193.stdout:Mar 24 00:20:21 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-l[70947]: 8: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x55ecb8737da1] 2024-03-24T00:20:21.873 INFO:journalctl@ceph.mds.l.smithi193.stdout:Mar 24 00:20:21 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-l[70947]: 9: (Server::handle_client_rename(boost::intrusive_ptr const&)+0x259d) [0x55ecb877497d] 2024-03-24T00:20:21.873 INFO:journalctl@ceph.mds.l.smithi193.stdout:Mar 24 00:20:21 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-l[70947]: 10: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x55ecb878a233] 2024-03-24T00:20:21.873 INFO:journalctl@ceph.mds.l.smithi193.stdout:Mar 24 00:20:21 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-l[70947]: 11: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x55ecb878c197] 2024-03-24T00:20:21.873 INFO:journalctl@ceph.mds.l.smithi193.stdout:Mar 24 00:20:21 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-l[70947]: 12: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x55ecb868b11b] 2024-03-24T00:20:21.874 INFO:journalctl@ceph.mds.l.smithi193.stdout:Mar 24 00:20:21 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-l[70947]: 13: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x55ecb868b6bc] 2024-03-24T00:20:21.874 INFO:journalctl@ceph.mds.l.smithi193.stdout:Mar 24 00:20:21 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-l[70947]: 14: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x55ecb86751ff] 2024-03-24T00:20:21.874 INFO:journalctl@ceph.mds.l.smithi193.stdout:Mar 24 00:20:21 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-l[70947]: 15: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7f8e429d7488] 2024-03-24T00:20:21.874 INFO:journalctl@ceph.mds.l.smithi193.stdout:Mar 24 00:20:21 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-l[70947]: 16: (DispatchQueue::entry()+0x51f) [0x7f8e429d460f] 2024-03-24T00:20:21.874 INFO:journalctl@ceph.mds.l.smithi193.stdout:Mar 24 00:20:21 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-l[70947]: 17: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f8e42aa7e81] 2024-03-24T00:20:21.874 INFO:journalctl@ceph.mds.l.smithi193.stdout:Mar 24 00:20:21 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-l[70947]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7f8e414c21ca] 2024-03-24T00:20:21.874 INFO:journalctl@ceph.mds.l.smithi193.stdout:Mar 24 00:20:21 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-l[70947]: 19: clone() 2024-03-24T00:20:21.874 INFO:journalctl@ceph.mds.l.smithi193.stdout:Mar 24 00:20:21 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-l[70947]: 2024-03-24T00:20:21.535+0000 7f8e3811b700 -1 *** Caught signal (Aborted) ** 2024-03-24T00:20:21.874 INFO:journalctl@ceph.mds.l.smithi193.stdout:Mar 24 00:20:21 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-l[70947]: in thread 7f8e3811b700 thread_name:ms_dispatch 2024-03-24T00:20:21.874 INFO:journalctl@ceph.mds.l.smithi193.stdout:Mar 24 00:20:21 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-l[70947]: 2024-03-24T00:20:21.874 INFO:journalctl@ceph.mds.l.smithi193.stdout:Mar 24 00:20:21 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-l[70947]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-24T00:20:21.874 INFO:journalctl@ceph.mds.l.smithi193.stdout:Mar 24 00:20:21 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-l[70947]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f8e414ccd20] 2024-03-24T00:20:21.874 INFO:journalctl@ceph.mds.l.smithi193.stdout:Mar 24 00:20:21 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-l[70947]: 2: gsignal() 2024-03-24T00:20:21.875 INFO:journalctl@ceph.mds.l.smithi193.stdout:Mar 24 00:20:21 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-l[70947]: 3: abort() 2024-03-24T00:20:21.875 INFO:journalctl@ceph.mds.l.smithi193.stdout:Mar 24 00:20:21 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-l[70947]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x18f) [0x7f8e427454fb] 2024-03-24T00:20:21.875 INFO:journalctl@ceph.mds.l.smithi193.stdout:Mar 24 00:20:21 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-l[70947]: 5: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7f8e42745667] 2024-03-24T00:20:21.875 INFO:journalctl@ceph.mds.l.smithi193.stdout:Mar 24 00:20:21 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-l[70947]: 6: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x55ecb88bd223] 2024-03-24T00:20:21.875 INFO:journalctl@ceph.mds.l.smithi193.stdout:Mar 24 00:20:21 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-l[70947]: 7: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x55ecb87375d6] 2024-03-24T00:20:21.875 INFO:journalctl@ceph.mds.l.smithi193.stdout:Mar 24 00:20:21 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-l[70947]: 8: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x55ecb8737da1] 2024-03-24T00:20:21.875 INFO:journalctl@ceph.mds.l.smithi193.stdout:Mar 24 00:20:21 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-l[70947]: 9: (Server::handle_client_rename(boost::intrusive_ptr const&)+0x259d) [0x55ecb877497d] 2024-03-24T00:20:21.875 INFO:journalctl@ceph.mds.l.smithi193.stdout:Mar 24 00:20:21 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-l[70947]: 10: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x55ecb878a233] 2024-03-24T00:20:21.875 INFO:journalctl@ceph.mds.l.smithi193.stdout:Mar 24 00:20:21 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-l[70947]: 11: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x55ecb878c197] 2024-03-24T00:20:21.875 INFO:journalctl@ceph.mds.l.smithi193.stdout:Mar 24 00:20:21 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-l[70947]: 12: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x55ecb868b11b] 2024-03-24T00:20:21.875 INFO:journalctl@ceph.mds.l.smithi193.stdout:Mar 24 00:20:21 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-l[70947]: 13: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x55ecb868b6bc] 2024-03-24T00:20:21.875 INFO:journalctl@ceph.mds.l.smithi193.stdout:Mar 24 00:20:21 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-l[70947]: 14: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x55ecb86751ff] 2024-03-24T00:20:21.875 INFO:journalctl@ceph.mds.l.smithi193.stdout:Mar 24 00:20:21 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-l[70947]: 15: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7f8e429d7488] 2024-03-24T00:20:21.875 INFO:journalctl@ceph.mds.l.smithi193.stdout:Mar 24 00:20:21 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-l[70947]: 16: (DispatchQueue::entry()+0x51f) [0x7f8e429d460f] 2024-03-24T00:20:21.875 INFO:journalctl@ceph.mds.l.smithi193.stdout:Mar 24 00:20:21 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-l[70947]: 17: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f8e42aa7e81] 2024-03-24T00:20:21.875 INFO:journalctl@ceph.mds.l.smithi193.stdout:Mar 24 00:20:21 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-l[70947]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7f8e414c21ca] 2024-03-24T00:20:21.876 INFO:journalctl@ceph.mds.l.smithi193.stdout:Mar 24 00:20:21 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-l[70947]: 19: clone() 2024-03-24T00:20:21.876 INFO:journalctl@ceph.mds.l.smithi193.stdout:Mar 24 00:20:21 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-l[70947]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-03-24T00:20:21.876 INFO:journalctl@ceph.mds.l.smithi193.stdout:Mar 24 00:20:21 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-l[70947]: 2024-03-24T00:20:21.876 INFO:journalctl@ceph.mds.l.smithi193.stdout:Mar 24 00:20:21 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-l[70947]: -1> 2024-03-24T00:20:21.533+0000 7f8e3811b700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: In function 'void Locker::set_xlocks_done(MutationImpl*, bool)' thread 7f8e3811b700 time 2024-03-24T00:20:21.532675+0000 2024-03-24T00:20:21.876 INFO:journalctl@ceph.mds.l.smithi193.stdout:Mar 24 00:20:21 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-l[70947]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: 626: FAILED ceph_assert(obj->is_auth()) 2024-03-24T00:20:21.876 INFO:journalctl@ceph.mds.l.smithi193.stdout:Mar 24 00:20:21 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-l[70947]: 2024-03-24T00:20:21.876 INFO:journalctl@ceph.mds.l.smithi193.stdout:Mar 24 00:20:21 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-l[70947]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-24T00:20:21.876 INFO:journalctl@ceph.mds.l.smithi193.stdout:Mar 24 00:20:21 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-l[70947]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x135) [0x7f8e427454a1] 2024-03-24T00:20:21.876 INFO:journalctl@ceph.mds.l.smithi193.stdout:Mar 24 00:20:21 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-l[70947]: 2: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7f8e42745667] 2024-03-24T00:20:21.876 INFO:journalctl@ceph.mds.l.smithi193.stdout:Mar 24 00:20:21 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-l[70947]: 3: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x55ecb88bd223] 2024-03-24T00:20:21.876 INFO:journalctl@ceph.mds.l.smithi193.stdout:Mar 24 00:20:21 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-l[70947]: 4: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x55ecb87375d6] 2024-03-24T00:20:21.876 INFO:journalctl@ceph.mds.l.smithi193.stdout:Mar 24 00:20:21 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-l[70947]: 5: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x55ecb8737da1] 2024-03-24T00:20:21.876 INFO:journalctl@ceph.mds.l.smithi193.stdout:Mar 24 00:20:21 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-l[70947]: 6: (Server::handle_client_rename(boost::intrusive_ptr const&)+0x259d) [0x55ecb877497d] 2024-03-24T00:20:21.876 INFO:journalctl@ceph.mds.l.smithi193.stdout:Mar 24 00:20:21 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-l[70947]: 7: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x55ecb878a233] 2024-03-24T00:20:21.876 INFO:journalctl@ceph.mds.l.smithi193.stdout:Mar 24 00:20:21 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-l[70947]: 8: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x55ecb878c197] 2024-03-24T00:20:21.877 INFO:journalctl@ceph.mds.l.smithi193.stdout:Mar 24 00:20:21 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-l[70947]: 9: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x55ecb868b11b] 2024-03-24T00:20:21.877 INFO:journalctl@ceph.mds.l.smithi193.stdout:Mar 24 00:20:21 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-l[70947]: 10: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x55ecb868b6bc] 2024-03-24T00:20:21.877 INFO:journalctl@ceph.mds.l.smithi193.stdout:Mar 24 00:20:21 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-l[70947]: 11: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x55ecb86751ff] 2024-03-24T00:20:21.877 INFO:journalctl@ceph.mds.l.smithi193.stdout:Mar 24 00:20:21 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-l[70947]: 12: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7f8e429d7488] 2024-03-24T00:20:21.877 INFO:journalctl@ceph.mds.l.smithi193.stdout:Mar 24 00:20:21 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-l[70947]: 13: (DispatchQueue::entry()+0x51f) [0x7f8e429d460f] 2024-03-24T00:20:21.877 INFO:journalctl@ceph.mds.l.smithi193.stdout:Mar 24 00:20:21 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-l[70947]: 14: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f8e42aa7e81] 2024-03-24T00:20:21.877 INFO:journalctl@ceph.mds.l.smithi193.stdout:Mar 24 00:20:21 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-l[70947]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7f8e414c21ca] 2024-03-24T00:20:21.877 INFO:journalctl@ceph.mds.l.smithi193.stdout:Mar 24 00:20:21 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-l[70947]: 16: clone() 2024-03-24T00:20:21.877 INFO:journalctl@ceph.mds.l.smithi193.stdout:Mar 24 00:20:21 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-l[70947]: 2024-03-24T00:20:21.877 INFO:journalctl@ceph.mds.l.smithi193.stdout:Mar 24 00:20:21 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-l[70947]: 0> 2024-03-24T00:20:21.535+0000 7f8e3811b700 -1 *** Caught signal (Aborted) ** 2024-03-24T00:20:21.877 INFO:journalctl@ceph.mds.l.smithi193.stdout:Mar 24 00:20:21 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-l[70947]: in thread 7f8e3811b700 thread_name:ms_dispatch 2024-03-24T00:20:21.877 INFO:journalctl@ceph.mds.l.smithi193.stdout:Mar 24 00:20:21 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-l[70947]: 2024-03-24T00:20:21.877 INFO:journalctl@ceph.mds.l.smithi193.stdout:Mar 24 00:20:21 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-l[70947]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-24T00:20:21.877 INFO:journalctl@ceph.mds.l.smithi193.stdout:Mar 24 00:20:21 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-l[70947]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f8e414ccd20] 2024-03-24T00:20:21.877 INFO:journalctl@ceph.mds.l.smithi193.stdout:Mar 24 00:20:21 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-l[70947]: 2: gsignal() 2024-03-24T00:20:21.878 INFO:journalctl@ceph.mds.l.smithi193.stdout:Mar 24 00:20:21 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-l[70947]: 3: abort() 2024-03-24T00:20:21.878 INFO:journalctl@ceph.mds.l.smithi193.stdout:Mar 24 00:20:21 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-l[70947]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x18f) [0x7f8e427454fb] 2024-03-24T00:20:21.878 INFO:journalctl@ceph.mds.l.smithi193.stdout:Mar 24 00:20:21 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-l[70947]: 5: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7f8e42745667] 2024-03-24T00:20:21.878 INFO:journalctl@ceph.mds.l.smithi193.stdout:Mar 24 00:20:21 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-l[70947]: 6: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x55ecb88bd223] 2024-03-24T00:20:21.878 INFO:journalctl@ceph.mds.l.smithi193.stdout:Mar 24 00:20:21 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-l[70947]: 7: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x55ecb87375d6] 2024-03-24T00:20:21.878 INFO:journalctl@ceph.mds.l.smithi193.stdout:Mar 24 00:20:21 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-l[70947]: 8: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x55ecb8737da1] 2024-03-24T00:20:21.878 INFO:journalctl@ceph.mds.l.smithi193.stdout:Mar 24 00:20:21 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-l[70947]: 9: (Server::handle_client_rename(boost::intrusive_ptr const&)+0x259d) [0x55ecb877497d] 2024-03-24T00:20:21.878 INFO:journalctl@ceph.mds.l.smithi193.stdout:Mar 24 00:20:21 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-l[70947]: 10: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x55ecb878a233] 2024-03-24T00:20:21.878 INFO:journalctl@ceph.mds.l.smithi193.stdout:Mar 24 00:20:21 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-l[70947]: 11: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x55ecb878c197] 2024-03-24T00:20:21.878 INFO:journalctl@ceph.mds.l.smithi193.stdout:Mar 24 00:20:21 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-l[70947]: 12: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x55ecb868b11b] 2024-03-24T00:20:21.878 INFO:journalctl@ceph.mds.l.smithi193.stdout:Mar 24 00:20:21 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-l[70947]: 13: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x55ecb868b6bc] 2024-03-24T00:20:21.878 INFO:journalctl@ceph.mds.l.smithi193.stdout:Mar 24 00:20:21 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-l[70947]: 14: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x55ecb86751ff] 2024-03-24T00:20:21.878 INFO:journalctl@ceph.mds.l.smithi193.stdout:Mar 24 00:20:21 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-l[70947]: 15: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7f8e429d7488] 2024-03-24T00:20:21.879 INFO:journalctl@ceph.mds.l.smithi193.stdout:Mar 24 00:20:21 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-l[70947]: 16: (DispatchQueue::entry()+0x51f) [0x7f8e429d460f] 2024-03-24T00:20:21.879 INFO:journalctl@ceph.mds.l.smithi193.stdout:Mar 24 00:20:21 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-l[70947]: 17: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f8e42aa7e81] 2024-03-24T00:20:21.879 INFO:journalctl@ceph.mds.l.smithi193.stdout:Mar 24 00:20:21 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-l[70947]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7f8e414c21ca] 2024-03-24T00:20:21.879 INFO:journalctl@ceph.mds.l.smithi193.stdout:Mar 24 00:20:21 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-l[70947]: 19: clone() 2024-03-24T00:20:21.879 INFO:journalctl@ceph.mds.l.smithi193.stdout:Mar 24 00:20:21 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-l[70947]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-03-24T00:20:21.879 INFO:journalctl@ceph.mds.l.smithi193.stdout:Mar 24 00:20:21 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-l[70947]: 2024-03-24T00:20:21.879 INFO:journalctl@ceph.mds.l.smithi193.stdout:Mar 24 00:20:21 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-l[70947]: -1> 2024-03-24T00:20:21.533+0000 7f8e3811b700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: In function 'void Locker::set_xlocks_done(MutationImpl*, bool)' thread 7f8e3811b700 time 2024-03-24T00:20:21.532675+0000 2024-03-24T00:20:21.879 INFO:journalctl@ceph.mds.l.smithi193.stdout:Mar 24 00:20:21 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-l[70947]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: 626: FAILED ceph_assert(obj->is_auth()) 2024-03-24T00:20:21.879 INFO:journalctl@ceph.mds.l.smithi193.stdout:Mar 24 00:20:21 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-l[70947]: 2024-03-24T00:20:21.879 INFO:journalctl@ceph.mds.l.smithi193.stdout:Mar 24 00:20:21 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-l[70947]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-24T00:20:21.879 INFO:journalctl@ceph.mds.l.smithi193.stdout:Mar 24 00:20:21 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-l[70947]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x135) [0x7f8e427454a1] 2024-03-24T00:20:21.879 INFO:journalctl@ceph.mds.l.smithi193.stdout:Mar 24 00:20:21 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-l[70947]: 2: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7f8e42745667] 2024-03-24T00:20:21.879 INFO:journalctl@ceph.mds.l.smithi193.stdout:Mar 24 00:20:21 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-l[70947]: 3: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x55ecb88bd223] 2024-03-24T00:20:21.879 INFO:journalctl@ceph.mds.l.smithi193.stdout:Mar 24 00:20:21 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-l[70947]: 4: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x55ecb87375d6] 2024-03-24T00:20:21.880 INFO:journalctl@ceph.mds.l.smithi193.stdout:Mar 24 00:20:21 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-l[70947]: 5: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x55ecb8737da1] 2024-03-24T00:20:21.880 INFO:journalctl@ceph.mds.l.smithi193.stdout:Mar 24 00:20:21 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-l[70947]: 6: (Server::handle_client_rename(boost::intrusive_ptr const&)+0x259d) [0x55ecb877497d] 2024-03-24T00:20:21.880 INFO:journalctl@ceph.mds.l.smithi193.stdout:Mar 24 00:20:21 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-l[70947]: 7: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x55ecb878a233] 2024-03-24T00:20:21.880 INFO:journalctl@ceph.mds.l.smithi193.stdout:Mar 24 00:20:21 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-l[70947]: 8: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x55ecb878c197] 2024-03-24T00:20:21.880 INFO:journalctl@ceph.mds.l.smithi193.stdout:Mar 24 00:20:21 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-l[70947]: 9: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x55ecb868b11b] 2024-03-24T00:20:21.880 INFO:journalctl@ceph.mds.l.smithi193.stdout:Mar 24 00:20:21 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-l[70947]: 10: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x55ecb868b6bc] 2024-03-24T00:20:21.880 INFO:journalctl@ceph.mds.l.smithi193.stdout:Mar 24 00:20:21 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-l[70947]: 11: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x55ecb86751ff] 2024-03-24T00:20:21.880 INFO:journalctl@ceph.mds.l.smithi193.stdout:Mar 24 00:20:21 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-l[70947]: 12: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7f8e429d7488] 2024-03-24T00:20:21.880 INFO:journalctl@ceph.mds.l.smithi193.stdout:Mar 24 00:20:21 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-l[70947]: 13: (DispatchQueue::entry()+0x51f) [0x7f8e429d460f] 2024-03-24T00:20:21.880 INFO:journalctl@ceph.mds.l.smithi193.stdout:Mar 24 00:20:21 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-l[70947]: 14: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f8e42aa7e81] 2024-03-24T00:20:21.880 INFO:journalctl@ceph.mds.l.smithi193.stdout:Mar 24 00:20:21 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-l[70947]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7f8e414c21ca] 2024-03-24T00:20:21.880 INFO:journalctl@ceph.mds.l.smithi193.stdout:Mar 24 00:20:21 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-l[70947]: 16: clone() 2024-03-24T00:20:21.880 INFO:journalctl@ceph.mds.l.smithi193.stdout:Mar 24 00:20:21 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-l[70947]: 2024-03-24T00:20:21.880 INFO:journalctl@ceph.mds.l.smithi193.stdout:Mar 24 00:20:21 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-l[70947]: 0> 2024-03-24T00:20:21.535+0000 7f8e3811b700 -1 *** Caught signal (Aborted) ** 2024-03-24T00:20:21.880 INFO:journalctl@ceph.mds.l.smithi193.stdout:Mar 24 00:20:21 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-l[70947]: in thread 7f8e3811b700 thread_name:ms_dispatch 2024-03-24T00:20:21.881 INFO:journalctl@ceph.mds.l.smithi193.stdout:Mar 24 00:20:21 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-l[70947]: 2024-03-24T00:20:21.881 INFO:journalctl@ceph.mds.l.smithi193.stdout:Mar 24 00:20:21 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-l[70947]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-24T00:20:21.881 INFO:journalctl@ceph.mds.l.smithi193.stdout:Mar 24 00:20:21 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-l[70947]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f8e414ccd20] 2024-03-24T00:20:21.881 INFO:journalctl@ceph.mds.l.smithi193.stdout:Mar 24 00:20:21 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-l[70947]: 2: gsignal() 2024-03-24T00:20:21.881 INFO:journalctl@ceph.mds.l.smithi193.stdout:Mar 24 00:20:21 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-l[70947]: 3: abort() 2024-03-24T00:20:21.881 INFO:journalctl@ceph.mds.l.smithi193.stdout:Mar 24 00:20:21 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-l[70947]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x18f) [0x7f8e427454fb] 2024-03-24T00:20:21.881 INFO:journalctl@ceph.mds.l.smithi193.stdout:Mar 24 00:20:21 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-l[70947]: 5: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7f8e42745667] 2024-03-24T00:20:21.881 INFO:journalctl@ceph.mds.l.smithi193.stdout:Mar 24 00:20:21 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-l[70947]: 6: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x55ecb88bd223] 2024-03-24T00:20:21.881 INFO:journalctl@ceph.mds.l.smithi193.stdout:Mar 24 00:20:21 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-l[70947]: 7: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x55ecb87375d6] 2024-03-24T00:20:21.881 INFO:journalctl@ceph.mds.l.smithi193.stdout:Mar 24 00:20:21 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-l[70947]: 8: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x55ecb8737da1] 2024-03-24T00:20:21.881 INFO:journalctl@ceph.mds.l.smithi193.stdout:Mar 24 00:20:21 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-l[70947]: 9: (Server::handle_client_rename(boost::intrusive_ptr const&)+0x259d) [0x55ecb877497d] 2024-03-24T00:20:21.881 INFO:journalctl@ceph.mds.l.smithi193.stdout:Mar 24 00:20:21 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-l[70947]: 10: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x55ecb878a233] 2024-03-24T00:20:21.881 INFO:journalctl@ceph.mds.l.smithi193.stdout:Mar 24 00:20:21 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-l[70947]: 11: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x55ecb878c197] 2024-03-24T00:20:21.882 INFO:journalctl@ceph.mds.l.smithi193.stdout:Mar 24 00:20:21 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-l[70947]: 12: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x55ecb868b11b] 2024-03-24T00:20:21.882 INFO:journalctl@ceph.mds.l.smithi193.stdout:Mar 24 00:20:21 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-l[70947]: 13: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x55ecb868b6bc] 2024-03-24T00:20:21.882 INFO:journalctl@ceph.mds.l.smithi193.stdout:Mar 24 00:20:21 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-l[70947]: 14: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x55ecb86751ff] 2024-03-24T00:20:21.882 INFO:journalctl@ceph.mds.l.smithi193.stdout:Mar 24 00:20:21 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-l[70947]: 15: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7f8e429d7488] 2024-03-24T00:20:21.882 INFO:journalctl@ceph.mds.l.smithi193.stdout:Mar 24 00:20:21 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-l[70947]: 16: (DispatchQueue::entry()+0x51f) [0x7f8e429d460f] 2024-03-24T00:20:21.882 INFO:journalctl@ceph.mds.l.smithi193.stdout:Mar 24 00:20:21 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-l[70947]: 17: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f8e42aa7e81] 2024-03-24T00:20:21.882 INFO:journalctl@ceph.mds.l.smithi193.stdout:Mar 24 00:20:21 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-l[70947]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7f8e414c21ca] 2024-03-24T00:20:21.882 INFO:journalctl@ceph.mds.l.smithi193.stdout:Mar 24 00:20:21 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-l[70947]: 19: clone() 2024-03-24T00:20:21.882 INFO:journalctl@ceph.mds.l.smithi193.stdout:Mar 24 00:20:21 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-l[70947]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-03-24T00:20:21.882 INFO:journalctl@ceph.mds.l.smithi193.stdout:Mar 24 00:20:21 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-l[70947]: 2024-03-24T00:20:21.882 INFO:journalctl@ceph.mds.l.smithi193.stdout:Mar 24 00:20:21 smithi193 podman[71141]: 2024-03-24 00:20:21.612389652 +0000 UTC m=+0.020392748 container died 85d07539cdbb4c91a033882ae0632ed4070ba7d9cd6e7785198c28d300fe8a17 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-l, org.label-schema.schema-version=1.0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, org.label-schema.build-date=20240319, GIT_CLEAN=True, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux ) 2024-03-24T00:20:22.497 INFO:journalctl@ceph.mds.l.smithi193.stdout:Mar 24 00:20:22 smithi193 podman[71141]: 2024-03-24 00:20:22.245469583 +0000 UTC m=+0.653472664 container remove 85d07539cdbb4c91a033882ae0632ed4070ba7d9cd6e7785198c28d300fe8a17 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-l, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, ceph=True, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240319, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, io.buildah.version=1.33.5) 2024-03-24T00:20:22.497 INFO:journalctl@ceph.mds.l.smithi193.stdout:Mar 24 00:20:22 smithi193 systemd[1]: ceph-d79a5420-e971-11ee-95cd-87774f69a715@mds.l.service: Main process exited, code=exited, status=134/n/a 2024-03-24T00:20:22.498 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:20:22 smithi193 ceph-mon[36878]: mds.? [v2:172.21.15.193:6832/3097400619,v1:172.21.15.193:6833/3097400619] up:rejoin 2024-03-24T00:20:22.498 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:20:22 smithi193 ceph-mon[36878]: fsmap cephfs:5/5 4 up:standby-replay 3 up:standby 1 up:rejoin 4 up:active 2024-03-24T00:20:22.498 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:20:22 smithi193 ceph-mon[36878]: daemon mds.l is now active in filesystem cephfs as rank 3 2024-03-24T00:20:22.498 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:20:22 smithi193 ceph-mon[36878]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-03-24T00:20:22.498 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:20:22 smithi193 ceph-mon[36878]: Cluster is now healthy 2024-03-24T00:20:22.498 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:20:22 smithi193 ceph-mon[36878]: mds.? [v2:172.21.15.193:6832/3097400619,v1:172.21.15.193:6833/3097400619] up:active 2024-03-24T00:20:22.498 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:20:22 smithi193 ceph-mon[36878]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-03-24T00:20:22.498 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:20:22 smithi193 ceph-mon[36878]: pgmap v541: 97 pgs: 97 active+clean; 99 MiB data, 643 MiB used, 1.0 TiB / 1.0 TiB avail; 93 KiB/s rd, 749 KiB/s wr, 55 op/s 2024-03-24T00:20:22.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:20:22 smithi118 ceph-mon[31535]: mds.? [v2:172.21.15.193:6832/3097400619,v1:172.21.15.193:6833/3097400619] up:rejoin 2024-03-24T00:20:22.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:20:22 smithi118 ceph-mon[31535]: fsmap cephfs:5/5 4 up:standby-replay 3 up:standby 1 up:rejoin 4 up:active 2024-03-24T00:20:22.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:20:22 smithi118 ceph-mon[31535]: daemon mds.l is now active in filesystem cephfs as rank 3 2024-03-24T00:20:22.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:20:22 smithi118 ceph-mon[31535]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-03-24T00:20:22.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:20:22 smithi118 ceph-mon[31535]: Cluster is now healthy 2024-03-24T00:20:22.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:20:22 smithi118 ceph-mon[31535]: mds.? [v2:172.21.15.193:6832/3097400619,v1:172.21.15.193:6833/3097400619] up:active 2024-03-24T00:20:22.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:20:22 smithi118 ceph-mon[31535]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-03-24T00:20:22.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:20:22 smithi118 ceph-mon[31535]: pgmap v541: 97 pgs: 97 active+clean; 99 MiB data, 643 MiB used, 1.0 TiB / 1.0 TiB avail; 93 KiB/s rd, 749 KiB/s wr, 55 op/s 2024-03-24T00:20:22.704 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:20:22 smithi189 ceph-mon[36187]: mds.? [v2:172.21.15.193:6832/3097400619,v1:172.21.15.193:6833/3097400619] up:rejoin 2024-03-24T00:20:22.704 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:20:22 smithi189 ceph-mon[36187]: fsmap cephfs:5/5 4 up:standby-replay 3 up:standby 1 up:rejoin 4 up:active 2024-03-24T00:20:22.704 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:20:22 smithi189 ceph-mon[36187]: daemon mds.l is now active in filesystem cephfs as rank 3 2024-03-24T00:20:22.704 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:20:22 smithi189 ceph-mon[36187]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-03-24T00:20:22.704 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:20:22 smithi189 ceph-mon[36187]: Cluster is now healthy 2024-03-24T00:20:22.704 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:20:22 smithi189 ceph-mon[36187]: mds.? [v2:172.21.15.193:6832/3097400619,v1:172.21.15.193:6833/3097400619] up:active 2024-03-24T00:20:22.704 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:20:22 smithi189 ceph-mon[36187]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-03-24T00:20:22.704 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:20:22 smithi189 ceph-mon[36187]: pgmap v541: 97 pgs: 97 active+clean; 99 MiB data, 643 MiB used, 1.0 TiB / 1.0 TiB avail; 93 KiB/s rd, 749 KiB/s wr, 55 op/s 2024-03-24T00:20:22.960 INFO:journalctl@ceph.mds.l.smithi193.stdout:Mar 24 00:20:22 smithi193 systemd[1]: ceph-d79a5420-e971-11ee-95cd-87774f69a715@mds.l.service: Failed with result 'exit-code'. 2024-03-24T00:20:25.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:20:24 smithi118 ceph-mon[31535]: pgmap v542: 97 pgs: 97 active+clean; 100 MiB data, 663 MiB used, 1.0 TiB / 1.0 TiB avail; 136 KiB/s rd, 833 KiB/s wr, 109 op/s 2024-03-24T00:20:25.203 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:20:24 smithi189 ceph-mon[36187]: pgmap v542: 97 pgs: 97 active+clean; 100 MiB data, 663 MiB used, 1.0 TiB / 1.0 TiB avail; 136 KiB/s rd, 833 KiB/s wr, 109 op/s 2024-03-24T00:20:25.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:20:24 smithi193 ceph-mon[36878]: pgmap v542: 97 pgs: 97 active+clean; 100 MiB data, 663 MiB used, 1.0 TiB / 1.0 TiB avail; 136 KiB/s rd, 833 KiB/s wr, 109 op/s 2024-03-24T00:20:26.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:20:25 smithi118 ceph-mon[31535]: pgmap v543: 97 pgs: 97 active+clean; 100 MiB data, 664 MiB used, 1.0 TiB / 1.0 TiB avail; 146 KiB/s rd, 832 KiB/s wr, 129 op/s 2024-03-24T00:20:26.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:20:25 smithi118 ceph-mon[31535]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-03-24T00:20:26.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:20:25 smithi189 ceph-mon[36187]: pgmap v543: 97 pgs: 97 active+clean; 100 MiB data, 664 MiB used, 1.0 TiB / 1.0 TiB avail; 146 KiB/s rd, 832 KiB/s wr, 129 op/s 2024-03-24T00:20:26.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:20:25 smithi189 ceph-mon[36187]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-03-24T00:20:26.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:20:25 smithi193 ceph-mon[36878]: pgmap v543: 97 pgs: 97 active+clean; 100 MiB data, 664 MiB used, 1.0 TiB / 1.0 TiB avail; 146 KiB/s rd, 832 KiB/s wr, 129 op/s 2024-03-24T00:20:26.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:20:25 smithi193 ceph-mon[36878]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-03-24T00:20:26.844 INFO:tasks.workunit.client.0.smithi118.stdout: 35 0 105 0 306 276 14 2024-03-24T00:20:29.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:20:28 smithi118 ceph-mon[31535]: pgmap v544: 97 pgs: 97 active+clean; 101 MiB data, 671 MiB used, 1.0 TiB / 1.0 TiB avail; 249 KiB/s rd, 551 KiB/s wr, 153 op/s 2024-03-24T00:20:29.203 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:20:28 smithi189 ceph-mon[36187]: pgmap v544: 97 pgs: 97 active+clean; 101 MiB data, 671 MiB used, 1.0 TiB / 1.0 TiB avail; 249 KiB/s rd, 551 KiB/s wr, 153 op/s 2024-03-24T00:20:29.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:20:28 smithi193 ceph-mon[36878]: pgmap v544: 97 pgs: 97 active+clean; 101 MiB data, 671 MiB used, 1.0 TiB / 1.0 TiB avail; 249 KiB/s rd, 551 KiB/s wr, 153 op/s 2024-03-24T00:20:30.098 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:20:29 smithi118 ceph-mon[31535]: pgmap v545: 97 pgs: 97 active+clean; 101 MiB data, 671 MiB used, 1.0 TiB / 1.0 TiB avail; 217 KiB/s rd, 423 KiB/s wr, 133 op/s 2024-03-24T00:20:30.203 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:20:29 smithi189 ceph-mon[36187]: pgmap v545: 97 pgs: 97 active+clean; 101 MiB data, 671 MiB used, 1.0 TiB / 1.0 TiB avail; 217 KiB/s rd, 423 KiB/s wr, 133 op/s 2024-03-24T00:20:30.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:20:29 smithi193 ceph-mon[36878]: pgmap v545: 97 pgs: 97 active+clean; 101 MiB data, 671 MiB used, 1.0 TiB / 1.0 TiB avail; 217 KiB/s rd, 423 KiB/s wr, 133 op/s 2024-03-24T00:20:32.035 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:20:32.034+0000 7fd627e00640 1 -- 172.21.15.118:0/736842230 >> [v2:172.21.15.193:6832/1023428412,v1:172.21.15.193:6833/1023428412] conn(0x55c55390a6a0 msgr2=0x55c55390cac0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.193:6832/1023428412 2024-03-24T00:20:32.035 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:20:32.034+0000 7fd613e00640 1 client.26594 ms_handle_refused on v2:172.21.15.193:6832/1023428412 2024-03-24T00:20:32.035 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:20:32.034+0000 7fd627e00640 1 --2- 172.21.15.118:0/736842230 >> [v2:172.21.15.193:6832/1023428412,v1:172.21.15.193:6833/1023428412] conn(0x55c55390a6a0 0x55c55390cac0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-03-24T00:20:32.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:20:31 smithi118 ceph-mon[31535]: pgmap v546: 97 pgs: 97 active+clean; 101 MiB data, 671 MiB used, 1.0 TiB / 1.0 TiB avail; 207 KiB/s rd, 295 KiB/s wr, 122 op/s 2024-03-24T00:20:32.203 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:20:31 smithi189 ceph-mon[36187]: pgmap v546: 97 pgs: 97 active+clean; 101 MiB data, 671 MiB used, 1.0 TiB / 1.0 TiB avail; 207 KiB/s rd, 295 KiB/s wr, 122 op/s 2024-03-24T00:20:32.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:20:31 smithi193 ceph-mon[36878]: pgmap v546: 97 pgs: 97 active+clean; 101 MiB data, 671 MiB used, 1.0 TiB / 1.0 TiB avail; 207 KiB/s rd, 295 KiB/s wr, 122 op/s 2024-03-24T00:20:33.074 INFO:journalctl@ceph.mds.l.smithi193.stdout:Mar 24 00:20:32 smithi193 systemd[1]: ceph-d79a5420-e971-11ee-95cd-87774f69a715@mds.l.service: Scheduled restart job, restart counter is at 2. 2024-03-24T00:20:33.074 INFO:journalctl@ceph.mds.l.smithi193.stdout:Mar 24 00:20:32 smithi193 systemd[1]: Stopped Ceph mds.l for d79a5420-e971-11ee-95cd-87774f69a715. 2024-03-24T00:20:33.074 INFO:journalctl@ceph.mds.l.smithi193.stdout:Mar 24 00:20:32 smithi193 systemd[1]: Starting Ceph mds.l for d79a5420-e971-11ee-95cd-87774f69a715... 2024-03-24T00:20:33.074 INFO:journalctl@ceph.mds.l.smithi193.stdout:Mar 24 00:20:32 smithi193 podman[71242]: 2024-03-24 00:20:32.839543029 +0000 UTC m=+0.020042785 image pull c664afe56bb01db4f9afa9532c20598e201e664f2e9ba2be50db0d52674a26d2 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc 2024-03-24T00:20:33.075 INFO:journalctl@ceph.mds.l.smithi193.stdout:Mar 24 00:20:32 smithi193 podman[71242]: 2024-03-24 00:20:32.954372289 +0000 UTC m=+0.134872055 container create 194e6e07f7e41b4829f11e21b1107d5d88b1246ccbfeeff413c34cb65d828d46 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-l, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240319, org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, GIT_BRANCH=HEAD, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.schema-version=1.0, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, org.label-schema.license=GPLv2, ceph=True) 2024-03-24T00:20:33.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:20:33 smithi193 ceph-mon[36878]: daemon mds.l restarted 2024-03-24T00:20:33.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:20:33 smithi193 ceph-mon[36878]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-03-24T00:20:33.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:20:33 smithi193 ceph-mon[36878]: osdmap e94: 12 total, 12 up, 12 in 2024-03-24T00:20:33.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:20:33 smithi193 ceph-mon[36878]: mds.? [v2:172.21.15.193:6832/1233062505,v1:172.21.15.193:6833/1233062505] up:boot 2024-03-24T00:20:33.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:20:33 smithi193 ceph-mon[36878]: Standby daemon mds.f assigned to filesystem cephfs as rank 3 2024-03-24T00:20:33.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:20:33 smithi193 ceph-mon[36878]: fsmap cephfs:4/5 {0=b=up:active,1=g=up:active,2=c=up:active,4=e=up:active} 5 up:standby-replay 3 up:standby, 1 failed 2024-03-24T00:20:33.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:20:33 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "l"}]: dispatch 2024-03-24T00:20:33.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:20:33 smithi193 ceph-mon[36878]: fsmap cephfs:5/5 4 up:standby-replay 3 up:standby 1 up:replay 4 up:active 2024-03-24T00:20:33.461 INFO:journalctl@ceph.mds.l.smithi193.stdout:Mar 24 00:20:33 smithi193 podman[71242]: 2024-03-24 00:20:33.19873727 +0000 UTC m=+0.379237044 container init 194e6e07f7e41b4829f11e21b1107d5d88b1246ccbfeeff413c34cb65d828d46 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-l, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , ceph=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, org.label-schema.build-date=20240319, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-24T00:20:33.462 INFO:journalctl@ceph.mds.l.smithi193.stdout:Mar 24 00:20:33 smithi193 podman[71242]: 2024-03-24 00:20:33.201530436 +0000 UTC m=+0.382030201 container start 194e6e07f7e41b4829f11e21b1107d5d88b1246ccbfeeff413c34cb65d828d46 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-l, org.label-schema.schema-version=1.0, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.build-date=20240319, org.label-schema.vendor=CentOS, ceph=True, CEPH_POINT_RELEASE=, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD) 2024-03-24T00:20:33.462 INFO:journalctl@ceph.mds.l.smithi193.stdout:Mar 24 00:20:33 smithi193 ceph-mds[71260]: -- 172.21.15.193:0/3311541828 <== mon.1 v2:172.21.15.193:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x55ce2fd69a40 con 0x55ce2fd3cc00 2024-03-24T00:20:33.462 INFO:journalctl@ceph.mds.l.smithi193.stdout:Mar 24 00:20:33 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-l[71256]: starting mds.l at 2024-03-24T00:20:33.462 INFO:journalctl@ceph.mds.l.smithi193.stdout:Mar 24 00:20:33 smithi193 bash[71242]: 194e6e07f7e41b4829f11e21b1107d5d88b1246ccbfeeff413c34cb65d828d46 2024-03-24T00:20:33.462 INFO:journalctl@ceph.mds.l.smithi193.stdout:Mar 24 00:20:33 smithi193 systemd[1]: Started Ceph mds.l for d79a5420-e971-11ee-95cd-87774f69a715. 2024-03-24T00:20:33.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:20:33 smithi118 ceph-mon[31535]: daemon mds.l restarted 2024-03-24T00:20:33.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:20:33 smithi118 ceph-mon[31535]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-03-24T00:20:33.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:20:33 smithi118 ceph-mon[31535]: osdmap e94: 12 total, 12 up, 12 in 2024-03-24T00:20:33.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:20:33 smithi118 ceph-mon[31535]: mds.? [v2:172.21.15.193:6832/1233062505,v1:172.21.15.193:6833/1233062505] up:boot 2024-03-24T00:20:33.697 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:20:33 smithi118 ceph-mon[31535]: Standby daemon mds.f assigned to filesystem cephfs as rank 3 2024-03-24T00:20:33.697 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:20:33 smithi118 ceph-mon[31535]: fsmap cephfs:4/5 {0=b=up:active,1=g=up:active,2=c=up:active,4=e=up:active} 5 up:standby-replay 3 up:standby, 1 failed 2024-03-24T00:20:33.697 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:20:33 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "l"}]: dispatch 2024-03-24T00:20:33.697 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:20:33 smithi118 ceph-mon[31535]: fsmap cephfs:5/5 4 up:standby-replay 3 up:standby 1 up:replay 4 up:active 2024-03-24T00:20:33.703 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:20:33 smithi189 ceph-mon[36187]: daemon mds.l restarted 2024-03-24T00:20:33.704 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:20:33 smithi189 ceph-mon[36187]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-03-24T00:20:33.704 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:20:33 smithi189 ceph-mon[36187]: osdmap e94: 12 total, 12 up, 12 in 2024-03-24T00:20:33.704 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:20:33 smithi189 ceph-mon[36187]: mds.? [v2:172.21.15.193:6832/1233062505,v1:172.21.15.193:6833/1233062505] up:boot 2024-03-24T00:20:33.704 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:20:33 smithi189 ceph-mon[36187]: Standby daemon mds.f assigned to filesystem cephfs as rank 3 2024-03-24T00:20:33.704 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:20:33 smithi189 ceph-mon[36187]: fsmap cephfs:4/5 {0=b=up:active,1=g=up:active,2=c=up:active,4=e=up:active} 5 up:standby-replay 3 up:standby, 1 failed 2024-03-24T00:20:33.704 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:20:33 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "l"}]: dispatch 2024-03-24T00:20:33.704 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:20:33 smithi189 ceph-mon[36187]: fsmap cephfs:5/5 4 up:standby-replay 3 up:standby 1 up:replay 4 up:active 2024-03-24T00:20:34.461 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:20:34.460+0000 7f089ea00640 1 -- 172.21.15.118:0/1509174086 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f08981073a0 msgr2=0x7f08981097f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:20:34.462 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:20:34.460+0000 7f089ea00640 1 --2- 172.21.15.118:0/1509174086 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f08981073a0 0x7f08981097f0 secure :-1 s=READY pgs=411 cs=0 l=1 rev1=1 crypto rx=0x7f0890009ca0 tx=0x7f089002c0f0 comp rx=0 tx=0).stop 2024-03-24T00:20:34.462 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:20:34.460+0000 7f089ea00640 1 -- 172.21.15.118:0/1509174086 shutdown_connections 2024-03-24T00:20:34.462 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:20:34.460+0000 7f089ea00640 1 --2- 172.21.15.118:0/1509174086 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f08981073a0 0x7f08981097f0 unknown :-1 s=CLOSED pgs=411 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:20:34.462 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:20:34.460+0000 7f089ea00640 1 --2- 172.21.15.118:0/1509174086 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f0898104a70 0x7f0898106e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:20:34.462 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:20:34.460+0000 7f089ea00640 1 --2- 172.21.15.118:0/1509174086 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f0898102140 0x7f0898104530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:20:34.462 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:20:34.460+0000 7f089ea00640 1 -- 172.21.15.118:0/1509174086 >> 172.21.15.118:0/1509174086 conn(0x7f08980fbb30 msgr2=0x7f08980fdf70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:20:34.462 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:20:34.460+0000 7f089ea00640 1 -- 172.21.15.118:0/1509174086 shutdown_connections 2024-03-24T00:20:34.462 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:20:34.460+0000 7f089ea00640 1 -- 172.21.15.118:0/1509174086 wait complete. 2024-03-24T00:20:34.464 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:20:34.461+0000 7f089ea00640 1 Processor -- start 2024-03-24T00:20:34.464 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:20:34.461+0000 7f089ea00640 1 -- start start 2024-03-24T00:20:34.464 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:20:34.461+0000 7f089ea00640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f0898102140 0x7f08981978a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:20:34.464 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:20:34.461+0000 7f089ea00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f0898104a70 0x7f0898197de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:20:34.465 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:20:34.461+0000 7f089ea00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f08981073a0 0x7f089819ee50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:20:34.465 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:20:34.461+0000 7f089ea00640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f089810d340 con 0x7f0898102140 2024-03-24T00:20:34.465 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:20:34.461+0000 7f089ea00640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f089810d1c0 con 0x7f08981073a0 2024-03-24T00:20:34.465 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:20:34.461+0000 7f089ea00640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f089810d4c0 con 0x7f0898104a70 2024-03-24T00:20:34.465 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:20:34.462+0000 7f0897e00640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f0898102140 0x7f08981978a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:20:34.465 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:20:34.462+0000 7f089ca00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f08981073a0 0x7f089819ee50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:20:34.465 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:20:34.462+0000 7f089ca00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f08981073a0 0x7f089819ee50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.118:44112/0 (socket says 172.21.15.118:44112) 2024-03-24T00:20:34.465 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:20:34.462+0000 7f089ca00640 1 -- 172.21.15.118:0/2028022490 learned_addr learned my addr 172.21.15.118:0/2028022490 (peer_addr_for_me v2:172.21.15.118:0/0) 2024-03-24T00:20:34.465 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:20:34.462+0000 7f088fe00640 1 --2- 172.21.15.118:0/2028022490 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f0898104a70 0x7f0898197de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:20:34.465 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:20:34.462+0000 7f089ca00640 1 -- 172.21.15.118:0/2028022490 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f0898104a70 msgr2=0x7f0898197de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:20:34.465 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:20:34.462+0000 7f089ca00640 1 --2- 172.21.15.118:0/2028022490 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f0898104a70 0x7f0898197de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:20:34.465 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:20:34.462+0000 7f089ca00640 1 -- 172.21.15.118:0/2028022490 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f0898102140 msgr2=0x7f08981978a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:20:34.465 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:20:34.462+0000 7f089ca00640 1 --2- 172.21.15.118:0/2028022490 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f0898102140 0x7f08981978a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:20:34.465 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:20:34.462+0000 7f089ca00640 1 -- 172.21.15.118:0/2028022490 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f089819f4c0 con 0x7f08981073a0 2024-03-24T00:20:34.466 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:20:34.462+0000 7f089ca00640 1 --2- 172.21.15.118:0/2028022490 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f08981073a0 0x7f089819ee50 secure :-1 s=READY pgs=406 cs=0 l=1 rev1=1 crypto rx=0x7f0890005e30 tx=0x7f0890032ec0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:20:34.466 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:20:34.463+0000 7f0895600640 1 -- 172.21.15.118:0/2028022490 <== mon.2 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0890041020 con 0x7f08981073a0 2024-03-24T00:20:34.466 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:20:34.463+0000 7f0895600640 1 -- 172.21.15.118:0/2028022490 <== mon.2 v2:172.21.15.189:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f0890034cd0 con 0x7f08981073a0 2024-03-24T00:20:34.466 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:20:34.463+0000 7f0895600640 1 -- 172.21.15.118:0/2028022490 <== mon.2 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0890039040 con 0x7f08981073a0 2024-03-24T00:20:34.466 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:20:34.463+0000 7f089ea00640 1 -- 172.21.15.118:0/2028022490 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f08980fff30 con 0x7f08981073a0 2024-03-24T00:20:34.466 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:20:34.463+0000 7f089ea00640 1 -- 172.21.15.118:0/2028022490 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f08981003d0 con 0x7f08981073a0 2024-03-24T00:20:34.475 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:20:34.467+0000 7f08a1812740 1 Processor -- start 2024-03-24T00:20:34.475 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:20:34.467+0000 7f0895600640 1 -- 172.21.15.118:0/2028022490 <== mon.2 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 20) v1 ==== 137291+0+0 (secure 0 0 0) 0x7f0890044020 con 0x7f08981073a0 2024-03-24T00:20:34.475 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:20:34.468+0000 7f08a1812740 1 -- start start 2024-03-24T00:20:34.475 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:20:34.468+0000 7f08a1812740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56135b7061e0 0x56135b706600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:20:34.475 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:20:34.468+0000 7f08a1812740 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x56135b706bd0 0x56135b70b6d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:20:34.475 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:20:34.468+0000 7f08a1812740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56135b70bd40 0x56135b70e1d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:20:34.475 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:20:34.468+0000 7f08a1812740 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x56135b6c93b0 con 0x56135b706bd0 2024-03-24T00:20:34.476 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:20:34.468+0000 7f08a1812740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x56135b66b830 con 0x56135b7061e0 2024-03-24T00:20:34.476 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:20:34.468+0000 7f08a1812740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x56135b6145d0 con 0x56135b70bd40 2024-03-24T00:20:34.476 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:20:34.468+0000 7f0897e00640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x56135b706bd0 0x56135b70b6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:20:34.476 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:20:34.468+0000 7f0897e00640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x56135b706bd0 0x56135b70b6d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.118:3300/0 says I am v2:172.21.15.118:38936/0 (socket says 172.21.15.118:38936) 2024-03-24T00:20:34.476 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:20:34.468+0000 7f0897e00640 1 -- 172.21.15.118:0/381989698 learned_addr learned my addr 172.21.15.118:0/381989698 (peer_addr_for_me v2:172.21.15.118:0/0) 2024-03-24T00:20:34.476 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:20:34.468+0000 7f088fe00640 1 --2- 172.21.15.118:0/381989698 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56135b7061e0 0x56135b706600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:20:34.476 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:20:34.468+0000 7f088fe00640 1 --2- 172.21.15.118:0/381989698 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56135b70bd40 0x56135b70e1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:20:34.476 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:20:34.469+0000 7f0897e00640 1 -- 172.21.15.118:0/381989698 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56135b70bd40 msgr2=0x56135b70e1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:20:34.476 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:20:34.469+0000 7f0897e00640 1 --2- 172.21.15.118:0/381989698 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56135b70bd40 0x56135b70e1d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:20:34.477 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:20:34.469+0000 7f0897e00640 1 -- 172.21.15.118:0/381989698 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56135b7061e0 msgr2=0x56135b706600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:20:34.477 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:20:34.469+0000 7f0897e00640 1 --2- 172.21.15.118:0/381989698 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56135b7061e0 0x56135b706600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:20:34.477 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:20:34.469+0000 7f0897e00640 1 -- 172.21.15.118:0/381989698 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x56135b62c2c0 con 0x56135b706bd0 2024-03-24T00:20:34.477 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:20:34.469+0000 7f088fe00640 1 --2- 172.21.15.118:0/381989698 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56135b70bd40 0x56135b70e1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-24T00:20:34.477 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:20:34.469+0000 7f0895600640 1 --2- 172.21.15.118:0/2028022490 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7f087c0a2920 0x7f087c0a4de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:20:34.477 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:20:34.469+0000 7f0897e00640 1 --2- 172.21.15.118:0/381989698 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x56135b706bd0 0x56135b70b6d0 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7f0888009380 tx=0x7f0888009840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4a35f98f124fa30e server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:20:34.477 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:20:34.469+0000 7f0895600640 1 -- 172.21.15.118:0/2028022490 <== mon.2 v2:172.21.15.189:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 8621+0+0 (secure 0 0 0) 0x7f08900f0b00 con 0x7f08981073a0 2024-03-24T00:20:34.477 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:20:34.469+0000 7f088fe00640 1 --2- 172.21.15.118:0/2028022490 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7f087c0a2920 0x7f087c0a4de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:20:34.477 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:20:34.469+0000 7f088d200640 1 -- 172.21.15.118:0/381989698 <== mon.0 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0888005020 con 0x56135b706bd0 2024-03-24T00:20:34.477 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:20:34.469+0000 7f088d200640 1 -- 172.21.15.118:0/381989698 <== mon.0 v2:172.21.15.118:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f08880047a0 con 0x56135b706bd0 2024-03-24T00:20:34.477 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:20:34.469+0000 7f088d200640 1 -- 172.21.15.118:0/381989698 <== mon.0 v2:172.21.15.118:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f088801a880 con 0x56135b706bd0 2024-03-24T00:20:34.477 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:20:34.470+0000 7f08a1812740 1 -- 172.21.15.118:0/381989698 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x56135b706bd0 msgr2=0x56135b70b6d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:20:34.477 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:20:34.470+0000 7f08a1812740 1 --2- 172.21.15.118:0/381989698 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x56135b706bd0 0x56135b70b6d0 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7f0888009380 tx=0x7f0888009840 comp rx=0 tx=0).stop 2024-03-24T00:20:34.478 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:20:34.470+0000 7f088fe00640 1 --2- 172.21.15.118:0/2028022490 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7f087c0a2920 0x7f087c0a4de0 secure :-1 s=READY pgs=354 cs=0 l=1 rev1=1 crypto rx=0x56135b707430 tx=0x7f0880014040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:20:34.478 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:20:34.470+0000 7f08a1812740 1 -- 172.21.15.118:0/381989698 shutdown_connections 2024-03-24T00:20:34.478 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:20:34.470+0000 7f08a1812740 1 --2- 172.21.15.118:0/381989698 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56135b70bd40 0x56135b70e1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:20:34.478 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:20:34.470+0000 7f08a1812740 1 --2- 172.21.15.118:0/381989698 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x56135b706bd0 0x56135b70b6d0 unknown :-1 s=CLOSED pgs=211 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:20:34.478 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:20:34.470+0000 7f08a1812740 1 --2- 172.21.15.118:0/381989698 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56135b7061e0 0x56135b706600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:20:34.478 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:20:34.470+0000 7f08a1812740 1 -- 172.21.15.118:0/381989698 >> 172.21.15.118:0/381989698 conn(0x56135b701750 msgr2=0x56135b701b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:20:34.478 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:20:34.470+0000 7f08a1812740 1 -- 172.21.15.118:0/381989698 shutdown_connections 2024-03-24T00:20:34.478 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:20:34.470+0000 7f08a1812740 1 -- 172.21.15.118:0/381989698 wait complete. 2024-03-24T00:20:34.478 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:20:34.471+0000 7f08a1812740 1 -- start start 2024-03-24T00:20:34.478 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:20:34.471+0000 7f08a1812740 1 Processor -- start 2024-03-24T00:20:34.478 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:20:34.471+0000 7f08a1812740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56135b7061e0 0x56135b710340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:20:34.478 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:20:34.471+0000 7f08a1812740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56135b706bd0 0x56135b710880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:20:34.478 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:20:34.471+0000 7f08a1812740 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x56135b70bd40 0x56135b7178c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:20:34.479 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:20:34.471+0000 7f08a1812740 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x56135b627a20 con 0x56135b70bd40 2024-03-24T00:20:34.479 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:20:34.471+0000 7f08a1812740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x56135b6678d0 con 0x56135b7061e0 2024-03-24T00:20:34.479 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:20:34.471+0000 7f08a1812740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x56135b70ecf0 con 0x56135b706bd0 2024-03-24T00:20:34.479 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:20:34.471+0000 7f08a1812740 10 client.0 _ll_register_callbacks cb 0x56135b6641c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T00:20:34.479 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:20:34.472+0000 7f0897e00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56135b7061e0 0x56135b710340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:20:34.479 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:20:34.472+0000 7f0897e00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56135b7061e0 0x56135b710340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.118:44132/0 (socket says 172.21.15.118:44132) 2024-03-24T00:20:34.479 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:20:34.472+0000 7f0897e00640 1 -- 172.21.15.118:0/545458555 learned_addr learned my addr 172.21.15.118:0/545458555 (peer_addr_for_me v2:172.21.15.118:0/0) 2024-03-24T00:20:34.479 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:20:34.472+0000 7f089ca00640 1 --2- 172.21.15.118:0/545458555 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x56135b70bd40 0x56135b7178c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:20:34.479 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:20:34.472+0000 7f088fe00640 1 --2- 172.21.15.118:0/545458555 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56135b706bd0 0x56135b710880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:20:34.479 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:20:34.472+0000 7f0897e00640 1 -- 172.21.15.118:0/545458555 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56135b706bd0 msgr2=0x56135b710880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:20:34.479 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:20:34.472+0000 7f0897e00640 1 --2- 172.21.15.118:0/545458555 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56135b706bd0 0x56135b710880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:20:34.479 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:20:34.472+0000 7f0897e00640 1 -- 172.21.15.118:0/545458555 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x56135b70bd40 msgr2=0x56135b7178c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:20:34.479 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:20:34.472+0000 7f0897e00640 1 --2- 172.21.15.118:0/545458555 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x56135b70bd40 0x56135b7178c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:20:34.480 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:20:34.472+0000 7f0897e00640 1 -- 172.21.15.118:0/545458555 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x56135b380210 con 0x56135b7061e0 2024-03-24T00:20:34.480 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:20:34.472+0000 7f0897e00640 1 --2- 172.21.15.118:0/545458555 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56135b7061e0 0x56135b710340 secure :-1 s=READY pgs=407 cs=0 l=1 rev1=1 crypto rx=0x7f0888000c00 tx=0x7f0888004f20 comp rx=0 tx=0).ready entity=mon.2 client_cookie=6e69c4e40a554f68 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:20:34.480 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:20:34.472+0000 7f0885600640 10 client.0 ms_handle_connect on v2:172.21.15.189:3300/0 2024-03-24T00:20:34.480 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:20:34.473+0000 7f0885600640 1 -- 172.21.15.118:0/545458555 <== mon.2 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0888027030 con 0x56135b7061e0 2024-03-24T00:20:34.480 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:20:34.473+0000 7f0885600640 1 -- 172.21.15.118:0/545458555 <== mon.2 v2:172.21.15.189:3300/0 2 ==== osd_map(94..94 src has 1..94) v4 ==== 8621+0+0 (secure 0 0 0) 0x7f088801f050 con 0x56135b7061e0 2024-03-24T00:20:34.480 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:20:34.473+0000 7f08a1812740 1 -- 172.21.15.118:0/545458555 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f08880047a0 con 0x56135b7061e0 2024-03-24T00:20:34.480 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:20:34.473+0000 7f0885600640 1 -- 172.21.15.118:0/545458555 <== mon.2 v2:172.21.15.189:3300/0 3 ==== mon_get_version_reply(handle=1 version=52) v2 ==== 24+0+0 (secure 0 0 0) 0x7f0888033050 con 0x56135b7061e0 2024-03-24T00:20:34.480 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:20:34.473+0000 7f08a1812740 10 client.26657 fetch_fsmap learned FSMap version 52 2024-03-24T00:20:34.480 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:20:34.473+0000 7f08a1812740 1 -- 172.21.15.118:0/545458555 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,fsmap=52,monmap=4+}) v3 -- 0x56135b44c620 con 0x56135b7061e0 2024-03-24T00:20:34.480 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:20:34.473+0000 7f0885600640 1 -- 172.21.15.118:0/545458555 <== mon.2 v2:172.21.15.189:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f08880047a0 con 0x56135b7061e0 2024-03-24T00:20:34.480 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:20:34.473+0000 7f0885600640 1 -- 172.21.15.118:0/545458555 <== mon.2 v2:172.21.15.189:3300/0 5 ==== fsmap(e 52) v1 ==== 7946+0+0 (secure 0 0 0) 0x7f0888023440 con 0x56135b7061e0 2024-03-24T00:20:34.480 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:20:34.473+0000 7f08a1812740 10 client.26657 fetch_fsmap finished waiting for FSMap version 52 2024-03-24T00:20:34.480 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:20:34.473+0000 7f08a1812740 10 client.26657 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-24T00:20:34.480 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:20:34.473+0000 7f08a1812740 20 client.26657 populate_metadata read hostname 'smithi118' 2024-03-24T00:20:34.481 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:20:34.474+0000 7f08a1812740 1 --2- 172.21.15.118:0/545458555 >> [v2:172.21.15.189:6838/4074484168,v1:172.21.15.189:6839/4074484168] conn(0x56135b71bd80 0x56135b71e170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:20:34.481 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:20:34.474+0000 7f08a1812740 4 client.26657 mds_command: new command op to 24569 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-24T00:20:34.481 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:20:34.474+0000 7f08a1812740 1 -- 172.21.15.118:0/545458555 --> [v2:172.21.15.189:6838/4074484168,v1:172.21.15.189:6839/4074484168] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x56135b678d70 con 0x56135b71bd80 2024-03-24T00:20:34.481 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:20:34.474+0000 7f088fe00640 1 --2- 172.21.15.118:0/545458555 >> [v2:172.21.15.189:6838/4074484168,v1:172.21.15.189:6839/4074484168] conn(0x56135b71bd80 0x56135b71e170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:20:34.481 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:20:34.474+0000 7f088fe00640 1 --2- 172.21.15.118:0/545458555 >> [v2:172.21.15.189:6838/4074484168,v1:172.21.15.189:6839/4074484168] conn(0x56135b71bd80 0x56135b71e170 crc :-1 s=READY pgs=152 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=8b1dc8b27451b557 server_cookie=cf73536bfb09c680 in_seq=0 out_seq=0 2024-03-24T00:20:34.481 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:20:34.474+0000 7f0885600640 10 client.26657 ms_handle_connect on v2:172.21.15.189:6838/4074484168 2024-03-24T00:20:34.482 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:20:34.475+0000 7f0885600640 1 -- 172.21.15.118:0/545458555 <== mds.0 v2:172.21.15.189:6838/4074484168 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x56135b678d70 con 0x56135b71bd80 2024-03-24T00:20:34.482 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:20:34.475+0000 7f0885600640 10 client.26657 handle_command_reply: tid=0 2024-03-24T00:20:34.483 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:20:34.475+0000 7f08a1812740 1 client.26657 shutdown 2024-03-24T00:20:34.483 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:20:34.475+0000 7f08a1812740 1 -- 172.21.15.118:0/545458555 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56135b7061e0 msgr2=0x56135b710340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:20:34.483 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:20:34.475+0000 7f08a1812740 1 --2- 172.21.15.118:0/545458555 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56135b7061e0 0x56135b710340 secure :-1 s=READY pgs=407 cs=0 l=1 rev1=1 crypto rx=0x7f0888000c00 tx=0x7f0888004f20 comp rx=0 tx=0).stop 2024-03-24T00:20:34.483 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:20:34.475+0000 7f08a1812740 1 -- 172.21.15.118:0/545458555 shutdown_connections 2024-03-24T00:20:34.483 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:20:34.475+0000 7f08a1812740 1 --2- 172.21.15.118:0/545458555 >> [v2:172.21.15.189:6838/4074484168,v1:172.21.15.189:6839/4074484168] conn(0x56135b71bd80 0x56135b71e170 crc :-1 s=READY pgs=152 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:20:34.483 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:20:34.475+0000 7f08a1812740 1 --2- 172.21.15.118:0/545458555 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x56135b70bd40 0x56135b7178c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:20:34.483 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:20:34.475+0000 7f08a1812740 1 --2- 172.21.15.118:0/545458555 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56135b706bd0 0x56135b710880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:20:34.483 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:20:34.475+0000 7f0885600640 0 client.26657 ms_handle_reset on v2:172.21.15.189:6838/4074484168 2024-03-24T00:20:34.483 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:20:34.475+0000 7f08a1812740 1 --2- 172.21.15.118:0/545458555 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56135b7061e0 0x56135b710340 unknown :-1 s=CLOSED pgs=407 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:20:34.483 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:20:34.475+0000 7f08a1812740 1 -- 172.21.15.118:0/545458555 >> 172.21.15.118:0/545458555 conn(0x56135b701750 msgr2=0x56135b700950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:20:34.483 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:20:34.476+0000 7f08a1812740 1 -- 172.21.15.118:0/545458555 shutdown_connections 2024-03-24T00:20:34.483 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:20:34.476+0000 7f08a1812740 1 -- 172.21.15.118:0/545458555 wait complete. 2024-03-24T00:20:34.483 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:20:34.476+0000 7f08a1812740 20 client.26657 trim_cache size 0 max 16384 2024-03-24T00:20:34.494 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:20:34.491+0000 7f08a1812740 1 Processor -- start 2024-03-24T00:20:34.494 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:20:34.491+0000 7f08a1812740 1 -- start start 2024-03-24T00:20:34.494 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:20:34.491+0000 7f08a1812740 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x56135b7061e0 0x56135b70b4c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:20:34.494 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:20:34.491+0000 7f08a1812740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56135b706bd0 0x56135b722a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:20:34.495 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:20:34.491+0000 7f08a1812740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56135b70bd40 0x56135b722f70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:20:34.495 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:20:34.491+0000 7f08a1812740 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f08981a7110 con 0x56135b7061e0 2024-03-24T00:20:34.495 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:20:34.491+0000 7f08a1812740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f08981a8490 con 0x56135b70bd40 2024-03-24T00:20:34.495 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:20:34.491+0000 7f08a1812740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f08981a7770 con 0x56135b706bd0 2024-03-24T00:20:34.495 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:20:34.491+0000 7f0897e00640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x56135b7061e0 0x56135b70b4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:20:34.496 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:20:34.491+0000 7f0897e00640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x56135b7061e0 0x56135b70b4c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.118:3300/0 says I am v2:172.21.15.118:38952/0 (socket says 172.21.15.118:38952) 2024-03-24T00:20:34.496 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:20:34.491+0000 7f0897e00640 1 -- 172.21.15.118:0/846891000 learned_addr learned my addr 172.21.15.118:0/846891000 (peer_addr_for_me v2:172.21.15.118:0/0) 2024-03-24T00:20:34.496 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:20:34.491+0000 7f088fe00640 1 --2- 172.21.15.118:0/846891000 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56135b706bd0 0x56135b722a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:20:34.496 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:20:34.491+0000 7f089ca00640 1 --2- 172.21.15.118:0/846891000 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56135b70bd40 0x56135b722f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:20:34.496 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:20:34.491+0000 7f088fe00640 1 -- 172.21.15.118:0/846891000 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56135b70bd40 msgr2=0x56135b722f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:20:34.496 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:20:34.491+0000 7f088fe00640 1 --2- 172.21.15.118:0/846891000 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56135b70bd40 0x56135b722f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:20:34.496 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:20:34.492+0000 7f088fe00640 1 -- 172.21.15.118:0/846891000 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x56135b7061e0 msgr2=0x56135b70b4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:20:34.496 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:20:34.492+0000 7f088fe00640 1 --2- 172.21.15.118:0/846891000 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x56135b7061e0 0x56135b70b4c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:20:34.496 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:20:34.492+0000 7f088fe00640 1 -- 172.21.15.118:0/846891000 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x56135b70ebd0 con 0x56135b706bd0 2024-03-24T00:20:34.496 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:20:34.492+0000 7f0897e00640 1 --2- 172.21.15.118:0/846891000 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x56135b7061e0 0x56135b70b4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T00:20:34.496 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:20:34.492+0000 7f088fe00640 1 --2- 172.21.15.118:0/846891000 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56135b706bd0 0x56135b722a30 secure :-1 s=READY pgs=412 cs=0 l=1 rev1=1 crypto rx=0x7f0880012fd0 tx=0x7f0880016000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=a0da7b8f4c649685 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:20:34.496 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:20:34.492+0000 7f0887e00640 1 -- 172.21.15.118:0/846891000 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f088001ece0 con 0x56135b706bd0 2024-03-24T00:20:34.496 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:20:34.492+0000 7f0887e00640 1 -- 172.21.15.118:0/846891000 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f088002e070 con 0x56135b706bd0 2024-03-24T00:20:34.497 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:20:34.492+0000 7f0887e00640 1 -- 172.21.15.118:0/846891000 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f088001d810 con 0x56135b706bd0 2024-03-24T00:20:34.497 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:20:34.493+0000 7f08a1812740 1 -- 172.21.15.118:0/846891000 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56135b706bd0 msgr2=0x56135b722a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:20:34.497 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:20:34.493+0000 7f08a1812740 1 --2- 172.21.15.118:0/846891000 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56135b706bd0 0x56135b722a30 secure :-1 s=READY pgs=412 cs=0 l=1 rev1=1 crypto rx=0x7f0880012fd0 tx=0x7f0880016000 comp rx=0 tx=0).stop 2024-03-24T00:20:34.497 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:20:34.493+0000 7f08a1812740 1 -- 172.21.15.118:0/846891000 shutdown_connections 2024-03-24T00:20:34.497 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:20:34.493+0000 7f08a1812740 1 --2- 172.21.15.118:0/846891000 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56135b70bd40 0x56135b722f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:20:34.497 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:20:34.493+0000 7f08a1812740 1 --2- 172.21.15.118:0/846891000 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56135b706bd0 0x56135b722a30 unknown :-1 s=CLOSED pgs=412 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:20:34.497 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:20:34.493+0000 7f08a1812740 1 --2- 172.21.15.118:0/846891000 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x56135b7061e0 0x56135b70b4c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:20:34.497 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:20:34.493+0000 7f08a1812740 1 -- 172.21.15.118:0/846891000 >> 172.21.15.118:0/846891000 conn(0x56135b701750 msgr2=0x56135b7049b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:20:34.497 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:20:34.493+0000 7f08a1812740 1 -- 172.21.15.118:0/846891000 shutdown_connections 2024-03-24T00:20:34.497 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:20:34.493+0000 7f08a1812740 1 -- 172.21.15.118:0/846891000 wait complete. 2024-03-24T00:20:34.497 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:20:34.493+0000 7f08a1812740 1 -- start start 2024-03-24T00:20:34.497 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:20:34.493+0000 7f08a1812740 1 Processor -- start 2024-03-24T00:20:34.500 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:20:34.494+0000 7f08a1812740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56135b7061e0 0x56135b714f10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:20:34.500 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:20:34.494+0000 7f08a1812740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56135b706bd0 0x56135b715450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:20:34.500 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:20:34.494+0000 7f08a1812740 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x56135b70bd40 0x56135b716460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:20:34.500 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:20:34.494+0000 7f08a1812740 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f08981a5890 con 0x56135b70bd40 2024-03-24T00:20:34.500 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:20:34.494+0000 7f08a1812740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f08981a7e30 con 0x56135b7061e0 2024-03-24T00:20:34.501 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:20:34.494+0000 7f08a1812740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f08981a4c50 con 0x56135b706bd0 2024-03-24T00:20:34.501 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:20:34.494+0000 7f08a1812740 10 client.0 _ll_register_callbacks cb 0x7f08880092a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T00:20:34.501 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:20:34.494+0000 7f088fe00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56135b706bd0 0x56135b715450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:20:34.501 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:20:34.494+0000 7f089ca00640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x56135b70bd40 0x56135b716460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:20:34.501 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:20:34.494+0000 7f0897e00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56135b7061e0 0x56135b714f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:20:34.501 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:20:34.494+0000 7f088fe00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56135b706bd0 0x56135b715450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.118:39596/0 (socket says 172.21.15.118:39596) 2024-03-24T00:20:34.501 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:20:34.494+0000 7f088fe00640 1 -- 172.21.15.118:0/875879641 learned_addr learned my addr 172.21.15.118:0/875879641 (peer_addr_for_me v2:172.21.15.118:0/0) 2024-03-24T00:20:34.501 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:20:34.494+0000 7f088fe00640 1 -- 172.21.15.118:0/875879641 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56135b7061e0 msgr2=0x56135b714f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:20:34.501 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:20:34.494+0000 7f088fe00640 1 --2- 172.21.15.118:0/875879641 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56135b7061e0 0x56135b714f10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:20:34.501 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:20:34.494+0000 7f088fe00640 1 -- 172.21.15.118:0/875879641 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x56135b70bd40 msgr2=0x56135b716460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:20:34.501 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:20:34.494+0000 7f088fe00640 1 --2- 172.21.15.118:0/875879641 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x56135b70bd40 0x56135b716460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:20:34.501 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:20:34.494+0000 7f088fe00640 1 -- 172.21.15.118:0/875879641 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x56135b716a50 con 0x56135b706bd0 2024-03-24T00:20:34.501 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:20:34.495+0000 7f088fe00640 1 --2- 172.21.15.118:0/875879641 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56135b706bd0 0x56135b715450 secure :-1 s=READY pgs=413 cs=0 l=1 rev1=1 crypto rx=0x7f088000b690 tx=0x7f088001ece0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=20d065502015de15 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:20:34.502 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:20:34.495+0000 7f0885600640 10 client.0 ms_handle_connect on v2:172.21.15.193:3300/0 2024-03-24T00:20:34.502 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:20:34.495+0000 7f0885600640 1 -- 172.21.15.118:0/875879641 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0880036070 con 0x56135b706bd0 2024-03-24T00:20:34.502 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:20:34.495+0000 7f08a1812740 1 -- 172.21.15.118:0/875879641 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f088002e070 con 0x56135b706bd0 2024-03-24T00:20:34.502 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:20:34.495+0000 7f0885600640 1 -- 172.21.15.118:0/875879641 <== mon.1 v2:172.21.15.193:3300/0 2 ==== osd_map(94..94 src has 1..94) v4 ==== 8621+0+0 (secure 0 0 0) 0x7f0880029a20 con 0x56135b706bd0 2024-03-24T00:20:34.502 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:20:34.495+0000 7f0885600640 1 -- 172.21.15.118:0/875879641 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_get_version_reply(handle=1 version=52) v2 ==== 24+0+0 (secure 0 0 0) 0x7f0880029c20 con 0x56135b706bd0 2024-03-24T00:20:34.502 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:20:34.495+0000 7f08a1812740 10 client.26677 fetch_fsmap learned FSMap version 52 2024-03-24T00:20:34.502 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:20:34.495+0000 7f08a1812740 1 -- 172.21.15.118:0/875879641 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,fsmap=52,monmap=4+}) v3 -- 0x56135b777de0 con 0x56135b706bd0 2024-03-24T00:20:34.502 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:20:34.496+0000 7f0885600640 1 -- 172.21.15.118:0/875879641 <== mon.1 v2:172.21.15.193:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f088002e070 con 0x56135b706bd0 2024-03-24T00:20:34.502 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:20:34.496+0000 7f0885600640 1 -- 172.21.15.118:0/875879641 <== mon.1 v2:172.21.15.193:3300/0 5 ==== fsmap(e 52) v1 ==== 7946+0+0 (secure 0 0 0) 0x7f088003b540 con 0x56135b706bd0 2024-03-24T00:20:34.502 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:20:34.496+0000 7f08a1812740 10 client.26677 fetch_fsmap finished waiting for FSMap version 52 2024-03-24T00:20:34.502 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:20:34.496+0000 7f08a1812740 10 client.26677 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-24T00:20:34.502 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:20:34.496+0000 7f08a1812740 20 client.26677 populate_metadata read hostname 'smithi118' 2024-03-24T00:20:34.502 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:20:34.496+0000 7f08a1812740 1 --2- 172.21.15.118:0/875879641 >> [v2:172.21.15.189:6838/4074484168,v1:172.21.15.189:6839/4074484168] conn(0x56135b7786b0 0x56135b77aaf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:20:34.503 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:20:34.496+0000 7f08a1812740 4 client.26677 mds_command: new command op to 24569 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-03-24T00:20:34.503 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:20:34.496+0000 7f08a1812740 1 -- 172.21.15.118:0/875879641 --> [v2:172.21.15.189:6838/4074484168,v1:172.21.15.189:6839/4074484168] -- command(tid 0: {"prefix": "flush journal"}) v1 -- 0x56135b6fca80 con 0x56135b7786b0 2024-03-24T00:20:34.503 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:20:34.496+0000 7f0897e00640 1 --2- 172.21.15.118:0/875879641 >> [v2:172.21.15.189:6838/4074484168,v1:172.21.15.189:6839/4074484168] conn(0x56135b7786b0 0x56135b77aaf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:20:34.503 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:20:34.497+0000 7f0885600640 10 client.26677 ms_handle_connect on v2:172.21.15.189:6838/4074484168 2024-03-24T00:20:34.503 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:20:34.497+0000 7f0897e00640 1 --2- 172.21.15.118:0/875879641 >> [v2:172.21.15.189:6838/4074484168,v1:172.21.15.189:6839/4074484168] conn(0x56135b7786b0 0x56135b77aaf0 crc :-1 s=READY pgs=153 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=95803183d067410d server_cookie=7758d45f745c04c5 in_seq=0 out_seq=0 2024-03-24T00:20:34.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:20:34 smithi118 ceph-mon[31535]: pgmap v548: 97 pgs: 97 active+clean; 101 MiB data, 671 MiB used, 1.0 TiB / 1.0 TiB avail; 169 KiB/s rd, 175 KiB/s wr, 71 op/s 2024-03-24T00:20:34.703 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:20:34 smithi189 ceph-mon[36187]: pgmap v548: 97 pgs: 97 active+clean; 101 MiB data, 671 MiB used, 1.0 TiB / 1.0 TiB avail; 169 KiB/s rd, 175 KiB/s wr, 71 op/s 2024-03-24T00:20:34.710 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:20:34 smithi193 ceph-mon[36878]: pgmap v548: 97 pgs: 97 active+clean; 101 MiB data, 671 MiB used, 1.0 TiB / 1.0 TiB avail; 169 KiB/s rd, 175 KiB/s wr, 71 op/s 2024-03-24T00:20:35.597 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:20:35 smithi118 ceph-mon[31535]: mds.? [v2:172.21.15.193:6834/1699655293,v1:172.21.15.193:6835/1699655293] up:resolve 2024-03-24T00:20:35.598 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:20:35 smithi118 ceph-mon[31535]: fsmap cephfs:5/5 4 up:standby-replay 3 up:standby 1 up:resolve 4 up:active 2024-03-24T00:20:35.703 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:20:35 smithi189 ceph-mon[36187]: mds.? [v2:172.21.15.193:6834/1699655293,v1:172.21.15.193:6835/1699655293] up:resolve 2024-03-24T00:20:35.704 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:20:35 smithi189 ceph-mon[36187]: fsmap cephfs:5/5 4 up:standby-replay 3 up:standby 1 up:resolve 4 up:active 2024-03-24T00:20:35.710 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:20:35 smithi193 ceph-mon[36878]: mds.? [v2:172.21.15.193:6834/1699655293,v1:172.21.15.193:6835/1699655293] up:resolve 2024-03-24T00:20:35.710 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:20:35 smithi193 ceph-mon[36878]: fsmap cephfs:5/5 4 up:standby-replay 3 up:standby 1 up:resolve 4 up:active 2024-03-24T00:20:36.367 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:20:36.367+0000 7f0885600640 1 -- 172.21.15.118:0/875879641 <== mds.0 v2:172.21.15.189:6838/4074484168 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+44 (crc 0 0 0) 0x56135b6fca80 con 0x56135b7786b0 2024-03-24T00:20:36.367 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:20:36.367+0000 7f0885600640 10 client.26677 handle_command_reply: tid=0 2024-03-24T00:20:36.367 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:20:36.367+0000 7f08a1812740 1 client.26677 shutdown 2024-03-24T00:20:36.367 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:20:36.367+0000 7f08a1812740 1 -- 172.21.15.118:0/875879641 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56135b706bd0 msgr2=0x56135b715450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:20:36.367 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:20:36.367+0000 7f08a1812740 1 --2- 172.21.15.118:0/875879641 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56135b706bd0 0x56135b715450 secure :-1 s=READY pgs=413 cs=0 l=1 rev1=1 crypto rx=0x7f088000b690 tx=0x7f088001ece0 comp rx=0 tx=0).stop 2024-03-24T00:20:36.368 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:20:36.367+0000 7f08a1812740 1 -- 172.21.15.118:0/875879641 shutdown_connections 2024-03-24T00:20:36.368 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:20:36.367+0000 7f08a1812740 1 --2- 172.21.15.118:0/875879641 >> [v2:172.21.15.189:6838/4074484168,v1:172.21.15.189:6839/4074484168] conn(0x56135b7786b0 0x56135b77aaf0 crc :-1 s=READY pgs=153 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:20:36.368 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:20:36.367+0000 7f08a1812740 1 --2- 172.21.15.118:0/875879641 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x56135b70bd40 0x56135b716460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:20:36.368 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:20:36.367+0000 7f08a1812740 1 --2- 172.21.15.118:0/875879641 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56135b706bd0 0x56135b715450 unknown :-1 s=CLOSED pgs=413 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:20:36.368 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:20:36.367+0000 7f08a1812740 1 --2- 172.21.15.118:0/875879641 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56135b7061e0 0x56135b714f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:20:36.368 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:20:36.367+0000 7f0885600640 0 client.26677 ms_handle_reset on v2:172.21.15.189:6838/4074484168 2024-03-24T00:20:36.368 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:20:36.367+0000 7f08a1812740 1 -- 172.21.15.118:0/875879641 >> 172.21.15.118:0/875879641 conn(0x56135b701750 msgr2=0x56135b7049b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:20:36.368 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:20:36.367+0000 7f08a1812740 1 -- 172.21.15.118:0/875879641 shutdown_connections 2024-03-24T00:20:36.368 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:20:36.367+0000 7f08a1812740 1 -- 172.21.15.118:0/875879641 wait complete. 2024-03-24T00:20:36.368 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:20:36.368+0000 7f08a1812740 20 client.26677 trim_cache size 0 max 16384 2024-03-24T00:20:36.369 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stdout:{ 2024-03-24T00:20:36.369 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stdout: "message": "", 2024-03-24T00:20:36.369 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stdout: "return_code": 0 2024-03-24T00:20:36.369 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stdout:} 2024-03-24T00:20:36.370 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:20:36.369+0000 7f089ea00640 1 -- 172.21.15.118:0/2028022490 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7f087c0a2920 msgr2=0x7f087c0a4de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:20:36.370 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:20:36.369+0000 7f089ea00640 1 --2- 172.21.15.118:0/2028022490 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7f087c0a2920 0x7f087c0a4de0 secure :-1 s=READY pgs=354 cs=0 l=1 rev1=1 crypto rx=0x56135b707430 tx=0x7f0880014040 comp rx=0 tx=0).stop 2024-03-24T00:20:36.370 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:20:36.369+0000 7f089ea00640 1 -- 172.21.15.118:0/2028022490 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f08981073a0 msgr2=0x7f089819ee50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:20:36.370 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:20:36.369+0000 7f089ea00640 1 --2- 172.21.15.118:0/2028022490 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f08981073a0 0x7f089819ee50 secure :-1 s=READY pgs=406 cs=0 l=1 rev1=1 crypto rx=0x7f0890005e30 tx=0x7f0890032ec0 comp rx=0 tx=0).stop 2024-03-24T00:20:36.370 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:20:36.369+0000 7f089ea00640 1 -- 172.21.15.118:0/2028022490 shutdown_connections 2024-03-24T00:20:36.370 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:20:36.369+0000 7f089ea00640 1 --2- 172.21.15.118:0/2028022490 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7f087c0a2920 0x7f087c0a4de0 unknown :-1 s=CLOSED pgs=354 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:20:36.370 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:20:36.369+0000 7f089ea00640 1 --2- 172.21.15.118:0/2028022490 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f08981073a0 0x7f089819ee50 unknown :-1 s=CLOSED pgs=406 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:20:36.370 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:20:36.369+0000 7f089ea00640 1 --2- 172.21.15.118:0/2028022490 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f0898104a70 0x7f0898197de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:20:36.370 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:20:36.369+0000 7f089ea00640 1 --2- 172.21.15.118:0/2028022490 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f0898102140 0x7f08981978a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:20:36.370 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:20:36.369+0000 7f089ea00640 1 -- 172.21.15.118:0/2028022490 >> 172.21.15.118:0/2028022490 conn(0x7f08980fbb30 msgr2=0x7f0898102a40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:20:36.371 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:20:36.369+0000 7f089ea00640 1 -- 172.21.15.118:0/2028022490 shutdown_connections 2024-03-24T00:20:36.371 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:20:36.369+0000 7f089ea00640 1 -- 172.21.15.118:0/2028022490 wait complete. 2024-03-24T00:20:36.596 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:20:36 smithi193 ceph-mon[36878]: mds.? [v2:172.21.15.193:6834/1699655293,v1:172.21.15.193:6835/1699655293] up:reconnect 2024-03-24T00:20:36.596 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:20:36 smithi193 ceph-mon[36878]: fsmap cephfs:5/5 4 up:standby-replay 3 up:standby 1 up:reconnect 4 up:active 2024-03-24T00:20:36.596 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:20:36 smithi193 ceph-mon[36878]: reconnect by client.15159 192.168.0.1:0/3463739313 after 0.0139998 2024-03-24T00:20:36.596 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:20:36 smithi193 ceph-mon[36878]: pgmap v549: 97 pgs: 97 active+clean; 101 MiB data, 671 MiB used, 1.0 TiB / 1.0 TiB avail; 159 KiB/s rd, 171 KiB/s wr, 63 op/s 2024-03-24T00:20:36.596 INFO:journalctl@ceph.mds.f.smithi193.stdout:Mar 24 00:20:36 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-f[71102]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: In function 'void Locker::set_xlocks_done(MutationImpl*, bool)' thread 7fcc172d0700 time 2024-03-24T00:20:36.544253+0000 2024-03-24T00:20:36.597 INFO:journalctl@ceph.mds.f.smithi193.stdout:Mar 24 00:20:36 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-f[71102]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: 626: FAILED ceph_assert(obj->is_auth()) 2024-03-24T00:20:36.597 INFO:journalctl@ceph.mds.f.smithi193.stdout:Mar 24 00:20:36 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-f[71102]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-24T00:20:36.597 INFO:journalctl@ceph.mds.f.smithi193.stdout:Mar 24 00:20:36 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-f[71102]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x135) [0x7fcc218fa4a1] 2024-03-24T00:20:36.597 INFO:journalctl@ceph.mds.f.smithi193.stdout:Mar 24 00:20:36 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-f[71102]: 2: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7fcc218fa667] 2024-03-24T00:20:36.597 INFO:journalctl@ceph.mds.f.smithi193.stdout:Mar 24 00:20:36 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-f[71102]: 3: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x55c8b2347223] 2024-03-24T00:20:36.597 INFO:journalctl@ceph.mds.f.smithi193.stdout:Mar 24 00:20:36 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-f[71102]: 4: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x55c8b21c15d6] 2024-03-24T00:20:36.597 INFO:journalctl@ceph.mds.f.smithi193.stdout:Mar 24 00:20:36 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-f[71102]: 5: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x55c8b21c1da1] 2024-03-24T00:20:36.597 INFO:journalctl@ceph.mds.f.smithi193.stdout:Mar 24 00:20:36 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-f[71102]: 6: (Server::handle_client_rename(boost::intrusive_ptr const&)+0x259d) [0x55c8b21fe97d] 2024-03-24T00:20:36.597 INFO:journalctl@ceph.mds.f.smithi193.stdout:Mar 24 00:20:36 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-f[71102]: 7: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x55c8b2214233] 2024-03-24T00:20:36.597 INFO:journalctl@ceph.mds.f.smithi193.stdout:Mar 24 00:20:36 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-f[71102]: 8: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x55c8b2216197] 2024-03-24T00:20:36.597 INFO:journalctl@ceph.mds.f.smithi193.stdout:Mar 24 00:20:36 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-f[71102]: 9: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x55c8b211511b] 2024-03-24T00:20:36.597 INFO:journalctl@ceph.mds.f.smithi193.stdout:Mar 24 00:20:36 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-f[71102]: 10: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x55c8b21156bc] 2024-03-24T00:20:36.597 INFO:journalctl@ceph.mds.f.smithi193.stdout:Mar 24 00:20:36 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-f[71102]: 11: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x55c8b20ff1ff] 2024-03-24T00:20:36.597 INFO:journalctl@ceph.mds.f.smithi193.stdout:Mar 24 00:20:36 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-f[71102]: 12: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7fcc21b8c488] 2024-03-24T00:20:36.598 INFO:journalctl@ceph.mds.f.smithi193.stdout:Mar 24 00:20:36 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-f[71102]: 13: (DispatchQueue::entry()+0x51f) [0x7fcc21b8960f] 2024-03-24T00:20:36.598 INFO:journalctl@ceph.mds.f.smithi193.stdout:Mar 24 00:20:36 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-f[71102]: 14: (DispatchQueue::DispatchThread::entry()+0x11) [0x7fcc21c5ce81] 2024-03-24T00:20:36.598 INFO:journalctl@ceph.mds.f.smithi193.stdout:Mar 24 00:20:36 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-f[71102]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7fcc206771ca] 2024-03-24T00:20:36.598 INFO:journalctl@ceph.mds.f.smithi193.stdout:Mar 24 00:20:36 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-f[71102]: 16: clone() 2024-03-24T00:20:36.598 INFO:journalctl@ceph.mds.f.smithi193.stdout:Mar 24 00:20:36 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-f[71102]: 2024-03-24T00:20:36.545+0000 7fcc172d0700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: In function 'void Locker::set_xlocks_done(MutationImpl*, bool)' thread 7fcc172d0700 time 2024-03-24T00:20:36.544253+0000 2024-03-24T00:20:36.598 INFO:journalctl@ceph.mds.f.smithi193.stdout:Mar 24 00:20:36 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-f[71102]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: 626: FAILED ceph_assert(obj->is_auth()) 2024-03-24T00:20:36.598 INFO:journalctl@ceph.mds.f.smithi193.stdout:Mar 24 00:20:36 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-f[71102]: 2024-03-24T00:20:36.599 INFO:journalctl@ceph.mds.f.smithi193.stdout:Mar 24 00:20:36 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-f[71102]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-24T00:20:36.599 INFO:journalctl@ceph.mds.f.smithi193.stdout:Mar 24 00:20:36 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-f[71102]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x135) [0x7fcc218fa4a1] 2024-03-24T00:20:36.599 INFO:journalctl@ceph.mds.f.smithi193.stdout:Mar 24 00:20:36 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-f[71102]: 2: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7fcc218fa667] 2024-03-24T00:20:36.599 INFO:journalctl@ceph.mds.f.smithi193.stdout:Mar 24 00:20:36 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-f[71102]: 3: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x55c8b2347223] 2024-03-24T00:20:36.599 INFO:journalctl@ceph.mds.f.smithi193.stdout:Mar 24 00:20:36 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-f[71102]: 4: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x55c8b21c15d6] 2024-03-24T00:20:36.599 INFO:journalctl@ceph.mds.f.smithi193.stdout:Mar 24 00:20:36 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-f[71102]: 5: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x55c8b21c1da1] 2024-03-24T00:20:36.599 INFO:journalctl@ceph.mds.f.smithi193.stdout:Mar 24 00:20:36 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-f[71102]: 6: (Server::handle_client_rename(boost::intrusive_ptr const&)+0x259d) [0x55c8b21fe97d] 2024-03-24T00:20:36.599 INFO:journalctl@ceph.mds.f.smithi193.stdout:Mar 24 00:20:36 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-f[71102]: 7: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x55c8b2214233] 2024-03-24T00:20:36.599 INFO:journalctl@ceph.mds.f.smithi193.stdout:Mar 24 00:20:36 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-f[71102]: 8: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x55c8b2216197] 2024-03-24T00:20:36.599 INFO:journalctl@ceph.mds.f.smithi193.stdout:Mar 24 00:20:36 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-f[71102]: 9: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x55c8b211511b] 2024-03-24T00:20:36.599 INFO:journalctl@ceph.mds.f.smithi193.stdout:Mar 24 00:20:36 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-f[71102]: 10: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x55c8b21156bc] 2024-03-24T00:20:36.599 INFO:journalctl@ceph.mds.f.smithi193.stdout:Mar 24 00:20:36 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-f[71102]: 11: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x55c8b20ff1ff] 2024-03-24T00:20:36.599 INFO:journalctl@ceph.mds.f.smithi193.stdout:Mar 24 00:20:36 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-f[71102]: 12: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7fcc21b8c488] 2024-03-24T00:20:36.600 INFO:journalctl@ceph.mds.f.smithi193.stdout:Mar 24 00:20:36 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-f[71102]: 13: (DispatchQueue::entry()+0x51f) [0x7fcc21b8960f] 2024-03-24T00:20:36.600 INFO:journalctl@ceph.mds.f.smithi193.stdout:Mar 24 00:20:36 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-f[71102]: 14: (DispatchQueue::DispatchThread::entry()+0x11) [0x7fcc21c5ce81] 2024-03-24T00:20:36.600 INFO:journalctl@ceph.mds.f.smithi193.stdout:Mar 24 00:20:36 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-f[71102]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7fcc206771ca] 2024-03-24T00:20:36.600 INFO:journalctl@ceph.mds.f.smithi193.stdout:Mar 24 00:20:36 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-f[71102]: 16: clone() 2024-03-24T00:20:36.600 INFO:journalctl@ceph.mds.f.smithi193.stdout:Mar 24 00:20:36 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-f[71102]: 2024-03-24T00:20:36.600 INFO:journalctl@ceph.mds.f.smithi193.stdout:Mar 24 00:20:36 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-f[71102]: *** Caught signal (Aborted) ** 2024-03-24T00:20:36.600 INFO:journalctl@ceph.mds.f.smithi193.stdout:Mar 24 00:20:36 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-f[71102]: in thread 7fcc172d0700 thread_name:ms_dispatch 2024-03-24T00:20:36.600 INFO:journalctl@ceph.mds.f.smithi193.stdout:Mar 24 00:20:36 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-f[71102]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-24T00:20:36.600 INFO:journalctl@ceph.mds.f.smithi193.stdout:Mar 24 00:20:36 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-f[71102]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7fcc20681d20] 2024-03-24T00:20:36.600 INFO:journalctl@ceph.mds.f.smithi193.stdout:Mar 24 00:20:36 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-f[71102]: 2: gsignal() 2024-03-24T00:20:36.600 INFO:journalctl@ceph.mds.f.smithi193.stdout:Mar 24 00:20:36 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-f[71102]: 3: abort() 2024-03-24T00:20:36.600 INFO:journalctl@ceph.mds.f.smithi193.stdout:Mar 24 00:20:36 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-f[71102]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x18f) [0x7fcc218fa4fb] 2024-03-24T00:20:36.600 INFO:journalctl@ceph.mds.f.smithi193.stdout:Mar 24 00:20:36 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-f[71102]: 5: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7fcc218fa667] 2024-03-24T00:20:36.600 INFO:journalctl@ceph.mds.f.smithi193.stdout:Mar 24 00:20:36 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-f[71102]: 6: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x55c8b2347223] 2024-03-24T00:20:36.601 INFO:journalctl@ceph.mds.f.smithi193.stdout:Mar 24 00:20:36 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-f[71102]: 7: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x55c8b21c15d6] 2024-03-24T00:20:36.601 INFO:journalctl@ceph.mds.f.smithi193.stdout:Mar 24 00:20:36 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-f[71102]: 8: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x55c8b21c1da1] 2024-03-24T00:20:36.601 INFO:journalctl@ceph.mds.f.smithi193.stdout:Mar 24 00:20:36 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-f[71102]: 9: (Server::handle_client_rename(boost::intrusive_ptr const&)+0x259d) [0x55c8b21fe97d] 2024-03-24T00:20:36.601 INFO:journalctl@ceph.mds.f.smithi193.stdout:Mar 24 00:20:36 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-f[71102]: 10: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x55c8b2214233] 2024-03-24T00:20:36.601 INFO:journalctl@ceph.mds.f.smithi193.stdout:Mar 24 00:20:36 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-f[71102]: 11: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x55c8b2216197] 2024-03-24T00:20:36.601 INFO:journalctl@ceph.mds.f.smithi193.stdout:Mar 24 00:20:36 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-f[71102]: 12: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x55c8b211511b] 2024-03-24T00:20:36.601 INFO:journalctl@ceph.mds.f.smithi193.stdout:Mar 24 00:20:36 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-f[71102]: 13: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x55c8b21156bc] 2024-03-24T00:20:36.601 INFO:journalctl@ceph.mds.f.smithi193.stdout:Mar 24 00:20:36 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-f[71102]: 14: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x55c8b20ff1ff] 2024-03-24T00:20:36.601 INFO:journalctl@ceph.mds.f.smithi193.stdout:Mar 24 00:20:36 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-f[71102]: 15: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7fcc21b8c488] 2024-03-24T00:20:36.601 INFO:journalctl@ceph.mds.f.smithi193.stdout:Mar 24 00:20:36 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-f[71102]: 16: (DispatchQueue::entry()+0x51f) [0x7fcc21b8960f] 2024-03-24T00:20:36.601 INFO:journalctl@ceph.mds.f.smithi193.stdout:Mar 24 00:20:36 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-f[71102]: 17: (DispatchQueue::DispatchThread::entry()+0x11) [0x7fcc21c5ce81] 2024-03-24T00:20:36.601 INFO:journalctl@ceph.mds.f.smithi193.stdout:Mar 24 00:20:36 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-f[71102]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7fcc206771ca] 2024-03-24T00:20:36.601 INFO:journalctl@ceph.mds.f.smithi193.stdout:Mar 24 00:20:36 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-f[71102]: 19: clone() 2024-03-24T00:20:36.601 INFO:journalctl@ceph.mds.f.smithi193.stdout:Mar 24 00:20:36 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-f[71102]: 2024-03-24T00:20:36.547+0000 7fcc172d0700 -1 *** Caught signal (Aborted) ** 2024-03-24T00:20:36.602 INFO:journalctl@ceph.mds.f.smithi193.stdout:Mar 24 00:20:36 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-f[71102]: in thread 7fcc172d0700 thread_name:ms_dispatch 2024-03-24T00:20:36.602 INFO:journalctl@ceph.mds.f.smithi193.stdout:Mar 24 00:20:36 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-f[71102]: 2024-03-24T00:20:36.602 INFO:journalctl@ceph.mds.f.smithi193.stdout:Mar 24 00:20:36 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-f[71102]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-24T00:20:36.602 INFO:journalctl@ceph.mds.f.smithi193.stdout:Mar 24 00:20:36 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-f[71102]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7fcc20681d20] 2024-03-24T00:20:36.602 INFO:journalctl@ceph.mds.f.smithi193.stdout:Mar 24 00:20:36 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-f[71102]: 2: gsignal() 2024-03-24T00:20:36.602 INFO:journalctl@ceph.mds.f.smithi193.stdout:Mar 24 00:20:36 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-f[71102]: 3: abort() 2024-03-24T00:20:36.602 INFO:journalctl@ceph.mds.f.smithi193.stdout:Mar 24 00:20:36 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-f[71102]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x18f) [0x7fcc218fa4fb] 2024-03-24T00:20:36.602 INFO:journalctl@ceph.mds.f.smithi193.stdout:Mar 24 00:20:36 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-f[71102]: 5: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7fcc218fa667] 2024-03-24T00:20:36.603 INFO:journalctl@ceph.mds.f.smithi193.stdout:Mar 24 00:20:36 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-f[71102]: 6: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x55c8b2347223] 2024-03-24T00:20:36.603 INFO:journalctl@ceph.mds.f.smithi193.stdout:Mar 24 00:20:36 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-f[71102]: 7: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x55c8b21c15d6] 2024-03-24T00:20:36.603 INFO:journalctl@ceph.mds.f.smithi193.stdout:Mar 24 00:20:36 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-f[71102]: 8: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x55c8b21c1da1] 2024-03-24T00:20:36.603 INFO:journalctl@ceph.mds.f.smithi193.stdout:Mar 24 00:20:36 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-f[71102]: 9: (Server::handle_client_rename(boost::intrusive_ptr const&)+0x259d) [0x55c8b21fe97d] 2024-03-24T00:20:36.603 INFO:journalctl@ceph.mds.f.smithi193.stdout:Mar 24 00:20:36 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-f[71102]: 10: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x55c8b2214233] 2024-03-24T00:20:36.603 INFO:journalctl@ceph.mds.f.smithi193.stdout:Mar 24 00:20:36 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-f[71102]: 11: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x55c8b2216197] 2024-03-24T00:20:36.603 INFO:journalctl@ceph.mds.f.smithi193.stdout:Mar 24 00:20:36 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-f[71102]: 12: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x55c8b211511b] 2024-03-24T00:20:36.604 INFO:journalctl@ceph.mds.f.smithi193.stdout:Mar 24 00:20:36 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-f[71102]: 13: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x55c8b21156bc] 2024-03-24T00:20:36.604 INFO:journalctl@ceph.mds.f.smithi193.stdout:Mar 24 00:20:36 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-f[71102]: 14: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x55c8b20ff1ff] 2024-03-24T00:20:36.604 INFO:journalctl@ceph.mds.f.smithi193.stdout:Mar 24 00:20:36 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-f[71102]: 15: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7fcc21b8c488] 2024-03-24T00:20:36.604 INFO:journalctl@ceph.mds.f.smithi193.stdout:Mar 24 00:20:36 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-f[71102]: 16: (DispatchQueue::entry()+0x51f) [0x7fcc21b8960f] 2024-03-24T00:20:36.604 INFO:journalctl@ceph.mds.f.smithi193.stdout:Mar 24 00:20:36 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-f[71102]: 17: (DispatchQueue::DispatchThread::entry()+0x11) [0x7fcc21c5ce81] 2024-03-24T00:20:36.604 INFO:journalctl@ceph.mds.f.smithi193.stdout:Mar 24 00:20:36 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-f[71102]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7fcc206771ca] 2024-03-24T00:20:36.604 INFO:journalctl@ceph.mds.f.smithi193.stdout:Mar 24 00:20:36 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-f[71102]: 19: clone() 2024-03-24T00:20:36.604 INFO:journalctl@ceph.mds.f.smithi193.stdout:Mar 24 00:20:36 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-f[71102]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-03-24T00:20:36.604 INFO:journalctl@ceph.mds.f.smithi193.stdout:Mar 24 00:20:36 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-f[71102]: 2024-03-24T00:20:36.604 INFO:journalctl@ceph.mds.f.smithi193.stdout:Mar 24 00:20:36 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-f[71102]: -1> 2024-03-24T00:20:36.545+0000 7fcc172d0700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: In function 'void Locker::set_xlocks_done(MutationImpl*, bool)' thread 7fcc172d0700 time 2024-03-24T00:20:36.544253+0000 2024-03-24T00:20:36.604 INFO:journalctl@ceph.mds.f.smithi193.stdout:Mar 24 00:20:36 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-f[71102]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: 626: FAILED ceph_assert(obj->is_auth()) 2024-03-24T00:20:36.604 INFO:journalctl@ceph.mds.f.smithi193.stdout:Mar 24 00:20:36 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-f[71102]: 2024-03-24T00:20:36.604 INFO:journalctl@ceph.mds.f.smithi193.stdout:Mar 24 00:20:36 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-f[71102]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-24T00:20:36.604 INFO:journalctl@ceph.mds.f.smithi193.stdout:Mar 24 00:20:36 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-f[71102]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x135) [0x7fcc218fa4a1] 2024-03-24T00:20:36.605 INFO:journalctl@ceph.mds.f.smithi193.stdout:Mar 24 00:20:36 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-f[71102]: 2: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7fcc218fa667] 2024-03-24T00:20:36.605 INFO:journalctl@ceph.mds.f.smithi193.stdout:Mar 24 00:20:36 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-f[71102]: 3: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x55c8b2347223] 2024-03-24T00:20:36.605 INFO:journalctl@ceph.mds.f.smithi193.stdout:Mar 24 00:20:36 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-f[71102]: 4: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x55c8b21c15d6] 2024-03-24T00:20:36.605 INFO:journalctl@ceph.mds.f.smithi193.stdout:Mar 24 00:20:36 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-f[71102]: 5: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x55c8b21c1da1] 2024-03-24T00:20:36.605 INFO:journalctl@ceph.mds.f.smithi193.stdout:Mar 24 00:20:36 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-f[71102]: 6: (Server::handle_client_rename(boost::intrusive_ptr const&)+0x259d) [0x55c8b21fe97d] 2024-03-24T00:20:36.605 INFO:journalctl@ceph.mds.f.smithi193.stdout:Mar 24 00:20:36 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-f[71102]: 7: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x55c8b2214233] 2024-03-24T00:20:36.605 INFO:journalctl@ceph.mds.f.smithi193.stdout:Mar 24 00:20:36 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-f[71102]: 8: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x55c8b2216197] 2024-03-24T00:20:36.605 INFO:journalctl@ceph.mds.f.smithi193.stdout:Mar 24 00:20:36 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-f[71102]: 9: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x55c8b211511b] 2024-03-24T00:20:36.605 INFO:journalctl@ceph.mds.f.smithi193.stdout:Mar 24 00:20:36 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-f[71102]: 10: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x55c8b21156bc] 2024-03-24T00:20:36.605 INFO:journalctl@ceph.mds.f.smithi193.stdout:Mar 24 00:20:36 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-f[71102]: 11: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x55c8b20ff1ff] 2024-03-24T00:20:36.605 INFO:journalctl@ceph.mds.f.smithi193.stdout:Mar 24 00:20:36 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-f[71102]: 12: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7fcc21b8c488] 2024-03-24T00:20:36.605 INFO:journalctl@ceph.mds.f.smithi193.stdout:Mar 24 00:20:36 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-f[71102]: 13: (DispatchQueue::entry()+0x51f) [0x7fcc21b8960f] 2024-03-24T00:20:36.605 INFO:journalctl@ceph.mds.f.smithi193.stdout:Mar 24 00:20:36 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-f[71102]: 14: (DispatchQueue::DispatchThread::entry()+0x11) [0x7fcc21c5ce81] 2024-03-24T00:20:36.605 INFO:journalctl@ceph.mds.f.smithi193.stdout:Mar 24 00:20:36 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-f[71102]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7fcc206771ca] 2024-03-24T00:20:36.605 INFO:journalctl@ceph.mds.f.smithi193.stdout:Mar 24 00:20:36 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-f[71102]: 16: clone() 2024-03-24T00:20:36.606 INFO:journalctl@ceph.mds.f.smithi193.stdout:Mar 24 00:20:36 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-f[71102]: 2024-03-24T00:20:36.606 INFO:journalctl@ceph.mds.f.smithi193.stdout:Mar 24 00:20:36 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-f[71102]: 0> 2024-03-24T00:20:36.547+0000 7fcc172d0700 -1 *** Caught signal (Aborted) ** 2024-03-24T00:20:36.606 INFO:journalctl@ceph.mds.f.smithi193.stdout:Mar 24 00:20:36 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-f[71102]: in thread 7fcc172d0700 thread_name:ms_dispatch 2024-03-24T00:20:36.606 INFO:journalctl@ceph.mds.f.smithi193.stdout:Mar 24 00:20:36 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-f[71102]: 2024-03-24T00:20:36.606 INFO:journalctl@ceph.mds.f.smithi193.stdout:Mar 24 00:20:36 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-f[71102]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-24T00:20:36.606 INFO:journalctl@ceph.mds.f.smithi193.stdout:Mar 24 00:20:36 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-f[71102]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7fcc20681d20] 2024-03-24T00:20:36.606 INFO:journalctl@ceph.mds.f.smithi193.stdout:Mar 24 00:20:36 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-f[71102]: 2: gsignal() 2024-03-24T00:20:36.606 INFO:journalctl@ceph.mds.f.smithi193.stdout:Mar 24 00:20:36 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-f[71102]: 3: abort() 2024-03-24T00:20:36.606 INFO:journalctl@ceph.mds.f.smithi193.stdout:Mar 24 00:20:36 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-f[71102]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x18f) [0x7fcc218fa4fb] 2024-03-24T00:20:36.606 INFO:journalctl@ceph.mds.f.smithi193.stdout:Mar 24 00:20:36 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-f[71102]: 5: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7fcc218fa667] 2024-03-24T00:20:36.606 INFO:journalctl@ceph.mds.f.smithi193.stdout:Mar 24 00:20:36 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-f[71102]: 6: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x55c8b2347223] 2024-03-24T00:20:36.606 INFO:journalctl@ceph.mds.f.smithi193.stdout:Mar 24 00:20:36 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-f[71102]: 7: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x55c8b21c15d6] 2024-03-24T00:20:36.606 INFO:journalctl@ceph.mds.f.smithi193.stdout:Mar 24 00:20:36 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-f[71102]: 8: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x55c8b21c1da1] 2024-03-24T00:20:36.606 INFO:journalctl@ceph.mds.f.smithi193.stdout:Mar 24 00:20:36 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-f[71102]: 9: (Server::handle_client_rename(boost::intrusive_ptr const&)+0x259d) [0x55c8b21fe97d] 2024-03-24T00:20:36.607 INFO:journalctl@ceph.mds.f.smithi193.stdout:Mar 24 00:20:36 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-f[71102]: 10: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x55c8b2214233] 2024-03-24T00:20:36.607 INFO:journalctl@ceph.mds.f.smithi193.stdout:Mar 24 00:20:36 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-f[71102]: 11: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x55c8b2216197] 2024-03-24T00:20:36.607 INFO:journalctl@ceph.mds.f.smithi193.stdout:Mar 24 00:20:36 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-f[71102]: 12: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x55c8b211511b] 2024-03-24T00:20:36.607 INFO:journalctl@ceph.mds.f.smithi193.stdout:Mar 24 00:20:36 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-f[71102]: 13: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x55c8b21156bc] 2024-03-24T00:20:36.607 INFO:journalctl@ceph.mds.f.smithi193.stdout:Mar 24 00:20:36 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-f[71102]: 14: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x55c8b20ff1ff] 2024-03-24T00:20:36.607 INFO:journalctl@ceph.mds.f.smithi193.stdout:Mar 24 00:20:36 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-f[71102]: 15: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7fcc21b8c488] 2024-03-24T00:20:36.607 INFO:journalctl@ceph.mds.f.smithi193.stdout:Mar 24 00:20:36 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-f[71102]: 16: (DispatchQueue::entry()+0x51f) [0x7fcc21b8960f] 2024-03-24T00:20:36.607 INFO:journalctl@ceph.mds.f.smithi193.stdout:Mar 24 00:20:36 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-f[71102]: 17: (DispatchQueue::DispatchThread::entry()+0x11) [0x7fcc21c5ce81] 2024-03-24T00:20:36.607 INFO:journalctl@ceph.mds.f.smithi193.stdout:Mar 24 00:20:36 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-f[71102]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7fcc206771ca] 2024-03-24T00:20:36.607 INFO:journalctl@ceph.mds.f.smithi193.stdout:Mar 24 00:20:36 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-f[71102]: 19: clone() 2024-03-24T00:20:36.607 INFO:journalctl@ceph.mds.f.smithi193.stdout:Mar 24 00:20:36 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-f[71102]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-03-24T00:20:36.607 INFO:journalctl@ceph.mds.f.smithi193.stdout:Mar 24 00:20:36 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-f[71102]: 2024-03-24T00:20:36.607 INFO:journalctl@ceph.mds.f.smithi193.stdout:Mar 24 00:20:36 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-f[71102]: -1> 2024-03-24T00:20:36.545+0000 7fcc172d0700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: In function 'void Locker::set_xlocks_done(MutationImpl*, bool)' thread 7fcc172d0700 time 2024-03-24T00:20:36.544253+0000 2024-03-24T00:20:36.608 INFO:journalctl@ceph.mds.f.smithi193.stdout:Mar 24 00:20:36 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-f[71102]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: 626: FAILED ceph_assert(obj->is_auth()) 2024-03-24T00:20:36.608 INFO:journalctl@ceph.mds.f.smithi193.stdout:Mar 24 00:20:36 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-f[71102]: 2024-03-24T00:20:36.608 INFO:journalctl@ceph.mds.f.smithi193.stdout:Mar 24 00:20:36 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-f[71102]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-24T00:20:36.608 INFO:journalctl@ceph.mds.f.smithi193.stdout:Mar 24 00:20:36 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-f[71102]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x135) [0x7fcc218fa4a1] 2024-03-24T00:20:36.608 INFO:journalctl@ceph.mds.f.smithi193.stdout:Mar 24 00:20:36 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-f[71102]: 2: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7fcc218fa667] 2024-03-24T00:20:36.608 INFO:journalctl@ceph.mds.f.smithi193.stdout:Mar 24 00:20:36 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-f[71102]: 3: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x55c8b2347223] 2024-03-24T00:20:36.608 INFO:journalctl@ceph.mds.f.smithi193.stdout:Mar 24 00:20:36 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-f[71102]: 4: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x55c8b21c15d6] 2024-03-24T00:20:36.608 INFO:journalctl@ceph.mds.f.smithi193.stdout:Mar 24 00:20:36 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-f[71102]: 5: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x55c8b21c1da1] 2024-03-24T00:20:36.608 INFO:journalctl@ceph.mds.f.smithi193.stdout:Mar 24 00:20:36 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-f[71102]: 6: (Server::handle_client_rename(boost::intrusive_ptr const&)+0x259d) [0x55c8b21fe97d] 2024-03-24T00:20:36.608 INFO:journalctl@ceph.mds.f.smithi193.stdout:Mar 24 00:20:36 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-f[71102]: 7: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x55c8b2214233] 2024-03-24T00:20:36.608 INFO:journalctl@ceph.mds.f.smithi193.stdout:Mar 24 00:20:36 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-f[71102]: 8: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x55c8b2216197] 2024-03-24T00:20:36.608 INFO:journalctl@ceph.mds.f.smithi193.stdout:Mar 24 00:20:36 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-f[71102]: 9: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x55c8b211511b] 2024-03-24T00:20:36.608 INFO:journalctl@ceph.mds.f.smithi193.stdout:Mar 24 00:20:36 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-f[71102]: 10: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x55c8b21156bc] 2024-03-24T00:20:36.608 INFO:journalctl@ceph.mds.f.smithi193.stdout:Mar 24 00:20:36 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-f[71102]: 11: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x55c8b20ff1ff] 2024-03-24T00:20:36.609 INFO:journalctl@ceph.mds.f.smithi193.stdout:Mar 24 00:20:36 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-f[71102]: 12: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7fcc21b8c488] 2024-03-24T00:20:36.609 INFO:journalctl@ceph.mds.f.smithi193.stdout:Mar 24 00:20:36 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-f[71102]: 13: (DispatchQueue::entry()+0x51f) [0x7fcc21b8960f] 2024-03-24T00:20:36.609 INFO:journalctl@ceph.mds.f.smithi193.stdout:Mar 24 00:20:36 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-f[71102]: 14: (DispatchQueue::DispatchThread::entry()+0x11) [0x7fcc21c5ce81] 2024-03-24T00:20:36.609 INFO:journalctl@ceph.mds.f.smithi193.stdout:Mar 24 00:20:36 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-f[71102]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7fcc206771ca] 2024-03-24T00:20:36.609 INFO:journalctl@ceph.mds.f.smithi193.stdout:Mar 24 00:20:36 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-f[71102]: 16: clone() 2024-03-24T00:20:36.609 INFO:journalctl@ceph.mds.f.smithi193.stdout:Mar 24 00:20:36 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-f[71102]: 2024-03-24T00:20:36.609 INFO:journalctl@ceph.mds.f.smithi193.stdout:Mar 24 00:20:36 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-f[71102]: 0> 2024-03-24T00:20:36.547+0000 7fcc172d0700 -1 *** Caught signal (Aborted) ** 2024-03-24T00:20:36.609 INFO:journalctl@ceph.mds.f.smithi193.stdout:Mar 24 00:20:36 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-f[71102]: in thread 7fcc172d0700 thread_name:ms_dispatch 2024-03-24T00:20:36.609 INFO:journalctl@ceph.mds.f.smithi193.stdout:Mar 24 00:20:36 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-f[71102]: 2024-03-24T00:20:36.609 INFO:journalctl@ceph.mds.f.smithi193.stdout:Mar 24 00:20:36 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-f[71102]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-24T00:20:36.609 INFO:journalctl@ceph.mds.f.smithi193.stdout:Mar 24 00:20:36 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-f[71102]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7fcc20681d20] 2024-03-24T00:20:36.609 INFO:journalctl@ceph.mds.f.smithi193.stdout:Mar 24 00:20:36 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-f[71102]: 2: gsignal() 2024-03-24T00:20:36.609 INFO:journalctl@ceph.mds.f.smithi193.stdout:Mar 24 00:20:36 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-f[71102]: 3: abort() 2024-03-24T00:20:36.609 INFO:journalctl@ceph.mds.f.smithi193.stdout:Mar 24 00:20:36 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-f[71102]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x18f) [0x7fcc218fa4fb] 2024-03-24T00:20:36.609 INFO:journalctl@ceph.mds.f.smithi193.stdout:Mar 24 00:20:36 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-f[71102]: 5: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7fcc218fa667] 2024-03-24T00:20:36.610 INFO:journalctl@ceph.mds.f.smithi193.stdout:Mar 24 00:20:36 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-f[71102]: 6: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x55c8b2347223] 2024-03-24T00:20:36.610 INFO:journalctl@ceph.mds.f.smithi193.stdout:Mar 24 00:20:36 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-f[71102]: 7: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x55c8b21c15d6] 2024-03-24T00:20:36.610 INFO:journalctl@ceph.mds.f.smithi193.stdout:Mar 24 00:20:36 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-f[71102]: 8: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x55c8b21c1da1] 2024-03-24T00:20:36.610 INFO:journalctl@ceph.mds.f.smithi193.stdout:Mar 24 00:20:36 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-f[71102]: 9: (Server::handle_client_rename(boost::intrusive_ptr const&)+0x259d) [0x55c8b21fe97d] 2024-03-24T00:20:36.610 INFO:journalctl@ceph.mds.f.smithi193.stdout:Mar 24 00:20:36 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-f[71102]: 10: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x55c8b2214233] 2024-03-24T00:20:36.610 INFO:journalctl@ceph.mds.f.smithi193.stdout:Mar 24 00:20:36 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-f[71102]: 11: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x55c8b2216197] 2024-03-24T00:20:36.610 INFO:journalctl@ceph.mds.f.smithi193.stdout:Mar 24 00:20:36 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-f[71102]: 12: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x55c8b211511b] 2024-03-24T00:20:36.610 INFO:journalctl@ceph.mds.f.smithi193.stdout:Mar 24 00:20:36 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-f[71102]: 13: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x55c8b21156bc] 2024-03-24T00:20:36.610 INFO:journalctl@ceph.mds.f.smithi193.stdout:Mar 24 00:20:36 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-f[71102]: 14: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x55c8b20ff1ff] 2024-03-24T00:20:36.610 INFO:journalctl@ceph.mds.f.smithi193.stdout:Mar 24 00:20:36 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-f[71102]: 15: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7fcc21b8c488] 2024-03-24T00:20:36.610 INFO:journalctl@ceph.mds.f.smithi193.stdout:Mar 24 00:20:36 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-f[71102]: 16: (DispatchQueue::entry()+0x51f) [0x7fcc21b8960f] 2024-03-24T00:20:36.610 INFO:journalctl@ceph.mds.f.smithi193.stdout:Mar 24 00:20:36 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-f[71102]: 17: (DispatchQueue::DispatchThread::entry()+0x11) [0x7fcc21c5ce81] 2024-03-24T00:20:36.611 INFO:journalctl@ceph.mds.f.smithi193.stdout:Mar 24 00:20:36 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-f[71102]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7fcc206771ca] 2024-03-24T00:20:36.611 INFO:journalctl@ceph.mds.f.smithi193.stdout:Mar 24 00:20:36 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-f[71102]: 19: clone() 2024-03-24T00:20:36.611 INFO:journalctl@ceph.mds.f.smithi193.stdout:Mar 24 00:20:36 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-f[71102]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-03-24T00:20:36.611 INFO:journalctl@ceph.mds.f.smithi193.stdout:Mar 24 00:20:36 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-f[71102]: 2024-03-24T00:20:36.611 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:20:36 smithi118 ceph-mon[31535]: mds.? [v2:172.21.15.193:6834/1699655293,v1:172.21.15.193:6835/1699655293] up:reconnect 2024-03-24T00:20:36.611 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:20:36 smithi118 ceph-mon[31535]: fsmap cephfs:5/5 4 up:standby-replay 3 up:standby 1 up:reconnect 4 up:active 2024-03-24T00:20:36.611 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:20:36 smithi118 ceph-mon[31535]: reconnect by client.15159 192.168.0.1:0/3463739313 after 0.0139998 2024-03-24T00:20:36.611 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:20:36 smithi118 ceph-mon[31535]: pgmap v549: 97 pgs: 97 active+clean; 101 MiB data, 671 MiB used, 1.0 TiB / 1.0 TiB avail; 159 KiB/s rd, 171 KiB/s wr, 63 op/s 2024-03-24T00:20:36.704 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:20:36 smithi189 ceph-mon[36187]: mds.? [v2:172.21.15.193:6834/1699655293,v1:172.21.15.193:6835/1699655293] up:reconnect 2024-03-24T00:20:36.704 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:20:36 smithi189 ceph-mon[36187]: fsmap cephfs:5/5 4 up:standby-replay 3 up:standby 1 up:reconnect 4 up:active 2024-03-24T00:20:36.704 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:20:36 smithi189 ceph-mon[36187]: reconnect by client.15159 192.168.0.1:0/3463739313 after 0.0139998 2024-03-24T00:20:36.704 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:20:36 smithi189 ceph-mon[36187]: pgmap v549: 97 pgs: 97 active+clean; 101 MiB data, 671 MiB used, 1.0 TiB / 1.0 TiB avail; 159 KiB/s rd, 171 KiB/s wr, 63 op/s 2024-03-24T00:20:36.844 INFO:tasks.workunit.client.0.smithi118.stdout: 35 0 30 0 2 2513 20 2024-03-24T00:20:36.919 INFO:journalctl@ceph.mds.g.smithi118.stdout:Mar 24 00:20:36 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-g[73614]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: In function 'void Locker::set_xlocks_done(MutationImpl*, bool)' thread 7f2b33600700 time 2024-03-24T00:20:36.795598+0000 2024-03-24T00:20:36.919 INFO:journalctl@ceph.mds.g.smithi118.stdout:Mar 24 00:20:36 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-g[73614]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: 626: FAILED ceph_assert(obj->is_auth()) 2024-03-24T00:20:36.919 INFO:journalctl@ceph.mds.g.smithi118.stdout:Mar 24 00:20:36 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-g[73614]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-24T00:20:36.919 INFO:journalctl@ceph.mds.g.smithi118.stdout:Mar 24 00:20:36 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-g[73614]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x135) [0x7f2b3ee444a1] 2024-03-24T00:20:36.920 INFO:journalctl@ceph.mds.g.smithi118.stdout:Mar 24 00:20:36 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-g[73614]: 2: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7f2b3ee44667] 2024-03-24T00:20:36.920 INFO:journalctl@ceph.mds.g.smithi118.stdout:Mar 24 00:20:36 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-g[73614]: 3: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x561bcbc42223] 2024-03-24T00:20:36.920 INFO:journalctl@ceph.mds.g.smithi118.stdout:Mar 24 00:20:36 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-g[73614]: 4: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x561bcbabc5d6] 2024-03-24T00:20:36.920 INFO:journalctl@ceph.mds.g.smithi118.stdout:Mar 24 00:20:36 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-g[73614]: 5: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x561bcbabcda1] 2024-03-24T00:20:36.920 INFO:journalctl@ceph.mds.g.smithi118.stdout:Mar 24 00:20:36 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-g[73614]: 6: (Server::handle_client_rename(boost::intrusive_ptr const&)+0x259d) [0x561bcbaf997d] 2024-03-24T00:20:36.920 INFO:journalctl@ceph.mds.g.smithi118.stdout:Mar 24 00:20:36 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-g[73614]: 7: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x561bcbb0f233] 2024-03-24T00:20:36.920 INFO:journalctl@ceph.mds.g.smithi118.stdout:Mar 24 00:20:36 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-g[73614]: 8: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x561bcbb11197] 2024-03-24T00:20:36.920 INFO:journalctl@ceph.mds.g.smithi118.stdout:Mar 24 00:20:36 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-g[73614]: 9: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x561bcba1011b] 2024-03-24T00:20:36.920 INFO:journalctl@ceph.mds.g.smithi118.stdout:Mar 24 00:20:36 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-g[73614]: 10: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x561bcba106bc] 2024-03-24T00:20:36.920 INFO:journalctl@ceph.mds.g.smithi118.stdout:Mar 24 00:20:36 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-g[73614]: 11: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x561bcb9fa1ff] 2024-03-24T00:20:36.920 INFO:journalctl@ceph.mds.g.smithi118.stdout:Mar 24 00:20:36 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-g[73614]: 12: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7f2b3f0d6488] 2024-03-24T00:20:36.920 INFO:journalctl@ceph.mds.g.smithi118.stdout:Mar 24 00:20:36 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-g[73614]: 13: (DispatchQueue::entry()+0x51f) [0x7f2b3f0d360f] 2024-03-24T00:20:36.920 INFO:journalctl@ceph.mds.g.smithi118.stdout:Mar 24 00:20:36 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-g[73614]: 14: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f2b3f1a6e81] 2024-03-24T00:20:36.920 INFO:journalctl@ceph.mds.g.smithi118.stdout:Mar 24 00:20:36 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-g[73614]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7f2b3dbc11ca] 2024-03-24T00:20:36.920 INFO:journalctl@ceph.mds.g.smithi118.stdout:Mar 24 00:20:36 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-g[73614]: 16: clone() 2024-03-24T00:20:36.921 INFO:journalctl@ceph.mds.g.smithi118.stdout:Mar 24 00:20:36 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-g[73614]: 2024-03-24T00:20:36.797+0000 7f2b33600700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: In function 'void Locker::set_xlocks_done(MutationImpl*, bool)' thread 7f2b33600700 time 2024-03-24T00:20:36.795598+0000 2024-03-24T00:20:36.921 INFO:journalctl@ceph.mds.g.smithi118.stdout:Mar 24 00:20:36 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-g[73614]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: 626: FAILED ceph_assert(obj->is_auth()) 2024-03-24T00:20:36.921 INFO:journalctl@ceph.mds.g.smithi118.stdout:Mar 24 00:20:36 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-g[73614]: 2024-03-24T00:20:36.921 INFO:journalctl@ceph.mds.g.smithi118.stdout:Mar 24 00:20:36 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-g[73614]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-24T00:20:36.921 INFO:journalctl@ceph.mds.g.smithi118.stdout:Mar 24 00:20:36 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-g[73614]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x135) [0x7f2b3ee444a1] 2024-03-24T00:20:36.922 INFO:journalctl@ceph.mds.g.smithi118.stdout:Mar 24 00:20:36 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-g[73614]: 2: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7f2b3ee44667] 2024-03-24T00:20:36.922 INFO:journalctl@ceph.mds.g.smithi118.stdout:Mar 24 00:20:36 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-g[73614]: 3: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x561bcbc42223] 2024-03-24T00:20:36.922 INFO:journalctl@ceph.mds.g.smithi118.stdout:Mar 24 00:20:36 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-g[73614]: 4: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x561bcbabc5d6] 2024-03-24T00:20:36.922 INFO:journalctl@ceph.mds.g.smithi118.stdout:Mar 24 00:20:36 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-g[73614]: 5: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x561bcbabcda1] 2024-03-24T00:20:36.922 INFO:journalctl@ceph.mds.g.smithi118.stdout:Mar 24 00:20:36 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-g[73614]: 6: (Server::handle_client_rename(boost::intrusive_ptr const&)+0x259d) [0x561bcbaf997d] 2024-03-24T00:20:36.922 INFO:journalctl@ceph.mds.g.smithi118.stdout:Mar 24 00:20:36 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-g[73614]: 7: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x561bcbb0f233] 2024-03-24T00:20:36.922 INFO:journalctl@ceph.mds.g.smithi118.stdout:Mar 24 00:20:36 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-g[73614]: 8: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x561bcbb11197] 2024-03-24T00:20:36.922 INFO:journalctl@ceph.mds.g.smithi118.stdout:Mar 24 00:20:36 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-g[73614]: 9: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x561bcba1011b] 2024-03-24T00:20:36.922 INFO:journalctl@ceph.mds.g.smithi118.stdout:Mar 24 00:20:36 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-g[73614]: 10: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x561bcba106bc] 2024-03-24T00:20:36.922 INFO:journalctl@ceph.mds.g.smithi118.stdout:Mar 24 00:20:36 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-g[73614]: 11: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x561bcb9fa1ff] 2024-03-24T00:20:36.922 INFO:journalctl@ceph.mds.g.smithi118.stdout:Mar 24 00:20:36 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-g[73614]: 12: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7f2b3f0d6488] 2024-03-24T00:20:36.922 INFO:journalctl@ceph.mds.g.smithi118.stdout:Mar 24 00:20:36 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-g[73614]: 13: (DispatchQueue::entry()+0x51f) [0x7f2b3f0d360f] 2024-03-24T00:20:36.923 INFO:journalctl@ceph.mds.g.smithi118.stdout:Mar 24 00:20:36 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-g[73614]: 14: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f2b3f1a6e81] 2024-03-24T00:20:36.923 INFO:journalctl@ceph.mds.g.smithi118.stdout:Mar 24 00:20:36 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-g[73614]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7f2b3dbc11ca] 2024-03-24T00:20:36.923 INFO:journalctl@ceph.mds.g.smithi118.stdout:Mar 24 00:20:36 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-g[73614]: 16: clone() 2024-03-24T00:20:36.923 INFO:journalctl@ceph.mds.g.smithi118.stdout:Mar 24 00:20:36 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-g[73614]: 2024-03-24T00:20:36.923 INFO:journalctl@ceph.mds.g.smithi118.stdout:Mar 24 00:20:36 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-g[73614]: *** Caught signal (Aborted) ** 2024-03-24T00:20:36.923 INFO:journalctl@ceph.mds.g.smithi118.stdout:Mar 24 00:20:36 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-g[73614]: in thread 7f2b33600700 thread_name:ms_dispatch 2024-03-24T00:20:36.923 INFO:journalctl@ceph.mds.g.smithi118.stdout:Mar 24 00:20:36 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-g[73614]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-24T00:20:36.923 INFO:journalctl@ceph.mds.g.smithi118.stdout:Mar 24 00:20:36 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-g[73614]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f2b3dbcbd20] 2024-03-24T00:20:36.923 INFO:journalctl@ceph.mds.g.smithi118.stdout:Mar 24 00:20:36 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-g[73614]: 2: gsignal() 2024-03-24T00:20:36.923 INFO:journalctl@ceph.mds.g.smithi118.stdout:Mar 24 00:20:36 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-g[73614]: 3: abort() 2024-03-24T00:20:36.923 INFO:journalctl@ceph.mds.g.smithi118.stdout:Mar 24 00:20:36 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-g[73614]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x18f) [0x7f2b3ee444fb] 2024-03-24T00:20:36.923 INFO:journalctl@ceph.mds.g.smithi118.stdout:Mar 24 00:20:36 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-g[73614]: 5: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7f2b3ee44667] 2024-03-24T00:20:36.923 INFO:journalctl@ceph.mds.g.smithi118.stdout:Mar 24 00:20:36 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-g[73614]: 6: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x561bcbc42223] 2024-03-24T00:20:36.924 INFO:journalctl@ceph.mds.g.smithi118.stdout:Mar 24 00:20:36 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-g[73614]: 7: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x561bcbabc5d6] 2024-03-24T00:20:36.924 INFO:journalctl@ceph.mds.g.smithi118.stdout:Mar 24 00:20:36 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-g[73614]: 8: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x561bcbabcda1] 2024-03-24T00:20:36.924 INFO:journalctl@ceph.mds.g.smithi118.stdout:Mar 24 00:20:36 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-g[73614]: 9: (Server::handle_client_rename(boost::intrusive_ptr const&)+0x259d) [0x561bcbaf997d] 2024-03-24T00:20:36.924 INFO:journalctl@ceph.mds.g.smithi118.stdout:Mar 24 00:20:36 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-g[73614]: 10: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x561bcbb0f233] 2024-03-24T00:20:36.924 INFO:journalctl@ceph.mds.g.smithi118.stdout:Mar 24 00:20:36 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-g[73614]: 11: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x561bcbb11197] 2024-03-24T00:20:36.924 INFO:journalctl@ceph.mds.g.smithi118.stdout:Mar 24 00:20:36 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-g[73614]: 12: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x561bcba1011b] 2024-03-24T00:20:36.924 INFO:journalctl@ceph.mds.g.smithi118.stdout:Mar 24 00:20:36 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-g[73614]: 13: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x561bcba106bc] 2024-03-24T00:20:36.924 INFO:journalctl@ceph.mds.g.smithi118.stdout:Mar 24 00:20:36 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-g[73614]: 14: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x561bcb9fa1ff] 2024-03-24T00:20:36.924 INFO:journalctl@ceph.mds.g.smithi118.stdout:Mar 24 00:20:36 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-g[73614]: 15: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7f2b3f0d6488] 2024-03-24T00:20:36.924 INFO:journalctl@ceph.mds.g.smithi118.stdout:Mar 24 00:20:36 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-g[73614]: 16: (DispatchQueue::entry()+0x51f) [0x7f2b3f0d360f] 2024-03-24T00:20:36.924 INFO:journalctl@ceph.mds.g.smithi118.stdout:Mar 24 00:20:36 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-g[73614]: 17: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f2b3f1a6e81] 2024-03-24T00:20:36.924 INFO:journalctl@ceph.mds.g.smithi118.stdout:Mar 24 00:20:36 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-g[73614]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7f2b3dbc11ca] 2024-03-24T00:20:36.924 INFO:journalctl@ceph.mds.g.smithi118.stdout:Mar 24 00:20:36 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-g[73614]: 19: clone() 2024-03-24T00:20:36.924 INFO:journalctl@ceph.mds.g.smithi118.stdout:Mar 24 00:20:36 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-g[73614]: 2024-03-24T00:20:36.800+0000 7f2b33600700 -1 *** Caught signal (Aborted) ** 2024-03-24T00:20:36.925 INFO:journalctl@ceph.mds.g.smithi118.stdout:Mar 24 00:20:36 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-g[73614]: in thread 7f2b33600700 thread_name:ms_dispatch 2024-03-24T00:20:36.925 INFO:journalctl@ceph.mds.g.smithi118.stdout:Mar 24 00:20:36 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-g[73614]: 2024-03-24T00:20:36.925 INFO:journalctl@ceph.mds.g.smithi118.stdout:Mar 24 00:20:36 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-g[73614]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-24T00:20:36.925 INFO:journalctl@ceph.mds.g.smithi118.stdout:Mar 24 00:20:36 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-g[73614]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f2b3dbcbd20] 2024-03-24T00:20:36.925 INFO:journalctl@ceph.mds.g.smithi118.stdout:Mar 24 00:20:36 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-g[73614]: 2: gsignal() 2024-03-24T00:20:36.925 INFO:journalctl@ceph.mds.g.smithi118.stdout:Mar 24 00:20:36 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-g[73614]: 3: abort() 2024-03-24T00:20:36.925 INFO:journalctl@ceph.mds.g.smithi118.stdout:Mar 24 00:20:36 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-g[73614]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x18f) [0x7f2b3ee444fb] 2024-03-24T00:20:36.925 INFO:journalctl@ceph.mds.g.smithi118.stdout:Mar 24 00:20:36 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-g[73614]: 5: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7f2b3ee44667] 2024-03-24T00:20:36.925 INFO:journalctl@ceph.mds.g.smithi118.stdout:Mar 24 00:20:36 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-g[73614]: 6: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x561bcbc42223] 2024-03-24T00:20:36.925 INFO:journalctl@ceph.mds.g.smithi118.stdout:Mar 24 00:20:36 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-g[73614]: 7: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x561bcbabc5d6] 2024-03-24T00:20:36.925 INFO:journalctl@ceph.mds.g.smithi118.stdout:Mar 24 00:20:36 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-g[73614]: 8: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x561bcbabcda1] 2024-03-24T00:20:36.925 INFO:journalctl@ceph.mds.g.smithi118.stdout:Mar 24 00:20:36 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-g[73614]: 9: (Server::handle_client_rename(boost::intrusive_ptr const&)+0x259d) [0x561bcbaf997d] 2024-03-24T00:20:36.925 INFO:journalctl@ceph.mds.g.smithi118.stdout:Mar 24 00:20:36 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-g[73614]: 10: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x561bcbb0f233] 2024-03-24T00:20:36.925 INFO:journalctl@ceph.mds.g.smithi118.stdout:Mar 24 00:20:36 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-g[73614]: 11: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x561bcbb11197] 2024-03-24T00:20:36.925 INFO:journalctl@ceph.mds.g.smithi118.stdout:Mar 24 00:20:36 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-g[73614]: 12: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x561bcba1011b] 2024-03-24T00:20:36.926 INFO:journalctl@ceph.mds.g.smithi118.stdout:Mar 24 00:20:36 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-g[73614]: 13: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x561bcba106bc] 2024-03-24T00:20:36.926 INFO:journalctl@ceph.mds.g.smithi118.stdout:Mar 24 00:20:36 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-g[73614]: 14: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x561bcb9fa1ff] 2024-03-24T00:20:36.926 INFO:journalctl@ceph.mds.g.smithi118.stdout:Mar 24 00:20:36 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-g[73614]: 15: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7f2b3f0d6488] 2024-03-24T00:20:36.926 INFO:journalctl@ceph.mds.g.smithi118.stdout:Mar 24 00:20:36 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-g[73614]: 16: (DispatchQueue::entry()+0x51f) [0x7f2b3f0d360f] 2024-03-24T00:20:36.926 INFO:journalctl@ceph.mds.g.smithi118.stdout:Mar 24 00:20:36 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-g[73614]: 17: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f2b3f1a6e81] 2024-03-24T00:20:36.926 INFO:journalctl@ceph.mds.g.smithi118.stdout:Mar 24 00:20:36 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-g[73614]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7f2b3dbc11ca] 2024-03-24T00:20:36.926 INFO:journalctl@ceph.mds.g.smithi118.stdout:Mar 24 00:20:36 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-g[73614]: 19: clone() 2024-03-24T00:20:36.926 INFO:journalctl@ceph.mds.g.smithi118.stdout:Mar 24 00:20:36 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-g[73614]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-03-24T00:20:36.926 INFO:journalctl@ceph.mds.g.smithi118.stdout:Mar 24 00:20:36 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-g[73614]: 2024-03-24T00:20:36.926 INFO:journalctl@ceph.mds.g.smithi118.stdout:Mar 24 00:20:36 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-g[73614]: -1> 2024-03-24T00:20:36.797+0000 7f2b33600700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: In function 'void Locker::set_xlocks_done(MutationImpl*, bool)' thread 7f2b33600700 time 2024-03-24T00:20:36.795598+0000 2024-03-24T00:20:36.926 INFO:journalctl@ceph.mds.g.smithi118.stdout:Mar 24 00:20:36 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-g[73614]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: 626: FAILED ceph_assert(obj->is_auth()) 2024-03-24T00:20:36.926 INFO:journalctl@ceph.mds.g.smithi118.stdout:Mar 24 00:20:36 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-g[73614]: 2024-03-24T00:20:36.926 INFO:journalctl@ceph.mds.g.smithi118.stdout:Mar 24 00:20:36 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-g[73614]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-24T00:20:36.927 INFO:journalctl@ceph.mds.g.smithi118.stdout:Mar 24 00:20:36 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-g[73614]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x135) [0x7f2b3ee444a1] 2024-03-24T00:20:36.927 INFO:journalctl@ceph.mds.g.smithi118.stdout:Mar 24 00:20:36 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-g[73614]: 2: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7f2b3ee44667] 2024-03-24T00:20:36.927 INFO:journalctl@ceph.mds.g.smithi118.stdout:Mar 24 00:20:36 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-g[73614]: 3: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x561bcbc42223] 2024-03-24T00:20:36.927 INFO:journalctl@ceph.mds.g.smithi118.stdout:Mar 24 00:20:36 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-g[73614]: 4: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x561bcbabc5d6] 2024-03-24T00:20:36.928 INFO:journalctl@ceph.mds.g.smithi118.stdout:Mar 24 00:20:36 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-g[73614]: 5: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x561bcbabcda1] 2024-03-24T00:20:36.928 INFO:journalctl@ceph.mds.g.smithi118.stdout:Mar 24 00:20:36 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-g[73614]: 6: (Server::handle_client_rename(boost::intrusive_ptr const&)+0x259d) [0x561bcbaf997d] 2024-03-24T00:20:36.928 INFO:journalctl@ceph.mds.g.smithi118.stdout:Mar 24 00:20:36 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-g[73614]: 7: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x561bcbb0f233] 2024-03-24T00:20:36.928 INFO:journalctl@ceph.mds.g.smithi118.stdout:Mar 24 00:20:36 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-g[73614]: 8: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x561bcbb11197] 2024-03-24T00:20:36.928 INFO:journalctl@ceph.mds.g.smithi118.stdout:Mar 24 00:20:36 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-g[73614]: 9: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x561bcba1011b] 2024-03-24T00:20:36.928 INFO:journalctl@ceph.mds.g.smithi118.stdout:Mar 24 00:20:36 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-g[73614]: 10: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x561bcba106bc] 2024-03-24T00:20:36.928 INFO:journalctl@ceph.mds.g.smithi118.stdout:Mar 24 00:20:36 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-g[73614]: 11: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x561bcb9fa1ff] 2024-03-24T00:20:36.928 INFO:journalctl@ceph.mds.g.smithi118.stdout:Mar 24 00:20:36 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-g[73614]: 12: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7f2b3f0d6488] 2024-03-24T00:20:36.928 INFO:journalctl@ceph.mds.g.smithi118.stdout:Mar 24 00:20:36 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-g[73614]: 13: (DispatchQueue::entry()+0x51f) [0x7f2b3f0d360f] 2024-03-24T00:20:36.928 INFO:journalctl@ceph.mds.g.smithi118.stdout:Mar 24 00:20:36 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-g[73614]: 14: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f2b3f1a6e81] 2024-03-24T00:20:36.928 INFO:journalctl@ceph.mds.g.smithi118.stdout:Mar 24 00:20:36 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-g[73614]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7f2b3dbc11ca] 2024-03-24T00:20:36.928 INFO:journalctl@ceph.mds.g.smithi118.stdout:Mar 24 00:20:36 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-g[73614]: 16: clone() 2024-03-24T00:20:36.928 INFO:journalctl@ceph.mds.g.smithi118.stdout:Mar 24 00:20:36 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-g[73614]: 2024-03-24T00:20:36.928 INFO:journalctl@ceph.mds.g.smithi118.stdout:Mar 24 00:20:36 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-g[73614]: 0> 2024-03-24T00:20:36.800+0000 7f2b33600700 -1 *** Caught signal (Aborted) ** 2024-03-24T00:20:36.929 INFO:journalctl@ceph.mds.g.smithi118.stdout:Mar 24 00:20:36 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-g[73614]: in thread 7f2b33600700 thread_name:ms_dispatch 2024-03-24T00:20:36.929 INFO:journalctl@ceph.mds.g.smithi118.stdout:Mar 24 00:20:36 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-g[73614]: 2024-03-24T00:20:36.929 INFO:journalctl@ceph.mds.g.smithi118.stdout:Mar 24 00:20:36 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-g[73614]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-24T00:20:36.929 INFO:journalctl@ceph.mds.g.smithi118.stdout:Mar 24 00:20:36 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-g[73614]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f2b3dbcbd20] 2024-03-24T00:20:36.929 INFO:journalctl@ceph.mds.g.smithi118.stdout:Mar 24 00:20:36 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-g[73614]: 2: gsignal() 2024-03-24T00:20:36.929 INFO:journalctl@ceph.mds.g.smithi118.stdout:Mar 24 00:20:36 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-g[73614]: 3: abort() 2024-03-24T00:20:36.929 INFO:journalctl@ceph.mds.g.smithi118.stdout:Mar 24 00:20:36 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-g[73614]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x18f) [0x7f2b3ee444fb] 2024-03-24T00:20:36.929 INFO:journalctl@ceph.mds.g.smithi118.stdout:Mar 24 00:20:36 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-g[73614]: 5: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7f2b3ee44667] 2024-03-24T00:20:36.929 INFO:journalctl@ceph.mds.g.smithi118.stdout:Mar 24 00:20:36 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-g[73614]: 6: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x561bcbc42223] 2024-03-24T00:20:36.929 INFO:journalctl@ceph.mds.g.smithi118.stdout:Mar 24 00:20:36 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-g[73614]: 7: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x561bcbabc5d6] 2024-03-24T00:20:36.930 INFO:journalctl@ceph.mds.g.smithi118.stdout:Mar 24 00:20:36 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-g[73614]: 8: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x561bcbabcda1] 2024-03-24T00:20:36.930 INFO:journalctl@ceph.mds.g.smithi118.stdout:Mar 24 00:20:36 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-g[73614]: 9: (Server::handle_client_rename(boost::intrusive_ptr const&)+0x259d) [0x561bcbaf997d] 2024-03-24T00:20:36.930 INFO:journalctl@ceph.mds.g.smithi118.stdout:Mar 24 00:20:36 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-g[73614]: 10: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x561bcbb0f233] 2024-03-24T00:20:36.930 INFO:journalctl@ceph.mds.g.smithi118.stdout:Mar 24 00:20:36 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-g[73614]: 11: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x561bcbb11197] 2024-03-24T00:20:36.930 INFO:journalctl@ceph.mds.g.smithi118.stdout:Mar 24 00:20:36 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-g[73614]: 12: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x561bcba1011b] 2024-03-24T00:20:36.930 INFO:journalctl@ceph.mds.g.smithi118.stdout:Mar 24 00:20:36 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-g[73614]: 13: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x561bcba106bc] 2024-03-24T00:20:36.930 INFO:journalctl@ceph.mds.g.smithi118.stdout:Mar 24 00:20:36 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-g[73614]: 14: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x561bcb9fa1ff] 2024-03-24T00:20:36.930 INFO:journalctl@ceph.mds.g.smithi118.stdout:Mar 24 00:20:36 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-g[73614]: 15: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7f2b3f0d6488] 2024-03-24T00:20:36.930 INFO:journalctl@ceph.mds.g.smithi118.stdout:Mar 24 00:20:36 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-g[73614]: 16: (DispatchQueue::entry()+0x51f) [0x7f2b3f0d360f] 2024-03-24T00:20:36.930 INFO:journalctl@ceph.mds.g.smithi118.stdout:Mar 24 00:20:36 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-g[73614]: 17: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f2b3f1a6e81] 2024-03-24T00:20:36.930 INFO:journalctl@ceph.mds.g.smithi118.stdout:Mar 24 00:20:36 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-g[73614]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7f2b3dbc11ca] 2024-03-24T00:20:36.930 INFO:journalctl@ceph.mds.g.smithi118.stdout:Mar 24 00:20:36 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-g[73614]: 19: clone() 2024-03-24T00:20:36.931 INFO:journalctl@ceph.mds.g.smithi118.stdout:Mar 24 00:20:36 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-g[73614]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-03-24T00:20:36.931 INFO:journalctl@ceph.mds.g.smithi118.stdout:Mar 24 00:20:36 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-g[73614]: 2024-03-24T00:20:36.931 INFO:journalctl@ceph.mds.g.smithi118.stdout:Mar 24 00:20:36 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-g[73614]: -1> 2024-03-24T00:20:36.797+0000 7f2b33600700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: In function 'void Locker::set_xlocks_done(MutationImpl*, bool)' thread 7f2b33600700 time 2024-03-24T00:20:36.795598+0000 2024-03-24T00:20:36.931 INFO:journalctl@ceph.mds.g.smithi118.stdout:Mar 24 00:20:36 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-g[73614]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: 626: FAILED ceph_assert(obj->is_auth()) 2024-03-24T00:20:36.931 INFO:journalctl@ceph.mds.g.smithi118.stdout:Mar 24 00:20:36 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-g[73614]: 2024-03-24T00:20:36.931 INFO:journalctl@ceph.mds.g.smithi118.stdout:Mar 24 00:20:36 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-g[73614]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-24T00:20:36.931 INFO:journalctl@ceph.mds.g.smithi118.stdout:Mar 24 00:20:36 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-g[73614]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x135) [0x7f2b3ee444a1] 2024-03-24T00:20:36.931 INFO:journalctl@ceph.mds.g.smithi118.stdout:Mar 24 00:20:36 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-g[73614]: 2: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7f2b3ee44667] 2024-03-24T00:20:36.931 INFO:journalctl@ceph.mds.g.smithi118.stdout:Mar 24 00:20:36 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-g[73614]: 3: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x561bcbc42223] 2024-03-24T00:20:36.932 INFO:journalctl@ceph.mds.g.smithi118.stdout:Mar 24 00:20:36 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-g[73614]: 4: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x561bcbabc5d6] 2024-03-24T00:20:36.932 INFO:journalctl@ceph.mds.g.smithi118.stdout:Mar 24 00:20:36 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-g[73614]: 5: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x561bcbabcda1] 2024-03-24T00:20:36.932 INFO:journalctl@ceph.mds.g.smithi118.stdout:Mar 24 00:20:36 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-g[73614]: 6: (Server::handle_client_rename(boost::intrusive_ptr const&)+0x259d) [0x561bcbaf997d] 2024-03-24T00:20:36.932 INFO:journalctl@ceph.mds.g.smithi118.stdout:Mar 24 00:20:36 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-g[73614]: 7: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x561bcbb0f233] 2024-03-24T00:20:36.932 INFO:journalctl@ceph.mds.g.smithi118.stdout:Mar 24 00:20:36 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-g[73614]: 8: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x561bcbb11197] 2024-03-24T00:20:36.932 INFO:journalctl@ceph.mds.g.smithi118.stdout:Mar 24 00:20:36 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-g[73614]: 9: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x561bcba1011b] 2024-03-24T00:20:36.932 INFO:journalctl@ceph.mds.g.smithi118.stdout:Mar 24 00:20:36 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-g[73614]: 10: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x561bcba106bc] 2024-03-24T00:20:36.932 INFO:journalctl@ceph.mds.g.smithi118.stdout:Mar 24 00:20:36 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-g[73614]: 11: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x561bcb9fa1ff] 2024-03-24T00:20:36.932 INFO:journalctl@ceph.mds.g.smithi118.stdout:Mar 24 00:20:36 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-g[73614]: 12: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7f2b3f0d6488] 2024-03-24T00:20:36.932 INFO:journalctl@ceph.mds.g.smithi118.stdout:Mar 24 00:20:36 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-g[73614]: 13: (DispatchQueue::entry()+0x51f) [0x7f2b3f0d360f] 2024-03-24T00:20:36.932 INFO:journalctl@ceph.mds.g.smithi118.stdout:Mar 24 00:20:36 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-g[73614]: 14: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f2b3f1a6e81] 2024-03-24T00:20:36.932 INFO:journalctl@ceph.mds.g.smithi118.stdout:Mar 24 00:20:36 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-g[73614]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7f2b3dbc11ca] 2024-03-24T00:20:36.932 INFO:journalctl@ceph.mds.g.smithi118.stdout:Mar 24 00:20:36 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-g[73614]: 16: clone() 2024-03-24T00:20:36.932 INFO:journalctl@ceph.mds.g.smithi118.stdout:Mar 24 00:20:36 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-g[73614]: 2024-03-24T00:20:36.933 INFO:journalctl@ceph.mds.g.smithi118.stdout:Mar 24 00:20:36 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-g[73614]: 0> 2024-03-24T00:20:36.800+0000 7f2b33600700 -1 *** Caught signal (Aborted) ** 2024-03-24T00:20:36.933 INFO:journalctl@ceph.mds.g.smithi118.stdout:Mar 24 00:20:36 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-g[73614]: in thread 7f2b33600700 thread_name:ms_dispatch 2024-03-24T00:20:36.933 INFO:journalctl@ceph.mds.g.smithi118.stdout:Mar 24 00:20:36 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-g[73614]: 2024-03-24T00:20:36.933 INFO:journalctl@ceph.mds.g.smithi118.stdout:Mar 24 00:20:36 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-g[73614]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-24T00:20:36.933 INFO:journalctl@ceph.mds.g.smithi118.stdout:Mar 24 00:20:36 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-g[73614]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f2b3dbcbd20] 2024-03-24T00:20:36.933 INFO:journalctl@ceph.mds.g.smithi118.stdout:Mar 24 00:20:36 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-g[73614]: 2: gsignal() 2024-03-24T00:20:36.933 INFO:journalctl@ceph.mds.g.smithi118.stdout:Mar 24 00:20:36 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-g[73614]: 3: abort() 2024-03-24T00:20:36.933 INFO:journalctl@ceph.mds.g.smithi118.stdout:Mar 24 00:20:36 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-g[73614]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x18f) [0x7f2b3ee444fb] 2024-03-24T00:20:36.933 INFO:journalctl@ceph.mds.g.smithi118.stdout:Mar 24 00:20:36 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-g[73614]: 5: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7f2b3ee44667] 2024-03-24T00:20:36.933 INFO:journalctl@ceph.mds.g.smithi118.stdout:Mar 24 00:20:36 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-g[73614]: 6: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x561bcbc42223] 2024-03-24T00:20:36.933 INFO:journalctl@ceph.mds.g.smithi118.stdout:Mar 24 00:20:36 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-g[73614]: 7: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x561bcbabc5d6] 2024-03-24T00:20:36.934 INFO:journalctl@ceph.mds.g.smithi118.stdout:Mar 24 00:20:36 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-g[73614]: 8: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x561bcbabcda1] 2024-03-24T00:20:36.934 INFO:journalctl@ceph.mds.g.smithi118.stdout:Mar 24 00:20:36 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-g[73614]: 9: (Server::handle_client_rename(boost::intrusive_ptr const&)+0x259d) [0x561bcbaf997d] 2024-03-24T00:20:36.934 INFO:journalctl@ceph.mds.g.smithi118.stdout:Mar 24 00:20:36 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-g[73614]: 10: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x561bcbb0f233] 2024-03-24T00:20:36.934 INFO:journalctl@ceph.mds.g.smithi118.stdout:Mar 24 00:20:36 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-g[73614]: 11: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x561bcbb11197] 2024-03-24T00:20:36.934 INFO:journalctl@ceph.mds.g.smithi118.stdout:Mar 24 00:20:36 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-g[73614]: 12: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x561bcba1011b] 2024-03-24T00:20:36.934 INFO:journalctl@ceph.mds.g.smithi118.stdout:Mar 24 00:20:36 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-g[73614]: 13: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x561bcba106bc] 2024-03-24T00:20:36.934 INFO:journalctl@ceph.mds.g.smithi118.stdout:Mar 24 00:20:36 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-g[73614]: 14: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x561bcb9fa1ff] 2024-03-24T00:20:36.934 INFO:journalctl@ceph.mds.g.smithi118.stdout:Mar 24 00:20:36 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-g[73614]: 15: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7f2b3f0d6488] 2024-03-24T00:20:36.934 INFO:journalctl@ceph.mds.g.smithi118.stdout:Mar 24 00:20:36 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-g[73614]: 16: (DispatchQueue::entry()+0x51f) [0x7f2b3f0d360f] 2024-03-24T00:20:36.934 INFO:journalctl@ceph.mds.g.smithi118.stdout:Mar 24 00:20:36 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-g[73614]: 17: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f2b3f1a6e81] 2024-03-24T00:20:36.934 INFO:journalctl@ceph.mds.g.smithi118.stdout:Mar 24 00:20:36 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-g[73614]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7f2b3dbc11ca] 2024-03-24T00:20:36.934 INFO:journalctl@ceph.mds.g.smithi118.stdout:Mar 24 00:20:36 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-g[73614]: 19: clone() 2024-03-24T00:20:36.934 INFO:journalctl@ceph.mds.g.smithi118.stdout:Mar 24 00:20:36 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-g[73614]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-03-24T00:20:36.934 INFO:journalctl@ceph.mds.g.smithi118.stdout:Mar 24 00:20:36 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-g[73614]: 2024-03-24T00:20:36.935 INFO:journalctl@ceph.mds.g.smithi118.stdout:Mar 24 00:20:36 smithi118 podman[93788]: 2024-03-24 00:20:36.919202453 +0000 UTC m=+0.037948199 container died 2943a8e74123a676960a943dae30cd5c28d14b4754589981df7f4d82ed679c74 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-g, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, GIT_BRANCH=HEAD, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, ceph=True, org.label-schema.build-date=20240319, CEPH_POINT_RELEASE=) 2024-03-24T00:20:36.952 INFO:journalctl@ceph.mds.f.smithi193.stdout:Mar 24 00:20:36 smithi193 podman[71297]: 2024-03-24 00:20:36.62783854 +0000 UTC m=+0.019413935 container died 4b67230e0d857dd5235101666cd773f385288cea0ef4623a9d3c3deca41e9070 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-f, CEPH_POINT_RELEASE=, ceph=True, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240319, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_CLEAN=True, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD) 2024-03-24T00:20:37.488 INFO:journalctl@ceph.mds.f.smithi193.stdout:Mar 24 00:20:37 smithi193 podman[71297]: 2024-03-24 00:20:37.228876262 +0000 UTC m=+0.620451653 container remove 4b67230e0d857dd5235101666cd773f385288cea0ef4623a9d3c3deca41e9070 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-f, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, org.label-schema.build-date=20240319, org.label-schema.vendor=CentOS, GIT_CLEAN=True, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.schema-version=1.0, ceph=True, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux ) 2024-03-24T00:20:37.488 INFO:journalctl@ceph.mds.f.smithi193.stdout:Mar 24 00:20:37 smithi193 systemd[1]: ceph-d79a5420-e971-11ee-95cd-87774f69a715@mds.f.service: Main process exited, code=exited, status=134/n/a 2024-03-24T00:20:37.488 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:20:37 smithi193 ceph-mon[36878]: mds.? [v2:172.21.15.193:6834/1699655293,v1:172.21.15.193:6835/1699655293] up:rejoin 2024-03-24T00:20:37.489 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:20:37 smithi193 ceph-mon[36878]: fsmap cephfs:5/5 4 up:standby-replay 3 up:standby 1 up:rejoin 4 up:active 2024-03-24T00:20:37.489 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:20:37 smithi193 ceph-mon[36878]: daemon mds.f is now active in filesystem cephfs as rank 3 2024-03-24T00:20:37.489 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:20:37 smithi193 ceph-mon[36878]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-03-24T00:20:37.489 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:20:37 smithi193 ceph-mon[36878]: Cluster is now healthy 2024-03-24T00:20:37.489 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:20:37 smithi193 ceph-mon[36878]: mds.? [v2:172.21.15.193:6834/1699655293,v1:172.21.15.193:6835/1699655293] up:active 2024-03-24T00:20:37.489 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:20:37 smithi193 ceph-mon[36878]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-03-24T00:20:37.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:20:37 smithi118 ceph-mon[31535]: mds.? [v2:172.21.15.193:6834/1699655293,v1:172.21.15.193:6835/1699655293] up:rejoin 2024-03-24T00:20:37.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:20:37 smithi118 ceph-mon[31535]: fsmap cephfs:5/5 4 up:standby-replay 3 up:standby 1 up:rejoin 4 up:active 2024-03-24T00:20:37.697 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:20:37 smithi118 ceph-mon[31535]: daemon mds.f is now active in filesystem cephfs as rank 3 2024-03-24T00:20:37.697 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:20:37 smithi118 ceph-mon[31535]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-03-24T00:20:37.697 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:20:37 smithi118 ceph-mon[31535]: Cluster is now healthy 2024-03-24T00:20:37.697 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:20:37 smithi118 ceph-mon[31535]: mds.? [v2:172.21.15.193:6834/1699655293,v1:172.21.15.193:6835/1699655293] up:active 2024-03-24T00:20:37.697 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:20:37 smithi118 ceph-mon[31535]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-03-24T00:20:37.704 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:20:37 smithi189 ceph-mon[36187]: mds.? [v2:172.21.15.193:6834/1699655293,v1:172.21.15.193:6835/1699655293] up:rejoin 2024-03-24T00:20:37.704 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:20:37 smithi189 ceph-mon[36187]: fsmap cephfs:5/5 4 up:standby-replay 3 up:standby 1 up:rejoin 4 up:active 2024-03-24T00:20:37.704 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:20:37 smithi189 ceph-mon[36187]: daemon mds.f is now active in filesystem cephfs as rank 3 2024-03-24T00:20:37.704 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:20:37 smithi189 ceph-mon[36187]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-03-24T00:20:37.704 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:20:37 smithi189 ceph-mon[36187]: Cluster is now healthy 2024-03-24T00:20:37.704 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:20:37 smithi189 ceph-mon[36187]: mds.? [v2:172.21.15.193:6834/1699655293,v1:172.21.15.193:6835/1699655293] up:active 2024-03-24T00:20:37.704 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:20:37 smithi189 ceph-mon[36187]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-03-24T00:20:37.960 INFO:journalctl@ceph.mds.f.smithi193.stdout:Mar 24 00:20:37 smithi193 systemd[1]: ceph-d79a5420-e971-11ee-95cd-87774f69a715@mds.f.service: Failed with result 'exit-code'. 2024-03-24T00:20:38.196 INFO:journalctl@ceph.mds.g.smithi118.stdout:Mar 24 00:20:37 smithi118 podman[93788]: 2024-03-24 00:20:37.816608585 +0000 UTC m=+0.935354390 container remove 2943a8e74123a676960a943dae30cd5c28d14b4754589981df7f4d82ed679c74 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-g, GIT_CLEAN=True, org.label-schema.build-date=20240319, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, ceph=True) 2024-03-24T00:20:38.196 INFO:journalctl@ceph.mds.g.smithi118.stdout:Mar 24 00:20:37 smithi118 systemd[1]: ceph-d79a5420-e971-11ee-95cd-87774f69a715@mds.g.service: Main process exited, code=exited, status=134/n/a 2024-03-24T00:20:38.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:20:38 smithi118 ceph-mon[31535]: pgmap v550: 97 pgs: 97 active+clean; 104 MiB data, 678 MiB used, 1.0 TiB / 1.0 TiB avail; 40 KiB/s rd, 715 KiB/s wr, 101 op/s 2024-03-24T00:20:38.696 INFO:journalctl@ceph.mds.g.smithi118.stdout:Mar 24 00:20:38 smithi118 systemd[1]: ceph-d79a5420-e971-11ee-95cd-87774f69a715@mds.g.service: Failed with result 'exit-code'. 2024-03-24T00:20:38.697 INFO:journalctl@ceph.mds.g.smithi118.stdout:Mar 24 00:20:38 smithi118 systemd[1]: ceph-d79a5420-e971-11ee-95cd-87774f69a715@mds.g.service: Consumed 4min 56.335s CPU time. 2024-03-24T00:20:38.703 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:20:38 smithi189 ceph-mon[36187]: pgmap v550: 97 pgs: 97 active+clean; 104 MiB data, 678 MiB used, 1.0 TiB / 1.0 TiB avail; 40 KiB/s rd, 715 KiB/s wr, 101 op/s 2024-03-24T00:20:38.710 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:20:38 smithi193 ceph-mon[36878]: pgmap v550: 97 pgs: 97 active+clean; 104 MiB data, 678 MiB used, 1.0 TiB / 1.0 TiB avail; 40 KiB/s rd, 715 KiB/s wr, 101 op/s 2024-03-24T00:20:40.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:20:39 smithi118 ceph-mon[31535]: pgmap v551: 97 pgs: 97 active+clean; 109 MiB data, 680 MiB used, 1.0 TiB / 1.0 TiB avail; 63 KiB/s rd, 1.2 MiB/s wr, 153 op/s 2024-03-24T00:20:40.203 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:20:39 smithi189 ceph-mon[36187]: pgmap v551: 97 pgs: 97 active+clean; 109 MiB data, 680 MiB used, 1.0 TiB / 1.0 TiB avail; 63 KiB/s rd, 1.2 MiB/s wr, 153 op/s 2024-03-24T00:20:40.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:20:39 smithi193 ceph-mon[36878]: pgmap v551: 97 pgs: 97 active+clean; 109 MiB data, 680 MiB used, 1.0 TiB / 1.0 TiB avail; 63 KiB/s rd, 1.2 MiB/s wr, 153 op/s 2024-03-24T00:20:42.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:20:41 smithi118 ceph-mon[31535]: pgmap v552: 97 pgs: 97 active+clean; 112 MiB data, 682 MiB used, 1.0 TiB / 1.0 TiB avail; 81 KiB/s rd, 1.5 MiB/s wr, 170 op/s 2024-03-24T00:20:42.203 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:20:41 smithi189 ceph-mon[36187]: pgmap v552: 97 pgs: 97 active+clean; 112 MiB data, 682 MiB used, 1.0 TiB / 1.0 TiB avail; 81 KiB/s rd, 1.5 MiB/s wr, 170 op/s 2024-03-24T00:20:42.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:20:41 smithi193 ceph-mon[36878]: pgmap v552: 97 pgs: 97 active+clean; 112 MiB data, 682 MiB used, 1.0 TiB / 1.0 TiB avail; 81 KiB/s rd, 1.5 MiB/s wr, 170 op/s 2024-03-24T00:20:44.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:20:43 smithi118 ceph-mon[31535]: pgmap v553: 97 pgs: 97 active+clean; 113 MiB data, 683 MiB used, 1.0 TiB / 1.0 TiB avail; 262 KiB/s rd, 1.5 MiB/s wr, 172 op/s 2024-03-24T00:20:44.203 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:20:43 smithi189 ceph-mon[36187]: pgmap v553: 97 pgs: 97 active+clean; 113 MiB data, 683 MiB used, 1.0 TiB / 1.0 TiB avail; 262 KiB/s rd, 1.5 MiB/s wr, 172 op/s 2024-03-24T00:20:44.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:20:43 smithi193 ceph-mon[36878]: pgmap v553: 97 pgs: 97 active+clean; 113 MiB data, 683 MiB used, 1.0 TiB / 1.0 TiB avail; 262 KiB/s rd, 1.5 MiB/s wr, 172 op/s 2024-03-24T00:20:46.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:20:45 smithi118 ceph-mon[31535]: pgmap v554: 97 pgs: 97 active+clean; 113 MiB data, 683 MiB used, 1.0 TiB / 1.0 TiB avail; 231 KiB/s rd, 1.3 MiB/s wr, 153 op/s 2024-03-24T00:20:46.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:20:45 smithi189 ceph-mon[36187]: pgmap v554: 97 pgs: 97 active+clean; 113 MiB data, 683 MiB used, 1.0 TiB / 1.0 TiB avail; 231 KiB/s rd, 1.3 MiB/s wr, 153 op/s 2024-03-24T00:20:46.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:20:45 smithi193 ceph-mon[36878]: pgmap v554: 97 pgs: 97 active+clean; 113 MiB data, 683 MiB used, 1.0 TiB / 1.0 TiB avail; 231 KiB/s rd, 1.3 MiB/s wr, 153 op/s 2024-03-24T00:20:46.844 INFO:tasks.workunit.client.0.smithi118.stdout: 37 0 33 0 79 0 5 2024-03-24T00:20:47.049 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:20:47.050+0000 7fd627e00640 1 --2- 172.21.15.118:0/736842230 >> [v2:172.21.15.193:6832/1023428412,v1:172.21.15.193:6833/1023428412] conn(0x55c55390a6a0 0x55c55390cac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:20:47.050 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:20:47.050+0000 7fd627e00640 1 -- 172.21.15.118:0/736842230 >> [v2:172.21.15.193:6832/1023428412,v1:172.21.15.193:6833/1023428412] conn(0x55c55390a6a0 msgr2=0x55c55390cac0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-03-24T00:20:47.050 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:20:47.051+0000 7fd627e00640 1 -- 172.21.15.118:0/736842230 >> [v2:172.21.15.193:6832/1023428412,v1:172.21.15.193:6833/1023428412] conn(0x55c55390a6a0 msgr2=0x55c55390cac0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-03-24T00:20:47.050 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:20:47.051+0000 7fd627e00640 1 --2- 172.21.15.118:0/736842230 >> [v2:172.21.15.193:6832/1023428412,v1:172.21.15.193:6833/1023428412] conn(0x55c55390a6a0 0x55c55390cac0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-03-24T00:20:47.051 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:20:47.051+0000 7fd627e00640 1 --2- 172.21.15.118:0/736842230 >> [v2:172.21.15.193:6832/1023428412,v1:172.21.15.193:6833/1023428412] conn(0x55c55390a6a0 0x55c55390cac0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-03-24T00:20:47.970 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:20:47 smithi193 ceph-mon[36878]: pgmap v555: 97 pgs: 97 active+clean; 113 MiB data, 683 MiB used, 1.0 TiB / 1.0 TiB avail; 235 KiB/s rd, 1.3 MiB/s wr, 142 op/s 2024-03-24T00:20:47.970 INFO:journalctl@ceph.mds.f.smithi193.stdout:Mar 24 00:20:47 smithi193 systemd[1]: ceph-d79a5420-e971-11ee-95cd-87774f69a715@mds.f.service: Scheduled restart job, restart counter is at 3. 2024-03-24T00:20:47.970 INFO:journalctl@ceph.mds.f.smithi193.stdout:Mar 24 00:20:47 smithi193 systemd[1]: Stopped Ceph mds.f for d79a5420-e971-11ee-95cd-87774f69a715. 2024-03-24T00:20:47.970 INFO:journalctl@ceph.mds.f.smithi193.stdout:Mar 24 00:20:47 smithi193 systemd[1]: Starting Ceph mds.f for d79a5420-e971-11ee-95cd-87774f69a715... 2024-03-24T00:20:47.970 INFO:journalctl@ceph.mds.f.smithi193.stdout:Mar 24 00:20:47 smithi193 podman[71393]: 2024-03-24 00:20:47.849327602 +0000 UTC m=+0.023380908 image pull c664afe56bb01db4f9afa9532c20598e201e664f2e9ba2be50db0d52674a26d2 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc 2024-03-24T00:20:48.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:20:47 smithi118 ceph-mon[31535]: pgmap v555: 97 pgs: 97 active+clean; 113 MiB data, 683 MiB used, 1.0 TiB / 1.0 TiB avail; 235 KiB/s rd, 1.3 MiB/s wr, 142 op/s 2024-03-24T00:20:48.203 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:20:47 smithi189 ceph-mon[36187]: pgmap v555: 97 pgs: 97 active+clean; 113 MiB data, 683 MiB used, 1.0 TiB / 1.0 TiB avail; 235 KiB/s rd, 1.3 MiB/s wr, 142 op/s 2024-03-24T00:20:48.223 INFO:journalctl@ceph.mds.f.smithi193.stdout:Mar 24 00:20:47 smithi193 podman[71393]: 2024-03-24 00:20:47.971072696 +0000 UTC m=+0.145125988 container create 2357719800f568c5c2d6078a6306ce0671cc51e7d25ad7d5cf0380eb00130b6c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-f, maintainer=Guillaume Abrioux , RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240319, ceph=True, GIT_BRANCH=HEAD) 2024-03-24T00:20:48.711 INFO:journalctl@ceph.mds.f.smithi193.stdout:Mar 24 00:20:48 smithi193 podman[71393]: 2024-03-24 00:20:48.224041289 +0000 UTC m=+0.398094585 container init 2357719800f568c5c2d6078a6306ce0671cc51e7d25ad7d5cf0380eb00130b6c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-f, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240319, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, ceph=True, org.label-schema.schema-version=1.0, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux ) 2024-03-24T00:20:48.711 INFO:journalctl@ceph.mds.f.smithi193.stdout:Mar 24 00:20:48 smithi193 podman[71393]: 2024-03-24 00:20:48.227748778 +0000 UTC m=+0.401802063 container start 2357719800f568c5c2d6078a6306ce0671cc51e7d25ad7d5cf0380eb00130b6c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-f, GIT_CLEAN=True, org.label-schema.build-date=20240319, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0) 2024-03-24T00:20:48.711 INFO:journalctl@ceph.mds.f.smithi193.stdout:Mar 24 00:20:48 smithi193 ceph-mds[71413]: -- 172.21.15.193:0/2776929303 <== mon.1 v2:172.21.15.193:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x55be814cfa40 con 0x55be814a3000 2024-03-24T00:20:48.711 INFO:journalctl@ceph.mds.f.smithi193.stdout:Mar 24 00:20:48 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-f[71409]: starting mds.f at 2024-03-24T00:20:48.711 INFO:journalctl@ceph.mds.f.smithi193.stdout:Mar 24 00:20:48 smithi193 bash[71393]: 2357719800f568c5c2d6078a6306ce0671cc51e7d25ad7d5cf0380eb00130b6c 2024-03-24T00:20:48.711 INFO:journalctl@ceph.mds.f.smithi193.stdout:Mar 24 00:20:48 smithi193 systemd[1]: Started Ceph mds.f for d79a5420-e971-11ee-95cd-87774f69a715. 2024-03-24T00:20:48.823 INFO:journalctl@ceph.mds.g.smithi118.stdout:Mar 24 00:20:48 smithi118 systemd[1]: ceph-d79a5420-e971-11ee-95cd-87774f69a715@mds.g.service: Scheduled restart job, restart counter is at 1. 2024-03-24T00:20:48.824 INFO:journalctl@ceph.mds.g.smithi118.stdout:Mar 24 00:20:48 smithi118 systemd[1]: Stopped Ceph mds.g for d79a5420-e971-11ee-95cd-87774f69a715. 2024-03-24T00:20:48.824 INFO:journalctl@ceph.mds.g.smithi118.stdout:Mar 24 00:20:48 smithi118 systemd[1]: ceph-d79a5420-e971-11ee-95cd-87774f69a715@mds.g.service: Consumed 4min 56.335s CPU time. 2024-03-24T00:20:48.824 INFO:journalctl@ceph.mds.g.smithi118.stdout:Mar 24 00:20:48 smithi118 systemd[1]: Starting Ceph mds.g for d79a5420-e971-11ee-95cd-87774f69a715... 2024-03-24T00:20:49.188 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:20:48 smithi118 ceph-mon[31535]: daemon mds.f restarted 2024-03-24T00:20:49.188 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:20:48 smithi118 ceph-mon[31535]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-03-24T00:20:49.188 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:20:48 smithi118 ceph-mon[31535]: osdmap e95: 12 total, 12 up, 12 in 2024-03-24T00:20:49.188 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:20:48 smithi118 ceph-mon[31535]: mds.? [v2:172.21.15.193:6834/799001823,v1:172.21.15.193:6835/799001823] up:boot 2024-03-24T00:20:49.188 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:20:48 smithi118 ceph-mon[31535]: Standby daemon mds.l assigned to filesystem cephfs as rank 3 2024-03-24T00:20:49.188 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:20:48 smithi118 ceph-mon[31535]: fsmap cephfs:4/5 {0=b=up:active,1=g=up:active,2=c=up:active,4=e=up:active} 5 up:standby-replay 3 up:standby, 1 failed 2024-03-24T00:20:49.188 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:20:48 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-03-24T00:20:49.188 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:20:48 smithi118 ceph-mon[31535]: fsmap cephfs:5/5 4 up:standby-replay 3 up:standby 1 up:replay 4 up:active 2024-03-24T00:20:49.189 INFO:journalctl@ceph.mds.g.smithi118.stdout:Mar 24 00:20:48 smithi118 podman[93881]: 2024-03-24 00:20:48.729476839 +0000 UTC m=+0.033124155 image pull c664afe56bb01db4f9afa9532c20598e201e664f2e9ba2be50db0d52674a26d2 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc 2024-03-24T00:20:49.189 INFO:journalctl@ceph.mds.g.smithi118.stdout:Mar 24 00:20:48 smithi118 podman[93881]: 2024-03-24 00:20:48.867271812 +0000 UTC m=+0.170919164 container create b08cc13c723022d35cf73076cb46d7bd86645c672c376533811688dc205c3f7e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-g, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.build-date=20240319, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.schema-version=1.0, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2) 2024-03-24T00:20:49.189 INFO:journalctl@ceph.mds.g.smithi118.stdout:Mar 24 00:20:49 smithi118 podman[93881]: 2024-03-24 00:20:49.187961677 +0000 UTC m=+0.491609036 container init b08cc13c723022d35cf73076cb46d7bd86645c672c376533811688dc205c3f7e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-g, org.label-schema.build-date=20240319, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, ceph=True, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True) 2024-03-24T00:20:49.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:20:48 smithi189 ceph-mon[36187]: daemon mds.f restarted 2024-03-24T00:20:49.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:20:48 smithi189 ceph-mon[36187]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-03-24T00:20:49.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:20:48 smithi189 ceph-mon[36187]: osdmap e95: 12 total, 12 up, 12 in 2024-03-24T00:20:49.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:20:48 smithi189 ceph-mon[36187]: mds.? [v2:172.21.15.193:6834/799001823,v1:172.21.15.193:6835/799001823] up:boot 2024-03-24T00:20:49.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:20:48 smithi189 ceph-mon[36187]: Standby daemon mds.l assigned to filesystem cephfs as rank 3 2024-03-24T00:20:49.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:20:48 smithi189 ceph-mon[36187]: fsmap cephfs:4/5 {0=b=up:active,1=g=up:active,2=c=up:active,4=e=up:active} 5 up:standby-replay 3 up:standby, 1 failed 2024-03-24T00:20:49.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:20:48 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-03-24T00:20:49.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:20:48 smithi189 ceph-mon[36187]: fsmap cephfs:5/5 4 up:standby-replay 3 up:standby 1 up:replay 4 up:active 2024-03-24T00:20:49.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:20:48 smithi193 ceph-mon[36878]: daemon mds.f restarted 2024-03-24T00:20:49.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:20:48 smithi193 ceph-mon[36878]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-03-24T00:20:49.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:20:48 smithi193 ceph-mon[36878]: osdmap e95: 12 total, 12 up, 12 in 2024-03-24T00:20:49.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:20:48 smithi193 ceph-mon[36878]: mds.? [v2:172.21.15.193:6834/799001823,v1:172.21.15.193:6835/799001823] up:boot 2024-03-24T00:20:49.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:20:48 smithi193 ceph-mon[36878]: Standby daemon mds.l assigned to filesystem cephfs as rank 3 2024-03-24T00:20:49.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:20:48 smithi193 ceph-mon[36878]: fsmap cephfs:4/5 {0=b=up:active,1=g=up:active,2=c=up:active,4=e=up:active} 5 up:standby-replay 3 up:standby, 1 failed 2024-03-24T00:20:49.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:20:48 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-03-24T00:20:49.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:20:48 smithi193 ceph-mon[36878]: fsmap cephfs:5/5 4 up:standby-replay 3 up:standby 1 up:replay 4 up:active 2024-03-24T00:20:49.446 INFO:journalctl@ceph.mds.g.smithi118.stdout:Mar 24 00:20:49 smithi118 podman[93881]: 2024-03-24 00:20:49.197035023 +0000 UTC m=+0.500682345 container start b08cc13c723022d35cf73076cb46d7bd86645c672c376533811688dc205c3f7e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-g, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, GIT_CLEAN=True, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240319, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, ceph=True) 2024-03-24T00:20:49.446 INFO:journalctl@ceph.mds.g.smithi118.stdout:Mar 24 00:20:49 smithi118 ceph-mds[93900]: -- 172.21.15.118:0/2267212596 <== mon.1 v2:172.21.15.193:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x558bf6c65a40 con 0x558bf6c38800 2024-03-24T00:20:49.447 INFO:journalctl@ceph.mds.g.smithi118.stdout:Mar 24 00:20:49 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-g[93896]: starting mds.g at 2024-03-24T00:20:49.447 INFO:journalctl@ceph.mds.g.smithi118.stdout:Mar 24 00:20:49 smithi118 bash[93881]: b08cc13c723022d35cf73076cb46d7bd86645c672c376533811688dc205c3f7e 2024-03-24T00:20:49.447 INFO:journalctl@ceph.mds.g.smithi118.stdout:Mar 24 00:20:49 smithi118 systemd[1]: Started Ceph mds.g for d79a5420-e971-11ee-95cd-87774f69a715. 2024-03-24T00:20:49.515 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:20:49.515+0000 7fa69a000640 1 -- 172.21.15.118:0/2126226988 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa694102710 msgr2=0x7fa694102b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:20:49.516 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:20:49.515+0000 7fa69a000640 1 --2- 172.21.15.118:0/2126226988 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa694102710 0x7fa694102b90 secure :-1 s=READY pgs=409 cs=0 l=1 rev1=1 crypto rx=0x7fa680009900 tx=0x7fa68002bc40 comp rx=0 tx=0).stop 2024-03-24T00:20:49.516 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:20:49.515+0000 7fa69a000640 1 -- 172.21.15.118:0/2126226988 shutdown_connections 2024-03-24T00:20:49.516 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:20:49.515+0000 7fa69a000640 1 --2- 172.21.15.118:0/2126226988 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fa6941030d0 0x7fa694109960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:20:49.516 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:20:49.515+0000 7fa69a000640 1 --2- 172.21.15.118:0/2126226988 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa694102710 0x7fa694102b90 unknown :-1 s=CLOSED pgs=409 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:20:49.516 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:20:49.515+0000 7fa69a000640 1 --2- 172.21.15.118:0/2126226988 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fa694101520 0x7fa694101940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:20:49.516 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:20:49.515+0000 7fa69a000640 1 -- 172.21.15.118:0/2126226988 >> 172.21.15.118:0/2126226988 conn(0x7fa6940fcab0 msgr2=0x7fa6940feed0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:20:49.516 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:20:49.516+0000 7fa69a000640 1 -- 172.21.15.118:0/2126226988 shutdown_connections 2024-03-24T00:20:49.516 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:20:49.516+0000 7fa69a000640 1 -- 172.21.15.118:0/2126226988 wait complete. 2024-03-24T00:20:49.518 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:20:49.517+0000 7fa69a000640 1 Processor -- start 2024-03-24T00:20:49.519 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:20:49.517+0000 7fa69a000640 1 -- start start 2024-03-24T00:20:49.519 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:20:49.517+0000 7fa69a000640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fa694101520 0x7fa694075450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:20:49.519 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:20:49.517+0000 7fa69a000640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fa694102710 0x7fa694073a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:20:49.519 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:20:49.517+0000 7fa69a000640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa6941030d0 0x7fa694073fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:20:49.519 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:20:49.517+0000 7fa69a000640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa69410d200 con 0x7fa694102710 2024-03-24T00:20:49.519 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:20:49.517+0000 7fa69a000640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa69410d080 con 0x7fa6941030d0 2024-03-24T00:20:49.519 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:20:49.517+0000 7fa69a000640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa69410d380 con 0x7fa694101520 2024-03-24T00:20:49.519 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:20:49.517+0000 7fa698c00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fa694101520 0x7fa694075450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:20:49.519 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:20:49.517+0000 7fa698c00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fa694101520 0x7fa694075450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.118:57756/0 (socket says 172.21.15.118:57756) 2024-03-24T00:20:49.519 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:20:49.517+0000 7fa698c00640 1 -- 172.21.15.118:0/1753000363 learned_addr learned my addr 172.21.15.118:0/1753000363 (peer_addr_for_me v2:172.21.15.118:0/0) 2024-03-24T00:20:49.519 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:20:49.517+0000 7fa699600640 1 --2- 172.21.15.118:0/1753000363 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa6941030d0 0x7fa694073fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:20:49.519 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:20:49.517+0000 7fa693e00640 1 --2- 172.21.15.118:0/1753000363 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fa694102710 0x7fa694073a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:20:49.519 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:20:49.518+0000 7fa699600640 1 -- 172.21.15.118:0/1753000363 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fa694101520 msgr2=0x7fa694075450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:20:49.519 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:20:49.518+0000 7fa699600640 1 --2- 172.21.15.118:0/1753000363 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fa694101520 0x7fa694075450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:20:49.520 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:20:49.518+0000 7fa699600640 1 -- 172.21.15.118:0/1753000363 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fa694102710 msgr2=0x7fa694073a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:20:49.520 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:20:49.518+0000 7fa699600640 1 --2- 172.21.15.118:0/1753000363 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fa694102710 0x7fa694073a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:20:49.520 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:20:49.518+0000 7fa699600640 1 -- 172.21.15.118:0/1753000363 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa694074860 con 0x7fa6941030d0 2024-03-24T00:20:49.520 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:20:49.518+0000 7fa698c00640 1 --2- 172.21.15.118:0/1753000363 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fa694101520 0x7fa694075450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T00:20:49.520 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:20:49.518+0000 7fa699600640 1 --2- 172.21.15.118:0/1753000363 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa6941030d0 0x7fa694073fd0 secure :-1 s=READY pgs=410 cs=0 l=1 rev1=1 crypto rx=0x7fa68c009fd0 tx=0x7fa68c00ef10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:20:49.520 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:20:49.518+0000 7fa691600640 1 -- 172.21.15.118:0/1753000363 <== mon.2 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa68c00ce20 con 0x7fa6941030d0 2024-03-24T00:20:49.520 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:20:49.518+0000 7fa691600640 1 -- 172.21.15.118:0/1753000363 <== mon.2 v2:172.21.15.189:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fa68c005020 con 0x7fa6941030d0 2024-03-24T00:20:49.520 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:20:49.518+0000 7fa691600640 1 -- 172.21.15.118:0/1753000363 <== mon.2 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa68c010720 con 0x7fa6941030d0 2024-03-24T00:20:49.520 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:20:49.518+0000 7fa69a000640 1 -- 172.21.15.118:0/1753000363 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa6941a18f0 con 0x7fa6941030d0 2024-03-24T00:20:49.520 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:20:49.519+0000 7fa69a000640 1 -- 172.21.15.118:0/1753000363 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa6941a1dc0 con 0x7fa6941030d0 2024-03-24T00:20:49.522 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:20:49.523+0000 7fa69e224740 1 Processor -- start 2024-03-24T00:20:49.523 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:20:49.523+0000 7fa691600640 1 -- 172.21.15.118:0/1753000363 <== mon.2 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 20) v1 ==== 137291+0+0 (secure 0 0 0) 0x7fa68c005540 con 0x7fa6941030d0 2024-03-24T00:20:49.523 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:20:49.523+0000 7fa69e224740 1 -- start start 2024-03-24T00:20:49.524 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:20:49.523+0000 7fa69e224740 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55c112f233f0 0x55c112f23810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:20:49.524 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:20:49.523+0000 7fa69e224740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55c112f23de0 0x55c112f288e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:20:49.525 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:20:49.524+0000 7fa69e224740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55c112f28f50 0x55c112f2b3e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:20:49.525 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:20:49.524+0000 7fa69e224740 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c112ee63b0 con 0x55c112f233f0 2024-03-24T00:20:49.525 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:20:49.524+0000 7fa69e224740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c112e88830 con 0x55c112f23de0 2024-03-24T00:20:49.525 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:20:49.524+0000 7fa69e224740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c112e315d0 con 0x55c112f28f50 2024-03-24T00:20:49.525 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:20:49.524+0000 7fa693e00640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55c112f233f0 0x55c112f23810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:20:49.525 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:20:49.524+0000 7fa698c00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55c112f23de0 0x55c112f288e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:20:49.525 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:20:49.524+0000 7fa698c00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55c112f23de0 0x55c112f288e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.118:55270/0 (socket says 172.21.15.118:55270) 2024-03-24T00:20:49.525 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:20:49.524+0000 7fa698c00640 1 -- 172.21.15.118:0/228481141 learned_addr learned my addr 172.21.15.118:0/228481141 (peer_addr_for_me v2:172.21.15.118:0/0) 2024-03-24T00:20:49.525 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:20:49.524+0000 7fa693e00640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55c112f233f0 0x55c112f23810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.118:3300/0 says I am v2:172.21.15.118:59106/0 (socket says 172.21.15.118:59106) 2024-03-24T00:20:49.525 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:20:49.524+0000 7fa693e00640 1 --2- 172.21.15.118:0/228481141 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55c112f28f50 0x55c112f2b3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:20:49.525 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:20:49.524+0000 7fa698c00640 1 -- 172.21.15.118:0/228481141 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55c112f28f50 msgr2=0x55c112f2b3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:20:49.525 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:20:49.524+0000 7fa698c00640 1 --2- 172.21.15.118:0/228481141 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55c112f28f50 0x55c112f2b3e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:20:49.525 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:20:49.524+0000 7fa698c00640 1 -- 172.21.15.118:0/228481141 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55c112f233f0 msgr2=0x55c112f23810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:20:49.526 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:20:49.524+0000 7fa698c00640 1 --2- 172.21.15.118:0/228481141 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55c112f233f0 0x55c112f23810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:20:49.526 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:20:49.524+0000 7fa698c00640 1 -- 172.21.15.118:0/228481141 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55c112e492c0 con 0x55c112f23de0 2024-03-24T00:20:49.526 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:20:49.525+0000 7fa698c00640 1 --2- 172.21.15.118:0/228481141 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55c112f23de0 0x55c112f288e0 secure :-1 s=READY pgs=411 cs=0 l=1 rev1=1 crypto rx=0x7fa6880115c0 tx=0x7fa688011a80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=12723761c75b7ce7 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:20:49.526 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:20:49.525+0000 7fa691600640 1 --2- 172.21.15.118:0/1753000363 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7fa67c0a2870 0x7fa67c0a4d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:20:49.526 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:20:49.525+0000 7fa685c00640 1 -- 172.21.15.118:0/228481141 <== mon.2 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa688005020 con 0x55c112f23de0 2024-03-24T00:20:49.526 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:20:49.525+0000 7fa685c00640 1 -- 172.21.15.118:0/228481141 <== mon.2 v2:172.21.15.189:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fa6880047a0 con 0x55c112f23de0 2024-03-24T00:20:49.526 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:20:49.525+0000 7fa685c00640 1 -- 172.21.15.118:0/228481141 <== mon.2 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa68800fa00 con 0x55c112f23de0 2024-03-24T00:20:49.526 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:20:49.525+0000 7fa691600640 1 -- 172.21.15.118:0/1753000363 <== mon.2 v2:172.21.15.189:3300/0 5 ==== osd_map(96..96 src has 1..96) v4 ==== 8793+0+0 (secure 0 0 0) 0x7fa68c0cf320 con 0x7fa6941030d0 2024-03-24T00:20:49.526 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:20:49.525+0000 7fa693e00640 1 --2- 172.21.15.118:0/1753000363 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7fa67c0a2870 0x7fa67c0a4d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:20:49.530 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:20:49.526+0000 7fa69e224740 1 -- 172.21.15.118:0/228481141 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55c112f23de0 msgr2=0x55c112f288e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:20:49.530 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:20:49.526+0000 7fa693e00640 1 --2- 172.21.15.118:0/1753000363 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7fa67c0a2870 0x7fa67c0a4d30 secure :-1 s=READY pgs=357 cs=0 l=1 rev1=1 crypto rx=0x7fa68002c250 tx=0x7fa680000bc0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:20:49.530 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:20:49.526+0000 7fa69e224740 1 --2- 172.21.15.118:0/228481141 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55c112f23de0 0x55c112f288e0 secure :-1 s=READY pgs=411 cs=0 l=1 rev1=1 crypto rx=0x7fa6880115c0 tx=0x7fa688011a80 comp rx=0 tx=0).stop 2024-03-24T00:20:49.530 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:20:49.526+0000 7fa69e224740 1 -- 172.21.15.118:0/228481141 shutdown_connections 2024-03-24T00:20:49.530 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:20:49.526+0000 7fa69e224740 1 --2- 172.21.15.118:0/228481141 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55c112f28f50 0x55c112f2b3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:20:49.530 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:20:49.526+0000 7fa69e224740 1 --2- 172.21.15.118:0/228481141 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55c112f23de0 0x55c112f288e0 unknown :-1 s=CLOSED pgs=411 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:20:49.531 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:20:49.526+0000 7fa69e224740 1 --2- 172.21.15.118:0/228481141 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55c112f233f0 0x55c112f23810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:20:49.531 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:20:49.526+0000 7fa69e224740 1 -- 172.21.15.118:0/228481141 >> 172.21.15.118:0/228481141 conn(0x55c112f1e960 msgr2=0x55c112f1ed60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:20:49.531 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:20:49.526+0000 7fa69e224740 1 -- 172.21.15.118:0/228481141 shutdown_connections 2024-03-24T00:20:49.531 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:20:49.526+0000 7fa69e224740 1 -- 172.21.15.118:0/228481141 wait complete. 2024-03-24T00:20:49.531 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:20:49.526+0000 7fa69e224740 1 -- start start 2024-03-24T00:20:49.531 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:20:49.527+0000 7fa69e224740 1 Processor -- start 2024-03-24T00:20:49.531 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:20:49.527+0000 7fa69e224740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55c112f233f0 0x55c112f2d5d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:20:49.531 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:20:49.527+0000 7fa69e224740 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55c112f23de0 0x55c112f2db10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:20:49.531 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:20:49.527+0000 7fa69e224740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55c112f28f50 0x55c112f34ae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:20:49.531 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:20:49.527+0000 7fa69e224740 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c112e44a20 con 0x55c112f23de0 2024-03-24T00:20:49.531 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:20:49.527+0000 7fa69e224740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c112e848d0 con 0x55c112f233f0 2024-03-24T00:20:49.531 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:20:49.527+0000 7fa69e224740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c112f2bf00 con 0x55c112f28f50 2024-03-24T00:20:49.531 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:20:49.527+0000 7fa69e224740 10 client.0 _ll_register_callbacks cb 0x55c112e811c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T00:20:49.532 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:20:49.527+0000 7fa698c00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55c112f233f0 0x55c112f2d5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:20:49.532 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:20:49.527+0000 7fa699600640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55c112f28f50 0x55c112f34ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:20:49.532 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:20:49.527+0000 7fa698c00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55c112f233f0 0x55c112f2d5d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.118:55286/0 (socket says 172.21.15.118:55286) 2024-03-24T00:20:49.532 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:20:49.527+0000 7fa698c00640 1 -- 172.21.15.118:0/725687618 learned_addr learned my addr 172.21.15.118:0/725687618 (peer_addr_for_me v2:172.21.15.118:0/0) 2024-03-24T00:20:49.532 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:20:49.528+0000 7fa693e00640 1 --2- 172.21.15.118:0/725687618 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55c112f23de0 0x55c112f2db10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:20:49.532 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:20:49.528+0000 7fa698c00640 1 -- 172.21.15.118:0/725687618 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55c112f28f50 msgr2=0x55c112f34ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:20:49.532 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:20:49.528+0000 7fa698c00640 1 --2- 172.21.15.118:0/725687618 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55c112f28f50 0x55c112f34ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:20:49.532 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:20:49.528+0000 7fa698c00640 1 -- 172.21.15.118:0/725687618 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55c112f23de0 msgr2=0x55c112f2db10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:20:49.532 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:20:49.528+0000 7fa698c00640 1 --2- 172.21.15.118:0/725687618 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55c112f23de0 0x55c112f2db10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:20:49.532 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:20:49.528+0000 7fa698c00640 1 -- 172.21.15.118:0/725687618 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55c112b9d210 con 0x55c112f233f0 2024-03-24T00:20:49.532 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:20:49.528+0000 7fa699600640 1 --2- 172.21.15.118:0/725687618 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55c112f28f50 0x55c112f34ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T00:20:49.532 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:20:49.528+0000 7fa693e00640 1 --2- 172.21.15.118:0/725687618 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55c112f23de0 0x55c112f2db10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T00:20:49.532 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:20:49.528+0000 7fa698c00640 1 --2- 172.21.15.118:0/725687618 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55c112f233f0 0x55c112f2d5d0 secure :-1 s=READY pgs=412 cs=0 l=1 rev1=1 crypto rx=0x7fa6880058c0 tx=0x7fa688005230 comp rx=0 tx=0).ready entity=mon.2 client_cookie=7fe7a935048f1569 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:20:49.532 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:20:49.528+0000 7fa67a000640 10 client.0 ms_handle_connect on v2:172.21.15.189:3300/0 2024-03-24T00:20:49.532 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:20:49.528+0000 7fa67a000640 1 -- 172.21.15.118:0/725687618 <== mon.2 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa688028020 con 0x55c112f233f0 2024-03-24T00:20:49.533 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:20:49.528+0000 7fa69e224740 1 -- 172.21.15.118:0/725687618 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fa6880047a0 con 0x55c112f233f0 2024-03-24T00:20:49.533 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:20:49.528+0000 7fa67a000640 1 -- 172.21.15.118:0/725687618 <== mon.2 v2:172.21.15.189:3300/0 2 ==== osd_map(96..96 src has 1..96) v4 ==== 8793+0+0 (secure 0 0 0) 0x7fa6880053a0 con 0x55c112f233f0 2024-03-24T00:20:49.533 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:20:49.529+0000 7fa67a000640 1 -- 172.21.15.118:0/725687618 <== mon.2 v2:172.21.15.189:3300/0 3 ==== mon_get_version_reply(handle=1 version=59) v2 ==== 24+0+0 (secure 0 0 0) 0x7fa688033050 con 0x55c112f233f0 2024-03-24T00:20:49.533 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:20:49.529+0000 7fa69e224740 10 client.26687 fetch_fsmap learned FSMap version 59 2024-03-24T00:20:49.533 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:20:49.529+0000 7fa69e224740 1 -- 172.21.15.118:0/725687618 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,fsmap=59,monmap=4+}) v3 -- 0x55c112c69620 con 0x55c112f233f0 2024-03-24T00:20:49.533 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:20:49.529+0000 7fa67a000640 1 -- 172.21.15.118:0/725687618 <== mon.2 v2:172.21.15.189:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fa6880047a0 con 0x55c112f233f0 2024-03-24T00:20:49.533 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:20:49.529+0000 7fa67a000640 1 -- 172.21.15.118:0/725687618 <== mon.2 v2:172.21.15.189:3300/0 5 ==== fsmap(e 59) v1 ==== 7950+0+0 (secure 0 0 0) 0x7fa688023620 con 0x55c112f233f0 2024-03-24T00:20:49.533 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:20:49.529+0000 7fa69e224740 10 client.26687 fetch_fsmap finished waiting for FSMap version 59 2024-03-24T00:20:49.533 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:20:49.529+0000 7fa69e224740 10 client.26687 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-24T00:20:49.533 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:20:49.529+0000 7fa69e224740 20 client.26687 populate_metadata read hostname 'smithi118' 2024-03-24T00:20:49.533 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:20:49.529+0000 7fa69e224740 1 --2- 172.21.15.118:0/725687618 >> [v2:172.21.15.189:6838/4074484168,v1:172.21.15.189:6839/4074484168] conn(0x55c112f391a0 0x55c112f3b5d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:20:49.533 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:20:49.529+0000 7fa69e224740 4 client.26687 mds_command: new command op to 24569 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-24T00:20:49.533 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:20:49.529+0000 7fa69e224740 1 -- 172.21.15.118:0/725687618 --> [v2:172.21.15.189:6838/4074484168,v1:172.21.15.189:6839/4074484168] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55c112e95d70 con 0x55c112f391a0 2024-03-24T00:20:49.533 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:20:49.529+0000 7fa693e00640 1 --2- 172.21.15.118:0/725687618 >> [v2:172.21.15.189:6838/4074484168,v1:172.21.15.189:6839/4074484168] conn(0x55c112f391a0 0x55c112f3b5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:20:49.534 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:20:49.530+0000 7fa693e00640 1 --2- 172.21.15.118:0/725687618 >> [v2:172.21.15.189:6838/4074484168,v1:172.21.15.189:6839/4074484168] conn(0x55c112f391a0 0x55c112f3b5d0 crc :-1 s=READY pgs=162 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=9a1cf303dcbbb4c8 server_cookie=2d3aa79996b8b090 in_seq=0 out_seq=0 2024-03-24T00:20:49.534 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:20:49.530+0000 7fa67a000640 10 client.26687 ms_handle_connect on v2:172.21.15.189:6838/4074484168 2024-03-24T00:20:49.534 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:20:49.531+0000 7fa67a000640 1 -- 172.21.15.118:0/725687618 <== mds.0 v2:172.21.15.189:6838/4074484168 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x55c112e95d70 con 0x55c112f391a0 2024-03-24T00:20:49.534 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:20:49.531+0000 7fa67a000640 10 client.26687 handle_command_reply: tid=0 2024-03-24T00:20:49.534 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:20:49.531+0000 7fa69e224740 1 client.26687 shutdown 2024-03-24T00:20:49.535 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:20:49.531+0000 7fa69e224740 1 -- 172.21.15.118:0/725687618 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55c112f233f0 msgr2=0x55c112f2d5d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:20:49.535 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:20:49.531+0000 7fa69e224740 1 --2- 172.21.15.118:0/725687618 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55c112f233f0 0x55c112f2d5d0 secure :-1 s=READY pgs=412 cs=0 l=1 rev1=1 crypto rx=0x7fa6880058c0 tx=0x7fa688005230 comp rx=0 tx=0).stop 2024-03-24T00:20:49.535 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:20:49.531+0000 7fa69e224740 1 -- 172.21.15.118:0/725687618 shutdown_connections 2024-03-24T00:20:49.535 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:20:49.531+0000 7fa69e224740 1 --2- 172.21.15.118:0/725687618 >> [v2:172.21.15.189:6838/4074484168,v1:172.21.15.189:6839/4074484168] conn(0x55c112f391a0 0x55c112f3b5d0 crc :-1 s=READY pgs=162 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:20:49.535 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:20:49.531+0000 7fa69e224740 1 --2- 172.21.15.118:0/725687618 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55c112f28f50 0x55c112f34ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:20:49.535 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:20:49.531+0000 7fa69e224740 1 --2- 172.21.15.118:0/725687618 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55c112f23de0 0x55c112f2db10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:20:49.535 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:20:49.531+0000 7fa69e224740 1 --2- 172.21.15.118:0/725687618 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55c112f233f0 0x55c112f2d5d0 unknown :-1 s=CLOSED pgs=412 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:20:49.535 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:20:49.531+0000 7fa69e224740 1 -- 172.21.15.118:0/725687618 >> 172.21.15.118:0/725687618 conn(0x55c112f1e960 msgr2=0x55c112f1db70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:20:49.535 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:20:49.531+0000 7fa67a000640 0 client.26687 ms_handle_reset on v2:172.21.15.189:6838/4074484168 2024-03-24T00:20:49.535 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:20:49.531+0000 7fa69e224740 1 -- 172.21.15.118:0/725687618 shutdown_connections 2024-03-24T00:20:49.536 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:20:49.531+0000 7fa69e224740 1 -- 172.21.15.118:0/725687618 wait complete. 2024-03-24T00:20:49.536 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:20:49.531+0000 7fa69e224740 20 client.26687 trim_cache size 0 max 16384 2024-03-24T00:20:49.546 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:20:49.546+0000 7fa69e224740 1 Processor -- start 2024-03-24T00:20:49.547 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:20:49.546+0000 7fa69e224740 1 -- start start 2024-03-24T00:20:49.547 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:20:49.546+0000 7fa69e224740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55c112f233f0 0x55c112f35530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:20:49.547 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:20:49.547+0000 7fa69e224740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55c112f23de0 0x55c112f21bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:20:49.547 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:20:49.547+0000 7fa69e224740 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55c112f28f50 0x55c112f22100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:20:49.547 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:20:49.547+0000 7fa69e224740 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa6941abc50 con 0x55c112f28f50 2024-03-24T00:20:49.547 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:20:49.547+0000 7fa69e224740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa6941ac2b0 con 0x55c112f233f0 2024-03-24T00:20:49.547 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:20:49.547+0000 7fa69e224740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa6941ac970 con 0x55c112f23de0 2024-03-24T00:20:49.547 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:20:49.547+0000 7fa698c00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55c112f233f0 0x55c112f35530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:20:49.547 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:20:49.547+0000 7fa698c00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55c112f233f0 0x55c112f35530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.118:55300/0 (socket says 172.21.15.118:55300) 2024-03-24T00:20:49.547 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:20:49.547+0000 7fa698c00640 1 -- 172.21.15.118:0/4033412939 learned_addr learned my addr 172.21.15.118:0/4033412939 (peer_addr_for_me v2:172.21.15.118:0/0) 2024-03-24T00:20:49.547 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:20:49.547+0000 7fa693e00640 1 --2- 172.21.15.118:0/4033412939 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55c112f23de0 0x55c112f21bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:20:49.552 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:20:49.547+0000 7fa699600640 1 --2- 172.21.15.118:0/4033412939 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55c112f28f50 0x55c112f22100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:20:49.552 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:20:49.547+0000 7fa698c00640 1 -- 172.21.15.118:0/4033412939 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55c112f23de0 msgr2=0x55c112f21bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:20:49.552 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:20:49.547+0000 7fa698c00640 1 --2- 172.21.15.118:0/4033412939 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55c112f23de0 0x55c112f21bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:20:49.552 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:20:49.547+0000 7fa698c00640 1 -- 172.21.15.118:0/4033412939 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55c112f28f50 msgr2=0x55c112f22100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:20:49.552 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:20:49.547+0000 7fa698c00640 1 --2- 172.21.15.118:0/4033412939 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55c112f28f50 0x55c112f22100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:20:49.552 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:20:49.547+0000 7fa698c00640 1 -- 172.21.15.118:0/4033412939 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55c112f28400 con 0x55c112f233f0 2024-03-24T00:20:49.552 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:20:49.547+0000 7fa699600640 1 --2- 172.21.15.118:0/4033412939 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55c112f28f50 0x55c112f22100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-24T00:20:49.552 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:20:49.548+0000 7fa698c00640 1 --2- 172.21.15.118:0/4033412939 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55c112f233f0 0x55c112f35530 secure :-1 s=READY pgs=413 cs=0 l=1 rev1=1 crypto rx=0x7fa688023ac0 tx=0x7fa688004ea0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=66676a3b8542d8fc server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:20:49.552 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:20:49.548+0000 7fa685200640 1 -- 172.21.15.118:0/4033412939 <== mon.2 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa688028070 con 0x55c112f233f0 2024-03-24T00:20:49.552 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:20:49.548+0000 7fa685200640 1 -- 172.21.15.118:0/4033412939 <== mon.2 v2:172.21.15.189:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fa688023c30 con 0x55c112f233f0 2024-03-24T00:20:49.553 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:20:49.548+0000 7fa685200640 1 -- 172.21.15.118:0/4033412939 <== mon.2 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa68801d040 con 0x55c112f233f0 2024-03-24T00:20:49.553 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:20:49.549+0000 7fa69e224740 1 -- 172.21.15.118:0/4033412939 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55c112f233f0 msgr2=0x55c112f35530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:20:49.553 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:20:49.549+0000 7fa69e224740 1 --2- 172.21.15.118:0/4033412939 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55c112f233f0 0x55c112f35530 secure :-1 s=READY pgs=413 cs=0 l=1 rev1=1 crypto rx=0x7fa688023ac0 tx=0x7fa688004ea0 comp rx=0 tx=0).stop 2024-03-24T00:20:49.553 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:20:49.549+0000 7fa69e224740 1 -- 172.21.15.118:0/4033412939 shutdown_connections 2024-03-24T00:20:49.553 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:20:49.549+0000 7fa69e224740 1 --2- 172.21.15.118:0/4033412939 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55c112f28f50 0x55c112f22100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:20:49.553 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:20:49.549+0000 7fa69e224740 1 --2- 172.21.15.118:0/4033412939 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55c112f23de0 0x55c112f21bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:20:49.553 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:20:49.549+0000 7fa69e224740 1 --2- 172.21.15.118:0/4033412939 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55c112f233f0 0x55c112f35530 unknown :-1 s=CLOSED pgs=413 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:20:49.553 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:20:49.549+0000 7fa69e224740 1 -- 172.21.15.118:0/4033412939 >> 172.21.15.118:0/4033412939 conn(0x55c112f1e960 msgr2=0x55c112f25140 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:20:49.553 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:20:49.549+0000 7fa69e224740 1 -- 172.21.15.118:0/4033412939 shutdown_connections 2024-03-24T00:20:49.553 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:20:49.549+0000 7fa69e224740 1 -- 172.21.15.118:0/4033412939 wait complete. 2024-03-24T00:20:49.553 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:20:49.549+0000 7fa69e224740 1 -- start start 2024-03-24T00:20:49.553 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:20:49.549+0000 7fa69e224740 1 Processor -- start 2024-03-24T00:20:49.553 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:20:49.550+0000 7fa69e224740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55c112f233f0 0x55c112f314e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:20:49.554 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:20:49.550+0000 7fa69e224740 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55c112f23de0 0x55c112f31a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:20:49.554 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:20:49.550+0000 7fa69e224740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55c112f28f50 0x55c112f32a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:20:49.554 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:20:49.550+0000 7fa69e224740 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa6941a9110 con 0x55c112f23de0 2024-03-24T00:20:49.554 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:20:49.550+0000 7fa69e224740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa6941aa3b0 con 0x55c112f28f50 2024-03-24T00:20:49.554 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:20:49.550+0000 7fa69e224740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa6941a9710 con 0x55c112f233f0 2024-03-24T00:20:49.554 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:20:49.550+0000 7fa69e224740 10 client.0 _ll_register_callbacks cb 0x7fa68c0093a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T00:20:49.554 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:20:49.550+0000 7fa699600640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55c112f28f50 0x55c112f32a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:20:49.554 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:20:49.550+0000 7fa698c00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55c112f233f0 0x55c112f314e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:20:49.554 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:20:49.550+0000 7fa698c00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55c112f233f0 0x55c112f314e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.118:57786/0 (socket says 172.21.15.118:57786) 2024-03-24T00:20:49.554 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:20:49.550+0000 7fa699600640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55c112f28f50 0x55c112f32a30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.118:55314/0 (socket says 172.21.15.118:55314) 2024-03-24T00:20:49.554 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:20:49.550+0000 7fa698c00640 1 -- 172.21.15.118:0/111443530 learned_addr learned my addr 172.21.15.118:0/111443530 (peer_addr_for_me v2:172.21.15.118:0/0) 2024-03-24T00:20:49.554 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:20:49.550+0000 7fa693e00640 1 --2- 172.21.15.118:0/111443530 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55c112f23de0 0x55c112f31a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:20:49.554 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:20:49.550+0000 7fa698c00640 1 -- 172.21.15.118:0/111443530 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55c112f28f50 msgr2=0x55c112f32a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:20:49.554 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:20:49.550+0000 7fa698c00640 1 --2- 172.21.15.118:0/111443530 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55c112f28f50 0x55c112f32a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:20:49.555 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:20:49.550+0000 7fa698c00640 1 -- 172.21.15.118:0/111443530 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55c112f23de0 msgr2=0x55c112f31a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:20:49.555 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:20:49.550+0000 7fa698c00640 1 --2- 172.21.15.118:0/111443530 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55c112f23de0 0x55c112f31a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:20:49.555 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:20:49.550+0000 7fa698c00640 1 -- 172.21.15.118:0/111443530 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55c112f33020 con 0x55c112f233f0 2024-03-24T00:20:49.555 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:20:49.550+0000 7fa693e00640 1 --2- 172.21.15.118:0/111443530 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55c112f23de0 0x55c112f31a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T00:20:49.555 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:20:49.551+0000 7fa698c00640 1 --2- 172.21.15.118:0/111443530 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55c112f233f0 0x55c112f314e0 secure :-1 s=READY pgs=417 cs=0 l=1 rev1=1 crypto rx=0x7fa688004fd0 tx=0x7fa688005d30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=ed23e7876bdf090d server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:20:49.555 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:20:49.551+0000 7fa67a000640 10 client.0 ms_handle_connect on v2:172.21.15.193:3300/0 2024-03-24T00:20:49.555 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:20:49.551+0000 7fa67a000640 1 -- 172.21.15.118:0/111443530 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa688028070 con 0x55c112f233f0 2024-03-24T00:20:49.555 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:20:49.551+0000 7fa69e224740 1 -- 172.21.15.118:0/111443530 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fa688023c30 con 0x55c112f233f0 2024-03-24T00:20:49.555 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:20:49.551+0000 7fa67a000640 1 -- 172.21.15.118:0/111443530 <== mon.1 v2:172.21.15.193:3300/0 2 ==== osd_map(96..96 src has 1..96) v4 ==== 8793+0+0 (secure 0 0 0) 0x7fa688011240 con 0x55c112f233f0 2024-03-24T00:20:49.555 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:20:49.551+0000 7fa67a000640 1 -- 172.21.15.118:0/111443530 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_get_version_reply(handle=1 version=59) v2 ==== 24+0+0 (secure 0 0 0) 0x7fa68801d040 con 0x55c112f233f0 2024-03-24T00:20:49.555 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:20:49.551+0000 7fa69e224740 10 client.26707 fetch_fsmap learned FSMap version 59 2024-03-24T00:20:49.555 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:20:49.551+0000 7fa69e224740 1 -- 172.21.15.118:0/111443530 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,fsmap=59,monmap=4+}) v3 -- 0x55c112f342a0 con 0x55c112f233f0 2024-03-24T00:20:49.555 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:20:49.551+0000 7fa67a000640 1 -- 172.21.15.118:0/111443530 <== mon.1 v2:172.21.15.193:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fa688023c30 con 0x55c112f233f0 2024-03-24T00:20:49.556 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:20:49.552+0000 7fa67a000640 1 -- 172.21.15.118:0/111443530 <== mon.1 v2:172.21.15.193:3300/0 5 ==== fsmap(e 59) v1 ==== 7950+0+0 (secure 0 0 0) 0x7fa68800f980 con 0x55c112f233f0 2024-03-24T00:20:49.556 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:20:49.552+0000 7fa69e224740 10 client.26707 fetch_fsmap finished waiting for FSMap version 59 2024-03-24T00:20:49.556 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:20:49.552+0000 7fa69e224740 10 client.26707 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-24T00:20:49.556 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:20:49.552+0000 7fa69e224740 20 client.26707 populate_metadata read hostname 'smithi118' 2024-03-24T00:20:49.556 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:20:49.552+0000 7fa69e224740 1 --2- 172.21.15.118:0/111443530 >> [v2:172.21.15.189:6838/4074484168,v1:172.21.15.189:6839/4074484168] conn(0x55c112f9d630 0x55c112f9fa50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:20:49.556 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:20:49.552+0000 7fa69e224740 4 client.26707 mds_command: new command op to 24569 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-03-24T00:20:49.556 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:20:49.552+0000 7fa69e224740 1 -- 172.21.15.118:0/111443530 --> [v2:172.21.15.189:6838/4074484168,v1:172.21.15.189:6839/4074484168] -- command(tid 0: {"prefix": "flush journal"}) v1 -- 0x55c112e900c0 con 0x55c112f9d630 2024-03-24T00:20:49.556 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:20:49.552+0000 7fa693e00640 1 --2- 172.21.15.118:0/111443530 >> [v2:172.21.15.189:6838/4074484168,v1:172.21.15.189:6839/4074484168] conn(0x55c112f9d630 0x55c112f9fa50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:20:49.556 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:20:49.552+0000 7fa693e00640 1 --2- 172.21.15.118:0/111443530 >> [v2:172.21.15.189:6838/4074484168,v1:172.21.15.189:6839/4074484168] conn(0x55c112f9d630 0x55c112f9fa50 crc :-1 s=READY pgs=163 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=337dd6c9c74a2c82 server_cookie=e0f244927f2282b9 in_seq=0 out_seq=0 2024-03-24T00:20:49.556 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:20:49.552+0000 7fa67a000640 10 client.26707 ms_handle_connect on v2:172.21.15.189:6838/4074484168 2024-03-24T00:20:50.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:20:49 smithi118 ceph-mon[31535]: daemon mds.g restarted 2024-03-24T00:20:50.197 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:20:49 smithi118 ceph-mon[31535]: osdmap e96: 12 total, 12 up, 12 in 2024-03-24T00:20:50.197 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:20:49 smithi118 ceph-mon[31535]: mds.? [v2:172.21.15.193:6832/1233062505,v1:172.21.15.193:6833/1233062505] up:resolve 2024-03-24T00:20:50.197 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:20:49 smithi118 ceph-mon[31535]: mds.? [v2:172.21.15.118:6838/3808689299,v1:172.21.15.118:6839/3808689299] up:boot 2024-03-24T00:20:50.197 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:20:49 smithi118 ceph-mon[31535]: Standby daemon mds.h assigned to filesystem cephfs as rank 1 2024-03-24T00:20:50.197 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:20:49 smithi118 ceph-mon[31535]: fsmap cephfs:4/5 {0=b=up:active,2=c=up:active,3=l=up:resolve,4=e=up:active} 4 up:standby-replay 4 up:standby, 1 failed 2024-03-24T00:20:50.197 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:20:49 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2024-03-24T00:20:50.197 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:20:49 smithi118 ceph-mon[31535]: fsmap cephfs:5/5 3 up:standby-replay 4 up:standby 1 up:replay 1 up:resolve 3 up:active 2024-03-24T00:20:50.197 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:20:49 smithi118 ceph-mon[31535]: pgmap v558: 97 pgs: 97 active+clean; 113 MiB data, 683 MiB used, 1.0 TiB / 1.0 TiB avail; 258 KiB/s rd, 88 KiB/s wr, 26 op/s 2024-03-24T00:20:50.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:20:49 smithi189 ceph-mon[36187]: daemon mds.g restarted 2024-03-24T00:20:50.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:20:49 smithi189 ceph-mon[36187]: osdmap e96: 12 total, 12 up, 12 in 2024-03-24T00:20:50.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:20:49 smithi189 ceph-mon[36187]: mds.? [v2:172.21.15.193:6832/1233062505,v1:172.21.15.193:6833/1233062505] up:resolve 2024-03-24T00:20:50.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:20:49 smithi189 ceph-mon[36187]: mds.? [v2:172.21.15.118:6838/3808689299,v1:172.21.15.118:6839/3808689299] up:boot 2024-03-24T00:20:50.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:20:49 smithi189 ceph-mon[36187]: Standby daemon mds.h assigned to filesystem cephfs as rank 1 2024-03-24T00:20:50.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:20:49 smithi189 ceph-mon[36187]: fsmap cephfs:4/5 {0=b=up:active,2=c=up:active,3=l=up:resolve,4=e=up:active} 4 up:standby-replay 4 up:standby, 1 failed 2024-03-24T00:20:50.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:20:49 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2024-03-24T00:20:50.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:20:49 smithi189 ceph-mon[36187]: fsmap cephfs:5/5 3 up:standby-replay 4 up:standby 1 up:replay 1 up:resolve 3 up:active 2024-03-24T00:20:50.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:20:49 smithi189 ceph-mon[36187]: pgmap v558: 97 pgs: 97 active+clean; 113 MiB data, 683 MiB used, 1.0 TiB / 1.0 TiB avail; 258 KiB/s rd, 88 KiB/s wr, 26 op/s 2024-03-24T00:20:50.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:20:49 smithi193 ceph-mon[36878]: daemon mds.g restarted 2024-03-24T00:20:50.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:20:49 smithi193 ceph-mon[36878]: osdmap e96: 12 total, 12 up, 12 in 2024-03-24T00:20:50.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:20:49 smithi193 ceph-mon[36878]: mds.? [v2:172.21.15.193:6832/1233062505,v1:172.21.15.193:6833/1233062505] up:resolve 2024-03-24T00:20:50.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:20:49 smithi193 ceph-mon[36878]: mds.? [v2:172.21.15.118:6838/3808689299,v1:172.21.15.118:6839/3808689299] up:boot 2024-03-24T00:20:50.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:20:49 smithi193 ceph-mon[36878]: Standby daemon mds.h assigned to filesystem cephfs as rank 1 2024-03-24T00:20:50.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:20:49 smithi193 ceph-mon[36878]: fsmap cephfs:4/5 {0=b=up:active,2=c=up:active,3=l=up:resolve,4=e=up:active} 4 up:standby-replay 4 up:standby, 1 failed 2024-03-24T00:20:50.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:20:49 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2024-03-24T00:20:50.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:20:49 smithi193 ceph-mon[36878]: fsmap cephfs:5/5 3 up:standby-replay 4 up:standby 1 up:replay 1 up:resolve 3 up:active 2024-03-24T00:20:50.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:20:49 smithi193 ceph-mon[36878]: pgmap v558: 97 pgs: 97 active+clean; 113 MiB data, 683 MiB used, 1.0 TiB / 1.0 TiB avail; 258 KiB/s rd, 88 KiB/s wr, 26 op/s 2024-03-24T00:20:51.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:20:51 smithi118 ceph-mon[31535]: mds.? [v2:172.21.15.189:6836/3450187137,v1:172.21.15.189:6837/3450187137] up:resolve 2024-03-24T00:20:51.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:20:51 smithi118 ceph-mon[31535]: fsmap cephfs:5/5 3 up:standby-replay 4 up:standby 2 up:resolve 3 up:active 2024-03-24T00:20:51.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:20:51 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T00:20:51.703 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:20:51 smithi189 ceph-mon[36187]: mds.? [v2:172.21.15.189:6836/3450187137,v1:172.21.15.189:6837/3450187137] up:resolve 2024-03-24T00:20:51.704 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:20:51 smithi189 ceph-mon[36187]: fsmap cephfs:5/5 3 up:standby-replay 4 up:standby 2 up:resolve 3 up:active 2024-03-24T00:20:51.704 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:20:51 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T00:20:51.710 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:20:51 smithi193 ceph-mon[36878]: mds.? [v2:172.21.15.189:6836/3450187137,v1:172.21.15.189:6837/3450187137] up:resolve 2024-03-24T00:20:51.710 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:20:51 smithi193 ceph-mon[36878]: fsmap cephfs:5/5 3 up:standby-replay 4 up:standby 2 up:resolve 3 up:active 2024-03-24T00:20:51.710 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:20:51 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T00:20:52.347 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:20:52.347+0000 7fa67a000640 1 -- 172.21.15.118:0/111443530 <== mds.0 v2:172.21.15.189:6838/4074484168 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+44 (crc 0 0 0) 0x55c112e900c0 con 0x55c112f9d630 2024-03-24T00:20:52.347 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:20:52.347+0000 7fa67a000640 10 client.26707 handle_command_reply: tid=0 2024-03-24T00:20:52.347 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:20:52.347+0000 7fa69e224740 1 client.26707 shutdown 2024-03-24T00:20:52.348 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:20:52.348+0000 7fa69e224740 1 -- 172.21.15.118:0/111443530 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55c112f233f0 msgr2=0x55c112f314e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:20:52.348 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:20:52.348+0000 7fa69e224740 1 --2- 172.21.15.118:0/111443530 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55c112f233f0 0x55c112f314e0 secure :-1 s=READY pgs=417 cs=0 l=1 rev1=1 crypto rx=0x7fa688004fd0 tx=0x7fa688005d30 comp rx=0 tx=0).stop 2024-03-24T00:20:52.348 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:20:52.348+0000 7fa69e224740 1 -- 172.21.15.118:0/111443530 shutdown_connections 2024-03-24T00:20:52.348 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:20:52.348+0000 7fa69e224740 1 --2- 172.21.15.118:0/111443530 >> [v2:172.21.15.189:6838/4074484168,v1:172.21.15.189:6839/4074484168] conn(0x55c112f9d630 0x55c112f9fa50 crc :-1 s=READY pgs=163 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:20:52.348 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:20:52.348+0000 7fa69e224740 1 --2- 172.21.15.118:0/111443530 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55c112f28f50 0x55c112f32a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:20:52.348 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:20:52.348+0000 7fa69e224740 1 --2- 172.21.15.118:0/111443530 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55c112f23de0 0x55c112f31a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:20:52.348 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:20:52.348+0000 7fa69e224740 1 --2- 172.21.15.118:0/111443530 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55c112f233f0 0x55c112f314e0 unknown :-1 s=CLOSED pgs=417 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:20:52.348 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:20:52.348+0000 7fa69e224740 1 -- 172.21.15.118:0/111443530 >> 172.21.15.118:0/111443530 conn(0x55c112f1e960 msgr2=0x55c112f21bc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:20:52.349 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:20:52.348+0000 7fa67a000640 0 client.26707 ms_handle_reset on v2:172.21.15.189:6838/4074484168 2024-03-24T00:20:52.349 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:20:52.349+0000 7fa69e224740 1 -- 172.21.15.118:0/111443530 shutdown_connections 2024-03-24T00:20:52.349 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:20:52.349+0000 7fa69e224740 1 -- 172.21.15.118:0/111443530 wait complete. 2024-03-24T00:20:52.349 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:20:52.349+0000 7fa69e224740 20 client.26707 trim_cache size 0 max 16384 2024-03-24T00:20:52.349 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stdout:{ 2024-03-24T00:20:52.349 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stdout: "message": "", 2024-03-24T00:20:52.349 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stdout: "return_code": 0 2024-03-24T00:20:52.350 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stdout:} 2024-03-24T00:20:52.350 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:20:52.350+0000 7fa69a000640 1 -- 172.21.15.118:0/1753000363 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7fa67c0a2870 msgr2=0x7fa67c0a4d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:20:52.350 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:20:52.350+0000 7fa69a000640 1 --2- 172.21.15.118:0/1753000363 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7fa67c0a2870 0x7fa67c0a4d30 secure :-1 s=READY pgs=357 cs=0 l=1 rev1=1 crypto rx=0x7fa68002c250 tx=0x7fa680000bc0 comp rx=0 tx=0).stop 2024-03-24T00:20:52.350 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:20:52.350+0000 7fa69a000640 1 -- 172.21.15.118:0/1753000363 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa6941030d0 msgr2=0x7fa694073fd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:20:52.351 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:20:52.350+0000 7fa69a000640 1 --2- 172.21.15.118:0/1753000363 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa6941030d0 0x7fa694073fd0 secure :-1 s=READY pgs=410 cs=0 l=1 rev1=1 crypto rx=0x7fa68c009fd0 tx=0x7fa68c00ef10 comp rx=0 tx=0).stop 2024-03-24T00:20:52.351 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:20:52.350+0000 7fa69a000640 1 -- 172.21.15.118:0/1753000363 shutdown_connections 2024-03-24T00:20:52.351 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:20:52.350+0000 7fa69a000640 1 --2- 172.21.15.118:0/1753000363 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7fa67c0a2870 0x7fa67c0a4d30 unknown :-1 s=CLOSED pgs=357 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:20:52.351 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:20:52.350+0000 7fa69a000640 1 --2- 172.21.15.118:0/1753000363 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa6941030d0 0x7fa694073fd0 unknown :-1 s=CLOSED pgs=410 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:20:52.351 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:20:52.350+0000 7fa69a000640 1 --2- 172.21.15.118:0/1753000363 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fa694102710 0x7fa694073a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:20:52.351 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:20:52.350+0000 7fa69a000640 1 --2- 172.21.15.118:0/1753000363 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fa694101520 0x7fa694075450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:20:52.351 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:20:52.350+0000 7fa69a000640 1 -- 172.21.15.118:0/1753000363 >> 172.21.15.118:0/1753000363 conn(0x7fa6940fcab0 msgr2=0x7fa6941059b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:20:52.351 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:20:52.351+0000 7fa69a000640 1 -- 172.21.15.118:0/1753000363 shutdown_connections 2024-03-24T00:20:52.351 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:20:52.351+0000 7fa69a000640 1 -- 172.21.15.118:0/1753000363 wait complete. 2024-03-24T00:20:52.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:20:52 smithi118 ceph-mon[31535]: mds.? [v2:172.21.15.193:6832/1233062505,v1:172.21.15.193:6833/1233062505] up:reconnect 2024-03-24T00:20:52.697 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:20:52 smithi118 ceph-mon[31535]: mds.? [v2:172.21.15.189:6836/3450187137,v1:172.21.15.189:6837/3450187137] up:reconnect 2024-03-24T00:20:52.697 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:20:52 smithi118 ceph-mon[31535]: fsmap cephfs:5/5 3 up:standby-replay 4 up:standby 2 up:reconnect 3 up:active 2024-03-24T00:20:52.697 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:20:52 smithi118 ceph-mon[31535]: reconnect by client.15159 192.168.0.1:0/3463739313 after 0.0219996 2024-03-24T00:20:52.697 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:20:52 smithi118 ceph-mon[31535]: reconnect by client.15159 192.168.0.1:0/3463739313 after 0.0219996 2024-03-24T00:20:52.697 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:20:52 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T00:20:52.697 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:20:52 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T00:20:52.697 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:20:52 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T00:20:52.697 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:20:52 smithi118 ceph-mon[31535]: pgmap v559: 97 pgs: 97 active+clean; 113 MiB data, 683 MiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 7.4 KiB/s wr, 22 op/s 2024-03-24T00:20:52.704 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:20:52 smithi189 ceph-mon[36187]: mds.? [v2:172.21.15.193:6832/1233062505,v1:172.21.15.193:6833/1233062505] up:reconnect 2024-03-24T00:20:52.704 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:20:52 smithi189 ceph-mon[36187]: mds.? [v2:172.21.15.189:6836/3450187137,v1:172.21.15.189:6837/3450187137] up:reconnect 2024-03-24T00:20:52.704 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:20:52 smithi189 ceph-mon[36187]: fsmap cephfs:5/5 3 up:standby-replay 4 up:standby 2 up:reconnect 3 up:active 2024-03-24T00:20:52.704 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:20:52 smithi189 ceph-mon[36187]: reconnect by client.15159 192.168.0.1:0/3463739313 after 0.0219996 2024-03-24T00:20:52.704 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:20:52 smithi189 ceph-mon[36187]: reconnect by client.15159 192.168.0.1:0/3463739313 after 0.0219996 2024-03-24T00:20:52.704 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:20:52 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T00:20:52.704 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:20:52 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T00:20:52.704 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:20:52 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T00:20:52.704 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:20:52 smithi189 ceph-mon[36187]: pgmap v559: 97 pgs: 97 active+clean; 113 MiB data, 683 MiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 7.4 KiB/s wr, 22 op/s 2024-03-24T00:20:52.710 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:20:52 smithi193 ceph-mon[36878]: mds.? [v2:172.21.15.193:6832/1233062505,v1:172.21.15.193:6833/1233062505] up:reconnect 2024-03-24T00:20:52.710 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:20:52 smithi193 ceph-mon[36878]: mds.? [v2:172.21.15.189:6836/3450187137,v1:172.21.15.189:6837/3450187137] up:reconnect 2024-03-24T00:20:52.711 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:20:52 smithi193 ceph-mon[36878]: fsmap cephfs:5/5 3 up:standby-replay 4 up:standby 2 up:reconnect 3 up:active 2024-03-24T00:20:52.711 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:20:52 smithi193 ceph-mon[36878]: reconnect by client.15159 192.168.0.1:0/3463739313 after 0.0219996 2024-03-24T00:20:52.711 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:20:52 smithi193 ceph-mon[36878]: reconnect by client.15159 192.168.0.1:0/3463739313 after 0.0219996 2024-03-24T00:20:52.711 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:20:52 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T00:20:52.711 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:20:52 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T00:20:52.711 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:20:52 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T00:20:52.711 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:20:52 smithi193 ceph-mon[36878]: pgmap v559: 97 pgs: 97 active+clean; 113 MiB data, 683 MiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 7.4 KiB/s wr, 22 op/s 2024-03-24T00:20:53.555 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:20:53 smithi189 ceph-mon[36187]: mds.? [v2:172.21.15.193:6832/1233062505,v1:172.21.15.193:6833/1233062505] up:rejoin 2024-03-24T00:20:53.555 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:20:53 smithi189 ceph-mon[36187]: mds.? [v2:172.21.15.189:6836/3450187137,v1:172.21.15.189:6837/3450187137] up:rejoin 2024-03-24T00:20:53.555 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:20:53 smithi189 ceph-mon[36187]: fsmap cephfs:5/5 3 up:standby-replay 4 up:standby 2 up:rejoin 3 up:active 2024-03-24T00:20:53.556 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:20:53 smithi189 ceph-mon[36187]: daemon mds.l is now active in filesystem cephfs as rank 3 2024-03-24T00:20:53.556 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:20:53 smithi189 ceph-mon[36187]: daemon mds.h is now active in filesystem cephfs as rank 1 2024-03-24T00:20:53.557 INFO:journalctl@ceph.mds.h.smithi189.stdout:Mar 24 00:20:53 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-h[66671]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: In function 'void Locker::set_xlocks_done(MutationImpl*, bool)' thread 7f917fa17700 time 2024-03-24T00:20:53.476459+0000 2024-03-24T00:20:53.557 INFO:journalctl@ceph.mds.h.smithi189.stdout:Mar 24 00:20:53 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-h[66671]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: 626: FAILED ceph_assert(obj->is_auth()) 2024-03-24T00:20:53.557 INFO:journalctl@ceph.mds.h.smithi189.stdout:Mar 24 00:20:53 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-h[66671]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-24T00:20:53.557 INFO:journalctl@ceph.mds.h.smithi189.stdout:Mar 24 00:20:53 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-h[66671]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x135) [0x7f918a0414a1] 2024-03-24T00:20:53.557 INFO:journalctl@ceph.mds.h.smithi189.stdout:Mar 24 00:20:53 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-h[66671]: 2: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7f918a041667] 2024-03-24T00:20:53.557 INFO:journalctl@ceph.mds.h.smithi189.stdout:Mar 24 00:20:53 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-h[66671]: 3: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x56515efb0223] 2024-03-24T00:20:53.557 INFO:journalctl@ceph.mds.h.smithi189.stdout:Mar 24 00:20:53 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-h[66671]: 4: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x56515ee2a5d6] 2024-03-24T00:20:53.558 INFO:journalctl@ceph.mds.h.smithi189.stdout:Mar 24 00:20:53 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-h[66671]: 5: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x56515ee2ada1] 2024-03-24T00:20:53.558 INFO:journalctl@ceph.mds.h.smithi189.stdout:Mar 24 00:20:53 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-h[66671]: 6: (Server::handle_client_rename(boost::intrusive_ptr const&)+0x259d) [0x56515ee6797d] 2024-03-24T00:20:53.558 INFO:journalctl@ceph.mds.h.smithi189.stdout:Mar 24 00:20:53 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-h[66671]: 7: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x56515ee7d233] 2024-03-24T00:20:53.558 INFO:journalctl@ceph.mds.h.smithi189.stdout:Mar 24 00:20:53 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-h[66671]: 8: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x56515ee7f197] 2024-03-24T00:20:53.558 INFO:journalctl@ceph.mds.h.smithi189.stdout:Mar 24 00:20:53 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-h[66671]: 9: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x56515ed7e11b] 2024-03-24T00:20:53.558 INFO:journalctl@ceph.mds.h.smithi189.stdout:Mar 24 00:20:53 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-h[66671]: 10: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x56515ed7e6bc] 2024-03-24T00:20:53.558 INFO:journalctl@ceph.mds.h.smithi189.stdout:Mar 24 00:20:53 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-h[66671]: 11: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x56515ed681ff] 2024-03-24T00:20:53.558 INFO:journalctl@ceph.mds.h.smithi189.stdout:Mar 24 00:20:53 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-h[66671]: 12: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7f918a2d3488] 2024-03-24T00:20:53.559 INFO:journalctl@ceph.mds.h.smithi189.stdout:Mar 24 00:20:53 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-h[66671]: 13: (DispatchQueue::entry()+0x51f) [0x7f918a2d060f] 2024-03-24T00:20:53.559 INFO:journalctl@ceph.mds.h.smithi189.stdout:Mar 24 00:20:53 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-h[66671]: 14: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f918a3a3e81] 2024-03-24T00:20:53.559 INFO:journalctl@ceph.mds.h.smithi189.stdout:Mar 24 00:20:53 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-h[66671]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7f9188dbe1ca] 2024-03-24T00:20:53.559 INFO:journalctl@ceph.mds.h.smithi189.stdout:Mar 24 00:20:53 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-h[66671]: 16: clone() 2024-03-24T00:20:53.559 INFO:journalctl@ceph.mds.h.smithi189.stdout:Mar 24 00:20:53 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-h[66671]: 2024-03-24T00:20:53.478+0000 7f917fa17700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: In function 'void Locker::set_xlocks_done(MutationImpl*, bool)' thread 7f917fa17700 time 2024-03-24T00:20:53.476459+0000 2024-03-24T00:20:53.559 INFO:journalctl@ceph.mds.h.smithi189.stdout:Mar 24 00:20:53 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-h[66671]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: 626: FAILED ceph_assert(obj->is_auth()) 2024-03-24T00:20:53.559 INFO:journalctl@ceph.mds.h.smithi189.stdout:Mar 24 00:20:53 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-h[66671]: 2024-03-24T00:20:53.559 INFO:journalctl@ceph.mds.h.smithi189.stdout:Mar 24 00:20:53 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-h[66671]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-24T00:20:53.559 INFO:journalctl@ceph.mds.h.smithi189.stdout:Mar 24 00:20:53 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-h[66671]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x135) [0x7f918a0414a1] 2024-03-24T00:20:53.560 INFO:journalctl@ceph.mds.h.smithi189.stdout:Mar 24 00:20:53 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-h[66671]: 2: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7f918a041667] 2024-03-24T00:20:53.560 INFO:journalctl@ceph.mds.h.smithi189.stdout:Mar 24 00:20:53 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-h[66671]: 3: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x56515efb0223] 2024-03-24T00:20:53.560 INFO:journalctl@ceph.mds.h.smithi189.stdout:Mar 24 00:20:53 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-h[66671]: 4: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x56515ee2a5d6] 2024-03-24T00:20:53.560 INFO:journalctl@ceph.mds.h.smithi189.stdout:Mar 24 00:20:53 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-h[66671]: 5: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x56515ee2ada1] 2024-03-24T00:20:53.560 INFO:journalctl@ceph.mds.h.smithi189.stdout:Mar 24 00:20:53 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-h[66671]: 6: (Server::handle_client_rename(boost::intrusive_ptr const&)+0x259d) [0x56515ee6797d] 2024-03-24T00:20:53.560 INFO:journalctl@ceph.mds.h.smithi189.stdout:Mar 24 00:20:53 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-h[66671]: 7: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x56515ee7d233] 2024-03-24T00:20:53.560 INFO:journalctl@ceph.mds.h.smithi189.stdout:Mar 24 00:20:53 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-h[66671]: 8: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x56515ee7f197] 2024-03-24T00:20:53.560 INFO:journalctl@ceph.mds.h.smithi189.stdout:Mar 24 00:20:53 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-h[66671]: 9: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x56515ed7e11b] 2024-03-24T00:20:53.560 INFO:journalctl@ceph.mds.h.smithi189.stdout:Mar 24 00:20:53 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-h[66671]: 10: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x56515ed7e6bc] 2024-03-24T00:20:53.560 INFO:journalctl@ceph.mds.h.smithi189.stdout:Mar 24 00:20:53 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-h[66671]: 11: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x56515ed681ff] 2024-03-24T00:20:53.560 INFO:journalctl@ceph.mds.h.smithi189.stdout:Mar 24 00:20:53 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-h[66671]: 12: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7f918a2d3488] 2024-03-24T00:20:53.560 INFO:journalctl@ceph.mds.h.smithi189.stdout:Mar 24 00:20:53 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-h[66671]: 13: (DispatchQueue::entry()+0x51f) [0x7f918a2d060f] 2024-03-24T00:20:53.560 INFO:journalctl@ceph.mds.h.smithi189.stdout:Mar 24 00:20:53 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-h[66671]: 14: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f918a3a3e81] 2024-03-24T00:20:53.561 INFO:journalctl@ceph.mds.h.smithi189.stdout:Mar 24 00:20:53 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-h[66671]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7f9188dbe1ca] 2024-03-24T00:20:53.561 INFO:journalctl@ceph.mds.h.smithi189.stdout:Mar 24 00:20:53 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-h[66671]: 16: clone() 2024-03-24T00:20:53.561 INFO:journalctl@ceph.mds.h.smithi189.stdout:Mar 24 00:20:53 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-h[66671]: 2024-03-24T00:20:53.561 INFO:journalctl@ceph.mds.h.smithi189.stdout:Mar 24 00:20:53 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-h[66671]: *** Caught signal (Aborted) ** 2024-03-24T00:20:53.561 INFO:journalctl@ceph.mds.h.smithi189.stdout:Mar 24 00:20:53 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-h[66671]: in thread 7f917fa17700 thread_name:ms_dispatch 2024-03-24T00:20:53.561 INFO:journalctl@ceph.mds.h.smithi189.stdout:Mar 24 00:20:53 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-h[66671]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-24T00:20:53.561 INFO:journalctl@ceph.mds.h.smithi189.stdout:Mar 24 00:20:53 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-h[66671]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f9188dc8d20] 2024-03-24T00:20:53.561 INFO:journalctl@ceph.mds.h.smithi189.stdout:Mar 24 00:20:53 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-h[66671]: 2: gsignal() 2024-03-24T00:20:53.561 INFO:journalctl@ceph.mds.h.smithi189.stdout:Mar 24 00:20:53 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-h[66671]: 3: abort() 2024-03-24T00:20:53.561 INFO:journalctl@ceph.mds.h.smithi189.stdout:Mar 24 00:20:53 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-h[66671]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x18f) [0x7f918a0414fb] 2024-03-24T00:20:53.561 INFO:journalctl@ceph.mds.h.smithi189.stdout:Mar 24 00:20:53 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-h[66671]: 5: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7f918a041667] 2024-03-24T00:20:53.561 INFO:journalctl@ceph.mds.h.smithi189.stdout:Mar 24 00:20:53 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-h[66671]: 6: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x56515efb0223] 2024-03-24T00:20:53.561 INFO:journalctl@ceph.mds.h.smithi189.stdout:Mar 24 00:20:53 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-h[66671]: 7: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x56515ee2a5d6] 2024-03-24T00:20:53.561 INFO:journalctl@ceph.mds.h.smithi189.stdout:Mar 24 00:20:53 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-h[66671]: 8: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x56515ee2ada1] 2024-03-24T00:20:53.561 INFO:journalctl@ceph.mds.h.smithi189.stdout:Mar 24 00:20:53 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-h[66671]: 9: (Server::handle_client_rename(boost::intrusive_ptr const&)+0x259d) [0x56515ee6797d] 2024-03-24T00:20:53.562 INFO:journalctl@ceph.mds.h.smithi189.stdout:Mar 24 00:20:53 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-h[66671]: 10: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x56515ee7d233] 2024-03-24T00:20:53.562 INFO:journalctl@ceph.mds.h.smithi189.stdout:Mar 24 00:20:53 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-h[66671]: 11: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x56515ee7f197] 2024-03-24T00:20:53.562 INFO:journalctl@ceph.mds.h.smithi189.stdout:Mar 24 00:20:53 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-h[66671]: 12: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x56515ed7e11b] 2024-03-24T00:20:53.562 INFO:journalctl@ceph.mds.h.smithi189.stdout:Mar 24 00:20:53 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-h[66671]: 13: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x56515ed7e6bc] 2024-03-24T00:20:53.562 INFO:journalctl@ceph.mds.h.smithi189.stdout:Mar 24 00:20:53 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-h[66671]: 14: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x56515ed681ff] 2024-03-24T00:20:53.562 INFO:journalctl@ceph.mds.h.smithi189.stdout:Mar 24 00:20:53 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-h[66671]: 15: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7f918a2d3488] 2024-03-24T00:20:53.562 INFO:journalctl@ceph.mds.h.smithi189.stdout:Mar 24 00:20:53 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-h[66671]: 16: (DispatchQueue::entry()+0x51f) [0x7f918a2d060f] 2024-03-24T00:20:53.562 INFO:journalctl@ceph.mds.h.smithi189.stdout:Mar 24 00:20:53 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-h[66671]: 17: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f918a3a3e81] 2024-03-24T00:20:53.562 INFO:journalctl@ceph.mds.h.smithi189.stdout:Mar 24 00:20:53 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-h[66671]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7f9188dbe1ca] 2024-03-24T00:20:53.562 INFO:journalctl@ceph.mds.h.smithi189.stdout:Mar 24 00:20:53 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-h[66671]: 19: clone() 2024-03-24T00:20:53.562 INFO:journalctl@ceph.mds.h.smithi189.stdout:Mar 24 00:20:53 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-h[66671]: 2024-03-24T00:20:53.480+0000 7f917fa17700 -1 *** Caught signal (Aborted) ** 2024-03-24T00:20:53.562 INFO:journalctl@ceph.mds.h.smithi189.stdout:Mar 24 00:20:53 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-h[66671]: in thread 7f917fa17700 thread_name:ms_dispatch 2024-03-24T00:20:53.562 INFO:journalctl@ceph.mds.h.smithi189.stdout:Mar 24 00:20:53 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-h[66671]: 2024-03-24T00:20:53.562 INFO:journalctl@ceph.mds.h.smithi189.stdout:Mar 24 00:20:53 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-h[66671]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-24T00:20:53.563 INFO:journalctl@ceph.mds.h.smithi189.stdout:Mar 24 00:20:53 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-h[66671]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f9188dc8d20] 2024-03-24T00:20:53.563 INFO:journalctl@ceph.mds.h.smithi189.stdout:Mar 24 00:20:53 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-h[66671]: 2: gsignal() 2024-03-24T00:20:53.563 INFO:journalctl@ceph.mds.h.smithi189.stdout:Mar 24 00:20:53 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-h[66671]: 3: abort() 2024-03-24T00:20:53.563 INFO:journalctl@ceph.mds.h.smithi189.stdout:Mar 24 00:20:53 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-h[66671]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x18f) [0x7f918a0414fb] 2024-03-24T00:20:53.563 INFO:journalctl@ceph.mds.h.smithi189.stdout:Mar 24 00:20:53 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-h[66671]: 5: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7f918a041667] 2024-03-24T00:20:53.564 INFO:journalctl@ceph.mds.h.smithi189.stdout:Mar 24 00:20:53 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-h[66671]: 6: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x56515efb0223] 2024-03-24T00:20:53.564 INFO:journalctl@ceph.mds.h.smithi189.stdout:Mar 24 00:20:53 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-h[66671]: 7: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x56515ee2a5d6] 2024-03-24T00:20:53.564 INFO:journalctl@ceph.mds.h.smithi189.stdout:Mar 24 00:20:53 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-h[66671]: 8: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x56515ee2ada1] 2024-03-24T00:20:53.564 INFO:journalctl@ceph.mds.h.smithi189.stdout:Mar 24 00:20:53 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-h[66671]: 9: (Server::handle_client_rename(boost::intrusive_ptr const&)+0x259d) [0x56515ee6797d] 2024-03-24T00:20:53.564 INFO:journalctl@ceph.mds.h.smithi189.stdout:Mar 24 00:20:53 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-h[66671]: 10: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x56515ee7d233] 2024-03-24T00:20:53.564 INFO:journalctl@ceph.mds.h.smithi189.stdout:Mar 24 00:20:53 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-h[66671]: 11: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x56515ee7f197] 2024-03-24T00:20:53.564 INFO:journalctl@ceph.mds.h.smithi189.stdout:Mar 24 00:20:53 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-h[66671]: 12: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x56515ed7e11b] 2024-03-24T00:20:53.564 INFO:journalctl@ceph.mds.h.smithi189.stdout:Mar 24 00:20:53 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-h[66671]: 13: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x56515ed7e6bc] 2024-03-24T00:20:53.564 INFO:journalctl@ceph.mds.h.smithi189.stdout:Mar 24 00:20:53 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-h[66671]: 14: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x56515ed681ff] 2024-03-24T00:20:53.564 INFO:journalctl@ceph.mds.h.smithi189.stdout:Mar 24 00:20:53 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-h[66671]: 15: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7f918a2d3488] 2024-03-24T00:20:53.564 INFO:journalctl@ceph.mds.h.smithi189.stdout:Mar 24 00:20:53 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-h[66671]: 16: (DispatchQueue::entry()+0x51f) [0x7f918a2d060f] 2024-03-24T00:20:53.564 INFO:journalctl@ceph.mds.h.smithi189.stdout:Mar 24 00:20:53 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-h[66671]: 17: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f918a3a3e81] 2024-03-24T00:20:53.565 INFO:journalctl@ceph.mds.h.smithi189.stdout:Mar 24 00:20:53 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-h[66671]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7f9188dbe1ca] 2024-03-24T00:20:53.565 INFO:journalctl@ceph.mds.h.smithi189.stdout:Mar 24 00:20:53 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-h[66671]: 19: clone() 2024-03-24T00:20:53.565 INFO:journalctl@ceph.mds.h.smithi189.stdout:Mar 24 00:20:53 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-h[66671]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-03-24T00:20:53.565 INFO:journalctl@ceph.mds.h.smithi189.stdout:Mar 24 00:20:53 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-h[66671]: 2024-03-24T00:20:53.565 INFO:journalctl@ceph.mds.h.smithi189.stdout:Mar 24 00:20:53 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-h[66671]: -1> 2024-03-24T00:20:53.478+0000 7f917fa17700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: In function 'void Locker::set_xlocks_done(MutationImpl*, bool)' thread 7f917fa17700 time 2024-03-24T00:20:53.476459+0000 2024-03-24T00:20:53.565 INFO:journalctl@ceph.mds.h.smithi189.stdout:Mar 24 00:20:53 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-h[66671]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: 626: FAILED ceph_assert(obj->is_auth()) 2024-03-24T00:20:53.565 INFO:journalctl@ceph.mds.h.smithi189.stdout:Mar 24 00:20:53 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-h[66671]: 2024-03-24T00:20:53.565 INFO:journalctl@ceph.mds.h.smithi189.stdout:Mar 24 00:20:53 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-h[66671]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-24T00:20:53.565 INFO:journalctl@ceph.mds.h.smithi189.stdout:Mar 24 00:20:53 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-h[66671]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x135) [0x7f918a0414a1] 2024-03-24T00:20:53.565 INFO:journalctl@ceph.mds.h.smithi189.stdout:Mar 24 00:20:53 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-h[66671]: 2: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7f918a041667] 2024-03-24T00:20:53.565 INFO:journalctl@ceph.mds.h.smithi189.stdout:Mar 24 00:20:53 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-h[66671]: 3: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x56515efb0223] 2024-03-24T00:20:53.565 INFO:journalctl@ceph.mds.h.smithi189.stdout:Mar 24 00:20:53 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-h[66671]: 4: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x56515ee2a5d6] 2024-03-24T00:20:53.565 INFO:journalctl@ceph.mds.h.smithi189.stdout:Mar 24 00:20:53 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-h[66671]: 5: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x56515ee2ada1] 2024-03-24T00:20:53.565 INFO:journalctl@ceph.mds.h.smithi189.stdout:Mar 24 00:20:53 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-h[66671]: 6: (Server::handle_client_rename(boost::intrusive_ptr const&)+0x259d) [0x56515ee6797d] 2024-03-24T00:20:53.566 INFO:journalctl@ceph.mds.h.smithi189.stdout:Mar 24 00:20:53 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-h[66671]: 7: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x56515ee7d233] 2024-03-24T00:20:53.566 INFO:journalctl@ceph.mds.h.smithi189.stdout:Mar 24 00:20:53 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-h[66671]: 8: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x56515ee7f197] 2024-03-24T00:20:53.566 INFO:journalctl@ceph.mds.h.smithi189.stdout:Mar 24 00:20:53 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-h[66671]: 9: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x56515ed7e11b] 2024-03-24T00:20:53.566 INFO:journalctl@ceph.mds.h.smithi189.stdout:Mar 24 00:20:53 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-h[66671]: 10: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x56515ed7e6bc] 2024-03-24T00:20:53.566 INFO:journalctl@ceph.mds.h.smithi189.stdout:Mar 24 00:20:53 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-h[66671]: 11: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x56515ed681ff] 2024-03-24T00:20:53.566 INFO:journalctl@ceph.mds.h.smithi189.stdout:Mar 24 00:20:53 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-h[66671]: 12: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7f918a2d3488] 2024-03-24T00:20:53.566 INFO:journalctl@ceph.mds.h.smithi189.stdout:Mar 24 00:20:53 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-h[66671]: 13: (DispatchQueue::entry()+0x51f) [0x7f918a2d060f] 2024-03-24T00:20:53.566 INFO:journalctl@ceph.mds.h.smithi189.stdout:Mar 24 00:20:53 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-h[66671]: 14: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f918a3a3e81] 2024-03-24T00:20:53.566 INFO:journalctl@ceph.mds.h.smithi189.stdout:Mar 24 00:20:53 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-h[66671]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7f9188dbe1ca] 2024-03-24T00:20:53.566 INFO:journalctl@ceph.mds.h.smithi189.stdout:Mar 24 00:20:53 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-h[66671]: 16: clone() 2024-03-24T00:20:53.566 INFO:journalctl@ceph.mds.h.smithi189.stdout:Mar 24 00:20:53 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-h[66671]: 2024-03-24T00:20:53.566 INFO:journalctl@ceph.mds.h.smithi189.stdout:Mar 24 00:20:53 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-h[66671]: 0> 2024-03-24T00:20:53.480+0000 7f917fa17700 -1 *** Caught signal (Aborted) ** 2024-03-24T00:20:53.566 INFO:journalctl@ceph.mds.h.smithi189.stdout:Mar 24 00:20:53 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-h[66671]: in thread 7f917fa17700 thread_name:ms_dispatch 2024-03-24T00:20:53.566 INFO:journalctl@ceph.mds.h.smithi189.stdout:Mar 24 00:20:53 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-h[66671]: 2024-03-24T00:20:53.567 INFO:journalctl@ceph.mds.h.smithi189.stdout:Mar 24 00:20:53 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-h[66671]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-24T00:20:53.567 INFO:journalctl@ceph.mds.h.smithi189.stdout:Mar 24 00:20:53 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-h[66671]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f9188dc8d20] 2024-03-24T00:20:53.567 INFO:journalctl@ceph.mds.h.smithi189.stdout:Mar 24 00:20:53 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-h[66671]: 2: gsignal() 2024-03-24T00:20:53.567 INFO:journalctl@ceph.mds.h.smithi189.stdout:Mar 24 00:20:53 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-h[66671]: 3: abort() 2024-03-24T00:20:53.567 INFO:journalctl@ceph.mds.h.smithi189.stdout:Mar 24 00:20:53 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-h[66671]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x18f) [0x7f918a0414fb] 2024-03-24T00:20:53.567 INFO:journalctl@ceph.mds.h.smithi189.stdout:Mar 24 00:20:53 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-h[66671]: 5: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7f918a041667] 2024-03-24T00:20:53.567 INFO:journalctl@ceph.mds.h.smithi189.stdout:Mar 24 00:20:53 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-h[66671]: 6: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x56515efb0223] 2024-03-24T00:20:53.567 INFO:journalctl@ceph.mds.h.smithi189.stdout:Mar 24 00:20:53 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-h[66671]: 7: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x56515ee2a5d6] 2024-03-24T00:20:53.567 INFO:journalctl@ceph.mds.h.smithi189.stdout:Mar 24 00:20:53 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-h[66671]: 8: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x56515ee2ada1] 2024-03-24T00:20:53.567 INFO:journalctl@ceph.mds.h.smithi189.stdout:Mar 24 00:20:53 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-h[66671]: 9: (Server::handle_client_rename(boost::intrusive_ptr const&)+0x259d) [0x56515ee6797d] 2024-03-24T00:20:53.567 INFO:journalctl@ceph.mds.h.smithi189.stdout:Mar 24 00:20:53 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-h[66671]: 10: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x56515ee7d233] 2024-03-24T00:20:53.567 INFO:journalctl@ceph.mds.h.smithi189.stdout:Mar 24 00:20:53 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-h[66671]: 11: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x56515ee7f197] 2024-03-24T00:20:53.567 INFO:journalctl@ceph.mds.h.smithi189.stdout:Mar 24 00:20:53 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-h[66671]: 12: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x56515ed7e11b] 2024-03-24T00:20:53.567 INFO:journalctl@ceph.mds.h.smithi189.stdout:Mar 24 00:20:53 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-h[66671]: 13: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x56515ed7e6bc] 2024-03-24T00:20:53.568 INFO:journalctl@ceph.mds.h.smithi189.stdout:Mar 24 00:20:53 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-h[66671]: 14: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x56515ed681ff] 2024-03-24T00:20:53.568 INFO:journalctl@ceph.mds.h.smithi189.stdout:Mar 24 00:20:53 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-h[66671]: 15: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7f918a2d3488] 2024-03-24T00:20:53.568 INFO:journalctl@ceph.mds.h.smithi189.stdout:Mar 24 00:20:53 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-h[66671]: 16: (DispatchQueue::entry()+0x51f) [0x7f918a2d060f] 2024-03-24T00:20:53.568 INFO:journalctl@ceph.mds.h.smithi189.stdout:Mar 24 00:20:53 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-h[66671]: 17: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f918a3a3e81] 2024-03-24T00:20:53.568 INFO:journalctl@ceph.mds.h.smithi189.stdout:Mar 24 00:20:53 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-h[66671]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7f9188dbe1ca] 2024-03-24T00:20:53.568 INFO:journalctl@ceph.mds.h.smithi189.stdout:Mar 24 00:20:53 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-h[66671]: 19: clone() 2024-03-24T00:20:53.568 INFO:journalctl@ceph.mds.h.smithi189.stdout:Mar 24 00:20:53 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-h[66671]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-03-24T00:20:53.568 INFO:journalctl@ceph.mds.h.smithi189.stdout:Mar 24 00:20:53 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-h[66671]: 2024-03-24T00:20:53.568 INFO:journalctl@ceph.mds.h.smithi189.stdout:Mar 24 00:20:53 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-h[66671]: -1> 2024-03-24T00:20:53.478+0000 7f917fa17700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: In function 'void Locker::set_xlocks_done(MutationImpl*, bool)' thread 7f917fa17700 time 2024-03-24T00:20:53.476459+0000 2024-03-24T00:20:53.568 INFO:journalctl@ceph.mds.h.smithi189.stdout:Mar 24 00:20:53 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-h[66671]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: 626: FAILED ceph_assert(obj->is_auth()) 2024-03-24T00:20:53.568 INFO:journalctl@ceph.mds.h.smithi189.stdout:Mar 24 00:20:53 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-h[66671]: 2024-03-24T00:20:53.568 INFO:journalctl@ceph.mds.h.smithi189.stdout:Mar 24 00:20:53 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-h[66671]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-24T00:20:53.568 INFO:journalctl@ceph.mds.h.smithi189.stdout:Mar 24 00:20:53 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-h[66671]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x135) [0x7f918a0414a1] 2024-03-24T00:20:53.568 INFO:journalctl@ceph.mds.h.smithi189.stdout:Mar 24 00:20:53 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-h[66671]: 2: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7f918a041667] 2024-03-24T00:20:53.569 INFO:journalctl@ceph.mds.h.smithi189.stdout:Mar 24 00:20:53 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-h[66671]: 3: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x56515efb0223] 2024-03-24T00:20:53.569 INFO:journalctl@ceph.mds.h.smithi189.stdout:Mar 24 00:20:53 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-h[66671]: 4: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x56515ee2a5d6] 2024-03-24T00:20:53.569 INFO:journalctl@ceph.mds.h.smithi189.stdout:Mar 24 00:20:53 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-h[66671]: 5: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x56515ee2ada1] 2024-03-24T00:20:53.569 INFO:journalctl@ceph.mds.h.smithi189.stdout:Mar 24 00:20:53 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-h[66671]: 6: (Server::handle_client_rename(boost::intrusive_ptr const&)+0x259d) [0x56515ee6797d] 2024-03-24T00:20:53.569 INFO:journalctl@ceph.mds.h.smithi189.stdout:Mar 24 00:20:53 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-h[66671]: 7: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x56515ee7d233] 2024-03-24T00:20:53.569 INFO:journalctl@ceph.mds.h.smithi189.stdout:Mar 24 00:20:53 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-h[66671]: 8: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x56515ee7f197] 2024-03-24T00:20:53.569 INFO:journalctl@ceph.mds.h.smithi189.stdout:Mar 24 00:20:53 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-h[66671]: 9: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x56515ed7e11b] 2024-03-24T00:20:53.569 INFO:journalctl@ceph.mds.h.smithi189.stdout:Mar 24 00:20:53 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-h[66671]: 10: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x56515ed7e6bc] 2024-03-24T00:20:53.569 INFO:journalctl@ceph.mds.h.smithi189.stdout:Mar 24 00:20:53 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-h[66671]: 11: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x56515ed681ff] 2024-03-24T00:20:53.569 INFO:journalctl@ceph.mds.h.smithi189.stdout:Mar 24 00:20:53 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-h[66671]: 12: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7f918a2d3488] 2024-03-24T00:20:53.569 INFO:journalctl@ceph.mds.h.smithi189.stdout:Mar 24 00:20:53 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-h[66671]: 13: (DispatchQueue::entry()+0x51f) [0x7f918a2d060f] 2024-03-24T00:20:53.569 INFO:journalctl@ceph.mds.h.smithi189.stdout:Mar 24 00:20:53 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-h[66671]: 14: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f918a3a3e81] 2024-03-24T00:20:53.569 INFO:journalctl@ceph.mds.h.smithi189.stdout:Mar 24 00:20:53 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-h[66671]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7f9188dbe1ca] 2024-03-24T00:20:53.569 INFO:journalctl@ceph.mds.h.smithi189.stdout:Mar 24 00:20:53 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-h[66671]: 16: clone() 2024-03-24T00:20:53.569 INFO:journalctl@ceph.mds.h.smithi189.stdout:Mar 24 00:20:53 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-h[66671]: 2024-03-24T00:20:53.570 INFO:journalctl@ceph.mds.h.smithi189.stdout:Mar 24 00:20:53 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-h[66671]: 0> 2024-03-24T00:20:53.480+0000 7f917fa17700 -1 *** Caught signal (Aborted) ** 2024-03-24T00:20:53.570 INFO:journalctl@ceph.mds.h.smithi189.stdout:Mar 24 00:20:53 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-h[66671]: in thread 7f917fa17700 thread_name:ms_dispatch 2024-03-24T00:20:53.570 INFO:journalctl@ceph.mds.h.smithi189.stdout:Mar 24 00:20:53 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-h[66671]: 2024-03-24T00:20:53.570 INFO:journalctl@ceph.mds.h.smithi189.stdout:Mar 24 00:20:53 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-h[66671]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-24T00:20:53.570 INFO:journalctl@ceph.mds.h.smithi189.stdout:Mar 24 00:20:53 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-h[66671]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f9188dc8d20] 2024-03-24T00:20:53.570 INFO:journalctl@ceph.mds.h.smithi189.stdout:Mar 24 00:20:53 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-h[66671]: 2: gsignal() 2024-03-24T00:20:53.570 INFO:journalctl@ceph.mds.h.smithi189.stdout:Mar 24 00:20:53 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-h[66671]: 3: abort() 2024-03-24T00:20:53.570 INFO:journalctl@ceph.mds.h.smithi189.stdout:Mar 24 00:20:53 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-h[66671]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x18f) [0x7f918a0414fb] 2024-03-24T00:20:53.570 INFO:journalctl@ceph.mds.h.smithi189.stdout:Mar 24 00:20:53 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-h[66671]: 5: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7f918a041667] 2024-03-24T00:20:53.570 INFO:journalctl@ceph.mds.h.smithi189.stdout:Mar 24 00:20:53 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-h[66671]: 6: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x56515efb0223] 2024-03-24T00:20:53.570 INFO:journalctl@ceph.mds.h.smithi189.stdout:Mar 24 00:20:53 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-h[66671]: 7: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x56515ee2a5d6] 2024-03-24T00:20:53.570 INFO:journalctl@ceph.mds.h.smithi189.stdout:Mar 24 00:20:53 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-h[66671]: 8: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x56515ee2ada1] 2024-03-24T00:20:53.570 INFO:journalctl@ceph.mds.h.smithi189.stdout:Mar 24 00:20:53 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-h[66671]: 9: (Server::handle_client_rename(boost::intrusive_ptr const&)+0x259d) [0x56515ee6797d] 2024-03-24T00:20:53.570 INFO:journalctl@ceph.mds.h.smithi189.stdout:Mar 24 00:20:53 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-h[66671]: 10: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x56515ee7d233] 2024-03-24T00:20:53.570 INFO:journalctl@ceph.mds.h.smithi189.stdout:Mar 24 00:20:53 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-h[66671]: 11: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x56515ee7f197] 2024-03-24T00:20:53.571 INFO:journalctl@ceph.mds.h.smithi189.stdout:Mar 24 00:20:53 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-h[66671]: 12: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x56515ed7e11b] 2024-03-24T00:20:53.571 INFO:journalctl@ceph.mds.h.smithi189.stdout:Mar 24 00:20:53 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-h[66671]: 13: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x56515ed7e6bc] 2024-03-24T00:20:53.571 INFO:journalctl@ceph.mds.h.smithi189.stdout:Mar 24 00:20:53 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-h[66671]: 14: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x56515ed681ff] 2024-03-24T00:20:53.571 INFO:journalctl@ceph.mds.h.smithi189.stdout:Mar 24 00:20:53 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-h[66671]: 15: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7f918a2d3488] 2024-03-24T00:20:53.571 INFO:journalctl@ceph.mds.h.smithi189.stdout:Mar 24 00:20:53 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-h[66671]: 16: (DispatchQueue::entry()+0x51f) [0x7f918a2d060f] 2024-03-24T00:20:53.571 INFO:journalctl@ceph.mds.h.smithi189.stdout:Mar 24 00:20:53 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-h[66671]: 17: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f918a3a3e81] 2024-03-24T00:20:53.571 INFO:journalctl@ceph.mds.h.smithi189.stdout:Mar 24 00:20:53 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-h[66671]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7f9188dbe1ca] 2024-03-24T00:20:53.571 INFO:journalctl@ceph.mds.h.smithi189.stdout:Mar 24 00:20:53 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-h[66671]: 19: clone() 2024-03-24T00:20:53.571 INFO:journalctl@ceph.mds.h.smithi189.stdout:Mar 24 00:20:53 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-h[66671]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-03-24T00:20:53.571 INFO:journalctl@ceph.mds.h.smithi189.stdout:Mar 24 00:20:53 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-h[66671]: 2024-03-24T00:20:53.571 INFO:journalctl@ceph.mds.h.smithi189.stdout:Mar 24 00:20:53 smithi189 podman[69545]: 2024-03-24 00:20:53.55574506 +0000 UTC m=+0.025299018 container died 8cde5e7f81746198147a07f104c44245acf3afed80c4f5b597e0a01e660f8672 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-h, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, ceph=True, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.build-date=20240319) 2024-03-24T00:20:53.636 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:20:53 smithi118 ceph-mon[31535]: mds.? [v2:172.21.15.193:6832/1233062505,v1:172.21.15.193:6833/1233062505] up:rejoin 2024-03-24T00:20:53.636 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:20:53 smithi118 ceph-mon[31535]: mds.? [v2:172.21.15.189:6836/3450187137,v1:172.21.15.189:6837/3450187137] up:rejoin 2024-03-24T00:20:53.636 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:20:53 smithi118 ceph-mon[31535]: fsmap cephfs:5/5 3 up:standby-replay 4 up:standby 2 up:rejoin 3 up:active 2024-03-24T00:20:53.636 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:20:53 smithi118 ceph-mon[31535]: daemon mds.l is now active in filesystem cephfs as rank 3 2024-03-24T00:20:53.637 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:20:53 smithi118 ceph-mon[31535]: daemon mds.h is now active in filesystem cephfs as rank 1 2024-03-24T00:20:53.711 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:20:53 smithi193 ceph-mon[36878]: mds.? [v2:172.21.15.193:6832/1233062505,v1:172.21.15.193:6833/1233062505] up:rejoin 2024-03-24T00:20:53.711 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:20:53 smithi193 ceph-mon[36878]: mds.? [v2:172.21.15.189:6836/3450187137,v1:172.21.15.189:6837/3450187137] up:rejoin 2024-03-24T00:20:53.711 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:20:53 smithi193 ceph-mon[36878]: fsmap cephfs:5/5 3 up:standby-replay 4 up:standby 2 up:rejoin 3 up:active 2024-03-24T00:20:53.711 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:20:53 smithi193 ceph-mon[36878]: daemon mds.l is now active in filesystem cephfs as rank 3 2024-03-24T00:20:53.711 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:20:53 smithi193 ceph-mon[36878]: daemon mds.h is now active in filesystem cephfs as rank 1 2024-03-24T00:20:53.711 INFO:journalctl@ceph.mds.l.smithi193.stdout:Mar 24 00:20:53 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-l[71256]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: In function 'void Locker::set_xlocks_done(MutationImpl*, bool)' thread 7fc427a62700 time 2024-03-24T00:20:53.353475+0000 2024-03-24T00:20:53.712 INFO:journalctl@ceph.mds.l.smithi193.stdout:Mar 24 00:20:53 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-l[71256]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: 626: FAILED ceph_assert(obj->is_auth()) 2024-03-24T00:20:53.712 INFO:journalctl@ceph.mds.l.smithi193.stdout:Mar 24 00:20:53 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-l[71256]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-24T00:20:53.712 INFO:journalctl@ceph.mds.l.smithi193.stdout:Mar 24 00:20:53 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-l[71256]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x135) [0x7fc43208c4a1] 2024-03-24T00:20:53.712 INFO:journalctl@ceph.mds.l.smithi193.stdout:Mar 24 00:20:53 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-l[71256]: 2: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7fc43208c667] 2024-03-24T00:20:53.712 INFO:journalctl@ceph.mds.l.smithi193.stdout:Mar 24 00:20:53 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-l[71256]: 3: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x55ce2ed6b223] 2024-03-24T00:20:53.712 INFO:journalctl@ceph.mds.l.smithi193.stdout:Mar 24 00:20:53 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-l[71256]: 4: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x55ce2ebe55d6] 2024-03-24T00:20:53.712 INFO:journalctl@ceph.mds.l.smithi193.stdout:Mar 24 00:20:53 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-l[71256]: 5: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x55ce2ebe5da1] 2024-03-24T00:20:53.712 INFO:journalctl@ceph.mds.l.smithi193.stdout:Mar 24 00:20:53 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-l[71256]: 6: (Server::handle_client_rename(boost::intrusive_ptr const&)+0x259d) [0x55ce2ec2297d] 2024-03-24T00:20:53.712 INFO:journalctl@ceph.mds.l.smithi193.stdout:Mar 24 00:20:53 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-l[71256]: 7: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x55ce2ec38233] 2024-03-24T00:20:53.712 INFO:journalctl@ceph.mds.l.smithi193.stdout:Mar 24 00:20:53 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-l[71256]: 8: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x55ce2ec3a197] 2024-03-24T00:20:53.712 INFO:journalctl@ceph.mds.l.smithi193.stdout:Mar 24 00:20:53 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-l[71256]: 9: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x55ce2eb3911b] 2024-03-24T00:20:53.712 INFO:journalctl@ceph.mds.l.smithi193.stdout:Mar 24 00:20:53 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-l[71256]: 10: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x55ce2eb396bc] 2024-03-24T00:20:53.713 INFO:journalctl@ceph.mds.l.smithi193.stdout:Mar 24 00:20:53 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-l[71256]: 11: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x55ce2eb231ff] 2024-03-24T00:20:53.713 INFO:journalctl@ceph.mds.l.smithi193.stdout:Mar 24 00:20:53 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-l[71256]: 12: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7fc43231e488] 2024-03-24T00:20:53.713 INFO:journalctl@ceph.mds.l.smithi193.stdout:Mar 24 00:20:53 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-l[71256]: 13: (DispatchQueue::entry()+0x51f) [0x7fc43231b60f] 2024-03-24T00:20:53.713 INFO:journalctl@ceph.mds.l.smithi193.stdout:Mar 24 00:20:53 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-l[71256]: 14: (DispatchQueue::DispatchThread::entry()+0x11) [0x7fc4323eee81] 2024-03-24T00:20:53.713 INFO:journalctl@ceph.mds.l.smithi193.stdout:Mar 24 00:20:53 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-l[71256]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7fc430e091ca] 2024-03-24T00:20:53.713 INFO:journalctl@ceph.mds.l.smithi193.stdout:Mar 24 00:20:53 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-l[71256]: 16: clone() 2024-03-24T00:20:53.713 INFO:journalctl@ceph.mds.l.smithi193.stdout:Mar 24 00:20:53 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-l[71256]: 2024-03-24T00:20:53.354+0000 7fc427a62700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: In function 'void Locker::set_xlocks_done(MutationImpl*, bool)' thread 7fc427a62700 time 2024-03-24T00:20:53.353475+0000 2024-03-24T00:20:53.713 INFO:journalctl@ceph.mds.l.smithi193.stdout:Mar 24 00:20:53 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-l[71256]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: 626: FAILED ceph_assert(obj->is_auth()) 2024-03-24T00:20:53.713 INFO:journalctl@ceph.mds.l.smithi193.stdout:Mar 24 00:20:53 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-l[71256]: 2024-03-24T00:20:53.713 INFO:journalctl@ceph.mds.l.smithi193.stdout:Mar 24 00:20:53 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-l[71256]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-24T00:20:53.713 INFO:journalctl@ceph.mds.l.smithi193.stdout:Mar 24 00:20:53 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-l[71256]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x135) [0x7fc43208c4a1] 2024-03-24T00:20:53.713 INFO:journalctl@ceph.mds.l.smithi193.stdout:Mar 24 00:20:53 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-l[71256]: 2: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7fc43208c667] 2024-03-24T00:20:53.713 INFO:journalctl@ceph.mds.l.smithi193.stdout:Mar 24 00:20:53 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-l[71256]: 3: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x55ce2ed6b223] 2024-03-24T00:20:53.713 INFO:journalctl@ceph.mds.l.smithi193.stdout:Mar 24 00:20:53 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-l[71256]: 4: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x55ce2ebe55d6] 2024-03-24T00:20:53.714 INFO:journalctl@ceph.mds.l.smithi193.stdout:Mar 24 00:20:53 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-l[71256]: 5: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x55ce2ebe5da1] 2024-03-24T00:20:53.714 INFO:journalctl@ceph.mds.l.smithi193.stdout:Mar 24 00:20:53 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-l[71256]: 6: (Server::handle_client_rename(boost::intrusive_ptr const&)+0x259d) [0x55ce2ec2297d] 2024-03-24T00:20:53.714 INFO:journalctl@ceph.mds.l.smithi193.stdout:Mar 24 00:20:53 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-l[71256]: 7: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x55ce2ec38233] 2024-03-24T00:20:53.714 INFO:journalctl@ceph.mds.l.smithi193.stdout:Mar 24 00:20:53 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-l[71256]: 8: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x55ce2ec3a197] 2024-03-24T00:20:53.714 INFO:journalctl@ceph.mds.l.smithi193.stdout:Mar 24 00:20:53 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-l[71256]: 9: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x55ce2eb3911b] 2024-03-24T00:20:53.714 INFO:journalctl@ceph.mds.l.smithi193.stdout:Mar 24 00:20:53 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-l[71256]: 10: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x55ce2eb396bc] 2024-03-24T00:20:53.714 INFO:journalctl@ceph.mds.l.smithi193.stdout:Mar 24 00:20:53 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-l[71256]: 11: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x55ce2eb231ff] 2024-03-24T00:20:53.714 INFO:journalctl@ceph.mds.l.smithi193.stdout:Mar 24 00:20:53 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-l[71256]: 12: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7fc43231e488] 2024-03-24T00:20:53.714 INFO:journalctl@ceph.mds.l.smithi193.stdout:Mar 24 00:20:53 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-l[71256]: 13: (DispatchQueue::entry()+0x51f) [0x7fc43231b60f] 2024-03-24T00:20:53.714 INFO:journalctl@ceph.mds.l.smithi193.stdout:Mar 24 00:20:53 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-l[71256]: 14: (DispatchQueue::DispatchThread::entry()+0x11) [0x7fc4323eee81] 2024-03-24T00:20:53.714 INFO:journalctl@ceph.mds.l.smithi193.stdout:Mar 24 00:20:53 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-l[71256]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7fc430e091ca] 2024-03-24T00:20:53.714 INFO:journalctl@ceph.mds.l.smithi193.stdout:Mar 24 00:20:53 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-l[71256]: 16: clone() 2024-03-24T00:20:53.714 INFO:journalctl@ceph.mds.l.smithi193.stdout:Mar 24 00:20:53 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-l[71256]: 2024-03-24T00:20:53.715 INFO:journalctl@ceph.mds.l.smithi193.stdout:Mar 24 00:20:53 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-l[71256]: *** Caught signal (Aborted) ** 2024-03-24T00:20:53.715 INFO:journalctl@ceph.mds.l.smithi193.stdout:Mar 24 00:20:53 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-l[71256]: in thread 7fc427a62700 thread_name:ms_dispatch 2024-03-24T00:20:53.715 INFO:journalctl@ceph.mds.l.smithi193.stdout:Mar 24 00:20:53 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-l[71256]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-24T00:20:53.715 INFO:journalctl@ceph.mds.l.smithi193.stdout:Mar 24 00:20:53 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-l[71256]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7fc430e13d20] 2024-03-24T00:20:53.716 INFO:journalctl@ceph.mds.l.smithi193.stdout:Mar 24 00:20:53 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-l[71256]: 2: gsignal() 2024-03-24T00:20:53.716 INFO:journalctl@ceph.mds.l.smithi193.stdout:Mar 24 00:20:53 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-l[71256]: 3: abort() 2024-03-24T00:20:53.716 INFO:journalctl@ceph.mds.l.smithi193.stdout:Mar 24 00:20:53 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-l[71256]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x18f) [0x7fc43208c4fb] 2024-03-24T00:20:53.716 INFO:journalctl@ceph.mds.l.smithi193.stdout:Mar 24 00:20:53 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-l[71256]: 5: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7fc43208c667] 2024-03-24T00:20:53.716 INFO:journalctl@ceph.mds.l.smithi193.stdout:Mar 24 00:20:53 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-l[71256]: 6: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x55ce2ed6b223] 2024-03-24T00:20:53.716 INFO:journalctl@ceph.mds.l.smithi193.stdout:Mar 24 00:20:53 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-l[71256]: 7: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x55ce2ebe55d6] 2024-03-24T00:20:53.716 INFO:journalctl@ceph.mds.l.smithi193.stdout:Mar 24 00:20:53 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-l[71256]: 8: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x55ce2ebe5da1] 2024-03-24T00:20:53.716 INFO:journalctl@ceph.mds.l.smithi193.stdout:Mar 24 00:20:53 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-l[71256]: 9: (Server::handle_client_rename(boost::intrusive_ptr const&)+0x259d) [0x55ce2ec2297d] 2024-03-24T00:20:53.716 INFO:journalctl@ceph.mds.l.smithi193.stdout:Mar 24 00:20:53 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-l[71256]: 10: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x55ce2ec38233] 2024-03-24T00:20:53.716 INFO:journalctl@ceph.mds.l.smithi193.stdout:Mar 24 00:20:53 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-l[71256]: 11: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x55ce2ec3a197] 2024-03-24T00:20:53.716 INFO:journalctl@ceph.mds.l.smithi193.stdout:Mar 24 00:20:53 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-l[71256]: 12: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x55ce2eb3911b] 2024-03-24T00:20:53.716 INFO:journalctl@ceph.mds.l.smithi193.stdout:Mar 24 00:20:53 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-l[71256]: 13: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x55ce2eb396bc] 2024-03-24T00:20:53.717 INFO:journalctl@ceph.mds.l.smithi193.stdout:Mar 24 00:20:53 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-l[71256]: 14: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x55ce2eb231ff] 2024-03-24T00:20:53.717 INFO:journalctl@ceph.mds.l.smithi193.stdout:Mar 24 00:20:53 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-l[71256]: 15: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7fc43231e488] 2024-03-24T00:20:53.717 INFO:journalctl@ceph.mds.l.smithi193.stdout:Mar 24 00:20:53 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-l[71256]: 16: (DispatchQueue::entry()+0x51f) [0x7fc43231b60f] 2024-03-24T00:20:53.717 INFO:journalctl@ceph.mds.l.smithi193.stdout:Mar 24 00:20:53 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-l[71256]: 17: (DispatchQueue::DispatchThread::entry()+0x11) [0x7fc4323eee81] 2024-03-24T00:20:53.717 INFO:journalctl@ceph.mds.l.smithi193.stdout:Mar 24 00:20:53 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-l[71256]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7fc430e091ca] 2024-03-24T00:20:53.717 INFO:journalctl@ceph.mds.l.smithi193.stdout:Mar 24 00:20:53 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-l[71256]: 19: clone() 2024-03-24T00:20:53.717 INFO:journalctl@ceph.mds.l.smithi193.stdout:Mar 24 00:20:53 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-l[71256]: 2024-03-24T00:20:53.357+0000 7fc427a62700 -1 *** Caught signal (Aborted) ** 2024-03-24T00:20:53.717 INFO:journalctl@ceph.mds.l.smithi193.stdout:Mar 24 00:20:53 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-l[71256]: in thread 7fc427a62700 thread_name:ms_dispatch 2024-03-24T00:20:53.717 INFO:journalctl@ceph.mds.l.smithi193.stdout:Mar 24 00:20:53 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-l[71256]: 2024-03-24T00:20:53.717 INFO:journalctl@ceph.mds.l.smithi193.stdout:Mar 24 00:20:53 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-l[71256]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-24T00:20:53.717 INFO:journalctl@ceph.mds.l.smithi193.stdout:Mar 24 00:20:53 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-l[71256]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7fc430e13d20] 2024-03-24T00:20:53.717 INFO:journalctl@ceph.mds.l.smithi193.stdout:Mar 24 00:20:53 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-l[71256]: 2: gsignal() 2024-03-24T00:20:53.717 INFO:journalctl@ceph.mds.l.smithi193.stdout:Mar 24 00:20:53 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-l[71256]: 3: abort() 2024-03-24T00:20:53.717 INFO:journalctl@ceph.mds.l.smithi193.stdout:Mar 24 00:20:53 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-l[71256]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x18f) [0x7fc43208c4fb] 2024-03-24T00:20:53.718 INFO:journalctl@ceph.mds.l.smithi193.stdout:Mar 24 00:20:53 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-l[71256]: 5: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7fc43208c667] 2024-03-24T00:20:53.718 INFO:journalctl@ceph.mds.l.smithi193.stdout:Mar 24 00:20:53 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-l[71256]: 6: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x55ce2ed6b223] 2024-03-24T00:20:53.718 INFO:journalctl@ceph.mds.l.smithi193.stdout:Mar 24 00:20:53 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-l[71256]: 7: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x55ce2ebe55d6] 2024-03-24T00:20:53.718 INFO:journalctl@ceph.mds.l.smithi193.stdout:Mar 24 00:20:53 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-l[71256]: 8: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x55ce2ebe5da1] 2024-03-24T00:20:53.718 INFO:journalctl@ceph.mds.l.smithi193.stdout:Mar 24 00:20:53 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-l[71256]: 9: (Server::handle_client_rename(boost::intrusive_ptr const&)+0x259d) [0x55ce2ec2297d] 2024-03-24T00:20:53.718 INFO:journalctl@ceph.mds.l.smithi193.stdout:Mar 24 00:20:53 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-l[71256]: 10: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x55ce2ec38233] 2024-03-24T00:20:53.718 INFO:journalctl@ceph.mds.l.smithi193.stdout:Mar 24 00:20:53 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-l[71256]: 11: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x55ce2ec3a197] 2024-03-24T00:20:53.718 INFO:journalctl@ceph.mds.l.smithi193.stdout:Mar 24 00:20:53 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-l[71256]: 12: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x55ce2eb3911b] 2024-03-24T00:20:53.718 INFO:journalctl@ceph.mds.l.smithi193.stdout:Mar 24 00:20:53 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-l[71256]: 13: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x55ce2eb396bc] 2024-03-24T00:20:53.718 INFO:journalctl@ceph.mds.l.smithi193.stdout:Mar 24 00:20:53 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-l[71256]: 14: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x55ce2eb231ff] 2024-03-24T00:20:53.718 INFO:journalctl@ceph.mds.l.smithi193.stdout:Mar 24 00:20:53 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-l[71256]: 15: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7fc43231e488] 2024-03-24T00:20:53.718 INFO:journalctl@ceph.mds.l.smithi193.stdout:Mar 24 00:20:53 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-l[71256]: 16: (DispatchQueue::entry()+0x51f) [0x7fc43231b60f] 2024-03-24T00:20:53.718 INFO:journalctl@ceph.mds.l.smithi193.stdout:Mar 24 00:20:53 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-l[71256]: 17: (DispatchQueue::DispatchThread::entry()+0x11) [0x7fc4323eee81] 2024-03-24T00:20:53.719 INFO:journalctl@ceph.mds.l.smithi193.stdout:Mar 24 00:20:53 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-l[71256]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7fc430e091ca] 2024-03-24T00:20:53.719 INFO:journalctl@ceph.mds.l.smithi193.stdout:Mar 24 00:20:53 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-l[71256]: 19: clone() 2024-03-24T00:20:53.719 INFO:journalctl@ceph.mds.l.smithi193.stdout:Mar 24 00:20:53 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-l[71256]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-03-24T00:20:53.719 INFO:journalctl@ceph.mds.l.smithi193.stdout:Mar 24 00:20:53 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-l[71256]: 2024-03-24T00:20:53.719 INFO:journalctl@ceph.mds.l.smithi193.stdout:Mar 24 00:20:53 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-l[71256]: -1> 2024-03-24T00:20:53.354+0000 7fc427a62700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: In function 'void Locker::set_xlocks_done(MutationImpl*, bool)' thread 7fc427a62700 time 2024-03-24T00:20:53.353475+0000 2024-03-24T00:20:53.719 INFO:journalctl@ceph.mds.l.smithi193.stdout:Mar 24 00:20:53 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-l[71256]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: 626: FAILED ceph_assert(obj->is_auth()) 2024-03-24T00:20:53.719 INFO:journalctl@ceph.mds.l.smithi193.stdout:Mar 24 00:20:53 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-l[71256]: 2024-03-24T00:20:53.719 INFO:journalctl@ceph.mds.l.smithi193.stdout:Mar 24 00:20:53 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-l[71256]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-24T00:20:53.719 INFO:journalctl@ceph.mds.l.smithi193.stdout:Mar 24 00:20:53 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-l[71256]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x135) [0x7fc43208c4a1] 2024-03-24T00:20:53.719 INFO:journalctl@ceph.mds.l.smithi193.stdout:Mar 24 00:20:53 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-l[71256]: 2: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7fc43208c667] 2024-03-24T00:20:53.719 INFO:journalctl@ceph.mds.l.smithi193.stdout:Mar 24 00:20:53 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-l[71256]: 3: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x55ce2ed6b223] 2024-03-24T00:20:53.719 INFO:journalctl@ceph.mds.l.smithi193.stdout:Mar 24 00:20:53 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-l[71256]: 4: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x55ce2ebe55d6] 2024-03-24T00:20:53.719 INFO:journalctl@ceph.mds.l.smithi193.stdout:Mar 24 00:20:53 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-l[71256]: 5: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x55ce2ebe5da1] 2024-03-24T00:20:53.719 INFO:journalctl@ceph.mds.l.smithi193.stdout:Mar 24 00:20:53 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-l[71256]: 6: (Server::handle_client_rename(boost::intrusive_ptr const&)+0x259d) [0x55ce2ec2297d] 2024-03-24T00:20:53.719 INFO:journalctl@ceph.mds.l.smithi193.stdout:Mar 24 00:20:53 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-l[71256]: 7: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x55ce2ec38233] 2024-03-24T00:20:53.719 INFO:journalctl@ceph.mds.l.smithi193.stdout:Mar 24 00:20:53 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-l[71256]: 8: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x55ce2ec3a197] 2024-03-24T00:20:53.720 INFO:journalctl@ceph.mds.l.smithi193.stdout:Mar 24 00:20:53 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-l[71256]: 9: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x55ce2eb3911b] 2024-03-24T00:20:53.720 INFO:journalctl@ceph.mds.l.smithi193.stdout:Mar 24 00:20:53 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-l[71256]: 10: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x55ce2eb396bc] 2024-03-24T00:20:53.720 INFO:journalctl@ceph.mds.l.smithi193.stdout:Mar 24 00:20:53 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-l[71256]: 11: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x55ce2eb231ff] 2024-03-24T00:20:53.720 INFO:journalctl@ceph.mds.l.smithi193.stdout:Mar 24 00:20:53 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-l[71256]: 12: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7fc43231e488] 2024-03-24T00:20:53.720 INFO:journalctl@ceph.mds.l.smithi193.stdout:Mar 24 00:20:53 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-l[71256]: 13: (DispatchQueue::entry()+0x51f) [0x7fc43231b60f] 2024-03-24T00:20:53.720 INFO:journalctl@ceph.mds.l.smithi193.stdout:Mar 24 00:20:53 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-l[71256]: 14: (DispatchQueue::DispatchThread::entry()+0x11) [0x7fc4323eee81] 2024-03-24T00:20:53.720 INFO:journalctl@ceph.mds.l.smithi193.stdout:Mar 24 00:20:53 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-l[71256]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7fc430e091ca] 2024-03-24T00:20:53.720 INFO:journalctl@ceph.mds.l.smithi193.stdout:Mar 24 00:20:53 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-l[71256]: 16: clone() 2024-03-24T00:20:53.720 INFO:journalctl@ceph.mds.l.smithi193.stdout:Mar 24 00:20:53 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-l[71256]: 2024-03-24T00:20:53.720 INFO:journalctl@ceph.mds.l.smithi193.stdout:Mar 24 00:20:53 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-l[71256]: 0> 2024-03-24T00:20:53.357+0000 7fc427a62700 -1 *** Caught signal (Aborted) ** 2024-03-24T00:20:53.720 INFO:journalctl@ceph.mds.l.smithi193.stdout:Mar 24 00:20:53 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-l[71256]: in thread 7fc427a62700 thread_name:ms_dispatch 2024-03-24T00:20:53.720 INFO:journalctl@ceph.mds.l.smithi193.stdout:Mar 24 00:20:53 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-l[71256]: 2024-03-24T00:20:53.720 INFO:journalctl@ceph.mds.l.smithi193.stdout:Mar 24 00:20:53 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-l[71256]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-24T00:20:53.720 INFO:journalctl@ceph.mds.l.smithi193.stdout:Mar 24 00:20:53 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-l[71256]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7fc430e13d20] 2024-03-24T00:20:53.721 INFO:journalctl@ceph.mds.l.smithi193.stdout:Mar 24 00:20:53 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-l[71256]: 2: gsignal() 2024-03-24T00:20:53.721 INFO:journalctl@ceph.mds.l.smithi193.stdout:Mar 24 00:20:53 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-l[71256]: 3: abort() 2024-03-24T00:20:53.721 INFO:journalctl@ceph.mds.l.smithi193.stdout:Mar 24 00:20:53 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-l[71256]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x18f) [0x7fc43208c4fb] 2024-03-24T00:20:53.721 INFO:journalctl@ceph.mds.l.smithi193.stdout:Mar 24 00:20:53 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-l[71256]: 5: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7fc43208c667] 2024-03-24T00:20:53.721 INFO:journalctl@ceph.mds.l.smithi193.stdout:Mar 24 00:20:53 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-l[71256]: 6: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x55ce2ed6b223] 2024-03-24T00:20:53.721 INFO:journalctl@ceph.mds.l.smithi193.stdout:Mar 24 00:20:53 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-l[71256]: 7: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x55ce2ebe55d6] 2024-03-24T00:20:53.721 INFO:journalctl@ceph.mds.l.smithi193.stdout:Mar 24 00:20:53 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-l[71256]: 8: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x55ce2ebe5da1] 2024-03-24T00:20:53.721 INFO:journalctl@ceph.mds.l.smithi193.stdout:Mar 24 00:20:53 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-l[71256]: 9: (Server::handle_client_rename(boost::intrusive_ptr const&)+0x259d) [0x55ce2ec2297d] 2024-03-24T00:20:53.721 INFO:journalctl@ceph.mds.l.smithi193.stdout:Mar 24 00:20:53 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-l[71256]: 10: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x55ce2ec38233] 2024-03-24T00:20:53.721 INFO:journalctl@ceph.mds.l.smithi193.stdout:Mar 24 00:20:53 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-l[71256]: 11: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x55ce2ec3a197] 2024-03-24T00:20:53.721 INFO:journalctl@ceph.mds.l.smithi193.stdout:Mar 24 00:20:53 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-l[71256]: 12: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x55ce2eb3911b] 2024-03-24T00:20:53.721 INFO:journalctl@ceph.mds.l.smithi193.stdout:Mar 24 00:20:53 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-l[71256]: 13: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x55ce2eb396bc] 2024-03-24T00:20:53.721 INFO:journalctl@ceph.mds.l.smithi193.stdout:Mar 24 00:20:53 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-l[71256]: 14: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x55ce2eb231ff] 2024-03-24T00:20:53.721 INFO:journalctl@ceph.mds.l.smithi193.stdout:Mar 24 00:20:53 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-l[71256]: 15: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7fc43231e488] 2024-03-24T00:20:53.721 INFO:journalctl@ceph.mds.l.smithi193.stdout:Mar 24 00:20:53 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-l[71256]: 16: (DispatchQueue::entry()+0x51f) [0x7fc43231b60f] 2024-03-24T00:20:53.722 INFO:journalctl@ceph.mds.l.smithi193.stdout:Mar 24 00:20:53 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-l[71256]: 17: (DispatchQueue::DispatchThread::entry()+0x11) [0x7fc4323eee81] 2024-03-24T00:20:53.722 INFO:journalctl@ceph.mds.l.smithi193.stdout:Mar 24 00:20:53 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-l[71256]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7fc430e091ca] 2024-03-24T00:20:53.722 INFO:journalctl@ceph.mds.l.smithi193.stdout:Mar 24 00:20:53 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-l[71256]: 19: clone() 2024-03-24T00:20:53.722 INFO:journalctl@ceph.mds.l.smithi193.stdout:Mar 24 00:20:53 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-l[71256]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-03-24T00:20:53.722 INFO:journalctl@ceph.mds.l.smithi193.stdout:Mar 24 00:20:53 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-l[71256]: 2024-03-24T00:20:53.722 INFO:journalctl@ceph.mds.l.smithi193.stdout:Mar 24 00:20:53 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-l[71256]: -3> 2024-03-24T00:20:53.354+0000 7fc427a62700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: In function 'void Locker::set_xlocks_done(MutationImpl*, bool)' thread 7fc427a62700 time 2024-03-24T00:20:53.353475+0000 2024-03-24T00:20:53.722 INFO:journalctl@ceph.mds.l.smithi193.stdout:Mar 24 00:20:53 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-l[71256]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: 626: FAILED ceph_assert(obj->is_auth()) 2024-03-24T00:20:53.722 INFO:journalctl@ceph.mds.l.smithi193.stdout:Mar 24 00:20:53 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-l[71256]: 2024-03-24T00:20:53.722 INFO:journalctl@ceph.mds.l.smithi193.stdout:Mar 24 00:20:53 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-l[71256]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-24T00:20:53.722 INFO:journalctl@ceph.mds.l.smithi193.stdout:Mar 24 00:20:53 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-l[71256]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x135) [0x7fc43208c4a1] 2024-03-24T00:20:53.722 INFO:journalctl@ceph.mds.l.smithi193.stdout:Mar 24 00:20:53 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-l[71256]: 2: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7fc43208c667] 2024-03-24T00:20:53.722 INFO:journalctl@ceph.mds.l.smithi193.stdout:Mar 24 00:20:53 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-l[71256]: 3: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x55ce2ed6b223] 2024-03-24T00:20:53.722 INFO:journalctl@ceph.mds.l.smithi193.stdout:Mar 24 00:20:53 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-l[71256]: 4: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x55ce2ebe55d6] 2024-03-24T00:20:53.722 INFO:journalctl@ceph.mds.l.smithi193.stdout:Mar 24 00:20:53 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-l[71256]: 5: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x55ce2ebe5da1] 2024-03-24T00:20:53.722 INFO:journalctl@ceph.mds.l.smithi193.stdout:Mar 24 00:20:53 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-l[71256]: 6: (Server::handle_client_rename(boost::intrusive_ptr const&)+0x259d) [0x55ce2ec2297d] 2024-03-24T00:20:53.723 INFO:journalctl@ceph.mds.l.smithi193.stdout:Mar 24 00:20:53 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-l[71256]: 7: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x55ce2ec38233] 2024-03-24T00:20:53.723 INFO:journalctl@ceph.mds.l.smithi193.stdout:Mar 24 00:20:53 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-l[71256]: 8: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x55ce2ec3a197] 2024-03-24T00:20:53.723 INFO:journalctl@ceph.mds.l.smithi193.stdout:Mar 24 00:20:53 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-l[71256]: 9: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x55ce2eb3911b] 2024-03-24T00:20:53.723 INFO:journalctl@ceph.mds.l.smithi193.stdout:Mar 24 00:20:53 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-l[71256]: 10: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x55ce2eb396bc] 2024-03-24T00:20:53.723 INFO:journalctl@ceph.mds.l.smithi193.stdout:Mar 24 00:20:53 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-l[71256]: 11: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x55ce2eb231ff] 2024-03-24T00:20:53.723 INFO:journalctl@ceph.mds.l.smithi193.stdout:Mar 24 00:20:53 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-l[71256]: 12: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7fc43231e488] 2024-03-24T00:20:53.723 INFO:journalctl@ceph.mds.l.smithi193.stdout:Mar 24 00:20:53 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-l[71256]: 13: (DispatchQueue::entry()+0x51f) [0x7fc43231b60f] 2024-03-24T00:20:53.723 INFO:journalctl@ceph.mds.l.smithi193.stdout:Mar 24 00:20:53 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-l[71256]: 14: (DispatchQueue::DispatchThread::entry()+0x11) [0x7fc4323eee81] 2024-03-24T00:20:53.723 INFO:journalctl@ceph.mds.l.smithi193.stdout:Mar 24 00:20:53 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-l[71256]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7fc430e091ca] 2024-03-24T00:20:53.723 INFO:journalctl@ceph.mds.l.smithi193.stdout:Mar 24 00:20:53 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-l[71256]: 16: clone() 2024-03-24T00:20:53.723 INFO:journalctl@ceph.mds.l.smithi193.stdout:Mar 24 00:20:53 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-l[71256]: 2024-03-24T00:20:53.723 INFO:journalctl@ceph.mds.l.smithi193.stdout:Mar 24 00:20:53 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-l[71256]: -2> 2024-03-24T00:20:53.357+0000 7fc427a62700 -1 *** Caught signal (Aborted) ** 2024-03-24T00:20:53.723 INFO:journalctl@ceph.mds.l.smithi193.stdout:Mar 24 00:20:53 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-l[71256]: in thread 7fc427a62700 thread_name:ms_dispatch 2024-03-24T00:20:53.723 INFO:journalctl@ceph.mds.l.smithi193.stdout:Mar 24 00:20:53 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-l[71256]: 2024-03-24T00:20:53.724 INFO:journalctl@ceph.mds.l.smithi193.stdout:Mar 24 00:20:53 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-l[71256]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-24T00:20:53.724 INFO:journalctl@ceph.mds.l.smithi193.stdout:Mar 24 00:20:53 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-l[71256]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7fc430e13d20] 2024-03-24T00:20:53.724 INFO:journalctl@ceph.mds.l.smithi193.stdout:Mar 24 00:20:53 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-l[71256]: 2: gsignal() 2024-03-24T00:20:53.724 INFO:journalctl@ceph.mds.l.smithi193.stdout:Mar 24 00:20:53 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-l[71256]: 3: abort() 2024-03-24T00:20:53.724 INFO:journalctl@ceph.mds.l.smithi193.stdout:Mar 24 00:20:53 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-l[71256]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x18f) [0x7fc43208c4fb] 2024-03-24T00:20:53.724 INFO:journalctl@ceph.mds.l.smithi193.stdout:Mar 24 00:20:53 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-l[71256]: 5: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7fc43208c667] 2024-03-24T00:20:53.724 INFO:journalctl@ceph.mds.l.smithi193.stdout:Mar 24 00:20:53 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-l[71256]: 6: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x55ce2ed6b223] 2024-03-24T00:20:53.724 INFO:journalctl@ceph.mds.l.smithi193.stdout:Mar 24 00:20:53 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-l[71256]: 7: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x55ce2ebe55d6] 2024-03-24T00:20:53.724 INFO:journalctl@ceph.mds.l.smithi193.stdout:Mar 24 00:20:53 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-l[71256]: 8: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x55ce2ebe5da1] 2024-03-24T00:20:53.724 INFO:journalctl@ceph.mds.l.smithi193.stdout:Mar 24 00:20:53 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-l[71256]: 9: (Server::handle_client_rename(boost::intrusive_ptr const&)+0x259d) [0x55ce2ec2297d] 2024-03-24T00:20:53.724 INFO:journalctl@ceph.mds.l.smithi193.stdout:Mar 24 00:20:53 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-l[71256]: 10: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x55ce2ec38233] 2024-03-24T00:20:53.724 INFO:journalctl@ceph.mds.l.smithi193.stdout:Mar 24 00:20:53 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-l[71256]: 11: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x55ce2ec3a197] 2024-03-24T00:20:53.724 INFO:journalctl@ceph.mds.l.smithi193.stdout:Mar 24 00:20:53 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-l[71256]: 12: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x55ce2eb3911b] 2024-03-24T00:20:53.724 INFO:journalctl@ceph.mds.l.smithi193.stdout:Mar 24 00:20:53 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-l[71256]: 13: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x55ce2eb396bc] 2024-03-24T00:20:53.724 INFO:journalctl@ceph.mds.l.smithi193.stdout:Mar 24 00:20:53 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-l[71256]: 14: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x55ce2eb231ff] 2024-03-24T00:20:53.724 INFO:journalctl@ceph.mds.l.smithi193.stdout:Mar 24 00:20:53 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-l[71256]: 15: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7fc43231e488] 2024-03-24T00:20:53.725 INFO:journalctl@ceph.mds.l.smithi193.stdout:Mar 24 00:20:53 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-l[71256]: 16: (DispatchQueue::entry()+0x51f) [0x7fc43231b60f] 2024-03-24T00:20:53.725 INFO:journalctl@ceph.mds.l.smithi193.stdout:Mar 24 00:20:53 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-l[71256]: 17: (DispatchQueue::DispatchThread::entry()+0x11) [0x7fc4323eee81] 2024-03-24T00:20:53.725 INFO:journalctl@ceph.mds.l.smithi193.stdout:Mar 24 00:20:53 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-l[71256]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7fc430e091ca] 2024-03-24T00:20:53.725 INFO:journalctl@ceph.mds.l.smithi193.stdout:Mar 24 00:20:53 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-l[71256]: 19: clone() 2024-03-24T00:20:53.725 INFO:journalctl@ceph.mds.l.smithi193.stdout:Mar 24 00:20:53 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-l[71256]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-03-24T00:20:53.725 INFO:journalctl@ceph.mds.l.smithi193.stdout:Mar 24 00:20:53 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-l[71256]: 2024-03-24T00:20:53.725 INFO:journalctl@ceph.mds.l.smithi193.stdout:Mar 24 00:20:53 smithi193 podman[71578]: 2024-03-24 00:20:53.420273684 +0000 UTC m=+0.023717048 container died 194e6e07f7e41b4829f11e21b1107d5d88b1246ccbfeeff413c34cb65d828d46 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-l, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , ceph=True, CEPH_POINT_RELEASE=, GIT_CLEAN=True, io.buildah.version=1.33.5, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.build-date=20240319, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS) 2024-03-24T00:20:54.300 INFO:journalctl@ceph.mds.l.smithi193.stdout:Mar 24 00:20:54 smithi193 podman[71578]: 2024-03-24 00:20:54.030542762 +0000 UTC m=+0.633986123 container remove 194e6e07f7e41b4829f11e21b1107d5d88b1246ccbfeeff413c34cb65d828d46 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-l, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, org.label-schema.build-date=20240319, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, CEPH_POINT_RELEASE=, GIT_CLEAN=True, ceph=True, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux ) 2024-03-24T00:20:54.300 INFO:journalctl@ceph.mds.l.smithi193.stdout:Mar 24 00:20:54 smithi193 systemd[1]: ceph-d79a5420-e971-11ee-95cd-87774f69a715@mds.l.service: Main process exited, code=exited, status=134/n/a 2024-03-24T00:20:54.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:20:54 smithi118 ceph-mon[31535]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-03-24T00:20:54.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:20:54 smithi118 ceph-mon[31535]: Cluster is now healthy 2024-03-24T00:20:54.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:20:54 smithi118 ceph-mon[31535]: mds.? [v2:172.21.15.193:6832/1233062505,v1:172.21.15.193:6833/1233062505] up:active 2024-03-24T00:20:54.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:20:54 smithi118 ceph-mon[31535]: mds.? [v2:172.21.15.189:6836/3450187137,v1:172.21.15.189:6837/3450187137] up:active 2024-03-24T00:20:54.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:20:54 smithi118 ceph-mon[31535]: fsmap cephfs:5 3 up:standby-replay 4 up:standby 5 up:active 2024-03-24T00:20:54.447 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:20:54 smithi118 ceph-mon[31535]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-03-24T00:20:54.447 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:20:54 smithi118 ceph-mon[31535]: pgmap v560: 97 pgs: 97 active+clean; 113 MiB data, 684 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 40 KiB/s wr, 57 op/s 2024-03-24T00:20:54.605 INFO:journalctl@ceph.mds.h.smithi189.stdout:Mar 24 00:20:54 smithi189 podman[69545]: 2024-03-24 00:20:54.280129376 +0000 UTC m=+0.749683315 container remove 8cde5e7f81746198147a07f104c44245acf3afed80c4f5b597e0a01e660f8672 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-h, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240319, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, io.buildah.version=1.33.5) 2024-03-24T00:20:54.605 INFO:journalctl@ceph.mds.h.smithi189.stdout:Mar 24 00:20:54 smithi189 systemd[1]: ceph-d79a5420-e971-11ee-95cd-87774f69a715@mds.h.service: Main process exited, code=exited, status=134/n/a 2024-03-24T00:20:54.605 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:20:54 smithi189 ceph-mon[36187]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-03-24T00:20:54.605 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:20:54 smithi189 ceph-mon[36187]: Cluster is now healthy 2024-03-24T00:20:54.606 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:20:54 smithi189 ceph-mon[36187]: mds.? [v2:172.21.15.193:6832/1233062505,v1:172.21.15.193:6833/1233062505] up:active 2024-03-24T00:20:54.606 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:20:54 smithi189 ceph-mon[36187]: mds.? [v2:172.21.15.189:6836/3450187137,v1:172.21.15.189:6837/3450187137] up:active 2024-03-24T00:20:54.606 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:20:54 smithi189 ceph-mon[36187]: fsmap cephfs:5 3 up:standby-replay 4 up:standby 5 up:active 2024-03-24T00:20:54.606 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:20:54 smithi189 ceph-mon[36187]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-03-24T00:20:54.606 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:20:54 smithi189 ceph-mon[36187]: pgmap v560: 97 pgs: 97 active+clean; 113 MiB data, 684 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 40 KiB/s wr, 57 op/s 2024-03-24T00:20:54.710 INFO:journalctl@ceph.mds.l.smithi193.stdout:Mar 24 00:20:54 smithi193 systemd[1]: ceph-d79a5420-e971-11ee-95cd-87774f69a715@mds.l.service: Failed with result 'exit-code'. 2024-03-24T00:20:54.711 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:20:54 smithi193 ceph-mon[36878]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-03-24T00:20:54.711 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:20:54 smithi193 ceph-mon[36878]: Cluster is now healthy 2024-03-24T00:20:54.711 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:20:54 smithi193 ceph-mon[36878]: mds.? [v2:172.21.15.193:6832/1233062505,v1:172.21.15.193:6833/1233062505] up:active 2024-03-24T00:20:54.711 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:20:54 smithi193 ceph-mon[36878]: mds.? [v2:172.21.15.189:6836/3450187137,v1:172.21.15.189:6837/3450187137] up:active 2024-03-24T00:20:54.711 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:20:54 smithi193 ceph-mon[36878]: fsmap cephfs:5 3 up:standby-replay 4 up:standby 5 up:active 2024-03-24T00:20:54.711 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:20:54 smithi193 ceph-mon[36878]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-03-24T00:20:54.711 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:20:54 smithi193 ceph-mon[36878]: pgmap v560: 97 pgs: 97 active+clean; 113 MiB data, 684 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 40 KiB/s wr, 57 op/s 2024-03-24T00:20:54.954 INFO:journalctl@ceph.mds.h.smithi189.stdout:Mar 24 00:20:54 smithi189 systemd[1]: ceph-d79a5420-e971-11ee-95cd-87774f69a715@mds.h.service: Failed with result 'exit-code'. 2024-03-24T00:20:54.954 INFO:journalctl@ceph.mds.h.smithi189.stdout:Mar 24 00:20:54 smithi189 systemd[1]: ceph-d79a5420-e971-11ee-95cd-87774f69a715@mds.h.service: Consumed 1.546s CPU time. 2024-03-24T00:20:56.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:20:55 smithi118 ceph-mon[31535]: pgmap v561: 97 pgs: 97 active+clean; 113 MiB data, 696 MiB used, 1.0 TiB / 1.0 TiB avail; 72 KiB/s rd, 55 KiB/s wr, 90 op/s 2024-03-24T00:20:56.203 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:20:55 smithi189 ceph-mon[36187]: pgmap v561: 97 pgs: 97 active+clean; 113 MiB data, 696 MiB used, 1.0 TiB / 1.0 TiB avail; 72 KiB/s rd, 55 KiB/s wr, 90 op/s 2024-03-24T00:20:56.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:20:55 smithi193 ceph-mon[36878]: pgmap v561: 97 pgs: 97 active+clean; 113 MiB data, 696 MiB used, 1.0 TiB / 1.0 TiB avail; 72 KiB/s rd, 55 KiB/s wr, 90 op/s 2024-03-24T00:20:56.844 INFO:tasks.workunit.client.0.smithi118.stdout: 41 300 200 320 287 852 68 2024-03-24T00:20:58.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:20:57 smithi118 ceph-mon[31535]: pgmap v562: 97 pgs: 97 active+clean; 118 MiB data, 716 MiB used, 1.0 TiB / 1.0 TiB avail; 456 KiB/s rd, 658 KiB/s wr, 196 op/s 2024-03-24T00:20:58.203 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:20:57 smithi189 ceph-mon[36187]: pgmap v562: 97 pgs: 97 active+clean; 118 MiB data, 716 MiB used, 1.0 TiB / 1.0 TiB avail; 456 KiB/s rd, 658 KiB/s wr, 196 op/s 2024-03-24T00:20:58.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:20:57 smithi193 ceph-mon[36878]: pgmap v562: 97 pgs: 97 active+clean; 118 MiB data, 716 MiB used, 1.0 TiB / 1.0 TiB avail; 456 KiB/s rd, 658 KiB/s wr, 196 op/s 2024-03-24T00:21:00.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:20:59 smithi118 ceph-mon[31535]: pgmap v563: 97 pgs: 97 active+clean; 124 MiB data, 720 MiB used, 1.0 TiB / 1.0 TiB avail; 419 KiB/s rd, 1.2 MiB/s wr, 203 op/s 2024-03-24T00:21:00.203 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:20:59 smithi189 ceph-mon[36187]: pgmap v563: 97 pgs: 97 active+clean; 124 MiB data, 720 MiB used, 1.0 TiB / 1.0 TiB avail; 419 KiB/s rd, 1.2 MiB/s wr, 203 op/s 2024-03-24T00:21:00.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:20:59 smithi193 ceph-mon[36878]: pgmap v563: 97 pgs: 97 active+clean; 124 MiB data, 720 MiB used, 1.0 TiB / 1.0 TiB avail; 419 KiB/s rd, 1.2 MiB/s wr, 203 op/s 2024-03-24T00:21:02.066 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:21:02.065+0000 7fd627e00640 1 -- 172.21.15.118:0/736842230 >> [v2:172.21.15.193:6832/1023428412,v1:172.21.15.193:6833/1023428412] conn(0x55c55390a6a0 msgr2=0x55c55390cac0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.193:6832/1023428412 2024-03-24T00:21:02.066 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:21:02.065+0000 7fd613e00640 1 client.26594 ms_handle_refused on v2:172.21.15.193:6832/1023428412 2024-03-24T00:21:02.067 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:21:02.065+0000 7fd627e00640 1 --2- 172.21.15.118:0/736842230 >> [v2:172.21.15.193:6832/1023428412,v1:172.21.15.193:6833/1023428412] conn(0x55c55390a6a0 0x55c55390cac0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-03-24T00:21:02.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:21:01 smithi118 ceph-mon[31535]: pgmap v564: 97 pgs: 97 active+clean; 128 MiB data, 725 MiB used, 1.0 TiB / 1.0 TiB avail; 368 KiB/s rd, 1.4 MiB/s wr, 194 op/s 2024-03-24T00:21:02.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:21:01 smithi189 ceph-mon[36187]: pgmap v564: 97 pgs: 97 active+clean; 128 MiB data, 725 MiB used, 1.0 TiB / 1.0 TiB avail; 368 KiB/s rd, 1.4 MiB/s wr, 194 op/s 2024-03-24T00:21:02.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:21:01 smithi193 ceph-mon[36878]: pgmap v564: 97 pgs: 97 active+clean; 128 MiB data, 725 MiB used, 1.0 TiB / 1.0 TiB avail; 368 KiB/s rd, 1.4 MiB/s wr, 194 op/s 2024-03-24T00:21:04.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:21:03 smithi118 ceph-mon[31535]: pgmap v565: 97 pgs: 97 active+clean; 134 MiB data, 731 MiB used, 1.0 TiB / 1.0 TiB avail; 516 KiB/s rd, 1.9 MiB/s wr, 204 op/s 2024-03-24T00:21:04.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:21:03 smithi189 ceph-mon[36187]: pgmap v565: 97 pgs: 97 active+clean; 134 MiB data, 731 MiB used, 1.0 TiB / 1.0 TiB avail; 516 KiB/s rd, 1.9 MiB/s wr, 204 op/s 2024-03-24T00:21:04.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:21:03 smithi193 ceph-mon[36878]: pgmap v565: 97 pgs: 97 active+clean; 134 MiB data, 731 MiB used, 1.0 TiB / 1.0 TiB avail; 516 KiB/s rd, 1.9 MiB/s wr, 204 op/s 2024-03-24T00:21:04.722 INFO:journalctl@ceph.mds.l.smithi193.stdout:Mar 24 00:21:04 smithi193 systemd[1]: ceph-d79a5420-e971-11ee-95cd-87774f69a715@mds.l.service: Scheduled restart job, restart counter is at 3. 2024-03-24T00:21:04.722 INFO:journalctl@ceph.mds.l.smithi193.stdout:Mar 24 00:21:04 smithi193 systemd[1]: Stopped Ceph mds.l for d79a5420-e971-11ee-95cd-87774f69a715. 2024-03-24T00:21:04.722 INFO:journalctl@ceph.mds.l.smithi193.stdout:Mar 24 00:21:04 smithi193 systemd[1]: Starting Ceph mds.l for d79a5420-e971-11ee-95cd-87774f69a715... 2024-03-24T00:21:04.722 INFO:journalctl@ceph.mds.l.smithi193.stdout:Mar 24 00:21:04 smithi193 podman[71693]: 2024-03-24 00:21:04.599386077 +0000 UTC m=+0.017530477 image pull c664afe56bb01db4f9afa9532c20598e201e664f2e9ba2be50db0d52674a26d2 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc 2024-03-24T00:21:04.955 INFO:journalctl@ceph.mds.h.smithi189.stdout:Mar 24 00:21:04 smithi189 systemd[1]: ceph-d79a5420-e971-11ee-95cd-87774f69a715@mds.h.service: Scheduled restart job, restart counter is at 1. 2024-03-24T00:21:04.955 INFO:journalctl@ceph.mds.h.smithi189.stdout:Mar 24 00:21:04 smithi189 systemd[1]: Stopped Ceph mds.h for d79a5420-e971-11ee-95cd-87774f69a715. 2024-03-24T00:21:04.955 INFO:journalctl@ceph.mds.h.smithi189.stdout:Mar 24 00:21:04 smithi189 systemd[1]: ceph-d79a5420-e971-11ee-95cd-87774f69a715@mds.h.service: Consumed 1.546s CPU time. 2024-03-24T00:21:04.955 INFO:journalctl@ceph.mds.h.smithi189.stdout:Mar 24 00:21:04 smithi189 systemd[1]: Starting Ceph mds.h for d79a5420-e971-11ee-95cd-87774f69a715... 2024-03-24T00:21:04.956 INFO:journalctl@ceph.mds.h.smithi189.stdout:Mar 24 00:21:04 smithi189 podman[69639]: 2024-03-24 00:21:04.841542793 +0000 UTC m=+0.019735138 image pull c664afe56bb01db4f9afa9532c20598e201e664f2e9ba2be50db0d52674a26d2 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc 2024-03-24T00:21:04.956 INFO:journalctl@ceph.mds.h.smithi189.stdout:Mar 24 00:21:04 smithi189 podman[69639]: 2024-03-24 00:21:04.956050692 +0000 UTC m=+0.134243009 container create 30d8025ae60b5197cceebadd800d3c1406438009abbf9ed61785e4b539e8412f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-h, org.label-schema.schema-version=1.0, GIT_CLEAN=True, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240319, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5) 2024-03-24T00:21:04.979 INFO:journalctl@ceph.mds.l.smithi193.stdout:Mar 24 00:21:04 smithi193 podman[71693]: 2024-03-24 00:21:04.722975512 +0000 UTC m=+0.141119893 container create 2dd1e6ecd73ce6600b5bd2ff530b6697f813ec5675c7b76ac4db2c015e81cd35 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-l, org.label-schema.build-date=20240319, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, GIT_CLEAN=True, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.vendor=CentOS) 2024-03-24T00:21:04.979 INFO:journalctl@ceph.mds.l.smithi193.stdout:Mar 24 00:21:04 smithi193 podman[71693]: 2024-03-24 00:21:04.95090874 +0000 UTC m=+0.369053118 container init 2dd1e6ecd73ce6600b5bd2ff530b6697f813ec5675c7b76ac4db2c015e81cd35 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-l, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, io.buildah.version=1.33.5, GIT_CLEAN=True, ceph=True, org.label-schema.license=GPLv2, org.label-schema.build-date=20240319) 2024-03-24T00:21:04.979 INFO:journalctl@ceph.mds.l.smithi193.stdout:Mar 24 00:21:04 smithi193 podman[71693]: 2024-03-24 00:21:04.953582829 +0000 UTC m=+0.371727217 container start 2dd1e6ecd73ce6600b5bd2ff530b6697f813ec5675c7b76ac4db2c015e81cd35 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-l, ceph=True, GIT_CLEAN=True, org.label-schema.build-date=20240319, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_BRANCH=HEAD) 2024-03-24T00:21:05.222 INFO:journalctl@ceph.mds.h.smithi189.stdout:Mar 24 00:21:05 smithi189 podman[69639]: 2024-03-24 00:21:05.184204527 +0000 UTC m=+0.362396863 container init 30d8025ae60b5197cceebadd800d3c1406438009abbf9ed61785e4b539e8412f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-h, org.label-schema.build-date=20240319, CEPH_POINT_RELEASE=, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, GIT_CLEAN=True, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-24T00:21:05.222 INFO:journalctl@ceph.mds.h.smithi189.stdout:Mar 24 00:21:05 smithi189 podman[69639]: 2024-03-24 00:21:05.187320874 +0000 UTC m=+0.365513204 container start 30d8025ae60b5197cceebadd800d3c1406438009abbf9ed61785e4b539e8412f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-h, io.buildah.version=1.33.5, GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240319, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, maintainer=Guillaume Abrioux ) 2024-03-24T00:21:05.222 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:21:05 smithi189 ceph-mon[36187]: daemon mds.l restarted 2024-03-24T00:21:05.222 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:21:05 smithi189 ceph-mon[36187]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-03-24T00:21:05.222 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:21:05 smithi189 ceph-mon[36187]: osdmap e97: 12 total, 12 up, 12 in 2024-03-24T00:21:05.222 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:21:05 smithi189 ceph-mon[36187]: mds.? [v2:172.21.15.193:6832/4276594827,v1:172.21.15.193:6833/4276594827] up:boot 2024-03-24T00:21:05.222 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:21:05 smithi189 ceph-mon[36187]: Standby daemon mds.g assigned to filesystem cephfs as rank 3 2024-03-24T00:21:05.223 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:21:05 smithi189 ceph-mon[36187]: fsmap cephfs:4/5 {0=b=up:active,1=h=up:active,2=c=up:active,4=e=up:active} 5 up:standby-replay 3 up:standby, 1 failed 2024-03-24T00:21:05.223 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:21:05 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "l"}]: dispatch 2024-03-24T00:21:05.223 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:21:05 smithi189 ceph-mon[36187]: fsmap cephfs:5/5 4 up:standby-replay 3 up:standby 1 up:replay 4 up:active 2024-03-24T00:21:05.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:21:05 smithi118 ceph-mon[31535]: daemon mds.l restarted 2024-03-24T00:21:05.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:21:05 smithi118 ceph-mon[31535]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-03-24T00:21:05.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:21:05 smithi118 ceph-mon[31535]: osdmap e97: 12 total, 12 up, 12 in 2024-03-24T00:21:05.447 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:21:05 smithi118 ceph-mon[31535]: mds.? [v2:172.21.15.193:6832/4276594827,v1:172.21.15.193:6833/4276594827] up:boot 2024-03-24T00:21:05.447 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:21:05 smithi118 ceph-mon[31535]: Standby daemon mds.g assigned to filesystem cephfs as rank 3 2024-03-24T00:21:05.447 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:21:05 smithi118 ceph-mon[31535]: fsmap cephfs:4/5 {0=b=up:active,1=h=up:active,2=c=up:active,4=e=up:active} 5 up:standby-replay 3 up:standby, 1 failed 2024-03-24T00:21:05.447 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:21:05 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "l"}]: dispatch 2024-03-24T00:21:05.447 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:21:05 smithi118 ceph-mon[31535]: fsmap cephfs:5/5 4 up:standby-replay 3 up:standby 1 up:replay 4 up:active 2024-03-24T00:21:05.460 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:21:05 smithi193 ceph-mon[36878]: daemon mds.l restarted 2024-03-24T00:21:05.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:21:05 smithi193 ceph-mon[36878]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-03-24T00:21:05.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:21:05 smithi193 ceph-mon[36878]: osdmap e97: 12 total, 12 up, 12 in 2024-03-24T00:21:05.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:21:05 smithi193 ceph-mon[36878]: mds.? [v2:172.21.15.193:6832/4276594827,v1:172.21.15.193:6833/4276594827] up:boot 2024-03-24T00:21:05.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:21:05 smithi193 ceph-mon[36878]: Standby daemon mds.g assigned to filesystem cephfs as rank 3 2024-03-24T00:21:05.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:21:05 smithi193 ceph-mon[36878]: fsmap cephfs:4/5 {0=b=up:active,1=h=up:active,2=c=up:active,4=e=up:active} 5 up:standby-replay 3 up:standby, 1 failed 2024-03-24T00:21:05.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:21:05 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "l"}]: dispatch 2024-03-24T00:21:05.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:21:05 smithi193 ceph-mon[36878]: fsmap cephfs:5/5 4 up:standby-replay 3 up:standby 1 up:replay 4 up:active 2024-03-24T00:21:05.461 INFO:journalctl@ceph.mds.l.smithi193.stdout:Mar 24 00:21:04 smithi193 ceph-mds[71712]: -- 172.21.15.193:0/978074317 <== mon.1 v2:172.21.15.193:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x563c5cdd5a40 con 0x563c5cda9000 2024-03-24T00:21:05.461 INFO:journalctl@ceph.mds.l.smithi193.stdout:Mar 24 00:21:04 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-l[71708]: starting mds.l at 2024-03-24T00:21:05.461 INFO:journalctl@ceph.mds.l.smithi193.stdout:Mar 24 00:21:05 smithi193 bash[71693]: 2dd1e6ecd73ce6600b5bd2ff530b6697f813ec5675c7b76ac4db2c015e81cd35 2024-03-24T00:21:05.461 INFO:journalctl@ceph.mds.l.smithi193.stdout:Mar 24 00:21:05 smithi193 systemd[1]: Started Ceph mds.l for d79a5420-e971-11ee-95cd-87774f69a715. 2024-03-24T00:21:05.499 INFO:journalctl@ceph.mds.h.smithi189.stdout:Mar 24 00:21:05 smithi189 ceph-mds[69657]: -- 172.21.15.189:0/859973900 <== mon.2 v2:172.21.15.189:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x560788313a40 con 0x5607882e7000 2024-03-24T00:21:05.499 INFO:journalctl@ceph.mds.h.smithi189.stdout:Mar 24 00:21:05 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-h[69653]: starting mds.h at 2024-03-24T00:21:05.499 INFO:journalctl@ceph.mds.h.smithi189.stdout:Mar 24 00:21:05 smithi189 bash[69639]: 30d8025ae60b5197cceebadd800d3c1406438009abbf9ed61785e4b539e8412f 2024-03-24T00:21:05.499 INFO:journalctl@ceph.mds.h.smithi189.stdout:Mar 24 00:21:05 smithi189 systemd[1]: Started Ceph mds.h for d79a5420-e971-11ee-95cd-87774f69a715. 2024-03-24T00:21:05.501 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:21:05.500+0000 7f5fb7400640 1 -- 172.21.15.118:0/1514406428 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f5fb01028c0 msgr2=0x7f5fb0102d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:21:05.501 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:21:05.500+0000 7f5fb7400640 1 --2- 172.21.15.118:0/1514406428 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f5fb01028c0 0x7f5fb0102d40 secure :-1 s=READY pgs=420 cs=0 l=1 rev1=1 crypto rx=0x7f5f9c009900 tx=0x7f5f9c02bc30 comp rx=0 tx=0).stop 2024-03-24T00:21:05.501 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:21:05.500+0000 7f5fb7400640 1 -- 172.21.15.118:0/1514406428 shutdown_connections 2024-03-24T00:21:05.502 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:21:05.500+0000 7f5fb7400640 1 --2- 172.21.15.118:0/1514406428 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5fb0103280 0x7f5fb0109b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:21:05.502 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:21:05.500+0000 7f5fb7400640 1 --2- 172.21.15.118:0/1514406428 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f5fb01028c0 0x7f5fb0102d40 unknown :-1 s=CLOSED pgs=420 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:21:05.502 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:21:05.500+0000 7f5fb7400640 1 --2- 172.21.15.118:0/1514406428 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f5fb01016d0 0x7f5fb0101af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:21:05.502 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:21:05.500+0000 7f5fb7400640 1 -- 172.21.15.118:0/1514406428 >> 172.21.15.118:0/1514406428 conn(0x7f5fb00fcc60 msgr2=0x7f5fb00ff080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:21:05.502 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:21:05.500+0000 7f5fb7400640 1 -- 172.21.15.118:0/1514406428 shutdown_connections 2024-03-24T00:21:05.502 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:21:05.500+0000 7f5fb7400640 1 -- 172.21.15.118:0/1514406428 wait complete. 2024-03-24T00:21:05.503 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:21:05.501+0000 7f5fb7400640 1 Processor -- start 2024-03-24T00:21:05.503 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:21:05.501+0000 7f5fb7400640 1 -- start start 2024-03-24T00:21:05.503 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:21:05.502+0000 7f5fb7400640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5fb01016d0 0x7f5fb0075450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:21:05.503 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:21:05.502+0000 7f5fb7400640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f5fb01028c0 0x7f5fb0073a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:21:05.503 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:21:05.502+0000 7f5fb7400640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f5fb0103280 0x7f5fb0073fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:21:05.504 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:21:05.502+0000 7f5fb7400640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5fb010d3a0 con 0x7f5fb0103280 2024-03-24T00:21:05.504 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:21:05.502+0000 7f5fb7400640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5fb010d220 con 0x7f5fb01016d0 2024-03-24T00:21:05.504 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:21:05.502+0000 7f5fb7400640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5fb010d520 con 0x7f5fb01028c0 2024-03-24T00:21:05.504 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:21:05.502+0000 7f5fb4a00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5fb01016d0 0x7f5fb0075450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:21:05.504 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:21:05.502+0000 7f5fafe00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f5fb01028c0 0x7f5fb0073a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:21:05.504 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:21:05.502+0000 7f5fafe00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f5fb01028c0 0x7f5fb0073a90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.118:38146/0 (socket says 172.21.15.118:38146) 2024-03-24T00:21:05.504 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:21:05.502+0000 7f5fafe00640 1 -- 172.21.15.118:0/1931139018 learned_addr learned my addr 172.21.15.118:0/1931139018 (peer_addr_for_me v2:172.21.15.118:0/0) 2024-03-24T00:21:05.504 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:21:05.502+0000 7f5fb5400640 1 --2- 172.21.15.118:0/1931139018 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f5fb0103280 0x7f5fb0073fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:21:05.504 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:21:05.503+0000 7f5fafe00640 1 -- 172.21.15.118:0/1931139018 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5fb01016d0 msgr2=0x7f5fb0075450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:21:05.504 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:21:05.503+0000 7f5fafe00640 1 --2- 172.21.15.118:0/1931139018 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5fb01016d0 0x7f5fb0075450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:21:05.504 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:21:05.503+0000 7f5fafe00640 1 -- 172.21.15.118:0/1931139018 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f5fb0103280 msgr2=0x7f5fb0073fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:21:05.504 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:21:05.503+0000 7f5fafe00640 1 --2- 172.21.15.118:0/1931139018 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f5fb0103280 0x7f5fb0073fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:21:05.504 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:21:05.503+0000 7f5fafe00640 1 -- 172.21.15.118:0/1931139018 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5fb0074860 con 0x7f5fb01028c0 2024-03-24T00:21:05.505 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:21:05.503+0000 7f5fafe00640 1 --2- 172.21.15.118:0/1931139018 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f5fb01028c0 0x7f5fb0073a90 secure :-1 s=READY pgs=421 cs=0 l=1 rev1=1 crypto rx=0x7f5f9c000c00 tx=0x7f5f9c02c650 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:21:05.505 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:21:05.503+0000 7f5fad600640 1 -- 172.21.15.118:0/1931139018 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5f9c043020 con 0x7f5fb01028c0 2024-03-24T00:21:05.505 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:21:05.503+0000 7f5fad600640 1 -- 172.21.15.118:0/1931139018 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f5f9c0054e0 con 0x7f5fb01028c0 2024-03-24T00:21:05.505 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:21:05.503+0000 7f5fad600640 1 -- 172.21.15.118:0/1931139018 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5f9c003c40 con 0x7f5fb01028c0 2024-03-24T00:21:05.506 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:21:05.503+0000 7f5fb7400640 1 -- 172.21.15.118:0/1931139018 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5fb01a1a10 con 0x7f5fb01028c0 2024-03-24T00:21:05.506 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:21:05.504+0000 7f5fb7400640 1 -- 172.21.15.118:0/1931139018 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5fb01a1f60 con 0x7f5fb01028c0 2024-03-24T00:21:05.508 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:21:05.506+0000 7f5fba19e740 1 Processor -- start 2024-03-24T00:21:05.508 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:21:05.507+0000 7f5fad600640 1 -- 172.21.15.118:0/1931139018 <== mon.1 v2:172.21.15.193:3300/0 4 ==== mgrmap(e 20) v1 ==== 137291+0+0 (secure 0 0 0) 0x7f5f9c046020 con 0x7f5fb01028c0 2024-03-24T00:21:05.508 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:21:05.507+0000 7f5fba19e740 1 -- start start 2024-03-24T00:21:05.508 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:21:05.507+0000 7f5fba19e740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5634260861e0 0x563426086600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:21:05.508 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:21:05.507+0000 7f5fba19e740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x563426086bd0 0x56342608b6d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:21:05.508 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:21:05.507+0000 7f5fba19e740 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x56342608bd40 0x56342608e1d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:21:05.508 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:21:05.507+0000 7f5fba19e740 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x5634260493b0 con 0x56342608bd40 2024-03-24T00:21:05.508 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:21:05.507+0000 7f5fba19e740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x563425feb830 con 0x5634260861e0 2024-03-24T00:21:05.509 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:21:05.507+0000 7f5fba19e740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x563425f945d0 con 0x563426086bd0 2024-03-24T00:21:05.509 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:21:05.507+0000 7f5fafe00640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x56342608bd40 0x56342608e1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:21:05.509 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:21:05.507+0000 7f5fb5400640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5634260861e0 0x563426086600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:21:05.509 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:21:05.507+0000 7f5fafe00640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x56342608bd40 0x56342608e1d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.118:3300/0 says I am v2:172.21.15.118:58028/0 (socket says 172.21.15.118:58028) 2024-03-24T00:21:05.509 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:21:05.507+0000 7f5fb4a00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x563426086bd0 0x56342608b6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:21:05.509 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:21:05.507+0000 7f5fafe00640 1 -- 172.21.15.118:0/3560183651 learned_addr learned my addr 172.21.15.118:0/3560183651 (peer_addr_for_me v2:172.21.15.118:0/0) 2024-03-24T00:21:05.511 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:21:05.508+0000 7f5fb5400640 1 -- 172.21.15.118:0/3560183651 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x563426086bd0 msgr2=0x56342608b6d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:21:05.511 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:21:05.508+0000 7f5fb5400640 1 --2- 172.21.15.118:0/3560183651 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x563426086bd0 0x56342608b6d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:21:05.511 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:21:05.508+0000 7f5fb5400640 1 -- 172.21.15.118:0/3560183651 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x56342608bd40 msgr2=0x56342608e1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:21:05.511 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:21:05.508+0000 7f5fb5400640 1 --2- 172.21.15.118:0/3560183651 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x56342608bd40 0x56342608e1d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:21:05.511 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:21:05.508+0000 7f5fb5400640 1 -- 172.21.15.118:0/3560183651 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x563425fac2c0 con 0x5634260861e0 2024-03-24T00:21:05.511 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:21:05.508+0000 7f5fb5400640 1 --2- 172.21.15.118:0/3560183651 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5634260861e0 0x563426086600 secure :-1 s=READY pgs=416 cs=0 l=1 rev1=1 crypto rx=0x7f5fa8009f00 tx=0x7f5fa8011810 comp rx=0 tx=0).ready entity=mon.2 client_cookie=f9794084a914877b server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:21:05.511 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:21:05.508+0000 7f5fa1c00640 1 -- 172.21.15.118:0/3560183651 <== mon.2 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5fa8004d20 con 0x5634260861e0 2024-03-24T00:21:05.512 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:21:05.509+0000 7f5fa1c00640 1 -- 172.21.15.118:0/3560183651 <== mon.2 v2:172.21.15.189:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f5fa8004030 con 0x5634260861e0 2024-03-24T00:21:05.512 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:21:05.509+0000 7f5fa1c00640 1 -- 172.21.15.118:0/3560183651 <== mon.2 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5fa8005260 con 0x5634260861e0 2024-03-24T00:21:05.512 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:21:05.509+0000 7f5fba19e740 1 -- 172.21.15.118:0/3560183651 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5634260861e0 msgr2=0x563426086600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:21:05.512 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:21:05.509+0000 7f5fba19e740 1 --2- 172.21.15.118:0/3560183651 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5634260861e0 0x563426086600 secure :-1 s=READY pgs=416 cs=0 l=1 rev1=1 crypto rx=0x7f5fa8009f00 tx=0x7f5fa8011810 comp rx=0 tx=0).stop 2024-03-24T00:21:05.512 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:21:05.509+0000 7f5fad600640 1 --2- 172.21.15.118:0/1931139018 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7f5f980a28a0 0x7f5f980a4d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:21:05.512 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:21:05.509+0000 7f5fad600640 1 -- 172.21.15.118:0/1931139018 <== mon.1 v2:172.21.15.193:3300/0 5 ==== osd_map(98..98 src has 1..98) v4 ==== 8965+0+0 (secure 0 0 0) 0x7f5f9c0f0d80 con 0x7f5fb01028c0 2024-03-24T00:21:05.512 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:21:05.509+0000 7f5fba19e740 1 -- 172.21.15.118:0/3560183651 shutdown_connections 2024-03-24T00:21:05.512 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:21:05.510+0000 7f5fba19e740 1 --2- 172.21.15.118:0/3560183651 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x56342608bd40 0x56342608e1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:21:05.512 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:21:05.510+0000 7f5fba19e740 1 --2- 172.21.15.118:0/3560183651 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x563426086bd0 0x56342608b6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:21:05.512 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:21:05.510+0000 7f5fba19e740 1 --2- 172.21.15.118:0/3560183651 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5634260861e0 0x563426086600 unknown :-1 s=CLOSED pgs=416 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:21:05.512 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:21:05.510+0000 7f5fba19e740 1 -- 172.21.15.118:0/3560183651 >> 172.21.15.118:0/3560183651 conn(0x563426081750 msgr2=0x563426081b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:21:05.512 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:21:05.510+0000 7f5fb5400640 1 --2- 172.21.15.118:0/1931139018 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7f5f980a28a0 0x7f5f980a4d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:21:05.512 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:21:05.510+0000 7f5fba19e740 1 -- 172.21.15.118:0/3560183651 shutdown_connections 2024-03-24T00:21:05.512 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:21:05.510+0000 7f5fba19e740 1 -- 172.21.15.118:0/3560183651 wait complete. 2024-03-24T00:21:05.513 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:21:05.510+0000 7f5fba19e740 1 -- start start 2024-03-24T00:21:05.513 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:21:05.510+0000 7f5fb5400640 1 --2- 172.21.15.118:0/1931139018 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7f5f980a28a0 0x7f5f980a4d60 secure :-1 s=READY pgs=360 cs=0 l=1 rev1=1 crypto rx=0x7f5fa8009d40 tx=0x7f5fa8004870 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:21:05.513 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:21:05.511+0000 7f5fba19e740 1 Processor -- start 2024-03-24T00:21:05.517 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:21:05.511+0000 7f5fba19e740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5634260861e0 0x563426090380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:21:05.517 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:21:05.511+0000 7f5fba19e740 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x563426086bd0 0x5634260908c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:21:05.517 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:21:05.511+0000 7f5fba19e740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56342608bd40 0x563426097900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:21:05.517 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:21:05.511+0000 7f5fba19e740 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x563425fa7a20 con 0x563426086bd0 2024-03-24T00:21:05.517 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:21:05.511+0000 7f5fba19e740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x563425fe78d0 con 0x56342608bd40 2024-03-24T00:21:05.517 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:21:05.511+0000 7f5fba19e740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x56342608ecf0 con 0x5634260861e0 2024-03-24T00:21:05.517 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:21:05.511+0000 7f5fba19e740 10 client.0 _ll_register_callbacks cb 0x563425fe41c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T00:21:05.517 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:21:05.511+0000 7f5fb4a00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5634260861e0 0x563426090380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:21:05.517 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:21:05.511+0000 7f5fafe00640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x563426086bd0 0x5634260908c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:21:05.518 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:21:05.511+0000 7f5fb4a00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5634260861e0 0x563426090380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.118:38162/0 (socket says 172.21.15.118:38162) 2024-03-24T00:21:05.518 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:21:05.511+0000 7f5fb4a00640 1 -- 172.21.15.118:0/854929295 learned_addr learned my addr 172.21.15.118:0/854929295 (peer_addr_for_me v2:172.21.15.118:0/0) 2024-03-24T00:21:05.518 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:21:05.511+0000 7f5fb5400640 1 --2- 172.21.15.118:0/854929295 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56342608bd40 0x563426097900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:21:05.518 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:21:05.512+0000 7f5fb4a00640 1 -- 172.21.15.118:0/854929295 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56342608bd40 msgr2=0x563426097900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:21:05.518 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:21:05.512+0000 7f5fb4a00640 1 --2- 172.21.15.118:0/854929295 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56342608bd40 0x563426097900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:21:05.519 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:21:05.512+0000 7f5fb4a00640 1 -- 172.21.15.118:0/854929295 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x563426086bd0 msgr2=0x5634260908c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:21:05.519 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:21:05.512+0000 7f5fb4a00640 1 --2- 172.21.15.118:0/854929295 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x563426086bd0 0x5634260908c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:21:05.519 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:21:05.512+0000 7f5fb4a00640 1 -- 172.21.15.118:0/854929295 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x563425d00210 con 0x5634260861e0 2024-03-24T00:21:05.519 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:21:05.512+0000 7f5fb4a00640 1 --2- 172.21.15.118:0/854929295 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5634260861e0 0x563426090380 secure :-1 s=READY pgs=422 cs=0 l=1 rev1=1 crypto rx=0x7f5fa4012700 tx=0x7f5fa4012b00 comp rx=0 tx=0).ready entity=mon.1 client_cookie=9339e4492e91864 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:21:05.519 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:21:05.512+0000 7f5f96000640 10 client.0 ms_handle_connect on v2:172.21.15.193:3300/0 2024-03-24T00:21:05.519 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:21:05.512+0000 7f5f96000640 1 -- 172.21.15.118:0/854929295 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5fa400f560 con 0x5634260861e0 2024-03-24T00:21:05.519 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:21:05.512+0000 7f5f96000640 1 -- 172.21.15.118:0/854929295 <== mon.1 v2:172.21.15.193:3300/0 2 ==== osd_map(98..98 src has 1..98) v4 ==== 8965+0+0 (secure 0 0 0) 0x7f5fa400f700 con 0x5634260861e0 2024-03-24T00:21:05.520 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:21:05.512+0000 7f5fba19e740 1 -- 172.21.15.118:0/854929295 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f5fa8004030 con 0x5634260861e0 2024-03-24T00:21:05.520 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:21:05.513+0000 7f5f96000640 1 -- 172.21.15.118:0/854929295 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_get_version_reply(handle=1 version=68) v2 ==== 24+0+0 (secure 0 0 0) 0x7f5fa400f900 con 0x5634260861e0 2024-03-24T00:21:05.520 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:21:05.513+0000 7f5fba19e740 10 client.26731 fetch_fsmap learned FSMap version 68 2024-03-24T00:21:05.520 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:21:05.513+0000 7f5fba19e740 1 -- 172.21.15.118:0/854929295 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,fsmap=68,monmap=4+}) v3 -- 0x563425dcc620 con 0x5634260861e0 2024-03-24T00:21:05.520 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:21:05.513+0000 7f5f96000640 1 -- 172.21.15.118:0/854929295 <== mon.1 v2:172.21.15.193:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f5fa8004030 con 0x5634260861e0 2024-03-24T00:21:05.520 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:21:05.513+0000 7f5f96000640 1 -- 172.21.15.118:0/854929295 <== mon.1 v2:172.21.15.193:3300/0 5 ==== fsmap(e 68) v1 ==== 7950+0+0 (secure 0 0 0) 0x7f5fa4010b80 con 0x5634260861e0 2024-03-24T00:21:05.521 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:21:05.513+0000 7f5fba19e740 10 client.26731 fetch_fsmap finished waiting for FSMap version 68 2024-03-24T00:21:05.521 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:21:05.513+0000 7f5fba19e740 10 client.26731 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-24T00:21:05.521 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:21:05.513+0000 7f5fba19e740 20 client.26731 populate_metadata read hostname 'smithi118' 2024-03-24T00:21:05.521 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:21:05.513+0000 7f5fba19e740 1 --2- 172.21.15.118:0/854929295 >> [v2:172.21.15.189:6838/4074484168,v1:172.21.15.189:6839/4074484168] conn(0x56342609c300 0x56342609e750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:21:05.521 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:21:05.513+0000 7f5fba19e740 4 client.26731 mds_command: new command op to 24569 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-24T00:21:05.521 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:21:05.513+0000 7f5fba19e740 1 -- 172.21.15.118:0/854929295 --> [v2:172.21.15.189:6838/4074484168,v1:172.21.15.189:6839/4074484168] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x563425ff8d70 con 0x56342609c300 2024-03-24T00:21:05.521 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:21:05.513+0000 7f5fafe00640 1 --2- 172.21.15.118:0/854929295 >> [v2:172.21.15.189:6838/4074484168,v1:172.21.15.189:6839/4074484168] conn(0x56342609c300 0x56342609e750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:21:05.521 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:21:05.514+0000 7f5fafe00640 1 --2- 172.21.15.118:0/854929295 >> [v2:172.21.15.189:6838/4074484168,v1:172.21.15.189:6839/4074484168] conn(0x56342609c300 0x56342609e750 crc :-1 s=READY pgs=180 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=ab61e04620116338 server_cookie=4b095d06fb50cc97 in_seq=0 out_seq=0 2024-03-24T00:21:05.521 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:21:05.514+0000 7f5f96000640 10 client.26731 ms_handle_connect on v2:172.21.15.189:6838/4074484168 2024-03-24T00:21:05.522 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:21:05.515+0000 7f5f96000640 1 -- 172.21.15.118:0/854929295 <== mds.0 v2:172.21.15.189:6838/4074484168 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x563425ff8d70 con 0x56342609c300 2024-03-24T00:21:05.522 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:21:05.515+0000 7f5f96000640 10 client.26731 handle_command_reply: tid=0 2024-03-24T00:21:05.522 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:21:05.515+0000 7f5fba19e740 1 client.26731 shutdown 2024-03-24T00:21:05.522 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:21:05.515+0000 7f5fba19e740 1 -- 172.21.15.118:0/854929295 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5634260861e0 msgr2=0x563426090380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:21:05.522 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:21:05.515+0000 7f5fba19e740 1 --2- 172.21.15.118:0/854929295 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5634260861e0 0x563426090380 secure :-1 s=READY pgs=422 cs=0 l=1 rev1=1 crypto rx=0x7f5fa4012700 tx=0x7f5fa4012b00 comp rx=0 tx=0).stop 2024-03-24T00:21:05.523 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:21:05.515+0000 7f5fba19e740 1 -- 172.21.15.118:0/854929295 shutdown_connections 2024-03-24T00:21:05.523 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:21:05.515+0000 7f5fba19e740 1 --2- 172.21.15.118:0/854929295 >> [v2:172.21.15.189:6838/4074484168,v1:172.21.15.189:6839/4074484168] conn(0x56342609c300 0x56342609e750 crc :-1 s=READY pgs=180 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:21:05.523 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:21:05.515+0000 7f5fba19e740 1 --2- 172.21.15.118:0/854929295 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x56342608bd40 0x563426097900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:21:05.523 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:21:05.515+0000 7f5fba19e740 1 --2- 172.21.15.118:0/854929295 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x563426086bd0 0x5634260908c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:21:05.523 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:21:05.515+0000 7f5fba19e740 1 --2- 172.21.15.118:0/854929295 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5634260861e0 0x563426090380 unknown :-1 s=CLOSED pgs=422 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:21:05.523 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:21:05.515+0000 7f5f96000640 0 client.26731 ms_handle_reset on v2:172.21.15.189:6838/4074484168 2024-03-24T00:21:05.524 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:21:05.515+0000 7f5fba19e740 1 -- 172.21.15.118:0/854929295 >> 172.21.15.118:0/854929295 conn(0x563426081750 msgr2=0x563426080950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:21:05.524 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:21:05.515+0000 7f5fba19e740 1 -- 172.21.15.118:0/854929295 shutdown_connections 2024-03-24T00:21:05.524 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:21:05.515+0000 7f5fba19e740 1 -- 172.21.15.118:0/854929295 wait complete. 2024-03-24T00:21:05.524 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:21:05.515+0000 7f5fba19e740 20 client.26731 trim_cache size 0 max 16384 2024-03-24T00:21:05.532 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:21:05.530+0000 7f5fba19e740 1 Processor -- start 2024-03-24T00:21:05.532 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:21:05.531+0000 7f5fba19e740 1 -- start start 2024-03-24T00:21:05.532 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:21:05.531+0000 7f5fba19e740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5634260861e0 0x5634260807d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:21:05.532 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:21:05.531+0000 7f5fba19e740 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x563426086bd0 0x5634260978c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:21:05.532 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:21:05.531+0000 7f5fba19e740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56342608bd40 0x5634260a3050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:21:05.532 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:21:05.531+0000 7f5fba19e740 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5fb01abe60 con 0x563426086bd0 2024-03-24T00:21:05.532 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:21:05.531+0000 7f5fba19e740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5fb01ac520 con 0x5634260861e0 2024-03-24T00:21:05.532 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:21:05.531+0000 7f5fba19e740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5fb01acb80 con 0x56342608bd40 2024-03-24T00:21:05.532 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:21:05.531+0000 7f5fb5400640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56342608bd40 0x5634260a3050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:21:05.532 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:21:05.531+0000 7f5fb5400640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56342608bd40 0x5634260a3050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.118:38172/0 (socket says 172.21.15.118:38172) 2024-03-24T00:21:05.532 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:21:05.531+0000 7f5fb5400640 1 -- 172.21.15.118:0/4013251115 learned_addr learned my addr 172.21.15.118:0/4013251115 (peer_addr_for_me v2:172.21.15.118:0/0) 2024-03-24T00:21:05.533 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:21:05.531+0000 7f5fafe00640 1 --2- 172.21.15.118:0/4013251115 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x563426086bd0 0x5634260978c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:21:05.533 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:21:05.531+0000 7f5fb4a00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5634260861e0 0x5634260807d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:21:05.537 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:21:05.531+0000 7f5fb5400640 1 -- 172.21.15.118:0/4013251115 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5634260861e0 msgr2=0x5634260807d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:21:05.537 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:21:05.531+0000 7f5fb5400640 1 --2- 172.21.15.118:0/4013251115 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5634260861e0 0x5634260807d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:21:05.537 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:21:05.532+0000 7f5fb5400640 1 -- 172.21.15.118:0/4013251115 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x563426086bd0 msgr2=0x5634260978c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:21:05.537 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:21:05.532+0000 7f5fb5400640 1 --2- 172.21.15.118:0/4013251115 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x563426086bd0 0x5634260978c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:21:05.537 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:21:05.532+0000 7f5fb5400640 1 -- 172.21.15.118:0/4013251115 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5634260865c0 con 0x56342608bd40 2024-03-24T00:21:05.537 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:21:05.532+0000 7f5fb5400640 1 --2- 172.21.15.118:0/4013251115 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56342608bd40 0x5634260a3050 secure :-1 s=READY pgs=423 cs=0 l=1 rev1=1 crypto rx=0x563426087430 tx=0x7f5fa8023000 comp rx=0 tx=0).ready entity=mon.1 client_cookie=4c64032ec28b0be5 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:21:05.537 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:21:05.532+0000 7f5fa1200640 1 -- 172.21.15.118:0/4013251115 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5fa802c070 con 0x56342608bd40 2024-03-24T00:21:05.537 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:21:05.532+0000 7f5fa1200640 1 -- 172.21.15.118:0/4013251115 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f5fa8037070 con 0x56342608bd40 2024-03-24T00:21:05.537 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:21:05.532+0000 7f5fa1200640 1 -- 172.21.15.118:0/4013251115 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5fa801e050 con 0x56342608bd40 2024-03-24T00:21:05.537 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:21:05.533+0000 7f5fba19e740 1 -- 172.21.15.118:0/4013251115 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56342608bd40 msgr2=0x5634260a3050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:21:05.538 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:21:05.533+0000 7f5fba19e740 1 --2- 172.21.15.118:0/4013251115 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56342608bd40 0x5634260a3050 secure :-1 s=READY pgs=423 cs=0 l=1 rev1=1 crypto rx=0x563426087430 tx=0x7f5fa8023000 comp rx=0 tx=0).stop 2024-03-24T00:21:05.538 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:21:05.533+0000 7f5fba19e740 1 -- 172.21.15.118:0/4013251115 shutdown_connections 2024-03-24T00:21:05.538 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:21:05.533+0000 7f5fba19e740 1 --2- 172.21.15.118:0/4013251115 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56342608bd40 0x5634260a3050 unknown :-1 s=CLOSED pgs=423 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:21:05.538 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:21:05.533+0000 7f5fba19e740 1 --2- 172.21.15.118:0/4013251115 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x563426086bd0 0x5634260978c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:21:05.538 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:21:05.533+0000 7f5fba19e740 1 --2- 172.21.15.118:0/4013251115 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5634260861e0 0x5634260807d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:21:05.538 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:21:05.533+0000 7f5fba19e740 1 -- 172.21.15.118:0/4013251115 >> 172.21.15.118:0/4013251115 conn(0x563426081750 msgr2=0x5634260800c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:21:05.538 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:21:05.533+0000 7f5fba19e740 1 -- 172.21.15.118:0/4013251115 shutdown_connections 2024-03-24T00:21:05.538 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:21:05.533+0000 7f5fba19e740 1 -- 172.21.15.118:0/4013251115 wait complete. 2024-03-24T00:21:05.538 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:21:05.533+0000 7f5fba19e740 1 -- start start 2024-03-24T00:21:05.538 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:21:05.534+0000 7f5fba19e740 1 Processor -- start 2024-03-24T00:21:05.538 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:21:05.534+0000 7f5fba19e740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5634260861e0 0x563426095610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:21:05.538 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:21:05.534+0000 7f5fba19e740 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x563426086bd0 0x563426095b50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:21:05.539 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:21:05.534+0000 7f5fba19e740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56342608bd40 0x563426096b60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:21:05.539 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:21:05.534+0000 7f5fba19e740 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5fb01a9320 con 0x563426086bd0 2024-03-24T00:21:05.539 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:21:05.534+0000 7f5fba19e740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5fb01aa5c0 con 0x5634260861e0 2024-03-24T00:21:05.539 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:21:05.534+0000 7f5fba19e740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5fb01a9920 con 0x56342608bd40 2024-03-24T00:21:05.539 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:21:05.534+0000 7f5fba19e740 10 client.0 _ll_register_callbacks cb 0x7f5fa4009850 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T00:21:05.539 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:21:05.534+0000 7f5fb4a00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5634260861e0 0x563426095610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:21:05.539 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:21:05.534+0000 7f5fb4a00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5634260861e0 0x563426095610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.118:35054/0 (socket says 172.21.15.118:35054) 2024-03-24T00:21:05.539 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:21:05.534+0000 7f5fb4a00640 1 -- 172.21.15.118:0/3194366471 learned_addr learned my addr 172.21.15.118:0/3194366471 (peer_addr_for_me v2:172.21.15.118:0/0) 2024-03-24T00:21:05.539 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:21:05.534+0000 7f5fafe00640 1 --2- 172.21.15.118:0/3194366471 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x563426086bd0 0x563426095b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:21:05.539 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:21:05.534+0000 7f5fb5400640 1 --2- 172.21.15.118:0/3194366471 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56342608bd40 0x563426096b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:21:05.539 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:21:05.535+0000 7f5fb4a00640 1 -- 172.21.15.118:0/3194366471 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56342608bd40 msgr2=0x563426096b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:21:05.540 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:21:05.535+0000 7f5fb4a00640 1 --2- 172.21.15.118:0/3194366471 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56342608bd40 0x563426096b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:21:05.540 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:21:05.535+0000 7f5fb4a00640 1 -- 172.21.15.118:0/3194366471 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x563426086bd0 msgr2=0x563426095b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:21:05.540 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:21:05.535+0000 7f5fb4a00640 1 --2- 172.21.15.118:0/3194366471 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x563426086bd0 0x563426095b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:21:05.540 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:21:05.535+0000 7f5fb4a00640 1 -- 172.21.15.118:0/3194366471 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x563426097150 con 0x5634260861e0 2024-03-24T00:21:05.540 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:21:05.535+0000 7f5fb4a00640 1 --2- 172.21.15.118:0/3194366471 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5634260861e0 0x563426095610 secure :-1 s=READY pgs=417 cs=0 l=1 rev1=1 crypto rx=0x56342607ff10 tx=0x7f5fa4009d50 comp rx=0 tx=0).ready entity=mon.2 client_cookie=8211d79f3a8101ec server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:21:05.540 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:21:05.535+0000 7f5f96000640 10 client.0 ms_handle_connect on v2:172.21.15.189:3300/0 2024-03-24T00:21:05.540 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:21:05.535+0000 7f5f96000640 1 -- 172.21.15.118:0/3194366471 <== mon.2 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5fa402f060 con 0x5634260861e0 2024-03-24T00:21:05.541 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:21:05.535+0000 7f5fba19e740 1 -- 172.21.15.118:0/3194366471 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f5fa8037070 con 0x5634260861e0 2024-03-24T00:21:05.541 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:21:05.535+0000 7f5f96000640 1 -- 172.21.15.118:0/3194366471 <== mon.2 v2:172.21.15.189:3300/0 2 ==== osd_map(98..98 src has 1..98) v4 ==== 8965+0+0 (secure 0 0 0) 0x7f5fa4005320 con 0x5634260861e0 2024-03-24T00:21:05.541 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:21:05.535+0000 7f5f96000640 1 -- 172.21.15.118:0/3194366471 <== mon.2 v2:172.21.15.189:3300/0 3 ==== mon_get_version_reply(handle=1 version=68) v2 ==== 24+0+0 (secure 0 0 0) 0x7f5fa4036050 con 0x5634260861e0 2024-03-24T00:21:05.541 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:21:05.535+0000 7f5fba19e740 10 client.26723 fetch_fsmap learned FSMap version 68 2024-03-24T00:21:05.541 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:21:05.535+0000 7f5fba19e740 1 -- 172.21.15.118:0/3194366471 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,fsmap=68,monmap=4+}) v3 -- 0x5634260f82f0 con 0x5634260861e0 2024-03-24T00:21:05.541 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:21:05.536+0000 7f5f96000640 1 -- 172.21.15.118:0/3194366471 <== mon.2 v2:172.21.15.189:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f5fa8037070 con 0x5634260861e0 2024-03-24T00:21:05.541 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:21:05.536+0000 7f5f96000640 1 -- 172.21.15.118:0/3194366471 <== mon.2 v2:172.21.15.189:3300/0 5 ==== fsmap(e 68) v1 ==== 7950+0+0 (secure 0 0 0) 0x7f5fa401b5b0 con 0x5634260861e0 2024-03-24T00:21:05.541 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:21:05.536+0000 7f5fba19e740 10 client.26723 fetch_fsmap finished waiting for FSMap version 68 2024-03-24T00:21:05.542 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:21:05.536+0000 7f5fba19e740 10 client.26723 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-24T00:21:05.542 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:21:05.536+0000 7f5fba19e740 20 client.26723 populate_metadata read hostname 'smithi118' 2024-03-24T00:21:05.542 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:21:05.536+0000 7f5fba19e740 1 --2- 172.21.15.118:0/3194366471 >> [v2:172.21.15.189:6838/4074484168,v1:172.21.15.189:6839/4074484168] conn(0x5634260f8c70 0x5634260fb090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:21:05.542 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:21:05.536+0000 7f5fba19e740 4 client.26723 mds_command: new command op to 24569 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-03-24T00:21:05.542 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:21:05.536+0000 7f5fba19e740 1 -- 172.21.15.118:0/3194366471 --> [v2:172.21.15.189:6838/4074484168,v1:172.21.15.189:6839/4074484168] -- command(tid 0: {"prefix": "flush journal"}) v1 -- 0x563425ff30c0 con 0x5634260f8c70 2024-03-24T00:21:05.542 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:21:05.536+0000 7f5fafe00640 1 --2- 172.21.15.118:0/3194366471 >> [v2:172.21.15.189:6838/4074484168,v1:172.21.15.189:6839/4074484168] conn(0x5634260f8c70 0x5634260fb090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:21:05.543 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:21:05.537+0000 7f5fafe00640 1 --2- 172.21.15.118:0/3194366471 >> [v2:172.21.15.189:6838/4074484168,v1:172.21.15.189:6839/4074484168] conn(0x5634260f8c70 0x5634260fb090 crc :-1 s=READY pgs=181 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=115ed4ee8c5febe9 server_cookie=bf939b19690bb392 in_seq=0 out_seq=0 2024-03-24T00:21:05.543 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:21:05.537+0000 7f5f96000640 10 client.26723 ms_handle_connect on v2:172.21.15.189:6838/4074484168 2024-03-24T00:21:06.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:21:06 smithi118 ceph-mon[31535]: daemon mds.h restarted 2024-03-24T00:21:06.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:21:06 smithi118 ceph-mon[31535]: osdmap e98: 12 total, 12 up, 12 in 2024-03-24T00:21:06.447 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:21:06 smithi118 ceph-mon[31535]: mds.? [v2:172.21.15.189:6836/596160018,v1:172.21.15.189:6837/596160018] up:boot 2024-03-24T00:21:06.447 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:21:06 smithi118 ceph-mon[31535]: Standby daemon mds.f assigned to filesystem cephfs as rank 1 2024-03-24T00:21:06.447 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:21:06 smithi118 ceph-mon[31535]: fsmap cephfs:4/5 {0=b=up:active,2=c=up:active,3=g=up:replay,4=e=up:active} 4 up:standby-replay 4 up:standby, 1 failed 2024-03-24T00:21:06.447 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:21:06 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-03-24T00:21:06.447 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:21:06 smithi118 ceph-mon[31535]: fsmap cephfs:5/5 3 up:standby-replay 4 up:standby 2 up:replay 3 up:active 2024-03-24T00:21:06.447 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:21:06 smithi118 ceph-mon[31535]: pgmap v568: 97 pgs: 97 active+clean; 135 MiB data, 731 MiB used, 1.0 TiB / 1.0 TiB avail; 304 KiB/s rd, 2.2 MiB/s wr, 111 op/s 2024-03-24T00:21:06.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:21:06 smithi189 ceph-mon[36187]: daemon mds.h restarted 2024-03-24T00:21:06.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:21:06 smithi189 ceph-mon[36187]: osdmap e98: 12 total, 12 up, 12 in 2024-03-24T00:21:06.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:21:06 smithi189 ceph-mon[36187]: mds.? [v2:172.21.15.189:6836/596160018,v1:172.21.15.189:6837/596160018] up:boot 2024-03-24T00:21:06.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:21:06 smithi189 ceph-mon[36187]: Standby daemon mds.f assigned to filesystem cephfs as rank 1 2024-03-24T00:21:06.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:21:06 smithi189 ceph-mon[36187]: fsmap cephfs:4/5 {0=b=up:active,2=c=up:active,3=g=up:replay,4=e=up:active} 4 up:standby-replay 4 up:standby, 1 failed 2024-03-24T00:21:06.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:21:06 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-03-24T00:21:06.455 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:21:06 smithi189 ceph-mon[36187]: fsmap cephfs:5/5 3 up:standby-replay 4 up:standby 2 up:replay 3 up:active 2024-03-24T00:21:06.455 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:21:06 smithi189 ceph-mon[36187]: pgmap v568: 97 pgs: 97 active+clean; 135 MiB data, 731 MiB used, 1.0 TiB / 1.0 TiB avail; 304 KiB/s rd, 2.2 MiB/s wr, 111 op/s 2024-03-24T00:21:06.460 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:21:06 smithi193 ceph-mon[36878]: daemon mds.h restarted 2024-03-24T00:21:06.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:21:06 smithi193 ceph-mon[36878]: osdmap e98: 12 total, 12 up, 12 in 2024-03-24T00:21:06.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:21:06 smithi193 ceph-mon[36878]: mds.? [v2:172.21.15.189:6836/596160018,v1:172.21.15.189:6837/596160018] up:boot 2024-03-24T00:21:06.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:21:06 smithi193 ceph-mon[36878]: Standby daemon mds.f assigned to filesystem cephfs as rank 1 2024-03-24T00:21:06.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:21:06 smithi193 ceph-mon[36878]: fsmap cephfs:4/5 {0=b=up:active,2=c=up:active,3=g=up:replay,4=e=up:active} 4 up:standby-replay 4 up:standby, 1 failed 2024-03-24T00:21:06.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:21:06 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-03-24T00:21:06.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:21:06 smithi193 ceph-mon[36878]: fsmap cephfs:5/5 3 up:standby-replay 4 up:standby 2 up:replay 3 up:active 2024-03-24T00:21:06.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:21:06 smithi193 ceph-mon[36878]: pgmap v568: 97 pgs: 97 active+clean; 135 MiB data, 731 MiB used, 1.0 TiB / 1.0 TiB avail; 304 KiB/s rd, 2.2 MiB/s wr, 111 op/s 2024-03-24T00:21:06.844 INFO:tasks.workunit.client.0.smithi118.stdout: 41 0 0 0 0 0 0 2024-03-24T00:21:07.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:21:07 smithi118 ceph-mon[31535]: mds.? [v2:172.21.15.118:6838/3808689299,v1:172.21.15.118:6839/3808689299] up:resolve 2024-03-24T00:21:07.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:21:07 smithi118 ceph-mon[31535]: mds.? [v2:172.21.15.193:6834/799001823,v1:172.21.15.193:6835/799001823] up:resolve 2024-03-24T00:21:07.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:21:07 smithi118 ceph-mon[31535]: fsmap cephfs:5/5 3 up:standby-replay 4 up:standby 2 up:resolve 3 up:active 2024-03-24T00:21:07.704 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:21:07 smithi189 ceph-mon[36187]: mds.? [v2:172.21.15.118:6838/3808689299,v1:172.21.15.118:6839/3808689299] up:resolve 2024-03-24T00:21:07.704 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:21:07 smithi189 ceph-mon[36187]: mds.? [v2:172.21.15.193:6834/799001823,v1:172.21.15.193:6835/799001823] up:resolve 2024-03-24T00:21:07.704 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:21:07 smithi189 ceph-mon[36187]: fsmap cephfs:5/5 3 up:standby-replay 4 up:standby 2 up:resolve 3 up:active 2024-03-24T00:21:07.710 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:21:07 smithi193 ceph-mon[36878]: mds.? [v2:172.21.15.118:6838/3808689299,v1:172.21.15.118:6839/3808689299] up:resolve 2024-03-24T00:21:07.710 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:21:07 smithi193 ceph-mon[36878]: mds.? [v2:172.21.15.193:6834/799001823,v1:172.21.15.193:6835/799001823] up:resolve 2024-03-24T00:21:07.711 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:21:07 smithi193 ceph-mon[36878]: fsmap cephfs:5/5 3 up:standby-replay 4 up:standby 2 up:resolve 3 up:active 2024-03-24T00:21:08.337 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:21:08.336+0000 7f5f96000640 1 -- 172.21.15.118:0/3194366471 <== mds.0 v2:172.21.15.189:6838/4074484168 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+44 (crc 0 0 0) 0x563425ff30c0 con 0x5634260f8c70 2024-03-24T00:21:08.338 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:21:08.336+0000 7f5f96000640 10 client.26723 handle_command_reply: tid=0 2024-03-24T00:21:08.338 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:21:08.336+0000 7f5fba19e740 1 client.26723 shutdown 2024-03-24T00:21:08.338 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:21:08.337+0000 7f5fba19e740 1 -- 172.21.15.118:0/3194366471 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5634260861e0 msgr2=0x563426095610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:21:08.338 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:21:08.337+0000 7f5fba19e740 1 --2- 172.21.15.118:0/3194366471 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5634260861e0 0x563426095610 secure :-1 s=READY pgs=417 cs=0 l=1 rev1=1 crypto rx=0x56342607ff10 tx=0x7f5fa4009d50 comp rx=0 tx=0).stop 2024-03-24T00:21:08.339 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:21:08.337+0000 7f5fba19e740 1 -- 172.21.15.118:0/3194366471 shutdown_connections 2024-03-24T00:21:08.339 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:21:08.337+0000 7f5fba19e740 1 --2- 172.21.15.118:0/3194366471 >> [v2:172.21.15.189:6838/4074484168,v1:172.21.15.189:6839/4074484168] conn(0x5634260f8c70 0x5634260fb090 crc :-1 s=READY pgs=181 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:21:08.339 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:21:08.337+0000 7f5fba19e740 1 --2- 172.21.15.118:0/3194366471 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x56342608bd40 0x563426096b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:21:08.339 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:21:08.337+0000 7f5fba19e740 1 --2- 172.21.15.118:0/3194366471 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x563426086bd0 0x563426095b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:21:08.339 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:21:08.337+0000 7f5fba19e740 1 --2- 172.21.15.118:0/3194366471 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5634260861e0 0x563426095610 unknown :-1 s=CLOSED pgs=417 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:21:08.339 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:21:08.337+0000 7f5f96000640 0 client.26723 ms_handle_reset on v2:172.21.15.189:6838/4074484168 2024-03-24T00:21:08.339 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:21:08.337+0000 7f5fba19e740 1 -- 172.21.15.118:0/3194366471 >> 172.21.15.118:0/3194366471 conn(0x563426081750 msgr2=0x563426080950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:21:08.339 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:21:08.338+0000 7f5fba19e740 1 -- 172.21.15.118:0/3194366471 shutdown_connections 2024-03-24T00:21:08.339 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:21:08.338+0000 7f5fba19e740 1 -- 172.21.15.118:0/3194366471 wait complete. 2024-03-24T00:21:08.339 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:21:08.338+0000 7f5fba19e740 20 client.26723 trim_cache size 0 max 16384 2024-03-24T00:21:08.340 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stdout:{ 2024-03-24T00:21:08.340 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stdout: "message": "", 2024-03-24T00:21:08.340 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stdout: "return_code": 0 2024-03-24T00:21:08.340 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stdout:} 2024-03-24T00:21:08.340 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:21:08.339+0000 7f5fb7400640 1 -- 172.21.15.118:0/1931139018 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7f5f980a28a0 msgr2=0x7f5f980a4d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:21:08.341 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:21:08.339+0000 7f5fb7400640 1 --2- 172.21.15.118:0/1931139018 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7f5f980a28a0 0x7f5f980a4d60 secure :-1 s=READY pgs=360 cs=0 l=1 rev1=1 crypto rx=0x7f5fa8009d40 tx=0x7f5fa8004870 comp rx=0 tx=0).stop 2024-03-24T00:21:08.341 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:21:08.339+0000 7f5fb7400640 1 -- 172.21.15.118:0/1931139018 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f5fb01028c0 msgr2=0x7f5fb0073a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:21:08.341 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:21:08.339+0000 7f5fb7400640 1 --2- 172.21.15.118:0/1931139018 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f5fb01028c0 0x7f5fb0073a90 secure :-1 s=READY pgs=421 cs=0 l=1 rev1=1 crypto rx=0x7f5f9c000c00 tx=0x7f5f9c02c650 comp rx=0 tx=0).stop 2024-03-24T00:21:08.341 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:21:08.339+0000 7f5fb7400640 1 -- 172.21.15.118:0/1931139018 shutdown_connections 2024-03-24T00:21:08.341 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:21:08.339+0000 7f5fb7400640 1 --2- 172.21.15.118:0/1931139018 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7f5f980a28a0 0x7f5f980a4d60 unknown :-1 s=CLOSED pgs=360 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:21:08.341 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:21:08.339+0000 7f5fb7400640 1 --2- 172.21.15.118:0/1931139018 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f5fb0103280 0x7f5fb0073fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:21:08.341 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:21:08.339+0000 7f5fb7400640 1 --2- 172.21.15.118:0/1931139018 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f5fb01028c0 0x7f5fb0073a90 unknown :-1 s=CLOSED pgs=421 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:21:08.341 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:21:08.339+0000 7f5fb7400640 1 --2- 172.21.15.118:0/1931139018 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5fb01016d0 0x7f5fb0075450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:21:08.341 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:21:08.339+0000 7f5fb7400640 1 -- 172.21.15.118:0/1931139018 >> 172.21.15.118:0/1931139018 conn(0x7f5fb00fcc60 msgr2=0x7f5fb0105af0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:21:08.342 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:21:08.341+0000 7f5fb7400640 1 -- 172.21.15.118:0/1931139018 shutdown_connections 2024-03-24T00:21:08.342 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:21:08.341+0000 7f5fb7400640 1 -- 172.21.15.118:0/1931139018 wait complete. 2024-03-24T00:21:08.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:21:08 smithi118 ceph-mon[31535]: mds.? [v2:172.21.15.118:6838/3808689299,v1:172.21.15.118:6839/3808689299] up:reconnect 2024-03-24T00:21:08.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:21:08 smithi118 ceph-mon[31535]: mds.? [v2:172.21.15.193:6834/799001823,v1:172.21.15.193:6835/799001823] up:reconnect 2024-03-24T00:21:08.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:21:08 smithi118 ceph-mon[31535]: fsmap cephfs:5/5 3 up:standby-replay 4 up:standby 2 up:reconnect 3 up:active 2024-03-24T00:21:08.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:21:08 smithi118 ceph-mon[31535]: reconnect by client.15159 192.168.0.1:0/3463739313 after 0.0139998 2024-03-24T00:21:08.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:21:08 smithi118 ceph-mon[31535]: reconnect by client.15159 192.168.0.1:0/3463739313 after 0.0239994 2024-03-24T00:21:08.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:21:08 smithi118 ceph-mon[31535]: pgmap v569: 97 pgs: 97 active+clean; 135 MiB data, 732 MiB used, 1.0 TiB / 1.0 TiB avail; 294 KiB/s rd, 1.5 MiB/s wr, 157 op/s 2024-03-24T00:21:08.704 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:21:08 smithi189 ceph-mon[36187]: mds.? [v2:172.21.15.118:6838/3808689299,v1:172.21.15.118:6839/3808689299] up:reconnect 2024-03-24T00:21:08.704 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:21:08 smithi189 ceph-mon[36187]: mds.? [v2:172.21.15.193:6834/799001823,v1:172.21.15.193:6835/799001823] up:reconnect 2024-03-24T00:21:08.704 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:21:08 smithi189 ceph-mon[36187]: fsmap cephfs:5/5 3 up:standby-replay 4 up:standby 2 up:reconnect 3 up:active 2024-03-24T00:21:08.704 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:21:08 smithi189 ceph-mon[36187]: reconnect by client.15159 192.168.0.1:0/3463739313 after 0.0139998 2024-03-24T00:21:08.704 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:21:08 smithi189 ceph-mon[36187]: reconnect by client.15159 192.168.0.1:0/3463739313 after 0.0239994 2024-03-24T00:21:08.704 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:21:08 smithi189 ceph-mon[36187]: pgmap v569: 97 pgs: 97 active+clean; 135 MiB data, 732 MiB used, 1.0 TiB / 1.0 TiB avail; 294 KiB/s rd, 1.5 MiB/s wr, 157 op/s 2024-03-24T00:21:08.710 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:21:08 smithi193 ceph-mon[36878]: mds.? [v2:172.21.15.118:6838/3808689299,v1:172.21.15.118:6839/3808689299] up:reconnect 2024-03-24T00:21:08.710 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:21:08 smithi193 ceph-mon[36878]: mds.? [v2:172.21.15.193:6834/799001823,v1:172.21.15.193:6835/799001823] up:reconnect 2024-03-24T00:21:08.711 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:21:08 smithi193 ceph-mon[36878]: fsmap cephfs:5/5 3 up:standby-replay 4 up:standby 2 up:reconnect 3 up:active 2024-03-24T00:21:08.711 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:21:08 smithi193 ceph-mon[36878]: reconnect by client.15159 192.168.0.1:0/3463739313 after 0.0139998 2024-03-24T00:21:08.711 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:21:08 smithi193 ceph-mon[36878]: reconnect by client.15159 192.168.0.1:0/3463739313 after 0.0239994 2024-03-24T00:21:08.711 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:21:08 smithi193 ceph-mon[36878]: pgmap v569: 97 pgs: 97 active+clean; 135 MiB data, 732 MiB used, 1.0 TiB / 1.0 TiB avail; 294 KiB/s rd, 1.5 MiB/s wr, 157 op/s 2024-03-24T00:21:09.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:21:09 smithi118 ceph-mon[31535]: mds.? [v2:172.21.15.193:6834/799001823,v1:172.21.15.193:6835/799001823] up:rejoin 2024-03-24T00:21:09.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:21:09 smithi118 ceph-mon[31535]: mds.? [v2:172.21.15.118:6838/3808689299,v1:172.21.15.118:6839/3808689299] up:rejoin 2024-03-24T00:21:09.697 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:21:09 smithi118 ceph-mon[31535]: fsmap cephfs:5/5 3 up:standby-replay 4 up:standby 2 up:rejoin 3 up:active 2024-03-24T00:21:09.697 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:21:09 smithi118 ceph-mon[31535]: daemon mds.f is now active in filesystem cephfs as rank 1 2024-03-24T00:21:09.697 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:21:09 smithi118 ceph-mon[31535]: daemon mds.g is now active in filesystem cephfs as rank 3 2024-03-24T00:21:09.698 INFO:journalctl@ceph.mds.g.smithi118.stdout:Mar 24 00:21:09 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-g[93896]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: In function 'void Locker::set_xlocks_done(MutationImpl*, bool)' thread 7f1e57a00700 time 2024-03-24T00:21:09.349055+0000 2024-03-24T00:21:09.698 INFO:journalctl@ceph.mds.g.smithi118.stdout:Mar 24 00:21:09 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-g[93896]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: 626: FAILED ceph_assert(obj->is_auth()) 2024-03-24T00:21:09.698 INFO:journalctl@ceph.mds.g.smithi118.stdout:Mar 24 00:21:09 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-g[93896]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-24T00:21:09.698 INFO:journalctl@ceph.mds.g.smithi118.stdout:Mar 24 00:21:09 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-g[93896]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x135) [0x7f1e6333e4a1] 2024-03-24T00:21:09.699 INFO:journalctl@ceph.mds.g.smithi118.stdout:Mar 24 00:21:09 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-g[93896]: 2: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7f1e6333e667] 2024-03-24T00:21:09.699 INFO:journalctl@ceph.mds.g.smithi118.stdout:Mar 24 00:21:09 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-g[93896]: 3: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x558bf5b10223] 2024-03-24T00:21:09.699 INFO:journalctl@ceph.mds.g.smithi118.stdout:Mar 24 00:21:09 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-g[93896]: 4: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x558bf598a5d6] 2024-03-24T00:21:09.699 INFO:journalctl@ceph.mds.g.smithi118.stdout:Mar 24 00:21:09 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-g[93896]: 5: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x558bf598ada1] 2024-03-24T00:21:09.699 INFO:journalctl@ceph.mds.g.smithi118.stdout:Mar 24 00:21:09 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-g[93896]: 6: (Server::handle_client_rename(boost::intrusive_ptr const&)+0x259d) [0x558bf59c797d] 2024-03-24T00:21:09.699 INFO:journalctl@ceph.mds.g.smithi118.stdout:Mar 24 00:21:09 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-g[93896]: 7: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x558bf59dd233] 2024-03-24T00:21:09.699 INFO:journalctl@ceph.mds.g.smithi118.stdout:Mar 24 00:21:09 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-g[93896]: 8: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x558bf59df197] 2024-03-24T00:21:09.699 INFO:journalctl@ceph.mds.g.smithi118.stdout:Mar 24 00:21:09 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-g[93896]: 9: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x558bf58de11b] 2024-03-24T00:21:09.699 INFO:journalctl@ceph.mds.g.smithi118.stdout:Mar 24 00:21:09 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-g[93896]: 10: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x558bf58de6bc] 2024-03-24T00:21:09.700 INFO:journalctl@ceph.mds.g.smithi118.stdout:Mar 24 00:21:09 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-g[93896]: 11: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x558bf58c81ff] 2024-03-24T00:21:09.700 INFO:journalctl@ceph.mds.g.smithi118.stdout:Mar 24 00:21:09 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-g[93896]: 12: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7f1e635d0488] 2024-03-24T00:21:09.700 INFO:journalctl@ceph.mds.g.smithi118.stdout:Mar 24 00:21:09 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-g[93896]: 13: (DispatchQueue::entry()+0x51f) [0x7f1e635cd60f] 2024-03-24T00:21:09.700 INFO:journalctl@ceph.mds.g.smithi118.stdout:Mar 24 00:21:09 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-g[93896]: 14: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f1e636a0e81] 2024-03-24T00:21:09.700 INFO:journalctl@ceph.mds.g.smithi118.stdout:Mar 24 00:21:09 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-g[93896]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7f1e620bb1ca] 2024-03-24T00:21:09.700 INFO:journalctl@ceph.mds.g.smithi118.stdout:Mar 24 00:21:09 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-g[93896]: 16: clone() 2024-03-24T00:21:09.700 INFO:journalctl@ceph.mds.g.smithi118.stdout:Mar 24 00:21:09 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-g[93896]: 2024-03-24T00:21:09.349+0000 7f1e57a00700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: In function 'void Locker::set_xlocks_done(MutationImpl*, bool)' thread 7f1e57a00700 time 2024-03-24T00:21:09.349055+0000 2024-03-24T00:21:09.700 INFO:journalctl@ceph.mds.g.smithi118.stdout:Mar 24 00:21:09 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-g[93896]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: 626: FAILED ceph_assert(obj->is_auth()) 2024-03-24T00:21:09.700 INFO:journalctl@ceph.mds.g.smithi118.stdout:Mar 24 00:21:09 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-g[93896]: 2024-03-24T00:21:09.701 INFO:journalctl@ceph.mds.g.smithi118.stdout:Mar 24 00:21:09 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-g[93896]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-24T00:21:09.701 INFO:journalctl@ceph.mds.g.smithi118.stdout:Mar 24 00:21:09 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-g[93896]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x135) [0x7f1e6333e4a1] 2024-03-24T00:21:09.701 INFO:journalctl@ceph.mds.g.smithi118.stdout:Mar 24 00:21:09 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-g[93896]: 2: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7f1e6333e667] 2024-03-24T00:21:09.701 INFO:journalctl@ceph.mds.g.smithi118.stdout:Mar 24 00:21:09 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-g[93896]: 3: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x558bf5b10223] 2024-03-24T00:21:09.701 INFO:journalctl@ceph.mds.g.smithi118.stdout:Mar 24 00:21:09 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-g[93896]: 4: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x558bf598a5d6] 2024-03-24T00:21:09.701 INFO:journalctl@ceph.mds.g.smithi118.stdout:Mar 24 00:21:09 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-g[93896]: 5: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x558bf598ada1] 2024-03-24T00:21:09.701 INFO:journalctl@ceph.mds.g.smithi118.stdout:Mar 24 00:21:09 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-g[93896]: 6: (Server::handle_client_rename(boost::intrusive_ptr const&)+0x259d) [0x558bf59c797d] 2024-03-24T00:21:09.701 INFO:journalctl@ceph.mds.g.smithi118.stdout:Mar 24 00:21:09 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-g[93896]: 7: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x558bf59dd233] 2024-03-24T00:21:09.701 INFO:journalctl@ceph.mds.g.smithi118.stdout:Mar 24 00:21:09 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-g[93896]: 8: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x558bf59df197] 2024-03-24T00:21:09.701 INFO:journalctl@ceph.mds.g.smithi118.stdout:Mar 24 00:21:09 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-g[93896]: 9: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x558bf58de11b] 2024-03-24T00:21:09.701 INFO:journalctl@ceph.mds.g.smithi118.stdout:Mar 24 00:21:09 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-g[93896]: 10: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x558bf58de6bc] 2024-03-24T00:21:09.702 INFO:journalctl@ceph.mds.g.smithi118.stdout:Mar 24 00:21:09 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-g[93896]: 11: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x558bf58c81ff] 2024-03-24T00:21:09.702 INFO:journalctl@ceph.mds.g.smithi118.stdout:Mar 24 00:21:09 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-g[93896]: 12: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7f1e635d0488] 2024-03-24T00:21:09.702 INFO:journalctl@ceph.mds.g.smithi118.stdout:Mar 24 00:21:09 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-g[93896]: 13: (DispatchQueue::entry()+0x51f) [0x7f1e635cd60f] 2024-03-24T00:21:09.702 INFO:journalctl@ceph.mds.g.smithi118.stdout:Mar 24 00:21:09 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-g[93896]: 14: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f1e636a0e81] 2024-03-24T00:21:09.702 INFO:journalctl@ceph.mds.g.smithi118.stdout:Mar 24 00:21:09 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-g[93896]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7f1e620bb1ca] 2024-03-24T00:21:09.702 INFO:journalctl@ceph.mds.g.smithi118.stdout:Mar 24 00:21:09 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-g[93896]: 16: clone() 2024-03-24T00:21:09.702 INFO:journalctl@ceph.mds.g.smithi118.stdout:Mar 24 00:21:09 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-g[93896]: 2024-03-24T00:21:09.702 INFO:journalctl@ceph.mds.g.smithi118.stdout:Mar 24 00:21:09 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-g[93896]: *** Caught signal (Aborted) ** 2024-03-24T00:21:09.702 INFO:journalctl@ceph.mds.g.smithi118.stdout:Mar 24 00:21:09 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-g[93896]: in thread 7f1e57a00700 thread_name:ms_dispatch 2024-03-24T00:21:09.702 INFO:journalctl@ceph.mds.g.smithi118.stdout:Mar 24 00:21:09 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-g[93896]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-24T00:21:09.702 INFO:journalctl@ceph.mds.g.smithi118.stdout:Mar 24 00:21:09 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-g[93896]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f1e620c5d20] 2024-03-24T00:21:09.703 INFO:journalctl@ceph.mds.g.smithi118.stdout:Mar 24 00:21:09 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-g[93896]: 2: gsignal() 2024-03-24T00:21:09.703 INFO:journalctl@ceph.mds.g.smithi118.stdout:Mar 24 00:21:09 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-g[93896]: 3: abort() 2024-03-24T00:21:09.703 INFO:journalctl@ceph.mds.g.smithi118.stdout:Mar 24 00:21:09 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-g[93896]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x18f) [0x7f1e6333e4fb] 2024-03-24T00:21:09.703 INFO:journalctl@ceph.mds.g.smithi118.stdout:Mar 24 00:21:09 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-g[93896]: 5: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7f1e6333e667] 2024-03-24T00:21:09.703 INFO:journalctl@ceph.mds.g.smithi118.stdout:Mar 24 00:21:09 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-g[93896]: 6: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x558bf5b10223] 2024-03-24T00:21:09.703 INFO:journalctl@ceph.mds.g.smithi118.stdout:Mar 24 00:21:09 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-g[93896]: 7: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x558bf598a5d6] 2024-03-24T00:21:09.703 INFO:journalctl@ceph.mds.g.smithi118.stdout:Mar 24 00:21:09 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-g[93896]: 8: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x558bf598ada1] 2024-03-24T00:21:09.704 INFO:journalctl@ceph.mds.g.smithi118.stdout:Mar 24 00:21:09 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-g[93896]: 9: (Server::handle_client_rename(boost::intrusive_ptr const&)+0x259d) [0x558bf59c797d] 2024-03-24T00:21:09.704 INFO:journalctl@ceph.mds.g.smithi118.stdout:Mar 24 00:21:09 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-g[93896]: 10: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x558bf59dd233] 2024-03-24T00:21:09.704 INFO:journalctl@ceph.mds.g.smithi118.stdout:Mar 24 00:21:09 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-g[93896]: 11: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x558bf59df197] 2024-03-24T00:21:09.704 INFO:journalctl@ceph.mds.g.smithi118.stdout:Mar 24 00:21:09 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-g[93896]: 12: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x558bf58de11b] 2024-03-24T00:21:09.704 INFO:journalctl@ceph.mds.g.smithi118.stdout:Mar 24 00:21:09 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-g[93896]: 13: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x558bf58de6bc] 2024-03-24T00:21:09.704 INFO:journalctl@ceph.mds.g.smithi118.stdout:Mar 24 00:21:09 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-g[93896]: 14: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x558bf58c81ff] 2024-03-24T00:21:09.704 INFO:journalctl@ceph.mds.g.smithi118.stdout:Mar 24 00:21:09 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-g[93896]: 15: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7f1e635d0488] 2024-03-24T00:21:09.704 INFO:journalctl@ceph.mds.g.smithi118.stdout:Mar 24 00:21:09 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-g[93896]: 16: (DispatchQueue::entry()+0x51f) [0x7f1e635cd60f] 2024-03-24T00:21:09.704 INFO:journalctl@ceph.mds.g.smithi118.stdout:Mar 24 00:21:09 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-g[93896]: 17: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f1e636a0e81] 2024-03-24T00:21:09.704 INFO:journalctl@ceph.mds.g.smithi118.stdout:Mar 24 00:21:09 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-g[93896]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7f1e620bb1ca] 2024-03-24T00:21:09.704 INFO:journalctl@ceph.mds.g.smithi118.stdout:Mar 24 00:21:09 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-g[93896]: 19: clone() 2024-03-24T00:21:09.704 INFO:journalctl@ceph.mds.g.smithi118.stdout:Mar 24 00:21:09 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-g[93896]: 2024-03-24T00:21:09.352+0000 7f1e57a00700 -1 *** Caught signal (Aborted) ** 2024-03-24T00:21:09.704 INFO:journalctl@ceph.mds.g.smithi118.stdout:Mar 24 00:21:09 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-g[93896]: in thread 7f1e57a00700 thread_name:ms_dispatch 2024-03-24T00:21:09.705 INFO:journalctl@ceph.mds.g.smithi118.stdout:Mar 24 00:21:09 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-g[93896]: 2024-03-24T00:21:09.705 INFO:journalctl@ceph.mds.g.smithi118.stdout:Mar 24 00:21:09 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-g[93896]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-24T00:21:09.705 INFO:journalctl@ceph.mds.g.smithi118.stdout:Mar 24 00:21:09 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-g[93896]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f1e620c5d20] 2024-03-24T00:21:09.705 INFO:journalctl@ceph.mds.g.smithi118.stdout:Mar 24 00:21:09 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-g[93896]: 2: gsignal() 2024-03-24T00:21:09.705 INFO:journalctl@ceph.mds.g.smithi118.stdout:Mar 24 00:21:09 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-g[93896]: 3: abort() 2024-03-24T00:21:09.705 INFO:journalctl@ceph.mds.g.smithi118.stdout:Mar 24 00:21:09 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-g[93896]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x18f) [0x7f1e6333e4fb] 2024-03-24T00:21:09.705 INFO:journalctl@ceph.mds.g.smithi118.stdout:Mar 24 00:21:09 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-g[93896]: 5: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7f1e6333e667] 2024-03-24T00:21:09.705 INFO:journalctl@ceph.mds.g.smithi118.stdout:Mar 24 00:21:09 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-g[93896]: 6: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x558bf5b10223] 2024-03-24T00:21:09.705 INFO:journalctl@ceph.mds.g.smithi118.stdout:Mar 24 00:21:09 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-g[93896]: 7: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x558bf598a5d6] 2024-03-24T00:21:09.705 INFO:journalctl@ceph.mds.g.smithi118.stdout:Mar 24 00:21:09 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-g[93896]: 8: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x558bf598ada1] 2024-03-24T00:21:09.705 INFO:journalctl@ceph.mds.g.smithi118.stdout:Mar 24 00:21:09 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-g[93896]: 9: (Server::handle_client_rename(boost::intrusive_ptr const&)+0x259d) [0x558bf59c797d] 2024-03-24T00:21:09.705 INFO:journalctl@ceph.mds.g.smithi118.stdout:Mar 24 00:21:09 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-g[93896]: 10: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x558bf59dd233] 2024-03-24T00:21:09.705 INFO:journalctl@ceph.mds.g.smithi118.stdout:Mar 24 00:21:09 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-g[93896]: 11: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x558bf59df197] 2024-03-24T00:21:09.705 INFO:journalctl@ceph.mds.g.smithi118.stdout:Mar 24 00:21:09 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-g[93896]: 12: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x558bf58de11b] 2024-03-24T00:21:09.706 INFO:journalctl@ceph.mds.g.smithi118.stdout:Mar 24 00:21:09 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-g[93896]: 13: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x558bf58de6bc] 2024-03-24T00:21:09.706 INFO:journalctl@ceph.mds.g.smithi118.stdout:Mar 24 00:21:09 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-g[93896]: 14: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x558bf58c81ff] 2024-03-24T00:21:09.706 INFO:journalctl@ceph.mds.g.smithi118.stdout:Mar 24 00:21:09 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-g[93896]: 15: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7f1e635d0488] 2024-03-24T00:21:09.706 INFO:journalctl@ceph.mds.g.smithi118.stdout:Mar 24 00:21:09 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-g[93896]: 16: (DispatchQueue::entry()+0x51f) [0x7f1e635cd60f] 2024-03-24T00:21:09.706 INFO:journalctl@ceph.mds.g.smithi118.stdout:Mar 24 00:21:09 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-g[93896]: 17: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f1e636a0e81] 2024-03-24T00:21:09.706 INFO:journalctl@ceph.mds.g.smithi118.stdout:Mar 24 00:21:09 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-g[93896]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7f1e620bb1ca] 2024-03-24T00:21:09.706 INFO:journalctl@ceph.mds.g.smithi118.stdout:Mar 24 00:21:09 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-g[93896]: 19: clone() 2024-03-24T00:21:09.706 INFO:journalctl@ceph.mds.g.smithi118.stdout:Mar 24 00:21:09 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-g[93896]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-03-24T00:21:09.706 INFO:journalctl@ceph.mds.g.smithi118.stdout:Mar 24 00:21:09 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-g[93896]: 2024-03-24T00:21:09.707 INFO:journalctl@ceph.mds.g.smithi118.stdout:Mar 24 00:21:09 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-g[93896]: -1> 2024-03-24T00:21:09.349+0000 7f1e57a00700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: In function 'void Locker::set_xlocks_done(MutationImpl*, bool)' thread 7f1e57a00700 time 2024-03-24T00:21:09.349055+0000 2024-03-24T00:21:09.707 INFO:journalctl@ceph.mds.g.smithi118.stdout:Mar 24 00:21:09 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-g[93896]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: 626: FAILED ceph_assert(obj->is_auth()) 2024-03-24T00:21:09.707 INFO:journalctl@ceph.mds.g.smithi118.stdout:Mar 24 00:21:09 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-g[93896]: 2024-03-24T00:21:09.707 INFO:journalctl@ceph.mds.g.smithi118.stdout:Mar 24 00:21:09 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-g[93896]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-24T00:21:09.707 INFO:journalctl@ceph.mds.g.smithi118.stdout:Mar 24 00:21:09 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-g[93896]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x135) [0x7f1e6333e4a1] 2024-03-24T00:21:09.707 INFO:journalctl@ceph.mds.g.smithi118.stdout:Mar 24 00:21:09 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-g[93896]: 2: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7f1e6333e667] 2024-03-24T00:21:09.707 INFO:journalctl@ceph.mds.g.smithi118.stdout:Mar 24 00:21:09 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-g[93896]: 3: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x558bf5b10223] 2024-03-24T00:21:09.707 INFO:journalctl@ceph.mds.g.smithi118.stdout:Mar 24 00:21:09 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-g[93896]: 4: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x558bf598a5d6] 2024-03-24T00:21:09.707 INFO:journalctl@ceph.mds.g.smithi118.stdout:Mar 24 00:21:09 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-g[93896]: 5: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x558bf598ada1] 2024-03-24T00:21:09.707 INFO:journalctl@ceph.mds.g.smithi118.stdout:Mar 24 00:21:09 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-g[93896]: 6: (Server::handle_client_rename(boost::intrusive_ptr const&)+0x259d) [0x558bf59c797d] 2024-03-24T00:21:09.707 INFO:journalctl@ceph.mds.g.smithi118.stdout:Mar 24 00:21:09 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-g[93896]: 7: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x558bf59dd233] 2024-03-24T00:21:09.707 INFO:journalctl@ceph.mds.g.smithi118.stdout:Mar 24 00:21:09 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-g[93896]: 8: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x558bf59df197] 2024-03-24T00:21:09.707 INFO:journalctl@ceph.mds.g.smithi118.stdout:Mar 24 00:21:09 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-g[93896]: 9: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x558bf58de11b] 2024-03-24T00:21:09.707 INFO:journalctl@ceph.mds.g.smithi118.stdout:Mar 24 00:21:09 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-g[93896]: 10: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x558bf58de6bc] 2024-03-24T00:21:09.708 INFO:journalctl@ceph.mds.g.smithi118.stdout:Mar 24 00:21:09 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-g[93896]: 11: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x558bf58c81ff] 2024-03-24T00:21:09.708 INFO:journalctl@ceph.mds.g.smithi118.stdout:Mar 24 00:21:09 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-g[93896]: 12: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7f1e635d0488] 2024-03-24T00:21:09.708 INFO:journalctl@ceph.mds.g.smithi118.stdout:Mar 24 00:21:09 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-g[93896]: 13: (DispatchQueue::entry()+0x51f) [0x7f1e635cd60f] 2024-03-24T00:21:09.708 INFO:journalctl@ceph.mds.g.smithi118.stdout:Mar 24 00:21:09 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-g[93896]: 14: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f1e636a0e81] 2024-03-24T00:21:09.708 INFO:journalctl@ceph.mds.g.smithi118.stdout:Mar 24 00:21:09 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-g[93896]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7f1e620bb1ca] 2024-03-24T00:21:09.708 INFO:journalctl@ceph.mds.g.smithi118.stdout:Mar 24 00:21:09 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-g[93896]: 16: clone() 2024-03-24T00:21:09.708 INFO:journalctl@ceph.mds.g.smithi118.stdout:Mar 24 00:21:09 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-g[93896]: 2024-03-24T00:21:09.708 INFO:journalctl@ceph.mds.g.smithi118.stdout:Mar 24 00:21:09 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-g[93896]: 0> 2024-03-24T00:21:09.352+0000 7f1e57a00700 -1 *** Caught signal (Aborted) ** 2024-03-24T00:21:09.708 INFO:journalctl@ceph.mds.g.smithi118.stdout:Mar 24 00:21:09 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-g[93896]: in thread 7f1e57a00700 thread_name:ms_dispatch 2024-03-24T00:21:09.708 INFO:journalctl@ceph.mds.g.smithi118.stdout:Mar 24 00:21:09 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-g[93896]: 2024-03-24T00:21:09.708 INFO:journalctl@ceph.mds.g.smithi118.stdout:Mar 24 00:21:09 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-g[93896]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-24T00:21:09.708 INFO:journalctl@ceph.mds.g.smithi118.stdout:Mar 24 00:21:09 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-g[93896]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f1e620c5d20] 2024-03-24T00:21:09.708 INFO:journalctl@ceph.mds.g.smithi118.stdout:Mar 24 00:21:09 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-g[93896]: 2: gsignal() 2024-03-24T00:21:09.708 INFO:journalctl@ceph.mds.g.smithi118.stdout:Mar 24 00:21:09 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-g[93896]: 3: abort() 2024-03-24T00:21:09.709 INFO:journalctl@ceph.mds.g.smithi118.stdout:Mar 24 00:21:09 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-g[93896]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x18f) [0x7f1e6333e4fb] 2024-03-24T00:21:09.709 INFO:journalctl@ceph.mds.g.smithi118.stdout:Mar 24 00:21:09 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-g[93896]: 5: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7f1e6333e667] 2024-03-24T00:21:09.709 INFO:journalctl@ceph.mds.g.smithi118.stdout:Mar 24 00:21:09 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-g[93896]: 6: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x558bf5b10223] 2024-03-24T00:21:09.709 INFO:journalctl@ceph.mds.g.smithi118.stdout:Mar 24 00:21:09 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-g[93896]: 7: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x558bf598a5d6] 2024-03-24T00:21:09.710 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:21:09 smithi189 ceph-mon[36187]: mds.? [v2:172.21.15.193:6834/799001823,v1:172.21.15.193:6835/799001823] up:rejoin 2024-03-24T00:21:09.710 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:21:09 smithi189 ceph-mon[36187]: mds.? [v2:172.21.15.118:6838/3808689299,v1:172.21.15.118:6839/3808689299] up:rejoin 2024-03-24T00:21:09.710 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:21:09 smithi189 ceph-mon[36187]: fsmap cephfs:5/5 3 up:standby-replay 4 up:standby 2 up:rejoin 3 up:active 2024-03-24T00:21:09.710 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:21:09 smithi189 ceph-mon[36187]: daemon mds.f is now active in filesystem cephfs as rank 1 2024-03-24T00:21:09.710 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:21:09 smithi189 ceph-mon[36187]: daemon mds.g is now active in filesystem cephfs as rank 3 2024-03-24T00:21:09.711 INFO:journalctl@ceph.mds.g.smithi118.stdout:Mar 24 00:21:09 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-g[93896]: 8: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x558bf598ada1] 2024-03-24T00:21:09.711 INFO:journalctl@ceph.mds.g.smithi118.stdout:Mar 24 00:21:09 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-g[93896]: 9: (Server::handle_client_rename(boost::intrusive_ptr const&)+0x259d) [0x558bf59c797d] 2024-03-24T00:21:09.711 INFO:journalctl@ceph.mds.g.smithi118.stdout:Mar 24 00:21:09 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-g[93896]: 10: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x558bf59dd233] 2024-03-24T00:21:09.711 INFO:journalctl@ceph.mds.g.smithi118.stdout:Mar 24 00:21:09 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-g[93896]: 11: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x558bf59df197] 2024-03-24T00:21:09.711 INFO:journalctl@ceph.mds.g.smithi118.stdout:Mar 24 00:21:09 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-g[93896]: 12: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x558bf58de11b] 2024-03-24T00:21:09.711 INFO:journalctl@ceph.mds.g.smithi118.stdout:Mar 24 00:21:09 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-g[93896]: 13: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x558bf58de6bc] 2024-03-24T00:21:09.711 INFO:journalctl@ceph.mds.g.smithi118.stdout:Mar 24 00:21:09 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-g[93896]: 14: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x558bf58c81ff] 2024-03-24T00:21:09.711 INFO:journalctl@ceph.mds.g.smithi118.stdout:Mar 24 00:21:09 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-g[93896]: 15: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7f1e635d0488] 2024-03-24T00:21:09.712 INFO:journalctl@ceph.mds.g.smithi118.stdout:Mar 24 00:21:09 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-g[93896]: 16: (DispatchQueue::entry()+0x51f) [0x7f1e635cd60f] 2024-03-24T00:21:09.712 INFO:journalctl@ceph.mds.g.smithi118.stdout:Mar 24 00:21:09 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-g[93896]: 17: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f1e636a0e81] 2024-03-24T00:21:09.712 INFO:journalctl@ceph.mds.g.smithi118.stdout:Mar 24 00:21:09 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-g[93896]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7f1e620bb1ca] 2024-03-24T00:21:09.712 INFO:journalctl@ceph.mds.g.smithi118.stdout:Mar 24 00:21:09 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-g[93896]: 19: clone() 2024-03-24T00:21:09.712 INFO:journalctl@ceph.mds.g.smithi118.stdout:Mar 24 00:21:09 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-g[93896]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-03-24T00:21:09.712 INFO:journalctl@ceph.mds.g.smithi118.stdout:Mar 24 00:21:09 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-g[93896]: 2024-03-24T00:21:09.712 INFO:journalctl@ceph.mds.g.smithi118.stdout:Mar 24 00:21:09 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-g[93896]: -1> 2024-03-24T00:21:09.349+0000 7f1e57a00700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: In function 'void Locker::set_xlocks_done(MutationImpl*, bool)' thread 7f1e57a00700 time 2024-03-24T00:21:09.349055+0000 2024-03-24T00:21:09.712 INFO:journalctl@ceph.mds.g.smithi118.stdout:Mar 24 00:21:09 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-g[93896]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: 626: FAILED ceph_assert(obj->is_auth()) 2024-03-24T00:21:09.712 INFO:journalctl@ceph.mds.g.smithi118.stdout:Mar 24 00:21:09 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-g[93896]: 2024-03-24T00:21:09.712 INFO:journalctl@ceph.mds.g.smithi118.stdout:Mar 24 00:21:09 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-g[93896]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-24T00:21:09.712 INFO:journalctl@ceph.mds.g.smithi118.stdout:Mar 24 00:21:09 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-g[93896]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x135) [0x7f1e6333e4a1] 2024-03-24T00:21:09.712 INFO:journalctl@ceph.mds.g.smithi118.stdout:Mar 24 00:21:09 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-g[93896]: 2: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7f1e6333e667] 2024-03-24T00:21:09.712 INFO:journalctl@ceph.mds.g.smithi118.stdout:Mar 24 00:21:09 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-g[93896]: 3: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x558bf5b10223] 2024-03-24T00:21:09.713 INFO:journalctl@ceph.mds.g.smithi118.stdout:Mar 24 00:21:09 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-g[93896]: 4: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x558bf598a5d6] 2024-03-24T00:21:09.713 INFO:journalctl@ceph.mds.g.smithi118.stdout:Mar 24 00:21:09 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-g[93896]: 5: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x558bf598ada1] 2024-03-24T00:21:09.713 INFO:journalctl@ceph.mds.g.smithi118.stdout:Mar 24 00:21:09 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-g[93896]: 6: (Server::handle_client_rename(boost::intrusive_ptr const&)+0x259d) [0x558bf59c797d] 2024-03-24T00:21:09.713 INFO:journalctl@ceph.mds.g.smithi118.stdout:Mar 24 00:21:09 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-g[93896]: 7: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x558bf59dd233] 2024-03-24T00:21:09.713 INFO:journalctl@ceph.mds.g.smithi118.stdout:Mar 24 00:21:09 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-g[93896]: 8: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x558bf59df197] 2024-03-24T00:21:09.713 INFO:journalctl@ceph.mds.g.smithi118.stdout:Mar 24 00:21:09 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-g[93896]: 9: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x558bf58de11b] 2024-03-24T00:21:09.713 INFO:journalctl@ceph.mds.g.smithi118.stdout:Mar 24 00:21:09 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-g[93896]: 10: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x558bf58de6bc] 2024-03-24T00:21:09.713 INFO:journalctl@ceph.mds.g.smithi118.stdout:Mar 24 00:21:09 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-g[93896]: 11: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x558bf58c81ff] 2024-03-24T00:21:09.713 INFO:journalctl@ceph.mds.g.smithi118.stdout:Mar 24 00:21:09 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-g[93896]: 12: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7f1e635d0488] 2024-03-24T00:21:09.713 INFO:journalctl@ceph.mds.g.smithi118.stdout:Mar 24 00:21:09 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-g[93896]: 13: (DispatchQueue::entry()+0x51f) [0x7f1e635cd60f] 2024-03-24T00:21:09.713 INFO:journalctl@ceph.mds.g.smithi118.stdout:Mar 24 00:21:09 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-g[93896]: 14: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f1e636a0e81] 2024-03-24T00:21:09.713 INFO:journalctl@ceph.mds.g.smithi118.stdout:Mar 24 00:21:09 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-g[93896]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7f1e620bb1ca] 2024-03-24T00:21:09.713 INFO:journalctl@ceph.mds.g.smithi118.stdout:Mar 24 00:21:09 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-g[93896]: 16: clone() 2024-03-24T00:21:09.713 INFO:journalctl@ceph.mds.g.smithi118.stdout:Mar 24 00:21:09 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-g[93896]: 2024-03-24T00:21:09.714 INFO:journalctl@ceph.mds.g.smithi118.stdout:Mar 24 00:21:09 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-g[93896]: 0> 2024-03-24T00:21:09.352+0000 7f1e57a00700 -1 *** Caught signal (Aborted) ** 2024-03-24T00:21:09.714 INFO:journalctl@ceph.mds.g.smithi118.stdout:Mar 24 00:21:09 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-g[93896]: in thread 7f1e57a00700 thread_name:ms_dispatch 2024-03-24T00:21:09.714 INFO:journalctl@ceph.mds.g.smithi118.stdout:Mar 24 00:21:09 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-g[93896]: 2024-03-24T00:21:09.714 INFO:journalctl@ceph.mds.g.smithi118.stdout:Mar 24 00:21:09 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-g[93896]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-24T00:21:09.714 INFO:journalctl@ceph.mds.g.smithi118.stdout:Mar 24 00:21:09 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-g[93896]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f1e620c5d20] 2024-03-24T00:21:09.714 INFO:journalctl@ceph.mds.g.smithi118.stdout:Mar 24 00:21:09 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-g[93896]: 2: gsignal() 2024-03-24T00:21:09.714 INFO:journalctl@ceph.mds.g.smithi118.stdout:Mar 24 00:21:09 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-g[93896]: 3: abort() 2024-03-24T00:21:09.714 INFO:journalctl@ceph.mds.g.smithi118.stdout:Mar 24 00:21:09 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-g[93896]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x18f) [0x7f1e6333e4fb] 2024-03-24T00:21:09.714 INFO:journalctl@ceph.mds.g.smithi118.stdout:Mar 24 00:21:09 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-g[93896]: 5: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7f1e6333e667] 2024-03-24T00:21:09.714 INFO:journalctl@ceph.mds.g.smithi118.stdout:Mar 24 00:21:09 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-g[93896]: 6: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x558bf5b10223] 2024-03-24T00:21:09.714 INFO:journalctl@ceph.mds.g.smithi118.stdout:Mar 24 00:21:09 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-g[93896]: 7: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x558bf598a5d6] 2024-03-24T00:21:09.714 INFO:journalctl@ceph.mds.g.smithi118.stdout:Mar 24 00:21:09 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-g[93896]: 8: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x558bf598ada1] 2024-03-24T00:21:09.714 INFO:journalctl@ceph.mds.g.smithi118.stdout:Mar 24 00:21:09 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-g[93896]: 9: (Server::handle_client_rename(boost::intrusive_ptr const&)+0x259d) [0x558bf59c797d] 2024-03-24T00:21:09.714 INFO:journalctl@ceph.mds.g.smithi118.stdout:Mar 24 00:21:09 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-g[93896]: 10: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x558bf59dd233] 2024-03-24T00:21:09.715 INFO:journalctl@ceph.mds.g.smithi118.stdout:Mar 24 00:21:09 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-g[93896]: 11: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x558bf59df197] 2024-03-24T00:21:09.715 INFO:journalctl@ceph.mds.g.smithi118.stdout:Mar 24 00:21:09 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-g[93896]: 12: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x558bf58de11b] 2024-03-24T00:21:09.715 INFO:journalctl@ceph.mds.g.smithi118.stdout:Mar 24 00:21:09 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-g[93896]: 13: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x558bf58de6bc] 2024-03-24T00:21:09.715 INFO:journalctl@ceph.mds.g.smithi118.stdout:Mar 24 00:21:09 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-g[93896]: 14: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x558bf58c81ff] 2024-03-24T00:21:09.715 INFO:journalctl@ceph.mds.g.smithi118.stdout:Mar 24 00:21:09 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-g[93896]: 15: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7f1e635d0488] 2024-03-24T00:21:09.715 INFO:journalctl@ceph.mds.g.smithi118.stdout:Mar 24 00:21:09 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-g[93896]: 16: (DispatchQueue::entry()+0x51f) [0x7f1e635cd60f] 2024-03-24T00:21:09.715 INFO:journalctl@ceph.mds.g.smithi118.stdout:Mar 24 00:21:09 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-g[93896]: 17: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f1e636a0e81] 2024-03-24T00:21:09.715 INFO:journalctl@ceph.mds.g.smithi118.stdout:Mar 24 00:21:09 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-g[93896]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7f1e620bb1ca] 2024-03-24T00:21:09.715 INFO:journalctl@ceph.mds.g.smithi118.stdout:Mar 24 00:21:09 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-g[93896]: 19: clone() 2024-03-24T00:21:09.716 INFO:journalctl@ceph.mds.g.smithi118.stdout:Mar 24 00:21:09 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-g[93896]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-03-24T00:21:09.716 INFO:journalctl@ceph.mds.g.smithi118.stdout:Mar 24 00:21:09 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-g[93896]: 2024-03-24T00:21:09.716 INFO:journalctl@ceph.mds.g.smithi118.stdout:Mar 24 00:21:09 smithi118 podman[94168]: 2024-03-24 00:21:09.463435869 +0000 UTC m=+0.040913248 container died b08cc13c723022d35cf73076cb46d7bd86645c672c376533811688dc205c3f7e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-g, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, GIT_BRANCH=HEAD, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240319, GIT_CLEAN=True, org.label-schema.vendor=CentOS) 2024-03-24T00:21:09.716 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:21:09 smithi193 ceph-mon[36878]: mds.? [v2:172.21.15.193:6834/799001823,v1:172.21.15.193:6835/799001823] up:rejoin 2024-03-24T00:21:09.716 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:21:09 smithi193 ceph-mon[36878]: mds.? [v2:172.21.15.118:6838/3808689299,v1:172.21.15.118:6839/3808689299] up:rejoin 2024-03-24T00:21:09.716 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:21:09 smithi193 ceph-mon[36878]: fsmap cephfs:5/5 3 up:standby-replay 4 up:standby 2 up:rejoin 3 up:active 2024-03-24T00:21:09.717 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:21:09 smithi193 ceph-mon[36878]: daemon mds.f is now active in filesystem cephfs as rank 1 2024-03-24T00:21:09.717 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:21:09 smithi193 ceph-mon[36878]: daemon mds.g is now active in filesystem cephfs as rank 3 2024-03-24T00:21:09.718 INFO:journalctl@ceph.mds.f.smithi193.stdout:Mar 24 00:21:09 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-f[71409]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: In function 'void Locker::set_xlocks_done(MutationImpl*, bool)' thread 7f1aa164a700 time 2024-03-24T00:21:09.373126+0000 2024-03-24T00:21:09.718 INFO:journalctl@ceph.mds.f.smithi193.stdout:Mar 24 00:21:09 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-f[71409]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: 626: FAILED ceph_assert(obj->is_auth()) 2024-03-24T00:21:09.718 INFO:journalctl@ceph.mds.f.smithi193.stdout:Mar 24 00:21:09 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-f[71409]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-24T00:21:09.718 INFO:journalctl@ceph.mds.f.smithi193.stdout:Mar 24 00:21:09 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-f[71409]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x135) [0x7f1aabc744a1] 2024-03-24T00:21:09.718 INFO:journalctl@ceph.mds.f.smithi193.stdout:Mar 24 00:21:09 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-f[71409]: 2: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7f1aabc74667] 2024-03-24T00:21:09.718 INFO:journalctl@ceph.mds.f.smithi193.stdout:Mar 24 00:21:09 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-f[71409]: 3: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x55be80200223] 2024-03-24T00:21:09.719 INFO:journalctl@ceph.mds.f.smithi193.stdout:Mar 24 00:21:09 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-f[71409]: 4: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x55be8007a5d6] 2024-03-24T00:21:09.719 INFO:journalctl@ceph.mds.f.smithi193.stdout:Mar 24 00:21:09 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-f[71409]: 5: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x55be8007ada1] 2024-03-24T00:21:09.719 INFO:journalctl@ceph.mds.f.smithi193.stdout:Mar 24 00:21:09 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-f[71409]: 6: (Server::handle_client_rename(boost::intrusive_ptr const&)+0x259d) [0x55be800b797d] 2024-03-24T00:21:09.719 INFO:journalctl@ceph.mds.f.smithi193.stdout:Mar 24 00:21:09 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-f[71409]: 7: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x55be800cd233] 2024-03-24T00:21:09.719 INFO:journalctl@ceph.mds.f.smithi193.stdout:Mar 24 00:21:09 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-f[71409]: 8: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x55be800cf197] 2024-03-24T00:21:09.719 INFO:journalctl@ceph.mds.f.smithi193.stdout:Mar 24 00:21:09 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-f[71409]: 9: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x55be7ffce11b] 2024-03-24T00:21:09.719 INFO:journalctl@ceph.mds.f.smithi193.stdout:Mar 24 00:21:09 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-f[71409]: 10: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x55be7ffce6bc] 2024-03-24T00:21:09.719 INFO:journalctl@ceph.mds.f.smithi193.stdout:Mar 24 00:21:09 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-f[71409]: 11: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x55be7ffb81ff] 2024-03-24T00:21:09.719 INFO:journalctl@ceph.mds.f.smithi193.stdout:Mar 24 00:21:09 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-f[71409]: 12: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7f1aabf06488] 2024-03-24T00:21:09.719 INFO:journalctl@ceph.mds.f.smithi193.stdout:Mar 24 00:21:09 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-f[71409]: 13: (DispatchQueue::entry()+0x51f) [0x7f1aabf0360f] 2024-03-24T00:21:09.719 INFO:journalctl@ceph.mds.f.smithi193.stdout:Mar 24 00:21:09 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-f[71409]: 14: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f1aabfd6e81] 2024-03-24T00:21:09.719 INFO:journalctl@ceph.mds.f.smithi193.stdout:Mar 24 00:21:09 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-f[71409]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7f1aaa9f11ca] 2024-03-24T00:21:09.719 INFO:journalctl@ceph.mds.f.smithi193.stdout:Mar 24 00:21:09 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-f[71409]: 16: clone() 2024-03-24T00:21:09.719 INFO:journalctl@ceph.mds.f.smithi193.stdout:Mar 24 00:21:09 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-f[71409]: 2024-03-24T00:21:09.373+0000 7f1aa164a700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: In function 'void Locker::set_xlocks_done(MutationImpl*, bool)' thread 7f1aa164a700 time 2024-03-24T00:21:09.373126+0000 2024-03-24T00:21:09.720 INFO:journalctl@ceph.mds.f.smithi193.stdout:Mar 24 00:21:09 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-f[71409]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: 626: FAILED ceph_assert(obj->is_auth()) 2024-03-24T00:21:09.720 INFO:journalctl@ceph.mds.f.smithi193.stdout:Mar 24 00:21:09 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-f[71409]: 2024-03-24T00:21:09.720 INFO:journalctl@ceph.mds.f.smithi193.stdout:Mar 24 00:21:09 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-f[71409]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-24T00:21:09.720 INFO:journalctl@ceph.mds.f.smithi193.stdout:Mar 24 00:21:09 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-f[71409]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x135) [0x7f1aabc744a1] 2024-03-24T00:21:09.720 INFO:journalctl@ceph.mds.f.smithi193.stdout:Mar 24 00:21:09 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-f[71409]: 2: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7f1aabc74667] 2024-03-24T00:21:09.720 INFO:journalctl@ceph.mds.f.smithi193.stdout:Mar 24 00:21:09 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-f[71409]: 3: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x55be80200223] 2024-03-24T00:21:09.720 INFO:journalctl@ceph.mds.f.smithi193.stdout:Mar 24 00:21:09 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-f[71409]: 4: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x55be8007a5d6] 2024-03-24T00:21:09.720 INFO:journalctl@ceph.mds.f.smithi193.stdout:Mar 24 00:21:09 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-f[71409]: 5: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x55be8007ada1] 2024-03-24T00:21:09.720 INFO:journalctl@ceph.mds.f.smithi193.stdout:Mar 24 00:21:09 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-f[71409]: 6: (Server::handle_client_rename(boost::intrusive_ptr const&)+0x259d) [0x55be800b797d] 2024-03-24T00:21:09.720 INFO:journalctl@ceph.mds.f.smithi193.stdout:Mar 24 00:21:09 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-f[71409]: 7: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x55be800cd233] 2024-03-24T00:21:09.720 INFO:journalctl@ceph.mds.f.smithi193.stdout:Mar 24 00:21:09 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-f[71409]: 8: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x55be800cf197] 2024-03-24T00:21:09.720 INFO:journalctl@ceph.mds.f.smithi193.stdout:Mar 24 00:21:09 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-f[71409]: 9: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x55be7ffce11b] 2024-03-24T00:21:09.720 INFO:journalctl@ceph.mds.f.smithi193.stdout:Mar 24 00:21:09 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-f[71409]: 10: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x55be7ffce6bc] 2024-03-24T00:21:09.720 INFO:journalctl@ceph.mds.f.smithi193.stdout:Mar 24 00:21:09 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-f[71409]: 11: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x55be7ffb81ff] 2024-03-24T00:21:09.721 INFO:journalctl@ceph.mds.f.smithi193.stdout:Mar 24 00:21:09 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-f[71409]: 12: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7f1aabf06488] 2024-03-24T00:21:09.721 INFO:journalctl@ceph.mds.f.smithi193.stdout:Mar 24 00:21:09 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-f[71409]: 13: (DispatchQueue::entry()+0x51f) [0x7f1aabf0360f] 2024-03-24T00:21:09.721 INFO:journalctl@ceph.mds.f.smithi193.stdout:Mar 24 00:21:09 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-f[71409]: 14: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f1aabfd6e81] 2024-03-24T00:21:09.721 INFO:journalctl@ceph.mds.f.smithi193.stdout:Mar 24 00:21:09 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-f[71409]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7f1aaa9f11ca] 2024-03-24T00:21:09.721 INFO:journalctl@ceph.mds.f.smithi193.stdout:Mar 24 00:21:09 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-f[71409]: 16: clone() 2024-03-24T00:21:09.721 INFO:journalctl@ceph.mds.f.smithi193.stdout:Mar 24 00:21:09 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-f[71409]: 2024-03-24T00:21:09.721 INFO:journalctl@ceph.mds.f.smithi193.stdout:Mar 24 00:21:09 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-f[71409]: *** Caught signal (Aborted) ** 2024-03-24T00:21:09.721 INFO:journalctl@ceph.mds.f.smithi193.stdout:Mar 24 00:21:09 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-f[71409]: in thread 7f1aa164a700 thread_name:ms_dispatch 2024-03-24T00:21:09.721 INFO:journalctl@ceph.mds.f.smithi193.stdout:Mar 24 00:21:09 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-f[71409]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-24T00:21:09.721 INFO:journalctl@ceph.mds.f.smithi193.stdout:Mar 24 00:21:09 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-f[71409]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f1aaa9fbd20] 2024-03-24T00:21:09.721 INFO:journalctl@ceph.mds.f.smithi193.stdout:Mar 24 00:21:09 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-f[71409]: 2: gsignal() 2024-03-24T00:21:09.721 INFO:journalctl@ceph.mds.f.smithi193.stdout:Mar 24 00:21:09 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-f[71409]: 3: abort() 2024-03-24T00:21:09.721 INFO:journalctl@ceph.mds.f.smithi193.stdout:Mar 24 00:21:09 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-f[71409]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x18f) [0x7f1aabc744fb] 2024-03-24T00:21:09.721 INFO:journalctl@ceph.mds.f.smithi193.stdout:Mar 24 00:21:09 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-f[71409]: 5: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7f1aabc74667] 2024-03-24T00:21:09.722 INFO:journalctl@ceph.mds.f.smithi193.stdout:Mar 24 00:21:09 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-f[71409]: 6: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x55be80200223] 2024-03-24T00:21:09.722 INFO:journalctl@ceph.mds.f.smithi193.stdout:Mar 24 00:21:09 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-f[71409]: 7: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x55be8007a5d6] 2024-03-24T00:21:09.722 INFO:journalctl@ceph.mds.f.smithi193.stdout:Mar 24 00:21:09 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-f[71409]: 8: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x55be8007ada1] 2024-03-24T00:21:09.722 INFO:journalctl@ceph.mds.f.smithi193.stdout:Mar 24 00:21:09 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-f[71409]: 9: (Server::handle_client_rename(boost::intrusive_ptr const&)+0x259d) [0x55be800b797d] 2024-03-24T00:21:09.722 INFO:journalctl@ceph.mds.f.smithi193.stdout:Mar 24 00:21:09 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-f[71409]: 10: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x55be800cd233] 2024-03-24T00:21:09.722 INFO:journalctl@ceph.mds.f.smithi193.stdout:Mar 24 00:21:09 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-f[71409]: 11: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x55be800cf197] 2024-03-24T00:21:09.722 INFO:journalctl@ceph.mds.f.smithi193.stdout:Mar 24 00:21:09 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-f[71409]: 12: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x55be7ffce11b] 2024-03-24T00:21:09.722 INFO:journalctl@ceph.mds.f.smithi193.stdout:Mar 24 00:21:09 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-f[71409]: 13: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x55be7ffce6bc] 2024-03-24T00:21:09.722 INFO:journalctl@ceph.mds.f.smithi193.stdout:Mar 24 00:21:09 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-f[71409]: 14: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x55be7ffb81ff] 2024-03-24T00:21:09.722 INFO:journalctl@ceph.mds.f.smithi193.stdout:Mar 24 00:21:09 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-f[71409]: 15: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7f1aabf06488] 2024-03-24T00:21:09.722 INFO:journalctl@ceph.mds.f.smithi193.stdout:Mar 24 00:21:09 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-f[71409]: 16: (DispatchQueue::entry()+0x51f) [0x7f1aabf0360f] 2024-03-24T00:21:09.722 INFO:journalctl@ceph.mds.f.smithi193.stdout:Mar 24 00:21:09 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-f[71409]: 17: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f1aabfd6e81] 2024-03-24T00:21:09.722 INFO:journalctl@ceph.mds.f.smithi193.stdout:Mar 24 00:21:09 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-f[71409]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7f1aaa9f11ca] 2024-03-24T00:21:09.723 INFO:journalctl@ceph.mds.f.smithi193.stdout:Mar 24 00:21:09 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-f[71409]: 19: clone() 2024-03-24T00:21:09.723 INFO:journalctl@ceph.mds.f.smithi193.stdout:Mar 24 00:21:09 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-f[71409]: 2024-03-24T00:21:09.376+0000 7f1aa164a700 -1 *** Caught signal (Aborted) ** 2024-03-24T00:21:09.723 INFO:journalctl@ceph.mds.f.smithi193.stdout:Mar 24 00:21:09 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-f[71409]: in thread 7f1aa164a700 thread_name:ms_dispatch 2024-03-24T00:21:09.723 INFO:journalctl@ceph.mds.f.smithi193.stdout:Mar 24 00:21:09 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-f[71409]: 2024-03-24T00:21:09.723 INFO:journalctl@ceph.mds.f.smithi193.stdout:Mar 24 00:21:09 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-f[71409]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-24T00:21:09.723 INFO:journalctl@ceph.mds.f.smithi193.stdout:Mar 24 00:21:09 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-f[71409]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f1aaa9fbd20] 2024-03-24T00:21:09.723 INFO:journalctl@ceph.mds.f.smithi193.stdout:Mar 24 00:21:09 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-f[71409]: 2: gsignal() 2024-03-24T00:21:09.723 INFO:journalctl@ceph.mds.f.smithi193.stdout:Mar 24 00:21:09 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-f[71409]: 3: abort() 2024-03-24T00:21:09.723 INFO:journalctl@ceph.mds.f.smithi193.stdout:Mar 24 00:21:09 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-f[71409]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x18f) [0x7f1aabc744fb] 2024-03-24T00:21:09.723 INFO:journalctl@ceph.mds.f.smithi193.stdout:Mar 24 00:21:09 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-f[71409]: 5: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7f1aabc74667] 2024-03-24T00:21:09.723 INFO:journalctl@ceph.mds.f.smithi193.stdout:Mar 24 00:21:09 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-f[71409]: 6: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x55be80200223] 2024-03-24T00:21:09.723 INFO:journalctl@ceph.mds.f.smithi193.stdout:Mar 24 00:21:09 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-f[71409]: 7: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x55be8007a5d6] 2024-03-24T00:21:09.723 INFO:journalctl@ceph.mds.f.smithi193.stdout:Mar 24 00:21:09 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-f[71409]: 8: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x55be8007ada1] 2024-03-24T00:21:09.723 INFO:journalctl@ceph.mds.f.smithi193.stdout:Mar 24 00:21:09 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-f[71409]: 9: (Server::handle_client_rename(boost::intrusive_ptr const&)+0x259d) [0x55be800b797d] 2024-03-24T00:21:09.723 INFO:journalctl@ceph.mds.f.smithi193.stdout:Mar 24 00:21:09 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-f[71409]: 10: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x55be800cd233] 2024-03-24T00:21:09.724 INFO:journalctl@ceph.mds.f.smithi193.stdout:Mar 24 00:21:09 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-f[71409]: 11: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x55be800cf197] 2024-03-24T00:21:09.724 INFO:journalctl@ceph.mds.f.smithi193.stdout:Mar 24 00:21:09 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-f[71409]: 12: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x55be7ffce11b] 2024-03-24T00:21:09.724 INFO:journalctl@ceph.mds.f.smithi193.stdout:Mar 24 00:21:09 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-f[71409]: 13: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x55be7ffce6bc] 2024-03-24T00:21:09.724 INFO:journalctl@ceph.mds.f.smithi193.stdout:Mar 24 00:21:09 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-f[71409]: 14: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x55be7ffb81ff] 2024-03-24T00:21:09.724 INFO:journalctl@ceph.mds.f.smithi193.stdout:Mar 24 00:21:09 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-f[71409]: 15: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7f1aabf06488] 2024-03-24T00:21:09.724 INFO:journalctl@ceph.mds.f.smithi193.stdout:Mar 24 00:21:09 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-f[71409]: 16: (DispatchQueue::entry()+0x51f) [0x7f1aabf0360f] 2024-03-24T00:21:09.724 INFO:journalctl@ceph.mds.f.smithi193.stdout:Mar 24 00:21:09 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-f[71409]: 17: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f1aabfd6e81] 2024-03-24T00:21:09.724 INFO:journalctl@ceph.mds.f.smithi193.stdout:Mar 24 00:21:09 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-f[71409]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7f1aaa9f11ca] 2024-03-24T00:21:09.724 INFO:journalctl@ceph.mds.f.smithi193.stdout:Mar 24 00:21:09 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-f[71409]: 19: clone() 2024-03-24T00:21:09.724 INFO:journalctl@ceph.mds.f.smithi193.stdout:Mar 24 00:21:09 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-f[71409]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-03-24T00:21:09.724 INFO:journalctl@ceph.mds.f.smithi193.stdout:Mar 24 00:21:09 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-f[71409]: 2024-03-24T00:21:09.724 INFO:journalctl@ceph.mds.f.smithi193.stdout:Mar 24 00:21:09 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-f[71409]: -1> 2024-03-24T00:21:09.373+0000 7f1aa164a700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: In function 'void Locker::set_xlocks_done(MutationImpl*, bool)' thread 7f1aa164a700 time 2024-03-24T00:21:09.373126+0000 2024-03-24T00:21:09.724 INFO:journalctl@ceph.mds.f.smithi193.stdout:Mar 24 00:21:09 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-f[71409]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: 626: FAILED ceph_assert(obj->is_auth()) 2024-03-24T00:21:09.724 INFO:journalctl@ceph.mds.f.smithi193.stdout:Mar 24 00:21:09 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-f[71409]: 2024-03-24T00:21:09.725 INFO:journalctl@ceph.mds.f.smithi193.stdout:Mar 24 00:21:09 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-f[71409]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-24T00:21:09.725 INFO:journalctl@ceph.mds.f.smithi193.stdout:Mar 24 00:21:09 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-f[71409]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x135) [0x7f1aabc744a1] 2024-03-24T00:21:09.725 INFO:journalctl@ceph.mds.f.smithi193.stdout:Mar 24 00:21:09 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-f[71409]: 2: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7f1aabc74667] 2024-03-24T00:21:09.725 INFO:journalctl@ceph.mds.f.smithi193.stdout:Mar 24 00:21:09 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-f[71409]: 3: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x55be80200223] 2024-03-24T00:21:09.725 INFO:journalctl@ceph.mds.f.smithi193.stdout:Mar 24 00:21:09 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-f[71409]: 4: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x55be8007a5d6] 2024-03-24T00:21:09.725 INFO:journalctl@ceph.mds.f.smithi193.stdout:Mar 24 00:21:09 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-f[71409]: 5: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x55be8007ada1] 2024-03-24T00:21:09.725 INFO:journalctl@ceph.mds.f.smithi193.stdout:Mar 24 00:21:09 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-f[71409]: 6: (Server::handle_client_rename(boost::intrusive_ptr const&)+0x259d) [0x55be800b797d] 2024-03-24T00:21:09.725 INFO:journalctl@ceph.mds.f.smithi193.stdout:Mar 24 00:21:09 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-f[71409]: 7: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x55be800cd233] 2024-03-24T00:21:09.725 INFO:journalctl@ceph.mds.f.smithi193.stdout:Mar 24 00:21:09 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-f[71409]: 8: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x55be800cf197] 2024-03-24T00:21:09.725 INFO:journalctl@ceph.mds.f.smithi193.stdout:Mar 24 00:21:09 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-f[71409]: 9: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x55be7ffce11b] 2024-03-24T00:21:09.725 INFO:journalctl@ceph.mds.f.smithi193.stdout:Mar 24 00:21:09 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-f[71409]: 10: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x55be7ffce6bc] 2024-03-24T00:21:09.725 INFO:journalctl@ceph.mds.f.smithi193.stdout:Mar 24 00:21:09 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-f[71409]: 11: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x55be7ffb81ff] 2024-03-24T00:21:09.725 INFO:journalctl@ceph.mds.f.smithi193.stdout:Mar 24 00:21:09 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-f[71409]: 12: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7f1aabf06488] 2024-03-24T00:21:09.725 INFO:journalctl@ceph.mds.f.smithi193.stdout:Mar 24 00:21:09 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-f[71409]: 13: (DispatchQueue::entry()+0x51f) [0x7f1aabf0360f] 2024-03-24T00:21:09.726 INFO:journalctl@ceph.mds.f.smithi193.stdout:Mar 24 00:21:09 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-f[71409]: 14: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f1aabfd6e81] 2024-03-24T00:21:09.726 INFO:journalctl@ceph.mds.f.smithi193.stdout:Mar 24 00:21:09 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-f[71409]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7f1aaa9f11ca] 2024-03-24T00:21:09.726 INFO:journalctl@ceph.mds.f.smithi193.stdout:Mar 24 00:21:09 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-f[71409]: 16: clone() 2024-03-24T00:21:09.726 INFO:journalctl@ceph.mds.f.smithi193.stdout:Mar 24 00:21:09 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-f[71409]: 2024-03-24T00:21:09.726 INFO:journalctl@ceph.mds.f.smithi193.stdout:Mar 24 00:21:09 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-f[71409]: 0> 2024-03-24T00:21:09.376+0000 7f1aa164a700 -1 *** Caught signal (Aborted) ** 2024-03-24T00:21:09.726 INFO:journalctl@ceph.mds.f.smithi193.stdout:Mar 24 00:21:09 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-f[71409]: in thread 7f1aa164a700 thread_name:ms_dispatch 2024-03-24T00:21:09.726 INFO:journalctl@ceph.mds.f.smithi193.stdout:Mar 24 00:21:09 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-f[71409]: 2024-03-24T00:21:09.726 INFO:journalctl@ceph.mds.f.smithi193.stdout:Mar 24 00:21:09 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-f[71409]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-24T00:21:09.726 INFO:journalctl@ceph.mds.f.smithi193.stdout:Mar 24 00:21:09 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-f[71409]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f1aaa9fbd20] 2024-03-24T00:21:09.727 INFO:journalctl@ceph.mds.f.smithi193.stdout:Mar 24 00:21:09 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-f[71409]: 2: gsignal() 2024-03-24T00:21:09.727 INFO:journalctl@ceph.mds.f.smithi193.stdout:Mar 24 00:21:09 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-f[71409]: 3: abort() 2024-03-24T00:21:09.727 INFO:journalctl@ceph.mds.f.smithi193.stdout:Mar 24 00:21:09 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-f[71409]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x18f) [0x7f1aabc744fb] 2024-03-24T00:21:09.727 INFO:journalctl@ceph.mds.f.smithi193.stdout:Mar 24 00:21:09 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-f[71409]: 5: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7f1aabc74667] 2024-03-24T00:21:09.727 INFO:journalctl@ceph.mds.f.smithi193.stdout:Mar 24 00:21:09 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-f[71409]: 6: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x55be80200223] 2024-03-24T00:21:09.727 INFO:journalctl@ceph.mds.f.smithi193.stdout:Mar 24 00:21:09 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-f[71409]: 7: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x55be8007a5d6] 2024-03-24T00:21:09.727 INFO:journalctl@ceph.mds.f.smithi193.stdout:Mar 24 00:21:09 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-f[71409]: 8: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x55be8007ada1] 2024-03-24T00:21:09.727 INFO:journalctl@ceph.mds.f.smithi193.stdout:Mar 24 00:21:09 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-f[71409]: 9: (Server::handle_client_rename(boost::intrusive_ptr const&)+0x259d) [0x55be800b797d] 2024-03-24T00:21:09.727 INFO:journalctl@ceph.mds.f.smithi193.stdout:Mar 24 00:21:09 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-f[71409]: 10: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x55be800cd233] 2024-03-24T00:21:09.727 INFO:journalctl@ceph.mds.f.smithi193.stdout:Mar 24 00:21:09 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-f[71409]: 11: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x55be800cf197] 2024-03-24T00:21:09.727 INFO:journalctl@ceph.mds.f.smithi193.stdout:Mar 24 00:21:09 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-f[71409]: 12: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x55be7ffce11b] 2024-03-24T00:21:09.727 INFO:journalctl@ceph.mds.f.smithi193.stdout:Mar 24 00:21:09 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-f[71409]: 13: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x55be7ffce6bc] 2024-03-24T00:21:09.728 INFO:journalctl@ceph.mds.f.smithi193.stdout:Mar 24 00:21:09 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-f[71409]: 14: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x55be7ffb81ff] 2024-03-24T00:21:09.728 INFO:journalctl@ceph.mds.f.smithi193.stdout:Mar 24 00:21:09 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-f[71409]: 15: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7f1aabf06488] 2024-03-24T00:21:09.728 INFO:journalctl@ceph.mds.f.smithi193.stdout:Mar 24 00:21:09 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-f[71409]: 16: (DispatchQueue::entry()+0x51f) [0x7f1aabf0360f] 2024-03-24T00:21:09.728 INFO:journalctl@ceph.mds.f.smithi193.stdout:Mar 24 00:21:09 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-f[71409]: 17: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f1aabfd6e81] 2024-03-24T00:21:09.728 INFO:journalctl@ceph.mds.f.smithi193.stdout:Mar 24 00:21:09 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-f[71409]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7f1aaa9f11ca] 2024-03-24T00:21:09.728 INFO:journalctl@ceph.mds.f.smithi193.stdout:Mar 24 00:21:09 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-f[71409]: 19: clone() 2024-03-24T00:21:09.728 INFO:journalctl@ceph.mds.f.smithi193.stdout:Mar 24 00:21:09 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-f[71409]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-03-24T00:21:09.728 INFO:journalctl@ceph.mds.f.smithi193.stdout:Mar 24 00:21:09 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-f[71409]: 2024-03-24T00:21:09.728 INFO:journalctl@ceph.mds.f.smithi193.stdout:Mar 24 00:21:09 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-f[71409]: -5> 2024-03-24T00:21:09.373+0000 7f1aa164a700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: In function 'void Locker::set_xlocks_done(MutationImpl*, bool)' thread 7f1aa164a700 time 2024-03-24T00:21:09.373126+0000 2024-03-24T00:21:09.728 INFO:journalctl@ceph.mds.f.smithi193.stdout:Mar 24 00:21:09 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-f[71409]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: 626: FAILED ceph_assert(obj->is_auth()) 2024-03-24T00:21:09.728 INFO:journalctl@ceph.mds.f.smithi193.stdout:Mar 24 00:21:09 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-f[71409]: 2024-03-24T00:21:09.728 INFO:journalctl@ceph.mds.f.smithi193.stdout:Mar 24 00:21:09 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-f[71409]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-24T00:21:09.728 INFO:journalctl@ceph.mds.f.smithi193.stdout:Mar 24 00:21:09 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-f[71409]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x135) [0x7f1aabc744a1] 2024-03-24T00:21:09.728 INFO:journalctl@ceph.mds.f.smithi193.stdout:Mar 24 00:21:09 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-f[71409]: 2: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7f1aabc74667] 2024-03-24T00:21:09.728 INFO:journalctl@ceph.mds.f.smithi193.stdout:Mar 24 00:21:09 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-f[71409]: 3: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x55be80200223] 2024-03-24T00:21:09.729 INFO:journalctl@ceph.mds.f.smithi193.stdout:Mar 24 00:21:09 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-f[71409]: 4: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x55be8007a5d6] 2024-03-24T00:21:09.729 INFO:journalctl@ceph.mds.f.smithi193.stdout:Mar 24 00:21:09 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-f[71409]: 5: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x55be8007ada1] 2024-03-24T00:21:09.729 INFO:journalctl@ceph.mds.f.smithi193.stdout:Mar 24 00:21:09 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-f[71409]: 6: (Server::handle_client_rename(boost::intrusive_ptr const&)+0x259d) [0x55be800b797d] 2024-03-24T00:21:09.729 INFO:journalctl@ceph.mds.f.smithi193.stdout:Mar 24 00:21:09 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-f[71409]: 7: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x55be800cd233] 2024-03-24T00:21:09.729 INFO:journalctl@ceph.mds.f.smithi193.stdout:Mar 24 00:21:09 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-f[71409]: 8: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x55be800cf197] 2024-03-24T00:21:09.729 INFO:journalctl@ceph.mds.f.smithi193.stdout:Mar 24 00:21:09 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-f[71409]: 9: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x55be7ffce11b] 2024-03-24T00:21:09.729 INFO:journalctl@ceph.mds.f.smithi193.stdout:Mar 24 00:21:09 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-f[71409]: 10: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x55be7ffce6bc] 2024-03-24T00:21:09.729 INFO:journalctl@ceph.mds.f.smithi193.stdout:Mar 24 00:21:09 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-f[71409]: 11: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x55be7ffb81ff] 2024-03-24T00:21:09.729 INFO:journalctl@ceph.mds.f.smithi193.stdout:Mar 24 00:21:09 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-f[71409]: 12: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7f1aabf06488] 2024-03-24T00:21:09.729 INFO:journalctl@ceph.mds.f.smithi193.stdout:Mar 24 00:21:09 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-f[71409]: 13: (DispatchQueue::entry()+0x51f) [0x7f1aabf0360f] 2024-03-24T00:21:09.729 INFO:journalctl@ceph.mds.f.smithi193.stdout:Mar 24 00:21:09 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-f[71409]: 14: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f1aabfd6e81] 2024-03-24T00:21:09.729 INFO:journalctl@ceph.mds.f.smithi193.stdout:Mar 24 00:21:09 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-f[71409]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7f1aaa9f11ca] 2024-03-24T00:21:09.729 INFO:journalctl@ceph.mds.f.smithi193.stdout:Mar 24 00:21:09 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-f[71409]: 16: clone() 2024-03-24T00:21:09.730 INFO:journalctl@ceph.mds.f.smithi193.stdout:Mar 24 00:21:09 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-f[71409]: 2024-03-24T00:21:09.730 INFO:journalctl@ceph.mds.f.smithi193.stdout:Mar 24 00:21:09 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-f[71409]: -4> 2024-03-24T00:21:09.376+0000 7f1aa164a700 -1 *** Caught signal (Aborted) ** 2024-03-24T00:21:09.730 INFO:journalctl@ceph.mds.f.smithi193.stdout:Mar 24 00:21:09 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-f[71409]: in thread 7f1aa164a700 thread_name:ms_dispatch 2024-03-24T00:21:09.730 INFO:journalctl@ceph.mds.f.smithi193.stdout:Mar 24 00:21:09 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-f[71409]: 2024-03-24T00:21:09.730 INFO:journalctl@ceph.mds.f.smithi193.stdout:Mar 24 00:21:09 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-f[71409]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-24T00:21:09.730 INFO:journalctl@ceph.mds.f.smithi193.stdout:Mar 24 00:21:09 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-f[71409]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f1aaa9fbd20] 2024-03-24T00:21:09.730 INFO:journalctl@ceph.mds.f.smithi193.stdout:Mar 24 00:21:09 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-f[71409]: 2: gsignal() 2024-03-24T00:21:09.730 INFO:journalctl@ceph.mds.f.smithi193.stdout:Mar 24 00:21:09 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-f[71409]: 3: abort() 2024-03-24T00:21:09.730 INFO:journalctl@ceph.mds.f.smithi193.stdout:Mar 24 00:21:09 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-f[71409]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x18f) [0x7f1aabc744fb] 2024-03-24T00:21:09.730 INFO:journalctl@ceph.mds.f.smithi193.stdout:Mar 24 00:21:09 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-f[71409]: 5: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7f1aabc74667] 2024-03-24T00:21:09.730 INFO:journalctl@ceph.mds.f.smithi193.stdout:Mar 24 00:21:09 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-f[71409]: 6: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x55be80200223] 2024-03-24T00:21:09.730 INFO:journalctl@ceph.mds.f.smithi193.stdout:Mar 24 00:21:09 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-f[71409]: 7: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x55be8007a5d6] 2024-03-24T00:21:09.730 INFO:journalctl@ceph.mds.f.smithi193.stdout:Mar 24 00:21:09 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-f[71409]: 8: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x55be8007ada1] 2024-03-24T00:21:09.730 INFO:journalctl@ceph.mds.f.smithi193.stdout:Mar 24 00:21:09 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-f[71409]: 9: (Server::handle_client_rename(boost::intrusive_ptr const&)+0x259d) [0x55be800b797d] 2024-03-24T00:21:09.731 INFO:journalctl@ceph.mds.f.smithi193.stdout:Mar 24 00:21:09 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-f[71409]: 10: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x55be800cd233] 2024-03-24T00:21:09.731 INFO:journalctl@ceph.mds.f.smithi193.stdout:Mar 24 00:21:09 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-f[71409]: 11: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x55be800cf197] 2024-03-24T00:21:09.731 INFO:journalctl@ceph.mds.f.smithi193.stdout:Mar 24 00:21:09 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-f[71409]: 12: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x55be7ffce11b] 2024-03-24T00:21:09.731 INFO:journalctl@ceph.mds.f.smithi193.stdout:Mar 24 00:21:09 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-f[71409]: 13: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x55be7ffce6bc] 2024-03-24T00:21:09.731 INFO:journalctl@ceph.mds.f.smithi193.stdout:Mar 24 00:21:09 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-f[71409]: 14: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x55be7ffb81ff] 2024-03-24T00:21:09.731 INFO:journalctl@ceph.mds.f.smithi193.stdout:Mar 24 00:21:09 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-f[71409]: 15: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7f1aabf06488] 2024-03-24T00:21:09.731 INFO:journalctl@ceph.mds.f.smithi193.stdout:Mar 24 00:21:09 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-f[71409]: 16: (DispatchQueue::entry()+0x51f) [0x7f1aabf0360f] 2024-03-24T00:21:09.731 INFO:journalctl@ceph.mds.f.smithi193.stdout:Mar 24 00:21:09 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-f[71409]: 17: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f1aabfd6e81] 2024-03-24T00:21:09.731 INFO:journalctl@ceph.mds.f.smithi193.stdout:Mar 24 00:21:09 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-f[71409]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7f1aaa9f11ca] 2024-03-24T00:21:09.731 INFO:journalctl@ceph.mds.f.smithi193.stdout:Mar 24 00:21:09 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-f[71409]: 19: clone() 2024-03-24T00:21:09.731 INFO:journalctl@ceph.mds.f.smithi193.stdout:Mar 24 00:21:09 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-f[71409]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-03-24T00:21:09.731 INFO:journalctl@ceph.mds.f.smithi193.stdout:Mar 24 00:21:09 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-f[71409]: 2024-03-24T00:21:09.731 INFO:journalctl@ceph.mds.f.smithi193.stdout:Mar 24 00:21:09 smithi193 podman[71748]: 2024-03-24 00:21:09.446718312 +0000 UTC m=+0.021458745 container died 2357719800f568c5c2d6078a6306ce0671cc51e7d25ad7d5cf0380eb00130b6c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-f, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, ceph=True, org.label-schema.build-date=20240319, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=) 2024-03-24T00:21:10.491 INFO:journalctl@ceph.mds.f.smithi193.stdout:Mar 24 00:21:10 smithi193 podman[71748]: 2024-03-24 00:21:10.224363739 +0000 UTC m=+0.799104181 container remove 2357719800f568c5c2d6078a6306ce0671cc51e7d25ad7d5cf0380eb00130b6c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-f, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, org.label-schema.build-date=20240319, GIT_CLEAN=True, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30) 2024-03-24T00:21:10.491 INFO:journalctl@ceph.mds.f.smithi193.stdout:Mar 24 00:21:10 smithi193 systemd[1]: ceph-d79a5420-e971-11ee-95cd-87774f69a715@mds.f.service: Main process exited, code=exited, status=134/n/a 2024-03-24T00:21:10.491 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:21:10 smithi193 ceph-mon[36878]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-03-24T00:21:10.491 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:21:10 smithi193 ceph-mon[36878]: Cluster is now healthy 2024-03-24T00:21:10.491 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:21:10 smithi193 ceph-mon[36878]: mds.? [v2:172.21.15.193:6834/799001823,v1:172.21.15.193:6835/799001823] up:active 2024-03-24T00:21:10.491 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:21:10 smithi193 ceph-mon[36878]: mds.? [v2:172.21.15.118:6838/3808689299,v1:172.21.15.118:6839/3808689299] up:active 2024-03-24T00:21:10.491 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:21:10 smithi193 ceph-mon[36878]: fsmap cephfs:5 3 up:standby-replay 4 up:standby 5 up:active 2024-03-24T00:21:10.492 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:21:10 smithi193 ceph-mon[36878]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-03-24T00:21:10.492 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:21:10 smithi193 ceph-mon[36878]: pgmap v570: 97 pgs: 97 active+clean; 136 MiB data, 732 MiB used, 1.0 TiB / 1.0 TiB avail; 482 KiB/s rd, 1.0 MiB/s wr, 214 op/s 2024-03-24T00:21:10.520 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:21:10 smithi118 ceph-mon[31535]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-03-24T00:21:10.520 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:21:10 smithi118 ceph-mon[31535]: Cluster is now healthy 2024-03-24T00:21:10.520 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:21:10 smithi118 ceph-mon[31535]: mds.? [v2:172.21.15.193:6834/799001823,v1:172.21.15.193:6835/799001823] up:active 2024-03-24T00:21:10.520 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:21:10 smithi118 ceph-mon[31535]: mds.? [v2:172.21.15.118:6838/3808689299,v1:172.21.15.118:6839/3808689299] up:active 2024-03-24T00:21:10.520 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:21:10 smithi118 ceph-mon[31535]: fsmap cephfs:5 3 up:standby-replay 4 up:standby 5 up:active 2024-03-24T00:21:10.520 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:21:10 smithi118 ceph-mon[31535]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-03-24T00:21:10.520 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:21:10 smithi118 ceph-mon[31535]: pgmap v570: 97 pgs: 97 active+clean; 136 MiB data, 732 MiB used, 1.0 TiB / 1.0 TiB avail; 482 KiB/s rd, 1.0 MiB/s wr, 214 op/s 2024-03-24T00:21:10.521 INFO:journalctl@ceph.mds.g.smithi118.stdout:Mar 24 00:21:10 smithi118 podman[94168]: 2024-03-24 00:21:10.426368388 +0000 UTC m=+1.003845811 container remove b08cc13c723022d35cf73076cb46d7bd86645c672c376533811688dc205c3f7e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-g, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240319, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, GIT_CLEAN=True, ceph=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, maintainer=Guillaume Abrioux , RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c) 2024-03-24T00:21:10.521 INFO:journalctl@ceph.mds.g.smithi118.stdout:Mar 24 00:21:10 smithi118 systemd[1]: ceph-d79a5420-e971-11ee-95cd-87774f69a715@mds.g.service: Main process exited, code=exited, status=134/n/a 2024-03-24T00:21:10.704 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:21:10 smithi189 ceph-mon[36187]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-03-24T00:21:10.704 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:21:10 smithi189 ceph-mon[36187]: Cluster is now healthy 2024-03-24T00:21:10.704 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:21:10 smithi189 ceph-mon[36187]: mds.? [v2:172.21.15.193:6834/799001823,v1:172.21.15.193:6835/799001823] up:active 2024-03-24T00:21:10.704 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:21:10 smithi189 ceph-mon[36187]: mds.? [v2:172.21.15.118:6838/3808689299,v1:172.21.15.118:6839/3808689299] up:active 2024-03-24T00:21:10.704 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:21:10 smithi189 ceph-mon[36187]: fsmap cephfs:5 3 up:standby-replay 4 up:standby 5 up:active 2024-03-24T00:21:10.704 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:21:10 smithi189 ceph-mon[36187]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-03-24T00:21:10.704 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:21:10 smithi189 ceph-mon[36187]: pgmap v570: 97 pgs: 97 active+clean; 136 MiB data, 732 MiB used, 1.0 TiB / 1.0 TiB avail; 482 KiB/s rd, 1.0 MiB/s wr, 214 op/s 2024-03-24T00:21:10.960 INFO:journalctl@ceph.mds.f.smithi193.stdout:Mar 24 00:21:10 smithi193 systemd[1]: ceph-d79a5420-e971-11ee-95cd-87774f69a715@mds.f.service: Failed with result 'exit-code'. 2024-03-24T00:21:11.196 INFO:journalctl@ceph.mds.g.smithi118.stdout:Mar 24 00:21:10 smithi118 systemd[1]: ceph-d79a5420-e971-11ee-95cd-87774f69a715@mds.g.service: Failed with result 'exit-code'. 2024-03-24T00:21:11.196 INFO:journalctl@ceph.mds.g.smithi118.stdout:Mar 24 00:21:10 smithi118 systemd[1]: ceph-d79a5420-e971-11ee-95cd-87774f69a715@mds.g.service: Consumed 1.460s CPU time. 2024-03-24T00:21:11.203 INFO:journalctl@ceph.mds.b.smithi189.stdout:Mar 24 00:21:10 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-b[66975]: 2024-03-24T00:21:10.731+0000 7f4e07ab9700 -1 mds.pinger is_rank_lagging: rank=1 was never sent ping request. 2024-03-24T00:21:11.204 INFO:journalctl@ceph.mds.b.smithi189.stdout:Mar 24 00:21:10 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-b[66975]: 2024-03-24T00:21:10.731+0000 7f4e07ab9700 -1 mds.pinger is_rank_lagging: rank=3 was never sent ping request. 2024-03-24T00:21:12.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:21:11 smithi118 ceph-mon[31535]: pgmap v571: 97 pgs: 97 active+clean; 135 MiB data, 728 MiB used, 1.0 TiB / 1.0 TiB avail; 974 KiB/s rd, 261 KiB/s wr, 246 op/s 2024-03-24T00:21:12.203 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:21:11 smithi189 ceph-mon[36187]: pgmap v571: 97 pgs: 97 active+clean; 135 MiB data, 728 MiB used, 1.0 TiB / 1.0 TiB avail; 974 KiB/s rd, 261 KiB/s wr, 246 op/s 2024-03-24T00:21:12.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:21:11 smithi193 ceph-mon[36878]: pgmap v571: 97 pgs: 97 active+clean; 135 MiB data, 728 MiB used, 1.0 TiB / 1.0 TiB avail; 974 KiB/s rd, 261 KiB/s wr, 246 op/s 2024-03-24T00:21:14.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:21:13 smithi118 ceph-mon[31535]: pgmap v572: 97 pgs: 97 active+clean; 128 MiB data, 724 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 MiB/s rd, 310 KiB/s wr, 345 op/s 2024-03-24T00:21:14.203 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:21:13 smithi189 ceph-mon[36187]: pgmap v572: 97 pgs: 97 active+clean; 128 MiB data, 724 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 MiB/s rd, 310 KiB/s wr, 345 op/s 2024-03-24T00:21:14.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:21:13 smithi193 ceph-mon[36878]: pgmap v572: 97 pgs: 97 active+clean; 128 MiB data, 724 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 MiB/s rd, 310 KiB/s wr, 345 op/s 2024-03-24T00:21:16.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:21:15 smithi118 ceph-mon[31535]: pgmap v573: 97 pgs: 97 active+clean; 131 MiB data, 727 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 MiB/s rd, 518 KiB/s wr, 357 op/s 2024-03-24T00:21:16.203 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:21:15 smithi189 ceph-mon[36187]: pgmap v573: 97 pgs: 97 active+clean; 131 MiB data, 727 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 MiB/s rd, 518 KiB/s wr, 357 op/s 2024-03-24T00:21:16.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:21:15 smithi193 ceph-mon[36878]: pgmap v573: 97 pgs: 97 active+clean; 131 MiB data, 727 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 MiB/s rd, 518 KiB/s wr, 357 op/s 2024-03-24T00:21:16.844 INFO:tasks.workunit.client.0.smithi118.stdout: 43 1603 134 1516 92 1956 222 2024-03-24T00:21:17.075 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:21:17.074+0000 7fd627e00640 1 --2- 172.21.15.118:0/736842230 >> [v2:172.21.15.193:6832/1023428412,v1:172.21.15.193:6833/1023428412] conn(0x55c55390a6a0 0x55c55390cac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:21:17.075 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:21:17.075+0000 7fd627e00640 1 -- 172.21.15.118:0/736842230 >> [v2:172.21.15.193:6832/1023428412,v1:172.21.15.193:6833/1023428412] conn(0x55c55390a6a0 msgr2=0x55c55390cac0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-03-24T00:21:17.076 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:21:17.075+0000 7fd627e00640 1 -- 172.21.15.118:0/736842230 >> [v2:172.21.15.193:6832/1023428412,v1:172.21.15.193:6833/1023428412] conn(0x55c55390a6a0 msgr2=0x55c55390cac0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-03-24T00:21:17.076 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:21:17.075+0000 7fd627e00640 1 --2- 172.21.15.118:0/736842230 >> [v2:172.21.15.193:6832/1023428412,v1:172.21.15.193:6833/1023428412] conn(0x55c55390a6a0 0x55c55390cac0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-03-24T00:21:17.076 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:21:17.075+0000 7fd627e00640 1 --2- 172.21.15.118:0/736842230 >> [v2:172.21.15.193:6832/1023428412,v1:172.21.15.193:6833/1023428412] conn(0x55c55390a6a0 0x55c55390cac0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-03-24T00:21:18.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:21:17 smithi118 ceph-mon[31535]: pgmap v574: 97 pgs: 97 active+clean; 131 MiB data, 727 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 MiB/s rd, 459 KiB/s wr, 308 op/s 2024-03-24T00:21:18.203 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:21:17 smithi189 ceph-mon[36187]: pgmap v574: 97 pgs: 97 active+clean; 131 MiB data, 727 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 MiB/s rd, 459 KiB/s wr, 308 op/s 2024-03-24T00:21:18.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:21:17 smithi193 ceph-mon[36878]: pgmap v574: 97 pgs: 97 active+clean; 131 MiB data, 727 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 MiB/s rd, 459 KiB/s wr, 308 op/s 2024-03-24T00:21:20.198 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:21:19 smithi118 ceph-mon[31535]: pgmap v575: 97 pgs: 97 active+clean; 132 MiB data, 728 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 MiB/s rd, 549 KiB/s wr, 263 op/s 2024-03-24T00:21:20.203 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:21:19 smithi189 ceph-mon[36187]: pgmap v575: 97 pgs: 97 active+clean; 132 MiB data, 728 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 MiB/s rd, 549 KiB/s wr, 263 op/s 2024-03-24T00:21:20.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:21:19 smithi193 ceph-mon[36878]: pgmap v575: 97 pgs: 97 active+clean; 132 MiB data, 728 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 MiB/s rd, 549 KiB/s wr, 263 op/s 2024-03-24T00:21:20.982 INFO:journalctl@ceph.mds.f.smithi193.stdout:Mar 24 00:21:20 smithi193 systemd[1]: ceph-d79a5420-e971-11ee-95cd-87774f69a715@mds.f.service: Scheduled restart job, restart counter is at 4. 2024-03-24T00:21:20.982 INFO:journalctl@ceph.mds.f.smithi193.stdout:Mar 24 00:21:20 smithi193 systemd[1]: Stopped Ceph mds.f for d79a5420-e971-11ee-95cd-87774f69a715. 2024-03-24T00:21:20.983 INFO:journalctl@ceph.mds.f.smithi193.stdout:Mar 24 00:21:20 smithi193 systemd[1]: Starting Ceph mds.f for d79a5420-e971-11ee-95cd-87774f69a715... 2024-03-24T00:21:20.983 INFO:journalctl@ceph.mds.f.smithi193.stdout:Mar 24 00:21:20 smithi193 podman[71853]: 2024-03-24 00:21:20.854092739 +0000 UTC m=+0.023430387 image pull c664afe56bb01db4f9afa9532c20598e201e664f2e9ba2be50db0d52674a26d2 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc 2024-03-24T00:21:20.983 INFO:journalctl@ceph.mds.f.smithi193.stdout:Mar 24 00:21:20 smithi193 podman[71853]: 2024-03-24 00:21:20.983223157 +0000 UTC m=+0.152560797 container create 72641b2d5a1ac705eefede03dcb21c26a2e124304b9f7ca43ab321a67492be1f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-f, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, GIT_CLEAN=True, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, ceph=True, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240319, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-24T00:21:21.250 INFO:journalctl@ceph.mds.f.smithi193.stdout:Mar 24 00:21:21 smithi193 podman[71853]: 2024-03-24 00:21:21.219688651 +0000 UTC m=+0.389026294 container init 72641b2d5a1ac705eefede03dcb21c26a2e124304b9f7ca43ab321a67492be1f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-f, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.build-date=20240319, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c) 2024-03-24T00:21:21.250 INFO:journalctl@ceph.mds.f.smithi193.stdout:Mar 24 00:21:21 smithi193 podman[71853]: 2024-03-24 00:21:21.222267601 +0000 UTC m=+0.391605233 container start 72641b2d5a1ac705eefede03dcb21c26a2e124304b9f7ca43ab321a67492be1f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-f, maintainer=Guillaume Abrioux , GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_CLEAN=True, org.label-schema.schema-version=1.0, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, org.label-schema.build-date=20240319, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS) 2024-03-24T00:21:21.321 INFO:journalctl@ceph.mds.g.smithi118.stdout:Mar 24 00:21:20 smithi118 systemd[1]: ceph-d79a5420-e971-11ee-95cd-87774f69a715@mds.g.service: Scheduled restart job, restart counter is at 2. 2024-03-24T00:21:21.322 INFO:journalctl@ceph.mds.g.smithi118.stdout:Mar 24 00:21:20 smithi118 systemd[1]: Stopped Ceph mds.g for d79a5420-e971-11ee-95cd-87774f69a715. 2024-03-24T00:21:21.322 INFO:journalctl@ceph.mds.g.smithi118.stdout:Mar 24 00:21:20 smithi118 systemd[1]: ceph-d79a5420-e971-11ee-95cd-87774f69a715@mds.g.service: Consumed 1.460s CPU time. 2024-03-24T00:21:21.322 INFO:journalctl@ceph.mds.g.smithi118.stdout:Mar 24 00:21:20 smithi118 systemd[1]: Starting Ceph mds.g for d79a5420-e971-11ee-95cd-87774f69a715... 2024-03-24T00:21:21.489 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:21:21.488+0000 7ff973a00640 1 -- 172.21.15.118:0/1523486712 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff96c108ef0 msgr2=0x7ff96c109370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:21:21.489 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:21:21.489+0000 7ff973a00640 1 --2- 172.21.15.118:0/1523486712 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff96c108ef0 0x7ff96c109370 secure :-1 s=READY pgs=418 cs=0 l=1 rev1=1 crypto rx=0x7ff958009900 tx=0x7ff95802bc30 comp rx=0 tx=0).stop 2024-03-24T00:21:21.489 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:21:21.489+0000 7ff973a00640 1 -- 172.21.15.118:0/1523486712 shutdown_connections 2024-03-24T00:21:21.489 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:21:21.489+0000 7ff973a00640 1 --2- 172.21.15.118:0/1523486712 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7ff96c1098b0 0x7ff96c110140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:21:21.489 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:21:21.489+0000 7ff973a00640 1 --2- 172.21.15.118:0/1523486712 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff96c108ef0 0x7ff96c109370 unknown :-1 s=CLOSED pgs=418 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:21:21.489 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:21:21.489+0000 7ff973a00640 1 --2- 172.21.15.118:0/1523486712 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7ff96c107d00 0x7ff96c108120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:21:21.489 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:21:21.489+0000 7ff973a00640 1 -- 172.21.15.118:0/1523486712 >> 172.21.15.118:0/1523486712 conn(0x7ff96c076cb0 msgr2=0x7ff96c0790d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:21:21.489 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:21:21.489+0000 7ff973a00640 1 -- 172.21.15.118:0/1523486712 shutdown_connections 2024-03-24T00:21:21.489 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:21:21.489+0000 7ff973a00640 1 -- 172.21.15.118:0/1523486712 wait complete. 2024-03-24T00:21:21.492 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:21:21.490+0000 7ff973a00640 1 Processor -- start 2024-03-24T00:21:21.492 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:21:21.490+0000 7ff973a00640 1 -- start start 2024-03-24T00:21:21.493 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:21:21.490+0000 7ff973a00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff96c107d00 0x7ff96c19df70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:21:21.493 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:21:21.490+0000 7ff973a00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7ff96c108ef0 0x7ff96c19e4b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:21:21.493 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:21:21.490+0000 7ff973a00640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7ff96c1098b0 0x7ff96c1a5520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:21:21.493 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:21:21.490+0000 7ff973a00640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff96c1139b0 con 0x7ff96c1098b0 2024-03-24T00:21:21.493 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:21:21.490+0000 7ff973a00640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff96c113830 con 0x7ff96c107d00 2024-03-24T00:21:21.493 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:21:21.490+0000 7ff973a00640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff96c113b30 con 0x7ff96c108ef0 2024-03-24T00:21:21.494 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:21:21.491+0000 7ff971000640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff96c107d00 0x7ff96c19df70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:21:21.494 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:21:21.491+0000 7ff971000640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff96c107d00 0x7ff96c19df70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.118:41400/0 (socket says 172.21.15.118:41400) 2024-03-24T00:21:21.494 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:21:21.491+0000 7ff971000640 1 -- 172.21.15.118:0/696984240 learned_addr learned my addr 172.21.15.118:0/696984240 (peer_addr_for_me v2:172.21.15.118:0/0) 2024-03-24T00:21:21.494 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:21:21.491+0000 7ff96be00640 1 --2- 172.21.15.118:0/696984240 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7ff96c108ef0 0x7ff96c19e4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:21:21.494 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:21:21.491+0000 7ff971a00640 1 --2- 172.21.15.118:0/696984240 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7ff96c1098b0 0x7ff96c1a5520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:21:21.495 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:21:21.491+0000 7ff971000640 1 -- 172.21.15.118:0/696984240 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7ff96c108ef0 msgr2=0x7ff96c19e4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:21:21.495 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:21:21.491+0000 7ff971000640 1 --2- 172.21.15.118:0/696984240 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7ff96c108ef0 0x7ff96c19e4b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:21:21.495 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:21:21.491+0000 7ff971000640 1 -- 172.21.15.118:0/696984240 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7ff96c1098b0 msgr2=0x7ff96c1a5520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:21:21.495 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:21:21.491+0000 7ff971000640 1 --2- 172.21.15.118:0/696984240 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7ff96c1098b0 0x7ff96c1a5520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:21:21.495 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:21:21.491+0000 7ff971000640 1 -- 172.21.15.118:0/696984240 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff96c1a5b90 con 0x7ff96c107d00 2024-03-24T00:21:21.495 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:21:21.491+0000 7ff971000640 1 --2- 172.21.15.118:0/696984240 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff96c107d00 0x7ff96c19df70 secure :-1 s=READY pgs=419 cs=0 l=1 rev1=1 crypto rx=0x7ff96000e9c0 tx=0x7ff96000ee80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:21:21.495 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:21:21.491+0000 7ff969600640 1 -- 172.21.15.118:0/696984240 <== mon.2 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff96000cd80 con 0x7ff96c107d00 2024-03-24T00:21:21.495 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:21:21.492+0000 7ff969600640 1 -- 172.21.15.118:0/696984240 <== mon.2 v2:172.21.15.189:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7ff9600035f0 con 0x7ff96c107d00 2024-03-24T00:21:21.495 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:21:21.492+0000 7ff969600640 1 -- 172.21.15.118:0/696984240 <== mon.2 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff960010690 con 0x7ff96c107d00 2024-03-24T00:21:21.495 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:21:21.492+0000 7ff973a00640 1 -- 172.21.15.118:0/696984240 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff96c1a5e80 con 0x7ff96c107d00 2024-03-24T00:21:21.495 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:21:21.492+0000 7ff973a00640 1 -- 172.21.15.118:0/696984240 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff96c1a62f0 con 0x7ff96c107d00 2024-03-24T00:21:21.498 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:21:21.495+0000 7ff969600640 1 -- 172.21.15.118:0/696984240 <== mon.2 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 20) v1 ==== 137291+0+0 (secure 0 0 0) 0x7ff9600037a0 con 0x7ff96c107d00 2024-03-24T00:21:21.498 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:21:21.495+0000 7ff976783740 1 Processor -- start 2024-03-24T00:21:21.498 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:21:21.495+0000 7ff976783740 1 -- start start 2024-03-24T00:21:21.498 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:21:21.496+0000 7ff976783740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55d096e3b1e0 0x55d096e3b600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:21:21.498 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:21:21.496+0000 7ff976783740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55d096e3bbd0 0x55d096e406d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:21:21.498 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:21:21.496+0000 7ff976783740 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55d096e40d40 0x55d096e431d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:21:21.499 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:21:21.496+0000 7ff976783740 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d096dfe3b0 con 0x55d096e40d40 2024-03-24T00:21:21.499 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:21:21.496+0000 7ff976783740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d096da0830 con 0x55d096e3bbd0 2024-03-24T00:21:21.499 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:21:21.496+0000 7ff976783740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d096d495d0 con 0x55d096e3b1e0 2024-03-24T00:21:21.499 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:21:21.496+0000 7ff971000640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55d096e3bbd0 0x55d096e406d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:21:21.499 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:21:21.496+0000 7ff971a00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55d096e3b1e0 0x55d096e3b600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:21:21.499 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:21:21.496+0000 7ff971a00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55d096e3b1e0 0x55d096e3b600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.118:37268/0 (socket says 172.21.15.118:37268) 2024-03-24T00:21:21.499 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:21:21.496+0000 7ff971a00640 1 -- 172.21.15.118:0/3556625711 learned_addr learned my addr 172.21.15.118:0/3556625711 (peer_addr_for_me v2:172.21.15.118:0/0) 2024-03-24T00:21:21.499 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:21:21.496+0000 7ff96be00640 1 --2- 172.21.15.118:0/3556625711 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55d096e40d40 0x55d096e431d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:21:21.499 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:21:21.496+0000 7ff971a00640 1 -- 172.21.15.118:0/3556625711 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55d096e3bbd0 msgr2=0x55d096e406d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:21:21.499 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:21:21.497+0000 7ff971a00640 1 --2- 172.21.15.118:0/3556625711 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55d096e3bbd0 0x55d096e406d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:21:21.499 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:21:21.497+0000 7ff971a00640 1 -- 172.21.15.118:0/3556625711 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55d096e40d40 msgr2=0x55d096e431d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:21:21.499 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:21:21.497+0000 7ff971a00640 1 --2- 172.21.15.118:0/3556625711 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55d096e40d40 0x55d096e431d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:21:21.500 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:21:21.497+0000 7ff971a00640 1 -- 172.21.15.118:0/3556625711 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55d096d612c0 con 0x55d096e3b1e0 2024-03-24T00:21:21.500 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:21:21.497+0000 7ff96be00640 1 --2- 172.21.15.118:0/3556625711 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55d096e40d40 0x55d096e431d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T00:21:21.500 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:21:21.497+0000 7ff969600640 1 --2- 172.21.15.118:0/696984240 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7ff9540a2920 0x7ff9540a4de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:21:21.500 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:21:21.497+0000 7ff971a00640 1 --2- 172.21.15.118:0/3556625711 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55d096e3b1e0 0x55d096e3b600 secure :-1 s=READY pgs=426 cs=0 l=1 rev1=1 crypto rx=0x7ff964009fa0 tx=0x7ff96400f7f0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=e307ba6914a3585d server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:21:21.500 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:21:21.497+0000 7ff969600640 1 -- 172.21.15.118:0/696984240 <== mon.2 v2:172.21.15.189:3300/0 5 ==== osd_map(99..99 src has 1..99) v4 ==== 9051+0+0 (secure 0 0 0) 0x7ff9600cf1b0 con 0x7ff96c107d00 2024-03-24T00:21:21.500 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:21:21.497+0000 7ff95e000640 1 -- 172.21.15.118:0/3556625711 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff964015070 con 0x55d096e3b1e0 2024-03-24T00:21:21.500 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:21:21.497+0000 7ff971000640 1 --2- 172.21.15.118:0/696984240 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7ff9540a2920 0x7ff9540a4de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:21:21.500 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:21:21.497+0000 7ff95e000640 1 -- 172.21.15.118:0/3556625711 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7ff96401ab40 con 0x55d096e3b1e0 2024-03-24T00:21:21.500 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:21:21.497+0000 7ff95e000640 1 -- 172.21.15.118:0/3556625711 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff964011e30 con 0x55d096e3b1e0 2024-03-24T00:21:21.500 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:21:21.498+0000 7ff976783740 1 -- 172.21.15.118:0/3556625711 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55d096e3b1e0 msgr2=0x55d096e3b600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:21:21.500 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:21:21.498+0000 7ff976783740 1 --2- 172.21.15.118:0/3556625711 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55d096e3b1e0 0x55d096e3b600 secure :-1 s=READY pgs=426 cs=0 l=1 rev1=1 crypto rx=0x7ff964009fa0 tx=0x7ff96400f7f0 comp rx=0 tx=0).stop 2024-03-24T00:21:21.500 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:21:21.498+0000 7ff971000640 1 --2- 172.21.15.118:0/696984240 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7ff9540a2920 0x7ff9540a4de0 secure :-1 s=READY pgs=363 cs=0 l=1 rev1=1 crypto rx=0x7ff960097c20 tx=0x7ff9600d2bf0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:21:21.500 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:21:21.498+0000 7ff976783740 1 -- 172.21.15.118:0/3556625711 shutdown_connections 2024-03-24T00:21:21.501 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:21:21.498+0000 7ff976783740 1 --2- 172.21.15.118:0/3556625711 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55d096e40d40 0x55d096e431d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:21:21.501 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:21:21.498+0000 7ff976783740 1 --2- 172.21.15.118:0/3556625711 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55d096e3bbd0 0x55d096e406d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:21:21.501 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:21:21.498+0000 7ff976783740 1 --2- 172.21.15.118:0/3556625711 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55d096e3b1e0 0x55d096e3b600 unknown :-1 s=CLOSED pgs=426 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:21:21.501 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:21:21.498+0000 7ff976783740 1 -- 172.21.15.118:0/3556625711 >> 172.21.15.118:0/3556625711 conn(0x55d096e36750 msgr2=0x55d096e36b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:21:21.501 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:21:21.498+0000 7ff976783740 1 -- 172.21.15.118:0/3556625711 shutdown_connections 2024-03-24T00:21:21.501 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:21:21.498+0000 7ff976783740 1 -- 172.21.15.118:0/3556625711 wait complete. 2024-03-24T00:21:21.506 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:21:21.499+0000 7ff976783740 1 -- start start 2024-03-24T00:21:21.506 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:21:21.499+0000 7ff976783740 1 Processor -- start 2024-03-24T00:21:21.506 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:21:21.500+0000 7ff976783740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55d096e3b1e0 0x55d096e45480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:21:21.506 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:21:21.500+0000 7ff976783740 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55d096e3bbd0 0x55d096e459c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:21:21.506 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:21:21.500+0000 7ff976783740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55d096e40d40 0x55d096e4c990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:21:21.506 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:21:21.500+0000 7ff976783740 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d096d5ca20 con 0x55d096e3bbd0 2024-03-24T00:21:21.506 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:21:21.500+0000 7ff976783740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d096d9c8d0 con 0x55d096e40d40 2024-03-24T00:21:21.506 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:21:21.500+0000 7ff976783740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x55d096e43cf0 con 0x55d096e3b1e0 2024-03-24T00:21:21.506 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:21:21.500+0000 7ff976783740 10 client.0 _ll_register_callbacks cb 0x55d096d991c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T00:21:21.507 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:21:21.500+0000 7ff96be00640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55d096e3bbd0 0x55d096e459c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:21:21.507 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:21:21.500+0000 7ff971a00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55d096e40d40 0x55d096e4c990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:21:21.507 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:21:21.500+0000 7ff971a00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55d096e40d40 0x55d096e4c990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.118:41424/0 (socket says 172.21.15.118:41424) 2024-03-24T00:21:21.507 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:21:21.500+0000 7ff971a00640 1 -- 172.21.15.118:0/2716293753 learned_addr learned my addr 172.21.15.118:0/2716293753 (peer_addr_for_me v2:172.21.15.118:0/0) 2024-03-24T00:21:21.507 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:21:21.500+0000 7ff971a00640 1 --2- 172.21.15.118:0/2716293753 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55d096e3b1e0 0x55d096e45480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:21:21.507 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:21:21.501+0000 7ff971a00640 1 -- 172.21.15.118:0/2716293753 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55d096e3b1e0 msgr2=0x55d096e45480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:21:21.507 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:21:21.501+0000 7ff971a00640 1 --2- 172.21.15.118:0/2716293753 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55d096e3b1e0 0x55d096e45480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:21:21.507 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:21:21.501+0000 7ff971a00640 1 -- 172.21.15.118:0/2716293753 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55d096e3bbd0 msgr2=0x55d096e459c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:21:21.507 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:21:21.501+0000 7ff971a00640 1 --2- 172.21.15.118:0/2716293753 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55d096e3bbd0 0x55d096e459c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:21:21.507 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:21:21.501+0000 7ff971a00640 1 -- 172.21.15.118:0/2716293753 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55d096ab5210 con 0x55d096e40d40 2024-03-24T00:21:21.507 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:21:21.501+0000 7ff971a00640 1 --2- 172.21.15.118:0/2716293753 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55d096e40d40 0x55d096e4c990 secure :-1 s=READY pgs=420 cs=0 l=1 rev1=1 crypto rx=0x7ff96401bdf0 tx=0x7ff96400f7f0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=65c1a2df8056379 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:21:21.507 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:21:21.501+0000 7ff952a00640 10 client.0 ms_handle_connect on v2:172.21.15.189:3300/0 2024-03-24T00:21:21.507 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:21:21.501+0000 7ff952a00640 1 -- 172.21.15.118:0/2716293753 <== mon.2 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff964015070 con 0x55d096e40d40 2024-03-24T00:21:21.507 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:21:21.501+0000 7ff952a00640 1 -- 172.21.15.118:0/2716293753 <== mon.2 v2:172.21.15.189:3300/0 2 ==== osd_map(99..99 src has 1..99) v4 ==== 9051+0+0 (secure 0 0 0) 0x7ff964011db0 con 0x55d096e40d40 2024-03-24T00:21:21.508 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:21:21.501+0000 7ff976783740 1 -- 172.21.15.118:0/2716293753 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7ff96401ab40 con 0x55d096e40d40 2024-03-24T00:21:21.508 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:21:21.502+0000 7ff952a00640 1 -- 172.21.15.118:0/2716293753 <== mon.2 v2:172.21.15.189:3300/0 3 ==== mon_get_version_reply(handle=1 version=75) v2 ==== 24+0+0 (secure 0 0 0) 0x7ff964032050 con 0x55d096e40d40 2024-03-24T00:21:21.508 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:21:21.502+0000 7ff976783740 10 client.26741 fetch_fsmap learned FSMap version 75 2024-03-24T00:21:21.508 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:21:21.502+0000 7ff976783740 1 -- 172.21.15.118:0/2716293753 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,fsmap=75,monmap=4+}) v3 -- 0x55d096b81620 con 0x55d096e40d40 2024-03-24T00:21:21.508 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:21:21.502+0000 7ff952a00640 1 -- 172.21.15.118:0/2716293753 <== mon.2 v2:172.21.15.189:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7ff96401ab40 con 0x55d096e40d40 2024-03-24T00:21:21.508 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:21:21.502+0000 7ff952a00640 1 -- 172.21.15.118:0/2716293753 <== mon.2 v2:172.21.15.189:3300/0 5 ==== fsmap(e 75) v1 ==== 7946+0+0 (secure 0 0 0) 0x7ff964005290 con 0x55d096e40d40 2024-03-24T00:21:21.508 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:21:21.502+0000 7ff976783740 10 client.26741 fetch_fsmap finished waiting for FSMap version 75 2024-03-24T00:21:21.508 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:21:21.502+0000 7ff976783740 10 client.26741 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-24T00:21:21.508 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:21:21.502+0000 7ff976783740 20 client.26741 populate_metadata read hostname 'smithi118' 2024-03-24T00:21:21.508 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:21:21.502+0000 7ff976783740 1 --2- 172.21.15.118:0/2716293753 >> [v2:172.21.15.189:6838/4074484168,v1:172.21.15.189:6839/4074484168] conn(0x55d096e50f40 0x55d096e53360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:21:21.508 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:21:21.502+0000 7ff976783740 4 client.26741 mds_command: new command op to 24569 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-24T00:21:21.508 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:21:21.502+0000 7ff976783740 1 -- 172.21.15.118:0/2716293753 --> [v2:172.21.15.189:6838/4074484168,v1:172.21.15.189:6839/4074484168] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55d096dadd70 con 0x55d096e50f40 2024-03-24T00:21:21.508 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:21:21.503+0000 7ff96be00640 1 --2- 172.21.15.118:0/2716293753 >> [v2:172.21.15.189:6838/4074484168,v1:172.21.15.189:6839/4074484168] conn(0x55d096e50f40 0x55d096e53360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:21:21.509 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:21:21.503+0000 7ff96be00640 1 --2- 172.21.15.118:0/2716293753 >> [v2:172.21.15.189:6838/4074484168,v1:172.21.15.189:6839/4074484168] conn(0x55d096e50f40 0x55d096e53360 crc :-1 s=READY pgs=200 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=c6b205750843cf1f server_cookie=31596a048519a8b8 in_seq=0 out_seq=0 2024-03-24T00:21:21.509 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:21:21.503+0000 7ff952a00640 10 client.26741 ms_handle_connect on v2:172.21.15.189:6838/4074484168 2024-03-24T00:21:21.509 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:21:21.504+0000 7ff952a00640 1 -- 172.21.15.118:0/2716293753 <== mds.0 v2:172.21.15.189:6838/4074484168 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x55d096dadd70 con 0x55d096e50f40 2024-03-24T00:21:21.509 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:21:21.504+0000 7ff952a00640 10 client.26741 handle_command_reply: tid=0 2024-03-24T00:21:21.509 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:21:21.504+0000 7ff976783740 1 client.26741 shutdown 2024-03-24T00:21:21.509 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:21:21.504+0000 7ff976783740 1 -- 172.21.15.118:0/2716293753 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55d096e40d40 msgr2=0x55d096e4c990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:21:21.509 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:21:21.504+0000 7ff976783740 1 --2- 172.21.15.118:0/2716293753 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55d096e40d40 0x55d096e4c990 secure :-1 s=READY pgs=420 cs=0 l=1 rev1=1 crypto rx=0x7ff96401bdf0 tx=0x7ff96400f7f0 comp rx=0 tx=0).stop 2024-03-24T00:21:21.509 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:21:21.504+0000 7ff976783740 1 -- 172.21.15.118:0/2716293753 shutdown_connections 2024-03-24T00:21:21.509 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:21:21.504+0000 7ff976783740 1 --2- 172.21.15.118:0/2716293753 >> [v2:172.21.15.189:6838/4074484168,v1:172.21.15.189:6839/4074484168] conn(0x55d096e50f40 0x55d096e53360 crc :-1 s=READY pgs=200 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:21:21.509 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:21:21.504+0000 7ff976783740 1 --2- 172.21.15.118:0/2716293753 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55d096e40d40 0x55d096e4c990 unknown :-1 s=CLOSED pgs=420 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:21:21.509 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:21:21.504+0000 7ff976783740 1 --2- 172.21.15.118:0/2716293753 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55d096e3bbd0 0x55d096e459c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:21:21.509 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:21:21.504+0000 7ff976783740 1 --2- 172.21.15.118:0/2716293753 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55d096e3b1e0 0x55d096e45480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:21:21.510 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:21:21.504+0000 7ff976783740 1 -- 172.21.15.118:0/2716293753 >> 172.21.15.118:0/2716293753 conn(0x55d096e36750 msgr2=0x55d096e35950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:21:21.510 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:21:21.504+0000 7ff952a00640 0 client.26741 ms_handle_reset on v2:172.21.15.189:6838/4074484168 2024-03-24T00:21:21.510 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:21:21.504+0000 7ff976783740 1 -- 172.21.15.118:0/2716293753 shutdown_connections 2024-03-24T00:21:21.510 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:21:21.504+0000 7ff976783740 1 -- 172.21.15.118:0/2716293753 wait complete. 2024-03-24T00:21:21.510 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:21:21.504+0000 7ff976783740 20 client.26741 trim_cache size 0 max 16384 2024-03-24T00:21:21.521 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:21:21.522+0000 7ff976783740 1 Processor -- start 2024-03-24T00:21:21.521 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:21:21.522+0000 7ff976783740 1 -- start start 2024-03-24T00:21:21.523 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:21:21.522+0000 7ff976783740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55d096e3b1e0 0x55d096e3cf50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:21:21.523 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:21:21.522+0000 7ff976783740 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55d096e3bbd0 0x55d096e3d490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:21:21.523 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:21:21.522+0000 7ff976783740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55d096e40d40 0x55d096e4c950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:21:21.523 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:21:21.522+0000 7ff976783740 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff96c1afd40 con 0x55d096e3bbd0 2024-03-24T00:21:21.523 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:21:21.522+0000 7ff976783740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff96c1b10c0 con 0x55d096e40d40 2024-03-24T00:21:21.523 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:21:21.522+0000 7ff976783740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff96c1b0400 con 0x55d096e3b1e0 2024-03-24T00:21:21.523 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:21:21.522+0000 7ff971a00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55d096e3b1e0 0x55d096e3cf50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:21:21.523 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:21:21.522+0000 7ff96be00640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55d096e3bbd0 0x55d096e3d490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:21:21.523 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:21:21.523+0000 7ff971a00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55d096e3b1e0 0x55d096e3cf50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.118:37276/0 (socket says 172.21.15.118:37276) 2024-03-24T00:21:21.523 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:21:21.523+0000 7ff971a00640 1 -- 172.21.15.118:0/4033595069 learned_addr learned my addr 172.21.15.118:0/4033595069 (peer_addr_for_me v2:172.21.15.118:0/0) 2024-03-24T00:21:21.524 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:21:21.523+0000 7ff971a00640 1 --2- 172.21.15.118:0/4033595069 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55d096e40d40 0x55d096e4c950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:21:21.524 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:21:21.523+0000 7ff96be00640 1 -- 172.21.15.118:0/4033595069 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55d096e3b1e0 msgr2=0x55d096e3cf50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:21:21.524 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:21:21.523+0000 7ff96be00640 1 --2- 172.21.15.118:0/4033595069 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55d096e3b1e0 0x55d096e3cf50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:21:21.524 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:21:21.523+0000 7ff96be00640 1 -- 172.21.15.118:0/4033595069 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55d096e40d40 msgr2=0x55d096e4c950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:21:21.524 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:21:21.523+0000 7ff96be00640 1 --2- 172.21.15.118:0/4033595069 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55d096e40d40 0x55d096e4c950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:21:21.524 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:21:21.523+0000 7ff96be00640 1 -- 172.21.15.118:0/4033595069 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55d096e43e70 con 0x55d096e3bbd0 2024-03-24T00:21:21.524 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:21:21.523+0000 7ff971a00640 1 --2- 172.21.15.118:0/4033595069 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55d096e3b1e0 0x55d096e3cf50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T00:21:21.524 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:21:21.523+0000 7ff96be00640 1 --2- 172.21.15.118:0/4033595069 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55d096e3bbd0 0x55d096e3d490 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7ff958036040 tx=0x7ff95802df90 comp rx=0 tx=0).ready entity=mon.0 client_cookie=381bc0776b8385e5 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:21:21.527 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:21:21.524+0000 7ff95d600640 1 -- 172.21.15.118:0/4033595069 <== mon.0 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff95802cb20 con 0x55d096e3bbd0 2024-03-24T00:21:21.527 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:21:21.524+0000 7ff95d600640 1 -- 172.21.15.118:0/4033595069 <== mon.0 v2:172.21.15.118:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7ff958050030 con 0x55d096e3bbd0 2024-03-24T00:21:21.528 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:21:21.524+0000 7ff95d600640 1 -- 172.21.15.118:0/4033595069 <== mon.0 v2:172.21.15.118:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff958004eb0 con 0x55d096e3bbd0 2024-03-24T00:21:21.528 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:21:21.524+0000 7ff976783740 1 -- 172.21.15.118:0/4033595069 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55d096e3bbd0 msgr2=0x55d096e3d490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:21:21.528 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:21:21.524+0000 7ff976783740 1 --2- 172.21.15.118:0/4033595069 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55d096e3bbd0 0x55d096e3d490 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7ff958036040 tx=0x7ff95802df90 comp rx=0 tx=0).stop 2024-03-24T00:21:21.528 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:21:21.525+0000 7ff976783740 1 -- 172.21.15.118:0/4033595069 shutdown_connections 2024-03-24T00:21:21.528 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:21:21.525+0000 7ff976783740 1 --2- 172.21.15.118:0/4033595069 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55d096e40d40 0x55d096e4c950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:21:21.528 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:21:21.525+0000 7ff976783740 1 --2- 172.21.15.118:0/4033595069 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55d096e3bbd0 0x55d096e3d490 unknown :-1 s=CLOSED pgs=212 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:21:21.528 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:21:21.525+0000 7ff976783740 1 --2- 172.21.15.118:0/4033595069 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55d096e3b1e0 0x55d096e3cf50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:21:21.528 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:21:21.525+0000 7ff976783740 1 -- 172.21.15.118:0/4033595069 >> 172.21.15.118:0/4033595069 conn(0x55d096e36750 msgr2=0x55d096e36b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:21:21.528 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:21:21.525+0000 7ff976783740 1 -- 172.21.15.118:0/4033595069 shutdown_connections 2024-03-24T00:21:21.528 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:21:21.525+0000 7ff976783740 1 -- 172.21.15.118:0/4033595069 wait complete. 2024-03-24T00:21:21.528 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:21:21.525+0000 7ff976783740 1 -- start start 2024-03-24T00:21:21.528 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:21:21.525+0000 7ff976783740 1 Processor -- start 2024-03-24T00:21:21.528 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:21:21.526+0000 7ff976783740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55d096e3b1e0 0x55d096e4a900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:21:21.529 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:21:21.526+0000 7ff976783740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55d096e3bbd0 0x55d096e4ae40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:21:21.529 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:21:21.526+0000 7ff976783740 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55d096e40d40 0x55d096e4be50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:21:21.529 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:21:21.526+0000 7ff976783740 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff96c1ae4c0 con 0x55d096e40d40 2024-03-24T00:21:21.529 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:21:21.526+0000 7ff976783740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff96c1b0a60 con 0x55d096e3bbd0 2024-03-24T00:21:21.529 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:21:21.526+0000 7ff976783740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff96c1ad880 con 0x55d096e3b1e0 2024-03-24T00:21:21.529 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:21:21.526+0000 7ff976783740 10 client.0 _ll_register_callbacks cb 0x7ff9600d2ac0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T00:21:21.529 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:21:21.526+0000 7ff96be00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55d096e3bbd0 0x55d096e4ae40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:21:21.529 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:21:21.526+0000 7ff971a00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55d096e3b1e0 0x55d096e4a900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:21:21.529 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:21:21.526+0000 7ff96be00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55d096e3bbd0 0x55d096e4ae40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.118:41444/0 (socket says 172.21.15.118:41444) 2024-03-24T00:21:21.529 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:21:21.526+0000 7ff96be00640 1 -- 172.21.15.118:0/1441390391 learned_addr learned my addr 172.21.15.118:0/1441390391 (peer_addr_for_me v2:172.21.15.118:0/0) 2024-03-24T00:21:21.529 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:21:21.526+0000 7ff971a00640 1 --2- 172.21.15.118:0/1441390391 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55d096e40d40 0x55d096e4be50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:21:21.529 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:21:21.526+0000 7ff96be00640 1 -- 172.21.15.118:0/1441390391 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55d096e3b1e0 msgr2=0x55d096e4a900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:21:21.529 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:21:21.526+0000 7ff96be00640 1 --2- 172.21.15.118:0/1441390391 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55d096e3b1e0 0x55d096e4a900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:21:21.529 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:21:21.526+0000 7ff96be00640 1 -- 172.21.15.118:0/1441390391 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55d096e40d40 msgr2=0x55d096e4be50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:21:21.529 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:21:21.526+0000 7ff96be00640 1 --2- 172.21.15.118:0/1441390391 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55d096e40d40 0x55d096e4be50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:21:21.530 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:21:21.526+0000 7ff96be00640 1 -- 172.21.15.118:0/1441390391 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55d096e4c440 con 0x55d096e3bbd0 2024-03-24T00:21:21.530 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:21:21.527+0000 7ff96be00640 1 --2- 172.21.15.118:0/1441390391 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55d096e3bbd0 0x55d096e4ae40 secure :-1 s=READY pgs=421 cs=0 l=1 rev1=1 crypto rx=0x7ff95802c7b0 tx=0x7ff958004c70 comp rx=0 tx=0).ready entity=mon.2 client_cookie=34b10e70d592b903 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:21:21.530 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:21:21.527+0000 7ff952a00640 10 client.0 ms_handle_connect on v2:172.21.15.189:3300/0 2024-03-24T00:21:21.530 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:21:21.527+0000 7ff952a00640 1 -- 172.21.15.118:0/1441390391 <== mon.2 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff95802cb20 con 0x55d096e3bbd0 2024-03-24T00:21:21.530 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:21:21.527+0000 7ff952a00640 1 -- 172.21.15.118:0/1441390391 <== mon.2 v2:172.21.15.189:3300/0 2 ==== osd_map(99..99 src has 1..99) v4 ==== 9051+0+0 (secure 0 0 0) 0x7ff95802ccc0 con 0x55d096e3bbd0 2024-03-24T00:21:21.530 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:21:21.527+0000 7ff976783740 1 -- 172.21.15.118:0/1441390391 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7ff958050030 con 0x55d096e3bbd0 2024-03-24T00:21:21.530 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:21:21.527+0000 7ff952a00640 1 -- 172.21.15.118:0/1441390391 <== mon.2 v2:172.21.15.189:3300/0 3 ==== mon_get_version_reply(handle=1 version=75) v2 ==== 24+0+0 (secure 0 0 0) 0x7ff958004030 con 0x55d096e3bbd0 2024-03-24T00:21:21.530 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:21:21.527+0000 7ff976783740 10 client.26747 fetch_fsmap learned FSMap version 75 2024-03-24T00:21:21.530 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:21:21.527+0000 7ff976783740 1 -- 172.21.15.118:0/1441390391 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,fsmap=75,monmap=4+}) v3 -- 0x55d096ead0a0 con 0x55d096e3bbd0 2024-03-24T00:21:21.530 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:21:21.527+0000 7ff952a00640 1 -- 172.21.15.118:0/1441390391 <== mon.2 v2:172.21.15.189:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7ff958050030 con 0x55d096e3bbd0 2024-03-24T00:21:21.530 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:21:21.528+0000 7ff952a00640 1 -- 172.21.15.118:0/1441390391 <== mon.2 v2:172.21.15.189:3300/0 5 ==== fsmap(e 75) v1 ==== 7946+0+0 (secure 0 0 0) 0x7ff958004eb0 con 0x55d096e3bbd0 2024-03-24T00:21:21.530 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:21:21.528+0000 7ff976783740 10 client.26747 fetch_fsmap finished waiting for FSMap version 75 2024-03-24T00:21:21.530 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:21:21.528+0000 7ff976783740 10 client.26747 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-24T00:21:21.531 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:21:21.528+0000 7ff976783740 20 client.26747 populate_metadata read hostname 'smithi118' 2024-03-24T00:21:21.531 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:21:21.528+0000 7ff976783740 1 --2- 172.21.15.118:0/1441390391 >> [v2:172.21.15.189:6838/4074484168,v1:172.21.15.189:6839/4074484168] conn(0x55d096eada30 0x55d096eafe50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:21:21.531 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:21:21.528+0000 7ff976783740 4 client.26747 mds_command: new command op to 24569 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-03-24T00:21:21.531 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:21:21.528+0000 7ff976783740 1 -- 172.21.15.118:0/1441390391 --> [v2:172.21.15.189:6838/4074484168,v1:172.21.15.189:6839/4074484168] -- command(tid 0: {"prefix": "flush journal"}) v1 -- 0x55d096da80c0 con 0x55d096eada30 2024-03-24T00:21:21.531 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:21:21.528+0000 7ff971a00640 1 --2- 172.21.15.118:0/1441390391 >> [v2:172.21.15.189:6838/4074484168,v1:172.21.15.189:6839/4074484168] conn(0x55d096eada30 0x55d096eafe50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:21:21.532 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:21:21.528+0000 7ff952a00640 10 client.26747 ms_handle_connect on v2:172.21.15.189:6838/4074484168 2024-03-24T00:21:21.532 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:21:21.528+0000 7ff971a00640 1 --2- 172.21.15.118:0/1441390391 >> [v2:172.21.15.189:6838/4074484168,v1:172.21.15.189:6839/4074484168] conn(0x55d096eada30 0x55d096eafe50 crc :-1 s=READY pgs=201 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=cc795afcb4a6e4b7 server_cookie=5589f095cffdabb3 in_seq=0 out_seq=0 2024-03-24T00:21:21.680 INFO:journalctl@ceph.mds.g.smithi118.stdout:Mar 24 00:21:21 smithi118 podman[94258]: 2024-03-24 00:21:21.256128693 +0000 UTC m=+0.037674627 image pull c664afe56bb01db4f9afa9532c20598e201e664f2e9ba2be50db0d52674a26d2 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc 2024-03-24T00:21:21.680 INFO:journalctl@ceph.mds.g.smithi118.stdout:Mar 24 00:21:21 smithi118 podman[94258]: 2024-03-24 00:21:21.385906262 +0000 UTC m=+0.167452184 container create e5c87fc6a3bf4c109d5e01bdb0b1f592e72fa7f852e0ec79dc7ea1f1054cf96b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-g, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, ceph=True, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240319, GIT_CLEAN=True, CEPH_POINT_RELEASE=) 2024-03-24T00:21:21.680 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:21:21 smithi118 ceph-mon[31535]: daemon mds.f restarted 2024-03-24T00:21:21.681 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:21:21 smithi118 ceph-mon[31535]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-03-24T00:21:21.681 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:21:21 smithi118 ceph-mon[31535]: osdmap e99: 12 total, 12 up, 12 in 2024-03-24T00:21:21.681 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:21:21 smithi118 ceph-mon[31535]: mds.? [v2:172.21.15.193:6834/2814329894,v1:172.21.15.193:6835/2814329894] up:boot 2024-03-24T00:21:21.681 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:21:21 smithi118 ceph-mon[31535]: Standby daemon mds.l assigned to filesystem cephfs as rank 1 2024-03-24T00:21:21.681 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:21:21 smithi118 ceph-mon[31535]: fsmap cephfs:4/5 {0=b=up:active,2=c=up:active,3=g=up:active,4=e=up:active} 5 up:standby-replay 3 up:standby, 1 failed 2024-03-24T00:21:21.681 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:21:21 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-03-24T00:21:21.681 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:21:21 smithi118 ceph-mon[31535]: fsmap cephfs:5/5 4 up:standby-replay 3 up:standby 1 up:replay 4 up:active 2024-03-24T00:21:21.704 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:21:21 smithi189 ceph-mon[36187]: daemon mds.f restarted 2024-03-24T00:21:21.704 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:21:21 smithi189 ceph-mon[36187]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-03-24T00:21:21.704 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:21:21 smithi189 ceph-mon[36187]: osdmap e99: 12 total, 12 up, 12 in 2024-03-24T00:21:21.704 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:21:21 smithi189 ceph-mon[36187]: mds.? [v2:172.21.15.193:6834/2814329894,v1:172.21.15.193:6835/2814329894] up:boot 2024-03-24T00:21:21.704 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:21:21 smithi189 ceph-mon[36187]: Standby daemon mds.l assigned to filesystem cephfs as rank 1 2024-03-24T00:21:21.704 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:21:21 smithi189 ceph-mon[36187]: fsmap cephfs:4/5 {0=b=up:active,2=c=up:active,3=g=up:active,4=e=up:active} 5 up:standby-replay 3 up:standby, 1 failed 2024-03-24T00:21:21.704 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:21:21 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-03-24T00:21:21.704 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:21:21 smithi189 ceph-mon[36187]: fsmap cephfs:5/5 4 up:standby-replay 3 up:standby 1 up:replay 4 up:active 2024-03-24T00:21:21.710 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:21:21 smithi193 ceph-mon[36878]: daemon mds.f restarted 2024-03-24T00:21:21.711 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:21:21 smithi193 ceph-mon[36878]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-03-24T00:21:21.711 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:21:21 smithi193 ceph-mon[36878]: osdmap e99: 12 total, 12 up, 12 in 2024-03-24T00:21:21.711 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:21:21 smithi193 ceph-mon[36878]: mds.? [v2:172.21.15.193:6834/2814329894,v1:172.21.15.193:6835/2814329894] up:boot 2024-03-24T00:21:21.711 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:21:21 smithi193 ceph-mon[36878]: Standby daemon mds.l assigned to filesystem cephfs as rank 1 2024-03-24T00:21:21.711 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:21:21 smithi193 ceph-mon[36878]: fsmap cephfs:4/5 {0=b=up:active,2=c=up:active,3=g=up:active,4=e=up:active} 5 up:standby-replay 3 up:standby, 1 failed 2024-03-24T00:21:21.711 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:21:21 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-03-24T00:21:21.711 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:21:21 smithi193 ceph-mon[36878]: fsmap cephfs:5/5 4 up:standby-replay 3 up:standby 1 up:replay 4 up:active 2024-03-24T00:21:21.711 INFO:journalctl@ceph.mds.f.smithi193.stdout:Mar 24 00:21:21 smithi193 ceph-mds[71873]: -- 172.21.15.193:0/1927663480 <== mon.1 v2:172.21.15.193:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x55bcdb25ba40 con 0x55bcdb22ec00 2024-03-24T00:21:21.711 INFO:journalctl@ceph.mds.f.smithi193.stdout:Mar 24 00:21:21 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-f[71869]: starting mds.f at 2024-03-24T00:21:21.711 INFO:journalctl@ceph.mds.f.smithi193.stdout:Mar 24 00:21:21 smithi193 bash[71853]: 72641b2d5a1ac705eefede03dcb21c26a2e124304b9f7ca43ab321a67492be1f 2024-03-24T00:21:21.711 INFO:journalctl@ceph.mds.f.smithi193.stdout:Mar 24 00:21:21 smithi193 systemd[1]: Started Ceph mds.f for d79a5420-e971-11ee-95cd-87774f69a715. 2024-03-24T00:21:21.946 INFO:journalctl@ceph.mds.g.smithi118.stdout:Mar 24 00:21:21 smithi118 podman[94258]: 2024-03-24 00:21:21.680839183 +0000 UTC m=+0.462385082 container init e5c87fc6a3bf4c109d5e01bdb0b1f592e72fa7f852e0ec79dc7ea1f1054cf96b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-g, GIT_CLEAN=True, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, ceph=True, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.build-date=20240319, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30) 2024-03-24T00:21:21.947 INFO:journalctl@ceph.mds.g.smithi118.stdout:Mar 24 00:21:21 smithi118 podman[94258]: 2024-03-24 00:21:21.687777925 +0000 UTC m=+0.469323810 container start e5c87fc6a3bf4c109d5e01bdb0b1f592e72fa7f852e0ec79dc7ea1f1054cf96b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-g, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, GIT_CLEAN=True, org.label-schema.license=GPLv2, ceph=True, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, org.label-schema.build-date=20240319) 2024-03-24T00:21:21.947 INFO:journalctl@ceph.mds.g.smithi118.stdout:Mar 24 00:21:21 smithi118 ceph-mds[94322]: -- 172.21.15.118:0/1933484670 <== mon.0 v2:172.21.15.118:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x5567f9b49a40 con 0x5567f9b1cc00 2024-03-24T00:21:21.947 INFO:journalctl@ceph.mds.g.smithi118.stdout:Mar 24 00:21:21 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-g[94305]: starting mds.g at 2024-03-24T00:21:21.947 INFO:journalctl@ceph.mds.g.smithi118.stdout:Mar 24 00:21:21 smithi118 bash[94258]: e5c87fc6a3bf4c109d5e01bdb0b1f592e72fa7f852e0ec79dc7ea1f1054cf96b 2024-03-24T00:21:21.947 INFO:journalctl@ceph.mds.g.smithi118.stdout:Mar 24 00:21:21 smithi118 systemd[1]: Started Ceph mds.g for d79a5420-e971-11ee-95cd-87774f69a715. 2024-03-24T00:21:22.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:21:22 smithi118 ceph-mon[31535]: daemon mds.g restarted 2024-03-24T00:21:22.697 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:21:22 smithi118 ceph-mon[31535]: osdmap e100: 12 total, 12 up, 12 in 2024-03-24T00:21:22.697 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:21:22 smithi118 ceph-mon[31535]: mds.? [v2:172.21.15.193:6832/4276594827,v1:172.21.15.193:6833/4276594827] up:resolve 2024-03-24T00:21:22.697 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:21:22 smithi118 ceph-mon[31535]: mds.? [v2:172.21.15.118:6838/979323377,v1:172.21.15.118:6839/979323377] up:boot 2024-03-24T00:21:22.697 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:21:22 smithi118 ceph-mon[31535]: Standby daemon mds.h assigned to filesystem cephfs as rank 3 2024-03-24T00:21:22.697 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:21:22 smithi118 ceph-mon[31535]: fsmap cephfs:4/5 {0=b=up:active,1=l=up:resolve,2=c=up:active,4=e=up:active} 4 up:standby-replay 4 up:standby, 1 failed 2024-03-24T00:21:22.697 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:21:22 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2024-03-24T00:21:22.697 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:21:22 smithi118 ceph-mon[31535]: fsmap cephfs:5/5 3 up:standby-replay 4 up:standby 1 up:replay 1 up:resolve 3 up:active 2024-03-24T00:21:22.697 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:21:22 smithi118 ceph-mon[31535]: pgmap v578: 97 pgs: 97 active+clean; 133 MiB data, 730 MiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 735 KiB/s wr, 140 op/s 2024-03-24T00:21:22.704 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:21:22 smithi189 ceph-mon[36187]: daemon mds.g restarted 2024-03-24T00:21:22.704 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:21:22 smithi189 ceph-mon[36187]: osdmap e100: 12 total, 12 up, 12 in 2024-03-24T00:21:22.704 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:21:22 smithi189 ceph-mon[36187]: mds.? [v2:172.21.15.193:6832/4276594827,v1:172.21.15.193:6833/4276594827] up:resolve 2024-03-24T00:21:22.704 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:21:22 smithi189 ceph-mon[36187]: mds.? [v2:172.21.15.118:6838/979323377,v1:172.21.15.118:6839/979323377] up:boot 2024-03-24T00:21:22.704 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:21:22 smithi189 ceph-mon[36187]: Standby daemon mds.h assigned to filesystem cephfs as rank 3 2024-03-24T00:21:22.704 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:21:22 smithi189 ceph-mon[36187]: fsmap cephfs:4/5 {0=b=up:active,1=l=up:resolve,2=c=up:active,4=e=up:active} 4 up:standby-replay 4 up:standby, 1 failed 2024-03-24T00:21:22.704 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:21:22 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2024-03-24T00:21:22.704 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:21:22 smithi189 ceph-mon[36187]: fsmap cephfs:5/5 3 up:standby-replay 4 up:standby 1 up:replay 1 up:resolve 3 up:active 2024-03-24T00:21:22.704 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:21:22 smithi189 ceph-mon[36187]: pgmap v578: 97 pgs: 97 active+clean; 133 MiB data, 730 MiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 735 KiB/s wr, 140 op/s 2024-03-24T00:21:22.711 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:21:22 smithi193 ceph-mon[36878]: daemon mds.g restarted 2024-03-24T00:21:22.711 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:21:22 smithi193 ceph-mon[36878]: osdmap e100: 12 total, 12 up, 12 in 2024-03-24T00:21:22.711 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:21:22 smithi193 ceph-mon[36878]: mds.? [v2:172.21.15.193:6832/4276594827,v1:172.21.15.193:6833/4276594827] up:resolve 2024-03-24T00:21:22.711 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:21:22 smithi193 ceph-mon[36878]: mds.? [v2:172.21.15.118:6838/979323377,v1:172.21.15.118:6839/979323377] up:boot 2024-03-24T00:21:22.711 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:21:22 smithi193 ceph-mon[36878]: Standby daemon mds.h assigned to filesystem cephfs as rank 3 2024-03-24T00:21:22.711 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:21:22 smithi193 ceph-mon[36878]: fsmap cephfs:4/5 {0=b=up:active,1=l=up:resolve,2=c=up:active,4=e=up:active} 4 up:standby-replay 4 up:standby, 1 failed 2024-03-24T00:21:22.711 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:21:22 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2024-03-24T00:21:22.711 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:21:22 smithi193 ceph-mon[36878]: fsmap cephfs:5/5 3 up:standby-replay 4 up:standby 1 up:replay 1 up:resolve 3 up:active 2024-03-24T00:21:22.711 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:21:22 smithi193 ceph-mon[36878]: pgmap v578: 97 pgs: 97 active+clean; 133 MiB data, 730 MiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 735 KiB/s wr, 140 op/s 2024-03-24T00:21:24.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:21:24 smithi118 ceph-mon[31535]: mds.? [v2:172.21.15.189:6836/596160018,v1:172.21.15.189:6837/596160018] up:resolve 2024-03-24T00:21:24.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:21:24 smithi118 ceph-mon[31535]: fsmap cephfs:5/5 3 up:standby-replay 4 up:standby 2 up:resolve 3 up:active 2024-03-24T00:21:24.697 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:21:24 smithi118 ceph-mon[31535]: pgmap v579: 97 pgs: 97 active+clean; 138 MiB data, 739 MiB used, 1.0 TiB / 1.0 TiB avail; 149 KiB/s rd, 1.0 MiB/s wr, 158 op/s 2024-03-24T00:21:24.703 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:21:24 smithi189 ceph-mon[36187]: mds.? [v2:172.21.15.189:6836/596160018,v1:172.21.15.189:6837/596160018] up:resolve 2024-03-24T00:21:24.704 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:21:24 smithi189 ceph-mon[36187]: fsmap cephfs:5/5 3 up:standby-replay 4 up:standby 2 up:resolve 3 up:active 2024-03-24T00:21:24.704 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:21:24 smithi189 ceph-mon[36187]: pgmap v579: 97 pgs: 97 active+clean; 138 MiB data, 739 MiB used, 1.0 TiB / 1.0 TiB avail; 149 KiB/s rd, 1.0 MiB/s wr, 158 op/s 2024-03-24T00:21:24.710 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:21:24 smithi193 ceph-mon[36878]: mds.? [v2:172.21.15.189:6836/596160018,v1:172.21.15.189:6837/596160018] up:resolve 2024-03-24T00:21:24.710 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:21:24 smithi193 ceph-mon[36878]: fsmap cephfs:5/5 3 up:standby-replay 4 up:standby 2 up:resolve 3 up:active 2024-03-24T00:21:24.711 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:21:24 smithi193 ceph-mon[36878]: pgmap v579: 97 pgs: 97 active+clean; 138 MiB data, 739 MiB used, 1.0 TiB / 1.0 TiB avail; 149 KiB/s rd, 1.0 MiB/s wr, 158 op/s 2024-03-24T00:21:25.359 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:21:25.358+0000 7ff952a00640 1 -- 172.21.15.118:0/1441390391 <== mds.0 v2:172.21.15.189:6838/4074484168 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+44 (crc 0 0 0) 0x55d096da80c0 con 0x55d096eada30 2024-03-24T00:21:25.359 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:21:25.358+0000 7ff952a00640 10 client.26747 handle_command_reply: tid=0 2024-03-24T00:21:25.359 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:21:25.358+0000 7ff976783740 1 client.26747 shutdown 2024-03-24T00:21:25.359 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:21:25.359+0000 7ff976783740 1 -- 172.21.15.118:0/1441390391 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55d096e3bbd0 msgr2=0x55d096e4ae40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:21:25.359 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:21:25.359+0000 7ff976783740 1 --2- 172.21.15.118:0/1441390391 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55d096e3bbd0 0x55d096e4ae40 secure :-1 s=READY pgs=421 cs=0 l=1 rev1=1 crypto rx=0x7ff95802c7b0 tx=0x7ff958004c70 comp rx=0 tx=0).stop 2024-03-24T00:21:25.360 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:21:25.359+0000 7ff976783740 1 -- 172.21.15.118:0/1441390391 shutdown_connections 2024-03-24T00:21:25.360 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:21:25.359+0000 7ff976783740 1 --2- 172.21.15.118:0/1441390391 >> [v2:172.21.15.189:6838/4074484168,v1:172.21.15.189:6839/4074484168] conn(0x55d096eada30 0x55d096eafe50 crc :-1 s=READY pgs=201 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:21:25.360 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:21:25.359+0000 7ff976783740 1 --2- 172.21.15.118:0/1441390391 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55d096e40d40 0x55d096e4be50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:21:25.360 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:21:25.359+0000 7ff976783740 1 --2- 172.21.15.118:0/1441390391 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55d096e3bbd0 0x55d096e4ae40 unknown :-1 s=CLOSED pgs=421 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:21:25.360 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:21:25.359+0000 7ff976783740 1 --2- 172.21.15.118:0/1441390391 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55d096e3b1e0 0x55d096e4a900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:21:25.360 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:21:25.359+0000 7ff976783740 1 -- 172.21.15.118:0/1441390391 >> 172.21.15.118:0/1441390391 conn(0x55d096e36750 msgr2=0x55d096e3cf50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:21:25.360 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:21:25.359+0000 7ff952a00640 0 client.26747 ms_handle_reset on v2:172.21.15.189:6838/4074484168 2024-03-24T00:21:25.360 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:21:25.359+0000 7ff976783740 1 -- 172.21.15.118:0/1441390391 shutdown_connections 2024-03-24T00:21:25.360 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:21:25.359+0000 7ff976783740 1 -- 172.21.15.118:0/1441390391 wait complete. 2024-03-24T00:21:25.360 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:21:25.360+0000 7ff976783740 20 client.26747 trim_cache size 0 max 16384 2024-03-24T00:21:25.361 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stdout:{ 2024-03-24T00:21:25.361 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stdout: "message": "", 2024-03-24T00:21:25.361 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stdout: "return_code": 0 2024-03-24T00:21:25.361 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stdout:} 2024-03-24T00:21:25.363 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:21:25.360+0000 7ff973a00640 1 -- 172.21.15.118:0/696984240 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7ff9540a2920 msgr2=0x7ff9540a4de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:21:25.363 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:21:25.361+0000 7ff973a00640 1 --2- 172.21.15.118:0/696984240 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7ff9540a2920 0x7ff9540a4de0 secure :-1 s=READY pgs=363 cs=0 l=1 rev1=1 crypto rx=0x7ff960097c20 tx=0x7ff9600d2bf0 comp rx=0 tx=0).stop 2024-03-24T00:21:25.363 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:21:25.361+0000 7ff973a00640 1 -- 172.21.15.118:0/696984240 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff96c107d00 msgr2=0x7ff96c19df70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:21:25.363 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:21:25.361+0000 7ff973a00640 1 --2- 172.21.15.118:0/696984240 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff96c107d00 0x7ff96c19df70 secure :-1 s=READY pgs=419 cs=0 l=1 rev1=1 crypto rx=0x7ff96000e9c0 tx=0x7ff96000ee80 comp rx=0 tx=0).stop 2024-03-24T00:21:25.363 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:21:25.361+0000 7ff973a00640 1 -- 172.21.15.118:0/696984240 shutdown_connections 2024-03-24T00:21:25.364 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:21:25.361+0000 7ff973a00640 1 --2- 172.21.15.118:0/696984240 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7ff9540a2920 0x7ff9540a4de0 unknown :-1 s=CLOSED pgs=363 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:21:25.364 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:21:25.361+0000 7ff973a00640 1 --2- 172.21.15.118:0/696984240 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7ff96c1098b0 0x7ff96c1a5520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:21:25.364 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:21:25.361+0000 7ff973a00640 1 --2- 172.21.15.118:0/696984240 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7ff96c108ef0 0x7ff96c19e4b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:21:25.364 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:21:25.361+0000 7ff973a00640 1 --2- 172.21.15.118:0/696984240 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff96c107d00 0x7ff96c19df70 unknown :-1 s=CLOSED pgs=419 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:21:25.364 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:21:25.361+0000 7ff973a00640 1 -- 172.21.15.118:0/696984240 >> 172.21.15.118:0/696984240 conn(0x7ff96c076cb0 msgr2=0x7ff96c10c190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:21:25.365 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:21:25.361+0000 7ff973a00640 1 -- 172.21.15.118:0/696984240 shutdown_connections 2024-03-24T00:21:25.365 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:21:25.361+0000 7ff973a00640 1 -- 172.21.15.118:0/696984240 wait complete. 2024-03-24T00:21:25.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:21:25 smithi118 ceph-mon[31535]: mds.? [v2:172.21.15.193:6832/4276594827,v1:172.21.15.193:6833/4276594827] up:reconnect 2024-03-24T00:21:25.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:21:25 smithi118 ceph-mon[31535]: mds.? [v2:172.21.15.189:6836/596160018,v1:172.21.15.189:6837/596160018] up:reconnect 2024-03-24T00:21:25.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:21:25 smithi118 ceph-mon[31535]: fsmap cephfs:5/5 3 up:standby-replay 4 up:standby 2 up:reconnect 3 up:active 2024-03-24T00:21:25.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:21:25 smithi118 ceph-mon[31535]: reconnect by client.15159 192.168.0.1:0/3463739313 after 0.0439993 2024-03-24T00:21:25.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:21:25 smithi118 ceph-mon[31535]: reconnect by client.15159 192.168.0.1:0/3463739313 after 0.0589989 2024-03-24T00:21:25.703 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:21:25 smithi189 ceph-mon[36187]: mds.? [v2:172.21.15.193:6832/4276594827,v1:172.21.15.193:6833/4276594827] up:reconnect 2024-03-24T00:21:25.704 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:21:25 smithi189 ceph-mon[36187]: mds.? [v2:172.21.15.189:6836/596160018,v1:172.21.15.189:6837/596160018] up:reconnect 2024-03-24T00:21:25.704 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:21:25 smithi189 ceph-mon[36187]: fsmap cephfs:5/5 3 up:standby-replay 4 up:standby 2 up:reconnect 3 up:active 2024-03-24T00:21:25.704 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:21:25 smithi189 ceph-mon[36187]: reconnect by client.15159 192.168.0.1:0/3463739313 after 0.0439993 2024-03-24T00:21:25.704 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:21:25 smithi189 ceph-mon[36187]: reconnect by client.15159 192.168.0.1:0/3463739313 after 0.0589989 2024-03-24T00:21:25.710 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:21:25 smithi193 ceph-mon[36878]: mds.? [v2:172.21.15.193:6832/4276594827,v1:172.21.15.193:6833/4276594827] up:reconnect 2024-03-24T00:21:25.710 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:21:25 smithi193 ceph-mon[36878]: mds.? [v2:172.21.15.189:6836/596160018,v1:172.21.15.189:6837/596160018] up:reconnect 2024-03-24T00:21:25.711 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:21:25 smithi193 ceph-mon[36878]: fsmap cephfs:5/5 3 up:standby-replay 4 up:standby 2 up:reconnect 3 up:active 2024-03-24T00:21:25.711 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:21:25 smithi193 ceph-mon[36878]: reconnect by client.15159 192.168.0.1:0/3463739313 after 0.0439993 2024-03-24T00:21:25.711 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:21:25 smithi193 ceph-mon[36878]: reconnect by client.15159 192.168.0.1:0/3463739313 after 0.0589989 2024-03-24T00:21:26.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:21:26 smithi118 ceph-mon[31535]: mds.? [v2:172.21.15.193:6832/4276594827,v1:172.21.15.193:6833/4276594827] up:rejoin 2024-03-24T00:21:26.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:21:26 smithi118 ceph-mon[31535]: mds.? [v2:172.21.15.189:6836/596160018,v1:172.21.15.189:6837/596160018] up:rejoin 2024-03-24T00:21:26.697 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:21:26 smithi118 ceph-mon[31535]: fsmap cephfs:5/5 3 up:standby-replay 4 up:standby 2 up:rejoin 3 up:active 2024-03-24T00:21:26.697 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:21:26 smithi118 ceph-mon[31535]: daemon mds.l is now active in filesystem cephfs as rank 1 2024-03-24T00:21:26.697 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:21:26 smithi118 ceph-mon[31535]: daemon mds.h is now active in filesystem cephfs as rank 3 2024-03-24T00:21:26.697 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:21:26 smithi118 ceph-mon[31535]: pgmap v580: 97 pgs: 97 active+clean; 138 MiB data, 740 MiB used, 1.0 TiB / 1.0 TiB avail; 126 KiB/s rd, 1.1 MiB/s wr, 207 op/s 2024-03-24T00:21:26.704 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:21:26 smithi189 ceph-mon[36187]: mds.? [v2:172.21.15.193:6832/4276594827,v1:172.21.15.193:6833/4276594827] up:rejoin 2024-03-24T00:21:26.704 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:21:26 smithi189 ceph-mon[36187]: mds.? [v2:172.21.15.189:6836/596160018,v1:172.21.15.189:6837/596160018] up:rejoin 2024-03-24T00:21:26.704 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:21:26 smithi189 ceph-mon[36187]: fsmap cephfs:5/5 3 up:standby-replay 4 up:standby 2 up:rejoin 3 up:active 2024-03-24T00:21:26.704 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:21:26 smithi189 ceph-mon[36187]: daemon mds.l is now active in filesystem cephfs as rank 1 2024-03-24T00:21:26.704 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:21:26 smithi189 ceph-mon[36187]: daemon mds.h is now active in filesystem cephfs as rank 3 2024-03-24T00:21:26.704 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:21:26 smithi189 ceph-mon[36187]: pgmap v580: 97 pgs: 97 active+clean; 138 MiB data, 740 MiB used, 1.0 TiB / 1.0 TiB avail; 126 KiB/s rd, 1.1 MiB/s wr, 207 op/s 2024-03-24T00:21:26.705 INFO:journalctl@ceph.mds.h.smithi189.stdout:Mar 24 00:21:26 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-h[69653]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: In function 'void Locker::set_xlocks_done(MutationImpl*, bool)' thread 7f40dfa10700 time 2024-03-24T00:21:26.396713+0000 2024-03-24T00:21:26.705 INFO:journalctl@ceph.mds.h.smithi189.stdout:Mar 24 00:21:26 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-h[69653]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: 626: FAILED ceph_assert(obj->is_auth()) 2024-03-24T00:21:26.705 INFO:journalctl@ceph.mds.h.smithi189.stdout:Mar 24 00:21:26 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-h[69653]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-24T00:21:26.705 INFO:journalctl@ceph.mds.h.smithi189.stdout:Mar 24 00:21:26 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-h[69653]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x135) [0x7f40ea03a4a1] 2024-03-24T00:21:26.705 INFO:journalctl@ceph.mds.h.smithi189.stdout:Mar 24 00:21:26 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-h[69653]: 2: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7f40ea03a667] 2024-03-24T00:21:26.705 INFO:journalctl@ceph.mds.h.smithi189.stdout:Mar 24 00:21:26 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-h[69653]: 3: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x560786b64223] 2024-03-24T00:21:26.705 INFO:journalctl@ceph.mds.h.smithi189.stdout:Mar 24 00:21:26 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-h[69653]: 4: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x5607869de5d6] 2024-03-24T00:21:26.705 INFO:journalctl@ceph.mds.h.smithi189.stdout:Mar 24 00:21:26 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-h[69653]: 5: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x5607869deda1] 2024-03-24T00:21:26.705 INFO:journalctl@ceph.mds.h.smithi189.stdout:Mar 24 00:21:26 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-h[69653]: 6: (Server::handle_client_rename(boost::intrusive_ptr const&)+0x259d) [0x560786a1b97d] 2024-03-24T00:21:26.705 INFO:journalctl@ceph.mds.h.smithi189.stdout:Mar 24 00:21:26 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-h[69653]: 7: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x560786a31233] 2024-03-24T00:21:26.706 INFO:journalctl@ceph.mds.h.smithi189.stdout:Mar 24 00:21:26 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-h[69653]: 8: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x560786a33197] 2024-03-24T00:21:26.706 INFO:journalctl@ceph.mds.h.smithi189.stdout:Mar 24 00:21:26 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-h[69653]: 9: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x56078693211b] 2024-03-24T00:21:26.706 INFO:journalctl@ceph.mds.h.smithi189.stdout:Mar 24 00:21:26 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-h[69653]: 10: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x5607869326bc] 2024-03-24T00:21:26.706 INFO:journalctl@ceph.mds.h.smithi189.stdout:Mar 24 00:21:26 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-h[69653]: 11: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x56078691c1ff] 2024-03-24T00:21:26.706 INFO:journalctl@ceph.mds.h.smithi189.stdout:Mar 24 00:21:26 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-h[69653]: 12: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7f40ea2cc488] 2024-03-24T00:21:26.706 INFO:journalctl@ceph.mds.h.smithi189.stdout:Mar 24 00:21:26 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-h[69653]: 13: (DispatchQueue::entry()+0x51f) [0x7f40ea2c960f] 2024-03-24T00:21:26.706 INFO:journalctl@ceph.mds.h.smithi189.stdout:Mar 24 00:21:26 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-h[69653]: 14: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f40ea39ce81] 2024-03-24T00:21:26.706 INFO:journalctl@ceph.mds.h.smithi189.stdout:Mar 24 00:21:26 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-h[69653]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7f40e8db71ca] 2024-03-24T00:21:26.706 INFO:journalctl@ceph.mds.h.smithi189.stdout:Mar 24 00:21:26 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-h[69653]: 16: clone() 2024-03-24T00:21:26.706 INFO:journalctl@ceph.mds.h.smithi189.stdout:Mar 24 00:21:26 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-h[69653]: 2024-03-24T00:21:26.397+0000 7f40dfa10700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: In function 'void Locker::set_xlocks_done(MutationImpl*, bool)' thread 7f40dfa10700 time 2024-03-24T00:21:26.396713+0000 2024-03-24T00:21:26.706 INFO:journalctl@ceph.mds.h.smithi189.stdout:Mar 24 00:21:26 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-h[69653]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: 626: FAILED ceph_assert(obj->is_auth()) 2024-03-24T00:21:26.706 INFO:journalctl@ceph.mds.h.smithi189.stdout:Mar 24 00:21:26 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-h[69653]: 2024-03-24T00:21:26.706 INFO:journalctl@ceph.mds.h.smithi189.stdout:Mar 24 00:21:26 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-h[69653]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-24T00:21:26.707 INFO:journalctl@ceph.mds.h.smithi189.stdout:Mar 24 00:21:26 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-h[69653]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x135) [0x7f40ea03a4a1] 2024-03-24T00:21:26.707 INFO:journalctl@ceph.mds.h.smithi189.stdout:Mar 24 00:21:26 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-h[69653]: 2: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7f40ea03a667] 2024-03-24T00:21:26.707 INFO:journalctl@ceph.mds.h.smithi189.stdout:Mar 24 00:21:26 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-h[69653]: 3: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x560786b64223] 2024-03-24T00:21:26.707 INFO:journalctl@ceph.mds.h.smithi189.stdout:Mar 24 00:21:26 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-h[69653]: 4: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x5607869de5d6] 2024-03-24T00:21:26.707 INFO:journalctl@ceph.mds.h.smithi189.stdout:Mar 24 00:21:26 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-h[69653]: 5: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x5607869deda1] 2024-03-24T00:21:26.707 INFO:journalctl@ceph.mds.h.smithi189.stdout:Mar 24 00:21:26 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-h[69653]: 6: (Server::handle_client_rename(boost::intrusive_ptr const&)+0x259d) [0x560786a1b97d] 2024-03-24T00:21:26.707 INFO:journalctl@ceph.mds.h.smithi189.stdout:Mar 24 00:21:26 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-h[69653]: 7: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x560786a31233] 2024-03-24T00:21:26.707 INFO:journalctl@ceph.mds.h.smithi189.stdout:Mar 24 00:21:26 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-h[69653]: 8: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x560786a33197] 2024-03-24T00:21:26.707 INFO:journalctl@ceph.mds.h.smithi189.stdout:Mar 24 00:21:26 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-h[69653]: 9: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x56078693211b] 2024-03-24T00:21:26.707 INFO:journalctl@ceph.mds.h.smithi189.stdout:Mar 24 00:21:26 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-h[69653]: 10: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x5607869326bc] 2024-03-24T00:21:26.707 INFO:journalctl@ceph.mds.h.smithi189.stdout:Mar 24 00:21:26 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-h[69653]: 11: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x56078691c1ff] 2024-03-24T00:21:26.707 INFO:journalctl@ceph.mds.h.smithi189.stdout:Mar 24 00:21:26 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-h[69653]: 12: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7f40ea2cc488] 2024-03-24T00:21:26.707 INFO:journalctl@ceph.mds.h.smithi189.stdout:Mar 24 00:21:26 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-h[69653]: 13: (DispatchQueue::entry()+0x51f) [0x7f40ea2c960f] 2024-03-24T00:21:26.707 INFO:journalctl@ceph.mds.h.smithi189.stdout:Mar 24 00:21:26 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-h[69653]: 14: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f40ea39ce81] 2024-03-24T00:21:26.708 INFO:journalctl@ceph.mds.h.smithi189.stdout:Mar 24 00:21:26 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-h[69653]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7f40e8db71ca] 2024-03-24T00:21:26.708 INFO:journalctl@ceph.mds.h.smithi189.stdout:Mar 24 00:21:26 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-h[69653]: 16: clone() 2024-03-24T00:21:26.708 INFO:journalctl@ceph.mds.h.smithi189.stdout:Mar 24 00:21:26 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-h[69653]: 2024-03-24T00:21:26.708 INFO:journalctl@ceph.mds.h.smithi189.stdout:Mar 24 00:21:26 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-h[69653]: *** Caught signal (Aborted) ** 2024-03-24T00:21:26.708 INFO:journalctl@ceph.mds.h.smithi189.stdout:Mar 24 00:21:26 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-h[69653]: in thread 7f40dfa10700 thread_name:ms_dispatch 2024-03-24T00:21:26.708 INFO:journalctl@ceph.mds.h.smithi189.stdout:Mar 24 00:21:26 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-h[69653]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-24T00:21:26.708 INFO:journalctl@ceph.mds.h.smithi189.stdout:Mar 24 00:21:26 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-h[69653]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f40e8dc1d20] 2024-03-24T00:21:26.709 INFO:journalctl@ceph.mds.h.smithi189.stdout:Mar 24 00:21:26 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-h[69653]: 2: gsignal() 2024-03-24T00:21:26.709 INFO:journalctl@ceph.mds.h.smithi189.stdout:Mar 24 00:21:26 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-h[69653]: 3: abort() 2024-03-24T00:21:26.709 INFO:journalctl@ceph.mds.h.smithi189.stdout:Mar 24 00:21:26 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-h[69653]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x18f) [0x7f40ea03a4fb] 2024-03-24T00:21:26.709 INFO:journalctl@ceph.mds.h.smithi189.stdout:Mar 24 00:21:26 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-h[69653]: 5: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7f40ea03a667] 2024-03-24T00:21:26.709 INFO:journalctl@ceph.mds.h.smithi189.stdout:Mar 24 00:21:26 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-h[69653]: 6: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x560786b64223] 2024-03-24T00:21:26.709 INFO:journalctl@ceph.mds.h.smithi189.stdout:Mar 24 00:21:26 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-h[69653]: 7: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x5607869de5d6] 2024-03-24T00:21:26.709 INFO:journalctl@ceph.mds.h.smithi189.stdout:Mar 24 00:21:26 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-h[69653]: 8: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x5607869deda1] 2024-03-24T00:21:26.710 INFO:journalctl@ceph.mds.h.smithi189.stdout:Mar 24 00:21:26 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-h[69653]: 9: (Server::handle_client_rename(boost::intrusive_ptr const&)+0x259d) [0x560786a1b97d] 2024-03-24T00:21:26.710 INFO:journalctl@ceph.mds.h.smithi189.stdout:Mar 24 00:21:26 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-h[69653]: 10: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x560786a31233] 2024-03-24T00:21:26.710 INFO:journalctl@ceph.mds.h.smithi189.stdout:Mar 24 00:21:26 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-h[69653]: 11: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x560786a33197] 2024-03-24T00:21:26.710 INFO:journalctl@ceph.mds.h.smithi189.stdout:Mar 24 00:21:26 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-h[69653]: 12: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x56078693211b] 2024-03-24T00:21:26.710 INFO:journalctl@ceph.mds.h.smithi189.stdout:Mar 24 00:21:26 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-h[69653]: 13: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x5607869326bc] 2024-03-24T00:21:26.710 INFO:journalctl@ceph.mds.h.smithi189.stdout:Mar 24 00:21:26 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-h[69653]: 14: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x56078691c1ff] 2024-03-24T00:21:26.710 INFO:journalctl@ceph.mds.h.smithi189.stdout:Mar 24 00:21:26 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-h[69653]: 15: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7f40ea2cc488] 2024-03-24T00:21:26.710 INFO:journalctl@ceph.mds.h.smithi189.stdout:Mar 24 00:21:26 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-h[69653]: 16: (DispatchQueue::entry()+0x51f) [0x7f40ea2c960f] 2024-03-24T00:21:26.710 INFO:journalctl@ceph.mds.h.smithi189.stdout:Mar 24 00:21:26 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-h[69653]: 17: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f40ea39ce81] 2024-03-24T00:21:26.710 INFO:journalctl@ceph.mds.h.smithi189.stdout:Mar 24 00:21:26 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-h[69653]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7f40e8db71ca] 2024-03-24T00:21:26.710 INFO:journalctl@ceph.mds.h.smithi189.stdout:Mar 24 00:21:26 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-h[69653]: 19: clone() 2024-03-24T00:21:26.710 INFO:journalctl@ceph.mds.h.smithi189.stdout:Mar 24 00:21:26 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-h[69653]: 2024-03-24T00:21:26.399+0000 7f40dfa10700 -1 *** Caught signal (Aborted) ** 2024-03-24T00:21:26.710 INFO:journalctl@ceph.mds.h.smithi189.stdout:Mar 24 00:21:26 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-h[69653]: in thread 7f40dfa10700 thread_name:ms_dispatch 2024-03-24T00:21:26.711 INFO:journalctl@ceph.mds.h.smithi189.stdout:Mar 24 00:21:26 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-h[69653]: 2024-03-24T00:21:26.711 INFO:journalctl@ceph.mds.h.smithi189.stdout:Mar 24 00:21:26 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-h[69653]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-24T00:21:26.711 INFO:journalctl@ceph.mds.h.smithi189.stdout:Mar 24 00:21:26 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-h[69653]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f40e8dc1d20] 2024-03-24T00:21:26.711 INFO:journalctl@ceph.mds.h.smithi189.stdout:Mar 24 00:21:26 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-h[69653]: 2: gsignal() 2024-03-24T00:21:26.711 INFO:journalctl@ceph.mds.h.smithi189.stdout:Mar 24 00:21:26 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-h[69653]: 3: abort() 2024-03-24T00:21:26.711 INFO:journalctl@ceph.mds.h.smithi189.stdout:Mar 24 00:21:26 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-h[69653]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x18f) [0x7f40ea03a4fb] 2024-03-24T00:21:26.711 INFO:journalctl@ceph.mds.h.smithi189.stdout:Mar 24 00:21:26 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-h[69653]: 5: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7f40ea03a667] 2024-03-24T00:21:26.711 INFO:journalctl@ceph.mds.h.smithi189.stdout:Mar 24 00:21:26 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-h[69653]: 6: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x560786b64223] 2024-03-24T00:21:26.711 INFO:journalctl@ceph.mds.h.smithi189.stdout:Mar 24 00:21:26 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-h[69653]: 7: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x5607869de5d6] 2024-03-24T00:21:26.711 INFO:journalctl@ceph.mds.h.smithi189.stdout:Mar 24 00:21:26 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-h[69653]: 8: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x5607869deda1] 2024-03-24T00:21:26.711 INFO:journalctl@ceph.mds.h.smithi189.stdout:Mar 24 00:21:26 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-h[69653]: 9: (Server::handle_client_rename(boost::intrusive_ptr const&)+0x259d) [0x560786a1b97d] 2024-03-24T00:21:26.711 INFO:journalctl@ceph.mds.h.smithi189.stdout:Mar 24 00:21:26 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-h[69653]: 10: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x560786a31233] 2024-03-24T00:21:26.711 INFO:journalctl@ceph.mds.h.smithi189.stdout:Mar 24 00:21:26 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-h[69653]: 11: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x560786a33197] 2024-03-24T00:21:26.711 INFO:journalctl@ceph.mds.h.smithi189.stdout:Mar 24 00:21:26 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-h[69653]: 12: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x56078693211b] 2024-03-24T00:21:26.712 INFO:journalctl@ceph.mds.h.smithi189.stdout:Mar 24 00:21:26 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-h[69653]: 13: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x5607869326bc] 2024-03-24T00:21:26.712 INFO:journalctl@ceph.mds.h.smithi189.stdout:Mar 24 00:21:26 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-h[69653]: 14: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x56078691c1ff] 2024-03-24T00:21:26.712 INFO:journalctl@ceph.mds.h.smithi189.stdout:Mar 24 00:21:26 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-h[69653]: 15: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7f40ea2cc488] 2024-03-24T00:21:26.712 INFO:journalctl@ceph.mds.h.smithi189.stdout:Mar 24 00:21:26 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-h[69653]: 16: (DispatchQueue::entry()+0x51f) [0x7f40ea2c960f] 2024-03-24T00:21:26.712 INFO:journalctl@ceph.mds.h.smithi189.stdout:Mar 24 00:21:26 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-h[69653]: 17: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f40ea39ce81] 2024-03-24T00:21:26.712 INFO:journalctl@ceph.mds.h.smithi189.stdout:Mar 24 00:21:26 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-h[69653]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7f40e8db71ca] 2024-03-24T00:21:26.712 INFO:journalctl@ceph.mds.h.smithi189.stdout:Mar 24 00:21:26 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-h[69653]: 19: clone() 2024-03-24T00:21:26.712 INFO:journalctl@ceph.mds.h.smithi189.stdout:Mar 24 00:21:26 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-h[69653]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-03-24T00:21:26.712 INFO:journalctl@ceph.mds.h.smithi189.stdout:Mar 24 00:21:26 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-h[69653]: 2024-03-24T00:21:26.712 INFO:journalctl@ceph.mds.h.smithi189.stdout:Mar 24 00:21:26 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-h[69653]: -1> 2024-03-24T00:21:26.397+0000 7f40dfa10700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: In function 'void Locker::set_xlocks_done(MutationImpl*, bool)' thread 7f40dfa10700 time 2024-03-24T00:21:26.396713+0000 2024-03-24T00:21:26.712 INFO:journalctl@ceph.mds.h.smithi189.stdout:Mar 24 00:21:26 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-h[69653]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: 626: FAILED ceph_assert(obj->is_auth()) 2024-03-24T00:21:26.712 INFO:journalctl@ceph.mds.h.smithi189.stdout:Mar 24 00:21:26 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-h[69653]: 2024-03-24T00:21:26.712 INFO:journalctl@ceph.mds.h.smithi189.stdout:Mar 24 00:21:26 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-h[69653]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-24T00:21:26.712 INFO:journalctl@ceph.mds.h.smithi189.stdout:Mar 24 00:21:26 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-h[69653]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x135) [0x7f40ea03a4a1] 2024-03-24T00:21:26.712 INFO:journalctl@ceph.mds.h.smithi189.stdout:Mar 24 00:21:26 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-h[69653]: 2: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7f40ea03a667] 2024-03-24T00:21:26.713 INFO:journalctl@ceph.mds.h.smithi189.stdout:Mar 24 00:21:26 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-h[69653]: 3: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x560786b64223] 2024-03-24T00:21:26.713 INFO:journalctl@ceph.mds.h.smithi189.stdout:Mar 24 00:21:26 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-h[69653]: 4: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x5607869de5d6] 2024-03-24T00:21:26.713 INFO:journalctl@ceph.mds.h.smithi189.stdout:Mar 24 00:21:26 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-h[69653]: 5: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x5607869deda1] 2024-03-24T00:21:26.713 INFO:journalctl@ceph.mds.h.smithi189.stdout:Mar 24 00:21:26 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-h[69653]: 6: (Server::handle_client_rename(boost::intrusive_ptr const&)+0x259d) [0x560786a1b97d] 2024-03-24T00:21:26.713 INFO:journalctl@ceph.mds.h.smithi189.stdout:Mar 24 00:21:26 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-h[69653]: 7: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x560786a31233] 2024-03-24T00:21:26.713 INFO:journalctl@ceph.mds.h.smithi189.stdout:Mar 24 00:21:26 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-h[69653]: 8: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x560786a33197] 2024-03-24T00:21:26.713 INFO:journalctl@ceph.mds.h.smithi189.stdout:Mar 24 00:21:26 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-h[69653]: 9: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x56078693211b] 2024-03-24T00:21:26.713 INFO:journalctl@ceph.mds.h.smithi189.stdout:Mar 24 00:21:26 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-h[69653]: 10: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x5607869326bc] 2024-03-24T00:21:26.713 INFO:journalctl@ceph.mds.h.smithi189.stdout:Mar 24 00:21:26 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-h[69653]: 11: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x56078691c1ff] 2024-03-24T00:21:26.713 INFO:journalctl@ceph.mds.h.smithi189.stdout:Mar 24 00:21:26 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-h[69653]: 12: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7f40ea2cc488] 2024-03-24T00:21:26.713 INFO:journalctl@ceph.mds.h.smithi189.stdout:Mar 24 00:21:26 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-h[69653]: 13: (DispatchQueue::entry()+0x51f) [0x7f40ea2c960f] 2024-03-24T00:21:26.713 INFO:journalctl@ceph.mds.h.smithi189.stdout:Mar 24 00:21:26 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-h[69653]: 14: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f40ea39ce81] 2024-03-24T00:21:26.713 INFO:journalctl@ceph.mds.h.smithi189.stdout:Mar 24 00:21:26 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-h[69653]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7f40e8db71ca] 2024-03-24T00:21:26.713 INFO:journalctl@ceph.mds.h.smithi189.stdout:Mar 24 00:21:26 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-h[69653]: 16: clone() 2024-03-24T00:21:26.714 INFO:journalctl@ceph.mds.h.smithi189.stdout:Mar 24 00:21:26 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-h[69653]: 2024-03-24T00:21:26.714 INFO:journalctl@ceph.mds.h.smithi189.stdout:Mar 24 00:21:26 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-h[69653]: 0> 2024-03-24T00:21:26.399+0000 7f40dfa10700 -1 *** Caught signal (Aborted) ** 2024-03-24T00:21:26.714 INFO:journalctl@ceph.mds.h.smithi189.stdout:Mar 24 00:21:26 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-h[69653]: in thread 7f40dfa10700 thread_name:ms_dispatch 2024-03-24T00:21:26.714 INFO:journalctl@ceph.mds.h.smithi189.stdout:Mar 24 00:21:26 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-h[69653]: 2024-03-24T00:21:26.714 INFO:journalctl@ceph.mds.h.smithi189.stdout:Mar 24 00:21:26 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-h[69653]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-24T00:21:26.714 INFO:journalctl@ceph.mds.h.smithi189.stdout:Mar 24 00:21:26 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-h[69653]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f40e8dc1d20] 2024-03-24T00:21:26.714 INFO:journalctl@ceph.mds.h.smithi189.stdout:Mar 24 00:21:26 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-h[69653]: 2: gsignal() 2024-03-24T00:21:26.714 INFO:journalctl@ceph.mds.h.smithi189.stdout:Mar 24 00:21:26 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-h[69653]: 3: abort() 2024-03-24T00:21:26.714 INFO:journalctl@ceph.mds.h.smithi189.stdout:Mar 24 00:21:26 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-h[69653]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x18f) [0x7f40ea03a4fb] 2024-03-24T00:21:26.714 INFO:journalctl@ceph.mds.h.smithi189.stdout:Mar 24 00:21:26 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-h[69653]: 5: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7f40ea03a667] 2024-03-24T00:21:26.714 INFO:journalctl@ceph.mds.h.smithi189.stdout:Mar 24 00:21:26 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-h[69653]: 6: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x560786b64223] 2024-03-24T00:21:26.714 INFO:journalctl@ceph.mds.h.smithi189.stdout:Mar 24 00:21:26 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-h[69653]: 7: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x5607869de5d6] 2024-03-24T00:21:26.714 INFO:journalctl@ceph.mds.h.smithi189.stdout:Mar 24 00:21:26 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-h[69653]: 8: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x5607869deda1] 2024-03-24T00:21:26.715 INFO:journalctl@ceph.mds.h.smithi189.stdout:Mar 24 00:21:26 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-h[69653]: 9: (Server::handle_client_rename(boost::intrusive_ptr const&)+0x259d) [0x560786a1b97d] 2024-03-24T00:21:26.715 INFO:journalctl@ceph.mds.h.smithi189.stdout:Mar 24 00:21:26 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-h[69653]: 10: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x560786a31233] 2024-03-24T00:21:26.715 INFO:journalctl@ceph.mds.h.smithi189.stdout:Mar 24 00:21:26 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-h[69653]: 11: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x560786a33197] 2024-03-24T00:21:26.715 INFO:journalctl@ceph.mds.h.smithi189.stdout:Mar 24 00:21:26 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-h[69653]: 12: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x56078693211b] 2024-03-24T00:21:26.715 INFO:journalctl@ceph.mds.h.smithi189.stdout:Mar 24 00:21:26 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-h[69653]: 13: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x5607869326bc] 2024-03-24T00:21:26.715 INFO:journalctl@ceph.mds.h.smithi189.stdout:Mar 24 00:21:26 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-h[69653]: 14: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x56078691c1ff] 2024-03-24T00:21:26.715 INFO:journalctl@ceph.mds.h.smithi189.stdout:Mar 24 00:21:26 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-h[69653]: 15: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7f40ea2cc488] 2024-03-24T00:21:26.715 INFO:journalctl@ceph.mds.h.smithi189.stdout:Mar 24 00:21:26 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-h[69653]: 16: (DispatchQueue::entry()+0x51f) [0x7f40ea2c960f] 2024-03-24T00:21:26.715 INFO:journalctl@ceph.mds.h.smithi189.stdout:Mar 24 00:21:26 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-h[69653]: 17: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f40ea39ce81] 2024-03-24T00:21:26.715 INFO:journalctl@ceph.mds.h.smithi189.stdout:Mar 24 00:21:26 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-h[69653]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7f40e8db71ca] 2024-03-24T00:21:26.715 INFO:journalctl@ceph.mds.h.smithi189.stdout:Mar 24 00:21:26 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-h[69653]: 19: clone() 2024-03-24T00:21:26.715 INFO:journalctl@ceph.mds.h.smithi189.stdout:Mar 24 00:21:26 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-h[69653]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-03-24T00:21:26.715 INFO:journalctl@ceph.mds.h.smithi189.stdout:Mar 24 00:21:26 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-h[69653]: 2024-03-24T00:21:26.715 INFO:journalctl@ceph.mds.h.smithi189.stdout:Mar 24 00:21:26 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-h[69653]: -1> 2024-03-24T00:21:26.397+0000 7f40dfa10700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: In function 'void Locker::set_xlocks_done(MutationImpl*, bool)' thread 7f40dfa10700 time 2024-03-24T00:21:26.396713+0000 2024-03-24T00:21:26.715 INFO:journalctl@ceph.mds.h.smithi189.stdout:Mar 24 00:21:26 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-h[69653]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: 626: FAILED ceph_assert(obj->is_auth()) 2024-03-24T00:21:26.716 INFO:journalctl@ceph.mds.h.smithi189.stdout:Mar 24 00:21:26 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-h[69653]: 2024-03-24T00:21:26.716 INFO:journalctl@ceph.mds.h.smithi189.stdout:Mar 24 00:21:26 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-h[69653]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-24T00:21:26.716 INFO:journalctl@ceph.mds.h.smithi189.stdout:Mar 24 00:21:26 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-h[69653]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x135) [0x7f40ea03a4a1] 2024-03-24T00:21:26.716 INFO:journalctl@ceph.mds.h.smithi189.stdout:Mar 24 00:21:26 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-h[69653]: 2: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7f40ea03a667] 2024-03-24T00:21:26.716 INFO:journalctl@ceph.mds.h.smithi189.stdout:Mar 24 00:21:26 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-h[69653]: 3: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x560786b64223] 2024-03-24T00:21:26.716 INFO:journalctl@ceph.mds.h.smithi189.stdout:Mar 24 00:21:26 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-h[69653]: 4: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x5607869de5d6] 2024-03-24T00:21:26.716 INFO:journalctl@ceph.mds.h.smithi189.stdout:Mar 24 00:21:26 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-h[69653]: 5: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x5607869deda1] 2024-03-24T00:21:26.716 INFO:journalctl@ceph.mds.h.smithi189.stdout:Mar 24 00:21:26 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-h[69653]: 6: (Server::handle_client_rename(boost::intrusive_ptr const&)+0x259d) [0x560786a1b97d] 2024-03-24T00:21:26.716 INFO:journalctl@ceph.mds.h.smithi189.stdout:Mar 24 00:21:26 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-h[69653]: 7: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x560786a31233] 2024-03-24T00:21:26.716 INFO:journalctl@ceph.mds.h.smithi189.stdout:Mar 24 00:21:26 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-h[69653]: 8: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x560786a33197] 2024-03-24T00:21:26.716 INFO:journalctl@ceph.mds.h.smithi189.stdout:Mar 24 00:21:26 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-h[69653]: 9: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x56078693211b] 2024-03-24T00:21:26.716 INFO:journalctl@ceph.mds.h.smithi189.stdout:Mar 24 00:21:26 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-h[69653]: 10: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x5607869326bc] 2024-03-24T00:21:26.716 INFO:journalctl@ceph.mds.h.smithi189.stdout:Mar 24 00:21:26 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-h[69653]: 11: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x56078691c1ff] 2024-03-24T00:21:26.716 INFO:journalctl@ceph.mds.h.smithi189.stdout:Mar 24 00:21:26 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-h[69653]: 12: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7f40ea2cc488] 2024-03-24T00:21:26.717 INFO:journalctl@ceph.mds.h.smithi189.stdout:Mar 24 00:21:26 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-h[69653]: 13: (DispatchQueue::entry()+0x51f) [0x7f40ea2c960f] 2024-03-24T00:21:26.717 INFO:journalctl@ceph.mds.h.smithi189.stdout:Mar 24 00:21:26 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-h[69653]: 14: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f40ea39ce81] 2024-03-24T00:21:26.717 INFO:journalctl@ceph.mds.h.smithi189.stdout:Mar 24 00:21:26 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-h[69653]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7f40e8db71ca] 2024-03-24T00:21:26.717 INFO:journalctl@ceph.mds.h.smithi189.stdout:Mar 24 00:21:26 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-h[69653]: 16: clone() 2024-03-24T00:21:26.717 INFO:journalctl@ceph.mds.h.smithi189.stdout:Mar 24 00:21:26 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-h[69653]: 2024-03-24T00:21:26.717 INFO:journalctl@ceph.mds.h.smithi189.stdout:Mar 24 00:21:26 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-h[69653]: 0> 2024-03-24T00:21:26.399+0000 7f40dfa10700 -1 *** Caught signal (Aborted) ** 2024-03-24T00:21:26.717 INFO:journalctl@ceph.mds.h.smithi189.stdout:Mar 24 00:21:26 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-h[69653]: in thread 7f40dfa10700 thread_name:ms_dispatch 2024-03-24T00:21:26.717 INFO:journalctl@ceph.mds.h.smithi189.stdout:Mar 24 00:21:26 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-h[69653]: 2024-03-24T00:21:26.717 INFO:journalctl@ceph.mds.h.smithi189.stdout:Mar 24 00:21:26 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-h[69653]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-24T00:21:26.717 INFO:journalctl@ceph.mds.h.smithi189.stdout:Mar 24 00:21:26 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-h[69653]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f40e8dc1d20] 2024-03-24T00:21:26.717 INFO:journalctl@ceph.mds.h.smithi189.stdout:Mar 24 00:21:26 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-h[69653]: 2: gsignal() 2024-03-24T00:21:26.717 INFO:journalctl@ceph.mds.h.smithi189.stdout:Mar 24 00:21:26 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-h[69653]: 3: abort() 2024-03-24T00:21:26.717 INFO:journalctl@ceph.mds.h.smithi189.stdout:Mar 24 00:21:26 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-h[69653]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x18f) [0x7f40ea03a4fb] 2024-03-24T00:21:26.717 INFO:journalctl@ceph.mds.h.smithi189.stdout:Mar 24 00:21:26 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-h[69653]: 5: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7f40ea03a667] 2024-03-24T00:21:26.717 INFO:journalctl@ceph.mds.h.smithi189.stdout:Mar 24 00:21:26 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-h[69653]: 6: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x560786b64223] 2024-03-24T00:21:26.718 INFO:journalctl@ceph.mds.h.smithi189.stdout:Mar 24 00:21:26 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-h[69653]: 7: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x5607869de5d6] 2024-03-24T00:21:26.718 INFO:journalctl@ceph.mds.h.smithi189.stdout:Mar 24 00:21:26 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-h[69653]: 8: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x5607869deda1] 2024-03-24T00:21:26.718 INFO:journalctl@ceph.mds.h.smithi189.stdout:Mar 24 00:21:26 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-h[69653]: 9: (Server::handle_client_rename(boost::intrusive_ptr const&)+0x259d) [0x560786a1b97d] 2024-03-24T00:21:26.718 INFO:journalctl@ceph.mds.h.smithi189.stdout:Mar 24 00:21:26 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-h[69653]: 10: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x560786a31233] 2024-03-24T00:21:26.718 INFO:journalctl@ceph.mds.h.smithi189.stdout:Mar 24 00:21:26 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-h[69653]: 11: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x560786a33197] 2024-03-24T00:21:26.718 INFO:journalctl@ceph.mds.h.smithi189.stdout:Mar 24 00:21:26 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-h[69653]: 12: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x56078693211b] 2024-03-24T00:21:26.718 INFO:journalctl@ceph.mds.h.smithi189.stdout:Mar 24 00:21:26 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-h[69653]: 13: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x5607869326bc] 2024-03-24T00:21:26.718 INFO:journalctl@ceph.mds.h.smithi189.stdout:Mar 24 00:21:26 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-h[69653]: 14: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x56078691c1ff] 2024-03-24T00:21:26.718 INFO:journalctl@ceph.mds.h.smithi189.stdout:Mar 24 00:21:26 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-h[69653]: 15: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7f40ea2cc488] 2024-03-24T00:21:26.718 INFO:journalctl@ceph.mds.h.smithi189.stdout:Mar 24 00:21:26 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-h[69653]: 16: (DispatchQueue::entry()+0x51f) [0x7f40ea2c960f] 2024-03-24T00:21:26.718 INFO:journalctl@ceph.mds.h.smithi189.stdout:Mar 24 00:21:26 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-h[69653]: 17: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f40ea39ce81] 2024-03-24T00:21:26.718 INFO:journalctl@ceph.mds.h.smithi189.stdout:Mar 24 00:21:26 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-h[69653]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7f40e8db71ca] 2024-03-24T00:21:26.718 INFO:journalctl@ceph.mds.h.smithi189.stdout:Mar 24 00:21:26 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-h[69653]: 19: clone() 2024-03-24T00:21:26.719 INFO:journalctl@ceph.mds.h.smithi189.stdout:Mar 24 00:21:26 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-h[69653]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-03-24T00:21:26.719 INFO:journalctl@ceph.mds.h.smithi189.stdout:Mar 24 00:21:26 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-h[69653]: 2024-03-24T00:21:26.719 INFO:journalctl@ceph.mds.h.smithi189.stdout:Mar 24 00:21:26 smithi189 podman[69704]: 2024-03-24 00:21:26.47113912 +0000 UTC m=+0.029869668 container died 30d8025ae60b5197cceebadd800d3c1406438009abbf9ed61785e4b539e8412f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-h, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240319, ceph=True, org.label-schema.schema-version=1.0, GIT_CLEAN=True) 2024-03-24T00:21:26.719 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:21:26 smithi193 ceph-mon[36878]: mds.? [v2:172.21.15.193:6832/4276594827,v1:172.21.15.193:6833/4276594827] up:rejoin 2024-03-24T00:21:26.719 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:21:26 smithi193 ceph-mon[36878]: mds.? [v2:172.21.15.189:6836/596160018,v1:172.21.15.189:6837/596160018] up:rejoin 2024-03-24T00:21:26.719 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:21:26 smithi193 ceph-mon[36878]: fsmap cephfs:5/5 3 up:standby-replay 4 up:standby 2 up:rejoin 3 up:active 2024-03-24T00:21:26.719 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:21:26 smithi193 ceph-mon[36878]: daemon mds.l is now active in filesystem cephfs as rank 1 2024-03-24T00:21:26.719 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:21:26 smithi193 ceph-mon[36878]: daemon mds.h is now active in filesystem cephfs as rank 3 2024-03-24T00:21:26.719 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:21:26 smithi193 ceph-mon[36878]: pgmap v580: 97 pgs: 97 active+clean; 138 MiB data, 740 MiB used, 1.0 TiB / 1.0 TiB avail; 126 KiB/s rd, 1.1 MiB/s wr, 207 op/s 2024-03-24T00:21:26.721 INFO:journalctl@ceph.mds.l.smithi193.stdout:Mar 24 00:21:26 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-l[71708]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: In function 'void Locker::set_xlocks_done(MutationImpl*, bool)' thread 7f2fb4d34700 time 2024-03-24T00:21:26.449378+0000 2024-03-24T00:21:26.721 INFO:journalctl@ceph.mds.l.smithi193.stdout:Mar 24 00:21:26 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-l[71708]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: 626: FAILED ceph_assert(obj->is_auth()) 2024-03-24T00:21:26.721 INFO:journalctl@ceph.mds.l.smithi193.stdout:Mar 24 00:21:26 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-l[71708]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-24T00:21:26.721 INFO:journalctl@ceph.mds.l.smithi193.stdout:Mar 24 00:21:26 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-l[71708]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x135) [0x7f2fbf35e4a1] 2024-03-24T00:21:26.721 INFO:journalctl@ceph.mds.l.smithi193.stdout:Mar 24 00:21:26 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-l[71708]: 2: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7f2fbf35e667] 2024-03-24T00:21:26.721 INFO:journalctl@ceph.mds.l.smithi193.stdout:Mar 24 00:21:26 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-l[71708]: 3: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x563c5b892223] 2024-03-24T00:21:26.721 INFO:journalctl@ceph.mds.l.smithi193.stdout:Mar 24 00:21:26 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-l[71708]: 4: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x563c5b70c5d6] 2024-03-24T00:21:26.721 INFO:journalctl@ceph.mds.l.smithi193.stdout:Mar 24 00:21:26 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-l[71708]: 5: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x563c5b70cda1] 2024-03-24T00:21:26.721 INFO:journalctl@ceph.mds.l.smithi193.stdout:Mar 24 00:21:26 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-l[71708]: 6: (Server::handle_client_rename(boost::intrusive_ptr const&)+0x259d) [0x563c5b74997d] 2024-03-24T00:21:26.721 INFO:journalctl@ceph.mds.l.smithi193.stdout:Mar 24 00:21:26 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-l[71708]: 7: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x563c5b75f233] 2024-03-24T00:21:26.722 INFO:journalctl@ceph.mds.l.smithi193.stdout:Mar 24 00:21:26 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-l[71708]: 8: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x563c5b761197] 2024-03-24T00:21:26.722 INFO:journalctl@ceph.mds.l.smithi193.stdout:Mar 24 00:21:26 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-l[71708]: 9: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x563c5b66011b] 2024-03-24T00:21:26.722 INFO:journalctl@ceph.mds.l.smithi193.stdout:Mar 24 00:21:26 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-l[71708]: 10: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x563c5b6606bc] 2024-03-24T00:21:26.722 INFO:journalctl@ceph.mds.l.smithi193.stdout:Mar 24 00:21:26 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-l[71708]: 11: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x563c5b64a1ff] 2024-03-24T00:21:26.722 INFO:journalctl@ceph.mds.l.smithi193.stdout:Mar 24 00:21:26 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-l[71708]: 12: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7f2fbf5f0488] 2024-03-24T00:21:26.722 INFO:journalctl@ceph.mds.l.smithi193.stdout:Mar 24 00:21:26 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-l[71708]: 13: (DispatchQueue::entry()+0x51f) [0x7f2fbf5ed60f] 2024-03-24T00:21:26.722 INFO:journalctl@ceph.mds.l.smithi193.stdout:Mar 24 00:21:26 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-l[71708]: 14: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f2fbf6c0e81] 2024-03-24T00:21:26.722 INFO:journalctl@ceph.mds.l.smithi193.stdout:Mar 24 00:21:26 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-l[71708]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7f2fbe0db1ca] 2024-03-24T00:21:26.722 INFO:journalctl@ceph.mds.l.smithi193.stdout:Mar 24 00:21:26 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-l[71708]: 16: clone() 2024-03-24T00:21:26.722 INFO:journalctl@ceph.mds.l.smithi193.stdout:Mar 24 00:21:26 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-l[71708]: 2024-03-24T00:21:26.450+0000 7f2fb4d34700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: In function 'void Locker::set_xlocks_done(MutationImpl*, bool)' thread 7f2fb4d34700 time 2024-03-24T00:21:26.449378+0000 2024-03-24T00:21:26.722 INFO:journalctl@ceph.mds.l.smithi193.stdout:Mar 24 00:21:26 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-l[71708]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: 626: FAILED ceph_assert(obj->is_auth()) 2024-03-24T00:21:26.722 INFO:journalctl@ceph.mds.l.smithi193.stdout:Mar 24 00:21:26 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-l[71708]: 2024-03-24T00:21:26.722 INFO:journalctl@ceph.mds.l.smithi193.stdout:Mar 24 00:21:26 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-l[71708]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-24T00:21:26.723 INFO:journalctl@ceph.mds.l.smithi193.stdout:Mar 24 00:21:26 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-l[71708]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x135) [0x7f2fbf35e4a1] 2024-03-24T00:21:26.723 INFO:journalctl@ceph.mds.l.smithi193.stdout:Mar 24 00:21:26 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-l[71708]: 2: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7f2fbf35e667] 2024-03-24T00:21:26.723 INFO:journalctl@ceph.mds.l.smithi193.stdout:Mar 24 00:21:26 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-l[71708]: 3: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x563c5b892223] 2024-03-24T00:21:26.723 INFO:journalctl@ceph.mds.l.smithi193.stdout:Mar 24 00:21:26 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-l[71708]: 4: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x563c5b70c5d6] 2024-03-24T00:21:26.723 INFO:journalctl@ceph.mds.l.smithi193.stdout:Mar 24 00:21:26 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-l[71708]: 5: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x563c5b70cda1] 2024-03-24T00:21:26.723 INFO:journalctl@ceph.mds.l.smithi193.stdout:Mar 24 00:21:26 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-l[71708]: 6: (Server::handle_client_rename(boost::intrusive_ptr const&)+0x259d) [0x563c5b74997d] 2024-03-24T00:21:26.723 INFO:journalctl@ceph.mds.l.smithi193.stdout:Mar 24 00:21:26 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-l[71708]: 7: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x563c5b75f233] 2024-03-24T00:21:26.723 INFO:journalctl@ceph.mds.l.smithi193.stdout:Mar 24 00:21:26 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-l[71708]: 8: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x563c5b761197] 2024-03-24T00:21:26.723 INFO:journalctl@ceph.mds.l.smithi193.stdout:Mar 24 00:21:26 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-l[71708]: 9: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x563c5b66011b] 2024-03-24T00:21:26.723 INFO:journalctl@ceph.mds.l.smithi193.stdout:Mar 24 00:21:26 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-l[71708]: 10: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x563c5b6606bc] 2024-03-24T00:21:26.723 INFO:journalctl@ceph.mds.l.smithi193.stdout:Mar 24 00:21:26 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-l[71708]: 11: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x563c5b64a1ff] 2024-03-24T00:21:26.723 INFO:journalctl@ceph.mds.l.smithi193.stdout:Mar 24 00:21:26 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-l[71708]: 12: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7f2fbf5f0488] 2024-03-24T00:21:26.723 INFO:journalctl@ceph.mds.l.smithi193.stdout:Mar 24 00:21:26 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-l[71708]: 13: (DispatchQueue::entry()+0x51f) [0x7f2fbf5ed60f] 2024-03-24T00:21:26.723 INFO:journalctl@ceph.mds.l.smithi193.stdout:Mar 24 00:21:26 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-l[71708]: 14: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f2fbf6c0e81] 2024-03-24T00:21:26.723 INFO:journalctl@ceph.mds.l.smithi193.stdout:Mar 24 00:21:26 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-l[71708]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7f2fbe0db1ca] 2024-03-24T00:21:26.724 INFO:journalctl@ceph.mds.l.smithi193.stdout:Mar 24 00:21:26 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-l[71708]: 16: clone() 2024-03-24T00:21:26.724 INFO:journalctl@ceph.mds.l.smithi193.stdout:Mar 24 00:21:26 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-l[71708]: 2024-03-24T00:21:26.724 INFO:journalctl@ceph.mds.l.smithi193.stdout:Mar 24 00:21:26 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-l[71708]: *** Caught signal (Aborted) ** 2024-03-24T00:21:26.724 INFO:journalctl@ceph.mds.l.smithi193.stdout:Mar 24 00:21:26 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-l[71708]: in thread 7f2fb4d34700 thread_name:ms_dispatch 2024-03-24T00:21:26.724 INFO:journalctl@ceph.mds.l.smithi193.stdout:Mar 24 00:21:26 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-l[71708]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-24T00:21:26.724 INFO:journalctl@ceph.mds.l.smithi193.stdout:Mar 24 00:21:26 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-l[71708]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f2fbe0e5d20] 2024-03-24T00:21:26.724 INFO:journalctl@ceph.mds.l.smithi193.stdout:Mar 24 00:21:26 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-l[71708]: 2: gsignal() 2024-03-24T00:21:26.724 INFO:journalctl@ceph.mds.l.smithi193.stdout:Mar 24 00:21:26 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-l[71708]: 3: abort() 2024-03-24T00:21:26.724 INFO:journalctl@ceph.mds.l.smithi193.stdout:Mar 24 00:21:26 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-l[71708]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x18f) [0x7f2fbf35e4fb] 2024-03-24T00:21:26.724 INFO:journalctl@ceph.mds.l.smithi193.stdout:Mar 24 00:21:26 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-l[71708]: 5: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7f2fbf35e667] 2024-03-24T00:21:26.724 INFO:journalctl@ceph.mds.l.smithi193.stdout:Mar 24 00:21:26 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-l[71708]: 6: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x563c5b892223] 2024-03-24T00:21:26.724 INFO:journalctl@ceph.mds.l.smithi193.stdout:Mar 24 00:21:26 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-l[71708]: 7: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x563c5b70c5d6] 2024-03-24T00:21:26.724 INFO:journalctl@ceph.mds.l.smithi193.stdout:Mar 24 00:21:26 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-l[71708]: 8: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x563c5b70cda1] 2024-03-24T00:21:26.725 INFO:journalctl@ceph.mds.l.smithi193.stdout:Mar 24 00:21:26 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-l[71708]: 9: (Server::handle_client_rename(boost::intrusive_ptr const&)+0x259d) [0x563c5b74997d] 2024-03-24T00:21:26.725 INFO:journalctl@ceph.mds.l.smithi193.stdout:Mar 24 00:21:26 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-l[71708]: 10: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x563c5b75f233] 2024-03-24T00:21:26.725 INFO:journalctl@ceph.mds.l.smithi193.stdout:Mar 24 00:21:26 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-l[71708]: 11: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x563c5b761197] 2024-03-24T00:21:26.725 INFO:journalctl@ceph.mds.l.smithi193.stdout:Mar 24 00:21:26 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-l[71708]: 12: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x563c5b66011b] 2024-03-24T00:21:26.725 INFO:journalctl@ceph.mds.l.smithi193.stdout:Mar 24 00:21:26 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-l[71708]: 13: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x563c5b6606bc] 2024-03-24T00:21:26.725 INFO:journalctl@ceph.mds.l.smithi193.stdout:Mar 24 00:21:26 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-l[71708]: 14: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x563c5b64a1ff] 2024-03-24T00:21:26.725 INFO:journalctl@ceph.mds.l.smithi193.stdout:Mar 24 00:21:26 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-l[71708]: 15: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7f2fbf5f0488] 2024-03-24T00:21:26.725 INFO:journalctl@ceph.mds.l.smithi193.stdout:Mar 24 00:21:26 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-l[71708]: 16: (DispatchQueue::entry()+0x51f) [0x7f2fbf5ed60f] 2024-03-24T00:21:26.725 INFO:journalctl@ceph.mds.l.smithi193.stdout:Mar 24 00:21:26 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-l[71708]: 17: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f2fbf6c0e81] 2024-03-24T00:21:26.725 INFO:journalctl@ceph.mds.l.smithi193.stdout:Mar 24 00:21:26 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-l[71708]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7f2fbe0db1ca] 2024-03-24T00:21:26.725 INFO:journalctl@ceph.mds.l.smithi193.stdout:Mar 24 00:21:26 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-l[71708]: 19: clone() 2024-03-24T00:21:26.725 INFO:journalctl@ceph.mds.l.smithi193.stdout:Mar 24 00:21:26 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-l[71708]: 2024-03-24T00:21:26.452+0000 7f2fb4d34700 -1 *** Caught signal (Aborted) ** 2024-03-24T00:21:26.725 INFO:journalctl@ceph.mds.l.smithi193.stdout:Mar 24 00:21:26 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-l[71708]: in thread 7f2fb4d34700 thread_name:ms_dispatch 2024-03-24T00:21:26.725 INFO:journalctl@ceph.mds.l.smithi193.stdout:Mar 24 00:21:26 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-l[71708]: 2024-03-24T00:21:26.725 INFO:journalctl@ceph.mds.l.smithi193.stdout:Mar 24 00:21:26 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-l[71708]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-24T00:21:26.726 INFO:journalctl@ceph.mds.l.smithi193.stdout:Mar 24 00:21:26 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-l[71708]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f2fbe0e5d20] 2024-03-24T00:21:26.726 INFO:journalctl@ceph.mds.l.smithi193.stdout:Mar 24 00:21:26 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-l[71708]: 2: gsignal() 2024-03-24T00:21:26.726 INFO:journalctl@ceph.mds.l.smithi193.stdout:Mar 24 00:21:26 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-l[71708]: 3: abort() 2024-03-24T00:21:26.726 INFO:journalctl@ceph.mds.l.smithi193.stdout:Mar 24 00:21:26 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-l[71708]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x18f) [0x7f2fbf35e4fb] 2024-03-24T00:21:26.726 INFO:journalctl@ceph.mds.l.smithi193.stdout:Mar 24 00:21:26 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-l[71708]: 5: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7f2fbf35e667] 2024-03-24T00:21:26.726 INFO:journalctl@ceph.mds.l.smithi193.stdout:Mar 24 00:21:26 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-l[71708]: 6: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x563c5b892223] 2024-03-24T00:21:26.726 INFO:journalctl@ceph.mds.l.smithi193.stdout:Mar 24 00:21:26 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-l[71708]: 7: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x563c5b70c5d6] 2024-03-24T00:21:26.726 INFO:journalctl@ceph.mds.l.smithi193.stdout:Mar 24 00:21:26 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-l[71708]: 8: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x563c5b70cda1] 2024-03-24T00:21:26.726 INFO:journalctl@ceph.mds.l.smithi193.stdout:Mar 24 00:21:26 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-l[71708]: 9: (Server::handle_client_rename(boost::intrusive_ptr const&)+0x259d) [0x563c5b74997d] 2024-03-24T00:21:26.726 INFO:journalctl@ceph.mds.l.smithi193.stdout:Mar 24 00:21:26 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-l[71708]: 10: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x563c5b75f233] 2024-03-24T00:21:26.726 INFO:journalctl@ceph.mds.l.smithi193.stdout:Mar 24 00:21:26 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-l[71708]: 11: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x563c5b761197] 2024-03-24T00:21:26.726 INFO:journalctl@ceph.mds.l.smithi193.stdout:Mar 24 00:21:26 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-l[71708]: 12: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x563c5b66011b] 2024-03-24T00:21:26.726 INFO:journalctl@ceph.mds.l.smithi193.stdout:Mar 24 00:21:26 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-l[71708]: 13: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x563c5b6606bc] 2024-03-24T00:21:26.727 INFO:journalctl@ceph.mds.l.smithi193.stdout:Mar 24 00:21:26 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-l[71708]: 14: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x563c5b64a1ff] 2024-03-24T00:21:26.727 INFO:journalctl@ceph.mds.l.smithi193.stdout:Mar 24 00:21:26 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-l[71708]: 15: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7f2fbf5f0488] 2024-03-24T00:21:26.727 INFO:journalctl@ceph.mds.l.smithi193.stdout:Mar 24 00:21:26 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-l[71708]: 16: (DispatchQueue::entry()+0x51f) [0x7f2fbf5ed60f] 2024-03-24T00:21:26.727 INFO:journalctl@ceph.mds.l.smithi193.stdout:Mar 24 00:21:26 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-l[71708]: 17: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f2fbf6c0e81] 2024-03-24T00:21:26.727 INFO:journalctl@ceph.mds.l.smithi193.stdout:Mar 24 00:21:26 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-l[71708]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7f2fbe0db1ca] 2024-03-24T00:21:26.727 INFO:journalctl@ceph.mds.l.smithi193.stdout:Mar 24 00:21:26 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-l[71708]: 19: clone() 2024-03-24T00:21:26.727 INFO:journalctl@ceph.mds.l.smithi193.stdout:Mar 24 00:21:26 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-l[71708]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-03-24T00:21:26.727 INFO:journalctl@ceph.mds.l.smithi193.stdout:Mar 24 00:21:26 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-l[71708]: 2024-03-24T00:21:26.727 INFO:journalctl@ceph.mds.l.smithi193.stdout:Mar 24 00:21:26 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-l[71708]: -1> 2024-03-24T00:21:26.450+0000 7f2fb4d34700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: In function 'void Locker::set_xlocks_done(MutationImpl*, bool)' thread 7f2fb4d34700 time 2024-03-24T00:21:26.449378+0000 2024-03-24T00:21:26.727 INFO:journalctl@ceph.mds.l.smithi193.stdout:Mar 24 00:21:26 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-l[71708]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: 626: FAILED ceph_assert(obj->is_auth()) 2024-03-24T00:21:26.727 INFO:journalctl@ceph.mds.l.smithi193.stdout:Mar 24 00:21:26 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-l[71708]: 2024-03-24T00:21:26.727 INFO:journalctl@ceph.mds.l.smithi193.stdout:Mar 24 00:21:26 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-l[71708]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-24T00:21:26.727 INFO:journalctl@ceph.mds.l.smithi193.stdout:Mar 24 00:21:26 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-l[71708]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x135) [0x7f2fbf35e4a1] 2024-03-24T00:21:26.727 INFO:journalctl@ceph.mds.l.smithi193.stdout:Mar 24 00:21:26 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-l[71708]: 2: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7f2fbf35e667] 2024-03-24T00:21:26.728 INFO:journalctl@ceph.mds.l.smithi193.stdout:Mar 24 00:21:26 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-l[71708]: 3: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x563c5b892223] 2024-03-24T00:21:26.728 INFO:journalctl@ceph.mds.l.smithi193.stdout:Mar 24 00:21:26 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-l[71708]: 4: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x563c5b70c5d6] 2024-03-24T00:21:26.728 INFO:journalctl@ceph.mds.l.smithi193.stdout:Mar 24 00:21:26 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-l[71708]: 5: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x563c5b70cda1] 2024-03-24T00:21:26.728 INFO:journalctl@ceph.mds.l.smithi193.stdout:Mar 24 00:21:26 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-l[71708]: 6: (Server::handle_client_rename(boost::intrusive_ptr const&)+0x259d) [0x563c5b74997d] 2024-03-24T00:21:26.728 INFO:journalctl@ceph.mds.l.smithi193.stdout:Mar 24 00:21:26 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-l[71708]: 7: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x563c5b75f233] 2024-03-24T00:21:26.728 INFO:journalctl@ceph.mds.l.smithi193.stdout:Mar 24 00:21:26 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-l[71708]: 8: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x563c5b761197] 2024-03-24T00:21:26.728 INFO:journalctl@ceph.mds.l.smithi193.stdout:Mar 24 00:21:26 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-l[71708]: 9: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x563c5b66011b] 2024-03-24T00:21:26.728 INFO:journalctl@ceph.mds.l.smithi193.stdout:Mar 24 00:21:26 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-l[71708]: 10: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x563c5b6606bc] 2024-03-24T00:21:26.728 INFO:journalctl@ceph.mds.l.smithi193.stdout:Mar 24 00:21:26 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-l[71708]: 11: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x563c5b64a1ff] 2024-03-24T00:21:26.728 INFO:journalctl@ceph.mds.l.smithi193.stdout:Mar 24 00:21:26 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-l[71708]: 12: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7f2fbf5f0488] 2024-03-24T00:21:26.728 INFO:journalctl@ceph.mds.l.smithi193.stdout:Mar 24 00:21:26 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-l[71708]: 13: (DispatchQueue::entry()+0x51f) [0x7f2fbf5ed60f] 2024-03-24T00:21:26.728 INFO:journalctl@ceph.mds.l.smithi193.stdout:Mar 24 00:21:26 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-l[71708]: 14: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f2fbf6c0e81] 2024-03-24T00:21:26.728 INFO:journalctl@ceph.mds.l.smithi193.stdout:Mar 24 00:21:26 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-l[71708]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7f2fbe0db1ca] 2024-03-24T00:21:26.728 INFO:journalctl@ceph.mds.l.smithi193.stdout:Mar 24 00:21:26 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-l[71708]: 16: clone() 2024-03-24T00:21:26.729 INFO:journalctl@ceph.mds.l.smithi193.stdout:Mar 24 00:21:26 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-l[71708]: 2024-03-24T00:21:26.729 INFO:journalctl@ceph.mds.l.smithi193.stdout:Mar 24 00:21:26 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-l[71708]: 0> 2024-03-24T00:21:26.452+0000 7f2fb4d34700 -1 *** Caught signal (Aborted) ** 2024-03-24T00:21:26.729 INFO:journalctl@ceph.mds.l.smithi193.stdout:Mar 24 00:21:26 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-l[71708]: in thread 7f2fb4d34700 thread_name:ms_dispatch 2024-03-24T00:21:26.729 INFO:journalctl@ceph.mds.l.smithi193.stdout:Mar 24 00:21:26 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-l[71708]: 2024-03-24T00:21:26.729 INFO:journalctl@ceph.mds.l.smithi193.stdout:Mar 24 00:21:26 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-l[71708]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-24T00:21:26.729 INFO:journalctl@ceph.mds.l.smithi193.stdout:Mar 24 00:21:26 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-l[71708]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f2fbe0e5d20] 2024-03-24T00:21:26.729 INFO:journalctl@ceph.mds.l.smithi193.stdout:Mar 24 00:21:26 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-l[71708]: 2: gsignal() 2024-03-24T00:21:26.729 INFO:journalctl@ceph.mds.l.smithi193.stdout:Mar 24 00:21:26 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-l[71708]: 3: abort() 2024-03-24T00:21:26.729 INFO:journalctl@ceph.mds.l.smithi193.stdout:Mar 24 00:21:26 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-l[71708]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x18f) [0x7f2fbf35e4fb] 2024-03-24T00:21:26.729 INFO:journalctl@ceph.mds.l.smithi193.stdout:Mar 24 00:21:26 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-l[71708]: 5: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7f2fbf35e667] 2024-03-24T00:21:26.729 INFO:journalctl@ceph.mds.l.smithi193.stdout:Mar 24 00:21:26 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-l[71708]: 6: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x563c5b892223] 2024-03-24T00:21:26.729 INFO:journalctl@ceph.mds.l.smithi193.stdout:Mar 24 00:21:26 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-l[71708]: 7: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x563c5b70c5d6] 2024-03-24T00:21:26.729 INFO:journalctl@ceph.mds.l.smithi193.stdout:Mar 24 00:21:26 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-l[71708]: 8: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x563c5b70cda1] 2024-03-24T00:21:26.729 INFO:journalctl@ceph.mds.l.smithi193.stdout:Mar 24 00:21:26 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-l[71708]: 9: (Server::handle_client_rename(boost::intrusive_ptr const&)+0x259d) [0x563c5b74997d] 2024-03-24T00:21:26.729 INFO:journalctl@ceph.mds.l.smithi193.stdout:Mar 24 00:21:26 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-l[71708]: 10: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x563c5b75f233] 2024-03-24T00:21:26.730 INFO:journalctl@ceph.mds.l.smithi193.stdout:Mar 24 00:21:26 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-l[71708]: 11: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x563c5b761197] 2024-03-24T00:21:26.730 INFO:journalctl@ceph.mds.l.smithi193.stdout:Mar 24 00:21:26 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-l[71708]: 12: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x563c5b66011b] 2024-03-24T00:21:26.730 INFO:journalctl@ceph.mds.l.smithi193.stdout:Mar 24 00:21:26 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-l[71708]: 13: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x563c5b6606bc] 2024-03-24T00:21:26.730 INFO:journalctl@ceph.mds.l.smithi193.stdout:Mar 24 00:21:26 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-l[71708]: 14: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x563c5b64a1ff] 2024-03-24T00:21:26.730 INFO:journalctl@ceph.mds.l.smithi193.stdout:Mar 24 00:21:26 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-l[71708]: 15: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7f2fbf5f0488] 2024-03-24T00:21:26.730 INFO:journalctl@ceph.mds.l.smithi193.stdout:Mar 24 00:21:26 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-l[71708]: 16: (DispatchQueue::entry()+0x51f) [0x7f2fbf5ed60f] 2024-03-24T00:21:26.730 INFO:journalctl@ceph.mds.l.smithi193.stdout:Mar 24 00:21:26 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-l[71708]: 17: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f2fbf6c0e81] 2024-03-24T00:21:26.730 INFO:journalctl@ceph.mds.l.smithi193.stdout:Mar 24 00:21:26 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-l[71708]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7f2fbe0db1ca] 2024-03-24T00:21:26.730 INFO:journalctl@ceph.mds.l.smithi193.stdout:Mar 24 00:21:26 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-l[71708]: 19: clone() 2024-03-24T00:21:26.730 INFO:journalctl@ceph.mds.l.smithi193.stdout:Mar 24 00:21:26 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-l[71708]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-03-24T00:21:26.730 INFO:journalctl@ceph.mds.l.smithi193.stdout:Mar 24 00:21:26 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-l[71708]: 2024-03-24T00:21:26.730 INFO:journalctl@ceph.mds.l.smithi193.stdout:Mar 24 00:21:26 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-l[71708]: -1> 2024-03-24T00:21:26.450+0000 7f2fb4d34700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: In function 'void Locker::set_xlocks_done(MutationImpl*, bool)' thread 7f2fb4d34700 time 2024-03-24T00:21:26.449378+0000 2024-03-24T00:21:26.730 INFO:journalctl@ceph.mds.l.smithi193.stdout:Mar 24 00:21:26 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-l[71708]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: 626: FAILED ceph_assert(obj->is_auth()) 2024-03-24T00:21:26.730 INFO:journalctl@ceph.mds.l.smithi193.stdout:Mar 24 00:21:26 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-l[71708]: 2024-03-24T00:21:26.731 INFO:journalctl@ceph.mds.l.smithi193.stdout:Mar 24 00:21:26 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-l[71708]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-24T00:21:26.731 INFO:journalctl@ceph.mds.l.smithi193.stdout:Mar 24 00:21:26 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-l[71708]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x135) [0x7f2fbf35e4a1] 2024-03-24T00:21:26.731 INFO:journalctl@ceph.mds.l.smithi193.stdout:Mar 24 00:21:26 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-l[71708]: 2: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7f2fbf35e667] 2024-03-24T00:21:26.731 INFO:journalctl@ceph.mds.l.smithi193.stdout:Mar 24 00:21:26 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-l[71708]: 3: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x563c5b892223] 2024-03-24T00:21:26.731 INFO:journalctl@ceph.mds.l.smithi193.stdout:Mar 24 00:21:26 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-l[71708]: 4: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x563c5b70c5d6] 2024-03-24T00:21:26.731 INFO:journalctl@ceph.mds.l.smithi193.stdout:Mar 24 00:21:26 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-l[71708]: 5: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x563c5b70cda1] 2024-03-24T00:21:26.731 INFO:journalctl@ceph.mds.l.smithi193.stdout:Mar 24 00:21:26 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-l[71708]: 6: (Server::handle_client_rename(boost::intrusive_ptr const&)+0x259d) [0x563c5b74997d] 2024-03-24T00:21:26.731 INFO:journalctl@ceph.mds.l.smithi193.stdout:Mar 24 00:21:26 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-l[71708]: 7: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x563c5b75f233] 2024-03-24T00:21:26.731 INFO:journalctl@ceph.mds.l.smithi193.stdout:Mar 24 00:21:26 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-l[71708]: 8: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x563c5b761197] 2024-03-24T00:21:26.731 INFO:journalctl@ceph.mds.l.smithi193.stdout:Mar 24 00:21:26 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-l[71708]: 9: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x563c5b66011b] 2024-03-24T00:21:26.731 INFO:journalctl@ceph.mds.l.smithi193.stdout:Mar 24 00:21:26 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-l[71708]: 10: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x563c5b6606bc] 2024-03-24T00:21:26.731 INFO:journalctl@ceph.mds.l.smithi193.stdout:Mar 24 00:21:26 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-l[71708]: 11: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x563c5b64a1ff] 2024-03-24T00:21:26.731 INFO:journalctl@ceph.mds.l.smithi193.stdout:Mar 24 00:21:26 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-l[71708]: 12: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7f2fbf5f0488] 2024-03-24T00:21:26.731 INFO:journalctl@ceph.mds.l.smithi193.stdout:Mar 24 00:21:26 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-l[71708]: 13: (DispatchQueue::entry()+0x51f) [0x7f2fbf5ed60f] 2024-03-24T00:21:26.732 INFO:journalctl@ceph.mds.l.smithi193.stdout:Mar 24 00:21:26 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-l[71708]: 14: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f2fbf6c0e81] 2024-03-24T00:21:26.732 INFO:journalctl@ceph.mds.l.smithi193.stdout:Mar 24 00:21:26 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-l[71708]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7f2fbe0db1ca] 2024-03-24T00:21:26.732 INFO:journalctl@ceph.mds.l.smithi193.stdout:Mar 24 00:21:26 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-l[71708]: 16: clone() 2024-03-24T00:21:26.732 INFO:journalctl@ceph.mds.l.smithi193.stdout:Mar 24 00:21:26 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-l[71708]: 2024-03-24T00:21:26.732 INFO:journalctl@ceph.mds.l.smithi193.stdout:Mar 24 00:21:26 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-l[71708]: 0> 2024-03-24T00:21:26.452+0000 7f2fb4d34700 -1 *** Caught signal (Aborted) ** 2024-03-24T00:21:26.732 INFO:journalctl@ceph.mds.l.smithi193.stdout:Mar 24 00:21:26 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-l[71708]: in thread 7f2fb4d34700 thread_name:ms_dispatch 2024-03-24T00:21:26.732 INFO:journalctl@ceph.mds.l.smithi193.stdout:Mar 24 00:21:26 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-l[71708]: 2024-03-24T00:21:26.732 INFO:journalctl@ceph.mds.l.smithi193.stdout:Mar 24 00:21:26 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-l[71708]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-24T00:21:26.732 INFO:journalctl@ceph.mds.l.smithi193.stdout:Mar 24 00:21:26 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-l[71708]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f2fbe0e5d20] 2024-03-24T00:21:26.732 INFO:journalctl@ceph.mds.l.smithi193.stdout:Mar 24 00:21:26 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-l[71708]: 2: gsignal() 2024-03-24T00:21:26.732 INFO:journalctl@ceph.mds.l.smithi193.stdout:Mar 24 00:21:26 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-l[71708]: 3: abort() 2024-03-24T00:21:26.732 INFO:journalctl@ceph.mds.l.smithi193.stdout:Mar 24 00:21:26 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-l[71708]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x18f) [0x7f2fbf35e4fb] 2024-03-24T00:21:26.732 INFO:journalctl@ceph.mds.l.smithi193.stdout:Mar 24 00:21:26 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-l[71708]: 5: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7f2fbf35e667] 2024-03-24T00:21:26.732 INFO:journalctl@ceph.mds.l.smithi193.stdout:Mar 24 00:21:26 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-l[71708]: 6: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x563c5b892223] 2024-03-24T00:21:26.733 INFO:journalctl@ceph.mds.l.smithi193.stdout:Mar 24 00:21:26 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-l[71708]: 7: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x563c5b70c5d6] 2024-03-24T00:21:26.733 INFO:journalctl@ceph.mds.l.smithi193.stdout:Mar 24 00:21:26 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-l[71708]: 8: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x563c5b70cda1] 2024-03-24T00:21:26.733 INFO:journalctl@ceph.mds.l.smithi193.stdout:Mar 24 00:21:26 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-l[71708]: 9: (Server::handle_client_rename(boost::intrusive_ptr const&)+0x259d) [0x563c5b74997d] 2024-03-24T00:21:26.733 INFO:journalctl@ceph.mds.l.smithi193.stdout:Mar 24 00:21:26 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-l[71708]: 10: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x563c5b75f233] 2024-03-24T00:21:26.733 INFO:journalctl@ceph.mds.l.smithi193.stdout:Mar 24 00:21:26 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-l[71708]: 11: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x563c5b761197] 2024-03-24T00:21:26.733 INFO:journalctl@ceph.mds.l.smithi193.stdout:Mar 24 00:21:26 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-l[71708]: 12: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x563c5b66011b] 2024-03-24T00:21:26.733 INFO:journalctl@ceph.mds.l.smithi193.stdout:Mar 24 00:21:26 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-l[71708]: 13: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x563c5b6606bc] 2024-03-24T00:21:26.733 INFO:journalctl@ceph.mds.l.smithi193.stdout:Mar 24 00:21:26 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-l[71708]: 14: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x563c5b64a1ff] 2024-03-24T00:21:26.733 INFO:journalctl@ceph.mds.l.smithi193.stdout:Mar 24 00:21:26 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-l[71708]: 15: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7f2fbf5f0488] 2024-03-24T00:21:26.733 INFO:journalctl@ceph.mds.l.smithi193.stdout:Mar 24 00:21:26 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-l[71708]: 16: (DispatchQueue::entry()+0x51f) [0x7f2fbf5ed60f] 2024-03-24T00:21:26.733 INFO:journalctl@ceph.mds.l.smithi193.stdout:Mar 24 00:21:26 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-l[71708]: 17: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f2fbf6c0e81] 2024-03-24T00:21:26.733 INFO:journalctl@ceph.mds.l.smithi193.stdout:Mar 24 00:21:26 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-l[71708]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7f2fbe0db1ca] 2024-03-24T00:21:26.733 INFO:journalctl@ceph.mds.l.smithi193.stdout:Mar 24 00:21:26 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-l[71708]: 19: clone() 2024-03-24T00:21:26.733 INFO:journalctl@ceph.mds.l.smithi193.stdout:Mar 24 00:21:26 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-l[71708]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-03-24T00:21:26.734 INFO:journalctl@ceph.mds.l.smithi193.stdout:Mar 24 00:21:26 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-l[71708]: 2024-03-24T00:21:26.734 INFO:journalctl@ceph.mds.l.smithi193.stdout:Mar 24 00:21:26 smithi193 podman[71909]: 2024-03-24 00:21:26.518163208 +0000 UTC m=+0.024030866 container died 2dd1e6ecd73ce6600b5bd2ff530b6697f813ec5675c7b76ac4db2c015e81cd35 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-l, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, ceph=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.build-date=20240319, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-24T00:21:26.844 INFO:tasks.workunit.client.0.smithi118.stdout: 43 524 0 522 0 402 27 2024-03-24T00:21:27.454 INFO:journalctl@ceph.mds.h.smithi189.stdout:Mar 24 00:21:27 smithi189 podman[69704]: 2024-03-24 00:21:27.182888338 +0000 UTC m=+0.741618871 container remove 30d8025ae60b5197cceebadd800d3c1406438009abbf9ed61785e4b539e8412f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-h, org.label-schema.build-date=20240319, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, org.label-schema.license=GPLv2, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, ceph=True, GIT_CLEAN=True) 2024-03-24T00:21:27.454 INFO:journalctl@ceph.mds.h.smithi189.stdout:Mar 24 00:21:27 smithi189 systemd[1]: ceph-d79a5420-e971-11ee-95cd-87774f69a715@mds.h.service: Main process exited, code=exited, status=134/n/a 2024-03-24T00:21:27.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:21:27 smithi189 ceph-mon[36187]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-03-24T00:21:27.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:21:27 smithi189 ceph-mon[36187]: Cluster is now healthy 2024-03-24T00:21:27.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:21:27 smithi189 ceph-mon[36187]: mds.? [v2:172.21.15.193:6832/4276594827,v1:172.21.15.193:6833/4276594827] up:active 2024-03-24T00:21:27.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:21:27 smithi189 ceph-mon[36187]: mds.? [v2:172.21.15.189:6836/596160018,v1:172.21.15.189:6837/596160018] up:active 2024-03-24T00:21:27.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:21:27 smithi189 ceph-mon[36187]: fsmap cephfs:5 3 up:standby-replay 4 up:standby 5 up:active 2024-03-24T00:21:27.455 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:21:27 smithi189 ceph-mon[36187]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-03-24T00:21:27.477 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:21:27 smithi193 ceph-mon[36878]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-03-24T00:21:27.477 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:21:27 smithi193 ceph-mon[36878]: Cluster is now healthy 2024-03-24T00:21:27.477 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:21:27 smithi193 ceph-mon[36878]: mds.? [v2:172.21.15.193:6832/4276594827,v1:172.21.15.193:6833/4276594827] up:active 2024-03-24T00:21:27.477 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:21:27 smithi193 ceph-mon[36878]: mds.? [v2:172.21.15.189:6836/596160018,v1:172.21.15.189:6837/596160018] up:active 2024-03-24T00:21:27.478 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:21:27 smithi193 ceph-mon[36878]: fsmap cephfs:5 3 up:standby-replay 4 up:standby 5 up:active 2024-03-24T00:21:27.478 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:21:27 smithi193 ceph-mon[36878]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-03-24T00:21:27.478 INFO:journalctl@ceph.mds.l.smithi193.stdout:Mar 24 00:21:27 smithi193 podman[71909]: 2024-03-24 00:21:27.220831251 +0000 UTC m=+0.726698904 container remove 2dd1e6ecd73ce6600b5bd2ff530b6697f813ec5675c7b76ac4db2c015e81cd35 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-l, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.license=GPLv2, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240319, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c) 2024-03-24T00:21:27.478 INFO:journalctl@ceph.mds.l.smithi193.stdout:Mar 24 00:21:27 smithi193 systemd[1]: ceph-d79a5420-e971-11ee-95cd-87774f69a715@mds.l.service: Main process exited, code=exited, status=134/n/a 2024-03-24T00:21:27.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:21:27 smithi118 ceph-mon[31535]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-03-24T00:21:27.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:21:27 smithi118 ceph-mon[31535]: Cluster is now healthy 2024-03-24T00:21:27.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:21:27 smithi118 ceph-mon[31535]: mds.? [v2:172.21.15.193:6832/4276594827,v1:172.21.15.193:6833/4276594827] up:active 2024-03-24T00:21:27.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:21:27 smithi118 ceph-mon[31535]: mds.? [v2:172.21.15.189:6836/596160018,v1:172.21.15.189:6837/596160018] up:active 2024-03-24T00:21:27.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:21:27 smithi118 ceph-mon[31535]: fsmap cephfs:5 3 up:standby-replay 4 up:standby 5 up:active 2024-03-24T00:21:27.697 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:21:27 smithi118 ceph-mon[31535]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-03-24T00:21:27.954 INFO:journalctl@ceph.mds.h.smithi189.stdout:Mar 24 00:21:27 smithi189 systemd[1]: ceph-d79a5420-e971-11ee-95cd-87774f69a715@mds.h.service: Failed with result 'exit-code'. 2024-03-24T00:21:27.960 INFO:journalctl@ceph.mds.l.smithi193.stdout:Mar 24 00:21:27 smithi193 systemd[1]: ceph-d79a5420-e971-11ee-95cd-87774f69a715@mds.l.service: Failed with result 'exit-code'. 2024-03-24T00:21:28.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:21:28 smithi118 ceph-mon[31535]: pgmap v581: 97 pgs: 97 active+clean; 139 MiB data, 745 MiB used, 1.0 TiB / 1.0 TiB avail; 640 KiB/s rd, 1.0 MiB/s wr, 256 op/s 2024-03-24T00:21:28.703 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:21:28 smithi189 ceph-mon[36187]: pgmap v581: 97 pgs: 97 active+clean; 139 MiB data, 745 MiB used, 1.0 TiB / 1.0 TiB avail; 640 KiB/s rd, 1.0 MiB/s wr, 256 op/s 2024-03-24T00:21:28.710 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:21:28 smithi193 ceph-mon[36878]: pgmap v581: 97 pgs: 97 active+clean; 139 MiB data, 745 MiB used, 1.0 TiB / 1.0 TiB avail; 640 KiB/s rd, 1.0 MiB/s wr, 256 op/s 2024-03-24T00:21:30.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:21:29 smithi118 ceph-mon[31535]: pgmap v582: 97 pgs: 97 active+clean; 135 MiB data, 741 MiB used, 1.0 TiB / 1.0 TiB avail; 852 KiB/s rd, 834 KiB/s wr, 257 op/s 2024-03-24T00:21:30.203 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:21:29 smithi189 ceph-mon[36187]: pgmap v582: 97 pgs: 97 active+clean; 135 MiB data, 741 MiB used, 1.0 TiB / 1.0 TiB avail; 852 KiB/s rd, 834 KiB/s wr, 257 op/s 2024-03-24T00:21:30.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:21:29 smithi193 ceph-mon[36878]: pgmap v582: 97 pgs: 97 active+clean; 135 MiB data, 741 MiB used, 1.0 TiB / 1.0 TiB avail; 852 KiB/s rd, 834 KiB/s wr, 257 op/s 2024-03-24T00:21:32.091 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:21:32.091+0000 7fd627e00640 1 -- 172.21.15.118:0/736842230 >> [v2:172.21.15.193:6832/1023428412,v1:172.21.15.193:6833/1023428412] conn(0x55c55390a6a0 msgr2=0x55c55390cac0 unknown :-1 s=STATE_CONNECTING_RE l=0).process reconnect failed to v2:172.21.15.193:6832/1023428412 2024-03-24T00:21:32.093 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:21:32.092+0000 7fd627e00640 1 --2- 172.21.15.118:0/736842230 >> [v2:172.21.15.193:6832/1023428412,v1:172.21.15.193:6833/1023428412] conn(0x55c55390a6a0 0x55c55390cac0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-03-24T00:21:32.093 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:21:32.092+0000 7fd613e00640 1 client.26594 ms_handle_refused on v2:172.21.15.193:6832/1023428412 2024-03-24T00:21:32.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:21:31 smithi118 ceph-mon[31535]: pgmap v583: 97 pgs: 97 active+clean; 135 MiB data, 739 MiB used, 1.0 TiB / 1.0 TiB avail; 830 KiB/s rd, 653 KiB/s wr, 216 op/s 2024-03-24T00:21:32.203 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:21:31 smithi189 ceph-mon[36187]: pgmap v583: 97 pgs: 97 active+clean; 135 MiB data, 739 MiB used, 1.0 TiB / 1.0 TiB avail; 830 KiB/s rd, 653 KiB/s wr, 216 op/s 2024-03-24T00:21:32.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:21:31 smithi193 ceph-mon[36878]: pgmap v583: 97 pgs: 97 active+clean; 135 MiB data, 739 MiB used, 1.0 TiB / 1.0 TiB avail; 830 KiB/s rd, 653 KiB/s wr, 216 op/s 2024-03-24T00:21:34.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:21:33 smithi118 ceph-mon[31535]: pgmap v584: 97 pgs: 97 active+clean; 135 MiB data, 775 MiB used, 1.0 TiB / 1.0 TiB avail; 714 KiB/s rd, 548 KiB/s wr, 183 op/s 2024-03-24T00:21:34.203 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:21:33 smithi189 ceph-mon[36187]: pgmap v584: 97 pgs: 97 active+clean; 135 MiB data, 775 MiB used, 1.0 TiB / 1.0 TiB avail; 714 KiB/s rd, 548 KiB/s wr, 183 op/s 2024-03-24T00:21:34.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:21:33 smithi193 ceph-mon[36878]: pgmap v584: 97 pgs: 97 active+clean; 135 MiB data, 775 MiB used, 1.0 TiB / 1.0 TiB avail; 714 KiB/s rd, 548 KiB/s wr, 183 op/s 2024-03-24T00:21:36.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:21:35 smithi118 ceph-mon[31535]: pgmap v585: 97 pgs: 97 active+clean; 135 MiB data, 775 MiB used, 1.0 TiB / 1.0 TiB avail; 635 KiB/s rd, 97 KiB/s wr, 111 op/s 2024-03-24T00:21:36.203 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:21:35 smithi189 ceph-mon[36187]: pgmap v585: 97 pgs: 97 active+clean; 135 MiB data, 775 MiB used, 1.0 TiB / 1.0 TiB avail; 635 KiB/s rd, 97 KiB/s wr, 111 op/s 2024-03-24T00:21:36.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:21:35 smithi193 ceph-mon[36878]: pgmap v585: 97 pgs: 97 active+clean; 135 MiB data, 775 MiB used, 1.0 TiB / 1.0 TiB avail; 635 KiB/s rd, 97 KiB/s wr, 111 op/s 2024-03-24T00:21:36.845 INFO:tasks.workunit.client.0.smithi118.stdout: 43 21 0 22 0 0 64 2024-03-24T00:21:37.956 INFO:journalctl@ceph.mds.h.smithi189.stdout:Mar 24 00:21:37 smithi189 systemd[1]: ceph-d79a5420-e971-11ee-95cd-87774f69a715@mds.h.service: Scheduled restart job, restart counter is at 2. 2024-03-24T00:21:37.956 INFO:journalctl@ceph.mds.h.smithi189.stdout:Mar 24 00:21:37 smithi189 systemd[1]: Stopped Ceph mds.h for d79a5420-e971-11ee-95cd-87774f69a715. 2024-03-24T00:21:37.956 INFO:journalctl@ceph.mds.h.smithi189.stdout:Mar 24 00:21:37 smithi189 systemd[1]: Starting Ceph mds.h for d79a5420-e971-11ee-95cd-87774f69a715... 2024-03-24T00:21:37.956 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:21:37 smithi189 ceph-mon[36187]: pgmap v586: 97 pgs: 97 active+clean; 135 MiB data, 775 MiB used, 1.0 TiB / 1.0 TiB avail; 649 KiB/s rd, 119 KiB/s wr, 80 op/s 2024-03-24T00:21:37.971 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:21:37 smithi193 ceph-mon[36878]: pgmap v586: 97 pgs: 97 active+clean; 135 MiB data, 775 MiB used, 1.0 TiB / 1.0 TiB avail; 649 KiB/s rd, 119 KiB/s wr, 80 op/s 2024-03-24T00:21:37.971 INFO:journalctl@ceph.mds.l.smithi193.stdout:Mar 24 00:21:37 smithi193 systemd[1]: ceph-d79a5420-e971-11ee-95cd-87774f69a715@mds.l.service: Scheduled restart job, restart counter is at 4. 2024-03-24T00:21:37.971 INFO:journalctl@ceph.mds.l.smithi193.stdout:Mar 24 00:21:37 smithi193 systemd[1]: Stopped Ceph mds.l for d79a5420-e971-11ee-95cd-87774f69a715. 2024-03-24T00:21:37.971 INFO:journalctl@ceph.mds.l.smithi193.stdout:Mar 24 00:21:37 smithi193 systemd[1]: Starting Ceph mds.l for d79a5420-e971-11ee-95cd-87774f69a715... 2024-03-24T00:21:37.971 INFO:journalctl@ceph.mds.l.smithi193.stdout:Mar 24 00:21:37 smithi193 podman[72006]: 2024-03-24 00:21:37.844204388 +0000 UTC m=+0.018805248 image pull c664afe56bb01db4f9afa9532c20598e201e664f2e9ba2be50db0d52674a26d2 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc 2024-03-24T00:21:38.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:21:37 smithi118 ceph-mon[31535]: pgmap v586: 97 pgs: 97 active+clean; 135 MiB data, 775 MiB used, 1.0 TiB / 1.0 TiB avail; 649 KiB/s rd, 119 KiB/s wr, 80 op/s 2024-03-24T00:21:38.261 INFO:journalctl@ceph.mds.h.smithi189.stdout:Mar 24 00:21:37 smithi189 podman[69798]: 2024-03-24 00:21:37.85968742 +0000 UTC m=+0.021630064 image pull c664afe56bb01db4f9afa9532c20598e201e664f2e9ba2be50db0d52674a26d2 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc 2024-03-24T00:21:38.261 INFO:journalctl@ceph.mds.h.smithi189.stdout:Mar 24 00:21:38 smithi189 podman[69798]: 2024-03-24 00:21:38.008768886 +0000 UTC m=+0.170711510 container create 9706b49f92e8c2e438ca1e4fe4db03e0eee967249543d0df6906c3f11e450b28 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-h, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, org.label-schema.build-date=20240319, io.buildah.version=1.33.5, GIT_CLEAN=True) 2024-03-24T00:21:38.349 INFO:journalctl@ceph.mds.l.smithi193.stdout:Mar 24 00:21:37 smithi193 podman[72006]: 2024-03-24 00:21:37.971957677 +0000 UTC m=+0.146558525 container create 200b5ef45dd84c481ecdc9a61d2012b16dc125794c0fb337e89dec4988ba73bd (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-l, org.label-schema.build-date=20240319, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, GIT_CLEAN=True, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30) 2024-03-24T00:21:38.513 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:21:38.513+0000 7f93afe00640 1 -- 172.21.15.118:0/4038377379 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f93b0108d10 msgr2=0x7f93b0109190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:21:38.513 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:21:38.513+0000 7f93afe00640 1 --2- 172.21.15.118:0/4038377379 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f93b0108d10 0x7f93b0109190 secure :-1 s=READY pgs=429 cs=0 l=1 rev1=1 crypto rx=0x7f9398009900 tx=0x7f939802bc50 comp rx=0 tx=0).stop 2024-03-24T00:21:38.513 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:21:38.513+0000 7f93afe00640 1 -- 172.21.15.118:0/4038377379 shutdown_connections 2024-03-24T00:21:38.514 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:21:38.513+0000 7f93afe00640 1 --2- 172.21.15.118:0/4038377379 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f93b01096d0 0x7f93b010ff60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:21:38.514 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:21:38.513+0000 7f93afe00640 1 --2- 172.21.15.118:0/4038377379 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f93b0108d10 0x7f93b0109190 unknown :-1 s=CLOSED pgs=429 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:21:38.514 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:21:38.513+0000 7f93afe00640 1 --2- 172.21.15.118:0/4038377379 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f93b0107b20 0x7f93b0107f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:21:38.514 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:21:38.513+0000 7f93afe00640 1 -- 172.21.15.118:0/4038377379 >> 172.21.15.118:0/4038377379 conn(0x7f93b0076ae0 msgr2=0x7f93b0078f00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:21:38.514 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:21:38.513+0000 7f93afe00640 1 -- 172.21.15.118:0/4038377379 shutdown_connections 2024-03-24T00:21:38.514 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:21:38.513+0000 7f93afe00640 1 -- 172.21.15.118:0/4038377379 wait complete. 2024-03-24T00:21:38.516 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:21:38.514+0000 7f93afe00640 1 Processor -- start 2024-03-24T00:21:38.516 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:21:38.514+0000 7f93afe00640 1 -- start start 2024-03-24T00:21:38.516 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:21:38.515+0000 7f93afe00640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f93b0107b20 0x7f93b019de10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:21:38.516 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:21:38.515+0000 7f93afe00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f93b0108d10 0x7f93b019e350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:21:38.516 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:21:38.515+0000 7f93afe00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f93b01096d0 0x7f93b01a53c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:21:38.516 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:21:38.515+0000 7f93afe00640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f93b01137f0 con 0x7f93b0107b20 2024-03-24T00:21:38.516 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:21:38.515+0000 7f93afe00640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f93b0113670 con 0x7f93b01096d0 2024-03-24T00:21:38.517 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:21:38.515+0000 7f93afe00640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f93b0113970 con 0x7f93b0108d10 2024-03-24T00:21:38.517 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:21:38.515+0000 7f93a6000640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f93b0108d10 0x7f93b019e350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:21:38.517 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:21:38.515+0000 7f93af400640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f93b01096d0 0x7f93b01a53c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:21:38.517 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:21:38.515+0000 7f93aea00640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f93b0107b20 0x7f93b019de10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:21:38.517 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:21:38.515+0000 7f93af400640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f93b01096d0 0x7f93b01a53c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.118:56332/0 (socket says 172.21.15.118:56332) 2024-03-24T00:21:38.517 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:21:38.515+0000 7f93a6000640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f93b0108d10 0x7f93b019e350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.118:40520/0 (socket says 172.21.15.118:40520) 2024-03-24T00:21:38.517 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:21:38.515+0000 7f93af400640 1 -- 172.21.15.118:0/725018075 learned_addr learned my addr 172.21.15.118:0/725018075 (peer_addr_for_me v2:172.21.15.118:0/0) 2024-03-24T00:21:38.517 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:21:38.515+0000 7f93af400640 1 -- 172.21.15.118:0/725018075 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f93b0108d10 msgr2=0x7f93b019e350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:21:38.517 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:21:38.515+0000 7f93af400640 1 --2- 172.21.15.118:0/725018075 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f93b0108d10 0x7f93b019e350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:21:38.517 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:21:38.516+0000 7f93af400640 1 -- 172.21.15.118:0/725018075 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f93b0107b20 msgr2=0x7f93b019de10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:21:38.517 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:21:38.516+0000 7f93af400640 1 --2- 172.21.15.118:0/725018075 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f93b0107b20 0x7f93b019de10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:21:38.517 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:21:38.516+0000 7f93af400640 1 -- 172.21.15.118:0/725018075 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f93b01a5a30 con 0x7f93b01096d0 2024-03-24T00:21:38.517 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:21:38.516+0000 7f93a6000640 1 --2- 172.21.15.118:0/725018075 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f93b0108d10 0x7f93b019e350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T00:21:38.517 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:21:38.516+0000 7f93af400640 1 --2- 172.21.15.118:0/725018075 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f93b01096d0 0x7f93b01a53c0 secure :-1 s=READY pgs=424 cs=0 l=1 rev1=1 crypto rx=0x7f93a800eb10 tx=0x7f93a800c2d0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:21:38.518 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:21:38.516+0000 7f93b4a00640 1 -- 172.21.15.118:0/725018075 <== mon.2 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f93a8004b20 con 0x7f93b01096d0 2024-03-24T00:21:38.518 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:21:38.516+0000 7f93b4a00640 1 -- 172.21.15.118:0/725018075 <== mon.2 v2:172.21.15.189:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f93a8004cc0 con 0x7f93b01096d0 2024-03-24T00:21:38.518 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:21:38.516+0000 7f93afe00640 1 -- 172.21.15.118:0/725018075 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f93b01a5d20 con 0x7f93b01096d0 2024-03-24T00:21:38.518 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:21:38.516+0000 7f93b4a00640 1 -- 172.21.15.118:0/725018075 <== mon.2 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f93a8010760 con 0x7f93b01096d0 2024-03-24T00:21:38.518 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:21:38.516+0000 7f93afe00640 1 -- 172.21.15.118:0/725018075 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f93b01a61e0 con 0x7f93b01096d0 2024-03-24T00:21:38.519 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:21:38.519+0000 7f93b4a00640 1 -- 172.21.15.118:0/725018075 <== mon.2 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 20) v1 ==== 137291+0+0 (secure 0 0 0) 0x7f93a8005020 con 0x7f93b01096d0 2024-03-24T00:21:38.519 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:21:38.519+0000 7f93b8bb9740 1 Processor -- start 2024-03-24T00:21:38.520 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:21:38.520+0000 7f93b8bb9740 1 -- start start 2024-03-24T00:21:38.521 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:21:38.520+0000 7f93b8bb9740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55bcdfcad3f0 0x55bcdfcad810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:21:38.521 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:21:38.520+0000 7f93b8bb9740 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55bcdfcadde0 0x55bcdfcb28e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:21:38.521 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:21:38.520+0000 7f93b8bb9740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55bcdfcb2f50 0x55bcdfcb53e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:21:38.521 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:21:38.520+0000 7f93b8bb9740 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x55bcdfc703b0 con 0x55bcdfcadde0 2024-03-24T00:21:38.521 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:21:38.520+0000 7f93b8bb9740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x55bcdfc12830 con 0x55bcdfcad3f0 2024-03-24T00:21:38.521 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:21:38.520+0000 7f93b8bb9740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x55bcdfbbb5d0 con 0x55bcdfcb2f50 2024-03-24T00:21:38.521 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:21:38.520+0000 7f93aea00640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55bcdfcadde0 0x55bcdfcb28e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:21:38.521 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:21:38.520+0000 7f93aea00640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55bcdfcadde0 0x55bcdfcb28e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.118:3300/0 says I am v2:172.21.15.118:49888/0 (socket says 172.21.15.118:49888) 2024-03-24T00:21:38.521 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:21:38.520+0000 7f93aea00640 1 -- 172.21.15.118:0/1025249822 learned_addr learned my addr 172.21.15.118:0/1025249822 (peer_addr_for_me v2:172.21.15.118:0/0) 2024-03-24T00:21:38.521 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:21:38.520+0000 7f93a6000640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55bcdfcad3f0 0x55bcdfcad810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:21:38.521 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:21:38.520+0000 7f93a6000640 1 --2- 172.21.15.118:0/1025249822 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55bcdfcb2f50 0x55bcdfcb53e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:21:38.521 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:21:38.521+0000 7f93aea00640 1 -- 172.21.15.118:0/1025249822 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55bcdfcb2f50 msgr2=0x55bcdfcb53e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:21:38.522 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:21:38.521+0000 7f93aea00640 1 --2- 172.21.15.118:0/1025249822 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55bcdfcb2f50 0x55bcdfcb53e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:21:38.522 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:21:38.521+0000 7f93aea00640 1 -- 172.21.15.118:0/1025249822 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55bcdfcad3f0 msgr2=0x55bcdfcad810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:21:38.522 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:21:38.521+0000 7f93aea00640 1 --2- 172.21.15.118:0/1025249822 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55bcdfcad3f0 0x55bcdfcad810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:21:38.522 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:21:38.521+0000 7f93aea00640 1 -- 172.21.15.118:0/1025249822 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55bcdfbd32c0 con 0x55bcdfcadde0 2024-03-24T00:21:38.522 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:21:38.521+0000 7f93a6000640 1 --2- 172.21.15.118:0/1025249822 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55bcdfcad3f0 0x55bcdfcad810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T00:21:38.522 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:21:38.521+0000 7f93aea00640 1 --2- 172.21.15.118:0/1025249822 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55bcdfcadde0 0x55bcdfcb28e0 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7f93a0009ca0 tx=0x7f93a0011810 comp rx=0 tx=0).ready entity=mon.0 client_cookie=525d54b01151e5f5 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:21:38.526 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:21:38.521+0000 7f939fe00640 1 -- 172.21.15.118:0/1025249822 <== mon.0 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f93a0004220 con 0x55bcdfcadde0 2024-03-24T00:21:38.527 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:21:38.521+0000 7f939fe00640 1 -- 172.21.15.118:0/1025249822 <== mon.0 v2:172.21.15.118:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f93a00043c0 con 0x55bcdfcadde0 2024-03-24T00:21:38.527 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:21:38.521+0000 7f939fe00640 1 -- 172.21.15.118:0/1025249822 <== mon.0 v2:172.21.15.118:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f93a0019930 con 0x55bcdfcadde0 2024-03-24T00:21:38.527 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:21:38.521+0000 7f93b4a00640 1 --2- 172.21.15.118:0/725018075 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7f93940a27f0 0x7f93940a4cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:21:38.527 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:21:38.522+0000 7f93b4a00640 1 -- 172.21.15.118:0/725018075 <== mon.2 v2:172.21.15.189:3300/0 5 ==== osd_map(102..102 src has 1..102) v4 ==== 9309+0+0 (secure 0 0 0) 0x7f93a80cf5b0 con 0x7f93b01096d0 2024-03-24T00:21:38.527 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:21:38.522+0000 7f93a6000640 1 --2- 172.21.15.118:0/725018075 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7f93940a27f0 0x7f93940a4cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:21:38.527 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:21:38.522+0000 7f93b8bb9740 1 -- 172.21.15.118:0/1025249822 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55bcdfcadde0 msgr2=0x55bcdfcb28e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:21:38.527 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:21:38.522+0000 7f93b8bb9740 1 --2- 172.21.15.118:0/1025249822 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55bcdfcadde0 0x55bcdfcb28e0 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7f93a0009ca0 tx=0x7f93a0011810 comp rx=0 tx=0).stop 2024-03-24T00:21:38.527 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:21:38.522+0000 7f93b8bb9740 1 -- 172.21.15.118:0/1025249822 shutdown_connections 2024-03-24T00:21:38.527 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:21:38.522+0000 7f93b8bb9740 1 --2- 172.21.15.118:0/1025249822 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55bcdfcb2f50 0x55bcdfcb53e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:21:38.527 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:21:38.522+0000 7f93b8bb9740 1 --2- 172.21.15.118:0/1025249822 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55bcdfcadde0 0x55bcdfcb28e0 unknown :-1 s=CLOSED pgs=215 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:21:38.527 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:21:38.522+0000 7f93b8bb9740 1 --2- 172.21.15.118:0/1025249822 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55bcdfcad3f0 0x55bcdfcad810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:21:38.527 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:21:38.522+0000 7f93b8bb9740 1 -- 172.21.15.118:0/1025249822 >> 172.21.15.118:0/1025249822 conn(0x55bcdfca8960 msgr2=0x55bcdfca8d60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:21:38.528 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:21:38.522+0000 7f93b8bb9740 1 -- 172.21.15.118:0/1025249822 shutdown_connections 2024-03-24T00:21:38.528 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:21:38.522+0000 7f93b8bb9740 1 -- 172.21.15.118:0/1025249822 wait complete. 2024-03-24T00:21:38.528 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:21:38.522+0000 7f93a6000640 1 --2- 172.21.15.118:0/725018075 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7f93940a27f0 0x7f93940a4cb0 secure :-1 s=READY pgs=369 cs=0 l=1 rev1=1 crypto rx=0x7f939802c260 tx=0x7f9398005b70 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:21:38.528 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:21:38.523+0000 7f93b8bb9740 1 -- start start 2024-03-24T00:21:38.528 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:21:38.523+0000 7f93b8bb9740 1 Processor -- start 2024-03-24T00:21:38.528 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:21:38.523+0000 7f93b8bb9740 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55bcdfcad3f0 0x55bcdfcb7520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:21:38.528 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:21:38.523+0000 7f93b8bb9740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55bcdfcadde0 0x55bcdfcb7a60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:21:38.528 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:21:38.523+0000 7f93b8bb9740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55bcdfcb2f50 0x55bcdfcbeaa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:21:38.528 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:21:38.523+0000 7f93b8bb9740 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x55bcdfbcea20 con 0x55bcdfcad3f0 2024-03-24T00:21:38.528 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:21:38.523+0000 7f93b8bb9740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x55bcdfc0e8d0 con 0x55bcdfcadde0 2024-03-24T00:21:38.528 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:21:38.524+0000 7f93b8bb9740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x55bcdfcb5f00 con 0x55bcdfcb2f50 2024-03-24T00:21:38.528 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:21:38.524+0000 7f93b8bb9740 10 client.0 _ll_register_callbacks cb 0x55bcdfc0b1c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T00:21:38.529 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:21:38.524+0000 7f93aea00640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55bcdfcad3f0 0x55bcdfcb7520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:21:38.529 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:21:38.524+0000 7f93a6000640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55bcdfcadde0 0x55bcdfcb7a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:21:38.529 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:21:38.524+0000 7f93aea00640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55bcdfcad3f0 0x55bcdfcb7520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.118:3300/0 says I am v2:172.21.15.118:49896/0 (socket says 172.21.15.118:49896) 2024-03-24T00:21:38.529 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:21:38.524+0000 7f93a6000640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55bcdfcadde0 0x55bcdfcb7a60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.118:56354/0 (socket says 172.21.15.118:56354) 2024-03-24T00:21:38.529 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:21:38.524+0000 7f93af400640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55bcdfcb2f50 0x55bcdfcbeaa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:21:38.529 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:21:38.524+0000 7f93aea00640 1 -- 172.21.15.118:0/654303604 learned_addr learned my addr 172.21.15.118:0/654303604 (peer_addr_for_me v2:172.21.15.118:0/0) 2024-03-24T00:21:38.529 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:21:38.524+0000 7f93af400640 1 -- 172.21.15.118:0/654303604 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55bcdfcadde0 msgr2=0x55bcdfcb7a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:21:38.529 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:21:38.524+0000 7f93af400640 1 --2- 172.21.15.118:0/654303604 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55bcdfcadde0 0x55bcdfcb7a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:21:38.529 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:21:38.524+0000 7f93af400640 1 -- 172.21.15.118:0/654303604 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55bcdfcad3f0 msgr2=0x55bcdfcb7520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:21:38.529 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:21:38.524+0000 7f93af400640 1 --2- 172.21.15.118:0/654303604 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55bcdfcad3f0 0x55bcdfcb7520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:21:38.529 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:21:38.524+0000 7f93af400640 1 -- 172.21.15.118:0/654303604 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55bcdf927210 con 0x55bcdfcb2f50 2024-03-24T00:21:38.529 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:21:38.524+0000 7f93af400640 1 --2- 172.21.15.118:0/654303604 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55bcdfcb2f50 0x55bcdfcbeaa0 secure :-1 s=READY pgs=430 cs=0 l=1 rev1=1 crypto rx=0x7f93a80049d0 tx=0x7f93a809b420 comp rx=0 tx=0).ready entity=mon.1 client_cookie=88987d6a0443d64c server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:21:38.529 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:21:38.524+0000 7f939cc00640 10 client.0 ms_handle_connect on v2:172.21.15.193:3300/0 2024-03-24T00:21:38.530 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:21:38.525+0000 7f939cc00640 1 -- 172.21.15.118:0/654303604 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f93a809fc10 con 0x55bcdfcb2f50 2024-03-24T00:21:38.530 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:21:38.525+0000 7f939cc00640 1 -- 172.21.15.118:0/654303604 <== mon.1 v2:172.21.15.193:3300/0 2 ==== osd_map(102..102 src has 1..102) v4 ==== 9309+0+0 (secure 0 0 0) 0x7f93a80cf200 con 0x55bcdfcb2f50 2024-03-24T00:21:38.530 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:21:38.525+0000 7f93b8bb9740 1 -- 172.21.15.118:0/654303604 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f93a00043c0 con 0x55bcdfcb2f50 2024-03-24T00:21:38.530 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:21:38.525+0000 7f939cc00640 1 -- 172.21.15.118:0/654303604 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_get_version_reply(handle=1 version=86) v2 ==== 24+0+0 (secure 0 0 0) 0x7f93a80a4260 con 0x55bcdfcb2f50 2024-03-24T00:21:38.530 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:21:38.525+0000 7f93b8bb9740 10 client.26791 fetch_fsmap learned FSMap version 86 2024-03-24T00:21:38.530 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:21:38.525+0000 7f93b8bb9740 1 -- 172.21.15.118:0/654303604 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,fsmap=86,monmap=4+}) v3 -- 0x55bcdf9f3620 con 0x55bcdfcb2f50 2024-03-24T00:21:38.530 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:21:38.525+0000 7f939cc00640 1 -- 172.21.15.118:0/654303604 <== mon.1 v2:172.21.15.193:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f93a00043c0 con 0x55bcdfcb2f50 2024-03-24T00:21:38.530 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:21:38.525+0000 7f939cc00640 1 -- 172.21.15.118:0/654303604 <== mon.1 v2:172.21.15.193:3300/0 5 ==== fsmap(e 86) v1 ==== 7950+0+0 (secure 0 0 0) 0x7f93a809af10 con 0x55bcdfcb2f50 2024-03-24T00:21:38.530 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:21:38.525+0000 7f93b8bb9740 10 client.26791 fetch_fsmap finished waiting for FSMap version 86 2024-03-24T00:21:38.530 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:21:38.525+0000 7f93b8bb9740 10 client.26791 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-24T00:21:38.530 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:21:38.525+0000 7f93b8bb9740 20 client.26791 populate_metadata read hostname 'smithi118' 2024-03-24T00:21:38.530 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:21:38.525+0000 7f93b8bb9740 1 --2- 172.21.15.118:0/654303604 >> [v2:172.21.15.189:6838/4074484168,v1:172.21.15.189:6839/4074484168] conn(0x55bcdfcc34a0 0x55bcdfcc5890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:21:38.530 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:21:38.526+0000 7f93b8bb9740 4 client.26791 mds_command: new command op to 24569 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-24T00:21:38.530 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:21:38.526+0000 7f93b8bb9740 1 -- 172.21.15.118:0/654303604 --> [v2:172.21.15.189:6838/4074484168,v1:172.21.15.189:6839/4074484168] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55bcdfc1fd70 con 0x55bcdfcc34a0 2024-03-24T00:21:38.531 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:21:38.526+0000 7f93aea00640 1 --2- 172.21.15.118:0/654303604 >> [v2:172.21.15.189:6838/4074484168,v1:172.21.15.189:6839/4074484168] conn(0x55bcdfcc34a0 0x55bcdfcc5890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:21:38.531 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:21:38.526+0000 7f93aea00640 1 --2- 172.21.15.118:0/654303604 >> [v2:172.21.15.189:6838/4074484168,v1:172.21.15.189:6839/4074484168] conn(0x55bcdfcc34a0 0x55bcdfcc5890 crc :-1 s=READY pgs=220 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=87b860e714e48e88 server_cookie=a0f0280869ea5c95 in_seq=0 out_seq=0 2024-03-24T00:21:38.531 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:21:38.526+0000 7f939cc00640 10 client.26791 ms_handle_connect on v2:172.21.15.189:6838/4074484168 2024-03-24T00:21:38.532 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:21:38.527+0000 7f939cc00640 1 -- 172.21.15.118:0/654303604 <== mds.0 v2:172.21.15.189:6838/4074484168 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x55bcdfc1fd70 con 0x55bcdfcc34a0 2024-03-24T00:21:38.532 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:21:38.527+0000 7f939cc00640 10 client.26791 handle_command_reply: tid=0 2024-03-24T00:21:38.532 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:21:38.527+0000 7f93b8bb9740 1 client.26791 shutdown 2024-03-24T00:21:38.532 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:21:38.528+0000 7f93b8bb9740 1 -- 172.21.15.118:0/654303604 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55bcdfcb2f50 msgr2=0x55bcdfcbeaa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:21:38.532 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:21:38.528+0000 7f93b8bb9740 1 --2- 172.21.15.118:0/654303604 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55bcdfcb2f50 0x55bcdfcbeaa0 secure :-1 s=READY pgs=430 cs=0 l=1 rev1=1 crypto rx=0x7f93a80049d0 tx=0x7f93a809b420 comp rx=0 tx=0).stop 2024-03-24T00:21:38.532 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:21:38.528+0000 7f93b8bb9740 1 -- 172.21.15.118:0/654303604 shutdown_connections 2024-03-24T00:21:38.532 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:21:38.528+0000 7f93b8bb9740 1 --2- 172.21.15.118:0/654303604 >> [v2:172.21.15.189:6838/4074484168,v1:172.21.15.189:6839/4074484168] conn(0x55bcdfcc34a0 0x55bcdfcc5890 crc :-1 s=READY pgs=220 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:21:38.532 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:21:38.528+0000 7f93b8bb9740 1 --2- 172.21.15.118:0/654303604 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55bcdfcb2f50 0x55bcdfcbeaa0 unknown :-1 s=CLOSED pgs=430 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:21:38.532 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:21:38.528+0000 7f93b8bb9740 1 --2- 172.21.15.118:0/654303604 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55bcdfcadde0 0x55bcdfcb7a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:21:38.532 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:21:38.528+0000 7f93b8bb9740 1 --2- 172.21.15.118:0/654303604 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55bcdfcad3f0 0x55bcdfcb7520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:21:38.532 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:21:38.528+0000 7f93b8bb9740 1 -- 172.21.15.118:0/654303604 >> 172.21.15.118:0/654303604 conn(0x55bcdfca8960 msgr2=0x55bcdfca7b70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:21:38.532 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:21:38.528+0000 7f939cc00640 0 client.26791 ms_handle_reset on v2:172.21.15.189:6838/4074484168 2024-03-24T00:21:38.533 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:21:38.528+0000 7f93b8bb9740 1 -- 172.21.15.118:0/654303604 shutdown_connections 2024-03-24T00:21:38.533 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:21:38.528+0000 7f93b8bb9740 1 -- 172.21.15.118:0/654303604 wait complete. 2024-03-24T00:21:38.533 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:21:38.528+0000 7f93b8bb9740 20 client.26791 trim_cache size 0 max 16384 2024-03-24T00:21:38.543 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:21:38.543+0000 7f93b8bb9740 1 Processor -- start 2024-03-24T00:21:38.544 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:21:38.543+0000 7f93b8bb9740 1 -- start start 2024-03-24T00:21:38.544 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:21:38.543+0000 7f93b8bb9740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55bcdfcad3f0 0x55bcdfca8d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:21:38.544 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:21:38.543+0000 7f93b8bb9740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55bcdfcadde0 0x55bcdfcbea60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:21:38.544 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:21:38.543+0000 7f93b8bb9740 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55bcdfcb2f50 0x55bcdfcaf0a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:21:38.544 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:21:38.543+0000 7f93b8bb9740 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f93b01b0560 con 0x55bcdfcb2f50 2024-03-24T00:21:38.544 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:21:38.543+0000 7f93b8bb9740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f93b01b0bc0 con 0x55bcdfcad3f0 2024-03-24T00:21:38.544 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:21:38.543+0000 7f93b8bb9740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f93b01b1280 con 0x55bcdfcadde0 2024-03-24T00:21:38.544 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:21:38.543+0000 7f93aea00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55bcdfcad3f0 0x55bcdfca8d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:21:38.544 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:21:38.543+0000 7f93aea00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55bcdfcad3f0 0x55bcdfca8d40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.118:56358/0 (socket says 172.21.15.118:56358) 2024-03-24T00:21:38.544 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:21:38.543+0000 7f93af400640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55bcdfcb2f50 0x55bcdfcaf0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:21:38.545 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:21:38.543+0000 7f93aea00640 1 -- 172.21.15.118:0/1531893583 learned_addr learned my addr 172.21.15.118:0/1531893583 (peer_addr_for_me v2:172.21.15.118:0/0) 2024-03-24T00:21:38.545 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:21:38.543+0000 7f93a6000640 1 --2- 172.21.15.118:0/1531893583 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55bcdfcadde0 0x55bcdfcbea60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:21:38.545 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:21:38.544+0000 7f93aea00640 1 -- 172.21.15.118:0/1531893583 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55bcdfcadde0 msgr2=0x55bcdfcbea60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:21:38.545 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:21:38.544+0000 7f93aea00640 1 --2- 172.21.15.118:0/1531893583 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55bcdfcadde0 0x55bcdfcbea60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:21:38.545 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:21:38.544+0000 7f93aea00640 1 -- 172.21.15.118:0/1531893583 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55bcdfcb2f50 msgr2=0x55bcdfcaf0a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:21:38.545 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:21:38.544+0000 7f93aea00640 1 --2- 172.21.15.118:0/1531893583 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55bcdfcb2f50 0x55bcdfcaf0a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:21:38.545 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:21:38.544+0000 7f93aea00640 1 -- 172.21.15.118:0/1531893583 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55bcdfcb6040 con 0x55bcdfcad3f0 2024-03-24T00:21:38.545 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:21:38.544+0000 7f93aea00640 1 --2- 172.21.15.118:0/1531893583 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55bcdfcad3f0 0x55bcdfca8d40 secure :-1 s=READY pgs=425 cs=0 l=1 rev1=1 crypto rx=0x7f93a001b040 tx=0x7f93a0003dd0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=53ca0203976dbc0e server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:21:38.549 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:21:38.544+0000 7f939f400640 1 -- 172.21.15.118:0/1531893583 <== mon.2 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f93a0031070 con 0x55bcdfcad3f0 2024-03-24T00:21:38.549 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:21:38.544+0000 7f939f400640 1 -- 172.21.15.118:0/1531893583 <== mon.2 v2:172.21.15.189:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f93a002b2f0 con 0x55bcdfcad3f0 2024-03-24T00:21:38.549 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:21:38.544+0000 7f939f400640 1 -- 172.21.15.118:0/1531893583 <== mon.2 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f93a0003b00 con 0x55bcdfcad3f0 2024-03-24T00:21:38.549 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:21:38.545+0000 7f93b8bb9740 1 -- 172.21.15.118:0/1531893583 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55bcdfcad3f0 msgr2=0x55bcdfca8d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:21:38.549 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:21:38.545+0000 7f93b8bb9740 1 --2- 172.21.15.118:0/1531893583 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55bcdfcad3f0 0x55bcdfca8d40 secure :-1 s=READY pgs=425 cs=0 l=1 rev1=1 crypto rx=0x7f93a001b040 tx=0x7f93a0003dd0 comp rx=0 tx=0).stop 2024-03-24T00:21:38.549 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:21:38.545+0000 7f93b8bb9740 1 -- 172.21.15.118:0/1531893583 shutdown_connections 2024-03-24T00:21:38.549 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:21:38.545+0000 7f93b8bb9740 1 --2- 172.21.15.118:0/1531893583 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55bcdfcb2f50 0x55bcdfcaf0a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:21:38.549 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:21:38.545+0000 7f93b8bb9740 1 --2- 172.21.15.118:0/1531893583 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55bcdfcadde0 0x55bcdfcbea60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:21:38.550 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:21:38.545+0000 7f93b8bb9740 1 --2- 172.21.15.118:0/1531893583 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55bcdfcad3f0 0x55bcdfca8d40 unknown :-1 s=CLOSED pgs=425 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:21:38.550 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:21:38.545+0000 7f93b8bb9740 1 -- 172.21.15.118:0/1531893583 >> 172.21.15.118:0/1531893583 conn(0x55bcdfca8960 msgr2=0x55bcdfca36a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:21:38.550 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:21:38.545+0000 7f93b8bb9740 1 -- 172.21.15.118:0/1531893583 shutdown_connections 2024-03-24T00:21:38.550 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:21:38.545+0000 7f93b8bb9740 1 -- 172.21.15.118:0/1531893583 wait complete. 2024-03-24T00:21:38.550 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:21:38.545+0000 7f93b8bb9740 1 -- start start 2024-03-24T00:21:38.550 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:21:38.546+0000 7f93b8bb9740 1 Processor -- start 2024-03-24T00:21:38.550 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:21:38.546+0000 7f93b8bb9740 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55bcdfcad3f0 0x55bcdfcbc880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:21:38.550 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:21:38.546+0000 7f93b8bb9740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55bcdfcadde0 0x55bcdfcbcdc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:21:38.550 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:21:38.546+0000 7f93b8bb9740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55bcdfcb2f50 0x55bcdfcbddd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:21:38.550 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:21:38.546+0000 7f93b8bb9740 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f93b01ada20 con 0x55bcdfcad3f0 2024-03-24T00:21:38.550 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:21:38.546+0000 7f93b8bb9740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f93b01aecc0 con 0x55bcdfcadde0 2024-03-24T00:21:38.551 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:21:38.546+0000 7f93b8bb9740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f93b01ae020 con 0x55bcdfcb2f50 2024-03-24T00:21:38.551 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:21:38.546+0000 7f93b8bb9740 10 client.0 _ll_register_callbacks cb 0x7f93a809b890 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T00:21:38.551 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:21:38.546+0000 7f93aea00640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55bcdfcad3f0 0x55bcdfcbc880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:21:38.551 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:21:38.546+0000 7f93af400640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55bcdfcb2f50 0x55bcdfcbddd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:21:38.551 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:21:38.546+0000 7f93a6000640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55bcdfcadde0 0x55bcdfcbcdc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:21:38.551 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:21:38.546+0000 7f93af400640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55bcdfcb2f50 0x55bcdfcbddd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.118:40570/0 (socket says 172.21.15.118:40570) 2024-03-24T00:21:38.551 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:21:38.546+0000 7f93a6000640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55bcdfcadde0 0x55bcdfcbcdc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.118:56374/0 (socket says 172.21.15.118:56374) 2024-03-24T00:21:38.551 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:21:38.546+0000 7f93af400640 1 -- 172.21.15.118:0/1659455897 learned_addr learned my addr 172.21.15.118:0/1659455897 (peer_addr_for_me v2:172.21.15.118:0/0) 2024-03-24T00:21:38.551 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:21:38.547+0000 7f93af400640 1 -- 172.21.15.118:0/1659455897 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55bcdfcadde0 msgr2=0x55bcdfcbcdc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:21:38.551 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:21:38.547+0000 7f93af400640 1 --2- 172.21.15.118:0/1659455897 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55bcdfcadde0 0x55bcdfcbcdc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:21:38.551 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:21:38.547+0000 7f93af400640 1 -- 172.21.15.118:0/1659455897 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55bcdfcad3f0 msgr2=0x55bcdfcbc880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:21:38.551 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:21:38.547+0000 7f93af400640 1 --2- 172.21.15.118:0/1659455897 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55bcdfcad3f0 0x55bcdfcbc880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:21:38.551 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:21:38.547+0000 7f93af400640 1 -- 172.21.15.118:0/1659455897 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55bcdfcbe3c0 con 0x55bcdfcb2f50 2024-03-24T00:21:38.552 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:21:38.547+0000 7f93a6000640 1 --2- 172.21.15.118:0/1659455897 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55bcdfcadde0 0x55bcdfcbcdc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T00:21:38.552 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:21:38.547+0000 7f93aea00640 1 --2- 172.21.15.118:0/1659455897 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55bcdfcad3f0 0x55bcdfcbc880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T00:21:38.552 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:21:38.547+0000 7f93af400640 1 --2- 172.21.15.118:0/1659455897 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55bcdfcb2f50 0x55bcdfcbddd0 secure :-1 s=READY pgs=431 cs=0 l=1 rev1=1 crypto rx=0x7f93a80a4420 tx=0x7f93a809b550 comp rx=0 tx=0).ready entity=mon.1 client_cookie=857f655347af1778 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:21:38.552 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:21:38.547+0000 7f939cc00640 10 client.0 ms_handle_connect on v2:172.21.15.193:3300/0 2024-03-24T00:21:38.552 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:21:38.547+0000 7f939cc00640 1 -- 172.21.15.118:0/1659455897 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f93a809c1c0 con 0x55bcdfcb2f50 2024-03-24T00:21:38.552 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:21:38.547+0000 7f939cc00640 1 -- 172.21.15.118:0/1659455897 <== mon.1 v2:172.21.15.193:3300/0 2 ==== osd_map(102..102 src has 1..102) v4 ==== 9309+0+0 (secure 0 0 0) 0x7f93a80ba160 con 0x55bcdfcb2f50 2024-03-24T00:21:38.552 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:21:38.547+0000 7f93b8bb9740 1 -- 172.21.15.118:0/1659455897 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55bcdfc1a0c0 con 0x55bcdfcb2f50 2024-03-24T00:21:38.552 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:21:38.547+0000 7f939cc00640 1 -- 172.21.15.118:0/1659455897 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_get_version_reply(handle=1 version=86) v2 ==== 24+0+0 (secure 0 0 0) 0x7f93a80ba360 con 0x55bcdfcb2f50 2024-03-24T00:21:38.552 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:21:38.547+0000 7f93b8bb9740 10 client.26797 fetch_fsmap learned FSMap version 86 2024-03-24T00:21:38.552 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:21:38.547+0000 7f93b8bb9740 1 -- 172.21.15.118:0/1659455897 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,fsmap=86,monmap=4+}) v3 -- 0x55bcdfd1f500 con 0x55bcdfcb2f50 2024-03-24T00:21:38.552 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:21:38.548+0000 7f939cc00640 1 -- 172.21.15.118:0/1659455897 <== mon.1 v2:172.21.15.193:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x55bcdfc1a0c0 con 0x55bcdfcb2f50 2024-03-24T00:21:38.552 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:21:38.548+0000 7f939cc00640 1 -- 172.21.15.118:0/1659455897 <== mon.1 v2:172.21.15.193:3300/0 5 ==== fsmap(e 86) v1 ==== 7950+0+0 (secure 0 0 0) 0x7f93a80b4b80 con 0x55bcdfcb2f50 2024-03-24T00:21:38.552 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:21:38.548+0000 7f93b8bb9740 10 client.26797 fetch_fsmap finished waiting for FSMap version 86 2024-03-24T00:21:38.553 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:21:38.548+0000 7f93b8bb9740 10 client.26797 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-24T00:21:38.553 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:21:38.548+0000 7f93b8bb9740 20 client.26797 populate_metadata read hostname 'smithi118' 2024-03-24T00:21:38.553 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:21:38.548+0000 7f93b8bb9740 1 --2- 172.21.15.118:0/1659455897 >> [v2:172.21.15.189:6838/4074484168,v1:172.21.15.189:6839/4074484168] conn(0x55bcdfd1fce0 0x55bcdfd22100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:21:38.553 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:21:38.548+0000 7f93b8bb9740 4 client.26797 mds_command: new command op to 24569 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-03-24T00:21:38.553 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:21:38.548+0000 7f93b8bb9740 1 -- 172.21.15.118:0/1659455897 --> [v2:172.21.15.189:6838/4074484168,v1:172.21.15.189:6839/4074484168] -- command(tid 0: {"prefix": "flush journal"}) v1 -- 0x55bcdfb985d0 con 0x55bcdfd1fce0 2024-03-24T00:21:38.553 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:21:38.548+0000 7f93aea00640 1 --2- 172.21.15.118:0/1659455897 >> [v2:172.21.15.189:6838/4074484168,v1:172.21.15.189:6839/4074484168] conn(0x55bcdfd1fce0 0x55bcdfd22100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:21:38.553 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:21:38.548+0000 7f93aea00640 1 --2- 172.21.15.118:0/1659455897 >> [v2:172.21.15.189:6838/4074484168,v1:172.21.15.189:6839/4074484168] conn(0x55bcdfd1fce0 0x55bcdfd22100 crc :-1 s=READY pgs=221 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=dd4d5a3c0d8c7e7b server_cookie=98a6b0f76cc0362b in_seq=0 out_seq=0 2024-03-24T00:21:38.553 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:21:38.548+0000 7f939cc00640 10 client.26797 ms_handle_connect on v2:172.21.15.189:6838/4074484168 2024-03-24T00:21:38.704 INFO:journalctl@ceph.mds.h.smithi189.stdout:Mar 24 00:21:38 smithi189 podman[69798]: 2024-03-24 00:21:38.261491249 +0000 UTC m=+0.423433885 container init 9706b49f92e8c2e438ca1e4fe4db03e0eee967249543d0df6906c3f11e450b28 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-h, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240319, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS) 2024-03-24T00:21:38.704 INFO:journalctl@ceph.mds.h.smithi189.stdout:Mar 24 00:21:38 smithi189 podman[69798]: 2024-03-24 00:21:38.265187767 +0000 UTC m=+0.427130399 container start 9706b49f92e8c2e438ca1e4fe4db03e0eee967249543d0df6906c3f11e450b28 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-h, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240319, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, GIT_CLEAN=True, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5) 2024-03-24T00:21:38.704 INFO:journalctl@ceph.mds.h.smithi189.stdout:Mar 24 00:21:38 smithi189 ceph-mds[69816]: -- 172.21.15.189:0/3505677850 <== mon.2 v2:172.21.15.189:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x5592194c9a40 con 0x55921949c800 2024-03-24T00:21:38.704 INFO:journalctl@ceph.mds.h.smithi189.stdout:Mar 24 00:21:38 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-h[69812]: starting mds.h at 2024-03-24T00:21:38.704 INFO:journalctl@ceph.mds.h.smithi189.stdout:Mar 24 00:21:38 smithi189 bash[69798]: 9706b49f92e8c2e438ca1e4fe4db03e0eee967249543d0df6906c3f11e450b28 2024-03-24T00:21:38.705 INFO:journalctl@ceph.mds.h.smithi189.stdout:Mar 24 00:21:38 smithi189 systemd[1]: Started Ceph mds.h for d79a5420-e971-11ee-95cd-87774f69a715. 2024-03-24T00:21:38.710 INFO:journalctl@ceph.mds.l.smithi193.stdout:Mar 24 00:21:38 smithi193 podman[72006]: 2024-03-24 00:21:38.350173221 +0000 UTC m=+0.524774076 container init 200b5ef45dd84c481ecdc9a61d2012b16dc125794c0fb337e89dec4988ba73bd (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-l, org.label-schema.build-date=20240319, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, ceph=True, maintainer=Guillaume Abrioux , GIT_CLEAN=True) 2024-03-24T00:21:38.711 INFO:journalctl@ceph.mds.l.smithi193.stdout:Mar 24 00:21:38 smithi193 podman[72006]: 2024-03-24 00:21:38.35297475 +0000 UTC m=+0.527575595 container start 200b5ef45dd84c481ecdc9a61d2012b16dc125794c0fb337e89dec4988ba73bd (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-l, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240319) 2024-03-24T00:21:38.711 INFO:journalctl@ceph.mds.l.smithi193.stdout:Mar 24 00:21:38 smithi193 ceph-mds[72025]: -- 172.21.15.193:0/4208550513 <== mon.1 v2:172.21.15.193:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x5598cd51ba40 con 0x5598cd4ef000 2024-03-24T00:21:38.711 INFO:journalctl@ceph.mds.l.smithi193.stdout:Mar 24 00:21:38 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-l[72021]: starting mds.l at 2024-03-24T00:21:38.711 INFO:journalctl@ceph.mds.l.smithi193.stdout:Mar 24 00:21:38 smithi193 bash[72006]: 200b5ef45dd84c481ecdc9a61d2012b16dc125794c0fb337e89dec4988ba73bd 2024-03-24T00:21:38.711 INFO:journalctl@ceph.mds.l.smithi193.stdout:Mar 24 00:21:38 smithi193 systemd[1]: Started Ceph mds.l for d79a5420-e971-11ee-95cd-87774f69a715. 2024-03-24T00:21:39.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:21:38 smithi118 ceph-mon[31535]: daemon mds.h restarted 2024-03-24T00:21:39.197 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:21:38 smithi118 ceph-mon[31535]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-03-24T00:21:39.197 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:21:38 smithi118 ceph-mon[31535]: osdmap e101: 12 total, 12 up, 12 in 2024-03-24T00:21:39.197 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:21:38 smithi118 ceph-mon[31535]: mds.? [v2:172.21.15.189:6836/2350260652,v1:172.21.15.189:6837/2350260652] up:boot 2024-03-24T00:21:39.197 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:21:38 smithi118 ceph-mon[31535]: Standby daemon mds.g assigned to filesystem cephfs as rank 3 2024-03-24T00:21:39.197 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:21:38 smithi118 ceph-mon[31535]: fsmap cephfs:4/5 {0=b=up:active,1=l=up:active,2=c=up:active,4=e=up:active} 5 up:standby-replay 3 up:standby, 1 failed 2024-03-24T00:21:39.197 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:21:38 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-03-24T00:21:39.197 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:21:38 smithi118 ceph-mon[31535]: fsmap cephfs:5/5 4 up:standby-replay 3 up:standby 1 up:replay 4 up:active 2024-03-24T00:21:39.197 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:21:38 smithi118 ceph-mon[31535]: daemon mds.l restarted 2024-03-24T00:21:39.197 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:21:38 smithi118 ceph-mon[31535]: osdmap e102: 12 total, 12 up, 12 in 2024-03-24T00:21:39.197 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:21:38 smithi118 ceph-mon[31535]: mds.? [v2:172.21.15.193:6832/2276762432,v1:172.21.15.193:6833/2276762432] up:boot 2024-03-24T00:21:39.197 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:21:38 smithi118 ceph-mon[31535]: Standby daemon mds.f assigned to filesystem cephfs as rank 1 2024-03-24T00:21:39.197 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:21:38 smithi118 ceph-mon[31535]: fsmap cephfs:4/5 {0=b=up:active,2=c=up:active,3=g=up:replay,4=e=up:active} 4 up:standby-replay 4 up:standby, 1 failed 2024-03-24T00:21:39.198 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:21:38 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "l"}]: dispatch 2024-03-24T00:21:39.198 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:21:38 smithi118 ceph-mon[31535]: fsmap cephfs:5/5 3 up:standby-replay 4 up:standby 2 up:replay 3 up:active 2024-03-24T00:21:39.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:21:38 smithi189 ceph-mon[36187]: daemon mds.h restarted 2024-03-24T00:21:39.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:21:38 smithi189 ceph-mon[36187]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-03-24T00:21:39.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:21:38 smithi189 ceph-mon[36187]: osdmap e101: 12 total, 12 up, 12 in 2024-03-24T00:21:39.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:21:38 smithi189 ceph-mon[36187]: mds.? [v2:172.21.15.189:6836/2350260652,v1:172.21.15.189:6837/2350260652] up:boot 2024-03-24T00:21:39.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:21:38 smithi189 ceph-mon[36187]: Standby daemon mds.g assigned to filesystem cephfs as rank 3 2024-03-24T00:21:39.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:21:38 smithi189 ceph-mon[36187]: fsmap cephfs:4/5 {0=b=up:active,1=l=up:active,2=c=up:active,4=e=up:active} 5 up:standby-replay 3 up:standby, 1 failed 2024-03-24T00:21:39.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:21:38 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-03-24T00:21:39.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:21:38 smithi189 ceph-mon[36187]: fsmap cephfs:5/5 4 up:standby-replay 3 up:standby 1 up:replay 4 up:active 2024-03-24T00:21:39.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:21:38 smithi189 ceph-mon[36187]: daemon mds.l restarted 2024-03-24T00:21:39.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:21:38 smithi189 ceph-mon[36187]: osdmap e102: 12 total, 12 up, 12 in 2024-03-24T00:21:39.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:21:38 smithi189 ceph-mon[36187]: mds.? [v2:172.21.15.193:6832/2276762432,v1:172.21.15.193:6833/2276762432] up:boot 2024-03-24T00:21:39.205 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:21:38 smithi189 ceph-mon[36187]: Standby daemon mds.f assigned to filesystem cephfs as rank 1 2024-03-24T00:21:39.205 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:21:38 smithi189 ceph-mon[36187]: fsmap cephfs:4/5 {0=b=up:active,2=c=up:active,3=g=up:replay,4=e=up:active} 4 up:standby-replay 4 up:standby, 1 failed 2024-03-24T00:21:39.205 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:21:38 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "l"}]: dispatch 2024-03-24T00:21:39.205 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:21:38 smithi189 ceph-mon[36187]: fsmap cephfs:5/5 3 up:standby-replay 4 up:standby 2 up:replay 3 up:active 2024-03-24T00:21:39.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:21:38 smithi193 ceph-mon[36878]: daemon mds.h restarted 2024-03-24T00:21:39.212 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:21:38 smithi193 ceph-mon[36878]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-03-24T00:21:39.212 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:21:38 smithi193 ceph-mon[36878]: osdmap e101: 12 total, 12 up, 12 in 2024-03-24T00:21:39.212 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:21:38 smithi193 ceph-mon[36878]: mds.? [v2:172.21.15.189:6836/2350260652,v1:172.21.15.189:6837/2350260652] up:boot 2024-03-24T00:21:39.212 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:21:38 smithi193 ceph-mon[36878]: Standby daemon mds.g assigned to filesystem cephfs as rank 3 2024-03-24T00:21:39.213 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:21:38 smithi193 ceph-mon[36878]: fsmap cephfs:4/5 {0=b=up:active,1=l=up:active,2=c=up:active,4=e=up:active} 5 up:standby-replay 3 up:standby, 1 failed 2024-03-24T00:21:39.213 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:21:38 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-03-24T00:21:39.213 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:21:38 smithi193 ceph-mon[36878]: fsmap cephfs:5/5 4 up:standby-replay 3 up:standby 1 up:replay 4 up:active 2024-03-24T00:21:39.213 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:21:38 smithi193 ceph-mon[36878]: daemon mds.l restarted 2024-03-24T00:21:39.213 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:21:38 smithi193 ceph-mon[36878]: osdmap e102: 12 total, 12 up, 12 in 2024-03-24T00:21:39.213 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:21:38 smithi193 ceph-mon[36878]: mds.? [v2:172.21.15.193:6832/2276762432,v1:172.21.15.193:6833/2276762432] up:boot 2024-03-24T00:21:39.213 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:21:38 smithi193 ceph-mon[36878]: Standby daemon mds.f assigned to filesystem cephfs as rank 1 2024-03-24T00:21:39.214 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:21:38 smithi193 ceph-mon[36878]: fsmap cephfs:4/5 {0=b=up:active,2=c=up:active,3=g=up:replay,4=e=up:active} 4 up:standby-replay 4 up:standby, 1 failed 2024-03-24T00:21:39.214 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:21:38 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "l"}]: dispatch 2024-03-24T00:21:39.214 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:21:38 smithi193 ceph-mon[36878]: fsmap cephfs:5/5 3 up:standby-replay 4 up:standby 2 up:replay 3 up:active 2024-03-24T00:21:40.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:21:40 smithi118 ceph-mon[31535]: mds.? [v2:172.21.15.193:6834/2814329894,v1:172.21.15.193:6835/2814329894] up:resolve 2024-03-24T00:21:40.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:21:40 smithi118 ceph-mon[31535]: mds.? [v2:172.21.15.118:6838/979323377,v1:172.21.15.118:6839/979323377] up:resolve 2024-03-24T00:21:40.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:21:40 smithi118 ceph-mon[31535]: fsmap cephfs:5/5 3 up:standby-replay 4 up:standby 2 up:resolve 3 up:active 2024-03-24T00:21:40.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:21:40 smithi118 ceph-mon[31535]: pgmap v589: 97 pgs: 97 active+clean; 136 MiB data, 776 MiB used, 1.0 TiB / 1.0 TiB avail; 57 KiB/s rd, 76 KiB/s wr, 28 op/s 2024-03-24T00:21:40.704 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:21:40 smithi189 ceph-mon[36187]: mds.? [v2:172.21.15.193:6834/2814329894,v1:172.21.15.193:6835/2814329894] up:resolve 2024-03-24T00:21:40.704 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:21:40 smithi189 ceph-mon[36187]: mds.? [v2:172.21.15.118:6838/979323377,v1:172.21.15.118:6839/979323377] up:resolve 2024-03-24T00:21:40.704 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:21:40 smithi189 ceph-mon[36187]: fsmap cephfs:5/5 3 up:standby-replay 4 up:standby 2 up:resolve 3 up:active 2024-03-24T00:21:40.704 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:21:40 smithi189 ceph-mon[36187]: pgmap v589: 97 pgs: 97 active+clean; 136 MiB data, 776 MiB used, 1.0 TiB / 1.0 TiB avail; 57 KiB/s rd, 76 KiB/s wr, 28 op/s 2024-03-24T00:21:40.710 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:21:40 smithi193 ceph-mon[36878]: mds.? [v2:172.21.15.193:6834/2814329894,v1:172.21.15.193:6835/2814329894] up:resolve 2024-03-24T00:21:40.711 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:21:40 smithi193 ceph-mon[36878]: mds.? [v2:172.21.15.118:6838/979323377,v1:172.21.15.118:6839/979323377] up:resolve 2024-03-24T00:21:40.711 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:21:40 smithi193 ceph-mon[36878]: fsmap cephfs:5/5 3 up:standby-replay 4 up:standby 2 up:resolve 3 up:active 2024-03-24T00:21:40.711 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:21:40 smithi193 ceph-mon[36878]: pgmap v589: 97 pgs: 97 active+clean; 136 MiB data, 776 MiB used, 1.0 TiB / 1.0 TiB avail; 57 KiB/s rd, 76 KiB/s wr, 28 op/s 2024-03-24T00:21:41.463 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:21:41.462+0000 7f939cc00640 1 -- 172.21.15.118:0/1659455897 <== mds.0 v2:172.21.15.189:6838/4074484168 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+44 (crc 0 0 0) 0x55bcdfb985d0 con 0x55bcdfd1fce0 2024-03-24T00:21:41.464 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:21:41.462+0000 7f939cc00640 10 client.26797 handle_command_reply: tid=0 2024-03-24T00:21:41.464 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:21:41.462+0000 7f93b8bb9740 1 client.26797 shutdown 2024-03-24T00:21:41.464 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:21:41.463+0000 7f93b8bb9740 1 -- 172.21.15.118:0/1659455897 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55bcdfcb2f50 msgr2=0x55bcdfcbddd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:21:41.464 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:21:41.463+0000 7f93b8bb9740 1 --2- 172.21.15.118:0/1659455897 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55bcdfcb2f50 0x55bcdfcbddd0 secure :-1 s=READY pgs=431 cs=0 l=1 rev1=1 crypto rx=0x7f93a80a4420 tx=0x7f93a809b550 comp rx=0 tx=0).stop 2024-03-24T00:21:41.464 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:21:41.463+0000 7f93b8bb9740 1 -- 172.21.15.118:0/1659455897 shutdown_connections 2024-03-24T00:21:41.464 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:21:41.463+0000 7f93b8bb9740 1 --2- 172.21.15.118:0/1659455897 >> [v2:172.21.15.189:6838/4074484168,v1:172.21.15.189:6839/4074484168] conn(0x55bcdfd1fce0 0x55bcdfd22100 crc :-1 s=READY pgs=221 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:21:41.464 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:21:41.463+0000 7f93b8bb9740 1 --2- 172.21.15.118:0/1659455897 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55bcdfcb2f50 0x55bcdfcbddd0 unknown :-1 s=CLOSED pgs=431 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:21:41.464 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:21:41.463+0000 7f93b8bb9740 1 --2- 172.21.15.118:0/1659455897 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55bcdfcadde0 0x55bcdfcbcdc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:21:41.464 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:21:41.463+0000 7f93b8bb9740 1 --2- 172.21.15.118:0/1659455897 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55bcdfcad3f0 0x55bcdfcbc880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:21:41.464 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:21:41.463+0000 7f93b8bb9740 1 -- 172.21.15.118:0/1659455897 >> 172.21.15.118:0/1659455897 conn(0x55bcdfca8960 msgr2=0x55bcdfca8d40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:21:41.464 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:21:41.463+0000 7f939cc00640 0 client.26797 ms_handle_reset on v2:172.21.15.189:6838/4074484168 2024-03-24T00:21:41.465 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:21:41.463+0000 7f93b8bb9740 1 -- 172.21.15.118:0/1659455897 shutdown_connections 2024-03-24T00:21:41.465 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:21:41.463+0000 7f93b8bb9740 1 -- 172.21.15.118:0/1659455897 wait complete. 2024-03-24T00:21:41.465 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:21:41.464+0000 7f93b8bb9740 20 client.26797 trim_cache size 0 max 16384 2024-03-24T00:21:41.465 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stdout:{ 2024-03-24T00:21:41.465 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stdout: "message": "", 2024-03-24T00:21:41.466 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stdout: "return_code": 0 2024-03-24T00:21:41.466 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stdout:} 2024-03-24T00:21:41.466 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:21:41.465+0000 7f93afe00640 1 -- 172.21.15.118:0/725018075 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7f93940a27f0 msgr2=0x7f93940a4cb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:21:41.466 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:21:41.465+0000 7f93afe00640 1 --2- 172.21.15.118:0/725018075 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7f93940a27f0 0x7f93940a4cb0 secure :-1 s=READY pgs=369 cs=0 l=1 rev1=1 crypto rx=0x7f939802c260 tx=0x7f9398005b70 comp rx=0 tx=0).stop 2024-03-24T00:21:41.466 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:21:41.465+0000 7f93afe00640 1 -- 172.21.15.118:0/725018075 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f93b01096d0 msgr2=0x7f93b01a53c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:21:41.467 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:21:41.465+0000 7f93afe00640 1 --2- 172.21.15.118:0/725018075 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f93b01096d0 0x7f93b01a53c0 secure :-1 s=READY pgs=424 cs=0 l=1 rev1=1 crypto rx=0x7f93a800eb10 tx=0x7f93a800c2d0 comp rx=0 tx=0).stop 2024-03-24T00:21:41.467 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:21:41.465+0000 7f93afe00640 1 -- 172.21.15.118:0/725018075 shutdown_connections 2024-03-24T00:21:41.467 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:21:41.465+0000 7f93afe00640 1 --2- 172.21.15.118:0/725018075 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7f93940a27f0 0x7f93940a4cb0 unknown :-1 s=CLOSED pgs=369 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:21:41.467 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:21:41.465+0000 7f93afe00640 1 --2- 172.21.15.118:0/725018075 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f93b01096d0 0x7f93b01a53c0 unknown :-1 s=CLOSED pgs=424 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:21:41.467 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:21:41.465+0000 7f93afe00640 1 --2- 172.21.15.118:0/725018075 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f93b0108d10 0x7f93b019e350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:21:41.467 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:21:41.465+0000 7f93afe00640 1 --2- 172.21.15.118:0/725018075 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f93b0107b20 0x7f93b019de10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:21:41.467 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:21:41.465+0000 7f93afe00640 1 -- 172.21.15.118:0/725018075 >> 172.21.15.118:0/725018075 conn(0x7f93b0076ae0 msgr2=0x7f93b010c010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:21:41.467 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:21:41.465+0000 7f93afe00640 1 -- 172.21.15.118:0/725018075 shutdown_connections 2024-03-24T00:21:41.467 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:21:41.465+0000 7f93afe00640 1 -- 172.21.15.118:0/725018075 wait complete. 2024-03-24T00:21:41.618 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:21:41 smithi193 ceph-mon[36878]: mds.? [v2:172.21.15.118:6838/979323377,v1:172.21.15.118:6839/979323377] up:reconnect 2024-03-24T00:21:41.619 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:21:41 smithi193 ceph-mon[36878]: mds.? [v2:172.21.15.193:6834/2814329894,v1:172.21.15.193:6835/2814329894] up:reconnect 2024-03-24T00:21:41.619 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:21:41 smithi193 ceph-mon[36878]: fsmap cephfs:5/5 3 up:standby-replay 4 up:standby 2 up:reconnect 3 up:active 2024-03-24T00:21:41.619 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:21:41 smithi193 ceph-mon[36878]: reconnect by client.15159 192.168.0.1:0/3463739313 after 0.0289995 2024-03-24T00:21:41.619 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:21:41 smithi193 ceph-mon[36878]: reconnect by client.15159 192.168.0.1:0/3463739313 after 0.0369991 2024-03-24T00:21:41.619 INFO:journalctl@ceph.mds.f.smithi193.stdout:Mar 24 00:21:41 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-f[71869]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: In function 'void Locker::set_xlocks_done(MutationImpl*, bool)' thread 7fdf97a8c700 time 2024-03-24T00:21:41.612694+0000 2024-03-24T00:21:41.619 INFO:journalctl@ceph.mds.f.smithi193.stdout:Mar 24 00:21:41 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-f[71869]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: 626: FAILED ceph_assert(obj->is_auth()) 2024-03-24T00:21:41.619 INFO:journalctl@ceph.mds.f.smithi193.stdout:Mar 24 00:21:41 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-f[71869]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-24T00:21:41.619 INFO:journalctl@ceph.mds.f.smithi193.stdout:Mar 24 00:21:41 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-f[71869]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x135) [0x7fdfa20b64a1] 2024-03-24T00:21:41.619 INFO:journalctl@ceph.mds.f.smithi193.stdout:Mar 24 00:21:41 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-f[71869]: 2: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7fdfa20b6667] 2024-03-24T00:21:41.620 INFO:journalctl@ceph.mds.f.smithi193.stdout:Mar 24 00:21:41 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-f[71869]: 3: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x55bcda43f223] 2024-03-24T00:21:41.620 INFO:journalctl@ceph.mds.f.smithi193.stdout:Mar 24 00:21:41 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-f[71869]: 4: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x55bcda2b95d6] 2024-03-24T00:21:41.620 INFO:journalctl@ceph.mds.f.smithi193.stdout:Mar 24 00:21:41 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-f[71869]: 5: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x55bcda2b9da1] 2024-03-24T00:21:41.620 INFO:journalctl@ceph.mds.f.smithi193.stdout:Mar 24 00:21:41 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-f[71869]: 6: (Server::handle_client_rename(boost::intrusive_ptr const&)+0x259d) [0x55bcda2f697d] 2024-03-24T00:21:41.620 INFO:journalctl@ceph.mds.f.smithi193.stdout:Mar 24 00:21:41 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-f[71869]: 7: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x55bcda30c233] 2024-03-24T00:21:41.620 INFO:journalctl@ceph.mds.f.smithi193.stdout:Mar 24 00:21:41 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-f[71869]: 8: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x55bcda30e197] 2024-03-24T00:21:41.620 INFO:journalctl@ceph.mds.f.smithi193.stdout:Mar 24 00:21:41 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-f[71869]: 9: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x55bcda20d11b] 2024-03-24T00:21:41.620 INFO:journalctl@ceph.mds.f.smithi193.stdout:Mar 24 00:21:41 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-f[71869]: 10: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x55bcda20d6bc] 2024-03-24T00:21:41.620 INFO:journalctl@ceph.mds.f.smithi193.stdout:Mar 24 00:21:41 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-f[71869]: 11: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x55bcda1f71ff] 2024-03-24T00:21:41.620 INFO:journalctl@ceph.mds.f.smithi193.stdout:Mar 24 00:21:41 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-f[71869]: 12: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7fdfa2348488] 2024-03-24T00:21:41.620 INFO:journalctl@ceph.mds.f.smithi193.stdout:Mar 24 00:21:41 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-f[71869]: 13: (DispatchQueue::entry()+0x51f) [0x7fdfa234560f] 2024-03-24T00:21:41.620 INFO:journalctl@ceph.mds.f.smithi193.stdout:Mar 24 00:21:41 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-f[71869]: 14: (DispatchQueue::DispatchThread::entry()+0x11) [0x7fdfa2418e81] 2024-03-24T00:21:41.620 INFO:journalctl@ceph.mds.f.smithi193.stdout:Mar 24 00:21:41 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-f[71869]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7fdfa0e331ca] 2024-03-24T00:21:41.620 INFO:journalctl@ceph.mds.f.smithi193.stdout:Mar 24 00:21:41 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-f[71869]: 16: clone() 2024-03-24T00:21:41.621 INFO:journalctl@ceph.mds.f.smithi193.stdout:Mar 24 00:21:41 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-f[71869]: 2024-03-24T00:21:41.613+0000 7fdf97a8c700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: In function 'void Locker::set_xlocks_done(MutationImpl*, bool)' thread 7fdf97a8c700 time 2024-03-24T00:21:41.612694+0000 2024-03-24T00:21:41.621 INFO:journalctl@ceph.mds.f.smithi193.stdout:Mar 24 00:21:41 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-f[71869]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: 626: FAILED ceph_assert(obj->is_auth()) 2024-03-24T00:21:41.621 INFO:journalctl@ceph.mds.f.smithi193.stdout:Mar 24 00:21:41 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-f[71869]: 2024-03-24T00:21:41.621 INFO:journalctl@ceph.mds.f.smithi193.stdout:Mar 24 00:21:41 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-f[71869]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-24T00:21:41.621 INFO:journalctl@ceph.mds.f.smithi193.stdout:Mar 24 00:21:41 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-f[71869]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x135) [0x7fdfa20b64a1] 2024-03-24T00:21:41.621 INFO:journalctl@ceph.mds.f.smithi193.stdout:Mar 24 00:21:41 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-f[71869]: 2: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7fdfa20b6667] 2024-03-24T00:21:41.621 INFO:journalctl@ceph.mds.f.smithi193.stdout:Mar 24 00:21:41 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-f[71869]: 3: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x55bcda43f223] 2024-03-24T00:21:41.621 INFO:journalctl@ceph.mds.f.smithi193.stdout:Mar 24 00:21:41 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-f[71869]: 4: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x55bcda2b95d6] 2024-03-24T00:21:41.622 INFO:journalctl@ceph.mds.f.smithi193.stdout:Mar 24 00:21:41 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-f[71869]: 5: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x55bcda2b9da1] 2024-03-24T00:21:41.622 INFO:journalctl@ceph.mds.f.smithi193.stdout:Mar 24 00:21:41 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-f[71869]: 6: (Server::handle_client_rename(boost::intrusive_ptr const&)+0x259d) [0x55bcda2f697d] 2024-03-24T00:21:41.622 INFO:journalctl@ceph.mds.f.smithi193.stdout:Mar 24 00:21:41 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-f[71869]: 7: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x55bcda30c233] 2024-03-24T00:21:41.622 INFO:journalctl@ceph.mds.f.smithi193.stdout:Mar 24 00:21:41 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-f[71869]: 8: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x55bcda30e197] 2024-03-24T00:21:41.622 INFO:journalctl@ceph.mds.f.smithi193.stdout:Mar 24 00:21:41 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-f[71869]: 9: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x55bcda20d11b] 2024-03-24T00:21:41.622 INFO:journalctl@ceph.mds.f.smithi193.stdout:Mar 24 00:21:41 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-f[71869]: 10: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x55bcda20d6bc] 2024-03-24T00:21:41.622 INFO:journalctl@ceph.mds.f.smithi193.stdout:Mar 24 00:21:41 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-f[71869]: 11: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x55bcda1f71ff] 2024-03-24T00:21:41.622 INFO:journalctl@ceph.mds.f.smithi193.stdout:Mar 24 00:21:41 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-f[71869]: 12: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7fdfa2348488] 2024-03-24T00:21:41.622 INFO:journalctl@ceph.mds.f.smithi193.stdout:Mar 24 00:21:41 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-f[71869]: 13: (DispatchQueue::entry()+0x51f) [0x7fdfa234560f] 2024-03-24T00:21:41.622 INFO:journalctl@ceph.mds.f.smithi193.stdout:Mar 24 00:21:41 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-f[71869]: 14: (DispatchQueue::DispatchThread::entry()+0x11) [0x7fdfa2418e81] 2024-03-24T00:21:41.622 INFO:journalctl@ceph.mds.f.smithi193.stdout:Mar 24 00:21:41 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-f[71869]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7fdfa0e331ca] 2024-03-24T00:21:41.622 INFO:journalctl@ceph.mds.f.smithi193.stdout:Mar 24 00:21:41 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-f[71869]: 16: clone() 2024-03-24T00:21:41.622 INFO:journalctl@ceph.mds.f.smithi193.stdout:Mar 24 00:21:41 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-f[71869]: 2024-03-24T00:21:41.623 INFO:journalctl@ceph.mds.f.smithi193.stdout:Mar 24 00:21:41 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-f[71869]: *** Caught signal (Aborted) ** 2024-03-24T00:21:41.623 INFO:journalctl@ceph.mds.f.smithi193.stdout:Mar 24 00:21:41 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-f[71869]: in thread 7fdf97a8c700 thread_name:ms_dispatch 2024-03-24T00:21:41.623 INFO:journalctl@ceph.mds.f.smithi193.stdout:Mar 24 00:21:41 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-f[71869]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-24T00:21:41.623 INFO:journalctl@ceph.mds.f.smithi193.stdout:Mar 24 00:21:41 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-f[71869]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7fdfa0e3dd20] 2024-03-24T00:21:41.623 INFO:journalctl@ceph.mds.f.smithi193.stdout:Mar 24 00:21:41 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-f[71869]: 2: gsignal() 2024-03-24T00:21:41.623 INFO:journalctl@ceph.mds.f.smithi193.stdout:Mar 24 00:21:41 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-f[71869]: 3: abort() 2024-03-24T00:21:41.623 INFO:journalctl@ceph.mds.f.smithi193.stdout:Mar 24 00:21:41 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-f[71869]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x18f) [0x7fdfa20b64fb] 2024-03-24T00:21:41.623 INFO:journalctl@ceph.mds.f.smithi193.stdout:Mar 24 00:21:41 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-f[71869]: 5: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7fdfa20b6667] 2024-03-24T00:21:41.623 INFO:journalctl@ceph.mds.f.smithi193.stdout:Mar 24 00:21:41 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-f[71869]: 6: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x55bcda43f223] 2024-03-24T00:21:41.623 INFO:journalctl@ceph.mds.f.smithi193.stdout:Mar 24 00:21:41 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-f[71869]: 7: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x55bcda2b95d6] 2024-03-24T00:21:41.623 INFO:journalctl@ceph.mds.f.smithi193.stdout:Mar 24 00:21:41 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-f[71869]: 8: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x55bcda2b9da1] 2024-03-24T00:21:41.623 INFO:journalctl@ceph.mds.f.smithi193.stdout:Mar 24 00:21:41 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-f[71869]: 9: (Server::handle_client_rename(boost::intrusive_ptr const&)+0x259d) [0x55bcda2f697d] 2024-03-24T00:21:41.623 INFO:journalctl@ceph.mds.f.smithi193.stdout:Mar 24 00:21:41 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-f[71869]: 10: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x55bcda30c233] 2024-03-24T00:21:41.623 INFO:journalctl@ceph.mds.f.smithi193.stdout:Mar 24 00:21:41 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-f[71869]: 11: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x55bcda30e197] 2024-03-24T00:21:41.624 INFO:journalctl@ceph.mds.f.smithi193.stdout:Mar 24 00:21:41 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-f[71869]: 12: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x55bcda20d11b] 2024-03-24T00:21:41.624 INFO:journalctl@ceph.mds.f.smithi193.stdout:Mar 24 00:21:41 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-f[71869]: 13: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x55bcda20d6bc] 2024-03-24T00:21:41.624 INFO:journalctl@ceph.mds.f.smithi193.stdout:Mar 24 00:21:41 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-f[71869]: 14: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x55bcda1f71ff] 2024-03-24T00:21:41.624 INFO:journalctl@ceph.mds.f.smithi193.stdout:Mar 24 00:21:41 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-f[71869]: 15: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7fdfa2348488] 2024-03-24T00:21:41.624 INFO:journalctl@ceph.mds.f.smithi193.stdout:Mar 24 00:21:41 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-f[71869]: 16: (DispatchQueue::entry()+0x51f) [0x7fdfa234560f] 2024-03-24T00:21:41.624 INFO:journalctl@ceph.mds.f.smithi193.stdout:Mar 24 00:21:41 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-f[71869]: 17: (DispatchQueue::DispatchThread::entry()+0x11) [0x7fdfa2418e81] 2024-03-24T00:21:41.624 INFO:journalctl@ceph.mds.f.smithi193.stdout:Mar 24 00:21:41 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-f[71869]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7fdfa0e331ca] 2024-03-24T00:21:41.624 INFO:journalctl@ceph.mds.f.smithi193.stdout:Mar 24 00:21:41 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-f[71869]: 19: clone() 2024-03-24T00:21:41.624 INFO:journalctl@ceph.mds.f.smithi193.stdout:Mar 24 00:21:41 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-f[71869]: 2024-03-24T00:21:41.614+0000 7fdf97a8c700 -1 *** Caught signal (Aborted) ** 2024-03-24T00:21:41.624 INFO:journalctl@ceph.mds.f.smithi193.stdout:Mar 24 00:21:41 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-f[71869]: in thread 7fdf97a8c700 thread_name:ms_dispatch 2024-03-24T00:21:41.624 INFO:journalctl@ceph.mds.f.smithi193.stdout:Mar 24 00:21:41 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-f[71869]: 2024-03-24T00:21:41.624 INFO:journalctl@ceph.mds.f.smithi193.stdout:Mar 24 00:21:41 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-f[71869]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-24T00:21:41.624 INFO:journalctl@ceph.mds.f.smithi193.stdout:Mar 24 00:21:41 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-f[71869]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7fdfa0e3dd20] 2024-03-24T00:21:41.624 INFO:journalctl@ceph.mds.f.smithi193.stdout:Mar 24 00:21:41 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-f[71869]: 2: gsignal() 2024-03-24T00:21:41.624 INFO:journalctl@ceph.mds.f.smithi193.stdout:Mar 24 00:21:41 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-f[71869]: 3: abort() 2024-03-24T00:21:41.624 INFO:journalctl@ceph.mds.f.smithi193.stdout:Mar 24 00:21:41 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-f[71869]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x18f) [0x7fdfa20b64fb] 2024-03-24T00:21:41.625 INFO:journalctl@ceph.mds.f.smithi193.stdout:Mar 24 00:21:41 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-f[71869]: 5: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7fdfa20b6667] 2024-03-24T00:21:41.625 INFO:journalctl@ceph.mds.f.smithi193.stdout:Mar 24 00:21:41 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-f[71869]: 6: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x55bcda43f223] 2024-03-24T00:21:41.625 INFO:journalctl@ceph.mds.f.smithi193.stdout:Mar 24 00:21:41 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-f[71869]: 7: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x55bcda2b95d6] 2024-03-24T00:21:41.625 INFO:journalctl@ceph.mds.f.smithi193.stdout:Mar 24 00:21:41 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-f[71869]: 8: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x55bcda2b9da1] 2024-03-24T00:21:41.625 INFO:journalctl@ceph.mds.f.smithi193.stdout:Mar 24 00:21:41 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-f[71869]: 9: (Server::handle_client_rename(boost::intrusive_ptr const&)+0x259d) [0x55bcda2f697d] 2024-03-24T00:21:41.625 INFO:journalctl@ceph.mds.f.smithi193.stdout:Mar 24 00:21:41 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-f[71869]: 10: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x55bcda30c233] 2024-03-24T00:21:41.625 INFO:journalctl@ceph.mds.f.smithi193.stdout:Mar 24 00:21:41 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-f[71869]: 11: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x55bcda30e197] 2024-03-24T00:21:41.625 INFO:journalctl@ceph.mds.f.smithi193.stdout:Mar 24 00:21:41 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-f[71869]: 12: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x55bcda20d11b] 2024-03-24T00:21:41.625 INFO:journalctl@ceph.mds.f.smithi193.stdout:Mar 24 00:21:41 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-f[71869]: 13: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x55bcda20d6bc] 2024-03-24T00:21:41.625 INFO:journalctl@ceph.mds.f.smithi193.stdout:Mar 24 00:21:41 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-f[71869]: 14: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x55bcda1f71ff] 2024-03-24T00:21:41.626 INFO:journalctl@ceph.mds.f.smithi193.stdout:Mar 24 00:21:41 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-f[71869]: 15: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7fdfa2348488] 2024-03-24T00:21:41.626 INFO:journalctl@ceph.mds.f.smithi193.stdout:Mar 24 00:21:41 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-f[71869]: 16: (DispatchQueue::entry()+0x51f) [0x7fdfa234560f] 2024-03-24T00:21:41.626 INFO:journalctl@ceph.mds.f.smithi193.stdout:Mar 24 00:21:41 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-f[71869]: 17: (DispatchQueue::DispatchThread::entry()+0x11) [0x7fdfa2418e81] 2024-03-24T00:21:41.626 INFO:journalctl@ceph.mds.f.smithi193.stdout:Mar 24 00:21:41 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-f[71869]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7fdfa0e331ca] 2024-03-24T00:21:41.626 INFO:journalctl@ceph.mds.f.smithi193.stdout:Mar 24 00:21:41 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-f[71869]: 19: clone() 2024-03-24T00:21:41.626 INFO:journalctl@ceph.mds.f.smithi193.stdout:Mar 24 00:21:41 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-f[71869]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-03-24T00:21:41.626 INFO:journalctl@ceph.mds.f.smithi193.stdout:Mar 24 00:21:41 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-f[71869]: 2024-03-24T00:21:41.695 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:21:41 smithi118 ceph-mon[31535]: mds.? [v2:172.21.15.118:6838/979323377,v1:172.21.15.118:6839/979323377] up:reconnect 2024-03-24T00:21:41.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:21:41 smithi118 ceph-mon[31535]: mds.? [v2:172.21.15.193:6834/2814329894,v1:172.21.15.193:6835/2814329894] up:reconnect 2024-03-24T00:21:41.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:21:41 smithi118 ceph-mon[31535]: fsmap cephfs:5/5 3 up:standby-replay 4 up:standby 2 up:reconnect 3 up:active 2024-03-24T00:21:41.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:21:41 smithi118 ceph-mon[31535]: reconnect by client.15159 192.168.0.1:0/3463739313 after 0.0289995 2024-03-24T00:21:41.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:21:41 smithi118 ceph-mon[31535]: reconnect by client.15159 192.168.0.1:0/3463739313 after 0.0369991 2024-03-24T00:21:41.696 INFO:journalctl@ceph.mds.g.smithi118.stdout:Mar 24 00:21:41 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-g[94305]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: In function 'void Locker::set_xlocks_done(MutationImpl*, bool)' thread 7f57d9600700 time 2024-03-24T00:21:41.569478+0000 2024-03-24T00:21:41.697 INFO:journalctl@ceph.mds.g.smithi118.stdout:Mar 24 00:21:41 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-g[94305]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: 626: FAILED ceph_assert(obj->is_auth()) 2024-03-24T00:21:41.697 INFO:journalctl@ceph.mds.g.smithi118.stdout:Mar 24 00:21:41 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-g[94305]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-24T00:21:41.697 INFO:journalctl@ceph.mds.g.smithi118.stdout:Mar 24 00:21:41 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-g[94305]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x135) [0x7f57e4fe24a1] 2024-03-24T00:21:41.697 INFO:journalctl@ceph.mds.g.smithi118.stdout:Mar 24 00:21:41 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-g[94305]: 2: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7f57e4fe2667] 2024-03-24T00:21:41.697 INFO:journalctl@ceph.mds.g.smithi118.stdout:Mar 24 00:21:41 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-g[94305]: 3: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x5567f7189223] 2024-03-24T00:21:41.697 INFO:journalctl@ceph.mds.g.smithi118.stdout:Mar 24 00:21:41 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-g[94305]: 4: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x5567f70035d6] 2024-03-24T00:21:41.697 INFO:journalctl@ceph.mds.g.smithi118.stdout:Mar 24 00:21:41 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-g[94305]: 5: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x5567f7003da1] 2024-03-24T00:21:41.697 INFO:journalctl@ceph.mds.g.smithi118.stdout:Mar 24 00:21:41 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-g[94305]: 6: (Server::handle_client_rename(boost::intrusive_ptr const&)+0x259d) [0x5567f704097d] 2024-03-24T00:21:41.697 INFO:journalctl@ceph.mds.g.smithi118.stdout:Mar 24 00:21:41 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-g[94305]: 7: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x5567f7056233] 2024-03-24T00:21:41.697 INFO:journalctl@ceph.mds.g.smithi118.stdout:Mar 24 00:21:41 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-g[94305]: 8: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x5567f7058197] 2024-03-24T00:21:41.697 INFO:journalctl@ceph.mds.g.smithi118.stdout:Mar 24 00:21:41 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-g[94305]: 9: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x5567f6f5711b] 2024-03-24T00:21:41.697 INFO:journalctl@ceph.mds.g.smithi118.stdout:Mar 24 00:21:41 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-g[94305]: 10: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x5567f6f576bc] 2024-03-24T00:21:41.697 INFO:journalctl@ceph.mds.g.smithi118.stdout:Mar 24 00:21:41 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-g[94305]: 11: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x5567f6f411ff] 2024-03-24T00:21:41.697 INFO:journalctl@ceph.mds.g.smithi118.stdout:Mar 24 00:21:41 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-g[94305]: 12: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7f57e5274488] 2024-03-24T00:21:41.698 INFO:journalctl@ceph.mds.g.smithi118.stdout:Mar 24 00:21:41 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-g[94305]: 13: (DispatchQueue::entry()+0x51f) [0x7f57e527160f] 2024-03-24T00:21:41.698 INFO:journalctl@ceph.mds.g.smithi118.stdout:Mar 24 00:21:41 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-g[94305]: 14: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f57e5344e81] 2024-03-24T00:21:41.698 INFO:journalctl@ceph.mds.g.smithi118.stdout:Mar 24 00:21:41 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-g[94305]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7f57e3d5f1ca] 2024-03-24T00:21:41.698 INFO:journalctl@ceph.mds.g.smithi118.stdout:Mar 24 00:21:41 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-g[94305]: 16: clone() 2024-03-24T00:21:41.698 INFO:journalctl@ceph.mds.g.smithi118.stdout:Mar 24 00:21:41 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-g[94305]: 2024-03-24T00:21:41.570+0000 7f57d9600700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: In function 'void Locker::set_xlocks_done(MutationImpl*, bool)' thread 7f57d9600700 time 2024-03-24T00:21:41.569478+0000 2024-03-24T00:21:41.698 INFO:journalctl@ceph.mds.g.smithi118.stdout:Mar 24 00:21:41 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-g[94305]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: 626: FAILED ceph_assert(obj->is_auth()) 2024-03-24T00:21:41.698 INFO:journalctl@ceph.mds.g.smithi118.stdout:Mar 24 00:21:41 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-g[94305]: 2024-03-24T00:21:41.698 INFO:journalctl@ceph.mds.g.smithi118.stdout:Mar 24 00:21:41 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-g[94305]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-24T00:21:41.698 INFO:journalctl@ceph.mds.g.smithi118.stdout:Mar 24 00:21:41 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-g[94305]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x135) [0x7f57e4fe24a1] 2024-03-24T00:21:41.698 INFO:journalctl@ceph.mds.g.smithi118.stdout:Mar 24 00:21:41 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-g[94305]: 2: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7f57e4fe2667] 2024-03-24T00:21:41.698 INFO:journalctl@ceph.mds.g.smithi118.stdout:Mar 24 00:21:41 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-g[94305]: 3: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x5567f7189223] 2024-03-24T00:21:41.698 INFO:journalctl@ceph.mds.g.smithi118.stdout:Mar 24 00:21:41 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-g[94305]: 4: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x5567f70035d6] 2024-03-24T00:21:41.698 INFO:journalctl@ceph.mds.g.smithi118.stdout:Mar 24 00:21:41 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-g[94305]: 5: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x5567f7003da1] 2024-03-24T00:21:41.699 INFO:journalctl@ceph.mds.g.smithi118.stdout:Mar 24 00:21:41 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-g[94305]: 6: (Server::handle_client_rename(boost::intrusive_ptr const&)+0x259d) [0x5567f704097d] 2024-03-24T00:21:41.699 INFO:journalctl@ceph.mds.g.smithi118.stdout:Mar 24 00:21:41 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-g[94305]: 7: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x5567f7056233] 2024-03-24T00:21:41.699 INFO:journalctl@ceph.mds.g.smithi118.stdout:Mar 24 00:21:41 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-g[94305]: 8: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x5567f7058197] 2024-03-24T00:21:41.699 INFO:journalctl@ceph.mds.g.smithi118.stdout:Mar 24 00:21:41 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-g[94305]: 9: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x5567f6f5711b] 2024-03-24T00:21:41.699 INFO:journalctl@ceph.mds.g.smithi118.stdout:Mar 24 00:21:41 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-g[94305]: 10: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x5567f6f576bc] 2024-03-24T00:21:41.699 INFO:journalctl@ceph.mds.g.smithi118.stdout:Mar 24 00:21:41 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-g[94305]: 11: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x5567f6f411ff] 2024-03-24T00:21:41.699 INFO:journalctl@ceph.mds.g.smithi118.stdout:Mar 24 00:21:41 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-g[94305]: 12: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7f57e5274488] 2024-03-24T00:21:41.699 INFO:journalctl@ceph.mds.g.smithi118.stdout:Mar 24 00:21:41 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-g[94305]: 13: (DispatchQueue::entry()+0x51f) [0x7f57e527160f] 2024-03-24T00:21:41.699 INFO:journalctl@ceph.mds.g.smithi118.stdout:Mar 24 00:21:41 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-g[94305]: 14: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f57e5344e81] 2024-03-24T00:21:41.699 INFO:journalctl@ceph.mds.g.smithi118.stdout:Mar 24 00:21:41 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-g[94305]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7f57e3d5f1ca] 2024-03-24T00:21:41.699 INFO:journalctl@ceph.mds.g.smithi118.stdout:Mar 24 00:21:41 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-g[94305]: 16: clone() 2024-03-24T00:21:41.699 INFO:journalctl@ceph.mds.g.smithi118.stdout:Mar 24 00:21:41 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-g[94305]: 2024-03-24T00:21:41.699 INFO:journalctl@ceph.mds.g.smithi118.stdout:Mar 24 00:21:41 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-g[94305]: *** Caught signal (Aborted) ** 2024-03-24T00:21:41.699 INFO:journalctl@ceph.mds.g.smithi118.stdout:Mar 24 00:21:41 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-g[94305]: in thread 7f57d9600700 thread_name:ms_dispatch 2024-03-24T00:21:41.699 INFO:journalctl@ceph.mds.g.smithi118.stdout:Mar 24 00:21:41 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-g[94305]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-24T00:21:41.700 INFO:journalctl@ceph.mds.g.smithi118.stdout:Mar 24 00:21:41 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-g[94305]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f57e3d69d20] 2024-03-24T00:21:41.700 INFO:journalctl@ceph.mds.g.smithi118.stdout:Mar 24 00:21:41 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-g[94305]: 2: gsignal() 2024-03-24T00:21:41.701 INFO:journalctl@ceph.mds.g.smithi118.stdout:Mar 24 00:21:41 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-g[94305]: 3: abort() 2024-03-24T00:21:41.701 INFO:journalctl@ceph.mds.g.smithi118.stdout:Mar 24 00:21:41 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-g[94305]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x18f) [0x7f57e4fe24fb] 2024-03-24T00:21:41.701 INFO:journalctl@ceph.mds.g.smithi118.stdout:Mar 24 00:21:41 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-g[94305]: 5: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7f57e4fe2667] 2024-03-24T00:21:41.701 INFO:journalctl@ceph.mds.g.smithi118.stdout:Mar 24 00:21:41 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-g[94305]: 6: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x5567f7189223] 2024-03-24T00:21:41.701 INFO:journalctl@ceph.mds.g.smithi118.stdout:Mar 24 00:21:41 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-g[94305]: 7: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x5567f70035d6] 2024-03-24T00:21:41.701 INFO:journalctl@ceph.mds.g.smithi118.stdout:Mar 24 00:21:41 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-g[94305]: 8: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x5567f7003da1] 2024-03-24T00:21:41.701 INFO:journalctl@ceph.mds.g.smithi118.stdout:Mar 24 00:21:41 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-g[94305]: 9: (Server::handle_client_rename(boost::intrusive_ptr const&)+0x259d) [0x5567f704097d] 2024-03-24T00:21:41.701 INFO:journalctl@ceph.mds.g.smithi118.stdout:Mar 24 00:21:41 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-g[94305]: 10: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x5567f7056233] 2024-03-24T00:21:41.701 INFO:journalctl@ceph.mds.g.smithi118.stdout:Mar 24 00:21:41 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-g[94305]: 11: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x5567f7058197] 2024-03-24T00:21:41.701 INFO:journalctl@ceph.mds.g.smithi118.stdout:Mar 24 00:21:41 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-g[94305]: 12: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x5567f6f5711b] 2024-03-24T00:21:41.701 INFO:journalctl@ceph.mds.g.smithi118.stdout:Mar 24 00:21:41 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-g[94305]: 13: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x5567f6f576bc] 2024-03-24T00:21:41.701 INFO:journalctl@ceph.mds.g.smithi118.stdout:Mar 24 00:21:41 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-g[94305]: 14: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x5567f6f411ff] 2024-03-24T00:21:41.701 INFO:journalctl@ceph.mds.g.smithi118.stdout:Mar 24 00:21:41 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-g[94305]: 15: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7f57e5274488] 2024-03-24T00:21:41.702 INFO:journalctl@ceph.mds.g.smithi118.stdout:Mar 24 00:21:41 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-g[94305]: 16: (DispatchQueue::entry()+0x51f) [0x7f57e527160f] 2024-03-24T00:21:41.702 INFO:journalctl@ceph.mds.g.smithi118.stdout:Mar 24 00:21:41 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-g[94305]: 17: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f57e5344e81] 2024-03-24T00:21:41.702 INFO:journalctl@ceph.mds.g.smithi118.stdout:Mar 24 00:21:41 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-g[94305]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7f57e3d5f1ca] 2024-03-24T00:21:41.702 INFO:journalctl@ceph.mds.g.smithi118.stdout:Mar 24 00:21:41 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-g[94305]: 19: clone() 2024-03-24T00:21:41.702 INFO:journalctl@ceph.mds.g.smithi118.stdout:Mar 24 00:21:41 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-g[94305]: 2024-03-24T00:21:41.573+0000 7f57d9600700 -1 *** Caught signal (Aborted) ** 2024-03-24T00:21:41.702 INFO:journalctl@ceph.mds.g.smithi118.stdout:Mar 24 00:21:41 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-g[94305]: in thread 7f57d9600700 thread_name:ms_dispatch 2024-03-24T00:21:41.702 INFO:journalctl@ceph.mds.g.smithi118.stdout:Mar 24 00:21:41 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-g[94305]: 2024-03-24T00:21:41.702 INFO:journalctl@ceph.mds.g.smithi118.stdout:Mar 24 00:21:41 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-g[94305]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-24T00:21:41.702 INFO:journalctl@ceph.mds.g.smithi118.stdout:Mar 24 00:21:41 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-g[94305]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f57e3d69d20] 2024-03-24T00:21:41.702 INFO:journalctl@ceph.mds.g.smithi118.stdout:Mar 24 00:21:41 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-g[94305]: 2: gsignal() 2024-03-24T00:21:41.702 INFO:journalctl@ceph.mds.g.smithi118.stdout:Mar 24 00:21:41 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-g[94305]: 3: abort() 2024-03-24T00:21:41.702 INFO:journalctl@ceph.mds.g.smithi118.stdout:Mar 24 00:21:41 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-g[94305]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x18f) [0x7f57e4fe24fb] 2024-03-24T00:21:41.702 INFO:journalctl@ceph.mds.g.smithi118.stdout:Mar 24 00:21:41 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-g[94305]: 5: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7f57e4fe2667] 2024-03-24T00:21:41.702 INFO:journalctl@ceph.mds.g.smithi118.stdout:Mar 24 00:21:41 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-g[94305]: 6: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x5567f7189223] 2024-03-24T00:21:41.703 INFO:journalctl@ceph.mds.g.smithi118.stdout:Mar 24 00:21:41 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-g[94305]: 7: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x5567f70035d6] 2024-03-24T00:21:41.703 INFO:journalctl@ceph.mds.g.smithi118.stdout:Mar 24 00:21:41 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-g[94305]: 8: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x5567f7003da1] 2024-03-24T00:21:41.703 INFO:journalctl@ceph.mds.g.smithi118.stdout:Mar 24 00:21:41 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-g[94305]: 9: (Server::handle_client_rename(boost::intrusive_ptr const&)+0x259d) [0x5567f704097d] 2024-03-24T00:21:41.703 INFO:journalctl@ceph.mds.g.smithi118.stdout:Mar 24 00:21:41 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-g[94305]: 10: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x5567f7056233] 2024-03-24T00:21:41.703 INFO:journalctl@ceph.mds.g.smithi118.stdout:Mar 24 00:21:41 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-g[94305]: 11: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x5567f7058197] 2024-03-24T00:21:41.703 INFO:journalctl@ceph.mds.g.smithi118.stdout:Mar 24 00:21:41 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-g[94305]: 12: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x5567f6f5711b] 2024-03-24T00:21:41.703 INFO:journalctl@ceph.mds.g.smithi118.stdout:Mar 24 00:21:41 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-g[94305]: 13: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x5567f6f576bc] 2024-03-24T00:21:41.703 INFO:journalctl@ceph.mds.g.smithi118.stdout:Mar 24 00:21:41 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-g[94305]: 14: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x5567f6f411ff] 2024-03-24T00:21:41.703 INFO:journalctl@ceph.mds.g.smithi118.stdout:Mar 24 00:21:41 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-g[94305]: 15: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7f57e5274488] 2024-03-24T00:21:41.703 INFO:journalctl@ceph.mds.g.smithi118.stdout:Mar 24 00:21:41 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-g[94305]: 16: (DispatchQueue::entry()+0x51f) [0x7f57e527160f] 2024-03-24T00:21:41.703 INFO:journalctl@ceph.mds.g.smithi118.stdout:Mar 24 00:21:41 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-g[94305]: 17: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f57e5344e81] 2024-03-24T00:21:41.703 INFO:journalctl@ceph.mds.g.smithi118.stdout:Mar 24 00:21:41 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-g[94305]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7f57e3d5f1ca] 2024-03-24T00:21:41.703 INFO:journalctl@ceph.mds.g.smithi118.stdout:Mar 24 00:21:41 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-g[94305]: 19: clone() 2024-03-24T00:21:41.703 INFO:journalctl@ceph.mds.g.smithi118.stdout:Mar 24 00:21:41 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-g[94305]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-03-24T00:21:41.703 INFO:journalctl@ceph.mds.g.smithi118.stdout:Mar 24 00:21:41 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-g[94305]: 2024-03-24T00:21:41.704 INFO:journalctl@ceph.mds.g.smithi118.stdout:Mar 24 00:21:41 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-g[94305]: -1> 2024-03-24T00:21:41.570+0000 7f57d9600700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: In function 'void Locker::set_xlocks_done(MutationImpl*, bool)' thread 7f57d9600700 time 2024-03-24T00:21:41.569478+0000 2024-03-24T00:21:41.704 INFO:journalctl@ceph.mds.g.smithi118.stdout:Mar 24 00:21:41 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-g[94305]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: 626: FAILED ceph_assert(obj->is_auth()) 2024-03-24T00:21:41.704 INFO:journalctl@ceph.mds.g.smithi118.stdout:Mar 24 00:21:41 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-g[94305]: 2024-03-24T00:21:41.704 INFO:journalctl@ceph.mds.g.smithi118.stdout:Mar 24 00:21:41 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-g[94305]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-24T00:21:41.704 INFO:journalctl@ceph.mds.g.smithi118.stdout:Mar 24 00:21:41 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-g[94305]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x135) [0x7f57e4fe24a1] 2024-03-24T00:21:41.704 INFO:journalctl@ceph.mds.g.smithi118.stdout:Mar 24 00:21:41 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-g[94305]: 2: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7f57e4fe2667] 2024-03-24T00:21:41.704 INFO:journalctl@ceph.mds.g.smithi118.stdout:Mar 24 00:21:41 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-g[94305]: 3: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x5567f7189223] 2024-03-24T00:21:41.704 INFO:journalctl@ceph.mds.g.smithi118.stdout:Mar 24 00:21:41 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-g[94305]: 4: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x5567f70035d6] 2024-03-24T00:21:41.704 INFO:journalctl@ceph.mds.g.smithi118.stdout:Mar 24 00:21:41 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-g[94305]: 5: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x5567f7003da1] 2024-03-24T00:21:41.704 INFO:journalctl@ceph.mds.g.smithi118.stdout:Mar 24 00:21:41 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-g[94305]: 6: (Server::handle_client_rename(boost::intrusive_ptr const&)+0x259d) [0x5567f704097d] 2024-03-24T00:21:41.704 INFO:journalctl@ceph.mds.g.smithi118.stdout:Mar 24 00:21:41 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-g[94305]: 7: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x5567f7056233] 2024-03-24T00:21:41.704 INFO:journalctl@ceph.mds.g.smithi118.stdout:Mar 24 00:21:41 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-g[94305]: 8: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x5567f7058197] 2024-03-24T00:21:41.704 INFO:journalctl@ceph.mds.g.smithi118.stdout:Mar 24 00:21:41 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-g[94305]: 9: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x5567f6f5711b] 2024-03-24T00:21:41.704 INFO:journalctl@ceph.mds.g.smithi118.stdout:Mar 24 00:21:41 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-g[94305]: 10: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x5567f6f576bc] 2024-03-24T00:21:41.705 INFO:journalctl@ceph.mds.g.smithi118.stdout:Mar 24 00:21:41 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-g[94305]: 11: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x5567f6f411ff] 2024-03-24T00:21:41.705 INFO:journalctl@ceph.mds.g.smithi118.stdout:Mar 24 00:21:41 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-g[94305]: 12: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7f57e5274488] 2024-03-24T00:21:41.705 INFO:journalctl@ceph.mds.g.smithi118.stdout:Mar 24 00:21:41 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-g[94305]: 13: (DispatchQueue::entry()+0x51f) [0x7f57e527160f] 2024-03-24T00:21:41.705 INFO:journalctl@ceph.mds.g.smithi118.stdout:Mar 24 00:21:41 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-g[94305]: 14: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f57e5344e81] 2024-03-24T00:21:41.705 INFO:journalctl@ceph.mds.g.smithi118.stdout:Mar 24 00:21:41 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-g[94305]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7f57e3d5f1ca] 2024-03-24T00:21:41.705 INFO:journalctl@ceph.mds.g.smithi118.stdout:Mar 24 00:21:41 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-g[94305]: 16: clone() 2024-03-24T00:21:41.705 INFO:journalctl@ceph.mds.g.smithi118.stdout:Mar 24 00:21:41 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-g[94305]: 2024-03-24T00:21:41.705 INFO:journalctl@ceph.mds.g.smithi118.stdout:Mar 24 00:21:41 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-g[94305]: 0> 2024-03-24T00:21:41.573+0000 7f57d9600700 -1 *** Caught signal (Aborted) ** 2024-03-24T00:21:41.705 INFO:journalctl@ceph.mds.g.smithi118.stdout:Mar 24 00:21:41 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-g[94305]: in thread 7f57d9600700 thread_name:ms_dispatch 2024-03-24T00:21:41.705 INFO:journalctl@ceph.mds.g.smithi118.stdout:Mar 24 00:21:41 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-g[94305]: 2024-03-24T00:21:41.705 INFO:journalctl@ceph.mds.g.smithi118.stdout:Mar 24 00:21:41 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-g[94305]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-24T00:21:41.705 INFO:journalctl@ceph.mds.g.smithi118.stdout:Mar 24 00:21:41 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-g[94305]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f57e3d69d20] 2024-03-24T00:21:41.705 INFO:journalctl@ceph.mds.g.smithi118.stdout:Mar 24 00:21:41 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-g[94305]: 2: gsignal() 2024-03-24T00:21:41.705 INFO:journalctl@ceph.mds.g.smithi118.stdout:Mar 24 00:21:41 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-g[94305]: 3: abort() 2024-03-24T00:21:41.706 INFO:journalctl@ceph.mds.g.smithi118.stdout:Mar 24 00:21:41 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-g[94305]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x18f) [0x7f57e4fe24fb] 2024-03-24T00:21:41.706 INFO:journalctl@ceph.mds.g.smithi118.stdout:Mar 24 00:21:41 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-g[94305]: 5: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7f57e4fe2667] 2024-03-24T00:21:41.706 INFO:journalctl@ceph.mds.g.smithi118.stdout:Mar 24 00:21:41 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-g[94305]: 6: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x5567f7189223] 2024-03-24T00:21:41.706 INFO:journalctl@ceph.mds.g.smithi118.stdout:Mar 24 00:21:41 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-g[94305]: 7: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x5567f70035d6] 2024-03-24T00:21:41.706 INFO:journalctl@ceph.mds.g.smithi118.stdout:Mar 24 00:21:41 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-g[94305]: 8: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x5567f7003da1] 2024-03-24T00:21:41.706 INFO:journalctl@ceph.mds.g.smithi118.stdout:Mar 24 00:21:41 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-g[94305]: 9: (Server::handle_client_rename(boost::intrusive_ptr const&)+0x259d) [0x5567f704097d] 2024-03-24T00:21:41.706 INFO:journalctl@ceph.mds.g.smithi118.stdout:Mar 24 00:21:41 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-g[94305]: 10: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x5567f7056233] 2024-03-24T00:21:41.706 INFO:journalctl@ceph.mds.g.smithi118.stdout:Mar 24 00:21:41 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-g[94305]: 11: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x5567f7058197] 2024-03-24T00:21:41.706 INFO:journalctl@ceph.mds.g.smithi118.stdout:Mar 24 00:21:41 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-g[94305]: 12: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x5567f6f5711b] 2024-03-24T00:21:41.706 INFO:journalctl@ceph.mds.g.smithi118.stdout:Mar 24 00:21:41 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-g[94305]: 13: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x5567f6f576bc] 2024-03-24T00:21:41.706 INFO:journalctl@ceph.mds.g.smithi118.stdout:Mar 24 00:21:41 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-g[94305]: 14: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x5567f6f411ff] 2024-03-24T00:21:41.706 INFO:journalctl@ceph.mds.g.smithi118.stdout:Mar 24 00:21:41 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-g[94305]: 15: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7f57e5274488] 2024-03-24T00:21:41.706 INFO:journalctl@ceph.mds.g.smithi118.stdout:Mar 24 00:21:41 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-g[94305]: 16: (DispatchQueue::entry()+0x51f) [0x7f57e527160f] 2024-03-24T00:21:41.706 INFO:journalctl@ceph.mds.g.smithi118.stdout:Mar 24 00:21:41 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-g[94305]: 17: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f57e5344e81] 2024-03-24T00:21:41.707 INFO:journalctl@ceph.mds.g.smithi118.stdout:Mar 24 00:21:41 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-g[94305]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7f57e3d5f1ca] 2024-03-24T00:21:41.707 INFO:journalctl@ceph.mds.g.smithi118.stdout:Mar 24 00:21:41 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-g[94305]: 19: clone() 2024-03-24T00:21:41.707 INFO:journalctl@ceph.mds.g.smithi118.stdout:Mar 24 00:21:41 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-g[94305]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-03-24T00:21:41.707 INFO:journalctl@ceph.mds.g.smithi118.stdout:Mar 24 00:21:41 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-g[94305]: 2024-03-24T00:21:41.707 INFO:journalctl@ceph.mds.g.smithi118.stdout:Mar 24 00:21:41 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-g[94305]: -1> 2024-03-24T00:21:41.570+0000 7f57d9600700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: In function 'void Locker::set_xlocks_done(MutationImpl*, bool)' thread 7f57d9600700 time 2024-03-24T00:21:41.569478+0000 2024-03-24T00:21:41.707 INFO:journalctl@ceph.mds.g.smithi118.stdout:Mar 24 00:21:41 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-g[94305]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: 626: FAILED ceph_assert(obj->is_auth()) 2024-03-24T00:21:41.707 INFO:journalctl@ceph.mds.g.smithi118.stdout:Mar 24 00:21:41 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-g[94305]: 2024-03-24T00:21:41.707 INFO:journalctl@ceph.mds.g.smithi118.stdout:Mar 24 00:21:41 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-g[94305]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-24T00:21:41.707 INFO:journalctl@ceph.mds.g.smithi118.stdout:Mar 24 00:21:41 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-g[94305]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x135) [0x7f57e4fe24a1] 2024-03-24T00:21:41.707 INFO:journalctl@ceph.mds.g.smithi118.stdout:Mar 24 00:21:41 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-g[94305]: 2: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7f57e4fe2667] 2024-03-24T00:21:41.707 INFO:journalctl@ceph.mds.g.smithi118.stdout:Mar 24 00:21:41 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-g[94305]: 3: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x5567f7189223] 2024-03-24T00:21:41.707 INFO:journalctl@ceph.mds.g.smithi118.stdout:Mar 24 00:21:41 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-g[94305]: 4: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x5567f70035d6] 2024-03-24T00:21:41.707 INFO:journalctl@ceph.mds.g.smithi118.stdout:Mar 24 00:21:41 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-g[94305]: 5: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x5567f7003da1] 2024-03-24T00:21:41.707 INFO:journalctl@ceph.mds.g.smithi118.stdout:Mar 24 00:21:41 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-g[94305]: 6: (Server::handle_client_rename(boost::intrusive_ptr const&)+0x259d) [0x5567f704097d] 2024-03-24T00:21:41.707 INFO:journalctl@ceph.mds.g.smithi118.stdout:Mar 24 00:21:41 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-g[94305]: 7: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x5567f7056233] 2024-03-24T00:21:41.708 INFO:journalctl@ceph.mds.g.smithi118.stdout:Mar 24 00:21:41 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-g[94305]: 8: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x5567f7058197] 2024-03-24T00:21:41.708 INFO:journalctl@ceph.mds.g.smithi118.stdout:Mar 24 00:21:41 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-g[94305]: 9: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x5567f6f5711b] 2024-03-24T00:21:41.708 INFO:journalctl@ceph.mds.g.smithi118.stdout:Mar 24 00:21:41 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-g[94305]: 10: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x5567f6f576bc] 2024-03-24T00:21:41.708 INFO:journalctl@ceph.mds.g.smithi118.stdout:Mar 24 00:21:41 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-g[94305]: 11: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x5567f6f411ff] 2024-03-24T00:21:41.708 INFO:journalctl@ceph.mds.g.smithi118.stdout:Mar 24 00:21:41 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-g[94305]: 12: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7f57e5274488] 2024-03-24T00:21:41.708 INFO:journalctl@ceph.mds.g.smithi118.stdout:Mar 24 00:21:41 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-g[94305]: 13: (DispatchQueue::entry()+0x51f) [0x7f57e527160f] 2024-03-24T00:21:41.708 INFO:journalctl@ceph.mds.g.smithi118.stdout:Mar 24 00:21:41 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-g[94305]: 14: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f57e5344e81] 2024-03-24T00:21:41.708 INFO:journalctl@ceph.mds.g.smithi118.stdout:Mar 24 00:21:41 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-g[94305]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7f57e3d5f1ca] 2024-03-24T00:21:41.708 INFO:journalctl@ceph.mds.g.smithi118.stdout:Mar 24 00:21:41 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-g[94305]: 16: clone() 2024-03-24T00:21:41.708 INFO:journalctl@ceph.mds.g.smithi118.stdout:Mar 24 00:21:41 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-g[94305]: 2024-03-24T00:21:41.708 INFO:journalctl@ceph.mds.g.smithi118.stdout:Mar 24 00:21:41 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-g[94305]: 0> 2024-03-24T00:21:41.573+0000 7f57d9600700 -1 *** Caught signal (Aborted) ** 2024-03-24T00:21:41.708 INFO:journalctl@ceph.mds.g.smithi118.stdout:Mar 24 00:21:41 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-g[94305]: in thread 7f57d9600700 thread_name:ms_dispatch 2024-03-24T00:21:41.708 INFO:journalctl@ceph.mds.g.smithi118.stdout:Mar 24 00:21:41 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-g[94305]: 2024-03-24T00:21:41.708 INFO:journalctl@ceph.mds.g.smithi118.stdout:Mar 24 00:21:41 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-g[94305]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-24T00:21:41.708 INFO:journalctl@ceph.mds.g.smithi118.stdout:Mar 24 00:21:41 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-g[94305]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f57e3d69d20] 2024-03-24T00:21:41.709 INFO:journalctl@ceph.mds.g.smithi118.stdout:Mar 24 00:21:41 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-g[94305]: 2: gsignal() 2024-03-24T00:21:41.709 INFO:journalctl@ceph.mds.g.smithi118.stdout:Mar 24 00:21:41 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-g[94305]: 3: abort() 2024-03-24T00:21:41.709 INFO:journalctl@ceph.mds.g.smithi118.stdout:Mar 24 00:21:41 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-g[94305]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x18f) [0x7f57e4fe24fb] 2024-03-24T00:21:41.709 INFO:journalctl@ceph.mds.g.smithi118.stdout:Mar 24 00:21:41 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-g[94305]: 5: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7f57e4fe2667] 2024-03-24T00:21:41.709 INFO:journalctl@ceph.mds.g.smithi118.stdout:Mar 24 00:21:41 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-g[94305]: 6: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x5567f7189223] 2024-03-24T00:21:41.709 INFO:journalctl@ceph.mds.g.smithi118.stdout:Mar 24 00:21:41 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-g[94305]: 7: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x5567f70035d6] 2024-03-24T00:21:41.709 INFO:journalctl@ceph.mds.g.smithi118.stdout:Mar 24 00:21:41 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-g[94305]: 8: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x5567f7003da1] 2024-03-24T00:21:41.709 INFO:journalctl@ceph.mds.g.smithi118.stdout:Mar 24 00:21:41 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-g[94305]: 9: (Server::handle_client_rename(boost::intrusive_ptr const&)+0x259d) [0x5567f704097d] 2024-03-24T00:21:41.709 INFO:journalctl@ceph.mds.g.smithi118.stdout:Mar 24 00:21:41 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-g[94305]: 10: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x5567f7056233] 2024-03-24T00:21:41.709 INFO:journalctl@ceph.mds.g.smithi118.stdout:Mar 24 00:21:41 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-g[94305]: 11: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x5567f7058197] 2024-03-24T00:21:41.709 INFO:journalctl@ceph.mds.g.smithi118.stdout:Mar 24 00:21:41 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-g[94305]: 12: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x5567f6f5711b] 2024-03-24T00:21:41.709 INFO:journalctl@ceph.mds.g.smithi118.stdout:Mar 24 00:21:41 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-g[94305]: 13: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x5567f6f576bc] 2024-03-24T00:21:41.709 INFO:journalctl@ceph.mds.g.smithi118.stdout:Mar 24 00:21:41 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-g[94305]: 14: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x5567f6f411ff] 2024-03-24T00:21:41.709 INFO:journalctl@ceph.mds.g.smithi118.stdout:Mar 24 00:21:41 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-g[94305]: 15: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7f57e5274488] 2024-03-24T00:21:41.709 INFO:journalctl@ceph.mds.g.smithi118.stdout:Mar 24 00:21:41 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-g[94305]: 16: (DispatchQueue::entry()+0x51f) [0x7f57e527160f] 2024-03-24T00:21:41.710 INFO:journalctl@ceph.mds.g.smithi118.stdout:Mar 24 00:21:41 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-g[94305]: 17: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f57e5344e81] 2024-03-24T00:21:41.710 INFO:journalctl@ceph.mds.g.smithi118.stdout:Mar 24 00:21:41 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-g[94305]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7f57e3d5f1ca] 2024-03-24T00:21:41.710 INFO:journalctl@ceph.mds.g.smithi118.stdout:Mar 24 00:21:41 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-g[94305]: 19: clone() 2024-03-24T00:21:41.710 INFO:journalctl@ceph.mds.g.smithi118.stdout:Mar 24 00:21:41 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-g[94305]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-03-24T00:21:41.710 INFO:journalctl@ceph.mds.g.smithi118.stdout:Mar 24 00:21:41 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-g[94305]: 2024-03-24T00:21:41.710 INFO:journalctl@ceph.mds.g.smithi118.stdout:Mar 24 00:21:41 smithi118 podman[94417]: 2024-03-24 00:21:41.68913556 +0000 UTC m=+0.036517442 container died e5c87fc6a3bf4c109d5e01bdb0b1f592e72fa7f852e0ec79dc7ea1f1054cf96b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-g, org.label-schema.build-date=20240319, io.buildah.version=1.33.5, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.vendor=CentOS, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.schema-version=1.0, GIT_CLEAN=True, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=) 2024-03-24T00:21:41.710 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:21:41 smithi189 ceph-mon[36187]: mds.? [v2:172.21.15.118:6838/979323377,v1:172.21.15.118:6839/979323377] up:reconnect 2024-03-24T00:21:41.710 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:21:41 smithi189 ceph-mon[36187]: mds.? [v2:172.21.15.193:6834/2814329894,v1:172.21.15.193:6835/2814329894] up:reconnect 2024-03-24T00:21:41.710 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:21:41 smithi189 ceph-mon[36187]: fsmap cephfs:5/5 3 up:standby-replay 4 up:standby 2 up:reconnect 3 up:active 2024-03-24T00:21:41.711 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:21:41 smithi189 ceph-mon[36187]: reconnect by client.15159 192.168.0.1:0/3463739313 after 0.0289995 2024-03-24T00:21:41.711 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:21:41 smithi189 ceph-mon[36187]: reconnect by client.15159 192.168.0.1:0/3463739313 after 0.0369991 2024-03-24T00:21:41.711 INFO:journalctl@ceph.mds.f.smithi193.stdout:Mar 24 00:21:41 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-f[71869]: -1> 2024-03-24T00:21:41.613+0000 7fdf97a8c700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: In function 'void Locker::set_xlocks_done(MutationImpl*, bool)' thread 7fdf97a8c700 time 2024-03-24T00:21:41.612694+0000 2024-03-24T00:21:41.711 INFO:journalctl@ceph.mds.f.smithi193.stdout:Mar 24 00:21:41 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-f[71869]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: 626: FAILED ceph_assert(obj->is_auth()) 2024-03-24T00:21:41.711 INFO:journalctl@ceph.mds.f.smithi193.stdout:Mar 24 00:21:41 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-f[71869]: 2024-03-24T00:21:41.711 INFO:journalctl@ceph.mds.f.smithi193.stdout:Mar 24 00:21:41 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-f[71869]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-24T00:21:41.711 INFO:journalctl@ceph.mds.f.smithi193.stdout:Mar 24 00:21:41 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-f[71869]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x135) [0x7fdfa20b64a1] 2024-03-24T00:21:41.711 INFO:journalctl@ceph.mds.f.smithi193.stdout:Mar 24 00:21:41 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-f[71869]: 2: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7fdfa20b6667] 2024-03-24T00:21:41.711 INFO:journalctl@ceph.mds.f.smithi193.stdout:Mar 24 00:21:41 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-f[71869]: 3: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x55bcda43f223] 2024-03-24T00:21:41.711 INFO:journalctl@ceph.mds.f.smithi193.stdout:Mar 24 00:21:41 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-f[71869]: 4: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x55bcda2b95d6] 2024-03-24T00:21:41.712 INFO:journalctl@ceph.mds.f.smithi193.stdout:Mar 24 00:21:41 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-f[71869]: 5: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x55bcda2b9da1] 2024-03-24T00:21:41.712 INFO:journalctl@ceph.mds.f.smithi193.stdout:Mar 24 00:21:41 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-f[71869]: 6: (Server::handle_client_rename(boost::intrusive_ptr const&)+0x259d) [0x55bcda2f697d] 2024-03-24T00:21:41.712 INFO:journalctl@ceph.mds.f.smithi193.stdout:Mar 24 00:21:41 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-f[71869]: 7: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x55bcda30c233] 2024-03-24T00:21:41.712 INFO:journalctl@ceph.mds.f.smithi193.stdout:Mar 24 00:21:41 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-f[71869]: 8: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x55bcda30e197] 2024-03-24T00:21:41.712 INFO:journalctl@ceph.mds.f.smithi193.stdout:Mar 24 00:21:41 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-f[71869]: 9: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x55bcda20d11b] 2024-03-24T00:21:41.712 INFO:journalctl@ceph.mds.f.smithi193.stdout:Mar 24 00:21:41 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-f[71869]: 10: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x55bcda20d6bc] 2024-03-24T00:21:41.712 INFO:journalctl@ceph.mds.f.smithi193.stdout:Mar 24 00:21:41 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-f[71869]: 11: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x55bcda1f71ff] 2024-03-24T00:21:41.712 INFO:journalctl@ceph.mds.f.smithi193.stdout:Mar 24 00:21:41 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-f[71869]: 12: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7fdfa2348488] 2024-03-24T00:21:41.712 INFO:journalctl@ceph.mds.f.smithi193.stdout:Mar 24 00:21:41 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-f[71869]: 13: (DispatchQueue::entry()+0x51f) [0x7fdfa234560f] 2024-03-24T00:21:41.712 INFO:journalctl@ceph.mds.f.smithi193.stdout:Mar 24 00:21:41 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-f[71869]: 14: (DispatchQueue::DispatchThread::entry()+0x11) [0x7fdfa2418e81] 2024-03-24T00:21:41.712 INFO:journalctl@ceph.mds.f.smithi193.stdout:Mar 24 00:21:41 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-f[71869]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7fdfa0e331ca] 2024-03-24T00:21:41.712 INFO:journalctl@ceph.mds.f.smithi193.stdout:Mar 24 00:21:41 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-f[71869]: 16: clone() 2024-03-24T00:21:41.712 INFO:journalctl@ceph.mds.f.smithi193.stdout:Mar 24 00:21:41 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-f[71869]: 2024-03-24T00:21:41.712 INFO:journalctl@ceph.mds.f.smithi193.stdout:Mar 24 00:21:41 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-f[71869]: 0> 2024-03-24T00:21:41.614+0000 7fdf97a8c700 -1 *** Caught signal (Aborted) ** 2024-03-24T00:21:41.713 INFO:journalctl@ceph.mds.f.smithi193.stdout:Mar 24 00:21:41 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-f[71869]: in thread 7fdf97a8c700 thread_name:ms_dispatch 2024-03-24T00:21:41.713 INFO:journalctl@ceph.mds.f.smithi193.stdout:Mar 24 00:21:41 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-f[71869]: 2024-03-24T00:21:41.713 INFO:journalctl@ceph.mds.f.smithi193.stdout:Mar 24 00:21:41 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-f[71869]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-24T00:21:41.713 INFO:journalctl@ceph.mds.f.smithi193.stdout:Mar 24 00:21:41 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-f[71869]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7fdfa0e3dd20] 2024-03-24T00:21:41.713 INFO:journalctl@ceph.mds.f.smithi193.stdout:Mar 24 00:21:41 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-f[71869]: 2: gsignal() 2024-03-24T00:21:41.713 INFO:journalctl@ceph.mds.f.smithi193.stdout:Mar 24 00:21:41 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-f[71869]: 3: abort() 2024-03-24T00:21:41.713 INFO:journalctl@ceph.mds.f.smithi193.stdout:Mar 24 00:21:41 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-f[71869]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x18f) [0x7fdfa20b64fb] 2024-03-24T00:21:41.714 INFO:journalctl@ceph.mds.f.smithi193.stdout:Mar 24 00:21:41 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-f[71869]: 5: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7fdfa20b6667] 2024-03-24T00:21:41.714 INFO:journalctl@ceph.mds.f.smithi193.stdout:Mar 24 00:21:41 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-f[71869]: 6: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x55bcda43f223] 2024-03-24T00:21:41.714 INFO:journalctl@ceph.mds.f.smithi193.stdout:Mar 24 00:21:41 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-f[71869]: 7: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x55bcda2b95d6] 2024-03-24T00:21:41.714 INFO:journalctl@ceph.mds.f.smithi193.stdout:Mar 24 00:21:41 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-f[71869]: 8: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x55bcda2b9da1] 2024-03-24T00:21:41.714 INFO:journalctl@ceph.mds.f.smithi193.stdout:Mar 24 00:21:41 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-f[71869]: 9: (Server::handle_client_rename(boost::intrusive_ptr const&)+0x259d) [0x55bcda2f697d] 2024-03-24T00:21:41.714 INFO:journalctl@ceph.mds.f.smithi193.stdout:Mar 24 00:21:41 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-f[71869]: 10: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x55bcda30c233] 2024-03-24T00:21:41.714 INFO:journalctl@ceph.mds.f.smithi193.stdout:Mar 24 00:21:41 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-f[71869]: 11: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x55bcda30e197] 2024-03-24T00:21:41.714 INFO:journalctl@ceph.mds.f.smithi193.stdout:Mar 24 00:21:41 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-f[71869]: 12: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x55bcda20d11b] 2024-03-24T00:21:41.714 INFO:journalctl@ceph.mds.f.smithi193.stdout:Mar 24 00:21:41 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-f[71869]: 13: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x55bcda20d6bc] 2024-03-24T00:21:41.714 INFO:journalctl@ceph.mds.f.smithi193.stdout:Mar 24 00:21:41 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-f[71869]: 14: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x55bcda1f71ff] 2024-03-24T00:21:41.714 INFO:journalctl@ceph.mds.f.smithi193.stdout:Mar 24 00:21:41 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-f[71869]: 15: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7fdfa2348488] 2024-03-24T00:21:41.714 INFO:journalctl@ceph.mds.f.smithi193.stdout:Mar 24 00:21:41 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-f[71869]: 16: (DispatchQueue::entry()+0x51f) [0x7fdfa234560f] 2024-03-24T00:21:41.714 INFO:journalctl@ceph.mds.f.smithi193.stdout:Mar 24 00:21:41 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-f[71869]: 17: (DispatchQueue::DispatchThread::entry()+0x11) [0x7fdfa2418e81] 2024-03-24T00:21:41.715 INFO:journalctl@ceph.mds.f.smithi193.stdout:Mar 24 00:21:41 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-f[71869]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7fdfa0e331ca] 2024-03-24T00:21:41.715 INFO:journalctl@ceph.mds.f.smithi193.stdout:Mar 24 00:21:41 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-f[71869]: 19: clone() 2024-03-24T00:21:41.715 INFO:journalctl@ceph.mds.f.smithi193.stdout:Mar 24 00:21:41 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-f[71869]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-03-24T00:21:41.715 INFO:journalctl@ceph.mds.f.smithi193.stdout:Mar 24 00:21:41 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-f[71869]: 2024-03-24T00:21:41.715 INFO:journalctl@ceph.mds.f.smithi193.stdout:Mar 24 00:21:41 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-f[71869]: -5> 2024-03-24T00:21:41.613+0000 7fdf97a8c700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: In function 'void Locker::set_xlocks_done(MutationImpl*, bool)' thread 7fdf97a8c700 time 2024-03-24T00:21:41.612694+0000 2024-03-24T00:21:41.715 INFO:journalctl@ceph.mds.f.smithi193.stdout:Mar 24 00:21:41 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-f[71869]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: 626: FAILED ceph_assert(obj->is_auth()) 2024-03-24T00:21:41.715 INFO:journalctl@ceph.mds.f.smithi193.stdout:Mar 24 00:21:41 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-f[71869]: 2024-03-24T00:21:41.715 INFO:journalctl@ceph.mds.f.smithi193.stdout:Mar 24 00:21:41 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-f[71869]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-24T00:21:41.715 INFO:journalctl@ceph.mds.f.smithi193.stdout:Mar 24 00:21:41 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-f[71869]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x135) [0x7fdfa20b64a1] 2024-03-24T00:21:41.715 INFO:journalctl@ceph.mds.f.smithi193.stdout:Mar 24 00:21:41 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-f[71869]: 2: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7fdfa20b6667] 2024-03-24T00:21:41.715 INFO:journalctl@ceph.mds.f.smithi193.stdout:Mar 24 00:21:41 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-f[71869]: 3: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x55bcda43f223] 2024-03-24T00:21:41.715 INFO:journalctl@ceph.mds.f.smithi193.stdout:Mar 24 00:21:41 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-f[71869]: 4: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x55bcda2b95d6] 2024-03-24T00:21:41.715 INFO:journalctl@ceph.mds.f.smithi193.stdout:Mar 24 00:21:41 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-f[71869]: 5: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x55bcda2b9da1] 2024-03-24T00:21:41.715 INFO:journalctl@ceph.mds.f.smithi193.stdout:Mar 24 00:21:41 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-f[71869]: 6: (Server::handle_client_rename(boost::intrusive_ptr const&)+0x259d) [0x55bcda2f697d] 2024-03-24T00:21:41.715 INFO:journalctl@ceph.mds.f.smithi193.stdout:Mar 24 00:21:41 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-f[71869]: 7: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x55bcda30c233] 2024-03-24T00:21:41.716 INFO:journalctl@ceph.mds.f.smithi193.stdout:Mar 24 00:21:41 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-f[71869]: 8: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x55bcda30e197] 2024-03-24T00:21:41.716 INFO:journalctl@ceph.mds.f.smithi193.stdout:Mar 24 00:21:41 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-f[71869]: 9: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x55bcda20d11b] 2024-03-24T00:21:41.716 INFO:journalctl@ceph.mds.f.smithi193.stdout:Mar 24 00:21:41 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-f[71869]: 10: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x55bcda20d6bc] 2024-03-24T00:21:41.716 INFO:journalctl@ceph.mds.f.smithi193.stdout:Mar 24 00:21:41 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-f[71869]: 11: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x55bcda1f71ff] 2024-03-24T00:21:41.716 INFO:journalctl@ceph.mds.f.smithi193.stdout:Mar 24 00:21:41 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-f[71869]: 12: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7fdfa2348488] 2024-03-24T00:21:41.716 INFO:journalctl@ceph.mds.f.smithi193.stdout:Mar 24 00:21:41 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-f[71869]: 13: (DispatchQueue::entry()+0x51f) [0x7fdfa234560f] 2024-03-24T00:21:41.716 INFO:journalctl@ceph.mds.f.smithi193.stdout:Mar 24 00:21:41 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-f[71869]: 14: (DispatchQueue::DispatchThread::entry()+0x11) [0x7fdfa2418e81] 2024-03-24T00:21:41.716 INFO:journalctl@ceph.mds.f.smithi193.stdout:Mar 24 00:21:41 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-f[71869]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7fdfa0e331ca] 2024-03-24T00:21:41.716 INFO:journalctl@ceph.mds.f.smithi193.stdout:Mar 24 00:21:41 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-f[71869]: 16: clone() 2024-03-24T00:21:41.716 INFO:journalctl@ceph.mds.f.smithi193.stdout:Mar 24 00:21:41 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-f[71869]: 2024-03-24T00:21:41.716 INFO:journalctl@ceph.mds.f.smithi193.stdout:Mar 24 00:21:41 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-f[71869]: -4> 2024-03-24T00:21:41.614+0000 7fdf97a8c700 -1 *** Caught signal (Aborted) ** 2024-03-24T00:21:41.716 INFO:journalctl@ceph.mds.f.smithi193.stdout:Mar 24 00:21:41 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-f[71869]: in thread 7fdf97a8c700 thread_name:ms_dispatch 2024-03-24T00:21:41.716 INFO:journalctl@ceph.mds.f.smithi193.stdout:Mar 24 00:21:41 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-f[71869]: 2024-03-24T00:21:41.716 INFO:journalctl@ceph.mds.f.smithi193.stdout:Mar 24 00:21:41 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-f[71869]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-24T00:21:41.716 INFO:journalctl@ceph.mds.f.smithi193.stdout:Mar 24 00:21:41 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-f[71869]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7fdfa0e3dd20] 2024-03-24T00:21:41.717 INFO:journalctl@ceph.mds.f.smithi193.stdout:Mar 24 00:21:41 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-f[71869]: 2: gsignal() 2024-03-24T00:21:41.717 INFO:journalctl@ceph.mds.f.smithi193.stdout:Mar 24 00:21:41 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-f[71869]: 3: abort() 2024-03-24T00:21:41.717 INFO:journalctl@ceph.mds.f.smithi193.stdout:Mar 24 00:21:41 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-f[71869]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x18f) [0x7fdfa20b64fb] 2024-03-24T00:21:41.717 INFO:journalctl@ceph.mds.f.smithi193.stdout:Mar 24 00:21:41 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-f[71869]: 5: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7fdfa20b6667] 2024-03-24T00:21:41.717 INFO:journalctl@ceph.mds.f.smithi193.stdout:Mar 24 00:21:41 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-f[71869]: 6: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x55bcda43f223] 2024-03-24T00:21:41.717 INFO:journalctl@ceph.mds.f.smithi193.stdout:Mar 24 00:21:41 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-f[71869]: 7: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x55bcda2b95d6] 2024-03-24T00:21:41.717 INFO:journalctl@ceph.mds.f.smithi193.stdout:Mar 24 00:21:41 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-f[71869]: 8: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x55bcda2b9da1] 2024-03-24T00:21:41.717 INFO:journalctl@ceph.mds.f.smithi193.stdout:Mar 24 00:21:41 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-f[71869]: 9: (Server::handle_client_rename(boost::intrusive_ptr const&)+0x259d) [0x55bcda2f697d] 2024-03-24T00:21:41.717 INFO:journalctl@ceph.mds.f.smithi193.stdout:Mar 24 00:21:41 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-f[71869]: 10: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x55bcda30c233] 2024-03-24T00:21:41.717 INFO:journalctl@ceph.mds.f.smithi193.stdout:Mar 24 00:21:41 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-f[71869]: 11: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x55bcda30e197] 2024-03-24T00:21:41.717 INFO:journalctl@ceph.mds.f.smithi193.stdout:Mar 24 00:21:41 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-f[71869]: 12: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x55bcda20d11b] 2024-03-24T00:21:41.717 INFO:journalctl@ceph.mds.f.smithi193.stdout:Mar 24 00:21:41 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-f[71869]: 13: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x55bcda20d6bc] 2024-03-24T00:21:41.717 INFO:journalctl@ceph.mds.f.smithi193.stdout:Mar 24 00:21:41 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-f[71869]: 14: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x55bcda1f71ff] 2024-03-24T00:21:41.717 INFO:journalctl@ceph.mds.f.smithi193.stdout:Mar 24 00:21:41 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-f[71869]: 15: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7fdfa2348488] 2024-03-24T00:21:41.718 INFO:journalctl@ceph.mds.f.smithi193.stdout:Mar 24 00:21:41 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-f[71869]: 16: (DispatchQueue::entry()+0x51f) [0x7fdfa234560f] 2024-03-24T00:21:41.718 INFO:journalctl@ceph.mds.f.smithi193.stdout:Mar 24 00:21:41 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-f[71869]: 17: (DispatchQueue::DispatchThread::entry()+0x11) [0x7fdfa2418e81] 2024-03-24T00:21:41.718 INFO:journalctl@ceph.mds.f.smithi193.stdout:Mar 24 00:21:41 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-f[71869]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7fdfa0e331ca] 2024-03-24T00:21:41.718 INFO:journalctl@ceph.mds.f.smithi193.stdout:Mar 24 00:21:41 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-f[71869]: 19: clone() 2024-03-24T00:21:41.718 INFO:journalctl@ceph.mds.f.smithi193.stdout:Mar 24 00:21:41 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-f[71869]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-03-24T00:21:41.718 INFO:journalctl@ceph.mds.f.smithi193.stdout:Mar 24 00:21:41 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-f[71869]: 2024-03-24T00:21:41.718 INFO:journalctl@ceph.mds.f.smithi193.stdout:Mar 24 00:21:41 smithi193 podman[72060]: 2024-03-24 00:21:41.684052521 +0000 UTC m=+0.021878938 container died 72641b2d5a1ac705eefede03dcb21c26a2e124304b9f7ca43ab321a67492be1f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-f, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, ceph=True, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240319, org.label-schema.vendor=CentOS, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, maintainer=Guillaume Abrioux ) 2024-03-24T00:21:42.624 INFO:journalctl@ceph.mds.f.smithi193.stdout:Mar 24 00:21:42 smithi193 podman[72060]: 2024-03-24 00:21:42.366404035 +0000 UTC m=+0.704230436 container remove 72641b2d5a1ac705eefede03dcb21c26a2e124304b9f7ca43ab321a67492be1f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-f, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, ceph=True, org.label-schema.build-date=20240319, org.label-schema.schema-version=1.0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30) 2024-03-24T00:21:42.624 INFO:journalctl@ceph.mds.f.smithi193.stdout:Mar 24 00:21:42 smithi193 systemd[1]: ceph-d79a5420-e971-11ee-95cd-87774f69a715@mds.f.service: Main process exited, code=exited, status=134/n/a 2024-03-24T00:21:42.624 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:21:42 smithi193 ceph-mon[36878]: mds.? [v2:172.21.15.193:6834/2814329894,v1:172.21.15.193:6835/2814329894] up:rejoin 2024-03-24T00:21:42.625 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:21:42 smithi193 ceph-mon[36878]: mds.? [v2:172.21.15.118:6838/979323377,v1:172.21.15.118:6839/979323377] up:rejoin 2024-03-24T00:21:42.625 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:21:42 smithi193 ceph-mon[36878]: fsmap cephfs:5/5 3 up:standby-replay 4 up:standby 2 up:rejoin 3 up:active 2024-03-24T00:21:42.625 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:21:42 smithi193 ceph-mon[36878]: daemon mds.g is now active in filesystem cephfs as rank 3 2024-03-24T00:21:42.625 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:21:42 smithi193 ceph-mon[36878]: daemon mds.f is now active in filesystem cephfs as rank 1 2024-03-24T00:21:42.625 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:21:42 smithi193 ceph-mon[36878]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-03-24T00:21:42.625 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:21:42 smithi193 ceph-mon[36878]: Cluster is now healthy 2024-03-24T00:21:42.625 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:21:42 smithi193 ceph-mon[36878]: mds.? [v2:172.21.15.118:6838/979323377,v1:172.21.15.118:6839/979323377] up:active 2024-03-24T00:21:42.625 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:21:42 smithi193 ceph-mon[36878]: mds.? [v2:172.21.15.193:6834/2814329894,v1:172.21.15.193:6835/2814329894] up:active 2024-03-24T00:21:42.625 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:21:42 smithi193 ceph-mon[36878]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-03-24T00:21:42.626 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:21:42 smithi193 ceph-mon[36878]: pgmap v590: 97 pgs: 97 active+clean; 136 MiB data, 777 MiB used, 1.0 TiB / 1.0 TiB avail; 52 KiB/s rd, 101 KiB/s wr, 41 op/s 2024-03-24T00:21:42.696 INFO:journalctl@ceph.mds.g.smithi118.stdout:Mar 24 00:21:42 smithi118 podman[94417]: 2024-03-24 00:21:42.511220125 +0000 UTC m=+0.858602011 container remove e5c87fc6a3bf4c109d5e01bdb0b1f592e72fa7f852e0ec79dc7ea1f1054cf96b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-g, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_CLEAN=True, ceph=True, org.label-schema.build-date=20240319) 2024-03-24T00:21:42.696 INFO:journalctl@ceph.mds.g.smithi118.stdout:Mar 24 00:21:42 smithi118 systemd[1]: ceph-d79a5420-e971-11ee-95cd-87774f69a715@mds.g.service: Main process exited, code=exited, status=134/n/a 2024-03-24T00:21:42.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:21:42 smithi118 ceph-mon[31535]: mds.? [v2:172.21.15.193:6834/2814329894,v1:172.21.15.193:6835/2814329894] up:rejoin 2024-03-24T00:21:42.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:21:42 smithi118 ceph-mon[31535]: mds.? [v2:172.21.15.118:6838/979323377,v1:172.21.15.118:6839/979323377] up:rejoin 2024-03-24T00:21:42.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:21:42 smithi118 ceph-mon[31535]: fsmap cephfs:5/5 3 up:standby-replay 4 up:standby 2 up:rejoin 3 up:active 2024-03-24T00:21:42.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:21:42 smithi118 ceph-mon[31535]: daemon mds.g is now active in filesystem cephfs as rank 3 2024-03-24T00:21:42.697 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:21:42 smithi118 ceph-mon[31535]: daemon mds.f is now active in filesystem cephfs as rank 1 2024-03-24T00:21:42.697 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:21:42 smithi118 ceph-mon[31535]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-03-24T00:21:42.697 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:21:42 smithi118 ceph-mon[31535]: Cluster is now healthy 2024-03-24T00:21:42.697 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:21:42 smithi118 ceph-mon[31535]: mds.? [v2:172.21.15.118:6838/979323377,v1:172.21.15.118:6839/979323377] up:active 2024-03-24T00:21:42.697 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:21:42 smithi118 ceph-mon[31535]: mds.? [v2:172.21.15.193:6834/2814329894,v1:172.21.15.193:6835/2814329894] up:active 2024-03-24T00:21:42.697 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:21:42 smithi118 ceph-mon[31535]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-03-24T00:21:42.697 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:21:42 smithi118 ceph-mon[31535]: pgmap v590: 97 pgs: 97 active+clean; 136 MiB data, 777 MiB used, 1.0 TiB / 1.0 TiB avail; 52 KiB/s rd, 101 KiB/s wr, 41 op/s 2024-03-24T00:21:42.704 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:21:42 smithi189 ceph-mon[36187]: mds.? [v2:172.21.15.193:6834/2814329894,v1:172.21.15.193:6835/2814329894] up:rejoin 2024-03-24T00:21:42.704 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:21:42 smithi189 ceph-mon[36187]: mds.? [v2:172.21.15.118:6838/979323377,v1:172.21.15.118:6839/979323377] up:rejoin 2024-03-24T00:21:42.704 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:21:42 smithi189 ceph-mon[36187]: fsmap cephfs:5/5 3 up:standby-replay 4 up:standby 2 up:rejoin 3 up:active 2024-03-24T00:21:42.704 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:21:42 smithi189 ceph-mon[36187]: daemon mds.g is now active in filesystem cephfs as rank 3 2024-03-24T00:21:42.704 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:21:42 smithi189 ceph-mon[36187]: daemon mds.f is now active in filesystem cephfs as rank 1 2024-03-24T00:21:42.705 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:21:42 smithi189 ceph-mon[36187]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-03-24T00:21:42.705 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:21:42 smithi189 ceph-mon[36187]: Cluster is now healthy 2024-03-24T00:21:42.705 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:21:42 smithi189 ceph-mon[36187]: mds.? [v2:172.21.15.118:6838/979323377,v1:172.21.15.118:6839/979323377] up:active 2024-03-24T00:21:42.705 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:21:42 smithi189 ceph-mon[36187]: mds.? [v2:172.21.15.193:6834/2814329894,v1:172.21.15.193:6835/2814329894] up:active 2024-03-24T00:21:42.705 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:21:42 smithi189 ceph-mon[36187]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-03-24T00:21:42.705 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:21:42 smithi189 ceph-mon[36187]: pgmap v590: 97 pgs: 97 active+clean; 136 MiB data, 777 MiB used, 1.0 TiB / 1.0 TiB avail; 52 KiB/s rd, 101 KiB/s wr, 41 op/s 2024-03-24T00:21:42.961 INFO:journalctl@ceph.mds.f.smithi193.stdout:Mar 24 00:21:42 smithi193 systemd[1]: ceph-d79a5420-e971-11ee-95cd-87774f69a715@mds.f.service: Failed with result 'exit-code'. 2024-03-24T00:21:43.196 INFO:journalctl@ceph.mds.g.smithi118.stdout:Mar 24 00:21:42 smithi118 systemd[1]: ceph-d79a5420-e971-11ee-95cd-87774f69a715@mds.g.service: Failed with result 'exit-code'. 2024-03-24T00:21:43.196 INFO:journalctl@ceph.mds.g.smithi118.stdout:Mar 24 00:21:42 smithi118 systemd[1]: ceph-d79a5420-e971-11ee-95cd-87774f69a715@mds.g.service: Consumed 1.417s CPU time. 2024-03-24T00:21:44.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:21:43 smithi118 ceph-mon[31535]: pgmap v591: 97 pgs: 97 active+clean; 137 MiB data, 796 MiB used, 1.0 TiB / 1.0 TiB avail; 525 KiB/s rd, 236 KiB/s wr, 119 op/s 2024-03-24T00:21:44.203 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:21:43 smithi189 ceph-mon[36187]: pgmap v591: 97 pgs: 97 active+clean; 137 MiB data, 796 MiB used, 1.0 TiB / 1.0 TiB avail; 525 KiB/s rd, 236 KiB/s wr, 119 op/s 2024-03-24T00:21:44.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:21:43 smithi193 ceph-mon[36878]: pgmap v591: 97 pgs: 97 active+clean; 137 MiB data, 796 MiB used, 1.0 TiB / 1.0 TiB avail; 525 KiB/s rd, 236 KiB/s wr, 119 op/s 2024-03-24T00:21:46.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:21:46 smithi118 ceph-mon[31535]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-03-24T00:21:46.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:21:46 smithi118 ceph-mon[31535]: pgmap v592: 97 pgs: 97 active+clean; 136 MiB data, 796 MiB used, 1.0 TiB / 1.0 TiB avail; 826 KiB/s rd, 185 KiB/s wr, 157 op/s 2024-03-24T00:21:46.704 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:21:46 smithi189 ceph-mon[36187]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-03-24T00:21:46.704 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:21:46 smithi189 ceph-mon[36187]: pgmap v592: 97 pgs: 97 active+clean; 136 MiB data, 796 MiB used, 1.0 TiB / 1.0 TiB avail; 826 KiB/s rd, 185 KiB/s wr, 157 op/s 2024-03-24T00:21:46.711 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:21:46 smithi193 ceph-mon[36878]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-03-24T00:21:46.711 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:21:46 smithi193 ceph-mon[36878]: pgmap v592: 97 pgs: 97 active+clean; 136 MiB data, 796 MiB used, 1.0 TiB / 1.0 TiB avail; 826 KiB/s rd, 185 KiB/s wr, 157 op/s 2024-03-24T00:21:46.845 INFO:tasks.workunit.client.0.smithi118.stdout: 45 1779 47 863 21 844 37 2024-03-24T00:21:47.107 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:21:47.106+0000 7fd627e00640 1 --2- 172.21.15.118:0/736842230 >> [v2:172.21.15.193:6832/1023428412,v1:172.21.15.193:6833/1023428412] conn(0x55c55390a6a0 0x55c55390cac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:21:47.108 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:21:47.107+0000 7fd627e00640 1 -- 172.21.15.118:0/736842230 >> [v2:172.21.15.193:6832/1023428412,v1:172.21.15.193:6833/1023428412] conn(0x55c55390a6a0 msgr2=0x55c55390cac0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 13 2024-03-24T00:21:47.108 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:21:47.107+0000 7fd627e00640 1 -- 172.21.15.118:0/736842230 >> [v2:172.21.15.193:6832/1023428412,v1:172.21.15.193:6833/1023428412] conn(0x55c55390a6a0 msgr2=0x55c55390cac0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-03-24T00:21:47.108 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:21:47.107+0000 7fd627e00640 1 --2- 172.21.15.118:0/736842230 >> [v2:172.21.15.193:6832/1023428412,v1:172.21.15.193:6833/1023428412] conn(0x55c55390a6a0 0x55c55390cac0 crc :-1 s=SESSION_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-03-24T00:21:47.108 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T00:21:47.107+0000 7fd627e00640 1 --2- 172.21.15.118:0/736842230 >> [v2:172.21.15.193:6832/1023428412,v1:172.21.15.193:6833/1023428412] conn(0x55c55390a6a0 0x55c55390cac0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 15.000000 2024-03-24T00:21:48.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:21:47 smithi118 ceph-mon[31535]: pgmap v593: 97 pgs: 97 active+clean; 137 MiB data, 798 MiB used, 1.0 TiB / 1.0 TiB avail; 867 KiB/s rd, 245 KiB/s wr, 152 op/s 2024-03-24T00:21:48.203 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:21:47 smithi189 ceph-mon[36187]: pgmap v593: 97 pgs: 97 active+clean; 137 MiB data, 798 MiB used, 1.0 TiB / 1.0 TiB avail; 867 KiB/s rd, 245 KiB/s wr, 152 op/s 2024-03-24T00:21:48.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:21:47 smithi193 ceph-mon[36878]: pgmap v593: 97 pgs: 97 active+clean; 137 MiB data, 798 MiB used, 1.0 TiB / 1.0 TiB avail; 867 KiB/s rd, 245 KiB/s wr, 152 op/s 2024-03-24T00:21:50.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:21:49 smithi118 ceph-mon[31535]: pgmap v594: 97 pgs: 97 active+clean; 138 MiB data, 807 MiB used, 1.0 TiB / 1.0 TiB avail; 733 KiB/s rd, 282 KiB/s wr, 132 op/s 2024-03-24T00:21:50.203 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:21:49 smithi189 ceph-mon[36187]: pgmap v594: 97 pgs: 97 active+clean; 138 MiB data, 807 MiB used, 1.0 TiB / 1.0 TiB avail; 733 KiB/s rd, 282 KiB/s wr, 132 op/s 2024-03-24T00:21:50.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:21:49 smithi193 ceph-mon[36878]: pgmap v594: 97 pgs: 97 active+clean; 138 MiB data, 807 MiB used, 1.0 TiB / 1.0 TiB avail; 733 KiB/s rd, 282 KiB/s wr, 132 op/s 2024-03-24T00:21:51.379 ERROR:teuthology:Uncaught exception (Hub) Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/virtualenv/lib/python3.8/site-packages/paramiko/channel.py", line 697, in recv out = self.in_buffer.read(nbytes, self.timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/virtualenv/lib/python3.8/site-packages/paramiko/buffered_pipe.py", line 154, in read raise PipeTimeout() paramiko.buffered_pipe.PipeTimeout During handling of the above exception, another exception occurred: Traceback (most recent call last): File "src/gevent/greenlet.py", line 908, in gevent._gevent_cgreenlet.Greenlet.run File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 323, in copy_file_to copy_to_log(src, logger, capture=stream, quiet=quiet) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 276, in copy_to_log for line in f: File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/virtualenv/lib/python3.8/site-packages/paramiko/file.py", line 109, in __next__ line = self.readline() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/virtualenv/lib/python3.8/site-packages/paramiko/file.py", line 275, in readline new_data = self._read(n) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/virtualenv/lib/python3.8/site-packages/paramiko/channel.py", line 1359, in _read return self.channel.recv(size) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/virtualenv/lib/python3.8/site-packages/paramiko/channel.py", line 699, in recv raise socket.timeout() socket.timeout 2024-03-24T00:21:51.476 DEBUG:teuthology.orchestra.run:got remote process result: 124 2024-03-24T00:21:51.477 ERROR:tasks.fwd_scrub.fs.[cephfs]:exception: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/virtualenv/lib/python3.8/site-packages/paramiko/channel.py", line 697, in recv out = self.in_buffer.read(nbytes, self.timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/virtualenv/lib/python3.8/site-packages/paramiko/buffered_pipe.py", line 154, in read raise PipeTimeout() paramiko.buffered_pipe.PipeTimeout During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_ceph-c_3be9a5c9bc793e11e2800a8c0c696e8b46742033/qa/tasks/fwd_scrub.py", line 38, in _run self.do_scrub() File "/home/teuthworker/src/git.ceph.com_ceph-c_3be9a5c9bc793e11e2800a8c0c696e8b46742033/qa/tasks/fwd_scrub.py", line 55, in do_scrub self._scrub() File "/home/teuthworker/src/git.ceph.com_ceph-c_3be9a5c9bc793e11e2800a8c0c696e8b46742033/qa/tasks/fwd_scrub.py", line 79, in _scrub self._check_damage() File "/home/teuthworker/src/git.ceph.com_ceph-c_3be9a5c9bc793e11e2800a8c0c696e8b46742033/qa/tasks/fwd_scrub.py", line 82, in _check_damage rdmg = self.fs.get_damage() File "/home/teuthworker/src/git.ceph.com_ceph-c_3be9a5c9bc793e11e2800a8c0c696e8b46742033/qa/tasks/cephfs/filesystem.py", line 1777, in get_damage result[rank] = self.get_damage(rank=rank) File "/home/teuthworker/src/git.ceph.com_ceph-c_3be9a5c9bc793e11e2800a8c0c696e8b46742033/qa/tasks/cephfs/filesystem.py", line 1780, in get_damage return self.rank_tell(['damage', 'ls'], rank=rank) File "/home/teuthworker/src/git.ceph.com_ceph-c_3be9a5c9bc793e11e2800a8c0c696e8b46742033/qa/tasks/cephfs/filesystem.py", line 1319, in rank_tell out = self.get_ceph_cmd_stdout("tell", f"mds.{self.id}:{rank}", *command, timeout=timeout) File "/home/teuthworker/src/git.ceph.com_ceph-c_3be9a5c9bc793e11e2800a8c0c696e8b46742033/qa/tasks/ceph_test_case.py", line 55, in get_ceph_cmd_stdout return self.run_ceph_cmd(**kwargs).stdout.getvalue() File "/home/teuthworker/src/git.ceph.com_ceph-c_3be9a5c9bc793e11e2800a8c0c696e8b46742033/qa/tasks/ceph_test_case.py", line 30, in run_ceph_cmd return self.mon_manager.run_cluster_cmd(**kwargs) File "/home/teuthworker/src/git.ceph.com_ceph-c_3be9a5c9bc793e11e2800a8c0c696e8b46742033/qa/tasks/ceph_manager.py", line 1629, in run_cluster_cmd return self.controller.run(**kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 148, in wait greenlet.get(block=True,timeout=60) File "src/gevent/greenlet.py", line 805, in gevent._gevent_cgreenlet.Greenlet.get File "src/gevent/greenlet.py", line 373, in gevent._gevent_cgreenlet.Greenlet._raise_exception File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/virtualenv/lib/python3.8/site-packages/gevent/_compat.py", line 49, in reraise raise value.with_traceback(tb) File "src/gevent/greenlet.py", line 908, in gevent._gevent_cgreenlet.Greenlet.run File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 323, in copy_file_to copy_to_log(src, logger, capture=stream, quiet=quiet) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 276, in copy_to_log for line in f: File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/virtualenv/lib/python3.8/site-packages/paramiko/file.py", line 109, in __next__ line = self.readline() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/virtualenv/lib/python3.8/site-packages/paramiko/file.py", line 275, in readline new_data = self._read(n) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/virtualenv/lib/python3.8/site-packages/paramiko/channel.py", line 1359, in _read return self.channel.recv(size) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/virtualenv/lib/python3.8/site-packages/paramiko/channel.py", line 699, in recv raise socket.timeout() socket.timeout 2024-03-24T00:21:52.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:21:51 smithi118 ceph-mon[31535]: pgmap v595: 97 pgs: 97 active+clean; 138 MiB data, 807 MiB used, 1.0 TiB / 1.0 TiB avail; 695 KiB/s rd, 293 KiB/s wr, 120 op/s 2024-03-24T00:21:52.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:21:51 smithi189 ceph-mon[36187]: pgmap v595: 97 pgs: 97 active+clean; 138 MiB data, 807 MiB used, 1.0 TiB / 1.0 TiB avail; 695 KiB/s rd, 293 KiB/s wr, 120 op/s 2024-03-24T00:21:52.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:21:51 smithi193 ceph-mon[36878]: pgmap v595: 97 pgs: 97 active+clean; 138 MiB data, 807 MiB used, 1.0 TiB / 1.0 TiB avail; 695 KiB/s rd, 293 KiB/s wr, 120 op/s 2024-03-24T00:21:53.082 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:21:52 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T00:21:53.114 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:21:52 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-24T00:21:53.115 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:21:52 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-24T00:21:53.115 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:21:52 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-24T00:21:53.115 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:21:52 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-03-24T00:21:53.116 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:21:52 smithi193 ceph-mon[36878]: Adjusting osd_memory_target on smithi189 to 174.1M 2024-03-24T00:21:53.116 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:21:52 smithi193 ceph-mon[36878]: Unable to set osd_memory_target on smithi189 to 182635622: error parsing value: Value '182635622' is below minimum 939524096 2024-03-24T00:21:53.117 INFO:journalctl@ceph.mds.f.smithi193.stdout:Mar 24 00:21:52 smithi193 systemd[1]: ceph-d79a5420-e971-11ee-95cd-87774f69a715@mds.f.service: Scheduled restart job, restart counter is at 5. 2024-03-24T00:21:53.117 INFO:journalctl@ceph.mds.f.smithi193.stdout:Mar 24 00:21:52 smithi193 systemd[1]: Stopped Ceph mds.f for d79a5420-e971-11ee-95cd-87774f69a715. 2024-03-24T00:21:53.117 INFO:journalctl@ceph.mds.f.smithi193.stdout:Mar 24 00:21:52 smithi193 systemd[1]: Starting Ceph mds.f for d79a5420-e971-11ee-95cd-87774f69a715... 2024-03-24T00:21:53.117 INFO:journalctl@ceph.mds.f.smithi193.stdout:Mar 24 00:21:52 smithi193 podman[72258]: 2024-03-24 00:21:52.841902416 +0000 UTC m=+0.020996013 image pull c664afe56bb01db4f9afa9532c20598e201e664f2e9ba2be50db0d52674a26d2 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc 2024-03-24T00:21:53.118 INFO:journalctl@ceph.mds.f.smithi193.stdout:Mar 24 00:21:52 smithi193 podman[72258]: 2024-03-24 00:21:52.95970697 +0000 UTC m=+0.138800557 container create cfc9767be3a75f4aa40e1a53c49ee30e1b9a6187d38281688c96f804dbb0f0a1 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-f, maintainer=Guillaume Abrioux , ceph=True, GIT_CLEAN=True, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240319, org.label-schema.license=GPLv2, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c) 2024-03-24T00:21:53.196 INFO:journalctl@ceph.mds.g.smithi118.stdout:Mar 24 00:21:52 smithi118 systemd[1]: ceph-d79a5420-e971-11ee-95cd-87774f69a715@mds.g.service: Scheduled restart job, restart counter is at 3. 2024-03-24T00:21:53.196 INFO:journalctl@ceph.mds.g.smithi118.stdout:Mar 24 00:21:52 smithi118 systemd[1]: Stopped Ceph mds.g for d79a5420-e971-11ee-95cd-87774f69a715. 2024-03-24T00:21:53.196 INFO:journalctl@ceph.mds.g.smithi118.stdout:Mar 24 00:21:52 smithi118 systemd[1]: ceph-d79a5420-e971-11ee-95cd-87774f69a715@mds.g.service: Consumed 1.417s CPU time. 2024-03-24T00:21:53.196 INFO:journalctl@ceph.mds.g.smithi118.stdout:Mar 24 00:21:52 smithi118 systemd[1]: Starting Ceph mds.g for d79a5420-e971-11ee-95cd-87774f69a715... 2024-03-24T00:21:53.197 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:21:52 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T00:21:53.197 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:21:52 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-24T00:21:53.197 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:21:52 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-24T00:21:53.198 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:21:52 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-24T00:21:53.198 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:21:52 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-03-24T00:21:53.198 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:21:52 smithi118 ceph-mon[31535]: Adjusting osd_memory_target on smithi189 to 174.1M 2024-03-24T00:21:53.199 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:21:52 smithi118 ceph-mon[31535]: Unable to set osd_memory_target on smithi189 to 182635622: error parsing value: Value '182635622' is below minimum 939524096 2024-03-24T00:21:53.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:21:52 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T00:21:53.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:21:52 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-24T00:21:53.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:21:52 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-24T00:21:53.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:21:52 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-24T00:21:53.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:21:52 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-03-24T00:21:53.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:21:52 smithi189 ceph-mon[36187]: Adjusting osd_memory_target on smithi189 to 174.1M 2024-03-24T00:21:53.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:21:52 smithi189 ceph-mon[36187]: Unable to set osd_memory_target on smithi189 to 182635622: error parsing value: Value '182635622' is below minimum 939524096 2024-03-24T00:21:53.461 INFO:journalctl@ceph.mds.f.smithi193.stdout:Mar 24 00:21:53 smithi193 podman[72258]: 2024-03-24 00:21:53.203833422 +0000 UTC m=+0.382927022 container init cfc9767be3a75f4aa40e1a53c49ee30e1b9a6187d38281688c96f804dbb0f0a1 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-f, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240319, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , ceph=True, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5) 2024-03-24T00:21:53.461 INFO:journalctl@ceph.mds.f.smithi193.stdout:Mar 24 00:21:53 smithi193 podman[72258]: 2024-03-24 00:21:53.20642169 +0000 UTC m=+0.385515289 container start cfc9767be3a75f4aa40e1a53c49ee30e1b9a6187d38281688c96f804dbb0f0a1 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-f, org.label-schema.vendor=CentOS, ceph=True, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.build-date=20240319, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5) 2024-03-24T00:21:53.461 INFO:journalctl@ceph.mds.f.smithi193.stdout:Mar 24 00:21:53 smithi193 ceph-mds[72277]: -- 172.21.15.193:0/3642245555 <== mon.1 v2:172.21.15.193:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x55883cba7a40 con 0x55883cb7ac00 2024-03-24T00:21:53.461 INFO:journalctl@ceph.mds.f.smithi193.stdout:Mar 24 00:21:53 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-f[72273]: starting mds.f at 2024-03-24T00:21:53.461 INFO:journalctl@ceph.mds.f.smithi193.stdout:Mar 24 00:21:53 smithi193 bash[72258]: cfc9767be3a75f4aa40e1a53c49ee30e1b9a6187d38281688c96f804dbb0f0a1 2024-03-24T00:21:53.461 INFO:journalctl@ceph.mds.f.smithi193.stdout:Mar 24 00:21:53 smithi193 systemd[1]: Started Ceph mds.f for d79a5420-e971-11ee-95cd-87774f69a715. 2024-03-24T00:21:53.591 INFO:journalctl@ceph.mds.g.smithi118.stdout:Mar 24 00:21:53 smithi118 podman[94609]: 2024-03-24 00:21:53.230539584 +0000 UTC m=+0.037141691 image pull c664afe56bb01db4f9afa9532c20598e201e664f2e9ba2be50db0d52674a26d2 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc 2024-03-24T00:21:53.591 INFO:journalctl@ceph.mds.g.smithi118.stdout:Mar 24 00:21:53 smithi118 podman[94609]: 2024-03-24 00:21:53.412988707 +0000 UTC m=+0.219590855 container create 0498d0005714887d6ee54e9ea72d8f7bd3fe80cfc028a92db30adaa4b85c2076 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-g, ceph=True, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, org.label-schema.schema-version=1.0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, org.label-schema.build-date=20240319, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-24T00:21:54.197 INFO:journalctl@ceph.mds.g.smithi118.stdout:Mar 24 00:21:53 smithi118 podman[94609]: 2024-03-24 00:21:53.710984402 +0000 UTC m=+0.517586541 container init 0498d0005714887d6ee54e9ea72d8f7bd3fe80cfc028a92db30adaa4b85c2076 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-g, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, org.label-schema.license=GPLv2, GIT_CLEAN=True, org.label-schema.build-date=20240319) 2024-03-24T00:21:54.197 INFO:journalctl@ceph.mds.g.smithi118.stdout:Mar 24 00:21:53 smithi118 podman[94609]: 2024-03-24 00:21:53.720513708 +0000 UTC m=+0.527115856 container start 0498d0005714887d6ee54e9ea72d8f7bd3fe80cfc028a92db30adaa4b85c2076 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-g, org.label-schema.build-date=20240319, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, GIT_CLEAN=True, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30) 2024-03-24T00:21:54.197 INFO:journalctl@ceph.mds.g.smithi118.stdout:Mar 24 00:21:53 smithi118 ceph-mds[94659]: -- 172.21.15.118:0/2873470410 <== mon.2 v2:172.21.15.189:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x55c8b3371a40 con 0x55c8b3345000 2024-03-24T00:21:54.197 INFO:journalctl@ceph.mds.g.smithi118.stdout:Mar 24 00:21:53 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-g[94655]: starting mds.g at 2024-03-24T00:21:54.197 INFO:journalctl@ceph.mds.g.smithi118.stdout:Mar 24 00:21:53 smithi118 bash[94609]: 0498d0005714887d6ee54e9ea72d8f7bd3fe80cfc028a92db30adaa4b85c2076 2024-03-24T00:21:54.197 INFO:journalctl@ceph.mds.g.smithi118.stdout:Mar 24 00:21:53 smithi118 systemd[1]: Started Ceph mds.g for d79a5420-e971-11ee-95cd-87774f69a715. 2024-03-24T00:21:54.198 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:21:53 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-24T00:21:54.198 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:21:53 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-24T00:21:54.198 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:21:53 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-24T00:21:54.198 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:21:53 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-03-24T00:21:54.198 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:21:53 smithi118 ceph-mon[31535]: Adjusting osd_memory_target on smithi118 to 176.7M 2024-03-24T00:21:54.198 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:21:53 smithi118 ceph-mon[31535]: Unable to set osd_memory_target on smithi118 to 185365196: error parsing value: Value '185365196' is below minimum 939524096 2024-03-24T00:21:54.198 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:21:53 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T00:21:54.198 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:21:53 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T00:21:54.198 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:21:53 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T00:21:54.198 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:21:53 smithi118 ceph-mon[31535]: daemon mds.f restarted 2024-03-24T00:21:54.198 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:21:53 smithi118 ceph-mon[31535]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-03-24T00:21:54.199 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:21:53 smithi118 ceph-mon[31535]: osdmap e103: 12 total, 12 up, 12 in 2024-03-24T00:21:54.199 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:21:53 smithi118 ceph-mon[31535]: mds.? [v2:172.21.15.193:6834/2264220953,v1:172.21.15.193:6835/2264220953] up:boot 2024-03-24T00:21:54.199 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:21:53 smithi118 ceph-mon[31535]: Standby daemon mds.l assigned to filesystem cephfs as rank 1 2024-03-24T00:21:54.199 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:21:53 smithi118 ceph-mon[31535]: fsmap cephfs:4/5 {0=b=up:active,2=c=up:active,3=g=up:active,4=e=up:active} 5 up:standby-replay 3 up:standby, 1 failed 2024-03-24T00:21:54.199 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:21:53 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-03-24T00:21:54.199 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:21:53 smithi118 ceph-mon[31535]: fsmap cephfs:5/5 4 up:standby-replay 3 up:standby 1 up:replay 4 up:active 2024-03-24T00:21:54.199 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:21:53 smithi118 ceph-mon[31535]: daemon mds.g restarted 2024-03-24T00:21:54.199 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:21:53 smithi118 ceph-mon[31535]: osdmap e104: 12 total, 12 up, 12 in 2024-03-24T00:21:54.199 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:21:53 smithi118 ceph-mon[31535]: mds.? [v2:172.21.15.193:6832/2276762432,v1:172.21.15.193:6833/2276762432] up:resolve 2024-03-24T00:21:54.199 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:21:53 smithi118 ceph-mon[31535]: mds.? [v2:172.21.15.118:6838/3600700815,v1:172.21.15.118:6839/3600700815] up:boot 2024-03-24T00:21:54.199 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:21:53 smithi118 ceph-mon[31535]: Standby daemon mds.h assigned to filesystem cephfs as rank 3 2024-03-24T00:21:54.199 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:21:53 smithi118 ceph-mon[31535]: fsmap cephfs:4/5 {0=b=up:active,1=l=up:resolve,2=c=up:active,4=e=up:active} 4 up:standby-replay 4 up:standby, 1 failed 2024-03-24T00:21:54.199 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:21:53 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2024-03-24T00:21:54.199 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:21:53 smithi118 ceph-mon[31535]: fsmap cephfs:5/5 3 up:standby-replay 4 up:standby 1 up:replay 1 up:resolve 3 up:active 2024-03-24T00:21:54.199 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:21:53 smithi118 ceph-mon[31535]: pgmap v598: 97 pgs: 97 active+clean; 138 MiB data, 807 MiB used, 1.0 TiB / 1.0 TiB avail; 221 KiB/s rd, 278 KiB/s wr, 41 op/s 2024-03-24T00:21:54.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:21:53 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-24T00:21:54.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:21:53 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-24T00:21:54.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:21:53 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-24T00:21:54.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:21:53 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-03-24T00:21:54.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:21:53 smithi189 ceph-mon[36187]: Adjusting osd_memory_target on smithi118 to 176.7M 2024-03-24T00:21:54.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:21:53 smithi189 ceph-mon[36187]: Unable to set osd_memory_target on smithi118 to 185365196: error parsing value: Value '185365196' is below minimum 939524096 2024-03-24T00:21:54.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:21:53 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T00:21:54.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:21:53 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T00:21:54.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:21:53 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T00:21:54.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:21:53 smithi189 ceph-mon[36187]: daemon mds.f restarted 2024-03-24T00:21:54.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:21:53 smithi189 ceph-mon[36187]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-03-24T00:21:54.205 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:21:53 smithi189 ceph-mon[36187]: osdmap e103: 12 total, 12 up, 12 in 2024-03-24T00:21:54.205 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:21:53 smithi189 ceph-mon[36187]: mds.? [v2:172.21.15.193:6834/2264220953,v1:172.21.15.193:6835/2264220953] up:boot 2024-03-24T00:21:54.205 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:21:53 smithi189 ceph-mon[36187]: Standby daemon mds.l assigned to filesystem cephfs as rank 1 2024-03-24T00:21:54.205 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:21:53 smithi189 ceph-mon[36187]: fsmap cephfs:4/5 {0=b=up:active,2=c=up:active,3=g=up:active,4=e=up:active} 5 up:standby-replay 3 up:standby, 1 failed 2024-03-24T00:21:54.205 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:21:53 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-03-24T00:21:54.205 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:21:53 smithi189 ceph-mon[36187]: fsmap cephfs:5/5 4 up:standby-replay 3 up:standby 1 up:replay 4 up:active 2024-03-24T00:21:54.205 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:21:53 smithi189 ceph-mon[36187]: daemon mds.g restarted 2024-03-24T00:21:54.205 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:21:53 smithi189 ceph-mon[36187]: osdmap e104: 12 total, 12 up, 12 in 2024-03-24T00:21:54.205 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:21:53 smithi189 ceph-mon[36187]: mds.? [v2:172.21.15.193:6832/2276762432,v1:172.21.15.193:6833/2276762432] up:resolve 2024-03-24T00:21:54.205 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:21:53 smithi189 ceph-mon[36187]: mds.? [v2:172.21.15.118:6838/3600700815,v1:172.21.15.118:6839/3600700815] up:boot 2024-03-24T00:21:54.205 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:21:53 smithi189 ceph-mon[36187]: Standby daemon mds.h assigned to filesystem cephfs as rank 3 2024-03-24T00:21:54.205 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:21:53 smithi189 ceph-mon[36187]: fsmap cephfs:4/5 {0=b=up:active,1=l=up:resolve,2=c=up:active,4=e=up:active} 4 up:standby-replay 4 up:standby, 1 failed 2024-03-24T00:21:54.205 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:21:53 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2024-03-24T00:21:54.206 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:21:53 smithi189 ceph-mon[36187]: fsmap cephfs:5/5 3 up:standby-replay 4 up:standby 1 up:replay 1 up:resolve 3 up:active 2024-03-24T00:21:54.206 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:21:53 smithi189 ceph-mon[36187]: pgmap v598: 97 pgs: 97 active+clean; 138 MiB data, 807 MiB used, 1.0 TiB / 1.0 TiB avail; 221 KiB/s rd, 278 KiB/s wr, 41 op/s 2024-03-24T00:21:54.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:21:53 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-24T00:21:54.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:21:53 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-24T00:21:54.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:21:53 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-24T00:21:54.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:21:53 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-03-24T00:21:54.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:21:53 smithi193 ceph-mon[36878]: Adjusting osd_memory_target on smithi118 to 176.7M 2024-03-24T00:21:54.212 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:21:53 smithi193 ceph-mon[36878]: Unable to set osd_memory_target on smithi118 to 185365196: error parsing value: Value '185365196' is below minimum 939524096 2024-03-24T00:21:54.212 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:21:53 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T00:21:54.212 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:21:53 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T00:21:54.212 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:21:53 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T00:21:54.212 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:21:53 smithi193 ceph-mon[36878]: daemon mds.f restarted 2024-03-24T00:21:54.212 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:21:53 smithi193 ceph-mon[36878]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-03-24T00:21:54.212 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:21:53 smithi193 ceph-mon[36878]: osdmap e103: 12 total, 12 up, 12 in 2024-03-24T00:21:54.212 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:21:53 smithi193 ceph-mon[36878]: mds.? [v2:172.21.15.193:6834/2264220953,v1:172.21.15.193:6835/2264220953] up:boot 2024-03-24T00:21:54.212 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:21:53 smithi193 ceph-mon[36878]: Standby daemon mds.l assigned to filesystem cephfs as rank 1 2024-03-24T00:21:54.212 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:21:53 smithi193 ceph-mon[36878]: fsmap cephfs:4/5 {0=b=up:active,2=c=up:active,3=g=up:active,4=e=up:active} 5 up:standby-replay 3 up:standby, 1 failed 2024-03-24T00:21:54.212 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:21:53 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "f"}]: dispatch 2024-03-24T00:21:54.212 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:21:53 smithi193 ceph-mon[36878]: fsmap cephfs:5/5 4 up:standby-replay 3 up:standby 1 up:replay 4 up:active 2024-03-24T00:21:54.212 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:21:53 smithi193 ceph-mon[36878]: daemon mds.g restarted 2024-03-24T00:21:54.213 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:21:53 smithi193 ceph-mon[36878]: osdmap e104: 12 total, 12 up, 12 in 2024-03-24T00:21:54.213 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:21:53 smithi193 ceph-mon[36878]: mds.? [v2:172.21.15.193:6832/2276762432,v1:172.21.15.193:6833/2276762432] up:resolve 2024-03-24T00:21:54.213 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:21:53 smithi193 ceph-mon[36878]: mds.? [v2:172.21.15.118:6838/3600700815,v1:172.21.15.118:6839/3600700815] up:boot 2024-03-24T00:21:54.213 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:21:53 smithi193 ceph-mon[36878]: Standby daemon mds.h assigned to filesystem cephfs as rank 3 2024-03-24T00:21:54.213 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:21:53 smithi193 ceph-mon[36878]: fsmap cephfs:4/5 {0=b=up:active,1=l=up:resolve,2=c=up:active,4=e=up:active} 4 up:standby-replay 4 up:standby, 1 failed 2024-03-24T00:21:54.213 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:21:53 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2024-03-24T00:21:54.213 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:21:53 smithi193 ceph-mon[36878]: fsmap cephfs:5/5 3 up:standby-replay 4 up:standby 1 up:replay 1 up:resolve 3 up:active 2024-03-24T00:21:54.213 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:21:53 smithi193 ceph-mon[36878]: pgmap v598: 97 pgs: 97 active+clean; 138 MiB data, 807 MiB used, 1.0 TiB / 1.0 TiB avail; 221 KiB/s rd, 278 KiB/s wr, 41 op/s 2024-03-24T00:21:54.610 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:21:54.609+0000 7fd6b0000640 1 -- 172.21.15.118:0/3672915360 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd6a8103280 msgr2=0x7fd6a8109b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:21:54.610 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:21:54.609+0000 7fd6b0000640 1 --2- 172.21.15.118:0/3672915360 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd6a8103280 0x7fd6a8109b10 secure :-1 s=READY pgs=427 cs=0 l=1 rev1=1 crypto rx=0x7fd698009900 tx=0x7fd69802bc30 comp rx=0 tx=0).stop 2024-03-24T00:21:54.610 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:21:54.609+0000 7fd6b0000640 1 -- 172.21.15.118:0/3672915360 shutdown_connections 2024-03-24T00:21:54.610 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:21:54.609+0000 7fd6b0000640 1 --2- 172.21.15.118:0/3672915360 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd6a8103280 0x7fd6a8109b10 unknown :-1 s=CLOSED pgs=427 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:21:54.610 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:21:54.609+0000 7fd6b0000640 1 --2- 172.21.15.118:0/3672915360 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fd6a81028c0 0x7fd6a8102d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:21:54.611 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:21:54.609+0000 7fd6b0000640 1 --2- 172.21.15.118:0/3672915360 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fd6a81016d0 0x7fd6a8101af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:21:54.611 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:21:54.609+0000 7fd6b0000640 1 -- 172.21.15.118:0/3672915360 >> 172.21.15.118:0/3672915360 conn(0x7fd6a80fcc60 msgr2=0x7fd6a80ff080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:21:54.611 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:21:54.609+0000 7fd6b0000640 1 -- 172.21.15.118:0/3672915360 shutdown_connections 2024-03-24T00:21:54.611 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:21:54.609+0000 7fd6b0000640 1 -- 172.21.15.118:0/3672915360 wait complete. 2024-03-24T00:21:54.613 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:21:54.610+0000 7fd6b0000640 1 Processor -- start 2024-03-24T00:21:54.613 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:21:54.610+0000 7fd6b0000640 1 -- start start 2024-03-24T00:21:54.613 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:21:54.611+0000 7fd6b0000640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd6a81016d0 0x7fd6a8199ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:21:54.613 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:21:54.611+0000 7fd6b0000640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fd6a81028c0 0x7fd6a819a000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:21:54.613 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:21:54.611+0000 7fd6b0000640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fd6a8103280 0x7fd6a81a1070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:21:54.613 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:21:54.611+0000 7fd6b0000640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd6a810d380 con 0x7fd6a81028c0 2024-03-24T00:21:54.613 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:21:54.611+0000 7fd6b0000640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd6a810d200 con 0x7fd6a81016d0 2024-03-24T00:21:54.613 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:21:54.611+0000 7fd6b0000640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd6a810d500 con 0x7fd6a8103280 2024-03-24T00:21:54.614 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:21:54.611+0000 7fd6ad600640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd6a81016d0 0x7fd6a8199ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:21:54.614 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:21:54.611+0000 7fd6ae000640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fd6a8103280 0x7fd6a81a1070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:21:54.614 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:21:54.611+0000 7fd6ad600640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd6a81016d0 0x7fd6a8199ac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.118:36890/0 (socket says 172.21.15.118:36890) 2024-03-24T00:21:54.614 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:21:54.611+0000 7fd6ad600640 1 -- 172.21.15.118:0/1928872620 learned_addr learned my addr 172.21.15.118:0/1928872620 (peer_addr_for_me v2:172.21.15.118:0/0) 2024-03-24T00:21:54.614 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:21:54.611+0000 7fd6acc00640 1 --2- 172.21.15.118:0/1928872620 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fd6a81028c0 0x7fd6a819a000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:21:54.614 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:21:54.611+0000 7fd6ad600640 1 -- 172.21.15.118:0/1928872620 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fd6a8103280 msgr2=0x7fd6a81a1070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:21:54.614 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:21:54.611+0000 7fd6ad600640 1 --2- 172.21.15.118:0/1928872620 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fd6a8103280 0x7fd6a81a1070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:21:54.614 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:21:54.611+0000 7fd6ad600640 1 -- 172.21.15.118:0/1928872620 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fd6a81028c0 msgr2=0x7fd6a819a000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:21:54.614 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:21:54.611+0000 7fd6ad600640 1 --2- 172.21.15.118:0/1928872620 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fd6a81028c0 0x7fd6a819a000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:21:54.614 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:21:54.611+0000 7fd6ad600640 1 -- 172.21.15.118:0/1928872620 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd6a81a16e0 con 0x7fd6a81016d0 2024-03-24T00:21:54.614 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:21:54.611+0000 7fd6acc00640 1 --2- 172.21.15.118:0/1928872620 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fd6a81028c0 0x7fd6a819a000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T00:21:54.614 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:21:54.612+0000 7fd6ad600640 1 --2- 172.21.15.118:0/1928872620 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd6a81016d0 0x7fd6a8199ac0 secure :-1 s=READY pgs=428 cs=0 l=1 rev1=1 crypto rx=0x7fd6a000ebc0 tx=0x7fd6a000c6a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:21:54.614 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:21:54.612+0000 7fd6a6000640 1 -- 172.21.15.118:0/1928872620 <== mon.2 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd6a0003b70 con 0x7fd6a81016d0 2024-03-24T00:21:54.614 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:21:54.612+0000 7fd6a6000640 1 -- 172.21.15.118:0/1928872620 <== mon.2 v2:172.21.15.189:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fd6a0003d10 con 0x7fd6a81016d0 2024-03-24T00:21:54.615 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:21:54.612+0000 7fd6a6000640 1 -- 172.21.15.118:0/1928872620 <== mon.2 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd6a0019980 con 0x7fd6a81016d0 2024-03-24T00:21:54.615 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:21:54.612+0000 7fd6b0000640 1 -- 172.21.15.118:0/1928872620 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd6a81a19d0 con 0x7fd6a81016d0 2024-03-24T00:21:54.615 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:21:54.612+0000 7fd6b0000640 1 -- 172.21.15.118:0/1928872620 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd6a81a1f70 con 0x7fd6a81016d0 2024-03-24T00:21:54.617 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:21:54.616+0000 7fd6b2c01740 1 Processor -- start 2024-03-24T00:21:54.618 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:21:54.616+0000 7fd6a6000640 1 -- 172.21.15.118:0/1928872620 <== mon.2 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 20) v1 ==== 137291+0+0 (secure 0 0 0) 0x7fd6a0019b20 con 0x7fd6a81016d0 2024-03-24T00:21:54.618 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:21:54.617+0000 7fd6b2c01740 1 -- start start 2024-03-24T00:21:54.618 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:21:54.617+0000 7fd6b2c01740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55f0240ac1e0 0x55f0240ac600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:21:54.618 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:21:54.617+0000 7fd6b2c01740 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55f0240acbd0 0x55f0240b16d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:21:54.618 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:21:54.617+0000 7fd6b2c01740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55f0240b1d40 0x55f0240b41d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:21:54.618 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:21:54.617+0000 7fd6b2c01740 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f02406f3b0 con 0x55f0240acbd0 2024-03-24T00:21:54.618 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:21:54.617+0000 7fd6b2c01740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f024011830 con 0x55f0240b1d40 2024-03-24T00:21:54.618 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:21:54.617+0000 7fd6b2c01740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f023fba5d0 con 0x55f0240ac1e0 2024-03-24T00:21:54.621 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:21:54.617+0000 7fd6ae000640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55f0240ac1e0 0x55f0240ac600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:21:54.621 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:21:54.617+0000 7fd6ad600640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55f0240acbd0 0x55f0240b16d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:21:54.621 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:21:54.617+0000 7fd6acc00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55f0240b1d40 0x55f0240b41d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:21:54.621 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:21:54.617+0000 7fd6ae000640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55f0240ac1e0 0x55f0240ac600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.118:44116/0 (socket says 172.21.15.118:44116) 2024-03-24T00:21:54.621 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:21:54.617+0000 7fd6ae000640 1 -- 172.21.15.118:0/2474844522 learned_addr learned my addr 172.21.15.118:0/2474844522 (peer_addr_for_me v2:172.21.15.118:0/0) 2024-03-24T00:21:54.621 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:21:54.618+0000 7fd6ae000640 1 -- 172.21.15.118:0/2474844522 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55f0240b1d40 msgr2=0x55f0240b41d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:21:54.621 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:21:54.618+0000 7fd6ae000640 1 --2- 172.21.15.118:0/2474844522 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55f0240b1d40 0x55f0240b41d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:21:54.621 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:21:54.618+0000 7fd6ae000640 1 -- 172.21.15.118:0/2474844522 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55f0240acbd0 msgr2=0x55f0240b16d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:21:54.621 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:21:54.618+0000 7fd6ae000640 1 --2- 172.21.15.118:0/2474844522 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55f0240acbd0 0x55f0240b16d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:21:54.621 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:21:54.618+0000 7fd6ae000640 1 -- 172.21.15.118:0/2474844522 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55f023fd22c0 con 0x55f0240ac1e0 2024-03-24T00:21:54.622 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:21:54.618+0000 7fd6acc00640 1 --2- 172.21.15.118:0/2474844522 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55f0240b1d40 0x55f0240b41d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T00:21:54.622 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:21:54.618+0000 7fd6a6000640 1 --2- 172.21.15.118:0/1928872620 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7fd68c0a2920 0x7fd68c0a4de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:21:54.622 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:21:54.618+0000 7fd6ae000640 1 --2- 172.21.15.118:0/2474844522 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55f0240ac1e0 0x55f0240ac600 secure :-1 s=READY pgs=434 cs=0 l=1 rev1=1 crypto rx=0x7fd69802c8e0 tx=0x7fd69802c680 comp rx=0 tx=0).ready entity=mon.1 client_cookie=917e7db5c8ea2ee server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:21:54.622 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:21:54.618+0000 7fd6a6000640 1 -- 172.21.15.118:0/1928872620 <== mon.2 v2:172.21.15.189:3300/0 5 ==== osd_map(104..104 src has 1..104) v4 ==== 9481+0+0 (secure 0 0 0) 0x7fd6a00cf7b0 con 0x7fd6a81016d0 2024-03-24T00:21:54.622 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:21:54.618+0000 7fd6ad600640 1 --2- 172.21.15.118:0/1928872620 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7fd68c0a2920 0x7fd68c0a4de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:21:54.622 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:21:54.618+0000 7fd69e600640 1 -- 172.21.15.118:0/2474844522 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd69802c980 con 0x55f0240ac1e0 2024-03-24T00:21:54.622 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:21:54.618+0000 7fd69e600640 1 -- 172.21.15.118:0/2474844522 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fd69802cb20 con 0x55f0240ac1e0 2024-03-24T00:21:54.622 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:21:54.619+0000 7fd69e600640 1 -- 172.21.15.118:0/2474844522 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd6980046c0 con 0x55f0240ac1e0 2024-03-24T00:21:54.622 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:21:54.619+0000 7fd6b2c01740 1 -- 172.21.15.118:0/2474844522 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55f0240ac1e0 msgr2=0x55f0240ac600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:21:54.622 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:21:54.619+0000 7fd6ad600640 1 --2- 172.21.15.118:0/1928872620 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7fd68c0a2920 0x7fd68c0a4de0 secure :-1 s=READY pgs=372 cs=0 l=1 rev1=1 crypto rx=0x7fd6a00a3e80 tx=0x7fd6a00d5530 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:21:54.622 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:21:54.619+0000 7fd6b2c01740 1 --2- 172.21.15.118:0/2474844522 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55f0240ac1e0 0x55f0240ac600 secure :-1 s=READY pgs=434 cs=0 l=1 rev1=1 crypto rx=0x7fd69802c8e0 tx=0x7fd69802c680 comp rx=0 tx=0).stop 2024-03-24T00:21:54.623 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:21:54.619+0000 7fd6b2c01740 1 -- 172.21.15.118:0/2474844522 shutdown_connections 2024-03-24T00:21:54.623 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:21:54.619+0000 7fd6b2c01740 1 --2- 172.21.15.118:0/2474844522 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55f0240b1d40 0x55f0240b41d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:21:54.623 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:21:54.619+0000 7fd6b2c01740 1 --2- 172.21.15.118:0/2474844522 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55f0240acbd0 0x55f0240b16d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:21:54.623 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:21:54.619+0000 7fd6b2c01740 1 --2- 172.21.15.118:0/2474844522 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55f0240ac1e0 0x55f0240ac600 unknown :-1 s=CLOSED pgs=434 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:21:54.624 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:21:54.619+0000 7fd6b2c01740 1 -- 172.21.15.118:0/2474844522 >> 172.21.15.118:0/2474844522 conn(0x55f0240a7750 msgr2=0x55f0240a7b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:21:54.624 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:21:54.619+0000 7fd6b2c01740 1 -- 172.21.15.118:0/2474844522 shutdown_connections 2024-03-24T00:21:54.624 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:21:54.619+0000 7fd6b2c01740 1 -- 172.21.15.118:0/2474844522 wait complete. 2024-03-24T00:21:54.625 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:21:54.620+0000 7fd6b2c01740 1 -- start start 2024-03-24T00:21:54.625 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:21:54.620+0000 7fd6b2c01740 1 Processor -- start 2024-03-24T00:21:54.632 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:21:54.620+0000 7fd6b2c01740 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55f0240ac1e0 0x55f0240b63c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:21:54.632 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:21:54.620+0000 7fd6b2c01740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55f0240acbd0 0x55f0240b6900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:21:54.632 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:21:54.621+0000 7fd6b2c01740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55f0240b1d40 0x55f0240bd8d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:21:54.632 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:21:54.621+0000 7fd6b2c01740 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f023fcda20 con 0x55f0240ac1e0 2024-03-24T00:21:54.632 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:21:54.621+0000 7fd6b2c01740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f02400d8d0 con 0x55f0240acbd0 2024-03-24T00:21:54.632 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:21:54.621+0000 7fd6b2c01740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x55f0240b4c90 con 0x55f0240b1d40 2024-03-24T00:21:54.632 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:21:54.621+0000 7fd6b2c01740 10 client.0 _ll_register_callbacks cb 0x55f02400a1c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T00:21:54.632 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:21:54.621+0000 7fd6ae000640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55f0240ac1e0 0x55f0240b63c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:21:54.633 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:21:54.621+0000 7fd6ae000640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55f0240ac1e0 0x55f0240b63c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.118:3300/0 says I am v2:172.21.15.118:57422/0 (socket says 172.21.15.118:57422) 2024-03-24T00:21:54.633 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:21:54.621+0000 7fd6ae000640 1 -- 172.21.15.118:0/3895486018 learned_addr learned my addr 172.21.15.118:0/3895486018 (peer_addr_for_me v2:172.21.15.118:0/0) 2024-03-24T00:21:54.633 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:21:54.621+0000 7fd6acc00640 1 --2- 172.21.15.118:0/3895486018 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55f0240acbd0 0x55f0240b6900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:21:54.633 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:21:54.621+0000 7fd6ae000640 1 --2- 172.21.15.118:0/3895486018 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55f0240b1d40 0x55f0240bd8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:21:54.633 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:21:54.621+0000 7fd6acc00640 1 -- 172.21.15.118:0/3895486018 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55f0240b1d40 msgr2=0x55f0240bd8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:21:54.633 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:21:54.621+0000 7fd6acc00640 1 --2- 172.21.15.118:0/3895486018 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55f0240b1d40 0x55f0240bd8d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:21:54.633 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:21:54.621+0000 7fd6acc00640 1 -- 172.21.15.118:0/3895486018 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55f0240ac1e0 msgr2=0x55f0240b63c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:21:54.633 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:21:54.621+0000 7fd6acc00640 1 --2- 172.21.15.118:0/3895486018 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55f0240ac1e0 0x55f0240b63c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:21:54.633 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:21:54.621+0000 7fd6acc00640 1 -- 172.21.15.118:0/3895486018 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55f023d26210 con 0x55f0240acbd0 2024-03-24T00:21:54.633 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:21:54.621+0000 7fd6acc00640 1 --2- 172.21.15.118:0/3895486018 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55f0240acbd0 0x55f0240b6900 secure :-1 s=READY pgs=429 cs=0 l=1 rev1=1 crypto rx=0x7fd6a819afd0 tx=0x7fd694012b10 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a43e798ff246894a server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:21:54.633 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:21:54.621+0000 7fd692a00640 10 client.0 ms_handle_connect on v2:172.21.15.189:3300/0 2024-03-24T00:21:54.633 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:21:54.622+0000 7fd692a00640 1 -- 172.21.15.118:0/3895486018 <== mon.2 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd694011c70 con 0x55f0240acbd0 2024-03-24T00:21:54.633 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:21:54.622+0000 7fd6b2c01740 1 -- 172.21.15.118:0/3895486018 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fd69802cb20 con 0x55f0240acbd0 2024-03-24T00:21:54.633 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:21:54.622+0000 7fd692a00640 1 -- 172.21.15.118:0/3895486018 <== mon.2 v2:172.21.15.189:3300/0 2 ==== osd_map(104..104 src has 1..104) v4 ==== 9481+0+0 (secure 0 0 0) 0x7fd694011e10 con 0x55f0240acbd0 2024-03-24T00:21:54.634 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:21:54.622+0000 7fd692a00640 1 -- 172.21.15.118:0/3895486018 <== mon.2 v2:172.21.15.189:3300/0 3 ==== mon_get_version_reply(handle=1 version=95) v2 ==== 24+0+0 (secure 0 0 0) 0x7fd69400f460 con 0x55f0240acbd0 2024-03-24T00:21:54.634 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:21:54.622+0000 7fd6b2c01740 10 client.26807 fetch_fsmap learned FSMap version 95 2024-03-24T00:21:54.634 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:21:54.622+0000 7fd6b2c01740 1 -- 172.21.15.118:0/3895486018 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,fsmap=95,monmap=4+}) v3 -- 0x55f023df2620 con 0x55f0240acbd0 2024-03-24T00:21:54.634 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:21:54.622+0000 7fd692a00640 1 -- 172.21.15.118:0/3895486018 <== mon.2 v2:172.21.15.189:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fd69802cb20 con 0x55f0240acbd0 2024-03-24T00:21:54.634 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:21:54.622+0000 7fd692a00640 1 -- 172.21.15.118:0/3895486018 <== mon.2 v2:172.21.15.189:3300/0 5 ==== fsmap(e 95) v1 ==== 7946+0+0 (secure 0 0 0) 0x7fd69401c8a0 con 0x55f0240acbd0 2024-03-24T00:21:54.634 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:21:54.622+0000 7fd6b2c01740 10 client.26807 fetch_fsmap finished waiting for FSMap version 95 2024-03-24T00:21:54.634 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:21:54.622+0000 7fd6b2c01740 10 client.26807 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-24T00:21:54.634 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:21:54.623+0000 7fd6b2c01740 20 client.26807 populate_metadata read hostname 'smithi118' 2024-03-24T00:21:54.634 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:21:54.623+0000 7fd6b2c01740 1 --2- 172.21.15.118:0/3895486018 >> [v2:172.21.15.189:6838/4074484168,v1:172.21.15.189:6839/4074484168] conn(0x55f0240c2260 0x55f0240c4680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:21:54.634 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:21:54.623+0000 7fd6b2c01740 4 client.26807 mds_command: new command op to 24569 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-24T00:21:54.634 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:21:54.623+0000 7fd6b2c01740 1 -- 172.21.15.118:0/3895486018 --> [v2:172.21.15.189:6838/4074484168,v1:172.21.15.189:6839/4074484168] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55f02401ed70 con 0x55f0240c2260 2024-03-24T00:21:54.634 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:21:54.623+0000 7fd6ae000640 1 --2- 172.21.15.118:0/3895486018 >> [v2:172.21.15.189:6838/4074484168,v1:172.21.15.189:6839/4074484168] conn(0x55f0240c2260 0x55f0240c4680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:21:54.634 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:21:54.623+0000 7fd6ae000640 1 --2- 172.21.15.118:0/3895486018 >> [v2:172.21.15.189:6838/4074484168,v1:172.21.15.189:6839/4074484168] conn(0x55f0240c2260 0x55f0240c4680 crc :-1 s=READY pgs=236 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=c8d469133f2e76fb server_cookie=d968120e9956132f in_seq=0 out_seq=0 2024-03-24T00:21:54.635 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:21:54.623+0000 7fd692a00640 10 client.26807 ms_handle_connect on v2:172.21.15.189:6838/4074484168 2024-03-24T00:21:54.635 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:21:54.624+0000 7fd692a00640 1 -- 172.21.15.118:0/3895486018 <== mds.0 v2:172.21.15.189:6838/4074484168 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x55f02401ed70 con 0x55f0240c2260 2024-03-24T00:21:54.635 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:21:54.624+0000 7fd692a00640 10 client.26807 handle_command_reply: tid=0 2024-03-24T00:21:54.635 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:21:54.624+0000 7fd6b2c01740 1 client.26807 shutdown 2024-03-24T00:21:54.635 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:21:54.624+0000 7fd6b2c01740 1 -- 172.21.15.118:0/3895486018 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55f0240acbd0 msgr2=0x55f0240b6900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:21:54.635 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:21:54.624+0000 7fd6b2c01740 1 --2- 172.21.15.118:0/3895486018 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55f0240acbd0 0x55f0240b6900 secure :-1 s=READY pgs=429 cs=0 l=1 rev1=1 crypto rx=0x7fd6a819afd0 tx=0x7fd694012b10 comp rx=0 tx=0).stop 2024-03-24T00:21:54.635 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:21:54.624+0000 7fd6b2c01740 1 -- 172.21.15.118:0/3895486018 shutdown_connections 2024-03-24T00:21:54.635 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:21:54.624+0000 7fd6b2c01740 1 --2- 172.21.15.118:0/3895486018 >> [v2:172.21.15.189:6838/4074484168,v1:172.21.15.189:6839/4074484168] conn(0x55f0240c2260 0x55f0240c4680 crc :-1 s=READY pgs=236 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:21:54.635 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:21:54.625+0000 7fd6b2c01740 1 --2- 172.21.15.118:0/3895486018 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55f0240b1d40 0x55f0240bd8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:21:54.635 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:21:54.625+0000 7fd6b2c01740 1 --2- 172.21.15.118:0/3895486018 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55f0240acbd0 0x55f0240b6900 unknown :-1 s=CLOSED pgs=429 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:21:54.635 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:21:54.625+0000 7fd6b2c01740 1 --2- 172.21.15.118:0/3895486018 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55f0240ac1e0 0x55f0240b63c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:21:54.635 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:21:54.625+0000 7fd692a00640 0 client.26807 ms_handle_reset on v2:172.21.15.189:6838/4074484168 2024-03-24T00:21:54.635 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:21:54.625+0000 7fd6b2c01740 1 -- 172.21.15.118:0/3895486018 >> 172.21.15.118:0/3895486018 conn(0x55f0240a7750 msgr2=0x55f0240a6950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:21:54.636 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:21:54.625+0000 7fd6b2c01740 1 -- 172.21.15.118:0/3895486018 shutdown_connections 2024-03-24T00:21:54.636 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:21:54.625+0000 7fd6b2c01740 1 -- 172.21.15.118:0/3895486018 wait complete. 2024-03-24T00:21:54.636 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:21:54.625+0000 7fd6b2c01740 20 client.26807 trim_cache size 0 max 16384 2024-03-24T00:21:54.641 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:21:54.640+0000 7fd6b2c01740 1 Processor -- start 2024-03-24T00:21:54.641 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:21:54.640+0000 7fd6b2c01740 1 -- start start 2024-03-24T00:21:54.641 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:21:54.640+0000 7fd6b2c01740 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55f0240ac1e0 0x55f0240a60c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:21:54.641 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:21:54.640+0000 7fd6b2c01740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55f0240acbd0 0x55f0240bd890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:21:54.641 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:21:54.640+0000 7fd6b2c01740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55f0240b1d40 0x55f0240a6600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:21:54.642 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:21:54.640+0000 7fd6b2c01740 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd6a81ac330 con 0x55f0240ac1e0 2024-03-24T00:21:54.642 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:21:54.640+0000 7fd6b2c01740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd6a81ac9c0 con 0x55f0240acbd0 2024-03-24T00:21:54.642 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:21:54.640+0000 7fd6b2c01740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd6a81ad020 con 0x55f0240b1d40 2024-03-24T00:21:54.642 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:21:54.640+0000 7fd6ae000640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55f0240ac1e0 0x55f0240a60c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:21:54.642 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:21:54.640+0000 7fd6ae000640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55f0240ac1e0 0x55f0240a60c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.118:3300/0 says I am v2:172.21.15.118:57430/0 (socket says 172.21.15.118:57430) 2024-03-24T00:21:54.642 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:21:54.640+0000 7fd6ae000640 1 -- 172.21.15.118:0/1390392263 learned_addr learned my addr 172.21.15.118:0/1390392263 (peer_addr_for_me v2:172.21.15.118:0/0) 2024-03-24T00:21:54.642 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:21:54.640+0000 7fd6acc00640 1 --2- 172.21.15.118:0/1390392263 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55f0240acbd0 0x55f0240bd890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:21:54.642 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:21:54.640+0000 7fd6ae000640 1 --2- 172.21.15.118:0/1390392263 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55f0240b1d40 0x55f0240a6600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:21:54.642 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:21:54.641+0000 7fd6acc00640 1 -- 172.21.15.118:0/1390392263 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55f0240b1d40 msgr2=0x55f0240a6600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:21:54.642 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:21:54.641+0000 7fd6acc00640 1 --2- 172.21.15.118:0/1390392263 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55f0240b1d40 0x55f0240a6600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:21:54.642 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:21:54.641+0000 7fd6acc00640 1 -- 172.21.15.118:0/1390392263 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55f0240ac1e0 msgr2=0x55f0240a60c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:21:54.643 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:21:54.641+0000 7fd6acc00640 1 --2- 172.21.15.118:0/1390392263 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55f0240ac1e0 0x55f0240a60c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:21:54.643 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:21:54.641+0000 7fd6acc00640 1 -- 172.21.15.118:0/1390392263 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55f0240ad430 con 0x55f0240acbd0 2024-03-24T00:21:54.643 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:21:54.641+0000 7fd6ae000640 1 --2- 172.21.15.118:0/1390392263 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55f0240ac1e0 0x55f0240a60c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T00:21:54.646 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:21:54.641+0000 7fd6acc00640 1 --2- 172.21.15.118:0/1390392263 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55f0240acbd0 0x55f0240bd890 secure :-1 s=READY pgs=430 cs=0 l=1 rev1=1 crypto rx=0x7fd694009960 tx=0x7fd694009ed0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=9126b31b42d938e2 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:21:54.647 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:21:54.641+0000 7fd69dc00640 1 -- 172.21.15.118:0/1390392263 <== mon.2 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd69402e020 con 0x55f0240acbd0 2024-03-24T00:21:54.647 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:21:54.641+0000 7fd69dc00640 1 -- 172.21.15.118:0/1390392263 <== mon.2 v2:172.21.15.189:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fd6940055f0 con 0x55f0240acbd0 2024-03-24T00:21:54.647 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:21:54.641+0000 7fd69dc00640 1 -- 172.21.15.118:0/1390392263 <== mon.2 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd69401c9f0 con 0x55f0240acbd0 2024-03-24T00:21:54.647 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:21:54.642+0000 7fd6b2c01740 1 -- 172.21.15.118:0/1390392263 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55f0240acbd0 msgr2=0x55f0240bd890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:21:54.647 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:21:54.642+0000 7fd6b2c01740 1 --2- 172.21.15.118:0/1390392263 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55f0240acbd0 0x55f0240bd890 secure :-1 s=READY pgs=430 cs=0 l=1 rev1=1 crypto rx=0x7fd694009960 tx=0x7fd694009ed0 comp rx=0 tx=0).stop 2024-03-24T00:21:54.647 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:21:54.642+0000 7fd6b2c01740 1 -- 172.21.15.118:0/1390392263 shutdown_connections 2024-03-24T00:21:54.647 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:21:54.642+0000 7fd6b2c01740 1 --2- 172.21.15.118:0/1390392263 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55f0240b1d40 0x55f0240a6600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:21:54.647 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:21:54.642+0000 7fd6b2c01740 1 --2- 172.21.15.118:0/1390392263 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55f0240acbd0 0x55f0240bd890 unknown :-1 s=CLOSED pgs=430 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:21:54.647 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:21:54.642+0000 7fd6b2c01740 1 --2- 172.21.15.118:0/1390392263 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55f0240ac1e0 0x55f0240a60c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:21:54.647 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:21:54.642+0000 7fd6b2c01740 1 -- 172.21.15.118:0/1390392263 >> 172.21.15.118:0/1390392263 conn(0x55f0240a7750 msgr2=0x55f0240b63c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:21:54.647 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:21:54.642+0000 7fd6b2c01740 1 -- 172.21.15.118:0/1390392263 shutdown_connections 2024-03-24T00:21:54.648 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:21:54.642+0000 7fd6b2c01740 1 -- 172.21.15.118:0/1390392263 wait complete. 2024-03-24T00:21:54.648 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:21:54.642+0000 7fd6b2c01740 1 -- start start 2024-03-24T00:21:54.648 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:21:54.642+0000 7fd6b2c01740 1 Processor -- start 2024-03-24T00:21:54.648 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:21:54.643+0000 7fd6b2c01740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55f0240ac1e0 0x55f0240bb7c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:21:54.648 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:21:54.643+0000 7fd6b2c01740 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55f0240acbd0 0x55f0240bbd00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:21:54.648 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:21:54.643+0000 7fd6b2c01740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55f0240b1d40 0x55f0240bcd10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:21:54.648 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:21:54.643+0000 7fd6b2c01740 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd6a81a97f0 con 0x55f0240acbd0 2024-03-24T00:21:54.648 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:21:54.643+0000 7fd6b2c01740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd6a81aaa90 con 0x55f0240ac1e0 2024-03-24T00:21:54.648 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:21:54.643+0000 7fd6b2c01740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd6a81a9df0 con 0x55f0240b1d40 2024-03-24T00:21:54.648 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:21:54.643+0000 7fd6b2c01740 10 client.0 _ll_register_callbacks cb 0x7fd694009840 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T00:21:54.648 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:21:54.643+0000 7fd6ae000640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55f0240ac1e0 0x55f0240bb7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:21:54.648 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:21:54.643+0000 7fd6ae000640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55f0240ac1e0 0x55f0240bb7c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.118:36922/0 (socket says 172.21.15.118:36922) 2024-03-24T00:21:54.648 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:21:54.643+0000 7fd6ae000640 1 -- 172.21.15.118:0/2899931288 learned_addr learned my addr 172.21.15.118:0/2899931288 (peer_addr_for_me v2:172.21.15.118:0/0) 2024-03-24T00:21:54.649 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:21:54.643+0000 7fd6acc00640 1 --2- 172.21.15.118:0/2899931288 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55f0240acbd0 0x55f0240bbd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:21:54.649 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:21:54.643+0000 7fd6ae000640 1 --2- 172.21.15.118:0/2899931288 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55f0240b1d40 0x55f0240bcd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:21:54.649 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:21:54.643+0000 7fd6ae000640 1 -- 172.21.15.118:0/2899931288 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55f0240b1d40 msgr2=0x55f0240bcd10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:21:54.649 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:21:54.643+0000 7fd6ae000640 1 --2- 172.21.15.118:0/2899931288 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55f0240b1d40 0x55f0240bcd10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:21:54.649 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:21:54.643+0000 7fd6ae000640 1 -- 172.21.15.118:0/2899931288 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55f0240acbd0 msgr2=0x55f0240bbd00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:21:54.649 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:21:54.643+0000 7fd6ae000640 1 --2- 172.21.15.118:0/2899931288 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55f0240acbd0 0x55f0240bbd00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:21:54.649 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:21:54.643+0000 7fd6ae000640 1 -- 172.21.15.118:0/2899931288 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55f0240bd300 con 0x55f0240ac1e0 2024-03-24T00:21:54.649 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:21:54.644+0000 7fd6ae000640 1 --2- 172.21.15.118:0/2899931288 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55f0240ac1e0 0x55f0240bb7c0 secure :-1 s=READY pgs=431 cs=0 l=1 rev1=1 crypto rx=0x7fd6980059d0 tx=0x7fd69804b040 comp rx=0 tx=0).ready entity=mon.2 client_cookie=d305317da7d6efb8 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:21:54.649 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:21:54.644+0000 7fd692a00640 10 client.0 ms_handle_connect on v2:172.21.15.189:3300/0 2024-03-24T00:21:54.649 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:21:54.644+0000 7fd692a00640 1 -- 172.21.15.118:0/2899931288 <== mon.2 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd698004250 con 0x55f0240ac1e0 2024-03-24T00:21:54.649 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:21:54.644+0000 7fd6b2c01740 1 -- 172.21.15.118:0/2899931288 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fd6940055f0 con 0x55f0240ac1e0 2024-03-24T00:21:54.649 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:21:54.644+0000 7fd692a00640 1 -- 172.21.15.118:0/2899931288 <== mon.2 v2:172.21.15.189:3300/0 2 ==== osd_map(104..104 src has 1..104) v4 ==== 9481+0+0 (secure 0 0 0) 0x7fd6980043f0 con 0x55f0240ac1e0 2024-03-24T00:21:54.649 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:21:54.644+0000 7fd692a00640 1 -- 172.21.15.118:0/2899931288 <== mon.2 v2:172.21.15.189:3300/0 3 ==== mon_get_version_reply(handle=1 version=95) v2 ==== 24+0+0 (secure 0 0 0) 0x7fd6980566a0 con 0x55f0240ac1e0 2024-03-24T00:21:54.650 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:21:54.644+0000 7fd6b2c01740 10 client.26813 fetch_fsmap learned FSMap version 95 2024-03-24T00:21:54.650 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:21:54.644+0000 7fd6b2c01740 1 -- 172.21.15.118:0/2899931288 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,fsmap=95,monmap=4+}) v3 -- 0x55f02411e380 con 0x55f0240ac1e0 2024-03-24T00:21:54.650 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:21:54.644+0000 7fd692a00640 1 -- 172.21.15.118:0/2899931288 <== mon.2 v2:172.21.15.189:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fd6940055f0 con 0x55f0240ac1e0 2024-03-24T00:21:54.650 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:21:54.645+0000 7fd692a00640 1 -- 172.21.15.118:0/2899931288 <== mon.2 v2:172.21.15.189:3300/0 5 ==== fsmap(e 95) v1 ==== 7946+0+0 (secure 0 0 0) 0x7fd6980039e0 con 0x55f0240ac1e0 2024-03-24T00:21:54.650 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:21:54.645+0000 7fd6b2c01740 10 client.26813 fetch_fsmap finished waiting for FSMap version 95 2024-03-24T00:21:54.650 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:21:54.645+0000 7fd6b2c01740 10 client.26813 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-24T00:21:54.650 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:21:54.645+0000 7fd6b2c01740 20 client.26813 populate_metadata read hostname 'smithi118' 2024-03-24T00:21:54.650 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:21:54.645+0000 7fd6b2c01740 1 --2- 172.21.15.118:0/2899931288 >> [v2:172.21.15.189:6838/4074484168,v1:172.21.15.189:6839/4074484168] conn(0x55f02411e8e0 0x55f024120cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:21:54.650 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:21:54.645+0000 7fd6b2c01740 4 client.26813 mds_command: new command op to 24569 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-03-24T00:21:54.650 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:21:54.645+0000 7fd6b2c01740 1 -- 172.21.15.118:0/2899931288 --> [v2:172.21.15.189:6838/4074484168,v1:172.21.15.189:6839/4074484168] -- command(tid 0: {"prefix": "flush journal"}) v1 -- 0x55f0240190c0 con 0x55f02411e8e0 2024-03-24T00:21:54.650 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:21:54.645+0000 7fd6acc00640 1 --2- 172.21.15.118:0/2899931288 >> [v2:172.21.15.189:6838/4074484168,v1:172.21.15.189:6839/4074484168] conn(0x55f02411e8e0 0x55f024120cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:21:54.650 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:21:54.646+0000 7fd692a00640 10 client.26813 ms_handle_connect on v2:172.21.15.189:6838/4074484168 2024-03-24T00:21:54.650 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:21:54.646+0000 7fd6acc00640 1 --2- 172.21.15.118:0/2899931288 >> [v2:172.21.15.189:6838/4074484168,v1:172.21.15.189:6839/4074484168] conn(0x55f02411e8e0 0x55f024120cd0 crc :-1 s=READY pgs=237 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=d14693499bc896e6 server_cookie=d592f26eee8c4ca6 in_seq=0 out_seq=0 2024-03-24T00:21:56.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:21:55 smithi118 ceph-mon[31535]: mds.? [v2:172.21.15.189:6836/2350260652,v1:172.21.15.189:6837/2350260652] up:resolve 2024-03-24T00:21:56.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:21:55 smithi118 ceph-mon[31535]: fsmap cephfs:5/5 3 up:standby-replay 4 up:standby 2 up:resolve 3 up:active 2024-03-24T00:21:56.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:21:55 smithi118 ceph-mon[31535]: pgmap v599: 97 pgs: 97 active+clean; 139 MiB data, 807 MiB used, 1.0 TiB / 1.0 TiB avail; 36 KiB/s rd, 180 KiB/s wr, 26 op/s 2024-03-24T00:21:56.203 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:21:55 smithi189 ceph-mon[36187]: mds.? [v2:172.21.15.189:6836/2350260652,v1:172.21.15.189:6837/2350260652] up:resolve 2024-03-24T00:21:56.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:21:55 smithi189 ceph-mon[36187]: fsmap cephfs:5/5 3 up:standby-replay 4 up:standby 2 up:resolve 3 up:active 2024-03-24T00:21:56.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:21:55 smithi189 ceph-mon[36187]: pgmap v599: 97 pgs: 97 active+clean; 139 MiB data, 807 MiB used, 1.0 TiB / 1.0 TiB avail; 36 KiB/s rd, 180 KiB/s wr, 26 op/s 2024-03-24T00:21:56.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:21:55 smithi193 ceph-mon[36878]: mds.? [v2:172.21.15.189:6836/2350260652,v1:172.21.15.189:6837/2350260652] up:resolve 2024-03-24T00:21:56.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:21:55 smithi193 ceph-mon[36878]: fsmap cephfs:5/5 3 up:standby-replay 4 up:standby 2 up:resolve 3 up:active 2024-03-24T00:21:56.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:21:55 smithi193 ceph-mon[36878]: pgmap v599: 97 pgs: 97 active+clean; 139 MiB data, 807 MiB used, 1.0 TiB / 1.0 TiB avail; 36 KiB/s rd, 180 KiB/s wr, 26 op/s 2024-03-24T00:21:56.845 INFO:tasks.workunit.client.0.smithi118.stdout: 45 0 0 0 0 0 0 2024-03-24T00:21:56.968 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:21:56.967+0000 7fd692a00640 1 -- 172.21.15.118:0/2899931288 <== mds.0 v2:172.21.15.189:6838/4074484168 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+44 (crc 0 0 0) 0x55f0240190c0 con 0x55f02411e8e0 2024-03-24T00:21:56.968 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:21:56.967+0000 7fd692a00640 10 client.26813 handle_command_reply: tid=0 2024-03-24T00:21:56.968 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:21:56.967+0000 7fd6b2c01740 1 client.26813 shutdown 2024-03-24T00:21:56.968 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:21:56.967+0000 7fd6b2c01740 1 -- 172.21.15.118:0/2899931288 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55f0240ac1e0 msgr2=0x55f0240bb7c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:21:56.968 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:21:56.967+0000 7fd6b2c01740 1 --2- 172.21.15.118:0/2899931288 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55f0240ac1e0 0x55f0240bb7c0 secure :-1 s=READY pgs=431 cs=0 l=1 rev1=1 crypto rx=0x7fd6980059d0 tx=0x7fd69804b040 comp rx=0 tx=0).stop 2024-03-24T00:21:56.968 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:21:56.967+0000 7fd6b2c01740 1 -- 172.21.15.118:0/2899931288 shutdown_connections 2024-03-24T00:21:56.968 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:21:56.967+0000 7fd6b2c01740 1 --2- 172.21.15.118:0/2899931288 >> [v2:172.21.15.189:6838/4074484168,v1:172.21.15.189:6839/4074484168] conn(0x55f02411e8e0 0x55f024120cd0 crc :-1 s=READY pgs=237 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:21:56.968 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:21:56.967+0000 7fd6b2c01740 1 --2- 172.21.15.118:0/2899931288 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55f0240b1d40 0x55f0240bcd10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:21:56.969 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:21:56.967+0000 7fd6b2c01740 1 --2- 172.21.15.118:0/2899931288 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55f0240acbd0 0x55f0240bbd00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:21:56.969 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:21:56.967+0000 7fd6b2c01740 1 --2- 172.21.15.118:0/2899931288 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55f0240ac1e0 0x55f0240bb7c0 unknown :-1 s=CLOSED pgs=431 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:21:56.969 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:21:56.967+0000 7fd6b2c01740 1 -- 172.21.15.118:0/2899931288 >> 172.21.15.118:0/2899931288 conn(0x55f0240a7750 msgr2=0x55f0240b63a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:21:56.969 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:21:56.968+0000 7fd692a00640 0 client.26813 ms_handle_reset on v2:172.21.15.189:6838/4074484168 2024-03-24T00:21:56.969 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:21:56.968+0000 7fd6b2c01740 1 -- 172.21.15.118:0/2899931288 shutdown_connections 2024-03-24T00:21:56.969 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:21:56.968+0000 7fd6b2c01740 1 -- 172.21.15.118:0/2899931288 wait complete. 2024-03-24T00:21:56.970 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:21:56.968+0000 7fd6b2c01740 20 client.26813 trim_cache size 0 max 16384 2024-03-24T00:21:56.970 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stdout:{ 2024-03-24T00:21:56.970 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stdout: "message": "", 2024-03-24T00:21:56.970 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stdout: "return_code": 0 2024-03-24T00:21:56.970 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stdout:} 2024-03-24T00:21:56.971 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:21:56.970+0000 7fd6b0000640 1 -- 172.21.15.118:0/1928872620 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7fd68c0a2920 msgr2=0x7fd68c0a4de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:21:56.971 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:21:56.970+0000 7fd6b0000640 1 --2- 172.21.15.118:0/1928872620 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7fd68c0a2920 0x7fd68c0a4de0 secure :-1 s=READY pgs=372 cs=0 l=1 rev1=1 crypto rx=0x7fd6a00a3e80 tx=0x7fd6a00d5530 comp rx=0 tx=0).stop 2024-03-24T00:21:56.971 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:21:56.970+0000 7fd6b0000640 1 -- 172.21.15.118:0/1928872620 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd6a81016d0 msgr2=0x7fd6a8199ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:21:56.971 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:21:56.970+0000 7fd6b0000640 1 --2- 172.21.15.118:0/1928872620 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd6a81016d0 0x7fd6a8199ac0 secure :-1 s=READY pgs=428 cs=0 l=1 rev1=1 crypto rx=0x7fd6a000ebc0 tx=0x7fd6a000c6a0 comp rx=0 tx=0).stop 2024-03-24T00:21:56.971 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:21:56.970+0000 7fd6b0000640 1 -- 172.21.15.118:0/1928872620 shutdown_connections 2024-03-24T00:21:56.971 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:21:56.970+0000 7fd6b0000640 1 --2- 172.21.15.118:0/1928872620 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7fd68c0a2920 0x7fd68c0a4de0 unknown :-1 s=CLOSED pgs=372 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:21:56.971 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:21:56.970+0000 7fd6b0000640 1 --2- 172.21.15.118:0/1928872620 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fd6a8103280 0x7fd6a81a1070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:21:56.971 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:21:56.970+0000 7fd6b0000640 1 --2- 172.21.15.118:0/1928872620 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fd6a81028c0 0x7fd6a819a000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:21:56.971 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:21:56.970+0000 7fd6b0000640 1 --2- 172.21.15.118:0/1928872620 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd6a81016d0 0x7fd6a8199ac0 unknown :-1 s=CLOSED pgs=428 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:21:56.972 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:21:56.970+0000 7fd6b0000640 1 -- 172.21.15.118:0/1928872620 >> 172.21.15.118:0/1928872620 conn(0x7fd6a80fcc60 msgr2=0x7fd6a80738d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:21:56.972 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:21:56.970+0000 7fd6b0000640 1 -- 172.21.15.118:0/1928872620 shutdown_connections 2024-03-24T00:21:56.972 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:21:56.970+0000 7fd6b0000640 1 -- 172.21.15.118:0/1928872620 wait complete. 2024-03-24T00:21:57.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:21:56 smithi118 ceph-mon[31535]: mds.? [v2:172.21.15.193:6832/2276762432,v1:172.21.15.193:6833/2276762432] up:reconnect 2024-03-24T00:21:57.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:21:56 smithi118 ceph-mon[31535]: mds.? [v2:172.21.15.189:6836/2350260652,v1:172.21.15.189:6837/2350260652] up:reconnect 2024-03-24T00:21:57.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:21:56 smithi118 ceph-mon[31535]: fsmap cephfs:5/5 3 up:standby-replay 4 up:standby 2 up:reconnect 3 up:active 2024-03-24T00:21:57.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:21:56 smithi118 ceph-mon[31535]: reconnect by client.15159 192.168.0.1:0/3463739313 after 0.0259996 2024-03-24T00:21:57.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:21:56 smithi118 ceph-mon[31535]: reconnect by client.15159 192.168.0.1:0/3463739313 after 0.0399993 2024-03-24T00:21:57.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:21:56 smithi189 ceph-mon[36187]: mds.? [v2:172.21.15.193:6832/2276762432,v1:172.21.15.193:6833/2276762432] up:reconnect 2024-03-24T00:21:57.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:21:56 smithi189 ceph-mon[36187]: mds.? [v2:172.21.15.189:6836/2350260652,v1:172.21.15.189:6837/2350260652] up:reconnect 2024-03-24T00:21:57.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:21:56 smithi189 ceph-mon[36187]: fsmap cephfs:5/5 3 up:standby-replay 4 up:standby 2 up:reconnect 3 up:active 2024-03-24T00:21:57.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:21:56 smithi189 ceph-mon[36187]: reconnect by client.15159 192.168.0.1:0/3463739313 after 0.0259996 2024-03-24T00:21:57.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:21:56 smithi189 ceph-mon[36187]: reconnect by client.15159 192.168.0.1:0/3463739313 after 0.0399993 2024-03-24T00:21:57.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:21:56 smithi193 ceph-mon[36878]: mds.? [v2:172.21.15.193:6832/2276762432,v1:172.21.15.193:6833/2276762432] up:reconnect 2024-03-24T00:21:57.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:21:56 smithi193 ceph-mon[36878]: mds.? [v2:172.21.15.189:6836/2350260652,v1:172.21.15.189:6837/2350260652] up:reconnect 2024-03-24T00:21:57.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:21:56 smithi193 ceph-mon[36878]: fsmap cephfs:5/5 3 up:standby-replay 4 up:standby 2 up:reconnect 3 up:active 2024-03-24T00:21:57.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:21:56 smithi193 ceph-mon[36878]: reconnect by client.15159 192.168.0.1:0/3463739313 after 0.0259996 2024-03-24T00:21:57.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:21:56 smithi193 ceph-mon[36878]: reconnect by client.15159 192.168.0.1:0/3463739313 after 0.0399993 2024-03-24T00:21:58.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:21:57 smithi193 ceph-mon[36878]: mds.? [v2:172.21.15.193:6832/2276762432,v1:172.21.15.193:6833/2276762432] up:rejoin 2024-03-24T00:21:58.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:21:57 smithi193 ceph-mon[36878]: mds.? [v2:172.21.15.189:6836/2350260652,v1:172.21.15.189:6837/2350260652] up:rejoin 2024-03-24T00:21:58.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:21:57 smithi193 ceph-mon[36878]: fsmap cephfs:5/5 3 up:standby-replay 4 up:standby 2 up:rejoin 3 up:active 2024-03-24T00:21:58.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:21:57 smithi193 ceph-mon[36878]: daemon mds.h is now active in filesystem cephfs as rank 3 2024-03-24T00:21:58.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:21:57 smithi193 ceph-mon[36878]: daemon mds.l is now active in filesystem cephfs as rank 1 2024-03-24T00:21:58.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:21:57 smithi193 ceph-mon[36878]: pgmap v600: 97 pgs: 97 active+clean; 135 MiB data, 805 MiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 83 KiB/s wr, 39 op/s 2024-03-24T00:21:58.212 INFO:journalctl@ceph.mds.l.smithi193.stdout:Mar 24 00:21:58 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-l[72021]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: In function 'void Locker::set_xlocks_done(MutationImpl*, bool)' thread 7f6375d64700 time 2024-03-24T00:21:58.016977+0000 2024-03-24T00:21:58.212 INFO:journalctl@ceph.mds.l.smithi193.stdout:Mar 24 00:21:58 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-l[72021]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: 626: FAILED ceph_assert(obj->is_auth()) 2024-03-24T00:21:58.212 INFO:journalctl@ceph.mds.l.smithi193.stdout:Mar 24 00:21:58 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-l[72021]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-24T00:21:58.212 INFO:journalctl@ceph.mds.l.smithi193.stdout:Mar 24 00:21:58 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-l[72021]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x135) [0x7f638038e4a1] 2024-03-24T00:21:58.212 INFO:journalctl@ceph.mds.l.smithi193.stdout:Mar 24 00:21:58 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-l[72021]: 2: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7f638038e667] 2024-03-24T00:21:58.212 INFO:journalctl@ceph.mds.l.smithi193.stdout:Mar 24 00:21:58 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-l[72021]: 3: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x5598cb54c223] 2024-03-24T00:21:58.212 INFO:journalctl@ceph.mds.l.smithi193.stdout:Mar 24 00:21:58 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-l[72021]: 4: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x5598cb3c65d6] 2024-03-24T00:21:58.212 INFO:journalctl@ceph.mds.l.smithi193.stdout:Mar 24 00:21:58 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-l[72021]: 5: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x5598cb3c6da1] 2024-03-24T00:21:58.212 INFO:journalctl@ceph.mds.l.smithi193.stdout:Mar 24 00:21:58 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-l[72021]: 6: (Server::handle_client_rename(boost::intrusive_ptr const&)+0x259d) [0x5598cb40397d] 2024-03-24T00:21:58.213 INFO:journalctl@ceph.mds.l.smithi193.stdout:Mar 24 00:21:58 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-l[72021]: 7: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x5598cb419233] 2024-03-24T00:21:58.213 INFO:journalctl@ceph.mds.l.smithi193.stdout:Mar 24 00:21:58 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-l[72021]: 8: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x5598cb41b197] 2024-03-24T00:21:58.213 INFO:journalctl@ceph.mds.l.smithi193.stdout:Mar 24 00:21:58 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-l[72021]: 9: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x5598cb31a11b] 2024-03-24T00:21:58.213 INFO:journalctl@ceph.mds.l.smithi193.stdout:Mar 24 00:21:58 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-l[72021]: 10: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x5598cb31a6bc] 2024-03-24T00:21:58.213 INFO:journalctl@ceph.mds.l.smithi193.stdout:Mar 24 00:21:58 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-l[72021]: 11: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x5598cb3041ff] 2024-03-24T00:21:58.213 INFO:journalctl@ceph.mds.l.smithi193.stdout:Mar 24 00:21:58 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-l[72021]: 12: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7f6380620488] 2024-03-24T00:21:58.213 INFO:journalctl@ceph.mds.l.smithi193.stdout:Mar 24 00:21:58 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-l[72021]: 13: (DispatchQueue::entry()+0x51f) [0x7f638061d60f] 2024-03-24T00:21:58.213 INFO:journalctl@ceph.mds.l.smithi193.stdout:Mar 24 00:21:58 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-l[72021]: 14: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f63806f0e81] 2024-03-24T00:21:58.213 INFO:journalctl@ceph.mds.l.smithi193.stdout:Mar 24 00:21:58 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-l[72021]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7f637f10b1ca] 2024-03-24T00:21:58.213 INFO:journalctl@ceph.mds.l.smithi193.stdout:Mar 24 00:21:58 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-l[72021]: 16: clone() 2024-03-24T00:21:58.213 INFO:journalctl@ceph.mds.l.smithi193.stdout:Mar 24 00:21:58 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-l[72021]: 2024-03-24T00:21:58.017+0000 7f6375d64700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: In function 'void Locker::set_xlocks_done(MutationImpl*, bool)' thread 7f6375d64700 time 2024-03-24T00:21:58.016977+0000 2024-03-24T00:21:58.213 INFO:journalctl@ceph.mds.l.smithi193.stdout:Mar 24 00:21:58 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-l[72021]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: 626: FAILED ceph_assert(obj->is_auth()) 2024-03-24T00:21:58.213 INFO:journalctl@ceph.mds.l.smithi193.stdout:Mar 24 00:21:58 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-l[72021]: 2024-03-24T00:21:58.213 INFO:journalctl@ceph.mds.l.smithi193.stdout:Mar 24 00:21:58 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-l[72021]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-24T00:21:58.213 INFO:journalctl@ceph.mds.l.smithi193.stdout:Mar 24 00:21:58 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-l[72021]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x135) [0x7f638038e4a1] 2024-03-24T00:21:58.214 INFO:journalctl@ceph.mds.l.smithi193.stdout:Mar 24 00:21:58 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-l[72021]: 2: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7f638038e667] 2024-03-24T00:21:58.214 INFO:journalctl@ceph.mds.l.smithi193.stdout:Mar 24 00:21:58 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-l[72021]: 3: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x5598cb54c223] 2024-03-24T00:21:58.214 INFO:journalctl@ceph.mds.l.smithi193.stdout:Mar 24 00:21:58 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-l[72021]: 4: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x5598cb3c65d6] 2024-03-24T00:21:58.214 INFO:journalctl@ceph.mds.l.smithi193.stdout:Mar 24 00:21:58 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-l[72021]: 5: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x5598cb3c6da1] 2024-03-24T00:21:58.214 INFO:journalctl@ceph.mds.l.smithi193.stdout:Mar 24 00:21:58 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-l[72021]: 6: (Server::handle_client_rename(boost::intrusive_ptr const&)+0x259d) [0x5598cb40397d] 2024-03-24T00:21:58.214 INFO:journalctl@ceph.mds.l.smithi193.stdout:Mar 24 00:21:58 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-l[72021]: 7: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x5598cb419233] 2024-03-24T00:21:58.214 INFO:journalctl@ceph.mds.l.smithi193.stdout:Mar 24 00:21:58 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-l[72021]: 8: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x5598cb41b197] 2024-03-24T00:21:58.214 INFO:journalctl@ceph.mds.l.smithi193.stdout:Mar 24 00:21:58 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-l[72021]: 9: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x5598cb31a11b] 2024-03-24T00:21:58.214 INFO:journalctl@ceph.mds.l.smithi193.stdout:Mar 24 00:21:58 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-l[72021]: 10: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x5598cb31a6bc] 2024-03-24T00:21:58.214 INFO:journalctl@ceph.mds.l.smithi193.stdout:Mar 24 00:21:58 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-l[72021]: 11: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x5598cb3041ff] 2024-03-24T00:21:58.214 INFO:journalctl@ceph.mds.l.smithi193.stdout:Mar 24 00:21:58 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-l[72021]: 12: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7f6380620488] 2024-03-24T00:21:58.214 INFO:journalctl@ceph.mds.l.smithi193.stdout:Mar 24 00:21:58 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-l[72021]: 13: (DispatchQueue::entry()+0x51f) [0x7f638061d60f] 2024-03-24T00:21:58.214 INFO:journalctl@ceph.mds.l.smithi193.stdout:Mar 24 00:21:58 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-l[72021]: 14: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f63806f0e81] 2024-03-24T00:21:58.214 INFO:journalctl@ceph.mds.l.smithi193.stdout:Mar 24 00:21:58 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-l[72021]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7f637f10b1ca] 2024-03-24T00:21:58.215 INFO:journalctl@ceph.mds.l.smithi193.stdout:Mar 24 00:21:58 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-l[72021]: 16: clone() 2024-03-24T00:21:58.215 INFO:journalctl@ceph.mds.l.smithi193.stdout:Mar 24 00:21:58 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-l[72021]: 2024-03-24T00:21:58.215 INFO:journalctl@ceph.mds.l.smithi193.stdout:Mar 24 00:21:58 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-l[72021]: *** Caught signal (Aborted) ** 2024-03-24T00:21:58.215 INFO:journalctl@ceph.mds.l.smithi193.stdout:Mar 24 00:21:58 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-l[72021]: in thread 7f6375d64700 thread_name:ms_dispatch 2024-03-24T00:21:58.215 INFO:journalctl@ceph.mds.l.smithi193.stdout:Mar 24 00:21:58 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-l[72021]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-24T00:21:58.215 INFO:journalctl@ceph.mds.l.smithi193.stdout:Mar 24 00:21:58 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-l[72021]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f637f115d20] 2024-03-24T00:21:58.215 INFO:journalctl@ceph.mds.l.smithi193.stdout:Mar 24 00:21:58 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-l[72021]: 2: gsignal() 2024-03-24T00:21:58.215 INFO:journalctl@ceph.mds.l.smithi193.stdout:Mar 24 00:21:58 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-l[72021]: 3: abort() 2024-03-24T00:21:58.215 INFO:journalctl@ceph.mds.l.smithi193.stdout:Mar 24 00:21:58 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-l[72021]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x18f) [0x7f638038e4fb] 2024-03-24T00:21:58.215 INFO:journalctl@ceph.mds.l.smithi193.stdout:Mar 24 00:21:58 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-l[72021]: 5: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7f638038e667] 2024-03-24T00:21:58.215 INFO:journalctl@ceph.mds.l.smithi193.stdout:Mar 24 00:21:58 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-l[72021]: 6: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x5598cb54c223] 2024-03-24T00:21:58.215 INFO:journalctl@ceph.mds.l.smithi193.stdout:Mar 24 00:21:58 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-l[72021]: 7: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x5598cb3c65d6] 2024-03-24T00:21:58.215 INFO:journalctl@ceph.mds.l.smithi193.stdout:Mar 24 00:21:58 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-l[72021]: 8: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x5598cb3c6da1] 2024-03-24T00:21:58.215 INFO:journalctl@ceph.mds.l.smithi193.stdout:Mar 24 00:21:58 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-l[72021]: 9: (Server::handle_client_rename(boost::intrusive_ptr const&)+0x259d) [0x5598cb40397d] 2024-03-24T00:21:58.215 INFO:journalctl@ceph.mds.l.smithi193.stdout:Mar 24 00:21:58 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-l[72021]: 10: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x5598cb419233] 2024-03-24T00:21:58.216 INFO:journalctl@ceph.mds.l.smithi193.stdout:Mar 24 00:21:58 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-l[72021]: 11: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x5598cb41b197] 2024-03-24T00:21:58.216 INFO:journalctl@ceph.mds.l.smithi193.stdout:Mar 24 00:21:58 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-l[72021]: 12: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x5598cb31a11b] 2024-03-24T00:21:58.216 INFO:journalctl@ceph.mds.l.smithi193.stdout:Mar 24 00:21:58 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-l[72021]: 13: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x5598cb31a6bc] 2024-03-24T00:21:58.216 INFO:journalctl@ceph.mds.l.smithi193.stdout:Mar 24 00:21:58 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-l[72021]: 14: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x5598cb3041ff] 2024-03-24T00:21:58.216 INFO:journalctl@ceph.mds.l.smithi193.stdout:Mar 24 00:21:58 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-l[72021]: 15: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7f6380620488] 2024-03-24T00:21:58.216 INFO:journalctl@ceph.mds.l.smithi193.stdout:Mar 24 00:21:58 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-l[72021]: 16: (DispatchQueue::entry()+0x51f) [0x7f638061d60f] 2024-03-24T00:21:58.216 INFO:journalctl@ceph.mds.l.smithi193.stdout:Mar 24 00:21:58 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-l[72021]: 17: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f63806f0e81] 2024-03-24T00:21:58.216 INFO:journalctl@ceph.mds.l.smithi193.stdout:Mar 24 00:21:58 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-l[72021]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7f637f10b1ca] 2024-03-24T00:21:58.216 INFO:journalctl@ceph.mds.l.smithi193.stdout:Mar 24 00:21:58 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-l[72021]: 19: clone() 2024-03-24T00:21:58.216 INFO:journalctl@ceph.mds.l.smithi193.stdout:Mar 24 00:21:58 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-l[72021]: 2024-03-24T00:21:58.019+0000 7f6375d64700 -1 *** Caught signal (Aborted) ** 2024-03-24T00:21:58.216 INFO:journalctl@ceph.mds.l.smithi193.stdout:Mar 24 00:21:58 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-l[72021]: in thread 7f6375d64700 thread_name:ms_dispatch 2024-03-24T00:21:58.216 INFO:journalctl@ceph.mds.l.smithi193.stdout:Mar 24 00:21:58 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-l[72021]: 2024-03-24T00:21:58.216 INFO:journalctl@ceph.mds.l.smithi193.stdout:Mar 24 00:21:58 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-l[72021]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-24T00:21:58.216 INFO:journalctl@ceph.mds.l.smithi193.stdout:Mar 24 00:21:58 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-l[72021]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f637f115d20] 2024-03-24T00:21:58.216 INFO:journalctl@ceph.mds.l.smithi193.stdout:Mar 24 00:21:58 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-l[72021]: 2: gsignal() 2024-03-24T00:21:58.216 INFO:journalctl@ceph.mds.l.smithi193.stdout:Mar 24 00:21:58 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-l[72021]: 3: abort() 2024-03-24T00:21:58.217 INFO:journalctl@ceph.mds.l.smithi193.stdout:Mar 24 00:21:58 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-l[72021]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x18f) [0x7f638038e4fb] 2024-03-24T00:21:58.217 INFO:journalctl@ceph.mds.l.smithi193.stdout:Mar 24 00:21:58 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-l[72021]: 5: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7f638038e667] 2024-03-24T00:21:58.218 INFO:journalctl@ceph.mds.l.smithi193.stdout:Mar 24 00:21:58 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-l[72021]: 6: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x5598cb54c223] 2024-03-24T00:21:58.218 INFO:journalctl@ceph.mds.l.smithi193.stdout:Mar 24 00:21:58 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-l[72021]: 7: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x5598cb3c65d6] 2024-03-24T00:21:58.218 INFO:journalctl@ceph.mds.l.smithi193.stdout:Mar 24 00:21:58 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-l[72021]: 8: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x5598cb3c6da1] 2024-03-24T00:21:58.218 INFO:journalctl@ceph.mds.l.smithi193.stdout:Mar 24 00:21:58 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-l[72021]: 9: (Server::handle_client_rename(boost::intrusive_ptr const&)+0x259d) [0x5598cb40397d] 2024-03-24T00:21:58.218 INFO:journalctl@ceph.mds.l.smithi193.stdout:Mar 24 00:21:58 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-l[72021]: 10: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x5598cb419233] 2024-03-24T00:21:58.218 INFO:journalctl@ceph.mds.l.smithi193.stdout:Mar 24 00:21:58 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-l[72021]: 11: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x5598cb41b197] 2024-03-24T00:21:58.218 INFO:journalctl@ceph.mds.l.smithi193.stdout:Mar 24 00:21:58 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-l[72021]: 12: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x5598cb31a11b] 2024-03-24T00:21:58.218 INFO:journalctl@ceph.mds.l.smithi193.stdout:Mar 24 00:21:58 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-l[72021]: 13: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x5598cb31a6bc] 2024-03-24T00:21:58.218 INFO:journalctl@ceph.mds.l.smithi193.stdout:Mar 24 00:21:58 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-l[72021]: 14: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x5598cb3041ff] 2024-03-24T00:21:58.218 INFO:journalctl@ceph.mds.l.smithi193.stdout:Mar 24 00:21:58 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-l[72021]: 15: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7f6380620488] 2024-03-24T00:21:58.218 INFO:journalctl@ceph.mds.l.smithi193.stdout:Mar 24 00:21:58 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-l[72021]: 16: (DispatchQueue::entry()+0x51f) [0x7f638061d60f] 2024-03-24T00:21:58.218 INFO:journalctl@ceph.mds.l.smithi193.stdout:Mar 24 00:21:58 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-l[72021]: 17: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f63806f0e81] 2024-03-24T00:21:58.219 INFO:journalctl@ceph.mds.l.smithi193.stdout:Mar 24 00:21:58 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-l[72021]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7f637f10b1ca] 2024-03-24T00:21:58.219 INFO:journalctl@ceph.mds.l.smithi193.stdout:Mar 24 00:21:58 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-l[72021]: 19: clone() 2024-03-24T00:21:58.219 INFO:journalctl@ceph.mds.l.smithi193.stdout:Mar 24 00:21:58 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-l[72021]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-03-24T00:21:58.219 INFO:journalctl@ceph.mds.l.smithi193.stdout:Mar 24 00:21:58 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-l[72021]: 2024-03-24T00:21:58.219 INFO:journalctl@ceph.mds.l.smithi193.stdout:Mar 24 00:21:58 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-l[72021]: -1> 2024-03-24T00:21:58.017+0000 7f6375d64700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: In function 'void Locker::set_xlocks_done(MutationImpl*, bool)' thread 7f6375d64700 time 2024-03-24T00:21:58.016977+0000 2024-03-24T00:21:58.219 INFO:journalctl@ceph.mds.l.smithi193.stdout:Mar 24 00:21:58 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-l[72021]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: 626: FAILED ceph_assert(obj->is_auth()) 2024-03-24T00:21:58.219 INFO:journalctl@ceph.mds.l.smithi193.stdout:Mar 24 00:21:58 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-l[72021]: 2024-03-24T00:21:58.219 INFO:journalctl@ceph.mds.l.smithi193.stdout:Mar 24 00:21:58 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-l[72021]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-24T00:21:58.219 INFO:journalctl@ceph.mds.l.smithi193.stdout:Mar 24 00:21:58 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-l[72021]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x135) [0x7f638038e4a1] 2024-03-24T00:21:58.219 INFO:journalctl@ceph.mds.l.smithi193.stdout:Mar 24 00:21:58 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-l[72021]: 2: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7f638038e667] 2024-03-24T00:21:58.219 INFO:journalctl@ceph.mds.l.smithi193.stdout:Mar 24 00:21:58 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-l[72021]: 3: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x5598cb54c223] 2024-03-24T00:21:58.219 INFO:journalctl@ceph.mds.l.smithi193.stdout:Mar 24 00:21:58 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-l[72021]: 4: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x5598cb3c65d6] 2024-03-24T00:21:58.219 INFO:journalctl@ceph.mds.l.smithi193.stdout:Mar 24 00:21:58 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-l[72021]: 5: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x5598cb3c6da1] 2024-03-24T00:21:58.219 INFO:journalctl@ceph.mds.l.smithi193.stdout:Mar 24 00:21:58 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-l[72021]: 6: (Server::handle_client_rename(boost::intrusive_ptr const&)+0x259d) [0x5598cb40397d] 2024-03-24T00:21:58.219 INFO:journalctl@ceph.mds.l.smithi193.stdout:Mar 24 00:21:58 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-l[72021]: 7: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x5598cb419233] 2024-03-24T00:21:58.220 INFO:journalctl@ceph.mds.l.smithi193.stdout:Mar 24 00:21:58 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-l[72021]: 8: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x5598cb41b197] 2024-03-24T00:21:58.220 INFO:journalctl@ceph.mds.l.smithi193.stdout:Mar 24 00:21:58 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-l[72021]: 9: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x5598cb31a11b] 2024-03-24T00:21:58.220 INFO:journalctl@ceph.mds.l.smithi193.stdout:Mar 24 00:21:58 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-l[72021]: 10: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x5598cb31a6bc] 2024-03-24T00:21:58.220 INFO:journalctl@ceph.mds.l.smithi193.stdout:Mar 24 00:21:58 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-l[72021]: 11: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x5598cb3041ff] 2024-03-24T00:21:58.220 INFO:journalctl@ceph.mds.l.smithi193.stdout:Mar 24 00:21:58 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-l[72021]: 12: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7f6380620488] 2024-03-24T00:21:58.220 INFO:journalctl@ceph.mds.l.smithi193.stdout:Mar 24 00:21:58 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-l[72021]: 13: (DispatchQueue::entry()+0x51f) [0x7f638061d60f] 2024-03-24T00:21:58.220 INFO:journalctl@ceph.mds.l.smithi193.stdout:Mar 24 00:21:58 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-l[72021]: 14: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f63806f0e81] 2024-03-24T00:21:58.220 INFO:journalctl@ceph.mds.l.smithi193.stdout:Mar 24 00:21:58 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-l[72021]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7f637f10b1ca] 2024-03-24T00:21:58.220 INFO:journalctl@ceph.mds.l.smithi193.stdout:Mar 24 00:21:58 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-l[72021]: 16: clone() 2024-03-24T00:21:58.220 INFO:journalctl@ceph.mds.l.smithi193.stdout:Mar 24 00:21:58 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-l[72021]: 2024-03-24T00:21:58.220 INFO:journalctl@ceph.mds.l.smithi193.stdout:Mar 24 00:21:58 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-l[72021]: 0> 2024-03-24T00:21:58.019+0000 7f6375d64700 -1 *** Caught signal (Aborted) ** 2024-03-24T00:21:58.220 INFO:journalctl@ceph.mds.l.smithi193.stdout:Mar 24 00:21:58 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-l[72021]: in thread 7f6375d64700 thread_name:ms_dispatch 2024-03-24T00:21:58.220 INFO:journalctl@ceph.mds.l.smithi193.stdout:Mar 24 00:21:58 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-l[72021]: 2024-03-24T00:21:58.220 INFO:journalctl@ceph.mds.l.smithi193.stdout:Mar 24 00:21:58 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-l[72021]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-24T00:21:58.220 INFO:journalctl@ceph.mds.l.smithi193.stdout:Mar 24 00:21:58 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-l[72021]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f637f115d20] 2024-03-24T00:21:58.221 INFO:journalctl@ceph.mds.l.smithi193.stdout:Mar 24 00:21:58 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-l[72021]: 2: gsignal() 2024-03-24T00:21:58.221 INFO:journalctl@ceph.mds.l.smithi193.stdout:Mar 24 00:21:58 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-l[72021]: 3: abort() 2024-03-24T00:21:58.221 INFO:journalctl@ceph.mds.l.smithi193.stdout:Mar 24 00:21:58 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-l[72021]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x18f) [0x7f638038e4fb] 2024-03-24T00:21:58.221 INFO:journalctl@ceph.mds.l.smithi193.stdout:Mar 24 00:21:58 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-l[72021]: 5: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7f638038e667] 2024-03-24T00:21:58.221 INFO:journalctl@ceph.mds.l.smithi193.stdout:Mar 24 00:21:58 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-l[72021]: 6: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x5598cb54c223] 2024-03-24T00:21:58.221 INFO:journalctl@ceph.mds.l.smithi193.stdout:Mar 24 00:21:58 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-l[72021]: 7: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x5598cb3c65d6] 2024-03-24T00:21:58.221 INFO:journalctl@ceph.mds.l.smithi193.stdout:Mar 24 00:21:58 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-l[72021]: 8: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x5598cb3c6da1] 2024-03-24T00:21:58.221 INFO:journalctl@ceph.mds.l.smithi193.stdout:Mar 24 00:21:58 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-l[72021]: 9: (Server::handle_client_rename(boost::intrusive_ptr const&)+0x259d) [0x5598cb40397d] 2024-03-24T00:21:58.221 INFO:journalctl@ceph.mds.l.smithi193.stdout:Mar 24 00:21:58 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-l[72021]: 10: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x5598cb419233] 2024-03-24T00:21:58.221 INFO:journalctl@ceph.mds.l.smithi193.stdout:Mar 24 00:21:58 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-l[72021]: 11: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x5598cb41b197] 2024-03-24T00:21:58.221 INFO:journalctl@ceph.mds.l.smithi193.stdout:Mar 24 00:21:58 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-l[72021]: 12: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x5598cb31a11b] 2024-03-24T00:21:58.221 INFO:journalctl@ceph.mds.l.smithi193.stdout:Mar 24 00:21:58 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-l[72021]: 13: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x5598cb31a6bc] 2024-03-24T00:21:58.221 INFO:journalctl@ceph.mds.l.smithi193.stdout:Mar 24 00:21:58 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-l[72021]: 14: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x5598cb3041ff] 2024-03-24T00:21:58.221 INFO:journalctl@ceph.mds.l.smithi193.stdout:Mar 24 00:21:58 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-l[72021]: 15: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7f6380620488] 2024-03-24T00:21:58.221 INFO:journalctl@ceph.mds.l.smithi193.stdout:Mar 24 00:21:58 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-l[72021]: 16: (DispatchQueue::entry()+0x51f) [0x7f638061d60f] 2024-03-24T00:21:58.222 INFO:journalctl@ceph.mds.l.smithi193.stdout:Mar 24 00:21:58 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-l[72021]: 17: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f63806f0e81] 2024-03-24T00:21:58.222 INFO:journalctl@ceph.mds.l.smithi193.stdout:Mar 24 00:21:58 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-l[72021]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7f637f10b1ca] 2024-03-24T00:21:58.222 INFO:journalctl@ceph.mds.l.smithi193.stdout:Mar 24 00:21:58 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-l[72021]: 19: clone() 2024-03-24T00:21:58.222 INFO:journalctl@ceph.mds.l.smithi193.stdout:Mar 24 00:21:58 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-l[72021]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-03-24T00:21:58.222 INFO:journalctl@ceph.mds.l.smithi193.stdout:Mar 24 00:21:58 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-l[72021]: 2024-03-24T00:21:58.222 INFO:journalctl@ceph.mds.l.smithi193.stdout:Mar 24 00:21:58 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-l[72021]: -1> 2024-03-24T00:21:58.017+0000 7f6375d64700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: In function 'void Locker::set_xlocks_done(MutationImpl*, bool)' thread 7f6375d64700 time 2024-03-24T00:21:58.016977+0000 2024-03-24T00:21:58.222 INFO:journalctl@ceph.mds.l.smithi193.stdout:Mar 24 00:21:58 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-l[72021]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: 626: FAILED ceph_assert(obj->is_auth()) 2024-03-24T00:21:58.222 INFO:journalctl@ceph.mds.l.smithi193.stdout:Mar 24 00:21:58 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-l[72021]: 2024-03-24T00:21:58.222 INFO:journalctl@ceph.mds.l.smithi193.stdout:Mar 24 00:21:58 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-l[72021]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-24T00:21:58.222 INFO:journalctl@ceph.mds.l.smithi193.stdout:Mar 24 00:21:58 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-l[72021]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x135) [0x7f638038e4a1] 2024-03-24T00:21:58.222 INFO:journalctl@ceph.mds.l.smithi193.stdout:Mar 24 00:21:58 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-l[72021]: 2: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7f638038e667] 2024-03-24T00:21:58.222 INFO:journalctl@ceph.mds.l.smithi193.stdout:Mar 24 00:21:58 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-l[72021]: 3: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x5598cb54c223] 2024-03-24T00:21:58.222 INFO:journalctl@ceph.mds.l.smithi193.stdout:Mar 24 00:21:58 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-l[72021]: 4: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x5598cb3c65d6] 2024-03-24T00:21:58.223 INFO:journalctl@ceph.mds.l.smithi193.stdout:Mar 24 00:21:58 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-l[72021]: 5: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x5598cb3c6da1] 2024-03-24T00:21:58.223 INFO:journalctl@ceph.mds.l.smithi193.stdout:Mar 24 00:21:58 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-l[72021]: 6: (Server::handle_client_rename(boost::intrusive_ptr const&)+0x259d) [0x5598cb40397d] 2024-03-24T00:21:58.223 INFO:journalctl@ceph.mds.l.smithi193.stdout:Mar 24 00:21:58 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-l[72021]: 7: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x5598cb419233] 2024-03-24T00:21:58.223 INFO:journalctl@ceph.mds.l.smithi193.stdout:Mar 24 00:21:58 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-l[72021]: 8: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x5598cb41b197] 2024-03-24T00:21:58.223 INFO:journalctl@ceph.mds.l.smithi193.stdout:Mar 24 00:21:58 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-l[72021]: 9: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x5598cb31a11b] 2024-03-24T00:21:58.223 INFO:journalctl@ceph.mds.l.smithi193.stdout:Mar 24 00:21:58 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-l[72021]: 10: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x5598cb31a6bc] 2024-03-24T00:21:58.223 INFO:journalctl@ceph.mds.l.smithi193.stdout:Mar 24 00:21:58 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-l[72021]: 11: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x5598cb3041ff] 2024-03-24T00:21:58.223 INFO:journalctl@ceph.mds.l.smithi193.stdout:Mar 24 00:21:58 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-l[72021]: 12: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7f6380620488] 2024-03-24T00:21:58.223 INFO:journalctl@ceph.mds.l.smithi193.stdout:Mar 24 00:21:58 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-l[72021]: 13: (DispatchQueue::entry()+0x51f) [0x7f638061d60f] 2024-03-24T00:21:58.223 INFO:journalctl@ceph.mds.l.smithi193.stdout:Mar 24 00:21:58 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-l[72021]: 14: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f63806f0e81] 2024-03-24T00:21:58.223 INFO:journalctl@ceph.mds.l.smithi193.stdout:Mar 24 00:21:58 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-l[72021]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7f637f10b1ca] 2024-03-24T00:21:58.223 INFO:journalctl@ceph.mds.l.smithi193.stdout:Mar 24 00:21:58 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-l[72021]: 16: clone() 2024-03-24T00:21:58.223 INFO:journalctl@ceph.mds.l.smithi193.stdout:Mar 24 00:21:58 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-l[72021]: 2024-03-24T00:21:58.223 INFO:journalctl@ceph.mds.l.smithi193.stdout:Mar 24 00:21:58 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-l[72021]: 0> 2024-03-24T00:21:58.019+0000 7f6375d64700 -1 *** Caught signal (Aborted) ** 2024-03-24T00:21:58.224 INFO:journalctl@ceph.mds.l.smithi193.stdout:Mar 24 00:21:58 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-l[72021]: in thread 7f6375d64700 thread_name:ms_dispatch 2024-03-24T00:21:58.224 INFO:journalctl@ceph.mds.l.smithi193.stdout:Mar 24 00:21:58 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-l[72021]: 2024-03-24T00:21:58.224 INFO:journalctl@ceph.mds.l.smithi193.stdout:Mar 24 00:21:58 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-l[72021]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-24T00:21:58.224 INFO:journalctl@ceph.mds.l.smithi193.stdout:Mar 24 00:21:58 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-l[72021]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f637f115d20] 2024-03-24T00:21:58.224 INFO:journalctl@ceph.mds.l.smithi193.stdout:Mar 24 00:21:58 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-l[72021]: 2: gsignal() 2024-03-24T00:21:58.224 INFO:journalctl@ceph.mds.l.smithi193.stdout:Mar 24 00:21:58 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-l[72021]: 3: abort() 2024-03-24T00:21:58.224 INFO:journalctl@ceph.mds.l.smithi193.stdout:Mar 24 00:21:58 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-l[72021]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x18f) [0x7f638038e4fb] 2024-03-24T00:21:58.224 INFO:journalctl@ceph.mds.l.smithi193.stdout:Mar 24 00:21:58 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-l[72021]: 5: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7f638038e667] 2024-03-24T00:21:58.224 INFO:journalctl@ceph.mds.l.smithi193.stdout:Mar 24 00:21:58 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-l[72021]: 6: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x5598cb54c223] 2024-03-24T00:21:58.224 INFO:journalctl@ceph.mds.l.smithi193.stdout:Mar 24 00:21:58 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-l[72021]: 7: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x5598cb3c65d6] 2024-03-24T00:21:58.224 INFO:journalctl@ceph.mds.l.smithi193.stdout:Mar 24 00:21:58 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-l[72021]: 8: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x5598cb3c6da1] 2024-03-24T00:21:58.224 INFO:journalctl@ceph.mds.l.smithi193.stdout:Mar 24 00:21:58 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-l[72021]: 9: (Server::handle_client_rename(boost::intrusive_ptr const&)+0x259d) [0x5598cb40397d] 2024-03-24T00:21:58.224 INFO:journalctl@ceph.mds.l.smithi193.stdout:Mar 24 00:21:58 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-l[72021]: 10: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x5598cb419233] 2024-03-24T00:21:58.224 INFO:journalctl@ceph.mds.l.smithi193.stdout:Mar 24 00:21:58 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-l[72021]: 11: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x5598cb41b197] 2024-03-24T00:21:58.224 INFO:journalctl@ceph.mds.l.smithi193.stdout:Mar 24 00:21:58 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-l[72021]: 12: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x5598cb31a11b] 2024-03-24T00:21:58.224 INFO:journalctl@ceph.mds.l.smithi193.stdout:Mar 24 00:21:58 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-l[72021]: 13: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x5598cb31a6bc] 2024-03-24T00:21:58.225 INFO:journalctl@ceph.mds.l.smithi193.stdout:Mar 24 00:21:58 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-l[72021]: 14: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x5598cb3041ff] 2024-03-24T00:21:58.225 INFO:journalctl@ceph.mds.l.smithi193.stdout:Mar 24 00:21:58 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-l[72021]: 15: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7f6380620488] 2024-03-24T00:21:58.225 INFO:journalctl@ceph.mds.l.smithi193.stdout:Mar 24 00:21:58 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-l[72021]: 16: (DispatchQueue::entry()+0x51f) [0x7f638061d60f] 2024-03-24T00:21:58.225 INFO:journalctl@ceph.mds.l.smithi193.stdout:Mar 24 00:21:58 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-l[72021]: 17: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f63806f0e81] 2024-03-24T00:21:58.225 INFO:journalctl@ceph.mds.l.smithi193.stdout:Mar 24 00:21:58 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-l[72021]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7f637f10b1ca] 2024-03-24T00:21:58.225 INFO:journalctl@ceph.mds.l.smithi193.stdout:Mar 24 00:21:58 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-l[72021]: 19: clone() 2024-03-24T00:21:58.225 INFO:journalctl@ceph.mds.l.smithi193.stdout:Mar 24 00:21:58 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-l[72021]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-03-24T00:21:58.225 INFO:journalctl@ceph.mds.l.smithi193.stdout:Mar 24 00:21:58 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-l[72021]: 2024-03-24T00:21:58.225 INFO:journalctl@ceph.mds.l.smithi193.stdout:Mar 24 00:21:58 smithi193 podman[72357]: 2024-03-24 00:21:58.087143573 +0000 UTC m=+0.020749934 container died 200b5ef45dd84c481ecdc9a61d2012b16dc125794c0fb337e89dec4988ba73bd (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-l, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, ceph=True, io.buildah.version=1.33.5, org.label-schema.build-date=20240319, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.vendor=CentOS, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c) 2024-03-24T00:21:58.347 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:21:57 smithi118 ceph-mon[31535]: mds.? [v2:172.21.15.193:6832/2276762432,v1:172.21.15.193:6833/2276762432] up:rejoin 2024-03-24T00:21:58.347 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:21:57 smithi118 ceph-mon[31535]: mds.? [v2:172.21.15.189:6836/2350260652,v1:172.21.15.189:6837/2350260652] up:rejoin 2024-03-24T00:21:58.347 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:21:57 smithi118 ceph-mon[31535]: fsmap cephfs:5/5 3 up:standby-replay 4 up:standby 2 up:rejoin 3 up:active 2024-03-24T00:21:58.347 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:21:57 smithi118 ceph-mon[31535]: daemon mds.h is now active in filesystem cephfs as rank 3 2024-03-24T00:21:58.347 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:21:57 smithi118 ceph-mon[31535]: daemon mds.l is now active in filesystem cephfs as rank 1 2024-03-24T00:21:58.347 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:21:57 smithi118 ceph-mon[31535]: pgmap v600: 97 pgs: 97 active+clean; 135 MiB data, 805 MiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 83 KiB/s wr, 39 op/s 2024-03-24T00:21:58.422 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:21:57 smithi189 ceph-mon[36187]: mds.? [v2:172.21.15.193:6832/2276762432,v1:172.21.15.193:6833/2276762432] up:rejoin 2024-03-24T00:21:58.422 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:21:57 smithi189 ceph-mon[36187]: mds.? [v2:172.21.15.189:6836/2350260652,v1:172.21.15.189:6837/2350260652] up:rejoin 2024-03-24T00:21:58.422 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:21:57 smithi189 ceph-mon[36187]: fsmap cephfs:5/5 3 up:standby-replay 4 up:standby 2 up:rejoin 3 up:active 2024-03-24T00:21:58.422 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:21:57 smithi189 ceph-mon[36187]: daemon mds.h is now active in filesystem cephfs as rank 3 2024-03-24T00:21:58.422 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:21:57 smithi189 ceph-mon[36187]: daemon mds.l is now active in filesystem cephfs as rank 1 2024-03-24T00:21:58.423 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:21:57 smithi189 ceph-mon[36187]: pgmap v600: 97 pgs: 97 active+clean; 135 MiB data, 805 MiB used, 1.0 TiB / 1.0 TiB avail; 21 KiB/s rd, 83 KiB/s wr, 39 op/s 2024-03-24T00:21:58.423 INFO:journalctl@ceph.mds.h.smithi189.stdout:Mar 24 00:21:57 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-h[69812]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: In function 'void Locker::set_xlocks_done(MutationImpl*, bool)' thread 7fd2c67a5700 time 2024-03-24T00:21:57.977772+0000 2024-03-24T00:21:58.423 INFO:journalctl@ceph.mds.h.smithi189.stdout:Mar 24 00:21:57 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-h[69812]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: 626: FAILED ceph_assert(obj->is_auth()) 2024-03-24T00:21:58.423 INFO:journalctl@ceph.mds.h.smithi189.stdout:Mar 24 00:21:57 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-h[69812]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-24T00:21:58.423 INFO:journalctl@ceph.mds.h.smithi189.stdout:Mar 24 00:21:57 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-h[69812]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x135) [0x7fd2d0dcf4a1] 2024-03-24T00:21:58.423 INFO:journalctl@ceph.mds.h.smithi189.stdout:Mar 24 00:21:57 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-h[69812]: 2: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7fd2d0dcf667] 2024-03-24T00:21:58.423 INFO:journalctl@ceph.mds.h.smithi189.stdout:Mar 24 00:21:57 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-h[69812]: 3: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x559217940223] 2024-03-24T00:21:58.423 INFO:journalctl@ceph.mds.h.smithi189.stdout:Mar 24 00:21:57 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-h[69812]: 4: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x5592177ba5d6] 2024-03-24T00:21:58.423 INFO:journalctl@ceph.mds.h.smithi189.stdout:Mar 24 00:21:57 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-h[69812]: 5: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x5592177bada1] 2024-03-24T00:21:58.424 INFO:journalctl@ceph.mds.h.smithi189.stdout:Mar 24 00:21:57 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-h[69812]: 6: (Server::handle_client_rename(boost::intrusive_ptr const&)+0x259d) [0x5592177f797d] 2024-03-24T00:21:58.424 INFO:journalctl@ceph.mds.h.smithi189.stdout:Mar 24 00:21:57 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-h[69812]: 7: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x55921780d233] 2024-03-24T00:21:58.424 INFO:journalctl@ceph.mds.h.smithi189.stdout:Mar 24 00:21:57 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-h[69812]: 8: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x55921780f197] 2024-03-24T00:21:58.424 INFO:journalctl@ceph.mds.h.smithi189.stdout:Mar 24 00:21:57 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-h[69812]: 9: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x55921770e11b] 2024-03-24T00:21:58.424 INFO:journalctl@ceph.mds.h.smithi189.stdout:Mar 24 00:21:57 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-h[69812]: 10: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x55921770e6bc] 2024-03-24T00:21:58.424 INFO:journalctl@ceph.mds.h.smithi189.stdout:Mar 24 00:21:57 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-h[69812]: 11: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x5592176f81ff] 2024-03-24T00:21:58.424 INFO:journalctl@ceph.mds.h.smithi189.stdout:Mar 24 00:21:57 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-h[69812]: 12: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7fd2d1061488] 2024-03-24T00:21:58.424 INFO:journalctl@ceph.mds.h.smithi189.stdout:Mar 24 00:21:57 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-h[69812]: 13: (DispatchQueue::entry()+0x51f) [0x7fd2d105e60f] 2024-03-24T00:21:58.424 INFO:journalctl@ceph.mds.h.smithi189.stdout:Mar 24 00:21:57 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-h[69812]: 14: (DispatchQueue::DispatchThread::entry()+0x11) [0x7fd2d1131e81] 2024-03-24T00:21:58.424 INFO:journalctl@ceph.mds.h.smithi189.stdout:Mar 24 00:21:57 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-h[69812]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7fd2cfb4c1ca] 2024-03-24T00:21:58.424 INFO:journalctl@ceph.mds.h.smithi189.stdout:Mar 24 00:21:57 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-h[69812]: 16: clone() 2024-03-24T00:21:58.424 INFO:journalctl@ceph.mds.h.smithi189.stdout:Mar 24 00:21:57 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-h[69812]: *** Caught signal (Aborted) ** 2024-03-24T00:21:58.424 INFO:journalctl@ceph.mds.h.smithi189.stdout:Mar 24 00:21:57 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-h[69812]: in thread 7fd2c67a5700 thread_name:ms_dispatch 2024-03-24T00:21:58.425 INFO:journalctl@ceph.mds.h.smithi189.stdout:Mar 24 00:21:57 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-h[69812]: 2024-03-24T00:21:57.978+0000 7fd2c67a5700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: In function 'void Locker::set_xlocks_done(MutationImpl*, bool)' thread 7fd2c67a5700 time 2024-03-24T00:21:57.977772+0000 2024-03-24T00:21:58.425 INFO:journalctl@ceph.mds.h.smithi189.stdout:Mar 24 00:21:57 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-h[69812]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: 626: FAILED ceph_assert(obj->is_auth()) 2024-03-24T00:21:58.425 INFO:journalctl@ceph.mds.h.smithi189.stdout:Mar 24 00:21:57 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-h[69812]: 2024-03-24T00:21:58.425 INFO:journalctl@ceph.mds.h.smithi189.stdout:Mar 24 00:21:57 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-h[69812]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-24T00:21:58.425 INFO:journalctl@ceph.mds.h.smithi189.stdout:Mar 24 00:21:57 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-h[69812]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x135) [0x7fd2d0dcf4a1] 2024-03-24T00:21:58.425 INFO:journalctl@ceph.mds.h.smithi189.stdout:Mar 24 00:21:57 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-h[69812]: 2: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7fd2d0dcf667] 2024-03-24T00:21:58.425 INFO:journalctl@ceph.mds.h.smithi189.stdout:Mar 24 00:21:57 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-h[69812]: 3: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x559217940223] 2024-03-24T00:21:58.425 INFO:journalctl@ceph.mds.h.smithi189.stdout:Mar 24 00:21:57 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-h[69812]: 4: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x5592177ba5d6] 2024-03-24T00:21:58.425 INFO:journalctl@ceph.mds.h.smithi189.stdout:Mar 24 00:21:57 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-h[69812]: 5: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x5592177bada1] 2024-03-24T00:21:58.426 INFO:journalctl@ceph.mds.h.smithi189.stdout:Mar 24 00:21:57 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-h[69812]: 6: (Server::handle_client_rename(boost::intrusive_ptr const&)+0x259d) [0x5592177f797d] 2024-03-24T00:21:58.426 INFO:journalctl@ceph.mds.h.smithi189.stdout:Mar 24 00:21:57 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-h[69812]: 7: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x55921780d233] 2024-03-24T00:21:58.426 INFO:journalctl@ceph.mds.h.smithi189.stdout:Mar 24 00:21:57 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-h[69812]: 8: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x55921780f197] 2024-03-24T00:21:58.426 INFO:journalctl@ceph.mds.h.smithi189.stdout:Mar 24 00:21:57 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-h[69812]: 9: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x55921770e11b] 2024-03-24T00:21:58.426 INFO:journalctl@ceph.mds.h.smithi189.stdout:Mar 24 00:21:57 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-h[69812]: 10: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x55921770e6bc] 2024-03-24T00:21:58.426 INFO:journalctl@ceph.mds.h.smithi189.stdout:Mar 24 00:21:57 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-h[69812]: 11: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x5592176f81ff] 2024-03-24T00:21:58.426 INFO:journalctl@ceph.mds.h.smithi189.stdout:Mar 24 00:21:57 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-h[69812]: 12: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7fd2d1061488] 2024-03-24T00:21:58.426 INFO:journalctl@ceph.mds.h.smithi189.stdout:Mar 24 00:21:57 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-h[69812]: 13: (DispatchQueue::entry()+0x51f) [0x7fd2d105e60f] 2024-03-24T00:21:58.426 INFO:journalctl@ceph.mds.h.smithi189.stdout:Mar 24 00:21:57 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-h[69812]: 14: (DispatchQueue::DispatchThread::entry()+0x11) [0x7fd2d1131e81] 2024-03-24T00:21:58.426 INFO:journalctl@ceph.mds.h.smithi189.stdout:Mar 24 00:21:57 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-h[69812]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7fd2cfb4c1ca] 2024-03-24T00:21:58.426 INFO:journalctl@ceph.mds.h.smithi189.stdout:Mar 24 00:21:57 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-h[69812]: 16: clone() 2024-03-24T00:21:58.426 INFO:journalctl@ceph.mds.h.smithi189.stdout:Mar 24 00:21:57 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-h[69812]: 2024-03-24T00:21:58.426 INFO:journalctl@ceph.mds.h.smithi189.stdout:Mar 24 00:21:57 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-h[69812]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-24T00:21:58.426 INFO:journalctl@ceph.mds.h.smithi189.stdout:Mar 24 00:21:57 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-h[69812]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7fd2cfb56d20] 2024-03-24T00:21:58.427 INFO:journalctl@ceph.mds.h.smithi189.stdout:Mar 24 00:21:57 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-h[69812]: 2: gsignal() 2024-03-24T00:21:58.427 INFO:journalctl@ceph.mds.h.smithi189.stdout:Mar 24 00:21:57 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-h[69812]: 3: abort() 2024-03-24T00:21:58.427 INFO:journalctl@ceph.mds.h.smithi189.stdout:Mar 24 00:21:57 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-h[69812]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x18f) [0x7fd2d0dcf4fb] 2024-03-24T00:21:58.427 INFO:journalctl@ceph.mds.h.smithi189.stdout:Mar 24 00:21:57 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-h[69812]: 5: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7fd2d0dcf667] 2024-03-24T00:21:58.427 INFO:journalctl@ceph.mds.h.smithi189.stdout:Mar 24 00:21:57 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-h[69812]: 6: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x559217940223] 2024-03-24T00:21:58.427 INFO:journalctl@ceph.mds.h.smithi189.stdout:Mar 24 00:21:57 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-h[69812]: 7: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x5592177ba5d6] 2024-03-24T00:21:58.427 INFO:journalctl@ceph.mds.h.smithi189.stdout:Mar 24 00:21:57 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-h[69812]: 8: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x5592177bada1] 2024-03-24T00:21:58.427 INFO:journalctl@ceph.mds.h.smithi189.stdout:Mar 24 00:21:57 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-h[69812]: 9: (Server::handle_client_rename(boost::intrusive_ptr const&)+0x259d) [0x5592177f797d] 2024-03-24T00:21:58.427 INFO:journalctl@ceph.mds.h.smithi189.stdout:Mar 24 00:21:57 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-h[69812]: 10: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x55921780d233] 2024-03-24T00:21:58.427 INFO:journalctl@ceph.mds.h.smithi189.stdout:Mar 24 00:21:57 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-h[69812]: 11: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x55921780f197] 2024-03-24T00:21:58.427 INFO:journalctl@ceph.mds.h.smithi189.stdout:Mar 24 00:21:57 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-h[69812]: 12: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x55921770e11b] 2024-03-24T00:21:58.427 INFO:journalctl@ceph.mds.h.smithi189.stdout:Mar 24 00:21:57 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-h[69812]: 13: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x55921770e6bc] 2024-03-24T00:21:58.427 INFO:journalctl@ceph.mds.h.smithi189.stdout:Mar 24 00:21:57 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-h[69812]: 14: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x5592176f81ff] 2024-03-24T00:21:58.427 INFO:journalctl@ceph.mds.h.smithi189.stdout:Mar 24 00:21:57 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-h[69812]: 15: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7fd2d1061488] 2024-03-24T00:21:58.427 INFO:journalctl@ceph.mds.h.smithi189.stdout:Mar 24 00:21:57 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-h[69812]: 16: (DispatchQueue::entry()+0x51f) [0x7fd2d105e60f] 2024-03-24T00:21:58.428 INFO:journalctl@ceph.mds.h.smithi189.stdout:Mar 24 00:21:57 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-h[69812]: 17: (DispatchQueue::DispatchThread::entry()+0x11) [0x7fd2d1131e81] 2024-03-24T00:21:58.428 INFO:journalctl@ceph.mds.h.smithi189.stdout:Mar 24 00:21:57 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-h[69812]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7fd2cfb4c1ca] 2024-03-24T00:21:58.428 INFO:journalctl@ceph.mds.h.smithi189.stdout:Mar 24 00:21:57 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-h[69812]: 19: clone() 2024-03-24T00:21:58.428 INFO:journalctl@ceph.mds.h.smithi189.stdout:Mar 24 00:21:57 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-h[69812]: 2024-03-24T00:21:57.980+0000 7fd2c67a5700 -1 *** Caught signal (Aborted) ** 2024-03-24T00:21:58.428 INFO:journalctl@ceph.mds.h.smithi189.stdout:Mar 24 00:21:57 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-h[69812]: in thread 7fd2c67a5700 thread_name:ms_dispatch 2024-03-24T00:21:58.428 INFO:journalctl@ceph.mds.h.smithi189.stdout:Mar 24 00:21:57 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-h[69812]: 2024-03-24T00:21:58.428 INFO:journalctl@ceph.mds.h.smithi189.stdout:Mar 24 00:21:57 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-h[69812]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-24T00:21:58.428 INFO:journalctl@ceph.mds.h.smithi189.stdout:Mar 24 00:21:57 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-h[69812]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7fd2cfb56d20] 2024-03-24T00:21:58.428 INFO:journalctl@ceph.mds.h.smithi189.stdout:Mar 24 00:21:57 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-h[69812]: 2: gsignal() 2024-03-24T00:21:58.428 INFO:journalctl@ceph.mds.h.smithi189.stdout:Mar 24 00:21:57 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-h[69812]: 3: abort() 2024-03-24T00:21:58.428 INFO:journalctl@ceph.mds.h.smithi189.stdout:Mar 24 00:21:57 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-h[69812]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x18f) [0x7fd2d0dcf4fb] 2024-03-24T00:21:58.428 INFO:journalctl@ceph.mds.h.smithi189.stdout:Mar 24 00:21:57 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-h[69812]: 5: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7fd2d0dcf667] 2024-03-24T00:21:58.429 INFO:journalctl@ceph.mds.h.smithi189.stdout:Mar 24 00:21:57 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-h[69812]: 6: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x559217940223] 2024-03-24T00:21:58.429 INFO:journalctl@ceph.mds.h.smithi189.stdout:Mar 24 00:21:57 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-h[69812]: 7: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x5592177ba5d6] 2024-03-24T00:21:58.429 INFO:journalctl@ceph.mds.h.smithi189.stdout:Mar 24 00:21:57 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-h[69812]: 8: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x5592177bada1] 2024-03-24T00:21:58.429 INFO:journalctl@ceph.mds.h.smithi189.stdout:Mar 24 00:21:57 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-h[69812]: 9: (Server::handle_client_rename(boost::intrusive_ptr const&)+0x259d) [0x5592177f797d] 2024-03-24T00:21:58.429 INFO:journalctl@ceph.mds.h.smithi189.stdout:Mar 24 00:21:57 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-h[69812]: 10: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x55921780d233] 2024-03-24T00:21:58.430 INFO:journalctl@ceph.mds.h.smithi189.stdout:Mar 24 00:21:57 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-h[69812]: 11: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x55921780f197] 2024-03-24T00:21:58.430 INFO:journalctl@ceph.mds.h.smithi189.stdout:Mar 24 00:21:57 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-h[69812]: 12: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x55921770e11b] 2024-03-24T00:21:58.430 INFO:journalctl@ceph.mds.h.smithi189.stdout:Mar 24 00:21:57 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-h[69812]: 13: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x55921770e6bc] 2024-03-24T00:21:58.430 INFO:journalctl@ceph.mds.h.smithi189.stdout:Mar 24 00:21:57 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-h[69812]: 14: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x5592176f81ff] 2024-03-24T00:21:58.430 INFO:journalctl@ceph.mds.h.smithi189.stdout:Mar 24 00:21:57 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-h[69812]: 15: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7fd2d1061488] 2024-03-24T00:21:58.430 INFO:journalctl@ceph.mds.h.smithi189.stdout:Mar 24 00:21:57 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-h[69812]: 16: (DispatchQueue::entry()+0x51f) [0x7fd2d105e60f] 2024-03-24T00:21:58.430 INFO:journalctl@ceph.mds.h.smithi189.stdout:Mar 24 00:21:57 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-h[69812]: 17: (DispatchQueue::DispatchThread::entry()+0x11) [0x7fd2d1131e81] 2024-03-24T00:21:58.430 INFO:journalctl@ceph.mds.h.smithi189.stdout:Mar 24 00:21:57 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-h[69812]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7fd2cfb4c1ca] 2024-03-24T00:21:58.430 INFO:journalctl@ceph.mds.h.smithi189.stdout:Mar 24 00:21:57 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-h[69812]: 19: clone() 2024-03-24T00:21:58.430 INFO:journalctl@ceph.mds.h.smithi189.stdout:Mar 24 00:21:57 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-h[69812]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-03-24T00:21:58.430 INFO:journalctl@ceph.mds.h.smithi189.stdout:Mar 24 00:21:57 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-h[69812]: 2024-03-24T00:21:58.430 INFO:journalctl@ceph.mds.h.smithi189.stdout:Mar 24 00:21:57 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-h[69812]: -1> 2024-03-24T00:21:57.978+0000 7fd2c67a5700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: In function 'void Locker::set_xlocks_done(MutationImpl*, bool)' thread 7fd2c67a5700 time 2024-03-24T00:21:57.977772+0000 2024-03-24T00:21:58.430 INFO:journalctl@ceph.mds.h.smithi189.stdout:Mar 24 00:21:57 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-h[69812]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: 626: FAILED ceph_assert(obj->is_auth()) 2024-03-24T00:21:58.431 INFO:journalctl@ceph.mds.h.smithi189.stdout:Mar 24 00:21:57 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-h[69812]: 2024-03-24T00:21:58.431 INFO:journalctl@ceph.mds.h.smithi189.stdout:Mar 24 00:21:57 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-h[69812]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-24T00:21:58.431 INFO:journalctl@ceph.mds.h.smithi189.stdout:Mar 24 00:21:57 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-h[69812]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x135) [0x7fd2d0dcf4a1] 2024-03-24T00:21:58.431 INFO:journalctl@ceph.mds.h.smithi189.stdout:Mar 24 00:21:57 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-h[69812]: 2: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7fd2d0dcf667] 2024-03-24T00:21:58.431 INFO:journalctl@ceph.mds.h.smithi189.stdout:Mar 24 00:21:57 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-h[69812]: 3: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x559217940223] 2024-03-24T00:21:58.431 INFO:journalctl@ceph.mds.h.smithi189.stdout:Mar 24 00:21:57 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-h[69812]: 4: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x5592177ba5d6] 2024-03-24T00:21:58.431 INFO:journalctl@ceph.mds.h.smithi189.stdout:Mar 24 00:21:57 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-h[69812]: 5: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x5592177bada1] 2024-03-24T00:21:58.431 INFO:journalctl@ceph.mds.h.smithi189.stdout:Mar 24 00:21:57 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-h[69812]: 6: (Server::handle_client_rename(boost::intrusive_ptr const&)+0x259d) [0x5592177f797d] 2024-03-24T00:21:58.431 INFO:journalctl@ceph.mds.h.smithi189.stdout:Mar 24 00:21:57 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-h[69812]: 7: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x55921780d233] 2024-03-24T00:21:58.431 INFO:journalctl@ceph.mds.h.smithi189.stdout:Mar 24 00:21:57 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-h[69812]: 8: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x55921780f197] 2024-03-24T00:21:58.431 INFO:journalctl@ceph.mds.h.smithi189.stdout:Mar 24 00:21:57 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-h[69812]: 9: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x55921770e11b] 2024-03-24T00:21:58.431 INFO:journalctl@ceph.mds.h.smithi189.stdout:Mar 24 00:21:57 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-h[69812]: 10: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x55921770e6bc] 2024-03-24T00:21:58.431 INFO:journalctl@ceph.mds.h.smithi189.stdout:Mar 24 00:21:57 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-h[69812]: 11: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x5592176f81ff] 2024-03-24T00:21:58.432 INFO:journalctl@ceph.mds.h.smithi189.stdout:Mar 24 00:21:57 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-h[69812]: 12: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7fd2d1061488] 2024-03-24T00:21:58.432 INFO:journalctl@ceph.mds.h.smithi189.stdout:Mar 24 00:21:57 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-h[69812]: 13: (DispatchQueue::entry()+0x51f) [0x7fd2d105e60f] 2024-03-24T00:21:58.432 INFO:journalctl@ceph.mds.h.smithi189.stdout:Mar 24 00:21:57 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-h[69812]: 14: (DispatchQueue::DispatchThread::entry()+0x11) [0x7fd2d1131e81] 2024-03-24T00:21:58.432 INFO:journalctl@ceph.mds.h.smithi189.stdout:Mar 24 00:21:57 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-h[69812]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7fd2cfb4c1ca] 2024-03-24T00:21:58.432 INFO:journalctl@ceph.mds.h.smithi189.stdout:Mar 24 00:21:57 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-h[69812]: 16: clone() 2024-03-24T00:21:58.432 INFO:journalctl@ceph.mds.h.smithi189.stdout:Mar 24 00:21:57 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-h[69812]: 2024-03-24T00:21:58.432 INFO:journalctl@ceph.mds.h.smithi189.stdout:Mar 24 00:21:57 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-h[69812]: 0> 2024-03-24T00:21:57.980+0000 7fd2c67a5700 -1 *** Caught signal (Aborted) ** 2024-03-24T00:21:58.432 INFO:journalctl@ceph.mds.h.smithi189.stdout:Mar 24 00:21:57 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-h[69812]: in thread 7fd2c67a5700 thread_name:ms_dispatch 2024-03-24T00:21:58.432 INFO:journalctl@ceph.mds.h.smithi189.stdout:Mar 24 00:21:57 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-h[69812]: 2024-03-24T00:21:58.432 INFO:journalctl@ceph.mds.h.smithi189.stdout:Mar 24 00:21:57 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-h[69812]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-24T00:21:58.432 INFO:journalctl@ceph.mds.h.smithi189.stdout:Mar 24 00:21:57 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-h[69812]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7fd2cfb56d20] 2024-03-24T00:21:58.432 INFO:journalctl@ceph.mds.h.smithi189.stdout:Mar 24 00:21:57 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-h[69812]: 2: gsignal() 2024-03-24T00:21:58.432 INFO:journalctl@ceph.mds.h.smithi189.stdout:Mar 24 00:21:57 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-h[69812]: 3: abort() 2024-03-24T00:21:58.432 INFO:journalctl@ceph.mds.h.smithi189.stdout:Mar 24 00:21:57 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-h[69812]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x18f) [0x7fd2d0dcf4fb] 2024-03-24T00:21:58.432 INFO:journalctl@ceph.mds.h.smithi189.stdout:Mar 24 00:21:57 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-h[69812]: 5: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7fd2d0dcf667] 2024-03-24T00:21:58.433 INFO:journalctl@ceph.mds.h.smithi189.stdout:Mar 24 00:21:57 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-h[69812]: 6: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x559217940223] 2024-03-24T00:21:58.433 INFO:journalctl@ceph.mds.h.smithi189.stdout:Mar 24 00:21:57 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-h[69812]: 7: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x5592177ba5d6] 2024-03-24T00:21:58.433 INFO:journalctl@ceph.mds.h.smithi189.stdout:Mar 24 00:21:57 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-h[69812]: 8: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x5592177bada1] 2024-03-24T00:21:58.433 INFO:journalctl@ceph.mds.h.smithi189.stdout:Mar 24 00:21:57 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-h[69812]: 9: (Server::handle_client_rename(boost::intrusive_ptr const&)+0x259d) [0x5592177f797d] 2024-03-24T00:21:58.433 INFO:journalctl@ceph.mds.h.smithi189.stdout:Mar 24 00:21:57 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-h[69812]: 10: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x55921780d233] 2024-03-24T00:21:58.433 INFO:journalctl@ceph.mds.h.smithi189.stdout:Mar 24 00:21:57 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-h[69812]: 11: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x55921780f197] 2024-03-24T00:21:58.433 INFO:journalctl@ceph.mds.h.smithi189.stdout:Mar 24 00:21:57 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-h[69812]: 12: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x55921770e11b] 2024-03-24T00:21:58.433 INFO:journalctl@ceph.mds.h.smithi189.stdout:Mar 24 00:21:57 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-h[69812]: 13: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x55921770e6bc] 2024-03-24T00:21:58.433 INFO:journalctl@ceph.mds.h.smithi189.stdout:Mar 24 00:21:57 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-h[69812]: 14: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x5592176f81ff] 2024-03-24T00:21:58.433 INFO:journalctl@ceph.mds.h.smithi189.stdout:Mar 24 00:21:57 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-h[69812]: 15: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7fd2d1061488] 2024-03-24T00:21:58.433 INFO:journalctl@ceph.mds.h.smithi189.stdout:Mar 24 00:21:57 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-h[69812]: 16: (DispatchQueue::entry()+0x51f) [0x7fd2d105e60f] 2024-03-24T00:21:58.433 INFO:journalctl@ceph.mds.h.smithi189.stdout:Mar 24 00:21:57 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-h[69812]: 17: (DispatchQueue::DispatchThread::entry()+0x11) [0x7fd2d1131e81] 2024-03-24T00:21:58.433 INFO:journalctl@ceph.mds.h.smithi189.stdout:Mar 24 00:21:57 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-h[69812]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7fd2cfb4c1ca] 2024-03-24T00:21:58.434 INFO:journalctl@ceph.mds.h.smithi189.stdout:Mar 24 00:21:57 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-h[69812]: 19: clone() 2024-03-24T00:21:58.434 INFO:journalctl@ceph.mds.h.smithi189.stdout:Mar 24 00:21:57 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-h[69812]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-03-24T00:21:58.434 INFO:journalctl@ceph.mds.h.smithi189.stdout:Mar 24 00:21:57 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-h[69812]: 2024-03-24T00:21:58.434 INFO:journalctl@ceph.mds.h.smithi189.stdout:Mar 24 00:21:58 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-h[69812]: -1> 2024-03-24T00:21:57.978+0000 7fd2c67a5700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: In function 'void Locker::set_xlocks_done(MutationImpl*, bool)' thread 7fd2c67a5700 time 2024-03-24T00:21:57.977772+0000 2024-03-24T00:21:58.434 INFO:journalctl@ceph.mds.h.smithi189.stdout:Mar 24 00:21:58 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-h[69812]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: 626: FAILED ceph_assert(obj->is_auth()) 2024-03-24T00:21:58.434 INFO:journalctl@ceph.mds.h.smithi189.stdout:Mar 24 00:21:58 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-h[69812]: 2024-03-24T00:21:58.434 INFO:journalctl@ceph.mds.h.smithi189.stdout:Mar 24 00:21:58 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-h[69812]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-24T00:21:58.434 INFO:journalctl@ceph.mds.h.smithi189.stdout:Mar 24 00:21:58 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-h[69812]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x135) [0x7fd2d0dcf4a1] 2024-03-24T00:21:58.434 INFO:journalctl@ceph.mds.h.smithi189.stdout:Mar 24 00:21:58 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-h[69812]: 2: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7fd2d0dcf667] 2024-03-24T00:21:58.434 INFO:journalctl@ceph.mds.h.smithi189.stdout:Mar 24 00:21:58 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-h[69812]: 3: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x559217940223] 2024-03-24T00:21:58.434 INFO:journalctl@ceph.mds.h.smithi189.stdout:Mar 24 00:21:58 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-h[69812]: 4: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x5592177ba5d6] 2024-03-24T00:21:58.434 INFO:journalctl@ceph.mds.h.smithi189.stdout:Mar 24 00:21:58 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-h[69812]: 5: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x5592177bada1] 2024-03-24T00:21:58.434 INFO:journalctl@ceph.mds.h.smithi189.stdout:Mar 24 00:21:58 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-h[69812]: 6: (Server::handle_client_rename(boost::intrusive_ptr const&)+0x259d) [0x5592177f797d] 2024-03-24T00:21:58.434 INFO:journalctl@ceph.mds.h.smithi189.stdout:Mar 24 00:21:58 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-h[69812]: 7: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x55921780d233] 2024-03-24T00:21:58.435 INFO:journalctl@ceph.mds.h.smithi189.stdout:Mar 24 00:21:58 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-h[69812]: 8: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x55921780f197] 2024-03-24T00:21:58.435 INFO:journalctl@ceph.mds.h.smithi189.stdout:Mar 24 00:21:58 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-h[69812]: 9: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x55921770e11b] 2024-03-24T00:21:58.435 INFO:journalctl@ceph.mds.h.smithi189.stdout:Mar 24 00:21:58 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-h[69812]: 10: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x55921770e6bc] 2024-03-24T00:21:58.435 INFO:journalctl@ceph.mds.h.smithi189.stdout:Mar 24 00:21:58 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-h[69812]: 11: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x5592176f81ff] 2024-03-24T00:21:58.435 INFO:journalctl@ceph.mds.h.smithi189.stdout:Mar 24 00:21:58 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-h[69812]: 12: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7fd2d1061488] 2024-03-24T00:21:58.435 INFO:journalctl@ceph.mds.h.smithi189.stdout:Mar 24 00:21:58 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-h[69812]: 13: (DispatchQueue::entry()+0x51f) [0x7fd2d105e60f] 2024-03-24T00:21:58.435 INFO:journalctl@ceph.mds.h.smithi189.stdout:Mar 24 00:21:58 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-h[69812]: 14: (DispatchQueue::DispatchThread::entry()+0x11) [0x7fd2d1131e81] 2024-03-24T00:21:58.435 INFO:journalctl@ceph.mds.h.smithi189.stdout:Mar 24 00:21:58 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-h[69812]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7fd2cfb4c1ca] 2024-03-24T00:21:58.435 INFO:journalctl@ceph.mds.h.smithi189.stdout:Mar 24 00:21:58 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-h[69812]: 16: clone() 2024-03-24T00:21:58.435 INFO:journalctl@ceph.mds.h.smithi189.stdout:Mar 24 00:21:58 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-h[69812]: 2024-03-24T00:21:58.435 INFO:journalctl@ceph.mds.h.smithi189.stdout:Mar 24 00:21:58 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-h[69812]: 0> 2024-03-24T00:21:57.980+0000 7fd2c67a5700 -1 *** Caught signal (Aborted) ** 2024-03-24T00:21:58.435 INFO:journalctl@ceph.mds.h.smithi189.stdout:Mar 24 00:21:58 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-h[69812]: in thread 7fd2c67a5700 thread_name:ms_dispatch 2024-03-24T00:21:58.435 INFO:journalctl@ceph.mds.h.smithi189.stdout:Mar 24 00:21:58 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-h[69812]: 2024-03-24T00:21:58.435 INFO:journalctl@ceph.mds.h.smithi189.stdout:Mar 24 00:21:58 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-h[69812]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-24T00:21:58.436 INFO:journalctl@ceph.mds.h.smithi189.stdout:Mar 24 00:21:58 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-h[69812]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7fd2cfb56d20] 2024-03-24T00:21:58.436 INFO:journalctl@ceph.mds.h.smithi189.stdout:Mar 24 00:21:58 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-h[69812]: 2: gsignal() 2024-03-24T00:21:58.436 INFO:journalctl@ceph.mds.h.smithi189.stdout:Mar 24 00:21:58 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-h[69812]: 3: abort() 2024-03-24T00:21:58.436 INFO:journalctl@ceph.mds.h.smithi189.stdout:Mar 24 00:21:58 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-h[69812]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x18f) [0x7fd2d0dcf4fb] 2024-03-24T00:21:58.436 INFO:journalctl@ceph.mds.h.smithi189.stdout:Mar 24 00:21:58 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-h[69812]: 5: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7fd2d0dcf667] 2024-03-24T00:21:58.436 INFO:journalctl@ceph.mds.h.smithi189.stdout:Mar 24 00:21:58 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-h[69812]: 6: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x559217940223] 2024-03-24T00:21:58.436 INFO:journalctl@ceph.mds.h.smithi189.stdout:Mar 24 00:21:58 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-h[69812]: 7: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x5592177ba5d6] 2024-03-24T00:21:58.436 INFO:journalctl@ceph.mds.h.smithi189.stdout:Mar 24 00:21:58 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-h[69812]: 8: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x5592177bada1] 2024-03-24T00:21:58.436 INFO:journalctl@ceph.mds.h.smithi189.stdout:Mar 24 00:21:58 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-h[69812]: 9: (Server::handle_client_rename(boost::intrusive_ptr const&)+0x259d) [0x5592177f797d] 2024-03-24T00:21:58.436 INFO:journalctl@ceph.mds.h.smithi189.stdout:Mar 24 00:21:58 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-h[69812]: 10: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x55921780d233] 2024-03-24T00:21:58.436 INFO:journalctl@ceph.mds.h.smithi189.stdout:Mar 24 00:21:58 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-h[69812]: 11: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x55921780f197] 2024-03-24T00:21:58.436 INFO:journalctl@ceph.mds.h.smithi189.stdout:Mar 24 00:21:58 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-h[69812]: 12: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x55921770e11b] 2024-03-24T00:21:58.436 INFO:journalctl@ceph.mds.h.smithi189.stdout:Mar 24 00:21:58 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-h[69812]: 13: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x55921770e6bc] 2024-03-24T00:21:58.436 INFO:journalctl@ceph.mds.h.smithi189.stdout:Mar 24 00:21:58 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-h[69812]: 14: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x5592176f81ff] 2024-03-24T00:21:58.436 INFO:journalctl@ceph.mds.h.smithi189.stdout:Mar 24 00:21:58 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-h[69812]: 15: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7fd2d1061488] 2024-03-24T00:21:58.437 INFO:journalctl@ceph.mds.h.smithi189.stdout:Mar 24 00:21:58 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-h[69812]: 16: (DispatchQueue::entry()+0x51f) [0x7fd2d105e60f] 2024-03-24T00:21:58.437 INFO:journalctl@ceph.mds.h.smithi189.stdout:Mar 24 00:21:58 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-h[69812]: 17: (DispatchQueue::DispatchThread::entry()+0x11) [0x7fd2d1131e81] 2024-03-24T00:21:58.437 INFO:journalctl@ceph.mds.h.smithi189.stdout:Mar 24 00:21:58 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-h[69812]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7fd2cfb4c1ca] 2024-03-24T00:21:58.437 INFO:journalctl@ceph.mds.h.smithi189.stdout:Mar 24 00:21:58 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-h[69812]: 19: clone() 2024-03-24T00:21:58.437 INFO:journalctl@ceph.mds.h.smithi189.stdout:Mar 24 00:21:58 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-h[69812]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-03-24T00:21:58.437 INFO:journalctl@ceph.mds.h.smithi189.stdout:Mar 24 00:21:58 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-h[69812]: 2024-03-24T00:21:58.437 INFO:journalctl@ceph.mds.h.smithi189.stdout:Mar 24 00:21:58 smithi189 podman[69979]: 2024-03-24 00:21:58.041468376 +0000 UTC m=+0.024821059 container died 9706b49f92e8c2e438ca1e4fe4db03e0eee967249543d0df6906c3f11e450b28 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-h, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, org.label-schema.build-date=20240319, org.label-schema.schema-version=1.0) 2024-03-24T00:21:59.030 INFO:journalctl@ceph.mds.h.smithi189.stdout:Mar 24 00:21:58 smithi189 podman[69979]: 2024-03-24 00:21:58.726186093 +0000 UTC m=+0.709538756 container remove 9706b49f92e8c2e438ca1e4fe4db03e0eee967249543d0df6906c3f11e450b28 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-h, ceph=True, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.license=GPLv2, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240319) 2024-03-24T00:21:59.030 INFO:journalctl@ceph.mds.h.smithi189.stdout:Mar 24 00:21:58 smithi189 systemd[1]: ceph-d79a5420-e971-11ee-95cd-87774f69a715@mds.h.service: Main process exited, code=exited, status=134/n/a 2024-03-24T00:21:59.030 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:21:58 smithi189 ceph-mon[36187]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-03-24T00:21:59.031 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:21:58 smithi189 ceph-mon[36187]: Cluster is now healthy 2024-03-24T00:21:59.031 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:21:58 smithi189 ceph-mon[36187]: mds.? [v2:172.21.15.189:6836/2350260652,v1:172.21.15.189:6837/2350260652] up:active 2024-03-24T00:21:59.031 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:21:58 smithi189 ceph-mon[36187]: mds.? [v2:172.21.15.193:6832/2276762432,v1:172.21.15.193:6833/2276762432] up:active 2024-03-24T00:21:59.031 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:21:58 smithi189 ceph-mon[36187]: fsmap cephfs:5 3 up:standby-replay 4 up:standby 5 up:active 2024-03-24T00:21:59.031 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:21:58 smithi189 ceph-mon[36187]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-03-24T00:21:59.060 INFO:journalctl@ceph.mds.l.smithi193.stdout:Mar 24 00:21:58 smithi193 podman[72357]: 2024-03-24 00:21:58.803529327 +0000 UTC m=+0.737135684 container remove 200b5ef45dd84c481ecdc9a61d2012b16dc125794c0fb337e89dec4988ba73bd (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-l, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, ceph=True, org.label-schema.build-date=20240319, GIT_BRANCH=HEAD) 2024-03-24T00:21:59.060 INFO:journalctl@ceph.mds.l.smithi193.stdout:Mar 24 00:21:58 smithi193 systemd[1]: ceph-d79a5420-e971-11ee-95cd-87774f69a715@mds.l.service: Main process exited, code=exited, status=134/n/a 2024-03-24T00:21:59.061 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:21:58 smithi193 ceph-mon[36878]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-03-24T00:21:59.061 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:21:58 smithi193 ceph-mon[36878]: Cluster is now healthy 2024-03-24T00:21:59.061 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:21:58 smithi193 ceph-mon[36878]: mds.? [v2:172.21.15.189:6836/2350260652,v1:172.21.15.189:6837/2350260652] up:active 2024-03-24T00:21:59.061 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:21:58 smithi193 ceph-mon[36878]: mds.? [v2:172.21.15.193:6832/2276762432,v1:172.21.15.193:6833/2276762432] up:active 2024-03-24T00:21:59.061 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:21:58 smithi193 ceph-mon[36878]: fsmap cephfs:5 3 up:standby-replay 4 up:standby 5 up:active 2024-03-24T00:21:59.061 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:21:58 smithi193 ceph-mon[36878]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-03-24T00:21:59.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:21:58 smithi118 ceph-mon[31535]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-03-24T00:21:59.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:21:58 smithi118 ceph-mon[31535]: Cluster is now healthy 2024-03-24T00:21:59.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:21:58 smithi118 ceph-mon[31535]: mds.? [v2:172.21.15.189:6836/2350260652,v1:172.21.15.189:6837/2350260652] up:active 2024-03-24T00:21:59.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:21:58 smithi118 ceph-mon[31535]: mds.? [v2:172.21.15.193:6832/2276762432,v1:172.21.15.193:6833/2276762432] up:active 2024-03-24T00:21:59.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:21:58 smithi118 ceph-mon[31535]: fsmap cephfs:5 3 up:standby-replay 4 up:standby 5 up:active 2024-03-24T00:21:59.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:21:58 smithi118 ceph-mon[31535]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-03-24T00:21:59.454 INFO:journalctl@ceph.mds.h.smithi189.stdout:Mar 24 00:21:59 smithi189 systemd[1]: ceph-d79a5420-e971-11ee-95cd-87774f69a715@mds.h.service: Failed with result 'exit-code'. 2024-03-24T00:21:59.460 INFO:journalctl@ceph.mds.l.smithi193.stdout:Mar 24 00:21:59 smithi193 systemd[1]: ceph-d79a5420-e971-11ee-95cd-87774f69a715@mds.l.service: Failed with result 'exit-code'. 2024-03-24T00:22:00.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:21:59 smithi118 ceph-mon[31535]: pgmap v601: 97 pgs: 97 active+clean; 136 MiB data, 803 MiB used, 1.0 TiB / 1.0 TiB avail; 351 KiB/s rd, 169 KiB/s wr, 89 op/s 2024-03-24T00:22:00.453 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:21:59 smithi189 ceph-mon[36187]: pgmap v601: 97 pgs: 97 active+clean; 136 MiB data, 803 MiB used, 1.0 TiB / 1.0 TiB avail; 351 KiB/s rd, 169 KiB/s wr, 89 op/s 2024-03-24T00:22:00.460 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:21:59 smithi193 ceph-mon[36878]: pgmap v601: 97 pgs: 97 active+clean; 136 MiB data, 803 MiB used, 1.0 TiB / 1.0 TiB avail; 351 KiB/s rd, 169 KiB/s wr, 89 op/s 2024-03-24T00:22:02.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:22:01 smithi118 ceph-mon[31535]: pgmap v602: 97 pgs: 97 active+clean; 136 MiB data, 803 MiB used, 1.0 TiB / 1.0 TiB avail; 598 KiB/s rd, 157 KiB/s wr, 100 op/s 2024-03-24T00:22:02.203 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:22:01 smithi189 ceph-mon[36187]: pgmap v602: 97 pgs: 97 active+clean; 136 MiB data, 803 MiB used, 1.0 TiB / 1.0 TiB avail; 598 KiB/s rd, 157 KiB/s wr, 100 op/s 2024-03-24T00:22:02.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:22:01 smithi193 ceph-mon[36878]: pgmap v602: 97 pgs: 97 active+clean; 136 MiB data, 803 MiB used, 1.0 TiB / 1.0 TiB avail; 598 KiB/s rd, 157 KiB/s wr, 100 op/s 2024-03-24T00:22:04.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:22:03 smithi118 ceph-mon[31535]: pgmap v603: 97 pgs: 97 active+clean; 135 MiB data, 806 MiB used, 1.0 TiB / 1.0 TiB avail; 986 KiB/s rd, 138 KiB/s wr, 122 op/s 2024-03-24T00:22:04.203 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:22:03 smithi189 ceph-mon[36187]: pgmap v603: 97 pgs: 97 active+clean; 135 MiB data, 806 MiB used, 1.0 TiB / 1.0 TiB avail; 986 KiB/s rd, 138 KiB/s wr, 122 op/s 2024-03-24T00:22:04.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:22:03 smithi193 ceph-mon[36878]: pgmap v603: 97 pgs: 97 active+clean; 135 MiB data, 806 MiB used, 1.0 TiB / 1.0 TiB avail; 986 KiB/s rd, 138 KiB/s wr, 122 op/s 2024-03-24T00:22:06.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:22:05 smithi118 ceph-mon[31535]: pgmap v604: 97 pgs: 97 active+clean; 136 MiB data, 789 MiB used, 1.0 TiB / 1.0 TiB avail; 893 KiB/s rd, 138 KiB/s wr, 111 op/s 2024-03-24T00:22:06.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:22:05 smithi189 ceph-mon[36187]: pgmap v604: 97 pgs: 97 active+clean; 136 MiB data, 789 MiB used, 1.0 TiB / 1.0 TiB avail; 893 KiB/s rd, 138 KiB/s wr, 111 op/s 2024-03-24T00:22:06.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:22:05 smithi193 ceph-mon[36878]: pgmap v604: 97 pgs: 97 active+clean; 136 MiB data, 789 MiB used, 1.0 TiB / 1.0 TiB avail; 893 KiB/s rd, 138 KiB/s wr, 111 op/s 2024-03-24T00:22:06.845 INFO:tasks.workunit.client.0.smithi118.stdout: 45 492 0 528 0 127 114 2024-03-24T00:22:08.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:22:07 smithi118 ceph-mon[31535]: pgmap v605: 97 pgs: 97 active+clean; 136 MiB data, 789 MiB used, 1.0 TiB / 1.0 TiB avail; 882 KiB/s rd, 133 KiB/s wr, 107 op/s 2024-03-24T00:22:08.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:22:07 smithi189 ceph-mon[36187]: pgmap v605: 97 pgs: 97 active+clean; 136 MiB data, 789 MiB used, 1.0 TiB / 1.0 TiB avail; 882 KiB/s rd, 133 KiB/s wr, 107 op/s 2024-03-24T00:22:08.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:22:07 smithi193 ceph-mon[36878]: pgmap v605: 97 pgs: 97 active+clean; 136 MiB data, 789 MiB used, 1.0 TiB / 1.0 TiB avail; 882 KiB/s rd, 133 KiB/s wr, 107 op/s 2024-03-24T00:22:09.504 INFO:journalctl@ceph.mds.l.smithi193.stdout:Mar 24 00:22:09 smithi193 systemd[1]: ceph-d79a5420-e971-11ee-95cd-87774f69a715@mds.l.service: Scheduled restart job, restart counter is at 5. 2024-03-24T00:22:09.504 INFO:journalctl@ceph.mds.l.smithi193.stdout:Mar 24 00:22:09 smithi193 systemd[1]: Stopped Ceph mds.l for d79a5420-e971-11ee-95cd-87774f69a715. 2024-03-24T00:22:09.504 INFO:journalctl@ceph.mds.l.smithi193.stdout:Mar 24 00:22:09 smithi193 systemd[1]: Starting Ceph mds.l for d79a5420-e971-11ee-95cd-87774f69a715... 2024-03-24T00:22:09.505 INFO:journalctl@ceph.mds.l.smithi193.stdout:Mar 24 00:22:09 smithi193 podman[72458]: 2024-03-24 00:22:09.33624233 +0000 UTC m=+0.017429893 image pull c664afe56bb01db4f9afa9532c20598e201e664f2e9ba2be50db0d52674a26d2 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc 2024-03-24T00:22:09.539 INFO:journalctl@ceph.mds.h.smithi189.stdout:Mar 24 00:22:09 smithi189 systemd[1]: ceph-d79a5420-e971-11ee-95cd-87774f69a715@mds.h.service: Scheduled restart job, restart counter is at 3. 2024-03-24T00:22:09.539 INFO:journalctl@ceph.mds.h.smithi189.stdout:Mar 24 00:22:09 smithi189 systemd[1]: Stopped Ceph mds.h for d79a5420-e971-11ee-95cd-87774f69a715. 2024-03-24T00:22:09.539 INFO:journalctl@ceph.mds.h.smithi189.stdout:Mar 24 00:22:09 smithi189 systemd[1]: Starting Ceph mds.h for d79a5420-e971-11ee-95cd-87774f69a715... 2024-03-24T00:22:09.539 INFO:journalctl@ceph.mds.h.smithi189.stdout:Mar 24 00:22:09 smithi189 podman[70072]: 2024-03-24 00:22:09.340744635 +0000 UTC m=+0.018631743 image pull c664afe56bb01db4f9afa9532c20598e201e664f2e9ba2be50db0d52674a26d2 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc 2024-03-24T00:22:09.539 INFO:journalctl@ceph.mds.h.smithi189.stdout:Mar 24 00:22:09 smithi189 podman[70072]: 2024-03-24 00:22:09.443614417 +0000 UTC m=+0.121501517 container create dca63437cc7c74c655e371800e91eb29b294536dd147dea3f0df3cd301a8f382 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-h, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, ceph=True, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, org.label-schema.build-date=20240319, GIT_CLEAN=True, org.label-schema.schema-version=1.0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30) 2024-03-24T00:22:09.770 INFO:journalctl@ceph.mds.l.smithi193.stdout:Mar 24 00:22:09 smithi193 podman[72458]: 2024-03-24 00:22:09.504637766 +0000 UTC m=+0.185825315 container create 34b63e7de437dbf746219f051fd02d843764da00bec1829324daf12d71db4db5 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-l, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240319, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, ceph=True, org.label-schema.schema-version=1.0) 2024-03-24T00:22:09.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:22:09 smithi189 ceph-mon[36187]: daemon mds.h restarted 2024-03-24T00:22:09.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:22:09 smithi189 ceph-mon[36187]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-03-24T00:22:09.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:22:09 smithi189 ceph-mon[36187]: osdmap e105: 12 total, 12 up, 12 in 2024-03-24T00:22:09.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:22:09 smithi189 ceph-mon[36187]: mds.? [v2:172.21.15.189:6836/75274228,v1:172.21.15.189:6837/75274228] up:boot 2024-03-24T00:22:09.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:22:09 smithi189 ceph-mon[36187]: Standby daemon mds.g assigned to filesystem cephfs as rank 3 2024-03-24T00:22:09.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:22:09 smithi189 ceph-mon[36187]: fsmap cephfs:4/5 {0=b=up:active,1=l=up:active,2=c=up:active,4=e=up:active} 5 up:standby-replay 3 up:standby, 1 failed 2024-03-24T00:22:09.955 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:22:09 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-03-24T00:22:09.955 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:22:09 smithi189 ceph-mon[36187]: fsmap cephfs:5/5 4 up:standby-replay 3 up:standby 1 up:replay 4 up:active 2024-03-24T00:22:09.955 INFO:journalctl@ceph.mds.h.smithi189.stdout:Mar 24 00:22:09 smithi189 podman[70072]: 2024-03-24 00:22:09.671139642 +0000 UTC m=+0.349026737 container init dca63437cc7c74c655e371800e91eb29b294536dd147dea3f0df3cd301a8f382 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-h, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240319, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.license=GPLv2, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, ceph=True, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-24T00:22:09.955 INFO:journalctl@ceph.mds.h.smithi189.stdout:Mar 24 00:22:09 smithi189 podman[70072]: 2024-03-24 00:22:09.674426576 +0000 UTC m=+0.352313689 container start dca63437cc7c74c655e371800e91eb29b294536dd147dea3f0df3cd301a8f382 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-h, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240319, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_CLEAN=True, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5) 2024-03-24T00:22:09.955 INFO:journalctl@ceph.mds.h.smithi189.stdout:Mar 24 00:22:09 smithi189 ceph-mds[70091]: -- 172.21.15.189:0/2820817620 <== mon.2 v2:172.21.15.189:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x563aa1a3fa40 con 0x563aa1a12800 2024-03-24T00:22:09.955 INFO:journalctl@ceph.mds.h.smithi189.stdout:Mar 24 00:22:09 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-h[70087]: starting mds.h at 2024-03-24T00:22:09.955 INFO:journalctl@ceph.mds.h.smithi189.stdout:Mar 24 00:22:09 smithi189 bash[70072]: dca63437cc7c74c655e371800e91eb29b294536dd147dea3f0df3cd301a8f382 2024-03-24T00:22:09.955 INFO:journalctl@ceph.mds.h.smithi189.stdout:Mar 24 00:22:09 smithi189 systemd[1]: Started Ceph mds.h for d79a5420-e971-11ee-95cd-87774f69a715. 2024-03-24T00:22:10.127 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:22:10.127+0000 7efcc2600640 1 -- 172.21.15.118:0/527195362 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7efcbc0772a0 msgr2=0x7efcbc077700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:22:10.127 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:22:10.127+0000 7efcc2600640 1 --2- 172.21.15.118:0/527195362 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7efcbc0772a0 0x7efcbc077700 secure :-1 s=READY pgs=434 cs=0 l=1 rev1=1 crypto rx=0x7efca4009900 tx=0x7efca402bc40 comp rx=0 tx=0).stop 2024-03-24T00:22:10.128 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:22:10.127+0000 7efcc2600640 1 -- 172.21.15.118:0/527195362 shutdown_connections 2024-03-24T00:22:10.128 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:22:10.127+0000 7efcc2600640 1 --2- 172.21.15.118:0/527195362 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7efcbc077c40 0x7efcbc0784e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:22:10.128 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:22:10.127+0000 7efcc2600640 1 --2- 172.21.15.118:0/527195362 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7efcbc0772a0 0x7efcbc077700 unknown :-1 s=CLOSED pgs=434 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:22:10.128 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:22:10.127+0000 7efcc2600640 1 --2- 172.21.15.118:0/527195362 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7efcbc076060 0x7efcbc076480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:22:10.128 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:22:10.127+0000 7efcc2600640 1 -- 172.21.15.118:0/527195362 >> 172.21.15.118:0/527195362 conn(0x7efcbc0fee60 msgr2=0x7efcbc1012a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:22:10.128 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:22:10.127+0000 7efcc2600640 1 -- 172.21.15.118:0/527195362 shutdown_connections 2024-03-24T00:22:10.128 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:22:10.127+0000 7efcc2600640 1 -- 172.21.15.118:0/527195362 wait complete. 2024-03-24T00:22:10.130 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:22:10.128+0000 7efcc2600640 1 Processor -- start 2024-03-24T00:22:10.130 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:22:10.128+0000 7efcc2600640 1 -- start start 2024-03-24T00:22:10.130 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:22:10.129+0000 7efcc2600640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7efcbc076060 0x7efcbc19df60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:22:10.130 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:22:10.129+0000 7efcc2600640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7efcbc0772a0 0x7efcbc19e4a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:22:10.131 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:22:10.129+0000 7efcc2600640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7efcbc077c40 0x7efcbc1a54c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:22:10.131 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:22:10.129+0000 7efcc2600640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efcbc113980 con 0x7efcbc076060 2024-03-24T00:22:10.131 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:22:10.129+0000 7efcc2600640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efcbc113800 con 0x7efcbc077c40 2024-03-24T00:22:10.131 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:22:10.129+0000 7efcc2600640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efcbc113b00 con 0x7efcbc0772a0 2024-03-24T00:22:10.131 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:22:10.129+0000 7efcbb400640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7efcbc076060 0x7efcbc19df60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:22:10.131 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:22:10.129+0000 7efcb2a00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7efcbc0772a0 0x7efcbc19e4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:22:10.131 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:22:10.129+0000 7efcb2a00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7efcbc0772a0 0x7efcbc19e4a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.118:51396/0 (socket says 172.21.15.118:51396) 2024-03-24T00:22:10.131 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:22:10.129+0000 7efcb2a00640 1 -- 172.21.15.118:0/3881971746 learned_addr learned my addr 172.21.15.118:0/3881971746 (peer_addr_for_me v2:172.21.15.118:0/0) 2024-03-24T00:22:10.131 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:22:10.129+0000 7efcbbe00640 1 --2- 172.21.15.118:0/3881971746 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7efcbc077c40 0x7efcbc1a54c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:22:10.131 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:22:10.129+0000 7efcb2a00640 1 -- 172.21.15.118:0/3881971746 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7efcbc077c40 msgr2=0x7efcbc1a54c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:22:10.131 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:22:10.129+0000 7efcb2a00640 1 --2- 172.21.15.118:0/3881971746 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7efcbc077c40 0x7efcbc1a54c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:22:10.132 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:22:10.129+0000 7efcb2a00640 1 -- 172.21.15.118:0/3881971746 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7efcbc076060 msgr2=0x7efcbc19df60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:22:10.132 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:22:10.129+0000 7efcb2a00640 1 --2- 172.21.15.118:0/3881971746 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7efcbc076060 0x7efcbc19df60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:22:10.132 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:22:10.129+0000 7efcb2a00640 1 -- 172.21.15.118:0/3881971746 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efcbc1a5b30 con 0x7efcbc0772a0 2024-03-24T00:22:10.132 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:22:10.130+0000 7efcb2a00640 1 --2- 172.21.15.118:0/3881971746 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7efcbc0772a0 0x7efcbc19e4a0 secure :-1 s=READY pgs=437 cs=0 l=1 rev1=1 crypto rx=0x7efca4009230 tx=0x7efca402c660 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:22:10.132 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:22:10.130+0000 7efcb8c00640 1 -- 172.21.15.118:0/3881971746 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efca4041070 con 0x7efcbc0772a0 2024-03-24T00:22:10.132 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:22:10.130+0000 7efcb8c00640 1 -- 172.21.15.118:0/3881971746 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7efca4004450 con 0x7efcbc0772a0 2024-03-24T00:22:10.132 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:22:10.130+0000 7efcc2600640 1 -- 172.21.15.118:0/3881971746 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efcbc1a5dc0 con 0x7efcbc0772a0 2024-03-24T00:22:10.132 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:22:10.130+0000 7efcb8c00640 1 -- 172.21.15.118:0/3881971746 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efca4003e00 con 0x7efcbc0772a0 2024-03-24T00:22:10.132 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:22:10.130+0000 7efcc2600640 1 -- 172.21.15.118:0/3881971746 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efcbc1a6270 con 0x7efcbc0772a0 2024-03-24T00:22:10.135 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:22:10.133+0000 7efcb8c00640 1 -- 172.21.15.118:0/3881971746 <== mon.1 v2:172.21.15.193:3300/0 4 ==== mgrmap(e 20) v1 ==== 137291+0+0 (secure 0 0 0) 0x7efca4044070 con 0x7efcbc0772a0 2024-03-24T00:22:10.135 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:22:10.133+0000 7efcc53d3740 1 Processor -- start 2024-03-24T00:22:10.135 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:22:10.133+0000 7efcc53d3740 1 -- start start 2024-03-24T00:22:10.135 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:22:10.133+0000 7efcc53d3740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x555dd2bf61e0 0x555dd2bf6600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:22:10.135 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:22:10.133+0000 7efcc53d3740 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x555dd2bf6bd0 0x555dd2bfb6d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:22:10.135 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:22:10.134+0000 7efcc53d3740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x555dd2bfbd40 0x555dd2bfe1d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:22:10.135 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:22:10.134+0000 7efcc53d3740 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x555dd2bb93b0 con 0x555dd2bf6bd0 2024-03-24T00:22:10.135 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:22:10.134+0000 7efcc53d3740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x555dd2b5b830 con 0x555dd2bf61e0 2024-03-24T00:22:10.135 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:22:10.134+0000 7efcc53d3740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x555dd2b045d0 con 0x555dd2bfbd40 2024-03-24T00:22:10.135 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:22:10.134+0000 7efcbb400640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x555dd2bf6bd0 0x555dd2bfb6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:22:10.136 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:22:10.134+0000 7efcbb400640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x555dd2bf6bd0 0x555dd2bfb6d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.118:3300/0 says I am v2:172.21.15.118:36168/0 (socket says 172.21.15.118:36168) 2024-03-24T00:22:10.136 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:22:10.134+0000 7efcbbe00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x555dd2bf61e0 0x555dd2bf6600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:22:10.136 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:22:10.134+0000 7efcbb400640 1 -- 172.21.15.118:0/1154769827 learned_addr learned my addr 172.21.15.118:0/1154769827 (peer_addr_for_me v2:172.21.15.118:0/0) 2024-03-24T00:22:10.136 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:22:10.134+0000 7efcb2a00640 1 --2- 172.21.15.118:0/1154769827 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x555dd2bfbd40 0x555dd2bfe1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:22:10.136 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:22:10.134+0000 7efcbb400640 1 -- 172.21.15.118:0/1154769827 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x555dd2bfbd40 msgr2=0x555dd2bfe1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:22:10.136 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:22:10.134+0000 7efcbb400640 1 --2- 172.21.15.118:0/1154769827 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x555dd2bfbd40 0x555dd2bfe1d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:22:10.136 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:22:10.134+0000 7efcbb400640 1 -- 172.21.15.118:0/1154769827 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x555dd2bf61e0 msgr2=0x555dd2bf6600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:22:10.136 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:22:10.134+0000 7efcbb400640 1 --2- 172.21.15.118:0/1154769827 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x555dd2bf61e0 0x555dd2bf6600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:22:10.136 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:22:10.134+0000 7efcbb400640 1 -- 172.21.15.118:0/1154769827 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x555dd2b1c2c0 con 0x555dd2bf6bd0 2024-03-24T00:22:10.136 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:22:10.134+0000 7efcbbe00640 1 --2- 172.21.15.118:0/1154769827 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x555dd2bf61e0 0x555dd2bf6600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T00:22:10.136 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:22:10.135+0000 7efcbb400640 1 --2- 172.21.15.118:0/1154769827 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x555dd2bf6bd0 0x555dd2bfb6d0 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7efcac009cf0 tx=0x7efcac011810 comp rx=0 tx=0).ready entity=mon.0 client_cookie=edc20c5a866da7ad server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:22:10.136 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:22:10.135+0000 7efcb8c00640 1 --2- 172.21.15.118:0/3881971746 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7efca00a2920 0x7efca00a4de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:22:10.137 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:22:10.135+0000 7efcb0c00640 1 -- 172.21.15.118:0/1154769827 <== mon.0 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efcac01e070 con 0x555dd2bf6bd0 2024-03-24T00:22:10.137 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:22:10.135+0000 7efcb0c00640 1 -- 172.21.15.118:0/1154769827 <== mon.0 v2:172.21.15.118:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7efcac019b10 con 0x555dd2bf6bd0 2024-03-24T00:22:10.137 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:22:10.135+0000 7efcb0c00640 1 -- 172.21.15.118:0/1154769827 <== mon.0 v2:172.21.15.118:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efcac00ea60 con 0x555dd2bf6bd0 2024-03-24T00:22:10.137 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:22:10.135+0000 7efcb8c00640 1 -- 172.21.15.118:0/3881971746 <== mon.1 v2:172.21.15.193:3300/0 5 ==== osd_map(106..106 src has 1..106) v4 ==== 9653+0+0 (secure 0 0 0) 0x7efca40f0b10 con 0x7efcbc0772a0 2024-03-24T00:22:10.137 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:22:10.135+0000 7efcbbe00640 1 --2- 172.21.15.118:0/3881971746 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7efca00a2920 0x7efca00a4de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:22:10.141 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:22:10.136+0000 7efcc53d3740 1 -- 172.21.15.118:0/1154769827 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x555dd2bf6bd0 msgr2=0x555dd2bfb6d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:22:10.141 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:22:10.136+0000 7efcc53d3740 1 --2- 172.21.15.118:0/1154769827 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x555dd2bf6bd0 0x555dd2bfb6d0 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7efcac009cf0 tx=0x7efcac011810 comp rx=0 tx=0).stop 2024-03-24T00:22:10.141 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:22:10.136+0000 7efcc53d3740 1 -- 172.21.15.118:0/1154769827 shutdown_connections 2024-03-24T00:22:10.142 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:22:10.136+0000 7efcc53d3740 1 --2- 172.21.15.118:0/1154769827 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x555dd2bfbd40 0x555dd2bfe1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:22:10.142 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:22:10.136+0000 7efcc53d3740 1 --2- 172.21.15.118:0/1154769827 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x555dd2bf6bd0 0x555dd2bfb6d0 unknown :-1 s=CLOSED pgs=217 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:22:10.142 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:22:10.136+0000 7efcc53d3740 1 --2- 172.21.15.118:0/1154769827 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x555dd2bf61e0 0x555dd2bf6600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:22:10.142 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:22:10.136+0000 7efcc53d3740 1 -- 172.21.15.118:0/1154769827 >> 172.21.15.118:0/1154769827 conn(0x555dd2bf1750 msgr2=0x555dd2bf1b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:22:10.142 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:22:10.136+0000 7efcbbe00640 1 --2- 172.21.15.118:0/3881971746 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7efca00a2920 0x7efca00a4de0 secure :-1 s=READY pgs=375 cs=0 l=1 rev1=1 crypto rx=0x555dd2bf7430 tx=0x7efcb4009500 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:22:10.142 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:22:10.136+0000 7efcc53d3740 1 -- 172.21.15.118:0/1154769827 shutdown_connections 2024-03-24T00:22:10.142 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:22:10.136+0000 7efcc53d3740 1 -- 172.21.15.118:0/1154769827 wait complete. 2024-03-24T00:22:10.142 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:22:10.136+0000 7efcc53d3740 1 -- start start 2024-03-24T00:22:10.142 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:22:10.137+0000 7efcc53d3740 1 Processor -- start 2024-03-24T00:22:10.142 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:22:10.137+0000 7efcc53d3740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x555dd2bf61e0 0x555dd2c003f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:22:10.142 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:22:10.137+0000 7efcc53d3740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x555dd2bf6bd0 0x555dd2c00930 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:22:10.142 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:22:10.137+0000 7efcc53d3740 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x555dd2bfbd40 0x555dd2c07900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:22:10.143 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:22:10.137+0000 7efcc53d3740 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x555dd2b17a20 con 0x555dd2bfbd40 2024-03-24T00:22:10.143 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:22:10.137+0000 7efcc53d3740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x555dd2b578d0 con 0x555dd2bf6bd0 2024-03-24T00:22:10.143 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:22:10.137+0000 7efcc53d3740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x555dd2bfecf0 con 0x555dd2bf61e0 2024-03-24T00:22:10.143 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:22:10.137+0000 7efcc53d3740 10 client.0 _ll_register_callbacks cb 0x555dd2b541c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T00:22:10.143 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:22:10.137+0000 7efcbb400640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x555dd2bf61e0 0x555dd2c003f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:22:10.143 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:22:10.137+0000 7efcbb400640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x555dd2bf61e0 0x555dd2c003f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.118:51422/0 (socket says 172.21.15.118:51422) 2024-03-24T00:22:10.143 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:22:10.137+0000 7efcbb400640 1 -- 172.21.15.118:0/1364477963 learned_addr learned my addr 172.21.15.118:0/1364477963 (peer_addr_for_me v2:172.21.15.118:0/0) 2024-03-24T00:22:10.143 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:22:10.137+0000 7efcbbe00640 1 --2- 172.21.15.118:0/1364477963 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x555dd2bfbd40 0x555dd2c07900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:22:10.143 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:22:10.137+0000 7efcb2a00640 1 --2- 172.21.15.118:0/1364477963 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x555dd2bf6bd0 0x555dd2c00930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:22:10.143 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:22:10.138+0000 7efcbb400640 1 -- 172.21.15.118:0/1364477963 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x555dd2bf6bd0 msgr2=0x555dd2c00930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:22:10.143 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:22:10.138+0000 7efcbb400640 1 --2- 172.21.15.118:0/1364477963 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x555dd2bf6bd0 0x555dd2c00930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:22:10.143 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:22:10.138+0000 7efcbb400640 1 -- 172.21.15.118:0/1364477963 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x555dd2bfbd40 msgr2=0x555dd2c07900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:22:10.143 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:22:10.138+0000 7efcbb400640 1 --2- 172.21.15.118:0/1364477963 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x555dd2bfbd40 0x555dd2c07900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:22:10.143 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:22:10.138+0000 7efcbb400640 1 -- 172.21.15.118:0/1364477963 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x555dd2870210 con 0x555dd2bf61e0 2024-03-24T00:22:10.143 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:22:10.138+0000 7efcbb400640 1 --2- 172.21.15.118:0/1364477963 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x555dd2bf61e0 0x555dd2c003f0 secure :-1 s=READY pgs=438 cs=0 l=1 rev1=1 crypto rx=0x7efcac000c00 tx=0x7efcac019660 comp rx=0 tx=0).ready entity=mon.1 client_cookie=7c72992c1d83d443 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:22:10.144 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:22:10.138+0000 7efca9600640 10 client.0 ms_handle_connect on v2:172.21.15.193:3300/0 2024-03-24T00:22:10.144 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:22:10.138+0000 7efca9600640 1 -- 172.21.15.118:0/1364477963 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efcac01e050 con 0x555dd2bf61e0 2024-03-24T00:22:10.144 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:22:10.138+0000 7efca9600640 1 -- 172.21.15.118:0/1364477963 <== mon.1 v2:172.21.15.193:3300/0 2 ==== osd_map(106..106 src has 1..106) v4 ==== 9653+0+0 (secure 0 0 0) 0x7efcac005240 con 0x555dd2bf61e0 2024-03-24T00:22:10.144 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:22:10.138+0000 7efcc53d3740 1 -- 172.21.15.118:0/1364477963 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7efcac019b10 con 0x555dd2bf61e0 2024-03-24T00:22:10.144 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:22:10.138+0000 7efca9600640 1 -- 172.21.15.118:0/1364477963 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_get_version_reply(handle=1 version=104) v2 ==== 24+0+0 (secure 0 0 0) 0x7efcac033050 con 0x555dd2bf61e0 2024-03-24T00:22:10.144 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:22:10.138+0000 7efcc53d3740 10 client.26851 fetch_fsmap learned FSMap version 104 2024-03-24T00:22:10.144 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:22:10.138+0000 7efcc53d3740 1 -- 172.21.15.118:0/1364477963 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,fsmap=104,monmap=4+}) v3 -- 0x555dd293c620 con 0x555dd2bf61e0 2024-03-24T00:22:10.144 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:22:10.139+0000 7efca9600640 1 -- 172.21.15.118:0/1364477963 <== mon.1 v2:172.21.15.193:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7efcac019b10 con 0x555dd2bf61e0 2024-03-24T00:22:10.144 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:22:10.139+0000 7efca9600640 1 -- 172.21.15.118:0/1364477963 <== mon.1 v2:172.21.15.193:3300/0 5 ==== fsmap(e 104) v1 ==== 7946+0+0 (secure 0 0 0) 0x7efcac0046c0 con 0x555dd2bf61e0 2024-03-24T00:22:10.144 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:22:10.139+0000 7efcc53d3740 10 client.26851 fetch_fsmap finished waiting for FSMap version 104 2024-03-24T00:22:10.144 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:22:10.139+0000 7efcc53d3740 10 client.26851 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-24T00:22:10.144 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:22:10.139+0000 7efcc53d3740 20 client.26851 populate_metadata read hostname 'smithi118' 2024-03-24T00:22:10.144 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:22:10.139+0000 7efcc53d3740 1 --2- 172.21.15.118:0/1364477963 >> [v2:172.21.15.189:6838/4074484168,v1:172.21.15.189:6839/4074484168] conn(0x555dd2c0bf60 0x555dd2c0e350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:22:10.144 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:22:10.139+0000 7efcc53d3740 4 client.26851 mds_command: new command op to 24569 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-24T00:22:10.144 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:22:10.139+0000 7efcc53d3740 1 -- 172.21.15.118:0/1364477963 --> [v2:172.21.15.189:6838/4074484168,v1:172.21.15.189:6839/4074484168] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x555dd2b68d70 con 0x555dd2c0bf60 2024-03-24T00:22:10.145 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:22:10.139+0000 7efcb2a00640 1 --2- 172.21.15.118:0/1364477963 >> [v2:172.21.15.189:6838/4074484168,v1:172.21.15.189:6839/4074484168] conn(0x555dd2c0bf60 0x555dd2c0e350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:22:10.145 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:22:10.140+0000 7efca9600640 10 client.26851 ms_handle_connect on v2:172.21.15.189:6838/4074484168 2024-03-24T00:22:10.145 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:22:10.140+0000 7efcb2a00640 1 --2- 172.21.15.118:0/1364477963 >> [v2:172.21.15.189:6838/4074484168,v1:172.21.15.189:6839/4074484168] conn(0x555dd2c0bf60 0x555dd2c0e350 crc :-1 s=READY pgs=249 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=e966dcc21ab6d7a0 server_cookie=7fb1b404587e2207 in_seq=0 out_seq=0 2024-03-24T00:22:10.145 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:22:10.141+0000 7efca9600640 1 -- 172.21.15.118:0/1364477963 <== mds.0 v2:172.21.15.189:6838/4074484168 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x555dd2b68d70 con 0x555dd2c0bf60 2024-03-24T00:22:10.145 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:22:10.141+0000 7efca9600640 10 client.26851 handle_command_reply: tid=0 2024-03-24T00:22:10.145 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:22:10.141+0000 7efcc53d3740 1 client.26851 shutdown 2024-03-24T00:22:10.145 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:22:10.141+0000 7efcc53d3740 1 -- 172.21.15.118:0/1364477963 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x555dd2bf61e0 msgr2=0x555dd2c003f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:22:10.145 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:22:10.141+0000 7efcc53d3740 1 --2- 172.21.15.118:0/1364477963 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x555dd2bf61e0 0x555dd2c003f0 secure :-1 s=READY pgs=438 cs=0 l=1 rev1=1 crypto rx=0x7efcac000c00 tx=0x7efcac019660 comp rx=0 tx=0).stop 2024-03-24T00:22:10.145 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:22:10.141+0000 7efcc53d3740 1 -- 172.21.15.118:0/1364477963 shutdown_connections 2024-03-24T00:22:10.145 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:22:10.141+0000 7efcc53d3740 1 --2- 172.21.15.118:0/1364477963 >> [v2:172.21.15.189:6838/4074484168,v1:172.21.15.189:6839/4074484168] conn(0x555dd2c0bf60 0x555dd2c0e350 crc :-1 s=READY pgs=249 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:22:10.145 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:22:10.141+0000 7efcc53d3740 1 --2- 172.21.15.118:0/1364477963 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x555dd2bfbd40 0x555dd2c07900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:22:10.145 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:22:10.141+0000 7efcc53d3740 1 --2- 172.21.15.118:0/1364477963 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x555dd2bf6bd0 0x555dd2c00930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:22:10.145 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:22:10.141+0000 7efcc53d3740 1 --2- 172.21.15.118:0/1364477963 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x555dd2bf61e0 0x555dd2c003f0 unknown :-1 s=CLOSED pgs=438 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:22:10.145 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:22:10.141+0000 7efca9600640 0 client.26851 ms_handle_reset on v2:172.21.15.189:6838/4074484168 2024-03-24T00:22:10.146 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:22:10.141+0000 7efcc53d3740 1 -- 172.21.15.118:0/1364477963 >> 172.21.15.118:0/1364477963 conn(0x555dd2bf1750 msgr2=0x555dd2bf0950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:22:10.146 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:22:10.141+0000 7efcc53d3740 1 -- 172.21.15.118:0/1364477963 shutdown_connections 2024-03-24T00:22:10.146 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:22:10.141+0000 7efcc53d3740 1 -- 172.21.15.118:0/1364477963 wait complete. 2024-03-24T00:22:10.146 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:22:10.141+0000 7efcc53d3740 20 client.26851 trim_cache size 0 max 16384 2024-03-24T00:22:10.157 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:22:10.156+0000 7efcc53d3740 1 Processor -- start 2024-03-24T00:22:10.157 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:22:10.156+0000 7efcc53d3740 1 -- start start 2024-03-24T00:22:10.157 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:22:10.156+0000 7efcc53d3740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x555dd2bf1750 0x555dd2bf50c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:22:10.157 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:22:10.156+0000 7efcc53d3740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x555dd2bf61e0 0x555dd2c078c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:22:10.157 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:22:10.156+0000 7efcc53d3740 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x555dd2bf6bd0 0x555dd2c12c10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:22:10.157 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:22:10.157+0000 7efcc53d3740 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efcbc1b0920 con 0x555dd2bf6bd0 2024-03-24T00:22:10.157 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:22:10.157+0000 7efcc53d3740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efcbc1b0f80 con 0x555dd2bf61e0 2024-03-24T00:22:10.158 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:22:10.157+0000 7efcc53d3740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efcbc1b1610 con 0x555dd2bf1750 2024-03-24T00:22:10.158 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:22:10.157+0000 7efcb2a00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x555dd2bf61e0 0x555dd2c078c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:22:10.158 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:22:10.157+0000 7efcb2a00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x555dd2bf61e0 0x555dd2c078c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.118:53174/0 (socket says 172.21.15.118:53174) 2024-03-24T00:22:10.158 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:22:10.157+0000 7efcb2a00640 1 -- 172.21.15.118:0/3278257901 learned_addr learned my addr 172.21.15.118:0/3278257901 (peer_addr_for_me v2:172.21.15.118:0/0) 2024-03-24T00:22:10.158 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:22:10.157+0000 7efcbb400640 1 --2- 172.21.15.118:0/3278257901 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x555dd2bf1750 0x555dd2bf50c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:22:10.158 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:22:10.157+0000 7efcb2a00640 1 -- 172.21.15.118:0/3278257901 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x555dd2bf1750 msgr2=0x555dd2bf50c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:22:10.158 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:22:10.157+0000 7efcb2a00640 1 --2- 172.21.15.118:0/3278257901 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x555dd2bf1750 0x555dd2bf50c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:22:10.158 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:22:10.157+0000 7efcbbe00640 1 --2- 172.21.15.118:0/3278257901 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x555dd2bf6bd0 0x555dd2c12c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:22:10.158 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:22:10.157+0000 7efcb2a00640 1 -- 172.21.15.118:0/3278257901 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x555dd2bf6bd0 msgr2=0x555dd2c12c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:22:10.158 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:22:10.157+0000 7efcb2a00640 1 --2- 172.21.15.118:0/3278257901 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x555dd2bf6bd0 0x555dd2c12c10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:22:10.158 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:22:10.157+0000 7efcb2a00640 1 -- 172.21.15.118:0/3278257901 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x555dd2bfb1e0 con 0x555dd2bf61e0 2024-03-24T00:22:10.158 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:22:10.157+0000 7efcb2a00640 1 --2- 172.21.15.118:0/3278257901 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x555dd2bf61e0 0x555dd2c078c0 secure :-1 s=READY pgs=435 cs=0 l=1 rev1=1 crypto rx=0x7efca40bccd0 tx=0x7efca40f0040 comp rx=0 tx=0).ready entity=mon.2 client_cookie=22265ee901a950ae server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:22:10.162 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:22:10.158+0000 7efcabe00640 1 -- 172.21.15.118:0/3278257901 <== mon.2 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efca404b040 con 0x555dd2bf61e0 2024-03-24T00:22:10.162 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:22:10.158+0000 7efcabe00640 1 -- 172.21.15.118:0/3278257901 <== mon.2 v2:172.21.15.189:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7efca40f8040 con 0x555dd2bf61e0 2024-03-24T00:22:10.162 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:22:10.158+0000 7efcabe00640 1 -- 172.21.15.118:0/3278257901 <== mon.2 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efca40c56c0 con 0x555dd2bf61e0 2024-03-24T00:22:10.162 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:22:10.158+0000 7efcc53d3740 1 -- 172.21.15.118:0/3278257901 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x555dd2bf61e0 msgr2=0x555dd2c078c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:22:10.162 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:22:10.158+0000 7efcc53d3740 1 --2- 172.21.15.118:0/3278257901 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x555dd2bf61e0 0x555dd2c078c0 secure :-1 s=READY pgs=435 cs=0 l=1 rev1=1 crypto rx=0x7efca40bccd0 tx=0x7efca40f0040 comp rx=0 tx=0).stop 2024-03-24T00:22:10.162 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:22:10.158+0000 7efcc53d3740 1 -- 172.21.15.118:0/3278257901 shutdown_connections 2024-03-24T00:22:10.163 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:22:10.158+0000 7efcc53d3740 1 --2- 172.21.15.118:0/3278257901 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x555dd2bf6bd0 0x555dd2c12c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:22:10.163 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:22:10.158+0000 7efcc53d3740 1 --2- 172.21.15.118:0/3278257901 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x555dd2bf61e0 0x555dd2c078c0 unknown :-1 s=CLOSED pgs=435 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:22:10.163 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:22:10.158+0000 7efcc53d3740 1 --2- 172.21.15.118:0/3278257901 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x555dd2bf1750 0x555dd2bf50c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:22:10.163 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:22:10.158+0000 7efcc53d3740 1 -- 172.21.15.118:0/3278257901 >> 172.21.15.118:0/3278257901 conn(0x555dd2bf77d0 msgr2=0x555dd2bf49b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:22:10.163 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:22:10.158+0000 7efcc53d3740 1 -- 172.21.15.118:0/3278257901 shutdown_connections 2024-03-24T00:22:10.163 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:22:10.158+0000 7efcc53d3740 1 -- 172.21.15.118:0/3278257901 wait complete. 2024-03-24T00:22:10.163 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:22:10.159+0000 7efcc53d3740 1 -- start start 2024-03-24T00:22:10.163 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:22:10.159+0000 7efcc53d3740 1 Processor -- start 2024-03-24T00:22:10.163 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:22:10.159+0000 7efcc53d3740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x555dd2bf1750 0x555dd2c05120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:22:10.163 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:22:10.159+0000 7efcc53d3740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x555dd2bf61e0 0x555dd2c05660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:22:10.163 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:22:10.159+0000 7efcc53d3740 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x555dd2bf6bd0 0x555dd2c06670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:22:10.163 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:22:10.159+0000 7efcc53d3740 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efcbc1adde0 con 0x555dd2bf6bd0 2024-03-24T00:22:10.163 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:22:10.159+0000 7efcc53d3740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efcbc1af080 con 0x555dd2bf61e0 2024-03-24T00:22:10.163 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:22:10.159+0000 7efcc53d3740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efcbc1ae3e0 con 0x555dd2bf1750 2024-03-24T00:22:10.163 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:22:10.159+0000 7efcc53d3740 10 client.0 _ll_register_callbacks cb 0x7efcac01c070 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T00:22:10.164 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:22:10.160+0000 7efcbb400640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x555dd2bf1750 0x555dd2c05120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:22:10.164 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:22:10.160+0000 7efcbb400640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x555dd2bf1750 0x555dd2c05120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.118:51444/0 (socket says 172.21.15.118:51444) 2024-03-24T00:22:10.164 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:22:10.160+0000 7efcbb400640 1 -- 172.21.15.118:0/2698747803 learned_addr learned my addr 172.21.15.118:0/2698747803 (peer_addr_for_me v2:172.21.15.118:0/0) 2024-03-24T00:22:10.164 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:22:10.160+0000 7efcb2a00640 1 --2- 172.21.15.118:0/2698747803 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x555dd2bf61e0 0x555dd2c05660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:22:10.164 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:22:10.160+0000 7efcbbe00640 1 --2- 172.21.15.118:0/2698747803 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x555dd2bf6bd0 0x555dd2c06670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:22:10.164 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:22:10.160+0000 7efcbb400640 1 -- 172.21.15.118:0/2698747803 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x555dd2bf61e0 msgr2=0x555dd2c05660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:22:10.164 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:22:10.160+0000 7efcbb400640 1 --2- 172.21.15.118:0/2698747803 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x555dd2bf61e0 0x555dd2c05660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:22:10.164 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:22:10.160+0000 7efcbb400640 1 -- 172.21.15.118:0/2698747803 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x555dd2bf6bd0 msgr2=0x555dd2c06670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:22:10.164 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:22:10.160+0000 7efcbb400640 1 --2- 172.21.15.118:0/2698747803 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x555dd2bf6bd0 0x555dd2c06670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:22:10.164 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:22:10.160+0000 7efcbb400640 1 -- 172.21.15.118:0/2698747803 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x555dd2c06c60 con 0x555dd2bf1750 2024-03-24T00:22:10.164 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:22:10.160+0000 7efcbb400640 1 --2- 172.21.15.118:0/2698747803 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x555dd2bf1750 0x555dd2c05120 secure :-1 s=READY pgs=439 cs=0 l=1 rev1=1 crypto rx=0x7efcac0095f0 tx=0x7efcac009a60 comp rx=0 tx=0).ready entity=mon.1 client_cookie=e24426b447850642 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:22:10.164 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:22:10.160+0000 7efca9600640 10 client.0 ms_handle_connect on v2:172.21.15.193:3300/0 2024-03-24T00:22:10.164 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:22:10.160+0000 7efca9600640 1 -- 172.21.15.118:0/2698747803 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7efcac01e040 con 0x555dd2bf1750 2024-03-24T00:22:10.165 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:22:10.160+0000 7efcc53d3740 1 -- 172.21.15.118:0/2698747803 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7efca40f8040 con 0x555dd2bf1750 2024-03-24T00:22:10.165 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:22:10.160+0000 7efca9600640 1 -- 172.21.15.118:0/2698747803 <== mon.1 v2:172.21.15.193:3300/0 2 ==== osd_map(106..106 src has 1..106) v4 ==== 9653+0+0 (secure 0 0 0) 0x7efcac019df0 con 0x555dd2bf1750 2024-03-24T00:22:10.165 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:22:10.161+0000 7efca9600640 1 -- 172.21.15.118:0/2698747803 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_get_version_reply(handle=1 version=104) v2 ==== 24+0+0 (secure 0 0 0) 0x7efcac005ab0 con 0x555dd2bf1750 2024-03-24T00:22:10.165 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:22:10.161+0000 7efcc53d3740 10 client.26857 fetch_fsmap learned FSMap version 104 2024-03-24T00:22:10.165 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:22:10.161+0000 7efcc53d3740 1 -- 172.21.15.118:0/2698747803 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,fsmap=104,monmap=4+}) v3 -- 0x555dd2c67960 con 0x555dd2bf1750 2024-03-24T00:22:10.165 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:22:10.161+0000 7efca9600640 1 -- 172.21.15.118:0/2698747803 <== mon.1 v2:172.21.15.193:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7efca40f8040 con 0x555dd2bf1750 2024-03-24T00:22:10.165 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:22:10.161+0000 7efca9600640 1 -- 172.21.15.118:0/2698747803 <== mon.1 v2:172.21.15.193:3300/0 5 ==== fsmap(e 104) v1 ==== 7946+0+0 (secure 0 0 0) 0x7efcac004470 con 0x555dd2bf1750 2024-03-24T00:22:10.165 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:22:10.161+0000 7efcc53d3740 10 client.26857 fetch_fsmap finished waiting for FSMap version 104 2024-03-24T00:22:10.165 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:22:10.161+0000 7efcc53d3740 10 client.26857 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-24T00:22:10.165 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:22:10.161+0000 7efcc53d3740 20 client.26857 populate_metadata read hostname 'smithi118' 2024-03-24T00:22:10.165 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:22:10.161+0000 7efcc53d3740 1 --2- 172.21.15.118:0/2698747803 >> [v2:172.21.15.189:6838/4074484168,v1:172.21.15.189:6839/4074484168] conn(0x555dd2bfbd40 0x555dd2c6a220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:22:10.165 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:22:10.161+0000 7efcc53d3740 4 client.26857 mds_command: new command op to 24569 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-03-24T00:22:10.165 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:22:10.161+0000 7efcc53d3740 1 -- 172.21.15.118:0/2698747803 --> [v2:172.21.15.189:6838/4074484168,v1:172.21.15.189:6839/4074484168] -- command(tid 0: {"prefix": "flush journal"}) v1 -- 0x555dd2beca80 con 0x555dd2bfbd40 2024-03-24T00:22:10.166 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:22:10.161+0000 7efcb2a00640 1 --2- 172.21.15.118:0/2698747803 >> [v2:172.21.15.189:6838/4074484168,v1:172.21.15.189:6839/4074484168] conn(0x555dd2bfbd40 0x555dd2c6a220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:22:10.166 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:22:10.162+0000 7efcb2a00640 1 --2- 172.21.15.118:0/2698747803 >> [v2:172.21.15.189:6838/4074484168,v1:172.21.15.189:6839/4074484168] conn(0x555dd2bfbd40 0x555dd2c6a220 crc :-1 s=READY pgs=250 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=f3c3a3b71357c413 server_cookie=b54d6d45874eedc0 in_seq=0 out_seq=0 2024-03-24T00:22:10.166 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:22:10.162+0000 7efca9600640 10 client.26857 ms_handle_connect on v2:172.21.15.189:6838/4074484168 2024-03-24T00:22:10.197 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:22:09 smithi118 ceph-mon[31535]: daemon mds.h restarted 2024-03-24T00:22:10.197 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:22:09 smithi118 ceph-mon[31535]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-03-24T00:22:10.198 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:22:09 smithi118 ceph-mon[31535]: osdmap e105: 12 total, 12 up, 12 in 2024-03-24T00:22:10.198 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:22:09 smithi118 ceph-mon[31535]: mds.? [v2:172.21.15.189:6836/75274228,v1:172.21.15.189:6837/75274228] up:boot 2024-03-24T00:22:10.198 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:22:09 smithi118 ceph-mon[31535]: Standby daemon mds.g assigned to filesystem cephfs as rank 3 2024-03-24T00:22:10.198 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:22:09 smithi118 ceph-mon[31535]: fsmap cephfs:4/5 {0=b=up:active,1=l=up:active,2=c=up:active,4=e=up:active} 5 up:standby-replay 3 up:standby, 1 failed 2024-03-24T00:22:10.199 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:22:09 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-03-24T00:22:10.199 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:22:09 smithi118 ceph-mon[31535]: fsmap cephfs:5/5 4 up:standby-replay 3 up:standby 1 up:replay 4 up:active 2024-03-24T00:22:10.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:22:09 smithi193 ceph-mon[36878]: daemon mds.h restarted 2024-03-24T00:22:10.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:22:09 smithi193 ceph-mon[36878]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-03-24T00:22:10.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:22:09 smithi193 ceph-mon[36878]: osdmap e105: 12 total, 12 up, 12 in 2024-03-24T00:22:10.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:22:09 smithi193 ceph-mon[36878]: mds.? [v2:172.21.15.189:6836/75274228,v1:172.21.15.189:6837/75274228] up:boot 2024-03-24T00:22:10.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:22:09 smithi193 ceph-mon[36878]: Standby daemon mds.g assigned to filesystem cephfs as rank 3 2024-03-24T00:22:10.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:22:09 smithi193 ceph-mon[36878]: fsmap cephfs:4/5 {0=b=up:active,1=l=up:active,2=c=up:active,4=e=up:active} 5 up:standby-replay 3 up:standby, 1 failed 2024-03-24T00:22:10.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:22:09 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-03-24T00:22:10.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:22:09 smithi193 ceph-mon[36878]: fsmap cephfs:5/5 4 up:standby-replay 3 up:standby 1 up:replay 4 up:active 2024-03-24T00:22:10.211 INFO:journalctl@ceph.mds.l.smithi193.stdout:Mar 24 00:22:09 smithi193 podman[72458]: 2024-03-24 00:22:09.774874977 +0000 UTC m=+0.456062523 container init 34b63e7de437dbf746219f051fd02d843764da00bec1829324daf12d71db4db5 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-l, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240319, GIT_BRANCH=HEAD) 2024-03-24T00:22:10.212 INFO:journalctl@ceph.mds.l.smithi193.stdout:Mar 24 00:22:09 smithi193 podman[72458]: 2024-03-24 00:22:09.777637718 +0000 UTC m=+0.458825268 container start 34b63e7de437dbf746219f051fd02d843764da00bec1829324daf12d71db4db5 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-l, org.label-schema.schema-version=1.0, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, GIT_BRANCH=HEAD, GIT_CLEAN=True, ceph=True, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240319) 2024-03-24T00:22:10.212 INFO:journalctl@ceph.mds.l.smithi193.stdout:Mar 24 00:22:09 smithi193 ceph-mds[72477]: -- 172.21.15.193:0/2921879403 <== mon.1 v2:172.21.15.193:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x55770c70fa40 con 0x55770c6e3000 2024-03-24T00:22:10.212 INFO:journalctl@ceph.mds.l.smithi193.stdout:Mar 24 00:22:09 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-l[72473]: starting mds.l at 2024-03-24T00:22:10.212 INFO:journalctl@ceph.mds.l.smithi193.stdout:Mar 24 00:22:09 smithi193 bash[72458]: 34b63e7de437dbf746219f051fd02d843764da00bec1829324daf12d71db4db5 2024-03-24T00:22:10.212 INFO:journalctl@ceph.mds.l.smithi193.stdout:Mar 24 00:22:09 smithi193 systemd[1]: Started Ceph mds.l for d79a5420-e971-11ee-95cd-87774f69a715. 2024-03-24T00:22:11.026 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:22:10 smithi189 ceph-mon[36187]: daemon mds.l restarted 2024-03-24T00:22:11.026 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:22:10 smithi189 ceph-mon[36187]: pgmap v608: 97 pgs: 97 active+clean; 136 MiB data, 790 MiB used, 1.0 TiB / 1.0 TiB avail; 732 KiB/s rd, 97 KiB/s wr, 71 op/s 2024-03-24T00:22:11.026 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:22:10 smithi189 ceph-mon[36187]: osdmap e106: 12 total, 12 up, 12 in 2024-03-24T00:22:11.026 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:22:10 smithi189 ceph-mon[36187]: mds.? [v2:172.21.15.193:6832/2424695757,v1:172.21.15.193:6833/2424695757] up:boot 2024-03-24T00:22:11.026 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:22:10 smithi189 ceph-mon[36187]: Standby daemon mds.f assigned to filesystem cephfs as rank 1 2024-03-24T00:22:11.026 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:22:10 smithi189 ceph-mon[36187]: fsmap cephfs:4/5 {0=b=up:active,2=c=up:active,3=g=up:replay,4=e=up:active} 4 up:standby-replay 4 up:standby, 1 failed 2024-03-24T00:22:11.026 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:22:10 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "l"}]: dispatch 2024-03-24T00:22:11.026 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:22:10 smithi189 ceph-mon[36187]: fsmap cephfs:5/5 3 up:standby-replay 4 up:standby 2 up:replay 3 up:active 2024-03-24T00:22:11.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:22:10 smithi118 ceph-mon[31535]: daemon mds.l restarted 2024-03-24T00:22:11.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:22:10 smithi118 ceph-mon[31535]: pgmap v608: 97 pgs: 97 active+clean; 136 MiB data, 790 MiB used, 1.0 TiB / 1.0 TiB avail; 732 KiB/s rd, 97 KiB/s wr, 71 op/s 2024-03-24T00:22:11.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:22:10 smithi118 ceph-mon[31535]: osdmap e106: 12 total, 12 up, 12 in 2024-03-24T00:22:11.197 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:22:10 smithi118 ceph-mon[31535]: mds.? [v2:172.21.15.193:6832/2424695757,v1:172.21.15.193:6833/2424695757] up:boot 2024-03-24T00:22:11.197 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:22:10 smithi118 ceph-mon[31535]: Standby daemon mds.f assigned to filesystem cephfs as rank 1 2024-03-24T00:22:11.197 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:22:10 smithi118 ceph-mon[31535]: fsmap cephfs:4/5 {0=b=up:active,2=c=up:active,3=g=up:replay,4=e=up:active} 4 up:standby-replay 4 up:standby, 1 failed 2024-03-24T00:22:11.197 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:22:10 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "l"}]: dispatch 2024-03-24T00:22:11.197 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:22:10 smithi118 ceph-mon[31535]: fsmap cephfs:5/5 3 up:standby-replay 4 up:standby 2 up:replay 3 up:active 2024-03-24T00:22:11.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:22:10 smithi193 ceph-mon[36878]: daemon mds.l restarted 2024-03-24T00:22:11.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:22:10 smithi193 ceph-mon[36878]: pgmap v608: 97 pgs: 97 active+clean; 136 MiB data, 790 MiB used, 1.0 TiB / 1.0 TiB avail; 732 KiB/s rd, 97 KiB/s wr, 71 op/s 2024-03-24T00:22:11.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:22:10 smithi193 ceph-mon[36878]: osdmap e106: 12 total, 12 up, 12 in 2024-03-24T00:22:11.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:22:10 smithi193 ceph-mon[36878]: mds.? [v2:172.21.15.193:6832/2424695757,v1:172.21.15.193:6833/2424695757] up:boot 2024-03-24T00:22:11.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:22:10 smithi193 ceph-mon[36878]: Standby daemon mds.f assigned to filesystem cephfs as rank 1 2024-03-24T00:22:11.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:22:10 smithi193 ceph-mon[36878]: fsmap cephfs:4/5 {0=b=up:active,2=c=up:active,3=g=up:replay,4=e=up:active} 4 up:standby-replay 4 up:standby, 1 failed 2024-03-24T00:22:11.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:22:10 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "l"}]: dispatch 2024-03-24T00:22:11.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:22:10 smithi193 ceph-mon[36878]: fsmap cephfs:5/5 3 up:standby-replay 4 up:standby 2 up:replay 3 up:active 2024-03-24T00:22:12.099 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:22:11 smithi118 ceph-mon[31535]: mds.? [v2:172.21.15.118:6838/3600700815,v1:172.21.15.118:6839/3600700815] up:resolve 2024-03-24T00:22:12.099 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:22:11 smithi118 ceph-mon[31535]: mds.? [v2:172.21.15.193:6834/2264220953,v1:172.21.15.193:6835/2264220953] up:resolve 2024-03-24T00:22:12.099 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:22:11 smithi118 ceph-mon[31535]: fsmap cephfs:5/5 3 up:standby-replay 4 up:standby 2 up:resolve 3 up:active 2024-03-24T00:22:12.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:22:11 smithi189 ceph-mon[36187]: mds.? [v2:172.21.15.118:6838/3600700815,v1:172.21.15.118:6839/3600700815] up:resolve 2024-03-24T00:22:12.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:22:11 smithi189 ceph-mon[36187]: mds.? [v2:172.21.15.193:6834/2264220953,v1:172.21.15.193:6835/2264220953] up:resolve 2024-03-24T00:22:12.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:22:11 smithi189 ceph-mon[36187]: fsmap cephfs:5/5 3 up:standby-replay 4 up:standby 2 up:resolve 3 up:active 2024-03-24T00:22:12.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:22:11 smithi193 ceph-mon[36878]: mds.? [v2:172.21.15.118:6838/3600700815,v1:172.21.15.118:6839/3600700815] up:resolve 2024-03-24T00:22:12.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:22:11 smithi193 ceph-mon[36878]: mds.? [v2:172.21.15.193:6834/2264220953,v1:172.21.15.193:6835/2264220953] up:resolve 2024-03-24T00:22:12.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:22:11 smithi193 ceph-mon[36878]: fsmap cephfs:5/5 3 up:standby-replay 4 up:standby 2 up:resolve 3 up:active 2024-03-24T00:22:12.899 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:22:12.898+0000 7efca9600640 1 -- 172.21.15.118:0/2698747803 <== mds.0 v2:172.21.15.189:6838/4074484168 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+44 (crc 0 0 0) 0x555dd2beca80 con 0x555dd2bfbd40 2024-03-24T00:22:12.899 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:22:12.898+0000 7efca9600640 10 client.26857 handle_command_reply: tid=0 2024-03-24T00:22:12.899 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:22:12.899+0000 7efcc53d3740 1 client.26857 shutdown 2024-03-24T00:22:12.899 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:22:12.899+0000 7efcc53d3740 1 -- 172.21.15.118:0/2698747803 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x555dd2bf1750 msgr2=0x555dd2c05120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:22:12.899 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:22:12.899+0000 7efcc53d3740 1 --2- 172.21.15.118:0/2698747803 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x555dd2bf1750 0x555dd2c05120 secure :-1 s=READY pgs=439 cs=0 l=1 rev1=1 crypto rx=0x7efcac0095f0 tx=0x7efcac009a60 comp rx=0 tx=0).stop 2024-03-24T00:22:12.900 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:22:12.899+0000 7efcc53d3740 1 -- 172.21.15.118:0/2698747803 shutdown_connections 2024-03-24T00:22:12.900 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:22:12.899+0000 7efcc53d3740 1 --2- 172.21.15.118:0/2698747803 >> [v2:172.21.15.189:6838/4074484168,v1:172.21.15.189:6839/4074484168] conn(0x555dd2bfbd40 0x555dd2c6a220 crc :-1 s=READY pgs=250 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:22:12.901 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:22:12.899+0000 7efcc53d3740 1 --2- 172.21.15.118:0/2698747803 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x555dd2bf6bd0 0x555dd2c06670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:22:12.901 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:22:12.899+0000 7efcc53d3740 1 --2- 172.21.15.118:0/2698747803 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x555dd2bf61e0 0x555dd2c05660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:22:12.901 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:22:12.899+0000 7efcc53d3740 1 --2- 172.21.15.118:0/2698747803 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x555dd2bf1750 0x555dd2c05120 unknown :-1 s=CLOSED pgs=439 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:22:12.901 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:22:12.899+0000 7efcc53d3740 1 -- 172.21.15.118:0/2698747803 >> 172.21.15.118:0/2698747803 conn(0x555dd2bf77d0 msgr2=0x555dd2bf5240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:22:12.901 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:22:12.900+0000 7efca9600640 0 client.26857 ms_handle_reset on v2:172.21.15.189:6838/4074484168 2024-03-24T00:22:12.901 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:22:12.900+0000 7efcc53d3740 1 -- 172.21.15.118:0/2698747803 shutdown_connections 2024-03-24T00:22:12.901 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:22:12.900+0000 7efcc53d3740 1 -- 172.21.15.118:0/2698747803 wait complete. 2024-03-24T00:22:12.901 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:22:12.900+0000 7efcc53d3740 20 client.26857 trim_cache size 0 max 16384 2024-03-24T00:22:12.901 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stdout:{ 2024-03-24T00:22:12.901 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stdout: "message": "", 2024-03-24T00:22:12.901 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stdout: "return_code": 0 2024-03-24T00:22:12.902 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stdout:} 2024-03-24T00:22:12.902 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:22:12.902+0000 7efcc2600640 1 -- 172.21.15.118:0/3881971746 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7efca00a2920 msgr2=0x7efca00a4de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:22:12.903 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:22:12.902+0000 7efcc2600640 1 --2- 172.21.15.118:0/3881971746 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7efca00a2920 0x7efca00a4de0 secure :-1 s=READY pgs=375 cs=0 l=1 rev1=1 crypto rx=0x555dd2bf7430 tx=0x7efcb4009500 comp rx=0 tx=0).stop 2024-03-24T00:22:12.903 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:22:12.902+0000 7efcc2600640 1 -- 172.21.15.118:0/3881971746 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7efcbc0772a0 msgr2=0x7efcbc19e4a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:22:12.903 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:22:12.902+0000 7efcc2600640 1 --2- 172.21.15.118:0/3881971746 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7efcbc0772a0 0x7efcbc19e4a0 secure :-1 s=READY pgs=437 cs=0 l=1 rev1=1 crypto rx=0x7efca4009230 tx=0x7efca402c660 comp rx=0 tx=0).stop 2024-03-24T00:22:12.903 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:22:12.903+0000 7efcc2600640 1 -- 172.21.15.118:0/3881971746 shutdown_connections 2024-03-24T00:22:12.903 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:22:12.903+0000 7efcc2600640 1 --2- 172.21.15.118:0/3881971746 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7efca00a2920 0x7efca00a4de0 unknown :-1 s=CLOSED pgs=375 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:22:12.903 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:22:12.903+0000 7efcc2600640 1 --2- 172.21.15.118:0/3881971746 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7efcbc077c40 0x7efcbc1a54c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:22:12.903 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:22:12.903+0000 7efcc2600640 1 --2- 172.21.15.118:0/3881971746 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7efcbc0772a0 0x7efcbc19e4a0 unknown :-1 s=CLOSED pgs=437 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:22:12.903 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:22:12.903+0000 7efcc2600640 1 --2- 172.21.15.118:0/3881971746 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7efcbc076060 0x7efcbc19df60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:22:12.903 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:22:12.903+0000 7efcc2600640 1 -- 172.21.15.118:0/3881971746 >> 172.21.15.118:0/3881971746 conn(0x7efcbc0fee60 msgr2=0x7efcbc100960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:22:12.903 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:22:12.903+0000 7efcc2600640 1 -- 172.21.15.118:0/3881971746 shutdown_connections 2024-03-24T00:22:12.903 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:22:12.903+0000 7efcc2600640 1 -- 172.21.15.118:0/3881971746 wait complete. 2024-03-24T00:22:13.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:22:12 smithi118 ceph-mon[31535]: pgmap v609: 97 pgs: 97 active+clean; 136 MiB data, 790 MiB used, 1.0 TiB / 1.0 TiB avail; 135 KiB/s rd, 118 KiB/s wr, 36 op/s 2024-03-24T00:22:13.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:22:12 smithi118 ceph-mon[31535]: mds.? [v2:172.21.15.118:6838/3600700815,v1:172.21.15.118:6839/3600700815] up:reconnect 2024-03-24T00:22:13.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:22:12 smithi118 ceph-mon[31535]: mds.? [v2:172.21.15.193:6834/2264220953,v1:172.21.15.193:6835/2264220953] up:reconnect 2024-03-24T00:22:13.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:22:12 smithi118 ceph-mon[31535]: fsmap cephfs:5/5 3 up:standby-replay 4 up:standby 2 up:reconnect 3 up:active 2024-03-24T00:22:13.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:22:12 smithi118 ceph-mon[31535]: reconnect by client.15159 192.168.0.1:0/3463739313 after 0.0469993 2024-03-24T00:22:13.197 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:22:12 smithi118 ceph-mon[31535]: reconnect by client.15159 192.168.0.1:0/3463739313 after 0.0559987 2024-03-24T00:22:13.203 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:22:12 smithi189 ceph-mon[36187]: pgmap v609: 97 pgs: 97 active+clean; 136 MiB data, 790 MiB used, 1.0 TiB / 1.0 TiB avail; 135 KiB/s rd, 118 KiB/s wr, 36 op/s 2024-03-24T00:22:13.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:22:12 smithi189 ceph-mon[36187]: mds.? [v2:172.21.15.118:6838/3600700815,v1:172.21.15.118:6839/3600700815] up:reconnect 2024-03-24T00:22:13.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:22:12 smithi189 ceph-mon[36187]: mds.? [v2:172.21.15.193:6834/2264220953,v1:172.21.15.193:6835/2264220953] up:reconnect 2024-03-24T00:22:13.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:22:12 smithi189 ceph-mon[36187]: fsmap cephfs:5/5 3 up:standby-replay 4 up:standby 2 up:reconnect 3 up:active 2024-03-24T00:22:13.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:22:12 smithi189 ceph-mon[36187]: reconnect by client.15159 192.168.0.1:0/3463739313 after 0.0469993 2024-03-24T00:22:13.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:22:12 smithi189 ceph-mon[36187]: reconnect by client.15159 192.168.0.1:0/3463739313 after 0.0559987 2024-03-24T00:22:13.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:22:12 smithi193 ceph-mon[36878]: pgmap v609: 97 pgs: 97 active+clean; 136 MiB data, 790 MiB used, 1.0 TiB / 1.0 TiB avail; 135 KiB/s rd, 118 KiB/s wr, 36 op/s 2024-03-24T00:22:13.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:22:12 smithi193 ceph-mon[36878]: mds.? [v2:172.21.15.118:6838/3600700815,v1:172.21.15.118:6839/3600700815] up:reconnect 2024-03-24T00:22:13.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:22:12 smithi193 ceph-mon[36878]: mds.? [v2:172.21.15.193:6834/2264220953,v1:172.21.15.193:6835/2264220953] up:reconnect 2024-03-24T00:22:13.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:22:12 smithi193 ceph-mon[36878]: fsmap cephfs:5/5 3 up:standby-replay 4 up:standby 2 up:reconnect 3 up:active 2024-03-24T00:22:13.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:22:12 smithi193 ceph-mon[36878]: reconnect by client.15159 192.168.0.1:0/3463739313 after 0.0469993 2024-03-24T00:22:13.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:22:12 smithi193 ceph-mon[36878]: reconnect by client.15159 192.168.0.1:0/3463739313 after 0.0559987 2024-03-24T00:22:14.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:22:13 smithi118 ceph-mon[31535]: mds.? [v2:172.21.15.193:6834/2264220953,v1:172.21.15.193:6835/2264220953] up:rejoin 2024-03-24T00:22:14.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:22:13 smithi118 ceph-mon[31535]: mds.? [v2:172.21.15.118:6838/3600700815,v1:172.21.15.118:6839/3600700815] up:rejoin 2024-03-24T00:22:14.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:22:13 smithi118 ceph-mon[31535]: fsmap cephfs:5/5 3 up:standby-replay 4 up:standby 2 up:rejoin 3 up:active 2024-03-24T00:22:14.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:22:13 smithi118 ceph-mon[31535]: daemon mds.f is now active in filesystem cephfs as rank 1 2024-03-24T00:22:14.197 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:22:13 smithi118 ceph-mon[31535]: daemon mds.g is now active in filesystem cephfs as rank 3 2024-03-24T00:22:14.197 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:22:13 smithi118 ceph-mon[31535]: pgmap v610: 97 pgs: 97 active+clean; 136 MiB data, 792 MiB used, 1.0 TiB / 1.0 TiB avail; 54 KiB/s rd, 157 KiB/s wr, 60 op/s 2024-03-24T00:22:14.200 INFO:journalctl@ceph.mds.g.smithi118.stdout:Mar 24 00:22:13 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-g[94655]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: In function 'void Locker::set_xlocks_done(MutationImpl*, bool)' thread 7f4533000700 time 2024-03-24T00:22:13.938055+0000 2024-03-24T00:22:14.200 INFO:journalctl@ceph.mds.g.smithi118.stdout:Mar 24 00:22:13 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-g[94655]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: 626: FAILED ceph_assert(obj->is_auth()) 2024-03-24T00:22:14.200 INFO:journalctl@ceph.mds.g.smithi118.stdout:Mar 24 00:22:13 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-g[94655]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-24T00:22:14.201 INFO:journalctl@ceph.mds.g.smithi118.stdout:Mar 24 00:22:13 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-g[94655]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x135) [0x7f453e8724a1] 2024-03-24T00:22:14.201 INFO:journalctl@ceph.mds.g.smithi118.stdout:Mar 24 00:22:13 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-g[94655]: 2: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7f453e872667] 2024-03-24T00:22:14.201 INFO:journalctl@ceph.mds.g.smithi118.stdout:Mar 24 00:22:13 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-g[94655]: 3: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x55c8b16b8223] 2024-03-24T00:22:14.201 INFO:journalctl@ceph.mds.g.smithi118.stdout:Mar 24 00:22:13 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-g[94655]: 4: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x55c8b15325d6] 2024-03-24T00:22:14.201 INFO:journalctl@ceph.mds.g.smithi118.stdout:Mar 24 00:22:13 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-g[94655]: 5: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x55c8b1532da1] 2024-03-24T00:22:14.201 INFO:journalctl@ceph.mds.g.smithi118.stdout:Mar 24 00:22:13 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-g[94655]: 6: (Server::handle_client_rename(boost::intrusive_ptr const&)+0x259d) [0x55c8b156f97d] 2024-03-24T00:22:14.201 INFO:journalctl@ceph.mds.g.smithi118.stdout:Mar 24 00:22:13 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-g[94655]: 7: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x55c8b1585233] 2024-03-24T00:22:14.201 INFO:journalctl@ceph.mds.g.smithi118.stdout:Mar 24 00:22:13 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-g[94655]: 8: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x55c8b1587197] 2024-03-24T00:22:14.201 INFO:journalctl@ceph.mds.g.smithi118.stdout:Mar 24 00:22:13 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-g[94655]: 9: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x55c8b148611b] 2024-03-24T00:22:14.201 INFO:journalctl@ceph.mds.g.smithi118.stdout:Mar 24 00:22:13 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-g[94655]: 10: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x55c8b14866bc] 2024-03-24T00:22:14.201 INFO:journalctl@ceph.mds.g.smithi118.stdout:Mar 24 00:22:13 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-g[94655]: 11: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x55c8b14701ff] 2024-03-24T00:22:14.202 INFO:journalctl@ceph.mds.g.smithi118.stdout:Mar 24 00:22:13 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-g[94655]: 12: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7f453eb04488] 2024-03-24T00:22:14.202 INFO:journalctl@ceph.mds.g.smithi118.stdout:Mar 24 00:22:13 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-g[94655]: 13: (DispatchQueue::entry()+0x51f) [0x7f453eb0160f] 2024-03-24T00:22:14.202 INFO:journalctl@ceph.mds.g.smithi118.stdout:Mar 24 00:22:13 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-g[94655]: 14: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f453ebd4e81] 2024-03-24T00:22:14.202 INFO:journalctl@ceph.mds.g.smithi118.stdout:Mar 24 00:22:13 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-g[94655]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7f453d5ef1ca] 2024-03-24T00:22:14.202 INFO:journalctl@ceph.mds.g.smithi118.stdout:Mar 24 00:22:13 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-g[94655]: 16: clone() 2024-03-24T00:22:14.202 INFO:journalctl@ceph.mds.g.smithi118.stdout:Mar 24 00:22:13 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-g[94655]: 2024-03-24T00:22:13.939+0000 7f4533000700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: In function 'void Locker::set_xlocks_done(MutationImpl*, bool)' thread 7f4533000700 time 2024-03-24T00:22:13.938055+0000 2024-03-24T00:22:14.202 INFO:journalctl@ceph.mds.g.smithi118.stdout:Mar 24 00:22:13 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-g[94655]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: 626: FAILED ceph_assert(obj->is_auth()) 2024-03-24T00:22:14.202 INFO:journalctl@ceph.mds.g.smithi118.stdout:Mar 24 00:22:13 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-g[94655]: 2024-03-24T00:22:14.202 INFO:journalctl@ceph.mds.g.smithi118.stdout:Mar 24 00:22:13 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-g[94655]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-24T00:22:14.202 INFO:journalctl@ceph.mds.g.smithi118.stdout:Mar 24 00:22:13 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-g[94655]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x135) [0x7f453e8724a1] 2024-03-24T00:22:14.202 INFO:journalctl@ceph.mds.g.smithi118.stdout:Mar 24 00:22:13 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-g[94655]: 2: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7f453e872667] 2024-03-24T00:22:14.202 INFO:journalctl@ceph.mds.g.smithi118.stdout:Mar 24 00:22:13 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-g[94655]: 3: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x55c8b16b8223] 2024-03-24T00:22:14.202 INFO:journalctl@ceph.mds.g.smithi118.stdout:Mar 24 00:22:13 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-g[94655]: 4: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x55c8b15325d6] 2024-03-24T00:22:14.202 INFO:journalctl@ceph.mds.g.smithi118.stdout:Mar 24 00:22:13 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-g[94655]: 5: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x55c8b1532da1] 2024-03-24T00:22:14.203 INFO:journalctl@ceph.mds.g.smithi118.stdout:Mar 24 00:22:13 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-g[94655]: 6: (Server::handle_client_rename(boost::intrusive_ptr const&)+0x259d) [0x55c8b156f97d] 2024-03-24T00:22:14.203 INFO:journalctl@ceph.mds.g.smithi118.stdout:Mar 24 00:22:13 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-g[94655]: 7: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x55c8b1585233] 2024-03-24T00:22:14.203 INFO:journalctl@ceph.mds.g.smithi118.stdout:Mar 24 00:22:13 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-g[94655]: 8: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x55c8b1587197] 2024-03-24T00:22:14.203 INFO:journalctl@ceph.mds.g.smithi118.stdout:Mar 24 00:22:13 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-g[94655]: 9: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x55c8b148611b] 2024-03-24T00:22:14.203 INFO:journalctl@ceph.mds.g.smithi118.stdout:Mar 24 00:22:13 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-g[94655]: 10: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x55c8b14866bc] 2024-03-24T00:22:14.203 INFO:journalctl@ceph.mds.g.smithi118.stdout:Mar 24 00:22:13 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-g[94655]: 11: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x55c8b14701ff] 2024-03-24T00:22:14.203 INFO:journalctl@ceph.mds.g.smithi118.stdout:Mar 24 00:22:13 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-g[94655]: 12: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7f453eb04488] 2024-03-24T00:22:14.203 INFO:journalctl@ceph.mds.g.smithi118.stdout:Mar 24 00:22:13 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-g[94655]: 13: (DispatchQueue::entry()+0x51f) [0x7f453eb0160f] 2024-03-24T00:22:14.203 INFO:journalctl@ceph.mds.g.smithi118.stdout:Mar 24 00:22:13 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-g[94655]: 14: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f453ebd4e81] 2024-03-24T00:22:14.203 INFO:journalctl@ceph.mds.g.smithi118.stdout:Mar 24 00:22:13 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-g[94655]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7f453d5ef1ca] 2024-03-24T00:22:14.203 INFO:journalctl@ceph.mds.g.smithi118.stdout:Mar 24 00:22:13 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-g[94655]: 16: clone() 2024-03-24T00:22:14.203 INFO:journalctl@ceph.mds.g.smithi118.stdout:Mar 24 00:22:13 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-g[94655]: 2024-03-24T00:22:14.203 INFO:journalctl@ceph.mds.g.smithi118.stdout:Mar 24 00:22:13 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-g[94655]: *** Caught signal (Aborted) ** 2024-03-24T00:22:14.203 INFO:journalctl@ceph.mds.g.smithi118.stdout:Mar 24 00:22:13 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-g[94655]: in thread 7f4533000700 thread_name:ms_dispatch 2024-03-24T00:22:14.203 INFO:journalctl@ceph.mds.g.smithi118.stdout:Mar 24 00:22:13 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-g[94655]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-24T00:22:14.204 INFO:journalctl@ceph.mds.g.smithi118.stdout:Mar 24 00:22:13 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-g[94655]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f453d5f9d20] 2024-03-24T00:22:14.204 INFO:journalctl@ceph.mds.g.smithi118.stdout:Mar 24 00:22:13 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-g[94655]: 2: gsignal() 2024-03-24T00:22:14.204 INFO:journalctl@ceph.mds.g.smithi118.stdout:Mar 24 00:22:13 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-g[94655]: 3: abort() 2024-03-24T00:22:14.204 INFO:journalctl@ceph.mds.g.smithi118.stdout:Mar 24 00:22:13 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-g[94655]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x18f) [0x7f453e8724fb] 2024-03-24T00:22:14.204 INFO:journalctl@ceph.mds.g.smithi118.stdout:Mar 24 00:22:13 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-g[94655]: 5: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7f453e872667] 2024-03-24T00:22:14.204 INFO:journalctl@ceph.mds.g.smithi118.stdout:Mar 24 00:22:13 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-g[94655]: 6: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x55c8b16b8223] 2024-03-24T00:22:14.204 INFO:journalctl@ceph.mds.g.smithi118.stdout:Mar 24 00:22:13 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-g[94655]: 7: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x55c8b15325d6] 2024-03-24T00:22:14.204 INFO:journalctl@ceph.mds.g.smithi118.stdout:Mar 24 00:22:13 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-g[94655]: 8: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x55c8b1532da1] 2024-03-24T00:22:14.204 INFO:journalctl@ceph.mds.g.smithi118.stdout:Mar 24 00:22:13 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-g[94655]: 9: (Server::handle_client_rename(boost::intrusive_ptr const&)+0x259d) [0x55c8b156f97d] 2024-03-24T00:22:14.204 INFO:journalctl@ceph.mds.g.smithi118.stdout:Mar 24 00:22:13 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-g[94655]: 10: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x55c8b1585233] 2024-03-24T00:22:14.204 INFO:journalctl@ceph.mds.g.smithi118.stdout:Mar 24 00:22:13 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-g[94655]: 11: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x55c8b1587197] 2024-03-24T00:22:14.204 INFO:journalctl@ceph.mds.g.smithi118.stdout:Mar 24 00:22:13 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-g[94655]: 12: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x55c8b148611b] 2024-03-24T00:22:14.204 INFO:journalctl@ceph.mds.g.smithi118.stdout:Mar 24 00:22:13 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-g[94655]: 13: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x55c8b14866bc] 2024-03-24T00:22:14.204 INFO:journalctl@ceph.mds.g.smithi118.stdout:Mar 24 00:22:13 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-g[94655]: 14: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x55c8b14701ff] 2024-03-24T00:22:14.205 INFO:journalctl@ceph.mds.g.smithi118.stdout:Mar 24 00:22:13 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-g[94655]: 15: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7f453eb04488] 2024-03-24T00:22:14.205 INFO:journalctl@ceph.mds.g.smithi118.stdout:Mar 24 00:22:13 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-g[94655]: 16: (DispatchQueue::entry()+0x51f) [0x7f453eb0160f] 2024-03-24T00:22:14.205 INFO:journalctl@ceph.mds.g.smithi118.stdout:Mar 24 00:22:13 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-g[94655]: 17: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f453ebd4e81] 2024-03-24T00:22:14.205 INFO:journalctl@ceph.mds.g.smithi118.stdout:Mar 24 00:22:13 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-g[94655]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7f453d5ef1ca] 2024-03-24T00:22:14.205 INFO:journalctl@ceph.mds.g.smithi118.stdout:Mar 24 00:22:13 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-g[94655]: 19: clone() 2024-03-24T00:22:14.205 INFO:journalctl@ceph.mds.g.smithi118.stdout:Mar 24 00:22:13 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-g[94655]: 2024-03-24T00:22:13.941+0000 7f4533000700 -1 *** Caught signal (Aborted) ** 2024-03-24T00:22:14.205 INFO:journalctl@ceph.mds.g.smithi118.stdout:Mar 24 00:22:13 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-g[94655]: in thread 7f4533000700 thread_name:ms_dispatch 2024-03-24T00:22:14.205 INFO:journalctl@ceph.mds.g.smithi118.stdout:Mar 24 00:22:13 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-g[94655]: 2024-03-24T00:22:14.205 INFO:journalctl@ceph.mds.g.smithi118.stdout:Mar 24 00:22:13 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-g[94655]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-24T00:22:14.205 INFO:journalctl@ceph.mds.g.smithi118.stdout:Mar 24 00:22:13 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-g[94655]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f453d5f9d20] 2024-03-24T00:22:14.205 INFO:journalctl@ceph.mds.g.smithi118.stdout:Mar 24 00:22:13 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-g[94655]: 2: gsignal() 2024-03-24T00:22:14.205 INFO:journalctl@ceph.mds.g.smithi118.stdout:Mar 24 00:22:13 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-g[94655]: 3: abort() 2024-03-24T00:22:14.205 INFO:journalctl@ceph.mds.g.smithi118.stdout:Mar 24 00:22:13 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-g[94655]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x18f) [0x7f453e8724fb] 2024-03-24T00:22:14.205 INFO:journalctl@ceph.mds.g.smithi118.stdout:Mar 24 00:22:13 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-g[94655]: 5: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7f453e872667] 2024-03-24T00:22:14.205 INFO:journalctl@ceph.mds.g.smithi118.stdout:Mar 24 00:22:13 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-g[94655]: 6: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x55c8b16b8223] 2024-03-24T00:22:14.206 INFO:journalctl@ceph.mds.g.smithi118.stdout:Mar 24 00:22:13 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-g[94655]: 7: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x55c8b15325d6] 2024-03-24T00:22:14.206 INFO:journalctl@ceph.mds.g.smithi118.stdout:Mar 24 00:22:13 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-g[94655]: 8: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x55c8b1532da1] 2024-03-24T00:22:14.206 INFO:journalctl@ceph.mds.g.smithi118.stdout:Mar 24 00:22:13 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-g[94655]: 9: (Server::handle_client_rename(boost::intrusive_ptr const&)+0x259d) [0x55c8b156f97d] 2024-03-24T00:22:14.206 INFO:journalctl@ceph.mds.g.smithi118.stdout:Mar 24 00:22:13 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-g[94655]: 10: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x55c8b1585233] 2024-03-24T00:22:14.206 INFO:journalctl@ceph.mds.g.smithi118.stdout:Mar 24 00:22:13 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-g[94655]: 11: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x55c8b1587197] 2024-03-24T00:22:14.206 INFO:journalctl@ceph.mds.g.smithi118.stdout:Mar 24 00:22:13 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-g[94655]: 12: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x55c8b148611b] 2024-03-24T00:22:14.206 INFO:journalctl@ceph.mds.g.smithi118.stdout:Mar 24 00:22:13 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-g[94655]: 13: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x55c8b14866bc] 2024-03-24T00:22:14.206 INFO:journalctl@ceph.mds.g.smithi118.stdout:Mar 24 00:22:13 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-g[94655]: 14: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x55c8b14701ff] 2024-03-24T00:22:14.206 INFO:journalctl@ceph.mds.g.smithi118.stdout:Mar 24 00:22:13 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-g[94655]: 15: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7f453eb04488] 2024-03-24T00:22:14.206 INFO:journalctl@ceph.mds.g.smithi118.stdout:Mar 24 00:22:13 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-g[94655]: 16: (DispatchQueue::entry()+0x51f) [0x7f453eb0160f] 2024-03-24T00:22:14.206 INFO:journalctl@ceph.mds.g.smithi118.stdout:Mar 24 00:22:13 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-g[94655]: 17: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f453ebd4e81] 2024-03-24T00:22:14.206 INFO:journalctl@ceph.mds.g.smithi118.stdout:Mar 24 00:22:13 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-g[94655]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7f453d5ef1ca] 2024-03-24T00:22:14.207 INFO:journalctl@ceph.mds.g.smithi118.stdout:Mar 24 00:22:13 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-g[94655]: 19: clone() 2024-03-24T00:22:14.207 INFO:journalctl@ceph.mds.g.smithi118.stdout:Mar 24 00:22:13 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-g[94655]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-03-24T00:22:14.207 INFO:journalctl@ceph.mds.g.smithi118.stdout:Mar 24 00:22:13 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-g[94655]: 2024-03-24T00:22:14.207 INFO:journalctl@ceph.mds.g.smithi118.stdout:Mar 24 00:22:13 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-g[94655]: -1> 2024-03-24T00:22:13.939+0000 7f4533000700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: In function 'void Locker::set_xlocks_done(MutationImpl*, bool)' thread 7f4533000700 time 2024-03-24T00:22:13.938055+0000 2024-03-24T00:22:14.207 INFO:journalctl@ceph.mds.g.smithi118.stdout:Mar 24 00:22:13 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-g[94655]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: 626: FAILED ceph_assert(obj->is_auth()) 2024-03-24T00:22:14.207 INFO:journalctl@ceph.mds.g.smithi118.stdout:Mar 24 00:22:13 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-g[94655]: 2024-03-24T00:22:14.207 INFO:journalctl@ceph.mds.g.smithi118.stdout:Mar 24 00:22:13 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-g[94655]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-24T00:22:14.207 INFO:journalctl@ceph.mds.g.smithi118.stdout:Mar 24 00:22:13 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-g[94655]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x135) [0x7f453e8724a1] 2024-03-24T00:22:14.207 INFO:journalctl@ceph.mds.g.smithi118.stdout:Mar 24 00:22:13 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-g[94655]: 2: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7f453e872667] 2024-03-24T00:22:14.208 INFO:journalctl@ceph.mds.g.smithi118.stdout:Mar 24 00:22:13 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-g[94655]: 3: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x55c8b16b8223] 2024-03-24T00:22:14.208 INFO:journalctl@ceph.mds.g.smithi118.stdout:Mar 24 00:22:13 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-g[94655]: 4: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x55c8b15325d6] 2024-03-24T00:22:14.208 INFO:journalctl@ceph.mds.g.smithi118.stdout:Mar 24 00:22:13 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-g[94655]: 5: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x55c8b1532da1] 2024-03-24T00:22:14.208 INFO:journalctl@ceph.mds.g.smithi118.stdout:Mar 24 00:22:13 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-g[94655]: 6: (Server::handle_client_rename(boost::intrusive_ptr const&)+0x259d) [0x55c8b156f97d] 2024-03-24T00:22:14.208 INFO:journalctl@ceph.mds.g.smithi118.stdout:Mar 24 00:22:13 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-g[94655]: 7: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x55c8b1585233] 2024-03-24T00:22:14.208 INFO:journalctl@ceph.mds.g.smithi118.stdout:Mar 24 00:22:13 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-g[94655]: 8: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x55c8b1587197] 2024-03-24T00:22:14.208 INFO:journalctl@ceph.mds.g.smithi118.stdout:Mar 24 00:22:13 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-g[94655]: 9: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x55c8b148611b] 2024-03-24T00:22:14.208 INFO:journalctl@ceph.mds.g.smithi118.stdout:Mar 24 00:22:13 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-g[94655]: 10: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x55c8b14866bc] 2024-03-24T00:22:14.208 INFO:journalctl@ceph.mds.g.smithi118.stdout:Mar 24 00:22:13 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-g[94655]: 11: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x55c8b14701ff] 2024-03-24T00:22:14.208 INFO:journalctl@ceph.mds.g.smithi118.stdout:Mar 24 00:22:13 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-g[94655]: 12: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7f453eb04488] 2024-03-24T00:22:14.208 INFO:journalctl@ceph.mds.g.smithi118.stdout:Mar 24 00:22:13 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-g[94655]: 13: (DispatchQueue::entry()+0x51f) [0x7f453eb0160f] 2024-03-24T00:22:14.208 INFO:journalctl@ceph.mds.g.smithi118.stdout:Mar 24 00:22:13 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-g[94655]: 14: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f453ebd4e81] 2024-03-24T00:22:14.208 INFO:journalctl@ceph.mds.g.smithi118.stdout:Mar 24 00:22:13 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-g[94655]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7f453d5ef1ca] 2024-03-24T00:22:14.209 INFO:journalctl@ceph.mds.g.smithi118.stdout:Mar 24 00:22:13 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-g[94655]: 16: clone() 2024-03-24T00:22:14.209 INFO:journalctl@ceph.mds.g.smithi118.stdout:Mar 24 00:22:13 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-g[94655]: 2024-03-24T00:22:14.209 INFO:journalctl@ceph.mds.g.smithi118.stdout:Mar 24 00:22:13 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-g[94655]: 0> 2024-03-24T00:22:13.941+0000 7f4533000700 -1 *** Caught signal (Aborted) ** 2024-03-24T00:22:14.209 INFO:journalctl@ceph.mds.g.smithi118.stdout:Mar 24 00:22:13 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-g[94655]: in thread 7f4533000700 thread_name:ms_dispatch 2024-03-24T00:22:14.209 INFO:journalctl@ceph.mds.g.smithi118.stdout:Mar 24 00:22:13 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-g[94655]: 2024-03-24T00:22:14.209 INFO:journalctl@ceph.mds.g.smithi118.stdout:Mar 24 00:22:13 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-g[94655]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-24T00:22:14.209 INFO:journalctl@ceph.mds.g.smithi118.stdout:Mar 24 00:22:13 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-g[94655]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f453d5f9d20] 2024-03-24T00:22:14.209 INFO:journalctl@ceph.mds.g.smithi118.stdout:Mar 24 00:22:13 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-g[94655]: 2: gsignal() 2024-03-24T00:22:14.209 INFO:journalctl@ceph.mds.g.smithi118.stdout:Mar 24 00:22:13 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-g[94655]: 3: abort() 2024-03-24T00:22:14.209 INFO:journalctl@ceph.mds.g.smithi118.stdout:Mar 24 00:22:13 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-g[94655]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x18f) [0x7f453e8724fb] 2024-03-24T00:22:14.210 INFO:journalctl@ceph.mds.g.smithi118.stdout:Mar 24 00:22:13 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-g[94655]: 5: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7f453e872667] 2024-03-24T00:22:14.210 INFO:journalctl@ceph.mds.g.smithi118.stdout:Mar 24 00:22:13 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-g[94655]: 6: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x55c8b16b8223] 2024-03-24T00:22:14.210 INFO:journalctl@ceph.mds.g.smithi118.stdout:Mar 24 00:22:13 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-g[94655]: 7: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x55c8b15325d6] 2024-03-24T00:22:14.210 INFO:journalctl@ceph.mds.g.smithi118.stdout:Mar 24 00:22:13 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-g[94655]: 8: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x55c8b1532da1] 2024-03-24T00:22:14.210 INFO:journalctl@ceph.mds.g.smithi118.stdout:Mar 24 00:22:13 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-g[94655]: 9: (Server::handle_client_rename(boost::intrusive_ptr const&)+0x259d) [0x55c8b156f97d] 2024-03-24T00:22:14.210 INFO:journalctl@ceph.mds.g.smithi118.stdout:Mar 24 00:22:13 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-g[94655]: 10: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x55c8b1585233] 2024-03-24T00:22:14.210 INFO:journalctl@ceph.mds.g.smithi118.stdout:Mar 24 00:22:13 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-g[94655]: 11: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x55c8b1587197] 2024-03-24T00:22:14.210 INFO:journalctl@ceph.mds.g.smithi118.stdout:Mar 24 00:22:13 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-g[94655]: 12: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x55c8b148611b] 2024-03-24T00:22:14.210 INFO:journalctl@ceph.mds.g.smithi118.stdout:Mar 24 00:22:13 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-g[94655]: 13: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x55c8b14866bc] 2024-03-24T00:22:14.210 INFO:journalctl@ceph.mds.g.smithi118.stdout:Mar 24 00:22:13 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-g[94655]: 14: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x55c8b14701ff] 2024-03-24T00:22:14.210 INFO:journalctl@ceph.mds.g.smithi118.stdout:Mar 24 00:22:13 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-g[94655]: 15: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7f453eb04488] 2024-03-24T00:22:14.210 INFO:journalctl@ceph.mds.g.smithi118.stdout:Mar 24 00:22:13 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-g[94655]: 16: (DispatchQueue::entry()+0x51f) [0x7f453eb0160f] 2024-03-24T00:22:14.210 INFO:journalctl@ceph.mds.g.smithi118.stdout:Mar 24 00:22:13 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-g[94655]: 17: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f453ebd4e81] 2024-03-24T00:22:14.210 INFO:journalctl@ceph.mds.g.smithi118.stdout:Mar 24 00:22:13 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-g[94655]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7f453d5ef1ca] 2024-03-24T00:22:14.211 INFO:journalctl@ceph.mds.g.smithi118.stdout:Mar 24 00:22:13 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-g[94655]: 19: clone() 2024-03-24T00:22:14.211 INFO:journalctl@ceph.mds.g.smithi118.stdout:Mar 24 00:22:13 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-g[94655]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-03-24T00:22:14.211 INFO:journalctl@ceph.mds.g.smithi118.stdout:Mar 24 00:22:13 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-g[94655]: 2024-03-24T00:22:14.211 INFO:journalctl@ceph.mds.g.smithi118.stdout:Mar 24 00:22:13 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-g[94655]: -1> 2024-03-24T00:22:13.939+0000 7f4533000700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: In function 'void Locker::set_xlocks_done(MutationImpl*, bool)' thread 7f4533000700 time 2024-03-24T00:22:13.938055+0000 2024-03-24T00:22:14.211 INFO:journalctl@ceph.mds.g.smithi118.stdout:Mar 24 00:22:13 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-g[94655]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: 626: FAILED ceph_assert(obj->is_auth()) 2024-03-24T00:22:14.211 INFO:journalctl@ceph.mds.g.smithi118.stdout:Mar 24 00:22:13 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-g[94655]: 2024-03-24T00:22:14.211 INFO:journalctl@ceph.mds.g.smithi118.stdout:Mar 24 00:22:13 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-g[94655]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-24T00:22:14.211 INFO:journalctl@ceph.mds.g.smithi118.stdout:Mar 24 00:22:13 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-g[94655]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x135) [0x7f453e8724a1] 2024-03-24T00:22:14.211 INFO:journalctl@ceph.mds.g.smithi118.stdout:Mar 24 00:22:13 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-g[94655]: 2: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7f453e872667] 2024-03-24T00:22:14.211 INFO:journalctl@ceph.mds.g.smithi118.stdout:Mar 24 00:22:13 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-g[94655]: 3: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x55c8b16b8223] 2024-03-24T00:22:14.211 INFO:journalctl@ceph.mds.g.smithi118.stdout:Mar 24 00:22:13 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-g[94655]: 4: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x55c8b15325d6] 2024-03-24T00:22:14.212 INFO:journalctl@ceph.mds.g.smithi118.stdout:Mar 24 00:22:13 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-g[94655]: 5: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x55c8b1532da1] 2024-03-24T00:22:14.212 INFO:journalctl@ceph.mds.g.smithi118.stdout:Mar 24 00:22:13 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-g[94655]: 6: (Server::handle_client_rename(boost::intrusive_ptr const&)+0x259d) [0x55c8b156f97d] 2024-03-24T00:22:14.212 INFO:journalctl@ceph.mds.g.smithi118.stdout:Mar 24 00:22:13 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-g[94655]: 7: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x55c8b1585233] 2024-03-24T00:22:14.212 INFO:journalctl@ceph.mds.g.smithi118.stdout:Mar 24 00:22:13 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-g[94655]: 8: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x55c8b1587197] 2024-03-24T00:22:14.212 INFO:journalctl@ceph.mds.g.smithi118.stdout:Mar 24 00:22:13 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-g[94655]: 9: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x55c8b148611b] 2024-03-24T00:22:14.212 INFO:journalctl@ceph.mds.g.smithi118.stdout:Mar 24 00:22:13 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-g[94655]: 10: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x55c8b14866bc] 2024-03-24T00:22:14.212 INFO:journalctl@ceph.mds.g.smithi118.stdout:Mar 24 00:22:13 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-g[94655]: 11: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x55c8b14701ff] 2024-03-24T00:22:14.212 INFO:journalctl@ceph.mds.g.smithi118.stdout:Mar 24 00:22:13 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-g[94655]: 12: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7f453eb04488] 2024-03-24T00:22:14.212 INFO:journalctl@ceph.mds.g.smithi118.stdout:Mar 24 00:22:13 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-g[94655]: 13: (DispatchQueue::entry()+0x51f) [0x7f453eb0160f] 2024-03-24T00:22:14.212 INFO:journalctl@ceph.mds.g.smithi118.stdout:Mar 24 00:22:13 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-g[94655]: 14: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f453ebd4e81] 2024-03-24T00:22:14.212 INFO:journalctl@ceph.mds.g.smithi118.stdout:Mar 24 00:22:13 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-g[94655]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7f453d5ef1ca] 2024-03-24T00:22:14.212 INFO:journalctl@ceph.mds.g.smithi118.stdout:Mar 24 00:22:13 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-g[94655]: 16: clone() 2024-03-24T00:22:14.212 INFO:journalctl@ceph.mds.g.smithi118.stdout:Mar 24 00:22:13 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-g[94655]: 2024-03-24T00:22:14.213 INFO:journalctl@ceph.mds.g.smithi118.stdout:Mar 24 00:22:13 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-g[94655]: 0> 2024-03-24T00:22:13.941+0000 7f4533000700 -1 *** Caught signal (Aborted) ** 2024-03-24T00:22:14.213 INFO:journalctl@ceph.mds.g.smithi118.stdout:Mar 24 00:22:13 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-g[94655]: in thread 7f4533000700 thread_name:ms_dispatch 2024-03-24T00:22:14.213 INFO:journalctl@ceph.mds.g.smithi118.stdout:Mar 24 00:22:13 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-g[94655]: 2024-03-24T00:22:14.213 INFO:journalctl@ceph.mds.g.smithi118.stdout:Mar 24 00:22:13 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-g[94655]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-24T00:22:14.213 INFO:journalctl@ceph.mds.g.smithi118.stdout:Mar 24 00:22:13 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-g[94655]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f453d5f9d20] 2024-03-24T00:22:14.213 INFO:journalctl@ceph.mds.g.smithi118.stdout:Mar 24 00:22:13 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-g[94655]: 2: gsignal() 2024-03-24T00:22:14.213 INFO:journalctl@ceph.mds.g.smithi118.stdout:Mar 24 00:22:13 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-g[94655]: 3: abort() 2024-03-24T00:22:14.213 INFO:journalctl@ceph.mds.g.smithi118.stdout:Mar 24 00:22:13 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-g[94655]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x18f) [0x7f453e8724fb] 2024-03-24T00:22:14.213 INFO:journalctl@ceph.mds.g.smithi118.stdout:Mar 24 00:22:13 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-g[94655]: 5: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7f453e872667] 2024-03-24T00:22:14.213 INFO:journalctl@ceph.mds.g.smithi118.stdout:Mar 24 00:22:13 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-g[94655]: 6: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x55c8b16b8223] 2024-03-24T00:22:14.213 INFO:journalctl@ceph.mds.g.smithi118.stdout:Mar 24 00:22:13 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-g[94655]: 7: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x55c8b15325d6] 2024-03-24T00:22:14.213 INFO:journalctl@ceph.mds.g.smithi118.stdout:Mar 24 00:22:13 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-g[94655]: 8: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x55c8b1532da1] 2024-03-24T00:22:14.214 INFO:journalctl@ceph.mds.g.smithi118.stdout:Mar 24 00:22:13 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-g[94655]: 9: (Server::handle_client_rename(boost::intrusive_ptr const&)+0x259d) [0x55c8b156f97d] 2024-03-24T00:22:14.214 INFO:journalctl@ceph.mds.g.smithi118.stdout:Mar 24 00:22:13 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-g[94655]: 10: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x55c8b1585233] 2024-03-24T00:22:14.215 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:22:13 smithi193 ceph-mon[36878]: mds.? [v2:172.21.15.193:6834/2264220953,v1:172.21.15.193:6835/2264220953] up:rejoin 2024-03-24T00:22:14.215 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:22:13 smithi193 ceph-mon[36878]: mds.? [v2:172.21.15.118:6838/3600700815,v1:172.21.15.118:6839/3600700815] up:rejoin 2024-03-24T00:22:14.215 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:22:13 smithi193 ceph-mon[36878]: fsmap cephfs:5/5 3 up:standby-replay 4 up:standby 2 up:rejoin 3 up:active 2024-03-24T00:22:14.215 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:22:13 smithi193 ceph-mon[36878]: daemon mds.f is now active in filesystem cephfs as rank 1 2024-03-24T00:22:14.215 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:22:13 smithi193 ceph-mon[36878]: daemon mds.g is now active in filesystem cephfs as rank 3 2024-03-24T00:22:14.215 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:22:13 smithi193 ceph-mon[36878]: pgmap v610: 97 pgs: 97 active+clean; 136 MiB data, 792 MiB used, 1.0 TiB / 1.0 TiB avail; 54 KiB/s rd, 157 KiB/s wr, 60 op/s 2024-03-24T00:22:14.215 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:22:13 smithi189 ceph-mon[36187]: mds.? [v2:172.21.15.193:6834/2264220953,v1:172.21.15.193:6835/2264220953] up:rejoin 2024-03-24T00:22:14.215 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:22:13 smithi189 ceph-mon[36187]: mds.? [v2:172.21.15.118:6838/3600700815,v1:172.21.15.118:6839/3600700815] up:rejoin 2024-03-24T00:22:14.215 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:22:13 smithi189 ceph-mon[36187]: fsmap cephfs:5/5 3 up:standby-replay 4 up:standby 2 up:rejoin 3 up:active 2024-03-24T00:22:14.216 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:22:13 smithi189 ceph-mon[36187]: daemon mds.f is now active in filesystem cephfs as rank 1 2024-03-24T00:22:14.216 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:22:13 smithi189 ceph-mon[36187]: daemon mds.g is now active in filesystem cephfs as rank 3 2024-03-24T00:22:14.216 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:22:13 smithi189 ceph-mon[36187]: pgmap v610: 97 pgs: 97 active+clean; 136 MiB data, 792 MiB used, 1.0 TiB / 1.0 TiB avail; 54 KiB/s rd, 157 KiB/s wr, 60 op/s 2024-03-24T00:22:14.216 INFO:journalctl@ceph.mds.g.smithi118.stdout:Mar 24 00:22:13 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-g[94655]: 11: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x55c8b1587197] 2024-03-24T00:22:14.216 INFO:journalctl@ceph.mds.g.smithi118.stdout:Mar 24 00:22:13 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-g[94655]: 12: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x55c8b148611b] 2024-03-24T00:22:14.216 INFO:journalctl@ceph.mds.g.smithi118.stdout:Mar 24 00:22:13 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-g[94655]: 13: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x55c8b14866bc] 2024-03-24T00:22:14.216 INFO:journalctl@ceph.mds.g.smithi118.stdout:Mar 24 00:22:13 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-g[94655]: 14: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x55c8b14701ff] 2024-03-24T00:22:14.216 INFO:journalctl@ceph.mds.g.smithi118.stdout:Mar 24 00:22:13 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-g[94655]: 15: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7f453eb04488] 2024-03-24T00:22:14.216 INFO:journalctl@ceph.mds.g.smithi118.stdout:Mar 24 00:22:13 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-g[94655]: 16: (DispatchQueue::entry()+0x51f) [0x7f453eb0160f] 2024-03-24T00:22:14.216 INFO:journalctl@ceph.mds.g.smithi118.stdout:Mar 24 00:22:13 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-g[94655]: 17: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f453ebd4e81] 2024-03-24T00:22:14.216 INFO:journalctl@ceph.mds.g.smithi118.stdout:Mar 24 00:22:13 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-g[94655]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7f453d5ef1ca] 2024-03-24T00:22:14.216 INFO:journalctl@ceph.mds.g.smithi118.stdout:Mar 24 00:22:13 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-g[94655]: 19: clone() 2024-03-24T00:22:14.216 INFO:journalctl@ceph.mds.g.smithi118.stdout:Mar 24 00:22:13 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-g[94655]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-03-24T00:22:14.216 INFO:journalctl@ceph.mds.g.smithi118.stdout:Mar 24 00:22:13 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-g[94655]: 2024-03-24T00:22:14.217 INFO:journalctl@ceph.mds.g.smithi118.stdout:Mar 24 00:22:14 smithi118 podman[94787]: 2024-03-24 00:22:14.051927217 +0000 UTC m=+0.042572018 container died 0498d0005714887d6ee54e9ea72d8f7bd3fe80cfc028a92db30adaa4b85c2076 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-g, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_CLEAN=True, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.build-date=20240319, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c) 2024-03-24T00:22:14.218 INFO:journalctl@ceph.mds.f.smithi193.stdout:Mar 24 00:22:13 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-f[72273]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: In function 'void Locker::set_xlocks_done(MutationImpl*, bool)' thread 7f335b187700 time 2024-03-24T00:22:13.971324+0000 2024-03-24T00:22:14.218 INFO:journalctl@ceph.mds.f.smithi193.stdout:Mar 24 00:22:13 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-f[72273]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: 626: FAILED ceph_assert(obj->is_auth()) 2024-03-24T00:22:14.218 INFO:journalctl@ceph.mds.f.smithi193.stdout:Mar 24 00:22:13 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-f[72273]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-24T00:22:14.218 INFO:journalctl@ceph.mds.f.smithi193.stdout:Mar 24 00:22:13 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-f[72273]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x135) [0x7f33657b14a1] 2024-03-24T00:22:14.218 INFO:journalctl@ceph.mds.f.smithi193.stdout:Mar 24 00:22:13 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-f[72273]: 2: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7f33657b1667] 2024-03-24T00:22:14.218 INFO:journalctl@ceph.mds.f.smithi193.stdout:Mar 24 00:22:13 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-f[72273]: 3: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x55883bebb223] 2024-03-24T00:22:14.218 INFO:journalctl@ceph.mds.f.smithi193.stdout:Mar 24 00:22:13 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-f[72273]: 4: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x55883bd355d6] 2024-03-24T00:22:14.218 INFO:journalctl@ceph.mds.f.smithi193.stdout:Mar 24 00:22:13 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-f[72273]: 5: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x55883bd35da1] 2024-03-24T00:22:14.218 INFO:journalctl@ceph.mds.f.smithi193.stdout:Mar 24 00:22:13 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-f[72273]: 6: (Server::handle_client_rename(boost::intrusive_ptr const&)+0x259d) [0x55883bd7297d] 2024-03-24T00:22:14.218 INFO:journalctl@ceph.mds.f.smithi193.stdout:Mar 24 00:22:13 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-f[72273]: 7: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x55883bd88233] 2024-03-24T00:22:14.218 INFO:journalctl@ceph.mds.f.smithi193.stdout:Mar 24 00:22:13 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-f[72273]: 8: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x55883bd8a197] 2024-03-24T00:22:14.218 INFO:journalctl@ceph.mds.f.smithi193.stdout:Mar 24 00:22:13 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-f[72273]: 9: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x55883bc8911b] 2024-03-24T00:22:14.219 INFO:journalctl@ceph.mds.f.smithi193.stdout:Mar 24 00:22:13 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-f[72273]: 10: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x55883bc896bc] 2024-03-24T00:22:14.219 INFO:journalctl@ceph.mds.f.smithi193.stdout:Mar 24 00:22:13 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-f[72273]: 11: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x55883bc731ff] 2024-03-24T00:22:14.219 INFO:journalctl@ceph.mds.f.smithi193.stdout:Mar 24 00:22:13 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-f[72273]: 12: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7f3365a43488] 2024-03-24T00:22:14.219 INFO:journalctl@ceph.mds.f.smithi193.stdout:Mar 24 00:22:13 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-f[72273]: 13: (DispatchQueue::entry()+0x51f) [0x7f3365a4060f] 2024-03-24T00:22:14.219 INFO:journalctl@ceph.mds.f.smithi193.stdout:Mar 24 00:22:13 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-f[72273]: 14: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f3365b13e81] 2024-03-24T00:22:14.219 INFO:journalctl@ceph.mds.f.smithi193.stdout:Mar 24 00:22:13 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-f[72273]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7f336452e1ca] 2024-03-24T00:22:14.219 INFO:journalctl@ceph.mds.f.smithi193.stdout:Mar 24 00:22:13 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-f[72273]: 16: clone() 2024-03-24T00:22:14.219 INFO:journalctl@ceph.mds.f.smithi193.stdout:Mar 24 00:22:13 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-f[72273]: 2024-03-24T00:22:13.972+0000 7f335b187700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: In function 'void Locker::set_xlocks_done(MutationImpl*, bool)' thread 7f335b187700 time 2024-03-24T00:22:13.971324+0000 2024-03-24T00:22:14.219 INFO:journalctl@ceph.mds.f.smithi193.stdout:Mar 24 00:22:13 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-f[72273]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: 626: FAILED ceph_assert(obj->is_auth()) 2024-03-24T00:22:14.219 INFO:journalctl@ceph.mds.f.smithi193.stdout:Mar 24 00:22:13 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-f[72273]: 2024-03-24T00:22:14.219 INFO:journalctl@ceph.mds.f.smithi193.stdout:Mar 24 00:22:13 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-f[72273]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-24T00:22:14.219 INFO:journalctl@ceph.mds.f.smithi193.stdout:Mar 24 00:22:13 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-f[72273]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x135) [0x7f33657b14a1] 2024-03-24T00:22:14.219 INFO:journalctl@ceph.mds.f.smithi193.stdout:Mar 24 00:22:13 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-f[72273]: 2: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7f33657b1667] 2024-03-24T00:22:14.219 INFO:journalctl@ceph.mds.f.smithi193.stdout:Mar 24 00:22:13 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-f[72273]: 3: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x55883bebb223] 2024-03-24T00:22:14.220 INFO:journalctl@ceph.mds.f.smithi193.stdout:Mar 24 00:22:13 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-f[72273]: 4: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x55883bd355d6] 2024-03-24T00:22:14.220 INFO:journalctl@ceph.mds.f.smithi193.stdout:Mar 24 00:22:13 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-f[72273]: 5: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x55883bd35da1] 2024-03-24T00:22:14.220 INFO:journalctl@ceph.mds.f.smithi193.stdout:Mar 24 00:22:13 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-f[72273]: 6: (Server::handle_client_rename(boost::intrusive_ptr const&)+0x259d) [0x55883bd7297d] 2024-03-24T00:22:14.220 INFO:journalctl@ceph.mds.f.smithi193.stdout:Mar 24 00:22:13 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-f[72273]: 7: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x55883bd88233] 2024-03-24T00:22:14.220 INFO:journalctl@ceph.mds.f.smithi193.stdout:Mar 24 00:22:13 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-f[72273]: 8: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x55883bd8a197] 2024-03-24T00:22:14.220 INFO:journalctl@ceph.mds.f.smithi193.stdout:Mar 24 00:22:13 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-f[72273]: 9: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x55883bc8911b] 2024-03-24T00:22:14.220 INFO:journalctl@ceph.mds.f.smithi193.stdout:Mar 24 00:22:13 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-f[72273]: 10: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x55883bc896bc] 2024-03-24T00:22:14.220 INFO:journalctl@ceph.mds.f.smithi193.stdout:Mar 24 00:22:13 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-f[72273]: 11: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x55883bc731ff] 2024-03-24T00:22:14.220 INFO:journalctl@ceph.mds.f.smithi193.stdout:Mar 24 00:22:13 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-f[72273]: 12: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7f3365a43488] 2024-03-24T00:22:14.220 INFO:journalctl@ceph.mds.f.smithi193.stdout:Mar 24 00:22:13 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-f[72273]: 13: (DispatchQueue::entry()+0x51f) [0x7f3365a4060f] 2024-03-24T00:22:14.220 INFO:journalctl@ceph.mds.f.smithi193.stdout:Mar 24 00:22:13 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-f[72273]: 14: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f3365b13e81] 2024-03-24T00:22:14.220 INFO:journalctl@ceph.mds.f.smithi193.stdout:Mar 24 00:22:13 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-f[72273]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7f336452e1ca] 2024-03-24T00:22:14.220 INFO:journalctl@ceph.mds.f.smithi193.stdout:Mar 24 00:22:13 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-f[72273]: 16: clone() 2024-03-24T00:22:14.220 INFO:journalctl@ceph.mds.f.smithi193.stdout:Mar 24 00:22:13 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-f[72273]: 2024-03-24T00:22:14.220 INFO:journalctl@ceph.mds.f.smithi193.stdout:Mar 24 00:22:13 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-f[72273]: *** Caught signal (Aborted) ** 2024-03-24T00:22:14.220 INFO:journalctl@ceph.mds.f.smithi193.stdout:Mar 24 00:22:13 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-f[72273]: in thread 7f335b187700 thread_name:ms_dispatch 2024-03-24T00:22:14.221 INFO:journalctl@ceph.mds.f.smithi193.stdout:Mar 24 00:22:13 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-f[72273]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-24T00:22:14.221 INFO:journalctl@ceph.mds.f.smithi193.stdout:Mar 24 00:22:13 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-f[72273]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f3364538d20] 2024-03-24T00:22:14.221 INFO:journalctl@ceph.mds.f.smithi193.stdout:Mar 24 00:22:13 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-f[72273]: 2: gsignal() 2024-03-24T00:22:14.221 INFO:journalctl@ceph.mds.f.smithi193.stdout:Mar 24 00:22:13 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-f[72273]: 3: abort() 2024-03-24T00:22:14.221 INFO:journalctl@ceph.mds.f.smithi193.stdout:Mar 24 00:22:13 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-f[72273]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x18f) [0x7f33657b14fb] 2024-03-24T00:22:14.221 INFO:journalctl@ceph.mds.f.smithi193.stdout:Mar 24 00:22:13 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-f[72273]: 5: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7f33657b1667] 2024-03-24T00:22:14.221 INFO:journalctl@ceph.mds.f.smithi193.stdout:Mar 24 00:22:13 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-f[72273]: 6: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x55883bebb223] 2024-03-24T00:22:14.221 INFO:journalctl@ceph.mds.f.smithi193.stdout:Mar 24 00:22:13 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-f[72273]: 7: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x55883bd355d6] 2024-03-24T00:22:14.221 INFO:journalctl@ceph.mds.f.smithi193.stdout:Mar 24 00:22:13 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-f[72273]: 8: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x55883bd35da1] 2024-03-24T00:22:14.221 INFO:journalctl@ceph.mds.f.smithi193.stdout:Mar 24 00:22:13 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-f[72273]: 9: (Server::handle_client_rename(boost::intrusive_ptr const&)+0x259d) [0x55883bd7297d] 2024-03-24T00:22:14.221 INFO:journalctl@ceph.mds.f.smithi193.stdout:Mar 24 00:22:13 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-f[72273]: 10: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x55883bd88233] 2024-03-24T00:22:14.221 INFO:journalctl@ceph.mds.f.smithi193.stdout:Mar 24 00:22:13 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-f[72273]: 11: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x55883bd8a197] 2024-03-24T00:22:14.222 INFO:journalctl@ceph.mds.f.smithi193.stdout:Mar 24 00:22:13 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-f[72273]: 12: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x55883bc8911b] 2024-03-24T00:22:14.222 INFO:journalctl@ceph.mds.f.smithi193.stdout:Mar 24 00:22:13 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-f[72273]: 13: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x55883bc896bc] 2024-03-24T00:22:14.222 INFO:journalctl@ceph.mds.f.smithi193.stdout:Mar 24 00:22:13 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-f[72273]: 14: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x55883bc731ff] 2024-03-24T00:22:14.222 INFO:journalctl@ceph.mds.f.smithi193.stdout:Mar 24 00:22:13 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-f[72273]: 15: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7f3365a43488] 2024-03-24T00:22:14.222 INFO:journalctl@ceph.mds.f.smithi193.stdout:Mar 24 00:22:13 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-f[72273]: 16: (DispatchQueue::entry()+0x51f) [0x7f3365a4060f] 2024-03-24T00:22:14.222 INFO:journalctl@ceph.mds.f.smithi193.stdout:Mar 24 00:22:13 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-f[72273]: 17: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f3365b13e81] 2024-03-24T00:22:14.222 INFO:journalctl@ceph.mds.f.smithi193.stdout:Mar 24 00:22:13 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-f[72273]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7f336452e1ca] 2024-03-24T00:22:14.222 INFO:journalctl@ceph.mds.f.smithi193.stdout:Mar 24 00:22:13 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-f[72273]: 19: clone() 2024-03-24T00:22:14.222 INFO:journalctl@ceph.mds.f.smithi193.stdout:Mar 24 00:22:13 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-f[72273]: 2024-03-24T00:22:13.974+0000 7f335b187700 -1 *** Caught signal (Aborted) ** 2024-03-24T00:22:14.222 INFO:journalctl@ceph.mds.f.smithi193.stdout:Mar 24 00:22:13 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-f[72273]: in thread 7f335b187700 thread_name:ms_dispatch 2024-03-24T00:22:14.222 INFO:journalctl@ceph.mds.f.smithi193.stdout:Mar 24 00:22:13 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-f[72273]: 2024-03-24T00:22:14.223 INFO:journalctl@ceph.mds.f.smithi193.stdout:Mar 24 00:22:13 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-f[72273]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-24T00:22:14.223 INFO:journalctl@ceph.mds.f.smithi193.stdout:Mar 24 00:22:13 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-f[72273]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f3364538d20] 2024-03-24T00:22:14.223 INFO:journalctl@ceph.mds.f.smithi193.stdout:Mar 24 00:22:13 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-f[72273]: 2: gsignal() 2024-03-24T00:22:14.223 INFO:journalctl@ceph.mds.f.smithi193.stdout:Mar 24 00:22:13 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-f[72273]: 3: abort() 2024-03-24T00:22:14.223 INFO:journalctl@ceph.mds.f.smithi193.stdout:Mar 24 00:22:13 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-f[72273]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x18f) [0x7f33657b14fb] 2024-03-24T00:22:14.223 INFO:journalctl@ceph.mds.f.smithi193.stdout:Mar 24 00:22:13 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-f[72273]: 5: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7f33657b1667] 2024-03-24T00:22:14.223 INFO:journalctl@ceph.mds.f.smithi193.stdout:Mar 24 00:22:13 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-f[72273]: 6: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x55883bebb223] 2024-03-24T00:22:14.223 INFO:journalctl@ceph.mds.f.smithi193.stdout:Mar 24 00:22:13 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-f[72273]: 7: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x55883bd355d6] 2024-03-24T00:22:14.223 INFO:journalctl@ceph.mds.f.smithi193.stdout:Mar 24 00:22:13 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-f[72273]: 8: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x55883bd35da1] 2024-03-24T00:22:14.223 INFO:journalctl@ceph.mds.f.smithi193.stdout:Mar 24 00:22:13 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-f[72273]: 9: (Server::handle_client_rename(boost::intrusive_ptr const&)+0x259d) [0x55883bd7297d] 2024-03-24T00:22:14.223 INFO:journalctl@ceph.mds.f.smithi193.stdout:Mar 24 00:22:13 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-f[72273]: 10: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x55883bd88233] 2024-03-24T00:22:14.223 INFO:journalctl@ceph.mds.f.smithi193.stdout:Mar 24 00:22:13 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-f[72273]: 11: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x55883bd8a197] 2024-03-24T00:22:14.223 INFO:journalctl@ceph.mds.f.smithi193.stdout:Mar 24 00:22:13 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-f[72273]: 12: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x55883bc8911b] 2024-03-24T00:22:14.224 INFO:journalctl@ceph.mds.f.smithi193.stdout:Mar 24 00:22:13 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-f[72273]: 13: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x55883bc896bc] 2024-03-24T00:22:14.224 INFO:journalctl@ceph.mds.f.smithi193.stdout:Mar 24 00:22:13 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-f[72273]: 14: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x55883bc731ff] 2024-03-24T00:22:14.224 INFO:journalctl@ceph.mds.f.smithi193.stdout:Mar 24 00:22:13 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-f[72273]: 15: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7f3365a43488] 2024-03-24T00:22:14.224 INFO:journalctl@ceph.mds.f.smithi193.stdout:Mar 24 00:22:13 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-f[72273]: 16: (DispatchQueue::entry()+0x51f) [0x7f3365a4060f] 2024-03-24T00:22:14.224 INFO:journalctl@ceph.mds.f.smithi193.stdout:Mar 24 00:22:13 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-f[72273]: 17: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f3365b13e81] 2024-03-24T00:22:14.224 INFO:journalctl@ceph.mds.f.smithi193.stdout:Mar 24 00:22:13 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-f[72273]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7f336452e1ca] 2024-03-24T00:22:14.224 INFO:journalctl@ceph.mds.f.smithi193.stdout:Mar 24 00:22:13 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-f[72273]: 19: clone() 2024-03-24T00:22:14.224 INFO:journalctl@ceph.mds.f.smithi193.stdout:Mar 24 00:22:13 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-f[72273]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-03-24T00:22:14.224 INFO:journalctl@ceph.mds.f.smithi193.stdout:Mar 24 00:22:13 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-f[72273]: 2024-03-24T00:22:14.224 INFO:journalctl@ceph.mds.f.smithi193.stdout:Mar 24 00:22:14 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-f[72273]: -1> 2024-03-24T00:22:13.972+0000 7f335b187700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: In function 'void Locker::set_xlocks_done(MutationImpl*, bool)' thread 7f335b187700 time 2024-03-24T00:22:13.971324+0000 2024-03-24T00:22:14.224 INFO:journalctl@ceph.mds.f.smithi193.stdout:Mar 24 00:22:14 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-f[72273]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: 626: FAILED ceph_assert(obj->is_auth()) 2024-03-24T00:22:14.224 INFO:journalctl@ceph.mds.f.smithi193.stdout:Mar 24 00:22:14 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-f[72273]: 2024-03-24T00:22:14.224 INFO:journalctl@ceph.mds.f.smithi193.stdout:Mar 24 00:22:14 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-f[72273]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-24T00:22:14.224 INFO:journalctl@ceph.mds.f.smithi193.stdout:Mar 24 00:22:14 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-f[72273]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x135) [0x7f33657b14a1] 2024-03-24T00:22:14.225 INFO:journalctl@ceph.mds.f.smithi193.stdout:Mar 24 00:22:14 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-f[72273]: 2: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7f33657b1667] 2024-03-24T00:22:14.225 INFO:journalctl@ceph.mds.f.smithi193.stdout:Mar 24 00:22:14 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-f[72273]: 3: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x55883bebb223] 2024-03-24T00:22:14.225 INFO:journalctl@ceph.mds.f.smithi193.stdout:Mar 24 00:22:14 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-f[72273]: 4: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x55883bd355d6] 2024-03-24T00:22:14.225 INFO:journalctl@ceph.mds.f.smithi193.stdout:Mar 24 00:22:14 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-f[72273]: 5: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x55883bd35da1] 2024-03-24T00:22:14.225 INFO:journalctl@ceph.mds.f.smithi193.stdout:Mar 24 00:22:14 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-f[72273]: 6: (Server::handle_client_rename(boost::intrusive_ptr const&)+0x259d) [0x55883bd7297d] 2024-03-24T00:22:14.225 INFO:journalctl@ceph.mds.f.smithi193.stdout:Mar 24 00:22:14 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-f[72273]: 7: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x55883bd88233] 2024-03-24T00:22:14.225 INFO:journalctl@ceph.mds.f.smithi193.stdout:Mar 24 00:22:14 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-f[72273]: 8: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x55883bd8a197] 2024-03-24T00:22:14.225 INFO:journalctl@ceph.mds.f.smithi193.stdout:Mar 24 00:22:14 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-f[72273]: 9: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x55883bc8911b] 2024-03-24T00:22:14.225 INFO:journalctl@ceph.mds.f.smithi193.stdout:Mar 24 00:22:14 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-f[72273]: 10: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x55883bc896bc] 2024-03-24T00:22:14.225 INFO:journalctl@ceph.mds.f.smithi193.stdout:Mar 24 00:22:14 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-f[72273]: 11: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x55883bc731ff] 2024-03-24T00:22:14.225 INFO:journalctl@ceph.mds.f.smithi193.stdout:Mar 24 00:22:14 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-f[72273]: 12: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7f3365a43488] 2024-03-24T00:22:14.225 INFO:journalctl@ceph.mds.f.smithi193.stdout:Mar 24 00:22:14 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-f[72273]: 13: (DispatchQueue::entry()+0x51f) [0x7f3365a4060f] 2024-03-24T00:22:14.225 INFO:journalctl@ceph.mds.f.smithi193.stdout:Mar 24 00:22:14 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-f[72273]: 14: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f3365b13e81] 2024-03-24T00:22:14.225 INFO:journalctl@ceph.mds.f.smithi193.stdout:Mar 24 00:22:14 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-f[72273]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7f336452e1ca] 2024-03-24T00:22:14.226 INFO:journalctl@ceph.mds.f.smithi193.stdout:Mar 24 00:22:14 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-f[72273]: 16: clone() 2024-03-24T00:22:14.226 INFO:journalctl@ceph.mds.f.smithi193.stdout:Mar 24 00:22:14 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-f[72273]: 2024-03-24T00:22:14.226 INFO:journalctl@ceph.mds.f.smithi193.stdout:Mar 24 00:22:14 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-f[72273]: 0> 2024-03-24T00:22:13.974+0000 7f335b187700 -1 *** Caught signal (Aborted) ** 2024-03-24T00:22:14.226 INFO:journalctl@ceph.mds.f.smithi193.stdout:Mar 24 00:22:14 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-f[72273]: in thread 7f335b187700 thread_name:ms_dispatch 2024-03-24T00:22:14.226 INFO:journalctl@ceph.mds.f.smithi193.stdout:Mar 24 00:22:14 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-f[72273]: 2024-03-24T00:22:14.226 INFO:journalctl@ceph.mds.f.smithi193.stdout:Mar 24 00:22:14 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-f[72273]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-24T00:22:14.226 INFO:journalctl@ceph.mds.f.smithi193.stdout:Mar 24 00:22:14 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-f[72273]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f3364538d20] 2024-03-24T00:22:14.226 INFO:journalctl@ceph.mds.f.smithi193.stdout:Mar 24 00:22:14 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-f[72273]: 2: gsignal() 2024-03-24T00:22:14.226 INFO:journalctl@ceph.mds.f.smithi193.stdout:Mar 24 00:22:14 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-f[72273]: 3: abort() 2024-03-24T00:22:14.226 INFO:journalctl@ceph.mds.f.smithi193.stdout:Mar 24 00:22:14 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-f[72273]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x18f) [0x7f33657b14fb] 2024-03-24T00:22:14.226 INFO:journalctl@ceph.mds.f.smithi193.stdout:Mar 24 00:22:14 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-f[72273]: 5: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7f33657b1667] 2024-03-24T00:22:14.226 INFO:journalctl@ceph.mds.f.smithi193.stdout:Mar 24 00:22:14 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-f[72273]: 6: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x55883bebb223] 2024-03-24T00:22:14.226 INFO:journalctl@ceph.mds.f.smithi193.stdout:Mar 24 00:22:14 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-f[72273]: 7: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x55883bd355d6] 2024-03-24T00:22:14.227 INFO:journalctl@ceph.mds.f.smithi193.stdout:Mar 24 00:22:14 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-f[72273]: 8: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x55883bd35da1] 2024-03-24T00:22:14.227 INFO:journalctl@ceph.mds.f.smithi193.stdout:Mar 24 00:22:14 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-f[72273]: 9: (Server::handle_client_rename(boost::intrusive_ptr const&)+0x259d) [0x55883bd7297d] 2024-03-24T00:22:14.227 INFO:journalctl@ceph.mds.f.smithi193.stdout:Mar 24 00:22:14 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-f[72273]: 10: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x55883bd88233] 2024-03-24T00:22:14.227 INFO:journalctl@ceph.mds.f.smithi193.stdout:Mar 24 00:22:14 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-f[72273]: 11: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x55883bd8a197] 2024-03-24T00:22:14.227 INFO:journalctl@ceph.mds.f.smithi193.stdout:Mar 24 00:22:14 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-f[72273]: 12: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x55883bc8911b] 2024-03-24T00:22:14.227 INFO:journalctl@ceph.mds.f.smithi193.stdout:Mar 24 00:22:14 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-f[72273]: 13: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x55883bc896bc] 2024-03-24T00:22:14.227 INFO:journalctl@ceph.mds.f.smithi193.stdout:Mar 24 00:22:14 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-f[72273]: 14: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x55883bc731ff] 2024-03-24T00:22:14.227 INFO:journalctl@ceph.mds.f.smithi193.stdout:Mar 24 00:22:14 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-f[72273]: 15: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7f3365a43488] 2024-03-24T00:22:14.227 INFO:journalctl@ceph.mds.f.smithi193.stdout:Mar 24 00:22:14 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-f[72273]: 16: (DispatchQueue::entry()+0x51f) [0x7f3365a4060f] 2024-03-24T00:22:14.227 INFO:journalctl@ceph.mds.f.smithi193.stdout:Mar 24 00:22:14 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-f[72273]: 17: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f3365b13e81] 2024-03-24T00:22:14.227 INFO:journalctl@ceph.mds.f.smithi193.stdout:Mar 24 00:22:14 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-f[72273]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7f336452e1ca] 2024-03-24T00:22:14.227 INFO:journalctl@ceph.mds.f.smithi193.stdout:Mar 24 00:22:14 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-f[72273]: 19: clone() 2024-03-24T00:22:14.227 INFO:journalctl@ceph.mds.f.smithi193.stdout:Mar 24 00:22:14 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-f[72273]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-03-24T00:22:14.228 INFO:journalctl@ceph.mds.f.smithi193.stdout:Mar 24 00:22:14 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-f[72273]: 2024-03-24T00:22:14.228 INFO:journalctl@ceph.mds.f.smithi193.stdout:Mar 24 00:22:14 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-f[72273]: -5> 2024-03-24T00:22:13.972+0000 7f335b187700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: In function 'void Locker::set_xlocks_done(MutationImpl*, bool)' thread 7f335b187700 time 2024-03-24T00:22:13.971324+0000 2024-03-24T00:22:14.228 INFO:journalctl@ceph.mds.f.smithi193.stdout:Mar 24 00:22:14 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-f[72273]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: 626: FAILED ceph_assert(obj->is_auth()) 2024-03-24T00:22:14.228 INFO:journalctl@ceph.mds.f.smithi193.stdout:Mar 24 00:22:14 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-f[72273]: 2024-03-24T00:22:14.228 INFO:journalctl@ceph.mds.f.smithi193.stdout:Mar 24 00:22:14 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-f[72273]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-24T00:22:14.228 INFO:journalctl@ceph.mds.f.smithi193.stdout:Mar 24 00:22:14 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-f[72273]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x135) [0x7f33657b14a1] 2024-03-24T00:22:14.228 INFO:journalctl@ceph.mds.f.smithi193.stdout:Mar 24 00:22:14 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-f[72273]: 2: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7f33657b1667] 2024-03-24T00:22:14.228 INFO:journalctl@ceph.mds.f.smithi193.stdout:Mar 24 00:22:14 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-f[72273]: 3: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x55883bebb223] 2024-03-24T00:22:14.228 INFO:journalctl@ceph.mds.f.smithi193.stdout:Mar 24 00:22:14 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-f[72273]: 4: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x55883bd355d6] 2024-03-24T00:22:14.229 INFO:journalctl@ceph.mds.f.smithi193.stdout:Mar 24 00:22:14 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-f[72273]: 5: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x55883bd35da1] 2024-03-24T00:22:14.229 INFO:journalctl@ceph.mds.f.smithi193.stdout:Mar 24 00:22:14 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-f[72273]: 6: (Server::handle_client_rename(boost::intrusive_ptr const&)+0x259d) [0x55883bd7297d] 2024-03-24T00:22:14.229 INFO:journalctl@ceph.mds.f.smithi193.stdout:Mar 24 00:22:14 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-f[72273]: 7: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x55883bd88233] 2024-03-24T00:22:14.229 INFO:journalctl@ceph.mds.f.smithi193.stdout:Mar 24 00:22:14 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-f[72273]: 8: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x55883bd8a197] 2024-03-24T00:22:14.229 INFO:journalctl@ceph.mds.f.smithi193.stdout:Mar 24 00:22:14 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-f[72273]: 9: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x55883bc8911b] 2024-03-24T00:22:14.229 INFO:journalctl@ceph.mds.f.smithi193.stdout:Mar 24 00:22:14 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-f[72273]: 10: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x55883bc896bc] 2024-03-24T00:22:14.229 INFO:journalctl@ceph.mds.f.smithi193.stdout:Mar 24 00:22:14 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-f[72273]: 11: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x55883bc731ff] 2024-03-24T00:22:14.229 INFO:journalctl@ceph.mds.f.smithi193.stdout:Mar 24 00:22:14 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-f[72273]: 12: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7f3365a43488] 2024-03-24T00:22:14.229 INFO:journalctl@ceph.mds.f.smithi193.stdout:Mar 24 00:22:14 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-f[72273]: 13: (DispatchQueue::entry()+0x51f) [0x7f3365a4060f] 2024-03-24T00:22:14.229 INFO:journalctl@ceph.mds.f.smithi193.stdout:Mar 24 00:22:14 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-f[72273]: 14: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f3365b13e81] 2024-03-24T00:22:14.229 INFO:journalctl@ceph.mds.f.smithi193.stdout:Mar 24 00:22:14 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-f[72273]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7f336452e1ca] 2024-03-24T00:22:14.229 INFO:journalctl@ceph.mds.f.smithi193.stdout:Mar 24 00:22:14 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-f[72273]: 16: clone() 2024-03-24T00:22:14.229 INFO:journalctl@ceph.mds.f.smithi193.stdout:Mar 24 00:22:14 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-f[72273]: 2024-03-24T00:22:14.229 INFO:journalctl@ceph.mds.f.smithi193.stdout:Mar 24 00:22:14 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-f[72273]: -4> 2024-03-24T00:22:13.974+0000 7f335b187700 -1 *** Caught signal (Aborted) ** 2024-03-24T00:22:14.230 INFO:journalctl@ceph.mds.f.smithi193.stdout:Mar 24 00:22:14 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-f[72273]: in thread 7f335b187700 thread_name:ms_dispatch 2024-03-24T00:22:14.230 INFO:journalctl@ceph.mds.f.smithi193.stdout:Mar 24 00:22:14 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-f[72273]: 2024-03-24T00:22:14.230 INFO:journalctl@ceph.mds.f.smithi193.stdout:Mar 24 00:22:14 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-f[72273]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-24T00:22:14.230 INFO:journalctl@ceph.mds.f.smithi193.stdout:Mar 24 00:22:14 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-f[72273]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f3364538d20] 2024-03-24T00:22:14.230 INFO:journalctl@ceph.mds.f.smithi193.stdout:Mar 24 00:22:14 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-f[72273]: 2: gsignal() 2024-03-24T00:22:14.230 INFO:journalctl@ceph.mds.f.smithi193.stdout:Mar 24 00:22:14 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-f[72273]: 3: abort() 2024-03-24T00:22:14.230 INFO:journalctl@ceph.mds.f.smithi193.stdout:Mar 24 00:22:14 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-f[72273]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x18f) [0x7f33657b14fb] 2024-03-24T00:22:14.230 INFO:journalctl@ceph.mds.f.smithi193.stdout:Mar 24 00:22:14 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-f[72273]: 5: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7f33657b1667] 2024-03-24T00:22:14.230 INFO:journalctl@ceph.mds.f.smithi193.stdout:Mar 24 00:22:14 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-f[72273]: 6: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x55883bebb223] 2024-03-24T00:22:14.230 INFO:journalctl@ceph.mds.f.smithi193.stdout:Mar 24 00:22:14 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-f[72273]: 7: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x55883bd355d6] 2024-03-24T00:22:14.230 INFO:journalctl@ceph.mds.f.smithi193.stdout:Mar 24 00:22:14 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-f[72273]: 8: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x55883bd35da1] 2024-03-24T00:22:14.230 INFO:journalctl@ceph.mds.f.smithi193.stdout:Mar 24 00:22:14 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-f[72273]: 9: (Server::handle_client_rename(boost::intrusive_ptr const&)+0x259d) [0x55883bd7297d] 2024-03-24T00:22:14.230 INFO:journalctl@ceph.mds.f.smithi193.stdout:Mar 24 00:22:14 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-f[72273]: 10: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x55883bd88233] 2024-03-24T00:22:14.231 INFO:journalctl@ceph.mds.f.smithi193.stdout:Mar 24 00:22:14 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-f[72273]: 11: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x55883bd8a197] 2024-03-24T00:22:14.231 INFO:journalctl@ceph.mds.f.smithi193.stdout:Mar 24 00:22:14 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-f[72273]: 12: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x55883bc8911b] 2024-03-24T00:22:14.231 INFO:journalctl@ceph.mds.f.smithi193.stdout:Mar 24 00:22:14 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-f[72273]: 13: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x55883bc896bc] 2024-03-24T00:22:14.231 INFO:journalctl@ceph.mds.f.smithi193.stdout:Mar 24 00:22:14 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-f[72273]: 14: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x55883bc731ff] 2024-03-24T00:22:14.231 INFO:journalctl@ceph.mds.f.smithi193.stdout:Mar 24 00:22:14 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-f[72273]: 15: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7f3365a43488] 2024-03-24T00:22:14.231 INFO:journalctl@ceph.mds.f.smithi193.stdout:Mar 24 00:22:14 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-f[72273]: 16: (DispatchQueue::entry()+0x51f) [0x7f3365a4060f] 2024-03-24T00:22:14.231 INFO:journalctl@ceph.mds.f.smithi193.stdout:Mar 24 00:22:14 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-f[72273]: 17: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f3365b13e81] 2024-03-24T00:22:14.231 INFO:journalctl@ceph.mds.f.smithi193.stdout:Mar 24 00:22:14 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-f[72273]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7f336452e1ca] 2024-03-24T00:22:14.231 INFO:journalctl@ceph.mds.f.smithi193.stdout:Mar 24 00:22:14 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-f[72273]: 19: clone() 2024-03-24T00:22:14.231 INFO:journalctl@ceph.mds.f.smithi193.stdout:Mar 24 00:22:14 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-f[72273]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-03-24T00:22:14.231 INFO:journalctl@ceph.mds.f.smithi193.stdout:Mar 24 00:22:14 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-f[72273]: 2024-03-24T00:22:14.231 INFO:journalctl@ceph.mds.f.smithi193.stdout:Mar 24 00:22:14 smithi193 podman[72513]: 2024-03-24 00:22:14.050424196 +0000 UTC m=+0.020444572 container died cfc9767be3a75f4aa40e1a53c49ee30e1b9a6187d38281688c96f804dbb0f0a1 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-f, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240319, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_CLEAN=True) 2024-03-24T00:22:15.046 INFO:journalctl@ceph.mds.f.smithi193.stdout:Mar 24 00:22:14 smithi193 podman[72513]: 2024-03-24 00:22:14.789878623 +0000 UTC m=+0.759898975 container remove cfc9767be3a75f4aa40e1a53c49ee30e1b9a6187d38281688c96f804dbb0f0a1 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-f, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240319, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, org.label-schema.license=GPLv2, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-24T00:22:15.046 INFO:journalctl@ceph.mds.f.smithi193.stdout:Mar 24 00:22:14 smithi193 systemd[1]: ceph-d79a5420-e971-11ee-95cd-87774f69a715@mds.f.service: Main process exited, code=exited, status=134/n/a 2024-03-24T00:22:15.047 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:22:14 smithi193 ceph-mon[36878]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-03-24T00:22:15.047 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:22:14 smithi193 ceph-mon[36878]: Cluster is now healthy 2024-03-24T00:22:15.047 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:22:14 smithi193 ceph-mon[36878]: mds.? [v2:172.21.15.193:6834/2264220953,v1:172.21.15.193:6835/2264220953] up:active 2024-03-24T00:22:15.047 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:22:14 smithi193 ceph-mon[36878]: mds.? [v2:172.21.15.118:6838/3600700815,v1:172.21.15.118:6839/3600700815] up:active 2024-03-24T00:22:15.047 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:22:14 smithi193 ceph-mon[36878]: fsmap cephfs:5 3 up:standby-replay 4 up:standby 5 up:active 2024-03-24T00:22:15.047 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:22:14 smithi193 ceph-mon[36878]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-03-24T00:22:15.088 INFO:journalctl@ceph.mds.g.smithi118.stdout:Mar 24 00:22:14 smithi118 podman[94787]: 2024-03-24 00:22:14.863659358 +0000 UTC m=+0.854304163 container remove 0498d0005714887d6ee54e9ea72d8f7bd3fe80cfc028a92db30adaa4b85c2076 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-g, org.label-schema.build-date=20240319, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, ceph=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, GIT_CLEAN=True, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS) 2024-03-24T00:22:15.088 INFO:journalctl@ceph.mds.g.smithi118.stdout:Mar 24 00:22:14 smithi118 systemd[1]: ceph-d79a5420-e971-11ee-95cd-87774f69a715@mds.g.service: Main process exited, code=exited, status=134/n/a 2024-03-24T00:22:15.088 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:22:14 smithi118 ceph-mon[31535]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-03-24T00:22:15.088 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:22:14 smithi118 ceph-mon[31535]: Cluster is now healthy 2024-03-24T00:22:15.088 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:22:14 smithi118 ceph-mon[31535]: mds.? [v2:172.21.15.193:6834/2264220953,v1:172.21.15.193:6835/2264220953] up:active 2024-03-24T00:22:15.088 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:22:14 smithi118 ceph-mon[31535]: mds.? [v2:172.21.15.118:6838/3600700815,v1:172.21.15.118:6839/3600700815] up:active 2024-03-24T00:22:15.088 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:22:14 smithi118 ceph-mon[31535]: fsmap cephfs:5 3 up:standby-replay 4 up:standby 5 up:active 2024-03-24T00:22:15.089 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:22:14 smithi118 ceph-mon[31535]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-03-24T00:22:15.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:22:14 smithi189 ceph-mon[36187]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-03-24T00:22:15.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:22:14 smithi189 ceph-mon[36187]: Cluster is now healthy 2024-03-24T00:22:15.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:22:14 smithi189 ceph-mon[36187]: mds.? [v2:172.21.15.193:6834/2264220953,v1:172.21.15.193:6835/2264220953] up:active 2024-03-24T00:22:15.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:22:14 smithi189 ceph-mon[36187]: mds.? [v2:172.21.15.118:6838/3600700815,v1:172.21.15.118:6839/3600700815] up:active 2024-03-24T00:22:15.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:22:14 smithi189 ceph-mon[36187]: fsmap cephfs:5 3 up:standby-replay 4 up:standby 5 up:active 2024-03-24T00:22:15.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:22:14 smithi189 ceph-mon[36187]: fsmap cephfs:5 5 up:standby-replay 2 up:standby 5 up:active 2024-03-24T00:22:15.461 INFO:journalctl@ceph.mds.f.smithi193.stdout:Mar 24 00:22:15 smithi193 systemd[1]: ceph-d79a5420-e971-11ee-95cd-87774f69a715@mds.f.service: Failed with result 'exit-code'. 2024-03-24T00:22:15.600 INFO:journalctl@ceph.mds.g.smithi118.stdout:Mar 24 00:22:15 smithi118 systemd[1]: ceph-d79a5420-e971-11ee-95cd-87774f69a715@mds.g.service: Failed with result 'exit-code'. 2024-03-24T00:22:15.600 INFO:journalctl@ceph.mds.g.smithi118.stdout:Mar 24 00:22:15 smithi118 systemd[1]: ceph-d79a5420-e971-11ee-95cd-87774f69a715@mds.g.service: Consumed 1.431s CPU time. 2024-03-24T00:22:16.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:22:15 smithi118 ceph-mon[31535]: pgmap v611: 97 pgs: 97 active+clean; 138 MiB data, 797 MiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 319 KiB/s wr, 178 op/s 2024-03-24T00:22:16.204 INFO:journalctl@ceph.mds.b.smithi189.stdout:Mar 24 00:22:15 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-b[66975]: 2024-03-24T00:22:15.734+0000 7f4e07ab9700 -1 mds.pinger is_rank_lagging: rank=1 was never sent ping request. 2024-03-24T00:22:16.204 INFO:journalctl@ceph.mds.b.smithi189.stdout:Mar 24 00:22:15 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-b[66975]: 2024-03-24T00:22:15.734+0000 7f4e07ab9700 -1 mds.pinger is_rank_lagging: rank=3 was never sent ping request. 2024-03-24T00:22:16.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:22:15 smithi189 ceph-mon[36187]: pgmap v611: 97 pgs: 97 active+clean; 138 MiB data, 797 MiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 319 KiB/s wr, 178 op/s 2024-03-24T00:22:16.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:22:15 smithi193 ceph-mon[36878]: pgmap v611: 97 pgs: 97 active+clean; 138 MiB data, 797 MiB used, 1.0 TiB / 1.0 TiB avail; 1.8 MiB/s rd, 319 KiB/s wr, 178 op/s 2024-03-24T00:22:16.846 INFO:tasks.workunit.client.0.smithi118.stdout: 47 4722 53 3942 80 1769 160 2024-03-24T00:22:18.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:22:17 smithi118 ceph-mon[31535]: pgmap v612: 97 pgs: 97 active+clean; 138 MiB data, 803 MiB used, 1.0 TiB / 1.0 TiB avail; 5.3 MiB/s rd, 298 KiB/s wr, 382 op/s 2024-03-24T00:22:18.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:22:17 smithi189 ceph-mon[36187]: pgmap v612: 97 pgs: 97 active+clean; 138 MiB data, 803 MiB used, 1.0 TiB / 1.0 TiB avail; 5.3 MiB/s rd, 298 KiB/s wr, 382 op/s 2024-03-24T00:22:18.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:22:17 smithi193 ceph-mon[36878]: pgmap v612: 97 pgs: 97 active+clean; 138 MiB data, 803 MiB used, 1.0 TiB / 1.0 TiB avail; 5.3 MiB/s rd, 298 KiB/s wr, 382 op/s 2024-03-24T00:22:20.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:22:19 smithi118 ceph-mon[31535]: pgmap v613: 97 pgs: 97 active+clean; 144 MiB data, 806 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 MiB/s rd, 930 KiB/s wr, 466 op/s 2024-03-24T00:22:20.203 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:22:19 smithi189 ceph-mon[36187]: pgmap v613: 97 pgs: 97 active+clean; 144 MiB data, 806 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 MiB/s rd, 930 KiB/s wr, 466 op/s 2024-03-24T00:22:20.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:22:19 smithi193 ceph-mon[36878]: pgmap v613: 97 pgs: 97 active+clean; 144 MiB data, 806 MiB used, 1.0 TiB / 1.0 TiB avail; 6.2 MiB/s rd, 930 KiB/s wr, 466 op/s 2024-03-24T00:22:22.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:22:21 smithi118 ceph-mon[31535]: pgmap v614: 97 pgs: 97 active+clean; 145 MiB data, 808 MiB used, 1.0 TiB / 1.0 TiB avail; 5.3 MiB/s rd, 907 KiB/s wr, 410 op/s 2024-03-24T00:22:22.203 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:22:21 smithi189 ceph-mon[36187]: pgmap v614: 97 pgs: 97 active+clean; 145 MiB data, 808 MiB used, 1.0 TiB / 1.0 TiB avail; 5.3 MiB/s rd, 907 KiB/s wr, 410 op/s 2024-03-24T00:22:22.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:22:21 smithi193 ceph-mon[36878]: pgmap v614: 97 pgs: 97 active+clean; 145 MiB data, 808 MiB used, 1.0 TiB / 1.0 TiB avail; 5.3 MiB/s rd, 907 KiB/s wr, 410 op/s 2024-03-24T00:22:24.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:22:23 smithi118 ceph-mon[31535]: pgmap v615: 97 pgs: 97 active+clean; 149 MiB data, 790 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 MiB/s rd, 1.3 MiB/s wr, 445 op/s 2024-03-24T00:22:24.203 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:22:23 smithi189 ceph-mon[36187]: pgmap v615: 97 pgs: 97 active+clean; 149 MiB data, 790 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 MiB/s rd, 1.3 MiB/s wr, 445 op/s 2024-03-24T00:22:24.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:22:23 smithi193 ceph-mon[36878]: pgmap v615: 97 pgs: 97 active+clean; 149 MiB data, 790 MiB used, 1.0 TiB / 1.0 TiB avail; 5.4 MiB/s rd, 1.3 MiB/s wr, 445 op/s 2024-03-24T00:22:25.711 INFO:journalctl@ceph.mds.f.smithi193.stdout:Mar 24 00:22:25 smithi193 systemd[1]: ceph-d79a5420-e971-11ee-95cd-87774f69a715@mds.f.service: Scheduled restart job, restart counter is at 6. 2024-03-24T00:22:25.711 INFO:journalctl@ceph.mds.f.smithi193.stdout:Mar 24 00:22:25 smithi193 systemd[1]: Stopped Ceph mds.f for d79a5420-e971-11ee-95cd-87774f69a715. 2024-03-24T00:22:25.711 INFO:journalctl@ceph.mds.f.smithi193.stdout:Mar 24 00:22:25 smithi193 systemd[1]: ceph-d79a5420-e971-11ee-95cd-87774f69a715@mds.f.service: Start request repeated too quickly. 2024-03-24T00:22:25.711 INFO:journalctl@ceph.mds.f.smithi193.stdout:Mar 24 00:22:25 smithi193 systemd[1]: ceph-d79a5420-e971-11ee-95cd-87774f69a715@mds.f.service: Failed with result 'exit-code'. 2024-03-24T00:22:25.711 INFO:journalctl@ceph.mds.f.smithi193.stdout:Mar 24 00:22:25 smithi193 systemd[1]: Failed to start Ceph mds.f for d79a5420-e971-11ee-95cd-87774f69a715. 2024-03-24T00:22:25.838 INFO:journalctl@ceph.mds.g.smithi118.stdout:Mar 24 00:22:25 smithi118 systemd[1]: ceph-d79a5420-e971-11ee-95cd-87774f69a715@mds.g.service: Scheduled restart job, restart counter is at 4. 2024-03-24T00:22:25.838 INFO:journalctl@ceph.mds.g.smithi118.stdout:Mar 24 00:22:25 smithi118 systemd[1]: Stopped Ceph mds.g for d79a5420-e971-11ee-95cd-87774f69a715. 2024-03-24T00:22:25.838 INFO:journalctl@ceph.mds.g.smithi118.stdout:Mar 24 00:22:25 smithi118 systemd[1]: ceph-d79a5420-e971-11ee-95cd-87774f69a715@mds.g.service: Consumed 1.431s CPU time. 2024-03-24T00:22:25.838 INFO:journalctl@ceph.mds.g.smithi118.stdout:Mar 24 00:22:25 smithi118 systemd[1]: Starting Ceph mds.g for d79a5420-e971-11ee-95cd-87774f69a715... 2024-03-24T00:22:26.058 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:22:26.058+0000 7fc2ac600640 1 -- 172.21.15.118:0/1031829767 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc2a4076060 msgr2=0x7fc2a4076480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:22:26.059 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:22:26.058+0000 7fc2ac600640 1 --2- 172.21.15.118:0/1031829767 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc2a4076060 0x7fc2a4076480 secure :-1 s=READY pgs=436 cs=0 l=1 rev1=1 crypto rx=0x7fc298009900 tx=0x7fc29802bc40 comp rx=0 tx=0).stop 2024-03-24T00:22:26.059 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:22:26.058+0000 7fc2ac600640 1 -- 172.21.15.118:0/1031829767 shutdown_connections 2024-03-24T00:22:26.059 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:22:26.058+0000 7fc2ac600640 1 --2- 172.21.15.118:0/1031829767 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fc2a4077c40 0x7fc2a40784e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:22:26.059 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:22:26.058+0000 7fc2ac600640 1 --2- 172.21.15.118:0/1031829767 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fc2a40772a0 0x7fc2a4077700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:22:26.059 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:22:26.058+0000 7fc2ac600640 1 --2- 172.21.15.118:0/1031829767 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc2a4076060 0x7fc2a4076480 unknown :-1 s=CLOSED pgs=436 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:22:26.059 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:22:26.058+0000 7fc2ac600640 1 -- 172.21.15.118:0/1031829767 >> 172.21.15.118:0/1031829767 conn(0x7fc2a40fee90 msgr2=0x7fc2a41012b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:22:26.059 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:22:26.058+0000 7fc2ac600640 1 -- 172.21.15.118:0/1031829767 shutdown_connections 2024-03-24T00:22:26.059 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:22:26.058+0000 7fc2ac600640 1 -- 172.21.15.118:0/1031829767 wait complete. 2024-03-24T00:22:26.062 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:22:26.059+0000 7fc2ac600640 1 Processor -- start 2024-03-24T00:22:26.062 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:22:26.059+0000 7fc2ac600640 1 -- start start 2024-03-24T00:22:26.062 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:22:26.060+0000 7fc2ac600640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fc2a4076060 0x7fc2a419dee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:22:26.062 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:22:26.060+0000 7fc2ac600640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc2a40772a0 0x7fc2a419e420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:22:26.062 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:22:26.060+0000 7fc2ac600640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fc2a4077c40 0x7fc2a41a5490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:22:26.062 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:22:26.060+0000 7fc2ac600640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc2a41139d0 con 0x7fc2a4077c40 2024-03-24T00:22:26.062 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:22:26.060+0000 7fc2ac600640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc2a4113850 con 0x7fc2a40772a0 2024-03-24T00:22:26.062 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:22:26.060+0000 7fc2ac600640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc2a4113b50 con 0x7fc2a4076060 2024-03-24T00:22:26.062 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:22:26.060+0000 7fc2a9200640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc2a40772a0 0x7fc2a419e420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:22:26.062 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:22:26.060+0000 7fc2a9c00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fc2a4076060 0x7fc2a419dee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:22:26.062 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:22:26.060+0000 7fc2a9200640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc2a40772a0 0x7fc2a419e420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.118:50150/0 (socket says 172.21.15.118:50150) 2024-03-24T00:22:26.063 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:22:26.060+0000 7fc2a9c00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fc2a4076060 0x7fc2a419dee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.118:45862/0 (socket says 172.21.15.118:45862) 2024-03-24T00:22:26.063 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:22:26.060+0000 7fc2a9200640 1 -- 172.21.15.118:0/2122589690 learned_addr learned my addr 172.21.15.118:0/2122589690 (peer_addr_for_me v2:172.21.15.118:0/0) 2024-03-24T00:22:26.063 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:22:26.060+0000 7fc2aa600640 1 --2- 172.21.15.118:0/2122589690 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fc2a4077c40 0x7fc2a41a5490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:22:26.063 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:22:26.060+0000 7fc2a9200640 1 -- 172.21.15.118:0/2122589690 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fc2a4076060 msgr2=0x7fc2a419dee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:22:26.063 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:22:26.060+0000 7fc2a9200640 1 --2- 172.21.15.118:0/2122589690 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fc2a4076060 0x7fc2a419dee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:22:26.063 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:22:26.060+0000 7fc2a9200640 1 -- 172.21.15.118:0/2122589690 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fc2a4077c40 msgr2=0x7fc2a41a5490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:22:26.063 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:22:26.060+0000 7fc2a9200640 1 --2- 172.21.15.118:0/2122589690 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fc2a4077c40 0x7fc2a41a5490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:22:26.063 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:22:26.060+0000 7fc2a9200640 1 -- 172.21.15.118:0/2122589690 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc2a41a5b00 con 0x7fc2a40772a0 2024-03-24T00:22:26.063 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:22:26.060+0000 7fc2a9c00640 1 --2- 172.21.15.118:0/2122589690 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fc2a4076060 0x7fc2a419dee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T00:22:26.063 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:22:26.061+0000 7fc2a9200640 1 --2- 172.21.15.118:0/2122589690 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc2a40772a0 0x7fc2a419e420 secure :-1 s=READY pgs=437 cs=0 l=1 rev1=1 crypto rx=0x7fc29000e9f0 tx=0x7fc29000edf0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:22:26.064 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:22:26.061+0000 7fc2a2000640 1 -- 172.21.15.118:0/2122589690 <== mon.2 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc29000cd80 con 0x7fc2a40772a0 2024-03-24T00:22:26.064 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:22:26.061+0000 7fc2a2000640 1 -- 172.21.15.118:0/2122589690 <== mon.2 v2:172.21.15.189:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fc290005500 con 0x7fc2a40772a0 2024-03-24T00:22:26.064 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:22:26.061+0000 7fc2ac600640 1 -- 172.21.15.118:0/2122589690 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc2a41a5df0 con 0x7fc2a40772a0 2024-03-24T00:22:26.064 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:22:26.061+0000 7fc2a2000640 1 -- 172.21.15.118:0/2122589690 <== mon.2 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc290010690 con 0x7fc2a40772a0 2024-03-24T00:22:26.064 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:22:26.061+0000 7fc2ac600640 1 -- 172.21.15.118:0/2122589690 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc2a41a62b0 con 0x7fc2a40772a0 2024-03-24T00:22:26.066 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:22:26.064+0000 7fc2af364740 1 Processor -- start 2024-03-24T00:22:26.066 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:22:26.064+0000 7fc2af364740 1 -- start start 2024-03-24T00:22:26.066 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:22:26.065+0000 7fc2a2000640 1 -- 172.21.15.118:0/2122589690 <== mon.2 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 20) v1 ==== 137291+0+0 (secure 0 0 0) 0x7fc2900046c0 con 0x7fc2a40772a0 2024-03-24T00:22:26.066 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:22:26.065+0000 7fc2af364740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55c9d18fd1e0 0x55c9d18fd600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:22:26.066 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:22:26.065+0000 7fc2af364740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55c9d18fdbd0 0x55c9d19026d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:22:26.066 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:22:26.065+0000 7fc2af364740 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55c9d1902d40 0x55c9d19051d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:22:26.066 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:22:26.065+0000 7fc2af364740 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c9d18c03b0 con 0x55c9d1902d40 2024-03-24T00:22:26.066 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:22:26.065+0000 7fc2af364740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c9d1862830 con 0x55c9d18fd1e0 2024-03-24T00:22:26.066 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:22:26.065+0000 7fc2af364740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c9d180b5d0 con 0x55c9d18fdbd0 2024-03-24T00:22:26.067 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:22:26.065+0000 7fc2a9c00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55c9d18fdbd0 0x55c9d19026d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:22:26.067 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:22:26.065+0000 7fc2aa600640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55c9d18fd1e0 0x55c9d18fd600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:22:26.067 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:22:26.065+0000 7fc2a9c00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55c9d18fdbd0 0x55c9d19026d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.118:45864/0 (socket says 172.21.15.118:45864) 2024-03-24T00:22:26.067 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:22:26.065+0000 7fc2a9c00640 1 -- 172.21.15.118:0/678577839 learned_addr learned my addr 172.21.15.118:0/678577839 (peer_addr_for_me v2:172.21.15.118:0/0) 2024-03-24T00:22:26.067 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:22:26.065+0000 7fc2aa600640 1 --2- 172.21.15.118:0/678577839 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55c9d18fd1e0 0x55c9d18fd600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.118:50164/0 (socket says 172.21.15.118:50164) 2024-03-24T00:22:26.067 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:22:26.065+0000 7fc2a9200640 1 --2- 172.21.15.118:0/678577839 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55c9d1902d40 0x55c9d19051d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:22:26.067 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:22:26.065+0000 7fc2aa600640 1 -- 172.21.15.118:0/678577839 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55c9d18fdbd0 msgr2=0x55c9d19026d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:22:26.067 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:22:26.065+0000 7fc2aa600640 1 --2- 172.21.15.118:0/678577839 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55c9d18fdbd0 0x55c9d19026d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:22:26.067 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:22:26.065+0000 7fc2aa600640 1 -- 172.21.15.118:0/678577839 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55c9d1902d40 msgr2=0x55c9d19051d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:22:26.067 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:22:26.065+0000 7fc2aa600640 1 --2- 172.21.15.118:0/678577839 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55c9d1902d40 0x55c9d19051d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:22:26.067 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:22:26.065+0000 7fc2aa600640 1 -- 172.21.15.118:0/678577839 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55c9d18232c0 con 0x55c9d18fd1e0 2024-03-24T00:22:26.068 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:22:26.065+0000 7fc2a9c00640 1 --2- 172.21.15.118:0/678577839 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55c9d18fdbd0 0x55c9d19026d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T00:22:26.068 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:22:26.065+0000 7fc2a9200640 1 --2- 172.21.15.118:0/678577839 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55c9d1902d40 0x55c9d19051d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T00:22:26.068 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:22:26.066+0000 7fc2aa600640 1 --2- 172.21.15.118:0/678577839 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55c9d18fd1e0 0x55c9d18fd600 secure :-1 s=READY pgs=438 cs=0 l=1 rev1=1 crypto rx=0x7fc29c009f30 tx=0x7fc29c010800 comp rx=0 tx=0).ready entity=mon.2 client_cookie=49c5a8dbf88cb5dd server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:22:26.068 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:22:26.066+0000 7fc296a00640 1 -- 172.21.15.118:0/678577839 <== mon.2 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc29c016070 con 0x55c9d18fd1e0 2024-03-24T00:22:26.068 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:22:26.066+0000 7fc296a00640 1 -- 172.21.15.118:0/678577839 <== mon.2 v2:172.21.15.189:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fc29c004d20 con 0x55c9d18fd1e0 2024-03-24T00:22:26.068 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:22:26.066+0000 7fc296a00640 1 -- 172.21.15.118:0/678577839 <== mon.2 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc29c012870 con 0x55c9d18fd1e0 2024-03-24T00:22:26.068 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:22:26.066+0000 7fc2a2000640 1 --2- 172.21.15.118:0/2122589690 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7fc28c0a2920 0x7fc28c0a4de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:22:26.073 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:22:26.067+0000 7fc2a2000640 1 -- 172.21.15.118:0/2122589690 <== mon.2 v2:172.21.15.189:3300/0 5 ==== osd_map(106..106 src has 1..106) v4 ==== 9653+0+0 (secure 0 0 0) 0x7fc2900cf360 con 0x7fc2a40772a0 2024-03-24T00:22:26.073 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:22:26.067+0000 7fc2af364740 1 -- 172.21.15.118:0/678577839 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55c9d18fd1e0 msgr2=0x55c9d18fd600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:22:26.073 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:22:26.067+0000 7fc2af364740 1 --2- 172.21.15.118:0/678577839 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55c9d18fd1e0 0x55c9d18fd600 secure :-1 s=READY pgs=438 cs=0 l=1 rev1=1 crypto rx=0x7fc29c009f30 tx=0x7fc29c010800 comp rx=0 tx=0).stop 2024-03-24T00:22:26.073 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:22:26.067+0000 7fc2a9c00640 1 --2- 172.21.15.118:0/2122589690 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7fc28c0a2920 0x7fc28c0a4de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:22:26.073 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:22:26.067+0000 7fc2af364740 1 -- 172.21.15.118:0/678577839 shutdown_connections 2024-03-24T00:22:26.073 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:22:26.067+0000 7fc2af364740 1 --2- 172.21.15.118:0/678577839 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55c9d1902d40 0x55c9d19051d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:22:26.074 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:22:26.067+0000 7fc2af364740 1 --2- 172.21.15.118:0/678577839 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55c9d18fdbd0 0x55c9d19026d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:22:26.074 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:22:26.067+0000 7fc2af364740 1 --2- 172.21.15.118:0/678577839 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55c9d18fd1e0 0x55c9d18fd600 unknown :-1 s=CLOSED pgs=438 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:22:26.074 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:22:26.067+0000 7fc2af364740 1 -- 172.21.15.118:0/678577839 >> 172.21.15.118:0/678577839 conn(0x55c9d18f8750 msgr2=0x55c9d18f8b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:22:26.074 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:22:26.067+0000 7fc2af364740 1 -- 172.21.15.118:0/678577839 shutdown_connections 2024-03-24T00:22:26.074 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:22:26.067+0000 7fc2af364740 1 -- 172.21.15.118:0/678577839 wait complete. 2024-03-24T00:22:26.074 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:22:26.067+0000 7fc2af364740 1 -- start start 2024-03-24T00:22:26.074 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:22:26.067+0000 7fc2a9c00640 1 --2- 172.21.15.118:0/2122589690 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7fc28c0a2920 0x7fc28c0a4de0 secure :-1 s=READY pgs=376 cs=0 l=1 rev1=1 crypto rx=0x7fc29802efd0 tx=0x7fc29802cf90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:22:26.074 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:22:26.068+0000 7fc2af364740 1 Processor -- start 2024-03-24T00:22:26.074 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:22:26.068+0000 7fc2af364740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55c9d18fd1e0 0x55c9d19073c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:22:26.074 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:22:26.068+0000 7fc2af364740 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55c9d18fdbd0 0x55c9d1907900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:22:26.074 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:22:26.068+0000 7fc2af364740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55c9d1902d40 0x55c9d190e8d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:22:26.074 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:22:26.068+0000 7fc2af364740 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c9d181ea20 con 0x55c9d18fdbd0 2024-03-24T00:22:26.075 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:22:26.068+0000 7fc2af364740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c9d185e8d0 con 0x55c9d1902d40 2024-03-24T00:22:26.075 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:22:26.068+0000 7fc2af364740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x55c9d1905cf0 con 0x55c9d18fd1e0 2024-03-24T00:22:26.075 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:22:26.068+0000 7fc2af364740 10 client.0 _ll_register_callbacks cb 0x55c9d185b1c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T00:22:26.075 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:22:26.068+0000 7fc2a9c00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55c9d18fd1e0 0x55c9d19073c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:22:26.075 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:22:26.068+0000 7fc2a9c00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55c9d18fd1e0 0x55c9d19073c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.118:45876/0 (socket says 172.21.15.118:45876) 2024-03-24T00:22:26.075 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:22:26.068+0000 7fc2a9c00640 1 -- 172.21.15.118:0/1919802658 learned_addr learned my addr 172.21.15.118:0/1919802658 (peer_addr_for_me v2:172.21.15.118:0/0) 2024-03-24T00:22:26.075 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:22:26.068+0000 7fc2aa600640 1 --2- 172.21.15.118:0/1919802658 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55c9d1902d40 0x55c9d190e8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:22:26.075 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:22:26.068+0000 7fc2a9200640 1 --2- 172.21.15.118:0/1919802658 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55c9d18fdbd0 0x55c9d1907900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:22:26.075 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:22:26.069+0000 7fc2a9c00640 1 -- 172.21.15.118:0/1919802658 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55c9d1902d40 msgr2=0x55c9d190e8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:22:26.075 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:22:26.069+0000 7fc2a9c00640 1 --2- 172.21.15.118:0/1919802658 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55c9d1902d40 0x55c9d190e8d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:22:26.075 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:22:26.069+0000 7fc2a9c00640 1 -- 172.21.15.118:0/1919802658 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55c9d18fdbd0 msgr2=0x55c9d1907900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:22:26.075 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:22:26.069+0000 7fc2a9c00640 1 --2- 172.21.15.118:0/1919802658 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55c9d18fdbd0 0x55c9d1907900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:22:26.076 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:22:26.069+0000 7fc2a9c00640 1 -- 172.21.15.118:0/1919802658 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55c9d1577210 con 0x55c9d18fd1e0 2024-03-24T00:22:26.076 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:22:26.069+0000 7fc2a9c00640 1 --2- 172.21.15.118:0/1919802658 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55c9d18fd1e0 0x55c9d19073c0 secure :-1 s=READY pgs=440 cs=0 l=1 rev1=1 crypto rx=0x7fc29802c250 tx=0x7fc29802bc40 comp rx=0 tx=0).ready entity=mon.1 client_cookie=d92162cad937def server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:22:26.076 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:22:26.069+0000 7fc28b400640 10 client.0 ms_handle_connect on v2:172.21.15.193:3300/0 2024-03-24T00:22:26.076 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:22:26.069+0000 7fc28b400640 1 -- 172.21.15.118:0/1919802658 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc2980053c0 con 0x55c9d18fd1e0 2024-03-24T00:22:26.076 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:22:26.069+0000 7fc28b400640 1 -- 172.21.15.118:0/1919802658 <== mon.1 v2:172.21.15.193:3300/0 2 ==== osd_map(106..106 src has 1..106) v4 ==== 9653+0+0 (secure 0 0 0) 0x7fc298005560 con 0x55c9d18fd1e0 2024-03-24T00:22:26.076 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:22:26.069+0000 7fc2af364740 1 -- 172.21.15.118:0/1919802658 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fc29c004d20 con 0x55c9d18fd1e0 2024-03-24T00:22:26.076 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:22:26.069+0000 7fc28b400640 1 -- 172.21.15.118:0/1919802658 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_get_version_reply(handle=1 version=109) v2 ==== 24+0+0 (secure 0 0 0) 0x7fc2980035f0 con 0x55c9d18fd1e0 2024-03-24T00:22:26.076 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:22:26.069+0000 7fc2af364740 10 client.26869 fetch_fsmap learned FSMap version 109 2024-03-24T00:22:26.076 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:22:26.069+0000 7fc2af364740 1 -- 172.21.15.118:0/1919802658 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,fsmap=109,monmap=4+}) v3 -- 0x55c9d1643620 con 0x55c9d18fd1e0 2024-03-24T00:22:26.076 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:22:26.070+0000 7fc28b400640 1 -- 172.21.15.118:0/1919802658 <== mon.1 v2:172.21.15.193:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fc29c004d20 con 0x55c9d18fd1e0 2024-03-24T00:22:26.076 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:22:26.070+0000 7fc28b400640 1 -- 172.21.15.118:0/1919802658 <== mon.1 v2:172.21.15.193:3300/0 5 ==== fsmap(e 109) v1 ==== 7938+0+0 (secure 0 0 0) 0x7fc298050070 con 0x55c9d18fd1e0 2024-03-24T00:22:26.076 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:22:26.070+0000 7fc2af364740 10 client.26869 fetch_fsmap finished waiting for FSMap version 109 2024-03-24T00:22:26.076 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:22:26.070+0000 7fc2af364740 10 client.26869 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-24T00:22:26.077 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:22:26.070+0000 7fc2af364740 20 client.26869 populate_metadata read hostname 'smithi118' 2024-03-24T00:22:26.077 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:22:26.070+0000 7fc2af364740 1 --2- 172.21.15.118:0/1919802658 >> [v2:172.21.15.189:6838/4074484168,v1:172.21.15.189:6839/4074484168] conn(0x55c9d19131b0 0x55c9d19155c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:22:26.077 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:22:26.070+0000 7fc2af364740 4 client.26869 mds_command: new command op to 24569 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-24T00:22:26.077 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:22:26.070+0000 7fc2af364740 1 -- 172.21.15.118:0/1919802658 --> [v2:172.21.15.189:6838/4074484168,v1:172.21.15.189:6839/4074484168] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55c9d186fd70 con 0x55c9d19131b0 2024-03-24T00:22:26.077 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:22:26.070+0000 7fc2a9200640 1 --2- 172.21.15.118:0/1919802658 >> [v2:172.21.15.189:6838/4074484168,v1:172.21.15.189:6839/4074484168] conn(0x55c9d19131b0 0x55c9d19155c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:22:26.077 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:22:26.070+0000 7fc2a9200640 1 --2- 172.21.15.118:0/1919802658 >> [v2:172.21.15.189:6838/4074484168,v1:172.21.15.189:6839/4074484168] conn(0x55c9d19131b0 0x55c9d19155c0 crc :-1 s=READY pgs=259 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=b4cdfa19aa8966d server_cookie=4a7df4857c8a5219 in_seq=0 out_seq=0 2024-03-24T00:22:26.077 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:22:26.070+0000 7fc28b400640 10 client.26869 ms_handle_connect on v2:172.21.15.189:6838/4074484168 2024-03-24T00:22:26.077 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:22:26.071+0000 7fc28b400640 1 -- 172.21.15.118:0/1919802658 <== mds.0 v2:172.21.15.189:6838/4074484168 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x55c9d186fd70 con 0x55c9d19131b0 2024-03-24T00:22:26.077 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:22:26.071+0000 7fc28b400640 10 client.26869 handle_command_reply: tid=0 2024-03-24T00:22:26.077 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:22:26.072+0000 7fc2af364740 1 client.26869 shutdown 2024-03-24T00:22:26.077 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:22:26.072+0000 7fc2af364740 1 -- 172.21.15.118:0/1919802658 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55c9d18fd1e0 msgr2=0x55c9d19073c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:22:26.077 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:22:26.072+0000 7fc2af364740 1 --2- 172.21.15.118:0/1919802658 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55c9d18fd1e0 0x55c9d19073c0 secure :-1 s=READY pgs=440 cs=0 l=1 rev1=1 crypto rx=0x7fc29802c250 tx=0x7fc29802bc40 comp rx=0 tx=0).stop 2024-03-24T00:22:26.077 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:22:26.072+0000 7fc2af364740 1 -- 172.21.15.118:0/1919802658 shutdown_connections 2024-03-24T00:22:26.077 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:22:26.072+0000 7fc2af364740 1 --2- 172.21.15.118:0/1919802658 >> [v2:172.21.15.189:6838/4074484168,v1:172.21.15.189:6839/4074484168] conn(0x55c9d19131b0 0x55c9d19155c0 crc :-1 s=READY pgs=259 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:22:26.078 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:22:26.072+0000 7fc2af364740 1 --2- 172.21.15.118:0/1919802658 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55c9d1902d40 0x55c9d190e8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:22:26.078 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:22:26.072+0000 7fc2af364740 1 --2- 172.21.15.118:0/1919802658 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55c9d18fdbd0 0x55c9d1907900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:22:26.078 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:22:26.072+0000 7fc28b400640 0 client.26869 ms_handle_reset on v2:172.21.15.189:6838/4074484168 2024-03-24T00:22:26.078 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:22:26.072+0000 7fc2af364740 1 --2- 172.21.15.118:0/1919802658 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55c9d18fd1e0 0x55c9d19073c0 unknown :-1 s=CLOSED pgs=440 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:22:26.078 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:22:26.072+0000 7fc2af364740 1 -- 172.21.15.118:0/1919802658 >> 172.21.15.118:0/1919802658 conn(0x55c9d18f8750 msgr2=0x55c9d18f7950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:22:26.078 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:22:26.072+0000 7fc2af364740 1 -- 172.21.15.118:0/1919802658 shutdown_connections 2024-03-24T00:22:26.078 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:22:26.072+0000 7fc2af364740 1 -- 172.21.15.118:0/1919802658 wait complete. 2024-03-24T00:22:26.078 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:22:26.072+0000 7fc2af364740 20 client.26869 trim_cache size 0 max 16384 2024-03-24T00:22:26.089 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:22:26.087+0000 7fc2af364740 1 Processor -- start 2024-03-24T00:22:26.089 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:22:26.087+0000 7fc2af364740 1 -- start start 2024-03-24T00:22:26.090 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:22:26.087+0000 7fc2af364740 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55c9d18fd1e0 0x55c9d1906fd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:22:26.090 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:22:26.087+0000 7fc2af364740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55c9d18fdbd0 0x55c9d190e860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:22:26.090 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:22:26.087+0000 7fc2af364740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55c9d1902d40 0x55c9d1907510 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:22:26.090 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:22:26.087+0000 7fc2af364740 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc2a41afc60 con 0x55c9d18fd1e0 2024-03-24T00:22:26.090 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:22:26.087+0000 7fc2af364740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc2a41b15e0 con 0x55c9d18fdbd0 2024-03-24T00:22:26.090 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:22:26.087+0000 7fc2af364740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc2a41b02c0 con 0x55c9d1902d40 2024-03-24T00:22:26.090 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:22:26.088+0000 7fc2aa600640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55c9d1902d40 0x55c9d1907510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:22:26.090 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:22:26.088+0000 7fc2aa600640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55c9d1902d40 0x55c9d1907510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.118:45878/0 (socket says 172.21.15.118:45878) 2024-03-24T00:22:26.090 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:22:26.088+0000 7fc2aa600640 1 -- 172.21.15.118:0/843784904 learned_addr learned my addr 172.21.15.118:0/843784904 (peer_addr_for_me v2:172.21.15.118:0/0) 2024-03-24T00:22:26.090 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:22:26.088+0000 7fc2a9200640 1 --2- 172.21.15.118:0/843784904 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55c9d18fdbd0 0x55c9d190e860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:22:26.090 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:22:26.088+0000 7fc2a9c00640 1 --2- 172.21.15.118:0/843784904 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55c9d18fd1e0 0x55c9d1906fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:22:26.090 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:22:26.088+0000 7fc2aa600640 1 -- 172.21.15.118:0/843784904 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55c9d18fdbd0 msgr2=0x55c9d190e860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:22:26.091 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:22:26.088+0000 7fc2aa600640 1 --2- 172.21.15.118:0/843784904 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55c9d18fdbd0 0x55c9d190e860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:22:26.091 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:22:26.088+0000 7fc2aa600640 1 -- 172.21.15.118:0/843784904 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55c9d18fd1e0 msgr2=0x55c9d1906fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:22:26.091 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:22:26.088+0000 7fc2aa600640 1 --2- 172.21.15.118:0/843784904 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55c9d18fd1e0 0x55c9d1906fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:22:26.091 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:22:26.088+0000 7fc2aa600640 1 -- 172.21.15.118:0/843784904 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55c9d18fd5c0 con 0x55c9d1902d40 2024-03-24T00:22:26.091 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:22:26.088+0000 7fc2aa600640 1 --2- 172.21.15.118:0/843784904 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55c9d1902d40 0x55c9d1907510 secure :-1 s=READY pgs=441 cs=0 l=1 rev1=1 crypto rx=0x55c9d18fe430 tx=0x7fc29c004a00 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c6fec459e4222ca0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:22:26.091 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:22:26.088+0000 7fc296000640 1 -- 172.21.15.118:0/843784904 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc29c016050 con 0x55c9d1902d40 2024-03-24T00:22:26.091 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:22:26.089+0000 7fc296000640 1 -- 172.21.15.118:0/843784904 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fc29c012b30 con 0x55c9d1902d40 2024-03-24T00:22:26.091 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:22:26.089+0000 7fc296000640 1 -- 172.21.15.118:0/843784904 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc29c024cc0 con 0x55c9d1902d40 2024-03-24T00:22:26.091 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:22:26.089+0000 7fc2af364740 1 -- 172.21.15.118:0/843784904 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55c9d1902d40 msgr2=0x55c9d1907510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:22:26.091 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:22:26.089+0000 7fc2af364740 1 --2- 172.21.15.118:0/843784904 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55c9d1902d40 0x55c9d1907510 secure :-1 s=READY pgs=441 cs=0 l=1 rev1=1 crypto rx=0x55c9d18fe430 tx=0x7fc29c004a00 comp rx=0 tx=0).stop 2024-03-24T00:22:26.091 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:22:26.089+0000 7fc2af364740 1 -- 172.21.15.118:0/843784904 shutdown_connections 2024-03-24T00:22:26.091 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:22:26.089+0000 7fc2af364740 1 --2- 172.21.15.118:0/843784904 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55c9d1902d40 0x55c9d1907510 unknown :-1 s=CLOSED pgs=441 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:22:26.092 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:22:26.089+0000 7fc2af364740 1 --2- 172.21.15.118:0/843784904 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55c9d18fdbd0 0x55c9d190e860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:22:26.092 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:22:26.089+0000 7fc2af364740 1 --2- 172.21.15.118:0/843784904 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55c9d18fd1e0 0x55c9d1906fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:22:26.092 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:22:26.089+0000 7fc2af364740 1 -- 172.21.15.118:0/843784904 >> 172.21.15.118:0/843784904 conn(0x55c9d18f8750 msgr2=0x55c9d18fef30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:22:26.092 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:22:26.089+0000 7fc2af364740 1 -- 172.21.15.118:0/843784904 shutdown_connections 2024-03-24T00:22:26.092 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:22:26.089+0000 7fc2af364740 1 -- 172.21.15.118:0/843784904 wait complete. 2024-03-24T00:22:26.092 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:22:26.090+0000 7fc2af364740 1 -- start start 2024-03-24T00:22:26.092 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:22:26.090+0000 7fc2af364740 1 Processor -- start 2024-03-24T00:22:26.095 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:22:26.090+0000 7fc2af364740 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55c9d18fd1e0 0x55c9d190c0f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:22:26.095 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:22:26.090+0000 7fc2af364740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55c9d18fdbd0 0x55c9d190c630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:22:26.095 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:22:26.090+0000 7fc2af364740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55c9d1902d40 0x55c9d190d640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:22:26.095 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:22:26.090+0000 7fc2af364740 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc2a41b0f80 con 0x55c9d18fd1e0 2024-03-24T00:22:26.095 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:22:26.090+0000 7fc2af364740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc2a41b0920 con 0x55c9d1902d40 2024-03-24T00:22:26.095 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:22:26.090+0000 7fc2af364740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc2a41ade20 con 0x55c9d18fdbd0 2024-03-24T00:22:26.095 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:22:26.090+0000 7fc2af364740 10 client.0 _ll_register_callbacks cb 0x7fc298005b70 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T00:22:26.096 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:22:26.090+0000 7fc2a9200640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55c9d18fdbd0 0x55c9d190c630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:22:26.096 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:22:26.091+0000 7fc2a9200640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55c9d18fdbd0 0x55c9d190c630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.118:45894/0 (socket says 172.21.15.118:45894) 2024-03-24T00:22:26.096 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:22:26.091+0000 7fc2a9200640 1 -- 172.21.15.118:0/1538872295 learned_addr learned my addr 172.21.15.118:0/1538872295 (peer_addr_for_me v2:172.21.15.118:0/0) 2024-03-24T00:22:26.096 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:22:26.091+0000 7fc2aa600640 1 --2- 172.21.15.118:0/1538872295 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55c9d1902d40 0x55c9d190d640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:22:26.096 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:22:26.091+0000 7fc2a9c00640 1 --2- 172.21.15.118:0/1538872295 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55c9d18fd1e0 0x55c9d190c0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:22:26.096 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:22:26.091+0000 7fc2a9200640 1 -- 172.21.15.118:0/1538872295 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55c9d1902d40 msgr2=0x55c9d190d640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:22:26.096 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:22:26.091+0000 7fc2a9200640 1 --2- 172.21.15.118:0/1538872295 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55c9d1902d40 0x55c9d190d640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:22:26.096 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:22:26.091+0000 7fc2a9200640 1 -- 172.21.15.118:0/1538872295 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55c9d18fd1e0 msgr2=0x55c9d190c0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:22:26.096 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:22:26.091+0000 7fc2a9200640 1 --2- 172.21.15.118:0/1538872295 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55c9d18fd1e0 0x55c9d190c0f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:22:26.096 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:22:26.091+0000 7fc2a9200640 1 -- 172.21.15.118:0/1538872295 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55c9d190dc30 con 0x55c9d18fdbd0 2024-03-24T00:22:26.096 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:22:26.091+0000 7fc2a9200640 1 --2- 172.21.15.118:0/1538872295 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55c9d18fdbd0 0x55c9d190c630 secure :-1 s=READY pgs=442 cs=0 l=1 rev1=1 crypto rx=0x7fc290004c00 tx=0x7fc290004a40 comp rx=0 tx=0).ready entity=mon.1 client_cookie=cca063bd3969ee04 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:22:26.096 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:22:26.091+0000 7fc28b400640 10 client.0 ms_handle_connect on v2:172.21.15.193:3300/0 2024-03-24T00:22:26.096 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:22:26.091+0000 7fc28b400640 1 -- 172.21.15.118:0/1538872295 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fc29009be70 con 0x55c9d18fdbd0 2024-03-24T00:22:26.097 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:22:26.091+0000 7fc28b400640 1 -- 172.21.15.118:0/1538872295 <== mon.1 v2:172.21.15.193:3300/0 2 ==== osd_map(106..106 src has 1..106) v4 ==== 9653+0+0 (secure 0 0 0) 0x7fc290099a70 con 0x55c9d18fdbd0 2024-03-24T00:22:26.097 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:22:26.091+0000 7fc2af364740 1 -- 172.21.15.118:0/1538872295 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fc29c012b30 con 0x55c9d18fdbd0 2024-03-24T00:22:26.097 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:22:26.092+0000 7fc28b400640 1 -- 172.21.15.118:0/1538872295 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_get_version_reply(handle=1 version=109) v2 ==== 24+0+0 (secure 0 0 0) 0x7fc29009b770 con 0x55c9d18fdbd0 2024-03-24T00:22:26.097 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:22:26.092+0000 7fc2af364740 10 client.26875 fetch_fsmap learned FSMap version 109 2024-03-24T00:22:26.097 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:22:26.092+0000 7fc2af364740 1 -- 172.21.15.118:0/1538872295 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,fsmap=109,monmap=4+}) v3 -- 0x55c9d196ed10 con 0x55c9d18fdbd0 2024-03-24T00:22:26.097 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:22:26.092+0000 7fc28b400640 1 -- 172.21.15.118:0/1538872295 <== mon.1 v2:172.21.15.193:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fc29c012b30 con 0x55c9d18fdbd0 2024-03-24T00:22:26.097 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:22:26.092+0000 7fc28b400640 1 -- 172.21.15.118:0/1538872295 <== mon.1 v2:172.21.15.193:3300/0 5 ==== fsmap(e 109) v1 ==== 7938+0+0 (secure 0 0 0) 0x7fc290097de0 con 0x55c9d18fdbd0 2024-03-24T00:22:26.097 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:22:26.092+0000 7fc2af364740 10 client.26875 fetch_fsmap finished waiting for FSMap version 109 2024-03-24T00:22:26.097 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:22:26.092+0000 7fc2af364740 10 client.26875 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-24T00:22:26.097 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:22:26.092+0000 7fc2af364740 20 client.26875 populate_metadata read hostname 'smithi118' 2024-03-24T00:22:26.097 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:22:26.092+0000 7fc2af364740 1 --2- 172.21.15.118:0/1538872295 >> [v2:172.21.15.189:6838/4074484168,v1:172.21.15.189:6839/4074484168] conn(0x55c9d196f5a0 0x55c9d1971a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:22:26.097 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:22:26.092+0000 7fc2af364740 4 client.26875 mds_command: new command op to 24569 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-03-24T00:22:26.098 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:22:26.092+0000 7fc2af364740 1 -- 172.21.15.118:0/1538872295 --> [v2:172.21.15.189:6838/4074484168,v1:172.21.15.189:6839/4074484168] -- command(tid 0: {"prefix": "flush journal"}) v1 -- 0x55c9d186a0c0 con 0x55c9d196f5a0 2024-03-24T00:22:26.098 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:22:26.092+0000 7fc2a9c00640 1 --2- 172.21.15.118:0/1538872295 >> [v2:172.21.15.189:6838/4074484168,v1:172.21.15.189:6839/4074484168] conn(0x55c9d196f5a0 0x55c9d1971a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:22:26.098 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:22:26.093+0000 7fc2a9c00640 1 --2- 172.21.15.118:0/1538872295 >> [v2:172.21.15.189:6838/4074484168,v1:172.21.15.189:6839/4074484168] conn(0x55c9d196f5a0 0x55c9d1971a20 crc :-1 s=READY pgs=260 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=5d8c684863f4a503 server_cookie=5a352ce9c95c1cc5 in_seq=0 out_seq=0 2024-03-24T00:22:26.098 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:22:26.093+0000 7fc28b400640 10 client.26875 ms_handle_connect on v2:172.21.15.189:6838/4074484168 2024-03-24T00:22:26.168 INFO:journalctl@ceph.mds.g.smithi118.stdout:Mar 24 00:22:25 smithi118 podman[94876]: 2024-03-24 00:22:25.743459187 +0000 UTC m=+0.034493529 image pull c664afe56bb01db4f9afa9532c20598e201e664f2e9ba2be50db0d52674a26d2 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc 2024-03-24T00:22:26.168 INFO:journalctl@ceph.mds.g.smithi118.stdout:Mar 24 00:22:25 smithi118 podman[94876]: 2024-03-24 00:22:25.86555208 +0000 UTC m=+0.156586459 container create fdbf59c1210423ef0f7e07fe152cd76ea99cd862de938f70af5e35b932835521 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-g, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240319, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, GIT_CLEAN=True, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, ceph=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-24T00:22:26.168 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:22:25 smithi118 ceph-mon[31535]: pgmap v616: 97 pgs: 97 active+clean; 149 MiB data, 799 MiB used, 1.0 TiB / 1.0 TiB avail; 5.5 MiB/s rd, 1.2 MiB/s wr, 428 op/s 2024-03-24T00:22:26.203 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:22:25 smithi189 ceph-mon[36187]: pgmap v616: 97 pgs: 97 active+clean; 149 MiB data, 799 MiB used, 1.0 TiB / 1.0 TiB avail; 5.5 MiB/s rd, 1.2 MiB/s wr, 428 op/s 2024-03-24T00:22:26.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:22:25 smithi193 ceph-mon[36878]: pgmap v616: 97 pgs: 97 active+clean; 149 MiB data, 799 MiB used, 1.0 TiB / 1.0 TiB avail; 5.5 MiB/s rd, 1.2 MiB/s wr, 428 op/s 2024-03-24T00:22:26.447 INFO:journalctl@ceph.mds.g.smithi118.stdout:Mar 24 00:22:26 smithi118 podman[94876]: 2024-03-24 00:22:26.168617156 +0000 UTC m=+0.459651528 container init fdbf59c1210423ef0f7e07fe152cd76ea99cd862de938f70af5e35b932835521 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-g, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, ceph=True, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240319) 2024-03-24T00:22:26.447 INFO:journalctl@ceph.mds.g.smithi118.stdout:Mar 24 00:22:26 smithi118 podman[94876]: 2024-03-24 00:22:26.18098999 +0000 UTC m=+0.472024340 container start fdbf59c1210423ef0f7e07fe152cd76ea99cd862de938f70af5e35b932835521 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-g, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.vendor=CentOS, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240319, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, ceph=True, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-24T00:22:26.447 INFO:journalctl@ceph.mds.g.smithi118.stdout:Mar 24 00:22:26 smithi118 ceph-mds[94939]: -- 172.21.15.118:0/3955768562 <== mon.0 v2:172.21.15.118:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x5559693d3a40 con 0x5559693a6800 2024-03-24T00:22:26.447 INFO:journalctl@ceph.mds.g.smithi118.stdout:Mar 24 00:22:26 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-g[94891]: starting mds.g at 2024-03-24T00:22:26.448 INFO:journalctl@ceph.mds.g.smithi118.stdout:Mar 24 00:22:26 smithi118 bash[94876]: fdbf59c1210423ef0f7e07fe152cd76ea99cd862de938f70af5e35b932835521 2024-03-24T00:22:26.448 INFO:journalctl@ceph.mds.g.smithi118.stdout:Mar 24 00:22:26 smithi118 systemd[1]: Started Ceph mds.g for d79a5420-e971-11ee-95cd-87774f69a715. 2024-03-24T00:22:26.845 INFO:tasks.workunit.client.0.smithi118.stdout: 47 0 0 0 0 0 240 2024-03-24T00:22:27.197 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:22:26 smithi118 ceph-mon[31535]: daemon mds.g restarted 2024-03-24T00:22:27.197 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:22:26 smithi118 ceph-mon[31535]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-03-24T00:22:27.197 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:22:26 smithi118 ceph-mon[31535]: osdmap e107: 12 total, 12 up, 12 in 2024-03-24T00:22:27.197 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:22:26 smithi118 ceph-mon[31535]: mds.? [v2:172.21.15.118:6838/1710625324,v1:172.21.15.118:6839/1710625324] up:boot 2024-03-24T00:22:27.197 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:22:26 smithi118 ceph-mon[31535]: Standby daemon mds.h assigned to filesystem cephfs as rank 3 2024-03-24T00:22:27.197 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:22:26 smithi118 ceph-mon[31535]: fsmap cephfs:4/5 {0=b=up:active,1=f=up:active,2=c=up:active,4=e=up:active} 5 up:standby-replay 3 up:standby, 1 failed 2024-03-24T00:22:27.197 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:22:26 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2024-03-24T00:22:27.197 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:22:26 smithi118 ceph-mon[31535]: fsmap cephfs:5/5 4 up:standby-replay 3 up:standby 1 up:replay 4 up:active 2024-03-24T00:22:27.203 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:22:26 smithi189 ceph-mon[36187]: daemon mds.g restarted 2024-03-24T00:22:27.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:22:26 smithi189 ceph-mon[36187]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-03-24T00:22:27.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:22:26 smithi189 ceph-mon[36187]: osdmap e107: 12 total, 12 up, 12 in 2024-03-24T00:22:27.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:22:26 smithi189 ceph-mon[36187]: mds.? [v2:172.21.15.118:6838/1710625324,v1:172.21.15.118:6839/1710625324] up:boot 2024-03-24T00:22:27.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:22:26 smithi189 ceph-mon[36187]: Standby daemon mds.h assigned to filesystem cephfs as rank 3 2024-03-24T00:22:27.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:22:26 smithi189 ceph-mon[36187]: fsmap cephfs:4/5 {0=b=up:active,1=f=up:active,2=c=up:active,4=e=up:active} 5 up:standby-replay 3 up:standby, 1 failed 2024-03-24T00:22:27.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:22:26 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2024-03-24T00:22:27.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:22:26 smithi189 ceph-mon[36187]: fsmap cephfs:5/5 4 up:standby-replay 3 up:standby 1 up:replay 4 up:active 2024-03-24T00:22:27.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:22:26 smithi193 ceph-mon[36878]: daemon mds.g restarted 2024-03-24T00:22:27.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:22:26 smithi193 ceph-mon[36878]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-03-24T00:22:27.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:22:26 smithi193 ceph-mon[36878]: osdmap e107: 12 total, 12 up, 12 in 2024-03-24T00:22:27.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:22:26 smithi193 ceph-mon[36878]: mds.? [v2:172.21.15.118:6838/1710625324,v1:172.21.15.118:6839/1710625324] up:boot 2024-03-24T00:22:27.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:22:26 smithi193 ceph-mon[36878]: Standby daemon mds.h assigned to filesystem cephfs as rank 3 2024-03-24T00:22:27.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:22:26 smithi193 ceph-mon[36878]: fsmap cephfs:4/5 {0=b=up:active,1=f=up:active,2=c=up:active,4=e=up:active} 5 up:standby-replay 3 up:standby, 1 failed 2024-03-24T00:22:27.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:22:26 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2024-03-24T00:22:27.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:22:26 smithi193 ceph-mon[36878]: fsmap cephfs:5/5 4 up:standby-replay 3 up:standby 1 up:replay 4 up:active 2024-03-24T00:22:28.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:22:27 smithi118 ceph-mon[31535]: pgmap v618: 97 pgs: 97 active+clean; 149 MiB data, 799 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 1.3 MiB/s wr, 292 op/s 2024-03-24T00:22:28.203 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:22:27 smithi189 ceph-mon[36187]: pgmap v618: 97 pgs: 97 active+clean; 149 MiB data, 799 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 1.3 MiB/s wr, 292 op/s 2024-03-24T00:22:28.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:22:27 smithi193 ceph-mon[36878]: pgmap v618: 97 pgs: 97 active+clean; 149 MiB data, 799 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 MiB/s rd, 1.3 MiB/s wr, 292 op/s 2024-03-24T00:22:29.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:22:28 smithi118 ceph-mon[31535]: mds.? [v2:172.21.15.189:6836/75274228,v1:172.21.15.189:6837/75274228] up:resolve 2024-03-24T00:22:29.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:22:28 smithi118 ceph-mon[31535]: Replacing daemon mds.f as rank 1 with standby daemon mds.l 2024-03-24T00:22:29.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:22:28 smithi118 ceph-mon[31535]: MDS daemon mds.f is removed because it is dead or otherwise unavailable. 2024-03-24T00:22:29.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:22:28 smithi118 ceph-mon[31535]: fsmap cephfs:5/5 4 up:standby-replay 3 up:standby 1 up:resolve 4 up:active 2024-03-24T00:22:29.197 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:22:28 smithi118 ceph-mon[31535]: osdmap e108: 12 total, 12 up, 12 in 2024-03-24T00:22:29.197 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:22:28 smithi118 ceph-mon[31535]: fsmap cephfs:5/5 3 up:standby-replay 3 up:standby 1 up:replay 1 up:resolve 3 up:active 2024-03-24T00:22:29.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:22:28 smithi189 ceph-mon[36187]: mds.? [v2:172.21.15.189:6836/75274228,v1:172.21.15.189:6837/75274228] up:resolve 2024-03-24T00:22:29.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:22:28 smithi189 ceph-mon[36187]: Replacing daemon mds.f as rank 1 with standby daemon mds.l 2024-03-24T00:22:29.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:22:28 smithi189 ceph-mon[36187]: MDS daemon mds.f is removed because it is dead or otherwise unavailable. 2024-03-24T00:22:29.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:22:28 smithi189 ceph-mon[36187]: fsmap cephfs:5/5 4 up:standby-replay 3 up:standby 1 up:resolve 4 up:active 2024-03-24T00:22:29.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:22:28 smithi189 ceph-mon[36187]: osdmap e108: 12 total, 12 up, 12 in 2024-03-24T00:22:29.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:22:28 smithi189 ceph-mon[36187]: fsmap cephfs:5/5 3 up:standby-replay 3 up:standby 1 up:replay 1 up:resolve 3 up:active 2024-03-24T00:22:29.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:22:28 smithi193 ceph-mon[36878]: mds.? [v2:172.21.15.189:6836/75274228,v1:172.21.15.189:6837/75274228] up:resolve 2024-03-24T00:22:29.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:22:28 smithi193 ceph-mon[36878]: Replacing daemon mds.f as rank 1 with standby daemon mds.l 2024-03-24T00:22:29.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:22:28 smithi193 ceph-mon[36878]: MDS daemon mds.f is removed because it is dead or otherwise unavailable. 2024-03-24T00:22:29.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:22:28 smithi193 ceph-mon[36878]: fsmap cephfs:5/5 4 up:standby-replay 3 up:standby 1 up:resolve 4 up:active 2024-03-24T00:22:29.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:22:28 smithi193 ceph-mon[36878]: osdmap e108: 12 total, 12 up, 12 in 2024-03-24T00:22:29.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:22:28 smithi193 ceph-mon[36878]: fsmap cephfs:5/5 3 up:standby-replay 3 up:standby 1 up:replay 1 up:resolve 3 up:active 2024-03-24T00:22:30.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:22:29 smithi118 ceph-mon[31535]: pgmap v620: 97 pgs: 97 active+clean; 149 MiB data, 799 MiB used, 1.0 TiB / 1.0 TiB avail; 277 KiB/s rd, 665 KiB/s wr, 181 op/s 2024-03-24T00:22:30.203 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:22:29 smithi189 ceph-mon[36187]: pgmap v620: 97 pgs: 97 active+clean; 149 MiB data, 799 MiB used, 1.0 TiB / 1.0 TiB avail; 277 KiB/s rd, 665 KiB/s wr, 181 op/s 2024-03-24T00:22:30.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:22:29 smithi193 ceph-mon[36878]: pgmap v620: 97 pgs: 97 active+clean; 149 MiB data, 799 MiB used, 1.0 TiB / 1.0 TiB avail; 277 KiB/s rd, 665 KiB/s wr, 181 op/s 2024-03-24T00:22:31.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:22:30 smithi189 ceph-mon[36187]: mds.? [v2:172.21.15.193:6832/2424695757,v1:172.21.15.193:6833/2424695757] up:resolve 2024-03-24T00:22:31.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:22:30 smithi189 ceph-mon[36187]: fsmap cephfs:5/5 3 up:standby-replay 3 up:standby 2 up:resolve 3 up:active 2024-03-24T00:22:31.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:22:30 smithi193 ceph-mon[36878]: mds.? [v2:172.21.15.193:6832/2424695757,v1:172.21.15.193:6833/2424695757] up:resolve 2024-03-24T00:22:31.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:22:30 smithi193 ceph-mon[36878]: fsmap cephfs:5/5 3 up:standby-replay 3 up:standby 2 up:resolve 3 up:active 2024-03-24T00:22:31.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:22:30 smithi118 ceph-mon[31535]: mds.? [v2:172.21.15.193:6832/2424695757,v1:172.21.15.193:6833/2424695757] up:resolve 2024-03-24T00:22:31.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:22:30 smithi118 ceph-mon[31535]: fsmap cephfs:5/5 3 up:standby-replay 3 up:standby 2 up:resolve 3 up:active 2024-03-24T00:22:31.986 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:22:31.985+0000 7fc28b400640 1 -- 172.21.15.118:0/1538872295 <== mds.0 v2:172.21.15.189:6838/4074484168 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+44 (crc 0 0 0) 0x55c9d186a0c0 con 0x55c9d196f5a0 2024-03-24T00:22:31.986 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:22:31.985+0000 7fc28b400640 10 client.26875 handle_command_reply: tid=0 2024-03-24T00:22:31.986 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:22:31.985+0000 7fc2af364740 1 client.26875 shutdown 2024-03-24T00:22:31.988 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:22:31.986+0000 7fc2af364740 1 -- 172.21.15.118:0/1538872295 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55c9d18fdbd0 msgr2=0x55c9d190c630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:22:31.988 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:22:31.986+0000 7fc2af364740 1 --2- 172.21.15.118:0/1538872295 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55c9d18fdbd0 0x55c9d190c630 secure :-1 s=READY pgs=442 cs=0 l=1 rev1=1 crypto rx=0x7fc290004c00 tx=0x7fc290004a40 comp rx=0 tx=0).stop 2024-03-24T00:22:31.988 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:22:31.986+0000 7fc2af364740 1 -- 172.21.15.118:0/1538872295 shutdown_connections 2024-03-24T00:22:31.988 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:22:31.986+0000 7fc2af364740 1 --2- 172.21.15.118:0/1538872295 >> [v2:172.21.15.189:6838/4074484168,v1:172.21.15.189:6839/4074484168] conn(0x55c9d196f5a0 0x55c9d1971a20 crc :-1 s=READY pgs=260 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:22:31.988 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:22:31.986+0000 7fc2af364740 1 --2- 172.21.15.118:0/1538872295 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55c9d1902d40 0x55c9d190d640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:22:31.988 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:22:31.986+0000 7fc2af364740 1 --2- 172.21.15.118:0/1538872295 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55c9d18fdbd0 0x55c9d190c630 unknown :-1 s=CLOSED pgs=442 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:22:31.988 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:22:31.986+0000 7fc2af364740 1 --2- 172.21.15.118:0/1538872295 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55c9d18fd1e0 0x55c9d190c0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:22:31.989 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:22:31.986+0000 7fc2af364740 1 -- 172.21.15.118:0/1538872295 >> 172.21.15.118:0/1538872295 conn(0x55c9d18f8750 msgr2=0x55c9d1906fd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:22:31.989 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:22:31.986+0000 7fc28b400640 0 client.26875 ms_handle_reset on v2:172.21.15.189:6838/4074484168 2024-03-24T00:22:31.989 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:22:31.987+0000 7fc2af364740 1 -- 172.21.15.118:0/1538872295 shutdown_connections 2024-03-24T00:22:31.989 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:22:31.987+0000 7fc2af364740 1 -- 172.21.15.118:0/1538872295 wait complete. 2024-03-24T00:22:31.989 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:22:31.987+0000 7fc2af364740 20 client.26875 trim_cache size 0 max 16384 2024-03-24T00:22:31.989 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stdout:{ 2024-03-24T00:22:31.990 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stdout: "message": "", 2024-03-24T00:22:31.990 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stdout: "return_code": 0 2024-03-24T00:22:31.990 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stdout:} 2024-03-24T00:22:31.990 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:22:31.988+0000 7fc2ac600640 1 -- 172.21.15.118:0/2122589690 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7fc28c0a2920 msgr2=0x7fc28c0a4de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:22:31.991 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:22:31.988+0000 7fc2ac600640 1 --2- 172.21.15.118:0/2122589690 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7fc28c0a2920 0x7fc28c0a4de0 secure :-1 s=READY pgs=376 cs=0 l=1 rev1=1 crypto rx=0x7fc29802efd0 tx=0x7fc29802cf90 comp rx=0 tx=0).stop 2024-03-24T00:22:31.991 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:22:31.988+0000 7fc2ac600640 1 -- 172.21.15.118:0/2122589690 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc2a40772a0 msgr2=0x7fc2a419e420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:22:31.991 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:22:31.988+0000 7fc2ac600640 1 --2- 172.21.15.118:0/2122589690 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc2a40772a0 0x7fc2a419e420 secure :-1 s=READY pgs=437 cs=0 l=1 rev1=1 crypto rx=0x7fc29000e9f0 tx=0x7fc29000edf0 comp rx=0 tx=0).stop 2024-03-24T00:22:31.991 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:22:31.988+0000 7fc2ac600640 1 -- 172.21.15.118:0/2122589690 shutdown_connections 2024-03-24T00:22:31.991 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:22:31.988+0000 7fc2ac600640 1 --2- 172.21.15.118:0/2122589690 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7fc28c0a2920 0x7fc28c0a4de0 unknown :-1 s=CLOSED pgs=376 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:22:31.991 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:22:31.988+0000 7fc2ac600640 1 --2- 172.21.15.118:0/2122589690 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fc2a4077c40 0x7fc2a41a5490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:22:31.991 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:22:31.988+0000 7fc2ac600640 1 --2- 172.21.15.118:0/2122589690 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fc2a40772a0 0x7fc2a419e420 unknown :-1 s=CLOSED pgs=437 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:22:31.991 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:22:31.988+0000 7fc2ac600640 1 --2- 172.21.15.118:0/2122589690 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fc2a4076060 0x7fc2a419dee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:22:31.991 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:22:31.988+0000 7fc2ac600640 1 -- 172.21.15.118:0/2122589690 >> 172.21.15.118:0/2122589690 conn(0x7fc2a40fee90 msgr2=0x7fc2a4100990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:22:31.991 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:22:31.988+0000 7fc2ac600640 1 -- 172.21.15.118:0/2122589690 shutdown_connections 2024-03-24T00:22:31.991 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:22:31.988+0000 7fc2ac600640 1 -- 172.21.15.118:0/2122589690 wait complete. 2024-03-24T00:22:32.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:22:31 smithi193 ceph-mon[36878]: mds.? [v2:172.21.15.193:6832/2424695757,v1:172.21.15.193:6833/2424695757] up:reconnect 2024-03-24T00:22:32.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:22:31 smithi193 ceph-mon[36878]: mds.? [v2:172.21.15.189:6836/75274228,v1:172.21.15.189:6837/75274228] up:reconnect 2024-03-24T00:22:32.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:22:31 smithi193 ceph-mon[36878]: fsmap cephfs:5/5 3 up:standby-replay 3 up:standby 2 up:reconnect 3 up:active 2024-03-24T00:22:32.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:22:31 smithi193 ceph-mon[36878]: reconnect by client.15159 192.168.0.1:0/3463739313 after 0.0969985 2024-03-24T00:22:32.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:22:31 smithi193 ceph-mon[36878]: reconnect by client.15159 192.168.0.1:0/3463739313 after 0.113998 2024-03-24T00:22:32.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:22:31 smithi193 ceph-mon[36878]: pgmap v621: 97 pgs: 97 active+clean; 149 MiB data, 799 MiB used, 1.0 TiB / 1.0 TiB avail; 61 KiB/s rd, 48 KiB/s wr, 135 op/s 2024-03-24T00:22:32.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:22:31 smithi118 ceph-mon[31535]: mds.? [v2:172.21.15.193:6832/2424695757,v1:172.21.15.193:6833/2424695757] up:reconnect 2024-03-24T00:22:32.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:22:31 smithi118 ceph-mon[31535]: mds.? [v2:172.21.15.189:6836/75274228,v1:172.21.15.189:6837/75274228] up:reconnect 2024-03-24T00:22:32.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:22:31 smithi118 ceph-mon[31535]: fsmap cephfs:5/5 3 up:standby-replay 3 up:standby 2 up:reconnect 3 up:active 2024-03-24T00:22:32.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:22:31 smithi118 ceph-mon[31535]: reconnect by client.15159 192.168.0.1:0/3463739313 after 0.0969985 2024-03-24T00:22:32.447 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:22:31 smithi118 ceph-mon[31535]: reconnect by client.15159 192.168.0.1:0/3463739313 after 0.113998 2024-03-24T00:22:32.447 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:22:31 smithi118 ceph-mon[31535]: pgmap v621: 97 pgs: 97 active+clean; 149 MiB data, 799 MiB used, 1.0 TiB / 1.0 TiB avail; 61 KiB/s rd, 48 KiB/s wr, 135 op/s 2024-03-24T00:22:32.453 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:22:31 smithi189 ceph-mon[36187]: mds.? [v2:172.21.15.193:6832/2424695757,v1:172.21.15.193:6833/2424695757] up:reconnect 2024-03-24T00:22:32.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:22:31 smithi189 ceph-mon[36187]: mds.? [v2:172.21.15.189:6836/75274228,v1:172.21.15.189:6837/75274228] up:reconnect 2024-03-24T00:22:32.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:22:31 smithi189 ceph-mon[36187]: fsmap cephfs:5/5 3 up:standby-replay 3 up:standby 2 up:reconnect 3 up:active 2024-03-24T00:22:32.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:22:31 smithi189 ceph-mon[36187]: reconnect by client.15159 192.168.0.1:0/3463739313 after 0.0969985 2024-03-24T00:22:32.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:22:31 smithi189 ceph-mon[36187]: reconnect by client.15159 192.168.0.1:0/3463739313 after 0.113998 2024-03-24T00:22:32.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:22:31 smithi189 ceph-mon[36187]: pgmap v621: 97 pgs: 97 active+clean; 149 MiB data, 799 MiB used, 1.0 TiB / 1.0 TiB avail; 61 KiB/s rd, 48 KiB/s wr, 135 op/s 2024-03-24T00:22:33.370 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:22:32 smithi118 ceph-mon[31535]: mds.? [v2:172.21.15.193:6832/2424695757,v1:172.21.15.193:6833/2424695757] up:rejoin 2024-03-24T00:22:33.370 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:22:32 smithi118 ceph-mon[31535]: mds.? [v2:172.21.15.189:6836/75274228,v1:172.21.15.189:6837/75274228] up:rejoin 2024-03-24T00:22:33.371 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:22:32 smithi118 ceph-mon[31535]: fsmap cephfs:5/5 3 up:standby-replay 3 up:standby 2 up:rejoin 3 up:active 2024-03-24T00:22:33.371 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:22:32 smithi118 ceph-mon[31535]: daemon mds.h is now active in filesystem cephfs as rank 3 2024-03-24T00:22:33.371 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:22:32 smithi118 ceph-mon[31535]: daemon mds.l is now active in filesystem cephfs as rank 1 2024-03-24T00:22:33.378 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:22:32 smithi189 ceph-mon[36187]: mds.? [v2:172.21.15.193:6832/2424695757,v1:172.21.15.193:6833/2424695757] up:rejoin 2024-03-24T00:22:33.378 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:22:32 smithi189 ceph-mon[36187]: mds.? [v2:172.21.15.189:6836/75274228,v1:172.21.15.189:6837/75274228] up:rejoin 2024-03-24T00:22:33.378 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:22:32 smithi189 ceph-mon[36187]: fsmap cephfs:5/5 3 up:standby-replay 3 up:standby 2 up:rejoin 3 up:active 2024-03-24T00:22:33.378 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:22:32 smithi189 ceph-mon[36187]: daemon mds.h is now active in filesystem cephfs as rank 3 2024-03-24T00:22:33.378 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:22:32 smithi189 ceph-mon[36187]: daemon mds.l is now active in filesystem cephfs as rank 1 2024-03-24T00:22:33.379 INFO:journalctl@ceph.mds.h.smithi189.stdout:Mar 24 00:22:32 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-h[70087]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: In function 'void Locker::set_xlocks_done(MutationImpl*, bool)' thread 7fe0d5eb3700 time 2024-03-24T00:22:32.991146+0000 2024-03-24T00:22:33.379 INFO:journalctl@ceph.mds.h.smithi189.stdout:Mar 24 00:22:32 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-h[70087]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: 626: FAILED ceph_assert(obj->is_auth()) 2024-03-24T00:22:33.379 INFO:journalctl@ceph.mds.h.smithi189.stdout:Mar 24 00:22:32 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-h[70087]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-24T00:22:33.379 INFO:journalctl@ceph.mds.h.smithi189.stdout:Mar 24 00:22:32 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-h[70087]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x135) [0x7fe0e04dd4a1] 2024-03-24T00:22:33.379 INFO:journalctl@ceph.mds.h.smithi189.stdout:Mar 24 00:22:32 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-h[70087]: 2: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7fe0e04dd667] 2024-03-24T00:22:33.379 INFO:journalctl@ceph.mds.h.smithi189.stdout:Mar 24 00:22:32 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-h[70087]: 3: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x563aa0abc223] 2024-03-24T00:22:33.379 INFO:journalctl@ceph.mds.h.smithi189.stdout:Mar 24 00:22:32 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-h[70087]: 4: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x563aa09365d6] 2024-03-24T00:22:33.380 INFO:journalctl@ceph.mds.h.smithi189.stdout:Mar 24 00:22:32 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-h[70087]: 5: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x563aa0936da1] 2024-03-24T00:22:33.380 INFO:journalctl@ceph.mds.h.smithi189.stdout:Mar 24 00:22:32 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-h[70087]: 6: (Server::handle_client_rename(boost::intrusive_ptr const&)+0x259d) [0x563aa097397d] 2024-03-24T00:22:33.380 INFO:journalctl@ceph.mds.h.smithi189.stdout:Mar 24 00:22:32 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-h[70087]: 7: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x563aa0989233] 2024-03-24T00:22:33.380 INFO:journalctl@ceph.mds.h.smithi189.stdout:Mar 24 00:22:32 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-h[70087]: 8: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x563aa098b197] 2024-03-24T00:22:33.380 INFO:journalctl@ceph.mds.h.smithi189.stdout:Mar 24 00:22:32 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-h[70087]: 9: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x563aa088a11b] 2024-03-24T00:22:33.380 INFO:journalctl@ceph.mds.h.smithi189.stdout:Mar 24 00:22:32 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-h[70087]: 10: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x563aa088a6bc] 2024-03-24T00:22:33.380 INFO:journalctl@ceph.mds.h.smithi189.stdout:Mar 24 00:22:32 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-h[70087]: 11: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x563aa08741ff] 2024-03-24T00:22:33.380 INFO:journalctl@ceph.mds.h.smithi189.stdout:Mar 24 00:22:32 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-h[70087]: 12: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7fe0e076f488] 2024-03-24T00:22:33.380 INFO:journalctl@ceph.mds.h.smithi189.stdout:Mar 24 00:22:32 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-h[70087]: 13: (DispatchQueue::entry()+0x51f) [0x7fe0e076c60f] 2024-03-24T00:22:33.380 INFO:journalctl@ceph.mds.h.smithi189.stdout:Mar 24 00:22:32 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-h[70087]: 14: (DispatchQueue::DispatchThread::entry()+0x11) [0x7fe0e083fe81] 2024-03-24T00:22:33.380 INFO:journalctl@ceph.mds.h.smithi189.stdout:Mar 24 00:22:32 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-h[70087]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7fe0df25a1ca] 2024-03-24T00:22:33.380 INFO:journalctl@ceph.mds.h.smithi189.stdout:Mar 24 00:22:32 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-h[70087]: 16: clone() 2024-03-24T00:22:33.380 INFO:journalctl@ceph.mds.h.smithi189.stdout:Mar 24 00:22:32 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-h[70087]: 2024-03-24T00:22:32.992+0000 7fe0d5eb3700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: In function 'void Locker::set_xlocks_done(MutationImpl*, bool)' thread 7fe0d5eb3700 time 2024-03-24T00:22:32.991146+0000 2024-03-24T00:22:33.381 INFO:journalctl@ceph.mds.h.smithi189.stdout:Mar 24 00:22:32 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-h[70087]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: 626: FAILED ceph_assert(obj->is_auth()) 2024-03-24T00:22:33.381 INFO:journalctl@ceph.mds.h.smithi189.stdout:Mar 24 00:22:32 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-h[70087]: 2024-03-24T00:22:33.381 INFO:journalctl@ceph.mds.h.smithi189.stdout:Mar 24 00:22:32 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-h[70087]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-24T00:22:33.381 INFO:journalctl@ceph.mds.h.smithi189.stdout:Mar 24 00:22:32 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-h[70087]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x135) [0x7fe0e04dd4a1] 2024-03-24T00:22:33.381 INFO:journalctl@ceph.mds.h.smithi189.stdout:Mar 24 00:22:32 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-h[70087]: 2: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7fe0e04dd667] 2024-03-24T00:22:33.381 INFO:journalctl@ceph.mds.h.smithi189.stdout:Mar 24 00:22:32 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-h[70087]: 3: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x563aa0abc223] 2024-03-24T00:22:33.381 INFO:journalctl@ceph.mds.h.smithi189.stdout:Mar 24 00:22:32 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-h[70087]: 4: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x563aa09365d6] 2024-03-24T00:22:33.381 INFO:journalctl@ceph.mds.h.smithi189.stdout:Mar 24 00:22:32 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-h[70087]: 5: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x563aa0936da1] 2024-03-24T00:22:33.381 INFO:journalctl@ceph.mds.h.smithi189.stdout:Mar 24 00:22:32 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-h[70087]: 6: (Server::handle_client_rename(boost::intrusive_ptr const&)+0x259d) [0x563aa097397d] 2024-03-24T00:22:33.381 INFO:journalctl@ceph.mds.h.smithi189.stdout:Mar 24 00:22:32 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-h[70087]: 7: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x563aa0989233] 2024-03-24T00:22:33.381 INFO:journalctl@ceph.mds.h.smithi189.stdout:Mar 24 00:22:32 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-h[70087]: 8: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x563aa098b197] 2024-03-24T00:22:33.381 INFO:journalctl@ceph.mds.h.smithi189.stdout:Mar 24 00:22:32 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-h[70087]: 9: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x563aa088a11b] 2024-03-24T00:22:33.381 INFO:journalctl@ceph.mds.h.smithi189.stdout:Mar 24 00:22:32 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-h[70087]: 10: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x563aa088a6bc] 2024-03-24T00:22:33.381 INFO:journalctl@ceph.mds.h.smithi189.stdout:Mar 24 00:22:32 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-h[70087]: 11: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x563aa08741ff] 2024-03-24T00:22:33.381 INFO:journalctl@ceph.mds.h.smithi189.stdout:Mar 24 00:22:32 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-h[70087]: 12: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7fe0e076f488] 2024-03-24T00:22:33.382 INFO:journalctl@ceph.mds.h.smithi189.stdout:Mar 24 00:22:32 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-h[70087]: 13: (DispatchQueue::entry()+0x51f) [0x7fe0e076c60f] 2024-03-24T00:22:33.382 INFO:journalctl@ceph.mds.h.smithi189.stdout:Mar 24 00:22:32 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-h[70087]: 14: (DispatchQueue::DispatchThread::entry()+0x11) [0x7fe0e083fe81] 2024-03-24T00:22:33.382 INFO:journalctl@ceph.mds.h.smithi189.stdout:Mar 24 00:22:32 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-h[70087]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7fe0df25a1ca] 2024-03-24T00:22:33.382 INFO:journalctl@ceph.mds.h.smithi189.stdout:Mar 24 00:22:32 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-h[70087]: 16: clone() 2024-03-24T00:22:33.382 INFO:journalctl@ceph.mds.h.smithi189.stdout:Mar 24 00:22:32 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-h[70087]: 2024-03-24T00:22:33.382 INFO:journalctl@ceph.mds.h.smithi189.stdout:Mar 24 00:22:32 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-h[70087]: *** Caught signal (Aborted) ** 2024-03-24T00:22:33.382 INFO:journalctl@ceph.mds.h.smithi189.stdout:Mar 24 00:22:32 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-h[70087]: in thread 7fe0d5eb3700 thread_name:ms_dispatch 2024-03-24T00:22:33.382 INFO:journalctl@ceph.mds.h.smithi189.stdout:Mar 24 00:22:32 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-h[70087]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-24T00:22:33.382 INFO:journalctl@ceph.mds.h.smithi189.stdout:Mar 24 00:22:32 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-h[70087]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7fe0df264d20] 2024-03-24T00:22:33.383 INFO:journalctl@ceph.mds.h.smithi189.stdout:Mar 24 00:22:32 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-h[70087]: 2: gsignal() 2024-03-24T00:22:33.383 INFO:journalctl@ceph.mds.h.smithi189.stdout:Mar 24 00:22:32 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-h[70087]: 3: abort() 2024-03-24T00:22:33.383 INFO:journalctl@ceph.mds.h.smithi189.stdout:Mar 24 00:22:32 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-h[70087]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x18f) [0x7fe0e04dd4fb] 2024-03-24T00:22:33.383 INFO:journalctl@ceph.mds.h.smithi189.stdout:Mar 24 00:22:32 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-h[70087]: 5: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7fe0e04dd667] 2024-03-24T00:22:33.383 INFO:journalctl@ceph.mds.h.smithi189.stdout:Mar 24 00:22:32 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-h[70087]: 6: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x563aa0abc223] 2024-03-24T00:22:33.384 INFO:journalctl@ceph.mds.h.smithi189.stdout:Mar 24 00:22:32 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-h[70087]: 7: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x563aa09365d6] 2024-03-24T00:22:33.384 INFO:journalctl@ceph.mds.h.smithi189.stdout:Mar 24 00:22:32 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-h[70087]: 8: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x563aa0936da1] 2024-03-24T00:22:33.384 INFO:journalctl@ceph.mds.h.smithi189.stdout:Mar 24 00:22:32 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-h[70087]: 9: (Server::handle_client_rename(boost::intrusive_ptr const&)+0x259d) [0x563aa097397d] 2024-03-24T00:22:33.384 INFO:journalctl@ceph.mds.h.smithi189.stdout:Mar 24 00:22:32 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-h[70087]: 10: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x563aa0989233] 2024-03-24T00:22:33.384 INFO:journalctl@ceph.mds.h.smithi189.stdout:Mar 24 00:22:32 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-h[70087]: 11: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x563aa098b197] 2024-03-24T00:22:33.384 INFO:journalctl@ceph.mds.h.smithi189.stdout:Mar 24 00:22:32 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-h[70087]: 12: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x563aa088a11b] 2024-03-24T00:22:33.384 INFO:journalctl@ceph.mds.h.smithi189.stdout:Mar 24 00:22:32 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-h[70087]: 13: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x563aa088a6bc] 2024-03-24T00:22:33.384 INFO:journalctl@ceph.mds.h.smithi189.stdout:Mar 24 00:22:32 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-h[70087]: 14: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x563aa08741ff] 2024-03-24T00:22:33.384 INFO:journalctl@ceph.mds.h.smithi189.stdout:Mar 24 00:22:32 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-h[70087]: 15: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7fe0e076f488] 2024-03-24T00:22:33.384 INFO:journalctl@ceph.mds.h.smithi189.stdout:Mar 24 00:22:32 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-h[70087]: 16: (DispatchQueue::entry()+0x51f) [0x7fe0e076c60f] 2024-03-24T00:22:33.384 INFO:journalctl@ceph.mds.h.smithi189.stdout:Mar 24 00:22:32 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-h[70087]: 17: (DispatchQueue::DispatchThread::entry()+0x11) [0x7fe0e083fe81] 2024-03-24T00:22:33.384 INFO:journalctl@ceph.mds.h.smithi189.stdout:Mar 24 00:22:32 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-h[70087]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7fe0df25a1ca] 2024-03-24T00:22:33.384 INFO:journalctl@ceph.mds.h.smithi189.stdout:Mar 24 00:22:32 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-h[70087]: 19: clone() 2024-03-24T00:22:33.384 INFO:journalctl@ceph.mds.h.smithi189.stdout:Mar 24 00:22:32 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-h[70087]: 2024-03-24T00:22:32.994+0000 7fe0d5eb3700 -1 *** Caught signal (Aborted) ** 2024-03-24T00:22:33.384 INFO:journalctl@ceph.mds.h.smithi189.stdout:Mar 24 00:22:32 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-h[70087]: in thread 7fe0d5eb3700 thread_name:ms_dispatch 2024-03-24T00:22:33.385 INFO:journalctl@ceph.mds.h.smithi189.stdout:Mar 24 00:22:32 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-h[70087]: 2024-03-24T00:22:33.385 INFO:journalctl@ceph.mds.h.smithi189.stdout:Mar 24 00:22:32 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-h[70087]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-24T00:22:33.385 INFO:journalctl@ceph.mds.h.smithi189.stdout:Mar 24 00:22:32 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-h[70087]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7fe0df264d20] 2024-03-24T00:22:33.385 INFO:journalctl@ceph.mds.h.smithi189.stdout:Mar 24 00:22:32 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-h[70087]: 2: gsignal() 2024-03-24T00:22:33.385 INFO:journalctl@ceph.mds.h.smithi189.stdout:Mar 24 00:22:32 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-h[70087]: 3: abort() 2024-03-24T00:22:33.385 INFO:journalctl@ceph.mds.h.smithi189.stdout:Mar 24 00:22:32 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-h[70087]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x18f) [0x7fe0e04dd4fb] 2024-03-24T00:22:33.385 INFO:journalctl@ceph.mds.h.smithi189.stdout:Mar 24 00:22:32 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-h[70087]: 5: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7fe0e04dd667] 2024-03-24T00:22:33.385 INFO:journalctl@ceph.mds.h.smithi189.stdout:Mar 24 00:22:32 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-h[70087]: 6: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x563aa0abc223] 2024-03-24T00:22:33.385 INFO:journalctl@ceph.mds.h.smithi189.stdout:Mar 24 00:22:32 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-h[70087]: 7: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x563aa09365d6] 2024-03-24T00:22:33.385 INFO:journalctl@ceph.mds.h.smithi189.stdout:Mar 24 00:22:32 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-h[70087]: 8: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x563aa0936da1] 2024-03-24T00:22:33.385 INFO:journalctl@ceph.mds.h.smithi189.stdout:Mar 24 00:22:32 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-h[70087]: 9: (Server::handle_client_rename(boost::intrusive_ptr const&)+0x259d) [0x563aa097397d] 2024-03-24T00:22:33.385 INFO:journalctl@ceph.mds.h.smithi189.stdout:Mar 24 00:22:32 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-h[70087]: 10: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x563aa0989233] 2024-03-24T00:22:33.385 INFO:journalctl@ceph.mds.h.smithi189.stdout:Mar 24 00:22:32 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-h[70087]: 11: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x563aa098b197] 2024-03-24T00:22:33.386 INFO:journalctl@ceph.mds.h.smithi189.stdout:Mar 24 00:22:32 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-h[70087]: 12: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x563aa088a11b] 2024-03-24T00:22:33.386 INFO:journalctl@ceph.mds.h.smithi189.stdout:Mar 24 00:22:32 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-h[70087]: 13: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x563aa088a6bc] 2024-03-24T00:22:33.386 INFO:journalctl@ceph.mds.h.smithi189.stdout:Mar 24 00:22:32 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-h[70087]: 14: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x563aa08741ff] 2024-03-24T00:22:33.386 INFO:journalctl@ceph.mds.h.smithi189.stdout:Mar 24 00:22:32 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-h[70087]: 15: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7fe0e076f488] 2024-03-24T00:22:33.386 INFO:journalctl@ceph.mds.h.smithi189.stdout:Mar 24 00:22:32 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-h[70087]: 16: (DispatchQueue::entry()+0x51f) [0x7fe0e076c60f] 2024-03-24T00:22:33.386 INFO:journalctl@ceph.mds.h.smithi189.stdout:Mar 24 00:22:32 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-h[70087]: 17: (DispatchQueue::DispatchThread::entry()+0x11) [0x7fe0e083fe81] 2024-03-24T00:22:33.386 INFO:journalctl@ceph.mds.h.smithi189.stdout:Mar 24 00:22:32 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-h[70087]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7fe0df25a1ca] 2024-03-24T00:22:33.386 INFO:journalctl@ceph.mds.h.smithi189.stdout:Mar 24 00:22:32 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-h[70087]: 19: clone() 2024-03-24T00:22:33.386 INFO:journalctl@ceph.mds.h.smithi189.stdout:Mar 24 00:22:32 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-h[70087]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-03-24T00:22:33.386 INFO:journalctl@ceph.mds.h.smithi189.stdout:Mar 24 00:22:32 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-h[70087]: 2024-03-24T00:22:33.386 INFO:journalctl@ceph.mds.h.smithi189.stdout:Mar 24 00:22:33 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-h[70087]: -1> 2024-03-24T00:22:32.992+0000 7fe0d5eb3700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: In function 'void Locker::set_xlocks_done(MutationImpl*, bool)' thread 7fe0d5eb3700 time 2024-03-24T00:22:32.991146+0000 2024-03-24T00:22:33.386 INFO:journalctl@ceph.mds.h.smithi189.stdout:Mar 24 00:22:33 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-h[70087]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: 626: FAILED ceph_assert(obj->is_auth()) 2024-03-24T00:22:33.386 INFO:journalctl@ceph.mds.h.smithi189.stdout:Mar 24 00:22:33 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-h[70087]: 2024-03-24T00:22:33.386 INFO:journalctl@ceph.mds.h.smithi189.stdout:Mar 24 00:22:33 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-h[70087]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-24T00:22:33.387 INFO:journalctl@ceph.mds.h.smithi189.stdout:Mar 24 00:22:33 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-h[70087]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x135) [0x7fe0e04dd4a1] 2024-03-24T00:22:33.387 INFO:journalctl@ceph.mds.h.smithi189.stdout:Mar 24 00:22:33 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-h[70087]: 2: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7fe0e04dd667] 2024-03-24T00:22:33.387 INFO:journalctl@ceph.mds.h.smithi189.stdout:Mar 24 00:22:33 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-h[70087]: 3: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x563aa0abc223] 2024-03-24T00:22:33.387 INFO:journalctl@ceph.mds.h.smithi189.stdout:Mar 24 00:22:33 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-h[70087]: 4: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x563aa09365d6] 2024-03-24T00:22:33.387 INFO:journalctl@ceph.mds.h.smithi189.stdout:Mar 24 00:22:33 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-h[70087]: 5: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x563aa0936da1] 2024-03-24T00:22:33.387 INFO:journalctl@ceph.mds.h.smithi189.stdout:Mar 24 00:22:33 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-h[70087]: 6: (Server::handle_client_rename(boost::intrusive_ptr const&)+0x259d) [0x563aa097397d] 2024-03-24T00:22:33.387 INFO:journalctl@ceph.mds.h.smithi189.stdout:Mar 24 00:22:33 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-h[70087]: 7: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x563aa0989233] 2024-03-24T00:22:33.387 INFO:journalctl@ceph.mds.h.smithi189.stdout:Mar 24 00:22:33 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-h[70087]: 8: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x563aa098b197] 2024-03-24T00:22:33.387 INFO:journalctl@ceph.mds.h.smithi189.stdout:Mar 24 00:22:33 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-h[70087]: 9: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x563aa088a11b] 2024-03-24T00:22:33.387 INFO:journalctl@ceph.mds.h.smithi189.stdout:Mar 24 00:22:33 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-h[70087]: 10: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x563aa088a6bc] 2024-03-24T00:22:33.387 INFO:journalctl@ceph.mds.h.smithi189.stdout:Mar 24 00:22:33 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-h[70087]: 11: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x563aa08741ff] 2024-03-24T00:22:33.387 INFO:journalctl@ceph.mds.h.smithi189.stdout:Mar 24 00:22:33 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-h[70087]: 12: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7fe0e076f488] 2024-03-24T00:22:33.387 INFO:journalctl@ceph.mds.h.smithi189.stdout:Mar 24 00:22:33 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-h[70087]: 13: (DispatchQueue::entry()+0x51f) [0x7fe0e076c60f] 2024-03-24T00:22:33.387 INFO:journalctl@ceph.mds.h.smithi189.stdout:Mar 24 00:22:33 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-h[70087]: 14: (DispatchQueue::DispatchThread::entry()+0x11) [0x7fe0e083fe81] 2024-03-24T00:22:33.387 INFO:journalctl@ceph.mds.h.smithi189.stdout:Mar 24 00:22:33 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-h[70087]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7fe0df25a1ca] 2024-03-24T00:22:33.388 INFO:journalctl@ceph.mds.h.smithi189.stdout:Mar 24 00:22:33 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-h[70087]: 16: clone() 2024-03-24T00:22:33.388 INFO:journalctl@ceph.mds.h.smithi189.stdout:Mar 24 00:22:33 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-h[70087]: 2024-03-24T00:22:33.388 INFO:journalctl@ceph.mds.h.smithi189.stdout:Mar 24 00:22:33 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-h[70087]: 0> 2024-03-24T00:22:32.994+0000 7fe0d5eb3700 -1 *** Caught signal (Aborted) ** 2024-03-24T00:22:33.388 INFO:journalctl@ceph.mds.h.smithi189.stdout:Mar 24 00:22:33 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-h[70087]: in thread 7fe0d5eb3700 thread_name:ms_dispatch 2024-03-24T00:22:33.388 INFO:journalctl@ceph.mds.h.smithi189.stdout:Mar 24 00:22:33 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-h[70087]: 2024-03-24T00:22:33.388 INFO:journalctl@ceph.mds.h.smithi189.stdout:Mar 24 00:22:33 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-h[70087]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-24T00:22:33.388 INFO:journalctl@ceph.mds.h.smithi189.stdout:Mar 24 00:22:33 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-h[70087]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7fe0df264d20] 2024-03-24T00:22:33.388 INFO:journalctl@ceph.mds.h.smithi189.stdout:Mar 24 00:22:33 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-h[70087]: 2: gsignal() 2024-03-24T00:22:33.388 INFO:journalctl@ceph.mds.h.smithi189.stdout:Mar 24 00:22:33 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-h[70087]: 3: abort() 2024-03-24T00:22:33.388 INFO:journalctl@ceph.mds.h.smithi189.stdout:Mar 24 00:22:33 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-h[70087]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x18f) [0x7fe0e04dd4fb] 2024-03-24T00:22:33.388 INFO:journalctl@ceph.mds.h.smithi189.stdout:Mar 24 00:22:33 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-h[70087]: 5: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7fe0e04dd667] 2024-03-24T00:22:33.388 INFO:journalctl@ceph.mds.h.smithi189.stdout:Mar 24 00:22:33 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-h[70087]: 6: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x563aa0abc223] 2024-03-24T00:22:33.388 INFO:journalctl@ceph.mds.h.smithi189.stdout:Mar 24 00:22:33 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-h[70087]: 7: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x563aa09365d6] 2024-03-24T00:22:33.388 INFO:journalctl@ceph.mds.h.smithi189.stdout:Mar 24 00:22:33 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-h[70087]: 8: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x563aa0936da1] 2024-03-24T00:22:33.388 INFO:journalctl@ceph.mds.h.smithi189.stdout:Mar 24 00:22:33 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-h[70087]: 9: (Server::handle_client_rename(boost::intrusive_ptr const&)+0x259d) [0x563aa097397d] 2024-03-24T00:22:33.389 INFO:journalctl@ceph.mds.h.smithi189.stdout:Mar 24 00:22:33 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-h[70087]: 10: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x563aa0989233] 2024-03-24T00:22:33.389 INFO:journalctl@ceph.mds.h.smithi189.stdout:Mar 24 00:22:33 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-h[70087]: 11: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x563aa098b197] 2024-03-24T00:22:33.389 INFO:journalctl@ceph.mds.h.smithi189.stdout:Mar 24 00:22:33 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-h[70087]: 12: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x563aa088a11b] 2024-03-24T00:22:33.389 INFO:journalctl@ceph.mds.h.smithi189.stdout:Mar 24 00:22:33 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-h[70087]: 13: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x563aa088a6bc] 2024-03-24T00:22:33.389 INFO:journalctl@ceph.mds.h.smithi189.stdout:Mar 24 00:22:33 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-h[70087]: 14: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x563aa08741ff] 2024-03-24T00:22:33.389 INFO:journalctl@ceph.mds.h.smithi189.stdout:Mar 24 00:22:33 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-h[70087]: 15: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7fe0e076f488] 2024-03-24T00:22:33.389 INFO:journalctl@ceph.mds.h.smithi189.stdout:Mar 24 00:22:33 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-h[70087]: 16: (DispatchQueue::entry()+0x51f) [0x7fe0e076c60f] 2024-03-24T00:22:33.389 INFO:journalctl@ceph.mds.h.smithi189.stdout:Mar 24 00:22:33 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-h[70087]: 17: (DispatchQueue::DispatchThread::entry()+0x11) [0x7fe0e083fe81] 2024-03-24T00:22:33.389 INFO:journalctl@ceph.mds.h.smithi189.stdout:Mar 24 00:22:33 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-h[70087]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7fe0df25a1ca] 2024-03-24T00:22:33.389 INFO:journalctl@ceph.mds.h.smithi189.stdout:Mar 24 00:22:33 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-h[70087]: 19: clone() 2024-03-24T00:22:33.389 INFO:journalctl@ceph.mds.h.smithi189.stdout:Mar 24 00:22:33 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-h[70087]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-03-24T00:22:33.389 INFO:journalctl@ceph.mds.h.smithi189.stdout:Mar 24 00:22:33 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-h[70087]: 2024-03-24T00:22:33.389 INFO:journalctl@ceph.mds.h.smithi189.stdout:Mar 24 00:22:33 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-h[70087]: -1> 2024-03-24T00:22:32.992+0000 7fe0d5eb3700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: In function 'void Locker::set_xlocks_done(MutationImpl*, bool)' thread 7fe0d5eb3700 time 2024-03-24T00:22:32.991146+0000 2024-03-24T00:22:33.389 INFO:journalctl@ceph.mds.h.smithi189.stdout:Mar 24 00:22:33 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-h[70087]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: 626: FAILED ceph_assert(obj->is_auth()) 2024-03-24T00:22:33.389 INFO:journalctl@ceph.mds.h.smithi189.stdout:Mar 24 00:22:33 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-h[70087]: 2024-03-24T00:22:33.390 INFO:journalctl@ceph.mds.h.smithi189.stdout:Mar 24 00:22:33 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-h[70087]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-24T00:22:33.390 INFO:journalctl@ceph.mds.h.smithi189.stdout:Mar 24 00:22:33 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-h[70087]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x135) [0x7fe0e04dd4a1] 2024-03-24T00:22:33.390 INFO:journalctl@ceph.mds.h.smithi189.stdout:Mar 24 00:22:33 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-h[70087]: 2: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7fe0e04dd667] 2024-03-24T00:22:33.390 INFO:journalctl@ceph.mds.h.smithi189.stdout:Mar 24 00:22:33 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-h[70087]: 3: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x563aa0abc223] 2024-03-24T00:22:33.390 INFO:journalctl@ceph.mds.h.smithi189.stdout:Mar 24 00:22:33 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-h[70087]: 4: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x563aa09365d6] 2024-03-24T00:22:33.390 INFO:journalctl@ceph.mds.h.smithi189.stdout:Mar 24 00:22:33 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-h[70087]: 5: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x563aa0936da1] 2024-03-24T00:22:33.390 INFO:journalctl@ceph.mds.h.smithi189.stdout:Mar 24 00:22:33 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-h[70087]: 6: (Server::handle_client_rename(boost::intrusive_ptr const&)+0x259d) [0x563aa097397d] 2024-03-24T00:22:33.390 INFO:journalctl@ceph.mds.h.smithi189.stdout:Mar 24 00:22:33 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-h[70087]: 7: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x563aa0989233] 2024-03-24T00:22:33.390 INFO:journalctl@ceph.mds.h.smithi189.stdout:Mar 24 00:22:33 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-h[70087]: 8: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x563aa098b197] 2024-03-24T00:22:33.390 INFO:journalctl@ceph.mds.h.smithi189.stdout:Mar 24 00:22:33 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-h[70087]: 9: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x563aa088a11b] 2024-03-24T00:22:33.390 INFO:journalctl@ceph.mds.h.smithi189.stdout:Mar 24 00:22:33 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-h[70087]: 10: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x563aa088a6bc] 2024-03-24T00:22:33.390 INFO:journalctl@ceph.mds.h.smithi189.stdout:Mar 24 00:22:33 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-h[70087]: 11: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x563aa08741ff] 2024-03-24T00:22:33.391 INFO:journalctl@ceph.mds.h.smithi189.stdout:Mar 24 00:22:33 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-h[70087]: 12: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7fe0e076f488] 2024-03-24T00:22:33.391 INFO:journalctl@ceph.mds.h.smithi189.stdout:Mar 24 00:22:33 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-h[70087]: 13: (DispatchQueue::entry()+0x51f) [0x7fe0e076c60f] 2024-03-24T00:22:33.391 INFO:journalctl@ceph.mds.h.smithi189.stdout:Mar 24 00:22:33 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-h[70087]: 14: (DispatchQueue::DispatchThread::entry()+0x11) [0x7fe0e083fe81] 2024-03-24T00:22:33.391 INFO:journalctl@ceph.mds.h.smithi189.stdout:Mar 24 00:22:33 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-h[70087]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7fe0df25a1ca] 2024-03-24T00:22:33.391 INFO:journalctl@ceph.mds.h.smithi189.stdout:Mar 24 00:22:33 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-h[70087]: 16: clone() 2024-03-24T00:22:33.391 INFO:journalctl@ceph.mds.h.smithi189.stdout:Mar 24 00:22:33 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-h[70087]: 2024-03-24T00:22:33.391 INFO:journalctl@ceph.mds.h.smithi189.stdout:Mar 24 00:22:33 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-h[70087]: 0> 2024-03-24T00:22:32.994+0000 7fe0d5eb3700 -1 *** Caught signal (Aborted) ** 2024-03-24T00:22:33.391 INFO:journalctl@ceph.mds.h.smithi189.stdout:Mar 24 00:22:33 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-h[70087]: in thread 7fe0d5eb3700 thread_name:ms_dispatch 2024-03-24T00:22:33.391 INFO:journalctl@ceph.mds.h.smithi189.stdout:Mar 24 00:22:33 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-h[70087]: 2024-03-24T00:22:33.391 INFO:journalctl@ceph.mds.h.smithi189.stdout:Mar 24 00:22:33 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-h[70087]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-24T00:22:33.391 INFO:journalctl@ceph.mds.h.smithi189.stdout:Mar 24 00:22:33 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-h[70087]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7fe0df264d20] 2024-03-24T00:22:33.391 INFO:journalctl@ceph.mds.h.smithi189.stdout:Mar 24 00:22:33 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-h[70087]: 2: gsignal() 2024-03-24T00:22:33.391 INFO:journalctl@ceph.mds.h.smithi189.stdout:Mar 24 00:22:33 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-h[70087]: 3: abort() 2024-03-24T00:22:33.392 INFO:journalctl@ceph.mds.h.smithi189.stdout:Mar 24 00:22:33 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-h[70087]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x18f) [0x7fe0e04dd4fb] 2024-03-24T00:22:33.392 INFO:journalctl@ceph.mds.h.smithi189.stdout:Mar 24 00:22:33 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-h[70087]: 5: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7fe0e04dd667] 2024-03-24T00:22:33.392 INFO:journalctl@ceph.mds.h.smithi189.stdout:Mar 24 00:22:33 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-h[70087]: 6: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x563aa0abc223] 2024-03-24T00:22:33.392 INFO:journalctl@ceph.mds.h.smithi189.stdout:Mar 24 00:22:33 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-h[70087]: 7: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x563aa09365d6] 2024-03-24T00:22:33.392 INFO:journalctl@ceph.mds.h.smithi189.stdout:Mar 24 00:22:33 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-h[70087]: 8: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x563aa0936da1] 2024-03-24T00:22:33.392 INFO:journalctl@ceph.mds.h.smithi189.stdout:Mar 24 00:22:33 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-h[70087]: 9: (Server::handle_client_rename(boost::intrusive_ptr const&)+0x259d) [0x563aa097397d] 2024-03-24T00:22:33.392 INFO:journalctl@ceph.mds.h.smithi189.stdout:Mar 24 00:22:33 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-h[70087]: 10: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x563aa0989233] 2024-03-24T00:22:33.392 INFO:journalctl@ceph.mds.h.smithi189.stdout:Mar 24 00:22:33 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-h[70087]: 11: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x563aa098b197] 2024-03-24T00:22:33.392 INFO:journalctl@ceph.mds.h.smithi189.stdout:Mar 24 00:22:33 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-h[70087]: 12: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x563aa088a11b] 2024-03-24T00:22:33.392 INFO:journalctl@ceph.mds.h.smithi189.stdout:Mar 24 00:22:33 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-h[70087]: 13: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x563aa088a6bc] 2024-03-24T00:22:33.392 INFO:journalctl@ceph.mds.h.smithi189.stdout:Mar 24 00:22:33 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-h[70087]: 14: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x563aa08741ff] 2024-03-24T00:22:33.392 INFO:journalctl@ceph.mds.h.smithi189.stdout:Mar 24 00:22:33 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-h[70087]: 15: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7fe0e076f488] 2024-03-24T00:22:33.392 INFO:journalctl@ceph.mds.h.smithi189.stdout:Mar 24 00:22:33 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-h[70087]: 16: (DispatchQueue::entry()+0x51f) [0x7fe0e076c60f] 2024-03-24T00:22:33.392 INFO:journalctl@ceph.mds.h.smithi189.stdout:Mar 24 00:22:33 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-h[70087]: 17: (DispatchQueue::DispatchThread::entry()+0x11) [0x7fe0e083fe81] 2024-03-24T00:22:33.392 INFO:journalctl@ceph.mds.h.smithi189.stdout:Mar 24 00:22:33 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-h[70087]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7fe0df25a1ca] 2024-03-24T00:22:33.392 INFO:journalctl@ceph.mds.h.smithi189.stdout:Mar 24 00:22:33 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-h[70087]: 19: clone() 2024-03-24T00:22:33.393 INFO:journalctl@ceph.mds.h.smithi189.stdout:Mar 24 00:22:33 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-h[70087]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-03-24T00:22:33.393 INFO:journalctl@ceph.mds.h.smithi189.stdout:Mar 24 00:22:33 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-h[70087]: 2024-03-24T00:22:33.393 INFO:journalctl@ceph.mds.h.smithi189.stdout:Mar 24 00:22:33 smithi189 podman[70138]: 2024-03-24 00:22:33.065857464 +0000 UTC m=+0.030485810 container died dca63437cc7c74c655e371800e91eb29b294536dd147dea3f0df3cd301a8f382 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-h, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, CEPH_POINT_RELEASE=, GIT_CLEAN=True, io.buildah.version=1.33.5, org.label-schema.build-date=20240319, org.label-schema.schema-version=1.0) 2024-03-24T00:22:33.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:22:32 smithi193 ceph-mon[36878]: mds.? [v2:172.21.15.193:6832/2424695757,v1:172.21.15.193:6833/2424695757] up:rejoin 2024-03-24T00:22:33.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:22:32 smithi193 ceph-mon[36878]: mds.? [v2:172.21.15.189:6836/75274228,v1:172.21.15.189:6837/75274228] up:rejoin 2024-03-24T00:22:33.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:22:32 smithi193 ceph-mon[36878]: fsmap cephfs:5/5 3 up:standby-replay 3 up:standby 2 up:rejoin 3 up:active 2024-03-24T00:22:33.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:22:32 smithi193 ceph-mon[36878]: daemon mds.h is now active in filesystem cephfs as rank 3 2024-03-24T00:22:33.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:22:32 smithi193 ceph-mon[36878]: daemon mds.l is now active in filesystem cephfs as rank 1 2024-03-24T00:22:33.461 INFO:journalctl@ceph.mds.l.smithi193.stdout:Mar 24 00:22:33 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-l[72473]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: In function 'void Locker::set_xlocks_done(MutationImpl*, bool)' thread 7f508bbd4700 time 2024-03-24T00:22:33.039800+0000 2024-03-24T00:22:33.461 INFO:journalctl@ceph.mds.l.smithi193.stdout:Mar 24 00:22:33 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-l[72473]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: 626: FAILED ceph_assert(obj->is_auth()) 2024-03-24T00:22:33.462 INFO:journalctl@ceph.mds.l.smithi193.stdout:Mar 24 00:22:33 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-l[72473]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-24T00:22:33.462 INFO:journalctl@ceph.mds.l.smithi193.stdout:Mar 24 00:22:33 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-l[72473]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x135) [0x7f50961fe4a1] 2024-03-24T00:22:33.462 INFO:journalctl@ceph.mds.l.smithi193.stdout:Mar 24 00:22:33 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-l[72473]: 2: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7f50961fe667] 2024-03-24T00:22:33.462 INFO:journalctl@ceph.mds.l.smithi193.stdout:Mar 24 00:22:33 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-l[72473]: 3: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x55770b99c223] 2024-03-24T00:22:33.462 INFO:journalctl@ceph.mds.l.smithi193.stdout:Mar 24 00:22:33 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-l[72473]: 4: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x55770b8165d6] 2024-03-24T00:22:33.462 INFO:journalctl@ceph.mds.l.smithi193.stdout:Mar 24 00:22:33 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-l[72473]: 5: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x55770b816da1] 2024-03-24T00:22:33.462 INFO:journalctl@ceph.mds.l.smithi193.stdout:Mar 24 00:22:33 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-l[72473]: 6: (Server::handle_client_rename(boost::intrusive_ptr const&)+0x259d) [0x55770b85397d] 2024-03-24T00:22:33.462 INFO:journalctl@ceph.mds.l.smithi193.stdout:Mar 24 00:22:33 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-l[72473]: 7: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x55770b869233] 2024-03-24T00:22:33.462 INFO:journalctl@ceph.mds.l.smithi193.stdout:Mar 24 00:22:33 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-l[72473]: 8: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x55770b86b197] 2024-03-24T00:22:33.462 INFO:journalctl@ceph.mds.l.smithi193.stdout:Mar 24 00:22:33 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-l[72473]: 9: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x55770b76a11b] 2024-03-24T00:22:33.462 INFO:journalctl@ceph.mds.l.smithi193.stdout:Mar 24 00:22:33 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-l[72473]: 10: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x55770b76a6bc] 2024-03-24T00:22:33.462 INFO:journalctl@ceph.mds.l.smithi193.stdout:Mar 24 00:22:33 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-l[72473]: 11: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x55770b7541ff] 2024-03-24T00:22:33.462 INFO:journalctl@ceph.mds.l.smithi193.stdout:Mar 24 00:22:33 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-l[72473]: 12: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7f5096490488] 2024-03-24T00:22:33.463 INFO:journalctl@ceph.mds.l.smithi193.stdout:Mar 24 00:22:33 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-l[72473]: 13: (DispatchQueue::entry()+0x51f) [0x7f509648d60f] 2024-03-24T00:22:33.463 INFO:journalctl@ceph.mds.l.smithi193.stdout:Mar 24 00:22:33 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-l[72473]: 14: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f5096560e81] 2024-03-24T00:22:33.463 INFO:journalctl@ceph.mds.l.smithi193.stdout:Mar 24 00:22:33 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-l[72473]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7f5094f7b1ca] 2024-03-24T00:22:33.463 INFO:journalctl@ceph.mds.l.smithi193.stdout:Mar 24 00:22:33 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-l[72473]: 16: clone() 2024-03-24T00:22:33.463 INFO:journalctl@ceph.mds.l.smithi193.stdout:Mar 24 00:22:33 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-l[72473]: 2024-03-24T00:22:33.040+0000 7f508bbd4700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: In function 'void Locker::set_xlocks_done(MutationImpl*, bool)' thread 7f508bbd4700 time 2024-03-24T00:22:33.039800+0000 2024-03-24T00:22:33.463 INFO:journalctl@ceph.mds.l.smithi193.stdout:Mar 24 00:22:33 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-l[72473]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: 626: FAILED ceph_assert(obj->is_auth()) 2024-03-24T00:22:33.463 INFO:journalctl@ceph.mds.l.smithi193.stdout:Mar 24 00:22:33 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-l[72473]: 2024-03-24T00:22:33.464 INFO:journalctl@ceph.mds.l.smithi193.stdout:Mar 24 00:22:33 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-l[72473]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-24T00:22:33.464 INFO:journalctl@ceph.mds.l.smithi193.stdout:Mar 24 00:22:33 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-l[72473]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x135) [0x7f50961fe4a1] 2024-03-24T00:22:33.464 INFO:journalctl@ceph.mds.l.smithi193.stdout:Mar 24 00:22:33 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-l[72473]: 2: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7f50961fe667] 2024-03-24T00:22:33.464 INFO:journalctl@ceph.mds.l.smithi193.stdout:Mar 24 00:22:33 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-l[72473]: 3: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x55770b99c223] 2024-03-24T00:22:33.464 INFO:journalctl@ceph.mds.l.smithi193.stdout:Mar 24 00:22:33 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-l[72473]: 4: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x55770b8165d6] 2024-03-24T00:22:33.464 INFO:journalctl@ceph.mds.l.smithi193.stdout:Mar 24 00:22:33 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-l[72473]: 5: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x55770b816da1] 2024-03-24T00:22:33.464 INFO:journalctl@ceph.mds.l.smithi193.stdout:Mar 24 00:22:33 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-l[72473]: 6: (Server::handle_client_rename(boost::intrusive_ptr const&)+0x259d) [0x55770b85397d] 2024-03-24T00:22:33.464 INFO:journalctl@ceph.mds.l.smithi193.stdout:Mar 24 00:22:33 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-l[72473]: 7: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x55770b869233] 2024-03-24T00:22:33.464 INFO:journalctl@ceph.mds.l.smithi193.stdout:Mar 24 00:22:33 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-l[72473]: 8: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x55770b86b197] 2024-03-24T00:22:33.464 INFO:journalctl@ceph.mds.l.smithi193.stdout:Mar 24 00:22:33 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-l[72473]: 9: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x55770b76a11b] 2024-03-24T00:22:33.464 INFO:journalctl@ceph.mds.l.smithi193.stdout:Mar 24 00:22:33 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-l[72473]: 10: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x55770b76a6bc] 2024-03-24T00:22:33.464 INFO:journalctl@ceph.mds.l.smithi193.stdout:Mar 24 00:22:33 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-l[72473]: 11: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x55770b7541ff] 2024-03-24T00:22:33.464 INFO:journalctl@ceph.mds.l.smithi193.stdout:Mar 24 00:22:33 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-l[72473]: 12: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7f5096490488] 2024-03-24T00:22:33.464 INFO:journalctl@ceph.mds.l.smithi193.stdout:Mar 24 00:22:33 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-l[72473]: 13: (DispatchQueue::entry()+0x51f) [0x7f509648d60f] 2024-03-24T00:22:33.464 INFO:journalctl@ceph.mds.l.smithi193.stdout:Mar 24 00:22:33 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-l[72473]: 14: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f5096560e81] 2024-03-24T00:22:33.465 INFO:journalctl@ceph.mds.l.smithi193.stdout:Mar 24 00:22:33 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-l[72473]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7f5094f7b1ca] 2024-03-24T00:22:33.465 INFO:journalctl@ceph.mds.l.smithi193.stdout:Mar 24 00:22:33 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-l[72473]: 16: clone() 2024-03-24T00:22:33.465 INFO:journalctl@ceph.mds.l.smithi193.stdout:Mar 24 00:22:33 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-l[72473]: 2024-03-24T00:22:33.465 INFO:journalctl@ceph.mds.l.smithi193.stdout:Mar 24 00:22:33 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-l[72473]: *** Caught signal (Aborted) ** 2024-03-24T00:22:33.465 INFO:journalctl@ceph.mds.l.smithi193.stdout:Mar 24 00:22:33 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-l[72473]: in thread 7f508bbd4700 thread_name:ms_dispatch 2024-03-24T00:22:33.465 INFO:journalctl@ceph.mds.l.smithi193.stdout:Mar 24 00:22:33 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-l[72473]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-24T00:22:33.465 INFO:journalctl@ceph.mds.l.smithi193.stdout:Mar 24 00:22:33 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-l[72473]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f5094f85d20] 2024-03-24T00:22:33.465 INFO:journalctl@ceph.mds.l.smithi193.stdout:Mar 24 00:22:33 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-l[72473]: 2: gsignal() 2024-03-24T00:22:33.465 INFO:journalctl@ceph.mds.l.smithi193.stdout:Mar 24 00:22:33 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-l[72473]: 3: abort() 2024-03-24T00:22:33.465 INFO:journalctl@ceph.mds.l.smithi193.stdout:Mar 24 00:22:33 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-l[72473]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x18f) [0x7f50961fe4fb] 2024-03-24T00:22:33.465 INFO:journalctl@ceph.mds.l.smithi193.stdout:Mar 24 00:22:33 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-l[72473]: 5: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7f50961fe667] 2024-03-24T00:22:33.465 INFO:journalctl@ceph.mds.l.smithi193.stdout:Mar 24 00:22:33 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-l[72473]: 6: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x55770b99c223] 2024-03-24T00:22:33.465 INFO:journalctl@ceph.mds.l.smithi193.stdout:Mar 24 00:22:33 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-l[72473]: 7: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x55770b8165d6] 2024-03-24T00:22:33.465 INFO:journalctl@ceph.mds.l.smithi193.stdout:Mar 24 00:22:33 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-l[72473]: 8: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x55770b816da1] 2024-03-24T00:22:33.465 INFO:journalctl@ceph.mds.l.smithi193.stdout:Mar 24 00:22:33 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-l[72473]: 9: (Server::handle_client_rename(boost::intrusive_ptr const&)+0x259d) [0x55770b85397d] 2024-03-24T00:22:33.466 INFO:journalctl@ceph.mds.l.smithi193.stdout:Mar 24 00:22:33 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-l[72473]: 10: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x55770b869233] 2024-03-24T00:22:33.466 INFO:journalctl@ceph.mds.l.smithi193.stdout:Mar 24 00:22:33 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-l[72473]: 11: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x55770b86b197] 2024-03-24T00:22:33.466 INFO:journalctl@ceph.mds.l.smithi193.stdout:Mar 24 00:22:33 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-l[72473]: 12: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x55770b76a11b] 2024-03-24T00:22:33.466 INFO:journalctl@ceph.mds.l.smithi193.stdout:Mar 24 00:22:33 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-l[72473]: 13: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x55770b76a6bc] 2024-03-24T00:22:33.466 INFO:journalctl@ceph.mds.l.smithi193.stdout:Mar 24 00:22:33 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-l[72473]: 14: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x55770b7541ff] 2024-03-24T00:22:33.466 INFO:journalctl@ceph.mds.l.smithi193.stdout:Mar 24 00:22:33 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-l[72473]: 15: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7f5096490488] 2024-03-24T00:22:33.466 INFO:journalctl@ceph.mds.l.smithi193.stdout:Mar 24 00:22:33 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-l[72473]: 16: (DispatchQueue::entry()+0x51f) [0x7f509648d60f] 2024-03-24T00:22:33.466 INFO:journalctl@ceph.mds.l.smithi193.stdout:Mar 24 00:22:33 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-l[72473]: 17: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f5096560e81] 2024-03-24T00:22:33.466 INFO:journalctl@ceph.mds.l.smithi193.stdout:Mar 24 00:22:33 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-l[72473]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7f5094f7b1ca] 2024-03-24T00:22:33.466 INFO:journalctl@ceph.mds.l.smithi193.stdout:Mar 24 00:22:33 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-l[72473]: 19: clone() 2024-03-24T00:22:33.466 INFO:journalctl@ceph.mds.l.smithi193.stdout:Mar 24 00:22:33 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-l[72473]: 2024-03-24T00:22:33.042+0000 7f508bbd4700 -1 *** Caught signal (Aborted) ** 2024-03-24T00:22:33.466 INFO:journalctl@ceph.mds.l.smithi193.stdout:Mar 24 00:22:33 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-l[72473]: in thread 7f508bbd4700 thread_name:ms_dispatch 2024-03-24T00:22:33.466 INFO:journalctl@ceph.mds.l.smithi193.stdout:Mar 24 00:22:33 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-l[72473]: 2024-03-24T00:22:33.466 INFO:journalctl@ceph.mds.l.smithi193.stdout:Mar 24 00:22:33 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-l[72473]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-24T00:22:33.467 INFO:journalctl@ceph.mds.l.smithi193.stdout:Mar 24 00:22:33 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-l[72473]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f5094f85d20] 2024-03-24T00:22:33.467 INFO:journalctl@ceph.mds.l.smithi193.stdout:Mar 24 00:22:33 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-l[72473]: 2: gsignal() 2024-03-24T00:22:33.467 INFO:journalctl@ceph.mds.l.smithi193.stdout:Mar 24 00:22:33 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-l[72473]: 3: abort() 2024-03-24T00:22:33.467 INFO:journalctl@ceph.mds.l.smithi193.stdout:Mar 24 00:22:33 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-l[72473]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x18f) [0x7f50961fe4fb] 2024-03-24T00:22:33.467 INFO:journalctl@ceph.mds.l.smithi193.stdout:Mar 24 00:22:33 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-l[72473]: 5: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7f50961fe667] 2024-03-24T00:22:33.468 INFO:journalctl@ceph.mds.l.smithi193.stdout:Mar 24 00:22:33 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-l[72473]: 6: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x55770b99c223] 2024-03-24T00:22:33.468 INFO:journalctl@ceph.mds.l.smithi193.stdout:Mar 24 00:22:33 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-l[72473]: 7: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x55770b8165d6] 2024-03-24T00:22:33.468 INFO:journalctl@ceph.mds.l.smithi193.stdout:Mar 24 00:22:33 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-l[72473]: 8: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x55770b816da1] 2024-03-24T00:22:33.468 INFO:journalctl@ceph.mds.l.smithi193.stdout:Mar 24 00:22:33 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-l[72473]: 9: (Server::handle_client_rename(boost::intrusive_ptr const&)+0x259d) [0x55770b85397d] 2024-03-24T00:22:33.468 INFO:journalctl@ceph.mds.l.smithi193.stdout:Mar 24 00:22:33 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-l[72473]: 10: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x55770b869233] 2024-03-24T00:22:33.468 INFO:journalctl@ceph.mds.l.smithi193.stdout:Mar 24 00:22:33 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-l[72473]: 11: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x55770b86b197] 2024-03-24T00:22:33.468 INFO:journalctl@ceph.mds.l.smithi193.stdout:Mar 24 00:22:33 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-l[72473]: 12: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x55770b76a11b] 2024-03-24T00:22:33.468 INFO:journalctl@ceph.mds.l.smithi193.stdout:Mar 24 00:22:33 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-l[72473]: 13: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x55770b76a6bc] 2024-03-24T00:22:33.468 INFO:journalctl@ceph.mds.l.smithi193.stdout:Mar 24 00:22:33 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-l[72473]: 14: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x55770b7541ff] 2024-03-24T00:22:33.468 INFO:journalctl@ceph.mds.l.smithi193.stdout:Mar 24 00:22:33 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-l[72473]: 15: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7f5096490488] 2024-03-24T00:22:33.468 INFO:journalctl@ceph.mds.l.smithi193.stdout:Mar 24 00:22:33 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-l[72473]: 16: (DispatchQueue::entry()+0x51f) [0x7f509648d60f] 2024-03-24T00:22:33.468 INFO:journalctl@ceph.mds.l.smithi193.stdout:Mar 24 00:22:33 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-l[72473]: 17: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f5096560e81] 2024-03-24T00:22:33.469 INFO:journalctl@ceph.mds.l.smithi193.stdout:Mar 24 00:22:33 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-l[72473]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7f5094f7b1ca] 2024-03-24T00:22:33.469 INFO:journalctl@ceph.mds.l.smithi193.stdout:Mar 24 00:22:33 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-l[72473]: 19: clone() 2024-03-24T00:22:33.469 INFO:journalctl@ceph.mds.l.smithi193.stdout:Mar 24 00:22:33 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-l[72473]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-03-24T00:22:33.469 INFO:journalctl@ceph.mds.l.smithi193.stdout:Mar 24 00:22:33 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-l[72473]: 2024-03-24T00:22:33.469 INFO:journalctl@ceph.mds.l.smithi193.stdout:Mar 24 00:22:33 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-l[72473]: -1> 2024-03-24T00:22:33.040+0000 7f508bbd4700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: In function 'void Locker::set_xlocks_done(MutationImpl*, bool)' thread 7f508bbd4700 time 2024-03-24T00:22:33.039800+0000 2024-03-24T00:22:33.469 INFO:journalctl@ceph.mds.l.smithi193.stdout:Mar 24 00:22:33 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-l[72473]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: 626: FAILED ceph_assert(obj->is_auth()) 2024-03-24T00:22:33.469 INFO:journalctl@ceph.mds.l.smithi193.stdout:Mar 24 00:22:33 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-l[72473]: 2024-03-24T00:22:33.469 INFO:journalctl@ceph.mds.l.smithi193.stdout:Mar 24 00:22:33 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-l[72473]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-24T00:22:33.469 INFO:journalctl@ceph.mds.l.smithi193.stdout:Mar 24 00:22:33 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-l[72473]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x135) [0x7f50961fe4a1] 2024-03-24T00:22:33.469 INFO:journalctl@ceph.mds.l.smithi193.stdout:Mar 24 00:22:33 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-l[72473]: 2: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7f50961fe667] 2024-03-24T00:22:33.469 INFO:journalctl@ceph.mds.l.smithi193.stdout:Mar 24 00:22:33 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-l[72473]: 3: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x55770b99c223] 2024-03-24T00:22:33.469 INFO:journalctl@ceph.mds.l.smithi193.stdout:Mar 24 00:22:33 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-l[72473]: 4: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x55770b8165d6] 2024-03-24T00:22:33.469 INFO:journalctl@ceph.mds.l.smithi193.stdout:Mar 24 00:22:33 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-l[72473]: 5: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x55770b816da1] 2024-03-24T00:22:33.469 INFO:journalctl@ceph.mds.l.smithi193.stdout:Mar 24 00:22:33 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-l[72473]: 6: (Server::handle_client_rename(boost::intrusive_ptr const&)+0x259d) [0x55770b85397d] 2024-03-24T00:22:33.470 INFO:journalctl@ceph.mds.l.smithi193.stdout:Mar 24 00:22:33 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-l[72473]: 7: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x55770b869233] 2024-03-24T00:22:33.470 INFO:journalctl@ceph.mds.l.smithi193.stdout:Mar 24 00:22:33 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-l[72473]: 8: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x55770b86b197] 2024-03-24T00:22:33.470 INFO:journalctl@ceph.mds.l.smithi193.stdout:Mar 24 00:22:33 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-l[72473]: 9: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x55770b76a11b] 2024-03-24T00:22:33.470 INFO:journalctl@ceph.mds.l.smithi193.stdout:Mar 24 00:22:33 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-l[72473]: 10: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x55770b76a6bc] 2024-03-24T00:22:33.470 INFO:journalctl@ceph.mds.l.smithi193.stdout:Mar 24 00:22:33 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-l[72473]: 11: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x55770b7541ff] 2024-03-24T00:22:33.470 INFO:journalctl@ceph.mds.l.smithi193.stdout:Mar 24 00:22:33 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-l[72473]: 12: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7f5096490488] 2024-03-24T00:22:33.470 INFO:journalctl@ceph.mds.l.smithi193.stdout:Mar 24 00:22:33 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-l[72473]: 13: (DispatchQueue::entry()+0x51f) [0x7f509648d60f] 2024-03-24T00:22:33.470 INFO:journalctl@ceph.mds.l.smithi193.stdout:Mar 24 00:22:33 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-l[72473]: 14: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f5096560e81] 2024-03-24T00:22:33.470 INFO:journalctl@ceph.mds.l.smithi193.stdout:Mar 24 00:22:33 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-l[72473]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7f5094f7b1ca] 2024-03-24T00:22:33.470 INFO:journalctl@ceph.mds.l.smithi193.stdout:Mar 24 00:22:33 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-l[72473]: 16: clone() 2024-03-24T00:22:33.470 INFO:journalctl@ceph.mds.l.smithi193.stdout:Mar 24 00:22:33 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-l[72473]: 2024-03-24T00:22:33.470 INFO:journalctl@ceph.mds.l.smithi193.stdout:Mar 24 00:22:33 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-l[72473]: 0> 2024-03-24T00:22:33.042+0000 7f508bbd4700 -1 *** Caught signal (Aborted) ** 2024-03-24T00:22:33.470 INFO:journalctl@ceph.mds.l.smithi193.stdout:Mar 24 00:22:33 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-l[72473]: in thread 7f508bbd4700 thread_name:ms_dispatch 2024-03-24T00:22:33.470 INFO:journalctl@ceph.mds.l.smithi193.stdout:Mar 24 00:22:33 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-l[72473]: 2024-03-24T00:22:33.471 INFO:journalctl@ceph.mds.l.smithi193.stdout:Mar 24 00:22:33 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-l[72473]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-24T00:22:33.471 INFO:journalctl@ceph.mds.l.smithi193.stdout:Mar 24 00:22:33 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-l[72473]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f5094f85d20] 2024-03-24T00:22:33.471 INFO:journalctl@ceph.mds.l.smithi193.stdout:Mar 24 00:22:33 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-l[72473]: 2: gsignal() 2024-03-24T00:22:33.471 INFO:journalctl@ceph.mds.l.smithi193.stdout:Mar 24 00:22:33 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-l[72473]: 3: abort() 2024-03-24T00:22:33.471 INFO:journalctl@ceph.mds.l.smithi193.stdout:Mar 24 00:22:33 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-l[72473]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x18f) [0x7f50961fe4fb] 2024-03-24T00:22:33.471 INFO:journalctl@ceph.mds.l.smithi193.stdout:Mar 24 00:22:33 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-l[72473]: 5: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7f50961fe667] 2024-03-24T00:22:33.471 INFO:journalctl@ceph.mds.l.smithi193.stdout:Mar 24 00:22:33 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-l[72473]: 6: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x55770b99c223] 2024-03-24T00:22:33.471 INFO:journalctl@ceph.mds.l.smithi193.stdout:Mar 24 00:22:33 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-l[72473]: 7: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x55770b8165d6] 2024-03-24T00:22:33.472 INFO:journalctl@ceph.mds.l.smithi193.stdout:Mar 24 00:22:33 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-l[72473]: 8: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x55770b816da1] 2024-03-24T00:22:33.472 INFO:journalctl@ceph.mds.l.smithi193.stdout:Mar 24 00:22:33 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-l[72473]: 9: (Server::handle_client_rename(boost::intrusive_ptr const&)+0x259d) [0x55770b85397d] 2024-03-24T00:22:33.472 INFO:journalctl@ceph.mds.l.smithi193.stdout:Mar 24 00:22:33 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-l[72473]: 10: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x55770b869233] 2024-03-24T00:22:33.472 INFO:journalctl@ceph.mds.l.smithi193.stdout:Mar 24 00:22:33 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-l[72473]: 11: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x55770b86b197] 2024-03-24T00:22:33.472 INFO:journalctl@ceph.mds.l.smithi193.stdout:Mar 24 00:22:33 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-l[72473]: 12: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x55770b76a11b] 2024-03-24T00:22:33.472 INFO:journalctl@ceph.mds.l.smithi193.stdout:Mar 24 00:22:33 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-l[72473]: 13: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x55770b76a6bc] 2024-03-24T00:22:33.472 INFO:journalctl@ceph.mds.l.smithi193.stdout:Mar 24 00:22:33 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-l[72473]: 14: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x55770b7541ff] 2024-03-24T00:22:33.472 INFO:journalctl@ceph.mds.l.smithi193.stdout:Mar 24 00:22:33 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-l[72473]: 15: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7f5096490488] 2024-03-24T00:22:33.472 INFO:journalctl@ceph.mds.l.smithi193.stdout:Mar 24 00:22:33 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-l[72473]: 16: (DispatchQueue::entry()+0x51f) [0x7f509648d60f] 2024-03-24T00:22:33.472 INFO:journalctl@ceph.mds.l.smithi193.stdout:Mar 24 00:22:33 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-l[72473]: 17: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f5096560e81] 2024-03-24T00:22:33.472 INFO:journalctl@ceph.mds.l.smithi193.stdout:Mar 24 00:22:33 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-l[72473]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7f5094f7b1ca] 2024-03-24T00:22:33.472 INFO:journalctl@ceph.mds.l.smithi193.stdout:Mar 24 00:22:33 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-l[72473]: 19: clone() 2024-03-24T00:22:33.472 INFO:journalctl@ceph.mds.l.smithi193.stdout:Mar 24 00:22:33 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-l[72473]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-03-24T00:22:33.472 INFO:journalctl@ceph.mds.l.smithi193.stdout:Mar 24 00:22:33 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-l[72473]: 2024-03-24T00:22:33.473 INFO:journalctl@ceph.mds.l.smithi193.stdout:Mar 24 00:22:33 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-l[72473]: -1> 2024-03-24T00:22:33.040+0000 7f508bbd4700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: In function 'void Locker::set_xlocks_done(MutationImpl*, bool)' thread 7f508bbd4700 time 2024-03-24T00:22:33.039800+0000 2024-03-24T00:22:33.473 INFO:journalctl@ceph.mds.l.smithi193.stdout:Mar 24 00:22:33 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-l[72473]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: 626: FAILED ceph_assert(obj->is_auth()) 2024-03-24T00:22:33.473 INFO:journalctl@ceph.mds.l.smithi193.stdout:Mar 24 00:22:33 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-l[72473]: 2024-03-24T00:22:33.473 INFO:journalctl@ceph.mds.l.smithi193.stdout:Mar 24 00:22:33 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-l[72473]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-24T00:22:33.473 INFO:journalctl@ceph.mds.l.smithi193.stdout:Mar 24 00:22:33 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-l[72473]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x135) [0x7f50961fe4a1] 2024-03-24T00:22:33.473 INFO:journalctl@ceph.mds.l.smithi193.stdout:Mar 24 00:22:33 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-l[72473]: 2: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7f50961fe667] 2024-03-24T00:22:33.473 INFO:journalctl@ceph.mds.l.smithi193.stdout:Mar 24 00:22:33 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-l[72473]: 3: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x55770b99c223] 2024-03-24T00:22:33.473 INFO:journalctl@ceph.mds.l.smithi193.stdout:Mar 24 00:22:33 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-l[72473]: 4: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x55770b8165d6] 2024-03-24T00:22:33.473 INFO:journalctl@ceph.mds.l.smithi193.stdout:Mar 24 00:22:33 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-l[72473]: 5: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x55770b816da1] 2024-03-24T00:22:33.473 INFO:journalctl@ceph.mds.l.smithi193.stdout:Mar 24 00:22:33 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-l[72473]: 6: (Server::handle_client_rename(boost::intrusive_ptr const&)+0x259d) [0x55770b85397d] 2024-03-24T00:22:33.473 INFO:journalctl@ceph.mds.l.smithi193.stdout:Mar 24 00:22:33 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-l[72473]: 7: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x55770b869233] 2024-03-24T00:22:33.473 INFO:journalctl@ceph.mds.l.smithi193.stdout:Mar 24 00:22:33 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-l[72473]: 8: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x55770b86b197] 2024-03-24T00:22:33.473 INFO:journalctl@ceph.mds.l.smithi193.stdout:Mar 24 00:22:33 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-l[72473]: 9: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x55770b76a11b] 2024-03-24T00:22:33.473 INFO:journalctl@ceph.mds.l.smithi193.stdout:Mar 24 00:22:33 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-l[72473]: 10: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x55770b76a6bc] 2024-03-24T00:22:33.474 INFO:journalctl@ceph.mds.l.smithi193.stdout:Mar 24 00:22:33 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-l[72473]: 11: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x55770b7541ff] 2024-03-24T00:22:33.474 INFO:journalctl@ceph.mds.l.smithi193.stdout:Mar 24 00:22:33 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-l[72473]: 12: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7f5096490488] 2024-03-24T00:22:33.474 INFO:journalctl@ceph.mds.l.smithi193.stdout:Mar 24 00:22:33 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-l[72473]: 13: (DispatchQueue::entry()+0x51f) [0x7f509648d60f] 2024-03-24T00:22:33.474 INFO:journalctl@ceph.mds.l.smithi193.stdout:Mar 24 00:22:33 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-l[72473]: 14: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f5096560e81] 2024-03-24T00:22:33.474 INFO:journalctl@ceph.mds.l.smithi193.stdout:Mar 24 00:22:33 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-l[72473]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7f5094f7b1ca] 2024-03-24T00:22:33.474 INFO:journalctl@ceph.mds.l.smithi193.stdout:Mar 24 00:22:33 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-l[72473]: 16: clone() 2024-03-24T00:22:33.474 INFO:journalctl@ceph.mds.l.smithi193.stdout:Mar 24 00:22:33 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-l[72473]: 2024-03-24T00:22:33.474 INFO:journalctl@ceph.mds.l.smithi193.stdout:Mar 24 00:22:33 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-l[72473]: 0> 2024-03-24T00:22:33.042+0000 7f508bbd4700 -1 *** Caught signal (Aborted) ** 2024-03-24T00:22:33.474 INFO:journalctl@ceph.mds.l.smithi193.stdout:Mar 24 00:22:33 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-l[72473]: in thread 7f508bbd4700 thread_name:ms_dispatch 2024-03-24T00:22:33.474 INFO:journalctl@ceph.mds.l.smithi193.stdout:Mar 24 00:22:33 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-l[72473]: 2024-03-24T00:22:33.474 INFO:journalctl@ceph.mds.l.smithi193.stdout:Mar 24 00:22:33 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-l[72473]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-24T00:22:33.474 INFO:journalctl@ceph.mds.l.smithi193.stdout:Mar 24 00:22:33 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-l[72473]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f5094f85d20] 2024-03-24T00:22:33.474 INFO:journalctl@ceph.mds.l.smithi193.stdout:Mar 24 00:22:33 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-l[72473]: 2: gsignal() 2024-03-24T00:22:33.474 INFO:journalctl@ceph.mds.l.smithi193.stdout:Mar 24 00:22:33 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-l[72473]: 3: abort() 2024-03-24T00:22:33.474 INFO:journalctl@ceph.mds.l.smithi193.stdout:Mar 24 00:22:33 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-l[72473]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x18f) [0x7f50961fe4fb] 2024-03-24T00:22:33.475 INFO:journalctl@ceph.mds.l.smithi193.stdout:Mar 24 00:22:33 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-l[72473]: 5: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7f50961fe667] 2024-03-24T00:22:33.475 INFO:journalctl@ceph.mds.l.smithi193.stdout:Mar 24 00:22:33 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-l[72473]: 6: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x55770b99c223] 2024-03-24T00:22:33.475 INFO:journalctl@ceph.mds.l.smithi193.stdout:Mar 24 00:22:33 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-l[72473]: 7: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x55770b8165d6] 2024-03-24T00:22:33.475 INFO:journalctl@ceph.mds.l.smithi193.stdout:Mar 24 00:22:33 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-l[72473]: 8: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x55770b816da1] 2024-03-24T00:22:33.475 INFO:journalctl@ceph.mds.l.smithi193.stdout:Mar 24 00:22:33 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-l[72473]: 9: (Server::handle_client_rename(boost::intrusive_ptr const&)+0x259d) [0x55770b85397d] 2024-03-24T00:22:33.475 INFO:journalctl@ceph.mds.l.smithi193.stdout:Mar 24 00:22:33 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-l[72473]: 10: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x55770b869233] 2024-03-24T00:22:33.475 INFO:journalctl@ceph.mds.l.smithi193.stdout:Mar 24 00:22:33 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-l[72473]: 11: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x55770b86b197] 2024-03-24T00:22:33.475 INFO:journalctl@ceph.mds.l.smithi193.stdout:Mar 24 00:22:33 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-l[72473]: 12: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x55770b76a11b] 2024-03-24T00:22:33.475 INFO:journalctl@ceph.mds.l.smithi193.stdout:Mar 24 00:22:33 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-l[72473]: 13: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x55770b76a6bc] 2024-03-24T00:22:33.475 INFO:journalctl@ceph.mds.l.smithi193.stdout:Mar 24 00:22:33 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-l[72473]: 14: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x55770b7541ff] 2024-03-24T00:22:33.475 INFO:journalctl@ceph.mds.l.smithi193.stdout:Mar 24 00:22:33 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-l[72473]: 15: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7f5096490488] 2024-03-24T00:22:33.475 INFO:journalctl@ceph.mds.l.smithi193.stdout:Mar 24 00:22:33 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-l[72473]: 16: (DispatchQueue::entry()+0x51f) [0x7f509648d60f] 2024-03-24T00:22:33.475 INFO:journalctl@ceph.mds.l.smithi193.stdout:Mar 24 00:22:33 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-l[72473]: 17: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f5096560e81] 2024-03-24T00:22:33.475 INFO:journalctl@ceph.mds.l.smithi193.stdout:Mar 24 00:22:33 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-l[72473]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7f5094f7b1ca] 2024-03-24T00:22:33.476 INFO:journalctl@ceph.mds.l.smithi193.stdout:Mar 24 00:22:33 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-l[72473]: 19: clone() 2024-03-24T00:22:33.476 INFO:journalctl@ceph.mds.l.smithi193.stdout:Mar 24 00:22:33 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-l[72473]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-03-24T00:22:33.476 INFO:journalctl@ceph.mds.l.smithi193.stdout:Mar 24 00:22:33 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-l[72473]: 2024-03-24T00:22:33.476 INFO:journalctl@ceph.mds.l.smithi193.stdout:Mar 24 00:22:33 smithi193 podman[72569]: 2024-03-24 00:22:33.10677017 +0000 UTC m=+0.021044911 container died 34b63e7de437dbf746219f051fd02d843764da00bec1829324daf12d71db4db5 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-l, org.label-schema.vendor=CentOS, ceph=True, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.build-date=20240319, maintainer=Guillaume Abrioux , GIT_CLEAN=True, GIT_BRANCH=HEAD, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, org.label-schema.schema-version=1.0) 2024-03-24T00:22:33.982 INFO:journalctl@ceph.mds.h.smithi189.stdout:Mar 24 00:22:33 smithi189 podman[70138]: 2024-03-24 00:22:33.719796736 +0000 UTC m=+0.684425070 container remove dca63437cc7c74c655e371800e91eb29b294536dd147dea3f0df3cd301a8f382 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-h, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240319, org.label-schema.schema-version=1.0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, ceph=True, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD) 2024-03-24T00:22:33.982 INFO:journalctl@ceph.mds.h.smithi189.stdout:Mar 24 00:22:33 smithi189 systemd[1]: ceph-d79a5420-e971-11ee-95cd-87774f69a715@mds.h.service: Main process exited, code=exited, status=134/n/a 2024-03-24T00:22:34.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:22:33 smithi118 ceph-mon[31535]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-03-24T00:22:34.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:22:33 smithi118 ceph-mon[31535]: Cluster is now healthy 2024-03-24T00:22:34.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:22:33 smithi118 ceph-mon[31535]: mds.? [v2:172.21.15.189:6836/75274228,v1:172.21.15.189:6837/75274228] up:active 2024-03-24T00:22:34.197 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:22:33 smithi118 ceph-mon[31535]: mds.? [v2:172.21.15.193:6832/2424695757,v1:172.21.15.193:6833/2424695757] up:active 2024-03-24T00:22:34.197 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:22:33 smithi118 ceph-mon[31535]: fsmap cephfs:5 3 up:standby-replay 3 up:standby 5 up:active 2024-03-24T00:22:34.197 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:22:33 smithi118 ceph-mon[31535]: fsmap cephfs:5 5 up:standby-replay 1 up:standby 5 up:active 2024-03-24T00:22:34.197 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:22:33 smithi118 ceph-mon[31535]: pgmap v622: 97 pgs: 97 active+clean; 149 MiB data, 795 MiB used, 1.0 TiB / 1.0 TiB avail; 41 KiB/s rd, 58 KiB/s wr, 132 op/s 2024-03-24T00:22:34.210 INFO:journalctl@ceph.mds.l.smithi193.stdout:Mar 24 00:22:33 smithi193 podman[72569]: 2024-03-24 00:22:33.886906999 +0000 UTC m=+0.801181740 container remove 34b63e7de437dbf746219f051fd02d843764da00bec1829324daf12d71db4db5 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-l, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.build-date=20240319, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, ceph=True, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, GIT_CLEAN=True, maintainer=Guillaume Abrioux ) 2024-03-24T00:22:34.211 INFO:journalctl@ceph.mds.l.smithi193.stdout:Mar 24 00:22:33 smithi193 systemd[1]: ceph-d79a5420-e971-11ee-95cd-87774f69a715@mds.l.service: Main process exited, code=exited, status=134/n/a 2024-03-24T00:22:34.211 INFO:journalctl@ceph.mds.l.smithi193.stdout:Mar 24 00:22:34 smithi193 systemd[1]: ceph-d79a5420-e971-11ee-95cd-87774f69a715@mds.l.service: Failed with result 'exit-code'. 2024-03-24T00:22:34.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:22:33 smithi193 ceph-mon[36878]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-03-24T00:22:34.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:22:33 smithi193 ceph-mon[36878]: Cluster is now healthy 2024-03-24T00:22:34.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:22:33 smithi193 ceph-mon[36878]: mds.? [v2:172.21.15.189:6836/75274228,v1:172.21.15.189:6837/75274228] up:active 2024-03-24T00:22:34.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:22:33 smithi193 ceph-mon[36878]: mds.? [v2:172.21.15.193:6832/2424695757,v1:172.21.15.193:6833/2424695757] up:active 2024-03-24T00:22:34.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:22:33 smithi193 ceph-mon[36878]: fsmap cephfs:5 3 up:standby-replay 3 up:standby 5 up:active 2024-03-24T00:22:34.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:22:33 smithi193 ceph-mon[36878]: fsmap cephfs:5 5 up:standby-replay 1 up:standby 5 up:active 2024-03-24T00:22:34.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:22:33 smithi193 ceph-mon[36878]: pgmap v622: 97 pgs: 97 active+clean; 149 MiB data, 795 MiB used, 1.0 TiB / 1.0 TiB avail; 41 KiB/s rd, 58 KiB/s wr, 132 op/s 2024-03-24T00:22:34.454 INFO:journalctl@ceph.mds.h.smithi189.stdout:Mar 24 00:22:34 smithi189 systemd[1]: ceph-d79a5420-e971-11ee-95cd-87774f69a715@mds.h.service: Failed with result 'exit-code'. 2024-03-24T00:22:34.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:22:33 smithi189 ceph-mon[36187]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-03-24T00:22:34.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:22:33 smithi189 ceph-mon[36187]: Cluster is now healthy 2024-03-24T00:22:34.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:22:33 smithi189 ceph-mon[36187]: mds.? [v2:172.21.15.189:6836/75274228,v1:172.21.15.189:6837/75274228] up:active 2024-03-24T00:22:34.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:22:33 smithi189 ceph-mon[36187]: mds.? [v2:172.21.15.193:6832/2424695757,v1:172.21.15.193:6833/2424695757] up:active 2024-03-24T00:22:34.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:22:33 smithi189 ceph-mon[36187]: fsmap cephfs:5 3 up:standby-replay 3 up:standby 5 up:active 2024-03-24T00:22:34.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:22:33 smithi189 ceph-mon[36187]: fsmap cephfs:5 5 up:standby-replay 1 up:standby 5 up:active 2024-03-24T00:22:34.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:22:33 smithi189 ceph-mon[36187]: pgmap v622: 97 pgs: 97 active+clean; 149 MiB data, 795 MiB used, 1.0 TiB / 1.0 TiB avail; 41 KiB/s rd, 58 KiB/s wr, 132 op/s 2024-03-24T00:22:36.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:22:35 smithi118 ceph-mon[31535]: pgmap v623: 97 pgs: 97 active+clean; 141 MiB data, 791 MiB used, 1.0 TiB / 1.0 TiB avail; 197 KiB/s rd, 72 KiB/s wr, 139 op/s 2024-03-24T00:22:36.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:22:35 smithi189 ceph-mon[36187]: pgmap v623: 97 pgs: 97 active+clean; 141 MiB data, 791 MiB used, 1.0 TiB / 1.0 TiB avail; 197 KiB/s rd, 72 KiB/s wr, 139 op/s 2024-03-24T00:22:36.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:22:35 smithi193 ceph-mon[36878]: pgmap v623: 97 pgs: 97 active+clean; 141 MiB data, 791 MiB used, 1.0 TiB / 1.0 TiB avail; 197 KiB/s rd, 72 KiB/s wr, 139 op/s 2024-03-24T00:22:36.846 INFO:tasks.workunit.client.0.smithi118.stdout: 47 75 0 86 0 322 0 2024-03-24T00:22:38.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:22:37 smithi118 ceph-mon[31535]: pgmap v624: 97 pgs: 97 active+clean; 142 MiB data, 787 MiB used, 1.0 TiB / 1.0 TiB avail; 442 KiB/s rd, 109 KiB/s wr, 115 op/s 2024-03-24T00:22:38.203 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:22:37 smithi189 ceph-mon[36187]: pgmap v624: 97 pgs: 97 active+clean; 142 MiB data, 787 MiB used, 1.0 TiB / 1.0 TiB avail; 442 KiB/s rd, 109 KiB/s wr, 115 op/s 2024-03-24T00:22:38.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:22:37 smithi193 ceph-mon[36878]: pgmap v624: 97 pgs: 97 active+clean; 142 MiB data, 787 MiB used, 1.0 TiB / 1.0 TiB avail; 442 KiB/s rd, 109 KiB/s wr, 115 op/s 2024-03-24T00:22:40.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:22:39 smithi118 ceph-mon[31535]: pgmap v625: 97 pgs: 97 active+clean; 142 MiB data, 788 MiB used, 1.0 TiB / 1.0 TiB avail; 417 KiB/s rd, 96 KiB/s wr, 103 op/s 2024-03-24T00:22:40.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:22:39 smithi189 ceph-mon[36187]: pgmap v625: 97 pgs: 97 active+clean; 142 MiB data, 788 MiB used, 1.0 TiB / 1.0 TiB avail; 417 KiB/s rd, 96 KiB/s wr, 103 op/s 2024-03-24T00:22:40.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:22:39 smithi193 ceph-mon[36878]: pgmap v625: 97 pgs: 97 active+clean; 142 MiB data, 788 MiB used, 1.0 TiB / 1.0 TiB avail; 417 KiB/s rd, 96 KiB/s wr, 103 op/s 2024-03-24T00:22:42.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:22:41 smithi118 ceph-mon[31535]: pgmap v626: 97 pgs: 97 active+clean; 142 MiB data, 788 MiB used, 1.0 TiB / 1.0 TiB avail; 412 KiB/s rd, 85 KiB/s wr, 73 op/s 2024-03-24T00:22:42.203 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:22:41 smithi189 ceph-mon[36187]: pgmap v626: 97 pgs: 97 active+clean; 142 MiB data, 788 MiB used, 1.0 TiB / 1.0 TiB avail; 412 KiB/s rd, 85 KiB/s wr, 73 op/s 2024-03-24T00:22:42.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:22:41 smithi193 ceph-mon[36878]: pgmap v626: 97 pgs: 97 active+clean; 142 MiB data, 788 MiB used, 1.0 TiB / 1.0 TiB avail; 412 KiB/s rd, 85 KiB/s wr, 73 op/s 2024-03-24T00:22:44.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:22:43 smithi118 ceph-mon[31535]: pgmap v627: 97 pgs: 97 active+clean; 142 MiB data, 788 MiB used, 1.0 TiB / 1.0 TiB avail; 413 KiB/s rd, 85 KiB/s wr, 62 op/s 2024-03-24T00:22:44.203 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:22:43 smithi189 ceph-mon[36187]: pgmap v627: 97 pgs: 97 active+clean; 142 MiB data, 788 MiB used, 1.0 TiB / 1.0 TiB avail; 413 KiB/s rd, 85 KiB/s wr, 62 op/s 2024-03-24T00:22:44.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:22:43 smithi193 ceph-mon[36878]: pgmap v627: 97 pgs: 97 active+clean; 142 MiB data, 788 MiB used, 1.0 TiB / 1.0 TiB avail; 413 KiB/s rd, 85 KiB/s wr, 62 op/s 2024-03-24T00:22:44.571 INFO:journalctl@ceph.mds.h.smithi189.stdout:Mar 24 00:22:44 smithi189 systemd[1]: ceph-d79a5420-e971-11ee-95cd-87774f69a715@mds.h.service: Scheduled restart job, restart counter is at 4. 2024-03-24T00:22:44.571 INFO:journalctl@ceph.mds.h.smithi189.stdout:Mar 24 00:22:44 smithi189 systemd[1]: Stopped Ceph mds.h for d79a5420-e971-11ee-95cd-87774f69a715. 2024-03-24T00:22:44.571 INFO:journalctl@ceph.mds.h.smithi189.stdout:Mar 24 00:22:44 smithi189 systemd[1]: Starting Ceph mds.h for d79a5420-e971-11ee-95cd-87774f69a715... 2024-03-24T00:22:44.571 INFO:journalctl@ceph.mds.h.smithi189.stdout:Mar 24 00:22:44 smithi189 podman[70228]: 2024-03-24 00:22:44.345147734 +0000 UTC m=+0.018108393 image pull c664afe56bb01db4f9afa9532c20598e201e664f2e9ba2be50db0d52674a26d2 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc 2024-03-24T00:22:44.571 INFO:journalctl@ceph.mds.h.smithi189.stdout:Mar 24 00:22:44 smithi189 podman[70228]: 2024-03-24 00:22:44.447731308 +0000 UTC m=+0.120691958 container create aba7624bb5f575c9a5afe6e0c2e6e4f0a26e28da6cad1fb654d4b6a007abcb98 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-h, org.label-schema.schema-version=1.0, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, org.label-schema.build-date=20240319, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, ceph=True, GIT_CLEAN=True, org.label-schema.vendor=CentOS) 2024-03-24T00:22:44.711 INFO:journalctl@ceph.mds.l.smithi193.stdout:Mar 24 00:22:44 smithi193 systemd[1]: ceph-d79a5420-e971-11ee-95cd-87774f69a715@mds.l.service: Scheduled restart job, restart counter is at 6. 2024-03-24T00:22:44.711 INFO:journalctl@ceph.mds.l.smithi193.stdout:Mar 24 00:22:44 smithi193 systemd[1]: Stopped Ceph mds.l for d79a5420-e971-11ee-95cd-87774f69a715. 2024-03-24T00:22:44.711 INFO:journalctl@ceph.mds.l.smithi193.stdout:Mar 24 00:22:44 smithi193 systemd[1]: ceph-d79a5420-e971-11ee-95cd-87774f69a715@mds.l.service: Start request repeated too quickly. 2024-03-24T00:22:44.711 INFO:journalctl@ceph.mds.l.smithi193.stdout:Mar 24 00:22:44 smithi193 systemd[1]: ceph-d79a5420-e971-11ee-95cd-87774f69a715@mds.l.service: Failed with result 'exit-code'. 2024-03-24T00:22:44.711 INFO:journalctl@ceph.mds.l.smithi193.stdout:Mar 24 00:22:44 smithi193 systemd[1]: Failed to start Ceph mds.l for d79a5420-e971-11ee-95cd-87774f69a715. 2024-03-24T00:22:44.922 INFO:journalctl@ceph.mds.h.smithi189.stdout:Mar 24 00:22:44 smithi189 podman[70228]: 2024-03-24 00:22:44.683833233 +0000 UTC m=+0.356793894 container init aba7624bb5f575c9a5afe6e0c2e6e4f0a26e28da6cad1fb654d4b6a007abcb98 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-h, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, io.buildah.version=1.33.5, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, GIT_BRANCH=HEAD, GIT_CLEAN=True, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, org.label-schema.build-date=20240319, org.label-schema.vendor=CentOS) 2024-03-24T00:22:44.922 INFO:journalctl@ceph.mds.h.smithi189.stdout:Mar 24 00:22:44 smithi189 podman[70228]: 2024-03-24 00:22:44.686512437 +0000 UTC m=+0.359473086 container start aba7624bb5f575c9a5afe6e0c2e6e4f0a26e28da6cad1fb654d4b6a007abcb98 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-h, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240319, CEPH_POINT_RELEASE=, ceph=True) 2024-03-24T00:22:44.922 INFO:journalctl@ceph.mds.h.smithi189.stdout:Mar 24 00:22:44 smithi189 ceph-mds[70246]: -- 172.21.15.189:0/1434960970 <== mon.2 v2:172.21.15.189:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x556f5b913a40 con 0x556f5b8e6c00 2024-03-24T00:22:44.923 INFO:journalctl@ceph.mds.h.smithi189.stdout:Mar 24 00:22:44 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-h[70242]: starting mds.h at 2024-03-24T00:22:44.923 INFO:journalctl@ceph.mds.h.smithi189.stdout:Mar 24 00:22:44 smithi189 bash[70228]: aba7624bb5f575c9a5afe6e0c2e6e4f0a26e28da6cad1fb654d4b6a007abcb98 2024-03-24T00:22:44.923 INFO:journalctl@ceph.mds.h.smithi189.stdout:Mar 24 00:22:44 smithi189 systemd[1]: Started Ceph mds.h for d79a5420-e971-11ee-95cd-87774f69a715. 2024-03-24T00:22:45.137 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:22:45.137+0000 7fab87e00640 1 -- 172.21.15.118:0/3244527358 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fab80109000 msgr2=0x7fab8010f9e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:22:45.137 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:22:45.137+0000 7fab87e00640 1 --2- 172.21.15.118:0/3244527358 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fab80109000 0x7fab8010f9e0 secure :-1 s=READY pgs=441 cs=0 l=1 rev1=1 crypto rx=0x7fab6c009900 tx=0x7fab6c02bc50 comp rx=0 tx=0).stop 2024-03-24T00:22:45.137 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:22:45.137+0000 7fab87e00640 1 -- 172.21.15.118:0/3244527358 shutdown_connections 2024-03-24T00:22:45.137 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:22:45.137+0000 7fab87e00640 1 --2- 172.21.15.118:0/3244527358 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fab8010ff20 0x7fab80112310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:22:45.137 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:22:45.137+0000 7fab87e00640 1 --2- 172.21.15.118:0/3244527358 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fab80109000 0x7fab8010f9e0 unknown :-1 s=CLOSED pgs=441 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:22:45.137 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:22:45.137+0000 7fab87e00640 1 --2- 172.21.15.118:0/3244527358 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fab801086a0 0x7fab80108ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:22:45.137 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:22:45.137+0000 7fab87e00640 1 -- 172.21.15.118:0/3244527358 >> 172.21.15.118:0/3244527358 conn(0x7fab8006e8a0 msgr2=0x7fab80070ce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:22:45.137 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:22:45.137+0000 7fab87e00640 1 -- 172.21.15.118:0/3244527358 shutdown_connections 2024-03-24T00:22:45.137 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:22:45.137+0000 7fab87e00640 1 -- 172.21.15.118:0/3244527358 wait complete. 2024-03-24T00:22:45.139 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:22:45.138+0000 7fab87e00640 1 Processor -- start 2024-03-24T00:22:45.139 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:22:45.138+0000 7fab87e00640 1 -- start start 2024-03-24T00:22:45.139 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:22:45.138+0000 7fab87e00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fab801086a0 0x7fab801a2330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:22:45.139 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:22:45.138+0000 7fab87e00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fab80109000 0x7fab801a2870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:22:45.139 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:22:45.138+0000 7fab87e00640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fab8010ff20 0x7fab801a98e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:22:45.139 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:22:45.138+0000 7fab87e00640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fab80115b90 con 0x7fab8010ff20 2024-03-24T00:22:45.139 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:22:45.138+0000 7fab87e00640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fab80115a10 con 0x7fab801086a0 2024-03-24T00:22:45.139 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:22:45.138+0000 7fab87e00640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fab80115d10 con 0x7fab80109000 2024-03-24T00:22:45.139 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:22:45.139+0000 7fab85400640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fab801086a0 0x7fab801a2330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:22:45.139 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:22:45.139+0000 7fab85400640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fab801086a0 0x7fab801a2330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.118:50410/0 (socket says 172.21.15.118:50410) 2024-03-24T00:22:45.139 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:22:45.139+0000 7fab85400640 1 -- 172.21.15.118:0/2359644756 learned_addr learned my addr 172.21.15.118:0/2359644756 (peer_addr_for_me v2:172.21.15.118:0/0) 2024-03-24T00:22:45.140 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:22:45.139+0000 7fab84a00640 1 --2- 172.21.15.118:0/2359644756 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fab80109000 0x7fab801a2870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:22:45.140 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:22:45.139+0000 7fab85e00640 1 --2- 172.21.15.118:0/2359644756 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fab8010ff20 0x7fab801a98e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:22:45.140 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:22:45.139+0000 7fab85400640 1 -- 172.21.15.118:0/2359644756 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fab80109000 msgr2=0x7fab801a2870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:22:45.140 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:22:45.139+0000 7fab85400640 1 --2- 172.21.15.118:0/2359644756 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fab80109000 0x7fab801a2870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:22:45.140 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:22:45.139+0000 7fab85400640 1 -- 172.21.15.118:0/2359644756 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fab8010ff20 msgr2=0x7fab801a98e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:22:45.140 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:22:45.139+0000 7fab85400640 1 --2- 172.21.15.118:0/2359644756 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fab8010ff20 0x7fab801a98e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:22:45.140 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:22:45.139+0000 7fab85400640 1 -- 172.21.15.118:0/2359644756 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fab801a9f50 con 0x7fab801086a0 2024-03-24T00:22:45.140 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:22:45.139+0000 7fab85400640 1 --2- 172.21.15.118:0/2359644756 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fab801086a0 0x7fab801a2330 secure :-1 s=READY pgs=442 cs=0 l=1 rev1=1 crypto rx=0x7fab7400e9c0 tx=0x7fab7400ee80 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:22:45.141 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:22:45.140+0000 7fab7e000640 1 -- 172.21.15.118:0/2359644756 <== mon.2 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fab7400cd80 con 0x7fab801086a0 2024-03-24T00:22:45.141 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:22:45.140+0000 7fab7e000640 1 -- 172.21.15.118:0/2359644756 <== mon.2 v2:172.21.15.189:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fab740035f0 con 0x7fab801086a0 2024-03-24T00:22:45.141 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:22:45.140+0000 7fab7e000640 1 -- 172.21.15.118:0/2359644756 <== mon.2 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fab74010690 con 0x7fab801086a0 2024-03-24T00:22:45.141 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:22:45.140+0000 7fab87e00640 1 -- 172.21.15.118:0/2359644756 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fab801aa240 con 0x7fab801086a0 2024-03-24T00:22:45.141 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:22:45.140+0000 7fab87e00640 1 -- 172.21.15.118:0/2359644756 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fab801aa710 con 0x7fab801086a0 2024-03-24T00:22:45.143 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:22:45.143+0000 7fab8ac00740 1 Processor -- start 2024-03-24T00:22:45.144 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:22:45.143+0000 7fab8ac00740 1 -- start start 2024-03-24T00:22:45.144 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:22:45.143+0000 7fab8ac00740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x559d13f721e0 0x559d13f72600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:22:45.144 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:22:45.143+0000 7fab8ac00740 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x559d13f72bd0 0x559d13f776d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:22:45.144 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:22:45.143+0000 7fab8ac00740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x559d13f77d40 0x559d13f7a1d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:22:45.144 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:22:45.143+0000 7fab8ac00740 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x559d13f353b0 con 0x559d13f72bd0 2024-03-24T00:22:45.144 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:22:45.143+0000 7fab8ac00740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x559d13ed7830 con 0x559d13f77d40 2024-03-24T00:22:45.144 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:22:45.143+0000 7fab8ac00740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x559d13e805d0 con 0x559d13f721e0 2024-03-24T00:22:45.144 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:22:45.144+0000 7fab85e00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x559d13f721e0 0x559d13f72600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:22:45.144 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:22:45.144+0000 7fab85e00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x559d13f721e0 0x559d13f72600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.118:60212/0 (socket says 172.21.15.118:60212) 2024-03-24T00:22:45.144 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:22:45.144+0000 7fab85e00640 1 -- 172.21.15.118:0/817223382 learned_addr learned my addr 172.21.15.118:0/817223382 (peer_addr_for_me v2:172.21.15.118:0/0) 2024-03-24T00:22:45.144 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:22:45.144+0000 7fab84a00640 1 --2- 172.21.15.118:0/817223382 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x559d13f77d40 0x559d13f7a1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:22:45.144 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:22:45.144+0000 7fab7e000640 1 -- 172.21.15.118:0/2359644756 <== mon.2 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 20) v1 ==== 137291+0+0 (secure 0 0 0) 0x7fab740037a0 con 0x7fab801086a0 2024-03-24T00:22:45.144 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:22:45.144+0000 7fab85e00640 1 -- 172.21.15.118:0/817223382 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x559d13f77d40 msgr2=0x559d13f7a1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:22:45.144 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:22:45.144+0000 7fab85e00640 1 --2- 172.21.15.118:0/817223382 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x559d13f77d40 0x559d13f7a1d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:22:45.144 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:22:45.144+0000 7fab85e00640 1 -- 172.21.15.118:0/817223382 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x559d13f72bd0 msgr2=0x559d13f776d0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2024-03-24T00:22:45.145 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:22:45.144+0000 7fab85e00640 1 --2- 172.21.15.118:0/817223382 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x559d13f72bd0 0x559d13f776d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:22:45.145 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:22:45.144+0000 7fab85e00640 1 -- 172.21.15.118:0/817223382 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x559d13e982c0 con 0x559d13f721e0 2024-03-24T00:22:45.146 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:22:45.144+0000 7fab85e00640 1 --2- 172.21.15.118:0/817223382 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x559d13f721e0 0x559d13f72600 secure :-1 s=READY pgs=444 cs=0 l=1 rev1=1 crypto rx=0x7fab78009fd0 tx=0x7fab780118d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=19bc5f52d53db1f server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:22:45.146 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:22:45.145+0000 7fab72600640 1 -- 172.21.15.118:0/817223382 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fab78004d20 con 0x559d13f721e0 2024-03-24T00:22:45.146 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:22:45.145+0000 7fab72600640 1 -- 172.21.15.118:0/817223382 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fab78004050 con 0x559d13f721e0 2024-03-24T00:22:45.146 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:22:45.145+0000 7fab72600640 1 -- 172.21.15.118:0/817223382 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fab78005280 con 0x559d13f721e0 2024-03-24T00:22:45.146 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:22:45.145+0000 7fab8ac00740 1 -- 172.21.15.118:0/817223382 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x559d13f721e0 msgr2=0x559d13f72600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:22:45.146 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:22:45.145+0000 7fab8ac00740 1 --2- 172.21.15.118:0/817223382 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x559d13f721e0 0x559d13f72600 secure :-1 s=READY pgs=444 cs=0 l=1 rev1=1 crypto rx=0x7fab78009fd0 tx=0x7fab780118d0 comp rx=0 tx=0).stop 2024-03-24T00:22:45.146 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:22:45.145+0000 7fab7e000640 1 --2- 172.21.15.118:0/2359644756 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7fab680a2920 0x7fab680a4de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:22:45.147 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:22:45.146+0000 7fab8ac00740 1 -- 172.21.15.118:0/817223382 shutdown_connections 2024-03-24T00:22:45.147 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:22:45.146+0000 7fab8ac00740 1 --2- 172.21.15.118:0/817223382 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x559d13f77d40 0x559d13f7a1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:22:45.147 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:22:45.146+0000 7fab8ac00740 1 --2- 172.21.15.118:0/817223382 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x559d13f72bd0 0x559d13f776d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:22:45.147 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:22:45.146+0000 7fab8ac00740 1 --2- 172.21.15.118:0/817223382 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x559d13f721e0 0x559d13f72600 unknown :-1 s=CLOSED pgs=444 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:22:45.147 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:22:45.146+0000 7fab8ac00740 1 -- 172.21.15.118:0/817223382 >> 172.21.15.118:0/817223382 conn(0x559d13f6d750 msgr2=0x559d13f6db50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:22:45.147 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:22:45.146+0000 7fab7e000640 1 -- 172.21.15.118:0/2359644756 <== mon.2 v2:172.21.15.189:3300/0 5 ==== osd_map(109..109 src has 1..109) v4 ==== 9911+0+0 (secure 0 0 0) 0x7fab740cf3c0 con 0x7fab801086a0 2024-03-24T00:22:45.147 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:22:45.146+0000 7fab85400640 1 --2- 172.21.15.118:0/2359644756 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7fab680a2920 0x7fab680a4de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:22:45.147 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:22:45.146+0000 7fab8ac00740 1 -- 172.21.15.118:0/817223382 shutdown_connections 2024-03-24T00:22:45.147 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:22:45.146+0000 7fab8ac00740 1 -- 172.21.15.118:0/817223382 wait complete. 2024-03-24T00:22:45.147 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:22:45.146+0000 7fab85400640 1 --2- 172.21.15.118:0/2359644756 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7fab680a2920 0x7fab680a4de0 secure :-1 s=READY pgs=379 cs=0 l=1 rev1=1 crypto rx=0x7fab740980a0 tx=0x7fab740097d0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:22:45.147 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:22:45.146+0000 7fab8ac00740 1 -- start start 2024-03-24T00:22:45.147 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:22:45.147+0000 7fab8ac00740 1 Processor -- start 2024-03-24T00:22:45.151 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:22:45.147+0000 7fab8ac00740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x559d13f721e0 0x559d13f7c330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:22:45.151 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:22:45.147+0000 7fab8ac00740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x559d13f72bd0 0x559d13f7c870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:22:45.151 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:22:45.147+0000 7fab8ac00740 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x559d13f77d40 0x559d13f838b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:22:45.151 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:22:45.147+0000 7fab8ac00740 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x559d13e93a20 con 0x559d13f77d40 2024-03-24T00:22:45.151 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:22:45.147+0000 7fab8ac00740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x559d13ed38d0 con 0x559d13f721e0 2024-03-24T00:22:45.151 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:22:45.147+0000 7fab8ac00740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x559d13f7acf0 con 0x559d13f72bd0 2024-03-24T00:22:45.151 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:22:45.147+0000 7fab8ac00740 10 client.0 _ll_register_callbacks cb 0x559d13ed01c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T00:22:45.151 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:22:45.147+0000 7fab84a00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x559d13f72bd0 0x559d13f7c870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:22:45.151 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:22:45.147+0000 7fab84a00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x559d13f72bd0 0x559d13f7c870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.118:60222/0 (socket says 172.21.15.118:60222) 2024-03-24T00:22:45.152 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:22:45.147+0000 7fab84a00640 1 -- 172.21.15.118:0/607630947 learned_addr learned my addr 172.21.15.118:0/607630947 (peer_addr_for_me v2:172.21.15.118:0/0) 2024-03-24T00:22:45.152 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:22:45.147+0000 7fab85e00640 1 --2- 172.21.15.118:0/607630947 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x559d13f721e0 0x559d13f7c330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:22:45.152 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:22:45.147+0000 7fab85e00640 1 --2- 172.21.15.118:0/607630947 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x559d13f77d40 0x559d13f838b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:22:45.152 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:22:45.148+0000 7fab84a00640 1 -- 172.21.15.118:0/607630947 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x559d13f721e0 msgr2=0x559d13f7c330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:22:45.152 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:22:45.148+0000 7fab84a00640 1 --2- 172.21.15.118:0/607630947 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x559d13f721e0 0x559d13f7c330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:22:45.152 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:22:45.148+0000 7fab84a00640 1 -- 172.21.15.118:0/607630947 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x559d13f77d40 msgr2=0x559d13f838b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:22:45.152 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:22:45.148+0000 7fab84a00640 1 --2- 172.21.15.118:0/607630947 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x559d13f77d40 0x559d13f838b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:22:45.152 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:22:45.148+0000 7fab84a00640 1 -- 172.21.15.118:0/607630947 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x559d13bec210 con 0x559d13f72bd0 2024-03-24T00:22:45.152 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:22:45.148+0000 7fab85e00640 1 --2- 172.21.15.118:0/607630947 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x559d13f77d40 0x559d13f838b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-24T00:22:45.152 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:22:45.148+0000 7fab84a00640 1 --2- 172.21.15.118:0/607630947 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x559d13f72bd0 0x559d13f7c870 secure :-1 s=READY pgs=445 cs=0 l=1 rev1=1 crypto rx=0x7fab6c005ab0 tx=0x7fab6c004270 comp rx=0 tx=0).ready entity=mon.1 client_cookie=e982d6f3f3d4c031 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:22:45.152 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:22:45.148+0000 7fab66a00640 10 client.0 ms_handle_connect on v2:172.21.15.193:3300/0 2024-03-24T00:22:45.152 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:22:45.148+0000 7fab66a00640 1 -- 172.21.15.118:0/607630947 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fab6c005d50 con 0x559d13f72bd0 2024-03-24T00:22:45.152 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:22:45.148+0000 7fab8ac00740 1 -- 172.21.15.118:0/607630947 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fab78004050 con 0x559d13f72bd0 2024-03-24T00:22:45.152 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:22:45.148+0000 7fab66a00640 1 -- 172.21.15.118:0/607630947 <== mon.1 v2:172.21.15.193:3300/0 2 ==== osd_map(109..109 src has 1..109) v4 ==== 9911+0+0 (secure 0 0 0) 0x7fab6c0048c0 con 0x559d13f72bd0 2024-03-24T00:22:45.153 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:22:45.148+0000 7fab66a00640 1 -- 172.21.15.118:0/607630947 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_get_version_reply(handle=1 version=120) v2 ==== 24+0+0 (secure 0 0 0) 0x7fab6c004ac0 con 0x559d13f72bd0 2024-03-24T00:22:45.153 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:22:45.148+0000 7fab8ac00740 10 client.26899 fetch_fsmap learned FSMap version 120 2024-03-24T00:22:45.153 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:22:45.148+0000 7fab8ac00740 1 -- 172.21.15.118:0/607630947 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,fsmap=120,monmap=4+}) v3 -- 0x559d13cb8620 con 0x559d13f72bd0 2024-03-24T00:22:45.153 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:22:45.149+0000 7fab66a00640 1 -- 172.21.15.118:0/607630947 <== mon.1 v2:172.21.15.193:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fab78004050 con 0x559d13f72bd0 2024-03-24T00:22:45.153 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:22:45.149+0000 7fab66a00640 1 -- 172.21.15.118:0/607630947 <== mon.1 v2:172.21.15.193:3300/0 5 ==== fsmap(e 120) v1 ==== 7367+0+0 (secure 0 0 0) 0x7fab6c046070 con 0x559d13f72bd0 2024-03-24T00:22:45.153 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:22:45.149+0000 7fab8ac00740 10 client.26899 fetch_fsmap finished waiting for FSMap version 120 2024-03-24T00:22:45.153 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:22:45.149+0000 7fab8ac00740 10 client.26899 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-24T00:22:45.153 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:22:45.149+0000 7fab8ac00740 20 client.26899 populate_metadata read hostname 'smithi118' 2024-03-24T00:22:45.153 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:22:45.149+0000 7fab8ac00740 1 --2- 172.21.15.118:0/607630947 >> [v2:172.21.15.189:6838/4074484168,v1:172.21.15.189:6839/4074484168] conn(0x559d13f882a0 0x559d13f8a6d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:22:45.153 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:22:45.149+0000 7fab8ac00740 4 client.26899 mds_command: new command op to 24569 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-24T00:22:45.153 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:22:45.149+0000 7fab8ac00740 1 -- 172.21.15.118:0/607630947 --> [v2:172.21.15.189:6838/4074484168,v1:172.21.15.189:6839/4074484168] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x559d13ee4d70 con 0x559d13f882a0 2024-03-24T00:22:45.153 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:22:45.149+0000 7fab85e00640 1 --2- 172.21.15.118:0/607630947 >> [v2:172.21.15.189:6838/4074484168,v1:172.21.15.189:6839/4074484168] conn(0x559d13f882a0 0x559d13f8a6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:22:45.153 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:22:45.150+0000 7fab85e00640 1 --2- 172.21.15.118:0/607630947 >> [v2:172.21.15.189:6838/4074484168,v1:172.21.15.189:6839/4074484168] conn(0x559d13f882a0 0x559d13f8a6d0 crc :-1 s=READY pgs=277 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=99c1626de747270e server_cookie=331486dfc4f45472 in_seq=0 out_seq=0 2024-03-24T00:22:45.153 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:22:45.150+0000 7fab66a00640 10 client.26899 ms_handle_connect on v2:172.21.15.189:6838/4074484168 2024-03-24T00:22:45.154 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:22:45.150+0000 7fab66a00640 1 -- 172.21.15.118:0/607630947 <== mds.0 v2:172.21.15.189:6838/4074484168 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x559d13ee4d70 con 0x559d13f882a0 2024-03-24T00:22:45.154 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:22:45.150+0000 7fab66a00640 10 client.26899 handle_command_reply: tid=0 2024-03-24T00:22:45.154 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:22:45.150+0000 7fab8ac00740 1 client.26899 shutdown 2024-03-24T00:22:45.154 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:22:45.151+0000 7fab8ac00740 1 -- 172.21.15.118:0/607630947 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x559d13f72bd0 msgr2=0x559d13f7c870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:22:45.154 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:22:45.151+0000 7fab8ac00740 1 --2- 172.21.15.118:0/607630947 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x559d13f72bd0 0x559d13f7c870 secure :-1 s=READY pgs=445 cs=0 l=1 rev1=1 crypto rx=0x7fab6c005ab0 tx=0x7fab6c004270 comp rx=0 tx=0).stop 2024-03-24T00:22:45.154 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:22:45.151+0000 7fab8ac00740 1 -- 172.21.15.118:0/607630947 shutdown_connections 2024-03-24T00:22:45.154 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:22:45.151+0000 7fab8ac00740 1 --2- 172.21.15.118:0/607630947 >> [v2:172.21.15.189:6838/4074484168,v1:172.21.15.189:6839/4074484168] conn(0x559d13f882a0 0x559d13f8a6d0 crc :-1 s=READY pgs=277 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:22:45.154 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:22:45.151+0000 7fab8ac00740 1 --2- 172.21.15.118:0/607630947 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x559d13f77d40 0x559d13f838b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:22:45.154 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:22:45.151+0000 7fab8ac00740 1 --2- 172.21.15.118:0/607630947 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x559d13f72bd0 0x559d13f7c870 unknown :-1 s=CLOSED pgs=445 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:22:45.154 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:22:45.151+0000 7fab66a00640 0 client.26899 ms_handle_reset on v2:172.21.15.189:6838/4074484168 2024-03-24T00:22:45.154 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:22:45.151+0000 7fab8ac00740 1 --2- 172.21.15.118:0/607630947 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x559d13f721e0 0x559d13f7c330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:22:45.154 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:22:45.151+0000 7fab8ac00740 1 -- 172.21.15.118:0/607630947 >> 172.21.15.118:0/607630947 conn(0x559d13f6d750 msgr2=0x559d13f6c950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:22:45.154 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:22:45.151+0000 7fab8ac00740 1 -- 172.21.15.118:0/607630947 shutdown_connections 2024-03-24T00:22:45.154 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:22:45.151+0000 7fab8ac00740 1 -- 172.21.15.118:0/607630947 wait complete. 2024-03-24T00:22:45.155 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:22:45.151+0000 7fab8ac00740 20 client.26899 trim_cache size 0 max 16384 2024-03-24T00:22:45.165 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:22:45.166+0000 7fab8ac00740 1 Processor -- start 2024-03-24T00:22:45.165 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:22:45.166+0000 7fab8ac00740 1 -- start start 2024-03-24T00:22:45.167 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:22:45.166+0000 7fab8ac00740 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x559d13f721e0 0x559d13f7c330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:22:45.167 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:22:45.166+0000 7fab8ac00740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x559d13f72bd0 0x559d13f7c870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:22:45.167 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:22:45.166+0000 7fab8ac00740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x559d13f77d40 0x559d13f83870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:22:45.167 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:22:45.166+0000 7fab8ac00740 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fab801b5ae0 con 0x559d13f721e0 2024-03-24T00:22:45.167 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:22:45.166+0000 7fab8ac00740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fab801b4130 con 0x559d13f72bd0 2024-03-24T00:22:45.167 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:22:45.166+0000 7fab8ac00740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fab801b5450 con 0x559d13f77d40 2024-03-24T00:22:45.167 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:22:45.166+0000 7fab84a00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x559d13f72bd0 0x559d13f7c870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:22:45.167 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:22:45.166+0000 7fab85e00640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x559d13f721e0 0x559d13f7c330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:22:45.167 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:22:45.166+0000 7fab84a00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x559d13f72bd0 0x559d13f7c870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.118:50446/0 (socket says 172.21.15.118:50446) 2024-03-24T00:22:45.167 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:22:45.166+0000 7fab84a00640 1 -- 172.21.15.118:0/1515224841 learned_addr learned my addr 172.21.15.118:0/1515224841 (peer_addr_for_me v2:172.21.15.118:0/0) 2024-03-24T00:22:45.167 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:22:45.166+0000 7fab85e00640 1 --2- 172.21.15.118:0/1515224841 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x559d13f77d40 0x559d13f83870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:22:45.167 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:22:45.167+0000 7fab84a00640 1 -- 172.21.15.118:0/1515224841 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x559d13f77d40 msgr2=0x559d13f83870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:22:45.167 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:22:45.167+0000 7fab84a00640 1 --2- 172.21.15.118:0/1515224841 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x559d13f77d40 0x559d13f83870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:22:45.168 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:22:45.167+0000 7fab84a00640 1 -- 172.21.15.118:0/1515224841 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x559d13f721e0 msgr2=0x559d13f7c330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:22:45.168 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:22:45.167+0000 7fab84a00640 1 --2- 172.21.15.118:0/1515224841 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x559d13f721e0 0x559d13f7c330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:22:45.168 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:22:45.167+0000 7fab84a00640 1 -- 172.21.15.118:0/1515224841 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x559d13f73460 con 0x559d13f72bd0 2024-03-24T00:22:45.168 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:22:45.167+0000 7fab85e00640 1 --2- 172.21.15.118:0/1515224841 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x559d13f721e0 0x559d13f7c330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T00:22:45.168 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:22:45.167+0000 7fab84a00640 1 --2- 172.21.15.118:0/1515224841 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x559d13f72bd0 0x559d13f7c870 secure :-1 s=READY pgs=443 cs=0 l=1 rev1=1 crypto rx=0x7fab6c02c190 tx=0x7fab6c009830 comp rx=0 tx=0).ready entity=mon.2 client_cookie=20428a33dff4f1c6 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:22:45.168 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:22:45.167+0000 7fab71c00640 1 -- 172.21.15.118:0/1515224841 <== mon.2 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fab6c005d50 con 0x559d13f72bd0 2024-03-24T00:22:45.168 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:22:45.167+0000 7fab71c00640 1 -- 172.21.15.118:0/1515224841 <== mon.2 v2:172.21.15.189:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fab6c003d30 con 0x559d13f72bd0 2024-03-24T00:22:45.168 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:22:45.167+0000 7fab71c00640 1 -- 172.21.15.118:0/1515224841 <== mon.2 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fab6c02ccb0 con 0x559d13f72bd0 2024-03-24T00:22:45.171 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:22:45.168+0000 7fab8ac00740 1 -- 172.21.15.118:0/1515224841 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x559d13f72bd0 msgr2=0x559d13f7c870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:22:45.171 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:22:45.168+0000 7fab8ac00740 1 --2- 172.21.15.118:0/1515224841 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x559d13f72bd0 0x559d13f7c870 secure :-1 s=READY pgs=443 cs=0 l=1 rev1=1 crypto rx=0x7fab6c02c190 tx=0x7fab6c009830 comp rx=0 tx=0).stop 2024-03-24T00:22:45.171 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:22:45.168+0000 7fab8ac00740 1 -- 172.21.15.118:0/1515224841 shutdown_connections 2024-03-24T00:22:45.171 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:22:45.168+0000 7fab8ac00740 1 --2- 172.21.15.118:0/1515224841 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x559d13f77d40 0x559d13f83870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:22:45.171 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:22:45.168+0000 7fab8ac00740 1 --2- 172.21.15.118:0/1515224841 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x559d13f72bd0 0x559d13f7c870 unknown :-1 s=CLOSED pgs=443 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:22:45.171 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:22:45.168+0000 7fab8ac00740 1 --2- 172.21.15.118:0/1515224841 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x559d13f721e0 0x559d13f7c330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:22:45.171 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:22:45.168+0000 7fab8ac00740 1 -- 172.21.15.118:0/1515224841 >> 172.21.15.118:0/1515224841 conn(0x559d13f6d750 msgr2=0x559d13f7b0f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:22:45.171 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:22:45.168+0000 7fab8ac00740 1 -- 172.21.15.118:0/1515224841 shutdown_connections 2024-03-24T00:22:45.171 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:22:45.168+0000 7fab8ac00740 1 -- 172.21.15.118:0/1515224841 wait complete. 2024-03-24T00:22:45.171 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:22:45.168+0000 7fab8ac00740 1 -- start start 2024-03-24T00:22:45.171 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:22:45.169+0000 7fab8ac00740 1 Processor -- start 2024-03-24T00:22:45.172 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:22:45.169+0000 7fab8ac00740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x559d13f721e0 0x559d13f80a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:22:45.172 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:22:45.169+0000 7fab8ac00740 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x559d13f72bd0 0x559d13f80f50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:22:45.172 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:22:45.169+0000 7fab8ac00740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x559d13f77d40 0x559d13f81f10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:22:45.172 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:22:45.169+0000 7fab8ac00740 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fab801b4df0 con 0x559d13f72bd0 2024-03-24T00:22:45.172 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:22:45.169+0000 7fab8ac00740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fab801b4790 con 0x559d13f721e0 2024-03-24T00:22:45.172 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:22:45.169+0000 7fab8ac00740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fab801b2310 con 0x559d13f77d40 2024-03-24T00:22:45.172 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:22:45.169+0000 7fab8ac00740 10 client.0 _ll_register_callbacks cb 0x7fab6c009780 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T00:22:45.172 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:22:45.169+0000 7fab84a00640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x559d13f72bd0 0x559d13f80f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:22:45.172 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:22:45.169+0000 7fab85e00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x559d13f721e0 0x559d13f80a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:22:45.172 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:22:45.169+0000 7fab84a00640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x559d13f72bd0 0x559d13f80f50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.118:3300/0 says I am v2:172.21.15.118:43284/0 (socket says 172.21.15.118:43284) 2024-03-24T00:22:45.172 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:22:45.169+0000 7fab84a00640 1 -- 172.21.15.118:0/3539248607 learned_addr learned my addr 172.21.15.118:0/3539248607 (peer_addr_for_me v2:172.21.15.118:0/0) 2024-03-24T00:22:45.172 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:22:45.169+0000 7fab85e00640 1 --2- 172.21.15.118:0/3539248607 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x559d13f721e0 0x559d13f80a10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.118:50460/0 (socket says 172.21.15.118:50460) 2024-03-24T00:22:45.172 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:22:45.169+0000 7fab85e00640 1 --2- 172.21.15.118:0/3539248607 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x559d13f77d40 0x559d13f81f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:22:45.172 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:22:45.170+0000 7fab84a00640 1 -- 172.21.15.118:0/3539248607 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x559d13f77d40 msgr2=0x559d13f81f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:22:45.173 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:22:45.170+0000 7fab84a00640 1 --2- 172.21.15.118:0/3539248607 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x559d13f77d40 0x559d13f81f10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:22:45.173 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:22:45.170+0000 7fab84a00640 1 -- 172.21.15.118:0/3539248607 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x559d13f721e0 msgr2=0x559d13f80a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:22:45.173 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:22:45.170+0000 7fab84a00640 1 --2- 172.21.15.118:0/3539248607 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x559d13f721e0 0x559d13f80a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:22:45.173 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:22:45.170+0000 7fab84a00640 1 -- 172.21.15.118:0/3539248607 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x559d13f82500 con 0x559d13f72bd0 2024-03-24T00:22:45.173 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:22:45.170+0000 7fab84a00640 1 --2- 172.21.15.118:0/3539248607 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x559d13f72bd0 0x559d13f80f50 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7fab6c0059d0 tx=0x7fab6c005d50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=dff7f67aa0d1a6c8 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:22:45.173 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:22:45.170+0000 7fab66a00640 10 client.0 ms_handle_connect on v2:172.21.15.118:3300/0 2024-03-24T00:22:45.173 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:22:45.170+0000 7fab66a00640 1 -- 172.21.15.118:0/3539248607 <== mon.0 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fab6c003800 con 0x559d13f72bd0 2024-03-24T00:22:45.173 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:22:45.170+0000 7fab8ac00740 1 -- 172.21.15.118:0/3539248607 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x559d13edf0c0 con 0x559d13f72bd0 2024-03-24T00:22:45.173 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:22:45.170+0000 7fab66a00640 1 -- 172.21.15.118:0/3539248607 <== mon.0 v2:172.21.15.118:3300/0 2 ==== osd_map(109..109 src has 1..109) v4 ==== 9911+0+0 (secure 0 0 0) 0x7fab6c004ed0 con 0x559d13f72bd0 2024-03-24T00:22:45.173 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:22:45.170+0000 7fab66a00640 1 -- 172.21.15.118:0/3539248607 <== mon.0 v2:172.21.15.118:3300/0 3 ==== mon_get_version_reply(handle=1 version=120) v2 ==== 24+0+0 (secure 0 0 0) 0x7fab6c005110 con 0x559d13f72bd0 2024-03-24T00:22:45.173 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:22:45.170+0000 7fab8ac00740 10 client.16641 fetch_fsmap learned FSMap version 120 2024-03-24T00:22:45.173 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:22:45.171+0000 7fab8ac00740 1 -- 172.21.15.118:0/3539248607 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({config=0+,fsmap=120,monmap=4+}) v3 -- 0x559d13f83810 con 0x559d13f72bd0 2024-03-24T00:22:45.173 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:22:45.171+0000 7fab66a00640 1 -- 172.21.15.118:0/3539248607 <== mon.0 v2:172.21.15.118:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x559d13edf0c0 con 0x559d13f72bd0 2024-03-24T00:22:45.173 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:22:45.171+0000 7fab66a00640 1 -- 172.21.15.118:0/3539248607 <== mon.0 v2:172.21.15.118:3300/0 5 ==== fsmap(e 120) v1 ==== 7367+0+0 (secure 0 0 0) 0x7fab6c046040 con 0x559d13f72bd0 2024-03-24T00:22:45.174 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:22:45.171+0000 7fab8ac00740 10 client.16641 fetch_fsmap finished waiting for FSMap version 120 2024-03-24T00:22:45.174 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:22:45.171+0000 7fab8ac00740 10 client.16641 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-24T00:22:45.174 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:22:45.171+0000 7fab8ac00740 20 client.16641 populate_metadata read hostname 'smithi118' 2024-03-24T00:22:45.174 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:22:45.171+0000 7fab8ac00740 1 --2- 172.21.15.118:0/3539248607 >> [v2:172.21.15.189:6838/4074484168,v1:172.21.15.189:6839/4074484168] conn(0x559d13fe3ee0 0x559d13fe62d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:22:45.174 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:22:45.171+0000 7fab8ac00740 4 client.16641 mds_command: new command op to 24569 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-03-24T00:22:45.174 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:22:45.171+0000 7fab8ac00740 1 -- 172.21.15.118:0/3539248607 --> [v2:172.21.15.189:6838/4074484168,v1:172.21.15.189:6839/4074484168] -- command(tid 0: {"prefix": "flush journal"}) v1 -- 0x559d13e5d5d0 con 0x559d13fe3ee0 2024-03-24T00:22:45.174 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:22:45.171+0000 7fab85e00640 1 --2- 172.21.15.118:0/3539248607 >> [v2:172.21.15.189:6838/4074484168,v1:172.21.15.189:6839/4074484168] conn(0x559d13fe3ee0 0x559d13fe62d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:22:45.174 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:22:45.172+0000 7fab85e00640 1 --2- 172.21.15.118:0/3539248607 >> [v2:172.21.15.189:6838/4074484168,v1:172.21.15.189:6839/4074484168] conn(0x559d13fe3ee0 0x559d13fe62d0 crc :-1 s=READY pgs=278 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=c77e0396a6466a1f server_cookie=487532326a91ab05 in_seq=0 out_seq=0 2024-03-24T00:22:45.175 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:22:45.172+0000 7fab66a00640 10 client.16641 ms_handle_connect on v2:172.21.15.189:6838/4074484168 2024-03-24T00:22:45.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:22:44 smithi118 ceph-mon[31535]: daemon mds.h restarted 2024-03-24T00:22:45.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:22:44 smithi118 ceph-mon[31535]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-03-24T00:22:45.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:22:44 smithi118 ceph-mon[31535]: osdmap e109: 12 total, 12 up, 12 in 2024-03-24T00:22:45.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:22:44 smithi118 ceph-mon[31535]: mds.? [v2:172.21.15.189:6836/635914573,v1:172.21.15.189:6837/635914573] up:boot 2024-03-24T00:22:45.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:22:44 smithi118 ceph-mon[31535]: Standby daemon mds.d assigned to filesystem cephfs as rank 3 2024-03-24T00:22:45.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:22:44 smithi118 ceph-mon[31535]: fsmap cephfs:4/5 {0=b=up:active,1=l=up:active,2=c=up:active,4=e=up:active} 5 up:standby-replay 2 up:standby, 1 failed 2024-03-24T00:22:45.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:22:44 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-03-24T00:22:45.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:22:44 smithi118 ceph-mon[31535]: fsmap cephfs:5/5 4 up:standby-replay 2 up:standby 1 up:replay 4 up:active 2024-03-24T00:22:45.203 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:22:44 smithi189 ceph-mon[36187]: daemon mds.h restarted 2024-03-24T00:22:45.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:22:44 smithi189 ceph-mon[36187]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-03-24T00:22:45.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:22:44 smithi189 ceph-mon[36187]: osdmap e109: 12 total, 12 up, 12 in 2024-03-24T00:22:45.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:22:44 smithi189 ceph-mon[36187]: mds.? [v2:172.21.15.189:6836/635914573,v1:172.21.15.189:6837/635914573] up:boot 2024-03-24T00:22:45.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:22:44 smithi189 ceph-mon[36187]: Standby daemon mds.d assigned to filesystem cephfs as rank 3 2024-03-24T00:22:45.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:22:44 smithi189 ceph-mon[36187]: fsmap cephfs:4/5 {0=b=up:active,1=l=up:active,2=c=up:active,4=e=up:active} 5 up:standby-replay 2 up:standby, 1 failed 2024-03-24T00:22:45.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:22:44 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-03-24T00:22:45.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:22:44 smithi189 ceph-mon[36187]: fsmap cephfs:5/5 4 up:standby-replay 2 up:standby 1 up:replay 4 up:active 2024-03-24T00:22:45.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:22:44 smithi193 ceph-mon[36878]: daemon mds.h restarted 2024-03-24T00:22:45.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:22:44 smithi193 ceph-mon[36878]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-03-24T00:22:45.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:22:44 smithi193 ceph-mon[36878]: osdmap e109: 12 total, 12 up, 12 in 2024-03-24T00:22:45.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:22:44 smithi193 ceph-mon[36878]: mds.? [v2:172.21.15.189:6836/635914573,v1:172.21.15.189:6837/635914573] up:boot 2024-03-24T00:22:45.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:22:44 smithi193 ceph-mon[36878]: Standby daemon mds.d assigned to filesystem cephfs as rank 3 2024-03-24T00:22:45.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:22:44 smithi193 ceph-mon[36878]: fsmap cephfs:4/5 {0=b=up:active,1=l=up:active,2=c=up:active,4=e=up:active} 5 up:standby-replay 2 up:standby, 1 failed 2024-03-24T00:22:45.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:22:44 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-03-24T00:22:45.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:22:44 smithi193 ceph-mon[36878]: fsmap cephfs:5/5 4 up:standby-replay 2 up:standby 1 up:replay 4 up:active 2024-03-24T00:22:46.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:22:45 smithi118 ceph-mon[31535]: pgmap v629: 97 pgs: 97 active+clean; 142 MiB data, 788 MiB used, 1.0 TiB / 1.0 TiB avail; 316 KiB/s rd, 68 KiB/s wr, 42 op/s 2024-03-24T00:22:46.203 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:22:45 smithi189 ceph-mon[36187]: pgmap v629: 97 pgs: 97 active+clean; 142 MiB data, 788 MiB used, 1.0 TiB / 1.0 TiB avail; 316 KiB/s rd, 68 KiB/s wr, 42 op/s 2024-03-24T00:22:46.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:22:45 smithi193 ceph-mon[36878]: pgmap v629: 97 pgs: 97 active+clean; 142 MiB data, 788 MiB used, 1.0 TiB / 1.0 TiB avail; 316 KiB/s rd, 68 KiB/s wr, 42 op/s 2024-03-24T00:22:47.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:22:46 smithi118 ceph-mon[31535]: mds.? [v2:172.21.15.118:6836/1374462956,v1:172.21.15.118:6837/1374462956] up:resolve 2024-03-24T00:22:47.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:22:46 smithi118 ceph-mon[31535]: fsmap cephfs:5/5 4 up:standby-replay 2 up:standby 1 up:resolve 4 up:active 2024-03-24T00:22:47.203 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:22:46 smithi189 ceph-mon[36187]: mds.? [v2:172.21.15.118:6836/1374462956,v1:172.21.15.118:6837/1374462956] up:resolve 2024-03-24T00:22:47.203 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:22:46 smithi189 ceph-mon[36187]: fsmap cephfs:5/5 4 up:standby-replay 2 up:standby 1 up:resolve 4 up:active 2024-03-24T00:22:47.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:22:46 smithi193 ceph-mon[36878]: mds.? [v2:172.21.15.118:6836/1374462956,v1:172.21.15.118:6837/1374462956] up:resolve 2024-03-24T00:22:47.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:22:46 smithi193 ceph-mon[36878]: fsmap cephfs:5/5 4 up:standby-replay 2 up:standby 1 up:resolve 4 up:active 2024-03-24T00:22:48.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:22:47 smithi118 ceph-mon[31535]: pgmap v630: 97 pgs: 97 active+clean; 142 MiB data, 788 MiB used, 1.0 TiB / 1.0 TiB avail; 58 KiB/s rd, 21 KiB/s wr, 21 op/s 2024-03-24T00:22:48.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:22:47 smithi189 ceph-mon[36187]: pgmap v630: 97 pgs: 97 active+clean; 142 MiB data, 788 MiB used, 1.0 TiB / 1.0 TiB avail; 58 KiB/s rd, 21 KiB/s wr, 21 op/s 2024-03-24T00:22:48.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:22:47 smithi193 ceph-mon[36878]: pgmap v630: 97 pgs: 97 active+clean; 142 MiB data, 788 MiB used, 1.0 TiB / 1.0 TiB avail; 58 KiB/s rd, 21 KiB/s wr, 21 op/s 2024-03-24T00:22:50.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:22:49 smithi118 ceph-mon[31535]: pgmap v631: 97 pgs: 97 active+clean; 142 MiB data, 788 MiB used, 1.0 TiB / 1.0 TiB avail; 6.1 KiB/s rd, 19 KiB/s wr, 21 op/s 2024-03-24T00:22:50.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:22:49 smithi189 ceph-mon[36187]: pgmap v631: 97 pgs: 97 active+clean; 142 MiB data, 788 MiB used, 1.0 TiB / 1.0 TiB avail; 6.1 KiB/s rd, 19 KiB/s wr, 21 op/s 2024-03-24T00:22:50.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:22:49 smithi193 ceph-mon[36878]: pgmap v631: 97 pgs: 97 active+clean; 142 MiB data, 788 MiB used, 1.0 TiB / 1.0 TiB avail; 6.1 KiB/s rd, 19 KiB/s wr, 21 op/s 2024-03-24T00:22:51.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:22:51 smithi118 ceph-mon[31535]: Replacing daemon mds.l as rank 1 with standby daemon mds.g 2024-03-24T00:22:51.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:22:51 smithi118 ceph-mon[31535]: MDS daemon mds.l is removed because it is dead or otherwise unavailable. 2024-03-24T00:22:51.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:22:51 smithi118 ceph-mon[31535]: osdmap e110: 12 total, 12 up, 12 in 2024-03-24T00:22:51.947 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:22:51 smithi118 ceph-mon[31535]: fsmap cephfs:5/5 3 up:standby-replay 2 up:standby 1 up:replay 1 up:resolve 3 up:active 2024-03-24T00:22:51.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:22:51 smithi189 ceph-mon[36187]: Replacing daemon mds.l as rank 1 with standby daemon mds.g 2024-03-24T00:22:51.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:22:51 smithi189 ceph-mon[36187]: MDS daemon mds.l is removed because it is dead or otherwise unavailable. 2024-03-24T00:22:51.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:22:51 smithi189 ceph-mon[36187]: osdmap e110: 12 total, 12 up, 12 in 2024-03-24T00:22:51.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:22:51 smithi189 ceph-mon[36187]: fsmap cephfs:5/5 3 up:standby-replay 2 up:standby 1 up:replay 1 up:resolve 3 up:active 2024-03-24T00:22:51.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:22:51 smithi193 ceph-mon[36878]: Replacing daemon mds.l as rank 1 with standby daemon mds.g 2024-03-24T00:22:51.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:22:51 smithi193 ceph-mon[36878]: MDS daemon mds.l is removed because it is dead or otherwise unavailable. 2024-03-24T00:22:51.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:22:51 smithi193 ceph-mon[36878]: osdmap e110: 12 total, 12 up, 12 in 2024-03-24T00:22:51.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:22:51 smithi193 ceph-mon[36878]: fsmap cephfs:5/5 3 up:standby-replay 2 up:standby 1 up:replay 1 up:resolve 3 up:active 2024-03-24T00:22:52.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:22:52 smithi118 ceph-mon[31535]: pgmap v633: 97 pgs: 97 active+clean; 142 MiB data, 788 MiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 24 KiB/s wr, 23 op/s 2024-03-24T00:22:52.953 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:22:52 smithi189 ceph-mon[36187]: pgmap v633: 97 pgs: 97 active+clean; 142 MiB data, 788 MiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 24 KiB/s wr, 23 op/s 2024-03-24T00:22:52.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:22:52 smithi193 ceph-mon[36878]: pgmap v633: 97 pgs: 97 active+clean; 142 MiB data, 788 MiB used, 1.0 TiB / 1.0 TiB avail; 6.5 KiB/s rd, 24 KiB/s wr, 23 op/s 2024-03-24T00:22:53.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:22:53 smithi118 ceph-mon[31535]: mds.? [v2:172.21.15.118:6838/1710625324,v1:172.21.15.118:6839/1710625324] up:resolve 2024-03-24T00:22:53.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:22:53 smithi118 ceph-mon[31535]: fsmap cephfs:5/5 3 up:standby-replay 2 up:standby 2 up:resolve 3 up:active 2024-03-24T00:22:53.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:22:53 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T00:22:53.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:22:53 smithi189 ceph-mon[36187]: mds.? [v2:172.21.15.118:6838/1710625324,v1:172.21.15.118:6839/1710625324] up:resolve 2024-03-24T00:22:53.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:22:53 smithi189 ceph-mon[36187]: fsmap cephfs:5/5 3 up:standby-replay 2 up:standby 2 up:resolve 3 up:active 2024-03-24T00:22:53.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:22:53 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T00:22:53.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:22:53 smithi193 ceph-mon[36878]: mds.? [v2:172.21.15.118:6838/1710625324,v1:172.21.15.118:6839/1710625324] up:resolve 2024-03-24T00:22:53.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:22:53 smithi193 ceph-mon[36878]: fsmap cephfs:5/5 3 up:standby-replay 2 up:standby 2 up:resolve 3 up:active 2024-03-24T00:22:53.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:22:53 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T00:22:54.681 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:22:54.680+0000 7fab66a00640 1 -- 172.21.15.118:0/3539248607 <== mds.0 v2:172.21.15.189:6838/4074484168 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+44 (crc 0 0 0) 0x559d13e5d5d0 con 0x559d13fe3ee0 2024-03-24T00:22:54.681 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:22:54.680+0000 7fab66a00640 10 client.16641 handle_command_reply: tid=0 2024-03-24T00:22:54.681 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:22:54.680+0000 7fab8ac00740 1 client.16641 shutdown 2024-03-24T00:22:54.681 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:22:54.680+0000 7fab8ac00740 1 -- 172.21.15.118:0/3539248607 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x559d13f72bd0 msgr2=0x559d13f80f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:22:54.682 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:22:54.680+0000 7fab8ac00740 1 --2- 172.21.15.118:0/3539248607 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x559d13f72bd0 0x559d13f80f50 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7fab6c0059d0 tx=0x7fab6c005d50 comp rx=0 tx=0).stop 2024-03-24T00:22:54.682 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:22:54.681+0000 7fab8ac00740 1 -- 172.21.15.118:0/3539248607 shutdown_connections 2024-03-24T00:22:54.682 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:22:54.681+0000 7fab8ac00740 1 --2- 172.21.15.118:0/3539248607 >> [v2:172.21.15.189:6838/4074484168,v1:172.21.15.189:6839/4074484168] conn(0x559d13fe3ee0 0x559d13fe62d0 crc :-1 s=READY pgs=278 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:22:54.682 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:22:54.681+0000 7fab8ac00740 1 --2- 172.21.15.118:0/3539248607 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x559d13f77d40 0x559d13f81f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:22:54.682 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:22:54.681+0000 7fab66a00640 0 client.16641 ms_handle_reset on v2:172.21.15.189:6838/4074484168 2024-03-24T00:22:54.682 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:22:54.681+0000 7fab8ac00740 1 --2- 172.21.15.118:0/3539248607 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x559d13f72bd0 0x559d13f80f50 unknown :-1 s=CLOSED pgs=219 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:22:54.682 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:22:54.681+0000 7fab8ac00740 1 --2- 172.21.15.118:0/3539248607 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x559d13f721e0 0x559d13f80a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:22:54.682 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:22:54.681+0000 7fab8ac00740 1 -- 172.21.15.118:0/3539248607 >> 172.21.15.118:0/3539248607 conn(0x559d13f6d750 msgr2=0x559d13f73e50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:22:54.682 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:22:54.681+0000 7fab8ac00740 1 -- 172.21.15.118:0/3539248607 shutdown_connections 2024-03-24T00:22:54.683 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:22:54.681+0000 7fab8ac00740 1 -- 172.21.15.118:0/3539248607 wait complete. 2024-03-24T00:22:54.683 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:22:54.681+0000 7fab8ac00740 20 client.16641 trim_cache size 0 max 16384 2024-03-24T00:22:54.684 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stdout:{ 2024-03-24T00:22:54.684 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stdout: "message": "", 2024-03-24T00:22:54.684 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stdout: "return_code": 0 2024-03-24T00:22:54.684 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stdout:} 2024-03-24T00:22:54.685 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:22:54.682+0000 7fab87e00640 1 -- 172.21.15.118:0/2359644756 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7fab680a2920 msgr2=0x7fab680a4de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:22:54.685 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:22:54.682+0000 7fab87e00640 1 --2- 172.21.15.118:0/2359644756 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7fab680a2920 0x7fab680a4de0 secure :-1 s=READY pgs=379 cs=0 l=1 rev1=1 crypto rx=0x7fab740980a0 tx=0x7fab740097d0 comp rx=0 tx=0).stop 2024-03-24T00:22:54.685 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:22:54.682+0000 7fab87e00640 1 -- 172.21.15.118:0/2359644756 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fab801086a0 msgr2=0x7fab801a2330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:22:54.685 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:22:54.682+0000 7fab87e00640 1 --2- 172.21.15.118:0/2359644756 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fab801086a0 0x7fab801a2330 secure :-1 s=READY pgs=442 cs=0 l=1 rev1=1 crypto rx=0x7fab7400e9c0 tx=0x7fab7400ee80 comp rx=0 tx=0).stop 2024-03-24T00:22:54.685 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:22:54.682+0000 7fab87e00640 1 -- 172.21.15.118:0/2359644756 shutdown_connections 2024-03-24T00:22:54.685 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:22:54.682+0000 7fab87e00640 1 --2- 172.21.15.118:0/2359644756 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7fab680a2920 0x7fab680a4de0 unknown :-1 s=CLOSED pgs=379 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:22:54.685 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:22:54.682+0000 7fab87e00640 1 --2- 172.21.15.118:0/2359644756 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fab8010ff20 0x7fab801a98e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:22:54.685 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:22:54.682+0000 7fab87e00640 1 --2- 172.21.15.118:0/2359644756 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fab80109000 0x7fab801a2870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:22:54.685 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:22:54.682+0000 7fab87e00640 1 --2- 172.21.15.118:0/2359644756 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fab801086a0 0x7fab801a2330 unknown :-1 s=CLOSED pgs=442 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:22:54.685 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:22:54.682+0000 7fab87e00640 1 -- 172.21.15.118:0/2359644756 >> 172.21.15.118:0/2359644756 conn(0x7fab8006e8a0 msgr2=0x7fab8006ec80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:22:54.685 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:22:54.682+0000 7fab87e00640 1 -- 172.21.15.118:0/2359644756 shutdown_connections 2024-03-24T00:22:54.685 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:22:54.682+0000 7fab87e00640 1 -- 172.21.15.118:0/2359644756 wait complete. 2024-03-24T00:22:54.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:22:54 smithi118 ceph-mon[31535]: mds.? [v2:172.21.15.118:6836/1374462956,v1:172.21.15.118:6837/1374462956] up:reconnect 2024-03-24T00:22:54.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:22:54 smithi118 ceph-mon[31535]: mds.? [v2:172.21.15.118:6838/1710625324,v1:172.21.15.118:6839/1710625324] up:reconnect 2024-03-24T00:22:54.947 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:22:54 smithi118 ceph-mon[31535]: fsmap cephfs:5/5 3 up:standby-replay 2 up:standby 2 up:reconnect 3 up:active 2024-03-24T00:22:54.947 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:22:54 smithi118 ceph-mon[31535]: reconnect by client.15159 192.168.0.1:0/3463739313 after 0.0129997 2024-03-24T00:22:54.947 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:22:54 smithi118 ceph-mon[31535]: reconnect by client.15159 192.168.0.1:0/3463739313 after 0.0219995 2024-03-24T00:22:54.947 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:22:54 smithi118 ceph-mon[31535]: pgmap v634: 97 pgs: 97 active+clean; 142 MiB data, 788 MiB used, 1.0 TiB / 1.0 TiB avail; 5.3 KiB/s rd, 21 KiB/s wr, 21 op/s 2024-03-24T00:22:54.947 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:22:54 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-24T00:22:54.947 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:22:54 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-24T00:22:54.947 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:22:54 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-03-24T00:22:54.947 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:22:54 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-03-24T00:22:54.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:22:54 smithi189 ceph-mon[36187]: mds.? [v2:172.21.15.118:6836/1374462956,v1:172.21.15.118:6837/1374462956] up:reconnect 2024-03-24T00:22:54.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:22:54 smithi189 ceph-mon[36187]: mds.? [v2:172.21.15.118:6838/1710625324,v1:172.21.15.118:6839/1710625324] up:reconnect 2024-03-24T00:22:54.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:22:54 smithi189 ceph-mon[36187]: fsmap cephfs:5/5 3 up:standby-replay 2 up:standby 2 up:reconnect 3 up:active 2024-03-24T00:22:54.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:22:54 smithi189 ceph-mon[36187]: reconnect by client.15159 192.168.0.1:0/3463739313 after 0.0129997 2024-03-24T00:22:54.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:22:54 smithi189 ceph-mon[36187]: reconnect by client.15159 192.168.0.1:0/3463739313 after 0.0219995 2024-03-24T00:22:54.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:22:54 smithi189 ceph-mon[36187]: pgmap v634: 97 pgs: 97 active+clean; 142 MiB data, 788 MiB used, 1.0 TiB / 1.0 TiB avail; 5.3 KiB/s rd, 21 KiB/s wr, 21 op/s 2024-03-24T00:22:54.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:22:54 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-24T00:22:54.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:22:54 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-24T00:22:54.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:22:54 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-03-24T00:22:54.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:22:54 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-03-24T00:22:54.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:22:54 smithi193 ceph-mon[36878]: mds.? [v2:172.21.15.118:6836/1374462956,v1:172.21.15.118:6837/1374462956] up:reconnect 2024-03-24T00:22:54.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:22:54 smithi193 ceph-mon[36878]: mds.? [v2:172.21.15.118:6838/1710625324,v1:172.21.15.118:6839/1710625324] up:reconnect 2024-03-24T00:22:54.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:22:54 smithi193 ceph-mon[36878]: fsmap cephfs:5/5 3 up:standby-replay 2 up:standby 2 up:reconnect 3 up:active 2024-03-24T00:22:54.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:22:54 smithi193 ceph-mon[36878]: reconnect by client.15159 192.168.0.1:0/3463739313 after 0.0129997 2024-03-24T00:22:54.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:22:54 smithi193 ceph-mon[36878]: reconnect by client.15159 192.168.0.1:0/3463739313 after 0.0219995 2024-03-24T00:22:54.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:22:54 smithi193 ceph-mon[36878]: pgmap v634: 97 pgs: 97 active+clean; 142 MiB data, 788 MiB used, 1.0 TiB / 1.0 TiB avail; 5.3 KiB/s rd, 21 KiB/s wr, 21 op/s 2024-03-24T00:22:54.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:22:54 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-24T00:22:54.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:22:54 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-24T00:22:54.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:22:54 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-03-24T00:22:54.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:22:54 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-03-24T00:22:55.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:22:55 smithi118 ceph-mon[31535]: Adjusting osd_memory_target on smithi193 to 174.1M 2024-03-24T00:22:55.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:22:55 smithi118 ceph-mon[31535]: Unable to set osd_memory_target on smithi193 to 182639923: error parsing value: Value '182639923' is below minimum 939524096 2024-03-24T00:22:55.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:22:55 smithi118 ceph-mon[31535]: mds.? [v2:172.21.15.118:6838/1710625324,v1:172.21.15.118:6839/1710625324] up:rejoin 2024-03-24T00:22:55.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:22:55 smithi118 ceph-mon[31535]: mds.? [v2:172.21.15.118:6836/1374462956,v1:172.21.15.118:6837/1374462956] up:rejoin 2024-03-24T00:22:55.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:22:55 smithi118 ceph-mon[31535]: fsmap cephfs:5/5 3 up:standby-replay 2 up:standby 2 up:rejoin 3 up:active 2024-03-24T00:22:55.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:22:55 smithi118 ceph-mon[31535]: daemon mds.d is now active in filesystem cephfs as rank 3 2024-03-24T00:22:55.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:22:55 smithi118 ceph-mon[31535]: daemon mds.g is now active in filesystem cephfs as rank 1 2024-03-24T00:22:55.947 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:22:55 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T00:22:55.947 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:22:55 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T00:22:55.947 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:22:55 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T00:22:55.947 INFO:journalctl@ceph.mds.g.smithi118.stdout:Mar 24 00:22:55 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-g[94891]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: In function 'void Locker::set_xlocks_done(MutationImpl*, bool)' thread 7ff444e00700 time 2024-03-24T00:22:55.785567+0000 2024-03-24T00:22:55.947 INFO:journalctl@ceph.mds.g.smithi118.stdout:Mar 24 00:22:55 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-g[94891]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: 626: FAILED ceph_assert(obj->is_auth()) 2024-03-24T00:22:55.947 INFO:journalctl@ceph.mds.g.smithi118.stdout:Mar 24 00:22:55 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-g[94891]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-24T00:22:55.947 INFO:journalctl@ceph.mds.g.smithi118.stdout:Mar 24 00:22:55 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-g[94891]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x135) [0x7ff4506164a1] 2024-03-24T00:22:55.947 INFO:journalctl@ceph.mds.g.smithi118.stdout:Mar 24 00:22:55 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-g[94891]: 2: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7ff450616667] 2024-03-24T00:22:55.947 INFO:journalctl@ceph.mds.g.smithi118.stdout:Mar 24 00:22:55 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-g[94891]: 3: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x555967881223] 2024-03-24T00:22:55.948 INFO:journalctl@ceph.mds.g.smithi118.stdout:Mar 24 00:22:55 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-g[94891]: 4: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x5559676fb5d6] 2024-03-24T00:22:55.948 INFO:journalctl@ceph.mds.g.smithi118.stdout:Mar 24 00:22:55 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-g[94891]: 5: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x5559676fbda1] 2024-03-24T00:22:55.948 INFO:journalctl@ceph.mds.g.smithi118.stdout:Mar 24 00:22:55 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-g[94891]: 6: (Server::handle_client_rename(boost::intrusive_ptr const&)+0x259d) [0x55596773897d] 2024-03-24T00:22:55.948 INFO:journalctl@ceph.mds.g.smithi118.stdout:Mar 24 00:22:55 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-g[94891]: 7: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x55596774e233] 2024-03-24T00:22:55.948 INFO:journalctl@ceph.mds.g.smithi118.stdout:Mar 24 00:22:55 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-g[94891]: 8: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x555967750197] 2024-03-24T00:22:55.948 INFO:journalctl@ceph.mds.g.smithi118.stdout:Mar 24 00:22:55 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-g[94891]: 9: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x55596764f11b] 2024-03-24T00:22:55.948 INFO:journalctl@ceph.mds.g.smithi118.stdout:Mar 24 00:22:55 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-g[94891]: 10: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x55596764f6bc] 2024-03-24T00:22:55.948 INFO:journalctl@ceph.mds.g.smithi118.stdout:Mar 24 00:22:55 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-g[94891]: 11: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x5559676391ff] 2024-03-24T00:22:55.948 INFO:journalctl@ceph.mds.g.smithi118.stdout:Mar 24 00:22:55 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-g[94891]: 12: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7ff4508a8488] 2024-03-24T00:22:55.948 INFO:journalctl@ceph.mds.g.smithi118.stdout:Mar 24 00:22:55 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-g[94891]: 13: (DispatchQueue::entry()+0x51f) [0x7ff4508a560f] 2024-03-24T00:22:55.948 INFO:journalctl@ceph.mds.g.smithi118.stdout:Mar 24 00:22:55 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-g[94891]: 14: (DispatchQueue::DispatchThread::entry()+0x11) [0x7ff450978e81] 2024-03-24T00:22:55.948 INFO:journalctl@ceph.mds.g.smithi118.stdout:Mar 24 00:22:55 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-g[94891]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7ff44f3931ca] 2024-03-24T00:22:55.948 INFO:journalctl@ceph.mds.g.smithi118.stdout:Mar 24 00:22:55 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-g[94891]: 16: clone() 2024-03-24T00:22:55.949 INFO:journalctl@ceph.mds.d.smithi118.stdout:Mar 24 00:22:55 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-d[72983]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: In function 'void Locker::set_xlocks_done(MutationImpl*, bool)' thread 7f3585000700 time 2024-03-24T00:22:55.719746+0000 2024-03-24T00:22:55.949 INFO:journalctl@ceph.mds.d.smithi118.stdout:Mar 24 00:22:55 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-d[72983]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: 626: FAILED ceph_assert(obj->is_auth()) 2024-03-24T00:22:55.949 INFO:journalctl@ceph.mds.d.smithi118.stdout:Mar 24 00:22:55 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-d[72983]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-24T00:22:55.949 INFO:journalctl@ceph.mds.d.smithi118.stdout:Mar 24 00:22:55 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-d[72983]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x135) [0x7f35908054a1] 2024-03-24T00:22:55.949 INFO:journalctl@ceph.mds.d.smithi118.stdout:Mar 24 00:22:55 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-d[72983]: 2: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7f3590805667] 2024-03-24T00:22:55.949 INFO:journalctl@ceph.mds.d.smithi118.stdout:Mar 24 00:22:55 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-d[72983]: 3: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x555eae6fa223] 2024-03-24T00:22:55.949 INFO:journalctl@ceph.mds.d.smithi118.stdout:Mar 24 00:22:55 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-d[72983]: 4: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x555eae5745d6] 2024-03-24T00:22:55.949 INFO:journalctl@ceph.mds.d.smithi118.stdout:Mar 24 00:22:55 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-d[72983]: 5: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x555eae574da1] 2024-03-24T00:22:55.949 INFO:journalctl@ceph.mds.d.smithi118.stdout:Mar 24 00:22:55 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-d[72983]: 6: (Server::handle_client_rename(boost::intrusive_ptr const&)+0x259d) [0x555eae5b197d] 2024-03-24T00:22:55.949 INFO:journalctl@ceph.mds.d.smithi118.stdout:Mar 24 00:22:55 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-d[72983]: 7: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x555eae5c7233] 2024-03-24T00:22:55.949 INFO:journalctl@ceph.mds.d.smithi118.stdout:Mar 24 00:22:55 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-d[72983]: 8: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x555eae5c9197] 2024-03-24T00:22:55.949 INFO:journalctl@ceph.mds.d.smithi118.stdout:Mar 24 00:22:55 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-d[72983]: 9: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x555eae4c811b] 2024-03-24T00:22:55.950 INFO:journalctl@ceph.mds.d.smithi118.stdout:Mar 24 00:22:55 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-d[72983]: 10: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x555eae4c86bc] 2024-03-24T00:22:55.950 INFO:journalctl@ceph.mds.d.smithi118.stdout:Mar 24 00:22:55 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-d[72983]: 11: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x555eae4b21ff] 2024-03-24T00:22:55.950 INFO:journalctl@ceph.mds.d.smithi118.stdout:Mar 24 00:22:55 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-d[72983]: 12: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7f3590a97488] 2024-03-24T00:22:55.950 INFO:journalctl@ceph.mds.d.smithi118.stdout:Mar 24 00:22:55 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-d[72983]: 13: (DispatchQueue::entry()+0x51f) [0x7f3590a9460f] 2024-03-24T00:22:55.950 INFO:journalctl@ceph.mds.d.smithi118.stdout:Mar 24 00:22:55 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-d[72983]: 14: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f3590b67e81] 2024-03-24T00:22:55.950 INFO:journalctl@ceph.mds.d.smithi118.stdout:Mar 24 00:22:55 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-d[72983]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7f358f5821ca] 2024-03-24T00:22:55.950 INFO:journalctl@ceph.mds.d.smithi118.stdout:Mar 24 00:22:55 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-d[72983]: 16: clone() 2024-03-24T00:22:55.950 INFO:journalctl@ceph.mds.g.smithi118.stdout:Mar 24 00:22:55 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-g[94891]: 2024-03-24T00:22:55.786+0000 7ff444e00700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: In function 'void Locker::set_xlocks_done(MutationImpl*, bool)' thread 7ff444e00700 time 2024-03-24T00:22:55.785567+0000 2024-03-24T00:22:55.950 INFO:journalctl@ceph.mds.g.smithi118.stdout:Mar 24 00:22:55 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-g[94891]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: 626: FAILED ceph_assert(obj->is_auth()) 2024-03-24T00:22:55.951 INFO:journalctl@ceph.mds.g.smithi118.stdout:Mar 24 00:22:55 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-g[94891]: 2024-03-24T00:22:55.951 INFO:journalctl@ceph.mds.g.smithi118.stdout:Mar 24 00:22:55 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-g[94891]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-24T00:22:55.951 INFO:journalctl@ceph.mds.g.smithi118.stdout:Mar 24 00:22:55 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-g[94891]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x135) [0x7ff4506164a1] 2024-03-24T00:22:55.951 INFO:journalctl@ceph.mds.g.smithi118.stdout:Mar 24 00:22:55 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-g[94891]: 2: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7ff450616667] 2024-03-24T00:22:55.951 INFO:journalctl@ceph.mds.g.smithi118.stdout:Mar 24 00:22:55 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-g[94891]: 3: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x555967881223] 2024-03-24T00:22:55.951 INFO:journalctl@ceph.mds.g.smithi118.stdout:Mar 24 00:22:55 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-g[94891]: 4: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x5559676fb5d6] 2024-03-24T00:22:55.951 INFO:journalctl@ceph.mds.g.smithi118.stdout:Mar 24 00:22:55 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-g[94891]: 5: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x5559676fbda1] 2024-03-24T00:22:55.951 INFO:journalctl@ceph.mds.g.smithi118.stdout:Mar 24 00:22:55 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-g[94891]: 6: (Server::handle_client_rename(boost::intrusive_ptr const&)+0x259d) [0x55596773897d] 2024-03-24T00:22:55.951 INFO:journalctl@ceph.mds.g.smithi118.stdout:Mar 24 00:22:55 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-g[94891]: 7: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x55596774e233] 2024-03-24T00:22:55.951 INFO:journalctl@ceph.mds.g.smithi118.stdout:Mar 24 00:22:55 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-g[94891]: 8: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x555967750197] 2024-03-24T00:22:55.951 INFO:journalctl@ceph.mds.g.smithi118.stdout:Mar 24 00:22:55 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-g[94891]: 9: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x55596764f11b] 2024-03-24T00:22:55.951 INFO:journalctl@ceph.mds.g.smithi118.stdout:Mar 24 00:22:55 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-g[94891]: 10: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x55596764f6bc] 2024-03-24T00:22:55.951 INFO:journalctl@ceph.mds.g.smithi118.stdout:Mar 24 00:22:55 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-g[94891]: 11: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x5559676391ff] 2024-03-24T00:22:55.951 INFO:journalctl@ceph.mds.g.smithi118.stdout:Mar 24 00:22:55 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-g[94891]: 12: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7ff4508a8488] 2024-03-24T00:22:55.952 INFO:journalctl@ceph.mds.g.smithi118.stdout:Mar 24 00:22:55 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-g[94891]: 13: (DispatchQueue::entry()+0x51f) [0x7ff4508a560f] 2024-03-24T00:22:55.952 INFO:journalctl@ceph.mds.g.smithi118.stdout:Mar 24 00:22:55 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-g[94891]: 14: (DispatchQueue::DispatchThread::entry()+0x11) [0x7ff450978e81] 2024-03-24T00:22:55.952 INFO:journalctl@ceph.mds.g.smithi118.stdout:Mar 24 00:22:55 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-g[94891]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7ff44f3931ca] 2024-03-24T00:22:55.952 INFO:journalctl@ceph.mds.g.smithi118.stdout:Mar 24 00:22:55 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-g[94891]: 16: clone() 2024-03-24T00:22:55.952 INFO:journalctl@ceph.mds.g.smithi118.stdout:Mar 24 00:22:55 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-g[94891]: 2024-03-24T00:22:55.952 INFO:journalctl@ceph.mds.g.smithi118.stdout:Mar 24 00:22:55 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-g[94891]: *** Caught signal (Aborted) ** 2024-03-24T00:22:55.952 INFO:journalctl@ceph.mds.g.smithi118.stdout:Mar 24 00:22:55 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-g[94891]: in thread 7ff444e00700 thread_name:ms_dispatch 2024-03-24T00:22:55.952 INFO:journalctl@ceph.mds.g.smithi118.stdout:Mar 24 00:22:55 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-g[94891]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-24T00:22:55.952 INFO:journalctl@ceph.mds.g.smithi118.stdout:Mar 24 00:22:55 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-g[94891]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7ff44f39dd20] 2024-03-24T00:22:55.953 INFO:journalctl@ceph.mds.d.smithi118.stdout:Mar 24 00:22:55 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-d[72983]: 2024-03-24T00:22:55.721+0000 7f3585000700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: In function 'void Locker::set_xlocks_done(MutationImpl*, bool)' thread 7f3585000700 time 2024-03-24T00:22:55.719746+0000 2024-03-24T00:22:55.953 INFO:journalctl@ceph.mds.d.smithi118.stdout:Mar 24 00:22:55 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-d[72983]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: 626: FAILED ceph_assert(obj->is_auth()) 2024-03-24T00:22:55.953 INFO:journalctl@ceph.mds.d.smithi118.stdout:Mar 24 00:22:55 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-d[72983]: 2024-03-24T00:22:55.953 INFO:journalctl@ceph.mds.d.smithi118.stdout:Mar 24 00:22:55 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-d[72983]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-24T00:22:55.953 INFO:journalctl@ceph.mds.d.smithi118.stdout:Mar 24 00:22:55 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-d[72983]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x135) [0x7f35908054a1] 2024-03-24T00:22:55.953 INFO:journalctl@ceph.mds.d.smithi118.stdout:Mar 24 00:22:55 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-d[72983]: 2: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7f3590805667] 2024-03-24T00:22:55.953 INFO:journalctl@ceph.mds.d.smithi118.stdout:Mar 24 00:22:55 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-d[72983]: 3: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x555eae6fa223] 2024-03-24T00:22:55.953 INFO:journalctl@ceph.mds.d.smithi118.stdout:Mar 24 00:22:55 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-d[72983]: 4: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x555eae5745d6] 2024-03-24T00:22:55.953 INFO:journalctl@ceph.mds.d.smithi118.stdout:Mar 24 00:22:55 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-d[72983]: 5: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x555eae574da1] 2024-03-24T00:22:55.953 INFO:journalctl@ceph.mds.d.smithi118.stdout:Mar 24 00:22:55 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-d[72983]: 6: (Server::handle_client_rename(boost::intrusive_ptr const&)+0x259d) [0x555eae5b197d] 2024-03-24T00:22:55.953 INFO:journalctl@ceph.mds.d.smithi118.stdout:Mar 24 00:22:55 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-d[72983]: 7: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x555eae5c7233] 2024-03-24T00:22:55.954 INFO:journalctl@ceph.mds.d.smithi118.stdout:Mar 24 00:22:55 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-d[72983]: 8: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x555eae5c9197] 2024-03-24T00:22:55.954 INFO:journalctl@ceph.mds.d.smithi118.stdout:Mar 24 00:22:55 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-d[72983]: 9: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x555eae4c811b] 2024-03-24T00:22:55.954 INFO:journalctl@ceph.mds.d.smithi118.stdout:Mar 24 00:22:55 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-d[72983]: 10: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x555eae4c86bc] 2024-03-24T00:22:55.954 INFO:journalctl@ceph.mds.d.smithi118.stdout:Mar 24 00:22:55 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-d[72983]: 11: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x555eae4b21ff] 2024-03-24T00:22:55.954 INFO:journalctl@ceph.mds.d.smithi118.stdout:Mar 24 00:22:55 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-d[72983]: 12: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7f3590a97488] 2024-03-24T00:22:55.954 INFO:journalctl@ceph.mds.d.smithi118.stdout:Mar 24 00:22:55 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-d[72983]: 13: (DispatchQueue::entry()+0x51f) [0x7f3590a9460f] 2024-03-24T00:22:55.954 INFO:journalctl@ceph.mds.d.smithi118.stdout:Mar 24 00:22:55 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-d[72983]: 14: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f3590b67e81] 2024-03-24T00:22:55.954 INFO:journalctl@ceph.mds.d.smithi118.stdout:Mar 24 00:22:55 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-d[72983]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7f358f5821ca] 2024-03-24T00:22:55.954 INFO:journalctl@ceph.mds.d.smithi118.stdout:Mar 24 00:22:55 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-d[72983]: 16: clone() 2024-03-24T00:22:55.954 INFO:journalctl@ceph.mds.d.smithi118.stdout:Mar 24 00:22:55 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-d[72983]: 2024-03-24T00:22:55.954 INFO:journalctl@ceph.mds.d.smithi118.stdout:Mar 24 00:22:55 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-d[72983]: *** Caught signal (Aborted) ** 2024-03-24T00:22:55.954 INFO:journalctl@ceph.mds.d.smithi118.stdout:Mar 24 00:22:55 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-d[72983]: in thread 7f3585000700 thread_name:ms_dispatch 2024-03-24T00:22:55.955 INFO:journalctl@ceph.mds.d.smithi118.stdout:Mar 24 00:22:55 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-d[72983]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-24T00:22:55.955 INFO:journalctl@ceph.mds.d.smithi118.stdout:Mar 24 00:22:55 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-d[72983]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f358f58cd20] 2024-03-24T00:22:55.955 INFO:journalctl@ceph.mds.d.smithi118.stdout:Mar 24 00:22:55 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-d[72983]: 2: gsignal() 2024-03-24T00:22:55.955 INFO:journalctl@ceph.mds.d.smithi118.stdout:Mar 24 00:22:55 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-d[72983]: 3: abort() 2024-03-24T00:22:55.955 INFO:journalctl@ceph.mds.d.smithi118.stdout:Mar 24 00:22:55 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-d[72983]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x18f) [0x7f35908054fb] 2024-03-24T00:22:55.955 INFO:journalctl@ceph.mds.d.smithi118.stdout:Mar 24 00:22:55 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-d[72983]: 5: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7f3590805667] 2024-03-24T00:22:55.955 INFO:journalctl@ceph.mds.d.smithi118.stdout:Mar 24 00:22:55 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-d[72983]: 6: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x555eae6fa223] 2024-03-24T00:22:55.955 INFO:journalctl@ceph.mds.d.smithi118.stdout:Mar 24 00:22:55 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-d[72983]: 7: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x555eae5745d6] 2024-03-24T00:22:55.955 INFO:journalctl@ceph.mds.d.smithi118.stdout:Mar 24 00:22:55 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-d[72983]: 8: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x555eae574da1] 2024-03-24T00:22:55.955 INFO:journalctl@ceph.mds.d.smithi118.stdout:Mar 24 00:22:55 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-d[72983]: 9: (Server::handle_client_rename(boost::intrusive_ptr const&)+0x259d) [0x555eae5b197d] 2024-03-24T00:22:55.955 INFO:journalctl@ceph.mds.d.smithi118.stdout:Mar 24 00:22:55 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-d[72983]: 10: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x555eae5c7233] 2024-03-24T00:22:55.955 INFO:journalctl@ceph.mds.d.smithi118.stdout:Mar 24 00:22:55 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-d[72983]: 11: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x555eae5c9197] 2024-03-24T00:22:55.955 INFO:journalctl@ceph.mds.d.smithi118.stdout:Mar 24 00:22:55 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-d[72983]: 12: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x555eae4c811b] 2024-03-24T00:22:55.955 INFO:journalctl@ceph.mds.d.smithi118.stdout:Mar 24 00:22:55 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-d[72983]: 13: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x555eae4c86bc] 2024-03-24T00:22:55.955 INFO:journalctl@ceph.mds.d.smithi118.stdout:Mar 24 00:22:55 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-d[72983]: 14: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x555eae4b21ff] 2024-03-24T00:22:55.956 INFO:journalctl@ceph.mds.d.smithi118.stdout:Mar 24 00:22:55 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-d[72983]: 15: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7f3590a97488] 2024-03-24T00:22:55.956 INFO:journalctl@ceph.mds.d.smithi118.stdout:Mar 24 00:22:55 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-d[72983]: 16: (DispatchQueue::entry()+0x51f) [0x7f3590a9460f] 2024-03-24T00:22:55.956 INFO:journalctl@ceph.mds.d.smithi118.stdout:Mar 24 00:22:55 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-d[72983]: 17: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f3590b67e81] 2024-03-24T00:22:55.956 INFO:journalctl@ceph.mds.d.smithi118.stdout:Mar 24 00:22:55 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-d[72983]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7f358f5821ca] 2024-03-24T00:22:55.956 INFO:journalctl@ceph.mds.d.smithi118.stdout:Mar 24 00:22:55 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-d[72983]: 19: clone() 2024-03-24T00:22:55.956 INFO:journalctl@ceph.mds.d.smithi118.stdout:Mar 24 00:22:55 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-d[72983]: 2024-03-24T00:22:55.724+0000 7f3585000700 -1 *** Caught signal (Aborted) ** 2024-03-24T00:22:55.956 INFO:journalctl@ceph.mds.d.smithi118.stdout:Mar 24 00:22:55 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-d[72983]: in thread 7f3585000700 thread_name:ms_dispatch 2024-03-24T00:22:55.956 INFO:journalctl@ceph.mds.d.smithi118.stdout:Mar 24 00:22:55 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-d[72983]: 2024-03-24T00:22:55.956 INFO:journalctl@ceph.mds.d.smithi118.stdout:Mar 24 00:22:55 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-d[72983]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-24T00:22:55.956 INFO:journalctl@ceph.mds.d.smithi118.stdout:Mar 24 00:22:55 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-d[72983]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f358f58cd20] 2024-03-24T00:22:55.956 INFO:journalctl@ceph.mds.d.smithi118.stdout:Mar 24 00:22:55 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-d[72983]: 2: gsignal() 2024-03-24T00:22:55.956 INFO:journalctl@ceph.mds.d.smithi118.stdout:Mar 24 00:22:55 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-d[72983]: 3: abort() 2024-03-24T00:22:55.956 INFO:journalctl@ceph.mds.d.smithi118.stdout:Mar 24 00:22:55 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-d[72983]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x18f) [0x7f35908054fb] 2024-03-24T00:22:55.956 INFO:journalctl@ceph.mds.d.smithi118.stdout:Mar 24 00:22:55 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-d[72983]: 5: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7f3590805667] 2024-03-24T00:22:55.957 INFO:journalctl@ceph.mds.g.smithi118.stdout:Mar 24 00:22:55 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-g[94891]: 2: gsignal() 2024-03-24T00:22:55.957 INFO:journalctl@ceph.mds.g.smithi118.stdout:Mar 24 00:22:55 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-g[94891]: 3: abort() 2024-03-24T00:22:55.957 INFO:journalctl@ceph.mds.g.smithi118.stdout:Mar 24 00:22:55 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-g[94891]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x18f) [0x7ff4506164fb] 2024-03-24T00:22:55.957 INFO:journalctl@ceph.mds.g.smithi118.stdout:Mar 24 00:22:55 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-g[94891]: 5: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7ff450616667] 2024-03-24T00:22:55.957 INFO:journalctl@ceph.mds.g.smithi118.stdout:Mar 24 00:22:55 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-g[94891]: 6: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x555967881223] 2024-03-24T00:22:55.957 INFO:journalctl@ceph.mds.g.smithi118.stdout:Mar 24 00:22:55 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-g[94891]: 7: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x5559676fb5d6] 2024-03-24T00:22:55.957 INFO:journalctl@ceph.mds.g.smithi118.stdout:Mar 24 00:22:55 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-g[94891]: 8: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x5559676fbda1] 2024-03-24T00:22:55.957 INFO:journalctl@ceph.mds.g.smithi118.stdout:Mar 24 00:22:55 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-g[94891]: 9: (Server::handle_client_rename(boost::intrusive_ptr const&)+0x259d) [0x55596773897d] 2024-03-24T00:22:55.957 INFO:journalctl@ceph.mds.g.smithi118.stdout:Mar 24 00:22:55 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-g[94891]: 10: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x55596774e233] 2024-03-24T00:22:55.957 INFO:journalctl@ceph.mds.g.smithi118.stdout:Mar 24 00:22:55 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-g[94891]: 11: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x555967750197] 2024-03-24T00:22:55.958 INFO:journalctl@ceph.mds.g.smithi118.stdout:Mar 24 00:22:55 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-g[94891]: 12: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x55596764f11b] 2024-03-24T00:22:55.958 INFO:journalctl@ceph.mds.g.smithi118.stdout:Mar 24 00:22:55 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-g[94891]: 13: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x55596764f6bc] 2024-03-24T00:22:55.958 INFO:journalctl@ceph.mds.g.smithi118.stdout:Mar 24 00:22:55 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-g[94891]: 14: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x5559676391ff] 2024-03-24T00:22:55.958 INFO:journalctl@ceph.mds.g.smithi118.stdout:Mar 24 00:22:55 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-g[94891]: 15: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7ff4508a8488] 2024-03-24T00:22:55.958 INFO:journalctl@ceph.mds.g.smithi118.stdout:Mar 24 00:22:55 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-g[94891]: 16: (DispatchQueue::entry()+0x51f) [0x7ff4508a560f] 2024-03-24T00:22:55.958 INFO:journalctl@ceph.mds.g.smithi118.stdout:Mar 24 00:22:55 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-g[94891]: 17: (DispatchQueue::DispatchThread::entry()+0x11) [0x7ff450978e81] 2024-03-24T00:22:55.958 INFO:journalctl@ceph.mds.g.smithi118.stdout:Mar 24 00:22:55 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-g[94891]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7ff44f3931ca] 2024-03-24T00:22:55.958 INFO:journalctl@ceph.mds.g.smithi118.stdout:Mar 24 00:22:55 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-g[94891]: 19: clone() 2024-03-24T00:22:55.958 INFO:journalctl@ceph.mds.g.smithi118.stdout:Mar 24 00:22:55 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-g[94891]: 2024-03-24T00:22:55.789+0000 7ff444e00700 -1 *** Caught signal (Aborted) ** 2024-03-24T00:22:55.958 INFO:journalctl@ceph.mds.g.smithi118.stdout:Mar 24 00:22:55 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-g[94891]: in thread 7ff444e00700 thread_name:ms_dispatch 2024-03-24T00:22:55.958 INFO:journalctl@ceph.mds.g.smithi118.stdout:Mar 24 00:22:55 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-g[94891]: 2024-03-24T00:22:55.958 INFO:journalctl@ceph.mds.g.smithi118.stdout:Mar 24 00:22:55 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-g[94891]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-24T00:22:55.958 INFO:journalctl@ceph.mds.g.smithi118.stdout:Mar 24 00:22:55 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-g[94891]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7ff44f39dd20] 2024-03-24T00:22:55.958 INFO:journalctl@ceph.mds.g.smithi118.stdout:Mar 24 00:22:55 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-g[94891]: 2: gsignal() 2024-03-24T00:22:55.959 INFO:journalctl@ceph.mds.g.smithi118.stdout:Mar 24 00:22:55 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-g[94891]: 3: abort() 2024-03-24T00:22:55.959 INFO:journalctl@ceph.mds.g.smithi118.stdout:Mar 24 00:22:55 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-g[94891]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x18f) [0x7ff4506164fb] 2024-03-24T00:22:55.959 INFO:journalctl@ceph.mds.g.smithi118.stdout:Mar 24 00:22:55 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-g[94891]: 5: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7ff450616667] 2024-03-24T00:22:55.959 INFO:journalctl@ceph.mds.d.smithi118.stdout:Mar 24 00:22:55 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-d[72983]: 6: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x555eae6fa223] 2024-03-24T00:22:55.959 INFO:journalctl@ceph.mds.d.smithi118.stdout:Mar 24 00:22:55 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-d[72983]: 7: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x555eae5745d6] 2024-03-24T00:22:55.959 INFO:journalctl@ceph.mds.d.smithi118.stdout:Mar 24 00:22:55 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-d[72983]: 8: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x555eae574da1] 2024-03-24T00:22:55.959 INFO:journalctl@ceph.mds.d.smithi118.stdout:Mar 24 00:22:55 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-d[72983]: 9: (Server::handle_client_rename(boost::intrusive_ptr const&)+0x259d) [0x555eae5b197d] 2024-03-24T00:22:55.959 INFO:journalctl@ceph.mds.d.smithi118.stdout:Mar 24 00:22:55 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-d[72983]: 10: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x555eae5c7233] 2024-03-24T00:22:55.959 INFO:journalctl@ceph.mds.d.smithi118.stdout:Mar 24 00:22:55 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-d[72983]: 11: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x555eae5c9197] 2024-03-24T00:22:55.960 INFO:journalctl@ceph.mds.d.smithi118.stdout:Mar 24 00:22:55 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-d[72983]: 12: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x555eae4c811b] 2024-03-24T00:22:55.960 INFO:journalctl@ceph.mds.d.smithi118.stdout:Mar 24 00:22:55 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-d[72983]: 13: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x555eae4c86bc] 2024-03-24T00:22:55.960 INFO:journalctl@ceph.mds.d.smithi118.stdout:Mar 24 00:22:55 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-d[72983]: 14: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x555eae4b21ff] 2024-03-24T00:22:55.960 INFO:journalctl@ceph.mds.d.smithi118.stdout:Mar 24 00:22:55 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-d[72983]: 15: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7f3590a97488] 2024-03-24T00:22:55.960 INFO:journalctl@ceph.mds.d.smithi118.stdout:Mar 24 00:22:55 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-d[72983]: 16: (DispatchQueue::entry()+0x51f) [0x7f3590a9460f] 2024-03-24T00:22:55.960 INFO:journalctl@ceph.mds.d.smithi118.stdout:Mar 24 00:22:55 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-d[72983]: 17: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f3590b67e81] 2024-03-24T00:22:55.960 INFO:journalctl@ceph.mds.d.smithi118.stdout:Mar 24 00:22:55 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-d[72983]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7f358f5821ca] 2024-03-24T00:22:55.960 INFO:journalctl@ceph.mds.d.smithi118.stdout:Mar 24 00:22:55 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-d[72983]: 19: clone() 2024-03-24T00:22:55.960 INFO:journalctl@ceph.mds.d.smithi118.stdout:Mar 24 00:22:55 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-d[72983]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-03-24T00:22:55.960 INFO:journalctl@ceph.mds.d.smithi118.stdout:Mar 24 00:22:55 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-d[72983]: 2024-03-24T00:22:55.960 INFO:journalctl@ceph.mds.d.smithi118.stdout:Mar 24 00:22:55 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-d[72983]: -1> 2024-03-24T00:22:55.721+0000 7f3585000700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: In function 'void Locker::set_xlocks_done(MutationImpl*, bool)' thread 7f3585000700 time 2024-03-24T00:22:55.719746+0000 2024-03-24T00:22:55.960 INFO:journalctl@ceph.mds.d.smithi118.stdout:Mar 24 00:22:55 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-d[72983]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: 626: FAILED ceph_assert(obj->is_auth()) 2024-03-24T00:22:55.960 INFO:journalctl@ceph.mds.d.smithi118.stdout:Mar 24 00:22:55 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-d[72983]: 2024-03-24T00:22:55.961 INFO:journalctl@ceph.mds.d.smithi118.stdout:Mar 24 00:22:55 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-d[72983]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-24T00:22:55.961 INFO:journalctl@ceph.mds.d.smithi118.stdout:Mar 24 00:22:55 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-d[72983]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x135) [0x7f35908054a1] 2024-03-24T00:22:55.961 INFO:journalctl@ceph.mds.d.smithi118.stdout:Mar 24 00:22:55 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-d[72983]: 2: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7f3590805667] 2024-03-24T00:22:55.961 INFO:journalctl@ceph.mds.g.smithi118.stdout:Mar 24 00:22:55 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-g[94891]: 6: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x555967881223] 2024-03-24T00:22:55.961 INFO:journalctl@ceph.mds.g.smithi118.stdout:Mar 24 00:22:55 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-g[94891]: 7: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x5559676fb5d6] 2024-03-24T00:22:55.961 INFO:journalctl@ceph.mds.g.smithi118.stdout:Mar 24 00:22:55 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-g[94891]: 8: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x5559676fbda1] 2024-03-24T00:22:55.961 INFO:journalctl@ceph.mds.g.smithi118.stdout:Mar 24 00:22:55 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-g[94891]: 9: (Server::handle_client_rename(boost::intrusive_ptr const&)+0x259d) [0x55596773897d] 2024-03-24T00:22:55.961 INFO:journalctl@ceph.mds.g.smithi118.stdout:Mar 24 00:22:55 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-g[94891]: 10: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x55596774e233] 2024-03-24T00:22:55.961 INFO:journalctl@ceph.mds.g.smithi118.stdout:Mar 24 00:22:55 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-g[94891]: 11: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x555967750197] 2024-03-24T00:22:55.961 INFO:journalctl@ceph.mds.g.smithi118.stdout:Mar 24 00:22:55 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-g[94891]: 12: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x55596764f11b] 2024-03-24T00:22:55.962 INFO:journalctl@ceph.mds.g.smithi118.stdout:Mar 24 00:22:55 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-g[94891]: 13: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x55596764f6bc] 2024-03-24T00:22:55.962 INFO:journalctl@ceph.mds.g.smithi118.stdout:Mar 24 00:22:55 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-g[94891]: 14: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x5559676391ff] 2024-03-24T00:22:55.962 INFO:journalctl@ceph.mds.g.smithi118.stdout:Mar 24 00:22:55 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-g[94891]: 15: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7ff4508a8488] 2024-03-24T00:22:55.962 INFO:journalctl@ceph.mds.g.smithi118.stdout:Mar 24 00:22:55 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-g[94891]: 16: (DispatchQueue::entry()+0x51f) [0x7ff4508a560f] 2024-03-24T00:22:55.962 INFO:journalctl@ceph.mds.g.smithi118.stdout:Mar 24 00:22:55 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-g[94891]: 17: (DispatchQueue::DispatchThread::entry()+0x11) [0x7ff450978e81] 2024-03-24T00:22:55.962 INFO:journalctl@ceph.mds.g.smithi118.stdout:Mar 24 00:22:55 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-g[94891]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7ff44f3931ca] 2024-03-24T00:22:55.962 INFO:journalctl@ceph.mds.g.smithi118.stdout:Mar 24 00:22:55 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-g[94891]: 19: clone() 2024-03-24T00:22:55.962 INFO:journalctl@ceph.mds.g.smithi118.stdout:Mar 24 00:22:55 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-g[94891]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-03-24T00:22:55.962 INFO:journalctl@ceph.mds.g.smithi118.stdout:Mar 24 00:22:55 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-g[94891]: 2024-03-24T00:22:55.962 INFO:journalctl@ceph.mds.g.smithi118.stdout:Mar 24 00:22:55 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-g[94891]: -1> 2024-03-24T00:22:55.786+0000 7ff444e00700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: In function 'void Locker::set_xlocks_done(MutationImpl*, bool)' thread 7ff444e00700 time 2024-03-24T00:22:55.785567+0000 2024-03-24T00:22:55.962 INFO:journalctl@ceph.mds.g.smithi118.stdout:Mar 24 00:22:55 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-g[94891]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: 626: FAILED ceph_assert(obj->is_auth()) 2024-03-24T00:22:55.962 INFO:journalctl@ceph.mds.g.smithi118.stdout:Mar 24 00:22:55 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-g[94891]: 2024-03-24T00:22:55.962 INFO:journalctl@ceph.mds.g.smithi118.stdout:Mar 24 00:22:55 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-g[94891]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-24T00:22:55.963 INFO:journalctl@ceph.mds.g.smithi118.stdout:Mar 24 00:22:55 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-g[94891]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x135) [0x7ff4506164a1] 2024-03-24T00:22:55.963 INFO:journalctl@ceph.mds.g.smithi118.stdout:Mar 24 00:22:55 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-g[94891]: 2: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7ff450616667] 2024-03-24T00:22:55.963 INFO:journalctl@ceph.mds.d.smithi118.stdout:Mar 24 00:22:55 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-d[72983]: 3: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x555eae6fa223] 2024-03-24T00:22:55.963 INFO:journalctl@ceph.mds.d.smithi118.stdout:Mar 24 00:22:55 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-d[72983]: 4: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x555eae5745d6] 2024-03-24T00:22:55.963 INFO:journalctl@ceph.mds.d.smithi118.stdout:Mar 24 00:22:55 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-d[72983]: 5: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x555eae574da1] 2024-03-24T00:22:55.963 INFO:journalctl@ceph.mds.d.smithi118.stdout:Mar 24 00:22:55 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-d[72983]: 6: (Server::handle_client_rename(boost::intrusive_ptr const&)+0x259d) [0x555eae5b197d] 2024-03-24T00:22:55.963 INFO:journalctl@ceph.mds.d.smithi118.stdout:Mar 24 00:22:55 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-d[72983]: 7: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x555eae5c7233] 2024-03-24T00:22:55.963 INFO:journalctl@ceph.mds.d.smithi118.stdout:Mar 24 00:22:55 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-d[72983]: 8: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x555eae5c9197] 2024-03-24T00:22:55.963 INFO:journalctl@ceph.mds.d.smithi118.stdout:Mar 24 00:22:55 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-d[72983]: 9: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x555eae4c811b] 2024-03-24T00:22:55.963 INFO:journalctl@ceph.mds.d.smithi118.stdout:Mar 24 00:22:55 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-d[72983]: 10: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x555eae4c86bc] 2024-03-24T00:22:55.963 INFO:journalctl@ceph.mds.d.smithi118.stdout:Mar 24 00:22:55 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-d[72983]: 11: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x555eae4b21ff] 2024-03-24T00:22:55.964 INFO:journalctl@ceph.mds.d.smithi118.stdout:Mar 24 00:22:55 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-d[72983]: 12: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7f3590a97488] 2024-03-24T00:22:55.964 INFO:journalctl@ceph.mds.d.smithi118.stdout:Mar 24 00:22:55 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-d[72983]: 13: (DispatchQueue::entry()+0x51f) [0x7f3590a9460f] 2024-03-24T00:22:55.964 INFO:journalctl@ceph.mds.d.smithi118.stdout:Mar 24 00:22:55 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-d[72983]: 14: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f3590b67e81] 2024-03-24T00:22:55.964 INFO:journalctl@ceph.mds.d.smithi118.stdout:Mar 24 00:22:55 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-d[72983]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7f358f5821ca] 2024-03-24T00:22:55.964 INFO:journalctl@ceph.mds.d.smithi118.stdout:Mar 24 00:22:55 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-d[72983]: 16: clone() 2024-03-24T00:22:55.964 INFO:journalctl@ceph.mds.d.smithi118.stdout:Mar 24 00:22:55 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-d[72983]: 2024-03-24T00:22:55.964 INFO:journalctl@ceph.mds.d.smithi118.stdout:Mar 24 00:22:55 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-d[72983]: 0> 2024-03-24T00:22:55.724+0000 7f3585000700 -1 *** Caught signal (Aborted) ** 2024-03-24T00:22:55.964 INFO:journalctl@ceph.mds.d.smithi118.stdout:Mar 24 00:22:55 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-d[72983]: in thread 7f3585000700 thread_name:ms_dispatch 2024-03-24T00:22:55.964 INFO:journalctl@ceph.mds.d.smithi118.stdout:Mar 24 00:22:55 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-d[72983]: 2024-03-24T00:22:55.964 INFO:journalctl@ceph.mds.d.smithi118.stdout:Mar 24 00:22:55 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-d[72983]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-24T00:22:55.964 INFO:journalctl@ceph.mds.d.smithi118.stdout:Mar 24 00:22:55 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-d[72983]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f358f58cd20] 2024-03-24T00:22:55.964 INFO:journalctl@ceph.mds.d.smithi118.stdout:Mar 24 00:22:55 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-d[72983]: 2: gsignal() 2024-03-24T00:22:55.964 INFO:journalctl@ceph.mds.d.smithi118.stdout:Mar 24 00:22:55 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-d[72983]: 3: abort() 2024-03-24T00:22:55.964 INFO:journalctl@ceph.mds.d.smithi118.stdout:Mar 24 00:22:55 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-d[72983]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x18f) [0x7f35908054fb] 2024-03-24T00:22:55.964 INFO:journalctl@ceph.mds.d.smithi118.stdout:Mar 24 00:22:55 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-d[72983]: 5: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7f3590805667] 2024-03-24T00:22:55.964 INFO:journalctl@ceph.mds.d.smithi118.stdout:Mar 24 00:22:55 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-d[72983]: 6: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x555eae6fa223] 2024-03-24T00:22:55.965 INFO:journalctl@ceph.mds.d.smithi118.stdout:Mar 24 00:22:55 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-d[72983]: 7: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x555eae5745d6] 2024-03-24T00:22:55.965 INFO:journalctl@ceph.mds.g.smithi118.stdout:Mar 24 00:22:55 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-g[94891]: 3: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x555967881223] 2024-03-24T00:22:55.965 INFO:journalctl@ceph.mds.g.smithi118.stdout:Mar 24 00:22:55 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-g[94891]: 4: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x5559676fb5d6] 2024-03-24T00:22:55.965 INFO:journalctl@ceph.mds.g.smithi118.stdout:Mar 24 00:22:55 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-g[94891]: 5: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x5559676fbda1] 2024-03-24T00:22:55.965 INFO:journalctl@ceph.mds.g.smithi118.stdout:Mar 24 00:22:55 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-g[94891]: 6: (Server::handle_client_rename(boost::intrusive_ptr const&)+0x259d) [0x55596773897d] 2024-03-24T00:22:55.965 INFO:journalctl@ceph.mds.g.smithi118.stdout:Mar 24 00:22:55 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-g[94891]: 7: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x55596774e233] 2024-03-24T00:22:55.965 INFO:journalctl@ceph.mds.g.smithi118.stdout:Mar 24 00:22:55 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-g[94891]: 8: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x555967750197] 2024-03-24T00:22:55.965 INFO:journalctl@ceph.mds.g.smithi118.stdout:Mar 24 00:22:55 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-g[94891]: 9: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x55596764f11b] 2024-03-24T00:22:55.965 INFO:journalctl@ceph.mds.g.smithi118.stdout:Mar 24 00:22:55 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-g[94891]: 10: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x55596764f6bc] 2024-03-24T00:22:55.965 INFO:journalctl@ceph.mds.g.smithi118.stdout:Mar 24 00:22:55 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-g[94891]: 11: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x5559676391ff] 2024-03-24T00:22:55.965 INFO:journalctl@ceph.mds.g.smithi118.stdout:Mar 24 00:22:55 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-g[94891]: 12: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7ff4508a8488] 2024-03-24T00:22:55.965 INFO:journalctl@ceph.mds.g.smithi118.stdout:Mar 24 00:22:55 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-g[94891]: 13: (DispatchQueue::entry()+0x51f) [0x7ff4508a560f] 2024-03-24T00:22:55.966 INFO:journalctl@ceph.mds.g.smithi118.stdout:Mar 24 00:22:55 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-g[94891]: 14: (DispatchQueue::DispatchThread::entry()+0x11) [0x7ff450978e81] 2024-03-24T00:22:55.966 INFO:journalctl@ceph.mds.g.smithi118.stdout:Mar 24 00:22:55 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-g[94891]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7ff44f3931ca] 2024-03-24T00:22:55.966 INFO:journalctl@ceph.mds.g.smithi118.stdout:Mar 24 00:22:55 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-g[94891]: 16: clone() 2024-03-24T00:22:55.966 INFO:journalctl@ceph.mds.g.smithi118.stdout:Mar 24 00:22:55 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-g[94891]: 2024-03-24T00:22:55.966 INFO:journalctl@ceph.mds.g.smithi118.stdout:Mar 24 00:22:55 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-g[94891]: 0> 2024-03-24T00:22:55.789+0000 7ff444e00700 -1 *** Caught signal (Aborted) ** 2024-03-24T00:22:55.966 INFO:journalctl@ceph.mds.g.smithi118.stdout:Mar 24 00:22:55 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-g[94891]: in thread 7ff444e00700 thread_name:ms_dispatch 2024-03-24T00:22:55.966 INFO:journalctl@ceph.mds.g.smithi118.stdout:Mar 24 00:22:55 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-g[94891]: 2024-03-24T00:22:55.966 INFO:journalctl@ceph.mds.g.smithi118.stdout:Mar 24 00:22:55 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-g[94891]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-24T00:22:55.966 INFO:journalctl@ceph.mds.g.smithi118.stdout:Mar 24 00:22:55 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-g[94891]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7ff44f39dd20] 2024-03-24T00:22:55.966 INFO:journalctl@ceph.mds.g.smithi118.stdout:Mar 24 00:22:55 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-g[94891]: 2: gsignal() 2024-03-24T00:22:55.966 INFO:journalctl@ceph.mds.g.smithi118.stdout:Mar 24 00:22:55 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-g[94891]: 3: abort() 2024-03-24T00:22:55.966 INFO:journalctl@ceph.mds.g.smithi118.stdout:Mar 24 00:22:55 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-g[94891]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x18f) [0x7ff4506164fb] 2024-03-24T00:22:55.966 INFO:journalctl@ceph.mds.g.smithi118.stdout:Mar 24 00:22:55 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-g[94891]: 5: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7ff450616667] 2024-03-24T00:22:55.966 INFO:journalctl@ceph.mds.g.smithi118.stdout:Mar 24 00:22:55 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-g[94891]: 6: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x555967881223] 2024-03-24T00:22:55.966 INFO:journalctl@ceph.mds.g.smithi118.stdout:Mar 24 00:22:55 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-g[94891]: 7: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x5559676fb5d6] 2024-03-24T00:22:55.967 INFO:journalctl@ceph.mds.d.smithi118.stdout:Mar 24 00:22:55 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-d[72983]: 8: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x555eae574da1] 2024-03-24T00:22:55.967 INFO:journalctl@ceph.mds.d.smithi118.stdout:Mar 24 00:22:55 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-d[72983]: 9: (Server::handle_client_rename(boost::intrusive_ptr const&)+0x259d) [0x555eae5b197d] 2024-03-24T00:22:55.967 INFO:journalctl@ceph.mds.d.smithi118.stdout:Mar 24 00:22:55 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-d[72983]: 10: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x555eae5c7233] 2024-03-24T00:22:55.967 INFO:journalctl@ceph.mds.d.smithi118.stdout:Mar 24 00:22:55 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-d[72983]: 11: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x555eae5c9197] 2024-03-24T00:22:55.967 INFO:journalctl@ceph.mds.d.smithi118.stdout:Mar 24 00:22:55 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-d[72983]: 12: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x555eae4c811b] 2024-03-24T00:22:55.967 INFO:journalctl@ceph.mds.d.smithi118.stdout:Mar 24 00:22:55 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-d[72983]: 13: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x555eae4c86bc] 2024-03-24T00:22:55.967 INFO:journalctl@ceph.mds.d.smithi118.stdout:Mar 24 00:22:55 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-d[72983]: 14: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x555eae4b21ff] 2024-03-24T00:22:55.967 INFO:journalctl@ceph.mds.d.smithi118.stdout:Mar 24 00:22:55 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-d[72983]: 15: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7f3590a97488] 2024-03-24T00:22:55.967 INFO:journalctl@ceph.mds.d.smithi118.stdout:Mar 24 00:22:55 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-d[72983]: 16: (DispatchQueue::entry()+0x51f) [0x7f3590a9460f] 2024-03-24T00:22:55.967 INFO:journalctl@ceph.mds.d.smithi118.stdout:Mar 24 00:22:55 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-d[72983]: 17: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f3590b67e81] 2024-03-24T00:22:55.967 INFO:journalctl@ceph.mds.d.smithi118.stdout:Mar 24 00:22:55 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-d[72983]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7f358f5821ca] 2024-03-24T00:22:55.968 INFO:journalctl@ceph.mds.d.smithi118.stdout:Mar 24 00:22:55 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-d[72983]: 19: clone() 2024-03-24T00:22:55.968 INFO:journalctl@ceph.mds.d.smithi118.stdout:Mar 24 00:22:55 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-d[72983]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-03-24T00:22:55.968 INFO:journalctl@ceph.mds.d.smithi118.stdout:Mar 24 00:22:55 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-d[72983]: 2024-03-24T00:22:55.968 INFO:journalctl@ceph.mds.d.smithi118.stdout:Mar 24 00:22:55 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-d[72983]: -4> 2024-03-24T00:22:55.721+0000 7f3585000700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: In function 'void Locker::set_xlocks_done(MutationImpl*, bool)' thread 7f3585000700 time 2024-03-24T00:22:55.719746+0000 2024-03-24T00:22:55.968 INFO:journalctl@ceph.mds.d.smithi118.stdout:Mar 24 00:22:55 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-d[72983]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: 626: FAILED ceph_assert(obj->is_auth()) 2024-03-24T00:22:55.968 INFO:journalctl@ceph.mds.d.smithi118.stdout:Mar 24 00:22:55 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-d[72983]: 2024-03-24T00:22:55.968 INFO:journalctl@ceph.mds.d.smithi118.stdout:Mar 24 00:22:55 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-d[72983]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-24T00:22:55.968 INFO:journalctl@ceph.mds.d.smithi118.stdout:Mar 24 00:22:55 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-d[72983]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x135) [0x7f35908054a1] 2024-03-24T00:22:55.968 INFO:journalctl@ceph.mds.d.smithi118.stdout:Mar 24 00:22:55 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-d[72983]: 2: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7f3590805667] 2024-03-24T00:22:55.968 INFO:journalctl@ceph.mds.d.smithi118.stdout:Mar 24 00:22:55 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-d[72983]: 3: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x555eae6fa223] 2024-03-24T00:22:55.968 INFO:journalctl@ceph.mds.d.smithi118.stdout:Mar 24 00:22:55 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-d[72983]: 4: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x555eae5745d6] 2024-03-24T00:22:55.968 INFO:journalctl@ceph.mds.g.smithi118.stdout:Mar 24 00:22:55 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-g[94891]: 8: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x5559676fbda1] 2024-03-24T00:22:55.969 INFO:journalctl@ceph.mds.g.smithi118.stdout:Mar 24 00:22:55 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-g[94891]: 9: (Server::handle_client_rename(boost::intrusive_ptr const&)+0x259d) [0x55596773897d] 2024-03-24T00:22:55.969 INFO:journalctl@ceph.mds.g.smithi118.stdout:Mar 24 00:22:55 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-g[94891]: 10: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x55596774e233] 2024-03-24T00:22:55.969 INFO:journalctl@ceph.mds.g.smithi118.stdout:Mar 24 00:22:55 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-g[94891]: 11: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x555967750197] 2024-03-24T00:22:55.969 INFO:journalctl@ceph.mds.g.smithi118.stdout:Mar 24 00:22:55 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-g[94891]: 12: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x55596764f11b] 2024-03-24T00:22:55.969 INFO:journalctl@ceph.mds.g.smithi118.stdout:Mar 24 00:22:55 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-g[94891]: 13: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x55596764f6bc] 2024-03-24T00:22:55.969 INFO:journalctl@ceph.mds.g.smithi118.stdout:Mar 24 00:22:55 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-g[94891]: 14: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x5559676391ff] 2024-03-24T00:22:55.969 INFO:journalctl@ceph.mds.g.smithi118.stdout:Mar 24 00:22:55 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-g[94891]: 15: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7ff4508a8488] 2024-03-24T00:22:55.969 INFO:journalctl@ceph.mds.g.smithi118.stdout:Mar 24 00:22:55 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-g[94891]: 16: (DispatchQueue::entry()+0x51f) [0x7ff4508a560f] 2024-03-24T00:22:55.969 INFO:journalctl@ceph.mds.g.smithi118.stdout:Mar 24 00:22:55 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-g[94891]: 17: (DispatchQueue::DispatchThread::entry()+0x11) [0x7ff450978e81] 2024-03-24T00:22:55.969 INFO:journalctl@ceph.mds.g.smithi118.stdout:Mar 24 00:22:55 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-g[94891]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7ff44f3931ca] 2024-03-24T00:22:55.969 INFO:journalctl@ceph.mds.g.smithi118.stdout:Mar 24 00:22:55 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-g[94891]: 19: clone() 2024-03-24T00:22:55.969 INFO:journalctl@ceph.mds.g.smithi118.stdout:Mar 24 00:22:55 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-g[94891]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-03-24T00:22:55.969 INFO:journalctl@ceph.mds.g.smithi118.stdout:Mar 24 00:22:55 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-g[94891]: 2024-03-24T00:22:55.969 INFO:journalctl@ceph.mds.g.smithi118.stdout:Mar 24 00:22:55 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-g[94891]: -5> 2024-03-24T00:22:55.786+0000 7ff444e00700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: In function 'void Locker::set_xlocks_done(MutationImpl*, bool)' thread 7ff444e00700 time 2024-03-24T00:22:55.785567+0000 2024-03-24T00:22:55.970 INFO:journalctl@ceph.mds.g.smithi118.stdout:Mar 24 00:22:55 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-g[94891]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: 626: FAILED ceph_assert(obj->is_auth()) 2024-03-24T00:22:55.970 INFO:journalctl@ceph.mds.g.smithi118.stdout:Mar 24 00:22:55 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-g[94891]: 2024-03-24T00:22:55.970 INFO:journalctl@ceph.mds.g.smithi118.stdout:Mar 24 00:22:55 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-g[94891]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-24T00:22:55.970 INFO:journalctl@ceph.mds.g.smithi118.stdout:Mar 24 00:22:55 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-g[94891]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x135) [0x7ff4506164a1] 2024-03-24T00:22:55.970 INFO:journalctl@ceph.mds.g.smithi118.stdout:Mar 24 00:22:55 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-g[94891]: 2: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7ff450616667] 2024-03-24T00:22:55.970 INFO:journalctl@ceph.mds.g.smithi118.stdout:Mar 24 00:22:55 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-g[94891]: 3: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x555967881223] 2024-03-24T00:22:55.970 INFO:journalctl@ceph.mds.g.smithi118.stdout:Mar 24 00:22:55 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-g[94891]: 4: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x5559676fb5d6] 2024-03-24T00:22:55.970 INFO:journalctl@ceph.mds.d.smithi118.stdout:Mar 24 00:22:55 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-d[72983]: 5: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x555eae574da1] 2024-03-24T00:22:55.970 INFO:journalctl@ceph.mds.d.smithi118.stdout:Mar 24 00:22:55 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-d[72983]: 6: (Server::handle_client_rename(boost::intrusive_ptr const&)+0x259d) [0x555eae5b197d] 2024-03-24T00:22:55.970 INFO:journalctl@ceph.mds.d.smithi118.stdout:Mar 24 00:22:55 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-d[72983]: 7: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x555eae5c7233] 2024-03-24T00:22:55.970 INFO:journalctl@ceph.mds.d.smithi118.stdout:Mar 24 00:22:55 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-d[72983]: 8: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x555eae5c9197] 2024-03-24T00:22:55.971 INFO:journalctl@ceph.mds.d.smithi118.stdout:Mar 24 00:22:55 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-d[72983]: 9: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x555eae4c811b] 2024-03-24T00:22:55.971 INFO:journalctl@ceph.mds.d.smithi118.stdout:Mar 24 00:22:55 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-d[72983]: 10: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x555eae4c86bc] 2024-03-24T00:22:55.971 INFO:journalctl@ceph.mds.d.smithi118.stdout:Mar 24 00:22:55 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-d[72983]: 11: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x555eae4b21ff] 2024-03-24T00:22:55.971 INFO:journalctl@ceph.mds.d.smithi118.stdout:Mar 24 00:22:55 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-d[72983]: 12: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7f3590a97488] 2024-03-24T00:22:55.971 INFO:journalctl@ceph.mds.d.smithi118.stdout:Mar 24 00:22:55 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-d[72983]: 13: (DispatchQueue::entry()+0x51f) [0x7f3590a9460f] 2024-03-24T00:22:55.971 INFO:journalctl@ceph.mds.d.smithi118.stdout:Mar 24 00:22:55 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-d[72983]: 14: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f3590b67e81] 2024-03-24T00:22:55.971 INFO:journalctl@ceph.mds.d.smithi118.stdout:Mar 24 00:22:55 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-d[72983]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7f358f5821ca] 2024-03-24T00:22:55.971 INFO:journalctl@ceph.mds.d.smithi118.stdout:Mar 24 00:22:55 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-d[72983]: 16: clone() 2024-03-24T00:22:55.971 INFO:journalctl@ceph.mds.d.smithi118.stdout:Mar 24 00:22:55 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-d[72983]: 2024-03-24T00:22:55.971 INFO:journalctl@ceph.mds.d.smithi118.stdout:Mar 24 00:22:55 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-d[72983]: -3> 2024-03-24T00:22:55.724+0000 7f3585000700 -1 *** Caught signal (Aborted) ** 2024-03-24T00:22:55.971 INFO:journalctl@ceph.mds.d.smithi118.stdout:Mar 24 00:22:55 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-d[72983]: in thread 7f3585000700 thread_name:ms_dispatch 2024-03-24T00:22:55.971 INFO:journalctl@ceph.mds.d.smithi118.stdout:Mar 24 00:22:55 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-d[72983]: 2024-03-24T00:22:55.971 INFO:journalctl@ceph.mds.d.smithi118.stdout:Mar 24 00:22:55 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-d[72983]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-24T00:22:55.971 INFO:journalctl@ceph.mds.d.smithi118.stdout:Mar 24 00:22:55 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-d[72983]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f358f58cd20] 2024-03-24T00:22:55.971 INFO:journalctl@ceph.mds.d.smithi118.stdout:Mar 24 00:22:55 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-d[72983]: 2: gsignal() 2024-03-24T00:22:55.971 INFO:journalctl@ceph.mds.d.smithi118.stdout:Mar 24 00:22:55 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-d[72983]: 3: abort() 2024-03-24T00:22:55.972 INFO:journalctl@ceph.mds.d.smithi118.stdout:Mar 24 00:22:55 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-d[72983]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x18f) [0x7f35908054fb] 2024-03-24T00:22:55.972 INFO:journalctl@ceph.mds.d.smithi118.stdout:Mar 24 00:22:55 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-d[72983]: 5: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7f3590805667] 2024-03-24T00:22:55.972 INFO:journalctl@ceph.mds.d.smithi118.stdout:Mar 24 00:22:55 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-d[72983]: 6: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x555eae6fa223] 2024-03-24T00:22:55.972 INFO:journalctl@ceph.mds.d.smithi118.stdout:Mar 24 00:22:55 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-d[72983]: 7: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x555eae5745d6] 2024-03-24T00:22:55.972 INFO:journalctl@ceph.mds.d.smithi118.stdout:Mar 24 00:22:55 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-d[72983]: 8: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x555eae574da1] 2024-03-24T00:22:55.972 INFO:journalctl@ceph.mds.d.smithi118.stdout:Mar 24 00:22:55 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-d[72983]: 9: (Server::handle_client_rename(boost::intrusive_ptr const&)+0x259d) [0x555eae5b197d] 2024-03-24T00:22:55.972 INFO:journalctl@ceph.mds.d.smithi118.stdout:Mar 24 00:22:55 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-d[72983]: 10: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x555eae5c7233] 2024-03-24T00:22:55.972 INFO:journalctl@ceph.mds.g.smithi118.stdout:Mar 24 00:22:55 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-g[94891]: 5: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x5559676fbda1] 2024-03-24T00:22:55.972 INFO:journalctl@ceph.mds.g.smithi118.stdout:Mar 24 00:22:55 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-g[94891]: 6: (Server::handle_client_rename(boost::intrusive_ptr const&)+0x259d) [0x55596773897d] 2024-03-24T00:22:55.973 INFO:journalctl@ceph.mds.g.smithi118.stdout:Mar 24 00:22:55 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-g[94891]: 7: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x55596774e233] 2024-03-24T00:22:55.973 INFO:journalctl@ceph.mds.g.smithi118.stdout:Mar 24 00:22:55 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-g[94891]: 8: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x555967750197] 2024-03-24T00:22:55.973 INFO:journalctl@ceph.mds.g.smithi118.stdout:Mar 24 00:22:55 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-g[94891]: 9: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x55596764f11b] 2024-03-24T00:22:55.973 INFO:journalctl@ceph.mds.g.smithi118.stdout:Mar 24 00:22:55 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-g[94891]: 10: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x55596764f6bc] 2024-03-24T00:22:55.973 INFO:journalctl@ceph.mds.g.smithi118.stdout:Mar 24 00:22:55 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-g[94891]: 11: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x5559676391ff] 2024-03-24T00:22:55.973 INFO:journalctl@ceph.mds.g.smithi118.stdout:Mar 24 00:22:55 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-g[94891]: 12: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7ff4508a8488] 2024-03-24T00:22:55.973 INFO:journalctl@ceph.mds.g.smithi118.stdout:Mar 24 00:22:55 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-g[94891]: 13: (DispatchQueue::entry()+0x51f) [0x7ff4508a560f] 2024-03-24T00:22:55.973 INFO:journalctl@ceph.mds.g.smithi118.stdout:Mar 24 00:22:55 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-g[94891]: 14: (DispatchQueue::DispatchThread::entry()+0x11) [0x7ff450978e81] 2024-03-24T00:22:55.973 INFO:journalctl@ceph.mds.g.smithi118.stdout:Mar 24 00:22:55 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-g[94891]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7ff44f3931ca] 2024-03-24T00:22:55.973 INFO:journalctl@ceph.mds.g.smithi118.stdout:Mar 24 00:22:55 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-g[94891]: 16: clone() 2024-03-24T00:22:55.973 INFO:journalctl@ceph.mds.g.smithi118.stdout:Mar 24 00:22:55 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-g[94891]: 2024-03-24T00:22:55.973 INFO:journalctl@ceph.mds.g.smithi118.stdout:Mar 24 00:22:55 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-g[94891]: -4> 2024-03-24T00:22:55.789+0000 7ff444e00700 -1 *** Caught signal (Aborted) ** 2024-03-24T00:22:55.973 INFO:journalctl@ceph.mds.g.smithi118.stdout:Mar 24 00:22:55 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-g[94891]: in thread 7ff444e00700 thread_name:ms_dispatch 2024-03-24T00:22:55.973 INFO:journalctl@ceph.mds.g.smithi118.stdout:Mar 24 00:22:55 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-g[94891]: 2024-03-24T00:22:55.974 INFO:journalctl@ceph.mds.g.smithi118.stdout:Mar 24 00:22:55 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-g[94891]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-24T00:22:55.974 INFO:journalctl@ceph.mds.g.smithi118.stdout:Mar 24 00:22:55 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-g[94891]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7ff44f39dd20] 2024-03-24T00:22:55.974 INFO:journalctl@ceph.mds.g.smithi118.stdout:Mar 24 00:22:55 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-g[94891]: 2: gsignal() 2024-03-24T00:22:55.974 INFO:journalctl@ceph.mds.g.smithi118.stdout:Mar 24 00:22:55 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-g[94891]: 3: abort() 2024-03-24T00:22:55.974 INFO:journalctl@ceph.mds.g.smithi118.stdout:Mar 24 00:22:55 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-g[94891]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x18f) [0x7ff4506164fb] 2024-03-24T00:22:55.974 INFO:journalctl@ceph.mds.g.smithi118.stdout:Mar 24 00:22:55 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-g[94891]: 5: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7ff450616667] 2024-03-24T00:22:55.974 INFO:journalctl@ceph.mds.g.smithi118.stdout:Mar 24 00:22:55 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-g[94891]: 6: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x555967881223] 2024-03-24T00:22:55.974 INFO:journalctl@ceph.mds.g.smithi118.stdout:Mar 24 00:22:55 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-g[94891]: 7: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x5559676fb5d6] 2024-03-24T00:22:55.974 INFO:journalctl@ceph.mds.g.smithi118.stdout:Mar 24 00:22:55 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-g[94891]: 8: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x5559676fbda1] 2024-03-24T00:22:55.974 INFO:journalctl@ceph.mds.g.smithi118.stdout:Mar 24 00:22:55 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-g[94891]: 9: (Server::handle_client_rename(boost::intrusive_ptr const&)+0x259d) [0x55596773897d] 2024-03-24T00:22:55.974 INFO:journalctl@ceph.mds.g.smithi118.stdout:Mar 24 00:22:55 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-g[94891]: 10: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x55596774e233] 2024-03-24T00:22:55.975 INFO:journalctl@ceph.mds.d.smithi118.stdout:Mar 24 00:22:55 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-d[72983]: 11: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x555eae5c9197] 2024-03-24T00:22:55.975 INFO:journalctl@ceph.mds.d.smithi118.stdout:Mar 24 00:22:55 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-d[72983]: 12: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x555eae4c811b] 2024-03-24T00:22:55.975 INFO:journalctl@ceph.mds.d.smithi118.stdout:Mar 24 00:22:55 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-d[72983]: 13: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x555eae4c86bc] 2024-03-24T00:22:55.975 INFO:journalctl@ceph.mds.d.smithi118.stdout:Mar 24 00:22:55 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-d[72983]: 14: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x555eae4b21ff] 2024-03-24T00:22:55.975 INFO:journalctl@ceph.mds.d.smithi118.stdout:Mar 24 00:22:55 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-d[72983]: 15: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7f3590a97488] 2024-03-24T00:22:55.975 INFO:journalctl@ceph.mds.d.smithi118.stdout:Mar 24 00:22:55 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-d[72983]: 16: (DispatchQueue::entry()+0x51f) [0x7f3590a9460f] 2024-03-24T00:22:55.975 INFO:journalctl@ceph.mds.d.smithi118.stdout:Mar 24 00:22:55 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-d[72983]: 17: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f3590b67e81] 2024-03-24T00:22:55.976 INFO:journalctl@ceph.mds.d.smithi118.stdout:Mar 24 00:22:55 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-d[72983]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7f358f5821ca] 2024-03-24T00:22:55.976 INFO:journalctl@ceph.mds.d.smithi118.stdout:Mar 24 00:22:55 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-d[72983]: 19: clone() 2024-03-24T00:22:55.976 INFO:journalctl@ceph.mds.d.smithi118.stdout:Mar 24 00:22:55 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-d[72983]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-03-24T00:22:55.976 INFO:journalctl@ceph.mds.d.smithi118.stdout:Mar 24 00:22:55 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-d[72983]: 2024-03-24T00:22:55.976 INFO:journalctl@ceph.mds.d.smithi118.stdout:Mar 24 00:22:55 smithi118 podman[95187]: 2024-03-24 00:22:55.895252372 +0000 UTC m=+0.041587487 container died 92f7fd8f0d7a1188e7c2bdbc2ff52f5e634f544c8fb5568aa09480ae496c40bd (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-d, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, GIT_CLEAN=True, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240319, GIT_BRANCH=HEAD, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux ) 2024-03-24T00:22:55.976 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:22:55 smithi193 ceph-mon[36878]: Adjusting osd_memory_target on smithi193 to 174.1M 2024-03-24T00:22:55.976 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:22:55 smithi193 ceph-mon[36878]: Unable to set osd_memory_target on smithi193 to 182639923: error parsing value: Value '182639923' is below minimum 939524096 2024-03-24T00:22:55.976 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:22:55 smithi193 ceph-mon[36878]: mds.? [v2:172.21.15.118:6838/1710625324,v1:172.21.15.118:6839/1710625324] up:rejoin 2024-03-24T00:22:55.976 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:22:55 smithi193 ceph-mon[36878]: mds.? [v2:172.21.15.118:6836/1374462956,v1:172.21.15.118:6837/1374462956] up:rejoin 2024-03-24T00:22:55.976 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:22:55 smithi193 ceph-mon[36878]: fsmap cephfs:5/5 3 up:standby-replay 2 up:standby 2 up:rejoin 3 up:active 2024-03-24T00:22:55.976 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:22:55 smithi193 ceph-mon[36878]: daemon mds.d is now active in filesystem cephfs as rank 3 2024-03-24T00:22:55.976 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:22:55 smithi193 ceph-mon[36878]: daemon mds.g is now active in filesystem cephfs as rank 1 2024-03-24T00:22:55.976 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:22:55 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T00:22:55.977 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:22:55 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T00:22:55.977 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:22:55 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T00:22:55.977 INFO:journalctl@ceph.mds.b.smithi189.stdout:Mar 24 00:22:55 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-b[66975]: 2024-03-24T00:22:55.736+0000 7f4e07ab9700 -1 mds.pinger is_rank_lagging: rank=1 was never sent ping request. 2024-03-24T00:22:55.977 INFO:journalctl@ceph.mds.b.smithi189.stdout:Mar 24 00:22:55 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-b[66975]: 2024-03-24T00:22:55.736+0000 7f4e07ab9700 -1 mds.pinger is_rank_lagging: rank=3 was never sent ping request. 2024-03-24T00:22:55.977 INFO:journalctl@ceph.mds.g.smithi118.stdout:Mar 24 00:22:55 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-g[94891]: 11: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x555967750197] 2024-03-24T00:22:55.977 INFO:journalctl@ceph.mds.g.smithi118.stdout:Mar 24 00:22:55 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-g[94891]: 12: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x55596764f11b] 2024-03-24T00:22:55.977 INFO:journalctl@ceph.mds.g.smithi118.stdout:Mar 24 00:22:55 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-g[94891]: 13: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x55596764f6bc] 2024-03-24T00:22:55.977 INFO:journalctl@ceph.mds.g.smithi118.stdout:Mar 24 00:22:55 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-g[94891]: 14: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x5559676391ff] 2024-03-24T00:22:55.977 INFO:journalctl@ceph.mds.g.smithi118.stdout:Mar 24 00:22:55 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-g[94891]: 15: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7ff4508a8488] 2024-03-24T00:22:55.977 INFO:journalctl@ceph.mds.g.smithi118.stdout:Mar 24 00:22:55 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-g[94891]: 16: (DispatchQueue::entry()+0x51f) [0x7ff4508a560f] 2024-03-24T00:22:55.978 INFO:journalctl@ceph.mds.g.smithi118.stdout:Mar 24 00:22:55 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-g[94891]: 17: (DispatchQueue::DispatchThread::entry()+0x11) [0x7ff450978e81] 2024-03-24T00:22:55.978 INFO:journalctl@ceph.mds.g.smithi118.stdout:Mar 24 00:22:55 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-g[94891]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7ff44f3931ca] 2024-03-24T00:22:55.978 INFO:journalctl@ceph.mds.g.smithi118.stdout:Mar 24 00:22:55 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-g[94891]: 19: clone() 2024-03-24T00:22:55.978 INFO:journalctl@ceph.mds.g.smithi118.stdout:Mar 24 00:22:55 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-g[94891]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-03-24T00:22:55.978 INFO:journalctl@ceph.mds.g.smithi118.stdout:Mar 24 00:22:55 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-g[94891]: 2024-03-24T00:22:55.978 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:22:55 smithi189 ceph-mon[36187]: Adjusting osd_memory_target on smithi193 to 174.1M 2024-03-24T00:22:55.978 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:22:55 smithi189 ceph-mon[36187]: Unable to set osd_memory_target on smithi193 to 182639923: error parsing value: Value '182639923' is below minimum 939524096 2024-03-24T00:22:55.978 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:22:55 smithi189 ceph-mon[36187]: mds.? [v2:172.21.15.118:6838/1710625324,v1:172.21.15.118:6839/1710625324] up:rejoin 2024-03-24T00:22:55.978 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:22:55 smithi189 ceph-mon[36187]: mds.? [v2:172.21.15.118:6836/1374462956,v1:172.21.15.118:6837/1374462956] up:rejoin 2024-03-24T00:22:55.978 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:22:55 smithi189 ceph-mon[36187]: fsmap cephfs:5/5 3 up:standby-replay 2 up:standby 2 up:rejoin 3 up:active 2024-03-24T00:22:55.978 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:22:55 smithi189 ceph-mon[36187]: daemon mds.d is now active in filesystem cephfs as rank 3 2024-03-24T00:22:55.979 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:22:55 smithi189 ceph-mon[36187]: daemon mds.g is now active in filesystem cephfs as rank 1 2024-03-24T00:22:55.979 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:22:55 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T00:22:55.979 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:22:55 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T00:22:55.979 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:22:55 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T00:22:56.446 INFO:journalctl@ceph.mds.g.smithi118.stdout:Mar 24 00:22:56 smithi118 podman[95193]: 2024-03-24 00:22:56.261002266 +0000 UTC m=+0.383179141 container died fdbf59c1210423ef0f7e07fe152cd76ea99cd862de938f70af5e35b932835521 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-g, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, ceph=True, CEPH_POINT_RELEASE=, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240319, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS) 2024-03-24T00:22:56.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:22:56 smithi118 ceph-mon[31535]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-03-24T00:22:56.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:22:56 smithi118 ceph-mon[31535]: Cluster is now healthy 2024-03-24T00:22:56.947 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:22:56 smithi118 ceph-mon[31535]: mds.? [v2:172.21.15.118:6836/1374462956,v1:172.21.15.118:6837/1374462956] up:active 2024-03-24T00:22:56.947 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:22:56 smithi118 ceph-mon[31535]: mds.? [v2:172.21.15.118:6838/1710625324,v1:172.21.15.118:6839/1710625324] up:active 2024-03-24T00:22:56.947 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:22:56 smithi118 ceph-mon[31535]: fsmap cephfs:5 3 up:standby-replay 2 up:standby 5 up:active 2024-03-24T00:22:56.947 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:22:56 smithi118 ceph-mon[31535]: fsmap cephfs:5 5 up:standby-replay 5 up:active 2024-03-24T00:22:56.947 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:22:56 smithi118 ceph-mon[31535]: pgmap v635: 97 pgs: 97 active+clean; 142 MiB data, 788 MiB used, 1.0 TiB / 1.0 TiB avail; 6.0 KiB/s rd, 19 KiB/s wr, 21 op/s 2024-03-24T00:22:56.953 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:22:56 smithi189 ceph-mon[36187]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-03-24T00:22:56.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:22:56 smithi189 ceph-mon[36187]: Cluster is now healthy 2024-03-24T00:22:56.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:22:56 smithi189 ceph-mon[36187]: mds.? [v2:172.21.15.118:6836/1374462956,v1:172.21.15.118:6837/1374462956] up:active 2024-03-24T00:22:56.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:22:56 smithi189 ceph-mon[36187]: mds.? [v2:172.21.15.118:6838/1710625324,v1:172.21.15.118:6839/1710625324] up:active 2024-03-24T00:22:56.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:22:56 smithi189 ceph-mon[36187]: fsmap cephfs:5 3 up:standby-replay 2 up:standby 5 up:active 2024-03-24T00:22:56.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:22:56 smithi189 ceph-mon[36187]: fsmap cephfs:5 5 up:standby-replay 5 up:active 2024-03-24T00:22:56.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:22:56 smithi189 ceph-mon[36187]: pgmap v635: 97 pgs: 97 active+clean; 142 MiB data, 788 MiB used, 1.0 TiB / 1.0 TiB avail; 6.0 KiB/s rd, 19 KiB/s wr, 21 op/s 2024-03-24T00:22:56.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:22:56 smithi193 ceph-mon[36878]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-03-24T00:22:56.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:22:56 smithi193 ceph-mon[36878]: Cluster is now healthy 2024-03-24T00:22:56.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:22:56 smithi193 ceph-mon[36878]: mds.? [v2:172.21.15.118:6836/1374462956,v1:172.21.15.118:6837/1374462956] up:active 2024-03-24T00:22:56.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:22:56 smithi193 ceph-mon[36878]: mds.? [v2:172.21.15.118:6838/1710625324,v1:172.21.15.118:6839/1710625324] up:active 2024-03-24T00:22:56.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:22:56 smithi193 ceph-mon[36878]: fsmap cephfs:5 3 up:standby-replay 2 up:standby 5 up:active 2024-03-24T00:22:56.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:22:56 smithi193 ceph-mon[36878]: fsmap cephfs:5 5 up:standby-replay 5 up:active 2024-03-24T00:22:56.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:22:56 smithi193 ceph-mon[36878]: pgmap v635: 97 pgs: 97 active+clean; 142 MiB data, 788 MiB used, 1.0 TiB / 1.0 TiB avail; 6.0 KiB/s rd, 19 KiB/s wr, 21 op/s 2024-03-24T00:22:57.287 INFO:journalctl@ceph.mds.d.smithi118.stdout:Mar 24 00:22:56 smithi118 podman[95187]: 2024-03-24 00:22:56.950547291 +0000 UTC m=+1.096882453 container remove 92f7fd8f0d7a1188e7c2bdbc2ff52f5e634f544c8fb5568aa09480ae496c40bd (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-d, maintainer=Guillaume Abrioux , RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, org.label-schema.schema-version=1.0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.build-date=20240319, GIT_CLEAN=True, ceph=True, CEPH_POINT_RELEASE=) 2024-03-24T00:22:57.287 INFO:journalctl@ceph.mds.d.smithi118.stdout:Mar 24 00:22:56 smithi118 systemd[1]: ceph-d79a5420-e971-11ee-95cd-87774f69a715@mds.d.service: Main process exited, code=exited, status=134/n/a 2024-03-24T00:22:57.630 INFO:journalctl@ceph.mds.g.smithi118.stdout:Mar 24 00:22:57 smithi118 podman[95193]: 2024-03-24 00:22:57.286358526 +0000 UTC m=+1.408535427 container remove fdbf59c1210423ef0f7e07fe152cd76ea99cd862de938f70af5e35b932835521 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-g, CEPH_POINT_RELEASE=, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240319, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, GIT_CLEAN=True, ceph=True, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2) 2024-03-24T00:22:57.630 INFO:journalctl@ceph.mds.g.smithi118.stdout:Mar 24 00:22:57 smithi118 systemd[1]: ceph-d79a5420-e971-11ee-95cd-87774f69a715@mds.g.service: Main process exited, code=exited, status=134/n/a 2024-03-24T00:22:57.923 INFO:journalctl@ceph.mds.d.smithi118.stdout:Mar 24 00:22:57 smithi118 systemd[1]: ceph-d79a5420-e971-11ee-95cd-87774f69a715@mds.d.service: Failed with result 'exit-code'. 2024-03-24T00:22:57.923 INFO:journalctl@ceph.mds.d.smithi118.stdout:Mar 24 00:22:57 smithi118 systemd[1]: ceph-d79a5420-e971-11ee-95cd-87774f69a715@mds.d.service: Consumed 1.812s CPU time. 2024-03-24T00:22:57.923 INFO:journalctl@ceph.mds.g.smithi118.stdout:Mar 24 00:22:57 smithi118 systemd[1]: ceph-d79a5420-e971-11ee-95cd-87774f69a715@mds.g.service: Failed with result 'exit-code'. 2024-03-24T00:22:57.923 INFO:journalctl@ceph.mds.g.smithi118.stdout:Mar 24 00:22:57 smithi118 systemd[1]: ceph-d79a5420-e971-11ee-95cd-87774f69a715@mds.g.service: Consumed 1.702s CPU time. 2024-03-24T00:22:58.198 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:22:57 smithi118 ceph-mon[31535]: pgmap v636: 97 pgs: 97 active+clean; 142 MiB data, 789 MiB used, 1.0 TiB / 1.0 TiB avail; 232 KiB/s rd, 92 KiB/s wr, 58 op/s 2024-03-24T00:22:58.203 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:22:57 smithi189 ceph-mon[36187]: pgmap v636: 97 pgs: 97 active+clean; 142 MiB data, 789 MiB used, 1.0 TiB / 1.0 TiB avail; 232 KiB/s rd, 92 KiB/s wr, 58 op/s 2024-03-24T00:22:58.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:22:57 smithi193 ceph-mon[36878]: pgmap v636: 97 pgs: 97 active+clean; 142 MiB data, 789 MiB used, 1.0 TiB / 1.0 TiB avail; 232 KiB/s rd, 92 KiB/s wr, 58 op/s 2024-03-24T00:23:00.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:22:59 smithi118 ceph-mon[31535]: pgmap v637: 97 pgs: 97 active+clean; 142 MiB data, 789 MiB used, 1.0 TiB / 1.0 TiB avail; 317 KiB/s rd, 89 KiB/s wr, 71 op/s 2024-03-24T00:23:00.203 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:22:59 smithi189 ceph-mon[36187]: pgmap v637: 97 pgs: 97 active+clean; 142 MiB data, 789 MiB used, 1.0 TiB / 1.0 TiB avail; 317 KiB/s rd, 89 KiB/s wr, 71 op/s 2024-03-24T00:23:00.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:22:59 smithi193 ceph-mon[36878]: pgmap v637: 97 pgs: 97 active+clean; 142 MiB data, 789 MiB used, 1.0 TiB / 1.0 TiB avail; 317 KiB/s rd, 89 KiB/s wr, 71 op/s 2024-03-24T00:23:02.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:23:01 smithi118 ceph-mon[31535]: pgmap v638: 97 pgs: 97 active+clean; 143 MiB data, 808 MiB used, 1.0 TiB / 1.0 TiB avail; 352 KiB/s rd, 115 KiB/s wr, 77 op/s 2024-03-24T00:23:02.203 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:23:01 smithi189 ceph-mon[36187]: pgmap v638: 97 pgs: 97 active+clean; 143 MiB data, 808 MiB used, 1.0 TiB / 1.0 TiB avail; 352 KiB/s rd, 115 KiB/s wr, 77 op/s 2024-03-24T00:23:02.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:23:01 smithi193 ceph-mon[36878]: pgmap v638: 97 pgs: 97 active+clean; 143 MiB data, 808 MiB used, 1.0 TiB / 1.0 TiB avail; 352 KiB/s rd, 115 KiB/s wr, 77 op/s 2024-03-24T00:23:04.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:23:03 smithi118 ceph-mon[31535]: pgmap v639: 97 pgs: 97 active+clean; 144 MiB data, 809 MiB used, 1.0 TiB / 1.0 TiB avail; 323 KiB/s rd, 245 KiB/s wr, 75 op/s 2024-03-24T00:23:04.203 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:23:03 smithi189 ceph-mon[36187]: pgmap v639: 97 pgs: 97 active+clean; 144 MiB data, 809 MiB used, 1.0 TiB / 1.0 TiB avail; 323 KiB/s rd, 245 KiB/s wr, 75 op/s 2024-03-24T00:23:04.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:23:03 smithi193 ceph-mon[36878]: pgmap v639: 97 pgs: 97 active+clean; 144 MiB data, 809 MiB used, 1.0 TiB / 1.0 TiB avail; 323 KiB/s rd, 245 KiB/s wr, 75 op/s 2024-03-24T00:23:06.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:23:05 smithi118 ceph-mon[31535]: pgmap v640: 97 pgs: 97 active+clean; 145 MiB data, 810 MiB used, 1.0 TiB / 1.0 TiB avail; 323 KiB/s rd, 335 KiB/s wr, 78 op/s 2024-03-24T00:23:06.203 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:23:05 smithi189 ceph-mon[36187]: pgmap v640: 97 pgs: 97 active+clean; 145 MiB data, 810 MiB used, 1.0 TiB / 1.0 TiB avail; 323 KiB/s rd, 335 KiB/s wr, 78 op/s 2024-03-24T00:23:06.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:23:05 smithi193 ceph-mon[36878]: pgmap v640: 97 pgs: 97 active+clean; 145 MiB data, 810 MiB used, 1.0 TiB / 1.0 TiB avail; 323 KiB/s rd, 335 KiB/s wr, 78 op/s 2024-03-24T00:23:07.834 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:23:07.834+0000 7f78be000640 1 -- 172.21.15.118:0/1640781769 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f78b8104ae0 msgr2=0x7f78b8104f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:23:07.834 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:23:07.834+0000 7f78be000640 1 --2- 172.21.15.118:0/1640781769 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f78b8104ae0 0x7f78b8104f60 secure :-1 s=READY pgs=444 cs=0 l=1 rev1=1 crypto rx=0x7f78a4009900 tx=0x7f78a402bc30 comp rx=0 tx=0).stop 2024-03-24T00:23:07.834 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:23:07.834+0000 7f78be000640 1 -- 172.21.15.118:0/1640781769 shutdown_connections 2024-03-24T00:23:07.835 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:23:07.834+0000 7f78be000640 1 --2- 172.21.15.118:0/1640781769 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f78b81054a0 0x7f78b810bd30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:23:07.835 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:23:07.834+0000 7f78be000640 1 --2- 172.21.15.118:0/1640781769 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f78b8104ae0 0x7f78b8104f60 unknown :-1 s=CLOSED pgs=444 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:23:07.835 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:23:07.834+0000 7f78be000640 1 --2- 172.21.15.118:0/1640781769 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f78b81038f0 0x7f78b8103d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:23:07.835 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:23:07.834+0000 7f78be000640 1 -- 172.21.15.118:0/1640781769 >> 172.21.15.118:0/1640781769 conn(0x7f78b80fee60 msgr2=0x7f78b81012a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:23:07.835 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:23:07.834+0000 7f78be000640 1 -- 172.21.15.118:0/1640781769 shutdown_connections 2024-03-24T00:23:07.835 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:23:07.834+0000 7f78be000640 1 -- 172.21.15.118:0/1640781769 wait complete. 2024-03-24T00:23:07.836 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:23:07.835+0000 7f78be000640 1 Processor -- start 2024-03-24T00:23:07.836 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:23:07.836+0000 7f78be000640 1 -- start start 2024-03-24T00:23:07.836 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:23:07.836+0000 7f78be000640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f78b81038f0 0x7f78b8079450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:23:07.836 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:23:07.836+0000 7f78be000640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f78b8104ae0 0x7f78b8079990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:23:07.836 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:23:07.836+0000 7f78be000640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f78b81054a0 0x7f78b8075fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:23:07.836 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:23:07.836+0000 7f78be000640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f78b810f5d0 con 0x7f78b81038f0 2024-03-24T00:23:07.836 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:23:07.836+0000 7f78be000640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f78b810f450 con 0x7f78b8104ae0 2024-03-24T00:23:07.836 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:23:07.836+0000 7f78be000640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f78b810f750 con 0x7f78b81054a0 2024-03-24T00:23:07.837 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:23:07.836+0000 7f78b7400640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f78b81038f0 0x7f78b8079450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:23:07.837 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:23:07.836+0000 7f78b7400640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f78b81038f0 0x7f78b8079450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.118:3300/0 says I am v2:172.21.15.118:40690/0 (socket says 172.21.15.118:40690) 2024-03-24T00:23:07.837 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:23:07.836+0000 7f78b7400640 1 -- 172.21.15.118:0/2878504073 learned_addr learned my addr 172.21.15.118:0/2878504073 (peer_addr_for_me v2:172.21.15.118:0/0) 2024-03-24T00:23:07.838 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:23:07.837+0000 7f78b7400640 1 -- 172.21.15.118:0/2878504073 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f78b81054a0 msgr2=0x7f78b8075fc0 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-03-24T00:23:07.838 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:23:07.837+0000 7f78b7400640 1 --2- 172.21.15.118:0/2878504073 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f78b81054a0 0x7f78b8075fc0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:23:07.838 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:23:07.837+0000 7f78b7400640 1 -- 172.21.15.118:0/2878504073 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f78b8104ae0 msgr2=0x7f78b8079990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:23:07.838 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:23:07.837+0000 7f78b7400640 1 --2- 172.21.15.118:0/2878504073 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f78b8104ae0 0x7f78b8079990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:23:07.838 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:23:07.837+0000 7f78b7400640 1 -- 172.21.15.118:0/2878504073 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f78b8076880 con 0x7f78b81038f0 2024-03-24T00:23:07.838 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:23:07.837+0000 7f78b7400640 1 --2- 172.21.15.118:0/2878504073 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f78b81038f0 0x7f78b8079450 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7f78a800e9c0 tx=0x7f78a800ee80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:23:07.838 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:23:07.838+0000 7f78afe00640 1 -- 172.21.15.118:0/2878504073 <== mon.0 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f78a800cd80 con 0x7f78b81038f0 2024-03-24T00:23:07.838 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:23:07.838+0000 7f78afe00640 1 -- 172.21.15.118:0/2878504073 <== mon.0 v2:172.21.15.118:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f78a80035f0 con 0x7f78b81038f0 2024-03-24T00:23:07.838 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:23:07.838+0000 7f78afe00640 1 -- 172.21.15.118:0/2878504073 <== mon.0 v2:172.21.15.118:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f78a8010690 con 0x7f78b81038f0 2024-03-24T00:23:07.839 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:23:07.838+0000 7f78be000640 1 -- 172.21.15.118:0/2878504073 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f78b81a6060 con 0x7f78b81038f0 2024-03-24T00:23:07.839 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:23:07.838+0000 7f78be000640 1 -- 172.21.15.118:0/2878504073 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f78b81a6470 con 0x7f78b81038f0 2024-03-24T00:23:07.842 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:23:07.843+0000 7f78c0bf2740 1 Processor -- start 2024-03-24T00:23:07.843 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:23:07.843+0000 7f78c0bf2740 1 -- start start 2024-03-24T00:23:07.843 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:23:07.843+0000 7f78afe00640 1 -- 172.21.15.118:0/2878504073 <== mon.0 v2:172.21.15.118:3300/0 4 ==== mgrmap(e 20) v1 ==== 137291+0+0 (secure 0 0 0) 0x7f78a80037a0 con 0x7f78b81038f0 2024-03-24T00:23:07.843 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:23:07.843+0000 7f78c0bf2740 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55b9cb3fc1e0 0x55b9cb3fc600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:23:07.844 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:23:07.843+0000 7f78c0bf2740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55b9cb3fcbd0 0x55b9cb4016d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:23:07.844 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:23:07.843+0000 7f78c0bf2740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55b9cb401d40 0x55b9cb4041d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:23:07.844 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:23:07.843+0000 7f78c0bf2740 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b9cb3bf3b0 con 0x55b9cb3fc1e0 2024-03-24T00:23:07.844 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:23:07.843+0000 7f78c0bf2740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b9cb361830 con 0x55b9cb401d40 2024-03-24T00:23:07.844 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:23:07.843+0000 7f78c0bf2740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b9cb30a5d0 con 0x55b9cb3fcbd0 2024-03-24T00:23:07.844 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:23:07.844+0000 7f78b7e00640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55b9cb3fc1e0 0x55b9cb3fc600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:23:07.844 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:23:07.844+0000 7f78b6a00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55b9cb401d40 0x55b9cb4041d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:23:07.844 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:23:07.844+0000 7f78b6a00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55b9cb401d40 0x55b9cb4041d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.118:50198/0 (socket says 172.21.15.118:50198) 2024-03-24T00:23:07.844 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:23:07.844+0000 7f78b6a00640 1 -- 172.21.15.118:0/1469237441 learned_addr learned my addr 172.21.15.118:0/1469237441 (peer_addr_for_me v2:172.21.15.118:0/0) 2024-03-24T00:23:07.844 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:23:07.844+0000 7f78b7e00640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55b9cb3fc1e0 0x55b9cb3fc600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.118:3300/0 says I am v2:172.21.15.118:40702/0 (socket says 172.21.15.118:40702) 2024-03-24T00:23:07.845 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:23:07.844+0000 7f78b6a00640 1 -- 172.21.15.118:0/1469237441 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55b9cb3fcbd0 msgr2=0x55b9cb4016d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:23:07.846 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:23:07.844+0000 7f78b6a00640 1 --2- 172.21.15.118:0/1469237441 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55b9cb3fcbd0 0x55b9cb4016d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:23:07.846 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:23:07.844+0000 7f78b6a00640 1 -- 172.21.15.118:0/1469237441 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55b9cb3fc1e0 msgr2=0x55b9cb3fc600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:23:07.846 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:23:07.844+0000 7f78b6a00640 1 --2- 172.21.15.118:0/1469237441 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55b9cb3fc1e0 0x55b9cb3fc600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:23:07.846 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:23:07.844+0000 7f78b6a00640 1 -- 172.21.15.118:0/1469237441 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55b9cb3222c0 con 0x55b9cb401d40 2024-03-24T00:23:07.846 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:23:07.844+0000 7f78b7e00640 1 --2- 172.21.15.118:0/1469237441 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55b9cb3fc1e0 0x55b9cb3fc600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T00:23:07.846 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:23:07.845+0000 7f78afe00640 1 --2- 172.21.15.118:0/2878504073 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7f789c0a2920 0x7f789c0a4de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:23:07.846 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:23:07.845+0000 7f78b6a00640 1 --2- 172.21.15.118:0/1469237441 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55b9cb401d40 0x55b9cb4041d0 secure :-1 s=READY pgs=445 cs=0 l=1 rev1=1 crypto rx=0x7f78a402c170 tx=0x7f78a4009560 comp rx=0 tx=0).ready entity=mon.2 client_cookie=3f248232ef2e7fcf server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:23:07.846 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:23:07.845+0000 7f78afe00640 1 -- 172.21.15.118:0/2878504073 <== mon.0 v2:172.21.15.118:3300/0 5 ==== osd_map(110..110 src has 1..110) v4 ==== 9997+0+0 (secure 0 0 0) 0x7f78a80cf5e0 con 0x7f78b81038f0 2024-03-24T00:23:07.846 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:23:07.845+0000 7f78b7e00640 1 --2- 172.21.15.118:0/2878504073 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7f789c0a2920 0x7f789c0a4de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:23:07.846 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:23:07.845+0000 7f78a3e00640 1 -- 172.21.15.118:0/1469237441 <== mon.2 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f78a4041070 con 0x55b9cb401d40 2024-03-24T00:23:07.847 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:23:07.845+0000 7f78a3e00640 1 -- 172.21.15.118:0/1469237441 <== mon.2 v2:172.21.15.189:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f78a402cb80 con 0x55b9cb401d40 2024-03-24T00:23:07.847 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:23:07.845+0000 7f78a3e00640 1 -- 172.21.15.118:0/1469237441 <== mon.2 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f78a40046c0 con 0x55b9cb401d40 2024-03-24T00:23:07.848 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:23:07.846+0000 7f78c0bf2740 1 -- 172.21.15.118:0/1469237441 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55b9cb401d40 msgr2=0x55b9cb4041d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:23:07.848 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:23:07.846+0000 7f78c0bf2740 1 --2- 172.21.15.118:0/1469237441 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55b9cb401d40 0x55b9cb4041d0 secure :-1 s=READY pgs=445 cs=0 l=1 rev1=1 crypto rx=0x7f78a402c170 tx=0x7f78a4009560 comp rx=0 tx=0).stop 2024-03-24T00:23:07.848 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:23:07.846+0000 7f78c0bf2740 1 -- 172.21.15.118:0/1469237441 shutdown_connections 2024-03-24T00:23:07.848 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:23:07.846+0000 7f78c0bf2740 1 --2- 172.21.15.118:0/1469237441 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55b9cb401d40 0x55b9cb4041d0 unknown :-1 s=CLOSED pgs=445 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:23:07.848 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:23:07.846+0000 7f78c0bf2740 1 --2- 172.21.15.118:0/1469237441 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55b9cb3fcbd0 0x55b9cb4016d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:23:07.848 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:23:07.847+0000 7f78c0bf2740 1 --2- 172.21.15.118:0/1469237441 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55b9cb3fc1e0 0x55b9cb3fc600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:23:07.848 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:23:07.847+0000 7f78c0bf2740 1 -- 172.21.15.118:0/1469237441 >> 172.21.15.118:0/1469237441 conn(0x55b9cb3f7750 msgr2=0x55b9cb3f7b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:23:07.848 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:23:07.847+0000 7f78b7e00640 1 --2- 172.21.15.118:0/2878504073 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7f789c0a2920 0x7f789c0a4de0 secure :-1 s=READY pgs=382 cs=0 l=1 rev1=1 crypto rx=0x55b9cb3fd430 tx=0x7f78b000d040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:23:07.848 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:23:07.847+0000 7f78c0bf2740 1 -- 172.21.15.118:0/1469237441 shutdown_connections 2024-03-24T00:23:07.848 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:23:07.847+0000 7f78c0bf2740 1 -- 172.21.15.118:0/1469237441 wait complete. 2024-03-24T00:23:07.848 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:23:07.847+0000 7f78c0bf2740 1 -- start start 2024-03-24T00:23:07.848 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:23:07.848+0000 7f78c0bf2740 1 Processor -- start 2024-03-24T00:23:07.850 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:23:07.848+0000 7f78c0bf2740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55b9cb3fc1e0 0x55b9cb406390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:23:07.850 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:23:07.848+0000 7f78c0bf2740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55b9cb3fcbd0 0x55b9cb4068d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:23:07.850 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:23:07.849+0000 7f78c0bf2740 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55b9cb401d40 0x55b9cb40d910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:23:07.850 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:23:07.849+0000 7f78c0bf2740 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b9cb31da20 con 0x55b9cb401d40 2024-03-24T00:23:07.850 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:23:07.849+0000 7f78c0bf2740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b9cb35d8d0 con 0x55b9cb3fc1e0 2024-03-24T00:23:07.850 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:23:07.849+0000 7f78c0bf2740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x55b9cb404c90 con 0x55b9cb3fcbd0 2024-03-24T00:23:07.850 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:23:07.849+0000 7f78c0bf2740 10 client.0 _ll_register_callbacks cb 0x55b9cb35a1c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T00:23:07.850 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:23:07.849+0000 7f78b6a00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55b9cb3fcbd0 0x55b9cb4068d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:23:07.850 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:23:07.849+0000 7f78b6a00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55b9cb3fcbd0 0x55b9cb4068d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.118:46806/0 (socket says 172.21.15.118:46806) 2024-03-24T00:23:07.850 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:23:07.849+0000 7f78b6a00640 1 -- 172.21.15.118:0/927193030 learned_addr learned my addr 172.21.15.118:0/927193030 (peer_addr_for_me v2:172.21.15.118:0/0) 2024-03-24T00:23:07.851 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:23:07.849+0000 7f78b7e00640 1 --2- 172.21.15.118:0/927193030 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55b9cb401d40 0x55b9cb40d910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:23:07.851 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:23:07.849+0000 7f78b6a00640 1 -- 172.21.15.118:0/927193030 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55b9cb3fc1e0 msgr2=0x55b9cb406390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:23:07.851 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:23:07.849+0000 7f78b6a00640 1 --2- 172.21.15.118:0/927193030 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55b9cb3fc1e0 0x55b9cb406390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:23:07.851 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:23:07.849+0000 7f78b6a00640 1 -- 172.21.15.118:0/927193030 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55b9cb401d40 msgr2=0x55b9cb40d910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:23:07.851 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:23:07.849+0000 7f78b6a00640 1 --2- 172.21.15.118:0/927193030 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55b9cb401d40 0x55b9cb40d910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:23:07.851 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:23:07.849+0000 7f78b6a00640 1 -- 172.21.15.118:0/927193030 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55b9cb076210 con 0x55b9cb3fcbd0 2024-03-24T00:23:07.851 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:23:07.850+0000 7f78b6a00640 1 --2- 172.21.15.118:0/927193030 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55b9cb3fcbd0 0x55b9cb4068d0 secure :-1 s=READY pgs=446 cs=0 l=1 rev1=1 crypto rx=0x7f78a4009400 tx=0x7f78a402cda0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=36715ebcc040d4c7 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:23:07.851 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:23:07.850+0000 7f78a0c00640 10 client.0 ms_handle_connect on v2:172.21.15.193:3300/0 2024-03-24T00:23:07.851 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:23:07.850+0000 7f78a0c00640 1 -- 172.21.15.118:0/927193030 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f78a4041040 con 0x55b9cb3fcbd0 2024-03-24T00:23:07.851 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:23:07.850+0000 7f78a0c00640 1 -- 172.21.15.118:0/927193030 <== mon.1 v2:172.21.15.193:3300/0 2 ==== osd_map(110..110 src has 1..110) v4 ==== 9997+0+0 (secure 0 0 0) 0x7f78a4004ad0 con 0x55b9cb3fcbd0 2024-03-24T00:23:07.851 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:23:07.850+0000 7f78c0bf2740 1 -- 172.21.15.118:0/927193030 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f78a402cb80 con 0x55b9cb3fcbd0 2024-03-24T00:23:07.852 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:23:07.851+0000 7f78a0c00640 1 -- 172.21.15.118:0/927193030 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_get_version_reply(handle=1 version=127) v2 ==== 24+0+0 (secure 0 0 0) 0x7f78a4057050 con 0x55b9cb3fcbd0 2024-03-24T00:23:07.853 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:23:07.851+0000 7f78c0bf2740 10 client.26908 fetch_fsmap learned FSMap version 127 2024-03-24T00:23:07.853 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:23:07.851+0000 7f78c0bf2740 1 -- 172.21.15.118:0/927193030 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,fsmap=127,monmap=4+}) v3 -- 0x55b9cb142620 con 0x55b9cb3fcbd0 2024-03-24T00:23:07.853 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:23:07.851+0000 7f78a0c00640 1 -- 172.21.15.118:0/927193030 <== mon.1 v2:172.21.15.193:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f78a402cb80 con 0x55b9cb3fcbd0 2024-03-24T00:23:07.853 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:23:07.851+0000 7f78a0c00640 1 -- 172.21.15.118:0/927193030 <== mon.1 v2:172.21.15.193:3300/0 5 ==== fsmap(e 127) v1 ==== 6784+0+0 (secure 0 0 0) 0x7f78a4034440 con 0x55b9cb3fcbd0 2024-03-24T00:23:07.853 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:23:07.852+0000 7f78c0bf2740 10 client.26908 fetch_fsmap finished waiting for FSMap version 127 2024-03-24T00:23:07.853 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:23:07.852+0000 7f78c0bf2740 10 client.26908 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-24T00:23:07.853 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:23:07.852+0000 7f78c0bf2740 20 client.26908 populate_metadata read hostname 'smithi118' 2024-03-24T00:23:07.853 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:23:07.852+0000 7f78c0bf2740 1 --2- 172.21.15.118:0/927193030 >> [v2:172.21.15.189:6838/4074484168,v1:172.21.15.189:6839/4074484168] conn(0x55b9cb411ef0 0x55b9cb414300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:23:07.853 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:23:07.852+0000 7f78c0bf2740 4 client.26908 mds_command: new command op to 24569 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-24T00:23:07.853 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:23:07.852+0000 7f78c0bf2740 1 -- 172.21.15.118:0/927193030 --> [v2:172.21.15.189:6838/4074484168,v1:172.21.15.189:6839/4074484168] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55b9cb36ed70 con 0x55b9cb411ef0 2024-03-24T00:23:07.853 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:23:07.852+0000 7f78b7400640 1 --2- 172.21.15.118:0/927193030 >> [v2:172.21.15.189:6838/4074484168,v1:172.21.15.189:6839/4074484168] conn(0x55b9cb411ef0 0x55b9cb414300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:23:07.854 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:23:07.853+0000 7f78b7400640 1 --2- 172.21.15.118:0/927193030 >> [v2:172.21.15.189:6838/4074484168,v1:172.21.15.189:6839/4074484168] conn(0x55b9cb411ef0 0x55b9cb414300 crc :-1 s=READY pgs=297 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=45b4e159b59b3e78 server_cookie=23687255aeb374d5 in_seq=0 out_seq=0 2024-03-24T00:23:07.854 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:23:07.853+0000 7f78a0c00640 10 client.26908 ms_handle_connect on v2:172.21.15.189:6838/4074484168 2024-03-24T00:23:07.854 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:23:07.854+0000 7f78a0c00640 1 -- 172.21.15.118:0/927193030 <== mds.0 v2:172.21.15.189:6838/4074484168 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x55b9cb36ed70 con 0x55b9cb411ef0 2024-03-24T00:23:07.854 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:23:07.854+0000 7f78a0c00640 10 client.26908 handle_command_reply: tid=0 2024-03-24T00:23:07.854 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:23:07.854+0000 7f78c0bf2740 1 client.26908 shutdown 2024-03-24T00:23:07.855 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:23:07.855+0000 7f78c0bf2740 1 -- 172.21.15.118:0/927193030 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55b9cb3fcbd0 msgr2=0x55b9cb4068d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:23:07.855 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:23:07.855+0000 7f78c0bf2740 1 --2- 172.21.15.118:0/927193030 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55b9cb3fcbd0 0x55b9cb4068d0 secure :-1 s=READY pgs=446 cs=0 l=1 rev1=1 crypto rx=0x7f78a4009400 tx=0x7f78a402cda0 comp rx=0 tx=0).stop 2024-03-24T00:23:07.855 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:23:07.855+0000 7f78c0bf2740 1 -- 172.21.15.118:0/927193030 shutdown_connections 2024-03-24T00:23:07.855 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:23:07.855+0000 7f78c0bf2740 1 --2- 172.21.15.118:0/927193030 >> [v2:172.21.15.189:6838/4074484168,v1:172.21.15.189:6839/4074484168] conn(0x55b9cb411ef0 0x55b9cb414300 crc :-1 s=READY pgs=297 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:23:07.855 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:23:07.855+0000 7f78c0bf2740 1 --2- 172.21.15.118:0/927193030 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55b9cb401d40 0x55b9cb40d910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:23:07.855 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:23:07.855+0000 7f78c0bf2740 1 --2- 172.21.15.118:0/927193030 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55b9cb3fcbd0 0x55b9cb4068d0 unknown :-1 s=CLOSED pgs=446 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:23:07.855 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:23:07.855+0000 7f78c0bf2740 1 --2- 172.21.15.118:0/927193030 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55b9cb3fc1e0 0x55b9cb406390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:23:07.855 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:23:07.855+0000 7f78a0c00640 0 client.26908 ms_handle_reset on v2:172.21.15.189:6838/4074484168 2024-03-24T00:23:07.856 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:23:07.855+0000 7f78c0bf2740 1 -- 172.21.15.118:0/927193030 >> 172.21.15.118:0/927193030 conn(0x55b9cb3f7750 msgr2=0x55b9cb3f6950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:23:07.856 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:23:07.855+0000 7f78c0bf2740 1 -- 172.21.15.118:0/927193030 shutdown_connections 2024-03-24T00:23:07.856 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:23:07.855+0000 7f78c0bf2740 1 -- 172.21.15.118:0/927193030 wait complete. 2024-03-24T00:23:07.856 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:23:07.855+0000 7f78c0bf2740 20 client.26908 trim_cache size 0 max 16384 2024-03-24T00:23:07.875 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:23:07.875+0000 7f78c0bf2740 1 Processor -- start 2024-03-24T00:23:07.875 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:23:07.875+0000 7f78c0bf2740 1 -- start start 2024-03-24T00:23:07.875 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:23:07.875+0000 7f78c0bf2740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55b9cb3fc1e0 0x55b9cb3f25d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:23:07.875 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:23:07.875+0000 7f78c0bf2740 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55b9cb3fcbd0 0x55b9cb40d8d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:23:07.875 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:23:07.875+0000 7f78c0bf2740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55b9cb401d40 0x55b9cb3fa9b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:23:07.875 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:23:07.875+0000 7f78c0bf2740 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f78b81aeb20 con 0x55b9cb3fcbd0 2024-03-24T00:23:07.875 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:23:07.875+0000 7f78c0bf2740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f78b81b1160 con 0x55b9cb3fc1e0 2024-03-24T00:23:07.875 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:23:07.875+0000 7f78c0bf2740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f78b81af180 con 0x55b9cb401d40 2024-03-24T00:23:07.876 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:23:07.876+0000 7f78b7400640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55b9cb3fc1e0 0x55b9cb3f25d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:23:07.877 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:23:07.876+0000 7f78b7e00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55b9cb401d40 0x55b9cb3fa9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:23:07.877 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:23:07.876+0000 7f78b7400640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55b9cb3fc1e0 0x55b9cb3f25d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.118:50202/0 (socket says 172.21.15.118:50202) 2024-03-24T00:23:07.877 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:23:07.876+0000 7f78b7400640 1 -- 172.21.15.118:0/3037688182 learned_addr learned my addr 172.21.15.118:0/3037688182 (peer_addr_for_me v2:172.21.15.118:0/0) 2024-03-24T00:23:07.877 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:23:07.876+0000 7f78b6a00640 1 --2- 172.21.15.118:0/3037688182 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55b9cb3fcbd0 0x55b9cb40d8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:23:07.877 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:23:07.876+0000 7f78b7400640 1 -- 172.21.15.118:0/3037688182 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55b9cb401d40 msgr2=0x55b9cb3fa9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:23:07.877 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:23:07.876+0000 7f78b7400640 1 --2- 172.21.15.118:0/3037688182 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55b9cb401d40 0x55b9cb3fa9b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:23:07.877 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:23:07.876+0000 7f78b7400640 1 -- 172.21.15.118:0/3037688182 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55b9cb3fcbd0 msgr2=0x55b9cb40d8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:23:07.877 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:23:07.876+0000 7f78b7400640 1 --2- 172.21.15.118:0/3037688182 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55b9cb3fcbd0 0x55b9cb40d8d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:23:07.877 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:23:07.876+0000 7f78b7400640 1 -- 172.21.15.118:0/3037688182 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55b9cb3f81c0 con 0x55b9cb3fc1e0 2024-03-24T00:23:07.877 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:23:07.876+0000 7f78b6a00640 1 --2- 172.21.15.118:0/3037688182 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55b9cb3fcbd0 0x55b9cb40d8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T00:23:07.877 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:23:07.876+0000 7f78b7400640 1 --2- 172.21.15.118:0/3037688182 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55b9cb3fc1e0 0x55b9cb3f25d0 secure :-1 s=READY pgs=446 cs=0 l=1 rev1=1 crypto rx=0x7f78a809b5a0 tx=0x7f78a80cf8e0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=3332b21fbdce9511 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:23:07.877 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:23:07.876+0000 7f78a3400640 1 -- 172.21.15.118:0/3037688182 <== mon.2 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f78a809b8a0 con 0x55b9cb3fc1e0 2024-03-24T00:23:07.877 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:23:07.877+0000 7f78a3400640 1 -- 172.21.15.118:0/3037688182 <== mon.2 v2:172.21.15.189:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f78a809f9f0 con 0x55b9cb3fc1e0 2024-03-24T00:23:07.878 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:23:07.877+0000 7f78a3400640 1 -- 172.21.15.118:0/3037688182 <== mon.2 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f78a8097f10 con 0x55b9cb3fc1e0 2024-03-24T00:23:07.880 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:23:07.877+0000 7f78c0bf2740 1 -- 172.21.15.118:0/3037688182 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55b9cb3fc1e0 msgr2=0x55b9cb3f25d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:23:07.880 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:23:07.877+0000 7f78c0bf2740 1 --2- 172.21.15.118:0/3037688182 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55b9cb3fc1e0 0x55b9cb3f25d0 secure :-1 s=READY pgs=446 cs=0 l=1 rev1=1 crypto rx=0x7f78a809b5a0 tx=0x7f78a80cf8e0 comp rx=0 tx=0).stop 2024-03-24T00:23:07.881 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:23:07.877+0000 7f78c0bf2740 1 -- 172.21.15.118:0/3037688182 shutdown_connections 2024-03-24T00:23:07.881 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:23:07.877+0000 7f78c0bf2740 1 --2- 172.21.15.118:0/3037688182 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55b9cb401d40 0x55b9cb3fa9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:23:07.881 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:23:07.877+0000 7f78c0bf2740 1 --2- 172.21.15.118:0/3037688182 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55b9cb3fcbd0 0x55b9cb40d8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:23:07.881 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:23:07.877+0000 7f78c0bf2740 1 --2- 172.21.15.118:0/3037688182 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55b9cb3fc1e0 0x55b9cb3f25d0 unknown :-1 s=CLOSED pgs=446 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:23:07.881 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:23:07.877+0000 7f78c0bf2740 1 -- 172.21.15.118:0/3037688182 >> 172.21.15.118:0/3037688182 conn(0x55b9cb3f7750 msgr2=0x55b9cb3fde90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:23:07.881 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:23:07.878+0000 7f78c0bf2740 1 -- 172.21.15.118:0/3037688182 shutdown_connections 2024-03-24T00:23:07.881 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:23:07.878+0000 7f78c0bf2740 1 -- 172.21.15.118:0/3037688182 wait complete. 2024-03-24T00:23:07.881 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:23:07.878+0000 7f78c0bf2740 1 -- start start 2024-03-24T00:23:07.881 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:23:07.878+0000 7f78c0bf2740 1 Processor -- start 2024-03-24T00:23:07.881 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:23:07.879+0000 7f78c0bf2740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55b9cb3fc1e0 0x55b9cb40b170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:23:07.881 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:23:07.879+0000 7f78c0bf2740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55b9cb3fcbd0 0x55b9cb40b6b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:23:07.881 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:23:07.879+0000 7f78c0bf2740 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55b9cb401d40 0x55b9cb40c6c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:23:07.881 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:23:07.879+0000 7f78c0bf2740 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f78b81af7e0 con 0x55b9cb401d40 2024-03-24T00:23:07.882 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:23:07.879+0000 7f78c0bf2740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f78b81b0b00 con 0x55b9cb3fc1e0 2024-03-24T00:23:07.882 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:23:07.879+0000 7f78c0bf2740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f78b81afe40 con 0x55b9cb3fcbd0 2024-03-24T00:23:07.882 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:23:07.879+0000 7f78c0bf2740 10 client.0 _ll_register_callbacks cb 0x7f78b0009470 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T00:23:07.882 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:23:07.879+0000 7f78b7400640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55b9cb3fc1e0 0x55b9cb40b170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:23:07.882 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:23:07.879+0000 7f78b6a00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55b9cb3fcbd0 0x55b9cb40b6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:23:07.882 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:23:07.879+0000 7f78b7400640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55b9cb3fc1e0 0x55b9cb40b170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.118:50204/0 (socket says 172.21.15.118:50204) 2024-03-24T00:23:07.882 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:23:07.879+0000 7f78b7400640 1 -- 172.21.15.118:0/427172033 learned_addr learned my addr 172.21.15.118:0/427172033 (peer_addr_for_me v2:172.21.15.118:0/0) 2024-03-24T00:23:07.882 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:23:07.879+0000 7f78b7e00640 1 --2- 172.21.15.118:0/427172033 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55b9cb401d40 0x55b9cb40c6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:23:07.882 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:23:07.879+0000 7f78b7400640 1 -- 172.21.15.118:0/427172033 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55b9cb3fcbd0 msgr2=0x55b9cb40b6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:23:07.882 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:23:07.879+0000 7f78b7400640 1 --2- 172.21.15.118:0/427172033 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55b9cb3fcbd0 0x55b9cb40b6b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:23:07.882 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:23:07.879+0000 7f78b7400640 1 -- 172.21.15.118:0/427172033 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55b9cb401d40 msgr2=0x55b9cb40c6c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:23:07.882 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:23:07.879+0000 7f78b7400640 1 --2- 172.21.15.118:0/427172033 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55b9cb401d40 0x55b9cb40c6c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:23:07.883 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:23:07.879+0000 7f78b7400640 1 -- 172.21.15.118:0/427172033 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55b9cb40ccb0 con 0x55b9cb3fc1e0 2024-03-24T00:23:07.883 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:23:07.879+0000 7f78b6a00640 1 --2- 172.21.15.118:0/427172033 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55b9cb3fcbd0 0x55b9cb40b6b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T00:23:07.883 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:23:07.880+0000 7f78b7400640 1 --2- 172.21.15.118:0/427172033 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55b9cb3fc1e0 0x55b9cb40b170 secure :-1 s=READY pgs=447 cs=0 l=1 rev1=1 crypto rx=0x7f78a8097d00 tx=0x7f78a8099da0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=64d6921af2cbbc7c server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:23:07.883 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:23:07.880+0000 7f78a0c00640 10 client.0 ms_handle_connect on v2:172.21.15.189:3300/0 2024-03-24T00:23:07.883 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:23:07.880+0000 7f78a0c00640 1 -- 172.21.15.118:0/427172033 <== mon.2 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f78a807f060 con 0x55b9cb3fc1e0 2024-03-24T00:23:07.883 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:23:07.880+0000 7f78a0c00640 1 -- 172.21.15.118:0/427172033 <== mon.2 v2:172.21.15.189:3300/0 2 ==== osd_map(110..110 src has 1..110) v4 ==== 9997+0+0 (secure 0 0 0) 0x7f78a8097f10 con 0x55b9cb3fc1e0 2024-03-24T00:23:07.883 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:23:07.880+0000 7f78c0bf2740 1 -- 172.21.15.118:0/427172033 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f78a809f9f0 con 0x55b9cb3fc1e0 2024-03-24T00:23:07.883 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:23:07.880+0000 7f78a0c00640 1 -- 172.21.15.118:0/427172033 <== mon.2 v2:172.21.15.189:3300/0 3 ==== mon_get_version_reply(handle=1 version=127) v2 ==== 24+0+0 (secure 0 0 0) 0x7f78a80ba070 con 0x55b9cb3fc1e0 2024-03-24T00:23:07.883 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:23:07.880+0000 7f78c0bf2740 10 client.26903 fetch_fsmap learned FSMap version 127 2024-03-24T00:23:07.883 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:23:07.880+0000 7f78c0bf2740 1 -- 172.21.15.118:0/427172033 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,fsmap=127,monmap=4+}) v3 -- 0x55b9cb46d890 con 0x55b9cb3fc1e0 2024-03-24T00:23:07.883 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:23:07.880+0000 7f78a0c00640 1 -- 172.21.15.118:0/427172033 <== mon.2 v2:172.21.15.189:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f78a809f9f0 con 0x55b9cb3fc1e0 2024-03-24T00:23:07.883 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:23:07.880+0000 7f78a0c00640 1 -- 172.21.15.118:0/427172033 <== mon.2 v2:172.21.15.189:3300/0 5 ==== fsmap(e 127) v1 ==== 6784+0+0 (secure 0 0 0) 0x7f78a80a3bb0 con 0x55b9cb3fc1e0 2024-03-24T00:23:07.883 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:23:07.880+0000 7f78c0bf2740 10 client.26903 fetch_fsmap finished waiting for FSMap version 127 2024-03-24T00:23:07.884 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:23:07.881+0000 7f78c0bf2740 10 client.26903 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-24T00:23:07.884 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:23:07.881+0000 7f78c0bf2740 20 client.26903 populate_metadata read hostname 'smithi118' 2024-03-24T00:23:07.884 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:23:07.881+0000 7f78c0bf2740 1 --2- 172.21.15.118:0/427172033 >> [v2:172.21.15.189:6838/4074484168,v1:172.21.15.189:6839/4074484168] conn(0x55b9cb46e0f0 0x55b9cb4704e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:23:07.884 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:23:07.881+0000 7f78c0bf2740 4 client.26903 mds_command: new command op to 24569 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-03-24T00:23:07.884 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:23:07.881+0000 7f78c0bf2740 1 -- 172.21.15.118:0/427172033 --> [v2:172.21.15.189:6838/4074484168,v1:172.21.15.189:6839/4074484168] -- command(tid 0: {"prefix": "flush journal"}) v1 -- 0x55b9cb3690c0 con 0x55b9cb46e0f0 2024-03-24T00:23:07.884 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:23:07.882+0000 7f78b6a00640 1 --2- 172.21.15.118:0/427172033 >> [v2:172.21.15.189:6838/4074484168,v1:172.21.15.189:6839/4074484168] conn(0x55b9cb46e0f0 0x55b9cb4704e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:23:07.885 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:23:07.884+0000 7f78b6a00640 1 --2- 172.21.15.118:0/427172033 >> [v2:172.21.15.189:6838/4074484168,v1:172.21.15.189:6839/4074484168] conn(0x55b9cb46e0f0 0x55b9cb4704e0 crc :-1 s=READY pgs=298 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=df9f87814677e5ce server_cookie=36f1ae360e02b944 in_seq=0 out_seq=0 2024-03-24T00:23:07.885 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:23:07.884+0000 7f78a0c00640 10 client.26903 ms_handle_connect on v2:172.21.15.189:6838/4074484168 2024-03-24T00:23:08.126 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:23:07 smithi118 ceph-mon[31535]: pgmap v641: 97 pgs: 97 active+clean; 146 MiB data, 810 MiB used, 1.0 TiB / 1.0 TiB avail; 342 KiB/s rd, 354 KiB/s wr, 79 op/s 2024-03-24T00:23:08.127 INFO:journalctl@ceph.mds.d.smithi118.stdout:Mar 24 00:23:07 smithi118 systemd[1]: ceph-d79a5420-e971-11ee-95cd-87774f69a715@mds.d.service: Scheduled restart job, restart counter is at 1. 2024-03-24T00:23:08.127 INFO:journalctl@ceph.mds.d.smithi118.stdout:Mar 24 00:23:07 smithi118 systemd[1]: Stopped Ceph mds.d for d79a5420-e971-11ee-95cd-87774f69a715. 2024-03-24T00:23:08.127 INFO:journalctl@ceph.mds.d.smithi118.stdout:Mar 24 00:23:07 smithi118 systemd[1]: ceph-d79a5420-e971-11ee-95cd-87774f69a715@mds.d.service: Consumed 1.812s CPU time. 2024-03-24T00:23:08.127 INFO:journalctl@ceph.mds.d.smithi118.stdout:Mar 24 00:23:07 smithi118 systemd[1]: Starting Ceph mds.d for d79a5420-e971-11ee-95cd-87774f69a715... 2024-03-24T00:23:08.127 INFO:journalctl@ceph.mds.g.smithi118.stdout:Mar 24 00:23:07 smithi118 systemd[1]: ceph-d79a5420-e971-11ee-95cd-87774f69a715@mds.g.service: Scheduled restart job, restart counter is at 5. 2024-03-24T00:23:08.127 INFO:journalctl@ceph.mds.g.smithi118.stdout:Mar 24 00:23:07 smithi118 systemd[1]: Stopped Ceph mds.g for d79a5420-e971-11ee-95cd-87774f69a715. 2024-03-24T00:23:08.127 INFO:journalctl@ceph.mds.g.smithi118.stdout:Mar 24 00:23:07 smithi118 systemd[1]: ceph-d79a5420-e971-11ee-95cd-87774f69a715@mds.g.service: Consumed 1.702s CPU time. 2024-03-24T00:23:08.127 INFO:journalctl@ceph.mds.g.smithi118.stdout:Mar 24 00:23:07 smithi118 systemd[1]: Starting Ceph mds.g for d79a5420-e971-11ee-95cd-87774f69a715... 2024-03-24T00:23:08.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:23:07 smithi189 ceph-mon[36187]: pgmap v641: 97 pgs: 97 active+clean; 146 MiB data, 810 MiB used, 1.0 TiB / 1.0 TiB avail; 342 KiB/s rd, 354 KiB/s wr, 79 op/s 2024-03-24T00:23:08.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:23:07 smithi193 ceph-mon[36878]: pgmap v641: 97 pgs: 97 active+clean; 146 MiB data, 810 MiB used, 1.0 TiB / 1.0 TiB avail; 342 KiB/s rd, 354 KiB/s wr, 79 op/s 2024-03-24T00:23:08.442 INFO:journalctl@ceph.mds.g.smithi118.stdout:Mar 24 00:23:08 smithi118 podman[95403]: 2024-03-24 00:23:08.045019185 +0000 UTC m=+0.037118174 image pull c664afe56bb01db4f9afa9532c20598e201e664f2e9ba2be50db0d52674a26d2 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc 2024-03-24T00:23:08.442 INFO:journalctl@ceph.mds.g.smithi118.stdout:Mar 24 00:23:08 smithi118 podman[95403]: 2024-03-24 00:23:08.440485828 +0000 UTC m=+0.432584836 container create 3a65d19c304e930df858d4b40707683f229feaa70ee0d1e1ca3d5c91b819120f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-g, GIT_CLEAN=True, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, CEPH_POINT_RELEASE=, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240319) 2024-03-24T00:23:08.442 INFO:journalctl@ceph.mds.d.smithi118.stdout:Mar 24 00:23:08 smithi118 podman[95402]: 2024-03-24 00:23:08.033704157 +0000 UTC m=+0.035402223 image pull c664afe56bb01db4f9afa9532c20598e201e664f2e9ba2be50db0d52674a26d2 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc 2024-03-24T00:23:08.442 INFO:journalctl@ceph.mds.d.smithi118.stdout:Mar 24 00:23:08 smithi118 podman[95402]: 2024-03-24 00:23:08.179712814 +0000 UTC m=+0.181410897 container create 274eb342f15de2c333ada17081c1c027f0db79f869b5dda0e9e6b4a0e90f42d8 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-d, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.license=GPLv2, org.label-schema.build-date=20240319, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, ceph=True, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-24T00:23:08.696 INFO:journalctl@ceph.mds.d.smithi118.stdout:Mar 24 00:23:08 smithi118 podman[95402]: 2024-03-24 00:23:08.611336146 +0000 UTC m=+0.613034286 container init 274eb342f15de2c333ada17081c1c027f0db79f869b5dda0e9e6b4a0e90f42d8 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-d, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240319, io.buildah.version=1.33.5, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux ) 2024-03-24T00:23:08.696 INFO:journalctl@ceph.mds.d.smithi118.stdout:Mar 24 00:23:08 smithi118 podman[95402]: 2024-03-24 00:23:08.620018157 +0000 UTC m=+0.621716247 container start 274eb342f15de2c333ada17081c1c027f0db79f869b5dda0e9e6b4a0e90f42d8 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-d, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, io.buildah.version=1.33.5, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240319) 2024-03-24T00:23:08.696 INFO:journalctl@ceph.mds.d.smithi118.stdout:Mar 24 00:23:08 smithi118 ceph-mds[95437]: -- 172.21.15.118:0/1334898042 <== mon.2 v2:172.21.15.189:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x55e7e53f9a40 con 0x55e7e53cc800 2024-03-24T00:23:08.696 INFO:journalctl@ceph.mds.d.smithi118.stdout:Mar 24 00:23:08 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-d[95429]: starting mds.d at 2024-03-24T00:23:08.995 INFO:journalctl@ceph.mds.d.smithi118.stdout:Mar 24 00:23:08 smithi118 bash[95402]: 274eb342f15de2c333ada17081c1c027f0db79f869b5dda0e9e6b4a0e90f42d8 2024-03-24T00:23:08.995 INFO:journalctl@ceph.mds.d.smithi118.stdout:Mar 24 00:23:08 smithi118 systemd[1]: Started Ceph mds.d for d79a5420-e971-11ee-95cd-87774f69a715. 2024-03-24T00:23:08.995 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:23:08 smithi118 ceph-mon[31535]: daemon mds.d restarted 2024-03-24T00:23:08.995 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:23:08 smithi118 ceph-mon[31535]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-03-24T00:23:08.995 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:23:08 smithi118 ceph-mon[31535]: osdmap e111: 12 total, 12 up, 12 in 2024-03-24T00:23:08.995 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:23:08 smithi118 ceph-mon[31535]: mds.? [v2:172.21.15.118:6836/3985972391,v1:172.21.15.118:6837/3985972391] up:boot 2024-03-24T00:23:08.995 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:23:08 smithi118 ceph-mon[31535]: Standby daemon mds.j assigned to filesystem cephfs as rank 3 2024-03-24T00:23:08.995 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:23:08 smithi118 ceph-mon[31535]: fsmap cephfs:4/5 {0=b=up:active,1=g=up:active,2=c=up:active,4=e=up:active} 5 up:standby-replay 1 up:standby, 1 failed 2024-03-24T00:23:08.996 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:23:08 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2024-03-24T00:23:08.996 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:23:08 smithi118 ceph-mon[31535]: fsmap cephfs:5/5 4 up:standby-replay 1 up:standby 1 up:replay 4 up:active 2024-03-24T00:23:08.996 INFO:journalctl@ceph.mds.g.smithi118.stdout:Mar 24 00:23:08 smithi118 podman[95403]: 2024-03-24 00:23:08.885112093 +0000 UTC m=+0.877211108 container init 3a65d19c304e930df858d4b40707683f229feaa70ee0d1e1ca3d5c91b819120f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-g, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, org.label-schema.build-date=20240319, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.license=GPLv2, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30) 2024-03-24T00:23:08.996 INFO:journalctl@ceph.mds.g.smithi118.stdout:Mar 24 00:23:08 smithi118 podman[95403]: 2024-03-24 00:23:08.894505488 +0000 UTC m=+0.886604473 container start 3a65d19c304e930df858d4b40707683f229feaa70ee0d1e1ca3d5c91b819120f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-g, GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, ceph=True, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240319, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, org.label-schema.license=GPLv2) 2024-03-24T00:23:08.996 INFO:journalctl@ceph.mds.g.smithi118.stdout:Mar 24 00:23:08 smithi118 ceph-mds[95458]: -- 172.21.15.118:0/1200350249 <== mon.1 v2:172.21.15.193:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x5589cc5a5a40 con 0x5589cc578800 2024-03-24T00:23:08.996 INFO:journalctl@ceph.mds.g.smithi118.stdout:Mar 24 00:23:08 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-g[95433]: starting mds.g at 2024-03-24T00:23:09.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:23:08 smithi189 ceph-mon[36187]: daemon mds.d restarted 2024-03-24T00:23:09.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:23:08 smithi189 ceph-mon[36187]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-03-24T00:23:09.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:23:08 smithi189 ceph-mon[36187]: osdmap e111: 12 total, 12 up, 12 in 2024-03-24T00:23:09.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:23:08 smithi189 ceph-mon[36187]: mds.? [v2:172.21.15.118:6836/3985972391,v1:172.21.15.118:6837/3985972391] up:boot 2024-03-24T00:23:09.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:23:08 smithi189 ceph-mon[36187]: Standby daemon mds.j assigned to filesystem cephfs as rank 3 2024-03-24T00:23:09.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:23:08 smithi189 ceph-mon[36187]: fsmap cephfs:4/5 {0=b=up:active,1=g=up:active,2=c=up:active,4=e=up:active} 5 up:standby-replay 1 up:standby, 1 failed 2024-03-24T00:23:09.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:23:08 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2024-03-24T00:23:09.205 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:23:08 smithi189 ceph-mon[36187]: fsmap cephfs:5/5 4 up:standby-replay 1 up:standby 1 up:replay 4 up:active 2024-03-24T00:23:09.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:23:08 smithi193 ceph-mon[36878]: daemon mds.d restarted 2024-03-24T00:23:09.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:23:08 smithi193 ceph-mon[36878]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-03-24T00:23:09.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:23:08 smithi193 ceph-mon[36878]: osdmap e111: 12 total, 12 up, 12 in 2024-03-24T00:23:09.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:23:08 smithi193 ceph-mon[36878]: mds.? [v2:172.21.15.118:6836/3985972391,v1:172.21.15.118:6837/3985972391] up:boot 2024-03-24T00:23:09.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:23:08 smithi193 ceph-mon[36878]: Standby daemon mds.j assigned to filesystem cephfs as rank 3 2024-03-24T00:23:09.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:23:08 smithi193 ceph-mon[36878]: fsmap cephfs:4/5 {0=b=up:active,1=g=up:active,2=c=up:active,4=e=up:active} 5 up:standby-replay 1 up:standby, 1 failed 2024-03-24T00:23:09.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:23:08 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2024-03-24T00:23:09.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:23:08 smithi193 ceph-mon[36878]: fsmap cephfs:5/5 4 up:standby-replay 1 up:standby 1 up:replay 4 up:active 2024-03-24T00:23:09.446 INFO:journalctl@ceph.mds.g.smithi118.stdout:Mar 24 00:23:08 smithi118 bash[95403]: 3a65d19c304e930df858d4b40707683f229feaa70ee0d1e1ca3d5c91b819120f 2024-03-24T00:23:09.446 INFO:journalctl@ceph.mds.g.smithi118.stdout:Mar 24 00:23:09 smithi118 systemd[1]: Started Ceph mds.g for d79a5420-e971-11ee-95cd-87774f69a715. 2024-03-24T00:23:10.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:23:09 smithi118 ceph-mon[31535]: daemon mds.g restarted 2024-03-24T00:23:10.197 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:23:09 smithi118 ceph-mon[31535]: osdmap e112: 12 total, 12 up, 12 in 2024-03-24T00:23:10.197 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:23:09 smithi118 ceph-mon[31535]: mds.? [v2:172.21.15.118:6834/3194741373,v1:172.21.15.118:6835/3194741373] up:resolve 2024-03-24T00:23:10.197 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:23:09 smithi118 ceph-mon[31535]: mds.? [v2:172.21.15.118:6838/282013400,v1:172.21.15.118:6839/282013400] up:boot 2024-03-24T00:23:10.197 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:23:09 smithi118 ceph-mon[31535]: Standby daemon mds.h assigned to filesystem cephfs as rank 1 2024-03-24T00:23:10.197 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:23:09 smithi118 ceph-mon[31535]: fsmap cephfs:4/5 {0=b=up:active,2=c=up:active,3=j=up:resolve,4=e=up:active} 4 up:standby-replay 2 up:standby, 1 failed 2024-03-24T00:23:10.197 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:23:09 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2024-03-24T00:23:10.197 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:23:09 smithi118 ceph-mon[31535]: fsmap cephfs:5/5 3 up:standby-replay 2 up:standby 1 up:replay 1 up:resolve 3 up:active 2024-03-24T00:23:10.197 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:23:09 smithi118 ceph-mon[31535]: pgmap v644: 97 pgs: 97 active+clean; 146 MiB data, 810 MiB used, 1.0 TiB / 1.0 TiB avail; 66 KiB/s rd, 388 KiB/s wr, 39 op/s 2024-03-24T00:23:10.203 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:23:09 smithi189 ceph-mon[36187]: daemon mds.g restarted 2024-03-24T00:23:10.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:23:09 smithi189 ceph-mon[36187]: osdmap e112: 12 total, 12 up, 12 in 2024-03-24T00:23:10.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:23:09 smithi189 ceph-mon[36187]: mds.? [v2:172.21.15.118:6834/3194741373,v1:172.21.15.118:6835/3194741373] up:resolve 2024-03-24T00:23:10.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:23:09 smithi189 ceph-mon[36187]: mds.? [v2:172.21.15.118:6838/282013400,v1:172.21.15.118:6839/282013400] up:boot 2024-03-24T00:23:10.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:23:09 smithi189 ceph-mon[36187]: Standby daemon mds.h assigned to filesystem cephfs as rank 1 2024-03-24T00:23:10.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:23:09 smithi189 ceph-mon[36187]: fsmap cephfs:4/5 {0=b=up:active,2=c=up:active,3=j=up:resolve,4=e=up:active} 4 up:standby-replay 2 up:standby, 1 failed 2024-03-24T00:23:10.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:23:09 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2024-03-24T00:23:10.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:23:09 smithi189 ceph-mon[36187]: fsmap cephfs:5/5 3 up:standby-replay 2 up:standby 1 up:replay 1 up:resolve 3 up:active 2024-03-24T00:23:10.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:23:09 smithi189 ceph-mon[36187]: pgmap v644: 97 pgs: 97 active+clean; 146 MiB data, 810 MiB used, 1.0 TiB / 1.0 TiB avail; 66 KiB/s rd, 388 KiB/s wr, 39 op/s 2024-03-24T00:23:10.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:23:09 smithi193 ceph-mon[36878]: daemon mds.g restarted 2024-03-24T00:23:10.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:23:09 smithi193 ceph-mon[36878]: osdmap e112: 12 total, 12 up, 12 in 2024-03-24T00:23:10.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:23:09 smithi193 ceph-mon[36878]: mds.? [v2:172.21.15.118:6834/3194741373,v1:172.21.15.118:6835/3194741373] up:resolve 2024-03-24T00:23:10.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:23:09 smithi193 ceph-mon[36878]: mds.? [v2:172.21.15.118:6838/282013400,v1:172.21.15.118:6839/282013400] up:boot 2024-03-24T00:23:10.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:23:09 smithi193 ceph-mon[36878]: Standby daemon mds.h assigned to filesystem cephfs as rank 1 2024-03-24T00:23:10.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:23:09 smithi193 ceph-mon[36878]: fsmap cephfs:4/5 {0=b=up:active,2=c=up:active,3=j=up:resolve,4=e=up:active} 4 up:standby-replay 2 up:standby, 1 failed 2024-03-24T00:23:10.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:23:09 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "g"}]: dispatch 2024-03-24T00:23:10.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:23:09 smithi193 ceph-mon[36878]: fsmap cephfs:5/5 3 up:standby-replay 2 up:standby 1 up:replay 1 up:resolve 3 up:active 2024-03-24T00:23:10.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:23:09 smithi193 ceph-mon[36878]: pgmap v644: 97 pgs: 97 active+clean; 146 MiB data, 810 MiB used, 1.0 TiB / 1.0 TiB avail; 66 KiB/s rd, 388 KiB/s wr, 39 op/s 2024-03-24T00:23:11.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:23:10 smithi118 ceph-mon[31535]: mds.? [v2:172.21.15.189:6836/635914573,v1:172.21.15.189:6837/635914573] up:resolve 2024-03-24T00:23:11.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:23:10 smithi118 ceph-mon[31535]: fsmap cephfs:5/5 3 up:standby-replay 2 up:standby 2 up:resolve 3 up:active 2024-03-24T00:23:11.453 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:23:10 smithi189 ceph-mon[36187]: mds.? [v2:172.21.15.189:6836/635914573,v1:172.21.15.189:6837/635914573] up:resolve 2024-03-24T00:23:11.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:23:10 smithi189 ceph-mon[36187]: fsmap cephfs:5/5 3 up:standby-replay 2 up:standby 2 up:resolve 3 up:active 2024-03-24T00:23:11.460 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:23:10 smithi193 ceph-mon[36878]: mds.? [v2:172.21.15.189:6836/635914573,v1:172.21.15.189:6837/635914573] up:resolve 2024-03-24T00:23:11.460 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:23:10 smithi193 ceph-mon[36878]: fsmap cephfs:5/5 3 up:standby-replay 2 up:standby 2 up:resolve 3 up:active 2024-03-24T00:23:12.026 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:23:12.025+0000 7f78a0c00640 1 -- 172.21.15.118:0/427172033 <== mds.0 v2:172.21.15.189:6838/4074484168 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+44 (crc 0 0 0) 0x55b9cb3690c0 con 0x55b9cb46e0f0 2024-03-24T00:23:12.026 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:23:12.025+0000 7f78a0c00640 10 client.26903 handle_command_reply: tid=0 2024-03-24T00:23:12.027 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:23:12.025+0000 7f78c0bf2740 1 client.26903 shutdown 2024-03-24T00:23:12.028 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:23:12.026+0000 7f78c0bf2740 1 -- 172.21.15.118:0/427172033 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55b9cb3fc1e0 msgr2=0x55b9cb40b170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:23:12.028 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:23:12.026+0000 7f78c0bf2740 1 --2- 172.21.15.118:0/427172033 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55b9cb3fc1e0 0x55b9cb40b170 secure :-1 s=READY pgs=447 cs=0 l=1 rev1=1 crypto rx=0x7f78a8097d00 tx=0x7f78a8099da0 comp rx=0 tx=0).stop 2024-03-24T00:23:12.028 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:23:12.026+0000 7f78c0bf2740 1 -- 172.21.15.118:0/427172033 shutdown_connections 2024-03-24T00:23:12.028 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:23:12.026+0000 7f78c0bf2740 1 --2- 172.21.15.118:0/427172033 >> [v2:172.21.15.189:6838/4074484168,v1:172.21.15.189:6839/4074484168] conn(0x55b9cb46e0f0 0x55b9cb4704e0 crc :-1 s=READY pgs=298 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:23:12.028 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:23:12.026+0000 7f78c0bf2740 1 --2- 172.21.15.118:0/427172033 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55b9cb401d40 0x55b9cb40c6c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:23:12.028 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:23:12.026+0000 7f78c0bf2740 1 --2- 172.21.15.118:0/427172033 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55b9cb3fcbd0 0x55b9cb40b6b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:23:12.028 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:23:12.026+0000 7f78c0bf2740 1 --2- 172.21.15.118:0/427172033 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55b9cb3fc1e0 0x55b9cb40b170 unknown :-1 s=CLOSED pgs=447 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:23:12.028 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:23:12.026+0000 7f78c0bf2740 1 -- 172.21.15.118:0/427172033 >> 172.21.15.118:0/427172033 conn(0x55b9cb3f7750 msgr2=0x55b9cb40e360 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:23:12.028 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:23:12.026+0000 7f78a0c00640 0 client.26903 ms_handle_reset on v2:172.21.15.189:6838/4074484168 2024-03-24T00:23:12.028 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:23:12.027+0000 7f78c0bf2740 1 -- 172.21.15.118:0/427172033 shutdown_connections 2024-03-24T00:23:12.028 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:23:12.027+0000 7f78c0bf2740 1 -- 172.21.15.118:0/427172033 wait complete. 2024-03-24T00:23:12.029 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:23:12.027+0000 7f78c0bf2740 20 client.26903 trim_cache size 0 max 16384 2024-03-24T00:23:12.029 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stdout:{ 2024-03-24T00:23:12.029 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stdout: "message": "", 2024-03-24T00:23:12.029 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stdout: "return_code": 0 2024-03-24T00:23:12.029 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stdout:} 2024-03-24T00:23:12.029 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:23:12.028+0000 7f78be000640 1 -- 172.21.15.118:0/2878504073 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7f789c0a2920 msgr2=0x7f789c0a4de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:23:12.030 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:23:12.028+0000 7f78be000640 1 --2- 172.21.15.118:0/2878504073 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7f789c0a2920 0x7f789c0a4de0 secure :-1 s=READY pgs=382 cs=0 l=1 rev1=1 crypto rx=0x55b9cb3fd430 tx=0x7f78b000d040 comp rx=0 tx=0).stop 2024-03-24T00:23:12.030 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:23:12.029+0000 7f78be000640 1 -- 172.21.15.118:0/2878504073 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f78b81038f0 msgr2=0x7f78b8079450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:23:12.030 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:23:12.029+0000 7f78be000640 1 --2- 172.21.15.118:0/2878504073 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f78b81038f0 0x7f78b8079450 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7f78a800e9c0 tx=0x7f78a800ee80 comp rx=0 tx=0).stop 2024-03-24T00:23:12.030 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:23:12.029+0000 7f78be000640 1 -- 172.21.15.118:0/2878504073 shutdown_connections 2024-03-24T00:23:12.031 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:23:12.029+0000 7f78be000640 1 --2- 172.21.15.118:0/2878504073 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7f789c0a2920 0x7f789c0a4de0 unknown :-1 s=CLOSED pgs=382 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:23:12.031 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:23:12.029+0000 7f78be000640 1 --2- 172.21.15.118:0/2878504073 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f78b81054a0 0x7f78b8075fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:23:12.031 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:23:12.029+0000 7f78be000640 1 --2- 172.21.15.118:0/2878504073 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f78b8104ae0 0x7f78b8079990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:23:12.031 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:23:12.029+0000 7f78be000640 1 --2- 172.21.15.118:0/2878504073 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f78b81038f0 0x7f78b8079450 unknown :-1 s=CLOSED pgs=220 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:23:12.031 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:23:12.029+0000 7f78be000640 1 -- 172.21.15.118:0/2878504073 >> 172.21.15.118:0/2878504073 conn(0x7f78b80fee60 msgr2=0x7f78b8107d80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:23:12.031 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:23:12.029+0000 7f78be000640 1 -- 172.21.15.118:0/2878504073 shutdown_connections 2024-03-24T00:23:12.031 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:23:12.029+0000 7f78be000640 1 -- 172.21.15.118:0/2878504073 wait complete. 2024-03-24T00:23:12.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:23:11 smithi118 ceph-mon[31535]: mds.? [v2:172.21.15.189:6836/635914573,v1:172.21.15.189:6837/635914573] up:reconnect 2024-03-24T00:23:12.447 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:23:11 smithi118 ceph-mon[31535]: mds.? [v2:172.21.15.118:6834/3194741373,v1:172.21.15.118:6835/3194741373] up:reconnect 2024-03-24T00:23:12.447 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:23:11 smithi118 ceph-mon[31535]: fsmap cephfs:5/5 3 up:standby-replay 2 up:standby 2 up:reconnect 3 up:active 2024-03-24T00:23:12.447 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:23:11 smithi118 ceph-mon[31535]: reconnect by client.15159 192.168.0.1:0/3463739313 after 0.0229996 2024-03-24T00:23:12.447 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:23:11 smithi118 ceph-mon[31535]: reconnect by client.15159 192.168.0.1:0/3463739313 after 0.0319993 2024-03-24T00:23:12.447 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:23:11 smithi118 ceph-mon[31535]: pgmap v645: 97 pgs: 97 active+clean; 146 MiB data, 850 MiB used, 1.0 TiB / 1.0 TiB avail; 34 KiB/s rd, 168 KiB/s wr, 34 op/s 2024-03-24T00:23:12.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:23:11 smithi189 ceph-mon[36187]: mds.? [v2:172.21.15.189:6836/635914573,v1:172.21.15.189:6837/635914573] up:reconnect 2024-03-24T00:23:12.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:23:11 smithi189 ceph-mon[36187]: mds.? [v2:172.21.15.118:6834/3194741373,v1:172.21.15.118:6835/3194741373] up:reconnect 2024-03-24T00:23:12.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:23:11 smithi189 ceph-mon[36187]: fsmap cephfs:5/5 3 up:standby-replay 2 up:standby 2 up:reconnect 3 up:active 2024-03-24T00:23:12.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:23:11 smithi189 ceph-mon[36187]: reconnect by client.15159 192.168.0.1:0/3463739313 after 0.0229996 2024-03-24T00:23:12.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:23:11 smithi189 ceph-mon[36187]: reconnect by client.15159 192.168.0.1:0/3463739313 after 0.0319993 2024-03-24T00:23:12.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:23:11 smithi189 ceph-mon[36187]: pgmap v645: 97 pgs: 97 active+clean; 146 MiB data, 850 MiB used, 1.0 TiB / 1.0 TiB avail; 34 KiB/s rd, 168 KiB/s wr, 34 op/s 2024-03-24T00:23:12.460 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:23:11 smithi193 ceph-mon[36878]: mds.? [v2:172.21.15.189:6836/635914573,v1:172.21.15.189:6837/635914573] up:reconnect 2024-03-24T00:23:12.460 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:23:11 smithi193 ceph-mon[36878]: mds.? [v2:172.21.15.118:6834/3194741373,v1:172.21.15.118:6835/3194741373] up:reconnect 2024-03-24T00:23:12.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:23:11 smithi193 ceph-mon[36878]: fsmap cephfs:5/5 3 up:standby-replay 2 up:standby 2 up:reconnect 3 up:active 2024-03-24T00:23:12.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:23:11 smithi193 ceph-mon[36878]: reconnect by client.15159 192.168.0.1:0/3463739313 after 0.0229996 2024-03-24T00:23:12.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:23:11 smithi193 ceph-mon[36878]: reconnect by client.15159 192.168.0.1:0/3463739313 after 0.0319993 2024-03-24T00:23:12.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:23:11 smithi193 ceph-mon[36878]: pgmap v645: 97 pgs: 97 active+clean; 146 MiB data, 850 MiB used, 1.0 TiB / 1.0 TiB avail; 34 KiB/s rd, 168 KiB/s wr, 34 op/s 2024-03-24T00:23:13.348 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:23:12 smithi118 ceph-mon[31535]: mds.? [v2:172.21.15.189:6836/635914573,v1:172.21.15.189:6837/635914573] up:rejoin 2024-03-24T00:23:13.348 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:23:12 smithi118 ceph-mon[31535]: mds.? [v2:172.21.15.118:6834/3194741373,v1:172.21.15.118:6835/3194741373] up:rejoin 2024-03-24T00:23:13.348 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:23:12 smithi118 ceph-mon[31535]: fsmap cephfs:5/5 3 up:standby-replay 2 up:standby 2 up:rejoin 3 up:active 2024-03-24T00:23:13.348 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:23:12 smithi118 ceph-mon[31535]: daemon mds.j is now active in filesystem cephfs as rank 3 2024-03-24T00:23:13.348 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:23:12 smithi118 ceph-mon[31535]: daemon mds.h is now active in filesystem cephfs as rank 1 2024-03-24T00:23:13.349 INFO:journalctl@ceph.mds.j.smithi118.stdout:Mar 24 00:23:13 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-j[72625]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: In function 'void Locker::set_xlocks_done(MutationImpl*, bool)' thread 7fa129400700 time 2024-03-24T00:23:13.056234+0000 2024-03-24T00:23:13.349 INFO:journalctl@ceph.mds.j.smithi118.stdout:Mar 24 00:23:13 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-j[72625]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: 626: FAILED ceph_assert(obj->is_auth()) 2024-03-24T00:23:13.349 INFO:journalctl@ceph.mds.j.smithi118.stdout:Mar 24 00:23:13 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-j[72625]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-24T00:23:13.349 INFO:journalctl@ceph.mds.j.smithi118.stdout:Mar 24 00:23:13 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-j[72625]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x135) [0x7fa134c574a1] 2024-03-24T00:23:13.349 INFO:journalctl@ceph.mds.j.smithi118.stdout:Mar 24 00:23:13 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-j[72625]: 2: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7fa134c57667] 2024-03-24T00:23:13.349 INFO:journalctl@ceph.mds.j.smithi118.stdout:Mar 24 00:23:13 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-j[72625]: 3: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x55a5db5e1223] 2024-03-24T00:23:13.349 INFO:journalctl@ceph.mds.j.smithi118.stdout:Mar 24 00:23:13 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-j[72625]: 4: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x55a5db45b5d6] 2024-03-24T00:23:13.349 INFO:journalctl@ceph.mds.j.smithi118.stdout:Mar 24 00:23:13 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-j[72625]: 5: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x55a5db45bda1] 2024-03-24T00:23:13.349 INFO:journalctl@ceph.mds.j.smithi118.stdout:Mar 24 00:23:13 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-j[72625]: 6: (Server::handle_client_rename(boost::intrusive_ptr const&)+0x259d) [0x55a5db49897d] 2024-03-24T00:23:13.349 INFO:journalctl@ceph.mds.j.smithi118.stdout:Mar 24 00:23:13 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-j[72625]: 7: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x55a5db4ae233] 2024-03-24T00:23:13.349 INFO:journalctl@ceph.mds.j.smithi118.stdout:Mar 24 00:23:13 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-j[72625]: 8: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x55a5db4b0197] 2024-03-24T00:23:13.349 INFO:journalctl@ceph.mds.j.smithi118.stdout:Mar 24 00:23:13 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-j[72625]: 9: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x55a5db3af11b] 2024-03-24T00:23:13.349 INFO:journalctl@ceph.mds.j.smithi118.stdout:Mar 24 00:23:13 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-j[72625]: 10: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x55a5db3af6bc] 2024-03-24T00:23:13.349 INFO:journalctl@ceph.mds.j.smithi118.stdout:Mar 24 00:23:13 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-j[72625]: 11: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x55a5db3991ff] 2024-03-24T00:23:13.349 INFO:journalctl@ceph.mds.j.smithi118.stdout:Mar 24 00:23:13 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-j[72625]: 12: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7fa134ee9488] 2024-03-24T00:23:13.350 INFO:journalctl@ceph.mds.j.smithi118.stdout:Mar 24 00:23:13 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-j[72625]: 13: (DispatchQueue::entry()+0x51f) [0x7fa134ee660f] 2024-03-24T00:23:13.350 INFO:journalctl@ceph.mds.j.smithi118.stdout:Mar 24 00:23:13 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-j[72625]: 14: (DispatchQueue::DispatchThread::entry()+0x11) [0x7fa134fb9e81] 2024-03-24T00:23:13.350 INFO:journalctl@ceph.mds.j.smithi118.stdout:Mar 24 00:23:13 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-j[72625]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7fa1339d41ca] 2024-03-24T00:23:13.350 INFO:journalctl@ceph.mds.j.smithi118.stdout:Mar 24 00:23:13 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-j[72625]: 16: clone() 2024-03-24T00:23:13.350 INFO:journalctl@ceph.mds.j.smithi118.stdout:Mar 24 00:23:13 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-j[72625]: 2024-03-24T00:23:13.056+0000 7fa129400700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: In function 'void Locker::set_xlocks_done(MutationImpl*, bool)' thread 7fa129400700 time 2024-03-24T00:23:13.056234+0000 2024-03-24T00:23:13.350 INFO:journalctl@ceph.mds.j.smithi118.stdout:Mar 24 00:23:13 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-j[72625]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: 626: FAILED ceph_assert(obj->is_auth()) 2024-03-24T00:23:13.351 INFO:journalctl@ceph.mds.j.smithi118.stdout:Mar 24 00:23:13 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-j[72625]: 2024-03-24T00:23:13.351 INFO:journalctl@ceph.mds.j.smithi118.stdout:Mar 24 00:23:13 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-j[72625]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-24T00:23:13.351 INFO:journalctl@ceph.mds.j.smithi118.stdout:Mar 24 00:23:13 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-j[72625]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x135) [0x7fa134c574a1] 2024-03-24T00:23:13.351 INFO:journalctl@ceph.mds.j.smithi118.stdout:Mar 24 00:23:13 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-j[72625]: 2: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7fa134c57667] 2024-03-24T00:23:13.351 INFO:journalctl@ceph.mds.j.smithi118.stdout:Mar 24 00:23:13 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-j[72625]: 3: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x55a5db5e1223] 2024-03-24T00:23:13.351 INFO:journalctl@ceph.mds.j.smithi118.stdout:Mar 24 00:23:13 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-j[72625]: 4: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x55a5db45b5d6] 2024-03-24T00:23:13.351 INFO:journalctl@ceph.mds.j.smithi118.stdout:Mar 24 00:23:13 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-j[72625]: 5: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x55a5db45bda1] 2024-03-24T00:23:13.351 INFO:journalctl@ceph.mds.j.smithi118.stdout:Mar 24 00:23:13 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-j[72625]: 6: (Server::handle_client_rename(boost::intrusive_ptr const&)+0x259d) [0x55a5db49897d] 2024-03-24T00:23:13.351 INFO:journalctl@ceph.mds.j.smithi118.stdout:Mar 24 00:23:13 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-j[72625]: 7: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x55a5db4ae233] 2024-03-24T00:23:13.351 INFO:journalctl@ceph.mds.j.smithi118.stdout:Mar 24 00:23:13 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-j[72625]: 8: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x55a5db4b0197] 2024-03-24T00:23:13.351 INFO:journalctl@ceph.mds.j.smithi118.stdout:Mar 24 00:23:13 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-j[72625]: 9: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x55a5db3af11b] 2024-03-24T00:23:13.351 INFO:journalctl@ceph.mds.j.smithi118.stdout:Mar 24 00:23:13 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-j[72625]: 10: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x55a5db3af6bc] 2024-03-24T00:23:13.351 INFO:journalctl@ceph.mds.j.smithi118.stdout:Mar 24 00:23:13 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-j[72625]: 11: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x55a5db3991ff] 2024-03-24T00:23:13.351 INFO:journalctl@ceph.mds.j.smithi118.stdout:Mar 24 00:23:13 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-j[72625]: 12: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7fa134ee9488] 2024-03-24T00:23:13.352 INFO:journalctl@ceph.mds.j.smithi118.stdout:Mar 24 00:23:13 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-j[72625]: 13: (DispatchQueue::entry()+0x51f) [0x7fa134ee660f] 2024-03-24T00:23:13.352 INFO:journalctl@ceph.mds.j.smithi118.stdout:Mar 24 00:23:13 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-j[72625]: 14: (DispatchQueue::DispatchThread::entry()+0x11) [0x7fa134fb9e81] 2024-03-24T00:23:13.352 INFO:journalctl@ceph.mds.j.smithi118.stdout:Mar 24 00:23:13 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-j[72625]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7fa1339d41ca] 2024-03-24T00:23:13.352 INFO:journalctl@ceph.mds.j.smithi118.stdout:Mar 24 00:23:13 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-j[72625]: 16: clone() 2024-03-24T00:23:13.352 INFO:journalctl@ceph.mds.j.smithi118.stdout:Mar 24 00:23:13 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-j[72625]: 2024-03-24T00:23:13.352 INFO:journalctl@ceph.mds.j.smithi118.stdout:Mar 24 00:23:13 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-j[72625]: *** Caught signal (Aborted) ** 2024-03-24T00:23:13.352 INFO:journalctl@ceph.mds.j.smithi118.stdout:Mar 24 00:23:13 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-j[72625]: in thread 7fa129400700 thread_name:ms_dispatch 2024-03-24T00:23:13.352 INFO:journalctl@ceph.mds.j.smithi118.stdout:Mar 24 00:23:13 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-j[72625]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-24T00:23:13.352 INFO:journalctl@ceph.mds.j.smithi118.stdout:Mar 24 00:23:13 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-j[72625]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7fa1339ded20] 2024-03-24T00:23:13.352 INFO:journalctl@ceph.mds.j.smithi118.stdout:Mar 24 00:23:13 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-j[72625]: 2: gsignal() 2024-03-24T00:23:13.352 INFO:journalctl@ceph.mds.j.smithi118.stdout:Mar 24 00:23:13 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-j[72625]: 3: abort() 2024-03-24T00:23:13.352 INFO:journalctl@ceph.mds.j.smithi118.stdout:Mar 24 00:23:13 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-j[72625]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x18f) [0x7fa134c574fb] 2024-03-24T00:23:13.352 INFO:journalctl@ceph.mds.j.smithi118.stdout:Mar 24 00:23:13 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-j[72625]: 5: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7fa134c57667] 2024-03-24T00:23:13.352 INFO:journalctl@ceph.mds.j.smithi118.stdout:Mar 24 00:23:13 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-j[72625]: 6: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x55a5db5e1223] 2024-03-24T00:23:13.353 INFO:journalctl@ceph.mds.j.smithi118.stdout:Mar 24 00:23:13 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-j[72625]: 7: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x55a5db45b5d6] 2024-03-24T00:23:13.353 INFO:journalctl@ceph.mds.j.smithi118.stdout:Mar 24 00:23:13 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-j[72625]: 8: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x55a5db45bda1] 2024-03-24T00:23:13.353 INFO:journalctl@ceph.mds.j.smithi118.stdout:Mar 24 00:23:13 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-j[72625]: 9: (Server::handle_client_rename(boost::intrusive_ptr const&)+0x259d) [0x55a5db49897d] 2024-03-24T00:23:13.353 INFO:journalctl@ceph.mds.j.smithi118.stdout:Mar 24 00:23:13 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-j[72625]: 10: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x55a5db4ae233] 2024-03-24T00:23:13.353 INFO:journalctl@ceph.mds.j.smithi118.stdout:Mar 24 00:23:13 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-j[72625]: 11: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x55a5db4b0197] 2024-03-24T00:23:13.353 INFO:journalctl@ceph.mds.j.smithi118.stdout:Mar 24 00:23:13 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-j[72625]: 12: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x55a5db3af11b] 2024-03-24T00:23:13.353 INFO:journalctl@ceph.mds.j.smithi118.stdout:Mar 24 00:23:13 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-j[72625]: 13: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x55a5db3af6bc] 2024-03-24T00:23:13.353 INFO:journalctl@ceph.mds.j.smithi118.stdout:Mar 24 00:23:13 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-j[72625]: 14: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x55a5db3991ff] 2024-03-24T00:23:13.353 INFO:journalctl@ceph.mds.j.smithi118.stdout:Mar 24 00:23:13 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-j[72625]: 15: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7fa134ee9488] 2024-03-24T00:23:13.353 INFO:journalctl@ceph.mds.j.smithi118.stdout:Mar 24 00:23:13 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-j[72625]: 16: (DispatchQueue::entry()+0x51f) [0x7fa134ee660f] 2024-03-24T00:23:13.353 INFO:journalctl@ceph.mds.j.smithi118.stdout:Mar 24 00:23:13 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-j[72625]: 17: (DispatchQueue::DispatchThread::entry()+0x11) [0x7fa134fb9e81] 2024-03-24T00:23:13.353 INFO:journalctl@ceph.mds.j.smithi118.stdout:Mar 24 00:23:13 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-j[72625]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7fa1339d41ca] 2024-03-24T00:23:13.353 INFO:journalctl@ceph.mds.j.smithi118.stdout:Mar 24 00:23:13 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-j[72625]: 19: clone() 2024-03-24T00:23:13.353 INFO:journalctl@ceph.mds.j.smithi118.stdout:Mar 24 00:23:13 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-j[72625]: 2024-03-24T00:23:13.058+0000 7fa129400700 -1 *** Caught signal (Aborted) ** 2024-03-24T00:23:13.353 INFO:journalctl@ceph.mds.j.smithi118.stdout:Mar 24 00:23:13 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-j[72625]: in thread 7fa129400700 thread_name:ms_dispatch 2024-03-24T00:23:13.354 INFO:journalctl@ceph.mds.j.smithi118.stdout:Mar 24 00:23:13 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-j[72625]: 2024-03-24T00:23:13.354 INFO:journalctl@ceph.mds.j.smithi118.stdout:Mar 24 00:23:13 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-j[72625]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-24T00:23:13.354 INFO:journalctl@ceph.mds.j.smithi118.stdout:Mar 24 00:23:13 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-j[72625]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7fa1339ded20] 2024-03-24T00:23:13.354 INFO:journalctl@ceph.mds.j.smithi118.stdout:Mar 24 00:23:13 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-j[72625]: 2: gsignal() 2024-03-24T00:23:13.354 INFO:journalctl@ceph.mds.j.smithi118.stdout:Mar 24 00:23:13 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-j[72625]: 3: abort() 2024-03-24T00:23:13.354 INFO:journalctl@ceph.mds.j.smithi118.stdout:Mar 24 00:23:13 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-j[72625]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x18f) [0x7fa134c574fb] 2024-03-24T00:23:13.354 INFO:journalctl@ceph.mds.j.smithi118.stdout:Mar 24 00:23:13 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-j[72625]: 5: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7fa134c57667] 2024-03-24T00:23:13.355 INFO:journalctl@ceph.mds.j.smithi118.stdout:Mar 24 00:23:13 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-j[72625]: 6: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x55a5db5e1223] 2024-03-24T00:23:13.355 INFO:journalctl@ceph.mds.j.smithi118.stdout:Mar 24 00:23:13 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-j[72625]: 7: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x55a5db45b5d6] 2024-03-24T00:23:13.355 INFO:journalctl@ceph.mds.j.smithi118.stdout:Mar 24 00:23:13 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-j[72625]: 8: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x55a5db45bda1] 2024-03-24T00:23:13.355 INFO:journalctl@ceph.mds.j.smithi118.stdout:Mar 24 00:23:13 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-j[72625]: 9: (Server::handle_client_rename(boost::intrusive_ptr const&)+0x259d) [0x55a5db49897d] 2024-03-24T00:23:13.355 INFO:journalctl@ceph.mds.j.smithi118.stdout:Mar 24 00:23:13 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-j[72625]: 10: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x55a5db4ae233] 2024-03-24T00:23:13.355 INFO:journalctl@ceph.mds.j.smithi118.stdout:Mar 24 00:23:13 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-j[72625]: 11: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x55a5db4b0197] 2024-03-24T00:23:13.355 INFO:journalctl@ceph.mds.j.smithi118.stdout:Mar 24 00:23:13 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-j[72625]: 12: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x55a5db3af11b] 2024-03-24T00:23:13.356 INFO:journalctl@ceph.mds.j.smithi118.stdout:Mar 24 00:23:13 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-j[72625]: 13: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x55a5db3af6bc] 2024-03-24T00:23:13.356 INFO:journalctl@ceph.mds.j.smithi118.stdout:Mar 24 00:23:13 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-j[72625]: 14: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x55a5db3991ff] 2024-03-24T00:23:13.356 INFO:journalctl@ceph.mds.j.smithi118.stdout:Mar 24 00:23:13 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-j[72625]: 15: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7fa134ee9488] 2024-03-24T00:23:13.356 INFO:journalctl@ceph.mds.j.smithi118.stdout:Mar 24 00:23:13 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-j[72625]: 16: (DispatchQueue::entry()+0x51f) [0x7fa134ee660f] 2024-03-24T00:23:13.356 INFO:journalctl@ceph.mds.j.smithi118.stdout:Mar 24 00:23:13 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-j[72625]: 17: (DispatchQueue::DispatchThread::entry()+0x11) [0x7fa134fb9e81] 2024-03-24T00:23:13.356 INFO:journalctl@ceph.mds.j.smithi118.stdout:Mar 24 00:23:13 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-j[72625]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7fa1339d41ca] 2024-03-24T00:23:13.356 INFO:journalctl@ceph.mds.j.smithi118.stdout:Mar 24 00:23:13 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-j[72625]: 19: clone() 2024-03-24T00:23:13.356 INFO:journalctl@ceph.mds.j.smithi118.stdout:Mar 24 00:23:13 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-j[72625]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-03-24T00:23:13.356 INFO:journalctl@ceph.mds.j.smithi118.stdout:Mar 24 00:23:13 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-j[72625]: 2024-03-24T00:23:13.356 INFO:journalctl@ceph.mds.j.smithi118.stdout:Mar 24 00:23:13 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-j[72625]: -1> 2024-03-24T00:23:13.056+0000 7fa129400700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: In function 'void Locker::set_xlocks_done(MutationImpl*, bool)' thread 7fa129400700 time 2024-03-24T00:23:13.056234+0000 2024-03-24T00:23:13.356 INFO:journalctl@ceph.mds.j.smithi118.stdout:Mar 24 00:23:13 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-j[72625]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: 626: FAILED ceph_assert(obj->is_auth()) 2024-03-24T00:23:13.356 INFO:journalctl@ceph.mds.j.smithi118.stdout:Mar 24 00:23:13 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-j[72625]: 2024-03-24T00:23:13.356 INFO:journalctl@ceph.mds.j.smithi118.stdout:Mar 24 00:23:13 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-j[72625]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-24T00:23:13.356 INFO:journalctl@ceph.mds.j.smithi118.stdout:Mar 24 00:23:13 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-j[72625]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x135) [0x7fa134c574a1] 2024-03-24T00:23:13.357 INFO:journalctl@ceph.mds.j.smithi118.stdout:Mar 24 00:23:13 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-j[72625]: 2: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7fa134c57667] 2024-03-24T00:23:13.357 INFO:journalctl@ceph.mds.j.smithi118.stdout:Mar 24 00:23:13 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-j[72625]: 3: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x55a5db5e1223] 2024-03-24T00:23:13.357 INFO:journalctl@ceph.mds.j.smithi118.stdout:Mar 24 00:23:13 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-j[72625]: 4: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x55a5db45b5d6] 2024-03-24T00:23:13.357 INFO:journalctl@ceph.mds.j.smithi118.stdout:Mar 24 00:23:13 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-j[72625]: 5: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x55a5db45bda1] 2024-03-24T00:23:13.357 INFO:journalctl@ceph.mds.j.smithi118.stdout:Mar 24 00:23:13 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-j[72625]: 6: (Server::handle_client_rename(boost::intrusive_ptr const&)+0x259d) [0x55a5db49897d] 2024-03-24T00:23:13.357 INFO:journalctl@ceph.mds.j.smithi118.stdout:Mar 24 00:23:13 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-j[72625]: 7: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x55a5db4ae233] 2024-03-24T00:23:13.357 INFO:journalctl@ceph.mds.j.smithi118.stdout:Mar 24 00:23:13 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-j[72625]: 8: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x55a5db4b0197] 2024-03-24T00:23:13.357 INFO:journalctl@ceph.mds.j.smithi118.stdout:Mar 24 00:23:13 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-j[72625]: 9: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x55a5db3af11b] 2024-03-24T00:23:13.357 INFO:journalctl@ceph.mds.j.smithi118.stdout:Mar 24 00:23:13 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-j[72625]: 10: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x55a5db3af6bc] 2024-03-24T00:23:13.357 INFO:journalctl@ceph.mds.j.smithi118.stdout:Mar 24 00:23:13 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-j[72625]: 11: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x55a5db3991ff] 2024-03-24T00:23:13.357 INFO:journalctl@ceph.mds.j.smithi118.stdout:Mar 24 00:23:13 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-j[72625]: 12: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7fa134ee9488] 2024-03-24T00:23:13.357 INFO:journalctl@ceph.mds.j.smithi118.stdout:Mar 24 00:23:13 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-j[72625]: 13: (DispatchQueue::entry()+0x51f) [0x7fa134ee660f] 2024-03-24T00:23:13.357 INFO:journalctl@ceph.mds.j.smithi118.stdout:Mar 24 00:23:13 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-j[72625]: 14: (DispatchQueue::DispatchThread::entry()+0x11) [0x7fa134fb9e81] 2024-03-24T00:23:13.357 INFO:journalctl@ceph.mds.j.smithi118.stdout:Mar 24 00:23:13 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-j[72625]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7fa1339d41ca] 2024-03-24T00:23:13.358 INFO:journalctl@ceph.mds.j.smithi118.stdout:Mar 24 00:23:13 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-j[72625]: 16: clone() 2024-03-24T00:23:13.358 INFO:journalctl@ceph.mds.j.smithi118.stdout:Mar 24 00:23:13 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-j[72625]: 2024-03-24T00:23:13.358 INFO:journalctl@ceph.mds.j.smithi118.stdout:Mar 24 00:23:13 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-j[72625]: 0> 2024-03-24T00:23:13.058+0000 7fa129400700 -1 *** Caught signal (Aborted) ** 2024-03-24T00:23:13.358 INFO:journalctl@ceph.mds.j.smithi118.stdout:Mar 24 00:23:13 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-j[72625]: in thread 7fa129400700 thread_name:ms_dispatch 2024-03-24T00:23:13.358 INFO:journalctl@ceph.mds.j.smithi118.stdout:Mar 24 00:23:13 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-j[72625]: 2024-03-24T00:23:13.358 INFO:journalctl@ceph.mds.j.smithi118.stdout:Mar 24 00:23:13 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-j[72625]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-24T00:23:13.358 INFO:journalctl@ceph.mds.j.smithi118.stdout:Mar 24 00:23:13 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-j[72625]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7fa1339ded20] 2024-03-24T00:23:13.358 INFO:journalctl@ceph.mds.j.smithi118.stdout:Mar 24 00:23:13 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-j[72625]: 2: gsignal() 2024-03-24T00:23:13.358 INFO:journalctl@ceph.mds.j.smithi118.stdout:Mar 24 00:23:13 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-j[72625]: 3: abort() 2024-03-24T00:23:13.358 INFO:journalctl@ceph.mds.j.smithi118.stdout:Mar 24 00:23:13 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-j[72625]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x18f) [0x7fa134c574fb] 2024-03-24T00:23:13.358 INFO:journalctl@ceph.mds.j.smithi118.stdout:Mar 24 00:23:13 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-j[72625]: 5: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7fa134c57667] 2024-03-24T00:23:13.358 INFO:journalctl@ceph.mds.j.smithi118.stdout:Mar 24 00:23:13 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-j[72625]: 6: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x55a5db5e1223] 2024-03-24T00:23:13.358 INFO:journalctl@ceph.mds.j.smithi118.stdout:Mar 24 00:23:13 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-j[72625]: 7: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x55a5db45b5d6] 2024-03-24T00:23:13.359 INFO:journalctl@ceph.mds.j.smithi118.stdout:Mar 24 00:23:13 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-j[72625]: 8: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x55a5db45bda1] 2024-03-24T00:23:13.359 INFO:journalctl@ceph.mds.j.smithi118.stdout:Mar 24 00:23:13 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-j[72625]: 9: (Server::handle_client_rename(boost::intrusive_ptr const&)+0x259d) [0x55a5db49897d] 2024-03-24T00:23:13.359 INFO:journalctl@ceph.mds.j.smithi118.stdout:Mar 24 00:23:13 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-j[72625]: 10: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x55a5db4ae233] 2024-03-24T00:23:13.359 INFO:journalctl@ceph.mds.j.smithi118.stdout:Mar 24 00:23:13 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-j[72625]: 11: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x55a5db4b0197] 2024-03-24T00:23:13.359 INFO:journalctl@ceph.mds.j.smithi118.stdout:Mar 24 00:23:13 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-j[72625]: 12: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x55a5db3af11b] 2024-03-24T00:23:13.359 INFO:journalctl@ceph.mds.j.smithi118.stdout:Mar 24 00:23:13 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-j[72625]: 13: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x55a5db3af6bc] 2024-03-24T00:23:13.359 INFO:journalctl@ceph.mds.j.smithi118.stdout:Mar 24 00:23:13 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-j[72625]: 14: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x55a5db3991ff] 2024-03-24T00:23:13.359 INFO:journalctl@ceph.mds.j.smithi118.stdout:Mar 24 00:23:13 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-j[72625]: 15: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7fa134ee9488] 2024-03-24T00:23:13.359 INFO:journalctl@ceph.mds.j.smithi118.stdout:Mar 24 00:23:13 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-j[72625]: 16: (DispatchQueue::entry()+0x51f) [0x7fa134ee660f] 2024-03-24T00:23:13.359 INFO:journalctl@ceph.mds.j.smithi118.stdout:Mar 24 00:23:13 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-j[72625]: 17: (DispatchQueue::DispatchThread::entry()+0x11) [0x7fa134fb9e81] 2024-03-24T00:23:13.359 INFO:journalctl@ceph.mds.j.smithi118.stdout:Mar 24 00:23:13 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-j[72625]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7fa1339d41ca] 2024-03-24T00:23:13.359 INFO:journalctl@ceph.mds.j.smithi118.stdout:Mar 24 00:23:13 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-j[72625]: 19: clone() 2024-03-24T00:23:13.359 INFO:journalctl@ceph.mds.j.smithi118.stdout:Mar 24 00:23:13 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-j[72625]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-03-24T00:23:13.359 INFO:journalctl@ceph.mds.j.smithi118.stdout:Mar 24 00:23:13 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-j[72625]: 2024-03-24T00:23:13.360 INFO:journalctl@ceph.mds.j.smithi118.stdout:Mar 24 00:23:13 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-j[72625]: -1> 2024-03-24T00:23:13.056+0000 7fa129400700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: In function 'void Locker::set_xlocks_done(MutationImpl*, bool)' thread 7fa129400700 time 2024-03-24T00:23:13.056234+0000 2024-03-24T00:23:13.360 INFO:journalctl@ceph.mds.j.smithi118.stdout:Mar 24 00:23:13 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-j[72625]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: 626: FAILED ceph_assert(obj->is_auth()) 2024-03-24T00:23:13.360 INFO:journalctl@ceph.mds.j.smithi118.stdout:Mar 24 00:23:13 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-j[72625]: 2024-03-24T00:23:13.360 INFO:journalctl@ceph.mds.j.smithi118.stdout:Mar 24 00:23:13 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-j[72625]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-24T00:23:13.360 INFO:journalctl@ceph.mds.j.smithi118.stdout:Mar 24 00:23:13 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-j[72625]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x135) [0x7fa134c574a1] 2024-03-24T00:23:13.360 INFO:journalctl@ceph.mds.j.smithi118.stdout:Mar 24 00:23:13 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-j[72625]: 2: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7fa134c57667] 2024-03-24T00:23:13.360 INFO:journalctl@ceph.mds.j.smithi118.stdout:Mar 24 00:23:13 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-j[72625]: 3: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x55a5db5e1223] 2024-03-24T00:23:13.360 INFO:journalctl@ceph.mds.j.smithi118.stdout:Mar 24 00:23:13 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-j[72625]: 4: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x55a5db45b5d6] 2024-03-24T00:23:13.360 INFO:journalctl@ceph.mds.j.smithi118.stdout:Mar 24 00:23:13 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-j[72625]: 5: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x55a5db45bda1] 2024-03-24T00:23:13.360 INFO:journalctl@ceph.mds.j.smithi118.stdout:Mar 24 00:23:13 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-j[72625]: 6: (Server::handle_client_rename(boost::intrusive_ptr const&)+0x259d) [0x55a5db49897d] 2024-03-24T00:23:13.360 INFO:journalctl@ceph.mds.j.smithi118.stdout:Mar 24 00:23:13 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-j[72625]: 7: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x55a5db4ae233] 2024-03-24T00:23:13.360 INFO:journalctl@ceph.mds.j.smithi118.stdout:Mar 24 00:23:13 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-j[72625]: 8: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x55a5db4b0197] 2024-03-24T00:23:13.360 INFO:journalctl@ceph.mds.j.smithi118.stdout:Mar 24 00:23:13 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-j[72625]: 9: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x55a5db3af11b] 2024-03-24T00:23:13.360 INFO:journalctl@ceph.mds.j.smithi118.stdout:Mar 24 00:23:13 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-j[72625]: 10: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x55a5db3af6bc] 2024-03-24T00:23:13.360 INFO:journalctl@ceph.mds.j.smithi118.stdout:Mar 24 00:23:13 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-j[72625]: 11: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x55a5db3991ff] 2024-03-24T00:23:13.361 INFO:journalctl@ceph.mds.j.smithi118.stdout:Mar 24 00:23:13 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-j[72625]: 12: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7fa134ee9488] 2024-03-24T00:23:13.361 INFO:journalctl@ceph.mds.j.smithi118.stdout:Mar 24 00:23:13 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-j[72625]: 13: (DispatchQueue::entry()+0x51f) [0x7fa134ee660f] 2024-03-24T00:23:13.361 INFO:journalctl@ceph.mds.j.smithi118.stdout:Mar 24 00:23:13 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-j[72625]: 14: (DispatchQueue::DispatchThread::entry()+0x11) [0x7fa134fb9e81] 2024-03-24T00:23:13.361 INFO:journalctl@ceph.mds.j.smithi118.stdout:Mar 24 00:23:13 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-j[72625]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7fa1339d41ca] 2024-03-24T00:23:13.361 INFO:journalctl@ceph.mds.j.smithi118.stdout:Mar 24 00:23:13 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-j[72625]: 16: clone() 2024-03-24T00:23:13.361 INFO:journalctl@ceph.mds.j.smithi118.stdout:Mar 24 00:23:13 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-j[72625]: 2024-03-24T00:23:13.361 INFO:journalctl@ceph.mds.j.smithi118.stdout:Mar 24 00:23:13 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-j[72625]: 0> 2024-03-24T00:23:13.058+0000 7fa129400700 -1 *** Caught signal (Aborted) ** 2024-03-24T00:23:13.361 INFO:journalctl@ceph.mds.j.smithi118.stdout:Mar 24 00:23:13 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-j[72625]: in thread 7fa129400700 thread_name:ms_dispatch 2024-03-24T00:23:13.361 INFO:journalctl@ceph.mds.j.smithi118.stdout:Mar 24 00:23:13 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-j[72625]: 2024-03-24T00:23:13.361 INFO:journalctl@ceph.mds.j.smithi118.stdout:Mar 24 00:23:13 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-j[72625]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-24T00:23:13.361 INFO:journalctl@ceph.mds.j.smithi118.stdout:Mar 24 00:23:13 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-j[72625]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7fa1339ded20] 2024-03-24T00:23:13.361 INFO:journalctl@ceph.mds.j.smithi118.stdout:Mar 24 00:23:13 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-j[72625]: 2: gsignal() 2024-03-24T00:23:13.361 INFO:journalctl@ceph.mds.j.smithi118.stdout:Mar 24 00:23:13 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-j[72625]: 3: abort() 2024-03-24T00:23:13.361 INFO:journalctl@ceph.mds.j.smithi118.stdout:Mar 24 00:23:13 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-j[72625]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x18f) [0x7fa134c574fb] 2024-03-24T00:23:13.362 INFO:journalctl@ceph.mds.j.smithi118.stdout:Mar 24 00:23:13 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-j[72625]: 5: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7fa134c57667] 2024-03-24T00:23:13.362 INFO:journalctl@ceph.mds.j.smithi118.stdout:Mar 24 00:23:13 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-j[72625]: 6: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x55a5db5e1223] 2024-03-24T00:23:13.362 INFO:journalctl@ceph.mds.j.smithi118.stdout:Mar 24 00:23:13 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-j[72625]: 7: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x55a5db45b5d6] 2024-03-24T00:23:13.362 INFO:journalctl@ceph.mds.j.smithi118.stdout:Mar 24 00:23:13 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-j[72625]: 8: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x55a5db45bda1] 2024-03-24T00:23:13.362 INFO:journalctl@ceph.mds.j.smithi118.stdout:Mar 24 00:23:13 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-j[72625]: 9: (Server::handle_client_rename(boost::intrusive_ptr const&)+0x259d) [0x55a5db49897d] 2024-03-24T00:23:13.362 INFO:journalctl@ceph.mds.j.smithi118.stdout:Mar 24 00:23:13 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-j[72625]: 10: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x55a5db4ae233] 2024-03-24T00:23:13.362 INFO:journalctl@ceph.mds.j.smithi118.stdout:Mar 24 00:23:13 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-j[72625]: 11: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x55a5db4b0197] 2024-03-24T00:23:13.362 INFO:journalctl@ceph.mds.j.smithi118.stdout:Mar 24 00:23:13 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-j[72625]: 12: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x55a5db3af11b] 2024-03-24T00:23:13.362 INFO:journalctl@ceph.mds.j.smithi118.stdout:Mar 24 00:23:13 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-j[72625]: 13: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x55a5db3af6bc] 2024-03-24T00:23:13.362 INFO:journalctl@ceph.mds.j.smithi118.stdout:Mar 24 00:23:13 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-j[72625]: 14: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x55a5db3991ff] 2024-03-24T00:23:13.362 INFO:journalctl@ceph.mds.j.smithi118.stdout:Mar 24 00:23:13 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-j[72625]: 15: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7fa134ee9488] 2024-03-24T00:23:13.362 INFO:journalctl@ceph.mds.j.smithi118.stdout:Mar 24 00:23:13 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-j[72625]: 16: (DispatchQueue::entry()+0x51f) [0x7fa134ee660f] 2024-03-24T00:23:13.362 INFO:journalctl@ceph.mds.j.smithi118.stdout:Mar 24 00:23:13 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-j[72625]: 17: (DispatchQueue::DispatchThread::entry()+0x11) [0x7fa134fb9e81] 2024-03-24T00:23:13.363 INFO:journalctl@ceph.mds.j.smithi118.stdout:Mar 24 00:23:13 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-j[72625]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7fa1339d41ca] 2024-03-24T00:23:13.363 INFO:journalctl@ceph.mds.j.smithi118.stdout:Mar 24 00:23:13 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-j[72625]: 19: clone() 2024-03-24T00:23:13.363 INFO:journalctl@ceph.mds.j.smithi118.stdout:Mar 24 00:23:13 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-j[72625]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-03-24T00:23:13.363 INFO:journalctl@ceph.mds.j.smithi118.stdout:Mar 24 00:23:13 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-j[72625]: 2024-03-24T00:23:13.363 INFO:journalctl@ceph.mds.j.smithi118.stdout:Mar 24 00:23:13 smithi118 podman[95505]: 2024-03-24 00:23:13.180601104 +0000 UTC m=+0.040073306 container died 4b581125b505cb60f3ec5e80ce75c7ab756f93ffd0eadd1cd88571d4c2d74cc1 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-j, org.label-schema.schema-version=1.0, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.build-date=20240319, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS) 2024-03-24T00:23:13.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:23:12 smithi189 ceph-mon[36187]: mds.? [v2:172.21.15.189:6836/635914573,v1:172.21.15.189:6837/635914573] up:rejoin 2024-03-24T00:23:13.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:23:12 smithi189 ceph-mon[36187]: mds.? [v2:172.21.15.118:6834/3194741373,v1:172.21.15.118:6835/3194741373] up:rejoin 2024-03-24T00:23:13.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:23:12 smithi189 ceph-mon[36187]: fsmap cephfs:5/5 3 up:standby-replay 2 up:standby 2 up:rejoin 3 up:active 2024-03-24T00:23:13.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:23:12 smithi189 ceph-mon[36187]: daemon mds.j is now active in filesystem cephfs as rank 3 2024-03-24T00:23:13.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:23:12 smithi189 ceph-mon[36187]: daemon mds.h is now active in filesystem cephfs as rank 1 2024-03-24T00:23:13.455 INFO:journalctl@ceph.mds.h.smithi189.stdout:Mar 24 00:23:13 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-h[70242]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: In function 'void Locker::set_xlocks_done(MutationImpl*, bool)' thread 7ff34e917700 time 2024-03-24T00:23:13.096378+0000 2024-03-24T00:23:13.455 INFO:journalctl@ceph.mds.h.smithi189.stdout:Mar 24 00:23:13 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-h[70242]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: 626: FAILED ceph_assert(obj->is_auth()) 2024-03-24T00:23:13.455 INFO:journalctl@ceph.mds.h.smithi189.stdout:Mar 24 00:23:13 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-h[70242]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-24T00:23:13.455 INFO:journalctl@ceph.mds.h.smithi189.stdout:Mar 24 00:23:13 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-h[70242]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x135) [0x7ff358f414a1] 2024-03-24T00:23:13.455 INFO:journalctl@ceph.mds.h.smithi189.stdout:Mar 24 00:23:13 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-h[70242]: 2: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7ff358f41667] 2024-03-24T00:23:13.455 INFO:journalctl@ceph.mds.h.smithi189.stdout:Mar 24 00:23:13 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-h[70242]: 3: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x556f5a914223] 2024-03-24T00:23:13.455 INFO:journalctl@ceph.mds.h.smithi189.stdout:Mar 24 00:23:13 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-h[70242]: 4: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x556f5a78e5d6] 2024-03-24T00:23:13.455 INFO:journalctl@ceph.mds.h.smithi189.stdout:Mar 24 00:23:13 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-h[70242]: 5: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x556f5a78eda1] 2024-03-24T00:23:13.455 INFO:journalctl@ceph.mds.h.smithi189.stdout:Mar 24 00:23:13 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-h[70242]: 6: (Server::handle_client_rename(boost::intrusive_ptr const&)+0x259d) [0x556f5a7cb97d] 2024-03-24T00:23:13.456 INFO:journalctl@ceph.mds.h.smithi189.stdout:Mar 24 00:23:13 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-h[70242]: 7: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x556f5a7e1233] 2024-03-24T00:23:13.456 INFO:journalctl@ceph.mds.h.smithi189.stdout:Mar 24 00:23:13 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-h[70242]: 8: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x556f5a7e3197] 2024-03-24T00:23:13.456 INFO:journalctl@ceph.mds.h.smithi189.stdout:Mar 24 00:23:13 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-h[70242]: 9: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x556f5a6e211b] 2024-03-24T00:23:13.456 INFO:journalctl@ceph.mds.h.smithi189.stdout:Mar 24 00:23:13 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-h[70242]: 10: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x556f5a6e26bc] 2024-03-24T00:23:13.456 INFO:journalctl@ceph.mds.h.smithi189.stdout:Mar 24 00:23:13 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-h[70242]: 11: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x556f5a6cc1ff] 2024-03-24T00:23:13.456 INFO:journalctl@ceph.mds.h.smithi189.stdout:Mar 24 00:23:13 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-h[70242]: 12: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7ff3591d3488] 2024-03-24T00:23:13.456 INFO:journalctl@ceph.mds.h.smithi189.stdout:Mar 24 00:23:13 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-h[70242]: 13: (DispatchQueue::entry()+0x51f) [0x7ff3591d060f] 2024-03-24T00:23:13.456 INFO:journalctl@ceph.mds.h.smithi189.stdout:Mar 24 00:23:13 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-h[70242]: 14: (DispatchQueue::DispatchThread::entry()+0x11) [0x7ff3592a3e81] 2024-03-24T00:23:13.456 INFO:journalctl@ceph.mds.h.smithi189.stdout:Mar 24 00:23:13 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-h[70242]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7ff357cbe1ca] 2024-03-24T00:23:13.456 INFO:journalctl@ceph.mds.h.smithi189.stdout:Mar 24 00:23:13 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-h[70242]: 16: clone() 2024-03-24T00:23:13.456 INFO:journalctl@ceph.mds.h.smithi189.stdout:Mar 24 00:23:13 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-h[70242]: 2024-03-24T00:23:13.096+0000 7ff34e917700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: In function 'void Locker::set_xlocks_done(MutationImpl*, bool)' thread 7ff34e917700 time 2024-03-24T00:23:13.096378+0000 2024-03-24T00:23:13.456 INFO:journalctl@ceph.mds.h.smithi189.stdout:Mar 24 00:23:13 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-h[70242]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: 626: FAILED ceph_assert(obj->is_auth()) 2024-03-24T00:23:13.456 INFO:journalctl@ceph.mds.h.smithi189.stdout:Mar 24 00:23:13 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-h[70242]: 2024-03-24T00:23:13.456 INFO:journalctl@ceph.mds.h.smithi189.stdout:Mar 24 00:23:13 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-h[70242]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-24T00:23:13.457 INFO:journalctl@ceph.mds.h.smithi189.stdout:Mar 24 00:23:13 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-h[70242]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x135) [0x7ff358f414a1] 2024-03-24T00:23:13.457 INFO:journalctl@ceph.mds.h.smithi189.stdout:Mar 24 00:23:13 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-h[70242]: 2: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7ff358f41667] 2024-03-24T00:23:13.457 INFO:journalctl@ceph.mds.h.smithi189.stdout:Mar 24 00:23:13 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-h[70242]: 3: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x556f5a914223] 2024-03-24T00:23:13.457 INFO:journalctl@ceph.mds.h.smithi189.stdout:Mar 24 00:23:13 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-h[70242]: 4: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x556f5a78e5d6] 2024-03-24T00:23:13.457 INFO:journalctl@ceph.mds.h.smithi189.stdout:Mar 24 00:23:13 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-h[70242]: 5: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x556f5a78eda1] 2024-03-24T00:23:13.457 INFO:journalctl@ceph.mds.h.smithi189.stdout:Mar 24 00:23:13 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-h[70242]: 6: (Server::handle_client_rename(boost::intrusive_ptr const&)+0x259d) [0x556f5a7cb97d] 2024-03-24T00:23:13.457 INFO:journalctl@ceph.mds.h.smithi189.stdout:Mar 24 00:23:13 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-h[70242]: 7: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x556f5a7e1233] 2024-03-24T00:23:13.457 INFO:journalctl@ceph.mds.h.smithi189.stdout:Mar 24 00:23:13 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-h[70242]: 8: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x556f5a7e3197] 2024-03-24T00:23:13.457 INFO:journalctl@ceph.mds.h.smithi189.stdout:Mar 24 00:23:13 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-h[70242]: 9: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x556f5a6e211b] 2024-03-24T00:23:13.457 INFO:journalctl@ceph.mds.h.smithi189.stdout:Mar 24 00:23:13 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-h[70242]: 10: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x556f5a6e26bc] 2024-03-24T00:23:13.457 INFO:journalctl@ceph.mds.h.smithi189.stdout:Mar 24 00:23:13 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-h[70242]: 11: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x556f5a6cc1ff] 2024-03-24T00:23:13.457 INFO:journalctl@ceph.mds.h.smithi189.stdout:Mar 24 00:23:13 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-h[70242]: 12: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7ff3591d3488] 2024-03-24T00:23:13.457 INFO:journalctl@ceph.mds.h.smithi189.stdout:Mar 24 00:23:13 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-h[70242]: 13: (DispatchQueue::entry()+0x51f) [0x7ff3591d060f] 2024-03-24T00:23:13.457 INFO:journalctl@ceph.mds.h.smithi189.stdout:Mar 24 00:23:13 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-h[70242]: 14: (DispatchQueue::DispatchThread::entry()+0x11) [0x7ff3592a3e81] 2024-03-24T00:23:13.458 INFO:journalctl@ceph.mds.h.smithi189.stdout:Mar 24 00:23:13 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-h[70242]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7ff357cbe1ca] 2024-03-24T00:23:13.458 INFO:journalctl@ceph.mds.h.smithi189.stdout:Mar 24 00:23:13 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-h[70242]: 16: clone() 2024-03-24T00:23:13.458 INFO:journalctl@ceph.mds.h.smithi189.stdout:Mar 24 00:23:13 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-h[70242]: 2024-03-24T00:23:13.458 INFO:journalctl@ceph.mds.h.smithi189.stdout:Mar 24 00:23:13 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-h[70242]: *** Caught signal (Aborted) ** 2024-03-24T00:23:13.458 INFO:journalctl@ceph.mds.h.smithi189.stdout:Mar 24 00:23:13 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-h[70242]: in thread 7ff34e917700 thread_name:ms_dispatch 2024-03-24T00:23:13.458 INFO:journalctl@ceph.mds.h.smithi189.stdout:Mar 24 00:23:13 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-h[70242]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-24T00:23:13.458 INFO:journalctl@ceph.mds.h.smithi189.stdout:Mar 24 00:23:13 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-h[70242]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7ff357cc8d20] 2024-03-24T00:23:13.458 INFO:journalctl@ceph.mds.h.smithi189.stdout:Mar 24 00:23:13 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-h[70242]: 2: gsignal() 2024-03-24T00:23:13.458 INFO:journalctl@ceph.mds.h.smithi189.stdout:Mar 24 00:23:13 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-h[70242]: 3: abort() 2024-03-24T00:23:13.458 INFO:journalctl@ceph.mds.h.smithi189.stdout:Mar 24 00:23:13 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-h[70242]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x18f) [0x7ff358f414fb] 2024-03-24T00:23:13.458 INFO:journalctl@ceph.mds.h.smithi189.stdout:Mar 24 00:23:13 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-h[70242]: 5: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7ff358f41667] 2024-03-24T00:23:13.458 INFO:journalctl@ceph.mds.h.smithi189.stdout:Mar 24 00:23:13 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-h[70242]: 6: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x556f5a914223] 2024-03-24T00:23:13.458 INFO:journalctl@ceph.mds.h.smithi189.stdout:Mar 24 00:23:13 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-h[70242]: 7: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x556f5a78e5d6] 2024-03-24T00:23:13.459 INFO:journalctl@ceph.mds.h.smithi189.stdout:Mar 24 00:23:13 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-h[70242]: 8: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x556f5a78eda1] 2024-03-24T00:23:13.459 INFO:journalctl@ceph.mds.h.smithi189.stdout:Mar 24 00:23:13 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-h[70242]: 9: (Server::handle_client_rename(boost::intrusive_ptr const&)+0x259d) [0x556f5a7cb97d] 2024-03-24T00:23:13.459 INFO:journalctl@ceph.mds.h.smithi189.stdout:Mar 24 00:23:13 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-h[70242]: 10: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x556f5a7e1233] 2024-03-24T00:23:13.459 INFO:journalctl@ceph.mds.h.smithi189.stdout:Mar 24 00:23:13 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-h[70242]: 11: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x556f5a7e3197] 2024-03-24T00:23:13.459 INFO:journalctl@ceph.mds.h.smithi189.stdout:Mar 24 00:23:13 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-h[70242]: 12: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x556f5a6e211b] 2024-03-24T00:23:13.459 INFO:journalctl@ceph.mds.h.smithi189.stdout:Mar 24 00:23:13 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-h[70242]: 13: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x556f5a6e26bc] 2024-03-24T00:23:13.459 INFO:journalctl@ceph.mds.h.smithi189.stdout:Mar 24 00:23:13 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-h[70242]: 14: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x556f5a6cc1ff] 2024-03-24T00:23:13.459 INFO:journalctl@ceph.mds.h.smithi189.stdout:Mar 24 00:23:13 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-h[70242]: 15: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7ff3591d3488] 2024-03-24T00:23:13.459 INFO:journalctl@ceph.mds.h.smithi189.stdout:Mar 24 00:23:13 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-h[70242]: 16: (DispatchQueue::entry()+0x51f) [0x7ff3591d060f] 2024-03-24T00:23:13.459 INFO:journalctl@ceph.mds.h.smithi189.stdout:Mar 24 00:23:13 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-h[70242]: 17: (DispatchQueue::DispatchThread::entry()+0x11) [0x7ff3592a3e81] 2024-03-24T00:23:13.459 INFO:journalctl@ceph.mds.h.smithi189.stdout:Mar 24 00:23:13 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-h[70242]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7ff357cbe1ca] 2024-03-24T00:23:13.459 INFO:journalctl@ceph.mds.h.smithi189.stdout:Mar 24 00:23:13 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-h[70242]: 19: clone() 2024-03-24T00:23:13.459 INFO:journalctl@ceph.mds.h.smithi189.stdout:Mar 24 00:23:13 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-h[70242]: 2024-03-24T00:23:13.098+0000 7ff34e917700 -1 *** Caught signal (Aborted) ** 2024-03-24T00:23:13.459 INFO:journalctl@ceph.mds.h.smithi189.stdout:Mar 24 00:23:13 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-h[70242]: in thread 7ff34e917700 thread_name:ms_dispatch 2024-03-24T00:23:13.459 INFO:journalctl@ceph.mds.h.smithi189.stdout:Mar 24 00:23:13 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-h[70242]: 2024-03-24T00:23:13.460 INFO:journalctl@ceph.mds.h.smithi189.stdout:Mar 24 00:23:13 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-h[70242]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-24T00:23:13.460 INFO:journalctl@ceph.mds.h.smithi189.stdout:Mar 24 00:23:13 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-h[70242]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7ff357cc8d20] 2024-03-24T00:23:13.460 INFO:journalctl@ceph.mds.h.smithi189.stdout:Mar 24 00:23:13 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-h[70242]: 2: gsignal() 2024-03-24T00:23:13.460 INFO:journalctl@ceph.mds.h.smithi189.stdout:Mar 24 00:23:13 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-h[70242]: 3: abort() 2024-03-24T00:23:13.460 INFO:journalctl@ceph.mds.h.smithi189.stdout:Mar 24 00:23:13 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-h[70242]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x18f) [0x7ff358f414fb] 2024-03-24T00:23:13.460 INFO:journalctl@ceph.mds.h.smithi189.stdout:Mar 24 00:23:13 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-h[70242]: 5: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7ff358f41667] 2024-03-24T00:23:13.461 INFO:journalctl@ceph.mds.h.smithi189.stdout:Mar 24 00:23:13 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-h[70242]: 6: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x556f5a914223] 2024-03-24T00:23:13.461 INFO:journalctl@ceph.mds.h.smithi189.stdout:Mar 24 00:23:13 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-h[70242]: 7: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x556f5a78e5d6] 2024-03-24T00:23:13.461 INFO:journalctl@ceph.mds.h.smithi189.stdout:Mar 24 00:23:13 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-h[70242]: 8: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x556f5a78eda1] 2024-03-24T00:23:13.461 INFO:journalctl@ceph.mds.h.smithi189.stdout:Mar 24 00:23:13 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-h[70242]: 9: (Server::handle_client_rename(boost::intrusive_ptr const&)+0x259d) [0x556f5a7cb97d] 2024-03-24T00:23:13.461 INFO:journalctl@ceph.mds.h.smithi189.stdout:Mar 24 00:23:13 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-h[70242]: 10: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x556f5a7e1233] 2024-03-24T00:23:13.461 INFO:journalctl@ceph.mds.h.smithi189.stdout:Mar 24 00:23:13 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-h[70242]: 11: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x556f5a7e3197] 2024-03-24T00:23:13.461 INFO:journalctl@ceph.mds.h.smithi189.stdout:Mar 24 00:23:13 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-h[70242]: 12: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x556f5a6e211b] 2024-03-24T00:23:13.461 INFO:journalctl@ceph.mds.h.smithi189.stdout:Mar 24 00:23:13 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-h[70242]: 13: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x556f5a6e26bc] 2024-03-24T00:23:13.462 INFO:journalctl@ceph.mds.h.smithi189.stdout:Mar 24 00:23:13 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-h[70242]: 14: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x556f5a6cc1ff] 2024-03-24T00:23:13.462 INFO:journalctl@ceph.mds.h.smithi189.stdout:Mar 24 00:23:13 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-h[70242]: 15: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7ff3591d3488] 2024-03-24T00:23:13.462 INFO:journalctl@ceph.mds.h.smithi189.stdout:Mar 24 00:23:13 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-h[70242]: 16: (DispatchQueue::entry()+0x51f) [0x7ff3591d060f] 2024-03-24T00:23:13.462 INFO:journalctl@ceph.mds.h.smithi189.stdout:Mar 24 00:23:13 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-h[70242]: 17: (DispatchQueue::DispatchThread::entry()+0x11) [0x7ff3592a3e81] 2024-03-24T00:23:13.462 INFO:journalctl@ceph.mds.h.smithi189.stdout:Mar 24 00:23:13 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-h[70242]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7ff357cbe1ca] 2024-03-24T00:23:13.462 INFO:journalctl@ceph.mds.h.smithi189.stdout:Mar 24 00:23:13 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-h[70242]: 19: clone() 2024-03-24T00:23:13.462 INFO:journalctl@ceph.mds.h.smithi189.stdout:Mar 24 00:23:13 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-h[70242]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-03-24T00:23:13.462 INFO:journalctl@ceph.mds.h.smithi189.stdout:Mar 24 00:23:13 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-h[70242]: 2024-03-24T00:23:13.462 INFO:journalctl@ceph.mds.h.smithi189.stdout:Mar 24 00:23:13 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-h[70242]: -1> 2024-03-24T00:23:13.096+0000 7ff34e917700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: In function 'void Locker::set_xlocks_done(MutationImpl*, bool)' thread 7ff34e917700 time 2024-03-24T00:23:13.096378+0000 2024-03-24T00:23:13.462 INFO:journalctl@ceph.mds.h.smithi189.stdout:Mar 24 00:23:13 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-h[70242]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: 626: FAILED ceph_assert(obj->is_auth()) 2024-03-24T00:23:13.462 INFO:journalctl@ceph.mds.h.smithi189.stdout:Mar 24 00:23:13 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-h[70242]: 2024-03-24T00:23:13.462 INFO:journalctl@ceph.mds.h.smithi189.stdout:Mar 24 00:23:13 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-h[70242]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-24T00:23:13.462 INFO:journalctl@ceph.mds.h.smithi189.stdout:Mar 24 00:23:13 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-h[70242]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x135) [0x7ff358f414a1] 2024-03-24T00:23:13.462 INFO:journalctl@ceph.mds.h.smithi189.stdout:Mar 24 00:23:13 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-h[70242]: 2: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7ff358f41667] 2024-03-24T00:23:13.463 INFO:journalctl@ceph.mds.h.smithi189.stdout:Mar 24 00:23:13 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-h[70242]: 3: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x556f5a914223] 2024-03-24T00:23:13.463 INFO:journalctl@ceph.mds.h.smithi189.stdout:Mar 24 00:23:13 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-h[70242]: 4: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x556f5a78e5d6] 2024-03-24T00:23:13.463 INFO:journalctl@ceph.mds.h.smithi189.stdout:Mar 24 00:23:13 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-h[70242]: 5: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x556f5a78eda1] 2024-03-24T00:23:13.463 INFO:journalctl@ceph.mds.h.smithi189.stdout:Mar 24 00:23:13 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-h[70242]: 6: (Server::handle_client_rename(boost::intrusive_ptr const&)+0x259d) [0x556f5a7cb97d] 2024-03-24T00:23:13.463 INFO:journalctl@ceph.mds.h.smithi189.stdout:Mar 24 00:23:13 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-h[70242]: 7: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x556f5a7e1233] 2024-03-24T00:23:13.463 INFO:journalctl@ceph.mds.h.smithi189.stdout:Mar 24 00:23:13 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-h[70242]: 8: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x556f5a7e3197] 2024-03-24T00:23:13.463 INFO:journalctl@ceph.mds.h.smithi189.stdout:Mar 24 00:23:13 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-h[70242]: 9: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x556f5a6e211b] 2024-03-24T00:23:13.463 INFO:journalctl@ceph.mds.h.smithi189.stdout:Mar 24 00:23:13 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-h[70242]: 10: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x556f5a6e26bc] 2024-03-24T00:23:13.463 INFO:journalctl@ceph.mds.h.smithi189.stdout:Mar 24 00:23:13 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-h[70242]: 11: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x556f5a6cc1ff] 2024-03-24T00:23:13.463 INFO:journalctl@ceph.mds.h.smithi189.stdout:Mar 24 00:23:13 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-h[70242]: 12: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7ff3591d3488] 2024-03-24T00:23:13.463 INFO:journalctl@ceph.mds.h.smithi189.stdout:Mar 24 00:23:13 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-h[70242]: 13: (DispatchQueue::entry()+0x51f) [0x7ff3591d060f] 2024-03-24T00:23:13.463 INFO:journalctl@ceph.mds.h.smithi189.stdout:Mar 24 00:23:13 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-h[70242]: 14: (DispatchQueue::DispatchThread::entry()+0x11) [0x7ff3592a3e81] 2024-03-24T00:23:13.463 INFO:journalctl@ceph.mds.h.smithi189.stdout:Mar 24 00:23:13 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-h[70242]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7ff357cbe1ca] 2024-03-24T00:23:13.463 INFO:journalctl@ceph.mds.h.smithi189.stdout:Mar 24 00:23:13 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-h[70242]: 16: clone() 2024-03-24T00:23:13.463 INFO:journalctl@ceph.mds.h.smithi189.stdout:Mar 24 00:23:13 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-h[70242]: 2024-03-24T00:23:13.464 INFO:journalctl@ceph.mds.h.smithi189.stdout:Mar 24 00:23:13 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-h[70242]: 0> 2024-03-24T00:23:13.098+0000 7ff34e917700 -1 *** Caught signal (Aborted) ** 2024-03-24T00:23:13.464 INFO:journalctl@ceph.mds.h.smithi189.stdout:Mar 24 00:23:13 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-h[70242]: in thread 7ff34e917700 thread_name:ms_dispatch 2024-03-24T00:23:13.464 INFO:journalctl@ceph.mds.h.smithi189.stdout:Mar 24 00:23:13 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-h[70242]: 2024-03-24T00:23:13.464 INFO:journalctl@ceph.mds.h.smithi189.stdout:Mar 24 00:23:13 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-h[70242]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-24T00:23:13.464 INFO:journalctl@ceph.mds.h.smithi189.stdout:Mar 24 00:23:13 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-h[70242]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7ff357cc8d20] 2024-03-24T00:23:13.464 INFO:journalctl@ceph.mds.h.smithi189.stdout:Mar 24 00:23:13 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-h[70242]: 2: gsignal() 2024-03-24T00:23:13.464 INFO:journalctl@ceph.mds.h.smithi189.stdout:Mar 24 00:23:13 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-h[70242]: 3: abort() 2024-03-24T00:23:13.464 INFO:journalctl@ceph.mds.h.smithi189.stdout:Mar 24 00:23:13 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-h[70242]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x18f) [0x7ff358f414fb] 2024-03-24T00:23:13.464 INFO:journalctl@ceph.mds.h.smithi189.stdout:Mar 24 00:23:13 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-h[70242]: 5: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7ff358f41667] 2024-03-24T00:23:13.464 INFO:journalctl@ceph.mds.h.smithi189.stdout:Mar 24 00:23:13 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-h[70242]: 6: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x556f5a914223] 2024-03-24T00:23:13.464 INFO:journalctl@ceph.mds.h.smithi189.stdout:Mar 24 00:23:13 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-h[70242]: 7: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x556f5a78e5d6] 2024-03-24T00:23:13.464 INFO:journalctl@ceph.mds.h.smithi189.stdout:Mar 24 00:23:13 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-h[70242]: 8: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x556f5a78eda1] 2024-03-24T00:23:13.464 INFO:journalctl@ceph.mds.h.smithi189.stdout:Mar 24 00:23:13 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-h[70242]: 9: (Server::handle_client_rename(boost::intrusive_ptr const&)+0x259d) [0x556f5a7cb97d] 2024-03-24T00:23:13.465 INFO:journalctl@ceph.mds.h.smithi189.stdout:Mar 24 00:23:13 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-h[70242]: 10: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x556f5a7e1233] 2024-03-24T00:23:13.465 INFO:journalctl@ceph.mds.h.smithi189.stdout:Mar 24 00:23:13 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-h[70242]: 11: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x556f5a7e3197] 2024-03-24T00:23:13.465 INFO:journalctl@ceph.mds.h.smithi189.stdout:Mar 24 00:23:13 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-h[70242]: 12: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x556f5a6e211b] 2024-03-24T00:23:13.465 INFO:journalctl@ceph.mds.h.smithi189.stdout:Mar 24 00:23:13 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-h[70242]: 13: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x556f5a6e26bc] 2024-03-24T00:23:13.465 INFO:journalctl@ceph.mds.h.smithi189.stdout:Mar 24 00:23:13 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-h[70242]: 14: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x556f5a6cc1ff] 2024-03-24T00:23:13.465 INFO:journalctl@ceph.mds.h.smithi189.stdout:Mar 24 00:23:13 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-h[70242]: 15: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7ff3591d3488] 2024-03-24T00:23:13.465 INFO:journalctl@ceph.mds.h.smithi189.stdout:Mar 24 00:23:13 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-h[70242]: 16: (DispatchQueue::entry()+0x51f) [0x7ff3591d060f] 2024-03-24T00:23:13.465 INFO:journalctl@ceph.mds.h.smithi189.stdout:Mar 24 00:23:13 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-h[70242]: 17: (DispatchQueue::DispatchThread::entry()+0x11) [0x7ff3592a3e81] 2024-03-24T00:23:13.465 INFO:journalctl@ceph.mds.h.smithi189.stdout:Mar 24 00:23:13 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-h[70242]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7ff357cbe1ca] 2024-03-24T00:23:13.465 INFO:journalctl@ceph.mds.h.smithi189.stdout:Mar 24 00:23:13 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-h[70242]: 19: clone() 2024-03-24T00:23:13.465 INFO:journalctl@ceph.mds.h.smithi189.stdout:Mar 24 00:23:13 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-h[70242]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-03-24T00:23:13.465 INFO:journalctl@ceph.mds.h.smithi189.stdout:Mar 24 00:23:13 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-h[70242]: 2024-03-24T00:23:13.465 INFO:journalctl@ceph.mds.h.smithi189.stdout:Mar 24 00:23:13 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-h[70242]: -5> 2024-03-24T00:23:13.096+0000 7ff34e917700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: In function 'void Locker::set_xlocks_done(MutationImpl*, bool)' thread 7ff34e917700 time 2024-03-24T00:23:13.096378+0000 2024-03-24T00:23:13.465 INFO:journalctl@ceph.mds.h.smithi189.stdout:Mar 24 00:23:13 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-h[70242]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: 626: FAILED ceph_assert(obj->is_auth()) 2024-03-24T00:23:13.466 INFO:journalctl@ceph.mds.h.smithi189.stdout:Mar 24 00:23:13 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-h[70242]: 2024-03-24T00:23:13.466 INFO:journalctl@ceph.mds.h.smithi189.stdout:Mar 24 00:23:13 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-h[70242]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-24T00:23:13.466 INFO:journalctl@ceph.mds.h.smithi189.stdout:Mar 24 00:23:13 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-h[70242]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x135) [0x7ff358f414a1] 2024-03-24T00:23:13.466 INFO:journalctl@ceph.mds.h.smithi189.stdout:Mar 24 00:23:13 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-h[70242]: 2: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7ff358f41667] 2024-03-24T00:23:13.466 INFO:journalctl@ceph.mds.h.smithi189.stdout:Mar 24 00:23:13 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-h[70242]: 3: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x556f5a914223] 2024-03-24T00:23:13.466 INFO:journalctl@ceph.mds.h.smithi189.stdout:Mar 24 00:23:13 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-h[70242]: 4: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x556f5a78e5d6] 2024-03-24T00:23:13.466 INFO:journalctl@ceph.mds.h.smithi189.stdout:Mar 24 00:23:13 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-h[70242]: 5: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x556f5a78eda1] 2024-03-24T00:23:13.466 INFO:journalctl@ceph.mds.h.smithi189.stdout:Mar 24 00:23:13 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-h[70242]: 6: (Server::handle_client_rename(boost::intrusive_ptr const&)+0x259d) [0x556f5a7cb97d] 2024-03-24T00:23:13.466 INFO:journalctl@ceph.mds.h.smithi189.stdout:Mar 24 00:23:13 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-h[70242]: 7: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x556f5a7e1233] 2024-03-24T00:23:13.466 INFO:journalctl@ceph.mds.h.smithi189.stdout:Mar 24 00:23:13 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-h[70242]: 8: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x556f5a7e3197] 2024-03-24T00:23:13.466 INFO:journalctl@ceph.mds.h.smithi189.stdout:Mar 24 00:23:13 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-h[70242]: 9: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x556f5a6e211b] 2024-03-24T00:23:13.466 INFO:journalctl@ceph.mds.h.smithi189.stdout:Mar 24 00:23:13 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-h[70242]: 10: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x556f5a6e26bc] 2024-03-24T00:23:13.466 INFO:journalctl@ceph.mds.h.smithi189.stdout:Mar 24 00:23:13 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-h[70242]: 11: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x556f5a6cc1ff] 2024-03-24T00:23:13.466 INFO:journalctl@ceph.mds.h.smithi189.stdout:Mar 24 00:23:13 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-h[70242]: 12: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7ff3591d3488] 2024-03-24T00:23:13.467 INFO:journalctl@ceph.mds.h.smithi189.stdout:Mar 24 00:23:13 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-h[70242]: 13: (DispatchQueue::entry()+0x51f) [0x7ff3591d060f] 2024-03-24T00:23:13.467 INFO:journalctl@ceph.mds.h.smithi189.stdout:Mar 24 00:23:13 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-h[70242]: 14: (DispatchQueue::DispatchThread::entry()+0x11) [0x7ff3592a3e81] 2024-03-24T00:23:13.467 INFO:journalctl@ceph.mds.h.smithi189.stdout:Mar 24 00:23:13 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-h[70242]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7ff357cbe1ca] 2024-03-24T00:23:13.467 INFO:journalctl@ceph.mds.h.smithi189.stdout:Mar 24 00:23:13 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-h[70242]: 16: clone() 2024-03-24T00:23:13.467 INFO:journalctl@ceph.mds.h.smithi189.stdout:Mar 24 00:23:13 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-h[70242]: 2024-03-24T00:23:13.467 INFO:journalctl@ceph.mds.h.smithi189.stdout:Mar 24 00:23:13 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-h[70242]: -4> 2024-03-24T00:23:13.098+0000 7ff34e917700 -1 *** Caught signal (Aborted) ** 2024-03-24T00:23:13.467 INFO:journalctl@ceph.mds.h.smithi189.stdout:Mar 24 00:23:13 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-h[70242]: in thread 7ff34e917700 thread_name:ms_dispatch 2024-03-24T00:23:13.467 INFO:journalctl@ceph.mds.h.smithi189.stdout:Mar 24 00:23:13 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-h[70242]: 2024-03-24T00:23:13.467 INFO:journalctl@ceph.mds.h.smithi189.stdout:Mar 24 00:23:13 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-h[70242]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-24T00:23:13.467 INFO:journalctl@ceph.mds.h.smithi189.stdout:Mar 24 00:23:13 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-h[70242]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7ff357cc8d20] 2024-03-24T00:23:13.467 INFO:journalctl@ceph.mds.h.smithi189.stdout:Mar 24 00:23:13 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-h[70242]: 2: gsignal() 2024-03-24T00:23:13.467 INFO:journalctl@ceph.mds.h.smithi189.stdout:Mar 24 00:23:13 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-h[70242]: 3: abort() 2024-03-24T00:23:13.467 INFO:journalctl@ceph.mds.h.smithi189.stdout:Mar 24 00:23:13 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-h[70242]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x18f) [0x7ff358f414fb] 2024-03-24T00:23:13.467 INFO:journalctl@ceph.mds.h.smithi189.stdout:Mar 24 00:23:13 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-h[70242]: 5: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7ff358f41667] 2024-03-24T00:23:13.467 INFO:journalctl@ceph.mds.h.smithi189.stdout:Mar 24 00:23:13 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-h[70242]: 6: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x556f5a914223] 2024-03-24T00:23:13.468 INFO:journalctl@ceph.mds.h.smithi189.stdout:Mar 24 00:23:13 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-h[70242]: 7: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x556f5a78e5d6] 2024-03-24T00:23:13.468 INFO:journalctl@ceph.mds.h.smithi189.stdout:Mar 24 00:23:13 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-h[70242]: 8: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x556f5a78eda1] 2024-03-24T00:23:13.468 INFO:journalctl@ceph.mds.h.smithi189.stdout:Mar 24 00:23:13 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-h[70242]: 9: (Server::handle_client_rename(boost::intrusive_ptr const&)+0x259d) [0x556f5a7cb97d] 2024-03-24T00:23:13.468 INFO:journalctl@ceph.mds.h.smithi189.stdout:Mar 24 00:23:13 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-h[70242]: 10: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x556f5a7e1233] 2024-03-24T00:23:13.468 INFO:journalctl@ceph.mds.h.smithi189.stdout:Mar 24 00:23:13 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-h[70242]: 11: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x556f5a7e3197] 2024-03-24T00:23:13.468 INFO:journalctl@ceph.mds.h.smithi189.stdout:Mar 24 00:23:13 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-h[70242]: 12: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x556f5a6e211b] 2024-03-24T00:23:13.468 INFO:journalctl@ceph.mds.h.smithi189.stdout:Mar 24 00:23:13 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-h[70242]: 13: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x556f5a6e26bc] 2024-03-24T00:23:13.468 INFO:journalctl@ceph.mds.h.smithi189.stdout:Mar 24 00:23:13 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-h[70242]: 14: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x556f5a6cc1ff] 2024-03-24T00:23:13.468 INFO:journalctl@ceph.mds.h.smithi189.stdout:Mar 24 00:23:13 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-h[70242]: 15: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7ff3591d3488] 2024-03-24T00:23:13.468 INFO:journalctl@ceph.mds.h.smithi189.stdout:Mar 24 00:23:13 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-h[70242]: 16: (DispatchQueue::entry()+0x51f) [0x7ff3591d060f] 2024-03-24T00:23:13.468 INFO:journalctl@ceph.mds.h.smithi189.stdout:Mar 24 00:23:13 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-h[70242]: 17: (DispatchQueue::DispatchThread::entry()+0x11) [0x7ff3592a3e81] 2024-03-24T00:23:13.468 INFO:journalctl@ceph.mds.h.smithi189.stdout:Mar 24 00:23:13 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-h[70242]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7ff357cbe1ca] 2024-03-24T00:23:13.468 INFO:journalctl@ceph.mds.h.smithi189.stdout:Mar 24 00:23:13 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-h[70242]: 19: clone() 2024-03-24T00:23:13.468 INFO:journalctl@ceph.mds.h.smithi189.stdout:Mar 24 00:23:13 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-h[70242]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-03-24T00:23:13.468 INFO:journalctl@ceph.mds.h.smithi189.stdout:Mar 24 00:23:13 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-h[70242]: 2024-03-24T00:23:13.469 INFO:journalctl@ceph.mds.h.smithi189.stdout:Mar 24 00:23:13 smithi189 podman[70411]: 2024-03-24 00:23:13.190257558 +0000 UTC m=+0.028012534 container died aba7624bb5f575c9a5afe6e0c2e6e4f0a26e28da6cad1fb654d4b6a007abcb98 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-h, org.label-schema.license=GPLv2, GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.build-date=20240319, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, ceph=True) 2024-03-24T00:23:13.469 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:23:12 smithi193 ceph-mon[36878]: mds.? [v2:172.21.15.189:6836/635914573,v1:172.21.15.189:6837/635914573] up:rejoin 2024-03-24T00:23:13.469 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:23:12 smithi193 ceph-mon[36878]: mds.? [v2:172.21.15.118:6834/3194741373,v1:172.21.15.118:6835/3194741373] up:rejoin 2024-03-24T00:23:13.469 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:23:12 smithi193 ceph-mon[36878]: fsmap cephfs:5/5 3 up:standby-replay 2 up:standby 2 up:rejoin 3 up:active 2024-03-24T00:23:13.469 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:23:12 smithi193 ceph-mon[36878]: daemon mds.j is now active in filesystem cephfs as rank 3 2024-03-24T00:23:13.469 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:23:12 smithi193 ceph-mon[36878]: daemon mds.h is now active in filesystem cephfs as rank 1 2024-03-24T00:23:14.111 INFO:journalctl@ceph.mds.h.smithi189.stdout:Mar 24 00:23:13 smithi189 podman[70411]: 2024-03-24 00:23:13.85699559 +0000 UTC m=+0.694750553 container remove aba7624bb5f575c9a5afe6e0c2e6e4f0a26e28da6cad1fb654d4b6a007abcb98 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-h, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240319, org.label-schema.schema-version=1.0) 2024-03-24T00:23:14.112 INFO:journalctl@ceph.mds.h.smithi189.stdout:Mar 24 00:23:13 smithi189 systemd[1]: ceph-d79a5420-e971-11ee-95cd-87774f69a715@mds.h.service: Main process exited, code=exited, status=134/n/a 2024-03-24T00:23:14.112 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:23:13 smithi189 ceph-mon[36187]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-03-24T00:23:14.112 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:23:13 smithi189 ceph-mon[36187]: Cluster is now healthy 2024-03-24T00:23:14.112 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:23:13 smithi189 ceph-mon[36187]: mds.? [v2:172.21.15.118:6834/3194741373,v1:172.21.15.118:6835/3194741373] up:active 2024-03-24T00:23:14.112 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:23:13 smithi189 ceph-mon[36187]: mds.? [v2:172.21.15.189:6836/635914573,v1:172.21.15.189:6837/635914573] up:active 2024-03-24T00:23:14.112 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:23:13 smithi189 ceph-mon[36187]: fsmap cephfs:5 3 up:standby-replay 2 up:standby 5 up:active 2024-03-24T00:23:14.112 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:23:13 smithi189 ceph-mon[36187]: fsmap cephfs:5 5 up:standby-replay 5 up:active 2024-03-24T00:23:14.112 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:23:13 smithi189 ceph-mon[36187]: pgmap v646: 97 pgs: 97 active+clean; 146 MiB data, 851 MiB used, 1.0 TiB / 1.0 TiB avail; 166 KiB/s rd, 133 KiB/s wr, 67 op/s 2024-03-24T00:23:14.446 INFO:journalctl@ceph.mds.j.smithi118.stdout:Mar 24 00:23:14 smithi118 podman[95505]: 2024-03-24 00:23:14.017569422 +0000 UTC m=+0.877041646 container remove 4b581125b505cb60f3ec5e80ce75c7ab756f93ffd0eadd1cd88571d4c2d74cc1 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-j, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240319, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, CEPH_POINT_RELEASE=, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0) 2024-03-24T00:23:14.446 INFO:journalctl@ceph.mds.j.smithi118.stdout:Mar 24 00:23:14 smithi118 systemd[1]: ceph-d79a5420-e971-11ee-95cd-87774f69a715@mds.j.service: Main process exited, code=exited, status=134/n/a 2024-03-24T00:23:14.447 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:23:13 smithi118 ceph-mon[31535]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-03-24T00:23:14.447 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:23:13 smithi118 ceph-mon[31535]: Cluster is now healthy 2024-03-24T00:23:14.447 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:23:13 smithi118 ceph-mon[31535]: mds.? [v2:172.21.15.118:6834/3194741373,v1:172.21.15.118:6835/3194741373] up:active 2024-03-24T00:23:14.447 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:23:13 smithi118 ceph-mon[31535]: mds.? [v2:172.21.15.189:6836/635914573,v1:172.21.15.189:6837/635914573] up:active 2024-03-24T00:23:14.447 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:23:13 smithi118 ceph-mon[31535]: fsmap cephfs:5 3 up:standby-replay 2 up:standby 5 up:active 2024-03-24T00:23:14.447 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:23:13 smithi118 ceph-mon[31535]: fsmap cephfs:5 5 up:standby-replay 5 up:active 2024-03-24T00:23:14.447 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:23:13 smithi118 ceph-mon[31535]: pgmap v646: 97 pgs: 97 active+clean; 146 MiB data, 851 MiB used, 1.0 TiB / 1.0 TiB avail; 166 KiB/s rd, 133 KiB/s wr, 67 op/s 2024-03-24T00:23:14.453 INFO:journalctl@ceph.mds.h.smithi189.stdout:Mar 24 00:23:14 smithi189 systemd[1]: ceph-d79a5420-e971-11ee-95cd-87774f69a715@mds.h.service: Failed with result 'exit-code'. 2024-03-24T00:23:14.460 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:23:13 smithi193 ceph-mon[36878]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-03-24T00:23:14.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:23:13 smithi193 ceph-mon[36878]: Cluster is now healthy 2024-03-24T00:23:14.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:23:13 smithi193 ceph-mon[36878]: mds.? [v2:172.21.15.118:6834/3194741373,v1:172.21.15.118:6835/3194741373] up:active 2024-03-24T00:23:14.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:23:13 smithi193 ceph-mon[36878]: mds.? [v2:172.21.15.189:6836/635914573,v1:172.21.15.189:6837/635914573] up:active 2024-03-24T00:23:14.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:23:13 smithi193 ceph-mon[36878]: fsmap cephfs:5 3 up:standby-replay 2 up:standby 5 up:active 2024-03-24T00:23:14.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:23:13 smithi193 ceph-mon[36878]: fsmap cephfs:5 5 up:standby-replay 5 up:active 2024-03-24T00:23:14.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:23:13 smithi193 ceph-mon[36878]: pgmap v646: 97 pgs: 97 active+clean; 146 MiB data, 851 MiB used, 1.0 TiB / 1.0 TiB avail; 166 KiB/s rd, 133 KiB/s wr, 67 op/s 2024-03-24T00:23:14.906 INFO:journalctl@ceph.mds.j.smithi118.stdout:Mar 24 00:23:14 smithi118 systemd[1]: ceph-d79a5420-e971-11ee-95cd-87774f69a715@mds.j.service: Failed with result 'exit-code'. 2024-03-24T00:23:14.907 INFO:journalctl@ceph.mds.j.smithi118.stdout:Mar 24 00:23:14 smithi118 systemd[1]: ceph-d79a5420-e971-11ee-95cd-87774f69a715@mds.j.service: Consumed 1.903s CPU time. 2024-03-24T00:23:16.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:23:15 smithi118 ceph-mon[31535]: pgmap v647: 97 pgs: 97 active+clean; 147 MiB data, 852 MiB used, 1.0 TiB / 1.0 TiB avail; 837 KiB/s rd, 167 KiB/s wr, 124 op/s 2024-03-24T00:23:16.203 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:23:15 smithi189 ceph-mon[36187]: pgmap v647: 97 pgs: 97 active+clean; 147 MiB data, 852 MiB used, 1.0 TiB / 1.0 TiB avail; 837 KiB/s rd, 167 KiB/s wr, 124 op/s 2024-03-24T00:23:16.214 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:23:15 smithi193 ceph-mon[36878]: pgmap v647: 97 pgs: 97 active+clean; 147 MiB data, 852 MiB used, 1.0 TiB / 1.0 TiB avail; 837 KiB/s rd, 167 KiB/s wr, 124 op/s 2024-03-24T00:23:18.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:23:17 smithi118 ceph-mon[31535]: pgmap v648: 97 pgs: 97 active+clean; 147 MiB data, 852 MiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 149 KiB/s wr, 153 op/s 2024-03-24T00:23:18.203 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:23:17 smithi189 ceph-mon[36187]: pgmap v648: 97 pgs: 97 active+clean; 147 MiB data, 852 MiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 149 KiB/s wr, 153 op/s 2024-03-24T00:23:18.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:23:17 smithi193 ceph-mon[36878]: pgmap v648: 97 pgs: 97 active+clean; 147 MiB data, 852 MiB used, 1.0 TiB / 1.0 TiB avail; 1.3 MiB/s rd, 149 KiB/s wr, 153 op/s 2024-03-24T00:23:20.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:23:19 smithi118 ceph-mon[31535]: pgmap v649: 97 pgs: 97 active+clean; 147 MiB data, 852 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 139 KiB/s wr, 131 op/s 2024-03-24T00:23:20.203 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:23:19 smithi189 ceph-mon[36187]: pgmap v649: 97 pgs: 97 active+clean; 147 MiB data, 852 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 139 KiB/s wr, 131 op/s 2024-03-24T00:23:20.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:23:19 smithi193 ceph-mon[36878]: pgmap v649: 97 pgs: 97 active+clean; 147 MiB data, 852 MiB used, 1.0 TiB / 1.0 TiB avail; 1.2 MiB/s rd, 139 KiB/s wr, 131 op/s 2024-03-24T00:23:22.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:23:21 smithi118 ceph-mon[31535]: pgmap v650: 97 pgs: 97 active+clean; 147 MiB data, 852 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 126 KiB/s wr, 118 op/s 2024-03-24T00:23:22.203 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:23:21 smithi189 ceph-mon[36187]: pgmap v650: 97 pgs: 97 active+clean; 147 MiB data, 852 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 126 KiB/s wr, 118 op/s 2024-03-24T00:23:22.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:23:21 smithi193 ceph-mon[36878]: pgmap v650: 97 pgs: 97 active+clean; 147 MiB data, 852 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 126 KiB/s wr, 118 op/s 2024-03-24T00:23:24.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:23:23 smithi118 ceph-mon[31535]: pgmap v651: 97 pgs: 97 active+clean; 148 MiB data, 852 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 219 KiB/s wr, 119 op/s 2024-03-24T00:23:24.203 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:23:23 smithi189 ceph-mon[36187]: pgmap v651: 97 pgs: 97 active+clean; 148 MiB data, 852 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 219 KiB/s wr, 119 op/s 2024-03-24T00:23:24.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:23:23 smithi193 ceph-mon[36878]: pgmap v651: 97 pgs: 97 active+clean; 148 MiB data, 852 MiB used, 1.0 TiB / 1.0 TiB avail; 1.1 MiB/s rd, 219 KiB/s wr, 119 op/s 2024-03-24T00:23:24.561 INFO:journalctl@ceph.mds.h.smithi189.stdout:Mar 24 00:23:24 smithi189 systemd[1]: ceph-d79a5420-e971-11ee-95cd-87774f69a715@mds.h.service: Scheduled restart job, restart counter is at 5. 2024-03-24T00:23:24.562 INFO:journalctl@ceph.mds.h.smithi189.stdout:Mar 24 00:23:24 smithi189 systemd[1]: Stopped Ceph mds.h for d79a5420-e971-11ee-95cd-87774f69a715. 2024-03-24T00:23:24.562 INFO:journalctl@ceph.mds.h.smithi189.stdout:Mar 24 00:23:24 smithi189 systemd[1]: Starting Ceph mds.h for d79a5420-e971-11ee-95cd-87774f69a715... 2024-03-24T00:23:24.562 INFO:journalctl@ceph.mds.h.smithi189.stdout:Mar 24 00:23:24 smithi189 podman[70506]: 2024-03-24 00:23:24.33725935 +0000 UTC m=+0.018217048 image pull c664afe56bb01db4f9afa9532c20598e201e664f2e9ba2be50db0d52674a26d2 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc 2024-03-24T00:23:24.562 INFO:journalctl@ceph.mds.h.smithi189.stdout:Mar 24 00:23:24 smithi189 podman[70506]: 2024-03-24 00:23:24.449421068 +0000 UTC m=+0.130378745 container create a5b198163caeb41e0b9bdf8a2d51d4339aa10d06f4b1634a43af1f73d5255ee5 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-h, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, ceph=True, org.label-schema.build-date=20240319, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-24T00:23:24.930 INFO:journalctl@ceph.mds.h.smithi189.stdout:Mar 24 00:23:24 smithi189 podman[70506]: 2024-03-24 00:23:24.67733996 +0000 UTC m=+0.358297640 container init a5b198163caeb41e0b9bdf8a2d51d4339aa10d06f4b1634a43af1f73d5255ee5 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-h, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_CLEAN=True, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240319, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.vendor=CentOS) 2024-03-24T00:23:24.930 INFO:journalctl@ceph.mds.h.smithi189.stdout:Mar 24 00:23:24 smithi189 podman[70506]: 2024-03-24 00:23:24.680041662 +0000 UTC m=+0.360999355 container start a5b198163caeb41e0b9bdf8a2d51d4339aa10d06f4b1634a43af1f73d5255ee5 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-h, org.label-schema.build-date=20240319, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, GIT_CLEAN=True, io.buildah.version=1.33.5) 2024-03-24T00:23:24.930 INFO:journalctl@ceph.mds.h.smithi189.stdout:Mar 24 00:23:24 smithi189 ceph-mds[70525]: -- 172.21.15.189:0/4153531365 <== mon.2 v2:172.21.15.189:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x557c8d5c5a40 con 0x557c8d598c00 2024-03-24T00:23:24.930 INFO:journalctl@ceph.mds.h.smithi189.stdout:Mar 24 00:23:24 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-h[70521]: starting mds.h at 2024-03-24T00:23:24.930 INFO:journalctl@ceph.mds.h.smithi189.stdout:Mar 24 00:23:24 smithi189 bash[70506]: a5b198163caeb41e0b9bdf8a2d51d4339aa10d06f4b1634a43af1f73d5255ee5 2024-03-24T00:23:24.931 INFO:journalctl@ceph.mds.h.smithi189.stdout:Mar 24 00:23:24 smithi189 systemd[1]: Started Ceph mds.h for d79a5420-e971-11ee-95cd-87774f69a715. 2024-03-24T00:23:25.088 INFO:journalctl@ceph.mds.j.smithi118.stdout:Mar 24 00:23:24 smithi118 systemd[1]: ceph-d79a5420-e971-11ee-95cd-87774f69a715@mds.j.service: Scheduled restart job, restart counter is at 1. 2024-03-24T00:23:25.088 INFO:journalctl@ceph.mds.j.smithi118.stdout:Mar 24 00:23:24 smithi118 systemd[1]: Stopped Ceph mds.j for d79a5420-e971-11ee-95cd-87774f69a715. 2024-03-24T00:23:25.088 INFO:journalctl@ceph.mds.j.smithi118.stdout:Mar 24 00:23:24 smithi118 systemd[1]: ceph-d79a5420-e971-11ee-95cd-87774f69a715@mds.j.service: Consumed 1.903s CPU time. 2024-03-24T00:23:25.088 INFO:journalctl@ceph.mds.j.smithi118.stdout:Mar 24 00:23:24 smithi118 systemd[1]: Starting Ceph mds.j for d79a5420-e971-11ee-95cd-87774f69a715... 2024-03-24T00:23:25.089 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:23:24 smithi118 ceph-mon[31535]: daemon mds.h restarted 2024-03-24T00:23:25.089 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:23:24 smithi118 ceph-mon[31535]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-03-24T00:23:25.089 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:23:24 smithi118 ceph-mon[31535]: osdmap e113: 12 total, 12 up, 12 in 2024-03-24T00:23:25.089 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:23:24 smithi118 ceph-mon[31535]: mds.? [v2:172.21.15.189:6836/2642988219,v1:172.21.15.189:6837/2642988219] up:boot 2024-03-24T00:23:25.089 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:23:24 smithi118 ceph-mon[31535]: Standby daemon mds.d assigned to filesystem cephfs as rank 1 2024-03-24T00:23:25.089 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:23:24 smithi118 ceph-mon[31535]: fsmap cephfs:4/5 {0=b=up:active,2=c=up:active,3=j=up:active,4=e=up:active} 5 up:standby-replay 1 up:standby, 1 failed 2024-03-24T00:23:25.089 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:23:24 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-03-24T00:23:25.089 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:23:24 smithi118 ceph-mon[31535]: fsmap cephfs:5/5 4 up:standby-replay 1 up:standby 1 up:replay 4 up:active 2024-03-24T00:23:25.183 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:23:25.181+0000 7fd11fe00640 1 -- 172.21.15.118:0/4112029654 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fd120103720 msgr2=0x7fd120103b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:23:25.183 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:23:25.182+0000 7fd11fe00640 1 --2- 172.21.15.118:0/4112029654 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fd120103720 0x7fd120103b40 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7fd110009900 tx=0x7fd11002bc50 comp rx=0 tx=0).stop 2024-03-24T00:23:25.183 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:23:25.182+0000 7fd11fe00640 1 -- 172.21.15.118:0/4112029654 shutdown_connections 2024-03-24T00:23:25.183 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:23:25.182+0000 7fd11fe00640 1 --2- 172.21.15.118:0/4112029654 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fd1201052d0 0x7fd12010bb60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:23:25.183 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:23:25.182+0000 7fd11fe00640 1 --2- 172.21.15.118:0/4112029654 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd120104910 0x7fd120104d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:23:25.183 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:23:25.182+0000 7fd11fe00640 1 --2- 172.21.15.118:0/4112029654 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fd120103720 0x7fd120103b40 unknown :-1 s=CLOSED pgs=221 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:23:25.183 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:23:25.182+0000 7fd11fe00640 1 -- 172.21.15.118:0/4112029654 >> 172.21.15.118:0/4112029654 conn(0x7fd1200fecb0 msgr2=0x7fd1201010d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:23:25.184 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:23:25.182+0000 7fd11fe00640 1 -- 172.21.15.118:0/4112029654 shutdown_connections 2024-03-24T00:23:25.184 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:23:25.182+0000 7fd11fe00640 1 -- 172.21.15.118:0/4112029654 wait complete. 2024-03-24T00:23:25.186 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:23:25.183+0000 7fd11fe00640 1 Processor -- start 2024-03-24T00:23:25.186 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:23:25.183+0000 7fd11fe00640 1 -- start start 2024-03-24T00:23:25.186 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:23:25.183+0000 7fd11fe00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd120103720 0x7fd120199a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:23:25.186 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:23:25.183+0000 7fd11fe00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fd120104910 0x7fd120199fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:23:25.186 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:23:25.183+0000 7fd11fe00640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fd1201052d0 0x7fd1201a1030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:23:25.186 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:23:25.183+0000 7fd11fe00640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd12010f460 con 0x7fd1201052d0 2024-03-24T00:23:25.186 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:23:25.183+0000 7fd11fe00640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd12010f2e0 con 0x7fd120103720 2024-03-24T00:23:25.186 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:23:25.183+0000 7fd11fe00640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd12010f5e0 con 0x7fd120104910 2024-03-24T00:23:25.186 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:23:25.184+0000 7fd11e000640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fd120104910 0x7fd120199fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:23:25.187 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:23:25.184+0000 7fd11e000640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fd120104910 0x7fd120199fc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.118:56646/0 (socket says 172.21.15.118:56646) 2024-03-24T00:23:25.187 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:23:25.184+0000 7fd11f400640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fd1201052d0 0x7fd1201a1030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:23:25.187 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:23:25.184+0000 7fd11e000640 1 -- 172.21.15.118:0/541426089 learned_addr learned my addr 172.21.15.118:0/541426089 (peer_addr_for_me v2:172.21.15.118:0/0) 2024-03-24T00:23:25.187 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:23:25.184+0000 7fd11e000640 1 -- 172.21.15.118:0/541426089 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd120103720 msgr2=0x7fd120199a80 unknown :-1 s=STATE_CONNECTING_RE l=1).mark_down 2024-03-24T00:23:25.187 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:23:25.184+0000 7fd11e000640 1 --2- 172.21.15.118:0/541426089 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd120103720 0x7fd120199a80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:23:25.187 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:23:25.184+0000 7fd11e000640 1 -- 172.21.15.118:0/541426089 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fd1201052d0 msgr2=0x7fd1201a1030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:23:25.187 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:23:25.184+0000 7fd11e000640 1 --2- 172.21.15.118:0/541426089 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fd1201052d0 0x7fd1201a1030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:23:25.187 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:23:25.184+0000 7fd11e000640 1 -- 172.21.15.118:0/541426089 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd1201a16a0 con 0x7fd120104910 2024-03-24T00:23:25.187 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:23:25.184+0000 7fd11e000640 1 --2- 172.21.15.118:0/541426089 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fd120104910 0x7fd120199fc0 secure :-1 s=READY pgs=449 cs=0 l=1 rev1=1 crypto rx=0x7fd10c00ea60 tx=0x7fd10c00ef20 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:23:25.187 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:23:25.185+0000 7fd117400640 1 -- 172.21.15.118:0/541426089 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd10c00cde0 con 0x7fd120104910 2024-03-24T00:23:25.187 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:23:25.185+0000 7fd117400640 1 -- 172.21.15.118:0/541426089 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fd10c0035f0 con 0x7fd120104910 2024-03-24T00:23:25.187 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:23:25.185+0000 7fd117400640 1 -- 172.21.15.118:0/541426089 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd10c010670 con 0x7fd120104910 2024-03-24T00:23:25.187 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:23:25.185+0000 7fd11fe00640 1 -- 172.21.15.118:0/541426089 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd1201a1990 con 0x7fd120104910 2024-03-24T00:23:25.187 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:23:25.185+0000 7fd11fe00640 1 -- 172.21.15.118:0/541426089 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd1201a1e90 con 0x7fd120104910 2024-03-24T00:23:25.189 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:23:25.188+0000 7fd1285df740 1 Processor -- start 2024-03-24T00:23:25.189 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:23:25.188+0000 7fd1285df740 1 -- start start 2024-03-24T00:23:25.190 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:23:25.188+0000 7fd1285df740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55e14d5ca3f0 0x55e14d5ca810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:23:25.190 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:23:25.188+0000 7fd1285df740 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55e14d5cade0 0x55e14d5cf8e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:23:25.190 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:23:25.188+0000 7fd1285df740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55e14d5cff50 0x55e14d5d23e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:23:25.190 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:23:25.188+0000 7fd1285df740 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e14d58d3b0 con 0x55e14d5cade0 2024-03-24T00:23:25.190 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:23:25.188+0000 7fd1285df740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e14d52f830 con 0x55e14d5cff50 2024-03-24T00:23:25.190 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:23:25.189+0000 7fd1285df740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e14d4d85d0 con 0x55e14d5ca3f0 2024-03-24T00:23:25.190 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:23:25.189+0000 7fd11ea00640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55e14d5cade0 0x55e14d5cf8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:23:25.190 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:23:25.189+0000 7fd11ea00640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55e14d5cade0 0x55e14d5cf8e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.118:3300/0 says I am v2:172.21.15.118:58310/0 (socket says 172.21.15.118:58310) 2024-03-24T00:23:25.191 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:23:25.189+0000 7fd11ea00640 1 -- 172.21.15.118:0/130014628 learned_addr learned my addr 172.21.15.118:0/130014628 (peer_addr_for_me v2:172.21.15.118:0/0) 2024-03-24T00:23:25.191 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:23:25.189+0000 7fd117400640 1 -- 172.21.15.118:0/541426089 <== mon.1 v2:172.21.15.193:3300/0 4 ==== mgrmap(e 20) v1 ==== 137291+0+0 (secure 0 0 0) 0x7fd10c0037a0 con 0x7fd120104910 2024-03-24T00:23:25.191 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:23:25.189+0000 7fd11ea00640 1 -- 172.21.15.118:0/130014628 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55e14d5ca3f0 msgr2=0x55e14d5ca810 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-03-24T00:23:25.191 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:23:25.189+0000 7fd11ea00640 1 --2- 172.21.15.118:0/130014628 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55e14d5ca3f0 0x55e14d5ca810 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:23:25.191 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:23:25.189+0000 7fd11ea00640 1 -- 172.21.15.118:0/130014628 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55e14d5cff50 msgr2=0x55e14d5d23e0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-03-24T00:23:25.191 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:23:25.189+0000 7fd11ea00640 1 --2- 172.21.15.118:0/130014628 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55e14d5cff50 0x55e14d5d23e0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:23:25.191 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:23:25.189+0000 7fd11ea00640 1 -- 172.21.15.118:0/130014628 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55e14d4f02c0 con 0x55e14d5cade0 2024-03-24T00:23:25.192 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:23:25.190+0000 7fd11ea00640 1 --2- 172.21.15.118:0/130014628 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55e14d5cade0 0x55e14d5cf8e0 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7fd11002c7d0 tx=0x7fd11002c6a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1d7dff45800cba7e server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:23:25.193 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:23:25.190+0000 7fd10be00640 1 -- 172.21.15.118:0/130014628 <== mon.0 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd110041070 con 0x55e14d5cade0 2024-03-24T00:23:25.193 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:23:25.190+0000 7fd10be00640 1 -- 172.21.15.118:0/130014628 <== mon.0 v2:172.21.15.118:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fd110009520 con 0x55e14d5cade0 2024-03-24T00:23:25.193 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:23:25.190+0000 7fd10be00640 1 -- 172.21.15.118:0/130014628 <== mon.0 v2:172.21.15.118:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd1100046c0 con 0x55e14d5cade0 2024-03-24T00:23:25.193 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:23:25.191+0000 7fd1285df740 1 -- 172.21.15.118:0/130014628 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55e14d5cade0 msgr2=0x55e14d5cf8e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:23:25.193 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:23:25.191+0000 7fd1285df740 1 --2- 172.21.15.118:0/130014628 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55e14d5cade0 0x55e14d5cf8e0 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7fd11002c7d0 tx=0x7fd11002c6a0 comp rx=0 tx=0).stop 2024-03-24T00:23:25.193 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:23:25.191+0000 7fd117400640 1 --2- 172.21.15.118:0/541426089 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7fd1040a27f0 0x7fd1040a4cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:23:25.193 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:23:25.191+0000 7fd117400640 1 -- 172.21.15.118:0/541426089 <== mon.1 v2:172.21.15.193:3300/0 5 ==== osd_map(113..113 src has 1..113) v4 ==== 10255+0+0 (secure 0 0 0) 0x7fd10c0cf630 con 0x7fd120104910 2024-03-24T00:23:25.193 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:23:25.191+0000 7fd1285df740 1 -- 172.21.15.118:0/130014628 shutdown_connections 2024-03-24T00:23:25.193 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:23:25.191+0000 7fd1285df740 1 --2- 172.21.15.118:0/130014628 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55e14d5cff50 0x55e14d5d23e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:23:25.193 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:23:25.191+0000 7fd1285df740 1 --2- 172.21.15.118:0/130014628 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55e14d5cade0 0x55e14d5cf8e0 unknown :-1 s=CLOSED pgs=222 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:23:25.193 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:23:25.191+0000 7fd1285df740 1 --2- 172.21.15.118:0/130014628 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55e14d5ca3f0 0x55e14d5ca810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:23:25.193 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:23:25.191+0000 7fd1285df740 1 -- 172.21.15.118:0/130014628 >> 172.21.15.118:0/130014628 conn(0x55e14d5c5960 msgr2=0x55e14d5c5d60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:23:25.193 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:23:25.191+0000 7fd1285df740 1 -- 172.21.15.118:0/130014628 shutdown_connections 2024-03-24T00:23:25.194 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:23:25.191+0000 7fd11f400640 1 --2- 172.21.15.118:0/541426089 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7fd1040a27f0 0x7fd1040a4cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:23:25.194 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:23:25.191+0000 7fd1285df740 1 -- 172.21.15.118:0/130014628 wait complete. 2024-03-24T00:23:25.194 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:23:25.192+0000 7fd1285df740 1 -- start start 2024-03-24T00:23:25.194 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:23:25.192+0000 7fd11f400640 1 --2- 172.21.15.118:0/541426089 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7fd1040a27f0 0x7fd1040a4cb0 secure :-1 s=READY pgs=386 cs=0 l=1 rev1=1 crypto rx=0x55e14d5cb640 tx=0x7fd11800d040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:23:25.194 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:23:25.192+0000 7fd1285df740 1 Processor -- start 2024-03-24T00:23:25.197 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:23:25.192+0000 7fd1285df740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55e14d5ca3f0 0x55e14d5d45d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:23:25.197 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:23:25.192+0000 7fd1285df740 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55e14d5cade0 0x55e14d5d4b10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:23:25.197 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:23:25.192+0000 7fd1285df740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55e14d5cff50 0x55e14d5dbae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:23:25.197 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:23:25.192+0000 7fd1285df740 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e14d4eba20 con 0x55e14d5cade0 2024-03-24T00:23:25.197 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:23:25.192+0000 7fd1285df740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e14d52b8d0 con 0x55e14d5cff50 2024-03-24T00:23:25.197 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:23:25.192+0000 7fd1285df740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x55e14d5d2ea0 con 0x55e14d5ca3f0 2024-03-24T00:23:25.197 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:23:25.193+0000 7fd1285df740 10 client.0 _ll_register_callbacks cb 0x55e14d5281c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T00:23:25.197 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:23:25.193+0000 7fd11e000640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55e14d5cade0 0x55e14d5d4b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:23:25.198 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:23:25.193+0000 7fd11ea00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55e14d5ca3f0 0x55e14d5d45d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:23:25.198 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:23:25.193+0000 7fd11ea00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55e14d5ca3f0 0x55e14d5d45d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.118:56650/0 (socket says 172.21.15.118:56650) 2024-03-24T00:23:25.198 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:23:25.193+0000 7fd11ea00640 1 -- 172.21.15.118:0/3543269921 learned_addr learned my addr 172.21.15.118:0/3543269921 (peer_addr_for_me v2:172.21.15.118:0/0) 2024-03-24T00:23:25.198 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:23:25.193+0000 7fd11e000640 1 -- 172.21.15.118:0/3543269921 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55e14d5ca3f0 msgr2=0x55e14d5d45d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:23:25.198 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:23:25.193+0000 7fd11e000640 1 --2- 172.21.15.118:0/3543269921 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55e14d5ca3f0 0x55e14d5d45d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:23:25.198 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:23:25.193+0000 7fd11e000640 1 -- 172.21.15.118:0/3543269921 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55e14d5cff50 msgr2=0x55e14d5dbae0 unknown :-1 s=STATE_CONNECTING_RE l=0).mark_down 2024-03-24T00:23:25.198 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:23:25.193+0000 7fd11e000640 1 --2- 172.21.15.118:0/3543269921 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55e14d5cff50 0x55e14d5dbae0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:23:25.198 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:23:25.193+0000 7fd11e000640 1 -- 172.21.15.118:0/3543269921 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55e14d244210 con 0x55e14d5cade0 2024-03-24T00:23:25.198 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:23:25.193+0000 7fd11e000640 1 --2- 172.21.15.118:0/3543269921 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55e14d5cade0 0x55e14d5d4b10 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7fd10c01f000 tx=0x7fd10c09b0f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4f004fd37ef1a545 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:23:25.198 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:23:25.193+0000 7fd108c00640 10 client.0 ms_handle_connect on v2:172.21.15.118:3300/0 2024-03-24T00:23:25.198 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:23:25.194+0000 7fd108c00640 1 -- 172.21.15.118:0/3543269921 <== mon.0 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd10c09f8e0 con 0x55e14d5cade0 2024-03-24T00:23:25.198 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:23:25.194+0000 7fd108c00640 1 -- 172.21.15.118:0/3543269921 <== mon.0 v2:172.21.15.118:3300/0 2 ==== osd_map(113..113 src has 1..113) v4 ==== 10255+0+0 (secure 0 0 0) 0x7fd10c003b30 con 0x55e14d5cade0 2024-03-24T00:23:25.199 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:23:25.194+0000 7fd1285df740 1 -- 172.21.15.118:0/3543269921 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fd110009520 con 0x55e14d5cade0 2024-03-24T00:23:25.199 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:23:25.194+0000 7fd108c00640 1 -- 172.21.15.118:0/3543269921 <== mon.0 v2:172.21.15.118:3300/0 3 ==== mon_get_version_reply(handle=1 version=138) v2 ==== 24+0+0 (secure 0 0 0) 0x7fd10c0a3f30 con 0x55e14d5cade0 2024-03-24T00:23:25.199 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:23:25.194+0000 7fd1285df740 10 client.16683 fetch_fsmap learned FSMap version 138 2024-03-24T00:23:25.199 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:23:25.194+0000 7fd1285df740 1 -- 172.21.15.118:0/3543269921 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({config=0+,fsmap=138,monmap=4+}) v3 -- 0x55e14d310620 con 0x55e14d5cade0 2024-03-24T00:23:25.199 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:23:25.194+0000 7fd108c00640 1 -- 172.21.15.118:0/3543269921 <== mon.0 v2:172.21.15.118:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fd110009520 con 0x55e14d5cade0 2024-03-24T00:23:25.199 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:23:25.194+0000 7fd108c00640 1 -- 172.21.15.118:0/3543269921 <== mon.0 v2:172.21.15.118:3300/0 5 ==== fsmap(e 138) v1 ==== 6788+0+0 (secure 0 0 0) 0x7fd10c0afb70 con 0x55e14d5cade0 2024-03-24T00:23:25.199 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:23:25.194+0000 7fd1285df740 10 client.16683 fetch_fsmap finished waiting for FSMap version 138 2024-03-24T00:23:25.199 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:23:25.194+0000 7fd1285df740 10 client.16683 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-24T00:23:25.199 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:23:25.194+0000 7fd1285df740 20 client.16683 populate_metadata read hostname 'smithi118' 2024-03-24T00:23:25.199 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:23:25.194+0000 7fd1285df740 1 --2- 172.21.15.118:0/3543269921 >> [v2:172.21.15.189:6838/4074484168,v1:172.21.15.189:6839/4074484168] conn(0x55e14d5e0530 0x55e14d5e2920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:23:25.199 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:23:25.194+0000 7fd1285df740 4 client.16683 mds_command: new command op to 24569 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-24T00:23:25.199 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:23:25.194+0000 7fd1285df740 1 -- 172.21.15.118:0/3543269921 --> [v2:172.21.15.189:6838/4074484168,v1:172.21.15.189:6839/4074484168] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55e14d53cd70 con 0x55e14d5e0530 2024-03-24T00:23:25.199 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:23:25.196+0000 7fd11ea00640 1 --2- 172.21.15.118:0/3543269921 >> [v2:172.21.15.189:6838/4074484168,v1:172.21.15.189:6839/4074484168] conn(0x55e14d5e0530 0x55e14d5e2920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:23:25.201 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:23:25.201+0000 7fd11ea00640 1 --2- 172.21.15.118:0/3543269921 >> [v2:172.21.15.189:6838/4074484168,v1:172.21.15.189:6839/4074484168] conn(0x55e14d5e0530 0x55e14d5e2920 crc :-1 s=READY pgs=319 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=7fea2ed44faa1261 server_cookie=b96a7b4595defdf8 in_seq=0 out_seq=0 2024-03-24T00:23:25.201 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:23:25.201+0000 7fd108c00640 10 client.16683 ms_handle_connect on v2:172.21.15.189:6838/4074484168 2024-03-24T00:23:25.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:23:24 smithi189 ceph-mon[36187]: daemon mds.h restarted 2024-03-24T00:23:25.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:23:24 smithi189 ceph-mon[36187]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-03-24T00:23:25.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:23:24 smithi189 ceph-mon[36187]: osdmap e113: 12 total, 12 up, 12 in 2024-03-24T00:23:25.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:23:24 smithi189 ceph-mon[36187]: mds.? [v2:172.21.15.189:6836/2642988219,v1:172.21.15.189:6837/2642988219] up:boot 2024-03-24T00:23:25.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:23:24 smithi189 ceph-mon[36187]: Standby daemon mds.d assigned to filesystem cephfs as rank 1 2024-03-24T00:23:25.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:23:24 smithi189 ceph-mon[36187]: fsmap cephfs:4/5 {0=b=up:active,2=c=up:active,3=j=up:active,4=e=up:active} 5 up:standby-replay 1 up:standby, 1 failed 2024-03-24T00:23:25.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:23:24 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-03-24T00:23:25.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:23:24 smithi189 ceph-mon[36187]: fsmap cephfs:5/5 4 up:standby-replay 1 up:standby 1 up:replay 4 up:active 2024-03-24T00:23:25.206 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:23:25.203+0000 7fd108c00640 1 -- 172.21.15.118:0/3543269921 <== mds.0 v2:172.21.15.189:6838/4074484168 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x55e14d53cd70 con 0x55e14d5e0530 2024-03-24T00:23:25.206 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:23:25.203+0000 7fd108c00640 10 client.16683 handle_command_reply: tid=0 2024-03-24T00:23:25.206 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:23:25.203+0000 7fd1285df740 1 client.16683 shutdown 2024-03-24T00:23:25.206 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:23:25.204+0000 7fd1285df740 1 -- 172.21.15.118:0/3543269921 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55e14d5cade0 msgr2=0x55e14d5d4b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:23:25.206 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:23:25.204+0000 7fd1285df740 1 --2- 172.21.15.118:0/3543269921 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55e14d5cade0 0x55e14d5d4b10 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7fd10c01f000 tx=0x7fd10c09b0f0 comp rx=0 tx=0).stop 2024-03-24T00:23:25.206 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:23:25.204+0000 7fd1285df740 1 -- 172.21.15.118:0/3543269921 shutdown_connections 2024-03-24T00:23:25.206 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:23:25.204+0000 7fd1285df740 1 --2- 172.21.15.118:0/3543269921 >> [v2:172.21.15.189:6838/4074484168,v1:172.21.15.189:6839/4074484168] conn(0x55e14d5e0530 0x55e14d5e2920 crc :-1 s=READY pgs=319 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:23:25.207 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:23:25.204+0000 7fd1285df740 1 --2- 172.21.15.118:0/3543269921 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55e14d5cff50 0x55e14d5dbae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:23:25.207 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:23:25.204+0000 7fd1285df740 1 --2- 172.21.15.118:0/3543269921 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55e14d5cade0 0x55e14d5d4b10 unknown :-1 s=CLOSED pgs=223 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:23:25.207 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:23:25.204+0000 7fd1285df740 1 --2- 172.21.15.118:0/3543269921 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55e14d5ca3f0 0x55e14d5d45d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:23:25.207 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:23:25.204+0000 7fd108c00640 0 client.16683 ms_handle_reset on v2:172.21.15.189:6838/4074484168 2024-03-24T00:23:25.207 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:23:25.204+0000 7fd1285df740 1 -- 172.21.15.118:0/3543269921 >> 172.21.15.118:0/3543269921 conn(0x55e14d5c5960 msgr2=0x55e14d5c4b70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:23:25.207 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:23:25.204+0000 7fd1285df740 1 -- 172.21.15.118:0/3543269921 shutdown_connections 2024-03-24T00:23:25.207 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:23:25.204+0000 7fd1285df740 1 -- 172.21.15.118:0/3543269921 wait complete. 2024-03-24T00:23:25.207 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:23:25.204+0000 7fd1285df740 20 client.16683 trim_cache size 0 max 16384 2024-03-24T00:23:25.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:23:24 smithi193 ceph-mon[36878]: daemon mds.h restarted 2024-03-24T00:23:25.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:23:24 smithi193 ceph-mon[36878]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-03-24T00:23:25.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:23:24 smithi193 ceph-mon[36878]: osdmap e113: 12 total, 12 up, 12 in 2024-03-24T00:23:25.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:23:24 smithi193 ceph-mon[36878]: mds.? [v2:172.21.15.189:6836/2642988219,v1:172.21.15.189:6837/2642988219] up:boot 2024-03-24T00:23:25.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:23:24 smithi193 ceph-mon[36878]: Standby daemon mds.d assigned to filesystem cephfs as rank 1 2024-03-24T00:23:25.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:23:24 smithi193 ceph-mon[36878]: fsmap cephfs:4/5 {0=b=up:active,2=c=up:active,3=j=up:active,4=e=up:active} 5 up:standby-replay 1 up:standby, 1 failed 2024-03-24T00:23:25.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:23:24 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "h"}]: dispatch 2024-03-24T00:23:25.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:23:24 smithi193 ceph-mon[36878]: fsmap cephfs:5/5 4 up:standby-replay 1 up:standby 1 up:replay 4 up:active 2024-03-24T00:23:25.220 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:23:25.219+0000 7fd1285df740 1 Processor -- start 2024-03-24T00:23:25.220 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:23:25.219+0000 7fd1285df740 1 -- start start 2024-03-24T00:23:25.221 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:23:25.220+0000 7fd1285df740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55e14d5ca3f0 0x55e14d5cc1a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:23:25.222 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:23:25.220+0000 7fd1285df740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55e14d5cade0 0x55e14d5cc6e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:23:25.222 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:23:25.220+0000 7fd1285df740 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55e14d5cff50 0x55e14d5dbaa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:23:25.222 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:23:25.220+0000 7fd1285df740 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd1201aacf0 con 0x55e14d5cff50 2024-03-24T00:23:25.222 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:23:25.220+0000 7fd1285df740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd1201accd0 con 0x55e14d5cade0 2024-03-24T00:23:25.222 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:23:25.220+0000 7fd1285df740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd1201ab350 con 0x55e14d5ca3f0 2024-03-24T00:23:25.222 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:23:25.220+0000 7fd11f400640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55e14d5cff50 0x55e14d5dbaa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:23:25.222 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:23:25.220+0000 7fd11f400640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55e14d5cff50 0x55e14d5dbaa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.118:3300/0 says I am v2:172.21.15.118:58322/0 (socket says 172.21.15.118:58322) 2024-03-24T00:23:25.222 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:23:25.220+0000 7fd11f400640 1 -- 172.21.15.118:0/4181354890 learned_addr learned my addr 172.21.15.118:0/4181354890 (peer_addr_for_me v2:172.21.15.118:0/0) 2024-03-24T00:23:25.222 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:23:25.220+0000 7fd11e000640 1 --2- 172.21.15.118:0/4181354890 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55e14d5cade0 0x55e14d5cc6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:23:25.222 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:23:25.221+0000 7fd11f400640 1 -- 172.21.15.118:0/4181354890 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55e14d5ca3f0 msgr2=0x55e14d5cc1a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:23:25.222 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:23:25.221+0000 7fd11f400640 1 --2- 172.21.15.118:0/4181354890 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55e14d5ca3f0 0x55e14d5cc1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:23:25.223 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:23:25.221+0000 7fd11f400640 1 -- 172.21.15.118:0/4181354890 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55e14d5cade0 msgr2=0x55e14d5cc6e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:23:25.223 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:23:25.221+0000 7fd11f400640 1 --2- 172.21.15.118:0/4181354890 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55e14d5cade0 0x55e14d5cc6e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:23:25.223 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:23:25.221+0000 7fd11f400640 1 -- 172.21.15.118:0/4181354890 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55e14d5e2c90 con 0x55e14d5cff50 2024-03-24T00:23:25.223 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:23:25.221+0000 7fd11e000640 1 --2- 172.21.15.118:0/4181354890 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55e14d5cade0 0x55e14d5cc6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T00:23:25.223 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:23:25.221+0000 7fd11f400640 1 --2- 172.21.15.118:0/4181354890 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55e14d5cff50 0x55e14d5dbaa0 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7fd118007280 tx=0x7fd118011000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=61a888b2f8ac8783 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:23:25.226 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:23:25.221+0000 7fd10b400640 1 -- 172.21.15.118:0/4181354890 <== mon.0 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd118015840 con 0x55e14d5cff50 2024-03-24T00:23:25.226 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:23:25.221+0000 7fd10b400640 1 -- 172.21.15.118:0/4181354890 <== mon.0 v2:172.21.15.118:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fd118025070 con 0x55e14d5cff50 2024-03-24T00:23:25.227 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:23:25.221+0000 7fd10b400640 1 -- 172.21.15.118:0/4181354890 <== mon.0 v2:172.21.15.118:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd118020e30 con 0x55e14d5cff50 2024-03-24T00:23:25.227 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:23:25.222+0000 7fd1285df740 1 -- 172.21.15.118:0/4181354890 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55e14d5cff50 msgr2=0x55e14d5dbaa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:23:25.227 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:23:25.222+0000 7fd1285df740 1 --2- 172.21.15.118:0/4181354890 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55e14d5cff50 0x55e14d5dbaa0 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7fd118007280 tx=0x7fd118011000 comp rx=0 tx=0).stop 2024-03-24T00:23:25.227 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:23:25.222+0000 7fd1285df740 1 -- 172.21.15.118:0/4181354890 shutdown_connections 2024-03-24T00:23:25.227 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:23:25.222+0000 7fd1285df740 1 --2- 172.21.15.118:0/4181354890 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55e14d5cff50 0x55e14d5dbaa0 unknown :-1 s=CLOSED pgs=224 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:23:25.227 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:23:25.222+0000 7fd1285df740 1 --2- 172.21.15.118:0/4181354890 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55e14d5cade0 0x55e14d5cc6e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:23:25.227 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:23:25.222+0000 7fd1285df740 1 --2- 172.21.15.118:0/4181354890 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55e14d5ca3f0 0x55e14d5cc1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:23:25.227 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:23:25.222+0000 7fd1285df740 1 -- 172.21.15.118:0/4181354890 >> 172.21.15.118:0/4181354890 conn(0x55e14d5c5960 msgr2=0x55e14d5c5d40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:23:25.227 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:23:25.222+0000 7fd1285df740 1 -- 172.21.15.118:0/4181354890 shutdown_connections 2024-03-24T00:23:25.227 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:23:25.222+0000 7fd1285df740 1 -- 172.21.15.118:0/4181354890 wait complete. 2024-03-24T00:23:25.227 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:23:25.222+0000 7fd1285df740 1 -- start start 2024-03-24T00:23:25.227 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:23:25.223+0000 7fd1285df740 1 Processor -- start 2024-03-24T00:23:25.227 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:23:25.223+0000 7fd1285df740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55e14d5ca3f0 0x55e14d5d9080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:23:25.228 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:23:25.223+0000 7fd1285df740 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55e14d5cade0 0x55e14d5d95c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:23:25.228 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:23:25.223+0000 7fd1285df740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55e14d5cff50 0x55e14d5da5d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:23:25.228 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:23:25.223+0000 7fd1285df740 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd1201ab9b0 con 0x55e14d5cade0 2024-03-24T00:23:25.228 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:23:25.223+0000 7fd1285df740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd1201ac670 con 0x55e14d5ca3f0 2024-03-24T00:23:25.228 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:23:25.223+0000 7fd1285df740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd1201ac010 con 0x55e14d5cff50 2024-03-24T00:23:25.228 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:23:25.223+0000 7fd1285df740 10 client.0 _ll_register_callbacks cb 0x7fd10c009490 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T00:23:25.228 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:23:25.223+0000 7fd11ea00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55e14d5ca3f0 0x55e14d5d9080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:23:25.228 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:23:25.223+0000 7fd11e000640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55e14d5cade0 0x55e14d5d95c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:23:25.228 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:23:25.223+0000 7fd11ea00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55e14d5ca3f0 0x55e14d5d9080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.118:48054/0 (socket says 172.21.15.118:48054) 2024-03-24T00:23:25.228 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:23:25.223+0000 7fd11ea00640 1 -- 172.21.15.118:0/3347001817 learned_addr learned my addr 172.21.15.118:0/3347001817 (peer_addr_for_me v2:172.21.15.118:0/0) 2024-03-24T00:23:25.228 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:23:25.224+0000 7fd11e000640 1 -- 172.21.15.118:0/3347001817 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55e14d5cff50 msgr2=0x55e14d5da5d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:23:25.228 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:23:25.224+0000 7fd11e000640 1 --2- 172.21.15.118:0/3347001817 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55e14d5cff50 0x55e14d5da5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:23:25.228 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:23:25.224+0000 7fd11e000640 1 -- 172.21.15.118:0/3347001817 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55e14d5ca3f0 msgr2=0x55e14d5d9080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:23:25.228 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:23:25.224+0000 7fd11e000640 1 --2- 172.21.15.118:0/3347001817 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55e14d5ca3f0 0x55e14d5d9080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:23:25.228 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:23:25.224+0000 7fd11e000640 1 -- 172.21.15.118:0/3347001817 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55e14d5dabc0 con 0x55e14d5cade0 2024-03-24T00:23:25.229 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:23:25.224+0000 7fd11ea00640 1 --2- 172.21.15.118:0/3347001817 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55e14d5ca3f0 0x55e14d5d9080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T00:23:25.229 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:23:25.224+0000 7fd11e000640 1 --2- 172.21.15.118:0/3347001817 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55e14d5cade0 0x55e14d5d95c0 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7fd10c097380 tx=0x7fd10c09ae10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=efbb3b889b99d6ff server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:23:25.229 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:23:25.224+0000 7fd108c00640 10 client.0 ms_handle_connect on v2:172.21.15.118:3300/0 2024-03-24T00:23:25.229 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:23:25.224+0000 7fd108c00640 1 -- 172.21.15.118:0/3347001817 <== mon.0 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd10c09f8e0 con 0x55e14d5cade0 2024-03-24T00:23:25.229 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:23:25.224+0000 7fd108c00640 1 -- 172.21.15.118:0/3347001817 <== mon.0 v2:172.21.15.118:3300/0 2 ==== osd_map(113..113 src has 1..113) v4 ==== 10255+0+0 (secure 0 0 0) 0x7fd10c003b30 con 0x55e14d5cade0 2024-03-24T00:23:25.229 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:23:25.224+0000 7fd1285df740 1 -- 172.21.15.118:0/3347001817 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fd118025070 con 0x55e14d5cade0 2024-03-24T00:23:25.229 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:23:25.225+0000 7fd108c00640 1 -- 172.21.15.118:0/3347001817 <== mon.0 v2:172.21.15.118:3300/0 3 ==== mon_get_version_reply(handle=1 version=138) v2 ==== 24+0+0 (secure 0 0 0) 0x7fd10c081050 con 0x55e14d5cade0 2024-03-24T00:23:25.229 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:23:25.225+0000 7fd1285df740 10 client.16689 fetch_fsmap learned FSMap version 138 2024-03-24T00:23:25.229 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:23:25.225+0000 7fd1285df740 1 -- 172.21.15.118:0/3347001817 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({config=0+,fsmap=138,monmap=4+}) v3 -- 0x55e14d63bda0 con 0x55e14d5cade0 2024-03-24T00:23:25.229 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:23:25.225+0000 7fd108c00640 1 -- 172.21.15.118:0/3347001817 <== mon.0 v2:172.21.15.118:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fd118025070 con 0x55e14d5cade0 2024-03-24T00:23:25.229 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:23:25.225+0000 7fd108c00640 1 -- 172.21.15.118:0/3347001817 <== mon.0 v2:172.21.15.118:3300/0 5 ==== fsmap(e 138) v1 ==== 6788+0+0 (secure 0 0 0) 0x7fd10c097670 con 0x55e14d5cade0 2024-03-24T00:23:25.229 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:23:25.225+0000 7fd1285df740 10 client.16689 fetch_fsmap finished waiting for FSMap version 138 2024-03-24T00:23:25.229 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:23:25.225+0000 7fd1285df740 10 client.16689 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-24T00:23:25.230 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:23:25.225+0000 7fd1285df740 20 client.16689 populate_metadata read hostname 'smithi118' 2024-03-24T00:23:25.230 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:23:25.225+0000 7fd1285df740 1 --2- 172.21.15.118:0/3347001817 >> [v2:172.21.15.189:6838/4074484168,v1:172.21.15.189:6839/4074484168] conn(0x55e14d63c690 0x55e14d63eab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:23:25.230 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:23:25.225+0000 7fd1285df740 4 client.16689 mds_command: new command op to 24569 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-03-24T00:23:25.230 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:23:25.225+0000 7fd1285df740 1 -- 172.21.15.118:0/3347001817 --> [v2:172.21.15.189:6838/4074484168,v1:172.21.15.189:6839/4074484168] -- command(tid 0: {"prefix": "flush journal"}) v1 -- 0x55e14d5370c0 con 0x55e14d63c690 2024-03-24T00:23:25.230 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:23:25.226+0000 7fd11ea00640 1 --2- 172.21.15.118:0/3347001817 >> [v2:172.21.15.189:6838/4074484168,v1:172.21.15.189:6839/4074484168] conn(0x55e14d63c690 0x55e14d63eab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:23:25.230 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:23:25.226+0000 7fd11ea00640 1 --2- 172.21.15.118:0/3347001817 >> [v2:172.21.15.189:6838/4074484168,v1:172.21.15.189:6839/4074484168] conn(0x55e14d63c690 0x55e14d63eab0 crc :-1 s=READY pgs=320 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=5d2942bc8b152625 server_cookie=f3bc505bd1921abe in_seq=0 out_seq=0 2024-03-24T00:23:25.230 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:23:25.226+0000 7fd108c00640 10 client.16689 ms_handle_connect on v2:172.21.15.189:6838/4074484168 2024-03-24T00:23:25.440 INFO:journalctl@ceph.mds.j.smithi118.stdout:Mar 24 00:23:25 smithi118 podman[95611]: 2024-03-24 00:23:24.992265672 +0000 UTC m=+0.034598120 image pull c664afe56bb01db4f9afa9532c20598e201e664f2e9ba2be50db0d52674a26d2 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc 2024-03-24T00:23:25.440 INFO:journalctl@ceph.mds.j.smithi118.stdout:Mar 24 00:23:25 smithi118 podman[95611]: 2024-03-24 00:23:25.130126356 +0000 UTC m=+0.172458838 container create dc67f774391d71af5f11deab743293cfc5dfdd9caf2a15951b1ca1e76255d341 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-j, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, maintainer=Guillaume Abrioux , ceph=True, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240319, org.label-schema.license=GPLv2, GIT_CLEAN=True) 2024-03-24T00:23:25.440 INFO:journalctl@ceph.mds.j.smithi118.stdout:Mar 24 00:23:25 smithi118 podman[95611]: 2024-03-24 00:23:25.439544639 +0000 UTC m=+0.481877128 container init dc67f774391d71af5f11deab743293cfc5dfdd9caf2a15951b1ca1e76255d341 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-j, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, ceph=True, org.label-schema.build-date=20240319, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, io.buildah.version=1.33.5) 2024-03-24T00:23:25.697 INFO:journalctl@ceph.mds.j.smithi118.stdout:Mar 24 00:23:25 smithi118 podman[95611]: 2024-03-24 00:23:25.449566043 +0000 UTC m=+0.491898501 container start dc67f774391d71af5f11deab743293cfc5dfdd9caf2a15951b1ca1e76255d341 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-j, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240319, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, ceph=True, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0) 2024-03-24T00:23:25.697 INFO:journalctl@ceph.mds.j.smithi118.stdout:Mar 24 00:23:25 smithi118 ceph-mds[95675]: -- 172.21.15.118:0/1847518868 <== mon.0 v2:172.21.15.118:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x559192c27a40 con 0x559192bfa800 2024-03-24T00:23:25.697 INFO:journalctl@ceph.mds.j.smithi118.stdout:Mar 24 00:23:25 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-j[95671]: starting mds.j at 2024-03-24T00:23:25.697 INFO:journalctl@ceph.mds.j.smithi118.stdout:Mar 24 00:23:25 smithi118 bash[95611]: dc67f774391d71af5f11deab743293cfc5dfdd9caf2a15951b1ca1e76255d341 2024-03-24T00:23:25.698 INFO:journalctl@ceph.mds.j.smithi118.stdout:Mar 24 00:23:25 smithi118 systemd[1]: Started Ceph mds.j for d79a5420-e971-11ee-95cd-87774f69a715. 2024-03-24T00:23:26.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:23:25 smithi118 ceph-mon[31535]: daemon mds.j restarted 2024-03-24T00:23:26.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:23:25 smithi118 ceph-mon[31535]: osdmap e114: 12 total, 12 up, 12 in 2024-03-24T00:23:26.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:23:25 smithi118 ceph-mon[31535]: mds.? [v2:172.21.15.118:6836/3985972391,v1:172.21.15.118:6837/3985972391] up:resolve 2024-03-24T00:23:26.197 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:23:25 smithi118 ceph-mon[31535]: mds.? [v2:172.21.15.118:6834/2587469815,v1:172.21.15.118:6835/2587469815] up:boot 2024-03-24T00:23:26.197 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:23:25 smithi118 ceph-mon[31535]: Standby daemon mds.g assigned to filesystem cephfs as rank 3 2024-03-24T00:23:26.197 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:23:25 smithi118 ceph-mon[31535]: fsmap cephfs:4/5 {0=b=up:active,1=d=up:resolve,2=c=up:active,4=e=up:active} 4 up:standby-replay 2 up:standby, 1 failed 2024-03-24T00:23:26.197 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:23:25 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2024-03-24T00:23:26.197 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:23:25 smithi118 ceph-mon[31535]: fsmap cephfs:5/5 3 up:standby-replay 2 up:standby 1 up:replay 1 up:resolve 3 up:active 2024-03-24T00:23:26.197 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:23:25 smithi118 ceph-mon[31535]: pgmap v654: 97 pgs: 97 active+clean; 149 MiB data, 853 MiB used, 1.0 TiB / 1.0 TiB avail; 136 KiB/s rd, 333 KiB/s wr, 34 op/s 2024-03-24T00:23:26.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:23:25 smithi189 ceph-mon[36187]: daemon mds.j restarted 2024-03-24T00:23:26.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:23:25 smithi189 ceph-mon[36187]: osdmap e114: 12 total, 12 up, 12 in 2024-03-24T00:23:26.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:23:25 smithi189 ceph-mon[36187]: mds.? [v2:172.21.15.118:6836/3985972391,v1:172.21.15.118:6837/3985972391] up:resolve 2024-03-24T00:23:26.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:23:25 smithi189 ceph-mon[36187]: mds.? [v2:172.21.15.118:6834/2587469815,v1:172.21.15.118:6835/2587469815] up:boot 2024-03-24T00:23:26.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:23:25 smithi189 ceph-mon[36187]: Standby daemon mds.g assigned to filesystem cephfs as rank 3 2024-03-24T00:23:26.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:23:25 smithi189 ceph-mon[36187]: fsmap cephfs:4/5 {0=b=up:active,1=d=up:resolve,2=c=up:active,4=e=up:active} 4 up:standby-replay 2 up:standby, 1 failed 2024-03-24T00:23:26.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:23:25 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2024-03-24T00:23:26.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:23:25 smithi189 ceph-mon[36187]: fsmap cephfs:5/5 3 up:standby-replay 2 up:standby 1 up:replay 1 up:resolve 3 up:active 2024-03-24T00:23:26.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:23:25 smithi189 ceph-mon[36187]: pgmap v654: 97 pgs: 97 active+clean; 149 MiB data, 853 MiB used, 1.0 TiB / 1.0 TiB avail; 136 KiB/s rd, 333 KiB/s wr, 34 op/s 2024-03-24T00:23:26.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:23:25 smithi193 ceph-mon[36878]: daemon mds.j restarted 2024-03-24T00:23:26.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:23:25 smithi193 ceph-mon[36878]: osdmap e114: 12 total, 12 up, 12 in 2024-03-24T00:23:26.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:23:25 smithi193 ceph-mon[36878]: mds.? [v2:172.21.15.118:6836/3985972391,v1:172.21.15.118:6837/3985972391] up:resolve 2024-03-24T00:23:26.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:23:25 smithi193 ceph-mon[36878]: mds.? [v2:172.21.15.118:6834/2587469815,v1:172.21.15.118:6835/2587469815] up:boot 2024-03-24T00:23:26.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:23:25 smithi193 ceph-mon[36878]: Standby daemon mds.g assigned to filesystem cephfs as rank 3 2024-03-24T00:23:26.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:23:25 smithi193 ceph-mon[36878]: fsmap cephfs:4/5 {0=b=up:active,1=d=up:resolve,2=c=up:active,4=e=up:active} 4 up:standby-replay 2 up:standby, 1 failed 2024-03-24T00:23:26.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:23:25 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2024-03-24T00:23:26.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:23:25 smithi193 ceph-mon[36878]: fsmap cephfs:5/5 3 up:standby-replay 2 up:standby 1 up:replay 1 up:resolve 3 up:active 2024-03-24T00:23:26.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:23:25 smithi193 ceph-mon[36878]: pgmap v654: 97 pgs: 97 active+clean; 149 MiB data, 853 MiB used, 1.0 TiB / 1.0 TiB avail; 136 KiB/s rd, 333 KiB/s wr, 34 op/s 2024-03-24T00:23:28.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:23:27 smithi189 ceph-mon[36187]: mds.? [v2:172.21.15.118:6838/282013400,v1:172.21.15.118:6839/282013400] up:resolve 2024-03-24T00:23:28.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:23:27 smithi189 ceph-mon[36187]: fsmap cephfs:5/5 3 up:standby-replay 2 up:standby 2 up:resolve 3 up:active 2024-03-24T00:23:28.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:23:27 smithi189 ceph-mon[36187]: pgmap v655: 97 pgs: 97 active+clean; 150 MiB data, 855 MiB used, 1.0 TiB / 1.0 TiB avail; 36 KiB/s rd, 405 KiB/s wr, 50 op/s 2024-03-24T00:23:28.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:23:27 smithi193 ceph-mon[36878]: mds.? [v2:172.21.15.118:6838/282013400,v1:172.21.15.118:6839/282013400] up:resolve 2024-03-24T00:23:28.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:23:27 smithi193 ceph-mon[36878]: fsmap cephfs:5/5 3 up:standby-replay 2 up:standby 2 up:resolve 3 up:active 2024-03-24T00:23:28.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:23:27 smithi193 ceph-mon[36878]: pgmap v655: 97 pgs: 97 active+clean; 150 MiB data, 855 MiB used, 1.0 TiB / 1.0 TiB avail; 36 KiB/s rd, 405 KiB/s wr, 50 op/s 2024-03-24T00:23:28.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:23:27 smithi118 ceph-mon[31535]: mds.? [v2:172.21.15.118:6838/282013400,v1:172.21.15.118:6839/282013400] up:resolve 2024-03-24T00:23:28.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:23:27 smithi118 ceph-mon[31535]: fsmap cephfs:5/5 3 up:standby-replay 2 up:standby 2 up:resolve 3 up:active 2024-03-24T00:23:28.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:23:27 smithi118 ceph-mon[31535]: pgmap v655: 97 pgs: 97 active+clean; 150 MiB data, 855 MiB used, 1.0 TiB / 1.0 TiB avail; 36 KiB/s rd, 405 KiB/s wr, 50 op/s 2024-03-24T00:23:29.005 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:23:29.004+0000 7fd108c00640 1 -- 172.21.15.118:0/3347001817 <== mds.0 v2:172.21.15.189:6838/4074484168 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+44 (crc 0 0 0) 0x55e14d5370c0 con 0x55e14d63c690 2024-03-24T00:23:29.005 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:23:29.004+0000 7fd108c00640 10 client.16689 handle_command_reply: tid=0 2024-03-24T00:23:29.005 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:23:29.004+0000 7fd1285df740 1 client.16689 shutdown 2024-03-24T00:23:29.006 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:23:29.005+0000 7fd1285df740 1 -- 172.21.15.118:0/3347001817 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55e14d5cade0 msgr2=0x55e14d5d95c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:23:29.006 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:23:29.005+0000 7fd1285df740 1 --2- 172.21.15.118:0/3347001817 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55e14d5cade0 0x55e14d5d95c0 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7fd10c097380 tx=0x7fd10c09ae10 comp rx=0 tx=0).stop 2024-03-24T00:23:29.006 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:23:29.006+0000 7fd1285df740 1 -- 172.21.15.118:0/3347001817 shutdown_connections 2024-03-24T00:23:29.006 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:23:29.006+0000 7fd1285df740 1 --2- 172.21.15.118:0/3347001817 >> [v2:172.21.15.189:6838/4074484168,v1:172.21.15.189:6839/4074484168] conn(0x55e14d63c690 0x55e14d63eab0 crc :-1 s=READY pgs=320 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:23:29.006 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:23:29.006+0000 7fd1285df740 1 --2- 172.21.15.118:0/3347001817 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55e14d5cff50 0x55e14d5da5d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:23:29.007 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:23:29.006+0000 7fd1285df740 1 --2- 172.21.15.118:0/3347001817 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55e14d5cade0 0x55e14d5d95c0 unknown :-1 s=CLOSED pgs=225 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:23:29.007 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:23:29.006+0000 7fd1285df740 1 --2- 172.21.15.118:0/3347001817 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55e14d5ca3f0 0x55e14d5d9080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:23:29.007 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:23:29.006+0000 7fd1285df740 1 -- 172.21.15.118:0/3347001817 >> 172.21.15.118:0/3347001817 conn(0x55e14d5c5960 msgr2=0x55e14d5d4550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:23:29.007 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:23:29.006+0000 7fd108c00640 0 client.16689 ms_handle_reset on v2:172.21.15.189:6838/4074484168 2024-03-24T00:23:29.007 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:23:29.006+0000 7fd1285df740 1 -- 172.21.15.118:0/3347001817 shutdown_connections 2024-03-24T00:23:29.007 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:23:29.006+0000 7fd1285df740 1 -- 172.21.15.118:0/3347001817 wait complete. 2024-03-24T00:23:29.007 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:23:29.006+0000 7fd1285df740 20 client.16689 trim_cache size 0 max 16384 2024-03-24T00:23:29.007 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stdout:{ 2024-03-24T00:23:29.008 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stdout: "message": "", 2024-03-24T00:23:29.008 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stdout: "return_code": 0 2024-03-24T00:23:29.008 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stdout:} 2024-03-24T00:23:29.009 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:23:29.008+0000 7fd11fe00640 1 -- 172.21.15.118:0/541426089 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7fd1040a27f0 msgr2=0x7fd1040a4cb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:23:29.009 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:23:29.008+0000 7fd11fe00640 1 --2- 172.21.15.118:0/541426089 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7fd1040a27f0 0x7fd1040a4cb0 secure :-1 s=READY pgs=386 cs=0 l=1 rev1=1 crypto rx=0x55e14d5cb640 tx=0x7fd11800d040 comp rx=0 tx=0).stop 2024-03-24T00:23:29.009 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:23:29.008+0000 7fd11fe00640 1 -- 172.21.15.118:0/541426089 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fd120104910 msgr2=0x7fd120199fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:23:29.009 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:23:29.008+0000 7fd11fe00640 1 --2- 172.21.15.118:0/541426089 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fd120104910 0x7fd120199fc0 secure :-1 s=READY pgs=449 cs=0 l=1 rev1=1 crypto rx=0x7fd10c00ea60 tx=0x7fd10c00ef20 comp rx=0 tx=0).stop 2024-03-24T00:23:29.009 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:23:29.008+0000 7fd11fe00640 1 -- 172.21.15.118:0/541426089 shutdown_connections 2024-03-24T00:23:29.009 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:23:29.008+0000 7fd11fe00640 1 --2- 172.21.15.118:0/541426089 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7fd1040a27f0 0x7fd1040a4cb0 unknown :-1 s=CLOSED pgs=386 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:23:29.009 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:23:29.008+0000 7fd11fe00640 1 --2- 172.21.15.118:0/541426089 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fd1201052d0 0x7fd1201a1030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:23:29.009 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:23:29.008+0000 7fd11fe00640 1 --2- 172.21.15.118:0/541426089 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fd120104910 0x7fd120199fc0 unknown :-1 s=CLOSED pgs=449 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:23:29.009 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:23:29.008+0000 7fd11fe00640 1 --2- 172.21.15.118:0/541426089 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd120103720 0x7fd120199a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:23:29.010 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:23:29.008+0000 7fd11fe00640 1 -- 172.21.15.118:0/541426089 >> 172.21.15.118:0/541426089 conn(0x7fd1200fecb0 msgr2=0x7fd120107c10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:23:29.010 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:23:29.008+0000 7fd11fe00640 1 -- 172.21.15.118:0/541426089 shutdown_connections 2024-03-24T00:23:29.010 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:23:29.008+0000 7fd11fe00640 1 -- 172.21.15.118:0/541426089 wait complete. 2024-03-24T00:23:29.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:23:28 smithi189 ceph-mon[36187]: mds.? [v2:172.21.15.118:6838/282013400,v1:172.21.15.118:6839/282013400] up:reconnect 2024-03-24T00:23:29.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:23:28 smithi189 ceph-mon[36187]: mds.? [v2:172.21.15.118:6836/3985972391,v1:172.21.15.118:6837/3985972391] up:reconnect 2024-03-24T00:23:29.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:23:28 smithi189 ceph-mon[36187]: fsmap cephfs:5/5 3 up:standby-replay 2 up:standby 2 up:reconnect 3 up:active 2024-03-24T00:23:29.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:23:28 smithi189 ceph-mon[36187]: reconnect by client.15159 192.168.0.1:0/3463739313 after 0.0199995 2024-03-24T00:23:29.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:23:28 smithi193 ceph-mon[36878]: mds.? [v2:172.21.15.118:6838/282013400,v1:172.21.15.118:6839/282013400] up:reconnect 2024-03-24T00:23:29.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:23:28 smithi193 ceph-mon[36878]: mds.? [v2:172.21.15.118:6836/3985972391,v1:172.21.15.118:6837/3985972391] up:reconnect 2024-03-24T00:23:29.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:23:28 smithi193 ceph-mon[36878]: fsmap cephfs:5/5 3 up:standby-replay 2 up:standby 2 up:reconnect 3 up:active 2024-03-24T00:23:29.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:23:28 smithi193 ceph-mon[36878]: reconnect by client.15159 192.168.0.1:0/3463739313 after 0.0199995 2024-03-24T00:23:29.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:23:28 smithi118 ceph-mon[31535]: mds.? [v2:172.21.15.118:6838/282013400,v1:172.21.15.118:6839/282013400] up:reconnect 2024-03-24T00:23:29.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:23:28 smithi118 ceph-mon[31535]: mds.? [v2:172.21.15.118:6836/3985972391,v1:172.21.15.118:6837/3985972391] up:reconnect 2024-03-24T00:23:29.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:23:28 smithi118 ceph-mon[31535]: fsmap cephfs:5/5 3 up:standby-replay 2 up:standby 2 up:reconnect 3 up:active 2024-03-24T00:23:29.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:23:28 smithi118 ceph-mon[31535]: reconnect by client.15159 192.168.0.1:0/3463739313 after 0.0199995 2024-03-24T00:23:30.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:23:29 smithi189 ceph-mon[36187]: reconnect by client.15159 192.168.0.1:0/3463739313 after 0.0359992 2024-03-24T00:23:30.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:23:29 smithi189 ceph-mon[36187]: mds.? [v2:172.21.15.118:6836/3985972391,v1:172.21.15.118:6837/3985972391] up:rejoin 2024-03-24T00:23:30.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:23:29 smithi189 ceph-mon[36187]: mds.? [v2:172.21.15.118:6838/282013400,v1:172.21.15.118:6839/282013400] up:rejoin 2024-03-24T00:23:30.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:23:29 smithi189 ceph-mon[36187]: fsmap cephfs:5/5 3 up:standby-replay 2 up:standby 2 up:rejoin 3 up:active 2024-03-24T00:23:30.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:23:29 smithi189 ceph-mon[36187]: daemon mds.g is now active in filesystem cephfs as rank 3 2024-03-24T00:23:30.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:23:29 smithi189 ceph-mon[36187]: daemon mds.d is now active in filesystem cephfs as rank 1 2024-03-24T00:23:30.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:23:29 smithi189 ceph-mon[36187]: pgmap v656: 97 pgs: 97 active+clean; 150 MiB data, 855 MiB used, 1.0 TiB / 1.0 TiB avail; 42 KiB/s rd, 421 KiB/s wr, 67 op/s 2024-03-24T00:23:30.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:23:29 smithi193 ceph-mon[36878]: reconnect by client.15159 192.168.0.1:0/3463739313 after 0.0359992 2024-03-24T00:23:30.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:23:29 smithi193 ceph-mon[36878]: mds.? [v2:172.21.15.118:6836/3985972391,v1:172.21.15.118:6837/3985972391] up:rejoin 2024-03-24T00:23:30.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:23:29 smithi193 ceph-mon[36878]: mds.? [v2:172.21.15.118:6838/282013400,v1:172.21.15.118:6839/282013400] up:rejoin 2024-03-24T00:23:30.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:23:29 smithi193 ceph-mon[36878]: fsmap cephfs:5/5 3 up:standby-replay 2 up:standby 2 up:rejoin 3 up:active 2024-03-24T00:23:30.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:23:29 smithi193 ceph-mon[36878]: daemon mds.g is now active in filesystem cephfs as rank 3 2024-03-24T00:23:30.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:23:29 smithi193 ceph-mon[36878]: daemon mds.d is now active in filesystem cephfs as rank 1 2024-03-24T00:23:30.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:23:29 smithi193 ceph-mon[36878]: pgmap v656: 97 pgs: 97 active+clean; 150 MiB data, 855 MiB used, 1.0 TiB / 1.0 TiB avail; 42 KiB/s rd, 421 KiB/s wr, 67 op/s 2024-03-24T00:23:30.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:23:29 smithi118 ceph-mon[31535]: reconnect by client.15159 192.168.0.1:0/3463739313 after 0.0359992 2024-03-24T00:23:30.447 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:23:29 smithi118 ceph-mon[31535]: mds.? [v2:172.21.15.118:6836/3985972391,v1:172.21.15.118:6837/3985972391] up:rejoin 2024-03-24T00:23:30.447 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:23:29 smithi118 ceph-mon[31535]: mds.? [v2:172.21.15.118:6838/282013400,v1:172.21.15.118:6839/282013400] up:rejoin 2024-03-24T00:23:30.447 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:23:29 smithi118 ceph-mon[31535]: fsmap cephfs:5/5 3 up:standby-replay 2 up:standby 2 up:rejoin 3 up:active 2024-03-24T00:23:30.447 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:23:29 smithi118 ceph-mon[31535]: daemon mds.g is now active in filesystem cephfs as rank 3 2024-03-24T00:23:30.447 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:23:29 smithi118 ceph-mon[31535]: daemon mds.d is now active in filesystem cephfs as rank 1 2024-03-24T00:23:30.447 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:23:29 smithi118 ceph-mon[31535]: pgmap v656: 97 pgs: 97 active+clean; 150 MiB data, 855 MiB used, 1.0 TiB / 1.0 TiB avail; 42 KiB/s rd, 421 KiB/s wr, 67 op/s 2024-03-24T00:23:30.448 INFO:journalctl@ceph.mds.d.smithi118.stdout:Mar 24 00:23:30 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-d[95429]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: In function 'void Locker::set_xlocks_done(MutationImpl*, bool)' thread 7fa2b4400700 time 2024-03-24T00:23:30.064151+0000 2024-03-24T00:23:30.448 INFO:journalctl@ceph.mds.d.smithi118.stdout:Mar 24 00:23:30 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-d[95429]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: 626: FAILED ceph_assert(obj->is_auth()) 2024-03-24T00:23:30.448 INFO:journalctl@ceph.mds.d.smithi118.stdout:Mar 24 00:23:30 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-d[95429]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-24T00:23:30.448 INFO:journalctl@ceph.mds.d.smithi118.stdout:Mar 24 00:23:30 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-d[95429]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x135) [0x7fa2bfcbc4a1] 2024-03-24T00:23:30.448 INFO:journalctl@ceph.mds.d.smithi118.stdout:Mar 24 00:23:30 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-d[95429]: 2: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7fa2bfcbc667] 2024-03-24T00:23:30.448 INFO:journalctl@ceph.mds.d.smithi118.stdout:Mar 24 00:23:30 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-d[95429]: 3: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x55e7e2b5e223] 2024-03-24T00:23:30.448 INFO:journalctl@ceph.mds.d.smithi118.stdout:Mar 24 00:23:30 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-d[95429]: 4: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x55e7e29d85d6] 2024-03-24T00:23:30.448 INFO:journalctl@ceph.mds.d.smithi118.stdout:Mar 24 00:23:30 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-d[95429]: 5: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x55e7e29d8da1] 2024-03-24T00:23:30.448 INFO:journalctl@ceph.mds.d.smithi118.stdout:Mar 24 00:23:30 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-d[95429]: 6: (Server::handle_client_rename(boost::intrusive_ptr const&)+0x259d) [0x55e7e2a1597d] 2024-03-24T00:23:30.448 INFO:journalctl@ceph.mds.d.smithi118.stdout:Mar 24 00:23:30 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-d[95429]: 7: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x55e7e2a2b233] 2024-03-24T00:23:30.448 INFO:journalctl@ceph.mds.d.smithi118.stdout:Mar 24 00:23:30 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-d[95429]: 8: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x55e7e2a2d197] 2024-03-24T00:23:30.448 INFO:journalctl@ceph.mds.d.smithi118.stdout:Mar 24 00:23:30 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-d[95429]: 9: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x55e7e292c11b] 2024-03-24T00:23:30.448 INFO:journalctl@ceph.mds.d.smithi118.stdout:Mar 24 00:23:30 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-d[95429]: 10: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x55e7e292c6bc] 2024-03-24T00:23:30.448 INFO:journalctl@ceph.mds.d.smithi118.stdout:Mar 24 00:23:30 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-d[95429]: 11: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x55e7e29161ff] 2024-03-24T00:23:30.449 INFO:journalctl@ceph.mds.d.smithi118.stdout:Mar 24 00:23:30 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-d[95429]: 12: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7fa2bff4e488] 2024-03-24T00:23:30.449 INFO:journalctl@ceph.mds.d.smithi118.stdout:Mar 24 00:23:30 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-d[95429]: 13: (DispatchQueue::entry()+0x51f) [0x7fa2bff4b60f] 2024-03-24T00:23:30.449 INFO:journalctl@ceph.mds.d.smithi118.stdout:Mar 24 00:23:30 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-d[95429]: 14: (DispatchQueue::DispatchThread::entry()+0x11) [0x7fa2c001ee81] 2024-03-24T00:23:30.449 INFO:journalctl@ceph.mds.d.smithi118.stdout:Mar 24 00:23:30 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-d[95429]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7fa2bea391ca] 2024-03-24T00:23:30.449 INFO:journalctl@ceph.mds.d.smithi118.stdout:Mar 24 00:23:30 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-d[95429]: 16: clone() 2024-03-24T00:23:30.449 INFO:journalctl@ceph.mds.g.smithi118.stdout:Mar 24 00:23:30 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-g[95433]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: In function 'void Locker::set_xlocks_done(MutationImpl*, bool)' thread 7f7c60000700 time 2024-03-24T00:23:30.010332+0000 2024-03-24T00:23:30.449 INFO:journalctl@ceph.mds.g.smithi118.stdout:Mar 24 00:23:30 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-g[95433]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: 626: FAILED ceph_assert(obj->is_auth()) 2024-03-24T00:23:30.449 INFO:journalctl@ceph.mds.g.smithi118.stdout:Mar 24 00:23:30 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-g[95433]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-24T00:23:30.449 INFO:journalctl@ceph.mds.g.smithi118.stdout:Mar 24 00:23:30 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-g[95433]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x135) [0x7f7c6b8314a1] 2024-03-24T00:23:30.449 INFO:journalctl@ceph.mds.g.smithi118.stdout:Mar 24 00:23:30 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-g[95433]: 2: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7f7c6b831667] 2024-03-24T00:23:30.449 INFO:journalctl@ceph.mds.g.smithi118.stdout:Mar 24 00:23:30 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-g[95433]: 3: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x5589c9ab3223] 2024-03-24T00:23:30.450 INFO:journalctl@ceph.mds.g.smithi118.stdout:Mar 24 00:23:30 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-g[95433]: 4: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x5589c992d5d6] 2024-03-24T00:23:30.450 INFO:journalctl@ceph.mds.g.smithi118.stdout:Mar 24 00:23:30 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-g[95433]: 5: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x5589c992dda1] 2024-03-24T00:23:30.450 INFO:journalctl@ceph.mds.g.smithi118.stdout:Mar 24 00:23:30 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-g[95433]: 6: (Server::handle_client_rename(boost::intrusive_ptr const&)+0x259d) [0x5589c996a97d] 2024-03-24T00:23:30.450 INFO:journalctl@ceph.mds.g.smithi118.stdout:Mar 24 00:23:30 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-g[95433]: 7: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x5589c9980233] 2024-03-24T00:23:30.450 INFO:journalctl@ceph.mds.g.smithi118.stdout:Mar 24 00:23:30 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-g[95433]: 8: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x5589c9982197] 2024-03-24T00:23:30.450 INFO:journalctl@ceph.mds.g.smithi118.stdout:Mar 24 00:23:30 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-g[95433]: 9: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x5589c988111b] 2024-03-24T00:23:30.450 INFO:journalctl@ceph.mds.g.smithi118.stdout:Mar 24 00:23:30 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-g[95433]: 10: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x5589c98816bc] 2024-03-24T00:23:30.450 INFO:journalctl@ceph.mds.g.smithi118.stdout:Mar 24 00:23:30 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-g[95433]: 11: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x5589c986b1ff] 2024-03-24T00:23:30.450 INFO:journalctl@ceph.mds.g.smithi118.stdout:Mar 24 00:23:30 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-g[95433]: 12: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7f7c6bac3488] 2024-03-24T00:23:30.450 INFO:journalctl@ceph.mds.g.smithi118.stdout:Mar 24 00:23:30 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-g[95433]: 13: (DispatchQueue::entry()+0x51f) [0x7f7c6bac060f] 2024-03-24T00:23:30.450 INFO:journalctl@ceph.mds.g.smithi118.stdout:Mar 24 00:23:30 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-g[95433]: 14: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f7c6bb93e81] 2024-03-24T00:23:30.450 INFO:journalctl@ceph.mds.g.smithi118.stdout:Mar 24 00:23:30 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-g[95433]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7f7c6a5ae1ca] 2024-03-24T00:23:30.450 INFO:journalctl@ceph.mds.g.smithi118.stdout:Mar 24 00:23:30 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-g[95433]: 16: clone() 2024-03-24T00:23:30.451 INFO:journalctl@ceph.mds.d.smithi118.stdout:Mar 24 00:23:30 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-d[95429]: 2024-03-24T00:23:30.065+0000 7fa2b4400700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: In function 'void Locker::set_xlocks_done(MutationImpl*, bool)' thread 7fa2b4400700 time 2024-03-24T00:23:30.064151+0000 2024-03-24T00:23:30.451 INFO:journalctl@ceph.mds.d.smithi118.stdout:Mar 24 00:23:30 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-d[95429]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: 626: FAILED ceph_assert(obj->is_auth()) 2024-03-24T00:23:30.451 INFO:journalctl@ceph.mds.d.smithi118.stdout:Mar 24 00:23:30 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-d[95429]: 2024-03-24T00:23:30.451 INFO:journalctl@ceph.mds.d.smithi118.stdout:Mar 24 00:23:30 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-d[95429]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-24T00:23:30.451 INFO:journalctl@ceph.mds.d.smithi118.stdout:Mar 24 00:23:30 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-d[95429]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x135) [0x7fa2bfcbc4a1] 2024-03-24T00:23:30.451 INFO:journalctl@ceph.mds.d.smithi118.stdout:Mar 24 00:23:30 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-d[95429]: 2: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7fa2bfcbc667] 2024-03-24T00:23:30.451 INFO:journalctl@ceph.mds.d.smithi118.stdout:Mar 24 00:23:30 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-d[95429]: 3: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x55e7e2b5e223] 2024-03-24T00:23:30.451 INFO:journalctl@ceph.mds.d.smithi118.stdout:Mar 24 00:23:30 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-d[95429]: 4: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x55e7e29d85d6] 2024-03-24T00:23:30.451 INFO:journalctl@ceph.mds.d.smithi118.stdout:Mar 24 00:23:30 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-d[95429]: 5: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x55e7e29d8da1] 2024-03-24T00:23:30.451 INFO:journalctl@ceph.mds.d.smithi118.stdout:Mar 24 00:23:30 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-d[95429]: 6: (Server::handle_client_rename(boost::intrusive_ptr const&)+0x259d) [0x55e7e2a1597d] 2024-03-24T00:23:30.452 INFO:journalctl@ceph.mds.d.smithi118.stdout:Mar 24 00:23:30 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-d[95429]: 7: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x55e7e2a2b233] 2024-03-24T00:23:30.452 INFO:journalctl@ceph.mds.d.smithi118.stdout:Mar 24 00:23:30 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-d[95429]: 8: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x55e7e2a2d197] 2024-03-24T00:23:30.452 INFO:journalctl@ceph.mds.d.smithi118.stdout:Mar 24 00:23:30 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-d[95429]: 9: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x55e7e292c11b] 2024-03-24T00:23:30.452 INFO:journalctl@ceph.mds.d.smithi118.stdout:Mar 24 00:23:30 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-d[95429]: 10: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x55e7e292c6bc] 2024-03-24T00:23:30.452 INFO:journalctl@ceph.mds.d.smithi118.stdout:Mar 24 00:23:30 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-d[95429]: 11: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x55e7e29161ff] 2024-03-24T00:23:30.452 INFO:journalctl@ceph.mds.d.smithi118.stdout:Mar 24 00:23:30 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-d[95429]: 12: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7fa2bff4e488] 2024-03-24T00:23:30.452 INFO:journalctl@ceph.mds.d.smithi118.stdout:Mar 24 00:23:30 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-d[95429]: 13: (DispatchQueue::entry()+0x51f) [0x7fa2bff4b60f] 2024-03-24T00:23:30.452 INFO:journalctl@ceph.mds.d.smithi118.stdout:Mar 24 00:23:30 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-d[95429]: 14: (DispatchQueue::DispatchThread::entry()+0x11) [0x7fa2c001ee81] 2024-03-24T00:23:30.452 INFO:journalctl@ceph.mds.d.smithi118.stdout:Mar 24 00:23:30 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-d[95429]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7fa2bea391ca] 2024-03-24T00:23:30.452 INFO:journalctl@ceph.mds.d.smithi118.stdout:Mar 24 00:23:30 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-d[95429]: 16: clone() 2024-03-24T00:23:30.452 INFO:journalctl@ceph.mds.d.smithi118.stdout:Mar 24 00:23:30 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-d[95429]: 2024-03-24T00:23:30.452 INFO:journalctl@ceph.mds.d.smithi118.stdout:Mar 24 00:23:30 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-d[95429]: *** Caught signal (Aborted) ** 2024-03-24T00:23:30.452 INFO:journalctl@ceph.mds.d.smithi118.stdout:Mar 24 00:23:30 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-d[95429]: in thread 7fa2b4400700 thread_name:ms_dispatch 2024-03-24T00:23:30.452 INFO:journalctl@ceph.mds.d.smithi118.stdout:Mar 24 00:23:30 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-d[95429]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-24T00:23:30.452 INFO:journalctl@ceph.mds.d.smithi118.stdout:Mar 24 00:23:30 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-d[95429]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7fa2bea43d20] 2024-03-24T00:23:30.453 INFO:journalctl@ceph.mds.g.smithi118.stdout:Mar 24 00:23:30 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-g[95433]: 2024-03-24T00:23:30.012+0000 7f7c60000700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: In function 'void Locker::set_xlocks_done(MutationImpl*, bool)' thread 7f7c60000700 time 2024-03-24T00:23:30.010332+0000 2024-03-24T00:23:30.453 INFO:journalctl@ceph.mds.g.smithi118.stdout:Mar 24 00:23:30 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-g[95433]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: 626: FAILED ceph_assert(obj->is_auth()) 2024-03-24T00:23:30.453 INFO:journalctl@ceph.mds.g.smithi118.stdout:Mar 24 00:23:30 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-g[95433]: 2024-03-24T00:23:30.453 INFO:journalctl@ceph.mds.g.smithi118.stdout:Mar 24 00:23:30 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-g[95433]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-24T00:23:30.453 INFO:journalctl@ceph.mds.g.smithi118.stdout:Mar 24 00:23:30 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-g[95433]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x135) [0x7f7c6b8314a1] 2024-03-24T00:23:30.453 INFO:journalctl@ceph.mds.g.smithi118.stdout:Mar 24 00:23:30 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-g[95433]: 2: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7f7c6b831667] 2024-03-24T00:23:30.453 INFO:journalctl@ceph.mds.g.smithi118.stdout:Mar 24 00:23:30 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-g[95433]: 3: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x5589c9ab3223] 2024-03-24T00:23:30.453 INFO:journalctl@ceph.mds.g.smithi118.stdout:Mar 24 00:23:30 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-g[95433]: 4: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x5589c992d5d6] 2024-03-24T00:23:30.453 INFO:journalctl@ceph.mds.g.smithi118.stdout:Mar 24 00:23:30 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-g[95433]: 5: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x5589c992dda1] 2024-03-24T00:23:30.453 INFO:journalctl@ceph.mds.g.smithi118.stdout:Mar 24 00:23:30 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-g[95433]: 6: (Server::handle_client_rename(boost::intrusive_ptr const&)+0x259d) [0x5589c996a97d] 2024-03-24T00:23:30.454 INFO:journalctl@ceph.mds.g.smithi118.stdout:Mar 24 00:23:30 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-g[95433]: 7: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x5589c9980233] 2024-03-24T00:23:30.454 INFO:journalctl@ceph.mds.g.smithi118.stdout:Mar 24 00:23:30 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-g[95433]: 8: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x5589c9982197] 2024-03-24T00:23:30.454 INFO:journalctl@ceph.mds.g.smithi118.stdout:Mar 24 00:23:30 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-g[95433]: 9: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x5589c988111b] 2024-03-24T00:23:30.454 INFO:journalctl@ceph.mds.g.smithi118.stdout:Mar 24 00:23:30 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-g[95433]: 10: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x5589c98816bc] 2024-03-24T00:23:30.454 INFO:journalctl@ceph.mds.g.smithi118.stdout:Mar 24 00:23:30 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-g[95433]: 11: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x5589c986b1ff] 2024-03-24T00:23:30.454 INFO:journalctl@ceph.mds.g.smithi118.stdout:Mar 24 00:23:30 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-g[95433]: 12: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7f7c6bac3488] 2024-03-24T00:23:30.454 INFO:journalctl@ceph.mds.g.smithi118.stdout:Mar 24 00:23:30 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-g[95433]: 13: (DispatchQueue::entry()+0x51f) [0x7f7c6bac060f] 2024-03-24T00:23:30.454 INFO:journalctl@ceph.mds.g.smithi118.stdout:Mar 24 00:23:30 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-g[95433]: 14: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f7c6bb93e81] 2024-03-24T00:23:30.454 INFO:journalctl@ceph.mds.g.smithi118.stdout:Mar 24 00:23:30 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-g[95433]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7f7c6a5ae1ca] 2024-03-24T00:23:30.454 INFO:journalctl@ceph.mds.g.smithi118.stdout:Mar 24 00:23:30 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-g[95433]: 16: clone() 2024-03-24T00:23:30.454 INFO:journalctl@ceph.mds.g.smithi118.stdout:Mar 24 00:23:30 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-g[95433]: 2024-03-24T00:23:30.454 INFO:journalctl@ceph.mds.g.smithi118.stdout:Mar 24 00:23:30 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-g[95433]: *** Caught signal (Aborted) ** 2024-03-24T00:23:30.454 INFO:journalctl@ceph.mds.g.smithi118.stdout:Mar 24 00:23:30 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-g[95433]: in thread 7f7c60000700 thread_name:ms_dispatch 2024-03-24T00:23:30.455 INFO:journalctl@ceph.mds.g.smithi118.stdout:Mar 24 00:23:30 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-g[95433]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-24T00:23:30.455 INFO:journalctl@ceph.mds.g.smithi118.stdout:Mar 24 00:23:30 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-g[95433]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f7c6a5b8d20] 2024-03-24T00:23:30.455 INFO:journalctl@ceph.mds.d.smithi118.stdout:Mar 24 00:23:30 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-d[95429]: 2: gsignal() 2024-03-24T00:23:30.455 INFO:journalctl@ceph.mds.d.smithi118.stdout:Mar 24 00:23:30 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-d[95429]: 3: abort() 2024-03-24T00:23:30.455 INFO:journalctl@ceph.mds.d.smithi118.stdout:Mar 24 00:23:30 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-d[95429]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x18f) [0x7fa2bfcbc4fb] 2024-03-24T00:23:30.455 INFO:journalctl@ceph.mds.d.smithi118.stdout:Mar 24 00:23:30 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-d[95429]: 5: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7fa2bfcbc667] 2024-03-24T00:23:30.455 INFO:journalctl@ceph.mds.d.smithi118.stdout:Mar 24 00:23:30 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-d[95429]: 6: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x55e7e2b5e223] 2024-03-24T00:23:30.455 INFO:journalctl@ceph.mds.d.smithi118.stdout:Mar 24 00:23:30 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-d[95429]: 7: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x55e7e29d85d6] 2024-03-24T00:23:30.455 INFO:journalctl@ceph.mds.d.smithi118.stdout:Mar 24 00:23:30 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-d[95429]: 8: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x55e7e29d8da1] 2024-03-24T00:23:30.456 INFO:journalctl@ceph.mds.d.smithi118.stdout:Mar 24 00:23:30 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-d[95429]: 9: (Server::handle_client_rename(boost::intrusive_ptr const&)+0x259d) [0x55e7e2a1597d] 2024-03-24T00:23:30.456 INFO:journalctl@ceph.mds.d.smithi118.stdout:Mar 24 00:23:30 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-d[95429]: 10: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x55e7e2a2b233] 2024-03-24T00:23:30.456 INFO:journalctl@ceph.mds.d.smithi118.stdout:Mar 24 00:23:30 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-d[95429]: 11: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x55e7e2a2d197] 2024-03-24T00:23:30.456 INFO:journalctl@ceph.mds.d.smithi118.stdout:Mar 24 00:23:30 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-d[95429]: 12: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x55e7e292c11b] 2024-03-24T00:23:30.456 INFO:journalctl@ceph.mds.d.smithi118.stdout:Mar 24 00:23:30 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-d[95429]: 13: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x55e7e292c6bc] 2024-03-24T00:23:30.456 INFO:journalctl@ceph.mds.d.smithi118.stdout:Mar 24 00:23:30 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-d[95429]: 14: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x55e7e29161ff] 2024-03-24T00:23:30.456 INFO:journalctl@ceph.mds.d.smithi118.stdout:Mar 24 00:23:30 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-d[95429]: 15: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7fa2bff4e488] 2024-03-24T00:23:30.456 INFO:journalctl@ceph.mds.d.smithi118.stdout:Mar 24 00:23:30 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-d[95429]: 16: (DispatchQueue::entry()+0x51f) [0x7fa2bff4b60f] 2024-03-24T00:23:30.456 INFO:journalctl@ceph.mds.d.smithi118.stdout:Mar 24 00:23:30 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-d[95429]: 17: (DispatchQueue::DispatchThread::entry()+0x11) [0x7fa2c001ee81] 2024-03-24T00:23:30.456 INFO:journalctl@ceph.mds.d.smithi118.stdout:Mar 24 00:23:30 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-d[95429]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7fa2bea391ca] 2024-03-24T00:23:30.456 INFO:journalctl@ceph.mds.d.smithi118.stdout:Mar 24 00:23:30 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-d[95429]: 19: clone() 2024-03-24T00:23:30.456 INFO:journalctl@ceph.mds.d.smithi118.stdout:Mar 24 00:23:30 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-d[95429]: 2024-03-24T00:23:30.068+0000 7fa2b4400700 -1 *** Caught signal (Aborted) ** 2024-03-24T00:23:30.456 INFO:journalctl@ceph.mds.d.smithi118.stdout:Mar 24 00:23:30 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-d[95429]: in thread 7fa2b4400700 thread_name:ms_dispatch 2024-03-24T00:23:30.456 INFO:journalctl@ceph.mds.d.smithi118.stdout:Mar 24 00:23:30 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-d[95429]: 2024-03-24T00:23:30.456 INFO:journalctl@ceph.mds.d.smithi118.stdout:Mar 24 00:23:30 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-d[95429]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-24T00:23:30.457 INFO:journalctl@ceph.mds.d.smithi118.stdout:Mar 24 00:23:30 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-d[95429]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7fa2bea43d20] 2024-03-24T00:23:30.457 INFO:journalctl@ceph.mds.d.smithi118.stdout:Mar 24 00:23:30 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-d[95429]: 2: gsignal() 2024-03-24T00:23:30.457 INFO:journalctl@ceph.mds.d.smithi118.stdout:Mar 24 00:23:30 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-d[95429]: 3: abort() 2024-03-24T00:23:30.457 INFO:journalctl@ceph.mds.d.smithi118.stdout:Mar 24 00:23:30 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-d[95429]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x18f) [0x7fa2bfcbc4fb] 2024-03-24T00:23:30.457 INFO:journalctl@ceph.mds.d.smithi118.stdout:Mar 24 00:23:30 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-d[95429]: 5: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7fa2bfcbc667] 2024-03-24T00:23:30.457 INFO:journalctl@ceph.mds.g.smithi118.stdout:Mar 24 00:23:30 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-g[95433]: 2: gsignal() 2024-03-24T00:23:30.457 INFO:journalctl@ceph.mds.g.smithi118.stdout:Mar 24 00:23:30 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-g[95433]: 3: abort() 2024-03-24T00:23:30.457 INFO:journalctl@ceph.mds.g.smithi118.stdout:Mar 24 00:23:30 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-g[95433]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x18f) [0x7f7c6b8314fb] 2024-03-24T00:23:30.457 INFO:journalctl@ceph.mds.g.smithi118.stdout:Mar 24 00:23:30 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-g[95433]: 5: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7f7c6b831667] 2024-03-24T00:23:30.457 INFO:journalctl@ceph.mds.g.smithi118.stdout:Mar 24 00:23:30 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-g[95433]: 6: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x5589c9ab3223] 2024-03-24T00:23:30.458 INFO:journalctl@ceph.mds.g.smithi118.stdout:Mar 24 00:23:30 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-g[95433]: 7: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x5589c992d5d6] 2024-03-24T00:23:30.458 INFO:journalctl@ceph.mds.g.smithi118.stdout:Mar 24 00:23:30 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-g[95433]: 8: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x5589c992dda1] 2024-03-24T00:23:30.458 INFO:journalctl@ceph.mds.g.smithi118.stdout:Mar 24 00:23:30 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-g[95433]: 9: (Server::handle_client_rename(boost::intrusive_ptr const&)+0x259d) [0x5589c996a97d] 2024-03-24T00:23:30.458 INFO:journalctl@ceph.mds.g.smithi118.stdout:Mar 24 00:23:30 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-g[95433]: 10: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x5589c9980233] 2024-03-24T00:23:30.458 INFO:journalctl@ceph.mds.g.smithi118.stdout:Mar 24 00:23:30 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-g[95433]: 11: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x5589c9982197] 2024-03-24T00:23:30.458 INFO:journalctl@ceph.mds.g.smithi118.stdout:Mar 24 00:23:30 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-g[95433]: 12: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x5589c988111b] 2024-03-24T00:23:30.458 INFO:journalctl@ceph.mds.g.smithi118.stdout:Mar 24 00:23:30 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-g[95433]: 13: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x5589c98816bc] 2024-03-24T00:23:30.458 INFO:journalctl@ceph.mds.g.smithi118.stdout:Mar 24 00:23:30 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-g[95433]: 14: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x5589c986b1ff] 2024-03-24T00:23:30.458 INFO:journalctl@ceph.mds.g.smithi118.stdout:Mar 24 00:23:30 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-g[95433]: 15: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7f7c6bac3488] 2024-03-24T00:23:30.458 INFO:journalctl@ceph.mds.g.smithi118.stdout:Mar 24 00:23:30 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-g[95433]: 16: (DispatchQueue::entry()+0x51f) [0x7f7c6bac060f] 2024-03-24T00:23:30.458 INFO:journalctl@ceph.mds.g.smithi118.stdout:Mar 24 00:23:30 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-g[95433]: 17: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f7c6bb93e81] 2024-03-24T00:23:30.458 INFO:journalctl@ceph.mds.g.smithi118.stdout:Mar 24 00:23:30 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-g[95433]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7f7c6a5ae1ca] 2024-03-24T00:23:30.458 INFO:journalctl@ceph.mds.g.smithi118.stdout:Mar 24 00:23:30 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-g[95433]: 19: clone() 2024-03-24T00:23:30.458 INFO:journalctl@ceph.mds.g.smithi118.stdout:Mar 24 00:23:30 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-g[95433]: 2024-03-24T00:23:30.015+0000 7f7c60000700 -1 *** Caught signal (Aborted) ** 2024-03-24T00:23:30.459 INFO:journalctl@ceph.mds.g.smithi118.stdout:Mar 24 00:23:30 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-g[95433]: in thread 7f7c60000700 thread_name:ms_dispatch 2024-03-24T00:23:30.459 INFO:journalctl@ceph.mds.g.smithi118.stdout:Mar 24 00:23:30 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-g[95433]: 2024-03-24T00:23:30.459 INFO:journalctl@ceph.mds.g.smithi118.stdout:Mar 24 00:23:30 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-g[95433]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-24T00:23:30.459 INFO:journalctl@ceph.mds.g.smithi118.stdout:Mar 24 00:23:30 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-g[95433]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f7c6a5b8d20] 2024-03-24T00:23:30.459 INFO:journalctl@ceph.mds.g.smithi118.stdout:Mar 24 00:23:30 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-g[95433]: 2: gsignal() 2024-03-24T00:23:30.459 INFO:journalctl@ceph.mds.g.smithi118.stdout:Mar 24 00:23:30 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-g[95433]: 3: abort() 2024-03-24T00:23:30.459 INFO:journalctl@ceph.mds.g.smithi118.stdout:Mar 24 00:23:30 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-g[95433]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x18f) [0x7f7c6b8314fb] 2024-03-24T00:23:30.459 INFO:journalctl@ceph.mds.g.smithi118.stdout:Mar 24 00:23:30 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-g[95433]: 5: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7f7c6b831667] 2024-03-24T00:23:30.459 INFO:journalctl@ceph.mds.d.smithi118.stdout:Mar 24 00:23:30 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-d[95429]: 6: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x55e7e2b5e223] 2024-03-24T00:23:30.459 INFO:journalctl@ceph.mds.d.smithi118.stdout:Mar 24 00:23:30 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-d[95429]: 7: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x55e7e29d85d6] 2024-03-24T00:23:30.460 INFO:journalctl@ceph.mds.d.smithi118.stdout:Mar 24 00:23:30 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-d[95429]: 8: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x55e7e29d8da1] 2024-03-24T00:23:30.460 INFO:journalctl@ceph.mds.d.smithi118.stdout:Mar 24 00:23:30 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-d[95429]: 9: (Server::handle_client_rename(boost::intrusive_ptr const&)+0x259d) [0x55e7e2a1597d] 2024-03-24T00:23:30.460 INFO:journalctl@ceph.mds.d.smithi118.stdout:Mar 24 00:23:30 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-d[95429]: 10: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x55e7e2a2b233] 2024-03-24T00:23:30.460 INFO:journalctl@ceph.mds.d.smithi118.stdout:Mar 24 00:23:30 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-d[95429]: 11: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x55e7e2a2d197] 2024-03-24T00:23:30.460 INFO:journalctl@ceph.mds.d.smithi118.stdout:Mar 24 00:23:30 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-d[95429]: 12: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x55e7e292c11b] 2024-03-24T00:23:30.460 INFO:journalctl@ceph.mds.d.smithi118.stdout:Mar 24 00:23:30 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-d[95429]: 13: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x55e7e292c6bc] 2024-03-24T00:23:30.460 INFO:journalctl@ceph.mds.d.smithi118.stdout:Mar 24 00:23:30 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-d[95429]: 14: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x55e7e29161ff] 2024-03-24T00:23:30.460 INFO:journalctl@ceph.mds.d.smithi118.stdout:Mar 24 00:23:30 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-d[95429]: 15: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7fa2bff4e488] 2024-03-24T00:23:30.460 INFO:journalctl@ceph.mds.d.smithi118.stdout:Mar 24 00:23:30 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-d[95429]: 16: (DispatchQueue::entry()+0x51f) [0x7fa2bff4b60f] 2024-03-24T00:23:30.460 INFO:journalctl@ceph.mds.d.smithi118.stdout:Mar 24 00:23:30 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-d[95429]: 17: (DispatchQueue::DispatchThread::entry()+0x11) [0x7fa2c001ee81] 2024-03-24T00:23:30.460 INFO:journalctl@ceph.mds.d.smithi118.stdout:Mar 24 00:23:30 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-d[95429]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7fa2bea391ca] 2024-03-24T00:23:30.460 INFO:journalctl@ceph.mds.d.smithi118.stdout:Mar 24 00:23:30 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-d[95429]: 19: clone() 2024-03-24T00:23:30.460 INFO:journalctl@ceph.mds.d.smithi118.stdout:Mar 24 00:23:30 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-d[95429]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-03-24T00:23:30.460 INFO:journalctl@ceph.mds.d.smithi118.stdout:Mar 24 00:23:30 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-d[95429]: 2024-03-24T00:23:30.460 INFO:journalctl@ceph.mds.d.smithi118.stdout:Mar 24 00:23:30 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-d[95429]: -1> 2024-03-24T00:23:30.065+0000 7fa2b4400700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: In function 'void Locker::set_xlocks_done(MutationImpl*, bool)' thread 7fa2b4400700 time 2024-03-24T00:23:30.064151+0000 2024-03-24T00:23:30.461 INFO:journalctl@ceph.mds.d.smithi118.stdout:Mar 24 00:23:30 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-d[95429]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: 626: FAILED ceph_assert(obj->is_auth()) 2024-03-24T00:23:30.461 INFO:journalctl@ceph.mds.d.smithi118.stdout:Mar 24 00:23:30 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-d[95429]: 2024-03-24T00:23:30.461 INFO:journalctl@ceph.mds.d.smithi118.stdout:Mar 24 00:23:30 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-d[95429]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-24T00:23:30.461 INFO:journalctl@ceph.mds.d.smithi118.stdout:Mar 24 00:23:30 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-d[95429]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x135) [0x7fa2bfcbc4a1] 2024-03-24T00:23:30.461 INFO:journalctl@ceph.mds.d.smithi118.stdout:Mar 24 00:23:30 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-d[95429]: 2: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7fa2bfcbc667] 2024-03-24T00:23:30.461 INFO:journalctl@ceph.mds.g.smithi118.stdout:Mar 24 00:23:30 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-g[95433]: 6: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x5589c9ab3223] 2024-03-24T00:23:30.461 INFO:journalctl@ceph.mds.g.smithi118.stdout:Mar 24 00:23:30 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-g[95433]: 7: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x5589c992d5d6] 2024-03-24T00:23:30.461 INFO:journalctl@ceph.mds.g.smithi118.stdout:Mar 24 00:23:30 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-g[95433]: 8: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x5589c992dda1] 2024-03-24T00:23:30.461 INFO:journalctl@ceph.mds.g.smithi118.stdout:Mar 24 00:23:30 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-g[95433]: 9: (Server::handle_client_rename(boost::intrusive_ptr const&)+0x259d) [0x5589c996a97d] 2024-03-24T00:23:30.461 INFO:journalctl@ceph.mds.g.smithi118.stdout:Mar 24 00:23:30 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-g[95433]: 10: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x5589c9980233] 2024-03-24T00:23:30.461 INFO:journalctl@ceph.mds.g.smithi118.stdout:Mar 24 00:23:30 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-g[95433]: 11: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x5589c9982197] 2024-03-24T00:23:30.461 INFO:journalctl@ceph.mds.g.smithi118.stdout:Mar 24 00:23:30 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-g[95433]: 12: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x5589c988111b] 2024-03-24T00:23:30.461 INFO:journalctl@ceph.mds.g.smithi118.stdout:Mar 24 00:23:30 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-g[95433]: 13: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x5589c98816bc] 2024-03-24T00:23:30.462 INFO:journalctl@ceph.mds.g.smithi118.stdout:Mar 24 00:23:30 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-g[95433]: 14: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x5589c986b1ff] 2024-03-24T00:23:30.462 INFO:journalctl@ceph.mds.g.smithi118.stdout:Mar 24 00:23:30 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-g[95433]: 15: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7f7c6bac3488] 2024-03-24T00:23:30.462 INFO:journalctl@ceph.mds.g.smithi118.stdout:Mar 24 00:23:30 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-g[95433]: 16: (DispatchQueue::entry()+0x51f) [0x7f7c6bac060f] 2024-03-24T00:23:30.462 INFO:journalctl@ceph.mds.g.smithi118.stdout:Mar 24 00:23:30 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-g[95433]: 17: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f7c6bb93e81] 2024-03-24T00:23:30.462 INFO:journalctl@ceph.mds.g.smithi118.stdout:Mar 24 00:23:30 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-g[95433]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7f7c6a5ae1ca] 2024-03-24T00:23:30.462 INFO:journalctl@ceph.mds.g.smithi118.stdout:Mar 24 00:23:30 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-g[95433]: 19: clone() 2024-03-24T00:23:30.462 INFO:journalctl@ceph.mds.g.smithi118.stdout:Mar 24 00:23:30 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-g[95433]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-03-24T00:23:30.462 INFO:journalctl@ceph.mds.g.smithi118.stdout:Mar 24 00:23:30 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-g[95433]: 2024-03-24T00:23:30.462 INFO:journalctl@ceph.mds.g.smithi118.stdout:Mar 24 00:23:30 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-g[95433]: -1> 2024-03-24T00:23:30.012+0000 7f7c60000700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: In function 'void Locker::set_xlocks_done(MutationImpl*, bool)' thread 7f7c60000700 time 2024-03-24T00:23:30.010332+0000 2024-03-24T00:23:30.462 INFO:journalctl@ceph.mds.g.smithi118.stdout:Mar 24 00:23:30 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-g[95433]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: 626: FAILED ceph_assert(obj->is_auth()) 2024-03-24T00:23:30.462 INFO:journalctl@ceph.mds.g.smithi118.stdout:Mar 24 00:23:30 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-g[95433]: 2024-03-24T00:23:30.462 INFO:journalctl@ceph.mds.g.smithi118.stdout:Mar 24 00:23:30 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-g[95433]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-24T00:23:30.462 INFO:journalctl@ceph.mds.g.smithi118.stdout:Mar 24 00:23:30 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-g[95433]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x135) [0x7f7c6b8314a1] 2024-03-24T00:23:30.462 INFO:journalctl@ceph.mds.g.smithi118.stdout:Mar 24 00:23:30 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-g[95433]: 2: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7f7c6b831667] 2024-03-24T00:23:30.463 INFO:journalctl@ceph.mds.d.smithi118.stdout:Mar 24 00:23:30 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-d[95429]: 3: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x55e7e2b5e223] 2024-03-24T00:23:30.463 INFO:journalctl@ceph.mds.d.smithi118.stdout:Mar 24 00:23:30 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-d[95429]: 4: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x55e7e29d85d6] 2024-03-24T00:23:30.463 INFO:journalctl@ceph.mds.d.smithi118.stdout:Mar 24 00:23:30 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-d[95429]: 5: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x55e7e29d8da1] 2024-03-24T00:23:30.463 INFO:journalctl@ceph.mds.d.smithi118.stdout:Mar 24 00:23:30 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-d[95429]: 6: (Server::handle_client_rename(boost::intrusive_ptr const&)+0x259d) [0x55e7e2a1597d] 2024-03-24T00:23:30.463 INFO:journalctl@ceph.mds.d.smithi118.stdout:Mar 24 00:23:30 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-d[95429]: 7: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x55e7e2a2b233] 2024-03-24T00:23:30.463 INFO:journalctl@ceph.mds.d.smithi118.stdout:Mar 24 00:23:30 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-d[95429]: 8: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x55e7e2a2d197] 2024-03-24T00:23:30.463 INFO:journalctl@ceph.mds.d.smithi118.stdout:Mar 24 00:23:30 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-d[95429]: 9: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x55e7e292c11b] 2024-03-24T00:23:30.463 INFO:journalctl@ceph.mds.d.smithi118.stdout:Mar 24 00:23:30 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-d[95429]: 10: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x55e7e292c6bc] 2024-03-24T00:23:30.463 INFO:journalctl@ceph.mds.d.smithi118.stdout:Mar 24 00:23:30 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-d[95429]: 11: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x55e7e29161ff] 2024-03-24T00:23:30.463 INFO:journalctl@ceph.mds.d.smithi118.stdout:Mar 24 00:23:30 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-d[95429]: 12: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7fa2bff4e488] 2024-03-24T00:23:30.463 INFO:journalctl@ceph.mds.d.smithi118.stdout:Mar 24 00:23:30 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-d[95429]: 13: (DispatchQueue::entry()+0x51f) [0x7fa2bff4b60f] 2024-03-24T00:23:30.463 INFO:journalctl@ceph.mds.d.smithi118.stdout:Mar 24 00:23:30 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-d[95429]: 14: (DispatchQueue::DispatchThread::entry()+0x11) [0x7fa2c001ee81] 2024-03-24T00:23:30.463 INFO:journalctl@ceph.mds.d.smithi118.stdout:Mar 24 00:23:30 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-d[95429]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7fa2bea391ca] 2024-03-24T00:23:30.464 INFO:journalctl@ceph.mds.d.smithi118.stdout:Mar 24 00:23:30 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-d[95429]: 16: clone() 2024-03-24T00:23:30.464 INFO:journalctl@ceph.mds.d.smithi118.stdout:Mar 24 00:23:30 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-d[95429]: 2024-03-24T00:23:30.464 INFO:journalctl@ceph.mds.d.smithi118.stdout:Mar 24 00:23:30 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-d[95429]: 0> 2024-03-24T00:23:30.068+0000 7fa2b4400700 -1 *** Caught signal (Aborted) ** 2024-03-24T00:23:30.464 INFO:journalctl@ceph.mds.d.smithi118.stdout:Mar 24 00:23:30 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-d[95429]: in thread 7fa2b4400700 thread_name:ms_dispatch 2024-03-24T00:23:30.464 INFO:journalctl@ceph.mds.d.smithi118.stdout:Mar 24 00:23:30 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-d[95429]: 2024-03-24T00:23:30.464 INFO:journalctl@ceph.mds.d.smithi118.stdout:Mar 24 00:23:30 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-d[95429]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-24T00:23:30.464 INFO:journalctl@ceph.mds.d.smithi118.stdout:Mar 24 00:23:30 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-d[95429]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7fa2bea43d20] 2024-03-24T00:23:30.464 INFO:journalctl@ceph.mds.d.smithi118.stdout:Mar 24 00:23:30 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-d[95429]: 2: gsignal() 2024-03-24T00:23:30.464 INFO:journalctl@ceph.mds.d.smithi118.stdout:Mar 24 00:23:30 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-d[95429]: 3: abort() 2024-03-24T00:23:30.464 INFO:journalctl@ceph.mds.d.smithi118.stdout:Mar 24 00:23:30 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-d[95429]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x18f) [0x7fa2bfcbc4fb] 2024-03-24T00:23:30.464 INFO:journalctl@ceph.mds.d.smithi118.stdout:Mar 24 00:23:30 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-d[95429]: 5: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7fa2bfcbc667] 2024-03-24T00:23:30.464 INFO:journalctl@ceph.mds.d.smithi118.stdout:Mar 24 00:23:30 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-d[95429]: 6: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x55e7e2b5e223] 2024-03-24T00:23:30.464 INFO:journalctl@ceph.mds.d.smithi118.stdout:Mar 24 00:23:30 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-d[95429]: 7: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x55e7e29d85d6] 2024-03-24T00:23:30.465 INFO:journalctl@ceph.mds.g.smithi118.stdout:Mar 24 00:23:30 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-g[95433]: 3: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x5589c9ab3223] 2024-03-24T00:23:30.465 INFO:journalctl@ceph.mds.g.smithi118.stdout:Mar 24 00:23:30 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-g[95433]: 4: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x5589c992d5d6] 2024-03-24T00:23:30.465 INFO:journalctl@ceph.mds.g.smithi118.stdout:Mar 24 00:23:30 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-g[95433]: 5: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x5589c992dda1] 2024-03-24T00:23:30.465 INFO:journalctl@ceph.mds.g.smithi118.stdout:Mar 24 00:23:30 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-g[95433]: 6: (Server::handle_client_rename(boost::intrusive_ptr const&)+0x259d) [0x5589c996a97d] 2024-03-24T00:23:30.465 INFO:journalctl@ceph.mds.g.smithi118.stdout:Mar 24 00:23:30 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-g[95433]: 7: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x5589c9980233] 2024-03-24T00:23:30.465 INFO:journalctl@ceph.mds.g.smithi118.stdout:Mar 24 00:23:30 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-g[95433]: 8: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x5589c9982197] 2024-03-24T00:23:30.465 INFO:journalctl@ceph.mds.g.smithi118.stdout:Mar 24 00:23:30 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-g[95433]: 9: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x5589c988111b] 2024-03-24T00:23:30.465 INFO:journalctl@ceph.mds.g.smithi118.stdout:Mar 24 00:23:30 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-g[95433]: 10: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x5589c98816bc] 2024-03-24T00:23:30.465 INFO:journalctl@ceph.mds.g.smithi118.stdout:Mar 24 00:23:30 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-g[95433]: 11: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x5589c986b1ff] 2024-03-24T00:23:30.465 INFO:journalctl@ceph.mds.g.smithi118.stdout:Mar 24 00:23:30 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-g[95433]: 12: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7f7c6bac3488] 2024-03-24T00:23:30.465 INFO:journalctl@ceph.mds.g.smithi118.stdout:Mar 24 00:23:30 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-g[95433]: 13: (DispatchQueue::entry()+0x51f) [0x7f7c6bac060f] 2024-03-24T00:23:30.465 INFO:journalctl@ceph.mds.g.smithi118.stdout:Mar 24 00:23:30 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-g[95433]: 14: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f7c6bb93e81] 2024-03-24T00:23:30.465 INFO:journalctl@ceph.mds.g.smithi118.stdout:Mar 24 00:23:30 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-g[95433]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7f7c6a5ae1ca] 2024-03-24T00:23:30.466 INFO:journalctl@ceph.mds.g.smithi118.stdout:Mar 24 00:23:30 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-g[95433]: 16: clone() 2024-03-24T00:23:30.466 INFO:journalctl@ceph.mds.g.smithi118.stdout:Mar 24 00:23:30 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-g[95433]: 2024-03-24T00:23:30.466 INFO:journalctl@ceph.mds.g.smithi118.stdout:Mar 24 00:23:30 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-g[95433]: 0> 2024-03-24T00:23:30.015+0000 7f7c60000700 -1 *** Caught signal (Aborted) ** 2024-03-24T00:23:30.466 INFO:journalctl@ceph.mds.g.smithi118.stdout:Mar 24 00:23:30 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-g[95433]: in thread 7f7c60000700 thread_name:ms_dispatch 2024-03-24T00:23:30.466 INFO:journalctl@ceph.mds.g.smithi118.stdout:Mar 24 00:23:30 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-g[95433]: 2024-03-24T00:23:30.466 INFO:journalctl@ceph.mds.g.smithi118.stdout:Mar 24 00:23:30 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-g[95433]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-24T00:23:30.466 INFO:journalctl@ceph.mds.g.smithi118.stdout:Mar 24 00:23:30 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-g[95433]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f7c6a5b8d20] 2024-03-24T00:23:30.466 INFO:journalctl@ceph.mds.g.smithi118.stdout:Mar 24 00:23:30 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-g[95433]: 2: gsignal() 2024-03-24T00:23:30.466 INFO:journalctl@ceph.mds.g.smithi118.stdout:Mar 24 00:23:30 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-g[95433]: 3: abort() 2024-03-24T00:23:30.466 INFO:journalctl@ceph.mds.g.smithi118.stdout:Mar 24 00:23:30 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-g[95433]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x18f) [0x7f7c6b8314fb] 2024-03-24T00:23:30.466 INFO:journalctl@ceph.mds.g.smithi118.stdout:Mar 24 00:23:30 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-g[95433]: 5: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7f7c6b831667] 2024-03-24T00:23:30.466 INFO:journalctl@ceph.mds.g.smithi118.stdout:Mar 24 00:23:30 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-g[95433]: 6: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x5589c9ab3223] 2024-03-24T00:23:30.466 INFO:journalctl@ceph.mds.g.smithi118.stdout:Mar 24 00:23:30 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-g[95433]: 7: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x5589c992d5d6] 2024-03-24T00:23:30.467 INFO:journalctl@ceph.mds.d.smithi118.stdout:Mar 24 00:23:30 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-d[95429]: 8: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x55e7e29d8da1] 2024-03-24T00:23:30.467 INFO:journalctl@ceph.mds.d.smithi118.stdout:Mar 24 00:23:30 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-d[95429]: 9: (Server::handle_client_rename(boost::intrusive_ptr const&)+0x259d) [0x55e7e2a1597d] 2024-03-24T00:23:30.467 INFO:journalctl@ceph.mds.d.smithi118.stdout:Mar 24 00:23:30 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-d[95429]: 10: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x55e7e2a2b233] 2024-03-24T00:23:30.467 INFO:journalctl@ceph.mds.d.smithi118.stdout:Mar 24 00:23:30 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-d[95429]: 11: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x55e7e2a2d197] 2024-03-24T00:23:30.467 INFO:journalctl@ceph.mds.d.smithi118.stdout:Mar 24 00:23:30 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-d[95429]: 12: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x55e7e292c11b] 2024-03-24T00:23:30.467 INFO:journalctl@ceph.mds.d.smithi118.stdout:Mar 24 00:23:30 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-d[95429]: 13: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x55e7e292c6bc] 2024-03-24T00:23:30.467 INFO:journalctl@ceph.mds.d.smithi118.stdout:Mar 24 00:23:30 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-d[95429]: 14: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x55e7e29161ff] 2024-03-24T00:23:30.467 INFO:journalctl@ceph.mds.d.smithi118.stdout:Mar 24 00:23:30 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-d[95429]: 15: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7fa2bff4e488] 2024-03-24T00:23:30.467 INFO:journalctl@ceph.mds.d.smithi118.stdout:Mar 24 00:23:30 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-d[95429]: 16: (DispatchQueue::entry()+0x51f) [0x7fa2bff4b60f] 2024-03-24T00:23:30.467 INFO:journalctl@ceph.mds.d.smithi118.stdout:Mar 24 00:23:30 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-d[95429]: 17: (DispatchQueue::DispatchThread::entry()+0x11) [0x7fa2c001ee81] 2024-03-24T00:23:30.467 INFO:journalctl@ceph.mds.d.smithi118.stdout:Mar 24 00:23:30 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-d[95429]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7fa2bea391ca] 2024-03-24T00:23:30.467 INFO:journalctl@ceph.mds.d.smithi118.stdout:Mar 24 00:23:30 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-d[95429]: 19: clone() 2024-03-24T00:23:30.467 INFO:journalctl@ceph.mds.d.smithi118.stdout:Mar 24 00:23:30 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-d[95429]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-03-24T00:23:30.467 INFO:journalctl@ceph.mds.d.smithi118.stdout:Mar 24 00:23:30 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-d[95429]: 2024-03-24T00:23:30.468 INFO:journalctl@ceph.mds.d.smithi118.stdout:Mar 24 00:23:30 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-d[95429]: -5> 2024-03-24T00:23:30.065+0000 7fa2b4400700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: In function 'void Locker::set_xlocks_done(MutationImpl*, bool)' thread 7fa2b4400700 time 2024-03-24T00:23:30.064151+0000 2024-03-24T00:23:30.468 INFO:journalctl@ceph.mds.d.smithi118.stdout:Mar 24 00:23:30 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-d[95429]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: 626: FAILED ceph_assert(obj->is_auth()) 2024-03-24T00:23:30.468 INFO:journalctl@ceph.mds.d.smithi118.stdout:Mar 24 00:23:30 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-d[95429]: 2024-03-24T00:23:30.468 INFO:journalctl@ceph.mds.d.smithi118.stdout:Mar 24 00:23:30 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-d[95429]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-24T00:23:30.468 INFO:journalctl@ceph.mds.d.smithi118.stdout:Mar 24 00:23:30 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-d[95429]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x135) [0x7fa2bfcbc4a1] 2024-03-24T00:23:30.468 INFO:journalctl@ceph.mds.d.smithi118.stdout:Mar 24 00:23:30 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-d[95429]: 2: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7fa2bfcbc667] 2024-03-24T00:23:30.468 INFO:journalctl@ceph.mds.d.smithi118.stdout:Mar 24 00:23:30 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-d[95429]: 3: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x55e7e2b5e223] 2024-03-24T00:23:30.468 INFO:journalctl@ceph.mds.d.smithi118.stdout:Mar 24 00:23:30 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-d[95429]: 4: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x55e7e29d85d6] 2024-03-24T00:23:30.468 INFO:journalctl@ceph.mds.g.smithi118.stdout:Mar 24 00:23:30 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-g[95433]: 8: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x5589c992dda1] 2024-03-24T00:23:30.468 INFO:journalctl@ceph.mds.g.smithi118.stdout:Mar 24 00:23:30 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-g[95433]: 9: (Server::handle_client_rename(boost::intrusive_ptr const&)+0x259d) [0x5589c996a97d] 2024-03-24T00:23:30.468 INFO:journalctl@ceph.mds.g.smithi118.stdout:Mar 24 00:23:30 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-g[95433]: 10: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x5589c9980233] 2024-03-24T00:23:30.468 INFO:journalctl@ceph.mds.g.smithi118.stdout:Mar 24 00:23:30 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-g[95433]: 11: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x5589c9982197] 2024-03-24T00:23:30.469 INFO:journalctl@ceph.mds.g.smithi118.stdout:Mar 24 00:23:30 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-g[95433]: 12: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x5589c988111b] 2024-03-24T00:23:30.469 INFO:journalctl@ceph.mds.g.smithi118.stdout:Mar 24 00:23:30 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-g[95433]: 13: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x5589c98816bc] 2024-03-24T00:23:30.469 INFO:journalctl@ceph.mds.g.smithi118.stdout:Mar 24 00:23:30 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-g[95433]: 14: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x5589c986b1ff] 2024-03-24T00:23:30.469 INFO:journalctl@ceph.mds.g.smithi118.stdout:Mar 24 00:23:30 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-g[95433]: 15: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7f7c6bac3488] 2024-03-24T00:23:30.469 INFO:journalctl@ceph.mds.g.smithi118.stdout:Mar 24 00:23:30 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-g[95433]: 16: (DispatchQueue::entry()+0x51f) [0x7f7c6bac060f] 2024-03-24T00:23:30.469 INFO:journalctl@ceph.mds.g.smithi118.stdout:Mar 24 00:23:30 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-g[95433]: 17: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f7c6bb93e81] 2024-03-24T00:23:30.469 INFO:journalctl@ceph.mds.g.smithi118.stdout:Mar 24 00:23:30 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-g[95433]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7f7c6a5ae1ca] 2024-03-24T00:23:30.469 INFO:journalctl@ceph.mds.g.smithi118.stdout:Mar 24 00:23:30 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-g[95433]: 19: clone() 2024-03-24T00:23:30.469 INFO:journalctl@ceph.mds.g.smithi118.stdout:Mar 24 00:23:30 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-g[95433]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-03-24T00:23:30.469 INFO:journalctl@ceph.mds.g.smithi118.stdout:Mar 24 00:23:30 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-g[95433]: 2024-03-24T00:23:30.469 INFO:journalctl@ceph.mds.g.smithi118.stdout:Mar 24 00:23:30 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-g[95433]: -1> 2024-03-24T00:23:30.012+0000 7f7c60000700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: In function 'void Locker::set_xlocks_done(MutationImpl*, bool)' thread 7f7c60000700 time 2024-03-24T00:23:30.010332+0000 2024-03-24T00:23:30.469 INFO:journalctl@ceph.mds.g.smithi118.stdout:Mar 24 00:23:30 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-g[95433]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: 626: FAILED ceph_assert(obj->is_auth()) 2024-03-24T00:23:30.469 INFO:journalctl@ceph.mds.g.smithi118.stdout:Mar 24 00:23:30 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-g[95433]: 2024-03-24T00:23:30.469 INFO:journalctl@ceph.mds.g.smithi118.stdout:Mar 24 00:23:30 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-g[95433]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-24T00:23:30.469 INFO:journalctl@ceph.mds.g.smithi118.stdout:Mar 24 00:23:30 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-g[95433]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x135) [0x7f7c6b8314a1] 2024-03-24T00:23:30.469 INFO:journalctl@ceph.mds.g.smithi118.stdout:Mar 24 00:23:30 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-g[95433]: 2: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7f7c6b831667] 2024-03-24T00:23:30.470 INFO:journalctl@ceph.mds.g.smithi118.stdout:Mar 24 00:23:30 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-g[95433]: 3: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x5589c9ab3223] 2024-03-24T00:23:30.470 INFO:journalctl@ceph.mds.g.smithi118.stdout:Mar 24 00:23:30 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-g[95433]: 4: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x5589c992d5d6] 2024-03-24T00:23:30.470 INFO:journalctl@ceph.mds.d.smithi118.stdout:Mar 24 00:23:30 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-d[95429]: 5: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x55e7e29d8da1] 2024-03-24T00:23:30.470 INFO:journalctl@ceph.mds.d.smithi118.stdout:Mar 24 00:23:30 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-d[95429]: 6: (Server::handle_client_rename(boost::intrusive_ptr const&)+0x259d) [0x55e7e2a1597d] 2024-03-24T00:23:30.470 INFO:journalctl@ceph.mds.d.smithi118.stdout:Mar 24 00:23:30 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-d[95429]: 7: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x55e7e2a2b233] 2024-03-24T00:23:30.470 INFO:journalctl@ceph.mds.d.smithi118.stdout:Mar 24 00:23:30 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-d[95429]: 8: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x55e7e2a2d197] 2024-03-24T00:23:30.470 INFO:journalctl@ceph.mds.d.smithi118.stdout:Mar 24 00:23:30 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-d[95429]: 9: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x55e7e292c11b] 2024-03-24T00:23:30.470 INFO:journalctl@ceph.mds.d.smithi118.stdout:Mar 24 00:23:30 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-d[95429]: 10: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x55e7e292c6bc] 2024-03-24T00:23:30.470 INFO:journalctl@ceph.mds.d.smithi118.stdout:Mar 24 00:23:30 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-d[95429]: 11: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x55e7e29161ff] 2024-03-24T00:23:30.471 INFO:journalctl@ceph.mds.d.smithi118.stdout:Mar 24 00:23:30 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-d[95429]: 12: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7fa2bff4e488] 2024-03-24T00:23:30.471 INFO:journalctl@ceph.mds.d.smithi118.stdout:Mar 24 00:23:30 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-d[95429]: 13: (DispatchQueue::entry()+0x51f) [0x7fa2bff4b60f] 2024-03-24T00:23:30.471 INFO:journalctl@ceph.mds.d.smithi118.stdout:Mar 24 00:23:30 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-d[95429]: 14: (DispatchQueue::DispatchThread::entry()+0x11) [0x7fa2c001ee81] 2024-03-24T00:23:30.471 INFO:journalctl@ceph.mds.d.smithi118.stdout:Mar 24 00:23:30 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-d[95429]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7fa2bea391ca] 2024-03-24T00:23:30.471 INFO:journalctl@ceph.mds.d.smithi118.stdout:Mar 24 00:23:30 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-d[95429]: 16: clone() 2024-03-24T00:23:30.471 INFO:journalctl@ceph.mds.d.smithi118.stdout:Mar 24 00:23:30 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-d[95429]: 2024-03-24T00:23:30.471 INFO:journalctl@ceph.mds.d.smithi118.stdout:Mar 24 00:23:30 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-d[95429]: -4> 2024-03-24T00:23:30.068+0000 7fa2b4400700 -1 *** Caught signal (Aborted) ** 2024-03-24T00:23:30.471 INFO:journalctl@ceph.mds.d.smithi118.stdout:Mar 24 00:23:30 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-d[95429]: in thread 7fa2b4400700 thread_name:ms_dispatch 2024-03-24T00:23:30.471 INFO:journalctl@ceph.mds.d.smithi118.stdout:Mar 24 00:23:30 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-d[95429]: 2024-03-24T00:23:30.471 INFO:journalctl@ceph.mds.d.smithi118.stdout:Mar 24 00:23:30 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-d[95429]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-24T00:23:30.471 INFO:journalctl@ceph.mds.d.smithi118.stdout:Mar 24 00:23:30 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-d[95429]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7fa2bea43d20] 2024-03-24T00:23:30.471 INFO:journalctl@ceph.mds.d.smithi118.stdout:Mar 24 00:23:30 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-d[95429]: 2: gsignal() 2024-03-24T00:23:30.471 INFO:journalctl@ceph.mds.d.smithi118.stdout:Mar 24 00:23:30 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-d[95429]: 3: abort() 2024-03-24T00:23:30.471 INFO:journalctl@ceph.mds.d.smithi118.stdout:Mar 24 00:23:30 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-d[95429]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x18f) [0x7fa2bfcbc4fb] 2024-03-24T00:23:30.471 INFO:journalctl@ceph.mds.d.smithi118.stdout:Mar 24 00:23:30 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-d[95429]: 5: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7fa2bfcbc667] 2024-03-24T00:23:30.472 INFO:journalctl@ceph.mds.d.smithi118.stdout:Mar 24 00:23:30 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-d[95429]: 6: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x55e7e2b5e223] 2024-03-24T00:23:30.472 INFO:journalctl@ceph.mds.d.smithi118.stdout:Mar 24 00:23:30 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-d[95429]: 7: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x55e7e29d85d6] 2024-03-24T00:23:30.472 INFO:journalctl@ceph.mds.d.smithi118.stdout:Mar 24 00:23:30 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-d[95429]: 8: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x55e7e29d8da1] 2024-03-24T00:23:30.472 INFO:journalctl@ceph.mds.d.smithi118.stdout:Mar 24 00:23:30 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-d[95429]: 9: (Server::handle_client_rename(boost::intrusive_ptr const&)+0x259d) [0x55e7e2a1597d] 2024-03-24T00:23:30.472 INFO:journalctl@ceph.mds.d.smithi118.stdout:Mar 24 00:23:30 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-d[95429]: 10: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x55e7e2a2b233] 2024-03-24T00:23:30.472 INFO:journalctl@ceph.mds.g.smithi118.stdout:Mar 24 00:23:30 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-g[95433]: 5: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x5589c992dda1] 2024-03-24T00:23:30.472 INFO:journalctl@ceph.mds.g.smithi118.stdout:Mar 24 00:23:30 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-g[95433]: 6: (Server::handle_client_rename(boost::intrusive_ptr const&)+0x259d) [0x5589c996a97d] 2024-03-24T00:23:30.472 INFO:journalctl@ceph.mds.g.smithi118.stdout:Mar 24 00:23:30 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-g[95433]: 7: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x5589c9980233] 2024-03-24T00:23:30.473 INFO:journalctl@ceph.mds.g.smithi118.stdout:Mar 24 00:23:30 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-g[95433]: 8: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x5589c9982197] 2024-03-24T00:23:30.473 INFO:journalctl@ceph.mds.g.smithi118.stdout:Mar 24 00:23:30 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-g[95433]: 9: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x5589c988111b] 2024-03-24T00:23:30.473 INFO:journalctl@ceph.mds.g.smithi118.stdout:Mar 24 00:23:30 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-g[95433]: 10: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x5589c98816bc] 2024-03-24T00:23:30.473 INFO:journalctl@ceph.mds.g.smithi118.stdout:Mar 24 00:23:30 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-g[95433]: 11: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x5589c986b1ff] 2024-03-24T00:23:30.473 INFO:journalctl@ceph.mds.g.smithi118.stdout:Mar 24 00:23:30 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-g[95433]: 12: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7f7c6bac3488] 2024-03-24T00:23:30.473 INFO:journalctl@ceph.mds.g.smithi118.stdout:Mar 24 00:23:30 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-g[95433]: 13: (DispatchQueue::entry()+0x51f) [0x7f7c6bac060f] 2024-03-24T00:23:30.473 INFO:journalctl@ceph.mds.g.smithi118.stdout:Mar 24 00:23:30 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-g[95433]: 14: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f7c6bb93e81] 2024-03-24T00:23:30.473 INFO:journalctl@ceph.mds.g.smithi118.stdout:Mar 24 00:23:30 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-g[95433]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7f7c6a5ae1ca] 2024-03-24T00:23:30.473 INFO:journalctl@ceph.mds.g.smithi118.stdout:Mar 24 00:23:30 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-g[95433]: 16: clone() 2024-03-24T00:23:30.473 INFO:journalctl@ceph.mds.g.smithi118.stdout:Mar 24 00:23:30 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-g[95433]: 2024-03-24T00:23:30.473 INFO:journalctl@ceph.mds.g.smithi118.stdout:Mar 24 00:23:30 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-g[95433]: 0> 2024-03-24T00:23:30.015+0000 7f7c60000700 -1 *** Caught signal (Aborted) ** 2024-03-24T00:23:30.473 INFO:journalctl@ceph.mds.g.smithi118.stdout:Mar 24 00:23:30 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-g[95433]: in thread 7f7c60000700 thread_name:ms_dispatch 2024-03-24T00:23:30.473 INFO:journalctl@ceph.mds.g.smithi118.stdout:Mar 24 00:23:30 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-g[95433]: 2024-03-24T00:23:30.473 INFO:journalctl@ceph.mds.g.smithi118.stdout:Mar 24 00:23:30 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-g[95433]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-24T00:23:30.474 INFO:journalctl@ceph.mds.g.smithi118.stdout:Mar 24 00:23:30 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-g[95433]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f7c6a5b8d20] 2024-03-24T00:23:30.474 INFO:journalctl@ceph.mds.g.smithi118.stdout:Mar 24 00:23:30 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-g[95433]: 2: gsignal() 2024-03-24T00:23:30.474 INFO:journalctl@ceph.mds.g.smithi118.stdout:Mar 24 00:23:30 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-g[95433]: 3: abort() 2024-03-24T00:23:30.474 INFO:journalctl@ceph.mds.g.smithi118.stdout:Mar 24 00:23:30 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-g[95433]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x18f) [0x7f7c6b8314fb] 2024-03-24T00:23:30.474 INFO:journalctl@ceph.mds.g.smithi118.stdout:Mar 24 00:23:30 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-g[95433]: 5: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7f7c6b831667] 2024-03-24T00:23:30.474 INFO:journalctl@ceph.mds.g.smithi118.stdout:Mar 24 00:23:30 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-g[95433]: 6: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x5589c9ab3223] 2024-03-24T00:23:30.474 INFO:journalctl@ceph.mds.g.smithi118.stdout:Mar 24 00:23:30 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-g[95433]: 7: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x5589c992d5d6] 2024-03-24T00:23:30.474 INFO:journalctl@ceph.mds.g.smithi118.stdout:Mar 24 00:23:30 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-g[95433]: 8: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x5589c992dda1] 2024-03-24T00:23:30.474 INFO:journalctl@ceph.mds.g.smithi118.stdout:Mar 24 00:23:30 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-g[95433]: 9: (Server::handle_client_rename(boost::intrusive_ptr const&)+0x259d) [0x5589c996a97d] 2024-03-24T00:23:30.474 INFO:journalctl@ceph.mds.g.smithi118.stdout:Mar 24 00:23:30 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-g[95433]: 10: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x5589c9980233] 2024-03-24T00:23:30.474 INFO:journalctl@ceph.mds.d.smithi118.stdout:Mar 24 00:23:30 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-d[95429]: 11: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x55e7e2a2d197] 2024-03-24T00:23:30.475 INFO:journalctl@ceph.mds.d.smithi118.stdout:Mar 24 00:23:30 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-d[95429]: 12: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x55e7e292c11b] 2024-03-24T00:23:30.475 INFO:journalctl@ceph.mds.d.smithi118.stdout:Mar 24 00:23:30 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-d[95429]: 13: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x55e7e292c6bc] 2024-03-24T00:23:30.475 INFO:journalctl@ceph.mds.d.smithi118.stdout:Mar 24 00:23:30 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-d[95429]: 14: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x55e7e29161ff] 2024-03-24T00:23:30.475 INFO:journalctl@ceph.mds.d.smithi118.stdout:Mar 24 00:23:30 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-d[95429]: 15: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7fa2bff4e488] 2024-03-24T00:23:30.475 INFO:journalctl@ceph.mds.d.smithi118.stdout:Mar 24 00:23:30 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-d[95429]: 16: (DispatchQueue::entry()+0x51f) [0x7fa2bff4b60f] 2024-03-24T00:23:30.475 INFO:journalctl@ceph.mds.d.smithi118.stdout:Mar 24 00:23:30 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-d[95429]: 17: (DispatchQueue::DispatchThread::entry()+0x11) [0x7fa2c001ee81] 2024-03-24T00:23:30.475 INFO:journalctl@ceph.mds.d.smithi118.stdout:Mar 24 00:23:30 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-d[95429]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7fa2bea391ca] 2024-03-24T00:23:30.475 INFO:journalctl@ceph.mds.d.smithi118.stdout:Mar 24 00:23:30 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-d[95429]: 19: clone() 2024-03-24T00:23:30.475 INFO:journalctl@ceph.mds.d.smithi118.stdout:Mar 24 00:23:30 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-d[95429]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-03-24T00:23:30.475 INFO:journalctl@ceph.mds.d.smithi118.stdout:Mar 24 00:23:30 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-d[95429]: 2024-03-24T00:23:30.475 INFO:journalctl@ceph.mds.g.smithi118.stdout:Mar 24 00:23:30 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-g[95433]: 11: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x5589c9982197] 2024-03-24T00:23:30.476 INFO:journalctl@ceph.mds.g.smithi118.stdout:Mar 24 00:23:30 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-g[95433]: 12: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x5589c988111b] 2024-03-24T00:23:30.476 INFO:journalctl@ceph.mds.g.smithi118.stdout:Mar 24 00:23:30 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-g[95433]: 13: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x5589c98816bc] 2024-03-24T00:23:30.476 INFO:journalctl@ceph.mds.g.smithi118.stdout:Mar 24 00:23:30 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-g[95433]: 14: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x5589c986b1ff] 2024-03-24T00:23:30.476 INFO:journalctl@ceph.mds.g.smithi118.stdout:Mar 24 00:23:30 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-g[95433]: 15: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7f7c6bac3488] 2024-03-24T00:23:30.476 INFO:journalctl@ceph.mds.g.smithi118.stdout:Mar 24 00:23:30 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-g[95433]: 16: (DispatchQueue::entry()+0x51f) [0x7f7c6bac060f] 2024-03-24T00:23:30.476 INFO:journalctl@ceph.mds.g.smithi118.stdout:Mar 24 00:23:30 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-g[95433]: 17: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f7c6bb93e81] 2024-03-24T00:23:30.476 INFO:journalctl@ceph.mds.g.smithi118.stdout:Mar 24 00:23:30 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-g[95433]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7f7c6a5ae1ca] 2024-03-24T00:23:30.476 INFO:journalctl@ceph.mds.g.smithi118.stdout:Mar 24 00:23:30 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-g[95433]: 19: clone() 2024-03-24T00:23:30.476 INFO:journalctl@ceph.mds.g.smithi118.stdout:Mar 24 00:23:30 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-g[95433]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-03-24T00:23:30.476 INFO:journalctl@ceph.mds.g.smithi118.stdout:Mar 24 00:23:30 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-g[95433]: 2024-03-24T00:23:30.476 INFO:journalctl@ceph.mds.g.smithi118.stdout:Mar 24 00:23:30 smithi118 podman[95714]: 2024-03-24 00:23:30.138320964 +0000 UTC m=+0.041058595 container died 3a65d19c304e930df858d4b40707683f229feaa70ee0d1e1ca3d5c91b819120f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-g, CEPH_POINT_RELEASE=, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, GIT_BRANCH=HEAD, ceph=True, org.label-schema.build-date=20240319, GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.vendor=CentOS) 2024-03-24T00:23:31.196 INFO:journalctl@ceph.mds.g.smithi118.stdout:Mar 24 00:23:30 smithi118 podman[95714]: 2024-03-24 00:23:30.902738044 +0000 UTC m=+0.805475712 container remove 3a65d19c304e930df858d4b40707683f229feaa70ee0d1e1ca3d5c91b819120f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-g, ceph=True, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240319, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, io.buildah.version=1.33.5) 2024-03-24T00:23:31.196 INFO:journalctl@ceph.mds.g.smithi118.stdout:Mar 24 00:23:30 smithi118 systemd[1]: ceph-d79a5420-e971-11ee-95cd-87774f69a715@mds.g.service: Main process exited, code=exited, status=134/n/a 2024-03-24T00:23:31.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:23:30 smithi118 ceph-mon[31535]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-03-24T00:23:31.197 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:23:30 smithi118 ceph-mon[31535]: Cluster is now healthy 2024-03-24T00:23:31.197 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:23:30 smithi118 ceph-mon[31535]: mds.? [v2:172.21.15.118:6838/282013400,v1:172.21.15.118:6839/282013400] up:active 2024-03-24T00:23:31.197 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:23:30 smithi118 ceph-mon[31535]: mds.? [v2:172.21.15.118:6836/3985972391,v1:172.21.15.118:6837/3985972391] up:active 2024-03-24T00:23:31.197 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:23:30 smithi118 ceph-mon[31535]: fsmap cephfs:5 3 up:standby-replay 2 up:standby 5 up:active 2024-03-24T00:23:31.197 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:23:30 smithi118 ceph-mon[31535]: fsmap cephfs:5 5 up:standby-replay 5 up:active 2024-03-24T00:23:31.197 INFO:journalctl@ceph.mds.d.smithi118.stdout:Mar 24 00:23:30 smithi118 podman[95727]: 2024-03-24 00:23:30.927977295 +0000 UTC m=+0.767166432 container died 274eb342f15de2c333ada17081c1c027f0db79f869b5dda0e9e6b4a0e90f42d8 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-d, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, ceph=True, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, CEPH_POINT_RELEASE=, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.build-date=20240319, io.buildah.version=1.33.5) 2024-03-24T00:23:31.204 INFO:journalctl@ceph.mds.b.smithi189.stdout:Mar 24 00:23:30 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-b[66975]: 2024-03-24T00:23:30.739+0000 7f4e07ab9700 -1 mds.pinger is_rank_lagging: rank=1 was never sent ping request. 2024-03-24T00:23:31.204 INFO:journalctl@ceph.mds.b.smithi189.stdout:Mar 24 00:23:30 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-b[66975]: 2024-03-24T00:23:30.739+0000 7f4e07ab9700 -1 mds.pinger is_rank_lagging: rank=3 was never sent ping request. 2024-03-24T00:23:31.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:23:30 smithi189 ceph-mon[36187]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-03-24T00:23:31.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:23:30 smithi189 ceph-mon[36187]: Cluster is now healthy 2024-03-24T00:23:31.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:23:30 smithi189 ceph-mon[36187]: mds.? [v2:172.21.15.118:6838/282013400,v1:172.21.15.118:6839/282013400] up:active 2024-03-24T00:23:31.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:23:30 smithi189 ceph-mon[36187]: mds.? [v2:172.21.15.118:6836/3985972391,v1:172.21.15.118:6837/3985972391] up:active 2024-03-24T00:23:31.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:23:30 smithi189 ceph-mon[36187]: fsmap cephfs:5 3 up:standby-replay 2 up:standby 5 up:active 2024-03-24T00:23:31.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:23:30 smithi189 ceph-mon[36187]: fsmap cephfs:5 5 up:standby-replay 5 up:active 2024-03-24T00:23:31.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:23:30 smithi193 ceph-mon[36878]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-03-24T00:23:31.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:23:30 smithi193 ceph-mon[36878]: Cluster is now healthy 2024-03-24T00:23:31.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:23:30 smithi193 ceph-mon[36878]: mds.? [v2:172.21.15.118:6838/282013400,v1:172.21.15.118:6839/282013400] up:active 2024-03-24T00:23:31.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:23:30 smithi193 ceph-mon[36878]: mds.? [v2:172.21.15.118:6836/3985972391,v1:172.21.15.118:6837/3985972391] up:active 2024-03-24T00:23:31.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:23:30 smithi193 ceph-mon[36878]: fsmap cephfs:5 3 up:standby-replay 2 up:standby 5 up:active 2024-03-24T00:23:31.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:23:30 smithi193 ceph-mon[36878]: fsmap cephfs:5 5 up:standby-replay 5 up:active 2024-03-24T00:23:31.917 INFO:journalctl@ceph.mds.d.smithi118.stdout:Mar 24 00:23:31 smithi118 podman[95727]: 2024-03-24 00:23:31.649121744 +0000 UTC m=+1.488310897 container remove 274eb342f15de2c333ada17081c1c027f0db79f869b5dda0e9e6b4a0e90f42d8 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-d, org.label-schema.schema-version=1.0, GIT_CLEAN=True, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240319, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_BRANCH=HEAD, io.buildah.version=1.33.5) 2024-03-24T00:23:31.917 INFO:journalctl@ceph.mds.d.smithi118.stdout:Mar 24 00:23:31 smithi118 systemd[1]: ceph-d79a5420-e971-11ee-95cd-87774f69a715@mds.d.service: Main process exited, code=exited, status=134/n/a 2024-03-24T00:23:32.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:23:31 smithi118 ceph-mon[31535]: pgmap v657: 97 pgs: 97 active+clean; 150 MiB data, 853 MiB used, 1.0 TiB / 1.0 TiB avail; 160 KiB/s rd, 282 KiB/s wr, 72 op/s 2024-03-24T00:23:32.196 INFO:journalctl@ceph.mds.d.smithi118.stdout:Mar 24 00:23:32 smithi118 systemd[1]: ceph-d79a5420-e971-11ee-95cd-87774f69a715@mds.d.service: Failed with result 'exit-code'. 2024-03-24T00:23:32.196 INFO:journalctl@ceph.mds.d.smithi118.stdout:Mar 24 00:23:32 smithi118 systemd[1]: ceph-d79a5420-e971-11ee-95cd-87774f69a715@mds.d.service: Consumed 1.721s CPU time. 2024-03-24T00:23:32.197 INFO:journalctl@ceph.mds.g.smithi118.stdout:Mar 24 00:23:31 smithi118 systemd[1]: ceph-d79a5420-e971-11ee-95cd-87774f69a715@mds.g.service: Failed with result 'exit-code'. 2024-03-24T00:23:32.197 INFO:journalctl@ceph.mds.g.smithi118.stdout:Mar 24 00:23:31 smithi118 systemd[1]: ceph-d79a5420-e971-11ee-95cd-87774f69a715@mds.g.service: Consumed 1.580s CPU time. 2024-03-24T00:23:32.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:23:31 smithi193 ceph-mon[36878]: pgmap v657: 97 pgs: 97 active+clean; 150 MiB data, 853 MiB used, 1.0 TiB / 1.0 TiB avail; 160 KiB/s rd, 282 KiB/s wr, 72 op/s 2024-03-24T00:23:32.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:23:31 smithi189 ceph-mon[36187]: pgmap v657: 97 pgs: 97 active+clean; 150 MiB data, 853 MiB used, 1.0 TiB / 1.0 TiB avail; 160 KiB/s rd, 282 KiB/s wr, 72 op/s 2024-03-24T00:23:34.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:23:33 smithi118 ceph-mon[31535]: pgmap v658: 97 pgs: 97 active+clean; 146 MiB data, 849 MiB used, 1.0 TiB / 1.0 TiB avail; 370 KiB/s rd, 157 KiB/s wr, 85 op/s 2024-03-24T00:23:34.203 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:23:33 smithi189 ceph-mon[36187]: pgmap v658: 97 pgs: 97 active+clean; 146 MiB data, 849 MiB used, 1.0 TiB / 1.0 TiB avail; 370 KiB/s rd, 157 KiB/s wr, 85 op/s 2024-03-24T00:23:34.213 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:23:33 smithi193 ceph-mon[36878]: pgmap v658: 97 pgs: 97 active+clean; 146 MiB data, 849 MiB used, 1.0 TiB / 1.0 TiB avail; 370 KiB/s rd, 157 KiB/s wr, 85 op/s 2024-03-24T00:23:36.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:23:35 smithi118 ceph-mon[31535]: pgmap v659: 97 pgs: 97 active+clean; 146 MiB data, 849 MiB used, 1.0 TiB / 1.0 TiB avail; 437 KiB/s rd, 146 KiB/s wr, 95 op/s 2024-03-24T00:23:36.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:23:35 smithi189 ceph-mon[36187]: pgmap v659: 97 pgs: 97 active+clean; 146 MiB data, 849 MiB used, 1.0 TiB / 1.0 TiB avail; 437 KiB/s rd, 146 KiB/s wr, 95 op/s 2024-03-24T00:23:36.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:23:35 smithi193 ceph-mon[36878]: pgmap v659: 97 pgs: 97 active+clean; 146 MiB data, 849 MiB used, 1.0 TiB / 1.0 TiB avail; 437 KiB/s rd, 146 KiB/s wr, 95 op/s 2024-03-24T00:23:38.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:23:37 smithi118 ceph-mon[31535]: pgmap v660: 97 pgs: 97 active+clean; 146 MiB data, 849 MiB used, 1.0 TiB / 1.0 TiB avail; 378 KiB/s rd, 97 KiB/s wr, 81 op/s 2024-03-24T00:23:38.203 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:23:37 smithi189 ceph-mon[36187]: pgmap v660: 97 pgs: 97 active+clean; 146 MiB data, 849 MiB used, 1.0 TiB / 1.0 TiB avail; 378 KiB/s rd, 97 KiB/s wr, 81 op/s 2024-03-24T00:23:38.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:23:37 smithi193 ceph-mon[36878]: pgmap v660: 97 pgs: 97 active+clean; 146 MiB data, 849 MiB used, 1.0 TiB / 1.0 TiB avail; 378 KiB/s rd, 97 KiB/s wr, 81 op/s 2024-03-24T00:23:40.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:23:39 smithi118 ceph-mon[31535]: pgmap v661: 97 pgs: 97 active+clean; 146 MiB data, 849 MiB used, 1.0 TiB / 1.0 TiB avail; 378 KiB/s rd, 34 KiB/s wr, 65 op/s 2024-03-24T00:23:40.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:23:39 smithi189 ceph-mon[36187]: pgmap v661: 97 pgs: 97 active+clean; 146 MiB data, 849 MiB used, 1.0 TiB / 1.0 TiB avail; 378 KiB/s rd, 34 KiB/s wr, 65 op/s 2024-03-24T00:23:40.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:23:39 smithi193 ceph-mon[36878]: pgmap v661: 97 pgs: 97 active+clean; 146 MiB data, 849 MiB used, 1.0 TiB / 1.0 TiB avail; 378 KiB/s rd, 34 KiB/s wr, 65 op/s 2024-03-24T00:23:42.163 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:23:42.163+0000 7fda74a00640 1 -- 172.21.15.118:0/375317284 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fda70104ae0 msgr2=0x7fda70104f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:23:42.163 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:23:42.163+0000 7fda74a00640 1 --2- 172.21.15.118:0/375317284 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fda70104ae0 0x7fda70104f60 secure :-1 s=READY pgs=450 cs=0 l=1 rev1=1 crypto rx=0x7fda58009900 tx=0x7fda5802bc30 comp rx=0 tx=0).stop 2024-03-24T00:23:42.163 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:23:42.163+0000 7fda74a00640 1 -- 172.21.15.118:0/375317284 shutdown_connections 2024-03-24T00:23:42.163 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:23:42.163+0000 7fda74a00640 1 --2- 172.21.15.118:0/375317284 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fda701054a0 0x7fda7010bd30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:23:42.163 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:23:42.163+0000 7fda74a00640 1 --2- 172.21.15.118:0/375317284 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fda70104ae0 0x7fda70104f60 unknown :-1 s=CLOSED pgs=450 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:23:42.164 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:23:42.163+0000 7fda74a00640 1 --2- 172.21.15.118:0/375317284 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fda701038f0 0x7fda70103d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:23:42.164 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:23:42.163+0000 7fda74a00640 1 -- 172.21.15.118:0/375317284 >> 172.21.15.118:0/375317284 conn(0x7fda700fee60 msgr2=0x7fda701012a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:23:42.164 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:23:42.163+0000 7fda74a00640 1 -- 172.21.15.118:0/375317284 shutdown_connections 2024-03-24T00:23:42.164 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:23:42.163+0000 7fda74a00640 1 -- 172.21.15.118:0/375317284 wait complete. 2024-03-24T00:23:42.166 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:23:42.164+0000 7fda74a00640 1 Processor -- start 2024-03-24T00:23:42.166 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:23:42.164+0000 7fda74a00640 1 -- start start 2024-03-24T00:23:42.166 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:23:42.165+0000 7fda74a00640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fda701038f0 0x7fda70199b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:23:42.166 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:23:42.165+0000 7fda74a00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fda70104ae0 0x7fda7019a070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:23:42.166 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:23:42.165+0000 7fda74a00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fda701054a0 0x7fda701a10e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:23:42.166 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:23:42.165+0000 7fda74a00640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fda7010f5a0 con 0x7fda701038f0 2024-03-24T00:23:42.166 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:23:42.165+0000 7fda74a00640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fda7010f420 con 0x7fda70104ae0 2024-03-24T00:23:42.167 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:23:42.165+0000 7fda74a00640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fda7010f720 con 0x7fda701054a0 2024-03-24T00:23:42.167 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:23:42.165+0000 7fda6de00640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fda701038f0 0x7fda70199b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:23:42.167 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:23:42.165+0000 7fda6de00640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fda701038f0 0x7fda70199b30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.118:3300/0 says I am v2:172.21.15.118:33392/0 (socket says 172.21.15.118:33392) 2024-03-24T00:23:42.167 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:23:42.165+0000 7fda6de00640 1 -- 172.21.15.118:0/1602305341 learned_addr learned my addr 172.21.15.118:0/1602305341 (peer_addr_for_me v2:172.21.15.118:0/0) 2024-03-24T00:23:42.167 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:23:42.165+0000 7fda67e00640 1 --2- 172.21.15.118:0/1602305341 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fda70104ae0 0x7fda7019a070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:23:42.167 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:23:42.165+0000 7fda6e800640 1 --2- 172.21.15.118:0/1602305341 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fda701054a0 0x7fda701a10e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:23:42.167 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:23:42.165+0000 7fda6de00640 1 -- 172.21.15.118:0/1602305341 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fda701054a0 msgr2=0x7fda701a10e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:23:42.167 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:23:42.165+0000 7fda6de00640 1 --2- 172.21.15.118:0/1602305341 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fda701054a0 0x7fda701a10e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:23:42.167 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:23:42.165+0000 7fda6de00640 1 -- 172.21.15.118:0/1602305341 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fda70104ae0 msgr2=0x7fda7019a070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:23:42.167 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:23:42.165+0000 7fda6de00640 1 --2- 172.21.15.118:0/1602305341 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fda70104ae0 0x7fda7019a070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:23:42.167 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:23:42.165+0000 7fda6de00640 1 -- 172.21.15.118:0/1602305341 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fda701a1750 con 0x7fda701038f0 2024-03-24T00:23:42.167 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:23:42.166+0000 7fda6de00640 1 --2- 172.21.15.118:0/1602305341 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fda701038f0 0x7fda70199b30 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7fda6000e9c0 tx=0x7fda6000ee80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:23:42.167 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:23:42.166+0000 7fda66a00640 1 -- 172.21.15.118:0/1602305341 <== mon.0 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fda6000cd80 con 0x7fda701038f0 2024-03-24T00:23:42.168 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:23:42.166+0000 7fda66a00640 1 -- 172.21.15.118:0/1602305341 <== mon.0 v2:172.21.15.118:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fda600035f0 con 0x7fda701038f0 2024-03-24T00:23:42.168 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:23:42.166+0000 7fda74a00640 1 -- 172.21.15.118:0/1602305341 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fda701a1a40 con 0x7fda701038f0 2024-03-24T00:23:42.168 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:23:42.166+0000 7fda66a00640 1 -- 172.21.15.118:0/1602305341 <== mon.0 v2:172.21.15.118:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fda60010690 con 0x7fda701038f0 2024-03-24T00:23:42.168 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:23:42.166+0000 7fda74a00640 1 -- 172.21.15.118:0/1602305341 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fda701a1fb0 con 0x7fda701038f0 2024-03-24T00:23:42.170 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:23:42.169+0000 7fda7782c740 1 Processor -- start 2024-03-24T00:23:42.170 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:23:42.169+0000 7fda7782c740 1 -- start start 2024-03-24T00:23:42.170 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:23:42.169+0000 7fda66a00640 1 -- 172.21.15.118:0/1602305341 <== mon.0 v2:172.21.15.118:3300/0 4 ==== mgrmap(e 20) v1 ==== 137291+0+0 (secure 0 0 0) 0x7fda600037a0 con 0x7fda701038f0 2024-03-24T00:23:42.170 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:23:42.169+0000 7fda7782c740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55df5164b1e0 0x55df5164b600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:23:42.170 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:23:42.169+0000 7fda7782c740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55df5164bbd0 0x55df516506d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:23:42.170 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:23:42.169+0000 7fda7782c740 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55df51650d40 0x55df516531d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:23:42.170 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:23:42.170+0000 7fda7782c740 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x55df5160e3b0 con 0x55df51650d40 2024-03-24T00:23:42.170 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:23:42.170+0000 7fda7782c740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x55df515b0830 con 0x55df5164b1e0 2024-03-24T00:23:42.170 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:23:42.170+0000 7fda7782c740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x55df515595d0 con 0x55df5164bbd0 2024-03-24T00:23:42.170 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:23:42.170+0000 7fda6de00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55df5164bbd0 0x55df516506d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:23:42.171 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:23:42.170+0000 7fda6e800640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55df5164b1e0 0x55df5164b600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:23:42.171 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:23:42.170+0000 7fda6e800640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55df5164b1e0 0x55df5164b600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.118:41346/0 (socket says 172.21.15.118:41346) 2024-03-24T00:23:42.171 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:23:42.170+0000 7fda6de00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55df5164bbd0 0x55df516506d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.118:43938/0 (socket says 172.21.15.118:43938) 2024-03-24T00:23:42.171 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:23:42.170+0000 7fda6e800640 1 -- 172.21.15.118:0/38686105 learned_addr learned my addr 172.21.15.118:0/38686105 (peer_addr_for_me v2:172.21.15.118:0/0) 2024-03-24T00:23:42.171 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:23:42.170+0000 7fda67e00640 1 --2- 172.21.15.118:0/38686105 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55df51650d40 0x55df516531d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:23:42.171 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:23:42.170+0000 7fda6e800640 1 -- 172.21.15.118:0/38686105 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55df5164bbd0 msgr2=0x55df516506d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:23:42.171 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:23:42.170+0000 7fda6e800640 1 --2- 172.21.15.118:0/38686105 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55df5164bbd0 0x55df516506d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:23:42.171 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:23:42.170+0000 7fda6e800640 1 -- 172.21.15.118:0/38686105 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55df51650d40 msgr2=0x55df516531d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:23:42.171 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:23:42.170+0000 7fda6e800640 1 --2- 172.21.15.118:0/38686105 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55df51650d40 0x55df516531d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:23:42.171 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:23:42.170+0000 7fda6e800640 1 -- 172.21.15.118:0/38686105 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55df515712c0 con 0x55df5164b1e0 2024-03-24T00:23:42.175 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:23:42.171+0000 7fda6e800640 1 --2- 172.21.15.118:0/38686105 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55df5164b1e0 0x55df5164b600 secure :-1 s=READY pgs=452 cs=0 l=1 rev1=1 crypto rx=0x7fda68009fc0 tx=0x7fda68010860 comp rx=0 tx=0).ready entity=mon.2 client_cookie=bdb25cf49b230d98 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:23:42.175 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:23:42.171+0000 7fda5f000640 1 -- 172.21.15.118:0/38686105 <== mon.2 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fda68003b70 con 0x55df5164b1e0 2024-03-24T00:23:42.175 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:23:42.171+0000 7fda5f000640 1 -- 172.21.15.118:0/38686105 <== mon.2 v2:172.21.15.189:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fda68003d10 con 0x55df5164b1e0 2024-03-24T00:23:42.176 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:23:42.171+0000 7fda5f000640 1 -- 172.21.15.118:0/38686105 <== mon.2 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fda68011ab0 con 0x55df5164b1e0 2024-03-24T00:23:42.176 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:23:42.171+0000 7fda66a00640 1 --2- 172.21.15.118:0/1602305341 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7fda540a2920 0x7fda540a4de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:23:42.176 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:23:42.171+0000 7fda66a00640 1 -- 172.21.15.118:0/1602305341 <== mon.0 v2:172.21.15.118:3300/0 5 ==== osd_map(114..114 src has 1..114) v4 ==== 10341+0+0 (secure 0 0 0) 0x7fda600cf6e0 con 0x7fda701038f0 2024-03-24T00:23:42.176 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:23:42.171+0000 7fda6de00640 1 --2- 172.21.15.118:0/1602305341 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7fda540a2920 0x7fda540a4de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:23:42.176 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:23:42.172+0000 7fda7782c740 1 -- 172.21.15.118:0/38686105 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55df5164b1e0 msgr2=0x55df5164b600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:23:42.176 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:23:42.172+0000 7fda7782c740 1 --2- 172.21.15.118:0/38686105 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55df5164b1e0 0x55df5164b600 secure :-1 s=READY pgs=452 cs=0 l=1 rev1=1 crypto rx=0x7fda68009fc0 tx=0x7fda68010860 comp rx=0 tx=0).stop 2024-03-24T00:23:42.176 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:23:42.172+0000 7fda7782c740 1 -- 172.21.15.118:0/38686105 shutdown_connections 2024-03-24T00:23:42.176 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:23:42.172+0000 7fda7782c740 1 --2- 172.21.15.118:0/38686105 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55df51650d40 0x55df516531d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:23:42.176 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:23:42.172+0000 7fda7782c740 1 --2- 172.21.15.118:0/38686105 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55df5164bbd0 0x55df516506d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:23:42.176 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:23:42.172+0000 7fda7782c740 1 --2- 172.21.15.118:0/38686105 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55df5164b1e0 0x55df5164b600 unknown :-1 s=CLOSED pgs=452 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:23:42.176 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:23:42.172+0000 7fda7782c740 1 -- 172.21.15.118:0/38686105 >> 172.21.15.118:0/38686105 conn(0x55df51646750 msgr2=0x55df51646b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:23:42.176 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:23:42.172+0000 7fda6de00640 1 --2- 172.21.15.118:0/1602305341 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7fda540a2920 0x7fda540a4de0 secure :-1 s=READY pgs=388 cs=0 l=1 rev1=1 crypto rx=0x7fda600981b0 tx=0x7fda600d5530 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:23:42.176 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:23:42.172+0000 7fda7782c740 1 -- 172.21.15.118:0/38686105 shutdown_connections 2024-03-24T00:23:42.177 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:23:42.172+0000 7fda7782c740 1 -- 172.21.15.118:0/38686105 wait complete. 2024-03-24T00:23:42.177 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:23:42.172+0000 7fda7782c740 1 -- start start 2024-03-24T00:23:42.177 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:23:42.173+0000 7fda7782c740 1 Processor -- start 2024-03-24T00:23:42.177 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:23:42.173+0000 7fda7782c740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55df5164b1e0 0x55df516553d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:23:42.177 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:23:42.173+0000 7fda7782c740 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55df5164bbd0 0x55df51655910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:23:42.177 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:23:42.173+0000 7fda7782c740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55df51650d40 0x55df5165c8e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:23:42.177 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:23:42.173+0000 7fda7782c740 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x55df5156ca20 con 0x55df5164bbd0 2024-03-24T00:23:42.177 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:23:42.173+0000 7fda7782c740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x55df515ac8d0 con 0x55df5164b1e0 2024-03-24T00:23:42.177 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:23:42.173+0000 7fda7782c740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x55df51653cf0 con 0x55df51650d40 2024-03-24T00:23:42.177 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:23:42.173+0000 7fda7782c740 10 client.0 _ll_register_callbacks cb 0x55df515a91c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T00:23:42.177 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:23:42.173+0000 7fda6e800640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55df5164b1e0 0x55df516553d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:23:42.177 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:23:42.173+0000 7fda67e00640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55df5164bbd0 0x55df51655910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:23:42.177 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:23:42.173+0000 7fda6e800640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55df5164b1e0 0x55df516553d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.118:41354/0 (socket says 172.21.15.118:41354) 2024-03-24T00:23:42.177 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:23:42.173+0000 7fda6e800640 1 -- 172.21.15.118:0/2341795877 learned_addr learned my addr 172.21.15.118:0/2341795877 (peer_addr_for_me v2:172.21.15.118:0/0) 2024-03-24T00:23:42.177 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:23:42.173+0000 7fda67e00640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55df5164bbd0 0x55df51655910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.118:3300/0 says I am v2:172.21.15.118:33406/0 (socket says 172.21.15.118:33406) 2024-03-24T00:23:42.178 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:23:42.173+0000 7fda6e800640 1 --2- 172.21.15.118:0/2341795877 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55df51650d40 0x55df5165c8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:23:42.178 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:23:42.174+0000 7fda67e00640 1 -- 172.21.15.118:0/2341795877 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55df51650d40 msgr2=0x55df5165c8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:23:42.178 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:23:42.174+0000 7fda67e00640 1 --2- 172.21.15.118:0/2341795877 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55df51650d40 0x55df5165c8e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:23:42.178 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:23:42.174+0000 7fda67e00640 1 -- 172.21.15.118:0/2341795877 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55df5164b1e0 msgr2=0x55df516553d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:23:42.178 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:23:42.174+0000 7fda67e00640 1 --2- 172.21.15.118:0/2341795877 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55df5164b1e0 0x55df516553d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:23:42.178 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:23:42.174+0000 7fda67e00640 1 -- 172.21.15.118:0/2341795877 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55df512c5210 con 0x55df5164bbd0 2024-03-24T00:23:42.178 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:23:42.174+0000 7fda6e800640 1 --2- 172.21.15.118:0/2341795877 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55df51650d40 0x55df5165c8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T00:23:42.178 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:23:42.174+0000 7fda67e00640 1 --2- 172.21.15.118:0/2341795877 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55df5164bbd0 0x55df51655910 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7fda58036040 tx=0x7fda58004270 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cbcc37b5e98d9e25 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:23:42.178 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:23:42.174+0000 7fda53400640 10 client.0 ms_handle_connect on v2:172.21.15.118:3300/0 2024-03-24T00:23:42.178 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:23:42.174+0000 7fda53400640 1 -- 172.21.15.118:0/2341795877 <== mon.0 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fda58005d50 con 0x55df5164bbd0 2024-03-24T00:23:42.178 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:23:42.174+0000 7fda53400640 1 -- 172.21.15.118:0/2341795877 <== mon.0 v2:172.21.15.118:3300/0 2 ==== osd_map(114..114 src has 1..114) v4 ==== 10341+0+0 (secure 0 0 0) 0x7fda580046c0 con 0x55df5164bbd0 2024-03-24T00:23:42.178 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:23:42.174+0000 7fda7782c740 1 -- 172.21.15.118:0/2341795877 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fda68003d10 con 0x55df5164bbd0 2024-03-24T00:23:42.178 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:23:42.175+0000 7fda53400640 1 -- 172.21.15.118:0/2341795877 <== mon.0 v2:172.21.15.118:3300/0 3 ==== mon_get_version_reply(handle=1 version=145) v2 ==== 24+0+0 (secure 0 0 0) 0x7fda580048c0 con 0x55df5164bbd0 2024-03-24T00:23:42.179 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:23:42.175+0000 7fda7782c740 10 client.16707 fetch_fsmap learned FSMap version 145 2024-03-24T00:23:42.179 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:23:42.175+0000 7fda7782c740 1 -- 172.21.15.118:0/2341795877 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({config=0+,fsmap=145,monmap=4+}) v3 -- 0x55df51391620 con 0x55df5164bbd0 2024-03-24T00:23:42.179 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:23:42.175+0000 7fda53400640 1 -- 172.21.15.118:0/2341795877 <== mon.0 v2:172.21.15.118:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fda68003d10 con 0x55df5164bbd0 2024-03-24T00:23:42.179 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:23:42.175+0000 7fda53400640 1 -- 172.21.15.118:0/2341795877 <== mon.0 v2:172.21.15.118:3300/0 5 ==== fsmap(e 145) v1 ==== 6784+0+0 (secure 0 0 0) 0x7fda58003820 con 0x55df5164bbd0 2024-03-24T00:23:42.179 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:23:42.175+0000 7fda7782c740 10 client.16707 fetch_fsmap finished waiting for FSMap version 145 2024-03-24T00:23:42.179 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:23:42.175+0000 7fda7782c740 10 client.16707 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-24T00:23:42.179 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:23:42.175+0000 7fda7782c740 20 client.16707 populate_metadata read hostname 'smithi118' 2024-03-24T00:23:42.179 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:23:42.175+0000 7fda7782c740 1 --2- 172.21.15.118:0/2341795877 >> [v2:172.21.15.189:6838/4074484168,v1:172.21.15.189:6839/4074484168] conn(0x55df51661150 0x55df51663540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:23:42.179 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:23:42.175+0000 7fda7782c740 4 client.16707 mds_command: new command op to 24569 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-24T00:23:42.179 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:23:42.175+0000 7fda7782c740 1 -- 172.21.15.118:0/2341795877 --> [v2:172.21.15.189:6838/4074484168,v1:172.21.15.189:6839/4074484168] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55df515bdd70 con 0x55df51661150 2024-03-24T00:23:42.179 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:23:42.176+0000 7fda6e800640 1 --2- 172.21.15.118:0/2341795877 >> [v2:172.21.15.189:6838/4074484168,v1:172.21.15.189:6839/4074484168] conn(0x55df51661150 0x55df51663540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:23:42.180 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:23:42.176+0000 7fda6e800640 1 --2- 172.21.15.118:0/2341795877 >> [v2:172.21.15.189:6838/4074484168,v1:172.21.15.189:6839/4074484168] conn(0x55df51661150 0x55df51663540 crc :-1 s=READY pgs=337 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=e9e3d9703424ec6c server_cookie=c93912285ecd581a in_seq=0 out_seq=0 2024-03-24T00:23:42.180 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:23:42.176+0000 7fda53400640 10 client.16707 ms_handle_connect on v2:172.21.15.189:6838/4074484168 2024-03-24T00:23:42.180 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:23:42.177+0000 7fda53400640 1 -- 172.21.15.118:0/2341795877 <== mds.0 v2:172.21.15.189:6838/4074484168 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x55df515bdd70 con 0x55df51661150 2024-03-24T00:23:42.181 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:23:42.177+0000 7fda53400640 10 client.16707 handle_command_reply: tid=0 2024-03-24T00:23:42.181 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:23:42.177+0000 7fda7782c740 1 client.16707 shutdown 2024-03-24T00:23:42.181 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:23:42.178+0000 7fda7782c740 1 -- 172.21.15.118:0/2341795877 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55df5164bbd0 msgr2=0x55df51655910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:23:42.181 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:23:42.178+0000 7fda7782c740 1 --2- 172.21.15.118:0/2341795877 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55df5164bbd0 0x55df51655910 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7fda58036040 tx=0x7fda58004270 comp rx=0 tx=0).stop 2024-03-24T00:23:42.181 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:23:42.178+0000 7fda7782c740 1 -- 172.21.15.118:0/2341795877 shutdown_connections 2024-03-24T00:23:42.181 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:23:42.178+0000 7fda7782c740 1 --2- 172.21.15.118:0/2341795877 >> [v2:172.21.15.189:6838/4074484168,v1:172.21.15.189:6839/4074484168] conn(0x55df51661150 0x55df51663540 crc :-1 s=READY pgs=337 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:23:42.181 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:23:42.178+0000 7fda7782c740 1 --2- 172.21.15.118:0/2341795877 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55df51650d40 0x55df5165c8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:23:42.181 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:23:42.178+0000 7fda7782c740 1 --2- 172.21.15.118:0/2341795877 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55df5164bbd0 0x55df51655910 unknown :-1 s=CLOSED pgs=229 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:23:42.181 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:23:42.178+0000 7fda7782c740 1 --2- 172.21.15.118:0/2341795877 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55df5164b1e0 0x55df516553d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:23:42.181 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:23:42.178+0000 7fda53400640 0 client.16707 ms_handle_reset on v2:172.21.15.189:6838/4074484168 2024-03-24T00:23:42.181 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:23:42.178+0000 7fda7782c740 1 -- 172.21.15.118:0/2341795877 >> 172.21.15.118:0/2341795877 conn(0x55df51646750 msgr2=0x55df51645950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:23:42.181 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:23:42.178+0000 7fda7782c740 1 -- 172.21.15.118:0/2341795877 shutdown_connections 2024-03-24T00:23:42.181 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:23:42.178+0000 7fda7782c740 1 -- 172.21.15.118:0/2341795877 wait complete. 2024-03-24T00:23:42.181 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:23:42.178+0000 7fda7782c740 20 client.16707 trim_cache size 0 max 16384 2024-03-24T00:23:42.192 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:23:42.193+0000 7fda7782c740 1 Processor -- start 2024-03-24T00:23:42.193 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:23:42.193+0000 7fda7782c740 1 -- start start 2024-03-24T00:23:42.193 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:23:42.193+0000 7fda7782c740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55df5164b1e0 0x55df516499b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:23:42.193 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:23:42.193+0000 7fda7782c740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55df5164bbd0 0x55df5165c8a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:23:42.193 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:23:42.193+0000 7fda7782c740 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55df51650d40 0x55df51649ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:23:42.193 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:23:42.193+0000 7fda7782c740 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fda701aa990 con 0x55df51650d40 2024-03-24T00:23:42.193 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:23:42.193+0000 7fda7782c740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fda701acfd0 con 0x55df5164b1e0 2024-03-24T00:23:42.193 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:23:42.193+0000 7fda7782c740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fda701aaff0 con 0x55df5164bbd0 2024-03-24T00:23:42.195 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:23:42.193+0000 7fda67e00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55df5164bbd0 0x55df5165c8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:23:42.195 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:23:42.193+0000 7fda67e00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55df5164bbd0 0x55df5165c8a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.118:43966/0 (socket says 172.21.15.118:43966) 2024-03-24T00:23:42.195 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:23:42.193+0000 7fda67e00640 1 -- 172.21.15.118:0/3531267914 learned_addr learned my addr 172.21.15.118:0/3531267914 (peer_addr_for_me v2:172.21.15.118:0/0) 2024-03-24T00:23:42.195 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:23:42.193+0000 7fda6e800640 1 --2- 172.21.15.118:0/3531267914 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55df5164b1e0 0x55df516499b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:23:42.195 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:23:42.193+0000 7fda6e800640 1 --2- 172.21.15.118:0/3531267914 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55df51650d40 0x55df51649ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:23:42.196 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:23:42.194+0000 7fda67e00640 1 -- 172.21.15.118:0/3531267914 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55df5164b1e0 msgr2=0x55df516499b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:23:42.196 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:23:42.194+0000 7fda67e00640 1 --2- 172.21.15.118:0/3531267914 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55df5164b1e0 0x55df516499b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:23:42.196 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:23:42.194+0000 7fda67e00640 1 -- 172.21.15.118:0/3531267914 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55df51650d40 msgr2=0x55df51649ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:23:42.196 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:23:42.194+0000 7fda67e00640 1 --2- 172.21.15.118:0/3531267914 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55df51650d40 0x55df51649ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:23:42.196 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:23:42.194+0000 7fda67e00640 1 -- 172.21.15.118:0/3531267914 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55df516638b0 con 0x55df5164bbd0 2024-03-24T00:23:42.196 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:23:42.194+0000 7fda6e800640 1 --2- 172.21.15.118:0/3531267914 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55df5164b1e0 0x55df516499b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T00:23:42.196 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:23:42.194+0000 7fda67e00640 1 --2- 172.21.15.118:0/3531267914 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55df5164bbd0 0x55df5165c8a0 secure :-1 s=READY pgs=451 cs=0 l=1 rev1=1 crypto rx=0x7fda5802c7b0 tx=0x7fda58000bc0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=5c94af8dd8d1ce9c server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:23:42.196 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:23:42.194+0000 7fda5e600640 1 -- 172.21.15.118:0/3531267914 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fda5805c070 con 0x55df5164bbd0 2024-03-24T00:23:42.196 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:23:42.194+0000 7fda5e600640 1 -- 172.21.15.118:0/3531267914 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fda58003d90 con 0x55df5164bbd0 2024-03-24T00:23:42.196 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:23:42.194+0000 7fda5e600640 1 -- 172.21.15.118:0/3531267914 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fda58005290 con 0x55df5164bbd0 2024-03-24T00:23:42.196 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:23:42.195+0000 7fda7782c740 1 -- 172.21.15.118:0/3531267914 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55df5164bbd0 msgr2=0x55df5165c8a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:23:42.196 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:23:42.195+0000 7fda7782c740 1 --2- 172.21.15.118:0/3531267914 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55df5164bbd0 0x55df5165c8a0 secure :-1 s=READY pgs=451 cs=0 l=1 rev1=1 crypto rx=0x7fda5802c7b0 tx=0x7fda58000bc0 comp rx=0 tx=0).stop 2024-03-24T00:23:42.196 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:23:42.195+0000 7fda7782c740 1 -- 172.21.15.118:0/3531267914 shutdown_connections 2024-03-24T00:23:42.196 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:23:42.195+0000 7fda7782c740 1 --2- 172.21.15.118:0/3531267914 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55df51650d40 0x55df51649ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:23:42.197 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:23:42.195+0000 7fda7782c740 1 --2- 172.21.15.118:0/3531267914 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55df5164bbd0 0x55df5165c8a0 unknown :-1 s=CLOSED pgs=451 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:23:42.197 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:23:42.195+0000 7fda7782c740 1 --2- 172.21.15.118:0/3531267914 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55df5164b1e0 0x55df516499b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:23:42.197 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:23:42.195+0000 7fda7782c740 1 -- 172.21.15.118:0/3531267914 >> 172.21.15.118:0/3531267914 conn(0x55df51646750 msgr2=0x55df5164cef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:23:42.197 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:23:42.195+0000 7fda7782c740 1 -- 172.21.15.118:0/3531267914 shutdown_connections 2024-03-24T00:23:42.197 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:23:42.195+0000 7fda7782c740 1 -- 172.21.15.118:0/3531267914 wait complete. 2024-03-24T00:23:42.197 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:23:42.195+0000 7fda7782c740 1 -- start start 2024-03-24T00:23:42.197 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:23:42.196+0000 7fda7782c740 1 Processor -- start 2024-03-24T00:23:42.197 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:23:41 smithi118 ceph-mon[31535]: pgmap v662: 97 pgs: 97 active+clean; 146 MiB data, 849 MiB used, 1.0 TiB / 1.0 TiB avail; 374 KiB/s rd, 30 KiB/s wr, 54 op/s 2024-03-24T00:23:42.198 INFO:journalctl@ceph.mds.g.smithi118.stdout:Mar 24 00:23:41 smithi118 systemd[1]: ceph-d79a5420-e971-11ee-95cd-87774f69a715@mds.g.service: Scheduled restart job, restart counter is at 6. 2024-03-24T00:23:42.198 INFO:journalctl@ceph.mds.g.smithi118.stdout:Mar 24 00:23:41 smithi118 systemd[1]: Stopped Ceph mds.g for d79a5420-e971-11ee-95cd-87774f69a715. 2024-03-24T00:23:42.198 INFO:journalctl@ceph.mds.g.smithi118.stdout:Mar 24 00:23:41 smithi118 systemd[1]: ceph-d79a5420-e971-11ee-95cd-87774f69a715@mds.g.service: Consumed 1.580s CPU time. 2024-03-24T00:23:42.198 INFO:journalctl@ceph.mds.g.smithi118.stdout:Mar 24 00:23:41 smithi118 systemd[1]: ceph-d79a5420-e971-11ee-95cd-87774f69a715@mds.g.service: Start request repeated too quickly. 2024-03-24T00:23:42.198 INFO:journalctl@ceph.mds.g.smithi118.stdout:Mar 24 00:23:41 smithi118 systemd[1]: ceph-d79a5420-e971-11ee-95cd-87774f69a715@mds.g.service: Failed with result 'exit-code'. 2024-03-24T00:23:42.198 INFO:journalctl@ceph.mds.g.smithi118.stdout:Mar 24 00:23:41 smithi118 systemd[1]: Failed to start Ceph mds.g for d79a5420-e971-11ee-95cd-87774f69a715. 2024-03-24T00:23:42.200 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:23:42.196+0000 7fda7782c740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55df5164b1e0 0x55df5165c230 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:23:42.200 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:23:42.196+0000 7fda7782c740 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55df5164bbd0 0x55df5165c770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:23:42.200 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:23:42.196+0000 7fda7782c740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55df51650d40 0x55df5165a890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:23:42.201 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:23:42.196+0000 7fda7782c740 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fda701ab650 con 0x55df5164bbd0 2024-03-24T00:23:42.201 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:23:42.196+0000 7fda7782c740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fda701ac970 con 0x55df5164b1e0 2024-03-24T00:23:42.201 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:23:42.196+0000 7fda7782c740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fda701abcb0 con 0x55df51650d40 2024-03-24T00:23:42.201 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:23:42.196+0000 7fda7782c740 10 client.0 _ll_register_callbacks cb 0x7fda58009780 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T00:23:42.201 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:23:42.196+0000 7fda6e800640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55df5164b1e0 0x55df5165c230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:23:42.201 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:23:42.196+0000 7fda6e800640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55df5164b1e0 0x55df5165c230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.118:41378/0 (socket says 172.21.15.118:41378) 2024-03-24T00:23:42.201 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:23:42.196+0000 7fda6e800640 1 -- 172.21.15.118:0/722581818 learned_addr learned my addr 172.21.15.118:0/722581818 (peer_addr_for_me v2:172.21.15.118:0/0) 2024-03-24T00:23:42.201 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:23:42.196+0000 7fda6e800640 1 --2- 172.21.15.118:0/722581818 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55df51650d40 0x55df5165a890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:23:42.201 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:23:42.197+0000 7fda67e00640 1 --2- 172.21.15.118:0/722581818 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55df5164bbd0 0x55df5165c770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:23:42.201 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:23:42.197+0000 7fda6e800640 1 -- 172.21.15.118:0/722581818 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55df51650d40 msgr2=0x55df5165a890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:23:42.201 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:23:42.197+0000 7fda6e800640 1 --2- 172.21.15.118:0/722581818 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55df51650d40 0x55df5165a890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:23:42.201 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:23:42.197+0000 7fda6e800640 1 -- 172.21.15.118:0/722581818 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55df5164bbd0 msgr2=0x55df5165c770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:23:42.202 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:23:42.197+0000 7fda6e800640 1 --2- 172.21.15.118:0/722581818 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55df5164bbd0 0x55df5165c770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:23:42.202 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:23:42.197+0000 7fda6e800640 1 -- 172.21.15.118:0/722581818 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55df5165b0c0 con 0x55df5164b1e0 2024-03-24T00:23:42.202 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:23:42.197+0000 7fda67e00640 1 --2- 172.21.15.118:0/722581818 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55df5164bbd0 0x55df5165c770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T00:23:42.202 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:23:42.197+0000 7fda6e800640 1 --2- 172.21.15.118:0/722581818 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55df5164b1e0 0x55df5165c230 secure :-1 s=READY pgs=453 cs=0 l=1 rev1=1 crypto rx=0x7fda68000f80 tx=0x7fda68010860 comp rx=0 tx=0).ready entity=mon.2 client_cookie=efb264fa1340e645 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:23:42.202 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:23:42.197+0000 7fda53400640 10 client.0 ms_handle_connect on v2:172.21.15.189:3300/0 2024-03-24T00:23:42.202 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:23:42.197+0000 7fda53400640 1 -- 172.21.15.118:0/722581818 <== mon.2 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fda6802dbc0 con 0x55df5164b1e0 2024-03-24T00:23:42.202 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:23:42.197+0000 7fda7782c740 1 -- 172.21.15.118:0/722581818 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fda58003d90 con 0x55df5164b1e0 2024-03-24T00:23:42.202 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:23:42.197+0000 7fda53400640 1 -- 172.21.15.118:0/722581818 <== mon.2 v2:172.21.15.189:3300/0 2 ==== osd_map(114..114 src has 1..114) v4 ==== 10341+0+0 (secure 0 0 0) 0x7fda6802dd60 con 0x55df5164b1e0 2024-03-24T00:23:42.202 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:23:42.198+0000 7fda53400640 1 -- 172.21.15.118:0/722581818 <== mon.2 v2:172.21.15.189:3300/0 3 ==== mon_get_version_reply(handle=1 version=145) v2 ==== 24+0+0 (secure 0 0 0) 0x7fda68037b40 con 0x55df5164b1e0 2024-03-24T00:23:42.202 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:23:42.198+0000 7fda7782c740 10 client.26948 fetch_fsmap learned FSMap version 145 2024-03-24T00:23:42.202 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:23:42.198+0000 7fda7782c740 1 -- 172.21.15.118:0/722581818 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,fsmap=145,monmap=4+}) v3 -- 0x55df516c5d70 con 0x55df5164b1e0 2024-03-24T00:23:42.202 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:23:42.198+0000 7fda53400640 1 -- 172.21.15.118:0/722581818 <== mon.2 v2:172.21.15.189:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fda58003d90 con 0x55df5164b1e0 2024-03-24T00:23:42.202 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:23:42.198+0000 7fda53400640 1 -- 172.21.15.118:0/722581818 <== mon.2 v2:172.21.15.189:3300/0 5 ==== fsmap(e 145) v1 ==== 6784+0+0 (secure 0 0 0) 0x7fda68037ce0 con 0x55df5164b1e0 2024-03-24T00:23:42.202 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:23:42.198+0000 7fda7782c740 10 client.26948 fetch_fsmap finished waiting for FSMap version 145 2024-03-24T00:23:42.203 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:23:42.198+0000 7fda7782c740 10 client.26948 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-24T00:23:42.203 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:23:42.198+0000 7fda7782c740 20 client.26948 populate_metadata read hostname 'smithi118' 2024-03-24T00:23:42.203 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:23:42.198+0000 7fda7782c740 1 --2- 172.21.15.118:0/722581818 >> [v2:172.21.15.189:6838/4074484168,v1:172.21.15.189:6839/4074484168] conn(0x55df516c6610 0x55df516c8a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:23:42.203 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:23:42.198+0000 7fda7782c740 4 client.26948 mds_command: new command op to 24569 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-03-24T00:23:42.203 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:23:42.198+0000 7fda7782c740 1 -- 172.21.15.118:0/722581818 --> [v2:172.21.15.189:6838/4074484168,v1:172.21.15.189:6839/4074484168] -- command(tid 0: {"prefix": "flush journal"}) v1 -- 0x55df515b80c0 con 0x55df516c6610 2024-03-24T00:23:42.203 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:23:42.198+0000 7fda67e00640 1 --2- 172.21.15.118:0/722581818 >> [v2:172.21.15.189:6838/4074484168,v1:172.21.15.189:6839/4074484168] conn(0x55df516c6610 0x55df516c8a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:23:42.203 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:23:42.199+0000 7fda67e00640 1 --2- 172.21.15.118:0/722581818 >> [v2:172.21.15.189:6838/4074484168,v1:172.21.15.189:6839/4074484168] conn(0x55df516c6610 0x55df516c8a30 crc :-1 s=READY pgs=338 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=a5e1ee4f1044072e server_cookie=102907862ab610f6 in_seq=0 out_seq=0 2024-03-24T00:23:42.203 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:23:42.199+0000 7fda53400640 10 client.26948 ms_handle_connect on v2:172.21.15.189:6838/4074484168 2024-03-24T00:23:42.203 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:23:41 smithi189 ceph-mon[36187]: pgmap v662: 97 pgs: 97 active+clean; 146 MiB data, 849 MiB used, 1.0 TiB / 1.0 TiB avail; 374 KiB/s rd, 30 KiB/s wr, 54 op/s 2024-03-24T00:23:42.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:23:41 smithi193 ceph-mon[36878]: pgmap v662: 97 pgs: 97 active+clean; 146 MiB data, 849 MiB used, 1.0 TiB / 1.0 TiB avail; 374 KiB/s rd, 30 KiB/s wr, 54 op/s 2024-03-24T00:23:42.549 INFO:journalctl@ceph.mds.d.smithi118.stdout:Mar 24 00:23:42 smithi118 systemd[1]: ceph-d79a5420-e971-11ee-95cd-87774f69a715@mds.d.service: Scheduled restart job, restart counter is at 2. 2024-03-24T00:23:42.549 INFO:journalctl@ceph.mds.d.smithi118.stdout:Mar 24 00:23:42 smithi118 systemd[1]: Stopped Ceph mds.d for d79a5420-e971-11ee-95cd-87774f69a715. 2024-03-24T00:23:42.549 INFO:journalctl@ceph.mds.d.smithi118.stdout:Mar 24 00:23:42 smithi118 systemd[1]: ceph-d79a5420-e971-11ee-95cd-87774f69a715@mds.d.service: Consumed 1.721s CPU time. 2024-03-24T00:23:42.549 INFO:journalctl@ceph.mds.d.smithi118.stdout:Mar 24 00:23:42 smithi118 systemd[1]: Starting Ceph mds.d for d79a5420-e971-11ee-95cd-87774f69a715... 2024-03-24T00:23:42.870 INFO:journalctl@ceph.mds.d.smithi118.stdout:Mar 24 00:23:42 smithi118 podman[95893]: 2024-03-24 00:23:42.453492068 +0000 UTC m=+0.032075112 image pull c664afe56bb01db4f9afa9532c20598e201e664f2e9ba2be50db0d52674a26d2 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc 2024-03-24T00:23:42.871 INFO:journalctl@ceph.mds.d.smithi118.stdout:Mar 24 00:23:42 smithi118 podman[95893]: 2024-03-24 00:23:42.574956527 +0000 UTC m=+0.153539608 container create 3cd3f9060cca524fa1b913dbc4556047ba0927ead5677afa6b93e2890dddba84 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-d, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.build-date=20240319, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux ) 2024-03-24T00:23:42.871 INFO:journalctl@ceph.mds.d.smithi118.stdout:Mar 24 00:23:42 smithi118 podman[95893]: 2024-03-24 00:23:42.870324516 +0000 UTC m=+0.448907604 container init 3cd3f9060cca524fa1b913dbc4556047ba0927ead5677afa6b93e2890dddba84 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-d, org.label-schema.build-date=20240319, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, GIT_CLEAN=True, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, ceph=True) 2024-03-24T00:23:43.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:23:42 smithi118 ceph-mon[31535]: daemon mds.d restarted 2024-03-24T00:23:43.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:23:42 smithi118 ceph-mon[31535]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-03-24T00:23:43.197 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:23:42 smithi118 ceph-mon[31535]: osdmap e115: 12 total, 12 up, 12 in 2024-03-24T00:23:43.197 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:23:42 smithi118 ceph-mon[31535]: mds.? [v2:172.21.15.118:6836/1531167856,v1:172.21.15.118:6837/1531167856] up:boot 2024-03-24T00:23:43.197 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:23:42 smithi118 ceph-mon[31535]: Standby daemon mds.h assigned to filesystem cephfs as rank 1 2024-03-24T00:23:43.197 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:23:42 smithi118 ceph-mon[31535]: fsmap cephfs:4/5 {0=b=up:active,2=c=up:active,3=g=up:active,4=e=up:active} 5 up:standby-replay 1 up:standby, 1 failed 2024-03-24T00:23:43.197 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:23:42 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2024-03-24T00:23:43.197 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:23:42 smithi118 ceph-mon[31535]: fsmap cephfs:5/5 4 up:standby-replay 1 up:standby 1 up:replay 4 up:active 2024-03-24T00:23:43.197 INFO:journalctl@ceph.mds.d.smithi118.stdout:Mar 24 00:23:42 smithi118 podman[95893]: 2024-03-24 00:23:42.880819409 +0000 UTC m=+0.459402480 container start 3cd3f9060cca524fa1b913dbc4556047ba0927ead5677afa6b93e2890dddba84 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-d, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , ceph=True, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240319, io.buildah.version=1.33.5) 2024-03-24T00:23:43.197 INFO:journalctl@ceph.mds.d.smithi118.stdout:Mar 24 00:23:42 smithi118 ceph-mds[95912]: -- 172.21.15.118:0/1688984223 <== mon.2 v2:172.21.15.189:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x55c0c3207a40 con 0x55c0c31da800 2024-03-24T00:23:43.197 INFO:journalctl@ceph.mds.d.smithi118.stdout:Mar 24 00:23:42 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-d[95908]: starting mds.d at 2024-03-24T00:23:43.197 INFO:journalctl@ceph.mds.d.smithi118.stdout:Mar 24 00:23:42 smithi118 bash[95893]: 3cd3f9060cca524fa1b913dbc4556047ba0927ead5677afa6b93e2890dddba84 2024-03-24T00:23:43.197 INFO:journalctl@ceph.mds.d.smithi118.stdout:Mar 24 00:23:42 smithi118 systemd[1]: Started Ceph mds.d for d79a5420-e971-11ee-95cd-87774f69a715. 2024-03-24T00:23:43.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:23:42 smithi189 ceph-mon[36187]: daemon mds.d restarted 2024-03-24T00:23:43.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:23:42 smithi189 ceph-mon[36187]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-03-24T00:23:43.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:23:42 smithi189 ceph-mon[36187]: osdmap e115: 12 total, 12 up, 12 in 2024-03-24T00:23:43.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:23:42 smithi189 ceph-mon[36187]: mds.? [v2:172.21.15.118:6836/1531167856,v1:172.21.15.118:6837/1531167856] up:boot 2024-03-24T00:23:43.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:23:42 smithi189 ceph-mon[36187]: Standby daemon mds.h assigned to filesystem cephfs as rank 1 2024-03-24T00:23:43.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:23:42 smithi189 ceph-mon[36187]: fsmap cephfs:4/5 {0=b=up:active,2=c=up:active,3=g=up:active,4=e=up:active} 5 up:standby-replay 1 up:standby, 1 failed 2024-03-24T00:23:43.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:23:42 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2024-03-24T00:23:43.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:23:42 smithi189 ceph-mon[36187]: fsmap cephfs:5/5 4 up:standby-replay 1 up:standby 1 up:replay 4 up:active 2024-03-24T00:23:43.460 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:23:42 smithi193 ceph-mon[36878]: daemon mds.d restarted 2024-03-24T00:23:43.460 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:23:42 smithi193 ceph-mon[36878]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-03-24T00:23:43.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:23:42 smithi193 ceph-mon[36878]: osdmap e115: 12 total, 12 up, 12 in 2024-03-24T00:23:43.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:23:42 smithi193 ceph-mon[36878]: mds.? [v2:172.21.15.118:6836/1531167856,v1:172.21.15.118:6837/1531167856] up:boot 2024-03-24T00:23:43.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:23:42 smithi193 ceph-mon[36878]: Standby daemon mds.h assigned to filesystem cephfs as rank 1 2024-03-24T00:23:43.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:23:42 smithi193 ceph-mon[36878]: fsmap cephfs:4/5 {0=b=up:active,2=c=up:active,3=g=up:active,4=e=up:active} 5 up:standby-replay 1 up:standby, 1 failed 2024-03-24T00:23:43.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:23:42 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2024-03-24T00:23:43.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:23:42 smithi193 ceph-mon[36878]: fsmap cephfs:5/5 4 up:standby-replay 1 up:standby 1 up:replay 4 up:active 2024-03-24T00:23:44.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:23:43 smithi118 ceph-mon[31535]: pgmap v664: 97 pgs: 97 active+clean; 146 MiB data, 849 MiB used, 1.0 TiB / 1.0 TiB avail; 118 KiB/s rd, 52 KiB/s wr, 36 op/s 2024-03-24T00:23:44.453 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:23:43 smithi189 ceph-mon[36187]: pgmap v664: 97 pgs: 97 active+clean; 146 MiB data, 849 MiB used, 1.0 TiB / 1.0 TiB avail; 118 KiB/s rd, 52 KiB/s wr, 36 op/s 2024-03-24T00:23:44.460 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:23:43 smithi193 ceph-mon[36878]: pgmap v664: 97 pgs: 97 active+clean; 146 MiB data, 849 MiB used, 1.0 TiB / 1.0 TiB avail; 118 KiB/s rd, 52 KiB/s wr, 36 op/s 2024-03-24T00:23:45.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:23:45 smithi118 ceph-mon[31535]: mds.? [v2:172.21.15.189:6836/2642988219,v1:172.21.15.189:6837/2642988219] up:resolve 2024-03-24T00:23:45.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:23:45 smithi118 ceph-mon[31535]: Replacing daemon mds.g as rank 3 with standby daemon mds.j 2024-03-24T00:23:45.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:23:45 smithi118 ceph-mon[31535]: MDS daemon mds.g is removed because it is dead or otherwise unavailable. 2024-03-24T00:23:45.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:23:45 smithi118 ceph-mon[31535]: fsmap cephfs:5/5 4 up:standby-replay 1 up:standby 1 up:resolve 4 up:active 2024-03-24T00:23:45.447 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:23:45 smithi118 ceph-mon[31535]: osdmap e116: 12 total, 12 up, 12 in 2024-03-24T00:23:45.447 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:23:45 smithi118 ceph-mon[31535]: fsmap cephfs:5/5 3 up:standby-replay 1 up:standby 1 up:replay 1 up:resolve 3 up:active 2024-03-24T00:23:45.453 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:23:45 smithi189 ceph-mon[36187]: mds.? [v2:172.21.15.189:6836/2642988219,v1:172.21.15.189:6837/2642988219] up:resolve 2024-03-24T00:23:45.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:23:45 smithi189 ceph-mon[36187]: Replacing daemon mds.g as rank 3 with standby daemon mds.j 2024-03-24T00:23:45.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:23:45 smithi189 ceph-mon[36187]: MDS daemon mds.g is removed because it is dead or otherwise unavailable. 2024-03-24T00:23:45.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:23:45 smithi189 ceph-mon[36187]: fsmap cephfs:5/5 4 up:standby-replay 1 up:standby 1 up:resolve 4 up:active 2024-03-24T00:23:45.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:23:45 smithi189 ceph-mon[36187]: osdmap e116: 12 total, 12 up, 12 in 2024-03-24T00:23:45.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:23:45 smithi189 ceph-mon[36187]: fsmap cephfs:5/5 3 up:standby-replay 1 up:standby 1 up:replay 1 up:resolve 3 up:active 2024-03-24T00:23:45.460 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:23:44 smithi193 ceph-mon[36878]: mds.? [v2:172.21.15.189:6836/2642988219,v1:172.21.15.189:6837/2642988219] up:resolve 2024-03-24T00:23:45.460 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:23:44 smithi193 ceph-mon[36878]: Replacing daemon mds.g as rank 3 with standby daemon mds.j 2024-03-24T00:23:45.460 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:23:44 smithi193 ceph-mon[36878]: MDS daemon mds.g is removed because it is dead or otherwise unavailable. 2024-03-24T00:23:45.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:23:44 smithi193 ceph-mon[36878]: fsmap cephfs:5/5 4 up:standby-replay 1 up:standby 1 up:resolve 4 up:active 2024-03-24T00:23:45.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:23:44 smithi193 ceph-mon[36878]: osdmap e116: 12 total, 12 up, 12 in 2024-03-24T00:23:45.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:23:44 smithi193 ceph-mon[36878]: fsmap cephfs:5/5 3 up:standby-replay 1 up:standby 1 up:replay 1 up:resolve 3 up:active 2024-03-24T00:23:46.348 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:23:46 smithi118 ceph-mon[31535]: mds.? [v2:172.21.15.118:6834/2587469815,v1:172.21.15.118:6835/2587469815] up:resolve 2024-03-24T00:23:46.348 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:23:46 smithi118 ceph-mon[31535]: fsmap cephfs:5/5 3 up:standby-replay 1 up:standby 2 up:resolve 3 up:active 2024-03-24T00:23:46.348 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:23:46 smithi118 ceph-mon[31535]: pgmap v666: 97 pgs: 97 active+clean; 146 MiB data, 850 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 42 KiB/s wr, 25 op/s 2024-03-24T00:23:46.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:23:46 smithi189 ceph-mon[36187]: mds.? [v2:172.21.15.118:6834/2587469815,v1:172.21.15.118:6835/2587469815] up:resolve 2024-03-24T00:23:46.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:23:46 smithi189 ceph-mon[36187]: fsmap cephfs:5/5 3 up:standby-replay 1 up:standby 2 up:resolve 3 up:active 2024-03-24T00:23:46.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:23:46 smithi189 ceph-mon[36187]: pgmap v666: 97 pgs: 97 active+clean; 146 MiB data, 850 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 42 KiB/s wr, 25 op/s 2024-03-24T00:23:46.460 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:23:46 smithi193 ceph-mon[36878]: mds.? [v2:172.21.15.118:6834/2587469815,v1:172.21.15.118:6835/2587469815] up:resolve 2024-03-24T00:23:46.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:23:46 smithi193 ceph-mon[36878]: fsmap cephfs:5/5 3 up:standby-replay 1 up:standby 2 up:resolve 3 up:active 2024-03-24T00:23:46.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:23:46 smithi193 ceph-mon[36878]: pgmap v666: 97 pgs: 97 active+clean; 146 MiB data, 850 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 42 KiB/s wr, 25 op/s 2024-03-24T00:23:47.060 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:23:47.060+0000 7fda53400640 1 -- 172.21.15.118:0/722581818 <== mds.0 v2:172.21.15.189:6838/4074484168 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+44 (crc 0 0 0) 0x55df515b80c0 con 0x55df516c6610 2024-03-24T00:23:47.060 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:23:47.060+0000 7fda53400640 10 client.26948 handle_command_reply: tid=0 2024-03-24T00:23:47.060 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:23:47.060+0000 7fda7782c740 1 client.26948 shutdown 2024-03-24T00:23:47.061 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:23:47.061+0000 7fda7782c740 1 -- 172.21.15.118:0/722581818 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55df5164b1e0 msgr2=0x55df5165c230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:23:47.061 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:23:47.061+0000 7fda7782c740 1 --2- 172.21.15.118:0/722581818 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55df5164b1e0 0x55df5165c230 secure :-1 s=READY pgs=453 cs=0 l=1 rev1=1 crypto rx=0x7fda68000f80 tx=0x7fda68010860 comp rx=0 tx=0).stop 2024-03-24T00:23:47.061 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:23:47.061+0000 7fda7782c740 1 -- 172.21.15.118:0/722581818 shutdown_connections 2024-03-24T00:23:47.061 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:23:47.061+0000 7fda7782c740 1 --2- 172.21.15.118:0/722581818 >> [v2:172.21.15.189:6838/4074484168,v1:172.21.15.189:6839/4074484168] conn(0x55df516c6610 0x55df516c8a30 crc :-1 s=READY pgs=338 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:23:47.061 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:23:47.061+0000 7fda7782c740 1 --2- 172.21.15.118:0/722581818 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55df51650d40 0x55df5165a890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:23:47.061 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:23:47.061+0000 7fda7782c740 1 --2- 172.21.15.118:0/722581818 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55df5164bbd0 0x55df5165c770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:23:47.061 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:23:47.061+0000 7fda7782c740 1 --2- 172.21.15.118:0/722581818 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55df5164b1e0 0x55df5165c230 unknown :-1 s=CLOSED pgs=453 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:23:47.061 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:23:47.061+0000 7fda53400640 0 client.26948 ms_handle_reset on v2:172.21.15.189:6838/4074484168 2024-03-24T00:23:47.061 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:23:47.061+0000 7fda7782c740 1 -- 172.21.15.118:0/722581818 >> 172.21.15.118:0/722581818 conn(0x55df51646750 msgr2=0x55df516499b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:23:47.062 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:23:47.061+0000 7fda7782c740 1 -- 172.21.15.118:0/722581818 shutdown_connections 2024-03-24T00:23:47.062 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:23:47.061+0000 7fda7782c740 1 -- 172.21.15.118:0/722581818 wait complete. 2024-03-24T00:23:47.062 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:23:47.062+0000 7fda7782c740 20 client.26948 trim_cache size 0 max 16384 2024-03-24T00:23:47.062 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stdout:{ 2024-03-24T00:23:47.063 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stdout: "message": "", 2024-03-24T00:23:47.063 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stdout: "return_code": 0 2024-03-24T00:23:47.063 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stdout:} 2024-03-24T00:23:47.063 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:23:47.063+0000 7fda74a00640 1 -- 172.21.15.118:0/1602305341 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7fda540a2920 msgr2=0x7fda540a4de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:23:47.063 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:23:47.063+0000 7fda74a00640 1 --2- 172.21.15.118:0/1602305341 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7fda540a2920 0x7fda540a4de0 secure :-1 s=READY pgs=388 cs=0 l=1 rev1=1 crypto rx=0x7fda600981b0 tx=0x7fda600d5530 comp rx=0 tx=0).stop 2024-03-24T00:23:47.063 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:23:47.063+0000 7fda74a00640 1 -- 172.21.15.118:0/1602305341 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fda701038f0 msgr2=0x7fda70199b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:23:47.064 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:23:47.063+0000 7fda74a00640 1 --2- 172.21.15.118:0/1602305341 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fda701038f0 0x7fda70199b30 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7fda6000e9c0 tx=0x7fda6000ee80 comp rx=0 tx=0).stop 2024-03-24T00:23:47.064 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:23:47.063+0000 7fda74a00640 1 -- 172.21.15.118:0/1602305341 shutdown_connections 2024-03-24T00:23:47.064 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:23:47.063+0000 7fda74a00640 1 --2- 172.21.15.118:0/1602305341 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7fda540a2920 0x7fda540a4de0 unknown :-1 s=CLOSED pgs=388 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:23:47.064 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:23:47.063+0000 7fda74a00640 1 --2- 172.21.15.118:0/1602305341 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fda701054a0 0x7fda701a10e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:23:47.064 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:23:47.063+0000 7fda74a00640 1 --2- 172.21.15.118:0/1602305341 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fda70104ae0 0x7fda7019a070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:23:47.064 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:23:47.063+0000 7fda74a00640 1 --2- 172.21.15.118:0/1602305341 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fda701038f0 0x7fda70199b30 unknown :-1 s=CLOSED pgs=228 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:23:47.064 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:23:47.063+0000 7fda74a00640 1 -- 172.21.15.118:0/1602305341 >> 172.21.15.118:0/1602305341 conn(0x7fda700fee60 msgr2=0x7fda70107d10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:23:47.064 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:23:47.063+0000 7fda74a00640 1 -- 172.21.15.118:0/1602305341 shutdown_connections 2024-03-24T00:23:47.064 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:23:47.063+0000 7fda74a00640 1 -- 172.21.15.118:0/1602305341 wait complete. 2024-03-24T00:23:47.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:23:47 smithi118 ceph-mon[31535]: mds.? [v2:172.21.15.189:6836/2642988219,v1:172.21.15.189:6837/2642988219] up:reconnect 2024-03-24T00:23:47.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:23:47 smithi118 ceph-mon[31535]: mds.? [v2:172.21.15.118:6834/2587469815,v1:172.21.15.118:6835/2587469815] up:reconnect 2024-03-24T00:23:47.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:23:47 smithi118 ceph-mon[31535]: fsmap cephfs:5/5 3 up:standby-replay 1 up:standby 2 up:reconnect 3 up:active 2024-03-24T00:23:47.447 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:23:47 smithi118 ceph-mon[31535]: reconnect by client.15159 192.168.0.1:0/3463739313 after 0.0129998 2024-03-24T00:23:47.447 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:23:47 smithi118 ceph-mon[31535]: reconnect by client.15159 192.168.0.1:0/3463739313 after 0.0229995 2024-03-24T00:23:47.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:23:47 smithi189 ceph-mon[36187]: mds.? [v2:172.21.15.189:6836/2642988219,v1:172.21.15.189:6837/2642988219] up:reconnect 2024-03-24T00:23:47.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:23:47 smithi189 ceph-mon[36187]: mds.? [v2:172.21.15.118:6834/2587469815,v1:172.21.15.118:6835/2587469815] up:reconnect 2024-03-24T00:23:47.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:23:47 smithi189 ceph-mon[36187]: fsmap cephfs:5/5 3 up:standby-replay 1 up:standby 2 up:reconnect 3 up:active 2024-03-24T00:23:47.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:23:47 smithi189 ceph-mon[36187]: reconnect by client.15159 192.168.0.1:0/3463739313 after 0.0129998 2024-03-24T00:23:47.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:23:47 smithi189 ceph-mon[36187]: reconnect by client.15159 192.168.0.1:0/3463739313 after 0.0229995 2024-03-24T00:23:47.460 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:23:47 smithi193 ceph-mon[36878]: mds.? [v2:172.21.15.189:6836/2642988219,v1:172.21.15.189:6837/2642988219] up:reconnect 2024-03-24T00:23:47.460 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:23:47 smithi193 ceph-mon[36878]: mds.? [v2:172.21.15.118:6834/2587469815,v1:172.21.15.118:6835/2587469815] up:reconnect 2024-03-24T00:23:47.460 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:23:47 smithi193 ceph-mon[36878]: fsmap cephfs:5/5 3 up:standby-replay 1 up:standby 2 up:reconnect 3 up:active 2024-03-24T00:23:47.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:23:47 smithi193 ceph-mon[36878]: reconnect by client.15159 192.168.0.1:0/3463739313 after 0.0129998 2024-03-24T00:23:47.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:23:47 smithi193 ceph-mon[36878]: reconnect by client.15159 192.168.0.1:0/3463739313 after 0.0229995 2024-03-24T00:23:48.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:23:48 smithi118 ceph-mon[31535]: mds.? [v2:172.21.15.189:6836/2642988219,v1:172.21.15.189:6837/2642988219] up:rejoin 2024-03-24T00:23:48.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:23:48 smithi118 ceph-mon[31535]: mds.? [v2:172.21.15.118:6834/2587469815,v1:172.21.15.118:6835/2587469815] up:rejoin 2024-03-24T00:23:48.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:23:48 smithi118 ceph-mon[31535]: fsmap cephfs:5/5 3 up:standby-replay 1 up:standby 2 up:rejoin 3 up:active 2024-03-24T00:23:48.447 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:23:48 smithi118 ceph-mon[31535]: daemon mds.h is now active in filesystem cephfs as rank 1 2024-03-24T00:23:48.447 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:23:48 smithi118 ceph-mon[31535]: daemon mds.j is now active in filesystem cephfs as rank 3 2024-03-24T00:23:48.447 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:23:48 smithi118 ceph-mon[31535]: pgmap v667: 97 pgs: 97 active+clean; 146 MiB data, 850 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 42 KiB/s wr, 24 op/s 2024-03-24T00:23:48.447 INFO:journalctl@ceph.mds.j.smithi118.stdout:Mar 24 00:23:48 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-j[95671]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: In function 'void Locker::set_xlocks_done(MutationImpl*, bool)' thread 7ff24c400700 time 2024-03-24T00:23:48.082244+0000 2024-03-24T00:23:48.447 INFO:journalctl@ceph.mds.j.smithi118.stdout:Mar 24 00:23:48 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-j[95671]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: 626: FAILED ceph_assert(obj->is_auth()) 2024-03-24T00:23:48.447 INFO:journalctl@ceph.mds.j.smithi118.stdout:Mar 24 00:23:48 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-j[95671]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-24T00:23:48.447 INFO:journalctl@ceph.mds.j.smithi118.stdout:Mar 24 00:23:48 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-j[95671]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x135) [0x7ff257c964a1] 2024-03-24T00:23:48.447 INFO:journalctl@ceph.mds.j.smithi118.stdout:Mar 24 00:23:48 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-j[95671]: 2: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7ff257c96667] 2024-03-24T00:23:48.448 INFO:journalctl@ceph.mds.j.smithi118.stdout:Mar 24 00:23:48 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-j[95671]: 3: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x559190231223] 2024-03-24T00:23:48.448 INFO:journalctl@ceph.mds.j.smithi118.stdout:Mar 24 00:23:48 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-j[95671]: 4: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x5591900ab5d6] 2024-03-24T00:23:48.448 INFO:journalctl@ceph.mds.j.smithi118.stdout:Mar 24 00:23:48 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-j[95671]: 5: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x5591900abda1] 2024-03-24T00:23:48.448 INFO:journalctl@ceph.mds.j.smithi118.stdout:Mar 24 00:23:48 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-j[95671]: 6: (Server::handle_client_rename(boost::intrusive_ptr const&)+0x259d) [0x5591900e897d] 2024-03-24T00:23:48.448 INFO:journalctl@ceph.mds.j.smithi118.stdout:Mar 24 00:23:48 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-j[95671]: 7: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x5591900fe233] 2024-03-24T00:23:48.448 INFO:journalctl@ceph.mds.j.smithi118.stdout:Mar 24 00:23:48 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-j[95671]: 8: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x559190100197] 2024-03-24T00:23:48.448 INFO:journalctl@ceph.mds.j.smithi118.stdout:Mar 24 00:23:48 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-j[95671]: 9: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x55918ffff11b] 2024-03-24T00:23:48.448 INFO:journalctl@ceph.mds.j.smithi118.stdout:Mar 24 00:23:48 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-j[95671]: 10: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x55918ffff6bc] 2024-03-24T00:23:48.448 INFO:journalctl@ceph.mds.j.smithi118.stdout:Mar 24 00:23:48 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-j[95671]: 11: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x55918ffe91ff] 2024-03-24T00:23:48.448 INFO:journalctl@ceph.mds.j.smithi118.stdout:Mar 24 00:23:48 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-j[95671]: 12: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7ff257f28488] 2024-03-24T00:23:48.448 INFO:journalctl@ceph.mds.j.smithi118.stdout:Mar 24 00:23:48 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-j[95671]: 13: (DispatchQueue::entry()+0x51f) [0x7ff257f2560f] 2024-03-24T00:23:48.448 INFO:journalctl@ceph.mds.j.smithi118.stdout:Mar 24 00:23:48 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-j[95671]: 14: (DispatchQueue::DispatchThread::entry()+0x11) [0x7ff257ff8e81] 2024-03-24T00:23:48.448 INFO:journalctl@ceph.mds.j.smithi118.stdout:Mar 24 00:23:48 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-j[95671]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7ff256a131ca] 2024-03-24T00:23:48.448 INFO:journalctl@ceph.mds.j.smithi118.stdout:Mar 24 00:23:48 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-j[95671]: 16: clone() 2024-03-24T00:23:48.449 INFO:journalctl@ceph.mds.j.smithi118.stdout:Mar 24 00:23:48 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-j[95671]: 2024-03-24T00:23:48.083+0000 7ff24c400700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: In function 'void Locker::set_xlocks_done(MutationImpl*, bool)' thread 7ff24c400700 time 2024-03-24T00:23:48.082244+0000 2024-03-24T00:23:48.449 INFO:journalctl@ceph.mds.j.smithi118.stdout:Mar 24 00:23:48 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-j[95671]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: 626: FAILED ceph_assert(obj->is_auth()) 2024-03-24T00:23:48.449 INFO:journalctl@ceph.mds.j.smithi118.stdout:Mar 24 00:23:48 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-j[95671]: 2024-03-24T00:23:48.449 INFO:journalctl@ceph.mds.j.smithi118.stdout:Mar 24 00:23:48 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-j[95671]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-24T00:23:48.449 INFO:journalctl@ceph.mds.j.smithi118.stdout:Mar 24 00:23:48 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-j[95671]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x135) [0x7ff257c964a1] 2024-03-24T00:23:48.449 INFO:journalctl@ceph.mds.j.smithi118.stdout:Mar 24 00:23:48 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-j[95671]: 2: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7ff257c96667] 2024-03-24T00:23:48.449 INFO:journalctl@ceph.mds.j.smithi118.stdout:Mar 24 00:23:48 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-j[95671]: 3: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x559190231223] 2024-03-24T00:23:48.449 INFO:journalctl@ceph.mds.j.smithi118.stdout:Mar 24 00:23:48 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-j[95671]: 4: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x5591900ab5d6] 2024-03-24T00:23:48.449 INFO:journalctl@ceph.mds.j.smithi118.stdout:Mar 24 00:23:48 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-j[95671]: 5: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x5591900abda1] 2024-03-24T00:23:48.449 INFO:journalctl@ceph.mds.j.smithi118.stdout:Mar 24 00:23:48 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-j[95671]: 6: (Server::handle_client_rename(boost::intrusive_ptr const&)+0x259d) [0x5591900e897d] 2024-03-24T00:23:48.449 INFO:journalctl@ceph.mds.j.smithi118.stdout:Mar 24 00:23:48 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-j[95671]: 7: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x5591900fe233] 2024-03-24T00:23:48.450 INFO:journalctl@ceph.mds.j.smithi118.stdout:Mar 24 00:23:48 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-j[95671]: 8: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x559190100197] 2024-03-24T00:23:48.450 INFO:journalctl@ceph.mds.j.smithi118.stdout:Mar 24 00:23:48 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-j[95671]: 9: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x55918ffff11b] 2024-03-24T00:23:48.450 INFO:journalctl@ceph.mds.j.smithi118.stdout:Mar 24 00:23:48 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-j[95671]: 10: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x55918ffff6bc] 2024-03-24T00:23:48.450 INFO:journalctl@ceph.mds.j.smithi118.stdout:Mar 24 00:23:48 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-j[95671]: 11: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x55918ffe91ff] 2024-03-24T00:23:48.450 INFO:journalctl@ceph.mds.j.smithi118.stdout:Mar 24 00:23:48 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-j[95671]: 12: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7ff257f28488] 2024-03-24T00:23:48.450 INFO:journalctl@ceph.mds.j.smithi118.stdout:Mar 24 00:23:48 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-j[95671]: 13: (DispatchQueue::entry()+0x51f) [0x7ff257f2560f] 2024-03-24T00:23:48.450 INFO:journalctl@ceph.mds.j.smithi118.stdout:Mar 24 00:23:48 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-j[95671]: 14: (DispatchQueue::DispatchThread::entry()+0x11) [0x7ff257ff8e81] 2024-03-24T00:23:48.450 INFO:journalctl@ceph.mds.j.smithi118.stdout:Mar 24 00:23:48 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-j[95671]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7ff256a131ca] 2024-03-24T00:23:48.450 INFO:journalctl@ceph.mds.j.smithi118.stdout:Mar 24 00:23:48 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-j[95671]: 16: clone() 2024-03-24T00:23:48.450 INFO:journalctl@ceph.mds.j.smithi118.stdout:Mar 24 00:23:48 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-j[95671]: 2024-03-24T00:23:48.450 INFO:journalctl@ceph.mds.j.smithi118.stdout:Mar 24 00:23:48 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-j[95671]: *** Caught signal (Aborted) ** 2024-03-24T00:23:48.450 INFO:journalctl@ceph.mds.j.smithi118.stdout:Mar 24 00:23:48 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-j[95671]: in thread 7ff24c400700 thread_name:ms_dispatch 2024-03-24T00:23:48.450 INFO:journalctl@ceph.mds.j.smithi118.stdout:Mar 24 00:23:48 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-j[95671]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-24T00:23:48.450 INFO:journalctl@ceph.mds.j.smithi118.stdout:Mar 24 00:23:48 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-j[95671]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7ff256a1dd20] 2024-03-24T00:23:48.451 INFO:journalctl@ceph.mds.j.smithi118.stdout:Mar 24 00:23:48 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-j[95671]: 2: gsignal() 2024-03-24T00:23:48.451 INFO:journalctl@ceph.mds.j.smithi118.stdout:Mar 24 00:23:48 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-j[95671]: 3: abort() 2024-03-24T00:23:48.451 INFO:journalctl@ceph.mds.j.smithi118.stdout:Mar 24 00:23:48 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-j[95671]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x18f) [0x7ff257c964fb] 2024-03-24T00:23:48.451 INFO:journalctl@ceph.mds.j.smithi118.stdout:Mar 24 00:23:48 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-j[95671]: 5: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7ff257c96667] 2024-03-24T00:23:48.451 INFO:journalctl@ceph.mds.j.smithi118.stdout:Mar 24 00:23:48 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-j[95671]: 6: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x559190231223] 2024-03-24T00:23:48.451 INFO:journalctl@ceph.mds.j.smithi118.stdout:Mar 24 00:23:48 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-j[95671]: 7: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x5591900ab5d6] 2024-03-24T00:23:48.451 INFO:journalctl@ceph.mds.j.smithi118.stdout:Mar 24 00:23:48 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-j[95671]: 8: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x5591900abda1] 2024-03-24T00:23:48.451 INFO:journalctl@ceph.mds.j.smithi118.stdout:Mar 24 00:23:48 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-j[95671]: 9: (Server::handle_client_rename(boost::intrusive_ptr const&)+0x259d) [0x5591900e897d] 2024-03-24T00:23:48.451 INFO:journalctl@ceph.mds.j.smithi118.stdout:Mar 24 00:23:48 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-j[95671]: 10: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x5591900fe233] 2024-03-24T00:23:48.451 INFO:journalctl@ceph.mds.j.smithi118.stdout:Mar 24 00:23:48 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-j[95671]: 11: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x559190100197] 2024-03-24T00:23:48.451 INFO:journalctl@ceph.mds.j.smithi118.stdout:Mar 24 00:23:48 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-j[95671]: 12: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x55918ffff11b] 2024-03-24T00:23:48.451 INFO:journalctl@ceph.mds.j.smithi118.stdout:Mar 24 00:23:48 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-j[95671]: 13: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x55918ffff6bc] 2024-03-24T00:23:48.451 INFO:journalctl@ceph.mds.j.smithi118.stdout:Mar 24 00:23:48 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-j[95671]: 14: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x55918ffe91ff] 2024-03-24T00:23:48.452 INFO:journalctl@ceph.mds.j.smithi118.stdout:Mar 24 00:23:48 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-j[95671]: 15: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7ff257f28488] 2024-03-24T00:23:48.452 INFO:journalctl@ceph.mds.j.smithi118.stdout:Mar 24 00:23:48 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-j[95671]: 16: (DispatchQueue::entry()+0x51f) [0x7ff257f2560f] 2024-03-24T00:23:48.452 INFO:journalctl@ceph.mds.j.smithi118.stdout:Mar 24 00:23:48 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-j[95671]: 17: (DispatchQueue::DispatchThread::entry()+0x11) [0x7ff257ff8e81] 2024-03-24T00:23:48.452 INFO:journalctl@ceph.mds.j.smithi118.stdout:Mar 24 00:23:48 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-j[95671]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7ff256a131ca] 2024-03-24T00:23:48.452 INFO:journalctl@ceph.mds.j.smithi118.stdout:Mar 24 00:23:48 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-j[95671]: 19: clone() 2024-03-24T00:23:48.452 INFO:journalctl@ceph.mds.j.smithi118.stdout:Mar 24 00:23:48 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-j[95671]: 2024-03-24T00:23:48.086+0000 7ff24c400700 -1 *** Caught signal (Aborted) ** 2024-03-24T00:23:48.452 INFO:journalctl@ceph.mds.j.smithi118.stdout:Mar 24 00:23:48 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-j[95671]: in thread 7ff24c400700 thread_name:ms_dispatch 2024-03-24T00:23:48.452 INFO:journalctl@ceph.mds.j.smithi118.stdout:Mar 24 00:23:48 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-j[95671]: 2024-03-24T00:23:48.452 INFO:journalctl@ceph.mds.j.smithi118.stdout:Mar 24 00:23:48 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-j[95671]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-24T00:23:48.452 INFO:journalctl@ceph.mds.j.smithi118.stdout:Mar 24 00:23:48 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-j[95671]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7ff256a1dd20] 2024-03-24T00:23:48.452 INFO:journalctl@ceph.mds.j.smithi118.stdout:Mar 24 00:23:48 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-j[95671]: 2: gsignal() 2024-03-24T00:23:48.452 INFO:journalctl@ceph.mds.j.smithi118.stdout:Mar 24 00:23:48 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-j[95671]: 3: abort() 2024-03-24T00:23:48.452 INFO:journalctl@ceph.mds.j.smithi118.stdout:Mar 24 00:23:48 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-j[95671]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x18f) [0x7ff257c964fb] 2024-03-24T00:23:48.452 INFO:journalctl@ceph.mds.j.smithi118.stdout:Mar 24 00:23:48 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-j[95671]: 5: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7ff257c96667] 2024-03-24T00:23:48.453 INFO:journalctl@ceph.mds.j.smithi118.stdout:Mar 24 00:23:48 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-j[95671]: 6: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x559190231223] 2024-03-24T00:23:48.453 INFO:journalctl@ceph.mds.j.smithi118.stdout:Mar 24 00:23:48 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-j[95671]: 7: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x5591900ab5d6] 2024-03-24T00:23:48.453 INFO:journalctl@ceph.mds.j.smithi118.stdout:Mar 24 00:23:48 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-j[95671]: 8: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x5591900abda1] 2024-03-24T00:23:48.453 INFO:journalctl@ceph.mds.j.smithi118.stdout:Mar 24 00:23:48 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-j[95671]: 9: (Server::handle_client_rename(boost::intrusive_ptr const&)+0x259d) [0x5591900e897d] 2024-03-24T00:23:48.453 INFO:journalctl@ceph.mds.j.smithi118.stdout:Mar 24 00:23:48 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-j[95671]: 10: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x5591900fe233] 2024-03-24T00:23:48.453 INFO:journalctl@ceph.mds.j.smithi118.stdout:Mar 24 00:23:48 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-j[95671]: 11: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x559190100197] 2024-03-24T00:23:48.453 INFO:journalctl@ceph.mds.j.smithi118.stdout:Mar 24 00:23:48 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-j[95671]: 12: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x55918ffff11b] 2024-03-24T00:23:48.453 INFO:journalctl@ceph.mds.j.smithi118.stdout:Mar 24 00:23:48 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-j[95671]: 13: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x55918ffff6bc] 2024-03-24T00:23:48.453 INFO:journalctl@ceph.mds.j.smithi118.stdout:Mar 24 00:23:48 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-j[95671]: 14: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x55918ffe91ff] 2024-03-24T00:23:48.453 INFO:journalctl@ceph.mds.j.smithi118.stdout:Mar 24 00:23:48 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-j[95671]: 15: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7ff257f28488] 2024-03-24T00:23:48.453 INFO:journalctl@ceph.mds.j.smithi118.stdout:Mar 24 00:23:48 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-j[95671]: 16: (DispatchQueue::entry()+0x51f) [0x7ff257f2560f] 2024-03-24T00:23:48.453 INFO:journalctl@ceph.mds.j.smithi118.stdout:Mar 24 00:23:48 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-j[95671]: 17: (DispatchQueue::DispatchThread::entry()+0x11) [0x7ff257ff8e81] 2024-03-24T00:23:48.454 INFO:journalctl@ceph.mds.j.smithi118.stdout:Mar 24 00:23:48 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-j[95671]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7ff256a131ca] 2024-03-24T00:23:48.454 INFO:journalctl@ceph.mds.j.smithi118.stdout:Mar 24 00:23:48 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-j[95671]: 19: clone() 2024-03-24T00:23:48.454 INFO:journalctl@ceph.mds.j.smithi118.stdout:Mar 24 00:23:48 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-j[95671]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-03-24T00:23:48.454 INFO:journalctl@ceph.mds.j.smithi118.stdout:Mar 24 00:23:48 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-j[95671]: 2024-03-24T00:23:48.454 INFO:journalctl@ceph.mds.j.smithi118.stdout:Mar 24 00:23:48 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-j[95671]: -1> 2024-03-24T00:23:48.083+0000 7ff24c400700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: In function 'void Locker::set_xlocks_done(MutationImpl*, bool)' thread 7ff24c400700 time 2024-03-24T00:23:48.082244+0000 2024-03-24T00:23:48.454 INFO:journalctl@ceph.mds.j.smithi118.stdout:Mar 24 00:23:48 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-j[95671]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: 626: FAILED ceph_assert(obj->is_auth()) 2024-03-24T00:23:48.454 INFO:journalctl@ceph.mds.j.smithi118.stdout:Mar 24 00:23:48 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-j[95671]: 2024-03-24T00:23:48.454 INFO:journalctl@ceph.mds.j.smithi118.stdout:Mar 24 00:23:48 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-j[95671]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-24T00:23:48.454 INFO:journalctl@ceph.mds.j.smithi118.stdout:Mar 24 00:23:48 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-j[95671]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x135) [0x7ff257c964a1] 2024-03-24T00:23:48.454 INFO:journalctl@ceph.mds.j.smithi118.stdout:Mar 24 00:23:48 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-j[95671]: 2: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7ff257c96667] 2024-03-24T00:23:48.455 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:23:48 smithi189 ceph-mon[36187]: mds.? [v2:172.21.15.189:6836/2642988219,v1:172.21.15.189:6837/2642988219] up:rejoin 2024-03-24T00:23:48.455 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:23:48 smithi189 ceph-mon[36187]: mds.? [v2:172.21.15.118:6834/2587469815,v1:172.21.15.118:6835/2587469815] up:rejoin 2024-03-24T00:23:48.455 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:23:48 smithi189 ceph-mon[36187]: fsmap cephfs:5/5 3 up:standby-replay 1 up:standby 2 up:rejoin 3 up:active 2024-03-24T00:23:48.455 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:23:48 smithi189 ceph-mon[36187]: daemon mds.h is now active in filesystem cephfs as rank 1 2024-03-24T00:23:48.455 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:23:48 smithi189 ceph-mon[36187]: daemon mds.j is now active in filesystem cephfs as rank 3 2024-03-24T00:23:48.455 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:23:48 smithi189 ceph-mon[36187]: pgmap v667: 97 pgs: 97 active+clean; 146 MiB data, 850 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 42 KiB/s wr, 24 op/s 2024-03-24T00:23:48.456 INFO:journalctl@ceph.mds.j.smithi118.stdout:Mar 24 00:23:48 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-j[95671]: 3: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x559190231223] 2024-03-24T00:23:48.456 INFO:journalctl@ceph.mds.j.smithi118.stdout:Mar 24 00:23:48 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-j[95671]: 4: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x5591900ab5d6] 2024-03-24T00:23:48.456 INFO:journalctl@ceph.mds.j.smithi118.stdout:Mar 24 00:23:48 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-j[95671]: 5: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x5591900abda1] 2024-03-24T00:23:48.456 INFO:journalctl@ceph.mds.j.smithi118.stdout:Mar 24 00:23:48 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-j[95671]: 6: (Server::handle_client_rename(boost::intrusive_ptr const&)+0x259d) [0x5591900e897d] 2024-03-24T00:23:48.456 INFO:journalctl@ceph.mds.j.smithi118.stdout:Mar 24 00:23:48 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-j[95671]: 7: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x5591900fe233] 2024-03-24T00:23:48.456 INFO:journalctl@ceph.mds.j.smithi118.stdout:Mar 24 00:23:48 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-j[95671]: 8: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x559190100197] 2024-03-24T00:23:48.456 INFO:journalctl@ceph.mds.j.smithi118.stdout:Mar 24 00:23:48 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-j[95671]: 9: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x55918ffff11b] 2024-03-24T00:23:48.456 INFO:journalctl@ceph.mds.j.smithi118.stdout:Mar 24 00:23:48 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-j[95671]: 10: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x55918ffff6bc] 2024-03-24T00:23:48.456 INFO:journalctl@ceph.mds.j.smithi118.stdout:Mar 24 00:23:48 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-j[95671]: 11: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x55918ffe91ff] 2024-03-24T00:23:48.456 INFO:journalctl@ceph.mds.j.smithi118.stdout:Mar 24 00:23:48 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-j[95671]: 12: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7ff257f28488] 2024-03-24T00:23:48.457 INFO:journalctl@ceph.mds.j.smithi118.stdout:Mar 24 00:23:48 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-j[95671]: 13: (DispatchQueue::entry()+0x51f) [0x7ff257f2560f] 2024-03-24T00:23:48.457 INFO:journalctl@ceph.mds.j.smithi118.stdout:Mar 24 00:23:48 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-j[95671]: 14: (DispatchQueue::DispatchThread::entry()+0x11) [0x7ff257ff8e81] 2024-03-24T00:23:48.457 INFO:journalctl@ceph.mds.j.smithi118.stdout:Mar 24 00:23:48 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-j[95671]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7ff256a131ca] 2024-03-24T00:23:48.457 INFO:journalctl@ceph.mds.j.smithi118.stdout:Mar 24 00:23:48 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-j[95671]: 16: clone() 2024-03-24T00:23:48.457 INFO:journalctl@ceph.mds.j.smithi118.stdout:Mar 24 00:23:48 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-j[95671]: 2024-03-24T00:23:48.457 INFO:journalctl@ceph.mds.j.smithi118.stdout:Mar 24 00:23:48 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-j[95671]: 0> 2024-03-24T00:23:48.086+0000 7ff24c400700 -1 *** Caught signal (Aborted) ** 2024-03-24T00:23:48.457 INFO:journalctl@ceph.mds.j.smithi118.stdout:Mar 24 00:23:48 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-j[95671]: in thread 7ff24c400700 thread_name:ms_dispatch 2024-03-24T00:23:48.457 INFO:journalctl@ceph.mds.j.smithi118.stdout:Mar 24 00:23:48 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-j[95671]: 2024-03-24T00:23:48.457 INFO:journalctl@ceph.mds.j.smithi118.stdout:Mar 24 00:23:48 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-j[95671]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-24T00:23:48.457 INFO:journalctl@ceph.mds.j.smithi118.stdout:Mar 24 00:23:48 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-j[95671]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7ff256a1dd20] 2024-03-24T00:23:48.457 INFO:journalctl@ceph.mds.j.smithi118.stdout:Mar 24 00:23:48 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-j[95671]: 2: gsignal() 2024-03-24T00:23:48.457 INFO:journalctl@ceph.mds.j.smithi118.stdout:Mar 24 00:23:48 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-j[95671]: 3: abort() 2024-03-24T00:23:48.457 INFO:journalctl@ceph.mds.j.smithi118.stdout:Mar 24 00:23:48 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-j[95671]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x18f) [0x7ff257c964fb] 2024-03-24T00:23:48.457 INFO:journalctl@ceph.mds.j.smithi118.stdout:Mar 24 00:23:48 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-j[95671]: 5: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7ff257c96667] 2024-03-24T00:23:48.458 INFO:journalctl@ceph.mds.j.smithi118.stdout:Mar 24 00:23:48 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-j[95671]: 6: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x559190231223] 2024-03-24T00:23:48.458 INFO:journalctl@ceph.mds.j.smithi118.stdout:Mar 24 00:23:48 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-j[95671]: 7: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x5591900ab5d6] 2024-03-24T00:23:48.458 INFO:journalctl@ceph.mds.j.smithi118.stdout:Mar 24 00:23:48 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-j[95671]: 8: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x5591900abda1] 2024-03-24T00:23:48.458 INFO:journalctl@ceph.mds.j.smithi118.stdout:Mar 24 00:23:48 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-j[95671]: 9: (Server::handle_client_rename(boost::intrusive_ptr const&)+0x259d) [0x5591900e897d] 2024-03-24T00:23:48.458 INFO:journalctl@ceph.mds.j.smithi118.stdout:Mar 24 00:23:48 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-j[95671]: 10: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x5591900fe233] 2024-03-24T00:23:48.458 INFO:journalctl@ceph.mds.j.smithi118.stdout:Mar 24 00:23:48 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-j[95671]: 11: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x559190100197] 2024-03-24T00:23:48.458 INFO:journalctl@ceph.mds.j.smithi118.stdout:Mar 24 00:23:48 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-j[95671]: 12: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x55918ffff11b] 2024-03-24T00:23:48.458 INFO:journalctl@ceph.mds.j.smithi118.stdout:Mar 24 00:23:48 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-j[95671]: 13: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x55918ffff6bc] 2024-03-24T00:23:48.458 INFO:journalctl@ceph.mds.j.smithi118.stdout:Mar 24 00:23:48 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-j[95671]: 14: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x55918ffe91ff] 2024-03-24T00:23:48.458 INFO:journalctl@ceph.mds.j.smithi118.stdout:Mar 24 00:23:48 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-j[95671]: 15: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7ff257f28488] 2024-03-24T00:23:48.458 INFO:journalctl@ceph.mds.j.smithi118.stdout:Mar 24 00:23:48 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-j[95671]: 16: (DispatchQueue::entry()+0x51f) [0x7ff257f2560f] 2024-03-24T00:23:48.458 INFO:journalctl@ceph.mds.j.smithi118.stdout:Mar 24 00:23:48 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-j[95671]: 17: (DispatchQueue::DispatchThread::entry()+0x11) [0x7ff257ff8e81] 2024-03-24T00:23:48.458 INFO:journalctl@ceph.mds.j.smithi118.stdout:Mar 24 00:23:48 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-j[95671]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7ff256a131ca] 2024-03-24T00:23:48.458 INFO:journalctl@ceph.mds.j.smithi118.stdout:Mar 24 00:23:48 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-j[95671]: 19: clone() 2024-03-24T00:23:48.458 INFO:journalctl@ceph.mds.j.smithi118.stdout:Mar 24 00:23:48 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-j[95671]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-03-24T00:23:48.458 INFO:journalctl@ceph.mds.j.smithi118.stdout:Mar 24 00:23:48 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-j[95671]: 2024-03-24T00:23:48.459 INFO:journalctl@ceph.mds.j.smithi118.stdout:Mar 24 00:23:48 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-j[95671]: -1> 2024-03-24T00:23:48.083+0000 7ff24c400700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: In function 'void Locker::set_xlocks_done(MutationImpl*, bool)' thread 7ff24c400700 time 2024-03-24T00:23:48.082244+0000 2024-03-24T00:23:48.459 INFO:journalctl@ceph.mds.j.smithi118.stdout:Mar 24 00:23:48 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-j[95671]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: 626: FAILED ceph_assert(obj->is_auth()) 2024-03-24T00:23:48.459 INFO:journalctl@ceph.mds.j.smithi118.stdout:Mar 24 00:23:48 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-j[95671]: 2024-03-24T00:23:48.459 INFO:journalctl@ceph.mds.j.smithi118.stdout:Mar 24 00:23:48 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-j[95671]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-24T00:23:48.459 INFO:journalctl@ceph.mds.j.smithi118.stdout:Mar 24 00:23:48 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-j[95671]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x135) [0x7ff257c964a1] 2024-03-24T00:23:48.459 INFO:journalctl@ceph.mds.j.smithi118.stdout:Mar 24 00:23:48 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-j[95671]: 2: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7ff257c96667] 2024-03-24T00:23:48.459 INFO:journalctl@ceph.mds.j.smithi118.stdout:Mar 24 00:23:48 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-j[95671]: 3: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x559190231223] 2024-03-24T00:23:48.459 INFO:journalctl@ceph.mds.j.smithi118.stdout:Mar 24 00:23:48 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-j[95671]: 4: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x5591900ab5d6] 2024-03-24T00:23:48.460 INFO:journalctl@ceph.mds.h.smithi189.stdout:Mar 24 00:23:48 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-h[70521]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: In function 'void Locker::set_xlocks_done(MutationImpl*, bool)' thread 7f2b81978700 time 2024-03-24T00:23:48.108985+0000 2024-03-24T00:23:48.460 INFO:journalctl@ceph.mds.h.smithi189.stdout:Mar 24 00:23:48 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-h[70521]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: 626: FAILED ceph_assert(obj->is_auth()) 2024-03-24T00:23:48.460 INFO:journalctl@ceph.mds.h.smithi189.stdout:Mar 24 00:23:48 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-h[70521]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-24T00:23:48.460 INFO:journalctl@ceph.mds.h.smithi189.stdout:Mar 24 00:23:48 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-h[70521]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x135) [0x7f2b8bfa24a1] 2024-03-24T00:23:48.460 INFO:journalctl@ceph.mds.h.smithi189.stdout:Mar 24 00:23:48 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-h[70521]: 2: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7f2b8bfa2667] 2024-03-24T00:23:48.460 INFO:journalctl@ceph.mds.h.smithi189.stdout:Mar 24 00:23:48 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-h[70521]: 3: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x557c8c2c3223] 2024-03-24T00:23:48.460 INFO:journalctl@ceph.mds.h.smithi189.stdout:Mar 24 00:23:48 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-h[70521]: 4: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x557c8c13d5d6] 2024-03-24T00:23:48.460 INFO:journalctl@ceph.mds.h.smithi189.stdout:Mar 24 00:23:48 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-h[70521]: 5: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x557c8c13dda1] 2024-03-24T00:23:48.460 INFO:journalctl@ceph.mds.h.smithi189.stdout:Mar 24 00:23:48 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-h[70521]: 6: (Server::handle_client_rename(boost::intrusive_ptr const&)+0x259d) [0x557c8c17a97d] 2024-03-24T00:23:48.461 INFO:journalctl@ceph.mds.h.smithi189.stdout:Mar 24 00:23:48 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-h[70521]: 7: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x557c8c190233] 2024-03-24T00:23:48.461 INFO:journalctl@ceph.mds.h.smithi189.stdout:Mar 24 00:23:48 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-h[70521]: 8: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x557c8c192197] 2024-03-24T00:23:48.461 INFO:journalctl@ceph.mds.h.smithi189.stdout:Mar 24 00:23:48 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-h[70521]: 9: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x557c8c09111b] 2024-03-24T00:23:48.461 INFO:journalctl@ceph.mds.h.smithi189.stdout:Mar 24 00:23:48 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-h[70521]: 10: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x557c8c0916bc] 2024-03-24T00:23:48.461 INFO:journalctl@ceph.mds.h.smithi189.stdout:Mar 24 00:23:48 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-h[70521]: 11: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x557c8c07b1ff] 2024-03-24T00:23:48.461 INFO:journalctl@ceph.mds.h.smithi189.stdout:Mar 24 00:23:48 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-h[70521]: 12: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7f2b8c234488] 2024-03-24T00:23:48.461 INFO:journalctl@ceph.mds.h.smithi189.stdout:Mar 24 00:23:48 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-h[70521]: 13: (DispatchQueue::entry()+0x51f) [0x7f2b8c23160f] 2024-03-24T00:23:48.461 INFO:journalctl@ceph.mds.h.smithi189.stdout:Mar 24 00:23:48 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-h[70521]: 14: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f2b8c304e81] 2024-03-24T00:23:48.461 INFO:journalctl@ceph.mds.h.smithi189.stdout:Mar 24 00:23:48 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-h[70521]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7f2b8ad1f1ca] 2024-03-24T00:23:48.461 INFO:journalctl@ceph.mds.h.smithi189.stdout:Mar 24 00:23:48 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-h[70521]: 16: clone() 2024-03-24T00:23:48.461 INFO:journalctl@ceph.mds.h.smithi189.stdout:Mar 24 00:23:48 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-h[70521]: 2024-03-24T00:23:48.109+0000 7f2b81978700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: In function 'void Locker::set_xlocks_done(MutationImpl*, bool)' thread 7f2b81978700 time 2024-03-24T00:23:48.108985+0000 2024-03-24T00:23:48.461 INFO:journalctl@ceph.mds.h.smithi189.stdout:Mar 24 00:23:48 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-h[70521]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: 626: FAILED ceph_assert(obj->is_auth()) 2024-03-24T00:23:48.461 INFO:journalctl@ceph.mds.h.smithi189.stdout:Mar 24 00:23:48 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-h[70521]: 2024-03-24T00:23:48.461 INFO:journalctl@ceph.mds.h.smithi189.stdout:Mar 24 00:23:48 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-h[70521]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-24T00:23:48.461 INFO:journalctl@ceph.mds.h.smithi189.stdout:Mar 24 00:23:48 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-h[70521]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x135) [0x7f2b8bfa24a1] 2024-03-24T00:23:48.462 INFO:journalctl@ceph.mds.h.smithi189.stdout:Mar 24 00:23:48 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-h[70521]: 2: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7f2b8bfa2667] 2024-03-24T00:23:48.462 INFO:journalctl@ceph.mds.h.smithi189.stdout:Mar 24 00:23:48 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-h[70521]: 3: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x557c8c2c3223] 2024-03-24T00:23:48.462 INFO:journalctl@ceph.mds.h.smithi189.stdout:Mar 24 00:23:48 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-h[70521]: 4: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x557c8c13d5d6] 2024-03-24T00:23:48.462 INFO:journalctl@ceph.mds.h.smithi189.stdout:Mar 24 00:23:48 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-h[70521]: 5: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x557c8c13dda1] 2024-03-24T00:23:48.462 INFO:journalctl@ceph.mds.h.smithi189.stdout:Mar 24 00:23:48 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-h[70521]: 6: (Server::handle_client_rename(boost::intrusive_ptr const&)+0x259d) [0x557c8c17a97d] 2024-03-24T00:23:48.462 INFO:journalctl@ceph.mds.h.smithi189.stdout:Mar 24 00:23:48 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-h[70521]: 7: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x557c8c190233] 2024-03-24T00:23:48.462 INFO:journalctl@ceph.mds.h.smithi189.stdout:Mar 24 00:23:48 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-h[70521]: 8: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x557c8c192197] 2024-03-24T00:23:48.462 INFO:journalctl@ceph.mds.h.smithi189.stdout:Mar 24 00:23:48 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-h[70521]: 9: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x557c8c09111b] 2024-03-24T00:23:48.462 INFO:journalctl@ceph.mds.h.smithi189.stdout:Mar 24 00:23:48 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-h[70521]: 10: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x557c8c0916bc] 2024-03-24T00:23:48.462 INFO:journalctl@ceph.mds.h.smithi189.stdout:Mar 24 00:23:48 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-h[70521]: 11: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x557c8c07b1ff] 2024-03-24T00:23:48.462 INFO:journalctl@ceph.mds.h.smithi189.stdout:Mar 24 00:23:48 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-h[70521]: 12: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7f2b8c234488] 2024-03-24T00:23:48.462 INFO:journalctl@ceph.mds.h.smithi189.stdout:Mar 24 00:23:48 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-h[70521]: 13: (DispatchQueue::entry()+0x51f) [0x7f2b8c23160f] 2024-03-24T00:23:48.462 INFO:journalctl@ceph.mds.h.smithi189.stdout:Mar 24 00:23:48 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-h[70521]: 14: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f2b8c304e81] 2024-03-24T00:23:48.463 INFO:journalctl@ceph.mds.h.smithi189.stdout:Mar 24 00:23:48 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-h[70521]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7f2b8ad1f1ca] 2024-03-24T00:23:48.463 INFO:journalctl@ceph.mds.h.smithi189.stdout:Mar 24 00:23:48 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-h[70521]: 16: clone() 2024-03-24T00:23:48.463 INFO:journalctl@ceph.mds.h.smithi189.stdout:Mar 24 00:23:48 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-h[70521]: 2024-03-24T00:23:48.463 INFO:journalctl@ceph.mds.h.smithi189.stdout:Mar 24 00:23:48 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-h[70521]: *** Caught signal (Aborted) ** 2024-03-24T00:23:48.463 INFO:journalctl@ceph.mds.h.smithi189.stdout:Mar 24 00:23:48 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-h[70521]: in thread 7f2b81978700 thread_name:ms_dispatch 2024-03-24T00:23:48.463 INFO:journalctl@ceph.mds.h.smithi189.stdout:Mar 24 00:23:48 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-h[70521]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-24T00:23:48.463 INFO:journalctl@ceph.mds.h.smithi189.stdout:Mar 24 00:23:48 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-h[70521]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f2b8ad29d20] 2024-03-24T00:23:48.463 INFO:journalctl@ceph.mds.h.smithi189.stdout:Mar 24 00:23:48 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-h[70521]: 2: gsignal() 2024-03-24T00:23:48.463 INFO:journalctl@ceph.mds.h.smithi189.stdout:Mar 24 00:23:48 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-h[70521]: 3: abort() 2024-03-24T00:23:48.463 INFO:journalctl@ceph.mds.h.smithi189.stdout:Mar 24 00:23:48 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-h[70521]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x18f) [0x7f2b8bfa24fb] 2024-03-24T00:23:48.463 INFO:journalctl@ceph.mds.h.smithi189.stdout:Mar 24 00:23:48 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-h[70521]: 5: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7f2b8bfa2667] 2024-03-24T00:23:48.463 INFO:journalctl@ceph.mds.h.smithi189.stdout:Mar 24 00:23:48 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-h[70521]: 6: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x557c8c2c3223] 2024-03-24T00:23:48.463 INFO:journalctl@ceph.mds.h.smithi189.stdout:Mar 24 00:23:48 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-h[70521]: 7: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x557c8c13d5d6] 2024-03-24T00:23:48.463 INFO:journalctl@ceph.mds.h.smithi189.stdout:Mar 24 00:23:48 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-h[70521]: 8: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x557c8c13dda1] 2024-03-24T00:23:48.464 INFO:journalctl@ceph.mds.h.smithi189.stdout:Mar 24 00:23:48 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-h[70521]: 9: (Server::handle_client_rename(boost::intrusive_ptr const&)+0x259d) [0x557c8c17a97d] 2024-03-24T00:23:48.464 INFO:journalctl@ceph.mds.h.smithi189.stdout:Mar 24 00:23:48 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-h[70521]: 10: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x557c8c190233] 2024-03-24T00:23:48.464 INFO:journalctl@ceph.mds.h.smithi189.stdout:Mar 24 00:23:48 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-h[70521]: 11: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x557c8c192197] 2024-03-24T00:23:48.464 INFO:journalctl@ceph.mds.h.smithi189.stdout:Mar 24 00:23:48 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-h[70521]: 12: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x557c8c09111b] 2024-03-24T00:23:48.464 INFO:journalctl@ceph.mds.h.smithi189.stdout:Mar 24 00:23:48 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-h[70521]: 13: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x557c8c0916bc] 2024-03-24T00:23:48.464 INFO:journalctl@ceph.mds.h.smithi189.stdout:Mar 24 00:23:48 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-h[70521]: 14: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x557c8c07b1ff] 2024-03-24T00:23:48.464 INFO:journalctl@ceph.mds.h.smithi189.stdout:Mar 24 00:23:48 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-h[70521]: 15: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7f2b8c234488] 2024-03-24T00:23:48.464 INFO:journalctl@ceph.mds.h.smithi189.stdout:Mar 24 00:23:48 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-h[70521]: 16: (DispatchQueue::entry()+0x51f) [0x7f2b8c23160f] 2024-03-24T00:23:48.464 INFO:journalctl@ceph.mds.h.smithi189.stdout:Mar 24 00:23:48 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-h[70521]: 17: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f2b8c304e81] 2024-03-24T00:23:48.464 INFO:journalctl@ceph.mds.h.smithi189.stdout:Mar 24 00:23:48 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-h[70521]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7f2b8ad1f1ca] 2024-03-24T00:23:48.464 INFO:journalctl@ceph.mds.h.smithi189.stdout:Mar 24 00:23:48 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-h[70521]: 19: clone() 2024-03-24T00:23:48.464 INFO:journalctl@ceph.mds.h.smithi189.stdout:Mar 24 00:23:48 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-h[70521]: 2024-03-24T00:23:48.111+0000 7f2b81978700 -1 *** Caught signal (Aborted) ** 2024-03-24T00:23:48.464 INFO:journalctl@ceph.mds.h.smithi189.stdout:Mar 24 00:23:48 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-h[70521]: in thread 7f2b81978700 thread_name:ms_dispatch 2024-03-24T00:23:48.464 INFO:journalctl@ceph.mds.h.smithi189.stdout:Mar 24 00:23:48 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-h[70521]: 2024-03-24T00:23:48.464 INFO:journalctl@ceph.mds.h.smithi189.stdout:Mar 24 00:23:48 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-h[70521]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-24T00:23:48.464 INFO:journalctl@ceph.mds.h.smithi189.stdout:Mar 24 00:23:48 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-h[70521]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f2b8ad29d20] 2024-03-24T00:23:48.465 INFO:journalctl@ceph.mds.h.smithi189.stdout:Mar 24 00:23:48 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-h[70521]: 2: gsignal() 2024-03-24T00:23:48.465 INFO:journalctl@ceph.mds.h.smithi189.stdout:Mar 24 00:23:48 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-h[70521]: 3: abort() 2024-03-24T00:23:48.465 INFO:journalctl@ceph.mds.h.smithi189.stdout:Mar 24 00:23:48 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-h[70521]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x18f) [0x7f2b8bfa24fb] 2024-03-24T00:23:48.465 INFO:journalctl@ceph.mds.h.smithi189.stdout:Mar 24 00:23:48 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-h[70521]: 5: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7f2b8bfa2667] 2024-03-24T00:23:48.465 INFO:journalctl@ceph.mds.h.smithi189.stdout:Mar 24 00:23:48 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-h[70521]: 6: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x557c8c2c3223] 2024-03-24T00:23:48.465 INFO:journalctl@ceph.mds.h.smithi189.stdout:Mar 24 00:23:48 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-h[70521]: 7: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x557c8c13d5d6] 2024-03-24T00:23:48.465 INFO:journalctl@ceph.mds.h.smithi189.stdout:Mar 24 00:23:48 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-h[70521]: 8: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x557c8c13dda1] 2024-03-24T00:23:48.465 INFO:journalctl@ceph.mds.h.smithi189.stdout:Mar 24 00:23:48 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-h[70521]: 9: (Server::handle_client_rename(boost::intrusive_ptr const&)+0x259d) [0x557c8c17a97d] 2024-03-24T00:23:48.465 INFO:journalctl@ceph.mds.h.smithi189.stdout:Mar 24 00:23:48 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-h[70521]: 10: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x557c8c190233] 2024-03-24T00:23:48.465 INFO:journalctl@ceph.mds.h.smithi189.stdout:Mar 24 00:23:48 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-h[70521]: 11: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x557c8c192197] 2024-03-24T00:23:48.465 INFO:journalctl@ceph.mds.h.smithi189.stdout:Mar 24 00:23:48 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-h[70521]: 12: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x557c8c09111b] 2024-03-24T00:23:48.465 INFO:journalctl@ceph.mds.h.smithi189.stdout:Mar 24 00:23:48 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-h[70521]: 13: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x557c8c0916bc] 2024-03-24T00:23:48.465 INFO:journalctl@ceph.mds.h.smithi189.stdout:Mar 24 00:23:48 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-h[70521]: 14: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x557c8c07b1ff] 2024-03-24T00:23:48.465 INFO:journalctl@ceph.mds.h.smithi189.stdout:Mar 24 00:23:48 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-h[70521]: 15: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7f2b8c234488] 2024-03-24T00:23:48.465 INFO:journalctl@ceph.mds.h.smithi189.stdout:Mar 24 00:23:48 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-h[70521]: 16: (DispatchQueue::entry()+0x51f) [0x7f2b8c23160f] 2024-03-24T00:23:48.466 INFO:journalctl@ceph.mds.h.smithi189.stdout:Mar 24 00:23:48 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-h[70521]: 17: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f2b8c304e81] 2024-03-24T00:23:48.466 INFO:journalctl@ceph.mds.h.smithi189.stdout:Mar 24 00:23:48 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-h[70521]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7f2b8ad1f1ca] 2024-03-24T00:23:48.466 INFO:journalctl@ceph.mds.h.smithi189.stdout:Mar 24 00:23:48 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-h[70521]: 19: clone() 2024-03-24T00:23:48.466 INFO:journalctl@ceph.mds.h.smithi189.stdout:Mar 24 00:23:48 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-h[70521]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-03-24T00:23:48.466 INFO:journalctl@ceph.mds.h.smithi189.stdout:Mar 24 00:23:48 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-h[70521]: 2024-03-24T00:23:48.466 INFO:journalctl@ceph.mds.h.smithi189.stdout:Mar 24 00:23:48 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-h[70521]: -1> 2024-03-24T00:23:48.109+0000 7f2b81978700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: In function 'void Locker::set_xlocks_done(MutationImpl*, bool)' thread 7f2b81978700 time 2024-03-24T00:23:48.108985+0000 2024-03-24T00:23:48.466 INFO:journalctl@ceph.mds.h.smithi189.stdout:Mar 24 00:23:48 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-h[70521]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: 626: FAILED ceph_assert(obj->is_auth()) 2024-03-24T00:23:48.466 INFO:journalctl@ceph.mds.h.smithi189.stdout:Mar 24 00:23:48 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-h[70521]: 2024-03-24T00:23:48.466 INFO:journalctl@ceph.mds.h.smithi189.stdout:Mar 24 00:23:48 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-h[70521]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-24T00:23:48.466 INFO:journalctl@ceph.mds.h.smithi189.stdout:Mar 24 00:23:48 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-h[70521]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x135) [0x7f2b8bfa24a1] 2024-03-24T00:23:48.466 INFO:journalctl@ceph.mds.h.smithi189.stdout:Mar 24 00:23:48 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-h[70521]: 2: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7f2b8bfa2667] 2024-03-24T00:23:48.466 INFO:journalctl@ceph.mds.h.smithi189.stdout:Mar 24 00:23:48 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-h[70521]: 3: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x557c8c2c3223] 2024-03-24T00:23:48.466 INFO:journalctl@ceph.mds.h.smithi189.stdout:Mar 24 00:23:48 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-h[70521]: 4: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x557c8c13d5d6] 2024-03-24T00:23:48.466 INFO:journalctl@ceph.mds.h.smithi189.stdout:Mar 24 00:23:48 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-h[70521]: 5: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x557c8c13dda1] 2024-03-24T00:23:48.467 INFO:journalctl@ceph.mds.h.smithi189.stdout:Mar 24 00:23:48 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-h[70521]: 6: (Server::handle_client_rename(boost::intrusive_ptr const&)+0x259d) [0x557c8c17a97d] 2024-03-24T00:23:48.467 INFO:journalctl@ceph.mds.h.smithi189.stdout:Mar 24 00:23:48 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-h[70521]: 7: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x557c8c190233] 2024-03-24T00:23:48.467 INFO:journalctl@ceph.mds.h.smithi189.stdout:Mar 24 00:23:48 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-h[70521]: 8: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x557c8c192197] 2024-03-24T00:23:48.467 INFO:journalctl@ceph.mds.h.smithi189.stdout:Mar 24 00:23:48 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-h[70521]: 9: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x557c8c09111b] 2024-03-24T00:23:48.467 INFO:journalctl@ceph.mds.h.smithi189.stdout:Mar 24 00:23:48 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-h[70521]: 10: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x557c8c0916bc] 2024-03-24T00:23:48.467 INFO:journalctl@ceph.mds.h.smithi189.stdout:Mar 24 00:23:48 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-h[70521]: 11: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x557c8c07b1ff] 2024-03-24T00:23:48.467 INFO:journalctl@ceph.mds.h.smithi189.stdout:Mar 24 00:23:48 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-h[70521]: 12: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7f2b8c234488] 2024-03-24T00:23:48.467 INFO:journalctl@ceph.mds.h.smithi189.stdout:Mar 24 00:23:48 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-h[70521]: 13: (DispatchQueue::entry()+0x51f) [0x7f2b8c23160f] 2024-03-24T00:23:48.467 INFO:journalctl@ceph.mds.h.smithi189.stdout:Mar 24 00:23:48 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-h[70521]: 14: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f2b8c304e81] 2024-03-24T00:23:48.467 INFO:journalctl@ceph.mds.h.smithi189.stdout:Mar 24 00:23:48 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-h[70521]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7f2b8ad1f1ca] 2024-03-24T00:23:48.467 INFO:journalctl@ceph.mds.h.smithi189.stdout:Mar 24 00:23:48 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-h[70521]: 16: clone() 2024-03-24T00:23:48.467 INFO:journalctl@ceph.mds.h.smithi189.stdout:Mar 24 00:23:48 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-h[70521]: 2024-03-24T00:23:48.467 INFO:journalctl@ceph.mds.h.smithi189.stdout:Mar 24 00:23:48 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-h[70521]: 0> 2024-03-24T00:23:48.111+0000 7f2b81978700 -1 *** Caught signal (Aborted) ** 2024-03-24T00:23:48.467 INFO:journalctl@ceph.mds.h.smithi189.stdout:Mar 24 00:23:48 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-h[70521]: in thread 7f2b81978700 thread_name:ms_dispatch 2024-03-24T00:23:48.467 INFO:journalctl@ceph.mds.h.smithi189.stdout:Mar 24 00:23:48 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-h[70521]: 2024-03-24T00:23:48.467 INFO:journalctl@ceph.mds.h.smithi189.stdout:Mar 24 00:23:48 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-h[70521]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-24T00:23:48.468 INFO:journalctl@ceph.mds.h.smithi189.stdout:Mar 24 00:23:48 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-h[70521]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f2b8ad29d20] 2024-03-24T00:23:48.468 INFO:journalctl@ceph.mds.h.smithi189.stdout:Mar 24 00:23:48 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-h[70521]: 2: gsignal() 2024-03-24T00:23:48.468 INFO:journalctl@ceph.mds.h.smithi189.stdout:Mar 24 00:23:48 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-h[70521]: 3: abort() 2024-03-24T00:23:48.468 INFO:journalctl@ceph.mds.h.smithi189.stdout:Mar 24 00:23:48 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-h[70521]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x18f) [0x7f2b8bfa24fb] 2024-03-24T00:23:48.468 INFO:journalctl@ceph.mds.h.smithi189.stdout:Mar 24 00:23:48 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-h[70521]: 5: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7f2b8bfa2667] 2024-03-24T00:23:48.468 INFO:journalctl@ceph.mds.h.smithi189.stdout:Mar 24 00:23:48 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-h[70521]: 6: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x557c8c2c3223] 2024-03-24T00:23:48.468 INFO:journalctl@ceph.mds.h.smithi189.stdout:Mar 24 00:23:48 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-h[70521]: 7: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x557c8c13d5d6] 2024-03-24T00:23:48.468 INFO:journalctl@ceph.mds.h.smithi189.stdout:Mar 24 00:23:48 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-h[70521]: 8: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x557c8c13dda1] 2024-03-24T00:23:48.468 INFO:journalctl@ceph.mds.h.smithi189.stdout:Mar 24 00:23:48 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-h[70521]: 9: (Server::handle_client_rename(boost::intrusive_ptr const&)+0x259d) [0x557c8c17a97d] 2024-03-24T00:23:48.468 INFO:journalctl@ceph.mds.h.smithi189.stdout:Mar 24 00:23:48 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-h[70521]: 10: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x557c8c190233] 2024-03-24T00:23:48.468 INFO:journalctl@ceph.mds.h.smithi189.stdout:Mar 24 00:23:48 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-h[70521]: 11: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x557c8c192197] 2024-03-24T00:23:48.468 INFO:journalctl@ceph.mds.h.smithi189.stdout:Mar 24 00:23:48 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-h[70521]: 12: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x557c8c09111b] 2024-03-24T00:23:48.468 INFO:journalctl@ceph.mds.h.smithi189.stdout:Mar 24 00:23:48 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-h[70521]: 13: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x557c8c0916bc] 2024-03-24T00:23:48.468 INFO:journalctl@ceph.mds.h.smithi189.stdout:Mar 24 00:23:48 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-h[70521]: 14: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x557c8c07b1ff] 2024-03-24T00:23:48.468 INFO:journalctl@ceph.mds.h.smithi189.stdout:Mar 24 00:23:48 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-h[70521]: 15: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7f2b8c234488] 2024-03-24T00:23:48.468 INFO:journalctl@ceph.mds.h.smithi189.stdout:Mar 24 00:23:48 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-h[70521]: 16: (DispatchQueue::entry()+0x51f) [0x7f2b8c23160f] 2024-03-24T00:23:48.469 INFO:journalctl@ceph.mds.h.smithi189.stdout:Mar 24 00:23:48 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-h[70521]: 17: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f2b8c304e81] 2024-03-24T00:23:48.469 INFO:journalctl@ceph.mds.h.smithi189.stdout:Mar 24 00:23:48 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-h[70521]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7f2b8ad1f1ca] 2024-03-24T00:23:48.469 INFO:journalctl@ceph.mds.h.smithi189.stdout:Mar 24 00:23:48 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-h[70521]: 19: clone() 2024-03-24T00:23:48.469 INFO:journalctl@ceph.mds.h.smithi189.stdout:Mar 24 00:23:48 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-h[70521]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-03-24T00:23:48.469 INFO:journalctl@ceph.mds.h.smithi189.stdout:Mar 24 00:23:48 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-h[70521]: 2024-03-24T00:23:48.469 INFO:journalctl@ceph.mds.h.smithi189.stdout:Mar 24 00:23:48 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-h[70521]: -5> 2024-03-24T00:23:48.109+0000 7f2b81978700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: In function 'void Locker::set_xlocks_done(MutationImpl*, bool)' thread 7f2b81978700 time 2024-03-24T00:23:48.108985+0000 2024-03-24T00:23:48.469 INFO:journalctl@ceph.mds.h.smithi189.stdout:Mar 24 00:23:48 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-h[70521]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: 626: FAILED ceph_assert(obj->is_auth()) 2024-03-24T00:23:48.469 INFO:journalctl@ceph.mds.h.smithi189.stdout:Mar 24 00:23:48 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-h[70521]: 2024-03-24T00:23:48.469 INFO:journalctl@ceph.mds.h.smithi189.stdout:Mar 24 00:23:48 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-h[70521]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-24T00:23:48.469 INFO:journalctl@ceph.mds.h.smithi189.stdout:Mar 24 00:23:48 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-h[70521]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x135) [0x7f2b8bfa24a1] 2024-03-24T00:23:48.469 INFO:journalctl@ceph.mds.h.smithi189.stdout:Mar 24 00:23:48 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-h[70521]: 2: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7f2b8bfa2667] 2024-03-24T00:23:48.469 INFO:journalctl@ceph.mds.h.smithi189.stdout:Mar 24 00:23:48 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-h[70521]: 3: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x557c8c2c3223] 2024-03-24T00:23:48.469 INFO:journalctl@ceph.mds.h.smithi189.stdout:Mar 24 00:23:48 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-h[70521]: 4: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x557c8c13d5d6] 2024-03-24T00:23:48.469 INFO:journalctl@ceph.mds.h.smithi189.stdout:Mar 24 00:23:48 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-h[70521]: 5: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x557c8c13dda1] 2024-03-24T00:23:48.470 INFO:journalctl@ceph.mds.h.smithi189.stdout:Mar 24 00:23:48 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-h[70521]: 6: (Server::handle_client_rename(boost::intrusive_ptr const&)+0x259d) [0x557c8c17a97d] 2024-03-24T00:23:48.470 INFO:journalctl@ceph.mds.h.smithi189.stdout:Mar 24 00:23:48 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-h[70521]: 7: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x557c8c190233] 2024-03-24T00:23:48.470 INFO:journalctl@ceph.mds.h.smithi189.stdout:Mar 24 00:23:48 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-h[70521]: 8: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x557c8c192197] 2024-03-24T00:23:48.470 INFO:journalctl@ceph.mds.h.smithi189.stdout:Mar 24 00:23:48 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-h[70521]: 9: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x557c8c09111b] 2024-03-24T00:23:48.470 INFO:journalctl@ceph.mds.h.smithi189.stdout:Mar 24 00:23:48 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-h[70521]: 10: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x557c8c0916bc] 2024-03-24T00:23:48.470 INFO:journalctl@ceph.mds.h.smithi189.stdout:Mar 24 00:23:48 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-h[70521]: 11: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x557c8c07b1ff] 2024-03-24T00:23:48.470 INFO:journalctl@ceph.mds.h.smithi189.stdout:Mar 24 00:23:48 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-h[70521]: 12: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7f2b8c234488] 2024-03-24T00:23:48.470 INFO:journalctl@ceph.mds.h.smithi189.stdout:Mar 24 00:23:48 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-h[70521]: 13: (DispatchQueue::entry()+0x51f) [0x7f2b8c23160f] 2024-03-24T00:23:48.470 INFO:journalctl@ceph.mds.h.smithi189.stdout:Mar 24 00:23:48 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-h[70521]: 14: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f2b8c304e81] 2024-03-24T00:23:48.470 INFO:journalctl@ceph.mds.h.smithi189.stdout:Mar 24 00:23:48 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-h[70521]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7f2b8ad1f1ca] 2024-03-24T00:23:48.470 INFO:journalctl@ceph.mds.h.smithi189.stdout:Mar 24 00:23:48 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-h[70521]: 16: clone() 2024-03-24T00:23:48.470 INFO:journalctl@ceph.mds.h.smithi189.stdout:Mar 24 00:23:48 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-h[70521]: 2024-03-24T00:23:48.471 INFO:journalctl@ceph.mds.h.smithi189.stdout:Mar 24 00:23:48 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-h[70521]: -4> 2024-03-24T00:23:48.111+0000 7f2b81978700 -1 *** Caught signal (Aborted) ** 2024-03-24T00:23:48.471 INFO:journalctl@ceph.mds.h.smithi189.stdout:Mar 24 00:23:48 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-h[70521]: in thread 7f2b81978700 thread_name:ms_dispatch 2024-03-24T00:23:48.471 INFO:journalctl@ceph.mds.h.smithi189.stdout:Mar 24 00:23:48 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-h[70521]: 2024-03-24T00:23:48.471 INFO:journalctl@ceph.mds.h.smithi189.stdout:Mar 24 00:23:48 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-h[70521]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-24T00:23:48.471 INFO:journalctl@ceph.mds.h.smithi189.stdout:Mar 24 00:23:48 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-h[70521]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f2b8ad29d20] 2024-03-24T00:23:48.471 INFO:journalctl@ceph.mds.h.smithi189.stdout:Mar 24 00:23:48 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-h[70521]: 2: gsignal() 2024-03-24T00:23:48.471 INFO:journalctl@ceph.mds.h.smithi189.stdout:Mar 24 00:23:48 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-h[70521]: 3: abort() 2024-03-24T00:23:48.471 INFO:journalctl@ceph.mds.h.smithi189.stdout:Mar 24 00:23:48 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-h[70521]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x18f) [0x7f2b8bfa24fb] 2024-03-24T00:23:48.471 INFO:journalctl@ceph.mds.h.smithi189.stdout:Mar 24 00:23:48 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-h[70521]: 5: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7f2b8bfa2667] 2024-03-24T00:23:48.471 INFO:journalctl@ceph.mds.h.smithi189.stdout:Mar 24 00:23:48 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-h[70521]: 6: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x557c8c2c3223] 2024-03-24T00:23:48.471 INFO:journalctl@ceph.mds.h.smithi189.stdout:Mar 24 00:23:48 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-h[70521]: 7: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x557c8c13d5d6] 2024-03-24T00:23:48.471 INFO:journalctl@ceph.mds.h.smithi189.stdout:Mar 24 00:23:48 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-h[70521]: 8: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x557c8c13dda1] 2024-03-24T00:23:48.471 INFO:journalctl@ceph.mds.h.smithi189.stdout:Mar 24 00:23:48 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-h[70521]: 9: (Server::handle_client_rename(boost::intrusive_ptr const&)+0x259d) [0x557c8c17a97d] 2024-03-24T00:23:48.471 INFO:journalctl@ceph.mds.h.smithi189.stdout:Mar 24 00:23:48 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-h[70521]: 10: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x557c8c190233] 2024-03-24T00:23:48.471 INFO:journalctl@ceph.mds.h.smithi189.stdout:Mar 24 00:23:48 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-h[70521]: 11: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x557c8c192197] 2024-03-24T00:23:48.471 INFO:journalctl@ceph.mds.h.smithi189.stdout:Mar 24 00:23:48 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-h[70521]: 12: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x557c8c09111b] 2024-03-24T00:23:48.472 INFO:journalctl@ceph.mds.h.smithi189.stdout:Mar 24 00:23:48 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-h[70521]: 13: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x557c8c0916bc] 2024-03-24T00:23:48.472 INFO:journalctl@ceph.mds.h.smithi189.stdout:Mar 24 00:23:48 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-h[70521]: 14: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x557c8c07b1ff] 2024-03-24T00:23:48.472 INFO:journalctl@ceph.mds.h.smithi189.stdout:Mar 24 00:23:48 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-h[70521]: 15: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7f2b8c234488] 2024-03-24T00:23:48.472 INFO:journalctl@ceph.mds.h.smithi189.stdout:Mar 24 00:23:48 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-h[70521]: 16: (DispatchQueue::entry()+0x51f) [0x7f2b8c23160f] 2024-03-24T00:23:48.472 INFO:journalctl@ceph.mds.h.smithi189.stdout:Mar 24 00:23:48 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-h[70521]: 17: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f2b8c304e81] 2024-03-24T00:23:48.472 INFO:journalctl@ceph.mds.h.smithi189.stdout:Mar 24 00:23:48 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-h[70521]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7f2b8ad1f1ca] 2024-03-24T00:23:48.472 INFO:journalctl@ceph.mds.h.smithi189.stdout:Mar 24 00:23:48 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-h[70521]: 19: clone() 2024-03-24T00:23:48.472 INFO:journalctl@ceph.mds.h.smithi189.stdout:Mar 24 00:23:48 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-h[70521]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-03-24T00:23:48.472 INFO:journalctl@ceph.mds.h.smithi189.stdout:Mar 24 00:23:48 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-h[70521]: 2024-03-24T00:23:48.472 INFO:journalctl@ceph.mds.h.smithi189.stdout:Mar 24 00:23:48 smithi189 podman[70571]: 2024-03-24 00:23:48.201097335 +0000 UTC m=+0.026716367 container died a5b198163caeb41e0b9bdf8a2d51d4339aa10d06f4b1634a43af1f73d5255ee5 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-h, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, ceph=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240319, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-24T00:23:48.473 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:23:48 smithi193 ceph-mon[36878]: mds.? [v2:172.21.15.189:6836/2642988219,v1:172.21.15.189:6837/2642988219] up:rejoin 2024-03-24T00:23:48.473 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:23:48 smithi193 ceph-mon[36878]: mds.? [v2:172.21.15.118:6834/2587469815,v1:172.21.15.118:6835/2587469815] up:rejoin 2024-03-24T00:23:48.473 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:23:48 smithi193 ceph-mon[36878]: fsmap cephfs:5/5 3 up:standby-replay 1 up:standby 2 up:rejoin 3 up:active 2024-03-24T00:23:48.473 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:23:48 smithi193 ceph-mon[36878]: daemon mds.h is now active in filesystem cephfs as rank 1 2024-03-24T00:23:48.473 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:23:48 smithi193 ceph-mon[36878]: daemon mds.j is now active in filesystem cephfs as rank 3 2024-03-24T00:23:48.473 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:23:48 smithi193 ceph-mon[36878]: pgmap v667: 97 pgs: 97 active+clean; 146 MiB data, 850 MiB used, 1.0 TiB / 1.0 TiB avail; 13 KiB/s rd, 42 KiB/s wr, 24 op/s 2024-03-24T00:23:48.473 INFO:journalctl@ceph.mds.j.smithi118.stdout:Mar 24 00:23:48 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-j[95671]: 5: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x5591900abda1] 2024-03-24T00:23:48.473 INFO:journalctl@ceph.mds.j.smithi118.stdout:Mar 24 00:23:48 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-j[95671]: 6: (Server::handle_client_rename(boost::intrusive_ptr const&)+0x259d) [0x5591900e897d] 2024-03-24T00:23:48.473 INFO:journalctl@ceph.mds.j.smithi118.stdout:Mar 24 00:23:48 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-j[95671]: 7: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x5591900fe233] 2024-03-24T00:23:48.473 INFO:journalctl@ceph.mds.j.smithi118.stdout:Mar 24 00:23:48 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-j[95671]: 8: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x559190100197] 2024-03-24T00:23:48.474 INFO:journalctl@ceph.mds.j.smithi118.stdout:Mar 24 00:23:48 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-j[95671]: 9: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x55918ffff11b] 2024-03-24T00:23:48.474 INFO:journalctl@ceph.mds.j.smithi118.stdout:Mar 24 00:23:48 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-j[95671]: 10: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x55918ffff6bc] 2024-03-24T00:23:48.474 INFO:journalctl@ceph.mds.j.smithi118.stdout:Mar 24 00:23:48 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-j[95671]: 11: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x55918ffe91ff] 2024-03-24T00:23:48.474 INFO:journalctl@ceph.mds.j.smithi118.stdout:Mar 24 00:23:48 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-j[95671]: 12: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7ff257f28488] 2024-03-24T00:23:48.474 INFO:journalctl@ceph.mds.j.smithi118.stdout:Mar 24 00:23:48 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-j[95671]: 13: (DispatchQueue::entry()+0x51f) [0x7ff257f2560f] 2024-03-24T00:23:48.474 INFO:journalctl@ceph.mds.j.smithi118.stdout:Mar 24 00:23:48 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-j[95671]: 14: (DispatchQueue::DispatchThread::entry()+0x11) [0x7ff257ff8e81] 2024-03-24T00:23:48.474 INFO:journalctl@ceph.mds.j.smithi118.stdout:Mar 24 00:23:48 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-j[95671]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7ff256a131ca] 2024-03-24T00:23:48.474 INFO:journalctl@ceph.mds.j.smithi118.stdout:Mar 24 00:23:48 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-j[95671]: 16: clone() 2024-03-24T00:23:48.474 INFO:journalctl@ceph.mds.j.smithi118.stdout:Mar 24 00:23:48 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-j[95671]: 2024-03-24T00:23:48.474 INFO:journalctl@ceph.mds.j.smithi118.stdout:Mar 24 00:23:48 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-j[95671]: 0> 2024-03-24T00:23:48.086+0000 7ff24c400700 -1 *** Caught signal (Aborted) ** 2024-03-24T00:23:48.474 INFO:journalctl@ceph.mds.j.smithi118.stdout:Mar 24 00:23:48 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-j[95671]: in thread 7ff24c400700 thread_name:ms_dispatch 2024-03-24T00:23:48.474 INFO:journalctl@ceph.mds.j.smithi118.stdout:Mar 24 00:23:48 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-j[95671]: 2024-03-24T00:23:48.474 INFO:journalctl@ceph.mds.j.smithi118.stdout:Mar 24 00:23:48 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-j[95671]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-24T00:23:48.474 INFO:journalctl@ceph.mds.j.smithi118.stdout:Mar 24 00:23:48 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-j[95671]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7ff256a1dd20] 2024-03-24T00:23:48.474 INFO:journalctl@ceph.mds.j.smithi118.stdout:Mar 24 00:23:48 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-j[95671]: 2: gsignal() 2024-03-24T00:23:48.475 INFO:journalctl@ceph.mds.j.smithi118.stdout:Mar 24 00:23:48 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-j[95671]: 3: abort() 2024-03-24T00:23:48.475 INFO:journalctl@ceph.mds.j.smithi118.stdout:Mar 24 00:23:48 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-j[95671]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x18f) [0x7ff257c964fb] 2024-03-24T00:23:48.475 INFO:journalctl@ceph.mds.j.smithi118.stdout:Mar 24 00:23:48 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-j[95671]: 5: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7ff257c96667] 2024-03-24T00:23:48.475 INFO:journalctl@ceph.mds.j.smithi118.stdout:Mar 24 00:23:48 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-j[95671]: 6: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x559190231223] 2024-03-24T00:23:48.475 INFO:journalctl@ceph.mds.j.smithi118.stdout:Mar 24 00:23:48 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-j[95671]: 7: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x5591900ab5d6] 2024-03-24T00:23:48.475 INFO:journalctl@ceph.mds.j.smithi118.stdout:Mar 24 00:23:48 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-j[95671]: 8: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x5591900abda1] 2024-03-24T00:23:48.475 INFO:journalctl@ceph.mds.j.smithi118.stdout:Mar 24 00:23:48 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-j[95671]: 9: (Server::handle_client_rename(boost::intrusive_ptr const&)+0x259d) [0x5591900e897d] 2024-03-24T00:23:48.475 INFO:journalctl@ceph.mds.j.smithi118.stdout:Mar 24 00:23:48 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-j[95671]: 10: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x5591900fe233] 2024-03-24T00:23:48.475 INFO:journalctl@ceph.mds.j.smithi118.stdout:Mar 24 00:23:48 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-j[95671]: 11: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x559190100197] 2024-03-24T00:23:48.475 INFO:journalctl@ceph.mds.j.smithi118.stdout:Mar 24 00:23:48 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-j[95671]: 12: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x55918ffff11b] 2024-03-24T00:23:48.475 INFO:journalctl@ceph.mds.j.smithi118.stdout:Mar 24 00:23:48 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-j[95671]: 13: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x55918ffff6bc] 2024-03-24T00:23:48.475 INFO:journalctl@ceph.mds.j.smithi118.stdout:Mar 24 00:23:48 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-j[95671]: 14: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x55918ffe91ff] 2024-03-24T00:23:48.475 INFO:journalctl@ceph.mds.j.smithi118.stdout:Mar 24 00:23:48 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-j[95671]: 15: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7ff257f28488] 2024-03-24T00:23:48.475 INFO:journalctl@ceph.mds.j.smithi118.stdout:Mar 24 00:23:48 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-j[95671]: 16: (DispatchQueue::entry()+0x51f) [0x7ff257f2560f] 2024-03-24T00:23:48.475 INFO:journalctl@ceph.mds.j.smithi118.stdout:Mar 24 00:23:48 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-j[95671]: 17: (DispatchQueue::DispatchThread::entry()+0x11) [0x7ff257ff8e81] 2024-03-24T00:23:48.475 INFO:journalctl@ceph.mds.j.smithi118.stdout:Mar 24 00:23:48 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-j[95671]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7ff256a131ca] 2024-03-24T00:23:48.476 INFO:journalctl@ceph.mds.j.smithi118.stdout:Mar 24 00:23:48 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-j[95671]: 19: clone() 2024-03-24T00:23:48.476 INFO:journalctl@ceph.mds.j.smithi118.stdout:Mar 24 00:23:48 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-j[95671]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-03-24T00:23:48.476 INFO:journalctl@ceph.mds.j.smithi118.stdout:Mar 24 00:23:48 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-j[95671]: 2024-03-24T00:23:48.476 INFO:journalctl@ceph.mds.j.smithi118.stdout:Mar 24 00:23:48 smithi118 podman[95948]: 2024-03-24 00:23:48.202109812 +0000 UTC m=+0.036162900 container died dc67f774391d71af5f11deab743293cfc5dfdd9caf2a15951b1ca1e76255d341 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-j, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, io.buildah.version=1.33.5, org.label-schema.build-date=20240319, GIT_CLEAN=True, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , ceph=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2) 2024-03-24T00:23:49.016 INFO:journalctl@ceph.mds.h.smithi189.stdout:Mar 24 00:23:48 smithi189 podman[70571]: 2024-03-24 00:23:48.766285616 +0000 UTC m=+0.591904621 container remove a5b198163caeb41e0b9bdf8a2d51d4339aa10d06f4b1634a43af1f73d5255ee5 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-h, ceph=True, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240319, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_CLEAN=True, io.buildah.version=1.33.5) 2024-03-24T00:23:49.016 INFO:journalctl@ceph.mds.h.smithi189.stdout:Mar 24 00:23:48 smithi189 systemd[1]: ceph-d79a5420-e971-11ee-95cd-87774f69a715@mds.h.service: Main process exited, code=exited, status=134/n/a 2024-03-24T00:23:49.280 INFO:journalctl@ceph.mds.j.smithi118.stdout:Mar 24 00:23:49 smithi118 podman[95948]: 2024-03-24 00:23:49.055858018 +0000 UTC m=+0.889911143 container remove dc67f774391d71af5f11deab743293cfc5dfdd9caf2a15951b1ca1e76255d341 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-j, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240319, GIT_CLEAN=True, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, io.buildah.version=1.33.5, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, ceph=True, CEPH_POINT_RELEASE=) 2024-03-24T00:23:49.280 INFO:journalctl@ceph.mds.j.smithi118.stdout:Mar 24 00:23:49 smithi118 systemd[1]: ceph-d79a5420-e971-11ee-95cd-87774f69a715@mds.j.service: Main process exited, code=exited, status=134/n/a 2024-03-24T00:23:49.281 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:23:49 smithi118 ceph-mon[31535]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-03-24T00:23:49.281 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:23:49 smithi118 ceph-mon[31535]: Cluster is now healthy 2024-03-24T00:23:49.281 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:23:49 smithi118 ceph-mon[31535]: mds.? [v2:172.21.15.189:6836/2642988219,v1:172.21.15.189:6837/2642988219] up:active 2024-03-24T00:23:49.281 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:23:49 smithi118 ceph-mon[31535]: mds.? [v2:172.21.15.118:6834/2587469815,v1:172.21.15.118:6835/2587469815] up:active 2024-03-24T00:23:49.281 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:23:49 smithi118 ceph-mon[31535]: fsmap cephfs:5 3 up:standby-replay 1 up:standby 5 up:active 2024-03-24T00:23:49.281 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:23:49 smithi118 ceph-mon[31535]: fsmap cephfs:5 4 up:standby-replay 5 up:active 2024-03-24T00:23:49.454 INFO:journalctl@ceph.mds.h.smithi189.stdout:Mar 24 00:23:49 smithi189 systemd[1]: ceph-d79a5420-e971-11ee-95cd-87774f69a715@mds.h.service: Failed with result 'exit-code'. 2024-03-24T00:23:49.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:23:49 smithi189 ceph-mon[36187]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-03-24T00:23:49.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:23:49 smithi189 ceph-mon[36187]: Cluster is now healthy 2024-03-24T00:23:49.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:23:49 smithi189 ceph-mon[36187]: mds.? [v2:172.21.15.189:6836/2642988219,v1:172.21.15.189:6837/2642988219] up:active 2024-03-24T00:23:49.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:23:49 smithi189 ceph-mon[36187]: mds.? [v2:172.21.15.118:6834/2587469815,v1:172.21.15.118:6835/2587469815] up:active 2024-03-24T00:23:49.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:23:49 smithi189 ceph-mon[36187]: fsmap cephfs:5 3 up:standby-replay 1 up:standby 5 up:active 2024-03-24T00:23:49.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:23:49 smithi189 ceph-mon[36187]: fsmap cephfs:5 4 up:standby-replay 5 up:active 2024-03-24T00:23:49.460 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:23:49 smithi193 ceph-mon[36878]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-03-24T00:23:49.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:23:49 smithi193 ceph-mon[36878]: Cluster is now healthy 2024-03-24T00:23:49.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:23:49 smithi193 ceph-mon[36878]: mds.? [v2:172.21.15.189:6836/2642988219,v1:172.21.15.189:6837/2642988219] up:active 2024-03-24T00:23:49.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:23:49 smithi193 ceph-mon[36878]: mds.? [v2:172.21.15.118:6834/2587469815,v1:172.21.15.118:6835/2587469815] up:active 2024-03-24T00:23:49.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:23:49 smithi193 ceph-mon[36878]: fsmap cephfs:5 3 up:standby-replay 1 up:standby 5 up:active 2024-03-24T00:23:49.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:23:49 smithi193 ceph-mon[36878]: fsmap cephfs:5 4 up:standby-replay 5 up:active 2024-03-24T00:23:49.914 INFO:journalctl@ceph.mds.j.smithi118.stdout:Mar 24 00:23:49 smithi118 systemd[1]: ceph-d79a5420-e971-11ee-95cd-87774f69a715@mds.j.service: Failed with result 'exit-code'. 2024-03-24T00:23:49.914 INFO:journalctl@ceph.mds.j.smithi118.stdout:Mar 24 00:23:49 smithi118 systemd[1]: ceph-d79a5420-e971-11ee-95cd-87774f69a715@mds.j.service: Consumed 1.560s CPU time. 2024-03-24T00:23:50.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:23:50 smithi118 ceph-mon[31535]: pgmap v668: 97 pgs: 97 active+clean; 146 MiB data, 850 MiB used, 1.0 TiB / 1.0 TiB avail; 122 KiB/s rd, 39 KiB/s wr, 42 op/s 2024-03-24T00:23:50.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:23:50 smithi189 ceph-mon[36187]: pgmap v668: 97 pgs: 97 active+clean; 146 MiB data, 850 MiB used, 1.0 TiB / 1.0 TiB avail; 122 KiB/s rd, 39 KiB/s wr, 42 op/s 2024-03-24T00:23:50.460 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:23:50 smithi193 ceph-mon[36878]: pgmap v668: 97 pgs: 97 active+clean; 146 MiB data, 850 MiB used, 1.0 TiB / 1.0 TiB avail; 122 KiB/s rd, 39 KiB/s wr, 42 op/s 2024-03-24T00:23:52.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:23:51 smithi118 ceph-mon[31535]: pgmap v669: 97 pgs: 97 active+clean; 146 MiB data, 850 MiB used, 1.0 TiB / 1.0 TiB avail; 189 KiB/s rd, 17 KiB/s wr, 40 op/s 2024-03-24T00:23:52.203 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:23:51 smithi189 ceph-mon[36187]: pgmap v669: 97 pgs: 97 active+clean; 146 MiB data, 850 MiB used, 1.0 TiB / 1.0 TiB avail; 189 KiB/s rd, 17 KiB/s wr, 40 op/s 2024-03-24T00:23:52.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:23:51 smithi193 ceph-mon[36878]: pgmap v669: 97 pgs: 97 active+clean; 146 MiB data, 850 MiB used, 1.0 TiB / 1.0 TiB avail; 189 KiB/s rd, 17 KiB/s wr, 40 op/s 2024-03-24T00:23:54.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:23:53 smithi118 ceph-mon[31535]: pgmap v670: 97 pgs: 97 active+clean; 146 MiB data, 850 MiB used, 1.0 TiB / 1.0 TiB avail; 332 KiB/s rd, 14 KiB/s wr, 57 op/s 2024-03-24T00:23:54.203 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:23:53 smithi189 ceph-mon[36187]: pgmap v670: 97 pgs: 97 active+clean; 146 MiB data, 850 MiB used, 1.0 TiB / 1.0 TiB avail; 332 KiB/s rd, 14 KiB/s wr, 57 op/s 2024-03-24T00:23:54.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:23:53 smithi193 ceph-mon[36878]: pgmap v670: 97 pgs: 97 active+clean; 146 MiB data, 850 MiB used, 1.0 TiB / 1.0 TiB avail; 332 KiB/s rd, 14 KiB/s wr, 57 op/s 2024-03-24T00:23:55.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:23:55 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T00:23:55.453 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:23:55 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T00:23:55.460 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:23:55 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T00:23:56.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:23:56 smithi118 ceph-mon[31535]: pgmap v671: 97 pgs: 97 active+clean; 146 MiB data, 850 MiB used, 1.0 TiB / 1.0 TiB avail; 294 KiB/s rd, 24 KiB/s wr, 48 op/s 2024-03-24T00:23:56.453 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:23:56 smithi189 ceph-mon[36187]: pgmap v671: 97 pgs: 97 active+clean; 146 MiB data, 850 MiB used, 1.0 TiB / 1.0 TiB avail; 294 KiB/s rd, 24 KiB/s wr, 48 op/s 2024-03-24T00:23:56.460 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:23:56 smithi193 ceph-mon[36878]: pgmap v671: 97 pgs: 97 active+clean; 146 MiB data, 850 MiB used, 1.0 TiB / 1.0 TiB avail; 294 KiB/s rd, 24 KiB/s wr, 48 op/s 2024-03-24T00:23:57.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:23:57 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T00:23:57.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:23:57 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T00:23:57.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:23:57 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T00:23:57.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:23:57 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T00:23:57.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:23:57 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T00:23:57.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:23:57 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T00:23:57.460 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:23:57 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T00:23:57.460 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:23:57 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T00:23:57.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:23:57 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T00:23:58.361 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:23:58 smithi193 ceph-mon[36878]: pgmap v672: 97 pgs: 97 active+clean; 146 MiB data, 850 MiB used, 1.0 TiB / 1.0 TiB avail; 285 KiB/s rd, 19 KiB/s wr, 44 op/s 2024-03-24T00:23:58.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:23:58 smithi118 ceph-mon[31535]: pgmap v672: 97 pgs: 97 active+clean; 146 MiB data, 850 MiB used, 1.0 TiB / 1.0 TiB avail; 285 KiB/s rd, 19 KiB/s wr, 44 op/s 2024-03-24T00:23:58.453 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:23:58 smithi189 ceph-mon[36187]: pgmap v672: 97 pgs: 97 active+clean; 146 MiB data, 850 MiB used, 1.0 TiB / 1.0 TiB avail; 285 KiB/s rd, 19 KiB/s wr, 44 op/s 2024-03-24T00:23:59.704 INFO:journalctl@ceph.mds.h.smithi189.stdout:Mar 24 00:23:59 smithi189 systemd[1]: ceph-d79a5420-e971-11ee-95cd-87774f69a715@mds.h.service: Scheduled restart job, restart counter is at 6. 2024-03-24T00:23:59.704 INFO:journalctl@ceph.mds.h.smithi189.stdout:Mar 24 00:23:59 smithi189 systemd[1]: Stopped Ceph mds.h for d79a5420-e971-11ee-95cd-87774f69a715. 2024-03-24T00:23:59.704 INFO:journalctl@ceph.mds.h.smithi189.stdout:Mar 24 00:23:59 smithi189 systemd[1]: ceph-d79a5420-e971-11ee-95cd-87774f69a715@mds.h.service: Start request repeated too quickly. 2024-03-24T00:23:59.704 INFO:journalctl@ceph.mds.h.smithi189.stdout:Mar 24 00:23:59 smithi189 systemd[1]: ceph-d79a5420-e971-11ee-95cd-87774f69a715@mds.h.service: Failed with result 'exit-code'. 2024-03-24T00:23:59.704 INFO:journalctl@ceph.mds.h.smithi189.stdout:Mar 24 00:23:59 smithi189 systemd[1]: Failed to start Ceph mds.h for d79a5420-e971-11ee-95cd-87774f69a715. 2024-03-24T00:24:00.089 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:23:59 smithi118 ceph-mon[31535]: pgmap v673: 97 pgs: 97 active+clean; 146 MiB data, 850 MiB used, 1.0 TiB / 1.0 TiB avail; 284 KiB/s rd, 18 KiB/s wr, 45 op/s 2024-03-24T00:24:00.089 INFO:journalctl@ceph.mds.j.smithi118.stdout:Mar 24 00:23:59 smithi118 systemd[1]: ceph-d79a5420-e971-11ee-95cd-87774f69a715@mds.j.service: Scheduled restart job, restart counter is at 2. 2024-03-24T00:24:00.089 INFO:journalctl@ceph.mds.j.smithi118.stdout:Mar 24 00:23:59 smithi118 systemd[1]: Stopped Ceph mds.j for d79a5420-e971-11ee-95cd-87774f69a715. 2024-03-24T00:24:00.089 INFO:journalctl@ceph.mds.j.smithi118.stdout:Mar 24 00:23:59 smithi118 systemd[1]: ceph-d79a5420-e971-11ee-95cd-87774f69a715@mds.j.service: Consumed 1.560s CPU time. 2024-03-24T00:24:00.089 INFO:journalctl@ceph.mds.j.smithi118.stdout:Mar 24 00:23:59 smithi118 systemd[1]: Starting Ceph mds.j for d79a5420-e971-11ee-95cd-87774f69a715... 2024-03-24T00:24:00.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:23:59 smithi189 ceph-mon[36187]: pgmap v673: 97 pgs: 97 active+clean; 146 MiB data, 850 MiB used, 1.0 TiB / 1.0 TiB avail; 284 KiB/s rd, 18 KiB/s wr, 45 op/s 2024-03-24T00:24:00.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:23:59 smithi193 ceph-mon[36878]: pgmap v673: 97 pgs: 97 active+clean; 146 MiB data, 850 MiB used, 1.0 TiB / 1.0 TiB avail; 284 KiB/s rd, 18 KiB/s wr, 45 op/s 2024-03-24T00:24:00.213 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:24:00.212+0000 7f5aa4e00640 1 -- 172.21.15.118:0/3331386444 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f5aa0102700 msgr2=0x7f5aa0102b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:24:00.213 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:24:00.212+0000 7f5aa4e00640 1 --2- 172.21.15.118:0/3331386444 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f5aa0102700 0x7f5aa0102b80 secure :-1 s=READY pgs=453 cs=0 l=1 rev1=1 crypto rx=0x7f5a88009900 tx=0x7f5a8802bc30 comp rx=0 tx=0).stop 2024-03-24T00:24:00.214 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:24:00.212+0000 7f5aa4e00640 1 -- 172.21.15.118:0/3331386444 shutdown_connections 2024-03-24T00:24:00.214 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:24:00.212+0000 7f5aa4e00640 1 --2- 172.21.15.118:0/3331386444 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5aa01030c0 0x7f5aa0109950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:24:00.214 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:24:00.212+0000 7f5aa4e00640 1 --2- 172.21.15.118:0/3331386444 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f5aa0102700 0x7f5aa0102b80 unknown :-1 s=CLOSED pgs=453 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:24:00.214 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:24:00.212+0000 7f5aa4e00640 1 --2- 172.21.15.118:0/3331386444 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f5aa0101510 0x7f5aa0101930 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:24:00.214 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:24:00.212+0000 7f5aa4e00640 1 -- 172.21.15.118:0/3331386444 >> 172.21.15.118:0/3331386444 conn(0x7f5aa00fca80 msgr2=0x7f5aa00feec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:24:00.214 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:24:00.213+0000 7f5aa4e00640 1 -- 172.21.15.118:0/3331386444 shutdown_connections 2024-03-24T00:24:00.214 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:24:00.213+0000 7f5aa4e00640 1 -- 172.21.15.118:0/3331386444 wait complete. 2024-03-24T00:24:00.216 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:24:00.214+0000 7f5aa4e00640 1 Processor -- start 2024-03-24T00:24:00.216 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:24:00.214+0000 7f5aa4e00640 1 -- start start 2024-03-24T00:24:00.216 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:24:00.214+0000 7f5aa4e00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f5aa0101510 0x7f5aa01999a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:24:00.216 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:24:00.214+0000 7f5aa4e00640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f5aa0102700 0x7f5aa0199ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:24:00.216 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:24:00.214+0000 7f5aa4e00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5aa01030c0 0x7f5aa01a0f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:24:00.217 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:24:00.214+0000 7f5aa4e00640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5aa010d1f0 con 0x7f5aa0102700 2024-03-24T00:24:00.217 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:24:00.214+0000 7f5aa4e00640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5aa010d070 con 0x7f5aa01030c0 2024-03-24T00:24:00.217 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:24:00.214+0000 7f5aa4e00640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5aa010d370 con 0x7f5aa0101510 2024-03-24T00:24:00.217 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:24:00.214+0000 7f5a9f400640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f5aa0101510 0x7f5aa01999a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:24:00.217 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:24:00.214+0000 7f5a9ea00640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f5aa0102700 0x7f5aa0199ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:24:00.217 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:24:00.214+0000 7f5a9ea00640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f5aa0102700 0x7f5aa0199ee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.118:3300/0 says I am v2:172.21.15.118:38020/0 (socket says 172.21.15.118:38020) 2024-03-24T00:24:00.217 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:24:00.214+0000 7f5a9fe00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5aa01030c0 0x7f5aa01a0f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:24:00.217 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:24:00.214+0000 7f5a9ea00640 1 -- 172.21.15.118:0/756697505 learned_addr learned my addr 172.21.15.118:0/756697505 (peer_addr_for_me v2:172.21.15.118:0/0) 2024-03-24T00:24:00.217 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:24:00.215+0000 7f5a9ea00640 1 -- 172.21.15.118:0/756697505 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f5aa0101510 msgr2=0x7f5aa01999a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:24:00.217 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:24:00.215+0000 7f5a9ea00640 1 --2- 172.21.15.118:0/756697505 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f5aa0101510 0x7f5aa01999a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:24:00.217 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:24:00.215+0000 7f5a9ea00640 1 -- 172.21.15.118:0/756697505 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5aa01030c0 msgr2=0x7f5aa01a0f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:24:00.217 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:24:00.215+0000 7f5a9ea00640 1 --2- 172.21.15.118:0/756697505 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5aa01030c0 0x7f5aa01a0f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:24:00.217 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:24:00.215+0000 7f5a9ea00640 1 -- 172.21.15.118:0/756697505 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5aa01a15c0 con 0x7f5aa0102700 2024-03-24T00:24:00.218 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:24:00.215+0000 7f5a9fe00640 1 --2- 172.21.15.118:0/756697505 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5aa01030c0 0x7f5aa01a0f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T00:24:00.218 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:24:00.215+0000 7f5a9f400640 1 --2- 172.21.15.118:0/756697505 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f5aa0101510 0x7f5aa01999a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T00:24:00.218 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:24:00.215+0000 7f5a9ea00640 1 --2- 172.21.15.118:0/756697505 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f5aa0102700 0x7f5aa0199ee0 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7f5a88005ab0 tx=0x7f5a8802c6c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:24:00.218 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:24:00.215+0000 7f5a97e00640 1 -- 172.21.15.118:0/756697505 <== mon.0 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5a88043020 con 0x7f5aa0102700 2024-03-24T00:24:00.218 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:24:00.215+0000 7f5a97e00640 1 -- 172.21.15.118:0/756697505 <== mon.0 v2:172.21.15.118:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f5a880054e0 con 0x7f5aa0102700 2024-03-24T00:24:00.218 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:24:00.215+0000 7f5aa4e00640 1 -- 172.21.15.118:0/756697505 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5aa01a1850 con 0x7f5aa0102700 2024-03-24T00:24:00.218 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:24:00.215+0000 7f5a97e00640 1 -- 172.21.15.118:0/756697505 <== mon.0 v2:172.21.15.118:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5a88003ce0 con 0x7f5aa0102700 2024-03-24T00:24:00.218 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:24:00.216+0000 7f5aa4e00640 1 -- 172.21.15.118:0/756697505 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5aa01a1da0 con 0x7f5aa0102700 2024-03-24T00:24:00.220 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:24:00.219+0000 7f5aa9176740 1 Processor -- start 2024-03-24T00:24:00.220 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:24:00.220+0000 7f5aa9176740 1 -- start start 2024-03-24T00:24:00.222 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:24:00.220+0000 7f5aa9176740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55ba197ca3f0 0x55ba197ca810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:24:00.222 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:24:00.220+0000 7f5aa9176740 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55ba197cade0 0x55ba197cf8e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:24:00.222 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:24:00.220+0000 7f5aa9176740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55ba197cff50 0x55ba197d23e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:24:00.222 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:24:00.220+0000 7f5aa9176740 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ba1978d3b0 con 0x55ba197cade0 2024-03-24T00:24:00.222 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:24:00.220+0000 7f5aa9176740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ba1972f830 con 0x55ba197ca3f0 2024-03-24T00:24:00.222 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:24:00.220+0000 7f5aa9176740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ba196d85d0 con 0x55ba197cff50 2024-03-24T00:24:00.222 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:24:00.220+0000 7f5a97e00640 1 -- 172.21.15.118:0/756697505 <== mon.0 v2:172.21.15.118:3300/0 4 ==== mgrmap(e 20) v1 ==== 137291+0+0 (secure 0 0 0) 0x7f5a88046020 con 0x7f5aa0102700 2024-03-24T00:24:00.222 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:24:00.220+0000 7f5a9fe00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55ba197ca3f0 0x55ba197ca810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:24:00.223 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:24:00.220+0000 7f5a9fe00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55ba197ca3f0 0x55ba197ca810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.118:54308/0 (socket says 172.21.15.118:54308) 2024-03-24T00:24:00.223 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:24:00.220+0000 7f5a9fe00640 1 -- 172.21.15.118:0/1132830726 learned_addr learned my addr 172.21.15.118:0/1132830726 (peer_addr_for_me v2:172.21.15.118:0/0) 2024-03-24T00:24:00.223 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:24:00.221+0000 7f5a9f400640 1 --2- 172.21.15.118:0/1132830726 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55ba197cade0 0x55ba197cf8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:24:00.223 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:24:00.221+0000 7f5a9ea00640 1 --2- 172.21.15.118:0/1132830726 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55ba197cff50 0x55ba197d23e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:24:00.223 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:24:00.221+0000 7f5a9fe00640 1 -- 172.21.15.118:0/1132830726 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55ba197cff50 msgr2=0x55ba197d23e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:24:00.223 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:24:00.221+0000 7f5a9fe00640 1 --2- 172.21.15.118:0/1132830726 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55ba197cff50 0x55ba197d23e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:24:00.223 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:24:00.221+0000 7f5a9fe00640 1 -- 172.21.15.118:0/1132830726 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55ba197cade0 msgr2=0x55ba197cf8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:24:00.223 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:24:00.221+0000 7f5a9fe00640 1 --2- 172.21.15.118:0/1132830726 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55ba197cade0 0x55ba197cf8e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:24:00.223 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:24:00.221+0000 7f5a9fe00640 1 -- 172.21.15.118:0/1132830726 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55ba196f02c0 con 0x55ba197ca3f0 2024-03-24T00:24:00.223 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:24:00.221+0000 7f5a9fe00640 1 --2- 172.21.15.118:0/1132830726 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55ba197ca3f0 0x55ba197ca810 secure :-1 s=READY pgs=455 cs=0 l=1 rev1=1 crypto rx=0x7f5a980117e0 tx=0x7f5a98011ca0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=a1a148e1c0ed28aa server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:24:00.223 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:24:00.221+0000 7f5a94c00640 1 -- 172.21.15.118:0/1132830726 <== mon.2 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5a98011310 con 0x55ba197ca3f0 2024-03-24T00:24:00.223 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:24:00.221+0000 7f5a94c00640 1 -- 172.21.15.118:0/1132830726 <== mon.2 v2:172.21.15.189:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f5a98003e50 con 0x55ba197ca3f0 2024-03-24T00:24:00.223 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:24:00.221+0000 7f5a94c00640 1 -- 172.21.15.118:0/1132830726 <== mon.2 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5a98004590 con 0x55ba197ca3f0 2024-03-24T00:24:00.227 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:24:00.222+0000 7f5a97e00640 1 --2- 172.21.15.118:0/756697505 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7f5a840a27f0 0x7f5a840a4cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:24:00.227 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:24:00.222+0000 7f5aa9176740 1 -- 172.21.15.118:0/1132830726 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55ba197ca3f0 msgr2=0x55ba197ca810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:24:00.227 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:24:00.222+0000 7f5aa9176740 1 --2- 172.21.15.118:0/1132830726 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55ba197ca3f0 0x55ba197ca810 secure :-1 s=READY pgs=455 cs=0 l=1 rev1=1 crypto rx=0x7f5a980117e0 tx=0x7f5a98011ca0 comp rx=0 tx=0).stop 2024-03-24T00:24:00.228 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:24:00.222+0000 7f5a97e00640 1 -- 172.21.15.118:0/756697505 <== mon.0 v2:172.21.15.118:3300/0 5 ==== osd_map(116..116 src has 1..116) v4 ==== 10513+0+0 (secure 0 0 0) 0x7f5a880f1440 con 0x7f5aa0102700 2024-03-24T00:24:00.228 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:24:00.222+0000 7f5aa9176740 1 -- 172.21.15.118:0/1132830726 shutdown_connections 2024-03-24T00:24:00.228 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:24:00.222+0000 7f5aa9176740 1 --2- 172.21.15.118:0/1132830726 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55ba197cff50 0x55ba197d23e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:24:00.228 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:24:00.222+0000 7f5aa9176740 1 --2- 172.21.15.118:0/1132830726 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55ba197cade0 0x55ba197cf8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:24:00.228 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:24:00.222+0000 7f5aa9176740 1 --2- 172.21.15.118:0/1132830726 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55ba197ca3f0 0x55ba197ca810 unknown :-1 s=CLOSED pgs=455 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:24:00.228 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:24:00.222+0000 7f5aa9176740 1 -- 172.21.15.118:0/1132830726 >> 172.21.15.118:0/1132830726 conn(0x55ba197c5960 msgr2=0x55ba197c5d60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:24:00.228 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:24:00.222+0000 7f5a9f400640 1 --2- 172.21.15.118:0/756697505 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7f5a840a27f0 0x7f5a840a4cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:24:00.228 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:24:00.222+0000 7f5aa9176740 1 -- 172.21.15.118:0/1132830726 shutdown_connections 2024-03-24T00:24:00.228 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:24:00.222+0000 7f5aa9176740 1 -- 172.21.15.118:0/1132830726 wait complete. 2024-03-24T00:24:00.228 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:24:00.223+0000 7f5aa9176740 1 -- start start 2024-03-24T00:24:00.228 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:24:00.223+0000 7f5a9f400640 1 --2- 172.21.15.118:0/756697505 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7f5a840a27f0 0x7f5a840a4cb0 secure :-1 s=READY pgs=390 cs=0 l=1 rev1=1 crypto rx=0x55ba197c68d0 tx=0x7f5a90005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:24:00.228 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:24:00.223+0000 7f5aa9176740 1 Processor -- start 2024-03-24T00:24:00.228 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:24:00.223+0000 7f5aa9176740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55ba197ca3f0 0x55ba197d45e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:24:00.228 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:24:00.224+0000 7f5aa9176740 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55ba197cade0 0x55ba197d4b20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:24:00.229 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:24:00.224+0000 7f5aa9176740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55ba197cff50 0x55ba197dbb60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:24:00.229 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:24:00.224+0000 7f5aa9176740 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ba196eba20 con 0x55ba197cade0 2024-03-24T00:24:00.229 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:24:00.224+0000 7f5aa9176740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ba1972b8d0 con 0x55ba197cff50 2024-03-24T00:24:00.229 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:24:00.224+0000 7f5aa9176740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x55ba197d2f00 con 0x55ba197ca3f0 2024-03-24T00:24:00.229 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:24:00.224+0000 7f5aa9176740 10 client.0 _ll_register_callbacks cb 0x55ba197281c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T00:24:00.229 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:24:00.224+0000 7f5a9fe00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55ba197cff50 0x55ba197dbb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:24:00.229 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:24:00.224+0000 7f5a9ea00640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55ba197cade0 0x55ba197d4b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:24:00.229 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:24:00.224+0000 7f5a9fe00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55ba197cff50 0x55ba197dbb60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.118:54322/0 (socket says 172.21.15.118:54322) 2024-03-24T00:24:00.229 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:24:00.224+0000 7f5a9fe00640 1 -- 172.21.15.118:0/1466314365 learned_addr learned my addr 172.21.15.118:0/1466314365 (peer_addr_for_me v2:172.21.15.118:0/0) 2024-03-24T00:24:00.229 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:24:00.224+0000 7f5a9f400640 1 --2- 172.21.15.118:0/1466314365 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55ba197ca3f0 0x55ba197d45e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:24:00.229 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:24:00.224+0000 7f5a9ea00640 1 -- 172.21.15.118:0/1466314365 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55ba197ca3f0 msgr2=0x55ba197d45e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:24:00.229 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:24:00.224+0000 7f5a9ea00640 1 --2- 172.21.15.118:0/1466314365 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55ba197ca3f0 0x55ba197d45e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:24:00.229 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:24:00.224+0000 7f5a9ea00640 1 -- 172.21.15.118:0/1466314365 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55ba197cff50 msgr2=0x55ba197dbb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:24:00.229 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:24:00.224+0000 7f5a9ea00640 1 --2- 172.21.15.118:0/1466314365 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55ba197cff50 0x55ba197dbb60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:24:00.230 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:24:00.224+0000 7f5a9ea00640 1 -- 172.21.15.118:0/1466314365 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55ba19444210 con 0x55ba197cade0 2024-03-24T00:24:00.230 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:24:00.224+0000 7f5a9fe00640 1 --2- 172.21.15.118:0/1466314365 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55ba197cff50 0x55ba197dbb60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T00:24:00.230 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:24:00.225+0000 7f5a9ea00640 1 --2- 172.21.15.118:0/1466314365 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55ba197cade0 0x55ba197d4b20 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7f5a880ee6a0 tx=0x7f5a88003e80 comp rx=0 tx=0).ready entity=mon.0 client_cookie=998b2c0199c1e5f9 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:24:00.230 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:24:00.225+0000 7f5a8d600640 10 client.0 ms_handle_connect on v2:172.21.15.118:3300/0 2024-03-24T00:24:00.230 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:24:00.225+0000 7f5a8d600640 1 -- 172.21.15.118:0/1466314365 <== mon.0 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5a880bd030 con 0x55ba197cade0 2024-03-24T00:24:00.230 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:24:00.225+0000 7f5a8d600640 1 -- 172.21.15.118:0/1466314365 <== mon.0 v2:172.21.15.118:3300/0 2 ==== osd_map(116..116 src has 1..116) v4 ==== 10513+0+0 (secure 0 0 0) 0x7f5a880bb7f0 con 0x55ba197cade0 2024-03-24T00:24:00.230 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:24:00.225+0000 7f5aa9176740 1 -- 172.21.15.118:0/1466314365 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f5a98003e50 con 0x55ba197cade0 2024-03-24T00:24:00.230 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:24:00.225+0000 7f5a8d600640 1 -- 172.21.15.118:0/1466314365 <== mon.0 v2:172.21.15.118:3300/0 3 ==== mon_get_version_reply(handle=1 version=154) v2 ==== 24+0+0 (secure 0 0 0) 0x7f5a880bce50 con 0x55ba197cade0 2024-03-24T00:24:00.230 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:24:00.225+0000 7f5aa9176740 10 client.16728 fetch_fsmap learned FSMap version 154 2024-03-24T00:24:00.230 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:24:00.225+0000 7f5aa9176740 1 -- 172.21.15.118:0/1466314365 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({config=0+,fsmap=154,monmap=4+}) v3 -- 0x55ba19510620 con 0x55ba197cade0 2024-03-24T00:24:00.230 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:24:00.225+0000 7f5a8d600640 1 -- 172.21.15.118:0/1466314365 <== mon.0 v2:172.21.15.118:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f5a98003e50 con 0x55ba197cade0 2024-03-24T00:24:00.230 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:24:00.225+0000 7f5a8d600640 1 -- 172.21.15.118:0/1466314365 <== mon.0 v2:172.21.15.118:3300/0 5 ==== fsmap(e 154) v1 ==== 6221+0+0 (secure 0 0 0) 0x7f5a880c5800 con 0x55ba197cade0 2024-03-24T00:24:00.230 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:24:00.226+0000 7f5aa9176740 10 client.16728 fetch_fsmap finished waiting for FSMap version 154 2024-03-24T00:24:00.231 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:24:00.226+0000 7f5aa9176740 10 client.16728 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-24T00:24:00.231 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:24:00.226+0000 7f5aa9176740 20 client.16728 populate_metadata read hostname 'smithi118' 2024-03-24T00:24:00.231 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:24:00.226+0000 7f5aa9176740 1 --2- 172.21.15.118:0/1466314365 >> [v2:172.21.15.189:6838/4074484168,v1:172.21.15.189:6839/4074484168] conn(0x55ba197e0430 0x55ba197e2840 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:24:00.231 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:24:00.226+0000 7f5aa9176740 4 client.16728 mds_command: new command op to 24569 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-24T00:24:00.231 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:24:00.226+0000 7f5aa9176740 1 -- 172.21.15.118:0/1466314365 --> [v2:172.21.15.189:6838/4074484168,v1:172.21.15.189:6839/4074484168] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55ba1973cd70 con 0x55ba197e0430 2024-03-24T00:24:00.231 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:24:00.226+0000 7f5a9f400640 1 --2- 172.21.15.118:0/1466314365 >> [v2:172.21.15.189:6838/4074484168,v1:172.21.15.189:6839/4074484168] conn(0x55ba197e0430 0x55ba197e2840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:24:00.231 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:24:00.226+0000 7f5a9f400640 1 --2- 172.21.15.118:0/1466314365 >> [v2:172.21.15.189:6838/4074484168,v1:172.21.15.189:6839/4074484168] conn(0x55ba197e0430 0x55ba197e2840 crc :-1 s=READY pgs=357 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=16c381691fbc7c5a server_cookie=ef422da48680269b in_seq=0 out_seq=0 2024-03-24T00:24:00.231 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:24:00.226+0000 7f5a8d600640 10 client.16728 ms_handle_connect on v2:172.21.15.189:6838/4074484168 2024-03-24T00:24:00.232 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:24:00.227+0000 7f5a8d600640 1 -- 172.21.15.118:0/1466314365 <== mds.0 v2:172.21.15.189:6838/4074484168 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x55ba1973cd70 con 0x55ba197e0430 2024-03-24T00:24:00.232 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:24:00.227+0000 7f5a8d600640 10 client.16728 handle_command_reply: tid=0 2024-03-24T00:24:00.232 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:24:00.227+0000 7f5aa9176740 1 client.16728 shutdown 2024-03-24T00:24:00.232 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:24:00.228+0000 7f5aa9176740 1 -- 172.21.15.118:0/1466314365 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55ba197cade0 msgr2=0x55ba197d4b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:24:00.232 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:24:00.228+0000 7f5aa9176740 1 --2- 172.21.15.118:0/1466314365 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55ba197cade0 0x55ba197d4b20 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7f5a880ee6a0 tx=0x7f5a88003e80 comp rx=0 tx=0).stop 2024-03-24T00:24:00.233 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:24:00.228+0000 7f5aa9176740 1 -- 172.21.15.118:0/1466314365 shutdown_connections 2024-03-24T00:24:00.233 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:24:00.228+0000 7f5aa9176740 1 --2- 172.21.15.118:0/1466314365 >> [v2:172.21.15.189:6838/4074484168,v1:172.21.15.189:6839/4074484168] conn(0x55ba197e0430 0x55ba197e2840 crc :-1 s=READY pgs=357 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:24:00.233 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:24:00.228+0000 7f5aa9176740 1 --2- 172.21.15.118:0/1466314365 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55ba197cff50 0x55ba197dbb60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:24:00.233 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:24:00.228+0000 7f5aa9176740 1 --2- 172.21.15.118:0/1466314365 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55ba197cade0 0x55ba197d4b20 unknown :-1 s=CLOSED pgs=231 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:24:00.233 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:24:00.228+0000 7f5aa9176740 1 --2- 172.21.15.118:0/1466314365 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55ba197ca3f0 0x55ba197d45e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:24:00.233 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:24:00.228+0000 7f5aa9176740 1 -- 172.21.15.118:0/1466314365 >> 172.21.15.118:0/1466314365 conn(0x55ba197c5960 msgr2=0x55ba197c4b70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:24:00.233 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:24:00.228+0000 7f5a8d600640 0 client.16728 ms_handle_reset on v2:172.21.15.189:6838/4074484168 2024-03-24T00:24:00.233 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:24:00.228+0000 7f5aa9176740 1 -- 172.21.15.118:0/1466314365 shutdown_connections 2024-03-24T00:24:00.233 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:24:00.228+0000 7f5aa9176740 1 -- 172.21.15.118:0/1466314365 wait complete. 2024-03-24T00:24:00.233 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:24:00.228+0000 7f5aa9176740 20 client.16728 trim_cache size 0 max 16384 2024-03-24T00:24:00.247 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:24:00.245+0000 7f5aa9176740 1 Processor -- start 2024-03-24T00:24:00.247 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:24:00.245+0000 7f5aa9176740 1 -- start start 2024-03-24T00:24:00.247 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:24:00.245+0000 7f5aa9176740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55ba197ca3f0 0x55ba197cc100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:24:00.247 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:24:00.245+0000 7f5aa9176740 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55ba197cade0 0x55ba197cc640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:24:00.247 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:24:00.245+0000 7f5aa9176740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55ba197cff50 0x55ba197dbb20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:24:00.247 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:24:00.245+0000 7f5aa9176740 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5aa01ad910 con 0x55ba197cade0 2024-03-24T00:24:00.247 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:24:00.245+0000 7f5aa9176740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5aa01aac70 con 0x55ba197ca3f0 2024-03-24T00:24:00.247 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:24:00.245+0000 7f5aa9176740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5aa01acc50 con 0x55ba197cff50 2024-03-24T00:24:00.247 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:24:00.245+0000 7f5a9f400640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55ba197ca3f0 0x55ba197cc100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:24:00.248 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:24:00.245+0000 7f5a9ea00640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55ba197cade0 0x55ba197cc640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:24:00.248 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:24:00.245+0000 7f5a9fe00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55ba197cff50 0x55ba197dbb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:24:00.248 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:24:00.245+0000 7f5a9f400640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55ba197ca3f0 0x55ba197cc100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.118:54328/0 (socket says 172.21.15.118:54328) 2024-03-24T00:24:00.248 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:24:00.245+0000 7f5a9f400640 1 -- 172.21.15.118:0/1441817571 learned_addr learned my addr 172.21.15.118:0/1441817571 (peer_addr_for_me v2:172.21.15.118:0/0) 2024-03-24T00:24:00.248 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:24:00.246+0000 7f5a9fe00640 1 -- 172.21.15.118:0/1441817571 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55ba197ca3f0 msgr2=0x55ba197cc100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:24:00.248 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:24:00.246+0000 7f5a9fe00640 1 --2- 172.21.15.118:0/1441817571 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55ba197ca3f0 0x55ba197cc100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:24:00.248 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:24:00.246+0000 7f5a9fe00640 1 -- 172.21.15.118:0/1441817571 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55ba197cade0 msgr2=0x55ba197cc640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:24:00.248 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:24:00.246+0000 7f5a9fe00640 1 --2- 172.21.15.118:0/1441817571 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55ba197cade0 0x55ba197cc640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:24:00.248 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:24:00.246+0000 7f5a9fe00640 1 -- 172.21.15.118:0/1441817571 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55ba197e2e90 con 0x55ba197cff50 2024-03-24T00:24:00.248 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:24:00.246+0000 7f5a9f400640 1 --2- 172.21.15.118:0/1441817571 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55ba197ca3f0 0x55ba197cc100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T00:24:00.248 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:24:00.246+0000 7f5a9fe00640 1 --2- 172.21.15.118:0/1441817571 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55ba197cff50 0x55ba197dbb20 secure :-1 s=READY pgs=454 cs=0 l=1 rev1=1 crypto rx=0x7f5a98006010 tx=0x7f5a98003e40 comp rx=0 tx=0).ready entity=mon.1 client_cookie=223c2768ad8e841f server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:24:00.248 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:24:00.246+0000 7f5a8fe00640 1 -- 172.21.15.118:0/1441817571 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5a98011050 con 0x55ba197cff50 2024-03-24T00:24:00.248 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:24:00.246+0000 7f5a8fe00640 1 -- 172.21.15.118:0/1441817571 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f5a9800fe80 con 0x55ba197cff50 2024-03-24T00:24:00.248 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:24:00.246+0000 7f5a8fe00640 1 -- 172.21.15.118:0/1441817571 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5a980056b0 con 0x55ba197cff50 2024-03-24T00:24:00.252 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:24:00.247+0000 7f5aa9176740 1 -- 172.21.15.118:0/1441817571 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55ba197cff50 msgr2=0x55ba197dbb20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:24:00.252 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:24:00.247+0000 7f5aa9176740 1 --2- 172.21.15.118:0/1441817571 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55ba197cff50 0x55ba197dbb20 secure :-1 s=READY pgs=454 cs=0 l=1 rev1=1 crypto rx=0x7f5a98006010 tx=0x7f5a98003e40 comp rx=0 tx=0).stop 2024-03-24T00:24:00.252 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:24:00.247+0000 7f5aa9176740 1 -- 172.21.15.118:0/1441817571 shutdown_connections 2024-03-24T00:24:00.252 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:24:00.247+0000 7f5aa9176740 1 --2- 172.21.15.118:0/1441817571 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55ba197cff50 0x55ba197dbb20 unknown :-1 s=CLOSED pgs=454 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:24:00.252 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:24:00.247+0000 7f5aa9176740 1 --2- 172.21.15.118:0/1441817571 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55ba197cade0 0x55ba197cc640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:24:00.253 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:24:00.247+0000 7f5aa9176740 1 --2- 172.21.15.118:0/1441817571 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55ba197ca3f0 0x55ba197cc100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:24:00.253 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:24:00.247+0000 7f5aa9176740 1 -- 172.21.15.118:0/1441817571 >> 172.21.15.118:0/1441817571 conn(0x55ba197c5960 msgr2=0x55ba197c06a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:24:00.253 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:24:00.247+0000 7f5aa9176740 1 -- 172.21.15.118:0/1441817571 shutdown_connections 2024-03-24T00:24:00.253 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:24:00.247+0000 7f5aa9176740 1 -- 172.21.15.118:0/1441817571 wait complete. 2024-03-24T00:24:00.253 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:24:00.248+0000 7f5aa9176740 1 -- start start 2024-03-24T00:24:00.253 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:24:00.248+0000 7f5aa9176740 1 Processor -- start 2024-03-24T00:24:00.253 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:24:00.248+0000 7f5aa9176740 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55ba197ca3f0 0x55ba197d95a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:24:00.253 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:24:00.248+0000 7f5aa9176740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55ba197cade0 0x55ba197d9ae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:24:00.253 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:24:00.249+0000 7f5aa9176740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55ba197cff50 0x55ba197daaf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:24:00.253 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:24:00.249+0000 7f5aa9176740 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5aa01ac5f0 con 0x55ba197ca3f0 2024-03-24T00:24:00.253 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:24:00.249+0000 7f5aa9176740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5aa01ab2d0 con 0x55ba197cff50 2024-03-24T00:24:00.253 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:24:00.249+0000 7f5aa9176740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5aa01ab930 con 0x55ba197cade0 2024-03-24T00:24:00.253 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:24:00.249+0000 7f5aa9176740 10 client.0 _ll_register_callbacks cb 0x7f5a880f4a30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T00:24:00.253 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:24:00.249+0000 7f5a9ea00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55ba197cade0 0x55ba197d9ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:24:00.254 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:24:00.249+0000 7f5a9f400640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55ba197ca3f0 0x55ba197d95a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:24:00.254 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:24:00.249+0000 7f5a9ea00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55ba197cade0 0x55ba197d9ae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.118:45154/0 (socket says 172.21.15.118:45154) 2024-03-24T00:24:00.254 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:24:00.249+0000 7f5a9fe00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55ba197cff50 0x55ba197daaf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:24:00.254 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:24:00.249+0000 7f5a9ea00640 1 -- 172.21.15.118:0/4182944604 learned_addr learned my addr 172.21.15.118:0/4182944604 (peer_addr_for_me v2:172.21.15.118:0/0) 2024-03-24T00:24:00.254 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:24:00.249+0000 7f5a9ea00640 1 -- 172.21.15.118:0/4182944604 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55ba197cff50 msgr2=0x55ba197daaf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:24:00.254 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:24:00.249+0000 7f5a9ea00640 1 --2- 172.21.15.118:0/4182944604 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55ba197cff50 0x55ba197daaf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:24:00.254 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:24:00.249+0000 7f5a9ea00640 1 -- 172.21.15.118:0/4182944604 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55ba197ca3f0 msgr2=0x55ba197d95a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:24:00.254 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:24:00.249+0000 7f5a9ea00640 1 --2- 172.21.15.118:0/4182944604 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55ba197ca3f0 0x55ba197d95a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:24:00.254 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:24:00.249+0000 7f5a9ea00640 1 -- 172.21.15.118:0/4182944604 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55ba197db0e0 con 0x55ba197cade0 2024-03-24T00:24:00.254 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:24:00.249+0000 7f5a9f400640 1 --2- 172.21.15.118:0/4182944604 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55ba197ca3f0 0x55ba197d95a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T00:24:00.254 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:24:00.249+0000 7f5a9ea00640 1 --2- 172.21.15.118:0/4182944604 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55ba197cade0 0x55ba197d9ae0 secure :-1 s=READY pgs=455 cs=0 l=1 rev1=1 crypto rx=0x7f5a880b9e70 tx=0x7f5a880bd380 comp rx=0 tx=0).ready entity=mon.1 client_cookie=c5157cc2f7113404 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:24:00.254 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:24:00.249+0000 7f5a8d600640 10 client.0 ms_handle_connect on v2:172.21.15.193:3300/0 2024-03-24T00:24:00.255 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:24:00.250+0000 7f5a8d600640 1 -- 172.21.15.118:0/4182944604 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5a880bd030 con 0x55ba197cade0 2024-03-24T00:24:00.255 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:24:00.250+0000 7f5a8d600640 1 -- 172.21.15.118:0/4182944604 <== mon.1 v2:172.21.15.193:3300/0 2 ==== osd_map(116..116 src has 1..116) v4 ==== 10513+0+0 (secure 0 0 0) 0x7f5a880bb7f0 con 0x55ba197cade0 2024-03-24T00:24:00.255 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:24:00.250+0000 7f5aa9176740 1 -- 172.21.15.118:0/4182944604 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x55ba197370c0 con 0x55ba197cade0 2024-03-24T00:24:00.255 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:24:00.250+0000 7f5a8d600640 1 -- 172.21.15.118:0/4182944604 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_get_version_reply(handle=1 version=154) v2 ==== 24+0+0 (secure 0 0 0) 0x7f5a88004c80 con 0x55ba197cade0 2024-03-24T00:24:00.255 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:24:00.250+0000 7f5aa9176740 10 client.26983 fetch_fsmap learned FSMap version 154 2024-03-24T00:24:00.255 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:24:00.250+0000 7f5aa9176740 1 -- 172.21.15.118:0/4182944604 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,fsmap=154,monmap=4+}) v3 -- 0x55ba1983bfe0 con 0x55ba197cade0 2024-03-24T00:24:00.255 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:24:00.250+0000 7f5a8d600640 1 -- 172.21.15.118:0/4182944604 <== mon.1 v2:172.21.15.193:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x55ba197370c0 con 0x55ba197cade0 2024-03-24T00:24:00.255 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:24:00.250+0000 7f5a8d600640 1 -- 172.21.15.118:0/4182944604 <== mon.1 v2:172.21.15.193:3300/0 5 ==== fsmap(e 154) v1 ==== 6221+0+0 (secure 0 0 0) 0x7f5a880c1120 con 0x55ba197cade0 2024-03-24T00:24:00.255 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:24:00.250+0000 7f5aa9176740 10 client.26983 fetch_fsmap finished waiting for FSMap version 154 2024-03-24T00:24:00.255 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:24:00.250+0000 7f5aa9176740 10 client.26983 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-24T00:24:00.255 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:24:00.250+0000 7f5aa9176740 20 client.26983 populate_metadata read hostname 'smithi118' 2024-03-24T00:24:00.255 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:24:00.250+0000 7f5aa9176740 1 --2- 172.21.15.118:0/4182944604 >> [v2:172.21.15.189:6838/4074484168,v1:172.21.15.189:6839/4074484168] conn(0x55ba1983c740 0x55ba1983ec00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:24:00.255 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:24:00.250+0000 7f5aa9176740 4 client.26983 mds_command: new command op to 24569 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-03-24T00:24:00.255 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:24:00.250+0000 7f5aa9176740 1 -- 172.21.15.118:0/4182944604 --> [v2:172.21.15.189:6838/4074484168,v1:172.21.15.189:6839/4074484168] -- command(tid 0: {"prefix": "flush journal"}) v1 -- 0x55ba196b55d0 con 0x55ba1983c740 2024-03-24T00:24:00.256 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:24:00.251+0000 7f5a9f400640 1 --2- 172.21.15.118:0/4182944604 >> [v2:172.21.15.189:6838/4074484168,v1:172.21.15.189:6839/4074484168] conn(0x55ba1983c740 0x55ba1983ec00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:24:00.256 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:24:00.251+0000 7f5a9f400640 1 --2- 172.21.15.118:0/4182944604 >> [v2:172.21.15.189:6838/4074484168,v1:172.21.15.189:6839/4074484168] conn(0x55ba1983c740 0x55ba1983ec00 crc :-1 s=READY pgs=358 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=dc65edc6de270316 server_cookie=df15e307af842da2 in_seq=0 out_seq=0 2024-03-24T00:24:00.256 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:24:00.251+0000 7f5a8d600640 10 client.26983 ms_handle_connect on v2:172.21.15.189:6838/4074484168 2024-03-24T00:24:00.444 INFO:journalctl@ceph.mds.j.smithi118.stdout:Mar 24 00:24:00 smithi118 podman[96171]: 2024-03-24 00:23:59.994801398 +0000 UTC m=+0.033003190 image pull c664afe56bb01db4f9afa9532c20598e201e664f2e9ba2be50db0d52674a26d2 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc 2024-03-24T00:24:00.444 INFO:journalctl@ceph.mds.j.smithi118.stdout:Mar 24 00:24:00 smithi118 podman[96171]: 2024-03-24 00:24:00.139861845 +0000 UTC m=+0.178063674 container create a9ed700118ad9459d1fd074773f67f75ea0d1f85f557b884101969b4e9f2004c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-j, org.label-schema.vendor=CentOS, ceph=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, org.label-schema.build-date=20240319, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux ) 2024-03-24T00:24:00.444 INFO:journalctl@ceph.mds.j.smithi118.stdout:Mar 24 00:24:00 smithi118 podman[96171]: 2024-03-24 00:24:00.44337727 +0000 UTC m=+0.481579106 container init a9ed700118ad9459d1fd074773f67f75ea0d1f85f557b884101969b4e9f2004c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-j, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, org.label-schema.build-date=20240319, ceph=True, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30) 2024-03-24T00:24:00.696 INFO:journalctl@ceph.mds.j.smithi118.stdout:Mar 24 00:24:00 smithi118 podman[96171]: 2024-03-24 00:24:00.453114657 +0000 UTC m=+0.491316478 container start a9ed700118ad9459d1fd074773f67f75ea0d1f85f557b884101969b4e9f2004c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-j, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, GIT_CLEAN=True, maintainer=Guillaume Abrioux , ceph=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.build-date=20240319, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c) 2024-03-24T00:24:00.697 INFO:journalctl@ceph.mds.j.smithi118.stdout:Mar 24 00:24:00 smithi118 ceph-mds[96234]: -- 172.21.15.118:0/563884186 <== mon.1 v2:172.21.15.193:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x565098ee9a40 con 0x565098ebc800 2024-03-24T00:24:00.697 INFO:journalctl@ceph.mds.j.smithi118.stdout:Mar 24 00:24:00 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-j[96230]: starting mds.j at 2024-03-24T00:24:00.697 INFO:journalctl@ceph.mds.j.smithi118.stdout:Mar 24 00:24:00 smithi118 bash[96171]: a9ed700118ad9459d1fd074773f67f75ea0d1f85f557b884101969b4e9f2004c 2024-03-24T00:24:00.697 INFO:journalctl@ceph.mds.j.smithi118.stdout:Mar 24 00:24:00 smithi118 systemd[1]: Started Ceph mds.j for d79a5420-e971-11ee-95cd-87774f69a715. 2024-03-24T00:24:01.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:24:00 smithi118 ceph-mon[31535]: daemon mds.j restarted 2024-03-24T00:24:01.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:24:00 smithi118 ceph-mon[31535]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-03-24T00:24:01.197 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:24:00 smithi118 ceph-mon[31535]: osdmap e117: 12 total, 12 up, 12 in 2024-03-24T00:24:01.197 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:24:00 smithi118 ceph-mon[31535]: mds.? [v2:172.21.15.118:6834/3766636479,v1:172.21.15.118:6835/3766636479] up:boot 2024-03-24T00:24:01.197 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:24:00 smithi118 ceph-mon[31535]: Standby daemon mds.j assigned to filesystem cephfs as rank 3 2024-03-24T00:24:01.197 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:24:00 smithi118 ceph-mon[31535]: fsmap cephfs:4/5 {0=b=up:active,1=h=up:active,2=c=up:active,4=e=up:active} 4 up:standby-replay 1 up:standby, 1 failed 2024-03-24T00:24:01.197 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:24:00 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2024-03-24T00:24:01.197 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:24:00 smithi118 ceph-mon[31535]: fsmap cephfs:5/5 4 up:standby-replay 1 up:replay 4 up:active 2024-03-24T00:24:01.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:24:00 smithi189 ceph-mon[36187]: daemon mds.j restarted 2024-03-24T00:24:01.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:24:00 smithi189 ceph-mon[36187]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-03-24T00:24:01.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:24:00 smithi189 ceph-mon[36187]: osdmap e117: 12 total, 12 up, 12 in 2024-03-24T00:24:01.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:24:00 smithi189 ceph-mon[36187]: mds.? [v2:172.21.15.118:6834/3766636479,v1:172.21.15.118:6835/3766636479] up:boot 2024-03-24T00:24:01.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:24:00 smithi189 ceph-mon[36187]: Standby daemon mds.j assigned to filesystem cephfs as rank 3 2024-03-24T00:24:01.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:24:00 smithi189 ceph-mon[36187]: fsmap cephfs:4/5 {0=b=up:active,1=h=up:active,2=c=up:active,4=e=up:active} 4 up:standby-replay 1 up:standby, 1 failed 2024-03-24T00:24:01.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:24:00 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2024-03-24T00:24:01.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:24:00 smithi189 ceph-mon[36187]: fsmap cephfs:5/5 4 up:standby-replay 1 up:replay 4 up:active 2024-03-24T00:24:01.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:24:00 smithi193 ceph-mon[36878]: daemon mds.j restarted 2024-03-24T00:24:01.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:24:00 smithi193 ceph-mon[36878]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-03-24T00:24:01.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:24:00 smithi193 ceph-mon[36878]: osdmap e117: 12 total, 12 up, 12 in 2024-03-24T00:24:01.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:24:00 smithi193 ceph-mon[36878]: mds.? [v2:172.21.15.118:6834/3766636479,v1:172.21.15.118:6835/3766636479] up:boot 2024-03-24T00:24:01.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:24:00 smithi193 ceph-mon[36878]: Standby daemon mds.j assigned to filesystem cephfs as rank 3 2024-03-24T00:24:01.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:24:00 smithi193 ceph-mon[36878]: fsmap cephfs:4/5 {0=b=up:active,1=h=up:active,2=c=up:active,4=e=up:active} 4 up:standby-replay 1 up:standby, 1 failed 2024-03-24T00:24:01.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:24:00 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2024-03-24T00:24:01.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:24:00 smithi193 ceph-mon[36878]: fsmap cephfs:5/5 4 up:standby-replay 1 up:replay 4 up:active 2024-03-24T00:24:02.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:24:02 smithi118 ceph-mon[31535]: mds.? [v2:172.21.15.118:6834/3766636479,v1:172.21.15.118:6835/3766636479] up:resolve 2024-03-24T00:24:02.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:24:02 smithi118 ceph-mon[31535]: fsmap cephfs:5/5 4 up:standby-replay 1 up:resolve 4 up:active 2024-03-24T00:24:02.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:24:02 smithi118 ceph-mon[31535]: pgmap v675: 97 pgs: 97 active+clean; 146 MiB data, 850 MiB used, 1.0 TiB / 1.0 TiB avail; 182 KiB/s rd, 17 KiB/s wr, 29 op/s 2024-03-24T00:24:02.953 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:24:02 smithi189 ceph-mon[36187]: mds.? [v2:172.21.15.118:6834/3766636479,v1:172.21.15.118:6835/3766636479] up:resolve 2024-03-24T00:24:02.953 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:24:02 smithi189 ceph-mon[36187]: fsmap cephfs:5/5 4 up:standby-replay 1 up:resolve 4 up:active 2024-03-24T00:24:02.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:24:02 smithi189 ceph-mon[36187]: pgmap v675: 97 pgs: 97 active+clean; 146 MiB data, 850 MiB used, 1.0 TiB / 1.0 TiB avail; 182 KiB/s rd, 17 KiB/s wr, 29 op/s 2024-03-24T00:24:02.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:24:02 smithi193 ceph-mon[36878]: mds.? [v2:172.21.15.118:6834/3766636479,v1:172.21.15.118:6835/3766636479] up:resolve 2024-03-24T00:24:02.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:24:02 smithi193 ceph-mon[36878]: fsmap cephfs:5/5 4 up:standby-replay 1 up:resolve 4 up:active 2024-03-24T00:24:02.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:24:02 smithi193 ceph-mon[36878]: pgmap v675: 97 pgs: 97 active+clean; 146 MiB data, 850 MiB used, 1.0 TiB / 1.0 TiB avail; 182 KiB/s rd, 17 KiB/s wr, 29 op/s 2024-03-24T00:24:04.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:24:03 smithi118 ceph-mon[31535]: pgmap v676: 97 pgs: 97 active+clean; 146 MiB data, 854 MiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 27 KiB/s wr, 12 op/s 2024-03-24T00:24:04.203 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:24:03 smithi189 ceph-mon[36187]: pgmap v676: 97 pgs: 97 active+clean; 146 MiB data, 854 MiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 27 KiB/s wr, 12 op/s 2024-03-24T00:24:04.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:24:03 smithi193 ceph-mon[36878]: pgmap v676: 97 pgs: 97 active+clean; 146 MiB data, 854 MiB used, 1.0 TiB / 1.0 TiB avail; 20 KiB/s rd, 27 KiB/s wr, 12 op/s 2024-03-24T00:24:06.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:24:05 smithi118 ceph-mon[31535]: pgmap v677: 97 pgs: 97 active+clean; 146 MiB data, 854 MiB used, 1.0 TiB / 1.0 TiB avail; 10 KiB/s rd, 12 KiB/s wr, 13 op/s 2024-03-24T00:24:06.203 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:24:05 smithi189 ceph-mon[36187]: pgmap v677: 97 pgs: 97 active+clean; 146 MiB data, 854 MiB used, 1.0 TiB / 1.0 TiB avail; 10 KiB/s rd, 12 KiB/s wr, 13 op/s 2024-03-24T00:24:06.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:24:05 smithi193 ceph-mon[36878]: pgmap v677: 97 pgs: 97 active+clean; 146 MiB data, 854 MiB used, 1.0 TiB / 1.0 TiB avail; 10 KiB/s rd, 12 KiB/s wr, 13 op/s 2024-03-24T00:24:07.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:24:06 smithi118 ceph-mon[31535]: Replacing daemon mds.h as rank 1 with standby daemon mds.d 2024-03-24T00:24:07.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:24:06 smithi118 ceph-mon[31535]: MDS daemon mds.h is removed because it is dead or otherwise unavailable. 2024-03-24T00:24:07.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:24:06 smithi118 ceph-mon[31535]: osdmap e118: 12 total, 12 up, 12 in 2024-03-24T00:24:07.197 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:24:06 smithi118 ceph-mon[31535]: fsmap cephfs:5/5 3 up:standby-replay 1 up:replay 1 up:resolve 3 up:active 2024-03-24T00:24:07.203 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:24:06 smithi189 ceph-mon[36187]: Replacing daemon mds.h as rank 1 with standby daemon mds.d 2024-03-24T00:24:07.203 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:24:06 smithi189 ceph-mon[36187]: MDS daemon mds.h is removed because it is dead or otherwise unavailable. 2024-03-24T00:24:07.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:24:06 smithi189 ceph-mon[36187]: osdmap e118: 12 total, 12 up, 12 in 2024-03-24T00:24:07.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:24:06 smithi189 ceph-mon[36187]: fsmap cephfs:5/5 3 up:standby-replay 1 up:replay 1 up:resolve 3 up:active 2024-03-24T00:24:07.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:24:06 smithi193 ceph-mon[36878]: Replacing daemon mds.h as rank 1 with standby daemon mds.d 2024-03-24T00:24:07.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:24:06 smithi193 ceph-mon[36878]: MDS daemon mds.h is removed because it is dead or otherwise unavailable. 2024-03-24T00:24:07.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:24:06 smithi193 ceph-mon[36878]: osdmap e118: 12 total, 12 up, 12 in 2024-03-24T00:24:07.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:24:06 smithi193 ceph-mon[36878]: fsmap cephfs:5/5 3 up:standby-replay 1 up:replay 1 up:resolve 3 up:active 2024-03-24T00:24:08.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:24:07 smithi193 ceph-mon[36878]: pgmap v679: 97 pgs: 97 active+clean; 146 MiB data, 858 MiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 15 KiB/s wr, 14 op/s 2024-03-24T00:24:08.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:24:07 smithi118 ceph-mon[31535]: pgmap v679: 97 pgs: 97 active+clean; 146 MiB data, 858 MiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 15 KiB/s wr, 14 op/s 2024-03-24T00:24:08.453 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:24:07 smithi189 ceph-mon[36187]: pgmap v679: 97 pgs: 97 active+clean; 146 MiB data, 858 MiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 15 KiB/s wr, 14 op/s 2024-03-24T00:24:09.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:24:08 smithi193 ceph-mon[36878]: mds.? [v2:172.21.15.118:6836/1531167856,v1:172.21.15.118:6837/1531167856] up:resolve 2024-03-24T00:24:09.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:24:08 smithi193 ceph-mon[36878]: fsmap cephfs:5/5 3 up:standby-replay 2 up:resolve 3 up:active 2024-03-24T00:24:09.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:24:08 smithi118 ceph-mon[31535]: mds.? [v2:172.21.15.118:6836/1531167856,v1:172.21.15.118:6837/1531167856] up:resolve 2024-03-24T00:24:09.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:24:08 smithi118 ceph-mon[31535]: fsmap cephfs:5/5 3 up:standby-replay 2 up:resolve 3 up:active 2024-03-24T00:24:09.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:24:08 smithi189 ceph-mon[36187]: mds.? [v2:172.21.15.118:6836/1531167856,v1:172.21.15.118:6837/1531167856] up:resolve 2024-03-24T00:24:09.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:24:08 smithi189 ceph-mon[36187]: fsmap cephfs:5/5 3 up:standby-replay 2 up:resolve 3 up:active 2024-03-24T00:24:10.012 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:24:10.012+0000 7f5a8d600640 1 -- 172.21.15.118:0/4182944604 <== mds.0 v2:172.21.15.189:6838/4074484168 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+44 (crc 0 0 0) 0x55ba196b55d0 con 0x55ba1983c740 2024-03-24T00:24:10.012 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:24:10.012+0000 7f5a8d600640 10 client.26983 handle_command_reply: tid=0 2024-03-24T00:24:10.012 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:24:10.012+0000 7f5aa9176740 1 client.26983 shutdown 2024-03-24T00:24:10.012 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:24:10.012+0000 7f5aa9176740 1 -- 172.21.15.118:0/4182944604 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55ba197cade0 msgr2=0x55ba197d9ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:24:10.012 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:24:10.012+0000 7f5aa9176740 1 --2- 172.21.15.118:0/4182944604 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55ba197cade0 0x55ba197d9ae0 secure :-1 s=READY pgs=455 cs=0 l=1 rev1=1 crypto rx=0x7f5a880b9e70 tx=0x7f5a880bd380 comp rx=0 tx=0).stop 2024-03-24T00:24:10.013 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:24:10.013+0000 7f5aa9176740 1 -- 172.21.15.118:0/4182944604 shutdown_connections 2024-03-24T00:24:10.013 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:24:10.013+0000 7f5aa9176740 1 --2- 172.21.15.118:0/4182944604 >> [v2:172.21.15.189:6838/4074484168,v1:172.21.15.189:6839/4074484168] conn(0x55ba1983c740 0x55ba1983ec00 crc :-1 s=READY pgs=358 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:24:10.013 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:24:10.013+0000 7f5aa9176740 1 --2- 172.21.15.118:0/4182944604 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55ba197cff50 0x55ba197daaf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:24:10.014 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:24:10.013+0000 7f5aa9176740 1 --2- 172.21.15.118:0/4182944604 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55ba197cade0 0x55ba197d9ae0 unknown :-1 s=CLOSED pgs=455 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:24:10.014 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:24:10.013+0000 7f5aa9176740 1 --2- 172.21.15.118:0/4182944604 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55ba197ca3f0 0x55ba197d95a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:24:10.014 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:24:10.013+0000 7f5a8d600640 0 client.26983 ms_handle_reset on v2:172.21.15.189:6838/4074484168 2024-03-24T00:24:10.014 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:24:10.013+0000 7f5aa9176740 1 -- 172.21.15.118:0/4182944604 >> 172.21.15.118:0/4182944604 conn(0x55ba197c5960 msgr2=0x55ba197d44b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:24:10.014 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:24:10.013+0000 7f5aa9176740 1 -- 172.21.15.118:0/4182944604 shutdown_connections 2024-03-24T00:24:10.014 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:24:10.013+0000 7f5aa9176740 1 -- 172.21.15.118:0/4182944604 wait complete. 2024-03-24T00:24:10.014 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:24:10.013+0000 7f5aa9176740 20 client.26983 trim_cache size 0 max 16384 2024-03-24T00:24:10.014 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stdout:{ 2024-03-24T00:24:10.015 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stdout: "message": "", 2024-03-24T00:24:10.015 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stdout: "return_code": 0 2024-03-24T00:24:10.015 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stdout:} 2024-03-24T00:24:10.015 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:24:10.014+0000 7f5aa4e00640 1 -- 172.21.15.118:0/756697505 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7f5a840a27f0 msgr2=0x7f5a840a4cb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:24:10.015 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:24:10.014+0000 7f5aa4e00640 1 --2- 172.21.15.118:0/756697505 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7f5a840a27f0 0x7f5a840a4cb0 secure :-1 s=READY pgs=390 cs=0 l=1 rev1=1 crypto rx=0x55ba197c68d0 tx=0x7f5a90005f90 comp rx=0 tx=0).stop 2024-03-24T00:24:10.016 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:24:10.014+0000 7f5aa4e00640 1 -- 172.21.15.118:0/756697505 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f5aa0102700 msgr2=0x7f5aa0199ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:24:10.016 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:24:10.014+0000 7f5aa4e00640 1 --2- 172.21.15.118:0/756697505 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f5aa0102700 0x7f5aa0199ee0 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7f5a88005ab0 tx=0x7f5a8802c6c0 comp rx=0 tx=0).stop 2024-03-24T00:24:10.016 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:24:10.014+0000 7f5aa4e00640 1 -- 172.21.15.118:0/756697505 shutdown_connections 2024-03-24T00:24:10.016 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:24:10.014+0000 7f5aa4e00640 1 --2- 172.21.15.118:0/756697505 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7f5a840a27f0 0x7f5a840a4cb0 unknown :-1 s=CLOSED pgs=390 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:24:10.016 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:24:10.014+0000 7f5aa4e00640 1 --2- 172.21.15.118:0/756697505 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f5aa01030c0 0x7f5aa01a0f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:24:10.016 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:24:10.014+0000 7f5aa4e00640 1 --2- 172.21.15.118:0/756697505 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f5aa0102700 0x7f5aa0199ee0 unknown :-1 s=CLOSED pgs=230 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:24:10.016 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:24:10.014+0000 7f5aa4e00640 1 --2- 172.21.15.118:0/756697505 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f5aa0101510 0x7f5aa01999a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:24:10.016 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:24:10.014+0000 7f5aa4e00640 1 -- 172.21.15.118:0/756697505 >> 172.21.15.118:0/756697505 conn(0x7f5aa00fca80 msgr2=0x7f5aa00738d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:24:10.016 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:24:10.014+0000 7f5aa4e00640 1 -- 172.21.15.118:0/756697505 shutdown_connections 2024-03-24T00:24:10.016 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:24:10.015+0000 7f5aa4e00640 1 -- 172.21.15.118:0/756697505 wait complete. 2024-03-24T00:24:10.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:24:09 smithi118 ceph-mon[31535]: mds.? [v2:172.21.15.118:6834/3766636479,v1:172.21.15.118:6835/3766636479] up:reconnect 2024-03-24T00:24:10.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:24:09 smithi118 ceph-mon[31535]: mds.? [v2:172.21.15.118:6836/1531167856,v1:172.21.15.118:6837/1531167856] up:reconnect 2024-03-24T00:24:10.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:24:09 smithi118 ceph-mon[31535]: fsmap cephfs:5/5 3 up:standby-replay 2 up:reconnect 3 up:active 2024-03-24T00:24:10.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:24:09 smithi118 ceph-mon[31535]: reconnect by client.15159 192.168.0.1:0/3463739313 after 0.0249994 2024-03-24T00:24:10.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:24:09 smithi118 ceph-mon[31535]: reconnect by client.15159 192.168.0.1:0/3463739313 after 0.0259994 2024-03-24T00:24:10.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:24:09 smithi118 ceph-mon[31535]: pgmap v680: 97 pgs: 97 active+clean; 146 MiB data, 858 MiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 13 KiB/s wr, 15 op/s 2024-03-24T00:24:10.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:24:09 smithi189 ceph-mon[36187]: mds.? [v2:172.21.15.118:6834/3766636479,v1:172.21.15.118:6835/3766636479] up:reconnect 2024-03-24T00:24:10.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:24:09 smithi189 ceph-mon[36187]: mds.? [v2:172.21.15.118:6836/1531167856,v1:172.21.15.118:6837/1531167856] up:reconnect 2024-03-24T00:24:10.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:24:09 smithi189 ceph-mon[36187]: fsmap cephfs:5/5 3 up:standby-replay 2 up:reconnect 3 up:active 2024-03-24T00:24:10.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:24:09 smithi189 ceph-mon[36187]: reconnect by client.15159 192.168.0.1:0/3463739313 after 0.0249994 2024-03-24T00:24:10.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:24:09 smithi189 ceph-mon[36187]: reconnect by client.15159 192.168.0.1:0/3463739313 after 0.0259994 2024-03-24T00:24:10.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:24:09 smithi189 ceph-mon[36187]: pgmap v680: 97 pgs: 97 active+clean; 146 MiB data, 858 MiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 13 KiB/s wr, 15 op/s 2024-03-24T00:24:10.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:24:09 smithi193 ceph-mon[36878]: mds.? [v2:172.21.15.118:6834/3766636479,v1:172.21.15.118:6835/3766636479] up:reconnect 2024-03-24T00:24:10.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:24:09 smithi193 ceph-mon[36878]: mds.? [v2:172.21.15.118:6836/1531167856,v1:172.21.15.118:6837/1531167856] up:reconnect 2024-03-24T00:24:10.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:24:09 smithi193 ceph-mon[36878]: fsmap cephfs:5/5 3 up:standby-replay 2 up:reconnect 3 up:active 2024-03-24T00:24:10.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:24:09 smithi193 ceph-mon[36878]: reconnect by client.15159 192.168.0.1:0/3463739313 after 0.0249994 2024-03-24T00:24:10.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:24:09 smithi193 ceph-mon[36878]: reconnect by client.15159 192.168.0.1:0/3463739313 after 0.0259994 2024-03-24T00:24:10.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:24:09 smithi193 ceph-mon[36878]: pgmap v680: 97 pgs: 97 active+clean; 146 MiB data, 858 MiB used, 1.0 TiB / 1.0 TiB avail; 12 KiB/s rd, 13 KiB/s wr, 15 op/s 2024-03-24T00:24:11.447 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:24:10 smithi118 ceph-mon[31535]: mds.? [v2:172.21.15.118:6836/1531167856,v1:172.21.15.118:6837/1531167856] up:rejoin 2024-03-24T00:24:11.447 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:24:10 smithi118 ceph-mon[31535]: mds.? [v2:172.21.15.118:6834/3766636479,v1:172.21.15.118:6835/3766636479] up:rejoin 2024-03-24T00:24:11.447 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:24:10 smithi118 ceph-mon[31535]: fsmap cephfs:5/5 3 up:standby-replay 2 up:rejoin 3 up:active 2024-03-24T00:24:11.447 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:24:10 smithi118 ceph-mon[31535]: daemon mds.j is now active in filesystem cephfs as rank 3 2024-03-24T00:24:11.447 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:24:10 smithi118 ceph-mon[31535]: daemon mds.d is now active in filesystem cephfs as rank 1 2024-03-24T00:24:11.448 INFO:journalctl@ceph.mds.j.smithi118.stdout:Mar 24 00:24:11 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-j[96230]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: In function 'void Locker::set_xlocks_done(MutationImpl*, bool)' thread 7f7478400700 time 2024-03-24T00:24:11.024865+0000 2024-03-24T00:24:11.448 INFO:journalctl@ceph.mds.j.smithi118.stdout:Mar 24 00:24:11 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-j[96230]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: 626: FAILED ceph_assert(obj->is_auth()) 2024-03-24T00:24:11.448 INFO:journalctl@ceph.mds.j.smithi118.stdout:Mar 24 00:24:11 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-j[96230]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-24T00:24:11.448 INFO:journalctl@ceph.mds.j.smithi118.stdout:Mar 24 00:24:11 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-j[96230]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x135) [0x7f7483cde4a1] 2024-03-24T00:24:11.448 INFO:journalctl@ceph.mds.j.smithi118.stdout:Mar 24 00:24:11 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-j[96230]: 2: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7f7483cde667] 2024-03-24T00:24:11.448 INFO:journalctl@ceph.mds.j.smithi118.stdout:Mar 24 00:24:11 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-j[96230]: 3: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x565096ffa223] 2024-03-24T00:24:11.449 INFO:journalctl@ceph.mds.j.smithi118.stdout:Mar 24 00:24:11 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-j[96230]: 4: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x565096e745d6] 2024-03-24T00:24:11.449 INFO:journalctl@ceph.mds.j.smithi118.stdout:Mar 24 00:24:11 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-j[96230]: 5: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x565096e74da1] 2024-03-24T00:24:11.449 INFO:journalctl@ceph.mds.j.smithi118.stdout:Mar 24 00:24:11 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-j[96230]: 6: (Server::handle_client_rename(boost::intrusive_ptr const&)+0x259d) [0x565096eb197d] 2024-03-24T00:24:11.449 INFO:journalctl@ceph.mds.j.smithi118.stdout:Mar 24 00:24:11 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-j[96230]: 7: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x565096ec7233] 2024-03-24T00:24:11.449 INFO:journalctl@ceph.mds.j.smithi118.stdout:Mar 24 00:24:11 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-j[96230]: 8: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x565096ec9197] 2024-03-24T00:24:11.449 INFO:journalctl@ceph.mds.j.smithi118.stdout:Mar 24 00:24:11 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-j[96230]: 9: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x565096dc811b] 2024-03-24T00:24:11.449 INFO:journalctl@ceph.mds.j.smithi118.stdout:Mar 24 00:24:11 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-j[96230]: 10: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x565096dc86bc] 2024-03-24T00:24:11.449 INFO:journalctl@ceph.mds.j.smithi118.stdout:Mar 24 00:24:11 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-j[96230]: 11: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x565096db21ff] 2024-03-24T00:24:11.449 INFO:journalctl@ceph.mds.j.smithi118.stdout:Mar 24 00:24:11 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-j[96230]: 12: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7f7483f70488] 2024-03-24T00:24:11.449 INFO:journalctl@ceph.mds.j.smithi118.stdout:Mar 24 00:24:11 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-j[96230]: 13: (DispatchQueue::entry()+0x51f) [0x7f7483f6d60f] 2024-03-24T00:24:11.449 INFO:journalctl@ceph.mds.j.smithi118.stdout:Mar 24 00:24:11 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-j[96230]: 14: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f7484040e81] 2024-03-24T00:24:11.449 INFO:journalctl@ceph.mds.j.smithi118.stdout:Mar 24 00:24:11 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-j[96230]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7f7482a5b1ca] 2024-03-24T00:24:11.449 INFO:journalctl@ceph.mds.j.smithi118.stdout:Mar 24 00:24:11 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-j[96230]: 16: clone() 2024-03-24T00:24:11.449 INFO:journalctl@ceph.mds.j.smithi118.stdout:Mar 24 00:24:11 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-j[96230]: *** Caught signal (Aborted) ** 2024-03-24T00:24:11.450 INFO:journalctl@ceph.mds.j.smithi118.stdout:Mar 24 00:24:11 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-j[96230]: in thread 7f7478400700 thread_name:ms_dispatch 2024-03-24T00:24:11.450 INFO:journalctl@ceph.mds.d.smithi118.stdout:Mar 24 00:24:11 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-d[95908]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: In function 'void Locker::set_xlocks_done(MutationImpl*, bool)' thread 7f15e0600700 time 2024-03-24T00:24:11.062894+0000 2024-03-24T00:24:11.450 INFO:journalctl@ceph.mds.d.smithi118.stdout:Mar 24 00:24:11 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-d[95908]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: 626: FAILED ceph_assert(obj->is_auth()) 2024-03-24T00:24:11.450 INFO:journalctl@ceph.mds.d.smithi118.stdout:Mar 24 00:24:11 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-d[95908]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-24T00:24:11.450 INFO:journalctl@ceph.mds.d.smithi118.stdout:Mar 24 00:24:11 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-d[95908]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x135) [0x7f15ebe374a1] 2024-03-24T00:24:11.450 INFO:journalctl@ceph.mds.d.smithi118.stdout:Mar 24 00:24:11 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-d[95908]: 2: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7f15ebe37667] 2024-03-24T00:24:11.450 INFO:journalctl@ceph.mds.d.smithi118.stdout:Mar 24 00:24:11 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-d[95908]: 3: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x55c0c24bd223] 2024-03-24T00:24:11.450 INFO:journalctl@ceph.mds.d.smithi118.stdout:Mar 24 00:24:11 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-d[95908]: 4: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x55c0c23375d6] 2024-03-24T00:24:11.451 INFO:journalctl@ceph.mds.d.smithi118.stdout:Mar 24 00:24:11 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-d[95908]: 5: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x55c0c2337da1] 2024-03-24T00:24:11.451 INFO:journalctl@ceph.mds.d.smithi118.stdout:Mar 24 00:24:11 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-d[95908]: 6: (Server::handle_client_rename(boost::intrusive_ptr const&)+0x259d) [0x55c0c237497d] 2024-03-24T00:24:11.451 INFO:journalctl@ceph.mds.d.smithi118.stdout:Mar 24 00:24:11 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-d[95908]: 7: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x55c0c238a233] 2024-03-24T00:24:11.451 INFO:journalctl@ceph.mds.d.smithi118.stdout:Mar 24 00:24:11 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-d[95908]: 8: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x55c0c238c197] 2024-03-24T00:24:11.451 INFO:journalctl@ceph.mds.d.smithi118.stdout:Mar 24 00:24:11 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-d[95908]: 9: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x55c0c228b11b] 2024-03-24T00:24:11.451 INFO:journalctl@ceph.mds.d.smithi118.stdout:Mar 24 00:24:11 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-d[95908]: 10: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x55c0c228b6bc] 2024-03-24T00:24:11.451 INFO:journalctl@ceph.mds.d.smithi118.stdout:Mar 24 00:24:11 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-d[95908]: 11: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x55c0c22751ff] 2024-03-24T00:24:11.451 INFO:journalctl@ceph.mds.d.smithi118.stdout:Mar 24 00:24:11 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-d[95908]: 12: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7f15ec0c9488] 2024-03-24T00:24:11.451 INFO:journalctl@ceph.mds.d.smithi118.stdout:Mar 24 00:24:11 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-d[95908]: 13: (DispatchQueue::entry()+0x51f) [0x7f15ec0c660f] 2024-03-24T00:24:11.451 INFO:journalctl@ceph.mds.d.smithi118.stdout:Mar 24 00:24:11 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-d[95908]: 14: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f15ec199e81] 2024-03-24T00:24:11.451 INFO:journalctl@ceph.mds.d.smithi118.stdout:Mar 24 00:24:11 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-d[95908]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7f15eabb41ca] 2024-03-24T00:24:11.451 INFO:journalctl@ceph.mds.d.smithi118.stdout:Mar 24 00:24:11 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-d[95908]: 16: clone() 2024-03-24T00:24:11.452 INFO:journalctl@ceph.mds.j.smithi118.stdout:Mar 24 00:24:11 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-j[96230]: 2024-03-24T00:24:11.026+0000 7f7478400700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: In function 'void Locker::set_xlocks_done(MutationImpl*, bool)' thread 7f7478400700 time 2024-03-24T00:24:11.024865+0000 2024-03-24T00:24:11.452 INFO:journalctl@ceph.mds.j.smithi118.stdout:Mar 24 00:24:11 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-j[96230]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: 626: FAILED ceph_assert(obj->is_auth()) 2024-03-24T00:24:11.452 INFO:journalctl@ceph.mds.j.smithi118.stdout:Mar 24 00:24:11 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-j[96230]: 2024-03-24T00:24:11.452 INFO:journalctl@ceph.mds.j.smithi118.stdout:Mar 24 00:24:11 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-j[96230]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-24T00:24:11.452 INFO:journalctl@ceph.mds.j.smithi118.stdout:Mar 24 00:24:11 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-j[96230]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x135) [0x7f7483cde4a1] 2024-03-24T00:24:11.452 INFO:journalctl@ceph.mds.j.smithi118.stdout:Mar 24 00:24:11 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-j[96230]: 2: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7f7483cde667] 2024-03-24T00:24:11.452 INFO:journalctl@ceph.mds.j.smithi118.stdout:Mar 24 00:24:11 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-j[96230]: 3: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x565096ffa223] 2024-03-24T00:24:11.452 INFO:journalctl@ceph.mds.j.smithi118.stdout:Mar 24 00:24:11 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-j[96230]: 4: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x565096e745d6] 2024-03-24T00:24:11.452 INFO:journalctl@ceph.mds.j.smithi118.stdout:Mar 24 00:24:11 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-j[96230]: 5: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x565096e74da1] 2024-03-24T00:24:11.452 INFO:journalctl@ceph.mds.j.smithi118.stdout:Mar 24 00:24:11 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-j[96230]: 6: (Server::handle_client_rename(boost::intrusive_ptr const&)+0x259d) [0x565096eb197d] 2024-03-24T00:24:11.453 INFO:journalctl@ceph.mds.j.smithi118.stdout:Mar 24 00:24:11 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-j[96230]: 7: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x565096ec7233] 2024-03-24T00:24:11.453 INFO:journalctl@ceph.mds.j.smithi118.stdout:Mar 24 00:24:11 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-j[96230]: 8: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x565096ec9197] 2024-03-24T00:24:11.453 INFO:journalctl@ceph.mds.j.smithi118.stdout:Mar 24 00:24:11 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-j[96230]: 9: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x565096dc811b] 2024-03-24T00:24:11.453 INFO:journalctl@ceph.mds.j.smithi118.stdout:Mar 24 00:24:11 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-j[96230]: 10: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x565096dc86bc] 2024-03-24T00:24:11.453 INFO:journalctl@ceph.mds.j.smithi118.stdout:Mar 24 00:24:11 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-j[96230]: 11: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x565096db21ff] 2024-03-24T00:24:11.453 INFO:journalctl@ceph.mds.j.smithi118.stdout:Mar 24 00:24:11 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-j[96230]: 12: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7f7483f70488] 2024-03-24T00:24:11.453 INFO:journalctl@ceph.mds.j.smithi118.stdout:Mar 24 00:24:11 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-j[96230]: 13: (DispatchQueue::entry()+0x51f) [0x7f7483f6d60f] 2024-03-24T00:24:11.453 INFO:journalctl@ceph.mds.j.smithi118.stdout:Mar 24 00:24:11 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-j[96230]: 14: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f7484040e81] 2024-03-24T00:24:11.453 INFO:journalctl@ceph.mds.j.smithi118.stdout:Mar 24 00:24:11 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-j[96230]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7f7482a5b1ca] 2024-03-24T00:24:11.453 INFO:journalctl@ceph.mds.j.smithi118.stdout:Mar 24 00:24:11 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-j[96230]: 16: clone() 2024-03-24T00:24:11.453 INFO:journalctl@ceph.mds.j.smithi118.stdout:Mar 24 00:24:11 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-j[96230]: 2024-03-24T00:24:11.454 INFO:journalctl@ceph.mds.j.smithi118.stdout:Mar 24 00:24:11 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-j[96230]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-24T00:24:11.454 INFO:journalctl@ceph.mds.j.smithi118.stdout:Mar 24 00:24:11 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-j[96230]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f7482a65d20] 2024-03-24T00:24:11.454 INFO:journalctl@ceph.mds.d.smithi118.stdout:Mar 24 00:24:11 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-d[95908]: 2024-03-24T00:24:11.064+0000 7f15e0600700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: In function 'void Locker::set_xlocks_done(MutationImpl*, bool)' thread 7f15e0600700 time 2024-03-24T00:24:11.062894+0000 2024-03-24T00:24:11.454 INFO:journalctl@ceph.mds.d.smithi118.stdout:Mar 24 00:24:11 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-d[95908]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: 626: FAILED ceph_assert(obj->is_auth()) 2024-03-24T00:24:11.454 INFO:journalctl@ceph.mds.d.smithi118.stdout:Mar 24 00:24:11 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-d[95908]: 2024-03-24T00:24:11.454 INFO:journalctl@ceph.mds.d.smithi118.stdout:Mar 24 00:24:11 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-d[95908]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-24T00:24:11.454 INFO:journalctl@ceph.mds.d.smithi118.stdout:Mar 24 00:24:11 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-d[95908]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x135) [0x7f15ebe374a1] 2024-03-24T00:24:11.454 INFO:journalctl@ceph.mds.d.smithi118.stdout:Mar 24 00:24:11 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-d[95908]: 2: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7f15ebe37667] 2024-03-24T00:24:11.454 INFO:journalctl@ceph.mds.d.smithi118.stdout:Mar 24 00:24:11 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-d[95908]: 3: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x55c0c24bd223] 2024-03-24T00:24:11.454 INFO:journalctl@ceph.mds.d.smithi118.stdout:Mar 24 00:24:11 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-d[95908]: 4: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x55c0c23375d6] 2024-03-24T00:24:11.455 INFO:journalctl@ceph.mds.d.smithi118.stdout:Mar 24 00:24:11 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-d[95908]: 5: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x55c0c2337da1] 2024-03-24T00:24:11.455 INFO:journalctl@ceph.mds.d.smithi118.stdout:Mar 24 00:24:11 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-d[95908]: 6: (Server::handle_client_rename(boost::intrusive_ptr const&)+0x259d) [0x55c0c237497d] 2024-03-24T00:24:11.455 INFO:journalctl@ceph.mds.d.smithi118.stdout:Mar 24 00:24:11 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-d[95908]: 7: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x55c0c238a233] 2024-03-24T00:24:11.455 INFO:journalctl@ceph.mds.d.smithi118.stdout:Mar 24 00:24:11 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-d[95908]: 8: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x55c0c238c197] 2024-03-24T00:24:11.455 INFO:journalctl@ceph.mds.d.smithi118.stdout:Mar 24 00:24:11 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-d[95908]: 9: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x55c0c228b11b] 2024-03-24T00:24:11.455 INFO:journalctl@ceph.mds.d.smithi118.stdout:Mar 24 00:24:11 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-d[95908]: 10: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x55c0c228b6bc] 2024-03-24T00:24:11.455 INFO:journalctl@ceph.mds.d.smithi118.stdout:Mar 24 00:24:11 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-d[95908]: 11: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x55c0c22751ff] 2024-03-24T00:24:11.455 INFO:journalctl@ceph.mds.d.smithi118.stdout:Mar 24 00:24:11 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-d[95908]: 12: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7f15ec0c9488] 2024-03-24T00:24:11.455 INFO:journalctl@ceph.mds.d.smithi118.stdout:Mar 24 00:24:11 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-d[95908]: 13: (DispatchQueue::entry()+0x51f) [0x7f15ec0c660f] 2024-03-24T00:24:11.455 INFO:journalctl@ceph.mds.d.smithi118.stdout:Mar 24 00:24:11 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-d[95908]: 14: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f15ec199e81] 2024-03-24T00:24:11.455 INFO:journalctl@ceph.mds.d.smithi118.stdout:Mar 24 00:24:11 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-d[95908]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7f15eabb41ca] 2024-03-24T00:24:11.455 INFO:journalctl@ceph.mds.d.smithi118.stdout:Mar 24 00:24:11 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-d[95908]: 16: clone() 2024-03-24T00:24:11.455 INFO:journalctl@ceph.mds.d.smithi118.stdout:Mar 24 00:24:11 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-d[95908]: 2024-03-24T00:24:11.456 INFO:journalctl@ceph.mds.d.smithi118.stdout:Mar 24 00:24:11 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-d[95908]: *** Caught signal (Aborted) ** 2024-03-24T00:24:11.456 INFO:journalctl@ceph.mds.d.smithi118.stdout:Mar 24 00:24:11 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-d[95908]: in thread 7f15e0600700 thread_name:ms_dispatch 2024-03-24T00:24:11.456 INFO:journalctl@ceph.mds.d.smithi118.stdout:Mar 24 00:24:11 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-d[95908]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-24T00:24:11.456 INFO:journalctl@ceph.mds.d.smithi118.stdout:Mar 24 00:24:11 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-d[95908]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f15eabbed20] 2024-03-24T00:24:11.456 INFO:journalctl@ceph.mds.j.smithi118.stdout:Mar 24 00:24:11 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-j[96230]: 2: gsignal() 2024-03-24T00:24:11.456 INFO:journalctl@ceph.mds.j.smithi118.stdout:Mar 24 00:24:11 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-j[96230]: 3: abort() 2024-03-24T00:24:11.456 INFO:journalctl@ceph.mds.j.smithi118.stdout:Mar 24 00:24:11 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-j[96230]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x18f) [0x7f7483cde4fb] 2024-03-24T00:24:11.456 INFO:journalctl@ceph.mds.j.smithi118.stdout:Mar 24 00:24:11 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-j[96230]: 5: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7f7483cde667] 2024-03-24T00:24:11.456 INFO:journalctl@ceph.mds.j.smithi118.stdout:Mar 24 00:24:11 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-j[96230]: 6: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x565096ffa223] 2024-03-24T00:24:11.457 INFO:journalctl@ceph.mds.j.smithi118.stdout:Mar 24 00:24:11 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-j[96230]: 7: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x565096e745d6] 2024-03-24T00:24:11.457 INFO:journalctl@ceph.mds.j.smithi118.stdout:Mar 24 00:24:11 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-j[96230]: 8: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x565096e74da1] 2024-03-24T00:24:11.457 INFO:journalctl@ceph.mds.j.smithi118.stdout:Mar 24 00:24:11 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-j[96230]: 9: (Server::handle_client_rename(boost::intrusive_ptr const&)+0x259d) [0x565096eb197d] 2024-03-24T00:24:11.457 INFO:journalctl@ceph.mds.j.smithi118.stdout:Mar 24 00:24:11 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-j[96230]: 10: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x565096ec7233] 2024-03-24T00:24:11.457 INFO:journalctl@ceph.mds.j.smithi118.stdout:Mar 24 00:24:11 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-j[96230]: 11: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x565096ec9197] 2024-03-24T00:24:11.457 INFO:journalctl@ceph.mds.j.smithi118.stdout:Mar 24 00:24:11 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-j[96230]: 12: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x565096dc811b] 2024-03-24T00:24:11.457 INFO:journalctl@ceph.mds.j.smithi118.stdout:Mar 24 00:24:11 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-j[96230]: 13: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x565096dc86bc] 2024-03-24T00:24:11.457 INFO:journalctl@ceph.mds.j.smithi118.stdout:Mar 24 00:24:11 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-j[96230]: 14: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x565096db21ff] 2024-03-24T00:24:11.457 INFO:journalctl@ceph.mds.j.smithi118.stdout:Mar 24 00:24:11 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-j[96230]: 15: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7f7483f70488] 2024-03-24T00:24:11.457 INFO:journalctl@ceph.mds.j.smithi118.stdout:Mar 24 00:24:11 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-j[96230]: 16: (DispatchQueue::entry()+0x51f) [0x7f7483f6d60f] 2024-03-24T00:24:11.457 INFO:journalctl@ceph.mds.j.smithi118.stdout:Mar 24 00:24:11 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-j[96230]: 17: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f7484040e81] 2024-03-24T00:24:11.457 INFO:journalctl@ceph.mds.j.smithi118.stdout:Mar 24 00:24:11 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-j[96230]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7f7482a5b1ca] 2024-03-24T00:24:11.457 INFO:journalctl@ceph.mds.j.smithi118.stdout:Mar 24 00:24:11 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-j[96230]: 19: clone() 2024-03-24T00:24:11.458 INFO:journalctl@ceph.mds.j.smithi118.stdout:Mar 24 00:24:11 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-j[96230]: 2024-03-24T00:24:11.029+0000 7f7478400700 -1 *** Caught signal (Aborted) ** 2024-03-24T00:24:11.458 INFO:journalctl@ceph.mds.j.smithi118.stdout:Mar 24 00:24:11 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-j[96230]: in thread 7f7478400700 thread_name:ms_dispatch 2024-03-24T00:24:11.458 INFO:journalctl@ceph.mds.j.smithi118.stdout:Mar 24 00:24:11 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-j[96230]: 2024-03-24T00:24:11.458 INFO:journalctl@ceph.mds.j.smithi118.stdout:Mar 24 00:24:11 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-j[96230]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-24T00:24:11.458 INFO:journalctl@ceph.mds.j.smithi118.stdout:Mar 24 00:24:11 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-j[96230]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f7482a65d20] 2024-03-24T00:24:11.458 INFO:journalctl@ceph.mds.j.smithi118.stdout:Mar 24 00:24:11 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-j[96230]: 2: gsignal() 2024-03-24T00:24:11.458 INFO:journalctl@ceph.mds.j.smithi118.stdout:Mar 24 00:24:11 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-j[96230]: 3: abort() 2024-03-24T00:24:11.458 INFO:journalctl@ceph.mds.j.smithi118.stdout:Mar 24 00:24:11 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-j[96230]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x18f) [0x7f7483cde4fb] 2024-03-24T00:24:11.458 INFO:journalctl@ceph.mds.j.smithi118.stdout:Mar 24 00:24:11 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-j[96230]: 5: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7f7483cde667] 2024-03-24T00:24:11.458 INFO:journalctl@ceph.mds.d.smithi118.stdout:Mar 24 00:24:11 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-d[95908]: 2: gsignal() 2024-03-24T00:24:11.459 INFO:journalctl@ceph.mds.d.smithi118.stdout:Mar 24 00:24:11 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-d[95908]: 3: abort() 2024-03-24T00:24:11.459 INFO:journalctl@ceph.mds.d.smithi118.stdout:Mar 24 00:24:11 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-d[95908]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x18f) [0x7f15ebe374fb] 2024-03-24T00:24:11.459 INFO:journalctl@ceph.mds.d.smithi118.stdout:Mar 24 00:24:11 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-d[95908]: 5: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7f15ebe37667] 2024-03-24T00:24:11.459 INFO:journalctl@ceph.mds.d.smithi118.stdout:Mar 24 00:24:11 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-d[95908]: 6: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x55c0c24bd223] 2024-03-24T00:24:11.459 INFO:journalctl@ceph.mds.d.smithi118.stdout:Mar 24 00:24:11 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-d[95908]: 7: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x55c0c23375d6] 2024-03-24T00:24:11.459 INFO:journalctl@ceph.mds.d.smithi118.stdout:Mar 24 00:24:11 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-d[95908]: 8: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x55c0c2337da1] 2024-03-24T00:24:11.459 INFO:journalctl@ceph.mds.d.smithi118.stdout:Mar 24 00:24:11 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-d[95908]: 9: (Server::handle_client_rename(boost::intrusive_ptr const&)+0x259d) [0x55c0c237497d] 2024-03-24T00:24:11.459 INFO:journalctl@ceph.mds.d.smithi118.stdout:Mar 24 00:24:11 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-d[95908]: 10: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x55c0c238a233] 2024-03-24T00:24:11.459 INFO:journalctl@ceph.mds.d.smithi118.stdout:Mar 24 00:24:11 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-d[95908]: 11: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x55c0c238c197] 2024-03-24T00:24:11.459 INFO:journalctl@ceph.mds.d.smithi118.stdout:Mar 24 00:24:11 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-d[95908]: 12: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x55c0c228b11b] 2024-03-24T00:24:11.459 INFO:journalctl@ceph.mds.d.smithi118.stdout:Mar 24 00:24:11 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-d[95908]: 13: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x55c0c228b6bc] 2024-03-24T00:24:11.459 INFO:journalctl@ceph.mds.d.smithi118.stdout:Mar 24 00:24:11 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-d[95908]: 14: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x55c0c22751ff] 2024-03-24T00:24:11.459 INFO:journalctl@ceph.mds.d.smithi118.stdout:Mar 24 00:24:11 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-d[95908]: 15: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7f15ec0c9488] 2024-03-24T00:24:11.459 INFO:journalctl@ceph.mds.d.smithi118.stdout:Mar 24 00:24:11 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-d[95908]: 16: (DispatchQueue::entry()+0x51f) [0x7f15ec0c660f] 2024-03-24T00:24:11.459 INFO:journalctl@ceph.mds.d.smithi118.stdout:Mar 24 00:24:11 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-d[95908]: 17: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f15ec199e81] 2024-03-24T00:24:11.459 INFO:journalctl@ceph.mds.d.smithi118.stdout:Mar 24 00:24:11 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-d[95908]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7f15eabb41ca] 2024-03-24T00:24:11.460 INFO:journalctl@ceph.mds.d.smithi118.stdout:Mar 24 00:24:11 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-d[95908]: 19: clone() 2024-03-24T00:24:11.460 INFO:journalctl@ceph.mds.d.smithi118.stdout:Mar 24 00:24:11 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-d[95908]: 2024-03-24T00:24:11.067+0000 7f15e0600700 -1 *** Caught signal (Aborted) ** 2024-03-24T00:24:11.460 INFO:journalctl@ceph.mds.d.smithi118.stdout:Mar 24 00:24:11 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-d[95908]: in thread 7f15e0600700 thread_name:ms_dispatch 2024-03-24T00:24:11.460 INFO:journalctl@ceph.mds.d.smithi118.stdout:Mar 24 00:24:11 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-d[95908]: 2024-03-24T00:24:11.460 INFO:journalctl@ceph.mds.d.smithi118.stdout:Mar 24 00:24:11 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-d[95908]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-24T00:24:11.460 INFO:journalctl@ceph.mds.d.smithi118.stdout:Mar 24 00:24:11 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-d[95908]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f15eabbed20] 2024-03-24T00:24:11.460 INFO:journalctl@ceph.mds.d.smithi118.stdout:Mar 24 00:24:11 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-d[95908]: 2: gsignal() 2024-03-24T00:24:11.460 INFO:journalctl@ceph.mds.d.smithi118.stdout:Mar 24 00:24:11 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-d[95908]: 3: abort() 2024-03-24T00:24:11.460 INFO:journalctl@ceph.mds.d.smithi118.stdout:Mar 24 00:24:11 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-d[95908]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x18f) [0x7f15ebe374fb] 2024-03-24T00:24:11.460 INFO:journalctl@ceph.mds.d.smithi118.stdout:Mar 24 00:24:11 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-d[95908]: 5: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7f15ebe37667] 2024-03-24T00:24:11.460 INFO:journalctl@ceph.mds.j.smithi118.stdout:Mar 24 00:24:11 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-j[96230]: 6: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x565096ffa223] 2024-03-24T00:24:11.461 INFO:journalctl@ceph.mds.j.smithi118.stdout:Mar 24 00:24:11 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-j[96230]: 7: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x565096e745d6] 2024-03-24T00:24:11.461 INFO:journalctl@ceph.mds.j.smithi118.stdout:Mar 24 00:24:11 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-j[96230]: 8: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x565096e74da1] 2024-03-24T00:24:11.461 INFO:journalctl@ceph.mds.j.smithi118.stdout:Mar 24 00:24:11 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-j[96230]: 9: (Server::handle_client_rename(boost::intrusive_ptr const&)+0x259d) [0x565096eb197d] 2024-03-24T00:24:11.461 INFO:journalctl@ceph.mds.j.smithi118.stdout:Mar 24 00:24:11 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-j[96230]: 10: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x565096ec7233] 2024-03-24T00:24:11.461 INFO:journalctl@ceph.mds.j.smithi118.stdout:Mar 24 00:24:11 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-j[96230]: 11: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x565096ec9197] 2024-03-24T00:24:11.461 INFO:journalctl@ceph.mds.j.smithi118.stdout:Mar 24 00:24:11 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-j[96230]: 12: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x565096dc811b] 2024-03-24T00:24:11.461 INFO:journalctl@ceph.mds.j.smithi118.stdout:Mar 24 00:24:11 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-j[96230]: 13: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x565096dc86bc] 2024-03-24T00:24:11.461 INFO:journalctl@ceph.mds.j.smithi118.stdout:Mar 24 00:24:11 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-j[96230]: 14: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x565096db21ff] 2024-03-24T00:24:11.461 INFO:journalctl@ceph.mds.j.smithi118.stdout:Mar 24 00:24:11 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-j[96230]: 15: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7f7483f70488] 2024-03-24T00:24:11.461 INFO:journalctl@ceph.mds.j.smithi118.stdout:Mar 24 00:24:11 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-j[96230]: 16: (DispatchQueue::entry()+0x51f) [0x7f7483f6d60f] 2024-03-24T00:24:11.461 INFO:journalctl@ceph.mds.j.smithi118.stdout:Mar 24 00:24:11 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-j[96230]: 17: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f7484040e81] 2024-03-24T00:24:11.461 INFO:journalctl@ceph.mds.j.smithi118.stdout:Mar 24 00:24:11 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-j[96230]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7f7482a5b1ca] 2024-03-24T00:24:11.461 INFO:journalctl@ceph.mds.j.smithi118.stdout:Mar 24 00:24:11 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-j[96230]: 19: clone() 2024-03-24T00:24:11.461 INFO:journalctl@ceph.mds.j.smithi118.stdout:Mar 24 00:24:11 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-j[96230]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-03-24T00:24:11.461 INFO:journalctl@ceph.mds.j.smithi118.stdout:Mar 24 00:24:11 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-j[96230]: 2024-03-24T00:24:11.462 INFO:journalctl@ceph.mds.j.smithi118.stdout:Mar 24 00:24:11 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-j[96230]: -1> 2024-03-24T00:24:11.026+0000 7f7478400700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: In function 'void Locker::set_xlocks_done(MutationImpl*, bool)' thread 7f7478400700 time 2024-03-24T00:24:11.024865+0000 2024-03-24T00:24:11.462 INFO:journalctl@ceph.mds.j.smithi118.stdout:Mar 24 00:24:11 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-j[96230]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: 626: FAILED ceph_assert(obj->is_auth()) 2024-03-24T00:24:11.462 INFO:journalctl@ceph.mds.j.smithi118.stdout:Mar 24 00:24:11 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-j[96230]: 2024-03-24T00:24:11.462 INFO:journalctl@ceph.mds.j.smithi118.stdout:Mar 24 00:24:11 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-j[96230]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-24T00:24:11.462 INFO:journalctl@ceph.mds.j.smithi118.stdout:Mar 24 00:24:11 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-j[96230]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x135) [0x7f7483cde4a1] 2024-03-24T00:24:11.462 INFO:journalctl@ceph.mds.j.smithi118.stdout:Mar 24 00:24:11 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-j[96230]: 2: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7f7483cde667] 2024-03-24T00:24:11.462 INFO:journalctl@ceph.mds.d.smithi118.stdout:Mar 24 00:24:11 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-d[95908]: 6: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x55c0c24bd223] 2024-03-24T00:24:11.462 INFO:journalctl@ceph.mds.d.smithi118.stdout:Mar 24 00:24:11 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-d[95908]: 7: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x55c0c23375d6] 2024-03-24T00:24:11.462 INFO:journalctl@ceph.mds.d.smithi118.stdout:Mar 24 00:24:11 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-d[95908]: 8: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x55c0c2337da1] 2024-03-24T00:24:11.463 INFO:journalctl@ceph.mds.d.smithi118.stdout:Mar 24 00:24:11 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-d[95908]: 9: (Server::handle_client_rename(boost::intrusive_ptr const&)+0x259d) [0x55c0c237497d] 2024-03-24T00:24:11.463 INFO:journalctl@ceph.mds.d.smithi118.stdout:Mar 24 00:24:11 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-d[95908]: 10: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x55c0c238a233] 2024-03-24T00:24:11.463 INFO:journalctl@ceph.mds.d.smithi118.stdout:Mar 24 00:24:11 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-d[95908]: 11: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x55c0c238c197] 2024-03-24T00:24:11.463 INFO:journalctl@ceph.mds.d.smithi118.stdout:Mar 24 00:24:11 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-d[95908]: 12: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x55c0c228b11b] 2024-03-24T00:24:11.463 INFO:journalctl@ceph.mds.d.smithi118.stdout:Mar 24 00:24:11 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-d[95908]: 13: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x55c0c228b6bc] 2024-03-24T00:24:11.463 INFO:journalctl@ceph.mds.d.smithi118.stdout:Mar 24 00:24:11 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-d[95908]: 14: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x55c0c22751ff] 2024-03-24T00:24:11.463 INFO:journalctl@ceph.mds.d.smithi118.stdout:Mar 24 00:24:11 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-d[95908]: 15: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7f15ec0c9488] 2024-03-24T00:24:11.463 INFO:journalctl@ceph.mds.d.smithi118.stdout:Mar 24 00:24:11 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-d[95908]: 16: (DispatchQueue::entry()+0x51f) [0x7f15ec0c660f] 2024-03-24T00:24:11.463 INFO:journalctl@ceph.mds.d.smithi118.stdout:Mar 24 00:24:11 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-d[95908]: 17: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f15ec199e81] 2024-03-24T00:24:11.463 INFO:journalctl@ceph.mds.d.smithi118.stdout:Mar 24 00:24:11 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-d[95908]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7f15eabb41ca] 2024-03-24T00:24:11.463 INFO:journalctl@ceph.mds.d.smithi118.stdout:Mar 24 00:24:11 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-d[95908]: 19: clone() 2024-03-24T00:24:11.463 INFO:journalctl@ceph.mds.d.smithi118.stdout:Mar 24 00:24:11 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-d[95908]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-03-24T00:24:11.463 INFO:journalctl@ceph.mds.d.smithi118.stdout:Mar 24 00:24:11 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-d[95908]: 2024-03-24T00:24:11.464 INFO:journalctl@ceph.mds.d.smithi118.stdout:Mar 24 00:24:11 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-d[95908]: -1> 2024-03-24T00:24:11.064+0000 7f15e0600700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: In function 'void Locker::set_xlocks_done(MutationImpl*, bool)' thread 7f15e0600700 time 2024-03-24T00:24:11.062894+0000 2024-03-24T00:24:11.464 INFO:journalctl@ceph.mds.d.smithi118.stdout:Mar 24 00:24:11 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-d[95908]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: 626: FAILED ceph_assert(obj->is_auth()) 2024-03-24T00:24:11.464 INFO:journalctl@ceph.mds.d.smithi118.stdout:Mar 24 00:24:11 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-d[95908]: 2024-03-24T00:24:11.464 INFO:journalctl@ceph.mds.d.smithi118.stdout:Mar 24 00:24:11 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-d[95908]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-24T00:24:11.464 INFO:journalctl@ceph.mds.d.smithi118.stdout:Mar 24 00:24:11 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-d[95908]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x135) [0x7f15ebe374a1] 2024-03-24T00:24:11.464 INFO:journalctl@ceph.mds.d.smithi118.stdout:Mar 24 00:24:11 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-d[95908]: 2: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7f15ebe37667] 2024-03-24T00:24:11.464 INFO:journalctl@ceph.mds.j.smithi118.stdout:Mar 24 00:24:11 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-j[96230]: 3: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x565096ffa223] 2024-03-24T00:24:11.464 INFO:journalctl@ceph.mds.j.smithi118.stdout:Mar 24 00:24:11 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-j[96230]: 4: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x565096e745d6] 2024-03-24T00:24:11.464 INFO:journalctl@ceph.mds.j.smithi118.stdout:Mar 24 00:24:11 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-j[96230]: 5: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x565096e74da1] 2024-03-24T00:24:11.464 INFO:journalctl@ceph.mds.j.smithi118.stdout:Mar 24 00:24:11 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-j[96230]: 6: (Server::handle_client_rename(boost::intrusive_ptr const&)+0x259d) [0x565096eb197d] 2024-03-24T00:24:11.464 INFO:journalctl@ceph.mds.j.smithi118.stdout:Mar 24 00:24:11 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-j[96230]: 7: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x565096ec7233] 2024-03-24T00:24:11.464 INFO:journalctl@ceph.mds.j.smithi118.stdout:Mar 24 00:24:11 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-j[96230]: 8: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x565096ec9197] 2024-03-24T00:24:11.465 INFO:journalctl@ceph.mds.j.smithi118.stdout:Mar 24 00:24:11 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-j[96230]: 9: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x565096dc811b] 2024-03-24T00:24:11.465 INFO:journalctl@ceph.mds.j.smithi118.stdout:Mar 24 00:24:11 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-j[96230]: 10: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x565096dc86bc] 2024-03-24T00:24:11.465 INFO:journalctl@ceph.mds.j.smithi118.stdout:Mar 24 00:24:11 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-j[96230]: 11: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x565096db21ff] 2024-03-24T00:24:11.465 INFO:journalctl@ceph.mds.j.smithi118.stdout:Mar 24 00:24:11 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-j[96230]: 12: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7f7483f70488] 2024-03-24T00:24:11.465 INFO:journalctl@ceph.mds.j.smithi118.stdout:Mar 24 00:24:11 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-j[96230]: 13: (DispatchQueue::entry()+0x51f) [0x7f7483f6d60f] 2024-03-24T00:24:11.465 INFO:journalctl@ceph.mds.j.smithi118.stdout:Mar 24 00:24:11 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-j[96230]: 14: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f7484040e81] 2024-03-24T00:24:11.465 INFO:journalctl@ceph.mds.j.smithi118.stdout:Mar 24 00:24:11 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-j[96230]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7f7482a5b1ca] 2024-03-24T00:24:11.465 INFO:journalctl@ceph.mds.j.smithi118.stdout:Mar 24 00:24:11 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-j[96230]: 16: clone() 2024-03-24T00:24:11.465 INFO:journalctl@ceph.mds.j.smithi118.stdout:Mar 24 00:24:11 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-j[96230]: 2024-03-24T00:24:11.465 INFO:journalctl@ceph.mds.j.smithi118.stdout:Mar 24 00:24:11 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-j[96230]: 0> 2024-03-24T00:24:11.029+0000 7f7478400700 -1 *** Caught signal (Aborted) ** 2024-03-24T00:24:11.465 INFO:journalctl@ceph.mds.j.smithi118.stdout:Mar 24 00:24:11 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-j[96230]: in thread 7f7478400700 thread_name:ms_dispatch 2024-03-24T00:24:11.465 INFO:journalctl@ceph.mds.j.smithi118.stdout:Mar 24 00:24:11 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-j[96230]: 2024-03-24T00:24:11.465 INFO:journalctl@ceph.mds.j.smithi118.stdout:Mar 24 00:24:11 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-j[96230]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-24T00:24:11.465 INFO:journalctl@ceph.mds.j.smithi118.stdout:Mar 24 00:24:11 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-j[96230]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f7482a65d20] 2024-03-24T00:24:11.465 INFO:journalctl@ceph.mds.j.smithi118.stdout:Mar 24 00:24:11 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-j[96230]: 2: gsignal() 2024-03-24T00:24:11.465 INFO:journalctl@ceph.mds.j.smithi118.stdout:Mar 24 00:24:11 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-j[96230]: 3: abort() 2024-03-24T00:24:11.466 INFO:journalctl@ceph.mds.j.smithi118.stdout:Mar 24 00:24:11 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-j[96230]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x18f) [0x7f7483cde4fb] 2024-03-24T00:24:11.466 INFO:journalctl@ceph.mds.j.smithi118.stdout:Mar 24 00:24:11 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-j[96230]: 5: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7f7483cde667] 2024-03-24T00:24:11.466 INFO:journalctl@ceph.mds.j.smithi118.stdout:Mar 24 00:24:11 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-j[96230]: 6: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x565096ffa223] 2024-03-24T00:24:11.466 INFO:journalctl@ceph.mds.j.smithi118.stdout:Mar 24 00:24:11 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-j[96230]: 7: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x565096e745d6] 2024-03-24T00:24:11.466 INFO:journalctl@ceph.mds.d.smithi118.stdout:Mar 24 00:24:11 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-d[95908]: 3: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x55c0c24bd223] 2024-03-24T00:24:11.466 INFO:journalctl@ceph.mds.d.smithi118.stdout:Mar 24 00:24:11 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-d[95908]: 4: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x55c0c23375d6] 2024-03-24T00:24:11.466 INFO:journalctl@ceph.mds.d.smithi118.stdout:Mar 24 00:24:11 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-d[95908]: 5: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x55c0c2337da1] 2024-03-24T00:24:11.466 INFO:journalctl@ceph.mds.d.smithi118.stdout:Mar 24 00:24:11 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-d[95908]: 6: (Server::handle_client_rename(boost::intrusive_ptr const&)+0x259d) [0x55c0c237497d] 2024-03-24T00:24:11.466 INFO:journalctl@ceph.mds.d.smithi118.stdout:Mar 24 00:24:11 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-d[95908]: 7: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x55c0c238a233] 2024-03-24T00:24:11.466 INFO:journalctl@ceph.mds.d.smithi118.stdout:Mar 24 00:24:11 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-d[95908]: 8: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x55c0c238c197] 2024-03-24T00:24:11.466 INFO:journalctl@ceph.mds.d.smithi118.stdout:Mar 24 00:24:11 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-d[95908]: 9: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x55c0c228b11b] 2024-03-24T00:24:11.466 INFO:journalctl@ceph.mds.d.smithi118.stdout:Mar 24 00:24:11 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-d[95908]: 10: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x55c0c228b6bc] 2024-03-24T00:24:11.467 INFO:journalctl@ceph.mds.d.smithi118.stdout:Mar 24 00:24:11 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-d[95908]: 11: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x55c0c22751ff] 2024-03-24T00:24:11.467 INFO:journalctl@ceph.mds.d.smithi118.stdout:Mar 24 00:24:11 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-d[95908]: 12: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7f15ec0c9488] 2024-03-24T00:24:11.467 INFO:journalctl@ceph.mds.d.smithi118.stdout:Mar 24 00:24:11 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-d[95908]: 13: (DispatchQueue::entry()+0x51f) [0x7f15ec0c660f] 2024-03-24T00:24:11.467 INFO:journalctl@ceph.mds.d.smithi118.stdout:Mar 24 00:24:11 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-d[95908]: 14: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f15ec199e81] 2024-03-24T00:24:11.467 INFO:journalctl@ceph.mds.d.smithi118.stdout:Mar 24 00:24:11 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-d[95908]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7f15eabb41ca] 2024-03-24T00:24:11.467 INFO:journalctl@ceph.mds.d.smithi118.stdout:Mar 24 00:24:11 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-d[95908]: 16: clone() 2024-03-24T00:24:11.467 INFO:journalctl@ceph.mds.d.smithi118.stdout:Mar 24 00:24:11 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-d[95908]: 2024-03-24T00:24:11.467 INFO:journalctl@ceph.mds.d.smithi118.stdout:Mar 24 00:24:11 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-d[95908]: 0> 2024-03-24T00:24:11.067+0000 7f15e0600700 -1 *** Caught signal (Aborted) ** 2024-03-24T00:24:11.467 INFO:journalctl@ceph.mds.d.smithi118.stdout:Mar 24 00:24:11 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-d[95908]: in thread 7f15e0600700 thread_name:ms_dispatch 2024-03-24T00:24:11.467 INFO:journalctl@ceph.mds.d.smithi118.stdout:Mar 24 00:24:11 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-d[95908]: 2024-03-24T00:24:11.467 INFO:journalctl@ceph.mds.d.smithi118.stdout:Mar 24 00:24:11 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-d[95908]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-24T00:24:11.467 INFO:journalctl@ceph.mds.d.smithi118.stdout:Mar 24 00:24:11 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-d[95908]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f15eabbed20] 2024-03-24T00:24:11.467 INFO:journalctl@ceph.mds.d.smithi118.stdout:Mar 24 00:24:11 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-d[95908]: 2: gsignal() 2024-03-24T00:24:11.467 INFO:journalctl@ceph.mds.d.smithi118.stdout:Mar 24 00:24:11 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-d[95908]: 3: abort() 2024-03-24T00:24:11.468 INFO:journalctl@ceph.mds.d.smithi118.stdout:Mar 24 00:24:11 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-d[95908]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x18f) [0x7f15ebe374fb] 2024-03-24T00:24:11.468 INFO:journalctl@ceph.mds.d.smithi118.stdout:Mar 24 00:24:11 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-d[95908]: 5: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7f15ebe37667] 2024-03-24T00:24:11.468 INFO:journalctl@ceph.mds.d.smithi118.stdout:Mar 24 00:24:11 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-d[95908]: 6: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x55c0c24bd223] 2024-03-24T00:24:11.468 INFO:journalctl@ceph.mds.d.smithi118.stdout:Mar 24 00:24:11 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-d[95908]: 7: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x55c0c23375d6] 2024-03-24T00:24:11.468 INFO:journalctl@ceph.mds.j.smithi118.stdout:Mar 24 00:24:11 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-j[96230]: 8: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x565096e74da1] 2024-03-24T00:24:11.468 INFO:journalctl@ceph.mds.j.smithi118.stdout:Mar 24 00:24:11 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-j[96230]: 9: (Server::handle_client_rename(boost::intrusive_ptr const&)+0x259d) [0x565096eb197d] 2024-03-24T00:24:11.468 INFO:journalctl@ceph.mds.j.smithi118.stdout:Mar 24 00:24:11 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-j[96230]: 10: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x565096ec7233] 2024-03-24T00:24:11.468 INFO:journalctl@ceph.mds.j.smithi118.stdout:Mar 24 00:24:11 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-j[96230]: 11: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x565096ec9197] 2024-03-24T00:24:11.468 INFO:journalctl@ceph.mds.j.smithi118.stdout:Mar 24 00:24:11 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-j[96230]: 12: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x565096dc811b] 2024-03-24T00:24:11.468 INFO:journalctl@ceph.mds.j.smithi118.stdout:Mar 24 00:24:11 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-j[96230]: 13: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x565096dc86bc] 2024-03-24T00:24:11.468 INFO:journalctl@ceph.mds.j.smithi118.stdout:Mar 24 00:24:11 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-j[96230]: 14: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x565096db21ff] 2024-03-24T00:24:11.468 INFO:journalctl@ceph.mds.j.smithi118.stdout:Mar 24 00:24:11 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-j[96230]: 15: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7f7483f70488] 2024-03-24T00:24:11.468 INFO:journalctl@ceph.mds.j.smithi118.stdout:Mar 24 00:24:11 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-j[96230]: 16: (DispatchQueue::entry()+0x51f) [0x7f7483f6d60f] 2024-03-24T00:24:11.469 INFO:journalctl@ceph.mds.j.smithi118.stdout:Mar 24 00:24:11 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-j[96230]: 17: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f7484040e81] 2024-03-24T00:24:11.469 INFO:journalctl@ceph.mds.j.smithi118.stdout:Mar 24 00:24:11 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-j[96230]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7f7482a5b1ca] 2024-03-24T00:24:11.469 INFO:journalctl@ceph.mds.j.smithi118.stdout:Mar 24 00:24:11 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-j[96230]: 19: clone() 2024-03-24T00:24:11.469 INFO:journalctl@ceph.mds.j.smithi118.stdout:Mar 24 00:24:11 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-j[96230]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-03-24T00:24:11.469 INFO:journalctl@ceph.mds.j.smithi118.stdout:Mar 24 00:24:11 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-j[96230]: 2024-03-24T00:24:11.469 INFO:journalctl@ceph.mds.j.smithi118.stdout:Mar 24 00:24:11 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-j[96230]: -1> 2024-03-24T00:24:11.026+0000 7f7478400700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: In function 'void Locker::set_xlocks_done(MutationImpl*, bool)' thread 7f7478400700 time 2024-03-24T00:24:11.024865+0000 2024-03-24T00:24:11.469 INFO:journalctl@ceph.mds.j.smithi118.stdout:Mar 24 00:24:11 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-j[96230]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: 626: FAILED ceph_assert(obj->is_auth()) 2024-03-24T00:24:11.469 INFO:journalctl@ceph.mds.j.smithi118.stdout:Mar 24 00:24:11 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-j[96230]: 2024-03-24T00:24:11.469 INFO:journalctl@ceph.mds.j.smithi118.stdout:Mar 24 00:24:11 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-j[96230]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-24T00:24:11.469 INFO:journalctl@ceph.mds.j.smithi118.stdout:Mar 24 00:24:11 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-j[96230]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x135) [0x7f7483cde4a1] 2024-03-24T00:24:11.469 INFO:journalctl@ceph.mds.j.smithi118.stdout:Mar 24 00:24:11 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-j[96230]: 2: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7f7483cde667] 2024-03-24T00:24:11.469 INFO:journalctl@ceph.mds.j.smithi118.stdout:Mar 24 00:24:11 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-j[96230]: 3: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x565096ffa223] 2024-03-24T00:24:11.469 INFO:journalctl@ceph.mds.j.smithi118.stdout:Mar 24 00:24:11 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-j[96230]: 4: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x565096e745d6] 2024-03-24T00:24:11.470 INFO:journalctl@ceph.mds.d.smithi118.stdout:Mar 24 00:24:11 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-d[95908]: 8: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x55c0c2337da1] 2024-03-24T00:24:11.470 INFO:journalctl@ceph.mds.d.smithi118.stdout:Mar 24 00:24:11 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-d[95908]: 9: (Server::handle_client_rename(boost::intrusive_ptr const&)+0x259d) [0x55c0c237497d] 2024-03-24T00:24:11.470 INFO:journalctl@ceph.mds.d.smithi118.stdout:Mar 24 00:24:11 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-d[95908]: 10: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x55c0c238a233] 2024-03-24T00:24:11.470 INFO:journalctl@ceph.mds.d.smithi118.stdout:Mar 24 00:24:11 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-d[95908]: 11: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x55c0c238c197] 2024-03-24T00:24:11.470 INFO:journalctl@ceph.mds.d.smithi118.stdout:Mar 24 00:24:11 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-d[95908]: 12: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x55c0c228b11b] 2024-03-24T00:24:11.470 INFO:journalctl@ceph.mds.d.smithi118.stdout:Mar 24 00:24:11 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-d[95908]: 13: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x55c0c228b6bc] 2024-03-24T00:24:11.470 INFO:journalctl@ceph.mds.d.smithi118.stdout:Mar 24 00:24:11 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-d[95908]: 14: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x55c0c22751ff] 2024-03-24T00:24:11.470 INFO:journalctl@ceph.mds.d.smithi118.stdout:Mar 24 00:24:11 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-d[95908]: 15: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7f15ec0c9488] 2024-03-24T00:24:11.470 INFO:journalctl@ceph.mds.d.smithi118.stdout:Mar 24 00:24:11 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-d[95908]: 16: (DispatchQueue::entry()+0x51f) [0x7f15ec0c660f] 2024-03-24T00:24:11.470 INFO:journalctl@ceph.mds.d.smithi118.stdout:Mar 24 00:24:11 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-d[95908]: 17: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f15ec199e81] 2024-03-24T00:24:11.471 INFO:journalctl@ceph.mds.d.smithi118.stdout:Mar 24 00:24:11 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-d[95908]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7f15eabb41ca] 2024-03-24T00:24:11.471 INFO:journalctl@ceph.mds.d.smithi118.stdout:Mar 24 00:24:11 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-d[95908]: 19: clone() 2024-03-24T00:24:11.471 INFO:journalctl@ceph.mds.d.smithi118.stdout:Mar 24 00:24:11 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-d[95908]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-03-24T00:24:11.471 INFO:journalctl@ceph.mds.d.smithi118.stdout:Mar 24 00:24:11 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-d[95908]: 2024-03-24T00:24:11.471 INFO:journalctl@ceph.mds.d.smithi118.stdout:Mar 24 00:24:11 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-d[95908]: -5> 2024-03-24T00:24:11.064+0000 7f15e0600700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: In function 'void Locker::set_xlocks_done(MutationImpl*, bool)' thread 7f15e0600700 time 2024-03-24T00:24:11.062894+0000 2024-03-24T00:24:11.471 INFO:journalctl@ceph.mds.d.smithi118.stdout:Mar 24 00:24:11 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-d[95908]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: 626: FAILED ceph_assert(obj->is_auth()) 2024-03-24T00:24:11.471 INFO:journalctl@ceph.mds.d.smithi118.stdout:Mar 24 00:24:11 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-d[95908]: 2024-03-24T00:24:11.471 INFO:journalctl@ceph.mds.d.smithi118.stdout:Mar 24 00:24:11 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-d[95908]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-24T00:24:11.471 INFO:journalctl@ceph.mds.d.smithi118.stdout:Mar 24 00:24:11 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-d[95908]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x135) [0x7f15ebe374a1] 2024-03-24T00:24:11.471 INFO:journalctl@ceph.mds.d.smithi118.stdout:Mar 24 00:24:11 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-d[95908]: 2: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7f15ebe37667] 2024-03-24T00:24:11.471 INFO:journalctl@ceph.mds.d.smithi118.stdout:Mar 24 00:24:11 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-d[95908]: 3: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x55c0c24bd223] 2024-03-24T00:24:11.471 INFO:journalctl@ceph.mds.d.smithi118.stdout:Mar 24 00:24:11 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-d[95908]: 4: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x55c0c23375d6] 2024-03-24T00:24:11.472 INFO:journalctl@ceph.mds.j.smithi118.stdout:Mar 24 00:24:11 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-j[96230]: 5: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x565096e74da1] 2024-03-24T00:24:11.472 INFO:journalctl@ceph.mds.j.smithi118.stdout:Mar 24 00:24:11 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-j[96230]: 6: (Server::handle_client_rename(boost::intrusive_ptr const&)+0x259d) [0x565096eb197d] 2024-03-24T00:24:11.472 INFO:journalctl@ceph.mds.j.smithi118.stdout:Mar 24 00:24:11 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-j[96230]: 7: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x565096ec7233] 2024-03-24T00:24:11.472 INFO:journalctl@ceph.mds.j.smithi118.stdout:Mar 24 00:24:11 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-j[96230]: 8: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x565096ec9197] 2024-03-24T00:24:11.472 INFO:journalctl@ceph.mds.j.smithi118.stdout:Mar 24 00:24:11 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-j[96230]: 9: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x565096dc811b] 2024-03-24T00:24:11.472 INFO:journalctl@ceph.mds.j.smithi118.stdout:Mar 24 00:24:11 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-j[96230]: 10: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x565096dc86bc] 2024-03-24T00:24:11.472 INFO:journalctl@ceph.mds.j.smithi118.stdout:Mar 24 00:24:11 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-j[96230]: 11: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x565096db21ff] 2024-03-24T00:24:11.472 INFO:journalctl@ceph.mds.j.smithi118.stdout:Mar 24 00:24:11 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-j[96230]: 12: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7f7483f70488] 2024-03-24T00:24:11.472 INFO:journalctl@ceph.mds.j.smithi118.stdout:Mar 24 00:24:11 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-j[96230]: 13: (DispatchQueue::entry()+0x51f) [0x7f7483f6d60f] 2024-03-24T00:24:11.472 INFO:journalctl@ceph.mds.j.smithi118.stdout:Mar 24 00:24:11 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-j[96230]: 14: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f7484040e81] 2024-03-24T00:24:11.472 INFO:journalctl@ceph.mds.j.smithi118.stdout:Mar 24 00:24:11 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-j[96230]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7f7482a5b1ca] 2024-03-24T00:24:11.472 INFO:journalctl@ceph.mds.j.smithi118.stdout:Mar 24 00:24:11 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-j[96230]: 16: clone() 2024-03-24T00:24:11.472 INFO:journalctl@ceph.mds.j.smithi118.stdout:Mar 24 00:24:11 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-j[96230]: 2024-03-24T00:24:11.472 INFO:journalctl@ceph.mds.j.smithi118.stdout:Mar 24 00:24:11 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-j[96230]: 0> 2024-03-24T00:24:11.029+0000 7f7478400700 -1 *** Caught signal (Aborted) ** 2024-03-24T00:24:11.472 INFO:journalctl@ceph.mds.j.smithi118.stdout:Mar 24 00:24:11 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-j[96230]: in thread 7f7478400700 thread_name:ms_dispatch 2024-03-24T00:24:11.472 INFO:journalctl@ceph.mds.j.smithi118.stdout:Mar 24 00:24:11 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-j[96230]: 2024-03-24T00:24:11.472 INFO:journalctl@ceph.mds.j.smithi118.stdout:Mar 24 00:24:11 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-j[96230]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-24T00:24:11.473 INFO:journalctl@ceph.mds.j.smithi118.stdout:Mar 24 00:24:11 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-j[96230]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f7482a65d20] 2024-03-24T00:24:11.473 INFO:journalctl@ceph.mds.j.smithi118.stdout:Mar 24 00:24:11 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-j[96230]: 2: gsignal() 2024-03-24T00:24:11.473 INFO:journalctl@ceph.mds.j.smithi118.stdout:Mar 24 00:24:11 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-j[96230]: 3: abort() 2024-03-24T00:24:11.473 INFO:journalctl@ceph.mds.j.smithi118.stdout:Mar 24 00:24:11 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-j[96230]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x18f) [0x7f7483cde4fb] 2024-03-24T00:24:11.473 INFO:journalctl@ceph.mds.j.smithi118.stdout:Mar 24 00:24:11 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-j[96230]: 5: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7f7483cde667] 2024-03-24T00:24:11.473 INFO:journalctl@ceph.mds.j.smithi118.stdout:Mar 24 00:24:11 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-j[96230]: 6: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x565096ffa223] 2024-03-24T00:24:11.473 INFO:journalctl@ceph.mds.j.smithi118.stdout:Mar 24 00:24:11 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-j[96230]: 7: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x565096e745d6] 2024-03-24T00:24:11.473 INFO:journalctl@ceph.mds.j.smithi118.stdout:Mar 24 00:24:11 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-j[96230]: 8: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x565096e74da1] 2024-03-24T00:24:11.473 INFO:journalctl@ceph.mds.j.smithi118.stdout:Mar 24 00:24:11 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-j[96230]: 9: (Server::handle_client_rename(boost::intrusive_ptr const&)+0x259d) [0x565096eb197d] 2024-03-24T00:24:11.473 INFO:journalctl@ceph.mds.j.smithi118.stdout:Mar 24 00:24:11 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-j[96230]: 10: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x565096ec7233] 2024-03-24T00:24:11.473 INFO:journalctl@ceph.mds.d.smithi118.stdout:Mar 24 00:24:11 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-d[95908]: 5: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x55c0c2337da1] 2024-03-24T00:24:11.473 INFO:journalctl@ceph.mds.d.smithi118.stdout:Mar 24 00:24:11 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-d[95908]: 6: (Server::handle_client_rename(boost::intrusive_ptr const&)+0x259d) [0x55c0c237497d] 2024-03-24T00:24:11.474 INFO:journalctl@ceph.mds.d.smithi118.stdout:Mar 24 00:24:11 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-d[95908]: 7: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x55c0c238a233] 2024-03-24T00:24:11.474 INFO:journalctl@ceph.mds.d.smithi118.stdout:Mar 24 00:24:11 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-d[95908]: 8: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x55c0c238c197] 2024-03-24T00:24:11.474 INFO:journalctl@ceph.mds.d.smithi118.stdout:Mar 24 00:24:11 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-d[95908]: 9: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x55c0c228b11b] 2024-03-24T00:24:11.474 INFO:journalctl@ceph.mds.d.smithi118.stdout:Mar 24 00:24:11 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-d[95908]: 10: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x55c0c228b6bc] 2024-03-24T00:24:11.474 INFO:journalctl@ceph.mds.d.smithi118.stdout:Mar 24 00:24:11 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-d[95908]: 11: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x55c0c22751ff] 2024-03-24T00:24:11.474 INFO:journalctl@ceph.mds.d.smithi118.stdout:Mar 24 00:24:11 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-d[95908]: 12: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7f15ec0c9488] 2024-03-24T00:24:11.474 INFO:journalctl@ceph.mds.d.smithi118.stdout:Mar 24 00:24:11 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-d[95908]: 13: (DispatchQueue::entry()+0x51f) [0x7f15ec0c660f] 2024-03-24T00:24:11.474 INFO:journalctl@ceph.mds.d.smithi118.stdout:Mar 24 00:24:11 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-d[95908]: 14: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f15ec199e81] 2024-03-24T00:24:11.474 INFO:journalctl@ceph.mds.d.smithi118.stdout:Mar 24 00:24:11 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-d[95908]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7f15eabb41ca] 2024-03-24T00:24:11.474 INFO:journalctl@ceph.mds.d.smithi118.stdout:Mar 24 00:24:11 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-d[95908]: 16: clone() 2024-03-24T00:24:11.474 INFO:journalctl@ceph.mds.d.smithi118.stdout:Mar 24 00:24:11 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-d[95908]: 2024-03-24T00:24:11.474 INFO:journalctl@ceph.mds.d.smithi118.stdout:Mar 24 00:24:11 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-d[95908]: -4> 2024-03-24T00:24:11.067+0000 7f15e0600700 -1 *** Caught signal (Aborted) ** 2024-03-24T00:24:11.474 INFO:journalctl@ceph.mds.d.smithi118.stdout:Mar 24 00:24:11 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-d[95908]: in thread 7f15e0600700 thread_name:ms_dispatch 2024-03-24T00:24:11.474 INFO:journalctl@ceph.mds.d.smithi118.stdout:Mar 24 00:24:11 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-d[95908]: 2024-03-24T00:24:11.474 INFO:journalctl@ceph.mds.d.smithi118.stdout:Mar 24 00:24:11 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-d[95908]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-24T00:24:11.475 INFO:journalctl@ceph.mds.d.smithi118.stdout:Mar 24 00:24:11 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-d[95908]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f15eabbed20] 2024-03-24T00:24:11.475 INFO:journalctl@ceph.mds.d.smithi118.stdout:Mar 24 00:24:11 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-d[95908]: 2: gsignal() 2024-03-24T00:24:11.475 INFO:journalctl@ceph.mds.d.smithi118.stdout:Mar 24 00:24:11 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-d[95908]: 3: abort() 2024-03-24T00:24:11.475 INFO:journalctl@ceph.mds.d.smithi118.stdout:Mar 24 00:24:11 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-d[95908]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x18f) [0x7f15ebe374fb] 2024-03-24T00:24:11.475 INFO:journalctl@ceph.mds.d.smithi118.stdout:Mar 24 00:24:11 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-d[95908]: 5: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7f15ebe37667] 2024-03-24T00:24:11.475 INFO:journalctl@ceph.mds.d.smithi118.stdout:Mar 24 00:24:11 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-d[95908]: 6: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x55c0c24bd223] 2024-03-24T00:24:11.475 INFO:journalctl@ceph.mds.d.smithi118.stdout:Mar 24 00:24:11 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-d[95908]: 7: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x55c0c23375d6] 2024-03-24T00:24:11.475 INFO:journalctl@ceph.mds.d.smithi118.stdout:Mar 24 00:24:11 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-d[95908]: 8: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x55c0c2337da1] 2024-03-24T00:24:11.475 INFO:journalctl@ceph.mds.d.smithi118.stdout:Mar 24 00:24:11 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-d[95908]: 9: (Server::handle_client_rename(boost::intrusive_ptr const&)+0x259d) [0x55c0c237497d] 2024-03-24T00:24:11.475 INFO:journalctl@ceph.mds.d.smithi118.stdout:Mar 24 00:24:11 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-d[95908]: 10: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x55c0c238a233] 2024-03-24T00:24:11.476 INFO:journalctl@ceph.mds.j.smithi118.stdout:Mar 24 00:24:11 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-j[96230]: 11: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x565096ec9197] 2024-03-24T00:24:11.476 INFO:journalctl@ceph.mds.j.smithi118.stdout:Mar 24 00:24:11 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-j[96230]: 12: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x565096dc811b] 2024-03-24T00:24:11.476 INFO:journalctl@ceph.mds.j.smithi118.stdout:Mar 24 00:24:11 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-j[96230]: 13: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x565096dc86bc] 2024-03-24T00:24:11.476 INFO:journalctl@ceph.mds.j.smithi118.stdout:Mar 24 00:24:11 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-j[96230]: 14: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x565096db21ff] 2024-03-24T00:24:11.476 INFO:journalctl@ceph.mds.j.smithi118.stdout:Mar 24 00:24:11 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-j[96230]: 15: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7f7483f70488] 2024-03-24T00:24:11.476 INFO:journalctl@ceph.mds.j.smithi118.stdout:Mar 24 00:24:11 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-j[96230]: 16: (DispatchQueue::entry()+0x51f) [0x7f7483f6d60f] 2024-03-24T00:24:11.476 INFO:journalctl@ceph.mds.j.smithi118.stdout:Mar 24 00:24:11 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-j[96230]: 17: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f7484040e81] 2024-03-24T00:24:11.476 INFO:journalctl@ceph.mds.j.smithi118.stdout:Mar 24 00:24:11 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-j[96230]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7f7482a5b1ca] 2024-03-24T00:24:11.476 INFO:journalctl@ceph.mds.j.smithi118.stdout:Mar 24 00:24:11 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-j[96230]: 19: clone() 2024-03-24T00:24:11.476 INFO:journalctl@ceph.mds.j.smithi118.stdout:Mar 24 00:24:11 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-j[96230]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-03-24T00:24:11.476 INFO:journalctl@ceph.mds.j.smithi118.stdout:Mar 24 00:24:11 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-j[96230]: 2024-03-24T00:24:11.476 INFO:journalctl@ceph.mds.j.smithi118.stdout:Mar 24 00:24:11 smithi118 podman[96285]: 2024-03-24 00:24:11.169419817 +0000 UTC m=+0.046626782 container died a9ed700118ad9459d1fd074773f67f75ea0d1f85f557b884101969b4e9f2004c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-j, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, ceph=True, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.build-date=20240319, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-24T00:24:11.477 INFO:journalctl@ceph.mds.d.smithi118.stdout:Mar 24 00:24:11 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-d[95908]: 11: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x55c0c238c197] 2024-03-24T00:24:11.477 INFO:journalctl@ceph.mds.d.smithi118.stdout:Mar 24 00:24:11 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-d[95908]: 12: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x55c0c228b11b] 2024-03-24T00:24:11.477 INFO:journalctl@ceph.mds.d.smithi118.stdout:Mar 24 00:24:11 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-d[95908]: 13: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x55c0c228b6bc] 2024-03-24T00:24:11.477 INFO:journalctl@ceph.mds.d.smithi118.stdout:Mar 24 00:24:11 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-d[95908]: 14: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x55c0c22751ff] 2024-03-24T00:24:11.477 INFO:journalctl@ceph.mds.d.smithi118.stdout:Mar 24 00:24:11 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-d[95908]: 15: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7f15ec0c9488] 2024-03-24T00:24:11.477 INFO:journalctl@ceph.mds.d.smithi118.stdout:Mar 24 00:24:11 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-d[95908]: 16: (DispatchQueue::entry()+0x51f) [0x7f15ec0c660f] 2024-03-24T00:24:11.477 INFO:journalctl@ceph.mds.d.smithi118.stdout:Mar 24 00:24:11 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-d[95908]: 17: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f15ec199e81] 2024-03-24T00:24:11.477 INFO:journalctl@ceph.mds.d.smithi118.stdout:Mar 24 00:24:11 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-d[95908]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7f15eabb41ca] 2024-03-24T00:24:11.477 INFO:journalctl@ceph.mds.d.smithi118.stdout:Mar 24 00:24:11 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-d[95908]: 19: clone() 2024-03-24T00:24:11.477 INFO:journalctl@ceph.mds.d.smithi118.stdout:Mar 24 00:24:11 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-d[95908]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-03-24T00:24:11.477 INFO:journalctl@ceph.mds.d.smithi118.stdout:Mar 24 00:24:11 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-d[95908]: 2024-03-24T00:24:11.477 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:24:10 smithi193 ceph-mon[36878]: mds.? [v2:172.21.15.118:6836/1531167856,v1:172.21.15.118:6837/1531167856] up:rejoin 2024-03-24T00:24:11.477 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:24:10 smithi193 ceph-mon[36878]: mds.? [v2:172.21.15.118:6834/3766636479,v1:172.21.15.118:6835/3766636479] up:rejoin 2024-03-24T00:24:11.478 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:24:10 smithi193 ceph-mon[36878]: fsmap cephfs:5/5 3 up:standby-replay 2 up:rejoin 3 up:active 2024-03-24T00:24:11.478 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:24:10 smithi193 ceph-mon[36878]: daemon mds.j is now active in filesystem cephfs as rank 3 2024-03-24T00:24:11.478 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:24:10 smithi193 ceph-mon[36878]: daemon mds.d is now active in filesystem cephfs as rank 1 2024-03-24T00:24:11.478 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:24:10 smithi189 ceph-mon[36187]: mds.? [v2:172.21.15.118:6836/1531167856,v1:172.21.15.118:6837/1531167856] up:rejoin 2024-03-24T00:24:11.478 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:24:10 smithi189 ceph-mon[36187]: mds.? [v2:172.21.15.118:6834/3766636479,v1:172.21.15.118:6835/3766636479] up:rejoin 2024-03-24T00:24:11.478 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:24:10 smithi189 ceph-mon[36187]: fsmap cephfs:5/5 3 up:standby-replay 2 up:rejoin 3 up:active 2024-03-24T00:24:11.478 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:24:10 smithi189 ceph-mon[36187]: daemon mds.j is now active in filesystem cephfs as rank 3 2024-03-24T00:24:11.478 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:24:10 smithi189 ceph-mon[36187]: daemon mds.d is now active in filesystem cephfs as rank 1 2024-03-24T00:24:11.946 INFO:journalctl@ceph.mds.d.smithi118.stdout:Mar 24 00:24:11 smithi118 podman[96294]: 2024-03-24 00:24:11.58034731 +0000 UTC m=+0.403233410 container died 3cd3f9060cca524fa1b913dbc4556047ba0927ead5677afa6b93e2890dddba84 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-d, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240319, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, io.buildah.version=1.33.5, GIT_CLEAN=True, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, ceph=True) 2024-03-24T00:24:12.390 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:24:12 smithi118 ceph-mon[31535]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-03-24T00:24:12.390 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:24:12 smithi118 ceph-mon[31535]: Cluster is now healthy 2024-03-24T00:24:12.390 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:24:12 smithi118 ceph-mon[31535]: mds.? [v2:172.21.15.118:6834/3766636479,v1:172.21.15.118:6835/3766636479] up:active 2024-03-24T00:24:12.390 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:24:12 smithi118 ceph-mon[31535]: mds.? [v2:172.21.15.118:6836/1531167856,v1:172.21.15.118:6837/1531167856] up:active 2024-03-24T00:24:12.391 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:24:12 smithi118 ceph-mon[31535]: fsmap cephfs:5 3 up:standby-replay 5 up:active 2024-03-24T00:24:12.391 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:24:12 smithi118 ceph-mon[31535]: pgmap v681: 97 pgs: 97 active+clean; 146 MiB data, 859 MiB used, 1.0 TiB / 1.0 TiB avail; 33 KiB/s rd, 12 KiB/s wr, 17 op/s 2024-03-24T00:24:12.453 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:24:12 smithi189 ceph-mon[36187]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-03-24T00:24:12.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:24:12 smithi189 ceph-mon[36187]: Cluster is now healthy 2024-03-24T00:24:12.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:24:12 smithi189 ceph-mon[36187]: mds.? [v2:172.21.15.118:6834/3766636479,v1:172.21.15.118:6835/3766636479] up:active 2024-03-24T00:24:12.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:24:12 smithi189 ceph-mon[36187]: mds.? [v2:172.21.15.118:6836/1531167856,v1:172.21.15.118:6837/1531167856] up:active 2024-03-24T00:24:12.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:24:12 smithi189 ceph-mon[36187]: fsmap cephfs:5 3 up:standby-replay 5 up:active 2024-03-24T00:24:12.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:24:12 smithi189 ceph-mon[36187]: pgmap v681: 97 pgs: 97 active+clean; 146 MiB data, 859 MiB used, 1.0 TiB / 1.0 TiB avail; 33 KiB/s rd, 12 KiB/s wr, 17 op/s 2024-03-24T00:24:12.460 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:24:12 smithi193 ceph-mon[36878]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-03-24T00:24:12.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:24:12 smithi193 ceph-mon[36878]: Cluster is now healthy 2024-03-24T00:24:12.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:24:12 smithi193 ceph-mon[36878]: mds.? [v2:172.21.15.118:6834/3766636479,v1:172.21.15.118:6835/3766636479] up:active 2024-03-24T00:24:12.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:24:12 smithi193 ceph-mon[36878]: mds.? [v2:172.21.15.118:6836/1531167856,v1:172.21.15.118:6837/1531167856] up:active 2024-03-24T00:24:12.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:24:12 smithi193 ceph-mon[36878]: fsmap cephfs:5 3 up:standby-replay 5 up:active 2024-03-24T00:24:12.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:24:12 smithi193 ceph-mon[36878]: pgmap v681: 97 pgs: 97 active+clean; 146 MiB data, 859 MiB used, 1.0 TiB / 1.0 TiB avail; 33 KiB/s rd, 12 KiB/s wr, 17 op/s 2024-03-24T00:24:12.696 INFO:journalctl@ceph.mds.j.smithi118.stdout:Mar 24 00:24:12 smithi118 podman[96285]: 2024-03-24 00:24:12.587083151 +0000 UTC m=+1.464290148 container remove a9ed700118ad9459d1fd074773f67f75ea0d1f85f557b884101969b4e9f2004c (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-j, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.build-date=20240319, GIT_BRANCH=HEAD, GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2) 2024-03-24T00:24:12.696 INFO:journalctl@ceph.mds.j.smithi118.stdout:Mar 24 00:24:12 smithi118 systemd[1]: ceph-d79a5420-e971-11ee-95cd-87774f69a715@mds.j.service: Main process exited, code=exited, status=134/n/a 2024-03-24T00:24:12.977 INFO:journalctl@ceph.mds.d.smithi118.stdout:Mar 24 00:24:12 smithi118 podman[96294]: 2024-03-24 00:24:12.696150333 +0000 UTC m=+1.519036393 container remove 3cd3f9060cca524fa1b913dbc4556047ba0927ead5677afa6b93e2890dddba84 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-d, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, io.buildah.version=1.33.5, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.build-date=20240319, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, GIT_CLEAN=True) 2024-03-24T00:24:12.977 INFO:journalctl@ceph.mds.d.smithi118.stdout:Mar 24 00:24:12 smithi118 systemd[1]: ceph-d79a5420-e971-11ee-95cd-87774f69a715@mds.d.service: Main process exited, code=exited, status=134/n/a 2024-03-24T00:24:13.446 INFO:journalctl@ceph.mds.d.smithi118.stdout:Mar 24 00:24:13 smithi118 systemd[1]: ceph-d79a5420-e971-11ee-95cd-87774f69a715@mds.d.service: Failed with result 'exit-code'. 2024-03-24T00:24:13.446 INFO:journalctl@ceph.mds.d.smithi118.stdout:Mar 24 00:24:13 smithi118 systemd[1]: ceph-d79a5420-e971-11ee-95cd-87774f69a715@mds.d.service: Consumed 1.663s CPU time. 2024-03-24T00:24:13.447 INFO:journalctl@ceph.mds.j.smithi118.stdout:Mar 24 00:24:12 smithi118 systemd[1]: ceph-d79a5420-e971-11ee-95cd-87774f69a715@mds.j.service: Failed with result 'exit-code'. 2024-03-24T00:24:13.447 INFO:journalctl@ceph.mds.j.smithi118.stdout:Mar 24 00:24:12 smithi118 systemd[1]: ceph-d79a5420-e971-11ee-95cd-87774f69a715@mds.j.service: Consumed 1.401s CPU time. 2024-03-24T00:24:14.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:24:13 smithi189 ceph-mon[36187]: pgmap v682: 97 pgs: 97 active+clean; 146 MiB data, 859 MiB used, 1.0 TiB / 1.0 TiB avail; 141 KiB/s rd, 2.1 KiB/s wr, 32 op/s 2024-03-24T00:24:14.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:24:13 smithi193 ceph-mon[36878]: pgmap v682: 97 pgs: 97 active+clean; 146 MiB data, 859 MiB used, 1.0 TiB / 1.0 TiB avail; 141 KiB/s rd, 2.1 KiB/s wr, 32 op/s 2024-03-24T00:24:14.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:24:13 smithi118 ceph-mon[31535]: pgmap v682: 97 pgs: 97 active+clean; 146 MiB data, 859 MiB used, 1.0 TiB / 1.0 TiB avail; 141 KiB/s rd, 2.1 KiB/s wr, 32 op/s 2024-03-24T00:24:16.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:24:15 smithi189 ceph-mon[36187]: pgmap v683: 97 pgs: 97 active+clean; 146 MiB data, 863 MiB used, 1.0 TiB / 1.0 TiB avail; 165 KiB/s rd, 4.2 KiB/s wr, 42 op/s 2024-03-24T00:24:16.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:24:15 smithi193 ceph-mon[36878]: pgmap v683: 97 pgs: 97 active+clean; 146 MiB data, 863 MiB used, 1.0 TiB / 1.0 TiB avail; 165 KiB/s rd, 4.2 KiB/s wr, 42 op/s 2024-03-24T00:24:16.249 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:24:15 smithi118 ceph-mon[31535]: pgmap v683: 97 pgs: 97 active+clean; 146 MiB data, 863 MiB used, 1.0 TiB / 1.0 TiB avail; 165 KiB/s rd, 4.2 KiB/s wr, 42 op/s 2024-03-24T00:24:18.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:24:17 smithi118 ceph-mon[31535]: pgmap v684: 97 pgs: 97 active+clean; 146 MiB data, 863 MiB used, 1.0 TiB / 1.0 TiB avail; 147 KiB/s rd, 3.7 KiB/s wr, 38 op/s 2024-03-24T00:24:18.203 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:24:17 smithi189 ceph-mon[36187]: pgmap v684: 97 pgs: 97 active+clean; 146 MiB data, 863 MiB used, 1.0 TiB / 1.0 TiB avail; 147 KiB/s rd, 3.7 KiB/s wr, 38 op/s 2024-03-24T00:24:18.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:24:17 smithi193 ceph-mon[36878]: pgmap v684: 97 pgs: 97 active+clean; 146 MiB data, 863 MiB used, 1.0 TiB / 1.0 TiB avail; 147 KiB/s rd, 3.7 KiB/s wr, 38 op/s 2024-03-24T00:24:21.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:24:20 smithi118 ceph-mon[31535]: pgmap v685: 97 pgs: 97 active+clean; 146 MiB data, 863 MiB used, 1.0 TiB / 1.0 TiB avail; 143 KiB/s rd, 8.7 KiB/s wr, 36 op/s 2024-03-24T00:24:21.203 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:24:20 smithi189 ceph-mon[36187]: pgmap v685: 97 pgs: 97 active+clean; 146 MiB data, 863 MiB used, 1.0 TiB / 1.0 TiB avail; 143 KiB/s rd, 8.7 KiB/s wr, 36 op/s 2024-03-24T00:24:21.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:24:20 smithi193 ceph-mon[36878]: pgmap v685: 97 pgs: 97 active+clean; 146 MiB data, 863 MiB used, 1.0 TiB / 1.0 TiB avail; 143 KiB/s rd, 8.7 KiB/s wr, 36 op/s 2024-03-24T00:24:23.167 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:24:23.167+0000 7f6810a00640 1 -- 172.21.15.118:0/352758445 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f680c101fc0 msgr2=0x7f680c108850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:24:23.168 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:24:23.167+0000 7f6810a00640 1 --2- 172.21.15.118:0/352758445 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f680c101fc0 0x7f680c108850 secure :-1 s=READY pgs=456 cs=0 l=1 rev1=1 crypto rx=0x7f6804009ca0 tx=0x7f680402c0f0 comp rx=0 tx=0).stop 2024-03-24T00:24:23.168 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:24:23.167+0000 7f6810a00640 1 -- 172.21.15.118:0/352758445 shutdown_connections 2024-03-24T00:24:23.168 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:24:23.167+0000 7f6810a00640 1 --2- 172.21.15.118:0/352758445 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f680c101fc0 0x7f680c108850 unknown :-1 s=CLOSED pgs=456 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:24:23.168 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:24:23.167+0000 7f6810a00640 1 --2- 172.21.15.118:0/352758445 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f680c101600 0x7f680c101a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:24:23.168 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:24:23.167+0000 7f6810a00640 1 --2- 172.21.15.118:0/352758445 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f680c100410 0x7f680c100830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:24:23.168 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:24:23.167+0000 7f6810a00640 1 -- 172.21.15.118:0/352758445 >> 172.21.15.118:0/352758445 conn(0x7f680c0fb9a0 msgr2=0x7f680c0fddc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:24:23.168 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:24:23.167+0000 7f6810a00640 1 -- 172.21.15.118:0/352758445 shutdown_connections 2024-03-24T00:24:23.168 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:24:23.167+0000 7f6810a00640 1 -- 172.21.15.118:0/352758445 wait complete. 2024-03-24T00:24:23.171 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:24:23.168+0000 7f6810a00640 1 Processor -- start 2024-03-24T00:24:23.171 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:24:23.168+0000 7f6810a00640 1 -- start start 2024-03-24T00:24:23.171 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:24:23.169+0000 7f6810a00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f680c100410 0x7f680c1999e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:24:23.171 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:24:23.169+0000 7f6810a00640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f680c101600 0x7f680c199f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:24:23.171 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:24:23.169+0000 7f6810a00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f680c101fc0 0x7f680c1a0f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:24:23.171 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:24:23.169+0000 7f6810a00640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f680c10c070 con 0x7f680c101600 2024-03-24T00:24:23.171 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:24:23.169+0000 7f6810a00640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f680c10bef0 con 0x7f680c101fc0 2024-03-24T00:24:23.171 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:24:23.169+0000 7f6810a00640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f680c10c1f0 con 0x7f680c100410 2024-03-24T00:24:23.171 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:24:23.169+0000 7f680b400640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f680c100410 0x7f680c1999e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:24:23.172 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:24:23.169+0000 7f680b400640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f680c100410 0x7f680c1999e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.118:46796/0 (socket says 172.21.15.118:46796) 2024-03-24T00:24:23.172 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:24:23.169+0000 7f680b400640 1 -- 172.21.15.118:0/4203939711 learned_addr learned my addr 172.21.15.118:0/4203939711 (peer_addr_for_me v2:172.21.15.118:0/0) 2024-03-24T00:24:23.172 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:24:23.169+0000 7f680aa00640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f680c101600 0x7f680c199f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:24:23.172 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:24:23.169+0000 7f680be00640 1 --2- 172.21.15.118:0/4203939711 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f680c101fc0 0x7f680c1a0f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:24:23.172 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:24:23.169+0000 7f680b400640 1 -- 172.21.15.118:0/4203939711 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f680c101fc0 msgr2=0x7f680c1a0f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:24:23.172 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:24:23.170+0000 7f680b400640 1 --2- 172.21.15.118:0/4203939711 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f680c101fc0 0x7f680c1a0f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:24:23.172 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:24:23.170+0000 7f680b400640 1 -- 172.21.15.118:0/4203939711 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f680c101600 msgr2=0x7f680c199f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:24:23.172 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:24:23.170+0000 7f680b400640 1 --2- 172.21.15.118:0/4203939711 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f680c101600 0x7f680c199f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:24:23.172 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:24:23.170+0000 7f680b400640 1 -- 172.21.15.118:0/4203939711 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f680c1a1600 con 0x7f680c100410 2024-03-24T00:24:23.172 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:24:23.170+0000 7f680be00640 1 --2- 172.21.15.118:0/4203939711 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f680c101fc0 0x7f680c1a0f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T00:24:23.172 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:24:23.170+0000 7f680b400640 1 --2- 172.21.15.118:0/4203939711 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f680c100410 0x7f680c1999e0 secure :-1 s=READY pgs=457 cs=0 l=1 rev1=1 crypto rx=0x7f67fc00e9c0 tx=0x7f67fc00ee80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:24:23.172 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:24:23.170+0000 7f6803e00640 1 -- 172.21.15.118:0/4203939711 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f67fc00cd80 con 0x7f680c100410 2024-03-24T00:24:23.173 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:24:23.170+0000 7f6803e00640 1 -- 172.21.15.118:0/4203939711 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f67fc0035f0 con 0x7f680c100410 2024-03-24T00:24:23.173 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:24:23.170+0000 7f6810a00640 1 -- 172.21.15.118:0/4203939711 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f680c1a18f0 con 0x7f680c100410 2024-03-24T00:24:23.173 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:24:23.170+0000 7f6803e00640 1 -- 172.21.15.118:0/4203939711 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f67fc010690 con 0x7f680c100410 2024-03-24T00:24:23.173 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:24:23.170+0000 7f6810a00640 1 -- 172.21.15.118:0/4203939711 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f680c1a1e40 con 0x7f680c100410 2024-03-24T00:24:23.175 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:24:23.173+0000 7f6803e00640 1 -- 172.21.15.118:0/4203939711 <== mon.1 v2:172.21.15.193:3300/0 4 ==== mgrmap(e 20) v1 ==== 137291+0+0 (secure 0 0 0) 0x7f67fc0037a0 con 0x7f680c100410 2024-03-24T00:24:23.175 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:24:23.173+0000 7f6814d64740 1 Processor -- start 2024-03-24T00:24:23.175 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:24:23.174+0000 7f6814d64740 1 -- start start 2024-03-24T00:24:23.175 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:24:23.174+0000 7f6814d64740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55be4ba993f0 0x55be4ba99810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:24:23.176 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:24:23.174+0000 7f6814d64740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55be4ba99de0 0x55be4ba9e8e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:24:23.176 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:24:23.174+0000 7f6814d64740 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55be4ba9ef50 0x55be4baa13e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:24:23.176 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:24:23.174+0000 7f6814d64740 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x55be4ba5c3b0 con 0x55be4ba9ef50 2024-03-24T00:24:23.176 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:24:23.174+0000 7f6814d64740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x55be4b9fe830 con 0x55be4ba99de0 2024-03-24T00:24:23.176 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:24:23.174+0000 7f6814d64740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x55be4b9a75d0 con 0x55be4ba993f0 2024-03-24T00:24:23.176 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:24:23.174+0000 7f680b400640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55be4ba99de0 0x55be4ba9e8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:24:23.176 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:24:23.174+0000 7f680be00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55be4ba993f0 0x55be4ba99810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:24:23.176 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:24:23.174+0000 7f680be00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55be4ba993f0 0x55be4ba99810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.118:46800/0 (socket says 172.21.15.118:46800) 2024-03-24T00:24:23.176 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:24:23.174+0000 7f680be00640 1 -- 172.21.15.118:0/2437262792 learned_addr learned my addr 172.21.15.118:0/2437262792 (peer_addr_for_me v2:172.21.15.118:0/0) 2024-03-24T00:24:23.176 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:24:23.174+0000 7f680aa00640 1 --2- 172.21.15.118:0/2437262792 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55be4ba9ef50 0x55be4baa13e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:24:23.176 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:24:23.175+0000 7f680be00640 1 -- 172.21.15.118:0/2437262792 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55be4ba99de0 msgr2=0x55be4ba9e8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:24:23.177 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:24:23.175+0000 7f680be00640 1 --2- 172.21.15.118:0/2437262792 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55be4ba99de0 0x55be4ba9e8e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:24:23.177 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:24:23.175+0000 7f680be00640 1 -- 172.21.15.118:0/2437262792 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55be4ba9ef50 msgr2=0x55be4baa13e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:24:23.177 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:24:23.175+0000 7f680be00640 1 --2- 172.21.15.118:0/2437262792 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55be4ba9ef50 0x55be4baa13e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:24:23.177 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:24:23.175+0000 7f680be00640 1 -- 172.21.15.118:0/2437262792 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55be4b9bf2c0 con 0x55be4ba993f0 2024-03-24T00:24:23.177 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:24:23.175+0000 7f680aa00640 1 --2- 172.21.15.118:0/2437262792 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55be4ba9ef50 0x55be4baa13e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T00:24:23.177 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:24:23.175+0000 7f680b400640 1 --2- 172.21.15.118:0/2437262792 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55be4ba99de0 0x55be4ba9e8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T00:24:23.177 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:24:23.175+0000 7f6803e00640 1 --2- 172.21.15.118:0/4203939711 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7f67f00a27f0 0x7f67f00a4cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:24:23.177 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:24:23.175+0000 7f680be00640 1 --2- 172.21.15.118:0/2437262792 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55be4ba993f0 0x55be4ba99810 secure :-1 s=READY pgs=458 cs=0 l=1 rev1=1 crypto rx=0x7f680402c700 tx=0x7f68040323d0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=43cdfa5800a6b196 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:24:23.177 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:24:23.175+0000 7f6803e00640 1 -- 172.21.15.118:0/4203939711 <== mon.1 v2:172.21.15.193:3300/0 5 ==== osd_map(118..118 src has 1..118) v4 ==== 10685+0+0 (secure 0 0 0) 0x7f67fc0cf800 con 0x7f680c100410 2024-03-24T00:24:23.177 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:24:23.175+0000 7f680b400640 1 --2- 172.21.15.118:0/4203939711 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7f67f00a27f0 0x7f67f00a4cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:24:23.177 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:24:23.175+0000 7f67f7e00640 1 -- 172.21.15.118:0/2437262792 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6804043070 con 0x55be4ba993f0 2024-03-24T00:24:23.178 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:24:23.175+0000 7f67f7e00640 1 -- 172.21.15.118:0/2437262792 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f680402cca0 con 0x55be4ba993f0 2024-03-24T00:24:23.178 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:24:23.175+0000 7f67f7e00640 1 -- 172.21.15.118:0/2437262792 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f6804039040 con 0x55be4ba993f0 2024-03-24T00:24:23.182 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:24:23.176+0000 7f6814d64740 1 -- 172.21.15.118:0/2437262792 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55be4ba993f0 msgr2=0x55be4ba99810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:24:23.182 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:24:23.176+0000 7f680b400640 1 --2- 172.21.15.118:0/4203939711 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7f67f00a27f0 0x7f67f00a4cb0 secure :-1 s=READY pgs=392 cs=0 l=1 rev1=1 crypto rx=0x7f67fc09b4b0 tx=0x7f67fc0d5530 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:24:23.182 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:24:23.176+0000 7f6814d64740 1 --2- 172.21.15.118:0/2437262792 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55be4ba993f0 0x55be4ba99810 secure :-1 s=READY pgs=458 cs=0 l=1 rev1=1 crypto rx=0x7f680402c700 tx=0x7f68040323d0 comp rx=0 tx=0).stop 2024-03-24T00:24:23.183 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:24:23.176+0000 7f6814d64740 1 -- 172.21.15.118:0/2437262792 shutdown_connections 2024-03-24T00:24:23.183 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:24:23.176+0000 7f6814d64740 1 --2- 172.21.15.118:0/2437262792 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55be4ba9ef50 0x55be4baa13e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:24:23.183 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:24:23.176+0000 7f6814d64740 1 --2- 172.21.15.118:0/2437262792 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55be4ba99de0 0x55be4ba9e8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:24:23.183 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:24:23.176+0000 7f6814d64740 1 --2- 172.21.15.118:0/2437262792 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55be4ba993f0 0x55be4ba99810 unknown :-1 s=CLOSED pgs=458 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:24:23.183 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:24:23.176+0000 7f6814d64740 1 -- 172.21.15.118:0/2437262792 >> 172.21.15.118:0/2437262792 conn(0x55be4ba94960 msgr2=0x55be4ba94d60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:24:23.183 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:24:23.176+0000 7f6814d64740 1 -- 172.21.15.118:0/2437262792 shutdown_connections 2024-03-24T00:24:23.183 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:24:23.176+0000 7f6814d64740 1 -- 172.21.15.118:0/2437262792 wait complete. 2024-03-24T00:24:23.183 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:24:23.177+0000 7f6814d64740 1 -- start start 2024-03-24T00:24:23.183 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:24:23.177+0000 7f6814d64740 1 Processor -- start 2024-03-24T00:24:23.183 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:24:23.177+0000 7f6814d64740 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55be4ba993f0 0x55be4baa3540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:24:23.183 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:24:23.177+0000 7f6814d64740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55be4ba99de0 0x55be4baa3a80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:24:23.183 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:24:23.177+0000 7f6814d64740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55be4ba9ef50 0x55be4baaaac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:24:23.184 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:24:23.177+0000 7f6814d64740 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x55be4b9baa20 con 0x55be4ba993f0 2024-03-24T00:24:23.184 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:24:23.177+0000 7f6814d64740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x55be4b9fa8d0 con 0x55be4ba99de0 2024-03-24T00:24:23.184 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:24:23.177+0000 7f6814d64740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x55be4baa1ea0 con 0x55be4ba9ef50 2024-03-24T00:24:23.184 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:24:23.177+0000 7f6814d64740 10 client.0 _ll_register_callbacks cb 0x55be4b9f71c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T00:24:23.184 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:24:23.177+0000 7f680aa00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55be4ba99de0 0x55be4baa3a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:24:23.184 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:24:23.177+0000 7f680be00640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55be4ba993f0 0x55be4baa3540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:24:23.184 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:24:23.177+0000 7f680aa00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55be4ba99de0 0x55be4baa3a80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.118:51224/0 (socket says 172.21.15.118:51224) 2024-03-24T00:24:23.184 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:24:23.177+0000 7f680aa00640 1 -- 172.21.15.118:0/4104845652 learned_addr learned my addr 172.21.15.118:0/4104845652 (peer_addr_for_me v2:172.21.15.118:0/0) 2024-03-24T00:24:23.184 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:24:23.177+0000 7f680be00640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55be4ba993f0 0x55be4baa3540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.118:3300/0 says I am v2:172.21.15.118:41284/0 (socket says 172.21.15.118:41284) 2024-03-24T00:24:23.184 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:24:23.178+0000 7f680be00640 1 --2- 172.21.15.118:0/4104845652 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55be4ba9ef50 0x55be4baaaac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:24:23.184 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:24:23.178+0000 7f680aa00640 1 -- 172.21.15.118:0/4104845652 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55be4ba9ef50 msgr2=0x55be4baaaac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:24:23.184 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:24:23.178+0000 7f680aa00640 1 --2- 172.21.15.118:0/4104845652 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55be4ba9ef50 0x55be4baaaac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:24:23.185 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:24:23.178+0000 7f680aa00640 1 -- 172.21.15.118:0/4104845652 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55be4ba993f0 msgr2=0x55be4baa3540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:24:23.185 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:24:23.178+0000 7f680aa00640 1 --2- 172.21.15.118:0/4104845652 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55be4ba993f0 0x55be4baa3540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:24:23.185 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:24:23.178+0000 7f680aa00640 1 -- 172.21.15.118:0/4104845652 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55be4b713210 con 0x55be4ba99de0 2024-03-24T00:24:23.185 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:24:23.178+0000 7f680be00640 1 --2- 172.21.15.118:0/4104845652 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55be4ba993f0 0x55be4baa3540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T00:24:23.185 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:24:23.178+0000 7f680aa00640 1 --2- 172.21.15.118:0/4104845652 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55be4ba99de0 0x55be4baa3a80 secure :-1 s=READY pgs=457 cs=0 l=1 rev1=1 crypto rx=0x7f67f8012700 tx=0x7f67f8012bc0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=64f39c1b574ea099 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:24:23.185 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:24:23.178+0000 7f67f4c00640 10 client.0 ms_handle_connect on v2:172.21.15.189:3300/0 2024-03-24T00:24:23.185 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:24:23.178+0000 7f67f4c00640 1 -- 172.21.15.118:0/4104845652 <== mon.2 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f67f8004050 con 0x55be4ba99de0 2024-03-24T00:24:23.185 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:24:23.179+0000 7f6814d64740 1 -- 172.21.15.118:0/4104845652 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f680402cca0 con 0x55be4ba99de0 2024-03-24T00:24:23.185 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:24:23.179+0000 7f67f4c00640 1 -- 172.21.15.118:0/4104845652 <== mon.2 v2:172.21.15.189:3300/0 2 ==== osd_map(118..118 src has 1..118) v4 ==== 10685+0+0 (secure 0 0 0) 0x7f67f80041f0 con 0x55be4ba99de0 2024-03-24T00:24:23.185 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:24:23.179+0000 7f67f4c00640 1 -- 172.21.15.118:0/4104845652 <== mon.2 v2:172.21.15.189:3300/0 3 ==== mon_get_version_reply(handle=1 version=162) v2 ==== 24+0+0 (secure 0 0 0) 0x7f67f8024050 con 0x55be4ba99de0 2024-03-24T00:24:23.185 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:24:23.179+0000 7f6814d64740 10 client.26990 fetch_fsmap learned FSMap version 162 2024-03-24T00:24:23.186 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:24:23.179+0000 7f6814d64740 1 -- 172.21.15.118:0/4104845652 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,fsmap=162,monmap=4+}) v3 -- 0x55be4b7df620 con 0x55be4ba99de0 2024-03-24T00:24:23.186 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:24:23.179+0000 7f67f4c00640 1 -- 172.21.15.118:0/4104845652 <== mon.2 v2:172.21.15.189:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f680402cca0 con 0x55be4ba99de0 2024-03-24T00:24:23.186 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:24:23.179+0000 7f67f4c00640 1 -- 172.21.15.118:0/4104845652 <== mon.2 v2:172.21.15.189:3300/0 5 ==== fsmap(e 162) v1 ==== 5658+0+0 (secure 0 0 0) 0x7f67f8003ab0 con 0x55be4ba99de0 2024-03-24T00:24:23.186 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:24:23.179+0000 7f6814d64740 10 client.26990 fetch_fsmap finished waiting for FSMap version 162 2024-03-24T00:24:23.186 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:24:23.179+0000 7f6814d64740 10 client.26990 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-24T00:24:23.186 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:24:23.179+0000 7f6814d64740 20 client.26990 populate_metadata read hostname 'smithi118' 2024-03-24T00:24:23.186 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:24:23.179+0000 7f6814d64740 1 --2- 172.21.15.118:0/4104845652 >> [v2:172.21.15.189:6838/4074484168,v1:172.21.15.189:6839/4074484168] conn(0x55be4baaf6f0 0x55be4bab1b10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:24:23.186 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:24:23.179+0000 7f6814d64740 4 client.26990 mds_command: new command op to 24569 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-24T00:24:23.186 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:24:23.179+0000 7f6814d64740 1 -- 172.21.15.118:0/4104845652 --> [v2:172.21.15.189:6838/4074484168,v1:172.21.15.189:6839/4074484168] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x55be4ba0bd70 con 0x55be4baaf6f0 2024-03-24T00:24:23.187 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:24:23.180+0000 7f680be00640 1 --2- 172.21.15.118:0/4104845652 >> [v2:172.21.15.189:6838/4074484168,v1:172.21.15.189:6839/4074484168] conn(0x55be4baaf6f0 0x55be4bab1b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:24:23.187 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:24:23.180+0000 7f680be00640 1 --2- 172.21.15.118:0/4104845652 >> [v2:172.21.15.189:6838/4074484168,v1:172.21.15.189:6839/4074484168] conn(0x55be4baaf6f0 0x55be4bab1b10 crc :-1 s=READY pgs=378 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=9d5c41d0325b24ba server_cookie=ec2de6dfc26e697 in_seq=0 out_seq=0 2024-03-24T00:24:23.187 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:24:23.180+0000 7f67f4c00640 10 client.26990 ms_handle_connect on v2:172.21.15.189:6838/4074484168 2024-03-24T00:24:23.187 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:24:23.181+0000 7f67f4c00640 1 -- 172.21.15.118:0/4104845652 <== mds.0 v2:172.21.15.189:6838/4074484168 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x55be4ba0bd70 con 0x55be4baaf6f0 2024-03-24T00:24:23.187 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:24:23.181+0000 7f67f4c00640 10 client.26990 handle_command_reply: tid=0 2024-03-24T00:24:23.187 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:24:23.181+0000 7f6814d64740 1 client.26990 shutdown 2024-03-24T00:24:23.187 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:24:23.181+0000 7f6814d64740 1 -- 172.21.15.118:0/4104845652 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55be4ba99de0 msgr2=0x55be4baa3a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:24:23.187 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:24:23.181+0000 7f6814d64740 1 --2- 172.21.15.118:0/4104845652 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55be4ba99de0 0x55be4baa3a80 secure :-1 s=READY pgs=457 cs=0 l=1 rev1=1 crypto rx=0x7f67f8012700 tx=0x7f67f8012bc0 comp rx=0 tx=0).stop 2024-03-24T00:24:23.187 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:24:23.181+0000 7f6814d64740 1 -- 172.21.15.118:0/4104845652 shutdown_connections 2024-03-24T00:24:23.187 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:24:23.181+0000 7f6814d64740 1 --2- 172.21.15.118:0/4104845652 >> [v2:172.21.15.189:6838/4074484168,v1:172.21.15.189:6839/4074484168] conn(0x55be4baaf6f0 0x55be4bab1b10 crc :-1 s=READY pgs=378 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:24:23.187 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:24:23.181+0000 7f6814d64740 1 --2- 172.21.15.118:0/4104845652 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55be4ba9ef50 0x55be4baaaac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:24:23.188 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:24:23.181+0000 7f6814d64740 1 --2- 172.21.15.118:0/4104845652 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55be4ba99de0 0x55be4baa3a80 unknown :-1 s=CLOSED pgs=457 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:24:23.188 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:24:23.181+0000 7f6814d64740 1 --2- 172.21.15.118:0/4104845652 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55be4ba993f0 0x55be4baa3540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:24:23.188 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:24:23.181+0000 7f67f4c00640 0 client.26990 ms_handle_reset on v2:172.21.15.189:6838/4074484168 2024-03-24T00:24:23.188 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:24:23.181+0000 7f6814d64740 1 -- 172.21.15.118:0/4104845652 >> 172.21.15.118:0/4104845652 conn(0x55be4ba94960 msgr2=0x55be4ba93b70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:24:23.188 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:24:23.182+0000 7f6814d64740 1 -- 172.21.15.118:0/4104845652 shutdown_connections 2024-03-24T00:24:23.188 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:24:23.182+0000 7f6814d64740 1 -- 172.21.15.118:0/4104845652 wait complete. 2024-03-24T00:24:23.188 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:24:23.182+0000 7f6814d64740 20 client.26990 trim_cache size 0 max 16384 2024-03-24T00:24:23.197 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:24:23.197+0000 7f6814d64740 1 Processor -- start 2024-03-24T00:24:23.197 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:24:23.197+0000 7f6814d64740 1 -- start start 2024-03-24T00:24:23.197 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:24:22 smithi118 ceph-mon[31535]: pgmap v686: 97 pgs: 97 active+clean; 146 MiB data, 863 MiB used, 1.0 TiB / 1.0 TiB avail; 142 KiB/s rd, 8.7 KiB/s wr, 34 op/s 2024-03-24T00:24:23.198 INFO:journalctl@ceph.mds.j.smithi118.stdout:Mar 24 00:24:23 smithi118 systemd[1]: ceph-d79a5420-e971-11ee-95cd-87774f69a715@mds.j.service: Scheduled restart job, restart counter is at 3. 2024-03-24T00:24:23.199 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:24:23.198+0000 7f6814d64740 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55be4ba993f0 0x55be4ba8f6a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:24:23.199 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:24:23.198+0000 7f6814d64740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55be4ba99de0 0x55be4baaaa80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:24:23.199 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:24:23.198+0000 7f6814d64740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55be4ba9ef50 0x55be4baa3540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:24:23.200 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:24:23.198+0000 7f6814d64740 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f680c1ace30 con 0x55be4ba993f0 2024-03-24T00:24:23.200 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:24:23.198+0000 7f6814d64740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f680c1ad490 con 0x55be4ba99de0 2024-03-24T00:24:23.200 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:24:23.198+0000 7f6814d64740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f680c1aae50 con 0x55be4ba9ef50 2024-03-24T00:24:23.200 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:24:23.198+0000 7f680be00640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55be4ba993f0 0x55be4ba8f6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:24:23.200 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:24:23.198+0000 7f680be00640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55be4ba993f0 0x55be4ba8f6a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.118:3300/0 says I am v2:172.21.15.118:41290/0 (socket says 172.21.15.118:41290) 2024-03-24T00:24:23.200 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:24:23.198+0000 7f680be00640 1 -- 172.21.15.118:0/2996810568 learned_addr learned my addr 172.21.15.118:0/2996810568 (peer_addr_for_me v2:172.21.15.118:0/0) 2024-03-24T00:24:23.200 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:24:23.198+0000 7f680aa00640 1 --2- 172.21.15.118:0/2996810568 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55be4ba99de0 0x55be4baaaa80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:24:23.200 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:24:23.198+0000 7f680be00640 1 --2- 172.21.15.118:0/2996810568 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55be4ba9ef50 0x55be4baa3540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:24:23.200 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:24:23.199+0000 7f680aa00640 1 -- 172.21.15.118:0/2996810568 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55be4ba9ef50 msgr2=0x55be4baa3540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:24:23.200 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:24:23.199+0000 7f680aa00640 1 --2- 172.21.15.118:0/2996810568 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55be4ba9ef50 0x55be4baa3540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:24:23.200 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:24:23.199+0000 7f680aa00640 1 -- 172.21.15.118:0/2996810568 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55be4ba993f0 msgr2=0x55be4ba8f6a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:24:23.201 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:24:23.199+0000 7f680aa00640 1 --2- 172.21.15.118:0/2996810568 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55be4ba993f0 0x55be4ba8f6a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:24:23.201 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:24:23.199+0000 7f680aa00640 1 -- 172.21.15.118:0/2996810568 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55be4ba997f0 con 0x55be4ba99de0 2024-03-24T00:24:23.201 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:24:23.199+0000 7f680be00640 1 --2- 172.21.15.118:0/2996810568 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55be4ba993f0 0x55be4ba8f6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T00:24:23.201 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:24:23.199+0000 7f680aa00640 1 --2- 172.21.15.118:0/2996810568 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55be4ba99de0 0x55be4baaaa80 secure :-1 s=READY pgs=458 cs=0 l=1 rev1=1 crypto rx=0x7f67f800eff0 tx=0x7f67f8009d30 comp rx=0 tx=0).ready entity=mon.2 client_cookie=4a0ec827f8b3a620 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:24:23.201 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:24:23.199+0000 7f67f7400640 1 -- 172.21.15.118:0/2996810568 <== mon.2 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f67f8020830 con 0x55be4ba99de0 2024-03-24T00:24:23.201 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:24:23.199+0000 7f67f7400640 1 -- 172.21.15.118:0/2996810568 <== mon.2 v2:172.21.15.189:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f67f800f9a0 con 0x55be4ba99de0 2024-03-24T00:24:23.201 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:24:23.199+0000 7f67f7400640 1 -- 172.21.15.118:0/2996810568 <== mon.2 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f67f80209d0 con 0x55be4ba99de0 2024-03-24T00:24:23.205 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:24:23.200+0000 7f6814d64740 1 -- 172.21.15.118:0/2996810568 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55be4ba99de0 msgr2=0x55be4baaaa80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:24:23.205 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:24:23.200+0000 7f6814d64740 1 --2- 172.21.15.118:0/2996810568 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55be4ba99de0 0x55be4baaaa80 secure :-1 s=READY pgs=458 cs=0 l=1 rev1=1 crypto rx=0x7f67f800eff0 tx=0x7f67f8009d30 comp rx=0 tx=0).stop 2024-03-24T00:24:23.205 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:24:23.200+0000 7f6814d64740 1 -- 172.21.15.118:0/2996810568 shutdown_connections 2024-03-24T00:24:23.205 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:24:23.200+0000 7f6814d64740 1 --2- 172.21.15.118:0/2996810568 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55be4ba9ef50 0x55be4baa3540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:24:23.205 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:24:23.200+0000 7f6814d64740 1 --2- 172.21.15.118:0/2996810568 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55be4ba99de0 0x55be4baaaa80 unknown :-1 s=CLOSED pgs=458 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:24:23.205 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:24:23.200+0000 7f6814d64740 1 --2- 172.21.15.118:0/2996810568 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55be4ba993f0 0x55be4ba8f6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:24:23.205 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:24:23.200+0000 7f6814d64740 1 -- 172.21.15.118:0/2996810568 >> 172.21.15.118:0/2996810568 conn(0x55be4ba94960 msgr2=0x55be4ba94d40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:24:23.205 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:24:23.200+0000 7f6814d64740 1 -- 172.21.15.118:0/2996810568 shutdown_connections 2024-03-24T00:24:23.205 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:24:23.200+0000 7f6814d64740 1 -- 172.21.15.118:0/2996810568 wait complete. 2024-03-24T00:24:23.205 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:24:23.201+0000 7f6814d64740 1 -- start start 2024-03-24T00:24:23.205 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:24:23.201+0000 7f6814d64740 1 Processor -- start 2024-03-24T00:24:23.205 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:24:23.201+0000 7f6814d64740 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55be4ba993f0 0x55be4baa8200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:24:23.205 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:24:23.201+0000 7f6814d64740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55be4ba99de0 0x55be4baa8740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:24:23.206 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:24:23.201+0000 7f6814d64740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55be4ba9ef50 0x55be4baa9750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:24:23.206 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:24:23.201+0000 7f6814d64740 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f680c1ab4b0 con 0x55be4ba993f0 2024-03-24T00:24:23.206 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:24:23.201+0000 7f6814d64740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f680c1ac7d0 con 0x55be4ba9ef50 2024-03-24T00:24:23.206 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:24:23.201+0000 7f6814d64740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f680c1abb10 con 0x55be4ba99de0 2024-03-24T00:24:23.206 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:24:23.201+0000 7f6814d64740 10 client.0 _ll_register_callbacks cb 0x7f67f8009880 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T00:24:23.206 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:24:23.202+0000 7f680aa00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55be4ba99de0 0x55be4baa8740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:24:23.206 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:24:23.202+0000 7f680be00640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55be4ba993f0 0x55be4baa8200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:24:23.206 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:24:23.202+0000 7f680aa00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55be4ba99de0 0x55be4baa8740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.118:46836/0 (socket says 172.21.15.118:46836) 2024-03-24T00:24:23.206 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:24:23.202+0000 7f680aa00640 1 -- 172.21.15.118:0/2604137503 learned_addr learned my addr 172.21.15.118:0/2604137503 (peer_addr_for_me v2:172.21.15.118:0/0) 2024-03-24T00:24:23.206 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:24:23.202+0000 7f680be00640 1 --2- 172.21.15.118:0/2604137503 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55be4ba9ef50 0x55be4baa9750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:24:23.206 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:24:23.202+0000 7f680aa00640 1 -- 172.21.15.118:0/2604137503 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55be4ba9ef50 msgr2=0x55be4baa9750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:24:23.207 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:24:23.202+0000 7f680aa00640 1 --2- 172.21.15.118:0/2604137503 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55be4ba9ef50 0x55be4baa9750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:24:23.207 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:24:23.202+0000 7f680aa00640 1 -- 172.21.15.118:0/2604137503 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55be4ba993f0 msgr2=0x55be4baa8200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:24:23.207 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:24:23.202+0000 7f680aa00640 1 --2- 172.21.15.118:0/2604137503 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55be4ba993f0 0x55be4baa8200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:24:23.207 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:24:23.202+0000 7f680aa00640 1 -- 172.21.15.118:0/2604137503 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55be4baa9d40 con 0x55be4ba99de0 2024-03-24T00:24:23.207 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:24:23.202+0000 7f680be00640 1 --2- 172.21.15.118:0/2604137503 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55be4ba9ef50 0x55be4baa9750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T00:24:23.207 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:24:23.202+0000 7f680aa00640 1 --2- 172.21.15.118:0/2604137503 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55be4ba99de0 0x55be4baa8740 secure :-1 s=READY pgs=459 cs=0 l=1 rev1=1 crypto rx=0x7f67f8000c00 tx=0x7f67f8005970 comp rx=0 tx=0).ready entity=mon.1 client_cookie=b3357df02ced09a server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:24:23.207 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:24:23.202+0000 7f67f4c00640 10 client.0 ms_handle_connect on v2:172.21.15.193:3300/0 2024-03-24T00:24:23.207 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:24:23.203+0000 7f67f4c00640 1 -- 172.21.15.118:0/2604137503 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f67f8009e90 con 0x55be4ba99de0 2024-03-24T00:24:23.207 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:24:23.203+0000 7f67f4c00640 1 -- 172.21.15.118:0/2604137503 <== mon.1 v2:172.21.15.193:3300/0 2 ==== osd_map(118..118 src has 1..118) v4 ==== 10685+0+0 (secure 0 0 0) 0x7f67f8020a40 con 0x55be4ba99de0 2024-03-24T00:24:23.207 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:24:23.203+0000 7f6814d64740 1 -- 172.21.15.118:0/2604137503 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f67f800f9a0 con 0x55be4ba99de0 2024-03-24T00:24:23.208 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:24:23.203+0000 7f67f4c00640 1 -- 172.21.15.118:0/2604137503 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_get_version_reply(handle=1 version=162) v2 ==== 24+0+0 (secure 0 0 0) 0x7f67f803f050 con 0x55be4ba99de0 2024-03-24T00:24:23.208 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:24:23.203+0000 7f6814d64740 10 client.27007 fetch_fsmap learned FSMap version 162 2024-03-24T00:24:23.208 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:24:23.203+0000 7f6814d64740 1 -- 172.21.15.118:0/2604137503 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,fsmap=162,monmap=4+}) v3 -- 0x55be4bb0afe0 con 0x55be4ba99de0 2024-03-24T00:24:23.208 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:24:23.203+0000 7f67f4c00640 1 -- 172.21.15.118:0/2604137503 <== mon.1 v2:172.21.15.193:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f67f800f9a0 con 0x55be4ba99de0 2024-03-24T00:24:23.208 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:24:23.203+0000 7f67f4c00640 1 -- 172.21.15.118:0/2604137503 <== mon.1 v2:172.21.15.193:3300/0 5 ==== fsmap(e 162) v1 ==== 5658+0+0 (secure 0 0 0) 0x7f67f80054a0 con 0x55be4ba99de0 2024-03-24T00:24:23.208 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:24:23.203+0000 7f6814d64740 10 client.27007 fetch_fsmap finished waiting for FSMap version 162 2024-03-24T00:24:23.208 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:24:23.203+0000 7f6814d64740 10 client.27007 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-24T00:24:23.208 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:24:23.203+0000 7f6814d64740 20 client.27007 populate_metadata read hostname 'smithi118' 2024-03-24T00:24:23.208 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:24:23.203+0000 7f6814d64740 1 --2- 172.21.15.118:0/2604137503 >> [v2:172.21.15.189:6838/4074484168,v1:172.21.15.189:6839/4074484168] conn(0x55be4bb0b850 0x55be4bb0dcb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:24:23.208 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:24:23.203+0000 7f6814d64740 4 client.27007 mds_command: new command op to 24569 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-03-24T00:24:23.208 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:24:23.203+0000 7f6814d64740 1 -- 172.21.15.118:0/2604137503 --> [v2:172.21.15.189:6838/4074484168,v1:172.21.15.189:6839/4074484168] -- command(tid 0: {"prefix": "flush journal"}) v1 -- 0x55be4ba060c0 con 0x55be4bb0b850 2024-03-24T00:24:23.209 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:24:23.204+0000 7f680be00640 1 --2- 172.21.15.118:0/2604137503 >> [v2:172.21.15.189:6838/4074484168,v1:172.21.15.189:6839/4074484168] conn(0x55be4bb0b850 0x55be4bb0dcb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:24:23.209 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:24:23.204+0000 7f680be00640 1 --2- 172.21.15.118:0/2604137503 >> [v2:172.21.15.189:6838/4074484168,v1:172.21.15.189:6839/4074484168] conn(0x55be4bb0b850 0x55be4bb0dcb0 crc :-1 s=READY pgs=379 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=fb7bd8da11c8111c server_cookie=acc59f01118037b3 in_seq=0 out_seq=0 2024-03-24T00:24:23.209 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:24:23.204+0000 7f67f4c00640 10 client.27007 ms_handle_connect on v2:172.21.15.189:6838/4074484168 2024-03-24T00:24:23.209 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:24:22 smithi189 ceph-mon[36187]: pgmap v686: 97 pgs: 97 active+clean; 146 MiB data, 863 MiB used, 1.0 TiB / 1.0 TiB avail; 142 KiB/s rd, 8.7 KiB/s wr, 34 op/s 2024-03-24T00:24:23.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:24:22 smithi193 ceph-mon[36878]: pgmap v686: 97 pgs: 97 active+clean; 146 MiB data, 863 MiB used, 1.0 TiB / 1.0 TiB avail; 142 KiB/s rd, 8.7 KiB/s wr, 34 op/s 2024-03-24T00:24:23.568 INFO:journalctl@ceph.mds.d.smithi118.stdout:Mar 24 00:24:23 smithi118 systemd[1]: ceph-d79a5420-e971-11ee-95cd-87774f69a715@mds.d.service: Scheduled restart job, restart counter is at 3. 2024-03-24T00:24:23.569 INFO:journalctl@ceph.mds.d.smithi118.stdout:Mar 24 00:24:23 smithi118 systemd[1]: Stopped Ceph mds.d for d79a5420-e971-11ee-95cd-87774f69a715. 2024-03-24T00:24:23.569 INFO:journalctl@ceph.mds.d.smithi118.stdout:Mar 24 00:24:23 smithi118 systemd[1]: ceph-d79a5420-e971-11ee-95cd-87774f69a715@mds.d.service: Consumed 1.663s CPU time. 2024-03-24T00:24:23.569 INFO:journalctl@ceph.mds.d.smithi118.stdout:Mar 24 00:24:23 smithi118 systemd[1]: Starting Ceph mds.d for d79a5420-e971-11ee-95cd-87774f69a715... 2024-03-24T00:24:23.569 INFO:journalctl@ceph.mds.j.smithi118.stdout:Mar 24 00:24:23 smithi118 systemd[1]: Stopped Ceph mds.j for d79a5420-e971-11ee-95cd-87774f69a715. 2024-03-24T00:24:23.569 INFO:journalctl@ceph.mds.j.smithi118.stdout:Mar 24 00:24:23 smithi118 systemd[1]: ceph-d79a5420-e971-11ee-95cd-87774f69a715@mds.j.service: Consumed 1.401s CPU time. 2024-03-24T00:24:23.569 INFO:journalctl@ceph.mds.j.smithi118.stdout:Mar 24 00:24:23 smithi118 systemd[1]: Starting Ceph mds.j for d79a5420-e971-11ee-95cd-87774f69a715... 2024-03-24T00:24:23.904 INFO:journalctl@ceph.mds.j.smithi118.stdout:Mar 24 00:24:23 smithi118 podman[96510]: 2024-03-24 00:24:23.475136621 +0000 UTC m=+0.047071643 image pull c664afe56bb01db4f9afa9532c20598e201e664f2e9ba2be50db0d52674a26d2 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc 2024-03-24T00:24:23.904 INFO:journalctl@ceph.mds.d.smithi118.stdout:Mar 24 00:24:23 smithi118 podman[96511]: 2024-03-24 00:24:23.474516567 +0000 UTC m=+0.045665090 image pull c664afe56bb01db4f9afa9532c20598e201e664f2e9ba2be50db0d52674a26d2 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc 2024-03-24T00:24:23.905 INFO:journalctl@ceph.mds.d.smithi118.stdout:Mar 24 00:24:23 smithi118 podman[96511]: 2024-03-24 00:24:23.638957389 +0000 UTC m=+0.210105949 container create ff8d7eec8ba9a530f9c147d53d29c72fb675d996090dd6c3c96ae7997968c0b0 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-d, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240319, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_CLEAN=True, org.label-schema.schema-version=1.0, ceph=True) 2024-03-24T00:24:24.196 INFO:journalctl@ceph.mds.j.smithi118.stdout:Mar 24 00:24:24 smithi118 podman[96510]: 2024-03-24 00:24:24.067122545 +0000 UTC m=+0.639057567 container create 7244d2bf20660b74adf0f39669b2b63aaf0e1df2c7ba2ec622c5b2864eba0601 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-j, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, org.label-schema.license=GPLv2, ceph=True, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_BRANCH=HEAD, org.label-schema.build-date=20240319, GIT_CLEAN=True) 2024-03-24T00:24:24.197 INFO:journalctl@ceph.mds.d.smithi118.stdout:Mar 24 00:24:23 smithi118 podman[96511]: 2024-03-24 00:24:23.914427013 +0000 UTC m=+0.485575579 container init ff8d7eec8ba9a530f9c147d53d29c72fb675d996090dd6c3c96ae7997968c0b0 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-d, org.label-schema.schema-version=1.0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240319, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c) 2024-03-24T00:24:24.197 INFO:journalctl@ceph.mds.d.smithi118.stdout:Mar 24 00:24:23 smithi118 podman[96511]: 2024-03-24 00:24:23.927389977 +0000 UTC m=+0.498538564 container start ff8d7eec8ba9a530f9c147d53d29c72fb675d996090dd6c3c96ae7997968c0b0 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-d, GIT_BRANCH=HEAD, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240319, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True) 2024-03-24T00:24:24.197 INFO:journalctl@ceph.mds.d.smithi118.stdout:Mar 24 00:24:23 smithi118 ceph-mds[96542]: -- 172.21.15.118:0/2135331815 <== mon.1 v2:172.21.15.193:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x558fad891a40 con 0x558fad865000 2024-03-24T00:24:24.197 INFO:journalctl@ceph.mds.d.smithi118.stdout:Mar 24 00:24:23 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-d[96538]: starting mds.d at 2024-03-24T00:24:24.586 INFO:journalctl@ceph.mds.j.smithi118.stdout:Mar 24 00:24:24 smithi118 podman[96510]: 2024-03-24 00:24:24.330031883 +0000 UTC m=+0.901966940 container init 7244d2bf20660b74adf0f39669b2b63aaf0e1df2c7ba2ec622c5b2864eba0601 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-j, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240319, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, org.label-schema.license=GPLv2, GIT_CLEAN=True, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, ceph=True, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS) 2024-03-24T00:24:24.586 INFO:journalctl@ceph.mds.j.smithi118.stdout:Mar 24 00:24:24 smithi118 podman[96510]: 2024-03-24 00:24:24.34096986 +0000 UTC m=+0.912904914 container start 7244d2bf20660b74adf0f39669b2b63aaf0e1df2c7ba2ec622c5b2864eba0601 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-j, org.label-schema.license=GPLv2, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.build-date=20240319, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, ceph=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30) 2024-03-24T00:24:24.586 INFO:journalctl@ceph.mds.j.smithi118.stdout:Mar 24 00:24:24 smithi118 ceph-mds[96578]: -- 172.21.15.118:0/2183211195 <== mon.2 v2:172.21.15.189:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x55948d3a5a40 con 0x55948d378800 2024-03-24T00:24:24.586 INFO:journalctl@ceph.mds.j.smithi118.stdout:Mar 24 00:24:24 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-j[96574]: starting mds.j at 2024-03-24T00:24:24.586 INFO:journalctl@ceph.mds.j.smithi118.stdout:Mar 24 00:24:24 smithi118 bash[96510]: 7244d2bf20660b74adf0f39669b2b63aaf0e1df2c7ba2ec622c5b2864eba0601 2024-03-24T00:24:24.586 INFO:journalctl@ceph.mds.j.smithi118.stdout:Mar 24 00:24:24 smithi118 systemd[1]: Started Ceph mds.j for d79a5420-e971-11ee-95cd-87774f69a715. 2024-03-24T00:24:24.851 INFO:journalctl@ceph.mds.d.smithi118.stdout:Mar 24 00:24:24 smithi118 bash[96511]: ff8d7eec8ba9a530f9c147d53d29c72fb675d996090dd6c3c96ae7997968c0b0 2024-03-24T00:24:25.106 INFO:journalctl@ceph.mds.d.smithi118.stdout:Mar 24 00:24:24 smithi118 systemd[1]: Started Ceph mds.d for d79a5420-e971-11ee-95cd-87774f69a715. 2024-03-24T00:24:25.107 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:24:24 smithi118 ceph-mon[31535]: pgmap v687: 97 pgs: 97 active+clean; 146 MiB data, 863 MiB used, 1.0 TiB / 1.0 TiB avail; 124 KiB/s rd, 8.7 KiB/s wr, 32 op/s 2024-03-24T00:24:25.107 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:24:24 smithi118 ceph-mon[31535]: daemon mds.d restarted 2024-03-24T00:24:25.107 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:24:24 smithi118 ceph-mon[31535]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-03-24T00:24:25.107 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:24:24 smithi118 ceph-mon[31535]: osdmap e119: 12 total, 12 up, 12 in 2024-03-24T00:24:25.107 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:24:24 smithi118 ceph-mon[31535]: mds.? [v2:172.21.15.118:6834/1095918599,v1:172.21.15.118:6835/1095918599] up:boot 2024-03-24T00:24:25.107 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:24:24 smithi118 ceph-mon[31535]: Standby daemon mds.d assigned to filesystem cephfs as rank 1 2024-03-24T00:24:25.107 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:24:24 smithi118 ceph-mon[31535]: fsmap cephfs:4/5 {0=b=up:active,2=c=up:active,3=j=up:active,4=e=up:active} 3 up:standby-replay 1 up:standby, 1 failed 2024-03-24T00:24:25.107 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:24:24 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2024-03-24T00:24:25.107 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:24:24 smithi118 ceph-mon[31535]: fsmap cephfs:5/5 3 up:standby-replay 1 up:replay 4 up:active 2024-03-24T00:24:25.107 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:24:24 smithi118 ceph-mon[31535]: daemon mds.j restarted 2024-03-24T00:24:25.107 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:24:24 smithi118 ceph-mon[31535]: osdmap e120: 12 total, 12 up, 12 in 2024-03-24T00:24:25.108 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:24:24 smithi118 ceph-mon[31535]: mds.? [v2:172.21.15.118:6834/1095918599,v1:172.21.15.118:6835/1095918599] up:resolve 2024-03-24T00:24:25.108 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:24:24 smithi118 ceph-mon[31535]: mds.? [v2:172.21.15.118:6836/179741564,v1:172.21.15.118:6837/179741564] up:boot 2024-03-24T00:24:25.108 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:24:24 smithi118 ceph-mon[31535]: Standby daemon mds.j assigned to filesystem cephfs as rank 3 2024-03-24T00:24:25.108 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:24:24 smithi118 ceph-mon[31535]: fsmap cephfs:4/5 {0=b=up:active,1=d=up:resolve,2=c=up:active,4=e=up:active} 3 up:standby-replay 1 up:standby, 1 failed 2024-03-24T00:24:25.108 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:24:24 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2024-03-24T00:24:25.108 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:24:24 smithi118 ceph-mon[31535]: fsmap cephfs:5/5 3 up:standby-replay 1 up:replay 1 up:resolve 3 up:active 2024-03-24T00:24:25.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:24:24 smithi193 ceph-mon[36878]: pgmap v687: 97 pgs: 97 active+clean; 146 MiB data, 863 MiB used, 1.0 TiB / 1.0 TiB avail; 124 KiB/s rd, 8.7 KiB/s wr, 32 op/s 2024-03-24T00:24:25.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:24:24 smithi193 ceph-mon[36878]: daemon mds.d restarted 2024-03-24T00:24:25.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:24:24 smithi193 ceph-mon[36878]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-03-24T00:24:25.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:24:24 smithi193 ceph-mon[36878]: osdmap e119: 12 total, 12 up, 12 in 2024-03-24T00:24:25.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:24:24 smithi193 ceph-mon[36878]: mds.? [v2:172.21.15.118:6834/1095918599,v1:172.21.15.118:6835/1095918599] up:boot 2024-03-24T00:24:25.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:24:24 smithi193 ceph-mon[36878]: Standby daemon mds.d assigned to filesystem cephfs as rank 1 2024-03-24T00:24:25.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:24:24 smithi193 ceph-mon[36878]: fsmap cephfs:4/5 {0=b=up:active,2=c=up:active,3=j=up:active,4=e=up:active} 3 up:standby-replay 1 up:standby, 1 failed 2024-03-24T00:24:25.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:24:24 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2024-03-24T00:24:25.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:24:24 smithi193 ceph-mon[36878]: fsmap cephfs:5/5 3 up:standby-replay 1 up:replay 4 up:active 2024-03-24T00:24:25.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:24:24 smithi193 ceph-mon[36878]: daemon mds.j restarted 2024-03-24T00:24:25.212 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:24:24 smithi193 ceph-mon[36878]: osdmap e120: 12 total, 12 up, 12 in 2024-03-24T00:24:25.212 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:24:24 smithi193 ceph-mon[36878]: mds.? [v2:172.21.15.118:6834/1095918599,v1:172.21.15.118:6835/1095918599] up:resolve 2024-03-24T00:24:25.212 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:24:24 smithi193 ceph-mon[36878]: mds.? [v2:172.21.15.118:6836/179741564,v1:172.21.15.118:6837/179741564] up:boot 2024-03-24T00:24:25.212 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:24:24 smithi193 ceph-mon[36878]: Standby daemon mds.j assigned to filesystem cephfs as rank 3 2024-03-24T00:24:25.212 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:24:24 smithi193 ceph-mon[36878]: fsmap cephfs:4/5 {0=b=up:active,1=d=up:resolve,2=c=up:active,4=e=up:active} 3 up:standby-replay 1 up:standby, 1 failed 2024-03-24T00:24:25.212 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:24:24 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2024-03-24T00:24:25.212 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:24:24 smithi193 ceph-mon[36878]: fsmap cephfs:5/5 3 up:standby-replay 1 up:replay 1 up:resolve 3 up:active 2024-03-24T00:24:25.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:24:24 smithi189 ceph-mon[36187]: pgmap v687: 97 pgs: 97 active+clean; 146 MiB data, 863 MiB used, 1.0 TiB / 1.0 TiB avail; 124 KiB/s rd, 8.7 KiB/s wr, 32 op/s 2024-03-24T00:24:25.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:24:24 smithi189 ceph-mon[36187]: daemon mds.d restarted 2024-03-24T00:24:25.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:24:24 smithi189 ceph-mon[36187]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-03-24T00:24:25.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:24:24 smithi189 ceph-mon[36187]: osdmap e119: 12 total, 12 up, 12 in 2024-03-24T00:24:25.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:24:24 smithi189 ceph-mon[36187]: mds.? [v2:172.21.15.118:6834/1095918599,v1:172.21.15.118:6835/1095918599] up:boot 2024-03-24T00:24:25.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:24:24 smithi189 ceph-mon[36187]: Standby daemon mds.d assigned to filesystem cephfs as rank 1 2024-03-24T00:24:25.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:24:24 smithi189 ceph-mon[36187]: fsmap cephfs:4/5 {0=b=up:active,2=c=up:active,3=j=up:active,4=e=up:active} 3 up:standby-replay 1 up:standby, 1 failed 2024-03-24T00:24:25.455 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:24:24 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2024-03-24T00:24:25.455 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:24:24 smithi189 ceph-mon[36187]: fsmap cephfs:5/5 3 up:standby-replay 1 up:replay 4 up:active 2024-03-24T00:24:25.455 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:24:24 smithi189 ceph-mon[36187]: daemon mds.j restarted 2024-03-24T00:24:25.455 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:24:24 smithi189 ceph-mon[36187]: osdmap e120: 12 total, 12 up, 12 in 2024-03-24T00:24:25.455 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:24:24 smithi189 ceph-mon[36187]: mds.? [v2:172.21.15.118:6834/1095918599,v1:172.21.15.118:6835/1095918599] up:resolve 2024-03-24T00:24:25.455 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:24:24 smithi189 ceph-mon[36187]: mds.? [v2:172.21.15.118:6836/179741564,v1:172.21.15.118:6837/179741564] up:boot 2024-03-24T00:24:25.455 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:24:24 smithi189 ceph-mon[36187]: Standby daemon mds.j assigned to filesystem cephfs as rank 3 2024-03-24T00:24:25.455 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:24:24 smithi189 ceph-mon[36187]: fsmap cephfs:4/5 {0=b=up:active,1=d=up:resolve,2=c=up:active,4=e=up:active} 3 up:standby-replay 1 up:standby, 1 failed 2024-03-24T00:24:25.455 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:24:24 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2024-03-24T00:24:25.455 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:24:24 smithi189 ceph-mon[36187]: fsmap cephfs:5/5 3 up:standby-replay 1 up:replay 1 up:resolve 3 up:active 2024-03-24T00:24:26.687 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:24:26 smithi118 ceph-mon[31535]: mds.? [v2:172.21.15.118:6836/179741564,v1:172.21.15.118:6837/179741564] up:resolve 2024-03-24T00:24:26.687 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:24:26 smithi118 ceph-mon[31535]: fsmap cephfs:5/5 3 up:standby-replay 2 up:resolve 3 up:active 2024-03-24T00:24:26.703 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:24:26 smithi189 ceph-mon[36187]: mds.? [v2:172.21.15.118:6836/179741564,v1:172.21.15.118:6837/179741564] up:resolve 2024-03-24T00:24:26.704 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:24:26 smithi189 ceph-mon[36187]: fsmap cephfs:5/5 3 up:standby-replay 2 up:resolve 3 up:active 2024-03-24T00:24:26.710 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:24:26 smithi193 ceph-mon[36878]: mds.? [v2:172.21.15.118:6836/179741564,v1:172.21.15.118:6837/179741564] up:resolve 2024-03-24T00:24:26.711 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:24:26 smithi193 ceph-mon[36878]: fsmap cephfs:5/5 3 up:standby-replay 2 up:resolve 3 up:active 2024-03-24T00:24:27.467 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:24:27.467+0000 7f67f4c00640 1 -- 172.21.15.118:0/2604137503 <== mds.0 v2:172.21.15.189:6838/4074484168 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+44 (crc 0 0 0) 0x55be4ba060c0 con 0x55be4bb0b850 2024-03-24T00:24:27.467 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:24:27.467+0000 7f67f4c00640 10 client.27007 handle_command_reply: tid=0 2024-03-24T00:24:27.467 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:24:27.467+0000 7f6814d64740 1 client.27007 shutdown 2024-03-24T00:24:27.468 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:24:27.468+0000 7f6814d64740 1 -- 172.21.15.118:0/2604137503 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55be4ba99de0 msgr2=0x55be4baa8740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:24:27.468 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:24:27.468+0000 7f6814d64740 1 --2- 172.21.15.118:0/2604137503 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55be4ba99de0 0x55be4baa8740 secure :-1 s=READY pgs=459 cs=0 l=1 rev1=1 crypto rx=0x7f67f8000c00 tx=0x7f67f8005970 comp rx=0 tx=0).stop 2024-03-24T00:24:27.469 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:24:27.468+0000 7f6814d64740 1 -- 172.21.15.118:0/2604137503 shutdown_connections 2024-03-24T00:24:27.469 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:24:27.468+0000 7f6814d64740 1 --2- 172.21.15.118:0/2604137503 >> [v2:172.21.15.189:6838/4074484168,v1:172.21.15.189:6839/4074484168] conn(0x55be4bb0b850 0x55be4bb0dcb0 crc :-1 s=READY pgs=379 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:24:27.469 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:24:27.468+0000 7f6814d64740 1 --2- 172.21.15.118:0/2604137503 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x55be4ba9ef50 0x55be4baa9750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:24:27.469 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:24:27.468+0000 7f6814d64740 1 --2- 172.21.15.118:0/2604137503 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x55be4ba99de0 0x55be4baa8740 unknown :-1 s=CLOSED pgs=459 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:24:27.469 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:24:27.468+0000 7f6814d64740 1 --2- 172.21.15.118:0/2604137503 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x55be4ba993f0 0x55be4baa8200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:24:27.469 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:24:27.468+0000 7f6814d64740 1 -- 172.21.15.118:0/2604137503 >> 172.21.15.118:0/2604137503 conn(0x55be4ba94960 msgr2=0x55be4baa3520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:24:27.469 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:24:27.468+0000 7f67f4c00640 0 client.27007 ms_handle_reset on v2:172.21.15.189:6838/4074484168 2024-03-24T00:24:27.469 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:24:27.468+0000 7f6814d64740 1 -- 172.21.15.118:0/2604137503 shutdown_connections 2024-03-24T00:24:27.469 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:24:27.469+0000 7f6814d64740 1 -- 172.21.15.118:0/2604137503 wait complete. 2024-03-24T00:24:27.469 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:24:27.469+0000 7f6814d64740 20 client.27007 trim_cache size 0 max 16384 2024-03-24T00:24:27.470 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stdout:{ 2024-03-24T00:24:27.470 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stdout: "message": "", 2024-03-24T00:24:27.470 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stdout: "return_code": 0 2024-03-24T00:24:27.470 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stdout:} 2024-03-24T00:24:27.471 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:24:27.470+0000 7f6810a00640 1 -- 172.21.15.118:0/4203939711 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7f67f00a27f0 msgr2=0x7f67f00a4cb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:24:27.471 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:24:27.470+0000 7f6810a00640 1 --2- 172.21.15.118:0/4203939711 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7f67f00a27f0 0x7f67f00a4cb0 secure :-1 s=READY pgs=392 cs=0 l=1 rev1=1 crypto rx=0x7f67fc09b4b0 tx=0x7f67fc0d5530 comp rx=0 tx=0).stop 2024-03-24T00:24:27.471 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:24:27.470+0000 7f6810a00640 1 -- 172.21.15.118:0/4203939711 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f680c100410 msgr2=0x7f680c1999e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:24:27.471 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:24:27.470+0000 7f6810a00640 1 --2- 172.21.15.118:0/4203939711 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f680c100410 0x7f680c1999e0 secure :-1 s=READY pgs=457 cs=0 l=1 rev1=1 crypto rx=0x7f67fc00e9c0 tx=0x7f67fc00ee80 comp rx=0 tx=0).stop 2024-03-24T00:24:27.471 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:24:27.470+0000 7f6810a00640 1 -- 172.21.15.118:0/4203939711 shutdown_connections 2024-03-24T00:24:27.471 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:24:27.470+0000 7f6810a00640 1 --2- 172.21.15.118:0/4203939711 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7f67f00a27f0 0x7f67f00a4cb0 unknown :-1 s=CLOSED pgs=392 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:24:27.471 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:24:27.470+0000 7f6810a00640 1 --2- 172.21.15.118:0/4203939711 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f680c101fc0 0x7f680c1a0f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:24:27.471 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:24:27.470+0000 7f6810a00640 1 --2- 172.21.15.118:0/4203939711 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f680c101600 0x7f680c199f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:24:27.471 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:24:27.470+0000 7f6810a00640 1 --2- 172.21.15.118:0/4203939711 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f680c100410 0x7f680c1999e0 unknown :-1 s=CLOSED pgs=457 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:24:27.471 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:24:27.470+0000 7f6810a00640 1 -- 172.21.15.118:0/4203939711 >> 172.21.15.118:0/4203939711 conn(0x7f680c0fb9a0 msgr2=0x7f680c0fd420 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:24:27.471 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:24:27.470+0000 7f6810a00640 1 -- 172.21.15.118:0/4203939711 shutdown_connections 2024-03-24T00:24:27.471 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:24:27.471+0000 7f6810a00640 1 -- 172.21.15.118:0/4203939711 wait complete. 2024-03-24T00:24:27.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:24:27 smithi118 ceph-mon[31535]: pgmap v690: 97 pgs: 97 active+clean; 146 MiB data, 863 MiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 9.6 KiB/s wr, 8 op/s 2024-03-24T00:24:27.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:24:27 smithi118 ceph-mon[31535]: mds.? [v2:172.21.15.118:6836/179741564,v1:172.21.15.118:6837/179741564] up:reconnect 2024-03-24T00:24:27.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:24:27 smithi118 ceph-mon[31535]: mds.? [v2:172.21.15.118:6834/1095918599,v1:172.21.15.118:6835/1095918599] up:reconnect 2024-03-24T00:24:27.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:24:27 smithi118 ceph-mon[31535]: fsmap cephfs:5/5 3 up:standby-replay 2 up:reconnect 3 up:active 2024-03-24T00:24:27.697 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:24:27 smithi118 ceph-mon[31535]: reconnect by client.15159 192.168.0.1:0/3463739313 after 0.0279993 2024-03-24T00:24:27.697 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:24:27 smithi118 ceph-mon[31535]: reconnect by client.15159 192.168.0.1:0/3463739313 after 0.0289993 2024-03-24T00:24:27.704 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:24:27 smithi189 ceph-mon[36187]: pgmap v690: 97 pgs: 97 active+clean; 146 MiB data, 863 MiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 9.6 KiB/s wr, 8 op/s 2024-03-24T00:24:27.704 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:24:27 smithi189 ceph-mon[36187]: mds.? [v2:172.21.15.118:6836/179741564,v1:172.21.15.118:6837/179741564] up:reconnect 2024-03-24T00:24:27.704 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:24:27 smithi189 ceph-mon[36187]: mds.? [v2:172.21.15.118:6834/1095918599,v1:172.21.15.118:6835/1095918599] up:reconnect 2024-03-24T00:24:27.704 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:24:27 smithi189 ceph-mon[36187]: fsmap cephfs:5/5 3 up:standby-replay 2 up:reconnect 3 up:active 2024-03-24T00:24:27.704 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:24:27 smithi189 ceph-mon[36187]: reconnect by client.15159 192.168.0.1:0/3463739313 after 0.0279993 2024-03-24T00:24:27.704 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:24:27 smithi189 ceph-mon[36187]: reconnect by client.15159 192.168.0.1:0/3463739313 after 0.0289993 2024-03-24T00:24:27.710 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:24:27 smithi193 ceph-mon[36878]: pgmap v690: 97 pgs: 97 active+clean; 146 MiB data, 863 MiB used, 1.0 TiB / 1.0 TiB avail; 17 KiB/s rd, 9.6 KiB/s wr, 8 op/s 2024-03-24T00:24:27.710 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:24:27 smithi193 ceph-mon[36878]: mds.? [v2:172.21.15.118:6836/179741564,v1:172.21.15.118:6837/179741564] up:reconnect 2024-03-24T00:24:27.711 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:24:27 smithi193 ceph-mon[36878]: mds.? [v2:172.21.15.118:6834/1095918599,v1:172.21.15.118:6835/1095918599] up:reconnect 2024-03-24T00:24:27.711 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:24:27 smithi193 ceph-mon[36878]: fsmap cephfs:5/5 3 up:standby-replay 2 up:reconnect 3 up:active 2024-03-24T00:24:27.711 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:24:27 smithi193 ceph-mon[36878]: reconnect by client.15159 192.168.0.1:0/3463739313 after 0.0279993 2024-03-24T00:24:27.711 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:24:27 smithi193 ceph-mon[36878]: reconnect by client.15159 192.168.0.1:0/3463739313 after 0.0289993 2024-03-24T00:24:28.704 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:24:28 smithi189 ceph-mon[36187]: mds.? [v2:172.21.15.118:6836/179741564,v1:172.21.15.118:6837/179741564] up:rejoin 2024-03-24T00:24:28.704 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:24:28 smithi189 ceph-mon[36187]: mds.? [v2:172.21.15.118:6834/1095918599,v1:172.21.15.118:6835/1095918599] up:rejoin 2024-03-24T00:24:28.704 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:24:28 smithi189 ceph-mon[36187]: fsmap cephfs:5/5 3 up:standby-replay 2 up:rejoin 3 up:active 2024-03-24T00:24:28.704 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:24:28 smithi189 ceph-mon[36187]: daemon mds.j is now active in filesystem cephfs as rank 3 2024-03-24T00:24:28.704 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:24:28 smithi189 ceph-mon[36187]: daemon mds.d is now active in filesystem cephfs as rank 1 2024-03-24T00:24:28.710 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:24:28 smithi193 ceph-mon[36878]: mds.? [v2:172.21.15.118:6836/179741564,v1:172.21.15.118:6837/179741564] up:rejoin 2024-03-24T00:24:28.711 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:24:28 smithi193 ceph-mon[36878]: mds.? [v2:172.21.15.118:6834/1095918599,v1:172.21.15.118:6835/1095918599] up:rejoin 2024-03-24T00:24:28.711 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:24:28 smithi193 ceph-mon[36878]: fsmap cephfs:5/5 3 up:standby-replay 2 up:rejoin 3 up:active 2024-03-24T00:24:28.711 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:24:28 smithi193 ceph-mon[36878]: daemon mds.j is now active in filesystem cephfs as rank 3 2024-03-24T00:24:28.711 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:24:28 smithi193 ceph-mon[36878]: daemon mds.d is now active in filesystem cephfs as rank 1 2024-03-24T00:24:28.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:24:28 smithi118 ceph-mon[31535]: mds.? [v2:172.21.15.118:6836/179741564,v1:172.21.15.118:6837/179741564] up:rejoin 2024-03-24T00:24:28.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:24:28 smithi118 ceph-mon[31535]: mds.? [v2:172.21.15.118:6834/1095918599,v1:172.21.15.118:6835/1095918599] up:rejoin 2024-03-24T00:24:28.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:24:28 smithi118 ceph-mon[31535]: fsmap cephfs:5/5 3 up:standby-replay 2 up:rejoin 3 up:active 2024-03-24T00:24:28.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:24:28 smithi118 ceph-mon[31535]: daemon mds.j is now active in filesystem cephfs as rank 3 2024-03-24T00:24:28.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:24:28 smithi118 ceph-mon[31535]: daemon mds.d is now active in filesystem cephfs as rank 1 2024-03-24T00:24:28.947 INFO:journalctl@ceph.mds.j.smithi118.stdout:Mar 24 00:24:28 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-j[96574]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: In function 'void Locker::set_xlocks_done(MutationImpl*, bool)' thread 7f1aeae00700 time 2024-03-24T00:24:28.489891+0000 2024-03-24T00:24:28.947 INFO:journalctl@ceph.mds.j.smithi118.stdout:Mar 24 00:24:28 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-j[96574]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: 626: FAILED ceph_assert(obj->is_auth()) 2024-03-24T00:24:28.947 INFO:journalctl@ceph.mds.j.smithi118.stdout:Mar 24 00:24:28 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-j[96574]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-24T00:24:28.947 INFO:journalctl@ceph.mds.j.smithi118.stdout:Mar 24 00:24:28 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-j[96574]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x135) [0x7f1af67dd4a1] 2024-03-24T00:24:28.947 INFO:journalctl@ceph.mds.j.smithi118.stdout:Mar 24 00:24:28 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-j[96574]: 2: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7f1af67dd667] 2024-03-24T00:24:28.947 INFO:journalctl@ceph.mds.j.smithi118.stdout:Mar 24 00:24:28 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-j[96574]: 3: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x55948aee0223] 2024-03-24T00:24:28.947 INFO:journalctl@ceph.mds.j.smithi118.stdout:Mar 24 00:24:28 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-j[96574]: 4: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x55948ad5a5d6] 2024-03-24T00:24:28.948 INFO:journalctl@ceph.mds.j.smithi118.stdout:Mar 24 00:24:28 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-j[96574]: 5: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x55948ad5ada1] 2024-03-24T00:24:28.948 INFO:journalctl@ceph.mds.j.smithi118.stdout:Mar 24 00:24:28 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-j[96574]: 6: (Server::handle_client_rename(boost::intrusive_ptr const&)+0x259d) [0x55948ad9797d] 2024-03-24T00:24:28.948 INFO:journalctl@ceph.mds.j.smithi118.stdout:Mar 24 00:24:28 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-j[96574]: 7: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x55948adad233] 2024-03-24T00:24:28.948 INFO:journalctl@ceph.mds.j.smithi118.stdout:Mar 24 00:24:28 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-j[96574]: 8: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x55948adaf197] 2024-03-24T00:24:28.948 INFO:journalctl@ceph.mds.j.smithi118.stdout:Mar 24 00:24:28 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-j[96574]: 9: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x55948acae11b] 2024-03-24T00:24:28.948 INFO:journalctl@ceph.mds.j.smithi118.stdout:Mar 24 00:24:28 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-j[96574]: 10: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x55948acae6bc] 2024-03-24T00:24:28.948 INFO:journalctl@ceph.mds.j.smithi118.stdout:Mar 24 00:24:28 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-j[96574]: 11: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x55948ac981ff] 2024-03-24T00:24:28.948 INFO:journalctl@ceph.mds.j.smithi118.stdout:Mar 24 00:24:28 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-j[96574]: 12: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7f1af6a6f488] 2024-03-24T00:24:28.948 INFO:journalctl@ceph.mds.j.smithi118.stdout:Mar 24 00:24:28 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-j[96574]: 13: (DispatchQueue::entry()+0x51f) [0x7f1af6a6c60f] 2024-03-24T00:24:28.948 INFO:journalctl@ceph.mds.j.smithi118.stdout:Mar 24 00:24:28 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-j[96574]: 14: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f1af6b3fe81] 2024-03-24T00:24:28.948 INFO:journalctl@ceph.mds.j.smithi118.stdout:Mar 24 00:24:28 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-j[96574]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7f1af555a1ca] 2024-03-24T00:24:28.949 INFO:journalctl@ceph.mds.j.smithi118.stdout:Mar 24 00:24:28 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-j[96574]: 16: clone() 2024-03-24T00:24:28.949 INFO:journalctl@ceph.mds.d.smithi118.stdout:Mar 24 00:24:28 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-d[96538]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: In function 'void Locker::set_xlocks_done(MutationImpl*, bool)' thread 7f051e800700 time 2024-03-24T00:24:28.539897+0000 2024-03-24T00:24:28.949 INFO:journalctl@ceph.mds.d.smithi118.stdout:Mar 24 00:24:28 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-d[96538]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: 626: FAILED ceph_assert(obj->is_auth()) 2024-03-24T00:24:28.949 INFO:journalctl@ceph.mds.d.smithi118.stdout:Mar 24 00:24:28 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-d[96538]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-24T00:24:28.949 INFO:journalctl@ceph.mds.d.smithi118.stdout:Mar 24 00:24:28 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-d[96538]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x135) [0x7f052a11f4a1] 2024-03-24T00:24:28.949 INFO:journalctl@ceph.mds.d.smithi118.stdout:Mar 24 00:24:28 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-d[96538]: 2: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7f052a11f667] 2024-03-24T00:24:28.949 INFO:journalctl@ceph.mds.d.smithi118.stdout:Mar 24 00:24:28 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-d[96538]: 3: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x558fab2c8223] 2024-03-24T00:24:28.949 INFO:journalctl@ceph.mds.d.smithi118.stdout:Mar 24 00:24:28 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-d[96538]: 4: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x558fab1425d6] 2024-03-24T00:24:28.950 INFO:journalctl@ceph.mds.d.smithi118.stdout:Mar 24 00:24:28 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-d[96538]: 5: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x558fab142da1] 2024-03-24T00:24:28.950 INFO:journalctl@ceph.mds.d.smithi118.stdout:Mar 24 00:24:28 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-d[96538]: 6: (Server::handle_client_rename(boost::intrusive_ptr const&)+0x259d) [0x558fab17f97d] 2024-03-24T00:24:28.950 INFO:journalctl@ceph.mds.d.smithi118.stdout:Mar 24 00:24:28 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-d[96538]: 7: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x558fab195233] 2024-03-24T00:24:28.950 INFO:journalctl@ceph.mds.d.smithi118.stdout:Mar 24 00:24:28 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-d[96538]: 8: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x558fab197197] 2024-03-24T00:24:28.950 INFO:journalctl@ceph.mds.d.smithi118.stdout:Mar 24 00:24:28 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-d[96538]: 9: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x558fab09611b] 2024-03-24T00:24:28.950 INFO:journalctl@ceph.mds.d.smithi118.stdout:Mar 24 00:24:28 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-d[96538]: 10: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x558fab0966bc] 2024-03-24T00:24:28.950 INFO:journalctl@ceph.mds.d.smithi118.stdout:Mar 24 00:24:28 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-d[96538]: 11: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x558fab0801ff] 2024-03-24T00:24:28.950 INFO:journalctl@ceph.mds.d.smithi118.stdout:Mar 24 00:24:28 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-d[96538]: 12: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7f052a3b1488] 2024-03-24T00:24:28.950 INFO:journalctl@ceph.mds.d.smithi118.stdout:Mar 24 00:24:28 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-d[96538]: 13: (DispatchQueue::entry()+0x51f) [0x7f052a3ae60f] 2024-03-24T00:24:28.950 INFO:journalctl@ceph.mds.d.smithi118.stdout:Mar 24 00:24:28 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-d[96538]: 14: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f052a481e81] 2024-03-24T00:24:28.950 INFO:journalctl@ceph.mds.d.smithi118.stdout:Mar 24 00:24:28 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-d[96538]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7f0528e9c1ca] 2024-03-24T00:24:28.950 INFO:journalctl@ceph.mds.d.smithi118.stdout:Mar 24 00:24:28 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-d[96538]: 16: clone() 2024-03-24T00:24:28.951 INFO:journalctl@ceph.mds.j.smithi118.stdout:Mar 24 00:24:28 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-j[96574]: 2024-03-24T00:24:28.491+0000 7f1aeae00700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: In function 'void Locker::set_xlocks_done(MutationImpl*, bool)' thread 7f1aeae00700 time 2024-03-24T00:24:28.489891+0000 2024-03-24T00:24:28.951 INFO:journalctl@ceph.mds.j.smithi118.stdout:Mar 24 00:24:28 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-j[96574]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: 626: FAILED ceph_assert(obj->is_auth()) 2024-03-24T00:24:28.951 INFO:journalctl@ceph.mds.j.smithi118.stdout:Mar 24 00:24:28 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-j[96574]: 2024-03-24T00:24:28.951 INFO:journalctl@ceph.mds.j.smithi118.stdout:Mar 24 00:24:28 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-j[96574]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-24T00:24:28.951 INFO:journalctl@ceph.mds.j.smithi118.stdout:Mar 24 00:24:28 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-j[96574]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x135) [0x7f1af67dd4a1] 2024-03-24T00:24:28.951 INFO:journalctl@ceph.mds.j.smithi118.stdout:Mar 24 00:24:28 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-j[96574]: 2: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7f1af67dd667] 2024-03-24T00:24:28.951 INFO:journalctl@ceph.mds.j.smithi118.stdout:Mar 24 00:24:28 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-j[96574]: 3: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x55948aee0223] 2024-03-24T00:24:28.951 INFO:journalctl@ceph.mds.j.smithi118.stdout:Mar 24 00:24:28 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-j[96574]: 4: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x55948ad5a5d6] 2024-03-24T00:24:28.951 INFO:journalctl@ceph.mds.j.smithi118.stdout:Mar 24 00:24:28 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-j[96574]: 5: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x55948ad5ada1] 2024-03-24T00:24:28.952 INFO:journalctl@ceph.mds.j.smithi118.stdout:Mar 24 00:24:28 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-j[96574]: 6: (Server::handle_client_rename(boost::intrusive_ptr const&)+0x259d) [0x55948ad9797d] 2024-03-24T00:24:28.952 INFO:journalctl@ceph.mds.j.smithi118.stdout:Mar 24 00:24:28 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-j[96574]: 7: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x55948adad233] 2024-03-24T00:24:28.952 INFO:journalctl@ceph.mds.j.smithi118.stdout:Mar 24 00:24:28 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-j[96574]: 8: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x55948adaf197] 2024-03-24T00:24:28.952 INFO:journalctl@ceph.mds.j.smithi118.stdout:Mar 24 00:24:28 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-j[96574]: 9: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x55948acae11b] 2024-03-24T00:24:28.952 INFO:journalctl@ceph.mds.j.smithi118.stdout:Mar 24 00:24:28 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-j[96574]: 10: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x55948acae6bc] 2024-03-24T00:24:28.952 INFO:journalctl@ceph.mds.j.smithi118.stdout:Mar 24 00:24:28 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-j[96574]: 11: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x55948ac981ff] 2024-03-24T00:24:28.952 INFO:journalctl@ceph.mds.j.smithi118.stdout:Mar 24 00:24:28 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-j[96574]: 12: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7f1af6a6f488] 2024-03-24T00:24:28.952 INFO:journalctl@ceph.mds.j.smithi118.stdout:Mar 24 00:24:28 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-j[96574]: 13: (DispatchQueue::entry()+0x51f) [0x7f1af6a6c60f] 2024-03-24T00:24:28.952 INFO:journalctl@ceph.mds.j.smithi118.stdout:Mar 24 00:24:28 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-j[96574]: 14: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f1af6b3fe81] 2024-03-24T00:24:28.952 INFO:journalctl@ceph.mds.j.smithi118.stdout:Mar 24 00:24:28 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-j[96574]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7f1af555a1ca] 2024-03-24T00:24:28.952 INFO:journalctl@ceph.mds.j.smithi118.stdout:Mar 24 00:24:28 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-j[96574]: 16: clone() 2024-03-24T00:24:28.953 INFO:journalctl@ceph.mds.j.smithi118.stdout:Mar 24 00:24:28 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-j[96574]: 2024-03-24T00:24:28.953 INFO:journalctl@ceph.mds.j.smithi118.stdout:Mar 24 00:24:28 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-j[96574]: *** Caught signal (Aborted) ** 2024-03-24T00:24:28.953 INFO:journalctl@ceph.mds.j.smithi118.stdout:Mar 24 00:24:28 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-j[96574]: in thread 7f1aeae00700 thread_name:ms_dispatch 2024-03-24T00:24:28.953 INFO:journalctl@ceph.mds.j.smithi118.stdout:Mar 24 00:24:28 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-j[96574]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-24T00:24:28.953 INFO:journalctl@ceph.mds.j.smithi118.stdout:Mar 24 00:24:28 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-j[96574]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f1af5564d20] 2024-03-24T00:24:28.953 INFO:journalctl@ceph.mds.d.smithi118.stdout:Mar 24 00:24:28 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-d[96538]: 2024-03-24T00:24:28.541+0000 7f051e800700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: In function 'void Locker::set_xlocks_done(MutationImpl*, bool)' thread 7f051e800700 time 2024-03-24T00:24:28.539897+0000 2024-03-24T00:24:28.953 INFO:journalctl@ceph.mds.d.smithi118.stdout:Mar 24 00:24:28 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-d[96538]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: 626: FAILED ceph_assert(obj->is_auth()) 2024-03-24T00:24:28.953 INFO:journalctl@ceph.mds.d.smithi118.stdout:Mar 24 00:24:28 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-d[96538]: 2024-03-24T00:24:28.953 INFO:journalctl@ceph.mds.d.smithi118.stdout:Mar 24 00:24:28 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-d[96538]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-24T00:24:28.954 INFO:journalctl@ceph.mds.d.smithi118.stdout:Mar 24 00:24:28 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-d[96538]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x135) [0x7f052a11f4a1] 2024-03-24T00:24:28.954 INFO:journalctl@ceph.mds.d.smithi118.stdout:Mar 24 00:24:28 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-d[96538]: 2: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7f052a11f667] 2024-03-24T00:24:28.954 INFO:journalctl@ceph.mds.d.smithi118.stdout:Mar 24 00:24:28 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-d[96538]: 3: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x558fab2c8223] 2024-03-24T00:24:28.954 INFO:journalctl@ceph.mds.d.smithi118.stdout:Mar 24 00:24:28 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-d[96538]: 4: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x558fab1425d6] 2024-03-24T00:24:28.954 INFO:journalctl@ceph.mds.d.smithi118.stdout:Mar 24 00:24:28 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-d[96538]: 5: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x558fab142da1] 2024-03-24T00:24:28.954 INFO:journalctl@ceph.mds.d.smithi118.stdout:Mar 24 00:24:28 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-d[96538]: 6: (Server::handle_client_rename(boost::intrusive_ptr const&)+0x259d) [0x558fab17f97d] 2024-03-24T00:24:28.954 INFO:journalctl@ceph.mds.d.smithi118.stdout:Mar 24 00:24:28 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-d[96538]: 7: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x558fab195233] 2024-03-24T00:24:28.954 INFO:journalctl@ceph.mds.d.smithi118.stdout:Mar 24 00:24:28 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-d[96538]: 8: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x558fab197197] 2024-03-24T00:24:28.954 INFO:journalctl@ceph.mds.d.smithi118.stdout:Mar 24 00:24:28 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-d[96538]: 9: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x558fab09611b] 2024-03-24T00:24:28.954 INFO:journalctl@ceph.mds.d.smithi118.stdout:Mar 24 00:24:28 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-d[96538]: 10: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x558fab0966bc] 2024-03-24T00:24:28.954 INFO:journalctl@ceph.mds.d.smithi118.stdout:Mar 24 00:24:28 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-d[96538]: 11: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x558fab0801ff] 2024-03-24T00:24:28.954 INFO:journalctl@ceph.mds.d.smithi118.stdout:Mar 24 00:24:28 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-d[96538]: 12: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7f052a3b1488] 2024-03-24T00:24:28.954 INFO:journalctl@ceph.mds.d.smithi118.stdout:Mar 24 00:24:28 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-d[96538]: 13: (DispatchQueue::entry()+0x51f) [0x7f052a3ae60f] 2024-03-24T00:24:28.955 INFO:journalctl@ceph.mds.d.smithi118.stdout:Mar 24 00:24:28 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-d[96538]: 14: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f052a481e81] 2024-03-24T00:24:28.955 INFO:journalctl@ceph.mds.d.smithi118.stdout:Mar 24 00:24:28 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-d[96538]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7f0528e9c1ca] 2024-03-24T00:24:28.955 INFO:journalctl@ceph.mds.d.smithi118.stdout:Mar 24 00:24:28 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-d[96538]: 16: clone() 2024-03-24T00:24:28.955 INFO:journalctl@ceph.mds.d.smithi118.stdout:Mar 24 00:24:28 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-d[96538]: 2024-03-24T00:24:28.955 INFO:journalctl@ceph.mds.d.smithi118.stdout:Mar 24 00:24:28 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-d[96538]: *** Caught signal (Aborted) ** 2024-03-24T00:24:28.955 INFO:journalctl@ceph.mds.d.smithi118.stdout:Mar 24 00:24:28 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-d[96538]: in thread 7f051e800700 thread_name:ms_dispatch 2024-03-24T00:24:28.955 INFO:journalctl@ceph.mds.d.smithi118.stdout:Mar 24 00:24:28 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-d[96538]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-24T00:24:28.955 INFO:journalctl@ceph.mds.d.smithi118.stdout:Mar 24 00:24:28 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-d[96538]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f0528ea6d20] 2024-03-24T00:24:28.955 INFO:journalctl@ceph.mds.j.smithi118.stdout:Mar 24 00:24:28 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-j[96574]: 2: gsignal() 2024-03-24T00:24:28.956 INFO:journalctl@ceph.mds.j.smithi118.stdout:Mar 24 00:24:28 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-j[96574]: 3: abort() 2024-03-24T00:24:28.956 INFO:journalctl@ceph.mds.j.smithi118.stdout:Mar 24 00:24:28 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-j[96574]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x18f) [0x7f1af67dd4fb] 2024-03-24T00:24:28.956 INFO:journalctl@ceph.mds.j.smithi118.stdout:Mar 24 00:24:28 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-j[96574]: 5: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7f1af67dd667] 2024-03-24T00:24:28.956 INFO:journalctl@ceph.mds.j.smithi118.stdout:Mar 24 00:24:28 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-j[96574]: 6: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x55948aee0223] 2024-03-24T00:24:28.956 INFO:journalctl@ceph.mds.j.smithi118.stdout:Mar 24 00:24:28 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-j[96574]: 7: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x55948ad5a5d6] 2024-03-24T00:24:28.956 INFO:journalctl@ceph.mds.j.smithi118.stdout:Mar 24 00:24:28 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-j[96574]: 8: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x55948ad5ada1] 2024-03-24T00:24:28.956 INFO:journalctl@ceph.mds.j.smithi118.stdout:Mar 24 00:24:28 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-j[96574]: 9: (Server::handle_client_rename(boost::intrusive_ptr const&)+0x259d) [0x55948ad9797d] 2024-03-24T00:24:28.956 INFO:journalctl@ceph.mds.j.smithi118.stdout:Mar 24 00:24:28 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-j[96574]: 10: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x55948adad233] 2024-03-24T00:24:28.956 INFO:journalctl@ceph.mds.j.smithi118.stdout:Mar 24 00:24:28 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-j[96574]: 11: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x55948adaf197] 2024-03-24T00:24:28.956 INFO:journalctl@ceph.mds.j.smithi118.stdout:Mar 24 00:24:28 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-j[96574]: 12: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x55948acae11b] 2024-03-24T00:24:28.956 INFO:journalctl@ceph.mds.j.smithi118.stdout:Mar 24 00:24:28 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-j[96574]: 13: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x55948acae6bc] 2024-03-24T00:24:28.956 INFO:journalctl@ceph.mds.j.smithi118.stdout:Mar 24 00:24:28 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-j[96574]: 14: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x55948ac981ff] 2024-03-24T00:24:28.957 INFO:journalctl@ceph.mds.j.smithi118.stdout:Mar 24 00:24:28 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-j[96574]: 15: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7f1af6a6f488] 2024-03-24T00:24:28.957 INFO:journalctl@ceph.mds.j.smithi118.stdout:Mar 24 00:24:28 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-j[96574]: 16: (DispatchQueue::entry()+0x51f) [0x7f1af6a6c60f] 2024-03-24T00:24:28.957 INFO:journalctl@ceph.mds.j.smithi118.stdout:Mar 24 00:24:28 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-j[96574]: 17: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f1af6b3fe81] 2024-03-24T00:24:28.957 INFO:journalctl@ceph.mds.j.smithi118.stdout:Mar 24 00:24:28 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-j[96574]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7f1af555a1ca] 2024-03-24T00:24:28.957 INFO:journalctl@ceph.mds.j.smithi118.stdout:Mar 24 00:24:28 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-j[96574]: 19: clone() 2024-03-24T00:24:28.957 INFO:journalctl@ceph.mds.j.smithi118.stdout:Mar 24 00:24:28 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-j[96574]: 2024-03-24T00:24:28.494+0000 7f1aeae00700 -1 *** Caught signal (Aborted) ** 2024-03-24T00:24:28.957 INFO:journalctl@ceph.mds.j.smithi118.stdout:Mar 24 00:24:28 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-j[96574]: in thread 7f1aeae00700 thread_name:ms_dispatch 2024-03-24T00:24:28.957 INFO:journalctl@ceph.mds.j.smithi118.stdout:Mar 24 00:24:28 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-j[96574]: 2024-03-24T00:24:28.957 INFO:journalctl@ceph.mds.j.smithi118.stdout:Mar 24 00:24:28 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-j[96574]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-24T00:24:28.957 INFO:journalctl@ceph.mds.j.smithi118.stdout:Mar 24 00:24:28 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-j[96574]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f1af5564d20] 2024-03-24T00:24:28.957 INFO:journalctl@ceph.mds.j.smithi118.stdout:Mar 24 00:24:28 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-j[96574]: 2: gsignal() 2024-03-24T00:24:28.957 INFO:journalctl@ceph.mds.j.smithi118.stdout:Mar 24 00:24:28 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-j[96574]: 3: abort() 2024-03-24T00:24:28.957 INFO:journalctl@ceph.mds.j.smithi118.stdout:Mar 24 00:24:28 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-j[96574]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x18f) [0x7f1af67dd4fb] 2024-03-24T00:24:28.958 INFO:journalctl@ceph.mds.j.smithi118.stdout:Mar 24 00:24:28 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-j[96574]: 5: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7f1af67dd667] 2024-03-24T00:24:28.958 INFO:journalctl@ceph.mds.j.smithi118.stdout:Mar 24 00:24:28 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-j[96574]: 6: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x55948aee0223] 2024-03-24T00:24:28.958 INFO:journalctl@ceph.mds.j.smithi118.stdout:Mar 24 00:24:28 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-j[96574]: 7: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x55948ad5a5d6] 2024-03-24T00:24:28.958 INFO:journalctl@ceph.mds.j.smithi118.stdout:Mar 24 00:24:28 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-j[96574]: 8: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x55948ad5ada1] 2024-03-24T00:24:28.958 INFO:journalctl@ceph.mds.j.smithi118.stdout:Mar 24 00:24:28 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-j[96574]: 9: (Server::handle_client_rename(boost::intrusive_ptr const&)+0x259d) [0x55948ad9797d] 2024-03-24T00:24:28.958 INFO:journalctl@ceph.mds.j.smithi118.stdout:Mar 24 00:24:28 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-j[96574]: 10: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x55948adad233] 2024-03-24T00:24:28.958 INFO:journalctl@ceph.mds.j.smithi118.stdout:Mar 24 00:24:28 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-j[96574]: 11: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x55948adaf197] 2024-03-24T00:24:28.958 INFO:journalctl@ceph.mds.j.smithi118.stdout:Mar 24 00:24:28 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-j[96574]: 12: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x55948acae11b] 2024-03-24T00:24:28.958 INFO:journalctl@ceph.mds.j.smithi118.stdout:Mar 24 00:24:28 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-j[96574]: 13: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x55948acae6bc] 2024-03-24T00:24:28.958 INFO:journalctl@ceph.mds.j.smithi118.stdout:Mar 24 00:24:28 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-j[96574]: 14: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x55948ac981ff] 2024-03-24T00:24:28.958 INFO:journalctl@ceph.mds.j.smithi118.stdout:Mar 24 00:24:28 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-j[96574]: 15: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7f1af6a6f488] 2024-03-24T00:24:28.958 INFO:journalctl@ceph.mds.j.smithi118.stdout:Mar 24 00:24:28 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-j[96574]: 16: (DispatchQueue::entry()+0x51f) [0x7f1af6a6c60f] 2024-03-24T00:24:28.958 INFO:journalctl@ceph.mds.j.smithi118.stdout:Mar 24 00:24:28 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-j[96574]: 17: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f1af6b3fe81] 2024-03-24T00:24:28.958 INFO:journalctl@ceph.mds.j.smithi118.stdout:Mar 24 00:24:28 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-j[96574]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7f1af555a1ca] 2024-03-24T00:24:28.958 INFO:journalctl@ceph.mds.j.smithi118.stdout:Mar 24 00:24:28 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-j[96574]: 19: clone() 2024-03-24T00:24:28.959 INFO:journalctl@ceph.mds.j.smithi118.stdout:Mar 24 00:24:28 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-j[96574]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-03-24T00:24:28.959 INFO:journalctl@ceph.mds.j.smithi118.stdout:Mar 24 00:24:28 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-j[96574]: 2024-03-24T00:24:28.959 INFO:journalctl@ceph.mds.j.smithi118.stdout:Mar 24 00:24:28 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-j[96574]: -1> 2024-03-24T00:24:28.491+0000 7f1aeae00700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: In function 'void Locker::set_xlocks_done(MutationImpl*, bool)' thread 7f1aeae00700 time 2024-03-24T00:24:28.489891+0000 2024-03-24T00:24:28.959 INFO:journalctl@ceph.mds.j.smithi118.stdout:Mar 24 00:24:28 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-j[96574]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: 626: FAILED ceph_assert(obj->is_auth()) 2024-03-24T00:24:28.959 INFO:journalctl@ceph.mds.j.smithi118.stdout:Mar 24 00:24:28 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-j[96574]: 2024-03-24T00:24:28.959 INFO:journalctl@ceph.mds.j.smithi118.stdout:Mar 24 00:24:28 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-j[96574]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-24T00:24:28.959 INFO:journalctl@ceph.mds.j.smithi118.stdout:Mar 24 00:24:28 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-j[96574]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x135) [0x7f1af67dd4a1] 2024-03-24T00:24:28.959 INFO:journalctl@ceph.mds.j.smithi118.stdout:Mar 24 00:24:28 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-j[96574]: 2: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7f1af67dd667] 2024-03-24T00:24:28.959 INFO:journalctl@ceph.mds.d.smithi118.stdout:Mar 24 00:24:28 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-d[96538]: 2: gsignal() 2024-03-24T00:24:28.959 INFO:journalctl@ceph.mds.d.smithi118.stdout:Mar 24 00:24:28 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-d[96538]: 3: abort() 2024-03-24T00:24:28.960 INFO:journalctl@ceph.mds.d.smithi118.stdout:Mar 24 00:24:28 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-d[96538]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x18f) [0x7f052a11f4fb] 2024-03-24T00:24:28.960 INFO:journalctl@ceph.mds.d.smithi118.stdout:Mar 24 00:24:28 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-d[96538]: 5: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7f052a11f667] 2024-03-24T00:24:28.960 INFO:journalctl@ceph.mds.d.smithi118.stdout:Mar 24 00:24:28 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-d[96538]: 6: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x558fab2c8223] 2024-03-24T00:24:28.960 INFO:journalctl@ceph.mds.d.smithi118.stdout:Mar 24 00:24:28 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-d[96538]: 7: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x558fab1425d6] 2024-03-24T00:24:28.960 INFO:journalctl@ceph.mds.d.smithi118.stdout:Mar 24 00:24:28 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-d[96538]: 8: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x558fab142da1] 2024-03-24T00:24:28.960 INFO:journalctl@ceph.mds.d.smithi118.stdout:Mar 24 00:24:28 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-d[96538]: 9: (Server::handle_client_rename(boost::intrusive_ptr const&)+0x259d) [0x558fab17f97d] 2024-03-24T00:24:28.960 INFO:journalctl@ceph.mds.d.smithi118.stdout:Mar 24 00:24:28 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-d[96538]: 10: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x558fab195233] 2024-03-24T00:24:28.960 INFO:journalctl@ceph.mds.d.smithi118.stdout:Mar 24 00:24:28 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-d[96538]: 11: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x558fab197197] 2024-03-24T00:24:28.960 INFO:journalctl@ceph.mds.d.smithi118.stdout:Mar 24 00:24:28 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-d[96538]: 12: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x558fab09611b] 2024-03-24T00:24:28.960 INFO:journalctl@ceph.mds.d.smithi118.stdout:Mar 24 00:24:28 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-d[96538]: 13: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x558fab0966bc] 2024-03-24T00:24:28.960 INFO:journalctl@ceph.mds.d.smithi118.stdout:Mar 24 00:24:28 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-d[96538]: 14: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x558fab0801ff] 2024-03-24T00:24:28.960 INFO:journalctl@ceph.mds.d.smithi118.stdout:Mar 24 00:24:28 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-d[96538]: 15: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7f052a3b1488] 2024-03-24T00:24:28.960 INFO:journalctl@ceph.mds.d.smithi118.stdout:Mar 24 00:24:28 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-d[96538]: 16: (DispatchQueue::entry()+0x51f) [0x7f052a3ae60f] 2024-03-24T00:24:28.960 INFO:journalctl@ceph.mds.d.smithi118.stdout:Mar 24 00:24:28 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-d[96538]: 17: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f052a481e81] 2024-03-24T00:24:28.961 INFO:journalctl@ceph.mds.d.smithi118.stdout:Mar 24 00:24:28 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-d[96538]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7f0528e9c1ca] 2024-03-24T00:24:28.961 INFO:journalctl@ceph.mds.d.smithi118.stdout:Mar 24 00:24:28 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-d[96538]: 19: clone() 2024-03-24T00:24:28.961 INFO:journalctl@ceph.mds.d.smithi118.stdout:Mar 24 00:24:28 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-d[96538]: 2024-03-24T00:24:28.543+0000 7f051e800700 -1 *** Caught signal (Aborted) ** 2024-03-24T00:24:28.961 INFO:journalctl@ceph.mds.d.smithi118.stdout:Mar 24 00:24:28 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-d[96538]: in thread 7f051e800700 thread_name:ms_dispatch 2024-03-24T00:24:28.961 INFO:journalctl@ceph.mds.d.smithi118.stdout:Mar 24 00:24:28 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-d[96538]: 2024-03-24T00:24:28.961 INFO:journalctl@ceph.mds.d.smithi118.stdout:Mar 24 00:24:28 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-d[96538]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-24T00:24:28.961 INFO:journalctl@ceph.mds.d.smithi118.stdout:Mar 24 00:24:28 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-d[96538]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f0528ea6d20] 2024-03-24T00:24:28.961 INFO:journalctl@ceph.mds.d.smithi118.stdout:Mar 24 00:24:28 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-d[96538]: 2: gsignal() 2024-03-24T00:24:28.961 INFO:journalctl@ceph.mds.d.smithi118.stdout:Mar 24 00:24:28 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-d[96538]: 3: abort() 2024-03-24T00:24:28.961 INFO:journalctl@ceph.mds.d.smithi118.stdout:Mar 24 00:24:28 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-d[96538]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x18f) [0x7f052a11f4fb] 2024-03-24T00:24:28.961 INFO:journalctl@ceph.mds.d.smithi118.stdout:Mar 24 00:24:28 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-d[96538]: 5: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7f052a11f667] 2024-03-24T00:24:28.962 INFO:journalctl@ceph.mds.j.smithi118.stdout:Mar 24 00:24:28 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-j[96574]: 3: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x55948aee0223] 2024-03-24T00:24:28.962 INFO:journalctl@ceph.mds.j.smithi118.stdout:Mar 24 00:24:28 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-j[96574]: 4: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x55948ad5a5d6] 2024-03-24T00:24:28.962 INFO:journalctl@ceph.mds.j.smithi118.stdout:Mar 24 00:24:28 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-j[96574]: 5: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x55948ad5ada1] 2024-03-24T00:24:28.962 INFO:journalctl@ceph.mds.j.smithi118.stdout:Mar 24 00:24:28 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-j[96574]: 6: (Server::handle_client_rename(boost::intrusive_ptr const&)+0x259d) [0x55948ad9797d] 2024-03-24T00:24:28.962 INFO:journalctl@ceph.mds.j.smithi118.stdout:Mar 24 00:24:28 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-j[96574]: 7: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x55948adad233] 2024-03-24T00:24:28.962 INFO:journalctl@ceph.mds.j.smithi118.stdout:Mar 24 00:24:28 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-j[96574]: 8: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x55948adaf197] 2024-03-24T00:24:28.962 INFO:journalctl@ceph.mds.j.smithi118.stdout:Mar 24 00:24:28 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-j[96574]: 9: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x55948acae11b] 2024-03-24T00:24:28.962 INFO:journalctl@ceph.mds.j.smithi118.stdout:Mar 24 00:24:28 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-j[96574]: 10: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x55948acae6bc] 2024-03-24T00:24:28.962 INFO:journalctl@ceph.mds.j.smithi118.stdout:Mar 24 00:24:28 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-j[96574]: 11: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x55948ac981ff] 2024-03-24T00:24:28.962 INFO:journalctl@ceph.mds.j.smithi118.stdout:Mar 24 00:24:28 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-j[96574]: 12: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7f1af6a6f488] 2024-03-24T00:24:28.962 INFO:journalctl@ceph.mds.j.smithi118.stdout:Mar 24 00:24:28 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-j[96574]: 13: (DispatchQueue::entry()+0x51f) [0x7f1af6a6c60f] 2024-03-24T00:24:28.962 INFO:journalctl@ceph.mds.j.smithi118.stdout:Mar 24 00:24:28 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-j[96574]: 14: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f1af6b3fe81] 2024-03-24T00:24:28.962 INFO:journalctl@ceph.mds.j.smithi118.stdout:Mar 24 00:24:28 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-j[96574]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7f1af555a1ca] 2024-03-24T00:24:28.963 INFO:journalctl@ceph.mds.j.smithi118.stdout:Mar 24 00:24:28 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-j[96574]: 16: clone() 2024-03-24T00:24:28.963 INFO:journalctl@ceph.mds.j.smithi118.stdout:Mar 24 00:24:28 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-j[96574]: 2024-03-24T00:24:28.963 INFO:journalctl@ceph.mds.j.smithi118.stdout:Mar 24 00:24:28 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-j[96574]: 0> 2024-03-24T00:24:28.494+0000 7f1aeae00700 -1 *** Caught signal (Aborted) ** 2024-03-24T00:24:28.963 INFO:journalctl@ceph.mds.j.smithi118.stdout:Mar 24 00:24:28 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-j[96574]: in thread 7f1aeae00700 thread_name:ms_dispatch 2024-03-24T00:24:28.963 INFO:journalctl@ceph.mds.j.smithi118.stdout:Mar 24 00:24:28 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-j[96574]: 2024-03-24T00:24:28.963 INFO:journalctl@ceph.mds.j.smithi118.stdout:Mar 24 00:24:28 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-j[96574]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-24T00:24:28.963 INFO:journalctl@ceph.mds.j.smithi118.stdout:Mar 24 00:24:28 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-j[96574]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f1af5564d20] 2024-03-24T00:24:28.963 INFO:journalctl@ceph.mds.j.smithi118.stdout:Mar 24 00:24:28 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-j[96574]: 2: gsignal() 2024-03-24T00:24:28.963 INFO:journalctl@ceph.mds.j.smithi118.stdout:Mar 24 00:24:28 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-j[96574]: 3: abort() 2024-03-24T00:24:28.963 INFO:journalctl@ceph.mds.j.smithi118.stdout:Mar 24 00:24:28 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-j[96574]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x18f) [0x7f1af67dd4fb] 2024-03-24T00:24:28.963 INFO:journalctl@ceph.mds.j.smithi118.stdout:Mar 24 00:24:28 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-j[96574]: 5: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7f1af67dd667] 2024-03-24T00:24:28.963 INFO:journalctl@ceph.mds.j.smithi118.stdout:Mar 24 00:24:28 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-j[96574]: 6: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x55948aee0223] 2024-03-24T00:24:28.963 INFO:journalctl@ceph.mds.j.smithi118.stdout:Mar 24 00:24:28 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-j[96574]: 7: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x55948ad5a5d6] 2024-03-24T00:24:28.964 INFO:journalctl@ceph.mds.d.smithi118.stdout:Mar 24 00:24:28 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-d[96538]: 6: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x558fab2c8223] 2024-03-24T00:24:28.964 INFO:journalctl@ceph.mds.d.smithi118.stdout:Mar 24 00:24:28 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-d[96538]: 7: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x558fab1425d6] 2024-03-24T00:24:28.964 INFO:journalctl@ceph.mds.d.smithi118.stdout:Mar 24 00:24:28 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-d[96538]: 8: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x558fab142da1] 2024-03-24T00:24:28.964 INFO:journalctl@ceph.mds.d.smithi118.stdout:Mar 24 00:24:28 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-d[96538]: 9: (Server::handle_client_rename(boost::intrusive_ptr const&)+0x259d) [0x558fab17f97d] 2024-03-24T00:24:28.964 INFO:journalctl@ceph.mds.d.smithi118.stdout:Mar 24 00:24:28 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-d[96538]: 10: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x558fab195233] 2024-03-24T00:24:28.964 INFO:journalctl@ceph.mds.d.smithi118.stdout:Mar 24 00:24:28 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-d[96538]: 11: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x558fab197197] 2024-03-24T00:24:28.964 INFO:journalctl@ceph.mds.d.smithi118.stdout:Mar 24 00:24:28 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-d[96538]: 12: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x558fab09611b] 2024-03-24T00:24:28.964 INFO:journalctl@ceph.mds.d.smithi118.stdout:Mar 24 00:24:28 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-d[96538]: 13: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x558fab0966bc] 2024-03-24T00:24:28.964 INFO:journalctl@ceph.mds.d.smithi118.stdout:Mar 24 00:24:28 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-d[96538]: 14: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x558fab0801ff] 2024-03-24T00:24:28.965 INFO:journalctl@ceph.mds.d.smithi118.stdout:Mar 24 00:24:28 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-d[96538]: 15: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7f052a3b1488] 2024-03-24T00:24:28.965 INFO:journalctl@ceph.mds.d.smithi118.stdout:Mar 24 00:24:28 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-d[96538]: 16: (DispatchQueue::entry()+0x51f) [0x7f052a3ae60f] 2024-03-24T00:24:28.965 INFO:journalctl@ceph.mds.d.smithi118.stdout:Mar 24 00:24:28 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-d[96538]: 17: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f052a481e81] 2024-03-24T00:24:28.965 INFO:journalctl@ceph.mds.d.smithi118.stdout:Mar 24 00:24:28 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-d[96538]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7f0528e9c1ca] 2024-03-24T00:24:28.965 INFO:journalctl@ceph.mds.d.smithi118.stdout:Mar 24 00:24:28 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-d[96538]: 19: clone() 2024-03-24T00:24:28.965 INFO:journalctl@ceph.mds.d.smithi118.stdout:Mar 24 00:24:28 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-d[96538]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-03-24T00:24:28.965 INFO:journalctl@ceph.mds.d.smithi118.stdout:Mar 24 00:24:28 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-d[96538]: 2024-03-24T00:24:28.965 INFO:journalctl@ceph.mds.d.smithi118.stdout:Mar 24 00:24:28 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-d[96538]: -1> 2024-03-24T00:24:28.541+0000 7f051e800700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: In function 'void Locker::set_xlocks_done(MutationImpl*, bool)' thread 7f051e800700 time 2024-03-24T00:24:28.539897+0000 2024-03-24T00:24:28.965 INFO:journalctl@ceph.mds.d.smithi118.stdout:Mar 24 00:24:28 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-d[96538]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: 626: FAILED ceph_assert(obj->is_auth()) 2024-03-24T00:24:28.965 INFO:journalctl@ceph.mds.d.smithi118.stdout:Mar 24 00:24:28 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-d[96538]: 2024-03-24T00:24:28.965 INFO:journalctl@ceph.mds.d.smithi118.stdout:Mar 24 00:24:28 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-d[96538]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-24T00:24:28.965 INFO:journalctl@ceph.mds.d.smithi118.stdout:Mar 24 00:24:28 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-d[96538]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x135) [0x7f052a11f4a1] 2024-03-24T00:24:28.965 INFO:journalctl@ceph.mds.d.smithi118.stdout:Mar 24 00:24:28 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-d[96538]: 2: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7f052a11f667] 2024-03-24T00:24:28.966 INFO:journalctl@ceph.mds.j.smithi118.stdout:Mar 24 00:24:28 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-j[96574]: 8: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x55948ad5ada1] 2024-03-24T00:24:28.966 INFO:journalctl@ceph.mds.j.smithi118.stdout:Mar 24 00:24:28 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-j[96574]: 9: (Server::handle_client_rename(boost::intrusive_ptr const&)+0x259d) [0x55948ad9797d] 2024-03-24T00:24:28.966 INFO:journalctl@ceph.mds.j.smithi118.stdout:Mar 24 00:24:28 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-j[96574]: 10: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x55948adad233] 2024-03-24T00:24:28.966 INFO:journalctl@ceph.mds.j.smithi118.stdout:Mar 24 00:24:28 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-j[96574]: 11: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x55948adaf197] 2024-03-24T00:24:28.966 INFO:journalctl@ceph.mds.j.smithi118.stdout:Mar 24 00:24:28 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-j[96574]: 12: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x55948acae11b] 2024-03-24T00:24:28.966 INFO:journalctl@ceph.mds.j.smithi118.stdout:Mar 24 00:24:28 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-j[96574]: 13: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x55948acae6bc] 2024-03-24T00:24:28.966 INFO:journalctl@ceph.mds.j.smithi118.stdout:Mar 24 00:24:28 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-j[96574]: 14: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x55948ac981ff] 2024-03-24T00:24:28.966 INFO:journalctl@ceph.mds.j.smithi118.stdout:Mar 24 00:24:28 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-j[96574]: 15: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7f1af6a6f488] 2024-03-24T00:24:28.966 INFO:journalctl@ceph.mds.j.smithi118.stdout:Mar 24 00:24:28 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-j[96574]: 16: (DispatchQueue::entry()+0x51f) [0x7f1af6a6c60f] 2024-03-24T00:24:28.967 INFO:journalctl@ceph.mds.j.smithi118.stdout:Mar 24 00:24:28 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-j[96574]: 17: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f1af6b3fe81] 2024-03-24T00:24:28.967 INFO:journalctl@ceph.mds.j.smithi118.stdout:Mar 24 00:24:28 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-j[96574]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7f1af555a1ca] 2024-03-24T00:24:28.967 INFO:journalctl@ceph.mds.j.smithi118.stdout:Mar 24 00:24:28 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-j[96574]: 19: clone() 2024-03-24T00:24:28.967 INFO:journalctl@ceph.mds.j.smithi118.stdout:Mar 24 00:24:28 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-j[96574]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-03-24T00:24:28.967 INFO:journalctl@ceph.mds.j.smithi118.stdout:Mar 24 00:24:28 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-j[96574]: 2024-03-24T00:24:28.967 INFO:journalctl@ceph.mds.j.smithi118.stdout:Mar 24 00:24:28 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-j[96574]: -1> 2024-03-24T00:24:28.491+0000 7f1aeae00700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: In function 'void Locker::set_xlocks_done(MutationImpl*, bool)' thread 7f1aeae00700 time 2024-03-24T00:24:28.489891+0000 2024-03-24T00:24:28.967 INFO:journalctl@ceph.mds.j.smithi118.stdout:Mar 24 00:24:28 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-j[96574]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: 626: FAILED ceph_assert(obj->is_auth()) 2024-03-24T00:24:28.967 INFO:journalctl@ceph.mds.j.smithi118.stdout:Mar 24 00:24:28 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-j[96574]: 2024-03-24T00:24:28.967 INFO:journalctl@ceph.mds.j.smithi118.stdout:Mar 24 00:24:28 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-j[96574]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-24T00:24:28.967 INFO:journalctl@ceph.mds.j.smithi118.stdout:Mar 24 00:24:28 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-j[96574]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x135) [0x7f1af67dd4a1] 2024-03-24T00:24:28.967 INFO:journalctl@ceph.mds.j.smithi118.stdout:Mar 24 00:24:28 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-j[96574]: 2: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7f1af67dd667] 2024-03-24T00:24:28.968 INFO:journalctl@ceph.mds.j.smithi118.stdout:Mar 24 00:24:28 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-j[96574]: 3: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x55948aee0223] 2024-03-24T00:24:28.968 INFO:journalctl@ceph.mds.j.smithi118.stdout:Mar 24 00:24:28 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-j[96574]: 4: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x55948ad5a5d6] 2024-03-24T00:24:28.968 INFO:journalctl@ceph.mds.d.smithi118.stdout:Mar 24 00:24:28 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-d[96538]: 3: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x558fab2c8223] 2024-03-24T00:24:28.968 INFO:journalctl@ceph.mds.d.smithi118.stdout:Mar 24 00:24:28 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-d[96538]: 4: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x558fab1425d6] 2024-03-24T00:24:28.968 INFO:journalctl@ceph.mds.d.smithi118.stdout:Mar 24 00:24:28 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-d[96538]: 5: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x558fab142da1] 2024-03-24T00:24:28.968 INFO:journalctl@ceph.mds.d.smithi118.stdout:Mar 24 00:24:28 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-d[96538]: 6: (Server::handle_client_rename(boost::intrusive_ptr const&)+0x259d) [0x558fab17f97d] 2024-03-24T00:24:28.968 INFO:journalctl@ceph.mds.d.smithi118.stdout:Mar 24 00:24:28 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-d[96538]: 7: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x558fab195233] 2024-03-24T00:24:28.968 INFO:journalctl@ceph.mds.d.smithi118.stdout:Mar 24 00:24:28 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-d[96538]: 8: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x558fab197197] 2024-03-24T00:24:28.968 INFO:journalctl@ceph.mds.d.smithi118.stdout:Mar 24 00:24:28 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-d[96538]: 9: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x558fab09611b] 2024-03-24T00:24:28.968 INFO:journalctl@ceph.mds.d.smithi118.stdout:Mar 24 00:24:28 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-d[96538]: 10: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x558fab0966bc] 2024-03-24T00:24:28.969 INFO:journalctl@ceph.mds.d.smithi118.stdout:Mar 24 00:24:28 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-d[96538]: 11: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x558fab0801ff] 2024-03-24T00:24:28.969 INFO:journalctl@ceph.mds.d.smithi118.stdout:Mar 24 00:24:28 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-d[96538]: 12: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7f052a3b1488] 2024-03-24T00:24:28.969 INFO:journalctl@ceph.mds.d.smithi118.stdout:Mar 24 00:24:28 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-d[96538]: 13: (DispatchQueue::entry()+0x51f) [0x7f052a3ae60f] 2024-03-24T00:24:28.969 INFO:journalctl@ceph.mds.d.smithi118.stdout:Mar 24 00:24:28 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-d[96538]: 14: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f052a481e81] 2024-03-24T00:24:28.969 INFO:journalctl@ceph.mds.d.smithi118.stdout:Mar 24 00:24:28 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-d[96538]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7f0528e9c1ca] 2024-03-24T00:24:28.969 INFO:journalctl@ceph.mds.d.smithi118.stdout:Mar 24 00:24:28 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-d[96538]: 16: clone() 2024-03-24T00:24:28.969 INFO:journalctl@ceph.mds.d.smithi118.stdout:Mar 24 00:24:28 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-d[96538]: 2024-03-24T00:24:28.969 INFO:journalctl@ceph.mds.d.smithi118.stdout:Mar 24 00:24:28 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-d[96538]: 0> 2024-03-24T00:24:28.543+0000 7f051e800700 -1 *** Caught signal (Aborted) ** 2024-03-24T00:24:28.969 INFO:journalctl@ceph.mds.d.smithi118.stdout:Mar 24 00:24:28 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-d[96538]: in thread 7f051e800700 thread_name:ms_dispatch 2024-03-24T00:24:28.969 INFO:journalctl@ceph.mds.d.smithi118.stdout:Mar 24 00:24:28 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-d[96538]: 2024-03-24T00:24:28.969 INFO:journalctl@ceph.mds.d.smithi118.stdout:Mar 24 00:24:28 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-d[96538]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-24T00:24:28.969 INFO:journalctl@ceph.mds.d.smithi118.stdout:Mar 24 00:24:28 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-d[96538]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f0528ea6d20] 2024-03-24T00:24:28.970 INFO:journalctl@ceph.mds.d.smithi118.stdout:Mar 24 00:24:28 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-d[96538]: 2: gsignal() 2024-03-24T00:24:28.970 INFO:journalctl@ceph.mds.d.smithi118.stdout:Mar 24 00:24:28 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-d[96538]: 3: abort() 2024-03-24T00:24:28.970 INFO:journalctl@ceph.mds.d.smithi118.stdout:Mar 24 00:24:28 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-d[96538]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x18f) [0x7f052a11f4fb] 2024-03-24T00:24:28.970 INFO:journalctl@ceph.mds.d.smithi118.stdout:Mar 24 00:24:28 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-d[96538]: 5: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7f052a11f667] 2024-03-24T00:24:28.970 INFO:journalctl@ceph.mds.d.smithi118.stdout:Mar 24 00:24:28 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-d[96538]: 6: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x558fab2c8223] 2024-03-24T00:24:28.970 INFO:journalctl@ceph.mds.d.smithi118.stdout:Mar 24 00:24:28 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-d[96538]: 7: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x558fab1425d6] 2024-03-24T00:24:28.970 INFO:journalctl@ceph.mds.j.smithi118.stdout:Mar 24 00:24:28 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-j[96574]: 5: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x55948ad5ada1] 2024-03-24T00:24:28.971 INFO:journalctl@ceph.mds.j.smithi118.stdout:Mar 24 00:24:28 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-j[96574]: 6: (Server::handle_client_rename(boost::intrusive_ptr const&)+0x259d) [0x55948ad9797d] 2024-03-24T00:24:28.971 INFO:journalctl@ceph.mds.j.smithi118.stdout:Mar 24 00:24:28 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-j[96574]: 7: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x55948adad233] 2024-03-24T00:24:28.971 INFO:journalctl@ceph.mds.j.smithi118.stdout:Mar 24 00:24:28 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-j[96574]: 8: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x55948adaf197] 2024-03-24T00:24:28.971 INFO:journalctl@ceph.mds.j.smithi118.stdout:Mar 24 00:24:28 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-j[96574]: 9: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x55948acae11b] 2024-03-24T00:24:28.971 INFO:journalctl@ceph.mds.j.smithi118.stdout:Mar 24 00:24:28 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-j[96574]: 10: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x55948acae6bc] 2024-03-24T00:24:28.971 INFO:journalctl@ceph.mds.j.smithi118.stdout:Mar 24 00:24:28 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-j[96574]: 11: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x55948ac981ff] 2024-03-24T00:24:28.971 INFO:journalctl@ceph.mds.j.smithi118.stdout:Mar 24 00:24:28 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-j[96574]: 12: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7f1af6a6f488] 2024-03-24T00:24:28.971 INFO:journalctl@ceph.mds.j.smithi118.stdout:Mar 24 00:24:28 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-j[96574]: 13: (DispatchQueue::entry()+0x51f) [0x7f1af6a6c60f] 2024-03-24T00:24:28.971 INFO:journalctl@ceph.mds.j.smithi118.stdout:Mar 24 00:24:28 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-j[96574]: 14: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f1af6b3fe81] 2024-03-24T00:24:28.971 INFO:journalctl@ceph.mds.j.smithi118.stdout:Mar 24 00:24:28 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-j[96574]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7f1af555a1ca] 2024-03-24T00:24:28.971 INFO:journalctl@ceph.mds.j.smithi118.stdout:Mar 24 00:24:28 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-j[96574]: 16: clone() 2024-03-24T00:24:28.972 INFO:journalctl@ceph.mds.j.smithi118.stdout:Mar 24 00:24:28 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-j[96574]: 2024-03-24T00:24:28.972 INFO:journalctl@ceph.mds.j.smithi118.stdout:Mar 24 00:24:28 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-j[96574]: 0> 2024-03-24T00:24:28.494+0000 7f1aeae00700 -1 *** Caught signal (Aborted) ** 2024-03-24T00:24:28.972 INFO:journalctl@ceph.mds.j.smithi118.stdout:Mar 24 00:24:28 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-j[96574]: in thread 7f1aeae00700 thread_name:ms_dispatch 2024-03-24T00:24:28.972 INFO:journalctl@ceph.mds.j.smithi118.stdout:Mar 24 00:24:28 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-j[96574]: 2024-03-24T00:24:28.972 INFO:journalctl@ceph.mds.j.smithi118.stdout:Mar 24 00:24:28 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-j[96574]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-24T00:24:28.972 INFO:journalctl@ceph.mds.j.smithi118.stdout:Mar 24 00:24:28 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-j[96574]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f1af5564d20] 2024-03-24T00:24:28.972 INFO:journalctl@ceph.mds.j.smithi118.stdout:Mar 24 00:24:28 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-j[96574]: 2: gsignal() 2024-03-24T00:24:28.972 INFO:journalctl@ceph.mds.j.smithi118.stdout:Mar 24 00:24:28 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-j[96574]: 3: abort() 2024-03-24T00:24:28.972 INFO:journalctl@ceph.mds.j.smithi118.stdout:Mar 24 00:24:28 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-j[96574]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x18f) [0x7f1af67dd4fb] 2024-03-24T00:24:28.972 INFO:journalctl@ceph.mds.j.smithi118.stdout:Mar 24 00:24:28 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-j[96574]: 5: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7f1af67dd667] 2024-03-24T00:24:28.972 INFO:journalctl@ceph.mds.j.smithi118.stdout:Mar 24 00:24:28 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-j[96574]: 6: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x55948aee0223] 2024-03-24T00:24:28.972 INFO:journalctl@ceph.mds.j.smithi118.stdout:Mar 24 00:24:28 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-j[96574]: 7: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x55948ad5a5d6] 2024-03-24T00:24:28.972 INFO:journalctl@ceph.mds.j.smithi118.stdout:Mar 24 00:24:28 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-j[96574]: 8: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x55948ad5ada1] 2024-03-24T00:24:28.973 INFO:journalctl@ceph.mds.j.smithi118.stdout:Mar 24 00:24:28 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-j[96574]: 9: (Server::handle_client_rename(boost::intrusive_ptr const&)+0x259d) [0x55948ad9797d] 2024-03-24T00:24:28.973 INFO:journalctl@ceph.mds.j.smithi118.stdout:Mar 24 00:24:28 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-j[96574]: 10: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x55948adad233] 2024-03-24T00:24:28.973 INFO:journalctl@ceph.mds.j.smithi118.stdout:Mar 24 00:24:28 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-j[96574]: 11: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x55948adaf197] 2024-03-24T00:24:28.973 INFO:journalctl@ceph.mds.j.smithi118.stdout:Mar 24 00:24:28 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-j[96574]: 12: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x55948acae11b] 2024-03-24T00:24:28.973 INFO:journalctl@ceph.mds.j.smithi118.stdout:Mar 24 00:24:28 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-j[96574]: 13: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x55948acae6bc] 2024-03-24T00:24:28.973 INFO:journalctl@ceph.mds.j.smithi118.stdout:Mar 24 00:24:28 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-j[96574]: 14: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x55948ac981ff] 2024-03-24T00:24:28.973 INFO:journalctl@ceph.mds.j.smithi118.stdout:Mar 24 00:24:28 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-j[96574]: 15: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7f1af6a6f488] 2024-03-24T00:24:28.973 INFO:journalctl@ceph.mds.j.smithi118.stdout:Mar 24 00:24:28 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-j[96574]: 16: (DispatchQueue::entry()+0x51f) [0x7f1af6a6c60f] 2024-03-24T00:24:28.973 INFO:journalctl@ceph.mds.j.smithi118.stdout:Mar 24 00:24:28 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-j[96574]: 17: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f1af6b3fe81] 2024-03-24T00:24:28.973 INFO:journalctl@ceph.mds.j.smithi118.stdout:Mar 24 00:24:28 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-j[96574]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7f1af555a1ca] 2024-03-24T00:24:28.973 INFO:journalctl@ceph.mds.j.smithi118.stdout:Mar 24 00:24:28 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-j[96574]: 19: clone() 2024-03-24T00:24:28.973 INFO:journalctl@ceph.mds.j.smithi118.stdout:Mar 24 00:24:28 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-j[96574]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-03-24T00:24:28.973 INFO:journalctl@ceph.mds.j.smithi118.stdout:Mar 24 00:24:28 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-j[96574]: 2024-03-24T00:24:28.973 INFO:journalctl@ceph.mds.j.smithi118.stdout:Mar 24 00:24:28 smithi118 podman[96617]: 2024-03-24 00:24:28.619756806 +0000 UTC m=+0.043339867 container died 7244d2bf20660b74adf0f39669b2b63aaf0e1df2c7ba2ec622c5b2864eba0601 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-j, org.label-schema.schema-version=1.0, ceph=True, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240319, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30) 2024-03-24T00:24:28.974 INFO:journalctl@ceph.mds.d.smithi118.stdout:Mar 24 00:24:28 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-d[96538]: 8: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x558fab142da1] 2024-03-24T00:24:28.974 INFO:journalctl@ceph.mds.d.smithi118.stdout:Mar 24 00:24:28 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-d[96538]: 9: (Server::handle_client_rename(boost::intrusive_ptr const&)+0x259d) [0x558fab17f97d] 2024-03-24T00:24:28.974 INFO:journalctl@ceph.mds.d.smithi118.stdout:Mar 24 00:24:28 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-d[96538]: 10: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x558fab195233] 2024-03-24T00:24:28.974 INFO:journalctl@ceph.mds.d.smithi118.stdout:Mar 24 00:24:28 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-d[96538]: 11: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x558fab197197] 2024-03-24T00:24:28.975 INFO:journalctl@ceph.mds.d.smithi118.stdout:Mar 24 00:24:28 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-d[96538]: 12: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x558fab09611b] 2024-03-24T00:24:28.975 INFO:journalctl@ceph.mds.d.smithi118.stdout:Mar 24 00:24:28 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-d[96538]: 13: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x558fab0966bc] 2024-03-24T00:24:28.975 INFO:journalctl@ceph.mds.d.smithi118.stdout:Mar 24 00:24:28 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-d[96538]: 14: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x558fab0801ff] 2024-03-24T00:24:28.975 INFO:journalctl@ceph.mds.d.smithi118.stdout:Mar 24 00:24:28 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-d[96538]: 15: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7f052a3b1488] 2024-03-24T00:24:28.975 INFO:journalctl@ceph.mds.d.smithi118.stdout:Mar 24 00:24:28 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-d[96538]: 16: (DispatchQueue::entry()+0x51f) [0x7f052a3ae60f] 2024-03-24T00:24:28.975 INFO:journalctl@ceph.mds.d.smithi118.stdout:Mar 24 00:24:28 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-d[96538]: 17: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f052a481e81] 2024-03-24T00:24:28.975 INFO:journalctl@ceph.mds.d.smithi118.stdout:Mar 24 00:24:28 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-d[96538]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7f0528e9c1ca] 2024-03-24T00:24:28.975 INFO:journalctl@ceph.mds.d.smithi118.stdout:Mar 24 00:24:28 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-d[96538]: 19: clone() 2024-03-24T00:24:28.975 INFO:journalctl@ceph.mds.d.smithi118.stdout:Mar 24 00:24:28 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-d[96538]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-03-24T00:24:28.975 INFO:journalctl@ceph.mds.d.smithi118.stdout:Mar 24 00:24:28 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-d[96538]: 2024-03-24T00:24:28.975 INFO:journalctl@ceph.mds.d.smithi118.stdout:Mar 24 00:24:28 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-d[96538]: -5> 2024-03-24T00:24:28.541+0000 7f051e800700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: In function 'void Locker::set_xlocks_done(MutationImpl*, bool)' thread 7f051e800700 time 2024-03-24T00:24:28.539897+0000 2024-03-24T00:24:28.975 INFO:journalctl@ceph.mds.d.smithi118.stdout:Mar 24 00:24:28 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-d[96538]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: 626: FAILED ceph_assert(obj->is_auth()) 2024-03-24T00:24:28.976 INFO:journalctl@ceph.mds.d.smithi118.stdout:Mar 24 00:24:28 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-d[96538]: 2024-03-24T00:24:28.976 INFO:journalctl@ceph.mds.d.smithi118.stdout:Mar 24 00:24:28 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-d[96538]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-24T00:24:28.976 INFO:journalctl@ceph.mds.d.smithi118.stdout:Mar 24 00:24:28 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-d[96538]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x135) [0x7f052a11f4a1] 2024-03-24T00:24:28.976 INFO:journalctl@ceph.mds.d.smithi118.stdout:Mar 24 00:24:28 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-d[96538]: 2: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7f052a11f667] 2024-03-24T00:24:28.976 INFO:journalctl@ceph.mds.d.smithi118.stdout:Mar 24 00:24:28 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-d[96538]: 3: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x558fab2c8223] 2024-03-24T00:24:28.976 INFO:journalctl@ceph.mds.d.smithi118.stdout:Mar 24 00:24:28 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-d[96538]: 4: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x558fab1425d6] 2024-03-24T00:24:28.976 INFO:journalctl@ceph.mds.d.smithi118.stdout:Mar 24 00:24:28 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-d[96538]: 5: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x558fab142da1] 2024-03-24T00:24:28.976 INFO:journalctl@ceph.mds.d.smithi118.stdout:Mar 24 00:24:28 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-d[96538]: 6: (Server::handle_client_rename(boost::intrusive_ptr const&)+0x259d) [0x558fab17f97d] 2024-03-24T00:24:28.976 INFO:journalctl@ceph.mds.d.smithi118.stdout:Mar 24 00:24:28 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-d[96538]: 7: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x558fab195233] 2024-03-24T00:24:28.976 INFO:journalctl@ceph.mds.d.smithi118.stdout:Mar 24 00:24:28 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-d[96538]: 8: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x558fab197197] 2024-03-24T00:24:28.976 INFO:journalctl@ceph.mds.d.smithi118.stdout:Mar 24 00:24:28 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-d[96538]: 9: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x558fab09611b] 2024-03-24T00:24:28.976 INFO:journalctl@ceph.mds.d.smithi118.stdout:Mar 24 00:24:28 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-d[96538]: 10: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x558fab0966bc] 2024-03-24T00:24:28.976 INFO:journalctl@ceph.mds.d.smithi118.stdout:Mar 24 00:24:28 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-d[96538]: 11: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x558fab0801ff] 2024-03-24T00:24:28.977 INFO:journalctl@ceph.mds.d.smithi118.stdout:Mar 24 00:24:28 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-d[96538]: 12: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7f052a3b1488] 2024-03-24T00:24:28.977 INFO:journalctl@ceph.mds.d.smithi118.stdout:Mar 24 00:24:28 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-d[96538]: 13: (DispatchQueue::entry()+0x51f) [0x7f052a3ae60f] 2024-03-24T00:24:28.977 INFO:journalctl@ceph.mds.d.smithi118.stdout:Mar 24 00:24:28 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-d[96538]: 14: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f052a481e81] 2024-03-24T00:24:28.977 INFO:journalctl@ceph.mds.d.smithi118.stdout:Mar 24 00:24:28 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-d[96538]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7f0528e9c1ca] 2024-03-24T00:24:28.977 INFO:journalctl@ceph.mds.d.smithi118.stdout:Mar 24 00:24:28 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-d[96538]: 16: clone() 2024-03-24T00:24:28.977 INFO:journalctl@ceph.mds.d.smithi118.stdout:Mar 24 00:24:28 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-d[96538]: 2024-03-24T00:24:28.977 INFO:journalctl@ceph.mds.d.smithi118.stdout:Mar 24 00:24:28 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-d[96538]: -4> 2024-03-24T00:24:28.543+0000 7f051e800700 -1 *** Caught signal (Aborted) ** 2024-03-24T00:24:28.977 INFO:journalctl@ceph.mds.d.smithi118.stdout:Mar 24 00:24:28 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-d[96538]: in thread 7f051e800700 thread_name:ms_dispatch 2024-03-24T00:24:28.977 INFO:journalctl@ceph.mds.d.smithi118.stdout:Mar 24 00:24:28 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-d[96538]: 2024-03-24T00:24:28.978 INFO:journalctl@ceph.mds.d.smithi118.stdout:Mar 24 00:24:28 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-d[96538]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-24T00:24:28.978 INFO:journalctl@ceph.mds.d.smithi118.stdout:Mar 24 00:24:28 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-d[96538]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f0528ea6d20] 2024-03-24T00:24:28.978 INFO:journalctl@ceph.mds.d.smithi118.stdout:Mar 24 00:24:28 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-d[96538]: 2: gsignal() 2024-03-24T00:24:28.978 INFO:journalctl@ceph.mds.d.smithi118.stdout:Mar 24 00:24:28 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-d[96538]: 3: abort() 2024-03-24T00:24:28.978 INFO:journalctl@ceph.mds.d.smithi118.stdout:Mar 24 00:24:28 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-d[96538]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x18f) [0x7f052a11f4fb] 2024-03-24T00:24:28.978 INFO:journalctl@ceph.mds.d.smithi118.stdout:Mar 24 00:24:28 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-d[96538]: 5: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7f052a11f667] 2024-03-24T00:24:28.978 INFO:journalctl@ceph.mds.d.smithi118.stdout:Mar 24 00:24:28 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-d[96538]: 6: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x558fab2c8223] 2024-03-24T00:24:28.978 INFO:journalctl@ceph.mds.d.smithi118.stdout:Mar 24 00:24:28 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-d[96538]: 7: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x558fab1425d6] 2024-03-24T00:24:28.978 INFO:journalctl@ceph.mds.d.smithi118.stdout:Mar 24 00:24:28 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-d[96538]: 8: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x558fab142da1] 2024-03-24T00:24:28.978 INFO:journalctl@ceph.mds.d.smithi118.stdout:Mar 24 00:24:28 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-d[96538]: 9: (Server::handle_client_rename(boost::intrusive_ptr const&)+0x259d) [0x558fab17f97d] 2024-03-24T00:24:28.978 INFO:journalctl@ceph.mds.d.smithi118.stdout:Mar 24 00:24:28 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-d[96538]: 10: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x558fab195233] 2024-03-24T00:24:28.978 INFO:journalctl@ceph.mds.d.smithi118.stdout:Mar 24 00:24:28 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-d[96538]: 11: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x558fab197197] 2024-03-24T00:24:28.978 INFO:journalctl@ceph.mds.d.smithi118.stdout:Mar 24 00:24:28 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-d[96538]: 12: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x558fab09611b] 2024-03-24T00:24:28.979 INFO:journalctl@ceph.mds.d.smithi118.stdout:Mar 24 00:24:28 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-d[96538]: 13: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x558fab0966bc] 2024-03-24T00:24:28.979 INFO:journalctl@ceph.mds.d.smithi118.stdout:Mar 24 00:24:28 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-d[96538]: 14: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x558fab0801ff] 2024-03-24T00:24:28.979 INFO:journalctl@ceph.mds.d.smithi118.stdout:Mar 24 00:24:28 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-d[96538]: 15: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7f052a3b1488] 2024-03-24T00:24:28.979 INFO:journalctl@ceph.mds.d.smithi118.stdout:Mar 24 00:24:28 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-d[96538]: 16: (DispatchQueue::entry()+0x51f) [0x7f052a3ae60f] 2024-03-24T00:24:28.979 INFO:journalctl@ceph.mds.d.smithi118.stdout:Mar 24 00:24:28 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-d[96538]: 17: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f052a481e81] 2024-03-24T00:24:28.979 INFO:journalctl@ceph.mds.d.smithi118.stdout:Mar 24 00:24:28 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-d[96538]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7f0528e9c1ca] 2024-03-24T00:24:28.979 INFO:journalctl@ceph.mds.d.smithi118.stdout:Mar 24 00:24:28 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-d[96538]: 19: clone() 2024-03-24T00:24:28.979 INFO:journalctl@ceph.mds.d.smithi118.stdout:Mar 24 00:24:28 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-d[96538]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-03-24T00:24:28.979 INFO:journalctl@ceph.mds.d.smithi118.stdout:Mar 24 00:24:28 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-d[96538]: 2024-03-24T00:24:29.696 INFO:journalctl@ceph.mds.d.smithi118.stdout:Mar 24 00:24:29 smithi118 podman[96630]: 2024-03-24 00:24:29.494949218 +0000 UTC m=+0.862531669 container died ff8d7eec8ba9a530f9c147d53d29c72fb675d996090dd6c3c96ae7997968c0b0 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-d, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.build-date=20240319, maintainer=Guillaume Abrioux ) 2024-03-24T00:24:29.696 INFO:journalctl@ceph.mds.j.smithi118.stdout:Mar 24 00:24:29 smithi118 podman[96617]: 2024-03-24 00:24:29.42726444 +0000 UTC m=+0.850847538 container remove 7244d2bf20660b74adf0f39669b2b63aaf0e1df2c7ba2ec622c5b2864eba0601 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-j, GIT_CLEAN=True, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, org.label-schema.build-date=20240319, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True) 2024-03-24T00:24:29.696 INFO:journalctl@ceph.mds.j.smithi118.stdout:Mar 24 00:24:29 smithi118 systemd[1]: ceph-d79a5420-e971-11ee-95cd-87774f69a715@mds.j.service: Main process exited, code=exited, status=134/n/a 2024-03-24T00:24:29.697 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:24:29 smithi118 ceph-mon[31535]: pgmap v691: 97 pgs: 97 active+clean; 146 MiB data, 863 MiB used, 1.0 TiB / 1.0 TiB avail; 44 KiB/s rd, 2.9 KiB/s wr, 10 op/s 2024-03-24T00:24:29.697 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:24:29 smithi118 ceph-mon[31535]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-03-24T00:24:29.697 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:24:29 smithi118 ceph-mon[31535]: Cluster is now healthy 2024-03-24T00:24:29.697 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:24:29 smithi118 ceph-mon[31535]: mds.? [v2:172.21.15.118:6836/179741564,v1:172.21.15.118:6837/179741564] up:active 2024-03-24T00:24:29.697 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:24:29 smithi118 ceph-mon[31535]: mds.? [v2:172.21.15.118:6834/1095918599,v1:172.21.15.118:6835/1095918599] up:active 2024-03-24T00:24:29.697 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:24:29 smithi118 ceph-mon[31535]: fsmap cephfs:5 3 up:standby-replay 5 up:active 2024-03-24T00:24:29.710 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:24:29 smithi193 ceph-mon[36878]: pgmap v691: 97 pgs: 97 active+clean; 146 MiB data, 863 MiB used, 1.0 TiB / 1.0 TiB avail; 44 KiB/s rd, 2.9 KiB/s wr, 10 op/s 2024-03-24T00:24:29.711 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:24:29 smithi193 ceph-mon[36878]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-03-24T00:24:29.711 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:24:29 smithi193 ceph-mon[36878]: Cluster is now healthy 2024-03-24T00:24:29.711 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:24:29 smithi193 ceph-mon[36878]: mds.? [v2:172.21.15.118:6836/179741564,v1:172.21.15.118:6837/179741564] up:active 2024-03-24T00:24:29.711 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:24:29 smithi193 ceph-mon[36878]: mds.? [v2:172.21.15.118:6834/1095918599,v1:172.21.15.118:6835/1095918599] up:active 2024-03-24T00:24:29.711 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:24:29 smithi193 ceph-mon[36878]: fsmap cephfs:5 3 up:standby-replay 5 up:active 2024-03-24T00:24:29.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:24:29 smithi189 ceph-mon[36187]: pgmap v691: 97 pgs: 97 active+clean; 146 MiB data, 863 MiB used, 1.0 TiB / 1.0 TiB avail; 44 KiB/s rd, 2.9 KiB/s wr, 10 op/s 2024-03-24T00:24:29.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:24:29 smithi189 ceph-mon[36187]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-03-24T00:24:29.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:24:29 smithi189 ceph-mon[36187]: Cluster is now healthy 2024-03-24T00:24:29.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:24:29 smithi189 ceph-mon[36187]: mds.? [v2:172.21.15.118:6836/179741564,v1:172.21.15.118:6837/179741564] up:active 2024-03-24T00:24:29.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:24:29 smithi189 ceph-mon[36187]: mds.? [v2:172.21.15.118:6834/1095918599,v1:172.21.15.118:6835/1095918599] up:active 2024-03-24T00:24:29.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:24:29 smithi189 ceph-mon[36187]: fsmap cephfs:5 3 up:standby-replay 5 up:active 2024-03-24T00:24:30.561 INFO:journalctl@ceph.mds.d.smithi118.stdout:Mar 24 00:24:30 smithi118 podman[96630]: 2024-03-24 00:24:30.198314002 +0000 UTC m=+1.565896494 container remove ff8d7eec8ba9a530f9c147d53d29c72fb675d996090dd6c3c96ae7997968c0b0 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-d, GIT_BRANCH=HEAD, ceph=True, org.label-schema.build-date=20240319, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0) 2024-03-24T00:24:30.561 INFO:journalctl@ceph.mds.d.smithi118.stdout:Mar 24 00:24:30 smithi118 systemd[1]: ceph-d79a5420-e971-11ee-95cd-87774f69a715@mds.d.service: Main process exited, code=exited, status=134/n/a 2024-03-24T00:24:30.946 INFO:journalctl@ceph.mds.d.smithi118.stdout:Mar 24 00:24:30 smithi118 systemd[1]: ceph-d79a5420-e971-11ee-95cd-87774f69a715@mds.d.service: Failed with result 'exit-code'. 2024-03-24T00:24:30.946 INFO:journalctl@ceph.mds.d.smithi118.stdout:Mar 24 00:24:30 smithi118 systemd[1]: ceph-d79a5420-e971-11ee-95cd-87774f69a715@mds.d.service: Consumed 1.574s CPU time. 2024-03-24T00:24:30.946 INFO:journalctl@ceph.mds.j.smithi118.stdout:Mar 24 00:24:30 smithi118 systemd[1]: ceph-d79a5420-e971-11ee-95cd-87774f69a715@mds.j.service: Failed with result 'exit-code'. 2024-03-24T00:24:30.946 INFO:journalctl@ceph.mds.j.smithi118.stdout:Mar 24 00:24:30 smithi118 systemd[1]: ceph-d79a5420-e971-11ee-95cd-87774f69a715@mds.j.service: Consumed 1.368s CPU time. 2024-03-24T00:24:31.204 INFO:journalctl@ceph.mds.b.smithi189.stdout:Mar 24 00:24:30 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-b[66975]: 2024-03-24T00:24:30.741+0000 7f4e07ab9700 -1 mds.pinger is_rank_lagging: rank=1 was never sent ping request. 2024-03-24T00:24:31.204 INFO:journalctl@ceph.mds.b.smithi189.stdout:Mar 24 00:24:30 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-b[66975]: 2024-03-24T00:24:30.741+0000 7f4e07ab9700 -1 mds.pinger is_rank_lagging: rank=3 was never sent ping request. 2024-03-24T00:24:31.710 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:24:31 smithi193 ceph-mon[36878]: pgmap v692: 97 pgs: 97 active+clean; 146 MiB data, 863 MiB used, 1.0 TiB / 1.0 TiB avail; 86 KiB/s rd, 7.0 KiB/s wr, 28 op/s 2024-03-24T00:24:31.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:24:31 smithi118 ceph-mon[31535]: pgmap v692: 97 pgs: 97 active+clean; 146 MiB data, 863 MiB used, 1.0 TiB / 1.0 TiB avail; 86 KiB/s rd, 7.0 KiB/s wr, 28 op/s 2024-03-24T00:24:31.953 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:24:31 smithi189 ceph-mon[36187]: pgmap v692: 97 pgs: 97 active+clean; 146 MiB data, 863 MiB used, 1.0 TiB / 1.0 TiB avail; 86 KiB/s rd, 7.0 KiB/s wr, 28 op/s 2024-03-24T00:24:33.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:24:33 smithi118 ceph-mon[31535]: pgmap v693: 97 pgs: 97 active+clean; 146 MiB data, 881 MiB used, 1.0 TiB / 1.0 TiB avail; 164 KiB/s rd, 7.1 KiB/s wr, 36 op/s 2024-03-24T00:24:33.755 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:24:33 smithi189 ceph-mon[36187]: pgmap v693: 97 pgs: 97 active+clean; 146 MiB data, 881 MiB used, 1.0 TiB / 1.0 TiB avail; 164 KiB/s rd, 7.1 KiB/s wr, 36 op/s 2024-03-24T00:24:33.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:24:33 smithi193 ceph-mon[36878]: pgmap v693: 97 pgs: 97 active+clean; 146 MiB data, 881 MiB used, 1.0 TiB / 1.0 TiB avail; 164 KiB/s rd, 7.1 KiB/s wr, 36 op/s 2024-03-24T00:24:35.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:24:35 smithi118 ceph-mon[31535]: pgmap v694: 97 pgs: 97 active+clean; 146 MiB data, 881 MiB used, 1.0 TiB / 1.0 TiB avail; 182 KiB/s rd, 6.3 KiB/s wr, 44 op/s 2024-03-24T00:24:35.953 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:24:35 smithi189 ceph-mon[36187]: pgmap v694: 97 pgs: 97 active+clean; 146 MiB data, 881 MiB used, 1.0 TiB / 1.0 TiB avail; 182 KiB/s rd, 6.3 KiB/s wr, 44 op/s 2024-03-24T00:24:35.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:24:35 smithi193 ceph-mon[36878]: pgmap v694: 97 pgs: 97 active+clean; 146 MiB data, 881 MiB used, 1.0 TiB / 1.0 TiB avail; 182 KiB/s rd, 6.3 KiB/s wr, 44 op/s 2024-03-24T00:24:37.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:24:37 smithi118 ceph-mon[31535]: pgmap v695: 97 pgs: 97 active+clean; 146 MiB data, 881 MiB used, 1.0 TiB / 1.0 TiB avail; 161 KiB/s rd, 10 KiB/s wr, 38 op/s 2024-03-24T00:24:37.953 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:24:37 smithi189 ceph-mon[36187]: pgmap v695: 97 pgs: 97 active+clean; 146 MiB data, 881 MiB used, 1.0 TiB / 1.0 TiB avail; 161 KiB/s rd, 10 KiB/s wr, 38 op/s 2024-03-24T00:24:37.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:24:37 smithi193 ceph-mon[36878]: pgmap v695: 97 pgs: 97 active+clean; 146 MiB data, 881 MiB used, 1.0 TiB / 1.0 TiB avail; 161 KiB/s rd, 10 KiB/s wr, 38 op/s 2024-03-24T00:24:39.749 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:24:39 smithi118 ceph-mon[31535]: pgmap v696: 97 pgs: 97 active+clean; 146 MiB data, 881 MiB used, 1.0 TiB / 1.0 TiB avail; 152 KiB/s rd, 9.7 KiB/s wr, 37 op/s 2024-03-24T00:24:39.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:24:39 smithi189 ceph-mon[36187]: pgmap v696: 97 pgs: 97 active+clean; 146 MiB data, 881 MiB used, 1.0 TiB / 1.0 TiB avail; 152 KiB/s rd, 9.7 KiB/s wr, 37 op/s 2024-03-24T00:24:39.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:24:39 smithi193 ceph-mon[36878]: pgmap v696: 97 pgs: 97 active+clean; 146 MiB data, 881 MiB used, 1.0 TiB / 1.0 TiB avail; 152 KiB/s rd, 9.7 KiB/s wr, 37 op/s 2024-03-24T00:24:40.621 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:24:40.619+0000 7fde05600640 1 -- 172.21.15.118:0/2754570512 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fde00104af0 msgr2=0x7fde00104f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:24:40.621 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:24:40.619+0000 7fde05600640 1 --2- 172.21.15.118:0/2754570512 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fde00104af0 0x7fde00104f70 secure :-1 s=READY pgs=462 cs=0 l=1 rev1=1 crypto rx=0x7fddec009900 tx=0x7fddec02bc30 comp rx=0 tx=0).stop 2024-03-24T00:24:40.621 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:24:40.620+0000 7fde05600640 1 -- 172.21.15.118:0/2754570512 shutdown_connections 2024-03-24T00:24:40.621 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:24:40.620+0000 7fde05600640 1 --2- 172.21.15.118:0/2754570512 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fde001054b0 0x7fde0010bd40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:24:40.621 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:24:40.620+0000 7fde05600640 1 --2- 172.21.15.118:0/2754570512 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fde00104af0 0x7fde00104f70 unknown :-1 s=CLOSED pgs=462 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:24:40.621 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:24:40.620+0000 7fde05600640 1 --2- 172.21.15.118:0/2754570512 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fde00103900 0x7fde00103d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:24:40.621 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:24:40.620+0000 7fde05600640 1 -- 172.21.15.118:0/2754570512 >> 172.21.15.118:0/2754570512 conn(0x7fde000fee90 msgr2=0x7fde001012b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:24:40.621 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:24:40.620+0000 7fde05600640 1 -- 172.21.15.118:0/2754570512 shutdown_connections 2024-03-24T00:24:40.622 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:24:40.620+0000 7fde05600640 1 -- 172.21.15.118:0/2754570512 wait complete. 2024-03-24T00:24:40.623 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:24:40.621+0000 7fde05600640 1 Processor -- start 2024-03-24T00:24:40.623 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:24:40.621+0000 7fde05600640 1 -- start start 2024-03-24T00:24:40.624 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:24:40.621+0000 7fde05600640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fde00103900 0x7fde00199b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:24:40.624 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:24:40.621+0000 7fde05600640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fde00104af0 0x7fde0019a070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:24:40.624 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:24:40.621+0000 7fde05600640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fde001054b0 0x7fde001a10e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:24:40.624 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:24:40.621+0000 7fde05600640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fde0010f5d0 con 0x7fde00103900 2024-03-24T00:24:40.624 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:24:40.621+0000 7fde05600640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fde0010f450 con 0x7fde001054b0 2024-03-24T00:24:40.624 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:24:40.621+0000 7fde05600640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fde0010f750 con 0x7fde00104af0 2024-03-24T00:24:40.624 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:24:40.621+0000 7fddfe000640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fde00104af0 0x7fde0019a070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:24:40.624 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:24:40.621+0000 7fddff400640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fde001054b0 0x7fde001a10e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:24:40.624 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:24:40.621+0000 7fddfea00640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fde00103900 0x7fde00199b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:24:40.624 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:24:40.621+0000 7fddfe000640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fde00104af0 0x7fde0019a070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.118:51174/0 (socket says 172.21.15.118:51174) 2024-03-24T00:24:40.624 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:24:40.621+0000 7fddfe000640 1 -- 172.21.15.118:0/394825751 learned_addr learned my addr 172.21.15.118:0/394825751 (peer_addr_for_me v2:172.21.15.118:0/0) 2024-03-24T00:24:40.624 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:24:40.622+0000 7fddfe000640 1 -- 172.21.15.118:0/394825751 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fde001054b0 msgr2=0x7fde001a10e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:24:40.624 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:24:40.622+0000 7fddfe000640 1 --2- 172.21.15.118:0/394825751 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fde001054b0 0x7fde001a10e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:24:40.625 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:24:40.622+0000 7fddfe000640 1 -- 172.21.15.118:0/394825751 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fde00103900 msgr2=0x7fde00199b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:24:40.625 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:24:40.622+0000 7fddfe000640 1 --2- 172.21.15.118:0/394825751 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fde00103900 0x7fde00199b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:24:40.625 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:24:40.622+0000 7fddfe000640 1 -- 172.21.15.118:0/394825751 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fde001a1750 con 0x7fde00104af0 2024-03-24T00:24:40.625 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:24:40.622+0000 7fddfe000640 1 --2- 172.21.15.118:0/394825751 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fde00104af0 0x7fde0019a070 secure :-1 s=READY pgs=463 cs=0 l=1 rev1=1 crypto rx=0x7fddec000c00 tx=0x7fddec02c650 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:24:40.625 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:24:40.622+0000 7fddf7400640 1 -- 172.21.15.118:0/394825751 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fddec043020 con 0x7fde00104af0 2024-03-24T00:24:40.625 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:24:40.622+0000 7fddf7400640 1 -- 172.21.15.118:0/394825751 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fddec0054e0 con 0x7fde00104af0 2024-03-24T00:24:40.625 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:24:40.622+0000 7fddf7400640 1 -- 172.21.15.118:0/394825751 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fddec003c40 con 0x7fde00104af0 2024-03-24T00:24:40.625 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:24:40.622+0000 7fde05600640 1 -- 172.21.15.118:0/394825751 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fde001a19e0 con 0x7fde00104af0 2024-03-24T00:24:40.625 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:24:40.622+0000 7fde05600640 1 -- 172.21.15.118:0/394825751 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fde001a1f80 con 0x7fde00104af0 2024-03-24T00:24:40.626 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:24:40.625+0000 7fddf7400640 1 -- 172.21.15.118:0/394825751 <== mon.1 v2:172.21.15.193:3300/0 4 ==== mgrmap(e 20) v1 ==== 137291+0+0 (secure 0 0 0) 0x7fddec046020 con 0x7fde00104af0 2024-03-24T00:24:40.626 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:24:40.626+0000 7fde083f7740 1 Processor -- start 2024-03-24T00:24:40.626 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:24:40.626+0000 7fde083f7740 1 -- start start 2024-03-24T00:24:40.628 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:24:40.626+0000 7fde083f7740 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x560c41bdf1e0 0x560c41bdf600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:24:40.628 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:24:40.626+0000 7fde083f7740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x560c41bdfbd0 0x560c41be46d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:24:40.628 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:24:40.626+0000 7fde083f7740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x560c41be4d40 0x560c41be71d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:24:40.629 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:24:40.626+0000 7fde083f7740 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x560c41ba23b0 con 0x560c41bdf1e0 2024-03-24T00:24:40.629 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:24:40.626+0000 7fde083f7740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x560c41b44830 con 0x560c41bdfbd0 2024-03-24T00:24:40.629 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:24:40.626+0000 7fde083f7740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x560c41aed5d0 con 0x560c41be4d40 2024-03-24T00:24:40.629 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:24:40.626+0000 7fddff400640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x560c41bdf1e0 0x560c41bdf600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:24:40.629 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:24:40.627+0000 7fddff400640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x560c41bdf1e0 0x560c41bdf600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.118:3300/0 says I am v2:172.21.15.118:54404/0 (socket says 172.21.15.118:54404) 2024-03-24T00:24:40.629 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:24:40.627+0000 7fddff400640 1 -- 172.21.15.118:0/2249270482 learned_addr learned my addr 172.21.15.118:0/2249270482 (peer_addr_for_me v2:172.21.15.118:0/0) 2024-03-24T00:24:40.629 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:24:40.627+0000 7fddfea00640 1 --2- 172.21.15.118:0/2249270482 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x560c41bdfbd0 0x560c41be46d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:24:40.629 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:24:40.627+0000 7fddfe000640 1 --2- 172.21.15.118:0/2249270482 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x560c41be4d40 0x560c41be71d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:24:40.629 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:24:40.627+0000 7fddf7400640 1 --2- 172.21.15.118:0/394825751 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7fdde40a2920 0x7fdde40a4de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:24:40.629 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:24:40.627+0000 7fddfea00640 1 -- 172.21.15.118:0/2249270482 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x560c41be4d40 msgr2=0x560c41be71d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:24:40.629 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:24:40.627+0000 7fddfea00640 1 --2- 172.21.15.118:0/2249270482 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x560c41be4d40 0x560c41be71d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:24:40.629 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:24:40.627+0000 7fddfea00640 1 -- 172.21.15.118:0/2249270482 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x560c41bdf1e0 msgr2=0x560c41bdf600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:24:40.629 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:24:40.627+0000 7fddfea00640 1 --2- 172.21.15.118:0/2249270482 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x560c41bdf1e0 0x560c41bdf600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:24:40.630 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:24:40.627+0000 7fddfea00640 1 -- 172.21.15.118:0/2249270482 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x560c41b052c0 con 0x560c41bdfbd0 2024-03-24T00:24:40.630 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:24:40.627+0000 7fddf7400640 1 -- 172.21.15.118:0/394825751 <== mon.1 v2:172.21.15.193:3300/0 5 ==== osd_map(120..120 src has 1..120) v4 ==== 10857+0+0 (secure 0 0 0) 0x7fddec0f14c0 con 0x7fde00104af0 2024-03-24T00:24:40.630 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:24:40.627+0000 7fddfe000640 1 --2- 172.21.15.118:0/2249270482 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x560c41be4d40 0x560c41be71d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T00:24:40.630 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:24:40.627+0000 7fddff400640 1 --2- 172.21.15.118:0/2249270482 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x560c41bdf1e0 0x560c41bdf600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T00:24:40.630 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:24:40.627+0000 7fddff400640 1 --2- 172.21.15.118:0/394825751 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7fdde40a2920 0x7fdde40a4de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:24:40.630 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:24:40.627+0000 7fddfea00640 1 --2- 172.21.15.118:0/2249270482 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x560c41bdfbd0 0x560c41be46d0 secure :-1 s=READY pgs=461 cs=0 l=1 rev1=1 crypto rx=0x7fddf0009bf0 tx=0x7fddf0011810 comp rx=0 tx=0).ready entity=mon.2 client_cookie=3ed623c2be33e31f server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:24:40.630 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:24:40.628+0000 7fddeb400640 1 -- 172.21.15.118:0/2249270482 <== mon.2 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fddf0004050 con 0x560c41bdfbd0 2024-03-24T00:24:40.630 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:24:40.628+0000 7fddeb400640 1 -- 172.21.15.118:0/2249270482 <== mon.2 v2:172.21.15.189:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fddf0005500 con 0x560c41bdfbd0 2024-03-24T00:24:40.630 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:24:40.628+0000 7fddeb400640 1 -- 172.21.15.118:0/2249270482 <== mon.2 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fddf0004dd0 con 0x560c41bdfbd0 2024-03-24T00:24:40.635 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:24:40.628+0000 7fde083f7740 1 -- 172.21.15.118:0/2249270482 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x560c41bdfbd0 msgr2=0x560c41be46d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:24:40.635 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:24:40.628+0000 7fddff400640 1 --2- 172.21.15.118:0/394825751 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7fdde40a2920 0x7fdde40a4de0 secure :-1 s=READY pgs=395 cs=0 l=1 rev1=1 crypto rx=0x560c41be0430 tx=0x7fddf8004a40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:24:40.635 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:24:40.628+0000 7fde083f7740 1 --2- 172.21.15.118:0/2249270482 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x560c41bdfbd0 0x560c41be46d0 secure :-1 s=READY pgs=461 cs=0 l=1 rev1=1 crypto rx=0x7fddf0009bf0 tx=0x7fddf0011810 comp rx=0 tx=0).stop 2024-03-24T00:24:40.635 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:24:40.629+0000 7fde083f7740 1 -- 172.21.15.118:0/2249270482 shutdown_connections 2024-03-24T00:24:40.635 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:24:40.629+0000 7fde083f7740 1 --2- 172.21.15.118:0/2249270482 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x560c41be4d40 0x560c41be71d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:24:40.635 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:24:40.629+0000 7fde083f7740 1 --2- 172.21.15.118:0/2249270482 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x560c41bdfbd0 0x560c41be46d0 unknown :-1 s=CLOSED pgs=461 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:24:40.635 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:24:40.629+0000 7fde083f7740 1 --2- 172.21.15.118:0/2249270482 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x560c41bdf1e0 0x560c41bdf600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:24:40.635 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:24:40.629+0000 7fde083f7740 1 -- 172.21.15.118:0/2249270482 >> 172.21.15.118:0/2249270482 conn(0x560c41bda750 msgr2=0x560c41bdab50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:24:40.635 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:24:40.629+0000 7fde083f7740 1 -- 172.21.15.118:0/2249270482 shutdown_connections 2024-03-24T00:24:40.636 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:24:40.629+0000 7fde083f7740 1 -- 172.21.15.118:0/2249270482 wait complete. 2024-03-24T00:24:40.636 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:24:40.629+0000 7fde083f7740 1 -- start start 2024-03-24T00:24:40.636 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:24:40.629+0000 7fde083f7740 1 Processor -- start 2024-03-24T00:24:40.636 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:24:40.630+0000 7fde083f7740 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x560c41bdf1e0 0x560c41be9450 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:24:40.636 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:24:40.630+0000 7fde083f7740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x560c41bdfbd0 0x560c41be9990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:24:40.636 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:24:40.630+0000 7fde083f7740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x560c41be4d40 0x560c41bf08f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:24:40.636 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:24:40.630+0000 7fde083f7740 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x560c41b00a20 con 0x560c41bdf1e0 2024-03-24T00:24:40.636 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:24:40.630+0000 7fde083f7740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x560c41b408d0 con 0x560c41be4d40 2024-03-24T00:24:40.636 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:24:40.630+0000 7fde083f7740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x560c41be7cf0 con 0x560c41bdfbd0 2024-03-24T00:24:40.636 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:24:40.630+0000 7fde083f7740 10 client.0 _ll_register_callbacks cb 0x560c41b3d1c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T00:24:40.636 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:24:40.630+0000 7fddfe000640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x560c41bdfbd0 0x560c41be9990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:24:40.636 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:24:40.630+0000 7fddff400640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x560c41be4d40 0x560c41bf08f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:24:40.636 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:24:40.630+0000 7fddfe000640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x560c41bdfbd0 0x560c41be9990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.118:51186/0 (socket says 172.21.15.118:51186) 2024-03-24T00:24:40.636 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:24:40.630+0000 7fddfea00640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x560c41bdf1e0 0x560c41be9450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:24:40.637 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:24:40.630+0000 7fddfe000640 1 -- 172.21.15.118:0/3601143627 learned_addr learned my addr 172.21.15.118:0/3601143627 (peer_addr_for_me v2:172.21.15.118:0/0) 2024-03-24T00:24:40.637 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:24:40.630+0000 7fddfe000640 1 -- 172.21.15.118:0/3601143627 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x560c41be4d40 msgr2=0x560c41bf08f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:24:40.637 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:24:40.630+0000 7fddfe000640 1 --2- 172.21.15.118:0/3601143627 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x560c41be4d40 0x560c41bf08f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:24:40.637 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:24:40.630+0000 7fddfe000640 1 -- 172.21.15.118:0/3601143627 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x560c41bdf1e0 msgr2=0x560c41be9450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:24:40.637 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:24:40.630+0000 7fddfe000640 1 --2- 172.21.15.118:0/3601143627 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x560c41bdf1e0 0x560c41be9450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:24:40.637 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:24:40.630+0000 7fddfe000640 1 -- 172.21.15.118:0/3601143627 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x560c41859210 con 0x560c41bdfbd0 2024-03-24T00:24:40.637 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:24:40.631+0000 7fddfe000640 1 --2- 172.21.15.118:0/3601143627 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x560c41bdfbd0 0x560c41be9990 secure :-1 s=READY pgs=464 cs=0 l=1 rev1=1 crypto rx=0x7fddec006010 tx=0x7fddec009620 comp rx=0 tx=0).ready entity=mon.1 client_cookie=adda6c1b867811f8 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:24:40.637 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:24:40.631+0000 7fdddfe00640 10 client.0 ms_handle_connect on v2:172.21.15.193:3300/0 2024-03-24T00:24:40.637 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:24:40.631+0000 7fdddfe00640 1 -- 172.21.15.118:0/3601143627 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fddec0bcd20 con 0x560c41bdfbd0 2024-03-24T00:24:40.637 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:24:40.631+0000 7fdddfe00640 1 -- 172.21.15.118:0/3601143627 <== mon.1 v2:172.21.15.193:3300/0 2 ==== osd_map(120..120 src has 1..120) v4 ==== 10857+0+0 (secure 0 0 0) 0x7fddec0ee630 con 0x560c41bdfbd0 2024-03-24T00:24:40.637 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:24:40.631+0000 7fde083f7740 1 -- 172.21.15.118:0/3601143627 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fddf0005500 con 0x560c41bdfbd0 2024-03-24T00:24:40.637 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:24:40.631+0000 7fdddfe00640 1 -- 172.21.15.118:0/3601143627 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_get_version_reply(handle=1 version=170) v2 ==== 24+0+0 (secure 0 0 0) 0x7fddec004250 con 0x560c41bdfbd0 2024-03-24T00:24:40.637 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:24:40.631+0000 7fde083f7740 10 client.27028 fetch_fsmap learned FSMap version 170 2024-03-24T00:24:40.637 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:24:40.631+0000 7fde083f7740 1 -- 172.21.15.118:0/3601143627 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,fsmap=170,monmap=4+}) v3 -- 0x560c41925620 con 0x560c41bdfbd0 2024-03-24T00:24:40.638 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:24:40.632+0000 7fdddfe00640 1 -- 172.21.15.118:0/3601143627 <== mon.1 v2:172.21.15.193:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fddf0005500 con 0x560c41bdfbd0 2024-03-24T00:24:40.638 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:24:40.632+0000 7fdddfe00640 1 -- 172.21.15.118:0/3601143627 <== mon.1 v2:172.21.15.193:3300/0 5 ==== fsmap(e 170) v1 ==== 5658+0+0 (secure 0 0 0) 0x7fddec0c1120 con 0x560c41bdfbd0 2024-03-24T00:24:40.638 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:24:40.632+0000 7fde083f7740 10 client.27028 fetch_fsmap finished waiting for FSMap version 170 2024-03-24T00:24:40.638 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:24:40.632+0000 7fde083f7740 10 client.27028 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-24T00:24:40.638 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:24:40.632+0000 7fde083f7740 20 client.27028 populate_metadata read hostname 'smithi118' 2024-03-24T00:24:40.638 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:24:40.632+0000 7fde083f7740 1 --2- 172.21.15.118:0/3601143627 >> [v2:172.21.15.189:6838/4074484168,v1:172.21.15.189:6839/4074484168] conn(0x560c41bf5210 0x560c41bf7600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:24:40.638 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:24:40.632+0000 7fde083f7740 4 client.27028 mds_command: new command op to 24569 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-24T00:24:40.638 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:24:40.632+0000 7fde083f7740 1 -- 172.21.15.118:0/3601143627 --> [v2:172.21.15.189:6838/4074484168,v1:172.21.15.189:6839/4074484168] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x560c41b51d70 con 0x560c41bf5210 2024-03-24T00:24:40.638 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:24:40.632+0000 7fddfea00640 1 --2- 172.21.15.118:0/3601143627 >> [v2:172.21.15.189:6838/4074484168,v1:172.21.15.189:6839/4074484168] conn(0x560c41bf5210 0x560c41bf7600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:24:40.638 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:24:40.633+0000 7fdddfe00640 10 client.27028 ms_handle_connect on v2:172.21.15.189:6838/4074484168 2024-03-24T00:24:40.638 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:24:40.633+0000 7fddfea00640 1 --2- 172.21.15.118:0/3601143627 >> [v2:172.21.15.189:6838/4074484168,v1:172.21.15.189:6839/4074484168] conn(0x560c41bf5210 0x560c41bf7600 crc :-1 s=READY pgs=397 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=61b0e631c2f1988 server_cookie=97a4a5951655419 in_seq=0 out_seq=0 2024-03-24T00:24:40.638 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:24:40.634+0000 7fdddfe00640 1 -- 172.21.15.118:0/3601143627 <== mds.0 v2:172.21.15.189:6838/4074484168 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x560c41b51d70 con 0x560c41bf5210 2024-03-24T00:24:40.638 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:24:40.634+0000 7fdddfe00640 10 client.27028 handle_command_reply: tid=0 2024-03-24T00:24:40.638 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:24:40.634+0000 7fde083f7740 1 client.27028 shutdown 2024-03-24T00:24:40.639 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:24:40.634+0000 7fde083f7740 1 -- 172.21.15.118:0/3601143627 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x560c41bdfbd0 msgr2=0x560c41be9990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:24:40.639 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:24:40.634+0000 7fde083f7740 1 --2- 172.21.15.118:0/3601143627 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x560c41bdfbd0 0x560c41be9990 secure :-1 s=READY pgs=464 cs=0 l=1 rev1=1 crypto rx=0x7fddec006010 tx=0x7fddec009620 comp rx=0 tx=0).stop 2024-03-24T00:24:40.639 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:24:40.634+0000 7fde083f7740 1 -- 172.21.15.118:0/3601143627 shutdown_connections 2024-03-24T00:24:40.639 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:24:40.634+0000 7fde083f7740 1 --2- 172.21.15.118:0/3601143627 >> [v2:172.21.15.189:6838/4074484168,v1:172.21.15.189:6839/4074484168] conn(0x560c41bf5210 0x560c41bf7600 crc :-1 s=READY pgs=397 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:24:40.639 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:24:40.634+0000 7fde083f7740 1 --2- 172.21.15.118:0/3601143627 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x560c41be4d40 0x560c41bf08f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:24:40.639 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:24:40.634+0000 7fde083f7740 1 --2- 172.21.15.118:0/3601143627 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x560c41bdfbd0 0x560c41be9990 unknown :-1 s=CLOSED pgs=464 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:24:40.639 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:24:40.634+0000 7fdddfe00640 0 client.27028 ms_handle_reset on v2:172.21.15.189:6838/4074484168 2024-03-24T00:24:40.639 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:24:40.634+0000 7fde083f7740 1 --2- 172.21.15.118:0/3601143627 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x560c41bdf1e0 0x560c41be9450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:24:40.639 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:24:40.634+0000 7fde083f7740 1 -- 172.21.15.118:0/3601143627 >> 172.21.15.118:0/3601143627 conn(0x560c41bda750 msgr2=0x560c41bd9950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:24:40.639 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:24:40.634+0000 7fde083f7740 1 -- 172.21.15.118:0/3601143627 shutdown_connections 2024-03-24T00:24:40.639 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:24:40.634+0000 7fde083f7740 1 -- 172.21.15.118:0/3601143627 wait complete. 2024-03-24T00:24:40.639 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:24:40.634+0000 7fde083f7740 20 client.27028 trim_cache size 0 max 16384 2024-03-24T00:24:40.651 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:24:40.649+0000 7fde083f7740 1 Processor -- start 2024-03-24T00:24:40.651 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:24:40.649+0000 7fde083f7740 1 -- start start 2024-03-24T00:24:40.651 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:24:40.650+0000 7fde083f7740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x560c41bdf1e0 0x560c41bde0c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:24:40.652 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:24:40.650+0000 7fde083f7740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x560c41bdfbd0 0x560c41bf0810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:24:40.652 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:24:40.650+0000 7fde083f7740 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x560c41be4d40 0x560c41bfbeb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:24:40.652 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:24:40.650+0000 7fde083f7740 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fde001acfe0 con 0x560c41be4d40 2024-03-24T00:24:40.652 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:24:40.650+0000 7fde083f7740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fde001ad640 con 0x560c41bdf1e0 2024-03-24T00:24:40.652 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:24:40.650+0000 7fde083f7740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fde001ab000 con 0x560c41bdfbd0 2024-03-24T00:24:40.652 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:24:40.650+0000 7fddfea00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x560c41bdf1e0 0x560c41bde0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:24:40.652 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:24:40.650+0000 7fddfe000640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x560c41bdfbd0 0x560c41bf0810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:24:40.652 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:24:40.650+0000 7fddfea00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x560c41bdf1e0 0x560c41bde0c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.118:39844/0 (socket says 172.21.15.118:39844) 2024-03-24T00:24:40.652 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:24:40.650+0000 7fddff400640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x560c41be4d40 0x560c41bfbeb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:24:40.652 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:24:40.650+0000 7fddfea00640 1 -- 172.21.15.118:0/1084657449 learned_addr learned my addr 172.21.15.118:0/1084657449 (peer_addr_for_me v2:172.21.15.118:0/0) 2024-03-24T00:24:40.652 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:24:40.650+0000 7fddfea00640 1 -- 172.21.15.118:0/1084657449 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x560c41bdfbd0 msgr2=0x560c41bf0810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:24:40.653 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:24:40.650+0000 7fddfea00640 1 --2- 172.21.15.118:0/1084657449 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x560c41bdfbd0 0x560c41bf0810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:24:40.653 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:24:40.650+0000 7fddfea00640 1 -- 172.21.15.118:0/1084657449 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x560c41be4d40 msgr2=0x560c41bfbeb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:24:40.653 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:24:40.650+0000 7fddfea00640 1 --2- 172.21.15.118:0/1084657449 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x560c41be4d40 0x560c41bfbeb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:24:40.653 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:24:40.650+0000 7fddfea00640 1 -- 172.21.15.118:0/1084657449 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x560c41be7590 con 0x560c41bdf1e0 2024-03-24T00:24:40.653 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:24:40.650+0000 7fddfe000640 1 --2- 172.21.15.118:0/1084657449 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x560c41bdfbd0 0x560c41bf0810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T00:24:40.653 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:24:40.650+0000 7fddfea00640 1 --2- 172.21.15.118:0/1084657449 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x560c41bdf1e0 0x560c41bde0c0 secure :-1 s=READY pgs=462 cs=0 l=1 rev1=1 crypto rx=0x7fddf0014040 tx=0x7fddf0005d30 comp rx=0 tx=0).ready entity=mon.2 client_cookie=890f379167d7dcb1 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:24:40.653 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:24:40.651+0000 7fddeaa00640 1 -- 172.21.15.118:0/1084657449 <== mon.2 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fddf00037f0 con 0x560c41bdf1e0 2024-03-24T00:24:40.653 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:24:40.651+0000 7fddeaa00640 1 -- 172.21.15.118:0/1084657449 <== mon.2 v2:172.21.15.189:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fddf0004040 con 0x560c41bdf1e0 2024-03-24T00:24:40.653 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:24:40.651+0000 7fddeaa00640 1 -- 172.21.15.118:0/1084657449 <== mon.2 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fddf0019690 con 0x560c41bdf1e0 2024-03-24T00:24:40.657 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:24:40.651+0000 7fde083f7740 1 -- 172.21.15.118:0/1084657449 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x560c41bdf1e0 msgr2=0x560c41bde0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:24:40.657 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:24:40.651+0000 7fde083f7740 1 --2- 172.21.15.118:0/1084657449 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x560c41bdf1e0 0x560c41bde0c0 secure :-1 s=READY pgs=462 cs=0 l=1 rev1=1 crypto rx=0x7fddf0014040 tx=0x7fddf0005d30 comp rx=0 tx=0).stop 2024-03-24T00:24:40.657 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:24:40.651+0000 7fde083f7740 1 -- 172.21.15.118:0/1084657449 shutdown_connections 2024-03-24T00:24:40.657 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:24:40.651+0000 7fde083f7740 1 --2- 172.21.15.118:0/1084657449 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x560c41be4d40 0x560c41bfbeb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:24:40.657 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:24:40.651+0000 7fde083f7740 1 --2- 172.21.15.118:0/1084657449 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x560c41bdfbd0 0x560c41bf0810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:24:40.657 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:24:40.651+0000 7fde083f7740 1 --2- 172.21.15.118:0/1084657449 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x560c41bdf1e0 0x560c41bde0c0 unknown :-1 s=CLOSED pgs=462 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:24:40.657 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:24:40.651+0000 7fde083f7740 1 -- 172.21.15.118:0/1084657449 >> 172.21.15.118:0/1084657449 conn(0x560c41bda750 msgr2=0x560c41bdd9b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:24:40.657 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:24:40.651+0000 7fde083f7740 1 -- 172.21.15.118:0/1084657449 shutdown_connections 2024-03-24T00:24:40.658 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:24:40.651+0000 7fde083f7740 1 -- 172.21.15.118:0/1084657449 wait complete. 2024-03-24T00:24:40.658 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:24:40.652+0000 7fde083f7740 1 -- start start 2024-03-24T00:24:40.658 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:24:40.652+0000 7fde083f7740 1 Processor -- start 2024-03-24T00:24:40.658 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:24:40.652+0000 7fde083f7740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x560c41bdf1e0 0x560c41bee6e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:24:40.658 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:24:40.652+0000 7fde083f7740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x560c41bdfbd0 0x560c41beec20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:24:40.658 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:24:40.652+0000 7fde083f7740 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x560c41be4d40 0x560c41befbe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:24:40.658 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:24:40.652+0000 7fde083f7740 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fde001ab660 con 0x560c41be4d40 2024-03-24T00:24:40.658 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:24:40.652+0000 7fde083f7740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fde001ac980 con 0x560c41bdfbd0 2024-03-24T00:24:40.658 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:24:40.652+0000 7fde083f7740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fde001abcc0 con 0x560c41bdf1e0 2024-03-24T00:24:40.658 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:24:40.652+0000 7fde083f7740 10 client.0 _ll_register_callbacks cb 0x7fddec005b50 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T00:24:40.658 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:24:40.653+0000 7fddfea00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x560c41bdf1e0 0x560c41bee6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:24:40.658 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:24:40.653+0000 7fddfea00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x560c41bdf1e0 0x560c41bee6e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.118:51196/0 (socket says 172.21.15.118:51196) 2024-03-24T00:24:40.658 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:24:40.653+0000 7fddfea00640 1 -- 172.21.15.118:0/1432016403 learned_addr learned my addr 172.21.15.118:0/1432016403 (peer_addr_for_me v2:172.21.15.118:0/0) 2024-03-24T00:24:40.659 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:24:40.653+0000 7fddff400640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x560c41be4d40 0x560c41befbe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:24:40.659 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:24:40.653+0000 7fddfe000640 1 --2- 172.21.15.118:0/1432016403 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x560c41bdfbd0 0x560c41beec20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:24:40.659 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:24:40.653+0000 7fddfea00640 1 -- 172.21.15.118:0/1432016403 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x560c41bdfbd0 msgr2=0x560c41beec20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:24:40.659 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:24:40.653+0000 7fddfea00640 1 --2- 172.21.15.118:0/1432016403 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x560c41bdfbd0 0x560c41beec20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:24:40.659 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:24:40.653+0000 7fddfea00640 1 -- 172.21.15.118:0/1432016403 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x560c41be4d40 msgr2=0x560c41befbe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:24:40.659 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:24:40.653+0000 7fddfea00640 1 --2- 172.21.15.118:0/1432016403 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x560c41be4d40 0x560c41befbe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:24:40.659 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:24:40.653+0000 7fddfea00640 1 -- 172.21.15.118:0/1432016403 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x560c41bf01d0 con 0x560c41bdf1e0 2024-03-24T00:24:40.659 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:24:40.653+0000 7fddfea00640 1 --2- 172.21.15.118:0/1432016403 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x560c41bdf1e0 0x560c41bee6e0 secure :-1 s=READY pgs=465 cs=0 l=1 rev1=1 crypto rx=0x7fddf001b040 tx=0x7fddf0004420 comp rx=0 tx=0).ready entity=mon.1 client_cookie=e961dc1409eb9b4f server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:24:40.659 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:24:40.653+0000 7fdddfe00640 10 client.0 ms_handle_connect on v2:172.21.15.193:3300/0 2024-03-24T00:24:40.659 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:24:40.654+0000 7fdddfe00640 1 -- 172.21.15.118:0/1432016403 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fddf0004820 con 0x560c41bdf1e0 2024-03-24T00:24:40.659 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:24:40.654+0000 7fdddfe00640 1 -- 172.21.15.118:0/1432016403 <== mon.1 v2:172.21.15.193:3300/0 2 ==== osd_map(120..120 src has 1..120) v4 ==== 10857+0+0 (secure 0 0 0) 0x7fddf00035f0 con 0x560c41bdf1e0 2024-03-24T00:24:40.659 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:24:40.654+0000 7fde083f7740 1 -- 172.21.15.118:0/1432016403 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7fddf0004040 con 0x560c41bdf1e0 2024-03-24T00:24:40.659 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:24:40.654+0000 7fdddfe00640 1 -- 172.21.15.118:0/1432016403 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_get_version_reply(handle=1 version=170) v2 ==== 24+0+0 (secure 0 0 0) 0x7fddf00037f0 con 0x560c41bdf1e0 2024-03-24T00:24:40.659 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:24:40.654+0000 7fde083f7740 10 client.27034 fetch_fsmap learned FSMap version 170 2024-03-24T00:24:40.660 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:24:40.654+0000 7fde083f7740 1 -- 172.21.15.118:0/1432016403 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,fsmap=170,monmap=4+}) v3 -- 0x560c41c51210 con 0x560c41bdf1e0 2024-03-24T00:24:40.660 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:24:40.654+0000 7fdddfe00640 1 -- 172.21.15.118:0/1432016403 <== mon.1 v2:172.21.15.193:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fddf0004040 con 0x560c41bdf1e0 2024-03-24T00:24:40.660 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:24:40.654+0000 7fdddfe00640 1 -- 172.21.15.118:0/1432016403 <== mon.1 v2:172.21.15.193:3300/0 5 ==== fsmap(e 170) v1 ==== 5658+0+0 (secure 0 0 0) 0x7fddf0037070 con 0x560c41bdf1e0 2024-03-24T00:24:40.660 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:24:40.654+0000 7fde083f7740 10 client.27034 fetch_fsmap finished waiting for FSMap version 170 2024-03-24T00:24:40.660 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:24:40.654+0000 7fde083f7740 10 client.27034 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-24T00:24:40.660 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:24:40.654+0000 7fde083f7740 20 client.27034 populate_metadata read hostname 'smithi118' 2024-03-24T00:24:40.660 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:24:40.654+0000 7fde083f7740 1 --2- 172.21.15.118:0/1432016403 >> [v2:172.21.15.189:6838/4074484168,v1:172.21.15.189:6839/4074484168] conn(0x560c41c51b60 0x560c41c53f80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:24:40.660 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:24:40.654+0000 7fde083f7740 4 client.27034 mds_command: new command op to 24569 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-03-24T00:24:40.660 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:24:40.654+0000 7fde083f7740 1 -- 172.21.15.118:0/1432016403 --> [v2:172.21.15.189:6838/4074484168,v1:172.21.15.189:6839/4074484168] -- command(tid 0: {"prefix": "flush journal"}) v1 -- 0x560c41b4c0c0 con 0x560c41c51b60 2024-03-24T00:24:40.660 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:24:40.655+0000 7fddfe000640 1 --2- 172.21.15.118:0/1432016403 >> [v2:172.21.15.189:6838/4074484168,v1:172.21.15.189:6839/4074484168] conn(0x560c41c51b60 0x560c41c53f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:24:40.660 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:24:40.655+0000 7fddfe000640 1 --2- 172.21.15.118:0/1432016403 >> [v2:172.21.15.189:6838/4074484168,v1:172.21.15.189:6839/4074484168] conn(0x560c41c51b60 0x560c41c53f80 crc :-1 s=READY pgs=398 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=c703a238e3132a75 server_cookie=afc0f93e0d558764 in_seq=0 out_seq=0 2024-03-24T00:24:40.660 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:24:40.655+0000 7fdddfe00640 10 client.27034 ms_handle_connect on v2:172.21.15.189:6838/4074484168 2024-03-24T00:24:41.067 INFO:journalctl@ceph.mds.j.smithi118.stdout:Mar 24 00:24:40 smithi118 systemd[1]: ceph-d79a5420-e971-11ee-95cd-87774f69a715@mds.j.service: Scheduled restart job, restart counter is at 4. 2024-03-24T00:24:41.067 INFO:journalctl@ceph.mds.j.smithi118.stdout:Mar 24 00:24:40 smithi118 systemd[1]: Stopped Ceph mds.j for d79a5420-e971-11ee-95cd-87774f69a715. 2024-03-24T00:24:41.067 INFO:journalctl@ceph.mds.j.smithi118.stdout:Mar 24 00:24:40 smithi118 systemd[1]: ceph-d79a5420-e971-11ee-95cd-87774f69a715@mds.j.service: Consumed 1.368s CPU time. 2024-03-24T00:24:41.067 INFO:journalctl@ceph.mds.j.smithi118.stdout:Mar 24 00:24:40 smithi118 systemd[1]: Starting Ceph mds.j for d79a5420-e971-11ee-95cd-87774f69a715... 2024-03-24T00:24:41.068 INFO:journalctl@ceph.mds.d.smithi118.stdout:Mar 24 00:24:40 smithi118 systemd[1]: ceph-d79a5420-e971-11ee-95cd-87774f69a715@mds.d.service: Scheduled restart job, restart counter is at 4. 2024-03-24T00:24:41.068 INFO:journalctl@ceph.mds.d.smithi118.stdout:Mar 24 00:24:40 smithi118 systemd[1]: Stopped Ceph mds.d for d79a5420-e971-11ee-95cd-87774f69a715. 2024-03-24T00:24:41.068 INFO:journalctl@ceph.mds.d.smithi118.stdout:Mar 24 00:24:40 smithi118 systemd[1]: ceph-d79a5420-e971-11ee-95cd-87774f69a715@mds.d.service: Consumed 1.574s CPU time. 2024-03-24T00:24:41.068 INFO:journalctl@ceph.mds.d.smithi118.stdout:Mar 24 00:24:40 smithi118 systemd[1]: Starting Ceph mds.d for d79a5420-e971-11ee-95cd-87774f69a715... 2024-03-24T00:24:41.318 INFO:journalctl@ceph.mds.d.smithi118.stdout:Mar 24 00:24:41 smithi118 podman[96843]: 2024-03-24 00:24:40.973145669 +0000 UTC m=+0.034656704 image pull c664afe56bb01db4f9afa9532c20598e201e664f2e9ba2be50db0d52674a26d2 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc 2024-03-24T00:24:41.318 INFO:journalctl@ceph.mds.d.smithi118.stdout:Mar 24 00:24:41 smithi118 podman[96843]: 2024-03-24 00:24:41.10794808 +0000 UTC m=+0.169459151 container create 76fb46d4f076316c4871ba3ae6e8d8a3cde9e622ec7c972e9c4a6c2c72115c46 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-d, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, org.label-schema.schema-version=1.0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.build-date=20240319, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, GIT_CLEAN=True) 2024-03-24T00:24:41.319 INFO:journalctl@ceph.mds.j.smithi118.stdout:Mar 24 00:24:41 smithi118 podman[96844]: 2024-03-24 00:24:40.974738922 +0000 UTC m=+0.035629395 image pull c664afe56bb01db4f9afa9532c20598e201e664f2e9ba2be50db0d52674a26d2 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc 2024-03-24T00:24:41.572 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:24:41 smithi118 ceph-mon[31535]: pgmap v697: 97 pgs: 97 active+clean; 146 MiB data, 881 MiB used, 1.0 TiB / 1.0 TiB avail; 129 KiB/s rd, 9.0 KiB/s wr, 35 op/s 2024-03-24T00:24:41.572 INFO:journalctl@ceph.mds.j.smithi118.stdout:Mar 24 00:24:41 smithi118 podman[96844]: 2024-03-24 00:24:41.318438588 +0000 UTC m=+0.379329060 container create a714c7b16352e17f582f5c3983c5a7862dfa479508c596f53fef9d23d615a827 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-j, GIT_CLEAN=True, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240319, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, ceph=True, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, maintainer=Guillaume Abrioux ) 2024-03-24T00:24:41.933 INFO:journalctl@ceph.mds.d.smithi118.stdout:Mar 24 00:24:41 smithi118 podman[96843]: 2024-03-24 00:24:41.572687987 +0000 UTC m=+0.634199091 container init 76fb46d4f076316c4871ba3ae6e8d8a3cde9e622ec7c972e9c4a6c2c72115c46 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-d, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_CLEAN=True, org.label-schema.build-date=20240319, org.label-schema.license=GPLv2, io.buildah.version=1.33.5) 2024-03-24T00:24:41.933 INFO:journalctl@ceph.mds.d.smithi118.stdout:Mar 24 00:24:41 smithi118 podman[96843]: 2024-03-24 00:24:41.582445097 +0000 UTC m=+0.643956201 container start 76fb46d4f076316c4871ba3ae6e8d8a3cde9e622ec7c972e9c4a6c2c72115c46 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-d, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, org.label-schema.build-date=20240319, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , ceph=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS) 2024-03-24T00:24:41.933 INFO:journalctl@ceph.mds.d.smithi118.stdout:Mar 24 00:24:41 smithi118 ceph-mds[96880]: -- 172.21.15.118:0/2392880785 <== mon.2 v2:172.21.15.189:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x558dc8ce5a40 con 0x558dc8cb8c00 2024-03-24T00:24:41.933 INFO:journalctl@ceph.mds.d.smithi118.stdout:Mar 24 00:24:41 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-d[96871]: starting mds.d at 2024-03-24T00:24:41.933 INFO:journalctl@ceph.mds.d.smithi118.stdout:Mar 24 00:24:41 smithi118 bash[96843]: 76fb46d4f076316c4871ba3ae6e8d8a3cde9e622ec7c972e9c4a6c2c72115c46 2024-03-24T00:24:41.933 INFO:journalctl@ceph.mds.d.smithi118.stdout:Mar 24 00:24:41 smithi118 systemd[1]: Started Ceph mds.d for d79a5420-e971-11ee-95cd-87774f69a715. 2024-03-24T00:24:41.933 INFO:journalctl@ceph.mds.j.smithi118.stdout:Mar 24 00:24:41 smithi118 podman[96844]: 2024-03-24 00:24:41.681070796 +0000 UTC m=+0.741961272 container init a714c7b16352e17f582f5c3983c5a7862dfa479508c596f53fef9d23d615a827 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-j, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.build-date=20240319, CEPH_POINT_RELEASE=, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD) 2024-03-24T00:24:41.933 INFO:journalctl@ceph.mds.j.smithi118.stdout:Mar 24 00:24:41 smithi118 podman[96844]: 2024-03-24 00:24:41.688173094 +0000 UTC m=+0.749063556 container start a714c7b16352e17f582f5c3983c5a7862dfa479508c596f53fef9d23d615a827 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-j, GIT_CLEAN=True, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240319, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD) 2024-03-24T00:24:41.933 INFO:journalctl@ceph.mds.j.smithi118.stdout:Mar 24 00:24:41 smithi118 ceph-mds[96912]: -- 172.21.15.118:0/1783036576 <== mon.2 v2:172.21.15.189:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x56361fa41a40 con 0x56361fa14800 2024-03-24T00:24:41.934 INFO:journalctl@ceph.mds.j.smithi118.stdout:Mar 24 00:24:41 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-j[96876]: starting mds.j at 2024-03-24T00:24:41.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:24:41 smithi189 ceph-mon[36187]: pgmap v697: 97 pgs: 97 active+clean; 146 MiB data, 881 MiB used, 1.0 TiB / 1.0 TiB avail; 129 KiB/s rd, 9.0 KiB/s wr, 35 op/s 2024-03-24T00:24:41.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:24:41 smithi193 ceph-mon[36878]: pgmap v697: 97 pgs: 97 active+clean; 146 MiB data, 881 MiB used, 1.0 TiB / 1.0 TiB avail; 129 KiB/s rd, 9.0 KiB/s wr, 35 op/s 2024-03-24T00:24:42.196 INFO:journalctl@ceph.mds.j.smithi118.stdout:Mar 24 00:24:41 smithi118 bash[96844]: a714c7b16352e17f582f5c3983c5a7862dfa479508c596f53fef9d23d615a827 2024-03-24T00:24:42.196 INFO:journalctl@ceph.mds.j.smithi118.stdout:Mar 24 00:24:41 smithi118 systemd[1]: Started Ceph mds.j for d79a5420-e971-11ee-95cd-87774f69a715. 2024-03-24T00:24:42.947 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:24:42 smithi118 ceph-mon[31535]: daemon mds.d restarted 2024-03-24T00:24:42.947 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:24:42 smithi118 ceph-mon[31535]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-03-24T00:24:42.947 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:24:42 smithi118 ceph-mon[31535]: osdmap e121: 12 total, 12 up, 12 in 2024-03-24T00:24:42.947 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:24:42 smithi118 ceph-mon[31535]: mds.? [v2:172.21.15.118:6834/456491655,v1:172.21.15.118:6835/456491655] up:boot 2024-03-24T00:24:42.948 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:24:42 smithi118 ceph-mon[31535]: Standby daemon mds.d assigned to filesystem cephfs as rank 1 2024-03-24T00:24:42.948 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:24:42 smithi118 ceph-mon[31535]: fsmap cephfs:4/5 {0=b=up:active,2=c=up:active,3=j=up:active,4=e=up:active} 3 up:standby-replay 1 up:standby, 1 failed 2024-03-24T00:24:42.948 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:24:42 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2024-03-24T00:24:42.948 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:24:42 smithi118 ceph-mon[31535]: fsmap cephfs:5/5 3 up:standby-replay 1 up:replay 4 up:active 2024-03-24T00:24:42.948 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:24:42 smithi118 ceph-mon[31535]: daemon mds.j restarted 2024-03-24T00:24:42.948 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:24:42 smithi118 ceph-mon[31535]: osdmap e122: 12 total, 12 up, 12 in 2024-03-24T00:24:42.948 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:24:42 smithi118 ceph-mon[31535]: mds.? [v2:172.21.15.118:6834/456491655,v1:172.21.15.118:6835/456491655] up:resolve 2024-03-24T00:24:42.948 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:24:42 smithi118 ceph-mon[31535]: mds.? [v2:172.21.15.118:6836/2587227307,v1:172.21.15.118:6837/2587227307] up:boot 2024-03-24T00:24:42.948 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:24:42 smithi118 ceph-mon[31535]: Standby daemon mds.j assigned to filesystem cephfs as rank 3 2024-03-24T00:24:42.948 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:24:42 smithi118 ceph-mon[31535]: fsmap cephfs:4/5 {0=b=up:active,1=d=up:resolve,2=c=up:active,4=e=up:active} 3 up:standby-replay 1 up:standby, 1 failed 2024-03-24T00:24:42.948 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:24:42 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2024-03-24T00:24:42.948 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:24:42 smithi118 ceph-mon[31535]: fsmap cephfs:5/5 3 up:standby-replay 1 up:replay 1 up:resolve 3 up:active 2024-03-24T00:24:42.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:24:42 smithi189 ceph-mon[36187]: daemon mds.d restarted 2024-03-24T00:24:42.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:24:42 smithi189 ceph-mon[36187]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-03-24T00:24:42.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:24:42 smithi189 ceph-mon[36187]: osdmap e121: 12 total, 12 up, 12 in 2024-03-24T00:24:42.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:24:42 smithi189 ceph-mon[36187]: mds.? [v2:172.21.15.118:6834/456491655,v1:172.21.15.118:6835/456491655] up:boot 2024-03-24T00:24:42.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:24:42 smithi189 ceph-mon[36187]: Standby daemon mds.d assigned to filesystem cephfs as rank 1 2024-03-24T00:24:42.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:24:42 smithi189 ceph-mon[36187]: fsmap cephfs:4/5 {0=b=up:active,2=c=up:active,3=j=up:active,4=e=up:active} 3 up:standby-replay 1 up:standby, 1 failed 2024-03-24T00:24:42.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:24:42 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2024-03-24T00:24:42.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:24:42 smithi189 ceph-mon[36187]: fsmap cephfs:5/5 3 up:standby-replay 1 up:replay 4 up:active 2024-03-24T00:24:42.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:24:42 smithi189 ceph-mon[36187]: daemon mds.j restarted 2024-03-24T00:24:42.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:24:42 smithi189 ceph-mon[36187]: osdmap e122: 12 total, 12 up, 12 in 2024-03-24T00:24:42.955 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:24:42 smithi189 ceph-mon[36187]: mds.? [v2:172.21.15.118:6834/456491655,v1:172.21.15.118:6835/456491655] up:resolve 2024-03-24T00:24:42.955 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:24:42 smithi189 ceph-mon[36187]: mds.? [v2:172.21.15.118:6836/2587227307,v1:172.21.15.118:6837/2587227307] up:boot 2024-03-24T00:24:42.955 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:24:42 smithi189 ceph-mon[36187]: Standby daemon mds.j assigned to filesystem cephfs as rank 3 2024-03-24T00:24:42.955 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:24:42 smithi189 ceph-mon[36187]: fsmap cephfs:4/5 {0=b=up:active,1=d=up:resolve,2=c=up:active,4=e=up:active} 3 up:standby-replay 1 up:standby, 1 failed 2024-03-24T00:24:42.955 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:24:42 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2024-03-24T00:24:42.955 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:24:42 smithi189 ceph-mon[36187]: fsmap cephfs:5/5 3 up:standby-replay 1 up:replay 1 up:resolve 3 up:active 2024-03-24T00:24:42.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:24:42 smithi193 ceph-mon[36878]: daemon mds.d restarted 2024-03-24T00:24:42.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:24:42 smithi193 ceph-mon[36878]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-03-24T00:24:42.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:24:42 smithi193 ceph-mon[36878]: osdmap e121: 12 total, 12 up, 12 in 2024-03-24T00:24:42.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:24:42 smithi193 ceph-mon[36878]: mds.? [v2:172.21.15.118:6834/456491655,v1:172.21.15.118:6835/456491655] up:boot 2024-03-24T00:24:42.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:24:42 smithi193 ceph-mon[36878]: Standby daemon mds.d assigned to filesystem cephfs as rank 1 2024-03-24T00:24:42.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:24:42 smithi193 ceph-mon[36878]: fsmap cephfs:4/5 {0=b=up:active,2=c=up:active,3=j=up:active,4=e=up:active} 3 up:standby-replay 1 up:standby, 1 failed 2024-03-24T00:24:42.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:24:42 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2024-03-24T00:24:42.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:24:42 smithi193 ceph-mon[36878]: fsmap cephfs:5/5 3 up:standby-replay 1 up:replay 4 up:active 2024-03-24T00:24:42.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:24:42 smithi193 ceph-mon[36878]: daemon mds.j restarted 2024-03-24T00:24:42.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:24:42 smithi193 ceph-mon[36878]: osdmap e122: 12 total, 12 up, 12 in 2024-03-24T00:24:42.962 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:24:42 smithi193 ceph-mon[36878]: mds.? [v2:172.21.15.118:6834/456491655,v1:172.21.15.118:6835/456491655] up:resolve 2024-03-24T00:24:42.962 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:24:42 smithi193 ceph-mon[36878]: mds.? [v2:172.21.15.118:6836/2587227307,v1:172.21.15.118:6837/2587227307] up:boot 2024-03-24T00:24:42.962 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:24:42 smithi193 ceph-mon[36878]: Standby daemon mds.j assigned to filesystem cephfs as rank 3 2024-03-24T00:24:42.962 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:24:42 smithi193 ceph-mon[36878]: fsmap cephfs:4/5 {0=b=up:active,1=d=up:resolve,2=c=up:active,4=e=up:active} 3 up:standby-replay 1 up:standby, 1 failed 2024-03-24T00:24:42.962 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:24:42 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2024-03-24T00:24:42.962 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:24:42 smithi193 ceph-mon[36878]: fsmap cephfs:5/5 3 up:standby-replay 1 up:replay 1 up:resolve 3 up:active 2024-03-24T00:24:44.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:24:43 smithi118 ceph-mon[31535]: pgmap v700: 97 pgs: 97 active+clean; 146 MiB data, 881 MiB used, 1.0 TiB / 1.0 TiB avail; 10 KiB/s rd, 8.5 KiB/s wr, 7 op/s 2024-03-24T00:24:44.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:24:43 smithi118 ceph-mon[31535]: mds.? [v2:172.21.15.118:6836/2587227307,v1:172.21.15.118:6837/2587227307] up:resolve 2024-03-24T00:24:44.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:24:43 smithi118 ceph-mon[31535]: fsmap cephfs:5/5 3 up:standby-replay 2 up:resolve 3 up:active 2024-03-24T00:24:44.203 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:24:43 smithi189 ceph-mon[36187]: pgmap v700: 97 pgs: 97 active+clean; 146 MiB data, 881 MiB used, 1.0 TiB / 1.0 TiB avail; 10 KiB/s rd, 8.5 KiB/s wr, 7 op/s 2024-03-24T00:24:44.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:24:43 smithi189 ceph-mon[36187]: mds.? [v2:172.21.15.118:6836/2587227307,v1:172.21.15.118:6837/2587227307] up:resolve 2024-03-24T00:24:44.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:24:43 smithi189 ceph-mon[36187]: fsmap cephfs:5/5 3 up:standby-replay 2 up:resolve 3 up:active 2024-03-24T00:24:44.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:24:43 smithi193 ceph-mon[36878]: pgmap v700: 97 pgs: 97 active+clean; 146 MiB data, 881 MiB used, 1.0 TiB / 1.0 TiB avail; 10 KiB/s rd, 8.5 KiB/s wr, 7 op/s 2024-03-24T00:24:44.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:24:43 smithi193 ceph-mon[36878]: mds.? [v2:172.21.15.118:6836/2587227307,v1:172.21.15.118:6837/2587227307] up:resolve 2024-03-24T00:24:44.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:24:43 smithi193 ceph-mon[36878]: fsmap cephfs:5/5 3 up:standby-replay 2 up:resolve 3 up:active 2024-03-24T00:24:44.800 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:24:44.799+0000 7fdddfe00640 1 -- 172.21.15.118:0/1432016403 <== mds.0 v2:172.21.15.189:6838/4074484168 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+44 (crc 0 0 0) 0x560c41b4c0c0 con 0x560c41c51b60 2024-03-24T00:24:44.800 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:24:44.799+0000 7fdddfe00640 10 client.27034 handle_command_reply: tid=0 2024-03-24T00:24:44.801 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:24:44.799+0000 7fde083f7740 1 client.27034 shutdown 2024-03-24T00:24:44.801 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:24:44.800+0000 7fde083f7740 1 -- 172.21.15.118:0/1432016403 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x560c41bdf1e0 msgr2=0x560c41bee6e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:24:44.802 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:24:44.800+0000 7fde083f7740 1 --2- 172.21.15.118:0/1432016403 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x560c41bdf1e0 0x560c41bee6e0 secure :-1 s=READY pgs=465 cs=0 l=1 rev1=1 crypto rx=0x7fddf001b040 tx=0x7fddf0004420 comp rx=0 tx=0).stop 2024-03-24T00:24:44.802 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:24:44.800+0000 7fde083f7740 1 -- 172.21.15.118:0/1432016403 shutdown_connections 2024-03-24T00:24:44.802 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:24:44.800+0000 7fde083f7740 1 --2- 172.21.15.118:0/1432016403 >> [v2:172.21.15.189:6838/4074484168,v1:172.21.15.189:6839/4074484168] conn(0x560c41c51b60 0x560c41c53f80 crc :-1 s=READY pgs=398 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:24:44.802 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:24:44.800+0000 7fde083f7740 1 --2- 172.21.15.118:0/1432016403 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x560c41be4d40 0x560c41befbe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:24:44.802 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:24:44.800+0000 7fde083f7740 1 --2- 172.21.15.118:0/1432016403 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x560c41bdfbd0 0x560c41beec20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:24:44.802 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:24:44.800+0000 7fde083f7740 1 --2- 172.21.15.118:0/1432016403 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x560c41bdf1e0 0x560c41bee6e0 unknown :-1 s=CLOSED pgs=465 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:24:44.802 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:24:44.800+0000 7fde083f7740 1 -- 172.21.15.118:0/1432016403 >> 172.21.15.118:0/1432016403 conn(0x560c41bda750 msgr2=0x560c41bde240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:24:44.802 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:24:44.800+0000 7fdddfe00640 0 client.27034 ms_handle_reset on v2:172.21.15.189:6838/4074484168 2024-03-24T00:24:44.802 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:24:44.801+0000 7fde083f7740 1 -- 172.21.15.118:0/1432016403 shutdown_connections 2024-03-24T00:24:44.802 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:24:44.801+0000 7fde083f7740 1 -- 172.21.15.118:0/1432016403 wait complete. 2024-03-24T00:24:44.802 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:24:44.801+0000 7fde083f7740 20 client.27034 trim_cache size 0 max 16384 2024-03-24T00:24:44.803 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stdout:{ 2024-03-24T00:24:44.803 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stdout: "message": "", 2024-03-24T00:24:44.803 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stdout: "return_code": 0 2024-03-24T00:24:44.803 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stdout:} 2024-03-24T00:24:44.804 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:24:44.802+0000 7fde05600640 1 -- 172.21.15.118:0/394825751 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7fdde40a2920 msgr2=0x7fdde40a4de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:24:44.804 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:24:44.802+0000 7fde05600640 1 --2- 172.21.15.118:0/394825751 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7fdde40a2920 0x7fdde40a4de0 secure :-1 s=READY pgs=395 cs=0 l=1 rev1=1 crypto rx=0x560c41be0430 tx=0x7fddf8004a40 comp rx=0 tx=0).stop 2024-03-24T00:24:44.804 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:24:44.802+0000 7fde05600640 1 -- 172.21.15.118:0/394825751 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fde00104af0 msgr2=0x7fde0019a070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:24:44.804 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:24:44.802+0000 7fde05600640 1 --2- 172.21.15.118:0/394825751 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fde00104af0 0x7fde0019a070 secure :-1 s=READY pgs=463 cs=0 l=1 rev1=1 crypto rx=0x7fddec000c00 tx=0x7fddec02c650 comp rx=0 tx=0).stop 2024-03-24T00:24:44.804 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:24:44.802+0000 7fde05600640 1 -- 172.21.15.118:0/394825751 shutdown_connections 2024-03-24T00:24:44.804 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:24:44.802+0000 7fde05600640 1 --2- 172.21.15.118:0/394825751 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7fdde40a2920 0x7fdde40a4de0 unknown :-1 s=CLOSED pgs=395 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:24:44.804 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:24:44.802+0000 7fde05600640 1 --2- 172.21.15.118:0/394825751 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fde001054b0 0x7fde001a10e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:24:44.804 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:24:44.802+0000 7fde05600640 1 --2- 172.21.15.118:0/394825751 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fde00104af0 0x7fde0019a070 unknown :-1 s=CLOSED pgs=463 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:24:44.805 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:24:44.802+0000 7fde05600640 1 --2- 172.21.15.118:0/394825751 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fde00103900 0x7fde00199b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:24:44.805 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:24:44.802+0000 7fde05600640 1 -- 172.21.15.118:0/394825751 >> 172.21.15.118:0/394825751 conn(0x7fde000fee90 msgr2=0x7fde00107d90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:24:44.805 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:24:44.803+0000 7fde05600640 1 -- 172.21.15.118:0/394825751 shutdown_connections 2024-03-24T00:24:44.805 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:24:44.803+0000 7fde05600640 1 -- 172.21.15.118:0/394825751 wait complete. 2024-03-24T00:24:45.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:24:44 smithi118 ceph-mon[31535]: mds.? [v2:172.21.15.118:6834/456491655,v1:172.21.15.118:6835/456491655] up:reconnect 2024-03-24T00:24:45.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:24:44 smithi118 ceph-mon[31535]: mds.? [v2:172.21.15.118:6836/2587227307,v1:172.21.15.118:6837/2587227307] up:reconnect 2024-03-24T00:24:45.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:24:44 smithi118 ceph-mon[31535]: fsmap cephfs:5/5 3 up:standby-replay 2 up:reconnect 3 up:active 2024-03-24T00:24:45.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:24:44 smithi118 ceph-mon[31535]: reconnect by client.15159 192.168.0.1:0/3463739313 after 0.0209995 2024-03-24T00:24:45.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:24:44 smithi118 ceph-mon[31535]: reconnect by client.15159 192.168.0.1:0/3463739313 after 0.0209995 2024-03-24T00:24:45.203 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:24:44 smithi189 ceph-mon[36187]: mds.? [v2:172.21.15.118:6834/456491655,v1:172.21.15.118:6835/456491655] up:reconnect 2024-03-24T00:24:45.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:24:44 smithi189 ceph-mon[36187]: mds.? [v2:172.21.15.118:6836/2587227307,v1:172.21.15.118:6837/2587227307] up:reconnect 2024-03-24T00:24:45.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:24:44 smithi189 ceph-mon[36187]: fsmap cephfs:5/5 3 up:standby-replay 2 up:reconnect 3 up:active 2024-03-24T00:24:45.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:24:44 smithi189 ceph-mon[36187]: reconnect by client.15159 192.168.0.1:0/3463739313 after 0.0209995 2024-03-24T00:24:45.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:24:44 smithi189 ceph-mon[36187]: reconnect by client.15159 192.168.0.1:0/3463739313 after 0.0209995 2024-03-24T00:24:45.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:24:44 smithi193 ceph-mon[36878]: mds.? [v2:172.21.15.118:6834/456491655,v1:172.21.15.118:6835/456491655] up:reconnect 2024-03-24T00:24:45.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:24:44 smithi193 ceph-mon[36878]: mds.? [v2:172.21.15.118:6836/2587227307,v1:172.21.15.118:6837/2587227307] up:reconnect 2024-03-24T00:24:45.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:24:44 smithi193 ceph-mon[36878]: fsmap cephfs:5/5 3 up:standby-replay 2 up:reconnect 3 up:active 2024-03-24T00:24:45.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:24:44 smithi193 ceph-mon[36878]: reconnect by client.15159 192.168.0.1:0/3463739313 after 0.0209995 2024-03-24T00:24:45.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:24:44 smithi193 ceph-mon[36878]: reconnect by client.15159 192.168.0.1:0/3463739313 after 0.0209995 2024-03-24T00:24:46.098 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:24:45 smithi118 ceph-mon[31535]: pgmap v701: 97 pgs: 97 active+clean; 146 MiB data, 881 MiB used, 1.0 TiB / 1.0 TiB avail; 38 KiB/s rd, 13 KiB/s wr, 11 op/s 2024-03-24T00:24:46.098 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:24:45 smithi118 ceph-mon[31535]: mds.? [v2:172.21.15.118:6836/2587227307,v1:172.21.15.118:6837/2587227307] up:rejoin 2024-03-24T00:24:46.098 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:24:45 smithi118 ceph-mon[31535]: mds.? [v2:172.21.15.118:6834/456491655,v1:172.21.15.118:6835/456491655] up:rejoin 2024-03-24T00:24:46.098 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:24:45 smithi118 ceph-mon[31535]: fsmap cephfs:5/5 3 up:standby-replay 2 up:rejoin 3 up:active 2024-03-24T00:24:46.098 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:24:45 smithi118 ceph-mon[31535]: daemon mds.j is now active in filesystem cephfs as rank 3 2024-03-24T00:24:46.098 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:24:45 smithi118 ceph-mon[31535]: daemon mds.d is now active in filesystem cephfs as rank 1 2024-03-24T00:24:46.099 INFO:journalctl@ceph.mds.j.smithi118.stdout:Mar 24 00:24:45 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-j[96876]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: In function 'void Locker::set_xlocks_done(MutationImpl*, bool)' thread 7fa8a1200700 time 2024-03-24T00:24:45.825535+0000 2024-03-24T00:24:46.099 INFO:journalctl@ceph.mds.j.smithi118.stdout:Mar 24 00:24:45 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-j[96876]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: 626: FAILED ceph_assert(obj->is_auth()) 2024-03-24T00:24:46.099 INFO:journalctl@ceph.mds.j.smithi118.stdout:Mar 24 00:24:45 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-j[96876]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-24T00:24:46.099 INFO:journalctl@ceph.mds.j.smithi118.stdout:Mar 24 00:24:45 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-j[96876]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x135) [0x7fa8acb594a1] 2024-03-24T00:24:46.099 INFO:journalctl@ceph.mds.j.smithi118.stdout:Mar 24 00:24:45 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-j[96876]: 2: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7fa8acb59667] 2024-03-24T00:24:46.099 INFO:journalctl@ceph.mds.j.smithi118.stdout:Mar 24 00:24:45 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-j[96876]: 3: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x56361d7de223] 2024-03-24T00:24:46.099 INFO:journalctl@ceph.mds.j.smithi118.stdout:Mar 24 00:24:45 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-j[96876]: 4: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x56361d6585d6] 2024-03-24T00:24:46.100 INFO:journalctl@ceph.mds.j.smithi118.stdout:Mar 24 00:24:45 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-j[96876]: 5: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x56361d658da1] 2024-03-24T00:24:46.100 INFO:journalctl@ceph.mds.j.smithi118.stdout:Mar 24 00:24:45 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-j[96876]: 6: (Server::handle_client_rename(boost::intrusive_ptr const&)+0x259d) [0x56361d69597d] 2024-03-24T00:24:46.100 INFO:journalctl@ceph.mds.j.smithi118.stdout:Mar 24 00:24:45 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-j[96876]: 7: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x56361d6ab233] 2024-03-24T00:24:46.100 INFO:journalctl@ceph.mds.j.smithi118.stdout:Mar 24 00:24:45 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-j[96876]: 8: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x56361d6ad197] 2024-03-24T00:24:46.100 INFO:journalctl@ceph.mds.j.smithi118.stdout:Mar 24 00:24:45 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-j[96876]: 9: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x56361d5ac11b] 2024-03-24T00:24:46.100 INFO:journalctl@ceph.mds.j.smithi118.stdout:Mar 24 00:24:45 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-j[96876]: 10: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x56361d5ac6bc] 2024-03-24T00:24:46.100 INFO:journalctl@ceph.mds.j.smithi118.stdout:Mar 24 00:24:45 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-j[96876]: 11: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x56361d5961ff] 2024-03-24T00:24:46.100 INFO:journalctl@ceph.mds.j.smithi118.stdout:Mar 24 00:24:45 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-j[96876]: 12: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7fa8acdeb488] 2024-03-24T00:24:46.100 INFO:journalctl@ceph.mds.j.smithi118.stdout:Mar 24 00:24:45 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-j[96876]: 13: (DispatchQueue::entry()+0x51f) [0x7fa8acde860f] 2024-03-24T00:24:46.100 INFO:journalctl@ceph.mds.j.smithi118.stdout:Mar 24 00:24:45 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-j[96876]: 14: (DispatchQueue::DispatchThread::entry()+0x11) [0x7fa8acebbe81] 2024-03-24T00:24:46.100 INFO:journalctl@ceph.mds.j.smithi118.stdout:Mar 24 00:24:45 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-j[96876]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7fa8ab8d61ca] 2024-03-24T00:24:46.100 INFO:journalctl@ceph.mds.j.smithi118.stdout:Mar 24 00:24:45 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-j[96876]: 16: clone() 2024-03-24T00:24:46.100 INFO:journalctl@ceph.mds.d.smithi118.stdout:Mar 24 00:24:45 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-d[96871]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: In function 'void Locker::set_xlocks_done(MutationImpl*, bool)' thread 7f5001200700 time 2024-03-24T00:24:45.866817+0000 2024-03-24T00:24:46.101 INFO:journalctl@ceph.mds.d.smithi118.stdout:Mar 24 00:24:45 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-d[96871]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: 626: FAILED ceph_assert(obj->is_auth()) 2024-03-24T00:24:46.101 INFO:journalctl@ceph.mds.d.smithi118.stdout:Mar 24 00:24:45 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-d[96871]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-24T00:24:46.101 INFO:journalctl@ceph.mds.d.smithi118.stdout:Mar 24 00:24:45 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-d[96871]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x135) [0x7f500ca234a1] 2024-03-24T00:24:46.101 INFO:journalctl@ceph.mds.d.smithi118.stdout:Mar 24 00:24:45 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-d[96871]: 2: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7f500ca23667] 2024-03-24T00:24:46.101 INFO:journalctl@ceph.mds.d.smithi118.stdout:Mar 24 00:24:45 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-d[96871]: 3: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x558dc6e18223] 2024-03-24T00:24:46.101 INFO:journalctl@ceph.mds.d.smithi118.stdout:Mar 24 00:24:45 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-d[96871]: 4: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x558dc6c925d6] 2024-03-24T00:24:46.101 INFO:journalctl@ceph.mds.d.smithi118.stdout:Mar 24 00:24:45 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-d[96871]: 5: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x558dc6c92da1] 2024-03-24T00:24:46.101 INFO:journalctl@ceph.mds.d.smithi118.stdout:Mar 24 00:24:45 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-d[96871]: 6: (Server::handle_client_rename(boost::intrusive_ptr const&)+0x259d) [0x558dc6ccf97d] 2024-03-24T00:24:46.101 INFO:journalctl@ceph.mds.d.smithi118.stdout:Mar 24 00:24:45 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-d[96871]: 7: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x558dc6ce5233] 2024-03-24T00:24:46.101 INFO:journalctl@ceph.mds.d.smithi118.stdout:Mar 24 00:24:45 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-d[96871]: 8: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x558dc6ce7197] 2024-03-24T00:24:46.101 INFO:journalctl@ceph.mds.d.smithi118.stdout:Mar 24 00:24:45 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-d[96871]: 9: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x558dc6be611b] 2024-03-24T00:24:46.101 INFO:journalctl@ceph.mds.d.smithi118.stdout:Mar 24 00:24:45 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-d[96871]: 10: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x558dc6be66bc] 2024-03-24T00:24:46.101 INFO:journalctl@ceph.mds.d.smithi118.stdout:Mar 24 00:24:45 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-d[96871]: 11: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x558dc6bd01ff] 2024-03-24T00:24:46.101 INFO:journalctl@ceph.mds.d.smithi118.stdout:Mar 24 00:24:45 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-d[96871]: 12: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7f500ccb5488] 2024-03-24T00:24:46.102 INFO:journalctl@ceph.mds.d.smithi118.stdout:Mar 24 00:24:45 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-d[96871]: 13: (DispatchQueue::entry()+0x51f) [0x7f500ccb260f] 2024-03-24T00:24:46.102 INFO:journalctl@ceph.mds.d.smithi118.stdout:Mar 24 00:24:45 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-d[96871]: 14: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f500cd85e81] 2024-03-24T00:24:46.102 INFO:journalctl@ceph.mds.d.smithi118.stdout:Mar 24 00:24:45 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-d[96871]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7f500b7a01ca] 2024-03-24T00:24:46.102 INFO:journalctl@ceph.mds.d.smithi118.stdout:Mar 24 00:24:45 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-d[96871]: 16: clone() 2024-03-24T00:24:46.102 INFO:journalctl@ceph.mds.j.smithi118.stdout:Mar 24 00:24:45 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-j[96876]: 2024-03-24T00:24:45.825+0000 7fa8a1200700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: In function 'void Locker::set_xlocks_done(MutationImpl*, bool)' thread 7fa8a1200700 time 2024-03-24T00:24:45.825535+0000 2024-03-24T00:24:46.102 INFO:journalctl@ceph.mds.j.smithi118.stdout:Mar 24 00:24:45 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-j[96876]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: 626: FAILED ceph_assert(obj->is_auth()) 2024-03-24T00:24:46.102 INFO:journalctl@ceph.mds.j.smithi118.stdout:Mar 24 00:24:45 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-j[96876]: 2024-03-24T00:24:46.102 INFO:journalctl@ceph.mds.j.smithi118.stdout:Mar 24 00:24:45 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-j[96876]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-24T00:24:46.102 INFO:journalctl@ceph.mds.j.smithi118.stdout:Mar 24 00:24:45 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-j[96876]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x135) [0x7fa8acb594a1] 2024-03-24T00:24:46.102 INFO:journalctl@ceph.mds.j.smithi118.stdout:Mar 24 00:24:45 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-j[96876]: 2: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7fa8acb59667] 2024-03-24T00:24:46.103 INFO:journalctl@ceph.mds.j.smithi118.stdout:Mar 24 00:24:45 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-j[96876]: 3: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x56361d7de223] 2024-03-24T00:24:46.103 INFO:journalctl@ceph.mds.j.smithi118.stdout:Mar 24 00:24:45 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-j[96876]: 4: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x56361d6585d6] 2024-03-24T00:24:46.103 INFO:journalctl@ceph.mds.j.smithi118.stdout:Mar 24 00:24:45 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-j[96876]: 5: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x56361d658da1] 2024-03-24T00:24:46.103 INFO:journalctl@ceph.mds.j.smithi118.stdout:Mar 24 00:24:45 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-j[96876]: 6: (Server::handle_client_rename(boost::intrusive_ptr const&)+0x259d) [0x56361d69597d] 2024-03-24T00:24:46.103 INFO:journalctl@ceph.mds.j.smithi118.stdout:Mar 24 00:24:45 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-j[96876]: 7: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x56361d6ab233] 2024-03-24T00:24:46.103 INFO:journalctl@ceph.mds.j.smithi118.stdout:Mar 24 00:24:45 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-j[96876]: 8: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x56361d6ad197] 2024-03-24T00:24:46.103 INFO:journalctl@ceph.mds.j.smithi118.stdout:Mar 24 00:24:45 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-j[96876]: 9: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x56361d5ac11b] 2024-03-24T00:24:46.103 INFO:journalctl@ceph.mds.j.smithi118.stdout:Mar 24 00:24:45 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-j[96876]: 10: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x56361d5ac6bc] 2024-03-24T00:24:46.103 INFO:journalctl@ceph.mds.j.smithi118.stdout:Mar 24 00:24:45 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-j[96876]: 11: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x56361d5961ff] 2024-03-24T00:24:46.103 INFO:journalctl@ceph.mds.j.smithi118.stdout:Mar 24 00:24:45 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-j[96876]: 12: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7fa8acdeb488] 2024-03-24T00:24:46.103 INFO:journalctl@ceph.mds.j.smithi118.stdout:Mar 24 00:24:45 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-j[96876]: 13: (DispatchQueue::entry()+0x51f) [0x7fa8acde860f] 2024-03-24T00:24:46.103 INFO:journalctl@ceph.mds.j.smithi118.stdout:Mar 24 00:24:45 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-j[96876]: 14: (DispatchQueue::DispatchThread::entry()+0x11) [0x7fa8acebbe81] 2024-03-24T00:24:46.103 INFO:journalctl@ceph.mds.j.smithi118.stdout:Mar 24 00:24:45 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-j[96876]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7fa8ab8d61ca] 2024-03-24T00:24:46.104 INFO:journalctl@ceph.mds.j.smithi118.stdout:Mar 24 00:24:45 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-j[96876]: 16: clone() 2024-03-24T00:24:46.104 INFO:journalctl@ceph.mds.j.smithi118.stdout:Mar 24 00:24:45 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-j[96876]: 2024-03-24T00:24:46.104 INFO:journalctl@ceph.mds.j.smithi118.stdout:Mar 24 00:24:45 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-j[96876]: *** Caught signal (Aborted) ** 2024-03-24T00:24:46.104 INFO:journalctl@ceph.mds.j.smithi118.stdout:Mar 24 00:24:45 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-j[96876]: in thread 7fa8a1200700 thread_name:ms_dispatch 2024-03-24T00:24:46.104 INFO:journalctl@ceph.mds.j.smithi118.stdout:Mar 24 00:24:45 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-j[96876]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-24T00:24:46.104 INFO:journalctl@ceph.mds.j.smithi118.stdout:Mar 24 00:24:45 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-j[96876]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7fa8ab8e0d20] 2024-03-24T00:24:46.104 INFO:journalctl@ceph.mds.d.smithi118.stdout:Mar 24 00:24:45 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-d[96871]: 2024-03-24T00:24:45.867+0000 7f5001200700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: In function 'void Locker::set_xlocks_done(MutationImpl*, bool)' thread 7f5001200700 time 2024-03-24T00:24:45.866817+0000 2024-03-24T00:24:46.104 INFO:journalctl@ceph.mds.d.smithi118.stdout:Mar 24 00:24:45 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-d[96871]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: 626: FAILED ceph_assert(obj->is_auth()) 2024-03-24T00:24:46.104 INFO:journalctl@ceph.mds.d.smithi118.stdout:Mar 24 00:24:45 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-d[96871]: 2024-03-24T00:24:46.104 INFO:journalctl@ceph.mds.d.smithi118.stdout:Mar 24 00:24:45 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-d[96871]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-24T00:24:46.104 INFO:journalctl@ceph.mds.d.smithi118.stdout:Mar 24 00:24:45 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-d[96871]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x135) [0x7f500ca234a1] 2024-03-24T00:24:46.105 INFO:journalctl@ceph.mds.d.smithi118.stdout:Mar 24 00:24:45 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-d[96871]: 2: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7f500ca23667] 2024-03-24T00:24:46.105 INFO:journalctl@ceph.mds.d.smithi118.stdout:Mar 24 00:24:45 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-d[96871]: 3: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x558dc6e18223] 2024-03-24T00:24:46.105 INFO:journalctl@ceph.mds.d.smithi118.stdout:Mar 24 00:24:45 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-d[96871]: 4: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x558dc6c925d6] 2024-03-24T00:24:46.105 INFO:journalctl@ceph.mds.d.smithi118.stdout:Mar 24 00:24:45 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-d[96871]: 5: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x558dc6c92da1] 2024-03-24T00:24:46.105 INFO:journalctl@ceph.mds.d.smithi118.stdout:Mar 24 00:24:45 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-d[96871]: 6: (Server::handle_client_rename(boost::intrusive_ptr const&)+0x259d) [0x558dc6ccf97d] 2024-03-24T00:24:46.105 INFO:journalctl@ceph.mds.d.smithi118.stdout:Mar 24 00:24:45 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-d[96871]: 7: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x558dc6ce5233] 2024-03-24T00:24:46.105 INFO:journalctl@ceph.mds.d.smithi118.stdout:Mar 24 00:24:45 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-d[96871]: 8: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x558dc6ce7197] 2024-03-24T00:24:46.105 INFO:journalctl@ceph.mds.d.smithi118.stdout:Mar 24 00:24:45 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-d[96871]: 9: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x558dc6be611b] 2024-03-24T00:24:46.105 INFO:journalctl@ceph.mds.d.smithi118.stdout:Mar 24 00:24:45 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-d[96871]: 10: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x558dc6be66bc] 2024-03-24T00:24:46.105 INFO:journalctl@ceph.mds.d.smithi118.stdout:Mar 24 00:24:45 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-d[96871]: 11: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x558dc6bd01ff] 2024-03-24T00:24:46.105 INFO:journalctl@ceph.mds.d.smithi118.stdout:Mar 24 00:24:45 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-d[96871]: 12: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7f500ccb5488] 2024-03-24T00:24:46.105 INFO:journalctl@ceph.mds.d.smithi118.stdout:Mar 24 00:24:45 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-d[96871]: 13: (DispatchQueue::entry()+0x51f) [0x7f500ccb260f] 2024-03-24T00:24:46.105 INFO:journalctl@ceph.mds.d.smithi118.stdout:Mar 24 00:24:45 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-d[96871]: 14: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f500cd85e81] 2024-03-24T00:24:46.105 INFO:journalctl@ceph.mds.d.smithi118.stdout:Mar 24 00:24:45 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-d[96871]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7f500b7a01ca] 2024-03-24T00:24:46.106 INFO:journalctl@ceph.mds.d.smithi118.stdout:Mar 24 00:24:45 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-d[96871]: 16: clone() 2024-03-24T00:24:46.106 INFO:journalctl@ceph.mds.d.smithi118.stdout:Mar 24 00:24:45 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-d[96871]: 2024-03-24T00:24:46.106 INFO:journalctl@ceph.mds.d.smithi118.stdout:Mar 24 00:24:45 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-d[96871]: *** Caught signal (Aborted) ** 2024-03-24T00:24:46.106 INFO:journalctl@ceph.mds.d.smithi118.stdout:Mar 24 00:24:45 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-d[96871]: in thread 7f5001200700 thread_name:ms_dispatch 2024-03-24T00:24:46.106 INFO:journalctl@ceph.mds.d.smithi118.stdout:Mar 24 00:24:45 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-d[96871]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-24T00:24:46.106 INFO:journalctl@ceph.mds.d.smithi118.stdout:Mar 24 00:24:45 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-d[96871]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f500b7aad20] 2024-03-24T00:24:46.106 INFO:journalctl@ceph.mds.j.smithi118.stdout:Mar 24 00:24:45 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-j[96876]: 2: gsignal() 2024-03-24T00:24:46.106 INFO:journalctl@ceph.mds.j.smithi118.stdout:Mar 24 00:24:45 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-j[96876]: 3: abort() 2024-03-24T00:24:46.106 INFO:journalctl@ceph.mds.j.smithi118.stdout:Mar 24 00:24:45 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-j[96876]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x18f) [0x7fa8acb594fb] 2024-03-24T00:24:46.107 INFO:journalctl@ceph.mds.j.smithi118.stdout:Mar 24 00:24:45 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-j[96876]: 5: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7fa8acb59667] 2024-03-24T00:24:46.107 INFO:journalctl@ceph.mds.j.smithi118.stdout:Mar 24 00:24:45 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-j[96876]: 6: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x56361d7de223] 2024-03-24T00:24:46.107 INFO:journalctl@ceph.mds.j.smithi118.stdout:Mar 24 00:24:45 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-j[96876]: 7: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x56361d6585d6] 2024-03-24T00:24:46.107 INFO:journalctl@ceph.mds.j.smithi118.stdout:Mar 24 00:24:45 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-j[96876]: 8: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x56361d658da1] 2024-03-24T00:24:46.107 INFO:journalctl@ceph.mds.j.smithi118.stdout:Mar 24 00:24:45 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-j[96876]: 9: (Server::handle_client_rename(boost::intrusive_ptr const&)+0x259d) [0x56361d69597d] 2024-03-24T00:24:46.107 INFO:journalctl@ceph.mds.j.smithi118.stdout:Mar 24 00:24:45 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-j[96876]: 10: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x56361d6ab233] 2024-03-24T00:24:46.107 INFO:journalctl@ceph.mds.j.smithi118.stdout:Mar 24 00:24:45 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-j[96876]: 11: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x56361d6ad197] 2024-03-24T00:24:46.107 INFO:journalctl@ceph.mds.j.smithi118.stdout:Mar 24 00:24:45 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-j[96876]: 12: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x56361d5ac11b] 2024-03-24T00:24:46.107 INFO:journalctl@ceph.mds.j.smithi118.stdout:Mar 24 00:24:45 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-j[96876]: 13: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x56361d5ac6bc] 2024-03-24T00:24:46.107 INFO:journalctl@ceph.mds.j.smithi118.stdout:Mar 24 00:24:45 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-j[96876]: 14: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x56361d5961ff] 2024-03-24T00:24:46.107 INFO:journalctl@ceph.mds.j.smithi118.stdout:Mar 24 00:24:45 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-j[96876]: 15: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7fa8acdeb488] 2024-03-24T00:24:46.107 INFO:journalctl@ceph.mds.j.smithi118.stdout:Mar 24 00:24:45 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-j[96876]: 16: (DispatchQueue::entry()+0x51f) [0x7fa8acde860f] 2024-03-24T00:24:46.107 INFO:journalctl@ceph.mds.j.smithi118.stdout:Mar 24 00:24:45 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-j[96876]: 17: (DispatchQueue::DispatchThread::entry()+0x11) [0x7fa8acebbe81] 2024-03-24T00:24:46.107 INFO:journalctl@ceph.mds.j.smithi118.stdout:Mar 24 00:24:45 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-j[96876]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7fa8ab8d61ca] 2024-03-24T00:24:46.107 INFO:journalctl@ceph.mds.j.smithi118.stdout:Mar 24 00:24:45 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-j[96876]: 19: clone() 2024-03-24T00:24:46.108 INFO:journalctl@ceph.mds.j.smithi118.stdout:Mar 24 00:24:45 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-j[96876]: 2024-03-24T00:24:45.828+0000 7fa8a1200700 -1 *** Caught signal (Aborted) ** 2024-03-24T00:24:46.108 INFO:journalctl@ceph.mds.j.smithi118.stdout:Mar 24 00:24:45 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-j[96876]: in thread 7fa8a1200700 thread_name:ms_dispatch 2024-03-24T00:24:46.108 INFO:journalctl@ceph.mds.j.smithi118.stdout:Mar 24 00:24:45 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-j[96876]: 2024-03-24T00:24:46.108 INFO:journalctl@ceph.mds.j.smithi118.stdout:Mar 24 00:24:45 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-j[96876]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-24T00:24:46.108 INFO:journalctl@ceph.mds.j.smithi118.stdout:Mar 24 00:24:45 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-j[96876]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7fa8ab8e0d20] 2024-03-24T00:24:46.108 INFO:journalctl@ceph.mds.j.smithi118.stdout:Mar 24 00:24:45 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-j[96876]: 2: gsignal() 2024-03-24T00:24:46.108 INFO:journalctl@ceph.mds.j.smithi118.stdout:Mar 24 00:24:45 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-j[96876]: 3: abort() 2024-03-24T00:24:46.108 INFO:journalctl@ceph.mds.j.smithi118.stdout:Mar 24 00:24:45 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-j[96876]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x18f) [0x7fa8acb594fb] 2024-03-24T00:24:46.108 INFO:journalctl@ceph.mds.j.smithi118.stdout:Mar 24 00:24:45 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-j[96876]: 5: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7fa8acb59667] 2024-03-24T00:24:46.109 INFO:journalctl@ceph.mds.d.smithi118.stdout:Mar 24 00:24:45 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-d[96871]: 2: gsignal() 2024-03-24T00:24:46.109 INFO:journalctl@ceph.mds.d.smithi118.stdout:Mar 24 00:24:45 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-d[96871]: 3: abort() 2024-03-24T00:24:46.109 INFO:journalctl@ceph.mds.d.smithi118.stdout:Mar 24 00:24:45 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-d[96871]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x18f) [0x7f500ca234fb] 2024-03-24T00:24:46.109 INFO:journalctl@ceph.mds.d.smithi118.stdout:Mar 24 00:24:45 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-d[96871]: 5: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7f500ca23667] 2024-03-24T00:24:46.109 INFO:journalctl@ceph.mds.d.smithi118.stdout:Mar 24 00:24:45 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-d[96871]: 6: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x558dc6e18223] 2024-03-24T00:24:46.109 INFO:journalctl@ceph.mds.d.smithi118.stdout:Mar 24 00:24:45 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-d[96871]: 7: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x558dc6c925d6] 2024-03-24T00:24:46.109 INFO:journalctl@ceph.mds.d.smithi118.stdout:Mar 24 00:24:45 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-d[96871]: 8: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x558dc6c92da1] 2024-03-24T00:24:46.109 INFO:journalctl@ceph.mds.d.smithi118.stdout:Mar 24 00:24:45 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-d[96871]: 9: (Server::handle_client_rename(boost::intrusive_ptr const&)+0x259d) [0x558dc6ccf97d] 2024-03-24T00:24:46.109 INFO:journalctl@ceph.mds.d.smithi118.stdout:Mar 24 00:24:45 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-d[96871]: 10: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x558dc6ce5233] 2024-03-24T00:24:46.109 INFO:journalctl@ceph.mds.d.smithi118.stdout:Mar 24 00:24:45 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-d[96871]: 11: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x558dc6ce7197] 2024-03-24T00:24:46.109 INFO:journalctl@ceph.mds.d.smithi118.stdout:Mar 24 00:24:45 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-d[96871]: 12: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x558dc6be611b] 2024-03-24T00:24:46.109 INFO:journalctl@ceph.mds.d.smithi118.stdout:Mar 24 00:24:45 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-d[96871]: 13: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x558dc6be66bc] 2024-03-24T00:24:46.110 INFO:journalctl@ceph.mds.d.smithi118.stdout:Mar 24 00:24:45 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-d[96871]: 14: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x558dc6bd01ff] 2024-03-24T00:24:46.110 INFO:journalctl@ceph.mds.d.smithi118.stdout:Mar 24 00:24:45 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-d[96871]: 15: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7f500ccb5488] 2024-03-24T00:24:46.110 INFO:journalctl@ceph.mds.d.smithi118.stdout:Mar 24 00:24:45 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-d[96871]: 16: (DispatchQueue::entry()+0x51f) [0x7f500ccb260f] 2024-03-24T00:24:46.110 INFO:journalctl@ceph.mds.d.smithi118.stdout:Mar 24 00:24:45 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-d[96871]: 17: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f500cd85e81] 2024-03-24T00:24:46.110 INFO:journalctl@ceph.mds.d.smithi118.stdout:Mar 24 00:24:45 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-d[96871]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7f500b7a01ca] 2024-03-24T00:24:46.110 INFO:journalctl@ceph.mds.d.smithi118.stdout:Mar 24 00:24:45 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-d[96871]: 19: clone() 2024-03-24T00:24:46.110 INFO:journalctl@ceph.mds.d.smithi118.stdout:Mar 24 00:24:45 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-d[96871]: 2024-03-24T00:24:45.869+0000 7f5001200700 -1 *** Caught signal (Aborted) ** 2024-03-24T00:24:46.110 INFO:journalctl@ceph.mds.d.smithi118.stdout:Mar 24 00:24:45 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-d[96871]: in thread 7f5001200700 thread_name:ms_dispatch 2024-03-24T00:24:46.110 INFO:journalctl@ceph.mds.d.smithi118.stdout:Mar 24 00:24:45 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-d[96871]: 2024-03-24T00:24:46.110 INFO:journalctl@ceph.mds.d.smithi118.stdout:Mar 24 00:24:45 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-d[96871]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-24T00:24:46.110 INFO:journalctl@ceph.mds.d.smithi118.stdout:Mar 24 00:24:45 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-d[96871]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f500b7aad20] 2024-03-24T00:24:46.110 INFO:journalctl@ceph.mds.d.smithi118.stdout:Mar 24 00:24:45 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-d[96871]: 2: gsignal() 2024-03-24T00:24:46.110 INFO:journalctl@ceph.mds.d.smithi118.stdout:Mar 24 00:24:45 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-d[96871]: 3: abort() 2024-03-24T00:24:46.110 INFO:journalctl@ceph.mds.d.smithi118.stdout:Mar 24 00:24:45 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-d[96871]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x18f) [0x7f500ca234fb] 2024-03-24T00:24:46.110 INFO:journalctl@ceph.mds.d.smithi118.stdout:Mar 24 00:24:45 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-d[96871]: 5: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7f500ca23667] 2024-03-24T00:24:46.111 INFO:journalctl@ceph.mds.j.smithi118.stdout:Mar 24 00:24:45 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-j[96876]: 6: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x56361d7de223] 2024-03-24T00:24:46.111 INFO:journalctl@ceph.mds.j.smithi118.stdout:Mar 24 00:24:45 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-j[96876]: 7: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x56361d6585d6] 2024-03-24T00:24:46.111 INFO:journalctl@ceph.mds.j.smithi118.stdout:Mar 24 00:24:45 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-j[96876]: 8: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x56361d658da1] 2024-03-24T00:24:46.111 INFO:journalctl@ceph.mds.j.smithi118.stdout:Mar 24 00:24:45 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-j[96876]: 9: (Server::handle_client_rename(boost::intrusive_ptr const&)+0x259d) [0x56361d69597d] 2024-03-24T00:24:46.111 INFO:journalctl@ceph.mds.j.smithi118.stdout:Mar 24 00:24:45 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-j[96876]: 10: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x56361d6ab233] 2024-03-24T00:24:46.111 INFO:journalctl@ceph.mds.j.smithi118.stdout:Mar 24 00:24:45 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-j[96876]: 11: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x56361d6ad197] 2024-03-24T00:24:46.111 INFO:journalctl@ceph.mds.j.smithi118.stdout:Mar 24 00:24:45 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-j[96876]: 12: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x56361d5ac11b] 2024-03-24T00:24:46.112 INFO:journalctl@ceph.mds.j.smithi118.stdout:Mar 24 00:24:45 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-j[96876]: 13: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x56361d5ac6bc] 2024-03-24T00:24:46.112 INFO:journalctl@ceph.mds.j.smithi118.stdout:Mar 24 00:24:45 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-j[96876]: 14: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x56361d5961ff] 2024-03-24T00:24:46.112 INFO:journalctl@ceph.mds.j.smithi118.stdout:Mar 24 00:24:45 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-j[96876]: 15: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7fa8acdeb488] 2024-03-24T00:24:46.112 INFO:journalctl@ceph.mds.j.smithi118.stdout:Mar 24 00:24:45 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-j[96876]: 16: (DispatchQueue::entry()+0x51f) [0x7fa8acde860f] 2024-03-24T00:24:46.112 INFO:journalctl@ceph.mds.j.smithi118.stdout:Mar 24 00:24:45 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-j[96876]: 17: (DispatchQueue::DispatchThread::entry()+0x11) [0x7fa8acebbe81] 2024-03-24T00:24:46.112 INFO:journalctl@ceph.mds.j.smithi118.stdout:Mar 24 00:24:45 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-j[96876]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7fa8ab8d61ca] 2024-03-24T00:24:46.112 INFO:journalctl@ceph.mds.j.smithi118.stdout:Mar 24 00:24:45 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-j[96876]: 19: clone() 2024-03-24T00:24:46.112 INFO:journalctl@ceph.mds.j.smithi118.stdout:Mar 24 00:24:45 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-j[96876]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-03-24T00:24:46.112 INFO:journalctl@ceph.mds.j.smithi118.stdout:Mar 24 00:24:45 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-j[96876]: 2024-03-24T00:24:46.112 INFO:journalctl@ceph.mds.j.smithi118.stdout:Mar 24 00:24:45 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-j[96876]: -1> 2024-03-24T00:24:45.825+0000 7fa8a1200700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: In function 'void Locker::set_xlocks_done(MutationImpl*, bool)' thread 7fa8a1200700 time 2024-03-24T00:24:45.825535+0000 2024-03-24T00:24:46.112 INFO:journalctl@ceph.mds.j.smithi118.stdout:Mar 24 00:24:45 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-j[96876]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: 626: FAILED ceph_assert(obj->is_auth()) 2024-03-24T00:24:46.112 INFO:journalctl@ceph.mds.j.smithi118.stdout:Mar 24 00:24:45 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-j[96876]: 2024-03-24T00:24:46.112 INFO:journalctl@ceph.mds.j.smithi118.stdout:Mar 24 00:24:45 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-j[96876]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-24T00:24:46.113 INFO:journalctl@ceph.mds.j.smithi118.stdout:Mar 24 00:24:45 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-j[96876]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x135) [0x7fa8acb594a1] 2024-03-24T00:24:46.113 INFO:journalctl@ceph.mds.j.smithi118.stdout:Mar 24 00:24:45 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-j[96876]: 2: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7fa8acb59667] 2024-03-24T00:24:46.113 INFO:journalctl@ceph.mds.j.smithi118.stdout:Mar 24 00:24:45 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-j[96876]: 3: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x56361d7de223] 2024-03-24T00:24:46.113 INFO:journalctl@ceph.mds.j.smithi118.stdout:Mar 24 00:24:45 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-j[96876]: 4: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x56361d6585d6] 2024-03-24T00:24:46.113 INFO:journalctl@ceph.mds.j.smithi118.stdout:Mar 24 00:24:45 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-j[96876]: 5: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x56361d658da1] 2024-03-24T00:24:46.113 INFO:journalctl@ceph.mds.j.smithi118.stdout:Mar 24 00:24:45 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-j[96876]: 6: (Server::handle_client_rename(boost::intrusive_ptr const&)+0x259d) [0x56361d69597d] 2024-03-24T00:24:46.113 INFO:journalctl@ceph.mds.j.smithi118.stdout:Mar 24 00:24:45 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-j[96876]: 7: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x56361d6ab233] 2024-03-24T00:24:46.113 INFO:journalctl@ceph.mds.j.smithi118.stdout:Mar 24 00:24:45 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-j[96876]: 8: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x56361d6ad197] 2024-03-24T00:24:46.113 INFO:journalctl@ceph.mds.j.smithi118.stdout:Mar 24 00:24:45 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-j[96876]: 9: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x56361d5ac11b] 2024-03-24T00:24:46.113 INFO:journalctl@ceph.mds.j.smithi118.stdout:Mar 24 00:24:45 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-j[96876]: 10: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x56361d5ac6bc] 2024-03-24T00:24:46.113 INFO:journalctl@ceph.mds.j.smithi118.stdout:Mar 24 00:24:45 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-j[96876]: 11: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x56361d5961ff] 2024-03-24T00:24:46.113 INFO:journalctl@ceph.mds.j.smithi118.stdout:Mar 24 00:24:45 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-j[96876]: 12: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7fa8acdeb488] 2024-03-24T00:24:46.113 INFO:journalctl@ceph.mds.j.smithi118.stdout:Mar 24 00:24:45 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-j[96876]: 13: (DispatchQueue::entry()+0x51f) [0x7fa8acde860f] 2024-03-24T00:24:46.113 INFO:journalctl@ceph.mds.j.smithi118.stdout:Mar 24 00:24:45 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-j[96876]: 14: (DispatchQueue::DispatchThread::entry()+0x11) [0x7fa8acebbe81] 2024-03-24T00:24:46.113 INFO:journalctl@ceph.mds.j.smithi118.stdout:Mar 24 00:24:45 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-j[96876]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7fa8ab8d61ca] 2024-03-24T00:24:46.114 INFO:journalctl@ceph.mds.j.smithi118.stdout:Mar 24 00:24:45 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-j[96876]: 16: clone() 2024-03-24T00:24:46.114 INFO:journalctl@ceph.mds.j.smithi118.stdout:Mar 24 00:24:45 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-j[96876]: 2024-03-24T00:24:46.114 INFO:journalctl@ceph.mds.j.smithi118.stdout:Mar 24 00:24:45 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-j[96876]: 0> 2024-03-24T00:24:45.828+0000 7fa8a1200700 -1 *** Caught signal (Aborted) ** 2024-03-24T00:24:46.114 INFO:journalctl@ceph.mds.j.smithi118.stdout:Mar 24 00:24:45 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-j[96876]: in thread 7fa8a1200700 thread_name:ms_dispatch 2024-03-24T00:24:46.114 INFO:journalctl@ceph.mds.j.smithi118.stdout:Mar 24 00:24:45 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-j[96876]: 2024-03-24T00:24:46.114 INFO:journalctl@ceph.mds.j.smithi118.stdout:Mar 24 00:24:45 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-j[96876]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-24T00:24:46.114 INFO:journalctl@ceph.mds.j.smithi118.stdout:Mar 24 00:24:45 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-j[96876]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7fa8ab8e0d20] 2024-03-24T00:24:46.114 INFO:journalctl@ceph.mds.j.smithi118.stdout:Mar 24 00:24:45 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-j[96876]: 2: gsignal() 2024-03-24T00:24:46.114 INFO:journalctl@ceph.mds.j.smithi118.stdout:Mar 24 00:24:45 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-j[96876]: 3: abort() 2024-03-24T00:24:46.114 INFO:journalctl@ceph.mds.j.smithi118.stdout:Mar 24 00:24:45 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-j[96876]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x18f) [0x7fa8acb594fb] 2024-03-24T00:24:46.114 INFO:journalctl@ceph.mds.j.smithi118.stdout:Mar 24 00:24:45 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-j[96876]: 5: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7fa8acb59667] 2024-03-24T00:24:46.114 INFO:journalctl@ceph.mds.j.smithi118.stdout:Mar 24 00:24:45 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-j[96876]: 6: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x56361d7de223] 2024-03-24T00:24:46.114 INFO:journalctl@ceph.mds.j.smithi118.stdout:Mar 24 00:24:45 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-j[96876]: 7: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x56361d6585d6] 2024-03-24T00:24:46.115 INFO:journalctl@ceph.mds.d.smithi118.stdout:Mar 24 00:24:45 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-d[96871]: 6: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x558dc6e18223] 2024-03-24T00:24:46.115 INFO:journalctl@ceph.mds.d.smithi118.stdout:Mar 24 00:24:45 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-d[96871]: 7: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x558dc6c925d6] 2024-03-24T00:24:46.115 INFO:journalctl@ceph.mds.d.smithi118.stdout:Mar 24 00:24:45 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-d[96871]: 8: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x558dc6c92da1] 2024-03-24T00:24:46.115 INFO:journalctl@ceph.mds.d.smithi118.stdout:Mar 24 00:24:45 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-d[96871]: 9: (Server::handle_client_rename(boost::intrusive_ptr const&)+0x259d) [0x558dc6ccf97d] 2024-03-24T00:24:46.115 INFO:journalctl@ceph.mds.d.smithi118.stdout:Mar 24 00:24:45 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-d[96871]: 10: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x558dc6ce5233] 2024-03-24T00:24:46.115 INFO:journalctl@ceph.mds.d.smithi118.stdout:Mar 24 00:24:45 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-d[96871]: 11: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x558dc6ce7197] 2024-03-24T00:24:46.115 INFO:journalctl@ceph.mds.d.smithi118.stdout:Mar 24 00:24:45 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-d[96871]: 12: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x558dc6be611b] 2024-03-24T00:24:46.115 INFO:journalctl@ceph.mds.d.smithi118.stdout:Mar 24 00:24:45 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-d[96871]: 13: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x558dc6be66bc] 2024-03-24T00:24:46.115 INFO:journalctl@ceph.mds.d.smithi118.stdout:Mar 24 00:24:45 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-d[96871]: 14: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x558dc6bd01ff] 2024-03-24T00:24:46.115 INFO:journalctl@ceph.mds.d.smithi118.stdout:Mar 24 00:24:45 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-d[96871]: 15: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7f500ccb5488] 2024-03-24T00:24:46.115 INFO:journalctl@ceph.mds.d.smithi118.stdout:Mar 24 00:24:45 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-d[96871]: 16: (DispatchQueue::entry()+0x51f) [0x7f500ccb260f] 2024-03-24T00:24:46.115 INFO:journalctl@ceph.mds.d.smithi118.stdout:Mar 24 00:24:45 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-d[96871]: 17: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f500cd85e81] 2024-03-24T00:24:46.115 INFO:journalctl@ceph.mds.d.smithi118.stdout:Mar 24 00:24:45 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-d[96871]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7f500b7a01ca] 2024-03-24T00:24:46.116 INFO:journalctl@ceph.mds.d.smithi118.stdout:Mar 24 00:24:45 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-d[96871]: 19: clone() 2024-03-24T00:24:46.116 INFO:journalctl@ceph.mds.d.smithi118.stdout:Mar 24 00:24:45 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-d[96871]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-03-24T00:24:46.116 INFO:journalctl@ceph.mds.d.smithi118.stdout:Mar 24 00:24:45 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-d[96871]: 2024-03-24T00:24:46.116 INFO:journalctl@ceph.mds.d.smithi118.stdout:Mar 24 00:24:45 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-d[96871]: -1> 2024-03-24T00:24:45.867+0000 7f5001200700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: In function 'void Locker::set_xlocks_done(MutationImpl*, bool)' thread 7f5001200700 time 2024-03-24T00:24:45.866817+0000 2024-03-24T00:24:46.116 INFO:journalctl@ceph.mds.d.smithi118.stdout:Mar 24 00:24:45 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-d[96871]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: 626: FAILED ceph_assert(obj->is_auth()) 2024-03-24T00:24:46.116 INFO:journalctl@ceph.mds.d.smithi118.stdout:Mar 24 00:24:45 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-d[96871]: 2024-03-24T00:24:46.116 INFO:journalctl@ceph.mds.d.smithi118.stdout:Mar 24 00:24:45 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-d[96871]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-24T00:24:46.116 INFO:journalctl@ceph.mds.d.smithi118.stdout:Mar 24 00:24:45 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-d[96871]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x135) [0x7f500ca234a1] 2024-03-24T00:24:46.116 INFO:journalctl@ceph.mds.d.smithi118.stdout:Mar 24 00:24:45 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-d[96871]: 2: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7f500ca23667] 2024-03-24T00:24:46.116 INFO:journalctl@ceph.mds.j.smithi118.stdout:Mar 24 00:24:45 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-j[96876]: 8: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x56361d658da1] 2024-03-24T00:24:46.116 INFO:journalctl@ceph.mds.j.smithi118.stdout:Mar 24 00:24:45 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-j[96876]: 9: (Server::handle_client_rename(boost::intrusive_ptr const&)+0x259d) [0x56361d69597d] 2024-03-24T00:24:46.117 INFO:journalctl@ceph.mds.j.smithi118.stdout:Mar 24 00:24:45 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-j[96876]: 10: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x56361d6ab233] 2024-03-24T00:24:46.117 INFO:journalctl@ceph.mds.j.smithi118.stdout:Mar 24 00:24:45 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-j[96876]: 11: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x56361d6ad197] 2024-03-24T00:24:46.117 INFO:journalctl@ceph.mds.j.smithi118.stdout:Mar 24 00:24:45 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-j[96876]: 12: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x56361d5ac11b] 2024-03-24T00:24:46.117 INFO:journalctl@ceph.mds.j.smithi118.stdout:Mar 24 00:24:45 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-j[96876]: 13: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x56361d5ac6bc] 2024-03-24T00:24:46.117 INFO:journalctl@ceph.mds.j.smithi118.stdout:Mar 24 00:24:45 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-j[96876]: 14: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x56361d5961ff] 2024-03-24T00:24:46.117 INFO:journalctl@ceph.mds.j.smithi118.stdout:Mar 24 00:24:45 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-j[96876]: 15: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7fa8acdeb488] 2024-03-24T00:24:46.117 INFO:journalctl@ceph.mds.j.smithi118.stdout:Mar 24 00:24:45 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-j[96876]: 16: (DispatchQueue::entry()+0x51f) [0x7fa8acde860f] 2024-03-24T00:24:46.117 INFO:journalctl@ceph.mds.j.smithi118.stdout:Mar 24 00:24:45 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-j[96876]: 17: (DispatchQueue::DispatchThread::entry()+0x11) [0x7fa8acebbe81] 2024-03-24T00:24:46.117 INFO:journalctl@ceph.mds.j.smithi118.stdout:Mar 24 00:24:45 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-j[96876]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7fa8ab8d61ca] 2024-03-24T00:24:46.117 INFO:journalctl@ceph.mds.j.smithi118.stdout:Mar 24 00:24:45 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-j[96876]: 19: clone() 2024-03-24T00:24:46.117 INFO:journalctl@ceph.mds.j.smithi118.stdout:Mar 24 00:24:45 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-j[96876]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-03-24T00:24:46.117 INFO:journalctl@ceph.mds.j.smithi118.stdout:Mar 24 00:24:45 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-j[96876]: 2024-03-24T00:24:46.117 INFO:journalctl@ceph.mds.j.smithi118.stdout:Mar 24 00:24:45 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-j[96876]: -1> 2024-03-24T00:24:45.825+0000 7fa8a1200700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: In function 'void Locker::set_xlocks_done(MutationImpl*, bool)' thread 7fa8a1200700 time 2024-03-24T00:24:45.825535+0000 2024-03-24T00:24:46.118 INFO:journalctl@ceph.mds.j.smithi118.stdout:Mar 24 00:24:45 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-j[96876]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: 626: FAILED ceph_assert(obj->is_auth()) 2024-03-24T00:24:46.118 INFO:journalctl@ceph.mds.j.smithi118.stdout:Mar 24 00:24:45 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-j[96876]: 2024-03-24T00:24:46.118 INFO:journalctl@ceph.mds.j.smithi118.stdout:Mar 24 00:24:45 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-j[96876]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-24T00:24:46.118 INFO:journalctl@ceph.mds.j.smithi118.stdout:Mar 24 00:24:45 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-j[96876]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x135) [0x7fa8acb594a1] 2024-03-24T00:24:46.118 INFO:journalctl@ceph.mds.j.smithi118.stdout:Mar 24 00:24:45 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-j[96876]: 2: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7fa8acb59667] 2024-03-24T00:24:46.118 INFO:journalctl@ceph.mds.j.smithi118.stdout:Mar 24 00:24:45 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-j[96876]: 3: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x56361d7de223] 2024-03-24T00:24:46.118 INFO:journalctl@ceph.mds.j.smithi118.stdout:Mar 24 00:24:45 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-j[96876]: 4: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x56361d6585d6] 2024-03-24T00:24:46.118 INFO:journalctl@ceph.mds.d.smithi118.stdout:Mar 24 00:24:45 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-d[96871]: 3: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x558dc6e18223] 2024-03-24T00:24:46.118 INFO:journalctl@ceph.mds.d.smithi118.stdout:Mar 24 00:24:45 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-d[96871]: 4: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x558dc6c925d6] 2024-03-24T00:24:46.118 INFO:journalctl@ceph.mds.d.smithi118.stdout:Mar 24 00:24:45 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-d[96871]: 5: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x558dc6c92da1] 2024-03-24T00:24:46.118 INFO:journalctl@ceph.mds.d.smithi118.stdout:Mar 24 00:24:45 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-d[96871]: 6: (Server::handle_client_rename(boost::intrusive_ptr const&)+0x259d) [0x558dc6ccf97d] 2024-03-24T00:24:46.119 INFO:journalctl@ceph.mds.d.smithi118.stdout:Mar 24 00:24:45 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-d[96871]: 7: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x558dc6ce5233] 2024-03-24T00:24:46.119 INFO:journalctl@ceph.mds.d.smithi118.stdout:Mar 24 00:24:45 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-d[96871]: 8: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x558dc6ce7197] 2024-03-24T00:24:46.119 INFO:journalctl@ceph.mds.d.smithi118.stdout:Mar 24 00:24:45 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-d[96871]: 9: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x558dc6be611b] 2024-03-24T00:24:46.119 INFO:journalctl@ceph.mds.d.smithi118.stdout:Mar 24 00:24:45 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-d[96871]: 10: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x558dc6be66bc] 2024-03-24T00:24:46.119 INFO:journalctl@ceph.mds.d.smithi118.stdout:Mar 24 00:24:45 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-d[96871]: 11: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x558dc6bd01ff] 2024-03-24T00:24:46.119 INFO:journalctl@ceph.mds.d.smithi118.stdout:Mar 24 00:24:45 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-d[96871]: 12: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7f500ccb5488] 2024-03-24T00:24:46.119 INFO:journalctl@ceph.mds.d.smithi118.stdout:Mar 24 00:24:45 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-d[96871]: 13: (DispatchQueue::entry()+0x51f) [0x7f500ccb260f] 2024-03-24T00:24:46.119 INFO:journalctl@ceph.mds.d.smithi118.stdout:Mar 24 00:24:45 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-d[96871]: 14: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f500cd85e81] 2024-03-24T00:24:46.119 INFO:journalctl@ceph.mds.d.smithi118.stdout:Mar 24 00:24:45 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-d[96871]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7f500b7a01ca] 2024-03-24T00:24:46.119 INFO:journalctl@ceph.mds.d.smithi118.stdout:Mar 24 00:24:45 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-d[96871]: 16: clone() 2024-03-24T00:24:46.119 INFO:journalctl@ceph.mds.d.smithi118.stdout:Mar 24 00:24:45 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-d[96871]: 2024-03-24T00:24:46.119 INFO:journalctl@ceph.mds.d.smithi118.stdout:Mar 24 00:24:45 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-d[96871]: 0> 2024-03-24T00:24:45.869+0000 7f5001200700 -1 *** Caught signal (Aborted) ** 2024-03-24T00:24:46.119 INFO:journalctl@ceph.mds.d.smithi118.stdout:Mar 24 00:24:45 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-d[96871]: in thread 7f5001200700 thread_name:ms_dispatch 2024-03-24T00:24:46.119 INFO:journalctl@ceph.mds.d.smithi118.stdout:Mar 24 00:24:45 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-d[96871]: 2024-03-24T00:24:46.119 INFO:journalctl@ceph.mds.d.smithi118.stdout:Mar 24 00:24:45 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-d[96871]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-24T00:24:46.120 INFO:journalctl@ceph.mds.d.smithi118.stdout:Mar 24 00:24:45 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-d[96871]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f500b7aad20] 2024-03-24T00:24:46.120 INFO:journalctl@ceph.mds.d.smithi118.stdout:Mar 24 00:24:45 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-d[96871]: 2: gsignal() 2024-03-24T00:24:46.120 INFO:journalctl@ceph.mds.d.smithi118.stdout:Mar 24 00:24:45 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-d[96871]: 3: abort() 2024-03-24T00:24:46.120 INFO:journalctl@ceph.mds.d.smithi118.stdout:Mar 24 00:24:45 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-d[96871]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x18f) [0x7f500ca234fb] 2024-03-24T00:24:46.120 INFO:journalctl@ceph.mds.d.smithi118.stdout:Mar 24 00:24:45 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-d[96871]: 5: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7f500ca23667] 2024-03-24T00:24:46.120 INFO:journalctl@ceph.mds.d.smithi118.stdout:Mar 24 00:24:45 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-d[96871]: 6: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x558dc6e18223] 2024-03-24T00:24:46.120 INFO:journalctl@ceph.mds.d.smithi118.stdout:Mar 24 00:24:45 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-d[96871]: 7: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x558dc6c925d6] 2024-03-24T00:24:46.120 INFO:journalctl@ceph.mds.j.smithi118.stdout:Mar 24 00:24:45 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-j[96876]: 5: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x56361d658da1] 2024-03-24T00:24:46.121 INFO:journalctl@ceph.mds.j.smithi118.stdout:Mar 24 00:24:45 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-j[96876]: 6: (Server::handle_client_rename(boost::intrusive_ptr const&)+0x259d) [0x56361d69597d] 2024-03-24T00:24:46.121 INFO:journalctl@ceph.mds.j.smithi118.stdout:Mar 24 00:24:45 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-j[96876]: 7: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x56361d6ab233] 2024-03-24T00:24:46.121 INFO:journalctl@ceph.mds.j.smithi118.stdout:Mar 24 00:24:45 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-j[96876]: 8: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x56361d6ad197] 2024-03-24T00:24:46.121 INFO:journalctl@ceph.mds.j.smithi118.stdout:Mar 24 00:24:45 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-j[96876]: 9: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x56361d5ac11b] 2024-03-24T00:24:46.121 INFO:journalctl@ceph.mds.j.smithi118.stdout:Mar 24 00:24:45 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-j[96876]: 10: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x56361d5ac6bc] 2024-03-24T00:24:46.121 INFO:journalctl@ceph.mds.j.smithi118.stdout:Mar 24 00:24:45 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-j[96876]: 11: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x56361d5961ff] 2024-03-24T00:24:46.121 INFO:journalctl@ceph.mds.j.smithi118.stdout:Mar 24 00:24:45 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-j[96876]: 12: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7fa8acdeb488] 2024-03-24T00:24:46.121 INFO:journalctl@ceph.mds.j.smithi118.stdout:Mar 24 00:24:45 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-j[96876]: 13: (DispatchQueue::entry()+0x51f) [0x7fa8acde860f] 2024-03-24T00:24:46.121 INFO:journalctl@ceph.mds.j.smithi118.stdout:Mar 24 00:24:45 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-j[96876]: 14: (DispatchQueue::DispatchThread::entry()+0x11) [0x7fa8acebbe81] 2024-03-24T00:24:46.121 INFO:journalctl@ceph.mds.j.smithi118.stdout:Mar 24 00:24:45 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-j[96876]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7fa8ab8d61ca] 2024-03-24T00:24:46.121 INFO:journalctl@ceph.mds.j.smithi118.stdout:Mar 24 00:24:45 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-j[96876]: 16: clone() 2024-03-24T00:24:46.121 INFO:journalctl@ceph.mds.j.smithi118.stdout:Mar 24 00:24:45 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-j[96876]: 2024-03-24T00:24:46.121 INFO:journalctl@ceph.mds.j.smithi118.stdout:Mar 24 00:24:45 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-j[96876]: 0> 2024-03-24T00:24:45.828+0000 7fa8a1200700 -1 *** Caught signal (Aborted) ** 2024-03-24T00:24:46.121 INFO:journalctl@ceph.mds.j.smithi118.stdout:Mar 24 00:24:45 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-j[96876]: in thread 7fa8a1200700 thread_name:ms_dispatch 2024-03-24T00:24:46.121 INFO:journalctl@ceph.mds.j.smithi118.stdout:Mar 24 00:24:45 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-j[96876]: 2024-03-24T00:24:46.122 INFO:journalctl@ceph.mds.j.smithi118.stdout:Mar 24 00:24:45 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-j[96876]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-24T00:24:46.122 INFO:journalctl@ceph.mds.j.smithi118.stdout:Mar 24 00:24:45 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-j[96876]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7fa8ab8e0d20] 2024-03-24T00:24:46.122 INFO:journalctl@ceph.mds.j.smithi118.stdout:Mar 24 00:24:45 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-j[96876]: 2: gsignal() 2024-03-24T00:24:46.122 INFO:journalctl@ceph.mds.j.smithi118.stdout:Mar 24 00:24:45 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-j[96876]: 3: abort() 2024-03-24T00:24:46.122 INFO:journalctl@ceph.mds.j.smithi118.stdout:Mar 24 00:24:45 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-j[96876]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x18f) [0x7fa8acb594fb] 2024-03-24T00:24:46.122 INFO:journalctl@ceph.mds.j.smithi118.stdout:Mar 24 00:24:45 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-j[96876]: 5: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7fa8acb59667] 2024-03-24T00:24:46.122 INFO:journalctl@ceph.mds.j.smithi118.stdout:Mar 24 00:24:45 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-j[96876]: 6: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x56361d7de223] 2024-03-24T00:24:46.122 INFO:journalctl@ceph.mds.j.smithi118.stdout:Mar 24 00:24:45 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-j[96876]: 7: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x56361d6585d6] 2024-03-24T00:24:46.122 INFO:journalctl@ceph.mds.j.smithi118.stdout:Mar 24 00:24:45 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-j[96876]: 8: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x56361d658da1] 2024-03-24T00:24:46.122 INFO:journalctl@ceph.mds.j.smithi118.stdout:Mar 24 00:24:45 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-j[96876]: 9: (Server::handle_client_rename(boost::intrusive_ptr const&)+0x259d) [0x56361d69597d] 2024-03-24T00:24:46.122 INFO:journalctl@ceph.mds.j.smithi118.stdout:Mar 24 00:24:45 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-j[96876]: 10: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x56361d6ab233] 2024-03-24T00:24:46.122 INFO:journalctl@ceph.mds.j.smithi118.stdout:Mar 24 00:24:45 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-j[96876]: 11: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x56361d6ad197] 2024-03-24T00:24:46.122 INFO:journalctl@ceph.mds.j.smithi118.stdout:Mar 24 00:24:45 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-j[96876]: 12: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x56361d5ac11b] 2024-03-24T00:24:46.123 INFO:journalctl@ceph.mds.j.smithi118.stdout:Mar 24 00:24:45 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-j[96876]: 13: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x56361d5ac6bc] 2024-03-24T00:24:46.123 INFO:journalctl@ceph.mds.j.smithi118.stdout:Mar 24 00:24:45 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-j[96876]: 14: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x56361d5961ff] 2024-03-24T00:24:46.123 INFO:journalctl@ceph.mds.j.smithi118.stdout:Mar 24 00:24:45 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-j[96876]: 15: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7fa8acdeb488] 2024-03-24T00:24:46.123 INFO:journalctl@ceph.mds.j.smithi118.stdout:Mar 24 00:24:45 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-j[96876]: 16: (DispatchQueue::entry()+0x51f) [0x7fa8acde860f] 2024-03-24T00:24:46.123 INFO:journalctl@ceph.mds.j.smithi118.stdout:Mar 24 00:24:45 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-j[96876]: 17: (DispatchQueue::DispatchThread::entry()+0x11) [0x7fa8acebbe81] 2024-03-24T00:24:46.123 INFO:journalctl@ceph.mds.j.smithi118.stdout:Mar 24 00:24:45 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-j[96876]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7fa8ab8d61ca] 2024-03-24T00:24:46.123 INFO:journalctl@ceph.mds.j.smithi118.stdout:Mar 24 00:24:45 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-j[96876]: 19: clone() 2024-03-24T00:24:46.123 INFO:journalctl@ceph.mds.j.smithi118.stdout:Mar 24 00:24:45 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-j[96876]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-03-24T00:24:46.123 INFO:journalctl@ceph.mds.j.smithi118.stdout:Mar 24 00:24:45 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-j[96876]: 2024-03-24T00:24:46.123 INFO:journalctl@ceph.mds.j.smithi118.stdout:Mar 24 00:24:45 smithi118 podman[96951]: 2024-03-24 00:24:45.944337241 +0000 UTC m=+0.040316357 container died a714c7b16352e17f582f5c3983c5a7862dfa479508c596f53fef9d23d615a827 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-j, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240319, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, ceph=True, GIT_CLEAN=True, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c) 2024-03-24T00:24:46.124 INFO:journalctl@ceph.mds.d.smithi118.stdout:Mar 24 00:24:45 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-d[96871]: 8: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x558dc6c92da1] 2024-03-24T00:24:46.124 INFO:journalctl@ceph.mds.d.smithi118.stdout:Mar 24 00:24:45 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-d[96871]: 9: (Server::handle_client_rename(boost::intrusive_ptr const&)+0x259d) [0x558dc6ccf97d] 2024-03-24T00:24:46.124 INFO:journalctl@ceph.mds.d.smithi118.stdout:Mar 24 00:24:45 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-d[96871]: 10: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x558dc6ce5233] 2024-03-24T00:24:46.124 INFO:journalctl@ceph.mds.d.smithi118.stdout:Mar 24 00:24:45 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-d[96871]: 11: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x558dc6ce7197] 2024-03-24T00:24:46.124 INFO:journalctl@ceph.mds.d.smithi118.stdout:Mar 24 00:24:45 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-d[96871]: 12: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x558dc6be611b] 2024-03-24T00:24:46.124 INFO:journalctl@ceph.mds.d.smithi118.stdout:Mar 24 00:24:45 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-d[96871]: 13: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x558dc6be66bc] 2024-03-24T00:24:46.124 INFO:journalctl@ceph.mds.d.smithi118.stdout:Mar 24 00:24:45 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-d[96871]: 14: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x558dc6bd01ff] 2024-03-24T00:24:46.124 INFO:journalctl@ceph.mds.d.smithi118.stdout:Mar 24 00:24:45 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-d[96871]: 15: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7f500ccb5488] 2024-03-24T00:24:46.124 INFO:journalctl@ceph.mds.d.smithi118.stdout:Mar 24 00:24:45 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-d[96871]: 16: (DispatchQueue::entry()+0x51f) [0x7f500ccb260f] 2024-03-24T00:24:46.124 INFO:journalctl@ceph.mds.d.smithi118.stdout:Mar 24 00:24:45 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-d[96871]: 17: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f500cd85e81] 2024-03-24T00:24:46.124 INFO:journalctl@ceph.mds.d.smithi118.stdout:Mar 24 00:24:45 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-d[96871]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7f500b7a01ca] 2024-03-24T00:24:46.124 INFO:journalctl@ceph.mds.d.smithi118.stdout:Mar 24 00:24:45 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-d[96871]: 19: clone() 2024-03-24T00:24:46.125 INFO:journalctl@ceph.mds.d.smithi118.stdout:Mar 24 00:24:45 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-d[96871]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-03-24T00:24:46.125 INFO:journalctl@ceph.mds.d.smithi118.stdout:Mar 24 00:24:45 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-d[96871]: 2024-03-24T00:24:46.125 INFO:journalctl@ceph.mds.d.smithi118.stdout:Mar 24 00:24:45 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-d[96871]: -5> 2024-03-24T00:24:45.867+0000 7f5001200700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: In function 'void Locker::set_xlocks_done(MutationImpl*, bool)' thread 7f5001200700 time 2024-03-24T00:24:45.866817+0000 2024-03-24T00:24:46.125 INFO:journalctl@ceph.mds.d.smithi118.stdout:Mar 24 00:24:45 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-d[96871]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: 626: FAILED ceph_assert(obj->is_auth()) 2024-03-24T00:24:46.125 INFO:journalctl@ceph.mds.d.smithi118.stdout:Mar 24 00:24:45 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-d[96871]: 2024-03-24T00:24:46.125 INFO:journalctl@ceph.mds.d.smithi118.stdout:Mar 24 00:24:45 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-d[96871]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-24T00:24:46.125 INFO:journalctl@ceph.mds.d.smithi118.stdout:Mar 24 00:24:45 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-d[96871]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x135) [0x7f500ca234a1] 2024-03-24T00:24:46.125 INFO:journalctl@ceph.mds.d.smithi118.stdout:Mar 24 00:24:45 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-d[96871]: 2: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7f500ca23667] 2024-03-24T00:24:46.125 INFO:journalctl@ceph.mds.d.smithi118.stdout:Mar 24 00:24:45 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-d[96871]: 3: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x558dc6e18223] 2024-03-24T00:24:46.125 INFO:journalctl@ceph.mds.d.smithi118.stdout:Mar 24 00:24:45 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-d[96871]: 4: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x558dc6c925d6] 2024-03-24T00:24:46.125 INFO:journalctl@ceph.mds.d.smithi118.stdout:Mar 24 00:24:45 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-d[96871]: 5: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x558dc6c92da1] 2024-03-24T00:24:46.125 INFO:journalctl@ceph.mds.d.smithi118.stdout:Mar 24 00:24:45 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-d[96871]: 6: (Server::handle_client_rename(boost::intrusive_ptr const&)+0x259d) [0x558dc6ccf97d] 2024-03-24T00:24:46.125 INFO:journalctl@ceph.mds.d.smithi118.stdout:Mar 24 00:24:45 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-d[96871]: 7: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x558dc6ce5233] 2024-03-24T00:24:46.125 INFO:journalctl@ceph.mds.d.smithi118.stdout:Mar 24 00:24:45 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-d[96871]: 8: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x558dc6ce7197] 2024-03-24T00:24:46.126 INFO:journalctl@ceph.mds.d.smithi118.stdout:Mar 24 00:24:45 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-d[96871]: 9: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x558dc6be611b] 2024-03-24T00:24:46.126 INFO:journalctl@ceph.mds.d.smithi118.stdout:Mar 24 00:24:45 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-d[96871]: 10: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x558dc6be66bc] 2024-03-24T00:24:46.126 INFO:journalctl@ceph.mds.d.smithi118.stdout:Mar 24 00:24:45 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-d[96871]: 11: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x558dc6bd01ff] 2024-03-24T00:24:46.126 INFO:journalctl@ceph.mds.d.smithi118.stdout:Mar 24 00:24:45 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-d[96871]: 12: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7f500ccb5488] 2024-03-24T00:24:46.126 INFO:journalctl@ceph.mds.d.smithi118.stdout:Mar 24 00:24:45 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-d[96871]: 13: (DispatchQueue::entry()+0x51f) [0x7f500ccb260f] 2024-03-24T00:24:46.126 INFO:journalctl@ceph.mds.d.smithi118.stdout:Mar 24 00:24:45 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-d[96871]: 14: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f500cd85e81] 2024-03-24T00:24:46.126 INFO:journalctl@ceph.mds.d.smithi118.stdout:Mar 24 00:24:45 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-d[96871]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7f500b7a01ca] 2024-03-24T00:24:46.126 INFO:journalctl@ceph.mds.d.smithi118.stdout:Mar 24 00:24:45 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-d[96871]: 16: clone() 2024-03-24T00:24:46.126 INFO:journalctl@ceph.mds.d.smithi118.stdout:Mar 24 00:24:45 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-d[96871]: 2024-03-24T00:24:46.126 INFO:journalctl@ceph.mds.d.smithi118.stdout:Mar 24 00:24:45 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-d[96871]: -4> 2024-03-24T00:24:45.869+0000 7f5001200700 -1 *** Caught signal (Aborted) ** 2024-03-24T00:24:46.126 INFO:journalctl@ceph.mds.d.smithi118.stdout:Mar 24 00:24:45 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-d[96871]: in thread 7f5001200700 thread_name:ms_dispatch 2024-03-24T00:24:46.126 INFO:journalctl@ceph.mds.d.smithi118.stdout:Mar 24 00:24:45 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-d[96871]: 2024-03-24T00:24:46.126 INFO:journalctl@ceph.mds.d.smithi118.stdout:Mar 24 00:24:45 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-d[96871]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-24T00:24:46.126 INFO:journalctl@ceph.mds.d.smithi118.stdout:Mar 24 00:24:45 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-d[96871]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7f500b7aad20] 2024-03-24T00:24:46.127 INFO:journalctl@ceph.mds.d.smithi118.stdout:Mar 24 00:24:45 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-d[96871]: 2: gsignal() 2024-03-24T00:24:46.127 INFO:journalctl@ceph.mds.d.smithi118.stdout:Mar 24 00:24:45 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-d[96871]: 3: abort() 2024-03-24T00:24:46.127 INFO:journalctl@ceph.mds.d.smithi118.stdout:Mar 24 00:24:45 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-d[96871]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x18f) [0x7f500ca234fb] 2024-03-24T00:24:46.127 INFO:journalctl@ceph.mds.d.smithi118.stdout:Mar 24 00:24:45 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-d[96871]: 5: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7f500ca23667] 2024-03-24T00:24:46.127 INFO:journalctl@ceph.mds.d.smithi118.stdout:Mar 24 00:24:45 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-d[96871]: 6: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x558dc6e18223] 2024-03-24T00:24:46.127 INFO:journalctl@ceph.mds.d.smithi118.stdout:Mar 24 00:24:45 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-d[96871]: 7: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x558dc6c925d6] 2024-03-24T00:24:46.127 INFO:journalctl@ceph.mds.d.smithi118.stdout:Mar 24 00:24:45 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-d[96871]: 8: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x558dc6c92da1] 2024-03-24T00:24:46.127 INFO:journalctl@ceph.mds.d.smithi118.stdout:Mar 24 00:24:45 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-d[96871]: 9: (Server::handle_client_rename(boost::intrusive_ptr const&)+0x259d) [0x558dc6ccf97d] 2024-03-24T00:24:46.127 INFO:journalctl@ceph.mds.d.smithi118.stdout:Mar 24 00:24:45 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-d[96871]: 10: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x558dc6ce5233] 2024-03-24T00:24:46.127 INFO:journalctl@ceph.mds.d.smithi118.stdout:Mar 24 00:24:45 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-d[96871]: 11: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x558dc6ce7197] 2024-03-24T00:24:46.127 INFO:journalctl@ceph.mds.d.smithi118.stdout:Mar 24 00:24:45 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-d[96871]: 12: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x558dc6be611b] 2024-03-24T00:24:46.127 INFO:journalctl@ceph.mds.d.smithi118.stdout:Mar 24 00:24:45 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-d[96871]: 13: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x558dc6be66bc] 2024-03-24T00:24:46.127 INFO:journalctl@ceph.mds.d.smithi118.stdout:Mar 24 00:24:45 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-d[96871]: 14: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x558dc6bd01ff] 2024-03-24T00:24:46.127 INFO:journalctl@ceph.mds.d.smithi118.stdout:Mar 24 00:24:45 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-d[96871]: 15: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7f500ccb5488] 2024-03-24T00:24:46.128 INFO:journalctl@ceph.mds.d.smithi118.stdout:Mar 24 00:24:45 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-d[96871]: 16: (DispatchQueue::entry()+0x51f) [0x7f500ccb260f] 2024-03-24T00:24:46.128 INFO:journalctl@ceph.mds.d.smithi118.stdout:Mar 24 00:24:45 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-d[96871]: 17: (DispatchQueue::DispatchThread::entry()+0x11) [0x7f500cd85e81] 2024-03-24T00:24:46.128 INFO:journalctl@ceph.mds.d.smithi118.stdout:Mar 24 00:24:45 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-d[96871]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7f500b7a01ca] 2024-03-24T00:24:46.128 INFO:journalctl@ceph.mds.d.smithi118.stdout:Mar 24 00:24:45 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-d[96871]: 19: clone() 2024-03-24T00:24:46.128 INFO:journalctl@ceph.mds.d.smithi118.stdout:Mar 24 00:24:45 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-d[96871]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-03-24T00:24:46.128 INFO:journalctl@ceph.mds.d.smithi118.stdout:Mar 24 00:24:45 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-d[96871]: 2024-03-24T00:24:46.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:24:45 smithi189 ceph-mon[36187]: pgmap v701: 97 pgs: 97 active+clean; 146 MiB data, 881 MiB used, 1.0 TiB / 1.0 TiB avail; 38 KiB/s rd, 13 KiB/s wr, 11 op/s 2024-03-24T00:24:46.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:24:45 smithi189 ceph-mon[36187]: mds.? [v2:172.21.15.118:6836/2587227307,v1:172.21.15.118:6837/2587227307] up:rejoin 2024-03-24T00:24:46.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:24:45 smithi189 ceph-mon[36187]: mds.? [v2:172.21.15.118:6834/456491655,v1:172.21.15.118:6835/456491655] up:rejoin 2024-03-24T00:24:46.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:24:45 smithi189 ceph-mon[36187]: fsmap cephfs:5/5 3 up:standby-replay 2 up:rejoin 3 up:active 2024-03-24T00:24:46.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:24:45 smithi189 ceph-mon[36187]: daemon mds.j is now active in filesystem cephfs as rank 3 2024-03-24T00:24:46.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:24:45 smithi189 ceph-mon[36187]: daemon mds.d is now active in filesystem cephfs as rank 1 2024-03-24T00:24:46.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:24:45 smithi193 ceph-mon[36878]: pgmap v701: 97 pgs: 97 active+clean; 146 MiB data, 881 MiB used, 1.0 TiB / 1.0 TiB avail; 38 KiB/s rd, 13 KiB/s wr, 11 op/s 2024-03-24T00:24:46.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:24:45 smithi193 ceph-mon[36878]: mds.? [v2:172.21.15.118:6836/2587227307,v1:172.21.15.118:6837/2587227307] up:rejoin 2024-03-24T00:24:46.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:24:45 smithi193 ceph-mon[36878]: mds.? [v2:172.21.15.118:6834/456491655,v1:172.21.15.118:6835/456491655] up:rejoin 2024-03-24T00:24:46.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:24:45 smithi193 ceph-mon[36878]: fsmap cephfs:5/5 3 up:standby-replay 2 up:rejoin 3 up:active 2024-03-24T00:24:46.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:24:45 smithi193 ceph-mon[36878]: daemon mds.j is now active in filesystem cephfs as rank 3 2024-03-24T00:24:46.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:24:45 smithi193 ceph-mon[36878]: daemon mds.d is now active in filesystem cephfs as rank 1 2024-03-24T00:24:47.196 INFO:journalctl@ceph.mds.d.smithi118.stdout:Mar 24 00:24:46 smithi118 podman[96961]: 2024-03-24 00:24:46.84202909 +0000 UTC m=+0.874756214 container died 76fb46d4f076316c4871ba3ae6e8d8a3cde9e622ec7c972e9c4a6c2c72115c46 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-d, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240319, org.label-schema.license=GPLv2, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, GIT_BRANCH=HEAD) 2024-03-24T00:24:47.197 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:24:46 smithi118 ceph-mon[31535]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-03-24T00:24:47.197 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:24:46 smithi118 ceph-mon[31535]: Cluster is now healthy 2024-03-24T00:24:47.197 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:24:46 smithi118 ceph-mon[31535]: mds.? [v2:172.21.15.118:6836/2587227307,v1:172.21.15.118:6837/2587227307] up:active 2024-03-24T00:24:47.197 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:24:46 smithi118 ceph-mon[31535]: mds.? [v2:172.21.15.118:6834/456491655,v1:172.21.15.118:6835/456491655] up:active 2024-03-24T00:24:47.197 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:24:46 smithi118 ceph-mon[31535]: fsmap cephfs:5 3 up:standby-replay 5 up:active 2024-03-24T00:24:47.197 INFO:journalctl@ceph.mds.j.smithi118.stdout:Mar 24 00:24:46 smithi118 podman[96951]: 2024-03-24 00:24:46.763015092 +0000 UTC m=+0.858994244 container remove a714c7b16352e17f582f5c3983c5a7862dfa479508c596f53fef9d23d615a827 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-j, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, org.label-schema.build-date=20240319, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, CEPH_POINT_RELEASE=) 2024-03-24T00:24:47.197 INFO:journalctl@ceph.mds.j.smithi118.stdout:Mar 24 00:24:46 smithi118 systemd[1]: ceph-d79a5420-e971-11ee-95cd-87774f69a715@mds.j.service: Main process exited, code=exited, status=134/n/a 2024-03-24T00:24:47.203 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:24:46 smithi189 ceph-mon[36187]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-03-24T00:24:47.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:24:46 smithi189 ceph-mon[36187]: Cluster is now healthy 2024-03-24T00:24:47.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:24:46 smithi189 ceph-mon[36187]: mds.? [v2:172.21.15.118:6836/2587227307,v1:172.21.15.118:6837/2587227307] up:active 2024-03-24T00:24:47.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:24:46 smithi189 ceph-mon[36187]: mds.? [v2:172.21.15.118:6834/456491655,v1:172.21.15.118:6835/456491655] up:active 2024-03-24T00:24:47.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:24:46 smithi189 ceph-mon[36187]: fsmap cephfs:5 3 up:standby-replay 5 up:active 2024-03-24T00:24:47.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:24:46 smithi193 ceph-mon[36878]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-03-24T00:24:47.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:24:46 smithi193 ceph-mon[36878]: Cluster is now healthy 2024-03-24T00:24:47.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:24:46 smithi193 ceph-mon[36878]: mds.? [v2:172.21.15.118:6836/2587227307,v1:172.21.15.118:6837/2587227307] up:active 2024-03-24T00:24:47.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:24:46 smithi193 ceph-mon[36878]: mds.? [v2:172.21.15.118:6834/456491655,v1:172.21.15.118:6835/456491655] up:active 2024-03-24T00:24:47.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:24:46 smithi193 ceph-mon[36878]: fsmap cephfs:5 3 up:standby-replay 5 up:active 2024-03-24T00:24:47.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:24:47 smithi118 ceph-mon[31535]: pgmap v702: 97 pgs: 97 active+clean; 146 MiB data, 881 MiB used, 1.0 TiB / 1.0 TiB avail; 45 KiB/s rd, 15 KiB/s wr, 14 op/s 2024-03-24T00:24:47.946 INFO:journalctl@ceph.mds.d.smithi118.stdout:Mar 24 00:24:47 smithi118 podman[96961]: 2024-03-24 00:24:47.551041312 +0000 UTC m=+1.583768473 container remove 76fb46d4f076316c4871ba3ae6e8d8a3cde9e622ec7c972e9c4a6c2c72115c46 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-d, ceph=True, GIT_BRANCH=HEAD, GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240319, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, org.label-schema.license=GPLv2) 2024-03-24T00:24:47.947 INFO:journalctl@ceph.mds.d.smithi118.stdout:Mar 24 00:24:47 smithi118 systemd[1]: ceph-d79a5420-e971-11ee-95cd-87774f69a715@mds.d.service: Main process exited, code=exited, status=134/n/a 2024-03-24T00:24:47.947 INFO:journalctl@ceph.mds.j.smithi118.stdout:Mar 24 00:24:47 smithi118 systemd[1]: ceph-d79a5420-e971-11ee-95cd-87774f69a715@mds.j.service: Failed with result 'exit-code'. 2024-03-24T00:24:47.947 INFO:journalctl@ceph.mds.j.smithi118.stdout:Mar 24 00:24:47 smithi118 systemd[1]: ceph-d79a5420-e971-11ee-95cd-87774f69a715@mds.j.service: Consumed 1.398s CPU time. 2024-03-24T00:24:48.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:24:47 smithi189 ceph-mon[36187]: pgmap v702: 97 pgs: 97 active+clean; 146 MiB data, 881 MiB used, 1.0 TiB / 1.0 TiB avail; 45 KiB/s rd, 15 KiB/s wr, 14 op/s 2024-03-24T00:24:48.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:24:47 smithi193 ceph-mon[36878]: pgmap v702: 97 pgs: 97 active+clean; 146 MiB data, 881 MiB used, 1.0 TiB / 1.0 TiB avail; 45 KiB/s rd, 15 KiB/s wr, 14 op/s 2024-03-24T00:24:48.446 INFO:journalctl@ceph.mds.d.smithi118.stdout:Mar 24 00:24:48 smithi118 systemd[1]: ceph-d79a5420-e971-11ee-95cd-87774f69a715@mds.d.service: Failed with result 'exit-code'. 2024-03-24T00:24:48.446 INFO:journalctl@ceph.mds.d.smithi118.stdout:Mar 24 00:24:48 smithi118 systemd[1]: ceph-d79a5420-e971-11ee-95cd-87774f69a715@mds.d.service: Consumed 1.729s CPU time. 2024-03-24T00:24:50.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:24:49 smithi118 ceph-mon[31535]: pgmap v703: 97 pgs: 97 active+clean; 146 MiB data, 881 MiB used, 1.0 TiB / 1.0 TiB avail; 309 KiB/s rd, 16 KiB/s wr, 42 op/s 2024-03-24T00:24:50.203 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:24:49 smithi189 ceph-mon[36187]: pgmap v703: 97 pgs: 97 active+clean; 146 MiB data, 881 MiB used, 1.0 TiB / 1.0 TiB avail; 309 KiB/s rd, 16 KiB/s wr, 42 op/s 2024-03-24T00:24:50.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:24:49 smithi193 ceph-mon[36878]: pgmap v703: 97 pgs: 97 active+clean; 146 MiB data, 881 MiB used, 1.0 TiB / 1.0 TiB avail; 309 KiB/s rd, 16 KiB/s wr, 42 op/s 2024-03-24T00:24:52.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:24:51 smithi118 ceph-mon[31535]: pgmap v704: 97 pgs: 97 active+clean; 146 MiB data, 881 MiB used, 1.0 TiB / 1.0 TiB avail; 458 KiB/s rd, 20 KiB/s wr, 65 op/s 2024-03-24T00:24:52.203 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:24:51 smithi189 ceph-mon[36187]: pgmap v704: 97 pgs: 97 active+clean; 146 MiB data, 881 MiB used, 1.0 TiB / 1.0 TiB avail; 458 KiB/s rd, 20 KiB/s wr, 65 op/s 2024-03-24T00:24:52.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:24:51 smithi193 ceph-mon[36878]: pgmap v704: 97 pgs: 97 active+clean; 146 MiB data, 881 MiB used, 1.0 TiB / 1.0 TiB avail; 458 KiB/s rd, 20 KiB/s wr, 65 op/s 2024-03-24T00:24:54.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:24:53 smithi118 ceph-mon[31535]: pgmap v705: 97 pgs: 97 active+clean; 146 MiB data, 881 MiB used, 1.0 TiB / 1.0 TiB avail; 453 KiB/s rd, 16 KiB/s wr, 60 op/s 2024-03-24T00:24:54.203 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:24:53 smithi189 ceph-mon[36187]: pgmap v705: 97 pgs: 97 active+clean; 146 MiB data, 881 MiB used, 1.0 TiB / 1.0 TiB avail; 453 KiB/s rd, 16 KiB/s wr, 60 op/s 2024-03-24T00:24:54.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:24:53 smithi193 ceph-mon[36878]: pgmap v705: 97 pgs: 97 active+clean; 146 MiB data, 881 MiB used, 1.0 TiB / 1.0 TiB avail; 453 KiB/s rd, 16 KiB/s wr, 60 op/s 2024-03-24T00:24:56.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:24:55 smithi118 ceph-mon[31535]: pgmap v706: 97 pgs: 97 active+clean; 146 MiB data, 881 MiB used, 1.0 TiB / 1.0 TiB avail; 385 KiB/s rd, 14 KiB/s wr, 52 op/s 2024-03-24T00:24:56.203 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:24:55 smithi189 ceph-mon[36187]: pgmap v706: 97 pgs: 97 active+clean; 146 MiB data, 881 MiB used, 1.0 TiB / 1.0 TiB avail; 385 KiB/s rd, 14 KiB/s wr, 52 op/s 2024-03-24T00:24:56.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:24:55 smithi193 ceph-mon[36878]: pgmap v706: 97 pgs: 97 active+clean; 146 MiB data, 881 MiB used, 1.0 TiB / 1.0 TiB avail; 385 KiB/s rd, 14 KiB/s wr, 52 op/s 2024-03-24T00:24:57.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:24:56 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T00:24:57.203 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:24:56 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T00:24:57.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:24:56 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T00:24:57.949 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:24:57.948+0000 7f3929400640 1 -- 172.21.15.118:0/2217076209 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3924104af0 msgr2=0x7f3924104f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:24:57.949 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:24:57.948+0000 7f3929400640 1 --2- 172.21.15.118:0/2217076209 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3924104af0 0x7f3924104f70 secure :-1 s=READY pgs=465 cs=0 l=1 rev1=1 crypto rx=0x7f390c009900 tx=0x7f390c02bc30 comp rx=0 tx=0).stop 2024-03-24T00:24:57.949 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:24:57.949+0000 7f3929400640 1 -- 172.21.15.118:0/2217076209 shutdown_connections 2024-03-24T00:24:57.949 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:24:57.949+0000 7f3929400640 1 --2- 172.21.15.118:0/2217076209 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f39241054b0 0x7f392410bd40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:24:57.949 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:24:57.949+0000 7f3929400640 1 --2- 172.21.15.118:0/2217076209 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f3924104af0 0x7f3924104f70 unknown :-1 s=CLOSED pgs=465 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:24:57.949 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:24:57.949+0000 7f3929400640 1 --2- 172.21.15.118:0/2217076209 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f3924103900 0x7f3924103d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:24:57.949 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:24:57.949+0000 7f3929400640 1 -- 172.21.15.118:0/2217076209 >> 172.21.15.118:0/2217076209 conn(0x7f39240fee90 msgr2=0x7f39241012b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:24:57.950 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:24:57.949+0000 7f3929400640 1 -- 172.21.15.118:0/2217076209 shutdown_connections 2024-03-24T00:24:57.950 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:24:57.949+0000 7f3929400640 1 -- 172.21.15.118:0/2217076209 wait complete. 2024-03-24T00:24:57.951 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:24:57.950+0000 7f3929400640 1 Processor -- start 2024-03-24T00:24:57.951 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:24:57.950+0000 7f3929400640 1 -- start start 2024-03-24T00:24:57.951 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:24:57.951+0000 7f3929400640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f3924103900 0x7f3924199b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:24:57.951 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:24:57.951+0000 7f3929400640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f3924104af0 0x7f392419a040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:24:57.952 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:24:57.951+0000 7f3929400640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f39241054b0 0x7f39241a10b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:24:57.952 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:24:57.951+0000 7f3929400640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f392410f5b0 con 0x7f3924104af0 2024-03-24T00:24:57.952 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:24:57.951+0000 7f3929400640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f392410f430 con 0x7f39241054b0 2024-03-24T00:24:57.952 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:24:57.951+0000 7f3929400640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f392410f730 con 0x7f3924103900 2024-03-24T00:24:57.952 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:24:57.951+0000 7f3922800640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f3924103900 0x7f3924199b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:24:57.952 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:24:57.951+0000 7f3922800640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f3924103900 0x7f3924199b00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.118:35472/0 (socket says 172.21.15.118:35472) 2024-03-24T00:24:57.952 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:24:57.951+0000 7f3923200640 1 --2- 172.21.15.118:0/4172884086 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f39241054b0 0x7f39241a10b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:24:57.952 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:24:57.951+0000 7f3922800640 1 -- 172.21.15.118:0/4172884086 learned_addr learned my addr 172.21.15.118:0/4172884086 (peer_addr_for_me v2:172.21.15.118:0/0) 2024-03-24T00:24:57.952 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:24:57.951+0000 7f3921e00640 1 --2- 172.21.15.118:0/4172884086 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f3924104af0 0x7f392419a040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:24:57.952 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:24:57.952+0000 7f3923200640 1 -- 172.21.15.118:0/4172884086 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f3924103900 msgr2=0x7f3924199b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:24:57.952 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:24:57.952+0000 7f3923200640 1 --2- 172.21.15.118:0/4172884086 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f3924103900 0x7f3924199b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:24:57.952 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:24:57.952+0000 7f3923200640 1 -- 172.21.15.118:0/4172884086 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f3924104af0 msgr2=0x7f392419a040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:24:57.952 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:24:57.952+0000 7f3923200640 1 --2- 172.21.15.118:0/4172884086 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f3924104af0 0x7f392419a040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:24:57.953 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:24:57.952+0000 7f3923200640 1 -- 172.21.15.118:0/4172884086 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f39241a1720 con 0x7f39241054b0 2024-03-24T00:24:57.953 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:24:57.952+0000 7f3922800640 1 --2- 172.21.15.118:0/4172884086 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f3924103900 0x7f3924199b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T00:24:57.953 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:24:57.952+0000 7f3923200640 1 --2- 172.21.15.118:0/4172884086 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f39241054b0 0x7f39241a10b0 secure :-1 s=READY pgs=466 cs=0 l=1 rev1=1 crypto rx=0x7f391c00e970 tx=0x7f391c00ee30 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:24:57.953 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:24:57.952+0000 7f391b400640 1 -- 172.21.15.118:0/4172884086 <== mon.2 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f391c00ccc0 con 0x7f39241054b0 2024-03-24T00:24:57.954 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:24:57.952+0000 7f391b400640 1 -- 172.21.15.118:0/4172884086 <== mon.2 v2:172.21.15.189:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f391c005020 con 0x7f39241054b0 2024-03-24T00:24:57.954 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:24:57.952+0000 7f391b400640 1 -- 172.21.15.118:0/4172884086 <== mon.2 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f391c004a00 con 0x7f39241054b0 2024-03-24T00:24:57.954 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:24:57.952+0000 7f3929400640 1 -- 172.21.15.118:0/4172884086 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f39241a1a10 con 0x7f39241054b0 2024-03-24T00:24:57.954 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:24:57.952+0000 7f3929400640 1 -- 172.21.15.118:0/4172884086 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f39241a1ee0 con 0x7f39241054b0 2024-03-24T00:24:57.955 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:24:57.955+0000 7f391b400640 1 -- 172.21.15.118:0/4172884086 <== mon.2 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 20) v1 ==== 137291+0+0 (secure 0 0 0) 0x7f391c0051d0 con 0x7f39241054b0 2024-03-24T00:24:57.955 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:24:57.955+0000 7f392c000740 1 Processor -- start 2024-03-24T00:24:57.955 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:24:57.956+0000 7f392c000740 1 -- start start 2024-03-24T00:24:57.957 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:24:57.956+0000 7f392c000740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x563c8c1971e0 0x563c8c197600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:24:57.957 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:24:57.956+0000 7f392c000740 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x563c8c197bd0 0x563c8c19c6d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:24:57.957 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:24:57.956+0000 7f392c000740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x563c8c19cd40 0x563c8c19f1d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:24:57.957 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:24:57.956+0000 7f392c000740 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x563c8c15a3b0 con 0x563c8c197bd0 2024-03-24T00:24:57.957 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:24:57.956+0000 7f392c000740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x563c8c0fc830 con 0x563c8c1971e0 2024-03-24T00:24:57.957 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:24:57.956+0000 7f392c000740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x563c8c0a55d0 con 0x563c8c19cd40 2024-03-24T00:24:57.957 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:24:57.956+0000 7f3921e00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x563c8c1971e0 0x563c8c197600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:24:57.957 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:24:57.956+0000 7f3922800640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x563c8c197bd0 0x563c8c19c6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:24:57.957 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:24:57.956+0000 7f3922800640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x563c8c197bd0 0x563c8c19c6d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.118:3300/0 says I am v2:172.21.15.118:35260/0 (socket says 172.21.15.118:35260) 2024-03-24T00:24:57.957 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:24:57.956+0000 7f3921e00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x563c8c1971e0 0x563c8c197600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.118:59320/0 (socket says 172.21.15.118:59320) 2024-03-24T00:24:57.957 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:24:57.956+0000 7f3922800640 1 -- 172.21.15.118:0/2795771851 learned_addr learned my addr 172.21.15.118:0/2795771851 (peer_addr_for_me v2:172.21.15.118:0/0) 2024-03-24T00:24:57.957 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:24:57.957+0000 7f3921e00640 1 --2- 172.21.15.118:0/2795771851 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x563c8c19cd40 0x563c8c19f1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:24:57.957 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:24:57.957+0000 7f3922800640 1 -- 172.21.15.118:0/2795771851 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x563c8c19cd40 msgr2=0x563c8c19f1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:24:57.958 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:24:57.957+0000 7f3922800640 1 --2- 172.21.15.118:0/2795771851 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x563c8c19cd40 0x563c8c19f1d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:24:57.958 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:24:57.957+0000 7f3922800640 1 -- 172.21.15.118:0/2795771851 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x563c8c1971e0 msgr2=0x563c8c197600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:24:57.958 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:24:57.957+0000 7f3922800640 1 --2- 172.21.15.118:0/2795771851 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x563c8c1971e0 0x563c8c197600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:24:57.958 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:24:57.957+0000 7f3922800640 1 -- 172.21.15.118:0/2795771851 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x563c8c0bd2c0 con 0x563c8c197bd0 2024-03-24T00:24:57.962 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:24:57.957+0000 7f391b400640 1 --2- 172.21.15.118:0/4172884086 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7f39080a28a0 0x7f39080a4d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:24:57.962 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:24:57.957+0000 7f3922800640 1 --2- 172.21.15.118:0/2795771851 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x563c8c197bd0 0x563c8c19c6d0 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7f391400e880 tx=0x7f391400ed40 comp rx=0 tx=0).ready entity=mon.0 client_cookie=53f82b1843cb5b50 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:24:57.962 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:24:57.958+0000 7f391b400640 1 -- 172.21.15.118:0/4172884086 <== mon.2 v2:172.21.15.189:3300/0 5 ==== osd_map(122..122 src has 1..122) v4 ==== 11029+0+0 (secure 0 0 0) 0x7f391c0cf5d0 con 0x7f39241054b0 2024-03-24T00:24:57.962 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:24:57.958+0000 7f3913400640 1 -- 172.21.15.118:0/2795771851 <== mon.0 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3914004d20 con 0x563c8c197bd0 2024-03-24T00:24:57.962 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:24:57.958+0000 7f3913400640 1 -- 172.21.15.118:0/2795771851 <== mon.0 v2:172.21.15.118:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f3914003970 con 0x563c8c197bd0 2024-03-24T00:24:57.962 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:24:57.958+0000 7f3913400640 1 -- 172.21.15.118:0/2795771851 <== mon.0 v2:172.21.15.118:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f39140045c0 con 0x563c8c197bd0 2024-03-24T00:24:57.962 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:24:57.958+0000 7f3921e00640 1 --2- 172.21.15.118:0/4172884086 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7f39080a28a0 0x7f39080a4d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:24:57.962 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:24:57.958+0000 7f392c000740 1 -- 172.21.15.118:0/2795771851 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x563c8c197bd0 msgr2=0x563c8c19c6d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:24:57.963 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:24:57.958+0000 7f3921e00640 1 --2- 172.21.15.118:0/4172884086 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7f39080a28a0 0x7f39080a4d60 secure :-1 s=READY pgs=398 cs=0 l=1 rev1=1 crypto rx=0x7f390c02c240 tx=0x7f390c005a40 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:24:57.963 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:24:57.958+0000 7f392c000740 1 --2- 172.21.15.118:0/2795771851 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x563c8c197bd0 0x563c8c19c6d0 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7f391400e880 tx=0x7f391400ed40 comp rx=0 tx=0).stop 2024-03-24T00:24:57.963 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:24:57.959+0000 7f392c000740 1 -- 172.21.15.118:0/2795771851 shutdown_connections 2024-03-24T00:24:57.963 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:24:57.959+0000 7f392c000740 1 --2- 172.21.15.118:0/2795771851 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x563c8c19cd40 0x563c8c19f1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:24:57.963 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:24:57.959+0000 7f392c000740 1 --2- 172.21.15.118:0/2795771851 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x563c8c197bd0 0x563c8c19c6d0 unknown :-1 s=CLOSED pgs=233 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:24:57.963 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:24:57.959+0000 7f392c000740 1 --2- 172.21.15.118:0/2795771851 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x563c8c1971e0 0x563c8c197600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:24:57.963 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:24:57.959+0000 7f392c000740 1 -- 172.21.15.118:0/2795771851 >> 172.21.15.118:0/2795771851 conn(0x563c8c192750 msgr2=0x563c8c192b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:24:57.963 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:24:57.959+0000 7f392c000740 1 -- 172.21.15.118:0/2795771851 shutdown_connections 2024-03-24T00:24:57.963 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:24:57.959+0000 7f392c000740 1 -- 172.21.15.118:0/2795771851 wait complete. 2024-03-24T00:24:57.963 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:24:57.959+0000 7f392c000740 1 -- start start 2024-03-24T00:24:57.963 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:24:57.959+0000 7f392c000740 1 Processor -- start 2024-03-24T00:24:57.963 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:24:57.960+0000 7f392c000740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x563c8c1971e0 0x563c8c1a9b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:24:57.963 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:24:57.960+0000 7f392c000740 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x563c8c197bd0 0x563c8c1aa080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:24:57.963 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:24:57.960+0000 7f392c000740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x563c8c19cd40 0x563c8c1b10c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:24:57.964 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:24:57.960+0000 7f392c000740 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x563c8c0b8a20 con 0x563c8c197bd0 2024-03-24T00:24:57.964 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:24:57.960+0000 7f392c000740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x563c8c0f88d0 con 0x563c8c19cd40 2024-03-24T00:24:57.964 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:24:57.960+0000 7f392c000740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x563c8c19fcf0 con 0x563c8c1971e0 2024-03-24T00:24:57.964 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:24:57.960+0000 7f392c000740 10 client.0 _ll_register_callbacks cb 0x563c8c0f51c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T00:24:57.964 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:24:57.960+0000 7f3922800640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x563c8c1971e0 0x563c8c1a9b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:24:57.964 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:24:57.960+0000 7f3922800640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x563c8c1971e0 0x563c8c1a9b40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.118:35482/0 (socket says 172.21.15.118:35482) 2024-03-24T00:24:57.964 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:24:57.960+0000 7f3922800640 1 -- 172.21.15.118:0/1637684200 learned_addr learned my addr 172.21.15.118:0/1637684200 (peer_addr_for_me v2:172.21.15.118:0/0) 2024-03-24T00:24:57.964 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:24:57.960+0000 7f3921e00640 1 --2- 172.21.15.118:0/1637684200 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x563c8c197bd0 0x563c8c1aa080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:24:57.964 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:24:57.960+0000 7f3923200640 1 --2- 172.21.15.118:0/1637684200 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x563c8c19cd40 0x563c8c1b10c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:24:57.964 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:24:57.961+0000 7f3922800640 1 -- 172.21.15.118:0/1637684200 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x563c8c19cd40 msgr2=0x563c8c1b10c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:24:57.964 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:24:57.961+0000 7f3922800640 1 --2- 172.21.15.118:0/1637684200 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x563c8c19cd40 0x563c8c1b10c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:24:57.964 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:24:57.961+0000 7f3922800640 1 -- 172.21.15.118:0/1637684200 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x563c8c197bd0 msgr2=0x563c8c1aa080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:24:57.964 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:24:57.961+0000 7f3922800640 1 --2- 172.21.15.118:0/1637684200 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x563c8c197bd0 0x563c8c1aa080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:24:57.964 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:24:57.961+0000 7f3922800640 1 -- 172.21.15.118:0/1637684200 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x563c8be11210 con 0x563c8c1971e0 2024-03-24T00:24:57.965 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:24:57.961+0000 7f3922800640 1 --2- 172.21.15.118:0/1637684200 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x563c8c1971e0 0x563c8c1a9b40 secure :-1 s=READY pgs=468 cs=0 l=1 rev1=1 crypto rx=0x7f3914005c40 tx=0x7f3914011c50 comp rx=0 tx=0).ready entity=mon.1 client_cookie=29980648d129986b server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:24:57.965 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:24:57.961+0000 7f3907e00640 10 client.0 ms_handle_connect on v2:172.21.15.193:3300/0 2024-03-24T00:24:57.965 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:24:57.961+0000 7f3907e00640 1 -- 172.21.15.118:0/1637684200 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3914004e10 con 0x563c8c1971e0 2024-03-24T00:24:57.965 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:24:57.961+0000 7f392c000740 1 -- 172.21.15.118:0/1637684200 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f3914003970 con 0x563c8c1971e0 2024-03-24T00:24:57.965 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:24:57.961+0000 7f3907e00640 1 -- 172.21.15.118:0/1637684200 <== mon.1 v2:172.21.15.193:3300/0 2 ==== osd_map(122..122 src has 1..122) v4 ==== 11029+0+0 (secure 0 0 0) 0x7f3914004300 con 0x563c8c1971e0 2024-03-24T00:24:57.965 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:24:57.961+0000 7f3907e00640 1 -- 172.21.15.118:0/1637684200 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_get_version_reply(handle=1 version=178) v2 ==== 24+0+0 (secure 0 0 0) 0x7f3914009d80 con 0x563c8c1971e0 2024-03-24T00:24:57.965 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:24:57.961+0000 7f392c000740 10 client.27058 fetch_fsmap learned FSMap version 178 2024-03-24T00:24:57.965 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:24:57.961+0000 7f392c000740 1 -- 172.21.15.118:0/1637684200 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,fsmap=178,monmap=4+}) v3 -- 0x563c8bedd620 con 0x563c8c1971e0 2024-03-24T00:24:57.965 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:24:57.962+0000 7f3907e00640 1 -- 172.21.15.118:0/1637684200 <== mon.1 v2:172.21.15.193:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f3914003970 con 0x563c8c1971e0 2024-03-24T00:24:57.965 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:24:57.962+0000 7f3907e00640 1 -- 172.21.15.118:0/1637684200 <== mon.1 v2:172.21.15.193:3300/0 5 ==== fsmap(e 178) v1 ==== 5658+0+0 (secure 0 0 0) 0x7f3914022070 con 0x563c8c1971e0 2024-03-24T00:24:57.965 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:24:57.962+0000 7f392c000740 10 client.27058 fetch_fsmap finished waiting for FSMap version 178 2024-03-24T00:24:57.965 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:24:57.962+0000 7f392c000740 10 client.27058 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-24T00:24:57.965 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:24:57.962+0000 7f392c000740 20 client.27058 populate_metadata read hostname 'smithi118' 2024-03-24T00:24:57.966 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:24:57.962+0000 7f392c000740 1 --2- 172.21.15.118:0/1637684200 >> [v2:172.21.15.189:6838/4074484168,v1:172.21.15.189:6839/4074484168] conn(0x563c8c1b5ae0 0x563c8c1b7f60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:24:57.966 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:24:57.962+0000 7f392c000740 4 client.27058 mds_command: new command op to 24569 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-24T00:24:57.966 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:24:57.962+0000 7f392c000740 1 -- 172.21.15.118:0/1637684200 --> [v2:172.21.15.189:6838/4074484168,v1:172.21.15.189:6839/4074484168] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x563c8c109d70 con 0x563c8c1b5ae0 2024-03-24T00:24:57.966 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:24:57.962+0000 7f3921e00640 1 --2- 172.21.15.118:0/1637684200 >> [v2:172.21.15.189:6838/4074484168,v1:172.21.15.189:6839/4074484168] conn(0x563c8c1b5ae0 0x563c8c1b7f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:24:57.967 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:24:57.963+0000 7f3921e00640 1 --2- 172.21.15.118:0/1637684200 >> [v2:172.21.15.189:6838/4074484168,v1:172.21.15.189:6839/4074484168] conn(0x563c8c1b5ae0 0x563c8c1b7f60 crc :-1 s=READY pgs=418 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=bdd1dd89430a3625 server_cookie=590c8f86ab14f3f7 in_seq=0 out_seq=0 2024-03-24T00:24:57.967 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:24:57.963+0000 7f3907e00640 10 client.27058 ms_handle_connect on v2:172.21.15.189:6838/4074484168 2024-03-24T00:24:57.967 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:24:57.964+0000 7f3907e00640 1 -- 172.21.15.118:0/1637684200 <== mds.0 v2:172.21.15.189:6838/4074484168 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x563c8c109d70 con 0x563c8c1b5ae0 2024-03-24T00:24:57.967 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:24:57.964+0000 7f3907e00640 10 client.27058 handle_command_reply: tid=0 2024-03-24T00:24:57.967 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:24:57.964+0000 7f392c000740 1 client.27058 shutdown 2024-03-24T00:24:57.967 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:24:57.964+0000 7f392c000740 1 -- 172.21.15.118:0/1637684200 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x563c8c1971e0 msgr2=0x563c8c1a9b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:24:57.968 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:24:57.964+0000 7f392c000740 1 --2- 172.21.15.118:0/1637684200 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x563c8c1971e0 0x563c8c1a9b40 secure :-1 s=READY pgs=468 cs=0 l=1 rev1=1 crypto rx=0x7f3914005c40 tx=0x7f3914011c50 comp rx=0 tx=0).stop 2024-03-24T00:24:57.968 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:24:57.964+0000 7f392c000740 1 -- 172.21.15.118:0/1637684200 shutdown_connections 2024-03-24T00:24:57.968 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:24:57.964+0000 7f392c000740 1 --2- 172.21.15.118:0/1637684200 >> [v2:172.21.15.189:6838/4074484168,v1:172.21.15.189:6839/4074484168] conn(0x563c8c1b5ae0 0x563c8c1b7f60 crc :-1 s=READY pgs=418 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:24:57.968 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:24:57.964+0000 7f392c000740 1 --2- 172.21.15.118:0/1637684200 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x563c8c19cd40 0x563c8c1b10c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:24:57.968 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:24:57.964+0000 7f392c000740 1 --2- 172.21.15.118:0/1637684200 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x563c8c197bd0 0x563c8c1aa080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:24:57.968 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:24:57.964+0000 7f3907e00640 0 client.27058 ms_handle_reset on v2:172.21.15.189:6838/4074484168 2024-03-24T00:24:57.968 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:24:57.964+0000 7f392c000740 1 --2- 172.21.15.118:0/1637684200 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x563c8c1971e0 0x563c8c1a9b40 unknown :-1 s=CLOSED pgs=468 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:24:57.968 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:24:57.964+0000 7f392c000740 1 -- 172.21.15.118:0/1637684200 >> 172.21.15.118:0/1637684200 conn(0x563c8c192750 msgr2=0x563c8c191950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:24:57.968 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:24:57.964+0000 7f392c000740 1 -- 172.21.15.118:0/1637684200 shutdown_connections 2024-03-24T00:24:57.968 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:24:57.964+0000 7f392c000740 1 -- 172.21.15.118:0/1637684200 wait complete. 2024-03-24T00:24:57.968 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:24:57.965+0000 7f392c000740 20 client.27058 trim_cache size 0 max 16384 2024-03-24T00:24:57.980 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:24:57.980+0000 7f392c000740 1 Processor -- start 2024-03-24T00:24:57.980 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:24:57.980+0000 7f392c000740 1 -- start start 2024-03-24T00:24:57.982 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:24:57.981+0000 7f392c000740 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x563c8c192750 0x563c8c1a9810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:24:57.982 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:24:57.981+0000 7f392c000740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x563c8c1971e0 0x563c8c1b1080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:24:57.982 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:24:57.981+0000 7f392c000740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x563c8c197bd0 0x563c8c1a9d50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:24:57.982 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:24:57.981+0000 7f392c000740 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f39241a8e50 con 0x563c8c192750 2024-03-24T00:24:57.982 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:24:57.981+0000 7f392c000740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f39241a94b0 con 0x563c8c1971e0 2024-03-24T00:24:57.982 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:24:57.981+0000 7f392c000740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f39241a6e70 con 0x563c8c197bd0 2024-03-24T00:24:57.982 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:24:57.981+0000 7f3921e00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x563c8c1971e0 0x563c8c1b1080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:24:57.982 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:24:57.981+0000 7f3922800640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x563c8c192750 0x563c8c1a9810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:24:57.982 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:24:57.981+0000 7f3923200640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x563c8c197bd0 0x563c8c1a9d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:24:57.982 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:24:57.981+0000 7f3921e00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x563c8c1971e0 0x563c8c1b1080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.118:59330/0 (socket says 172.21.15.118:59330) 2024-03-24T00:24:57.982 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:24:57.981+0000 7f3923200640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x563c8c197bd0 0x563c8c1a9d50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.118:35484/0 (socket says 172.21.15.118:35484) 2024-03-24T00:24:57.982 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:24:57.981+0000 7f3921e00640 1 -- 172.21.15.118:0/2803492622 learned_addr learned my addr 172.21.15.118:0/2803492622 (peer_addr_for_me v2:172.21.15.118:0/0) 2024-03-24T00:24:57.983 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:24:57.981+0000 7f3923200640 1 -- 172.21.15.118:0/2803492622 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x563c8c1971e0 msgr2=0x563c8c1b1080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:24:57.983 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:24:57.981+0000 7f3923200640 1 --2- 172.21.15.118:0/2803492622 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x563c8c1971e0 0x563c8c1b1080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:24:57.983 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:24:57.981+0000 7f3923200640 1 -- 172.21.15.118:0/2803492622 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x563c8c192750 msgr2=0x563c8c1a9810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:24:57.983 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:24:57.981+0000 7f3923200640 1 --2- 172.21.15.118:0/2803492622 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x563c8c192750 0x563c8c1a9810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:24:57.983 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:24:57.981+0000 7f3923200640 1 -- 172.21.15.118:0/2803492622 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x563c8c1934c0 con 0x563c8c197bd0 2024-03-24T00:24:57.983 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:24:57.981+0000 7f3921e00640 1 --2- 172.21.15.118:0/2803492622 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x563c8c1971e0 0x563c8c1b1080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T00:24:57.983 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:24:57.982+0000 7f3923200640 1 --2- 172.21.15.118:0/2803492622 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x563c8c197bd0 0x563c8c1a9d50 secure :-1 s=READY pgs=469 cs=0 l=1 rev1=1 crypto rx=0x7f391c09b120 tx=0x7f391c005750 comp rx=0 tx=0).ready entity=mon.1 client_cookie=55ce215b48a8d590 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:24:57.983 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:24:57.982+0000 7f3912a00640 1 -- 172.21.15.118:0/2803492622 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f391c01c040 con 0x563c8c197bd0 2024-03-24T00:24:57.983 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:24:57.982+0000 7f3912a00640 1 -- 172.21.15.118:0/2803492622 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f391c0b4070 con 0x563c8c197bd0 2024-03-24T00:24:57.983 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:24:57.982+0000 7f3912a00640 1 -- 172.21.15.118:0/2803492622 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f391c09f170 con 0x563c8c197bd0 2024-03-24T00:24:57.986 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:24:57.983+0000 7f392c000740 1 -- 172.21.15.118:0/2803492622 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x563c8c197bd0 msgr2=0x563c8c1a9d50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:24:57.987 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:24:57.983+0000 7f392c000740 1 --2- 172.21.15.118:0/2803492622 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x563c8c197bd0 0x563c8c1a9d50 secure :-1 s=READY pgs=469 cs=0 l=1 rev1=1 crypto rx=0x7f391c09b120 tx=0x7f391c005750 comp rx=0 tx=0).stop 2024-03-24T00:24:57.987 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:24:57.983+0000 7f392c000740 1 -- 172.21.15.118:0/2803492622 shutdown_connections 2024-03-24T00:24:57.987 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:24:57.983+0000 7f392c000740 1 --2- 172.21.15.118:0/2803492622 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x563c8c197bd0 0x563c8c1a9d50 unknown :-1 s=CLOSED pgs=469 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:24:57.987 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:24:57.983+0000 7f392c000740 1 --2- 172.21.15.118:0/2803492622 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x563c8c1971e0 0x563c8c1b1080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:24:57.987 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:24:57.983+0000 7f392c000740 1 --2- 172.21.15.118:0/2803492622 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x563c8c192750 0x563c8c1a9810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:24:57.987 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:24:57.983+0000 7f392c000740 1 -- 172.21.15.118:0/2803492622 >> 172.21.15.118:0/2803492622 conn(0x563c8c19c810 msgr2=0x563c8c1959b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:24:57.987 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:24:57.983+0000 7f392c000740 1 -- 172.21.15.118:0/2803492622 shutdown_connections 2024-03-24T00:24:57.987 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:24:57.983+0000 7f392c000740 1 -- 172.21.15.118:0/2803492622 wait complete. 2024-03-24T00:24:57.987 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:24:57.983+0000 7f392c000740 1 -- start start 2024-03-24T00:24:57.987 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:24:57.983+0000 7f392c000740 1 Processor -- start 2024-03-24T00:24:57.987 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:24:57.984+0000 7f392c000740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x563c8c192750 0x563c8c1aebd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:24:57.987 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:24:57.984+0000 7f392c000740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x563c8c1971e0 0x563c8c1af110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:24:57.987 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:24:57.984+0000 7f392c000740 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x563c8c197bd0 0x563c8c1b0080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:24:57.988 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:24:57.984+0000 7f392c000740 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f39241a74d0 con 0x563c8c197bd0 2024-03-24T00:24:57.988 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:24:57.984+0000 7f392c000740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f39241a87f0 con 0x563c8c192750 2024-03-24T00:24:57.988 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:24:57.984+0000 7f392c000740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f39241a7b30 con 0x563c8c1971e0 2024-03-24T00:24:57.988 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:24:57.984+0000 7f392c000740 10 client.0 _ll_register_callbacks cb 0x7f391c005e30 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T00:24:57.988 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:24:57.984+0000 7f3922800640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x563c8c192750 0x563c8c1aebd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:24:57.988 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:24:57.984+0000 7f3922800640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x563c8c192750 0x563c8c1aebd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.118:59340/0 (socket says 172.21.15.118:59340) 2024-03-24T00:24:57.988 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:24:57.984+0000 7f3923200640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x563c8c197bd0 0x563c8c1b0080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:24:57.988 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:24:57.984+0000 7f3922800640 1 -- 172.21.15.118:0/1431104 learned_addr learned my addr 172.21.15.118:0/1431104 (peer_addr_for_me v2:172.21.15.118:0/0) 2024-03-24T00:24:57.988 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:24:57.984+0000 7f3921e00640 1 --2- 172.21.15.118:0/1431104 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x563c8c1971e0 0x563c8c1af110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:24:57.988 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:24:57.984+0000 7f3922800640 1 -- 172.21.15.118:0/1431104 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x563c8c1971e0 msgr2=0x563c8c1af110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:24:57.988 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:24:57.984+0000 7f3922800640 1 --2- 172.21.15.118:0/1431104 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x563c8c1971e0 0x563c8c1af110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:24:57.988 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:24:57.984+0000 7f3922800640 1 -- 172.21.15.118:0/1431104 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x563c8c197bd0 msgr2=0x563c8c1b0080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:24:57.988 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:24:57.984+0000 7f3922800640 1 --2- 172.21.15.118:0/1431104 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x563c8c197bd0 0x563c8c1b0080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:24:57.989 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:24:57.984+0000 7f3922800640 1 -- 172.21.15.118:0/1431104 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x563c8c1b0670 con 0x563c8c192750 2024-03-24T00:24:57.989 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:24:57.984+0000 7f3921e00640 1 --2- 172.21.15.118:0/1431104 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x563c8c1971e0 0x563c8c1af110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T00:24:57.989 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:24:57.985+0000 7f3922800640 1 --2- 172.21.15.118:0/1431104 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x563c8c192750 0x563c8c1aebd0 secure :-1 s=READY pgs=467 cs=0 l=1 rev1=1 crypto rx=0x7f3914002230 tx=0x7f3914009cf0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=9d83ee200b01f14 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:24:57.989 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:24:57.985+0000 7f3907e00640 10 client.0 ms_handle_connect on v2:172.21.15.189:3300/0 2024-03-24T00:24:57.989 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:24:57.985+0000 7f3907e00640 1 -- 172.21.15.118:0/1431104 <== mon.2 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3914005920 con 0x563c8c192750 2024-03-24T00:24:57.989 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:24:57.985+0000 7f392c000740 1 -- 172.21.15.118:0/1431104 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f391c0b4070 con 0x563c8c192750 2024-03-24T00:24:57.989 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:24:57.985+0000 7f3907e00640 1 -- 172.21.15.118:0/1431104 <== mon.2 v2:172.21.15.189:3300/0 2 ==== osd_map(122..122 src has 1..122) v4 ==== 11029+0+0 (secure 0 0 0) 0x7f391401a770 con 0x563c8c192750 2024-03-24T00:24:57.989 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:24:57.985+0000 7f3907e00640 1 -- 172.21.15.118:0/1431104 <== mon.2 v2:172.21.15.189:3300/0 3 ==== mon_get_version_reply(handle=1 version=178) v2 ==== 24+0+0 (secure 0 0 0) 0x7f3914003970 con 0x563c8c192750 2024-03-24T00:24:57.989 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:24:57.985+0000 7f392c000740 10 client.27053 fetch_fsmap learned FSMap version 178 2024-03-24T00:24:57.989 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:24:57.985+0000 7f392c000740 1 -- 172.21.15.118:0/1431104 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,fsmap=178,monmap=4+}) v3 -- 0x563c8c2117f0 con 0x563c8c192750 2024-03-24T00:24:57.989 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:24:57.986+0000 7f3907e00640 1 -- 172.21.15.118:0/1431104 <== mon.2 v2:172.21.15.189:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f391c0b4070 con 0x563c8c192750 2024-03-24T00:24:57.989 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:24:57.986+0000 7f3907e00640 1 -- 172.21.15.118:0/1431104 <== mon.2 v2:172.21.15.189:3300/0 5 ==== fsmap(e 178) v1 ==== 5658+0+0 (secure 0 0 0) 0x7f391403dce0 con 0x563c8c192750 2024-03-24T00:24:57.989 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:24:57.986+0000 7f392c000740 10 client.27053 fetch_fsmap finished waiting for FSMap version 178 2024-03-24T00:24:57.989 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:24:57.986+0000 7f392c000740 10 client.27053 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-24T00:24:57.990 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:24:57.986+0000 7f392c000740 20 client.27053 populate_metadata read hostname 'smithi118' 2024-03-24T00:24:57.990 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:24:57.986+0000 7f392c000740 1 --2- 172.21.15.118:0/1431104 >> [v2:172.21.15.189:6838/4074484168,v1:172.21.15.189:6839/4074484168] conn(0x563c8c19cd40 0x563c8c214060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:24:57.990 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:24:57.986+0000 7f392c000740 4 client.27053 mds_command: new command op to 24569 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-03-24T00:24:57.990 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:24:57.986+0000 7f392c000740 1 -- 172.21.15.118:0/1431104 --> [v2:172.21.15.189:6838/4074484168,v1:172.21.15.189:6839/4074484168] -- command(tid 0: {"prefix": "flush journal"}) v1 -- 0x563c8c1040c0 con 0x563c8c19cd40 2024-03-24T00:24:57.990 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:24:57.986+0000 7f3921e00640 1 --2- 172.21.15.118:0/1431104 >> [v2:172.21.15.189:6838/4074484168,v1:172.21.15.189:6839/4074484168] conn(0x563c8c19cd40 0x563c8c214060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:24:57.990 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:24:57.987+0000 7f3907e00640 10 client.27053 ms_handle_connect on v2:172.21.15.189:6838/4074484168 2024-03-24T00:24:57.990 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:24:57.987+0000 7f3921e00640 1 --2- 172.21.15.118:0/1431104 >> [v2:172.21.15.189:6838/4074484168,v1:172.21.15.189:6839/4074484168] conn(0x563c8c19cd40 0x563c8c214060 crc :-1 s=READY pgs=419 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=ccdc19ded8dffa16 server_cookie=2a4deafc19c58f89 in_seq=0 out_seq=0 2024-03-24T00:24:58.112 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:24:57 smithi193 ceph-mon[36878]: pgmap v707: 97 pgs: 97 active+clean; 147 MiB data, 881 MiB used, 1.0 TiB / 1.0 TiB avail; 375 KiB/s rd, 19 KiB/s wr, 48 op/s 2024-03-24T00:24:58.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:24:57 smithi118 ceph-mon[31535]: pgmap v707: 97 pgs: 97 active+clean; 147 MiB data, 881 MiB used, 1.0 TiB / 1.0 TiB avail; 375 KiB/s rd, 19 KiB/s wr, 48 op/s 2024-03-24T00:24:58.196 INFO:journalctl@ceph.mds.j.smithi118.stdout:Mar 24 00:24:57 smithi118 systemd[1]: ceph-d79a5420-e971-11ee-95cd-87774f69a715@mds.j.service: Scheduled restart job, restart counter is at 5. 2024-03-24T00:24:58.196 INFO:journalctl@ceph.mds.j.smithi118.stdout:Mar 24 00:24:57 smithi118 systemd[1]: Stopped Ceph mds.j for d79a5420-e971-11ee-95cd-87774f69a715. 2024-03-24T00:24:58.196 INFO:journalctl@ceph.mds.j.smithi118.stdout:Mar 24 00:24:57 smithi118 systemd[1]: ceph-d79a5420-e971-11ee-95cd-87774f69a715@mds.j.service: Consumed 1.398s CPU time. 2024-03-24T00:24:58.196 INFO:journalctl@ceph.mds.j.smithi118.stdout:Mar 24 00:24:57 smithi118 systemd[1]: Starting Ceph mds.j for d79a5420-e971-11ee-95cd-87774f69a715... 2024-03-24T00:24:58.203 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:24:57 smithi189 ceph-mon[36187]: pgmap v707: 97 pgs: 97 active+clean; 147 MiB data, 881 MiB used, 1.0 TiB / 1.0 TiB avail; 375 KiB/s rd, 19 KiB/s wr, 48 op/s 2024-03-24T00:24:58.517 INFO:journalctl@ceph.mds.d.smithi118.stdout:Mar 24 00:24:58 smithi118 systemd[1]: ceph-d79a5420-e971-11ee-95cd-87774f69a715@mds.d.service: Scheduled restart job, restart counter is at 5. 2024-03-24T00:24:58.517 INFO:journalctl@ceph.mds.d.smithi118.stdout:Mar 24 00:24:58 smithi118 systemd[1]: Stopped Ceph mds.d for d79a5420-e971-11ee-95cd-87774f69a715. 2024-03-24T00:24:58.518 INFO:journalctl@ceph.mds.d.smithi118.stdout:Mar 24 00:24:58 smithi118 systemd[1]: ceph-d79a5420-e971-11ee-95cd-87774f69a715@mds.d.service: Consumed 1.729s CPU time. 2024-03-24T00:24:58.518 INFO:journalctl@ceph.mds.d.smithi118.stdout:Mar 24 00:24:58 smithi118 systemd[1]: Starting Ceph mds.d for d79a5420-e971-11ee-95cd-87774f69a715... 2024-03-24T00:24:58.518 INFO:journalctl@ceph.mds.j.smithi118.stdout:Mar 24 00:24:58 smithi118 podman[97262]: 2024-03-24 00:24:58.219898649 +0000 UTC m=+0.034522601 image pull c664afe56bb01db4f9afa9532c20598e201e664f2e9ba2be50db0d52674a26d2 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc 2024-03-24T00:24:58.518 INFO:journalctl@ceph.mds.j.smithi118.stdout:Mar 24 00:24:58 smithi118 podman[97262]: 2024-03-24 00:24:58.351775271 +0000 UTC m=+0.166399260 container create 263e3b2b32d9ed7a147768b13e897003dae75c6a611458b7cb1f4a3392a16109 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-j, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.build-date=20240319, ceph=True, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD) 2024-03-24T00:24:58.623 INFO:journalctl@ceph.mds.j.smithi118.stdout:Mar 24 00:24:58 smithi118 podman[97262]: 2024-03-24 00:24:58.605419746 +0000 UTC m=+0.420043778 container init 263e3b2b32d9ed7a147768b13e897003dae75c6a611458b7cb1f4a3392a16109 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-j, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240319, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, GIT_CLEAN=True, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, ceph=True) 2024-03-24T00:24:58.946 INFO:journalctl@ceph.mds.j.smithi118.stdout:Mar 24 00:24:58 smithi118 podman[97262]: 2024-03-24 00:24:58.615160579 +0000 UTC m=+0.429784595 container start 263e3b2b32d9ed7a147768b13e897003dae75c6a611458b7cb1f4a3392a16109 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-j, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_BRANCH=HEAD, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, ceph=True, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240319, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-24T00:24:58.946 INFO:journalctl@ceph.mds.j.smithi118.stdout:Mar 24 00:24:58 smithi118 ceph-mds[97304]: -- 172.21.15.118:0/1974827287 <== mon.0 v2:172.21.15.118:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x559598deba40 con 0x559598dbe800 2024-03-24T00:24:58.946 INFO:journalctl@ceph.mds.j.smithi118.stdout:Mar 24 00:24:58 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-j[97290]: starting mds.j at 2024-03-24T00:24:58.947 INFO:journalctl@ceph.mds.j.smithi118.stdout:Mar 24 00:24:58 smithi118 bash[97262]: 263e3b2b32d9ed7a147768b13e897003dae75c6a611458b7cb1f4a3392a16109 2024-03-24T00:24:58.947 INFO:journalctl@ceph.mds.j.smithi118.stdout:Mar 24 00:24:58 smithi118 systemd[1]: Started Ceph mds.j for d79a5420-e971-11ee-95cd-87774f69a715. 2024-03-24T00:24:58.947 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:24:58 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T00:24:58.947 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:24:58 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T00:24:58.947 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:24:58 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T00:24:58.947 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:24:58 smithi118 ceph-mon[31535]: daemon mds.j restarted 2024-03-24T00:24:58.947 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:24:58 smithi118 ceph-mon[31535]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-03-24T00:24:58.947 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:24:58 smithi118 ceph-mon[31535]: osdmap e123: 12 total, 12 up, 12 in 2024-03-24T00:24:58.947 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:24:58 smithi118 ceph-mon[31535]: mds.? [v2:172.21.15.118:6834/395010504,v1:172.21.15.118:6835/395010504] up:boot 2024-03-24T00:24:58.947 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:24:58 smithi118 ceph-mon[31535]: Standby daemon mds.j assigned to filesystem cephfs as rank 3 2024-03-24T00:24:58.947 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:24:58 smithi118 ceph-mon[31535]: fsmap cephfs:4/5 {0=b=up:active,1=d=up:active,2=c=up:active,4=e=up:active} 3 up:standby-replay 1 up:standby, 1 failed 2024-03-24T00:24:58.947 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:24:58 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2024-03-24T00:24:58.948 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:24:58 smithi118 ceph-mon[31535]: fsmap cephfs:5/5 3 up:standby-replay 1 up:replay 4 up:active 2024-03-24T00:24:59.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:24:58 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T00:24:59.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:24:58 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T00:24:59.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:24:58 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T00:24:59.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:24:58 smithi189 ceph-mon[36187]: daemon mds.j restarted 2024-03-24T00:24:59.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:24:58 smithi189 ceph-mon[36187]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-03-24T00:24:59.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:24:58 smithi189 ceph-mon[36187]: osdmap e123: 12 total, 12 up, 12 in 2024-03-24T00:24:59.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:24:58 smithi189 ceph-mon[36187]: mds.? [v2:172.21.15.118:6834/395010504,v1:172.21.15.118:6835/395010504] up:boot 2024-03-24T00:24:59.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:24:58 smithi189 ceph-mon[36187]: Standby daemon mds.j assigned to filesystem cephfs as rank 3 2024-03-24T00:24:59.205 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:24:58 smithi189 ceph-mon[36187]: fsmap cephfs:4/5 {0=b=up:active,1=d=up:active,2=c=up:active,4=e=up:active} 3 up:standby-replay 1 up:standby, 1 failed 2024-03-24T00:24:59.205 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:24:58 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2024-03-24T00:24:59.205 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:24:58 smithi189 ceph-mon[36187]: fsmap cephfs:5/5 3 up:standby-replay 1 up:replay 4 up:active 2024-03-24T00:24:59.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:24:58 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T00:24:59.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:24:58 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T00:24:59.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:24:58 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T00:24:59.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:24:58 smithi193 ceph-mon[36878]: daemon mds.j restarted 2024-03-24T00:24:59.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:24:58 smithi193 ceph-mon[36878]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-03-24T00:24:59.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:24:58 smithi193 ceph-mon[36878]: osdmap e123: 12 total, 12 up, 12 in 2024-03-24T00:24:59.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:24:58 smithi193 ceph-mon[36878]: mds.? [v2:172.21.15.118:6834/395010504,v1:172.21.15.118:6835/395010504] up:boot 2024-03-24T00:24:59.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:24:58 smithi193 ceph-mon[36878]: Standby daemon mds.j assigned to filesystem cephfs as rank 3 2024-03-24T00:24:59.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:24:58 smithi193 ceph-mon[36878]: fsmap cephfs:4/5 {0=b=up:active,1=d=up:active,2=c=up:active,4=e=up:active} 3 up:standby-replay 1 up:standby, 1 failed 2024-03-24T00:24:59.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:24:58 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "j"}]: dispatch 2024-03-24T00:24:59.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:24:58 smithi193 ceph-mon[36878]: fsmap cephfs:5/5 3 up:standby-replay 1 up:replay 4 up:active 2024-03-24T00:24:59.323 INFO:journalctl@ceph.mds.d.smithi118.stdout:Mar 24 00:24:59 smithi118 podman[97359]: 2024-03-24 00:24:58.957038764 +0000 UTC m=+0.035862894 image pull c664afe56bb01db4f9afa9532c20598e201e664f2e9ba2be50db0d52674a26d2 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc 2024-03-24T00:24:59.324 INFO:journalctl@ceph.mds.d.smithi118.stdout:Mar 24 00:24:59 smithi118 podman[97359]: 2024-03-24 00:24:59.106127843 +0000 UTC m=+0.184951997 container create 1a8d39fc36fce7050e31d67be4b4bb268c7cea54024202f989e6aed27e64df53 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-d, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, io.buildah.version=1.33.5, GIT_CLEAN=True, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240319) 2024-03-24T00:24:59.696 INFO:journalctl@ceph.mds.d.smithi118.stdout:Mar 24 00:24:59 smithi118 podman[97359]: 2024-03-24 00:24:59.434752617 +0000 UTC m=+0.513576835 container init 1a8d39fc36fce7050e31d67be4b4bb268c7cea54024202f989e6aed27e64df53 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-d, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, GIT_BRANCH=HEAD, ceph=True, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240319) 2024-03-24T00:24:59.697 INFO:journalctl@ceph.mds.d.smithi118.stdout:Mar 24 00:24:59 smithi118 podman[97359]: 2024-03-24 00:24:59.445608284 +0000 UTC m=+0.524432465 container start 1a8d39fc36fce7050e31d67be4b4bb268c7cea54024202f989e6aed27e64df53 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-d, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240319, org.label-schema.license=GPLv2, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD) 2024-03-24T00:24:59.697 INFO:journalctl@ceph.mds.d.smithi118.stdout:Mar 24 00:24:59 smithi118 ceph-mds[97378]: -- 172.21.15.118:0/3996355047 <== mon.1 v2:172.21.15.193:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x5628e5491a40 con 0x5628e5464800 2024-03-24T00:24:59.697 INFO:journalctl@ceph.mds.d.smithi118.stdout:Mar 24 00:24:59 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-d[97374]: starting mds.d at 2024-03-24T00:24:59.697 INFO:journalctl@ceph.mds.d.smithi118.stdout:Mar 24 00:24:59 smithi118 bash[97359]: 1a8d39fc36fce7050e31d67be4b4bb268c7cea54024202f989e6aed27e64df53 2024-03-24T00:24:59.697 INFO:journalctl@ceph.mds.d.smithi118.stdout:Mar 24 00:24:59 smithi118 systemd[1]: Started Ceph mds.d for d79a5420-e971-11ee-95cd-87774f69a715. 2024-03-24T00:25:00.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:24:59 smithi118 ceph-mon[31535]: pgmap v708: 97 pgs: 97 active+clean; 147 MiB data, 881 MiB used, 1.0 TiB / 1.0 TiB avail; 370 KiB/s rd, 18 KiB/s wr, 46 op/s 2024-03-24T00:25:00.197 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:24:59 smithi118 ceph-mon[31535]: daemon mds.d restarted 2024-03-24T00:25:00.197 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:24:59 smithi118 ceph-mon[31535]: osdmap e124: 12 total, 12 up, 12 in 2024-03-24T00:25:00.197 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:24:59 smithi118 ceph-mon[31535]: mds.? [v2:172.21.15.118:6834/395010504,v1:172.21.15.118:6835/395010504] up:resolve 2024-03-24T00:25:00.197 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:24:59 smithi118 ceph-mon[31535]: mds.? [v2:172.21.15.118:6836/3350294232,v1:172.21.15.118:6837/3350294232] up:boot 2024-03-24T00:25:00.197 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:24:59 smithi118 ceph-mon[31535]: Standby daemon mds.d assigned to filesystem cephfs as rank 1 2024-03-24T00:25:00.197 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:24:59 smithi118 ceph-mon[31535]: fsmap cephfs:4/5 {0=b=up:active,2=c=up:active,3=j=up:resolve,4=e=up:active} 3 up:standby-replay 1 up:standby, 1 failed 2024-03-24T00:25:00.197 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:24:59 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2024-03-24T00:25:00.197 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:24:59 smithi118 ceph-mon[31535]: fsmap cephfs:5/5 3 up:standby-replay 1 up:replay 1 up:resolve 3 up:active 2024-03-24T00:25:00.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:24:59 smithi189 ceph-mon[36187]: pgmap v708: 97 pgs: 97 active+clean; 147 MiB data, 881 MiB used, 1.0 TiB / 1.0 TiB avail; 370 KiB/s rd, 18 KiB/s wr, 46 op/s 2024-03-24T00:25:00.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:24:59 smithi189 ceph-mon[36187]: daemon mds.d restarted 2024-03-24T00:25:00.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:24:59 smithi189 ceph-mon[36187]: osdmap e124: 12 total, 12 up, 12 in 2024-03-24T00:25:00.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:24:59 smithi189 ceph-mon[36187]: mds.? [v2:172.21.15.118:6834/395010504,v1:172.21.15.118:6835/395010504] up:resolve 2024-03-24T00:25:00.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:24:59 smithi189 ceph-mon[36187]: mds.? [v2:172.21.15.118:6836/3350294232,v1:172.21.15.118:6837/3350294232] up:boot 2024-03-24T00:25:00.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:24:59 smithi189 ceph-mon[36187]: Standby daemon mds.d assigned to filesystem cephfs as rank 1 2024-03-24T00:25:00.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:24:59 smithi189 ceph-mon[36187]: fsmap cephfs:4/5 {0=b=up:active,2=c=up:active,3=j=up:resolve,4=e=up:active} 3 up:standby-replay 1 up:standby, 1 failed 2024-03-24T00:25:00.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:24:59 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2024-03-24T00:25:00.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:24:59 smithi189 ceph-mon[36187]: fsmap cephfs:5/5 3 up:standby-replay 1 up:replay 1 up:resolve 3 up:active 2024-03-24T00:25:00.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:24:59 smithi193 ceph-mon[36878]: pgmap v708: 97 pgs: 97 active+clean; 147 MiB data, 881 MiB used, 1.0 TiB / 1.0 TiB avail; 370 KiB/s rd, 18 KiB/s wr, 46 op/s 2024-03-24T00:25:00.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:24:59 smithi193 ceph-mon[36878]: daemon mds.d restarted 2024-03-24T00:25:00.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:24:59 smithi193 ceph-mon[36878]: osdmap e124: 12 total, 12 up, 12 in 2024-03-24T00:25:00.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:24:59 smithi193 ceph-mon[36878]: mds.? [v2:172.21.15.118:6834/395010504,v1:172.21.15.118:6835/395010504] up:resolve 2024-03-24T00:25:00.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:24:59 smithi193 ceph-mon[36878]: mds.? [v2:172.21.15.118:6836/3350294232,v1:172.21.15.118:6837/3350294232] up:boot 2024-03-24T00:25:00.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:24:59 smithi193 ceph-mon[36878]: Standby daemon mds.d assigned to filesystem cephfs as rank 1 2024-03-24T00:25:00.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:24:59 smithi193 ceph-mon[36878]: fsmap cephfs:4/5 {0=b=up:active,2=c=up:active,3=j=up:resolve,4=e=up:active} 3 up:standby-replay 1 up:standby, 1 failed 2024-03-24T00:25:00.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:24:59 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "d"}]: dispatch 2024-03-24T00:25:00.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:24:59 smithi193 ceph-mon[36878]: fsmap cephfs:5/5 3 up:standby-replay 1 up:replay 1 up:resolve 3 up:active 2024-03-24T00:25:01.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:25:01 smithi118 ceph-mon[31535]: pgmap v711: 97 pgs: 97 active+clean; 147 MiB data, 881 MiB used, 1.0 TiB / 1.0 TiB avail; 26 KiB/s rd, 22 KiB/s wr, 9 op/s 2024-03-24T00:25:01.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:25:01 smithi118 ceph-mon[31535]: mds.? [v2:172.21.15.118:6836/3350294232,v1:172.21.15.118:6837/3350294232] up:resolve 2024-03-24T00:25:01.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:25:01 smithi118 ceph-mon[31535]: fsmap cephfs:5/5 3 up:standby-replay 2 up:resolve 3 up:active 2024-03-24T00:25:01.953 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:25:01 smithi189 ceph-mon[36187]: pgmap v711: 97 pgs: 97 active+clean; 147 MiB data, 881 MiB used, 1.0 TiB / 1.0 TiB avail; 26 KiB/s rd, 22 KiB/s wr, 9 op/s 2024-03-24T00:25:01.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:25:01 smithi189 ceph-mon[36187]: mds.? [v2:172.21.15.118:6836/3350294232,v1:172.21.15.118:6837/3350294232] up:resolve 2024-03-24T00:25:01.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:25:01 smithi189 ceph-mon[36187]: fsmap cephfs:5/5 3 up:standby-replay 2 up:resolve 3 up:active 2024-03-24T00:25:01.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:25:01 smithi193 ceph-mon[36878]: pgmap v711: 97 pgs: 97 active+clean; 147 MiB data, 881 MiB used, 1.0 TiB / 1.0 TiB avail; 26 KiB/s rd, 22 KiB/s wr, 9 op/s 2024-03-24T00:25:01.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:25:01 smithi193 ceph-mon[36878]: mds.? [v2:172.21.15.118:6836/3350294232,v1:172.21.15.118:6837/3350294232] up:resolve 2024-03-24T00:25:01.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:25:01 smithi193 ceph-mon[36878]: fsmap cephfs:5/5 3 up:standby-replay 2 up:resolve 3 up:active 2024-03-24T00:25:02.564 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:25:02.564+0000 7f3907e00640 1 -- 172.21.15.118:0/1431104 <== mds.0 v2:172.21.15.189:6838/4074484168 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+44 (crc 0 0 0) 0x563c8c1040c0 con 0x563c8c19cd40 2024-03-24T00:25:02.565 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:25:02.564+0000 7f3907e00640 10 client.27053 handle_command_reply: tid=0 2024-03-24T00:25:02.565 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:25:02.564+0000 7f392c000740 1 client.27053 shutdown 2024-03-24T00:25:02.566 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:25:02.565+0000 7f392c000740 1 -- 172.21.15.118:0/1431104 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x563c8c192750 msgr2=0x563c8c1aebd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:25:02.566 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:25:02.565+0000 7f392c000740 1 --2- 172.21.15.118:0/1431104 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x563c8c192750 0x563c8c1aebd0 secure :-1 s=READY pgs=467 cs=0 l=1 rev1=1 crypto rx=0x7f3914002230 tx=0x7f3914009cf0 comp rx=0 tx=0).stop 2024-03-24T00:25:02.566 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:25:02.565+0000 7f392c000740 1 -- 172.21.15.118:0/1431104 shutdown_connections 2024-03-24T00:25:02.567 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:25:02.565+0000 7f392c000740 1 --2- 172.21.15.118:0/1431104 >> [v2:172.21.15.189:6838/4074484168,v1:172.21.15.189:6839/4074484168] conn(0x563c8c19cd40 0x563c8c214060 crc :-1 s=READY pgs=419 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:25:02.567 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:25:02.565+0000 7f392c000740 1 --2- 172.21.15.118:0/1431104 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x563c8c197bd0 0x563c8c1b0080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:25:02.567 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:25:02.565+0000 7f392c000740 1 --2- 172.21.15.118:0/1431104 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x563c8c1971e0 0x563c8c1af110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:25:02.567 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:25:02.565+0000 7f392c000740 1 --2- 172.21.15.118:0/1431104 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x563c8c192750 0x563c8c1aebd0 unknown :-1 s=CLOSED pgs=467 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:25:02.567 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:25:02.565+0000 7f3907e00640 0 client.27053 ms_handle_reset on v2:172.21.15.189:6838/4074484168 2024-03-24T00:25:02.567 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:25:02.565+0000 7f392c000740 1 -- 172.21.15.118:0/1431104 >> 172.21.15.118:0/1431104 conn(0x563c8c19c810 msgr2=0x563c8c1a9810 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:25:02.567 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:25:02.566+0000 7f392c000740 1 -- 172.21.15.118:0/1431104 shutdown_connections 2024-03-24T00:25:02.567 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:25:02.566+0000 7f392c000740 1 -- 172.21.15.118:0/1431104 wait complete. 2024-03-24T00:25:02.567 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:25:02.566+0000 7f392c000740 20 client.27053 trim_cache size 0 max 16384 2024-03-24T00:25:02.568 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stdout:{ 2024-03-24T00:25:02.568 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stdout: "message": "", 2024-03-24T00:25:02.568 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stdout: "return_code": 0 2024-03-24T00:25:02.568 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stdout:} 2024-03-24T00:25:02.569 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:25:02.567+0000 7f3929400640 1 -- 172.21.15.118:0/4172884086 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7f39080a28a0 msgr2=0x7f39080a4d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:25:02.569 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:25:02.567+0000 7f3929400640 1 --2- 172.21.15.118:0/4172884086 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7f39080a28a0 0x7f39080a4d60 secure :-1 s=READY pgs=398 cs=0 l=1 rev1=1 crypto rx=0x7f390c02c240 tx=0x7f390c005a40 comp rx=0 tx=0).stop 2024-03-24T00:25:02.569 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:25:02.567+0000 7f3929400640 1 -- 172.21.15.118:0/4172884086 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f39241054b0 msgr2=0x7f39241a10b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:25:02.569 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:25:02.567+0000 7f3929400640 1 --2- 172.21.15.118:0/4172884086 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f39241054b0 0x7f39241a10b0 secure :-1 s=READY pgs=466 cs=0 l=1 rev1=1 crypto rx=0x7f391c00e970 tx=0x7f391c00ee30 comp rx=0 tx=0).stop 2024-03-24T00:25:02.569 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:25:02.568+0000 7f3929400640 1 -- 172.21.15.118:0/4172884086 shutdown_connections 2024-03-24T00:25:02.569 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:25:02.568+0000 7f3929400640 1 --2- 172.21.15.118:0/4172884086 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7f39080a28a0 0x7f39080a4d60 unknown :-1 s=CLOSED pgs=398 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:25:02.569 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:25:02.568+0000 7f3929400640 1 --2- 172.21.15.118:0/4172884086 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f39241054b0 0x7f39241a10b0 unknown :-1 s=CLOSED pgs=466 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:25:02.569 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:25:02.568+0000 7f3929400640 1 --2- 172.21.15.118:0/4172884086 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f3924104af0 0x7f392419a040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:25:02.569 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:25:02.568+0000 7f3929400640 1 --2- 172.21.15.118:0/4172884086 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f3924103900 0x7f3924199b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:25:02.569 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:25:02.568+0000 7f3929400640 1 -- 172.21.15.118:0/4172884086 >> 172.21.15.118:0/4172884086 conn(0x7f39240fee90 msgr2=0x7f3924107d20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:25:02.569 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:25:02.568+0000 7f3929400640 1 -- 172.21.15.118:0/4172884086 shutdown_connections 2024-03-24T00:25:02.569 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:25:02.568+0000 7f3929400640 1 -- 172.21.15.118:0/4172884086 wait complete. 2024-03-24T00:25:02.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:25:02 smithi118 ceph-mon[31535]: mds.? [v2:172.21.15.118:6834/395010504,v1:172.21.15.118:6835/395010504] up:reconnect 2024-03-24T00:25:02.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:25:02 smithi118 ceph-mon[31535]: mds.? [v2:172.21.15.118:6836/3350294232,v1:172.21.15.118:6837/3350294232] up:reconnect 2024-03-24T00:25:02.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:25:02 smithi118 ceph-mon[31535]: fsmap cephfs:5/5 3 up:standby-replay 2 up:reconnect 3 up:active 2024-03-24T00:25:02.947 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:25:02 smithi118 ceph-mon[31535]: reconnect by client.15159 192.168.0.1:0/3463739313 after 0.0169996 2024-03-24T00:25:02.947 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:25:02 smithi118 ceph-mon[31535]: reconnect by client.15159 192.168.0.1:0/3463739313 after 0.0249994 2024-03-24T00:25:02.953 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:25:02 smithi189 ceph-mon[36187]: mds.? [v2:172.21.15.118:6834/395010504,v1:172.21.15.118:6835/395010504] up:reconnect 2024-03-24T00:25:02.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:25:02 smithi189 ceph-mon[36187]: mds.? [v2:172.21.15.118:6836/3350294232,v1:172.21.15.118:6837/3350294232] up:reconnect 2024-03-24T00:25:02.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:25:02 smithi189 ceph-mon[36187]: fsmap cephfs:5/5 3 up:standby-replay 2 up:reconnect 3 up:active 2024-03-24T00:25:02.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:25:02 smithi189 ceph-mon[36187]: reconnect by client.15159 192.168.0.1:0/3463739313 after 0.0169996 2024-03-24T00:25:02.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:25:02 smithi189 ceph-mon[36187]: reconnect by client.15159 192.168.0.1:0/3463739313 after 0.0249994 2024-03-24T00:25:02.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:25:02 smithi193 ceph-mon[36878]: mds.? [v2:172.21.15.118:6834/395010504,v1:172.21.15.118:6835/395010504] up:reconnect 2024-03-24T00:25:02.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:25:02 smithi193 ceph-mon[36878]: mds.? [v2:172.21.15.118:6836/3350294232,v1:172.21.15.118:6837/3350294232] up:reconnect 2024-03-24T00:25:02.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:25:02 smithi193 ceph-mon[36878]: fsmap cephfs:5/5 3 up:standby-replay 2 up:reconnect 3 up:active 2024-03-24T00:25:02.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:25:02 smithi193 ceph-mon[36878]: reconnect by client.15159 192.168.0.1:0/3463739313 after 0.0169996 2024-03-24T00:25:02.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:25:02 smithi193 ceph-mon[36878]: reconnect by client.15159 192.168.0.1:0/3463739313 after 0.0249994 2024-03-24T00:25:03.855 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:25:03 smithi189 ceph-mon[36187]: pgmap v712: 97 pgs: 97 active+clean; 147 MiB data, 881 MiB used, 1.0 TiB / 1.0 TiB avail; 31 KiB/s rd, 22 KiB/s wr, 8 op/s 2024-03-24T00:25:03.855 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:25:03 smithi189 ceph-mon[36187]: mds.? [v2:172.21.15.118:6836/3350294232,v1:172.21.15.118:6837/3350294232] up:rejoin 2024-03-24T00:25:03.855 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:25:03 smithi189 ceph-mon[36187]: mds.? [v2:172.21.15.118:6834/395010504,v1:172.21.15.118:6835/395010504] up:rejoin 2024-03-24T00:25:03.855 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:25:03 smithi189 ceph-mon[36187]: fsmap cephfs:5/5 3 up:standby-replay 2 up:rejoin 3 up:active 2024-03-24T00:25:03.856 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:25:03 smithi189 ceph-mon[36187]: daemon mds.j is now active in filesystem cephfs as rank 3 2024-03-24T00:25:03.856 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:25:03 smithi189 ceph-mon[36187]: daemon mds.d is now active in filesystem cephfs as rank 1 2024-03-24T00:25:03.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:25:03 smithi118 ceph-mon[31535]: pgmap v712: 97 pgs: 97 active+clean; 147 MiB data, 881 MiB used, 1.0 TiB / 1.0 TiB avail; 31 KiB/s rd, 22 KiB/s wr, 8 op/s 2024-03-24T00:25:03.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:25:03 smithi118 ceph-mon[31535]: mds.? [v2:172.21.15.118:6836/3350294232,v1:172.21.15.118:6837/3350294232] up:rejoin 2024-03-24T00:25:03.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:25:03 smithi118 ceph-mon[31535]: mds.? [v2:172.21.15.118:6834/395010504,v1:172.21.15.118:6835/395010504] up:rejoin 2024-03-24T00:25:03.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:25:03 smithi118 ceph-mon[31535]: fsmap cephfs:5/5 3 up:standby-replay 2 up:rejoin 3 up:active 2024-03-24T00:25:03.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:25:03 smithi118 ceph-mon[31535]: daemon mds.j is now active in filesystem cephfs as rank 3 2024-03-24T00:25:03.947 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:25:03 smithi118 ceph-mon[31535]: daemon mds.d is now active in filesystem cephfs as rank 1 2024-03-24T00:25:03.947 INFO:journalctl@ceph.mds.d.smithi118.stdout:Mar 24 00:25:03 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-d[97374]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: In function 'void Locker::set_xlocks_done(MutationImpl*, bool)' thread 7ff343600700 time 2024-03-24T00:25:03.636500+0000 2024-03-24T00:25:03.947 INFO:journalctl@ceph.mds.d.smithi118.stdout:Mar 24 00:25:03 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-d[97374]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: 626: FAILED ceph_assert(obj->is_auth()) 2024-03-24T00:25:03.947 INFO:journalctl@ceph.mds.d.smithi118.stdout:Mar 24 00:25:03 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-d[97374]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-24T00:25:03.947 INFO:journalctl@ceph.mds.d.smithi118.stdout:Mar 24 00:25:03 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-d[97374]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x135) [0x7ff34ef694a1] 2024-03-24T00:25:03.947 INFO:journalctl@ceph.mds.d.smithi118.stdout:Mar 24 00:25:03 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-d[97374]: 2: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7ff34ef69667] 2024-03-24T00:25:03.947 INFO:journalctl@ceph.mds.d.smithi118.stdout:Mar 24 00:25:03 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-d[97374]: 3: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x5628e32b0223] 2024-03-24T00:25:03.948 INFO:journalctl@ceph.mds.d.smithi118.stdout:Mar 24 00:25:03 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-d[97374]: 4: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x5628e312a5d6] 2024-03-24T00:25:03.948 INFO:journalctl@ceph.mds.d.smithi118.stdout:Mar 24 00:25:03 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-d[97374]: 5: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x5628e312ada1] 2024-03-24T00:25:03.948 INFO:journalctl@ceph.mds.d.smithi118.stdout:Mar 24 00:25:03 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-d[97374]: 6: (Server::handle_client_rename(boost::intrusive_ptr const&)+0x259d) [0x5628e316797d] 2024-03-24T00:25:03.948 INFO:journalctl@ceph.mds.d.smithi118.stdout:Mar 24 00:25:03 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-d[97374]: 7: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x5628e317d233] 2024-03-24T00:25:03.948 INFO:journalctl@ceph.mds.d.smithi118.stdout:Mar 24 00:25:03 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-d[97374]: 8: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x5628e317f197] 2024-03-24T00:25:03.948 INFO:journalctl@ceph.mds.d.smithi118.stdout:Mar 24 00:25:03 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-d[97374]: 9: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x5628e307e11b] 2024-03-24T00:25:03.948 INFO:journalctl@ceph.mds.d.smithi118.stdout:Mar 24 00:25:03 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-d[97374]: 10: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x5628e307e6bc] 2024-03-24T00:25:03.948 INFO:journalctl@ceph.mds.d.smithi118.stdout:Mar 24 00:25:03 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-d[97374]: 11: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x5628e30681ff] 2024-03-24T00:25:03.948 INFO:journalctl@ceph.mds.d.smithi118.stdout:Mar 24 00:25:03 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-d[97374]: 12: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7ff34f1fb488] 2024-03-24T00:25:03.948 INFO:journalctl@ceph.mds.d.smithi118.stdout:Mar 24 00:25:03 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-d[97374]: 13: (DispatchQueue::entry()+0x51f) [0x7ff34f1f860f] 2024-03-24T00:25:03.948 INFO:journalctl@ceph.mds.d.smithi118.stdout:Mar 24 00:25:03 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-d[97374]: 14: (DispatchQueue::DispatchThread::entry()+0x11) [0x7ff34f2cbe81] 2024-03-24T00:25:03.948 INFO:journalctl@ceph.mds.d.smithi118.stdout:Mar 24 00:25:03 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-d[97374]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7ff34dce61ca] 2024-03-24T00:25:03.948 INFO:journalctl@ceph.mds.d.smithi118.stdout:Mar 24 00:25:03 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-d[97374]: 16: clone() 2024-03-24T00:25:03.949 INFO:journalctl@ceph.mds.j.smithi118.stdout:Mar 24 00:25:03 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-j[97290]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: In function 'void Locker::set_xlocks_done(MutationImpl*, bool)' thread 7fcee6a00700 time 2024-03-24T00:25:03.579717+0000 2024-03-24T00:25:03.949 INFO:journalctl@ceph.mds.j.smithi118.stdout:Mar 24 00:25:03 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-j[97290]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: 626: FAILED ceph_assert(obj->is_auth()) 2024-03-24T00:25:03.949 INFO:journalctl@ceph.mds.j.smithi118.stdout:Mar 24 00:25:03 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-j[97290]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-24T00:25:03.949 INFO:journalctl@ceph.mds.j.smithi118.stdout:Mar 24 00:25:03 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-j[97290]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x135) [0x7fcef22b94a1] 2024-03-24T00:25:03.949 INFO:journalctl@ceph.mds.j.smithi118.stdout:Mar 24 00:25:03 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-j[97290]: 2: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7fcef22b9667] 2024-03-24T00:25:03.949 INFO:journalctl@ceph.mds.j.smithi118.stdout:Mar 24 00:25:03 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-j[97290]: 3: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x559597dad223] 2024-03-24T00:25:03.949 INFO:journalctl@ceph.mds.j.smithi118.stdout:Mar 24 00:25:03 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-j[97290]: 4: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x559597c275d6] 2024-03-24T00:25:03.949 INFO:journalctl@ceph.mds.j.smithi118.stdout:Mar 24 00:25:03 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-j[97290]: 5: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x559597c27da1] 2024-03-24T00:25:03.949 INFO:journalctl@ceph.mds.j.smithi118.stdout:Mar 24 00:25:03 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-j[97290]: 6: (Server::handle_client_rename(boost::intrusive_ptr const&)+0x259d) [0x559597c6497d] 2024-03-24T00:25:03.949 INFO:journalctl@ceph.mds.j.smithi118.stdout:Mar 24 00:25:03 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-j[97290]: 7: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x559597c7a233] 2024-03-24T00:25:03.950 INFO:journalctl@ceph.mds.j.smithi118.stdout:Mar 24 00:25:03 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-j[97290]: 8: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x559597c7c197] 2024-03-24T00:25:03.950 INFO:journalctl@ceph.mds.j.smithi118.stdout:Mar 24 00:25:03 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-j[97290]: 9: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x559597b7b11b] 2024-03-24T00:25:03.950 INFO:journalctl@ceph.mds.j.smithi118.stdout:Mar 24 00:25:03 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-j[97290]: 10: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x559597b7b6bc] 2024-03-24T00:25:03.950 INFO:journalctl@ceph.mds.j.smithi118.stdout:Mar 24 00:25:03 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-j[97290]: 11: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x559597b651ff] 2024-03-24T00:25:03.950 INFO:journalctl@ceph.mds.j.smithi118.stdout:Mar 24 00:25:03 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-j[97290]: 12: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7fcef254b488] 2024-03-24T00:25:03.950 INFO:journalctl@ceph.mds.j.smithi118.stdout:Mar 24 00:25:03 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-j[97290]: 13: (DispatchQueue::entry()+0x51f) [0x7fcef254860f] 2024-03-24T00:25:03.950 INFO:journalctl@ceph.mds.j.smithi118.stdout:Mar 24 00:25:03 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-j[97290]: 14: (DispatchQueue::DispatchThread::entry()+0x11) [0x7fcef261be81] 2024-03-24T00:25:03.950 INFO:journalctl@ceph.mds.j.smithi118.stdout:Mar 24 00:25:03 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-j[97290]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7fcef10361ca] 2024-03-24T00:25:03.950 INFO:journalctl@ceph.mds.j.smithi118.stdout:Mar 24 00:25:03 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-j[97290]: 16: clone() 2024-03-24T00:25:03.951 INFO:journalctl@ceph.mds.d.smithi118.stdout:Mar 24 00:25:03 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-d[97374]: 2024-03-24T00:25:03.637+0000 7ff343600700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: In function 'void Locker::set_xlocks_done(MutationImpl*, bool)' thread 7ff343600700 time 2024-03-24T00:25:03.636500+0000 2024-03-24T00:25:03.951 INFO:journalctl@ceph.mds.d.smithi118.stdout:Mar 24 00:25:03 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-d[97374]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: 626: FAILED ceph_assert(obj->is_auth()) 2024-03-24T00:25:03.951 INFO:journalctl@ceph.mds.d.smithi118.stdout:Mar 24 00:25:03 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-d[97374]: 2024-03-24T00:25:03.951 INFO:journalctl@ceph.mds.d.smithi118.stdout:Mar 24 00:25:03 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-d[97374]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-24T00:25:03.951 INFO:journalctl@ceph.mds.d.smithi118.stdout:Mar 24 00:25:03 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-d[97374]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x135) [0x7ff34ef694a1] 2024-03-24T00:25:03.951 INFO:journalctl@ceph.mds.d.smithi118.stdout:Mar 24 00:25:03 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-d[97374]: 2: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7ff34ef69667] 2024-03-24T00:25:03.951 INFO:journalctl@ceph.mds.d.smithi118.stdout:Mar 24 00:25:03 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-d[97374]: 3: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x5628e32b0223] 2024-03-24T00:25:03.951 INFO:journalctl@ceph.mds.d.smithi118.stdout:Mar 24 00:25:03 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-d[97374]: 4: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x5628e312a5d6] 2024-03-24T00:25:03.951 INFO:journalctl@ceph.mds.d.smithi118.stdout:Mar 24 00:25:03 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-d[97374]: 5: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x5628e312ada1] 2024-03-24T00:25:03.951 INFO:journalctl@ceph.mds.d.smithi118.stdout:Mar 24 00:25:03 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-d[97374]: 6: (Server::handle_client_rename(boost::intrusive_ptr const&)+0x259d) [0x5628e316797d] 2024-03-24T00:25:03.951 INFO:journalctl@ceph.mds.d.smithi118.stdout:Mar 24 00:25:03 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-d[97374]: 7: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x5628e317d233] 2024-03-24T00:25:03.951 INFO:journalctl@ceph.mds.d.smithi118.stdout:Mar 24 00:25:03 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-d[97374]: 8: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x5628e317f197] 2024-03-24T00:25:03.951 INFO:journalctl@ceph.mds.d.smithi118.stdout:Mar 24 00:25:03 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-d[97374]: 9: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x5628e307e11b] 2024-03-24T00:25:03.952 INFO:journalctl@ceph.mds.d.smithi118.stdout:Mar 24 00:25:03 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-d[97374]: 10: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x5628e307e6bc] 2024-03-24T00:25:03.952 INFO:journalctl@ceph.mds.d.smithi118.stdout:Mar 24 00:25:03 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-d[97374]: 11: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x5628e30681ff] 2024-03-24T00:25:03.952 INFO:journalctl@ceph.mds.d.smithi118.stdout:Mar 24 00:25:03 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-d[97374]: 12: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7ff34f1fb488] 2024-03-24T00:25:03.952 INFO:journalctl@ceph.mds.d.smithi118.stdout:Mar 24 00:25:03 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-d[97374]: 13: (DispatchQueue::entry()+0x51f) [0x7ff34f1f860f] 2024-03-24T00:25:03.952 INFO:journalctl@ceph.mds.d.smithi118.stdout:Mar 24 00:25:03 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-d[97374]: 14: (DispatchQueue::DispatchThread::entry()+0x11) [0x7ff34f2cbe81] 2024-03-24T00:25:03.952 INFO:journalctl@ceph.mds.d.smithi118.stdout:Mar 24 00:25:03 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-d[97374]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7ff34dce61ca] 2024-03-24T00:25:03.952 INFO:journalctl@ceph.mds.d.smithi118.stdout:Mar 24 00:25:03 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-d[97374]: 16: clone() 2024-03-24T00:25:03.952 INFO:journalctl@ceph.mds.d.smithi118.stdout:Mar 24 00:25:03 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-d[97374]: 2024-03-24T00:25:03.952 INFO:journalctl@ceph.mds.d.smithi118.stdout:Mar 24 00:25:03 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-d[97374]: *** Caught signal (Aborted) ** 2024-03-24T00:25:03.952 INFO:journalctl@ceph.mds.d.smithi118.stdout:Mar 24 00:25:03 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-d[97374]: in thread 7ff343600700 thread_name:ms_dispatch 2024-03-24T00:25:03.952 INFO:journalctl@ceph.mds.d.smithi118.stdout:Mar 24 00:25:03 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-d[97374]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-24T00:25:03.952 INFO:journalctl@ceph.mds.d.smithi118.stdout:Mar 24 00:25:03 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-d[97374]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7ff34dcf0d20] 2024-03-24T00:25:03.953 INFO:journalctl@ceph.mds.j.smithi118.stdout:Mar 24 00:25:03 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-j[97290]: 2024-03-24T00:25:03.582+0000 7fcee6a00700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: In function 'void Locker::set_xlocks_done(MutationImpl*, bool)' thread 7fcee6a00700 time 2024-03-24T00:25:03.579717+0000 2024-03-24T00:25:03.953 INFO:journalctl@ceph.mds.j.smithi118.stdout:Mar 24 00:25:03 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-j[97290]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: 626: FAILED ceph_assert(obj->is_auth()) 2024-03-24T00:25:03.953 INFO:journalctl@ceph.mds.j.smithi118.stdout:Mar 24 00:25:03 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-j[97290]: 2024-03-24T00:25:03.953 INFO:journalctl@ceph.mds.j.smithi118.stdout:Mar 24 00:25:03 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-j[97290]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-24T00:25:03.953 INFO:journalctl@ceph.mds.j.smithi118.stdout:Mar 24 00:25:03 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-j[97290]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x135) [0x7fcef22b94a1] 2024-03-24T00:25:03.953 INFO:journalctl@ceph.mds.j.smithi118.stdout:Mar 24 00:25:03 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-j[97290]: 2: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7fcef22b9667] 2024-03-24T00:25:03.953 INFO:journalctl@ceph.mds.j.smithi118.stdout:Mar 24 00:25:03 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-j[97290]: 3: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x559597dad223] 2024-03-24T00:25:03.953 INFO:journalctl@ceph.mds.j.smithi118.stdout:Mar 24 00:25:03 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-j[97290]: 4: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x559597c275d6] 2024-03-24T00:25:03.953 INFO:journalctl@ceph.mds.j.smithi118.stdout:Mar 24 00:25:03 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-j[97290]: 5: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x559597c27da1] 2024-03-24T00:25:03.953 INFO:journalctl@ceph.mds.j.smithi118.stdout:Mar 24 00:25:03 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-j[97290]: 6: (Server::handle_client_rename(boost::intrusive_ptr const&)+0x259d) [0x559597c6497d] 2024-03-24T00:25:03.954 INFO:journalctl@ceph.mds.j.smithi118.stdout:Mar 24 00:25:03 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-j[97290]: 7: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x559597c7a233] 2024-03-24T00:25:03.954 INFO:journalctl@ceph.mds.j.smithi118.stdout:Mar 24 00:25:03 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-j[97290]: 8: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x559597c7c197] 2024-03-24T00:25:03.954 INFO:journalctl@ceph.mds.j.smithi118.stdout:Mar 24 00:25:03 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-j[97290]: 9: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x559597b7b11b] 2024-03-24T00:25:03.954 INFO:journalctl@ceph.mds.j.smithi118.stdout:Mar 24 00:25:03 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-j[97290]: 10: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x559597b7b6bc] 2024-03-24T00:25:03.954 INFO:journalctl@ceph.mds.j.smithi118.stdout:Mar 24 00:25:03 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-j[97290]: 11: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x559597b651ff] 2024-03-24T00:25:03.954 INFO:journalctl@ceph.mds.j.smithi118.stdout:Mar 24 00:25:03 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-j[97290]: 12: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7fcef254b488] 2024-03-24T00:25:03.954 INFO:journalctl@ceph.mds.j.smithi118.stdout:Mar 24 00:25:03 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-j[97290]: 13: (DispatchQueue::entry()+0x51f) [0x7fcef254860f] 2024-03-24T00:25:03.954 INFO:journalctl@ceph.mds.j.smithi118.stdout:Mar 24 00:25:03 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-j[97290]: 14: (DispatchQueue::DispatchThread::entry()+0x11) [0x7fcef261be81] 2024-03-24T00:25:03.954 INFO:journalctl@ceph.mds.j.smithi118.stdout:Mar 24 00:25:03 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-j[97290]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7fcef10361ca] 2024-03-24T00:25:03.954 INFO:journalctl@ceph.mds.j.smithi118.stdout:Mar 24 00:25:03 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-j[97290]: 16: clone() 2024-03-24T00:25:03.954 INFO:journalctl@ceph.mds.j.smithi118.stdout:Mar 24 00:25:03 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-j[97290]: 2024-03-24T00:25:03.954 INFO:journalctl@ceph.mds.j.smithi118.stdout:Mar 24 00:25:03 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-j[97290]: *** Caught signal (Aborted) ** 2024-03-24T00:25:03.954 INFO:journalctl@ceph.mds.j.smithi118.stdout:Mar 24 00:25:03 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-j[97290]: in thread 7fcee6a00700 thread_name:ms_dispatch 2024-03-24T00:25:03.954 INFO:journalctl@ceph.mds.j.smithi118.stdout:Mar 24 00:25:03 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-j[97290]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-24T00:25:03.955 INFO:journalctl@ceph.mds.j.smithi118.stdout:Mar 24 00:25:03 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-j[97290]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7fcef1040d20] 2024-03-24T00:25:03.955 INFO:journalctl@ceph.mds.d.smithi118.stdout:Mar 24 00:25:03 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-d[97374]: 2: gsignal() 2024-03-24T00:25:03.955 INFO:journalctl@ceph.mds.d.smithi118.stdout:Mar 24 00:25:03 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-d[97374]: 3: abort() 2024-03-24T00:25:03.955 INFO:journalctl@ceph.mds.d.smithi118.stdout:Mar 24 00:25:03 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-d[97374]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x18f) [0x7ff34ef694fb] 2024-03-24T00:25:03.955 INFO:journalctl@ceph.mds.d.smithi118.stdout:Mar 24 00:25:03 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-d[97374]: 5: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7ff34ef69667] 2024-03-24T00:25:03.955 INFO:journalctl@ceph.mds.d.smithi118.stdout:Mar 24 00:25:03 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-d[97374]: 6: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x5628e32b0223] 2024-03-24T00:25:03.955 INFO:journalctl@ceph.mds.d.smithi118.stdout:Mar 24 00:25:03 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-d[97374]: 7: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x5628e312a5d6] 2024-03-24T00:25:03.955 INFO:journalctl@ceph.mds.d.smithi118.stdout:Mar 24 00:25:03 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-d[97374]: 8: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x5628e312ada1] 2024-03-24T00:25:03.955 INFO:journalctl@ceph.mds.d.smithi118.stdout:Mar 24 00:25:03 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-d[97374]: 9: (Server::handle_client_rename(boost::intrusive_ptr const&)+0x259d) [0x5628e316797d] 2024-03-24T00:25:03.955 INFO:journalctl@ceph.mds.d.smithi118.stdout:Mar 24 00:25:03 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-d[97374]: 10: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x5628e317d233] 2024-03-24T00:25:03.956 INFO:journalctl@ceph.mds.d.smithi118.stdout:Mar 24 00:25:03 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-d[97374]: 11: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x5628e317f197] 2024-03-24T00:25:03.956 INFO:journalctl@ceph.mds.d.smithi118.stdout:Mar 24 00:25:03 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-d[97374]: 12: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x5628e307e11b] 2024-03-24T00:25:03.956 INFO:journalctl@ceph.mds.d.smithi118.stdout:Mar 24 00:25:03 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-d[97374]: 13: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x5628e307e6bc] 2024-03-24T00:25:03.956 INFO:journalctl@ceph.mds.d.smithi118.stdout:Mar 24 00:25:03 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-d[97374]: 14: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x5628e30681ff] 2024-03-24T00:25:03.956 INFO:journalctl@ceph.mds.d.smithi118.stdout:Mar 24 00:25:03 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-d[97374]: 15: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7ff34f1fb488] 2024-03-24T00:25:03.956 INFO:journalctl@ceph.mds.d.smithi118.stdout:Mar 24 00:25:03 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-d[97374]: 16: (DispatchQueue::entry()+0x51f) [0x7ff34f1f860f] 2024-03-24T00:25:03.956 INFO:journalctl@ceph.mds.d.smithi118.stdout:Mar 24 00:25:03 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-d[97374]: 17: (DispatchQueue::DispatchThread::entry()+0x11) [0x7ff34f2cbe81] 2024-03-24T00:25:03.956 INFO:journalctl@ceph.mds.d.smithi118.stdout:Mar 24 00:25:03 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-d[97374]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7ff34dce61ca] 2024-03-24T00:25:03.956 INFO:journalctl@ceph.mds.d.smithi118.stdout:Mar 24 00:25:03 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-d[97374]: 19: clone() 2024-03-24T00:25:03.956 INFO:journalctl@ceph.mds.d.smithi118.stdout:Mar 24 00:25:03 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-d[97374]: 2024-03-24T00:25:03.640+0000 7ff343600700 -1 *** Caught signal (Aborted) ** 2024-03-24T00:25:03.956 INFO:journalctl@ceph.mds.d.smithi118.stdout:Mar 24 00:25:03 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-d[97374]: in thread 7ff343600700 thread_name:ms_dispatch 2024-03-24T00:25:03.956 INFO:journalctl@ceph.mds.d.smithi118.stdout:Mar 24 00:25:03 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-d[97374]: 2024-03-24T00:25:03.957 INFO:journalctl@ceph.mds.d.smithi118.stdout:Mar 24 00:25:03 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-d[97374]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-24T00:25:03.957 INFO:journalctl@ceph.mds.d.smithi118.stdout:Mar 24 00:25:03 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-d[97374]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7ff34dcf0d20] 2024-03-24T00:25:03.957 INFO:journalctl@ceph.mds.d.smithi118.stdout:Mar 24 00:25:03 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-d[97374]: 2: gsignal() 2024-03-24T00:25:03.957 INFO:journalctl@ceph.mds.d.smithi118.stdout:Mar 24 00:25:03 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-d[97374]: 3: abort() 2024-03-24T00:25:03.957 INFO:journalctl@ceph.mds.d.smithi118.stdout:Mar 24 00:25:03 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-d[97374]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x18f) [0x7ff34ef694fb] 2024-03-24T00:25:03.957 INFO:journalctl@ceph.mds.d.smithi118.stdout:Mar 24 00:25:03 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-d[97374]: 5: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7ff34ef69667] 2024-03-24T00:25:03.958 INFO:journalctl@ceph.mds.j.smithi118.stdout:Mar 24 00:25:03 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-j[97290]: 2: gsignal() 2024-03-24T00:25:03.958 INFO:journalctl@ceph.mds.j.smithi118.stdout:Mar 24 00:25:03 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-j[97290]: 3: abort() 2024-03-24T00:25:03.958 INFO:journalctl@ceph.mds.j.smithi118.stdout:Mar 24 00:25:03 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-j[97290]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x18f) [0x7fcef22b94fb] 2024-03-24T00:25:03.959 INFO:journalctl@ceph.mds.j.smithi118.stdout:Mar 24 00:25:03 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-j[97290]: 5: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7fcef22b9667] 2024-03-24T00:25:03.959 INFO:journalctl@ceph.mds.j.smithi118.stdout:Mar 24 00:25:03 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-j[97290]: 6: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x559597dad223] 2024-03-24T00:25:03.959 INFO:journalctl@ceph.mds.j.smithi118.stdout:Mar 24 00:25:03 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-j[97290]: 7: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x559597c275d6] 2024-03-24T00:25:03.959 INFO:journalctl@ceph.mds.j.smithi118.stdout:Mar 24 00:25:03 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-j[97290]: 8: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x559597c27da1] 2024-03-24T00:25:03.959 INFO:journalctl@ceph.mds.j.smithi118.stdout:Mar 24 00:25:03 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-j[97290]: 9: (Server::handle_client_rename(boost::intrusive_ptr const&)+0x259d) [0x559597c6497d] 2024-03-24T00:25:03.959 INFO:journalctl@ceph.mds.j.smithi118.stdout:Mar 24 00:25:03 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-j[97290]: 10: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x559597c7a233] 2024-03-24T00:25:03.960 INFO:journalctl@ceph.mds.j.smithi118.stdout:Mar 24 00:25:03 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-j[97290]: 11: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x559597c7c197] 2024-03-24T00:25:03.960 INFO:journalctl@ceph.mds.j.smithi118.stdout:Mar 24 00:25:03 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-j[97290]: 12: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x559597b7b11b] 2024-03-24T00:25:03.960 INFO:journalctl@ceph.mds.j.smithi118.stdout:Mar 24 00:25:03 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-j[97290]: 13: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x559597b7b6bc] 2024-03-24T00:25:03.961 INFO:journalctl@ceph.mds.j.smithi118.stdout:Mar 24 00:25:03 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-j[97290]: 14: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x559597b651ff] 2024-03-24T00:25:03.961 INFO:journalctl@ceph.mds.j.smithi118.stdout:Mar 24 00:25:03 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-j[97290]: 15: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7fcef254b488] 2024-03-24T00:25:03.961 INFO:journalctl@ceph.mds.j.smithi118.stdout:Mar 24 00:25:03 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-j[97290]: 16: (DispatchQueue::entry()+0x51f) [0x7fcef254860f] 2024-03-24T00:25:03.961 INFO:journalctl@ceph.mds.j.smithi118.stdout:Mar 24 00:25:03 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-j[97290]: 17: (DispatchQueue::DispatchThread::entry()+0x11) [0x7fcef261be81] 2024-03-24T00:25:03.961 INFO:journalctl@ceph.mds.j.smithi118.stdout:Mar 24 00:25:03 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-j[97290]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7fcef10361ca] 2024-03-24T00:25:03.961 INFO:journalctl@ceph.mds.j.smithi118.stdout:Mar 24 00:25:03 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-j[97290]: 19: clone() 2024-03-24T00:25:03.961 INFO:journalctl@ceph.mds.j.smithi118.stdout:Mar 24 00:25:03 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-j[97290]: 2024-03-24T00:25:03.586+0000 7fcee6a00700 -1 *** Caught signal (Aborted) ** 2024-03-24T00:25:03.961 INFO:journalctl@ceph.mds.j.smithi118.stdout:Mar 24 00:25:03 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-j[97290]: in thread 7fcee6a00700 thread_name:ms_dispatch 2024-03-24T00:25:03.961 INFO:journalctl@ceph.mds.j.smithi118.stdout:Mar 24 00:25:03 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-j[97290]: 2024-03-24T00:25:03.961 INFO:journalctl@ceph.mds.j.smithi118.stdout:Mar 24 00:25:03 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-j[97290]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-24T00:25:03.961 INFO:journalctl@ceph.mds.j.smithi118.stdout:Mar 24 00:25:03 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-j[97290]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7fcef1040d20] 2024-03-24T00:25:03.962 INFO:journalctl@ceph.mds.j.smithi118.stdout:Mar 24 00:25:03 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-j[97290]: 2: gsignal() 2024-03-24T00:25:03.962 INFO:journalctl@ceph.mds.j.smithi118.stdout:Mar 24 00:25:03 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-j[97290]: 3: abort() 2024-03-24T00:25:03.962 INFO:journalctl@ceph.mds.j.smithi118.stdout:Mar 24 00:25:03 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-j[97290]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x18f) [0x7fcef22b94fb] 2024-03-24T00:25:03.962 INFO:journalctl@ceph.mds.j.smithi118.stdout:Mar 24 00:25:03 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-j[97290]: 5: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7fcef22b9667] 2024-03-24T00:25:03.962 INFO:journalctl@ceph.mds.d.smithi118.stdout:Mar 24 00:25:03 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-d[97374]: 6: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x5628e32b0223] 2024-03-24T00:25:03.962 INFO:journalctl@ceph.mds.d.smithi118.stdout:Mar 24 00:25:03 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-d[97374]: 7: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x5628e312a5d6] 2024-03-24T00:25:03.962 INFO:journalctl@ceph.mds.d.smithi118.stdout:Mar 24 00:25:03 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-d[97374]: 8: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x5628e312ada1] 2024-03-24T00:25:03.962 INFO:journalctl@ceph.mds.d.smithi118.stdout:Mar 24 00:25:03 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-d[97374]: 9: (Server::handle_client_rename(boost::intrusive_ptr const&)+0x259d) [0x5628e316797d] 2024-03-24T00:25:03.963 INFO:journalctl@ceph.mds.d.smithi118.stdout:Mar 24 00:25:03 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-d[97374]: 10: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x5628e317d233] 2024-03-24T00:25:03.963 INFO:journalctl@ceph.mds.d.smithi118.stdout:Mar 24 00:25:03 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-d[97374]: 11: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x5628e317f197] 2024-03-24T00:25:03.963 INFO:journalctl@ceph.mds.d.smithi118.stdout:Mar 24 00:25:03 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-d[97374]: 12: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x5628e307e11b] 2024-03-24T00:25:03.963 INFO:journalctl@ceph.mds.d.smithi118.stdout:Mar 24 00:25:03 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-d[97374]: 13: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x5628e307e6bc] 2024-03-24T00:25:03.963 INFO:journalctl@ceph.mds.d.smithi118.stdout:Mar 24 00:25:03 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-d[97374]: 14: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x5628e30681ff] 2024-03-24T00:25:03.963 INFO:journalctl@ceph.mds.d.smithi118.stdout:Mar 24 00:25:03 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-d[97374]: 15: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7ff34f1fb488] 2024-03-24T00:25:03.963 INFO:journalctl@ceph.mds.d.smithi118.stdout:Mar 24 00:25:03 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-d[97374]: 16: (DispatchQueue::entry()+0x51f) [0x7ff34f1f860f] 2024-03-24T00:25:03.963 INFO:journalctl@ceph.mds.d.smithi118.stdout:Mar 24 00:25:03 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-d[97374]: 17: (DispatchQueue::DispatchThread::entry()+0x11) [0x7ff34f2cbe81] 2024-03-24T00:25:03.963 INFO:journalctl@ceph.mds.d.smithi118.stdout:Mar 24 00:25:03 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-d[97374]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7ff34dce61ca] 2024-03-24T00:25:03.963 INFO:journalctl@ceph.mds.d.smithi118.stdout:Mar 24 00:25:03 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-d[97374]: 19: clone() 2024-03-24T00:25:03.963 INFO:journalctl@ceph.mds.d.smithi118.stdout:Mar 24 00:25:03 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-d[97374]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-03-24T00:25:03.963 INFO:journalctl@ceph.mds.d.smithi118.stdout:Mar 24 00:25:03 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-d[97374]: 2024-03-24T00:25:03.963 INFO:journalctl@ceph.mds.d.smithi118.stdout:Mar 24 00:25:03 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-d[97374]: -1> 2024-03-24T00:25:03.637+0000 7ff343600700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: In function 'void Locker::set_xlocks_done(MutationImpl*, bool)' thread 7ff343600700 time 2024-03-24T00:25:03.636500+0000 2024-03-24T00:25:03.964 INFO:journalctl@ceph.mds.d.smithi118.stdout:Mar 24 00:25:03 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-d[97374]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: 626: FAILED ceph_assert(obj->is_auth()) 2024-03-24T00:25:03.964 INFO:journalctl@ceph.mds.d.smithi118.stdout:Mar 24 00:25:03 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-d[97374]: 2024-03-24T00:25:03.964 INFO:journalctl@ceph.mds.d.smithi118.stdout:Mar 24 00:25:03 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-d[97374]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-24T00:25:03.964 INFO:journalctl@ceph.mds.d.smithi118.stdout:Mar 24 00:25:03 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-d[97374]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x135) [0x7ff34ef694a1] 2024-03-24T00:25:03.964 INFO:journalctl@ceph.mds.d.smithi118.stdout:Mar 24 00:25:03 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-d[97374]: 2: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7ff34ef69667] 2024-03-24T00:25:03.964 INFO:journalctl@ceph.mds.j.smithi118.stdout:Mar 24 00:25:03 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-j[97290]: 6: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x559597dad223] 2024-03-24T00:25:03.964 INFO:journalctl@ceph.mds.j.smithi118.stdout:Mar 24 00:25:03 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-j[97290]: 7: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x559597c275d6] 2024-03-24T00:25:03.965 INFO:journalctl@ceph.mds.j.smithi118.stdout:Mar 24 00:25:03 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-j[97290]: 8: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x559597c27da1] 2024-03-24T00:25:03.965 INFO:journalctl@ceph.mds.j.smithi118.stdout:Mar 24 00:25:03 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-j[97290]: 9: (Server::handle_client_rename(boost::intrusive_ptr const&)+0x259d) [0x559597c6497d] 2024-03-24T00:25:03.965 INFO:journalctl@ceph.mds.j.smithi118.stdout:Mar 24 00:25:03 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-j[97290]: 10: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x559597c7a233] 2024-03-24T00:25:03.965 INFO:journalctl@ceph.mds.j.smithi118.stdout:Mar 24 00:25:03 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-j[97290]: 11: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x559597c7c197] 2024-03-24T00:25:03.965 INFO:journalctl@ceph.mds.j.smithi118.stdout:Mar 24 00:25:03 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-j[97290]: 12: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x559597b7b11b] 2024-03-24T00:25:03.965 INFO:journalctl@ceph.mds.j.smithi118.stdout:Mar 24 00:25:03 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-j[97290]: 13: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x559597b7b6bc] 2024-03-24T00:25:03.965 INFO:journalctl@ceph.mds.j.smithi118.stdout:Mar 24 00:25:03 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-j[97290]: 14: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x559597b651ff] 2024-03-24T00:25:03.965 INFO:journalctl@ceph.mds.j.smithi118.stdout:Mar 24 00:25:03 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-j[97290]: 15: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7fcef254b488] 2024-03-24T00:25:03.965 INFO:journalctl@ceph.mds.j.smithi118.stdout:Mar 24 00:25:03 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-j[97290]: 16: (DispatchQueue::entry()+0x51f) [0x7fcef254860f] 2024-03-24T00:25:03.965 INFO:journalctl@ceph.mds.j.smithi118.stdout:Mar 24 00:25:03 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-j[97290]: 17: (DispatchQueue::DispatchThread::entry()+0x11) [0x7fcef261be81] 2024-03-24T00:25:03.965 INFO:journalctl@ceph.mds.j.smithi118.stdout:Mar 24 00:25:03 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-j[97290]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7fcef10361ca] 2024-03-24T00:25:03.966 INFO:journalctl@ceph.mds.j.smithi118.stdout:Mar 24 00:25:03 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-j[97290]: 19: clone() 2024-03-24T00:25:03.966 INFO:journalctl@ceph.mds.j.smithi118.stdout:Mar 24 00:25:03 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-j[97290]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-03-24T00:25:03.966 INFO:journalctl@ceph.mds.j.smithi118.stdout:Mar 24 00:25:03 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-j[97290]: 2024-03-24T00:25:03.966 INFO:journalctl@ceph.mds.j.smithi118.stdout:Mar 24 00:25:03 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-j[97290]: -1> 2024-03-24T00:25:03.582+0000 7fcee6a00700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: In function 'void Locker::set_xlocks_done(MutationImpl*, bool)' thread 7fcee6a00700 time 2024-03-24T00:25:03.579717+0000 2024-03-24T00:25:03.966 INFO:journalctl@ceph.mds.j.smithi118.stdout:Mar 24 00:25:03 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-j[97290]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: 626: FAILED ceph_assert(obj->is_auth()) 2024-03-24T00:25:03.966 INFO:journalctl@ceph.mds.j.smithi118.stdout:Mar 24 00:25:03 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-j[97290]: 2024-03-24T00:25:03.966 INFO:journalctl@ceph.mds.j.smithi118.stdout:Mar 24 00:25:03 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-j[97290]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-24T00:25:03.966 INFO:journalctl@ceph.mds.j.smithi118.stdout:Mar 24 00:25:03 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-j[97290]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x135) [0x7fcef22b94a1] 2024-03-24T00:25:03.966 INFO:journalctl@ceph.mds.j.smithi118.stdout:Mar 24 00:25:03 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-j[97290]: 2: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7fcef22b9667] 2024-03-24T00:25:03.967 INFO:journalctl@ceph.mds.d.smithi118.stdout:Mar 24 00:25:03 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-d[97374]: 3: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x5628e32b0223] 2024-03-24T00:25:03.967 INFO:journalctl@ceph.mds.d.smithi118.stdout:Mar 24 00:25:03 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-d[97374]: 4: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x5628e312a5d6] 2024-03-24T00:25:03.968 INFO:journalctl@ceph.mds.d.smithi118.stdout:Mar 24 00:25:03 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-d[97374]: 5: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x5628e312ada1] 2024-03-24T00:25:03.968 INFO:journalctl@ceph.mds.d.smithi118.stdout:Mar 24 00:25:03 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-d[97374]: 6: (Server::handle_client_rename(boost::intrusive_ptr const&)+0x259d) [0x5628e316797d] 2024-03-24T00:25:03.968 INFO:journalctl@ceph.mds.d.smithi118.stdout:Mar 24 00:25:03 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-d[97374]: 7: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x5628e317d233] 2024-03-24T00:25:03.968 INFO:journalctl@ceph.mds.d.smithi118.stdout:Mar 24 00:25:03 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-d[97374]: 8: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x5628e317f197] 2024-03-24T00:25:03.969 INFO:journalctl@ceph.mds.d.smithi118.stdout:Mar 24 00:25:03 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-d[97374]: 9: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x5628e307e11b] 2024-03-24T00:25:03.969 INFO:journalctl@ceph.mds.d.smithi118.stdout:Mar 24 00:25:03 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-d[97374]: 10: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x5628e307e6bc] 2024-03-24T00:25:03.969 INFO:journalctl@ceph.mds.d.smithi118.stdout:Mar 24 00:25:03 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-d[97374]: 11: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x5628e30681ff] 2024-03-24T00:25:03.970 INFO:journalctl@ceph.mds.d.smithi118.stdout:Mar 24 00:25:03 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-d[97374]: 12: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7ff34f1fb488] 2024-03-24T00:25:03.970 INFO:journalctl@ceph.mds.d.smithi118.stdout:Mar 24 00:25:03 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-d[97374]: 13: (DispatchQueue::entry()+0x51f) [0x7ff34f1f860f] 2024-03-24T00:25:03.970 INFO:journalctl@ceph.mds.d.smithi118.stdout:Mar 24 00:25:03 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-d[97374]: 14: (DispatchQueue::DispatchThread::entry()+0x11) [0x7ff34f2cbe81] 2024-03-24T00:25:03.970 INFO:journalctl@ceph.mds.d.smithi118.stdout:Mar 24 00:25:03 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-d[97374]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7ff34dce61ca] 2024-03-24T00:25:03.970 INFO:journalctl@ceph.mds.d.smithi118.stdout:Mar 24 00:25:03 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-d[97374]: 16: clone() 2024-03-24T00:25:03.970 INFO:journalctl@ceph.mds.d.smithi118.stdout:Mar 24 00:25:03 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-d[97374]: 2024-03-24T00:25:03.970 INFO:journalctl@ceph.mds.d.smithi118.stdout:Mar 24 00:25:03 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-d[97374]: 0> 2024-03-24T00:25:03.640+0000 7ff343600700 -1 *** Caught signal (Aborted) ** 2024-03-24T00:25:03.970 INFO:journalctl@ceph.mds.d.smithi118.stdout:Mar 24 00:25:03 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-d[97374]: in thread 7ff343600700 thread_name:ms_dispatch 2024-03-24T00:25:03.971 INFO:journalctl@ceph.mds.d.smithi118.stdout:Mar 24 00:25:03 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-d[97374]: 2024-03-24T00:25:03.971 INFO:journalctl@ceph.mds.d.smithi118.stdout:Mar 24 00:25:03 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-d[97374]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-24T00:25:03.971 INFO:journalctl@ceph.mds.d.smithi118.stdout:Mar 24 00:25:03 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-d[97374]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7ff34dcf0d20] 2024-03-24T00:25:03.971 INFO:journalctl@ceph.mds.d.smithi118.stdout:Mar 24 00:25:03 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-d[97374]: 2: gsignal() 2024-03-24T00:25:03.971 INFO:journalctl@ceph.mds.d.smithi118.stdout:Mar 24 00:25:03 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-d[97374]: 3: abort() 2024-03-24T00:25:03.971 INFO:journalctl@ceph.mds.d.smithi118.stdout:Mar 24 00:25:03 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-d[97374]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x18f) [0x7ff34ef694fb] 2024-03-24T00:25:03.971 INFO:journalctl@ceph.mds.d.smithi118.stdout:Mar 24 00:25:03 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-d[97374]: 5: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7ff34ef69667] 2024-03-24T00:25:03.971 INFO:journalctl@ceph.mds.d.smithi118.stdout:Mar 24 00:25:03 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-d[97374]: 6: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x5628e32b0223] 2024-03-24T00:25:03.971 INFO:journalctl@ceph.mds.d.smithi118.stdout:Mar 24 00:25:03 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-d[97374]: 7: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x5628e312a5d6] 2024-03-24T00:25:03.971 INFO:journalctl@ceph.mds.j.smithi118.stdout:Mar 24 00:25:03 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-j[97290]: 3: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x559597dad223] 2024-03-24T00:25:03.971 INFO:journalctl@ceph.mds.j.smithi118.stdout:Mar 24 00:25:03 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-j[97290]: 4: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x559597c275d6] 2024-03-24T00:25:03.972 INFO:journalctl@ceph.mds.j.smithi118.stdout:Mar 24 00:25:03 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-j[97290]: 5: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x559597c27da1] 2024-03-24T00:25:03.972 INFO:journalctl@ceph.mds.j.smithi118.stdout:Mar 24 00:25:03 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-j[97290]: 6: (Server::handle_client_rename(boost::intrusive_ptr const&)+0x259d) [0x559597c6497d] 2024-03-24T00:25:03.972 INFO:journalctl@ceph.mds.j.smithi118.stdout:Mar 24 00:25:03 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-j[97290]: 7: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x559597c7a233] 2024-03-24T00:25:03.972 INFO:journalctl@ceph.mds.j.smithi118.stdout:Mar 24 00:25:03 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-j[97290]: 8: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x559597c7c197] 2024-03-24T00:25:03.972 INFO:journalctl@ceph.mds.j.smithi118.stdout:Mar 24 00:25:03 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-j[97290]: 9: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x559597b7b11b] 2024-03-24T00:25:03.972 INFO:journalctl@ceph.mds.j.smithi118.stdout:Mar 24 00:25:03 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-j[97290]: 10: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x559597b7b6bc] 2024-03-24T00:25:03.972 INFO:journalctl@ceph.mds.j.smithi118.stdout:Mar 24 00:25:03 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-j[97290]: 11: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x559597b651ff] 2024-03-24T00:25:03.972 INFO:journalctl@ceph.mds.j.smithi118.stdout:Mar 24 00:25:03 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-j[97290]: 12: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7fcef254b488] 2024-03-24T00:25:03.972 INFO:journalctl@ceph.mds.j.smithi118.stdout:Mar 24 00:25:03 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-j[97290]: 13: (DispatchQueue::entry()+0x51f) [0x7fcef254860f] 2024-03-24T00:25:03.972 INFO:journalctl@ceph.mds.j.smithi118.stdout:Mar 24 00:25:03 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-j[97290]: 14: (DispatchQueue::DispatchThread::entry()+0x11) [0x7fcef261be81] 2024-03-24T00:25:03.972 INFO:journalctl@ceph.mds.j.smithi118.stdout:Mar 24 00:25:03 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-j[97290]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7fcef10361ca] 2024-03-24T00:25:03.972 INFO:journalctl@ceph.mds.j.smithi118.stdout:Mar 24 00:25:03 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-j[97290]: 16: clone() 2024-03-24T00:25:03.972 INFO:journalctl@ceph.mds.j.smithi118.stdout:Mar 24 00:25:03 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-j[97290]: 2024-03-24T00:25:03.972 INFO:journalctl@ceph.mds.j.smithi118.stdout:Mar 24 00:25:03 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-j[97290]: 0> 2024-03-24T00:25:03.586+0000 7fcee6a00700 -1 *** Caught signal (Aborted) ** 2024-03-24T00:25:03.972 INFO:journalctl@ceph.mds.j.smithi118.stdout:Mar 24 00:25:03 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-j[97290]: in thread 7fcee6a00700 thread_name:ms_dispatch 2024-03-24T00:25:03.973 INFO:journalctl@ceph.mds.j.smithi118.stdout:Mar 24 00:25:03 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-j[97290]: 2024-03-24T00:25:03.973 INFO:journalctl@ceph.mds.j.smithi118.stdout:Mar 24 00:25:03 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-j[97290]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-24T00:25:03.973 INFO:journalctl@ceph.mds.j.smithi118.stdout:Mar 24 00:25:03 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-j[97290]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7fcef1040d20] 2024-03-24T00:25:03.973 INFO:journalctl@ceph.mds.j.smithi118.stdout:Mar 24 00:25:03 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-j[97290]: 2: gsignal() 2024-03-24T00:25:03.973 INFO:journalctl@ceph.mds.j.smithi118.stdout:Mar 24 00:25:03 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-j[97290]: 3: abort() 2024-03-24T00:25:03.973 INFO:journalctl@ceph.mds.j.smithi118.stdout:Mar 24 00:25:03 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-j[97290]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x18f) [0x7fcef22b94fb] 2024-03-24T00:25:03.973 INFO:journalctl@ceph.mds.j.smithi118.stdout:Mar 24 00:25:03 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-j[97290]: 5: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7fcef22b9667] 2024-03-24T00:25:03.973 INFO:journalctl@ceph.mds.j.smithi118.stdout:Mar 24 00:25:03 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-j[97290]: 6: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x559597dad223] 2024-03-24T00:25:03.973 INFO:journalctl@ceph.mds.j.smithi118.stdout:Mar 24 00:25:03 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-j[97290]: 7: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x559597c275d6] 2024-03-24T00:25:03.973 INFO:journalctl@ceph.mds.d.smithi118.stdout:Mar 24 00:25:03 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-d[97374]: 8: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x5628e312ada1] 2024-03-24T00:25:03.973 INFO:journalctl@ceph.mds.d.smithi118.stdout:Mar 24 00:25:03 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-d[97374]: 9: (Server::handle_client_rename(boost::intrusive_ptr const&)+0x259d) [0x5628e316797d] 2024-03-24T00:25:03.973 INFO:journalctl@ceph.mds.d.smithi118.stdout:Mar 24 00:25:03 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-d[97374]: 10: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x5628e317d233] 2024-03-24T00:25:03.974 INFO:journalctl@ceph.mds.d.smithi118.stdout:Mar 24 00:25:03 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-d[97374]: 11: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x5628e317f197] 2024-03-24T00:25:03.974 INFO:journalctl@ceph.mds.d.smithi118.stdout:Mar 24 00:25:03 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-d[97374]: 12: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x5628e307e11b] 2024-03-24T00:25:03.974 INFO:journalctl@ceph.mds.d.smithi118.stdout:Mar 24 00:25:03 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-d[97374]: 13: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x5628e307e6bc] 2024-03-24T00:25:03.974 INFO:journalctl@ceph.mds.d.smithi118.stdout:Mar 24 00:25:03 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-d[97374]: 14: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x5628e30681ff] 2024-03-24T00:25:03.974 INFO:journalctl@ceph.mds.d.smithi118.stdout:Mar 24 00:25:03 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-d[97374]: 15: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7ff34f1fb488] 2024-03-24T00:25:03.974 INFO:journalctl@ceph.mds.d.smithi118.stdout:Mar 24 00:25:03 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-d[97374]: 16: (DispatchQueue::entry()+0x51f) [0x7ff34f1f860f] 2024-03-24T00:25:03.974 INFO:journalctl@ceph.mds.d.smithi118.stdout:Mar 24 00:25:03 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-d[97374]: 17: (DispatchQueue::DispatchThread::entry()+0x11) [0x7ff34f2cbe81] 2024-03-24T00:25:03.974 INFO:journalctl@ceph.mds.d.smithi118.stdout:Mar 24 00:25:03 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-d[97374]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7ff34dce61ca] 2024-03-24T00:25:03.974 INFO:journalctl@ceph.mds.d.smithi118.stdout:Mar 24 00:25:03 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-d[97374]: 19: clone() 2024-03-24T00:25:03.974 INFO:journalctl@ceph.mds.d.smithi118.stdout:Mar 24 00:25:03 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-d[97374]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-03-24T00:25:03.975 INFO:journalctl@ceph.mds.d.smithi118.stdout:Mar 24 00:25:03 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-d[97374]: 2024-03-24T00:25:03.975 INFO:journalctl@ceph.mds.d.smithi118.stdout:Mar 24 00:25:03 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-d[97374]: -5> 2024-03-24T00:25:03.637+0000 7ff343600700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: In function 'void Locker::set_xlocks_done(MutationImpl*, bool)' thread 7ff343600700 time 2024-03-24T00:25:03.636500+0000 2024-03-24T00:25:03.975 INFO:journalctl@ceph.mds.d.smithi118.stdout:Mar 24 00:25:03 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-d[97374]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: 626: FAILED ceph_assert(obj->is_auth()) 2024-03-24T00:25:03.975 INFO:journalctl@ceph.mds.d.smithi118.stdout:Mar 24 00:25:03 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-d[97374]: 2024-03-24T00:25:03.975 INFO:journalctl@ceph.mds.d.smithi118.stdout:Mar 24 00:25:03 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-d[97374]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-24T00:25:03.975 INFO:journalctl@ceph.mds.d.smithi118.stdout:Mar 24 00:25:03 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-d[97374]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x135) [0x7ff34ef694a1] 2024-03-24T00:25:03.975 INFO:journalctl@ceph.mds.d.smithi118.stdout:Mar 24 00:25:03 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-d[97374]: 2: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7ff34ef69667] 2024-03-24T00:25:03.975 INFO:journalctl@ceph.mds.d.smithi118.stdout:Mar 24 00:25:03 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-d[97374]: 3: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x5628e32b0223] 2024-03-24T00:25:03.975 INFO:journalctl@ceph.mds.d.smithi118.stdout:Mar 24 00:25:03 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-d[97374]: 4: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x5628e312a5d6] 2024-03-24T00:25:03.975 INFO:journalctl@ceph.mds.j.smithi118.stdout:Mar 24 00:25:03 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-j[97290]: 8: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x559597c27da1] 2024-03-24T00:25:03.975 INFO:journalctl@ceph.mds.j.smithi118.stdout:Mar 24 00:25:03 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-j[97290]: 9: (Server::handle_client_rename(boost::intrusive_ptr const&)+0x259d) [0x559597c6497d] 2024-03-24T00:25:03.976 INFO:journalctl@ceph.mds.j.smithi118.stdout:Mar 24 00:25:03 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-j[97290]: 10: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x559597c7a233] 2024-03-24T00:25:03.976 INFO:journalctl@ceph.mds.j.smithi118.stdout:Mar 24 00:25:03 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-j[97290]: 11: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x559597c7c197] 2024-03-24T00:25:03.976 INFO:journalctl@ceph.mds.j.smithi118.stdout:Mar 24 00:25:03 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-j[97290]: 12: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x559597b7b11b] 2024-03-24T00:25:03.976 INFO:journalctl@ceph.mds.j.smithi118.stdout:Mar 24 00:25:03 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-j[97290]: 13: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x559597b7b6bc] 2024-03-24T00:25:03.976 INFO:journalctl@ceph.mds.j.smithi118.stdout:Mar 24 00:25:03 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-j[97290]: 14: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x559597b651ff] 2024-03-24T00:25:03.976 INFO:journalctl@ceph.mds.j.smithi118.stdout:Mar 24 00:25:03 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-j[97290]: 15: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7fcef254b488] 2024-03-24T00:25:03.976 INFO:journalctl@ceph.mds.j.smithi118.stdout:Mar 24 00:25:03 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-j[97290]: 16: (DispatchQueue::entry()+0x51f) [0x7fcef254860f] 2024-03-24T00:25:03.976 INFO:journalctl@ceph.mds.j.smithi118.stdout:Mar 24 00:25:03 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-j[97290]: 17: (DispatchQueue::DispatchThread::entry()+0x11) [0x7fcef261be81] 2024-03-24T00:25:03.976 INFO:journalctl@ceph.mds.j.smithi118.stdout:Mar 24 00:25:03 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-j[97290]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7fcef10361ca] 2024-03-24T00:25:03.976 INFO:journalctl@ceph.mds.j.smithi118.stdout:Mar 24 00:25:03 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-j[97290]: 19: clone() 2024-03-24T00:25:03.976 INFO:journalctl@ceph.mds.j.smithi118.stdout:Mar 24 00:25:03 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-j[97290]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-03-24T00:25:03.976 INFO:journalctl@ceph.mds.j.smithi118.stdout:Mar 24 00:25:03 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-j[97290]: 2024-03-24T00:25:03.976 INFO:journalctl@ceph.mds.j.smithi118.stdout:Mar 24 00:25:03 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-j[97290]: -1> 2024-03-24T00:25:03.582+0000 7fcee6a00700 -1 /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: In function 'void Locker::set_xlocks_done(MutationImpl*, bool)' thread 7fcee6a00700 time 2024-03-24T00:25:03.579717+0000 2024-03-24T00:25:03.976 INFO:journalctl@ceph.mds.j.smithi118.stdout:Mar 24 00:25:03 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-j[97290]: /home/jenkins-build/build/workspace/ceph-dev-new-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos8/DIST/centos8/MACHINE_SIZE/gigantic/release/19.0.0-2145-g3be9a5c9/rpm/el8/BUILD/ceph-19.0.0-2145-g3be9a5c9/src/mds/Locker.cc: 626: FAILED ceph_assert(obj->is_auth()) 2024-03-24T00:25:03.977 INFO:journalctl@ceph.mds.j.smithi118.stdout:Mar 24 00:25:03 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-j[97290]: 2024-03-24T00:25:03.977 INFO:journalctl@ceph.mds.j.smithi118.stdout:Mar 24 00:25:03 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-j[97290]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-24T00:25:03.977 INFO:journalctl@ceph.mds.j.smithi118.stdout:Mar 24 00:25:03 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-j[97290]: 1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x135) [0x7fcef22b94a1] 2024-03-24T00:25:03.977 INFO:journalctl@ceph.mds.j.smithi118.stdout:Mar 24 00:25:03 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-j[97290]: 2: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7fcef22b9667] 2024-03-24T00:25:03.977 INFO:journalctl@ceph.mds.j.smithi118.stdout:Mar 24 00:25:03 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-j[97290]: 3: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x559597dad223] 2024-03-24T00:25:03.977 INFO:journalctl@ceph.mds.j.smithi118.stdout:Mar 24 00:25:03 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-j[97290]: 4: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x559597c275d6] 2024-03-24T00:25:03.977 INFO:journalctl@ceph.mds.d.smithi118.stdout:Mar 24 00:25:03 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-d[97374]: 5: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x5628e312ada1] 2024-03-24T00:25:03.977 INFO:journalctl@ceph.mds.d.smithi118.stdout:Mar 24 00:25:03 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-d[97374]: 6: (Server::handle_client_rename(boost::intrusive_ptr const&)+0x259d) [0x5628e316797d] 2024-03-24T00:25:03.977 INFO:journalctl@ceph.mds.d.smithi118.stdout:Mar 24 00:25:03 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-d[97374]: 7: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x5628e317d233] 2024-03-24T00:25:03.977 INFO:journalctl@ceph.mds.d.smithi118.stdout:Mar 24 00:25:03 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-d[97374]: 8: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x5628e317f197] 2024-03-24T00:25:03.978 INFO:journalctl@ceph.mds.d.smithi118.stdout:Mar 24 00:25:03 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-d[97374]: 9: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x5628e307e11b] 2024-03-24T00:25:03.978 INFO:journalctl@ceph.mds.d.smithi118.stdout:Mar 24 00:25:03 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-d[97374]: 10: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x5628e307e6bc] 2024-03-24T00:25:03.978 INFO:journalctl@ceph.mds.d.smithi118.stdout:Mar 24 00:25:03 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-d[97374]: 11: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x5628e30681ff] 2024-03-24T00:25:03.978 INFO:journalctl@ceph.mds.d.smithi118.stdout:Mar 24 00:25:03 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-d[97374]: 12: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7ff34f1fb488] 2024-03-24T00:25:03.978 INFO:journalctl@ceph.mds.d.smithi118.stdout:Mar 24 00:25:03 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-d[97374]: 13: (DispatchQueue::entry()+0x51f) [0x7ff34f1f860f] 2024-03-24T00:25:03.978 INFO:journalctl@ceph.mds.d.smithi118.stdout:Mar 24 00:25:03 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-d[97374]: 14: (DispatchQueue::DispatchThread::entry()+0x11) [0x7ff34f2cbe81] 2024-03-24T00:25:03.978 INFO:journalctl@ceph.mds.d.smithi118.stdout:Mar 24 00:25:03 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-d[97374]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7ff34dce61ca] 2024-03-24T00:25:03.978 INFO:journalctl@ceph.mds.d.smithi118.stdout:Mar 24 00:25:03 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-d[97374]: 16: clone() 2024-03-24T00:25:03.978 INFO:journalctl@ceph.mds.d.smithi118.stdout:Mar 24 00:25:03 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-d[97374]: 2024-03-24T00:25:03.978 INFO:journalctl@ceph.mds.d.smithi118.stdout:Mar 24 00:25:03 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-d[97374]: -4> 2024-03-24T00:25:03.640+0000 7ff343600700 -1 *** Caught signal (Aborted) ** 2024-03-24T00:25:03.978 INFO:journalctl@ceph.mds.d.smithi118.stdout:Mar 24 00:25:03 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-d[97374]: in thread 7ff343600700 thread_name:ms_dispatch 2024-03-24T00:25:03.978 INFO:journalctl@ceph.mds.d.smithi118.stdout:Mar 24 00:25:03 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-d[97374]: 2024-03-24T00:25:03.978 INFO:journalctl@ceph.mds.d.smithi118.stdout:Mar 24 00:25:03 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-d[97374]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-24T00:25:03.978 INFO:journalctl@ceph.mds.d.smithi118.stdout:Mar 24 00:25:03 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-d[97374]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7ff34dcf0d20] 2024-03-24T00:25:03.979 INFO:journalctl@ceph.mds.d.smithi118.stdout:Mar 24 00:25:03 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-d[97374]: 2: gsignal() 2024-03-24T00:25:03.979 INFO:journalctl@ceph.mds.d.smithi118.stdout:Mar 24 00:25:03 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-d[97374]: 3: abort() 2024-03-24T00:25:03.979 INFO:journalctl@ceph.mds.d.smithi118.stdout:Mar 24 00:25:03 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-d[97374]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x18f) [0x7ff34ef694fb] 2024-03-24T00:25:03.979 INFO:journalctl@ceph.mds.d.smithi118.stdout:Mar 24 00:25:03 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-d[97374]: 5: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7ff34ef69667] 2024-03-24T00:25:03.979 INFO:journalctl@ceph.mds.d.smithi118.stdout:Mar 24 00:25:03 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-d[97374]: 6: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x5628e32b0223] 2024-03-24T00:25:03.979 INFO:journalctl@ceph.mds.d.smithi118.stdout:Mar 24 00:25:03 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-d[97374]: 7: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x5628e312a5d6] 2024-03-24T00:25:03.979 INFO:journalctl@ceph.mds.d.smithi118.stdout:Mar 24 00:25:03 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-d[97374]: 8: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x5628e312ada1] 2024-03-24T00:25:03.979 INFO:journalctl@ceph.mds.d.smithi118.stdout:Mar 24 00:25:03 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-d[97374]: 9: (Server::handle_client_rename(boost::intrusive_ptr const&)+0x259d) [0x5628e316797d] 2024-03-24T00:25:03.979 INFO:journalctl@ceph.mds.d.smithi118.stdout:Mar 24 00:25:03 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-d[97374]: 10: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x5628e317d233] 2024-03-24T00:25:03.979 INFO:journalctl@ceph.mds.j.smithi118.stdout:Mar 24 00:25:03 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-j[97290]: 5: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x559597c27da1] 2024-03-24T00:25:03.980 INFO:journalctl@ceph.mds.j.smithi118.stdout:Mar 24 00:25:03 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-j[97290]: 6: (Server::handle_client_rename(boost::intrusive_ptr const&)+0x259d) [0x559597c6497d] 2024-03-24T00:25:03.980 INFO:journalctl@ceph.mds.j.smithi118.stdout:Mar 24 00:25:03 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-j[97290]: 7: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x559597c7a233] 2024-03-24T00:25:03.980 INFO:journalctl@ceph.mds.j.smithi118.stdout:Mar 24 00:25:03 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-j[97290]: 8: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x559597c7c197] 2024-03-24T00:25:03.980 INFO:journalctl@ceph.mds.j.smithi118.stdout:Mar 24 00:25:03 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-j[97290]: 9: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x559597b7b11b] 2024-03-24T00:25:03.980 INFO:journalctl@ceph.mds.j.smithi118.stdout:Mar 24 00:25:03 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-j[97290]: 10: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x559597b7b6bc] 2024-03-24T00:25:03.980 INFO:journalctl@ceph.mds.j.smithi118.stdout:Mar 24 00:25:03 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-j[97290]: 11: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x559597b651ff] 2024-03-24T00:25:03.980 INFO:journalctl@ceph.mds.j.smithi118.stdout:Mar 24 00:25:03 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-j[97290]: 12: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7fcef254b488] 2024-03-24T00:25:03.980 INFO:journalctl@ceph.mds.j.smithi118.stdout:Mar 24 00:25:03 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-j[97290]: 13: (DispatchQueue::entry()+0x51f) [0x7fcef254860f] 2024-03-24T00:25:03.980 INFO:journalctl@ceph.mds.j.smithi118.stdout:Mar 24 00:25:03 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-j[97290]: 14: (DispatchQueue::DispatchThread::entry()+0x11) [0x7fcef261be81] 2024-03-24T00:25:03.980 INFO:journalctl@ceph.mds.j.smithi118.stdout:Mar 24 00:25:03 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-j[97290]: 15: /lib64/libpthread.so.0(+0x81ca) [0x7fcef10361ca] 2024-03-24T00:25:03.980 INFO:journalctl@ceph.mds.j.smithi118.stdout:Mar 24 00:25:03 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-j[97290]: 16: clone() 2024-03-24T00:25:03.980 INFO:journalctl@ceph.mds.j.smithi118.stdout:Mar 24 00:25:03 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-j[97290]: 2024-03-24T00:25:03.980 INFO:journalctl@ceph.mds.j.smithi118.stdout:Mar 24 00:25:03 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-j[97290]: 0> 2024-03-24T00:25:03.586+0000 7fcee6a00700 -1 *** Caught signal (Aborted) ** 2024-03-24T00:25:03.980 INFO:journalctl@ceph.mds.j.smithi118.stdout:Mar 24 00:25:03 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-j[97290]: in thread 7fcee6a00700 thread_name:ms_dispatch 2024-03-24T00:25:03.981 INFO:journalctl@ceph.mds.j.smithi118.stdout:Mar 24 00:25:03 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-j[97290]: 2024-03-24T00:25:03.981 INFO:journalctl@ceph.mds.j.smithi118.stdout:Mar 24 00:25:03 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-j[97290]: ceph version 19.0.0-2145-g3be9a5c9 (3be9a5c9bc793e11e2800a8c0c696e8b46742033) squid (dev) 2024-03-24T00:25:03.981 INFO:journalctl@ceph.mds.j.smithi118.stdout:Mar 24 00:25:03 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-j[97290]: 1: /lib64/libpthread.so.0(+0x12d20) [0x7fcef1040d20] 2024-03-24T00:25:03.981 INFO:journalctl@ceph.mds.j.smithi118.stdout:Mar 24 00:25:03 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-j[97290]: 2: gsignal() 2024-03-24T00:25:03.981 INFO:journalctl@ceph.mds.j.smithi118.stdout:Mar 24 00:25:03 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-j[97290]: 3: abort() 2024-03-24T00:25:03.981 INFO:journalctl@ceph.mds.j.smithi118.stdout:Mar 24 00:25:03 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-j[97290]: 4: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x18f) [0x7fcef22b94fb] 2024-03-24T00:25:03.981 INFO:journalctl@ceph.mds.j.smithi118.stdout:Mar 24 00:25:03 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-j[97290]: 5: /usr/lib64/ceph/libceph-common.so.2(+0x2d2667) [0x7fcef22b9667] 2024-03-24T00:25:03.981 INFO:journalctl@ceph.mds.j.smithi118.stdout:Mar 24 00:25:03 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-j[97290]: 6: (Locker::set_xlocks_done(MutationImpl*, bool)+0x493) [0x559597dad223] 2024-03-24T00:25:03.981 INFO:journalctl@ceph.mds.j.smithi118.stdout:Mar 24 00:25:03 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-j[97290]: 7: (Server::early_reply(boost::intrusive_ptr const&, CInode*, CDentry*)+0x3e6) [0x559597c275d6] 2024-03-24T00:25:03.981 INFO:journalctl@ceph.mds.j.smithi118.stdout:Mar 24 00:25:03 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-j[97290]: 8: (Server::journal_and_reply(boost::intrusive_ptr const&, CInode*, CDentry*, LogEvent*, MDSLogContextBase*)+0xc1) [0x559597c27da1] 2024-03-24T00:25:03.981 INFO:journalctl@ceph.mds.j.smithi118.stdout:Mar 24 00:25:03 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-j[97290]: 9: (Server::handle_client_rename(boost::intrusive_ptr const&)+0x259d) [0x559597c6497d] 2024-03-24T00:25:03.981 INFO:journalctl@ceph.mds.j.smithi118.stdout:Mar 24 00:25:03 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-j[97290]: 10: (Server::handle_peer_request_reply(boost::intrusive_ptr const&)+0x733) [0x559597c7a233] 2024-03-24T00:25:03.981 INFO:journalctl@ceph.mds.d.smithi118.stdout:Mar 24 00:25:03 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-d[97374]: 11: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x5628e317f197] 2024-03-24T00:25:03.982 INFO:journalctl@ceph.mds.d.smithi118.stdout:Mar 24 00:25:03 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-d[97374]: 12: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x5628e307e11b] 2024-03-24T00:25:03.982 INFO:journalctl@ceph.mds.d.smithi118.stdout:Mar 24 00:25:03 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-d[97374]: 13: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x5628e307e6bc] 2024-03-24T00:25:03.982 INFO:journalctl@ceph.mds.d.smithi118.stdout:Mar 24 00:25:03 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-d[97374]: 14: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x5628e30681ff] 2024-03-24T00:25:03.982 INFO:journalctl@ceph.mds.d.smithi118.stdout:Mar 24 00:25:03 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-d[97374]: 15: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7ff34f1fb488] 2024-03-24T00:25:03.982 INFO:journalctl@ceph.mds.d.smithi118.stdout:Mar 24 00:25:03 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-d[97374]: 16: (DispatchQueue::entry()+0x51f) [0x7ff34f1f860f] 2024-03-24T00:25:03.982 INFO:journalctl@ceph.mds.d.smithi118.stdout:Mar 24 00:25:03 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-d[97374]: 17: (DispatchQueue::DispatchThread::entry()+0x11) [0x7ff34f2cbe81] 2024-03-24T00:25:03.982 INFO:journalctl@ceph.mds.d.smithi118.stdout:Mar 24 00:25:03 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-d[97374]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7ff34dce61ca] 2024-03-24T00:25:03.982 INFO:journalctl@ceph.mds.d.smithi118.stdout:Mar 24 00:25:03 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-d[97374]: 19: clone() 2024-03-24T00:25:03.982 INFO:journalctl@ceph.mds.d.smithi118.stdout:Mar 24 00:25:03 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-d[97374]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-03-24T00:25:03.982 INFO:journalctl@ceph.mds.d.smithi118.stdout:Mar 24 00:25:03 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-d[97374]: 2024-03-24T00:25:03.983 INFO:journalctl@ceph.mds.j.smithi118.stdout:Mar 24 00:25:03 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-j[97290]: 11: (Server::dispatch(boost::intrusive_ptr const&)+0x97) [0x559597c7c197] 2024-03-24T00:25:03.983 INFO:journalctl@ceph.mds.j.smithi118.stdout:Mar 24 00:25:03 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-j[97290]: 12: (MDSRank::_dispatch(boost::intrusive_ptr const&, bool)+0x5fb) [0x559597b7b11b] 2024-03-24T00:25:03.983 INFO:journalctl@ceph.mds.j.smithi118.stdout:Mar 24 00:25:03 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-j[97290]: 13: (MDSRankDispatcher::ms_dispatch(boost::intrusive_ptr const&)+0x5c) [0x559597b7b6bc] 2024-03-24T00:25:03.983 INFO:journalctl@ceph.mds.j.smithi118.stdout:Mar 24 00:25:03 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-j[97290]: 14: (MDSDaemon::ms_dispatch2(boost::intrusive_ptr const&)+0x1bf) [0x559597b651ff] 2024-03-24T00:25:03.983 INFO:journalctl@ceph.mds.j.smithi118.stdout:Mar 24 00:25:03 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-j[97290]: 15: (Messenger::ms_deliver_dispatch(boost::intrusive_ptr const&)+0x478) [0x7fcef254b488] 2024-03-24T00:25:03.983 INFO:journalctl@ceph.mds.j.smithi118.stdout:Mar 24 00:25:03 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-j[97290]: 16: (DispatchQueue::entry()+0x51f) [0x7fcef254860f] 2024-03-24T00:25:03.983 INFO:journalctl@ceph.mds.j.smithi118.stdout:Mar 24 00:25:03 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-j[97290]: 17: (DispatchQueue::DispatchThread::entry()+0x11) [0x7fcef261be81] 2024-03-24T00:25:03.983 INFO:journalctl@ceph.mds.j.smithi118.stdout:Mar 24 00:25:03 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-j[97290]: 18: /lib64/libpthread.so.0(+0x81ca) [0x7fcef10361ca] 2024-03-24T00:25:03.983 INFO:journalctl@ceph.mds.j.smithi118.stdout:Mar 24 00:25:03 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-j[97290]: 19: clone() 2024-03-24T00:25:03.983 INFO:journalctl@ceph.mds.j.smithi118.stdout:Mar 24 00:25:03 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-j[97290]: NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. 2024-03-24T00:25:03.983 INFO:journalctl@ceph.mds.j.smithi118.stdout:Mar 24 00:25:03 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-j[97290]: 2024-03-24T00:25:03.984 INFO:journalctl@ceph.mds.j.smithi118.stdout:Mar 24 00:25:03 smithi118 podman[97417]: 2024-03-24 00:25:03.708667348 +0000 UTC m=+0.040345983 container died 263e3b2b32d9ed7a147768b13e897003dae75c6a611458b7cb1f4a3392a16109 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-j, io.buildah.version=1.33.5, org.label-schema.build-date=20240319, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, ceph=True, GIT_CLEAN=True, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2) 2024-03-24T00:25:03.984 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:25:03 smithi193 ceph-mon[36878]: pgmap v712: 97 pgs: 97 active+clean; 147 MiB data, 881 MiB used, 1.0 TiB / 1.0 TiB avail; 31 KiB/s rd, 22 KiB/s wr, 8 op/s 2024-03-24T00:25:03.984 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:25:03 smithi193 ceph-mon[36878]: mds.? [v2:172.21.15.118:6836/3350294232,v1:172.21.15.118:6837/3350294232] up:rejoin 2024-03-24T00:25:03.984 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:25:03 smithi193 ceph-mon[36878]: mds.? [v2:172.21.15.118:6834/395010504,v1:172.21.15.118:6835/395010504] up:rejoin 2024-03-24T00:25:03.984 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:25:03 smithi193 ceph-mon[36878]: fsmap cephfs:5/5 3 up:standby-replay 2 up:rejoin 3 up:active 2024-03-24T00:25:03.984 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:25:03 smithi193 ceph-mon[36878]: daemon mds.j is now active in filesystem cephfs as rank 3 2024-03-24T00:25:03.984 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:25:03 smithi193 ceph-mon[36878]: daemon mds.d is now active in filesystem cephfs as rank 1 2024-03-24T00:25:04.446 INFO:journalctl@ceph.mds.d.smithi118.stdout:Mar 24 00:25:04 smithi118 podman[97429]: 2024-03-24 00:25:04.202725359 +0000 UTC m=+0.475387194 container died 1a8d39fc36fce7050e31d67be4b4bb268c7cea54024202f989e6aed27e64df53 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-d, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240319, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c) 2024-03-24T00:25:04.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:25:04 smithi118 ceph-mon[31535]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-03-24T00:25:04.947 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:25:04 smithi118 ceph-mon[31535]: Cluster is now healthy 2024-03-24T00:25:04.947 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:25:04 smithi118 ceph-mon[31535]: mds.? [v2:172.21.15.118:6834/395010504,v1:172.21.15.118:6835/395010504] up:active 2024-03-24T00:25:04.947 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:25:04 smithi118 ceph-mon[31535]: mds.? [v2:172.21.15.118:6836/3350294232,v1:172.21.15.118:6837/3350294232] up:active 2024-03-24T00:25:04.947 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:25:04 smithi118 ceph-mon[31535]: fsmap cephfs:5 3 up:standby-replay 5 up:active 2024-03-24T00:25:04.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:25:04 smithi189 ceph-mon[36187]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-03-24T00:25:04.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:25:04 smithi189 ceph-mon[36187]: Cluster is now healthy 2024-03-24T00:25:04.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:25:04 smithi189 ceph-mon[36187]: mds.? [v2:172.21.15.118:6834/395010504,v1:172.21.15.118:6835/395010504] up:active 2024-03-24T00:25:04.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:25:04 smithi189 ceph-mon[36187]: mds.? [v2:172.21.15.118:6836/3350294232,v1:172.21.15.118:6837/3350294232] up:active 2024-03-24T00:25:04.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:25:04 smithi189 ceph-mon[36187]: fsmap cephfs:5 3 up:standby-replay 5 up:active 2024-03-24T00:25:04.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:25:04 smithi193 ceph-mon[36878]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-03-24T00:25:04.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:25:04 smithi193 ceph-mon[36878]: Cluster is now healthy 2024-03-24T00:25:04.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:25:04 smithi193 ceph-mon[36878]: mds.? [v2:172.21.15.118:6834/395010504,v1:172.21.15.118:6835/395010504] up:active 2024-03-24T00:25:04.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:25:04 smithi193 ceph-mon[36878]: mds.? [v2:172.21.15.118:6836/3350294232,v1:172.21.15.118:6837/3350294232] up:active 2024-03-24T00:25:04.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:25:04 smithi193 ceph-mon[36878]: fsmap cephfs:5 3 up:standby-replay 5 up:active 2024-03-24T00:25:05.446 INFO:journalctl@ceph.mds.d.smithi118.stdout:Mar 24 00:25:05 smithi118 podman[97429]: 2024-03-24 00:25:05.111235701 +0000 UTC m=+1.383897566 container remove 1a8d39fc36fce7050e31d67be4b4bb268c7cea54024202f989e6aed27e64df53 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-d, org.label-schema.license=GPLv2, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240319, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS) 2024-03-24T00:25:05.446 INFO:journalctl@ceph.mds.d.smithi118.stdout:Mar 24 00:25:05 smithi118 systemd[1]: ceph-d79a5420-e971-11ee-95cd-87774f69a715@mds.d.service: Main process exited, code=exited, status=134/n/a 2024-03-24T00:25:05.446 INFO:journalctl@ceph.mds.j.smithi118.stdout:Mar 24 00:25:05 smithi118 podman[97417]: 2024-03-24 00:25:05.296621725 +0000 UTC m=+1.628300397 container remove 263e3b2b32d9ed7a147768b13e897003dae75c6a611458b7cb1f4a3392a16109 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-j, io.buildah.version=1.33.5, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, ceph=True, org.label-schema.build-date=20240319, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, GIT_CLEAN=True, org.label-schema.vendor=CentOS, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-24T00:25:05.446 INFO:journalctl@ceph.mds.j.smithi118.stdout:Mar 24 00:25:05 smithi118 systemd[1]: ceph-d79a5420-e971-11ee-95cd-87774f69a715@mds.j.service: Main process exited, code=exited, status=134/n/a 2024-03-24T00:25:05.881 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:25:05 smithi118 ceph-mon[31535]: pgmap v713: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 102 KiB/s rd, 2.9 KiB/s wr, 21 op/s 2024-03-24T00:25:05.882 INFO:journalctl@ceph.mds.d.smithi118.stdout:Mar 24 00:25:05 smithi118 systemd[1]: ceph-d79a5420-e971-11ee-95cd-87774f69a715@mds.d.service: Failed with result 'exit-code'. 2024-03-24T00:25:05.882 INFO:journalctl@ceph.mds.d.smithi118.stdout:Mar 24 00:25:05 smithi118 systemd[1]: ceph-d79a5420-e971-11ee-95cd-87774f69a715@mds.d.service: Consumed 1.632s CPU time. 2024-03-24T00:25:05.882 INFO:journalctl@ceph.mds.j.smithi118.stdout:Mar 24 00:25:05 smithi118 systemd[1]: ceph-d79a5420-e971-11ee-95cd-87774f69a715@mds.j.service: Failed with result 'exit-code'. 2024-03-24T00:25:05.882 INFO:journalctl@ceph.mds.j.smithi118.stdout:Mar 24 00:25:05 smithi118 systemd[1]: ceph-d79a5420-e971-11ee-95cd-87774f69a715@mds.j.service: Consumed 1.397s CPU time. 2024-03-24T00:25:05.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:25:05 smithi189 ceph-mon[36187]: pgmap v713: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 102 KiB/s rd, 2.9 KiB/s wr, 21 op/s 2024-03-24T00:25:05.954 INFO:journalctl@ceph.mds.b.smithi189.stdout:Mar 24 00:25:05 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-b[66975]: 2024-03-24T00:25:05.742+0000 7f4e07ab9700 -1 mds.pinger is_rank_lagging: rank=1 was never sent ping request. 2024-03-24T00:25:05.954 INFO:journalctl@ceph.mds.b.smithi189.stdout:Mar 24 00:25:05 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-b[66975]: 2024-03-24T00:25:05.742+0000 7f4e07ab9700 -1 mds.pinger is_rank_lagging: rank=3 was never sent ping request. 2024-03-24T00:25:05.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:25:05 smithi193 ceph-mon[36878]: pgmap v713: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 102 KiB/s rd, 2.9 KiB/s wr, 21 op/s 2024-03-24T00:25:07.864 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:25:07 smithi118 ceph-mon[31535]: pgmap v714: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 170 KiB/s rd, 7.1 KiB/s wr, 36 op/s 2024-03-24T00:25:07.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:25:07 smithi189 ceph-mon[36187]: pgmap v714: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 170 KiB/s rd, 7.1 KiB/s wr, 36 op/s 2024-03-24T00:25:07.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:25:07 smithi193 ceph-mon[36878]: pgmap v714: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 170 KiB/s rd, 7.1 KiB/s wr, 36 op/s 2024-03-24T00:25:09.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:25:09 smithi118 ceph-mon[31535]: pgmap v715: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 294 KiB/s rd, 7.2 KiB/s wr, 51 op/s 2024-03-24T00:25:09.953 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:25:09 smithi189 ceph-mon[36187]: pgmap v715: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 294 KiB/s rd, 7.2 KiB/s wr, 51 op/s 2024-03-24T00:25:09.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:25:09 smithi193 ceph-mon[36878]: pgmap v715: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 294 KiB/s rd, 7.2 KiB/s wr, 51 op/s 2024-03-24T00:25:11.897 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:25:11 smithi118 ceph-mon[31535]: pgmap v716: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 297 KiB/s rd, 17 KiB/s wr, 48 op/s 2024-03-24T00:25:11.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:25:11 smithi189 ceph-mon[36187]: pgmap v716: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 297 KiB/s rd, 17 KiB/s wr, 48 op/s 2024-03-24T00:25:11.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:25:11 smithi193 ceph-mon[36878]: pgmap v716: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 297 KiB/s rd, 17 KiB/s wr, 48 op/s 2024-03-24T00:25:13.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:25:13 smithi118 ceph-mon[31535]: pgmap v717: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 258 KiB/s rd, 14 KiB/s wr, 41 op/s 2024-03-24T00:25:13.953 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:25:13 smithi189 ceph-mon[36187]: pgmap v717: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 258 KiB/s rd, 14 KiB/s wr, 41 op/s 2024-03-24T00:25:13.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:25:13 smithi193 ceph-mon[36878]: pgmap v717: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 258 KiB/s rd, 14 KiB/s wr, 41 op/s 2024-03-24T00:25:15.720 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:25:15.720+0000 7f81f7e00640 1 -- 172.21.15.118:0/3864536383 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f81f8103720 msgr2=0x7f81f8103b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:25:15.721 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:25:15.720+0000 7f81f7e00640 1 --2- 172.21.15.118:0/3864536383 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f81f8103720 0x7f81f8103b40 secure :-1 s=READY pgs=468 cs=0 l=1 rev1=1 crypto rx=0x7f81e8009ca0 tx=0x7f81e802c100 comp rx=0 tx=0).stop 2024-03-24T00:25:15.721 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:25:15.720+0000 7f81f7e00640 1 -- 172.21.15.118:0/3864536383 shutdown_connections 2024-03-24T00:25:15.721 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:25:15.720+0000 7f81f7e00640 1 --2- 172.21.15.118:0/3864536383 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f81f81052d0 0x7f81f810bb60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:25:15.721 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:25:15.720+0000 7f81f7e00640 1 --2- 172.21.15.118:0/3864536383 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f81f8104910 0x7f81f8104d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:25:15.721 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:25:15.720+0000 7f81f7e00640 1 --2- 172.21.15.118:0/3864536383 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f81f8103720 0x7f81f8103b40 unknown :-1 s=CLOSED pgs=468 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:25:15.721 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:25:15.720+0000 7f81f7e00640 1 -- 172.21.15.118:0/3864536383 >> 172.21.15.118:0/3864536383 conn(0x7f81f80fecb0 msgr2=0x7f81f81010d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:25:15.721 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:25:15.721+0000 7f81f7e00640 1 -- 172.21.15.118:0/3864536383 shutdown_connections 2024-03-24T00:25:15.721 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:25:15.721+0000 7f81f7e00640 1 -- 172.21.15.118:0/3864536383 wait complete. 2024-03-24T00:25:15.723 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:25:15.721+0000 7f81f7e00640 1 Processor -- start 2024-03-24T00:25:15.723 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:25:15.722+0000 7f81f7e00640 1 -- start start 2024-03-24T00:25:15.723 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:25:15.722+0000 7f81f7e00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f81f8103720 0x7f81f8199a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:25:15.723 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:25:15.722+0000 7f81f7e00640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f81f8104910 0x7f81f8199f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:25:15.723 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:25:15.722+0000 7f81f7e00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f81f81052d0 0x7f81f81a0fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:25:15.724 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:25:15.722+0000 7f81f7e00640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f81f810f3a0 con 0x7f81f8104910 2024-03-24T00:25:15.724 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:25:15.722+0000 7f81f7e00640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f81f810f220 con 0x7f81f81052d0 2024-03-24T00:25:15.724 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:25:15.722+0000 7f81f7e00640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f81f810f520 con 0x7f81f8103720 2024-03-24T00:25:15.724 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:25:15.722+0000 7f81f6a00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f81f8103720 0x7f81f8199a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:25:15.724 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:25:15.722+0000 7f81f6a00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f81f8103720 0x7f81f8199a10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.118:35306/0 (socket says 172.21.15.118:35306) 2024-03-24T00:25:15.724 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:25:15.722+0000 7f81f6000640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f81f8104910 0x7f81f8199f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:25:15.724 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:25:15.722+0000 7f81f6a00640 1 -- 172.21.15.118:0/1693767697 learned_addr learned my addr 172.21.15.118:0/1693767697 (peer_addr_for_me v2:172.21.15.118:0/0) 2024-03-24T00:25:15.724 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:25:15.722+0000 7f81f7400640 1 --2- 172.21.15.118:0/1693767697 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f81f81052d0 0x7f81f81a0fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:25:15.724 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:25:15.723+0000 7f81f7400640 1 -- 172.21.15.118:0/1693767697 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f81f8103720 msgr2=0x7f81f8199a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:25:15.724 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:25:15.723+0000 7f81f7400640 1 --2- 172.21.15.118:0/1693767697 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f81f8103720 0x7f81f8199a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:25:15.724 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:25:15.723+0000 7f81f7400640 1 -- 172.21.15.118:0/1693767697 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f81f8104910 msgr2=0x7f81f8199f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:25:15.724 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:25:15.723+0000 7f81f7400640 1 --2- 172.21.15.118:0/1693767697 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f81f8104910 0x7f81f8199f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:25:15.724 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:25:15.723+0000 7f81f7400640 1 -- 172.21.15.118:0/1693767697 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f81f81a1630 con 0x7f81f81052d0 2024-03-24T00:25:15.724 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:25:15.723+0000 7f81f6000640 1 --2- 172.21.15.118:0/1693767697 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f81f8104910 0x7f81f8199f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T00:25:15.724 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:25:15.723+0000 7f81f6a00640 1 --2- 172.21.15.118:0/1693767697 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f81f8103720 0x7f81f8199a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T00:25:15.725 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:25:15.723+0000 7f81f7400640 1 --2- 172.21.15.118:0/1693767697 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f81f81052d0 0x7f81f81a0fc0 secure :-1 s=READY pgs=469 cs=0 l=1 rev1=1 crypto rx=0x7f81f000e970 tx=0x7f81f000ee30 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:25:15.725 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:25:15.723+0000 7f81ef400640 1 -- 172.21.15.118:0/1693767697 <== mon.2 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f81f000ccc0 con 0x7f81f81052d0 2024-03-24T00:25:15.725 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:25:15.723+0000 7f81ef400640 1 -- 172.21.15.118:0/1693767697 <== mon.2 v2:172.21.15.189:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f81f0005020 con 0x7f81f81052d0 2024-03-24T00:25:15.725 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:25:15.723+0000 7f81f7e00640 1 -- 172.21.15.118:0/1693767697 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f81f81a1920 con 0x7f81f81052d0 2024-03-24T00:25:15.725 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:25:15.723+0000 7f81ef400640 1 -- 172.21.15.118:0/1693767697 <== mon.2 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f81f0004a00 con 0x7f81f81052d0 2024-03-24T00:25:15.725 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:25:15.723+0000 7f81f7e00640 1 -- 172.21.15.118:0/1693767697 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f81f81a1e70 con 0x7f81f81052d0 2024-03-24T00:25:15.726 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:25:15.726+0000 7f81ef400640 1 -- 172.21.15.118:0/1693767697 <== mon.2 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 20) v1 ==== 137291+0+0 (secure 0 0 0) 0x7f81f00051d0 con 0x7f81f81052d0 2024-03-24T00:25:15.726 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:25:15.727+0000 7f82007d1740 1 Processor -- start 2024-03-24T00:25:15.726 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:25:15.727+0000 7f82007d1740 1 -- start start 2024-03-24T00:25:15.728 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:25:15.727+0000 7f82007d1740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5622651a13f0 0x5622651a1810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:25:15.728 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:25:15.727+0000 7f82007d1740 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5622651a1de0 0x5622651a68e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:25:15.728 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:25:15.727+0000 7f82007d1740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5622651a6f50 0x5622651a93e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:25:15.728 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:25:15.727+0000 7f82007d1740 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x5622651643b0 con 0x5622651a1de0 2024-03-24T00:25:15.728 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:25:15.727+0000 7f82007d1740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x562265106830 con 0x5622651a13f0 2024-03-24T00:25:15.728 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:25:15.727+0000 7f82007d1740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x5622650af5d0 con 0x5622651a6f50 2024-03-24T00:25:15.729 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:25:15.727+0000 7f81f6000640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5622651a13f0 0x5622651a1810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:25:15.729 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:25:15.727+0000 7f81f6a00640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5622651a1de0 0x5622651a68e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:25:15.729 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:25:15.727+0000 7f81f6000640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5622651a13f0 0x5622651a1810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.118:58514/0 (socket says 172.21.15.118:58514) 2024-03-24T00:25:15.729 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:25:15.727+0000 7f81f6000640 1 -- 172.21.15.118:0/4109920994 learned_addr learned my addr 172.21.15.118:0/4109920994 (peer_addr_for_me v2:172.21.15.118:0/0) 2024-03-24T00:25:15.729 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:25:15.727+0000 7f81f6000640 1 --2- 172.21.15.118:0/4109920994 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5622651a6f50 0x5622651a93e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:25:15.729 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:25:15.728+0000 7f81f6a00640 1 -- 172.21.15.118:0/4109920994 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5622651a6f50 msgr2=0x5622651a93e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:25:15.729 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:25:15.728+0000 7f81f6a00640 1 --2- 172.21.15.118:0/4109920994 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5622651a6f50 0x5622651a93e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:25:15.729 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:25:15.728+0000 7f81f6a00640 1 -- 172.21.15.118:0/4109920994 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5622651a13f0 msgr2=0x5622651a1810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:25:15.729 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:25:15.728+0000 7f81f6a00640 1 --2- 172.21.15.118:0/4109920994 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5622651a13f0 0x5622651a1810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:25:15.729 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:25:15.728+0000 7f81f6a00640 1 -- 172.21.15.118:0/4109920994 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x5622650c72c0 con 0x5622651a1de0 2024-03-24T00:25:15.729 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:25:15.728+0000 7f81f6000640 1 --2- 172.21.15.118:0/4109920994 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5622651a6f50 0x5622651a93e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T00:25:15.729 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:25:15.728+0000 7f81ef400640 1 --2- 172.21.15.118:0/1693767697 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7f81dc0a27f0 0x7f81dc0a4cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:25:15.729 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:25:15.728+0000 7f81f6a00640 1 --2- 172.21.15.118:0/4109920994 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5622651a1de0 0x5622651a68e0 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7f81e8000f80 tx=0x7f81e80323e0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9e5ea680369250a3 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:25:15.729 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:25:15.728+0000 7f81ef400640 1 -- 172.21.15.118:0/1693767697 <== mon.2 v2:172.21.15.189:3300/0 5 ==== osd_map(124..124 src has 1..124) v4 ==== 11201+0+0 (secure 0 0 0) 0x7f81f00cf5a0 con 0x7f81f81052d0 2024-03-24T00:25:15.730 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:25:15.728+0000 7f81f6000640 1 --2- 172.21.15.118:0/1693767697 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7f81dc0a27f0 0x7f81dc0a4cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:25:15.730 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:25:15.728+0000 7f81e7e00640 1 -- 172.21.15.118:0/4109920994 <== mon.0 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f81e8043070 con 0x5622651a1de0 2024-03-24T00:25:15.730 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:25:15.729+0000 7f81e7e00640 1 -- 172.21.15.118:0/4109920994 <== mon.0 v2:172.21.15.118:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f81e8003970 con 0x5622651a1de0 2024-03-24T00:25:15.730 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:25:15.729+0000 7f81e7e00640 1 -- 172.21.15.118:0/4109920994 <== mon.0 v2:172.21.15.118:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f81e8039040 con 0x5622651a1de0 2024-03-24T00:25:15.734 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:25:15.729+0000 7f82007d1740 1 -- 172.21.15.118:0/4109920994 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5622651a1de0 msgr2=0x5622651a68e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:25:15.734 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:25:15.729+0000 7f81f6000640 1 --2- 172.21.15.118:0/1693767697 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7f81dc0a27f0 0x7f81dc0a4cb0 secure :-1 s=READY pgs=401 cs=0 l=1 rev1=1 crypto rx=0x5622651a2640 tx=0x7f81e0015040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:25:15.734 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:25:15.729+0000 7f82007d1740 1 --2- 172.21.15.118:0/4109920994 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5622651a1de0 0x5622651a68e0 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7f81e8000f80 tx=0x7f81e80323e0 comp rx=0 tx=0).stop 2024-03-24T00:25:15.734 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:25:15.729+0000 7f82007d1740 1 -- 172.21.15.118:0/4109920994 shutdown_connections 2024-03-24T00:25:15.734 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:25:15.729+0000 7f82007d1740 1 --2- 172.21.15.118:0/4109920994 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5622651a6f50 0x5622651a93e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:25:15.734 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:25:15.729+0000 7f82007d1740 1 --2- 172.21.15.118:0/4109920994 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5622651a1de0 0x5622651a68e0 unknown :-1 s=CLOSED pgs=235 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:25:15.734 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:25:15.729+0000 7f82007d1740 1 --2- 172.21.15.118:0/4109920994 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5622651a13f0 0x5622651a1810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:25:15.734 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:25:15.729+0000 7f82007d1740 1 -- 172.21.15.118:0/4109920994 >> 172.21.15.118:0/4109920994 conn(0x56226519c960 msgr2=0x56226519cd60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:25:15.734 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:25:15.729+0000 7f82007d1740 1 -- 172.21.15.118:0/4109920994 shutdown_connections 2024-03-24T00:25:15.734 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:25:15.729+0000 7f82007d1740 1 -- 172.21.15.118:0/4109920994 wait complete. 2024-03-24T00:25:15.734 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:25:15.730+0000 7f82007d1740 1 -- start start 2024-03-24T00:25:15.734 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:25:15.730+0000 7f82007d1740 1 Processor -- start 2024-03-24T00:25:15.735 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:25:15.730+0000 7f82007d1740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5622651a13f0 0x5622651ab4f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:25:15.735 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:25:15.731+0000 7f82007d1740 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5622651a1de0 0x5622651aba30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:25:15.735 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:25:15.731+0000 7f82007d1740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5622651a6f50 0x5622651b2a70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:25:15.735 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:25:15.731+0000 7f82007d1740 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x5622650c2a20 con 0x5622651a1de0 2024-03-24T00:25:15.735 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:25:15.731+0000 7f82007d1740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x5622651028d0 con 0x5622651a13f0 2024-03-24T00:25:15.735 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:25:15.731+0000 7f82007d1740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x5622651a9ea0 con 0x5622651a6f50 2024-03-24T00:25:15.735 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:25:15.731+0000 7f82007d1740 10 client.0 _ll_register_callbacks cb 0x5622650ff1c0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T00:25:15.735 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:25:15.731+0000 7f81f6a00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5622651a13f0 0x5622651ab4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:25:15.735 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:25:15.731+0000 7f81f7400640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5622651a6f50 0x5622651b2a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:25:15.735 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:25:15.731+0000 7f81f6a00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5622651a13f0 0x5622651ab4f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.118:58518/0 (socket says 172.21.15.118:58518) 2024-03-24T00:25:15.735 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:25:15.731+0000 7f81f6a00640 1 -- 172.21.15.118:0/1145513718 learned_addr learned my addr 172.21.15.118:0/1145513718 (peer_addr_for_me v2:172.21.15.118:0/0) 2024-03-24T00:25:15.735 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:25:15.731+0000 7f81f6000640 1 --2- 172.21.15.118:0/1145513718 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5622651a1de0 0x5622651aba30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:25:15.735 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:25:15.731+0000 7f81f6a00640 1 -- 172.21.15.118:0/1145513718 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5622651a6f50 msgr2=0x5622651b2a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:25:15.736 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:25:15.731+0000 7f81f6a00640 1 --2- 172.21.15.118:0/1145513718 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5622651a6f50 0x5622651b2a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:25:15.736 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:25:15.731+0000 7f81f6a00640 1 -- 172.21.15.118:0/1145513718 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5622651a1de0 msgr2=0x5622651aba30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:25:15.736 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:25:15.731+0000 7f81f6a00640 1 --2- 172.21.15.118:0/1145513718 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5622651a1de0 0x5622651aba30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:25:15.736 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:25:15.731+0000 7f81f6a00640 1 -- 172.21.15.118:0/1145513718 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x562264e1b210 con 0x5622651a13f0 2024-03-24T00:25:15.736 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:25:15.731+0000 7f81f7400640 1 --2- 172.21.15.118:0/1145513718 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5622651a6f50 0x5622651b2a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T00:25:15.736 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:25:15.732+0000 7f81f6a00640 1 --2- 172.21.15.118:0/1145513718 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5622651a13f0 0x5622651ab4f0 secure :-1 s=READY pgs=470 cs=0 l=1 rev1=1 crypto rx=0x7f81e802c710 tx=0x7f81e8032a20 comp rx=0 tx=0).ready entity=mon.2 client_cookie=4d83aeaaca73548a server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:25:15.736 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:25:15.732+0000 7f81e4c00640 10 client.0 ms_handle_connect on v2:172.21.15.189:3300/0 2024-03-24T00:25:15.736 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:25:15.732+0000 7f81e4c00640 1 -- 172.21.15.118:0/1145513718 <== mon.2 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f81e804c030 con 0x5622651a13f0 2024-03-24T00:25:15.736 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:25:15.732+0000 7f82007d1740 1 -- 172.21.15.118:0/1145513718 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f81e8003970 con 0x5622651a13f0 2024-03-24T00:25:15.736 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:25:15.732+0000 7f81e4c00640 1 -- 172.21.15.118:0/1145513718 <== mon.2 v2:172.21.15.189:3300/0 2 ==== osd_map(124..124 src has 1..124) v4 ==== 11201+0+0 (secure 0 0 0) 0x7f81e8034960 con 0x5622651a13f0 2024-03-24T00:25:15.736 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:25:15.732+0000 7f81e4c00640 1 -- 172.21.15.118:0/1145513718 <== mon.2 v2:172.21.15.189:3300/0 3 ==== mon_get_version_reply(handle=1 version=186) v2 ==== 24+0+0 (secure 0 0 0) 0x7f81e8059050 con 0x5622651a13f0 2024-03-24T00:25:15.736 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:25:15.732+0000 7f82007d1740 10 client.27077 fetch_fsmap learned FSMap version 186 2024-03-24T00:25:15.736 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:25:15.732+0000 7f82007d1740 1 -- 172.21.15.118:0/1145513718 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,fsmap=186,monmap=4+}) v3 -- 0x562264ee7620 con 0x5622651a13f0 2024-03-24T00:25:15.736 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:25:15.733+0000 7f81e4c00640 1 -- 172.21.15.118:0/1145513718 <== mon.2 v2:172.21.15.189:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f81e8003970 con 0x5622651a13f0 2024-03-24T00:25:15.736 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:25:15.733+0000 7f81e4c00640 1 -- 172.21.15.118:0/1145513718 <== mon.2 v2:172.21.15.189:3300/0 5 ==== fsmap(e 186) v1 ==== 5658+0+0 (secure 0 0 0) 0x7f81e8054080 con 0x5622651a13f0 2024-03-24T00:25:15.737 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:25:15.733+0000 7f82007d1740 10 client.27077 fetch_fsmap finished waiting for FSMap version 186 2024-03-24T00:25:15.737 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:25:15.733+0000 7f82007d1740 10 client.27077 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-24T00:25:15.737 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:25:15.733+0000 7f82007d1740 20 client.27077 populate_metadata read hostname 'smithi118' 2024-03-24T00:25:15.737 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:25:15.733+0000 7f82007d1740 1 --2- 172.21.15.118:0/1145513718 >> [v2:172.21.15.189:6838/4074484168,v1:172.21.15.189:6839/4074484168] conn(0x5622651b71e0 0x5622651b9620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:25:15.737 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:25:15.733+0000 7f82007d1740 4 client.27077 mds_command: new command op to 24569 tid=0 multi_id=0 [{"prefix": "get_command_descriptions"}] 2024-03-24T00:25:15.737 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:25:15.733+0000 7f82007d1740 1 -- 172.21.15.118:0/1145513718 --> [v2:172.21.15.189:6838/4074484168,v1:172.21.15.189:6839/4074484168] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x562265113d70 con 0x5622651b71e0 2024-03-24T00:25:15.737 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:25:15.733+0000 7f81f6000640 1 --2- 172.21.15.118:0/1145513718 >> [v2:172.21.15.189:6838/4074484168,v1:172.21.15.189:6839/4074484168] conn(0x5622651b71e0 0x5622651b9620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:25:15.737 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:25:15.734+0000 7f81f6000640 1 --2- 172.21.15.118:0/1145513718 >> [v2:172.21.15.189:6838/4074484168,v1:172.21.15.189:6839/4074484168] conn(0x5622651b71e0 0x5622651b9620 crc :-1 s=READY pgs=438 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=a4ef1f9f8a5078fa server_cookie=c59b18957c6cf6e6 in_seq=0 out_seq=0 2024-03-24T00:25:15.737 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:25:15.734+0000 7f81e4c00640 10 client.27077 ms_handle_connect on v2:172.21.15.189:6838/4074484168 2024-03-24T00:25:15.738 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:25:15.734+0000 7f81e4c00640 1 -- 172.21.15.118:0/1145513718 <== mds.0 v2:172.21.15.189:6838/4074484168 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+21660 (crc 0 0 0) 0x562265113d70 con 0x5622651b71e0 2024-03-24T00:25:15.738 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:25:15.734+0000 7f81e4c00640 10 client.27077 handle_command_reply: tid=0 2024-03-24T00:25:15.738 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:25:15.735+0000 7f82007d1740 1 client.27077 shutdown 2024-03-24T00:25:15.739 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:25:15.735+0000 7f82007d1740 1 -- 172.21.15.118:0/1145513718 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5622651a13f0 msgr2=0x5622651ab4f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:25:15.739 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:25:15.735+0000 7f82007d1740 1 --2- 172.21.15.118:0/1145513718 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5622651a13f0 0x5622651ab4f0 secure :-1 s=READY pgs=470 cs=0 l=1 rev1=1 crypto rx=0x7f81e802c710 tx=0x7f81e8032a20 comp rx=0 tx=0).stop 2024-03-24T00:25:15.739 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:25:15.735+0000 7f82007d1740 1 -- 172.21.15.118:0/1145513718 shutdown_connections 2024-03-24T00:25:15.739 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:25:15.735+0000 7f82007d1740 1 --2- 172.21.15.118:0/1145513718 >> [v2:172.21.15.189:6838/4074484168,v1:172.21.15.189:6839/4074484168] conn(0x5622651b71e0 0x5622651b9620 crc :-1 s=READY pgs=438 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:25:15.739 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:25:15.735+0000 7f82007d1740 1 --2- 172.21.15.118:0/1145513718 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5622651a6f50 0x5622651b2a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:25:15.739 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:25:15.735+0000 7f82007d1740 1 --2- 172.21.15.118:0/1145513718 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5622651a1de0 0x5622651aba30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:25:15.739 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:25:15.735+0000 7f81e4c00640 0 client.27077 ms_handle_reset on v2:172.21.15.189:6838/4074484168 2024-03-24T00:25:15.739 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:25:15.735+0000 7f82007d1740 1 --2- 172.21.15.118:0/1145513718 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5622651a13f0 0x5622651ab4f0 unknown :-1 s=CLOSED pgs=470 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:25:15.739 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:25:15.735+0000 7f82007d1740 1 -- 172.21.15.118:0/1145513718 >> 172.21.15.118:0/1145513718 conn(0x56226519c960 msgr2=0x56226519bb70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:25:15.739 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:25:15.735+0000 7f82007d1740 1 -- 172.21.15.118:0/1145513718 shutdown_connections 2024-03-24T00:25:15.739 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:25:15.735+0000 7f82007d1740 1 -- 172.21.15.118:0/1145513718 wait complete. 2024-03-24T00:25:15.739 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:25:15.735+0000 7f82007d1740 20 client.27077 trim_cache size 0 max 16384 2024-03-24T00:25:15.750 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:25:15.750+0000 7f82007d1740 1 Processor -- start 2024-03-24T00:25:15.750 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:25:15.750+0000 7f82007d1740 1 -- start start 2024-03-24T00:25:15.750 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:25:15.751+0000 7f82007d1740 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5622651a13f0 0x56226519fbc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:25:15.751 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:25:15.751+0000 7f82007d1740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5622651a1de0 0x5622651a0100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:25:15.751 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:25:15.751+0000 7f82007d1740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5622651a6f50 0x5622651bdf20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:25:15.751 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:25:15.751+0000 7f82007d1740 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f81f81ad130 con 0x5622651a13f0 2024-03-24T00:25:15.751 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:25:15.751+0000 7f82007d1740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f81f81ad790 con 0x5622651a6f50 2024-03-24T00:25:15.751 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:25:15.751+0000 7f82007d1740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f81f81ab150 con 0x5622651a1de0 2024-03-24T00:25:15.753 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:25:15.751+0000 7f81f6a00640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5622651a13f0 0x56226519fbc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:25:15.753 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:25:15.751+0000 7f81f7400640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5622651a6f50 0x5622651bdf20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:25:15.753 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:25:15.751+0000 7f81f6000640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5622651a1de0 0x5622651a0100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:25:15.753 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:25:15.751+0000 7f81f6a00640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5622651a13f0 0x56226519fbc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.118:3300/0 says I am v2:172.21.15.118:40292/0 (socket says 172.21.15.118:40292) 2024-03-24T00:25:15.753 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:25:15.751+0000 7f81f6a00640 1 -- 172.21.15.118:0/933271706 learned_addr learned my addr 172.21.15.118:0/933271706 (peer_addr_for_me v2:172.21.15.118:0/0) 2024-03-24T00:25:15.753 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:25:15.751+0000 7f81f7400640 1 --2- 172.21.15.118:0/933271706 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5622651a6f50 0x5622651bdf20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.118:58522/0 (socket says 172.21.15.118:58522) 2024-03-24T00:25:15.753 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:25:15.751+0000 7f81f6000640 1 -- 172.21.15.118:0/933271706 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5622651a6f50 msgr2=0x5622651bdf20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:25:15.754 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:25:15.751+0000 7f81f6000640 1 --2- 172.21.15.118:0/933271706 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5622651a6f50 0x5622651bdf20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:25:15.754 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:25:15.751+0000 7f81f6000640 1 -- 172.21.15.118:0/933271706 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5622651a13f0 msgr2=0x56226519fbc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:25:15.754 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:25:15.751+0000 7f81f6000640 1 --2- 172.21.15.118:0/933271706 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5622651a13f0 0x56226519fbc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:25:15.754 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:25:15.751+0000 7f81f6000640 1 -- 172.21.15.118:0/933271706 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x56226519af80 con 0x5622651a1de0 2024-03-24T00:25:15.754 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:25:15.751+0000 7f81f6a00640 1 --2- 172.21.15.118:0/933271706 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5622651a13f0 0x56226519fbc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T00:25:15.754 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:25:15.751+0000 7f81f7400640 1 --2- 172.21.15.118:0/933271706 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5622651a6f50 0x5622651bdf20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T00:25:15.754 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:25:15.752+0000 7f81f6000640 1 --2- 172.21.15.118:0/933271706 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5622651a1de0 0x5622651a0100 secure :-1 s=READY pgs=473 cs=0 l=1 rev1=1 crypto rx=0x5622651b9ea0 tx=0x7f81e00046c0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=3fb416c491b62897 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:25:15.754 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:25:15.752+0000 7f81e7400640 1 -- 172.21.15.118:0/933271706 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f81e0030020 con 0x5622651a1de0 2024-03-24T00:25:15.754 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:25:15.752+0000 7f81e7400640 1 -- 172.21.15.118:0/933271706 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f81e0005500 con 0x5622651a1de0 2024-03-24T00:25:15.754 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:25:15.752+0000 7f81e7400640 1 -- 172.21.15.118:0/933271706 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f81e00224d0 con 0x5622651a1de0 2024-03-24T00:25:15.754 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:25:15.753+0000 7f82007d1740 1 -- 172.21.15.118:0/933271706 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5622651a1de0 msgr2=0x5622651a0100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T00:25:15.754 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:25:15.753+0000 7f82007d1740 1 --2- 172.21.15.118:0/933271706 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5622651a1de0 0x5622651a0100 secure :-1 s=READY pgs=473 cs=0 l=1 rev1=1 crypto rx=0x5622651b9ea0 tx=0x7f81e00046c0 comp rx=0 tx=0).stop 2024-03-24T00:25:15.754 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:25:15.753+0000 7f82007d1740 1 -- 172.21.15.118:0/933271706 shutdown_connections 2024-03-24T00:25:15.754 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:25:15.753+0000 7f82007d1740 1 --2- 172.21.15.118:0/933271706 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5622651a6f50 0x5622651bdf20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:25:15.755 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:25:15.753+0000 7f82007d1740 1 --2- 172.21.15.118:0/933271706 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5622651a1de0 0x5622651a0100 unknown :-1 s=CLOSED pgs=473 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:25:15.755 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:25:15.753+0000 7f82007d1740 1 --2- 172.21.15.118:0/933271706 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5622651a13f0 0x56226519fbc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:25:15.755 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:25:15.753+0000 7f82007d1740 1 -- 172.21.15.118:0/933271706 >> 172.21.15.118:0/933271706 conn(0x56226519c960 msgr2=0x5622651a6700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T00:25:15.755 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:25:15.753+0000 7f82007d1740 1 -- 172.21.15.118:0/933271706 shutdown_connections 2024-03-24T00:25:15.755 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:25:15.753+0000 7f82007d1740 1 -- 172.21.15.118:0/933271706 wait complete. 2024-03-24T00:25:15.755 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:25:15.753+0000 7f82007d1740 1 -- start start 2024-03-24T00:25:15.755 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:25:15.753+0000 7f82007d1740 1 Processor -- start 2024-03-24T00:25:15.757 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:25:15.754+0000 7f82007d1740 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5622651a13f0 0x5622651af760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:25:15.758 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:25:15.754+0000 7f82007d1740 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5622651a1de0 0x5622651afca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:25:15.758 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:25:15.754+0000 7f82007d1740 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5622651a6f50 0x5622651b0cb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:25:15.758 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:25:15.754+0000 7f82007d1740 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f81f81ab7b0 con 0x5622651a1de0 2024-03-24T00:25:15.758 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:25:15.754+0000 7f82007d1740 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f81f81acad0 con 0x5622651a6f50 2024-03-24T00:25:15.758 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:25:15.754+0000 7f82007d1740 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f81f81abe10 con 0x5622651a13f0 2024-03-24T00:25:15.758 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:25:15.754+0000 7f82007d1740 10 client.0 _ll_register_callbacks cb 0x7f81e0009ec0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2024-03-24T00:25:15.758 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:25:15.754+0000 7f81f6a00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5622651a13f0 0x5622651af760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:25:15.758 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:25:15.754+0000 7f81f6000640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5622651a1de0 0x5622651afca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:25:15.758 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:25:15.754+0000 7f81f6a00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5622651a13f0 0x5622651af760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.118:35344/0 (socket says 172.21.15.118:35344) 2024-03-24T00:25:15.758 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:25:15.754+0000 7f81f7400640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5622651a6f50 0x5622651b0cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:25:15.758 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:25:15.754+0000 7f81f6a00640 1 -- 172.21.15.118:0/659686424 learned_addr learned my addr 172.21.15.118:0/659686424 (peer_addr_for_me v2:172.21.15.118:0/0) 2024-03-24T00:25:15.758 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:25:15.754+0000 7f81f6a00640 1 -- 172.21.15.118:0/659686424 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5622651a6f50 msgr2=0x5622651b0cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:25:15.759 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:25:15.754+0000 7f81f6a00640 1 --2- 172.21.15.118:0/659686424 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5622651a6f50 0x5622651b0cb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:25:15.759 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:25:15.754+0000 7f81f6a00640 1 -- 172.21.15.118:0/659686424 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5622651a1de0 msgr2=0x5622651afca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2024-03-24T00:25:15.759 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:25:15.754+0000 7f81f6a00640 1 --2- 172.21.15.118:0/659686424 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x5622651a1de0 0x5622651afca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:25:15.759 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:25:15.754+0000 7f81f6a00640 1 -- 172.21.15.118:0/659686424 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x5622651b12a0 con 0x5622651a13f0 2024-03-24T00:25:15.759 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:25:15.754+0000 7f81f7400640 1 --2- 172.21.15.118:0/659686424 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x5622651a6f50 0x5622651b0cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T00:25:15.759 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:25:15.755+0000 7f81f6a00640 1 --2- 172.21.15.118:0/659686424 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x5622651a13f0 0x5622651af760 secure :-1 s=READY pgs=474 cs=0 l=1 rev1=1 crypto rx=0x7f81e8000f80 tx=0x7f81e8054bd0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=9767383b959f1a8b server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:25:15.759 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:25:15.755+0000 7f81e4c00640 10 client.0 ms_handle_connect on v2:172.21.15.193:3300/0 2024-03-24T00:25:15.759 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:25:15.755+0000 7f81e4c00640 1 -- 172.21.15.118:0/659686424 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f81e804c070 con 0x5622651a13f0 2024-03-24T00:25:15.759 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:25:15.755+0000 7f81e4c00640 1 -- 172.21.15.118:0/659686424 <== mon.1 v2:172.21.15.193:3300/0 2 ==== osd_map(124..124 src has 1..124) v4 ==== 11201+0+0 (secure 0 0 0) 0x7f81e8046360 con 0x5622651a13f0 2024-03-24T00:25:15.759 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:25:15.755+0000 7f82007d1740 1 -- 172.21.15.118:0/659686424 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f81e0005500 con 0x5622651a13f0 2024-03-24T00:25:15.759 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:25:15.755+0000 7f81e4c00640 1 -- 172.21.15.118:0/659686424 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_get_version_reply(handle=1 version=186) v2 ==== 24+0+0 (secure 0 0 0) 0x7f81e8039070 con 0x5622651a13f0 2024-03-24T00:25:15.759 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:25:15.755+0000 7f82007d1740 10 client.27094 fetch_fsmap learned FSMap version 186 2024-03-24T00:25:15.759 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:25:15.755+0000 7f82007d1740 1 -- 172.21.15.118:0/659686424 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,fsmap=186,monmap=4+}) v3 -- 0x5622651b25b0 con 0x5622651a13f0 2024-03-24T00:25:15.759 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:25:15.755+0000 7f81e4c00640 1 -- 172.21.15.118:0/659686424 <== mon.1 v2:172.21.15.193:3300/0 4 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f81e0005500 con 0x5622651a13f0 2024-03-24T00:25:15.760 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:25:15.756+0000 7f81e4c00640 1 -- 172.21.15.118:0/659686424 <== mon.1 v2:172.21.15.193:3300/0 5 ==== fsmap(e 186) v1 ==== 5658+0+0 (secure 0 0 0) 0x7f81e805faf0 con 0x5622651a13f0 2024-03-24T00:25:15.760 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:25:15.756+0000 7f82007d1740 10 client.27094 fetch_fsmap finished waiting for FSMap version 186 2024-03-24T00:25:15.760 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:25:15.756+0000 7f82007d1740 10 client.27094 resolve_mds: resolved cephfs:0 to role '1:0' aka daemon mds.b 2024-03-24T00:25:15.760 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:25:15.756+0000 7f82007d1740 20 client.27094 populate_metadata read hostname 'smithi118' 2024-03-24T00:25:15.760 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:25:15.756+0000 7f82007d1740 1 --2- 172.21.15.118:0/659686424 >> [v2:172.21.15.189:6838/4074484168,v1:172.21.15.189:6839/4074484168] conn(0x5622651b71e0 0x562265215110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:25:15.760 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:25:15.756+0000 7f82007d1740 4 client.27094 mds_command: new command op to 24569 tid=0 multi_id=0 [{"prefix": "flush journal"}] 2024-03-24T00:25:15.760 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:25:15.756+0000 7f82007d1740 1 -- 172.21.15.118:0/659686424 --> [v2:172.21.15.189:6838/4074484168,v1:172.21.15.189:6839/4074484168] -- command(tid 0: {"prefix": "flush journal"}) v1 -- 0x56226510e0c0 con 0x5622651b71e0 2024-03-24T00:25:15.760 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:25:15.756+0000 7f81f6000640 1 --2- 172.21.15.118:0/659686424 >> [v2:172.21.15.189:6838/4074484168,v1:172.21.15.189:6839/4074484168] conn(0x5622651b71e0 0x562265215110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:25:15.760 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:25:15.757+0000 7f81f6000640 1 --2- 172.21.15.118:0/659686424 >> [v2:172.21.15.189:6838/4074484168,v1:172.21.15.189:6839/4074484168] conn(0x5622651b71e0 0x562265215110 crc :-1 s=READY pgs=439 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=15e8ff9b10268bee server_cookie=e03295acb13a4178 in_seq=0 out_seq=0 2024-03-24T00:25:15.760 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:25:15.757+0000 7f81e4c00640 10 client.27094 ms_handle_connect on v2:172.21.15.189:6838/4074484168 2024-03-24T00:25:15.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:25:15 smithi118 ceph-mon[31535]: pgmap v718: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 255 KiB/s rd, 14 KiB/s wr, 42 op/s 2024-03-24T00:25:15.946 INFO:journalctl@ceph.mds.d.smithi118.stdout:Mar 24 00:25:15 smithi118 systemd[1]: ceph-d79a5420-e971-11ee-95cd-87774f69a715@mds.d.service: Scheduled restart job, restart counter is at 6. 2024-03-24T00:25:15.947 INFO:journalctl@ceph.mds.d.smithi118.stdout:Mar 24 00:25:15 smithi118 systemd[1]: Stopped Ceph mds.d for d79a5420-e971-11ee-95cd-87774f69a715. 2024-03-24T00:25:15.947 INFO:journalctl@ceph.mds.d.smithi118.stdout:Mar 24 00:25:15 smithi118 systemd[1]: ceph-d79a5420-e971-11ee-95cd-87774f69a715@mds.d.service: Consumed 1.632s CPU time. 2024-03-24T00:25:15.947 INFO:journalctl@ceph.mds.d.smithi118.stdout:Mar 24 00:25:15 smithi118 systemd[1]: ceph-d79a5420-e971-11ee-95cd-87774f69a715@mds.d.service: Start request repeated too quickly. 2024-03-24T00:25:15.947 INFO:journalctl@ceph.mds.d.smithi118.stdout:Mar 24 00:25:15 smithi118 systemd[1]: ceph-d79a5420-e971-11ee-95cd-87774f69a715@mds.d.service: Failed with result 'exit-code'. 2024-03-24T00:25:15.947 INFO:journalctl@ceph.mds.d.smithi118.stdout:Mar 24 00:25:15 smithi118 systemd[1]: Failed to start Ceph mds.d for d79a5420-e971-11ee-95cd-87774f69a715. 2024-03-24T00:25:15.953 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:25:15 smithi189 ceph-mon[36187]: pgmap v718: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 255 KiB/s rd, 14 KiB/s wr, 42 op/s 2024-03-24T00:25:15.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:25:15 smithi193 ceph-mon[36878]: pgmap v718: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 255 KiB/s rd, 14 KiB/s wr, 42 op/s 2024-03-24T00:25:16.446 INFO:journalctl@ceph.mds.j.smithi118.stdout:Mar 24 00:25:15 smithi118 systemd[1]: ceph-d79a5420-e971-11ee-95cd-87774f69a715@mds.j.service: Scheduled restart job, restart counter is at 6. 2024-03-24T00:25:16.447 INFO:journalctl@ceph.mds.j.smithi118.stdout:Mar 24 00:25:15 smithi118 systemd[1]: Stopped Ceph mds.j for d79a5420-e971-11ee-95cd-87774f69a715. 2024-03-24T00:25:16.447 INFO:journalctl@ceph.mds.j.smithi118.stdout:Mar 24 00:25:15 smithi118 systemd[1]: ceph-d79a5420-e971-11ee-95cd-87774f69a715@mds.j.service: Consumed 1.397s CPU time. 2024-03-24T00:25:16.447 INFO:journalctl@ceph.mds.j.smithi118.stdout:Mar 24 00:25:15 smithi118 systemd[1]: ceph-d79a5420-e971-11ee-95cd-87774f69a715@mds.j.service: Start request repeated too quickly. 2024-03-24T00:25:16.447 INFO:journalctl@ceph.mds.j.smithi118.stdout:Mar 24 00:25:15 smithi118 systemd[1]: ceph-d79a5420-e971-11ee-95cd-87774f69a715@mds.j.service: Failed with result 'exit-code'. 2024-03-24T00:25:16.447 INFO:journalctl@ceph.mds.j.smithi118.stdout:Mar 24 00:25:15 smithi118 systemd[1]: Failed to start Ceph mds.j for d79a5420-e971-11ee-95cd-87774f69a715. 2024-03-24T00:25:17.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:25:17 smithi118 ceph-mon[31535]: pgmap v719: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 194 KiB/s rd, 14 KiB/s wr, 33 op/s 2024-03-24T00:25:17.959 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:25:17 smithi189 ceph-mon[36187]: pgmap v719: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 194 KiB/s rd, 14 KiB/s wr, 33 op/s 2024-03-24T00:25:17.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:25:17 smithi193 ceph-mon[36878]: pgmap v719: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 194 KiB/s rd, 14 KiB/s wr, 33 op/s 2024-03-24T00:25:19.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:25:19 smithi118 ceph-mon[31535]: pgmap v720: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 148 KiB/s rd, 11 KiB/s wr, 25 op/s 2024-03-24T00:25:19.953 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:25:19 smithi189 ceph-mon[36187]: pgmap v720: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 148 KiB/s rd, 11 KiB/s wr, 25 op/s 2024-03-24T00:25:19.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:25:19 smithi193 ceph-mon[36878]: pgmap v720: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 148 KiB/s rd, 11 KiB/s wr, 25 op/s 2024-03-24T00:25:21.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:25:21 smithi118 ceph-mon[31535]: pgmap v721: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 37 KiB/s rd, 12 KiB/s wr, 12 op/s 2024-03-24T00:25:21.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:25:21 smithi189 ceph-mon[36187]: pgmap v721: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 37 KiB/s rd, 12 KiB/s wr, 12 op/s 2024-03-24T00:25:21.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:25:21 smithi193 ceph-mon[36878]: pgmap v721: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 37 KiB/s rd, 12 KiB/s wr, 12 op/s 2024-03-24T00:25:22.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:25:22 smithi118 ceph-mon[31535]: fsmap cephfs:5 3 up:standby-replay 5 up:active 2024-03-24T00:25:22.953 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:25:22 smithi189 ceph-mon[36187]: fsmap cephfs:5 3 up:standby-replay 5 up:active 2024-03-24T00:25:22.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:25:22 smithi193 ceph-mon[36878]: fsmap cephfs:5 3 up:standby-replay 5 up:active 2024-03-24T00:25:23.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:25:23 smithi118 ceph-mon[31535]: pgmap v722: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 1.9 KiB/s wr, 7 op/s 2024-03-24T00:25:23.953 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:25:23 smithi189 ceph-mon[36187]: pgmap v722: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 1.9 KiB/s wr, 7 op/s 2024-03-24T00:25:23.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:25:23 smithi193 ceph-mon[36878]: pgmap v722: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 1.9 KiB/s wr, 7 op/s 2024-03-24T00:25:25.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:25:25 smithi118 ceph-mon[31535]: pgmap v723: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 1.9 KiB/s wr, 9 op/s 2024-03-24T00:25:25.953 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:25:25 smithi189 ceph-mon[36187]: pgmap v723: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 1.9 KiB/s wr, 9 op/s 2024-03-24T00:25:25.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:25:25 smithi193 ceph-mon[36878]: pgmap v723: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 1.9 KiB/s wr, 9 op/s 2024-03-24T00:25:27.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:25:27 smithi118 ceph-mon[31535]: pgmap v724: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 1.9 KiB/s wr, 8 op/s 2024-03-24T00:25:27.953 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:25:27 smithi189 ceph-mon[36187]: pgmap v724: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 1.9 KiB/s wr, 8 op/s 2024-03-24T00:25:27.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:25:27 smithi193 ceph-mon[36878]: pgmap v724: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 1.9 KiB/s wr, 8 op/s 2024-03-24T00:25:29.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:25:29 smithi118 ceph-mon[31535]: pgmap v725: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 1.9 KiB/s wr, 8 op/s 2024-03-24T00:25:29.953 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:25:29 smithi189 ceph-mon[36187]: pgmap v725: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 1.9 KiB/s wr, 8 op/s 2024-03-24T00:25:29.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:25:29 smithi193 ceph-mon[36878]: pgmap v725: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 1.9 KiB/s wr, 8 op/s 2024-03-24T00:25:31.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:25:31 smithi189 ceph-mon[36187]: pgmap v726: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 1.2 KiB/s wr, 7 op/s 2024-03-24T00:25:31.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:25:31 smithi193 ceph-mon[36878]: pgmap v726: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 1.2 KiB/s wr, 7 op/s 2024-03-24T00:25:32.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:25:31 smithi118 ceph-mon[31535]: pgmap v726: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.8 KiB/s rd, 1.2 KiB/s wr, 7 op/s 2024-03-24T00:25:33.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:25:33 smithi189 ceph-mon[36187]: pgmap v727: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 85 B/s wr, 5 op/s 2024-03-24T00:25:33.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:25:33 smithi193 ceph-mon[36878]: pgmap v727: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 85 B/s wr, 5 op/s 2024-03-24T00:25:34.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:25:33 smithi118 ceph-mon[31535]: pgmap v727: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 85 B/s wr, 5 op/s 2024-03-24T00:25:35.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:25:35 smithi193 ceph-mon[36878]: pgmap v728: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T00:25:36.024 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:25:35 smithi118 ceph-mon[31535]: pgmap v728: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T00:25:36.203 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:25:35 smithi189 ceph-mon[36187]: pgmap v728: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T00:25:38.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:25:37 smithi118 ceph-mon[31535]: pgmap v729: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T00:25:38.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:25:37 smithi189 ceph-mon[36187]: pgmap v729: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T00:25:38.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:25:37 smithi193 ceph-mon[36878]: pgmap v729: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T00:25:40.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:25:39 smithi118 ceph-mon[31535]: pgmap v730: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 5 op/s 2024-03-24T00:25:40.203 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:25:39 smithi189 ceph-mon[36187]: pgmap v730: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 5 op/s 2024-03-24T00:25:40.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:25:39 smithi193 ceph-mon[36878]: pgmap v730: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 5 op/s 2024-03-24T00:25:42.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:25:41 smithi118 ceph-mon[31535]: pgmap v731: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T00:25:42.203 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:25:41 smithi189 ceph-mon[36187]: pgmap v731: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T00:25:42.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:25:41 smithi193 ceph-mon[36878]: pgmap v731: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T00:25:44.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:25:43 smithi118 ceph-mon[31535]: pgmap v732: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:25:44.203 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:25:43 smithi189 ceph-mon[36187]: pgmap v732: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:25:44.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:25:43 smithi193 ceph-mon[36878]: pgmap v732: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:25:46.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:25:45 smithi118 ceph-mon[31535]: pgmap v733: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T00:25:46.203 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:25:45 smithi189 ceph-mon[36187]: pgmap v733: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T00:25:46.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:25:45 smithi193 ceph-mon[36878]: pgmap v733: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T00:25:48.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:25:47 smithi118 ceph-mon[31535]: pgmap v734: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T00:25:48.203 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:25:47 smithi189 ceph-mon[36187]: pgmap v734: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T00:25:48.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:25:47 smithi193 ceph-mon[36878]: pgmap v734: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T00:25:50.198 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:25:49 smithi118 ceph-mon[31535]: pgmap v735: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 5 op/s 2024-03-24T00:25:50.203 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:25:49 smithi189 ceph-mon[36187]: pgmap v735: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 5 op/s 2024-03-24T00:25:50.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:25:49 smithi193 ceph-mon[36878]: pgmap v735: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 5 op/s 2024-03-24T00:25:52.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:25:51 smithi118 ceph-mon[31535]: pgmap v736: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T00:25:52.203 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:25:51 smithi189 ceph-mon[36187]: pgmap v736: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T00:25:52.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:25:51 smithi193 ceph-mon[36878]: pgmap v736: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T00:25:54.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:25:53 smithi118 ceph-mon[31535]: pgmap v737: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:25:54.203 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:25:53 smithi189 ceph-mon[36187]: pgmap v737: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:25:54.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:25:53 smithi193 ceph-mon[36878]: pgmap v737: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:25:56.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:25:55 smithi118 ceph-mon[31535]: pgmap v738: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T00:25:56.203 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:25:55 smithi189 ceph-mon[36187]: pgmap v738: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T00:25:56.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:25:55 smithi193 ceph-mon[36878]: pgmap v738: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T00:25:58.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:25:57 smithi118 ceph-mon[31535]: pgmap v739: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T00:25:58.203 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:25:57 smithi189 ceph-mon[36187]: pgmap v739: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T00:25:58.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:25:57 smithi193 ceph-mon[36878]: pgmap v739: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T00:25:59.017 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:25:58 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T00:25:59.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:25:58 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T00:25:59.203 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:25:58 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T00:25:59.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:25:59 smithi189 ceph-mon[36187]: pgmap v740: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 5 op/s 2024-03-24T00:25:59.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:25:59 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T00:25:59.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:25:59 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T00:25:59.955 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:25:59 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T00:25:59.955 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:25:59 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T00:26:00.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:25:59 smithi118 ceph-mon[31535]: pgmap v740: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 5 op/s 2024-03-24T00:26:00.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:25:59 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T00:26:00.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:25:59 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T00:26:00.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:25:59 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T00:26:00.197 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:25:59 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T00:26:00.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:25:59 smithi193 ceph-mon[36878]: pgmap v740: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 5 op/s 2024-03-24T00:26:00.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:25:59 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T00:26:00.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:25:59 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T00:26:00.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:25:59 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T00:26:00.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:25:59 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T00:26:01.697 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:26:01 smithi118 ceph-mon[31535]: pgmap v741: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T00:26:01.697 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:26:01 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T00:26:01.697 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:26:01 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T00:26:01.697 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:26:01 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T00:26:01.697 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:26:01 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T00:26:01.703 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:26:01 smithi189 ceph-mon[36187]: pgmap v741: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T00:26:01.704 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:26:01 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T00:26:01.704 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:26:01 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T00:26:01.704 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:26:01 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T00:26:01.704 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:26:01 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T00:26:01.710 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:26:01 smithi193 ceph-mon[36878]: pgmap v741: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T00:26:01.711 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:26:01 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T00:26:01.711 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:26:01 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T00:26:01.711 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:26:01 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T00:26:01.711 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:26:01 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T00:26:04.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:26:03 smithi118 ceph-mon[31535]: pgmap v742: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:26:04.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:26:03 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T00:26:04.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:26:03 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T00:26:04.203 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:26:03 smithi189 ceph-mon[36187]: pgmap v742: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:26:04.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:26:03 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T00:26:04.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:26:03 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T00:26:04.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:26:03 smithi193 ceph-mon[36878]: pgmap v742: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:26:04.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:26:03 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T00:26:04.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:26:03 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T00:26:05.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:26:04 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T00:26:05.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:26:04 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T00:26:05.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:26:04 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T00:26:05.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:26:04 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T00:26:05.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:26:04 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T00:26:05.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:26:04 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T00:26:05.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:26:04 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T00:26:05.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:26:04 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T00:26:05.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:26:04 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T00:26:06.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:26:05 smithi118 ceph-mon[31535]: pgmap v743: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T00:26:06.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:26:05 smithi118 ceph-mon[31535]: pgmap v744: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 6 op/s 2024-03-24T00:26:06.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:26:05 smithi118 ceph-mon[31535]: Health check failed: 6 failed cephadm daemon(s) (CEPHADM_FAILED_DAEMON) 2024-03-24T00:26:06.203 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:26:05 smithi189 ceph-mon[36187]: pgmap v743: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T00:26:06.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:26:05 smithi189 ceph-mon[36187]: pgmap v744: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 6 op/s 2024-03-24T00:26:06.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:26:05 smithi189 ceph-mon[36187]: Health check failed: 6 failed cephadm daemon(s) (CEPHADM_FAILED_DAEMON) 2024-03-24T00:26:06.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:26:05 smithi193 ceph-mon[36878]: pgmap v743: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T00:26:06.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:26:05 smithi193 ceph-mon[36878]: pgmap v744: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 6 op/s 2024-03-24T00:26:06.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:26:05 smithi193 ceph-mon[36878]: Health check failed: 6 failed cephadm daemon(s) (CEPHADM_FAILED_DAEMON) 2024-03-24T00:26:08.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:26:07 smithi118 ceph-mon[31535]: pgmap v745: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T00:26:08.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:26:07 smithi189 ceph-mon[36187]: pgmap v745: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T00:26:08.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:26:07 smithi193 ceph-mon[36878]: pgmap v745: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T00:26:10.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:26:09 smithi118 ceph-mon[31535]: pgmap v746: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 0 B/s wr, 6 op/s 2024-03-24T00:26:10.203 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:26:09 smithi189 ceph-mon[36187]: pgmap v746: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 0 B/s wr, 6 op/s 2024-03-24T00:26:10.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:26:09 smithi193 ceph-mon[36878]: pgmap v746: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 0 B/s wr, 6 op/s 2024-03-24T00:26:12.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:26:11 smithi118 ceph-mon[31535]: pgmap v747: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 0 B/s wr, 6 op/s 2024-03-24T00:26:12.203 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:26:11 smithi189 ceph-mon[36187]: pgmap v747: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 0 B/s wr, 6 op/s 2024-03-24T00:26:12.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:26:11 smithi193 ceph-mon[36878]: pgmap v747: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 0 B/s wr, 6 op/s 2024-03-24T00:26:14.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:26:13 smithi118 ceph-mon[31535]: pgmap v748: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 0 B/s wr, 6 op/s 2024-03-24T00:26:14.203 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:26:13 smithi189 ceph-mon[36187]: pgmap v748: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 0 B/s wr, 6 op/s 2024-03-24T00:26:14.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:26:13 smithi193 ceph-mon[36878]: pgmap v748: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 0 B/s wr, 6 op/s 2024-03-24T00:26:16.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:26:15 smithi118 ceph-mon[31535]: pgmap v749: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 0 B/s wr, 7 op/s 2024-03-24T00:26:16.203 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:26:15 smithi189 ceph-mon[36187]: pgmap v749: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 0 B/s wr, 7 op/s 2024-03-24T00:26:16.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:26:15 smithi193 ceph-mon[36878]: pgmap v749: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 0 B/s wr, 7 op/s 2024-03-24T00:26:18.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:26:17 smithi118 ceph-mon[31535]: pgmap v750: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 0 B/s wr, 5 op/s 2024-03-24T00:26:18.203 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:26:17 smithi189 ceph-mon[36187]: pgmap v750: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 0 B/s wr, 5 op/s 2024-03-24T00:26:18.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:26:17 smithi193 ceph-mon[36878]: pgmap v750: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 0 B/s wr, 5 op/s 2024-03-24T00:26:20.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:26:19 smithi118 ceph-mon[31535]: pgmap v751: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 0 B/s wr, 6 op/s 2024-03-24T00:26:20.203 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:26:19 smithi189 ceph-mon[36187]: pgmap v751: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 0 B/s wr, 6 op/s 2024-03-24T00:26:20.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:26:19 smithi193 ceph-mon[36878]: pgmap v751: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 0 B/s wr, 6 op/s 2024-03-24T00:26:22.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:26:21 smithi118 ceph-mon[31535]: pgmap v752: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T00:26:22.203 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:26:21 smithi189 ceph-mon[36187]: pgmap v752: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T00:26:22.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:26:21 smithi193 ceph-mon[36878]: pgmap v752: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T00:26:24.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:26:23 smithi118 ceph-mon[31535]: pgmap v753: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:26:24.203 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:26:23 smithi189 ceph-mon[36187]: pgmap v753: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:26:24.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:26:23 smithi193 ceph-mon[36878]: pgmap v753: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:26:26.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:26:25 smithi118 ceph-mon[31535]: pgmap v754: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-24T00:26:26.203 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:26:25 smithi189 ceph-mon[36187]: pgmap v754: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-24T00:26:26.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:26:25 smithi193 ceph-mon[36878]: pgmap v754: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-24T00:26:28.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:26:27 smithi118 ceph-mon[31535]: pgmap v755: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:26:28.203 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:26:27 smithi189 ceph-mon[36187]: pgmap v755: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:26:28.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:26:27 smithi193 ceph-mon[36878]: pgmap v755: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:26:30.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:26:29 smithi118 ceph-mon[31535]: pgmap v756: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T00:26:30.203 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:26:29 smithi189 ceph-mon[36187]: pgmap v756: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T00:26:30.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:26:29 smithi193 ceph-mon[36878]: pgmap v756: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T00:26:32.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:26:31 smithi118 ceph-mon[31535]: pgmap v757: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T00:26:32.203 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:26:31 smithi189 ceph-mon[36187]: pgmap v757: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T00:26:32.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:26:31 smithi193 ceph-mon[36878]: pgmap v757: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T00:26:33.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:26:32 smithi118 ceph-mon[31535]: pgmap v758: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:26:33.453 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:26:32 smithi189 ceph-mon[36187]: pgmap v758: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:26:33.460 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:26:32 smithi193 ceph-mon[36878]: pgmap v758: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:26:35.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:26:35 smithi118 ceph-mon[31535]: pgmap v759: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-24T00:26:35.703 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:26:35 smithi189 ceph-mon[36187]: pgmap v759: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-24T00:26:35.710 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:26:35 smithi193 ceph-mon[36878]: pgmap v759: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-24T00:26:37.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:26:37 smithi118 ceph-mon[31535]: pgmap v760: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:26:37.703 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:26:37 smithi189 ceph-mon[36187]: pgmap v760: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:26:37.710 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:26:37 smithi193 ceph-mon[36878]: pgmap v760: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:26:39.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:26:39 smithi118 ceph-mon[31535]: pgmap v761: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T00:26:39.703 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:26:39 smithi189 ceph-mon[36187]: pgmap v761: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T00:26:39.710 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:26:39 smithi193 ceph-mon[36878]: pgmap v761: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T00:26:41.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:26:41 smithi118 ceph-mon[31535]: pgmap v762: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T00:26:41.703 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:26:41 smithi189 ceph-mon[36187]: pgmap v762: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T00:26:41.710 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:26:41 smithi193 ceph-mon[36878]: pgmap v762: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T00:26:43.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:26:43 smithi118 ceph-mon[31535]: pgmap v763: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:26:43.703 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:26:43 smithi189 ceph-mon[36187]: pgmap v763: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:26:43.710 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:26:43 smithi193 ceph-mon[36878]: pgmap v763: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:26:45.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:26:45 smithi118 ceph-mon[31535]: pgmap v764: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-24T00:26:45.704 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:26:45 smithi189 ceph-mon[36187]: pgmap v764: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-24T00:26:45.710 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:26:45 smithi193 ceph-mon[36878]: pgmap v764: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-24T00:26:47.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:26:47 smithi118 ceph-mon[31535]: pgmap v765: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:26:47.703 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:26:47 smithi189 ceph-mon[36187]: pgmap v765: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:26:47.710 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:26:47 smithi193 ceph-mon[36878]: pgmap v765: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:26:49.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:26:49 smithi118 ceph-mon[31535]: pgmap v766: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T00:26:49.703 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:26:49 smithi189 ceph-mon[36187]: pgmap v766: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T00:26:49.710 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:26:49 smithi193 ceph-mon[36878]: pgmap v766: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T00:26:51.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:26:51 smithi118 ceph-mon[31535]: pgmap v767: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T00:26:51.703 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:26:51 smithi189 ceph-mon[36187]: pgmap v767: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T00:26:51.710 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:26:51 smithi193 ceph-mon[36878]: pgmap v767: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T00:26:53.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:26:53 smithi118 ceph-mon[31535]: pgmap v768: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:26:53.703 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:26:53 smithi189 ceph-mon[36187]: pgmap v768: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:26:53.710 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:26:53 smithi193 ceph-mon[36878]: pgmap v768: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:26:55.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:26:55 smithi118 ceph-mon[31535]: pgmap v769: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-24T00:26:55.704 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:26:55 smithi189 ceph-mon[36187]: pgmap v769: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-24T00:26:55.710 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:26:55 smithi193 ceph-mon[36878]: pgmap v769: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-24T00:26:57.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:26:57 smithi118 ceph-mon[31535]: pgmap v770: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:26:57.703 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:26:57 smithi189 ceph-mon[36187]: pgmap v770: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:26:57.710 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:26:57 smithi193 ceph-mon[36878]: pgmap v770: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:26:59.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:26:59 smithi118 ceph-mon[31535]: pgmap v771: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T00:26:59.703 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:26:59 smithi189 ceph-mon[36187]: pgmap v771: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T00:26:59.710 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:26:59 smithi193 ceph-mon[36878]: pgmap v771: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T00:27:01.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:27:01 smithi118 ceph-mon[31535]: pgmap v772: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T00:27:01.704 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:27:01 smithi189 ceph-mon[36187]: pgmap v772: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T00:27:01.710 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:27:01 smithi193 ceph-mon[36878]: pgmap v772: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T00:27:03.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:27:03 smithi118 ceph-mon[31535]: pgmap v773: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:27:03.703 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:27:03 smithi189 ceph-mon[36187]: pgmap v773: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:27:03.710 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:27:03 smithi193 ceph-mon[36878]: pgmap v773: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:27:05.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:27:05 smithi118 ceph-mon[31535]: pgmap v774: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-24T00:27:05.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:27:05 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T00:27:05.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:27:05 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T00:27:05.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:27:05 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T00:27:05.703 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:27:05 smithi189 ceph-mon[36187]: pgmap v774: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-24T00:27:05.704 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:27:05 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T00:27:05.704 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:27:05 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T00:27:05.704 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:27:05 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T00:27:05.710 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:27:05 smithi193 ceph-mon[36878]: pgmap v774: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-24T00:27:05.710 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:27:05 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T00:27:05.711 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:27:05 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T00:27:05.711 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:27:05 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T00:27:06.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:27:06 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T00:27:06.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:27:06 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T00:27:06.697 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:27:06 smithi118 ceph-mon[31535]: pgmap v775: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 5 op/s 2024-03-24T00:27:06.697 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:27:06 smithi118 ceph-mon[31535]: pgmap v776: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 6 op/s 2024-03-24T00:27:06.697 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:27:06 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T00:27:06.703 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:27:06 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T00:27:06.704 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:27:06 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T00:27:06.704 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:27:06 smithi189 ceph-mon[36187]: pgmap v775: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 5 op/s 2024-03-24T00:27:06.704 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:27:06 smithi189 ceph-mon[36187]: pgmap v776: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 6 op/s 2024-03-24T00:27:06.704 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:27:06 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T00:27:06.710 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:27:06 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T00:27:06.711 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:27:06 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T00:27:06.711 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:27:06 smithi193 ceph-mon[36878]: pgmap v775: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 5 op/s 2024-03-24T00:27:06.711 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:27:06 smithi193 ceph-mon[36878]: pgmap v776: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 6 op/s 2024-03-24T00:27:06.711 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:27:06 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T00:27:09.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:27:08 smithi118 ceph-mon[31535]: pgmap v777: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 5 op/s 2024-03-24T00:27:09.203 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:27:08 smithi189 ceph-mon[36187]: pgmap v777: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 5 op/s 2024-03-24T00:27:09.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:27:08 smithi193 ceph-mon[36878]: pgmap v777: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 5 op/s 2024-03-24T00:27:11.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:27:10 smithi118 ceph-mon[31535]: pgmap v778: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 6 op/s 2024-03-24T00:27:11.203 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:27:10 smithi189 ceph-mon[36187]: pgmap v778: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 6 op/s 2024-03-24T00:27:11.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:27:10 smithi193 ceph-mon[36878]: pgmap v778: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 6 op/s 2024-03-24T00:27:13.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:27:12 smithi118 ceph-mon[31535]: pgmap v779: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 6 op/s 2024-03-24T00:27:13.203 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:27:12 smithi189 ceph-mon[36187]: pgmap v779: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 6 op/s 2024-03-24T00:27:13.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:27:12 smithi193 ceph-mon[36878]: pgmap v779: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 6 op/s 2024-03-24T00:27:15.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:27:14 smithi118 ceph-mon[31535]: pgmap v780: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 4 op/s 2024-03-24T00:27:15.203 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:27:14 smithi189 ceph-mon[36187]: pgmap v780: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 4 op/s 2024-03-24T00:27:15.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:27:14 smithi193 ceph-mon[36878]: pgmap v780: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 4 op/s 2024-03-24T00:27:17.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:27:16 smithi118 ceph-mon[31535]: pgmap v781: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 5 op/s 2024-03-24T00:27:17.203 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:27:16 smithi189 ceph-mon[36187]: pgmap v781: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 5 op/s 2024-03-24T00:27:17.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:27:16 smithi193 ceph-mon[36878]: pgmap v781: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 5 op/s 2024-03-24T00:27:19.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:27:18 smithi118 ceph-mon[31535]: pgmap v782: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T00:27:19.203 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:27:18 smithi189 ceph-mon[36187]: pgmap v782: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T00:27:19.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:27:18 smithi193 ceph-mon[36878]: pgmap v782: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T00:27:21.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:27:20 smithi118 ceph-mon[31535]: pgmap v783: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-03-24T00:27:21.203 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:27:20 smithi189 ceph-mon[36187]: pgmap v783: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-03-24T00:27:21.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:27:20 smithi193 ceph-mon[36878]: pgmap v783: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-03-24T00:27:23.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:27:22 smithi118 ceph-mon[31535]: pgmap v784: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:27:23.203 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:27:22 smithi189 ceph-mon[36187]: pgmap v784: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:27:23.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:27:22 smithi193 ceph-mon[36878]: pgmap v784: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:27:25.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:27:24 smithi118 ceph-mon[31535]: pgmap v785: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T00:27:25.203 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:27:24 smithi189 ceph-mon[36187]: pgmap v785: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T00:27:25.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:27:24 smithi193 ceph-mon[36878]: pgmap v785: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T00:27:27.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:27:26 smithi118 ceph-mon[31535]: pgmap v786: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T00:27:27.203 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:27:26 smithi189 ceph-mon[36187]: pgmap v786: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T00:27:27.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:27:26 smithi193 ceph-mon[36878]: pgmap v786: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T00:27:29.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:27:28 smithi118 ceph-mon[31535]: pgmap v787: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T00:27:29.203 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:27:28 smithi189 ceph-mon[36187]: pgmap v787: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T00:27:29.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:27:28 smithi193 ceph-mon[36878]: pgmap v787: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T00:27:31.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:27:30 smithi118 ceph-mon[31535]: pgmap v788: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-03-24T00:27:31.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:27:30 smithi189 ceph-mon[36187]: pgmap v788: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-03-24T00:27:31.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:27:30 smithi193 ceph-mon[36878]: pgmap v788: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-03-24T00:27:33.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:27:32 smithi118 ceph-mon[31535]: pgmap v789: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:27:33.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:27:32 smithi189 ceph-mon[36187]: pgmap v789: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:27:33.214 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:27:32 smithi193 ceph-mon[36878]: pgmap v789: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:27:35.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:27:34 smithi118 ceph-mon[31535]: pgmap v790: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T00:27:35.203 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:27:34 smithi189 ceph-mon[36187]: pgmap v790: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T00:27:35.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:27:34 smithi193 ceph-mon[36878]: pgmap v790: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T00:27:37.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:27:36 smithi118 ceph-mon[31535]: pgmap v791: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T00:27:37.203 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:27:36 smithi189 ceph-mon[36187]: pgmap v791: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T00:27:37.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:27:36 smithi193 ceph-mon[36878]: pgmap v791: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T00:27:39.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:27:38 smithi118 ceph-mon[31535]: pgmap v792: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T00:27:39.203 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:27:38 smithi189 ceph-mon[36187]: pgmap v792: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T00:27:39.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:27:38 smithi193 ceph-mon[36878]: pgmap v792: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T00:27:41.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:27:40 smithi118 ceph-mon[31535]: pgmap v793: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-03-24T00:27:41.203 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:27:40 smithi189 ceph-mon[36187]: pgmap v793: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-03-24T00:27:41.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:27:40 smithi193 ceph-mon[36878]: pgmap v793: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-03-24T00:27:43.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:27:42 smithi118 ceph-mon[31535]: pgmap v794: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:27:43.203 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:27:42 smithi189 ceph-mon[36187]: pgmap v794: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:27:43.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:27:42 smithi193 ceph-mon[36878]: pgmap v794: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:27:45.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:27:44 smithi118 ceph-mon[31535]: pgmap v795: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T00:27:45.203 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:27:44 smithi189 ceph-mon[36187]: pgmap v795: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T00:27:45.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:27:44 smithi193 ceph-mon[36878]: pgmap v795: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T00:27:47.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:27:46 smithi118 ceph-mon[31535]: pgmap v796: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T00:27:47.203 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:27:46 smithi189 ceph-mon[36187]: pgmap v796: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T00:27:47.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:27:46 smithi193 ceph-mon[36878]: pgmap v796: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T00:27:49.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:27:48 smithi118 ceph-mon[31535]: pgmap v797: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T00:27:49.203 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:27:48 smithi189 ceph-mon[36187]: pgmap v797: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T00:27:49.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:27:48 smithi193 ceph-mon[36878]: pgmap v797: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T00:27:51.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:27:50 smithi118 ceph-mon[31535]: pgmap v798: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-03-24T00:27:51.203 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:27:50 smithi189 ceph-mon[36187]: pgmap v798: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-03-24T00:27:51.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:27:50 smithi193 ceph-mon[36878]: pgmap v798: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-03-24T00:27:53.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:27:52 smithi118 ceph-mon[31535]: pgmap v799: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:27:53.203 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:27:52 smithi189 ceph-mon[36187]: pgmap v799: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:27:53.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:27:52 smithi193 ceph-mon[36878]: pgmap v799: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:27:55.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:27:54 smithi118 ceph-mon[31535]: pgmap v800: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T00:27:55.203 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:27:54 smithi189 ceph-mon[36187]: pgmap v800: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T00:27:55.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:27:54 smithi193 ceph-mon[36878]: pgmap v800: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T00:27:57.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:27:56 smithi118 ceph-mon[31535]: pgmap v801: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T00:27:57.203 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:27:56 smithi189 ceph-mon[36187]: pgmap v801: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T00:27:57.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:27:56 smithi193 ceph-mon[36878]: pgmap v801: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T00:27:59.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:27:58 smithi118 ceph-mon[31535]: pgmap v802: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T00:27:59.203 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:27:58 smithi189 ceph-mon[36187]: pgmap v802: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T00:27:59.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:27:58 smithi193 ceph-mon[36878]: pgmap v802: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T00:28:01.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:28:00 smithi118 ceph-mon[31535]: pgmap v803: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-03-24T00:28:01.203 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:28:00 smithi189 ceph-mon[36187]: pgmap v803: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-03-24T00:28:01.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:28:00 smithi193 ceph-mon[36878]: pgmap v803: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-03-24T00:28:03.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:28:02 smithi118 ceph-mon[31535]: pgmap v804: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:28:03.203 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:28:02 smithi189 ceph-mon[36187]: pgmap v804: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:28:03.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:28:02 smithi193 ceph-mon[36878]: pgmap v804: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:28:05.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:28:04 smithi118 ceph-mon[31535]: pgmap v805: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T00:28:05.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:28:04 smithi189 ceph-mon[36187]: pgmap v805: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T00:28:05.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:28:04 smithi193 ceph-mon[36878]: pgmap v805: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T00:28:06.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:28:05 smithi118 ceph-mon[31535]: 3 slow requests, 3 included below; oldest blocked for > 181.617521 secs 2024-03-24T00:28:06.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:28:05 smithi118 ceph-mon[31535]: slow request 181.617521 seconds old, received at 2024-03-24T00:25:03.572253+0000: peer_request:client.15159:154789 currently acquired locks 2024-03-24T00:28:06.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:28:05 smithi118 ceph-mon[31535]: slow request 181.566274 seconds old, received at 2024-03-24T00:25:03.623500+0000: peer_request:client.15159:158090 currently acquired locks 2024-03-24T00:28:06.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:28:05 smithi118 ceph-mon[31535]: slow request 181.559596 seconds old, received at 2024-03-24T00:25:03.630178+0000: client_request(client.15159:173196 rename #0x100000010bf/comment-15.xml #0x100000010bf/comment-15.xml.tmp 2024-03-24T00:25:03.629366+0000 caller_uid=1000, caller_gid=1301{6,36,1000,1301,}) currently failed to xlock, waiting 2024-03-24T00:28:06.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:28:05 smithi189 ceph-mon[36187]: 3 slow requests, 3 included below; oldest blocked for > 181.617521 secs 2024-03-24T00:28:06.205 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:28:05 smithi189 ceph-mon[36187]: slow request 181.617521 seconds old, received at 2024-03-24T00:25:03.572253+0000: peer_request:client.15159:154789 currently acquired locks 2024-03-24T00:28:06.205 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:28:05 smithi189 ceph-mon[36187]: slow request 181.566274 seconds old, received at 2024-03-24T00:25:03.623500+0000: peer_request:client.15159:158090 currently acquired locks 2024-03-24T00:28:06.205 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:28:05 smithi189 ceph-mon[36187]: slow request 181.559596 seconds old, received at 2024-03-24T00:25:03.630178+0000: client_request(client.15159:173196 rename #0x100000010bf/comment-15.xml #0x100000010bf/comment-15.xml.tmp 2024-03-24T00:25:03.629366+0000 caller_uid=1000, caller_gid=1301{6,36,1000,1301,}) currently failed to xlock, waiting 2024-03-24T00:28:06.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:28:05 smithi193 ceph-mon[36878]: 3 slow requests, 3 included below; oldest blocked for > 181.617521 secs 2024-03-24T00:28:06.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:28:05 smithi193 ceph-mon[36878]: slow request 181.617521 seconds old, received at 2024-03-24T00:25:03.572253+0000: peer_request:client.15159:154789 currently acquired locks 2024-03-24T00:28:06.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:28:05 smithi193 ceph-mon[36878]: slow request 181.566274 seconds old, received at 2024-03-24T00:25:03.623500+0000: peer_request:client.15159:158090 currently acquired locks 2024-03-24T00:28:06.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:28:05 smithi193 ceph-mon[36878]: slow request 181.559596 seconds old, received at 2024-03-24T00:25:03.630178+0000: client_request(client.15159:173196 rename #0x100000010bf/comment-15.xml #0x100000010bf/comment-15.xml.tmp 2024-03-24T00:25:03.629366+0000 caller_uid=1000, caller_gid=1301{6,36,1000,1301,}) currently failed to xlock, waiting 2024-03-24T00:28:07.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:28:06 smithi118 ceph-mon[31535]: pgmap v806: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T00:28:07.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:28:06 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T00:28:07.203 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:28:06 smithi189 ceph-mon[36187]: pgmap v806: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T00:28:07.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:28:06 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T00:28:07.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:28:06 smithi193 ceph-mon[36878]: pgmap v806: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T00:28:07.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:28:06 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T00:28:08.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:28:07 smithi118 ceph-mon[31535]: Health check failed: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2024-03-24T00:28:08.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:28:07 smithi118 ceph-mon[31535]: mds.? [v2:172.21.15.189:6838/4074484168,v1:172.21.15.189:6839/4074484168] up:active 2024-03-24T00:28:08.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:28:07 smithi118 ceph-mon[31535]: fsmap cephfs:5 3 up:standby-replay 5 up:active 2024-03-24T00:28:08.197 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:28:07 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T00:28:08.197 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:28:07 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T00:28:08.197 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:28:07 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T00:28:08.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:28:07 smithi189 ceph-mon[36187]: Health check failed: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2024-03-24T00:28:08.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:28:07 smithi189 ceph-mon[36187]: mds.? [v2:172.21.15.189:6838/4074484168,v1:172.21.15.189:6839/4074484168] up:active 2024-03-24T00:28:08.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:28:07 smithi189 ceph-mon[36187]: fsmap cephfs:5 3 up:standby-replay 5 up:active 2024-03-24T00:28:08.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:28:07 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T00:28:08.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:28:07 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T00:28:08.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:28:07 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T00:28:08.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:28:07 smithi193 ceph-mon[36878]: Health check failed: 1 MDSs report slow requests (MDS_SLOW_REQUEST) 2024-03-24T00:28:08.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:28:07 smithi193 ceph-mon[36878]: mds.? [v2:172.21.15.189:6838/4074484168,v1:172.21.15.189:6839/4074484168] up:active 2024-03-24T00:28:08.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:28:07 smithi193 ceph-mon[36878]: fsmap cephfs:5 3 up:standby-replay 5 up:active 2024-03-24T00:28:08.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:28:07 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T00:28:08.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:28:07 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T00:28:08.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:28:07 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T00:28:09.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:28:08 smithi118 ceph-mon[31535]: pgmap v807: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 5 op/s 2024-03-24T00:28:09.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:28:08 smithi118 ceph-mon[31535]: pgmap v808: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 5 op/s 2024-03-24T00:28:09.203 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:28:08 smithi189 ceph-mon[36187]: pgmap v807: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 5 op/s 2024-03-24T00:28:09.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:28:08 smithi189 ceph-mon[36187]: pgmap v808: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 5 op/s 2024-03-24T00:28:09.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:28:08 smithi193 ceph-mon[36878]: pgmap v807: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 5 op/s 2024-03-24T00:28:09.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:28:08 smithi193 ceph-mon[36878]: pgmap v808: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 5 op/s 2024-03-24T00:28:11.160 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:28:10 smithi118 ceph-mon[31535]: pgmap v809: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-03-24T00:28:11.160 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:28:10 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 186.617583 secs 2024-03-24T00:28:11.203 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:28:10 smithi189 ceph-mon[36187]: pgmap v809: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-03-24T00:28:11.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:28:10 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 186.617583 secs 2024-03-24T00:28:11.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:28:10 smithi193 ceph-mon[36878]: pgmap v809: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-03-24T00:28:11.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:28:10 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 186.617583 secs 2024-03-24T00:28:13.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:28:12 smithi118 ceph-mon[31535]: pgmap v810: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-03-24T00:28:13.203 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:28:12 smithi189 ceph-mon[36187]: pgmap v810: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-03-24T00:28:13.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:28:12 smithi193 ceph-mon[36878]: pgmap v810: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-03-24T00:28:15.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:28:14 smithi118 ceph-mon[31535]: pgmap v811: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-03-24T00:28:15.203 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:28:14 smithi189 ceph-mon[36187]: pgmap v811: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-03-24T00:28:15.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:28:14 smithi193 ceph-mon[36878]: pgmap v811: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-03-24T00:28:15.751 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:28:15.750+0000 7f81ee000640 1 -- 172.21.15.118:0/1693767697 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f81c80034f0 con 0x7f81f81052d0 2024-03-24T00:28:15.752 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:28:15.751+0000 7f81ef400640 1 -- 172.21.15.118:0/1693767697 <== mon.2 v2:172.21.15.189:3300/0 6 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (secure 0 0 0) 0x7f81c80034f0 con 0x7f81f81052d0 2024-03-24T00:28:15.781 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:28:15.780+0000 7f81db400640 1 -- 172.21.15.118:0/659686424 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f81c40034c0 con 0x5622651a13f0 2024-03-24T00:28:15.781 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:28:15.780+0000 7f81e4c00640 1 -- 172.21.15.118:0/659686424 <== mon.1 v2:172.21.15.193:3300/0 6 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 409+0+0 (secure 0 0 0) 0x7f81c40034c0 con 0x5622651a13f0 2024-03-24T00:28:16.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:28:15 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 191.617660 secs 2024-03-24T00:28:16.203 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:28:15 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 191.617660 secs 2024-03-24T00:28:16.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:28:15 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 191.617660 secs 2024-03-24T00:28:17.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:28:16 smithi118 ceph-mon[31535]: pgmap v812: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-03-24T00:28:17.203 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:28:16 smithi189 ceph-mon[36187]: pgmap v812: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-03-24T00:28:17.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:28:16 smithi193 ceph-mon[36878]: pgmap v812: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-03-24T00:28:19.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:28:18 smithi118 ceph-mon[31535]: pgmap v813: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 5 op/s 2024-03-24T00:28:19.203 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:28:18 smithi189 ceph-mon[36187]: pgmap v813: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 5 op/s 2024-03-24T00:28:19.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:28:18 smithi193 ceph-mon[36878]: pgmap v813: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 5 op/s 2024-03-24T00:28:21.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:28:20 smithi118 ceph-mon[31535]: pgmap v814: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-24T00:28:21.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:28:20 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 196.617714 secs 2024-03-24T00:28:21.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:28:20 smithi189 ceph-mon[36187]: pgmap v814: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-24T00:28:21.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:28:20 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 196.617714 secs 2024-03-24T00:28:21.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:28:20 smithi193 ceph-mon[36878]: pgmap v814: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-24T00:28:21.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:28:20 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 196.617714 secs 2024-03-24T00:28:23.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:28:22 smithi118 ceph-mon[31535]: pgmap v815: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:28:23.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:28:22 smithi189 ceph-mon[36187]: pgmap v815: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:28:23.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:28:22 smithi193 ceph-mon[36878]: pgmap v815: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:28:25.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:28:24 smithi118 ceph-mon[31535]: pgmap v816: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T00:28:25.203 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:28:24 smithi189 ceph-mon[36187]: pgmap v816: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T00:28:25.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:28:24 smithi193 ceph-mon[36878]: pgmap v816: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T00:28:26.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:28:25 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 201.617775 secs 2024-03-24T00:28:26.203 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:28:25 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 201.617775 secs 2024-03-24T00:28:26.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:28:25 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 201.617775 secs 2024-03-24T00:28:27.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:28:26 smithi118 ceph-mon[31535]: pgmap v817: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T00:28:27.203 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:28:26 smithi189 ceph-mon[36187]: pgmap v817: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T00:28:27.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:28:26 smithi193 ceph-mon[36878]: pgmap v817: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T00:28:29.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:28:28 smithi118 ceph-mon[31535]: pgmap v818: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:28:29.203 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:28:28 smithi189 ceph-mon[36187]: pgmap v818: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:28:29.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:28:28 smithi193 ceph-mon[36878]: pgmap v818: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:28:31.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:28:30 smithi118 ceph-mon[31535]: pgmap v819: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-24T00:28:31.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:28:30 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 206.617861 secs 2024-03-24T00:28:31.203 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:28:30 smithi189 ceph-mon[36187]: pgmap v819: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-24T00:28:31.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:28:30 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 206.617861 secs 2024-03-24T00:28:31.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:28:30 smithi193 ceph-mon[36878]: pgmap v819: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-24T00:28:31.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:28:30 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 206.617861 secs 2024-03-24T00:28:32.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:28:31 smithi118 ceph-mon[31535]: pgmap v820: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:28:32.453 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:28:31 smithi189 ceph-mon[36187]: pgmap v820: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:28:32.460 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:28:31 smithi193 ceph-mon[36878]: pgmap v820: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:28:34.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:28:34 smithi118 ceph-mon[31535]: pgmap v821: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T00:28:34.704 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:28:34 smithi189 ceph-mon[36187]: pgmap v821: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T00:28:34.710 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:28:34 smithi193 ceph-mon[36878]: pgmap v821: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T00:28:35.604 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:28:35 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 211.617922 secs 2024-03-24T00:28:35.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:28:35 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 211.617922 secs 2024-03-24T00:28:35.710 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:28:35 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 211.617922 secs 2024-03-24T00:28:36.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:28:36 smithi118 ceph-mon[31535]: pgmap v822: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T00:28:36.703 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:28:36 smithi189 ceph-mon[36187]: pgmap v822: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T00:28:36.710 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:28:36 smithi193 ceph-mon[36878]: pgmap v822: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T00:28:38.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:28:38 smithi118 ceph-mon[31535]: pgmap v823: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:28:38.703 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:28:38 smithi189 ceph-mon[36187]: pgmap v823: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:28:38.710 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:28:38 smithi193 ceph-mon[36878]: pgmap v823: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:28:40.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:28:40 smithi118 ceph-mon[31535]: pgmap v824: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-24T00:28:40.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:28:40 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 216.618013 secs 2024-03-24T00:28:40.703 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:28:40 smithi189 ceph-mon[36187]: pgmap v824: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-24T00:28:40.704 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:28:40 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 216.618013 secs 2024-03-24T00:28:40.710 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:28:40 smithi193 ceph-mon[36878]: pgmap v824: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-24T00:28:40.710 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:28:40 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 216.618013 secs 2024-03-24T00:28:42.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:28:42 smithi118 ceph-mon[31535]: pgmap v825: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:28:42.703 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:28:42 smithi189 ceph-mon[36187]: pgmap v825: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:28:42.710 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:28:42 smithi193 ceph-mon[36878]: pgmap v825: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:28:44.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:28:44 smithi118 ceph-mon[31535]: pgmap v826: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T00:28:44.704 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:28:44 smithi189 ceph-mon[36187]: pgmap v826: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T00:28:44.710 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:28:44 smithi193 ceph-mon[36878]: pgmap v826: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T00:28:45.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:28:45 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 221.618095 secs 2024-03-24T00:28:45.703 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:28:45 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 221.618095 secs 2024-03-24T00:28:45.710 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:28:45 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 221.618095 secs 2024-03-24T00:28:46.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:28:46 smithi118 ceph-mon[31535]: pgmap v827: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T00:28:46.704 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:28:46 smithi189 ceph-mon[36187]: pgmap v827: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T00:28:46.710 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:28:46 smithi193 ceph-mon[36878]: pgmap v827: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T00:28:48.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:28:48 smithi118 ceph-mon[31535]: pgmap v828: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:28:48.703 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:28:48 smithi189 ceph-mon[36187]: pgmap v828: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:28:48.710 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:28:48 smithi193 ceph-mon[36878]: pgmap v828: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:28:50.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:28:50 smithi118 ceph-mon[31535]: pgmap v829: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-24T00:28:50.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:28:50 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 226.618184 secs 2024-03-24T00:28:50.703 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:28:50 smithi189 ceph-mon[36187]: pgmap v829: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-24T00:28:50.704 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:28:50 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 226.618184 secs 2024-03-24T00:28:50.710 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:28:50 smithi193 ceph-mon[36878]: pgmap v829: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-24T00:28:50.710 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:28:50 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 226.618184 secs 2024-03-24T00:28:52.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:28:52 smithi118 ceph-mon[31535]: pgmap v830: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:28:52.705 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:28:52 smithi189 ceph-mon[36187]: pgmap v830: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:28:52.710 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:28:52 smithi193 ceph-mon[36878]: pgmap v830: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:28:54.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:28:54 smithi118 ceph-mon[31535]: pgmap v831: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T00:28:54.703 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:28:54 smithi189 ceph-mon[36187]: pgmap v831: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T00:28:54.710 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:28:54 smithi193 ceph-mon[36878]: pgmap v831: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T00:28:55.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:28:55 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 231.618269 secs 2024-03-24T00:28:55.703 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:28:55 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 231.618269 secs 2024-03-24T00:28:55.710 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:28:55 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 231.618269 secs 2024-03-24T00:28:56.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:28:56 smithi118 ceph-mon[31535]: pgmap v832: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T00:28:56.703 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:28:56 smithi189 ceph-mon[36187]: pgmap v832: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T00:28:56.710 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:28:56 smithi193 ceph-mon[36878]: pgmap v832: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T00:28:58.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:28:58 smithi118 ceph-mon[31535]: pgmap v833: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:28:58.703 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:28:58 smithi189 ceph-mon[36187]: pgmap v833: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:28:58.710 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:28:58 smithi193 ceph-mon[36878]: pgmap v833: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:29:00.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:29:00 smithi118 ceph-mon[31535]: pgmap v834: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-24T00:29:00.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:29:00 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 236.618352 secs 2024-03-24T00:29:00.703 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:29:00 smithi189 ceph-mon[36187]: pgmap v834: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-24T00:29:00.704 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:29:00 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 236.618352 secs 2024-03-24T00:29:00.710 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:29:00 smithi193 ceph-mon[36878]: pgmap v834: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-24T00:29:00.710 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:29:00 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 236.618352 secs 2024-03-24T00:29:02.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:29:02 smithi118 ceph-mon[31535]: pgmap v835: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:29:02.703 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:29:02 smithi189 ceph-mon[36187]: pgmap v835: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:29:02.710 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:29:02 smithi193 ceph-mon[36878]: pgmap v835: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:29:04.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:29:04 smithi118 ceph-mon[31535]: pgmap v836: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T00:29:04.703 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:29:04 smithi189 ceph-mon[36187]: pgmap v836: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T00:29:04.710 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:29:04 smithi193 ceph-mon[36878]: pgmap v836: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T00:29:05.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:29:05 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 241.618440 secs 2024-03-24T00:29:05.703 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:29:05 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 241.618440 secs 2024-03-24T00:29:05.710 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:29:05 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 241.618440 secs 2024-03-24T00:29:06.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:29:06 smithi118 ceph-mon[31535]: pgmap v837: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T00:29:06.703 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:29:06 smithi189 ceph-mon[36187]: pgmap v837: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T00:29:06.710 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:29:06 smithi193 ceph-mon[36878]: pgmap v837: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T00:29:08.704 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:29:08 smithi189 ceph-mon[36187]: pgmap v838: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:29:08.704 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:29:08 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T00:29:08.710 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:29:08 smithi193 ceph-mon[36878]: pgmap v838: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:29:08.711 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:29:08 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T00:29:08.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:29:08 smithi118 ceph-mon[31535]: pgmap v838: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:29:08.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:29:08 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T00:29:09.711 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:29:09 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T00:29:09.711 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:29:09 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T00:29:09.711 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:29:09 smithi193 ceph-mon[36878]: pgmap v839: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 6 op/s 2024-03-24T00:29:09.711 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:29:09 smithi193 ceph-mon[36878]: pgmap v840: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 4 op/s 2024-03-24T00:29:09.711 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:29:09 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T00:29:09.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:29:09 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T00:29:09.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:29:09 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T00:29:09.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:29:09 smithi118 ceph-mon[31535]: pgmap v839: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 6 op/s 2024-03-24T00:29:09.947 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:29:09 smithi118 ceph-mon[31535]: pgmap v840: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 4 op/s 2024-03-24T00:29:09.947 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:29:09 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T00:29:09.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:29:09 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T00:29:09.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:29:09 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T00:29:09.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:29:09 smithi189 ceph-mon[36187]: pgmap v839: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 6 op/s 2024-03-24T00:29:09.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:29:09 smithi189 ceph-mon[36187]: pgmap v840: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 4 op/s 2024-03-24T00:29:09.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:29:09 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T00:29:10.711 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:29:10 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 246.618501 secs 2024-03-24T00:29:10.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:29:10 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 246.618501 secs 2024-03-24T00:29:10.953 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:29:10 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 246.618501 secs 2024-03-24T00:29:11.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:29:11 smithi118 ceph-mon[31535]: pgmap v841: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 6 op/s 2024-03-24T00:29:11.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:29:11 smithi189 ceph-mon[36187]: pgmap v841: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 6 op/s 2024-03-24T00:29:11.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:29:11 smithi193 ceph-mon[36878]: pgmap v841: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 6 op/s 2024-03-24T00:29:13.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:29:13 smithi118 ceph-mon[31535]: pgmap v842: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 5 op/s 2024-03-24T00:29:13.453 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:29:13 smithi189 ceph-mon[36187]: pgmap v842: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 5 op/s 2024-03-24T00:29:13.460 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:29:13 smithi193 ceph-mon[36878]: pgmap v842: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 5 op/s 2024-03-24T00:29:15.250 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:29:15 smithi118 ceph-mon[31535]: pgmap v843: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 6 op/s 2024-03-24T00:29:15.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:29:15 smithi189 ceph-mon[36187]: pgmap v843: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 6 op/s 2024-03-24T00:29:15.460 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:29:15 smithi193 ceph-mon[36878]: pgmap v843: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 6 op/s 2024-03-24T00:29:16.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:29:16 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 251.618587 secs 2024-03-24T00:29:16.453 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:29:16 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 251.618587 secs 2024-03-24T00:29:16.460 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:29:16 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 251.618587 secs 2024-03-24T00:29:17.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:29:17 smithi118 ceph-mon[31535]: pgmap v844: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 6 op/s 2024-03-24T00:29:17.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:29:17 smithi189 ceph-mon[36187]: pgmap v844: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 6 op/s 2024-03-24T00:29:17.462 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:29:17 smithi193 ceph-mon[36878]: pgmap v844: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 6 op/s 2024-03-24T00:29:19.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:29:19 smithi118 ceph-mon[31535]: pgmap v845: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 5 op/s 2024-03-24T00:29:19.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:29:19 smithi189 ceph-mon[36187]: pgmap v845: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 5 op/s 2024-03-24T00:29:19.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:29:19 smithi193 ceph-mon[36878]: pgmap v845: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 5 op/s 2024-03-24T00:29:20.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:29:20 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 256.618646 secs 2024-03-24T00:29:20.704 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:29:20 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 256.618646 secs 2024-03-24T00:29:20.710 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:29:20 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 256.618646 secs 2024-03-24T00:29:21.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:29:21 smithi118 ceph-mon[31535]: pgmap v846: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T00:29:21.703 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:29:21 smithi189 ceph-mon[36187]: pgmap v846: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T00:29:21.710 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:29:21 smithi193 ceph-mon[36878]: pgmap v846: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T00:29:23.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:29:23 smithi118 ceph-mon[31535]: pgmap v847: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 5 op/s 2024-03-24T00:29:23.453 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:29:23 smithi189 ceph-mon[36187]: pgmap v847: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 5 op/s 2024-03-24T00:29:23.460 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:29:23 smithi193 ceph-mon[36878]: pgmap v847: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 5 op/s 2024-03-24T00:29:25.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:29:25 smithi118 ceph-mon[31535]: pgmap v848: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T00:29:25.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:29:25 smithi189 ceph-mon[36187]: pgmap v848: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T00:29:25.460 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:29:25 smithi193 ceph-mon[36878]: pgmap v848: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T00:29:26.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:29:26 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 261.618682 secs 2024-03-24T00:29:26.453 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:29:26 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 261.618682 secs 2024-03-24T00:29:26.460 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:29:26 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 261.618682 secs 2024-03-24T00:29:27.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:29:27 smithi118 ceph-mon[31535]: pgmap v849: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:29:27.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:29:27 smithi189 ceph-mon[36187]: pgmap v849: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:29:27.460 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:29:27 smithi193 ceph-mon[36878]: pgmap v849: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:29:29.249 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:29:29 smithi118 ceph-mon[31535]: pgmap v850: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T00:29:29.453 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:29:29 smithi189 ceph-mon[36187]: pgmap v850: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T00:29:29.460 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:29:29 smithi193 ceph-mon[36878]: pgmap v850: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T00:29:30.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:29:30 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 266.618774 secs 2024-03-24T00:29:30.703 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:29:30 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 266.618774 secs 2024-03-24T00:29:30.710 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:29:30 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 266.618774 secs 2024-03-24T00:29:31.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:29:31 smithi118 ceph-mon[31535]: pgmap v851: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T00:29:31.703 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:29:31 smithi189 ceph-mon[36187]: pgmap v851: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T00:29:31.710 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:29:31 smithi193 ceph-mon[36878]: pgmap v851: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T00:29:33.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:29:33 smithi118 ceph-mon[31535]: pgmap v852: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 5 op/s 2024-03-24T00:29:33.453 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:29:33 smithi189 ceph-mon[36187]: pgmap v852: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 5 op/s 2024-03-24T00:29:33.460 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:29:33 smithi193 ceph-mon[36878]: pgmap v852: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 5 op/s 2024-03-24T00:29:35.255 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:29:35 smithi189 ceph-mon[36187]: pgmap v853: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T00:29:35.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:29:35 smithi118 ceph-mon[31535]: pgmap v853: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T00:29:35.460 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:29:35 smithi193 ceph-mon[36878]: pgmap v853: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T00:29:36.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:29:36 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 271.618863 secs 2024-03-24T00:29:36.453 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:29:36 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 271.618863 secs 2024-03-24T00:29:36.460 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:29:36 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 271.618863 secs 2024-03-24T00:29:37.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:29:37 smithi118 ceph-mon[31535]: pgmap v854: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:29:37.453 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:29:37 smithi189 ceph-mon[36187]: pgmap v854: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:29:37.460 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:29:37 smithi193 ceph-mon[36878]: pgmap v854: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:29:39.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:29:39 smithi118 ceph-mon[31535]: pgmap v855: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T00:29:39.453 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:29:39 smithi189 ceph-mon[36187]: pgmap v855: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T00:29:39.460 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:29:39 smithi193 ceph-mon[36878]: pgmap v855: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T00:29:40.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:29:40 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 276.618951 secs 2024-03-24T00:29:40.703 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:29:40 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 276.618951 secs 2024-03-24T00:29:40.710 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:29:40 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 276.618951 secs 2024-03-24T00:29:41.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:29:41 smithi118 ceph-mon[31535]: pgmap v856: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T00:29:41.703 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:29:41 smithi189 ceph-mon[36187]: pgmap v856: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T00:29:41.710 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:29:41 smithi193 ceph-mon[36878]: pgmap v856: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T00:29:43.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:29:43 smithi118 ceph-mon[31535]: pgmap v857: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 5 op/s 2024-03-24T00:29:43.453 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:29:43 smithi189 ceph-mon[36187]: pgmap v857: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 5 op/s 2024-03-24T00:29:43.460 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:29:43 smithi193 ceph-mon[36878]: pgmap v857: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 5 op/s 2024-03-24T00:29:45.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:29:45 smithi118 ceph-mon[31535]: pgmap v858: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T00:29:45.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:29:45 smithi189 ceph-mon[36187]: pgmap v858: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T00:29:45.460 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:29:45 smithi193 ceph-mon[36878]: pgmap v858: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T00:29:46.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:29:46 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 281.619032 secs 2024-03-24T00:29:46.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:29:46 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 281.619032 secs 2024-03-24T00:29:46.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:29:46 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 281.619032 secs 2024-03-24T00:29:48.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:29:47 smithi118 ceph-mon[31535]: pgmap v859: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:29:48.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:29:47 smithi189 ceph-mon[36187]: pgmap v859: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:29:48.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:29:47 smithi193 ceph-mon[36878]: pgmap v859: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:29:49.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:29:49 smithi118 ceph-mon[31535]: pgmap v860: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T00:29:49.453 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:29:49 smithi189 ceph-mon[36187]: pgmap v860: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T00:29:49.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:29:49 smithi193 ceph-mon[36878]: pgmap v860: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T00:29:50.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:29:50 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 286.619120 secs 2024-03-24T00:29:50.704 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:29:50 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 286.619120 secs 2024-03-24T00:29:50.710 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:29:50 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 286.619120 secs 2024-03-24T00:29:51.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:29:51 smithi118 ceph-mon[31535]: pgmap v861: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T00:29:51.703 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:29:51 smithi189 ceph-mon[36187]: pgmap v861: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T00:29:51.710 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:29:51 smithi193 ceph-mon[36878]: pgmap v861: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T00:29:53.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:29:53 smithi118 ceph-mon[31535]: pgmap v862: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 5 op/s 2024-03-24T00:29:53.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:29:53 smithi189 ceph-mon[36187]: pgmap v862: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 5 op/s 2024-03-24T00:29:53.460 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:29:53 smithi193 ceph-mon[36878]: pgmap v862: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 5 op/s 2024-03-24T00:29:55.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:29:55 smithi118 ceph-mon[31535]: pgmap v863: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T00:29:55.453 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:29:55 smithi189 ceph-mon[36187]: pgmap v863: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T00:29:55.460 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:29:55 smithi193 ceph-mon[36878]: pgmap v863: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T00:29:56.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:29:56 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 291.619212 secs 2024-03-24T00:29:56.453 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:29:56 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 291.619212 secs 2024-03-24T00:29:56.460 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:29:56 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 291.619212 secs 2024-03-24T00:29:57.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:29:57 smithi118 ceph-mon[31535]: pgmap v864: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:29:57.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:29:57 smithi189 ceph-mon[36187]: pgmap v864: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:29:57.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:29:57 smithi193 ceph-mon[36878]: pgmap v864: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:29:59.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:29:59 smithi118 ceph-mon[31535]: pgmap v865: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T00:29:59.453 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:29:59 smithi189 ceph-mon[36187]: pgmap v865: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T00:29:59.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:29:59 smithi193 ceph-mon[36878]: pgmap v865: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T00:30:00.351 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:30:00 smithi118 ceph-mon[31535]: Health detail: HEALTH_WARN 6 failed cephadm daemon(s); 1 MDSs report slow requests 2024-03-24T00:30:00.351 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:30:00 smithi118 ceph-mon[31535]: [WRN] CEPHADM_FAILED_DAEMON: 6 failed cephadm daemon(s) 2024-03-24T00:30:00.351 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:30:00 smithi118 ceph-mon[31535]: daemon mds.j on smithi118 is in error state 2024-03-24T00:30:00.351 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:30:00 smithi118 ceph-mon[31535]: daemon mds.d on smithi118 is in error state 2024-03-24T00:30:00.351 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:30:00 smithi118 ceph-mon[31535]: daemon mds.g on smithi118 is in error state 2024-03-24T00:30:00.351 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:30:00 smithi118 ceph-mon[31535]: daemon mds.h on smithi189 is in error state 2024-03-24T00:30:00.351 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:30:00 smithi118 ceph-mon[31535]: daemon mds.l on smithi193 is in error state 2024-03-24T00:30:00.351 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:30:00 smithi118 ceph-mon[31535]: daemon mds.f on smithi193 is in error state 2024-03-24T00:30:00.351 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:30:00 smithi118 ceph-mon[31535]: [WRN] MDS_SLOW_REQUEST: 1 MDSs report slow requests 2024-03-24T00:30:00.351 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:30:00 smithi118 ceph-mon[31535]: mds.b(mds.0): 3 slow requests are blocked > 180 secs 2024-03-24T00:30:00.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:30:00 smithi189 ceph-mon[36187]: Health detail: HEALTH_WARN 6 failed cephadm daemon(s); 1 MDSs report slow requests 2024-03-24T00:30:00.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:30:00 smithi189 ceph-mon[36187]: [WRN] CEPHADM_FAILED_DAEMON: 6 failed cephadm daemon(s) 2024-03-24T00:30:00.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:30:00 smithi189 ceph-mon[36187]: daemon mds.j on smithi118 is in error state 2024-03-24T00:30:00.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:30:00 smithi189 ceph-mon[36187]: daemon mds.d on smithi118 is in error state 2024-03-24T00:30:00.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:30:00 smithi189 ceph-mon[36187]: daemon mds.g on smithi118 is in error state 2024-03-24T00:30:00.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:30:00 smithi189 ceph-mon[36187]: daemon mds.h on smithi189 is in error state 2024-03-24T00:30:00.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:30:00 smithi189 ceph-mon[36187]: daemon mds.l on smithi193 is in error state 2024-03-24T00:30:00.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:30:00 smithi189 ceph-mon[36187]: daemon mds.f on smithi193 is in error state 2024-03-24T00:30:00.455 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:30:00 smithi189 ceph-mon[36187]: [WRN] MDS_SLOW_REQUEST: 1 MDSs report slow requests 2024-03-24T00:30:00.455 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:30:00 smithi189 ceph-mon[36187]: mds.b(mds.0): 3 slow requests are blocked > 180 secs 2024-03-24T00:30:00.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:30:00 smithi193 ceph-mon[36878]: Health detail: HEALTH_WARN 6 failed cephadm daemon(s); 1 MDSs report slow requests 2024-03-24T00:30:00.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:30:00 smithi193 ceph-mon[36878]: [WRN] CEPHADM_FAILED_DAEMON: 6 failed cephadm daemon(s) 2024-03-24T00:30:00.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:30:00 smithi193 ceph-mon[36878]: daemon mds.j on smithi118 is in error state 2024-03-24T00:30:00.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:30:00 smithi193 ceph-mon[36878]: daemon mds.d on smithi118 is in error state 2024-03-24T00:30:00.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:30:00 smithi193 ceph-mon[36878]: daemon mds.g on smithi118 is in error state 2024-03-24T00:30:00.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:30:00 smithi193 ceph-mon[36878]: daemon mds.h on smithi189 is in error state 2024-03-24T00:30:00.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:30:00 smithi193 ceph-mon[36878]: daemon mds.l on smithi193 is in error state 2024-03-24T00:30:00.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:30:00 smithi193 ceph-mon[36878]: daemon mds.f on smithi193 is in error state 2024-03-24T00:30:00.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:30:00 smithi193 ceph-mon[36878]: [WRN] MDS_SLOW_REQUEST: 1 MDSs report slow requests 2024-03-24T00:30:00.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:30:00 smithi193 ceph-mon[36878]: mds.b(mds.0): 3 slow requests are blocked > 180 secs 2024-03-24T00:30:01.447 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:30:01 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 296.619301 secs 2024-03-24T00:30:01.447 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:30:01 smithi118 ceph-mon[31535]: pgmap v866: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T00:30:01.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:30:01 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 296.619301 secs 2024-03-24T00:30:01.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:30:01 smithi189 ceph-mon[36187]: pgmap v866: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T00:30:01.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:30:01 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 296.619301 secs 2024-03-24T00:30:01.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:30:01 smithi193 ceph-mon[36878]: pgmap v866: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T00:30:03.263 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:30:03 smithi193 ceph-mon[36878]: pgmap v867: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 5 op/s 2024-03-24T00:30:03.288 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:30:03 smithi118 ceph-mon[31535]: pgmap v867: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 5 op/s 2024-03-24T00:30:03.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:30:03 smithi189 ceph-mon[36187]: pgmap v867: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 5 op/s 2024-03-24T00:30:05.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:30:05 smithi118 ceph-mon[31535]: pgmap v868: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T00:30:05.453 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:30:05 smithi189 ceph-mon[36187]: pgmap v868: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T00:30:05.460 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:30:05 smithi193 ceph-mon[36878]: pgmap v868: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T00:30:06.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:30:06 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 301.619397 secs 2024-03-24T00:30:06.453 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:30:06 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 301.619397 secs 2024-03-24T00:30:06.460 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:30:06 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 301.619397 secs 2024-03-24T00:30:07.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:30:07 smithi118 ceph-mon[31535]: pgmap v869: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:30:07.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:30:07 smithi189 ceph-mon[36187]: pgmap v869: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:30:07.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:30:07 smithi193 ceph-mon[36878]: pgmap v869: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:30:09.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:30:09 smithi118 ceph-mon[31535]: pgmap v870: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T00:30:09.453 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:30:09 smithi189 ceph-mon[36187]: pgmap v870: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T00:30:09.460 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:30:09 smithi193 ceph-mon[36878]: pgmap v870: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T00:30:10.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:30:10 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T00:30:10.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:30:10 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T00:30:10.460 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:30:10 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T00:30:11.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:30:11 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 306.619460 secs 2024-03-24T00:30:11.447 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:30:11 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T00:30:11.447 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:30:11 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T00:30:11.447 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:30:11 smithi118 ceph-mon[31535]: pgmap v871: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 6 op/s 2024-03-24T00:30:11.447 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:30:11 smithi118 ceph-mon[31535]: pgmap v872: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 6 op/s 2024-03-24T00:30:11.447 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:30:11 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T00:30:11.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:30:11 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 306.619460 secs 2024-03-24T00:30:11.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:30:11 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T00:30:11.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:30:11 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T00:30:11.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:30:11 smithi189 ceph-mon[36187]: pgmap v871: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 6 op/s 2024-03-24T00:30:11.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:30:11 smithi189 ceph-mon[36187]: pgmap v872: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 6 op/s 2024-03-24T00:30:11.455 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:30:11 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T00:30:11.460 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:30:11 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 306.619460 secs 2024-03-24T00:30:11.460 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:30:11 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T00:30:11.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:30:11 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T00:30:11.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:30:11 smithi193 ceph-mon[36878]: pgmap v871: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 6 op/s 2024-03-24T00:30:11.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:30:11 smithi193 ceph-mon[36878]: pgmap v872: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 6 op/s 2024-03-24T00:30:11.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:30:11 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T00:30:13.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:30:13 smithi118 ceph-mon[31535]: pgmap v873: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 5 op/s 2024-03-24T00:30:13.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:30:13 smithi189 ceph-mon[36187]: pgmap v873: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 5 op/s 2024-03-24T00:30:13.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:30:13 smithi193 ceph-mon[36878]: pgmap v873: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 5 op/s 2024-03-24T00:30:15.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:30:15 smithi118 ceph-mon[31535]: pgmap v874: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 6 op/s 2024-03-24T00:30:15.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:30:15 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 311.619519 secs 2024-03-24T00:30:15.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:30:15 smithi189 ceph-mon[36187]: pgmap v874: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 6 op/s 2024-03-24T00:30:15.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:30:15 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 311.619519 secs 2024-03-24T00:30:15.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:30:15 smithi193 ceph-mon[36878]: pgmap v874: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 6 op/s 2024-03-24T00:30:15.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:30:15 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 311.619519 secs 2024-03-24T00:30:17.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:30:17 smithi118 ceph-mon[31535]: pgmap v875: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 6 op/s 2024-03-24T00:30:17.953 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:30:17 smithi189 ceph-mon[36187]: pgmap v875: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 6 op/s 2024-03-24T00:30:17.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:30:17 smithi193 ceph-mon[36878]: pgmap v875: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 6 op/s 2024-03-24T00:30:19.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:30:19 smithi118 ceph-mon[31535]: pgmap v876: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 5 op/s 2024-03-24T00:30:19.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:30:19 smithi189 ceph-mon[36187]: pgmap v876: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 5 op/s 2024-03-24T00:30:19.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:30:19 smithi193 ceph-mon[36878]: pgmap v876: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 5 op/s 2024-03-24T00:30:20.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:30:20 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 316.619608 secs 2024-03-24T00:30:20.953 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:30:20 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 316.619608 secs 2024-03-24T00:30:20.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:30:20 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 316.619608 secs 2024-03-24T00:30:21.947 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:30:21 smithi118 ceph-mon[31535]: pgmap v877: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 5 op/s 2024-03-24T00:30:21.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:30:21 smithi189 ceph-mon[36187]: pgmap v877: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 5 op/s 2024-03-24T00:30:21.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:30:21 smithi193 ceph-mon[36878]: pgmap v877: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 5 op/s 2024-03-24T00:30:23.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:30:23 smithi118 ceph-mon[31535]: pgmap v878: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T00:30:23.953 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:30:23 smithi189 ceph-mon[36187]: pgmap v878: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T00:30:23.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:30:23 smithi193 ceph-mon[36878]: pgmap v878: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T00:30:25.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:30:25 smithi118 ceph-mon[31535]: pgmap v879: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-03-24T00:30:25.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:30:25 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 321.619692 secs 2024-03-24T00:30:25.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:30:25 smithi189 ceph-mon[36187]: pgmap v879: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-03-24T00:30:25.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:30:25 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 321.619692 secs 2024-03-24T00:30:25.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:30:25 smithi193 ceph-mon[36878]: pgmap v879: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-03-24T00:30:25.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:30:25 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 321.619692 secs 2024-03-24T00:30:27.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:30:27 smithi118 ceph-mon[31535]: pgmap v880: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:30:27.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:30:27 smithi189 ceph-mon[36187]: pgmap v880: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:30:27.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:30:27 smithi193 ceph-mon[36878]: pgmap v880: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:30:29.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:30:29 smithi118 ceph-mon[31535]: pgmap v881: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T00:30:29.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:30:29 smithi189 ceph-mon[36187]: pgmap v881: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T00:30:29.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:30:29 smithi193 ceph-mon[36878]: pgmap v881: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T00:30:30.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:30:30 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 326.619753 secs 2024-03-24T00:30:30.953 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:30:30 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 326.619753 secs 2024-03-24T00:30:30.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:30:30 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 326.619753 secs 2024-03-24T00:30:31.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:30:31 smithi118 ceph-mon[31535]: pgmap v882: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T00:30:31.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:30:31 smithi189 ceph-mon[36187]: pgmap v882: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T00:30:31.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:30:31 smithi193 ceph-mon[36878]: pgmap v882: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T00:30:33.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:30:33 smithi118 ceph-mon[31535]: pgmap v883: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T00:30:33.953 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:30:33 smithi189 ceph-mon[36187]: pgmap v883: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T00:30:33.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:30:33 smithi193 ceph-mon[36878]: pgmap v883: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T00:30:35.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:30:35 smithi118 ceph-mon[31535]: pgmap v884: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-03-24T00:30:35.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:30:35 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 331.619812 secs 2024-03-24T00:30:35.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:30:35 smithi189 ceph-mon[36187]: pgmap v884: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-03-24T00:30:35.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:30:35 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 331.619812 secs 2024-03-24T00:30:35.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:30:35 smithi193 ceph-mon[36878]: pgmap v884: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-03-24T00:30:35.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:30:35 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 331.619812 secs 2024-03-24T00:30:38.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:30:37 smithi118 ceph-mon[31535]: pgmap v885: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:30:38.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:30:37 smithi189 ceph-mon[36187]: pgmap v885: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:30:38.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:30:37 smithi193 ceph-mon[36878]: pgmap v885: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:30:40.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:30:39 smithi118 ceph-mon[31535]: pgmap v886: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T00:30:40.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:30:39 smithi189 ceph-mon[36187]: pgmap v886: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T00:30:40.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:30:39 smithi193 ceph-mon[36878]: pgmap v886: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T00:30:41.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:30:40 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 336.619885 secs 2024-03-24T00:30:41.203 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:30:40 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 336.619885 secs 2024-03-24T00:30:41.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:30:40 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 336.619885 secs 2024-03-24T00:30:42.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:30:41 smithi118 ceph-mon[31535]: pgmap v887: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T00:30:42.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:30:41 smithi189 ceph-mon[36187]: pgmap v887: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T00:30:42.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:30:41 smithi193 ceph-mon[36878]: pgmap v887: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T00:30:44.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:30:43 smithi118 ceph-mon[31535]: pgmap v888: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T00:30:44.203 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:30:43 smithi189 ceph-mon[36187]: pgmap v888: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T00:30:44.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:30:43 smithi193 ceph-mon[36878]: pgmap v888: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T00:30:46.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:30:45 smithi118 ceph-mon[31535]: pgmap v889: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-03-24T00:30:46.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:30:45 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 341.619898 secs 2024-03-24T00:30:46.203 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:30:45 smithi189 ceph-mon[36187]: pgmap v889: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-03-24T00:30:46.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:30:45 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 341.619898 secs 2024-03-24T00:30:46.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:30:45 smithi193 ceph-mon[36878]: pgmap v889: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-03-24T00:30:46.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:30:45 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 341.619898 secs 2024-03-24T00:30:48.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:30:47 smithi118 ceph-mon[31535]: pgmap v890: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:30:48.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:30:47 smithi189 ceph-mon[36187]: pgmap v890: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:30:48.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:30:47 smithi193 ceph-mon[36878]: pgmap v890: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:30:50.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:30:49 smithi118 ceph-mon[31535]: pgmap v891: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T00:30:50.203 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:30:49 smithi189 ceph-mon[36187]: pgmap v891: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T00:30:50.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:30:49 smithi193 ceph-mon[36878]: pgmap v891: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T00:30:51.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:30:50 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 346.619985 secs 2024-03-24T00:30:51.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:30:50 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 346.619985 secs 2024-03-24T00:30:51.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:30:50 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 346.619985 secs 2024-03-24T00:30:52.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:30:51 smithi118 ceph-mon[31535]: pgmap v892: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T00:30:52.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:30:51 smithi189 ceph-mon[36187]: pgmap v892: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T00:30:52.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:30:51 smithi193 ceph-mon[36878]: pgmap v892: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T00:30:54.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:30:53 smithi118 ceph-mon[31535]: pgmap v893: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T00:30:54.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:30:53 smithi189 ceph-mon[36187]: pgmap v893: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T00:30:54.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:30:53 smithi193 ceph-mon[36878]: pgmap v893: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T00:30:56.197 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:30:55 smithi118 ceph-mon[31535]: pgmap v894: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-03-24T00:30:56.197 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:30:55 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 351.620000 secs 2024-03-24T00:30:56.203 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:30:55 smithi189 ceph-mon[36187]: pgmap v894: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-03-24T00:30:56.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:30:55 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 351.620000 secs 2024-03-24T00:30:56.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:30:55 smithi193 ceph-mon[36878]: pgmap v894: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-03-24T00:30:56.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:30:55 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 351.620000 secs 2024-03-24T00:30:58.198 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:30:57 smithi118 ceph-mon[31535]: pgmap v895: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-03-24T00:30:58.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:30:57 smithi189 ceph-mon[36187]: pgmap v895: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-03-24T00:30:58.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:30:57 smithi193 ceph-mon[36878]: pgmap v895: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-03-24T00:31:00.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:30:59 smithi118 ceph-mon[31535]: pgmap v896: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-03-24T00:31:00.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:30:59 smithi189 ceph-mon[36187]: pgmap v896: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-03-24T00:31:00.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:30:59 smithi193 ceph-mon[36878]: pgmap v896: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-03-24T00:31:01.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:31:00 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 356.620088 secs 2024-03-24T00:31:01.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:31:00 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 356.620088 secs 2024-03-24T00:31:01.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:31:00 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 356.620088 secs 2024-03-24T00:31:02.197 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:31:01 smithi118 ceph-mon[31535]: pgmap v897: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 6 op/s 2024-03-24T00:31:02.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:31:01 smithi189 ceph-mon[36187]: pgmap v897: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 6 op/s 2024-03-24T00:31:02.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:31:01 smithi193 ceph-mon[36878]: pgmap v897: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 6 op/s 2024-03-24T00:31:04.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:31:03 smithi118 ceph-mon[31535]: pgmap v898: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 5 op/s 2024-03-24T00:31:04.203 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:31:03 smithi189 ceph-mon[36187]: pgmap v898: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 5 op/s 2024-03-24T00:31:04.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:31:03 smithi193 ceph-mon[36878]: pgmap v898: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 5 op/s 2024-03-24T00:31:06.197 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:31:05 smithi118 ceph-mon[31535]: pgmap v899: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-03-24T00:31:06.197 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:31:05 smithi118 ceph-mon[31535]: 3 slow requests, 3 included below; oldest blocked for > 361.620152 secs 2024-03-24T00:31:06.197 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:31:05 smithi118 ceph-mon[31535]: slow request 361.620151 seconds old, received at 2024-03-24T00:25:03.572253+0000: peer_request:client.15159:154789 currently acquired locks 2024-03-24T00:31:06.197 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:31:05 smithi118 ceph-mon[31535]: slow request 361.568905 seconds old, received at 2024-03-24T00:25:03.623500+0000: peer_request:client.15159:158090 currently acquired locks 2024-03-24T00:31:06.197 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:31:05 smithi118 ceph-mon[31535]: slow request 361.562226 seconds old, received at 2024-03-24T00:25:03.630178+0000: client_request(client.15159:173196 rename #0x100000010bf/comment-15.xml #0x100000010bf/comment-15.xml.tmp 2024-03-24T00:25:03.629366+0000 caller_uid=1000, caller_gid=1301{6,36,1000,1301,}) currently failed to xlock, waiting 2024-03-24T00:31:06.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:31:05 smithi189 ceph-mon[36187]: pgmap v899: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-03-24T00:31:06.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:31:05 smithi189 ceph-mon[36187]: 3 slow requests, 3 included below; oldest blocked for > 361.620152 secs 2024-03-24T00:31:06.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:31:05 smithi189 ceph-mon[36187]: slow request 361.620151 seconds old, received at 2024-03-24T00:25:03.572253+0000: peer_request:client.15159:154789 currently acquired locks 2024-03-24T00:31:06.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:31:05 smithi189 ceph-mon[36187]: slow request 361.568905 seconds old, received at 2024-03-24T00:25:03.623500+0000: peer_request:client.15159:158090 currently acquired locks 2024-03-24T00:31:06.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:31:05 smithi189 ceph-mon[36187]: slow request 361.562226 seconds old, received at 2024-03-24T00:25:03.630178+0000: client_request(client.15159:173196 rename #0x100000010bf/comment-15.xml #0x100000010bf/comment-15.xml.tmp 2024-03-24T00:25:03.629366+0000 caller_uid=1000, caller_gid=1301{6,36,1000,1301,}) currently failed to xlock, waiting 2024-03-24T00:31:06.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:31:05 smithi193 ceph-mon[36878]: pgmap v899: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-03-24T00:31:06.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:31:05 smithi193 ceph-mon[36878]: 3 slow requests, 3 included below; oldest blocked for > 361.620152 secs 2024-03-24T00:31:06.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:31:05 smithi193 ceph-mon[36878]: slow request 361.620151 seconds old, received at 2024-03-24T00:25:03.572253+0000: peer_request:client.15159:154789 currently acquired locks 2024-03-24T00:31:06.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:31:05 smithi193 ceph-mon[36878]: slow request 361.568905 seconds old, received at 2024-03-24T00:25:03.623500+0000: peer_request:client.15159:158090 currently acquired locks 2024-03-24T00:31:06.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:31:05 smithi193 ceph-mon[36878]: slow request 361.562226 seconds old, received at 2024-03-24T00:25:03.630178+0000: client_request(client.15159:173196 rename #0x100000010bf/comment-15.xml #0x100000010bf/comment-15.xml.tmp 2024-03-24T00:25:03.629366+0000 caller_uid=1000, caller_gid=1301{6,36,1000,1301,}) currently failed to xlock, waiting 2024-03-24T00:31:08.206 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:31:07 smithi189 ceph-mon[36187]: pgmap v900: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:31:08.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:31:07 smithi193 ceph-mon[36878]: pgmap v900: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:31:08.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:31:07 smithi118 ceph-mon[31535]: pgmap v900: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:31:10.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:31:09 smithi189 ceph-mon[36187]: pgmap v901: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-03-24T00:31:10.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:31:09 smithi193 ceph-mon[36878]: pgmap v901: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-03-24T00:31:10.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:31:09 smithi118 ceph-mon[31535]: pgmap v901: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-03-24T00:31:11.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:31:10 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 366.620213 secs 2024-03-24T00:31:11.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:31:10 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T00:31:11.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:31:10 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 366.620213 secs 2024-03-24T00:31:11.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:31:10 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T00:31:11.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:31:10 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 366.620213 secs 2024-03-24T00:31:11.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:31:10 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T00:31:12.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:31:11 smithi193 ceph-mon[36878]: pgmap v902: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 6 op/s 2024-03-24T00:31:12.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:31:11 smithi118 ceph-mon[31535]: pgmap v902: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 6 op/s 2024-03-24T00:31:12.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:31:11 smithi189 ceph-mon[36187]: pgmap v902: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 6 op/s 2024-03-24T00:31:13.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:31:12 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T00:31:13.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:31:12 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T00:31:13.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:31:12 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T00:31:13.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:31:12 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T00:31:13.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:31:12 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T00:31:13.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:31:12 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T00:31:13.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:31:12 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T00:31:13.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:31:12 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T00:31:13.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:31:12 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T00:31:14.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:31:13 smithi118 ceph-mon[31535]: pgmap v903: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-03-24T00:31:14.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:31:13 smithi118 ceph-mon[31535]: pgmap v904: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-03-24T00:31:14.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:31:13 smithi189 ceph-mon[36187]: pgmap v903: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-03-24T00:31:14.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:31:13 smithi189 ceph-mon[36187]: pgmap v904: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-03-24T00:31:14.460 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:31:13 smithi193 ceph-mon[36878]: pgmap v903: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-03-24T00:31:14.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:31:13 smithi193 ceph-mon[36878]: pgmap v904: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-03-24T00:31:15.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:31:14 smithi118 ceph-mon[31535]: pgmap v905: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-03-24T00:31:15.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:31:14 smithi189 ceph-mon[36187]: pgmap v905: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-03-24T00:31:15.460 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:31:14 smithi193 ceph-mon[36878]: pgmap v905: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-03-24T00:31:15.780 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:31:15.779+0000 7f81ee000640 1 -- 172.21.15.118:0/1693767697 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f81c8005850 con 0x7f81f81052d0 2024-03-24T00:31:15.780 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:31:15.779+0000 7f81ef400640 1 -- 172.21.15.118:0/1693767697 <== mon.2 v2:172.21.15.189:3300/0 7 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (secure 0 0 0) 0x7f81c8005850 con 0x7f81f81052d0 2024-03-24T00:31:15.807 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:31:15.807+0000 7f81db400640 1 -- 172.21.15.118:0/659686424 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f81c4005850 con 0x5622651a13f0 2024-03-24T00:31:15.808 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:31:15.807+0000 7f81e4c00640 1 -- 172.21.15.118:0/659686424 <== mon.1 v2:172.21.15.193:3300/0 7 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 409+0+0 (secure 0 0 0) 0x7f81c4005850 con 0x5622651a13f0 2024-03-24T00:31:16.348 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:31:15 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 371.620298 secs 2024-03-24T00:31:16.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:31:15 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 371.620298 secs 2024-03-24T00:31:16.460 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:31:15 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 371.620298 secs 2024-03-24T00:31:17.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:31:16 smithi118 ceph-mon[31535]: pgmap v906: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 6 op/s 2024-03-24T00:31:17.453 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:31:16 smithi189 ceph-mon[36187]: pgmap v906: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 6 op/s 2024-03-24T00:31:17.460 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:31:16 smithi193 ceph-mon[36878]: pgmap v906: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 6 op/s 2024-03-24T00:31:19.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:31:19 smithi118 ceph-mon[31535]: pgmap v907: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 5 op/s 2024-03-24T00:31:19.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:31:19 smithi189 ceph-mon[36187]: pgmap v907: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 5 op/s 2024-03-24T00:31:19.462 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:31:19 smithi193 ceph-mon[36878]: pgmap v907: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 5 op/s 2024-03-24T00:31:21.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:31:21 smithi118 ceph-mon[31535]: pgmap v908: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-03-24T00:31:21.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:31:21 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 376.620361 secs 2024-03-24T00:31:21.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:31:21 smithi189 ceph-mon[36187]: pgmap v908: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-03-24T00:31:21.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:31:21 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 376.620361 secs 2024-03-24T00:31:21.460 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:31:21 smithi193 ceph-mon[36878]: pgmap v908: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-03-24T00:31:21.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:31:21 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 376.620361 secs 2024-03-24T00:31:23.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:31:23 smithi118 ceph-mon[31535]: pgmap v909: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 5 op/s 2024-03-24T00:31:23.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:31:23 smithi189 ceph-mon[36187]: pgmap v909: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 5 op/s 2024-03-24T00:31:23.460 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:31:23 smithi193 ceph-mon[36878]: pgmap v909: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 5 op/s 2024-03-24T00:31:25.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:31:25 smithi118 ceph-mon[31535]: pgmap v910: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T00:31:25.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:31:25 smithi189 ceph-mon[36187]: pgmap v910: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T00:31:25.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:31:25 smithi193 ceph-mon[36878]: pgmap v910: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T00:31:26.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:31:26 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 381.620449 secs 2024-03-24T00:31:26.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:31:26 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 381.620449 secs 2024-03-24T00:31:26.460 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:31:26 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 381.620449 secs 2024-03-24T00:31:27.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:31:27 smithi118 ceph-mon[31535]: pgmap v911: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T00:31:27.453 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:31:27 smithi189 ceph-mon[36187]: pgmap v911: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T00:31:27.460 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:31:27 smithi193 ceph-mon[36878]: pgmap v911: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T00:31:29.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:31:29 smithi118 ceph-mon[31535]: pgmap v912: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 5 op/s 2024-03-24T00:31:29.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:31:29 smithi189 ceph-mon[36187]: pgmap v912: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 5 op/s 2024-03-24T00:31:29.460 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:31:29 smithi193 ceph-mon[36878]: pgmap v912: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 5 op/s 2024-03-24T00:31:31.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:31:31 smithi118 ceph-mon[31535]: pgmap v913: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T00:31:31.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:31:31 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 386.620512 secs 2024-03-24T00:31:31.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:31:31 smithi189 ceph-mon[36187]: pgmap v913: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T00:31:31.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:31:31 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 386.620512 secs 2024-03-24T00:31:31.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:31:31 smithi193 ceph-mon[36878]: pgmap v913: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T00:31:31.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:31:31 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 386.620512 secs 2024-03-24T00:31:33.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:31:33 smithi118 ceph-mon[31535]: pgmap v914: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:31:33.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:31:33 smithi189 ceph-mon[36187]: pgmap v914: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:31:33.460 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:31:33 smithi193 ceph-mon[36878]: pgmap v914: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:31:35.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:31:35 smithi118 ceph-mon[31535]: pgmap v915: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T00:31:35.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:31:35 smithi189 ceph-mon[36187]: pgmap v915: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T00:31:35.460 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:31:35 smithi193 ceph-mon[36878]: pgmap v915: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T00:31:36.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:31:36 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 391.620597 secs 2024-03-24T00:31:36.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:31:36 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 391.620597 secs 2024-03-24T00:31:36.460 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:31:36 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 391.620597 secs 2024-03-24T00:31:37.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:31:37 smithi118 ceph-mon[31535]: pgmap v916: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T00:31:37.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:31:37 smithi189 ceph-mon[36187]: pgmap v916: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T00:31:37.460 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:31:37 smithi193 ceph-mon[36878]: pgmap v916: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T00:31:39.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:31:39 smithi118 ceph-mon[31535]: pgmap v917: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 5 op/s 2024-03-24T00:31:39.453 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:31:39 smithi189 ceph-mon[36187]: pgmap v917: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 5 op/s 2024-03-24T00:31:39.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:31:39 smithi193 ceph-mon[36878]: pgmap v917: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 5 op/s 2024-03-24T00:31:41.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:31:41 smithi118 ceph-mon[31535]: pgmap v918: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T00:31:41.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:31:41 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 396.620686 secs 2024-03-24T00:31:41.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:31:41 smithi189 ceph-mon[36187]: pgmap v918: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T00:31:41.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:31:41 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 396.620686 secs 2024-03-24T00:31:41.460 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:31:41 smithi193 ceph-mon[36878]: pgmap v918: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T00:31:41.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:31:41 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 396.620686 secs 2024-03-24T00:31:43.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:31:43 smithi118 ceph-mon[31535]: pgmap v919: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:31:43.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:31:43 smithi189 ceph-mon[36187]: pgmap v919: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:31:43.460 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:31:43 smithi193 ceph-mon[36878]: pgmap v919: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:31:45.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:31:45 smithi118 ceph-mon[31535]: pgmap v920: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T00:31:45.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:31:45 smithi189 ceph-mon[36187]: pgmap v920: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T00:31:45.460 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:31:45 smithi193 ceph-mon[36878]: pgmap v920: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T00:31:46.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:31:46 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 401.620773 secs 2024-03-24T00:31:46.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:31:46 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 401.620773 secs 2024-03-24T00:31:46.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:31:46 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 401.620773 secs 2024-03-24T00:31:47.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:31:47 smithi118 ceph-mon[31535]: pgmap v921: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T00:31:47.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:31:47 smithi189 ceph-mon[36187]: pgmap v921: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T00:31:47.460 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:31:47 smithi193 ceph-mon[36878]: pgmap v921: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T00:31:49.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:31:49 smithi118 ceph-mon[31535]: pgmap v922: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 5 op/s 2024-03-24T00:31:49.453 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:31:49 smithi189 ceph-mon[36187]: pgmap v922: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 5 op/s 2024-03-24T00:31:49.460 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:31:49 smithi193 ceph-mon[36878]: pgmap v922: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 5 op/s 2024-03-24T00:31:51.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:31:51 smithi118 ceph-mon[31535]: pgmap v923: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T00:31:51.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:31:51 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 406.620862 secs 2024-03-24T00:31:51.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:31:51 smithi189 ceph-mon[36187]: pgmap v923: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T00:31:51.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:31:51 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 406.620862 secs 2024-03-24T00:31:51.460 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:31:51 smithi193 ceph-mon[36878]: pgmap v923: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T00:31:51.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:31:51 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 406.620862 secs 2024-03-24T00:31:53.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:31:53 smithi118 ceph-mon[31535]: pgmap v924: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:31:53.453 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:31:53 smithi189 ceph-mon[36187]: pgmap v924: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:31:53.460 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:31:53 smithi193 ceph-mon[36878]: pgmap v924: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:31:55.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:31:55 smithi118 ceph-mon[31535]: pgmap v925: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T00:31:55.453 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:31:55 smithi189 ceph-mon[36187]: pgmap v925: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T00:31:55.460 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:31:55 smithi193 ceph-mon[36878]: pgmap v925: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T00:31:56.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:31:56 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 411.620950 secs 2024-03-24T00:31:56.453 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:31:56 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 411.620950 secs 2024-03-24T00:31:56.460 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:31:56 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 411.620950 secs 2024-03-24T00:31:57.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:31:57 smithi118 ceph-mon[31535]: pgmap v926: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T00:31:57.453 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:31:57 smithi189 ceph-mon[36187]: pgmap v926: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T00:31:57.460 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:31:57 smithi193 ceph-mon[36878]: pgmap v926: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T00:31:59.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:31:59 smithi118 ceph-mon[31535]: pgmap v927: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 5 op/s 2024-03-24T00:31:59.453 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:31:59 smithi189 ceph-mon[36187]: pgmap v927: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 5 op/s 2024-03-24T00:31:59.460 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:31:59 smithi193 ceph-mon[36878]: pgmap v927: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 5 op/s 2024-03-24T00:32:01.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:32:01 smithi118 ceph-mon[31535]: pgmap v928: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T00:32:01.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:32:01 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 416.621016 secs 2024-03-24T00:32:01.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:32:01 smithi189 ceph-mon[36187]: pgmap v928: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T00:32:01.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:32:01 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 416.621016 secs 2024-03-24T00:32:01.460 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:32:01 smithi193 ceph-mon[36878]: pgmap v928: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T00:32:01.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:32:01 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 416.621016 secs 2024-03-24T00:32:04.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:32:03 smithi118 ceph-mon[31535]: pgmap v929: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:32:04.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:32:03 smithi189 ceph-mon[36187]: pgmap v929: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:32:04.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:32:03 smithi193 ceph-mon[36878]: pgmap v929: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:32:06.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:32:05 smithi118 ceph-mon[31535]: pgmap v930: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T00:32:06.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:32:05 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 421.621099 secs 2024-03-24T00:32:06.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:32:05 smithi189 ceph-mon[36187]: pgmap v930: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T00:32:06.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:32:05 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 421.621099 secs 2024-03-24T00:32:06.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:32:05 smithi193 ceph-mon[36878]: pgmap v930: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T00:32:06.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:32:05 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 421.621099 secs 2024-03-24T00:32:08.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:32:07 smithi118 ceph-mon[31535]: pgmap v931: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T00:32:08.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:32:07 smithi189 ceph-mon[36187]: pgmap v931: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T00:32:08.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:32:07 smithi193 ceph-mon[36878]: pgmap v931: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T00:32:10.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:32:09 smithi118 ceph-mon[31535]: pgmap v932: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 5 op/s 2024-03-24T00:32:10.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:32:09 smithi189 ceph-mon[36187]: pgmap v932: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 5 op/s 2024-03-24T00:32:10.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:32:09 smithi193 ceph-mon[36878]: pgmap v932: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 5 op/s 2024-03-24T00:32:11.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:32:10 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 426.621157 secs 2024-03-24T00:32:11.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:32:10 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 426.621157 secs 2024-03-24T00:32:11.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:32:10 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 426.621157 secs 2024-03-24T00:32:12.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:32:11 smithi118 ceph-mon[31535]: pgmap v933: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T00:32:12.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:32:11 smithi189 ceph-mon[36187]: pgmap v933: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T00:32:12.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:32:11 smithi193 ceph-mon[36878]: pgmap v933: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T00:32:13.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:32:12 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T00:32:13.203 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:32:12 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T00:32:13.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:32:12 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T00:32:13.957 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:32:13 smithi189 ceph-mon[36187]: pgmap v934: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:32:13.957 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:32:13 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-24T00:32:13.957 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:32:13 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-24T00:32:13.957 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:32:13 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-24T00:32:13.957 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:32:13 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-03-24T00:32:13.957 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:32:13 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-24T00:32:13.957 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:32:13 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-24T00:32:13.957 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:32:13 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-24T00:32:13.958 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:32:13 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-03-24T00:32:13.958 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:32:13 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T00:32:13.958 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:32:13 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T00:32:13.958 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:32:13 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T00:32:14.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:32:13 smithi193 ceph-mon[36878]: pgmap v934: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:32:14.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:32:13 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-24T00:32:14.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:32:13 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-24T00:32:14.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:32:13 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-24T00:32:14.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:32:13 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-03-24T00:32:14.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:32:13 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-24T00:32:14.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:32:13 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-24T00:32:14.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:32:13 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-24T00:32:14.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:32:13 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-03-24T00:32:14.212 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:32:13 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T00:32:14.212 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:32:13 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T00:32:14.212 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:32:13 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T00:32:14.447 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:32:13 smithi118 ceph-mon[31535]: pgmap v934: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:32:14.447 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:32:13 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-24T00:32:14.447 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:32:13 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-24T00:32:14.447 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:32:13 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-24T00:32:14.447 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:32:13 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-03-24T00:32:14.447 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:32:13 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-24T00:32:14.447 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:32:13 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-24T00:32:14.447 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:32:13 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-24T00:32:14.448 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:32:13 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-03-24T00:32:14.448 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:32:13 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T00:32:14.448 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:32:13 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T00:32:14.448 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:32:13 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T00:32:15.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:32:14 smithi193 ceph-mon[36878]: Adjusting osd_memory_target on smithi189 to 174.1M 2024-03-24T00:32:15.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:32:14 smithi193 ceph-mon[36878]: Unable to set osd_memory_target on smithi189 to 182635622: error parsing value: Value '182635622' is below minimum 939524096 2024-03-24T00:32:15.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:32:14 smithi193 ceph-mon[36878]: Adjusting osd_memory_target on smithi118 to 176.7M 2024-03-24T00:32:15.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:32:14 smithi193 ceph-mon[36878]: Unable to set osd_memory_target on smithi118 to 185365196: error parsing value: Value '185365196' is below minimum 939524096 2024-03-24T00:32:15.212 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:32:14 smithi193 ceph-mon[36878]: pgmap v935: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 6 op/s 2024-03-24T00:32:15.212 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:32:14 smithi193 ceph-mon[36878]: pgmap v936: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 5 op/s 2024-03-24T00:32:15.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:32:14 smithi118 ceph-mon[31535]: Adjusting osd_memory_target on smithi189 to 174.1M 2024-03-24T00:32:15.447 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:32:14 smithi118 ceph-mon[31535]: Unable to set osd_memory_target on smithi189 to 182635622: error parsing value: Value '182635622' is below minimum 939524096 2024-03-24T00:32:15.447 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:32:14 smithi118 ceph-mon[31535]: Adjusting osd_memory_target on smithi118 to 176.7M 2024-03-24T00:32:15.447 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:32:14 smithi118 ceph-mon[31535]: Unable to set osd_memory_target on smithi118 to 185365196: error parsing value: Value '185365196' is below minimum 939524096 2024-03-24T00:32:15.447 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:32:14 smithi118 ceph-mon[31535]: pgmap v935: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 6 op/s 2024-03-24T00:32:15.447 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:32:14 smithi118 ceph-mon[31535]: pgmap v936: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 5 op/s 2024-03-24T00:32:15.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:32:14 smithi189 ceph-mon[36187]: Adjusting osd_memory_target on smithi189 to 174.1M 2024-03-24T00:32:15.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:32:14 smithi189 ceph-mon[36187]: Unable to set osd_memory_target on smithi189 to 182635622: error parsing value: Value '182635622' is below minimum 939524096 2024-03-24T00:32:15.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:32:14 smithi189 ceph-mon[36187]: Adjusting osd_memory_target on smithi118 to 176.7M 2024-03-24T00:32:15.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:32:14 smithi189 ceph-mon[36187]: Unable to set osd_memory_target on smithi118 to 185365196: error parsing value: Value '185365196' is below minimum 939524096 2024-03-24T00:32:15.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:32:14 smithi189 ceph-mon[36187]: pgmap v935: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 6 op/s 2024-03-24T00:32:15.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:32:14 smithi189 ceph-mon[36187]: pgmap v936: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 5 op/s 2024-03-24T00:32:16.348 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:32:15 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 431.621215 secs 2024-03-24T00:32:16.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:32:15 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 431.621215 secs 2024-03-24T00:32:16.460 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:32:15 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 431.621215 secs 2024-03-24T00:32:17.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:32:16 smithi118 ceph-mon[31535]: pgmap v937: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-03-24T00:32:17.453 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:32:16 smithi189 ceph-mon[36187]: pgmap v937: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-03-24T00:32:17.460 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:32:16 smithi193 ceph-mon[36878]: pgmap v937: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-03-24T00:32:19.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:32:18 smithi118 ceph-mon[31535]: pgmap v938: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 5 op/s 2024-03-24T00:32:19.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:32:18 smithi189 ceph-mon[36187]: pgmap v938: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 5 op/s 2024-03-24T00:32:19.460 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:32:18 smithi193 ceph-mon[36878]: pgmap v938: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 5 op/s 2024-03-24T00:32:21.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:32:20 smithi118 ceph-mon[31535]: pgmap v939: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-03-24T00:32:21.447 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:32:20 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 436.621305 secs 2024-03-24T00:32:21.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:32:20 smithi189 ceph-mon[36187]: pgmap v939: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-03-24T00:32:21.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:32:20 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 436.621305 secs 2024-03-24T00:32:21.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:32:20 smithi193 ceph-mon[36878]: pgmap v939: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-03-24T00:32:21.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:32:20 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 436.621305 secs 2024-03-24T00:32:23.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:32:22 smithi118 ceph-mon[31535]: pgmap v940: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-03-24T00:32:23.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:32:22 smithi189 ceph-mon[36187]: pgmap v940: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-03-24T00:32:23.460 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:32:22 smithi193 ceph-mon[36878]: pgmap v940: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-03-24T00:32:25.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:32:24 smithi118 ceph-mon[31535]: pgmap v941: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 5 op/s 2024-03-24T00:32:25.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:32:24 smithi189 ceph-mon[36187]: pgmap v941: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 5 op/s 2024-03-24T00:32:25.460 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:32:24 smithi193 ceph-mon[36878]: pgmap v941: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 5 op/s 2024-03-24T00:32:26.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:32:25 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 441.621367 secs 2024-03-24T00:32:26.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:32:25 smithi118 ceph-mon[31535]: pgmap v942: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T00:32:26.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:32:25 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 441.621367 secs 2024-03-24T00:32:26.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:32:25 smithi189 ceph-mon[36187]: pgmap v942: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T00:32:26.460 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:32:25 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 441.621367 secs 2024-03-24T00:32:26.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:32:25 smithi193 ceph-mon[36878]: pgmap v942: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T00:32:28.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:32:28 smithi118 ceph-mon[31535]: pgmap v943: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T00:32:28.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:32:28 smithi189 ceph-mon[36187]: pgmap v943: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T00:32:28.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:32:28 smithi193 ceph-mon[36878]: pgmap v943: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T00:32:30.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:32:30 smithi118 ceph-mon[31535]: pgmap v944: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-03-24T00:32:30.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:32:30 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 446.621425 secs 2024-03-24T00:32:30.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:32:30 smithi189 ceph-mon[36187]: pgmap v944: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-03-24T00:32:30.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:32:30 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 446.621425 secs 2024-03-24T00:32:30.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:32:30 smithi193 ceph-mon[36878]: pgmap v944: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-03-24T00:32:30.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:32:30 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 446.621425 secs 2024-03-24T00:32:32.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:32:32 smithi118 ceph-mon[31535]: pgmap v945: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:32:32.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:32:32 smithi189 ceph-mon[36187]: pgmap v945: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:32:32.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:32:32 smithi193 ceph-mon[36878]: pgmap v945: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:32:34.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:32:34 smithi118 ceph-mon[31535]: pgmap v946: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T00:32:34.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:32:34 smithi189 ceph-mon[36187]: pgmap v946: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T00:32:34.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:32:34 smithi193 ceph-mon[36878]: pgmap v946: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T00:32:35.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:32:35 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 451.621513 secs 2024-03-24T00:32:35.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:32:35 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 451.621513 secs 2024-03-24T00:32:35.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:32:35 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 451.621513 secs 2024-03-24T00:32:36.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:32:36 smithi118 ceph-mon[31535]: pgmap v947: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T00:32:36.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:32:36 smithi189 ceph-mon[36187]: pgmap v947: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T00:32:36.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:32:36 smithi193 ceph-mon[36878]: pgmap v947: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T00:32:38.936 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:32:38 smithi118 ceph-mon[31535]: pgmap v948: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T00:32:38.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:32:38 smithi189 ceph-mon[36187]: pgmap v948: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T00:32:38.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:32:38 smithi193 ceph-mon[36878]: pgmap v948: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T00:32:40.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:32:40 smithi118 ceph-mon[31535]: pgmap v949: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T00:32:40.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:32:40 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 456.621575 secs 2024-03-24T00:32:40.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:32:40 smithi189 ceph-mon[36187]: pgmap v949: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T00:32:40.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:32:40 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 456.621575 secs 2024-03-24T00:32:40.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:32:40 smithi193 ceph-mon[36878]: pgmap v949: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T00:32:40.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:32:40 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 456.621575 secs 2024-03-24T00:32:42.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:32:42 smithi118 ceph-mon[31535]: pgmap v950: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:32:42.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:32:42 smithi189 ceph-mon[36187]: pgmap v950: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:32:42.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:32:42 smithi193 ceph-mon[36878]: pgmap v950: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:32:44.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:32:44 smithi118 ceph-mon[31535]: pgmap v951: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T00:32:44.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:32:44 smithi189 ceph-mon[36187]: pgmap v951: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T00:32:44.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:32:44 smithi193 ceph-mon[36878]: pgmap v951: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T00:32:45.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:32:45 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 461.621665 secs 2024-03-24T00:32:45.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:32:45 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 461.621665 secs 2024-03-24T00:32:45.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:32:45 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 461.621665 secs 2024-03-24T00:32:46.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:32:46 smithi118 ceph-mon[31535]: pgmap v952: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T00:32:46.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:32:46 smithi189 ceph-mon[36187]: pgmap v952: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T00:32:46.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:32:46 smithi193 ceph-mon[36878]: pgmap v952: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T00:32:48.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:32:48 smithi118 ceph-mon[31535]: pgmap v953: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T00:32:48.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:32:48 smithi189 ceph-mon[36187]: pgmap v953: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T00:32:48.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:32:48 smithi193 ceph-mon[36878]: pgmap v953: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T00:32:50.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:32:50 smithi118 ceph-mon[31535]: pgmap v954: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T00:32:50.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:32:50 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 466.621752 secs 2024-03-24T00:32:50.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:32:50 smithi189 ceph-mon[36187]: pgmap v954: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T00:32:50.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:32:50 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 466.621752 secs 2024-03-24T00:32:50.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:32:50 smithi193 ceph-mon[36878]: pgmap v954: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T00:32:50.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:32:50 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 466.621752 secs 2024-03-24T00:32:52.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:32:52 smithi118 ceph-mon[31535]: pgmap v955: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:32:52.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:32:52 smithi189 ceph-mon[36187]: pgmap v955: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:32:52.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:32:52 smithi193 ceph-mon[36878]: pgmap v955: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:32:54.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:32:54 smithi118 ceph-mon[31535]: pgmap v956: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T00:32:54.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:32:54 smithi189 ceph-mon[36187]: pgmap v956: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T00:32:54.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:32:54 smithi193 ceph-mon[36878]: pgmap v956: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T00:32:55.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:32:55 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 471.621813 secs 2024-03-24T00:32:55.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:32:55 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 471.621813 secs 2024-03-24T00:32:55.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:32:55 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 471.621813 secs 2024-03-24T00:32:56.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:32:56 smithi118 ceph-mon[31535]: pgmap v957: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T00:32:56.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:32:56 smithi189 ceph-mon[36187]: pgmap v957: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T00:32:56.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:32:56 smithi193 ceph-mon[36878]: pgmap v957: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T00:32:58.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:32:58 smithi118 ceph-mon[31535]: pgmap v958: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T00:32:58.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:32:58 smithi189 ceph-mon[36187]: pgmap v958: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T00:32:58.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:32:58 smithi193 ceph-mon[36878]: pgmap v958: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T00:33:00.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:33:00 smithi118 ceph-mon[31535]: pgmap v959: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-03-24T00:33:00.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:33:00 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 476.621902 secs 2024-03-24T00:33:00.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:33:00 smithi189 ceph-mon[36187]: pgmap v959: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-03-24T00:33:00.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:33:00 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 476.621902 secs 2024-03-24T00:33:00.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:33:00 smithi193 ceph-mon[36878]: pgmap v959: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-03-24T00:33:00.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:33:00 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 476.621902 secs 2024-03-24T00:33:02.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:33:02 smithi189 ceph-mon[36187]: pgmap v960: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:33:02.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:33:02 smithi193 ceph-mon[36878]: pgmap v960: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:33:03.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:33:02 smithi118 ceph-mon[31535]: pgmap v960: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:33:04.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:33:04 smithi189 ceph-mon[36187]: pgmap v961: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T00:33:04.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:33:04 smithi193 ceph-mon[36878]: pgmap v961: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T00:33:05.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:33:04 smithi118 ceph-mon[31535]: pgmap v961: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T00:33:05.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:33:05 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 481.621943 secs 2024-03-24T00:33:06.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:33:05 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 481.621943 secs 2024-03-24T00:33:06.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:33:05 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 481.621943 secs 2024-03-24T00:33:07.096 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:33:06 smithi118 ceph-mon[31535]: pgmap v962: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T00:33:07.105 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:33:06 smithi189 ceph-mon[36187]: pgmap v962: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T00:33:07.213 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:33:06 smithi193 ceph-mon[36878]: pgmap v962: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T00:33:09.197 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:33:08 smithi118 ceph-mon[31535]: pgmap v963: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T00:33:09.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:33:08 smithi189 ceph-mon[36187]: pgmap v963: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T00:33:09.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:33:08 smithi193 ceph-mon[36878]: pgmap v963: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T00:33:11.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:33:10 smithi118 ceph-mon[31535]: pgmap v964: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-03-24T00:33:11.197 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:33:10 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 486.622005 secs 2024-03-24T00:33:11.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:33:10 smithi189 ceph-mon[36187]: pgmap v964: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-03-24T00:33:11.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:33:10 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 486.622005 secs 2024-03-24T00:33:11.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:33:10 smithi193 ceph-mon[36878]: pgmap v964: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-03-24T00:33:11.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:33:10 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 486.622005 secs 2024-03-24T00:33:13.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:33:12 smithi118 ceph-mon[31535]: pgmap v965: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:33:13.203 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:33:12 smithi189 ceph-mon[36187]: pgmap v965: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:33:13.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:33:12 smithi193 ceph-mon[36878]: pgmap v965: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:33:15.197 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:33:14 smithi118 ceph-mon[31535]: pgmap v966: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T00:33:15.197 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:33:14 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T00:33:15.197 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:33:14 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-24T00:33:15.197 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:33:14 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-24T00:33:15.197 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:33:14 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-03-24T00:33:15.198 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:33:14 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-03-24T00:33:15.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:33:14 smithi189 ceph-mon[36187]: pgmap v966: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T00:33:15.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:33:14 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T00:33:15.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:33:14 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-24T00:33:15.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:33:14 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-24T00:33:15.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:33:14 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-03-24T00:33:15.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:33:14 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-03-24T00:33:15.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:33:14 smithi193 ceph-mon[36878]: pgmap v966: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T00:33:15.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:33:14 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T00:33:15.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:33:14 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-24T00:33:15.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:33:14 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-24T00:33:15.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:33:14 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-03-24T00:33:15.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:33:14 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-03-24T00:33:16.098 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:33:15 smithi118 ceph-mon[31535]: Adjusting osd_memory_target on smithi193 to 174.1M 2024-03-24T00:33:16.099 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:33:15 smithi118 ceph-mon[31535]: Unable to set osd_memory_target on smithi193 to 182639923: error parsing value: Value '182639923' is below minimum 939524096 2024-03-24T00:33:16.099 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:33:15 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T00:33:16.099 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:33:15 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T00:33:16.099 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:33:15 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T00:33:16.099 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:33:15 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 491.622071 secs 2024-03-24T00:33:16.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:33:15 smithi189 ceph-mon[36187]: Adjusting osd_memory_target on smithi193 to 174.1M 2024-03-24T00:33:16.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:33:15 smithi189 ceph-mon[36187]: Unable to set osd_memory_target on smithi193 to 182639923: error parsing value: Value '182639923' is below minimum 939524096 2024-03-24T00:33:16.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:33:15 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T00:33:16.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:33:15 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T00:33:16.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:33:15 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T00:33:16.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:33:15 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 491.622071 secs 2024-03-24T00:33:16.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:33:15 smithi193 ceph-mon[36878]: Adjusting osd_memory_target on smithi193 to 174.1M 2024-03-24T00:33:16.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:33:15 smithi193 ceph-mon[36878]: Unable to set osd_memory_target on smithi193 to 182639923: error parsing value: Value '182639923' is below minimum 939524096 2024-03-24T00:33:16.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:33:15 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T00:33:16.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:33:15 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T00:33:16.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:33:15 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T00:33:16.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:33:15 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 491.622071 secs 2024-03-24T00:33:17.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:33:16 smithi118 ceph-mon[31535]: pgmap v967: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 6 op/s 2024-03-24T00:33:17.197 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:33:16 smithi118 ceph-mon[31535]: pgmap v968: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-03-24T00:33:17.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:33:16 smithi189 ceph-mon[36187]: pgmap v967: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 6 op/s 2024-03-24T00:33:17.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:33:16 smithi189 ceph-mon[36187]: pgmap v968: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-03-24T00:33:17.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:33:16 smithi193 ceph-mon[36878]: pgmap v967: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 6 op/s 2024-03-24T00:33:17.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:33:16 smithi193 ceph-mon[36878]: pgmap v968: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-03-24T00:33:19.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:33:18 smithi118 ceph-mon[31535]: pgmap v969: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 5 op/s 2024-03-24T00:33:19.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:33:18 smithi189 ceph-mon[36187]: pgmap v969: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 5 op/s 2024-03-24T00:33:19.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:33:18 smithi193 ceph-mon[36878]: pgmap v969: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 5 op/s 2024-03-24T00:33:21.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:33:20 smithi118 ceph-mon[31535]: pgmap v970: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 5 op/s 2024-03-24T00:33:21.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:33:20 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 496.622155 secs 2024-03-24T00:33:21.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:33:20 smithi189 ceph-mon[36187]: pgmap v970: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 5 op/s 2024-03-24T00:33:21.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:33:20 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 496.622155 secs 2024-03-24T00:33:21.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:33:20 smithi193 ceph-mon[36878]: pgmap v970: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 5 op/s 2024-03-24T00:33:21.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:33:20 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 496.622155 secs 2024-03-24T00:33:23.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:33:22 smithi118 ceph-mon[31535]: pgmap v971: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-03-24T00:33:23.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:33:22 smithi189 ceph-mon[36187]: pgmap v971: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-03-24T00:33:23.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:33:22 smithi193 ceph-mon[36878]: pgmap v971: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-03-24T00:33:25.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:33:24 smithi118 ceph-mon[31535]: pgmap v972: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 5 op/s 2024-03-24T00:33:25.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:33:24 smithi189 ceph-mon[36187]: pgmap v972: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 5 op/s 2024-03-24T00:33:25.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:33:24 smithi193 ceph-mon[36878]: pgmap v972: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 5 op/s 2024-03-24T00:33:26.040 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:33:25 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 501.622216 secs 2024-03-24T00:33:26.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:33:25 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 501.622216 secs 2024-03-24T00:33:26.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:33:25 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 501.622216 secs 2024-03-24T00:33:27.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:33:26 smithi118 ceph-mon[31535]: pgmap v973: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 5 op/s 2024-03-24T00:33:27.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:33:26 smithi189 ceph-mon[36187]: pgmap v973: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 5 op/s 2024-03-24T00:33:27.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:33:26 smithi193 ceph-mon[36878]: pgmap v973: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 5 op/s 2024-03-24T00:33:29.197 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:33:28 smithi118 ceph-mon[31535]: pgmap v974: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:33:29.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:33:28 smithi189 ceph-mon[36187]: pgmap v974: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:33:29.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:33:28 smithi193 ceph-mon[36878]: pgmap v974: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:33:30.802 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:33:30.802+0000 7f81ee000640 1 -- 172.21.15.118:0/1693767697 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- auth(proto 2 132 bytes epoch 0) v1 -- 0x7f81c8005b50 con 0x7f81f81052d0 2024-03-24T00:33:30.803 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:33:30.803+0000 7f81ef400640 1 -- 172.21.15.118:0/1693767697 <== mon.2 v2:172.21.15.189:3300/0 8 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 233+0+0 (secure 0 0 0) 0x7f81c8005b50 con 0x7f81f81052d0 2024-03-24T00:33:30.828 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:33:30.828+0000 7f81db400640 1 -- 172.21.15.118:0/659686424 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- auth(proto 2 132 bytes epoch 0) v1 -- 0x7f81c4005b30 con 0x5622651a13f0 2024-03-24T00:33:30.829 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:33:30.829+0000 7f81e4c00640 1 -- 172.21.15.118:0/659686424 <== mon.1 v2:172.21.15.193:3300/0 8 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 233+0+0 (secure 0 0 0) 0x7f81c4005b30 con 0x5622651a13f0 2024-03-24T00:33:31.199 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:33:30 smithi118 ceph-mon[31535]: pgmap v975: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T00:33:31.199 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:33:30 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 506.622279 secs 2024-03-24T00:33:31.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:33:30 smithi189 ceph-mon[36187]: pgmap v975: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T00:33:31.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:33:30 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 506.622279 secs 2024-03-24T00:33:31.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:33:30 smithi193 ceph-mon[36878]: pgmap v975: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T00:33:31.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:33:30 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 506.622279 secs 2024-03-24T00:33:33.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:33:32 smithi118 ceph-mon[31535]: pgmap v976: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T00:33:33.203 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:33:32 smithi189 ceph-mon[36187]: pgmap v976: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T00:33:33.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:33:32 smithi193 ceph-mon[36878]: pgmap v976: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T00:33:35.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:33:34 smithi118 ceph-mon[31535]: pgmap v977: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 5 op/s 2024-03-24T00:33:35.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:33:34 smithi189 ceph-mon[36187]: pgmap v977: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 5 op/s 2024-03-24T00:33:35.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:33:34 smithi193 ceph-mon[36878]: pgmap v977: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 5 op/s 2024-03-24T00:33:36.112 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:33:35 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 511.622341 secs 2024-03-24T00:33:36.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:33:35 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 511.622341 secs 2024-03-24T00:33:36.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:33:35 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 511.622341 secs 2024-03-24T00:33:37.105 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:33:36 smithi189 ceph-mon[36187]: pgmap v978: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T00:33:37.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:33:36 smithi118 ceph-mon[31535]: pgmap v978: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T00:33:37.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:33:36 smithi193 ceph-mon[36878]: pgmap v978: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T00:33:39.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:33:38 smithi118 ceph-mon[31535]: pgmap v979: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:33:39.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:33:38 smithi189 ceph-mon[36187]: pgmap v979: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:33:39.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:33:38 smithi193 ceph-mon[36878]: pgmap v979: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:33:41.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:33:40 smithi118 ceph-mon[31535]: pgmap v980: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T00:33:41.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:33:40 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 516.622429 secs 2024-03-24T00:33:41.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:33:40 smithi189 ceph-mon[36187]: pgmap v980: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T00:33:41.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:33:40 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 516.622429 secs 2024-03-24T00:33:41.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:33:40 smithi193 ceph-mon[36878]: pgmap v980: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T00:33:41.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:33:40 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 516.622429 secs 2024-03-24T00:33:43.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:33:42 smithi118 ceph-mon[31535]: pgmap v981: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T00:33:43.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:33:42 smithi189 ceph-mon[36187]: pgmap v981: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T00:33:43.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:33:42 smithi193 ceph-mon[36878]: pgmap v981: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T00:33:45.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:33:44 smithi118 ceph-mon[31535]: pgmap v982: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 5 op/s 2024-03-24T00:33:45.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:33:44 smithi189 ceph-mon[36187]: pgmap v982: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 5 op/s 2024-03-24T00:33:45.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:33:44 smithi193 ceph-mon[36878]: pgmap v982: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 5 op/s 2024-03-24T00:33:46.098 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:33:45 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 521.622519 secs 2024-03-24T00:33:46.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:33:45 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 521.622519 secs 2024-03-24T00:33:46.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:33:45 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 521.622519 secs 2024-03-24T00:33:47.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:33:46 smithi118 ceph-mon[31535]: pgmap v983: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T00:33:47.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:33:46 smithi189 ceph-mon[36187]: pgmap v983: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T00:33:47.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:33:46 smithi193 ceph-mon[36878]: pgmap v983: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T00:33:49.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:33:48 smithi118 ceph-mon[31535]: pgmap v984: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:33:49.203 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:33:48 smithi189 ceph-mon[36187]: pgmap v984: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:33:49.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:33:48 smithi193 ceph-mon[36878]: pgmap v984: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:33:51.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:33:50 smithi118 ceph-mon[31535]: pgmap v985: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T00:33:51.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:33:50 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 526.622610 secs 2024-03-24T00:33:51.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:33:50 smithi189 ceph-mon[36187]: pgmap v985: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T00:33:51.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:33:50 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 526.622610 secs 2024-03-24T00:33:51.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:33:50 smithi193 ceph-mon[36878]: pgmap v985: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T00:33:51.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:33:50 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 526.622610 secs 2024-03-24T00:33:53.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:33:52 smithi118 ceph-mon[31535]: pgmap v986: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T00:33:53.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:33:52 smithi189 ceph-mon[36187]: pgmap v986: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T00:33:53.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:33:52 smithi193 ceph-mon[36878]: pgmap v986: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T00:33:55.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:33:54 smithi118 ceph-mon[31535]: pgmap v987: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T00:33:55.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:33:54 smithi189 ceph-mon[36187]: pgmap v987: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T00:33:55.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:33:54 smithi193 ceph-mon[36878]: pgmap v987: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T00:33:56.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:33:55 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 531.622696 secs 2024-03-24T00:33:56.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:33:55 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 531.622696 secs 2024-03-24T00:33:56.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:33:55 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 531.622696 secs 2024-03-24T00:33:57.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:33:56 smithi118 ceph-mon[31535]: pgmap v988: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T00:33:57.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:33:56 smithi189 ceph-mon[36187]: pgmap v988: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T00:33:57.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:33:56 smithi193 ceph-mon[36878]: pgmap v988: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T00:33:59.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:33:58 smithi118 ceph-mon[31535]: pgmap v989: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:33:59.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:33:58 smithi189 ceph-mon[36187]: pgmap v989: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:33:59.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:33:58 smithi193 ceph-mon[36878]: pgmap v989: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:34:01.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:34:00 smithi118 ceph-mon[31535]: pgmap v990: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T00:34:01.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:34:00 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 536.622783 secs 2024-03-24T00:34:01.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:34:00 smithi189 ceph-mon[36187]: pgmap v990: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T00:34:01.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:34:00 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 536.622783 secs 2024-03-24T00:34:01.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:34:00 smithi193 ceph-mon[36878]: pgmap v990: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T00:34:01.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:34:00 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 536.622783 secs 2024-03-24T00:34:03.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:34:02 smithi118 ceph-mon[31535]: pgmap v991: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T00:34:03.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:34:02 smithi189 ceph-mon[36187]: pgmap v991: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T00:34:03.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:34:02 smithi193 ceph-mon[36878]: pgmap v991: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T00:34:05.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:34:04 smithi118 ceph-mon[31535]: pgmap v992: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T00:34:05.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:34:04 smithi189 ceph-mon[36187]: pgmap v992: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T00:34:05.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:34:04 smithi193 ceph-mon[36878]: pgmap v992: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T00:34:06.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:34:05 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 541.622871 secs 2024-03-24T00:34:06.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:34:05 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 541.622871 secs 2024-03-24T00:34:06.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:34:05 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 541.622871 secs 2024-03-24T00:34:07.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:34:06 smithi118 ceph-mon[31535]: pgmap v993: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T00:34:07.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:34:06 smithi189 ceph-mon[36187]: pgmap v993: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T00:34:07.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:34:06 smithi193 ceph-mon[36878]: pgmap v993: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T00:34:09.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:34:08 smithi118 ceph-mon[31535]: pgmap v994: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:34:09.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:34:08 smithi189 ceph-mon[36187]: pgmap v994: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:34:09.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:34:08 smithi193 ceph-mon[36878]: pgmap v994: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:34:11.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:34:10 smithi118 ceph-mon[31535]: pgmap v995: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T00:34:11.197 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:34:10 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 546.622960 secs 2024-03-24T00:34:11.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:34:10 smithi189 ceph-mon[36187]: pgmap v995: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T00:34:11.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:34:10 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 546.622960 secs 2024-03-24T00:34:11.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:34:10 smithi193 ceph-mon[36878]: pgmap v995: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T00:34:11.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:34:10 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 546.622960 secs 2024-03-24T00:34:13.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:34:12 smithi118 ceph-mon[31535]: pgmap v996: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T00:34:13.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:34:12 smithi189 ceph-mon[36187]: pgmap v996: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T00:34:13.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:34:12 smithi193 ceph-mon[36878]: pgmap v996: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T00:34:15.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:34:14 smithi118 ceph-mon[31535]: pgmap v997: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T00:34:15.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:34:14 smithi189 ceph-mon[36187]: pgmap v997: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T00:34:15.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:34:14 smithi193 ceph-mon[36878]: pgmap v997: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T00:34:15.809 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:34:15.809+0000 7f81ee000640 1 -- 172.21.15.118:0/1693767697 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f81c8005ea0 con 0x7f81f81052d0 2024-03-24T00:34:15.810 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:34:15.810+0000 7f81ef400640 1 -- 172.21.15.118:0/1693767697 <== mon.2 v2:172.21.15.189:3300/0 9 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (secure 0 0 0) 0x7f81c8005ea0 con 0x7f81f81052d0 2024-03-24T00:34:15.835 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:34:15.835+0000 7f81db400640 1 -- 172.21.15.118:0/659686424 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f81c4005de0 con 0x5622651a13f0 2024-03-24T00:34:15.835 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:34:15.836+0000 7f81e4c00640 1 -- 172.21.15.118:0/659686424 <== mon.1 v2:172.21.15.193:3300/0 9 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 409+0+0 (secure 0 0 0) 0x7f81c4005de0 con 0x5622651a13f0 2024-03-24T00:34:16.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:34:15 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 551.623049 secs 2024-03-24T00:34:16.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:34:15 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T00:34:16.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:34:15 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 551.623049 secs 2024-03-24T00:34:16.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:34:15 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T00:34:16.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:34:15 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 551.623049 secs 2024-03-24T00:34:16.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:34:15 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T00:34:17.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:34:16 smithi118 ceph-mon[31535]: pgmap v998: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T00:34:17.197 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:34:16 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T00:34:17.197 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:34:16 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T00:34:17.197 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:34:16 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T00:34:17.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:34:16 smithi189 ceph-mon[36187]: pgmap v998: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T00:34:17.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:34:16 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T00:34:17.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:34:16 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T00:34:17.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:34:16 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T00:34:17.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:34:16 smithi193 ceph-mon[36878]: pgmap v998: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T00:34:17.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:34:16 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T00:34:17.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:34:16 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T00:34:17.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:34:16 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T00:34:18.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:34:17 smithi118 ceph-mon[31535]: pgmap v999: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 5 op/s 2024-03-24T00:34:18.197 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:34:17 smithi118 ceph-mon[31535]: pgmap v1000: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-03-24T00:34:18.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:34:17 smithi189 ceph-mon[36187]: pgmap v999: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 5 op/s 2024-03-24T00:34:18.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:34:17 smithi189 ceph-mon[36187]: pgmap v1000: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-03-24T00:34:18.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:34:17 smithi193 ceph-mon[36878]: pgmap v999: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 5 op/s 2024-03-24T00:34:18.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:34:17 smithi193 ceph-mon[36878]: pgmap v1000: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-03-24T00:34:20.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:34:19 smithi118 ceph-mon[31535]: pgmap v1001: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 5 op/s 2024-03-24T00:34:20.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:34:19 smithi189 ceph-mon[36187]: pgmap v1001: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 5 op/s 2024-03-24T00:34:20.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:34:19 smithi193 ceph-mon[36878]: pgmap v1001: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 5 op/s 2024-03-24T00:34:21.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:34:20 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 556.623095 secs 2024-03-24T00:34:21.214 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:34:20 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 556.623095 secs 2024-03-24T00:34:21.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:34:20 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 556.623095 secs 2024-03-24T00:34:22.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:34:21 smithi189 ceph-mon[36187]: pgmap v1002: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-03-24T00:34:22.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:34:21 smithi193 ceph-mon[36878]: pgmap v1002: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-03-24T00:34:22.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:34:21 smithi118 ceph-mon[31535]: pgmap v1002: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-03-24T00:34:24.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:34:23 smithi193 ceph-mon[36878]: pgmap v1003: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 5 op/s 2024-03-24T00:34:24.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:34:23 smithi118 ceph-mon[31535]: pgmap v1003: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 5 op/s 2024-03-24T00:34:24.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:34:23 smithi189 ceph-mon[36187]: pgmap v1003: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 5 op/s 2024-03-24T00:34:26.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:34:25 smithi193 ceph-mon[36878]: pgmap v1004: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-03-24T00:34:26.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:34:25 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 561.623155 secs 2024-03-24T00:34:26.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:34:25 smithi118 ceph-mon[31535]: pgmap v1004: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-03-24T00:34:26.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:34:25 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 561.623155 secs 2024-03-24T00:34:26.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:34:25 smithi189 ceph-mon[36187]: pgmap v1004: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-03-24T00:34:26.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:34:25 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 561.623155 secs 2024-03-24T00:34:28.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:34:27 smithi118 ceph-mon[31535]: pgmap v1005: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 5 op/s 2024-03-24T00:34:28.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:34:27 smithi189 ceph-mon[36187]: pgmap v1005: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 5 op/s 2024-03-24T00:34:28.460 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:34:27 smithi193 ceph-mon[36878]: pgmap v1005: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 5 op/s 2024-03-24T00:34:30.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:34:29 smithi118 ceph-mon[31535]: pgmap v1006: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T00:34:30.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:34:29 smithi189 ceph-mon[36187]: pgmap v1006: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T00:34:30.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:34:29 smithi193 ceph-mon[36878]: pgmap v1006: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T00:34:31.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:34:30 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 566.623169 secs 2024-03-24T00:34:31.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:34:30 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 566.623169 secs 2024-03-24T00:34:31.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:34:30 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 566.623169 secs 2024-03-24T00:34:32.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:34:31 smithi118 ceph-mon[31535]: pgmap v1007: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T00:34:32.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:34:31 smithi189 ceph-mon[36187]: pgmap v1007: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T00:34:32.460 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:34:31 smithi193 ceph-mon[36878]: pgmap v1007: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T00:34:34.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:34:33 smithi118 ceph-mon[31535]: pgmap v1008: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T00:34:34.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:34:33 smithi189 ceph-mon[36187]: pgmap v1008: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T00:34:34.460 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:34:33 smithi193 ceph-mon[36878]: pgmap v1008: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T00:34:36.262 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:34:35 smithi193 ceph-mon[36878]: pgmap v1009: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-03-24T00:34:36.262 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:34:35 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 571.623230 secs 2024-03-24T00:34:36.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:34:35 smithi118 ceph-mon[31535]: pgmap v1009: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-03-24T00:34:36.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:34:35 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 571.623230 secs 2024-03-24T00:34:36.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:34:35 smithi189 ceph-mon[36187]: pgmap v1009: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-03-24T00:34:36.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:34:35 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 571.623230 secs 2024-03-24T00:34:38.256 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:34:37 smithi189 ceph-mon[36187]: pgmap v1010: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:34:38.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:34:37 smithi118 ceph-mon[31535]: pgmap v1010: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:34:38.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:34:37 smithi193 ceph-mon[36878]: pgmap v1010: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:34:40.281 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:34:40 smithi118 ceph-mon[31535]: pgmap v1011: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T00:34:40.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:34:40 smithi189 ceph-mon[36187]: pgmap v1011: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T00:34:40.460 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:34:40 smithi193 ceph-mon[36878]: pgmap v1011: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T00:34:41.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:34:41 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 576.623316 secs 2024-03-24T00:34:41.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:34:41 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 576.623316 secs 2024-03-24T00:34:41.460 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:34:41 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 576.623316 secs 2024-03-24T00:34:42.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:34:42 smithi118 ceph-mon[31535]: pgmap v1012: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T00:34:42.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:34:42 smithi189 ceph-mon[36187]: pgmap v1012: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T00:34:42.460 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:34:42 smithi193 ceph-mon[36878]: pgmap v1012: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T00:34:43.352 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:34:43 smithi118 ceph-mon[31535]: pgmap v1013: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T00:34:43.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:34:43 smithi189 ceph-mon[36187]: pgmap v1013: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T00:34:43.460 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:34:43 smithi193 ceph-mon[36878]: pgmap v1013: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T00:34:45.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:34:45 smithi118 ceph-mon[31535]: pgmap v1014: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T00:34:45.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:34:45 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 581.623374 secs 2024-03-24T00:34:45.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:34:45 smithi189 ceph-mon[36187]: pgmap v1014: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T00:34:45.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:34:45 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 581.623374 secs 2024-03-24T00:34:45.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:34:45 smithi193 ceph-mon[36878]: pgmap v1014: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T00:34:45.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:34:45 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 581.623374 secs 2024-03-24T00:34:47.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:34:47 smithi118 ceph-mon[31535]: pgmap v1015: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:34:47.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:34:47 smithi189 ceph-mon[36187]: pgmap v1015: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:34:47.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:34:47 smithi193 ceph-mon[36878]: pgmap v1015: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:34:49.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:34:49 smithi118 ceph-mon[31535]: pgmap v1016: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T00:34:49.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:34:49 smithi189 ceph-mon[36187]: pgmap v1016: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T00:34:49.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:34:49 smithi193 ceph-mon[36878]: pgmap v1016: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T00:34:50.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:34:50 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 586.623433 secs 2024-03-24T00:34:50.953 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:34:50 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 586.623433 secs 2024-03-24T00:34:50.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:34:50 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 586.623433 secs 2024-03-24T00:34:51.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:34:51 smithi118 ceph-mon[31535]: pgmap v1017: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T00:34:51.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:34:51 smithi189 ceph-mon[36187]: pgmap v1017: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T00:34:51.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:34:51 smithi193 ceph-mon[36878]: pgmap v1017: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T00:34:53.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:34:53 smithi118 ceph-mon[31535]: pgmap v1018: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T00:34:53.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:34:53 smithi189 ceph-mon[36187]: pgmap v1018: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T00:34:53.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:34:53 smithi193 ceph-mon[36878]: pgmap v1018: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T00:34:55.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:34:55 smithi118 ceph-mon[31535]: pgmap v1019: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T00:34:55.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:34:55 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 591.623493 secs 2024-03-24T00:34:55.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:34:55 smithi189 ceph-mon[36187]: pgmap v1019: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T00:34:55.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:34:55 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 591.623493 secs 2024-03-24T00:34:55.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:34:55 smithi193 ceph-mon[36878]: pgmap v1019: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T00:34:55.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:34:55 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 591.623493 secs 2024-03-24T00:34:57.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:34:57 smithi189 ceph-mon[36187]: pgmap v1020: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:34:57.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:34:57 smithi193 ceph-mon[36878]: pgmap v1020: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:34:58.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:34:57 smithi118 ceph-mon[31535]: pgmap v1020: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:34:59.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:34:59 smithi189 ceph-mon[36187]: pgmap v1021: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T00:34:59.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:34:59 smithi193 ceph-mon[36878]: pgmap v1021: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T00:35:00.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:34:59 smithi118 ceph-mon[31535]: pgmap v1021: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T00:35:00.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:35:00 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 596.623554 secs 2024-03-24T00:35:01.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:35:00 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 596.623554 secs 2024-03-24T00:35:01.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:35:00 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 596.623554 secs 2024-03-24T00:35:02.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:35:01 smithi118 ceph-mon[31535]: pgmap v1022: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T00:35:02.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:35:01 smithi189 ceph-mon[36187]: pgmap v1022: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T00:35:02.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:35:01 smithi193 ceph-mon[36878]: pgmap v1022: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T00:35:04.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:35:03 smithi118 ceph-mon[31535]: pgmap v1023: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T00:35:04.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:35:03 smithi189 ceph-mon[36187]: pgmap v1023: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T00:35:04.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:35:03 smithi193 ceph-mon[36878]: pgmap v1023: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T00:35:06.013 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:35:05 smithi193 ceph-mon[36878]: pgmap v1024: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-03-24T00:35:06.013 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:35:05 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 601.623571 secs 2024-03-24T00:35:06.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:35:05 smithi118 ceph-mon[31535]: pgmap v1024: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-03-24T00:35:06.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:35:05 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 601.623571 secs 2024-03-24T00:35:06.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:35:05 smithi189 ceph-mon[36187]: pgmap v1024: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-03-24T00:35:06.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:35:05 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 601.623571 secs 2024-03-24T00:35:08.005 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:35:07 smithi189 ceph-mon[36187]: pgmap v1025: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:35:08.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:35:07 smithi118 ceph-mon[31535]: pgmap v1025: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:35:08.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:35:07 smithi193 ceph-mon[36878]: pgmap v1025: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:35:10.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:35:09 smithi118 ceph-mon[31535]: pgmap v1026: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T00:35:10.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:35:09 smithi189 ceph-mon[36187]: pgmap v1026: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T00:35:10.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:35:09 smithi193 ceph-mon[36878]: pgmap v1026: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T00:35:11.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:35:10 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 606.623632 secs 2024-03-24T00:35:11.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:35:10 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 606.623632 secs 2024-03-24T00:35:11.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:35:10 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 606.623632 secs 2024-03-24T00:35:12.024 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:35:11 smithi118 ceph-mon[31535]: pgmap v1027: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T00:35:12.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:35:11 smithi189 ceph-mon[36187]: pgmap v1027: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T00:35:12.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:35:11 smithi193 ceph-mon[36878]: pgmap v1027: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T00:35:14.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:35:13 smithi118 ceph-mon[31535]: pgmap v1028: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T00:35:14.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:35:13 smithi189 ceph-mon[36187]: pgmap v1028: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T00:35:14.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:35:13 smithi193 ceph-mon[36878]: pgmap v1028: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T00:35:16.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:35:15 smithi118 ceph-mon[31535]: pgmap v1029: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-03-24T00:35:16.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:35:15 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 611.623720 secs 2024-03-24T00:35:16.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:35:15 smithi189 ceph-mon[36187]: pgmap v1029: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-03-24T00:35:16.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:35:15 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 611.623720 secs 2024-03-24T00:35:16.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:35:15 smithi193 ceph-mon[36878]: pgmap v1029: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-03-24T00:35:16.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:35:15 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 611.623720 secs 2024-03-24T00:35:18.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:35:17 smithi118 ceph-mon[31535]: pgmap v1030: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:35:18.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:35:17 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T00:35:18.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:35:17 smithi189 ceph-mon[36187]: pgmap v1030: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:35:18.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:35:17 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T00:35:18.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:35:17 smithi193 ceph-mon[36878]: pgmap v1030: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:35:18.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:35:17 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T00:35:20.197 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:35:19 smithi118 ceph-mon[31535]: pgmap v1031: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T00:35:20.197 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:35:19 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T00:35:20.197 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:35:19 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T00:35:20.197 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:35:19 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T00:35:20.197 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:35:19 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T00:35:20.197 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:35:19 smithi118 ceph-mon[31535]: pgmap v1032: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T00:35:20.197 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:35:19 smithi118 ceph-mon[31535]: pgmap v1033: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 5 op/s 2024-03-24T00:35:20.197 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:35:19 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T00:35:20.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:35:19 smithi189 ceph-mon[36187]: pgmap v1031: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T00:35:20.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:35:19 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T00:35:20.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:35:19 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T00:35:20.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:35:19 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T00:35:20.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:35:19 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T00:35:20.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:35:19 smithi189 ceph-mon[36187]: pgmap v1032: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T00:35:20.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:35:19 smithi189 ceph-mon[36187]: pgmap v1033: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 5 op/s 2024-03-24T00:35:20.205 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:35:19 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T00:35:20.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:35:19 smithi193 ceph-mon[36878]: pgmap v1031: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T00:35:20.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:35:19 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T00:35:20.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:35:19 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T00:35:20.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:35:19 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T00:35:20.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:35:19 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T00:35:20.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:35:19 smithi193 ceph-mon[36878]: pgmap v1032: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T00:35:20.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:35:19 smithi193 ceph-mon[36878]: pgmap v1033: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 5 op/s 2024-03-24T00:35:20.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:35:19 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T00:35:21.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:35:20 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 616.623804 secs 2024-03-24T00:35:21.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:35:20 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 616.623804 secs 2024-03-24T00:35:21.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:35:20 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 616.623804 secs 2024-03-24T00:35:22.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:35:21 smithi118 ceph-mon[31535]: pgmap v1034: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 6 op/s 2024-03-24T00:35:22.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:35:21 smithi189 ceph-mon[36187]: pgmap v1034: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 6 op/s 2024-03-24T00:35:22.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:35:21 smithi193 ceph-mon[36878]: pgmap v1034: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 6 op/s 2024-03-24T00:35:24.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:35:23 smithi193 ceph-mon[36878]: pgmap v1035: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 4 op/s 2024-03-24T00:35:24.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:35:23 smithi118 ceph-mon[31535]: pgmap v1035: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 4 op/s 2024-03-24T00:35:24.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:35:23 smithi189 ceph-mon[36187]: pgmap v1035: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 4 op/s 2024-03-24T00:35:26.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:35:25 smithi118 ceph-mon[31535]: pgmap v1036: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-03-24T00:35:26.447 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:35:25 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 621.623888 secs 2024-03-24T00:35:26.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:35:25 smithi189 ceph-mon[36187]: pgmap v1036: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-03-24T00:35:26.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:35:25 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 621.623888 secs 2024-03-24T00:35:26.460 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:35:25 smithi193 ceph-mon[36878]: pgmap v1036: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-03-24T00:35:26.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:35:25 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 621.623888 secs 2024-03-24T00:35:28.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:35:27 smithi118 ceph-mon[31535]: pgmap v1037: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 5 op/s 2024-03-24T00:35:28.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:35:27 smithi189 ceph-mon[36187]: pgmap v1037: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 5 op/s 2024-03-24T00:35:28.460 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:35:27 smithi193 ceph-mon[36878]: pgmap v1037: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 5 op/s 2024-03-24T00:35:30.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:35:29 smithi118 ceph-mon[31535]: pgmap v1038: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 5 op/s 2024-03-24T00:35:30.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:35:29 smithi189 ceph-mon[36187]: pgmap v1038: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 5 op/s 2024-03-24T00:35:30.460 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:35:29 smithi193 ceph-mon[36878]: pgmap v1038: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 5 op/s 2024-03-24T00:35:31.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:35:30 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 626.623975 secs 2024-03-24T00:35:31.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:35:30 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 626.623975 secs 2024-03-24T00:35:31.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:35:30 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 626.623975 secs 2024-03-24T00:35:32.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:35:31 smithi118 ceph-mon[31535]: pgmap v1039: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T00:35:32.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:35:31 smithi189 ceph-mon[36187]: pgmap v1039: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T00:35:32.460 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:35:31 smithi193 ceph-mon[36878]: pgmap v1039: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T00:35:34.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:35:33 smithi118 ceph-mon[31535]: pgmap v1040: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 5 op/s 2024-03-24T00:35:34.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:35:33 smithi189 ceph-mon[36187]: pgmap v1040: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 5 op/s 2024-03-24T00:35:34.460 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:35:33 smithi193 ceph-mon[36878]: pgmap v1040: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 5 op/s 2024-03-24T00:35:36.263 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:35:35 smithi193 ceph-mon[36878]: pgmap v1041: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T00:35:36.263 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:35:35 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 631.624016 secs 2024-03-24T00:35:36.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:35:35 smithi118 ceph-mon[31535]: pgmap v1041: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T00:35:36.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:35:35 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 631.624016 secs 2024-03-24T00:35:36.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:35:35 smithi189 ceph-mon[36187]: pgmap v1041: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T00:35:36.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:35:35 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 631.624016 secs 2024-03-24T00:35:38.256 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:35:38 smithi189 ceph-mon[36187]: pgmap v1042: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:35:38.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:35:38 smithi118 ceph-mon[31535]: pgmap v1042: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:35:38.460 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:35:38 smithi193 ceph-mon[36878]: pgmap v1042: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:35:40.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:35:40 smithi118 ceph-mon[31535]: pgmap v1043: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T00:35:40.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:35:40 smithi189 ceph-mon[36187]: pgmap v1043: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T00:35:40.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:35:40 smithi193 ceph-mon[36878]: pgmap v1043: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T00:35:41.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:35:41 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 636.624066 secs 2024-03-24T00:35:41.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:35:41 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 636.624066 secs 2024-03-24T00:35:41.460 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:35:41 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 636.624066 secs 2024-03-24T00:35:42.348 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:35:42 smithi118 ceph-mon[31535]: pgmap v1044: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T00:35:42.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:35:42 smithi189 ceph-mon[36187]: pgmap v1044: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T00:35:42.460 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:35:42 smithi193 ceph-mon[36878]: pgmap v1044: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T00:35:43.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:35:43 smithi118 ceph-mon[31535]: pgmap v1045: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 5 op/s 2024-03-24T00:35:43.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:35:43 smithi189 ceph-mon[36187]: pgmap v1045: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 5 op/s 2024-03-24T00:35:43.460 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:35:43 smithi193 ceph-mon[36878]: pgmap v1045: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 5 op/s 2024-03-24T00:35:46.329 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:35:45 smithi118 ceph-mon[31535]: pgmap v1046: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T00:35:46.329 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:35:45 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 641.624153 secs 2024-03-24T00:35:46.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:35:45 smithi189 ceph-mon[36187]: pgmap v1046: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T00:35:46.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:35:45 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 641.624153 secs 2024-03-24T00:35:46.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:35:45 smithi193 ceph-mon[36878]: pgmap v1046: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T00:35:46.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:35:45 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 641.624153 secs 2024-03-24T00:35:48.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:35:47 smithi118 ceph-mon[31535]: pgmap v1047: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:35:48.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:35:47 smithi189 ceph-mon[36187]: pgmap v1047: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:35:48.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:35:47 smithi193 ceph-mon[36878]: pgmap v1047: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:35:50.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:35:49 smithi118 ceph-mon[31535]: pgmap v1048: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T00:35:50.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:35:49 smithi189 ceph-mon[36187]: pgmap v1048: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T00:35:50.460 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:35:49 smithi193 ceph-mon[36878]: pgmap v1048: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T00:35:51.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:35:50 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 646.624210 secs 2024-03-24T00:35:51.453 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:35:50 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 646.624210 secs 2024-03-24T00:35:51.460 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:35:50 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 646.624210 secs 2024-03-24T00:35:52.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:35:51 smithi118 ceph-mon[31535]: pgmap v1049: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T00:35:52.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:35:51 smithi189 ceph-mon[36187]: pgmap v1049: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T00:35:52.460 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:35:51 smithi193 ceph-mon[36878]: pgmap v1049: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T00:35:54.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:35:53 smithi118 ceph-mon[31535]: pgmap v1050: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 5 op/s 2024-03-24T00:35:54.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:35:53 smithi189 ceph-mon[36187]: pgmap v1050: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 5 op/s 2024-03-24T00:35:54.460 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:35:53 smithi193 ceph-mon[36878]: pgmap v1050: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 5 op/s 2024-03-24T00:35:56.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:35:56 smithi118 ceph-mon[31535]: pgmap v1051: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T00:35:56.447 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:35:56 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 651.624246 secs 2024-03-24T00:35:56.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:35:56 smithi189 ceph-mon[36187]: pgmap v1051: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T00:35:56.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:35:56 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 651.624246 secs 2024-03-24T00:35:56.460 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:35:56 smithi193 ceph-mon[36878]: pgmap v1051: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T00:35:56.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:35:56 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 651.624246 secs 2024-03-24T00:35:58.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:35:58 smithi118 ceph-mon[31535]: pgmap v1052: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:35:58.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:35:58 smithi189 ceph-mon[36187]: pgmap v1052: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:35:58.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:35:58 smithi193 ceph-mon[36878]: pgmap v1052: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:36:00.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:36:00 smithi118 ceph-mon[31535]: pgmap v1053: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T00:36:00.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:36:00 smithi189 ceph-mon[36187]: pgmap v1053: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T00:36:00.460 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:36:00 smithi193 ceph-mon[36878]: pgmap v1053: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T00:36:01.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:36:01 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 656.624327 secs 2024-03-24T00:36:01.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:36:01 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 656.624327 secs 2024-03-24T00:36:01.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:36:01 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 656.624327 secs 2024-03-24T00:36:02.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:36:02 smithi189 ceph-mon[36187]: pgmap v1054: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 6 op/s 2024-03-24T00:36:02.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:36:02 smithi118 ceph-mon[31535]: pgmap v1054: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 6 op/s 2024-03-24T00:36:02.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:36:02 smithi193 ceph-mon[36878]: pgmap v1054: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 6 op/s 2024-03-24T00:36:03.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:36:03 smithi118 ceph-mon[31535]: pgmap v1055: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 5 op/s 2024-03-24T00:36:03.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:36:03 smithi189 ceph-mon[36187]: pgmap v1055: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 5 op/s 2024-03-24T00:36:03.460 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:36:03 smithi193 ceph-mon[36878]: pgmap v1055: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 5 op/s 2024-03-24T00:36:06.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:36:05 smithi118 ceph-mon[31535]: pgmap v1056: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 6 op/s 2024-03-24T00:36:06.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:36:05 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 661.624389 secs 2024-03-24T00:36:06.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:36:05 smithi189 ceph-mon[36187]: pgmap v1056: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 6 op/s 2024-03-24T00:36:06.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:36:05 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 661.624389 secs 2024-03-24T00:36:06.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:36:05 smithi193 ceph-mon[36878]: pgmap v1056: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 6 op/s 2024-03-24T00:36:06.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:36:05 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 661.624389 secs 2024-03-24T00:36:08.255 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:36:07 smithi189 ceph-mon[36187]: pgmap v1057: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-03-24T00:36:08.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:36:07 smithi118 ceph-mon[31535]: pgmap v1057: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-03-24T00:36:08.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:36:07 smithi193 ceph-mon[36878]: pgmap v1057: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-03-24T00:36:10.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:36:09 smithi118 ceph-mon[31535]: pgmap v1058: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-03-24T00:36:10.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:36:09 smithi189 ceph-mon[36187]: pgmap v1058: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-03-24T00:36:10.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:36:09 smithi193 ceph-mon[36878]: pgmap v1058: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-03-24T00:36:11.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:36:10 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 666.624479 secs 2024-03-24T00:36:11.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:36:10 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 666.624479 secs 2024-03-24T00:36:11.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:36:10 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 666.624479 secs 2024-03-24T00:36:12.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:36:12 smithi118 ceph-mon[31535]: pgmap v1059: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 6 op/s 2024-03-24T00:36:12.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:36:12 smithi189 ceph-mon[36187]: pgmap v1059: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 6 op/s 2024-03-24T00:36:12.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:36:12 smithi193 ceph-mon[36878]: pgmap v1059: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 6 op/s 2024-03-24T00:36:14.249 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:36:14 smithi118 ceph-mon[31535]: pgmap v1060: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 5 op/s 2024-03-24T00:36:14.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:36:14 smithi189 ceph-mon[36187]: pgmap v1060: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 5 op/s 2024-03-24T00:36:14.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:36:14 smithi193 ceph-mon[36878]: pgmap v1060: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 5 op/s 2024-03-24T00:36:16.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:36:16 smithi118 ceph-mon[31535]: pgmap v1061: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T00:36:16.447 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:36:16 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 671.624522 secs 2024-03-24T00:36:16.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:36:16 smithi189 ceph-mon[36187]: pgmap v1061: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T00:36:16.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:36:16 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 671.624522 secs 2024-03-24T00:36:16.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:36:16 smithi193 ceph-mon[36878]: pgmap v1061: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T00:36:16.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:36:16 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 671.624522 secs 2024-03-24T00:36:18.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:36:18 smithi118 ceph-mon[31535]: pgmap v1062: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:36:18.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:36:18 smithi189 ceph-mon[36187]: pgmap v1062: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:36:18.460 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:36:18 smithi193 ceph-mon[36878]: pgmap v1062: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:36:20.369 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:36:20 smithi193 ceph-mon[36878]: pgmap v1063: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T00:36:20.369 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:36:20 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T00:36:20.369 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:36:20 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T00:36:20.369 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:36:20 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T00:36:20.369 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:36:20 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T00:36:20.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:36:20 smithi118 ceph-mon[31535]: pgmap v1063: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T00:36:20.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:36:20 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T00:36:20.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:36:20 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T00:36:20.447 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:36:20 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T00:36:20.447 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:36:20 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T00:36:20.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:36:20 smithi189 ceph-mon[36187]: pgmap v1063: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T00:36:20.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:36:20 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T00:36:20.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:36:20 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T00:36:20.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:36:20 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T00:36:20.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:36:20 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T00:36:21.299 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:36:21 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T00:36:21.300 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:36:21 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 676.624548 secs 2024-03-24T00:36:21.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:36:21 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T00:36:21.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:36:21 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 676.624548 secs 2024-03-24T00:36:21.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:36:21 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T00:36:21.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:36:21 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 676.624548 secs 2024-03-24T00:36:22.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:36:22 smithi118 ceph-mon[31535]: pgmap v1064: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T00:36:22.447 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:36:22 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T00:36:22.447 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:36:22 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T00:36:22.447 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:36:22 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T00:36:22.447 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:36:22 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T00:36:22.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:36:22 smithi189 ceph-mon[36187]: pgmap v1064: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T00:36:22.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:36:22 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T00:36:22.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:36:22 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T00:36:22.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:36:22 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T00:36:22.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:36:22 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T00:36:22.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:36:22 smithi193 ceph-mon[36878]: pgmap v1064: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T00:36:22.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:36:22 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T00:36:22.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:36:22 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T00:36:22.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:36:22 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T00:36:22.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:36:22 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T00:36:23.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:36:23 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T00:36:23.947 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:36:23 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T00:36:23.947 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:36:23 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T00:36:23.947 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:36:23 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T00:36:23.947 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:36:23 smithi118 ceph-mon[31535]: pgmap v1065: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 5 op/s 2024-03-24T00:36:23.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:36:23 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T00:36:23.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:36:23 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T00:36:23.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:36:23 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T00:36:23.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:36:23 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T00:36:23.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:36:23 smithi189 ceph-mon[36187]: pgmap v1065: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 5 op/s 2024-03-24T00:36:23.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:36:23 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T00:36:23.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:36:23 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T00:36:23.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:36:23 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T00:36:23.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:36:23 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T00:36:23.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:36:23 smithi193 ceph-mon[36878]: pgmap v1065: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 5 op/s 2024-03-24T00:36:25.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:36:24 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T00:36:25.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:36:24 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T00:36:25.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:36:24 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T00:36:25.447 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:36:24 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T00:36:25.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:36:24 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T00:36:25.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:36:24 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T00:36:26.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:36:25 smithi118 ceph-mon[31535]: pgmap v1066: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T00:36:26.447 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:36:25 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 681.624632 secs 2024-03-24T00:36:26.447 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:36:25 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T00:36:26.447 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:36:25 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T00:36:26.447 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:36:25 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T00:36:26.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:36:25 smithi189 ceph-mon[36187]: pgmap v1066: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T00:36:26.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:36:25 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 681.624632 secs 2024-03-24T00:36:26.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:36:25 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T00:36:26.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:36:25 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T00:36:26.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:36:25 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T00:36:26.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:36:25 smithi193 ceph-mon[36878]: pgmap v1066: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T00:36:26.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:36:25 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 681.624632 secs 2024-03-24T00:36:26.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:36:25 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T00:36:26.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:36:25 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T00:36:26.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:36:25 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T00:36:27.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:36:26 smithi118 ceph-mon[31535]: pgmap v1067: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T00:36:27.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:36:26 smithi118 ceph-mon[31535]: pgmap v1068: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-03-24T00:36:27.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:36:26 smithi189 ceph-mon[36187]: pgmap v1067: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T00:36:27.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:36:26 smithi189 ceph-mon[36187]: pgmap v1068: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-03-24T00:36:27.460 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:36:26 smithi193 ceph-mon[36878]: pgmap v1067: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T00:36:27.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:36:26 smithi193 ceph-mon[36878]: pgmap v1068: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-03-24T00:36:29.336 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:36:28 smithi118 ceph-mon[31535]: pgmap v1069: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 5 op/s 2024-03-24T00:36:29.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:36:28 smithi189 ceph-mon[36187]: pgmap v1069: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 5 op/s 2024-03-24T00:36:29.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:36:28 smithi193 ceph-mon[36878]: pgmap v1069: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 5 op/s 2024-03-24T00:36:31.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:36:30 smithi118 ceph-mon[31535]: pgmap v1070: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-03-24T00:36:31.447 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:36:30 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 686.624721 secs 2024-03-24T00:36:31.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:36:30 smithi189 ceph-mon[36187]: pgmap v1070: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-03-24T00:36:31.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:36:30 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 686.624721 secs 2024-03-24T00:36:31.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:36:30 smithi193 ceph-mon[36878]: pgmap v1070: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-03-24T00:36:31.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:36:30 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 686.624721 secs 2024-03-24T00:36:33.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:36:32 smithi118 ceph-mon[31535]: pgmap v1071: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 6 op/s 2024-03-24T00:36:33.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:36:32 smithi189 ceph-mon[36187]: pgmap v1071: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 6 op/s 2024-03-24T00:36:33.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:36:32 smithi193 ceph-mon[36878]: pgmap v1071: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 6 op/s 2024-03-24T00:36:35.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:36:35 smithi118 ceph-mon[31535]: pgmap v1072: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 5 op/s 2024-03-24T00:36:35.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:36:35 smithi189 ceph-mon[36187]: pgmap v1072: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 5 op/s 2024-03-24T00:36:35.460 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:36:35 smithi193 ceph-mon[36878]: pgmap v1072: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 5 op/s 2024-03-24T00:36:36.362 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:36:36 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 691.624808 secs 2024-03-24T00:36:36.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:36:36 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 691.624808 secs 2024-03-24T00:36:36.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:36:36 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 691.624808 secs 2024-03-24T00:36:37.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:36:37 smithi118 ceph-mon[31535]: pgmap v1073: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 5 op/s 2024-03-24T00:36:37.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:36:37 smithi189 ceph-mon[36187]: pgmap v1073: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 5 op/s 2024-03-24T00:36:37.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:36:37 smithi193 ceph-mon[36878]: pgmap v1073: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 5 op/s 2024-03-24T00:36:38.355 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:36:38 smithi189 ceph-mon[36187]: pgmap v1074: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:36:38.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:36:38 smithi118 ceph-mon[31535]: pgmap v1074: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:36:38.460 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:36:38 smithi193 ceph-mon[36878]: pgmap v1074: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:36:40.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:36:40 smithi118 ceph-mon[31535]: pgmap v1075: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-24T00:36:40.697 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:36:40 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 696.624867 secs 2024-03-24T00:36:40.704 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:36:40 smithi189 ceph-mon[36187]: pgmap v1075: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-24T00:36:40.704 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:36:40 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 696.624867 secs 2024-03-24T00:36:40.711 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:36:40 smithi193 ceph-mon[36878]: pgmap v1075: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-24T00:36:40.711 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:36:40 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 696.624867 secs 2024-03-24T00:36:42.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:36:42 smithi118 ceph-mon[31535]: pgmap v1076: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:36:42.704 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:36:42 smithi189 ceph-mon[36187]: pgmap v1076: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:36:42.711 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:36:42 smithi193 ceph-mon[36878]: pgmap v1076: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:36:44.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:36:44 smithi118 ceph-mon[31535]: pgmap v1077: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T00:36:44.704 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:36:44 smithi189 ceph-mon[36187]: pgmap v1077: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T00:36:44.711 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:36:44 smithi193 ceph-mon[36878]: pgmap v1077: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T00:36:45.598 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:36:45 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 701.624952 secs 2024-03-24T00:36:45.704 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:36:45 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 701.624952 secs 2024-03-24T00:36:45.710 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:36:45 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 701.624952 secs 2024-03-24T00:36:46.625 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:36:46 smithi118 ceph-mon[31535]: pgmap v1078: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T00:36:46.704 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:36:46 smithi189 ceph-mon[36187]: pgmap v1078: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T00:36:46.711 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:36:46 smithi193 ceph-mon[36878]: pgmap v1078: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T00:36:48.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:36:48 smithi118 ceph-mon[31535]: pgmap v1079: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:36:48.704 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:36:48 smithi189 ceph-mon[36187]: pgmap v1079: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:36:48.711 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:36:48 smithi193 ceph-mon[36878]: pgmap v1079: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:36:50.698 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:36:50 smithi118 ceph-mon[31535]: pgmap v1080: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-24T00:36:50.699 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:36:50 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 706.625042 secs 2024-03-24T00:36:50.704 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:36:50 smithi189 ceph-mon[36187]: pgmap v1080: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-24T00:36:50.704 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:36:50 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 706.625042 secs 2024-03-24T00:36:50.711 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:36:50 smithi193 ceph-mon[36878]: pgmap v1080: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-24T00:36:50.711 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:36:50 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 706.625042 secs 2024-03-24T00:36:52.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:36:52 smithi118 ceph-mon[31535]: pgmap v1081: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:36:52.704 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:36:52 smithi189 ceph-mon[36187]: pgmap v1081: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:36:52.711 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:36:52 smithi193 ceph-mon[36878]: pgmap v1081: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:36:54.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:36:54 smithi118 ceph-mon[31535]: pgmap v1082: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T00:36:54.704 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:36:54 smithi189 ceph-mon[36187]: pgmap v1082: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T00:36:54.710 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:36:54 smithi193 ceph-mon[36878]: pgmap v1082: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T00:36:55.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:36:55 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 711.625126 secs 2024-03-24T00:36:55.704 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:36:55 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 711.625126 secs 2024-03-24T00:36:55.710 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:36:55 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 711.625126 secs 2024-03-24T00:36:56.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:36:56 smithi118 ceph-mon[31535]: pgmap v1083: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T00:36:56.704 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:36:56 smithi189 ceph-mon[36187]: pgmap v1083: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T00:36:56.710 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:36:56 smithi193 ceph-mon[36878]: pgmap v1083: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T00:36:58.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:36:58 smithi118 ceph-mon[31535]: pgmap v1084: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:36:58.704 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:36:58 smithi189 ceph-mon[36187]: pgmap v1084: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:36:58.711 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:36:58 smithi193 ceph-mon[36878]: pgmap v1084: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:37:00.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:37:00 smithi118 ceph-mon[31535]: pgmap v1085: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-24T00:37:00.697 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:37:00 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 716.625209 secs 2024-03-24T00:37:00.704 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:37:00 smithi189 ceph-mon[36187]: pgmap v1085: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-24T00:37:00.704 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:37:00 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 716.625209 secs 2024-03-24T00:37:00.711 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:37:00 smithi193 ceph-mon[36878]: pgmap v1085: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-24T00:37:00.711 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:37:00 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 716.625209 secs 2024-03-24T00:37:02.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:37:02 smithi118 ceph-mon[31535]: pgmap v1086: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:37:02.704 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:37:02 smithi189 ceph-mon[36187]: pgmap v1086: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:37:02.710 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:37:02 smithi193 ceph-mon[36878]: pgmap v1086: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:37:04.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:37:04 smithi118 ceph-mon[31535]: pgmap v1087: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T00:37:04.704 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:37:04 smithi189 ceph-mon[36187]: pgmap v1087: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T00:37:04.711 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:37:04 smithi193 ceph-mon[36878]: pgmap v1087: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T00:37:05.697 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:37:05 smithi118 ceph-mon[31535]: 3 slow requests, 3 included below; oldest blocked for > 721.625246 secs 2024-03-24T00:37:05.697 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:37:05 smithi118 ceph-mon[31535]: slow request 721.625246 seconds old, received at 2024-03-24T00:25:03.572253+0000: peer_request:client.15159:154789 currently acquired locks 2024-03-24T00:37:05.697 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:37:05 smithi118 ceph-mon[31535]: slow request 721.573999 seconds old, received at 2024-03-24T00:25:03.623500+0000: peer_request:client.15159:158090 currently acquired locks 2024-03-24T00:37:05.697 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:37:05 smithi118 ceph-mon[31535]: slow request 721.567321 seconds old, received at 2024-03-24T00:25:03.630178+0000: client_request(client.15159:173196 rename #0x100000010bf/comment-15.xml #0x100000010bf/comment-15.xml.tmp 2024-03-24T00:25:03.629366+0000 caller_uid=1000, caller_gid=1301{6,36,1000,1301,}) currently failed to xlock, waiting 2024-03-24T00:37:05.704 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:37:05 smithi189 ceph-mon[36187]: 3 slow requests, 3 included below; oldest blocked for > 721.625246 secs 2024-03-24T00:37:05.704 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:37:05 smithi189 ceph-mon[36187]: slow request 721.625246 seconds old, received at 2024-03-24T00:25:03.572253+0000: peer_request:client.15159:154789 currently acquired locks 2024-03-24T00:37:05.704 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:37:05 smithi189 ceph-mon[36187]: slow request 721.573999 seconds old, received at 2024-03-24T00:25:03.623500+0000: peer_request:client.15159:158090 currently acquired locks 2024-03-24T00:37:05.704 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:37:05 smithi189 ceph-mon[36187]: slow request 721.567321 seconds old, received at 2024-03-24T00:25:03.630178+0000: client_request(client.15159:173196 rename #0x100000010bf/comment-15.xml #0x100000010bf/comment-15.xml.tmp 2024-03-24T00:25:03.629366+0000 caller_uid=1000, caller_gid=1301{6,36,1000,1301,}) currently failed to xlock, waiting 2024-03-24T00:37:05.710 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:37:05 smithi193 ceph-mon[36878]: 3 slow requests, 3 included below; oldest blocked for > 721.625246 secs 2024-03-24T00:37:05.711 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:37:05 smithi193 ceph-mon[36878]: slow request 721.625246 seconds old, received at 2024-03-24T00:25:03.572253+0000: peer_request:client.15159:154789 currently acquired locks 2024-03-24T00:37:05.711 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:37:05 smithi193 ceph-mon[36878]: slow request 721.573999 seconds old, received at 2024-03-24T00:25:03.623500+0000: peer_request:client.15159:158090 currently acquired locks 2024-03-24T00:37:05.711 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:37:05 smithi193 ceph-mon[36878]: slow request 721.567321 seconds old, received at 2024-03-24T00:25:03.630178+0000: client_request(client.15159:173196 rename #0x100000010bf/comment-15.xml #0x100000010bf/comment-15.xml.tmp 2024-03-24T00:25:03.629366+0000 caller_uid=1000, caller_gid=1301{6,36,1000,1301,}) currently failed to xlock, waiting 2024-03-24T00:37:06.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:37:06 smithi118 ceph-mon[31535]: pgmap v1088: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T00:37:06.704 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:37:06 smithi189 ceph-mon[36187]: pgmap v1088: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T00:37:06.710 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:37:06 smithi193 ceph-mon[36878]: pgmap v1088: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T00:37:08.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:37:08 smithi118 ceph-mon[31535]: pgmap v1089: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:37:08.704 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:37:08 smithi189 ceph-mon[36187]: pgmap v1089: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:37:08.711 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:37:08 smithi193 ceph-mon[36878]: pgmap v1089: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:37:10.697 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:37:10 smithi118 ceph-mon[31535]: pgmap v1090: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-24T00:37:10.697 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:37:10 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 726.625258 secs 2024-03-24T00:37:10.704 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:37:10 smithi189 ceph-mon[36187]: pgmap v1090: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-24T00:37:10.704 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:37:10 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 726.625258 secs 2024-03-24T00:37:10.711 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:37:10 smithi193 ceph-mon[36878]: pgmap v1090: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-24T00:37:10.711 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:37:10 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 726.625258 secs 2024-03-24T00:37:12.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:37:12 smithi118 ceph-mon[31535]: pgmap v1091: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:37:12.704 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:37:12 smithi189 ceph-mon[36187]: pgmap v1091: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:37:12.710 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:37:12 smithi193 ceph-mon[36878]: pgmap v1091: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:37:14.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:37:14 smithi118 ceph-mon[31535]: pgmap v1092: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T00:37:14.704 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:37:14 smithi189 ceph-mon[36187]: pgmap v1092: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T00:37:14.710 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:37:14 smithi193 ceph-mon[36878]: pgmap v1092: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T00:37:15.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:37:15 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 731.625342 secs 2024-03-24T00:37:15.704 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:37:15 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 731.625342 secs 2024-03-24T00:37:15.711 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:37:15 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 731.625342 secs 2024-03-24T00:37:15.839 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:37:15.839+0000 7f81ee000640 1 -- 172.21.15.118:0/1693767697 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f81c80060e0 con 0x7f81f81052d0 2024-03-24T00:37:15.929 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:37:15.839+0000 7f81ef400640 1 -- 172.21.15.118:0/1693767697 <== mon.2 v2:172.21.15.189:3300/0 10 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (secure 0 0 0) 0x7f81c80060e0 con 0x7f81f81052d0 2024-03-24T00:37:15.929 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:37:15.862+0000 7f81db400640 1 -- 172.21.15.118:0/659686424 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f81c4006040 con 0x5622651a13f0 2024-03-24T00:37:15.929 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:37:15.863+0000 7f81e4c00640 1 -- 172.21.15.118:0/659686424 <== mon.1 v2:172.21.15.193:3300/0 10 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 409+0+0 (secure 0 0 0) 0x7f81e8090100 con 0x5622651a13f0 2024-03-24T00:37:16.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:37:16 smithi118 ceph-mon[31535]: pgmap v1093: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T00:37:16.704 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:37:16 smithi189 ceph-mon[36187]: pgmap v1093: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T00:37:16.711 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:37:16 smithi193 ceph-mon[36878]: pgmap v1093: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T00:37:18.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:37:18 smithi118 ceph-mon[31535]: pgmap v1094: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:37:18.704 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:37:18 smithi189 ceph-mon[36187]: pgmap v1094: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:37:18.710 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:37:18 smithi193 ceph-mon[36878]: pgmap v1094: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:37:20.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:37:20 smithi118 ceph-mon[31535]: pgmap v1095: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-24T00:37:20.697 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:37:20 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 736.625405 secs 2024-03-24T00:37:20.704 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:37:20 smithi189 ceph-mon[36187]: pgmap v1095: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-24T00:37:20.704 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:37:20 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 736.625405 secs 2024-03-24T00:37:20.711 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:37:20 smithi193 ceph-mon[36878]: pgmap v1095: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-24T00:37:20.711 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:37:20 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 736.625405 secs 2024-03-24T00:37:22.704 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:37:22 smithi189 ceph-mon[36187]: pgmap v1096: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:37:22.710 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:37:22 smithi193 ceph-mon[36878]: pgmap v1096: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:37:22.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:37:22 smithi118 ceph-mon[31535]: pgmap v1096: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:37:24.704 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:37:24 smithi189 ceph-mon[36187]: pgmap v1097: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T00:37:24.710 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:37:24 smithi193 ceph-mon[36878]: pgmap v1097: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T00:37:24.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:37:24 smithi118 ceph-mon[31535]: pgmap v1097: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T00:37:25.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:37:25 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 741.625465 secs 2024-03-24T00:37:25.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:37:25 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 741.625465 secs 2024-03-24T00:37:25.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:37:25 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 741.625465 secs 2024-03-24T00:37:26.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:37:26 smithi118 ceph-mon[31535]: pgmap v1098: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T00:37:26.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:37:26 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T00:37:26.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:37:26 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T00:37:26.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:37:26 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T00:37:26.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:37:26 smithi189 ceph-mon[36187]: pgmap v1098: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T00:37:26.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:37:26 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T00:37:26.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:37:26 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T00:37:26.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:37:26 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T00:37:26.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:37:26 smithi193 ceph-mon[36878]: pgmap v1098: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T00:37:26.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:37:26 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T00:37:26.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:37:26 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T00:37:26.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:37:26 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T00:37:27.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:37:27 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T00:37:27.947 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:37:27 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T00:37:27.947 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:37:27 smithi118 ceph-mon[31535]: pgmap v1099: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 5 op/s 2024-03-24T00:37:27.947 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:37:27 smithi118 ceph-mon[31535]: pgmap v1100: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 6 op/s 2024-03-24T00:37:27.947 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:37:27 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T00:37:27.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:37:27 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T00:37:27.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:37:27 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T00:37:27.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:37:27 smithi189 ceph-mon[36187]: pgmap v1099: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 5 op/s 2024-03-24T00:37:27.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:37:27 smithi189 ceph-mon[36187]: pgmap v1100: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 6 op/s 2024-03-24T00:37:27.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:37:27 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T00:37:27.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:37:27 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T00:37:27.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:37:27 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T00:37:27.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:37:27 smithi193 ceph-mon[36878]: pgmap v1099: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 5 op/s 2024-03-24T00:37:27.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:37:27 smithi193 ceph-mon[36878]: pgmap v1100: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 6 op/s 2024-03-24T00:37:27.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:37:27 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T00:37:30.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:37:29 smithi189 ceph-mon[36187]: pgmap v1101: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 4 op/s 2024-03-24T00:37:30.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:37:29 smithi193 ceph-mon[36878]: pgmap v1101: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 4 op/s 2024-03-24T00:37:30.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:37:29 smithi118 ceph-mon[31535]: pgmap v1101: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 4 op/s 2024-03-24T00:37:31.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:37:30 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 746.625556 secs 2024-03-24T00:37:31.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:37:30 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 746.625556 secs 2024-03-24T00:37:31.288 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:37:30 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 746.625556 secs 2024-03-24T00:37:32.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:37:31 smithi193 ceph-mon[36878]: pgmap v1102: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 6 op/s 2024-03-24T00:37:32.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:37:31 smithi118 ceph-mon[31535]: pgmap v1102: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 6 op/s 2024-03-24T00:37:32.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:37:31 smithi189 ceph-mon[36187]: pgmap v1102: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 6 op/s 2024-03-24T00:37:34.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:37:33 smithi118 ceph-mon[31535]: pgmap v1103: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 5 op/s 2024-03-24T00:37:34.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:37:33 smithi189 ceph-mon[36187]: pgmap v1103: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 5 op/s 2024-03-24T00:37:34.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:37:33 smithi193 ceph-mon[36878]: pgmap v1103: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 5 op/s 2024-03-24T00:37:36.263 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:37:35 smithi193 ceph-mon[36878]: pgmap v1104: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 6 op/s 2024-03-24T00:37:36.263 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:37:35 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 751.625616 secs 2024-03-24T00:37:36.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:37:35 smithi118 ceph-mon[31535]: pgmap v1104: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 6 op/s 2024-03-24T00:37:36.447 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:37:35 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 751.625616 secs 2024-03-24T00:37:36.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:37:35 smithi189 ceph-mon[36187]: pgmap v1104: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 6 op/s 2024-03-24T00:37:36.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:37:35 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 751.625616 secs 2024-03-24T00:37:38.256 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:37:37 smithi189 ceph-mon[36187]: pgmap v1105: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 5 op/s 2024-03-24T00:37:38.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:37:37 smithi118 ceph-mon[31535]: pgmap v1105: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 5 op/s 2024-03-24T00:37:38.460 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:37:37 smithi193 ceph-mon[36878]: pgmap v1105: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 5 op/s 2024-03-24T00:37:40.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:37:39 smithi118 ceph-mon[31535]: pgmap v1106: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T00:37:40.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:37:39 smithi189 ceph-mon[36187]: pgmap v1106: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T00:37:40.460 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:37:39 smithi193 ceph-mon[36878]: pgmap v1106: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T00:37:41.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:37:40 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 756.625705 secs 2024-03-24T00:37:41.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:37:40 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 756.625705 secs 2024-03-24T00:37:41.460 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:37:40 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 756.625705 secs 2024-03-24T00:37:42.459 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:37:41 smithi189 ceph-mon[36187]: pgmap v1107: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T00:37:42.459 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:37:41 smithi118 ceph-mon[31535]: pgmap v1107: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T00:37:42.460 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:37:41 smithi193 ceph-mon[36878]: pgmap v1107: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T00:37:44.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:37:43 smithi118 ceph-mon[31535]: pgmap v1108: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 5 op/s 2024-03-24T00:37:44.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:37:43 smithi189 ceph-mon[36187]: pgmap v1108: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 5 op/s 2024-03-24T00:37:44.460 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:37:43 smithi193 ceph-mon[36878]: pgmap v1108: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 5 op/s 2024-03-24T00:37:46.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:37:46 smithi118 ceph-mon[31535]: pgmap v1109: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T00:37:46.447 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:37:46 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 761.625795 secs 2024-03-24T00:37:46.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:37:46 smithi189 ceph-mon[36187]: pgmap v1109: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T00:37:46.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:37:46 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 761.625795 secs 2024-03-24T00:37:46.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:37:46 smithi193 ceph-mon[36878]: pgmap v1109: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T00:37:46.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:37:46 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 761.625795 secs 2024-03-24T00:37:48.352 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:37:48 smithi118 ceph-mon[31535]: pgmap v1110: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:37:48.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:37:48 smithi189 ceph-mon[36187]: pgmap v1110: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:37:48.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:37:48 smithi193 ceph-mon[36878]: pgmap v1110: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:37:50.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:37:50 smithi118 ceph-mon[31535]: pgmap v1111: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T00:37:50.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:37:50 smithi189 ceph-mon[36187]: pgmap v1111: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T00:37:50.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:37:50 smithi193 ceph-mon[36878]: pgmap v1111: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T00:37:51.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:37:51 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 766.625883 secs 2024-03-24T00:37:51.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:37:51 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 766.625883 secs 2024-03-24T00:37:51.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:37:51 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 766.625883 secs 2024-03-24T00:37:52.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:37:52 smithi118 ceph-mon[31535]: pgmap v1112: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T00:37:52.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:37:52 smithi189 ceph-mon[36187]: pgmap v1112: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T00:37:52.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:37:52 smithi193 ceph-mon[36878]: pgmap v1112: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T00:37:54.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:37:54 smithi118 ceph-mon[31535]: pgmap v1113: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 5 op/s 2024-03-24T00:37:54.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:37:54 smithi189 ceph-mon[36187]: pgmap v1113: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 5 op/s 2024-03-24T00:37:54.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:37:54 smithi193 ceph-mon[36878]: pgmap v1113: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 5 op/s 2024-03-24T00:37:56.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:37:56 smithi118 ceph-mon[31535]: pgmap v1114: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T00:37:56.447 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:37:56 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 771.625973 secs 2024-03-24T00:37:56.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:37:56 smithi189 ceph-mon[36187]: pgmap v1114: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T00:37:56.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:37:56 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 771.625973 secs 2024-03-24T00:37:56.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:37:56 smithi193 ceph-mon[36878]: pgmap v1114: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T00:37:56.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:37:56 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 771.625973 secs 2024-03-24T00:37:58.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:37:58 smithi118 ceph-mon[31535]: pgmap v1115: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:37:58.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:37:58 smithi189 ceph-mon[36187]: pgmap v1115: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:37:58.460 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:37:58 smithi193 ceph-mon[36878]: pgmap v1115: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:38:00.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:38:00 smithi118 ceph-mon[31535]: pgmap v1116: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T00:38:00.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:38:00 smithi189 ceph-mon[36187]: pgmap v1116: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T00:38:00.460 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:38:00 smithi193 ceph-mon[36878]: pgmap v1116: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T00:38:01.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:38:01 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 776.626066 secs 2024-03-24T00:38:01.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:38:01 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 776.626066 secs 2024-03-24T00:38:01.460 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:38:01 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 776.626066 secs 2024-03-24T00:38:02.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:38:02 smithi118 ceph-mon[31535]: pgmap v1117: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T00:38:02.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:38:02 smithi189 ceph-mon[36187]: pgmap v1117: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T00:38:02.460 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:38:02 smithi193 ceph-mon[36878]: pgmap v1117: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T00:38:04.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:38:04 smithi118 ceph-mon[31535]: pgmap v1118: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 5 op/s 2024-03-24T00:38:04.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:38:04 smithi189 ceph-mon[36187]: pgmap v1118: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 5 op/s 2024-03-24T00:38:04.460 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:38:04 smithi193 ceph-mon[36878]: pgmap v1118: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 5 op/s 2024-03-24T00:38:05.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:38:05 smithi118 ceph-mon[31535]: pgmap v1119: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T00:38:05.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:38:05 smithi189 ceph-mon[36187]: pgmap v1119: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T00:38:05.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:38:05 smithi193 ceph-mon[36878]: pgmap v1119: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T00:38:06.361 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:38:06 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 781.626157 secs 2024-03-24T00:38:06.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:38:06 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 781.626157 secs 2024-03-24T00:38:06.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:38:06 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 781.626157 secs 2024-03-24T00:38:07.375 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:38:07 smithi193 ceph-mon[36878]: pgmap v1120: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:38:07.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:38:07 smithi118 ceph-mon[31535]: pgmap v1120: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:38:07.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:38:07 smithi189 ceph-mon[36187]: pgmap v1120: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:38:10.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:38:09 smithi118 ceph-mon[31535]: pgmap v1121: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T00:38:10.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:38:09 smithi189 ceph-mon[36187]: pgmap v1121: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T00:38:10.460 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:38:09 smithi193 ceph-mon[36878]: pgmap v1121: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T00:38:11.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:38:10 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 786.626219 secs 2024-03-24T00:38:11.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:38:10 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 786.626219 secs 2024-03-24T00:38:11.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:38:10 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 786.626219 secs 2024-03-24T00:38:12.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:38:11 smithi118 ceph-mon[31535]: pgmap v1122: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T00:38:12.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:38:11 smithi189 ceph-mon[36187]: pgmap v1122: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T00:38:12.460 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:38:11 smithi193 ceph-mon[36878]: pgmap v1122: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T00:38:14.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:38:13 smithi118 ceph-mon[31535]: pgmap v1123: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 5 op/s 2024-03-24T00:38:14.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:38:13 smithi189 ceph-mon[36187]: pgmap v1123: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 5 op/s 2024-03-24T00:38:14.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:38:13 smithi193 ceph-mon[36878]: pgmap v1123: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 5 op/s 2024-03-24T00:38:16.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:38:15 smithi118 ceph-mon[31535]: pgmap v1124: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T00:38:16.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:38:15 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 791.626281 secs 2024-03-24T00:38:16.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:38:15 smithi189 ceph-mon[36187]: pgmap v1124: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T00:38:16.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:38:15 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 791.626281 secs 2024-03-24T00:38:16.460 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:38:15 smithi193 ceph-mon[36878]: pgmap v1124: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T00:38:16.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:38:15 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 791.626281 secs 2024-03-24T00:38:18.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:38:17 smithi118 ceph-mon[31535]: pgmap v1125: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:38:18.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:38:17 smithi189 ceph-mon[36187]: pgmap v1125: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:38:18.460 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:38:17 smithi193 ceph-mon[36878]: pgmap v1125: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:38:20.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:38:19 smithi118 ceph-mon[31535]: pgmap v1126: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-03-24T00:38:20.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:38:19 smithi189 ceph-mon[36187]: pgmap v1126: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-03-24T00:38:20.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:38:19 smithi193 ceph-mon[36878]: pgmap v1126: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-03-24T00:38:21.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:38:21 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 796.626343 secs 2024-03-24T00:38:21.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:38:21 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 796.626343 secs 2024-03-24T00:38:21.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:38:21 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 796.626343 secs 2024-03-24T00:38:22.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:38:22 smithi118 ceph-mon[31535]: pgmap v1127: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 6 op/s 2024-03-24T00:38:22.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:38:22 smithi189 ceph-mon[36187]: pgmap v1127: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 6 op/s 2024-03-24T00:38:22.460 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:38:22 smithi193 ceph-mon[36878]: pgmap v1127: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 6 op/s 2024-03-24T00:38:24.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:38:24 smithi118 ceph-mon[31535]: pgmap v1128: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 5 op/s 2024-03-24T00:38:24.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:38:24 smithi189 ceph-mon[36187]: pgmap v1128: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 5 op/s 2024-03-24T00:38:24.460 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:38:24 smithi193 ceph-mon[36878]: pgmap v1128: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 5 op/s 2024-03-24T00:38:26.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:38:26 smithi118 ceph-mon[31535]: pgmap v1129: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 6 op/s 2024-03-24T00:38:26.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:38:26 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 801.626403 secs 2024-03-24T00:38:26.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:38:26 smithi189 ceph-mon[36187]: pgmap v1129: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 6 op/s 2024-03-24T00:38:26.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:38:26 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 801.626403 secs 2024-03-24T00:38:26.460 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:38:26 smithi193 ceph-mon[36878]: pgmap v1129: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 6 op/s 2024-03-24T00:38:26.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:38:26 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 801.626403 secs 2024-03-24T00:38:28.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:38:28 smithi118 ceph-mon[31535]: pgmap v1130: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-03-24T00:38:28.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:38:28 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T00:38:28.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:38:28 smithi189 ceph-mon[36187]: pgmap v1130: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-03-24T00:38:28.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:38:28 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T00:38:28.460 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:38:28 smithi193 ceph-mon[36878]: pgmap v1130: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-03-24T00:38:28.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:38:28 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T00:38:29.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:38:29 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T00:38:29.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:38:29 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T00:38:29.447 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:38:29 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T00:38:29.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:38:29 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T00:38:29.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:38:29 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T00:38:29.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:38:29 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T00:38:29.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:38:29 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T00:38:29.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:38:29 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T00:38:29.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:38:29 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T00:38:30.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:38:30 smithi118 ceph-mon[31535]: pgmap v1131: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 6 op/s 2024-03-24T00:38:30.447 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:38:30 smithi118 ceph-mon[31535]: pgmap v1132: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-03-24T00:38:30.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:38:30 smithi189 ceph-mon[36187]: pgmap v1131: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 6 op/s 2024-03-24T00:38:30.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:38:30 smithi189 ceph-mon[36187]: pgmap v1132: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-03-24T00:38:30.460 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:38:30 smithi193 ceph-mon[36878]: pgmap v1131: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 6 op/s 2024-03-24T00:38:30.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:38:30 smithi193 ceph-mon[36878]: pgmap v1132: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-03-24T00:38:31.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:38:31 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 806.626464 secs 2024-03-24T00:38:31.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:38:31 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 806.626464 secs 2024-03-24T00:38:31.460 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:38:31 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 806.626464 secs 2024-03-24T00:38:32.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:38:32 smithi118 ceph-mon[31535]: pgmap v1133: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-03-24T00:38:32.455 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:38:32 smithi189 ceph-mon[36187]: pgmap v1133: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-03-24T00:38:32.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:38:32 smithi193 ceph-mon[36878]: pgmap v1133: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-03-24T00:38:34.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:38:34 smithi118 ceph-mon[31535]: pgmap v1134: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 5 op/s 2024-03-24T00:38:34.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:38:34 smithi189 ceph-mon[36187]: pgmap v1134: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 5 op/s 2024-03-24T00:38:34.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:38:34 smithi193 ceph-mon[36878]: pgmap v1134: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 5 op/s 2024-03-24T00:38:36.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:38:36 smithi118 ceph-mon[31535]: pgmap v1135: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-03-24T00:38:36.447 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:38:36 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 811.626554 secs 2024-03-24T00:38:36.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:38:36 smithi189 ceph-mon[36187]: pgmap v1135: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-03-24T00:38:36.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:38:36 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 811.626554 secs 2024-03-24T00:38:36.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:38:36 smithi193 ceph-mon[36878]: pgmap v1135: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-03-24T00:38:36.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:38:36 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 811.626554 secs 2024-03-24T00:38:38.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:38:38 smithi118 ceph-mon[31535]: pgmap v1136: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-03-24T00:38:38.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:38:38 smithi189 ceph-mon[36187]: pgmap v1136: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-03-24T00:38:38.460 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:38:38 smithi193 ceph-mon[36878]: pgmap v1136: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-03-24T00:38:39.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:38:39 smithi118 ceph-mon[31535]: pgmap v1137: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 5 op/s 2024-03-24T00:38:39.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:38:39 smithi189 ceph-mon[36187]: pgmap v1137: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 5 op/s 2024-03-24T00:38:39.460 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:38:39 smithi193 ceph-mon[36878]: pgmap v1137: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 5 op/s 2024-03-24T00:38:40.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:38:40 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 816.626614 secs 2024-03-24T00:38:40.704 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:38:40 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 816.626614 secs 2024-03-24T00:38:40.710 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:38:40 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 816.626614 secs 2024-03-24T00:38:41.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:38:41 smithi118 ceph-mon[31535]: pgmap v1138: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T00:38:41.704 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:38:41 smithi189 ceph-mon[36187]: pgmap v1138: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T00:38:41.711 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:38:41 smithi193 ceph-mon[36878]: pgmap v1138: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T00:38:43.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:38:43 smithi118 ceph-mon[31535]: pgmap v1139: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:38:43.704 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:38:43 smithi189 ceph-mon[36187]: pgmap v1139: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:38:43.710 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:38:43 smithi193 ceph-mon[36878]: pgmap v1139: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:38:45.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:38:45 smithi118 ceph-mon[31535]: pgmap v1140: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-24T00:38:45.697 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:38:45 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 821.626700 secs 2024-03-24T00:38:45.704 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:38:45 smithi189 ceph-mon[36187]: pgmap v1140: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-24T00:38:45.704 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:38:45 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 821.626700 secs 2024-03-24T00:38:45.710 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:38:45 smithi193 ceph-mon[36878]: pgmap v1140: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-24T00:38:45.711 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:38:45 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 821.626700 secs 2024-03-24T00:38:47.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:38:47 smithi118 ceph-mon[31535]: pgmap v1141: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:38:47.704 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:38:47 smithi189 ceph-mon[36187]: pgmap v1141: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:38:47.710 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:38:47 smithi193 ceph-mon[36878]: pgmap v1141: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:38:49.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:38:49 smithi118 ceph-mon[31535]: pgmap v1142: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T00:38:49.703 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:38:49 smithi189 ceph-mon[36187]: pgmap v1142: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T00:38:49.710 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:38:49 smithi193 ceph-mon[36878]: pgmap v1142: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T00:38:50.704 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:38:50 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 826.626789 secs 2024-03-24T00:38:50.710 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:38:50 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 826.626789 secs 2024-03-24T00:38:50.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:38:50 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 826.626789 secs 2024-03-24T00:38:51.711 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:38:51 smithi193 ceph-mon[36878]: pgmap v1143: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T00:38:51.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:38:51 smithi118 ceph-mon[31535]: pgmap v1143: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T00:38:51.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:38:51 smithi189 ceph-mon[36187]: pgmap v1143: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T00:38:53.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:38:53 smithi118 ceph-mon[31535]: pgmap v1144: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:38:53.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:38:53 smithi189 ceph-mon[36187]: pgmap v1144: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:38:53.460 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:38:53 smithi193 ceph-mon[36878]: pgmap v1144: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:38:55.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:38:55 smithi118 ceph-mon[31535]: pgmap v1145: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-24T00:38:55.697 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:38:55 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 831.626876 secs 2024-03-24T00:38:55.704 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:38:55 smithi189 ceph-mon[36187]: pgmap v1145: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-24T00:38:55.704 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:38:55 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 831.626876 secs 2024-03-24T00:38:55.711 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:38:55 smithi193 ceph-mon[36878]: pgmap v1145: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-24T00:38:55.711 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:38:55 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 831.626876 secs 2024-03-24T00:38:57.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:38:57 smithi118 ceph-mon[31535]: pgmap v1146: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:38:57.704 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:38:57 smithi189 ceph-mon[36187]: pgmap v1146: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:38:57.710 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:38:57 smithi193 ceph-mon[36878]: pgmap v1146: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:38:59.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:38:59 smithi118 ceph-mon[31535]: pgmap v1147: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T00:38:59.704 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:38:59 smithi189 ceph-mon[36187]: pgmap v1147: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T00:38:59.710 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:38:59 smithi193 ceph-mon[36878]: pgmap v1147: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T00:39:00.711 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:39:00 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 836.626965 secs 2024-03-24T00:39:00.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:39:00 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 836.626965 secs 2024-03-24T00:39:00.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:39:00 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 836.626965 secs 2024-03-24T00:39:01.711 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:39:01 smithi193 ceph-mon[36878]: pgmap v1148: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T00:39:01.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:39:01 smithi118 ceph-mon[31535]: pgmap v1148: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T00:39:01.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:39:01 smithi189 ceph-mon[36187]: pgmap v1148: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T00:39:03.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:39:03 smithi118 ceph-mon[31535]: pgmap v1149: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:39:03.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:39:03 smithi189 ceph-mon[36187]: pgmap v1149: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:39:03.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:39:03 smithi193 ceph-mon[36878]: pgmap v1149: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:39:05.947 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:39:05 smithi118 ceph-mon[31535]: pgmap v1150: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-24T00:39:05.947 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:39:05 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 841.627041 secs 2024-03-24T00:39:05.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:39:05 smithi189 ceph-mon[36187]: pgmap v1150: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-24T00:39:05.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:39:05 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 841.627041 secs 2024-03-24T00:39:05.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:39:05 smithi193 ceph-mon[36878]: pgmap v1150: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-24T00:39:05.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:39:05 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 841.627041 secs 2024-03-24T00:39:07.763 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:39:07 smithi193 ceph-mon[36878]: pgmap v1151: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:39:07.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:39:07 smithi118 ceph-mon[31535]: pgmap v1151: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:39:07.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:39:07 smithi189 ceph-mon[36187]: pgmap v1151: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:39:09.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:39:09 smithi118 ceph-mon[31535]: pgmap v1152: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T00:39:09.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:39:09 smithi189 ceph-mon[36187]: pgmap v1152: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T00:39:09.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:39:09 smithi193 ceph-mon[36878]: pgmap v1152: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T00:39:10.855 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:39:10 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 846.627082 secs 2024-03-24T00:39:10.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:39:10 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 846.627082 secs 2024-03-24T00:39:10.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:39:10 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 846.627082 secs 2024-03-24T00:39:11.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:39:11 smithi118 ceph-mon[31535]: pgmap v1153: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T00:39:11.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:39:11 smithi189 ceph-mon[36187]: pgmap v1153: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T00:39:11.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:39:11 smithi193 ceph-mon[36878]: pgmap v1153: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T00:39:13.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:39:13 smithi118 ceph-mon[31535]: pgmap v1154: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:39:13.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:39:13 smithi189 ceph-mon[36187]: pgmap v1154: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:39:13.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:39:13 smithi193 ceph-mon[36878]: pgmap v1154: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:39:15.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:39:15 smithi118 ceph-mon[31535]: pgmap v1155: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-24T00:39:15.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:39:15 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 851.627143 secs 2024-03-24T00:39:15.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:39:15 smithi189 ceph-mon[36187]: pgmap v1155: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-24T00:39:15.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:39:15 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 851.627143 secs 2024-03-24T00:39:15.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:39:15 smithi193 ceph-mon[36878]: pgmap v1155: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-24T00:39:15.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:39:15 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 851.627143 secs 2024-03-24T00:39:17.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:39:17 smithi118 ceph-mon[31535]: pgmap v1156: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:39:17.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:39:17 smithi189 ceph-mon[36187]: pgmap v1156: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:39:17.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:39:17 smithi193 ceph-mon[36878]: pgmap v1156: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:39:19.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:39:19 smithi118 ceph-mon[31535]: pgmap v1157: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T00:39:19.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:39:19 smithi189 ceph-mon[36187]: pgmap v1157: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T00:39:19.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:39:19 smithi193 ceph-mon[36878]: pgmap v1157: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T00:39:20.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:39:20 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 856.627180 secs 2024-03-24T00:39:20.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:39:20 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 856.627180 secs 2024-03-24T00:39:20.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:39:20 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 856.627180 secs 2024-03-24T00:39:21.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:39:21 smithi118 ceph-mon[31535]: pgmap v1158: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T00:39:21.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:39:21 smithi189 ceph-mon[36187]: pgmap v1158: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T00:39:21.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:39:21 smithi193 ceph-mon[36878]: pgmap v1158: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T00:39:23.928 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:39:23 smithi118 ceph-mon[31535]: pgmap v1159: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:39:23.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:39:23 smithi189 ceph-mon[36187]: pgmap v1159: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:39:23.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:39:23 smithi193 ceph-mon[36878]: pgmap v1159: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:39:25.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:39:25 smithi118 ceph-mon[31535]: pgmap v1160: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-24T00:39:25.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:39:25 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 861.627242 secs 2024-03-24T00:39:25.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:39:25 smithi189 ceph-mon[36187]: pgmap v1160: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-24T00:39:25.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:39:25 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 861.627242 secs 2024-03-24T00:39:25.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:39:25 smithi193 ceph-mon[36878]: pgmap v1160: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-24T00:39:25.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:39:25 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 861.627242 secs 2024-03-24T00:39:27.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:39:27 smithi118 ceph-mon[31535]: pgmap v1161: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:39:27.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:39:27 smithi189 ceph-mon[36187]: pgmap v1161: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:39:27.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:39:27 smithi193 ceph-mon[36878]: pgmap v1161: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:39:29.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:39:29 smithi118 ceph-mon[31535]: pgmap v1162: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T00:39:29.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:39:29 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T00:39:29.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:39:29 smithi189 ceph-mon[36187]: pgmap v1162: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T00:39:29.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:39:29 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T00:39:29.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:39:29 smithi193 ceph-mon[36878]: pgmap v1162: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T00:39:29.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:39:29 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T00:39:30.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:39:30 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T00:39:30.947 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:39:30 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T00:39:30.947 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:39:30 smithi118 ceph-mon[31535]: pgmap v1163: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 6 op/s 2024-03-24T00:39:30.947 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:39:30 smithi118 ceph-mon[31535]: pgmap v1164: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 6 op/s 2024-03-24T00:39:30.947 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:39:30 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T00:39:30.947 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:39:30 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 866.627278 secs 2024-03-24T00:39:30.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:39:30 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T00:39:30.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:39:30 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T00:39:30.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:39:30 smithi189 ceph-mon[36187]: pgmap v1163: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 6 op/s 2024-03-24T00:39:30.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:39:30 smithi189 ceph-mon[36187]: pgmap v1164: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 6 op/s 2024-03-24T00:39:30.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:39:30 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T00:39:30.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:39:30 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 866.627278 secs 2024-03-24T00:39:30.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:39:30 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T00:39:30.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:39:30 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T00:39:30.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:39:30 smithi193 ceph-mon[36878]: pgmap v1163: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 6 op/s 2024-03-24T00:39:30.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:39:30 smithi193 ceph-mon[36878]: pgmap v1164: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 6 op/s 2024-03-24T00:39:30.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:39:30 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T00:39:30.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:39:30 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 866.627278 secs 2024-03-24T00:39:33.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:39:32 smithi118 ceph-mon[31535]: pgmap v1165: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 6 op/s 2024-03-24T00:39:33.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:39:32 smithi189 ceph-mon[36187]: pgmap v1165: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 6 op/s 2024-03-24T00:39:33.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:39:32 smithi193 ceph-mon[36878]: pgmap v1165: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 6 op/s 2024-03-24T00:39:35.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:39:34 smithi118 ceph-mon[31535]: pgmap v1166: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 5 op/s 2024-03-24T00:39:35.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:39:34 smithi189 ceph-mon[36187]: pgmap v1166: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 5 op/s 2024-03-24T00:39:35.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:39:34 smithi193 ceph-mon[36878]: pgmap v1166: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 5 op/s 2024-03-24T00:39:36.447 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:39:36 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 871.627371 secs 2024-03-24T00:39:36.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:39:36 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 871.627371 secs 2024-03-24T00:39:36.460 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:39:36 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 871.627371 secs 2024-03-24T00:39:37.361 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:39:37 smithi193 ceph-mon[36878]: pgmap v1167: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 6 op/s 2024-03-24T00:39:37.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:39:37 smithi118 ceph-mon[31535]: pgmap v1167: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 6 op/s 2024-03-24T00:39:37.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:39:37 smithi189 ceph-mon[36187]: pgmap v1167: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 6 op/s 2024-03-24T00:39:39.288 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:39:39 smithi118 ceph-mon[31535]: pgmap v1168: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 5 op/s 2024-03-24T00:39:39.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:39:39 smithi189 ceph-mon[36187]: pgmap v1168: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 5 op/s 2024-03-24T00:39:39.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:39:39 smithi193 ceph-mon[36878]: pgmap v1168: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 5 op/s 2024-03-24T00:39:41.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:39:41 smithi118 ceph-mon[31535]: pgmap v1169: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T00:39:41.447 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:39:41 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 876.627409 secs 2024-03-24T00:39:41.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:39:41 smithi189 ceph-mon[36187]: pgmap v1169: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T00:39:41.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:39:41 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 876.627409 secs 2024-03-24T00:39:41.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:39:41 smithi193 ceph-mon[36878]: pgmap v1169: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T00:39:41.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:39:41 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 876.627409 secs 2024-03-24T00:39:43.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:39:43 smithi118 ceph-mon[31535]: pgmap v1170: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-03-24T00:39:43.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:39:43 smithi189 ceph-mon[36187]: pgmap v1170: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-03-24T00:39:43.460 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:39:43 smithi193 ceph-mon[36878]: pgmap v1170: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-03-24T00:39:45.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:39:45 smithi118 ceph-mon[31535]: pgmap v1171: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-03-24T00:39:45.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:39:45 smithi189 ceph-mon[36187]: pgmap v1171: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-03-24T00:39:45.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:39:45 smithi193 ceph-mon[36878]: pgmap v1171: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-03-24T00:39:46.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:39:46 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 881.627496 secs 2024-03-24T00:39:46.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:39:46 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 881.627496 secs 2024-03-24T00:39:46.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:39:46 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 881.627496 secs 2024-03-24T00:39:47.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:39:47 smithi118 ceph-mon[31535]: pgmap v1172: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 5 op/s 2024-03-24T00:39:47.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:39:47 smithi189 ceph-mon[36187]: pgmap v1172: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 5 op/s 2024-03-24T00:39:47.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:39:47 smithi193 ceph-mon[36878]: pgmap v1172: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 5 op/s 2024-03-24T00:39:49.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:39:49 smithi118 ceph-mon[31535]: pgmap v1173: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 5 op/s 2024-03-24T00:39:49.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:39:49 smithi189 ceph-mon[36187]: pgmap v1173: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 5 op/s 2024-03-24T00:39:49.460 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:39:49 smithi193 ceph-mon[36878]: pgmap v1173: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 5 op/s 2024-03-24T00:39:51.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:39:51 smithi118 ceph-mon[31535]: pgmap v1174: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T00:39:51.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:39:51 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 886.627585 secs 2024-03-24T00:39:51.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:39:51 smithi189 ceph-mon[36187]: pgmap v1174: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T00:39:51.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:39:51 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 886.627585 secs 2024-03-24T00:39:51.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:39:51 smithi193 ceph-mon[36878]: pgmap v1174: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T00:39:51.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:39:51 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 886.627585 secs 2024-03-24T00:39:53.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:39:53 smithi118 ceph-mon[31535]: pgmap v1175: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:39:53.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:39:53 smithi189 ceph-mon[36187]: pgmap v1175: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:39:53.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:39:53 smithi193 ceph-mon[36878]: pgmap v1175: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:39:55.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:39:55 smithi118 ceph-mon[31535]: pgmap v1176: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T00:39:55.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:39:55 smithi189 ceph-mon[36187]: pgmap v1176: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T00:39:55.460 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:39:55 smithi193 ceph-mon[36878]: pgmap v1176: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T00:39:56.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:39:56 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 891.627645 secs 2024-03-24T00:39:56.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:39:56 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 891.627645 secs 2024-03-24T00:39:56.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:39:56 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 891.627645 secs 2024-03-24T00:39:57.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:39:57 smithi118 ceph-mon[31535]: pgmap v1177: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T00:39:57.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:39:57 smithi189 ceph-mon[36187]: pgmap v1177: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T00:39:57.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:39:57 smithi193 ceph-mon[36878]: pgmap v1177: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T00:39:59.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:39:59 smithi118 ceph-mon[31535]: pgmap v1178: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 5 op/s 2024-03-24T00:39:59.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:39:59 smithi189 ceph-mon[36187]: pgmap v1178: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 5 op/s 2024-03-24T00:39:59.460 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:39:59 smithi193 ceph-mon[36878]: pgmap v1178: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 5 op/s 2024-03-24T00:40:00.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:40:00 smithi118 ceph-mon[31535]: overall HEALTH_WARN 6 failed cephadm daemon(s); 1 MDSs report slow requests 2024-03-24T00:40:00.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:40:00 smithi189 ceph-mon[36187]: overall HEALTH_WARN 6 failed cephadm daemon(s); 1 MDSs report slow requests 2024-03-24T00:40:00.460 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:40:00 smithi193 ceph-mon[36878]: overall HEALTH_WARN 6 failed cephadm daemon(s); 1 MDSs report slow requests 2024-03-24T00:40:01.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:40:01 smithi118 ceph-mon[31535]: pgmap v1179: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T00:40:01.447 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:40:01 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 896.627660 secs 2024-03-24T00:40:01.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:40:01 smithi189 ceph-mon[36187]: pgmap v1179: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T00:40:01.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:40:01 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 896.627660 secs 2024-03-24T00:40:01.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:40:01 smithi193 ceph-mon[36878]: pgmap v1179: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T00:40:01.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:40:01 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 896.627660 secs 2024-03-24T00:40:03.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:40:03 smithi118 ceph-mon[31535]: pgmap v1180: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:40:03.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:40:03 smithi189 ceph-mon[36187]: pgmap v1180: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:40:03.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:40:03 smithi193 ceph-mon[36878]: pgmap v1180: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:40:04.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:40:04 smithi118 ceph-mon[31535]: pgmap v1181: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T00:40:04.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:40:04 smithi189 ceph-mon[36187]: pgmap v1181: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T00:40:04.460 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:40:04 smithi193 ceph-mon[36878]: pgmap v1181: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T00:40:05.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:40:05 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 901.627716 secs 2024-03-24T00:40:05.704 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:40:05 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 901.627716 secs 2024-03-24T00:40:05.710 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:40:05 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 901.627716 secs 2024-03-24T00:40:06.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:40:06 smithi118 ceph-mon[31535]: pgmap v1182: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T00:40:06.704 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:40:06 smithi189 ceph-mon[36187]: pgmap v1182: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T00:40:06.711 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:40:06 smithi193 ceph-mon[36878]: pgmap v1182: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T00:40:09.413 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:40:09 smithi193 ceph-mon[36878]: pgmap v1183: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 5 op/s 2024-03-24T00:40:09.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:40:09 smithi118 ceph-mon[31535]: pgmap v1183: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 5 op/s 2024-03-24T00:40:09.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:40:09 smithi189 ceph-mon[36187]: pgmap v1183: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 5 op/s 2024-03-24T00:40:11.378 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:40:11 smithi189 ceph-mon[36187]: pgmap v1184: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T00:40:11.378 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:40:11 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 906.627805 secs 2024-03-24T00:40:11.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:40:11 smithi118 ceph-mon[31535]: pgmap v1184: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T00:40:11.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:40:11 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 906.627805 secs 2024-03-24T00:40:11.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:40:11 smithi193 ceph-mon[36878]: pgmap v1184: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T00:40:11.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:40:11 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 906.627805 secs 2024-03-24T00:40:13.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:40:13 smithi118 ceph-mon[31535]: pgmap v1185: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:40:13.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:40:13 smithi189 ceph-mon[36187]: pgmap v1185: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:40:13.460 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:40:13 smithi193 ceph-mon[36878]: pgmap v1185: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:40:15.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:40:15 smithi118 ceph-mon[31535]: pgmap v1186: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T00:40:15.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:40:15 smithi189 ceph-mon[36187]: pgmap v1186: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T00:40:15.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:40:15 smithi193 ceph-mon[36878]: pgmap v1186: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T00:40:15.730 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:40:15.730+0000 7f81f6000640 1 -- 172.21.15.118:0/1693767697 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7f81dc0a27f0 msgr2=0x7f81dc0a4cb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 12 2024-03-24T00:40:15.730 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:40:15.730+0000 7f81f6000640 1 -- 172.21.15.118:0/1693767697 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7f81dc0a27f0 msgr2=0x7f81dc0a4cb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-03-24T00:40:15.731 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:40:15.730+0000 7f81f6000640 1 --2- 172.21.15.118:0/1693767697 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7f81dc0a27f0 0x7f81dc0a4cb0 secure :-1 s=READY pgs=401 cs=0 l=1 rev1=1 crypto rx=0x5622651a2640 tx=0x7f81e0015040 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-03-24T00:40:15.731 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:40:15.730+0000 7f81f6000640 1 --2- 172.21.15.118:0/1693767697 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7f81dc0a27f0 0x7f81dc0a4cb0 secure :-1 s=READY pgs=401 cs=0 l=1 rev1=1 crypto rx=0x5622651a2640 tx=0x7f81e0015040 comp rx=0 tx=0).stop 2024-03-24T00:40:15.731 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:40:15.730+0000 7f81f6000640 1 -- 172.21.15.118:0/1693767697 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7f81dc0a27f0 msgr2=0x7f81dc0a4cb0 unknown :-1 s=STATE_CLOSED l=1).tick see no progress in more than 10000000 us during connecting to v2:172.21.15.118:6800/2112912943, fault. 2024-03-24T00:40:15.731 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:40:15.730+0000 7f81ef400640 1 -- 172.21.15.118:0/1693767697 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7f81dc0a27f0 msgr2=0x7f81dc0a4cb0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-03-24T00:40:15.731 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:40:15.730+0000 7f81ef400640 1 --2- 172.21.15.118:0/1693767697 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7f81dc0a27f0 0x7f81dc0a4cb0 unknown :-1 s=CLOSED pgs=401 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:40:15.731 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:40:15.730+0000 7f81ef400640 1 --2- 172.21.15.118:0/1693767697 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7f81dc0ae1d0 0x7f81dc0b05c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:40:15.731 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:40:15.731+0000 7f81f6000640 1 --2- 172.21.15.118:0/1693767697 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7f81dc0ae1d0 0x7f81dc0b05c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:40:15.732 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:40:15.732+0000 7f81f6000640 1 --2- 172.21.15.118:0/1693767697 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7f81dc0ae1d0 0x7f81dc0b05c0 secure :-1 s=READY pgs=421 cs=0 l=1 rev1=1 crypto rx=0x7f81dc0a2bd0 tx=0x7f81e0024040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:40:15.757 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:40:15.757+0000 7f81f6000640 1 -- 172.21.15.118:0/659686424 >> [v2:172.21.15.189:6838/4074484168,v1:172.21.15.189:6839/4074484168] conn(0x5622651b71e0 msgr2=0x562265215110 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-03-24T00:40:15.757 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:40:15.757+0000 7f81f6000640 1 -- 172.21.15.118:0/659686424 >> [v2:172.21.15.189:6838/4074484168,v1:172.21.15.189:6839/4074484168] conn(0x5622651b71e0 msgr2=0x562265215110 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-03-24T00:40:15.757 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:40:15.757+0000 7f81f6000640 1 --2- 172.21.15.118:0/659686424 >> [v2:172.21.15.189:6838/4074484168,v1:172.21.15.189:6839/4074484168] conn(0x5622651b71e0 0x562265215110 crc :-1 s=READY pgs=439 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-03-24T00:40:15.757 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:40:15.757+0000 7f81f6000640 1 --2- 172.21.15.118:0/659686424 >> [v2:172.21.15.189:6838/4074484168,v1:172.21.15.189:6839/4074484168] conn(0x5622651b71e0 0x562265215110 unknown :-1 s=READY pgs=439 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault initiating reconnect 2024-03-24T00:40:15.757 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:40:15.757+0000 7f81f6000640 1 --2- 172.21.15.118:0/659686424 >> [v2:172.21.15.189:6838/4074484168,v1:172.21.15.189:6839/4074484168] conn(0x5622651b71e0 0x562265215110 unknown :-1 s=BANNER_CONNECTING pgs=439 cs=1 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:40:15.758 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:40:15.758+0000 7f81f6000640 1 --2- 172.21.15.118:0/659686424 >> [v2:172.21.15.189:6838/4074484168,v1:172.21.15.189:6839/4074484168] conn(0x5622651b71e0 0x562265215110 crc :-1 s=SESSION_RECONNECTING pgs=439 cs=1 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_session_reset received session reset full=1 2024-03-24T00:40:15.758 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:40:15.758+0000 7f81f6000640 1 --2- 172.21.15.118:0/659686424 >> [v2:172.21.15.189:6838/4074484168,v1:172.21.15.189:6839/4074484168] conn(0x5622651b71e0 0x562265215110 crc :-1 s=SESSION_RECONNECTING pgs=439 cs=1 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).reset_session 2024-03-24T00:40:15.758 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:40:15.758+0000 7f81e4c00640 0 client.27094 ms_handle_remote_reset on v2:172.21.15.189:6838/4074484168 2024-03-24T00:40:15.758 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:40:15.758+0000 7f81f6000640 1 --2- 172.21.15.118:0/659686424 >> [v2:172.21.15.189:6838/4074484168,v1:172.21.15.189:6839/4074484168] conn(0x5622651b71e0 0x562265215110 crc :-1 s=READY pgs=561 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=76f3d534289740f0 server_cookie=b31a2335a59552de in_seq=0 out_seq=0 2024-03-24T00:40:15.758 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:40:15.758+0000 7f81e4c00640 10 client.27094 ms_handle_connect on v2:172.21.15.189:6838/4074484168 2024-03-24T00:40:15.868 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:40:15.868+0000 7f81ee000640 1 -- 172.21.15.118:0/1693767697 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f81c80063c0 con 0x7f81f81052d0 2024-03-24T00:40:15.868 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:40:15.868+0000 7f81ef400640 1 -- 172.21.15.118:0/1693767697 <== mon.2 v2:172.21.15.189:3300/0 11 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (secure 0 0 0) 0x7f81c80063c0 con 0x7f81f81052d0 2024-03-24T00:40:15.890 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:40:15.890+0000 7f81db400640 1 -- 172.21.15.118:0/659686424 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f81c4006040 con 0x5622651a13f0 2024-03-24T00:40:15.890 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:40:15.890+0000 7f81e4c00640 1 -- 172.21.15.118:0/659686424 <== mon.1 v2:172.21.15.193:3300/0 11 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 409+0+0 (secure 0 0 0) 0x7f81c4006040 con 0x5622651a13f0 2024-03-24T00:40:16.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:40:16 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 911.627889 secs 2024-03-24T00:40:16.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:40:16 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 911.627889 secs 2024-03-24T00:40:16.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:40:16 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 911.627889 secs 2024-03-24T00:40:17.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:40:17 smithi118 ceph-mon[31535]: pgmap v1187: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T00:40:17.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:40:17 smithi189 ceph-mon[36187]: pgmap v1187: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T00:40:17.460 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:40:17 smithi193 ceph-mon[36878]: pgmap v1187: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T00:40:19.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:40:19 smithi118 ceph-mon[31535]: pgmap v1188: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 5 op/s 2024-03-24T00:40:19.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:40:19 smithi189 ceph-mon[36187]: pgmap v1188: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 5 op/s 2024-03-24T00:40:19.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:40:19 smithi193 ceph-mon[36878]: pgmap v1188: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 5 op/s 2024-03-24T00:40:21.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:40:21 smithi118 ceph-mon[31535]: pgmap v1189: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T00:40:21.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:40:21 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 916.627981 secs 2024-03-24T00:40:21.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:40:21 smithi189 ceph-mon[36187]: pgmap v1189: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T00:40:21.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:40:21 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 916.627981 secs 2024-03-24T00:40:21.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:40:21 smithi193 ceph-mon[36878]: pgmap v1189: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T00:40:21.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:40:21 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 916.627981 secs 2024-03-24T00:40:23.321 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:40:23 smithi118 ceph-mon[31535]: pgmap v1190: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:40:23.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:40:23 smithi189 ceph-mon[36187]: pgmap v1190: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:40:23.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:40:23 smithi193 ceph-mon[36878]: pgmap v1190: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:40:25.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:40:25 smithi118 ceph-mon[31535]: pgmap v1191: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T00:40:25.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:40:25 smithi189 ceph-mon[36187]: pgmap v1191: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T00:40:25.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:40:25 smithi193 ceph-mon[36878]: pgmap v1191: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T00:40:26.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:40:26 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 921.628072 secs 2024-03-24T00:40:26.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:40:26 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 921.628072 secs 2024-03-24T00:40:26.460 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:40:26 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 921.628072 secs 2024-03-24T00:40:27.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:40:27 smithi118 ceph-mon[31535]: pgmap v1192: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T00:40:27.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:40:27 smithi189 ceph-mon[36187]: pgmap v1192: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T00:40:27.460 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:40:27 smithi193 ceph-mon[36878]: pgmap v1192: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T00:40:29.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:40:29 smithi118 ceph-mon[31535]: pgmap v1193: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 5 op/s 2024-03-24T00:40:29.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:40:29 smithi189 ceph-mon[36187]: pgmap v1193: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 5 op/s 2024-03-24T00:40:29.460 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:40:29 smithi193 ceph-mon[36878]: pgmap v1193: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 5 op/s 2024-03-24T00:40:31.447 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:40:31 smithi118 ceph-mon[31535]: pgmap v1194: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T00:40:31.447 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:40:31 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T00:40:31.447 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:40:31 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 926.628154 secs 2024-03-24T00:40:31.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:40:31 smithi189 ceph-mon[36187]: pgmap v1194: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T00:40:31.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:40:31 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T00:40:31.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:40:31 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 926.628154 secs 2024-03-24T00:40:31.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:40:31 smithi193 ceph-mon[36878]: pgmap v1194: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T00:40:31.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:40:31 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T00:40:31.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:40:31 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 926.628154 secs 2024-03-24T00:40:32.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:40:32 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T00:40:32.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:40:32 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T00:40:32.447 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:40:32 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T00:40:32.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:40:32 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T00:40:32.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:40:32 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T00:40:32.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:40:32 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T00:40:32.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:40:32 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T00:40:32.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:40:32 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T00:40:32.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:40:32 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T00:40:33.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:40:33 smithi118 ceph-mon[31535]: pgmap v1195: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 5 op/s 2024-03-24T00:40:33.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:40:33 smithi118 ceph-mon[31535]: pgmap v1196: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-03-24T00:40:33.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:40:33 smithi189 ceph-mon[36187]: pgmap v1195: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 5 op/s 2024-03-24T00:40:33.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:40:33 smithi189 ceph-mon[36187]: pgmap v1196: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-03-24T00:40:33.460 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:40:33 smithi193 ceph-mon[36878]: pgmap v1195: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 5 op/s 2024-03-24T00:40:33.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:40:33 smithi193 ceph-mon[36878]: pgmap v1196: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-03-24T00:40:34.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:40:34 smithi118 ceph-mon[31535]: pgmap v1197: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 5 op/s 2024-03-24T00:40:34.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:40:34 smithi189 ceph-mon[36187]: pgmap v1197: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 5 op/s 2024-03-24T00:40:34.460 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:40:34 smithi193 ceph-mon[36878]: pgmap v1197: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 5 op/s 2024-03-24T00:40:35.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:40:35 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 931.628239 secs 2024-03-24T00:40:35.704 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:40:35 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 931.628239 secs 2024-03-24T00:40:35.710 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:40:35 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 931.628239 secs 2024-03-24T00:40:36.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:40:36 smithi118 ceph-mon[31535]: pgmap v1198: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-03-24T00:40:36.704 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:40:36 smithi189 ceph-mon[36187]: pgmap v1198: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-03-24T00:40:36.711 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:40:36 smithi193 ceph-mon[36878]: pgmap v1198: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-03-24T00:40:38.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:40:38 smithi118 ceph-mon[31535]: pgmap v1199: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 5 op/s 2024-03-24T00:40:38.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:40:38 smithi189 ceph-mon[36187]: pgmap v1199: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 5 op/s 2024-03-24T00:40:38.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:40:38 smithi193 ceph-mon[36878]: pgmap v1199: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 5 op/s 2024-03-24T00:40:40.763 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:40:40 smithi193 ceph-mon[36878]: pgmap v1200: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-03-24T00:40:40.763 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:40:40 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 936.628295 secs 2024-03-24T00:40:40.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:40:40 smithi118 ceph-mon[31535]: pgmap v1200: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-03-24T00:40:40.947 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:40:40 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 936.628295 secs 2024-03-24T00:40:40.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:40:40 smithi189 ceph-mon[36187]: pgmap v1200: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-03-24T00:40:40.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:40:40 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 936.628295 secs 2024-03-24T00:40:42.756 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:40:42 smithi189 ceph-mon[36187]: pgmap v1201: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 5 op/s 2024-03-24T00:40:42.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:40:42 smithi118 ceph-mon[31535]: pgmap v1201: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 5 op/s 2024-03-24T00:40:42.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:40:42 smithi193 ceph-mon[36878]: pgmap v1201: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 5 op/s 2024-03-24T00:40:44.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:40:44 smithi118 ceph-mon[31535]: pgmap v1202: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T00:40:44.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:40:44 smithi189 ceph-mon[36187]: pgmap v1202: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T00:40:44.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:40:44 smithi193 ceph-mon[36878]: pgmap v1202: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T00:40:45.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:40:45 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 941.628394 secs 2024-03-24T00:40:45.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:40:45 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 941.628394 secs 2024-03-24T00:40:45.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:40:45 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 941.628394 secs 2024-03-24T00:40:46.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:40:46 smithi118 ceph-mon[31535]: pgmap v1203: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T00:40:46.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:40:46 smithi189 ceph-mon[36187]: pgmap v1203: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T00:40:46.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:40:46 smithi193 ceph-mon[36878]: pgmap v1203: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T00:40:48.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:40:48 smithi118 ceph-mon[31535]: pgmap v1204: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T00:40:48.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:40:48 smithi189 ceph-mon[36187]: pgmap v1204: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T00:40:48.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:40:48 smithi193 ceph-mon[36878]: pgmap v1204: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T00:40:50.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:40:50 smithi118 ceph-mon[31535]: pgmap v1205: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-03-24T00:40:50.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:40:50 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 946.628481 secs 2024-03-24T00:40:50.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:40:50 smithi189 ceph-mon[36187]: pgmap v1205: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-03-24T00:40:50.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:40:50 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 946.628481 secs 2024-03-24T00:40:50.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:40:50 smithi193 ceph-mon[36878]: pgmap v1205: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-03-24T00:40:50.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:40:50 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 946.628481 secs 2024-03-24T00:40:52.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:40:52 smithi118 ceph-mon[31535]: pgmap v1206: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:40:52.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:40:52 smithi189 ceph-mon[36187]: pgmap v1206: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:40:52.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:40:52 smithi193 ceph-mon[36878]: pgmap v1206: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:40:54.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:40:54 smithi118 ceph-mon[31535]: pgmap v1207: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T00:40:54.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:40:54 smithi189 ceph-mon[36187]: pgmap v1207: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T00:40:54.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:40:54 smithi193 ceph-mon[36878]: pgmap v1207: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T00:40:55.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:40:55 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 951.628566 secs 2024-03-24T00:40:55.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:40:55 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 951.628566 secs 2024-03-24T00:40:55.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:40:55 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 951.628566 secs 2024-03-24T00:40:56.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:40:56 smithi118 ceph-mon[31535]: pgmap v1208: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T00:40:56.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:40:56 smithi189 ceph-mon[36187]: pgmap v1208: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T00:40:56.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:40:56 smithi193 ceph-mon[36878]: pgmap v1208: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T00:40:58.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:40:58 smithi118 ceph-mon[31535]: pgmap v1209: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T00:40:58.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:40:58 smithi189 ceph-mon[36187]: pgmap v1209: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T00:40:58.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:40:58 smithi193 ceph-mon[36878]: pgmap v1209: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T00:41:00.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:41:00 smithi118 ceph-mon[31535]: pgmap v1210: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-03-24T00:41:00.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:41:00 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 956.628649 secs 2024-03-24T00:41:00.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:41:00 smithi189 ceph-mon[36187]: pgmap v1210: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-03-24T00:41:00.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:41:00 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 956.628649 secs 2024-03-24T00:41:00.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:41:00 smithi193 ceph-mon[36878]: pgmap v1210: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-03-24T00:41:00.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:41:00 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 956.628649 secs 2024-03-24T00:41:02.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:41:02 smithi118 ceph-mon[31535]: pgmap v1211: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:41:02.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:41:02 smithi189 ceph-mon[36187]: pgmap v1211: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:41:02.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:41:02 smithi193 ceph-mon[36878]: pgmap v1211: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:41:04.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:41:04 smithi118 ceph-mon[31535]: pgmap v1212: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T00:41:04.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:41:04 smithi189 ceph-mon[36187]: pgmap v1212: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T00:41:04.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:41:04 smithi193 ceph-mon[36878]: pgmap v1212: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T00:41:05.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:41:05 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 961.628739 secs 2024-03-24T00:41:05.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:41:05 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 961.628739 secs 2024-03-24T00:41:05.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:41:05 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 961.628739 secs 2024-03-24T00:41:06.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:41:06 smithi118 ceph-mon[31535]: pgmap v1213: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T00:41:06.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:41:06 smithi189 ceph-mon[36187]: pgmap v1213: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T00:41:06.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:41:06 smithi193 ceph-mon[36878]: pgmap v1213: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T00:41:08.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:41:08 smithi118 ceph-mon[31535]: pgmap v1214: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T00:41:08.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:41:08 smithi189 ceph-mon[36187]: pgmap v1214: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T00:41:08.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:41:08 smithi193 ceph-mon[36878]: pgmap v1214: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T00:41:10.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:41:10 smithi118 ceph-mon[31535]: pgmap v1215: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-03-24T00:41:10.947 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:41:10 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 966.628826 secs 2024-03-24T00:41:10.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:41:10 smithi189 ceph-mon[36187]: pgmap v1215: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-03-24T00:41:10.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:41:10 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 966.628826 secs 2024-03-24T00:41:10.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:41:10 smithi193 ceph-mon[36878]: pgmap v1215: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-03-24T00:41:10.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:41:10 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 966.628826 secs 2024-03-24T00:41:12.949 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:41:12 smithi118 ceph-mon[31535]: pgmap v1216: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:41:12.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:41:12 smithi189 ceph-mon[36187]: pgmap v1216: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:41:12.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:41:12 smithi193 ceph-mon[36878]: pgmap v1216: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:41:14.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:41:14 smithi118 ceph-mon[31535]: pgmap v1217: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T00:41:14.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:41:14 smithi189 ceph-mon[36187]: pgmap v1217: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T00:41:14.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:41:14 smithi193 ceph-mon[36878]: pgmap v1217: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T00:41:15.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:41:15 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 971.628915 secs 2024-03-24T00:41:15.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:41:15 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 971.628915 secs 2024-03-24T00:41:15.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:41:15 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 971.628915 secs 2024-03-24T00:41:16.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:41:16 smithi118 ceph-mon[31535]: pgmap v1218: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T00:41:16.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:41:16 smithi189 ceph-mon[36187]: pgmap v1218: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T00:41:16.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:41:16 smithi193 ceph-mon[36878]: pgmap v1218: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T00:41:18.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:41:18 smithi118 ceph-mon[31535]: pgmap v1219: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T00:41:18.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:41:18 smithi189 ceph-mon[36187]: pgmap v1219: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T00:41:18.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:41:18 smithi193 ceph-mon[36878]: pgmap v1219: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T00:41:20.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:41:20 smithi118 ceph-mon[31535]: pgmap v1220: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-03-24T00:41:20.947 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:41:20 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 976.629001 secs 2024-03-24T00:41:20.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:41:20 smithi189 ceph-mon[36187]: pgmap v1220: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-03-24T00:41:20.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:41:20 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 976.629001 secs 2024-03-24T00:41:20.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:41:20 smithi193 ceph-mon[36878]: pgmap v1220: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-03-24T00:41:20.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:41:20 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 976.629001 secs 2024-03-24T00:41:22.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:41:22 smithi118 ceph-mon[31535]: pgmap v1221: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:41:22.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:41:22 smithi189 ceph-mon[36187]: pgmap v1221: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:41:22.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:41:22 smithi193 ceph-mon[36878]: pgmap v1221: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:41:24.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:41:24 smithi118 ceph-mon[31535]: pgmap v1222: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T00:41:24.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:41:24 smithi189 ceph-mon[36187]: pgmap v1222: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T00:41:24.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:41:24 smithi193 ceph-mon[36878]: pgmap v1222: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T00:41:25.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:41:25 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 981.629080 secs 2024-03-24T00:41:25.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:41:25 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 981.629080 secs 2024-03-24T00:41:25.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:41:25 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 981.629080 secs 2024-03-24T00:41:26.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:41:26 smithi118 ceph-mon[31535]: pgmap v1223: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T00:41:26.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:41:26 smithi189 ceph-mon[36187]: pgmap v1223: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T00:41:26.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:41:26 smithi193 ceph-mon[36878]: pgmap v1223: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T00:41:28.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:41:28 smithi118 ceph-mon[31535]: pgmap v1224: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T00:41:28.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:41:28 smithi189 ceph-mon[36187]: pgmap v1224: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T00:41:28.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:41:28 smithi193 ceph-mon[36878]: pgmap v1224: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T00:41:30.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:41:30 smithi118 ceph-mon[31535]: pgmap v1225: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-03-24T00:41:30.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:41:30 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 986.629096 secs 2024-03-24T00:41:30.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:41:30 smithi189 ceph-mon[36187]: pgmap v1225: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-03-24T00:41:30.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:41:30 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 986.629096 secs 2024-03-24T00:41:30.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:41:30 smithi193 ceph-mon[36878]: pgmap v1225: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-03-24T00:41:30.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:41:30 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 986.629096 secs 2024-03-24T00:41:32.947 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:41:32 smithi118 ceph-mon[31535]: pgmap v1226: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:41:32.947 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:41:32 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T00:41:32.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:41:32 smithi189 ceph-mon[36187]: pgmap v1226: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:41:32.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:41:32 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T00:41:32.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:41:32 smithi193 ceph-mon[36878]: pgmap v1226: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:41:32.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:41:32 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T00:41:34.697 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:41:34 smithi118 ceph-mon[31535]: pgmap v1227: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T00:41:34.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:41:34 smithi189 ceph-mon[36187]: pgmap v1227: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T00:41:34.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:41:34 smithi193 ceph-mon[36878]: pgmap v1227: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T00:41:35.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:41:35 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 991.629158 secs 2024-03-24T00:41:35.953 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:41:35 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 991.629158 secs 2024-03-24T00:41:35.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:41:35 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 991.629158 secs 2024-03-24T00:41:36.947 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:41:36 smithi118 ceph-mon[31535]: pgmap v1228: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T00:41:36.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:41:36 smithi189 ceph-mon[36187]: pgmap v1228: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T00:41:36.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:41:36 smithi193 ceph-mon[36878]: pgmap v1228: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T00:41:38.947 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:41:38 smithi118 ceph-mon[31535]: pgmap v1229: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T00:41:38.948 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:41:38 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T00:41:38.948 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:41:38 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T00:41:38.948 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:41:38 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T00:41:38.948 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:41:38 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T00:41:38.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:41:38 smithi189 ceph-mon[36187]: pgmap v1229: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T00:41:38.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:41:38 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T00:41:38.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:41:38 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T00:41:38.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:41:38 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T00:41:38.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:41:38 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T00:41:38.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:41:38 smithi193 ceph-mon[36878]: pgmap v1229: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T00:41:38.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:41:38 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T00:41:38.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:41:38 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T00:41:38.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:41:38 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T00:41:38.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:41:38 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T00:41:39.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:41:39 smithi118 ceph-mon[31535]: pgmap v1230: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 6 op/s 2024-03-24T00:41:39.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:41:39 smithi118 ceph-mon[31535]: pgmap v1231: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 4 op/s 2024-03-24T00:41:39.947 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:41:39 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T00:41:39.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:41:39 smithi189 ceph-mon[36187]: pgmap v1230: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 6 op/s 2024-03-24T00:41:39.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:41:39 smithi189 ceph-mon[36187]: pgmap v1231: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 4 op/s 2024-03-24T00:41:39.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:41:39 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T00:41:39.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:41:39 smithi193 ceph-mon[36878]: pgmap v1230: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 6 op/s 2024-03-24T00:41:39.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:41:39 smithi193 ceph-mon[36878]: pgmap v1231: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 4 op/s 2024-03-24T00:41:39.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:41:39 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T00:41:40.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:41:40 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 996.629212 secs 2024-03-24T00:41:40.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:41:40 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 996.629212 secs 2024-03-24T00:41:40.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:41:40 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 996.629212 secs 2024-03-24T00:41:41.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:41:41 smithi118 ceph-mon[31535]: pgmap v1232: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T00:41:41.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:41:41 smithi189 ceph-mon[36187]: pgmap v1232: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T00:41:41.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:41:41 smithi193 ceph-mon[36878]: pgmap v1232: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T00:41:43.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:41:43 smithi118 ceph-mon[31535]: pgmap v1233: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 5 op/s 2024-03-24T00:41:43.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:41:43 smithi189 ceph-mon[36187]: pgmap v1233: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 5 op/s 2024-03-24T00:41:43.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:41:43 smithi193 ceph-mon[36878]: pgmap v1233: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 5 op/s 2024-03-24T00:41:45.882 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:41:45.881+0000 7f81ee000640 1 -- 172.21.15.118:0/1693767697 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- auth(proto 2 132 bytes epoch 0) v1 -- 0x7f81c80065e0 con 0x7f81f81052d0 2024-03-24T00:41:45.883 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:41:45.881+0000 7f81ef400640 1 -- 172.21.15.118:0/1693767697 <== mon.2 v2:172.21.15.189:3300/0 12 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 233+0+0 (secure 0 0 0) 0x7f81c80065e0 con 0x7f81f81052d0 2024-03-24T00:41:45.903 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:41:45.902+0000 7f81db400640 1 -- 172.21.15.118:0/659686424 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- auth(proto 2 132 bytes epoch 0) v1 -- 0x7f81c4006300 con 0x5622651a13f0 2024-03-24T00:41:45.904 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:41:45.903+0000 7f81e4c00640 1 -- 172.21.15.118:0/659686424 <== mon.1 v2:172.21.15.193:3300/0 12 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 233+0+0 (secure 0 0 0) 0x7f81c4006300 con 0x5622651a13f0 2024-03-24T00:41:45.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:41:45 smithi189 ceph-mon[36187]: pgmap v1234: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T00:41:45.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:41:45 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 1001.629274 secs 2024-03-24T00:41:45.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:41:45 smithi193 ceph-mon[36878]: pgmap v1234: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T00:41:45.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:41:45 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 1001.629274 secs 2024-03-24T00:41:46.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:41:45 smithi118 ceph-mon[31535]: pgmap v1234: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T00:41:46.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:41:45 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 1001.629274 secs 2024-03-24T00:41:47.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:41:47 smithi189 ceph-mon[36187]: pgmap v1235: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 5 op/s 2024-03-24T00:41:47.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:41:47 smithi193 ceph-mon[36878]: pgmap v1235: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 5 op/s 2024-03-24T00:41:48.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:41:47 smithi118 ceph-mon[31535]: pgmap v1235: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 5 op/s 2024-03-24T00:41:49.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:41:49 smithi193 ceph-mon[36878]: pgmap v1236: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 5 op/s 2024-03-24T00:41:50.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:41:49 smithi118 ceph-mon[31535]: pgmap v1236: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 5 op/s 2024-03-24T00:41:50.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:41:49 smithi189 ceph-mon[36187]: pgmap v1236: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 5 op/s 2024-03-24T00:41:51.098 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:41:50 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 1006.629333 secs 2024-03-24T00:41:51.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:41:50 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 1006.629333 secs 2024-03-24T00:41:51.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:41:50 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 1006.629333 secs 2024-03-24T00:41:52.124 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:41:51 smithi118 ceph-mon[31535]: pgmap v1237: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T00:41:52.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:41:51 smithi189 ceph-mon[36187]: pgmap v1237: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T00:41:52.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:41:51 smithi193 ceph-mon[36878]: pgmap v1237: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T00:41:53.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:41:53 smithi118 ceph-mon[31535]: pgmap v1238: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 5 op/s 2024-03-24T00:41:54.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:41:53 smithi189 ceph-mon[36187]: pgmap v1238: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 5 op/s 2024-03-24T00:41:54.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:41:53 smithi193 ceph-mon[36878]: pgmap v1238: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 5 op/s 2024-03-24T00:41:56.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:41:55 smithi118 ceph-mon[31535]: pgmap v1239: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T00:41:56.197 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:41:55 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 1011.629392 secs 2024-03-24T00:41:56.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:41:55 smithi189 ceph-mon[36187]: pgmap v1239: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T00:41:56.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:41:55 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 1011.629392 secs 2024-03-24T00:41:56.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:41:55 smithi193 ceph-mon[36878]: pgmap v1239: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T00:41:56.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:41:55 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 1011.629392 secs 2024-03-24T00:41:58.114 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:41:57 smithi118 ceph-mon[31535]: pgmap v1240: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:41:58.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:41:57 smithi189 ceph-mon[36187]: pgmap v1240: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:41:58.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:41:57 smithi193 ceph-mon[36878]: pgmap v1240: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:42:00.088 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:41:59 smithi118 ceph-mon[31535]: pgmap v1241: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T00:42:00.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:41:59 smithi189 ceph-mon[36187]: pgmap v1241: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T00:42:00.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:41:59 smithi193 ceph-mon[36878]: pgmap v1241: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T00:42:01.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:42:00 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 1016.629450 secs 2024-03-24T00:42:01.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:42:00 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 1016.629450 secs 2024-03-24T00:42:01.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:42:00 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 1016.629450 secs 2024-03-24T00:42:02.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:42:01 smithi118 ceph-mon[31535]: pgmap v1242: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T00:42:02.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:42:01 smithi189 ceph-mon[36187]: pgmap v1242: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T00:42:02.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:42:01 smithi193 ceph-mon[36878]: pgmap v1242: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T00:42:04.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:42:03 smithi118 ceph-mon[31535]: pgmap v1243: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 5 op/s 2024-03-24T00:42:04.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:42:03 smithi189 ceph-mon[36187]: pgmap v1243: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 5 op/s 2024-03-24T00:42:04.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:42:03 smithi193 ceph-mon[36878]: pgmap v1243: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 5 op/s 2024-03-24T00:42:06.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:42:05 smithi118 ceph-mon[31535]: pgmap v1244: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T00:42:06.197 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:42:05 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 1021.629511 secs 2024-03-24T00:42:06.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:42:05 smithi189 ceph-mon[36187]: pgmap v1244: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T00:42:06.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:42:05 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 1021.629511 secs 2024-03-24T00:42:06.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:42:05 smithi193 ceph-mon[36878]: pgmap v1244: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T00:42:06.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:42:05 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 1021.629511 secs 2024-03-24T00:42:08.172 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:42:07 smithi118 ceph-mon[31535]: pgmap v1245: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:42:08.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:42:07 smithi189 ceph-mon[36187]: pgmap v1245: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:42:08.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:42:07 smithi193 ceph-mon[36878]: pgmap v1245: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:42:10.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:42:09 smithi118 ceph-mon[31535]: pgmap v1246: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T00:42:10.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:42:09 smithi189 ceph-mon[36187]: pgmap v1246: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T00:42:10.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:42:09 smithi193 ceph-mon[36878]: pgmap v1246: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T00:42:11.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:42:10 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 1026.629549 secs 2024-03-24T00:42:11.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:42:10 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 1026.629549 secs 2024-03-24T00:42:11.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:42:10 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 1026.629549 secs 2024-03-24T00:42:12.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:42:11 smithi118 ceph-mon[31535]: pgmap v1247: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T00:42:12.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:42:11 smithi189 ceph-mon[36187]: pgmap v1247: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T00:42:12.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:42:11 smithi193 ceph-mon[36878]: pgmap v1247: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T00:42:14.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:42:13 smithi118 ceph-mon[31535]: pgmap v1248: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 5 op/s 2024-03-24T00:42:14.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:42:13 smithi189 ceph-mon[36187]: pgmap v1248: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 5 op/s 2024-03-24T00:42:14.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:42:13 smithi193 ceph-mon[36878]: pgmap v1248: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 5 op/s 2024-03-24T00:42:16.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:42:15 smithi118 ceph-mon[31535]: pgmap v1249: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T00:42:16.197 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:42:15 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 1031.629584 secs 2024-03-24T00:42:16.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:42:15 smithi189 ceph-mon[36187]: pgmap v1249: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T00:42:16.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:42:15 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 1031.629584 secs 2024-03-24T00:42:16.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:42:15 smithi193 ceph-mon[36878]: pgmap v1249: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T00:42:16.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:42:15 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 1031.629584 secs 2024-03-24T00:42:18.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:42:17 smithi118 ceph-mon[31535]: pgmap v1250: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:42:18.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:42:17 smithi189 ceph-mon[36187]: pgmap v1250: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:42:18.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:42:17 smithi193 ceph-mon[36878]: pgmap v1250: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:42:20.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:42:19 smithi118 ceph-mon[31535]: pgmap v1251: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T00:42:20.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:42:19 smithi189 ceph-mon[36187]: pgmap v1251: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T00:42:20.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:42:19 smithi193 ceph-mon[36878]: pgmap v1251: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T00:42:21.098 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:42:20 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 1036.629645 secs 2024-03-24T00:42:21.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:42:20 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 1036.629645 secs 2024-03-24T00:42:21.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:42:20 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 1036.629645 secs 2024-03-24T00:42:22.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:42:21 smithi118 ceph-mon[31535]: pgmap v1252: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T00:42:22.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:42:21 smithi189 ceph-mon[36187]: pgmap v1252: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T00:42:22.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:42:21 smithi193 ceph-mon[36878]: pgmap v1252: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T00:42:24.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:42:23 smithi118 ceph-mon[31535]: pgmap v1253: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 5 op/s 2024-03-24T00:42:24.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:42:23 smithi189 ceph-mon[36187]: pgmap v1253: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 5 op/s 2024-03-24T00:42:24.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:42:23 smithi193 ceph-mon[36878]: pgmap v1253: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 5 op/s 2024-03-24T00:42:26.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:42:25 smithi118 ceph-mon[31535]: pgmap v1254: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T00:42:26.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:42:25 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 1041.629704 secs 2024-03-24T00:42:26.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:42:25 smithi189 ceph-mon[36187]: pgmap v1254: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T00:42:26.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:42:25 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 1041.629704 secs 2024-03-24T00:42:26.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:42:25 smithi193 ceph-mon[36878]: pgmap v1254: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T00:42:26.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:42:25 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 1041.629704 secs 2024-03-24T00:42:28.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:42:27 smithi118 ceph-mon[31535]: pgmap v1255: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:42:28.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:42:27 smithi189 ceph-mon[36187]: pgmap v1255: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:42:28.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:42:27 smithi193 ceph-mon[36878]: pgmap v1255: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:42:30.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:42:29 smithi118 ceph-mon[31535]: pgmap v1256: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T00:42:30.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:42:29 smithi189 ceph-mon[36187]: pgmap v1256: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T00:42:30.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:42:29 smithi193 ceph-mon[36878]: pgmap v1256: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T00:42:31.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:42:30 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 1046.629763 secs 2024-03-24T00:42:31.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:42:30 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 1046.629763 secs 2024-03-24T00:42:31.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:42:30 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 1046.629763 secs 2024-03-24T00:42:32.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:42:31 smithi118 ceph-mon[31535]: pgmap v1257: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T00:42:32.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:42:31 smithi189 ceph-mon[36187]: pgmap v1257: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T00:42:32.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:42:31 smithi193 ceph-mon[36878]: pgmap v1257: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T00:42:34.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:42:33 smithi118 ceph-mon[31535]: pgmap v1258: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 5 op/s 2024-03-24T00:42:34.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:42:33 smithi189 ceph-mon[36187]: pgmap v1258: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 5 op/s 2024-03-24T00:42:34.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:42:33 smithi193 ceph-mon[36878]: pgmap v1258: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 5 op/s 2024-03-24T00:42:36.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:42:35 smithi118 ceph-mon[31535]: pgmap v1259: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T00:42:36.197 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:42:35 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 1051.629852 secs 2024-03-24T00:42:36.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:42:35 smithi189 ceph-mon[36187]: pgmap v1259: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T00:42:36.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:42:35 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 1051.629852 secs 2024-03-24T00:42:36.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:42:35 smithi193 ceph-mon[36878]: pgmap v1259: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T00:42:36.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:42:35 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 1051.629852 secs 2024-03-24T00:42:38.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:42:37 smithi118 ceph-mon[31535]: pgmap v1260: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:42:38.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:42:37 smithi189 ceph-mon[36187]: pgmap v1260: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:42:38.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:42:37 smithi193 ceph-mon[36878]: pgmap v1260: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:42:40.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:42:39 smithi118 ceph-mon[31535]: pgmap v1261: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T00:42:40.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:42:39 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T00:42:40.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:42:39 smithi189 ceph-mon[36187]: pgmap v1261: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T00:42:40.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:42:39 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T00:42:40.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:42:39 smithi193 ceph-mon[36878]: pgmap v1261: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T00:42:40.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:42:39 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T00:42:40.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:42:40 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-24T00:42:40.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:42:40 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-24T00:42:40.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:42:40 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-24T00:42:40.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:42:40 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-03-24T00:42:40.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:42:40 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 1056.629919 secs 2024-03-24T00:42:41.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:42:40 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-24T00:42:41.197 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:42:40 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-24T00:42:41.197 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:42:40 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-24T00:42:41.197 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:42:40 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-03-24T00:42:41.197 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:42:40 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 1056.629919 secs 2024-03-24T00:42:41.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:42:40 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-24T00:42:41.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:42:40 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-24T00:42:41.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:42:40 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-24T00:42:41.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:42:40 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-03-24T00:42:41.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:42:40 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 1056.629919 secs 2024-03-24T00:42:41.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:42:41 smithi193 ceph-mon[36878]: Adjusting osd_memory_target on smithi118 to 176.7M 2024-03-24T00:42:41.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:42:41 smithi193 ceph-mon[36878]: Unable to set osd_memory_target on smithi118 to 185365196: error parsing value: Value '185365196' is below minimum 939524096 2024-03-24T00:42:41.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:42:41 smithi193 ceph-mon[36878]: pgmap v1262: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T00:42:42.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:42:41 smithi118 ceph-mon[31535]: Adjusting osd_memory_target on smithi118 to 176.7M 2024-03-24T00:42:42.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:42:41 smithi118 ceph-mon[31535]: Unable to set osd_memory_target on smithi118 to 185365196: error parsing value: Value '185365196' is below minimum 939524096 2024-03-24T00:42:42.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:42:41 smithi118 ceph-mon[31535]: pgmap v1262: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T00:42:42.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:42:41 smithi189 ceph-mon[36187]: Adjusting osd_memory_target on smithi118 to 176.7M 2024-03-24T00:42:42.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:42:41 smithi189 ceph-mon[36187]: Unable to set osd_memory_target on smithi118 to 185365196: error parsing value: Value '185365196' is below minimum 939524096 2024-03-24T00:42:42.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:42:41 smithi189 ceph-mon[36187]: pgmap v1262: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T00:42:44.197 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:42:43 smithi118 ceph-mon[31535]: pgmap v1263: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 5 op/s 2024-03-24T00:42:44.197 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:42:43 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T00:42:44.197 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:42:43 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T00:42:44.197 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:42:43 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T00:42:44.197 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:42:43 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T00:42:44.197 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:42:43 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-24T00:42:44.197 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:42:43 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-24T00:42:44.197 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:42:43 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-24T00:42:44.197 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:42:43 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-03-24T00:42:44.198 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:42:43 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T00:42:44.198 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:42:43 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T00:42:44.198 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:42:43 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T00:42:44.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:42:43 smithi189 ceph-mon[36187]: pgmap v1263: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 5 op/s 2024-03-24T00:42:44.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:42:43 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T00:42:44.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:42:43 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T00:42:44.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:42:43 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T00:42:44.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:42:43 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T00:42:44.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:42:43 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-24T00:42:44.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:42:43 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-24T00:42:44.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:42:43 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-24T00:42:44.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:42:43 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-03-24T00:42:44.205 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:42:43 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T00:42:44.205 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:42:43 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T00:42:44.205 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:42:43 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T00:42:44.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:42:43 smithi193 ceph-mon[36878]: pgmap v1263: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 5 op/s 2024-03-24T00:42:44.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:42:43 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T00:42:44.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:42:43 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T00:42:44.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:42:43 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T00:42:44.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:42:43 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T00:42:44.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:42:43 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-24T00:42:44.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:42:43 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-24T00:42:44.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:42:43 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-24T00:42:44.212 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:42:43 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-03-24T00:42:44.212 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:42:43 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T00:42:44.212 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:42:43 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T00:42:44.212 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:42:43 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T00:42:45.197 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:42:44 smithi118 ceph-mon[31535]: Adjusting osd_memory_target on smithi189 to 174.1M 2024-03-24T00:42:45.197 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:42:44 smithi118 ceph-mon[31535]: Unable to set osd_memory_target on smithi189 to 182635622: error parsing value: Value '182635622' is below minimum 939524096 2024-03-24T00:42:45.197 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:42:44 smithi118 ceph-mon[31535]: pgmap v1264: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 6 op/s 2024-03-24T00:42:45.197 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:42:44 smithi118 ceph-mon[31535]: pgmap v1265: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T00:42:45.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:42:44 smithi189 ceph-mon[36187]: Adjusting osd_memory_target on smithi189 to 174.1M 2024-03-24T00:42:45.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:42:44 smithi189 ceph-mon[36187]: Unable to set osd_memory_target on smithi189 to 182635622: error parsing value: Value '182635622' is below minimum 939524096 2024-03-24T00:42:45.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:42:44 smithi189 ceph-mon[36187]: pgmap v1264: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 6 op/s 2024-03-24T00:42:45.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:42:44 smithi189 ceph-mon[36187]: pgmap v1265: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T00:42:45.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:42:44 smithi193 ceph-mon[36878]: Adjusting osd_memory_target on smithi189 to 174.1M 2024-03-24T00:42:45.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:42:44 smithi193 ceph-mon[36878]: Unable to set osd_memory_target on smithi189 to 182635622: error parsing value: Value '182635622' is below minimum 939524096 2024-03-24T00:42:45.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:42:44 smithi193 ceph-mon[36878]: pgmap v1264: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 6 op/s 2024-03-24T00:42:45.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:42:44 smithi193 ceph-mon[36878]: pgmap v1265: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T00:42:46.168 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:42:45 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 1061.629979 secs 2024-03-24T00:42:46.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:42:45 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 1061.629979 secs 2024-03-24T00:42:46.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:42:45 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 1061.629979 secs 2024-03-24T00:42:47.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:42:46 smithi118 ceph-mon[31535]: pgmap v1266: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 6 op/s 2024-03-24T00:42:47.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:42:46 smithi189 ceph-mon[36187]: pgmap v1266: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 6 op/s 2024-03-24T00:42:47.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:42:46 smithi193 ceph-mon[36878]: pgmap v1266: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 6 op/s 2024-03-24T00:42:49.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:42:48 smithi118 ceph-mon[31535]: pgmap v1267: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T00:42:49.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:42:48 smithi189 ceph-mon[36187]: pgmap v1267: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T00:42:49.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:42:48 smithi193 ceph-mon[36878]: pgmap v1267: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T00:42:51.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:42:50 smithi118 ceph-mon[31535]: pgmap v1268: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 6 op/s 2024-03-24T00:42:51.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:42:50 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 1066.630035 secs 2024-03-24T00:42:51.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:42:50 smithi189 ceph-mon[36187]: pgmap v1268: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 6 op/s 2024-03-24T00:42:51.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:42:50 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 1066.630035 secs 2024-03-24T00:42:51.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:42:50 smithi193 ceph-mon[36878]: pgmap v1268: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 6 op/s 2024-03-24T00:42:51.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:42:50 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 1066.630035 secs 2024-03-24T00:42:53.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:42:52 smithi118 ceph-mon[31535]: pgmap v1269: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 5 op/s 2024-03-24T00:42:53.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:42:52 smithi189 ceph-mon[36187]: pgmap v1269: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 5 op/s 2024-03-24T00:42:53.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:42:52 smithi193 ceph-mon[36878]: pgmap v1269: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 5 op/s 2024-03-24T00:42:55.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:42:54 smithi118 ceph-mon[31535]: pgmap v1270: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 5 op/s 2024-03-24T00:42:55.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:42:54 smithi189 ceph-mon[36187]: pgmap v1270: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 5 op/s 2024-03-24T00:42:55.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:42:54 smithi193 ceph-mon[36878]: pgmap v1270: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 5 op/s 2024-03-24T00:42:56.197 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:42:55 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 1071.630098 secs 2024-03-24T00:42:56.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:42:55 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 1071.630098 secs 2024-03-24T00:42:56.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:42:55 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 1071.630098 secs 2024-03-24T00:42:57.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:42:56 smithi118 ceph-mon[31535]: pgmap v1271: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T00:42:57.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:42:56 smithi189 ceph-mon[36187]: pgmap v1271: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T00:42:57.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:42:56 smithi193 ceph-mon[36878]: pgmap v1271: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T00:42:59.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:42:58 smithi118 ceph-mon[31535]: pgmap v1272: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T00:42:59.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:42:58 smithi189 ceph-mon[36187]: pgmap v1272: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T00:42:59.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:42:58 smithi193 ceph-mon[36878]: pgmap v1272: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T00:43:01.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:43:00 smithi118 ceph-mon[31535]: pgmap v1273: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-03-24T00:43:01.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:43:00 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 1076.630158 secs 2024-03-24T00:43:01.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:43:00 smithi189 ceph-mon[36187]: pgmap v1273: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-03-24T00:43:01.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:43:00 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 1076.630158 secs 2024-03-24T00:43:01.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:43:00 smithi193 ceph-mon[36878]: pgmap v1273: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-03-24T00:43:01.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:43:00 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 1076.630158 secs 2024-03-24T00:43:03.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:43:02 smithi118 ceph-mon[31535]: pgmap v1274: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:43:03.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:43:02 smithi189 ceph-mon[36187]: pgmap v1274: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:43:03.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:43:02 smithi193 ceph-mon[36878]: pgmap v1274: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:43:05.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:43:04 smithi118 ceph-mon[31535]: pgmap v1275: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T00:43:05.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:43:04 smithi189 ceph-mon[36187]: pgmap v1275: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T00:43:05.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:43:04 smithi193 ceph-mon[36878]: pgmap v1275: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T00:43:06.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:43:05 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 1081.630220 secs 2024-03-24T00:43:06.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:43:05 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 1081.630220 secs 2024-03-24T00:43:06.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:43:05 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 1081.630220 secs 2024-03-24T00:43:07.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:43:06 smithi189 ceph-mon[36187]: pgmap v1276: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T00:43:07.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:43:06 smithi193 ceph-mon[36878]: pgmap v1276: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T00:43:07.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:43:06 smithi118 ceph-mon[31535]: pgmap v1276: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T00:43:09.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:43:08 smithi189 ceph-mon[36187]: pgmap v1277: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T00:43:09.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:43:08 smithi193 ceph-mon[36878]: pgmap v1277: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T00:43:09.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:43:08 smithi118 ceph-mon[31535]: pgmap v1277: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T00:43:11.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:43:10 smithi193 ceph-mon[36878]: pgmap v1278: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-03-24T00:43:11.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:43:10 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 1086.630281 secs 2024-03-24T00:43:11.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:43:10 smithi118 ceph-mon[31535]: pgmap v1278: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-03-24T00:43:11.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:43:10 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 1086.630281 secs 2024-03-24T00:43:11.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:43:10 smithi189 ceph-mon[36187]: pgmap v1278: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-03-24T00:43:11.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:43:10 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 1086.630281 secs 2024-03-24T00:43:13.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:43:12 smithi193 ceph-mon[36878]: pgmap v1279: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:43:13.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:43:12 smithi118 ceph-mon[31535]: pgmap v1279: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:43:13.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:43:12 smithi189 ceph-mon[36187]: pgmap v1279: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:43:15.256 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:43:14 smithi189 ceph-mon[36187]: pgmap v1280: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T00:43:15.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:43:14 smithi118 ceph-mon[31535]: pgmap v1280: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T00:43:15.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:43:14 smithi193 ceph-mon[36878]: pgmap v1280: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T00:43:15.897 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:43:15.896+0000 7f81ee000640 1 -- 172.21.15.118:0/1693767697 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f81c80036c0 con 0x7f81f81052d0 2024-03-24T00:43:15.897 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:43:15.896+0000 7f81ef400640 1 -- 172.21.15.118:0/1693767697 <== mon.2 v2:172.21.15.189:3300/0 13 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (secure 0 0 0) 0x7f81c80036c0 con 0x7f81f81052d0 2024-03-24T00:43:15.917 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:43:15.916+0000 7f81db400640 1 -- 172.21.15.118:0/659686424 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f81c40064d0 con 0x5622651a13f0 2024-03-24T00:43:15.917 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:43:15.916+0000 7f81e4c00640 1 -- 172.21.15.118:0/659686424 <== mon.1 v2:172.21.15.193:3300/0 13 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 409+0+0 (secure 0 0 0) 0x7f81c40064d0 con 0x5622651a13f0 2024-03-24T00:43:16.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:43:15 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 1091.630342 secs 2024-03-24T00:43:16.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:43:15 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 1091.630342 secs 2024-03-24T00:43:16.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:43:15 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 1091.630342 secs 2024-03-24T00:43:17.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:43:16 smithi118 ceph-mon[31535]: pgmap v1281: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T00:43:17.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:43:16 smithi189 ceph-mon[36187]: pgmap v1281: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T00:43:17.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:43:16 smithi193 ceph-mon[36878]: pgmap v1281: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T00:43:19.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:43:18 smithi118 ceph-mon[31535]: pgmap v1282: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T00:43:19.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:43:18 smithi189 ceph-mon[36187]: pgmap v1282: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T00:43:19.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:43:18 smithi193 ceph-mon[36878]: pgmap v1282: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T00:43:21.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:43:20 smithi118 ceph-mon[31535]: pgmap v1283: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-03-24T00:43:21.447 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:43:20 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 1096.630432 secs 2024-03-24T00:43:21.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:43:20 smithi189 ceph-mon[36187]: pgmap v1283: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-03-24T00:43:21.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:43:20 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 1096.630432 secs 2024-03-24T00:43:21.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:43:20 smithi193 ceph-mon[36878]: pgmap v1283: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-03-24T00:43:21.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:43:20 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 1096.630432 secs 2024-03-24T00:43:23.249 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:43:22 smithi118 ceph-mon[31535]: pgmap v1284: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:43:23.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:43:22 smithi189 ceph-mon[36187]: pgmap v1284: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:43:23.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:43:22 smithi193 ceph-mon[36878]: pgmap v1284: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:43:25.447 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:43:24 smithi118 ceph-mon[31535]: pgmap v1285: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T00:43:25.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:43:24 smithi189 ceph-mon[36187]: pgmap v1285: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T00:43:25.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:43:24 smithi193 ceph-mon[36878]: pgmap v1285: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T00:43:26.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:43:26 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 1101.630494 secs 2024-03-24T00:43:26.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:43:26 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 1101.630494 secs 2024-03-24T00:43:26.460 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:43:26 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 1101.630494 secs 2024-03-24T00:43:27.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:43:27 smithi118 ceph-mon[31535]: pgmap v1286: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T00:43:27.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:43:27 smithi189 ceph-mon[36187]: pgmap v1286: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T00:43:27.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:43:27 smithi193 ceph-mon[36878]: pgmap v1286: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T00:43:28.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:43:28 smithi118 ceph-mon[31535]: pgmap v1287: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T00:43:28.704 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:43:28 smithi189 ceph-mon[36187]: pgmap v1287: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T00:43:28.710 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:43:28 smithi193 ceph-mon[36878]: pgmap v1287: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T00:43:30.499 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:43:30 smithi118 ceph-mon[31535]: pgmap v1288: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-03-24T00:43:30.499 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:43:30 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 1106.630509 secs 2024-03-24T00:43:30.704 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:43:30 smithi189 ceph-mon[36187]: pgmap v1288: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-03-24T00:43:30.704 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:43:30 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 1106.630509 secs 2024-03-24T00:43:30.711 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:43:30 smithi193 ceph-mon[36878]: pgmap v1288: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-03-24T00:43:30.711 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:43:30 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 1106.630509 secs 2024-03-24T00:43:32.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:43:32 smithi118 ceph-mon[31535]: pgmap v1289: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:43:32.703 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:43:32 smithi189 ceph-mon[36187]: pgmap v1289: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:43:32.710 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:43:32 smithi193 ceph-mon[36878]: pgmap v1289: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:43:34.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:43:34 smithi118 ceph-mon[31535]: pgmap v1290: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T00:43:34.704 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:43:34 smithi189 ceph-mon[36187]: pgmap v1290: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T00:43:34.711 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:43:34 smithi193 ceph-mon[36878]: pgmap v1290: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T00:43:35.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:43:35 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 1111.630585 secs 2024-03-24T00:43:35.704 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:43:35 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 1111.630585 secs 2024-03-24T00:43:35.710 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:43:35 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 1111.630585 secs 2024-03-24T00:43:36.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:43:36 smithi118 ceph-mon[31535]: pgmap v1291: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T00:43:36.704 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:43:36 smithi189 ceph-mon[36187]: pgmap v1291: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T00:43:36.710 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:43:36 smithi193 ceph-mon[36878]: pgmap v1291: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T00:43:38.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:43:38 smithi118 ceph-mon[31535]: pgmap v1292: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T00:43:38.704 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:43:38 smithi189 ceph-mon[36187]: pgmap v1292: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T00:43:38.710 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:43:38 smithi193 ceph-mon[36878]: pgmap v1292: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T00:43:40.704 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:43:40 smithi189 ceph-mon[36187]: pgmap v1293: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-03-24T00:43:40.704 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:43:40 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 1116.630668 secs 2024-03-24T00:43:40.711 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:43:40 smithi193 ceph-mon[36878]: pgmap v1293: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-03-24T00:43:40.711 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:43:40 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 1116.630668 secs 2024-03-24T00:43:40.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:43:40 smithi118 ceph-mon[31535]: pgmap v1293: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-03-24T00:43:40.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:43:40 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 1116.630668 secs 2024-03-24T00:43:42.704 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:43:42 smithi189 ceph-mon[36187]: pgmap v1294: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:43:42.710 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:43:42 smithi193 ceph-mon[36878]: pgmap v1294: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:43:42.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:43:42 smithi118 ceph-mon[31535]: pgmap v1294: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:43:44.711 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:43:44 smithi193 ceph-mon[36878]: pgmap v1295: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-03-24T00:43:44.711 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:43:44 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T00:43:44.711 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:43:44 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-24T00:43:44.711 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:43:44 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-24T00:43:44.711 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:43:44 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-03-24T00:43:44.711 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:43:44 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-03-24T00:43:44.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:43:44 smithi118 ceph-mon[31535]: pgmap v1295: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-03-24T00:43:44.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:43:44 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T00:43:44.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:43:44 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-24T00:43:44.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:43:44 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-24T00:43:44.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:43:44 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-03-24T00:43:44.947 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:43:44 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-03-24T00:43:44.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:43:44 smithi189 ceph-mon[36187]: pgmap v1295: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-03-24T00:43:44.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:43:44 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T00:43:44.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:43:44 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-24T00:43:44.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:43:44 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-24T00:43:44.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:43:44 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-03-24T00:43:44.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:43:44 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-03-24T00:43:45.859 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:43:45 smithi189 ceph-mon[36187]: Adjusting osd_memory_target on smithi193 to 174.1M 2024-03-24T00:43:45.859 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:43:45 smithi189 ceph-mon[36187]: Unable to set osd_memory_target on smithi193 to 182639923: error parsing value: Value '182639923' is below minimum 939524096 2024-03-24T00:43:45.859 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:43:45 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T00:43:45.859 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:43:45 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T00:43:45.860 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:43:45 smithi189 ceph-mon[36187]: pgmap v1296: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 6 op/s 2024-03-24T00:43:45.860 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:43:45 smithi189 ceph-mon[36187]: pgmap v1297: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 6 op/s 2024-03-24T00:43:45.860 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:43:45 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T00:43:45.860 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:43:45 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 1121.630755 secs 2024-03-24T00:43:45.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:43:45 smithi118 ceph-mon[31535]: Adjusting osd_memory_target on smithi193 to 174.1M 2024-03-24T00:43:45.947 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:43:45 smithi118 ceph-mon[31535]: Unable to set osd_memory_target on smithi193 to 182639923: error parsing value: Value '182639923' is below minimum 939524096 2024-03-24T00:43:45.947 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:43:45 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T00:43:45.947 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:43:45 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T00:43:45.947 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:43:45 smithi118 ceph-mon[31535]: pgmap v1296: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 6 op/s 2024-03-24T00:43:45.947 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:43:45 smithi118 ceph-mon[31535]: pgmap v1297: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 6 op/s 2024-03-24T00:43:45.947 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:43:45 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T00:43:45.947 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:43:45 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 1121.630755 secs 2024-03-24T00:43:45.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:43:45 smithi193 ceph-mon[36878]: Adjusting osd_memory_target on smithi193 to 174.1M 2024-03-24T00:43:45.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:43:45 smithi193 ceph-mon[36878]: Unable to set osd_memory_target on smithi193 to 182639923: error parsing value: Value '182639923' is below minimum 939524096 2024-03-24T00:43:45.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:43:45 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T00:43:45.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:43:45 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T00:43:45.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:43:45 smithi193 ceph-mon[36878]: pgmap v1296: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 6 op/s 2024-03-24T00:43:45.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:43:45 smithi193 ceph-mon[36878]: pgmap v1297: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 6 op/s 2024-03-24T00:43:45.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:43:45 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T00:43:45.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:43:45 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 1121.630755 secs 2024-03-24T00:43:48.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:43:47 smithi118 ceph-mon[31535]: pgmap v1298: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-03-24T00:43:48.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:43:47 smithi189 ceph-mon[36187]: pgmap v1298: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-03-24T00:43:48.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:43:47 smithi193 ceph-mon[36878]: pgmap v1298: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-03-24T00:43:50.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:43:49 smithi118 ceph-mon[31535]: pgmap v1299: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:43:50.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:43:49 smithi189 ceph-mon[36187]: pgmap v1299: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:43:50.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:43:49 smithi193 ceph-mon[36878]: pgmap v1299: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:43:51.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:43:50 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 1126.630842 secs 2024-03-24T00:43:51.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:43:50 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 1126.630842 secs 2024-03-24T00:43:51.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:43:50 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 1126.630842 secs 2024-03-24T00:43:52.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:43:51 smithi118 ceph-mon[31535]: pgmap v1300: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 6 op/s 2024-03-24T00:43:52.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:43:51 smithi189 ceph-mon[36187]: pgmap v1300: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 6 op/s 2024-03-24T00:43:52.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:43:51 smithi193 ceph-mon[36878]: pgmap v1300: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 6 op/s 2024-03-24T00:43:54.197 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:43:53 smithi118 ceph-mon[31535]: pgmap v1301: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 5 op/s 2024-03-24T00:43:54.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:43:53 smithi189 ceph-mon[36187]: pgmap v1301: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 5 op/s 2024-03-24T00:43:54.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:43:53 smithi193 ceph-mon[36878]: pgmap v1301: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 5 op/s 2024-03-24T00:43:56.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:43:55 smithi118 ceph-mon[31535]: pgmap v1302: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 5 op/s 2024-03-24T00:43:56.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:43:55 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 1131.630858 secs 2024-03-24T00:43:56.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:43:55 smithi189 ceph-mon[36187]: pgmap v1302: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 5 op/s 2024-03-24T00:43:56.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:43:55 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 1131.630858 secs 2024-03-24T00:43:56.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:43:55 smithi193 ceph-mon[36878]: pgmap v1302: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 5 op/s 2024-03-24T00:43:56.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:43:55 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 1131.630858 secs 2024-03-24T00:43:58.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:43:57 smithi118 ceph-mon[31535]: pgmap v1303: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:43:58.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:43:57 smithi189 ceph-mon[36187]: pgmap v1303: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:43:58.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:43:57 smithi193 ceph-mon[36878]: pgmap v1303: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:44:00.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:43:59 smithi118 ceph-mon[31535]: pgmap v1304: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T00:44:00.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:43:59 smithi189 ceph-mon[36187]: pgmap v1304: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T00:44:00.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:43:59 smithi193 ceph-mon[36878]: pgmap v1304: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T00:44:01.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:44:00 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 1136.630895 secs 2024-03-24T00:44:01.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:44:00 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 1136.630895 secs 2024-03-24T00:44:01.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:44:00 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 1136.630895 secs 2024-03-24T00:44:02.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:44:01 smithi193 ceph-mon[36878]: pgmap v1305: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T00:44:02.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:44:01 smithi118 ceph-mon[31535]: pgmap v1305: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T00:44:02.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:44:01 smithi189 ceph-mon[36187]: pgmap v1305: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T00:44:03.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:44:03 smithi118 ceph-mon[31535]: pgmap v1306: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 5 op/s 2024-03-24T00:44:03.704 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:44:03 smithi189 ceph-mon[36187]: pgmap v1306: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 5 op/s 2024-03-24T00:44:03.710 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:44:03 smithi193 ceph-mon[36878]: pgmap v1306: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 5 op/s 2024-03-24T00:44:06.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:44:05 smithi118 ceph-mon[31535]: pgmap v1307: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T00:44:06.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:44:05 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 1141.630958 secs 2024-03-24T00:44:06.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:44:05 smithi189 ceph-mon[36187]: pgmap v1307: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T00:44:06.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:44:05 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 1141.630958 secs 2024-03-24T00:44:06.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:44:05 smithi193 ceph-mon[36878]: pgmap v1307: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T00:44:06.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:44:05 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 1141.630958 secs 2024-03-24T00:44:08.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:44:07 smithi118 ceph-mon[31535]: pgmap v1308: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:44:08.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:44:07 smithi189 ceph-mon[36187]: pgmap v1308: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:44:08.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:44:07 smithi193 ceph-mon[36878]: pgmap v1308: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:44:10.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:44:09 smithi118 ceph-mon[31535]: pgmap v1309: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T00:44:10.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:44:09 smithi189 ceph-mon[36187]: pgmap v1309: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T00:44:10.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:44:09 smithi193 ceph-mon[36878]: pgmap v1309: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T00:44:11.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:44:10 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 1146.631021 secs 2024-03-24T00:44:11.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:44:10 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 1146.631021 secs 2024-03-24T00:44:11.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:44:10 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 1146.631021 secs 2024-03-24T00:44:12.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:44:11 smithi118 ceph-mon[31535]: pgmap v1310: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T00:44:12.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:44:11 smithi189 ceph-mon[36187]: pgmap v1310: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T00:44:12.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:44:11 smithi193 ceph-mon[36878]: pgmap v1310: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T00:44:14.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:44:13 smithi118 ceph-mon[31535]: pgmap v1311: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 5 op/s 2024-03-24T00:44:14.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:44:13 smithi189 ceph-mon[36187]: pgmap v1311: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 5 op/s 2024-03-24T00:44:14.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:44:13 smithi193 ceph-mon[36878]: pgmap v1311: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 5 op/s 2024-03-24T00:44:16.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:44:15 smithi189 ceph-mon[36187]: pgmap v1312: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T00:44:16.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:44:15 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 1151.631099 secs 2024-03-24T00:44:16.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:44:15 smithi193 ceph-mon[36878]: pgmap v1312: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T00:44:16.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:44:15 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 1151.631099 secs 2024-03-24T00:44:16.280 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:44:15 smithi118 ceph-mon[31535]: pgmap v1312: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T00:44:16.281 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:44:15 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 1151.631099 secs 2024-03-24T00:44:18.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:44:17 smithi189 ceph-mon[36187]: pgmap v1313: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:44:18.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:44:17 smithi193 ceph-mon[36878]: pgmap v1313: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:44:18.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:44:17 smithi118 ceph-mon[31535]: pgmap v1313: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:44:20.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:44:19 smithi193 ceph-mon[36878]: pgmap v1314: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T00:44:20.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:44:19 smithi118 ceph-mon[31535]: pgmap v1314: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T00:44:20.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:44:19 smithi189 ceph-mon[36187]: pgmap v1314: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T00:44:21.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:44:20 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 1156.631134 secs 2024-03-24T00:44:21.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:44:20 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 1156.631134 secs 2024-03-24T00:44:21.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:44:20 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 1156.631134 secs 2024-03-24T00:44:22.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:44:21 smithi118 ceph-mon[31535]: pgmap v1315: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T00:44:22.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:44:21 smithi189 ceph-mon[36187]: pgmap v1315: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T00:44:22.460 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:44:21 smithi193 ceph-mon[36878]: pgmap v1315: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T00:44:24.249 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:44:23 smithi118 ceph-mon[31535]: pgmap v1316: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 5 op/s 2024-03-24T00:44:24.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:44:23 smithi189 ceph-mon[36187]: pgmap v1316: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 5 op/s 2024-03-24T00:44:24.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:44:23 smithi193 ceph-mon[36878]: pgmap v1316: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 5 op/s 2024-03-24T00:44:26.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:44:25 smithi118 ceph-mon[31535]: pgmap v1317: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T00:44:26.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:44:25 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 1161.631197 secs 2024-03-24T00:44:26.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:44:25 smithi189 ceph-mon[36187]: pgmap v1317: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T00:44:26.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:44:25 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 1161.631197 secs 2024-03-24T00:44:26.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:44:25 smithi193 ceph-mon[36878]: pgmap v1317: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T00:44:26.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:44:25 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 1161.631197 secs 2024-03-24T00:44:28.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:44:27 smithi118 ceph-mon[31535]: pgmap v1318: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:44:28.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:44:27 smithi189 ceph-mon[36187]: pgmap v1318: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:44:28.460 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:44:27 smithi193 ceph-mon[36878]: pgmap v1318: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:44:30.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:44:29 smithi118 ceph-mon[31535]: pgmap v1319: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T00:44:30.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:44:29 smithi189 ceph-mon[36187]: pgmap v1319: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T00:44:30.460 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:44:29 smithi193 ceph-mon[36878]: pgmap v1319: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T00:44:31.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:44:30 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 1166.631260 secs 2024-03-24T00:44:31.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:44:30 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 1166.631260 secs 2024-03-24T00:44:31.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:44:30 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 1166.631260 secs 2024-03-24T00:44:32.348 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:44:31 smithi118 ceph-mon[31535]: pgmap v1320: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T00:44:32.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:44:31 smithi189 ceph-mon[36187]: pgmap v1320: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T00:44:32.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:44:31 smithi193 ceph-mon[36878]: pgmap v1320: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T00:44:34.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:44:34 smithi118 ceph-mon[31535]: pgmap v1321: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 5 op/s 2024-03-24T00:44:34.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:44:34 smithi189 ceph-mon[36187]: pgmap v1321: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 5 op/s 2024-03-24T00:44:34.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:44:34 smithi193 ceph-mon[36878]: pgmap v1321: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 5 op/s 2024-03-24T00:44:36.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:44:36 smithi118 ceph-mon[31535]: pgmap v1322: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T00:44:36.447 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:44:36 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 1171.631322 secs 2024-03-24T00:44:36.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:44:36 smithi189 ceph-mon[36187]: pgmap v1322: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T00:44:36.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:44:36 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 1171.631322 secs 2024-03-24T00:44:36.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:44:36 smithi193 ceph-mon[36878]: pgmap v1322: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T00:44:36.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:44:36 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 1171.631322 secs 2024-03-24T00:44:38.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:44:38 smithi118 ceph-mon[31535]: pgmap v1323: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:44:38.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:44:38 smithi189 ceph-mon[36187]: pgmap v1323: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:44:38.460 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:44:38 smithi193 ceph-mon[36878]: pgmap v1323: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:44:40.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:44:40 smithi118 ceph-mon[31535]: pgmap v1324: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T00:44:40.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:44:40 smithi189 ceph-mon[36187]: pgmap v1324: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T00:44:40.460 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:44:40 smithi193 ceph-mon[36878]: pgmap v1324: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T00:44:41.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:44:41 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 1176.631383 secs 2024-03-24T00:44:41.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:44:41 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 1176.631383 secs 2024-03-24T00:44:41.460 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:44:41 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 1176.631383 secs 2024-03-24T00:44:42.362 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:44:42 smithi193 ceph-mon[36878]: pgmap v1325: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T00:44:42.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:44:42 smithi118 ceph-mon[31535]: pgmap v1325: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T00:44:42.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:44:42 smithi189 ceph-mon[36187]: pgmap v1325: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T00:44:44.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:44:44 smithi118 ceph-mon[31535]: pgmap v1326: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 5 op/s 2024-03-24T00:44:44.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:44:44 smithi189 ceph-mon[36187]: pgmap v1326: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 5 op/s 2024-03-24T00:44:44.460 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:44:44 smithi193 ceph-mon[36878]: pgmap v1326: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 5 op/s 2024-03-24T00:44:46.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:44:46 smithi118 ceph-mon[31535]: pgmap v1327: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T00:44:46.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:44:46 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T00:44:46.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:44:46 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 1181.631448 secs 2024-03-24T00:44:46.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:44:46 smithi189 ceph-mon[36187]: pgmap v1327: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T00:44:46.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:44:46 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T00:44:46.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:44:46 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 1181.631448 secs 2024-03-24T00:44:46.460 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:44:46 smithi193 ceph-mon[36878]: pgmap v1327: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T00:44:46.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:44:46 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T00:44:46.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:44:46 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 1181.631448 secs 2024-03-24T00:44:47.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:44:47 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T00:44:47.447 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:44:47 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T00:44:47.447 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:44:47 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T00:44:47.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:44:47 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T00:44:47.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:44:47 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T00:44:47.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:44:47 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T00:44:47.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:44:47 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T00:44:47.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:44:47 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T00:44:47.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:44:47 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T00:44:48.447 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:44:48 smithi118 ceph-mon[31535]: pgmap v1328: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 5 op/s 2024-03-24T00:44:48.447 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:44:48 smithi118 ceph-mon[31535]: pgmap v1329: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-03-24T00:44:48.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:44:48 smithi189 ceph-mon[36187]: pgmap v1328: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 5 op/s 2024-03-24T00:44:48.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:44:48 smithi189 ceph-mon[36187]: pgmap v1329: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-03-24T00:44:48.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:44:48 smithi193 ceph-mon[36878]: pgmap v1328: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 5 op/s 2024-03-24T00:44:48.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:44:48 smithi193 ceph-mon[36878]: pgmap v1329: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-03-24T00:44:50.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:44:50 smithi118 ceph-mon[31535]: pgmap v1330: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 5 op/s 2024-03-24T00:44:50.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:44:50 smithi189 ceph-mon[36187]: pgmap v1330: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 5 op/s 2024-03-24T00:44:50.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:44:50 smithi193 ceph-mon[36878]: pgmap v1330: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 5 op/s 2024-03-24T00:44:51.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:44:51 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 1186.631507 secs 2024-03-24T00:44:51.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:44:51 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 1186.631507 secs 2024-03-24T00:44:51.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:44:51 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 1186.631507 secs 2024-03-24T00:44:52.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:44:52 smithi118 ceph-mon[31535]: pgmap v1331: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-03-24T00:44:52.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:44:52 smithi189 ceph-mon[36187]: pgmap v1331: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-03-24T00:44:52.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:44:52 smithi193 ceph-mon[36878]: pgmap v1331: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-03-24T00:44:54.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:44:54 smithi118 ceph-mon[31535]: pgmap v1332: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 5 op/s 2024-03-24T00:44:54.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:44:54 smithi189 ceph-mon[36187]: pgmap v1332: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 5 op/s 2024-03-24T00:44:54.460 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:44:54 smithi193 ceph-mon[36878]: pgmap v1332: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 5 op/s 2024-03-24T00:44:56.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:44:56 smithi118 ceph-mon[31535]: pgmap v1333: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-03-24T00:44:56.447 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:44:56 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 1191.631595 secs 2024-03-24T00:44:56.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:44:56 smithi189 ceph-mon[36187]: pgmap v1333: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-03-24T00:44:56.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:44:56 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 1191.631595 secs 2024-03-24T00:44:56.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:44:56 smithi193 ceph-mon[36878]: pgmap v1333: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-03-24T00:44:56.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:44:56 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 1191.631595 secs 2024-03-24T00:44:58.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:44:58 smithi118 ceph-mon[31535]: pgmap v1334: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 5 op/s 2024-03-24T00:44:58.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:44:58 smithi189 ceph-mon[36187]: pgmap v1334: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 5 op/s 2024-03-24T00:44:58.460 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:44:58 smithi193 ceph-mon[36878]: pgmap v1334: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 5 op/s 2024-03-24T00:45:00.449 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:45:00 smithi118 ceph-mon[31535]: pgmap v1335: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T00:45:00.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:45:00 smithi189 ceph-mon[36187]: pgmap v1335: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T00:45:00.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:45:00 smithi193 ceph-mon[36878]: pgmap v1335: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T00:45:01.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:45:01 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 1196.631631 secs 2024-03-24T00:45:01.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:45:01 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 1196.631631 secs 2024-03-24T00:45:01.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:45:01 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 1196.631631 secs 2024-03-24T00:45:02.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:45:02 smithi118 ceph-mon[31535]: pgmap v1336: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T00:45:02.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:45:02 smithi189 ceph-mon[36187]: pgmap v1336: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T00:45:02.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:45:02 smithi193 ceph-mon[36878]: pgmap v1336: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T00:45:04.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:45:04 smithi118 ceph-mon[31535]: pgmap v1337: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T00:45:04.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:45:04 smithi189 ceph-mon[36187]: pgmap v1337: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T00:45:04.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:45:04 smithi193 ceph-mon[36878]: pgmap v1337: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T00:45:06.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:45:06 smithi118 ceph-mon[31535]: pgmap v1338: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-03-24T00:45:06.447 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:45:06 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 1201.631690 secs 2024-03-24T00:45:06.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:45:06 smithi189 ceph-mon[36187]: pgmap v1338: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-03-24T00:45:06.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:45:06 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 1201.631690 secs 2024-03-24T00:45:06.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:45:06 smithi193 ceph-mon[36878]: pgmap v1338: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-03-24T00:45:06.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:45:06 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 1201.631690 secs 2024-03-24T00:45:08.447 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:45:08 smithi118 ceph-mon[31535]: pgmap v1339: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:45:08.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:45:08 smithi189 ceph-mon[36187]: pgmap v1339: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:45:08.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:45:08 smithi193 ceph-mon[36878]: pgmap v1339: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:45:10.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:45:10 smithi118 ceph-mon[31535]: pgmap v1340: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T00:45:10.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:45:10 smithi189 ceph-mon[36187]: pgmap v1340: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T00:45:10.460 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:45:10 smithi193 ceph-mon[36878]: pgmap v1340: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T00:45:11.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:45:11 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 1206.631779 secs 2024-03-24T00:45:11.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:45:11 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 1206.631779 secs 2024-03-24T00:45:11.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:45:11 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 1206.631779 secs 2024-03-24T00:45:12.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:45:12 smithi118 ceph-mon[31535]: pgmap v1341: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T00:45:12.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:45:12 smithi189 ceph-mon[36187]: pgmap v1341: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T00:45:12.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:45:12 smithi193 ceph-mon[36878]: pgmap v1341: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T00:45:14.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:45:14 smithi118 ceph-mon[31535]: pgmap v1342: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T00:45:14.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:45:14 smithi189 ceph-mon[36187]: pgmap v1342: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T00:45:14.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:45:14 smithi193 ceph-mon[36878]: pgmap v1342: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T00:45:16.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:45:16 smithi118 ceph-mon[31535]: pgmap v1343: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-03-24T00:45:16.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:45:16 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 1211.631842 secs 2024-03-24T00:45:16.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:45:16 smithi189 ceph-mon[36187]: pgmap v1343: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-03-24T00:45:16.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:45:16 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 1211.631842 secs 2024-03-24T00:45:16.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:45:16 smithi193 ceph-mon[36878]: pgmap v1343: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-03-24T00:45:16.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:45:16 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 1211.631842 secs 2024-03-24T00:45:18.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:45:18 smithi118 ceph-mon[31535]: pgmap v1344: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:45:18.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:45:18 smithi189 ceph-mon[36187]: pgmap v1344: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:45:18.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:45:18 smithi193 ceph-mon[36878]: pgmap v1344: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:45:20.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:45:20 smithi118 ceph-mon[31535]: pgmap v1345: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T00:45:20.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:45:20 smithi189 ceph-mon[36187]: pgmap v1345: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T00:45:20.460 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:45:20 smithi193 ceph-mon[36878]: pgmap v1345: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T00:45:21.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:45:21 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 1216.631957 secs 2024-03-24T00:45:21.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:45:21 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 1216.631957 secs 2024-03-24T00:45:21.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:45:21 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 1216.631957 secs 2024-03-24T00:45:22.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:45:22 smithi118 ceph-mon[31535]: pgmap v1346: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T00:45:22.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:45:22 smithi189 ceph-mon[36187]: pgmap v1346: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T00:45:22.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:45:22 smithi193 ceph-mon[36878]: pgmap v1346: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T00:45:23.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:45:23 smithi118 ceph-mon[31535]: pgmap v1347: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T00:45:23.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:45:23 smithi189 ceph-mon[36187]: pgmap v1347: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T00:45:23.460 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:45:23 smithi193 ceph-mon[36878]: pgmap v1347: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T00:45:25.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:45:25 smithi118 ceph-mon[31535]: pgmap v1348: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-03-24T00:45:25.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:45:25 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 1221.632047 secs 2024-03-24T00:45:25.704 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:45:25 smithi189 ceph-mon[36187]: pgmap v1348: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-03-24T00:45:25.704 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:45:25 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 1221.632047 secs 2024-03-24T00:45:25.711 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:45:25 smithi193 ceph-mon[36878]: pgmap v1348: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-03-24T00:45:25.711 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:45:25 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 1221.632047 secs 2024-03-24T00:45:27.704 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:45:27 smithi189 ceph-mon[36187]: pgmap v1349: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-03-24T00:45:27.711 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:45:27 smithi193 ceph-mon[36878]: pgmap v1349: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-03-24T00:45:27.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:45:27 smithi118 ceph-mon[31535]: pgmap v1349: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-03-24T00:45:29.704 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:45:29 smithi189 ceph-mon[36187]: pgmap v1350: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-03-24T00:45:29.710 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:45:29 smithi193 ceph-mon[36878]: pgmap v1350: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-03-24T00:45:29.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:45:29 smithi118 ceph-mon[31535]: pgmap v1350: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-03-24T00:45:30.711 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:45:30 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 1226.632099 secs 2024-03-24T00:45:30.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:45:30 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 1226.632099 secs 2024-03-24T00:45:30.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:45:30 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 1226.632099 secs 2024-03-24T00:45:31.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:45:31 smithi118 ceph-mon[31535]: pgmap v1351: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 6 op/s 2024-03-24T00:45:31.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:45:31 smithi189 ceph-mon[36187]: pgmap v1351: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 6 op/s 2024-03-24T00:45:31.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:45:31 smithi193 ceph-mon[36878]: pgmap v1351: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 6 op/s 2024-03-24T00:45:33.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:45:33 smithi118 ceph-mon[31535]: pgmap v1352: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 5 op/s 2024-03-24T00:45:33.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:45:33 smithi189 ceph-mon[36187]: pgmap v1352: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 5 op/s 2024-03-24T00:45:33.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:45:33 smithi193 ceph-mon[36878]: pgmap v1352: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 5 op/s 2024-03-24T00:45:35.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:45:35 smithi118 ceph-mon[31535]: pgmap v1353: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-03-24T00:45:35.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:45:35 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 1231.632186 secs 2024-03-24T00:45:35.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:45:35 smithi189 ceph-mon[36187]: pgmap v1353: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-03-24T00:45:35.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:45:35 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 1231.632186 secs 2024-03-24T00:45:35.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:45:35 smithi193 ceph-mon[36878]: pgmap v1353: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-03-24T00:45:35.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:45:35 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 1231.632186 secs 2024-03-24T00:45:37.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:45:37 smithi118 ceph-mon[31535]: pgmap v1354: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:45:37.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:45:37 smithi189 ceph-mon[36187]: pgmap v1354: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:45:37.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:45:37 smithi193 ceph-mon[36878]: pgmap v1354: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:45:39.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:45:39 smithi118 ceph-mon[31535]: pgmap v1355: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T00:45:39.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:45:39 smithi189 ceph-mon[36187]: pgmap v1355: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T00:45:39.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:45:39 smithi193 ceph-mon[36878]: pgmap v1355: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T00:45:40.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:45:40 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 1236.632277 secs 2024-03-24T00:45:40.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:45:40 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 1236.632277 secs 2024-03-24T00:45:40.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:45:40 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 1236.632277 secs 2024-03-24T00:45:41.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:45:41 smithi118 ceph-mon[31535]: pgmap v1356: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T00:45:41.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:45:41 smithi189 ceph-mon[36187]: pgmap v1356: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T00:45:41.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:45:41 smithi193 ceph-mon[36878]: pgmap v1356: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T00:45:43.762 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:45:43 smithi193 ceph-mon[36878]: pgmap v1357: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 5 op/s 2024-03-24T00:45:43.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:45:43 smithi118 ceph-mon[31535]: pgmap v1357: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 5 op/s 2024-03-24T00:45:43.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:45:43 smithi189 ceph-mon[36187]: pgmap v1357: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 5 op/s 2024-03-24T00:45:45.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:45:45 smithi118 ceph-mon[31535]: pgmap v1358: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-03-24T00:45:45.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:45:45 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 1241.632353 secs 2024-03-24T00:45:45.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:45:45 smithi189 ceph-mon[36187]: pgmap v1358: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-03-24T00:45:45.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:45:45 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 1241.632353 secs 2024-03-24T00:45:45.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:45:45 smithi193 ceph-mon[36878]: pgmap v1358: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-03-24T00:45:45.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:45:45 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 1241.632353 secs 2024-03-24T00:45:47.756 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:45:47 smithi189 ceph-mon[36187]: pgmap v1359: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-03-24T00:45:47.756 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:45:47 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T00:45:47.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:45:47 smithi118 ceph-mon[31535]: pgmap v1359: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-03-24T00:45:47.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:45:47 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T00:45:47.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:45:47 smithi193 ceph-mon[36878]: pgmap v1359: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-03-24T00:45:47.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:45:47 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T00:45:48.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:45:48 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T00:45:48.947 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:45:48 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T00:45:48.947 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:45:48 smithi118 ceph-mon[31535]: pgmap v1360: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 6 op/s 2024-03-24T00:45:48.947 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:45:48 smithi118 ceph-mon[31535]: pgmap v1361: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 5 op/s 2024-03-24T00:45:48.947 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:45:48 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T00:45:48.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:45:48 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T00:45:48.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:45:48 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T00:45:48.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:45:48 smithi189 ceph-mon[36187]: pgmap v1360: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 6 op/s 2024-03-24T00:45:48.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:45:48 smithi189 ceph-mon[36187]: pgmap v1361: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 5 op/s 2024-03-24T00:45:48.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:45:48 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T00:45:48.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:45:48 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T00:45:48.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:45:48 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T00:45:48.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:45:48 smithi193 ceph-mon[36878]: pgmap v1360: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 6 op/s 2024-03-24T00:45:48.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:45:48 smithi193 ceph-mon[36878]: pgmap v1361: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 5 op/s 2024-03-24T00:45:48.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:45:48 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T00:45:51.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:45:50 smithi118 ceph-mon[31535]: pgmap v1362: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 6 op/s 2024-03-24T00:45:51.197 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:45:50 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 1246.632415 secs 2024-03-24T00:45:51.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:45:50 smithi189 ceph-mon[36187]: pgmap v1362: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 6 op/s 2024-03-24T00:45:51.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:45:50 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 1246.632415 secs 2024-03-24T00:45:51.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:45:50 smithi193 ceph-mon[36878]: pgmap v1362: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 6 op/s 2024-03-24T00:45:51.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:45:50 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 1246.632415 secs 2024-03-24T00:45:53.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:45:52 smithi118 ceph-mon[31535]: pgmap v1363: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 5 op/s 2024-03-24T00:45:53.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:45:52 smithi189 ceph-mon[36187]: pgmap v1363: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 5 op/s 2024-03-24T00:45:53.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:45:52 smithi193 ceph-mon[36878]: pgmap v1363: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 5 op/s 2024-03-24T00:45:55.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:45:54 smithi189 ceph-mon[36187]: pgmap v1364: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 5 op/s 2024-03-24T00:45:55.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:45:54 smithi193 ceph-mon[36878]: pgmap v1364: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 5 op/s 2024-03-24T00:45:55.348 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:45:54 smithi118 ceph-mon[31535]: pgmap v1364: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 5 op/s 2024-03-24T00:45:56.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:45:55 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 1251.632501 secs 2024-03-24T00:45:56.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:45:55 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 1251.632501 secs 2024-03-24T00:45:56.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:45:55 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 1251.632501 secs 2024-03-24T00:45:57.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:45:56 smithi118 ceph-mon[31535]: pgmap v1365: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-03-24T00:45:57.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:45:56 smithi189 ceph-mon[36187]: pgmap v1365: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-03-24T00:45:57.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:45:56 smithi193 ceph-mon[36878]: pgmap v1365: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-03-24T00:45:59.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:45:58 smithi118 ceph-mon[31535]: pgmap v1366: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 5 op/s 2024-03-24T00:45:59.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:45:58 smithi189 ceph-mon[36187]: pgmap v1366: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 5 op/s 2024-03-24T00:45:59.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:45:58 smithi193 ceph-mon[36878]: pgmap v1366: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 5 op/s 2024-03-24T00:46:01.376 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:46:00 smithi193 ceph-mon[36878]: pgmap v1367: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T00:46:01.377 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:46:00 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 1256.632588 secs 2024-03-24T00:46:01.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:46:00 smithi118 ceph-mon[31535]: pgmap v1367: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T00:46:01.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:46:00 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 1256.632588 secs 2024-03-24T00:46:01.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:46:00 smithi189 ceph-mon[36187]: pgmap v1367: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T00:46:01.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:46:00 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 1256.632588 secs 2024-03-24T00:46:03.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:46:02 smithi118 ceph-mon[31535]: pgmap v1368: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:46:03.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:46:02 smithi189 ceph-mon[36187]: pgmap v1368: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:46:03.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:46:02 smithi193 ceph-mon[36878]: pgmap v1368: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:46:05.336 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:46:04 smithi118 ceph-mon[31535]: pgmap v1369: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T00:46:05.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:46:04 smithi189 ceph-mon[36187]: pgmap v1369: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T00:46:05.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:46:04 smithi193 ceph-mon[36878]: pgmap v1369: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T00:46:06.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:46:05 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 1261.632602 secs 2024-03-24T00:46:06.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:46:05 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 1261.632602 secs 2024-03-24T00:46:06.460 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:46:05 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 1261.632602 secs 2024-03-24T00:46:07.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:46:06 smithi118 ceph-mon[31535]: pgmap v1370: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T00:46:07.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:46:06 smithi189 ceph-mon[36187]: pgmap v1370: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T00:46:07.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:46:06 smithi193 ceph-mon[36878]: pgmap v1370: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T00:46:09.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:46:08 smithi118 ceph-mon[31535]: pgmap v1371: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 5 op/s 2024-03-24T00:46:09.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:46:08 smithi189 ceph-mon[36187]: pgmap v1371: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 5 op/s 2024-03-24T00:46:09.462 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:46:08 smithi193 ceph-mon[36878]: pgmap v1371: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 5 op/s 2024-03-24T00:46:11.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:46:11 smithi118 ceph-mon[31535]: pgmap v1372: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T00:46:11.447 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:46:11 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 1266.632664 secs 2024-03-24T00:46:11.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:46:11 smithi189 ceph-mon[36187]: pgmap v1372: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T00:46:11.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:46:11 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 1266.632664 secs 2024-03-24T00:46:11.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:46:11 smithi193 ceph-mon[36878]: pgmap v1372: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T00:46:11.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:46:11 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 1266.632664 secs 2024-03-24T00:46:13.263 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:46:13 smithi193 ceph-mon[36878]: pgmap v1373: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:46:13.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:46:13 smithi118 ceph-mon[31535]: pgmap v1373: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:46:13.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:46:13 smithi189 ceph-mon[36187]: pgmap v1373: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:46:15.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:46:15 smithi118 ceph-mon[31535]: pgmap v1374: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T00:46:15.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:46:15 smithi189 ceph-mon[36187]: pgmap v1374: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T00:46:15.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:46:15 smithi193 ceph-mon[36878]: pgmap v1374: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T00:46:15.926 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:46:15.925+0000 7f81ee000640 1 -- 172.21.15.118:0/1693767697 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f81c80036c0 con 0x7f81f81052d0 2024-03-24T00:46:15.926 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:46:15.925+0000 7f81ef400640 1 -- 172.21.15.118:0/1693767697 <== mon.2 v2:172.21.15.189:3300/0 14 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (secure 0 0 0) 0x7f81c80036c0 con 0x7f81f81052d0 2024-03-24T00:46:15.943 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:46:15.942+0000 7f81db400640 1 -- 172.21.15.118:0/659686424 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f81c40064d0 con 0x5622651a13f0 2024-03-24T00:46:15.943 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:46:15.942+0000 7f81e4c00640 1 -- 172.21.15.118:0/659686424 <== mon.1 v2:172.21.15.193:3300/0 14 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 409+0+0 (secure 0 0 0) 0x7f81c40064d0 con 0x5622651a13f0 2024-03-24T00:46:16.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:46:16 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 1271.632753 secs 2024-03-24T00:46:16.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:46:16 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 1271.632753 secs 2024-03-24T00:46:16.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:46:16 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 1271.632753 secs 2024-03-24T00:46:17.355 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:46:17 smithi189 ceph-mon[36187]: pgmap v1375: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T00:46:17.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:46:17 smithi118 ceph-mon[31535]: pgmap v1375: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T00:46:17.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:46:17 smithi193 ceph-mon[36878]: pgmap v1375: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T00:46:19.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:46:19 smithi118 ceph-mon[31535]: pgmap v1376: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 5 op/s 2024-03-24T00:46:19.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:46:19 smithi189 ceph-mon[36187]: pgmap v1376: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 5 op/s 2024-03-24T00:46:19.462 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:46:19 smithi193 ceph-mon[36878]: pgmap v1376: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 5 op/s 2024-03-24T00:46:21.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:46:21 smithi118 ceph-mon[31535]: pgmap v1377: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T00:46:21.447 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:46:21 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 1276.632813 secs 2024-03-24T00:46:21.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:46:21 smithi189 ceph-mon[36187]: pgmap v1377: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T00:46:21.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:46:21 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 1276.632813 secs 2024-03-24T00:46:21.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:46:21 smithi193 ceph-mon[36878]: pgmap v1377: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T00:46:21.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:46:21 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 1276.632813 secs 2024-03-24T00:46:23.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:46:23 smithi118 ceph-mon[31535]: pgmap v1378: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:46:23.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:46:23 smithi189 ceph-mon[36187]: pgmap v1378: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:46:23.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:46:23 smithi193 ceph-mon[36878]: pgmap v1378: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:46:25.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:46:25 smithi118 ceph-mon[31535]: pgmap v1379: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T00:46:25.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:46:25 smithi189 ceph-mon[36187]: pgmap v1379: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T00:46:25.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:46:25 smithi193 ceph-mon[36878]: pgmap v1379: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T00:46:26.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:46:26 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 1281.632904 secs 2024-03-24T00:46:26.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:46:26 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 1281.632904 secs 2024-03-24T00:46:26.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:46:26 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 1281.632904 secs 2024-03-24T00:46:27.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:46:27 smithi118 ceph-mon[31535]: pgmap v1380: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T00:46:27.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:46:27 smithi189 ceph-mon[36187]: pgmap v1380: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T00:46:27.460 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:46:27 smithi193 ceph-mon[36878]: pgmap v1380: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T00:46:29.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:46:29 smithi118 ceph-mon[31535]: pgmap v1381: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 5 op/s 2024-03-24T00:46:29.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:46:29 smithi189 ceph-mon[36187]: pgmap v1381: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 5 op/s 2024-03-24T00:46:29.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:46:29 smithi193 ceph-mon[36878]: pgmap v1381: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 5 op/s 2024-03-24T00:46:31.447 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:46:31 smithi118 ceph-mon[31535]: pgmap v1382: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T00:46:31.447 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:46:31 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 1286.632995 secs 2024-03-24T00:46:31.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:46:31 smithi189 ceph-mon[36187]: pgmap v1382: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T00:46:31.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:46:31 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 1286.632995 secs 2024-03-24T00:46:31.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:46:31 smithi193 ceph-mon[36878]: pgmap v1382: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T00:46:31.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:46:31 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 1286.632995 secs 2024-03-24T00:46:33.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:46:33 smithi118 ceph-mon[31535]: pgmap v1383: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:46:33.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:46:33 smithi189 ceph-mon[36187]: pgmap v1383: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:46:33.460 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:46:33 smithi193 ceph-mon[36878]: pgmap v1383: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:46:35.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:46:35 smithi118 ceph-mon[31535]: pgmap v1384: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T00:46:35.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:46:35 smithi189 ceph-mon[36187]: pgmap v1384: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T00:46:35.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:46:35 smithi193 ceph-mon[36878]: pgmap v1384: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T00:46:36.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:46:36 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 1291.633087 secs 2024-03-24T00:46:36.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:46:36 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 1291.633087 secs 2024-03-24T00:46:36.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:46:36 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 1291.633087 secs 2024-03-24T00:46:37.348 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:46:37 smithi118 ceph-mon[31535]: pgmap v1385: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T00:46:37.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:46:37 smithi189 ceph-mon[36187]: pgmap v1385: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T00:46:37.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:46:37 smithi193 ceph-mon[36878]: pgmap v1385: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T00:46:39.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:46:39 smithi118 ceph-mon[31535]: pgmap v1386: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 5 op/s 2024-03-24T00:46:39.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:46:39 smithi189 ceph-mon[36187]: pgmap v1386: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 5 op/s 2024-03-24T00:46:39.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:46:39 smithi193 ceph-mon[36878]: pgmap v1386: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 5 op/s 2024-03-24T00:46:41.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:46:41 smithi118 ceph-mon[31535]: pgmap v1387: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T00:46:41.447 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:46:41 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 1296.633175 secs 2024-03-24T00:46:41.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:46:41 smithi189 ceph-mon[36187]: pgmap v1387: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T00:46:41.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:46:41 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 1296.633175 secs 2024-03-24T00:46:41.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:46:41 smithi193 ceph-mon[36878]: pgmap v1387: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T00:46:41.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:46:41 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 1296.633175 secs 2024-03-24T00:46:43.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:46:43 smithi118 ceph-mon[31535]: pgmap v1388: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:46:43.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:46:43 smithi189 ceph-mon[36187]: pgmap v1388: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:46:43.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:46:43 smithi193 ceph-mon[36878]: pgmap v1388: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:46:45.447 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:46:45 smithi118 ceph-mon[31535]: pgmap v1389: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T00:46:45.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:46:45 smithi189 ceph-mon[36187]: pgmap v1389: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T00:46:45.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:46:45 smithi193 ceph-mon[36878]: pgmap v1389: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T00:46:46.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:46:46 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 1301.633261 secs 2024-03-24T00:46:46.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:46:46 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 1301.633261 secs 2024-03-24T00:46:46.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:46:46 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 1301.633261 secs 2024-03-24T00:46:47.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:46:47 smithi118 ceph-mon[31535]: pgmap v1390: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T00:46:47.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:46:47 smithi189 ceph-mon[36187]: pgmap v1390: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T00:46:47.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:46:47 smithi193 ceph-mon[36878]: pgmap v1390: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T00:46:49.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:46:49 smithi118 ceph-mon[31535]: pgmap v1391: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 5 op/s 2024-03-24T00:46:49.447 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:46:49 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T00:46:49.447 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:46:49 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T00:46:49.447 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:46:49 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T00:46:49.447 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:46:49 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T00:46:49.447 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:46:49 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T00:46:49.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:46:49 smithi189 ceph-mon[36187]: pgmap v1391: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 5 op/s 2024-03-24T00:46:49.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:46:49 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T00:46:49.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:46:49 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T00:46:49.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:46:49 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T00:46:49.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:46:49 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T00:46:49.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:46:49 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T00:46:49.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:46:49 smithi193 ceph-mon[36878]: pgmap v1391: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 5 op/s 2024-03-24T00:46:49.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:46:49 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T00:46:49.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:46:49 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T00:46:49.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:46:49 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T00:46:49.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:46:49 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T00:46:49.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:46:49 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T00:46:51.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:46:51 smithi118 ceph-mon[31535]: pgmap v1392: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T00:46:51.447 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:46:51 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 1306.633281 secs 2024-03-24T00:46:51.447 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:46:51 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T00:46:51.447 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:46:51 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T00:46:51.447 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:46:51 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T00:46:51.447 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:46:51 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T00:46:51.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:46:51 smithi189 ceph-mon[36187]: pgmap v1392: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T00:46:51.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:46:51 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 1306.633281 secs 2024-03-24T00:46:51.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:46:51 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T00:46:51.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:46:51 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T00:46:51.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:46:51 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T00:46:51.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:46:51 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T00:46:51.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:46:51 smithi193 ceph-mon[36878]: pgmap v1392: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T00:46:51.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:46:51 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 1306.633281 secs 2024-03-24T00:46:51.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:46:51 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T00:46:51.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:46:51 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T00:46:51.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:46:51 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T00:46:51.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:46:51 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T00:46:53.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:46:53 smithi118 ceph-mon[31535]: pgmap v1393: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:46:53.447 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:46:53 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T00:46:53.447 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:46:53 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T00:46:53.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:46:53 smithi189 ceph-mon[36187]: pgmap v1393: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:46:53.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:46:53 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T00:46:53.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:46:53 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T00:46:53.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:46:53 smithi193 ceph-mon[36878]: pgmap v1393: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:46:53.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:46:53 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T00:46:53.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:46:53 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T00:46:55.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:46:55 smithi118 ceph-mon[31535]: pgmap v1394: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T00:46:55.447 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:46:55 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T00:46:55.447 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:46:55 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T00:46:55.447 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:46:55 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T00:46:55.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:46:55 smithi189 ceph-mon[36187]: pgmap v1394: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T00:46:55.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:46:55 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T00:46:55.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:46:55 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T00:46:55.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:46:55 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T00:46:55.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:46:55 smithi193 ceph-mon[36878]: pgmap v1394: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T00:46:55.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:46:55 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T00:46:55.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:46:55 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T00:46:55.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:46:55 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T00:46:56.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:46:56 smithi118 ceph-mon[31535]: pgmap v1395: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 6 op/s 2024-03-24T00:46:56.447 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:46:56 smithi118 ceph-mon[31535]: pgmap v1396: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-03-24T00:46:56.447 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:46:56 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 1311.633360 secs 2024-03-24T00:46:56.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:46:56 smithi189 ceph-mon[36187]: pgmap v1395: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 6 op/s 2024-03-24T00:46:56.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:46:56 smithi189 ceph-mon[36187]: pgmap v1396: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-03-24T00:46:56.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:46:56 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 1311.633360 secs 2024-03-24T00:46:56.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:46:56 smithi193 ceph-mon[36878]: pgmap v1395: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 6 op/s 2024-03-24T00:46:56.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:46:56 smithi193 ceph-mon[36878]: pgmap v1396: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-03-24T00:46:56.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:46:56 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 1311.633360 secs 2024-03-24T00:46:58.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:46:58 smithi118 ceph-mon[31535]: pgmap v1397: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 6 op/s 2024-03-24T00:46:58.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:46:58 smithi189 ceph-mon[36187]: pgmap v1397: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 6 op/s 2024-03-24T00:46:58.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:46:58 smithi193 ceph-mon[36878]: pgmap v1397: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 6 op/s 2024-03-24T00:47:00.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:47:00 smithi118 ceph-mon[31535]: pgmap v1398: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 5 op/s 2024-03-24T00:47:00.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:47:00 smithi189 ceph-mon[36187]: pgmap v1398: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 5 op/s 2024-03-24T00:47:00.460 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:47:00 smithi193 ceph-mon[36878]: pgmap v1398: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 5 op/s 2024-03-24T00:47:01.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:47:01 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 1316.633422 secs 2024-03-24T00:47:01.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:47:01 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 1316.633422 secs 2024-03-24T00:47:01.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:47:01 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 1316.633422 secs 2024-03-24T00:47:02.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:47:02 smithi118 ceph-mon[31535]: pgmap v1399: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-03-24T00:47:02.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:47:02 smithi189 ceph-mon[36187]: pgmap v1399: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-03-24T00:47:02.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:47:02 smithi193 ceph-mon[36878]: pgmap v1399: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-03-24T00:47:03.447 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:47:03 smithi118 ceph-mon[31535]: pgmap v1400: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-03-24T00:47:03.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:47:03 smithi189 ceph-mon[36187]: pgmap v1400: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-03-24T00:47:03.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:47:03 smithi193 ceph-mon[36878]: pgmap v1400: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-03-24T00:47:05.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:47:05 smithi118 ceph-mon[31535]: pgmap v1401: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 5 op/s 2024-03-24T00:47:05.704 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:47:05 smithi189 ceph-mon[36187]: pgmap v1401: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 5 op/s 2024-03-24T00:47:05.711 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:47:05 smithi193 ceph-mon[36878]: pgmap v1401: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 5 op/s 2024-03-24T00:47:06.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:47:06 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 1321.633511 secs 2024-03-24T00:47:06.704 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:47:06 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 1321.633511 secs 2024-03-24T00:47:06.710 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:47:06 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 1321.633511 secs 2024-03-24T00:47:07.598 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:47:07 smithi118 ceph-mon[31535]: pgmap v1402: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:47:07.704 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:47:07 smithi189 ceph-mon[36187]: pgmap v1402: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:47:07.711 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:47:07 smithi193 ceph-mon[36878]: pgmap v1402: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:47:09.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:47:09 smithi118 ceph-mon[31535]: pgmap v1403: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T00:47:09.704 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:47:09 smithi189 ceph-mon[36187]: pgmap v1403: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T00:47:09.711 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:47:09 smithi193 ceph-mon[36878]: pgmap v1403: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T00:47:10.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:47:10 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 1326.633596 secs 2024-03-24T00:47:10.704 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:47:10 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 1326.633596 secs 2024-03-24T00:47:10.711 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:47:10 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 1326.633596 secs 2024-03-24T00:47:11.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:47:11 smithi118 ceph-mon[31535]: pgmap v1404: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T00:47:11.704 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:47:11 smithi189 ceph-mon[36187]: pgmap v1404: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T00:47:11.711 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:47:11 smithi193 ceph-mon[36878]: pgmap v1404: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T00:47:13.704 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:47:13 smithi189 ceph-mon[36187]: pgmap v1405: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:47:13.711 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:47:13 smithi193 ceph-mon[36878]: pgmap v1405: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:47:13.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:47:13 smithi118 ceph-mon[31535]: pgmap v1405: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:47:15.704 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:47:15 smithi189 ceph-mon[36187]: pgmap v1406: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-24T00:47:15.704 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:47:15 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 1331.633685 secs 2024-03-24T00:47:15.711 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:47:15 smithi193 ceph-mon[36878]: pgmap v1406: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-24T00:47:15.711 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:47:15 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 1331.633685 secs 2024-03-24T00:47:15.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:47:15 smithi118 ceph-mon[31535]: pgmap v1406: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-24T00:47:15.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:47:15 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 1331.633685 secs 2024-03-24T00:47:17.711 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:47:17 smithi193 ceph-mon[36878]: pgmap v1407: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:47:17.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:47:17 smithi118 ceph-mon[31535]: pgmap v1407: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:47:17.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:47:17 smithi189 ceph-mon[36187]: pgmap v1407: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:47:19.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:47:19 smithi118 ceph-mon[31535]: pgmap v1408: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T00:47:19.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:47:19 smithi189 ceph-mon[36187]: pgmap v1408: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T00:47:19.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:47:19 smithi193 ceph-mon[36878]: pgmap v1408: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T00:47:20.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:47:20 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 1336.633755 secs 2024-03-24T00:47:20.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:47:20 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 1336.633755 secs 2024-03-24T00:47:20.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:47:20 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 1336.633755 secs 2024-03-24T00:47:21.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:47:21 smithi118 ceph-mon[31535]: pgmap v1409: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T00:47:21.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:47:21 smithi189 ceph-mon[36187]: pgmap v1409: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T00:47:21.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:47:21 smithi193 ceph-mon[36878]: pgmap v1409: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T00:47:23.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:47:23 smithi118 ceph-mon[31535]: pgmap v1410: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:47:23.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:47:23 smithi189 ceph-mon[36187]: pgmap v1410: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:47:23.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:47:23 smithi193 ceph-mon[36878]: pgmap v1410: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:47:25.749 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:47:25 smithi118 ceph-mon[31535]: pgmap v1411: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-24T00:47:25.750 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:47:25 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 1341.633845 secs 2024-03-24T00:47:25.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:47:25 smithi189 ceph-mon[36187]: pgmap v1411: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-24T00:47:25.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:47:25 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 1341.633845 secs 2024-03-24T00:47:25.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:47:25 smithi193 ceph-mon[36878]: pgmap v1411: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-24T00:47:25.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:47:25 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 1341.633845 secs 2024-03-24T00:47:27.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:47:27 smithi118 ceph-mon[31535]: pgmap v1412: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:47:27.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:47:27 smithi189 ceph-mon[36187]: pgmap v1412: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:47:27.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:47:27 smithi193 ceph-mon[36878]: pgmap v1412: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:47:29.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:47:29 smithi118 ceph-mon[31535]: pgmap v1413: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T00:47:29.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:47:29 smithi189 ceph-mon[36187]: pgmap v1413: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T00:47:29.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:47:29 smithi193 ceph-mon[36878]: pgmap v1413: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T00:47:30.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:47:30 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 1346.633934 secs 2024-03-24T00:47:30.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:47:30 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 1346.633934 secs 2024-03-24T00:47:30.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:47:30 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 1346.633934 secs 2024-03-24T00:47:31.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:47:31 smithi118 ceph-mon[31535]: pgmap v1414: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T00:47:31.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:47:31 smithi189 ceph-mon[36187]: pgmap v1414: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T00:47:31.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:47:31 smithi193 ceph-mon[36878]: pgmap v1414: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T00:47:33.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:47:33 smithi118 ceph-mon[31535]: pgmap v1415: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:47:33.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:47:33 smithi189 ceph-mon[36187]: pgmap v1415: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:47:33.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:47:33 smithi193 ceph-mon[36878]: pgmap v1415: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:47:35.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:47:35 smithi118 ceph-mon[31535]: pgmap v1416: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-24T00:47:35.947 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:47:35 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 1351.633994 secs 2024-03-24T00:47:35.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:47:35 smithi189 ceph-mon[36187]: pgmap v1416: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-24T00:47:35.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:47:35 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 1351.633994 secs 2024-03-24T00:47:35.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:47:35 smithi193 ceph-mon[36878]: pgmap v1416: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-24T00:47:35.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:47:35 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 1351.633994 secs 2024-03-24T00:47:37.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:47:37 smithi118 ceph-mon[31535]: pgmap v1417: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:47:37.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:47:37 smithi189 ceph-mon[36187]: pgmap v1417: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:47:37.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:47:37 smithi193 ceph-mon[36878]: pgmap v1417: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:47:39.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:47:39 smithi118 ceph-mon[31535]: pgmap v1418: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T00:47:39.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:47:39 smithi189 ceph-mon[36187]: pgmap v1418: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T00:47:39.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:47:39 smithi193 ceph-mon[36878]: pgmap v1418: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T00:47:40.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:47:40 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 1356.634080 secs 2024-03-24T00:47:40.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:47:40 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 1356.634080 secs 2024-03-24T00:47:40.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:47:40 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 1356.634080 secs 2024-03-24T00:47:41.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:47:41 smithi118 ceph-mon[31535]: pgmap v1419: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T00:47:41.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:47:41 smithi189 ceph-mon[36187]: pgmap v1419: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T00:47:41.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:47:41 smithi193 ceph-mon[36878]: pgmap v1419: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T00:47:43.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:47:43 smithi118 ceph-mon[31535]: pgmap v1420: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:47:43.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:47:43 smithi189 ceph-mon[36187]: pgmap v1420: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:47:43.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:47:43 smithi193 ceph-mon[36878]: pgmap v1420: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:47:45.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:47:45 smithi118 ceph-mon[31535]: pgmap v1421: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-24T00:47:45.947 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:47:45 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 1361.634086 secs 2024-03-24T00:47:45.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:47:45 smithi189 ceph-mon[36187]: pgmap v1421: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-24T00:47:45.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:47:45 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 1361.634086 secs 2024-03-24T00:47:45.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:47:45 smithi193 ceph-mon[36878]: pgmap v1421: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-24T00:47:45.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:47:45 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 1361.634086 secs 2024-03-24T00:47:47.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:47:47 smithi118 ceph-mon[31535]: pgmap v1422: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:47:47.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:47:47 smithi189 ceph-mon[36187]: pgmap v1422: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:47:47.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:47:47 smithi193 ceph-mon[36878]: pgmap v1422: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:47:49.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:47:49 smithi118 ceph-mon[31535]: pgmap v1423: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T00:47:49.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:47:49 smithi189 ceph-mon[36187]: pgmap v1423: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T00:47:49.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:47:49 smithi193 ceph-mon[36878]: pgmap v1423: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T00:47:50.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:47:50 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 1366.634149 secs 2024-03-24T00:47:50.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:47:50 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 1366.634149 secs 2024-03-24T00:47:50.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:47:50 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 1366.634149 secs 2024-03-24T00:47:51.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:47:51 smithi118 ceph-mon[31535]: pgmap v1424: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T00:47:51.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:47:51 smithi189 ceph-mon[36187]: pgmap v1424: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T00:47:51.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:47:51 smithi193 ceph-mon[36878]: pgmap v1424: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T00:47:53.881 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:47:53 smithi118 ceph-mon[31535]: pgmap v1425: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:47:53.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:47:53 smithi189 ceph-mon[36187]: pgmap v1425: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:47:53.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:47:53 smithi193 ceph-mon[36878]: pgmap v1425: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:47:54.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:47:54 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T00:47:54.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:47:54 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T00:47:54.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:47:54 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T00:47:55.848 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:47:55 smithi118 ceph-mon[31535]: pgmap v1426: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-24T00:47:55.848 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:47:55 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 1371.634213 secs 2024-03-24T00:47:55.849 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:47:55 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T00:47:55.849 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:47:55 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T00:47:55.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:47:55 smithi189 ceph-mon[36187]: pgmap v1426: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-24T00:47:55.978 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:47:55 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 1371.634213 secs 2024-03-24T00:47:55.979 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:47:55 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T00:47:55.979 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:47:55 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T00:47:55.979 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:47:55 smithi193 ceph-mon[36878]: pgmap v1426: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-24T00:47:55.979 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:47:55 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 1371.634213 secs 2024-03-24T00:47:55.979 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:47:55 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T00:47:55.979 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:47:55 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T00:47:56.947 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:47:56 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T00:47:56.947 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:47:56 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T00:47:56.947 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:47:56 smithi118 ceph-mon[31535]: pgmap v1427: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 5 op/s 2024-03-24T00:47:56.947 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:47:56 smithi118 ceph-mon[31535]: pgmap v1428: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 6 op/s 2024-03-24T00:47:56.947 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:47:56 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T00:47:56.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:47:56 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T00:47:56.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:47:56 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T00:47:56.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:47:56 smithi189 ceph-mon[36187]: pgmap v1427: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 5 op/s 2024-03-24T00:47:56.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:47:56 smithi189 ceph-mon[36187]: pgmap v1428: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 6 op/s 2024-03-24T00:47:56.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:47:56 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T00:47:56.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:47:56 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T00:47:56.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:47:56 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T00:47:56.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:47:56 smithi193 ceph-mon[36878]: pgmap v1427: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 5 op/s 2024-03-24T00:47:56.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:47:56 smithi193 ceph-mon[36878]: pgmap v1428: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 6 op/s 2024-03-24T00:47:56.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:47:56 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T00:47:59.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:47:59 smithi118 ceph-mon[31535]: pgmap v1429: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 5 op/s 2024-03-24T00:47:59.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:47:59 smithi189 ceph-mon[36187]: pgmap v1429: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 5 op/s 2024-03-24T00:47:59.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:47:59 smithi193 ceph-mon[36878]: pgmap v1429: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 5 op/s 2024-03-24T00:48:01.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:48:01 smithi118 ceph-mon[31535]: pgmap v1430: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 6 op/s 2024-03-24T00:48:01.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:48:01 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 1376.634253 secs 2024-03-24T00:48:01.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:48:01 smithi189 ceph-mon[36187]: pgmap v1430: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 6 op/s 2024-03-24T00:48:01.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:48:01 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 1376.634253 secs 2024-03-24T00:48:01.460 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:48:01 smithi193 ceph-mon[36878]: pgmap v1430: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 6 op/s 2024-03-24T00:48:01.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:48:01 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 1376.634253 secs 2024-03-24T00:48:03.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:48:03 smithi118 ceph-mon[31535]: pgmap v1431: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 6 op/s 2024-03-24T00:48:03.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:48:03 smithi189 ceph-mon[36187]: pgmap v1431: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 6 op/s 2024-03-24T00:48:03.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:48:03 smithi193 ceph-mon[36878]: pgmap v1431: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 6 op/s 2024-03-24T00:48:05.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:48:05 smithi118 ceph-mon[31535]: pgmap v1432: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 5 op/s 2024-03-24T00:48:05.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:48:05 smithi189 ceph-mon[36187]: pgmap v1432: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 5 op/s 2024-03-24T00:48:05.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:48:05 smithi193 ceph-mon[36878]: pgmap v1432: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 5 op/s 2024-03-24T00:48:06.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:48:06 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 1381.634296 secs 2024-03-24T00:48:06.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:48:06 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 1381.634296 secs 2024-03-24T00:48:06.460 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:48:06 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 1381.634296 secs 2024-03-24T00:48:07.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:48:07 smithi118 ceph-mon[31535]: pgmap v1433: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 5 op/s 2024-03-24T00:48:07.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:48:07 smithi189 ceph-mon[36187]: pgmap v1433: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 5 op/s 2024-03-24T00:48:07.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:48:07 smithi193 ceph-mon[36878]: pgmap v1433: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 5 op/s 2024-03-24T00:48:09.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:48:09 smithi118 ceph-mon[31535]: pgmap v1434: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 5 op/s 2024-03-24T00:48:09.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:48:09 smithi189 ceph-mon[36187]: pgmap v1434: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 5 op/s 2024-03-24T00:48:09.460 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:48:09 smithi193 ceph-mon[36878]: pgmap v1434: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 5 op/s 2024-03-24T00:48:11.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:48:11 smithi118 ceph-mon[31535]: pgmap v1435: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T00:48:11.447 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:48:11 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 1386.634381 secs 2024-03-24T00:48:11.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:48:11 smithi189 ceph-mon[36187]: pgmap v1435: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T00:48:11.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:48:11 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 1386.634381 secs 2024-03-24T00:48:11.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:48:11 smithi193 ceph-mon[36878]: pgmap v1435: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T00:48:11.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:48:11 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 1386.634381 secs 2024-03-24T00:48:13.336 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:48:13 smithi118 ceph-mon[31535]: pgmap v1436: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:48:13.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:48:13 smithi189 ceph-mon[36187]: pgmap v1436: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:48:13.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:48:13 smithi193 ceph-mon[36878]: pgmap v1436: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:48:15.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:48:15 smithi118 ceph-mon[31535]: pgmap v1437: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T00:48:15.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:48:15 smithi189 ceph-mon[36187]: pgmap v1437: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T00:48:15.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:48:15 smithi193 ceph-mon[36878]: pgmap v1437: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T00:48:16.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:48:16 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 1391.634471 secs 2024-03-24T00:48:16.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:48:16 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 1391.634471 secs 2024-03-24T00:48:16.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:48:16 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 1391.634471 secs 2024-03-24T00:48:17.355 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:48:17 smithi189 ceph-mon[36187]: pgmap v1438: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T00:48:17.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:48:17 smithi118 ceph-mon[31535]: pgmap v1438: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T00:48:17.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:48:17 smithi193 ceph-mon[36878]: pgmap v1438: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T00:48:19.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:48:19 smithi118 ceph-mon[31535]: pgmap v1439: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 5 op/s 2024-03-24T00:48:19.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:48:19 smithi189 ceph-mon[36187]: pgmap v1439: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 5 op/s 2024-03-24T00:48:19.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:48:19 smithi193 ceph-mon[36878]: pgmap v1439: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 5 op/s 2024-03-24T00:48:21.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:48:21 smithi118 ceph-mon[31535]: pgmap v1440: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T00:48:21.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:48:21 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 1396.634558 secs 2024-03-24T00:48:21.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:48:21 smithi189 ceph-mon[36187]: pgmap v1440: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T00:48:21.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:48:21 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 1396.634558 secs 2024-03-24T00:48:21.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:48:21 smithi193 ceph-mon[36878]: pgmap v1440: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T00:48:21.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:48:21 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 1396.634558 secs 2024-03-24T00:48:23.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:48:23 smithi118 ceph-mon[31535]: pgmap v1441: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:48:23.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:48:23 smithi189 ceph-mon[36187]: pgmap v1441: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:48:23.460 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:48:23 smithi193 ceph-mon[36878]: pgmap v1441: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:48:25.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:48:25 smithi118 ceph-mon[31535]: pgmap v1442: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T00:48:25.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:48:25 smithi189 ceph-mon[36187]: pgmap v1442: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T00:48:25.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:48:25 smithi193 ceph-mon[36878]: pgmap v1442: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T00:48:26.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:48:26 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 1401.634644 secs 2024-03-24T00:48:26.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:48:26 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 1401.634644 secs 2024-03-24T00:48:26.460 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:48:26 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 1401.634644 secs 2024-03-24T00:48:27.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:48:27 smithi118 ceph-mon[31535]: pgmap v1443: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T00:48:27.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:48:27 smithi189 ceph-mon[36187]: pgmap v1443: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T00:48:27.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:48:27 smithi193 ceph-mon[36878]: pgmap v1443: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T00:48:29.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:48:29 smithi118 ceph-mon[31535]: pgmap v1444: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 5 op/s 2024-03-24T00:48:29.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:48:29 smithi189 ceph-mon[36187]: pgmap v1444: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 5 op/s 2024-03-24T00:48:29.460 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:48:29 smithi193 ceph-mon[36878]: pgmap v1444: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 5 op/s 2024-03-24T00:48:31.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:48:31 smithi118 ceph-mon[31535]: pgmap v1445: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T00:48:31.447 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:48:31 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 1406.634708 secs 2024-03-24T00:48:31.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:48:31 smithi189 ceph-mon[36187]: pgmap v1445: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T00:48:31.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:48:31 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 1406.634708 secs 2024-03-24T00:48:31.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:48:31 smithi193 ceph-mon[36878]: pgmap v1445: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T00:48:31.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:48:31 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 1406.634708 secs 2024-03-24T00:48:33.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:48:33 smithi118 ceph-mon[31535]: pgmap v1446: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:48:33.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:48:33 smithi189 ceph-mon[36187]: pgmap v1446: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:48:33.460 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:48:33 smithi193 ceph-mon[36878]: pgmap v1446: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:48:35.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:48:35 smithi118 ceph-mon[31535]: pgmap v1447: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T00:48:35.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:48:35 smithi189 ceph-mon[36187]: pgmap v1447: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T00:48:35.460 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:48:35 smithi193 ceph-mon[36878]: pgmap v1447: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T00:48:36.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:48:36 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 1411.634794 secs 2024-03-24T00:48:36.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:48:36 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 1411.634794 secs 2024-03-24T00:48:36.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:48:36 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 1411.634794 secs 2024-03-24T00:48:37.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:48:37 smithi118 ceph-mon[31535]: pgmap v1448: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T00:48:37.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:48:37 smithi189 ceph-mon[36187]: pgmap v1448: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T00:48:37.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:48:37 smithi193 ceph-mon[36878]: pgmap v1448: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T00:48:39.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:48:39 smithi118 ceph-mon[31535]: pgmap v1449: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 5 op/s 2024-03-24T00:48:39.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:48:39 smithi189 ceph-mon[36187]: pgmap v1449: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 5 op/s 2024-03-24T00:48:39.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:48:39 smithi193 ceph-mon[36878]: pgmap v1449: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 5 op/s 2024-03-24T00:48:41.249 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:48:41 smithi118 ceph-mon[31535]: pgmap v1450: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T00:48:41.249 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:48:41 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 1416.634878 secs 2024-03-24T00:48:41.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:48:41 smithi189 ceph-mon[36187]: pgmap v1450: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T00:48:41.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:48:41 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 1416.634878 secs 2024-03-24T00:48:41.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:48:41 smithi193 ceph-mon[36878]: pgmap v1450: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T00:48:41.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:48:41 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 1416.634878 secs 2024-03-24T00:48:43.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:48:43 smithi118 ceph-mon[31535]: pgmap v1451: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:48:43.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:48:43 smithi189 ceph-mon[36187]: pgmap v1451: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:48:43.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:48:43 smithi193 ceph-mon[36878]: pgmap v1451: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:48:45.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:48:45 smithi118 ceph-mon[31535]: pgmap v1452: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T00:48:45.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:48:45 smithi189 ceph-mon[36187]: pgmap v1452: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T00:48:45.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:48:45 smithi193 ceph-mon[36878]: pgmap v1452: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T00:48:46.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:48:46 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 1421.634938 secs 2024-03-24T00:48:46.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:48:46 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 1421.634938 secs 2024-03-24T00:48:46.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:48:46 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 1421.634938 secs 2024-03-24T00:48:47.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:48:47 smithi118 ceph-mon[31535]: pgmap v1453: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T00:48:47.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:48:47 smithi189 ceph-mon[36187]: pgmap v1453: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T00:48:47.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:48:47 smithi193 ceph-mon[36878]: pgmap v1453: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T00:48:49.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:48:49 smithi118 ceph-mon[31535]: pgmap v1454: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 5 op/s 2024-03-24T00:48:49.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:48:49 smithi189 ceph-mon[36187]: pgmap v1454: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 5 op/s 2024-03-24T00:48:49.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:48:49 smithi193 ceph-mon[36878]: pgmap v1454: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 5 op/s 2024-03-24T00:48:51.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:48:51 smithi118 ceph-mon[31535]: pgmap v1455: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T00:48:51.447 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:48:51 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 1426.635029 secs 2024-03-24T00:48:51.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:48:51 smithi189 ceph-mon[36187]: pgmap v1455: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T00:48:51.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:48:51 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 1426.635029 secs 2024-03-24T00:48:51.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:48:51 smithi193 ceph-mon[36878]: pgmap v1455: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T00:48:51.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:48:51 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 1426.635029 secs 2024-03-24T00:48:53.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:48:53 smithi118 ceph-mon[31535]: pgmap v1456: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:48:53.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:48:53 smithi189 ceph-mon[36187]: pgmap v1456: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:48:53.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:48:53 smithi193 ceph-mon[36878]: pgmap v1456: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:48:55.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:48:55 smithi118 ceph-mon[31535]: pgmap v1457: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T00:48:55.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:48:55 smithi189 ceph-mon[36187]: pgmap v1457: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T00:48:55.460 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:48:55 smithi193 ceph-mon[36878]: pgmap v1457: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T00:48:56.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:48:56 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 1431.635095 secs 2024-03-24T00:48:56.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:48:56 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 1431.635095 secs 2024-03-24T00:48:56.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:48:56 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 1431.635095 secs 2024-03-24T00:48:57.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:48:57 smithi118 ceph-mon[31535]: pgmap v1458: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T00:48:57.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:48:57 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T00:48:57.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:48:57 smithi189 ceph-mon[36187]: pgmap v1458: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T00:48:57.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:48:57 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T00:48:57.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:48:57 smithi193 ceph-mon[36878]: pgmap v1458: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T00:48:57.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:48:57 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T00:48:58.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:48:58 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T00:48:58.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:48:58 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T00:48:58.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:48:58 smithi189 ceph-mon[36187]: pgmap v1459: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 5 op/s 2024-03-24T00:48:58.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:48:58 smithi189 ceph-mon[36187]: pgmap v1460: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 5 op/s 2024-03-24T00:48:58.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:48:58 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T00:48:58.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:48:58 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T00:48:58.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:48:58 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T00:48:58.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:48:58 smithi193 ceph-mon[36878]: pgmap v1459: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 5 op/s 2024-03-24T00:48:58.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:48:58 smithi193 ceph-mon[36878]: pgmap v1460: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 5 op/s 2024-03-24T00:48:58.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:48:58 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T00:48:58.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:48:58 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T00:48:58.697 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:48:58 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T00:48:58.697 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:48:58 smithi118 ceph-mon[31535]: pgmap v1459: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 5 op/s 2024-03-24T00:48:58.697 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:48:58 smithi118 ceph-mon[31535]: pgmap v1460: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 5 op/s 2024-03-24T00:48:58.697 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:48:58 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T00:49:00.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:49:00 smithi118 ceph-mon[31535]: pgmap v1461: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-03-24T00:49:00.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:49:00 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 1436.635183 secs 2024-03-24T00:49:00.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:49:00 smithi189 ceph-mon[36187]: pgmap v1461: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-03-24T00:49:00.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:49:00 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 1436.635183 secs 2024-03-24T00:49:00.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:49:00 smithi193 ceph-mon[36878]: pgmap v1461: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-03-24T00:49:00.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:49:00 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 1436.635183 secs 2024-03-24T00:49:02.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:49:02 smithi118 ceph-mon[31535]: pgmap v1462: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-03-24T00:49:02.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:49:02 smithi189 ceph-mon[36187]: pgmap v1462: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-03-24T00:49:02.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:49:02 smithi193 ceph-mon[36878]: pgmap v1462: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-03-24T00:49:04.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:49:04 smithi118 ceph-mon[31535]: pgmap v1463: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 5 op/s 2024-03-24T00:49:04.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:49:04 smithi189 ceph-mon[36187]: pgmap v1463: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 5 op/s 2024-03-24T00:49:04.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:49:04 smithi193 ceph-mon[36878]: pgmap v1463: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 5 op/s 2024-03-24T00:49:05.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:49:05 smithi118 ceph-mon[31535]: 3 slow requests, 3 included below; oldest blocked for > 1441.635242 secs 2024-03-24T00:49:05.947 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:49:05 smithi118 ceph-mon[31535]: slow request 1441.635241 seconds old, received at 2024-03-24T00:25:03.572253+0000: peer_request:client.15159:154789 currently acquired locks 2024-03-24T00:49:05.947 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:49:05 smithi118 ceph-mon[31535]: slow request 1441.583995 seconds old, received at 2024-03-24T00:25:03.623500+0000: peer_request:client.15159:158090 currently acquired locks 2024-03-24T00:49:05.947 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:49:05 smithi118 ceph-mon[31535]: slow request 1441.577316 seconds old, received at 2024-03-24T00:25:03.630178+0000: client_request(client.15159:173196 rename #0x100000010bf/comment-15.xml #0x100000010bf/comment-15.xml.tmp 2024-03-24T00:25:03.629366+0000 caller_uid=1000, caller_gid=1301{6,36,1000,1301,}) currently failed to xlock, waiting 2024-03-24T00:49:05.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:49:05 smithi189 ceph-mon[36187]: 3 slow requests, 3 included below; oldest blocked for > 1441.635242 secs 2024-03-24T00:49:05.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:49:05 smithi189 ceph-mon[36187]: slow request 1441.635241 seconds old, received at 2024-03-24T00:25:03.572253+0000: peer_request:client.15159:154789 currently acquired locks 2024-03-24T00:49:05.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:49:05 smithi189 ceph-mon[36187]: slow request 1441.583995 seconds old, received at 2024-03-24T00:25:03.623500+0000: peer_request:client.15159:158090 currently acquired locks 2024-03-24T00:49:05.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:49:05 smithi189 ceph-mon[36187]: slow request 1441.577316 seconds old, received at 2024-03-24T00:25:03.630178+0000: client_request(client.15159:173196 rename #0x100000010bf/comment-15.xml #0x100000010bf/comment-15.xml.tmp 2024-03-24T00:25:03.629366+0000 caller_uid=1000, caller_gid=1301{6,36,1000,1301,}) currently failed to xlock, waiting 2024-03-24T00:49:05.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:49:05 smithi193 ceph-mon[36878]: 3 slow requests, 3 included below; oldest blocked for > 1441.635242 secs 2024-03-24T00:49:05.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:49:05 smithi193 ceph-mon[36878]: slow request 1441.635241 seconds old, received at 2024-03-24T00:25:03.572253+0000: peer_request:client.15159:154789 currently acquired locks 2024-03-24T00:49:05.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:49:05 smithi193 ceph-mon[36878]: slow request 1441.583995 seconds old, received at 2024-03-24T00:25:03.623500+0000: peer_request:client.15159:158090 currently acquired locks 2024-03-24T00:49:05.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:49:05 smithi193 ceph-mon[36878]: slow request 1441.577316 seconds old, received at 2024-03-24T00:25:03.630178+0000: client_request(client.15159:173196 rename #0x100000010bf/comment-15.xml #0x100000010bf/comment-15.xml.tmp 2024-03-24T00:25:03.629366+0000 caller_uid=1000, caller_gid=1301{6,36,1000,1301,}) currently failed to xlock, waiting 2024-03-24T00:49:06.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:49:06 smithi118 ceph-mon[31535]: pgmap v1464: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-03-24T00:49:06.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:49:06 smithi189 ceph-mon[36187]: pgmap v1464: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-03-24T00:49:06.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:49:06 smithi193 ceph-mon[36878]: pgmap v1464: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-03-24T00:49:08.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:49:08 smithi118 ceph-mon[31535]: pgmap v1465: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 5 op/s 2024-03-24T00:49:08.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:49:08 smithi189 ceph-mon[36187]: pgmap v1465: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 5 op/s 2024-03-24T00:49:08.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:49:08 smithi193 ceph-mon[36878]: pgmap v1465: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 5 op/s 2024-03-24T00:49:10.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:49:10 smithi118 ceph-mon[31535]: pgmap v1466: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-03-24T00:49:10.947 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:49:10 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 1446.635305 secs 2024-03-24T00:49:10.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:49:10 smithi189 ceph-mon[36187]: pgmap v1466: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-03-24T00:49:10.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:49:10 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 1446.635305 secs 2024-03-24T00:49:10.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:49:10 smithi193 ceph-mon[36878]: pgmap v1466: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-03-24T00:49:10.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:49:10 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 1446.635305 secs 2024-03-24T00:49:12.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:49:12 smithi118 ceph-mon[31535]: pgmap v1467: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:49:12.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:49:12 smithi189 ceph-mon[36187]: pgmap v1467: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:49:12.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:49:12 smithi193 ceph-mon[36878]: pgmap v1467: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:49:14.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:49:14 smithi118 ceph-mon[31535]: pgmap v1468: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T00:49:14.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:49:14 smithi189 ceph-mon[36187]: pgmap v1468: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T00:49:14.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:49:14 smithi193 ceph-mon[36878]: pgmap v1468: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T00:49:15.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:49:15 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 1451.635363 secs 2024-03-24T00:49:15.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:49:15 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 1451.635363 secs 2024-03-24T00:49:15.958 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:49:15.958+0000 7f81ee000640 1 -- 172.21.15.118:0/1693767697 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f81c80036c0 con 0x7f81f81052d0 2024-03-24T00:49:15.958 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:49:15.958+0000 7f81ef400640 1 -- 172.21.15.118:0/1693767697 <== mon.2 v2:172.21.15.189:3300/0 15 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (secure 0 0 0) 0x7f81c80036c0 con 0x7f81f81052d0 2024-03-24T00:49:15.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:49:15 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 1451.635363 secs 2024-03-24T00:49:15.970 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:49:15.970+0000 7f81db400640 1 -- 172.21.15.118:0/659686424 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f81c40064d0 con 0x5622651a13f0 2024-03-24T00:49:15.971 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:49:15.971+0000 7f81e4c00640 1 -- 172.21.15.118:0/659686424 <== mon.1 v2:172.21.15.193:3300/0 15 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 409+0+0 (secure 0 0 0) 0x7f81c40064d0 con 0x5622651a13f0 2024-03-24T00:49:16.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:49:16 smithi118 ceph-mon[31535]: pgmap v1469: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T00:49:16.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:49:16 smithi189 ceph-mon[36187]: pgmap v1469: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T00:49:16.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:49:16 smithi193 ceph-mon[36878]: pgmap v1469: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T00:49:18.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:49:18 smithi118 ceph-mon[31535]: pgmap v1470: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T00:49:18.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:49:18 smithi189 ceph-mon[36187]: pgmap v1470: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T00:49:18.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:49:18 smithi193 ceph-mon[36878]: pgmap v1470: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T00:49:20.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:49:20 smithi118 ceph-mon[31535]: pgmap v1471: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-03-24T00:49:20.947 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:49:20 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 1456.635404 secs 2024-03-24T00:49:20.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:49:20 smithi189 ceph-mon[36187]: pgmap v1471: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-03-24T00:49:20.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:49:20 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 1456.635404 secs 2024-03-24T00:49:20.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:49:20 smithi193 ceph-mon[36878]: pgmap v1471: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-03-24T00:49:20.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:49:20 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 1456.635404 secs 2024-03-24T00:49:22.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:49:22 smithi118 ceph-mon[31535]: pgmap v1472: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:49:22.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:49:22 smithi189 ceph-mon[36187]: pgmap v1472: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:49:22.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:49:22 smithi193 ceph-mon[36878]: pgmap v1472: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:49:24.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:49:24 smithi118 ceph-mon[31535]: pgmap v1473: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T00:49:24.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:49:24 smithi189 ceph-mon[36187]: pgmap v1473: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T00:49:24.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:49:24 smithi193 ceph-mon[36878]: pgmap v1473: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T00:49:25.848 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:49:25 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 1461.635465 secs 2024-03-24T00:49:25.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:49:25 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 1461.635465 secs 2024-03-24T00:49:25.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:49:25 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 1461.635465 secs 2024-03-24T00:49:26.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:49:26 smithi118 ceph-mon[31535]: pgmap v1474: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T00:49:26.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:49:26 smithi189 ceph-mon[36187]: pgmap v1474: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T00:49:26.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:49:26 smithi193 ceph-mon[36878]: pgmap v1474: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T00:49:28.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:49:28 smithi118 ceph-mon[31535]: pgmap v1475: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T00:49:28.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:49:28 smithi189 ceph-mon[36187]: pgmap v1475: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T00:49:28.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:49:28 smithi193 ceph-mon[36878]: pgmap v1475: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T00:49:30.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:49:30 smithi118 ceph-mon[31535]: pgmap v1476: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-03-24T00:49:30.947 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:49:30 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 1466.635524 secs 2024-03-24T00:49:30.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:49:30 smithi189 ceph-mon[36187]: pgmap v1476: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-03-24T00:49:30.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:49:30 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 1466.635524 secs 2024-03-24T00:49:30.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:49:30 smithi193 ceph-mon[36878]: pgmap v1476: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-03-24T00:49:30.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:49:30 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 1466.635524 secs 2024-03-24T00:49:32.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:49:32 smithi118 ceph-mon[31535]: pgmap v1477: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:49:32.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:49:32 smithi189 ceph-mon[36187]: pgmap v1477: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:49:32.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:49:32 smithi193 ceph-mon[36878]: pgmap v1477: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:49:34.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:49:34 smithi118 ceph-mon[31535]: pgmap v1478: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T00:49:34.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:49:34 smithi189 ceph-mon[36187]: pgmap v1478: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T00:49:34.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:49:34 smithi193 ceph-mon[36878]: pgmap v1478: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T00:49:35.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:49:35 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 1471.635612 secs 2024-03-24T00:49:35.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:49:35 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 1471.635612 secs 2024-03-24T00:49:35.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:49:35 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 1471.635612 secs 2024-03-24T00:49:36.947 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:49:36 smithi118 ceph-mon[31535]: pgmap v1479: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T00:49:36.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:49:36 smithi189 ceph-mon[36187]: pgmap v1479: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T00:49:36.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:49:36 smithi193 ceph-mon[36878]: pgmap v1479: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T00:49:38.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:49:38 smithi118 ceph-mon[31535]: pgmap v1480: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T00:49:38.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:49:38 smithi189 ceph-mon[36187]: pgmap v1480: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T00:49:38.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:49:38 smithi193 ceph-mon[36878]: pgmap v1480: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T00:49:40.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:49:40 smithi118 ceph-mon[31535]: pgmap v1481: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-03-24T00:49:40.947 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:49:40 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 1476.635701 secs 2024-03-24T00:49:40.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:49:40 smithi189 ceph-mon[36187]: pgmap v1481: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-03-24T00:49:40.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:49:40 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 1476.635701 secs 2024-03-24T00:49:40.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:49:40 smithi193 ceph-mon[36878]: pgmap v1481: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-03-24T00:49:40.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:49:40 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 1476.635701 secs 2024-03-24T00:49:42.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:49:42 smithi118 ceph-mon[31535]: pgmap v1482: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:49:42.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:49:42 smithi189 ceph-mon[36187]: pgmap v1482: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:49:42.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:49:42 smithi193 ceph-mon[36878]: pgmap v1482: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:49:44.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:49:44 smithi118 ceph-mon[31535]: pgmap v1483: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T00:49:44.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:49:44 smithi189 ceph-mon[36187]: pgmap v1483: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T00:49:44.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:49:44 smithi193 ceph-mon[36878]: pgmap v1483: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T00:49:45.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:49:45 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 1481.635789 secs 2024-03-24T00:49:45.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:49:45 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 1481.635789 secs 2024-03-24T00:49:45.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:49:45 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 1481.635789 secs 2024-03-24T00:49:46.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:49:46 smithi118 ceph-mon[31535]: pgmap v1484: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T00:49:46.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:49:46 smithi189 ceph-mon[36187]: pgmap v1484: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T00:49:46.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:49:46 smithi193 ceph-mon[36878]: pgmap v1484: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T00:49:48.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:49:48 smithi118 ceph-mon[31535]: pgmap v1485: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T00:49:48.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:49:48 smithi189 ceph-mon[36187]: pgmap v1485: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T00:49:48.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:49:48 smithi193 ceph-mon[36878]: pgmap v1485: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T00:49:50.947 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:49:50 smithi118 ceph-mon[31535]: pgmap v1486: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-03-24T00:49:50.947 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:49:50 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 1486.635803 secs 2024-03-24T00:49:50.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:49:50 smithi189 ceph-mon[36187]: pgmap v1486: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-03-24T00:49:50.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:49:50 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 1486.635803 secs 2024-03-24T00:49:50.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:49:50 smithi193 ceph-mon[36878]: pgmap v1486: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-03-24T00:49:50.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:49:50 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 1486.635803 secs 2024-03-24T00:49:52.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:49:52 smithi118 ceph-mon[31535]: pgmap v1487: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:49:52.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:49:52 smithi189 ceph-mon[36187]: pgmap v1487: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:49:52.962 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:49:52 smithi193 ceph-mon[36878]: pgmap v1487: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:49:54.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:49:54 smithi118 ceph-mon[31535]: pgmap v1488: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T00:49:54.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:49:54 smithi189 ceph-mon[36187]: pgmap v1488: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T00:49:54.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:49:54 smithi193 ceph-mon[36878]: pgmap v1488: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T00:49:55.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:49:55 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 1491.635867 secs 2024-03-24T00:49:55.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:49:55 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 1491.635867 secs 2024-03-24T00:49:55.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:49:55 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 1491.635867 secs 2024-03-24T00:49:56.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:49:56 smithi118 ceph-mon[31535]: pgmap v1489: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T00:49:56.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:49:56 smithi189 ceph-mon[36187]: pgmap v1489: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T00:49:56.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:49:56 smithi193 ceph-mon[36878]: pgmap v1489: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T00:49:58.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:49:58 smithi118 ceph-mon[31535]: pgmap v1490: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T00:49:58.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:49:58 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T00:49:58.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:49:58 smithi189 ceph-mon[36187]: pgmap v1490: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T00:49:58.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:49:58 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T00:49:58.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:49:58 smithi193 ceph-mon[36878]: pgmap v1490: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T00:49:58.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:49:58 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T00:49:59.947 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:49:59 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T00:49:59.947 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:49:59 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T00:49:59.947 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:49:59 smithi118 ceph-mon[31535]: pgmap v1491: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 6 op/s 2024-03-24T00:49:59.947 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:49:59 smithi118 ceph-mon[31535]: pgmap v1492: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 4 op/s 2024-03-24T00:49:59.947 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:49:59 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T00:49:59.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:49:59 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T00:49:59.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:49:59 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T00:49:59.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:49:59 smithi189 ceph-mon[36187]: pgmap v1491: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 6 op/s 2024-03-24T00:49:59.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:49:59 smithi189 ceph-mon[36187]: pgmap v1492: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 4 op/s 2024-03-24T00:49:59.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:49:59 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T00:49:59.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:49:59 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T00:49:59.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:49:59 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T00:49:59.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:49:59 smithi193 ceph-mon[36878]: pgmap v1491: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 6 op/s 2024-03-24T00:49:59.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:49:59 smithi193 ceph-mon[36878]: pgmap v1492: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 4 op/s 2024-03-24T00:49:59.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:49:59 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T00:50:00.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:50:00 smithi118 ceph-mon[31535]: overall HEALTH_WARN 6 failed cephadm daemon(s); 1 MDSs report slow requests 2024-03-24T00:50:00.947 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:50:00 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 1496.635955 secs 2024-03-24T00:50:00.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:50:00 smithi189 ceph-mon[36187]: overall HEALTH_WARN 6 failed cephadm daemon(s); 1 MDSs report slow requests 2024-03-24T00:50:00.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:50:00 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 1496.635955 secs 2024-03-24T00:50:00.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:50:00 smithi193 ceph-mon[36878]: overall HEALTH_WARN 6 failed cephadm daemon(s); 1 MDSs report slow requests 2024-03-24T00:50:00.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:50:00 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 1496.635955 secs 2024-03-24T00:50:00.965 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:50:00.965+0000 7f81ee000640 1 -- 172.21.15.118:0/1693767697 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- auth(proto 2 132 bytes epoch 0) v1 -- 0x7f81c80036c0 con 0x7f81f81052d0 2024-03-24T00:50:00.966 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:50:00.966+0000 7f81ef400640 1 -- 172.21.15.118:0/1693767697 <== mon.2 v2:172.21.15.189:3300/0 16 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 233+0+0 (secure 0 0 0) 0x7f81c80036c0 con 0x7f81f81052d0 2024-03-24T00:50:00.977 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:50:00.977+0000 7f81db400640 1 -- 172.21.15.118:0/659686424 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- auth(proto 2 132 bytes epoch 0) v1 -- 0x7f81c40064d0 con 0x5622651a13f0 2024-03-24T00:50:00.978 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:50:00.978+0000 7f81e4c00640 1 -- 172.21.15.118:0/659686424 <== mon.1 v2:172.21.15.193:3300/0 16 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 233+0+0 (secure 0 0 0) 0x7f81c40064d0 con 0x5622651a13f0 2024-03-24T00:50:01.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:50:01 smithi118 ceph-mon[31535]: pgmap v1493: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-03-24T00:50:01.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:50:01 smithi189 ceph-mon[36187]: pgmap v1493: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-03-24T00:50:01.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:50:01 smithi193 ceph-mon[36878]: pgmap v1493: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-03-24T00:50:04.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:50:03 smithi118 ceph-mon[31535]: pgmap v1494: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 5 op/s 2024-03-24T00:50:04.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:50:03 smithi189 ceph-mon[36187]: pgmap v1494: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 5 op/s 2024-03-24T00:50:04.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:50:03 smithi193 ceph-mon[36878]: pgmap v1494: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 5 op/s 2024-03-24T00:50:06.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:50:05 smithi118 ceph-mon[31535]: pgmap v1495: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-03-24T00:50:06.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:50:05 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 1501.636046 secs 2024-03-24T00:50:06.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:50:05 smithi189 ceph-mon[36187]: pgmap v1495: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-03-24T00:50:06.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:50:05 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 1501.636046 secs 2024-03-24T00:50:06.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:50:05 smithi193 ceph-mon[36878]: pgmap v1495: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-03-24T00:50:06.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:50:05 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 1501.636046 secs 2024-03-24T00:50:08.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:50:07 smithi118 ceph-mon[31535]: pgmap v1496: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 5 op/s 2024-03-24T00:50:08.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:50:07 smithi189 ceph-mon[36187]: pgmap v1496: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 5 op/s 2024-03-24T00:50:08.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:50:07 smithi193 ceph-mon[36878]: pgmap v1496: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 5 op/s 2024-03-24T00:50:10.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:50:09 smithi118 ceph-mon[31535]: pgmap v1497: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 5 op/s 2024-03-24T00:50:10.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:50:09 smithi189 ceph-mon[36187]: pgmap v1497: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 5 op/s 2024-03-24T00:50:10.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:50:09 smithi193 ceph-mon[36878]: pgmap v1497: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 5 op/s 2024-03-24T00:50:11.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:50:10 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 1506.636086 secs 2024-03-24T00:50:11.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:50:10 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 1506.636086 secs 2024-03-24T00:50:11.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:50:10 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 1506.636086 secs 2024-03-24T00:50:12.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:50:11 smithi193 ceph-mon[36878]: pgmap v1498: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T00:50:12.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:50:11 smithi118 ceph-mon[31535]: pgmap v1498: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T00:50:12.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:50:11 smithi189 ceph-mon[36187]: pgmap v1498: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T00:50:14.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:50:13 smithi193 ceph-mon[36878]: pgmap v1499: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 5 op/s 2024-03-24T00:50:14.447 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:50:13 smithi118 ceph-mon[31535]: pgmap v1499: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 5 op/s 2024-03-24T00:50:14.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:50:13 smithi189 ceph-mon[36187]: pgmap v1499: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 5 op/s 2024-03-24T00:50:16.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:50:15 smithi118 ceph-mon[31535]: pgmap v1500: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T00:50:16.447 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:50:15 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 1511.636146 secs 2024-03-24T00:50:16.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:50:15 smithi189 ceph-mon[36187]: pgmap v1500: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T00:50:16.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:50:15 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 1511.636146 secs 2024-03-24T00:50:16.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:50:15 smithi193 ceph-mon[36878]: pgmap v1500: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T00:50:16.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:50:15 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 1511.636146 secs 2024-03-24T00:50:18.255 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:50:17 smithi189 ceph-mon[36187]: pgmap v1501: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:50:18.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:50:17 smithi118 ceph-mon[31535]: pgmap v1501: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:50:18.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:50:17 smithi193 ceph-mon[36878]: pgmap v1501: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:50:20.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:50:19 smithi118 ceph-mon[31535]: pgmap v1502: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T00:50:20.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:50:19 smithi189 ceph-mon[36187]: pgmap v1502: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T00:50:20.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:50:19 smithi193 ceph-mon[36878]: pgmap v1502: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T00:50:21.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:50:20 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 1516.636208 secs 2024-03-24T00:50:21.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:50:20 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 1516.636208 secs 2024-03-24T00:50:21.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:50:20 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 1516.636208 secs 2024-03-24T00:50:22.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:50:21 smithi118 ceph-mon[31535]: pgmap v1503: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T00:50:22.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:50:21 smithi189 ceph-mon[36187]: pgmap v1503: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T00:50:22.460 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:50:21 smithi193 ceph-mon[36878]: pgmap v1503: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T00:50:23.947 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:50:23 smithi118 ceph-mon[31535]: pgmap v1504: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 5 op/s 2024-03-24T00:50:23.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:50:23 smithi189 ceph-mon[36187]: pgmap v1504: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 5 op/s 2024-03-24T00:50:23.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:50:23 smithi193 ceph-mon[36878]: pgmap v1504: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 5 op/s 2024-03-24T00:50:26.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:50:25 smithi118 ceph-mon[31535]: pgmap v1505: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T00:50:26.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:50:25 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 1521.636270 secs 2024-03-24T00:50:26.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:50:25 smithi189 ceph-mon[36187]: pgmap v1505: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T00:50:26.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:50:25 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 1521.636270 secs 2024-03-24T00:50:26.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:50:25 smithi193 ceph-mon[36878]: pgmap v1505: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T00:50:26.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:50:25 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 1521.636270 secs 2024-03-24T00:50:28.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:50:27 smithi118 ceph-mon[31535]: pgmap v1506: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:50:28.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:50:27 smithi189 ceph-mon[36187]: pgmap v1506: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:50:28.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:50:27 smithi193 ceph-mon[36878]: pgmap v1506: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:50:30.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:50:29 smithi189 ceph-mon[36187]: pgmap v1507: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T00:50:30.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:50:29 smithi193 ceph-mon[36878]: pgmap v1507: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T00:50:30.447 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:50:29 smithi118 ceph-mon[31535]: pgmap v1507: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T00:50:31.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:50:30 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 1526.636331 secs 2024-03-24T00:50:31.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:50:30 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 1526.636331 secs 2024-03-24T00:50:31.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:50:30 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 1526.636331 secs 2024-03-24T00:50:32.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:50:31 smithi193 ceph-mon[36878]: pgmap v1508: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T00:50:32.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:50:31 smithi118 ceph-mon[31535]: pgmap v1508: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T00:50:32.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:50:31 smithi189 ceph-mon[36187]: pgmap v1508: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T00:50:34.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:50:33 smithi118 ceph-mon[31535]: pgmap v1509: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 5 op/s 2024-03-24T00:50:34.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:50:33 smithi189 ceph-mon[36187]: pgmap v1509: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 5 op/s 2024-03-24T00:50:34.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:50:33 smithi193 ceph-mon[36878]: pgmap v1509: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 5 op/s 2024-03-24T00:50:36.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:50:35 smithi118 ceph-mon[31535]: pgmap v1510: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T00:50:36.447 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:50:35 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 1531.636419 secs 2024-03-24T00:50:36.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:50:35 smithi189 ceph-mon[36187]: pgmap v1510: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T00:50:36.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:50:35 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 1531.636419 secs 2024-03-24T00:50:36.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:50:35 smithi193 ceph-mon[36878]: pgmap v1510: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T00:50:36.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:50:35 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 1531.636419 secs 2024-03-24T00:50:38.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:50:37 smithi118 ceph-mon[31535]: pgmap v1511: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:50:38.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:50:37 smithi189 ceph-mon[36187]: pgmap v1511: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:50:38.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:50:37 smithi193 ceph-mon[36878]: pgmap v1511: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:50:40.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:50:39 smithi118 ceph-mon[31535]: pgmap v1512: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T00:50:40.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:50:39 smithi189 ceph-mon[36187]: pgmap v1512: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T00:50:40.460 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:50:39 smithi193 ceph-mon[36878]: pgmap v1512: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T00:50:41.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:50:40 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 1536.636506 secs 2024-03-24T00:50:41.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:50:40 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 1536.636506 secs 2024-03-24T00:50:41.460 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:50:40 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 1536.636506 secs 2024-03-24T00:50:42.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:50:41 smithi118 ceph-mon[31535]: pgmap v1513: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T00:50:42.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:50:41 smithi189 ceph-mon[36187]: pgmap v1513: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T00:50:42.460 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:50:41 smithi193 ceph-mon[36878]: pgmap v1513: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T00:50:44.262 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:50:43 smithi193 ceph-mon[36878]: pgmap v1514: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 5 op/s 2024-03-24T00:50:44.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:50:43 smithi118 ceph-mon[31535]: pgmap v1514: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 5 op/s 2024-03-24T00:50:44.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:50:43 smithi189 ceph-mon[36187]: pgmap v1514: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 5 op/s 2024-03-24T00:50:46.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:50:46 smithi118 ceph-mon[31535]: pgmap v1515: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T00:50:46.197 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:50:46 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 1541.636569 secs 2024-03-24T00:50:46.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:50:46 smithi189 ceph-mon[36187]: pgmap v1515: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T00:50:46.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:50:46 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 1541.636569 secs 2024-03-24T00:50:46.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:50:46 smithi193 ceph-mon[36878]: pgmap v1515: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T00:50:46.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:50:46 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 1541.636569 secs 2024-03-24T00:50:48.256 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:50:48 smithi189 ceph-mon[36187]: pgmap v1516: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:50:48.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:50:48 smithi118 ceph-mon[31535]: pgmap v1516: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:50:48.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:50:48 smithi193 ceph-mon[36878]: pgmap v1516: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:50:50.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:50:50 smithi118 ceph-mon[31535]: pgmap v1517: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T00:50:50.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:50:50 smithi189 ceph-mon[36187]: pgmap v1517: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T00:50:50.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:50:50 smithi193 ceph-mon[36878]: pgmap v1517: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T00:50:51.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:50:51 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 1546.636612 secs 2024-03-24T00:50:51.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:50:51 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 1546.636612 secs 2024-03-24T00:50:51.460 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:50:51 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 1546.636612 secs 2024-03-24T00:50:52.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:50:52 smithi118 ceph-mon[31535]: pgmap v1518: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T00:50:52.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:50:52 smithi189 ceph-mon[36187]: pgmap v1518: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T00:50:52.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:50:52 smithi193 ceph-mon[36878]: pgmap v1518: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T00:50:54.447 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:50:54 smithi118 ceph-mon[31535]: pgmap v1519: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 5 op/s 2024-03-24T00:50:54.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:50:54 smithi189 ceph-mon[36187]: pgmap v1519: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 5 op/s 2024-03-24T00:50:54.460 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:50:54 smithi193 ceph-mon[36878]: pgmap v1519: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 5 op/s 2024-03-24T00:50:56.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:50:56 smithi118 ceph-mon[31535]: pgmap v1520: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T00:50:56.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:50:56 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 1551.636702 secs 2024-03-24T00:50:56.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:50:56 smithi189 ceph-mon[36187]: pgmap v1520: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T00:50:56.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:50:56 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 1551.636702 secs 2024-03-24T00:50:56.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:50:56 smithi193 ceph-mon[36878]: pgmap v1520: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T00:50:56.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:50:56 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 1551.636702 secs 2024-03-24T00:50:58.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:50:58 smithi118 ceph-mon[31535]: pgmap v1521: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:50:58.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:50:58 smithi189 ceph-mon[36187]: pgmap v1521: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:50:58.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:50:58 smithi193 ceph-mon[36878]: pgmap v1521: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:51:00.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:51:00 smithi118 ceph-mon[31535]: pgmap v1522: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T00:51:00.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:51:00 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T00:51:00.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:51:00 smithi189 ceph-mon[36187]: pgmap v1522: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T00:51:00.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:51:00 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T00:51:00.460 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:51:00 smithi193 ceph-mon[36878]: pgmap v1522: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T00:51:00.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:51:00 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T00:51:01.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:51:01 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 1556.636743 secs 2024-03-24T00:51:01.447 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:51:01 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T00:51:01.447 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:51:01 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T00:51:01.447 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:51:01 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T00:51:01.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:51:01 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 1556.636743 secs 2024-03-24T00:51:01.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:51:01 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T00:51:01.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:51:01 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T00:51:01.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:51:01 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T00:51:01.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:51:01 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 1556.636743 secs 2024-03-24T00:51:01.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:51:01 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T00:51:01.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:51:01 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T00:51:01.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:51:01 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T00:51:02.447 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:51:02 smithi118 ceph-mon[31535]: pgmap v1523: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 6 op/s 2024-03-24T00:51:02.447 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:51:02 smithi118 ceph-mon[31535]: pgmap v1524: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-03-24T00:51:02.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:51:02 smithi189 ceph-mon[36187]: pgmap v1523: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 6 op/s 2024-03-24T00:51:02.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:51:02 smithi189 ceph-mon[36187]: pgmap v1524: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-03-24T00:51:02.460 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:51:02 smithi193 ceph-mon[36878]: pgmap v1523: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 6 op/s 2024-03-24T00:51:02.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:51:02 smithi193 ceph-mon[36878]: pgmap v1524: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-03-24T00:51:04.345 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:51:04 smithi118 ceph-mon[31535]: pgmap v1525: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 5 op/s 2024-03-24T00:51:04.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:51:04 smithi189 ceph-mon[36187]: pgmap v1525: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 5 op/s 2024-03-24T00:51:04.462 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:51:04 smithi193 ceph-mon[36878]: pgmap v1525: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 5 op/s 2024-03-24T00:51:06.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:51:06 smithi118 ceph-mon[31535]: pgmap v1526: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-03-24T00:51:06.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:51:06 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 1561.636829 secs 2024-03-24T00:51:06.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:51:06 smithi189 ceph-mon[36187]: pgmap v1526: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-03-24T00:51:06.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:51:06 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 1561.636829 secs 2024-03-24T00:51:06.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:51:06 smithi193 ceph-mon[36878]: pgmap v1526: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-03-24T00:51:06.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:51:06 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 1561.636829 secs 2024-03-24T00:51:08.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:51:08 smithi118 ceph-mon[31535]: pgmap v1527: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-03-24T00:51:08.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:51:08 smithi189 ceph-mon[36187]: pgmap v1527: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-03-24T00:51:08.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:51:08 smithi193 ceph-mon[36878]: pgmap v1527: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-03-24T00:51:10.447 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:51:10 smithi118 ceph-mon[31535]: pgmap v1528: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 5 op/s 2024-03-24T00:51:10.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:51:10 smithi189 ceph-mon[36187]: pgmap v1528: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 5 op/s 2024-03-24T00:51:10.460 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:51:10 smithi193 ceph-mon[36878]: pgmap v1528: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 5 op/s 2024-03-24T00:51:11.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:51:11 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 1566.636916 secs 2024-03-24T00:51:11.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:51:11 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 1566.636916 secs 2024-03-24T00:51:11.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:51:11 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 1566.636916 secs 2024-03-24T00:51:12.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:51:12 smithi118 ceph-mon[31535]: pgmap v1529: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T00:51:12.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:51:12 smithi189 ceph-mon[36187]: pgmap v1529: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T00:51:12.460 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:51:12 smithi193 ceph-mon[36878]: pgmap v1529: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T00:51:14.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:51:14 smithi118 ceph-mon[31535]: pgmap v1530: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-03-24T00:51:14.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:51:14 smithi189 ceph-mon[36187]: pgmap v1530: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-03-24T00:51:14.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:51:14 smithi193 ceph-mon[36878]: pgmap v1530: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-03-24T00:51:16.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:51:16 smithi118 ceph-mon[31535]: pgmap v1531: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-24T00:51:16.447 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:51:16 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 1571.637006 secs 2024-03-24T00:51:16.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:51:16 smithi189 ceph-mon[36187]: pgmap v1531: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-24T00:51:16.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:51:16 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 1571.637006 secs 2024-03-24T00:51:16.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:51:16 smithi193 ceph-mon[36878]: pgmap v1531: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-24T00:51:16.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:51:16 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 1571.637006 secs 2024-03-24T00:51:18.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:51:18 smithi118 ceph-mon[31535]: pgmap v1532: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-03-24T00:51:18.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:51:18 smithi189 ceph-mon[36187]: pgmap v1532: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-03-24T00:51:18.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:51:18 smithi193 ceph-mon[36878]: pgmap v1532: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-03-24T00:51:20.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:51:20 smithi118 ceph-mon[31535]: pgmap v1533: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-03-24T00:51:20.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:51:20 smithi189 ceph-mon[36187]: pgmap v1533: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-03-24T00:51:20.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:51:20 smithi193 ceph-mon[36878]: pgmap v1533: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-03-24T00:51:21.447 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:51:21 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 1576.637107 secs 2024-03-24T00:51:21.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:51:21 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 1576.637107 secs 2024-03-24T00:51:21.460 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:51:21 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 1576.637107 secs 2024-03-24T00:51:22.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:51:22 smithi118 ceph-mon[31535]: pgmap v1534: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T00:51:22.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:51:22 smithi189 ceph-mon[36187]: pgmap v1534: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T00:51:22.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:51:22 smithi193 ceph-mon[36878]: pgmap v1534: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T00:51:24.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:51:24 smithi118 ceph-mon[31535]: pgmap v1535: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:51:24.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:51:24 smithi189 ceph-mon[36187]: pgmap v1535: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:51:24.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:51:24 smithi193 ceph-mon[36878]: pgmap v1535: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:51:26.447 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:51:26 smithi118 ceph-mon[31535]: pgmap v1536: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-24T00:51:26.448 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:51:26 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 1581.637167 secs 2024-03-24T00:51:26.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:51:26 smithi189 ceph-mon[36187]: pgmap v1536: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-24T00:51:26.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:51:26 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 1581.637167 secs 2024-03-24T00:51:26.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:51:26 smithi193 ceph-mon[36878]: pgmap v1536: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-24T00:51:26.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:51:26 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 1581.637167 secs 2024-03-24T00:51:28.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:51:28 smithi118 ceph-mon[31535]: pgmap v1537: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:51:28.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:51:28 smithi189 ceph-mon[36187]: pgmap v1537: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:51:28.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:51:28 smithi193 ceph-mon[36878]: pgmap v1537: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:51:30.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:51:30 smithi118 ceph-mon[31535]: pgmap v1538: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T00:51:30.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:51:30 smithi189 ceph-mon[36187]: pgmap v1538: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T00:51:30.460 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:51:30 smithi193 ceph-mon[36878]: pgmap v1538: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T00:51:31.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:51:31 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 1586.637227 secs 2024-03-24T00:51:31.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:51:31 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 1586.637227 secs 2024-03-24T00:51:31.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:51:31 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 1586.637227 secs 2024-03-24T00:51:32.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:51:32 smithi118 ceph-mon[31535]: pgmap v1539: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T00:51:32.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:51:32 smithi189 ceph-mon[36187]: pgmap v1539: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T00:51:32.460 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:51:32 smithi193 ceph-mon[36878]: pgmap v1539: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T00:51:34.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:51:34 smithi118 ceph-mon[31535]: pgmap v1540: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:51:34.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:51:34 smithi189 ceph-mon[36187]: pgmap v1540: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:51:34.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:51:34 smithi193 ceph-mon[36878]: pgmap v1540: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:51:36.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:51:36 smithi118 ceph-mon[31535]: pgmap v1541: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-24T00:51:36.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:51:36 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 1591.637312 secs 2024-03-24T00:51:36.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:51:36 smithi189 ceph-mon[36187]: pgmap v1541: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-24T00:51:36.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:51:36 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 1591.637312 secs 2024-03-24T00:51:36.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:51:36 smithi193 ceph-mon[36878]: pgmap v1541: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-24T00:51:36.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:51:36 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 1591.637312 secs 2024-03-24T00:51:38.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:51:38 smithi118 ceph-mon[31535]: pgmap v1542: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:51:38.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:51:38 smithi189 ceph-mon[36187]: pgmap v1542: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:51:38.460 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:51:38 smithi193 ceph-mon[36878]: pgmap v1542: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:51:40.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:51:40 smithi118 ceph-mon[31535]: pgmap v1543: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T00:51:40.455 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:51:40 smithi189 ceph-mon[36187]: pgmap v1543: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T00:51:40.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:51:40 smithi193 ceph-mon[36878]: pgmap v1543: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T00:51:41.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:51:41 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 1596.637400 secs 2024-03-24T00:51:41.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:51:41 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 1596.637400 secs 2024-03-24T00:51:41.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:51:41 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 1596.637400 secs 2024-03-24T00:51:42.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:51:42 smithi118 ceph-mon[31535]: pgmap v1544: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T00:51:42.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:51:42 smithi189 ceph-mon[36187]: pgmap v1544: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T00:51:42.460 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:51:42 smithi193 ceph-mon[36878]: pgmap v1544: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T00:51:44.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:51:44 smithi118 ceph-mon[31535]: pgmap v1545: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:51:44.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:51:44 smithi189 ceph-mon[36187]: pgmap v1545: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:51:44.460 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:51:44 smithi193 ceph-mon[36878]: pgmap v1545: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:51:46.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:51:46 smithi118 ceph-mon[31535]: pgmap v1546: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-24T00:51:46.447 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:51:46 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 1601.637416 secs 2024-03-24T00:51:46.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:51:46 smithi189 ceph-mon[36187]: pgmap v1546: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-24T00:51:46.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:51:46 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 1601.637416 secs 2024-03-24T00:51:46.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:51:46 smithi193 ceph-mon[36878]: pgmap v1546: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-24T00:51:46.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:51:46 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 1601.637416 secs 2024-03-24T00:51:48.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:51:48 smithi118 ceph-mon[31535]: pgmap v1547: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:51:48.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:51:48 smithi189 ceph-mon[36187]: pgmap v1547: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:51:48.460 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:51:48 smithi193 ceph-mon[36878]: pgmap v1547: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:51:50.447 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:51:50 smithi118 ceph-mon[31535]: pgmap v1548: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T00:51:50.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:51:50 smithi189 ceph-mon[36187]: pgmap v1548: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T00:51:50.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:51:50 smithi193 ceph-mon[36878]: pgmap v1548: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T00:51:51.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:51:51 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 1606.637503 secs 2024-03-24T00:51:51.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:51:51 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 1606.637503 secs 2024-03-24T00:51:51.460 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:51:51 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 1606.637503 secs 2024-03-24T00:51:52.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:51:52 smithi118 ceph-mon[31535]: pgmap v1549: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T00:51:52.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:51:52 smithi189 ceph-mon[36187]: pgmap v1549: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T00:51:52.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:51:52 smithi193 ceph-mon[36878]: pgmap v1549: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T00:51:54.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:51:54 smithi189 ceph-mon[36187]: pgmap v1550: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T00:51:54.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:51:54 smithi193 ceph-mon[36878]: pgmap v1550: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T00:51:54.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:51:54 smithi118 ceph-mon[31535]: pgmap v1550: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T00:51:56.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:51:56 smithi189 ceph-mon[36187]: pgmap v1551: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-24T00:51:56.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:51:56 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 1611.637593 secs 2024-03-24T00:51:56.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:51:56 smithi193 ceph-mon[36878]: pgmap v1551: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-24T00:51:56.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:51:56 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 1611.637593 secs 2024-03-24T00:51:56.499 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:51:56 smithi118 ceph-mon[31535]: pgmap v1551: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-24T00:51:56.499 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:51:56 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 1611.637593 secs 2024-03-24T00:51:58.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:51:58 smithi193 ceph-mon[36878]: pgmap v1552: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:51:58.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:51:58 smithi118 ceph-mon[31535]: pgmap v1552: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:51:58.704 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:51:58 smithi189 ceph-mon[36187]: pgmap v1552: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:52:00.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:52:00 smithi193 ceph-mon[36878]: pgmap v1553: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T00:52:00.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:52:00 smithi118 ceph-mon[31535]: pgmap v1553: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T00:52:00.704 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:52:00 smithi189 ceph-mon[36187]: pgmap v1553: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T00:52:01.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:52:01 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 1616.637653 secs 2024-03-24T00:52:01.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:52:01 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T00:52:01.704 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:52:01 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 1616.637653 secs 2024-03-24T00:52:01.704 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:52:01 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T00:52:01.711 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:52:01 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 1616.637653 secs 2024-03-24T00:52:01.711 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:52:01 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T00:52:02.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:52:02 smithi118 ceph-mon[31535]: pgmap v1554: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T00:52:02.697 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:52:02 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T00:52:02.697 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:52:02 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T00:52:02.697 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:52:02 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T00:52:02.704 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:52:02 smithi189 ceph-mon[36187]: pgmap v1554: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T00:52:02.704 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:52:02 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T00:52:02.704 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:52:02 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T00:52:02.704 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:52:02 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T00:52:02.711 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:52:02 smithi193 ceph-mon[36878]: pgmap v1554: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T00:52:02.711 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:52:02 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T00:52:02.711 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:52:02 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T00:52:02.711 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:52:02 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T00:52:03.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:52:03 smithi118 ceph-mon[31535]: pgmap v1555: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 5 op/s 2024-03-24T00:52:03.697 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:52:03 smithi118 ceph-mon[31535]: pgmap v1556: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 5 op/s 2024-03-24T00:52:03.704 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:52:03 smithi189 ceph-mon[36187]: pgmap v1555: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 5 op/s 2024-03-24T00:52:03.704 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:52:03 smithi189 ceph-mon[36187]: pgmap v1556: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 5 op/s 2024-03-24T00:52:03.711 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:52:03 smithi193 ceph-mon[36878]: pgmap v1555: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 5 op/s 2024-03-24T00:52:03.711 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:52:03 smithi193 ceph-mon[36878]: pgmap v1556: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 5 op/s 2024-03-24T00:52:04.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:52:04 smithi118 ceph-mon[31535]: pgmap v1557: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 5 op/s 2024-03-24T00:52:04.704 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:52:04 smithi189 ceph-mon[36187]: pgmap v1557: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 5 op/s 2024-03-24T00:52:04.711 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:52:04 smithi193 ceph-mon[36878]: pgmap v1557: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 5 op/s 2024-03-24T00:52:05.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:52:05 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 1621.637714 secs 2024-03-24T00:52:05.704 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:52:05 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 1621.637714 secs 2024-03-24T00:52:05.711 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:52:05 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 1621.637714 secs 2024-03-24T00:52:06.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:52:06 smithi118 ceph-mon[31535]: pgmap v1558: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-03-24T00:52:06.704 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:52:06 smithi189 ceph-mon[36187]: pgmap v1558: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-03-24T00:52:06.710 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:52:06 smithi193 ceph-mon[36878]: pgmap v1558: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-03-24T00:52:09.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:52:08 smithi118 ceph-mon[31535]: pgmap v1559: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 5 op/s 2024-03-24T00:52:09.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:52:08 smithi189 ceph-mon[36187]: pgmap v1559: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 5 op/s 2024-03-24T00:52:09.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:52:08 smithi193 ceph-mon[36878]: pgmap v1559: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 5 op/s 2024-03-24T00:52:11.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:52:10 smithi118 ceph-mon[31535]: pgmap v1560: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-03-24T00:52:11.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:52:10 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 1626.637776 secs 2024-03-24T00:52:11.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:52:10 smithi189 ceph-mon[36187]: pgmap v1560: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-03-24T00:52:11.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:52:10 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 1626.637776 secs 2024-03-24T00:52:11.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:52:10 smithi193 ceph-mon[36878]: pgmap v1560: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-03-24T00:52:11.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:52:10 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 1626.637776 secs 2024-03-24T00:52:13.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:52:12 smithi118 ceph-mon[31535]: pgmap v1561: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 5 op/s 2024-03-24T00:52:13.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:52:12 smithi189 ceph-mon[36187]: pgmap v1561: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 5 op/s 2024-03-24T00:52:13.460 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:52:12 smithi193 ceph-mon[36878]: pgmap v1561: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 5 op/s 2024-03-24T00:52:15.262 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:52:14 smithi193 ceph-mon[36878]: pgmap v1562: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T00:52:15.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:52:14 smithi118 ceph-mon[31535]: pgmap v1562: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T00:52:15.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:52:14 smithi189 ceph-mon[36187]: pgmap v1562: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T00:52:15.987 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:52:15.986+0000 7f81ee000640 1 -- 172.21.15.118:0/1693767697 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f81c80036c0 con 0x7f81f81052d0 2024-03-24T00:52:15.987 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:52:15.987+0000 7f81ef400640 1 -- 172.21.15.118:0/1693767697 <== mon.2 v2:172.21.15.189:3300/0 17 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (secure 0 0 0) 0x7f81c80036c0 con 0x7f81f81052d0 2024-03-24T00:52:15.997 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:52:15.997+0000 7f81db400640 1 -- 172.21.15.118:0/659686424 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f81c40064d0 con 0x5622651a13f0 2024-03-24T00:52:15.998 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:52:15.998+0000 7f81e4c00640 1 -- 172.21.15.118:0/659686424 <== mon.1 v2:172.21.15.193:3300/0 17 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 409+0+0 (secure 0 0 0) 0x7f81c40064d0 con 0x5622651a13f0 2024-03-24T00:52:16.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:52:15 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 1631.637835 secs 2024-03-24T00:52:16.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:52:15 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 1631.637835 secs 2024-03-24T00:52:16.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:52:15 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 1631.637835 secs 2024-03-24T00:52:17.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:52:16 smithi118 ceph-mon[31535]: pgmap v1563: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T00:52:17.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:52:16 smithi189 ceph-mon[36187]: pgmap v1563: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T00:52:17.460 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:52:16 smithi193 ceph-mon[36878]: pgmap v1563: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T00:52:19.249 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:52:18 smithi118 ceph-mon[31535]: pgmap v1564: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 5 op/s 2024-03-24T00:52:19.255 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:52:18 smithi189 ceph-mon[36187]: pgmap v1564: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 5 op/s 2024-03-24T00:52:19.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:52:18 smithi193 ceph-mon[36878]: pgmap v1564: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 5 op/s 2024-03-24T00:52:21.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:52:20 smithi118 ceph-mon[31535]: pgmap v1565: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T00:52:21.447 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:52:21 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 1636.637923 secs 2024-03-24T00:52:21.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:52:20 smithi189 ceph-mon[36187]: pgmap v1565: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T00:52:21.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:52:20 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 1636.637923 secs 2024-03-24T00:52:21.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:52:20 smithi193 ceph-mon[36878]: pgmap v1565: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T00:52:21.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:52:20 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 1636.637923 secs 2024-03-24T00:52:23.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:52:23 smithi118 ceph-mon[31535]: pgmap v1566: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:52:23.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:52:23 smithi189 ceph-mon[36187]: pgmap v1566: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:52:23.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:52:23 smithi193 ceph-mon[36878]: pgmap v1566: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:52:25.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:52:25 smithi118 ceph-mon[31535]: pgmap v1567: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T00:52:25.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:52:25 smithi189 ceph-mon[36187]: pgmap v1567: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T00:52:25.460 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:52:25 smithi193 ceph-mon[36878]: pgmap v1567: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T00:52:26.348 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:52:26 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 1641.637967 secs 2024-03-24T00:52:26.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:52:26 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 1641.637967 secs 2024-03-24T00:52:26.460 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:52:26 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 1641.637967 secs 2024-03-24T00:52:27.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:52:27 smithi118 ceph-mon[31535]: pgmap v1568: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T00:52:27.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:52:27 smithi189 ceph-mon[36187]: pgmap v1568: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T00:52:27.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:52:27 smithi193 ceph-mon[36878]: pgmap v1568: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T00:52:29.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:52:29 smithi118 ceph-mon[31535]: pgmap v1569: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 5 op/s 2024-03-24T00:52:29.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:52:29 smithi189 ceph-mon[36187]: pgmap v1569: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 5 op/s 2024-03-24T00:52:29.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:52:29 smithi193 ceph-mon[36878]: pgmap v1569: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 5 op/s 2024-03-24T00:52:31.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:52:31 smithi118 ceph-mon[31535]: pgmap v1570: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T00:52:31.447 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:52:31 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 1646.638027 secs 2024-03-24T00:52:31.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:52:31 smithi189 ceph-mon[36187]: pgmap v1570: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T00:52:31.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:52:31 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 1646.638027 secs 2024-03-24T00:52:31.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:52:31 smithi193 ceph-mon[36878]: pgmap v1570: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T00:52:31.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:52:31 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 1646.638027 secs 2024-03-24T00:52:33.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:52:33 smithi118 ceph-mon[31535]: pgmap v1571: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:52:33.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:52:33 smithi189 ceph-mon[36187]: pgmap v1571: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:52:33.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:52:33 smithi193 ceph-mon[36878]: pgmap v1571: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:52:35.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:52:35 smithi118 ceph-mon[31535]: pgmap v1572: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T00:52:35.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:52:35 smithi189 ceph-mon[36187]: pgmap v1572: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T00:52:35.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:52:35 smithi193 ceph-mon[36878]: pgmap v1572: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T00:52:36.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:52:36 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 1651.638090 secs 2024-03-24T00:52:36.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:52:36 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 1651.638090 secs 2024-03-24T00:52:36.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:52:36 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 1651.638090 secs 2024-03-24T00:52:37.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:52:37 smithi118 ceph-mon[31535]: pgmap v1573: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T00:52:37.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:52:37 smithi189 ceph-mon[36187]: pgmap v1573: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T00:52:37.460 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:52:37 smithi193 ceph-mon[36878]: pgmap v1573: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T00:52:39.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:52:39 smithi118 ceph-mon[31535]: pgmap v1574: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 5 op/s 2024-03-24T00:52:39.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:52:39 smithi189 ceph-mon[36187]: pgmap v1574: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 5 op/s 2024-03-24T00:52:39.460 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:52:39 smithi193 ceph-mon[36878]: pgmap v1574: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 5 op/s 2024-03-24T00:52:41.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:52:41 smithi118 ceph-mon[31535]: pgmap v1575: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T00:52:41.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:52:41 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 1656.638154 secs 2024-03-24T00:52:41.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:52:41 smithi189 ceph-mon[36187]: pgmap v1575: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T00:52:41.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:52:41 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 1656.638154 secs 2024-03-24T00:52:41.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:52:41 smithi193 ceph-mon[36878]: pgmap v1575: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T00:52:41.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:52:41 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 1656.638154 secs 2024-03-24T00:52:43.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:52:43 smithi118 ceph-mon[31535]: pgmap v1576: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:52:43.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:52:43 smithi189 ceph-mon[36187]: pgmap v1576: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:52:43.460 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:52:43 smithi193 ceph-mon[36878]: pgmap v1576: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:52:45.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:52:45 smithi118 ceph-mon[31535]: pgmap v1577: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T00:52:45.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:52:45 smithi189 ceph-mon[36187]: pgmap v1577: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T00:52:45.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:52:45 smithi193 ceph-mon[36878]: pgmap v1577: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T00:52:46.447 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:52:46 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 1661.638240 secs 2024-03-24T00:52:46.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:52:46 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 1661.638240 secs 2024-03-24T00:52:46.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:52:46 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 1661.638240 secs 2024-03-24T00:52:47.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:52:47 smithi118 ceph-mon[31535]: pgmap v1578: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T00:52:47.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:52:47 smithi189 ceph-mon[36187]: pgmap v1578: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T00:52:47.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:52:47 smithi193 ceph-mon[36878]: pgmap v1578: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T00:52:49.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:52:49 smithi118 ceph-mon[31535]: pgmap v1579: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 5 op/s 2024-03-24T00:52:49.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:52:49 smithi189 ceph-mon[36187]: pgmap v1579: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 5 op/s 2024-03-24T00:52:49.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:52:49 smithi193 ceph-mon[36878]: pgmap v1579: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 5 op/s 2024-03-24T00:52:51.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:52:51 smithi118 ceph-mon[31535]: pgmap v1580: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T00:52:51.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:52:51 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 1666.638303 secs 2024-03-24T00:52:51.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:52:51 smithi189 ceph-mon[36187]: pgmap v1580: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T00:52:51.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:52:51 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 1666.638303 secs 2024-03-24T00:52:51.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:52:51 smithi193 ceph-mon[36878]: pgmap v1580: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T00:52:51.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:52:51 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 1666.638303 secs 2024-03-24T00:52:53.197 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:52:53 smithi118 ceph-mon[31535]: pgmap v1581: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:52:53.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:52:53 smithi189 ceph-mon[36187]: pgmap v1581: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:52:53.460 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:52:53 smithi193 ceph-mon[36878]: pgmap v1581: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:52:55.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:52:55 smithi118 ceph-mon[31535]: pgmap v1582: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T00:52:55.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:52:55 smithi189 ceph-mon[36187]: pgmap v1582: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T00:52:55.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:52:55 smithi193 ceph-mon[36878]: pgmap v1582: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T00:52:56.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:52:56 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 1671.638318 secs 2024-03-24T00:52:56.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:52:56 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 1671.638318 secs 2024-03-24T00:52:56.460 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:52:56 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 1671.638318 secs 2024-03-24T00:52:57.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:52:57 smithi118 ceph-mon[31535]: pgmap v1583: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T00:52:57.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:52:57 smithi189 ceph-mon[36187]: pgmap v1583: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T00:52:57.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:52:57 smithi193 ceph-mon[36878]: pgmap v1583: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T00:52:59.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:52:59 smithi118 ceph-mon[31535]: pgmap v1584: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 5 op/s 2024-03-24T00:52:59.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:52:59 smithi189 ceph-mon[36187]: pgmap v1584: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 5 op/s 2024-03-24T00:52:59.460 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:52:59 smithi193 ceph-mon[36878]: pgmap v1584: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 5 op/s 2024-03-24T00:53:01.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:53:01 smithi118 ceph-mon[31535]: pgmap v1585: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T00:53:01.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:53:01 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 1676.638379 secs 2024-03-24T00:53:01.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:53:01 smithi189 ceph-mon[36187]: pgmap v1585: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T00:53:01.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:53:01 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 1676.638379 secs 2024-03-24T00:53:01.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:53:01 smithi193 ceph-mon[36878]: pgmap v1585: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T00:53:01.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:53:01 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 1676.638379 secs 2024-03-24T00:53:03.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:53:03 smithi118 ceph-mon[31535]: pgmap v1586: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:53:03.447 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:53:03 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T00:53:03.447 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:53:03 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-24T00:53:03.447 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:53:03 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-24T00:53:03.447 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:53:03 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-24T00:53:03.447 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:53:03 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-03-24T00:53:03.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:53:03 smithi189 ceph-mon[36187]: pgmap v1586: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:53:03.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:53:03 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T00:53:03.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:53:03 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-24T00:53:03.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:53:03 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-24T00:53:03.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:53:03 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-24T00:53:03.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:53:03 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-03-24T00:53:03.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:53:03 smithi193 ceph-mon[36878]: pgmap v1586: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:53:03.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:53:03 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T00:53:03.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:53:03 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-24T00:53:03.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:53:03 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-24T00:53:03.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:53:03 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-24T00:53:03.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:53:03 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-03-24T00:53:04.447 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:53:04 smithi118 ceph-mon[31535]: Adjusting osd_memory_target on smithi189 to 174.1M 2024-03-24T00:53:04.447 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:53:04 smithi118 ceph-mon[31535]: Unable to set osd_memory_target on smithi189 to 182635622: error parsing value: Value '182635622' is below minimum 939524096 2024-03-24T00:53:04.447 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:53:04 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-24T00:53:04.447 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:53:04 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-24T00:53:04.447 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:53:04 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-24T00:53:04.447 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:53:04 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-03-24T00:53:04.447 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:53:04 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T00:53:04.448 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:53:04 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T00:53:04.448 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:53:04 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T00:53:04.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:53:04 smithi189 ceph-mon[36187]: Adjusting osd_memory_target on smithi189 to 174.1M 2024-03-24T00:53:04.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:53:04 smithi189 ceph-mon[36187]: Unable to set osd_memory_target on smithi189 to 182635622: error parsing value: Value '182635622' is below minimum 939524096 2024-03-24T00:53:04.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:53:04 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-24T00:53:04.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:53:04 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-24T00:53:04.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:53:04 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-24T00:53:04.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:53:04 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-03-24T00:53:04.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:53:04 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T00:53:04.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:53:04 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T00:53:04.455 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:53:04 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T00:53:04.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:53:04 smithi193 ceph-mon[36878]: Adjusting osd_memory_target on smithi189 to 174.1M 2024-03-24T00:53:04.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:53:04 smithi193 ceph-mon[36878]: Unable to set osd_memory_target on smithi189 to 182635622: error parsing value: Value '182635622' is below minimum 939524096 2024-03-24T00:53:04.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:53:04 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-24T00:53:04.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:53:04 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-24T00:53:04.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:53:04 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-24T00:53:04.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:53:04 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-03-24T00:53:04.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:53:04 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T00:53:04.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:53:04 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T00:53:04.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:53:04 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T00:53:05.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:53:05 smithi118 ceph-mon[31535]: Adjusting osd_memory_target on smithi118 to 176.7M 2024-03-24T00:53:05.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:53:05 smithi118 ceph-mon[31535]: Unable to set osd_memory_target on smithi118 to 185365196: error parsing value: Value '185365196' is below minimum 939524096 2024-03-24T00:53:05.447 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:53:05 smithi118 ceph-mon[31535]: pgmap v1587: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 6 op/s 2024-03-24T00:53:05.447 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:53:05 smithi118 ceph-mon[31535]: pgmap v1588: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 5 op/s 2024-03-24T00:53:05.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:53:05 smithi189 ceph-mon[36187]: Adjusting osd_memory_target on smithi118 to 176.7M 2024-03-24T00:53:05.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:53:05 smithi189 ceph-mon[36187]: Unable to set osd_memory_target on smithi118 to 185365196: error parsing value: Value '185365196' is below minimum 939524096 2024-03-24T00:53:05.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:53:05 smithi189 ceph-mon[36187]: pgmap v1587: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 6 op/s 2024-03-24T00:53:05.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:53:05 smithi189 ceph-mon[36187]: pgmap v1588: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 5 op/s 2024-03-24T00:53:05.460 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:53:05 smithi193 ceph-mon[36878]: Adjusting osd_memory_target on smithi118 to 176.7M 2024-03-24T00:53:05.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:53:05 smithi193 ceph-mon[36878]: Unable to set osd_memory_target on smithi118 to 185365196: error parsing value: Value '185365196' is below minimum 939524096 2024-03-24T00:53:05.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:53:05 smithi193 ceph-mon[36878]: pgmap v1587: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 6 op/s 2024-03-24T00:53:05.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:53:05 smithi193 ceph-mon[36878]: pgmap v1588: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 5 op/s 2024-03-24T00:53:06.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:53:06 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 1681.638469 secs 2024-03-24T00:53:06.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:53:06 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 1681.638469 secs 2024-03-24T00:53:06.460 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:53:06 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 1681.638469 secs 2024-03-24T00:53:07.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:53:07 smithi118 ceph-mon[31535]: pgmap v1589: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-03-24T00:53:07.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:53:07 smithi189 ceph-mon[36187]: pgmap v1589: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-03-24T00:53:07.460 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:53:07 smithi193 ceph-mon[36878]: pgmap v1589: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-03-24T00:53:09.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:53:09 smithi118 ceph-mon[31535]: pgmap v1590: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 5 op/s 2024-03-24T00:53:09.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:53:09 smithi189 ceph-mon[36187]: pgmap v1590: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 5 op/s 2024-03-24T00:53:09.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:53:09 smithi193 ceph-mon[36878]: pgmap v1590: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 5 op/s 2024-03-24T00:53:11.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:53:11 smithi118 ceph-mon[31535]: pgmap v1591: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-03-24T00:53:11.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:53:11 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 1686.638558 secs 2024-03-24T00:53:11.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:53:11 smithi189 ceph-mon[36187]: pgmap v1591: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-03-24T00:53:11.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:53:11 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 1686.638558 secs 2024-03-24T00:53:11.460 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:53:11 smithi193 ceph-mon[36878]: pgmap v1591: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-03-24T00:53:11.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:53:11 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 1686.638558 secs 2024-03-24T00:53:13.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:53:13 smithi118 ceph-mon[31535]: pgmap v1592: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-03-24T00:53:13.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:53:13 smithi189 ceph-mon[36187]: pgmap v1592: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-03-24T00:53:13.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:53:13 smithi193 ceph-mon[36878]: pgmap v1592: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-03-24T00:53:15.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:53:15 smithi118 ceph-mon[31535]: pgmap v1593: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 5 op/s 2024-03-24T00:53:15.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:53:15 smithi189 ceph-mon[36187]: pgmap v1593: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 5 op/s 2024-03-24T00:53:15.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:53:15 smithi193 ceph-mon[36878]: pgmap v1593: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 5 op/s 2024-03-24T00:53:16.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:53:16 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 1691.638648 secs 2024-03-24T00:53:16.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:53:16 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 1691.638648 secs 2024-03-24T00:53:16.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:53:16 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 1691.638648 secs 2024-03-24T00:53:17.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:53:17 smithi118 ceph-mon[31535]: pgmap v1594: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T00:53:17.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:53:17 smithi189 ceph-mon[36187]: pgmap v1594: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T00:53:17.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:53:17 smithi193 ceph-mon[36878]: pgmap v1594: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T00:53:19.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:53:19 smithi118 ceph-mon[31535]: pgmap v1595: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T00:53:19.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:53:19 smithi189 ceph-mon[36187]: pgmap v1595: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T00:53:19.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:53:19 smithi193 ceph-mon[36878]: pgmap v1595: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T00:53:21.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:53:21 smithi118 ceph-mon[31535]: pgmap v1596: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T00:53:21.447 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:53:21 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 1696.638669 secs 2024-03-24T00:53:21.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:53:21 smithi189 ceph-mon[36187]: pgmap v1596: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T00:53:21.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:53:21 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 1696.638669 secs 2024-03-24T00:53:21.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:53:21 smithi193 ceph-mon[36878]: pgmap v1596: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T00:53:21.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:53:21 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 1696.638669 secs 2024-03-24T00:53:23.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:53:23 smithi189 ceph-mon[36187]: pgmap v1597: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:53:23.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:53:23 smithi193 ceph-mon[36878]: pgmap v1597: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:53:23.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:53:23 smithi118 ceph-mon[31535]: pgmap v1597: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:53:25.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:53:25 smithi189 ceph-mon[36187]: pgmap v1598: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T00:53:25.460 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:53:25 smithi193 ceph-mon[36878]: pgmap v1598: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T00:53:25.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:53:25 smithi118 ceph-mon[31535]: pgmap v1598: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T00:53:26.598 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:53:26 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 1701.638713 secs 2024-03-24T00:53:26.704 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:53:26 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 1701.638713 secs 2024-03-24T00:53:26.711 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:53:26 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 1701.638713 secs 2024-03-24T00:53:27.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:53:27 smithi118 ceph-mon[31535]: pgmap v1599: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T00:53:27.704 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:53:27 smithi189 ceph-mon[36187]: pgmap v1599: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T00:53:27.711 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:53:27 smithi193 ceph-mon[36878]: pgmap v1599: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T00:53:29.697 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:53:29 smithi118 ceph-mon[31535]: pgmap v1600: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T00:53:29.704 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:53:29 smithi189 ceph-mon[36187]: pgmap v1600: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T00:53:29.710 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:53:29 smithi193 ceph-mon[36878]: pgmap v1600: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T00:53:31.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:53:31 smithi118 ceph-mon[31535]: pgmap v1601: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-03-24T00:53:31.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:53:31 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 1706.638800 secs 2024-03-24T00:53:31.704 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:53:31 smithi189 ceph-mon[36187]: pgmap v1601: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-03-24T00:53:31.704 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:53:31 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 1706.638800 secs 2024-03-24T00:53:31.711 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:53:31 smithi193 ceph-mon[36878]: pgmap v1601: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-03-24T00:53:31.711 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:53:31 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 1706.638800 secs 2024-03-24T00:53:33.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:53:33 smithi118 ceph-mon[31535]: pgmap v1602: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:53:33.704 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:53:33 smithi189 ceph-mon[36187]: pgmap v1602: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:53:33.711 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:53:33 smithi193 ceph-mon[36878]: pgmap v1602: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:53:35.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:53:35 smithi118 ceph-mon[31535]: pgmap v1603: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T00:53:35.704 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:53:35 smithi189 ceph-mon[36187]: pgmap v1603: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T00:53:35.711 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:53:35 smithi193 ceph-mon[36878]: pgmap v1603: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T00:53:36.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:53:36 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 1711.638860 secs 2024-03-24T00:53:36.704 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:53:36 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 1711.638860 secs 2024-03-24T00:53:36.710 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:53:36 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 1711.638860 secs 2024-03-24T00:53:37.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:53:37 smithi118 ceph-mon[31535]: pgmap v1604: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T00:53:37.704 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:53:37 smithi189 ceph-mon[36187]: pgmap v1604: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T00:53:37.710 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:53:37 smithi193 ceph-mon[36878]: pgmap v1604: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T00:53:39.697 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:53:39 smithi118 ceph-mon[31535]: pgmap v1605: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T00:53:39.704 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:53:39 smithi189 ceph-mon[36187]: pgmap v1605: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T00:53:39.711 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:53:39 smithi193 ceph-mon[36878]: pgmap v1605: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T00:53:40.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:53:40 smithi118 ceph-mon[31535]: pgmap v1606: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-03-24T00:53:40.704 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:53:40 smithi189 ceph-mon[36187]: pgmap v1606: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-03-24T00:53:40.710 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:53:40 smithi193 ceph-mon[36878]: pgmap v1606: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-03-24T00:53:41.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:53:41 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 1716.638949 secs 2024-03-24T00:53:41.704 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:53:41 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 1716.638949 secs 2024-03-24T00:53:41.711 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:53:41 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 1716.638949 secs 2024-03-24T00:53:42.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:53:42 smithi118 ceph-mon[31535]: pgmap v1607: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:53:42.704 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:53:42 smithi189 ceph-mon[36187]: pgmap v1607: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:53:42.710 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:53:42 smithi193 ceph-mon[36878]: pgmap v1607: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:53:44.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:53:44 smithi118 ceph-mon[31535]: pgmap v1608: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T00:53:44.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:53:44 smithi189 ceph-mon[36187]: pgmap v1608: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T00:53:44.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:53:44 smithi193 ceph-mon[36878]: pgmap v1608: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T00:53:45.862 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:53:45 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 1721.639039 secs 2024-03-24T00:53:45.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:53:45 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 1721.639039 secs 2024-03-24T00:53:45.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:53:45 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 1721.639039 secs 2024-03-24T00:53:46.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:53:46 smithi118 ceph-mon[31535]: pgmap v1609: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T00:53:46.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:53:46 smithi189 ceph-mon[36187]: pgmap v1609: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T00:53:46.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:53:46 smithi193 ceph-mon[36878]: pgmap v1609: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T00:53:48.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:53:48 smithi118 ceph-mon[31535]: pgmap v1610: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T00:53:48.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:53:48 smithi189 ceph-mon[36187]: pgmap v1610: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T00:53:48.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:53:48 smithi193 ceph-mon[36878]: pgmap v1610: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T00:53:50.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:53:50 smithi118 ceph-mon[31535]: pgmap v1611: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-03-24T00:53:50.947 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:53:50 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 1726.639126 secs 2024-03-24T00:53:50.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:53:50 smithi189 ceph-mon[36187]: pgmap v1611: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-03-24T00:53:50.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:53:50 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 1726.639126 secs 2024-03-24T00:53:50.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:53:50 smithi193 ceph-mon[36878]: pgmap v1611: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-03-24T00:53:50.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:53:50 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 1726.639126 secs 2024-03-24T00:53:52.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:53:52 smithi118 ceph-mon[31535]: pgmap v1612: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:53:52.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:53:52 smithi189 ceph-mon[36187]: pgmap v1612: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:53:52.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:53:52 smithi193 ceph-mon[36878]: pgmap v1612: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:53:54.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:53:54 smithi118 ceph-mon[31535]: pgmap v1613: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T00:53:54.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:53:54 smithi189 ceph-mon[36187]: pgmap v1613: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T00:53:54.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:53:54 smithi193 ceph-mon[36878]: pgmap v1613: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T00:53:55.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:53:55 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 1731.639190 secs 2024-03-24T00:53:55.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:53:55 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 1731.639190 secs 2024-03-24T00:53:55.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:53:55 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 1731.639190 secs 2024-03-24T00:53:56.848 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:53:56 smithi118 ceph-mon[31535]: pgmap v1614: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T00:53:56.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:53:56 smithi189 ceph-mon[36187]: pgmap v1614: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T00:53:56.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:53:56 smithi193 ceph-mon[36878]: pgmap v1614: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T00:53:58.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:53:58 smithi118 ceph-mon[31535]: pgmap v1615: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T00:53:58.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:53:58 smithi189 ceph-mon[36187]: pgmap v1615: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T00:53:58.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:53:58 smithi193 ceph-mon[36878]: pgmap v1615: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T00:54:00.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:54:00 smithi118 ceph-mon[31535]: pgmap v1616: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-03-24T00:54:00.947 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:54:00 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 1736.639252 secs 2024-03-24T00:54:00.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:54:00 smithi189 ceph-mon[36187]: pgmap v1616: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-03-24T00:54:00.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:54:00 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 1736.639252 secs 2024-03-24T00:54:00.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:54:00 smithi193 ceph-mon[36878]: pgmap v1616: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-03-24T00:54:00.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:54:00 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 1736.639252 secs 2024-03-24T00:54:02.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:54:02 smithi118 ceph-mon[31535]: pgmap v1617: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:54:02.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:54:02 smithi189 ceph-mon[36187]: pgmap v1617: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:54:02.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:54:02 smithi193 ceph-mon[36878]: pgmap v1617: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:54:04.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:54:04 smithi118 ceph-mon[31535]: pgmap v1618: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T00:54:04.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:54:04 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T00:54:04.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:54:04 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-24T00:54:04.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:54:04 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-24T00:54:04.947 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:54:04 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-03-24T00:54:04.947 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:54:04 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-03-24T00:54:04.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:54:04 smithi189 ceph-mon[36187]: pgmap v1618: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T00:54:04.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:54:04 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T00:54:04.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:54:04 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-24T00:54:04.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:54:04 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-24T00:54:04.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:54:04 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-03-24T00:54:04.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:54:04 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-03-24T00:54:04.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:54:04 smithi193 ceph-mon[36878]: pgmap v1618: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T00:54:04.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:54:04 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T00:54:04.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:54:04 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-24T00:54:04.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:54:04 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-24T00:54:04.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:54:04 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-03-24T00:54:04.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:54:04 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-03-24T00:54:05.947 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:54:05 smithi118 ceph-mon[31535]: Adjusting osd_memory_target on smithi193 to 174.1M 2024-03-24T00:54:05.947 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:54:05 smithi118 ceph-mon[31535]: Unable to set osd_memory_target on smithi193 to 182639923: error parsing value: Value '182639923' is below minimum 939524096 2024-03-24T00:54:05.947 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:54:05 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T00:54:05.947 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:54:05 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T00:54:05.947 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:54:05 smithi118 ceph-mon[31535]: pgmap v1619: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 6 op/s 2024-03-24T00:54:05.947 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:54:05 smithi118 ceph-mon[31535]: pgmap v1620: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-03-24T00:54:05.947 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:54:05 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T00:54:05.947 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:54:05 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 1741.639313 secs 2024-03-24T00:54:05.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:54:05 smithi189 ceph-mon[36187]: Adjusting osd_memory_target on smithi193 to 174.1M 2024-03-24T00:54:05.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:54:05 smithi189 ceph-mon[36187]: Unable to set osd_memory_target on smithi193 to 182639923: error parsing value: Value '182639923' is below minimum 939524096 2024-03-24T00:54:05.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:54:05 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T00:54:05.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:54:05 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T00:54:05.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:54:05 smithi189 ceph-mon[36187]: pgmap v1619: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 6 op/s 2024-03-24T00:54:05.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:54:05 smithi189 ceph-mon[36187]: pgmap v1620: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-03-24T00:54:05.955 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:54:05 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T00:54:05.955 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:54:05 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 1741.639313 secs 2024-03-24T00:54:05.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:54:05 smithi193 ceph-mon[36878]: Adjusting osd_memory_target on smithi193 to 174.1M 2024-03-24T00:54:05.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:54:05 smithi193 ceph-mon[36878]: Unable to set osd_memory_target on smithi193 to 182639923: error parsing value: Value '182639923' is below minimum 939524096 2024-03-24T00:54:05.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:54:05 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T00:54:05.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:54:05 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T00:54:05.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:54:05 smithi193 ceph-mon[36878]: pgmap v1619: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 6 op/s 2024-03-24T00:54:05.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:54:05 smithi193 ceph-mon[36878]: pgmap v1620: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-03-24T00:54:05.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:54:05 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T00:54:05.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:54:05 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 1741.639313 secs 2024-03-24T00:54:08.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:54:07 smithi118 ceph-mon[31535]: pgmap v1621: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 5 op/s 2024-03-24T00:54:08.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:54:07 smithi189 ceph-mon[36187]: pgmap v1621: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 5 op/s 2024-03-24T00:54:08.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:54:07 smithi193 ceph-mon[36878]: pgmap v1621: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 5 op/s 2024-03-24T00:54:10.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:54:09 smithi189 ceph-mon[36187]: pgmap v1622: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 4 op/s 2024-03-24T00:54:10.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:54:09 smithi193 ceph-mon[36878]: pgmap v1622: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 4 op/s 2024-03-24T00:54:10.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:54:09 smithi118 ceph-mon[31535]: pgmap v1622: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 4 op/s 2024-03-24T00:54:11.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:54:10 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 1746.639377 secs 2024-03-24T00:54:11.224 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:54:10 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 1746.639377 secs 2024-03-24T00:54:11.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:54:10 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 1746.639377 secs 2024-03-24T00:54:12.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:54:11 smithi193 ceph-mon[36878]: pgmap v1623: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-03-24T00:54:12.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:54:11 smithi118 ceph-mon[31535]: pgmap v1623: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-03-24T00:54:12.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:54:11 smithi189 ceph-mon[36187]: pgmap v1623: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-03-24T00:54:14.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:54:13 smithi118 ceph-mon[31535]: pgmap v1624: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 5 op/s 2024-03-24T00:54:14.455 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:54:13 smithi189 ceph-mon[36187]: pgmap v1624: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 5 op/s 2024-03-24T00:54:14.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:54:13 smithi193 ceph-mon[36878]: pgmap v1624: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 5 op/s 2024-03-24T00:54:16.263 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:54:15 smithi193 ceph-mon[36878]: pgmap v1625: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T00:54:16.263 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:54:15 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 1751.639437 secs 2024-03-24T00:54:16.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:54:15 smithi118 ceph-mon[31535]: pgmap v1625: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T00:54:16.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:54:15 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 1751.639437 secs 2024-03-24T00:54:16.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:54:15 smithi189 ceph-mon[36187]: pgmap v1625: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T00:54:16.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:54:15 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 1751.639437 secs 2024-03-24T00:54:18.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:54:17 smithi118 ceph-mon[31535]: pgmap v1626: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-03-24T00:54:18.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:54:17 smithi189 ceph-mon[36187]: pgmap v1626: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-03-24T00:54:18.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:54:17 smithi193 ceph-mon[36878]: pgmap v1626: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-03-24T00:54:20.255 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:54:19 smithi189 ceph-mon[36187]: pgmap v1627: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-03-24T00:54:20.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:54:19 smithi118 ceph-mon[31535]: pgmap v1627: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-03-24T00:54:20.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:54:19 smithi193 ceph-mon[36878]: pgmap v1627: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-03-24T00:54:21.447 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:54:20 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 1756.639500 secs 2024-03-24T00:54:21.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:54:20 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 1756.639500 secs 2024-03-24T00:54:21.460 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:54:20 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 1756.639500 secs 2024-03-24T00:54:22.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:54:21 smithi118 ceph-mon[31535]: pgmap v1628: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 6 op/s 2024-03-24T00:54:22.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:54:21 smithi189 ceph-mon[36187]: pgmap v1628: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 6 op/s 2024-03-24T00:54:22.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:54:21 smithi193 ceph-mon[36878]: pgmap v1628: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 6 op/s 2024-03-24T00:54:24.249 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:54:23 smithi118 ceph-mon[31535]: pgmap v1629: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 5 op/s 2024-03-24T00:54:24.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:54:23 smithi189 ceph-mon[36187]: pgmap v1629: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 5 op/s 2024-03-24T00:54:24.460 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:54:23 smithi193 ceph-mon[36878]: pgmap v1629: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 5 op/s 2024-03-24T00:54:26.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:54:25 smithi118 ceph-mon[31535]: pgmap v1630: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T00:54:26.447 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:54:25 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 1761.639564 secs 2024-03-24T00:54:26.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:54:25 smithi189 ceph-mon[36187]: pgmap v1630: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T00:54:26.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:54:25 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 1761.639564 secs 2024-03-24T00:54:26.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:54:25 smithi193 ceph-mon[36878]: pgmap v1630: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T00:54:26.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:54:25 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 1761.639564 secs 2024-03-24T00:54:28.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:54:28 smithi118 ceph-mon[31535]: pgmap v1631: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:54:28.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:54:28 smithi189 ceph-mon[36187]: pgmap v1631: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:54:28.460 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:54:28 smithi193 ceph-mon[36878]: pgmap v1631: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:54:30.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:54:30 smithi118 ceph-mon[31535]: pgmap v1632: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T00:54:30.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:54:30 smithi189 ceph-mon[36187]: pgmap v1632: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T00:54:30.460 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:54:30 smithi193 ceph-mon[36878]: pgmap v1632: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T00:54:31.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:54:31 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 1766.639653 secs 2024-03-24T00:54:31.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:54:31 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 1766.639653 secs 2024-03-24T00:54:31.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:54:31 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 1766.639653 secs 2024-03-24T00:54:32.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:54:32 smithi118 ceph-mon[31535]: pgmap v1633: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T00:54:32.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:54:32 smithi189 ceph-mon[36187]: pgmap v1633: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T00:54:32.460 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:54:32 smithi193 ceph-mon[36878]: pgmap v1633: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T00:54:34.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:54:34 smithi118 ceph-mon[31535]: pgmap v1634: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 5 op/s 2024-03-24T00:54:34.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:54:34 smithi189 ceph-mon[36187]: pgmap v1634: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 5 op/s 2024-03-24T00:54:34.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:54:34 smithi193 ceph-mon[36878]: pgmap v1634: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 5 op/s 2024-03-24T00:54:36.447 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:54:36 smithi118 ceph-mon[31535]: pgmap v1635: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T00:54:36.447 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:54:36 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 1771.639694 secs 2024-03-24T00:54:36.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:54:36 smithi189 ceph-mon[36187]: pgmap v1635: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T00:54:36.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:54:36 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 1771.639694 secs 2024-03-24T00:54:36.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:54:36 smithi193 ceph-mon[36878]: pgmap v1635: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T00:54:36.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:54:36 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 1771.639694 secs 2024-03-24T00:54:38.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:54:38 smithi118 ceph-mon[31535]: pgmap v1636: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:54:38.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:54:38 smithi189 ceph-mon[36187]: pgmap v1636: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:54:38.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:54:38 smithi193 ceph-mon[36878]: pgmap v1636: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:54:40.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:54:40 smithi118 ceph-mon[31535]: pgmap v1637: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T00:54:40.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:54:40 smithi189 ceph-mon[36187]: pgmap v1637: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T00:54:40.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:54:40 smithi193 ceph-mon[36878]: pgmap v1637: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T00:54:41.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:54:41 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 1776.639737 secs 2024-03-24T00:54:41.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:54:41 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 1776.639737 secs 2024-03-24T00:54:41.460 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:54:41 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 1776.639737 secs 2024-03-24T00:54:42.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:54:42 smithi118 ceph-mon[31535]: pgmap v1638: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T00:54:42.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:54:42 smithi189 ceph-mon[36187]: pgmap v1638: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T00:54:42.460 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:54:42 smithi193 ceph-mon[36878]: pgmap v1638: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T00:54:44.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:54:44 smithi118 ceph-mon[31535]: pgmap v1639: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 5 op/s 2024-03-24T00:54:44.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:54:44 smithi189 ceph-mon[36187]: pgmap v1639: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 5 op/s 2024-03-24T00:54:44.460 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:54:44 smithi193 ceph-mon[36878]: pgmap v1639: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 5 op/s 2024-03-24T00:54:46.447 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:54:46 smithi118 ceph-mon[31535]: pgmap v1640: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T00:54:46.447 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:54:46 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 1781.639826 secs 2024-03-24T00:54:46.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:54:46 smithi189 ceph-mon[36187]: pgmap v1640: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T00:54:46.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:54:46 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 1781.639826 secs 2024-03-24T00:54:46.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:54:46 smithi193 ceph-mon[36878]: pgmap v1640: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T00:54:46.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:54:46 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 1781.639826 secs 2024-03-24T00:54:48.447 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:54:48 smithi118 ceph-mon[31535]: pgmap v1641: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:54:48.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:54:48 smithi189 ceph-mon[36187]: pgmap v1641: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:54:48.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:54:48 smithi193 ceph-mon[36878]: pgmap v1641: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:54:50.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:54:50 smithi118 ceph-mon[31535]: pgmap v1642: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T00:54:50.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:54:50 smithi189 ceph-mon[36187]: pgmap v1642: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T00:54:50.460 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:54:50 smithi193 ceph-mon[36878]: pgmap v1642: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T00:54:51.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:54:51 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 1786.639916 secs 2024-03-24T00:54:51.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:54:51 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 1786.639916 secs 2024-03-24T00:54:51.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:54:51 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 1786.639916 secs 2024-03-24T00:54:52.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:54:52 smithi118 ceph-mon[31535]: pgmap v1643: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T00:54:52.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:54:52 smithi189 ceph-mon[36187]: pgmap v1643: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T00:54:52.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:54:52 smithi193 ceph-mon[36878]: pgmap v1643: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T00:54:54.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:54:54 smithi118 ceph-mon[31535]: pgmap v1644: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 5 op/s 2024-03-24T00:54:54.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:54:54 smithi189 ceph-mon[36187]: pgmap v1644: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 5 op/s 2024-03-24T00:54:54.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:54:54 smithi193 ceph-mon[36878]: pgmap v1644: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 5 op/s 2024-03-24T00:54:56.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:54:56 smithi118 ceph-mon[31535]: pgmap v1645: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T00:54:56.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:54:56 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 1791.640005 secs 2024-03-24T00:54:56.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:54:56 smithi189 ceph-mon[36187]: pgmap v1645: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T00:54:56.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:54:56 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 1791.640005 secs 2024-03-24T00:54:56.460 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:54:56 smithi193 ceph-mon[36878]: pgmap v1645: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T00:54:56.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:54:56 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 1791.640005 secs 2024-03-24T00:54:58.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:54:58 smithi118 ceph-mon[31535]: pgmap v1646: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:54:58.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:54:58 smithi189 ceph-mon[36187]: pgmap v1646: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:54:58.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:54:58 smithi193 ceph-mon[36878]: pgmap v1646: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:55:00.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:55:00 smithi118 ceph-mon[31535]: pgmap v1647: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T00:55:00.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:55:00 smithi189 ceph-mon[36187]: pgmap v1647: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T00:55:00.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:55:00 smithi193 ceph-mon[36878]: pgmap v1647: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T00:55:01.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:55:01 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 1796.640098 secs 2024-03-24T00:55:01.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:55:01 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 1796.640098 secs 2024-03-24T00:55:01.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:55:01 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 1796.640098 secs 2024-03-24T00:55:02.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:55:02 smithi118 ceph-mon[31535]: pgmap v1648: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T00:55:02.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:55:02 smithi189 ceph-mon[36187]: pgmap v1648: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T00:55:02.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:55:02 smithi193 ceph-mon[36878]: pgmap v1648: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T00:55:04.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:55:04 smithi118 ceph-mon[31535]: pgmap v1649: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 5 op/s 2024-03-24T00:55:04.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:55:04 smithi189 ceph-mon[36187]: pgmap v1649: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 5 op/s 2024-03-24T00:55:04.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:55:04 smithi193 ceph-mon[36878]: pgmap v1649: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 5 op/s 2024-03-24T00:55:06.447 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:55:06 smithi118 ceph-mon[31535]: pgmap v1650: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T00:55:06.447 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:55:06 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T00:55:06.447 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:55:06 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 1801.640167 secs 2024-03-24T00:55:06.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:55:06 smithi189 ceph-mon[36187]: pgmap v1650: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T00:55:06.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:55:06 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T00:55:06.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:55:06 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 1801.640167 secs 2024-03-24T00:55:06.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:55:06 smithi193 ceph-mon[36878]: pgmap v1650: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T00:55:06.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:55:06 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T00:55:06.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:55:06 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 1801.640167 secs 2024-03-24T00:55:07.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:55:07 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T00:55:07.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:55:07 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T00:55:07.447 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:55:07 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T00:55:07.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:55:07 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T00:55:07.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:55:07 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T00:55:07.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:55:07 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T00:55:07.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:55:07 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T00:55:07.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:55:07 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T00:55:07.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:55:07 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T00:55:08.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:55:08 smithi118 ceph-mon[31535]: pgmap v1651: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 5 op/s 2024-03-24T00:55:08.447 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:55:08 smithi118 ceph-mon[31535]: pgmap v1652: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-03-24T00:55:08.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:55:08 smithi189 ceph-mon[36187]: pgmap v1651: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 5 op/s 2024-03-24T00:55:08.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:55:08 smithi189 ceph-mon[36187]: pgmap v1652: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-03-24T00:55:08.460 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:55:08 smithi193 ceph-mon[36878]: pgmap v1651: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 5 op/s 2024-03-24T00:55:08.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:55:08 smithi193 ceph-mon[36878]: pgmap v1652: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-03-24T00:55:10.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:55:10 smithi118 ceph-mon[31535]: pgmap v1653: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 5 op/s 2024-03-24T00:55:10.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:55:10 smithi189 ceph-mon[36187]: pgmap v1653: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 5 op/s 2024-03-24T00:55:10.460 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:55:10 smithi193 ceph-mon[36878]: pgmap v1653: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 5 op/s 2024-03-24T00:55:11.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:55:11 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 1806.640227 secs 2024-03-24T00:55:11.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:55:11 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 1806.640227 secs 2024-03-24T00:55:11.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:55:11 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 1806.640227 secs 2024-03-24T00:55:12.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:55:12 smithi118 ceph-mon[31535]: pgmap v1654: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-03-24T00:55:12.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:55:12 smithi189 ceph-mon[36187]: pgmap v1654: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-03-24T00:55:12.460 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:55:12 smithi193 ceph-mon[36878]: pgmap v1654: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-03-24T00:55:14.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:55:14 smithi118 ceph-mon[31535]: pgmap v1655: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 5 op/s 2024-03-24T00:55:14.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:55:14 smithi189 ceph-mon[36187]: pgmap v1655: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 5 op/s 2024-03-24T00:55:14.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:55:14 smithi193 ceph-mon[36878]: pgmap v1655: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 5 op/s 2024-03-24T00:55:15.734 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:55:15.732+0000 7f81f6000640 1 -- 172.21.15.118:0/1693767697 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7f81dc0ae1d0 msgr2=0x7f81dc0b05c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 12 2024-03-24T00:55:15.734 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:55:15.732+0000 7f81f6000640 1 -- 172.21.15.118:0/1693767697 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7f81dc0ae1d0 msgr2=0x7f81dc0b05c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-03-24T00:55:15.734 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:55:15.732+0000 7f81f6000640 1 --2- 172.21.15.118:0/1693767697 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7f81dc0ae1d0 0x7f81dc0b05c0 secure :-1 s=READY pgs=421 cs=0 l=1 rev1=1 crypto rx=0x7f81dc0a2bd0 tx=0x7f81e0024040 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-03-24T00:55:15.734 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:55:15.732+0000 7f81f6000640 1 --2- 172.21.15.118:0/1693767697 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7f81dc0ae1d0 0x7f81dc0b05c0 secure :-1 s=READY pgs=421 cs=0 l=1 rev1=1 crypto rx=0x7f81dc0a2bd0 tx=0x7f81e0024040 comp rx=0 tx=0).stop 2024-03-24T00:55:15.734 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:55:15.732+0000 7f81f6000640 1 -- 172.21.15.118:0/1693767697 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7f81dc0ae1d0 msgr2=0x7f81dc0b05c0 unknown :-1 s=STATE_CLOSED l=1).tick see no progress in more than 10000000 us during connecting to v2:172.21.15.118:6800/2112912943, fault. 2024-03-24T00:55:15.734 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:55:15.732+0000 7f81ef400640 1 -- 172.21.15.118:0/1693767697 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7f81dc0ae1d0 msgr2=0x7f81dc0b05c0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-03-24T00:55:15.734 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:55:15.732+0000 7f81ef400640 1 --2- 172.21.15.118:0/1693767697 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7f81dc0ae1d0 0x7f81dc0b05c0 unknown :-1 s=CLOSED pgs=421 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T00:55:15.735 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:55:15.732+0000 7f81ef400640 1 --2- 172.21.15.118:0/1693767697 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7f81dc0a3000 0x7f81dc0a33e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T00:55:15.735 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:55:15.733+0000 7f81f6000640 1 --2- 172.21.15.118:0/1693767697 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7f81dc0a3000 0x7f81dc0a33e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:55:15.735 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:55:15.734+0000 7f81f6000640 1 --2- 172.21.15.118:0/1693767697 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7f81dc0a3000 0x7f81dc0a33e0 secure :-1 s=READY pgs=441 cs=0 l=1 rev1=1 crypto rx=0x562265215970 tx=0x7f81e0015040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T00:55:15.758 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:55:15.756+0000 7f81f6000640 1 -- 172.21.15.118:0/659686424 >> [v2:172.21.15.189:6838/4074484168,v1:172.21.15.189:6839/4074484168] conn(0x5622651b71e0 msgr2=0x562265215110 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2024-03-24T00:55:15.758 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:55:15.756+0000 7f81f6000640 1 -- 172.21.15.118:0/659686424 >> [v2:172.21.15.189:6838/4074484168,v1:172.21.15.189:6839/4074484168] conn(0x5622651b71e0 msgr2=0x562265215110 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2024-03-24T00:55:15.758 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:55:15.756+0000 7f81f6000640 1 --2- 172.21.15.118:0/659686424 >> [v2:172.21.15.189:6838/4074484168,v1:172.21.15.189:6839/4074484168] conn(0x5622651b71e0 0x562265215110 crc :-1 s=READY pgs=561 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-03-24T00:55:15.758 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:55:15.757+0000 7f81f6000640 1 --2- 172.21.15.118:0/659686424 >> [v2:172.21.15.189:6838/4074484168,v1:172.21.15.189:6839/4074484168] conn(0x5622651b71e0 0x562265215110 unknown :-1 s=READY pgs=561 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault initiating reconnect 2024-03-24T00:55:15.758 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:55:15.757+0000 7f81f6000640 1 --2- 172.21.15.118:0/659686424 >> [v2:172.21.15.189:6838/4074484168,v1:172.21.15.189:6839/4074484168] conn(0x5622651b71e0 0x562265215110 unknown :-1 s=BANNER_CONNECTING pgs=561 cs=1 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T00:55:15.759 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:55:15.758+0000 7f81f6000640 1 --2- 172.21.15.118:0/659686424 >> [v2:172.21.15.189:6838/4074484168,v1:172.21.15.189:6839/4074484168] conn(0x5622651b71e0 0x562265215110 crc :-1 s=READY pgs=561 cs=1 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=76f3d534289740f0 server_cookie=b31a2335a59552de in_seq=0 out_seq=0 2024-03-24T00:55:15.759 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:55:15.758+0000 7f81e4c00640 10 client.27094 ms_handle_connect on v2:172.21.15.189:6838/4074484168 2024-03-24T00:55:16.015 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:55:16.015+0000 7f81ee000640 1 -- 172.21.15.118:0/1693767697 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f81c80036c0 con 0x7f81f81052d0 2024-03-24T00:55:16.016 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:55:16.015+0000 7f81ef400640 1 -- 172.21.15.118:0/1693767697 <== mon.2 v2:172.21.15.189:3300/0 18 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (secure 0 0 0) 0x7f81c80036c0 con 0x7f81f81052d0 2024-03-24T00:55:16.024 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:55:16.024+0000 7f81db400640 1 -- 172.21.15.118:0/659686424 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f81c40064d0 con 0x5622651a13f0 2024-03-24T00:55:16.025 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:55:16.024+0000 7f81e4c00640 1 -- 172.21.15.118:0/659686424 <== mon.1 v2:172.21.15.193:3300/0 18 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 409+0+0 (secure 0 0 0) 0x7f81c40064d0 con 0x5622651a13f0 2024-03-24T00:55:16.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:55:16 smithi118 ceph-mon[31535]: pgmap v1656: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-03-24T00:55:16.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:55:16 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 1811.640288 secs 2024-03-24T00:55:16.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:55:16 smithi189 ceph-mon[36187]: pgmap v1656: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-03-24T00:55:16.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:55:16 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 1811.640288 secs 2024-03-24T00:55:16.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:55:16 smithi193 ceph-mon[36878]: pgmap v1656: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-03-24T00:55:16.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:55:16 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 1811.640288 secs 2024-03-24T00:55:18.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:55:18 smithi118 ceph-mon[31535]: pgmap v1657: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 5 op/s 2024-03-24T00:55:18.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:55:18 smithi189 ceph-mon[36187]: pgmap v1657: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 5 op/s 2024-03-24T00:55:18.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:55:18 smithi193 ceph-mon[36878]: pgmap v1657: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 5 op/s 2024-03-24T00:55:20.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:55:20 smithi118 ceph-mon[31535]: pgmap v1658: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T00:55:20.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:55:20 smithi189 ceph-mon[36187]: pgmap v1658: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T00:55:20.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:55:20 smithi193 ceph-mon[36878]: pgmap v1658: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T00:55:21.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:55:21 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 1816.640379 secs 2024-03-24T00:55:21.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:55:21 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 1816.640379 secs 2024-03-24T00:55:21.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:55:21 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 1816.640379 secs 2024-03-24T00:55:22.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:55:22 smithi118 ceph-mon[31535]: pgmap v1659: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T00:55:22.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:55:22 smithi189 ceph-mon[36187]: pgmap v1659: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T00:55:22.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:55:22 smithi193 ceph-mon[36878]: pgmap v1659: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T00:55:24.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:55:24 smithi118 ceph-mon[31535]: pgmap v1660: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:55:24.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:55:24 smithi189 ceph-mon[36187]: pgmap v1660: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:55:24.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:55:24 smithi193 ceph-mon[36878]: pgmap v1660: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:55:26.249 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:55:26 smithi118 ceph-mon[31535]: pgmap v1661: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-24T00:55:26.249 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:55:26 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 1821.640468 secs 2024-03-24T00:55:26.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:55:26 smithi189 ceph-mon[36187]: pgmap v1661: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-24T00:55:26.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:55:26 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 1821.640468 secs 2024-03-24T00:55:26.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:55:26 smithi193 ceph-mon[36878]: pgmap v1661: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-24T00:55:26.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:55:26 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 1821.640468 secs 2024-03-24T00:55:28.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:55:28 smithi118 ceph-mon[31535]: pgmap v1662: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:55:28.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:55:28 smithi189 ceph-mon[36187]: pgmap v1662: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:55:28.462 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:55:28 smithi193 ceph-mon[36878]: pgmap v1662: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:55:30.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:55:30 smithi118 ceph-mon[31535]: pgmap v1663: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T00:55:30.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:55:30 smithi189 ceph-mon[36187]: pgmap v1663: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T00:55:30.460 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:55:30 smithi193 ceph-mon[36878]: pgmap v1663: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T00:55:31.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:55:31 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 1826.640479 secs 2024-03-24T00:55:31.460 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:55:31 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 1826.640479 secs 2024-03-24T00:55:31.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:55:31 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 1826.640479 secs 2024-03-24T00:55:32.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:55:32 smithi189 ceph-mon[36187]: pgmap v1664: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T00:55:32.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:55:32 smithi193 ceph-mon[36878]: pgmap v1664: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T00:55:32.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:55:32 smithi118 ceph-mon[31535]: pgmap v1664: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T00:55:34.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:55:34 smithi193 ceph-mon[36878]: pgmap v1665: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:55:34.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:55:34 smithi118 ceph-mon[31535]: pgmap v1665: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:55:34.704 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:55:34 smithi189 ceph-mon[36187]: pgmap v1665: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:55:36.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:55:36 smithi118 ceph-mon[31535]: pgmap v1666: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-24T00:55:36.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:55:36 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 1831.640494 secs 2024-03-24T00:55:36.704 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:55:36 smithi189 ceph-mon[36187]: pgmap v1666: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-24T00:55:36.704 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:55:36 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 1831.640494 secs 2024-03-24T00:55:36.711 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:55:36 smithi193 ceph-mon[36878]: pgmap v1666: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-24T00:55:36.711 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:55:36 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 1831.640494 secs 2024-03-24T00:55:38.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:55:38 smithi118 ceph-mon[31535]: pgmap v1667: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:55:38.704 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:55:38 smithi189 ceph-mon[36187]: pgmap v1667: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:55:38.711 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:55:38 smithi193 ceph-mon[36878]: pgmap v1667: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:55:40.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:55:40 smithi118 ceph-mon[31535]: pgmap v1668: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T00:55:40.704 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:55:40 smithi189 ceph-mon[36187]: pgmap v1668: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T00:55:40.710 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:55:40 smithi193 ceph-mon[36878]: pgmap v1668: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T00:55:41.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:55:41 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 1836.640554 secs 2024-03-24T00:55:41.704 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:55:41 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 1836.640554 secs 2024-03-24T00:55:41.711 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:55:41 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 1836.640554 secs 2024-03-24T00:55:42.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:55:42 smithi118 ceph-mon[31535]: pgmap v1669: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T00:55:42.704 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:55:42 smithi189 ceph-mon[36187]: pgmap v1669: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T00:55:42.711 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:55:42 smithi193 ceph-mon[36878]: pgmap v1669: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T00:55:44.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:55:44 smithi118 ceph-mon[31535]: pgmap v1670: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:55:44.704 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:55:44 smithi189 ceph-mon[36187]: pgmap v1670: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:55:44.711 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:55:44 smithi193 ceph-mon[36878]: pgmap v1670: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:55:46.512 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:55:46 smithi193 ceph-mon[36878]: pgmap v1671: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-24T00:55:46.512 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:55:46 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 1841.640594 secs 2024-03-24T00:55:46.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:55:46 smithi118 ceph-mon[31535]: pgmap v1671: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-24T00:55:46.697 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:55:46 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 1841.640594 secs 2024-03-24T00:55:46.704 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:55:46 smithi189 ceph-mon[36187]: pgmap v1671: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-24T00:55:46.704 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:55:46 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 1841.640594 secs 2024-03-24T00:55:48.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:55:48 smithi118 ceph-mon[31535]: pgmap v1672: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:55:48.704 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:55:48 smithi189 ceph-mon[36187]: pgmap v1672: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:55:48.711 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:55:48 smithi193 ceph-mon[36878]: pgmap v1672: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:55:50.506 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:55:50 smithi189 ceph-mon[36187]: pgmap v1673: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T00:55:50.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:55:50 smithi118 ceph-mon[31535]: pgmap v1673: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T00:55:50.711 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:55:50 smithi193 ceph-mon[36878]: pgmap v1673: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T00:55:51.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:55:51 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 1846.640682 secs 2024-03-24T00:55:51.704 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:55:51 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 1846.640682 secs 2024-03-24T00:55:51.711 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:55:51 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 1846.640682 secs 2024-03-24T00:55:52.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:55:52 smithi118 ceph-mon[31535]: pgmap v1674: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T00:55:52.704 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:55:52 smithi189 ceph-mon[36187]: pgmap v1674: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T00:55:52.711 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:55:52 smithi193 ceph-mon[36878]: pgmap v1674: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T00:55:54.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:55:54 smithi118 ceph-mon[31535]: pgmap v1675: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:55:54.704 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:55:54 smithi189 ceph-mon[36187]: pgmap v1675: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:55:54.710 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:55:54 smithi193 ceph-mon[36878]: pgmap v1675: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:55:56.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:55:56 smithi118 ceph-mon[31535]: pgmap v1676: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-24T00:55:56.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:55:56 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 1851.640767 secs 2024-03-24T00:55:56.704 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:55:56 smithi189 ceph-mon[36187]: pgmap v1676: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-24T00:55:56.704 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:55:56 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 1851.640767 secs 2024-03-24T00:55:56.711 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:55:56 smithi193 ceph-mon[36878]: pgmap v1676: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-24T00:55:56.711 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:55:56 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 1851.640767 secs 2024-03-24T00:55:58.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:55:58 smithi118 ceph-mon[31535]: pgmap v1677: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:55:58.704 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:55:58 smithi189 ceph-mon[36187]: pgmap v1677: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:55:58.711 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:55:58 smithi193 ceph-mon[36878]: pgmap v1677: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:55:59.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:55:59 smithi118 ceph-mon[31535]: pgmap v1678: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T00:55:59.704 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:55:59 smithi189 ceph-mon[36187]: pgmap v1678: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T00:55:59.710 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:55:59 smithi193 ceph-mon[36878]: pgmap v1678: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T00:56:00.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:56:00 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 1856.640824 secs 2024-03-24T00:56:00.704 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:56:00 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 1856.640824 secs 2024-03-24T00:56:00.710 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:56:00 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 1856.640824 secs 2024-03-24T00:56:01.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:56:01 smithi118 ceph-mon[31535]: pgmap v1679: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T00:56:01.704 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:56:01 smithi189 ceph-mon[36187]: pgmap v1679: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T00:56:01.710 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:56:01 smithi193 ceph-mon[36878]: pgmap v1679: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T00:56:04.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:56:03 smithi118 ceph-mon[31535]: pgmap v1680: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:56:04.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:56:03 smithi189 ceph-mon[36187]: pgmap v1680: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:56:04.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:56:03 smithi193 ceph-mon[36878]: pgmap v1680: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:56:06.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:56:05 smithi118 ceph-mon[31535]: pgmap v1681: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-24T00:56:06.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:56:05 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 1861.640909 secs 2024-03-24T00:56:06.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:56:05 smithi189 ceph-mon[36187]: pgmap v1681: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-24T00:56:06.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:56:05 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 1861.640909 secs 2024-03-24T00:56:06.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:56:05 smithi193 ceph-mon[36878]: pgmap v1681: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-24T00:56:06.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:56:05 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 1861.640909 secs 2024-03-24T00:56:07.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:56:06 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T00:56:07.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:56:06 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T00:56:07.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:56:06 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T00:56:08.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:56:07 smithi118 ceph-mon[31535]: pgmap v1682: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:56:08.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:56:07 smithi189 ceph-mon[36187]: pgmap v1682: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:56:08.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:56:07 smithi193 ceph-mon[36878]: pgmap v1682: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:56:09.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:56:08 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T00:56:09.197 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:56:08 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T00:56:09.197 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:56:08 smithi118 ceph-mon[31535]: pgmap v1683: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 6 op/s 2024-03-24T00:56:09.197 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:56:08 smithi118 ceph-mon[31535]: pgmap v1684: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 5 op/s 2024-03-24T00:56:09.197 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:56:08 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T00:56:09.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:56:08 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T00:56:09.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:56:08 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T00:56:09.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:56:08 smithi189 ceph-mon[36187]: pgmap v1683: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 6 op/s 2024-03-24T00:56:09.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:56:08 smithi189 ceph-mon[36187]: pgmap v1684: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 5 op/s 2024-03-24T00:56:09.205 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:56:08 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T00:56:09.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:56:08 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T00:56:09.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:56:08 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T00:56:09.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:56:08 smithi193 ceph-mon[36878]: pgmap v1683: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 6 op/s 2024-03-24T00:56:09.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:56:08 smithi193 ceph-mon[36878]: pgmap v1684: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 5 op/s 2024-03-24T00:56:09.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:56:08 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T00:56:11.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:56:10 smithi118 ceph-mon[31535]: pgmap v1685: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-03-24T00:56:11.447 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:56:10 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 1866.640995 secs 2024-03-24T00:56:11.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:56:10 smithi189 ceph-mon[36187]: pgmap v1685: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-03-24T00:56:11.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:56:10 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 1866.640995 secs 2024-03-24T00:56:11.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:56:10 smithi193 ceph-mon[36878]: pgmap v1685: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-03-24T00:56:11.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:56:10 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 1866.640995 secs 2024-03-24T00:56:13.447 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:56:12 smithi118 ceph-mon[31535]: pgmap v1686: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-03-24T00:56:13.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:56:12 smithi189 ceph-mon[36187]: pgmap v1686: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-03-24T00:56:13.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:56:12 smithi193 ceph-mon[36878]: pgmap v1686: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-03-24T00:56:15.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:56:14 smithi118 ceph-mon[31535]: pgmap v1687: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 5 op/s 2024-03-24T00:56:15.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:56:14 smithi189 ceph-mon[36187]: pgmap v1687: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 5 op/s 2024-03-24T00:56:15.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:56:14 smithi193 ceph-mon[36878]: pgmap v1687: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 5 op/s 2024-03-24T00:56:16.362 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:56:15 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 1871.641038 secs 2024-03-24T00:56:16.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:56:15 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 1871.641038 secs 2024-03-24T00:56:16.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:56:16 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 1871.641038 secs 2024-03-24T00:56:17.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:56:17 smithi118 ceph-mon[31535]: pgmap v1688: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-03-24T00:56:17.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:56:17 smithi189 ceph-mon[36187]: pgmap v1688: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-03-24T00:56:17.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:56:17 smithi193 ceph-mon[36878]: pgmap v1688: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-03-24T00:56:19.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:56:19 smithi118 ceph-mon[31535]: pgmap v1689: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 5 op/s 2024-03-24T00:56:19.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:56:19 smithi189 ceph-mon[36187]: pgmap v1689: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 5 op/s 2024-03-24T00:56:19.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:56:19 smithi193 ceph-mon[36878]: pgmap v1689: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 5 op/s 2024-03-24T00:56:21.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:56:21 smithi118 ceph-mon[31535]: pgmap v1690: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 6 op/s 2024-03-24T00:56:21.447 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:56:21 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 1876.641078 secs 2024-03-24T00:56:21.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:56:21 smithi189 ceph-mon[36187]: pgmap v1690: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 6 op/s 2024-03-24T00:56:21.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:56:21 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 1876.641078 secs 2024-03-24T00:56:21.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:56:21 smithi193 ceph-mon[36878]: pgmap v1690: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 6 op/s 2024-03-24T00:56:21.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:56:21 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 1876.641078 secs 2024-03-24T00:56:23.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:56:23 smithi118 ceph-mon[31535]: pgmap v1691: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-03-24T00:56:23.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:56:23 smithi189 ceph-mon[36187]: pgmap v1691: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-03-24T00:56:23.460 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:56:23 smithi193 ceph-mon[36878]: pgmap v1691: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-03-24T00:56:25.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:56:25 smithi118 ceph-mon[31535]: pgmap v1692: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-03-24T00:56:25.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:56:25 smithi189 ceph-mon[36187]: pgmap v1692: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-03-24T00:56:25.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:56:25 smithi193 ceph-mon[36878]: pgmap v1692: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-03-24T00:56:26.348 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:56:26 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 1881.641141 secs 2024-03-24T00:56:26.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:56:26 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 1881.641141 secs 2024-03-24T00:56:26.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:56:26 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 1881.641141 secs 2024-03-24T00:56:27.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:56:27 smithi118 ceph-mon[31535]: pgmap v1693: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 5 op/s 2024-03-24T00:56:27.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:56:27 smithi189 ceph-mon[36187]: pgmap v1693: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 5 op/s 2024-03-24T00:56:27.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:56:27 smithi193 ceph-mon[36878]: pgmap v1693: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 5 op/s 2024-03-24T00:56:29.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:56:29 smithi118 ceph-mon[31535]: pgmap v1694: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 5 op/s 2024-03-24T00:56:29.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:56:29 smithi189 ceph-mon[36187]: pgmap v1694: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 5 op/s 2024-03-24T00:56:29.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:56:29 smithi193 ceph-mon[36878]: pgmap v1694: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 5 op/s 2024-03-24T00:56:31.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:56:31 smithi118 ceph-mon[31535]: pgmap v1695: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 6 op/s 2024-03-24T00:56:31.447 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:56:31 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 1886.641199 secs 2024-03-24T00:56:31.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:56:31 smithi189 ceph-mon[36187]: pgmap v1695: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 6 op/s 2024-03-24T00:56:31.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:56:31 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 1886.641199 secs 2024-03-24T00:56:31.460 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:56:31 smithi193 ceph-mon[36878]: pgmap v1695: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 6 op/s 2024-03-24T00:56:31.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:56:31 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 1886.641199 secs 2024-03-24T00:56:33.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:56:33 smithi118 ceph-mon[31535]: pgmap v1696: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:56:33.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:56:33 smithi189 ceph-mon[36187]: pgmap v1696: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:56:33.460 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:56:33 smithi193 ceph-mon[36878]: pgmap v1696: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:56:35.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:56:35 smithi118 ceph-mon[31535]: pgmap v1697: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T00:56:35.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:56:35 smithi189 ceph-mon[36187]: pgmap v1697: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T00:56:35.460 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:56:35 smithi193 ceph-mon[36878]: pgmap v1697: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T00:56:36.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:56:36 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 1891.641261 secs 2024-03-24T00:56:36.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:56:36 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 1891.641261 secs 2024-03-24T00:56:36.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:56:36 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 1891.641261 secs 2024-03-24T00:56:37.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:56:37 smithi118 ceph-mon[31535]: pgmap v1698: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T00:56:37.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:56:37 smithi189 ceph-mon[36187]: pgmap v1698: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T00:56:37.460 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:56:37 smithi193 ceph-mon[36878]: pgmap v1698: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T00:56:39.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:56:39 smithi118 ceph-mon[31535]: pgmap v1699: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 5 op/s 2024-03-24T00:56:39.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:56:39 smithi189 ceph-mon[36187]: pgmap v1699: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 5 op/s 2024-03-24T00:56:39.460 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:56:39 smithi193 ceph-mon[36878]: pgmap v1699: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 5 op/s 2024-03-24T00:56:41.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:56:41 smithi118 ceph-mon[31535]: pgmap v1700: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T00:56:41.447 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:56:41 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 1896.641346 secs 2024-03-24T00:56:41.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:56:41 smithi189 ceph-mon[36187]: pgmap v1700: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T00:56:41.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:56:41 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 1896.641346 secs 2024-03-24T00:56:41.460 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:56:41 smithi193 ceph-mon[36878]: pgmap v1700: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T00:56:41.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:56:41 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 1896.641346 secs 2024-03-24T00:56:43.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:56:43 smithi118 ceph-mon[31535]: pgmap v1701: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:56:43.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:56:43 smithi189 ceph-mon[36187]: pgmap v1701: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:56:43.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:56:43 smithi193 ceph-mon[36878]: pgmap v1701: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:56:45.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:56:45 smithi118 ceph-mon[31535]: pgmap v1702: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T00:56:45.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:56:45 smithi189 ceph-mon[36187]: pgmap v1702: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T00:56:45.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:56:45 smithi193 ceph-mon[36878]: pgmap v1702: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T00:56:46.362 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:56:46 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 1901.641432 secs 2024-03-24T00:56:46.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:56:46 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 1901.641432 secs 2024-03-24T00:56:46.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:56:46 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 1901.641432 secs 2024-03-24T00:56:47.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:56:47 smithi118 ceph-mon[31535]: pgmap v1703: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T00:56:47.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:56:47 smithi189 ceph-mon[36187]: pgmap v1703: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T00:56:47.460 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:56:47 smithi193 ceph-mon[36878]: pgmap v1703: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T00:56:49.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:56:49 smithi118 ceph-mon[31535]: pgmap v1704: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 5 op/s 2024-03-24T00:56:49.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:56:49 smithi189 ceph-mon[36187]: pgmap v1704: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 5 op/s 2024-03-24T00:56:49.460 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:56:49 smithi193 ceph-mon[36878]: pgmap v1704: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 5 op/s 2024-03-24T00:56:51.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:56:51 smithi118 ceph-mon[31535]: pgmap v1705: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T00:56:51.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:56:51 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 1906.641522 secs 2024-03-24T00:56:51.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:56:51 smithi189 ceph-mon[36187]: pgmap v1705: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T00:56:51.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:56:51 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 1906.641522 secs 2024-03-24T00:56:51.460 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:56:51 smithi193 ceph-mon[36878]: pgmap v1705: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T00:56:51.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:56:51 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 1906.641522 secs 2024-03-24T00:56:53.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:56:53 smithi118 ceph-mon[31535]: pgmap v1706: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:56:53.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:56:53 smithi189 ceph-mon[36187]: pgmap v1706: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:56:53.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:56:53 smithi193 ceph-mon[36878]: pgmap v1706: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:56:55.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:56:55 smithi118 ceph-mon[31535]: pgmap v1707: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T00:56:55.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:56:55 smithi189 ceph-mon[36187]: pgmap v1707: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T00:56:55.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:56:55 smithi193 ceph-mon[36878]: pgmap v1707: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T00:56:56.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:56:56 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 1911.641613 secs 2024-03-24T00:56:56.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:56:56 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 1911.641613 secs 2024-03-24T00:56:56.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:56:56 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 1911.641613 secs 2024-03-24T00:56:57.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:56:57 smithi118 ceph-mon[31535]: pgmap v1708: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T00:56:57.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:56:57 smithi189 ceph-mon[36187]: pgmap v1708: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T00:56:57.460 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:56:57 smithi193 ceph-mon[36878]: pgmap v1708: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T00:56:59.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:56:59 smithi118 ceph-mon[31535]: pgmap v1709: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 5 op/s 2024-03-24T00:56:59.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:56:59 smithi189 ceph-mon[36187]: pgmap v1709: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 5 op/s 2024-03-24T00:56:59.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:56:59 smithi193 ceph-mon[36878]: pgmap v1709: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 5 op/s 2024-03-24T00:57:01.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:57:01 smithi118 ceph-mon[31535]: pgmap v1710: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T00:57:01.447 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:57:01 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 1916.641699 secs 2024-03-24T00:57:01.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:57:01 smithi189 ceph-mon[36187]: pgmap v1710: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T00:57:01.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:57:01 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 1916.641699 secs 2024-03-24T00:57:01.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:57:01 smithi193 ceph-mon[36878]: pgmap v1710: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T00:57:01.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:57:01 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 1916.641699 secs 2024-03-24T00:57:03.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:57:03 smithi118 ceph-mon[31535]: pgmap v1711: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:57:03.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:57:03 smithi189 ceph-mon[36187]: pgmap v1711: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:57:03.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:57:03 smithi193 ceph-mon[36878]: pgmap v1711: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:57:05.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:57:05 smithi118 ceph-mon[31535]: pgmap v1712: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T00:57:05.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:57:05 smithi189 ceph-mon[36187]: pgmap v1712: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T00:57:05.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:57:05 smithi193 ceph-mon[36878]: pgmap v1712: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T00:57:06.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:57:06 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 1921.641780 secs 2024-03-24T00:57:06.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:57:06 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 1921.641780 secs 2024-03-24T00:57:06.460 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:57:06 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 1921.641780 secs 2024-03-24T00:57:07.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:57:07 smithi118 ceph-mon[31535]: pgmap v1713: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T00:57:07.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:57:07 smithi189 ceph-mon[36187]: pgmap v1713: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T00:57:07.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:57:07 smithi193 ceph-mon[36878]: pgmap v1713: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T00:57:09.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:57:09 smithi118 ceph-mon[31535]: pgmap v1714: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 5 op/s 2024-03-24T00:57:09.447 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:57:09 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T00:57:09.447 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:57:09 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T00:57:09.447 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:57:09 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T00:57:09.447 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:57:09 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T00:57:09.447 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:57:09 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T00:57:09.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:57:09 smithi189 ceph-mon[36187]: pgmap v1714: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 5 op/s 2024-03-24T00:57:09.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:57:09 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T00:57:09.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:57:09 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T00:57:09.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:57:09 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T00:57:09.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:57:09 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T00:57:09.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:57:09 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T00:57:09.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:57:09 smithi193 ceph-mon[36878]: pgmap v1714: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 5 op/s 2024-03-24T00:57:09.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:57:09 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T00:57:09.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:57:09 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T00:57:09.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:57:09 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T00:57:09.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:57:09 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T00:57:09.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:57:09 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T00:57:11.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:57:11 smithi118 ceph-mon[31535]: pgmap v1715: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T00:57:11.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:57:11 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 1926.641830 secs 2024-03-24T00:57:11.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:57:11 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T00:57:11.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:57:11 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T00:57:11.447 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:57:11 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T00:57:11.447 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:57:11 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T00:57:11.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:57:11 smithi189 ceph-mon[36187]: pgmap v1715: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T00:57:11.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:57:11 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 1926.641830 secs 2024-03-24T00:57:11.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:57:11 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T00:57:11.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:57:11 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T00:57:11.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:57:11 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T00:57:11.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:57:11 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T00:57:11.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:57:11 smithi193 ceph-mon[36878]: pgmap v1715: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T00:57:11.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:57:11 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 1926.641830 secs 2024-03-24T00:57:11.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:57:11 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T00:57:11.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:57:11 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T00:57:11.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:57:11 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T00:57:11.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:57:11 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T00:57:13.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:57:13 smithi118 ceph-mon[31535]: pgmap v1716: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:57:13.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:57:13 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T00:57:13.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:57:13 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T00:57:13.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:57:13 smithi189 ceph-mon[36187]: pgmap v1716: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:57:13.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:57:13 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T00:57:13.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:57:13 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T00:57:13.460 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:57:13 smithi193 ceph-mon[36878]: pgmap v1716: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:57:13.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:57:13 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T00:57:13.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:57:13 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T00:57:15.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:57:15 smithi118 ceph-mon[31535]: pgmap v1717: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T00:57:15.447 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:57:15 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T00:57:15.447 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:57:15 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T00:57:15.447 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:57:15 smithi118 ceph-mon[31535]: pgmap v1718: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 6 op/s 2024-03-24T00:57:15.447 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:57:15 smithi118 ceph-mon[31535]: pgmap v1719: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-03-24T00:57:15.447 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:57:15 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T00:57:15.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:57:15 smithi189 ceph-mon[36187]: pgmap v1717: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T00:57:15.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:57:15 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T00:57:15.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:57:15 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T00:57:15.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:57:15 smithi189 ceph-mon[36187]: pgmap v1718: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 6 op/s 2024-03-24T00:57:15.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:57:15 smithi189 ceph-mon[36187]: pgmap v1719: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-03-24T00:57:15.455 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:57:15 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T00:57:15.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:57:15 smithi193 ceph-mon[36878]: pgmap v1717: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T00:57:15.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:57:15 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T00:57:15.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:57:15 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T00:57:15.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:57:15 smithi193 ceph-mon[36878]: pgmap v1718: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 6 op/s 2024-03-24T00:57:15.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:57:15 smithi193 ceph-mon[36878]: pgmap v1719: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-03-24T00:57:15.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:57:15 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T00:57:16.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:57:16 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 1931.641870 secs 2024-03-24T00:57:16.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:57:16 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 1931.641870 secs 2024-03-24T00:57:16.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:57:16 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 1931.641870 secs 2024-03-24T00:57:17.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:57:17 smithi118 ceph-mon[31535]: pgmap v1720: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 6 op/s 2024-03-24T00:57:17.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:57:17 smithi189 ceph-mon[36187]: pgmap v1720: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 6 op/s 2024-03-24T00:57:17.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:57:17 smithi193 ceph-mon[36878]: pgmap v1720: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 6 op/s 2024-03-24T00:57:18.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:57:18 smithi118 ceph-mon[31535]: pgmap v1721: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 5 op/s 2024-03-24T00:57:18.704 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:57:18 smithi189 ceph-mon[36187]: pgmap v1721: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 5 op/s 2024-03-24T00:57:18.711 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:57:18 smithi193 ceph-mon[36878]: pgmap v1721: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 5 op/s 2024-03-24T00:57:20.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:57:20 smithi118 ceph-mon[31535]: pgmap v1722: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-03-24T00:57:20.704 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:57:20 smithi189 ceph-mon[36187]: pgmap v1722: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-03-24T00:57:20.711 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:57:20 smithi193 ceph-mon[36878]: pgmap v1722: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-03-24T00:57:21.605 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:57:21 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 1936.641957 secs 2024-03-24T00:57:21.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:57:21 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 1936.641957 secs 2024-03-24T00:57:21.711 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:57:21 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 1936.641957 secs 2024-03-24T00:57:22.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:57:22 smithi118 ceph-mon[31535]: pgmap v1723: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-03-24T00:57:22.704 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:57:22 smithi189 ceph-mon[36187]: pgmap v1723: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-03-24T00:57:22.710 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:57:22 smithi193 ceph-mon[36878]: pgmap v1723: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-03-24T00:57:24.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:57:24 smithi118 ceph-mon[31535]: pgmap v1724: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 5 op/s 2024-03-24T00:57:24.704 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:57:24 smithi189 ceph-mon[36187]: pgmap v1724: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 5 op/s 2024-03-24T00:57:24.710 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:57:24 smithi193 ceph-mon[36878]: pgmap v1724: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 5 op/s 2024-03-24T00:57:25.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:57:25 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 1941.642045 secs 2024-03-24T00:57:25.704 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:57:25 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 1941.642045 secs 2024-03-24T00:57:25.711 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:57:25 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 1941.642045 secs 2024-03-24T00:57:26.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:57:26 smithi118 ceph-mon[31535]: pgmap v1725: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:57:26.704 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:57:26 smithi189 ceph-mon[36187]: pgmap v1725: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:57:26.711 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:57:26 smithi193 ceph-mon[36878]: pgmap v1725: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:57:28.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:57:28 smithi118 ceph-mon[31535]: pgmap v1726: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T00:57:28.704 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:57:28 smithi189 ceph-mon[36187]: pgmap v1726: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T00:57:28.711 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:57:28 smithi193 ceph-mon[36878]: pgmap v1726: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T00:57:30.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:57:30 smithi118 ceph-mon[31535]: pgmap v1727: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T00:57:30.704 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:57:30 smithi189 ceph-mon[36187]: pgmap v1727: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T00:57:30.711 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:57:30 smithi193 ceph-mon[36878]: pgmap v1727: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T00:57:31.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:57:31 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 1946.642097 secs 2024-03-24T00:57:31.704 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:57:31 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 1946.642097 secs 2024-03-24T00:57:31.711 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:57:31 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 1946.642097 secs 2024-03-24T00:57:32.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:57:32 smithi118 ceph-mon[31535]: pgmap v1728: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:57:32.704 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:57:32 smithi189 ceph-mon[36187]: pgmap v1728: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:57:32.711 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:57:32 smithi193 ceph-mon[36878]: pgmap v1728: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:57:34.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:57:34 smithi118 ceph-mon[31535]: pgmap v1729: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-24T00:57:34.704 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:57:34 smithi189 ceph-mon[36187]: pgmap v1729: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-24T00:57:34.710 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:57:34 smithi193 ceph-mon[36878]: pgmap v1729: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-24T00:57:35.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:57:35 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 1951.642161 secs 2024-03-24T00:57:35.704 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:57:35 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 1951.642161 secs 2024-03-24T00:57:35.710 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:57:35 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 1951.642161 secs 2024-03-24T00:57:36.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:57:36 smithi118 ceph-mon[31535]: pgmap v1730: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:57:36.704 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:57:36 smithi189 ceph-mon[36187]: pgmap v1730: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:57:36.710 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:57:36 smithi193 ceph-mon[36878]: pgmap v1730: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:57:38.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:57:38 smithi118 ceph-mon[31535]: pgmap v1731: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T00:57:38.704 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:57:38 smithi189 ceph-mon[36187]: pgmap v1731: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T00:57:38.710 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:57:38 smithi193 ceph-mon[36878]: pgmap v1731: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T00:57:40.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:57:40 smithi118 ceph-mon[31535]: pgmap v1732: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T00:57:40.704 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:57:40 smithi189 ceph-mon[36187]: pgmap v1732: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T00:57:40.711 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:57:40 smithi193 ceph-mon[36878]: pgmap v1732: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T00:57:41.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:57:41 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 1956.642220 secs 2024-03-24T00:57:41.704 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:57:41 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 1956.642220 secs 2024-03-24T00:57:41.711 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:57:41 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 1956.642220 secs 2024-03-24T00:57:42.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:57:42 smithi118 ceph-mon[31535]: pgmap v1733: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:57:42.704 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:57:42 smithi189 ceph-mon[36187]: pgmap v1733: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:57:42.710 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:57:42 smithi193 ceph-mon[36878]: pgmap v1733: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:57:44.697 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:57:44 smithi118 ceph-mon[31535]: pgmap v1734: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-24T00:57:44.704 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:57:44 smithi189 ceph-mon[36187]: pgmap v1734: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-24T00:57:44.711 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:57:44 smithi193 ceph-mon[36878]: pgmap v1734: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-24T00:57:45.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:57:45 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 1961.642282 secs 2024-03-24T00:57:45.704 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:57:45 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 1961.642282 secs 2024-03-24T00:57:45.711 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:57:45 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 1961.642282 secs 2024-03-24T00:57:46.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:57:46 smithi118 ceph-mon[31535]: pgmap v1735: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:57:46.704 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:57:46 smithi189 ceph-mon[36187]: pgmap v1735: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:57:46.710 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:57:46 smithi193 ceph-mon[36878]: pgmap v1735: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:57:48.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:57:48 smithi118 ceph-mon[31535]: pgmap v1736: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T00:57:48.704 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:57:48 smithi189 ceph-mon[36187]: pgmap v1736: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T00:57:48.710 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:57:48 smithi193 ceph-mon[36878]: pgmap v1736: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T00:57:50.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:57:50 smithi118 ceph-mon[31535]: pgmap v1737: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T00:57:50.704 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:57:50 smithi189 ceph-mon[36187]: pgmap v1737: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T00:57:50.711 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:57:50 smithi193 ceph-mon[36878]: pgmap v1737: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T00:57:51.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:57:51 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 1966.642390 secs 2024-03-24T00:57:51.704 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:57:51 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 1966.642390 secs 2024-03-24T00:57:51.711 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:57:51 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 1966.642390 secs 2024-03-24T00:57:52.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:57:52 smithi118 ceph-mon[31535]: pgmap v1738: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:57:52.704 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:57:52 smithi189 ceph-mon[36187]: pgmap v1738: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:57:52.710 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:57:52 smithi193 ceph-mon[36878]: pgmap v1738: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:57:54.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:57:54 smithi118 ceph-mon[31535]: pgmap v1739: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-24T00:57:54.704 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:57:54 smithi189 ceph-mon[36187]: pgmap v1739: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-24T00:57:54.710 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:57:54 smithi193 ceph-mon[36878]: pgmap v1739: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-24T00:57:55.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:57:55 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 1971.642475 secs 2024-03-24T00:57:55.704 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:57:55 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 1971.642475 secs 2024-03-24T00:57:55.711 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:57:55 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 1971.642475 secs 2024-03-24T00:57:56.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:57:56 smithi118 ceph-mon[31535]: pgmap v1740: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:57:56.704 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:57:56 smithi189 ceph-mon[36187]: pgmap v1740: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:57:56.710 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:57:56 smithi193 ceph-mon[36878]: pgmap v1740: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:57:58.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:57:58 smithi118 ceph-mon[31535]: pgmap v1741: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T00:57:58.704 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:57:58 smithi189 ceph-mon[36187]: pgmap v1741: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T00:57:58.711 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:57:58 smithi193 ceph-mon[36878]: pgmap v1741: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T00:58:00.697 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:58:00 smithi118 ceph-mon[31535]: pgmap v1742: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T00:58:00.704 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:58:00 smithi189 ceph-mon[36187]: pgmap v1742: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T00:58:00.711 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:58:00 smithi193 ceph-mon[36878]: pgmap v1742: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T00:58:01.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:58:01 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 1976.642537 secs 2024-03-24T00:58:01.704 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:58:01 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 1976.642537 secs 2024-03-24T00:58:01.711 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:58:01 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 1976.642537 secs 2024-03-24T00:58:02.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:58:02 smithi118 ceph-mon[31535]: pgmap v1743: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:58:02.704 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:58:02 smithi189 ceph-mon[36187]: pgmap v1743: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:58:02.711 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:58:02 smithi193 ceph-mon[36878]: pgmap v1743: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:58:04.697 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:58:04 smithi118 ceph-mon[31535]: pgmap v1744: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-24T00:58:04.704 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:58:04 smithi189 ceph-mon[36187]: pgmap v1744: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-24T00:58:04.711 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:58:04 smithi193 ceph-mon[36878]: pgmap v1744: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-24T00:58:05.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:58:05 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 1981.642626 secs 2024-03-24T00:58:05.704 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:58:05 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 1981.642626 secs 2024-03-24T00:58:05.711 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:58:05 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 1981.642626 secs 2024-03-24T00:58:06.697 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:58:06 smithi118 ceph-mon[31535]: pgmap v1745: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:58:06.704 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:58:06 smithi189 ceph-mon[36187]: pgmap v1745: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:58:06.711 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:58:06 smithi193 ceph-mon[36878]: pgmap v1745: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:58:08.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:58:08 smithi118 ceph-mon[31535]: pgmap v1746: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T00:58:08.704 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:58:08 smithi189 ceph-mon[36187]: pgmap v1746: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T00:58:08.711 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:58:08 smithi193 ceph-mon[36878]: pgmap v1746: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T00:58:10.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:58:10 smithi118 ceph-mon[31535]: pgmap v1747: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T00:58:10.704 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:58:10 smithi189 ceph-mon[36187]: pgmap v1747: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T00:58:10.711 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:58:10 smithi193 ceph-mon[36878]: pgmap v1747: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T00:58:11.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:58:11 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 1986.642709 secs 2024-03-24T00:58:11.704 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:58:11 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 1986.642709 secs 2024-03-24T00:58:11.711 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:58:11 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 1986.642709 secs 2024-03-24T00:58:12.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:58:12 smithi118 ceph-mon[31535]: pgmap v1748: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:58:12.704 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:58:12 smithi189 ceph-mon[36187]: pgmap v1748: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:58:12.711 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:58:12 smithi193 ceph-mon[36878]: pgmap v1748: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:58:14.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:58:14 smithi118 ceph-mon[31535]: pgmap v1749: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-24T00:58:14.704 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:58:14 smithi189 ceph-mon[36187]: pgmap v1749: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-24T00:58:14.711 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:58:14 smithi193 ceph-mon[36878]: pgmap v1749: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-24T00:58:15.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:58:15 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T00:58:15.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:58:15 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T00:58:15.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:58:15 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T00:58:15.704 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:58:15 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T00:58:15.704 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:58:15 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T00:58:15.704 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:58:15 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T00:58:15.711 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:58:15 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T00:58:15.711 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:58:15 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T00:58:15.711 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:58:15 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T00:58:16.046 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:58:16.044+0000 7f81ee000640 1 -- 172.21.15.118:0/1693767697 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- auth(proto 2 132 bytes epoch 0) v1 -- 0x7f81c80036c0 con 0x7f81f81052d0 2024-03-24T00:58:16.046 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:58:16.044+0000 7f81ef400640 1 -- 172.21.15.118:0/1693767697 <== mon.2 v2:172.21.15.189:3300/0 19 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 783+0+0 (secure 0 0 0) 0x7f81c80036c0 con 0x7f81f81052d0 2024-03-24T00:58:16.047 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:58:16.045+0000 7f81ef400640 1 -- 172.21.15.118:0/1693767697 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f81c80065e0 con 0x7f81f81052d0 2024-03-24T00:58:16.047 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:58:16.045+0000 7f81ef400640 1 -- 172.21.15.118:0/1693767697 <== mon.2 v2:172.21.15.189:3300/0 20 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (secure 0 0 0) 0x7f81c80065e0 con 0x7f81f81052d0 2024-03-24T00:58:16.052 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:58:16.051+0000 7f81db400640 1 -- 172.21.15.118:0/659686424 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- auth(proto 2 132 bytes epoch 0) v1 -- 0x7f81c40064d0 con 0x5622651a13f0 2024-03-24T00:58:16.053 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:58:16.051+0000 7f81e4c00640 1 -- 172.21.15.118:0/659686424 <== mon.1 v2:172.21.15.193:3300/0 19 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 612+0+0 (secure 0 0 0) 0x7f81c40064d0 con 0x5622651a13f0 2024-03-24T00:58:16.053 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:58:16.052+0000 7f81e4c00640 1 -- 172.21.15.118:0/659686424 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f81c4006300 con 0x5622651a13f0 2024-03-24T00:58:16.053 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T00:58:16.052+0000 7f81e4c00640 1 -- 172.21.15.118:0/659686424 <== mon.1 v2:172.21.15.193:3300/0 20 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 409+0+0 (secure 0 0 0) 0x7f81c4006300 con 0x5622651a13f0 2024-03-24T00:58:16.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:58:16 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 1991.642807 secs 2024-03-24T00:58:16.697 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:58:16 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T00:58:16.697 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:58:16 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T00:58:16.697 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:58:16 smithi118 ceph-mon[31535]: pgmap v1750: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 5 op/s 2024-03-24T00:58:16.697 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:58:16 smithi118 ceph-mon[31535]: pgmap v1751: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 6 op/s 2024-03-24T00:58:16.697 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:58:16 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T00:58:16.704 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:58:16 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 1991.642807 secs 2024-03-24T00:58:16.704 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:58:16 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T00:58:16.704 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:58:16 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T00:58:16.704 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:58:16 smithi189 ceph-mon[36187]: pgmap v1750: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 5 op/s 2024-03-24T00:58:16.704 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:58:16 smithi189 ceph-mon[36187]: pgmap v1751: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 6 op/s 2024-03-24T00:58:16.704 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:58:16 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T00:58:16.711 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:58:16 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 1991.642807 secs 2024-03-24T00:58:16.711 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:58:16 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T00:58:16.711 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:58:16 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T00:58:16.711 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:58:16 smithi193 ceph-mon[36878]: pgmap v1750: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 5 op/s 2024-03-24T00:58:16.711 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:58:16 smithi193 ceph-mon[36878]: pgmap v1751: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 6 op/s 2024-03-24T00:58:16.711 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:58:16 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T00:58:19.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:58:18 smithi118 ceph-mon[31535]: pgmap v1752: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 5 op/s 2024-03-24T00:58:19.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:58:18 smithi189 ceph-mon[36187]: pgmap v1752: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 5 op/s 2024-03-24T00:58:19.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:58:18 smithi193 ceph-mon[36878]: pgmap v1752: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 5 op/s 2024-03-24T00:58:21.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:58:20 smithi118 ceph-mon[31535]: pgmap v1753: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 6 op/s 2024-03-24T00:58:21.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:58:20 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 1996.642847 secs 2024-03-24T00:58:21.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:58:20 smithi189 ceph-mon[36187]: pgmap v1753: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 6 op/s 2024-03-24T00:58:21.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:58:20 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 1996.642847 secs 2024-03-24T00:58:21.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:58:20 smithi193 ceph-mon[36878]: pgmap v1753: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 6 op/s 2024-03-24T00:58:21.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:58:20 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 1996.642847 secs 2024-03-24T00:58:23.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:58:22 smithi118 ceph-mon[31535]: pgmap v1754: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 6 op/s 2024-03-24T00:58:23.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:58:22 smithi189 ceph-mon[36187]: pgmap v1754: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 6 op/s 2024-03-24T00:58:23.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:58:22 smithi193 ceph-mon[36878]: pgmap v1754: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 6 op/s 2024-03-24T00:58:25.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:58:24 smithi118 ceph-mon[31535]: pgmap v1755: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 4 op/s 2024-03-24T00:58:25.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:58:24 smithi189 ceph-mon[36187]: pgmap v1755: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 4 op/s 2024-03-24T00:58:25.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:58:24 smithi193 ceph-mon[36878]: pgmap v1755: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 4 op/s 2024-03-24T00:58:26.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:58:25 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 2001.642934 secs 2024-03-24T00:58:26.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:58:25 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 2001.642934 secs 2024-03-24T00:58:26.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:58:25 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 2001.642934 secs 2024-03-24T00:58:27.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:58:26 smithi118 ceph-mon[31535]: pgmap v1756: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 5 op/s 2024-03-24T00:58:27.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:58:26 smithi189 ceph-mon[36187]: pgmap v1756: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 5 op/s 2024-03-24T00:58:27.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:58:26 smithi193 ceph-mon[36878]: pgmap v1756: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 5 op/s 2024-03-24T00:58:29.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:58:28 smithi118 ceph-mon[31535]: pgmap v1757: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 5 op/s 2024-03-24T00:58:29.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:58:28 smithi189 ceph-mon[36187]: pgmap v1757: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 5 op/s 2024-03-24T00:58:29.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:58:28 smithi193 ceph-mon[36878]: pgmap v1757: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 5 op/s 2024-03-24T00:58:31.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:58:30 smithi118 ceph-mon[31535]: pgmap v1758: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T00:58:31.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:58:30 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 2006.643024 secs 2024-03-24T00:58:31.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:58:30 smithi189 ceph-mon[36187]: pgmap v1758: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T00:58:31.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:58:30 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 2006.643024 secs 2024-03-24T00:58:31.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:58:30 smithi193 ceph-mon[36878]: pgmap v1758: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T00:58:31.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:58:30 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 2006.643024 secs 2024-03-24T00:58:33.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:58:32 smithi118 ceph-mon[31535]: pgmap v1759: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:58:33.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:58:32 smithi189 ceph-mon[36187]: pgmap v1759: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:58:33.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:58:32 smithi193 ceph-mon[36878]: pgmap v1759: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:58:35.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:58:34 smithi118 ceph-mon[31535]: pgmap v1760: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-03-24T00:58:35.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:58:34 smithi189 ceph-mon[36187]: pgmap v1760: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-03-24T00:58:35.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:58:34 smithi193 ceph-mon[36878]: pgmap v1760: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-03-24T00:58:36.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:58:35 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 2011.643101 secs 2024-03-24T00:58:36.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:58:35 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 2011.643101 secs 2024-03-24T00:58:36.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:58:35 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 2011.643101 secs 2024-03-24T00:58:37.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:58:36 smithi118 ceph-mon[31535]: pgmap v1761: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 6 op/s 2024-03-24T00:58:37.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:58:36 smithi189 ceph-mon[36187]: pgmap v1761: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 6 op/s 2024-03-24T00:58:37.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:58:36 smithi193 ceph-mon[36878]: pgmap v1761: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 6 op/s 2024-03-24T00:58:39.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:58:38 smithi118 ceph-mon[31535]: pgmap v1762: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 5 op/s 2024-03-24T00:58:39.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:58:38 smithi189 ceph-mon[36187]: pgmap v1762: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 5 op/s 2024-03-24T00:58:39.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:58:38 smithi193 ceph-mon[36878]: pgmap v1762: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 5 op/s 2024-03-24T00:58:41.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:58:40 smithi118 ceph-mon[31535]: pgmap v1763: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 6 op/s 2024-03-24T00:58:41.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:58:40 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 2016.643182 secs 2024-03-24T00:58:41.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:58:40 smithi189 ceph-mon[36187]: pgmap v1763: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 6 op/s 2024-03-24T00:58:41.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:58:40 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 2016.643182 secs 2024-03-24T00:58:41.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:58:40 smithi193 ceph-mon[36878]: pgmap v1763: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 6 op/s 2024-03-24T00:58:41.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:58:40 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 2016.643182 secs 2024-03-24T00:58:43.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:58:42 smithi118 ceph-mon[31535]: pgmap v1764: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-03-24T00:58:43.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:58:42 smithi189 ceph-mon[36187]: pgmap v1764: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-03-24T00:58:43.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:58:42 smithi193 ceph-mon[36878]: pgmap v1764: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-03-24T00:58:45.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:58:44 smithi118 ceph-mon[31535]: pgmap v1765: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-03-24T00:58:45.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:58:44 smithi189 ceph-mon[36187]: pgmap v1765: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-03-24T00:58:45.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:58:44 smithi193 ceph-mon[36878]: pgmap v1765: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-03-24T00:58:46.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:58:45 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 2021.643245 secs 2024-03-24T00:58:46.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:58:45 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 2021.643245 secs 2024-03-24T00:58:46.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:58:45 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 2021.643245 secs 2024-03-24T00:58:47.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:58:46 smithi189 ceph-mon[36187]: pgmap v1766: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T00:58:47.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:58:46 smithi193 ceph-mon[36878]: pgmap v1766: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T00:58:47.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:58:46 smithi118 ceph-mon[31535]: pgmap v1766: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T00:58:49.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:58:48 smithi193 ceph-mon[36878]: pgmap v1767: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 5 op/s 2024-03-24T00:58:49.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:58:48 smithi118 ceph-mon[31535]: pgmap v1767: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 5 op/s 2024-03-24T00:58:49.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:58:48 smithi189 ceph-mon[36187]: pgmap v1767: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 5 op/s 2024-03-24T00:58:51.256 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:58:50 smithi189 ceph-mon[36187]: pgmap v1768: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T00:58:51.256 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:58:50 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 2026.643259 secs 2024-03-24T00:58:51.288 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:58:50 smithi118 ceph-mon[31535]: pgmap v1768: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T00:58:51.288 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:58:50 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 2026.643259 secs 2024-03-24T00:58:51.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:58:50 smithi193 ceph-mon[36878]: pgmap v1768: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T00:58:51.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:58:50 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 2026.643259 secs 2024-03-24T00:58:53.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:58:52 smithi118 ceph-mon[31535]: pgmap v1769: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:58:53.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:58:52 smithi189 ceph-mon[36187]: pgmap v1769: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:58:53.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:58:52 smithi193 ceph-mon[36878]: pgmap v1769: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:58:55.447 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:58:54 smithi118 ceph-mon[31535]: pgmap v1770: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T00:58:55.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:58:54 smithi189 ceph-mon[36187]: pgmap v1770: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T00:58:55.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:58:54 smithi193 ceph-mon[36878]: pgmap v1770: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T00:58:56.349 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:58:55 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 2031.643344 secs 2024-03-24T00:58:56.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:58:55 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 2031.643344 secs 2024-03-24T00:58:56.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:58:55 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 2031.643344 secs 2024-03-24T00:58:57.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:58:56 smithi118 ceph-mon[31535]: pgmap v1771: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T00:58:57.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:58:56 smithi189 ceph-mon[36187]: pgmap v1771: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T00:58:57.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:58:56 smithi193 ceph-mon[36878]: pgmap v1771: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T00:58:59.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:58:58 smithi118 ceph-mon[31535]: pgmap v1772: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 5 op/s 2024-03-24T00:58:59.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:58:58 smithi189 ceph-mon[36187]: pgmap v1772: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 5 op/s 2024-03-24T00:58:59.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:58:58 smithi193 ceph-mon[36878]: pgmap v1772: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 5 op/s 2024-03-24T00:59:01.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:59:00 smithi118 ceph-mon[31535]: pgmap v1773: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T00:59:01.447 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:59:00 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 2036.643403 secs 2024-03-24T00:59:01.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:59:00 smithi189 ceph-mon[36187]: pgmap v1773: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T00:59:01.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:59:00 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 2036.643403 secs 2024-03-24T00:59:01.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:59:00 smithi193 ceph-mon[36878]: pgmap v1773: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T00:59:01.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:59:00 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 2036.643403 secs 2024-03-24T00:59:03.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:59:02 smithi118 ceph-mon[31535]: pgmap v1774: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:59:03.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:59:02 smithi189 ceph-mon[36187]: pgmap v1774: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:59:03.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:59:02 smithi193 ceph-mon[36878]: pgmap v1774: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:59:05.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:59:04 smithi118 ceph-mon[31535]: pgmap v1775: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T00:59:05.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:59:04 smithi189 ceph-mon[36187]: pgmap v1775: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T00:59:05.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:59:04 smithi193 ceph-mon[36878]: pgmap v1775: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T00:59:06.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:59:06 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 2041.643463 secs 2024-03-24T00:59:06.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:59:06 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 2041.643463 secs 2024-03-24T00:59:06.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:59:06 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 2041.643463 secs 2024-03-24T00:59:07.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:59:07 smithi118 ceph-mon[31535]: pgmap v1776: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T00:59:07.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:59:07 smithi189 ceph-mon[36187]: pgmap v1776: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T00:59:07.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:59:07 smithi193 ceph-mon[36878]: pgmap v1776: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T00:59:09.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:59:09 smithi118 ceph-mon[31535]: pgmap v1777: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 5 op/s 2024-03-24T00:59:09.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:59:09 smithi189 ceph-mon[36187]: pgmap v1777: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 5 op/s 2024-03-24T00:59:09.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:59:09 smithi193 ceph-mon[36878]: pgmap v1777: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 5 op/s 2024-03-24T00:59:11.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:59:11 smithi118 ceph-mon[31535]: pgmap v1778: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T00:59:11.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:59:11 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 2046.643553 secs 2024-03-24T00:59:11.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:59:11 smithi189 ceph-mon[36187]: pgmap v1778: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T00:59:11.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:59:11 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 2046.643553 secs 2024-03-24T00:59:11.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:59:11 smithi193 ceph-mon[36878]: pgmap v1778: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T00:59:11.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:59:11 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 2046.643553 secs 2024-03-24T00:59:13.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:59:13 smithi118 ceph-mon[31535]: pgmap v1779: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:59:13.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:59:13 smithi189 ceph-mon[36187]: pgmap v1779: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:59:13.460 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:59:13 smithi193 ceph-mon[36878]: pgmap v1779: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:59:15.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:59:15 smithi118 ceph-mon[31535]: pgmap v1780: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T00:59:15.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:59:15 smithi189 ceph-mon[36187]: pgmap v1780: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T00:59:15.460 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:59:15 smithi193 ceph-mon[36878]: pgmap v1780: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T00:59:16.362 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:59:16 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 2051.643641 secs 2024-03-24T00:59:16.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:59:16 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 2051.643641 secs 2024-03-24T00:59:16.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:59:16 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 2051.643641 secs 2024-03-24T00:59:17.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:59:17 smithi118 ceph-mon[31535]: pgmap v1781: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T00:59:17.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:59:17 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T00:59:17.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:59:17 smithi189 ceph-mon[36187]: pgmap v1781: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T00:59:17.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:59:17 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T00:59:17.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:59:17 smithi193 ceph-mon[36878]: pgmap v1781: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T00:59:17.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:59:17 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T00:59:18.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:59:18 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T00:59:18.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:59:18 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T00:59:18.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:59:18 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T00:59:18.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:59:18 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T00:59:18.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:59:18 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T00:59:18.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:59:18 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T00:59:18.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:59:18 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T00:59:18.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:59:18 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T00:59:18.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:59:18 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T00:59:19.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:59:19 smithi118 ceph-mon[31535]: pgmap v1782: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 5 op/s 2024-03-24T00:59:19.447 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:59:19 smithi118 ceph-mon[31535]: pgmap v1783: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 5 op/s 2024-03-24T00:59:19.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:59:19 smithi189 ceph-mon[36187]: pgmap v1782: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 5 op/s 2024-03-24T00:59:19.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:59:19 smithi189 ceph-mon[36187]: pgmap v1783: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 5 op/s 2024-03-24T00:59:19.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:59:19 smithi193 ceph-mon[36878]: pgmap v1782: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 5 op/s 2024-03-24T00:59:19.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:59:19 smithi193 ceph-mon[36878]: pgmap v1783: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 5 op/s 2024-03-24T00:59:21.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:59:21 smithi118 ceph-mon[31535]: pgmap v1784: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-03-24T00:59:21.447 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:59:21 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 2056.643702 secs 2024-03-24T00:59:21.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:59:21 smithi189 ceph-mon[36187]: pgmap v1784: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-03-24T00:59:21.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:59:21 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 2056.643702 secs 2024-03-24T00:59:21.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:59:21 smithi193 ceph-mon[36878]: pgmap v1784: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-03-24T00:59:21.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:59:21 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 2056.643702 secs 2024-03-24T00:59:23.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:59:23 smithi118 ceph-mon[31535]: pgmap v1785: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-03-24T00:59:23.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:59:23 smithi189 ceph-mon[36187]: pgmap v1785: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-03-24T00:59:23.460 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:59:23 smithi193 ceph-mon[36878]: pgmap v1785: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-03-24T00:59:25.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:59:25 smithi118 ceph-mon[31535]: pgmap v1786: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-03-24T00:59:25.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:59:25 smithi189 ceph-mon[36187]: pgmap v1786: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-03-24T00:59:25.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:59:25 smithi193 ceph-mon[36878]: pgmap v1786: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-03-24T00:59:26.348 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:59:26 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 2061.643792 secs 2024-03-24T00:59:26.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:59:26 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 2061.643792 secs 2024-03-24T00:59:26.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:59:26 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 2061.643792 secs 2024-03-24T00:59:27.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:59:27 smithi118 ceph-mon[31535]: pgmap v1787: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-03-24T00:59:27.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:59:27 smithi189 ceph-mon[36187]: pgmap v1787: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-03-24T00:59:27.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:59:27 smithi193 ceph-mon[36878]: pgmap v1787: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-03-24T00:59:29.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:59:29 smithi118 ceph-mon[31535]: pgmap v1788: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 5 op/s 2024-03-24T00:59:29.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:59:29 smithi189 ceph-mon[36187]: pgmap v1788: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 5 op/s 2024-03-24T00:59:29.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:59:29 smithi193 ceph-mon[36878]: pgmap v1788: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 5 op/s 2024-03-24T00:59:31.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:59:31 smithi118 ceph-mon[31535]: pgmap v1789: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-24T00:59:31.447 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:59:31 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 2066.643881 secs 2024-03-24T00:59:31.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:59:31 smithi189 ceph-mon[36187]: pgmap v1789: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-24T00:59:31.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:59:31 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 2066.643881 secs 2024-03-24T00:59:31.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:59:31 smithi193 ceph-mon[36878]: pgmap v1789: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-24T00:59:31.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:59:31 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 2066.643881 secs 2024-03-24T00:59:33.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:59:33 smithi118 ceph-mon[31535]: pgmap v1790: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:59:33.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:59:33 smithi189 ceph-mon[36187]: pgmap v1790: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:59:33.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:59:33 smithi193 ceph-mon[36878]: pgmap v1790: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:59:35.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:59:35 smithi118 ceph-mon[31535]: pgmap v1791: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T00:59:35.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:59:35 smithi189 ceph-mon[36187]: pgmap v1791: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T00:59:35.460 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:59:35 smithi193 ceph-mon[36878]: pgmap v1791: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T00:59:36.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:59:36 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 2071.643967 secs 2024-03-24T00:59:36.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:59:36 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 2071.643967 secs 2024-03-24T00:59:36.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:59:36 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 2071.643967 secs 2024-03-24T00:59:37.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:59:37 smithi118 ceph-mon[31535]: pgmap v1792: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T00:59:37.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:59:37 smithi189 ceph-mon[36187]: pgmap v1792: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T00:59:37.460 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:59:37 smithi193 ceph-mon[36878]: pgmap v1792: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T00:59:39.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:59:39 smithi118 ceph-mon[31535]: pgmap v1793: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:59:39.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:59:39 smithi189 ceph-mon[36187]: pgmap v1793: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:59:39.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:59:39 smithi193 ceph-mon[36878]: pgmap v1793: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:59:41.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:59:41 smithi118 ceph-mon[31535]: pgmap v1794: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-24T00:59:41.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:59:41 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 2076.644061 secs 2024-03-24T00:59:41.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:59:41 smithi189 ceph-mon[36187]: pgmap v1794: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-24T00:59:41.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:59:41 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 2076.644061 secs 2024-03-24T00:59:41.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:59:41 smithi193 ceph-mon[36878]: pgmap v1794: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-24T00:59:41.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:59:41 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 2076.644061 secs 2024-03-24T00:59:43.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:59:43 smithi118 ceph-mon[31535]: pgmap v1795: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:59:43.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:59:43 smithi189 ceph-mon[36187]: pgmap v1795: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:59:43.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:59:43 smithi193 ceph-mon[36878]: pgmap v1795: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:59:45.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:59:45 smithi118 ceph-mon[31535]: pgmap v1796: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T00:59:45.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:59:45 smithi189 ceph-mon[36187]: pgmap v1796: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T00:59:45.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:59:45 smithi193 ceph-mon[36878]: pgmap v1796: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T00:59:46.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:59:46 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 2081.644097 secs 2024-03-24T00:59:46.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:59:46 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 2081.644097 secs 2024-03-24T00:59:46.460 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:59:46 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 2081.644097 secs 2024-03-24T00:59:47.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:59:47 smithi118 ceph-mon[31535]: pgmap v1797: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T00:59:47.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:59:47 smithi189 ceph-mon[36187]: pgmap v1797: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T00:59:47.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:59:47 smithi193 ceph-mon[36878]: pgmap v1797: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T00:59:49.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:59:49 smithi118 ceph-mon[31535]: pgmap v1798: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:59:49.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:59:49 smithi189 ceph-mon[36187]: pgmap v1798: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:59:49.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:59:49 smithi193 ceph-mon[36878]: pgmap v1798: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:59:51.447 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:59:51 smithi118 ceph-mon[31535]: pgmap v1799: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-24T00:59:51.447 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:59:51 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 2086.644182 secs 2024-03-24T00:59:51.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:59:51 smithi189 ceph-mon[36187]: pgmap v1799: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-24T00:59:51.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:59:51 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 2086.644182 secs 2024-03-24T00:59:51.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:59:51 smithi193 ceph-mon[36878]: pgmap v1799: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-24T00:59:51.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:59:51 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 2086.644182 secs 2024-03-24T00:59:53.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:59:53 smithi118 ceph-mon[31535]: pgmap v1800: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:59:53.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:59:53 smithi189 ceph-mon[36187]: pgmap v1800: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:59:53.460 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:59:53 smithi193 ceph-mon[36878]: pgmap v1800: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:59:55.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:59:55 smithi118 ceph-mon[31535]: pgmap v1801: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T00:59:55.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:59:55 smithi189 ceph-mon[36187]: pgmap v1801: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T00:59:55.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:59:55 smithi193 ceph-mon[36878]: pgmap v1801: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T00:59:56.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:59:56 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 2091.644241 secs 2024-03-24T00:59:56.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:59:56 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 2091.644241 secs 2024-03-24T00:59:56.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:59:56 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 2091.644241 secs 2024-03-24T00:59:57.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:59:57 smithi118 ceph-mon[31535]: pgmap v1802: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T00:59:57.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:59:57 smithi189 ceph-mon[36187]: pgmap v1802: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T00:59:57.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:59:57 smithi193 ceph-mon[36878]: pgmap v1802: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T00:59:59.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 00:59:59 smithi118 ceph-mon[31535]: pgmap v1803: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:59:59.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 00:59:59 smithi189 ceph-mon[36187]: pgmap v1803: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T00:59:59.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 00:59:59 smithi193 ceph-mon[36878]: pgmap v1803: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:00:00.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:00:00 smithi118 ceph-mon[31535]: overall HEALTH_WARN 6 failed cephadm daemon(s); 1 MDSs report slow requests 2024-03-24T01:00:00.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:00:00 smithi189 ceph-mon[36187]: overall HEALTH_WARN 6 failed cephadm daemon(s); 1 MDSs report slow requests 2024-03-24T01:00:00.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:00:00 smithi193 ceph-mon[36878]: overall HEALTH_WARN 6 failed cephadm daemon(s); 1 MDSs report slow requests 2024-03-24T01:00:01.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:00:01 smithi118 ceph-mon[31535]: pgmap v1804: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-24T01:00:01.447 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:00:01 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 2096.644258 secs 2024-03-24T01:00:01.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:00:01 smithi189 ceph-mon[36187]: pgmap v1804: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-24T01:00:01.455 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:00:01 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 2096.644258 secs 2024-03-24T01:00:01.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:00:01 smithi193 ceph-mon[36878]: pgmap v1804: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-24T01:00:01.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:00:01 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 2096.644258 secs 2024-03-24T01:00:03.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:00:03 smithi118 ceph-mon[31535]: pgmap v1805: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:00:03.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:00:03 smithi189 ceph-mon[36187]: pgmap v1805: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:00:03.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:00:03 smithi193 ceph-mon[36878]: pgmap v1805: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:00:05.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:00:05 smithi118 ceph-mon[31535]: pgmap v1806: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T01:00:05.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:00:05 smithi189 ceph-mon[36187]: pgmap v1806: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T01:00:05.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:00:05 smithi193 ceph-mon[36878]: pgmap v1806: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T01:00:06.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:00:06 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 2101.644320 secs 2024-03-24T01:00:06.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:00:06 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 2101.644320 secs 2024-03-24T01:00:06.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:00:06 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 2101.644320 secs 2024-03-24T01:00:07.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:00:07 smithi118 ceph-mon[31535]: pgmap v1807: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T01:00:07.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:00:07 smithi189 ceph-mon[36187]: pgmap v1807: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T01:00:07.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:00:07 smithi193 ceph-mon[36878]: pgmap v1807: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T01:00:09.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:00:09 smithi118 ceph-mon[31535]: pgmap v1808: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:00:09.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:00:09 smithi193 ceph-mon[36878]: pgmap v1808: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:00:09.704 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:00:09 smithi189 ceph-mon[36187]: pgmap v1808: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:00:11.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:00:11 smithi118 ceph-mon[31535]: pgmap v1809: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-24T01:00:11.697 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:00:11 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 2106.644384 secs 2024-03-24T01:00:11.704 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:00:11 smithi189 ceph-mon[36187]: pgmap v1809: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-24T01:00:11.704 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:00:11 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 2106.644384 secs 2024-03-24T01:00:11.711 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:00:11 smithi193 ceph-mon[36878]: pgmap v1809: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-24T01:00:11.711 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:00:11 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 2106.644384 secs 2024-03-24T01:00:13.698 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:00:13 smithi118 ceph-mon[31535]: pgmap v1810: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:00:13.704 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:00:13 smithi189 ceph-mon[36187]: pgmap v1810: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:00:13.711 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:00:13 smithi193 ceph-mon[36878]: pgmap v1810: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:00:15.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:00:15 smithi118 ceph-mon[31535]: pgmap v1811: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T01:00:15.704 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:00:15 smithi189 ceph-mon[36187]: pgmap v1811: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T01:00:15.713 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:00:15 smithi193 ceph-mon[36878]: pgmap v1811: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T01:00:16.612 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:00:16 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 2111.644443 secs 2024-03-24T01:00:16.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:00:16 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 2111.644443 secs 2024-03-24T01:00:16.704 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:00:16 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 2111.644443 secs 2024-03-24T01:00:17.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:00:17 smithi118 ceph-mon[31535]: pgmap v1812: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T01:00:17.704 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:00:17 smithi189 ceph-mon[36187]: pgmap v1812: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T01:00:17.711 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:00:17 smithi193 ceph-mon[36878]: pgmap v1812: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T01:00:18.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:00:18 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T01:00:18.704 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:00:18 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T01:00:18.711 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:00:18 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T01:00:19.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:00:19 smithi118 ceph-mon[31535]: pgmap v1813: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:00:19.697 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:00:19 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T01:00:19.697 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:00:19 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T01:00:19.697 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:00:19 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T01:00:19.704 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:00:19 smithi189 ceph-mon[36187]: pgmap v1813: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:00:19.704 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:00:19 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T01:00:19.704 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:00:19 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T01:00:19.704 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:00:19 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T01:00:19.711 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:00:19 smithi193 ceph-mon[36878]: pgmap v1813: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:00:19.711 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:00:19 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T01:00:19.711 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:00:19 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T01:00:19.711 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:00:19 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T01:00:20.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:00:20 smithi118 ceph-mon[31535]: pgmap v1814: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 6 op/s 2024-03-24T01:00:20.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:00:20 smithi118 ceph-mon[31535]: pgmap v1815: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 4 op/s 2024-03-24T01:00:20.704 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:00:20 smithi189 ceph-mon[36187]: pgmap v1814: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 6 op/s 2024-03-24T01:00:20.704 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:00:20 smithi189 ceph-mon[36187]: pgmap v1815: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 4 op/s 2024-03-24T01:00:20.711 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:00:20 smithi193 ceph-mon[36878]: pgmap v1814: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 6 op/s 2024-03-24T01:00:20.711 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:00:20 smithi193 ceph-mon[36878]: pgmap v1815: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 4 op/s 2024-03-24T01:00:21.605 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:00:21 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 2116.644528 secs 2024-03-24T01:00:21.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:00:21 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 2116.644528 secs 2024-03-24T01:00:21.711 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:00:21 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 2116.644528 secs 2024-03-24T01:00:22.617 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:00:22 smithi189 ceph-mon[36187]: pgmap v1816: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-03-24T01:00:22.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:00:22 smithi118 ceph-mon[31535]: pgmap v1816: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-03-24T01:00:22.711 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:00:22 smithi193 ceph-mon[36878]: pgmap v1816: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-03-24T01:00:24.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:00:24 smithi118 ceph-mon[31535]: pgmap v1817: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 5 op/s 2024-03-24T01:00:24.704 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:00:24 smithi189 ceph-mon[36187]: pgmap v1817: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 5 op/s 2024-03-24T01:00:24.712 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:00:24 smithi193 ceph-mon[36878]: pgmap v1817: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 5 op/s 2024-03-24T01:00:26.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:00:26 smithi118 ceph-mon[31535]: pgmap v1818: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-03-24T01:00:26.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:00:26 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 2121.644624 secs 2024-03-24T01:00:26.704 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:00:26 smithi189 ceph-mon[36187]: pgmap v1818: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-03-24T01:00:26.704 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:00:26 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 2121.644624 secs 2024-03-24T01:00:26.711 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:00:26 smithi193 ceph-mon[36878]: pgmap v1818: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-03-24T01:00:26.711 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:00:26 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 2121.644624 secs 2024-03-24T01:00:28.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:00:28 smithi118 ceph-mon[31535]: pgmap v1819: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-03-24T01:00:28.704 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:00:28 smithi189 ceph-mon[36187]: pgmap v1819: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-03-24T01:00:28.711 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:00:28 smithi193 ceph-mon[36878]: pgmap v1819: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-03-24T01:00:30.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:00:30 smithi118 ceph-mon[31535]: pgmap v1820: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 5 op/s 2024-03-24T01:00:30.704 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:00:30 smithi189 ceph-mon[36187]: pgmap v1820: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 5 op/s 2024-03-24T01:00:30.711 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:00:30 smithi193 ceph-mon[36878]: pgmap v1820: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 5 op/s 2024-03-24T01:00:31.697 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:00:31 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 2126.644712 secs 2024-03-24T01:00:31.704 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:00:31 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 2126.644712 secs 2024-03-24T01:00:31.711 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:00:31 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 2126.644712 secs 2024-03-24T01:00:32.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:00:32 smithi118 ceph-mon[31535]: pgmap v1821: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T01:00:32.704 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:00:32 smithi189 ceph-mon[36187]: pgmap v1821: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T01:00:32.711 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:00:32 smithi193 ceph-mon[36878]: pgmap v1821: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T01:00:34.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:00:34 smithi118 ceph-mon[31535]: pgmap v1822: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 5 op/s 2024-03-24T01:00:34.704 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:00:34 smithi189 ceph-mon[36187]: pgmap v1822: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 5 op/s 2024-03-24T01:00:34.711 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:00:34 smithi193 ceph-mon[36878]: pgmap v1822: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 5 op/s 2024-03-24T01:00:36.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:00:36 smithi118 ceph-mon[31535]: pgmap v1823: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T01:00:36.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:00:36 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 2131.644800 secs 2024-03-24T01:00:36.704 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:00:36 smithi189 ceph-mon[36187]: pgmap v1823: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T01:00:36.704 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:00:36 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 2131.644800 secs 2024-03-24T01:00:36.711 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:00:36 smithi193 ceph-mon[36878]: pgmap v1823: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T01:00:36.711 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:00:36 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 2131.644800 secs 2024-03-24T01:00:38.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:00:38 smithi118 ceph-mon[31535]: pgmap v1824: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:00:38.704 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:00:38 smithi189 ceph-mon[36187]: pgmap v1824: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:00:38.711 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:00:38 smithi193 ceph-mon[36878]: pgmap v1824: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:00:40.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:00:40 smithi118 ceph-mon[31535]: pgmap v1825: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T01:00:40.704 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:00:40 smithi189 ceph-mon[36187]: pgmap v1825: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T01:00:40.711 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:00:40 smithi193 ceph-mon[36878]: pgmap v1825: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T01:00:41.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:00:41 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 2136.644889 secs 2024-03-24T01:00:41.704 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:00:41 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 2136.644889 secs 2024-03-24T01:00:41.711 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:00:41 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 2136.644889 secs 2024-03-24T01:00:42.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:00:42 smithi118 ceph-mon[31535]: pgmap v1826: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T01:00:42.704 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:00:42 smithi189 ceph-mon[36187]: pgmap v1826: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T01:00:42.711 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:00:42 smithi193 ceph-mon[36878]: pgmap v1826: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T01:00:44.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:00:44 smithi118 ceph-mon[31535]: pgmap v1827: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 5 op/s 2024-03-24T01:00:44.704 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:00:44 smithi189 ceph-mon[36187]: pgmap v1827: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 5 op/s 2024-03-24T01:00:44.711 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:00:44 smithi193 ceph-mon[36878]: pgmap v1827: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 5 op/s 2024-03-24T01:00:46.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:00:46 smithi118 ceph-mon[31535]: pgmap v1828: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T01:00:46.697 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:00:46 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 2141.644977 secs 2024-03-24T01:00:46.704 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:00:46 smithi189 ceph-mon[36187]: pgmap v1828: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T01:00:46.704 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:00:46 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 2141.644977 secs 2024-03-24T01:00:46.711 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:00:46 smithi193 ceph-mon[36878]: pgmap v1828: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T01:00:46.711 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:00:46 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 2141.644977 secs 2024-03-24T01:00:48.697 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:00:48 smithi118 ceph-mon[31535]: pgmap v1829: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:00:48.704 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:00:48 smithi189 ceph-mon[36187]: pgmap v1829: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:00:48.711 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:00:48 smithi193 ceph-mon[36878]: pgmap v1829: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:00:50.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:00:50 smithi118 ceph-mon[31535]: pgmap v1830: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T01:00:50.704 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:00:50 smithi189 ceph-mon[36187]: pgmap v1830: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T01:00:50.711 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:00:50 smithi193 ceph-mon[36878]: pgmap v1830: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T01:00:51.605 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:00:51 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 2146.645066 secs 2024-03-24T01:00:51.606 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:00:51 smithi189 ceph-mon[36187]: pgmap v1831: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T01:00:51.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:00:51 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 2146.645066 secs 2024-03-24T01:00:51.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:00:51 smithi118 ceph-mon[31535]: pgmap v1831: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T01:00:51.711 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:00:51 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 2146.645066 secs 2024-03-24T01:00:51.711 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:00:51 smithi193 ceph-mon[36878]: pgmap v1831: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T01:00:54.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:00:53 smithi118 ceph-mon[31535]: pgmap v1832: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 5 op/s 2024-03-24T01:00:54.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:00:53 smithi189 ceph-mon[36187]: pgmap v1832: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 5 op/s 2024-03-24T01:00:54.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:00:53 smithi193 ceph-mon[36878]: pgmap v1832: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 5 op/s 2024-03-24T01:00:56.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:00:55 smithi118 ceph-mon[31535]: pgmap v1833: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T01:00:56.197 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:00:55 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 2151.645151 secs 2024-03-24T01:00:56.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:00:55 smithi189 ceph-mon[36187]: pgmap v1833: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T01:00:56.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:00:55 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 2151.645151 secs 2024-03-24T01:00:56.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:00:55 smithi193 ceph-mon[36878]: pgmap v1833: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T01:00:56.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:00:55 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 2151.645151 secs 2024-03-24T01:00:58.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:00:57 smithi118 ceph-mon[31535]: pgmap v1834: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:00:58.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:00:57 smithi189 ceph-mon[36187]: pgmap v1834: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:00:58.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:00:57 smithi193 ceph-mon[36878]: pgmap v1834: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:01:00.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:00:59 smithi118 ceph-mon[31535]: pgmap v1835: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T01:01:00.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:00:59 smithi189 ceph-mon[36187]: pgmap v1835: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T01:01:00.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:00:59 smithi193 ceph-mon[36878]: pgmap v1835: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T01:01:01.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:01:00 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 2156.645214 secs 2024-03-24T01:01:01.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:01:00 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 2156.645214 secs 2024-03-24T01:01:01.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:01:00 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 2156.645214 secs 2024-03-24T01:01:02.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:01:01 smithi118 ceph-mon[31535]: pgmap v1836: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T01:01:02.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:01:01 smithi189 ceph-mon[36187]: pgmap v1836: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T01:01:02.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:01:01 smithi193 ceph-mon[36878]: pgmap v1836: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T01:01:04.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:01:03 smithi118 ceph-mon[31535]: pgmap v1837: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 5 op/s 2024-03-24T01:01:04.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:01:03 smithi189 ceph-mon[36187]: pgmap v1837: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 5 op/s 2024-03-24T01:01:04.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:01:03 smithi193 ceph-mon[36878]: pgmap v1837: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 5 op/s 2024-03-24T01:01:06.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:01:05 smithi189 ceph-mon[36187]: pgmap v1838: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T01:01:06.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:01:05 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 2161.645302 secs 2024-03-24T01:01:06.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:01:05 smithi193 ceph-mon[36878]: pgmap v1838: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T01:01:06.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:01:05 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 2161.645302 secs 2024-03-24T01:01:06.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:01:05 smithi118 ceph-mon[31535]: pgmap v1838: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T01:01:06.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:01:05 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 2161.645302 secs 2024-03-24T01:01:08.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:01:07 smithi189 ceph-mon[36187]: pgmap v1839: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:01:08.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:01:07 smithi193 ceph-mon[36878]: pgmap v1839: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:01:08.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:01:07 smithi118 ceph-mon[31535]: pgmap v1839: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:01:10.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:01:09 smithi118 ceph-mon[31535]: pgmap v1840: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T01:01:10.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:01:09 smithi193 ceph-mon[36878]: pgmap v1840: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T01:01:10.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:01:09 smithi189 ceph-mon[36187]: pgmap v1840: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T01:01:11.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:01:10 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 2166.645368 secs 2024-03-24T01:01:11.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:01:10 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 2166.645368 secs 2024-03-24T01:01:11.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:01:10 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 2166.645368 secs 2024-03-24T01:01:12.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:01:11 smithi118 ceph-mon[31535]: pgmap v1841: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T01:01:12.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:01:11 smithi189 ceph-mon[36187]: pgmap v1841: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T01:01:12.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:01:11 smithi193 ceph-mon[36878]: pgmap v1841: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T01:01:14.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:01:13 smithi118 ceph-mon[31535]: pgmap v1842: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 5 op/s 2024-03-24T01:01:14.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:01:13 smithi189 ceph-mon[36187]: pgmap v1842: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 5 op/s 2024-03-24T01:01:14.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:01:13 smithi193 ceph-mon[36878]: pgmap v1842: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 5 op/s 2024-03-24T01:01:16.074 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T01:01:16.074+0000 7f81ee000640 1 -- 172.21.15.118:0/1693767697 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f81c80065b0 con 0x7f81f81052d0 2024-03-24T01:01:16.075 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T01:01:16.074+0000 7f81ef400640 1 -- 172.21.15.118:0/1693767697 <== mon.2 v2:172.21.15.189:3300/0 21 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (secure 0 0 0) 0x7f81c80065b0 con 0x7f81f81052d0 2024-03-24T01:01:16.078 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T01:01:16.078+0000 7f81db400640 1 -- 172.21.15.118:0/659686424 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f81c4006300 con 0x5622651a13f0 2024-03-24T01:01:16.079 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T01:01:16.079+0000 7f81e4c00640 1 -- 172.21.15.118:0/659686424 <== mon.1 v2:172.21.15.193:3300/0 21 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 409+0+0 (secure 0 0 0) 0x7f81c4006300 con 0x5622651a13f0 2024-03-24T01:01:16.262 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:01:15 smithi193 ceph-mon[36878]: pgmap v1843: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T01:01:16.262 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:01:15 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 2171.645457 secs 2024-03-24T01:01:16.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:01:15 smithi118 ceph-mon[31535]: pgmap v1843: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T01:01:16.447 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:01:15 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 2171.645457 secs 2024-03-24T01:01:16.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:01:15 smithi189 ceph-mon[36187]: pgmap v1843: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T01:01:16.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:01:15 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 2171.645457 secs 2024-03-24T01:01:18.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:01:17 smithi118 ceph-mon[31535]: pgmap v1844: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:01:18.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:01:17 smithi189 ceph-mon[36187]: pgmap v1844: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:01:18.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:01:17 smithi193 ceph-mon[36878]: pgmap v1844: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:01:20.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:01:19 smithi118 ceph-mon[31535]: pgmap v1845: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T01:01:20.447 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:01:19 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T01:01:20.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:01:19 smithi189 ceph-mon[36187]: pgmap v1845: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T01:01:20.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:01:19 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T01:01:20.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:01:19 smithi193 ceph-mon[36878]: pgmap v1845: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T01:01:20.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:01:19 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T01:01:21.355 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:01:20 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 2176.645541 secs 2024-03-24T01:01:21.355 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:01:20 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T01:01:21.355 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:01:20 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T01:01:21.356 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:01:20 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T01:01:21.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:01:20 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 2176.645541 secs 2024-03-24T01:01:21.447 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:01:20 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T01:01:21.447 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:01:20 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T01:01:21.447 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:01:20 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T01:01:21.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:01:20 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 2176.645541 secs 2024-03-24T01:01:21.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:01:20 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T01:01:21.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:01:20 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T01:01:21.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:01:20 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T01:01:22.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:01:21 smithi118 ceph-mon[31535]: pgmap v1846: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 6 op/s 2024-03-24T01:01:22.447 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:01:21 smithi118 ceph-mon[31535]: pgmap v1847: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 6 op/s 2024-03-24T01:01:22.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:01:21 smithi189 ceph-mon[36187]: pgmap v1846: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 6 op/s 2024-03-24T01:01:22.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:01:21 smithi189 ceph-mon[36187]: pgmap v1847: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 6 op/s 2024-03-24T01:01:22.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:01:21 smithi193 ceph-mon[36878]: pgmap v1846: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 6 op/s 2024-03-24T01:01:22.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:01:21 smithi193 ceph-mon[36878]: pgmap v1847: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 6 op/s 2024-03-24T01:01:24.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:01:24 smithi118 ceph-mon[31535]: pgmap v1848: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 5 op/s 2024-03-24T01:01:24.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:01:24 smithi189 ceph-mon[36187]: pgmap v1848: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 5 op/s 2024-03-24T01:01:24.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:01:24 smithi193 ceph-mon[36878]: pgmap v1848: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 5 op/s 2024-03-24T01:01:26.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:01:26 smithi118 ceph-mon[31535]: pgmap v1849: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 6 op/s 2024-03-24T01:01:26.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:01:26 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 2181.645629 secs 2024-03-24T01:01:26.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:01:26 smithi189 ceph-mon[36187]: pgmap v1849: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 6 op/s 2024-03-24T01:01:26.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:01:26 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 2181.645629 secs 2024-03-24T01:01:26.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:01:26 smithi193 ceph-mon[36878]: pgmap v1849: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 6 op/s 2024-03-24T01:01:26.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:01:26 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 2181.645629 secs 2024-03-24T01:01:28.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:01:28 smithi118 ceph-mon[31535]: pgmap v1850: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 6 op/s 2024-03-24T01:01:28.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:01:28 smithi189 ceph-mon[36187]: pgmap v1850: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 6 op/s 2024-03-24T01:01:28.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:01:28 smithi193 ceph-mon[36878]: pgmap v1850: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 6 op/s 2024-03-24T01:01:30.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:01:30 smithi118 ceph-mon[31535]: pgmap v1851: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 6 op/s 2024-03-24T01:01:30.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:01:30 smithi189 ceph-mon[36187]: pgmap v1851: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 6 op/s 2024-03-24T01:01:30.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:01:30 smithi193 ceph-mon[36878]: pgmap v1851: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 6 op/s 2024-03-24T01:01:31.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:01:31 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 2186.645715 secs 2024-03-24T01:01:31.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:01:31 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 2186.645715 secs 2024-03-24T01:01:31.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:01:31 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 2186.645715 secs 2024-03-24T01:01:32.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:01:32 smithi118 ceph-mon[31535]: pgmap v1852: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 5 op/s 2024-03-24T01:01:32.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:01:32 smithi189 ceph-mon[36187]: pgmap v1852: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 5 op/s 2024-03-24T01:01:32.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:01:32 smithi193 ceph-mon[36878]: pgmap v1852: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 5 op/s 2024-03-24T01:01:34.450 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:01:34 smithi118 ceph-mon[31535]: pgmap v1853: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:01:34.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:01:34 smithi189 ceph-mon[36187]: pgmap v1853: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:01:34.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:01:34 smithi193 ceph-mon[36878]: pgmap v1853: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:01:36.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:01:36 smithi118 ceph-mon[31535]: pgmap v1854: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-24T01:01:36.447 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:01:36 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 2191.645804 secs 2024-03-24T01:01:36.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:01:36 smithi189 ceph-mon[36187]: pgmap v1854: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-24T01:01:36.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:01:36 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 2191.645804 secs 2024-03-24T01:01:36.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:01:36 smithi193 ceph-mon[36878]: pgmap v1854: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-24T01:01:36.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:01:36 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 2191.645804 secs 2024-03-24T01:01:38.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:01:38 smithi118 ceph-mon[31535]: pgmap v1855: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:01:38.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:01:38 smithi189 ceph-mon[36187]: pgmap v1855: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:01:38.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:01:38 smithi193 ceph-mon[36878]: pgmap v1855: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:01:40.447 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:01:40 smithi118 ceph-mon[31535]: pgmap v1856: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T01:01:40.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:01:40 smithi189 ceph-mon[36187]: pgmap v1856: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T01:01:40.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:01:40 smithi193 ceph-mon[36878]: pgmap v1856: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T01:01:41.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:01:41 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 2196.645865 secs 2024-03-24T01:01:41.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:01:41 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 2196.645865 secs 2024-03-24T01:01:41.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:01:41 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 2196.645865 secs 2024-03-24T01:01:42.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:01:42 smithi118 ceph-mon[31535]: pgmap v1857: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T01:01:42.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:01:42 smithi189 ceph-mon[36187]: pgmap v1857: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T01:01:42.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:01:42 smithi193 ceph-mon[36878]: pgmap v1857: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T01:01:44.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:01:44 smithi118 ceph-mon[31535]: pgmap v1858: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:01:44.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:01:44 smithi189 ceph-mon[36187]: pgmap v1858: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:01:44.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:01:44 smithi193 ceph-mon[36878]: pgmap v1858: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:01:46.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:01:46 smithi118 ceph-mon[31535]: pgmap v1859: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-24T01:01:46.447 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:01:46 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 2201.645953 secs 2024-03-24T01:01:46.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:01:46 smithi189 ceph-mon[36187]: pgmap v1859: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-24T01:01:46.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:01:46 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 2201.645953 secs 2024-03-24T01:01:46.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:01:46 smithi193 ceph-mon[36878]: pgmap v1859: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-24T01:01:46.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:01:46 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 2201.645953 secs 2024-03-24T01:01:48.449 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:01:48 smithi118 ceph-mon[31535]: pgmap v1860: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:01:48.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:01:48 smithi189 ceph-mon[36187]: pgmap v1860: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:01:48.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:01:48 smithi193 ceph-mon[36878]: pgmap v1860: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:01:50.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:01:50 smithi118 ceph-mon[31535]: pgmap v1861: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T01:01:50.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:01:50 smithi189 ceph-mon[36187]: pgmap v1861: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T01:01:50.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:01:50 smithi193 ceph-mon[36878]: pgmap v1861: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T01:01:51.355 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:01:51 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 2206.645995 secs 2024-03-24T01:01:51.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:01:51 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 2206.645995 secs 2024-03-24T01:01:51.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:01:51 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 2206.645995 secs 2024-03-24T01:01:52.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:01:52 smithi118 ceph-mon[31535]: pgmap v1862: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T01:01:52.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:01:52 smithi189 ceph-mon[36187]: pgmap v1862: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T01:01:52.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:01:52 smithi193 ceph-mon[36878]: pgmap v1862: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T01:01:54.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:01:54 smithi118 ceph-mon[31535]: pgmap v1863: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:01:54.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:01:54 smithi189 ceph-mon[36187]: pgmap v1863: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:01:54.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:01:54 smithi193 ceph-mon[36878]: pgmap v1863: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:01:56.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:01:56 smithi118 ceph-mon[31535]: pgmap v1864: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-24T01:01:56.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:01:56 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 2211.646085 secs 2024-03-24T01:01:56.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:01:56 smithi189 ceph-mon[36187]: pgmap v1864: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-24T01:01:56.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:01:56 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 2211.646085 secs 2024-03-24T01:01:56.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:01:56 smithi193 ceph-mon[36878]: pgmap v1864: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-24T01:01:56.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:01:56 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 2211.646085 secs 2024-03-24T01:01:58.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:01:58 smithi118 ceph-mon[31535]: pgmap v1865: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:01:58.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:01:58 smithi189 ceph-mon[36187]: pgmap v1865: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:01:58.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:01:58 smithi193 ceph-mon[36878]: pgmap v1865: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:02:00.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:02:00 smithi118 ceph-mon[31535]: pgmap v1866: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T01:02:00.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:02:00 smithi189 ceph-mon[36187]: pgmap v1866: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T01:02:00.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:02:00 smithi193 ceph-mon[36878]: pgmap v1866: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T01:02:01.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:02:01 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 2216.646174 secs 2024-03-24T01:02:01.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:02:01 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 2216.646174 secs 2024-03-24T01:02:01.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:02:01 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 2216.646174 secs 2024-03-24T01:02:02.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:02:02 smithi118 ceph-mon[31535]: pgmap v1867: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T01:02:02.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:02:02 smithi189 ceph-mon[36187]: pgmap v1867: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T01:02:02.460 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:02:02 smithi193 ceph-mon[36878]: pgmap v1867: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T01:02:04.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:02:04 smithi118 ceph-mon[31535]: pgmap v1868: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:02:04.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:02:04 smithi189 ceph-mon[36187]: pgmap v1868: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:02:04.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:02:04 smithi193 ceph-mon[36878]: pgmap v1868: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:02:06.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:02:06 smithi118 ceph-mon[31535]: pgmap v1869: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-24T01:02:06.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:02:06 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 2221.646260 secs 2024-03-24T01:02:06.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:02:06 smithi189 ceph-mon[36187]: pgmap v1869: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-24T01:02:06.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:02:06 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 2221.646260 secs 2024-03-24T01:02:06.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:02:06 smithi193 ceph-mon[36878]: pgmap v1869: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-24T01:02:06.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:02:06 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 2221.646260 secs 2024-03-24T01:02:08.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:02:08 smithi118 ceph-mon[31535]: pgmap v1870: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:02:08.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:02:08 smithi189 ceph-mon[36187]: pgmap v1870: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:02:08.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:02:08 smithi193 ceph-mon[36878]: pgmap v1870: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:02:10.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:02:10 smithi118 ceph-mon[31535]: pgmap v1871: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T01:02:10.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:02:10 smithi189 ceph-mon[36187]: pgmap v1871: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T01:02:10.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:02:10 smithi193 ceph-mon[36878]: pgmap v1871: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T01:02:11.348 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:02:11 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 2226.646346 secs 2024-03-24T01:02:11.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:02:11 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 2226.646346 secs 2024-03-24T01:02:11.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:02:11 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 2226.646346 secs 2024-03-24T01:02:12.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:02:12 smithi118 ceph-mon[31535]: pgmap v1872: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T01:02:12.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:02:12 smithi189 ceph-mon[36187]: pgmap v1872: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T01:02:12.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:02:12 smithi193 ceph-mon[36878]: pgmap v1872: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T01:02:14.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:02:14 smithi118 ceph-mon[31535]: pgmap v1873: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:02:14.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:02:14 smithi189 ceph-mon[36187]: pgmap v1873: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:02:14.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:02:14 smithi193 ceph-mon[36878]: pgmap v1873: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:02:16.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:02:16 smithi118 ceph-mon[31535]: pgmap v1874: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-24T01:02:16.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:02:16 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 2231.646435 secs 2024-03-24T01:02:16.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:02:16 smithi189 ceph-mon[36187]: pgmap v1874: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-24T01:02:16.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:02:16 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 2231.646435 secs 2024-03-24T01:02:16.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:02:16 smithi193 ceph-mon[36878]: pgmap v1874: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-24T01:02:16.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:02:16 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 2231.646435 secs 2024-03-24T01:02:18.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:02:18 smithi118 ceph-mon[31535]: pgmap v1875: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:02:18.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:02:18 smithi189 ceph-mon[36187]: pgmap v1875: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:02:18.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:02:18 smithi193 ceph-mon[36878]: pgmap v1875: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:02:20.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:02:20 smithi118 ceph-mon[31535]: pgmap v1876: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T01:02:20.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:02:20 smithi189 ceph-mon[36187]: pgmap v1876: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T01:02:20.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:02:20 smithi193 ceph-mon[36878]: pgmap v1876: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T01:02:21.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:02:21 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 2236.646495 secs 2024-03-24T01:02:21.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:02:21 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T01:02:21.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:02:21 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 2236.646495 secs 2024-03-24T01:02:21.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:02:21 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T01:02:21.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:02:21 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 2236.646495 secs 2024-03-24T01:02:21.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:02:21 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T01:02:22.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:02:22 smithi118 ceph-mon[31535]: pgmap v1877: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T01:02:22.447 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:02:22 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T01:02:22.447 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:02:22 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T01:02:22.447 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:02:22 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T01:02:22.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:02:22 smithi189 ceph-mon[36187]: pgmap v1877: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T01:02:22.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:02:22 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T01:02:22.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:02:22 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T01:02:22.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:02:22 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T01:02:22.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:02:22 smithi193 ceph-mon[36878]: pgmap v1877: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T01:02:22.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:02:22 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T01:02:22.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:02:22 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T01:02:22.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:02:22 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T01:02:23.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:02:23 smithi118 ceph-mon[31535]: pgmap v1878: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 5 op/s 2024-03-24T01:02:23.447 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:02:23 smithi118 ceph-mon[31535]: pgmap v1879: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-03-24T01:02:23.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:02:23 smithi189 ceph-mon[36187]: pgmap v1878: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 5 op/s 2024-03-24T01:02:23.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:02:23 smithi189 ceph-mon[36187]: pgmap v1879: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-03-24T01:02:23.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:02:23 smithi193 ceph-mon[36878]: pgmap v1878: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 5 op/s 2024-03-24T01:02:23.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:02:23 smithi193 ceph-mon[36878]: pgmap v1879: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-03-24T01:02:25.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:02:25 smithi118 ceph-mon[31535]: pgmap v1880: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 4 op/s 2024-03-24T01:02:25.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:02:25 smithi189 ceph-mon[36187]: pgmap v1880: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 4 op/s 2024-03-24T01:02:25.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:02:25 smithi193 ceph-mon[36878]: pgmap v1880: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 4 op/s 2024-03-24T01:02:26.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:02:26 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 2241.646539 secs 2024-03-24T01:02:26.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:02:26 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 2241.646539 secs 2024-03-24T01:02:26.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:02:26 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 2241.646539 secs 2024-03-24T01:02:27.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:02:27 smithi193 ceph-mon[36878]: pgmap v1881: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-03-24T01:02:27.697 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:02:27 smithi118 ceph-mon[31535]: pgmap v1881: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-03-24T01:02:27.704 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:02:27 smithi189 ceph-mon[36187]: pgmap v1881: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-03-24T01:02:29.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:02:29 smithi193 ceph-mon[36878]: pgmap v1882: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 5 op/s 2024-03-24T01:02:29.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:02:29 smithi118 ceph-mon[31535]: pgmap v1882: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 5 op/s 2024-03-24T01:02:29.704 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:02:29 smithi189 ceph-mon[36187]: pgmap v1882: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 5 op/s 2024-03-24T01:02:31.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:02:31 smithi118 ceph-mon[31535]: pgmap v1883: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-03-24T01:02:31.697 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:02:31 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 2246.646627 secs 2024-03-24T01:02:31.704 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:02:31 smithi189 ceph-mon[36187]: pgmap v1883: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-03-24T01:02:31.704 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:02:31 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 2246.646627 secs 2024-03-24T01:02:31.711 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:02:31 smithi193 ceph-mon[36878]: pgmap v1883: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-03-24T01:02:31.711 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:02:31 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 2246.646627 secs 2024-03-24T01:02:33.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:02:33 smithi118 ceph-mon[31535]: pgmap v1884: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 5 op/s 2024-03-24T01:02:33.704 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:02:33 smithi189 ceph-mon[36187]: pgmap v1884: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 5 op/s 2024-03-24T01:02:33.711 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:02:33 smithi193 ceph-mon[36878]: pgmap v1884: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 5 op/s 2024-03-24T01:02:35.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:02:35 smithi118 ceph-mon[31535]: pgmap v1885: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T01:02:35.704 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:02:35 smithi189 ceph-mon[36187]: pgmap v1885: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T01:02:35.710 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:02:35 smithi193 ceph-mon[36878]: pgmap v1885: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T01:02:36.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:02:36 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 2251.646713 secs 2024-03-24T01:02:36.704 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:02:36 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 2251.646713 secs 2024-03-24T01:02:36.711 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:02:36 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 2251.646713 secs 2024-03-24T01:02:37.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:02:37 smithi118 ceph-mon[31535]: pgmap v1886: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T01:02:37.704 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:02:37 smithi189 ceph-mon[36187]: pgmap v1886: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T01:02:37.711 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:02:37 smithi193 ceph-mon[36878]: pgmap v1886: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T01:02:39.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:02:39 smithi118 ceph-mon[31535]: pgmap v1887: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 5 op/s 2024-03-24T01:02:39.704 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:02:39 smithi189 ceph-mon[36187]: pgmap v1887: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 5 op/s 2024-03-24T01:02:39.711 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:02:39 smithi193 ceph-mon[36878]: pgmap v1887: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 5 op/s 2024-03-24T01:02:41.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:02:41 smithi118 ceph-mon[31535]: pgmap v1888: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T01:02:41.697 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:02:41 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 2256.646799 secs 2024-03-24T01:02:41.704 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:02:41 smithi189 ceph-mon[36187]: pgmap v1888: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T01:02:41.704 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:02:41 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 2256.646799 secs 2024-03-24T01:02:41.711 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:02:41 smithi193 ceph-mon[36878]: pgmap v1888: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T01:02:41.711 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:02:41 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 2256.646799 secs 2024-03-24T01:02:43.698 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:02:43 smithi118 ceph-mon[31535]: pgmap v1889: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:02:43.704 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:02:43 smithi189 ceph-mon[36187]: pgmap v1889: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:02:43.711 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:02:43 smithi193 ceph-mon[36878]: pgmap v1889: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:02:45.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:02:45 smithi118 ceph-mon[31535]: pgmap v1890: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-03-24T01:02:45.704 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:02:45 smithi189 ceph-mon[36187]: pgmap v1890: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-03-24T01:02:45.711 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:02:45 smithi193 ceph-mon[36878]: pgmap v1890: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-03-24T01:02:46.612 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:02:46 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 2261.646888 secs 2024-03-24T01:02:46.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:02:46 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 2261.646888 secs 2024-03-24T01:02:46.704 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:02:46 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 2261.646888 secs 2024-03-24T01:02:47.697 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:02:47 smithi118 ceph-mon[31535]: pgmap v1891: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 6 op/s 2024-03-24T01:02:47.704 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:02:47 smithi189 ceph-mon[36187]: pgmap v1891: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 6 op/s 2024-03-24T01:02:47.711 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:02:47 smithi193 ceph-mon[36878]: pgmap v1891: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 6 op/s 2024-03-24T01:02:49.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:02:49 smithi118 ceph-mon[31535]: pgmap v1892: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 5 op/s 2024-03-24T01:02:49.704 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:02:49 smithi189 ceph-mon[36187]: pgmap v1892: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 5 op/s 2024-03-24T01:02:49.711 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:02:49 smithi193 ceph-mon[36878]: pgmap v1892: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 5 op/s 2024-03-24T01:02:51.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:02:51 smithi118 ceph-mon[31535]: pgmap v1893: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 6 op/s 2024-03-24T01:02:51.697 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:02:51 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 2266.646974 secs 2024-03-24T01:02:51.704 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:02:51 smithi189 ceph-mon[36187]: pgmap v1893: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 6 op/s 2024-03-24T01:02:51.704 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:02:51 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 2266.646974 secs 2024-03-24T01:02:51.711 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:02:51 smithi193 ceph-mon[36878]: pgmap v1893: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 6 op/s 2024-03-24T01:02:51.711 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:02:51 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 2266.646974 secs 2024-03-24T01:02:53.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:02:53 smithi118 ceph-mon[31535]: pgmap v1894: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-03-24T01:02:53.704 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:02:53 smithi189 ceph-mon[36187]: pgmap v1894: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-03-24T01:02:53.711 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:02:53 smithi193 ceph-mon[36878]: pgmap v1894: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-03-24T01:02:55.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:02:55 smithi118 ceph-mon[31535]: pgmap v1895: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-03-24T01:02:55.704 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:02:55 smithi189 ceph-mon[36187]: pgmap v1895: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-03-24T01:02:55.711 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:02:55 smithi193 ceph-mon[36878]: pgmap v1895: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-03-24T01:02:56.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:02:56 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 2271.647066 secs 2024-03-24T01:02:56.704 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:02:56 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 2271.647066 secs 2024-03-24T01:02:56.711 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:02:56 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 2271.647066 secs 2024-03-24T01:02:57.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:02:57 smithi118 ceph-mon[31535]: pgmap v1896: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T01:02:57.704 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:02:57 smithi189 ceph-mon[36187]: pgmap v1896: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T01:02:57.711 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:02:57 smithi193 ceph-mon[36878]: pgmap v1896: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T01:02:59.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:02:59 smithi118 ceph-mon[31535]: pgmap v1897: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 5 op/s 2024-03-24T01:02:59.704 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:02:59 smithi189 ceph-mon[36187]: pgmap v1897: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 5 op/s 2024-03-24T01:02:59.710 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:02:59 smithi193 ceph-mon[36878]: pgmap v1897: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 5 op/s 2024-03-24T01:03:01.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:03:01 smithi118 ceph-mon[31535]: pgmap v1898: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T01:03:01.697 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:03:01 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 2276.647097 secs 2024-03-24T01:03:01.704 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:03:01 smithi189 ceph-mon[36187]: pgmap v1898: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T01:03:01.704 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:03:01 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 2276.647097 secs 2024-03-24T01:03:01.711 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:03:01 smithi193 ceph-mon[36878]: pgmap v1898: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T01:03:01.711 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:03:01 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 2276.647097 secs 2024-03-24T01:03:03.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:03:03 smithi118 ceph-mon[31535]: pgmap v1899: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:03:03.704 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:03:03 smithi189 ceph-mon[36187]: pgmap v1899: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:03:03.710 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:03:03 smithi193 ceph-mon[36878]: pgmap v1899: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:03:05.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:03:05 smithi118 ceph-mon[31535]: pgmap v1900: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T01:03:05.704 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:03:05 smithi189 ceph-mon[36187]: pgmap v1900: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T01:03:05.711 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:03:05 smithi193 ceph-mon[36878]: pgmap v1900: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T01:03:06.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:03:06 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 2281.647157 secs 2024-03-24T01:03:06.704 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:03:06 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 2281.647157 secs 2024-03-24T01:03:06.711 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:03:06 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 2281.647157 secs 2024-03-24T01:03:07.697 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:03:07 smithi118 ceph-mon[31535]: pgmap v1901: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T01:03:07.704 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:03:07 smithi189 ceph-mon[36187]: pgmap v1901: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T01:03:07.711 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:03:07 smithi193 ceph-mon[36878]: pgmap v1901: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T01:03:09.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:03:09 smithi118 ceph-mon[31535]: pgmap v1902: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 5 op/s 2024-03-24T01:03:09.704 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:03:09 smithi189 ceph-mon[36187]: pgmap v1902: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 5 op/s 2024-03-24T01:03:09.711 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:03:09 smithi193 ceph-mon[36878]: pgmap v1902: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 5 op/s 2024-03-24T01:03:11.697 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:03:11 smithi118 ceph-mon[31535]: pgmap v1903: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T01:03:11.697 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:03:11 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 2286.647219 secs 2024-03-24T01:03:11.704 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:03:11 smithi189 ceph-mon[36187]: pgmap v1903: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T01:03:11.704 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:03:11 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 2286.647219 secs 2024-03-24T01:03:11.711 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:03:11 smithi193 ceph-mon[36878]: pgmap v1903: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T01:03:11.711 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:03:11 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 2286.647219 secs 2024-03-24T01:03:13.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:03:13 smithi118 ceph-mon[31535]: pgmap v1904: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:03:13.704 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:03:13 smithi189 ceph-mon[36187]: pgmap v1904: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:03:13.711 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:03:13 smithi193 ceph-mon[36878]: pgmap v1904: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:03:15.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:03:15 smithi118 ceph-mon[31535]: pgmap v1905: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T01:03:15.704 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:03:15 smithi189 ceph-mon[36187]: pgmap v1905: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T01:03:15.711 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:03:15 smithi193 ceph-mon[36878]: pgmap v1905: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T01:03:16.612 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:03:16 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 2291.647277 secs 2024-03-24T01:03:16.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:03:16 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 2291.647277 secs 2024-03-24T01:03:16.704 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:03:16 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 2291.647277 secs 2024-03-24T01:03:17.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:03:17 smithi118 ceph-mon[31535]: pgmap v1906: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T01:03:17.704 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:03:17 smithi189 ceph-mon[36187]: pgmap v1906: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T01:03:17.710 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:03:17 smithi193 ceph-mon[36878]: pgmap v1906: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T01:03:19.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:03:19 smithi118 ceph-mon[31535]: pgmap v1907: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 5 op/s 2024-03-24T01:03:19.704 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:03:19 smithi189 ceph-mon[36187]: pgmap v1907: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 5 op/s 2024-03-24T01:03:19.710 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:03:19 smithi193 ceph-mon[36878]: pgmap v1907: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 5 op/s 2024-03-24T01:03:21.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:03:21 smithi118 ceph-mon[31535]: pgmap v1908: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T01:03:21.697 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:03:21 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 2296.647358 secs 2024-03-24T01:03:21.704 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:03:21 smithi189 ceph-mon[36187]: pgmap v1908: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T01:03:21.704 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:03:21 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 2296.647358 secs 2024-03-24T01:03:21.711 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:03:21 smithi193 ceph-mon[36878]: pgmap v1908: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T01:03:21.711 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:03:21 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 2296.647358 secs 2024-03-24T01:03:22.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:03:22 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T01:03:22.704 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:03:22 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T01:03:22.710 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:03:22 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T01:03:23.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:03:23 smithi118 ceph-mon[31535]: pgmap v1909: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:03:23.697 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:03:23 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-24T01:03:23.697 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:03:23 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-24T01:03:23.697 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:03:23 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-24T01:03:23.697 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:03:23 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-03-24T01:03:23.704 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:03:23 smithi189 ceph-mon[36187]: pgmap v1909: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:03:23.704 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:03:23 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-24T01:03:23.704 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:03:23 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-24T01:03:23.704 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:03:23 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-24T01:03:23.704 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:03:23 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-03-24T01:03:23.711 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:03:23 smithi193 ceph-mon[36878]: pgmap v1909: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:03:23.711 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:03:23 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-24T01:03:23.711 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:03:23 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-24T01:03:23.711 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:03:23 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-24T01:03:23.711 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:03:23 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-03-24T01:03:24.697 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:03:24 smithi118 ceph-mon[31535]: Adjusting osd_memory_target on smithi189 to 174.1M 2024-03-24T01:03:24.697 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:03:24 smithi118 ceph-mon[31535]: Unable to set osd_memory_target on smithi189 to 182635622: error parsing value: Value '182635622' is below minimum 939524096 2024-03-24T01:03:24.697 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:03:24 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-24T01:03:24.697 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:03:24 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-24T01:03:24.697 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:03:24 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-24T01:03:24.697 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:03:24 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-03-24T01:03:24.697 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:03:24 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T01:03:24.697 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:03:24 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T01:03:24.697 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:03:24 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T01:03:24.704 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:03:24 smithi189 ceph-mon[36187]: Adjusting osd_memory_target on smithi189 to 174.1M 2024-03-24T01:03:24.704 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:03:24 smithi189 ceph-mon[36187]: Unable to set osd_memory_target on smithi189 to 182635622: error parsing value: Value '182635622' is below minimum 939524096 2024-03-24T01:03:24.704 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:03:24 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-24T01:03:24.704 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:03:24 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-24T01:03:24.704 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:03:24 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-24T01:03:24.704 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:03:24 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-03-24T01:03:24.704 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:03:24 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T01:03:24.704 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:03:24 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T01:03:24.705 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:03:24 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T01:03:24.711 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:03:24 smithi193 ceph-mon[36878]: Adjusting osd_memory_target on smithi189 to 174.1M 2024-03-24T01:03:24.711 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:03:24 smithi193 ceph-mon[36878]: Unable to set osd_memory_target on smithi189 to 182635622: error parsing value: Value '182635622' is below minimum 939524096 2024-03-24T01:03:24.711 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:03:24 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-24T01:03:24.711 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:03:24 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-24T01:03:24.711 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:03:24 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-24T01:03:24.711 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:03:24 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-03-24T01:03:24.711 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:03:24 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T01:03:24.711 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:03:24 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T01:03:24.711 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:03:24 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T01:03:25.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:03:25 smithi118 ceph-mon[31535]: Adjusting osd_memory_target on smithi118 to 176.7M 2024-03-24T01:03:25.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:03:25 smithi118 ceph-mon[31535]: Unable to set osd_memory_target on smithi118 to 185365196: error parsing value: Value '185365196' is below minimum 939524096 2024-03-24T01:03:25.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:03:25 smithi118 ceph-mon[31535]: pgmap v1910: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 6 op/s 2024-03-24T01:03:25.697 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:03:25 smithi118 ceph-mon[31535]: pgmap v1911: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 5 op/s 2024-03-24T01:03:25.704 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:03:25 smithi189 ceph-mon[36187]: Adjusting osd_memory_target on smithi118 to 176.7M 2024-03-24T01:03:25.704 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:03:25 smithi189 ceph-mon[36187]: Unable to set osd_memory_target on smithi118 to 185365196: error parsing value: Value '185365196' is below minimum 939524096 2024-03-24T01:03:25.704 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:03:25 smithi189 ceph-mon[36187]: pgmap v1910: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 6 op/s 2024-03-24T01:03:25.704 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:03:25 smithi189 ceph-mon[36187]: pgmap v1911: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 5 op/s 2024-03-24T01:03:25.711 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:03:25 smithi193 ceph-mon[36878]: Adjusting osd_memory_target on smithi118 to 176.7M 2024-03-24T01:03:25.711 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:03:25 smithi193 ceph-mon[36878]: Unable to set osd_memory_target on smithi118 to 185365196: error parsing value: Value '185365196' is below minimum 939524096 2024-03-24T01:03:25.711 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:03:25 smithi193 ceph-mon[36878]: pgmap v1910: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 6 op/s 2024-03-24T01:03:25.711 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:03:25 smithi193 ceph-mon[36878]: pgmap v1911: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 5 op/s 2024-03-24T01:03:26.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:03:26 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 2301.647415 secs 2024-03-24T01:03:26.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:03:26 smithi118 ceph-mon[31535]: pgmap v1912: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 6 op/s 2024-03-24T01:03:26.704 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:03:26 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 2301.647415 secs 2024-03-24T01:03:26.704 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:03:26 smithi189 ceph-mon[36187]: pgmap v1912: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 6 op/s 2024-03-24T01:03:26.711 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:03:26 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 2301.647415 secs 2024-03-24T01:03:26.711 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:03:26 smithi193 ceph-mon[36878]: pgmap v1912: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 6 op/s 2024-03-24T01:03:28.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:03:28 smithi118 ceph-mon[31535]: pgmap v1913: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 5 op/s 2024-03-24T01:03:28.955 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:03:28 smithi189 ceph-mon[36187]: pgmap v1913: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 5 op/s 2024-03-24T01:03:28.962 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:03:28 smithi193 ceph-mon[36878]: pgmap v1913: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 5 op/s 2024-03-24T01:03:30.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:03:30 smithi118 ceph-mon[31535]: pgmap v1914: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 6 op/s 2024-03-24T01:03:30.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:03:30 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 2306.647470 secs 2024-03-24T01:03:30.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:03:30 smithi189 ceph-mon[36187]: pgmap v1914: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 6 op/s 2024-03-24T01:03:30.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:03:30 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 2306.647470 secs 2024-03-24T01:03:30.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:03:30 smithi193 ceph-mon[36878]: pgmap v1914: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 6 op/s 2024-03-24T01:03:30.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:03:30 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 2306.647470 secs 2024-03-24T01:03:32.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:03:32 smithi118 ceph-mon[31535]: pgmap v1915: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 6 op/s 2024-03-24T01:03:32.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:03:32 smithi189 ceph-mon[36187]: pgmap v1915: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 6 op/s 2024-03-24T01:03:32.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:03:32 smithi193 ceph-mon[36878]: pgmap v1915: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 6 op/s 2024-03-24T01:03:34.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:03:34 smithi118 ceph-mon[31535]: pgmap v1916: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 5 op/s 2024-03-24T01:03:34.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:03:34 smithi189 ceph-mon[36187]: pgmap v1916: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 5 op/s 2024-03-24T01:03:34.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:03:34 smithi193 ceph-mon[36878]: pgmap v1916: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 5 op/s 2024-03-24T01:03:35.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:03:35 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 2311.647530 secs 2024-03-24T01:03:35.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:03:35 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 2311.647530 secs 2024-03-24T01:03:35.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:03:35 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 2311.647530 secs 2024-03-24T01:03:36.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:03:36 smithi118 ceph-mon[31535]: pgmap v1917: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T01:03:36.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:03:36 smithi189 ceph-mon[36187]: pgmap v1917: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T01:03:36.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:03:36 smithi193 ceph-mon[36878]: pgmap v1917: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T01:03:38.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:03:38 smithi118 ceph-mon[31535]: pgmap v1918: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:03:38.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:03:38 smithi189 ceph-mon[36187]: pgmap v1918: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:03:38.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:03:38 smithi193 ceph-mon[36878]: pgmap v1918: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:03:40.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:03:40 smithi118 ceph-mon[31535]: pgmap v1919: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-24T01:03:40.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:03:40 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 2316.647589 secs 2024-03-24T01:03:40.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:03:40 smithi189 ceph-mon[36187]: pgmap v1919: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-24T01:03:40.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:03:40 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 2316.647589 secs 2024-03-24T01:03:40.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:03:40 smithi193 ceph-mon[36878]: pgmap v1919: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-24T01:03:40.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:03:40 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 2316.647589 secs 2024-03-24T01:03:42.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:03:42 smithi118 ceph-mon[31535]: pgmap v1920: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:03:42.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:03:42 smithi189 ceph-mon[36187]: pgmap v1920: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:03:42.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:03:42 smithi193 ceph-mon[36878]: pgmap v1920: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:03:45.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:03:45 smithi118 ceph-mon[31535]: pgmap v1921: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T01:03:45.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:03:45 smithi189 ceph-mon[36187]: pgmap v1921: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T01:03:45.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:03:45 smithi193 ceph-mon[36878]: pgmap v1921: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T01:03:46.362 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:03:46 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 2321.647632 secs 2024-03-24T01:03:46.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:03:46 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 2321.647632 secs 2024-03-24T01:03:46.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:03:46 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 2321.647632 secs 2024-03-24T01:03:47.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:03:47 smithi118 ceph-mon[31535]: pgmap v1922: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T01:03:47.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:03:47 smithi189 ceph-mon[36187]: pgmap v1922: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T01:03:47.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:03:47 smithi193 ceph-mon[36878]: pgmap v1922: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T01:03:49.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:03:49 smithi118 ceph-mon[31535]: pgmap v1923: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:03:49.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:03:49 smithi189 ceph-mon[36187]: pgmap v1923: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:03:49.460 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:03:49 smithi193 ceph-mon[36878]: pgmap v1923: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:03:51.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:03:51 smithi118 ceph-mon[31535]: pgmap v1924: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-24T01:03:51.447 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:03:51 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 2326.647719 secs 2024-03-24T01:03:51.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:03:51 smithi189 ceph-mon[36187]: pgmap v1924: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-24T01:03:51.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:03:51 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 2326.647719 secs 2024-03-24T01:03:51.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:03:51 smithi193 ceph-mon[36878]: pgmap v1924: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-24T01:03:51.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:03:51 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 2326.647719 secs 2024-03-24T01:03:53.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:03:53 smithi118 ceph-mon[31535]: pgmap v1925: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:03:53.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:03:53 smithi189 ceph-mon[36187]: pgmap v1925: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:03:53.460 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:03:53 smithi193 ceph-mon[36878]: pgmap v1925: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:03:55.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:03:55 smithi118 ceph-mon[31535]: pgmap v1926: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T01:03:55.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:03:55 smithi189 ceph-mon[36187]: pgmap v1926: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T01:03:55.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:03:55 smithi193 ceph-mon[36878]: pgmap v1926: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T01:03:56.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:03:56 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 2331.647802 secs 2024-03-24T01:03:56.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:03:56 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 2331.647802 secs 2024-03-24T01:03:56.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:03:56 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 2331.647802 secs 2024-03-24T01:03:57.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:03:57 smithi118 ceph-mon[31535]: pgmap v1927: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T01:03:57.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:03:57 smithi189 ceph-mon[36187]: pgmap v1927: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T01:03:57.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:03:57 smithi193 ceph-mon[36878]: pgmap v1927: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T01:03:59.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:03:59 smithi118 ceph-mon[31535]: pgmap v1928: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T01:03:59.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:03:59 smithi189 ceph-mon[36187]: pgmap v1928: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T01:03:59.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:03:59 smithi193 ceph-mon[36878]: pgmap v1928: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T01:04:01.348 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:04:01 smithi118 ceph-mon[31535]: pgmap v1929: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-24T01:04:01.349 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:04:01 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 2336.647892 secs 2024-03-24T01:04:01.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:04:01 smithi189 ceph-mon[36187]: pgmap v1929: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-24T01:04:01.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:04:01 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 2336.647892 secs 2024-03-24T01:04:01.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:04:01 smithi193 ceph-mon[36878]: pgmap v1929: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-24T01:04:01.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:04:01 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 2336.647892 secs 2024-03-24T01:04:03.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:04:03 smithi118 ceph-mon[31535]: pgmap v1930: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:04:03.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:04:03 smithi189 ceph-mon[36187]: pgmap v1930: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:04:03.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:04:03 smithi193 ceph-mon[36878]: pgmap v1930: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:04:05.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:04:05 smithi118 ceph-mon[31535]: pgmap v1931: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T01:04:05.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:04:05 smithi189 ceph-mon[36187]: pgmap v1931: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T01:04:05.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:04:05 smithi193 ceph-mon[36878]: pgmap v1931: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T01:04:06.450 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:04:06 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 2341.647982 secs 2024-03-24T01:04:06.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:04:06 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 2341.647982 secs 2024-03-24T01:04:06.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:04:06 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 2341.647982 secs 2024-03-24T01:04:07.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:04:07 smithi118 ceph-mon[31535]: pgmap v1932: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T01:04:07.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:04:07 smithi189 ceph-mon[36187]: pgmap v1932: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T01:04:07.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:04:07 smithi193 ceph-mon[36878]: pgmap v1932: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T01:04:09.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:04:09 smithi118 ceph-mon[31535]: pgmap v1933: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T01:04:09.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:04:09 smithi189 ceph-mon[36187]: pgmap v1933: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T01:04:09.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:04:09 smithi193 ceph-mon[36878]: pgmap v1933: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T01:04:11.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:04:11 smithi118 ceph-mon[31535]: pgmap v1934: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-03-24T01:04:11.447 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:04:11 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 2346.648071 secs 2024-03-24T01:04:11.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:04:11 smithi189 ceph-mon[36187]: pgmap v1934: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-03-24T01:04:11.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:04:11 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 2346.648071 secs 2024-03-24T01:04:11.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:04:11 smithi193 ceph-mon[36878]: pgmap v1934: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-03-24T01:04:11.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:04:11 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 2346.648071 secs 2024-03-24T01:04:13.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:04:13 smithi118 ceph-mon[31535]: pgmap v1935: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:04:13.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:04:13 smithi189 ceph-mon[36187]: pgmap v1935: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:04:13.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:04:13 smithi193 ceph-mon[36878]: pgmap v1935: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:04:15.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:04:15 smithi118 ceph-mon[31535]: pgmap v1936: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T01:04:15.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:04:15 smithi189 ceph-mon[36187]: pgmap v1936: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T01:04:15.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:04:15 smithi193 ceph-mon[36878]: pgmap v1936: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T01:04:16.103 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T01:04:16.103+0000 7f81ee000640 1 -- 172.21.15.118:0/1693767697 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f81c80065b0 con 0x7f81f81052d0 2024-03-24T01:04:16.104 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T01:04:16.104+0000 7f81ef400640 1 -- 172.21.15.118:0/1693767697 <== mon.2 v2:172.21.15.189:3300/0 22 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (secure 0 0 0) 0x7f81c80065b0 con 0x7f81f81052d0 2024-03-24T01:04:16.105 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T01:04:16.105+0000 7f81db400640 1 -- 172.21.15.118:0/659686424 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f81c4006300 con 0x5622651a13f0 2024-03-24T01:04:16.106 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T01:04:16.106+0000 7f81e4c00640 1 -- 172.21.15.118:0/659686424 <== mon.1 v2:172.21.15.193:3300/0 22 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 409+0+0 (secure 0 0 0) 0x7f81c4006300 con 0x5622651a13f0 2024-03-24T01:04:16.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:04:16 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 2351.648155 secs 2024-03-24T01:04:16.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:04:16 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 2351.648155 secs 2024-03-24T01:04:16.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:04:16 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 2351.648155 secs 2024-03-24T01:04:17.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:04:17 smithi118 ceph-mon[31535]: pgmap v1937: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T01:04:17.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:04:17 smithi189 ceph-mon[36187]: pgmap v1937: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T01:04:17.460 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:04:17 smithi193 ceph-mon[36878]: pgmap v1937: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T01:04:19.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:04:19 smithi118 ceph-mon[31535]: pgmap v1938: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T01:04:19.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:04:19 smithi189 ceph-mon[36187]: pgmap v1938: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T01:04:19.460 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:04:19 smithi193 ceph-mon[36878]: pgmap v1938: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T01:04:21.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:04:21 smithi118 ceph-mon[31535]: pgmap v1939: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-03-24T01:04:21.447 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:04:21 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 2356.648193 secs 2024-03-24T01:04:21.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:04:21 smithi189 ceph-mon[36187]: pgmap v1939: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-03-24T01:04:21.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:04:21 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 2356.648193 secs 2024-03-24T01:04:21.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:04:21 smithi193 ceph-mon[36878]: pgmap v1939: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-03-24T01:04:21.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:04:21 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 2356.648193 secs 2024-03-24T01:04:23.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:04:23 smithi118 ceph-mon[31535]: pgmap v1940: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:04:23.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:04:23 smithi189 ceph-mon[36187]: pgmap v1940: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:04:23.460 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:04:23 smithi193 ceph-mon[36878]: pgmap v1940: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:04:24.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:04:24 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T01:04:24.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:04:24 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T01:04:24.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:04:24 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T01:04:25.447 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:04:25 smithi118 ceph-mon[31535]: pgmap v1941: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T01:04:25.447 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:04:25 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-24T01:04:25.447 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:04:25 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-24T01:04:25.447 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:04:25 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-03-24T01:04:25.447 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:04:25 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-03-24T01:04:25.447 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:04:25 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T01:04:25.447 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:04:25 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T01:04:25.447 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:04:25 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T01:04:25.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:04:25 smithi189 ceph-mon[36187]: pgmap v1941: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T01:04:25.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:04:25 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-24T01:04:25.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:04:25 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-24T01:04:25.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:04:25 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-03-24T01:04:25.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:04:25 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-03-24T01:04:25.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:04:25 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T01:04:25.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:04:25 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T01:04:25.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:04:25 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T01:04:25.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:04:25 smithi193 ceph-mon[36878]: pgmap v1941: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T01:04:25.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:04:25 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-24T01:04:25.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:04:25 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-24T01:04:25.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:04:25 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-03-24T01:04:25.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:04:25 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-03-24T01:04:25.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:04:25 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T01:04:25.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:04:25 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T01:04:25.462 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:04:25 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T01:04:26.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:04:26 smithi118 ceph-mon[31535]: Adjusting osd_memory_target on smithi193 to 174.1M 2024-03-24T01:04:26.447 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:04:26 smithi118 ceph-mon[31535]: Unable to set osd_memory_target on smithi193 to 182639923: error parsing value: Value '182639923' is below minimum 939524096 2024-03-24T01:04:26.447 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:04:26 smithi118 ceph-mon[31535]: pgmap v1942: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 6 op/s 2024-03-24T01:04:26.447 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:04:26 smithi118 ceph-mon[31535]: pgmap v1943: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-03-24T01:04:26.447 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:04:26 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 2361.648259 secs 2024-03-24T01:04:26.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:04:26 smithi189 ceph-mon[36187]: Adjusting osd_memory_target on smithi193 to 174.1M 2024-03-24T01:04:26.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:04:26 smithi189 ceph-mon[36187]: Unable to set osd_memory_target on smithi193 to 182639923: error parsing value: Value '182639923' is below minimum 939524096 2024-03-24T01:04:26.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:04:26 smithi189 ceph-mon[36187]: pgmap v1942: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 6 op/s 2024-03-24T01:04:26.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:04:26 smithi189 ceph-mon[36187]: pgmap v1943: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-03-24T01:04:26.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:04:26 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 2361.648259 secs 2024-03-24T01:04:26.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:04:26 smithi193 ceph-mon[36878]: Adjusting osd_memory_target on smithi193 to 174.1M 2024-03-24T01:04:26.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:04:26 smithi193 ceph-mon[36878]: Unable to set osd_memory_target on smithi193 to 182639923: error parsing value: Value '182639923' is below minimum 939524096 2024-03-24T01:04:26.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:04:26 smithi193 ceph-mon[36878]: pgmap v1942: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 6 op/s 2024-03-24T01:04:26.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:04:26 smithi193 ceph-mon[36878]: pgmap v1943: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-03-24T01:04:26.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:04:26 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 2361.648259 secs 2024-03-24T01:04:28.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:04:28 smithi118 ceph-mon[31535]: pgmap v1944: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 5 op/s 2024-03-24T01:04:28.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:04:28 smithi189 ceph-mon[36187]: pgmap v1944: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 5 op/s 2024-03-24T01:04:28.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:04:28 smithi193 ceph-mon[36878]: pgmap v1944: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 5 op/s 2024-03-24T01:04:30.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:04:30 smithi118 ceph-mon[31535]: pgmap v1945: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 5 op/s 2024-03-24T01:04:30.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:04:30 smithi189 ceph-mon[36187]: pgmap v1945: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 5 op/s 2024-03-24T01:04:30.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:04:30 smithi193 ceph-mon[36878]: pgmap v1945: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 5 op/s 2024-03-24T01:04:31.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:04:31 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 2366.648318 secs 2024-03-24T01:04:31.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:04:31 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 2366.648318 secs 2024-03-24T01:04:31.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:04:31 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 2366.648318 secs 2024-03-24T01:04:32.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:04:32 smithi118 ceph-mon[31535]: pgmap v1946: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-03-24T01:04:32.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:04:32 smithi189 ceph-mon[36187]: pgmap v1946: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-03-24T01:04:32.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:04:32 smithi193 ceph-mon[36878]: pgmap v1946: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-03-24T01:04:34.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:04:34 smithi118 ceph-mon[31535]: pgmap v1947: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 5 op/s 2024-03-24T01:04:34.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:04:34 smithi189 ceph-mon[36187]: pgmap v1947: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 5 op/s 2024-03-24T01:04:34.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:04:34 smithi193 ceph-mon[36878]: pgmap v1947: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 5 op/s 2024-03-24T01:04:36.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:04:36 smithi118 ceph-mon[31535]: pgmap v1948: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 5 op/s 2024-03-24T01:04:36.447 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:04:36 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 2371.648388 secs 2024-03-24T01:04:36.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:04:36 smithi189 ceph-mon[36187]: pgmap v1948: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 5 op/s 2024-03-24T01:04:36.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:04:36 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 2371.648388 secs 2024-03-24T01:04:36.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:04:36 smithi193 ceph-mon[36878]: pgmap v1948: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 5 op/s 2024-03-24T01:04:36.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:04:36 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 2371.648388 secs 2024-03-24T01:04:38.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:04:38 smithi118 ceph-mon[31535]: pgmap v1949: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:04:38.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:04:38 smithi189 ceph-mon[36187]: pgmap v1949: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:04:38.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:04:38 smithi193 ceph-mon[36878]: pgmap v1949: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:04:40.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:04:40 smithi118 ceph-mon[31535]: pgmap v1950: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T01:04:40.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:04:40 smithi189 ceph-mon[36187]: pgmap v1950: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T01:04:40.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:04:40 smithi193 ceph-mon[36878]: pgmap v1950: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T01:04:41.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:04:41 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 2376.648451 secs 2024-03-24T01:04:41.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:04:41 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 2376.648451 secs 2024-03-24T01:04:41.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:04:41 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 2376.648451 secs 2024-03-24T01:04:42.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:04:42 smithi193 ceph-mon[36878]: pgmap v1951: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T01:04:42.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:04:42 smithi118 ceph-mon[31535]: pgmap v1951: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T01:04:42.704 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:04:42 smithi189 ceph-mon[36187]: pgmap v1951: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T01:04:44.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:04:44 smithi193 ceph-mon[36878]: pgmap v1952: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 5 op/s 2024-03-24T01:04:44.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:04:44 smithi118 ceph-mon[31535]: pgmap v1952: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 5 op/s 2024-03-24T01:04:44.704 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:04:44 smithi189 ceph-mon[36187]: pgmap v1952: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 5 op/s 2024-03-24T01:04:46.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:04:46 smithi118 ceph-mon[31535]: pgmap v1953: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T01:04:46.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:04:46 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 2381.648512 secs 2024-03-24T01:04:46.704 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:04:46 smithi189 ceph-mon[36187]: pgmap v1953: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T01:04:46.704 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:04:46 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 2381.648512 secs 2024-03-24T01:04:46.711 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:04:46 smithi193 ceph-mon[36878]: pgmap v1953: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T01:04:46.711 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:04:46 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 2381.648512 secs 2024-03-24T01:04:48.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:04:48 smithi118 ceph-mon[31535]: pgmap v1954: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:04:48.704 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:04:48 smithi189 ceph-mon[36187]: pgmap v1954: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:04:48.711 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:04:48 smithi193 ceph-mon[36878]: pgmap v1954: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:04:50.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:04:50 smithi118 ceph-mon[31535]: pgmap v1955: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T01:04:50.704 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:04:50 smithi189 ceph-mon[36187]: pgmap v1955: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T01:04:50.710 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:04:50 smithi193 ceph-mon[36878]: pgmap v1955: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T01:04:51.605 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:04:51 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 2386.648571 secs 2024-03-24T01:04:51.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:04:51 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 2386.648571 secs 2024-03-24T01:04:51.711 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:04:51 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 2386.648571 secs 2024-03-24T01:04:52.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:04:52 smithi118 ceph-mon[31535]: pgmap v1956: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T01:04:52.704 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:04:52 smithi189 ceph-mon[36187]: pgmap v1956: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T01:04:52.710 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:04:52 smithi193 ceph-mon[36878]: pgmap v1956: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T01:04:54.697 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:04:54 smithi118 ceph-mon[31535]: pgmap v1957: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 5 op/s 2024-03-24T01:04:54.704 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:04:54 smithi189 ceph-mon[36187]: pgmap v1957: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 5 op/s 2024-03-24T01:04:54.711 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:04:54 smithi193 ceph-mon[36878]: pgmap v1957: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 5 op/s 2024-03-24T01:04:56.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:04:56 smithi118 ceph-mon[31535]: pgmap v1958: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T01:04:56.697 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:04:56 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 2391.648661 secs 2024-03-24T01:04:56.704 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:04:56 smithi189 ceph-mon[36187]: pgmap v1958: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T01:04:56.704 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:04:56 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 2391.648661 secs 2024-03-24T01:04:56.711 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:04:56 smithi193 ceph-mon[36878]: pgmap v1958: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T01:04:56.711 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:04:56 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 2391.648661 secs 2024-03-24T01:04:58.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:04:58 smithi118 ceph-mon[31535]: pgmap v1959: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:04:58.704 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:04:58 smithi189 ceph-mon[36187]: pgmap v1959: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:04:58.711 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:04:58 smithi193 ceph-mon[36878]: pgmap v1959: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:05:00.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:05:00 smithi118 ceph-mon[31535]: pgmap v1960: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T01:05:00.704 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:05:00 smithi189 ceph-mon[36187]: pgmap v1960: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T01:05:00.711 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:05:00 smithi193 ceph-mon[36878]: pgmap v1960: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T01:05:01.598 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:05:01 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 2396.648720 secs 2024-03-24T01:05:01.704 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:05:01 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 2396.648720 secs 2024-03-24T01:05:01.711 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:05:01 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 2396.648720 secs 2024-03-24T01:05:02.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:05:02 smithi118 ceph-mon[31535]: pgmap v1961: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T01:05:02.704 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:05:02 smithi189 ceph-mon[36187]: pgmap v1961: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T01:05:02.711 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:05:02 smithi193 ceph-mon[36878]: pgmap v1961: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T01:05:04.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:05:04 smithi118 ceph-mon[31535]: pgmap v1962: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 5 op/s 2024-03-24T01:05:04.704 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:05:04 smithi189 ceph-mon[36187]: pgmap v1962: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 5 op/s 2024-03-24T01:05:04.710 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:05:04 smithi193 ceph-mon[36878]: pgmap v1962: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 5 op/s 2024-03-24T01:05:06.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:05:06 smithi118 ceph-mon[31535]: pgmap v1963: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T01:05:06.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:05:06 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 2401.648759 secs 2024-03-24T01:05:06.704 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:05:06 smithi189 ceph-mon[36187]: pgmap v1963: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T01:05:06.704 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:05:06 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 2401.648759 secs 2024-03-24T01:05:06.711 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:05:06 smithi193 ceph-mon[36878]: pgmap v1963: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T01:05:06.711 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:05:06 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 2401.648759 secs 2024-03-24T01:05:08.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:05:08 smithi118 ceph-mon[31535]: pgmap v1964: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:05:08.704 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:05:08 smithi189 ceph-mon[36187]: pgmap v1964: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:05:08.710 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:05:08 smithi193 ceph-mon[36878]: pgmap v1964: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:05:10.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:05:10 smithi118 ceph-mon[31535]: pgmap v1965: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T01:05:10.704 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:05:10 smithi189 ceph-mon[36187]: pgmap v1965: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T01:05:10.711 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:05:10 smithi193 ceph-mon[36878]: pgmap v1965: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T01:05:11.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:05:11 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 2406.648841 secs 2024-03-24T01:05:11.704 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:05:11 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 2406.648841 secs 2024-03-24T01:05:11.710 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:05:11 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 2406.648841 secs 2024-03-24T01:05:12.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:05:12 smithi118 ceph-mon[31535]: pgmap v1966: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T01:05:12.704 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:05:12 smithi189 ceph-mon[36187]: pgmap v1966: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T01:05:12.710 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:05:12 smithi193 ceph-mon[36878]: pgmap v1966: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T01:05:14.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:05:14 smithi118 ceph-mon[31535]: pgmap v1967: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 5 op/s 2024-03-24T01:05:14.704 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:05:14 smithi189 ceph-mon[36187]: pgmap v1967: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 5 op/s 2024-03-24T01:05:14.711 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:05:14 smithi193 ceph-mon[36878]: pgmap v1967: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 5 op/s 2024-03-24T01:05:16.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:05:16 smithi118 ceph-mon[31535]: pgmap v1968: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T01:05:16.697 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:05:16 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 2411.648856 secs 2024-03-24T01:05:16.704 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:05:16 smithi189 ceph-mon[36187]: pgmap v1968: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T01:05:16.704 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:05:16 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 2411.648856 secs 2024-03-24T01:05:16.711 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:05:16 smithi193 ceph-mon[36878]: pgmap v1968: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T01:05:16.711 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:05:16 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 2411.648856 secs 2024-03-24T01:05:18.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:05:18 smithi118 ceph-mon[31535]: pgmap v1969: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:05:18.704 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:05:18 smithi189 ceph-mon[36187]: pgmap v1969: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:05:18.711 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:05:18 smithi193 ceph-mon[36878]: pgmap v1969: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:05:20.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:05:20 smithi118 ceph-mon[31535]: pgmap v1970: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T01:05:20.704 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:05:20 smithi189 ceph-mon[36187]: pgmap v1970: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T01:05:20.711 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:05:20 smithi193 ceph-mon[36878]: pgmap v1970: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T01:05:21.605 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:05:21 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 2416.648944 secs 2024-03-24T01:05:21.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:05:21 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 2416.648944 secs 2024-03-24T01:05:21.711 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:05:21 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 2416.648944 secs 2024-03-24T01:05:22.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:05:22 smithi118 ceph-mon[31535]: pgmap v1971: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T01:05:22.704 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:05:22 smithi189 ceph-mon[36187]: pgmap v1971: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T01:05:22.710 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:05:22 smithi193 ceph-mon[36878]: pgmap v1971: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T01:05:24.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:05:24 smithi118 ceph-mon[31535]: pgmap v1972: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 5 op/s 2024-03-24T01:05:24.704 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:05:24 smithi189 ceph-mon[36187]: pgmap v1972: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 5 op/s 2024-03-24T01:05:24.710 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:05:24 smithi193 ceph-mon[36878]: pgmap v1972: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 5 op/s 2024-03-24T01:05:25.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:05:25 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T01:05:25.704 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:05:25 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T01:05:25.711 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:05:25 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T01:05:26.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:05:26 smithi118 ceph-mon[31535]: pgmap v1973: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T01:05:26.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:05:26 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 2421.649032 secs 2024-03-24T01:05:26.704 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:05:26 smithi189 ceph-mon[36187]: pgmap v1973: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T01:05:26.704 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:05:26 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 2421.649032 secs 2024-03-24T01:05:26.711 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:05:26 smithi193 ceph-mon[36878]: pgmap v1973: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T01:05:26.711 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:05:26 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 2421.649032 secs 2024-03-24T01:05:28.697 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:05:28 smithi118 ceph-mon[31535]: pgmap v1974: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:05:28.697 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:05:28 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T01:05:28.697 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:05:28 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T01:05:28.698 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:05:28 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T01:05:28.698 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:05:28 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T01:05:28.698 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:05:28 smithi118 ceph-mon[31535]: pgmap v1975: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T01:05:28.698 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:05:28 smithi118 ceph-mon[31535]: pgmap v1976: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-03-24T01:05:28.698 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:05:28 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T01:05:28.704 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:05:28 smithi189 ceph-mon[36187]: pgmap v1974: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:05:28.704 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:05:28 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T01:05:28.704 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:05:28 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T01:05:28.704 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:05:28 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T01:05:28.705 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:05:28 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T01:05:28.705 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:05:28 smithi189 ceph-mon[36187]: pgmap v1975: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T01:05:28.705 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:05:28 smithi189 ceph-mon[36187]: pgmap v1976: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-03-24T01:05:28.705 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:05:28 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T01:05:28.711 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:05:28 smithi193 ceph-mon[36878]: pgmap v1974: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:05:28.711 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:05:28 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T01:05:28.711 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:05:28 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T01:05:28.711 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:05:28 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T01:05:28.711 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:05:28 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T01:05:28.711 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:05:28 smithi193 ceph-mon[36878]: pgmap v1975: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T01:05:28.711 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:05:28 smithi193 ceph-mon[36878]: pgmap v1976: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-03-24T01:05:28.711 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:05:28 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T01:05:29.704 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:05:29 smithi189 ceph-mon[36187]: pgmap v1977: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-03-24T01:05:29.710 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:05:29 smithi193 ceph-mon[36878]: pgmap v1977: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-03-24T01:05:29.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:05:29 smithi118 ceph-mon[31535]: pgmap v1977: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-03-24T01:05:30.704 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:05:30 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 2426.649095 secs 2024-03-24T01:05:30.710 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:05:30 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 2426.649095 secs 2024-03-24T01:05:30.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:05:30 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 2426.649095 secs 2024-03-24T01:05:31.848 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:05:31 smithi118 ceph-mon[31535]: pgmap v1978: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 6 op/s 2024-03-24T01:05:31.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:05:31 smithi189 ceph-mon[36187]: pgmap v1978: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 6 op/s 2024-03-24T01:05:31.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:05:31 smithi193 ceph-mon[36878]: pgmap v1978: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 6 op/s 2024-03-24T01:05:33.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:05:33 smithi118 ceph-mon[31535]: pgmap v1979: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 5 op/s 2024-03-24T01:05:33.704 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:05:33 smithi189 ceph-mon[36187]: pgmap v1979: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 5 op/s 2024-03-24T01:05:33.711 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:05:33 smithi193 ceph-mon[36878]: pgmap v1979: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 5 op/s 2024-03-24T01:05:35.704 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:05:35 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 2431.649157 secs 2024-03-24T01:05:35.704 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:05:35 smithi189 ceph-mon[36187]: pgmap v1980: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-03-24T01:05:35.711 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:05:35 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 2431.649157 secs 2024-03-24T01:05:35.711 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:05:35 smithi193 ceph-mon[36878]: pgmap v1980: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-03-24T01:05:35.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:05:35 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 2431.649157 secs 2024-03-24T01:05:35.947 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:05:35 smithi118 ceph-mon[31535]: pgmap v1980: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-03-24T01:05:37.711 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:05:37 smithi193 ceph-mon[36878]: pgmap v1981: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 5 op/s 2024-03-24T01:05:37.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:05:37 smithi118 ceph-mon[31535]: pgmap v1981: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 5 op/s 2024-03-24T01:05:37.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:05:37 smithi189 ceph-mon[36187]: pgmap v1981: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 5 op/s 2024-03-24T01:05:39.711 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:05:39 smithi193 ceph-mon[36878]: pgmap v1982: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-24T01:05:39.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:05:39 smithi118 ceph-mon[31535]: pgmap v1982: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-24T01:05:39.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:05:39 smithi189 ceph-mon[36187]: pgmap v1982: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-24T01:05:40.711 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:05:40 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 2436.649195 secs 2024-03-24T01:05:40.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:05:40 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 2436.649195 secs 2024-03-24T01:05:40.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:05:40 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 2436.649195 secs 2024-03-24T01:05:41.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:05:41 smithi118 ceph-mon[31535]: pgmap v1983: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:05:41.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:05:41 smithi189 ceph-mon[36187]: pgmap v1983: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:05:41.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:05:41 smithi193 ceph-mon[36878]: pgmap v1983: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:05:43.711 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:05:43 smithi193 ceph-mon[36878]: pgmap v1984: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T01:05:43.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:05:43 smithi118 ceph-mon[31535]: pgmap v1984: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T01:05:43.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:05:43 smithi189 ceph-mon[36187]: pgmap v1984: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T01:05:45.711 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:05:45 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 2441.649257 secs 2024-03-24T01:05:45.711 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:05:45 smithi193 ceph-mon[36878]: pgmap v1985: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T01:05:45.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:05:45 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 2441.649257 secs 2024-03-24T01:05:45.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:05:45 smithi118 ceph-mon[31535]: pgmap v1985: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T01:05:45.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:05:45 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 2441.649257 secs 2024-03-24T01:05:45.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:05:45 smithi189 ceph-mon[36187]: pgmap v1985: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T01:05:47.711 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:05:47 smithi193 ceph-mon[36878]: pgmap v1986: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:05:47.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:05:47 smithi118 ceph-mon[31535]: pgmap v1986: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:05:47.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:05:47 smithi189 ceph-mon[36187]: pgmap v1986: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:05:49.711 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:05:49 smithi193 ceph-mon[36878]: pgmap v1987: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-24T01:05:49.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:05:49 smithi118 ceph-mon[31535]: pgmap v1987: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-24T01:05:49.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:05:49 smithi189 ceph-mon[36187]: pgmap v1987: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-24T01:05:50.711 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:05:50 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 2446.649318 secs 2024-03-24T01:05:50.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:05:50 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 2446.649318 secs 2024-03-24T01:05:50.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:05:50 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 2446.649318 secs 2024-03-24T01:05:51.855 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:05:51 smithi189 ceph-mon[36187]: pgmap v1988: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:05:51.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:05:51 smithi118 ceph-mon[31535]: pgmap v1988: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:05:51.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:05:51 smithi193 ceph-mon[36878]: pgmap v1988: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:05:53.711 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:05:53 smithi193 ceph-mon[36878]: pgmap v1989: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-03-24T01:05:53.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:05:53 smithi118 ceph-mon[31535]: pgmap v1989: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-03-24T01:05:53.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:05:53 smithi189 ceph-mon[36187]: pgmap v1989: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-03-24T01:05:55.711 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:05:55 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 2451.649402 secs 2024-03-24T01:05:55.711 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:05:55 smithi193 ceph-mon[36878]: pgmap v1990: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 6 op/s 2024-03-24T01:05:55.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:05:55 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 2451.649402 secs 2024-03-24T01:05:55.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:05:55 smithi118 ceph-mon[31535]: pgmap v1990: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 6 op/s 2024-03-24T01:05:55.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:05:55 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 2451.649402 secs 2024-03-24T01:05:55.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:05:55 smithi189 ceph-mon[36187]: pgmap v1990: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 6 op/s 2024-03-24T01:05:57.711 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:05:57 smithi193 ceph-mon[36878]: pgmap v1991: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-03-24T01:05:57.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:05:57 smithi118 ceph-mon[31535]: pgmap v1991: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-03-24T01:05:57.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:05:57 smithi189 ceph-mon[36187]: pgmap v1991: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-03-24T01:05:59.711 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:05:59 smithi193 ceph-mon[36878]: pgmap v1992: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-24T01:05:59.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:05:59 smithi118 ceph-mon[31535]: pgmap v1992: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-24T01:05:59.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:05:59 smithi189 ceph-mon[36187]: pgmap v1992: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-24T01:06:00.711 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:06:00 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 2456.649491 secs 2024-03-24T01:06:00.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:06:00 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 2456.649491 secs 2024-03-24T01:06:00.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:06:00 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 2456.649491 secs 2024-03-24T01:06:01.711 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:06:01 smithi193 ceph-mon[36878]: pgmap v1993: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-03-24T01:06:01.848 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:06:01 smithi118 ceph-mon[31535]: pgmap v1993: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-03-24T01:06:01.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:06:01 smithi189 ceph-mon[36187]: pgmap v1993: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-03-24T01:06:03.711 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:06:03 smithi193 ceph-mon[36878]: pgmap v1994: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-03-24T01:06:03.928 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:06:03 smithi118 ceph-mon[31535]: pgmap v1994: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-03-24T01:06:03.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:06:03 smithi189 ceph-mon[36187]: pgmap v1994: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-03-24T01:06:05.711 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:06:05 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 2461.649531 secs 2024-03-24T01:06:05.711 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:06:05 smithi193 ceph-mon[36878]: pgmap v1995: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T01:06:05.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:06:05 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 2461.649531 secs 2024-03-24T01:06:05.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:06:05 smithi118 ceph-mon[31535]: pgmap v1995: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T01:06:05.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:06:05 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 2461.649531 secs 2024-03-24T01:06:05.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:06:05 smithi189 ceph-mon[36187]: pgmap v1995: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T01:06:07.711 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:06:07 smithi193 ceph-mon[36878]: pgmap v1996: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:06:07.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:06:07 smithi118 ceph-mon[31535]: pgmap v1996: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:06:07.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:06:07 smithi189 ceph-mon[36187]: pgmap v1996: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:06:09.711 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:06:09 smithi193 ceph-mon[36878]: pgmap v1997: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-24T01:06:09.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:06:09 smithi118 ceph-mon[31535]: pgmap v1997: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-24T01:06:09.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:06:09 smithi189 ceph-mon[36187]: pgmap v1997: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-24T01:06:10.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:06:10 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 2466.649619 secs 2024-03-24T01:06:10.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:06:10 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 2466.649619 secs 2024-03-24T01:06:10.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:06:10 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 2466.649619 secs 2024-03-24T01:06:11.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:06:11 smithi118 ceph-mon[31535]: pgmap v1998: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:06:11.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:06:11 smithi189 ceph-mon[36187]: pgmap v1998: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:06:11.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:06:11 smithi193 ceph-mon[36878]: pgmap v1998: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:06:13.711 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:06:13 smithi193 ceph-mon[36878]: pgmap v1999: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T01:06:13.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:06:13 smithi118 ceph-mon[31535]: pgmap v1999: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T01:06:13.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:06:13 smithi189 ceph-mon[36187]: pgmap v1999: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T01:06:15.711 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:06:15 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 2471.649680 secs 2024-03-24T01:06:15.711 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:06:15 smithi193 ceph-mon[36878]: pgmap v2000: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T01:06:15.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:06:15 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 2471.649680 secs 2024-03-24T01:06:15.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:06:15 smithi118 ceph-mon[31535]: pgmap v2000: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T01:06:15.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:06:15 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 2471.649680 secs 2024-03-24T01:06:15.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:06:15 smithi189 ceph-mon[36187]: pgmap v2000: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T01:06:17.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:06:17 smithi118 ceph-mon[31535]: pgmap v2001: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:06:17.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:06:17 smithi189 ceph-mon[36187]: pgmap v2001: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:06:17.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:06:17 smithi193 ceph-mon[36878]: pgmap v2001: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:06:19.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:06:19 smithi118 ceph-mon[31535]: pgmap v2002: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-24T01:06:19.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:06:19 smithi189 ceph-mon[36187]: pgmap v2002: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-24T01:06:19.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:06:19 smithi193 ceph-mon[36878]: pgmap v2002: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-24T01:06:20.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:06:20 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 2476.649769 secs 2024-03-24T01:06:20.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:06:20 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 2476.649769 secs 2024-03-24T01:06:20.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:06:20 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 2476.649769 secs 2024-03-24T01:06:21.854 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:06:21 smithi189 ceph-mon[36187]: pgmap v2003: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:06:21.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:06:21 smithi118 ceph-mon[31535]: pgmap v2003: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:06:21.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:06:21 smithi193 ceph-mon[36878]: pgmap v2003: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:06:23.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:06:23 smithi118 ceph-mon[31535]: pgmap v2004: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T01:06:23.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:06:23 smithi189 ceph-mon[36187]: pgmap v2004: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T01:06:23.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:06:23 smithi193 ceph-mon[36878]: pgmap v2004: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T01:06:25.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:06:25 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 2481.649858 secs 2024-03-24T01:06:25.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:06:25 smithi118 ceph-mon[31535]: pgmap v2005: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T01:06:25.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:06:25 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 2481.649858 secs 2024-03-24T01:06:25.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:06:25 smithi189 ceph-mon[36187]: pgmap v2005: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T01:06:25.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:06:25 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 2481.649858 secs 2024-03-24T01:06:25.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:06:25 smithi193 ceph-mon[36878]: pgmap v2005: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T01:06:27.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:06:27 smithi118 ceph-mon[31535]: pgmap v2006: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:06:27.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:06:27 smithi189 ceph-mon[36187]: pgmap v2006: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:06:27.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:06:27 smithi193 ceph-mon[36878]: pgmap v2006: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:06:28.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:06:28 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T01:06:28.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:06:28 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T01:06:28.962 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:06:28 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T01:06:30.197 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:06:29 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T01:06:30.197 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:06:29 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T01:06:30.197 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:06:29 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T01:06:30.197 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:06:29 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T01:06:30.197 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:06:29 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T01:06:30.197 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:06:29 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T01:06:30.197 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:06:29 smithi118 ceph-mon[31535]: pgmap v2007: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 6 op/s 2024-03-24T01:06:30.197 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:06:29 smithi118 ceph-mon[31535]: pgmap v2008: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 5 op/s 2024-03-24T01:06:30.197 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:06:29 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T01:06:30.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:06:29 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T01:06:30.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:06:29 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T01:06:30.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:06:29 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T01:06:30.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:06:29 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T01:06:30.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:06:29 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T01:06:30.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:06:29 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T01:06:30.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:06:29 smithi189 ceph-mon[36187]: pgmap v2007: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 6 op/s 2024-03-24T01:06:30.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:06:29 smithi189 ceph-mon[36187]: pgmap v2008: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 5 op/s 2024-03-24T01:06:30.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:06:29 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T01:06:30.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:06:29 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T01:06:30.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:06:29 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T01:06:30.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:06:29 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T01:06:30.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:06:29 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T01:06:30.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:06:29 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T01:06:30.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:06:29 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T01:06:30.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:06:29 smithi193 ceph-mon[36878]: pgmap v2007: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 6 op/s 2024-03-24T01:06:30.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:06:29 smithi193 ceph-mon[36878]: pgmap v2008: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 5 op/s 2024-03-24T01:06:30.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:06:29 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T01:06:31.125 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T01:06:31.125+0000 7f81db400640 1 -- 172.21.15.118:0/659686424 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- auth(proto 2 132 bytes epoch 0) v1 -- 0x7f81c4006300 con 0x5622651a13f0 2024-03-24T01:06:31.125 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T01:06:31.125+0000 7f81ee000640 1 -- 172.21.15.118:0/1693767697 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- auth(proto 2 132 bytes epoch 0) v1 -- 0x7f81c80065b0 con 0x7f81f81052d0 2024-03-24T01:06:31.126 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T01:06:31.125+0000 7f81e4c00640 1 -- 172.21.15.118:0/659686424 <== mon.1 v2:172.21.15.193:3300/0 23 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 233+0+0 (secure 0 0 0) 0x7f81c4006300 con 0x5622651a13f0 2024-03-24T01:06:31.126 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T01:06:31.125+0000 7f81ef400640 1 -- 172.21.15.118:0/1693767697 <== mon.2 v2:172.21.15.189:3300/0 23 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 233+0+0 (secure 0 0 0) 0x7f81c80065b0 con 0x7f81f81052d0 2024-03-24T01:06:31.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:06:30 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 2486.649948 secs 2024-03-24T01:06:31.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:06:30 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 2486.649948 secs 2024-03-24T01:06:31.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:06:30 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 2486.649948 secs 2024-03-24T01:06:32.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:06:31 smithi118 ceph-mon[31535]: pgmap v2009: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 6 op/s 2024-03-24T01:06:32.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:06:31 smithi189 ceph-mon[36187]: pgmap v2009: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 6 op/s 2024-03-24T01:06:32.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:06:31 smithi193 ceph-mon[36878]: pgmap v2009: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 6 op/s 2024-03-24T01:06:34.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:06:34 smithi118 ceph-mon[31535]: pgmap v2010: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 5 op/s 2024-03-24T01:06:34.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:06:34 smithi189 ceph-mon[36187]: pgmap v2010: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 5 op/s 2024-03-24T01:06:34.460 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:06:34 smithi193 ceph-mon[36878]: pgmap v2010: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 5 op/s 2024-03-24T01:06:36.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:06:36 smithi118 ceph-mon[31535]: pgmap v2011: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 6 op/s 2024-03-24T01:06:36.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:06:36 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 2491.650034 secs 2024-03-24T01:06:36.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:06:36 smithi189 ceph-mon[36187]: pgmap v2011: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 6 op/s 2024-03-24T01:06:36.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:06:36 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 2491.650034 secs 2024-03-24T01:06:36.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:06:36 smithi193 ceph-mon[36878]: pgmap v2011: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 6 op/s 2024-03-24T01:06:36.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:06:36 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 2491.650034 secs 2024-03-24T01:06:38.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:06:38 smithi118 ceph-mon[31535]: pgmap v2012: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 6 op/s 2024-03-24T01:06:38.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:06:38 smithi189 ceph-mon[36187]: pgmap v2012: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 6 op/s 2024-03-24T01:06:38.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:06:38 smithi193 ceph-mon[36878]: pgmap v2012: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 6 op/s 2024-03-24T01:06:40.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:06:40 smithi118 ceph-mon[31535]: pgmap v2013: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 5 op/s 2024-03-24T01:06:40.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:06:40 smithi189 ceph-mon[36187]: pgmap v2013: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 5 op/s 2024-03-24T01:06:40.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:06:40 smithi193 ceph-mon[36878]: pgmap v2013: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 5 op/s 2024-03-24T01:06:41.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:06:41 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 2496.650069 secs 2024-03-24T01:06:41.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:06:41 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 2496.650069 secs 2024-03-24T01:06:41.460 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:06:41 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 2496.650069 secs 2024-03-24T01:06:42.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:06:42 smithi118 ceph-mon[31535]: pgmap v2014: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T01:06:42.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:06:42 smithi189 ceph-mon[36187]: pgmap v2014: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T01:06:42.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:06:42 smithi193 ceph-mon[36878]: pgmap v2014: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T01:06:44.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:06:44 smithi118 ceph-mon[31535]: pgmap v2015: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 5 op/s 2024-03-24T01:06:44.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:06:44 smithi189 ceph-mon[36187]: pgmap v2015: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 5 op/s 2024-03-24T01:06:44.462 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:06:44 smithi193 ceph-mon[36878]: pgmap v2015: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 5 op/s 2024-03-24T01:06:46.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:06:46 smithi118 ceph-mon[31535]: pgmap v2016: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T01:06:46.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:06:46 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 2501.650129 secs 2024-03-24T01:06:46.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:06:46 smithi189 ceph-mon[36187]: pgmap v2016: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T01:06:46.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:06:46 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 2501.650129 secs 2024-03-24T01:06:46.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:06:46 smithi193 ceph-mon[36878]: pgmap v2016: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T01:06:46.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:06:46 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 2501.650129 secs 2024-03-24T01:06:48.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:06:48 smithi118 ceph-mon[31535]: pgmap v2017: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:06:48.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:06:48 smithi189 ceph-mon[36187]: pgmap v2017: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:06:48.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:06:48 smithi193 ceph-mon[36878]: pgmap v2017: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:06:50.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:06:50 smithi118 ceph-mon[31535]: pgmap v2018: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T01:06:50.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:06:50 smithi189 ceph-mon[36187]: pgmap v2018: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T01:06:50.460 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:06:50 smithi193 ceph-mon[36878]: pgmap v2018: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T01:06:51.349 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:06:51 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 2506.650191 secs 2024-03-24T01:06:51.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:06:51 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 2506.650191 secs 2024-03-24T01:06:51.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:06:51 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 2506.650191 secs 2024-03-24T01:06:52.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:06:52 smithi118 ceph-mon[31535]: pgmap v2019: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T01:06:52.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:06:52 smithi189 ceph-mon[36187]: pgmap v2019: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T01:06:52.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:06:52 smithi193 ceph-mon[36878]: pgmap v2019: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T01:06:54.447 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:06:54 smithi118 ceph-mon[31535]: pgmap v2020: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 5 op/s 2024-03-24T01:06:54.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:06:54 smithi189 ceph-mon[36187]: pgmap v2020: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 5 op/s 2024-03-24T01:06:54.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:06:54 smithi193 ceph-mon[36878]: pgmap v2020: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 5 op/s 2024-03-24T01:06:56.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:06:56 smithi118 ceph-mon[31535]: pgmap v2021: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T01:06:56.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:06:56 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 2511.650253 secs 2024-03-24T01:06:56.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:06:56 smithi189 ceph-mon[36187]: pgmap v2021: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T01:06:56.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:06:56 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 2511.650253 secs 2024-03-24T01:06:56.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:06:56 smithi193 ceph-mon[36878]: pgmap v2021: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T01:06:56.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:06:56 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 2511.650253 secs 2024-03-24T01:06:58.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:06:58 smithi118 ceph-mon[31535]: pgmap v2022: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:06:58.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:06:58 smithi189 ceph-mon[36187]: pgmap v2022: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:06:58.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:06:58 smithi193 ceph-mon[36878]: pgmap v2022: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:07:00.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:07:00 smithi118 ceph-mon[31535]: pgmap v2023: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T01:07:00.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:07:00 smithi189 ceph-mon[36187]: pgmap v2023: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T01:07:00.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:07:00 smithi193 ceph-mon[36878]: pgmap v2023: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T01:07:01.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:07:01 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 2516.650313 secs 2024-03-24T01:07:01.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:07:01 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 2516.650313 secs 2024-03-24T01:07:01.460 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:07:01 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 2516.650313 secs 2024-03-24T01:07:02.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:07:02 smithi118 ceph-mon[31535]: pgmap v2024: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T01:07:02.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:07:02 smithi189 ceph-mon[36187]: pgmap v2024: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T01:07:02.460 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:07:02 smithi193 ceph-mon[36878]: pgmap v2024: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T01:07:04.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:07:04 smithi118 ceph-mon[31535]: pgmap v2025: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 5 op/s 2024-03-24T01:07:04.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:07:04 smithi189 ceph-mon[36187]: pgmap v2025: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 5 op/s 2024-03-24T01:07:04.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:07:04 smithi193 ceph-mon[36878]: pgmap v2025: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 5 op/s 2024-03-24T01:07:06.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:07:06 smithi118 ceph-mon[31535]: pgmap v2026: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T01:07:06.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:07:06 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 2521.650350 secs 2024-03-24T01:07:06.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:07:06 smithi189 ceph-mon[36187]: pgmap v2026: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T01:07:06.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:07:06 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 2521.650350 secs 2024-03-24T01:07:06.460 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:07:06 smithi193 ceph-mon[36878]: pgmap v2026: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T01:07:06.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:07:06 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 2521.650350 secs 2024-03-24T01:07:08.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:07:08 smithi118 ceph-mon[31535]: pgmap v2027: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:07:08.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:07:08 smithi189 ceph-mon[36187]: pgmap v2027: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:07:08.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:07:08 smithi193 ceph-mon[36878]: pgmap v2027: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:07:10.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:07:10 smithi118 ceph-mon[31535]: pgmap v2028: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T01:07:10.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:07:10 smithi189 ceph-mon[36187]: pgmap v2028: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T01:07:10.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:07:10 smithi193 ceph-mon[36878]: pgmap v2028: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T01:07:11.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:07:11 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 2526.650435 secs 2024-03-24T01:07:11.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:07:11 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 2526.650435 secs 2024-03-24T01:07:11.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:07:11 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 2526.650435 secs 2024-03-24T01:07:12.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:07:12 smithi118 ceph-mon[31535]: pgmap v2029: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T01:07:12.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:07:12 smithi189 ceph-mon[36187]: pgmap v2029: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T01:07:12.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:07:12 smithi193 ceph-mon[36878]: pgmap v2029: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T01:07:14.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:07:14 smithi118 ceph-mon[31535]: pgmap v2030: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 5 op/s 2024-03-24T01:07:14.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:07:14 smithi189 ceph-mon[36187]: pgmap v2030: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 5 op/s 2024-03-24T01:07:14.460 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:07:14 smithi193 ceph-mon[36878]: pgmap v2030: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 5 op/s 2024-03-24T01:07:16.133 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T01:07:16.132+0000 7f81ee000640 1 -- 172.21.15.118:0/1693767697 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f81c80065b0 con 0x7f81f81052d0 2024-03-24T01:07:16.134 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T01:07:16.132+0000 7f81db400640 1 -- 172.21.15.118:0/659686424 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f81c40062d0 con 0x5622651a13f0 2024-03-24T01:07:16.134 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T01:07:16.133+0000 7f81ef400640 1 -- 172.21.15.118:0/1693767697 <== mon.2 v2:172.21.15.189:3300/0 24 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (secure 0 0 0) 0x7f81c80065b0 con 0x7f81f81052d0 2024-03-24T01:07:16.134 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T01:07:16.133+0000 7f81e4c00640 1 -- 172.21.15.118:0/659686424 <== mon.1 v2:172.21.15.193:3300/0 24 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 409+0+0 (secure 0 0 0) 0x7f81c40062d0 con 0x5622651a13f0 2024-03-24T01:07:16.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:07:16 smithi118 ceph-mon[31535]: pgmap v2031: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T01:07:16.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:07:16 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 2531.650497 secs 2024-03-24T01:07:16.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:07:16 smithi189 ceph-mon[36187]: pgmap v2031: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T01:07:16.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:07:16 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 2531.650497 secs 2024-03-24T01:07:16.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:07:16 smithi193 ceph-mon[36878]: pgmap v2031: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T01:07:16.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:07:16 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 2531.650497 secs 2024-03-24T01:07:18.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:07:18 smithi118 ceph-mon[31535]: pgmap v2032: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:07:18.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:07:18 smithi189 ceph-mon[36187]: pgmap v2032: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:07:18.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:07:18 smithi193 ceph-mon[36878]: pgmap v2032: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:07:20.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:07:20 smithi118 ceph-mon[31535]: pgmap v2033: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T01:07:20.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:07:20 smithi189 ceph-mon[36187]: pgmap v2033: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T01:07:20.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:07:20 smithi193 ceph-mon[36878]: pgmap v2033: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T01:07:21.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:07:21 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 2536.650584 secs 2024-03-24T01:07:21.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:07:21 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 2536.650584 secs 2024-03-24T01:07:21.460 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:07:21 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 2536.650584 secs 2024-03-24T01:07:22.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:07:22 smithi193 ceph-mon[36878]: pgmap v2034: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T01:07:22.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:07:22 smithi118 ceph-mon[31535]: pgmap v2034: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T01:07:22.704 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:07:22 smithi189 ceph-mon[36187]: pgmap v2034: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T01:07:24.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:07:24 smithi193 ceph-mon[36878]: pgmap v2035: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 5 op/s 2024-03-24T01:07:24.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:07:24 smithi118 ceph-mon[31535]: pgmap v2035: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 5 op/s 2024-03-24T01:07:24.704 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:07:24 smithi189 ceph-mon[36187]: pgmap v2035: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 5 op/s 2024-03-24T01:07:26.697 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:07:26 smithi118 ceph-mon[31535]: pgmap v2036: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 6 op/s 2024-03-24T01:07:26.697 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:07:26 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 2541.650675 secs 2024-03-24T01:07:26.704 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:07:26 smithi189 ceph-mon[36187]: pgmap v2036: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 6 op/s 2024-03-24T01:07:26.704 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:07:26 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 2541.650675 secs 2024-03-24T01:07:26.710 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:07:26 smithi193 ceph-mon[36878]: pgmap v2036: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 6 op/s 2024-03-24T01:07:26.711 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:07:26 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 2541.650675 secs 2024-03-24T01:07:28.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:07:28 smithi118 ceph-mon[31535]: pgmap v2037: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-03-24T01:07:28.704 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:07:28 smithi189 ceph-mon[36187]: pgmap v2037: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-03-24T01:07:28.710 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:07:28 smithi193 ceph-mon[36878]: pgmap v2037: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-03-24T01:07:30.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:07:30 smithi193 ceph-mon[36878]: pgmap v2038: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-03-24T01:07:30.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:07:30 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T01:07:30.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:07:30 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T01:07:30.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:07:30 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T01:07:30.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:07:30 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T01:07:30.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:07:30 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T01:07:30.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:07:30 smithi118 ceph-mon[31535]: pgmap v2038: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-03-24T01:07:30.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:07:30 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T01:07:30.697 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:07:30 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T01:07:30.697 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:07:30 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T01:07:30.697 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:07:30 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T01:07:30.697 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:07:30 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T01:07:30.704 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:07:30 smithi189 ceph-mon[36187]: pgmap v2038: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-03-24T01:07:30.704 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:07:30 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T01:07:30.704 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:07:30 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T01:07:30.704 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:07:30 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T01:07:30.704 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:07:30 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T01:07:30.704 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:07:30 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T01:07:31.511 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:07:31 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 2546.650744 secs 2024-03-24T01:07:31.704 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:07:31 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 2546.650744 secs 2024-03-24T01:07:31.711 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:07:31 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 2546.650744 secs 2024-03-24T01:07:32.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:07:32 smithi118 ceph-mon[31535]: pgmap v2039: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 5 op/s 2024-03-24T01:07:32.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:07:32 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T01:07:32.697 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:07:32 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T01:07:32.697 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:07:32 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T01:07:32.697 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:07:32 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T01:07:32.704 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:07:32 smithi189 ceph-mon[36187]: pgmap v2039: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 5 op/s 2024-03-24T01:07:32.704 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:07:32 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T01:07:32.704 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:07:32 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T01:07:32.704 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:07:32 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T01:07:32.704 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:07:32 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T01:07:32.711 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:07:32 smithi193 ceph-mon[36878]: pgmap v2039: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 5 op/s 2024-03-24T01:07:32.711 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:07:32 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T01:07:32.711 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:07:32 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T01:07:32.711 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:07:32 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T01:07:32.711 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:07:32 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T01:07:34.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:07:34 smithi118 ceph-mon[31535]: pgmap v2040: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 5 op/s 2024-03-24T01:07:34.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:07:34 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T01:07:34.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:07:34 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T01:07:34.704 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:07:34 smithi189 ceph-mon[36187]: pgmap v2040: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 5 op/s 2024-03-24T01:07:34.704 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:07:34 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T01:07:34.704 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:07:34 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T01:07:34.711 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:07:34 smithi193 ceph-mon[36878]: pgmap v2040: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 5 op/s 2024-03-24T01:07:34.711 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:07:34 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T01:07:34.711 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:07:34 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T01:07:36.697 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:07:36 smithi118 ceph-mon[31535]: pgmap v2041: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 6 op/s 2024-03-24T01:07:36.697 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:07:36 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 2551.650832 secs 2024-03-24T01:07:36.697 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:07:36 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T01:07:36.697 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:07:36 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T01:07:36.697 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:07:36 smithi118 ceph-mon[31535]: pgmap v2042: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T01:07:36.697 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:07:36 smithi118 ceph-mon[31535]: pgmap v2043: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-03-24T01:07:36.697 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:07:36 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T01:07:36.704 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:07:36 smithi189 ceph-mon[36187]: pgmap v2041: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 6 op/s 2024-03-24T01:07:36.704 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:07:36 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 2551.650832 secs 2024-03-24T01:07:36.704 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:07:36 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T01:07:36.704 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:07:36 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T01:07:36.704 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:07:36 smithi189 ceph-mon[36187]: pgmap v2042: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T01:07:36.704 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:07:36 smithi189 ceph-mon[36187]: pgmap v2043: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-03-24T01:07:36.704 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:07:36 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T01:07:36.711 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:07:36 smithi193 ceph-mon[36878]: pgmap v2041: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 6 op/s 2024-03-24T01:07:36.711 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:07:36 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 2551.650832 secs 2024-03-24T01:07:36.711 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:07:36 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T01:07:36.711 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:07:36 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T01:07:36.711 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:07:36 smithi193 ceph-mon[36878]: pgmap v2042: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T01:07:36.711 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:07:36 smithi193 ceph-mon[36878]: pgmap v2043: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-03-24T01:07:36.711 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:07:36 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T01:07:37.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:07:37 smithi118 ceph-mon[31535]: pgmap v2044: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-03-24T01:07:37.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:07:37 smithi189 ceph-mon[36187]: pgmap v2044: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-03-24T01:07:37.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:07:37 smithi193 ceph-mon[36878]: pgmap v2044: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-03-24T01:07:39.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:07:39 smithi118 ceph-mon[31535]: pgmap v2045: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-03-24T01:07:39.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:07:39 smithi189 ceph-mon[36187]: pgmap v2045: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-03-24T01:07:39.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:07:39 smithi193 ceph-mon[36878]: pgmap v2045: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-03-24T01:07:40.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:07:40 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 2556.650918 secs 2024-03-24T01:07:40.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:07:40 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 2556.650918 secs 2024-03-24T01:07:40.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:07:40 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 2556.650918 secs 2024-03-24T01:07:41.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:07:41 smithi118 ceph-mon[31535]: pgmap v2046: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 5 op/s 2024-03-24T01:07:41.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:07:41 smithi189 ceph-mon[36187]: pgmap v2046: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 5 op/s 2024-03-24T01:07:41.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:07:41 smithi193 ceph-mon[36878]: pgmap v2046: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 5 op/s 2024-03-24T01:07:43.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:07:43 smithi118 ceph-mon[31535]: pgmap v2047: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 5 op/s 2024-03-24T01:07:43.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:07:43 smithi189 ceph-mon[36187]: pgmap v2047: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 5 op/s 2024-03-24T01:07:43.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:07:43 smithi193 ceph-mon[36878]: pgmap v2047: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 5 op/s 2024-03-24T01:07:45.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:07:45 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 2561.651005 secs 2024-03-24T01:07:45.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:07:45 smithi118 ceph-mon[31535]: pgmap v2048: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 5 op/s 2024-03-24T01:07:45.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:07:45 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 2561.651005 secs 2024-03-24T01:07:45.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:07:45 smithi189 ceph-mon[36187]: pgmap v2048: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 5 op/s 2024-03-24T01:07:45.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:07:45 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 2561.651005 secs 2024-03-24T01:07:45.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:07:45 smithi193 ceph-mon[36878]: pgmap v2048: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 5 op/s 2024-03-24T01:07:47.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:07:47 smithi118 ceph-mon[31535]: pgmap v2049: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:07:47.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:07:47 smithi189 ceph-mon[36187]: pgmap v2049: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:07:47.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:07:47 smithi193 ceph-mon[36878]: pgmap v2049: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:07:49.762 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:07:49 smithi193 ceph-mon[36878]: pgmap v2050: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-24T01:07:49.947 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:07:49 smithi118 ceph-mon[31535]: pgmap v2050: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-24T01:07:49.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:07:49 smithi189 ceph-mon[36187]: pgmap v2050: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-24T01:07:50.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:07:50 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 2566.651087 secs 2024-03-24T01:07:50.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:07:50 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 2566.651087 secs 2024-03-24T01:07:50.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:07:50 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 2566.651087 secs 2024-03-24T01:07:51.855 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:07:51 smithi189 ceph-mon[36187]: pgmap v2051: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:07:51.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:07:51 smithi118 ceph-mon[31535]: pgmap v2051: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:07:51.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:07:51 smithi193 ceph-mon[36878]: pgmap v2051: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:07:53.821 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:07:53 smithi118 ceph-mon[31535]: pgmap v2052: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T01:07:53.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:07:53 smithi189 ceph-mon[36187]: pgmap v2052: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T01:07:53.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:07:53 smithi193 ceph-mon[36878]: pgmap v2052: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T01:07:55.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:07:55 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 2571.651176 secs 2024-03-24T01:07:55.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:07:55 smithi118 ceph-mon[31535]: pgmap v2053: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T01:07:55.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:07:55 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 2571.651176 secs 2024-03-24T01:07:55.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:07:55 smithi189 ceph-mon[36187]: pgmap v2053: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T01:07:55.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:07:55 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 2571.651176 secs 2024-03-24T01:07:55.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:07:55 smithi193 ceph-mon[36878]: pgmap v2053: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T01:07:57.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:07:57 smithi118 ceph-mon[31535]: pgmap v2054: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:07:57.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:07:57 smithi189 ceph-mon[36187]: pgmap v2054: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:07:57.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:07:57 smithi193 ceph-mon[36878]: pgmap v2054: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:07:59.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:07:59 smithi118 ceph-mon[31535]: pgmap v2055: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-24T01:07:59.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:07:59 smithi189 ceph-mon[36187]: pgmap v2055: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-24T01:07:59.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:07:59 smithi193 ceph-mon[36878]: pgmap v2055: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-24T01:08:00.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:08:00 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 2576.651259 secs 2024-03-24T01:08:00.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:08:00 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 2576.651259 secs 2024-03-24T01:08:00.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:08:00 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 2576.651259 secs 2024-03-24T01:08:01.848 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:08:01 smithi118 ceph-mon[31535]: pgmap v2056: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:08:01.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:08:01 smithi189 ceph-mon[36187]: pgmap v2056: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:08:01.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:08:01 smithi193 ceph-mon[36878]: pgmap v2056: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:08:04.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:08:04 smithi118 ceph-mon[31535]: pgmap v2057: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T01:08:04.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:08:04 smithi189 ceph-mon[36187]: pgmap v2057: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T01:08:04.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:08:04 smithi193 ceph-mon[36878]: pgmap v2057: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T01:08:05.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:08:05 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 2581.651348 secs 2024-03-24T01:08:05.947 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:08:05 smithi118 ceph-mon[31535]: pgmap v2058: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T01:08:05.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:08:05 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 2581.651348 secs 2024-03-24T01:08:05.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:08:05 smithi189 ceph-mon[36187]: pgmap v2058: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T01:08:05.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:08:05 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 2581.651348 secs 2024-03-24T01:08:05.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:08:05 smithi193 ceph-mon[36878]: pgmap v2058: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T01:08:07.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:08:07 smithi118 ceph-mon[31535]: pgmap v2059: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:08:07.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:08:07 smithi189 ceph-mon[36187]: pgmap v2059: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:08:07.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:08:07 smithi193 ceph-mon[36878]: pgmap v2059: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:08:09.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:08:09 smithi118 ceph-mon[31535]: pgmap v2060: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-24T01:08:09.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:08:09 smithi189 ceph-mon[36187]: pgmap v2060: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-24T01:08:09.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:08:09 smithi193 ceph-mon[36878]: pgmap v2060: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-24T01:08:10.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:08:10 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 2586.651364 secs 2024-03-24T01:08:10.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:08:10 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 2586.651364 secs 2024-03-24T01:08:10.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:08:10 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 2586.651364 secs 2024-03-24T01:08:11.928 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:08:11 smithi118 ceph-mon[31535]: pgmap v2061: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:08:11.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:08:11 smithi189 ceph-mon[36187]: pgmap v2061: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:08:11.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:08:11 smithi193 ceph-mon[36878]: pgmap v2061: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:08:13.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:08:13 smithi118 ceph-mon[31535]: pgmap v2062: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T01:08:13.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:08:13 smithi189 ceph-mon[36187]: pgmap v2062: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T01:08:13.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:08:13 smithi193 ceph-mon[36878]: pgmap v2062: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T01:08:15.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:08:15 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 2591.651451 secs 2024-03-24T01:08:15.947 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:08:15 smithi118 ceph-mon[31535]: pgmap v2063: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T01:08:15.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:08:15 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 2591.651451 secs 2024-03-24T01:08:15.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:08:15 smithi189 ceph-mon[36187]: pgmap v2063: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T01:08:15.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:08:15 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 2591.651451 secs 2024-03-24T01:08:15.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:08:15 smithi193 ceph-mon[36878]: pgmap v2063: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T01:08:17.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:08:17 smithi118 ceph-mon[31535]: pgmap v2064: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:08:17.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:08:17 smithi189 ceph-mon[36187]: pgmap v2064: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:08:17.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:08:17 smithi193 ceph-mon[36878]: pgmap v2064: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:08:19.762 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:08:19 smithi193 ceph-mon[36878]: pgmap v2065: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-24T01:08:19.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:08:19 smithi118 ceph-mon[31535]: pgmap v2065: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-24T01:08:19.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:08:19 smithi189 ceph-mon[36187]: pgmap v2065: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-24T01:08:20.947 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:08:20 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 2596.651492 secs 2024-03-24T01:08:20.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:08:20 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 2596.651492 secs 2024-03-24T01:08:20.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:08:20 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 2596.651492 secs 2024-03-24T01:08:21.855 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:08:21 smithi189 ceph-mon[36187]: pgmap v2066: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:08:21.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:08:21 smithi118 ceph-mon[31535]: pgmap v2066: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:08:21.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:08:21 smithi193 ceph-mon[36878]: pgmap v2066: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:08:23.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:08:23 smithi118 ceph-mon[31535]: pgmap v2067: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T01:08:23.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:08:23 smithi189 ceph-mon[36187]: pgmap v2067: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T01:08:23.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:08:23 smithi193 ceph-mon[36878]: pgmap v2067: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T01:08:25.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:08:25 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 2601.651553 secs 2024-03-24T01:08:25.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:08:25 smithi118 ceph-mon[31535]: pgmap v2068: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T01:08:25.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:08:25 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 2601.651553 secs 2024-03-24T01:08:25.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:08:25 smithi189 ceph-mon[36187]: pgmap v2068: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T01:08:25.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:08:25 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 2601.651553 secs 2024-03-24T01:08:25.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:08:25 smithi193 ceph-mon[36878]: pgmap v2068: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T01:08:28.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:08:28 smithi118 ceph-mon[31535]: pgmap v2069: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:08:28.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:08:28 smithi189 ceph-mon[36187]: pgmap v2069: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:08:28.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:08:28 smithi193 ceph-mon[36878]: pgmap v2069: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:08:30.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:08:30 smithi118 ceph-mon[31535]: pgmap v2070: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-24T01:08:30.947 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:08:30 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 2606.651639 secs 2024-03-24T01:08:30.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:08:30 smithi189 ceph-mon[36187]: pgmap v2070: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-24T01:08:30.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:08:30 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 2606.651639 secs 2024-03-24T01:08:30.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:08:30 smithi193 ceph-mon[36878]: pgmap v2070: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-24T01:08:30.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:08:30 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 2606.651639 secs 2024-03-24T01:08:32.750 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:08:32 smithi118 ceph-mon[31535]: pgmap v2071: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:08:32.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:08:32 smithi189 ceph-mon[36187]: pgmap v2071: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:08:32.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:08:32 smithi193 ceph-mon[36878]: pgmap v2071: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:08:34.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:08:34 smithi118 ceph-mon[31535]: pgmap v2072: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T01:08:34.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:08:34 smithi189 ceph-mon[36187]: pgmap v2072: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T01:08:34.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:08:34 smithi193 ceph-mon[36878]: pgmap v2072: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T01:08:35.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:08:35 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 2611.651730 secs 2024-03-24T01:08:35.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:08:35 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 2611.651730 secs 2024-03-24T01:08:35.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:08:35 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 2611.651730 secs 2024-03-24T01:08:36.947 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:08:36 smithi118 ceph-mon[31535]: pgmap v2073: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T01:08:36.947 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:08:36 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T01:08:36.947 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:08:36 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T01:08:36.947 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:08:36 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T01:08:36.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:08:36 smithi189 ceph-mon[36187]: pgmap v2073: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T01:08:36.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:08:36 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T01:08:36.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:08:36 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T01:08:36.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:08:36 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T01:08:36.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:08:36 smithi193 ceph-mon[36878]: pgmap v2073: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T01:08:36.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:08:36 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T01:08:36.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:08:36 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T01:08:36.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:08:36 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T01:08:37.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:08:37 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T01:08:37.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:08:37 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T01:08:37.947 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:08:37 smithi118 ceph-mon[31535]: pgmap v2074: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 5 op/s 2024-03-24T01:08:37.947 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:08:37 smithi118 ceph-mon[31535]: pgmap v2075: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 6 op/s 2024-03-24T01:08:37.947 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:08:37 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T01:08:37.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:08:37 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T01:08:37.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:08:37 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T01:08:37.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:08:37 smithi189 ceph-mon[36187]: pgmap v2074: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 5 op/s 2024-03-24T01:08:37.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:08:37 smithi189 ceph-mon[36187]: pgmap v2075: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 6 op/s 2024-03-24T01:08:37.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:08:37 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T01:08:37.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:08:37 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T01:08:37.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:08:37 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T01:08:37.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:08:37 smithi193 ceph-mon[36878]: pgmap v2074: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 5 op/s 2024-03-24T01:08:37.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:08:37 smithi193 ceph-mon[36878]: pgmap v2075: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 6 op/s 2024-03-24T01:08:37.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:08:37 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T01:08:40.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:08:40 smithi118 ceph-mon[31535]: pgmap v2076: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 5 op/s 2024-03-24T01:08:40.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:08:40 smithi189 ceph-mon[36187]: pgmap v2076: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 5 op/s 2024-03-24T01:08:40.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:08:40 smithi193 ceph-mon[36878]: pgmap v2076: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 5 op/s 2024-03-24T01:08:41.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:08:41 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 2616.651819 secs 2024-03-24T01:08:41.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:08:41 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 2616.651819 secs 2024-03-24T01:08:41.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:08:41 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 2616.651819 secs 2024-03-24T01:08:42.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:08:42 smithi118 ceph-mon[31535]: pgmap v2077: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 6 op/s 2024-03-24T01:08:42.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:08:42 smithi189 ceph-mon[36187]: pgmap v2077: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 6 op/s 2024-03-24T01:08:42.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:08:42 smithi193 ceph-mon[36878]: pgmap v2077: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 6 op/s 2024-03-24T01:08:44.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:08:44 smithi118 ceph-mon[31535]: pgmap v2078: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 5 op/s 2024-03-24T01:08:44.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:08:44 smithi189 ceph-mon[36187]: pgmap v2078: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 5 op/s 2024-03-24T01:08:44.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:08:44 smithi193 ceph-mon[36878]: pgmap v2078: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 5 op/s 2024-03-24T01:08:46.449 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:08:46 smithi118 ceph-mon[31535]: pgmap v2079: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 6 op/s 2024-03-24T01:08:46.449 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:08:46 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 2621.651912 secs 2024-03-24T01:08:46.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:08:46 smithi189 ceph-mon[36187]: pgmap v2079: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 6 op/s 2024-03-24T01:08:46.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:08:46 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 2621.651912 secs 2024-03-24T01:08:46.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:08:46 smithi193 ceph-mon[36878]: pgmap v2079: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 6 op/s 2024-03-24T01:08:46.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:08:46 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 2621.651912 secs 2024-03-24T01:08:48.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:08:48 smithi118 ceph-mon[31535]: pgmap v2080: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 5 op/s 2024-03-24T01:08:48.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:08:48 smithi189 ceph-mon[36187]: pgmap v2080: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 5 op/s 2024-03-24T01:08:48.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:08:48 smithi193 ceph-mon[36878]: pgmap v2080: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 5 op/s 2024-03-24T01:08:50.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:08:50 smithi118 ceph-mon[31535]: pgmap v2081: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T01:08:50.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:08:50 smithi189 ceph-mon[36187]: pgmap v2081: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T01:08:50.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:08:50 smithi193 ceph-mon[36878]: pgmap v2081: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T01:08:51.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:08:51 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 2626.652002 secs 2024-03-24T01:08:51.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:08:51 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 2626.652002 secs 2024-03-24T01:08:51.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:08:51 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 2626.652002 secs 2024-03-24T01:08:52.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:08:52 smithi118 ceph-mon[31535]: pgmap v2082: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T01:08:52.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:08:52 smithi189 ceph-mon[36187]: pgmap v2082: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T01:08:52.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:08:52 smithi193 ceph-mon[36878]: pgmap v2082: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T01:08:54.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:08:54 smithi118 ceph-mon[31535]: pgmap v2083: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 5 op/s 2024-03-24T01:08:54.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:08:54 smithi189 ceph-mon[36187]: pgmap v2083: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 5 op/s 2024-03-24T01:08:54.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:08:54 smithi193 ceph-mon[36878]: pgmap v2083: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 5 op/s 2024-03-24T01:08:56.249 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:08:56 smithi118 ceph-mon[31535]: pgmap v2084: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T01:08:56.250 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:08:56 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 2631.652089 secs 2024-03-24T01:08:56.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:08:56 smithi189 ceph-mon[36187]: pgmap v2084: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T01:08:56.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:08:56 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 2631.652089 secs 2024-03-24T01:08:56.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:08:56 smithi193 ceph-mon[36878]: pgmap v2084: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T01:08:56.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:08:56 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 2631.652089 secs 2024-03-24T01:08:58.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:08:58 smithi118 ceph-mon[31535]: pgmap v2085: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:08:58.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:08:58 smithi189 ceph-mon[36187]: pgmap v2085: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:08:58.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:08:58 smithi193 ceph-mon[36878]: pgmap v2085: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:09:00.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:09:00 smithi118 ceph-mon[31535]: pgmap v2086: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T01:09:00.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:09:00 smithi189 ceph-mon[36187]: pgmap v2086: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T01:09:00.460 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:09:00 smithi193 ceph-mon[36878]: pgmap v2086: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T01:09:01.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:09:01 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 2636.652177 secs 2024-03-24T01:09:01.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:09:01 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 2636.652177 secs 2024-03-24T01:09:01.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:09:01 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 2636.652177 secs 2024-03-24T01:09:02.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:09:02 smithi118 ceph-mon[31535]: pgmap v2087: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T01:09:02.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:09:02 smithi189 ceph-mon[36187]: pgmap v2087: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T01:09:02.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:09:02 smithi193 ceph-mon[36878]: pgmap v2087: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T01:09:04.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:09:04 smithi118 ceph-mon[31535]: pgmap v2088: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 5 op/s 2024-03-24T01:09:04.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:09:04 smithi189 ceph-mon[36187]: pgmap v2088: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 5 op/s 2024-03-24T01:09:04.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:09:04 smithi193 ceph-mon[36878]: pgmap v2088: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 5 op/s 2024-03-24T01:09:06.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:09:06 smithi118 ceph-mon[31535]: pgmap v2089: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T01:09:06.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:09:06 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 2641.652264 secs 2024-03-24T01:09:06.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:09:06 smithi189 ceph-mon[36187]: pgmap v2089: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T01:09:06.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:09:06 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 2641.652264 secs 2024-03-24T01:09:06.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:09:06 smithi193 ceph-mon[36878]: pgmap v2089: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T01:09:06.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:09:06 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 2641.652264 secs 2024-03-24T01:09:08.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:09:08 smithi118 ceph-mon[31535]: pgmap v2090: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:09:08.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:09:08 smithi189 ceph-mon[36187]: pgmap v2090: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:09:08.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:09:08 smithi193 ceph-mon[36878]: pgmap v2090: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:09:10.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:09:10 smithi118 ceph-mon[31535]: pgmap v2091: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T01:09:10.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:09:10 smithi189 ceph-mon[36187]: pgmap v2091: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T01:09:10.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:09:10 smithi193 ceph-mon[36878]: pgmap v2091: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T01:09:11.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:09:11 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 2646.652324 secs 2024-03-24T01:09:11.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:09:11 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 2646.652324 secs 2024-03-24T01:09:11.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:09:11 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 2646.652324 secs 2024-03-24T01:09:12.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:09:12 smithi118 ceph-mon[31535]: pgmap v2092: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T01:09:12.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:09:12 smithi189 ceph-mon[36187]: pgmap v2092: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T01:09:12.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:09:12 smithi193 ceph-mon[36878]: pgmap v2092: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T01:09:14.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:09:14 smithi118 ceph-mon[31535]: pgmap v2093: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 5 op/s 2024-03-24T01:09:14.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:09:14 smithi189 ceph-mon[36187]: pgmap v2093: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 5 op/s 2024-03-24T01:09:14.460 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:09:14 smithi193 ceph-mon[36878]: pgmap v2093: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 5 op/s 2024-03-24T01:09:16.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:09:16 smithi118 ceph-mon[31535]: pgmap v2094: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T01:09:16.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:09:16 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 2651.652388 secs 2024-03-24T01:09:16.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:09:16 smithi189 ceph-mon[36187]: pgmap v2094: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T01:09:16.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:09:16 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 2651.652388 secs 2024-03-24T01:09:16.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:09:16 smithi193 ceph-mon[36878]: pgmap v2094: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T01:09:16.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:09:16 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 2651.652388 secs 2024-03-24T01:09:18.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:09:18 smithi189 ceph-mon[36187]: pgmap v2095: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:09:18.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:09:18 smithi193 ceph-mon[36878]: pgmap v2095: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:09:18.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:09:18 smithi118 ceph-mon[31535]: pgmap v2095: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:09:20.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:09:20 smithi189 ceph-mon[36187]: pgmap v2096: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T01:09:20.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:09:20 smithi193 ceph-mon[36878]: pgmap v2096: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T01:09:20.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:09:20 smithi118 ceph-mon[31535]: pgmap v2096: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T01:09:21.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:09:21 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 2656.652450 secs 2024-03-24T01:09:21.704 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:09:21 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 2656.652450 secs 2024-03-24T01:09:21.710 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:09:21 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 2656.652450 secs 2024-03-24T01:09:22.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:09:22 smithi118 ceph-mon[31535]: pgmap v2097: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T01:09:22.704 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:09:22 smithi189 ceph-mon[36187]: pgmap v2097: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T01:09:22.711 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:09:22 smithi193 ceph-mon[36878]: pgmap v2097: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T01:09:24.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:09:24 smithi118 ceph-mon[31535]: pgmap v2098: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 5 op/s 2024-03-24T01:09:24.704 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:09:24 smithi189 ceph-mon[36187]: pgmap v2098: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 5 op/s 2024-03-24T01:09:24.711 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:09:24 smithi193 ceph-mon[36878]: pgmap v2098: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 5 op/s 2024-03-24T01:09:26.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:09:26 smithi118 ceph-mon[31535]: pgmap v2099: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T01:09:26.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:09:26 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 2661.652515 secs 2024-03-24T01:09:26.704 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:09:26 smithi189 ceph-mon[36187]: pgmap v2099: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T01:09:26.704 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:09:26 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 2661.652515 secs 2024-03-24T01:09:26.711 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:09:26 smithi193 ceph-mon[36878]: pgmap v2099: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T01:09:26.711 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:09:26 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 2661.652515 secs 2024-03-24T01:09:28.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:09:28 smithi118 ceph-mon[31535]: pgmap v2100: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:09:28.704 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:09:28 smithi189 ceph-mon[36187]: pgmap v2100: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:09:28.711 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:09:28 smithi193 ceph-mon[36878]: pgmap v2100: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:09:30.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:09:30 smithi118 ceph-mon[31535]: pgmap v2101: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T01:09:30.704 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:09:30 smithi189 ceph-mon[36187]: pgmap v2101: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T01:09:30.711 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:09:30 smithi193 ceph-mon[36878]: pgmap v2101: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T01:09:31.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:09:31 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 2666.652601 secs 2024-03-24T01:09:31.704 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:09:31 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 2666.652601 secs 2024-03-24T01:09:31.710 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:09:31 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 2666.652601 secs 2024-03-24T01:09:32.598 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:09:32 smithi118 ceph-mon[31535]: pgmap v2102: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T01:09:32.704 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:09:32 smithi189 ceph-mon[36187]: pgmap v2102: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T01:09:32.711 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:09:32 smithi193 ceph-mon[36878]: pgmap v2102: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T01:09:34.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:09:34 smithi118 ceph-mon[31535]: pgmap v2103: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 5 op/s 2024-03-24T01:09:34.704 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:09:34 smithi189 ceph-mon[36187]: pgmap v2103: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 5 op/s 2024-03-24T01:09:34.711 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:09:34 smithi193 ceph-mon[36878]: pgmap v2103: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 5 op/s 2024-03-24T01:09:36.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:09:36 smithi118 ceph-mon[31535]: pgmap v2104: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T01:09:36.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:09:36 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 2671.652660 secs 2024-03-24T01:09:36.704 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:09:36 smithi189 ceph-mon[36187]: pgmap v2104: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T01:09:36.704 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:09:36 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 2671.652660 secs 2024-03-24T01:09:36.711 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:09:36 smithi193 ceph-mon[36878]: pgmap v2104: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T01:09:36.711 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:09:36 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 2671.652660 secs 2024-03-24T01:09:38.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:09:38 smithi118 ceph-mon[31535]: pgmap v2105: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:09:38.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:09:38 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T01:09:38.704 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:09:38 smithi189 ceph-mon[36187]: pgmap v2105: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:09:38.704 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:09:38 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T01:09:38.710 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:09:38 smithi193 ceph-mon[36878]: pgmap v2105: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:09:38.711 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:09:38 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T01:09:39.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:09:39 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T01:09:39.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:09:39 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T01:09:39.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:09:39 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T01:09:39.704 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:09:39 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T01:09:39.704 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:09:39 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T01:09:39.704 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:09:39 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T01:09:39.711 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:09:39 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T01:09:39.711 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:09:39 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T01:09:39.711 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:09:39 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T01:09:40.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:09:40 smithi118 ceph-mon[31535]: pgmap v2106: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 6 op/s 2024-03-24T01:09:40.704 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:09:40 smithi189 ceph-mon[36187]: pgmap v2106: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 6 op/s 2024-03-24T01:09:40.711 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:09:40 smithi193 ceph-mon[36878]: pgmap v2106: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 6 op/s 2024-03-24T01:09:41.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:09:41 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 2676.652722 secs 2024-03-24T01:09:41.704 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:09:41 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 2676.652722 secs 2024-03-24T01:09:41.711 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:09:41 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 2676.652722 secs 2024-03-24T01:09:42.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:09:42 smithi118 ceph-mon[31535]: pgmap v2107: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 6 op/s 2024-03-24T01:09:42.704 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:09:42 smithi189 ceph-mon[36187]: pgmap v2107: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 6 op/s 2024-03-24T01:09:42.711 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:09:42 smithi193 ceph-mon[36878]: pgmap v2107: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 6 op/s 2024-03-24T01:09:44.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:09:44 smithi118 ceph-mon[31535]: pgmap v2108: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 5 op/s 2024-03-24T01:09:44.704 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:09:44 smithi189 ceph-mon[36187]: pgmap v2108: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 5 op/s 2024-03-24T01:09:44.711 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:09:44 smithi193 ceph-mon[36878]: pgmap v2108: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 5 op/s 2024-03-24T01:09:46.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:09:46 smithi118 ceph-mon[31535]: pgmap v2109: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 6 op/s 2024-03-24T01:09:46.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:09:46 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 2681.652806 secs 2024-03-24T01:09:46.704 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:09:46 smithi189 ceph-mon[36187]: pgmap v2109: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 6 op/s 2024-03-24T01:09:46.704 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:09:46 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 2681.652806 secs 2024-03-24T01:09:46.711 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:09:46 smithi193 ceph-mon[36878]: pgmap v2109: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 6 op/s 2024-03-24T01:09:46.711 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:09:46 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 2681.652806 secs 2024-03-24T01:09:48.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:09:48 smithi118 ceph-mon[31535]: pgmap v2110: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 5 op/s 2024-03-24T01:09:48.704 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:09:48 smithi189 ceph-mon[36187]: pgmap v2110: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 5 op/s 2024-03-24T01:09:48.711 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:09:48 smithi193 ceph-mon[36878]: pgmap v2110: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 5 op/s 2024-03-24T01:09:50.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:09:50 smithi118 ceph-mon[31535]: pgmap v2111: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 6 op/s 2024-03-24T01:09:50.704 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:09:50 smithi189 ceph-mon[36187]: pgmap v2111: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 6 op/s 2024-03-24T01:09:50.711 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:09:50 smithi193 ceph-mon[36878]: pgmap v2111: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 6 op/s 2024-03-24T01:09:51.605 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:09:51 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 2686.652888 secs 2024-03-24T01:09:51.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:09:51 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 2686.652888 secs 2024-03-24T01:09:51.710 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:09:51 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 2686.652888 secs 2024-03-24T01:09:52.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:09:52 smithi118 ceph-mon[31535]: pgmap v2112: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 6 op/s 2024-03-24T01:09:52.704 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:09:52 smithi189 ceph-mon[36187]: pgmap v2112: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 6 op/s 2024-03-24T01:09:52.710 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:09:52 smithi193 ceph-mon[36878]: pgmap v2112: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 6 op/s 2024-03-24T01:09:54.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:09:54 smithi118 ceph-mon[31535]: pgmap v2113: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 5 op/s 2024-03-24T01:09:54.704 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:09:54 smithi189 ceph-mon[36187]: pgmap v2113: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 5 op/s 2024-03-24T01:09:54.711 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:09:54 smithi193 ceph-mon[36878]: pgmap v2113: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 5 op/s 2024-03-24T01:09:56.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:09:56 smithi118 ceph-mon[31535]: pgmap v2114: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-24T01:09:56.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:09:56 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 2691.652978 secs 2024-03-24T01:09:56.704 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:09:56 smithi189 ceph-mon[36187]: pgmap v2114: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-24T01:09:56.704 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:09:56 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 2691.652978 secs 2024-03-24T01:09:56.711 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:09:56 smithi193 ceph-mon[36878]: pgmap v2114: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-24T01:09:56.711 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:09:56 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 2691.652978 secs 2024-03-24T01:09:58.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:09:58 smithi118 ceph-mon[31535]: pgmap v2115: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-03-24T01:09:58.704 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:09:58 smithi189 ceph-mon[36187]: pgmap v2115: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-03-24T01:09:58.711 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:09:58 smithi193 ceph-mon[36878]: pgmap v2115: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-03-24T01:10:00.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:10:00 smithi118 ceph-mon[31535]: pgmap v2116: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-03-24T01:10:00.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:10:00 smithi118 ceph-mon[31535]: overall HEALTH_WARN 6 failed cephadm daemon(s); 1 MDSs report slow requests 2024-03-24T01:10:00.704 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:10:00 smithi189 ceph-mon[36187]: pgmap v2116: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-03-24T01:10:00.704 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:10:00 smithi189 ceph-mon[36187]: overall HEALTH_WARN 6 failed cephadm daemon(s); 1 MDSs report slow requests 2024-03-24T01:10:00.711 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:10:00 smithi193 ceph-mon[36878]: pgmap v2116: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-03-24T01:10:00.711 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:10:00 smithi193 ceph-mon[36878]: overall HEALTH_WARN 6 failed cephadm daemon(s); 1 MDSs report slow requests 2024-03-24T01:10:01.697 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:10:01 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 2696.653040 secs 2024-03-24T01:10:01.704 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:10:01 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 2696.653040 secs 2024-03-24T01:10:01.711 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:10:01 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 2696.653040 secs 2024-03-24T01:10:02.598 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:10:02 smithi118 ceph-mon[31535]: pgmap v2117: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T01:10:02.704 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:10:02 smithi189 ceph-mon[36187]: pgmap v2117: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T01:10:02.711 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:10:02 smithi193 ceph-mon[36878]: pgmap v2117: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T01:10:04.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:10:04 smithi118 ceph-mon[31535]: pgmap v2118: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T01:10:04.704 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:10:04 smithi189 ceph-mon[36187]: pgmap v2118: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T01:10:04.711 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:10:04 smithi193 ceph-mon[36878]: pgmap v2118: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T01:10:06.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:10:06 smithi118 ceph-mon[31535]: pgmap v2119: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-03-24T01:10:06.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:10:06 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 2701.653096 secs 2024-03-24T01:10:06.704 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:10:06 smithi189 ceph-mon[36187]: pgmap v2119: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-03-24T01:10:06.704 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:10:06 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 2701.653096 secs 2024-03-24T01:10:06.711 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:10:06 smithi193 ceph-mon[36878]: pgmap v2119: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-03-24T01:10:06.711 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:10:06 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 2701.653096 secs 2024-03-24T01:10:08.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:10:08 smithi118 ceph-mon[31535]: pgmap v2120: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:10:08.704 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:10:08 smithi189 ceph-mon[36187]: pgmap v2120: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:10:08.711 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:10:08 smithi193 ceph-mon[36878]: pgmap v2120: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:10:10.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:10:10 smithi118 ceph-mon[31535]: pgmap v2121: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T01:10:10.704 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:10:10 smithi189 ceph-mon[36187]: pgmap v2121: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T01:10:10.710 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:10:10 smithi193 ceph-mon[36878]: pgmap v2121: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T01:10:11.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:10:11 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 2706.653181 secs 2024-03-24T01:10:11.704 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:10:11 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 2706.653181 secs 2024-03-24T01:10:11.710 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:10:11 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 2706.653181 secs 2024-03-24T01:10:12.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:10:12 smithi118 ceph-mon[31535]: pgmap v2122: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T01:10:12.704 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:10:12 smithi189 ceph-mon[36187]: pgmap v2122: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T01:10:12.711 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:10:12 smithi193 ceph-mon[36878]: pgmap v2122: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T01:10:14.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:10:14 smithi118 ceph-mon[31535]: pgmap v2123: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T01:10:14.704 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:10:14 smithi189 ceph-mon[36187]: pgmap v2123: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T01:10:14.711 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:10:14 smithi193 ceph-mon[36878]: pgmap v2123: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T01:10:15.736 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T01:10:15.735+0000 7f81f6000640 1 -- 172.21.15.118:0/1693767697 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7f81dc0a3000 msgr2=0x7f81dc0a33e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 12 2024-03-24T01:10:15.736 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T01:10:15.735+0000 7f81f6000640 1 -- 172.21.15.118:0/1693767697 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7f81dc0a3000 msgr2=0x7f81dc0a33e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-03-24T01:10:15.736 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T01:10:15.735+0000 7f81f6000640 1 --2- 172.21.15.118:0/1693767697 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7f81dc0a3000 0x7f81dc0a33e0 secure :-1 s=READY pgs=441 cs=0 l=1 rev1=1 crypto rx=0x562265215970 tx=0x7f81e0015040 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-03-24T01:10:15.737 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T01:10:15.735+0000 7f81f6000640 1 --2- 172.21.15.118:0/1693767697 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7f81dc0a3000 0x7f81dc0a33e0 secure :-1 s=READY pgs=441 cs=0 l=1 rev1=1 crypto rx=0x562265215970 tx=0x7f81e0015040 comp rx=0 tx=0).stop 2024-03-24T01:10:15.737 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T01:10:15.735+0000 7f81f6000640 1 -- 172.21.15.118:0/1693767697 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7f81dc0a3000 msgr2=0x7f81dc0a33e0 unknown :-1 s=STATE_CLOSED l=1).tick see no progress in more than 10000000 us during connecting to v2:172.21.15.118:6800/2112912943, fault. 2024-03-24T01:10:15.737 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T01:10:15.735+0000 7f81ef400640 1 -- 172.21.15.118:0/1693767697 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7f81dc0a3000 msgr2=0x7f81dc0a33e0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-03-24T01:10:15.737 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T01:10:15.735+0000 7f81ef400640 1 --2- 172.21.15.118:0/1693767697 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7f81dc0a3000 0x7f81dc0a33e0 unknown :-1 s=CLOSED pgs=441 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:10:15.737 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T01:10:15.735+0000 7f81ef400640 1 --2- 172.21.15.118:0/1693767697 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7f81dc0a4c80 0x7f81dc0b05c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:10:15.737 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T01:10:15.736+0000 7f81f6000640 1 --2- 172.21.15.118:0/1693767697 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7f81dc0a4c80 0x7f81dc0b05c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:10:15.739 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T01:10:15.738+0000 7f81f6000640 1 --2- 172.21.15.118:0/1693767697 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7f81dc0a4c80 0x7f81dc0b05c0 secure :-1 s=READY pgs=462 cs=0 l=1 rev1=1 crypto rx=0x7f81e0009b70 tx=0x7f81e003a040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:10:15.760 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T01:10:15.759+0000 7f81f6000640 1 -- 172.21.15.118:0/659686424 >> [v2:172.21.15.189:6838/4074484168,v1:172.21.15.189:6839/4074484168] conn(0x5622651b71e0 msgr2=0x562265215110 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).tick idle (900000983) for more than 900000000 us, fault. 2024-03-24T01:10:15.760 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T01:10:15.759+0000 7f81f6000640 1 --2- 172.21.15.118:0/659686424 >> [v2:172.21.15.189:6838/4074484168,v1:172.21.15.189:6839/4074484168] conn(0x5622651b71e0 0x562265215110 unknown :-1 s=READY pgs=561 cs=1 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault initiating reconnect 2024-03-24T01:10:15.760 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T01:10:15.759+0000 7f81f6000640 1 --2- 172.21.15.118:0/659686424 >> [v2:172.21.15.189:6838/4074484168,v1:172.21.15.189:6839/4074484168] conn(0x5622651b71e0 0x562265215110 unknown :-1 s=BANNER_CONNECTING pgs=561 cs=2 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:10:15.761 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T01:10:15.760+0000 7f81f6000640 1 --2- 172.21.15.118:0/659686424 >> [v2:172.21.15.189:6838/4074484168,v1:172.21.15.189:6839/4074484168] conn(0x5622651b71e0 0x562265215110 crc :-1 s=READY pgs=561 cs=2 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=76f3d534289740f0 server_cookie=b31a2335a59552de in_seq=0 out_seq=0 2024-03-24T01:10:15.761 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T01:10:15.760+0000 7f81e4c00640 10 client.27094 ms_handle_connect on v2:172.21.15.189:6838/4074484168 2024-03-24T01:10:16.161 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T01:10:16.160+0000 7f81db400640 1 -- 172.21.15.118:0/659686424 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f81c40062d0 con 0x5622651a13f0 2024-03-24T01:10:16.162 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T01:10:16.160+0000 7f81ee000640 1 -- 172.21.15.118:0/1693767697 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f81c80065b0 con 0x7f81f81052d0 2024-03-24T01:10:16.162 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T01:10:16.161+0000 7f81ef400640 1 -- 172.21.15.118:0/1693767697 <== mon.2 v2:172.21.15.189:3300/0 25 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (secure 0 0 0) 0x7f81c80065b0 con 0x7f81f81052d0 2024-03-24T01:10:16.162 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T01:10:16.161+0000 7f81e4c00640 1 -- 172.21.15.118:0/659686424 <== mon.1 v2:172.21.15.193:3300/0 25 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 409+0+0 (secure 0 0 0) 0x7f81c40062d0 con 0x5622651a13f0 2024-03-24T01:10:16.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:10:16 smithi118 ceph-mon[31535]: pgmap v2124: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-03-24T01:10:16.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:10:16 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 2711.653196 secs 2024-03-24T01:10:16.704 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:10:16 smithi189 ceph-mon[36187]: pgmap v2124: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-03-24T01:10:16.704 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:10:16 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 2711.653196 secs 2024-03-24T01:10:16.711 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:10:16 smithi193 ceph-mon[36878]: pgmap v2124: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-03-24T01:10:16.711 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:10:16 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 2711.653196 secs 2024-03-24T01:10:18.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:10:18 smithi118 ceph-mon[31535]: pgmap v2125: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:10:18.704 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:10:18 smithi189 ceph-mon[36187]: pgmap v2125: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:10:18.710 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:10:18 smithi193 ceph-mon[36878]: pgmap v2125: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:10:20.697 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:10:20 smithi118 ceph-mon[31535]: pgmap v2126: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T01:10:20.704 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:10:20 smithi189 ceph-mon[36187]: pgmap v2126: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T01:10:20.711 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:10:20 smithi193 ceph-mon[36878]: pgmap v2126: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T01:10:21.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:10:21 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 2716.653256 secs 2024-03-24T01:10:21.704 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:10:21 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 2716.653256 secs 2024-03-24T01:10:21.711 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:10:21 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 2716.653256 secs 2024-03-24T01:10:22.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:10:22 smithi118 ceph-mon[31535]: pgmap v2127: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T01:10:22.704 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:10:22 smithi189 ceph-mon[36187]: pgmap v2127: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T01:10:22.711 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:10:22 smithi193 ceph-mon[36878]: pgmap v2127: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T01:10:24.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:10:24 smithi118 ceph-mon[31535]: pgmap v2128: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T01:10:24.704 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:10:24 smithi189 ceph-mon[36187]: pgmap v2128: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T01:10:24.710 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:10:24 smithi193 ceph-mon[36878]: pgmap v2128: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T01:10:26.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:10:26 smithi118 ceph-mon[31535]: pgmap v2129: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-03-24T01:10:26.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:10:26 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 2721.653340 secs 2024-03-24T01:10:26.704 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:10:26 smithi189 ceph-mon[36187]: pgmap v2129: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-03-24T01:10:26.704 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:10:26 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 2721.653340 secs 2024-03-24T01:10:26.711 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:10:26 smithi193 ceph-mon[36878]: pgmap v2129: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-03-24T01:10:26.711 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:10:26 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 2721.653340 secs 2024-03-24T01:10:28.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:10:28 smithi118 ceph-mon[31535]: pgmap v2130: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:10:28.704 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:10:28 smithi189 ceph-mon[36187]: pgmap v2130: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:10:28.710 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:10:28 smithi193 ceph-mon[36878]: pgmap v2130: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:10:30.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:10:30 smithi118 ceph-mon[31535]: pgmap v2131: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T01:10:30.704 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:10:30 smithi189 ceph-mon[36187]: pgmap v2131: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T01:10:30.710 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:10:30 smithi193 ceph-mon[36878]: pgmap v2131: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T01:10:31.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:10:31 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 2726.653430 secs 2024-03-24T01:10:31.704 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:10:31 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 2726.653430 secs 2024-03-24T01:10:31.711 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:10:31 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 2726.653430 secs 2024-03-24T01:10:32.598 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:10:32 smithi118 ceph-mon[31535]: pgmap v2132: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T01:10:32.704 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:10:32 smithi189 ceph-mon[36187]: pgmap v2132: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T01:10:32.711 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:10:32 smithi193 ceph-mon[36878]: pgmap v2132: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T01:10:34.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:10:34 smithi118 ceph-mon[31535]: pgmap v2133: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T01:10:34.704 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:10:34 smithi189 ceph-mon[36187]: pgmap v2133: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T01:10:34.710 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:10:34 smithi193 ceph-mon[36878]: pgmap v2133: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T01:10:36.698 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:10:36 smithi118 ceph-mon[31535]: pgmap v2134: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-03-24T01:10:36.698 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:10:36 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 2731.653517 secs 2024-03-24T01:10:36.704 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:10:36 smithi189 ceph-mon[36187]: pgmap v2134: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-03-24T01:10:36.704 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:10:36 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 2731.653517 secs 2024-03-24T01:10:36.711 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:10:36 smithi193 ceph-mon[36878]: pgmap v2134: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-03-24T01:10:36.711 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:10:36 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 2731.653517 secs 2024-03-24T01:10:38.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:10:38 smithi118 ceph-mon[31535]: pgmap v2135: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:10:38.704 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:10:38 smithi189 ceph-mon[36187]: pgmap v2135: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:10:38.710 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:10:38 smithi193 ceph-mon[36878]: pgmap v2135: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:10:39.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:10:39 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T01:10:39.704 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:10:39 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T01:10:39.711 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:10:39 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T01:10:40.704 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:10:40 smithi189 ceph-mon[36187]: pgmap v2136: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T01:10:40.704 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:10:40 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T01:10:40.704 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:10:40 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T01:10:40.704 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:10:40 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T01:10:40.710 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:10:40 smithi193 ceph-mon[36878]: pgmap v2136: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T01:10:40.711 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:10:40 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T01:10:40.711 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:10:40 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T01:10:40.711 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:10:40 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T01:10:40.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:10:40 smithi118 ceph-mon[31535]: pgmap v2136: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T01:10:40.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:10:40 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T01:10:40.947 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:10:40 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T01:10:40.947 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:10:40 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T01:10:41.704 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:10:41 smithi189 ceph-mon[36187]: pgmap v2137: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 6 op/s 2024-03-24T01:10:41.704 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:10:41 smithi189 ceph-mon[36187]: pgmap v2138: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-03-24T01:10:41.704 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:10:41 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 2736.653609 secs 2024-03-24T01:10:41.711 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:10:41 smithi193 ceph-mon[36878]: pgmap v2137: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 6 op/s 2024-03-24T01:10:41.711 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:10:41 smithi193 ceph-mon[36878]: pgmap v2138: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-03-24T01:10:41.711 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:10:41 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 2736.653609 secs 2024-03-24T01:10:41.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:10:41 smithi118 ceph-mon[31535]: pgmap v2137: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 6 op/s 2024-03-24T01:10:41.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:10:41 smithi118 ceph-mon[31535]: pgmap v2138: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-03-24T01:10:41.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:10:41 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 2736.653609 secs 2024-03-24T01:10:43.711 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:10:43 smithi193 ceph-mon[36878]: pgmap v2139: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 5 op/s 2024-03-24T01:10:43.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:10:43 smithi118 ceph-mon[31535]: pgmap v2139: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 5 op/s 2024-03-24T01:10:43.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:10:43 smithi189 ceph-mon[36187]: pgmap v2139: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 5 op/s 2024-03-24T01:10:45.711 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:10:45 smithi193 ceph-mon[36878]: pgmap v2140: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 5 op/s 2024-03-24T01:10:45.711 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:10:45 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 2741.653692 secs 2024-03-24T01:10:45.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:10:45 smithi118 ceph-mon[31535]: pgmap v2140: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 5 op/s 2024-03-24T01:10:45.947 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:10:45 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 2741.653692 secs 2024-03-24T01:10:45.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:10:45 smithi189 ceph-mon[36187]: pgmap v2140: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 5 op/s 2024-03-24T01:10:45.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:10:45 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 2741.653692 secs 2024-03-24T01:10:47.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:10:47 smithi118 ceph-mon[31535]: pgmap v2141: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-03-24T01:10:47.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:10:47 smithi189 ceph-mon[36187]: pgmap v2141: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-03-24T01:10:47.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:10:47 smithi193 ceph-mon[36878]: pgmap v2141: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-03-24T01:10:48.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:10:48 smithi118 ceph-mon[31535]: pgmap v2142: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 5 op/s 2024-03-24T01:10:48.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:10:48 smithi189 ceph-mon[36187]: pgmap v2142: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 5 op/s 2024-03-24T01:10:48.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:10:48 smithi193 ceph-mon[36878]: pgmap v2142: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 5 op/s 2024-03-24T01:10:51.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:10:51 smithi118 ceph-mon[31535]: pgmap v2143: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 5 op/s 2024-03-24T01:10:51.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:10:51 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 2746.653734 secs 2024-03-24T01:10:51.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:10:51 smithi189 ceph-mon[36187]: pgmap v2143: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 5 op/s 2024-03-24T01:10:51.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:10:51 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 2746.653734 secs 2024-03-24T01:10:51.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:10:51 smithi193 ceph-mon[36878]: pgmap v2143: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 5 op/s 2024-03-24T01:10:51.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:10:51 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 2746.653734 secs 2024-03-24T01:10:53.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:10:53 smithi118 ceph-mon[31535]: pgmap v2144: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:10:53.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:10:53 smithi189 ceph-mon[36187]: pgmap v2144: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:10:53.460 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:10:53 smithi193 ceph-mon[36878]: pgmap v2144: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:10:55.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:10:55 smithi118 ceph-mon[31535]: pgmap v2145: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T01:10:55.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:10:55 smithi189 ceph-mon[36187]: pgmap v2145: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T01:10:55.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:10:55 smithi193 ceph-mon[36878]: pgmap v2145: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T01:10:56.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:10:56 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 2751.653773 secs 2024-03-24T01:10:56.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:10:56 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 2751.653773 secs 2024-03-24T01:10:56.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:10:56 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 2751.653773 secs 2024-03-24T01:10:57.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:10:57 smithi118 ceph-mon[31535]: pgmap v2146: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T01:10:57.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:10:57 smithi189 ceph-mon[36187]: pgmap v2146: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T01:10:57.460 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:10:57 smithi193 ceph-mon[36878]: pgmap v2146: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T01:10:59.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:10:59 smithi118 ceph-mon[31535]: pgmap v2147: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 5 op/s 2024-03-24T01:10:59.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:10:59 smithi189 ceph-mon[36187]: pgmap v2147: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 5 op/s 2024-03-24T01:10:59.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:10:59 smithi193 ceph-mon[36878]: pgmap v2147: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 5 op/s 2024-03-24T01:11:01.447 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:11:01 smithi118 ceph-mon[31535]: pgmap v2148: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T01:11:01.447 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:11:01 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 2756.653863 secs 2024-03-24T01:11:01.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:11:01 smithi189 ceph-mon[36187]: pgmap v2148: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T01:11:01.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:11:01 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 2756.653863 secs 2024-03-24T01:11:01.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:11:01 smithi193 ceph-mon[36878]: pgmap v2148: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T01:11:01.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:11:01 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 2756.653863 secs 2024-03-24T01:11:03.197 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:11:03 smithi118 ceph-mon[31535]: pgmap v2149: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:11:03.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:11:03 smithi189 ceph-mon[36187]: pgmap v2149: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:11:03.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:11:03 smithi193 ceph-mon[36878]: pgmap v2149: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:11:05.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:11:05 smithi118 ceph-mon[31535]: pgmap v2150: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T01:11:05.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:11:05 smithi189 ceph-mon[36187]: pgmap v2150: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T01:11:05.460 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:11:05 smithi193 ceph-mon[36878]: pgmap v2150: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T01:11:06.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:11:06 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 2761.653923 secs 2024-03-24T01:11:06.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:11:06 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 2761.653923 secs 2024-03-24T01:11:06.460 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:11:06 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 2761.653923 secs 2024-03-24T01:11:07.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:11:07 smithi118 ceph-mon[31535]: pgmap v2151: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T01:11:07.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:11:07 smithi189 ceph-mon[36187]: pgmap v2151: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T01:11:07.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:11:07 smithi193 ceph-mon[36878]: pgmap v2151: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T01:11:09.361 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:11:09 smithi189 ceph-mon[36187]: pgmap v2152: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 5 op/s 2024-03-24T01:11:09.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:11:09 smithi118 ceph-mon[31535]: pgmap v2152: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 5 op/s 2024-03-24T01:11:09.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:11:09 smithi193 ceph-mon[36878]: pgmap v2152: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 5 op/s 2024-03-24T01:11:11.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:11:11 smithi118 ceph-mon[31535]: pgmap v2153: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T01:11:11.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:11:11 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 2766.654010 secs 2024-03-24T01:11:11.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:11:11 smithi189 ceph-mon[36187]: pgmap v2153: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T01:11:11.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:11:11 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 2766.654010 secs 2024-03-24T01:11:11.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:11:11 smithi193 ceph-mon[36878]: pgmap v2153: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T01:11:11.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:11:11 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 2766.654010 secs 2024-03-24T01:11:13.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:11:13 smithi118 ceph-mon[31535]: pgmap v2154: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:11:13.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:11:13 smithi189 ceph-mon[36187]: pgmap v2154: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:11:13.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:11:13 smithi193 ceph-mon[36878]: pgmap v2154: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:11:15.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:11:15 smithi118 ceph-mon[31535]: pgmap v2155: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-03-24T01:11:15.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:11:15 smithi189 ceph-mon[36187]: pgmap v2155: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-03-24T01:11:15.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:11:15 smithi193 ceph-mon[36878]: pgmap v2155: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-03-24T01:11:16.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:11:16 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 2771.654114 secs 2024-03-24T01:11:16.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:11:16 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 2771.654114 secs 2024-03-24T01:11:16.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:11:16 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 2771.654114 secs 2024-03-24T01:11:17.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:11:17 smithi118 ceph-mon[31535]: pgmap v2156: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 5 op/s 2024-03-24T01:11:17.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:11:17 smithi189 ceph-mon[36187]: pgmap v2156: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 5 op/s 2024-03-24T01:11:17.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:11:17 smithi193 ceph-mon[36878]: pgmap v2156: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 5 op/s 2024-03-24T01:11:19.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:11:19 smithi118 ceph-mon[31535]: pgmap v2157: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-03-24T01:11:19.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:11:19 smithi189 ceph-mon[36187]: pgmap v2157: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-03-24T01:11:19.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:11:19 smithi193 ceph-mon[36878]: pgmap v2157: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-03-24T01:11:21.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:11:21 smithi118 ceph-mon[31535]: pgmap v2158: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 6 op/s 2024-03-24T01:11:21.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:11:21 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 2776.654176 secs 2024-03-24T01:11:21.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:11:21 smithi189 ceph-mon[36187]: pgmap v2158: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 6 op/s 2024-03-24T01:11:21.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:11:21 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 2776.654176 secs 2024-03-24T01:11:21.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:11:21 smithi193 ceph-mon[36878]: pgmap v2158: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 6 op/s 2024-03-24T01:11:21.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:11:21 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 2776.654176 secs 2024-03-24T01:11:23.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:11:23 smithi118 ceph-mon[31535]: pgmap v2159: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-03-24T01:11:23.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:11:23 smithi189 ceph-mon[36187]: pgmap v2159: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-03-24T01:11:23.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:11:23 smithi193 ceph-mon[36878]: pgmap v2159: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-03-24T01:11:25.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:11:25 smithi118 ceph-mon[31535]: pgmap v2160: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-03-24T01:11:25.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:11:25 smithi189 ceph-mon[36187]: pgmap v2160: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-03-24T01:11:25.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:11:25 smithi193 ceph-mon[36878]: pgmap v2160: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-03-24T01:11:26.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:11:26 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 2781.654239 secs 2024-03-24T01:11:26.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:11:26 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 2781.654239 secs 2024-03-24T01:11:26.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:11:26 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 2781.654239 secs 2024-03-24T01:11:27.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:11:27 smithi118 ceph-mon[31535]: pgmap v2161: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T01:11:27.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:11:27 smithi189 ceph-mon[36187]: pgmap v2161: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T01:11:27.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:11:27 smithi193 ceph-mon[36878]: pgmap v2161: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T01:11:29.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:11:29 smithi118 ceph-mon[31535]: pgmap v2162: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T01:11:29.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:11:29 smithi189 ceph-mon[36187]: pgmap v2162: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T01:11:29.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:11:29 smithi193 ceph-mon[36878]: pgmap v2162: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T01:11:31.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:11:31 smithi118 ceph-mon[31535]: pgmap v2163: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T01:11:31.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:11:31 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 2786.654327 secs 2024-03-24T01:11:31.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:11:31 smithi189 ceph-mon[36187]: pgmap v2163: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T01:11:31.455 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:11:31 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 2786.654327 secs 2024-03-24T01:11:31.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:11:31 smithi193 ceph-mon[36878]: pgmap v2163: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T01:11:31.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:11:31 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 2786.654327 secs 2024-03-24T01:11:33.447 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:11:33 smithi118 ceph-mon[31535]: pgmap v2164: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:11:35.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:11:33 smithi193 ceph-mon[36878]: pgmap v2164: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:11:35.211 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:11:33 smithi189 ceph-mon[36187]: pgmap v2164: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:11:35.447 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:11:35 smithi118 ceph-mon[31535]: pgmap v2165: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T01:11:35.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:11:35 smithi189 ceph-mon[36187]: pgmap v2165: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T01:11:35.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:11:35 smithi193 ceph-mon[36878]: pgmap v2165: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T01:11:36.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:11:36 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 2791.654416 secs 2024-03-24T01:11:36.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:11:36 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 2791.654416 secs 2024-03-24T01:11:36.460 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:11:36 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 2791.654416 secs 2024-03-24T01:11:37.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:11:37 smithi118 ceph-mon[31535]: pgmap v2166: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T01:11:37.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:11:37 smithi189 ceph-mon[36187]: pgmap v2166: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T01:11:37.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:11:37 smithi193 ceph-mon[36878]: pgmap v2166: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T01:11:39.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:11:39 smithi118 ceph-mon[31535]: pgmap v2167: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T01:11:39.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:11:39 smithi189 ceph-mon[36187]: pgmap v2167: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T01:11:39.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:11:39 smithi193 ceph-mon[36878]: pgmap v2167: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T01:11:41.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:11:41 smithi118 ceph-mon[31535]: pgmap v2168: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T01:11:41.447 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:11:41 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T01:11:41.447 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:11:41 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 2796.654478 secs 2024-03-24T01:11:41.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:11:41 smithi189 ceph-mon[36187]: pgmap v2168: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T01:11:41.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:11:41 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T01:11:41.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:11:41 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 2796.654478 secs 2024-03-24T01:11:41.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:11:41 smithi193 ceph-mon[36878]: pgmap v2168: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T01:11:41.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:11:41 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T01:11:41.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:11:41 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 2796.654478 secs 2024-03-24T01:11:43.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:11:43 smithi118 ceph-mon[31535]: pgmap v2169: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:11:43.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:11:43 smithi189 ceph-mon[36187]: pgmap v2169: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:11:43.460 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:11:43 smithi193 ceph-mon[36878]: pgmap v2169: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:11:45.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:11:45 smithi118 ceph-mon[31535]: pgmap v2170: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T01:11:45.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:11:45 smithi189 ceph-mon[36187]: pgmap v2170: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T01:11:45.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:11:45 smithi193 ceph-mon[36878]: pgmap v2170: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T01:11:46.447 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:11:46 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 2801.654569 secs 2024-03-24T01:11:46.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:11:46 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 2801.654569 secs 2024-03-24T01:11:46.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:11:46 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 2801.654569 secs 2024-03-24T01:11:47.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:11:47 smithi189 ceph-mon[36187]: pgmap v2171: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T01:11:47.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:11:47 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T01:11:47.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:11:47 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T01:11:47.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:11:47 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T01:11:47.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:11:47 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T01:11:47.455 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:11:47 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T01:11:47.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:11:47 smithi193 ceph-mon[36878]: pgmap v2171: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T01:11:47.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:11:47 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T01:11:47.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:11:47 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T01:11:47.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:11:47 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T01:11:47.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:11:47 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T01:11:47.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:11:47 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T01:11:47.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:11:47 smithi118 ceph-mon[31535]: pgmap v2171: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T01:11:47.697 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:11:47 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T01:11:47.697 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:11:47 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T01:11:47.697 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:11:47 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T01:11:47.697 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:11:47 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T01:11:47.697 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:11:47 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T01:11:48.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:11:48 smithi189 ceph-mon[36187]: pgmap v2172: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 5 op/s 2024-03-24T01:11:48.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:11:48 smithi189 ceph-mon[36187]: pgmap v2173: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 5 op/s 2024-03-24T01:11:48.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:11:48 smithi193 ceph-mon[36878]: pgmap v2172: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 5 op/s 2024-03-24T01:11:48.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:11:48 smithi193 ceph-mon[36878]: pgmap v2173: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 5 op/s 2024-03-24T01:11:48.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:11:48 smithi118 ceph-mon[31535]: pgmap v2172: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 5 op/s 2024-03-24T01:11:48.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:11:48 smithi118 ceph-mon[31535]: pgmap v2173: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 5 op/s 2024-03-24T01:11:50.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:11:50 smithi118 ceph-mon[31535]: pgmap v2174: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 5 op/s 2024-03-24T01:11:50.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:11:50 smithi193 ceph-mon[36878]: pgmap v2174: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 5 op/s 2024-03-24T01:11:50.704 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:11:50 smithi189 ceph-mon[36187]: pgmap v2174: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 5 op/s 2024-03-24T01:11:51.605 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:11:51 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 2806.654660 secs 2024-03-24T01:11:51.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:11:51 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 2806.654660 secs 2024-03-24T01:11:51.711 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:11:51 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 2806.654660 secs 2024-03-24T01:11:52.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:11:52 smithi118 ceph-mon[31535]: pgmap v2175: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T01:11:52.704 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:11:52 smithi189 ceph-mon[36187]: pgmap v2175: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T01:11:52.711 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:11:52 smithi193 ceph-mon[36878]: pgmap v2175: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T01:11:54.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:11:54 smithi118 ceph-mon[31535]: pgmap v2176: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 5 op/s 2024-03-24T01:11:54.704 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:11:54 smithi189 ceph-mon[36187]: pgmap v2176: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 5 op/s 2024-03-24T01:11:54.711 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:11:54 smithi193 ceph-mon[36878]: pgmap v2176: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 5 op/s 2024-03-24T01:11:56.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:11:56 smithi118 ceph-mon[31535]: pgmap v2177: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T01:11:56.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:11:56 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 2811.654746 secs 2024-03-24T01:11:56.704 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:11:56 smithi189 ceph-mon[36187]: pgmap v2177: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T01:11:56.704 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:11:56 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 2811.654746 secs 2024-03-24T01:11:56.711 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:11:56 smithi193 ceph-mon[36878]: pgmap v2177: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T01:11:56.711 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:11:56 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 2811.654746 secs 2024-03-24T01:11:58.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:11:58 smithi118 ceph-mon[31535]: pgmap v2178: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 5 op/s 2024-03-24T01:11:58.704 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:11:58 smithi189 ceph-mon[36187]: pgmap v2178: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 5 op/s 2024-03-24T01:11:58.710 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:11:58 smithi193 ceph-mon[36878]: pgmap v2178: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 5 op/s 2024-03-24T01:12:00.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:12:00 smithi118 ceph-mon[31535]: pgmap v2179: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T01:12:00.704 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:12:00 smithi189 ceph-mon[36187]: pgmap v2179: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T01:12:00.710 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:12:00 smithi193 ceph-mon[36878]: pgmap v2179: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T01:12:01.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:12:01 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 2816.654802 secs 2024-03-24T01:12:01.704 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:12:01 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 2816.654802 secs 2024-03-24T01:12:01.710 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:12:01 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 2816.654802 secs 2024-03-24T01:12:02.599 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:12:02 smithi118 ceph-mon[31535]: pgmap v2180: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T01:12:02.704 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:12:02 smithi189 ceph-mon[36187]: pgmap v2180: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T01:12:02.711 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:12:02 smithi193 ceph-mon[36878]: pgmap v2180: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T01:12:04.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:12:04 smithi118 ceph-mon[31535]: pgmap v2181: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 5 op/s 2024-03-24T01:12:04.704 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:12:04 smithi189 ceph-mon[36187]: pgmap v2181: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 5 op/s 2024-03-24T01:12:04.711 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:12:04 smithi193 ceph-mon[36878]: pgmap v2181: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 5 op/s 2024-03-24T01:12:06.697 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:12:06 smithi118 ceph-mon[31535]: pgmap v2182: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T01:12:06.697 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:12:06 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 2821.654880 secs 2024-03-24T01:12:06.704 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:12:06 smithi189 ceph-mon[36187]: pgmap v2182: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T01:12:06.704 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:12:06 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 2821.654880 secs 2024-03-24T01:12:06.711 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:12:06 smithi193 ceph-mon[36878]: pgmap v2182: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T01:12:06.711 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:12:06 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 2821.654880 secs 2024-03-24T01:12:08.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:12:08 smithi118 ceph-mon[31535]: pgmap v2183: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:12:08.704 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:12:08 smithi189 ceph-mon[36187]: pgmap v2183: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:12:08.710 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:12:08 smithi193 ceph-mon[36878]: pgmap v2183: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:12:10.697 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:12:10 smithi118 ceph-mon[31535]: pgmap v2184: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T01:12:10.704 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:12:10 smithi189 ceph-mon[36187]: pgmap v2184: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T01:12:10.711 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:12:10 smithi193 ceph-mon[36878]: pgmap v2184: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T01:12:11.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:12:11 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 2826.654967 secs 2024-03-24T01:12:11.704 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:12:11 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 2826.654967 secs 2024-03-24T01:12:11.711 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:12:11 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 2826.654967 secs 2024-03-24T01:12:12.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:12:12 smithi118 ceph-mon[31535]: pgmap v2185: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T01:12:12.704 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:12:12 smithi189 ceph-mon[36187]: pgmap v2185: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T01:12:12.711 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:12:12 smithi193 ceph-mon[36878]: pgmap v2185: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T01:12:14.662 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:12:14 smithi118 ceph-mon[31535]: pgmap v2186: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T01:12:14.704 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:12:14 smithi189 ceph-mon[36187]: pgmap v2186: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T01:12:14.710 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:12:14 smithi193 ceph-mon[36878]: pgmap v2186: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T01:12:16.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:12:16 smithi118 ceph-mon[31535]: pgmap v2187: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T01:12:16.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:12:16 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 2831.655061 secs 2024-03-24T01:12:16.704 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:12:16 smithi189 ceph-mon[36187]: pgmap v2187: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T01:12:16.704 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:12:16 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 2831.655061 secs 2024-03-24T01:12:16.711 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:12:16 smithi193 ceph-mon[36878]: pgmap v2187: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T01:12:16.711 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:12:16 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 2831.655061 secs 2024-03-24T01:12:18.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:12:18 smithi118 ceph-mon[31535]: pgmap v2188: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:12:18.704 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:12:18 smithi189 ceph-mon[36187]: pgmap v2188: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:12:18.711 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:12:18 smithi193 ceph-mon[36878]: pgmap v2188: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:12:20.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:12:20 smithi118 ceph-mon[31535]: pgmap v2189: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T01:12:20.704 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:12:20 smithi189 ceph-mon[36187]: pgmap v2189: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T01:12:20.711 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:12:20 smithi193 ceph-mon[36878]: pgmap v2189: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T01:12:21.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:12:21 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 2836.655142 secs 2024-03-24T01:12:21.704 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:12:21 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 2836.655142 secs 2024-03-24T01:12:21.710 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:12:21 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 2836.655142 secs 2024-03-24T01:12:22.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:12:22 smithi118 ceph-mon[31535]: pgmap v2190: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T01:12:22.704 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:12:22 smithi189 ceph-mon[36187]: pgmap v2190: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T01:12:22.711 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:12:22 smithi193 ceph-mon[36878]: pgmap v2190: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T01:12:24.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:12:24 smithi118 ceph-mon[31535]: pgmap v2191: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T01:12:24.704 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:12:24 smithi189 ceph-mon[36187]: pgmap v2191: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T01:12:24.711 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:12:24 smithi193 ceph-mon[36878]: pgmap v2191: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T01:12:26.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:12:26 smithi118 ceph-mon[31535]: pgmap v2192: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T01:12:26.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:12:26 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 2841.655231 secs 2024-03-24T01:12:26.704 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:12:26 smithi189 ceph-mon[36187]: pgmap v2192: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T01:12:26.704 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:12:26 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 2841.655231 secs 2024-03-24T01:12:26.711 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:12:26 smithi193 ceph-mon[36878]: pgmap v2192: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T01:12:26.711 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:12:26 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 2841.655231 secs 2024-03-24T01:12:28.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:12:28 smithi118 ceph-mon[31535]: pgmap v2193: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:12:28.704 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:12:28 smithi189 ceph-mon[36187]: pgmap v2193: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:12:28.710 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:12:28 smithi193 ceph-mon[36878]: pgmap v2193: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:12:30.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:12:30 smithi118 ceph-mon[31535]: pgmap v2194: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T01:12:30.704 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:12:30 smithi189 ceph-mon[36187]: pgmap v2194: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T01:12:30.710 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:12:30 smithi193 ceph-mon[36878]: pgmap v2194: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T01:12:31.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:12:31 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 2846.655290 secs 2024-03-24T01:12:31.704 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:12:31 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 2846.655290 secs 2024-03-24T01:12:31.710 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:12:31 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 2846.655290 secs 2024-03-24T01:12:32.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:12:32 smithi118 ceph-mon[31535]: pgmap v2195: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T01:12:32.704 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:12:32 smithi189 ceph-mon[36187]: pgmap v2195: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T01:12:32.710 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:12:32 smithi193 ceph-mon[36878]: pgmap v2195: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T01:12:34.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:12:34 smithi118 ceph-mon[31535]: pgmap v2196: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T01:12:34.704 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:12:34 smithi189 ceph-mon[36187]: pgmap v2196: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T01:12:34.711 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:12:34 smithi193 ceph-mon[36878]: pgmap v2196: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T01:12:36.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:12:36 smithi118 ceph-mon[31535]: pgmap v2197: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T01:12:36.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:12:36 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 2851.655374 secs 2024-03-24T01:12:36.704 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:12:36 smithi189 ceph-mon[36187]: pgmap v2197: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T01:12:36.704 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:12:36 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 2851.655374 secs 2024-03-24T01:12:36.710 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:12:36 smithi193 ceph-mon[36878]: pgmap v2197: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T01:12:36.711 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:12:36 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 2851.655374 secs 2024-03-24T01:12:38.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:12:38 smithi118 ceph-mon[31535]: pgmap v2198: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:12:38.704 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:12:38 smithi189 ceph-mon[36187]: pgmap v2198: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:12:38.711 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:12:38 smithi193 ceph-mon[36878]: pgmap v2198: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:12:40.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:12:40 smithi118 ceph-mon[31535]: pgmap v2199: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T01:12:40.704 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:12:40 smithi189 ceph-mon[36187]: pgmap v2199: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T01:12:40.711 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:12:40 smithi193 ceph-mon[36878]: pgmap v2199: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T01:12:41.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:12:41 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 2856.655459 secs 2024-03-24T01:12:41.704 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:12:41 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 2856.655459 secs 2024-03-24T01:12:41.710 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:12:41 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 2856.655459 secs 2024-03-24T01:12:42.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:12:42 smithi118 ceph-mon[31535]: pgmap v2200: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T01:12:42.704 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:12:42 smithi189 ceph-mon[36187]: pgmap v2200: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T01:12:42.710 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:12:42 smithi193 ceph-mon[36878]: pgmap v2200: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T01:12:44.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:12:44 smithi118 ceph-mon[31535]: pgmap v2201: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T01:12:44.704 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:12:44 smithi189 ceph-mon[36187]: pgmap v2201: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T01:12:44.710 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:12:44 smithi193 ceph-mon[36878]: pgmap v2201: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T01:12:46.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:12:46 smithi118 ceph-mon[31535]: pgmap v2202: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T01:12:46.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:12:46 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 2861.655543 secs 2024-03-24T01:12:46.704 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:12:46 smithi189 ceph-mon[36187]: pgmap v2202: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T01:12:46.704 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:12:46 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 2861.655543 secs 2024-03-24T01:12:46.711 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:12:46 smithi193 ceph-mon[36878]: pgmap v2202: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T01:12:46.711 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:12:46 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 2861.655543 secs 2024-03-24T01:12:47.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:12:47 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T01:12:47.704 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:12:47 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T01:12:47.711 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:12:47 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T01:12:48.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:12:48 smithi118 ceph-mon[31535]: pgmap v2203: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:12:48.704 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:12:48 smithi189 ceph-mon[36187]: pgmap v2203: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:12:48.711 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:12:48 smithi193 ceph-mon[36878]: pgmap v2203: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:12:50.704 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:12:50 smithi189 ceph-mon[36187]: pgmap v2204: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T01:12:50.711 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:12:50 smithi193 ceph-mon[36878]: pgmap v2204: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T01:12:50.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:12:50 smithi118 ceph-mon[31535]: pgmap v2204: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T01:12:51.524 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:12:51 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 2866.655615 secs 2024-03-24T01:12:51.568 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:12:51 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 2866.655615 secs 2024-03-24T01:12:51.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:12:51 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 2866.655615 secs 2024-03-24T01:12:52.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:12:52 smithi118 ceph-mon[31535]: pgmap v2205: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T01:12:52.947 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:12:52 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T01:12:52.947 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:12:52 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T01:12:52.947 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:12:52 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T01:12:52.947 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:12:52 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T01:12:52.947 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:12:52 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T01:12:52.947 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:12:52 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T01:12:52.947 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:12:52 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T01:12:52.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:12:52 smithi189 ceph-mon[36187]: pgmap v2205: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T01:12:52.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:12:52 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T01:12:52.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:12:52 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T01:12:52.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:12:52 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T01:12:52.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:12:52 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T01:12:52.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:12:52 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T01:12:52.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:12:52 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T01:12:52.955 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:12:52 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T01:12:52.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:12:52 smithi193 ceph-mon[36878]: pgmap v2205: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T01:12:52.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:12:52 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T01:12:52.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:12:52 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T01:12:52.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:12:52 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T01:12:52.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:12:52 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T01:12:52.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:12:52 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T01:12:52.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:12:52 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T01:12:52.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:12:52 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T01:12:53.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:12:53 smithi118 ceph-mon[31535]: pgmap v2206: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 5 op/s 2024-03-24T01:12:53.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:12:53 smithi118 ceph-mon[31535]: pgmap v2207: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 5 op/s 2024-03-24T01:12:53.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:12:53 smithi189 ceph-mon[36187]: pgmap v2206: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 5 op/s 2024-03-24T01:12:53.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:12:53 smithi189 ceph-mon[36187]: pgmap v2207: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 5 op/s 2024-03-24T01:12:53.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:12:53 smithi193 ceph-mon[36878]: pgmap v2206: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 5 op/s 2024-03-24T01:12:53.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:12:53 smithi193 ceph-mon[36878]: pgmap v2207: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 5 op/s 2024-03-24T01:12:55.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:12:55 smithi118 ceph-mon[31535]: pgmap v2208: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 5 op/s 2024-03-24T01:12:55.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:12:55 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 2871.655673 secs 2024-03-24T01:12:55.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:12:55 smithi189 ceph-mon[36187]: pgmap v2208: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 5 op/s 2024-03-24T01:12:55.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:12:55 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 2871.655673 secs 2024-03-24T01:12:55.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:12:55 smithi193 ceph-mon[36878]: pgmap v2208: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 5 op/s 2024-03-24T01:12:55.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:12:55 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 2871.655673 secs 2024-03-24T01:12:57.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:12:57 smithi118 ceph-mon[31535]: pgmap v2209: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 6 op/s 2024-03-24T01:12:57.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:12:57 smithi189 ceph-mon[36187]: pgmap v2209: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 6 op/s 2024-03-24T01:12:57.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:12:57 smithi193 ceph-mon[36878]: pgmap v2209: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 6 op/s 2024-03-24T01:12:59.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:12:59 smithi118 ceph-mon[31535]: pgmap v2210: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 5 op/s 2024-03-24T01:12:59.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:12:59 smithi189 ceph-mon[36187]: pgmap v2210: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 5 op/s 2024-03-24T01:12:59.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:12:59 smithi193 ceph-mon[36878]: pgmap v2210: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 5 op/s 2024-03-24T01:13:00.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:13:00 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 2876.655760 secs 2024-03-24T01:13:00.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:13:00 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 2876.655760 secs 2024-03-24T01:13:00.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:13:00 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 2876.655760 secs 2024-03-24T01:13:01.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:13:01 smithi118 ceph-mon[31535]: pgmap v2211: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 6 op/s 2024-03-24T01:13:01.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:13:01 smithi189 ceph-mon[36187]: pgmap v2211: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 6 op/s 2024-03-24T01:13:01.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:13:01 smithi193 ceph-mon[36878]: pgmap v2211: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 6 op/s 2024-03-24T01:13:03.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:13:03 smithi118 ceph-mon[31535]: pgmap v2212: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 5 op/s 2024-03-24T01:13:03.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:13:03 smithi189 ceph-mon[36187]: pgmap v2212: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 5 op/s 2024-03-24T01:13:03.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:13:03 smithi193 ceph-mon[36878]: pgmap v2212: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 5 op/s 2024-03-24T01:13:04.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:13:04 smithi118 ceph-mon[31535]: pgmap v2213: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T01:13:04.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:13:04 smithi189 ceph-mon[36187]: pgmap v2213: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T01:13:04.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:13:04 smithi193 ceph-mon[36878]: pgmap v2213: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T01:13:05.921 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:13:05 smithi118 ceph-mon[31535]: 3 slow requests, 3 included below; oldest blocked for > 2881.655848 secs 2024-03-24T01:13:05.921 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:13:05 smithi118 ceph-mon[31535]: slow request 2881.655848 seconds old, received at 2024-03-24T00:25:03.572253+0000: peer_request:client.15159:154789 currently acquired locks 2024-03-24T01:13:05.921 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:13:05 smithi118 ceph-mon[31535]: slow request 2881.604601 seconds old, received at 2024-03-24T00:25:03.623500+0000: peer_request:client.15159:158090 currently acquired locks 2024-03-24T01:13:05.921 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:13:05 smithi118 ceph-mon[31535]: slow request 2881.597923 seconds old, received at 2024-03-24T00:25:03.630178+0000: client_request(client.15159:173196 rename #0x100000010bf/comment-15.xml #0x100000010bf/comment-15.xml.tmp 2024-03-24T00:25:03.629366+0000 caller_uid=1000, caller_gid=1301{6,36,1000,1301,}) currently failed to xlock, waiting 2024-03-24T01:13:05.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:13:05 smithi189 ceph-mon[36187]: 3 slow requests, 3 included below; oldest blocked for > 2881.655848 secs 2024-03-24T01:13:05.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:13:05 smithi189 ceph-mon[36187]: slow request 2881.655848 seconds old, received at 2024-03-24T00:25:03.572253+0000: peer_request:client.15159:154789 currently acquired locks 2024-03-24T01:13:05.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:13:05 smithi189 ceph-mon[36187]: slow request 2881.604601 seconds old, received at 2024-03-24T00:25:03.623500+0000: peer_request:client.15159:158090 currently acquired locks 2024-03-24T01:13:05.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:13:05 smithi189 ceph-mon[36187]: slow request 2881.597923 seconds old, received at 2024-03-24T00:25:03.630178+0000: client_request(client.15159:173196 rename #0x100000010bf/comment-15.xml #0x100000010bf/comment-15.xml.tmp 2024-03-24T00:25:03.629366+0000 caller_uid=1000, caller_gid=1301{6,36,1000,1301,}) currently failed to xlock, waiting 2024-03-24T01:13:05.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:13:05 smithi193 ceph-mon[36878]: 3 slow requests, 3 included below; oldest blocked for > 2881.655848 secs 2024-03-24T01:13:05.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:13:05 smithi193 ceph-mon[36878]: slow request 2881.655848 seconds old, received at 2024-03-24T00:25:03.572253+0000: peer_request:client.15159:154789 currently acquired locks 2024-03-24T01:13:05.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:13:05 smithi193 ceph-mon[36878]: slow request 2881.604601 seconds old, received at 2024-03-24T00:25:03.623500+0000: peer_request:client.15159:158090 currently acquired locks 2024-03-24T01:13:05.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:13:05 smithi193 ceph-mon[36878]: slow request 2881.597923 seconds old, received at 2024-03-24T00:25:03.630178+0000: client_request(client.15159:173196 rename #0x100000010bf/comment-15.xml #0x100000010bf/comment-15.xml.tmp 2024-03-24T00:25:03.629366+0000 caller_uid=1000, caller_gid=1301{6,36,1000,1301,}) currently failed to xlock, waiting 2024-03-24T01:13:06.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:13:06 smithi118 ceph-mon[31535]: pgmap v2214: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T01:13:06.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:13:06 smithi189 ceph-mon[36187]: pgmap v2214: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T01:13:06.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:13:06 smithi193 ceph-mon[36878]: pgmap v2214: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T01:13:09.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:13:08 smithi118 ceph-mon[31535]: pgmap v2215: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 5 op/s 2024-03-24T01:13:09.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:13:08 smithi189 ceph-mon[36187]: pgmap v2215: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 5 op/s 2024-03-24T01:13:09.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:13:08 smithi193 ceph-mon[36878]: pgmap v2215: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 5 op/s 2024-03-24T01:13:11.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:13:10 smithi118 ceph-mon[31535]: pgmap v2216: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T01:13:11.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:13:10 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 2886.655938 secs 2024-03-24T01:13:11.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:13:10 smithi189 ceph-mon[36187]: pgmap v2216: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T01:13:11.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:13:10 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 2886.655938 secs 2024-03-24T01:13:11.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:13:10 smithi193 ceph-mon[36878]: pgmap v2216: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T01:13:11.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:13:10 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 2886.655938 secs 2024-03-24T01:13:13.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:13:12 smithi118 ceph-mon[31535]: pgmap v2217: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:13:13.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:13:12 smithi189 ceph-mon[36187]: pgmap v2217: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:13:13.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:13:12 smithi193 ceph-mon[36878]: pgmap v2217: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:13:15.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:13:14 smithi118 ceph-mon[31535]: pgmap v2218: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T01:13:15.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:13:14 smithi189 ceph-mon[36187]: pgmap v2218: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T01:13:15.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:13:14 smithi193 ceph-mon[36878]: pgmap v2218: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T01:13:16.191 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T01:13:16.189+0000 7f81ee000640 1 -- 172.21.15.118:0/1693767697 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f81c80065b0 con 0x7f81f81052d0 2024-03-24T01:13:16.191 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T01:13:16.189+0000 7f81db400640 1 -- 172.21.15.118:0/659686424 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f81c40062d0 con 0x5622651a13f0 2024-03-24T01:13:16.191 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T01:13:16.190+0000 7f81e4c00640 1 -- 172.21.15.118:0/659686424 <== mon.1 v2:172.21.15.193:3300/0 26 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 409+0+0 (secure 0 0 0) 0x7f81c40062d0 con 0x5622651a13f0 2024-03-24T01:13:16.191 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T01:13:16.190+0000 7f81ef400640 1 -- 172.21.15.118:0/1693767697 <== mon.2 v2:172.21.15.189:3300/0 26 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (secure 0 0 0) 0x7f81c80065b0 con 0x7f81f81052d0 2024-03-24T01:13:16.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:13:15 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 2891.656027 secs 2024-03-24T01:13:16.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:13:15 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 2891.656027 secs 2024-03-24T01:13:16.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:13:15 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 2891.656027 secs 2024-03-24T01:13:17.197 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:13:16 smithi118 ceph-mon[31535]: pgmap v2219: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T01:13:17.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:13:16 smithi189 ceph-mon[36187]: pgmap v2219: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T01:13:17.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:13:16 smithi193 ceph-mon[36878]: pgmap v2219: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T01:13:19.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:13:18 smithi118 ceph-mon[31535]: pgmap v2220: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 5 op/s 2024-03-24T01:13:19.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:13:18 smithi189 ceph-mon[36187]: pgmap v2220: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 5 op/s 2024-03-24T01:13:19.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:13:18 smithi193 ceph-mon[36878]: pgmap v2220: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 5 op/s 2024-03-24T01:13:21.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:13:20 smithi118 ceph-mon[31535]: pgmap v2221: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T01:13:21.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:13:20 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 2896.656100 secs 2024-03-24T01:13:21.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:13:20 smithi189 ceph-mon[36187]: pgmap v2221: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T01:13:21.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:13:20 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 2896.656100 secs 2024-03-24T01:13:21.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:13:20 smithi193 ceph-mon[36878]: pgmap v2221: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T01:13:21.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:13:20 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 2896.656100 secs 2024-03-24T01:13:23.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:13:22 smithi118 ceph-mon[31535]: pgmap v2222: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:13:23.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:13:22 smithi189 ceph-mon[36187]: pgmap v2222: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:13:23.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:13:22 smithi193 ceph-mon[36878]: pgmap v2222: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:13:25.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:13:24 smithi118 ceph-mon[31535]: pgmap v2223: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T01:13:25.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:13:24 smithi189 ceph-mon[36187]: pgmap v2223: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T01:13:25.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:13:24 smithi193 ceph-mon[36878]: pgmap v2223: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T01:13:26.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:13:25 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 2901.656165 secs 2024-03-24T01:13:26.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:13:25 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 2901.656165 secs 2024-03-24T01:13:26.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:13:25 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 2901.656165 secs 2024-03-24T01:13:27.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:13:26 smithi118 ceph-mon[31535]: pgmap v2224: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T01:13:27.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:13:26 smithi189 ceph-mon[36187]: pgmap v2224: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T01:13:27.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:13:26 smithi193 ceph-mon[36878]: pgmap v2224: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T01:13:29.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:13:28 smithi118 ceph-mon[31535]: pgmap v2225: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 5 op/s 2024-03-24T01:13:29.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:13:28 smithi189 ceph-mon[36187]: pgmap v2225: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 5 op/s 2024-03-24T01:13:29.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:13:28 smithi193 ceph-mon[36878]: pgmap v2225: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 5 op/s 2024-03-24T01:13:31.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:13:30 smithi118 ceph-mon[31535]: pgmap v2226: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T01:13:31.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:13:30 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 2906.656250 secs 2024-03-24T01:13:31.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:13:30 smithi189 ceph-mon[36187]: pgmap v2226: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T01:13:31.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:13:30 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 2906.656250 secs 2024-03-24T01:13:31.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:13:30 smithi193 ceph-mon[36878]: pgmap v2226: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T01:13:31.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:13:30 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 2906.656250 secs 2024-03-24T01:13:33.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:13:32 smithi118 ceph-mon[31535]: pgmap v2227: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:13:33.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:13:32 smithi189 ceph-mon[36187]: pgmap v2227: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:13:33.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:13:32 smithi193 ceph-mon[36878]: pgmap v2227: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:13:35.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:13:34 smithi118 ceph-mon[31535]: pgmap v2228: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T01:13:35.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:13:34 smithi189 ceph-mon[36187]: pgmap v2228: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T01:13:35.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:13:34 smithi193 ceph-mon[36878]: pgmap v2228: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T01:13:36.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:13:35 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 2911.656337 secs 2024-03-24T01:13:36.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:13:35 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 2911.656337 secs 2024-03-24T01:13:36.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:13:35 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 2911.656337 secs 2024-03-24T01:13:37.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:13:36 smithi118 ceph-mon[31535]: pgmap v2229: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T01:13:37.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:13:36 smithi189 ceph-mon[36187]: pgmap v2229: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T01:13:37.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:13:36 smithi193 ceph-mon[36878]: pgmap v2229: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T01:13:39.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:13:38 smithi118 ceph-mon[31535]: pgmap v2230: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 5 op/s 2024-03-24T01:13:39.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:13:38 smithi189 ceph-mon[36187]: pgmap v2230: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 5 op/s 2024-03-24T01:13:39.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:13:38 smithi193 ceph-mon[36878]: pgmap v2230: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 5 op/s 2024-03-24T01:13:41.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:13:40 smithi118 ceph-mon[31535]: pgmap v2231: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T01:13:41.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:13:40 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 2916.656395 secs 2024-03-24T01:13:41.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:13:40 smithi189 ceph-mon[36187]: pgmap v2231: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T01:13:41.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:13:40 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 2916.656395 secs 2024-03-24T01:13:41.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:13:40 smithi193 ceph-mon[36878]: pgmap v2231: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T01:13:41.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:13:40 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 2916.656395 secs 2024-03-24T01:13:43.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:13:42 smithi118 ceph-mon[31535]: pgmap v2232: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:13:43.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:13:42 smithi189 ceph-mon[36187]: pgmap v2232: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:13:43.213 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:13:42 smithi193 ceph-mon[36878]: pgmap v2232: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:13:45.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:13:44 smithi118 ceph-mon[31535]: pgmap v2233: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T01:13:45.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:13:44 smithi189 ceph-mon[36187]: pgmap v2233: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T01:13:45.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:13:44 smithi193 ceph-mon[36878]: pgmap v2233: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T01:13:46.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:13:45 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 2921.656492 secs 2024-03-24T01:13:46.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:13:45 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 2921.656492 secs 2024-03-24T01:13:46.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:13:45 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 2921.656492 secs 2024-03-24T01:13:47.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:13:46 smithi193 ceph-mon[36878]: pgmap v2234: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T01:13:47.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:13:46 smithi118 ceph-mon[31535]: pgmap v2234: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T01:13:47.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:13:46 smithi189 ceph-mon[36187]: pgmap v2234: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T01:13:49.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:13:48 smithi193 ceph-mon[36878]: pgmap v2235: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 5 op/s 2024-03-24T01:13:49.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:13:48 smithi118 ceph-mon[31535]: pgmap v2235: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 5 op/s 2024-03-24T01:13:49.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:13:48 smithi189 ceph-mon[36187]: pgmap v2235: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 5 op/s 2024-03-24T01:13:51.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:13:50 smithi118 ceph-mon[31535]: pgmap v2236: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T01:13:51.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:13:50 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 2926.656580 secs 2024-03-24T01:13:51.256 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:13:50 smithi189 ceph-mon[36187]: pgmap v2236: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T01:13:51.256 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:13:50 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 2926.656580 secs 2024-03-24T01:13:51.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:13:50 smithi193 ceph-mon[36878]: pgmap v2236: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T01:13:51.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:13:50 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 2926.656580 secs 2024-03-24T01:13:53.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:13:52 smithi118 ceph-mon[31535]: pgmap v2237: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:13:53.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:13:52 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T01:13:53.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:13:52 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-24T01:13:53.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:13:52 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-24T01:13:53.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:13:52 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-24T01:13:53.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:13:52 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-03-24T01:13:53.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:13:52 smithi189 ceph-mon[36187]: pgmap v2237: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:13:53.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:13:52 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T01:13:53.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:13:52 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-24T01:13:53.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:13:52 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-24T01:13:53.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:13:52 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-24T01:13:53.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:13:52 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-03-24T01:13:53.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:13:52 smithi193 ceph-mon[36878]: pgmap v2237: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:13:53.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:13:52 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T01:13:53.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:13:52 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-24T01:13:53.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:13:52 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-24T01:13:53.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:13:52 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-24T01:13:53.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:13:52 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-03-24T01:13:54.447 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:13:53 smithi118 ceph-mon[31535]: Adjusting osd_memory_target on smithi189 to 174.1M 2024-03-24T01:13:54.447 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:13:53 smithi118 ceph-mon[31535]: Unable to set osd_memory_target on smithi189 to 182635622: error parsing value: Value '182635622' is below minimum 939524096 2024-03-24T01:13:54.447 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:13:53 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-24T01:13:54.447 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:13:53 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-24T01:13:54.447 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:13:53 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-24T01:13:54.447 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:13:53 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-03-24T01:13:54.447 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:13:53 smithi118 ceph-mon[31535]: Adjusting osd_memory_target on smithi118 to 176.7M 2024-03-24T01:13:54.447 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:13:53 smithi118 ceph-mon[31535]: Unable to set osd_memory_target on smithi118 to 185365196: error parsing value: Value '185365196' is below minimum 939524096 2024-03-24T01:13:54.447 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:13:53 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T01:13:54.447 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:13:53 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T01:13:54.447 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:13:53 smithi118 ceph-mon[31535]: pgmap v2238: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 6 op/s 2024-03-24T01:13:54.448 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:13:53 smithi118 ceph-mon[31535]: pgmap v2239: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 6 op/s 2024-03-24T01:13:54.448 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:13:53 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T01:13:54.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:13:53 smithi189 ceph-mon[36187]: Adjusting osd_memory_target on smithi189 to 174.1M 2024-03-24T01:13:54.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:13:53 smithi189 ceph-mon[36187]: Unable to set osd_memory_target on smithi189 to 182635622: error parsing value: Value '182635622' is below minimum 939524096 2024-03-24T01:13:54.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:13:53 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-24T01:13:54.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:13:53 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-24T01:13:54.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:13:53 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-24T01:13:54.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:13:53 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-03-24T01:13:54.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:13:53 smithi189 ceph-mon[36187]: Adjusting osd_memory_target on smithi118 to 176.7M 2024-03-24T01:13:54.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:13:53 smithi189 ceph-mon[36187]: Unable to set osd_memory_target on smithi118 to 185365196: error parsing value: Value '185365196' is below minimum 939524096 2024-03-24T01:13:54.455 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:13:53 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T01:13:54.455 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:13:53 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T01:13:54.455 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:13:53 smithi189 ceph-mon[36187]: pgmap v2238: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 6 op/s 2024-03-24T01:13:54.455 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:13:53 smithi189 ceph-mon[36187]: pgmap v2239: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 6 op/s 2024-03-24T01:13:54.455 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:13:53 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T01:13:54.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:13:53 smithi193 ceph-mon[36878]: Adjusting osd_memory_target on smithi189 to 174.1M 2024-03-24T01:13:54.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:13:53 smithi193 ceph-mon[36878]: Unable to set osd_memory_target on smithi189 to 182635622: error parsing value: Value '182635622' is below minimum 939524096 2024-03-24T01:13:54.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:13:53 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-24T01:13:54.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:13:53 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-24T01:13:54.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:13:53 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-24T01:13:54.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:13:53 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-03-24T01:13:54.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:13:53 smithi193 ceph-mon[36878]: Adjusting osd_memory_target on smithi118 to 176.7M 2024-03-24T01:13:54.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:13:53 smithi193 ceph-mon[36878]: Unable to set osd_memory_target on smithi118 to 185365196: error parsing value: Value '185365196' is below minimum 939524096 2024-03-24T01:13:54.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:13:53 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T01:13:54.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:13:53 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T01:13:54.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:13:53 smithi193 ceph-mon[36878]: pgmap v2238: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 6 op/s 2024-03-24T01:13:54.462 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:13:53 smithi193 ceph-mon[36878]: pgmap v2239: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 6 op/s 2024-03-24T01:13:54.462 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:13:53 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T01:13:56.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:13:56 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 2931.656669 secs 2024-03-24T01:13:56.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:13:56 smithi118 ceph-mon[31535]: pgmap v2240: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 6 op/s 2024-03-24T01:13:56.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:13:56 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 2931.656669 secs 2024-03-24T01:13:56.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:13:56 smithi189 ceph-mon[36187]: pgmap v2240: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 6 op/s 2024-03-24T01:13:56.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:13:56 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 2931.656669 secs 2024-03-24T01:13:56.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:13:56 smithi193 ceph-mon[36878]: pgmap v2240: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 6 op/s 2024-03-24T01:13:58.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:13:58 smithi118 ceph-mon[31535]: pgmap v2241: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 5 op/s 2024-03-24T01:13:58.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:13:58 smithi189 ceph-mon[36187]: pgmap v2241: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 5 op/s 2024-03-24T01:13:58.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:13:58 smithi193 ceph-mon[36878]: pgmap v2241: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 5 op/s 2024-03-24T01:14:00.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:14:00 smithi118 ceph-mon[31535]: pgmap v2242: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 6 op/s 2024-03-24T01:14:00.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:14:00 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 2936.656750 secs 2024-03-24T01:14:00.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:14:00 smithi189 ceph-mon[36187]: pgmap v2242: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 6 op/s 2024-03-24T01:14:00.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:14:00 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 2936.656750 secs 2024-03-24T01:14:00.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:14:00 smithi193 ceph-mon[36878]: pgmap v2242: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 6 op/s 2024-03-24T01:14:00.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:14:00 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 2936.656750 secs 2024-03-24T01:14:02.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:14:02 smithi118 ceph-mon[31535]: pgmap v2243: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 6 op/s 2024-03-24T01:14:02.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:14:02 smithi189 ceph-mon[36187]: pgmap v2243: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 6 op/s 2024-03-24T01:14:02.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:14:02 smithi193 ceph-mon[36878]: pgmap v2243: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 6 op/s 2024-03-24T01:14:04.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:14:04 smithi118 ceph-mon[31535]: pgmap v2244: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 5 op/s 2024-03-24T01:14:04.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:14:04 smithi189 ceph-mon[36187]: pgmap v2244: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 5 op/s 2024-03-24T01:14:04.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:14:04 smithi193 ceph-mon[36878]: pgmap v2244: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 5 op/s 2024-03-24T01:14:05.848 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:14:05 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 2941.656837 secs 2024-03-24T01:14:05.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:14:05 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 2941.656837 secs 2024-03-24T01:14:05.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:14:05 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 2941.656837 secs 2024-03-24T01:14:06.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:14:06 smithi118 ceph-mon[31535]: pgmap v2245: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T01:14:06.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:14:06 smithi189 ceph-mon[36187]: pgmap v2245: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T01:14:06.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:14:06 smithi193 ceph-mon[36878]: pgmap v2245: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T01:14:08.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:14:08 smithi118 ceph-mon[31535]: pgmap v2246: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T01:14:08.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:14:08 smithi189 ceph-mon[36187]: pgmap v2246: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T01:14:08.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:14:08 smithi193 ceph-mon[36878]: pgmap v2246: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T01:14:10.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:14:10 smithi118 ceph-mon[31535]: pgmap v2247: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-03-24T01:14:10.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:14:10 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 2946.656924 secs 2024-03-24T01:14:10.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:14:10 smithi189 ceph-mon[36187]: pgmap v2247: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-03-24T01:14:10.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:14:10 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 2946.656924 secs 2024-03-24T01:14:10.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:14:10 smithi193 ceph-mon[36878]: pgmap v2247: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-03-24T01:14:10.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:14:10 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 2946.656924 secs 2024-03-24T01:14:12.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:14:12 smithi118 ceph-mon[31535]: pgmap v2248: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:14:12.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:14:12 smithi189 ceph-mon[36187]: pgmap v2248: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:14:12.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:14:12 smithi193 ceph-mon[36878]: pgmap v2248: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:14:13.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:14:13 smithi118 ceph-mon[31535]: pgmap v2249: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T01:14:13.971 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:14:13 smithi193 ceph-mon[36878]: pgmap v2249: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T01:14:13.971 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:14:13 smithi189 ceph-mon[36187]: pgmap v2249: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T01:14:15.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:14:15 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 2951.656967 secs 2024-03-24T01:14:15.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:14:15 smithi118 ceph-mon[31535]: pgmap v2250: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T01:14:15.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:14:15 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 2951.656967 secs 2024-03-24T01:14:15.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:14:15 smithi189 ceph-mon[36187]: pgmap v2250: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T01:14:15.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:14:15 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 2951.656967 secs 2024-03-24T01:14:15.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:14:15 smithi193 ceph-mon[36878]: pgmap v2250: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T01:14:18.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:14:18 smithi118 ceph-mon[31535]: pgmap v2251: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T01:14:18.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:14:18 smithi189 ceph-mon[36187]: pgmap v2251: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T01:14:18.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:14:18 smithi193 ceph-mon[36878]: pgmap v2251: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T01:14:20.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:14:20 smithi118 ceph-mon[31535]: pgmap v2252: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-03-24T01:14:20.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:14:20 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 2956.657003 secs 2024-03-24T01:14:20.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:14:20 smithi189 ceph-mon[36187]: pgmap v2252: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-03-24T01:14:20.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:14:20 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 2956.657003 secs 2024-03-24T01:14:20.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:14:20 smithi193 ceph-mon[36878]: pgmap v2252: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-03-24T01:14:20.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:14:20 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 2956.657003 secs 2024-03-24T01:14:22.756 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:14:22 smithi189 ceph-mon[36187]: pgmap v2253: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:14:22.762 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:14:22 smithi193 ceph-mon[36878]: pgmap v2253: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:14:22.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:14:22 smithi118 ceph-mon[31535]: pgmap v2253: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:14:24.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:14:24 smithi118 ceph-mon[31535]: pgmap v2254: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T01:14:24.955 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:14:24 smithi189 ceph-mon[36187]: pgmap v2254: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T01:14:24.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:14:24 smithi193 ceph-mon[36878]: pgmap v2254: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T01:14:25.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:14:25 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 2961.657092 secs 2024-03-24T01:14:25.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:14:25 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 2961.657092 secs 2024-03-24T01:14:25.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:14:25 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 2961.657092 secs 2024-03-24T01:14:26.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:14:26 smithi118 ceph-mon[31535]: pgmap v2255: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T01:14:26.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:14:26 smithi189 ceph-mon[36187]: pgmap v2255: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T01:14:26.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:14:26 smithi193 ceph-mon[36878]: pgmap v2255: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T01:14:28.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:14:28 smithi118 ceph-mon[31535]: pgmap v2256: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T01:14:28.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:14:28 smithi189 ceph-mon[36187]: pgmap v2256: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T01:14:28.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:14:28 smithi193 ceph-mon[36878]: pgmap v2256: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T01:14:30.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:14:30 smithi118 ceph-mon[31535]: pgmap v2257: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-03-24T01:14:30.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:14:30 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 2966.657126 secs 2024-03-24T01:14:30.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:14:30 smithi189 ceph-mon[36187]: pgmap v2257: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-03-24T01:14:30.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:14:30 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 2966.657126 secs 2024-03-24T01:14:30.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:14:30 smithi193 ceph-mon[36878]: pgmap v2257: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-03-24T01:14:30.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:14:30 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 2966.657126 secs 2024-03-24T01:14:31.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:14:31 smithi118 ceph-mon[31535]: pgmap v2258: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:14:31.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:14:31 smithi189 ceph-mon[36187]: pgmap v2258: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:14:31.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:14:31 smithi193 ceph-mon[36878]: pgmap v2258: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:14:34.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:14:34 smithi118 ceph-mon[31535]: pgmap v2259: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T01:14:34.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:14:34 smithi189 ceph-mon[36187]: pgmap v2259: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T01:14:34.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:14:34 smithi193 ceph-mon[36878]: pgmap v2259: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T01:14:35.848 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:14:35 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 2971.657214 secs 2024-03-24T01:14:35.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:14:35 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 2971.657214 secs 2024-03-24T01:14:35.963 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:14:35 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 2971.657214 secs 2024-03-24T01:14:36.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:14:36 smithi118 ceph-mon[31535]: pgmap v2260: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T01:14:36.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:14:36 smithi189 ceph-mon[36187]: pgmap v2260: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T01:14:36.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:14:36 smithi193 ceph-mon[36878]: pgmap v2260: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T01:14:38.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:14:38 smithi118 ceph-mon[31535]: pgmap v2261: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T01:14:38.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:14:38 smithi189 ceph-mon[36187]: pgmap v2261: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T01:14:38.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:14:38 smithi193 ceph-mon[36878]: pgmap v2261: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T01:14:40.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:14:40 smithi118 ceph-mon[31535]: pgmap v2262: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-03-24T01:14:40.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:14:40 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 2976.657299 secs 2024-03-24T01:14:40.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:14:40 smithi189 ceph-mon[36187]: pgmap v2262: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-03-24T01:14:40.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:14:40 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 2976.657299 secs 2024-03-24T01:14:40.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:14:40 smithi193 ceph-mon[36878]: pgmap v2262: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-03-24T01:14:40.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:14:40 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 2976.657299 secs 2024-03-24T01:14:42.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:14:42 smithi118 ceph-mon[31535]: pgmap v2263: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:14:42.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:14:42 smithi189 ceph-mon[36187]: pgmap v2263: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:14:42.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:14:42 smithi193 ceph-mon[36878]: pgmap v2263: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:14:44.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:14:44 smithi118 ceph-mon[31535]: pgmap v2264: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T01:14:44.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:14:44 smithi189 ceph-mon[36187]: pgmap v2264: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T01:14:44.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:14:44 smithi193 ceph-mon[36878]: pgmap v2264: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T01:14:45.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:14:45 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 2981.657361 secs 2024-03-24T01:14:45.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:14:45 smithi118 ceph-mon[31535]: pgmap v2265: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T01:14:45.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:14:45 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 2981.657361 secs 2024-03-24T01:14:45.955 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:14:45 smithi189 ceph-mon[36187]: pgmap v2265: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T01:14:45.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:14:45 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 2981.657361 secs 2024-03-24T01:14:45.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:14:45 smithi193 ceph-mon[36878]: pgmap v2265: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T01:14:46.205 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T01:14:46.205+0000 7f81ee000640 1 -- 172.21.15.118:0/1693767697 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- auth(proto 2 132 bytes epoch 0) v1 -- 0x7f81c80065b0 con 0x7f81f81052d0 2024-03-24T01:14:46.205 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T01:14:46.205+0000 7f81db400640 1 -- 172.21.15.118:0/659686424 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- auth(proto 2 132 bytes epoch 0) v1 -- 0x7f81c40062d0 con 0x5622651a13f0 2024-03-24T01:14:46.206 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T01:14:46.205+0000 7f81e4c00640 1 -- 172.21.15.118:0/659686424 <== mon.1 v2:172.21.15.193:3300/0 27 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 233+0+0 (secure 0 0 0) 0x7f81c40062d0 con 0x5622651a13f0 2024-03-24T01:14:46.206 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T01:14:46.205+0000 7f81ef400640 1 -- 172.21.15.118:0/1693767697 <== mon.2 v2:172.21.15.189:3300/0 27 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 233+0+0 (secure 0 0 0) 0x7f81c80065b0 con 0x7f81f81052d0 2024-03-24T01:14:48.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:14:48 smithi118 ceph-mon[31535]: pgmap v2266: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T01:14:48.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:14:48 smithi189 ceph-mon[36187]: pgmap v2266: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T01:14:48.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:14:48 smithi193 ceph-mon[36878]: pgmap v2266: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T01:14:50.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:14:50 smithi118 ceph-mon[31535]: pgmap v2267: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-03-24T01:14:50.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:14:50 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 2986.657420 secs 2024-03-24T01:14:50.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:14:50 smithi189 ceph-mon[36187]: pgmap v2267: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-03-24T01:14:50.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:14:50 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 2986.657420 secs 2024-03-24T01:14:50.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:14:50 smithi193 ceph-mon[36878]: pgmap v2267: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-03-24T01:14:50.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:14:50 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 2986.657420 secs 2024-03-24T01:14:52.756 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:14:52 smithi189 ceph-mon[36187]: pgmap v2268: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:14:52.763 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:14:52 smithi193 ceph-mon[36878]: pgmap v2268: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:14:52.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:14:52 smithi118 ceph-mon[31535]: pgmap v2268: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:14:54.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:14:54 smithi118 ceph-mon[31535]: pgmap v2269: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T01:14:54.947 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:14:54 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T01:14:54.947 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:14:54 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-24T01:14:54.947 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:14:54 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-24T01:14:54.947 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:14:54 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-03-24T01:14:54.947 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:14:54 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-03-24T01:14:54.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:14:54 smithi189 ceph-mon[36187]: pgmap v2269: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T01:14:54.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:14:54 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T01:14:54.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:14:54 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-24T01:14:54.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:14:54 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-24T01:14:54.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:14:54 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-03-24T01:14:54.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:14:54 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-03-24T01:14:54.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:14:54 smithi193 ceph-mon[36878]: pgmap v2269: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T01:14:54.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:14:54 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T01:14:54.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:14:54 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-24T01:14:54.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:14:54 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-24T01:14:54.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:14:54 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-03-24T01:14:54.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:14:54 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-03-24T01:14:55.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:14:55 smithi118 ceph-mon[31535]: Adjusting osd_memory_target on smithi193 to 174.1M 2024-03-24T01:14:55.947 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:14:55 smithi118 ceph-mon[31535]: Unable to set osd_memory_target on smithi193 to 182639923: error parsing value: Value '182639923' is below minimum 939524096 2024-03-24T01:14:55.947 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:14:55 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T01:14:55.947 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:14:55 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T01:14:55.947 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:14:55 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T01:14:55.947 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:14:55 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 2991.657483 secs 2024-03-24T01:14:55.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:14:55 smithi189 ceph-mon[36187]: Adjusting osd_memory_target on smithi193 to 174.1M 2024-03-24T01:14:55.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:14:55 smithi189 ceph-mon[36187]: Unable to set osd_memory_target on smithi193 to 182639923: error parsing value: Value '182639923' is below minimum 939524096 2024-03-24T01:14:55.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:14:55 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T01:14:55.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:14:55 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T01:14:55.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:14:55 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T01:14:55.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:14:55 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 2991.657483 secs 2024-03-24T01:14:55.962 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:14:55 smithi193 ceph-mon[36878]: Adjusting osd_memory_target on smithi193 to 174.1M 2024-03-24T01:14:55.962 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:14:55 smithi193 ceph-mon[36878]: Unable to set osd_memory_target on smithi193 to 182639923: error parsing value: Value '182639923' is below minimum 939524096 2024-03-24T01:14:55.962 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:14:55 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T01:14:55.962 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:14:55 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T01:14:55.963 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:14:55 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T01:14:55.963 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:14:55 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 2991.657483 secs 2024-03-24T01:14:56.920 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:14:56 smithi118 ceph-mon[31535]: pgmap v2270: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 6 op/s 2024-03-24T01:14:56.920 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:14:56 smithi118 ceph-mon[31535]: pgmap v2271: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 6 op/s 2024-03-24T01:14:56.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:14:56 smithi189 ceph-mon[36187]: pgmap v2270: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 6 op/s 2024-03-24T01:14:56.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:14:56 smithi189 ceph-mon[36187]: pgmap v2271: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 6 op/s 2024-03-24T01:14:56.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:14:56 smithi193 ceph-mon[36878]: pgmap v2270: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 6 op/s 2024-03-24T01:14:56.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:14:56 smithi193 ceph-mon[36878]: pgmap v2271: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 6 op/s 2024-03-24T01:14:57.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:14:57 smithi118 ceph-mon[31535]: pgmap v2272: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 5 op/s 2024-03-24T01:14:57.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:14:57 smithi189 ceph-mon[36187]: pgmap v2272: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 5 op/s 2024-03-24T01:14:57.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:14:57 smithi193 ceph-mon[36878]: pgmap v2272: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 5 op/s 2024-03-24T01:15:00.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:15:00 smithi118 ceph-mon[31535]: pgmap v2273: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 5 op/s 2024-03-24T01:15:00.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:15:00 smithi189 ceph-mon[36187]: pgmap v2273: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 5 op/s 2024-03-24T01:15:00.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:15:00 smithi193 ceph-mon[36878]: pgmap v2273: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 5 op/s 2024-03-24T01:15:01.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:15:01 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 2996.657544 secs 2024-03-24T01:15:01.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:15:01 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 2996.657544 secs 2024-03-24T01:15:01.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:15:01 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 2996.657544 secs 2024-03-24T01:15:02.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:15:02 smithi118 ceph-mon[31535]: pgmap v2274: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 6 op/s 2024-03-24T01:15:02.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:15:02 smithi189 ceph-mon[36187]: pgmap v2274: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 6 op/s 2024-03-24T01:15:02.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:15:02 smithi193 ceph-mon[36878]: pgmap v2274: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 6 op/s 2024-03-24T01:15:04.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:15:04 smithi118 ceph-mon[31535]: pgmap v2275: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 5 op/s 2024-03-24T01:15:04.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:15:04 smithi189 ceph-mon[36187]: pgmap v2275: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 5 op/s 2024-03-24T01:15:04.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:15:04 smithi193 ceph-mon[36878]: pgmap v2275: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 5 op/s 2024-03-24T01:15:06.447 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:15:06 smithi118 ceph-mon[31535]: pgmap v2276: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 5 op/s 2024-03-24T01:15:06.447 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:15:06 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 3001.657606 secs 2024-03-24T01:15:06.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:15:06 smithi189 ceph-mon[36187]: pgmap v2276: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 5 op/s 2024-03-24T01:15:06.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:15:06 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 3001.657606 secs 2024-03-24T01:15:06.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:15:06 smithi193 ceph-mon[36878]: pgmap v2276: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 5 op/s 2024-03-24T01:15:06.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:15:06 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 3001.657606 secs 2024-03-24T01:15:08.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:15:08 smithi118 ceph-mon[31535]: pgmap v2277: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:15:08.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:15:08 smithi189 ceph-mon[36187]: pgmap v2277: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:15:08.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:15:08 smithi193 ceph-mon[36878]: pgmap v2277: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:15:10.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:15:10 smithi118 ceph-mon[31535]: pgmap v2278: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T01:15:10.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:15:10 smithi189 ceph-mon[36187]: pgmap v2278: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T01:15:10.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:15:10 smithi193 ceph-mon[36878]: pgmap v2278: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T01:15:11.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:15:11 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 3006.657696 secs 2024-03-24T01:15:11.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:15:11 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 3006.657696 secs 2024-03-24T01:15:11.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:15:11 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 3006.657696 secs 2024-03-24T01:15:12.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:15:12 smithi118 ceph-mon[31535]: pgmap v2279: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T01:15:12.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:15:12 smithi189 ceph-mon[36187]: pgmap v2279: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T01:15:12.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:15:12 smithi193 ceph-mon[36878]: pgmap v2279: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T01:15:14.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:15:14 smithi118 ceph-mon[31535]: pgmap v2280: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 5 op/s 2024-03-24T01:15:14.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:15:14 smithi189 ceph-mon[36187]: pgmap v2280: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 5 op/s 2024-03-24T01:15:14.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:15:14 smithi193 ceph-mon[36878]: pgmap v2280: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 5 op/s 2024-03-24T01:15:16.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:15:16 smithi118 ceph-mon[31535]: pgmap v2281: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T01:15:16.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:15:16 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 3011.657709 secs 2024-03-24T01:15:16.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:15:16 smithi189 ceph-mon[36187]: pgmap v2281: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T01:15:16.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:15:16 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 3011.657709 secs 2024-03-24T01:15:16.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:15:16 smithi193 ceph-mon[36878]: pgmap v2281: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T01:15:16.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:15:16 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 3011.657709 secs 2024-03-24T01:15:18.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:15:18 smithi118 ceph-mon[31535]: pgmap v2282: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:15:18.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:15:18 smithi189 ceph-mon[36187]: pgmap v2282: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:15:18.460 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:15:18 smithi193 ceph-mon[36878]: pgmap v2282: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:15:20.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:15:20 smithi118 ceph-mon[31535]: pgmap v2283: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T01:15:20.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:15:20 smithi189 ceph-mon[36187]: pgmap v2283: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T01:15:20.460 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:15:20 smithi193 ceph-mon[36878]: pgmap v2283: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T01:15:21.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:15:21 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 3016.657792 secs 2024-03-24T01:15:21.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:15:21 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 3016.657792 secs 2024-03-24T01:15:21.460 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:15:21 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 3016.657792 secs 2024-03-24T01:15:22.362 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:15:22 smithi193 ceph-mon[36878]: pgmap v2284: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T01:15:22.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:15:22 smithi118 ceph-mon[31535]: pgmap v2284: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T01:15:22.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:15:22 smithi189 ceph-mon[36187]: pgmap v2284: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T01:15:24.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:15:24 smithi118 ceph-mon[31535]: pgmap v2285: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 5 op/s 2024-03-24T01:15:24.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:15:24 smithi189 ceph-mon[36187]: pgmap v2285: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 5 op/s 2024-03-24T01:15:24.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:15:24 smithi193 ceph-mon[36878]: pgmap v2285: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 5 op/s 2024-03-24T01:15:26.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:15:26 smithi118 ceph-mon[31535]: pgmap v2286: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T01:15:26.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:15:26 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 3021.657881 secs 2024-03-24T01:15:26.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:15:26 smithi189 ceph-mon[36187]: pgmap v2286: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T01:15:26.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:15:26 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 3021.657881 secs 2024-03-24T01:15:26.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:15:26 smithi193 ceph-mon[36878]: pgmap v2286: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T01:15:26.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:15:26 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 3021.657881 secs 2024-03-24T01:15:28.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:15:28 smithi118 ceph-mon[31535]: pgmap v2287: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:15:28.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:15:28 smithi189 ceph-mon[36187]: pgmap v2287: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:15:28.460 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:15:28 smithi193 ceph-mon[36878]: pgmap v2287: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:15:30.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:15:30 smithi118 ceph-mon[31535]: pgmap v2288: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T01:15:30.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:15:30 smithi189 ceph-mon[36187]: pgmap v2288: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T01:15:30.460 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:15:30 smithi193 ceph-mon[36878]: pgmap v2288: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T01:15:31.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:15:31 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 3026.657971 secs 2024-03-24T01:15:31.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:15:31 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 3026.657971 secs 2024-03-24T01:15:31.460 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:15:31 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 3026.657971 secs 2024-03-24T01:15:32.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:15:32 smithi118 ceph-mon[31535]: pgmap v2289: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T01:15:32.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:15:32 smithi189 ceph-mon[36187]: pgmap v2289: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T01:15:32.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:15:32 smithi193 ceph-mon[36878]: pgmap v2289: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T01:15:34.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:15:34 smithi118 ceph-mon[31535]: pgmap v2290: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T01:15:34.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:15:34 smithi189 ceph-mon[36187]: pgmap v2290: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T01:15:34.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:15:34 smithi193 ceph-mon[36878]: pgmap v2290: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T01:15:36.447 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:15:36 smithi118 ceph-mon[31535]: pgmap v2291: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T01:15:36.447 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:15:36 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 3031.657988 secs 2024-03-24T01:15:36.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:15:36 smithi189 ceph-mon[36187]: pgmap v2291: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T01:15:36.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:15:36 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 3031.657988 secs 2024-03-24T01:15:36.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:15:36 smithi193 ceph-mon[36878]: pgmap v2291: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T01:15:36.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:15:36 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 3031.657988 secs 2024-03-24T01:15:38.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:15:38 smithi118 ceph-mon[31535]: pgmap v2292: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:15:38.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:15:38 smithi189 ceph-mon[36187]: pgmap v2292: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:15:38.460 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:15:38 smithi193 ceph-mon[36878]: pgmap v2292: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:15:40.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:15:40 smithi118 ceph-mon[31535]: pgmap v2293: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T01:15:40.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:15:40 smithi189 ceph-mon[36187]: pgmap v2293: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T01:15:40.460 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:15:40 smithi193 ceph-mon[36878]: pgmap v2293: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T01:15:41.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:15:41 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 3036.658078 secs 2024-03-24T01:15:41.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:15:41 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 3036.658078 secs 2024-03-24T01:15:41.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:15:41 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 3036.658078 secs 2024-03-24T01:15:42.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:15:42 smithi118 ceph-mon[31535]: pgmap v2294: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T01:15:42.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:15:42 smithi189 ceph-mon[36187]: pgmap v2294: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T01:15:42.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:15:42 smithi193 ceph-mon[36878]: pgmap v2294: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T01:15:44.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:15:44 smithi118 ceph-mon[31535]: pgmap v2295: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T01:15:44.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:15:44 smithi189 ceph-mon[36187]: pgmap v2295: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T01:15:44.460 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:15:44 smithi193 ceph-mon[36878]: pgmap v2295: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T01:15:46.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:15:46 smithi118 ceph-mon[31535]: pgmap v2296: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T01:15:46.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:15:46 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 3041.658162 secs 2024-03-24T01:15:46.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:15:46 smithi189 ceph-mon[36187]: pgmap v2296: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T01:15:46.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:15:46 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 3041.658162 secs 2024-03-24T01:15:46.460 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:15:46 smithi193 ceph-mon[36878]: pgmap v2296: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T01:15:46.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:15:46 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 3041.658162 secs 2024-03-24T01:15:48.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:15:48 smithi118 ceph-mon[31535]: pgmap v2297: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:15:48.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:15:48 smithi189 ceph-mon[36187]: pgmap v2297: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:15:48.460 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:15:48 smithi193 ceph-mon[36878]: pgmap v2297: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:15:50.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:15:50 smithi118 ceph-mon[31535]: pgmap v2298: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T01:15:50.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:15:50 smithi189 ceph-mon[36187]: pgmap v2298: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T01:15:50.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:15:50 smithi193 ceph-mon[36878]: pgmap v2298: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T01:15:51.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:15:51 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 3046.658223 secs 2024-03-24T01:15:51.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:15:51 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 3046.658223 secs 2024-03-24T01:15:51.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:15:51 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 3046.658223 secs 2024-03-24T01:15:52.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:15:52 smithi118 ceph-mon[31535]: pgmap v2299: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T01:15:52.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:15:52 smithi189 ceph-mon[36187]: pgmap v2299: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T01:15:52.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:15:52 smithi193 ceph-mon[36878]: pgmap v2299: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T01:15:54.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:15:54 smithi189 ceph-mon[36187]: pgmap v2300: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T01:15:54.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:15:54 smithi193 ceph-mon[36878]: pgmap v2300: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T01:15:54.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:15:54 smithi118 ceph-mon[31535]: pgmap v2300: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T01:15:56.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:15:56 smithi189 ceph-mon[36187]: pgmap v2301: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T01:15:56.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:15:56 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 3051.658309 secs 2024-03-24T01:15:56.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:15:56 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T01:15:56.460 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:15:56 smithi193 ceph-mon[36878]: pgmap v2301: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T01:15:56.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:15:56 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 3051.658309 secs 2024-03-24T01:15:56.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:15:56 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T01:15:56.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:15:56 smithi118 ceph-mon[31535]: pgmap v2301: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T01:15:56.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:15:56 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 3051.658309 secs 2024-03-24T01:15:56.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:15:56 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T01:15:57.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:15:57 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T01:15:57.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:15:57 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T01:15:57.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:15:57 smithi193 ceph-mon[36878]: pgmap v2302: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 5 op/s 2024-03-24T01:15:57.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:15:57 smithi193 ceph-mon[36878]: pgmap v2303: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-03-24T01:15:57.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:15:57 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T01:15:57.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:15:57 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T01:15:57.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:15:57 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T01:15:57.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:15:57 smithi118 ceph-mon[31535]: pgmap v2302: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 5 op/s 2024-03-24T01:15:57.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:15:57 smithi118 ceph-mon[31535]: pgmap v2303: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-03-24T01:15:57.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:15:57 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T01:15:57.704 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:15:57 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T01:15:57.704 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:15:57 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T01:15:57.704 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:15:57 smithi189 ceph-mon[36187]: pgmap v2302: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 5 op/s 2024-03-24T01:15:57.704 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:15:57 smithi189 ceph-mon[36187]: pgmap v2303: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-03-24T01:15:57.704 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:15:57 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T01:15:58.872 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:15:58 smithi118 ceph-mon[31535]: pgmap v2304: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 5 op/s 2024-03-24T01:15:58.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:15:58 smithi189 ceph-mon[36187]: pgmap v2304: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 5 op/s 2024-03-24T01:15:58.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:15:58 smithi193 ceph-mon[36878]: pgmap v2304: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 5 op/s 2024-03-24T01:16:00.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:16:00 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 3056.658393 secs 2024-03-24T01:16:00.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:16:00 smithi118 ceph-mon[31535]: pgmap v2305: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-03-24T01:16:00.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:16:00 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 3056.658393 secs 2024-03-24T01:16:00.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:16:00 smithi189 ceph-mon[36187]: pgmap v2305: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-03-24T01:16:00.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:16:00 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 3056.658393 secs 2024-03-24T01:16:00.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:16:00 smithi193 ceph-mon[36878]: pgmap v2305: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-03-24T01:16:02.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:16:02 smithi118 ceph-mon[31535]: pgmap v2306: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 5 op/s 2024-03-24T01:16:02.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:16:02 smithi189 ceph-mon[36187]: pgmap v2306: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 5 op/s 2024-03-24T01:16:02.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:16:02 smithi193 ceph-mon[36878]: pgmap v2306: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 5 op/s 2024-03-24T01:16:04.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:16:04 smithi118 ceph-mon[31535]: pgmap v2307: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-03-24T01:16:04.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:16:04 smithi189 ceph-mon[36187]: pgmap v2307: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-03-24T01:16:04.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:16:04 smithi193 ceph-mon[36878]: pgmap v2307: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-03-24T01:16:05.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:16:05 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 3061.658455 secs 2024-03-24T01:16:05.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:16:05 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 3061.658455 secs 2024-03-24T01:16:05.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:16:05 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 3061.658455 secs 2024-03-24T01:16:06.848 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:16:06 smithi118 ceph-mon[31535]: pgmap v2308: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 5 op/s 2024-03-24T01:16:06.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:16:06 smithi189 ceph-mon[36187]: pgmap v2308: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 5 op/s 2024-03-24T01:16:06.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:16:06 smithi193 ceph-mon[36878]: pgmap v2308: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 5 op/s 2024-03-24T01:16:08.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:16:08 smithi118 ceph-mon[31535]: pgmap v2309: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T01:16:08.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:16:08 smithi189 ceph-mon[36187]: pgmap v2309: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T01:16:08.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:16:08 smithi193 ceph-mon[36878]: pgmap v2309: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T01:16:10.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:16:10 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 3066.658492 secs 2024-03-24T01:16:10.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:16:10 smithi118 ceph-mon[31535]: pgmap v2310: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T01:16:10.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:16:10 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 3066.658492 secs 2024-03-24T01:16:10.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:16:10 smithi189 ceph-mon[36187]: pgmap v2310: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T01:16:10.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:16:10 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 3066.658492 secs 2024-03-24T01:16:10.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:16:10 smithi193 ceph-mon[36878]: pgmap v2310: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T01:16:12.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:16:12 smithi118 ceph-mon[31535]: pgmap v2311: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T01:16:12.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:16:12 smithi189 ceph-mon[36187]: pgmap v2311: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T01:16:12.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:16:12 smithi193 ceph-mon[36878]: pgmap v2311: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T01:16:14.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:16:14 smithi118 ceph-mon[31535]: pgmap v2312: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-03-24T01:16:14.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:16:14 smithi189 ceph-mon[36187]: pgmap v2312: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-03-24T01:16:14.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:16:14 smithi193 ceph-mon[36878]: pgmap v2312: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-03-24T01:16:15.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:16:15 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 3071.658553 secs 2024-03-24T01:16:15.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:16:15 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 3071.658553 secs 2024-03-24T01:16:15.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:16:15 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 3071.658553 secs 2024-03-24T01:16:16.219 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T01:16:16.219+0000 7f81ee000640 1 -- 172.21.15.118:0/1693767697 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f81c80065b0 con 0x7f81f81052d0 2024-03-24T01:16:16.220 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T01:16:16.219+0000 7f81db400640 1 -- 172.21.15.118:0/659686424 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f81c4006290 con 0x5622651a13f0 2024-03-24T01:16:16.220 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T01:16:16.220+0000 7f81e4c00640 1 -- 172.21.15.118:0/659686424 <== mon.1 v2:172.21.15.193:3300/0 28 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 409+0+0 (secure 0 0 0) 0x7f81c4006290 con 0x5622651a13f0 2024-03-24T01:16:16.220 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T01:16:16.220+0000 7f81ef400640 1 -- 172.21.15.118:0/1693767697 <== mon.2 v2:172.21.15.189:3300/0 28 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (secure 0 0 0) 0x7f81c80065b0 con 0x7f81f81052d0 2024-03-24T01:16:16.950 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:16:16 smithi118 ceph-mon[31535]: pgmap v2313: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:16:16.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:16:16 smithi189 ceph-mon[36187]: pgmap v2313: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:16:16.962 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:16:16 smithi193 ceph-mon[36878]: pgmap v2313: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:16:18.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:16:18 smithi118 ceph-mon[31535]: pgmap v2314: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T01:16:18.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:16:18 smithi189 ceph-mon[36187]: pgmap v2314: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T01:16:18.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:16:18 smithi193 ceph-mon[36878]: pgmap v2314: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T01:16:20.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:16:20 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 3076.658612 secs 2024-03-24T01:16:20.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:16:20 smithi118 ceph-mon[31535]: pgmap v2315: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T01:16:20.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:16:20 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 3076.658612 secs 2024-03-24T01:16:20.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:16:20 smithi189 ceph-mon[36187]: pgmap v2315: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T01:16:20.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:16:20 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 3076.658612 secs 2024-03-24T01:16:20.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:16:20 smithi193 ceph-mon[36878]: pgmap v2315: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T01:16:22.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:16:22 smithi118 ceph-mon[31535]: pgmap v2316: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T01:16:22.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:16:22 smithi189 ceph-mon[36187]: pgmap v2316: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T01:16:22.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:16:22 smithi193 ceph-mon[36878]: pgmap v2316: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T01:16:24.947 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:16:24 smithi118 ceph-mon[31535]: pgmap v2317: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-03-24T01:16:24.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:16:24 smithi189 ceph-mon[36187]: pgmap v2317: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-03-24T01:16:24.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:16:24 smithi193 ceph-mon[36878]: pgmap v2317: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-03-24T01:16:25.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:16:25 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 3081.658703 secs 2024-03-24T01:16:25.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:16:25 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 3081.658703 secs 2024-03-24T01:16:25.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:16:25 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 3081.658703 secs 2024-03-24T01:16:26.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:16:26 smithi118 ceph-mon[31535]: pgmap v2318: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:16:26.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:16:26 smithi189 ceph-mon[36187]: pgmap v2318: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:16:26.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:16:26 smithi193 ceph-mon[36878]: pgmap v2318: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:16:28.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:16:28 smithi118 ceph-mon[31535]: pgmap v2319: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T01:16:28.956 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:16:28 smithi189 ceph-mon[36187]: pgmap v2319: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T01:16:28.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:16:28 smithi193 ceph-mon[36878]: pgmap v2319: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T01:16:30.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:16:30 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 3086.658763 secs 2024-03-24T01:16:30.953 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:16:30 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 3086.658763 secs 2024-03-24T01:16:30.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:16:30 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 3086.658763 secs 2024-03-24T01:16:31.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:16:31 smithi118 ceph-mon[31535]: pgmap v2320: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T01:16:31.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:16:31 smithi189 ceph-mon[36187]: pgmap v2320: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T01:16:31.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:16:31 smithi193 ceph-mon[36878]: pgmap v2320: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T01:16:32.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:16:32 smithi118 ceph-mon[31535]: pgmap v2321: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T01:16:32.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:16:32 smithi189 ceph-mon[36187]: pgmap v2321: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T01:16:32.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:16:32 smithi193 ceph-mon[36878]: pgmap v2321: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T01:16:34.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:16:34 smithi118 ceph-mon[31535]: pgmap v2322: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-03-24T01:16:34.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:16:34 smithi189 ceph-mon[36187]: pgmap v2322: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-03-24T01:16:34.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:16:34 smithi193 ceph-mon[36878]: pgmap v2322: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-03-24T01:16:35.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:16:35 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 3091.658854 secs 2024-03-24T01:16:35.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:16:35 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 3091.658854 secs 2024-03-24T01:16:35.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:16:35 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 3091.658854 secs 2024-03-24T01:16:36.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:16:36 smithi118 ceph-mon[31535]: pgmap v2323: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:16:36.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:16:36 smithi189 ceph-mon[36187]: pgmap v2323: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:16:36.962 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:16:36 smithi193 ceph-mon[36878]: pgmap v2323: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:16:38.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:16:38 smithi118 ceph-mon[31535]: pgmap v2324: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T01:16:38.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:16:38 smithi189 ceph-mon[36187]: pgmap v2324: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T01:16:38.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:16:38 smithi193 ceph-mon[36878]: pgmap v2324: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T01:16:40.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:16:40 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 3096.658913 secs 2024-03-24T01:16:40.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:16:40 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 3096.658913 secs 2024-03-24T01:16:40.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:16:40 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 3096.658913 secs 2024-03-24T01:16:41.880 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:16:41 smithi118 ceph-mon[31535]: pgmap v2325: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 6 op/s 2024-03-24T01:16:41.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:16:41 smithi189 ceph-mon[36187]: pgmap v2325: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 6 op/s 2024-03-24T01:16:41.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:16:41 smithi193 ceph-mon[36878]: pgmap v2325: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 6 op/s 2024-03-24T01:16:42.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:16:42 smithi118 ceph-mon[31535]: pgmap v2326: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 5 op/s 2024-03-24T01:16:42.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:16:42 smithi189 ceph-mon[36187]: pgmap v2326: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 5 op/s 2024-03-24T01:16:42.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:16:42 smithi193 ceph-mon[36878]: pgmap v2326: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 5 op/s 2024-03-24T01:16:44.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:16:44 smithi118 ceph-mon[31535]: pgmap v2327: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-03-24T01:16:44.953 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:16:44 smithi189 ceph-mon[36187]: pgmap v2327: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-03-24T01:16:44.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:16:44 smithi193 ceph-mon[36878]: pgmap v2327: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-03-24T01:16:45.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:16:45 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 3101.658952 secs 2024-03-24T01:16:45.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:16:45 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 3101.658952 secs 2024-03-24T01:16:45.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:16:45 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 3101.658952 secs 2024-03-24T01:16:46.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:16:46 smithi118 ceph-mon[31535]: pgmap v2328: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-03-24T01:16:46.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:16:46 smithi189 ceph-mon[36187]: pgmap v2328: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-03-24T01:16:46.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:16:46 smithi193 ceph-mon[36878]: pgmap v2328: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-03-24T01:16:48.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:16:48 smithi118 ceph-mon[31535]: pgmap v2329: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-03-24T01:16:48.953 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:16:48 smithi189 ceph-mon[36187]: pgmap v2329: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-03-24T01:16:48.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:16:48 smithi193 ceph-mon[36878]: pgmap v2329: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-03-24T01:16:50.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:16:50 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 3106.659039 secs 2024-03-24T01:16:50.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:16:50 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 3106.659039 secs 2024-03-24T01:16:50.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:16:50 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 3106.659039 secs 2024-03-24T01:16:51.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:16:51 smithi118 ceph-mon[31535]: pgmap v2330: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 6 op/s 2024-03-24T01:16:51.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:16:51 smithi189 ceph-mon[36187]: pgmap v2330: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 6 op/s 2024-03-24T01:16:51.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:16:51 smithi193 ceph-mon[36878]: pgmap v2330: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 6 op/s 2024-03-24T01:16:52.854 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:16:52 smithi189 ceph-mon[36187]: pgmap v2331: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T01:16:52.861 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:16:52 smithi193 ceph-mon[36878]: pgmap v2331: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T01:16:52.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:16:52 smithi118 ceph-mon[31535]: pgmap v2331: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T01:16:54.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:16:54 smithi118 ceph-mon[31535]: pgmap v2332: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-03-24T01:16:54.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:16:54 smithi189 ceph-mon[36187]: pgmap v2332: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-03-24T01:16:54.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:16:54 smithi193 ceph-mon[36878]: pgmap v2332: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-03-24T01:16:55.947 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:16:55 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 3111.659107 secs 2024-03-24T01:16:55.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:16:55 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 3111.659107 secs 2024-03-24T01:16:55.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:16:55 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 3111.659107 secs 2024-03-24T01:16:56.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:16:56 smithi118 ceph-mon[31535]: pgmap v2333: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:16:56.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:16:56 smithi189 ceph-mon[36187]: pgmap v2333: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:16:56.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:16:56 smithi193 ceph-mon[36878]: pgmap v2333: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:16:57.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:16:57 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T01:16:57.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:16:57 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T01:16:57.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:16:57 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T01:16:58.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:16:58 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T01:16:58.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:16:58 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T01:16:58.947 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:16:58 smithi118 ceph-mon[31535]: pgmap v2334: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 6 op/s 2024-03-24T01:16:58.947 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:16:58 smithi118 ceph-mon[31535]: pgmap v2335: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-03-24T01:16:58.947 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:16:58 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T01:16:58.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:16:58 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T01:16:58.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:16:58 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T01:16:58.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:16:58 smithi189 ceph-mon[36187]: pgmap v2334: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 6 op/s 2024-03-24T01:16:58.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:16:58 smithi189 ceph-mon[36187]: pgmap v2335: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-03-24T01:16:58.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:16:58 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T01:16:58.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:16:58 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T01:16:58.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:16:58 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T01:16:58.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:16:58 smithi193 ceph-mon[36878]: pgmap v2334: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 6 op/s 2024-03-24T01:16:58.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:16:58 smithi193 ceph-mon[36878]: pgmap v2335: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-03-24T01:16:58.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:16:58 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T01:17:01.336 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:17:01 smithi118 ceph-mon[31535]: pgmap v2336: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-03-24T01:17:01.337 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:17:01 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 3116.659197 secs 2024-03-24T01:17:01.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:17:01 smithi189 ceph-mon[36187]: pgmap v2336: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-03-24T01:17:01.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:17:01 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 3116.659197 secs 2024-03-24T01:17:01.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:17:01 smithi193 ceph-mon[36878]: pgmap v2336: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-03-24T01:17:01.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:17:01 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 3116.659197 secs 2024-03-24T01:17:03.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:17:03 smithi118 ceph-mon[31535]: pgmap v2337: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 5 op/s 2024-03-24T01:17:03.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:17:03 smithi189 ceph-mon[36187]: pgmap v2337: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 5 op/s 2024-03-24T01:17:03.460 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:17:03 smithi193 ceph-mon[36878]: pgmap v2337: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 5 op/s 2024-03-24T01:17:05.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:17:05 smithi118 ceph-mon[31535]: pgmap v2338: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-03-24T01:17:05.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:17:05 smithi189 ceph-mon[36187]: pgmap v2338: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-03-24T01:17:05.460 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:17:05 smithi193 ceph-mon[36878]: pgmap v2338: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-03-24T01:17:06.348 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:17:06 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 3121.659258 secs 2024-03-24T01:17:06.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:17:06 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 3121.659258 secs 2024-03-24T01:17:06.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:17:06 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 3121.659258 secs 2024-03-24T01:17:07.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:17:07 smithi118 ceph-mon[31535]: pgmap v2339: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-03-24T01:17:07.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:17:07 smithi189 ceph-mon[36187]: pgmap v2339: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-03-24T01:17:07.460 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:17:07 smithi193 ceph-mon[36878]: pgmap v2339: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-03-24T01:17:09.447 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:17:09 smithi118 ceph-mon[31535]: pgmap v2340: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 5 op/s 2024-03-24T01:17:09.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:17:09 smithi189 ceph-mon[36187]: pgmap v2340: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 5 op/s 2024-03-24T01:17:09.460 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:17:09 smithi193 ceph-mon[36878]: pgmap v2340: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 5 op/s 2024-03-24T01:17:11.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:17:11 smithi118 ceph-mon[31535]: pgmap v2341: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T01:17:11.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:17:11 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 3126.659339 secs 2024-03-24T01:17:11.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:17:11 smithi189 ceph-mon[36187]: pgmap v2341: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T01:17:11.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:17:11 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 3126.659339 secs 2024-03-24T01:17:11.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:17:11 smithi193 ceph-mon[36878]: pgmap v2341: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T01:17:11.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:17:11 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 3126.659339 secs 2024-03-24T01:17:13.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:17:13 smithi118 ceph-mon[31535]: pgmap v2342: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:17:13.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:17:13 smithi189 ceph-mon[36187]: pgmap v2342: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:17:13.460 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:17:13 smithi193 ceph-mon[36878]: pgmap v2342: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:17:15.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:17:15 smithi118 ceph-mon[31535]: pgmap v2343: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-24T01:17:15.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:17:15 smithi189 ceph-mon[36187]: pgmap v2343: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-24T01:17:15.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:17:15 smithi193 ceph-mon[36878]: pgmap v2343: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-24T01:17:16.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:17:16 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 3131.659430 secs 2024-03-24T01:17:16.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:17:16 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 3131.659430 secs 2024-03-24T01:17:16.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:17:16 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 3131.659430 secs 2024-03-24T01:17:17.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:17:17 smithi118 ceph-mon[31535]: pgmap v2344: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:17:17.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:17:17 smithi189 ceph-mon[36187]: pgmap v2344: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:17:17.460 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:17:17 smithi193 ceph-mon[36878]: pgmap v2344: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:17:19.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:17:19 smithi118 ceph-mon[31535]: pgmap v2345: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T01:17:19.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:17:19 smithi189 ceph-mon[36187]: pgmap v2345: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T01:17:19.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:17:19 smithi193 ceph-mon[36878]: pgmap v2345: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T01:17:21.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:17:21 smithi118 ceph-mon[31535]: pgmap v2346: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T01:17:21.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:17:21 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 3136.659518 secs 2024-03-24T01:17:21.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:17:21 smithi189 ceph-mon[36187]: pgmap v2346: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T01:17:21.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:17:21 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 3136.659518 secs 2024-03-24T01:17:21.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:17:21 smithi193 ceph-mon[36878]: pgmap v2346: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T01:17:21.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:17:21 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 3136.659518 secs 2024-03-24T01:17:23.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:17:23 smithi118 ceph-mon[31535]: pgmap v2347: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:17:23.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:17:23 smithi189 ceph-mon[36187]: pgmap v2347: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:17:23.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:17:23 smithi193 ceph-mon[36878]: pgmap v2347: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:17:25.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:17:25 smithi118 ceph-mon[31535]: pgmap v2348: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-24T01:17:25.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:17:25 smithi189 ceph-mon[36187]: pgmap v2348: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-24T01:17:25.460 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:17:25 smithi193 ceph-mon[36878]: pgmap v2348: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-24T01:17:26.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:17:26 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 3141.659611 secs 2024-03-24T01:17:26.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:17:26 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 3141.659611 secs 2024-03-24T01:17:26.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:17:26 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 3141.659611 secs 2024-03-24T01:17:27.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:17:27 smithi118 ceph-mon[31535]: pgmap v2349: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:17:27.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:17:27 smithi189 ceph-mon[36187]: pgmap v2349: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:17:27.460 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:17:27 smithi193 ceph-mon[36878]: pgmap v2349: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:17:29.449 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:17:29 smithi118 ceph-mon[31535]: pgmap v2350: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T01:17:29.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:17:29 smithi189 ceph-mon[36187]: pgmap v2350: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T01:17:29.460 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:17:29 smithi193 ceph-mon[36878]: pgmap v2350: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T01:17:31.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:17:31 smithi118 ceph-mon[31535]: pgmap v2351: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T01:17:31.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:17:31 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 3146.659673 secs 2024-03-24T01:17:31.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:17:31 smithi189 ceph-mon[36187]: pgmap v2351: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T01:17:31.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:17:31 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 3146.659673 secs 2024-03-24T01:17:31.460 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:17:31 smithi193 ceph-mon[36878]: pgmap v2351: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T01:17:31.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:17:31 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 3146.659673 secs 2024-03-24T01:17:33.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:17:33 smithi118 ceph-mon[31535]: pgmap v2352: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:17:33.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:17:33 smithi189 ceph-mon[36187]: pgmap v2352: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:17:33.460 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:17:33 smithi193 ceph-mon[36878]: pgmap v2352: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:17:35.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:17:35 smithi118 ceph-mon[31535]: pgmap v2353: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-24T01:17:35.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:17:35 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 3151.659759 secs 2024-03-24T01:17:35.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:17:35 smithi189 ceph-mon[36187]: pgmap v2353: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-24T01:17:35.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:17:35 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 3151.659759 secs 2024-03-24T01:17:35.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:17:35 smithi193 ceph-mon[36878]: pgmap v2353: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-24T01:17:35.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:17:35 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 3151.659759 secs 2024-03-24T01:17:37.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:17:37 smithi118 ceph-mon[31535]: pgmap v2354: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-03-24T01:17:37.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:17:37 smithi189 ceph-mon[36187]: pgmap v2354: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-03-24T01:17:37.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:17:37 smithi193 ceph-mon[36878]: pgmap v2354: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-03-24T01:17:39.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:17:39 smithi118 ceph-mon[31535]: pgmap v2355: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-03-24T01:17:39.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:17:39 smithi189 ceph-mon[36187]: pgmap v2355: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-03-24T01:17:39.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:17:39 smithi193 ceph-mon[36878]: pgmap v2355: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-03-24T01:17:40.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:17:40 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 3156.659822 secs 2024-03-24T01:17:40.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:17:40 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 3156.659822 secs 2024-03-24T01:17:40.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:17:40 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 3156.659822 secs 2024-03-24T01:17:41.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:17:41 smithi118 ceph-mon[31535]: pgmap v2356: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 6 op/s 2024-03-24T01:17:41.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:17:41 smithi189 ceph-mon[36187]: pgmap v2356: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 6 op/s 2024-03-24T01:17:41.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:17:41 smithi193 ceph-mon[36878]: pgmap v2356: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 6 op/s 2024-03-24T01:17:43.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:17:43 smithi118 ceph-mon[31535]: pgmap v2357: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-03-24T01:17:43.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:17:43 smithi189 ceph-mon[36187]: pgmap v2357: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-03-24T01:17:43.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:17:43 smithi193 ceph-mon[36878]: pgmap v2357: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-03-24T01:17:45.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:17:45 smithi118 ceph-mon[31535]: pgmap v2358: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-24T01:17:45.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:17:45 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 3161.659909 secs 2024-03-24T01:17:45.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:17:45 smithi189 ceph-mon[36187]: pgmap v2358: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-24T01:17:45.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:17:45 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 3161.659909 secs 2024-03-24T01:17:45.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:17:45 smithi193 ceph-mon[36878]: pgmap v2358: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-24T01:17:45.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:17:45 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 3161.659909 secs 2024-03-24T01:17:47.928 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:17:47 smithi118 ceph-mon[31535]: pgmap v2359: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:17:47.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:17:47 smithi189 ceph-mon[36187]: pgmap v2359: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:17:47.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:17:47 smithi193 ceph-mon[36878]: pgmap v2359: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:17:49.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:17:49 smithi118 ceph-mon[31535]: pgmap v2360: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T01:17:49.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:17:49 smithi189 ceph-mon[36187]: pgmap v2360: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T01:17:49.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:17:49 smithi193 ceph-mon[36878]: pgmap v2360: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T01:17:50.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:17:50 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 3166.659945 secs 2024-03-24T01:17:50.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:17:50 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 3166.659945 secs 2024-03-24T01:17:50.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:17:50 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 3166.659945 secs 2024-03-24T01:17:51.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:17:51 smithi118 ceph-mon[31535]: pgmap v2361: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T01:17:51.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:17:51 smithi189 ceph-mon[36187]: pgmap v2361: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T01:17:51.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:17:51 smithi193 ceph-mon[36878]: pgmap v2361: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T01:17:52.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:17:52 smithi118 ceph-mon[31535]: pgmap v2362: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:17:52.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:17:52 smithi189 ceph-mon[36187]: pgmap v2362: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:17:52.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:17:52 smithi193 ceph-mon[36878]: pgmap v2362: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:17:55.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:17:55 smithi118 ceph-mon[31535]: pgmap v2363: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-24T01:17:55.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:17:55 smithi189 ceph-mon[36187]: pgmap v2363: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-24T01:17:55.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:17:55 smithi193 ceph-mon[36878]: pgmap v2363: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-24T01:17:56.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:17:56 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 3171.659988 secs 2024-03-24T01:17:56.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:17:56 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 3171.659988 secs 2024-03-24T01:17:56.460 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:17:56 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 3171.659988 secs 2024-03-24T01:17:57.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:17:57 smithi118 ceph-mon[31535]: pgmap v2364: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:17:57.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:17:57 smithi189 ceph-mon[36187]: pgmap v2364: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:17:57.460 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:17:57 smithi193 ceph-mon[36878]: pgmap v2364: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:17:59.386 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:17:59 smithi193 ceph-mon[36878]: pgmap v2365: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T01:17:59.386 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:17:59 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T01:17:59.386 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:17:59 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T01:17:59.386 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:17:59 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T01:17:59.386 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:17:59 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T01:17:59.387 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:17:59 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T01:17:59.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:17:59 smithi118 ceph-mon[31535]: pgmap v2365: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T01:17:59.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:17:59 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T01:17:59.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:17:59 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T01:17:59.447 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:17:59 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T01:17:59.447 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:17:59 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T01:17:59.447 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:17:59 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T01:17:59.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:17:59 smithi189 ceph-mon[36187]: pgmap v2365: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T01:17:59.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:17:59 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T01:17:59.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:17:59 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T01:17:59.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:17:59 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T01:17:59.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:17:59 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T01:17:59.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:17:59 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T01:18:01.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:18:01 smithi118 ceph-mon[31535]: pgmap v2366: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T01:18:01.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:18:01 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 3176.660059 secs 2024-03-24T01:18:01.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:18:01 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T01:18:01.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:18:01 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T01:18:01.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:18:01 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T01:18:01.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:18:01 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T01:18:01.704 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:18:01 smithi189 ceph-mon[36187]: pgmap v2366: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T01:18:01.704 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:18:01 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 3176.660059 secs 2024-03-24T01:18:01.704 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:18:01 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T01:18:01.704 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:18:01 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T01:18:01.704 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:18:01 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T01:18:01.704 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:18:01 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T01:18:01.710 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:18:01 smithi193 ceph-mon[36878]: pgmap v2366: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T01:18:01.711 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:18:01 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 3176.660059 secs 2024-03-24T01:18:01.711 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:18:01 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T01:18:01.711 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:18:01 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T01:18:01.711 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:18:01 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T01:18:01.711 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:18:01 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T01:18:03.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:18:03 smithi118 ceph-mon[31535]: pgmap v2367: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:18:03.697 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:18:03 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T01:18:03.697 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:18:03 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T01:18:03.704 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:18:03 smithi189 ceph-mon[36187]: pgmap v2367: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:18:03.704 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:18:03 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T01:18:03.704 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:18:03 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T01:18:03.711 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:18:03 smithi193 ceph-mon[36878]: pgmap v2367: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:18:03.711 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:18:03 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T01:18:03.711 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:18:03 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T01:18:04.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:18:04 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T01:18:04.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:18:04 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T01:18:04.704 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:18:04 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T01:18:04.704 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:18:04 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T01:18:04.711 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:18:04 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T01:18:04.711 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:18:04 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T01:18:05.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:18:05 smithi118 ceph-mon[31535]: pgmap v2368: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-24T01:18:05.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:18:05 smithi118 ceph-mon[31535]: pgmap v2369: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T01:18:05.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:18:05 smithi118 ceph-mon[31535]: pgmap v2370: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-03-24T01:18:05.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:18:05 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T01:18:05.704 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:18:05 smithi189 ceph-mon[36187]: pgmap v2368: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-24T01:18:05.704 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:18:05 smithi189 ceph-mon[36187]: pgmap v2369: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T01:18:05.704 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:18:05 smithi189 ceph-mon[36187]: pgmap v2370: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-03-24T01:18:05.704 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:18:05 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T01:18:05.711 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:18:05 smithi193 ceph-mon[36878]: pgmap v2368: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-24T01:18:05.711 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:18:05 smithi193 ceph-mon[36878]: pgmap v2369: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T01:18:05.711 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:18:05 smithi193 ceph-mon[36878]: pgmap v2370: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-03-24T01:18:05.711 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:18:05 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T01:18:06.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:18:06 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 3181.660114 secs 2024-03-24T01:18:06.704 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:18:06 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 3181.660114 secs 2024-03-24T01:18:06.711 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:18:06 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 3181.660114 secs 2024-03-24T01:18:07.704 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:18:07 smithi189 ceph-mon[36187]: pgmap v2371: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 5 op/s 2024-03-24T01:18:07.711 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:18:07 smithi193 ceph-mon[36878]: pgmap v2371: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 5 op/s 2024-03-24T01:18:07.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:18:07 smithi118 ceph-mon[31535]: pgmap v2371: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 5 op/s 2024-03-24T01:18:09.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:18:09 smithi118 ceph-mon[31535]: pgmap v2372: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 5 op/s 2024-03-24T01:18:09.953 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:18:09 smithi189 ceph-mon[36187]: pgmap v2372: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 5 op/s 2024-03-24T01:18:09.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:18:09 smithi193 ceph-mon[36878]: pgmap v2372: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 5 op/s 2024-03-24T01:18:11.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:18:11 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 3186.660204 secs 2024-03-24T01:18:11.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:18:11 smithi118 ceph-mon[31535]: pgmap v2373: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-03-24T01:18:11.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:18:11 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 3186.660204 secs 2024-03-24T01:18:11.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:18:11 smithi189 ceph-mon[36187]: pgmap v2373: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-03-24T01:18:11.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:18:11 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 3186.660204 secs 2024-03-24T01:18:11.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:18:11 smithi193 ceph-mon[36878]: pgmap v2373: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-03-24T01:18:13.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:18:13 smithi118 ceph-mon[31535]: pgmap v2374: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-03-24T01:18:13.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:18:13 smithi189 ceph-mon[36187]: pgmap v2374: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-03-24T01:18:13.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:18:13 smithi193 ceph-mon[36878]: pgmap v2374: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-03-24T01:18:15.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:18:15 smithi118 ceph-mon[31535]: pgmap v2375: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 5 op/s 2024-03-24T01:18:15.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:18:15 smithi189 ceph-mon[36187]: pgmap v2375: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 5 op/s 2024-03-24T01:18:15.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:18:15 smithi193 ceph-mon[36878]: pgmap v2375: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 5 op/s 2024-03-24T01:18:16.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:18:16 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 3191.660266 secs 2024-03-24T01:18:16.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:18:16 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 3191.660266 secs 2024-03-24T01:18:16.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:18:16 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 3191.660266 secs 2024-03-24T01:18:17.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:18:17 smithi118 ceph-mon[31535]: pgmap v2376: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:18:17.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:18:17 smithi189 ceph-mon[36187]: pgmap v2376: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:18:17.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:18:17 smithi193 ceph-mon[36878]: pgmap v2376: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:18:19.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:18:19 smithi118 ceph-mon[31535]: pgmap v2377: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T01:18:19.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:18:19 smithi189 ceph-mon[36187]: pgmap v2377: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T01:18:19.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:18:19 smithi193 ceph-mon[36878]: pgmap v2377: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T01:18:21.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:18:21 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 3196.660371 secs 2024-03-24T01:18:21.947 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:18:21 smithi118 ceph-mon[31535]: pgmap v2378: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T01:18:21.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:18:21 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 3196.660371 secs 2024-03-24T01:18:21.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:18:21 smithi189 ceph-mon[36187]: pgmap v2378: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T01:18:21.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:18:21 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 3196.660371 secs 2024-03-24T01:18:21.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:18:21 smithi193 ceph-mon[36878]: pgmap v2378: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T01:18:23.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:18:23 smithi118 ceph-mon[31535]: pgmap v2379: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T01:18:23.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:18:23 smithi189 ceph-mon[36187]: pgmap v2379: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T01:18:23.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:18:23 smithi193 ceph-mon[36878]: pgmap v2379: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T01:18:25.756 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:18:25 smithi189 ceph-mon[36187]: pgmap v2380: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-03-24T01:18:25.756 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:18:25 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 3201.660432 secs 2024-03-24T01:18:25.762 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:18:25 smithi193 ceph-mon[36878]: pgmap v2380: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-03-24T01:18:25.762 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:18:25 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 3201.660432 secs 2024-03-24T01:18:25.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:18:25 smithi118 ceph-mon[31535]: pgmap v2380: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-03-24T01:18:25.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:18:25 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 3201.660432 secs 2024-03-24T01:18:27.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:18:27 smithi118 ceph-mon[31535]: pgmap v2381: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:18:27.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:18:27 smithi189 ceph-mon[36187]: pgmap v2381: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:18:27.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:18:27 smithi193 ceph-mon[36878]: pgmap v2381: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:18:29.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:18:29 smithi118 ceph-mon[31535]: pgmap v2382: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T01:18:29.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:18:29 smithi189 ceph-mon[36187]: pgmap v2382: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T01:18:29.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:18:29 smithi193 ceph-mon[36878]: pgmap v2382: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T01:18:30.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:18:30 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 3206.660470 secs 2024-03-24T01:18:30.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:18:30 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 3206.660470 secs 2024-03-24T01:18:30.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:18:30 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 3206.660470 secs 2024-03-24T01:18:31.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:18:31 smithi118 ceph-mon[31535]: pgmap v2383: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T01:18:31.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:18:31 smithi189 ceph-mon[36187]: pgmap v2383: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T01:18:31.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:18:31 smithi193 ceph-mon[36878]: pgmap v2383: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T01:18:33.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:18:33 smithi118 ceph-mon[31535]: pgmap v2384: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T01:18:33.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:18:33 smithi189 ceph-mon[36187]: pgmap v2384: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T01:18:33.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:18:33 smithi193 ceph-mon[36878]: pgmap v2384: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T01:18:35.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:18:35 smithi118 ceph-mon[31535]: pgmap v2385: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-03-24T01:18:35.947 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:18:35 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 3211.660549 secs 2024-03-24T01:18:35.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:18:35 smithi189 ceph-mon[36187]: pgmap v2385: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-03-24T01:18:35.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:18:35 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 3211.660549 secs 2024-03-24T01:18:35.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:18:35 smithi193 ceph-mon[36878]: pgmap v2385: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-03-24T01:18:35.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:18:35 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 3211.660549 secs 2024-03-24T01:18:37.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:18:37 smithi118 ceph-mon[31535]: pgmap v2386: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:18:37.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:18:37 smithi189 ceph-mon[36187]: pgmap v2386: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:18:37.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:18:37 smithi193 ceph-mon[36878]: pgmap v2386: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:18:39.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:18:39 smithi118 ceph-mon[31535]: pgmap v2387: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T01:18:39.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:18:39 smithi189 ceph-mon[36187]: pgmap v2387: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T01:18:39.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:18:39 smithi193 ceph-mon[36878]: pgmap v2387: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T01:18:40.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:18:40 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 3216.660608 secs 2024-03-24T01:18:40.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:18:40 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 3216.660608 secs 2024-03-24T01:18:40.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:18:40 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 3216.660608 secs 2024-03-24T01:18:41.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:18:41 smithi118 ceph-mon[31535]: pgmap v2388: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T01:18:41.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:18:41 smithi189 ceph-mon[36187]: pgmap v2388: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T01:18:41.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:18:41 smithi193 ceph-mon[36878]: pgmap v2388: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T01:18:43.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:18:43 smithi118 ceph-mon[31535]: pgmap v2389: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T01:18:43.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:18:43 smithi189 ceph-mon[36187]: pgmap v2389: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T01:18:43.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:18:43 smithi193 ceph-mon[36878]: pgmap v2389: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T01:18:45.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:18:45 smithi118 ceph-mon[31535]: pgmap v2390: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-03-24T01:18:45.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:18:45 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 3221.660622 secs 2024-03-24T01:18:45.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:18:45 smithi189 ceph-mon[36187]: pgmap v2390: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-03-24T01:18:45.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:18:45 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 3221.660622 secs 2024-03-24T01:18:45.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:18:45 smithi193 ceph-mon[36878]: pgmap v2390: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-03-24T01:18:45.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:18:45 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 3221.660622 secs 2024-03-24T01:18:46.947 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:18:46 smithi118 ceph-mon[31535]: pgmap v2391: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-03-24T01:18:46.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:18:46 smithi189 ceph-mon[36187]: pgmap v2391: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-03-24T01:18:46.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:18:46 smithi193 ceph-mon[36878]: pgmap v2391: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-03-24T01:18:49.880 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:18:49 smithi118 ceph-mon[31535]: pgmap v2392: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-03-24T01:18:49.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:18:49 smithi189 ceph-mon[36187]: pgmap v2392: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-03-24T01:18:49.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:18:49 smithi193 ceph-mon[36878]: pgmap v2392: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-03-24T01:18:50.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:18:50 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 3226.660711 secs 2024-03-24T01:18:50.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:18:50 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 3226.660711 secs 2024-03-24T01:18:50.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:18:50 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 3226.660711 secs 2024-03-24T01:18:51.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:18:51 smithi118 ceph-mon[31535]: pgmap v2393: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 6 op/s 2024-03-24T01:18:51.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:18:51 smithi189 ceph-mon[36187]: pgmap v2393: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 6 op/s 2024-03-24T01:18:51.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:18:51 smithi193 ceph-mon[36878]: pgmap v2393: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 6 op/s 2024-03-24T01:18:53.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:18:53 smithi118 ceph-mon[31535]: pgmap v2394: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 5 op/s 2024-03-24T01:18:53.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:18:53 smithi189 ceph-mon[36187]: pgmap v2394: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 5 op/s 2024-03-24T01:18:53.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:18:53 smithi193 ceph-mon[36878]: pgmap v2394: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 5 op/s 2024-03-24T01:18:55.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:18:55 smithi118 ceph-mon[31535]: pgmap v2395: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-03-24T01:18:55.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:18:55 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 3231.660800 secs 2024-03-24T01:18:55.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:18:55 smithi189 ceph-mon[36187]: pgmap v2395: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-03-24T01:18:55.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:18:55 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 3231.660800 secs 2024-03-24T01:18:55.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:18:55 smithi193 ceph-mon[36878]: pgmap v2395: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-03-24T01:18:55.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:18:55 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 3231.660800 secs 2024-03-24T01:18:57.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:18:57 smithi118 ceph-mon[31535]: pgmap v2396: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:18:57.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:18:57 smithi189 ceph-mon[36187]: pgmap v2396: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:18:57.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:18:57 smithi193 ceph-mon[36878]: pgmap v2396: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:18:58.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:18:58 smithi118 ceph-mon[31535]: pgmap v2397: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T01:18:58.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:18:58 smithi189 ceph-mon[36187]: pgmap v2397: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T01:18:58.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:18:58 smithi193 ceph-mon[36878]: pgmap v2397: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T01:19:00.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:19:00 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 3236.660887 secs 2024-03-24T01:19:00.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:19:00 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 3236.660887 secs 2024-03-24T01:19:00.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:19:00 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 3236.660887 secs 2024-03-24T01:19:01.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:19:01 smithi118 ceph-mon[31535]: pgmap v2398: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T01:19:01.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:19:01 smithi189 ceph-mon[36187]: pgmap v2398: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T01:19:01.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:19:01 smithi193 ceph-mon[36878]: pgmap v2398: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T01:19:03.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:19:03 smithi118 ceph-mon[31535]: pgmap v2399: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T01:19:03.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:19:03 smithi189 ceph-mon[36187]: pgmap v2399: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T01:19:03.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:19:03 smithi193 ceph-mon[36878]: pgmap v2399: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T01:19:05.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:19:05 smithi118 ceph-mon[31535]: pgmap v2400: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-03-24T01:19:05.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:19:05 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T01:19:05.947 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:19:05 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 3241.660952 secs 2024-03-24T01:19:05.947 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:19:05 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T01:19:05.947 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:19:05 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T01:19:05.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:19:05 smithi189 ceph-mon[36187]: pgmap v2400: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-03-24T01:19:05.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:19:05 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T01:19:05.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:19:05 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 3241.660952 secs 2024-03-24T01:19:05.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:19:05 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T01:19:05.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:19:05 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T01:19:05.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:19:05 smithi193 ceph-mon[36878]: pgmap v2400: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-03-24T01:19:05.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:19:05 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T01:19:05.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:19:05 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 3241.660952 secs 2024-03-24T01:19:05.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:19:05 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T01:19:05.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:19:05 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T01:19:06.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:19:06 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T01:19:06.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:19:06 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T01:19:06.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:19:06 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T01:19:06.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:19:06 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T01:19:06.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:19:06 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T01:19:06.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:19:06 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T01:19:06.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:19:06 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T01:19:06.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:19:06 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T01:19:06.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:19:06 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T01:19:07.848 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:19:07 smithi118 ceph-mon[31535]: pgmap v2401: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 5 op/s 2024-03-24T01:19:07.848 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:19:07 smithi118 ceph-mon[31535]: pgmap v2402: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 6 op/s 2024-03-24T01:19:07.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:19:07 smithi189 ceph-mon[36187]: pgmap v2401: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 5 op/s 2024-03-24T01:19:07.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:19:07 smithi189 ceph-mon[36187]: pgmap v2402: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 6 op/s 2024-03-24T01:19:07.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:19:07 smithi193 ceph-mon[36878]: pgmap v2401: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 5 op/s 2024-03-24T01:19:07.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:19:07 smithi193 ceph-mon[36878]: pgmap v2402: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 6 op/s 2024-03-24T01:19:08.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:19:08 smithi118 ceph-mon[31535]: pgmap v2403: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 6 op/s 2024-03-24T01:19:08.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:19:08 smithi189 ceph-mon[36187]: pgmap v2403: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 6 op/s 2024-03-24T01:19:08.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:19:08 smithi193 ceph-mon[36878]: pgmap v2403: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 6 op/s 2024-03-24T01:19:11.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:19:11 smithi118 ceph-mon[31535]: pgmap v2404: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 6 op/s 2024-03-24T01:19:11.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:19:11 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 3246.661041 secs 2024-03-24T01:19:11.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:19:11 smithi189 ceph-mon[36187]: pgmap v2404: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 6 op/s 2024-03-24T01:19:11.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:19:11 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 3246.661041 secs 2024-03-24T01:19:11.460 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:19:11 smithi193 ceph-mon[36878]: pgmap v2404: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 6 op/s 2024-03-24T01:19:11.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:19:11 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 3246.661041 secs 2024-03-24T01:19:13.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:19:13 smithi118 ceph-mon[31535]: pgmap v2405: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 5 op/s 2024-03-24T01:19:13.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:19:13 smithi189 ceph-mon[36187]: pgmap v2405: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 5 op/s 2024-03-24T01:19:13.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:19:13 smithi193 ceph-mon[36878]: pgmap v2405: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 5 op/s 2024-03-24T01:19:15.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:19:15 smithi118 ceph-mon[31535]: pgmap v2406: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 6 op/s 2024-03-24T01:19:15.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:19:15 smithi189 ceph-mon[36187]: pgmap v2406: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 6 op/s 2024-03-24T01:19:15.460 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:19:15 smithi193 ceph-mon[36878]: pgmap v2406: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 6 op/s 2024-03-24T01:19:16.250 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T01:19:16.250+0000 7f81db400640 1 -- 172.21.15.118:0/659686424 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f81c4006290 con 0x5622651a13f0 2024-03-24T01:19:16.250 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T01:19:16.250+0000 7f81ee000640 1 -- 172.21.15.118:0/1693767697 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f81c80065b0 con 0x7f81f81052d0 2024-03-24T01:19:16.251 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T01:19:16.250+0000 7f81e4c00640 1 -- 172.21.15.118:0/659686424 <== mon.1 v2:172.21.15.193:3300/0 29 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 409+0+0 (secure 0 0 0) 0x7f81c4006290 con 0x5622651a13f0 2024-03-24T01:19:16.251 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T01:19:16.250+0000 7f81ef400640 1 -- 172.21.15.118:0/1693767697 <== mon.2 v2:172.21.15.189:3300/0 29 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (secure 0 0 0) 0x7f81c80065b0 con 0x7f81f81052d0 2024-03-24T01:19:16.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:19:16 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 3251.661131 secs 2024-03-24T01:19:16.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:19:16 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 3251.661131 secs 2024-03-24T01:19:16.460 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:19:16 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 3251.661131 secs 2024-03-24T01:19:17.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:19:17 smithi118 ceph-mon[31535]: pgmap v2407: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 5 op/s 2024-03-24T01:19:17.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:19:17 smithi189 ceph-mon[36187]: pgmap v2407: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 5 op/s 2024-03-24T01:19:17.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:19:17 smithi193 ceph-mon[36878]: pgmap v2407: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 5 op/s 2024-03-24T01:19:19.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:19:19 smithi118 ceph-mon[31535]: pgmap v2408: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T01:19:19.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:19:19 smithi189 ceph-mon[36187]: pgmap v2408: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T01:19:19.460 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:19:19 smithi193 ceph-mon[36878]: pgmap v2408: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T01:19:21.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:19:21 smithi118 ceph-mon[31535]: pgmap v2409: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T01:19:21.447 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:19:21 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 3256.661218 secs 2024-03-24T01:19:21.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:19:21 smithi189 ceph-mon[36187]: pgmap v2409: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T01:19:21.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:19:21 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 3256.661218 secs 2024-03-24T01:19:21.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:19:21 smithi193 ceph-mon[36878]: pgmap v2409: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T01:19:21.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:19:21 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 3256.661218 secs 2024-03-24T01:19:23.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:19:23 smithi118 ceph-mon[31535]: pgmap v2410: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:19:23.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:19:23 smithi189 ceph-mon[36187]: pgmap v2410: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:19:23.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:19:23 smithi193 ceph-mon[36878]: pgmap v2410: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:19:25.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:19:25 smithi118 ceph-mon[31535]: pgmap v2411: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-24T01:19:25.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:19:25 smithi189 ceph-mon[36187]: pgmap v2411: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-24T01:19:25.460 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:19:25 smithi193 ceph-mon[36878]: pgmap v2411: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-24T01:19:26.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:19:26 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 3261.661304 secs 2024-03-24T01:19:26.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:19:26 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 3261.661304 secs 2024-03-24T01:19:26.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:19:26 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 3261.661304 secs 2024-03-24T01:19:27.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:19:27 smithi118 ceph-mon[31535]: pgmap v2412: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:19:27.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:19:27 smithi189 ceph-mon[36187]: pgmap v2412: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:19:27.460 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:19:27 smithi193 ceph-mon[36878]: pgmap v2412: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:19:29.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:19:29 smithi118 ceph-mon[31535]: pgmap v2413: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T01:19:29.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:19:29 smithi189 ceph-mon[36187]: pgmap v2413: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T01:19:29.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:19:29 smithi193 ceph-mon[36878]: pgmap v2413: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T01:19:31.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:19:31 smithi118 ceph-mon[31535]: pgmap v2414: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T01:19:31.447 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:19:31 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 3266.661398 secs 2024-03-24T01:19:31.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:19:31 smithi189 ceph-mon[36187]: pgmap v2414: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T01:19:31.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:19:31 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 3266.661398 secs 2024-03-24T01:19:31.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:19:31 smithi193 ceph-mon[36878]: pgmap v2414: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T01:19:31.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:19:31 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 3266.661398 secs 2024-03-24T01:19:33.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:19:33 smithi118 ceph-mon[31535]: pgmap v2415: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:19:33.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:19:33 smithi189 ceph-mon[36187]: pgmap v2415: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:19:33.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:19:33 smithi193 ceph-mon[36878]: pgmap v2415: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:19:35.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:19:35 smithi118 ceph-mon[31535]: pgmap v2416: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-24T01:19:35.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:19:35 smithi189 ceph-mon[36187]: pgmap v2416: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-24T01:19:35.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:19:35 smithi193 ceph-mon[36878]: pgmap v2416: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-24T01:19:36.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:19:36 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 3271.661488 secs 2024-03-24T01:19:36.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:19:36 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 3271.661488 secs 2024-03-24T01:19:36.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:19:36 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 3271.661488 secs 2024-03-24T01:19:37.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:19:37 smithi118 ceph-mon[31535]: pgmap v2417: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:19:37.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:19:37 smithi193 ceph-mon[36878]: pgmap v2417: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:19:37.704 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:19:37 smithi189 ceph-mon[36187]: pgmap v2417: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:19:39.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:19:39 smithi193 ceph-mon[36878]: pgmap v2418: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T01:19:39.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:19:39 smithi118 ceph-mon[31535]: pgmap v2418: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T01:19:39.704 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:19:39 smithi189 ceph-mon[36187]: pgmap v2418: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T01:19:41.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:19:41 smithi118 ceph-mon[31535]: pgmap v2419: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T01:19:41.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:19:41 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 3276.661573 secs 2024-03-24T01:19:41.704 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:19:41 smithi189 ceph-mon[36187]: pgmap v2419: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T01:19:41.704 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:19:41 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 3276.661573 secs 2024-03-24T01:19:41.711 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:19:41 smithi193 ceph-mon[36878]: pgmap v2419: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T01:19:41.711 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:19:41 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 3276.661573 secs 2024-03-24T01:19:43.697 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:19:43 smithi118 ceph-mon[31535]: pgmap v2420: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:19:43.704 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:19:43 smithi189 ceph-mon[36187]: pgmap v2420: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:19:43.710 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:19:43 smithi193 ceph-mon[36878]: pgmap v2420: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:19:45.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:19:45 smithi118 ceph-mon[31535]: pgmap v2421: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-24T01:19:45.704 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:19:45 smithi189 ceph-mon[36187]: pgmap v2421: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-24T01:19:45.710 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:19:45 smithi193 ceph-mon[36878]: pgmap v2421: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-24T01:19:46.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:19:46 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 3281.661613 secs 2024-03-24T01:19:46.704 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:19:46 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 3281.661613 secs 2024-03-24T01:19:46.711 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:19:46 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 3281.661613 secs 2024-03-24T01:19:47.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:19:47 smithi118 ceph-mon[31535]: pgmap v2422: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:19:47.704 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:19:47 smithi189 ceph-mon[36187]: pgmap v2422: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:19:47.711 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:19:47 smithi193 ceph-mon[36878]: pgmap v2422: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:19:49.499 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:19:49 smithi118 ceph-mon[31535]: pgmap v2423: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T01:19:49.704 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:19:49 smithi189 ceph-mon[36187]: pgmap v2423: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T01:19:49.711 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:19:49 smithi193 ceph-mon[36878]: pgmap v2423: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T01:19:51.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:19:51 smithi118 ceph-mon[31535]: pgmap v2424: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T01:19:51.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:19:51 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 3286.661703 secs 2024-03-24T01:19:51.704 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:19:51 smithi189 ceph-mon[36187]: pgmap v2424: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T01:19:51.704 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:19:51 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 3286.661703 secs 2024-03-24T01:19:51.711 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:19:51 smithi193 ceph-mon[36878]: pgmap v2424: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T01:19:51.711 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:19:51 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 3286.661703 secs 2024-03-24T01:19:53.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:19:53 smithi118 ceph-mon[31535]: pgmap v2425: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:19:53.704 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:19:53 smithi189 ceph-mon[36187]: pgmap v2425: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:19:53.713 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:19:53 smithi193 ceph-mon[36878]: pgmap v2425: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:19:55.506 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:19:55 smithi189 ceph-mon[36187]: pgmap v2426: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-24T01:19:55.513 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:19:55 smithi193 ceph-mon[36878]: pgmap v2426: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-24T01:19:55.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:19:55 smithi118 ceph-mon[31535]: pgmap v2426: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-24T01:19:56.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:19:56 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 3291.661791 secs 2024-03-24T01:19:56.704 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:19:56 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 3291.661791 secs 2024-03-24T01:19:56.710 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:19:56 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 3291.661791 secs 2024-03-24T01:19:57.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:19:57 smithi118 ceph-mon[31535]: pgmap v2427: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:19:57.704 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:19:57 smithi189 ceph-mon[36187]: pgmap v2427: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:19:57.711 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:19:57 smithi193 ceph-mon[36878]: pgmap v2427: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:19:59.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:19:59 smithi118 ceph-mon[31535]: pgmap v2428: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T01:19:59.704 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:19:59 smithi189 ceph-mon[36187]: pgmap v2428: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T01:19:59.710 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:19:59 smithi193 ceph-mon[36878]: pgmap v2428: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T01:20:00.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:20:00 smithi118 ceph-mon[31535]: overall HEALTH_WARN 6 failed cephadm daemon(s); 1 MDSs report slow requests 2024-03-24T01:20:00.704 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:20:00 smithi189 ceph-mon[36187]: overall HEALTH_WARN 6 failed cephadm daemon(s); 1 MDSs report slow requests 2024-03-24T01:20:00.711 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:20:00 smithi193 ceph-mon[36878]: overall HEALTH_WARN 6 failed cephadm daemon(s); 1 MDSs report slow requests 2024-03-24T01:20:01.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:20:01 smithi118 ceph-mon[31535]: pgmap v2429: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T01:20:01.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:20:01 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 3296.661873 secs 2024-03-24T01:20:01.704 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:20:01 smithi189 ceph-mon[36187]: pgmap v2429: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T01:20:01.704 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:20:01 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 3296.661873 secs 2024-03-24T01:20:01.710 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:20:01 smithi193 ceph-mon[36878]: pgmap v2429: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T01:20:01.711 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:20:01 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 3296.661873 secs 2024-03-24T01:20:03.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:20:03 smithi118 ceph-mon[31535]: pgmap v2430: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:20:03.704 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:20:03 smithi189 ceph-mon[36187]: pgmap v2430: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:20:03.710 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:20:03 smithi193 ceph-mon[36878]: pgmap v2430: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:20:05.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:20:05 smithi118 ceph-mon[31535]: pgmap v2431: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-24T01:20:05.947 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:20:05 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 3301.661960 secs 2024-03-24T01:20:05.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:20:05 smithi189 ceph-mon[36187]: pgmap v2431: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-24T01:20:05.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:20:05 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 3301.661960 secs 2024-03-24T01:20:05.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:20:05 smithi193 ceph-mon[36878]: pgmap v2431: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-24T01:20:05.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:20:05 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 3301.661960 secs 2024-03-24T01:20:06.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:20:06 smithi118 ceph-mon[31535]: pgmap v2432: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:20:06.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:20:06 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T01:20:06.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:20:06 smithi189 ceph-mon[36187]: pgmap v2432: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:20:06.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:20:06 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T01:20:06.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:20:06 smithi193 ceph-mon[36878]: pgmap v2432: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:20:06.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:20:06 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T01:20:07.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:20:07 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T01:20:07.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:20:07 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T01:20:07.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:20:07 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T01:20:07.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:20:07 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T01:20:07.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:20:07 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T01:20:07.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:20:07 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T01:20:07.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:20:07 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T01:20:07.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:20:07 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T01:20:07.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:20:07 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T01:20:08.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:20:08 smithi189 ceph-mon[36187]: pgmap v2433: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 5 op/s 2024-03-24T01:20:08.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:20:08 smithi189 ceph-mon[36187]: pgmap v2434: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 5 op/s 2024-03-24T01:20:08.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:20:08 smithi193 ceph-mon[36878]: pgmap v2433: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 5 op/s 2024-03-24T01:20:08.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:20:08 smithi193 ceph-mon[36878]: pgmap v2434: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 5 op/s 2024-03-24T01:20:09.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:20:08 smithi118 ceph-mon[31535]: pgmap v2433: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 5 op/s 2024-03-24T01:20:09.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:20:08 smithi118 ceph-mon[31535]: pgmap v2434: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 5 op/s 2024-03-24T01:20:10.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:20:10 smithi189 ceph-mon[36187]: pgmap v2435: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 6 op/s 2024-03-24T01:20:10.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:20:10 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 3306.662021 secs 2024-03-24T01:20:10.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:20:10 smithi193 ceph-mon[36878]: pgmap v2435: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 6 op/s 2024-03-24T01:20:10.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:20:10 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 3306.662021 secs 2024-03-24T01:20:11.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:20:10 smithi118 ceph-mon[31535]: pgmap v2435: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 6 op/s 2024-03-24T01:20:11.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:20:10 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 3306.662021 secs 2024-03-24T01:20:12.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:20:12 smithi193 ceph-mon[36878]: pgmap v2436: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 6 op/s 2024-03-24T01:20:13.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:20:12 smithi118 ceph-mon[31535]: pgmap v2436: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 6 op/s 2024-03-24T01:20:13.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:20:12 smithi189 ceph-mon[36187]: pgmap v2436: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 6 op/s 2024-03-24T01:20:14.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:20:14 smithi193 ceph-mon[36878]: pgmap v2437: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 4 op/s 2024-03-24T01:20:15.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:20:14 smithi118 ceph-mon[31535]: pgmap v2437: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 4 op/s 2024-03-24T01:20:15.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:20:14 smithi189 ceph-mon[36187]: pgmap v2437: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 4 op/s 2024-03-24T01:20:16.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:20:15 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 3311.662110 secs 2024-03-24T01:20:16.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:20:15 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 3311.662110 secs 2024-03-24T01:20:16.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:20:15 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 3311.662110 secs 2024-03-24T01:20:17.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:20:16 smithi118 ceph-mon[31535]: pgmap v2438: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 6 op/s 2024-03-24T01:20:17.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:20:16 smithi189 ceph-mon[36187]: pgmap v2438: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 6 op/s 2024-03-24T01:20:17.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:20:16 smithi193 ceph-mon[36878]: pgmap v2438: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 6 op/s 2024-03-24T01:20:19.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:20:18 smithi118 ceph-mon[31535]: pgmap v2439: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 5 op/s 2024-03-24T01:20:19.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:20:18 smithi189 ceph-mon[36187]: pgmap v2439: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 5 op/s 2024-03-24T01:20:19.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:20:18 smithi193 ceph-mon[36878]: pgmap v2439: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 5 op/s 2024-03-24T01:20:21.016 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:20:20 smithi118 ceph-mon[31535]: pgmap v2440: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-03-24T01:20:21.016 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:20:20 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 3316.662199 secs 2024-03-24T01:20:21.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:20:20 smithi189 ceph-mon[36187]: pgmap v2440: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-03-24T01:20:21.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:20:20 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 3316.662199 secs 2024-03-24T01:20:21.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:20:20 smithi193 ceph-mon[36878]: pgmap v2440: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-03-24T01:20:21.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:20:20 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 3316.662199 secs 2024-03-24T01:20:23.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:20:22 smithi118 ceph-mon[31535]: pgmap v2441: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:20:23.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:20:22 smithi189 ceph-mon[36187]: pgmap v2441: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:20:23.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:20:22 smithi193 ceph-mon[36878]: pgmap v2441: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:20:25.005 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:20:24 smithi189 ceph-mon[36187]: pgmap v2442: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T01:20:25.013 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:20:24 smithi193 ceph-mon[36878]: pgmap v2442: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T01:20:25.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:20:24 smithi118 ceph-mon[31535]: pgmap v2442: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T01:20:26.022 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:20:25 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 3321.662288 secs 2024-03-24T01:20:26.025 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:20:25 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 3321.662288 secs 2024-03-24T01:20:26.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:20:25 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 3321.662288 secs 2024-03-24T01:20:27.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:20:26 smithi118 ceph-mon[31535]: pgmap v2443: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T01:20:27.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:20:26 smithi189 ceph-mon[36187]: pgmap v2443: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T01:20:27.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:20:26 smithi193 ceph-mon[36878]: pgmap v2443: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T01:20:29.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:20:28 smithi118 ceph-mon[31535]: pgmap v2444: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T01:20:29.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:20:28 smithi189 ceph-mon[36187]: pgmap v2444: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T01:20:29.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:20:28 smithi193 ceph-mon[36878]: pgmap v2444: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T01:20:31.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:20:30 smithi118 ceph-mon[31535]: pgmap v2445: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-03-24T01:20:31.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:20:30 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 3326.662365 secs 2024-03-24T01:20:31.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:20:30 smithi189 ceph-mon[36187]: pgmap v2445: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-03-24T01:20:31.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:20:30 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 3326.662365 secs 2024-03-24T01:20:31.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:20:30 smithi193 ceph-mon[36878]: pgmap v2445: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-03-24T01:20:31.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:20:30 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 3326.662365 secs 2024-03-24T01:20:33.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:20:32 smithi118 ceph-mon[31535]: pgmap v2446: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:20:33.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:20:32 smithi189 ceph-mon[36187]: pgmap v2446: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:20:33.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:20:32 smithi193 ceph-mon[36878]: pgmap v2446: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:20:35.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:20:34 smithi118 ceph-mon[31535]: pgmap v2447: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T01:20:35.203 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:20:34 smithi189 ceph-mon[36187]: pgmap v2447: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T01:20:35.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:20:34 smithi193 ceph-mon[36878]: pgmap v2447: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T01:20:36.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:20:35 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 3331.662452 secs 2024-03-24T01:20:36.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:20:35 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 3331.662452 secs 2024-03-24T01:20:36.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:20:35 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 3331.662452 secs 2024-03-24T01:20:37.098 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:20:36 smithi118 ceph-mon[31535]: pgmap v2448: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T01:20:37.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:20:36 smithi189 ceph-mon[36187]: pgmap v2448: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T01:20:37.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:20:36 smithi193 ceph-mon[36878]: pgmap v2448: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T01:20:39.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:20:38 smithi118 ceph-mon[31535]: pgmap v2449: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T01:20:39.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:20:38 smithi189 ceph-mon[36187]: pgmap v2449: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T01:20:39.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:20:38 smithi193 ceph-mon[36878]: pgmap v2449: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T01:20:41.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:20:40 smithi118 ceph-mon[31535]: pgmap v2450: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-03-24T01:20:41.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:20:40 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 3336.662539 secs 2024-03-24T01:20:41.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:20:40 smithi189 ceph-mon[36187]: pgmap v2450: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-03-24T01:20:41.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:20:40 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 3336.662539 secs 2024-03-24T01:20:41.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:20:40 smithi193 ceph-mon[36878]: pgmap v2450: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-03-24T01:20:41.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:20:40 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 3336.662539 secs 2024-03-24T01:20:43.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:20:42 smithi118 ceph-mon[31535]: pgmap v2451: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:20:43.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:20:42 smithi189 ceph-mon[36187]: pgmap v2451: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:20:43.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:20:42 smithi193 ceph-mon[36878]: pgmap v2451: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:20:45.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:20:44 smithi118 ceph-mon[31535]: pgmap v2452: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T01:20:45.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:20:44 smithi189 ceph-mon[36187]: pgmap v2452: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T01:20:45.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:20:44 smithi193 ceph-mon[36878]: pgmap v2452: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T01:20:46.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:20:45 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 3341.662630 secs 2024-03-24T01:20:46.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:20:45 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 3341.662630 secs 2024-03-24T01:20:46.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:20:45 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 3341.662630 secs 2024-03-24T01:20:47.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:20:46 smithi118 ceph-mon[31535]: pgmap v2453: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T01:20:47.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:20:46 smithi189 ceph-mon[36187]: pgmap v2453: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T01:20:47.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:20:46 smithi193 ceph-mon[36878]: pgmap v2453: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T01:20:49.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:20:48 smithi118 ceph-mon[31535]: pgmap v2454: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T01:20:49.203 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:20:48 smithi189 ceph-mon[36187]: pgmap v2454: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T01:20:49.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:20:48 smithi193 ceph-mon[36878]: pgmap v2454: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T01:20:51.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:20:50 smithi118 ceph-mon[31535]: pgmap v2455: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T01:20:51.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:20:50 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 3346.662689 secs 2024-03-24T01:20:51.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:20:50 smithi189 ceph-mon[36187]: pgmap v2455: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T01:20:51.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:20:50 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 3346.662689 secs 2024-03-24T01:20:51.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:20:50 smithi193 ceph-mon[36878]: pgmap v2455: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T01:20:51.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:20:50 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 3346.662689 secs 2024-03-24T01:20:53.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:20:52 smithi118 ceph-mon[31535]: pgmap v2456: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:20:53.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:20:52 smithi189 ceph-mon[36187]: pgmap v2456: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:20:53.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:20:52 smithi193 ceph-mon[36878]: pgmap v2456: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:20:55.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:20:54 smithi118 ceph-mon[31535]: pgmap v2457: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T01:20:55.203 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:20:54 smithi189 ceph-mon[36187]: pgmap v2457: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T01:20:55.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:20:54 smithi193 ceph-mon[36878]: pgmap v2457: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T01:20:56.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:20:55 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 3351.662776 secs 2024-03-24T01:20:56.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:20:55 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 3351.662776 secs 2024-03-24T01:20:56.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:20:55 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 3351.662776 secs 2024-03-24T01:20:57.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:20:56 smithi118 ceph-mon[31535]: pgmap v2458: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T01:20:57.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:20:56 smithi189 ceph-mon[36187]: pgmap v2458: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T01:20:57.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:20:56 smithi193 ceph-mon[36878]: pgmap v2458: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T01:20:59.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:20:58 smithi118 ceph-mon[31535]: pgmap v2459: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T01:20:59.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:20:58 smithi189 ceph-mon[36187]: pgmap v2459: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T01:20:59.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:20:58 smithi193 ceph-mon[36878]: pgmap v2459: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T01:21:01.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:21:00 smithi118 ceph-mon[31535]: pgmap v2460: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-03-24T01:21:01.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:21:00 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 3356.662864 secs 2024-03-24T01:21:01.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:21:00 smithi189 ceph-mon[36187]: pgmap v2460: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-03-24T01:21:01.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:21:00 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 3356.662864 secs 2024-03-24T01:21:01.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:21:00 smithi193 ceph-mon[36878]: pgmap v2460: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-03-24T01:21:01.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:21:00 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 3356.662864 secs 2024-03-24T01:21:03.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:21:02 smithi118 ceph-mon[31535]: pgmap v2461: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:21:03.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:21:02 smithi189 ceph-mon[36187]: pgmap v2461: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:21:03.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:21:02 smithi193 ceph-mon[36878]: pgmap v2461: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:21:04.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:21:04 smithi118 ceph-mon[31535]: pgmap v2462: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T01:21:04.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:21:04 smithi189 ceph-mon[36187]: pgmap v2462: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T01:21:04.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:21:04 smithi193 ceph-mon[36878]: pgmap v2462: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T01:21:05.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:21:05 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 3361.662954 secs 2024-03-24T01:21:05.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:21:05 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 3361.662954 secs 2024-03-24T01:21:05.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:21:05 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 3361.662954 secs 2024-03-24T01:21:06.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:21:06 smithi118 ceph-mon[31535]: pgmap v2463: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T01:21:06.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:21:06 smithi189 ceph-mon[36187]: pgmap v2463: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T01:21:06.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:21:06 smithi193 ceph-mon[36878]: pgmap v2463: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T01:21:09.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:21:08 smithi118 ceph-mon[31535]: pgmap v2464: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T01:21:09.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:21:08 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T01:21:09.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:21:08 smithi189 ceph-mon[36187]: pgmap v2464: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T01:21:09.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:21:08 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T01:21:09.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:21:08 smithi193 ceph-mon[36878]: pgmap v2464: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T01:21:09.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:21:08 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T01:21:10.169 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:21:09 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T01:21:10.169 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:21:09 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T01:21:10.169 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:21:09 smithi118 ceph-mon[31535]: pgmap v2465: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 7 op/s 2024-03-24T01:21:10.169 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:21:09 smithi118 ceph-mon[31535]: pgmap v2466: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 6 op/s 2024-03-24T01:21:10.169 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:21:09 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T01:21:10.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:21:09 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T01:21:10.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:21:09 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T01:21:10.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:21:09 smithi189 ceph-mon[36187]: pgmap v2465: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 7 op/s 2024-03-24T01:21:10.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:21:09 smithi189 ceph-mon[36187]: pgmap v2466: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 6 op/s 2024-03-24T01:21:10.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:21:09 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T01:21:10.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:21:09 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T01:21:10.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:21:09 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T01:21:10.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:21:09 smithi193 ceph-mon[36878]: pgmap v2465: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 7 op/s 2024-03-24T01:21:10.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:21:09 smithi193 ceph-mon[36878]: pgmap v2466: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 6 op/s 2024-03-24T01:21:10.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:21:09 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T01:21:11.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:21:10 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 3366.663038 secs 2024-03-24T01:21:11.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:21:10 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 3366.663038 secs 2024-03-24T01:21:11.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:21:10 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 3366.663038 secs 2024-03-24T01:21:12.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:21:11 smithi118 ceph-mon[31535]: pgmap v2467: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 6 op/s 2024-03-24T01:21:12.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:21:11 smithi189 ceph-mon[36187]: pgmap v2467: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 6 op/s 2024-03-24T01:21:12.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:21:11 smithi193 ceph-mon[36878]: pgmap v2467: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 6 op/s 2024-03-24T01:21:14.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:21:14 smithi118 ceph-mon[31535]: pgmap v2468: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 6 op/s 2024-03-24T01:21:14.704 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:21:14 smithi189 ceph-mon[36187]: pgmap v2468: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 6 op/s 2024-03-24T01:21:14.711 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:21:14 smithi193 ceph-mon[36878]: pgmap v2468: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 6 op/s 2024-03-24T01:21:16.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:21:16 smithi118 ceph-mon[31535]: pgmap v2469: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 6 op/s 2024-03-24T01:21:16.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:21:16 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 3371.663099 secs 2024-03-24T01:21:16.704 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:21:16 smithi189 ceph-mon[36187]: pgmap v2469: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 6 op/s 2024-03-24T01:21:16.704 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:21:16 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 3371.663099 secs 2024-03-24T01:21:16.711 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:21:16 smithi193 ceph-mon[36878]: pgmap v2469: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 6 op/s 2024-03-24T01:21:16.711 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:21:16 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 3371.663099 secs 2024-03-24T01:21:18.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:21:18 smithi118 ceph-mon[31535]: pgmap v2470: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 5 op/s 2024-03-24T01:21:18.704 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:21:18 smithi189 ceph-mon[36187]: pgmap v2470: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 5 op/s 2024-03-24T01:21:18.711 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:21:18 smithi193 ceph-mon[36878]: pgmap v2470: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 5 op/s 2024-03-24T01:21:20.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:21:20 smithi118 ceph-mon[31535]: pgmap v2471: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 5 op/s 2024-03-24T01:21:20.704 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:21:20 smithi189 ceph-mon[36187]: pgmap v2471: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 5 op/s 2024-03-24T01:21:20.711 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:21:20 smithi193 ceph-mon[36878]: pgmap v2471: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 5 op/s 2024-03-24T01:21:21.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:21:21 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 3376.663187 secs 2024-03-24T01:21:21.704 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:21:21 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 3376.663187 secs 2024-03-24T01:21:21.711 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:21:21 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 3376.663187 secs 2024-03-24T01:21:22.697 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:21:22 smithi118 ceph-mon[31535]: pgmap v2472: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:21:22.704 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:21:22 smithi189 ceph-mon[36187]: pgmap v2472: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:21:22.711 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:21:22 smithi193 ceph-mon[36878]: pgmap v2472: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:21:24.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:21:24 smithi118 ceph-mon[31535]: pgmap v2473: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T01:21:24.704 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:21:24 smithi189 ceph-mon[36187]: pgmap v2473: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T01:21:24.711 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:21:24 smithi193 ceph-mon[36878]: pgmap v2473: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T01:21:26.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:21:26 smithi118 ceph-mon[31535]: pgmap v2474: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T01:21:26.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:21:26 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 3381.663222 secs 2024-03-24T01:21:26.704 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:21:26 smithi189 ceph-mon[36187]: pgmap v2474: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T01:21:26.704 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:21:26 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 3381.663222 secs 2024-03-24T01:21:26.711 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:21:26 smithi193 ceph-mon[36878]: pgmap v2474: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T01:21:26.711 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:21:26 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 3381.663222 secs 2024-03-24T01:21:28.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:21:28 smithi118 ceph-mon[31535]: pgmap v2475: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:21:28.704 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:21:28 smithi189 ceph-mon[36187]: pgmap v2475: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:21:28.710 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:21:28 smithi193 ceph-mon[36878]: pgmap v2475: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:21:30.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:21:30 smithi118 ceph-mon[31535]: pgmap v2476: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-24T01:21:30.704 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:21:30 smithi189 ceph-mon[36187]: pgmap v2476: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-24T01:21:30.710 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:21:30 smithi193 ceph-mon[36878]: pgmap v2476: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-24T01:21:31.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:21:31 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 3386.663285 secs 2024-03-24T01:21:31.704 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:21:31 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 3386.663285 secs 2024-03-24T01:21:31.711 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:21:31 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 3386.663285 secs 2024-03-24T01:21:32.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:21:32 smithi118 ceph-mon[31535]: pgmap v2477: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:21:32.704 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:21:32 smithi189 ceph-mon[36187]: pgmap v2477: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:21:32.711 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:21:32 smithi193 ceph-mon[36878]: pgmap v2477: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:21:34.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:21:34 smithi118 ceph-mon[31535]: pgmap v2478: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T01:21:34.704 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:21:34 smithi189 ceph-mon[36187]: pgmap v2478: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T01:21:34.711 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:21:34 smithi193 ceph-mon[36878]: pgmap v2478: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T01:21:36.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:21:36 smithi118 ceph-mon[31535]: pgmap v2479: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T01:21:36.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:21:36 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 3391.663344 secs 2024-03-24T01:21:36.704 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:21:36 smithi189 ceph-mon[36187]: pgmap v2479: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T01:21:36.704 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:21:36 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 3391.663344 secs 2024-03-24T01:21:36.711 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:21:36 smithi193 ceph-mon[36878]: pgmap v2479: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T01:21:36.711 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:21:36 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 3391.663344 secs 2024-03-24T01:21:38.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:21:38 smithi118 ceph-mon[31535]: pgmap v2480: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:21:38.704 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:21:38 smithi189 ceph-mon[36187]: pgmap v2480: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:21:38.710 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:21:38 smithi193 ceph-mon[36878]: pgmap v2480: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:21:40.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:21:40 smithi118 ceph-mon[31535]: pgmap v2481: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-24T01:21:40.703 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:21:40 smithi189 ceph-mon[36187]: pgmap v2481: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-24T01:21:40.710 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:21:40 smithi193 ceph-mon[36878]: pgmap v2481: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-24T01:21:41.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:21:41 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 3396.663431 secs 2024-03-24T01:21:41.704 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:21:41 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 3396.663431 secs 2024-03-24T01:21:41.710 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:21:41 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 3396.663431 secs 2024-03-24T01:21:42.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:21:42 smithi118 ceph-mon[31535]: pgmap v2482: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:21:42.704 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:21:42 smithi189 ceph-mon[36187]: pgmap v2482: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:21:42.710 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:21:42 smithi193 ceph-mon[36878]: pgmap v2482: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:21:44.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:21:44 smithi118 ceph-mon[31535]: pgmap v2483: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T01:21:44.704 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:21:44 smithi189 ceph-mon[36187]: pgmap v2483: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T01:21:44.710 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:21:44 smithi193 ceph-mon[36878]: pgmap v2483: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T01:21:46.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:21:46 smithi118 ceph-mon[31535]: pgmap v2484: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T01:21:46.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:21:46 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 3401.663520 secs 2024-03-24T01:21:46.704 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:21:46 smithi189 ceph-mon[36187]: pgmap v2484: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T01:21:46.704 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:21:46 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 3401.663520 secs 2024-03-24T01:21:46.711 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:21:46 smithi193 ceph-mon[36878]: pgmap v2484: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T01:21:46.711 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:21:46 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 3401.663520 secs 2024-03-24T01:21:48.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:21:48 smithi118 ceph-mon[31535]: pgmap v2485: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:21:48.703 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:21:48 smithi189 ceph-mon[36187]: pgmap v2485: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:21:48.711 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:21:48 smithi193 ceph-mon[36878]: pgmap v2485: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:21:50.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:21:50 smithi118 ceph-mon[31535]: pgmap v2486: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-24T01:21:50.704 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:21:50 smithi189 ceph-mon[36187]: pgmap v2486: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-24T01:21:50.711 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:21:50 smithi193 ceph-mon[36878]: pgmap v2486: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-24T01:21:51.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:21:51 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 3406.663610 secs 2024-03-24T01:21:51.704 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:21:51 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 3406.663610 secs 2024-03-24T01:21:51.710 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:21:51 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 3406.663610 secs 2024-03-24T01:21:52.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:21:52 smithi118 ceph-mon[31535]: pgmap v2487: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:21:52.704 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:21:52 smithi189 ceph-mon[36187]: pgmap v2487: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:21:52.710 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:21:52 smithi193 ceph-mon[36878]: pgmap v2487: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:21:54.499 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:21:54 smithi118 ceph-mon[31535]: pgmap v2488: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T01:21:54.704 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:21:54 smithi189 ceph-mon[36187]: pgmap v2488: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T01:21:54.715 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:21:54 smithi193 ceph-mon[36878]: pgmap v2488: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T01:21:56.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:21:56 smithi118 ceph-mon[31535]: pgmap v2489: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T01:21:56.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:21:56 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 3411.663687 secs 2024-03-24T01:21:56.704 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:21:56 smithi189 ceph-mon[36187]: pgmap v2489: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T01:21:56.704 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:21:56 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 3411.663687 secs 2024-03-24T01:21:56.710 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:21:56 smithi193 ceph-mon[36878]: pgmap v2489: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T01:21:56.711 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:21:56 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 3411.663687 secs 2024-03-24T01:21:58.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:21:58 smithi118 ceph-mon[31535]: pgmap v2490: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:21:58.704 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:21:58 smithi189 ceph-mon[36187]: pgmap v2490: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:21:58.710 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:21:58 smithi193 ceph-mon[36878]: pgmap v2490: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:22:00.697 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:22:00 smithi118 ceph-mon[31535]: pgmap v2491: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-24T01:22:00.704 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:22:00 smithi189 ceph-mon[36187]: pgmap v2491: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-24T01:22:00.710 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:22:00 smithi193 ceph-mon[36878]: pgmap v2491: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-24T01:22:01.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:22:01 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 3416.663726 secs 2024-03-24T01:22:01.704 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:22:01 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 3416.663726 secs 2024-03-24T01:22:01.710 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:22:01 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 3416.663726 secs 2024-03-24T01:22:02.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:22:02 smithi118 ceph-mon[31535]: pgmap v2492: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:22:02.704 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:22:02 smithi189 ceph-mon[36187]: pgmap v2492: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:22:02.710 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:22:02 smithi193 ceph-mon[36878]: pgmap v2492: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:22:04.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:22:04 smithi118 ceph-mon[31535]: pgmap v2493: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T01:22:04.704 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:22:04 smithi189 ceph-mon[36187]: pgmap v2493: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T01:22:04.710 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:22:04 smithi193 ceph-mon[36878]: pgmap v2493: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T01:22:06.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:22:06 smithi118 ceph-mon[31535]: pgmap v2494: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T01:22:06.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:22:06 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 3421.663812 secs 2024-03-24T01:22:06.704 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:22:06 smithi189 ceph-mon[36187]: pgmap v2494: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T01:22:06.704 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:22:06 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 3421.663812 secs 2024-03-24T01:22:06.711 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:22:06 smithi193 ceph-mon[36878]: pgmap v2494: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T01:22:06.711 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:22:06 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 3421.663812 secs 2024-03-24T01:22:08.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:22:08 smithi118 ceph-mon[31535]: pgmap v2495: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:22:08.704 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:22:08 smithi189 ceph-mon[36187]: pgmap v2495: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:22:08.710 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:22:08 smithi193 ceph-mon[36878]: pgmap v2495: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:22:10.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:22:10 smithi118 ceph-mon[31535]: pgmap v2496: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-24T01:22:10.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:22:10 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T01:22:10.704 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:22:10 smithi189 ceph-mon[36187]: pgmap v2496: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-24T01:22:10.704 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:22:10 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T01:22:10.711 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:22:10 smithi193 ceph-mon[36878]: pgmap v2496: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-24T01:22:10.711 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:22:10 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T01:22:11.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:22:11 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 3426.663879 secs 2024-03-24T01:22:11.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:22:11 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T01:22:11.697 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:22:11 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T01:22:11.697 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:22:11 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T01:22:11.704 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:22:11 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 3426.663879 secs 2024-03-24T01:22:11.704 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:22:11 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T01:22:11.704 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:22:11 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T01:22:11.704 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:22:11 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T01:22:11.711 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:22:11 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 3426.663879 secs 2024-03-24T01:22:11.711 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:22:11 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T01:22:11.711 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:22:11 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T01:22:11.711 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:22:11 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T01:22:12.704 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:22:12 smithi189 ceph-mon[36187]: pgmap v2497: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 5 op/s 2024-03-24T01:22:12.704 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:22:12 smithi189 ceph-mon[36187]: pgmap v2498: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-03-24T01:22:12.711 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:22:12 smithi193 ceph-mon[36878]: pgmap v2497: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 5 op/s 2024-03-24T01:22:12.711 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:22:12 smithi193 ceph-mon[36878]: pgmap v2498: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-03-24T01:22:12.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:22:12 smithi118 ceph-mon[31535]: pgmap v2497: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 5 op/s 2024-03-24T01:22:12.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:22:12 smithi118 ceph-mon[31535]: pgmap v2498: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-03-24T01:22:14.704 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:22:14 smithi189 ceph-mon[36187]: pgmap v2499: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 5 op/s 2024-03-24T01:22:14.711 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:22:14 smithi193 ceph-mon[36878]: pgmap v2499: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 5 op/s 2024-03-24T01:22:14.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:22:14 smithi118 ceph-mon[31535]: pgmap v2499: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 5 op/s 2024-03-24T01:22:16.279 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T01:22:16.279+0000 7f81ee000640 1 -- 172.21.15.118:0/1693767697 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f81c80065b0 con 0x7f81f81052d0 2024-03-24T01:22:16.279 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T01:22:16.279+0000 7f81db400640 1 -- 172.21.15.118:0/659686424 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f81c4006290 con 0x5622651a13f0 2024-03-24T01:22:16.279 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T01:22:16.279+0000 7f81ef400640 1 -- 172.21.15.118:0/1693767697 <== mon.2 v2:172.21.15.189:3300/0 30 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (secure 0 0 0) 0x7f81c80065b0 con 0x7f81f81052d0 2024-03-24T01:22:16.280 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T01:22:16.280+0000 7f81e4c00640 1 -- 172.21.15.118:0/659686424 <== mon.1 v2:172.21.15.193:3300/0 30 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 409+0+0 (secure 0 0 0) 0x7f81c4006290 con 0x5622651a13f0 2024-03-24T01:22:16.711 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:22:16 smithi193 ceph-mon[36878]: pgmap v2500: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-03-24T01:22:16.711 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:22:16 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 3431.663965 secs 2024-03-24T01:22:16.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:22:16 smithi118 ceph-mon[31535]: pgmap v2500: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-03-24T01:22:16.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:22:16 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 3431.663965 secs 2024-03-24T01:22:16.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:22:16 smithi189 ceph-mon[36187]: pgmap v2500: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-03-24T01:22:16.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:22:16 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 3431.663965 secs 2024-03-24T01:22:18.711 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:22:18 smithi193 ceph-mon[36878]: pgmap v2501: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-03-24T01:22:18.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:22:18 smithi118 ceph-mon[31535]: pgmap v2501: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-03-24T01:22:18.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:22:18 smithi189 ceph-mon[36187]: pgmap v2501: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-03-24T01:22:20.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:22:20 smithi118 ceph-mon[31535]: pgmap v2502: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 4 op/s 2024-03-24T01:22:20.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:22:20 smithi189 ceph-mon[36187]: pgmap v2502: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 4 op/s 2024-03-24T01:22:20.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:22:20 smithi193 ceph-mon[36878]: pgmap v2502: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 4 op/s 2024-03-24T01:22:21.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:22:21 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 3436.663976 secs 2024-03-24T01:22:21.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:22:21 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 3436.663976 secs 2024-03-24T01:22:21.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:22:21 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 3436.663976 secs 2024-03-24T01:22:22.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:22:22 smithi118 ceph-mon[31535]: pgmap v2503: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 5 op/s 2024-03-24T01:22:22.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:22:22 smithi189 ceph-mon[36187]: pgmap v2503: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 5 op/s 2024-03-24T01:22:22.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:22:22 smithi193 ceph-mon[36878]: pgmap v2503: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 5 op/s 2024-03-24T01:22:24.921 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:22:24 smithi118 ceph-mon[31535]: pgmap v2504: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T01:22:24.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:22:24 smithi189 ceph-mon[36187]: pgmap v2504: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T01:22:24.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:22:24 smithi193 ceph-mon[36878]: pgmap v2504: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T01:22:26.755 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:22:26 smithi189 ceph-mon[36187]: pgmap v2505: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-03-24T01:22:26.755 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:22:26 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 3441.664065 secs 2024-03-24T01:22:26.763 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:22:26 smithi193 ceph-mon[36878]: pgmap v2505: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-03-24T01:22:26.763 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:22:26 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 3441.664065 secs 2024-03-24T01:22:26.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:22:26 smithi118 ceph-mon[31535]: pgmap v2505: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-03-24T01:22:26.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:22:26 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 3441.664065 secs 2024-03-24T01:22:28.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:22:28 smithi118 ceph-mon[31535]: pgmap v2506: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:22:28.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:22:28 smithi189 ceph-mon[36187]: pgmap v2506: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:22:28.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:22:28 smithi193 ceph-mon[36878]: pgmap v2506: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:22:30.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:22:30 smithi118 ceph-mon[31535]: pgmap v2507: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T01:22:30.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:22:30 smithi189 ceph-mon[36187]: pgmap v2507: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T01:22:30.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:22:30 smithi193 ceph-mon[36878]: pgmap v2507: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T01:22:31.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:22:31 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 3446.664124 secs 2024-03-24T01:22:31.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:22:31 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 3446.664124 secs 2024-03-24T01:22:31.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:22:31 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 3446.664124 secs 2024-03-24T01:22:32.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:22:32 smithi118 ceph-mon[31535]: pgmap v2508: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T01:22:32.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:22:32 smithi189 ceph-mon[36187]: pgmap v2508: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T01:22:32.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:22:32 smithi193 ceph-mon[36878]: pgmap v2508: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T01:22:34.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:22:34 smithi118 ceph-mon[31535]: pgmap v2509: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T01:22:34.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:22:34 smithi189 ceph-mon[36187]: pgmap v2509: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T01:22:34.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:22:34 smithi193 ceph-mon[36878]: pgmap v2509: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T01:22:36.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:22:36 smithi118 ceph-mon[31535]: pgmap v2510: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-03-24T01:22:36.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:22:36 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 3451.664167 secs 2024-03-24T01:22:36.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:22:36 smithi189 ceph-mon[36187]: pgmap v2510: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-03-24T01:22:36.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:22:36 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 3451.664167 secs 2024-03-24T01:22:36.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:22:36 smithi193 ceph-mon[36878]: pgmap v2510: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-03-24T01:22:36.962 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:22:36 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 3451.664167 secs 2024-03-24T01:22:38.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:22:38 smithi118 ceph-mon[31535]: pgmap v2511: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:22:38.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:22:38 smithi189 ceph-mon[36187]: pgmap v2511: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:22:38.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:22:38 smithi193 ceph-mon[36878]: pgmap v2511: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:22:40.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:22:40 smithi118 ceph-mon[31535]: pgmap v2512: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T01:22:40.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:22:40 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 3456.664228 secs 2024-03-24T01:22:40.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:22:40 smithi189 ceph-mon[36187]: pgmap v2512: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T01:22:40.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:22:40 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 3456.664228 secs 2024-03-24T01:22:40.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:22:40 smithi193 ceph-mon[36878]: pgmap v2512: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T01:22:40.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:22:40 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 3456.664228 secs 2024-03-24T01:22:42.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:22:42 smithi118 ceph-mon[31535]: pgmap v2513: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T01:22:42.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:22:42 smithi189 ceph-mon[36187]: pgmap v2513: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T01:22:42.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:22:42 smithi193 ceph-mon[36878]: pgmap v2513: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T01:22:44.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:22:44 smithi118 ceph-mon[31535]: pgmap v2514: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T01:22:44.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:22:44 smithi189 ceph-mon[36187]: pgmap v2514: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T01:22:44.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:22:44 smithi193 ceph-mon[36878]: pgmap v2514: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T01:22:45.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:22:45 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 3461.664318 secs 2024-03-24T01:22:45.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:22:45 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 3461.664318 secs 2024-03-24T01:22:45.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:22:45 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 3461.664318 secs 2024-03-24T01:22:46.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:22:46 smithi118 ceph-mon[31535]: pgmap v2515: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-03-24T01:22:46.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:22:46 smithi189 ceph-mon[36187]: pgmap v2515: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-03-24T01:22:46.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:22:46 smithi193 ceph-mon[36878]: pgmap v2515: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-03-24T01:22:48.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:22:48 smithi118 ceph-mon[31535]: pgmap v2516: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:22:48.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:22:48 smithi189 ceph-mon[36187]: pgmap v2516: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:22:48.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:22:48 smithi193 ceph-mon[36878]: pgmap v2516: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:22:50.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:22:50 smithi118 ceph-mon[31535]: pgmap v2517: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T01:22:50.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:22:50 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 3466.664381 secs 2024-03-24T01:22:50.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:22:50 smithi189 ceph-mon[36187]: pgmap v2517: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T01:22:50.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:22:50 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 3466.664381 secs 2024-03-24T01:22:50.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:22:50 smithi193 ceph-mon[36878]: pgmap v2517: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T01:22:50.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:22:50 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 3466.664381 secs 2024-03-24T01:22:52.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:22:52 smithi118 ceph-mon[31535]: pgmap v2518: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T01:22:52.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:22:52 smithi189 ceph-mon[36187]: pgmap v2518: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T01:22:52.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:22:52 smithi193 ceph-mon[36878]: pgmap v2518: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T01:22:54.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:22:54 smithi118 ceph-mon[31535]: pgmap v2519: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T01:22:54.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:22:54 smithi189 ceph-mon[36187]: pgmap v2519: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T01:22:54.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:22:54 smithi193 ceph-mon[36878]: pgmap v2519: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T01:22:55.950 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:22:55 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 3471.664470 secs 2024-03-24T01:22:55.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:22:55 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 3471.664470 secs 2024-03-24T01:22:55.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:22:55 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 3471.664470 secs 2024-03-24T01:22:56.854 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:22:56 smithi189 ceph-mon[36187]: pgmap v2520: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-03-24T01:22:56.862 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:22:56 smithi193 ceph-mon[36878]: pgmap v2520: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-03-24T01:22:56.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:22:56 smithi118 ceph-mon[31535]: pgmap v2520: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-03-24T01:22:58.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:22:58 smithi118 ceph-mon[31535]: pgmap v2521: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:22:58.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:22:58 smithi189 ceph-mon[36187]: pgmap v2521: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:22:58.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:22:58 smithi193 ceph-mon[36878]: pgmap v2521: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:23:00.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:23:00 smithi118 ceph-mon[31535]: pgmap v2522: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T01:23:00.947 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:23:00 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 3476.664530 secs 2024-03-24T01:23:00.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:23:00 smithi189 ceph-mon[36187]: pgmap v2522: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T01:23:00.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:23:00 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 3476.664530 secs 2024-03-24T01:23:00.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:23:00 smithi193 ceph-mon[36878]: pgmap v2522: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T01:23:00.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:23:00 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 3476.664530 secs 2024-03-24T01:23:01.286 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T01:23:01.285+0000 7f81db400640 1 -- 172.21.15.118:0/659686424 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- auth(proto 2 132 bytes epoch 0) v1 -- 0x7f81c4006290 con 0x5622651a13f0 2024-03-24T01:23:01.368 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T01:23:01.285+0000 7f81ee000640 1 -- 172.21.15.118:0/1693767697 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- auth(proto 2 132 bytes epoch 0) v1 -- 0x7f81c80065b0 con 0x7f81f81052d0 2024-03-24T01:23:01.369 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T01:23:01.286+0000 7f81ef400640 1 -- 172.21.15.118:0/1693767697 <== mon.2 v2:172.21.15.189:3300/0 31 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 233+0+0 (secure 0 0 0) 0x7f81c80065b0 con 0x7f81f81052d0 2024-03-24T01:23:01.369 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T01:23:01.286+0000 7f81e4c00640 1 -- 172.21.15.118:0/659686424 <== mon.1 v2:172.21.15.193:3300/0 31 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 233+0+0 (secure 0 0 0) 0x7f81c4006290 con 0x5622651a13f0 2024-03-24T01:23:02.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:23:02 smithi118 ceph-mon[31535]: pgmap v2523: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T01:23:02.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:23:02 smithi189 ceph-mon[36187]: pgmap v2523: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T01:23:02.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:23:02 smithi193 ceph-mon[36878]: pgmap v2523: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T01:23:04.950 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:23:04 smithi118 ceph-mon[31535]: pgmap v2524: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 5 op/s 2024-03-24T01:23:04.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:23:04 smithi189 ceph-mon[36187]: pgmap v2524: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 5 op/s 2024-03-24T01:23:04.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:23:04 smithi193 ceph-mon[36878]: pgmap v2524: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 5 op/s 2024-03-24T01:23:05.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:23:05 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 3481.664569 secs 2024-03-24T01:23:05.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:23:05 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 3481.664569 secs 2024-03-24T01:23:05.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:23:05 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 3481.664569 secs 2024-03-24T01:23:06.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:23:06 smithi118 ceph-mon[31535]: pgmap v2525: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-03-24T01:23:06.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:23:06 smithi189 ceph-mon[36187]: pgmap v2525: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-03-24T01:23:06.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:23:06 smithi193 ceph-mon[36878]: pgmap v2525: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-03-24T01:23:08.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:23:08 smithi118 ceph-mon[31535]: pgmap v2526: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:23:08.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:23:08 smithi189 ceph-mon[36187]: pgmap v2526: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:23:08.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:23:08 smithi193 ceph-mon[36878]: pgmap v2526: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:23:10.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:23:10 smithi118 ceph-mon[31535]: pgmap v2527: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T01:23:10.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:23:10 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 3486.664631 secs 2024-03-24T01:23:10.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:23:10 smithi189 ceph-mon[36187]: pgmap v2527: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T01:23:10.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:23:10 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 3486.664631 secs 2024-03-24T01:23:10.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:23:10 smithi193 ceph-mon[36878]: pgmap v2527: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T01:23:10.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:23:10 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 3486.664631 secs 2024-03-24T01:23:11.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:23:11 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T01:23:11.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:23:11 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T01:23:11.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:23:11 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T01:23:12.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:23:12 smithi118 ceph-mon[31535]: pgmap v2528: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T01:23:12.947 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:23:12 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T01:23:12.947 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:23:12 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T01:23:12.947 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:23:12 smithi118 ceph-mon[31535]: pgmap v2529: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 5 op/s 2024-03-24T01:23:12.947 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:23:12 smithi118 ceph-mon[31535]: pgmap v2530: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 5 op/s 2024-03-24T01:23:12.947 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:23:12 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T01:23:12.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:23:12 smithi189 ceph-mon[36187]: pgmap v2528: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T01:23:12.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:23:12 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T01:23:12.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:23:12 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T01:23:12.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:23:12 smithi189 ceph-mon[36187]: pgmap v2529: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 5 op/s 2024-03-24T01:23:12.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:23:12 smithi189 ceph-mon[36187]: pgmap v2530: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 5 op/s 2024-03-24T01:23:12.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:23:12 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T01:23:12.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:23:12 smithi193 ceph-mon[36878]: pgmap v2528: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T01:23:12.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:23:12 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T01:23:12.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:23:12 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T01:23:12.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:23:12 smithi193 ceph-mon[36878]: pgmap v2529: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 5 op/s 2024-03-24T01:23:12.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:23:12 smithi193 ceph-mon[36878]: pgmap v2530: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 5 op/s 2024-03-24T01:23:12.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:23:12 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T01:23:15.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:23:15 smithi118 ceph-mon[31535]: pgmap v2531: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-03-24T01:23:15.704 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:23:15 smithi189 ceph-mon[36187]: pgmap v2531: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-03-24T01:23:15.711 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:23:15 smithi193 ceph-mon[36878]: pgmap v2531: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-03-24T01:23:16.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:23:16 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 3491.664688 secs 2024-03-24T01:23:16.704 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:23:16 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 3491.664688 secs 2024-03-24T01:23:16.710 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:23:16 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 3491.664688 secs 2024-03-24T01:23:17.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:23:17 smithi118 ceph-mon[31535]: pgmap v2532: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-03-24T01:23:17.704 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:23:17 smithi189 ceph-mon[36187]: pgmap v2532: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-03-24T01:23:17.711 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:23:17 smithi193 ceph-mon[36878]: pgmap v2532: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-03-24T01:23:19.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:23:19 smithi118 ceph-mon[31535]: pgmap v2533: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-03-24T01:23:19.704 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:23:19 smithi189 ceph-mon[36187]: pgmap v2533: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-03-24T01:23:19.710 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:23:19 smithi193 ceph-mon[36878]: pgmap v2533: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-03-24T01:23:21.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:23:21 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 3496.664749 secs 2024-03-24T01:23:21.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:23:21 smithi118 ceph-mon[31535]: pgmap v2534: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-03-24T01:23:21.704 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:23:21 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 3496.664749 secs 2024-03-24T01:23:21.704 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:23:21 smithi189 ceph-mon[36187]: pgmap v2534: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-03-24T01:23:21.711 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:23:21 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 3496.664749 secs 2024-03-24T01:23:21.711 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:23:21 smithi193 ceph-mon[36878]: pgmap v2534: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-03-24T01:23:23.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:23:23 smithi118 ceph-mon[31535]: pgmap v2535: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 5 op/s 2024-03-24T01:23:23.704 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:23:23 smithi189 ceph-mon[36187]: pgmap v2535: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 5 op/s 2024-03-24T01:23:23.711 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:23:23 smithi193 ceph-mon[36878]: pgmap v2535: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 5 op/s 2024-03-24T01:23:25.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:23:25 smithi118 ceph-mon[31535]: pgmap v2536: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-24T01:23:25.704 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:23:25 smithi189 ceph-mon[36187]: pgmap v2536: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-24T01:23:25.710 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:23:25 smithi193 ceph-mon[36878]: pgmap v2536: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-24T01:23:26.605 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:23:26 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 3501.664804 secs 2024-03-24T01:23:26.612 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:23:26 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 3501.664804 secs 2024-03-24T01:23:26.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:23:26 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 3501.664804 secs 2024-03-24T01:23:27.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:23:27 smithi118 ceph-mon[31535]: pgmap v2537: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:23:27.704 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:23:27 smithi189 ceph-mon[36187]: pgmap v2537: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:23:27.710 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:23:27 smithi193 ceph-mon[36878]: pgmap v2537: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:23:29.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:23:29 smithi118 ceph-mon[31535]: pgmap v2538: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T01:23:29.704 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:23:29 smithi189 ceph-mon[36187]: pgmap v2538: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T01:23:29.710 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:23:29 smithi193 ceph-mon[36878]: pgmap v2538: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T01:23:31.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:23:31 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 3506.664863 secs 2024-03-24T01:23:31.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:23:31 smithi118 ceph-mon[31535]: pgmap v2539: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T01:23:31.704 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:23:31 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 3506.664863 secs 2024-03-24T01:23:31.704 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:23:31 smithi189 ceph-mon[36187]: pgmap v2539: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T01:23:31.711 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:23:31 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 3506.664863 secs 2024-03-24T01:23:31.711 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:23:31 smithi193 ceph-mon[36878]: pgmap v2539: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T01:23:33.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:23:33 smithi118 ceph-mon[31535]: pgmap v2540: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:23:33.704 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:23:33 smithi189 ceph-mon[36187]: pgmap v2540: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:23:33.711 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:23:33 smithi193 ceph-mon[36878]: pgmap v2540: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:23:35.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:23:35 smithi118 ceph-mon[31535]: pgmap v2541: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-24T01:23:35.704 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:23:35 smithi189 ceph-mon[36187]: pgmap v2541: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-24T01:23:35.710 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:23:35 smithi193 ceph-mon[36878]: pgmap v2541: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-24T01:23:36.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:23:36 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 3511.664918 secs 2024-03-24T01:23:36.704 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:23:36 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 3511.664918 secs 2024-03-24T01:23:36.710 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:23:36 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 3511.664918 secs 2024-03-24T01:23:37.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:23:37 smithi118 ceph-mon[31535]: pgmap v2542: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:23:37.704 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:23:37 smithi189 ceph-mon[36187]: pgmap v2542: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:23:37.712 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:23:37 smithi193 ceph-mon[36878]: pgmap v2542: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:23:39.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:23:39 smithi118 ceph-mon[31535]: pgmap v2543: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T01:23:39.704 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:23:39 smithi189 ceph-mon[36187]: pgmap v2543: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T01:23:39.711 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:23:39 smithi193 ceph-mon[36878]: pgmap v2543: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T01:23:41.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:23:41 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 3516.665007 secs 2024-03-24T01:23:41.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:23:41 smithi118 ceph-mon[31535]: pgmap v2544: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T01:23:41.704 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:23:41 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 3516.665007 secs 2024-03-24T01:23:41.704 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:23:41 smithi189 ceph-mon[36187]: pgmap v2544: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T01:23:41.711 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:23:41 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 3516.665007 secs 2024-03-24T01:23:41.711 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:23:41 smithi193 ceph-mon[36878]: pgmap v2544: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T01:23:43.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:23:43 smithi118 ceph-mon[31535]: pgmap v2545: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:23:43.703 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:23:43 smithi189 ceph-mon[36187]: pgmap v2545: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:23:43.711 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:23:43 smithi193 ceph-mon[36878]: pgmap v2545: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:23:45.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:23:45 smithi118 ceph-mon[31535]: pgmap v2546: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-24T01:23:45.704 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:23:45 smithi189 ceph-mon[36187]: pgmap v2546: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-24T01:23:45.710 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:23:45 smithi193 ceph-mon[36878]: pgmap v2546: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-24T01:23:46.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:23:46 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 3521.665096 secs 2024-03-24T01:23:46.704 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:23:46 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 3521.665096 secs 2024-03-24T01:23:46.711 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:23:46 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 3521.665096 secs 2024-03-24T01:23:47.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:23:47 smithi118 ceph-mon[31535]: pgmap v2547: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:23:47.704 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:23:47 smithi189 ceph-mon[36187]: pgmap v2547: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:23:47.710 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:23:47 smithi193 ceph-mon[36878]: pgmap v2547: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:23:49.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:23:49 smithi118 ceph-mon[31535]: pgmap v2548: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T01:23:49.704 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:23:49 smithi189 ceph-mon[36187]: pgmap v2548: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T01:23:49.710 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:23:49 smithi193 ceph-mon[36878]: pgmap v2548: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T01:23:51.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:23:51 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 3526.665157 secs 2024-03-24T01:23:51.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:23:51 smithi118 ceph-mon[31535]: pgmap v2549: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T01:23:51.704 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:23:51 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 3526.665157 secs 2024-03-24T01:23:51.704 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:23:51 smithi189 ceph-mon[36187]: pgmap v2549: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T01:23:51.711 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:23:51 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 3526.665157 secs 2024-03-24T01:23:51.711 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:23:51 smithi193 ceph-mon[36878]: pgmap v2549: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T01:23:53.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:23:53 smithi118 ceph-mon[31535]: pgmap v2550: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:23:53.704 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:23:53 smithi189 ceph-mon[36187]: pgmap v2550: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:23:53.710 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:23:53 smithi193 ceph-mon[36878]: pgmap v2550: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:23:55.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:23:55 smithi118 ceph-mon[31535]: pgmap v2551: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-24T01:23:55.704 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:23:55 smithi189 ceph-mon[36187]: pgmap v2551: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-24T01:23:55.710 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:23:55 smithi193 ceph-mon[36878]: pgmap v2551: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-24T01:23:56.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:23:56 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 3531.665219 secs 2024-03-24T01:23:56.704 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:23:56 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 3531.665219 secs 2024-03-24T01:23:56.711 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:23:56 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 3531.665219 secs 2024-03-24T01:23:57.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:23:57 smithi118 ceph-mon[31535]: pgmap v2552: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:23:57.704 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:23:57 smithi189 ceph-mon[36187]: pgmap v2552: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:23:57.711 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:23:57 smithi193 ceph-mon[36878]: pgmap v2552: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:23:59.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:23:59 smithi118 ceph-mon[31535]: pgmap v2553: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T01:23:59.704 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:23:59 smithi189 ceph-mon[36187]: pgmap v2553: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T01:23:59.710 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:23:59 smithi193 ceph-mon[36878]: pgmap v2553: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T01:24:01.704 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:24:01 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 3536.665273 secs 2024-03-24T01:24:01.704 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:24:01 smithi189 ceph-mon[36187]: pgmap v2554: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T01:24:01.711 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:24:01 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 3536.665273 secs 2024-03-24T01:24:01.711 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:24:01 smithi193 ceph-mon[36878]: pgmap v2554: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T01:24:01.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:24:01 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 3536.665273 secs 2024-03-24T01:24:01.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:24:01 smithi118 ceph-mon[31535]: pgmap v2554: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T01:24:03.704 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:24:03 smithi189 ceph-mon[36187]: pgmap v2555: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:24:03.711 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:24:03 smithi193 ceph-mon[36878]: pgmap v2555: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:24:03.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:24:03 smithi118 ceph-mon[31535]: pgmap v2555: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:24:05.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:24:04 smithi118 ceph-mon[31535]: pgmap v2556: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-24T01:24:05.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:24:04 smithi189 ceph-mon[36187]: pgmap v2556: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-24T01:24:05.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:24:04 smithi193 ceph-mon[36878]: pgmap v2556: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-24T01:24:06.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:24:05 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 3541.665359 secs 2024-03-24T01:24:06.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:24:05 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 3541.665359 secs 2024-03-24T01:24:06.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:24:05 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 3541.665359 secs 2024-03-24T01:24:07.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:24:06 smithi118 ceph-mon[31535]: pgmap v2557: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:24:07.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:24:06 smithi189 ceph-mon[36187]: pgmap v2557: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:24:07.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:24:06 smithi193 ceph-mon[36878]: pgmap v2557: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:24:09.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:24:09 smithi118 ceph-mon[31535]: pgmap v2558: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T01:24:09.704 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:24:09 smithi189 ceph-mon[36187]: pgmap v2558: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T01:24:09.711 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:24:09 smithi193 ceph-mon[36878]: pgmap v2558: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T01:24:11.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:24:11 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 3546.665445 secs 2024-03-24T01:24:11.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:24:11 smithi118 ceph-mon[31535]: pgmap v2559: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T01:24:11.704 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:24:11 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 3546.665445 secs 2024-03-24T01:24:11.704 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:24:11 smithi189 ceph-mon[36187]: pgmap v2559: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T01:24:11.711 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:24:11 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 3546.665445 secs 2024-03-24T01:24:11.711 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:24:11 smithi193 ceph-mon[36878]: pgmap v2559: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T01:24:13.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:24:13 smithi118 ceph-mon[31535]: pgmap v2560: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:24:13.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:24:13 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T01:24:13.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:24:13 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-24T01:24:13.697 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:24:13 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-24T01:24:13.697 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:24:13 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-24T01:24:13.697 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:24:13 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-03-24T01:24:13.704 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:24:13 smithi189 ceph-mon[36187]: pgmap v2560: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:24:13.704 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:24:13 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T01:24:13.704 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:24:13 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-24T01:24:13.704 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:24:13 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-24T01:24:13.704 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:24:13 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-24T01:24:13.704 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:24:13 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-03-24T01:24:13.711 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:24:13 smithi193 ceph-mon[36878]: pgmap v2560: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:24:13.711 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:24:13 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T01:24:13.711 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:24:13 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-24T01:24:13.711 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:24:13 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-24T01:24:13.711 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:24:13 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-24T01:24:13.711 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:24:13 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-03-24T01:24:14.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:24:14 smithi118 ceph-mon[31535]: Adjusting osd_memory_target on smithi189 to 174.1M 2024-03-24T01:24:14.697 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:24:14 smithi118 ceph-mon[31535]: Unable to set osd_memory_target on smithi189 to 182635622: error parsing value: Value '182635622' is below minimum 939524096 2024-03-24T01:24:14.697 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:24:14 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-24T01:24:14.697 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:24:14 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-24T01:24:14.697 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:24:14 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-24T01:24:14.697 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:24:14 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-03-24T01:24:14.697 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:24:14 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T01:24:14.697 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:24:14 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T01:24:14.697 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:24:14 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T01:24:14.704 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:24:14 smithi189 ceph-mon[36187]: Adjusting osd_memory_target on smithi189 to 174.1M 2024-03-24T01:24:14.705 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:24:14 smithi189 ceph-mon[36187]: Unable to set osd_memory_target on smithi189 to 182635622: error parsing value: Value '182635622' is below minimum 939524096 2024-03-24T01:24:14.705 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:24:14 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-24T01:24:14.705 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:24:14 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-24T01:24:14.705 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:24:14 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-24T01:24:14.705 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:24:14 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-03-24T01:24:14.705 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:24:14 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T01:24:14.705 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:24:14 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T01:24:14.705 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:24:14 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T01:24:14.711 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:24:14 smithi193 ceph-mon[36878]: Adjusting osd_memory_target on smithi189 to 174.1M 2024-03-24T01:24:14.711 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:24:14 smithi193 ceph-mon[36878]: Unable to set osd_memory_target on smithi189 to 182635622: error parsing value: Value '182635622' is below minimum 939524096 2024-03-24T01:24:14.711 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:24:14 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-24T01:24:14.711 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:24:14 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-24T01:24:14.711 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:24:14 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-24T01:24:14.711 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:24:14 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-03-24T01:24:14.711 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:24:14 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T01:24:14.711 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:24:14 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T01:24:14.711 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:24:14 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T01:24:15.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:24:15 smithi118 ceph-mon[31535]: Adjusting osd_memory_target on smithi118 to 176.7M 2024-03-24T01:24:15.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:24:15 smithi118 ceph-mon[31535]: Unable to set osd_memory_target on smithi118 to 185365196: error parsing value: Value '185365196' is below minimum 939524096 2024-03-24T01:24:15.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:24:15 smithi118 ceph-mon[31535]: pgmap v2561: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 6 op/s 2024-03-24T01:24:15.697 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:24:15 smithi118 ceph-mon[31535]: pgmap v2562: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 4 op/s 2024-03-24T01:24:15.704 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:24:15 smithi189 ceph-mon[36187]: Adjusting osd_memory_target on smithi118 to 176.7M 2024-03-24T01:24:15.704 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:24:15 smithi189 ceph-mon[36187]: Unable to set osd_memory_target on smithi118 to 185365196: error parsing value: Value '185365196' is below minimum 939524096 2024-03-24T01:24:15.704 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:24:15 smithi189 ceph-mon[36187]: pgmap v2561: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 6 op/s 2024-03-24T01:24:15.704 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:24:15 smithi189 ceph-mon[36187]: pgmap v2562: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 4 op/s 2024-03-24T01:24:15.711 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:24:15 smithi193 ceph-mon[36878]: Adjusting osd_memory_target on smithi118 to 176.7M 2024-03-24T01:24:15.711 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:24:15 smithi193 ceph-mon[36878]: Unable to set osd_memory_target on smithi118 to 185365196: error parsing value: Value '185365196' is below minimum 939524096 2024-03-24T01:24:15.711 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:24:15 smithi193 ceph-mon[36878]: pgmap v2561: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 6 op/s 2024-03-24T01:24:15.711 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:24:15 smithi193 ceph-mon[36878]: pgmap v2562: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 4 op/s 2024-03-24T01:24:16.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:24:16 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 3551.665480 secs 2024-03-24T01:24:16.704 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:24:16 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 3551.665480 secs 2024-03-24T01:24:16.710 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:24:16 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 3551.665480 secs 2024-03-24T01:24:17.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:24:17 smithi118 ceph-mon[31535]: pgmap v2563: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 6 op/s 2024-03-24T01:24:17.704 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:24:17 smithi189 ceph-mon[36187]: pgmap v2563: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 6 op/s 2024-03-24T01:24:17.711 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:24:17 smithi193 ceph-mon[36878]: pgmap v2563: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 6 op/s 2024-03-24T01:24:19.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:24:19 smithi118 ceph-mon[31535]: pgmap v2564: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 5 op/s 2024-03-24T01:24:19.704 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:24:19 smithi189 ceph-mon[36187]: pgmap v2564: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 5 op/s 2024-03-24T01:24:19.711 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:24:19 smithi193 ceph-mon[36878]: pgmap v2564: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 5 op/s 2024-03-24T01:24:21.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:24:21 smithi118 ceph-mon[31535]: pgmap v2565: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 6 op/s 2024-03-24T01:24:21.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:24:21 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 3556.665491 secs 2024-03-24T01:24:21.704 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:24:21 smithi189 ceph-mon[36187]: pgmap v2565: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 6 op/s 2024-03-24T01:24:21.704 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:24:21 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 3556.665491 secs 2024-03-24T01:24:21.711 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:24:21 smithi193 ceph-mon[36878]: pgmap v2565: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 6 op/s 2024-03-24T01:24:21.711 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:24:21 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 3556.665491 secs 2024-03-24T01:24:23.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:24:23 smithi118 ceph-mon[31535]: pgmap v2566: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 6 op/s 2024-03-24T01:24:23.704 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:24:23 smithi189 ceph-mon[36187]: pgmap v2566: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 6 op/s 2024-03-24T01:24:23.711 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:24:23 smithi193 ceph-mon[36878]: pgmap v2566: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 6 op/s 2024-03-24T01:24:25.197 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:24:24 smithi118 ceph-mon[31535]: pgmap v2567: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 5 op/s 2024-03-24T01:24:25.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:24:24 smithi189 ceph-mon[36187]: pgmap v2567: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 5 op/s 2024-03-24T01:24:25.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:24:24 smithi193 ceph-mon[36878]: pgmap v2567: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 5 op/s 2024-03-24T01:24:26.105 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:24:25 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 3561.665548 secs 2024-03-24T01:24:26.112 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:24:25 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 3561.665548 secs 2024-03-24T01:24:26.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:24:25 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 3561.665548 secs 2024-03-24T01:24:27.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:24:26 smithi118 ceph-mon[31535]: pgmap v2568: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T01:24:27.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:24:26 smithi189 ceph-mon[36187]: pgmap v2568: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T01:24:27.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:24:26 smithi193 ceph-mon[36878]: pgmap v2568: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T01:24:29.197 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:24:28 smithi118 ceph-mon[31535]: pgmap v2569: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 5 op/s 2024-03-24T01:24:29.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:24:28 smithi189 ceph-mon[36187]: pgmap v2569: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 5 op/s 2024-03-24T01:24:29.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:24:28 smithi193 ceph-mon[36878]: pgmap v2569: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 5 op/s 2024-03-24T01:24:31.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:24:30 smithi118 ceph-mon[31535]: pgmap v2570: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T01:24:31.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:24:30 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 3566.665637 secs 2024-03-24T01:24:31.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:24:30 smithi189 ceph-mon[36187]: pgmap v2570: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T01:24:31.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:24:30 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 3566.665637 secs 2024-03-24T01:24:31.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:24:30 smithi193 ceph-mon[36878]: pgmap v2570: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T01:24:31.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:24:30 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 3566.665637 secs 2024-03-24T01:24:33.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:24:32 smithi118 ceph-mon[31535]: pgmap v2571: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:24:33.203 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:24:32 smithi189 ceph-mon[36187]: pgmap v2571: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:24:33.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:24:32 smithi193 ceph-mon[36878]: pgmap v2571: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:24:35.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:24:34 smithi118 ceph-mon[31535]: pgmap v2572: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T01:24:35.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:24:34 smithi189 ceph-mon[36187]: pgmap v2572: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T01:24:35.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:24:34 smithi193 ceph-mon[36878]: pgmap v2572: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T01:24:36.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:24:35 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 3571.665678 secs 2024-03-24T01:24:36.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:24:35 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 3571.665678 secs 2024-03-24T01:24:36.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:24:35 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 3571.665678 secs 2024-03-24T01:24:37.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:24:36 smithi118 ceph-mon[31535]: pgmap v2573: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T01:24:37.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:24:36 smithi189 ceph-mon[36187]: pgmap v2573: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T01:24:37.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:24:36 smithi193 ceph-mon[36878]: pgmap v2573: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T01:24:39.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:24:38 smithi118 ceph-mon[31535]: pgmap v2574: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 5 op/s 2024-03-24T01:24:39.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:24:38 smithi189 ceph-mon[36187]: pgmap v2574: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 5 op/s 2024-03-24T01:24:39.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:24:38 smithi193 ceph-mon[36878]: pgmap v2574: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 5 op/s 2024-03-24T01:24:41.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:24:40 smithi118 ceph-mon[31535]: pgmap v2575: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T01:24:41.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:24:40 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 3576.665716 secs 2024-03-24T01:24:41.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:24:40 smithi189 ceph-mon[36187]: pgmap v2575: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T01:24:41.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:24:40 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 3576.665716 secs 2024-03-24T01:24:41.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:24:40 smithi193 ceph-mon[36878]: pgmap v2575: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T01:24:41.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:24:40 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 3576.665716 secs 2024-03-24T01:24:43.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:24:42 smithi118 ceph-mon[31535]: pgmap v2576: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:24:43.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:24:42 smithi189 ceph-mon[36187]: pgmap v2576: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:24:43.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:24:42 smithi193 ceph-mon[36878]: pgmap v2576: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:24:45.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:24:44 smithi118 ceph-mon[31535]: pgmap v2577: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T01:24:45.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:24:44 smithi189 ceph-mon[36187]: pgmap v2577: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T01:24:45.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:24:44 smithi193 ceph-mon[36878]: pgmap v2577: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T01:24:46.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:24:45 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 3581.665806 secs 2024-03-24T01:24:46.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:24:45 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 3581.665806 secs 2024-03-24T01:24:46.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:24:45 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 3581.665806 secs 2024-03-24T01:24:47.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:24:46 smithi118 ceph-mon[31535]: pgmap v2578: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T01:24:47.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:24:46 smithi189 ceph-mon[36187]: pgmap v2578: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T01:24:47.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:24:46 smithi193 ceph-mon[36878]: pgmap v2578: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T01:24:49.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:24:48 smithi118 ceph-mon[31535]: pgmap v2579: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 5 op/s 2024-03-24T01:24:49.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:24:48 smithi189 ceph-mon[36187]: pgmap v2579: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 5 op/s 2024-03-24T01:24:49.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:24:48 smithi193 ceph-mon[36878]: pgmap v2579: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 5 op/s 2024-03-24T01:24:51.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:24:50 smithi118 ceph-mon[31535]: pgmap v2580: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T01:24:51.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:24:50 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 3586.665893 secs 2024-03-24T01:24:51.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:24:50 smithi189 ceph-mon[36187]: pgmap v2580: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T01:24:51.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:24:50 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 3586.665893 secs 2024-03-24T01:24:51.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:24:50 smithi193 ceph-mon[36878]: pgmap v2580: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T01:24:51.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:24:50 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 3586.665893 secs 2024-03-24T01:24:53.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:24:52 smithi118 ceph-mon[31535]: pgmap v2581: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:24:53.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:24:52 smithi189 ceph-mon[36187]: pgmap v2581: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:24:53.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:24:52 smithi193 ceph-mon[36878]: pgmap v2581: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:24:55.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:24:54 smithi118 ceph-mon[31535]: pgmap v2582: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-03-24T01:24:55.203 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:24:54 smithi189 ceph-mon[36187]: pgmap v2582: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-03-24T01:24:55.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:24:54 smithi193 ceph-mon[36878]: pgmap v2582: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-03-24T01:24:56.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:24:55 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 3591.665981 secs 2024-03-24T01:24:56.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:24:55 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 3591.665981 secs 2024-03-24T01:24:56.447 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:24:55 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 3591.665981 secs 2024-03-24T01:24:57.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:24:56 smithi189 ceph-mon[36187]: pgmap v2583: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 6 op/s 2024-03-24T01:24:57.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:24:56 smithi193 ceph-mon[36878]: pgmap v2583: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 6 op/s 2024-03-24T01:24:57.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:24:56 smithi118 ceph-mon[31535]: pgmap v2583: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 6 op/s 2024-03-24T01:24:59.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:24:58 smithi193 ceph-mon[36878]: pgmap v2584: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 5 op/s 2024-03-24T01:24:59.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:24:58 smithi118 ceph-mon[31535]: pgmap v2584: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 5 op/s 2024-03-24T01:24:59.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:24:58 smithi189 ceph-mon[36187]: pgmap v2584: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 5 op/s 2024-03-24T01:25:01.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:25:00 smithi193 ceph-mon[36878]: pgmap v2585: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 6 op/s 2024-03-24T01:25:01.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:25:00 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 3596.666071 secs 2024-03-24T01:25:01.250 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:25:00 smithi118 ceph-mon[31535]: pgmap v2585: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 6 op/s 2024-03-24T01:25:01.250 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:25:00 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 3596.666071 secs 2024-03-24T01:25:01.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:25:00 smithi189 ceph-mon[36187]: pgmap v2585: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 6 op/s 2024-03-24T01:25:01.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:25:00 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 3596.666071 secs 2024-03-24T01:25:03.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:25:02 smithi118 ceph-mon[31535]: pgmap v2586: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-03-24T01:25:03.455 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:25:02 smithi189 ceph-mon[36187]: pgmap v2586: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-03-24T01:25:03.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:25:02 smithi193 ceph-mon[36878]: pgmap v2586: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-03-24T01:25:05.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:25:04 smithi118 ceph-mon[31535]: pgmap v2587: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-03-24T01:25:05.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:25:04 smithi189 ceph-mon[36187]: pgmap v2587: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-03-24T01:25:05.460 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:25:04 smithi193 ceph-mon[36878]: pgmap v2587: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-03-24T01:25:06.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:25:05 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 3601.666133 secs 2024-03-24T01:25:06.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:25:05 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 3601.666133 secs 2024-03-24T01:25:06.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:25:05 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 3601.666133 secs 2024-03-24T01:25:07.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:25:06 smithi118 ceph-mon[31535]: pgmap v2588: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T01:25:07.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:25:06 smithi189 ceph-mon[36187]: pgmap v2588: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T01:25:07.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:25:06 smithi193 ceph-mon[36878]: pgmap v2588: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T01:25:09.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:25:08 smithi118 ceph-mon[31535]: pgmap v2589: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 5 op/s 2024-03-24T01:25:09.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:25:08 smithi189 ceph-mon[36187]: pgmap v2589: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 5 op/s 2024-03-24T01:25:09.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:25:08 smithi193 ceph-mon[36878]: pgmap v2589: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 5 op/s 2024-03-24T01:25:11.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:25:10 smithi118 ceph-mon[31535]: pgmap v2590: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T01:25:11.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:25:10 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 3606.666224 secs 2024-03-24T01:25:11.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:25:10 smithi189 ceph-mon[36187]: pgmap v2590: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T01:25:11.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:25:10 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 3606.666224 secs 2024-03-24T01:25:11.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:25:10 smithi193 ceph-mon[36878]: pgmap v2590: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T01:25:11.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:25:10 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 3606.666224 secs 2024-03-24T01:25:13.249 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:25:12 smithi118 ceph-mon[31535]: pgmap v2591: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:25:13.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:25:12 smithi189 ceph-mon[36187]: pgmap v2591: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:25:13.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:25:12 smithi193 ceph-mon[36878]: pgmap v2591: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:25:15.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:25:14 smithi118 ceph-mon[31535]: pgmap v2592: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T01:25:15.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:25:14 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T01:25:15.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:25:14 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-24T01:25:15.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:25:14 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-24T01:25:15.447 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:25:14 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-03-24T01:25:15.447 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:25:14 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-03-24T01:25:15.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:25:14 smithi189 ceph-mon[36187]: pgmap v2592: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T01:25:15.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:25:14 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T01:25:15.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:25:14 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-24T01:25:15.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:25:14 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-24T01:25:15.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:25:14 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-03-24T01:25:15.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:25:14 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-03-24T01:25:15.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:25:14 smithi193 ceph-mon[36878]: pgmap v2592: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T01:25:15.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:25:14 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T01:25:15.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:25:14 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-24T01:25:15.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:25:14 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-24T01:25:15.462 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:25:14 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-03-24T01:25:15.462 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:25:14 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-03-24T01:25:15.741 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T01:25:15.740+0000 7f81f6000640 1 -- 172.21.15.118:0/1693767697 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7f81dc0a4c80 msgr2=0x7f81dc0b05c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 12 2024-03-24T01:25:15.741 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T01:25:15.740+0000 7f81f6000640 1 -- 172.21.15.118:0/1693767697 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7f81dc0a4c80 msgr2=0x7f81dc0b05c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-03-24T01:25:15.741 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T01:25:15.740+0000 7f81f6000640 1 --2- 172.21.15.118:0/1693767697 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7f81dc0a4c80 0x7f81dc0b05c0 secure :-1 s=READY pgs=462 cs=0 l=1 rev1=1 crypto rx=0x7f81e0009b70 tx=0x7f81e003a040 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-03-24T01:25:15.741 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T01:25:15.740+0000 7f81f6000640 1 --2- 172.21.15.118:0/1693767697 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7f81dc0a4c80 0x7f81dc0b05c0 secure :-1 s=READY pgs=462 cs=0 l=1 rev1=1 crypto rx=0x7f81e0009b70 tx=0x7f81e003a040 comp rx=0 tx=0).stop 2024-03-24T01:25:15.741 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T01:25:15.740+0000 7f81f6000640 1 -- 172.21.15.118:0/1693767697 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7f81dc0a4c80 msgr2=0x7f81dc0b05c0 unknown :-1 s=STATE_CLOSED l=1).tick see no progress in more than 10000000 us during connecting to v2:172.21.15.118:6800/2112912943, fault. 2024-03-24T01:25:15.741 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T01:25:15.740+0000 7f81ef400640 1 -- 172.21.15.118:0/1693767697 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7f81dc0a4c80 msgr2=0x7f81dc0b05c0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-03-24T01:25:15.742 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T01:25:15.740+0000 7f81ef400640 1 --2- 172.21.15.118:0/1693767697 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7f81dc0a4c80 0x7f81dc0b05c0 unknown :-1 s=CLOSED pgs=462 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:25:15.742 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T01:25:15.740+0000 7f81ef400640 1 --2- 172.21.15.118:0/1693767697 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7f81dc0a33e0 0x7f81dc0a3bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:25:15.742 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T01:25:15.741+0000 7f81f6000640 1 --2- 172.21.15.118:0/1693767697 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7f81dc0a33e0 0x7f81dc0a3bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:25:15.743 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T01:25:15.742+0000 7f81f6000640 1 --2- 172.21.15.118:0/1693767697 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7f81dc0a33e0 0x7f81dc0a3bd0 secure :-1 s=READY pgs=480 cs=0 l=1 rev1=1 crypto rx=0x562265215970 tx=0x7f81e004e040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:25:15.761 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T01:25:15.761+0000 7f81f6000640 1 -- 172.21.15.118:0/659686424 >> [v2:172.21.15.189:6838/4074484168,v1:172.21.15.189:6839/4074484168] conn(0x5622651b71e0 msgr2=0x562265215110 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).tick idle (900000840) for more than 900000000 us, fault. 2024-03-24T01:25:15.761 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T01:25:15.761+0000 7f81f6000640 1 --2- 172.21.15.118:0/659686424 >> [v2:172.21.15.189:6838/4074484168,v1:172.21.15.189:6839/4074484168] conn(0x5622651b71e0 0x562265215110 unknown :-1 s=READY pgs=561 cs=2 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault initiating reconnect 2024-03-24T01:25:15.762 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T01:25:15.761+0000 7f81f6000640 1 --2- 172.21.15.118:0/659686424 >> [v2:172.21.15.189:6838/4074484168,v1:172.21.15.189:6839/4074484168] conn(0x5622651b71e0 0x562265215110 unknown :-1 s=BANNER_CONNECTING pgs=561 cs=3 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:25:15.762 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T01:25:15.762+0000 7f81f6000640 1 --2- 172.21.15.118:0/659686424 >> [v2:172.21.15.189:6838/4074484168,v1:172.21.15.189:6839/4074484168] conn(0x5622651b71e0 0x562265215110 crc :-1 s=READY pgs=561 cs=3 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=76f3d534289740f0 server_cookie=b31a2335a59552de in_seq=0 out_seq=0 2024-03-24T01:25:15.763 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T01:25:15.762+0000 7f81e4c00640 10 client.27094 ms_handle_connect on v2:172.21.15.189:6838/4074484168 2024-03-24T01:25:16.197 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:25:16 smithi118 ceph-mon[31535]: Adjusting osd_memory_target on smithi193 to 174.1M 2024-03-24T01:25:16.197 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:25:16 smithi118 ceph-mon[31535]: Unable to set osd_memory_target on smithi193 to 182639923: error parsing value: Value '182639923' is below minimum 939524096 2024-03-24T01:25:16.197 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:25:16 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 3611.666311 secs 2024-03-24T01:25:16.197 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:25:16 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T01:25:16.197 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:25:16 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T01:25:16.197 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:25:16 smithi118 ceph-mon[31535]: pgmap v2593: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 6 op/s 2024-03-24T01:25:16.197 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:25:16 smithi118 ceph-mon[31535]: pgmap v2594: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 6 op/s 2024-03-24T01:25:16.197 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:25:16 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T01:25:16.309 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T01:25:16.308+0000 7f81db400640 1 -- 172.21.15.118:0/659686424 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f81c4006290 con 0x5622651a13f0 2024-03-24T01:25:16.309 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T01:25:16.308+0000 7f81ee000640 1 -- 172.21.15.118:0/1693767697 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f81c80065b0 con 0x7f81f81052d0 2024-03-24T01:25:16.309 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T01:25:16.309+0000 7f81e4c00640 1 -- 172.21.15.118:0/659686424 <== mon.1 v2:172.21.15.193:3300/0 32 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 409+0+0 (secure 0 0 0) 0x7f81c4006290 con 0x5622651a13f0 2024-03-24T01:25:16.310 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T01:25:16.309+0000 7f81ef400640 1 -- 172.21.15.118:0/1693767697 <== mon.2 v2:172.21.15.189:3300/0 32 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (secure 0 0 0) 0x7f81c80065b0 con 0x7f81f81052d0 2024-03-24T01:25:16.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:25:16 smithi189 ceph-mon[36187]: Adjusting osd_memory_target on smithi193 to 174.1M 2024-03-24T01:25:16.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:25:16 smithi189 ceph-mon[36187]: Unable to set osd_memory_target on smithi193 to 182639923: error parsing value: Value '182639923' is below minimum 939524096 2024-03-24T01:25:16.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:25:16 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 3611.666311 secs 2024-03-24T01:25:16.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:25:16 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T01:25:16.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:25:16 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T01:25:16.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:25:16 smithi189 ceph-mon[36187]: pgmap v2593: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 6 op/s 2024-03-24T01:25:16.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:25:16 smithi189 ceph-mon[36187]: pgmap v2594: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 6 op/s 2024-03-24T01:25:16.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:25:16 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T01:25:16.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:25:16 smithi193 ceph-mon[36878]: Adjusting osd_memory_target on smithi193 to 174.1M 2024-03-24T01:25:16.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:25:16 smithi193 ceph-mon[36878]: Unable to set osd_memory_target on smithi193 to 182639923: error parsing value: Value '182639923' is below minimum 939524096 2024-03-24T01:25:16.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:25:16 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 3611.666311 secs 2024-03-24T01:25:16.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:25:16 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T01:25:16.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:25:16 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T01:25:16.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:25:16 smithi193 ceph-mon[36878]: pgmap v2593: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 6 op/s 2024-03-24T01:25:16.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:25:16 smithi193 ceph-mon[36878]: pgmap v2594: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 6 op/s 2024-03-24T01:25:16.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:25:16 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T01:25:18.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:25:18 smithi118 ceph-mon[31535]: pgmap v2595: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 5 op/s 2024-03-24T01:25:18.704 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:25:18 smithi189 ceph-mon[36187]: pgmap v2595: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 5 op/s 2024-03-24T01:25:18.711 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:25:18 smithi193 ceph-mon[36878]: pgmap v2595: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 5 op/s 2024-03-24T01:25:20.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:25:20 smithi118 ceph-mon[31535]: pgmap v2596: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 6 op/s 2024-03-24T01:25:20.704 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:25:20 smithi189 ceph-mon[36187]: pgmap v2596: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 6 op/s 2024-03-24T01:25:20.711 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:25:20 smithi193 ceph-mon[36878]: pgmap v2596: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 6 op/s 2024-03-24T01:25:21.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:25:21 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 3616.666400 secs 2024-03-24T01:25:21.704 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:25:21 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 3616.666400 secs 2024-03-24T01:25:21.710 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:25:21 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 3616.666400 secs 2024-03-24T01:25:22.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:25:22 smithi118 ceph-mon[31535]: pgmap v2597: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 6 op/s 2024-03-24T01:25:22.704 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:25:22 smithi189 ceph-mon[36187]: pgmap v2597: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 6 op/s 2024-03-24T01:25:22.711 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:25:22 smithi193 ceph-mon[36878]: pgmap v2597: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 6 op/s 2024-03-24T01:25:24.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:25:24 smithi118 ceph-mon[31535]: pgmap v2598: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 6 op/s 2024-03-24T01:25:24.704 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:25:24 smithi189 ceph-mon[36187]: pgmap v2598: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 6 op/s 2024-03-24T01:25:24.710 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:25:24 smithi193 ceph-mon[36878]: pgmap v2598: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 6 op/s 2024-03-24T01:25:26.704 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:25:26 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 3621.666460 secs 2024-03-24T01:25:26.704 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:25:26 smithi189 ceph-mon[36187]: pgmap v2599: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 5 op/s 2024-03-24T01:25:26.711 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:25:26 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 3621.666460 secs 2024-03-24T01:25:26.711 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:25:26 smithi193 ceph-mon[36878]: pgmap v2599: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 5 op/s 2024-03-24T01:25:26.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:25:26 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 3621.666460 secs 2024-03-24T01:25:26.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:25:26 smithi118 ceph-mon[31535]: pgmap v2599: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 5 op/s 2024-03-24T01:25:28.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:25:28 smithi118 ceph-mon[31535]: pgmap v2600: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T01:25:28.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:25:28 smithi189 ceph-mon[36187]: pgmap v2600: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T01:25:28.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:25:28 smithi193 ceph-mon[36878]: pgmap v2600: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T01:25:30.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:25:30 smithi118 ceph-mon[31535]: pgmap v2601: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-03-24T01:25:30.953 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:25:30 smithi189 ceph-mon[36187]: pgmap v2601: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-03-24T01:25:30.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:25:30 smithi193 ceph-mon[36878]: pgmap v2601: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-03-24T01:25:31.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:25:31 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 3626.666572 secs 2024-03-24T01:25:31.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:25:31 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 3626.666572 secs 2024-03-24T01:25:31.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:25:31 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 3626.666572 secs 2024-03-24T01:25:32.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:25:32 smithi118 ceph-mon[31535]: pgmap v2602: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:25:32.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:25:32 smithi189 ceph-mon[36187]: pgmap v2602: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:25:32.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:25:32 smithi193 ceph-mon[36878]: pgmap v2602: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:25:34.872 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:25:34 smithi118 ceph-mon[31535]: pgmap v2603: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T01:25:34.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:25:34 smithi189 ceph-mon[36187]: pgmap v2603: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T01:25:34.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:25:34 smithi193 ceph-mon[36878]: pgmap v2603: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T01:25:36.949 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:25:36 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 3631.666612 secs 2024-03-24T01:25:36.950 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:25:36 smithi118 ceph-mon[31535]: pgmap v2604: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T01:25:36.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:25:36 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 3631.666612 secs 2024-03-24T01:25:36.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:25:36 smithi189 ceph-mon[36187]: pgmap v2604: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T01:25:36.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:25:36 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 3631.666612 secs 2024-03-24T01:25:36.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:25:36 smithi193 ceph-mon[36878]: pgmap v2604: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T01:25:38.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:25:38 smithi118 ceph-mon[31535]: pgmap v2605: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T01:25:38.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:25:38 smithi189 ceph-mon[36187]: pgmap v2605: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T01:25:38.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:25:38 smithi193 ceph-mon[36878]: pgmap v2605: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T01:25:40.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:25:40 smithi118 ceph-mon[31535]: pgmap v2606: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-03-24T01:25:40.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:25:40 smithi189 ceph-mon[36187]: pgmap v2606: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-03-24T01:25:40.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:25:40 smithi193 ceph-mon[36878]: pgmap v2606: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-03-24T01:25:41.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:25:41 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 3636.666671 secs 2024-03-24T01:25:41.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:25:41 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 3636.666671 secs 2024-03-24T01:25:41.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:25:41 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 3636.666671 secs 2024-03-24T01:25:42.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:25:42 smithi118 ceph-mon[31535]: pgmap v2607: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:25:42.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:25:42 smithi189 ceph-mon[36187]: pgmap v2607: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:25:42.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:25:42 smithi193 ceph-mon[36878]: pgmap v2607: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:25:44.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:25:44 smithi118 ceph-mon[31535]: pgmap v2608: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T01:25:44.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:25:44 smithi189 ceph-mon[36187]: pgmap v2608: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T01:25:44.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:25:44 smithi193 ceph-mon[36878]: pgmap v2608: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T01:25:46.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:25:46 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 3641.666759 secs 2024-03-24T01:25:46.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:25:46 smithi118 ceph-mon[31535]: pgmap v2609: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T01:25:46.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:25:46 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 3641.666759 secs 2024-03-24T01:25:46.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:25:46 smithi189 ceph-mon[36187]: pgmap v2609: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T01:25:46.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:25:46 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 3641.666759 secs 2024-03-24T01:25:46.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:25:46 smithi193 ceph-mon[36878]: pgmap v2609: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T01:25:48.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:25:48 smithi118 ceph-mon[31535]: pgmap v2610: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T01:25:48.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:25:48 smithi189 ceph-mon[36187]: pgmap v2610: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T01:25:48.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:25:48 smithi193 ceph-mon[36878]: pgmap v2610: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T01:25:50.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:25:50 smithi118 ceph-mon[31535]: pgmap v2611: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-03-24T01:25:50.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:25:50 smithi189 ceph-mon[36187]: pgmap v2611: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-03-24T01:25:50.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:25:50 smithi193 ceph-mon[36878]: pgmap v2611: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-03-24T01:25:51.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:25:51 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 3646.666846 secs 2024-03-24T01:25:51.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:25:51 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 3646.666846 secs 2024-03-24T01:25:51.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:25:51 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 3646.666846 secs 2024-03-24T01:25:52.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:25:52 smithi118 ceph-mon[31535]: pgmap v2612: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:25:52.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:25:52 smithi189 ceph-mon[36187]: pgmap v2612: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:25:52.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:25:52 smithi193 ceph-mon[36878]: pgmap v2612: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:25:54.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:25:54 smithi118 ceph-mon[31535]: pgmap v2613: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T01:25:54.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:25:54 smithi189 ceph-mon[36187]: pgmap v2613: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T01:25:54.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:25:54 smithi193 ceph-mon[36878]: pgmap v2613: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T01:25:55.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:25:55 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 3651.666904 secs 2024-03-24T01:25:55.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:25:55 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 3651.666904 secs 2024-03-24T01:25:55.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:25:55 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 3651.666904 secs 2024-03-24T01:25:56.855 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:25:56 smithi189 ceph-mon[36187]: pgmap v2614: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T01:25:56.862 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:25:56 smithi193 ceph-mon[36878]: pgmap v2614: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T01:25:56.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:25:56 smithi118 ceph-mon[31535]: pgmap v2614: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T01:25:58.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:25:58 smithi118 ceph-mon[31535]: pgmap v2615: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T01:25:58.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:25:58 smithi189 ceph-mon[36187]: pgmap v2615: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T01:25:58.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:25:58 smithi193 ceph-mon[36878]: pgmap v2615: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T01:26:00.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:26:00 smithi118 ceph-mon[31535]: pgmap v2616: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-03-24T01:26:00.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:26:00 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 3656.666994 secs 2024-03-24T01:26:00.953 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:26:00 smithi189 ceph-mon[36187]: pgmap v2616: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-03-24T01:26:00.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:26:00 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 3656.666994 secs 2024-03-24T01:26:00.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:26:00 smithi193 ceph-mon[36878]: pgmap v2616: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-03-24T01:26:00.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:26:00 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 3656.666994 secs 2024-03-24T01:26:02.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:26:02 smithi118 ceph-mon[31535]: pgmap v2617: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:26:02.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:26:02 smithi189 ceph-mon[36187]: pgmap v2617: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:26:02.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:26:02 smithi193 ceph-mon[36878]: pgmap v2617: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:26:04.947 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:26:04 smithi118 ceph-mon[31535]: pgmap v2618: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T01:26:04.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:26:04 smithi189 ceph-mon[36187]: pgmap v2618: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T01:26:04.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:26:04 smithi193 ceph-mon[36878]: pgmap v2618: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T01:26:05.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:26:05 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 3661.667083 secs 2024-03-24T01:26:05.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:26:05 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 3661.667083 secs 2024-03-24T01:26:05.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:26:05 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 3661.667083 secs 2024-03-24T01:26:06.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:26:06 smithi118 ceph-mon[31535]: pgmap v2619: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T01:26:06.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:26:06 smithi189 ceph-mon[36187]: pgmap v2619: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T01:26:06.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:26:06 smithi193 ceph-mon[36878]: pgmap v2619: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T01:26:08.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:26:08 smithi118 ceph-mon[31535]: pgmap v2620: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T01:26:08.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:26:08 smithi189 ceph-mon[36187]: pgmap v2620: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T01:26:08.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:26:08 smithi193 ceph-mon[36878]: pgmap v2620: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T01:26:10.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:26:10 smithi118 ceph-mon[31535]: pgmap v2621: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-03-24T01:26:10.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:26:10 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 3666.667170 secs 2024-03-24T01:26:10.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:26:10 smithi189 ceph-mon[36187]: pgmap v2621: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-03-24T01:26:10.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:26:10 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 3666.667170 secs 2024-03-24T01:26:10.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:26:10 smithi193 ceph-mon[36878]: pgmap v2621: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-03-24T01:26:10.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:26:10 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 3666.667170 secs 2024-03-24T01:26:12.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:26:12 smithi118 ceph-mon[31535]: pgmap v2622: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-03-24T01:26:12.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:26:12 smithi189 ceph-mon[36187]: pgmap v2622: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-03-24T01:26:12.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:26:12 smithi193 ceph-mon[36878]: pgmap v2622: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-03-24T01:26:14.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:26:14 smithi118 ceph-mon[31535]: pgmap v2623: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-03-24T01:26:14.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:26:14 smithi189 ceph-mon[36187]: pgmap v2623: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-03-24T01:26:14.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:26:14 smithi193 ceph-mon[36878]: pgmap v2623: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-03-24T01:26:15.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:26:15 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 3671.667253 secs 2024-03-24T01:26:15.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:26:15 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 3671.667253 secs 2024-03-24T01:26:15.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:26:15 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 3671.667253 secs 2024-03-24T01:26:16.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:26:16 smithi118 ceph-mon[31535]: pgmap v2624: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 6 op/s 2024-03-24T01:26:16.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:26:16 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T01:26:16.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:26:16 smithi189 ceph-mon[36187]: pgmap v2624: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 6 op/s 2024-03-24T01:26:16.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:26:16 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T01:26:16.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:26:16 smithi193 ceph-mon[36878]: pgmap v2624: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 6 op/s 2024-03-24T01:26:16.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:26:16 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T01:26:17.880 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:26:17 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T01:26:17.880 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:26:17 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T01:26:17.881 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:26:17 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T01:26:17.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:26:17 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T01:26:17.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:26:17 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T01:26:17.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:26:17 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T01:26:17.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:26:17 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T01:26:17.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:26:17 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T01:26:17.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:26:17 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T01:26:18.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:26:18 smithi118 ceph-mon[31535]: pgmap v2625: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-03-24T01:26:18.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:26:18 smithi118 ceph-mon[31535]: pgmap v2626: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-03-24T01:26:18.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:26:18 smithi189 ceph-mon[36187]: pgmap v2625: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-03-24T01:26:18.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:26:18 smithi189 ceph-mon[36187]: pgmap v2626: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-03-24T01:26:18.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:26:18 smithi193 ceph-mon[36878]: pgmap v2625: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-03-24T01:26:18.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:26:18 smithi193 ceph-mon[36878]: pgmap v2626: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-03-24T01:26:20.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:26:20 smithi118 ceph-mon[31535]: pgmap v2627: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 5 op/s 2024-03-24T01:26:20.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:26:20 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 3676.667339 secs 2024-03-24T01:26:20.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:26:20 smithi189 ceph-mon[36187]: pgmap v2627: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 5 op/s 2024-03-24T01:26:20.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:26:20 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 3676.667339 secs 2024-03-24T01:26:20.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:26:20 smithi193 ceph-mon[36878]: pgmap v2627: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 5 op/s 2024-03-24T01:26:20.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:26:20 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 3676.667339 secs 2024-03-24T01:26:22.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:26:22 smithi118 ceph-mon[31535]: pgmap v2628: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 6 op/s 2024-03-24T01:26:22.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:26:22 smithi189 ceph-mon[36187]: pgmap v2628: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 6 op/s 2024-03-24T01:26:22.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:26:22 smithi193 ceph-mon[36878]: pgmap v2628: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 6 op/s 2024-03-24T01:26:24.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:26:24 smithi118 ceph-mon[31535]: pgmap v2629: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 5 op/s 2024-03-24T01:26:24.953 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:26:24 smithi189 ceph-mon[36187]: pgmap v2629: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 5 op/s 2024-03-24T01:26:24.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:26:24 smithi193 ceph-mon[36878]: pgmap v2629: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 5 op/s 2024-03-24T01:26:25.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:26:25 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 3681.667423 secs 2024-03-24T01:26:25.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:26:25 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 3681.667423 secs 2024-03-24T01:26:25.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:26:25 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 3681.667423 secs 2024-03-24T01:26:26.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:26:26 smithi118 ceph-mon[31535]: pgmap v2630: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 6 op/s 2024-03-24T01:26:26.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:26:26 smithi189 ceph-mon[36187]: pgmap v2630: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 6 op/s 2024-03-24T01:26:26.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:26:26 smithi193 ceph-mon[36878]: pgmap v2630: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 6 op/s 2024-03-24T01:26:28.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:26:28 smithi118 ceph-mon[31535]: pgmap v2631: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 5 op/s 2024-03-24T01:26:28.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:26:28 smithi189 ceph-mon[36187]: pgmap v2631: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 5 op/s 2024-03-24T01:26:28.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:26:28 smithi193 ceph-mon[36878]: pgmap v2631: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 5 op/s 2024-03-24T01:26:30.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:26:30 smithi118 ceph-mon[31535]: pgmap v2632: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T01:26:30.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:26:30 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 3686.667482 secs 2024-03-24T01:26:30.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:26:30 smithi189 ceph-mon[36187]: pgmap v2632: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T01:26:30.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:26:30 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 3686.667482 secs 2024-03-24T01:26:30.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:26:30 smithi193 ceph-mon[36878]: pgmap v2632: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T01:26:30.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:26:30 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 3686.667482 secs 2024-03-24T01:26:32.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:26:32 smithi118 ceph-mon[31535]: pgmap v2633: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T01:26:32.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:26:32 smithi189 ceph-mon[36187]: pgmap v2633: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T01:26:32.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:26:32 smithi193 ceph-mon[36878]: pgmap v2633: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T01:26:33.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:26:33 smithi118 ceph-mon[31535]: pgmap v2634: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 5 op/s 2024-03-24T01:26:33.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:26:33 smithi189 ceph-mon[36187]: pgmap v2634: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 5 op/s 2024-03-24T01:26:33.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:26:33 smithi193 ceph-mon[36878]: pgmap v2634: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 5 op/s 2024-03-24T01:26:36.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:26:36 smithi118 ceph-mon[31535]: pgmap v2635: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T01:26:36.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:26:36 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 3691.667523 secs 2024-03-24T01:26:36.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:26:36 smithi189 ceph-mon[36187]: pgmap v2635: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T01:26:36.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:26:36 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 3691.667523 secs 2024-03-24T01:26:36.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:26:36 smithi193 ceph-mon[36878]: pgmap v2635: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T01:26:36.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:26:36 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 3691.667523 secs 2024-03-24T01:26:38.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:26:38 smithi118 ceph-mon[31535]: pgmap v2636: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:26:38.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:26:38 smithi189 ceph-mon[36187]: pgmap v2636: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:26:38.460 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:26:38 smithi193 ceph-mon[36878]: pgmap v2636: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:26:40.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:26:40 smithi118 ceph-mon[31535]: pgmap v2637: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T01:26:40.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:26:40 smithi189 ceph-mon[36187]: pgmap v2637: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T01:26:40.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:26:40 smithi193 ceph-mon[36878]: pgmap v2637: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T01:26:41.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:26:41 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 3696.667535 secs 2024-03-24T01:26:41.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:26:41 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 3696.667535 secs 2024-03-24T01:26:41.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:26:41 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 3696.667535 secs 2024-03-24T01:26:42.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:26:42 smithi118 ceph-mon[31535]: pgmap v2638: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T01:26:42.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:26:42 smithi189 ceph-mon[36187]: pgmap v2638: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T01:26:42.460 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:26:42 smithi193 ceph-mon[36878]: pgmap v2638: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T01:26:44.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:26:44 smithi118 ceph-mon[31535]: pgmap v2639: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 5 op/s 2024-03-24T01:26:44.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:26:44 smithi189 ceph-mon[36187]: pgmap v2639: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 5 op/s 2024-03-24T01:26:44.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:26:44 smithi193 ceph-mon[36878]: pgmap v2639: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 5 op/s 2024-03-24T01:26:46.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:26:46 smithi118 ceph-mon[31535]: pgmap v2640: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T01:26:46.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:26:46 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 3701.667626 secs 2024-03-24T01:26:46.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:26:46 smithi189 ceph-mon[36187]: pgmap v2640: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T01:26:46.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:26:46 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 3701.667626 secs 2024-03-24T01:26:46.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:26:46 smithi193 ceph-mon[36878]: pgmap v2640: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T01:26:46.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:26:46 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 3701.667626 secs 2024-03-24T01:26:48.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:26:48 smithi118 ceph-mon[31535]: pgmap v2641: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:26:48.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:26:48 smithi189 ceph-mon[36187]: pgmap v2641: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:26:48.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:26:48 smithi193 ceph-mon[36878]: pgmap v2641: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:26:50.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:26:50 smithi118 ceph-mon[31535]: pgmap v2642: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T01:26:50.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:26:50 smithi189 ceph-mon[36187]: pgmap v2642: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T01:26:50.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:26:50 smithi193 ceph-mon[36878]: pgmap v2642: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T01:26:51.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:26:51 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 3706.667680 secs 2024-03-24T01:26:51.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:26:51 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 3706.667680 secs 2024-03-24T01:26:51.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:26:51 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 3706.667680 secs 2024-03-24T01:26:52.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:26:52 smithi118 ceph-mon[31535]: pgmap v2643: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T01:26:52.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:26:52 smithi189 ceph-mon[36187]: pgmap v2643: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T01:26:52.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:26:52 smithi193 ceph-mon[36878]: pgmap v2643: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T01:26:54.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:26:54 smithi118 ceph-mon[31535]: pgmap v2644: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 5 op/s 2024-03-24T01:26:54.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:26:54 smithi189 ceph-mon[36187]: pgmap v2644: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 5 op/s 2024-03-24T01:26:54.460 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:26:54 smithi193 ceph-mon[36878]: pgmap v2644: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 5 op/s 2024-03-24T01:26:56.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:26:56 smithi118 ceph-mon[31535]: pgmap v2645: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T01:26:56.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:26:56 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 3711.667742 secs 2024-03-24T01:26:56.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:26:56 smithi189 ceph-mon[36187]: pgmap v2645: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T01:26:56.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:26:56 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 3711.667742 secs 2024-03-24T01:26:56.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:26:56 smithi193 ceph-mon[36878]: pgmap v2645: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T01:26:56.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:26:56 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 3711.667742 secs 2024-03-24T01:26:58.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:26:58 smithi118 ceph-mon[31535]: pgmap v2646: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:26:58.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:26:58 smithi189 ceph-mon[36187]: pgmap v2646: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:26:58.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:26:58 smithi193 ceph-mon[36878]: pgmap v2646: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:27:00.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:27:00 smithi118 ceph-mon[31535]: pgmap v2647: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T01:27:00.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:27:00 smithi189 ceph-mon[36187]: pgmap v2647: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T01:27:00.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:27:00 smithi193 ceph-mon[36878]: pgmap v2647: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T01:27:01.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:27:01 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 3716.667828 secs 2024-03-24T01:27:01.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:27:01 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 3716.667828 secs 2024-03-24T01:27:01.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:27:01 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 3716.667828 secs 2024-03-24T01:27:02.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:27:02 smithi118 ceph-mon[31535]: pgmap v2648: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T01:27:02.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:27:02 smithi189 ceph-mon[36187]: pgmap v2648: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T01:27:02.460 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:27:02 smithi193 ceph-mon[36878]: pgmap v2648: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T01:27:04.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:27:04 smithi118 ceph-mon[31535]: pgmap v2649: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 5 op/s 2024-03-24T01:27:04.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:27:04 smithi189 ceph-mon[36187]: pgmap v2649: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 5 op/s 2024-03-24T01:27:04.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:27:04 smithi193 ceph-mon[36878]: pgmap v2649: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 5 op/s 2024-03-24T01:27:06.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:27:06 smithi118 ceph-mon[31535]: pgmap v2650: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T01:27:06.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:27:06 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 3721.667916 secs 2024-03-24T01:27:06.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:27:06 smithi189 ceph-mon[36187]: pgmap v2650: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T01:27:06.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:27:06 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 3721.667916 secs 2024-03-24T01:27:06.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:27:06 smithi193 ceph-mon[36878]: pgmap v2650: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T01:27:06.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:27:06 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 3721.667916 secs 2024-03-24T01:27:08.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:27:08 smithi118 ceph-mon[31535]: pgmap v2651: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:27:08.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:27:08 smithi189 ceph-mon[36187]: pgmap v2651: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:27:08.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:27:08 smithi193 ceph-mon[36878]: pgmap v2651: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:27:10.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:27:10 smithi118 ceph-mon[31535]: pgmap v2652: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T01:27:10.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:27:10 smithi189 ceph-mon[36187]: pgmap v2652: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T01:27:10.460 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:27:10 smithi193 ceph-mon[36878]: pgmap v2652: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T01:27:11.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:27:11 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 3726.667999 secs 2024-03-24T01:27:11.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:27:11 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 3726.667999 secs 2024-03-24T01:27:11.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:27:11 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 3726.667999 secs 2024-03-24T01:27:12.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:27:12 smithi118 ceph-mon[31535]: pgmap v2653: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T01:27:12.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:27:12 smithi189 ceph-mon[36187]: pgmap v2653: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T01:27:12.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:27:12 smithi193 ceph-mon[36878]: pgmap v2653: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T01:27:14.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:27:14 smithi118 ceph-mon[31535]: pgmap v2654: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 5 op/s 2024-03-24T01:27:14.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:27:14 smithi189 ceph-mon[36187]: pgmap v2654: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 5 op/s 2024-03-24T01:27:14.460 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:27:14 smithi193 ceph-mon[36878]: pgmap v2654: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 5 op/s 2024-03-24T01:27:16.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:27:16 smithi118 ceph-mon[31535]: pgmap v2655: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T01:27:16.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:27:16 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 3731.668084 secs 2024-03-24T01:27:16.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:27:16 smithi189 ceph-mon[36187]: pgmap v2655: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T01:27:16.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:27:16 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 3731.668084 secs 2024-03-24T01:27:16.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:27:16 smithi193 ceph-mon[36878]: pgmap v2655: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T01:27:16.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:27:16 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 3731.668084 secs 2024-03-24T01:27:18.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:27:18 smithi118 ceph-mon[31535]: pgmap v2656: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:27:18.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:27:18 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T01:27:18.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:27:18 smithi189 ceph-mon[36187]: pgmap v2656: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:27:18.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:27:18 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T01:27:18.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:27:18 smithi193 ceph-mon[36878]: pgmap v2656: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:27:18.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:27:18 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T01:27:19.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:27:19 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T01:27:19.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:27:19 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T01:27:19.447 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:27:19 smithi118 ceph-mon[31535]: pgmap v2657: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 6 op/s 2024-03-24T01:27:19.447 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:27:19 smithi118 ceph-mon[31535]: pgmap v2658: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 5 op/s 2024-03-24T01:27:19.447 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:27:19 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T01:27:19.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:27:19 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T01:27:19.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:27:19 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T01:27:19.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:27:19 smithi189 ceph-mon[36187]: pgmap v2657: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 6 op/s 2024-03-24T01:27:19.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:27:19 smithi189 ceph-mon[36187]: pgmap v2658: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 5 op/s 2024-03-24T01:27:19.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:27:19 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T01:27:19.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:27:19 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T01:27:19.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:27:19 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T01:27:19.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:27:19 smithi193 ceph-mon[36878]: pgmap v2657: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 6 op/s 2024-03-24T01:27:19.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:27:19 smithi193 ceph-mon[36878]: pgmap v2658: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 5 op/s 2024-03-24T01:27:19.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:27:19 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T01:27:20.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:27:20 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 3736.668164 secs 2024-03-24T01:27:20.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:27:20 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 3736.668164 secs 2024-03-24T01:27:20.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:27:20 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 3736.668164 secs 2024-03-24T01:27:21.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:27:21 smithi118 ceph-mon[31535]: pgmap v2659: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 6 op/s 2024-03-24T01:27:21.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:27:21 smithi189 ceph-mon[36187]: pgmap v2659: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 6 op/s 2024-03-24T01:27:21.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:27:21 smithi193 ceph-mon[36878]: pgmap v2659: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 6 op/s 2024-03-24T01:27:23.929 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:27:23 smithi118 ceph-mon[31535]: pgmap v2660: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 5 op/s 2024-03-24T01:27:23.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:27:23 smithi189 ceph-mon[36187]: pgmap v2660: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 5 op/s 2024-03-24T01:27:23.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:27:23 smithi193 ceph-mon[36878]: pgmap v2660: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 5 op/s 2024-03-24T01:27:25.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:27:25 smithi118 ceph-mon[31535]: pgmap v2661: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 6 op/s 2024-03-24T01:27:25.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:27:25 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 3741.668253 secs 2024-03-24T01:27:25.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:27:25 smithi189 ceph-mon[36187]: pgmap v2661: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 6 op/s 2024-03-24T01:27:25.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:27:25 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 3741.668253 secs 2024-03-24T01:27:25.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:27:25 smithi193 ceph-mon[36878]: pgmap v2661: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 6 op/s 2024-03-24T01:27:25.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:27:25 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 3741.668253 secs 2024-03-24T01:27:27.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:27:27 smithi118 ceph-mon[31535]: pgmap v2662: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 6 op/s 2024-03-24T01:27:27.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:27:27 smithi189 ceph-mon[36187]: pgmap v2662: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 6 op/s 2024-03-24T01:27:27.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:27:27 smithi193 ceph-mon[36878]: pgmap v2662: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 6 op/s 2024-03-24T01:27:29.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:27:29 smithi118 ceph-mon[31535]: pgmap v2663: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 5 op/s 2024-03-24T01:27:29.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:27:29 smithi189 ceph-mon[36187]: pgmap v2663: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 5 op/s 2024-03-24T01:27:29.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:27:29 smithi193 ceph-mon[36878]: pgmap v2663: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 5 op/s 2024-03-24T01:27:30.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:27:30 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 3746.668342 secs 2024-03-24T01:27:30.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:27:30 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 3746.668342 secs 2024-03-24T01:27:30.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:27:30 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 3746.668342 secs 2024-03-24T01:27:31.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:27:31 smithi118 ceph-mon[31535]: pgmap v2664: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 6 op/s 2024-03-24T01:27:31.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:27:31 smithi189 ceph-mon[36187]: pgmap v2664: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 6 op/s 2024-03-24T01:27:31.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:27:31 smithi193 ceph-mon[36878]: pgmap v2664: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 6 op/s 2024-03-24T01:27:33.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:27:33 smithi118 ceph-mon[31535]: pgmap v2665: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 5 op/s 2024-03-24T01:27:33.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:27:33 smithi189 ceph-mon[36187]: pgmap v2665: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 5 op/s 2024-03-24T01:27:33.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:27:33 smithi193 ceph-mon[36878]: pgmap v2665: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 5 op/s 2024-03-24T01:27:35.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:27:35 smithi118 ceph-mon[31535]: pgmap v2666: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-03-24T01:27:35.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:27:35 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 3751.668404 secs 2024-03-24T01:27:35.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:27:35 smithi189 ceph-mon[36187]: pgmap v2666: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-03-24T01:27:35.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:27:35 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 3751.668404 secs 2024-03-24T01:27:35.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:27:35 smithi193 ceph-mon[36878]: pgmap v2666: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-03-24T01:27:35.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:27:35 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 3751.668404 secs 2024-03-24T01:27:37.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:27:37 smithi118 ceph-mon[31535]: pgmap v2667: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-03-24T01:27:37.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:27:37 smithi189 ceph-mon[36187]: pgmap v2667: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-03-24T01:27:37.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:27:37 smithi193 ceph-mon[36878]: pgmap v2667: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-03-24T01:27:39.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:27:39 smithi118 ceph-mon[31535]: pgmap v2668: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-03-24T01:27:39.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:27:39 smithi189 ceph-mon[36187]: pgmap v2668: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-03-24T01:27:39.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:27:39 smithi193 ceph-mon[36878]: pgmap v2668: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-03-24T01:27:40.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:27:40 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 3756.668490 secs 2024-03-24T01:27:40.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:27:40 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 3756.668490 secs 2024-03-24T01:27:40.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:27:40 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 3756.668490 secs 2024-03-24T01:27:41.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:27:41 smithi118 ceph-mon[31535]: pgmap v2669: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 6 op/s 2024-03-24T01:27:41.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:27:41 smithi189 ceph-mon[36187]: pgmap v2669: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 6 op/s 2024-03-24T01:27:41.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:27:41 smithi193 ceph-mon[36878]: pgmap v2669: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 6 op/s 2024-03-24T01:27:43.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:27:43 smithi118 ceph-mon[31535]: pgmap v2670: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T01:27:43.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:27:43 smithi189 ceph-mon[36187]: pgmap v2670: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T01:27:43.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:27:43 smithi193 ceph-mon[36878]: pgmap v2670: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T01:27:45.948 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:27:45 smithi118 ceph-mon[31535]: pgmap v2671: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-03-24T01:27:45.948 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:27:45 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 3761.668574 secs 2024-03-24T01:27:45.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:27:45 smithi189 ceph-mon[36187]: pgmap v2671: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-03-24T01:27:45.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:27:45 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 3761.668574 secs 2024-03-24T01:27:45.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:27:45 smithi193 ceph-mon[36878]: pgmap v2671: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-03-24T01:27:45.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:27:45 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 3761.668574 secs 2024-03-24T01:27:47.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:27:47 smithi118 ceph-mon[31535]: pgmap v2672: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:27:47.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:27:47 smithi189 ceph-mon[36187]: pgmap v2672: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:27:47.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:27:47 smithi193 ceph-mon[36878]: pgmap v2672: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:27:49.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:27:49 smithi118 ceph-mon[31535]: pgmap v2673: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T01:27:49.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:27:49 smithi189 ceph-mon[36187]: pgmap v2673: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T01:27:49.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:27:49 smithi193 ceph-mon[36878]: pgmap v2673: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T01:27:50.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:27:50 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 3766.668636 secs 2024-03-24T01:27:50.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:27:50 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 3766.668636 secs 2024-03-24T01:27:50.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:27:50 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 3766.668636 secs 2024-03-24T01:27:51.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:27:51 smithi118 ceph-mon[31535]: pgmap v2674: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T01:27:51.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:27:51 smithi189 ceph-mon[36187]: pgmap v2674: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T01:27:51.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:27:51 smithi193 ceph-mon[36878]: pgmap v2674: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T01:27:52.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:27:52 smithi189 ceph-mon[36187]: pgmap v2675: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T01:27:52.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:27:52 smithi193 ceph-mon[36878]: pgmap v2675: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T01:27:53.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:27:52 smithi118 ceph-mon[31535]: pgmap v2675: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T01:27:55.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:27:55 smithi118 ceph-mon[31535]: pgmap v2676: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T01:27:55.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:27:55 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 3771.668677 secs 2024-03-24T01:27:55.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:27:55 smithi189 ceph-mon[36187]: pgmap v2676: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T01:27:55.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:27:55 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 3771.668677 secs 2024-03-24T01:27:55.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:27:55 smithi193 ceph-mon[36878]: pgmap v2676: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T01:27:55.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:27:55 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 3771.668677 secs 2024-03-24T01:27:57.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:27:57 smithi118 ceph-mon[31535]: pgmap v2677: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:27:57.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:27:57 smithi189 ceph-mon[36187]: pgmap v2677: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:27:57.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:27:57 smithi193 ceph-mon[36878]: pgmap v2677: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:27:59.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:27:59 smithi118 ceph-mon[31535]: pgmap v2678: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T01:27:59.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:27:59 smithi189 ceph-mon[36187]: pgmap v2678: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T01:27:59.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:27:59 smithi193 ceph-mon[36878]: pgmap v2678: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T01:28:00.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:28:00 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 3776.668765 secs 2024-03-24T01:28:00.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:28:00 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 3776.668765 secs 2024-03-24T01:28:00.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:28:00 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 3776.668765 secs 2024-03-24T01:28:01.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:28:01 smithi118 ceph-mon[31535]: pgmap v2679: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T01:28:01.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:28:01 smithi189 ceph-mon[36187]: pgmap v2679: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T01:28:01.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:28:01 smithi193 ceph-mon[36878]: pgmap v2679: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T01:28:03.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:28:03 smithi118 ceph-mon[31535]: pgmap v2680: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T01:28:03.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:28:03 smithi189 ceph-mon[36187]: pgmap v2680: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T01:28:03.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:28:03 smithi193 ceph-mon[36878]: pgmap v2680: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T01:28:05.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:28:04 smithi118 ceph-mon[31535]: pgmap v2681: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T01:28:05.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:28:04 smithi189 ceph-mon[36187]: pgmap v2681: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T01:28:05.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:28:04 smithi193 ceph-mon[36878]: pgmap v2681: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T01:28:06.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:28:05 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 3781.668826 secs 2024-03-24T01:28:06.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:28:05 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 3781.668826 secs 2024-03-24T01:28:06.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:28:05 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 3781.668826 secs 2024-03-24T01:28:07.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:28:06 smithi118 ceph-mon[31535]: pgmap v2682: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:28:07.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:28:06 smithi189 ceph-mon[36187]: pgmap v2682: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:28:07.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:28:06 smithi193 ceph-mon[36878]: pgmap v2682: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:28:09.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:28:09 smithi118 ceph-mon[31535]: pgmap v2683: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T01:28:09.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:28:09 smithi189 ceph-mon[36187]: pgmap v2683: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T01:28:09.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:28:09 smithi193 ceph-mon[36878]: pgmap v2683: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T01:28:10.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:28:10 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 3786.668912 secs 2024-03-24T01:28:10.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:28:10 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 3786.668912 secs 2024-03-24T01:28:10.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:28:10 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 3786.668912 secs 2024-03-24T01:28:11.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:28:11 smithi118 ceph-mon[31535]: pgmap v2684: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T01:28:11.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:28:11 smithi189 ceph-mon[36187]: pgmap v2684: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T01:28:11.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:28:11 smithi193 ceph-mon[36878]: pgmap v2684: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T01:28:13.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:28:13 smithi118 ceph-mon[31535]: pgmap v2685: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T01:28:13.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:28:13 smithi189 ceph-mon[36187]: pgmap v2685: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T01:28:13.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:28:13 smithi193 ceph-mon[36878]: pgmap v2685: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T01:28:15.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:28:15 smithi118 ceph-mon[31535]: pgmap v2686: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-03-24T01:28:15.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:28:15 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 3791.668927 secs 2024-03-24T01:28:15.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:28:15 smithi189 ceph-mon[36187]: pgmap v2686: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-03-24T01:28:15.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:28:15 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 3791.668927 secs 2024-03-24T01:28:15.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:28:15 smithi193 ceph-mon[36878]: pgmap v2686: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-03-24T01:28:15.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:28:15 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 3791.668927 secs 2024-03-24T01:28:16.339 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T01:28:16.339+0000 7f81ee000640 1 -- 172.21.15.118:0/1693767697 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f81c80065b0 con 0x7f81f81052d0 2024-03-24T01:28:16.339 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T01:28:16.339+0000 7f81db400640 1 -- 172.21.15.118:0/659686424 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f81c4006290 con 0x5622651a13f0 2024-03-24T01:28:16.340 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T01:28:16.339+0000 7f81ef400640 1 -- 172.21.15.118:0/1693767697 <== mon.2 v2:172.21.15.189:3300/0 33 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (secure 0 0 0) 0x7f81c80065b0 con 0x7f81f81052d0 2024-03-24T01:28:16.340 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T01:28:16.339+0000 7f81e4c00640 1 -- 172.21.15.118:0/659686424 <== mon.1 v2:172.21.15.193:3300/0 33 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 409+0+0 (secure 0 0 0) 0x7f81c4006290 con 0x5622651a13f0 2024-03-24T01:28:17.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:28:17 smithi118 ceph-mon[31535]: pgmap v2687: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:28:17.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:28:17 smithi189 ceph-mon[36187]: pgmap v2687: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:28:17.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:28:17 smithi193 ceph-mon[36878]: pgmap v2687: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:28:19.865 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:28:19 smithi193 ceph-mon[36878]: pgmap v2688: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T01:28:19.865 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:28:19 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T01:28:19.865 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:28:19 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T01:28:19.865 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:28:19 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T01:28:19.865 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:28:19 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T01:28:19.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:28:19 smithi118 ceph-mon[31535]: pgmap v2688: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T01:28:19.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:28:19 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T01:28:19.947 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:28:19 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T01:28:19.947 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:28:19 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T01:28:19.947 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:28:19 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T01:28:19.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:28:19 smithi189 ceph-mon[36187]: pgmap v2688: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T01:28:19.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:28:19 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T01:28:19.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:28:19 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T01:28:19.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:28:19 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T01:28:19.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:28:19 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T01:28:20.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:28:20 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T01:28:20.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:28:20 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 3796.668966 secs 2024-03-24T01:28:20.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:28:20 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T01:28:20.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:28:20 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 3796.668966 secs 2024-03-24T01:28:20.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:28:20 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T01:28:20.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:28:20 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 3796.668966 secs 2024-03-24T01:28:21.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:28:21 smithi118 ceph-mon[31535]: pgmap v2689: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T01:28:21.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:28:21 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T01:28:21.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:28:21 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T01:28:21.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:28:21 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T01:28:21.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:28:21 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T01:28:21.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:28:21 smithi189 ceph-mon[36187]: pgmap v2689: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T01:28:21.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:28:21 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T01:28:21.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:28:21 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T01:28:21.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:28:21 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T01:28:21.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:28:21 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T01:28:21.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:28:21 smithi193 ceph-mon[36878]: pgmap v2689: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T01:28:21.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:28:21 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T01:28:21.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:28:21 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T01:28:21.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:28:21 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T01:28:21.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:28:21 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T01:28:23.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:28:23 smithi118 ceph-mon[31535]: pgmap v2690: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T01:28:23.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:28:23 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T01:28:23.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:28:23 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T01:28:23.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:28:23 smithi189 ceph-mon[36187]: pgmap v2690: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T01:28:23.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:28:23 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T01:28:23.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:28:23 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T01:28:23.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:28:23 smithi193 ceph-mon[36878]: pgmap v2690: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T01:28:23.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:28:23 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T01:28:23.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:28:23 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T01:28:25.888 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:28:25 smithi118 ceph-mon[31535]: pgmap v2691: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-03-24T01:28:25.889 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:28:25 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T01:28:25.889 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:28:25 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T01:28:25.889 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:28:25 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T01:28:25.889 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:28:25 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 3801.668979 secs 2024-03-24T01:28:25.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:28:25 smithi189 ceph-mon[36187]: pgmap v2691: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-03-24T01:28:25.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:28:25 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T01:28:25.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:28:25 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T01:28:25.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:28:25 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T01:28:25.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:28:25 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 3801.668979 secs 2024-03-24T01:28:25.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:28:25 smithi193 ceph-mon[36878]: pgmap v2691: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-03-24T01:28:25.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:28:25 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T01:28:25.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:28:25 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T01:28:25.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:28:25 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T01:28:25.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:28:25 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 3801.668979 secs 2024-03-24T01:28:26.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:28:26 smithi118 ceph-mon[31535]: pgmap v2692: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T01:28:26.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:28:26 smithi118 ceph-mon[31535]: pgmap v2693: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-03-24T01:28:26.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:28:26 smithi189 ceph-mon[36187]: pgmap v2692: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T01:28:26.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:28:26 smithi189 ceph-mon[36187]: pgmap v2693: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-03-24T01:28:26.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:28:26 smithi193 ceph-mon[36878]: pgmap v2692: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T01:28:26.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:28:26 smithi193 ceph-mon[36878]: pgmap v2693: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-03-24T01:28:28.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:28:28 smithi118 ceph-mon[31535]: pgmap v2694: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 5 op/s 2024-03-24T01:28:28.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:28:28 smithi189 ceph-mon[36187]: pgmap v2694: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 5 op/s 2024-03-24T01:28:28.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:28:28 smithi193 ceph-mon[36878]: pgmap v2694: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 5 op/s 2024-03-24T01:28:30.947 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:28:30 smithi118 ceph-mon[31535]: pgmap v2695: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 5 op/s 2024-03-24T01:28:30.947 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:28:30 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 3806.669038 secs 2024-03-24T01:28:30.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:28:30 smithi189 ceph-mon[36187]: pgmap v2695: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 5 op/s 2024-03-24T01:28:30.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:28:30 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 3806.669038 secs 2024-03-24T01:28:30.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:28:30 smithi193 ceph-mon[36878]: pgmap v2695: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 5 op/s 2024-03-24T01:28:30.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:28:30 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 3806.669038 secs 2024-03-24T01:28:32.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:28:32 smithi118 ceph-mon[31535]: pgmap v2696: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 6 op/s 2024-03-24T01:28:32.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:28:32 smithi189 ceph-mon[36187]: pgmap v2696: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 6 op/s 2024-03-24T01:28:32.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:28:32 smithi193 ceph-mon[36878]: pgmap v2696: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 6 op/s 2024-03-24T01:28:34.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:28:34 smithi118 ceph-mon[31535]: pgmap v2697: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 4 op/s 2024-03-24T01:28:34.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:28:34 smithi189 ceph-mon[36187]: pgmap v2697: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 4 op/s 2024-03-24T01:28:34.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:28:34 smithi193 ceph-mon[36878]: pgmap v2697: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 4 op/s 2024-03-24T01:28:35.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:28:35 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 3811.669098 secs 2024-03-24T01:28:35.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:28:35 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 3811.669098 secs 2024-03-24T01:28:35.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:28:35 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 3811.669098 secs 2024-03-24T01:28:36.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:28:36 smithi118 ceph-mon[31535]: pgmap v2698: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 5 op/s 2024-03-24T01:28:36.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:28:36 smithi189 ceph-mon[36187]: pgmap v2698: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 5 op/s 2024-03-24T01:28:36.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:28:36 smithi193 ceph-mon[36878]: pgmap v2698: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 5 op/s 2024-03-24T01:28:38.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:28:38 smithi118 ceph-mon[31535]: pgmap v2699: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:28:38.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:28:38 smithi189 ceph-mon[36187]: pgmap v2699: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:28:38.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:28:38 smithi193 ceph-mon[36878]: pgmap v2699: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:28:40.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:28:40 smithi118 ceph-mon[31535]: pgmap v2700: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T01:28:40.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:28:40 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 3816.669188 secs 2024-03-24T01:28:40.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:28:40 smithi189 ceph-mon[36187]: pgmap v2700: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T01:28:40.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:28:40 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 3816.669188 secs 2024-03-24T01:28:40.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:28:40 smithi193 ceph-mon[36878]: pgmap v2700: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T01:28:40.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:28:40 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 3816.669188 secs 2024-03-24T01:28:42.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:28:42 smithi118 ceph-mon[31535]: pgmap v2701: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T01:28:42.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:28:42 smithi189 ceph-mon[36187]: pgmap v2701: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T01:28:42.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:28:42 smithi193 ceph-mon[36878]: pgmap v2701: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T01:28:44.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:28:44 smithi118 ceph-mon[31535]: pgmap v2702: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 5 op/s 2024-03-24T01:28:44.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:28:44 smithi189 ceph-mon[36187]: pgmap v2702: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 5 op/s 2024-03-24T01:28:44.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:28:44 smithi193 ceph-mon[36878]: pgmap v2702: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 5 op/s 2024-03-24T01:28:45.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:28:45 smithi189 ceph-mon[36187]: pgmap v2703: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T01:28:45.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:28:45 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 3821.669251 secs 2024-03-24T01:28:45.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:28:45 smithi193 ceph-mon[36878]: pgmap v2703: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T01:28:45.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:28:45 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 3821.669251 secs 2024-03-24T01:28:46.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:28:45 smithi118 ceph-mon[31535]: pgmap v2703: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T01:28:46.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:28:45 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 3821.669251 secs 2024-03-24T01:28:48.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:28:47 smithi118 ceph-mon[31535]: pgmap v2704: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:28:48.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:28:47 smithi189 ceph-mon[36187]: pgmap v2704: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:28:48.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:28:47 smithi193 ceph-mon[36878]: pgmap v2704: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:28:50.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:28:49 smithi118 ceph-mon[31535]: pgmap v2705: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T01:28:50.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:28:49 smithi189 ceph-mon[36187]: pgmap v2705: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T01:28:50.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:28:49 smithi193 ceph-mon[36878]: pgmap v2705: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T01:28:51.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:28:50 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 3826.669336 secs 2024-03-24T01:28:51.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:28:50 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 3826.669336 secs 2024-03-24T01:28:51.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:28:50 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 3826.669336 secs 2024-03-24T01:28:52.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:28:51 smithi118 ceph-mon[31535]: pgmap v2706: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T01:28:52.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:28:51 smithi189 ceph-mon[36187]: pgmap v2706: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T01:28:52.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:28:51 smithi193 ceph-mon[36878]: pgmap v2706: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T01:28:54.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:28:53 smithi189 ceph-mon[36187]: pgmap v2707: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 5 op/s 2024-03-24T01:28:54.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:28:53 smithi193 ceph-mon[36878]: pgmap v2707: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 5 op/s 2024-03-24T01:28:54.447 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:28:53 smithi118 ceph-mon[31535]: pgmap v2707: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 5 op/s 2024-03-24T01:28:56.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:28:55 smithi189 ceph-mon[36187]: pgmap v2708: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T01:28:56.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:28:55 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 3831.669416 secs 2024-03-24T01:28:56.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:28:55 smithi193 ceph-mon[36878]: pgmap v2708: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T01:28:56.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:28:55 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 3831.669416 secs 2024-03-24T01:28:56.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:28:55 smithi118 ceph-mon[31535]: pgmap v2708: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T01:28:56.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:28:55 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 3831.669416 secs 2024-03-24T01:28:58.350 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:28:57 smithi193 ceph-mon[36878]: pgmap v2709: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:28:58.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:28:57 smithi118 ceph-mon[31535]: pgmap v2709: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:28:58.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:28:57 smithi189 ceph-mon[36187]: pgmap v2709: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:29:00.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:28:59 smithi118 ceph-mon[31535]: pgmap v2710: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T01:29:00.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:28:59 smithi189 ceph-mon[36187]: pgmap v2710: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T01:29:00.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:28:59 smithi193 ceph-mon[36878]: pgmap v2710: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T01:29:01.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:29:00 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 3836.669478 secs 2024-03-24T01:29:01.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:29:00 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 3836.669478 secs 2024-03-24T01:29:01.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:29:00 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 3836.669478 secs 2024-03-24T01:29:02.447 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:29:01 smithi118 ceph-mon[31535]: pgmap v2711: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T01:29:02.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:29:01 smithi189 ceph-mon[36187]: pgmap v2711: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T01:29:02.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:29:01 smithi193 ceph-mon[36878]: pgmap v2711: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T01:29:04.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:29:03 smithi118 ceph-mon[31535]: pgmap v2712: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 5 op/s 2024-03-24T01:29:04.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:29:03 smithi189 ceph-mon[36187]: pgmap v2712: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 5 op/s 2024-03-24T01:29:04.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:29:03 smithi193 ceph-mon[36878]: pgmap v2712: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 5 op/s 2024-03-24T01:29:06.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:29:05 smithi118 ceph-mon[31535]: pgmap v2713: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T01:29:06.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:29:05 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 3841.669520 secs 2024-03-24T01:29:06.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:29:05 smithi189 ceph-mon[36187]: pgmap v2713: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T01:29:06.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:29:05 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 3841.669520 secs 2024-03-24T01:29:06.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:29:05 smithi193 ceph-mon[36878]: pgmap v2713: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T01:29:06.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:29:05 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 3841.669520 secs 2024-03-24T01:29:08.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:29:07 smithi118 ceph-mon[31535]: pgmap v2714: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:29:08.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:29:07 smithi189 ceph-mon[36187]: pgmap v2714: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:29:08.460 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:29:07 smithi193 ceph-mon[36878]: pgmap v2714: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:29:10.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:29:10 smithi118 ceph-mon[31535]: pgmap v2715: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T01:29:10.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:29:10 smithi189 ceph-mon[36187]: pgmap v2715: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T01:29:10.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:29:10 smithi193 ceph-mon[36878]: pgmap v2715: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T01:29:11.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:29:11 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 3846.669609 secs 2024-03-24T01:29:11.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:29:11 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 3846.669609 secs 2024-03-24T01:29:11.460 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:29:11 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 3846.669609 secs 2024-03-24T01:29:12.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:29:12 smithi118 ceph-mon[31535]: pgmap v2716: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T01:29:12.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:29:12 smithi189 ceph-mon[36187]: pgmap v2716: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T01:29:12.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:29:12 smithi193 ceph-mon[36878]: pgmap v2716: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T01:29:14.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:29:14 smithi118 ceph-mon[31535]: pgmap v2717: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 5 op/s 2024-03-24T01:29:14.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:29:14 smithi189 ceph-mon[36187]: pgmap v2717: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 5 op/s 2024-03-24T01:29:14.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:29:14 smithi193 ceph-mon[36878]: pgmap v2717: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 5 op/s 2024-03-24T01:29:16.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:29:16 smithi118 ceph-mon[31535]: pgmap v2718: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T01:29:16.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:29:16 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 3851.669650 secs 2024-03-24T01:29:16.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:29:16 smithi189 ceph-mon[36187]: pgmap v2718: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T01:29:16.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:29:16 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 3851.669650 secs 2024-03-24T01:29:16.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:29:16 smithi193 ceph-mon[36878]: pgmap v2718: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T01:29:16.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:29:16 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 3851.669650 secs 2024-03-24T01:29:18.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:29:18 smithi118 ceph-mon[31535]: pgmap v2719: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:29:18.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:29:18 smithi189 ceph-mon[36187]: pgmap v2719: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:29:18.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:29:18 smithi193 ceph-mon[36878]: pgmap v2719: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:29:20.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:29:20 smithi118 ceph-mon[31535]: pgmap v2720: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T01:29:20.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:29:20 smithi189 ceph-mon[36187]: pgmap v2720: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T01:29:20.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:29:20 smithi193 ceph-mon[36878]: pgmap v2720: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T01:29:21.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:29:21 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 3856.669738 secs 2024-03-24T01:29:21.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:29:21 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 3856.669738 secs 2024-03-24T01:29:21.460 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:29:21 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 3856.669738 secs 2024-03-24T01:29:22.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:29:22 smithi118 ceph-mon[31535]: pgmap v2721: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T01:29:22.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:29:22 smithi189 ceph-mon[36187]: pgmap v2721: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T01:29:22.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:29:22 smithi193 ceph-mon[36878]: pgmap v2721: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T01:29:24.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:29:24 smithi118 ceph-mon[31535]: pgmap v2722: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 5 op/s 2024-03-24T01:29:24.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:29:24 smithi189 ceph-mon[36187]: pgmap v2722: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 5 op/s 2024-03-24T01:29:24.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:29:24 smithi193 ceph-mon[36878]: pgmap v2722: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 5 op/s 2024-03-24T01:29:26.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:29:26 smithi118 ceph-mon[31535]: pgmap v2723: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T01:29:26.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:29:26 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T01:29:26.447 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:29:26 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 3861.669828 secs 2024-03-24T01:29:26.447 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:29:26 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T01:29:26.447 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:29:26 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T01:29:26.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:29:26 smithi189 ceph-mon[36187]: pgmap v2723: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T01:29:26.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:29:26 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T01:29:26.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:29:26 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 3861.669828 secs 2024-03-24T01:29:26.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:29:26 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T01:29:26.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:29:26 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T01:29:26.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:29:26 smithi193 ceph-mon[36878]: pgmap v2723: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T01:29:26.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:29:26 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T01:29:26.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:29:26 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 3861.669828 secs 2024-03-24T01:29:26.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:29:26 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T01:29:26.462 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:29:26 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T01:29:27.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:29:27 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T01:29:27.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:29:27 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T01:29:27.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:29:27 smithi118 ceph-mon[31535]: pgmap v2724: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 5 op/s 2024-03-24T01:29:27.447 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:29:27 smithi118 ceph-mon[31535]: pgmap v2725: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 6 op/s 2024-03-24T01:29:27.447 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:29:27 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T01:29:27.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:29:27 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T01:29:27.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:29:27 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T01:29:27.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:29:27 smithi189 ceph-mon[36187]: pgmap v2724: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 5 op/s 2024-03-24T01:29:27.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:29:27 smithi189 ceph-mon[36187]: pgmap v2725: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 6 op/s 2024-03-24T01:29:27.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:29:27 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T01:29:27.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:29:27 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T01:29:27.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:29:27 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T01:29:27.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:29:27 smithi193 ceph-mon[36878]: pgmap v2724: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 5 op/s 2024-03-24T01:29:27.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:29:27 smithi193 ceph-mon[36878]: pgmap v2725: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 6 op/s 2024-03-24T01:29:27.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:29:27 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T01:29:29.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:29:29 smithi118 ceph-mon[31535]: pgmap v2726: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 6 op/s 2024-03-24T01:29:29.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:29:29 smithi189 ceph-mon[36187]: pgmap v2726: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 6 op/s 2024-03-24T01:29:29.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:29:29 smithi193 ceph-mon[36878]: pgmap v2726: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 6 op/s 2024-03-24T01:29:31.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:29:31 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 3866.669920 secs 2024-03-24T01:29:31.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:29:31 smithi118 ceph-mon[31535]: pgmap v2727: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 6 op/s 2024-03-24T01:29:31.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:29:31 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 3866.669920 secs 2024-03-24T01:29:31.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:29:31 smithi189 ceph-mon[36187]: pgmap v2727: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 6 op/s 2024-03-24T01:29:31.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:29:31 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 3866.669920 secs 2024-03-24T01:29:31.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:29:31 smithi193 ceph-mon[36878]: pgmap v2727: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 6 op/s 2024-03-24T01:29:33.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:29:33 smithi118 ceph-mon[31535]: pgmap v2728: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 5 op/s 2024-03-24T01:29:33.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:29:33 smithi189 ceph-mon[36187]: pgmap v2728: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 5 op/s 2024-03-24T01:29:33.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:29:33 smithi193 ceph-mon[36878]: pgmap v2728: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 5 op/s 2024-03-24T01:29:35.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:29:35 smithi118 ceph-mon[31535]: pgmap v2729: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 6 op/s 2024-03-24T01:29:35.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:29:35 smithi189 ceph-mon[36187]: pgmap v2729: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 6 op/s 2024-03-24T01:29:35.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:29:35 smithi193 ceph-mon[36878]: pgmap v2729: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 6 op/s 2024-03-24T01:29:36.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:29:36 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 3871.670003 secs 2024-03-24T01:29:36.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:29:36 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 3871.670003 secs 2024-03-24T01:29:36.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:29:36 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 3871.670003 secs 2024-03-24T01:29:37.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:29:37 smithi118 ceph-mon[31535]: pgmap v2730: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 5 op/s 2024-03-24T01:29:37.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:29:37 smithi189 ceph-mon[36187]: pgmap v2730: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 5 op/s 2024-03-24T01:29:37.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:29:37 smithi193 ceph-mon[36878]: pgmap v2730: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 5 op/s 2024-03-24T01:29:39.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:29:39 smithi118 ceph-mon[31535]: pgmap v2731: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T01:29:39.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:29:39 smithi189 ceph-mon[36187]: pgmap v2731: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T01:29:39.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:29:39 smithi193 ceph-mon[36878]: pgmap v2731: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T01:29:41.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:29:41 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 3876.670092 secs 2024-03-24T01:29:41.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:29:41 smithi118 ceph-mon[31535]: pgmap v2732: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T01:29:41.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:29:41 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 3876.670092 secs 2024-03-24T01:29:41.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:29:41 smithi189 ceph-mon[36187]: pgmap v2732: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T01:29:41.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:29:41 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 3876.670092 secs 2024-03-24T01:29:41.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:29:41 smithi193 ceph-mon[36878]: pgmap v2732: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T01:29:43.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:29:43 smithi118 ceph-mon[31535]: pgmap v2733: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T01:29:43.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:29:43 smithi189 ceph-mon[36187]: pgmap v2733: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T01:29:43.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:29:43 smithi193 ceph-mon[36878]: pgmap v2733: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T01:29:45.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:29:45 smithi118 ceph-mon[31535]: pgmap v2734: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-03-24T01:29:45.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:29:45 smithi189 ceph-mon[36187]: pgmap v2734: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-03-24T01:29:45.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:29:45 smithi193 ceph-mon[36878]: pgmap v2734: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-03-24T01:29:46.848 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:29:46 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 3881.670132 secs 2024-03-24T01:29:46.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:29:46 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 3881.670132 secs 2024-03-24T01:29:46.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:29:46 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 3881.670132 secs 2024-03-24T01:29:47.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:29:47 smithi118 ceph-mon[31535]: pgmap v2735: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:29:47.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:29:47 smithi189 ceph-mon[36187]: pgmap v2735: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:29:47.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:29:47 smithi193 ceph-mon[36878]: pgmap v2735: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:29:49.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:29:49 smithi118 ceph-mon[31535]: pgmap v2736: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T01:29:49.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:29:49 smithi189 ceph-mon[36187]: pgmap v2736: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T01:29:49.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:29:49 smithi193 ceph-mon[36878]: pgmap v2736: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T01:29:51.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:29:51 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 3886.670193 secs 2024-03-24T01:29:51.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:29:51 smithi118 ceph-mon[31535]: pgmap v2737: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T01:29:51.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:29:51 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 3886.670193 secs 2024-03-24T01:29:51.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:29:51 smithi189 ceph-mon[36187]: pgmap v2737: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T01:29:51.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:29:51 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 3886.670193 secs 2024-03-24T01:29:51.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:29:51 smithi193 ceph-mon[36878]: pgmap v2737: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T01:29:53.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:29:53 smithi118 ceph-mon[31535]: pgmap v2738: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T01:29:53.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:29:53 smithi189 ceph-mon[36187]: pgmap v2738: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T01:29:53.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:29:53 smithi193 ceph-mon[36878]: pgmap v2738: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T01:29:55.947 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:29:55 smithi118 ceph-mon[31535]: pgmap v2739: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T01:29:55.947 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:29:55 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 3891.670255 secs 2024-03-24T01:29:55.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:29:55 smithi189 ceph-mon[36187]: pgmap v2739: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T01:29:55.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:29:55 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 3891.670255 secs 2024-03-24T01:29:55.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:29:55 smithi193 ceph-mon[36878]: pgmap v2739: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T01:29:55.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:29:55 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 3891.670255 secs 2024-03-24T01:29:57.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:29:57 smithi118 ceph-mon[31535]: pgmap v2740: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:29:57.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:29:57 smithi189 ceph-mon[36187]: pgmap v2740: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:29:57.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:29:57 smithi193 ceph-mon[36878]: pgmap v2740: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:29:59.947 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:29:59 smithi118 ceph-mon[31535]: pgmap v2741: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T01:29:59.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:29:59 smithi189 ceph-mon[36187]: pgmap v2741: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T01:29:59.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:29:59 smithi193 ceph-mon[36878]: pgmap v2741: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T01:30:00.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:30:00 smithi118 ceph-mon[31535]: overall HEALTH_WARN 6 failed cephadm daemon(s); 1 MDSs report slow requests 2024-03-24T01:30:00.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:30:00 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 3896.670339 secs 2024-03-24T01:30:00.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:30:00 smithi189 ceph-mon[36187]: overall HEALTH_WARN 6 failed cephadm daemon(s); 1 MDSs report slow requests 2024-03-24T01:30:00.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:30:00 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 3896.670339 secs 2024-03-24T01:30:00.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:30:00 smithi193 ceph-mon[36878]: overall HEALTH_WARN 6 failed cephadm daemon(s); 1 MDSs report slow requests 2024-03-24T01:30:00.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:30:00 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 3896.670339 secs 2024-03-24T01:30:01.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:30:01 smithi118 ceph-mon[31535]: pgmap v2742: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T01:30:01.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:30:01 smithi189 ceph-mon[36187]: pgmap v2742: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T01:30:01.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:30:01 smithi193 ceph-mon[36878]: pgmap v2742: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T01:30:03.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:30:03 smithi118 ceph-mon[31535]: pgmap v2743: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T01:30:03.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:30:03 smithi189 ceph-mon[36187]: pgmap v2743: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T01:30:03.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:30:03 smithi193 ceph-mon[36878]: pgmap v2743: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T01:30:05.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:30:05 smithi118 ceph-mon[31535]: pgmap v2744: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-03-24T01:30:05.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:30:05 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 3901.670399 secs 2024-03-24T01:30:05.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:30:05 smithi189 ceph-mon[36187]: pgmap v2744: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-03-24T01:30:05.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:30:05 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 3901.670399 secs 2024-03-24T01:30:05.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:30:05 smithi193 ceph-mon[36878]: pgmap v2744: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-03-24T01:30:05.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:30:05 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 3901.670399 secs 2024-03-24T01:30:07.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:30:07 smithi118 ceph-mon[31535]: pgmap v2745: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:30:07.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:30:07 smithi189 ceph-mon[36187]: pgmap v2745: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:30:07.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:30:07 smithi193 ceph-mon[36878]: pgmap v2745: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:30:09.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:30:09 smithi118 ceph-mon[31535]: pgmap v2746: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T01:30:09.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:30:09 smithi189 ceph-mon[36187]: pgmap v2746: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T01:30:09.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:30:09 smithi193 ceph-mon[36878]: pgmap v2746: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T01:30:10.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:30:10 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 3906.670484 secs 2024-03-24T01:30:10.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:30:10 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 3906.670484 secs 2024-03-24T01:30:10.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:30:10 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 3906.670484 secs 2024-03-24T01:30:11.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:30:11 smithi118 ceph-mon[31535]: pgmap v2747: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T01:30:11.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:30:11 smithi189 ceph-mon[36187]: pgmap v2747: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T01:30:11.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:30:11 smithi193 ceph-mon[36878]: pgmap v2747: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T01:30:13.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:30:13 smithi118 ceph-mon[31535]: pgmap v2748: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T01:30:13.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:30:13 smithi189 ceph-mon[36187]: pgmap v2748: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T01:30:13.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:30:13 smithi193 ceph-mon[36878]: pgmap v2748: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T01:30:15.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:30:15 smithi118 ceph-mon[31535]: pgmap v2749: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-03-24T01:30:15.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:30:15 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 3911.670526 secs 2024-03-24T01:30:15.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:30:15 smithi189 ceph-mon[36187]: pgmap v2749: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-03-24T01:30:15.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:30:15 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 3911.670526 secs 2024-03-24T01:30:15.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:30:15 smithi193 ceph-mon[36878]: pgmap v2749: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-03-24T01:30:15.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:30:15 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 3911.670526 secs 2024-03-24T01:30:17.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:30:17 smithi118 ceph-mon[31535]: pgmap v2750: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:30:17.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:30:17 smithi189 ceph-mon[36187]: pgmap v2750: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:30:17.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:30:17 smithi193 ceph-mon[36878]: pgmap v2750: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:30:19.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:30:19 smithi118 ceph-mon[31535]: pgmap v2751: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T01:30:19.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:30:19 smithi189 ceph-mon[36187]: pgmap v2751: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T01:30:19.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:30:19 smithi193 ceph-mon[36878]: pgmap v2751: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T01:30:20.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:30:20 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 3916.670586 secs 2024-03-24T01:30:20.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:30:20 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 3916.670586 secs 2024-03-24T01:30:20.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:30:20 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 3916.670586 secs 2024-03-24T01:30:21.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:30:21 smithi118 ceph-mon[31535]: pgmap v2752: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T01:30:21.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:30:21 smithi189 ceph-mon[36187]: pgmap v2752: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T01:30:21.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:30:21 smithi193 ceph-mon[36878]: pgmap v2752: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T01:30:23.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:30:23 smithi118 ceph-mon[31535]: pgmap v2753: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T01:30:23.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:30:23 smithi189 ceph-mon[36187]: pgmap v2753: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T01:30:23.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:30:23 smithi193 ceph-mon[36878]: pgmap v2753: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T01:30:25.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:30:25 smithi118 ceph-mon[31535]: pgmap v2754: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-03-24T01:30:25.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:30:25 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 3921.670647 secs 2024-03-24T01:30:25.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:30:25 smithi189 ceph-mon[36187]: pgmap v2754: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-03-24T01:30:25.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:30:25 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 3921.670647 secs 2024-03-24T01:30:25.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:30:25 smithi193 ceph-mon[36878]: pgmap v2754: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-03-24T01:30:25.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:30:25 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 3921.670647 secs 2024-03-24T01:30:27.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:30:27 smithi118 ceph-mon[31535]: pgmap v2755: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:30:27.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:30:27 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T01:30:27.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:30:27 smithi189 ceph-mon[36187]: pgmap v2755: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:30:27.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:30:27 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T01:30:27.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:30:27 smithi193 ceph-mon[36878]: pgmap v2755: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:30:27.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:30:27 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T01:30:28.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:30:28 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T01:30:28.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:30:28 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T01:30:28.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:30:28 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T01:30:28.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:30:28 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T01:30:28.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:30:28 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T01:30:28.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:30:28 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T01:30:28.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:30:28 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T01:30:28.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:30:28 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T01:30:28.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:30:28 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T01:30:29.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:30:29 smithi118 ceph-mon[31535]: pgmap v2756: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 6 op/s 2024-03-24T01:30:29.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:30:29 smithi118 ceph-mon[31535]: pgmap v2757: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 5 op/s 2024-03-24T01:30:29.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:30:29 smithi189 ceph-mon[36187]: pgmap v2756: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 6 op/s 2024-03-24T01:30:29.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:30:29 smithi189 ceph-mon[36187]: pgmap v2757: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 5 op/s 2024-03-24T01:30:29.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:30:29 smithi193 ceph-mon[36878]: pgmap v2756: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 6 op/s 2024-03-24T01:30:29.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:30:29 smithi193 ceph-mon[36878]: pgmap v2757: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 5 op/s 2024-03-24T01:30:30.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:30:30 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 3926.670706 secs 2024-03-24T01:30:30.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:30:30 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 3926.670706 secs 2024-03-24T01:30:30.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:30:30 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 3926.670706 secs 2024-03-24T01:30:31.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:30:31 smithi118 ceph-mon[31535]: pgmap v2758: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-03-24T01:30:31.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:30:31 smithi189 ceph-mon[36187]: pgmap v2758: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-03-24T01:30:31.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:30:31 smithi193 ceph-mon[36878]: pgmap v2758: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-03-24T01:30:33.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:30:33 smithi118 ceph-mon[31535]: pgmap v2759: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 5 op/s 2024-03-24T01:30:33.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:30:33 smithi189 ceph-mon[36187]: pgmap v2759: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 5 op/s 2024-03-24T01:30:33.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:30:33 smithi193 ceph-mon[36878]: pgmap v2759: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 5 op/s 2024-03-24T01:30:35.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:30:35 smithi189 ceph-mon[36187]: pgmap v2760: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 4 op/s 2024-03-24T01:30:35.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:30:35 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 3931.670759 secs 2024-03-24T01:30:35.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:30:35 smithi193 ceph-mon[36878]: pgmap v2760: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 4 op/s 2024-03-24T01:30:35.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:30:35 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 3931.670759 secs 2024-03-24T01:30:36.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:30:35 smithi118 ceph-mon[31535]: pgmap v2760: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 4 op/s 2024-03-24T01:30:36.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:30:35 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 3931.670759 secs 2024-03-24T01:30:37.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:30:37 smithi189 ceph-mon[36187]: pgmap v2761: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-03-24T01:30:37.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:30:37 smithi193 ceph-mon[36878]: pgmap v2761: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-03-24T01:30:38.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:30:37 smithi118 ceph-mon[31535]: pgmap v2761: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-03-24T01:30:39.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:30:39 smithi193 ceph-mon[36878]: pgmap v2762: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 5 op/s 2024-03-24T01:30:40.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:30:39 smithi118 ceph-mon[31535]: pgmap v2762: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 5 op/s 2024-03-24T01:30:40.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:30:39 smithi189 ceph-mon[36187]: pgmap v2762: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 5 op/s 2024-03-24T01:30:41.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:30:40 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 3936.670847 secs 2024-03-24T01:30:41.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:30:40 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 3936.670847 secs 2024-03-24T01:30:41.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:30:40 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 3936.670847 secs 2024-03-24T01:30:42.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:30:41 smithi118 ceph-mon[31535]: pgmap v2763: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T01:30:42.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:30:41 smithi189 ceph-mon[36187]: pgmap v2763: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T01:30:42.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:30:41 smithi193 ceph-mon[36878]: pgmap v2763: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T01:30:43.096 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:30:42 smithi118 ceph-mon[31535]: pgmap v2764: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:30:43.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:30:42 smithi189 ceph-mon[36187]: pgmap v2764: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:30:43.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:30:42 smithi193 ceph-mon[36878]: pgmap v2764: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:30:45.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:30:44 smithi118 ceph-mon[31535]: pgmap v2765: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T01:30:45.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:30:44 smithi189 ceph-mon[36187]: pgmap v2765: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T01:30:45.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:30:44 smithi193 ceph-mon[36878]: pgmap v2765: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T01:30:46.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:30:45 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 3941.670907 secs 2024-03-24T01:30:46.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:30:45 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 3941.670907 secs 2024-03-24T01:30:46.460 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:30:45 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 3941.670907 secs 2024-03-24T01:30:47.348 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:30:46 smithi118 ceph-mon[31535]: pgmap v2766: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T01:30:47.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:30:46 smithi189 ceph-mon[36187]: pgmap v2766: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T01:30:47.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:30:46 smithi193 ceph-mon[36878]: pgmap v2766: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T01:30:49.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:30:48 smithi118 ceph-mon[31535]: pgmap v2767: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 5 op/s 2024-03-24T01:30:49.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:30:48 smithi189 ceph-mon[36187]: pgmap v2767: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 5 op/s 2024-03-24T01:30:49.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:30:48 smithi193 ceph-mon[36878]: pgmap v2767: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 5 op/s 2024-03-24T01:30:51.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:30:50 smithi118 ceph-mon[31535]: pgmap v2768: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T01:30:51.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:30:50 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 3946.670948 secs 2024-03-24T01:30:51.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:30:50 smithi189 ceph-mon[36187]: pgmap v2768: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T01:30:51.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:30:50 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 3946.670948 secs 2024-03-24T01:30:51.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:30:50 smithi193 ceph-mon[36878]: pgmap v2768: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T01:30:51.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:30:50 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 3946.670948 secs 2024-03-24T01:30:53.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:30:53 smithi118 ceph-mon[31535]: pgmap v2769: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:30:53.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:30:53 smithi189 ceph-mon[36187]: pgmap v2769: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:30:53.460 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:30:53 smithi193 ceph-mon[36878]: pgmap v2769: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:30:55.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:30:55 smithi118 ceph-mon[31535]: pgmap v2770: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T01:30:55.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:30:55 smithi189 ceph-mon[36187]: pgmap v2770: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T01:30:55.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:30:55 smithi193 ceph-mon[36878]: pgmap v2770: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T01:30:56.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:30:56 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 3951.671036 secs 2024-03-24T01:30:56.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:30:56 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 3951.671036 secs 2024-03-24T01:30:56.460 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:30:56 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 3951.671036 secs 2024-03-24T01:30:57.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:30:57 smithi118 ceph-mon[31535]: pgmap v2771: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T01:30:57.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:30:57 smithi189 ceph-mon[36187]: pgmap v2771: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T01:30:57.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:30:57 smithi193 ceph-mon[36878]: pgmap v2771: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T01:30:59.390 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:30:59 smithi118 ceph-mon[31535]: pgmap v2772: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 5 op/s 2024-03-24T01:30:59.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:30:59 smithi189 ceph-mon[36187]: pgmap v2772: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 5 op/s 2024-03-24T01:30:59.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:30:59 smithi193 ceph-mon[36878]: pgmap v2772: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 5 op/s 2024-03-24T01:31:01.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:31:01 smithi118 ceph-mon[31535]: pgmap v2773: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T01:31:01.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:31:01 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 3956.671121 secs 2024-03-24T01:31:01.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:31:01 smithi189 ceph-mon[36187]: pgmap v2773: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T01:31:01.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:31:01 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 3956.671121 secs 2024-03-24T01:31:01.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:31:01 smithi193 ceph-mon[36878]: pgmap v2773: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T01:31:01.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:31:01 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 3956.671121 secs 2024-03-24T01:31:03.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:31:03 smithi118 ceph-mon[31535]: pgmap v2774: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:31:03.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:31:03 smithi189 ceph-mon[36187]: pgmap v2774: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:31:03.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:31:03 smithi193 ceph-mon[36878]: pgmap v2774: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:31:05.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:31:05 smithi118 ceph-mon[31535]: pgmap v2775: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T01:31:05.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:31:05 smithi189 ceph-mon[36187]: pgmap v2775: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T01:31:05.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:31:05 smithi193 ceph-mon[36878]: pgmap v2775: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T01:31:06.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:31:06 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 3961.671180 secs 2024-03-24T01:31:06.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:31:06 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 3961.671180 secs 2024-03-24T01:31:06.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:31:06 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 3961.671180 secs 2024-03-24T01:31:07.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:31:07 smithi118 ceph-mon[31535]: pgmap v2776: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T01:31:07.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:31:07 smithi189 ceph-mon[36187]: pgmap v2776: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T01:31:07.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:31:07 smithi193 ceph-mon[36878]: pgmap v2776: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T01:31:09.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:31:09 smithi118 ceph-mon[31535]: pgmap v2777: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 5 op/s 2024-03-24T01:31:09.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:31:09 smithi189 ceph-mon[36187]: pgmap v2777: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 5 op/s 2024-03-24T01:31:09.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:31:09 smithi193 ceph-mon[36878]: pgmap v2777: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 5 op/s 2024-03-24T01:31:11.447 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:31:11 smithi118 ceph-mon[31535]: pgmap v2778: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T01:31:11.447 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:31:11 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 3966.671239 secs 2024-03-24T01:31:11.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:31:11 smithi189 ceph-mon[36187]: pgmap v2778: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T01:31:11.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:31:11 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 3966.671239 secs 2024-03-24T01:31:11.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:31:11 smithi193 ceph-mon[36878]: pgmap v2778: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T01:31:11.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:31:11 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 3966.671239 secs 2024-03-24T01:31:13.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:31:13 smithi118 ceph-mon[31535]: pgmap v2779: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:31:13.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:31:13 smithi189 ceph-mon[36187]: pgmap v2779: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:31:13.460 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:31:13 smithi193 ceph-mon[36878]: pgmap v2779: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:31:15.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:31:15 smithi118 ceph-mon[31535]: pgmap v2780: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T01:31:15.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:31:15 smithi189 ceph-mon[36187]: pgmap v2780: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T01:31:15.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:31:15 smithi193 ceph-mon[36878]: pgmap v2780: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T01:31:16.367 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T01:31:16.366+0000 7f81db400640 1 -- 172.21.15.118:0/659686424 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- auth(proto 2 132 bytes epoch 0) v1 -- 0x7f81c4006290 con 0x5622651a13f0 2024-03-24T01:31:16.367 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T01:31:16.366+0000 7f81ee000640 1 -- 172.21.15.118:0/1693767697 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- auth(proto 2 132 bytes epoch 0) v1 -- 0x7f81c80065b0 con 0x7f81f81052d0 2024-03-24T01:31:16.367 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T01:31:16.366+0000 7f81e4c00640 1 -- 172.21.15.118:0/659686424 <== mon.1 v2:172.21.15.193:3300/0 34 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 612+0+0 (secure 0 0 0) 0x7f81c4006290 con 0x5622651a13f0 2024-03-24T01:31:16.367 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T01:31:16.367+0000 7f81e4c00640 1 -- 172.21.15.118:0/659686424 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f81c40064d0 con 0x5622651a13f0 2024-03-24T01:31:16.367 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T01:31:16.367+0000 7f81ef400640 1 -- 172.21.15.118:0/1693767697 <== mon.2 v2:172.21.15.189:3300/0 34 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 783+0+0 (secure 0 0 0) 0x7f81c80065b0 con 0x7f81f81052d0 2024-03-24T01:31:16.367 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T01:31:16.367+0000 7f81ef400640 1 -- 172.21.15.118:0/1693767697 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f81c80036c0 con 0x7f81f81052d0 2024-03-24T01:31:16.367 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T01:31:16.367+0000 7f81e4c00640 1 -- 172.21.15.118:0/659686424 <== mon.1 v2:172.21.15.193:3300/0 35 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 409+0+0 (secure 0 0 0) 0x7f81c40064d0 con 0x5622651a13f0 2024-03-24T01:31:16.368 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T01:31:16.367+0000 7f81ef400640 1 -- 172.21.15.118:0/1693767697 <== mon.2 v2:172.21.15.189:3300/0 35 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (secure 0 0 0) 0x7f81c80036c0 con 0x7f81f81052d0 2024-03-24T01:31:16.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:31:16 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 3971.671282 secs 2024-03-24T01:31:16.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:31:16 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 3971.671282 secs 2024-03-24T01:31:16.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:31:16 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 3971.671282 secs 2024-03-24T01:31:17.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:31:17 smithi118 ceph-mon[31535]: pgmap v2781: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T01:31:17.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:31:17 smithi189 ceph-mon[36187]: pgmap v2781: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T01:31:17.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:31:17 smithi193 ceph-mon[36878]: pgmap v2781: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T01:31:19.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:31:19 smithi118 ceph-mon[31535]: pgmap v2782: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 5 op/s 2024-03-24T01:31:19.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:31:19 smithi189 ceph-mon[36187]: pgmap v2782: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 5 op/s 2024-03-24T01:31:19.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:31:19 smithi193 ceph-mon[36878]: pgmap v2782: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 5 op/s 2024-03-24T01:31:21.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:31:21 smithi118 ceph-mon[31535]: pgmap v2783: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T01:31:21.447 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:31:21 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 3976.671367 secs 2024-03-24T01:31:21.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:31:21 smithi189 ceph-mon[36187]: pgmap v2783: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T01:31:21.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:31:21 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 3976.671367 secs 2024-03-24T01:31:21.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:31:21 smithi193 ceph-mon[36878]: pgmap v2783: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T01:31:21.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:31:21 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 3976.671367 secs 2024-03-24T01:31:23.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:31:23 smithi118 ceph-mon[31535]: pgmap v2784: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:31:23.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:31:23 smithi189 ceph-mon[36187]: pgmap v2784: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:31:23.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:31:23 smithi193 ceph-mon[36878]: pgmap v2784: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:31:25.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:31:25 smithi118 ceph-mon[31535]: pgmap v2785: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T01:31:25.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:31:25 smithi189 ceph-mon[36187]: pgmap v2785: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T01:31:25.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:31:25 smithi193 ceph-mon[36878]: pgmap v2785: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T01:31:26.447 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:31:26 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 3981.671457 secs 2024-03-24T01:31:26.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:31:26 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 3981.671457 secs 2024-03-24T01:31:26.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:31:26 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 3981.671457 secs 2024-03-24T01:31:27.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:31:27 smithi118 ceph-mon[31535]: pgmap v2786: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T01:31:27.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:31:27 smithi189 ceph-mon[36187]: pgmap v2786: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T01:31:27.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:31:27 smithi193 ceph-mon[36878]: pgmap v2786: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T01:31:29.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:31:29 smithi118 ceph-mon[31535]: pgmap v2787: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 5 op/s 2024-03-24T01:31:29.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:31:29 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T01:31:29.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:31:29 smithi189 ceph-mon[36187]: pgmap v2787: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 5 op/s 2024-03-24T01:31:29.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:31:29 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T01:31:29.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:31:29 smithi193 ceph-mon[36878]: pgmap v2787: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 5 op/s 2024-03-24T01:31:29.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:31:29 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T01:31:30.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:31:30 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T01:31:30.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:31:30 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T01:31:30.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:31:30 smithi118 ceph-mon[31535]: pgmap v2788: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 6 op/s 2024-03-24T01:31:30.447 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:31:30 smithi118 ceph-mon[31535]: pgmap v2789: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-03-24T01:31:30.447 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:31:30 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T01:31:30.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:31:30 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T01:31:30.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:31:30 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T01:31:30.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:31:30 smithi189 ceph-mon[36187]: pgmap v2788: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 6 op/s 2024-03-24T01:31:30.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:31:30 smithi189 ceph-mon[36187]: pgmap v2789: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-03-24T01:31:30.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:31:30 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T01:31:30.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:31:30 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T01:31:30.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:31:30 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T01:31:30.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:31:30 smithi193 ceph-mon[36878]: pgmap v2788: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 6 op/s 2024-03-24T01:31:30.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:31:30 smithi193 ceph-mon[36878]: pgmap v2789: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-03-24T01:31:30.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:31:30 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T01:31:31.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:31:31 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 3986.671547 secs 2024-03-24T01:31:31.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:31:31 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 3986.671547 secs 2024-03-24T01:31:31.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:31:31 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 3986.671547 secs 2024-03-24T01:31:32.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:31:32 smithi118 ceph-mon[31535]: pgmap v2790: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-03-24T01:31:32.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:31:32 smithi189 ceph-mon[36187]: pgmap v2790: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-03-24T01:31:32.460 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:31:32 smithi193 ceph-mon[36878]: pgmap v2790: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-03-24T01:31:34.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:31:34 smithi118 ceph-mon[31535]: pgmap v2791: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 5 op/s 2024-03-24T01:31:34.704 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:31:34 smithi189 ceph-mon[36187]: pgmap v2791: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 5 op/s 2024-03-24T01:31:34.711 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:31:34 smithi193 ceph-mon[36878]: pgmap v2791: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 5 op/s 2024-03-24T01:31:36.697 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:31:36 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 3991.671602 secs 2024-03-24T01:31:36.697 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:31:36 smithi118 ceph-mon[31535]: pgmap v2792: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-03-24T01:31:36.704 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:31:36 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 3991.671602 secs 2024-03-24T01:31:36.704 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:31:36 smithi189 ceph-mon[36187]: pgmap v2792: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-03-24T01:31:36.711 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:31:36 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 3991.671602 secs 2024-03-24T01:31:36.711 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:31:36 smithi193 ceph-mon[36878]: pgmap v2792: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-03-24T01:31:38.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:31:38 smithi118 ceph-mon[31535]: pgmap v2793: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 5 op/s 2024-03-24T01:31:38.704 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:31:38 smithi189 ceph-mon[36187]: pgmap v2793: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 5 op/s 2024-03-24T01:31:38.711 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:31:38 smithi193 ceph-mon[36878]: pgmap v2793: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 5 op/s 2024-03-24T01:31:40.697 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:31:40 smithi118 ceph-mon[31535]: pgmap v2794: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 5 op/s 2024-03-24T01:31:40.704 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:31:40 smithi189 ceph-mon[36187]: pgmap v2794: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 5 op/s 2024-03-24T01:31:40.711 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:31:40 smithi193 ceph-mon[36878]: pgmap v2794: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 5 op/s 2024-03-24T01:31:41.704 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:31:41 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 3996.671688 secs 2024-03-24T01:31:41.711 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:31:41 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 3996.671688 secs 2024-03-24T01:31:41.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:31:41 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 3996.671688 secs 2024-03-24T01:31:42.704 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:31:42 smithi189 ceph-mon[36187]: pgmap v2795: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:31:42.711 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:31:42 smithi193 ceph-mon[36878]: pgmap v2795: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:31:42.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:31:42 smithi118 ceph-mon[31535]: pgmap v2795: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:31:44.711 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:31:44 smithi193 ceph-mon[36878]: pgmap v2796: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T01:31:44.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:31:44 smithi118 ceph-mon[31535]: pgmap v2796: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T01:31:44.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:31:44 smithi189 ceph-mon[36187]: pgmap v2796: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T01:31:46.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:31:46 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 4001.671777 secs 2024-03-24T01:31:46.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:31:46 smithi118 ceph-mon[31535]: pgmap v2797: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T01:31:46.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:31:46 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 4001.671777 secs 2024-03-24T01:31:46.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:31:46 smithi189 ceph-mon[36187]: pgmap v2797: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T01:31:46.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:31:46 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 4001.671777 secs 2024-03-24T01:31:46.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:31:46 smithi193 ceph-mon[36878]: pgmap v2797: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T01:31:48.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:31:48 smithi118 ceph-mon[31535]: pgmap v2798: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T01:31:48.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:31:48 smithi189 ceph-mon[36187]: pgmap v2798: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T01:31:48.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:31:48 smithi193 ceph-mon[36878]: pgmap v2798: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T01:31:50.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:31:50 smithi118 ceph-mon[31535]: pgmap v2799: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-03-24T01:31:50.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:31:50 smithi189 ceph-mon[36187]: pgmap v2799: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-03-24T01:31:50.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:31:50 smithi193 ceph-mon[36878]: pgmap v2799: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-03-24T01:31:51.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:31:51 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 4006.671867 secs 2024-03-24T01:31:51.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:31:51 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 4006.671867 secs 2024-03-24T01:31:51.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:31:51 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 4006.671867 secs 2024-03-24T01:31:52.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:31:52 smithi118 ceph-mon[31535]: pgmap v2800: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:31:52.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:31:52 smithi189 ceph-mon[36187]: pgmap v2800: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:31:52.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:31:52 smithi193 ceph-mon[36878]: pgmap v2800: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:31:54.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:31:54 smithi118 ceph-mon[31535]: pgmap v2801: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T01:31:54.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:31:54 smithi189 ceph-mon[36187]: pgmap v2801: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T01:31:54.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:31:54 smithi193 ceph-mon[36878]: pgmap v2801: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T01:31:56.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:31:56 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 4011.671955 secs 2024-03-24T01:31:56.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:31:56 smithi118 ceph-mon[31535]: pgmap v2802: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T01:31:56.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:31:56 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 4011.671955 secs 2024-03-24T01:31:56.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:31:56 smithi189 ceph-mon[36187]: pgmap v2802: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T01:31:56.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:31:56 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 4011.671955 secs 2024-03-24T01:31:56.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:31:56 smithi193 ceph-mon[36878]: pgmap v2802: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T01:31:58.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:31:58 smithi118 ceph-mon[31535]: pgmap v2803: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T01:31:58.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:31:58 smithi189 ceph-mon[36187]: pgmap v2803: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T01:31:58.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:31:58 smithi193 ceph-mon[36878]: pgmap v2803: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T01:32:00.755 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:32:00 smithi189 ceph-mon[36187]: pgmap v2804: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-03-24T01:32:00.786 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:32:00 smithi193 ceph-mon[36878]: pgmap v2804: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-03-24T01:32:00.920 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:32:00 smithi118 ceph-mon[31535]: pgmap v2804: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-03-24T01:32:01.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:32:01 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 4016.672041 secs 2024-03-24T01:32:01.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:32:01 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 4016.672041 secs 2024-03-24T01:32:01.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:32:01 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 4016.672041 secs 2024-03-24T01:32:02.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:32:02 smithi118 ceph-mon[31535]: pgmap v2805: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:32:02.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:32:02 smithi189 ceph-mon[36187]: pgmap v2805: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:32:02.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:32:02 smithi193 ceph-mon[36878]: pgmap v2805: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:32:04.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:32:04 smithi118 ceph-mon[31535]: pgmap v2806: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T01:32:04.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:32:04 smithi189 ceph-mon[36187]: pgmap v2806: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T01:32:04.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:32:04 smithi193 ceph-mon[36878]: pgmap v2806: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T01:32:06.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:32:06 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 4021.672114 secs 2024-03-24T01:32:06.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:32:06 smithi118 ceph-mon[31535]: pgmap v2807: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T01:32:06.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:32:06 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 4021.672114 secs 2024-03-24T01:32:06.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:32:06 smithi189 ceph-mon[36187]: pgmap v2807: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T01:32:06.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:32:06 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 4021.672114 secs 2024-03-24T01:32:06.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:32:06 smithi193 ceph-mon[36878]: pgmap v2807: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T01:32:08.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:32:08 smithi118 ceph-mon[31535]: pgmap v2808: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T01:32:08.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:32:08 smithi189 ceph-mon[36187]: pgmap v2808: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T01:32:08.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:32:08 smithi193 ceph-mon[36878]: pgmap v2808: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T01:32:10.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:32:10 smithi118 ceph-mon[31535]: pgmap v2809: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-03-24T01:32:10.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:32:10 smithi189 ceph-mon[36187]: pgmap v2809: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-03-24T01:32:10.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:32:10 smithi193 ceph-mon[36878]: pgmap v2809: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-03-24T01:32:11.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:32:11 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 4026.672174 secs 2024-03-24T01:32:11.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:32:11 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 4026.672174 secs 2024-03-24T01:32:11.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:32:11 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 4026.672174 secs 2024-03-24T01:32:12.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:32:12 smithi118 ceph-mon[31535]: pgmap v2810: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:32:12.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:32:12 smithi189 ceph-mon[36187]: pgmap v2810: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:32:12.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:32:12 smithi193 ceph-mon[36878]: pgmap v2810: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:32:14.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:32:14 smithi118 ceph-mon[31535]: pgmap v2811: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T01:32:14.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:32:14 smithi189 ceph-mon[36187]: pgmap v2811: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T01:32:14.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:32:14 smithi193 ceph-mon[36878]: pgmap v2811: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T01:32:16.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:32:16 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 4031.672211 secs 2024-03-24T01:32:16.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:32:16 smithi118 ceph-mon[31535]: pgmap v2812: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T01:32:16.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:32:16 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 4031.672211 secs 2024-03-24T01:32:16.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:32:16 smithi189 ceph-mon[36187]: pgmap v2812: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T01:32:16.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:32:16 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 4031.672211 secs 2024-03-24T01:32:16.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:32:16 smithi193 ceph-mon[36878]: pgmap v2812: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T01:32:18.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:32:18 smithi118 ceph-mon[31535]: pgmap v2813: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T01:32:18.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:32:18 smithi189 ceph-mon[36187]: pgmap v2813: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T01:32:18.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:32:18 smithi193 ceph-mon[36878]: pgmap v2813: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T01:32:20.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:32:20 smithi118 ceph-mon[31535]: pgmap v2814: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-03-24T01:32:20.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:32:20 smithi189 ceph-mon[36187]: pgmap v2814: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-03-24T01:32:20.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:32:20 smithi193 ceph-mon[36878]: pgmap v2814: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-03-24T01:32:21.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:32:21 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 4036.672272 secs 2024-03-24T01:32:21.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:32:21 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 4036.672272 secs 2024-03-24T01:32:21.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:32:21 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 4036.672272 secs 2024-03-24T01:32:22.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:32:22 smithi118 ceph-mon[31535]: pgmap v2815: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:32:22.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:32:22 smithi189 ceph-mon[36187]: pgmap v2815: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:32:22.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:32:22 smithi193 ceph-mon[36878]: pgmap v2815: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:32:24.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:32:24 smithi118 ceph-mon[31535]: pgmap v2816: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T01:32:24.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:32:24 smithi189 ceph-mon[36187]: pgmap v2816: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T01:32:24.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:32:24 smithi193 ceph-mon[36878]: pgmap v2816: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T01:32:26.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:32:26 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 4041.672331 secs 2024-03-24T01:32:26.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:32:26 smithi118 ceph-mon[31535]: pgmap v2817: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T01:32:26.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:32:26 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 4041.672331 secs 2024-03-24T01:32:26.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:32:26 smithi189 ceph-mon[36187]: pgmap v2817: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T01:32:26.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:32:26 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 4041.672331 secs 2024-03-24T01:32:26.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:32:26 smithi193 ceph-mon[36878]: pgmap v2817: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T01:32:28.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:32:28 smithi118 ceph-mon[31535]: pgmap v2818: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T01:32:28.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:32:28 smithi189 ceph-mon[36187]: pgmap v2818: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T01:32:28.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:32:28 smithi193 ceph-mon[36878]: pgmap v2818: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T01:32:30.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:32:30 smithi118 ceph-mon[31535]: pgmap v2819: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-03-24T01:32:30.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:32:30 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T01:32:30.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:32:30 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 4046.672399 secs 2024-03-24T01:32:30.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:32:30 smithi189 ceph-mon[36187]: pgmap v2819: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-03-24T01:32:30.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:32:30 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T01:32:30.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:32:30 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 4046.672399 secs 2024-03-24T01:32:30.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:32:30 smithi193 ceph-mon[36878]: pgmap v2819: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-03-24T01:32:30.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:32:30 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T01:32:30.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:32:30 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 4046.672399 secs 2024-03-24T01:32:31.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:32:31 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T01:32:31.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:32:31 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T01:32:31.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:32:31 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T01:32:31.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:32:31 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T01:32:31.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:32:31 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T01:32:31.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:32:31 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T01:32:31.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:32:31 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T01:32:31.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:32:31 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T01:32:31.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:32:31 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T01:32:32.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:32:32 smithi118 ceph-mon[31535]: pgmap v2820: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 5 op/s 2024-03-24T01:32:32.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:32:32 smithi118 ceph-mon[31535]: pgmap v2821: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 6 op/s 2024-03-24T01:32:32.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:32:32 smithi189 ceph-mon[36187]: pgmap v2820: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 5 op/s 2024-03-24T01:32:32.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:32:32 smithi189 ceph-mon[36187]: pgmap v2821: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 6 op/s 2024-03-24T01:32:32.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:32:32 smithi193 ceph-mon[36878]: pgmap v2820: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 5 op/s 2024-03-24T01:32:32.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:32:32 smithi193 ceph-mon[36878]: pgmap v2821: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 6 op/s 2024-03-24T01:32:34.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:32:34 smithi118 ceph-mon[31535]: pgmap v2822: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 5 op/s 2024-03-24T01:32:34.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:32:34 smithi189 ceph-mon[36187]: pgmap v2822: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 5 op/s 2024-03-24T01:32:34.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:32:34 smithi193 ceph-mon[36878]: pgmap v2822: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 5 op/s 2024-03-24T01:32:35.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:32:35 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 4051.672484 secs 2024-03-24T01:32:35.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:32:35 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 4051.672484 secs 2024-03-24T01:32:35.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:32:35 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 4051.672484 secs 2024-03-24T01:32:36.947 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:32:36 smithi118 ceph-mon[31535]: pgmap v2823: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 6 op/s 2024-03-24T01:32:36.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:32:36 smithi189 ceph-mon[36187]: pgmap v2823: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 6 op/s 2024-03-24T01:32:36.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:32:36 smithi193 ceph-mon[36878]: pgmap v2823: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 6 op/s 2024-03-24T01:32:38.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:32:38 smithi118 ceph-mon[31535]: pgmap v2824: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-03-24T01:32:38.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:32:38 smithi189 ceph-mon[36187]: pgmap v2824: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-03-24T01:32:38.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:32:38 smithi193 ceph-mon[36878]: pgmap v2824: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-03-24T01:32:40.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:32:40 smithi118 ceph-mon[31535]: pgmap v2825: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:32:40.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:32:40 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 4056.672571 secs 2024-03-24T01:32:40.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:32:40 smithi189 ceph-mon[36187]: pgmap v2825: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:32:40.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:32:40 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 4056.672571 secs 2024-03-24T01:32:40.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:32:40 smithi193 ceph-mon[36878]: pgmap v2825: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:32:40.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:32:40 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 4056.672571 secs 2024-03-24T01:32:42.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:32:42 smithi118 ceph-mon[31535]: pgmap v2826: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T01:32:42.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:32:42 smithi189 ceph-mon[36187]: pgmap v2826: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T01:32:42.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:32:42 smithi193 ceph-mon[36878]: pgmap v2826: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T01:32:44.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:32:44 smithi118 ceph-mon[31535]: pgmap v2827: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 5 op/s 2024-03-24T01:32:44.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:32:44 smithi189 ceph-mon[36187]: pgmap v2827: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 5 op/s 2024-03-24T01:32:44.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:32:44 smithi193 ceph-mon[36878]: pgmap v2827: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 5 op/s 2024-03-24T01:32:45.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:32:45 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 4061.672659 secs 2024-03-24T01:32:45.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:32:45 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 4061.672659 secs 2024-03-24T01:32:45.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:32:45 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 4061.672659 secs 2024-03-24T01:32:46.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:32:46 smithi118 ceph-mon[31535]: pgmap v2828: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T01:32:46.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:32:46 smithi189 ceph-mon[36187]: pgmap v2828: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T01:32:46.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:32:46 smithi193 ceph-mon[36878]: pgmap v2828: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T01:32:48.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:32:48 smithi118 ceph-mon[31535]: pgmap v2829: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:32:48.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:32:48 smithi189 ceph-mon[36187]: pgmap v2829: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:32:48.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:32:48 smithi193 ceph-mon[36878]: pgmap v2829: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:32:50.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:32:50 smithi118 ceph-mon[31535]: pgmap v2830: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T01:32:50.947 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:32:50 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 4066.672719 secs 2024-03-24T01:32:50.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:32:50 smithi189 ceph-mon[36187]: pgmap v2830: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T01:32:50.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:32:50 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 4066.672719 secs 2024-03-24T01:32:50.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:32:50 smithi193 ceph-mon[36878]: pgmap v2830: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T01:32:50.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:32:50 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 4066.672719 secs 2024-03-24T01:32:52.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:32:52 smithi118 ceph-mon[31535]: pgmap v2831: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T01:32:52.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:32:52 smithi189 ceph-mon[36187]: pgmap v2831: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T01:32:52.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:32:52 smithi193 ceph-mon[36878]: pgmap v2831: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T01:32:54.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:32:54 smithi118 ceph-mon[31535]: pgmap v2832: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 5 op/s 2024-03-24T01:32:54.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:32:54 smithi189 ceph-mon[36187]: pgmap v2832: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 5 op/s 2024-03-24T01:32:54.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:32:54 smithi193 ceph-mon[36878]: pgmap v2832: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 5 op/s 2024-03-24T01:32:55.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:32:55 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 4071.672809 secs 2024-03-24T01:32:55.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:32:55 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 4071.672809 secs 2024-03-24T01:32:55.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:32:55 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 4071.672809 secs 2024-03-24T01:32:56.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:32:56 smithi118 ceph-mon[31535]: pgmap v2833: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T01:32:56.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:32:56 smithi189 ceph-mon[36187]: pgmap v2833: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T01:32:56.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:32:56 smithi193 ceph-mon[36878]: pgmap v2833: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T01:32:58.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:32:58 smithi118 ceph-mon[31535]: pgmap v2834: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:32:58.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:32:58 smithi189 ceph-mon[36187]: pgmap v2834: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:32:58.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:32:58 smithi193 ceph-mon[36878]: pgmap v2834: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:33:00.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:33:00 smithi118 ceph-mon[31535]: pgmap v2835: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T01:33:00.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:33:00 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 4076.672896 secs 2024-03-24T01:33:00.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:33:00 smithi189 ceph-mon[36187]: pgmap v2835: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T01:33:00.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:33:00 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 4076.672896 secs 2024-03-24T01:33:00.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:33:00 smithi193 ceph-mon[36878]: pgmap v2835: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T01:33:00.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:33:00 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 4076.672896 secs 2024-03-24T01:33:02.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:33:02 smithi189 ceph-mon[36187]: pgmap v2836: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T01:33:02.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:33:02 smithi193 ceph-mon[36878]: pgmap v2836: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T01:33:03.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:33:02 smithi118 ceph-mon[31535]: pgmap v2836: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T01:33:04.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:33:04 smithi189 ceph-mon[36187]: pgmap v2837: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 5 op/s 2024-03-24T01:33:04.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:33:04 smithi193 ceph-mon[36878]: pgmap v2837: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 5 op/s 2024-03-24T01:33:05.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:33:04 smithi118 ceph-mon[31535]: pgmap v2837: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 5 op/s 2024-03-24T01:33:06.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:33:05 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 4081.672979 secs 2024-03-24T01:33:06.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:33:05 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 4081.672979 secs 2024-03-24T01:33:06.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:33:05 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 4081.672979 secs 2024-03-24T01:33:06.968 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:33:06 smithi118 ceph-mon[31535]: pgmap v2838: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T01:33:07.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:33:06 smithi189 ceph-mon[36187]: pgmap v2838: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T01:33:07.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:33:06 smithi193 ceph-mon[36878]: pgmap v2838: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T01:33:09.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:33:08 smithi118 ceph-mon[31535]: pgmap v2839: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:33:09.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:33:08 smithi189 ceph-mon[36187]: pgmap v2839: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:33:09.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:33:08 smithi193 ceph-mon[36878]: pgmap v2839: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:33:11.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:33:10 smithi118 ceph-mon[31535]: pgmap v2840: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T01:33:11.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:33:10 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 4086.673067 secs 2024-03-24T01:33:11.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:33:10 smithi189 ceph-mon[36187]: pgmap v2840: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T01:33:11.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:33:10 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 4086.673067 secs 2024-03-24T01:33:11.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:33:10 smithi193 ceph-mon[36878]: pgmap v2840: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T01:33:11.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:33:10 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 4086.673067 secs 2024-03-24T01:33:13.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:33:12 smithi118 ceph-mon[31535]: pgmap v2841: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T01:33:13.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:33:12 smithi189 ceph-mon[36187]: pgmap v2841: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T01:33:13.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:33:12 smithi193 ceph-mon[36878]: pgmap v2841: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T01:33:15.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:33:14 smithi118 ceph-mon[31535]: pgmap v2842: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 5 op/s 2024-03-24T01:33:15.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:33:14 smithi189 ceph-mon[36187]: pgmap v2842: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 5 op/s 2024-03-24T01:33:15.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:33:14 smithi193 ceph-mon[36878]: pgmap v2842: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 5 op/s 2024-03-24T01:33:16.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:33:15 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 4091.673128 secs 2024-03-24T01:33:16.205 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:33:15 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 4091.673128 secs 2024-03-24T01:33:16.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:33:15 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 4091.673128 secs 2024-03-24T01:33:17.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:33:16 smithi118 ceph-mon[31535]: pgmap v2843: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T01:33:17.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:33:16 smithi189 ceph-mon[36187]: pgmap v2843: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T01:33:17.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:33:16 smithi193 ceph-mon[36878]: pgmap v2843: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T01:33:19.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:33:18 smithi118 ceph-mon[31535]: pgmap v2844: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:33:19.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:33:18 smithi189 ceph-mon[36187]: pgmap v2844: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:33:19.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:33:18 smithi193 ceph-mon[36878]: pgmap v2844: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:33:20.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:33:19 smithi118 ceph-mon[31535]: pgmap v2845: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-03-24T01:33:20.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:33:19 smithi189 ceph-mon[36187]: pgmap v2845: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-03-24T01:33:20.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:33:19 smithi193 ceph-mon[36878]: pgmap v2845: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-03-24T01:33:21.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:33:20 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 4096.673190 secs 2024-03-24T01:33:21.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:33:20 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 4096.673190 secs 2024-03-24T01:33:21.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:33:20 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 4096.673190 secs 2024-03-24T01:33:22.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:33:21 smithi118 ceph-mon[31535]: pgmap v2846: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 5 op/s 2024-03-24T01:33:22.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:33:21 smithi189 ceph-mon[36187]: pgmap v2846: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 5 op/s 2024-03-24T01:33:22.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:33:21 smithi193 ceph-mon[36878]: pgmap v2846: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 5 op/s 2024-03-24T01:33:24.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:33:24 smithi118 ceph-mon[31535]: pgmap v2847: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 5 op/s 2024-03-24T01:33:24.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:33:24 smithi189 ceph-mon[36187]: pgmap v2847: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 5 op/s 2024-03-24T01:33:24.460 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:33:24 smithi193 ceph-mon[36878]: pgmap v2847: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 5 op/s 2024-03-24T01:33:26.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:33:26 smithi118 ceph-mon[31535]: pgmap v2848: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 6 op/s 2024-03-24T01:33:26.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:33:26 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 4101.673247 secs 2024-03-24T01:33:26.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:33:26 smithi189 ceph-mon[36187]: pgmap v2848: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 6 op/s 2024-03-24T01:33:26.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:33:26 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 4101.673247 secs 2024-03-24T01:33:26.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:33:26 smithi193 ceph-mon[36878]: pgmap v2848: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 6 op/s 2024-03-24T01:33:26.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:33:26 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 4101.673247 secs 2024-03-24T01:33:28.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:33:28 smithi118 ceph-mon[31535]: pgmap v2849: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-03-24T01:33:28.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:33:28 smithi189 ceph-mon[36187]: pgmap v2849: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-03-24T01:33:28.460 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:33:28 smithi193 ceph-mon[36878]: pgmap v2849: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-03-24T01:33:30.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:33:30 smithi118 ceph-mon[31535]: pgmap v2850: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-03-24T01:33:30.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:33:30 smithi189 ceph-mon[36187]: pgmap v2850: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-03-24T01:33:30.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:33:30 smithi193 ceph-mon[36878]: pgmap v2850: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-03-24T01:33:31.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:33:31 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 4106.673302 secs 2024-03-24T01:33:31.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:33:31 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 4106.673302 secs 2024-03-24T01:33:31.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:33:31 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 4106.673302 secs 2024-03-24T01:33:32.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:33:32 smithi118 ceph-mon[31535]: pgmap v2851: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T01:33:32.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:33:32 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T01:33:32.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:33:32 smithi189 ceph-mon[36187]: pgmap v2851: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T01:33:32.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:33:32 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T01:33:32.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:33:32 smithi193 ceph-mon[36878]: pgmap v2851: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T01:33:32.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:33:32 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T01:33:33.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:33:33 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T01:33:33.447 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:33:33 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T01:33:33.447 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:33:33 smithi118 ceph-mon[31535]: pgmap v2852: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 5 op/s 2024-03-24T01:33:33.447 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:33:33 smithi118 ceph-mon[31535]: pgmap v2853: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 5 op/s 2024-03-24T01:33:33.447 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:33:33 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T01:33:33.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:33:33 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T01:33:33.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:33:33 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T01:33:33.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:33:33 smithi189 ceph-mon[36187]: pgmap v2852: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 5 op/s 2024-03-24T01:33:33.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:33:33 smithi189 ceph-mon[36187]: pgmap v2853: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 5 op/s 2024-03-24T01:33:33.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:33:33 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T01:33:33.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:33:33 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T01:33:33.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:33:33 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T01:33:33.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:33:33 smithi193 ceph-mon[36878]: pgmap v2852: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 5 op/s 2024-03-24T01:33:33.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:33:33 smithi193 ceph-mon[36878]: pgmap v2853: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 5 op/s 2024-03-24T01:33:33.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:33:33 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T01:33:35.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:33:35 smithi118 ceph-mon[31535]: pgmap v2854: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 6 op/s 2024-03-24T01:33:35.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:33:35 smithi189 ceph-mon[36187]: pgmap v2854: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 6 op/s 2024-03-24T01:33:35.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:33:35 smithi193 ceph-mon[36878]: pgmap v2854: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 6 op/s 2024-03-24T01:33:36.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:33:36 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 4111.673403 secs 2024-03-24T01:33:36.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:33:36 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 4111.673403 secs 2024-03-24T01:33:36.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:33:36 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 4111.673403 secs 2024-03-24T01:33:37.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:33:37 smithi118 ceph-mon[31535]: pgmap v2855: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 6 op/s 2024-03-24T01:33:37.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:33:37 smithi189 ceph-mon[36187]: pgmap v2855: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 6 op/s 2024-03-24T01:33:37.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:33:37 smithi193 ceph-mon[36878]: pgmap v2855: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 6 op/s 2024-03-24T01:33:39.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:33:39 smithi118 ceph-mon[31535]: pgmap v2856: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 5 op/s 2024-03-24T01:33:39.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:33:39 smithi189 ceph-mon[36187]: pgmap v2856: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 5 op/s 2024-03-24T01:33:39.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:33:39 smithi193 ceph-mon[36878]: pgmap v2856: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 5 op/s 2024-03-24T01:33:41.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:33:41 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 4116.673466 secs 2024-03-24T01:33:41.947 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:33:41 smithi118 ceph-mon[31535]: pgmap v2857: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 6 op/s 2024-03-24T01:33:41.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:33:41 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 4116.673466 secs 2024-03-24T01:33:41.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:33:41 smithi189 ceph-mon[36187]: pgmap v2857: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 6 op/s 2024-03-24T01:33:41.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:33:41 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 4116.673466 secs 2024-03-24T01:33:41.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:33:41 smithi193 ceph-mon[36878]: pgmap v2857: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 6 op/s 2024-03-24T01:33:43.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:33:43 smithi118 ceph-mon[31535]: pgmap v2858: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 5 op/s 2024-03-24T01:33:43.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:33:43 smithi189 ceph-mon[36187]: pgmap v2858: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 5 op/s 2024-03-24T01:33:43.962 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:33:43 smithi193 ceph-mon[36878]: pgmap v2858: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 5 op/s 2024-03-24T01:33:45.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:33:45 smithi118 ceph-mon[31535]: pgmap v2859: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-03-24T01:33:45.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:33:45 smithi189 ceph-mon[36187]: pgmap v2859: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-03-24T01:33:45.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:33:45 smithi193 ceph-mon[36878]: pgmap v2859: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-03-24T01:33:46.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:33:46 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 4121.673525 secs 2024-03-24T01:33:46.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:33:46 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 4121.673525 secs 2024-03-24T01:33:46.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:33:46 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 4121.673525 secs 2024-03-24T01:33:47.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:33:47 smithi118 ceph-mon[31535]: pgmap v2860: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:33:47.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:33:47 smithi189 ceph-mon[36187]: pgmap v2860: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:33:47.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:33:47 smithi193 ceph-mon[36878]: pgmap v2860: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:33:49.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:33:49 smithi118 ceph-mon[31535]: pgmap v2861: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T01:33:49.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:33:49 smithi189 ceph-mon[36187]: pgmap v2861: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T01:33:49.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:33:49 smithi193 ceph-mon[36878]: pgmap v2861: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T01:33:50.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:33:50 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 4126.673616 secs 2024-03-24T01:33:50.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:33:50 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 4126.673616 secs 2024-03-24T01:33:50.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:33:50 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 4126.673616 secs 2024-03-24T01:33:51.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:33:51 smithi118 ceph-mon[31535]: pgmap v2862: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T01:33:51.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:33:51 smithi189 ceph-mon[36187]: pgmap v2862: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T01:33:51.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:33:51 smithi193 ceph-mon[36878]: pgmap v2862: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T01:33:53.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:33:53 smithi118 ceph-mon[31535]: pgmap v2863: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T01:33:53.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:33:53 smithi189 ceph-mon[36187]: pgmap v2863: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T01:33:53.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:33:53 smithi193 ceph-mon[36878]: pgmap v2863: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T01:33:55.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:33:55 smithi118 ceph-mon[31535]: pgmap v2864: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-03-24T01:33:55.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:33:55 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 4131.673677 secs 2024-03-24T01:33:55.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:33:55 smithi189 ceph-mon[36187]: pgmap v2864: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-03-24T01:33:55.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:33:55 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 4131.673677 secs 2024-03-24T01:33:55.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:33:55 smithi193 ceph-mon[36878]: pgmap v2864: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-03-24T01:33:55.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:33:55 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 4131.673677 secs 2024-03-24T01:33:57.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:33:57 smithi118 ceph-mon[31535]: pgmap v2865: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:33:57.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:33:57 smithi189 ceph-mon[36187]: pgmap v2865: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:33:57.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:33:57 smithi193 ceph-mon[36878]: pgmap v2865: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:33:59.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:33:59 smithi118 ceph-mon[31535]: pgmap v2866: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T01:33:59.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:33:59 smithi189 ceph-mon[36187]: pgmap v2866: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T01:33:59.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:33:59 smithi193 ceph-mon[36878]: pgmap v2866: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T01:34:00.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:34:00 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 4136.673736 secs 2024-03-24T01:34:00.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:34:00 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 4136.673736 secs 2024-03-24T01:34:00.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:34:00 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 4136.673736 secs 2024-03-24T01:34:01.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:34:01 smithi118 ceph-mon[31535]: pgmap v2867: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T01:34:01.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:34:01 smithi189 ceph-mon[36187]: pgmap v2867: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T01:34:01.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:34:01 smithi193 ceph-mon[36878]: pgmap v2867: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T01:34:03.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:34:03 smithi118 ceph-mon[31535]: pgmap v2868: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T01:34:03.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:34:03 smithi189 ceph-mon[36187]: pgmap v2868: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T01:34:03.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:34:03 smithi193 ceph-mon[36878]: pgmap v2868: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T01:34:05.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:34:05 smithi118 ceph-mon[31535]: pgmap v2869: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-03-24T01:34:05.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:34:05 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 4141.673798 secs 2024-03-24T01:34:05.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:34:05 smithi189 ceph-mon[36187]: pgmap v2869: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-03-24T01:34:05.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:34:05 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 4141.673798 secs 2024-03-24T01:34:05.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:34:05 smithi193 ceph-mon[36878]: pgmap v2869: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-03-24T01:34:05.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:34:05 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 4141.673798 secs 2024-03-24T01:34:07.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:34:07 smithi118 ceph-mon[31535]: pgmap v2870: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:34:07.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:34:07 smithi189 ceph-mon[36187]: pgmap v2870: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:34:07.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:34:07 smithi193 ceph-mon[36878]: pgmap v2870: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:34:09.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:34:09 smithi118 ceph-mon[31535]: pgmap v2871: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T01:34:09.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:34:09 smithi189 ceph-mon[36187]: pgmap v2871: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T01:34:09.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:34:09 smithi193 ceph-mon[36878]: pgmap v2871: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T01:34:10.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:34:10 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 4146.673882 secs 2024-03-24T01:34:10.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:34:10 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 4146.673882 secs 2024-03-24T01:34:10.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:34:10 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 4146.673882 secs 2024-03-24T01:34:11.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:34:11 smithi118 ceph-mon[31535]: pgmap v2872: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T01:34:11.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:34:11 smithi189 ceph-mon[36187]: pgmap v2872: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T01:34:11.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:34:11 smithi193 ceph-mon[36878]: pgmap v2872: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T01:34:13.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:34:13 smithi118 ceph-mon[31535]: pgmap v2873: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T01:34:13.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:34:13 smithi189 ceph-mon[36187]: pgmap v2873: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T01:34:13.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:34:13 smithi193 ceph-mon[36878]: pgmap v2873: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T01:34:15.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:34:15 smithi118 ceph-mon[31535]: pgmap v2874: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-03-24T01:34:15.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:34:15 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 4151.673969 secs 2024-03-24T01:34:15.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:34:15 smithi189 ceph-mon[36187]: pgmap v2874: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-03-24T01:34:15.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:34:15 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 4151.673969 secs 2024-03-24T01:34:15.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:34:15 smithi193 ceph-mon[36878]: pgmap v2874: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-03-24T01:34:15.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:34:15 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 4151.673969 secs 2024-03-24T01:34:16.395 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T01:34:16.395+0000 7f81ee000640 1 -- 172.21.15.118:0/1693767697 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f81c80036c0 con 0x7f81f81052d0 2024-03-24T01:34:16.396 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T01:34:16.395+0000 7f81db400640 1 -- 172.21.15.118:0/659686424 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f81c40064d0 con 0x5622651a13f0 2024-03-24T01:34:16.396 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T01:34:16.396+0000 7f81e4c00640 1 -- 172.21.15.118:0/659686424 <== mon.1 v2:172.21.15.193:3300/0 36 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 409+0+0 (secure 0 0 0) 0x7f81c40064d0 con 0x5622651a13f0 2024-03-24T01:34:16.396 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T01:34:16.396+0000 7f81ef400640 1 -- 172.21.15.118:0/1693767697 <== mon.2 v2:172.21.15.189:3300/0 36 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (secure 0 0 0) 0x7f81c80036c0 con 0x7f81f81052d0 2024-03-24T01:34:17.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:34:17 smithi118 ceph-mon[31535]: pgmap v2875: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:34:17.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:34:17 smithi189 ceph-mon[36187]: pgmap v2875: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:34:17.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:34:17 smithi193 ceph-mon[36878]: pgmap v2875: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:34:19.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:34:19 smithi118 ceph-mon[31535]: pgmap v2876: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T01:34:19.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:34:19 smithi189 ceph-mon[36187]: pgmap v2876: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T01:34:19.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:34:19 smithi193 ceph-mon[36878]: pgmap v2876: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T01:34:20.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:34:20 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 4156.674030 secs 2024-03-24T01:34:20.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:34:20 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 4156.674030 secs 2024-03-24T01:34:20.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:34:20 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 4156.674030 secs 2024-03-24T01:34:21.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:34:21 smithi118 ceph-mon[31535]: pgmap v2877: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T01:34:21.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:34:21 smithi189 ceph-mon[36187]: pgmap v2877: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T01:34:21.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:34:21 smithi193 ceph-mon[36878]: pgmap v2877: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T01:34:23.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:34:23 smithi189 ceph-mon[36187]: pgmap v2878: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T01:34:23.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:34:23 smithi193 ceph-mon[36878]: pgmap v2878: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T01:34:24.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:34:23 smithi118 ceph-mon[31535]: pgmap v2878: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T01:34:25.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:34:25 smithi189 ceph-mon[36187]: pgmap v2879: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-03-24T01:34:25.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:34:25 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 4161.674094 secs 2024-03-24T01:34:25.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:34:25 smithi193 ceph-mon[36878]: pgmap v2879: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-03-24T01:34:25.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:34:25 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 4161.674094 secs 2024-03-24T01:34:26.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:34:25 smithi118 ceph-mon[31535]: pgmap v2879: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-03-24T01:34:26.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:34:25 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 4161.674094 secs 2024-03-24T01:34:27.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:34:27 smithi193 ceph-mon[36878]: pgmap v2880: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:34:28.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:34:27 smithi118 ceph-mon[31535]: pgmap v2880: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:34:28.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:34:27 smithi189 ceph-mon[36187]: pgmap v2880: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:34:29.962 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:34:29 smithi193 ceph-mon[36878]: pgmap v2881: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T01:34:30.006 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:34:29 smithi189 ceph-mon[36187]: pgmap v2881: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T01:34:30.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:34:29 smithi118 ceph-mon[31535]: pgmap v2881: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T01:34:31.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:34:30 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 4166.674154 secs 2024-03-24T01:34:31.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:34:30 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 4166.674154 secs 2024-03-24T01:34:31.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:34:30 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 4166.674154 secs 2024-03-24T01:34:32.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:34:31 smithi118 ceph-mon[31535]: pgmap v2882: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T01:34:32.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:34:31 smithi189 ceph-mon[36187]: pgmap v2882: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T01:34:32.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:34:31 smithi193 ceph-mon[36878]: pgmap v2882: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T01:34:34.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:34:33 smithi118 ceph-mon[31535]: pgmap v2883: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T01:34:34.197 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:34:33 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T01:34:34.197 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:34:33 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-24T01:34:34.197 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:34:33 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-24T01:34:34.197 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:34:33 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-24T01:34:34.197 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:34:33 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-03-24T01:34:34.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:34:33 smithi189 ceph-mon[36187]: pgmap v2883: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T01:34:34.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:34:33 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T01:34:34.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:34:33 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-24T01:34:34.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:34:33 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-24T01:34:34.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:34:33 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-24T01:34:34.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:34:33 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-03-24T01:34:34.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:34:33 smithi193 ceph-mon[36878]: pgmap v2883: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T01:34:34.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:34:33 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T01:34:34.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:34:33 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-24T01:34:34.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:34:33 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-24T01:34:34.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:34:33 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-24T01:34:34.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:34:33 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-03-24T01:34:35.197 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:34:34 smithi118 ceph-mon[31535]: Adjusting osd_memory_target on smithi189 to 174.1M 2024-03-24T01:34:35.197 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:34:34 smithi118 ceph-mon[31535]: Unable to set osd_memory_target on smithi189 to 182635622: error parsing value: Value '182635622' is below minimum 939524096 2024-03-24T01:34:35.197 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:34:34 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-24T01:34:35.197 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:34:34 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-24T01:34:35.197 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:34:34 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-24T01:34:35.197 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:34:34 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-03-24T01:34:35.198 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:34:34 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T01:34:35.198 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:34:34 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T01:34:35.198 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:34:34 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T01:34:35.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:34:34 smithi189 ceph-mon[36187]: Adjusting osd_memory_target on smithi189 to 174.1M 2024-03-24T01:34:35.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:34:34 smithi189 ceph-mon[36187]: Unable to set osd_memory_target on smithi189 to 182635622: error parsing value: Value '182635622' is below minimum 939524096 2024-03-24T01:34:35.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:34:34 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-24T01:34:35.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:34:34 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-24T01:34:35.205 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:34:34 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-24T01:34:35.205 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:34:34 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-03-24T01:34:35.205 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:34:34 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T01:34:35.205 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:34:34 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T01:34:35.205 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:34:34 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T01:34:35.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:34:34 smithi193 ceph-mon[36878]: Adjusting osd_memory_target on smithi189 to 174.1M 2024-03-24T01:34:35.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:34:34 smithi193 ceph-mon[36878]: Unable to set osd_memory_target on smithi189 to 182635622: error parsing value: Value '182635622' is below minimum 939524096 2024-03-24T01:34:35.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:34:34 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-24T01:34:35.212 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:34:34 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-24T01:34:35.212 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:34:34 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-24T01:34:35.212 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:34:34 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-03-24T01:34:35.212 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:34:34 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T01:34:35.212 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:34:34 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T01:34:35.212 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:34:34 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T01:34:36.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:34:35 smithi118 ceph-mon[31535]: Adjusting osd_memory_target on smithi118 to 176.7M 2024-03-24T01:34:36.197 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:34:35 smithi118 ceph-mon[31535]: Unable to set osd_memory_target on smithi118 to 185365196: error parsing value: Value '185365196' is below minimum 939524096 2024-03-24T01:34:36.197 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:34:35 smithi118 ceph-mon[31535]: pgmap v2884: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 6 op/s 2024-03-24T01:34:36.197 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:34:35 smithi118 ceph-mon[31535]: pgmap v2885: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 5 op/s 2024-03-24T01:34:36.197 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:34:35 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 4171.674211 secs 2024-03-24T01:34:36.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:34:35 smithi189 ceph-mon[36187]: Adjusting osd_memory_target on smithi118 to 176.7M 2024-03-24T01:34:36.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:34:35 smithi189 ceph-mon[36187]: Unable to set osd_memory_target on smithi118 to 185365196: error parsing value: Value '185365196' is below minimum 939524096 2024-03-24T01:34:36.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:34:35 smithi189 ceph-mon[36187]: pgmap v2884: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 6 op/s 2024-03-24T01:34:36.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:34:35 smithi189 ceph-mon[36187]: pgmap v2885: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 5 op/s 2024-03-24T01:34:36.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:34:35 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 4171.674211 secs 2024-03-24T01:34:36.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:34:35 smithi193 ceph-mon[36878]: Adjusting osd_memory_target on smithi118 to 176.7M 2024-03-24T01:34:36.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:34:35 smithi193 ceph-mon[36878]: Unable to set osd_memory_target on smithi118 to 185365196: error parsing value: Value '185365196' is below minimum 939524096 2024-03-24T01:34:36.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:34:35 smithi193 ceph-mon[36878]: pgmap v2884: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 6 op/s 2024-03-24T01:34:36.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:34:35 smithi193 ceph-mon[36878]: pgmap v2885: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 5 op/s 2024-03-24T01:34:36.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:34:35 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 4171.674211 secs 2024-03-24T01:34:38.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:34:37 smithi118 ceph-mon[31535]: pgmap v2886: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-03-24T01:34:38.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:34:37 smithi189 ceph-mon[36187]: pgmap v2886: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-03-24T01:34:38.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:34:37 smithi193 ceph-mon[36878]: pgmap v2886: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-03-24T01:34:40.153 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:34:39 smithi118 ceph-mon[31535]: pgmap v2887: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 5 op/s 2024-03-24T01:34:40.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:34:39 smithi189 ceph-mon[36187]: pgmap v2887: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 5 op/s 2024-03-24T01:34:40.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:34:39 smithi193 ceph-mon[36878]: pgmap v2887: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 5 op/s 2024-03-24T01:34:41.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:34:40 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 4176.674273 secs 2024-03-24T01:34:41.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:34:40 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 4176.674273 secs 2024-03-24T01:34:41.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:34:40 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 4176.674273 secs 2024-03-24T01:34:42.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:34:41 smithi118 ceph-mon[31535]: pgmap v2888: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-03-24T01:34:42.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:34:41 smithi189 ceph-mon[36187]: pgmap v2888: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-03-24T01:34:42.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:34:41 smithi193 ceph-mon[36878]: pgmap v2888: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-03-24T01:34:43.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:34:42 smithi118 ceph-mon[31535]: pgmap v2889: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 5 op/s 2024-03-24T01:34:43.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:34:42 smithi189 ceph-mon[36187]: pgmap v2889: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 5 op/s 2024-03-24T01:34:43.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:34:42 smithi193 ceph-mon[36878]: pgmap v2889: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 5 op/s 2024-03-24T01:34:45.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:34:45 smithi118 ceph-mon[31535]: pgmap v2890: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 5 op/s 2024-03-24T01:34:45.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:34:45 smithi189 ceph-mon[36187]: pgmap v2890: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 5 op/s 2024-03-24T01:34:45.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:34:45 smithi193 ceph-mon[36878]: pgmap v2890: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 5 op/s 2024-03-24T01:34:46.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:34:46 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 4181.674336 secs 2024-03-24T01:34:46.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:34:46 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 4181.674336 secs 2024-03-24T01:34:46.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:34:46 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 4181.674336 secs 2024-03-24T01:34:47.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:34:47 smithi118 ceph-mon[31535]: pgmap v2891: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T01:34:47.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:34:47 smithi189 ceph-mon[36187]: pgmap v2891: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T01:34:47.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:34:47 smithi193 ceph-mon[36878]: pgmap v2891: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T01:34:49.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:34:49 smithi118 ceph-mon[31535]: pgmap v2892: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 5 op/s 2024-03-24T01:34:49.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:34:49 smithi189 ceph-mon[36187]: pgmap v2892: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 5 op/s 2024-03-24T01:34:49.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:34:49 smithi193 ceph-mon[36878]: pgmap v2892: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 5 op/s 2024-03-24T01:34:51.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:34:51 smithi118 ceph-mon[31535]: pgmap v2893: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T01:34:51.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:34:51 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 4186.674397 secs 2024-03-24T01:34:51.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:34:51 smithi189 ceph-mon[36187]: pgmap v2893: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T01:34:51.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:34:51 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 4186.674397 secs 2024-03-24T01:34:51.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:34:51 smithi193 ceph-mon[36878]: pgmap v2893: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T01:34:51.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:34:51 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 4186.674397 secs 2024-03-24T01:34:53.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:34:53 smithi118 ceph-mon[31535]: pgmap v2894: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:34:53.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:34:53 smithi189 ceph-mon[36187]: pgmap v2894: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:34:53.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:34:53 smithi193 ceph-mon[36878]: pgmap v2894: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:34:55.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:34:55 smithi118 ceph-mon[31535]: pgmap v2895: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T01:34:55.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:34:55 smithi189 ceph-mon[36187]: pgmap v2895: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T01:34:55.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:34:55 smithi193 ceph-mon[36878]: pgmap v2895: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T01:34:56.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:34:56 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 4191.674483 secs 2024-03-24T01:34:56.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:34:56 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 4191.674483 secs 2024-03-24T01:34:56.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:34:56 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 4191.674483 secs 2024-03-24T01:34:57.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:34:57 smithi118 ceph-mon[31535]: pgmap v2896: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T01:34:57.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:34:57 smithi189 ceph-mon[36187]: pgmap v2896: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T01:34:57.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:34:57 smithi193 ceph-mon[36878]: pgmap v2896: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T01:34:59.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:34:59 smithi118 ceph-mon[31535]: pgmap v2897: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 5 op/s 2024-03-24T01:34:59.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:34:59 smithi189 ceph-mon[36187]: pgmap v2897: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 5 op/s 2024-03-24T01:34:59.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:34:59 smithi193 ceph-mon[36878]: pgmap v2897: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 5 op/s 2024-03-24T01:35:01.447 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:35:01 smithi118 ceph-mon[31535]: pgmap v2898: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T01:35:01.447 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:35:01 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 4196.674572 secs 2024-03-24T01:35:01.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:35:01 smithi189 ceph-mon[36187]: pgmap v2898: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T01:35:01.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:35:01 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 4196.674572 secs 2024-03-24T01:35:01.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:35:01 smithi193 ceph-mon[36878]: pgmap v2898: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T01:35:01.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:35:01 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 4196.674572 secs 2024-03-24T01:35:03.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:35:03 smithi118 ceph-mon[31535]: pgmap v2899: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:35:03.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:35:03 smithi189 ceph-mon[36187]: pgmap v2899: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:35:03.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:35:03 smithi193 ceph-mon[36878]: pgmap v2899: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:35:05.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:35:05 smithi118 ceph-mon[31535]: pgmap v2900: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T01:35:05.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:35:05 smithi189 ceph-mon[36187]: pgmap v2900: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T01:35:05.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:35:05 smithi193 ceph-mon[36878]: pgmap v2900: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T01:35:06.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:35:06 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 4201.674634 secs 2024-03-24T01:35:06.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:35:06 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 4201.674634 secs 2024-03-24T01:35:06.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:35:06 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 4201.674634 secs 2024-03-24T01:35:07.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:35:07 smithi118 ceph-mon[31535]: pgmap v2901: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T01:35:07.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:35:07 smithi189 ceph-mon[36187]: pgmap v2901: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T01:35:07.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:35:07 smithi193 ceph-mon[36878]: pgmap v2901: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T01:35:09.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:35:09 smithi118 ceph-mon[31535]: pgmap v2902: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 5 op/s 2024-03-24T01:35:09.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:35:09 smithi189 ceph-mon[36187]: pgmap v2902: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 5 op/s 2024-03-24T01:35:09.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:35:09 smithi193 ceph-mon[36878]: pgmap v2902: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 5 op/s 2024-03-24T01:35:11.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:35:11 smithi118 ceph-mon[31535]: pgmap v2903: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T01:35:11.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:35:11 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 4206.674694 secs 2024-03-24T01:35:11.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:35:11 smithi189 ceph-mon[36187]: pgmap v2903: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T01:35:11.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:35:11 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 4206.674694 secs 2024-03-24T01:35:11.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:35:11 smithi193 ceph-mon[36878]: pgmap v2903: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T01:35:11.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:35:11 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 4206.674694 secs 2024-03-24T01:35:13.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:35:13 smithi118 ceph-mon[31535]: pgmap v2904: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:35:13.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:35:13 smithi189 ceph-mon[36187]: pgmap v2904: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:35:13.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:35:13 smithi193 ceph-mon[36878]: pgmap v2904: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:35:15.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:35:15 smithi118 ceph-mon[31535]: pgmap v2905: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T01:35:15.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:35:15 smithi189 ceph-mon[36187]: pgmap v2905: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T01:35:15.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:35:15 smithi193 ceph-mon[36878]: pgmap v2905: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T01:35:16.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:35:16 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 4211.674783 secs 2024-03-24T01:35:16.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:35:16 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 4211.674783 secs 2024-03-24T01:35:16.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:35:16 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 4211.674783 secs 2024-03-24T01:35:17.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:35:17 smithi118 ceph-mon[31535]: pgmap v2906: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T01:35:17.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:35:17 smithi189 ceph-mon[36187]: pgmap v2906: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T01:35:17.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:35:17 smithi193 ceph-mon[36878]: pgmap v2906: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T01:35:19.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:35:19 smithi118 ceph-mon[31535]: pgmap v2907: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 5 op/s 2024-03-24T01:35:19.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:35:19 smithi189 ceph-mon[36187]: pgmap v2907: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 5 op/s 2024-03-24T01:35:19.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:35:19 smithi193 ceph-mon[36878]: pgmap v2907: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 5 op/s 2024-03-24T01:35:21.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:35:21 smithi118 ceph-mon[31535]: pgmap v2908: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T01:35:21.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:35:21 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 4216.674871 secs 2024-03-24T01:35:21.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:35:21 smithi189 ceph-mon[36187]: pgmap v2908: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T01:35:21.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:35:21 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 4216.674871 secs 2024-03-24T01:35:21.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:35:21 smithi193 ceph-mon[36878]: pgmap v2908: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T01:35:21.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:35:21 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 4216.674871 secs 2024-03-24T01:35:23.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:35:23 smithi118 ceph-mon[31535]: pgmap v2909: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:35:23.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:35:23 smithi189 ceph-mon[36187]: pgmap v2909: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:35:23.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:35:23 smithi193 ceph-mon[36878]: pgmap v2909: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:35:25.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:35:25 smithi118 ceph-mon[31535]: pgmap v2910: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T01:35:25.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:35:25 smithi189 ceph-mon[36187]: pgmap v2910: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T01:35:25.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:35:25 smithi193 ceph-mon[36878]: pgmap v2910: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T01:35:26.462 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:35:26 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 4221.674959 secs 2024-03-24T01:35:26.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:35:26 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 4221.674959 secs 2024-03-24T01:35:26.704 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:35:26 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 4221.674959 secs 2024-03-24T01:35:27.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:35:27 smithi193 ceph-mon[36878]: pgmap v2911: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T01:35:27.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:35:27 smithi118 ceph-mon[31535]: pgmap v2911: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T01:35:27.704 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:35:27 smithi189 ceph-mon[36187]: pgmap v2911: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T01:35:29.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:35:29 smithi193 ceph-mon[36878]: pgmap v2912: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 5 op/s 2024-03-24T01:35:29.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:35:29 smithi118 ceph-mon[31535]: pgmap v2912: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 5 op/s 2024-03-24T01:35:29.704 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:35:29 smithi189 ceph-mon[36187]: pgmap v2912: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 5 op/s 2024-03-24T01:35:31.507 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:35:31 smithi189 ceph-mon[36187]: pgmap v2913: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T01:35:31.507 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:35:31 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 4226.675001 secs 2024-03-24T01:35:31.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:35:31 smithi118 ceph-mon[31535]: pgmap v2913: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T01:35:31.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:35:31 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 4226.675001 secs 2024-03-24T01:35:31.711 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:35:31 smithi193 ceph-mon[36878]: pgmap v2913: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T01:35:31.711 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:35:31 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 4226.675001 secs 2024-03-24T01:35:33.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:35:33 smithi118 ceph-mon[31535]: pgmap v2914: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:35:33.704 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:35:33 smithi189 ceph-mon[36187]: pgmap v2914: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:35:33.711 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:35:33 smithi193 ceph-mon[36878]: pgmap v2914: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:35:35.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:35:35 smithi118 ceph-mon[31535]: pgmap v2915: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T01:35:35.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:35:35 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T01:35:35.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:35:35 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-24T01:35:35.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:35:35 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-24T01:35:35.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:35:35 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-03-24T01:35:35.697 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:35:35 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-03-24T01:35:35.704 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:35:35 smithi189 ceph-mon[36187]: pgmap v2915: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T01:35:35.704 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:35:35 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T01:35:35.704 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:35:35 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-24T01:35:35.704 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:35:35 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-24T01:35:35.704 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:35:35 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-03-24T01:35:35.704 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:35:35 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-03-24T01:35:35.711 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:35:35 smithi193 ceph-mon[36878]: pgmap v2915: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T01:35:35.711 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:35:35 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T01:35:35.711 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:35:35 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-24T01:35:35.711 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:35:35 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-24T01:35:35.711 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:35:35 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-03-24T01:35:35.711 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:35:35 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-03-24T01:35:36.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:35:36 smithi118 ceph-mon[31535]: Adjusting osd_memory_target on smithi193 to 174.1M 2024-03-24T01:35:36.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:35:36 smithi118 ceph-mon[31535]: Unable to set osd_memory_target on smithi193 to 182639923: error parsing value: Value '182639923' is below minimum 939524096 2024-03-24T01:35:36.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:35:36 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 4231.675086 secs 2024-03-24T01:35:36.704 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:35:36 smithi189 ceph-mon[36187]: Adjusting osd_memory_target on smithi193 to 174.1M 2024-03-24T01:35:36.704 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:35:36 smithi189 ceph-mon[36187]: Unable to set osd_memory_target on smithi193 to 182639923: error parsing value: Value '182639923' is below minimum 939524096 2024-03-24T01:35:36.704 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:35:36 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 4231.675086 secs 2024-03-24T01:35:36.711 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:35:36 smithi193 ceph-mon[36878]: Adjusting osd_memory_target on smithi193 to 174.1M 2024-03-24T01:35:36.711 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:35:36 smithi193 ceph-mon[36878]: Unable to set osd_memory_target on smithi193 to 182639923: error parsing value: Value '182639923' is below minimum 939524096 2024-03-24T01:35:36.711 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:35:36 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 4231.675086 secs 2024-03-24T01:35:37.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:35:37 smithi118 ceph-mon[31535]: pgmap v2916: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T01:35:37.697 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:35:37 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T01:35:37.697 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:35:37 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T01:35:37.697 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:35:37 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T01:35:37.697 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:35:37 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T01:35:37.697 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:35:37 smithi118 ceph-mon[31535]: pgmap v2917: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T01:35:37.697 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:35:37 smithi118 ceph-mon[31535]: pgmap v2918: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 6 op/s 2024-03-24T01:35:37.697 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:35:37 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T01:35:37.704 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:35:37 smithi189 ceph-mon[36187]: pgmap v2916: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T01:35:37.704 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:35:37 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T01:35:37.704 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:35:37 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T01:35:37.704 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:35:37 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T01:35:37.704 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:35:37 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T01:35:37.704 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:35:37 smithi189 ceph-mon[36187]: pgmap v2917: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T01:35:37.704 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:35:37 smithi189 ceph-mon[36187]: pgmap v2918: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 6 op/s 2024-03-24T01:35:37.704 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:35:37 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T01:35:37.711 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:35:37 smithi193 ceph-mon[36878]: pgmap v2916: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T01:35:37.711 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:35:37 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T01:35:37.711 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:35:37 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T01:35:37.711 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:35:37 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T01:35:37.711 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:35:37 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T01:35:37.711 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:35:37 smithi193 ceph-mon[36878]: pgmap v2917: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T01:35:37.711 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:35:37 smithi193 ceph-mon[36878]: pgmap v2918: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 6 op/s 2024-03-24T01:35:37.711 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:35:37 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T01:35:39.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:35:39 smithi118 ceph-mon[31535]: pgmap v2919: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 5 op/s 2024-03-24T01:35:39.704 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:35:39 smithi189 ceph-mon[36187]: pgmap v2919: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 5 op/s 2024-03-24T01:35:39.711 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:35:39 smithi193 ceph-mon[36878]: pgmap v2919: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 5 op/s 2024-03-24T01:35:41.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:35:41 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 4236.675145 secs 2024-03-24T01:35:41.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:35:41 smithi118 ceph-mon[31535]: pgmap v2920: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-03-24T01:35:41.704 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:35:41 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 4236.675145 secs 2024-03-24T01:35:41.704 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:35:41 smithi189 ceph-mon[36187]: pgmap v2920: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-03-24T01:35:41.711 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:35:41 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 4236.675145 secs 2024-03-24T01:35:41.711 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:35:41 smithi193 ceph-mon[36878]: pgmap v2920: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-03-24T01:35:43.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:35:43 smithi118 ceph-mon[31535]: pgmap v2921: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-03-24T01:35:43.704 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:35:43 smithi189 ceph-mon[36187]: pgmap v2921: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-03-24T01:35:43.711 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:35:43 smithi193 ceph-mon[36878]: pgmap v2921: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-03-24T01:35:45.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:35:45 smithi118 ceph-mon[31535]: pgmap v2922: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-03-24T01:35:45.704 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:35:45 smithi189 ceph-mon[36187]: pgmap v2922: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-03-24T01:35:45.711 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:35:45 smithi193 ceph-mon[36878]: pgmap v2922: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-03-24T01:35:46.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:35:46 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 4241.675230 secs 2024-03-24T01:35:46.704 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:35:46 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 4241.675230 secs 2024-03-24T01:35:46.711 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:35:46 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 4241.675230 secs 2024-03-24T01:35:47.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:35:47 smithi118 ceph-mon[31535]: pgmap v2923: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 5 op/s 2024-03-24T01:35:47.704 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:35:47 smithi189 ceph-mon[36187]: pgmap v2923: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 5 op/s 2024-03-24T01:35:47.711 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:35:47 smithi193 ceph-mon[36878]: pgmap v2923: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 5 op/s 2024-03-24T01:35:49.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:35:49 smithi118 ceph-mon[31535]: pgmap v2924: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T01:35:49.704 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:35:49 smithi189 ceph-mon[36187]: pgmap v2924: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T01:35:49.711 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:35:49 smithi193 ceph-mon[36878]: pgmap v2924: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T01:35:51.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:35:51 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 4246.675246 secs 2024-03-24T01:35:51.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:35:51 smithi118 ceph-mon[31535]: pgmap v2925: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T01:35:51.704 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:35:51 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 4246.675246 secs 2024-03-24T01:35:51.704 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:35:51 smithi189 ceph-mon[36187]: pgmap v2925: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T01:35:51.711 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:35:51 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 4246.675246 secs 2024-03-24T01:35:51.711 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:35:51 smithi193 ceph-mon[36878]: pgmap v2925: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T01:35:53.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:35:53 smithi118 ceph-mon[31535]: pgmap v2926: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:35:53.704 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:35:53 smithi189 ceph-mon[36187]: pgmap v2926: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:35:53.711 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:35:53 smithi193 ceph-mon[36878]: pgmap v2926: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:35:55.697 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:35:55 smithi118 ceph-mon[31535]: pgmap v2927: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-24T01:35:55.704 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:35:55 smithi189 ceph-mon[36187]: pgmap v2927: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-24T01:35:55.711 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:35:55 smithi193 ceph-mon[36878]: pgmap v2927: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-24T01:35:56.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:35:56 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 4251.675330 secs 2024-03-24T01:35:56.704 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:35:56 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 4251.675330 secs 2024-03-24T01:35:56.711 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:35:56 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 4251.675330 secs 2024-03-24T01:35:57.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:35:57 smithi118 ceph-mon[31535]: pgmap v2928: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:35:57.704 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:35:57 smithi189 ceph-mon[36187]: pgmap v2928: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:35:57.711 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:35:57 smithi193 ceph-mon[36878]: pgmap v2928: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:35:59.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:35:59 smithi118 ceph-mon[31535]: pgmap v2929: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T01:35:59.704 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:35:59 smithi189 ceph-mon[36187]: pgmap v2929: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T01:35:59.711 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:35:59 smithi193 ceph-mon[36878]: pgmap v2929: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T01:36:01.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:36:01 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 4256.675393 secs 2024-03-24T01:36:01.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:36:01 smithi118 ceph-mon[31535]: pgmap v2930: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T01:36:01.704 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:36:01 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 4256.675393 secs 2024-03-24T01:36:01.704 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:36:01 smithi189 ceph-mon[36187]: pgmap v2930: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T01:36:01.711 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:36:01 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 4256.675393 secs 2024-03-24T01:36:01.711 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:36:01 smithi193 ceph-mon[36878]: pgmap v2930: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T01:36:03.704 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:36:03 smithi189 ceph-mon[36187]: pgmap v2931: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:36:03.711 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:36:03 smithi193 ceph-mon[36878]: pgmap v2931: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:36:03.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:36:03 smithi118 ceph-mon[31535]: pgmap v2931: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:36:05.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:36:05 smithi118 ceph-mon[31535]: pgmap v2932: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-24T01:36:05.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:36:05 smithi189 ceph-mon[36187]: pgmap v2932: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-24T01:36:05.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:36:05 smithi193 ceph-mon[36878]: pgmap v2932: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-24T01:36:06.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:36:06 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 4261.675482 secs 2024-03-24T01:36:06.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:36:06 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 4261.675482 secs 2024-03-24T01:36:06.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:36:06 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 4261.675482 secs 2024-03-24T01:36:07.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:36:07 smithi118 ceph-mon[31535]: pgmap v2933: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:36:07.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:36:07 smithi189 ceph-mon[36187]: pgmap v2933: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:36:07.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:36:07 smithi193 ceph-mon[36878]: pgmap v2933: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:36:09.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:36:09 smithi118 ceph-mon[31535]: pgmap v2934: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T01:36:09.704 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:36:09 smithi189 ceph-mon[36187]: pgmap v2934: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T01:36:09.711 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:36:09 smithi193 ceph-mon[36878]: pgmap v2934: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T01:36:11.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:36:11 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 4266.675541 secs 2024-03-24T01:36:11.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:36:11 smithi118 ceph-mon[31535]: pgmap v2935: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T01:36:11.704 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:36:11 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 4266.675541 secs 2024-03-24T01:36:11.704 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:36:11 smithi189 ceph-mon[36187]: pgmap v2935: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T01:36:11.711 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:36:11 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 4266.675541 secs 2024-03-24T01:36:11.711 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:36:11 smithi193 ceph-mon[36878]: pgmap v2935: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T01:36:13.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:36:13 smithi118 ceph-mon[31535]: pgmap v2936: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:36:13.704 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:36:13 smithi189 ceph-mon[36187]: pgmap v2936: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:36:13.711 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:36:13 smithi193 ceph-mon[36878]: pgmap v2936: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:36:15.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:36:15 smithi118 ceph-mon[31535]: pgmap v2937: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-24T01:36:15.704 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:36:15 smithi189 ceph-mon[36187]: pgmap v2937: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-24T01:36:15.711 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:36:15 smithi193 ceph-mon[36878]: pgmap v2937: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-24T01:36:16.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:36:16 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 4271.675631 secs 2024-03-24T01:36:16.704 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:36:16 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 4271.675631 secs 2024-03-24T01:36:16.711 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:36:16 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 4271.675631 secs 2024-03-24T01:36:17.697 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:36:17 smithi118 ceph-mon[31535]: pgmap v2938: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:36:17.704 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:36:17 smithi189 ceph-mon[36187]: pgmap v2938: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:36:17.711 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:36:17 smithi193 ceph-mon[36878]: pgmap v2938: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:36:19.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:36:19 smithi118 ceph-mon[31535]: pgmap v2939: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T01:36:19.704 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:36:19 smithi189 ceph-mon[36187]: pgmap v2939: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T01:36:19.711 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:36:19 smithi193 ceph-mon[36878]: pgmap v2939: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T01:36:21.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:36:21 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 4276.675721 secs 2024-03-24T01:36:21.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:36:21 smithi118 ceph-mon[31535]: pgmap v2940: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T01:36:21.704 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:36:21 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 4276.675721 secs 2024-03-24T01:36:21.704 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:36:21 smithi189 ceph-mon[36187]: pgmap v2940: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T01:36:21.711 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:36:21 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 4276.675721 secs 2024-03-24T01:36:21.711 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:36:21 smithi193 ceph-mon[36878]: pgmap v2940: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T01:36:23.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:36:23 smithi118 ceph-mon[31535]: pgmap v2941: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:36:23.704 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:36:23 smithi189 ceph-mon[36187]: pgmap v2941: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:36:23.711 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:36:23 smithi193 ceph-mon[36878]: pgmap v2941: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:36:25.704 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:36:25 smithi189 ceph-mon[36187]: pgmap v2942: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-24T01:36:25.711 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:36:25 smithi193 ceph-mon[36878]: pgmap v2942: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-24T01:36:25.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:36:25 smithi118 ceph-mon[31535]: pgmap v2942: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-24T01:36:26.704 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:36:26 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 4281.675781 secs 2024-03-24T01:36:26.711 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:36:26 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 4281.675781 secs 2024-03-24T01:36:26.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:36:26 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 4281.675781 secs 2024-03-24T01:36:27.711 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:36:27 smithi193 ceph-mon[36878]: pgmap v2943: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:36:27.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:36:27 smithi118 ceph-mon[31535]: pgmap v2943: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:36:27.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:36:27 smithi189 ceph-mon[36187]: pgmap v2943: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:36:29.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:36:29 smithi118 ceph-mon[31535]: pgmap v2944: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T01:36:29.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:36:29 smithi189 ceph-mon[36187]: pgmap v2944: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T01:36:29.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:36:29 smithi193 ceph-mon[36878]: pgmap v2944: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T01:36:31.756 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:36:31 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 4286.675870 secs 2024-03-24T01:36:31.756 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:36:31 smithi189 ceph-mon[36187]: pgmap v2945: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T01:36:31.762 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:36:31 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 4286.675870 secs 2024-03-24T01:36:31.762 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:36:31 smithi193 ceph-mon[36878]: pgmap v2945: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T01:36:31.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:36:31 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 4286.675870 secs 2024-03-24T01:36:31.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:36:31 smithi118 ceph-mon[31535]: pgmap v2945: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T01:36:33.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:36:33 smithi118 ceph-mon[31535]: pgmap v2946: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:36:33.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:36:33 smithi189 ceph-mon[36187]: pgmap v2946: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:36:33.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:36:33 smithi193 ceph-mon[36878]: pgmap v2946: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:36:35.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:36:35 smithi118 ceph-mon[31535]: pgmap v2947: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-24T01:36:35.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:36:35 smithi189 ceph-mon[36187]: pgmap v2947: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-24T01:36:35.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:36:35 smithi193 ceph-mon[36878]: pgmap v2947: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-24T01:36:36.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:36:36 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 4291.675960 secs 2024-03-24T01:36:36.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:36:36 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 4291.675960 secs 2024-03-24T01:36:36.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:36:36 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 4291.675960 secs 2024-03-24T01:36:37.947 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:36:37 smithi118 ceph-mon[31535]: pgmap v2948: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:36:37.947 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:36:37 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T01:36:37.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:36:37 smithi189 ceph-mon[36187]: pgmap v2948: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:36:37.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:36:37 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T01:36:37.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:36:37 smithi193 ceph-mon[36878]: pgmap v2948: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:36:37.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:36:37 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T01:36:39.197 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:36:38 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T01:36:39.197 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:36:38 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T01:36:39.197 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:36:38 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T01:36:39.197 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:36:38 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T01:36:39.197 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:36:38 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T01:36:39.197 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:36:38 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T01:36:39.197 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:36:38 smithi118 ceph-mon[31535]: pgmap v2949: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 6 op/s 2024-03-24T01:36:39.197 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:36:38 smithi118 ceph-mon[31535]: pgmap v2950: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 5 op/s 2024-03-24T01:36:39.197 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:36:38 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T01:36:39.205 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:36:38 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T01:36:39.205 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:36:38 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T01:36:39.205 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:36:38 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T01:36:39.206 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:36:38 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T01:36:39.206 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:36:38 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T01:36:39.206 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:36:38 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T01:36:39.206 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:36:38 smithi189 ceph-mon[36187]: pgmap v2949: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 6 op/s 2024-03-24T01:36:39.206 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:36:38 smithi189 ceph-mon[36187]: pgmap v2950: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 5 op/s 2024-03-24T01:36:39.207 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:36:38 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T01:36:39.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:36:38 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T01:36:39.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:36:38 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T01:36:39.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:36:38 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T01:36:39.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:36:38 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T01:36:39.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:36:38 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T01:36:39.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:36:38 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T01:36:39.212 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:36:38 smithi193 ceph-mon[36878]: pgmap v2949: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 6 op/s 2024-03-24T01:36:39.212 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:36:38 smithi193 ceph-mon[36878]: pgmap v2950: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 5 op/s 2024-03-24T01:36:39.212 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:36:38 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T01:36:41.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:36:41 smithi118 ceph-mon[31535]: pgmap v2951: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 6 op/s 2024-03-24T01:36:41.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:36:41 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 4296.676018 secs 2024-03-24T01:36:41.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:36:41 smithi189 ceph-mon[36187]: pgmap v2951: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 6 op/s 2024-03-24T01:36:41.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:36:41 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 4296.676018 secs 2024-03-24T01:36:41.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:36:41 smithi193 ceph-mon[36878]: pgmap v2951: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 6 op/s 2024-03-24T01:36:41.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:36:41 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 4296.676018 secs 2024-03-24T01:36:43.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:36:43 smithi118 ceph-mon[31535]: pgmap v2952: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 6 op/s 2024-03-24T01:36:43.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:36:43 smithi189 ceph-mon[36187]: pgmap v2952: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 6 op/s 2024-03-24T01:36:43.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:36:43 smithi193 ceph-mon[36878]: pgmap v2952: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 6 op/s 2024-03-24T01:36:45.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:36:45 smithi118 ceph-mon[31535]: pgmap v2953: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 5 op/s 2024-03-24T01:36:45.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:36:45 smithi189 ceph-mon[36187]: pgmap v2953: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 5 op/s 2024-03-24T01:36:45.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:36:45 smithi193 ceph-mon[36878]: pgmap v2953: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 5 op/s 2024-03-24T01:36:46.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:36:46 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 4301.676065 secs 2024-03-24T01:36:46.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:36:46 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 4301.676065 secs 2024-03-24T01:36:46.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:36:46 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 4301.676065 secs 2024-03-24T01:36:47.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:36:47 smithi118 ceph-mon[31535]: pgmap v2954: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 6 op/s 2024-03-24T01:36:47.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:36:47 smithi189 ceph-mon[36187]: pgmap v2954: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 6 op/s 2024-03-24T01:36:47.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:36:47 smithi193 ceph-mon[36878]: pgmap v2954: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 6 op/s 2024-03-24T01:36:49.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:36:49 smithi118 ceph-mon[31535]: pgmap v2955: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 5 op/s 2024-03-24T01:36:49.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:36:49 smithi189 ceph-mon[36187]: pgmap v2955: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 5 op/s 2024-03-24T01:36:49.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:36:49 smithi193 ceph-mon[36878]: pgmap v2955: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 5 op/s 2024-03-24T01:36:51.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:36:51 smithi118 ceph-mon[31535]: pgmap v2956: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T01:36:51.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:36:51 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 4306.676124 secs 2024-03-24T01:36:51.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:36:51 smithi189 ceph-mon[36187]: pgmap v2956: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T01:36:51.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:36:51 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 4306.676124 secs 2024-03-24T01:36:51.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:36:51 smithi193 ceph-mon[36878]: pgmap v2956: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T01:36:51.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:36:51 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 4306.676124 secs 2024-03-24T01:36:53.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:36:53 smithi118 ceph-mon[31535]: pgmap v2957: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:36:53.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:36:53 smithi189 ceph-mon[36187]: pgmap v2957: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:36:53.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:36:53 smithi193 ceph-mon[36878]: pgmap v2957: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:36:55.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:36:55 smithi118 ceph-mon[31535]: pgmap v2958: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T01:36:55.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:36:55 smithi189 ceph-mon[36187]: pgmap v2958: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T01:36:55.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:36:55 smithi193 ceph-mon[36878]: pgmap v2958: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T01:36:56.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:36:56 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 4311.676200 secs 2024-03-24T01:36:56.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:36:56 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 4311.676200 secs 2024-03-24T01:36:56.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:36:56 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 4311.676200 secs 2024-03-24T01:36:57.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:36:57 smithi118 ceph-mon[31535]: pgmap v2959: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T01:36:57.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:36:57 smithi189 ceph-mon[36187]: pgmap v2959: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T01:36:57.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:36:57 smithi193 ceph-mon[36878]: pgmap v2959: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T01:36:59.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:36:59 smithi118 ceph-mon[31535]: pgmap v2960: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 5 op/s 2024-03-24T01:36:59.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:36:59 smithi189 ceph-mon[36187]: pgmap v2960: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 5 op/s 2024-03-24T01:36:59.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:36:59 smithi193 ceph-mon[36878]: pgmap v2960: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 5 op/s 2024-03-24T01:37:01.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:37:01 smithi189 ceph-mon[36187]: pgmap v2961: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T01:37:01.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:37:01 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 4316.676260 secs 2024-03-24T01:37:01.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:37:01 smithi193 ceph-mon[36878]: pgmap v2961: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T01:37:01.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:37:01 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 4316.676260 secs 2024-03-24T01:37:01.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:37:01 smithi118 ceph-mon[31535]: pgmap v2961: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T01:37:01.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:37:01 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 4316.676260 secs 2024-03-24T01:37:03.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:37:03 smithi189 ceph-mon[36187]: pgmap v2962: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:37:03.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:37:03 smithi193 ceph-mon[36878]: pgmap v2962: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:37:03.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:37:03 smithi118 ceph-mon[31535]: pgmap v2962: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:37:05.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:37:05 smithi193 ceph-mon[36878]: pgmap v2963: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T01:37:05.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:37:05 smithi118 ceph-mon[31535]: pgmap v2963: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T01:37:05.704 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:37:05 smithi189 ceph-mon[36187]: pgmap v2963: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T01:37:06.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:37:06 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 4321.676348 secs 2024-03-24T01:37:06.704 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:37:06 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 4321.676348 secs 2024-03-24T01:37:06.711 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:37:06 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 4321.676348 secs 2024-03-24T01:37:07.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:37:07 smithi118 ceph-mon[31535]: pgmap v2964: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T01:37:07.704 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:37:07 smithi189 ceph-mon[36187]: pgmap v2964: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T01:37:07.711 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:37:07 smithi193 ceph-mon[36878]: pgmap v2964: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T01:37:09.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:37:09 smithi118 ceph-mon[31535]: pgmap v2965: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 5 op/s 2024-03-24T01:37:09.704 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:37:09 smithi189 ceph-mon[36187]: pgmap v2965: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 5 op/s 2024-03-24T01:37:09.711 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:37:09 smithi193 ceph-mon[36878]: pgmap v2965: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 5 op/s 2024-03-24T01:37:11.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:37:11 smithi118 ceph-mon[31535]: pgmap v2966: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T01:37:11.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:37:11 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 4326.676398 secs 2024-03-24T01:37:11.704 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:37:11 smithi189 ceph-mon[36187]: pgmap v2966: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T01:37:11.704 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:37:11 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 4326.676398 secs 2024-03-24T01:37:11.711 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:37:11 smithi193 ceph-mon[36878]: pgmap v2966: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T01:37:11.711 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:37:11 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 4326.676398 secs 2024-03-24T01:37:13.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:37:13 smithi118 ceph-mon[31535]: pgmap v2967: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:37:13.704 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:37:13 smithi189 ceph-mon[36187]: pgmap v2967: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:37:13.711 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:37:13 smithi193 ceph-mon[36878]: pgmap v2967: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:37:15.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:37:15 smithi118 ceph-mon[31535]: pgmap v2968: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T01:37:15.704 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:37:15 smithi189 ceph-mon[36187]: pgmap v2968: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T01:37:15.711 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:37:15 smithi193 ceph-mon[36878]: pgmap v2968: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T01:37:16.426 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T01:37:16.424+0000 7f81db400640 1 -- 172.21.15.118:0/659686424 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f81c40064d0 con 0x5622651a13f0 2024-03-24T01:37:16.426 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T01:37:16.424+0000 7f81ee000640 1 -- 172.21.15.118:0/1693767697 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f81c80036c0 con 0x7f81f81052d0 2024-03-24T01:37:16.426 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T01:37:16.425+0000 7f81ef400640 1 -- 172.21.15.118:0/1693767697 <== mon.2 v2:172.21.15.189:3300/0 37 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (secure 0 0 0) 0x7f81c80036c0 con 0x7f81f81052d0 2024-03-24T01:37:16.426 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T01:37:16.425+0000 7f81e4c00640 1 -- 172.21.15.118:0/659686424 <== mon.1 v2:172.21.15.193:3300/0 37 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 409+0+0 (secure 0 0 0) 0x7f81c40064d0 con 0x5622651a13f0 2024-03-24T01:37:16.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:37:16 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 4331.676483 secs 2024-03-24T01:37:16.704 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:37:16 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 4331.676483 secs 2024-03-24T01:37:16.711 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:37:16 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 4331.676483 secs 2024-03-24T01:37:17.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:37:17 smithi118 ceph-mon[31535]: pgmap v2969: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T01:37:17.704 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:37:17 smithi189 ceph-mon[36187]: pgmap v2969: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T01:37:17.711 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:37:17 smithi193 ceph-mon[36878]: pgmap v2969: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T01:37:19.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:37:19 smithi118 ceph-mon[31535]: pgmap v2970: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 5 op/s 2024-03-24T01:37:19.704 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:37:19 smithi189 ceph-mon[36187]: pgmap v2970: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 5 op/s 2024-03-24T01:37:19.711 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:37:19 smithi193 ceph-mon[36878]: pgmap v2970: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 5 op/s 2024-03-24T01:37:21.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:37:21 smithi118 ceph-mon[31535]: pgmap v2971: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T01:37:21.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:37:21 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 4336.676567 secs 2024-03-24T01:37:21.704 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:37:21 smithi189 ceph-mon[36187]: pgmap v2971: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T01:37:21.704 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:37:21 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 4336.676567 secs 2024-03-24T01:37:21.711 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:37:21 smithi193 ceph-mon[36878]: pgmap v2971: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T01:37:21.711 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:37:21 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 4336.676567 secs 2024-03-24T01:37:23.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:37:23 smithi118 ceph-mon[31535]: pgmap v2972: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:37:23.704 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:37:23 smithi189 ceph-mon[36187]: pgmap v2972: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:37:23.711 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:37:23 smithi193 ceph-mon[36878]: pgmap v2972: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:37:25.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:37:25 smithi118 ceph-mon[31535]: pgmap v2973: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T01:37:25.704 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:37:25 smithi189 ceph-mon[36187]: pgmap v2973: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T01:37:25.711 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:37:25 smithi193 ceph-mon[36878]: pgmap v2973: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T01:37:26.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:37:26 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 4341.676627 secs 2024-03-24T01:37:26.704 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:37:26 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 4341.676627 secs 2024-03-24T01:37:26.711 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:37:26 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 4341.676627 secs 2024-03-24T01:37:27.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:37:27 smithi118 ceph-mon[31535]: pgmap v2974: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T01:37:27.704 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:37:27 smithi189 ceph-mon[36187]: pgmap v2974: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T01:37:27.711 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:37:27 smithi193 ceph-mon[36878]: pgmap v2974: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T01:37:29.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:37:29 smithi118 ceph-mon[31535]: pgmap v2975: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 5 op/s 2024-03-24T01:37:29.704 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:37:29 smithi189 ceph-mon[36187]: pgmap v2975: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 5 op/s 2024-03-24T01:37:29.711 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:37:29 smithi193 ceph-mon[36878]: pgmap v2975: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 5 op/s 2024-03-24T01:37:31.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:37:31 smithi118 ceph-mon[31535]: pgmap v2976: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T01:37:31.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:37:31 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 4346.676713 secs 2024-03-24T01:37:31.704 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:37:31 smithi189 ceph-mon[36187]: pgmap v2976: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T01:37:31.704 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:37:31 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 4346.676713 secs 2024-03-24T01:37:31.711 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:37:31 smithi193 ceph-mon[36878]: pgmap v2976: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T01:37:31.711 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:37:31 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 4346.676713 secs 2024-03-24T01:37:33.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:37:33 smithi118 ceph-mon[31535]: pgmap v2977: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:37:33.704 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:37:33 smithi189 ceph-mon[36187]: pgmap v2977: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:37:33.711 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:37:33 smithi193 ceph-mon[36878]: pgmap v2977: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:37:35.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:37:35 smithi118 ceph-mon[31535]: pgmap v2978: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T01:37:35.704 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:37:35 smithi189 ceph-mon[36187]: pgmap v2978: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T01:37:35.711 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:37:35 smithi193 ceph-mon[36878]: pgmap v2978: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T01:37:36.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:37:36 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 4351.676773 secs 2024-03-24T01:37:36.704 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:37:36 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 4351.676773 secs 2024-03-24T01:37:36.711 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:37:36 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 4351.676773 secs 2024-03-24T01:37:37.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:37:37 smithi118 ceph-mon[31535]: pgmap v2979: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T01:37:37.704 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:37:37 smithi189 ceph-mon[36187]: pgmap v2979: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T01:37:37.711 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:37:37 smithi193 ceph-mon[36878]: pgmap v2979: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T01:37:38.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:37:38 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T01:37:38.704 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:37:38 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T01:37:38.711 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:37:38 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T01:37:39.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:37:39 smithi118 ceph-mon[31535]: pgmap v2980: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 5 op/s 2024-03-24T01:37:39.704 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:37:39 smithi189 ceph-mon[36187]: pgmap v2980: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 5 op/s 2024-03-24T01:37:39.711 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:37:39 smithi193 ceph-mon[36878]: pgmap v2980: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 5 op/s 2024-03-24T01:37:40.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:37:40 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T01:37:40.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:37:40 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T01:37:40.697 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:37:40 smithi118 ceph-mon[31535]: pgmap v2981: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 6 op/s 2024-03-24T01:37:40.697 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:37:40 smithi118 ceph-mon[31535]: pgmap v2982: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-03-24T01:37:40.697 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:37:40 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T01:37:40.704 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:37:40 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T01:37:40.704 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:37:40 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T01:37:40.704 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:37:40 smithi189 ceph-mon[36187]: pgmap v2981: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 6 op/s 2024-03-24T01:37:40.704 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:37:40 smithi189 ceph-mon[36187]: pgmap v2982: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-03-24T01:37:40.704 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:37:40 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T01:37:40.711 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:37:40 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T01:37:40.711 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:37:40 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T01:37:40.711 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:37:40 smithi193 ceph-mon[36878]: pgmap v2981: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 6 op/s 2024-03-24T01:37:40.711 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:37:40 smithi193 ceph-mon[36878]: pgmap v2982: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-03-24T01:37:40.711 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:37:40 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T01:37:41.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:37:41 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 4356.676827 secs 2024-03-24T01:37:41.704 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:37:41 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 4356.676827 secs 2024-03-24T01:37:41.711 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:37:41 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 4356.676827 secs 2024-03-24T01:37:42.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:37:42 smithi118 ceph-mon[31535]: pgmap v2983: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-03-24T01:37:42.704 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:37:42 smithi189 ceph-mon[36187]: pgmap v2983: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-03-24T01:37:42.711 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:37:42 smithi193 ceph-mon[36878]: pgmap v2983: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-03-24T01:37:44.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:37:44 smithi118 ceph-mon[31535]: pgmap v2984: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 5 op/s 2024-03-24T01:37:44.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:37:44 smithi189 ceph-mon[36187]: pgmap v2984: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 5 op/s 2024-03-24T01:37:44.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:37:44 smithi193 ceph-mon[36878]: pgmap v2984: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 5 op/s 2024-03-24T01:37:46.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:37:46 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 4361.676914 secs 2024-03-24T01:37:46.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:37:46 smithi118 ceph-mon[31535]: pgmap v2985: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-03-24T01:37:46.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:37:46 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 4361.676914 secs 2024-03-24T01:37:46.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:37:46 smithi189 ceph-mon[36187]: pgmap v2985: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-03-24T01:37:46.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:37:46 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 4361.676914 secs 2024-03-24T01:37:46.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:37:46 smithi193 ceph-mon[36878]: pgmap v2985: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-03-24T01:37:48.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:37:48 smithi118 ceph-mon[31535]: pgmap v2986: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 5 op/s 2024-03-24T01:37:48.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:37:48 smithi189 ceph-mon[36187]: pgmap v2986: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 5 op/s 2024-03-24T01:37:48.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:37:48 smithi193 ceph-mon[36878]: pgmap v2986: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 5 op/s 2024-03-24T01:37:50.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:37:50 smithi118 ceph-mon[31535]: pgmap v2987: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 5 op/s 2024-03-24T01:37:50.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:37:50 smithi189 ceph-mon[36187]: pgmap v2987: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 5 op/s 2024-03-24T01:37:50.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:37:50 smithi193 ceph-mon[36878]: pgmap v2987: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 5 op/s 2024-03-24T01:37:51.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:37:51 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 4366.677001 secs 2024-03-24T01:37:51.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:37:51 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 4366.677001 secs 2024-03-24T01:37:51.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:37:51 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 4366.677001 secs 2024-03-24T01:37:52.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:37:52 smithi118 ceph-mon[31535]: pgmap v2988: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:37:52.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:37:52 smithi189 ceph-mon[36187]: pgmap v2988: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:37:52.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:37:52 smithi193 ceph-mon[36878]: pgmap v2988: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:37:54.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:37:54 smithi118 ceph-mon[31535]: pgmap v2989: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T01:37:54.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:37:54 smithi189 ceph-mon[36187]: pgmap v2989: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T01:37:54.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:37:54 smithi193 ceph-mon[36878]: pgmap v2989: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T01:37:55.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:37:55 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 4371.677082 secs 2024-03-24T01:37:55.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:37:55 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 4371.677082 secs 2024-03-24T01:37:55.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:37:55 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 4371.677082 secs 2024-03-24T01:37:56.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:37:56 smithi118 ceph-mon[31535]: pgmap v2990: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T01:37:56.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:37:56 smithi189 ceph-mon[36187]: pgmap v2990: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T01:37:56.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:37:56 smithi193 ceph-mon[36878]: pgmap v2990: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T01:37:58.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:37:58 smithi118 ceph-mon[31535]: pgmap v2991: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T01:37:58.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:37:58 smithi189 ceph-mon[36187]: pgmap v2991: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T01:37:58.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:37:58 smithi193 ceph-mon[36878]: pgmap v2991: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T01:38:00.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:38:00 smithi118 ceph-mon[31535]: pgmap v2992: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T01:38:00.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:38:00 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 4376.677139 secs 2024-03-24T01:38:00.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:38:00 smithi189 ceph-mon[36187]: pgmap v2992: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T01:38:00.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:38:00 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 4376.677139 secs 2024-03-24T01:38:00.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:38:00 smithi193 ceph-mon[36878]: pgmap v2992: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T01:38:00.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:38:00 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 4376.677139 secs 2024-03-24T01:38:02.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:38:02 smithi118 ceph-mon[31535]: pgmap v2993: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:38:02.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:38:02 smithi189 ceph-mon[36187]: pgmap v2993: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:38:02.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:38:02 smithi193 ceph-mon[36878]: pgmap v2993: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:38:04.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:38:04 smithi118 ceph-mon[31535]: pgmap v2994: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T01:38:04.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:38:04 smithi189 ceph-mon[36187]: pgmap v2994: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T01:38:04.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:38:04 smithi193 ceph-mon[36878]: pgmap v2994: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T01:38:05.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:38:05 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 4381.677200 secs 2024-03-24T01:38:05.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:38:05 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 4381.677200 secs 2024-03-24T01:38:05.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:38:05 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 4381.677200 secs 2024-03-24T01:38:06.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:38:06 smithi118 ceph-mon[31535]: pgmap v2995: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T01:38:06.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:38:06 smithi189 ceph-mon[36187]: pgmap v2995: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T01:38:06.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:38:06 smithi193 ceph-mon[36878]: pgmap v2995: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T01:38:08.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:38:08 smithi118 ceph-mon[31535]: pgmap v2996: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T01:38:08.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:38:08 smithi189 ceph-mon[36187]: pgmap v2996: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T01:38:08.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:38:08 smithi193 ceph-mon[36878]: pgmap v2996: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T01:38:10.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:38:10 smithi118 ceph-mon[31535]: pgmap v2997: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-03-24T01:38:10.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:38:10 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 4386.677262 secs 2024-03-24T01:38:10.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:38:10 smithi189 ceph-mon[36187]: pgmap v2997: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-03-24T01:38:10.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:38:10 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 4386.677262 secs 2024-03-24T01:38:10.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:38:10 smithi193 ceph-mon[36878]: pgmap v2997: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-03-24T01:38:10.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:38:10 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 4386.677262 secs 2024-03-24T01:38:12.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:38:12 smithi118 ceph-mon[31535]: pgmap v2998: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:38:12.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:38:12 smithi189 ceph-mon[36187]: pgmap v2998: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:38:12.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:38:12 smithi193 ceph-mon[36878]: pgmap v2998: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:38:14.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:38:14 smithi118 ceph-mon[31535]: pgmap v2999: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T01:38:14.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:38:14 smithi189 ceph-mon[36187]: pgmap v2999: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T01:38:14.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:38:14 smithi193 ceph-mon[36878]: pgmap v2999: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T01:38:15.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:38:15 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 4391.677322 secs 2024-03-24T01:38:15.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:38:15 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 4391.677322 secs 2024-03-24T01:38:15.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:38:15 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 4391.677322 secs 2024-03-24T01:38:16.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:38:16 smithi118 ceph-mon[31535]: pgmap v3000: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T01:38:16.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:38:16 smithi189 ceph-mon[36187]: pgmap v3000: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T01:38:16.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:38:16 smithi193 ceph-mon[36878]: pgmap v3000: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T01:38:18.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:38:18 smithi118 ceph-mon[31535]: pgmap v3001: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T01:38:18.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:38:18 smithi189 ceph-mon[36187]: pgmap v3001: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T01:38:18.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:38:18 smithi193 ceph-mon[36878]: pgmap v3001: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T01:38:20.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:38:20 smithi118 ceph-mon[31535]: pgmap v3002: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-03-24T01:38:20.947 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:38:20 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 4396.677383 secs 2024-03-24T01:38:20.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:38:20 smithi189 ceph-mon[36187]: pgmap v3002: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-03-24T01:38:20.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:38:20 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 4396.677383 secs 2024-03-24T01:38:20.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:38:20 smithi193 ceph-mon[36878]: pgmap v3002: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-03-24T01:38:20.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:38:20 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 4396.677383 secs 2024-03-24T01:38:22.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:38:22 smithi118 ceph-mon[31535]: pgmap v3003: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:38:22.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:38:22 smithi189 ceph-mon[36187]: pgmap v3003: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:38:22.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:38:22 smithi193 ceph-mon[36878]: pgmap v3003: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:38:24.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:38:24 smithi118 ceph-mon[31535]: pgmap v3004: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T01:38:24.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:38:24 smithi189 ceph-mon[36187]: pgmap v3004: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T01:38:24.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:38:24 smithi193 ceph-mon[36878]: pgmap v3004: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T01:38:25.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:38:25 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 4401.677445 secs 2024-03-24T01:38:25.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:38:25 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 4401.677445 secs 2024-03-24T01:38:26.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:38:25 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 4401.677445 secs 2024-03-24T01:38:26.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:38:26 smithi189 ceph-mon[36187]: pgmap v3005: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 6 op/s 2024-03-24T01:38:26.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:38:26 smithi193 ceph-mon[36878]: pgmap v3005: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 6 op/s 2024-03-24T01:38:27.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:38:26 smithi118 ceph-mon[31535]: pgmap v3005: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 6 op/s 2024-03-24T01:38:28.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:38:28 smithi193 ceph-mon[36878]: pgmap v3006: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 5 op/s 2024-03-24T01:38:29.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:38:28 smithi118 ceph-mon[31535]: pgmap v3006: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 5 op/s 2024-03-24T01:38:29.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:38:28 smithi189 ceph-mon[36187]: pgmap v3006: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 5 op/s 2024-03-24T01:38:31.066 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:38:30 smithi118 ceph-mon[31535]: pgmap v3007: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-03-24T01:38:31.066 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:38:30 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 4406.677533 secs 2024-03-24T01:38:31.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:38:30 smithi189 ceph-mon[36187]: pgmap v3007: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-03-24T01:38:31.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:38:30 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 4406.677533 secs 2024-03-24T01:38:31.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:38:30 smithi193 ceph-mon[36878]: pgmap v3007: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-03-24T01:38:31.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:38:30 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 4406.677533 secs 2024-03-24T01:38:33.006 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:38:32 smithi189 ceph-mon[36187]: pgmap v3008: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-03-24T01:38:33.013 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:38:32 smithi193 ceph-mon[36878]: pgmap v3008: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-03-24T01:38:33.112 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:38:32 smithi118 ceph-mon[31535]: pgmap v3008: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-03-24T01:38:35.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:38:34 smithi118 ceph-mon[31535]: pgmap v3009: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-03-24T01:38:35.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:38:34 smithi189 ceph-mon[36187]: pgmap v3009: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-03-24T01:38:35.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:38:34 smithi193 ceph-mon[36878]: pgmap v3009: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-03-24T01:38:36.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:38:35 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 4411.677622 secs 2024-03-24T01:38:36.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:38:35 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 4411.677622 secs 2024-03-24T01:38:36.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:38:35 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 4411.677622 secs 2024-03-24T01:38:37.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:38:36 smithi118 ceph-mon[31535]: pgmap v3010: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 6 op/s 2024-03-24T01:38:37.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:38:36 smithi189 ceph-mon[36187]: pgmap v3010: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 6 op/s 2024-03-24T01:38:37.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:38:36 smithi193 ceph-mon[36878]: pgmap v3010: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 6 op/s 2024-03-24T01:38:39.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:38:38 smithi118 ceph-mon[31535]: pgmap v3011: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T01:38:39.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:38:38 smithi189 ceph-mon[36187]: pgmap v3011: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T01:38:39.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:38:38 smithi193 ceph-mon[36878]: pgmap v3011: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T01:38:40.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:38:40 smithi193 ceph-mon[36878]: pgmap v3012: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-03-24T01:38:40.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:38:40 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T01:38:40.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:38:40 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 4416.677690 secs 2024-03-24T01:38:40.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:38:40 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T01:38:40.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:38:40 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T01:38:40.962 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:38:40 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T01:38:40.962 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:38:40 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T01:38:41.163 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:38:40 smithi189 ceph-mon[36187]: pgmap v3012: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-03-24T01:38:41.163 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:38:40 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T01:38:41.163 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:38:40 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 4416.677690 secs 2024-03-24T01:38:41.163 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:38:40 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T01:38:41.163 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:38:40 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T01:38:41.163 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:38:40 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T01:38:41.163 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:38:40 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T01:38:41.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:38:40 smithi118 ceph-mon[31535]: pgmap v3012: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-03-24T01:38:41.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:38:40 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T01:38:41.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:38:40 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 4416.677690 secs 2024-03-24T01:38:41.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:38:40 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T01:38:41.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:38:40 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T01:38:41.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:38:40 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T01:38:41.197 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:38:40 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T01:38:43.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:38:42 smithi118 ceph-mon[31535]: pgmap v3013: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:38:43.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:38:42 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T01:38:43.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:38:42 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T01:38:43.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:38:42 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T01:38:43.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:38:42 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T01:38:43.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:38:42 smithi189 ceph-mon[36187]: pgmap v3013: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:38:43.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:38:42 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T01:38:43.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:38:42 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T01:38:43.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:38:42 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T01:38:43.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:38:42 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T01:38:43.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:38:42 smithi193 ceph-mon[36878]: pgmap v3013: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:38:43.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:38:42 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T01:38:43.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:38:42 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T01:38:43.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:38:42 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T01:38:43.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:38:42 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T01:38:45.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:38:44 smithi118 ceph-mon[31535]: pgmap v3014: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T01:38:45.197 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:38:44 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T01:38:45.197 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:38:44 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T01:38:45.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:38:44 smithi189 ceph-mon[36187]: pgmap v3014: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T01:38:45.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:38:44 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T01:38:45.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:38:44 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T01:38:45.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:38:44 smithi193 ceph-mon[36878]: pgmap v3014: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T01:38:45.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:38:44 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T01:38:45.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:38:44 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T01:38:46.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:38:45 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 4421.677775 secs 2024-03-24T01:38:46.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:38:45 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 4421.677775 secs 2024-03-24T01:38:46.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:38:45 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 4421.677775 secs 2024-03-24T01:38:47.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:38:46 smithi118 ceph-mon[31535]: pgmap v3015: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T01:38:47.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:38:46 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T01:38:47.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:38:46 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T01:38:47.197 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:38:46 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T01:38:47.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:38:46 smithi189 ceph-mon[36187]: pgmap v3015: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T01:38:47.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:38:46 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T01:38:47.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:38:46 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T01:38:47.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:38:46 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T01:38:47.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:38:46 smithi193 ceph-mon[36878]: pgmap v3015: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T01:38:47.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:38:46 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T01:38:47.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:38:46 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T01:38:47.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:38:46 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T01:38:48.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:38:47 smithi118 ceph-mon[31535]: pgmap v3016: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T01:38:48.197 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:38:47 smithi118 ceph-mon[31535]: pgmap v3017: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 6 op/s 2024-03-24T01:38:48.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:38:47 smithi189 ceph-mon[36187]: pgmap v3016: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T01:38:48.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:38:47 smithi189 ceph-mon[36187]: pgmap v3017: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 6 op/s 2024-03-24T01:38:48.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:38:47 smithi193 ceph-mon[36878]: pgmap v3016: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T01:38:48.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:38:47 smithi193 ceph-mon[36878]: pgmap v3017: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 6 op/s 2024-03-24T01:38:50.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:38:49 smithi118 ceph-mon[31535]: pgmap v3018: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 4 op/s 2024-03-24T01:38:50.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:38:49 smithi189 ceph-mon[36187]: pgmap v3018: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 4 op/s 2024-03-24T01:38:50.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:38:49 smithi193 ceph-mon[36878]: pgmap v3018: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 4 op/s 2024-03-24T01:38:51.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:38:50 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 4426.677817 secs 2024-03-24T01:38:51.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:38:50 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 4426.677817 secs 2024-03-24T01:38:51.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:38:50 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 4426.677817 secs 2024-03-24T01:38:52.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:38:51 smithi118 ceph-mon[31535]: pgmap v3019: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-03-24T01:38:52.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:38:51 smithi189 ceph-mon[36187]: pgmap v3019: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-03-24T01:38:52.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:38:51 smithi193 ceph-mon[36878]: pgmap v3019: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-03-24T01:38:54.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:38:53 smithi118 ceph-mon[31535]: pgmap v3020: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 5 op/s 2024-03-24T01:38:54.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:38:53 smithi189 ceph-mon[36187]: pgmap v3020: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 5 op/s 2024-03-24T01:38:54.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:38:53 smithi193 ceph-mon[36878]: pgmap v3020: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 5 op/s 2024-03-24T01:38:56.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:38:55 smithi118 ceph-mon[31535]: pgmap v3021: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 6 op/s 2024-03-24T01:38:56.197 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:38:55 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 4431.677905 secs 2024-03-24T01:38:56.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:38:55 smithi189 ceph-mon[36187]: pgmap v3021: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 6 op/s 2024-03-24T01:38:56.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:38:55 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 4431.677905 secs 2024-03-24T01:38:56.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:38:55 smithi193 ceph-mon[36878]: pgmap v3021: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 6 op/s 2024-03-24T01:38:56.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:38:55 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 4431.677905 secs 2024-03-24T01:38:58.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:38:57 smithi118 ceph-mon[31535]: pgmap v3022: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 5 op/s 2024-03-24T01:38:58.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:38:57 smithi189 ceph-mon[36187]: pgmap v3022: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 5 op/s 2024-03-24T01:38:58.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:38:57 smithi193 ceph-mon[36878]: pgmap v3022: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 5 op/s 2024-03-24T01:39:00.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:38:59 smithi118 ceph-mon[31535]: pgmap v3023: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 5 op/s 2024-03-24T01:39:00.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:38:59 smithi189 ceph-mon[36187]: pgmap v3023: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 5 op/s 2024-03-24T01:39:00.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:38:59 smithi193 ceph-mon[36878]: pgmap v3023: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 5 op/s 2024-03-24T01:39:01.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:39:00 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 4436.677968 secs 2024-03-24T01:39:01.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:39:00 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 4436.677968 secs 2024-03-24T01:39:01.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:39:00 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 4436.677968 secs 2024-03-24T01:39:02.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:39:01 smithi118 ceph-mon[31535]: pgmap v3024: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T01:39:02.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:39:01 smithi189 ceph-mon[36187]: pgmap v3024: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T01:39:02.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:39:01 smithi193 ceph-mon[36878]: pgmap v3024: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T01:39:04.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:39:03 smithi118 ceph-mon[31535]: pgmap v3025: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:39:04.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:39:03 smithi189 ceph-mon[36187]: pgmap v3025: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:39:04.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:39:03 smithi193 ceph-mon[36878]: pgmap v3025: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:39:05.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:39:04 smithi118 ceph-mon[31535]: pgmap v3026: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T01:39:05.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:39:04 smithi189 ceph-mon[36187]: pgmap v3026: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T01:39:05.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:39:04 smithi193 ceph-mon[36878]: pgmap v3026: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T01:39:06.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:39:05 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 4441.678064 secs 2024-03-24T01:39:06.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:39:05 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 4441.678064 secs 2024-03-24T01:39:06.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:39:05 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 4441.678064 secs 2024-03-24T01:39:07.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:39:06 smithi118 ceph-mon[31535]: pgmap v3027: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T01:39:07.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:39:06 smithi189 ceph-mon[36187]: pgmap v3027: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T01:39:07.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:39:06 smithi193 ceph-mon[36878]: pgmap v3027: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T01:39:09.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:39:08 smithi118 ceph-mon[31535]: pgmap v3028: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 5 op/s 2024-03-24T01:39:09.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:39:08 smithi189 ceph-mon[36187]: pgmap v3028: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 5 op/s 2024-03-24T01:39:09.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:39:08 smithi193 ceph-mon[36878]: pgmap v3028: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 5 op/s 2024-03-24T01:39:11.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:39:10 smithi189 ceph-mon[36187]: pgmap v3029: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T01:39:11.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:39:10 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 4446.678124 secs 2024-03-24T01:39:11.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:39:10 smithi193 ceph-mon[36878]: pgmap v3029: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T01:39:11.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:39:10 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 4446.678124 secs 2024-03-24T01:39:11.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:39:10 smithi118 ceph-mon[31535]: pgmap v3029: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T01:39:11.447 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:39:10 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 4446.678124 secs 2024-03-24T01:39:13.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:39:12 smithi189 ceph-mon[36187]: pgmap v3030: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:39:13.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:39:12 smithi193 ceph-mon[36878]: pgmap v3030: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:39:13.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:39:12 smithi118 ceph-mon[31535]: pgmap v3030: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:39:15.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:39:14 smithi193 ceph-mon[36878]: pgmap v3031: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T01:39:15.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:39:14 smithi118 ceph-mon[31535]: pgmap v3031: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T01:39:15.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:39:14 smithi189 ceph-mon[36187]: pgmap v3031: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T01:39:16.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:39:15 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 4451.678209 secs 2024-03-24T01:39:16.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:39:15 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 4451.678209 secs 2024-03-24T01:39:16.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:39:15 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 4451.678209 secs 2024-03-24T01:39:17.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:39:16 smithi118 ceph-mon[31535]: pgmap v3032: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T01:39:17.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:39:16 smithi189 ceph-mon[36187]: pgmap v3032: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T01:39:17.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:39:16 smithi193 ceph-mon[36878]: pgmap v3032: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T01:39:19.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:39:18 smithi118 ceph-mon[31535]: pgmap v3033: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 5 op/s 2024-03-24T01:39:19.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:39:18 smithi189 ceph-mon[36187]: pgmap v3033: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 5 op/s 2024-03-24T01:39:19.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:39:18 smithi193 ceph-mon[36878]: pgmap v3033: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 5 op/s 2024-03-24T01:39:21.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:39:20 smithi118 ceph-mon[31535]: pgmap v3034: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T01:39:21.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:39:20 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 4456.678271 secs 2024-03-24T01:39:21.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:39:20 smithi189 ceph-mon[36187]: pgmap v3034: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T01:39:21.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:39:20 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 4456.678271 secs 2024-03-24T01:39:21.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:39:20 smithi193 ceph-mon[36878]: pgmap v3034: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T01:39:21.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:39:20 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 4456.678271 secs 2024-03-24T01:39:23.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:39:22 smithi118 ceph-mon[31535]: pgmap v3035: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:39:23.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:39:22 smithi189 ceph-mon[36187]: pgmap v3035: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:39:23.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:39:22 smithi193 ceph-mon[36878]: pgmap v3035: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:39:25.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:39:24 smithi118 ceph-mon[31535]: pgmap v3036: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T01:39:25.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:39:24 smithi189 ceph-mon[36187]: pgmap v3036: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T01:39:25.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:39:24 smithi193 ceph-mon[36878]: pgmap v3036: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T01:39:26.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:39:25 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 4461.678332 secs 2024-03-24T01:39:26.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:39:25 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 4461.678332 secs 2024-03-24T01:39:26.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:39:25 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 4461.678332 secs 2024-03-24T01:39:27.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:39:27 smithi118 ceph-mon[31535]: pgmap v3037: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T01:39:27.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:39:27 smithi189 ceph-mon[36187]: pgmap v3037: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T01:39:27.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:39:27 smithi193 ceph-mon[36878]: pgmap v3037: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T01:39:29.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:39:29 smithi118 ceph-mon[31535]: pgmap v3038: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 5 op/s 2024-03-24T01:39:29.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:39:29 smithi189 ceph-mon[36187]: pgmap v3038: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 5 op/s 2024-03-24T01:39:29.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:39:29 smithi193 ceph-mon[36878]: pgmap v3038: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 5 op/s 2024-03-24T01:39:31.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:39:31 smithi118 ceph-mon[31535]: pgmap v3039: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T01:39:31.447 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:39:31 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 4466.678393 secs 2024-03-24T01:39:31.447 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T01:39:31.447+0000 7f81db400640 1 -- 172.21.15.118:0/659686424 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- auth(proto 2 132 bytes epoch 0) v1 -- 0x7f81c40064d0 con 0x5622651a13f0 2024-03-24T01:39:31.448 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T01:39:31.447+0000 7f81ee000640 1 -- 172.21.15.118:0/1693767697 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- auth(proto 2 132 bytes epoch 0) v1 -- 0x7f81c80036c0 con 0x7f81f81052d0 2024-03-24T01:39:31.448 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T01:39:31.448+0000 7f81e4c00640 1 -- 172.21.15.118:0/659686424 <== mon.1 v2:172.21.15.193:3300/0 38 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 233+0+0 (secure 0 0 0) 0x7f81c40064d0 con 0x5622651a13f0 2024-03-24T01:39:31.448 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T01:39:31.448+0000 7f81ef400640 1 -- 172.21.15.118:0/1693767697 <== mon.2 v2:172.21.15.189:3300/0 38 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 233+0+0 (secure 0 0 0) 0x7f81c80036c0 con 0x7f81f81052d0 2024-03-24T01:39:31.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:39:31 smithi189 ceph-mon[36187]: pgmap v3039: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T01:39:31.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:39:31 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 4466.678393 secs 2024-03-24T01:39:31.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:39:31 smithi193 ceph-mon[36878]: pgmap v3039: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T01:39:31.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:39:31 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 4466.678393 secs 2024-03-24T01:39:33.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:39:33 smithi118 ceph-mon[31535]: pgmap v3040: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:39:33.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:39:33 smithi189 ceph-mon[36187]: pgmap v3040: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:39:33.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:39:33 smithi193 ceph-mon[36878]: pgmap v3040: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:39:35.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:39:35 smithi118 ceph-mon[31535]: pgmap v3041: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T01:39:35.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:39:35 smithi189 ceph-mon[36187]: pgmap v3041: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T01:39:35.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:39:35 smithi193 ceph-mon[36878]: pgmap v3041: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T01:39:36.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:39:36 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 4471.678454 secs 2024-03-24T01:39:36.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:39:36 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 4471.678454 secs 2024-03-24T01:39:36.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:39:36 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 4471.678454 secs 2024-03-24T01:39:37.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:39:37 smithi118 ceph-mon[31535]: pgmap v3042: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T01:39:37.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:39:37 smithi189 ceph-mon[36187]: pgmap v3042: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T01:39:37.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:39:37 smithi193 ceph-mon[36878]: pgmap v3042: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T01:39:39.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:39:39 smithi118 ceph-mon[31535]: pgmap v3043: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 5 op/s 2024-03-24T01:39:39.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:39:39 smithi189 ceph-mon[36187]: pgmap v3043: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 5 op/s 2024-03-24T01:39:39.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:39:39 smithi193 ceph-mon[36878]: pgmap v3043: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 5 op/s 2024-03-24T01:39:41.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:39:41 smithi118 ceph-mon[31535]: pgmap v3044: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T01:39:41.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:39:41 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 4476.678543 secs 2024-03-24T01:39:41.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:39:41 smithi189 ceph-mon[36187]: pgmap v3044: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T01:39:41.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:39:41 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 4476.678543 secs 2024-03-24T01:39:41.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:39:41 smithi193 ceph-mon[36878]: pgmap v3044: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T01:39:41.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:39:41 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 4476.678543 secs 2024-03-24T01:39:43.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:39:43 smithi118 ceph-mon[31535]: pgmap v3045: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:39:43.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:39:43 smithi189 ceph-mon[36187]: pgmap v3045: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:39:43.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:39:43 smithi193 ceph-mon[36878]: pgmap v3045: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:39:45.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:39:45 smithi118 ceph-mon[31535]: pgmap v3046: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T01:39:45.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:39:45 smithi189 ceph-mon[36187]: pgmap v3046: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T01:39:45.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:39:45 smithi193 ceph-mon[36878]: pgmap v3046: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T01:39:46.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:39:46 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 4481.678632 secs 2024-03-24T01:39:46.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:39:46 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 4481.678632 secs 2024-03-24T01:39:46.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:39:46 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 4481.678632 secs 2024-03-24T01:39:47.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:39:47 smithi118 ceph-mon[31535]: pgmap v3047: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T01:39:47.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:39:47 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T01:39:47.447 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:39:47 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T01:39:47.447 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:39:47 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T01:39:47.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:39:47 smithi189 ceph-mon[36187]: pgmap v3047: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T01:39:47.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:39:47 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T01:39:47.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:39:47 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T01:39:47.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:39:47 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T01:39:47.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:39:47 smithi193 ceph-mon[36878]: pgmap v3047: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T01:39:47.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:39:47 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T01:39:47.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:39:47 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T01:39:47.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:39:47 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T01:39:48.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:39:48 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T01:39:48.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:39:48 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T01:39:48.197 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:39:48 smithi118 ceph-mon[31535]: pgmap v3048: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 5 op/s 2024-03-24T01:39:48.197 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:39:48 smithi118 ceph-mon[31535]: pgmap v3049: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 5 op/s 2024-03-24T01:39:48.197 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:39:48 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T01:39:48.455 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:39:48 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T01:39:48.455 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:39:48 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T01:39:48.455 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:39:48 smithi189 ceph-mon[36187]: pgmap v3048: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 5 op/s 2024-03-24T01:39:48.455 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:39:48 smithi189 ceph-mon[36187]: pgmap v3049: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 5 op/s 2024-03-24T01:39:48.455 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:39:48 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T01:39:48.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:39:48 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T01:39:48.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:39:48 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T01:39:48.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:39:48 smithi193 ceph-mon[36878]: pgmap v3048: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 5 op/s 2024-03-24T01:39:48.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:39:48 smithi193 ceph-mon[36878]: pgmap v3049: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 5 op/s 2024-03-24T01:39:48.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:39:48 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T01:39:50.936 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:39:50 smithi118 ceph-mon[31535]: pgmap v3050: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 6 op/s 2024-03-24T01:39:50.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:39:50 smithi189 ceph-mon[36187]: pgmap v3050: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 6 op/s 2024-03-24T01:39:50.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:39:50 smithi193 ceph-mon[36878]: pgmap v3050: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 6 op/s 2024-03-24T01:39:51.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:39:51 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 4486.678718 secs 2024-03-24T01:39:51.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:39:51 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 4486.678718 secs 2024-03-24T01:39:51.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:39:51 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 4486.678718 secs 2024-03-24T01:39:52.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:39:52 smithi118 ceph-mon[31535]: pgmap v3051: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 6 op/s 2024-03-24T01:39:52.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:39:52 smithi189 ceph-mon[36187]: pgmap v3051: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 6 op/s 2024-03-24T01:39:52.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:39:52 smithi193 ceph-mon[36878]: pgmap v3051: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 6 op/s 2024-03-24T01:39:54.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:39:54 smithi118 ceph-mon[31535]: pgmap v3052: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 5 op/s 2024-03-24T01:39:54.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:39:54 smithi189 ceph-mon[36187]: pgmap v3052: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 5 op/s 2024-03-24T01:39:54.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:39:54 smithi193 ceph-mon[36878]: pgmap v3052: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 5 op/s 2024-03-24T01:39:56.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:39:56 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 4491.678806 secs 2024-03-24T01:39:56.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:39:56 smithi118 ceph-mon[31535]: pgmap v3053: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 6 op/s 2024-03-24T01:39:56.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:39:56 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 4491.678806 secs 2024-03-24T01:39:56.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:39:56 smithi189 ceph-mon[36187]: pgmap v3053: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 6 op/s 2024-03-24T01:39:56.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:39:56 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 4491.678806 secs 2024-03-24T01:39:56.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:39:56 smithi193 ceph-mon[36878]: pgmap v3053: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 6 op/s 2024-03-24T01:39:58.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:39:58 smithi118 ceph-mon[31535]: pgmap v3054: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 5 op/s 2024-03-24T01:39:58.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:39:58 smithi189 ceph-mon[36187]: pgmap v3054: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 5 op/s 2024-03-24T01:39:58.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:39:58 smithi193 ceph-mon[36878]: pgmap v3054: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 5 op/s 2024-03-24T01:40:00.947 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:40:00 smithi118 ceph-mon[31535]: pgmap v3055: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-03-24T01:40:00.947 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:40:00 smithi118 ceph-mon[31535]: overall HEALTH_WARN 6 failed cephadm daemon(s); 1 MDSs report slow requests 2024-03-24T01:40:00.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:40:00 smithi189 ceph-mon[36187]: pgmap v3055: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-03-24T01:40:00.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:40:00 smithi189 ceph-mon[36187]: overall HEALTH_WARN 6 failed cephadm daemon(s); 1 MDSs report slow requests 2024-03-24T01:40:00.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:40:00 smithi193 ceph-mon[36878]: pgmap v3055: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-03-24T01:40:00.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:40:00 smithi193 ceph-mon[36878]: overall HEALTH_WARN 6 failed cephadm daemon(s); 1 MDSs report slow requests 2024-03-24T01:40:01.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:40:01 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 4496.678896 secs 2024-03-24T01:40:01.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:40:01 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 4496.678896 secs 2024-03-24T01:40:01.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:40:01 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 4496.678896 secs 2024-03-24T01:40:02.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:40:02 smithi118 ceph-mon[31535]: pgmap v3056: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-03-24T01:40:02.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:40:02 smithi189 ceph-mon[36187]: pgmap v3056: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-03-24T01:40:02.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:40:02 smithi193 ceph-mon[36878]: pgmap v3056: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-03-24T01:40:04.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:40:04 smithi118 ceph-mon[31535]: pgmap v3057: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-03-24T01:40:04.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:40:04 smithi189 ceph-mon[36187]: pgmap v3057: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-03-24T01:40:04.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:40:04 smithi193 ceph-mon[36878]: pgmap v3057: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-03-24T01:40:06.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:40:06 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 4501.678984 secs 2024-03-24T01:40:06.947 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:40:06 smithi118 ceph-mon[31535]: pgmap v3058: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 6 op/s 2024-03-24T01:40:06.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:40:06 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 4501.678984 secs 2024-03-24T01:40:06.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:40:06 smithi189 ceph-mon[36187]: pgmap v3058: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 6 op/s 2024-03-24T01:40:06.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:40:06 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 4501.678984 secs 2024-03-24T01:40:06.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:40:06 smithi193 ceph-mon[36878]: pgmap v3058: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 6 op/s 2024-03-24T01:40:08.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:40:08 smithi118 ceph-mon[31535]: pgmap v3059: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 5 op/s 2024-03-24T01:40:08.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:40:08 smithi189 ceph-mon[36187]: pgmap v3059: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 5 op/s 2024-03-24T01:40:08.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:40:08 smithi193 ceph-mon[36878]: pgmap v3059: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 5 op/s 2024-03-24T01:40:10.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:40:10 smithi118 ceph-mon[31535]: pgmap v3060: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-03-24T01:40:10.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:40:10 smithi189 ceph-mon[36187]: pgmap v3060: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-03-24T01:40:10.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:40:10 smithi193 ceph-mon[36878]: pgmap v3060: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-03-24T01:40:11.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:40:11 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 4506.679046 secs 2024-03-24T01:40:11.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:40:11 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 4506.679046 secs 2024-03-24T01:40:11.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:40:11 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 4506.679046 secs 2024-03-24T01:40:12.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:40:12 smithi118 ceph-mon[31535]: pgmap v3061: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:40:12.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:40:12 smithi189 ceph-mon[36187]: pgmap v3061: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:40:12.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:40:12 smithi193 ceph-mon[36878]: pgmap v3061: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:40:14.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:40:14 smithi118 ceph-mon[31535]: pgmap v3062: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T01:40:14.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:40:14 smithi189 ceph-mon[36187]: pgmap v3062: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T01:40:14.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:40:14 smithi193 ceph-mon[36878]: pgmap v3062: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T01:40:15.745 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T01:40:15.744+0000 7f81f6000640 1 -- 172.21.15.118:0/1693767697 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7f81dc0a33e0 msgr2=0x7f81dc0a3bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 12 2024-03-24T01:40:15.745 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T01:40:15.744+0000 7f81f6000640 1 -- 172.21.15.118:0/1693767697 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7f81dc0a33e0 msgr2=0x7f81dc0a3bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-03-24T01:40:15.745 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T01:40:15.744+0000 7f81f6000640 1 --2- 172.21.15.118:0/1693767697 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7f81dc0a33e0 0x7f81dc0a3bd0 secure :-1 s=READY pgs=480 cs=0 l=1 rev1=1 crypto rx=0x562265215970 tx=0x7f81e004e040 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-03-24T01:40:15.745 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T01:40:15.744+0000 7f81f6000640 1 --2- 172.21.15.118:0/1693767697 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7f81dc0a33e0 0x7f81dc0a3bd0 secure :-1 s=READY pgs=480 cs=0 l=1 rev1=1 crypto rx=0x562265215970 tx=0x7f81e004e040 comp rx=0 tx=0).stop 2024-03-24T01:40:15.745 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T01:40:15.744+0000 7f81f6000640 1 -- 172.21.15.118:0/1693767697 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7f81dc0a33e0 msgr2=0x7f81dc0a3bd0 unknown :-1 s=STATE_CLOSED l=1).tick see no progress in more than 10000000 us during connecting to v2:172.21.15.118:6800/2112912943, fault. 2024-03-24T01:40:15.745 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T01:40:15.744+0000 7f81ef400640 1 -- 172.21.15.118:0/1693767697 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7f81dc0a33e0 msgr2=0x7f81dc0a3bd0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-03-24T01:40:15.745 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T01:40:15.744+0000 7f81ef400640 1 --2- 172.21.15.118:0/1693767697 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7f81dc0a33e0 0x7f81dc0a3bd0 unknown :-1 s=CLOSED pgs=480 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:40:15.746 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T01:40:15.744+0000 7f81ef400640 1 --2- 172.21.15.118:0/1693767697 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7f81dc0a4320 0x7f81dc0b05c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:40:15.746 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T01:40:15.745+0000 7f81f6000640 1 --2- 172.21.15.118:0/1693767697 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7f81dc0a4320 0x7f81dc0b05c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:40:15.747 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T01:40:15.747+0000 7f81f6000640 1 --2- 172.21.15.118:0/1693767697 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7f81dc0a4320 0x7f81dc0b05c0 secure :-1 s=READY pgs=501 cs=0 l=1 rev1=1 crypto rx=0x7f81e00047d0 tx=0x7f81e0044040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:40:15.763 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T01:40:15.763+0000 7f81f6000640 1 -- 172.21.15.118:0/659686424 >> [v2:172.21.15.189:6838/4074484168,v1:172.21.15.189:6839/4074484168] conn(0x5622651b71e0 msgr2=0x562265215110 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).tick idle (900000773) for more than 900000000 us, fault. 2024-03-24T01:40:15.763 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T01:40:15.763+0000 7f81f6000640 1 --2- 172.21.15.118:0/659686424 >> [v2:172.21.15.189:6838/4074484168,v1:172.21.15.189:6839/4074484168] conn(0x5622651b71e0 0x562265215110 unknown :-1 s=READY pgs=561 cs=3 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault initiating reconnect 2024-03-24T01:40:15.763 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T01:40:15.763+0000 7f81f6000640 1 --2- 172.21.15.118:0/659686424 >> [v2:172.21.15.189:6838/4074484168,v1:172.21.15.189:6839/4074484168] conn(0x5622651b71e0 0x562265215110 unknown :-1 s=BANNER_CONNECTING pgs=561 cs=4 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:40:15.764 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T01:40:15.764+0000 7f81f6000640 1 --2- 172.21.15.118:0/659686424 >> [v2:172.21.15.189:6838/4074484168,v1:172.21.15.189:6839/4074484168] conn(0x5622651b71e0 0x562265215110 crc :-1 s=READY pgs=561 cs=4 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=76f3d534289740f0 server_cookie=b31a2335a59552de in_seq=0 out_seq=0 2024-03-24T01:40:15.764 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T01:40:15.764+0000 7f81e4c00640 10 client.27094 ms_handle_connect on v2:172.21.15.189:6838/4074484168 2024-03-24T01:40:15.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:40:15 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 4511.679109 secs 2024-03-24T01:40:15.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:40:15 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 4511.679109 secs 2024-03-24T01:40:15.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:40:15 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 4511.679109 secs 2024-03-24T01:40:16.455 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T01:40:16.455+0000 7f81ee000640 1 -- 172.21.15.118:0/1693767697 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f81c80036c0 con 0x7f81f81052d0 2024-03-24T01:40:16.455 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T01:40:16.455+0000 7f81db400640 1 -- 172.21.15.118:0/659686424 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f81c40064d0 con 0x5622651a13f0 2024-03-24T01:40:16.456 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T01:40:16.455+0000 7f81e4c00640 1 -- 172.21.15.118:0/659686424 <== mon.1 v2:172.21.15.193:3300/0 39 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 409+0+0 (secure 0 0 0) 0x7f81c40064d0 con 0x5622651a13f0 2024-03-24T01:40:16.456 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T01:40:16.455+0000 7f81ef400640 1 -- 172.21.15.118:0/1693767697 <== mon.2 v2:172.21.15.189:3300/0 39 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (secure 0 0 0) 0x7f81c80036c0 con 0x7f81f81052d0 2024-03-24T01:40:16.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:40:16 smithi118 ceph-mon[31535]: pgmap v3063: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T01:40:16.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:40:16 smithi189 ceph-mon[36187]: pgmap v3063: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T01:40:16.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:40:16 smithi193 ceph-mon[36878]: pgmap v3063: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T01:40:18.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:40:18 smithi118 ceph-mon[31535]: pgmap v3064: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T01:40:18.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:40:18 smithi189 ceph-mon[36187]: pgmap v3064: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T01:40:18.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:40:18 smithi193 ceph-mon[36878]: pgmap v3064: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T01:40:20.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:40:20 smithi118 ceph-mon[31535]: pgmap v3065: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-03-24T01:40:20.947 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:40:20 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 4516.679171 secs 2024-03-24T01:40:20.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:40:20 smithi189 ceph-mon[36187]: pgmap v3065: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-03-24T01:40:20.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:40:20 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 4516.679171 secs 2024-03-24T01:40:20.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:40:20 smithi193 ceph-mon[36878]: pgmap v3065: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-03-24T01:40:20.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:40:20 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 4516.679171 secs 2024-03-24T01:40:22.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:40:22 smithi118 ceph-mon[31535]: pgmap v3066: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:40:22.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:40:22 smithi189 ceph-mon[36187]: pgmap v3066: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:40:22.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:40:22 smithi193 ceph-mon[36878]: pgmap v3066: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:40:24.947 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:40:24 smithi118 ceph-mon[31535]: pgmap v3067: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T01:40:24.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:40:24 smithi189 ceph-mon[36187]: pgmap v3067: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T01:40:24.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:40:24 smithi193 ceph-mon[36878]: pgmap v3067: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T01:40:25.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:40:25 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 4521.679261 secs 2024-03-24T01:40:25.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:40:25 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 4521.679261 secs 2024-03-24T01:40:25.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:40:25 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 4521.679261 secs 2024-03-24T01:40:26.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:40:26 smithi118 ceph-mon[31535]: pgmap v3068: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T01:40:26.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:40:26 smithi189 ceph-mon[36187]: pgmap v3068: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T01:40:26.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:40:26 smithi193 ceph-mon[36878]: pgmap v3068: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T01:40:28.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:40:28 smithi118 ceph-mon[31535]: pgmap v3069: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T01:40:28.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:40:28 smithi189 ceph-mon[36187]: pgmap v3069: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T01:40:28.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:40:28 smithi193 ceph-mon[36878]: pgmap v3069: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T01:40:30.947 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:40:30 smithi118 ceph-mon[31535]: pgmap v3070: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-03-24T01:40:30.947 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:40:30 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 4526.679324 secs 2024-03-24T01:40:30.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:40:30 smithi189 ceph-mon[36187]: pgmap v3070: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-03-24T01:40:30.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:40:30 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 4526.679324 secs 2024-03-24T01:40:30.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:40:30 smithi193 ceph-mon[36878]: pgmap v3070: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-03-24T01:40:30.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:40:30 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 4526.679324 secs 2024-03-24T01:40:32.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:40:32 smithi118 ceph-mon[31535]: pgmap v3071: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:40:32.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:40:32 smithi189 ceph-mon[36187]: pgmap v3071: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:40:32.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:40:32 smithi193 ceph-mon[36878]: pgmap v3071: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:40:34.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:40:34 smithi118 ceph-mon[31535]: pgmap v3072: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T01:40:34.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:40:34 smithi189 ceph-mon[36187]: pgmap v3072: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T01:40:34.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:40:34 smithi193 ceph-mon[36878]: pgmap v3072: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T01:40:35.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:40:35 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 4531.679339 secs 2024-03-24T01:40:35.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:40:35 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 4531.679339 secs 2024-03-24T01:40:35.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:40:35 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 4531.679339 secs 2024-03-24T01:40:36.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:40:36 smithi118 ceph-mon[31535]: pgmap v3073: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T01:40:36.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:40:36 smithi189 ceph-mon[36187]: pgmap v3073: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T01:40:36.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:40:36 smithi193 ceph-mon[36878]: pgmap v3073: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T01:40:38.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:40:38 smithi118 ceph-mon[31535]: pgmap v3074: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T01:40:38.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:40:38 smithi189 ceph-mon[36187]: pgmap v3074: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T01:40:38.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:40:38 smithi193 ceph-mon[36878]: pgmap v3074: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T01:40:40.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:40:40 smithi118 ceph-mon[31535]: pgmap v3075: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-03-24T01:40:40.947 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:40:40 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 4536.679401 secs 2024-03-24T01:40:40.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:40:40 smithi189 ceph-mon[36187]: pgmap v3075: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-03-24T01:40:40.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:40:40 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 4536.679401 secs 2024-03-24T01:40:40.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:40:40 smithi193 ceph-mon[36878]: pgmap v3075: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-03-24T01:40:40.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:40:40 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 4536.679401 secs 2024-03-24T01:40:42.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:40:42 smithi189 ceph-mon[36187]: pgmap v3076: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:40:42.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:40:42 smithi193 ceph-mon[36878]: pgmap v3076: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:40:43.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:40:42 smithi118 ceph-mon[31535]: pgmap v3076: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:40:44.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:40:44 smithi189 ceph-mon[36187]: pgmap v3077: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T01:40:44.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:40:44 smithi193 ceph-mon[36878]: pgmap v3077: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T01:40:45.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:40:44 smithi118 ceph-mon[31535]: pgmap v3077: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T01:40:46.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:40:45 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 4541.679441 secs 2024-03-24T01:40:46.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:40:45 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 4541.679441 secs 2024-03-24T01:40:46.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:40:45 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 4541.679441 secs 2024-03-24T01:40:47.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:40:46 smithi118 ceph-mon[31535]: pgmap v3078: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T01:40:47.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:40:46 smithi189 ceph-mon[36187]: pgmap v3078: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T01:40:47.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:40:46 smithi193 ceph-mon[36878]: pgmap v3078: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T01:40:49.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:40:48 smithi118 ceph-mon[31535]: pgmap v3079: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T01:40:49.197 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:40:48 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T01:40:49.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:40:48 smithi189 ceph-mon[36187]: pgmap v3079: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T01:40:49.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:40:48 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T01:40:49.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:40:48 smithi193 ceph-mon[36878]: pgmap v3079: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T01:40:49.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:40:48 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T01:40:50.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:40:49 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T01:40:50.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:40:49 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T01:40:50.197 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:40:49 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T01:40:50.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:40:49 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T01:40:50.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:40:49 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T01:40:50.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:40:49 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T01:40:50.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:40:49 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T01:40:50.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:40:49 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T01:40:50.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:40:49 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T01:40:51.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:40:50 smithi118 ceph-mon[31535]: pgmap v3080: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 6 op/s 2024-03-24T01:40:51.197 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:40:50 smithi118 ceph-mon[31535]: pgmap v3081: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 5 op/s 2024-03-24T01:40:51.197 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:40:50 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 4546.679503 secs 2024-03-24T01:40:51.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:40:50 smithi189 ceph-mon[36187]: pgmap v3080: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 6 op/s 2024-03-24T01:40:51.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:40:50 smithi189 ceph-mon[36187]: pgmap v3081: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 5 op/s 2024-03-24T01:40:51.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:40:50 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 4546.679503 secs 2024-03-24T01:40:51.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:40:50 smithi193 ceph-mon[36878]: pgmap v3080: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 6 op/s 2024-03-24T01:40:51.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:40:50 smithi193 ceph-mon[36878]: pgmap v3081: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 5 op/s 2024-03-24T01:40:51.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:40:50 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 4546.679503 secs 2024-03-24T01:40:53.000 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:40:52 smithi118 ceph-mon[31535]: pgmap v3082: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-03-24T01:40:53.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:40:52 smithi189 ceph-mon[36187]: pgmap v3082: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-03-24T01:40:53.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:40:52 smithi193 ceph-mon[36878]: pgmap v3082: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-03-24T01:40:55.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:40:54 smithi118 ceph-mon[31535]: pgmap v3083: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-03-24T01:40:55.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:40:54 smithi189 ceph-mon[36187]: pgmap v3083: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-03-24T01:40:55.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:40:54 smithi193 ceph-mon[36878]: pgmap v3083: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-03-24T01:40:56.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:40:55 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 4551.679563 secs 2024-03-24T01:40:56.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:40:55 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 4551.679563 secs 2024-03-24T01:40:56.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:40:55 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 4551.679563 secs 2024-03-24T01:40:57.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:40:56 smithi118 ceph-mon[31535]: pgmap v3084: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-03-24T01:40:57.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:40:56 smithi189 ceph-mon[36187]: pgmap v3084: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-03-24T01:40:57.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:40:56 smithi193 ceph-mon[36878]: pgmap v3084: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-03-24T01:40:59.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:40:58 smithi118 ceph-mon[31535]: pgmap v3085: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 5 op/s 2024-03-24T01:40:59.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:40:58 smithi189 ceph-mon[36187]: pgmap v3085: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 5 op/s 2024-03-24T01:40:59.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:40:58 smithi193 ceph-mon[36878]: pgmap v3085: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 5 op/s 2024-03-24T01:41:01.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:41:00 smithi118 ceph-mon[31535]: pgmap v3086: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 5 op/s 2024-03-24T01:41:01.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:41:00 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 4556.679652 secs 2024-03-24T01:41:01.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:41:00 smithi189 ceph-mon[36187]: pgmap v3086: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 5 op/s 2024-03-24T01:41:01.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:41:00 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 4556.679652 secs 2024-03-24T01:41:01.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:41:00 smithi193 ceph-mon[36878]: pgmap v3086: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 5 op/s 2024-03-24T01:41:01.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:41:00 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 4556.679652 secs 2024-03-24T01:41:03.197 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:41:02 smithi118 ceph-mon[31535]: pgmap v3087: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T01:41:03.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:41:02 smithi189 ceph-mon[36187]: pgmap v3087: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T01:41:03.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:41:02 smithi193 ceph-mon[36878]: pgmap v3087: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T01:41:05.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:41:04 smithi118 ceph-mon[31535]: pgmap v3088: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T01:41:05.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:41:04 smithi189 ceph-mon[36187]: pgmap v3088: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T01:41:05.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:41:04 smithi193 ceph-mon[36878]: pgmap v3088: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T01:41:06.105 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:41:05 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 4561.679740 secs 2024-03-24T01:41:06.112 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:41:05 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 4561.679740 secs 2024-03-24T01:41:06.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:41:05 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 4561.679740 secs 2024-03-24T01:41:07.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:41:06 smithi118 ceph-mon[31535]: pgmap v3089: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T01:41:07.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:41:06 smithi189 ceph-mon[36187]: pgmap v3089: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T01:41:07.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:41:06 smithi193 ceph-mon[36878]: pgmap v3089: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T01:41:09.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:41:08 smithi118 ceph-mon[31535]: pgmap v3090: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:41:09.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:41:08 smithi189 ceph-mon[36187]: pgmap v3090: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:41:09.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:41:08 smithi193 ceph-mon[36878]: pgmap v3090: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:41:11.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:41:10 smithi118 ceph-mon[31535]: pgmap v3091: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T01:41:11.197 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:41:10 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 4566.679829 secs 2024-03-24T01:41:11.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:41:10 smithi189 ceph-mon[36187]: pgmap v3091: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T01:41:11.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:41:10 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 4566.679829 secs 2024-03-24T01:41:11.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:41:10 smithi193 ceph-mon[36878]: pgmap v3091: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T01:41:11.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:41:10 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 4566.679829 secs 2024-03-24T01:41:13.197 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:41:12 smithi118 ceph-mon[31535]: pgmap v3092: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T01:41:13.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:41:12 smithi189 ceph-mon[36187]: pgmap v3092: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T01:41:13.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:41:12 smithi193 ceph-mon[36878]: pgmap v3092: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T01:41:15.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:41:14 smithi118 ceph-mon[31535]: pgmap v3093: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T01:41:15.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:41:14 smithi189 ceph-mon[36187]: pgmap v3093: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T01:41:15.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:41:14 smithi193 ceph-mon[36878]: pgmap v3093: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T01:41:16.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:41:15 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 4571.679916 secs 2024-03-24T01:41:16.205 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:41:15 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 4571.679916 secs 2024-03-24T01:41:16.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:41:15 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 4571.679916 secs 2024-03-24T01:41:17.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:41:16 smithi118 ceph-mon[31535]: pgmap v3094: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T01:41:17.206 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:41:16 smithi189 ceph-mon[36187]: pgmap v3094: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T01:41:17.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:41:16 smithi193 ceph-mon[36878]: pgmap v3094: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T01:41:19.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:41:18 smithi118 ceph-mon[31535]: pgmap v3095: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:41:19.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:41:18 smithi189 ceph-mon[36187]: pgmap v3095: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:41:19.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:41:18 smithi193 ceph-mon[36878]: pgmap v3095: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:41:21.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:41:20 smithi118 ceph-mon[31535]: pgmap v3096: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T01:41:21.197 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:41:20 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 4576.680003 secs 2024-03-24T01:41:21.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:41:20 smithi189 ceph-mon[36187]: pgmap v3096: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T01:41:21.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:41:20 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 4576.680003 secs 2024-03-24T01:41:21.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:41:20 smithi193 ceph-mon[36878]: pgmap v3096: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T01:41:21.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:41:20 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 4576.680003 secs 2024-03-24T01:41:23.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:41:22 smithi118 ceph-mon[31535]: pgmap v3097: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T01:41:23.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:41:22 smithi189 ceph-mon[36187]: pgmap v3097: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T01:41:23.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:41:22 smithi193 ceph-mon[36878]: pgmap v3097: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T01:41:25.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:41:24 smithi118 ceph-mon[31535]: pgmap v3098: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T01:41:25.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:41:24 smithi189 ceph-mon[36187]: pgmap v3098: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T01:41:25.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:41:24 smithi193 ceph-mon[36878]: pgmap v3098: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T01:41:26.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:41:25 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 4581.680043 secs 2024-03-24T01:41:26.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:41:25 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 4581.680043 secs 2024-03-24T01:41:26.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:41:25 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 4581.680043 secs 2024-03-24T01:41:27.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:41:26 smithi118 ceph-mon[31535]: pgmap v3099: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T01:41:27.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:41:26 smithi189 ceph-mon[36187]: pgmap v3099: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T01:41:27.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:41:26 smithi193 ceph-mon[36878]: pgmap v3099: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T01:41:29.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:41:28 smithi118 ceph-mon[31535]: pgmap v3100: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:41:29.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:41:28 smithi189 ceph-mon[36187]: pgmap v3100: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:41:29.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:41:28 smithi193 ceph-mon[36878]: pgmap v3100: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:41:30.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:41:29 smithi118 ceph-mon[31535]: pgmap v3101: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T01:41:30.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:41:29 smithi189 ceph-mon[36187]: pgmap v3101: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T01:41:30.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:41:29 smithi193 ceph-mon[36878]: pgmap v3101: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T01:41:31.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:41:30 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 4586.680099 secs 2024-03-24T01:41:31.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:41:30 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 4586.680099 secs 2024-03-24T01:41:31.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:41:30 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 4586.680099 secs 2024-03-24T01:41:32.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:41:31 smithi118 ceph-mon[31535]: pgmap v3102: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T01:41:32.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:41:31 smithi189 ceph-mon[36187]: pgmap v3102: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T01:41:32.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:41:31 smithi193 ceph-mon[36878]: pgmap v3102: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T01:41:34.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:41:34 smithi118 ceph-mon[31535]: pgmap v3103: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T01:41:34.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:41:34 smithi189 ceph-mon[36187]: pgmap v3103: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T01:41:34.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:41:34 smithi193 ceph-mon[36878]: pgmap v3103: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T01:41:36.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:41:36 smithi118 ceph-mon[31535]: pgmap v3104: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T01:41:36.447 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:41:36 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 4591.680181 secs 2024-03-24T01:41:36.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:41:36 smithi189 ceph-mon[36187]: pgmap v3104: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T01:41:36.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:41:36 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 4591.680181 secs 2024-03-24T01:41:36.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:41:36 smithi193 ceph-mon[36878]: pgmap v3104: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T01:41:36.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:41:36 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 4591.680181 secs 2024-03-24T01:41:38.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:41:38 smithi118 ceph-mon[31535]: pgmap v3105: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:41:38.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:41:38 smithi189 ceph-mon[36187]: pgmap v3105: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:41:38.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:41:38 smithi193 ceph-mon[36878]: pgmap v3105: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:41:40.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:41:40 smithi118 ceph-mon[31535]: pgmap v3106: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-24T01:41:40.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:41:40 smithi189 ceph-mon[36187]: pgmap v3106: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-24T01:41:40.462 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:41:40 smithi193 ceph-mon[36878]: pgmap v3106: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-24T01:41:41.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:41:41 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 4596.680222 secs 2024-03-24T01:41:41.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:41:41 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 4596.680222 secs 2024-03-24T01:41:41.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:41:41 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 4596.680222 secs 2024-03-24T01:41:42.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:41:42 smithi118 ceph-mon[31535]: pgmap v3107: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T01:41:42.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:41:42 smithi189 ceph-mon[36187]: pgmap v3107: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T01:41:42.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:41:42 smithi193 ceph-mon[36878]: pgmap v3107: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T01:41:44.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:41:44 smithi118 ceph-mon[31535]: pgmap v3108: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T01:41:44.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:41:44 smithi189 ceph-mon[36187]: pgmap v3108: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T01:41:44.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:41:44 smithi193 ceph-mon[36878]: pgmap v3108: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T01:41:46.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:41:46 smithi118 ceph-mon[31535]: pgmap v3109: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T01:41:46.447 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:41:46 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 4601.680239 secs 2024-03-24T01:41:46.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:41:46 smithi189 ceph-mon[36187]: pgmap v3109: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T01:41:46.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:41:46 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 4601.680239 secs 2024-03-24T01:41:46.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:41:46 smithi193 ceph-mon[36878]: pgmap v3109: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T01:41:46.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:41:46 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 4601.680239 secs 2024-03-24T01:41:48.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:41:48 smithi118 ceph-mon[31535]: pgmap v3110: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:41:48.455 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:41:48 smithi189 ceph-mon[36187]: pgmap v3110: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:41:48.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:41:48 smithi193 ceph-mon[36878]: pgmap v3110: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:41:50.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:41:50 smithi118 ceph-mon[31535]: pgmap v3111: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-24T01:41:50.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:41:50 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T01:41:50.455 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:41:50 smithi189 ceph-mon[36187]: pgmap v3111: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-24T01:41:50.455 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:41:50 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T01:41:50.462 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:41:50 smithi193 ceph-mon[36878]: pgmap v3111: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-24T01:41:50.462 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:41:50 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T01:41:51.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:41:51 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 4606.680324 secs 2024-03-24T01:41:51.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:41:51 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 4606.680324 secs 2024-03-24T01:41:51.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:41:51 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 4606.680324 secs 2024-03-24T01:41:52.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:41:52 smithi118 ceph-mon[31535]: pgmap v3112: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-03-24T01:41:52.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:41:52 smithi189 ceph-mon[36187]: pgmap v3112: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-03-24T01:41:52.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:41:52 smithi193 ceph-mon[36878]: pgmap v3112: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-03-24T01:41:54.447 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:41:54 smithi118 ceph-mon[31535]: pgmap v3113: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-03-24T01:41:54.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:41:54 smithi189 ceph-mon[36187]: pgmap v3113: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-03-24T01:41:54.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:41:54 smithi193 ceph-mon[36878]: pgmap v3113: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-03-24T01:41:56.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:41:56 smithi118 ceph-mon[31535]: pgmap v3114: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 6 op/s 2024-03-24T01:41:56.447 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:41:56 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 4611.680413 secs 2024-03-24T01:41:56.447 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:41:56 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T01:41:56.447 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:41:56 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T01:41:56.447 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:41:56 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T01:41:56.447 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:41:56 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T01:41:56.447 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:41:56 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T01:41:56.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:41:56 smithi189 ceph-mon[36187]: pgmap v3114: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 6 op/s 2024-03-24T01:41:56.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:41:56 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 4611.680413 secs 2024-03-24T01:41:56.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:41:56 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T01:41:56.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:41:56 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T01:41:56.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:41:56 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T01:41:56.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:41:56 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T01:41:56.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:41:56 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T01:41:56.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:41:56 smithi193 ceph-mon[36878]: pgmap v3114: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 6 op/s 2024-03-24T01:41:56.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:41:56 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 4611.680413 secs 2024-03-24T01:41:56.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:41:56 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T01:41:56.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:41:56 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T01:41:56.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:41:56 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T01:41:56.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:41:56 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T01:41:56.462 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:41:56 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T01:41:57.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:41:57 smithi118 ceph-mon[31535]: pgmap v3115: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 5 op/s 2024-03-24T01:41:57.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:41:57 smithi118 ceph-mon[31535]: pgmap v3116: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T01:41:57.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:41:57 smithi189 ceph-mon[36187]: pgmap v3115: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 5 op/s 2024-03-24T01:41:57.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:41:57 smithi189 ceph-mon[36187]: pgmap v3116: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T01:41:57.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:41:57 smithi193 ceph-mon[36878]: pgmap v3115: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 5 op/s 2024-03-24T01:41:57.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:41:57 smithi193 ceph-mon[36878]: pgmap v3116: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T01:41:59.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:41:59 smithi118 ceph-mon[31535]: pgmap v3117: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 4 op/s 2024-03-24T01:41:59.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:41:59 smithi189 ceph-mon[36187]: pgmap v3117: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 4 op/s 2024-03-24T01:41:59.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:41:59 smithi193 ceph-mon[36878]: pgmap v3117: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 4 op/s 2024-03-24T01:42:01.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:42:01 smithi118 ceph-mon[31535]: pgmap v3118: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 6 op/s 2024-03-24T01:42:01.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:42:01 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 4616.680476 secs 2024-03-24T01:42:01.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:42:01 smithi189 ceph-mon[36187]: pgmap v3118: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 6 op/s 2024-03-24T01:42:01.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:42:01 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 4616.680476 secs 2024-03-24T01:42:01.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:42:01 smithi193 ceph-mon[36878]: pgmap v3118: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 6 op/s 2024-03-24T01:42:01.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:42:01 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 4616.680476 secs 2024-03-24T01:42:03.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:42:03 smithi118 ceph-mon[31535]: pgmap v3119: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 5 op/s 2024-03-24T01:42:03.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:42:03 smithi189 ceph-mon[36187]: pgmap v3119: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 5 op/s 2024-03-24T01:42:03.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:42:03 smithi193 ceph-mon[36878]: pgmap v3119: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 5 op/s 2024-03-24T01:42:05.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:42:05 smithi118 ceph-mon[31535]: pgmap v3120: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 5 op/s 2024-03-24T01:42:05.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:42:05 smithi189 ceph-mon[36187]: pgmap v3120: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 5 op/s 2024-03-24T01:42:05.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:42:05 smithi193 ceph-mon[36878]: pgmap v3120: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 5 op/s 2024-03-24T01:42:06.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:42:06 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 4621.680563 secs 2024-03-24T01:42:06.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:42:06 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 4621.680563 secs 2024-03-24T01:42:06.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:42:06 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 4621.680563 secs 2024-03-24T01:42:07.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:42:07 smithi118 ceph-mon[31535]: pgmap v3121: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 5 op/s 2024-03-24T01:42:07.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:42:07 smithi189 ceph-mon[36187]: pgmap v3121: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 5 op/s 2024-03-24T01:42:07.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:42:07 smithi193 ceph-mon[36878]: pgmap v3121: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 5 op/s 2024-03-24T01:42:09.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:42:09 smithi118 ceph-mon[31535]: pgmap v3122: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 5 op/s 2024-03-24T01:42:09.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:42:09 smithi189 ceph-mon[36187]: pgmap v3122: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 5 op/s 2024-03-24T01:42:09.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:42:09 smithi193 ceph-mon[36878]: pgmap v3122: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 5 op/s 2024-03-24T01:42:11.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:42:11 smithi118 ceph-mon[31535]: pgmap v3123: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T01:42:11.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:42:11 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 4626.680624 secs 2024-03-24T01:42:11.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:42:11 smithi189 ceph-mon[36187]: pgmap v3123: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T01:42:11.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:42:11 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 4626.680624 secs 2024-03-24T01:42:11.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:42:11 smithi193 ceph-mon[36878]: pgmap v3123: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T01:42:11.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:42:11 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 4626.680624 secs 2024-03-24T01:42:13.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:42:13 smithi118 ceph-mon[31535]: pgmap v3124: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:42:13.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:42:13 smithi189 ceph-mon[36187]: pgmap v3124: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:42:13.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:42:13 smithi193 ceph-mon[36878]: pgmap v3124: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:42:15.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:42:15 smithi118 ceph-mon[31535]: pgmap v3125: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T01:42:15.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:42:15 smithi189 ceph-mon[36187]: pgmap v3125: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T01:42:15.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:42:15 smithi193 ceph-mon[36878]: pgmap v3125: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T01:42:16.462 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:42:16 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 4631.680712 secs 2024-03-24T01:42:16.698 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:42:16 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 4631.680712 secs 2024-03-24T01:42:16.704 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:42:16 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 4631.680712 secs 2024-03-24T01:42:17.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:42:17 smithi193 ceph-mon[36878]: pgmap v3126: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T01:42:17.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:42:17 smithi118 ceph-mon[31535]: pgmap v3126: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T01:42:17.704 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:42:17 smithi189 ceph-mon[36187]: pgmap v3126: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T01:42:19.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:42:19 smithi193 ceph-mon[36878]: pgmap v3127: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 5 op/s 2024-03-24T01:42:19.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:42:19 smithi118 ceph-mon[31535]: pgmap v3127: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 5 op/s 2024-03-24T01:42:19.704 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:42:19 smithi189 ceph-mon[36187]: pgmap v3127: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 5 op/s 2024-03-24T01:42:21.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:42:21 smithi118 ceph-mon[31535]: pgmap v3128: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T01:42:21.697 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:42:21 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 4636.680797 secs 2024-03-24T01:42:21.704 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:42:21 smithi189 ceph-mon[36187]: pgmap v3128: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T01:42:21.704 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:42:21 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 4636.680797 secs 2024-03-24T01:42:21.711 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:42:21 smithi193 ceph-mon[36878]: pgmap v3128: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T01:42:21.711 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:42:21 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 4636.680797 secs 2024-03-24T01:42:23.499 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:42:23 smithi118 ceph-mon[31535]: pgmap v3129: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:42:23.704 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:42:23 smithi189 ceph-mon[36187]: pgmap v3129: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:42:23.711 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:42:23 smithi193 ceph-mon[36878]: pgmap v3129: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:42:25.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:42:25 smithi118 ceph-mon[31535]: pgmap v3130: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T01:42:25.704 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:42:25 smithi189 ceph-mon[36187]: pgmap v3130: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T01:42:25.711 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:42:25 smithi193 ceph-mon[36878]: pgmap v3130: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T01:42:26.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:42:26 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 4641.680885 secs 2024-03-24T01:42:26.704 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:42:26 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 4641.680885 secs 2024-03-24T01:42:26.711 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:42:26 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 4641.680885 secs 2024-03-24T01:42:27.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:42:27 smithi118 ceph-mon[31535]: pgmap v3131: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T01:42:27.704 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:42:27 smithi189 ceph-mon[36187]: pgmap v3131: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T01:42:27.711 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:42:27 smithi193 ceph-mon[36878]: pgmap v3131: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T01:42:29.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:42:29 smithi118 ceph-mon[31535]: pgmap v3132: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 5 op/s 2024-03-24T01:42:29.704 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:42:29 smithi189 ceph-mon[36187]: pgmap v3132: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 5 op/s 2024-03-24T01:42:29.711 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:42:29 smithi193 ceph-mon[36878]: pgmap v3132: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 5 op/s 2024-03-24T01:42:31.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:42:31 smithi118 ceph-mon[31535]: pgmap v3133: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T01:42:31.697 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:42:31 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 4646.680947 secs 2024-03-24T01:42:31.704 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:42:31 smithi189 ceph-mon[36187]: pgmap v3133: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T01:42:31.704 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:42:31 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 4646.680947 secs 2024-03-24T01:42:31.711 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:42:31 smithi193 ceph-mon[36878]: pgmap v3133: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T01:42:31.711 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:42:31 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 4646.680947 secs 2024-03-24T01:42:33.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:42:33 smithi118 ceph-mon[31535]: pgmap v3134: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:42:33.704 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:42:33 smithi189 ceph-mon[36187]: pgmap v3134: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:42:33.711 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:42:33 smithi193 ceph-mon[36878]: pgmap v3134: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:42:35.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:42:35 smithi118 ceph-mon[31535]: pgmap v3135: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T01:42:35.704 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:42:35 smithi189 ceph-mon[36187]: pgmap v3135: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T01:42:35.711 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:42:35 smithi193 ceph-mon[36878]: pgmap v3135: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T01:42:36.605 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:42:36 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 4651.681005 secs 2024-03-24T01:42:36.612 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:42:36 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 4651.681005 secs 2024-03-24T01:42:36.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:42:36 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 4651.681005 secs 2024-03-24T01:42:37.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:42:37 smithi118 ceph-mon[31535]: pgmap v3136: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T01:42:37.704 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:42:37 smithi189 ceph-mon[36187]: pgmap v3136: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T01:42:37.711 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:42:37 smithi193 ceph-mon[36878]: pgmap v3136: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T01:42:39.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:42:39 smithi118 ceph-mon[31535]: pgmap v3137: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 5 op/s 2024-03-24T01:42:39.704 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:42:39 smithi189 ceph-mon[36187]: pgmap v3137: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 5 op/s 2024-03-24T01:42:39.711 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:42:39 smithi193 ceph-mon[36878]: pgmap v3137: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 5 op/s 2024-03-24T01:42:41.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:42:41 smithi118 ceph-mon[31535]: pgmap v3138: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T01:42:41.697 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:42:41 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 4656.681091 secs 2024-03-24T01:42:41.704 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:42:41 smithi189 ceph-mon[36187]: pgmap v3138: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T01:42:41.704 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:42:41 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 4656.681091 secs 2024-03-24T01:42:41.711 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:42:41 smithi193 ceph-mon[36878]: pgmap v3138: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T01:42:41.711 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:42:41 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 4656.681091 secs 2024-03-24T01:42:43.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:42:43 smithi118 ceph-mon[31535]: pgmap v3139: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:42:43.704 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:42:43 smithi189 ceph-mon[36187]: pgmap v3139: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:42:43.711 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:42:43 smithi193 ceph-mon[36878]: pgmap v3139: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:42:45.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:42:45 smithi118 ceph-mon[31535]: pgmap v3140: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T01:42:45.704 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:42:45 smithi189 ceph-mon[36187]: pgmap v3140: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T01:42:45.711 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:42:45 smithi193 ceph-mon[36878]: pgmap v3140: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T01:42:46.697 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:42:46 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 4661.681147 secs 2024-03-24T01:42:46.705 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:42:46 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 4661.681147 secs 2024-03-24T01:42:46.711 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:42:46 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 4661.681147 secs 2024-03-24T01:42:47.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:42:47 smithi118 ceph-mon[31535]: pgmap v3141: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T01:42:47.704 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:42:47 smithi189 ceph-mon[36187]: pgmap v3141: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T01:42:47.712 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:42:47 smithi193 ceph-mon[36878]: pgmap v3141: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T01:42:49.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:42:49 smithi118 ceph-mon[31535]: pgmap v3142: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 5 op/s 2024-03-24T01:42:49.704 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:42:49 smithi189 ceph-mon[36187]: pgmap v3142: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 5 op/s 2024-03-24T01:42:49.711 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:42:49 smithi193 ceph-mon[36878]: pgmap v3142: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 5 op/s 2024-03-24T01:42:51.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:42:51 smithi118 ceph-mon[31535]: pgmap v3143: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T01:42:51.697 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:42:51 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 4666.681209 secs 2024-03-24T01:42:51.704 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:42:51 smithi189 ceph-mon[36187]: pgmap v3143: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T01:42:51.704 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:42:51 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 4666.681209 secs 2024-03-24T01:42:51.711 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:42:51 smithi193 ceph-mon[36878]: pgmap v3143: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T01:42:51.711 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:42:51 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 4666.681209 secs 2024-03-24T01:42:53.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:42:53 smithi118 ceph-mon[31535]: pgmap v3144: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:42:53.704 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:42:53 smithi189 ceph-mon[36187]: pgmap v3144: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:42:53.711 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:42:53 smithi193 ceph-mon[36878]: pgmap v3144: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:42:55.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:42:55 smithi118 ceph-mon[31535]: pgmap v3145: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T01:42:55.704 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:42:55 smithi189 ceph-mon[36187]: pgmap v3145: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T01:42:55.711 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:42:55 smithi193 ceph-mon[36878]: pgmap v3145: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T01:42:56.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:42:56 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 4671.681269 secs 2024-03-24T01:42:56.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:42:56 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T01:42:56.704 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:42:56 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 4671.681269 secs 2024-03-24T01:42:56.704 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:42:56 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T01:42:56.711 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:42:56 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 4671.681269 secs 2024-03-24T01:42:56.711 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:42:56 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T01:42:57.590 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:42:57 smithi189 ceph-mon[36187]: pgmap v3146: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T01:42:57.646 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:42:57 smithi193 ceph-mon[36878]: pgmap v3146: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T01:42:57.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:42:57 smithi118 ceph-mon[31535]: pgmap v3146: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T01:42:59.455 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:42:59 smithi189 ceph-mon[36187]: pgmap v3147: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 5 op/s 2024-03-24T01:42:59.496 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:42:59 smithi193 ceph-mon[36878]: pgmap v3147: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 5 op/s 2024-03-24T01:42:59.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:42:59 smithi118 ceph-mon[31535]: pgmap v3147: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 5 op/s 2024-03-24T01:43:01.697 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:43:01 smithi118 ceph-mon[31535]: pgmap v3148: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T01:43:01.697 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:43:01 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 4676.681335 secs 2024-03-24T01:43:01.697 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:43:01 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T01:43:01.697 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:43:01 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T01:43:01.697 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:43:01 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T01:43:01.697 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:43:01 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T01:43:01.697 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:43:01 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T01:43:01.697 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:43:01 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T01:43:01.697 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:43:01 smithi118 ceph-mon[31535]: pgmap v3149: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 5 op/s 2024-03-24T01:43:01.697 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:43:01 smithi118 ceph-mon[31535]: pgmap v3150: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 6 op/s 2024-03-24T01:43:01.698 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:43:01 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T01:43:01.705 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:43:01 smithi189 ceph-mon[36187]: pgmap v3148: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T01:43:01.705 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:43:01 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 4676.681335 secs 2024-03-24T01:43:01.705 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:43:01 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T01:43:01.705 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:43:01 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T01:43:01.705 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:43:01 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T01:43:01.706 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:43:01 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T01:43:01.706 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:43:01 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T01:43:01.706 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:43:01 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T01:43:01.706 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:43:01 smithi189 ceph-mon[36187]: pgmap v3149: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 5 op/s 2024-03-24T01:43:01.706 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:43:01 smithi189 ceph-mon[36187]: pgmap v3150: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 6 op/s 2024-03-24T01:43:01.706 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:43:01 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T01:43:01.711 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:43:01 smithi193 ceph-mon[36878]: pgmap v3148: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T01:43:01.711 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:43:01 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 4676.681335 secs 2024-03-24T01:43:01.711 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:43:01 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T01:43:01.711 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:43:01 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T01:43:01.712 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:43:01 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T01:43:01.712 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:43:01 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T01:43:01.712 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:43:01 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T01:43:01.712 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:43:01 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T01:43:01.712 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:43:01 smithi193 ceph-mon[36878]: pgmap v3149: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 5 op/s 2024-03-24T01:43:01.712 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:43:01 smithi193 ceph-mon[36878]: pgmap v3150: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 6 op/s 2024-03-24T01:43:01.712 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:43:01 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T01:43:04.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:43:03 smithi118 ceph-mon[31535]: pgmap v3151: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 5 op/s 2024-03-24T01:43:04.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:43:03 smithi189 ceph-mon[36187]: pgmap v3151: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 5 op/s 2024-03-24T01:43:04.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:43:03 smithi193 ceph-mon[36878]: pgmap v3151: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 5 op/s 2024-03-24T01:43:05.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:43:05 smithi118 ceph-mon[31535]: pgmap v3152: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 6 op/s 2024-03-24T01:43:05.704 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:43:05 smithi189 ceph-mon[36187]: pgmap v3152: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 6 op/s 2024-03-24T01:43:05.711 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:43:05 smithi193 ceph-mon[36878]: pgmap v3152: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 6 op/s 2024-03-24T01:43:06.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:43:06 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 4681.681393 secs 2024-03-24T01:43:06.704 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:43:06 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 4681.681393 secs 2024-03-24T01:43:06.711 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:43:06 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 4681.681393 secs 2024-03-24T01:43:07.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:43:07 smithi118 ceph-mon[31535]: pgmap v3153: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 5 op/s 2024-03-24T01:43:07.704 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:43:07 smithi189 ceph-mon[36187]: pgmap v3153: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 5 op/s 2024-03-24T01:43:07.711 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:43:07 smithi193 ceph-mon[36878]: pgmap v3153: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 5 op/s 2024-03-24T01:43:10.105 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:43:09 smithi118 ceph-mon[31535]: pgmap v3154: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 5 op/s 2024-03-24T01:43:10.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:43:09 smithi189 ceph-mon[36187]: pgmap v3154: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 5 op/s 2024-03-24T01:43:10.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:43:09 smithi193 ceph-mon[36878]: pgmap v3154: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 5 op/s 2024-03-24T01:43:11.133 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:43:10 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 4686.681455 secs 2024-03-24T01:43:11.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:43:10 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 4686.681455 secs 2024-03-24T01:43:11.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:43:10 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 4686.681455 secs 2024-03-24T01:43:12.197 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:43:11 smithi118 ceph-mon[31535]: pgmap v3155: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 5 op/s 2024-03-24T01:43:12.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:43:11 smithi189 ceph-mon[36187]: pgmap v3155: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 5 op/s 2024-03-24T01:43:12.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:43:11 smithi193 ceph-mon[36878]: pgmap v3155: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 5 op/s 2024-03-24T01:43:14.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:43:13 smithi118 ceph-mon[31535]: pgmap v3156: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T01:43:14.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:43:13 smithi189 ceph-mon[36187]: pgmap v3156: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T01:43:14.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:43:13 smithi193 ceph-mon[36878]: pgmap v3156: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T01:43:16.197 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:43:15 smithi118 ceph-mon[31535]: pgmap v3157: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-03-24T01:43:16.199 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:43:15 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 4691.681517 secs 2024-03-24T01:43:16.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:43:15 smithi189 ceph-mon[36187]: pgmap v3157: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-03-24T01:43:16.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:43:15 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 4691.681517 secs 2024-03-24T01:43:16.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:43:15 smithi193 ceph-mon[36878]: pgmap v3157: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-03-24T01:43:16.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:43:15 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 4691.681517 secs 2024-03-24T01:43:16.484 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T01:43:16.484+0000 7f81ee000640 1 -- 172.21.15.118:0/1693767697 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f81c80036c0 con 0x7f81f81052d0 2024-03-24T01:43:16.485 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T01:43:16.484+0000 7f81db400640 1 -- 172.21.15.118:0/659686424 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f81c40064d0 con 0x5622651a13f0 2024-03-24T01:43:16.485 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T01:43:16.485+0000 7f81ef400640 1 -- 172.21.15.118:0/1693767697 <== mon.2 v2:172.21.15.189:3300/0 40 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (secure 0 0 0) 0x7f81c80036c0 con 0x7f81f81052d0 2024-03-24T01:43:16.485 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T01:43:16.485+0000 7f81e4c00640 1 -- 172.21.15.118:0/659686424 <== mon.1 v2:172.21.15.193:3300/0 40 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 409+0+0 (secure 0 0 0) 0x7f81c40064d0 con 0x5622651a13f0 2024-03-24T01:43:18.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:43:17 smithi118 ceph-mon[31535]: pgmap v3158: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:43:18.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:43:17 smithi189 ceph-mon[36187]: pgmap v3158: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:43:18.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:43:17 smithi193 ceph-mon[36878]: pgmap v3158: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:43:20.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:43:19 smithi118 ceph-mon[31535]: pgmap v3159: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T01:43:20.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:43:19 smithi189 ceph-mon[36187]: pgmap v3159: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T01:43:20.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:43:19 smithi193 ceph-mon[36878]: pgmap v3159: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T01:43:21.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:43:20 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 4696.681597 secs 2024-03-24T01:43:21.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:43:20 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 4696.681597 secs 2024-03-24T01:43:21.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:43:20 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 4696.681597 secs 2024-03-24T01:43:22.174 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:43:21 smithi118 ceph-mon[31535]: pgmap v3160: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T01:43:22.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:43:21 smithi189 ceph-mon[36187]: pgmap v3160: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T01:43:22.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:43:21 smithi193 ceph-mon[36878]: pgmap v3160: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T01:43:23.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:43:22 smithi118 ceph-mon[31535]: pgmap v3161: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T01:43:23.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:43:22 smithi189 ceph-mon[36187]: pgmap v3161: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T01:43:23.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:43:22 smithi193 ceph-mon[36878]: pgmap v3161: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T01:43:26.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:43:25 smithi118 ceph-mon[31535]: pgmap v3162: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-03-24T01:43:26.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:43:25 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 4701.681639 secs 2024-03-24T01:43:26.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:43:25 smithi189 ceph-mon[36187]: pgmap v3162: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-03-24T01:43:26.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:43:25 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 4701.681639 secs 2024-03-24T01:43:26.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:43:25 smithi193 ceph-mon[36878]: pgmap v3162: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-03-24T01:43:26.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:43:25 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 4701.681639 secs 2024-03-24T01:43:28.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:43:27 smithi118 ceph-mon[31535]: pgmap v3163: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:43:28.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:43:27 smithi189 ceph-mon[36187]: pgmap v3163: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:43:28.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:43:27 smithi193 ceph-mon[36878]: pgmap v3163: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:43:30.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:43:29 smithi118 ceph-mon[31535]: pgmap v3164: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T01:43:30.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:43:29 smithi189 ceph-mon[36187]: pgmap v3164: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T01:43:30.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:43:29 smithi193 ceph-mon[36878]: pgmap v3164: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T01:43:31.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:43:30 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 4706.681701 secs 2024-03-24T01:43:31.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:43:30 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 4706.681701 secs 2024-03-24T01:43:31.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:43:30 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 4706.681701 secs 2024-03-24T01:43:32.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:43:31 smithi118 ceph-mon[31535]: pgmap v3165: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T01:43:32.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:43:31 smithi189 ceph-mon[36187]: pgmap v3165: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T01:43:32.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:43:31 smithi193 ceph-mon[36878]: pgmap v3165: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T01:43:34.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:43:33 smithi118 ceph-mon[31535]: pgmap v3166: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T01:43:34.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:43:33 smithi189 ceph-mon[36187]: pgmap v3166: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T01:43:34.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:43:33 smithi193 ceph-mon[36878]: pgmap v3166: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T01:43:35.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:43:34 smithi118 ceph-mon[31535]: pgmap v3167: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-03-24T01:43:35.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:43:34 smithi189 ceph-mon[36187]: pgmap v3167: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-03-24T01:43:35.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:43:34 smithi193 ceph-mon[36878]: pgmap v3167: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-03-24T01:43:36.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:43:35 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 4711.681791 secs 2024-03-24T01:43:36.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:43:35 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 4711.681791 secs 2024-03-24T01:43:36.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:43:35 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 4711.681791 secs 2024-03-24T01:43:37.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:43:36 smithi118 ceph-mon[31535]: pgmap v3168: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:43:37.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:43:36 smithi189 ceph-mon[36187]: pgmap v3168: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:43:37.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:43:36 smithi193 ceph-mon[36878]: pgmap v3168: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:43:40.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:43:39 smithi118 ceph-mon[31535]: pgmap v3169: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T01:43:40.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:43:39 smithi189 ceph-mon[36187]: pgmap v3169: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T01:43:40.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:43:39 smithi193 ceph-mon[36878]: pgmap v3169: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T01:43:41.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:43:40 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 4716.681875 secs 2024-03-24T01:43:41.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:43:40 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 4716.681875 secs 2024-03-24T01:43:41.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:43:40 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 4716.681875 secs 2024-03-24T01:43:42.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:43:41 smithi118 ceph-mon[31535]: pgmap v3170: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T01:43:42.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:43:41 smithi189 ceph-mon[36187]: pgmap v3170: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T01:43:42.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:43:41 smithi193 ceph-mon[36878]: pgmap v3170: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T01:43:44.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:43:43 smithi118 ceph-mon[31535]: pgmap v3171: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T01:43:44.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:43:43 smithi189 ceph-mon[36187]: pgmap v3171: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T01:43:44.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:43:43 smithi193 ceph-mon[36878]: pgmap v3171: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T01:43:46.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:43:45 smithi118 ceph-mon[31535]: pgmap v3172: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-03-24T01:43:46.197 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:43:45 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 4721.681961 secs 2024-03-24T01:43:46.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:43:45 smithi189 ceph-mon[36187]: pgmap v3172: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-03-24T01:43:46.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:43:45 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 4721.681961 secs 2024-03-24T01:43:46.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:43:45 smithi193 ceph-mon[36878]: pgmap v3172: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-03-24T01:43:46.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:43:45 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 4721.681961 secs 2024-03-24T01:43:47.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:43:46 smithi118 ceph-mon[31535]: pgmap v3173: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:43:47.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:43:46 smithi189 ceph-mon[36187]: pgmap v3173: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:43:47.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:43:46 smithi193 ceph-mon[36878]: pgmap v3173: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:43:50.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:43:49 smithi118 ceph-mon[31535]: pgmap v3174: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T01:43:50.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:43:49 smithi189 ceph-mon[36187]: pgmap v3174: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T01:43:50.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:43:49 smithi193 ceph-mon[36878]: pgmap v3174: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T01:43:51.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:43:50 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 4726.682050 secs 2024-03-24T01:43:51.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:43:50 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 4726.682050 secs 2024-03-24T01:43:51.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:43:50 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 4726.682050 secs 2024-03-24T01:43:52.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:43:51 smithi118 ceph-mon[31535]: pgmap v3175: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T01:43:52.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:43:51 smithi189 ceph-mon[36187]: pgmap v3175: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T01:43:52.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:43:51 smithi193 ceph-mon[36878]: pgmap v3175: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T01:43:54.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:43:53 smithi118 ceph-mon[31535]: pgmap v3176: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T01:43:54.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:43:53 smithi189 ceph-mon[36187]: pgmap v3176: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T01:43:54.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:43:53 smithi193 ceph-mon[36878]: pgmap v3176: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T01:43:55.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:43:54 smithi118 ceph-mon[31535]: pgmap v3177: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-03-24T01:43:55.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:43:54 smithi189 ceph-mon[36187]: pgmap v3177: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-03-24T01:43:55.212 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:43:54 smithi193 ceph-mon[36878]: pgmap v3177: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-03-24T01:43:56.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:43:55 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 4731.682113 secs 2024-03-24T01:43:56.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:43:55 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 4731.682113 secs 2024-03-24T01:43:56.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:43:55 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 4731.682113 secs 2024-03-24T01:43:57.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:43:56 smithi118 ceph-mon[31535]: pgmap v3178: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:43:57.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:43:56 smithi189 ceph-mon[36187]: pgmap v3178: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:43:57.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:43:56 smithi193 ceph-mon[36878]: pgmap v3178: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:44:00.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:43:59 smithi118 ceph-mon[31535]: pgmap v3179: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T01:44:00.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:43:59 smithi189 ceph-mon[36187]: pgmap v3179: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T01:44:00.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:43:59 smithi193 ceph-mon[36878]: pgmap v3179: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T01:44:01.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:44:00 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 4736.682197 secs 2024-03-24T01:44:01.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:44:00 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 4736.682197 secs 2024-03-24T01:44:01.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:44:00 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 4736.682197 secs 2024-03-24T01:44:02.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:44:01 smithi118 ceph-mon[31535]: pgmap v3180: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T01:44:02.197 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:44:01 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T01:44:02.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:44:01 smithi189 ceph-mon[36187]: pgmap v3180: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T01:44:02.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:44:01 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T01:44:02.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:44:01 smithi193 ceph-mon[36878]: pgmap v3180: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T01:44:02.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:44:01 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T01:44:03.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:44:02 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T01:44:03.197 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:44:02 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T01:44:03.197 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:44:02 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T01:44:03.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:44:02 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T01:44:03.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:44:02 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T01:44:03.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:44:02 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T01:44:03.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:44:02 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T01:44:03.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:44:02 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T01:44:03.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:44:02 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T01:44:04.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:44:03 smithi118 ceph-mon[31535]: pgmap v3181: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 5 op/s 2024-03-24T01:44:04.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:44:03 smithi118 ceph-mon[31535]: pgmap v3182: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 5 op/s 2024-03-24T01:44:04.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:44:03 smithi189 ceph-mon[36187]: pgmap v3181: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 5 op/s 2024-03-24T01:44:04.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:44:03 smithi189 ceph-mon[36187]: pgmap v3182: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 5 op/s 2024-03-24T01:44:04.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:44:03 smithi193 ceph-mon[36878]: pgmap v3181: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 5 op/s 2024-03-24T01:44:04.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:44:03 smithi193 ceph-mon[36878]: pgmap v3182: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 5 op/s 2024-03-24T01:44:05.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:44:05 smithi118 ceph-mon[31535]: pgmap v3183: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-03-24T01:44:05.704 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:44:05 smithi189 ceph-mon[36187]: pgmap v3183: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-03-24T01:44:05.711 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:44:05 smithi193 ceph-mon[36878]: pgmap v3183: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-03-24T01:44:06.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:44:06 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 4741.682286 secs 2024-03-24T01:44:06.704 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:44:06 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 4741.682286 secs 2024-03-24T01:44:06.711 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:44:06 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 4741.682286 secs 2024-03-24T01:44:07.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:44:07 smithi118 ceph-mon[31535]: pgmap v3184: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-03-24T01:44:07.704 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:44:07 smithi189 ceph-mon[36187]: pgmap v3184: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-03-24T01:44:07.711 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:44:07 smithi193 ceph-mon[36878]: pgmap v3184: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-03-24T01:44:09.702 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:44:09 smithi118 ceph-mon[31535]: pgmap v3185: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-03-24T01:44:09.705 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:44:09 smithi189 ceph-mon[36187]: pgmap v3185: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-03-24T01:44:09.719 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:44:09 smithi193 ceph-mon[36878]: pgmap v3185: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-03-24T01:44:11.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:44:11 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 4746.682367 secs 2024-03-24T01:44:11.697 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:44:11 smithi118 ceph-mon[31535]: pgmap v3186: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-03-24T01:44:11.704 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:44:11 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 4746.682367 secs 2024-03-24T01:44:11.704 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:44:11 smithi189 ceph-mon[36187]: pgmap v3186: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-03-24T01:44:11.711 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:44:11 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 4746.682367 secs 2024-03-24T01:44:11.711 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:44:11 smithi193 ceph-mon[36878]: pgmap v3186: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-03-24T01:44:13.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:44:13 smithi118 ceph-mon[31535]: pgmap v3187: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 5 op/s 2024-03-24T01:44:13.704 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:44:13 smithi189 ceph-mon[36187]: pgmap v3187: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 5 op/s 2024-03-24T01:44:13.711 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:44:13 smithi193 ceph-mon[36878]: pgmap v3187: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 5 op/s 2024-03-24T01:44:15.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:44:15 smithi118 ceph-mon[31535]: pgmap v3188: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-24T01:44:15.704 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:44:15 smithi189 ceph-mon[36187]: pgmap v3188: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-24T01:44:15.711 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:44:15 smithi193 ceph-mon[36878]: pgmap v3188: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-24T01:44:16.704 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:44:16 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 4751.682427 secs 2024-03-24T01:44:16.711 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:44:16 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 4751.682427 secs 2024-03-24T01:44:16.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:44:16 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 4751.682427 secs 2024-03-24T01:44:17.704 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:44:17 smithi189 ceph-mon[36187]: pgmap v3189: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:44:17.711 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:44:17 smithi193 ceph-mon[36878]: pgmap v3189: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:44:17.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:44:17 smithi118 ceph-mon[31535]: pgmap v3189: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:44:19.711 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:44:19 smithi193 ceph-mon[36878]: pgmap v3190: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T01:44:19.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:44:19 smithi118 ceph-mon[31535]: pgmap v3190: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T01:44:19.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:44:19 smithi189 ceph-mon[36187]: pgmap v3190: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T01:44:21.711 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:44:21 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 4756.682483 secs 2024-03-24T01:44:21.711 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:44:21 smithi193 ceph-mon[36878]: pgmap v3191: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T01:44:21.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:44:21 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 4756.682483 secs 2024-03-24T01:44:21.947 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:44:21 smithi118 ceph-mon[31535]: pgmap v3191: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T01:44:21.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:44:21 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 4756.682483 secs 2024-03-24T01:44:21.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:44:21 smithi189 ceph-mon[36187]: pgmap v3191: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T01:44:23.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:44:23 smithi118 ceph-mon[31535]: pgmap v3192: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:44:23.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:44:23 smithi189 ceph-mon[36187]: pgmap v3192: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:44:23.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:44:23 smithi193 ceph-mon[36878]: pgmap v3192: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:44:25.749 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:44:25 smithi118 ceph-mon[31535]: pgmap v3193: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-24T01:44:25.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:44:25 smithi189 ceph-mon[36187]: pgmap v3193: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-24T01:44:25.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:44:25 smithi193 ceph-mon[36878]: pgmap v3193: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-24T01:44:26.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:44:26 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 4761.682542 secs 2024-03-24T01:44:26.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:44:26 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 4761.682542 secs 2024-03-24T01:44:26.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:44:26 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 4761.682542 secs 2024-03-24T01:44:27.929 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:44:27 smithi118 ceph-mon[31535]: pgmap v3194: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:44:27.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:44:27 smithi189 ceph-mon[36187]: pgmap v3194: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:44:27.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:44:27 smithi193 ceph-mon[36878]: pgmap v3194: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:44:29.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:44:29 smithi118 ceph-mon[31535]: pgmap v3195: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T01:44:29.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:44:29 smithi189 ceph-mon[36187]: pgmap v3195: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T01:44:29.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:44:29 smithi193 ceph-mon[36878]: pgmap v3195: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T01:44:31.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:44:31 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 4766.682631 secs 2024-03-24T01:44:31.947 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:44:31 smithi118 ceph-mon[31535]: pgmap v3196: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T01:44:31.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:44:31 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 4766.682631 secs 2024-03-24T01:44:31.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:44:31 smithi189 ceph-mon[36187]: pgmap v3196: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T01:44:31.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:44:31 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 4766.682631 secs 2024-03-24T01:44:31.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:44:31 smithi193 ceph-mon[36878]: pgmap v3196: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T01:44:33.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:44:33 smithi118 ceph-mon[31535]: pgmap v3197: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:44:33.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:44:33 smithi189 ceph-mon[36187]: pgmap v3197: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:44:33.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:44:33 smithi193 ceph-mon[36878]: pgmap v3197: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:44:35.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:44:35 smithi118 ceph-mon[31535]: pgmap v3198: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-24T01:44:35.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:44:35 smithi189 ceph-mon[36187]: pgmap v3198: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-24T01:44:35.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:44:35 smithi193 ceph-mon[36878]: pgmap v3198: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-24T01:44:36.855 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:44:36 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 4771.682672 secs 2024-03-24T01:44:36.862 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:44:36 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 4771.682672 secs 2024-03-24T01:44:36.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:44:36 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 4771.682672 secs 2024-03-24T01:44:37.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:44:37 smithi118 ceph-mon[31535]: pgmap v3199: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:44:37.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:44:37 smithi189 ceph-mon[36187]: pgmap v3199: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:44:37.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:44:37 smithi193 ceph-mon[36878]: pgmap v3199: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:44:39.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:44:39 smithi118 ceph-mon[31535]: pgmap v3200: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T01:44:39.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:44:39 smithi189 ceph-mon[36187]: pgmap v3200: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T01:44:39.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:44:39 smithi193 ceph-mon[36878]: pgmap v3200: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T01:44:41.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:44:41 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 4776.682745 secs 2024-03-24T01:44:41.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:44:41 smithi118 ceph-mon[31535]: pgmap v3201: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T01:44:41.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:44:41 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 4776.682745 secs 2024-03-24T01:44:41.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:44:41 smithi189 ceph-mon[36187]: pgmap v3201: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T01:44:41.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:44:41 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 4776.682745 secs 2024-03-24T01:44:41.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:44:41 smithi193 ceph-mon[36878]: pgmap v3201: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T01:44:43.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:44:43 smithi118 ceph-mon[31535]: pgmap v3202: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:44:43.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:44:43 smithi189 ceph-mon[36187]: pgmap v3202: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:44:43.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:44:43 smithi193 ceph-mon[36878]: pgmap v3202: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:44:45.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:44:45 smithi118 ceph-mon[31535]: pgmap v3203: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-24T01:44:45.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:44:45 smithi189 ceph-mon[36187]: pgmap v3203: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-24T01:44:45.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:44:45 smithi193 ceph-mon[36878]: pgmap v3203: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-24T01:44:46.872 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:44:46 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 4781.682830 secs 2024-03-24T01:44:46.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:44:46 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 4781.682830 secs 2024-03-24T01:44:46.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:44:46 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 4781.682830 secs 2024-03-24T01:44:47.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:44:47 smithi118 ceph-mon[31535]: pgmap v3204: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:44:47.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:44:47 smithi189 ceph-mon[36187]: pgmap v3204: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:44:47.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:44:47 smithi193 ceph-mon[36878]: pgmap v3204: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:44:49.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:44:49 smithi118 ceph-mon[31535]: pgmap v3205: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T01:44:49.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:44:49 smithi189 ceph-mon[36187]: pgmap v3205: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T01:44:49.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:44:49 smithi193 ceph-mon[36878]: pgmap v3205: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T01:44:51.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:44:51 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 4786.682918 secs 2024-03-24T01:44:51.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:44:51 smithi118 ceph-mon[31535]: pgmap v3206: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T01:44:51.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:44:51 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 4786.682918 secs 2024-03-24T01:44:51.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:44:51 smithi189 ceph-mon[36187]: pgmap v3206: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T01:44:51.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:44:51 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 4786.682918 secs 2024-03-24T01:44:51.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:44:51 smithi193 ceph-mon[36878]: pgmap v3206: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T01:44:53.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:44:53 smithi118 ceph-mon[31535]: pgmap v3207: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:44:53.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:44:53 smithi189 ceph-mon[36187]: pgmap v3207: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:44:53.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:44:53 smithi193 ceph-mon[36878]: pgmap v3207: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:44:55.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:44:55 smithi118 ceph-mon[31535]: pgmap v3208: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-24T01:44:55.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:44:55 smithi189 ceph-mon[36187]: pgmap v3208: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-24T01:44:55.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:44:55 smithi193 ceph-mon[36878]: pgmap v3208: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-24T01:44:56.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:44:56 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 4791.683001 secs 2024-03-24T01:44:56.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:44:56 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 4791.683001 secs 2024-03-24T01:44:56.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:44:56 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 4791.683001 secs 2024-03-24T01:44:57.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:44:57 smithi118 ceph-mon[31535]: pgmap v3209: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:44:57.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:44:57 smithi189 ceph-mon[36187]: pgmap v3209: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:44:57.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:44:57 smithi193 ceph-mon[36878]: pgmap v3209: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:44:59.947 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:44:59 smithi118 ceph-mon[31535]: pgmap v3210: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T01:44:59.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:44:59 smithi189 ceph-mon[36187]: pgmap v3210: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T01:44:59.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:44:59 smithi193 ceph-mon[36878]: pgmap v3210: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T01:45:01.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:45:01 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 4796.683086 secs 2024-03-24T01:45:01.947 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:45:01 smithi118 ceph-mon[31535]: pgmap v3211: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T01:45:01.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:45:01 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 4796.683086 secs 2024-03-24T01:45:01.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:45:01 smithi189 ceph-mon[36187]: pgmap v3211: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T01:45:01.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:45:01 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 4796.683086 secs 2024-03-24T01:45:01.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:45:01 smithi193 ceph-mon[36878]: pgmap v3211: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T01:45:02.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:45:02 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T01:45:02.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:45:02 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T01:45:02.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:45:02 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T01:45:03.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:45:03 smithi118 ceph-mon[31535]: pgmap v3212: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:45:03.947 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:45:03 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-24T01:45:03.947 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:45:03 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-24T01:45:03.947 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:45:03 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-24T01:45:03.947 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:45:03 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-03-24T01:45:03.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:45:03 smithi189 ceph-mon[36187]: pgmap v3212: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:45:03.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:45:03 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-24T01:45:03.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:45:03 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-24T01:45:03.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:45:03 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-24T01:45:03.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:45:03 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-03-24T01:45:03.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:45:03 smithi193 ceph-mon[36878]: pgmap v3212: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:45:03.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:45:03 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-24T01:45:03.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:45:03 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-24T01:45:03.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:45:03 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-24T01:45:03.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:45:03 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-03-24T01:45:04.947 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:45:04 smithi118 ceph-mon[31535]: Adjusting osd_memory_target on smithi189 to 174.1M 2024-03-24T01:45:04.947 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:45:04 smithi118 ceph-mon[31535]: Unable to set osd_memory_target on smithi189 to 182635622: error parsing value: Value '182635622' is below minimum 939524096 2024-03-24T01:45:04.947 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:45:04 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-24T01:45:04.947 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:45:04 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-24T01:45:04.947 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:45:04 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-24T01:45:04.947 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:45:04 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-03-24T01:45:04.947 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:45:04 smithi118 ceph-mon[31535]: Adjusting osd_memory_target on smithi118 to 176.7M 2024-03-24T01:45:04.947 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:45:04 smithi118 ceph-mon[31535]: Unable to set osd_memory_target on smithi118 to 185365196: error parsing value: Value '185365196' is below minimum 939524096 2024-03-24T01:45:04.948 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:45:04 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T01:45:04.948 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:45:04 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T01:45:04.948 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:45:04 smithi118 ceph-mon[31535]: pgmap v3213: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 6 op/s 2024-03-24T01:45:04.948 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:45:04 smithi118 ceph-mon[31535]: pgmap v3214: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 4 op/s 2024-03-24T01:45:04.948 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:45:04 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T01:45:04.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:45:04 smithi189 ceph-mon[36187]: Adjusting osd_memory_target on smithi189 to 174.1M 2024-03-24T01:45:04.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:45:04 smithi189 ceph-mon[36187]: Unable to set osd_memory_target on smithi189 to 182635622: error parsing value: Value '182635622' is below minimum 939524096 2024-03-24T01:45:04.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:45:04 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-24T01:45:04.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:45:04 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-24T01:45:04.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:45:04 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-24T01:45:04.955 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:45:04 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-03-24T01:45:04.955 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:45:04 smithi189 ceph-mon[36187]: Adjusting osd_memory_target on smithi118 to 176.7M 2024-03-24T01:45:04.955 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:45:04 smithi189 ceph-mon[36187]: Unable to set osd_memory_target on smithi118 to 185365196: error parsing value: Value '185365196' is below minimum 939524096 2024-03-24T01:45:04.955 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:45:04 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T01:45:04.955 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:45:04 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T01:45:04.955 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:45:04 smithi189 ceph-mon[36187]: pgmap v3213: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 6 op/s 2024-03-24T01:45:04.955 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:45:04 smithi189 ceph-mon[36187]: pgmap v3214: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 4 op/s 2024-03-24T01:45:04.955 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:45:04 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T01:45:04.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:45:04 smithi193 ceph-mon[36878]: Adjusting osd_memory_target on smithi189 to 174.1M 2024-03-24T01:45:04.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:45:04 smithi193 ceph-mon[36878]: Unable to set osd_memory_target on smithi189 to 182635622: error parsing value: Value '182635622' is below minimum 939524096 2024-03-24T01:45:04.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:45:04 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-24T01:45:04.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:45:04 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-24T01:45:04.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:45:04 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-24T01:45:04.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:45:04 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-03-24T01:45:04.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:45:04 smithi193 ceph-mon[36878]: Adjusting osd_memory_target on smithi118 to 176.7M 2024-03-24T01:45:04.962 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:45:04 smithi193 ceph-mon[36878]: Unable to set osd_memory_target on smithi118 to 185365196: error parsing value: Value '185365196' is below minimum 939524096 2024-03-24T01:45:04.962 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:45:04 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T01:45:04.962 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:45:04 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T01:45:04.962 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:45:04 smithi193 ceph-mon[36878]: pgmap v3213: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 6 op/s 2024-03-24T01:45:04.962 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:45:04 smithi193 ceph-mon[36878]: pgmap v3214: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 4 op/s 2024-03-24T01:45:04.962 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:45:04 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T01:45:05.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:45:05 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 4801.683149 secs 2024-03-24T01:45:05.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:45:05 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 4801.683149 secs 2024-03-24T01:45:05.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:45:05 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 4801.683149 secs 2024-03-24T01:45:06.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:45:06 smithi193 ceph-mon[36878]: pgmap v3215: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-03-24T01:45:07.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:45:06 smithi118 ceph-mon[31535]: pgmap v3215: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-03-24T01:45:07.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:45:06 smithi189 ceph-mon[36187]: pgmap v3215: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-03-24T01:45:09.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:45:08 smithi118 ceph-mon[31535]: pgmap v3216: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 5 op/s 2024-03-24T01:45:09.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:45:08 smithi189 ceph-mon[36187]: pgmap v3216: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 5 op/s 2024-03-24T01:45:09.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:45:08 smithi193 ceph-mon[36878]: pgmap v3216: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 5 op/s 2024-03-24T01:45:11.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:45:10 smithi118 ceph-mon[31535]: pgmap v3217: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-03-24T01:45:11.197 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:45:10 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 4806.683237 secs 2024-03-24T01:45:11.205 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:45:10 smithi189 ceph-mon[36187]: pgmap v3217: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-03-24T01:45:11.205 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:45:10 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 4806.683237 secs 2024-03-24T01:45:11.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:45:10 smithi193 ceph-mon[36878]: pgmap v3217: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-03-24T01:45:11.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:45:10 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 4806.683237 secs 2024-03-24T01:45:12.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:45:11 smithi118 ceph-mon[31535]: pgmap v3218: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-03-24T01:45:12.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:45:11 smithi189 ceph-mon[36187]: pgmap v3218: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-03-24T01:45:12.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:45:11 smithi193 ceph-mon[36878]: pgmap v3218: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-03-24T01:45:15.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:45:14 smithi118 ceph-mon[31535]: pgmap v3219: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 5 op/s 2024-03-24T01:45:15.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:45:14 smithi189 ceph-mon[36187]: pgmap v3219: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 5 op/s 2024-03-24T01:45:15.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:45:14 smithi193 ceph-mon[36878]: pgmap v3219: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 5 op/s 2024-03-24T01:45:16.198 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:45:15 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 4811.683299 secs 2024-03-24T01:45:16.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:45:15 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 4811.683299 secs 2024-03-24T01:45:16.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:45:15 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 4811.683299 secs 2024-03-24T01:45:17.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:45:16 smithi118 ceph-mon[31535]: pgmap v3220: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T01:45:17.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:45:16 smithi189 ceph-mon[36187]: pgmap v3220: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T01:45:17.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:45:16 smithi193 ceph-mon[36878]: pgmap v3220: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T01:45:18.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:45:17 smithi118 ceph-mon[31535]: pgmap v3221: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T01:45:18.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:45:17 smithi189 ceph-mon[36187]: pgmap v3221: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T01:45:18.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:45:17 smithi193 ceph-mon[36878]: pgmap v3221: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T01:45:21.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:45:20 smithi118 ceph-mon[31535]: pgmap v3222: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-03-24T01:45:21.197 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:45:20 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 4816.683362 secs 2024-03-24T01:45:21.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:45:20 smithi189 ceph-mon[36187]: pgmap v3222: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-03-24T01:45:21.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:45:20 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 4816.683362 secs 2024-03-24T01:45:21.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:45:20 smithi193 ceph-mon[36878]: pgmap v3222: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-03-24T01:45:21.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:45:20 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 4816.683362 secs 2024-03-24T01:45:22.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:45:21 smithi189 ceph-mon[36187]: pgmap v3223: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:45:22.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:45:21 smithi193 ceph-mon[36878]: pgmap v3223: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:45:22.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:45:21 smithi118 ceph-mon[31535]: pgmap v3223: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:45:25.150 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:45:24 smithi118 ceph-mon[31535]: pgmap v3224: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T01:45:25.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:45:24 smithi189 ceph-mon[36187]: pgmap v3224: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T01:45:25.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:45:24 smithi193 ceph-mon[36878]: pgmap v3224: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T01:45:26.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:45:25 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 4821.683423 secs 2024-03-24T01:45:26.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:45:25 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 4821.683423 secs 2024-03-24T01:45:26.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:45:25 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 4821.683423 secs 2024-03-24T01:45:27.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:45:26 smithi118 ceph-mon[31535]: pgmap v3225: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T01:45:27.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:45:26 smithi189 ceph-mon[36187]: pgmap v3225: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T01:45:27.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:45:26 smithi193 ceph-mon[36878]: pgmap v3225: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T01:45:28.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:45:27 smithi118 ceph-mon[31535]: pgmap v3226: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T01:45:28.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:45:27 smithi189 ceph-mon[36187]: pgmap v3226: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T01:45:28.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:45:27 smithi193 ceph-mon[36878]: pgmap v3226: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T01:45:31.197 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:45:30 smithi118 ceph-mon[31535]: pgmap v3227: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-03-24T01:45:31.197 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:45:30 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 4826.683462 secs 2024-03-24T01:45:31.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:45:30 smithi189 ceph-mon[36187]: pgmap v3227: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-03-24T01:45:31.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:45:30 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 4826.683462 secs 2024-03-24T01:45:31.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:45:30 smithi193 ceph-mon[36878]: pgmap v3227: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-03-24T01:45:31.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:45:30 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 4826.683462 secs 2024-03-24T01:45:32.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:45:31 smithi118 ceph-mon[31535]: pgmap v3228: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:45:32.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:45:31 smithi189 ceph-mon[36187]: pgmap v3228: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:45:32.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:45:31 smithi193 ceph-mon[36878]: pgmap v3228: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:45:34.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:45:33 smithi189 ceph-mon[36187]: pgmap v3229: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T01:45:34.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:45:33 smithi193 ceph-mon[36878]: pgmap v3229: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T01:45:34.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:45:33 smithi118 ceph-mon[31535]: pgmap v3229: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T01:45:35.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:45:35 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 4831.683522 secs 2024-03-24T01:45:35.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:45:35 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 4831.683522 secs 2024-03-24T01:45:36.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:45:35 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 4831.683522 secs 2024-03-24T01:45:36.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:45:36 smithi193 ceph-mon[36878]: pgmap v3230: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T01:45:37.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:45:36 smithi118 ceph-mon[31535]: pgmap v3230: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T01:45:37.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:45:36 smithi189 ceph-mon[36187]: pgmap v3230: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T01:45:38.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:45:37 smithi189 ceph-mon[36187]: pgmap v3231: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 5 op/s 2024-03-24T01:45:38.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:45:37 smithi193 ceph-mon[36878]: pgmap v3231: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 5 op/s 2024-03-24T01:45:38.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:45:37 smithi118 ceph-mon[31535]: pgmap v3231: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 5 op/s 2024-03-24T01:45:40.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:45:39 smithi189 ceph-mon[36187]: pgmap v3232: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-03-24T01:45:40.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:45:39 smithi193 ceph-mon[36878]: pgmap v3232: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-03-24T01:45:40.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:45:39 smithi118 ceph-mon[31535]: pgmap v3232: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-03-24T01:45:41.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:45:40 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 4836.683564 secs 2024-03-24T01:45:41.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:45:40 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 4836.683564 secs 2024-03-24T01:45:41.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:45:40 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 4836.683564 secs 2024-03-24T01:45:42.447 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:45:41 smithi118 ceph-mon[31535]: pgmap v3233: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:45:42.455 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:45:41 smithi189 ceph-mon[36187]: pgmap v3233: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:45:42.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:45:41 smithi193 ceph-mon[36878]: pgmap v3233: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:45:44.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:45:43 smithi189 ceph-mon[36187]: pgmap v3234: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T01:45:44.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:45:43 smithi193 ceph-mon[36878]: pgmap v3234: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T01:45:44.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:45:43 smithi118 ceph-mon[31535]: pgmap v3234: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T01:45:45.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:45:45 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 4841.683622 secs 2024-03-24T01:45:45.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:45:45 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 4841.683622 secs 2024-03-24T01:45:46.197 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:45:45 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 4841.683622 secs 2024-03-24T01:45:46.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:45:46 smithi193 ceph-mon[36878]: pgmap v3235: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T01:45:47.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:45:46 smithi118 ceph-mon[31535]: pgmap v3235: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T01:45:47.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:45:46 smithi189 ceph-mon[36187]: pgmap v3235: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T01:45:48.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:45:47 smithi189 ceph-mon[36187]: pgmap v3236: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 5 op/s 2024-03-24T01:45:48.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:45:47 smithi193 ceph-mon[36878]: pgmap v3236: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 5 op/s 2024-03-24T01:45:48.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:45:47 smithi118 ceph-mon[31535]: pgmap v3236: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 5 op/s 2024-03-24T01:45:50.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:45:49 smithi189 ceph-mon[36187]: pgmap v3237: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T01:45:50.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:45:49 smithi193 ceph-mon[36878]: pgmap v3237: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T01:45:50.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:45:49 smithi118 ceph-mon[31535]: pgmap v3237: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T01:45:51.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:45:50 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 4846.683710 secs 2024-03-24T01:45:51.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:45:50 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 4846.683710 secs 2024-03-24T01:45:51.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:45:50 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 4846.683710 secs 2024-03-24T01:45:52.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:45:51 smithi118 ceph-mon[31535]: pgmap v3238: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:45:52.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:45:51 smithi189 ceph-mon[36187]: pgmap v3238: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:45:52.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:45:51 smithi193 ceph-mon[36878]: pgmap v3238: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:45:54.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:45:53 smithi189 ceph-mon[36187]: pgmap v3239: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T01:45:54.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:45:53 smithi193 ceph-mon[36878]: pgmap v3239: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T01:45:54.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:45:53 smithi118 ceph-mon[31535]: pgmap v3239: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T01:45:55.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:45:55 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 4851.683768 secs 2024-03-24T01:45:55.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:45:55 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 4851.683768 secs 2024-03-24T01:45:56.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:45:55 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 4851.683768 secs 2024-03-24T01:45:56.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:45:56 smithi193 ceph-mon[36878]: pgmap v3240: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T01:45:57.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:45:56 smithi118 ceph-mon[31535]: pgmap v3240: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T01:45:57.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:45:56 smithi189 ceph-mon[36187]: pgmap v3240: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T01:45:58.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:45:57 smithi189 ceph-mon[36187]: pgmap v3241: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 5 op/s 2024-03-24T01:45:58.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:45:57 smithi193 ceph-mon[36878]: pgmap v3241: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 5 op/s 2024-03-24T01:45:58.447 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:45:57 smithi118 ceph-mon[31535]: pgmap v3241: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 5 op/s 2024-03-24T01:46:00.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:45:59 smithi189 ceph-mon[36187]: pgmap v3242: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T01:46:00.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:45:59 smithi193 ceph-mon[36878]: pgmap v3242: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T01:46:00.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:45:59 smithi118 ceph-mon[31535]: pgmap v3242: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T01:46:01.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:46:00 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 4856.683832 secs 2024-03-24T01:46:01.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:46:00 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 4856.683832 secs 2024-03-24T01:46:01.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:46:00 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 4856.683832 secs 2024-03-24T01:46:02.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:46:01 smithi193 ceph-mon[36878]: pgmap v3243: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:46:02.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:46:01 smithi118 ceph-mon[31535]: pgmap v3243: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:46:02.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:46:01 smithi189 ceph-mon[36187]: pgmap v3243: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:46:04.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:46:03 smithi189 ceph-mon[36187]: pgmap v3244: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T01:46:04.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:46:03 smithi193 ceph-mon[36878]: pgmap v3244: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T01:46:04.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:46:03 smithi118 ceph-mon[31535]: pgmap v3244: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T01:46:05.209 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:46:04 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T01:46:05.209 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:46:04 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-24T01:46:05.209 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:46:04 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-24T01:46:05.209 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:46:04 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-03-24T01:46:05.209 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:46:04 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-03-24T01:46:05.209 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:46:04 smithi118 ceph-mon[31535]: Adjusting osd_memory_target on smithi193 to 174.1M 2024-03-24T01:46:05.209 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:46:04 smithi118 ceph-mon[31535]: Unable to set osd_memory_target on smithi193 to 182639923: error parsing value: Value '182639923' is below minimum 939524096 2024-03-24T01:46:05.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:46:04 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T01:46:05.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:46:04 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-24T01:46:05.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:46:04 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-24T01:46:05.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:46:04 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-03-24T01:46:05.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:46:04 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-03-24T01:46:05.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:46:04 smithi193 ceph-mon[36878]: Adjusting osd_memory_target on smithi193 to 174.1M 2024-03-24T01:46:05.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:46:04 smithi193 ceph-mon[36878]: Unable to set osd_memory_target on smithi193 to 182639923: error parsing value: Value '182639923' is below minimum 939524096 2024-03-24T01:46:05.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:46:04 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T01:46:05.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:46:04 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-24T01:46:05.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:46:04 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-24T01:46:05.455 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:46:04 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-03-24T01:46:05.455 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:46:04 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-03-24T01:46:05.455 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:46:04 smithi189 ceph-mon[36187]: Adjusting osd_memory_target on smithi193 to 174.1M 2024-03-24T01:46:05.455 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:46:04 smithi189 ceph-mon[36187]: Unable to set osd_memory_target on smithi193 to 182639923: error parsing value: Value '182639923' is below minimum 939524096 2024-03-24T01:46:06.355 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:46:05 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 4861.683895 secs 2024-03-24T01:46:06.355 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:46:05 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T01:46:06.355 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:46:05 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T01:46:06.355 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:46:05 smithi189 ceph-mon[36187]: pgmap v3245: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 6 op/s 2024-03-24T01:46:06.355 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:46:05 smithi189 ceph-mon[36187]: pgmap v3246: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-03-24T01:46:06.356 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:46:05 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T01:46:06.447 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:46:05 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 4861.683895 secs 2024-03-24T01:46:06.447 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:46:05 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T01:46:06.447 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:46:05 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T01:46:06.447 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:46:05 smithi118 ceph-mon[31535]: pgmap v3245: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 6 op/s 2024-03-24T01:46:06.447 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:46:05 smithi118 ceph-mon[31535]: pgmap v3246: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-03-24T01:46:06.447 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:46:05 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T01:46:06.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:46:05 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 4861.683895 secs 2024-03-24T01:46:06.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:46:05 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T01:46:06.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:46:05 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T01:46:06.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:46:05 smithi193 ceph-mon[36878]: pgmap v3245: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 6 op/s 2024-03-24T01:46:06.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:46:05 smithi193 ceph-mon[36878]: pgmap v3246: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-03-24T01:46:06.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:46:05 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T01:46:08.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:46:08 smithi118 ceph-mon[31535]: pgmap v3247: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 5 op/s 2024-03-24T01:46:08.704 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:46:08 smithi189 ceph-mon[36187]: pgmap v3247: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 5 op/s 2024-03-24T01:46:08.711 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:46:08 smithi193 ceph-mon[36878]: pgmap v3247: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 5 op/s 2024-03-24T01:46:10.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:46:10 smithi118 ceph-mon[31535]: pgmap v3248: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-03-24T01:46:10.704 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:46:10 smithi189 ceph-mon[36187]: pgmap v3248: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-03-24T01:46:10.711 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:46:10 smithi193 ceph-mon[36878]: pgmap v3248: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-03-24T01:46:11.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:46:11 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 4866.683982 secs 2024-03-24T01:46:11.704 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:46:11 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 4866.683982 secs 2024-03-24T01:46:11.711 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:46:11 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 4866.683982 secs 2024-03-24T01:46:12.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:46:12 smithi118 ceph-mon[31535]: pgmap v3249: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-03-24T01:46:12.704 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:46:12 smithi189 ceph-mon[36187]: pgmap v3249: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-03-24T01:46:12.711 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:46:12 smithi193 ceph-mon[36878]: pgmap v3249: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-03-24T01:46:14.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:46:14 smithi118 ceph-mon[31535]: pgmap v3250: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-03-24T01:46:14.704 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:46:14 smithi189 ceph-mon[36187]: pgmap v3250: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-03-24T01:46:14.711 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:46:14 smithi193 ceph-mon[36878]: pgmap v3250: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-03-24T01:46:16.513 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T01:46:16.513+0000 7f81ee000640 1 -- 172.21.15.118:0/1693767697 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f81c80036c0 con 0x7f81f81052d0 2024-03-24T01:46:16.513 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T01:46:16.513+0000 7f81db400640 1 -- 172.21.15.118:0/659686424 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f81c40064d0 con 0x5622651a13f0 2024-03-24T01:46:16.514 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T01:46:16.514+0000 7f81e4c00640 1 -- 172.21.15.118:0/659686424 <== mon.1 v2:172.21.15.193:3300/0 41 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 409+0+0 (secure 0 0 0) 0x7f81c40064d0 con 0x5622651a13f0 2024-03-24T01:46:16.514 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T01:46:16.514+0000 7f81ef400640 1 -- 172.21.15.118:0/1693767697 <== mon.2 v2:172.21.15.189:3300/0 41 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (secure 0 0 0) 0x7f81c80036c0 con 0x7f81f81052d0 2024-03-24T01:46:16.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:46:16 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 4871.684038 secs 2024-03-24T01:46:16.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:46:16 smithi118 ceph-mon[31535]: pgmap v3251: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 5 op/s 2024-03-24T01:46:16.704 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:46:16 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 4871.684038 secs 2024-03-24T01:46:16.704 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:46:16 smithi189 ceph-mon[36187]: pgmap v3251: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 5 op/s 2024-03-24T01:46:16.711 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:46:16 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 4871.684038 secs 2024-03-24T01:46:16.711 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:46:16 smithi193 ceph-mon[36878]: pgmap v3251: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 5 op/s 2024-03-24T01:46:18.704 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:46:18 smithi189 ceph-mon[36187]: pgmap v3252: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:46:18.711 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:46:18 smithi193 ceph-mon[36878]: pgmap v3252: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:46:18.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:46:18 smithi118 ceph-mon[31535]: pgmap v3252: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:46:20.704 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:46:20 smithi189 ceph-mon[36187]: pgmap v3253: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-24T01:46:20.711 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:46:20 smithi193 ceph-mon[36878]: pgmap v3253: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-24T01:46:20.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:46:20 smithi118 ceph-mon[31535]: pgmap v3253: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-24T01:46:21.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:46:21 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 4876.684092 secs 2024-03-24T01:46:21.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:46:21 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 4876.684092 secs 2024-03-24T01:46:21.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:46:21 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 4876.684092 secs 2024-03-24T01:46:22.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:46:22 smithi118 ceph-mon[31535]: pgmap v3254: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:46:22.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:46:22 smithi189 ceph-mon[36187]: pgmap v3254: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:46:22.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:46:22 smithi193 ceph-mon[36878]: pgmap v3254: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:46:24.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:46:24 smithi118 ceph-mon[31535]: pgmap v3255: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T01:46:24.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:46:24 smithi189 ceph-mon[36187]: pgmap v3255: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T01:46:24.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:46:24 smithi193 ceph-mon[36878]: pgmap v3255: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T01:46:26.750 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:46:26 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 4881.684149 secs 2024-03-24T01:46:26.750 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:46:26 smithi118 ceph-mon[31535]: pgmap v3256: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T01:46:26.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:46:26 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 4881.684149 secs 2024-03-24T01:46:26.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:46:26 smithi189 ceph-mon[36187]: pgmap v3256: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T01:46:26.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:46:26 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 4881.684149 secs 2024-03-24T01:46:26.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:46:26 smithi193 ceph-mon[36878]: pgmap v3256: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T01:46:28.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:46:28 smithi118 ceph-mon[31535]: pgmap v3257: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:46:28.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:46:28 smithi189 ceph-mon[36187]: pgmap v3257: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:46:28.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:46:28 smithi193 ceph-mon[36878]: pgmap v3257: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:46:30.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:46:30 smithi118 ceph-mon[31535]: pgmap v3258: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-24T01:46:30.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:46:30 smithi189 ceph-mon[36187]: pgmap v3258: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-24T01:46:30.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:46:30 smithi193 ceph-mon[36878]: pgmap v3258: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-24T01:46:31.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:46:31 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 4886.684234 secs 2024-03-24T01:46:31.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:46:31 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 4886.684234 secs 2024-03-24T01:46:31.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:46:31 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 4886.684234 secs 2024-03-24T01:46:32.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:46:32 smithi118 ceph-mon[31535]: pgmap v3259: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:46:32.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:46:32 smithi189 ceph-mon[36187]: pgmap v3259: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:46:32.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:46:32 smithi193 ceph-mon[36878]: pgmap v3259: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:46:34.947 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:46:34 smithi118 ceph-mon[31535]: pgmap v3260: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T01:46:34.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:46:34 smithi189 ceph-mon[36187]: pgmap v3260: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T01:46:34.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:46:34 smithi193 ceph-mon[36878]: pgmap v3260: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T01:46:36.755 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:46:36 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 4891.684317 secs 2024-03-24T01:46:36.755 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:46:36 smithi189 ceph-mon[36187]: pgmap v3261: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T01:46:36.762 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:46:36 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 4891.684317 secs 2024-03-24T01:46:36.762 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:46:36 smithi193 ceph-mon[36878]: pgmap v3261: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T01:46:36.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:46:36 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 4891.684317 secs 2024-03-24T01:46:36.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:46:36 smithi118 ceph-mon[31535]: pgmap v3261: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T01:46:38.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:46:38 smithi118 ceph-mon[31535]: pgmap v3262: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:46:38.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:46:38 smithi189 ceph-mon[36187]: pgmap v3262: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:46:38.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:46:38 smithi193 ceph-mon[36878]: pgmap v3262: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:46:40.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:46:40 smithi118 ceph-mon[31535]: pgmap v3263: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-24T01:46:40.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:46:40 smithi189 ceph-mon[36187]: pgmap v3263: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-24T01:46:40.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:46:40 smithi193 ceph-mon[36878]: pgmap v3263: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-24T01:46:41.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:46:41 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 4896.684378 secs 2024-03-24T01:46:41.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:46:41 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 4896.684378 secs 2024-03-24T01:46:41.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:46:41 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 4896.684378 secs 2024-03-24T01:46:42.947 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:46:42 smithi118 ceph-mon[31535]: pgmap v3264: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:46:42.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:46:42 smithi189 ceph-mon[36187]: pgmap v3264: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:46:42.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:46:42 smithi193 ceph-mon[36878]: pgmap v3264: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:46:44.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:46:44 smithi118 ceph-mon[31535]: pgmap v3265: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T01:46:44.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:46:44 smithi189 ceph-mon[36187]: pgmap v3265: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T01:46:44.963 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:46:44 smithi193 ceph-mon[36878]: pgmap v3265: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T01:46:46.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:46:46 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 4901.684435 secs 2024-03-24T01:46:46.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:46:46 smithi118 ceph-mon[31535]: pgmap v3266: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T01:46:46.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:46:46 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 4901.684435 secs 2024-03-24T01:46:46.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:46:46 smithi189 ceph-mon[36187]: pgmap v3266: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T01:46:46.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:46:46 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 4901.684435 secs 2024-03-24T01:46:46.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:46:46 smithi193 ceph-mon[36878]: pgmap v3266: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T01:46:48.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:46:48 smithi118 ceph-mon[31535]: pgmap v3267: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:46:48.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:46:48 smithi189 ceph-mon[36187]: pgmap v3267: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:46:48.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:46:48 smithi193 ceph-mon[36878]: pgmap v3267: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:46:50.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:46:50 smithi118 ceph-mon[31535]: pgmap v3268: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-24T01:46:50.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:46:50 smithi189 ceph-mon[36187]: pgmap v3268: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-24T01:46:50.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:46:50 smithi193 ceph-mon[36878]: pgmap v3268: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-24T01:46:51.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:46:51 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 4906.684476 secs 2024-03-24T01:46:51.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:46:51 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 4906.684476 secs 2024-03-24T01:46:51.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:46:51 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 4906.684476 secs 2024-03-24T01:46:52.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:46:52 smithi118 ceph-mon[31535]: pgmap v3269: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:46:52.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:46:52 smithi189 ceph-mon[36187]: pgmap v3269: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:46:52.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:46:52 smithi193 ceph-mon[36878]: pgmap v3269: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:46:54.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:46:54 smithi118 ceph-mon[31535]: pgmap v3270: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T01:46:54.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:46:54 smithi189 ceph-mon[36187]: pgmap v3270: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T01:46:54.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:46:54 smithi193 ceph-mon[36878]: pgmap v3270: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T01:46:56.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:46:56 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 4911.684566 secs 2024-03-24T01:46:56.947 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:46:56 smithi118 ceph-mon[31535]: pgmap v3271: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T01:46:56.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:46:56 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 4911.684566 secs 2024-03-24T01:46:56.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:46:56 smithi189 ceph-mon[36187]: pgmap v3271: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T01:46:56.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:46:56 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 4911.684566 secs 2024-03-24T01:46:56.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:46:56 smithi193 ceph-mon[36878]: pgmap v3271: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T01:46:58.947 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:46:58 smithi118 ceph-mon[31535]: pgmap v3272: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:46:58.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:46:58 smithi189 ceph-mon[36187]: pgmap v3272: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:46:58.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:46:58 smithi193 ceph-mon[36878]: pgmap v3272: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:47:00.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:47:00 smithi118 ceph-mon[31535]: pgmap v3273: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-24T01:47:00.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:47:00 smithi189 ceph-mon[36187]: pgmap v3273: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-24T01:47:00.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:47:00 smithi193 ceph-mon[36878]: pgmap v3273: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-24T01:47:01.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:47:01 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 4916.684655 secs 2024-03-24T01:47:01.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:47:01 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 4916.684655 secs 2024-03-24T01:47:01.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:47:01 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 4916.684655 secs 2024-03-24T01:47:02.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:47:02 smithi118 ceph-mon[31535]: pgmap v3274: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:47:02.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:47:02 smithi189 ceph-mon[36187]: pgmap v3274: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:47:02.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:47:02 smithi193 ceph-mon[36878]: pgmap v3274: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:47:04.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:47:04 smithi118 ceph-mon[31535]: pgmap v3275: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T01:47:04.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:47:04 smithi189 ceph-mon[36187]: pgmap v3275: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T01:47:04.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:47:04 smithi193 ceph-mon[36878]: pgmap v3275: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T01:47:06.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:47:06 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 4921.684712 secs 2024-03-24T01:47:06.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:47:06 smithi118 ceph-mon[31535]: pgmap v3276: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T01:47:06.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:47:06 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T01:47:06.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:47:06 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 4921.684712 secs 2024-03-24T01:47:06.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:47:06 smithi189 ceph-mon[36187]: pgmap v3276: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T01:47:06.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:47:06 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T01:47:06.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:47:06 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 4921.684712 secs 2024-03-24T01:47:06.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:47:06 smithi193 ceph-mon[36878]: pgmap v3276: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T01:47:06.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:47:06 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T01:47:07.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:47:07 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T01:47:07.947 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:47:07 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T01:47:07.947 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:47:07 smithi118 ceph-mon[31535]: pgmap v3277: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 5 op/s 2024-03-24T01:47:07.947 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:47:07 smithi118 ceph-mon[31535]: pgmap v3278: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-03-24T01:47:07.947 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:47:07 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T01:47:07.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:47:07 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T01:47:07.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:47:07 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T01:47:07.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:47:07 smithi189 ceph-mon[36187]: pgmap v3277: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 5 op/s 2024-03-24T01:47:07.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:47:07 smithi189 ceph-mon[36187]: pgmap v3278: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-03-24T01:47:07.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:47:07 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T01:47:07.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:47:07 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T01:47:07.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:47:07 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T01:47:07.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:47:07 smithi193 ceph-mon[36878]: pgmap v3277: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 5 op/s 2024-03-24T01:47:07.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:47:07 smithi193 ceph-mon[36878]: pgmap v3278: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-03-24T01:47:07.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:47:07 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T01:47:09.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:47:08 smithi118 ceph-mon[31535]: pgmap v3279: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 4 op/s 2024-03-24T01:47:09.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:47:08 smithi189 ceph-mon[36187]: pgmap v3279: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 4 op/s 2024-03-24T01:47:09.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:47:08 smithi193 ceph-mon[36878]: pgmap v3279: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 4 op/s 2024-03-24T01:47:10.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:47:10 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 4926.684797 secs 2024-03-24T01:47:11.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:47:10 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 4926.684797 secs 2024-03-24T01:47:11.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:47:10 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 4926.684797 secs 2024-03-24T01:47:12.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:47:11 smithi118 ceph-mon[31535]: pgmap v3280: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-03-24T01:47:12.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:47:11 smithi189 ceph-mon[36187]: pgmap v3280: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-03-24T01:47:12.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:47:11 smithi193 ceph-mon[36878]: pgmap v3280: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-03-24T01:47:13.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:47:12 smithi118 ceph-mon[31535]: pgmap v3281: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 5 op/s 2024-03-24T01:47:13.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:47:12 smithi189 ceph-mon[36187]: pgmap v3281: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 5 op/s 2024-03-24T01:47:13.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:47:12 smithi193 ceph-mon[36878]: pgmap v3281: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 5 op/s 2024-03-24T01:47:15.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:47:14 smithi118 ceph-mon[31535]: pgmap v3282: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-03-24T01:47:15.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:47:14 smithi189 ceph-mon[36187]: pgmap v3282: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-03-24T01:47:15.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:47:14 smithi193 ceph-mon[36878]: pgmap v3282: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-03-24T01:47:16.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:47:15 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 4931.684856 secs 2024-03-24T01:47:16.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:47:15 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 4931.684856 secs 2024-03-24T01:47:16.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:47:15 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 4931.684856 secs 2024-03-24T01:47:17.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:47:16 smithi118 ceph-mon[31535]: pgmap v3283: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 5 op/s 2024-03-24T01:47:17.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:47:16 smithi189 ceph-mon[36187]: pgmap v3283: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 5 op/s 2024-03-24T01:47:17.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:47:16 smithi193 ceph-mon[36878]: pgmap v3283: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 5 op/s 2024-03-24T01:47:19.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:47:18 smithi118 ceph-mon[31535]: pgmap v3284: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-03-24T01:47:19.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:47:18 smithi189 ceph-mon[36187]: pgmap v3284: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-03-24T01:47:19.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:47:18 smithi193 ceph-mon[36878]: pgmap v3284: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-03-24T01:47:20.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:47:20 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 4936.684946 secs 2024-03-24T01:47:21.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:47:20 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 4936.684946 secs 2024-03-24T01:47:21.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:47:20 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 4936.684946 secs 2024-03-24T01:47:22.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:47:21 smithi118 ceph-mon[31535]: pgmap v3285: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 6 op/s 2024-03-24T01:47:22.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:47:21 smithi189 ceph-mon[36187]: pgmap v3285: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 6 op/s 2024-03-24T01:47:22.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:47:21 smithi193 ceph-mon[36878]: pgmap v3285: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 6 op/s 2024-03-24T01:47:23.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:47:22 smithi118 ceph-mon[31535]: pgmap v3286: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 5 op/s 2024-03-24T01:47:23.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:47:22 smithi189 ceph-mon[36187]: pgmap v3286: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 5 op/s 2024-03-24T01:47:23.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:47:22 smithi193 ceph-mon[36878]: pgmap v3286: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 5 op/s 2024-03-24T01:47:25.447 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:47:24 smithi118 ceph-mon[31535]: pgmap v3287: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 6 op/s 2024-03-24T01:47:25.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:47:24 smithi189 ceph-mon[36187]: pgmap v3287: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 6 op/s 2024-03-24T01:47:25.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:47:24 smithi193 ceph-mon[36878]: pgmap v3287: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 6 op/s 2024-03-24T01:47:26.349 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:47:25 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 4941.685034 secs 2024-03-24T01:47:26.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:47:25 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 4941.685034 secs 2024-03-24T01:47:26.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:47:25 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 4941.685034 secs 2024-03-24T01:47:27.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:47:26 smithi118 ceph-mon[31535]: pgmap v3288: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-03-24T01:47:27.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:47:26 smithi189 ceph-mon[36187]: pgmap v3288: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-03-24T01:47:27.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:47:26 smithi193 ceph-mon[36878]: pgmap v3288: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-03-24T01:47:29.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:47:28 smithi118 ceph-mon[31535]: pgmap v3289: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-03-24T01:47:29.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:47:28 smithi189 ceph-mon[36187]: pgmap v3289: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-03-24T01:47:29.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:47:28 smithi193 ceph-mon[36878]: pgmap v3289: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-03-24T01:47:30.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:47:30 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 4946.685118 secs 2024-03-24T01:47:31.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:47:30 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 4946.685118 secs 2024-03-24T01:47:31.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:47:30 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 4946.685118 secs 2024-03-24T01:47:32.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:47:31 smithi118 ceph-mon[31535]: pgmap v3290: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T01:47:32.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:47:31 smithi189 ceph-mon[36187]: pgmap v3290: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T01:47:32.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:47:31 smithi193 ceph-mon[36878]: pgmap v3290: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T01:47:33.447 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:47:32 smithi118 ceph-mon[31535]: pgmap v3291: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 5 op/s 2024-03-24T01:47:33.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:47:32 smithi189 ceph-mon[36187]: pgmap v3291: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 5 op/s 2024-03-24T01:47:33.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:47:32 smithi193 ceph-mon[36878]: pgmap v3291: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 5 op/s 2024-03-24T01:47:35.320 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:47:34 smithi118 ceph-mon[31535]: pgmap v3292: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T01:47:35.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:47:34 smithi189 ceph-mon[36187]: pgmap v3292: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T01:47:35.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:47:34 smithi193 ceph-mon[36878]: pgmap v3292: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T01:47:36.355 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:47:35 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 4951.685180 secs 2024-03-24T01:47:36.362 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:47:35 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 4951.685180 secs 2024-03-24T01:47:36.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:47:35 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 4951.685180 secs 2024-03-24T01:47:37.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:47:36 smithi118 ceph-mon[31535]: pgmap v3293: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:47:37.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:47:36 smithi189 ceph-mon[36187]: pgmap v3293: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:47:37.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:47:36 smithi193 ceph-mon[36878]: pgmap v3293: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:47:39.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:47:38 smithi118 ceph-mon[31535]: pgmap v3294: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T01:47:39.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:47:38 smithi189 ceph-mon[36187]: pgmap v3294: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T01:47:39.462 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:47:38 smithi193 ceph-mon[36878]: pgmap v3294: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T01:47:40.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:47:40 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 4956.685240 secs 2024-03-24T01:47:41.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:47:40 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 4956.685240 secs 2024-03-24T01:47:41.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:47:40 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 4956.685240 secs 2024-03-24T01:47:42.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:47:41 smithi118 ceph-mon[31535]: pgmap v3295: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T01:47:42.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:47:41 smithi189 ceph-mon[36187]: pgmap v3295: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T01:47:42.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:47:41 smithi193 ceph-mon[36878]: pgmap v3295: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T01:47:43.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:47:42 smithi118 ceph-mon[31535]: pgmap v3296: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 5 op/s 2024-03-24T01:47:43.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:47:42 smithi189 ceph-mon[36187]: pgmap v3296: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 5 op/s 2024-03-24T01:47:43.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:47:42 smithi193 ceph-mon[36878]: pgmap v3296: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 5 op/s 2024-03-24T01:47:45.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:47:44 smithi118 ceph-mon[31535]: pgmap v3297: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T01:47:45.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:47:44 smithi189 ceph-mon[36187]: pgmap v3297: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T01:47:45.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:47:44 smithi193 ceph-mon[36878]: pgmap v3297: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T01:47:46.447 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:47:45 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 4961.685298 secs 2024-03-24T01:47:46.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:47:45 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 4961.685298 secs 2024-03-24T01:47:46.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:47:45 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 4961.685298 secs 2024-03-24T01:47:46.528 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T01:47:46.527+0000 7f81db400640 1 -- 172.21.15.118:0/659686424 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- auth(proto 2 132 bytes epoch 0) v1 -- 0x7f81c40064d0 con 0x5622651a13f0 2024-03-24T01:47:46.528 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T01:47:46.527+0000 7f81ee000640 1 -- 172.21.15.118:0/1693767697 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- auth(proto 2 132 bytes epoch 0) v1 -- 0x7f81c80036c0 con 0x7f81f81052d0 2024-03-24T01:47:46.528 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T01:47:46.527+0000 7f81e4c00640 1 -- 172.21.15.118:0/659686424 <== mon.1 v2:172.21.15.193:3300/0 42 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 233+0+0 (secure 0 0 0) 0x7f81c40064d0 con 0x5622651a13f0 2024-03-24T01:47:46.529 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T01:47:46.527+0000 7f81ef400640 1 -- 172.21.15.118:0/1693767697 <== mon.2 v2:172.21.15.189:3300/0 42 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 233+0+0 (secure 0 0 0) 0x7f81c80036c0 con 0x7f81f81052d0 2024-03-24T01:47:47.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:47:46 smithi118 ceph-mon[31535]: pgmap v3298: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:47:47.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:47:46 smithi189 ceph-mon[36187]: pgmap v3298: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:47:47.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:47:46 smithi193 ceph-mon[36878]: pgmap v3298: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:47:49.447 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:47:48 smithi118 ceph-mon[31535]: pgmap v3299: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T01:47:49.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:47:48 smithi189 ceph-mon[36187]: pgmap v3299: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T01:47:49.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:47:48 smithi193 ceph-mon[36878]: pgmap v3299: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T01:47:50.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:47:50 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 4966.685360 secs 2024-03-24T01:47:51.193 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:47:50 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 4966.685360 secs 2024-03-24T01:47:51.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:47:50 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 4966.685360 secs 2024-03-24T01:47:52.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:47:51 smithi118 ceph-mon[31535]: pgmap v3300: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T01:47:52.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:47:51 smithi189 ceph-mon[36187]: pgmap v3300: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T01:47:52.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:47:51 smithi193 ceph-mon[36878]: pgmap v3300: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T01:47:53.240 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:47:52 smithi118 ceph-mon[31535]: pgmap v3301: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 5 op/s 2024-03-24T01:47:53.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:47:52 smithi189 ceph-mon[36187]: pgmap v3301: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 5 op/s 2024-03-24T01:47:53.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:47:52 smithi193 ceph-mon[36878]: pgmap v3301: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 5 op/s 2024-03-24T01:47:55.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:47:54 smithi118 ceph-mon[31535]: pgmap v3302: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 6 op/s 2024-03-24T01:47:55.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:47:54 smithi189 ceph-mon[36187]: pgmap v3302: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 6 op/s 2024-03-24T01:47:55.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:47:54 smithi193 ceph-mon[36878]: pgmap v3302: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 6 op/s 2024-03-24T01:47:56.348 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:47:55 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 4971.685420 secs 2024-03-24T01:47:56.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:47:55 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 4971.685420 secs 2024-03-24T01:47:56.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:47:55 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 4971.685420 secs 2024-03-24T01:47:57.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:47:56 smithi118 ceph-mon[31535]: pgmap v3303: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-03-24T01:47:57.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:47:56 smithi189 ceph-mon[36187]: pgmap v3303: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-03-24T01:47:57.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:47:56 smithi193 ceph-mon[36878]: pgmap v3303: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-03-24T01:47:59.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:47:58 smithi118 ceph-mon[31535]: pgmap v3304: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-03-24T01:47:59.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:47:58 smithi189 ceph-mon[36187]: pgmap v3304: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-03-24T01:47:59.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:47:58 smithi193 ceph-mon[36878]: pgmap v3304: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-03-24T01:48:00.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:48:00 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 4976.685509 secs 2024-03-24T01:48:01.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:48:00 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 4976.685509 secs 2024-03-24T01:48:01.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:48:00 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 4976.685509 secs 2024-03-24T01:48:02.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:48:01 smithi118 ceph-mon[31535]: pgmap v3305: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 5 op/s 2024-03-24T01:48:02.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:48:01 smithi189 ceph-mon[36187]: pgmap v3305: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 5 op/s 2024-03-24T01:48:02.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:48:01 smithi193 ceph-mon[36878]: pgmap v3305: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 5 op/s 2024-03-24T01:48:03.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:48:02 smithi118 ceph-mon[31535]: pgmap v3306: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 5 op/s 2024-03-24T01:48:03.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:48:02 smithi189 ceph-mon[36187]: pgmap v3306: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 5 op/s 2024-03-24T01:48:03.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:48:02 smithi193 ceph-mon[36878]: pgmap v3306: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 5 op/s 2024-03-24T01:48:05.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:48:05 smithi118 ceph-mon[31535]: pgmap v3307: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 6 op/s 2024-03-24T01:48:05.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:48:05 smithi189 ceph-mon[36187]: pgmap v3307: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 6 op/s 2024-03-24T01:48:05.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:48:05 smithi193 ceph-mon[36878]: pgmap v3307: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 6 op/s 2024-03-24T01:48:06.855 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:48:06 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 4981.685569 secs 2024-03-24T01:48:06.861 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:48:06 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 4981.685569 secs 2024-03-24T01:48:06.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:48:06 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 4981.685569 secs 2024-03-24T01:48:07.947 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:48:07 smithi118 ceph-mon[31535]: pgmap v3308: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:48:07.947 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:48:07 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T01:48:07.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:48:07 smithi189 ceph-mon[36187]: pgmap v3308: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:48:07.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:48:07 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T01:48:07.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:48:07 smithi193 ceph-mon[36878]: pgmap v3308: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:48:07.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:48:07 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T01:48:08.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:48:08 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T01:48:08.947 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:48:08 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T01:48:08.947 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:48:08 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T01:48:08.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:48:08 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T01:48:08.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:48:08 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T01:48:08.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:48:08 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T01:48:08.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:48:08 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T01:48:08.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:48:08 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T01:48:08.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:48:08 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T01:48:09.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:48:09 smithi118 ceph-mon[31535]: pgmap v3309: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 6 op/s 2024-03-24T01:48:09.947 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:48:09 smithi118 ceph-mon[31535]: pgmap v3310: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 5 op/s 2024-03-24T01:48:09.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:48:09 smithi189 ceph-mon[36187]: pgmap v3309: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 6 op/s 2024-03-24T01:48:09.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:48:09 smithi189 ceph-mon[36187]: pgmap v3310: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 5 op/s 2024-03-24T01:48:09.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:48:09 smithi193 ceph-mon[36878]: pgmap v3309: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 6 op/s 2024-03-24T01:48:09.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:48:09 smithi193 ceph-mon[36878]: pgmap v3310: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 5 op/s 2024-03-24T01:48:11.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:48:11 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 4986.685656 secs 2024-03-24T01:48:11.947 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:48:11 smithi118 ceph-mon[31535]: pgmap v3311: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-03-24T01:48:11.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:48:11 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 4986.685656 secs 2024-03-24T01:48:11.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:48:11 smithi189 ceph-mon[36187]: pgmap v3311: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-03-24T01:48:11.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:48:11 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 4986.685656 secs 2024-03-24T01:48:11.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:48:11 smithi193 ceph-mon[36878]: pgmap v3311: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-03-24T01:48:13.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:48:13 smithi118 ceph-mon[31535]: pgmap v3312: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 5 op/s 2024-03-24T01:48:13.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:48:13 smithi189 ceph-mon[36187]: pgmap v3312: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 5 op/s 2024-03-24T01:48:13.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:48:13 smithi193 ceph-mon[36878]: pgmap v3312: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 5 op/s 2024-03-24T01:48:15.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:48:15 smithi118 ceph-mon[31535]: pgmap v3313: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-03-24T01:48:15.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:48:15 smithi189 ceph-mon[36187]: pgmap v3313: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-03-24T01:48:15.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:48:15 smithi193 ceph-mon[36878]: pgmap v3313: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-03-24T01:48:16.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:48:16 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 4991.685708 secs 2024-03-24T01:48:16.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:48:16 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 4991.685708 secs 2024-03-24T01:48:16.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:48:16 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 4991.685708 secs 2024-03-24T01:48:17.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:48:17 smithi118 ceph-mon[31535]: pgmap v3314: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-03-24T01:48:17.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:48:17 smithi189 ceph-mon[36187]: pgmap v3314: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-03-24T01:48:17.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:48:17 smithi193 ceph-mon[36878]: pgmap v3314: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-03-24T01:48:19.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:48:19 smithi118 ceph-mon[31535]: pgmap v3315: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 5 op/s 2024-03-24T01:48:19.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:48:19 smithi189 ceph-mon[36187]: pgmap v3315: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 5 op/s 2024-03-24T01:48:19.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:48:19 smithi193 ceph-mon[36878]: pgmap v3315: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 5 op/s 2024-03-24T01:48:21.912 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:48:21 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 4996.685794 secs 2024-03-24T01:48:21.913 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:48:21 smithi118 ceph-mon[31535]: pgmap v3316: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T01:48:21.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:48:21 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 4996.685794 secs 2024-03-24T01:48:21.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:48:21 smithi189 ceph-mon[36187]: pgmap v3316: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T01:48:21.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:48:21 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 4996.685794 secs 2024-03-24T01:48:21.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:48:21 smithi193 ceph-mon[36878]: pgmap v3316: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T01:48:23.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:48:23 smithi118 ceph-mon[31535]: pgmap v3317: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:48:23.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:48:23 smithi189 ceph-mon[36187]: pgmap v3317: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:48:23.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:48:23 smithi193 ceph-mon[36878]: pgmap v3317: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:48:25.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:48:25 smithi118 ceph-mon[31535]: pgmap v3318: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-24T01:48:25.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:48:25 smithi189 ceph-mon[36187]: pgmap v3318: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-24T01:48:25.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:48:25 smithi193 ceph-mon[36878]: pgmap v3318: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-24T01:48:26.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:48:26 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 5001.685880 secs 2024-03-24T01:48:26.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:48:26 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 5001.685880 secs 2024-03-24T01:48:26.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:48:26 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 5001.685880 secs 2024-03-24T01:48:27.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:48:27 smithi118 ceph-mon[31535]: pgmap v3319: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:48:27.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:48:27 smithi189 ceph-mon[36187]: pgmap v3319: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:48:27.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:48:27 smithi193 ceph-mon[36878]: pgmap v3319: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:48:29.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:48:29 smithi118 ceph-mon[31535]: pgmap v3320: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T01:48:29.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:48:29 smithi189 ceph-mon[36187]: pgmap v3320: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T01:48:29.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:48:29 smithi193 ceph-mon[36878]: pgmap v3320: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T01:48:31.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:48:31 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 5006.685970 secs 2024-03-24T01:48:31.947 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:48:31 smithi118 ceph-mon[31535]: pgmap v3321: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T01:48:31.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:48:31 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 5006.685970 secs 2024-03-24T01:48:31.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:48:31 smithi189 ceph-mon[36187]: pgmap v3321: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T01:48:31.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:48:31 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 5006.685970 secs 2024-03-24T01:48:31.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:48:31 smithi193 ceph-mon[36878]: pgmap v3321: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T01:48:33.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:48:33 smithi118 ceph-mon[31535]: pgmap v3322: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:48:33.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:48:33 smithi189 ceph-mon[36187]: pgmap v3322: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:48:33.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:48:33 smithi193 ceph-mon[36878]: pgmap v3322: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:48:35.947 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:48:35 smithi118 ceph-mon[31535]: pgmap v3323: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-24T01:48:35.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:48:35 smithi189 ceph-mon[36187]: pgmap v3323: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-24T01:48:35.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:48:35 smithi193 ceph-mon[36878]: pgmap v3323: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-24T01:48:36.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:48:36 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 5011.686032 secs 2024-03-24T01:48:36.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:48:36 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 5011.686032 secs 2024-03-24T01:48:36.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:48:36 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 5011.686032 secs 2024-03-24T01:48:37.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:48:37 smithi118 ceph-mon[31535]: pgmap v3324: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:48:37.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:48:37 smithi189 ceph-mon[36187]: pgmap v3324: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:48:37.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:48:37 smithi193 ceph-mon[36878]: pgmap v3324: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:48:39.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:48:39 smithi118 ceph-mon[31535]: pgmap v3325: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T01:48:39.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:48:39 smithi189 ceph-mon[36187]: pgmap v3325: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T01:48:39.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:48:39 smithi193 ceph-mon[36878]: pgmap v3325: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T01:48:40.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:48:40 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 5016.686112 secs 2024-03-24T01:48:40.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:48:40 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 5016.686112 secs 2024-03-24T01:48:40.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:48:40 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 5016.686112 secs 2024-03-24T01:48:41.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:48:41 smithi118 ceph-mon[31535]: pgmap v3326: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T01:48:41.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:48:41 smithi189 ceph-mon[36187]: pgmap v3326: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T01:48:41.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:48:41 smithi193 ceph-mon[36878]: pgmap v3326: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T01:48:43.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:48:43 smithi189 ceph-mon[36187]: pgmap v3327: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:48:43.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:48:43 smithi193 ceph-mon[36878]: pgmap v3327: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:48:44.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:48:43 smithi118 ceph-mon[31535]: pgmap v3327: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:48:45.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:48:45 smithi189 ceph-mon[36187]: pgmap v3328: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-24T01:48:45.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:48:45 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 5021.686173 secs 2024-03-24T01:48:45.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:48:45 smithi193 ceph-mon[36878]: pgmap v3328: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-24T01:48:45.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:48:45 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 5021.686173 secs 2024-03-24T01:48:46.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:48:45 smithi118 ceph-mon[31535]: pgmap v3328: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-24T01:48:46.197 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:48:45 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 5021.686173 secs 2024-03-24T01:48:47.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:48:47 smithi193 ceph-mon[36878]: pgmap v3329: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:48:48.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:48:47 smithi118 ceph-mon[31535]: pgmap v3329: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:48:48.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:48:47 smithi189 ceph-mon[36187]: pgmap v3329: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:48:49.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:48:49 smithi193 ceph-mon[36878]: pgmap v3330: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T01:48:50.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:48:49 smithi118 ceph-mon[31535]: pgmap v3330: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T01:48:50.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:48:49 smithi189 ceph-mon[36187]: pgmap v3330: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T01:48:51.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:48:50 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 5026.686188 secs 2024-03-24T01:48:51.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:48:50 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 5026.686188 secs 2024-03-24T01:48:51.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:48:50 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 5026.686188 secs 2024-03-24T01:48:52.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:48:51 smithi118 ceph-mon[31535]: pgmap v3331: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T01:48:52.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:48:51 smithi189 ceph-mon[36187]: pgmap v3331: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T01:48:52.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:48:51 smithi193 ceph-mon[36878]: pgmap v3331: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T01:48:54.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:48:53 smithi118 ceph-mon[31535]: pgmap v3332: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:48:54.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:48:53 smithi189 ceph-mon[36187]: pgmap v3332: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:48:54.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:48:53 smithi193 ceph-mon[36878]: pgmap v3332: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:48:56.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:48:55 smithi118 ceph-mon[31535]: pgmap v3333: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-24T01:48:56.197 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:48:55 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 5031.686276 secs 2024-03-24T01:48:56.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:48:55 smithi189 ceph-mon[36187]: pgmap v3333: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-24T01:48:56.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:48:55 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 5031.686276 secs 2024-03-24T01:48:56.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:48:55 smithi193 ceph-mon[36878]: pgmap v3333: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-24T01:48:56.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:48:55 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 5031.686276 secs 2024-03-24T01:48:58.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:48:57 smithi118 ceph-mon[31535]: pgmap v3334: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:48:58.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:48:57 smithi189 ceph-mon[36187]: pgmap v3334: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:48:58.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:48:57 smithi193 ceph-mon[36878]: pgmap v3334: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:49:00.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:48:59 smithi118 ceph-mon[31535]: pgmap v3335: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T01:49:00.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:48:59 smithi189 ceph-mon[36187]: pgmap v3335: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T01:49:00.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:48:59 smithi193 ceph-mon[36878]: pgmap v3335: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T01:49:01.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:49:00 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 5036.686379 secs 2024-03-24T01:49:01.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:49:00 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 5036.686379 secs 2024-03-24T01:49:01.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:49:00 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 5036.686379 secs 2024-03-24T01:49:02.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:49:01 smithi118 ceph-mon[31535]: pgmap v3336: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T01:49:02.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:49:01 smithi189 ceph-mon[36187]: pgmap v3336: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T01:49:02.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:49:01 smithi193 ceph-mon[36878]: pgmap v3336: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T01:49:04.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:49:03 smithi118 ceph-mon[31535]: pgmap v3337: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:49:04.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:49:03 smithi189 ceph-mon[36187]: pgmap v3337: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:49:04.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:49:03 smithi193 ceph-mon[36878]: pgmap v3337: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:49:06.013 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:49:05 smithi193 ceph-mon[36878]: pgmap v3338: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-24T01:49:06.013 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:49:05 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 5041.686440 secs 2024-03-24T01:49:06.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:49:05 smithi118 ceph-mon[31535]: pgmap v3338: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-24T01:49:06.197 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:49:05 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 5041.686440 secs 2024-03-24T01:49:06.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:49:05 smithi189 ceph-mon[36187]: pgmap v3338: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-24T01:49:06.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:49:05 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 5041.686440 secs 2024-03-24T01:49:08.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:49:07 smithi118 ceph-mon[31535]: pgmap v3339: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:49:08.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:49:07 smithi189 ceph-mon[36187]: pgmap v3339: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:49:08.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:49:07 smithi193 ceph-mon[36878]: pgmap v3339: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:49:09.019 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:49:08 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T01:49:09.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:49:08 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T01:49:09.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:49:08 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T01:49:09.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:49:09 smithi193 ceph-mon[36878]: pgmap v3340: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T01:49:09.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:49:09 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T01:49:09.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:49:09 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T01:49:09.962 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:49:09 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T01:49:09.962 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:49:09 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T01:49:10.129 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:49:09 smithi189 ceph-mon[36187]: pgmap v3340: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T01:49:10.129 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:49:09 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T01:49:10.129 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:49:09 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T01:49:10.129 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:49:09 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T01:49:10.129 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:49:09 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T01:49:10.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:49:09 smithi118 ceph-mon[31535]: pgmap v3340: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T01:49:10.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:49:09 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T01:49:10.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:49:09 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T01:49:10.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:49:09 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T01:49:10.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:49:09 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T01:49:11.087 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:49:10 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 5046.686506 secs 2024-03-24T01:49:11.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:49:10 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 5046.686506 secs 2024-03-24T01:49:11.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:49:10 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 5046.686506 secs 2024-03-24T01:49:12.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:49:11 smithi118 ceph-mon[31535]: pgmap v3341: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T01:49:12.197 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:49:11 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T01:49:12.197 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:49:11 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T01:49:12.197 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:49:11 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T01:49:12.197 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:49:11 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T01:49:12.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:49:11 smithi189 ceph-mon[36187]: pgmap v3341: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T01:49:12.205 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:49:11 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T01:49:12.205 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:49:11 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T01:49:12.205 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:49:11 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T01:49:12.205 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:49:11 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T01:49:12.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:49:11 smithi193 ceph-mon[36878]: pgmap v3341: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T01:49:12.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:49:11 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T01:49:12.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:49:11 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T01:49:12.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:49:11 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T01:49:12.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:49:11 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T01:49:14.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:49:13 smithi118 ceph-mon[31535]: pgmap v3342: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:49:14.197 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:49:13 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T01:49:14.197 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:49:13 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T01:49:14.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:49:13 smithi189 ceph-mon[36187]: pgmap v3342: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:49:14.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:49:13 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T01:49:14.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:49:13 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T01:49:14.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:49:13 smithi193 ceph-mon[36878]: pgmap v3342: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:49:14.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:49:13 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T01:49:14.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:49:13 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T01:49:16.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:49:15 smithi118 ceph-mon[31535]: pgmap v3343: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-24T01:49:16.197 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:49:15 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T01:49:16.197 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:49:15 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T01:49:16.197 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:49:15 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T01:49:16.197 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:49:15 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 5051.686569 secs 2024-03-24T01:49:16.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:49:15 smithi189 ceph-mon[36187]: pgmap v3343: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-24T01:49:16.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:49:15 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T01:49:16.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:49:15 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T01:49:16.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:49:15 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T01:49:16.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:49:15 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 5051.686569 secs 2024-03-24T01:49:16.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:49:15 smithi193 ceph-mon[36878]: pgmap v3343: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-24T01:49:16.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:49:15 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T01:49:16.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:49:15 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T01:49:16.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:49:15 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T01:49:16.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:49:15 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 5051.686569 secs 2024-03-24T01:49:16.543 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T01:49:16.541+0000 7f81db400640 1 -- 172.21.15.118:0/659686424 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f81c4006460 con 0x5622651a13f0 2024-03-24T01:49:16.543 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T01:49:16.541+0000 7f81ee000640 1 -- 172.21.15.118:0/1693767697 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f81c80036c0 con 0x7f81f81052d0 2024-03-24T01:49:16.543 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T01:49:16.542+0000 7f81e4c00640 1 -- 172.21.15.118:0/659686424 <== mon.1 v2:172.21.15.193:3300/0 43 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 409+0+0 (secure 0 0 0) 0x7f81c4006460 con 0x5622651a13f0 2024-03-24T01:49:16.543 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T01:49:16.542+0000 7f81ef400640 1 -- 172.21.15.118:0/1693767697 <== mon.2 v2:172.21.15.189:3300/0 43 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (secure 0 0 0) 0x7f81c80036c0 con 0x7f81f81052d0 2024-03-24T01:49:17.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:49:16 smithi118 ceph-mon[31535]: pgmap v3344: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 5 op/s 2024-03-24T01:49:17.197 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:49:16 smithi118 ceph-mon[31535]: pgmap v3345: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 6 op/s 2024-03-24T01:49:17.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:49:16 smithi189 ceph-mon[36187]: pgmap v3344: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 5 op/s 2024-03-24T01:49:17.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:49:16 smithi189 ceph-mon[36187]: pgmap v3345: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 6 op/s 2024-03-24T01:49:17.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:49:16 smithi193 ceph-mon[36878]: pgmap v3344: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 5 op/s 2024-03-24T01:49:17.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:49:16 smithi193 ceph-mon[36878]: pgmap v3345: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 6 op/s 2024-03-24T01:49:19.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:49:18 smithi118 ceph-mon[31535]: pgmap v3346: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 5 op/s 2024-03-24T01:49:19.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:49:18 smithi189 ceph-mon[36187]: pgmap v3346: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 5 op/s 2024-03-24T01:49:19.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:49:18 smithi193 ceph-mon[36878]: pgmap v3346: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.3 KiB/s rd, 5 op/s 2024-03-24T01:49:21.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:49:20 smithi118 ceph-mon[31535]: pgmap v3347: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 6 op/s 2024-03-24T01:49:21.197 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:49:20 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 5056.686655 secs 2024-03-24T01:49:21.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:49:20 smithi189 ceph-mon[36187]: pgmap v3347: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 6 op/s 2024-03-24T01:49:21.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:49:20 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 5056.686655 secs 2024-03-24T01:49:21.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:49:20 smithi193 ceph-mon[36878]: pgmap v3347: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 6 op/s 2024-03-24T01:49:21.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:49:20 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 5056.686655 secs 2024-03-24T01:49:23.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:49:22 smithi118 ceph-mon[31535]: pgmap v3348: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 6 op/s 2024-03-24T01:49:23.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:49:22 smithi189 ceph-mon[36187]: pgmap v3348: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 6 op/s 2024-03-24T01:49:23.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:49:22 smithi193 ceph-mon[36878]: pgmap v3348: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 6 op/s 2024-03-24T01:49:25.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:49:24 smithi118 ceph-mon[31535]: pgmap v3349: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T01:49:25.205 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:49:24 smithi189 ceph-mon[36187]: pgmap v3349: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T01:49:25.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:49:24 smithi193 ceph-mon[36878]: pgmap v3349: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T01:49:26.197 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:49:25 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 5061.686744 secs 2024-03-24T01:49:26.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:49:25 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 5061.686744 secs 2024-03-24T01:49:26.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:49:25 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 5061.686744 secs 2024-03-24T01:49:27.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:49:26 smithi118 ceph-mon[31535]: pgmap v3350: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 5 op/s 2024-03-24T01:49:27.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:49:26 smithi189 ceph-mon[36187]: pgmap v3350: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 5 op/s 2024-03-24T01:49:27.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:49:26 smithi193 ceph-mon[36878]: pgmap v3350: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 5 op/s 2024-03-24T01:49:29.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:49:28 smithi118 ceph-mon[31535]: pgmap v3351: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:49:29.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:49:28 smithi189 ceph-mon[36187]: pgmap v3351: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:49:29.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:49:28 smithi193 ceph-mon[36878]: pgmap v3351: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:49:31.197 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:49:30 smithi118 ceph-mon[31535]: pgmap v3352: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-24T01:49:31.197 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:49:30 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 5066.686784 secs 2024-03-24T01:49:31.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:49:30 smithi189 ceph-mon[36187]: pgmap v3352: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-24T01:49:31.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:49:30 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 5066.686784 secs 2024-03-24T01:49:31.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:49:30 smithi193 ceph-mon[36878]: pgmap v3352: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-24T01:49:31.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:49:30 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 5066.686784 secs 2024-03-24T01:49:33.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:49:32 smithi118 ceph-mon[31535]: pgmap v3353: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:49:33.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:49:32 smithi189 ceph-mon[36187]: pgmap v3353: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:49:33.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:49:32 smithi193 ceph-mon[36878]: pgmap v3353: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:49:35.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:49:34 smithi118 ceph-mon[31535]: pgmap v3354: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T01:49:35.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:49:34 smithi189 ceph-mon[36187]: pgmap v3354: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T01:49:35.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:49:34 smithi193 ceph-mon[36878]: pgmap v3354: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T01:49:36.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:49:35 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 5071.686873 secs 2024-03-24T01:49:36.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:49:35 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 5071.686873 secs 2024-03-24T01:49:36.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:49:35 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 5071.686873 secs 2024-03-24T01:49:37.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:49:36 smithi189 ceph-mon[36187]: pgmap v3355: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T01:49:37.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:49:36 smithi193 ceph-mon[36878]: pgmap v3355: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T01:49:37.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:49:36 smithi118 ceph-mon[31535]: pgmap v3355: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T01:49:38.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:49:37 smithi189 ceph-mon[36187]: pgmap v3356: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:49:38.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:49:37 smithi193 ceph-mon[36878]: pgmap v3356: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:49:38.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:49:37 smithi118 ceph-mon[31535]: pgmap v3356: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:49:40.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:49:40 smithi118 ceph-mon[31535]: pgmap v3357: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-24T01:49:40.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:49:40 smithi189 ceph-mon[36187]: pgmap v3357: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-24T01:49:40.462 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:49:40 smithi193 ceph-mon[36878]: pgmap v3357: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-24T01:49:41.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:49:41 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 5076.686956 secs 2024-03-24T01:49:41.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:49:41 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 5076.686956 secs 2024-03-24T01:49:41.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:49:41 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 5076.686956 secs 2024-03-24T01:49:42.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:49:42 smithi118 ceph-mon[31535]: pgmap v3358: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:49:42.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:49:42 smithi189 ceph-mon[36187]: pgmap v3358: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:49:42.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:49:42 smithi193 ceph-mon[36878]: pgmap v3358: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:49:44.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:49:44 smithi118 ceph-mon[31535]: pgmap v3359: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T01:49:44.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:49:44 smithi189 ceph-mon[36187]: pgmap v3359: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T01:49:44.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:49:44 smithi193 ceph-mon[36878]: pgmap v3359: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T01:49:46.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:49:46 smithi118 ceph-mon[31535]: pgmap v3360: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T01:49:46.447 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:49:46 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 5081.686998 secs 2024-03-24T01:49:46.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:49:46 smithi189 ceph-mon[36187]: pgmap v3360: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T01:49:46.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:49:46 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 5081.686998 secs 2024-03-24T01:49:46.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:49:46 smithi193 ceph-mon[36878]: pgmap v3360: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T01:49:46.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:49:46 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 5081.686998 secs 2024-03-24T01:49:48.447 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:49:48 smithi118 ceph-mon[31535]: pgmap v3361: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:49:48.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:49:48 smithi189 ceph-mon[36187]: pgmap v3361: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:49:48.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:49:48 smithi193 ceph-mon[36878]: pgmap v3361: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:49:50.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:49:50 smithi118 ceph-mon[31535]: pgmap v3362: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-24T01:49:50.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:49:50 smithi189 ceph-mon[36187]: pgmap v3362: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-24T01:49:50.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:49:50 smithi193 ceph-mon[36878]: pgmap v3362: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-24T01:49:51.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:49:51 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 5086.687086 secs 2024-03-24T01:49:51.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:49:51 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 5086.687086 secs 2024-03-24T01:49:51.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:49:51 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 5086.687086 secs 2024-03-24T01:49:52.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:49:52 smithi118 ceph-mon[31535]: pgmap v3363: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:49:52.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:49:52 smithi189 ceph-mon[36187]: pgmap v3363: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:49:52.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:49:52 smithi193 ceph-mon[36878]: pgmap v3363: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:49:54.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:49:54 smithi118 ceph-mon[31535]: pgmap v3364: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T01:49:54.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:49:54 smithi189 ceph-mon[36187]: pgmap v3364: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T01:49:54.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:49:54 smithi193 ceph-mon[36878]: pgmap v3364: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T01:49:56.447 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:49:56 smithi118 ceph-mon[31535]: pgmap v3365: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T01:49:56.447 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:49:56 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 5091.687148 secs 2024-03-24T01:49:56.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:49:56 smithi189 ceph-mon[36187]: pgmap v3365: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T01:49:56.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:49:56 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 5091.687148 secs 2024-03-24T01:49:56.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:49:56 smithi193 ceph-mon[36878]: pgmap v3365: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T01:49:56.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:49:56 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 5091.687148 secs 2024-03-24T01:49:58.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:49:58 smithi118 ceph-mon[31535]: pgmap v3366: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:49:58.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:49:58 smithi189 ceph-mon[36187]: pgmap v3366: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:49:58.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:49:58 smithi193 ceph-mon[36878]: pgmap v3366: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:50:00.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:50:00 smithi118 ceph-mon[31535]: pgmap v3367: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-24T01:50:00.447 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:50:00 smithi118 ceph-mon[31535]: overall HEALTH_WARN 6 failed cephadm daemon(s); 1 MDSs report slow requests 2024-03-24T01:50:00.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:50:00 smithi189 ceph-mon[36187]: pgmap v3367: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-24T01:50:00.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:50:00 smithi189 ceph-mon[36187]: overall HEALTH_WARN 6 failed cephadm daemon(s); 1 MDSs report slow requests 2024-03-24T01:50:00.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:50:00 smithi193 ceph-mon[36878]: pgmap v3367: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-24T01:50:00.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:50:00 smithi193 ceph-mon[36878]: overall HEALTH_WARN 6 failed cephadm daemon(s); 1 MDSs report slow requests 2024-03-24T01:50:01.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:50:01 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 5096.687209 secs 2024-03-24T01:50:01.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:50:01 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 5096.687209 secs 2024-03-24T01:50:01.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:50:01 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 5096.687209 secs 2024-03-24T01:50:02.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:50:02 smithi118 ceph-mon[31535]: pgmap v3368: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:50:02.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:50:02 smithi189 ceph-mon[36187]: pgmap v3368: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:50:02.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:50:02 smithi193 ceph-mon[36878]: pgmap v3368: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:50:04.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:50:04 smithi189 ceph-mon[36187]: pgmap v3369: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T01:50:04.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:50:04 smithi193 ceph-mon[36878]: pgmap v3369: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T01:50:04.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:50:04 smithi118 ceph-mon[31535]: pgmap v3369: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T01:50:06.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:50:06 smithi193 ceph-mon[36878]: pgmap v3370: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T01:50:06.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:50:06 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 5101.687291 secs 2024-03-24T01:50:06.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:50:06 smithi118 ceph-mon[31535]: pgmap v3370: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T01:50:06.697 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:50:06 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 5101.687291 secs 2024-03-24T01:50:06.704 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:50:06 smithi189 ceph-mon[36187]: pgmap v3370: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T01:50:06.704 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:50:06 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 5101.687291 secs 2024-03-24T01:50:08.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:50:08 smithi118 ceph-mon[31535]: pgmap v3371: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:50:08.704 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:50:08 smithi189 ceph-mon[36187]: pgmap v3371: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:50:08.711 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:50:08 smithi193 ceph-mon[36878]: pgmap v3371: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:50:10.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:50:10 smithi118 ceph-mon[31535]: pgmap v3372: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-24T01:50:10.704 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:50:10 smithi189 ceph-mon[36187]: pgmap v3372: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-24T01:50:10.711 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:50:10 smithi193 ceph-mon[36878]: pgmap v3372: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-24T01:50:11.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:50:11 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 5106.687355 secs 2024-03-24T01:50:11.704 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:50:11 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 5106.687355 secs 2024-03-24T01:50:11.711 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:50:11 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 5106.687355 secs 2024-03-24T01:50:12.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:50:12 smithi118 ceph-mon[31535]: pgmap v3373: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:50:12.704 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:50:12 smithi189 ceph-mon[36187]: pgmap v3373: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:50:12.711 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:50:12 smithi193 ceph-mon[36878]: pgmap v3373: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:50:14.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:50:14 smithi118 ceph-mon[31535]: pgmap v3374: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T01:50:14.570 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:50:14 smithi193 ceph-mon[36878]: pgmap v3374: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T01:50:14.704 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:50:14 smithi189 ceph-mon[36187]: pgmap v3374: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T01:50:16.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:50:16 smithi118 ceph-mon[31535]: pgmap v3375: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T01:50:16.697 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:50:16 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 5111.687446 secs 2024-03-24T01:50:16.697 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:50:16 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T01:50:16.697 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:50:16 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T01:50:16.697 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:50:16 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T01:50:16.704 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:50:16 smithi189 ceph-mon[36187]: pgmap v3375: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T01:50:16.704 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:50:16 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 5111.687446 secs 2024-03-24T01:50:16.704 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:50:16 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T01:50:16.704 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:50:16 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T01:50:16.704 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:50:16 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T01:50:16.711 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:50:16 smithi193 ceph-mon[36878]: pgmap v3375: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T01:50:16.711 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:50:16 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 5111.687446 secs 2024-03-24T01:50:16.711 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:50:16 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T01:50:16.711 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:50:16 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T01:50:16.711 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:50:16 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T01:50:17.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:50:17 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T01:50:17.697 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:50:17 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T01:50:17.697 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:50:17 smithi118 ceph-mon[31535]: pgmap v3376: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 5 op/s 2024-03-24T01:50:17.697 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:50:17 smithi118 ceph-mon[31535]: pgmap v3377: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 6 op/s 2024-03-24T01:50:17.697 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:50:17 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T01:50:17.704 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:50:17 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T01:50:17.704 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:50:17 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T01:50:17.704 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:50:17 smithi189 ceph-mon[36187]: pgmap v3376: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 5 op/s 2024-03-24T01:50:17.704 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:50:17 smithi189 ceph-mon[36187]: pgmap v3377: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 6 op/s 2024-03-24T01:50:17.704 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:50:17 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T01:50:17.711 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:50:17 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T01:50:17.711 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:50:17 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T01:50:17.711 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:50:17 smithi193 ceph-mon[36878]: pgmap v3376: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 5 op/s 2024-03-24T01:50:17.711 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:50:17 smithi193 ceph-mon[36878]: pgmap v3377: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 6 op/s 2024-03-24T01:50:17.711 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:50:17 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T01:50:20.197 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:50:19 smithi118 ceph-mon[31535]: pgmap v3378: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 4 op/s 2024-03-24T01:50:20.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:50:19 smithi189 ceph-mon[36187]: pgmap v3378: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 4 op/s 2024-03-24T01:50:20.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:50:19 smithi193 ceph-mon[36878]: pgmap v3378: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 4 op/s 2024-03-24T01:50:21.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:50:20 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 5116.687506 secs 2024-03-24T01:50:21.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:50:20 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 5116.687506 secs 2024-03-24T01:50:21.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:50:20 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 5116.687506 secs 2024-03-24T01:50:22.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:50:21 smithi118 ceph-mon[31535]: pgmap v3379: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 6 op/s 2024-03-24T01:50:22.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:50:21 smithi189 ceph-mon[36187]: pgmap v3379: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 6 op/s 2024-03-24T01:50:22.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:50:21 smithi193 ceph-mon[36878]: pgmap v3379: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 6 op/s 2024-03-24T01:50:24.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:50:23 smithi118 ceph-mon[31535]: pgmap v3380: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 5 op/s 2024-03-24T01:50:24.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:50:23 smithi189 ceph-mon[36187]: pgmap v3380: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 5 op/s 2024-03-24T01:50:24.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:50:23 smithi193 ceph-mon[36878]: pgmap v3380: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 5 op/s 2024-03-24T01:50:26.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:50:25 smithi118 ceph-mon[31535]: pgmap v3381: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 6 op/s 2024-03-24T01:50:26.197 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:50:25 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 5121.687593 secs 2024-03-24T01:50:26.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:50:25 smithi189 ceph-mon[36187]: pgmap v3381: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 6 op/s 2024-03-24T01:50:26.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:50:25 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 5121.687593 secs 2024-03-24T01:50:26.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:50:25 smithi193 ceph-mon[36878]: pgmap v3381: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 6 op/s 2024-03-24T01:50:26.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:50:25 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 5121.687593 secs 2024-03-24T01:50:28.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:50:27 smithi118 ceph-mon[31535]: pgmap v3382: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 5 op/s 2024-03-24T01:50:28.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:50:27 smithi189 ceph-mon[36187]: pgmap v3382: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 5 op/s 2024-03-24T01:50:28.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:50:27 smithi193 ceph-mon[36878]: pgmap v3382: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 5 op/s 2024-03-24T01:50:30.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:50:29 smithi118 ceph-mon[31535]: pgmap v3383: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-03-24T01:50:30.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:50:29 smithi189 ceph-mon[36187]: pgmap v3383: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-03-24T01:50:30.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:50:29 smithi193 ceph-mon[36878]: pgmap v3383: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-03-24T01:50:31.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:50:30 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 5126.687654 secs 2024-03-24T01:50:31.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:50:30 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 5126.687654 secs 2024-03-24T01:50:31.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:50:30 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 5126.687654 secs 2024-03-24T01:50:32.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:50:31 smithi118 ceph-mon[31535]: pgmap v3384: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 6 op/s 2024-03-24T01:50:32.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:50:31 smithi189 ceph-mon[36187]: pgmap v3384: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 6 op/s 2024-03-24T01:50:32.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:50:31 smithi193 ceph-mon[36878]: pgmap v3384: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 6 op/s 2024-03-24T01:50:34.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:50:33 smithi118 ceph-mon[31535]: pgmap v3385: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 5 op/s 2024-03-24T01:50:34.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:50:33 smithi189 ceph-mon[36187]: pgmap v3385: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 5 op/s 2024-03-24T01:50:34.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:50:33 smithi193 ceph-mon[36878]: pgmap v3385: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 5 op/s 2024-03-24T01:50:36.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:50:35 smithi118 ceph-mon[31535]: pgmap v3386: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-03-24T01:50:36.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:50:35 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 5131.687741 secs 2024-03-24T01:50:36.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:50:35 smithi189 ceph-mon[36187]: pgmap v3386: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-03-24T01:50:36.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:50:35 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 5131.687741 secs 2024-03-24T01:50:36.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:50:35 smithi193 ceph-mon[36878]: pgmap v3386: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-03-24T01:50:36.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:50:35 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 5131.687741 secs 2024-03-24T01:50:38.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:50:37 smithi118 ceph-mon[31535]: pgmap v3387: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-03-24T01:50:38.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:50:37 smithi189 ceph-mon[36187]: pgmap v3387: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-03-24T01:50:38.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:50:37 smithi193 ceph-mon[36878]: pgmap v3387: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-03-24T01:50:40.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:50:39 smithi118 ceph-mon[31535]: pgmap v3388: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-03-24T01:50:40.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:50:39 smithi189 ceph-mon[36187]: pgmap v3388: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-03-24T01:50:40.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:50:39 smithi193 ceph-mon[36878]: pgmap v3388: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-03-24T01:50:41.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:50:40 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 5136.687805 secs 2024-03-24T01:50:41.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:50:40 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 5136.687805 secs 2024-03-24T01:50:41.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:50:40 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 5136.687805 secs 2024-03-24T01:50:42.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:50:41 smithi118 ceph-mon[31535]: pgmap v3389: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T01:50:42.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:50:41 smithi189 ceph-mon[36187]: pgmap v3389: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T01:50:42.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:50:41 smithi193 ceph-mon[36878]: pgmap v3389: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T01:50:44.197 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:50:43 smithi118 ceph-mon[31535]: pgmap v3390: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T01:50:44.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:50:43 smithi189 ceph-mon[36187]: pgmap v3390: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T01:50:44.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:50:43 smithi193 ceph-mon[36878]: pgmap v3390: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T01:50:46.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:50:45 smithi118 ceph-mon[31535]: pgmap v3391: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-03-24T01:50:46.197 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:50:45 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 5141.687892 secs 2024-03-24T01:50:46.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:50:45 smithi189 ceph-mon[36187]: pgmap v3391: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-03-24T01:50:46.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:50:45 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 5141.687892 secs 2024-03-24T01:50:46.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:50:45 smithi193 ceph-mon[36878]: pgmap v3391: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-03-24T01:50:46.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:50:45 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 5141.687892 secs 2024-03-24T01:50:48.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:50:47 smithi118 ceph-mon[31535]: pgmap v3392: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:50:48.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:50:47 smithi189 ceph-mon[36187]: pgmap v3392: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:50:48.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:50:47 smithi193 ceph-mon[36878]: pgmap v3392: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:50:50.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:50:49 smithi118 ceph-mon[31535]: pgmap v3393: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T01:50:50.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:50:49 smithi189 ceph-mon[36187]: pgmap v3393: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T01:50:50.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:50:49 smithi193 ceph-mon[36878]: pgmap v3393: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T01:50:51.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:50:50 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 5146.687945 secs 2024-03-24T01:50:51.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:50:50 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 5146.687945 secs 2024-03-24T01:50:51.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:50:50 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 5146.687945 secs 2024-03-24T01:50:52.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:50:51 smithi118 ceph-mon[31535]: pgmap v3394: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T01:50:52.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:50:51 smithi189 ceph-mon[36187]: pgmap v3394: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T01:50:52.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:50:51 smithi193 ceph-mon[36878]: pgmap v3394: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T01:50:54.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:50:53 smithi189 ceph-mon[36187]: pgmap v3395: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T01:50:54.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:50:53 smithi193 ceph-mon[36878]: pgmap v3395: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T01:50:54.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:50:53 smithi118 ceph-mon[31535]: pgmap v3395: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T01:50:56.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:50:55 smithi189 ceph-mon[36187]: pgmap v3396: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-03-24T01:50:56.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:50:55 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 5151.688033 secs 2024-03-24T01:50:56.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:50:55 smithi193 ceph-mon[36878]: pgmap v3396: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-03-24T01:50:56.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:50:55 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 5151.688033 secs 2024-03-24T01:50:56.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:50:55 smithi118 ceph-mon[31535]: pgmap v3396: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-03-24T01:50:56.447 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:50:55 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 5151.688033 secs 2024-03-24T01:50:58.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:50:57 smithi193 ceph-mon[36878]: pgmap v3397: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:50:58.249 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:50:57 smithi118 ceph-mon[31535]: pgmap v3397: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:50:58.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:50:57 smithi189 ceph-mon[36187]: pgmap v3397: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:51:00.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:50:59 smithi118 ceph-mon[31535]: pgmap v3398: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T01:51:00.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:50:59 smithi189 ceph-mon[36187]: pgmap v3398: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T01:51:00.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:50:59 smithi193 ceph-mon[36878]: pgmap v3398: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T01:51:01.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:51:00 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 5156.688116 secs 2024-03-24T01:51:01.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:51:00 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 5156.688116 secs 2024-03-24T01:51:01.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:51:00 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 5156.688116 secs 2024-03-24T01:51:02.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:51:01 smithi118 ceph-mon[31535]: pgmap v3399: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T01:51:02.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:51:01 smithi189 ceph-mon[36187]: pgmap v3399: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T01:51:02.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:51:01 smithi193 ceph-mon[36878]: pgmap v3399: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T01:51:03.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:51:02 smithi118 ceph-mon[31535]: pgmap v3400: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T01:51:03.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:51:02 smithi189 ceph-mon[36187]: pgmap v3400: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T01:51:03.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:51:02 smithi193 ceph-mon[36878]: pgmap v3400: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T01:51:06.156 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:51:05 smithi189 ceph-mon[36187]: pgmap v3401: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-03-24T01:51:06.156 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:51:05 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 5161.688205 secs 2024-03-24T01:51:06.162 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:51:05 smithi193 ceph-mon[36878]: pgmap v3401: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-03-24T01:51:06.162 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:51:05 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 5161.688205 secs 2024-03-24T01:51:06.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:51:05 smithi118 ceph-mon[31535]: pgmap v3401: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-03-24T01:51:06.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:51:05 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 5161.688205 secs 2024-03-24T01:51:08.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:51:07 smithi118 ceph-mon[31535]: pgmap v3402: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:51:08.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:51:07 smithi189 ceph-mon[36187]: pgmap v3402: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:51:08.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:51:07 smithi193 ceph-mon[36878]: pgmap v3402: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:51:10.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:51:09 smithi118 ceph-mon[31535]: pgmap v3403: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T01:51:10.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:51:09 smithi189 ceph-mon[36187]: pgmap v3403: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T01:51:10.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:51:09 smithi193 ceph-mon[36878]: pgmap v3403: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T01:51:11.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:51:10 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 5166.688267 secs 2024-03-24T01:51:11.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:51:10 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 5166.688267 secs 2024-03-24T01:51:11.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:51:10 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 5166.688267 secs 2024-03-24T01:51:12.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:51:11 smithi118 ceph-mon[31535]: pgmap v3404: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T01:51:12.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:51:11 smithi189 ceph-mon[36187]: pgmap v3404: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T01:51:12.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:51:11 smithi193 ceph-mon[36878]: pgmap v3404: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T01:51:14.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:51:13 smithi118 ceph-mon[31535]: pgmap v3405: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T01:51:14.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:51:13 smithi189 ceph-mon[36187]: pgmap v3405: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T01:51:14.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:51:13 smithi193 ceph-mon[36878]: pgmap v3405: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T01:51:16.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:51:15 smithi118 ceph-mon[31535]: pgmap v3406: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-03-24T01:51:16.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:51:15 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 5171.688355 secs 2024-03-24T01:51:16.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:51:15 smithi189 ceph-mon[36187]: pgmap v3406: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-03-24T01:51:16.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:51:15 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 5171.688355 secs 2024-03-24T01:51:16.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:51:15 smithi193 ceph-mon[36878]: pgmap v3406: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-03-24T01:51:16.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:51:15 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 5171.688355 secs 2024-03-24T01:51:18.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:51:17 smithi118 ceph-mon[31535]: pgmap v3407: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:51:18.197 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:51:17 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T01:51:18.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:51:17 smithi189 ceph-mon[36187]: pgmap v3407: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:51:18.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:51:17 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T01:51:18.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:51:17 smithi193 ceph-mon[36878]: pgmap v3407: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:51:18.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:51:17 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T01:51:19.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:51:18 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T01:51:19.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:51:18 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T01:51:19.197 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:51:18 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T01:51:19.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:51:18 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T01:51:19.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:51:18 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T01:51:19.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:51:18 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T01:51:19.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:51:18 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T01:51:19.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:51:18 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T01:51:19.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:51:18 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T01:51:20.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:51:19 smithi118 ceph-mon[31535]: pgmap v3408: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 6 op/s 2024-03-24T01:51:20.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:51:19 smithi118 ceph-mon[31535]: pgmap v3409: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 6 op/s 2024-03-24T01:51:20.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:51:19 smithi189 ceph-mon[36187]: pgmap v3408: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 6 op/s 2024-03-24T01:51:20.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:51:19 smithi189 ceph-mon[36187]: pgmap v3409: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 6 op/s 2024-03-24T01:51:20.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:51:19 smithi193 ceph-mon[36878]: pgmap v3408: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 6 op/s 2024-03-24T01:51:20.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:51:19 smithi193 ceph-mon[36878]: pgmap v3409: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 6 op/s 2024-03-24T01:51:21.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:51:20 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 5176.688441 secs 2024-03-24T01:51:21.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:51:20 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 5176.688441 secs 2024-03-24T01:51:21.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:51:20 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 5176.688441 secs 2024-03-24T01:51:22.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:51:21 smithi118 ceph-mon[31535]: pgmap v3410: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 6 op/s 2024-03-24T01:51:22.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:51:21 smithi189 ceph-mon[36187]: pgmap v3410: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 6 op/s 2024-03-24T01:51:22.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:51:21 smithi193 ceph-mon[36878]: pgmap v3410: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 6 op/s 2024-03-24T01:51:24.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:51:23 smithi118 ceph-mon[31535]: pgmap v3411: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 5 op/s 2024-03-24T01:51:24.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:51:23 smithi189 ceph-mon[36187]: pgmap v3411: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 5 op/s 2024-03-24T01:51:24.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:51:23 smithi193 ceph-mon[36878]: pgmap v3411: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 5 op/s 2024-03-24T01:51:26.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:51:25 smithi118 ceph-mon[31535]: pgmap v3412: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 6 op/s 2024-03-24T01:51:26.197 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:51:25 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 5181.688501 secs 2024-03-24T01:51:26.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:51:25 smithi189 ceph-mon[36187]: pgmap v3412: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 6 op/s 2024-03-24T01:51:26.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:51:25 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 5181.688501 secs 2024-03-24T01:51:26.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:51:25 smithi193 ceph-mon[36878]: pgmap v3412: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 6 op/s 2024-03-24T01:51:26.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:51:25 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 5181.688501 secs 2024-03-24T01:51:28.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:51:27 smithi118 ceph-mon[31535]: pgmap v3413: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 6 op/s 2024-03-24T01:51:28.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:51:27 smithi189 ceph-mon[36187]: pgmap v3413: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 6 op/s 2024-03-24T01:51:28.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:51:27 smithi193 ceph-mon[36878]: pgmap v3413: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 6 op/s 2024-03-24T01:51:30.197 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:51:29 smithi118 ceph-mon[31535]: pgmap v3414: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 5 op/s 2024-03-24T01:51:30.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:51:29 smithi189 ceph-mon[36187]: pgmap v3414: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 5 op/s 2024-03-24T01:51:30.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:51:29 smithi193 ceph-mon[36878]: pgmap v3414: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 5 op/s 2024-03-24T01:51:31.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:51:30 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 5186.688588 secs 2024-03-24T01:51:31.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:51:30 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 5186.688588 secs 2024-03-24T01:51:31.352 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:51:30 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 5186.688588 secs 2024-03-24T01:51:32.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:51:31 smithi193 ceph-mon[36878]: pgmap v3415: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T01:51:32.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:51:31 smithi118 ceph-mon[31535]: pgmap v3415: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T01:51:32.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:51:31 smithi189 ceph-mon[36187]: pgmap v3415: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T01:51:34.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:51:33 smithi193 ceph-mon[36878]: pgmap v3416: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:51:34.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:51:33 smithi118 ceph-mon[31535]: pgmap v3416: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:51:34.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:51:33 smithi189 ceph-mon[36187]: pgmap v3416: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:51:36.256 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:51:35 smithi189 ceph-mon[36187]: pgmap v3417: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-24T01:51:36.256 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:51:35 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 5191.688677 secs 2024-03-24T01:51:36.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:51:35 smithi118 ceph-mon[31535]: pgmap v3417: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-24T01:51:36.447 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:51:35 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 5191.688677 secs 2024-03-24T01:51:36.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:51:35 smithi193 ceph-mon[36878]: pgmap v3417: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-24T01:51:36.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:51:35 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 5191.688677 secs 2024-03-24T01:51:38.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:51:37 smithi118 ceph-mon[31535]: pgmap v3418: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:51:38.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:51:37 smithi189 ceph-mon[36187]: pgmap v3418: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:51:38.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:51:37 smithi193 ceph-mon[36878]: pgmap v3418: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:51:40.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:51:39 smithi118 ceph-mon[31535]: pgmap v3419: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T01:51:40.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:51:39 smithi189 ceph-mon[36187]: pgmap v3419: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T01:51:40.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:51:39 smithi193 ceph-mon[36878]: pgmap v3419: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T01:51:41.003 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:51:40 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 5196.688766 secs 2024-03-24T01:51:41.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:51:40 smithi118 ceph-mon[31535]: pgmap v3420: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T01:51:41.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:51:40 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 5196.688766 secs 2024-03-24T01:51:41.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:51:40 smithi189 ceph-mon[36187]: pgmap v3420: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T01:51:41.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:51:40 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 5196.688766 secs 2024-03-24T01:51:41.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:51:40 smithi193 ceph-mon[36878]: pgmap v3420: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T01:51:43.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:51:43 smithi118 ceph-mon[31535]: pgmap v3421: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:51:43.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:51:43 smithi189 ceph-mon[36187]: pgmap v3421: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:51:43.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:51:43 smithi193 ceph-mon[36878]: pgmap v3421: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:51:45.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:51:45 smithi118 ceph-mon[31535]: pgmap v3422: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-24T01:51:45.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:51:45 smithi189 ceph-mon[36187]: pgmap v3422: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-24T01:51:45.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:51:45 smithi193 ceph-mon[36878]: pgmap v3422: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-24T01:51:46.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:51:46 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 5201.688781 secs 2024-03-24T01:51:46.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:51:46 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 5201.688781 secs 2024-03-24T01:51:46.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:51:46 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 5201.688781 secs 2024-03-24T01:51:47.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:51:47 smithi118 ceph-mon[31535]: pgmap v3423: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:51:47.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:51:47 smithi189 ceph-mon[36187]: pgmap v3423: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:51:47.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:51:47 smithi193 ceph-mon[36878]: pgmap v3423: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:51:49.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:51:49 smithi118 ceph-mon[31535]: pgmap v3424: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T01:51:49.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:51:49 smithi189 ceph-mon[36187]: pgmap v3424: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T01:51:49.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:51:49 smithi193 ceph-mon[36878]: pgmap v3424: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T01:51:51.947 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:51:51 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 5206.688845 secs 2024-03-24T01:51:51.947 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:51:51 smithi118 ceph-mon[31535]: pgmap v3425: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T01:51:51.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:51:51 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 5206.688845 secs 2024-03-24T01:51:51.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:51:51 smithi189 ceph-mon[36187]: pgmap v3425: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T01:51:51.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:51:51 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 5206.688845 secs 2024-03-24T01:51:51.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:51:51 smithi193 ceph-mon[36878]: pgmap v3425: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T01:51:53.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:51:53 smithi118 ceph-mon[31535]: pgmap v3426: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:51:53.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:51:53 smithi189 ceph-mon[36187]: pgmap v3426: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:51:53.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:51:53 smithi193 ceph-mon[36878]: pgmap v3426: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:51:55.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:51:55 smithi118 ceph-mon[31535]: pgmap v3427: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-24T01:51:55.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:51:55 smithi189 ceph-mon[36187]: pgmap v3427: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-24T01:51:55.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:51:55 smithi193 ceph-mon[36878]: pgmap v3427: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-24T01:51:56.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:51:56 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 5211.688928 secs 2024-03-24T01:51:56.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:51:56 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 5211.688928 secs 2024-03-24T01:51:56.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:51:56 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 5211.688928 secs 2024-03-24T01:51:57.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:51:57 smithi118 ceph-mon[31535]: pgmap v3428: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:51:57.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:51:57 smithi189 ceph-mon[36187]: pgmap v3428: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:51:57.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:51:57 smithi193 ceph-mon[36878]: pgmap v3428: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:51:59.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:51:59 smithi118 ceph-mon[31535]: pgmap v3429: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T01:51:59.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:51:59 smithi189 ceph-mon[36187]: pgmap v3429: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T01:51:59.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:51:59 smithi193 ceph-mon[36878]: pgmap v3429: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T01:52:01.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:52:01 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 5216.689016 secs 2024-03-24T01:52:01.947 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:52:01 smithi118 ceph-mon[31535]: pgmap v3430: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T01:52:01.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:52:01 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 5216.689016 secs 2024-03-24T01:52:01.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:52:01 smithi189 ceph-mon[36187]: pgmap v3430: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T01:52:01.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:52:01 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 5216.689016 secs 2024-03-24T01:52:01.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:52:01 smithi193 ceph-mon[36878]: pgmap v3430: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T01:52:03.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:52:03 smithi118 ceph-mon[31535]: pgmap v3431: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:52:03.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:52:03 smithi189 ceph-mon[36187]: pgmap v3431: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:52:03.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:52:03 smithi193 ceph-mon[36878]: pgmap v3431: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:52:06.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:52:05 smithi118 ceph-mon[31535]: pgmap v3432: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-24T01:52:06.218 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:52:05 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 5221.689112 secs 2024-03-24T01:52:06.218 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:52:05 smithi193 ceph-mon[36878]: pgmap v3432: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-24T01:52:06.219 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:52:05 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 5221.689112 secs 2024-03-24T01:52:06.219 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:52:05 smithi189 ceph-mon[36187]: pgmap v3432: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-24T01:52:06.219 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:52:05 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 5221.689112 secs 2024-03-24T01:52:08.006 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:52:07 smithi189 ceph-mon[36187]: pgmap v3433: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:52:08.013 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:52:07 smithi193 ceph-mon[36878]: pgmap v3433: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:52:08.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:52:07 smithi118 ceph-mon[31535]: pgmap v3433: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:52:10.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:52:09 smithi118 ceph-mon[31535]: pgmap v3434: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T01:52:10.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:52:09 smithi189 ceph-mon[36187]: pgmap v3434: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T01:52:10.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:52:09 smithi193 ceph-mon[36878]: pgmap v3434: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T01:52:11.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:52:10 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 5226.689203 secs 2024-03-24T01:52:11.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:52:10 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 5226.689203 secs 2024-03-24T01:52:11.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:52:10 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 5226.689203 secs 2024-03-24T01:52:12.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:52:11 smithi118 ceph-mon[31535]: pgmap v3435: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T01:52:12.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:52:11 smithi189 ceph-mon[36187]: pgmap v3435: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T01:52:12.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:52:11 smithi193 ceph-mon[36878]: pgmap v3435: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T01:52:14.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:52:13 smithi118 ceph-mon[31535]: pgmap v3436: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:52:14.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:52:13 smithi189 ceph-mon[36187]: pgmap v3436: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:52:14.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:52:13 smithi193 ceph-mon[36878]: pgmap v3436: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:52:16.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:52:15 smithi118 ceph-mon[31535]: pgmap v3437: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-24T01:52:16.197 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:52:15 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 5231.689286 secs 2024-03-24T01:52:16.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:52:15 smithi189 ceph-mon[36187]: pgmap v3437: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-24T01:52:16.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:52:15 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 5231.689286 secs 2024-03-24T01:52:16.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:52:15 smithi193 ceph-mon[36878]: pgmap v3437: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-24T01:52:16.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:52:15 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 5231.689286 secs 2024-03-24T01:52:16.572 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T01:52:16.571+0000 7f81ee000640 1 -- 172.21.15.118:0/1693767697 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f81c80036c0 con 0x7f81f81052d0 2024-03-24T01:52:16.572 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T01:52:16.571+0000 7f81db400640 1 -- 172.21.15.118:0/659686424 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f81c4006460 con 0x5622651a13f0 2024-03-24T01:52:16.572 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T01:52:16.572+0000 7f81ef400640 1 -- 172.21.15.118:0/1693767697 <== mon.2 v2:172.21.15.189:3300/0 44 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (secure 0 0 0) 0x7f81c80036c0 con 0x7f81f81052d0 2024-03-24T01:52:16.572 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T01:52:16.572+0000 7f81e4c00640 1 -- 172.21.15.118:0/659686424 <== mon.1 v2:172.21.15.193:3300/0 44 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 409+0+0 (secure 0 0 0) 0x7f81c4006460 con 0x5622651a13f0 2024-03-24T01:52:18.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:52:17 smithi118 ceph-mon[31535]: pgmap v3438: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:52:18.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:52:17 smithi189 ceph-mon[36187]: pgmap v3438: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:52:18.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:52:17 smithi193 ceph-mon[36878]: pgmap v3438: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:52:19.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:52:18 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T01:52:19.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:52:18 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T01:52:19.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:52:18 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T01:52:20.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:52:19 smithi118 ceph-mon[31535]: pgmap v3439: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T01:52:20.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:52:19 smithi189 ceph-mon[36187]: pgmap v3439: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T01:52:20.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:52:19 smithi193 ceph-mon[36878]: pgmap v3439: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T01:52:21.197 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:52:20 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T01:52:21.197 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:52:20 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T01:52:21.197 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:52:20 smithi118 ceph-mon[31535]: pgmap v3440: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 6 op/s 2024-03-24T01:52:21.197 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:52:20 smithi118 ceph-mon[31535]: pgmap v3441: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-03-24T01:52:21.197 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:52:20 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T01:52:21.197 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:52:20 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 5236.689377 secs 2024-03-24T01:52:21.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:52:20 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T01:52:21.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:52:20 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T01:52:21.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:52:20 smithi189 ceph-mon[36187]: pgmap v3440: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 6 op/s 2024-03-24T01:52:21.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:52:20 smithi189 ceph-mon[36187]: pgmap v3441: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-03-24T01:52:21.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:52:20 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T01:52:21.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:52:20 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 5236.689377 secs 2024-03-24T01:52:21.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:52:20 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T01:52:21.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:52:20 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T01:52:21.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:52:20 smithi193 ceph-mon[36878]: pgmap v3440: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 6 op/s 2024-03-24T01:52:21.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:52:20 smithi193 ceph-mon[36878]: pgmap v3441: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-03-24T01:52:21.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:52:20 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T01:52:21.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:52:20 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 5236.689377 secs 2024-03-24T01:52:23.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:52:22 smithi118 ceph-mon[31535]: pgmap v3442: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-03-24T01:52:23.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:52:22 smithi189 ceph-mon[36187]: pgmap v3442: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-03-24T01:52:23.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:52:22 smithi193 ceph-mon[36878]: pgmap v3442: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-03-24T01:52:24.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:52:23 smithi118 ceph-mon[31535]: pgmap v3443: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 5 op/s 2024-03-24T01:52:24.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:52:23 smithi189 ceph-mon[36187]: pgmap v3443: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 5 op/s 2024-03-24T01:52:24.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:52:23 smithi193 ceph-mon[36878]: pgmap v3443: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 5 op/s 2024-03-24T01:52:26.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:52:25 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 5241.689463 secs 2024-03-24T01:52:26.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:52:25 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 5241.689463 secs 2024-03-24T01:52:26.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:52:25 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 5241.689463 secs 2024-03-24T01:52:27.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:52:26 smithi118 ceph-mon[31535]: pgmap v3444: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-03-24T01:52:27.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:52:26 smithi189 ceph-mon[36187]: pgmap v3444: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-03-24T01:52:27.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:52:26 smithi193 ceph-mon[36878]: pgmap v3444: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-03-24T01:52:28.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:52:28 smithi118 ceph-mon[31535]: pgmap v3445: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 5 op/s 2024-03-24T01:52:28.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:52:28 smithi189 ceph-mon[36187]: pgmap v3445: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 5 op/s 2024-03-24T01:52:28.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:52:28 smithi193 ceph-mon[36878]: pgmap v3445: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 5 op/s 2024-03-24T01:52:30.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:52:30 smithi118 ceph-mon[31535]: pgmap v3446: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 5 op/s 2024-03-24T01:52:30.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:52:30 smithi189 ceph-mon[36187]: pgmap v3446: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 5 op/s 2024-03-24T01:52:30.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:52:30 smithi193 ceph-mon[36878]: pgmap v3446: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 5 op/s 2024-03-24T01:52:31.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:52:31 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 5246.689551 secs 2024-03-24T01:52:31.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:52:31 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 5246.689551 secs 2024-03-24T01:52:31.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:52:31 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 5246.689551 secs 2024-03-24T01:52:32.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:52:32 smithi118 ceph-mon[31535]: pgmap v3447: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:52:32.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:52:32 smithi189 ceph-mon[36187]: pgmap v3447: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:52:32.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:52:32 smithi193 ceph-mon[36878]: pgmap v3447: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:52:34.329 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:52:34 smithi118 ceph-mon[31535]: pgmap v3448: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T01:52:34.457 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:52:34 smithi189 ceph-mon[36187]: pgmap v3448: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T01:52:34.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:52:34 smithi193 ceph-mon[36878]: pgmap v3448: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T01:52:36.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:52:35 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 5251.689611 secs 2024-03-24T01:52:36.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:52:35 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 5251.689611 secs 2024-03-24T01:52:36.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:52:35 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 5251.689611 secs 2024-03-24T01:52:37.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:52:36 smithi118 ceph-mon[31535]: pgmap v3449: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T01:52:37.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:52:36 smithi189 ceph-mon[36187]: pgmap v3449: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T01:52:37.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:52:36 smithi193 ceph-mon[36878]: pgmap v3449: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T01:52:38.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:52:38 smithi118 ceph-mon[31535]: pgmap v3450: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 5 op/s 2024-03-24T01:52:38.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:52:38 smithi189 ceph-mon[36187]: pgmap v3450: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 5 op/s 2024-03-24T01:52:38.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:52:38 smithi193 ceph-mon[36878]: pgmap v3450: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 5 op/s 2024-03-24T01:52:40.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:52:40 smithi118 ceph-mon[31535]: pgmap v3451: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T01:52:40.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:52:40 smithi189 ceph-mon[36187]: pgmap v3451: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T01:52:40.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:52:40 smithi193 ceph-mon[36878]: pgmap v3451: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T01:52:41.447 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:52:41 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 5256.689701 secs 2024-03-24T01:52:41.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:52:41 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 5256.689701 secs 2024-03-24T01:52:41.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:52:41 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 5256.689701 secs 2024-03-24T01:52:42.447 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:52:42 smithi118 ceph-mon[31535]: pgmap v3452: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:52:42.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:52:42 smithi189 ceph-mon[36187]: pgmap v3452: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:52:42.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:52:42 smithi193 ceph-mon[36878]: pgmap v3452: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:52:44.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:52:44 smithi118 ceph-mon[31535]: pgmap v3453: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T01:52:44.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:52:44 smithi189 ceph-mon[36187]: pgmap v3453: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T01:52:44.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:52:44 smithi193 ceph-mon[36878]: pgmap v3453: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T01:52:46.105 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:52:45 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 5261.689785 secs 2024-03-24T01:52:46.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:52:45 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 5261.689785 secs 2024-03-24T01:52:46.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:52:45 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 5261.689785 secs 2024-03-24T01:52:47.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:52:46 smithi118 ceph-mon[31535]: pgmap v3454: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T01:52:47.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:52:46 smithi189 ceph-mon[36187]: pgmap v3454: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T01:52:47.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:52:46 smithi193 ceph-mon[36878]: pgmap v3454: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T01:52:48.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:52:48 smithi118 ceph-mon[31535]: pgmap v3455: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 5 op/s 2024-03-24T01:52:48.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:52:48 smithi189 ceph-mon[36187]: pgmap v3455: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 5 op/s 2024-03-24T01:52:48.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:52:48 smithi193 ceph-mon[36878]: pgmap v3455: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 5 op/s 2024-03-24T01:52:50.447 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:52:50 smithi118 ceph-mon[31535]: pgmap v3456: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T01:52:50.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:52:50 smithi189 ceph-mon[36187]: pgmap v3456: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T01:52:50.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:52:50 smithi193 ceph-mon[36878]: pgmap v3456: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T01:52:51.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:52:51 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 5266.689870 secs 2024-03-24T01:52:51.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:52:51 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 5266.689870 secs 2024-03-24T01:52:51.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:52:51 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 5266.689870 secs 2024-03-24T01:52:52.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:52:52 smithi118 ceph-mon[31535]: pgmap v3457: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:52:52.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:52:52 smithi189 ceph-mon[36187]: pgmap v3457: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:52:52.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:52:52 smithi193 ceph-mon[36878]: pgmap v3457: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:52:54.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:52:54 smithi118 ceph-mon[31535]: pgmap v3458: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T01:52:54.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:52:54 smithi189 ceph-mon[36187]: pgmap v3458: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T01:52:54.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:52:54 smithi193 ceph-mon[36878]: pgmap v3458: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T01:52:56.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:52:55 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 5271.689956 secs 2024-03-24T01:52:56.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:52:55 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 5271.689956 secs 2024-03-24T01:52:56.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:52:55 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 5271.689956 secs 2024-03-24T01:52:57.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:52:56 smithi118 ceph-mon[31535]: pgmap v3459: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T01:52:57.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:52:56 smithi189 ceph-mon[36187]: pgmap v3459: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T01:52:57.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:52:56 smithi193 ceph-mon[36878]: pgmap v3459: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T01:52:58.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:52:58 smithi118 ceph-mon[31535]: pgmap v3460: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 5 op/s 2024-03-24T01:52:58.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:52:58 smithi189 ceph-mon[36187]: pgmap v3460: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 5 op/s 2024-03-24T01:52:58.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:52:58 smithi193 ceph-mon[36878]: pgmap v3460: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 5 op/s 2024-03-24T01:53:00.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:53:00 smithi118 ceph-mon[31535]: pgmap v3461: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T01:53:00.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:53:00 smithi189 ceph-mon[36187]: pgmap v3461: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T01:53:00.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:53:00 smithi193 ceph-mon[36878]: pgmap v3461: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T01:53:01.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:53:01 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 5276.690048 secs 2024-03-24T01:53:01.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:53:01 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 5276.690048 secs 2024-03-24T01:53:01.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:53:01 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 5276.690048 secs 2024-03-24T01:53:02.348 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:53:02 smithi118 ceph-mon[31535]: pgmap v3462: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:53:02.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:53:02 smithi189 ceph-mon[36187]: pgmap v3462: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:53:02.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:53:02 smithi193 ceph-mon[36878]: pgmap v3462: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:53:04.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:53:04 smithi118 ceph-mon[31535]: pgmap v3463: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T01:53:04.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:53:04 smithi189 ceph-mon[36187]: pgmap v3463: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T01:53:04.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:53:04 smithi193 ceph-mon[36878]: pgmap v3463: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T01:53:06.073 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:53:05 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 5281.690084 secs 2024-03-24T01:53:06.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:53:05 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 5281.690084 secs 2024-03-24T01:53:06.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:53:05 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 5281.690084 secs 2024-03-24T01:53:07.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:53:06 smithi118 ceph-mon[31535]: pgmap v3464: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T01:53:07.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:53:06 smithi189 ceph-mon[36187]: pgmap v3464: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T01:53:07.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:53:06 smithi193 ceph-mon[36878]: pgmap v3464: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T01:53:08.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:53:08 smithi118 ceph-mon[31535]: pgmap v3465: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 5 op/s 2024-03-24T01:53:08.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:53:08 smithi189 ceph-mon[36187]: pgmap v3465: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 5 op/s 2024-03-24T01:53:08.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:53:08 smithi193 ceph-mon[36878]: pgmap v3465: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 5 op/s 2024-03-24T01:53:10.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:53:10 smithi118 ceph-mon[31535]: pgmap v3466: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T01:53:10.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:53:10 smithi189 ceph-mon[36187]: pgmap v3466: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T01:53:10.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:53:10 smithi193 ceph-mon[36878]: pgmap v3466: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T01:53:11.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:53:11 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 5286.690121 secs 2024-03-24T01:53:11.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:53:11 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 5286.690121 secs 2024-03-24T01:53:11.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:53:11 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 5286.690121 secs 2024-03-24T01:53:12.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:53:12 smithi118 ceph-mon[31535]: pgmap v3467: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:53:12.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:53:12 smithi189 ceph-mon[36187]: pgmap v3467: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:53:12.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:53:12 smithi193 ceph-mon[36878]: pgmap v3467: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:53:14.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:53:14 smithi118 ceph-mon[31535]: pgmap v3468: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T01:53:14.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:53:14 smithi189 ceph-mon[36187]: pgmap v3468: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T01:53:14.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:53:14 smithi193 ceph-mon[36878]: pgmap v3468: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T01:53:16.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:53:15 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 5291.690161 secs 2024-03-24T01:53:16.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:53:15 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 5291.690161 secs 2024-03-24T01:53:16.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:53:15 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 5291.690161 secs 2024-03-24T01:53:17.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:53:16 smithi118 ceph-mon[31535]: pgmap v3469: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T01:53:17.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:53:16 smithi189 ceph-mon[36187]: pgmap v3469: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T01:53:17.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:53:16 smithi193 ceph-mon[36878]: pgmap v3469: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T01:53:18.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:53:18 smithi118 ceph-mon[31535]: pgmap v3470: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 5 op/s 2024-03-24T01:53:18.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:53:18 smithi189 ceph-mon[36187]: pgmap v3470: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 5 op/s 2024-03-24T01:53:18.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:53:18 smithi193 ceph-mon[36878]: pgmap v3470: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 5 op/s 2024-03-24T01:53:21.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:53:20 smithi118 ceph-mon[31535]: pgmap v3471: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T01:53:21.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:53:20 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T01:53:21.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:53:20 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 5296.690227 secs 2024-03-24T01:53:21.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:53:20 smithi189 ceph-mon[36187]: pgmap v3471: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T01:53:21.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:53:20 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T01:53:21.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:53:20 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 5296.690227 secs 2024-03-24T01:53:21.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:53:20 smithi193 ceph-mon[36878]: pgmap v3471: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T01:53:21.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:53:20 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T01:53:21.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:53:20 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 5296.690227 secs 2024-03-24T01:53:22.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:53:21 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T01:53:22.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:53:21 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T01:53:22.197 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:53:21 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T01:53:22.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:53:21 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T01:53:22.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:53:21 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T01:53:22.205 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:53:21 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T01:53:22.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:53:21 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T01:53:22.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:53:21 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T01:53:22.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:53:21 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T01:53:23.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:53:22 smithi118 ceph-mon[31535]: pgmap v3472: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 5 op/s 2024-03-24T01:53:23.197 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:53:22 smithi118 ceph-mon[31535]: pgmap v3473: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 6 op/s 2024-03-24T01:53:23.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:53:22 smithi189 ceph-mon[36187]: pgmap v3472: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 5 op/s 2024-03-24T01:53:23.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:53:22 smithi189 ceph-mon[36187]: pgmap v3473: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 6 op/s 2024-03-24T01:53:23.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:53:22 smithi193 ceph-mon[36878]: pgmap v3472: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 5 op/s 2024-03-24T01:53:23.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:53:22 smithi193 ceph-mon[36878]: pgmap v3473: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 6 op/s 2024-03-24T01:53:25.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:53:24 smithi118 ceph-mon[31535]: pgmap v3474: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 5 op/s 2024-03-24T01:53:25.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:53:24 smithi189 ceph-mon[36187]: pgmap v3474: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 5 op/s 2024-03-24T01:53:25.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:53:24 smithi193 ceph-mon[36878]: pgmap v3474: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 5 op/s 2024-03-24T01:53:26.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:53:25 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 5301.690310 secs 2024-03-24T01:53:26.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:53:25 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 5301.690310 secs 2024-03-24T01:53:26.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:53:25 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 5301.690310 secs 2024-03-24T01:53:27.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:53:26 smithi118 ceph-mon[31535]: pgmap v3475: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 6 op/s 2024-03-24T01:53:27.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:53:26 smithi189 ceph-mon[36187]: pgmap v3475: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 6 op/s 2024-03-24T01:53:27.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:53:26 smithi193 ceph-mon[36878]: pgmap v3475: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 6 op/s 2024-03-24T01:53:29.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:53:28 smithi118 ceph-mon[31535]: pgmap v3476: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 5 op/s 2024-03-24T01:53:29.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:53:28 smithi189 ceph-mon[36187]: pgmap v3476: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 5 op/s 2024-03-24T01:53:29.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:53:28 smithi193 ceph-mon[36878]: pgmap v3476: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 5 op/s 2024-03-24T01:53:31.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:53:30 smithi118 ceph-mon[31535]: pgmap v3477: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 6 op/s 2024-03-24T01:53:31.197 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:53:30 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 5306.690398 secs 2024-03-24T01:53:31.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:53:30 smithi189 ceph-mon[36187]: pgmap v3477: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 6 op/s 2024-03-24T01:53:31.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:53:30 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 5306.690398 secs 2024-03-24T01:53:31.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:53:30 smithi193 ceph-mon[36878]: pgmap v3477: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 6 op/s 2024-03-24T01:53:31.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:53:30 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 5306.690398 secs 2024-03-24T01:53:33.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:53:32 smithi118 ceph-mon[31535]: pgmap v3478: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 5 op/s 2024-03-24T01:53:33.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:53:32 smithi189 ceph-mon[36187]: pgmap v3478: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 5 op/s 2024-03-24T01:53:33.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:53:32 smithi193 ceph-mon[36878]: pgmap v3478: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 5 op/s 2024-03-24T01:53:35.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:53:34 smithi118 ceph-mon[31535]: pgmap v3479: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T01:53:35.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:53:34 smithi189 ceph-mon[36187]: pgmap v3479: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T01:53:35.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:53:34 smithi193 ceph-mon[36878]: pgmap v3479: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T01:53:36.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:53:35 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 5311.690441 secs 2024-03-24T01:53:36.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:53:35 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 5311.690441 secs 2024-03-24T01:53:36.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:53:35 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 5311.690441 secs 2024-03-24T01:53:37.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:53:36 smithi118 ceph-mon[31535]: pgmap v3480: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T01:53:37.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:53:36 smithi189 ceph-mon[36187]: pgmap v3480: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T01:53:37.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:53:36 smithi193 ceph-mon[36878]: pgmap v3480: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T01:53:39.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:53:38 smithi118 ceph-mon[31535]: pgmap v3481: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:53:39.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:53:38 smithi189 ceph-mon[36187]: pgmap v3481: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:53:39.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:53:38 smithi193 ceph-mon[36878]: pgmap v3481: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:53:41.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:53:40 smithi118 ceph-mon[31535]: pgmap v3482: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-24T01:53:41.197 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:53:40 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 5316.690528 secs 2024-03-24T01:53:41.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:53:40 smithi189 ceph-mon[36187]: pgmap v3482: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-24T01:53:41.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:53:40 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 5316.690528 secs 2024-03-24T01:53:41.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:53:40 smithi193 ceph-mon[36878]: pgmap v3482: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-24T01:53:41.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:53:40 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 5316.690528 secs 2024-03-24T01:53:43.198 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:53:42 smithi118 ceph-mon[31535]: pgmap v3483: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:53:43.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:53:42 smithi189 ceph-mon[36187]: pgmap v3483: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:53:43.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:53:42 smithi193 ceph-mon[36878]: pgmap v3483: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:53:45.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:53:44 smithi118 ceph-mon[31535]: pgmap v3484: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T01:53:45.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:53:44 smithi189 ceph-mon[36187]: pgmap v3484: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T01:53:45.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:53:44 smithi193 ceph-mon[36878]: pgmap v3484: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T01:53:46.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:53:45 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 5321.690590 secs 2024-03-24T01:53:46.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:53:45 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 5321.690590 secs 2024-03-24T01:53:46.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:53:45 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 5321.690590 secs 2024-03-24T01:53:47.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:53:46 smithi118 ceph-mon[31535]: pgmap v3485: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T01:53:47.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:53:46 smithi189 ceph-mon[36187]: pgmap v3485: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T01:53:47.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:53:46 smithi193 ceph-mon[36878]: pgmap v3485: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T01:53:49.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:53:48 smithi118 ceph-mon[31535]: pgmap v3486: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:53:49.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:53:48 smithi189 ceph-mon[36187]: pgmap v3486: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:53:49.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:53:48 smithi193 ceph-mon[36878]: pgmap v3486: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:53:51.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:53:50 smithi118 ceph-mon[31535]: pgmap v3487: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-24T01:53:51.197 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:53:50 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 5326.690673 secs 2024-03-24T01:53:51.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:53:50 smithi189 ceph-mon[36187]: pgmap v3487: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-24T01:53:51.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:53:50 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 5326.690673 secs 2024-03-24T01:53:51.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:53:50 smithi193 ceph-mon[36878]: pgmap v3487: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-24T01:53:51.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:53:50 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 5326.690673 secs 2024-03-24T01:53:53.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:53:52 smithi118 ceph-mon[31535]: pgmap v3488: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:53:53.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:53:52 smithi189 ceph-mon[36187]: pgmap v3488: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:53:53.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:53:52 smithi193 ceph-mon[36878]: pgmap v3488: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:53:55.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:53:54 smithi118 ceph-mon[31535]: pgmap v3489: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T01:53:55.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:53:54 smithi189 ceph-mon[36187]: pgmap v3489: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T01:53:55.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:53:54 smithi193 ceph-mon[36878]: pgmap v3489: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T01:53:56.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:53:55 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 5331.690733 secs 2024-03-24T01:53:56.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:53:55 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 5331.690733 secs 2024-03-24T01:53:56.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:53:55 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 5331.690733 secs 2024-03-24T01:53:57.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:53:56 smithi118 ceph-mon[31535]: pgmap v3490: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T01:53:57.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:53:56 smithi189 ceph-mon[36187]: pgmap v3490: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T01:53:57.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:53:56 smithi193 ceph-mon[36878]: pgmap v3490: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T01:53:59.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:53:58 smithi118 ceph-mon[31535]: pgmap v3491: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:53:59.205 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:53:58 smithi189 ceph-mon[36187]: pgmap v3491: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:53:59.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:53:58 smithi193 ceph-mon[36878]: pgmap v3491: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:54:01.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:54:00 smithi118 ceph-mon[31535]: pgmap v3492: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-24T01:54:01.197 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:54:00 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 5336.690812 secs 2024-03-24T01:54:01.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:54:00 smithi189 ceph-mon[36187]: pgmap v3492: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-24T01:54:01.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:54:00 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 5336.690812 secs 2024-03-24T01:54:01.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:54:00 smithi193 ceph-mon[36878]: pgmap v3492: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-24T01:54:01.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:54:00 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 5336.690812 secs 2024-03-24T01:54:03.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:54:02 smithi118 ceph-mon[31535]: pgmap v3493: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:54:03.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:54:02 smithi189 ceph-mon[36187]: pgmap v3493: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:54:03.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:54:02 smithi193 ceph-mon[36878]: pgmap v3493: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:54:05.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:54:04 smithi118 ceph-mon[31535]: pgmap v3494: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T01:54:05.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:54:04 smithi189 ceph-mon[36187]: pgmap v3494: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T01:54:05.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:54:04 smithi193 ceph-mon[36878]: pgmap v3494: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T01:54:06.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:54:05 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 5341.690898 secs 2024-03-24T01:54:06.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:54:05 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 5341.690898 secs 2024-03-24T01:54:06.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:54:05 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 5341.690898 secs 2024-03-24T01:54:07.197 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:54:06 smithi118 ceph-mon[31535]: pgmap v3495: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T01:54:07.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:54:06 smithi189 ceph-mon[36187]: pgmap v3495: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T01:54:07.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:54:06 smithi193 ceph-mon[36878]: pgmap v3495: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T01:54:09.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:54:08 smithi118 ceph-mon[31535]: pgmap v3496: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:54:09.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:54:08 smithi189 ceph-mon[36187]: pgmap v3496: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:54:09.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:54:08 smithi193 ceph-mon[36878]: pgmap v3496: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:54:11.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:54:10 smithi118 ceph-mon[31535]: pgmap v3497: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-24T01:54:11.197 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:54:10 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 5346.690943 secs 2024-03-24T01:54:11.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:54:10 smithi189 ceph-mon[36187]: pgmap v3497: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-24T01:54:11.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:54:10 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 5346.690943 secs 2024-03-24T01:54:11.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:54:10 smithi193 ceph-mon[36878]: pgmap v3497: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-24T01:54:11.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:54:10 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 5346.690943 secs 2024-03-24T01:54:13.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:54:12 smithi118 ceph-mon[31535]: pgmap v3498: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:54:13.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:54:12 smithi189 ceph-mon[36187]: pgmap v3498: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:54:13.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:54:12 smithi193 ceph-mon[36878]: pgmap v3498: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:54:15.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:54:14 smithi118 ceph-mon[31535]: pgmap v3499: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T01:54:15.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:54:14 smithi189 ceph-mon[36187]: pgmap v3499: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T01:54:15.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:54:14 smithi193 ceph-mon[36878]: pgmap v3499: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T01:54:16.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:54:15 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 5351.691025 secs 2024-03-24T01:54:16.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:54:15 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 5351.691025 secs 2024-03-24T01:54:16.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:54:15 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 5351.691025 secs 2024-03-24T01:54:17.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:54:16 smithi193 ceph-mon[36878]: pgmap v3500: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T01:54:17.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:54:16 smithi118 ceph-mon[31535]: pgmap v3500: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T01:54:17.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:54:16 smithi189 ceph-mon[36187]: pgmap v3500: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T01:54:19.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:54:18 smithi193 ceph-mon[36878]: pgmap v3501: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:54:19.288 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:54:18 smithi118 ceph-mon[31535]: pgmap v3501: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:54:19.458 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:54:18 smithi189 ceph-mon[36187]: pgmap v3501: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:54:21.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:54:20 smithi118 ceph-mon[31535]: pgmap v3502: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-24T01:54:21.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:54:20 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 5356.691114 secs 2024-03-24T01:54:21.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:54:20 smithi189 ceph-mon[36187]: pgmap v3502: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-24T01:54:21.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:54:20 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 5356.691114 secs 2024-03-24T01:54:21.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:54:20 smithi193 ceph-mon[36878]: pgmap v3502: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-24T01:54:21.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:54:20 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 5356.691114 secs 2024-03-24T01:54:22.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:54:21 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T01:54:22.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:54:21 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T01:54:22.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:54:21 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T01:54:23.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:54:22 smithi118 ceph-mon[31535]: pgmap v3503: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:54:23.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:54:22 smithi189 ceph-mon[36187]: pgmap v3503: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:54:23.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:54:22 smithi193 ceph-mon[36878]: pgmap v3503: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:54:24.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:54:23 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T01:54:24.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:54:23 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T01:54:24.447 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:54:23 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T01:54:24.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:54:23 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T01:54:24.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:54:23 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T01:54:24.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:54:23 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T01:54:24.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:54:23 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T01:54:24.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:54:23 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T01:54:24.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:54:23 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T01:54:25.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:54:24 smithi118 ceph-mon[31535]: pgmap v3504: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 6 op/s 2024-03-24T01:54:25.447 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:54:24 smithi118 ceph-mon[31535]: pgmap v3505: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 6 op/s 2024-03-24T01:54:25.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:54:24 smithi189 ceph-mon[36187]: pgmap v3504: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 6 op/s 2024-03-24T01:54:25.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:54:24 smithi189 ceph-mon[36187]: pgmap v3505: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 6 op/s 2024-03-24T01:54:25.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:54:24 smithi193 ceph-mon[36878]: pgmap v3504: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 6 op/s 2024-03-24T01:54:25.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:54:24 smithi193 ceph-mon[36878]: pgmap v3505: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 6 op/s 2024-03-24T01:54:26.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:54:25 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 5361.691174 secs 2024-03-24T01:54:26.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:54:25 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 5361.691174 secs 2024-03-24T01:54:26.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:54:25 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 5361.691174 secs 2024-03-24T01:54:27.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:54:26 smithi118 ceph-mon[31535]: pgmap v3506: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 6 op/s 2024-03-24T01:54:27.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:54:26 smithi189 ceph-mon[36187]: pgmap v3506: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 6 op/s 2024-03-24T01:54:27.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:54:26 smithi193 ceph-mon[36878]: pgmap v3506: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 6 op/s 2024-03-24T01:54:29.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:54:28 smithi118 ceph-mon[31535]: pgmap v3507: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 6 op/s 2024-03-24T01:54:29.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:54:28 smithi189 ceph-mon[36187]: pgmap v3507: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 6 op/s 2024-03-24T01:54:29.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:54:28 smithi193 ceph-mon[36878]: pgmap v3507: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 6 op/s 2024-03-24T01:54:31.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:54:31 smithi118 ceph-mon[31535]: pgmap v3508: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 5 op/s 2024-03-24T01:54:31.447 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:54:31 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 5366.691257 secs 2024-03-24T01:54:31.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:54:31 smithi189 ceph-mon[36187]: pgmap v3508: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 5 op/s 2024-03-24T01:54:31.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:54:31 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 5366.691257 secs 2024-03-24T01:54:31.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:54:31 smithi193 ceph-mon[36878]: pgmap v3508: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 5 op/s 2024-03-24T01:54:31.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:54:31 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 5366.691257 secs 2024-03-24T01:54:33.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:54:33 smithi118 ceph-mon[31535]: pgmap v3509: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 6 op/s 2024-03-24T01:54:33.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:54:33 smithi189 ceph-mon[36187]: pgmap v3509: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 6 op/s 2024-03-24T01:54:33.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:54:33 smithi193 ceph-mon[36878]: pgmap v3509: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 6 op/s 2024-03-24T01:54:34.447 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:54:34 smithi118 ceph-mon[31535]: pgmap v3510: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 5 op/s 2024-03-24T01:54:34.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:54:34 smithi189 ceph-mon[36187]: pgmap v3510: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 5 op/s 2024-03-24T01:54:34.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:54:34 smithi193 ceph-mon[36878]: pgmap v3510: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 5 op/s 2024-03-24T01:54:36.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:54:36 smithi118 ceph-mon[31535]: pgmap v3511: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T01:54:36.447 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:54:36 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 5371.691318 secs 2024-03-24T01:54:36.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:54:36 smithi189 ceph-mon[36187]: pgmap v3511: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T01:54:36.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:54:36 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 5371.691318 secs 2024-03-24T01:54:36.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:54:36 smithi193 ceph-mon[36878]: pgmap v3511: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T01:54:36.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:54:36 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 5371.691318 secs 2024-03-24T01:54:38.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:54:38 smithi118 ceph-mon[31535]: pgmap v3512: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:54:38.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:54:38 smithi189 ceph-mon[36187]: pgmap v3512: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:54:38.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:54:38 smithi193 ceph-mon[36878]: pgmap v3512: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:54:40.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:54:40 smithi118 ceph-mon[31535]: pgmap v3513: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-24T01:54:40.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:54:40 smithi189 ceph-mon[36187]: pgmap v3513: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-24T01:54:40.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:54:40 smithi193 ceph-mon[36878]: pgmap v3513: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-24T01:54:41.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:54:41 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 5376.691408 secs 2024-03-24T01:54:41.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:54:41 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 5376.691408 secs 2024-03-24T01:54:41.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:54:41 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 5376.691408 secs 2024-03-24T01:54:42.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:54:42 smithi118 ceph-mon[31535]: pgmap v3514: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T01:54:42.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:54:42 smithi189 ceph-mon[36187]: pgmap v3514: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T01:54:42.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:54:42 smithi193 ceph-mon[36878]: pgmap v3514: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T01:54:44.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:54:44 smithi118 ceph-mon[31535]: pgmap v3515: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-03-24T01:54:44.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:54:44 smithi189 ceph-mon[36187]: pgmap v3515: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-03-24T01:54:44.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:54:44 smithi193 ceph-mon[36878]: pgmap v3515: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-03-24T01:54:46.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:54:46 smithi118 ceph-mon[31535]: pgmap v3516: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 6 op/s 2024-03-24T01:54:46.447 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:54:46 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 5381.691486 secs 2024-03-24T01:54:46.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:54:46 smithi189 ceph-mon[36187]: pgmap v3516: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 6 op/s 2024-03-24T01:54:46.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:54:46 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 5381.691486 secs 2024-03-24T01:54:46.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:54:46 smithi193 ceph-mon[36878]: pgmap v3516: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 6 op/s 2024-03-24T01:54:46.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:54:46 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 5381.691486 secs 2024-03-24T01:54:48.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:54:48 smithi118 ceph-mon[31535]: pgmap v3517: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-03-24T01:54:48.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:54:48 smithi189 ceph-mon[36187]: pgmap v3517: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-03-24T01:54:48.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:54:48 smithi193 ceph-mon[36878]: pgmap v3517: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-03-24T01:54:50.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:54:50 smithi118 ceph-mon[31535]: pgmap v3518: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-24T01:54:50.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:54:50 smithi189 ceph-mon[36187]: pgmap v3518: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-24T01:54:50.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:54:50 smithi193 ceph-mon[36878]: pgmap v3518: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-24T01:54:51.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:54:51 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 5386.691576 secs 2024-03-24T01:54:51.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:54:51 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 5386.691576 secs 2024-03-24T01:54:51.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:54:51 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 5386.691576 secs 2024-03-24T01:54:52.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:54:52 smithi118 ceph-mon[31535]: pgmap v3519: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-03-24T01:54:52.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:54:52 smithi189 ceph-mon[36187]: pgmap v3519: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-03-24T01:54:52.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:54:52 smithi193 ceph-mon[36878]: pgmap v3519: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-03-24T01:54:54.449 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:54:54 smithi118 ceph-mon[31535]: pgmap v3520: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-03-24T01:54:54.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:54:54 smithi189 ceph-mon[36187]: pgmap v3520: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-03-24T01:54:54.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:54:54 smithi193 ceph-mon[36878]: pgmap v3520: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-03-24T01:54:56.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:54:56 smithi118 ceph-mon[31535]: pgmap v3521: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T01:54:56.447 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:54:56 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 5391.691616 secs 2024-03-24T01:54:56.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:54:56 smithi189 ceph-mon[36187]: pgmap v3521: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T01:54:56.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:54:56 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 5391.691616 secs 2024-03-24T01:54:56.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:54:56 smithi193 ceph-mon[36878]: pgmap v3521: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T01:54:56.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:54:56 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 5391.691616 secs 2024-03-24T01:54:58.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:54:58 smithi118 ceph-mon[31535]: pgmap v3522: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:54:58.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:54:58 smithi189 ceph-mon[36187]: pgmap v3522: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:54:58.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:54:58 smithi193 ceph-mon[36878]: pgmap v3522: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:55:00.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:55:00 smithi118 ceph-mon[31535]: pgmap v3523: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-24T01:55:00.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:55:00 smithi189 ceph-mon[36187]: pgmap v3523: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-24T01:55:00.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:55:00 smithi193 ceph-mon[36878]: pgmap v3523: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-24T01:55:01.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:55:01 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 5396.691708 secs 2024-03-24T01:55:01.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:55:01 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 5396.691708 secs 2024-03-24T01:55:01.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:55:01 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 5396.691708 secs 2024-03-24T01:55:02.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:55:02 smithi118 ceph-mon[31535]: pgmap v3524: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:55:02.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:55:02 smithi189 ceph-mon[36187]: pgmap v3524: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:55:02.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:55:02 smithi193 ceph-mon[36878]: pgmap v3524: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:55:04.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:55:04 smithi118 ceph-mon[31535]: pgmap v3525: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T01:55:04.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:55:04 smithi189 ceph-mon[36187]: pgmap v3525: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T01:55:04.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:55:04 smithi193 ceph-mon[36878]: pgmap v3525: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T01:55:06.447 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:55:06 smithi118 ceph-mon[31535]: pgmap v3526: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T01:55:06.447 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:55:06 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 5401.691793 secs 2024-03-24T01:55:06.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:55:06 smithi189 ceph-mon[36187]: pgmap v3526: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T01:55:06.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:55:06 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 5401.691793 secs 2024-03-24T01:55:06.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:55:06 smithi193 ceph-mon[36878]: pgmap v3526: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T01:55:06.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:55:06 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 5401.691793 secs 2024-03-24T01:55:08.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:55:08 smithi118 ceph-mon[31535]: pgmap v3527: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:55:08.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:55:08 smithi189 ceph-mon[36187]: pgmap v3527: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:55:08.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:55:08 smithi193 ceph-mon[36878]: pgmap v3527: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:55:10.447 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:55:10 smithi118 ceph-mon[31535]: pgmap v3528: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-24T01:55:10.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:55:10 smithi189 ceph-mon[36187]: pgmap v3528: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-24T01:55:10.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:55:10 smithi193 ceph-mon[36878]: pgmap v3528: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-24T01:55:11.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:55:11 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 5406.691883 secs 2024-03-24T01:55:11.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:55:11 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 5406.691883 secs 2024-03-24T01:55:11.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:55:11 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 5406.691883 secs 2024-03-24T01:55:12.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:55:12 smithi118 ceph-mon[31535]: pgmap v3529: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:55:12.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:55:12 smithi189 ceph-mon[36187]: pgmap v3529: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:55:12.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:55:12 smithi193 ceph-mon[36878]: pgmap v3529: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:55:14.447 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:55:14 smithi118 ceph-mon[31535]: pgmap v3530: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T01:55:14.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:55:14 smithi189 ceph-mon[36187]: pgmap v3530: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T01:55:14.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:55:14 smithi193 ceph-mon[36878]: pgmap v3530: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T01:55:15.750 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T01:55:15.749+0000 7f81f6000640 1 -- 172.21.15.118:0/1693767697 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7f81dc0a4320 msgr2=0x7f81dc0b05c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 12 2024-03-24T01:55:15.751 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T01:55:15.750+0000 7f81f6000640 1 -- 172.21.15.118:0/1693767697 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7f81dc0a4320 msgr2=0x7f81dc0b05c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-03-24T01:55:15.751 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T01:55:15.750+0000 7f81f6000640 1 --2- 172.21.15.118:0/1693767697 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7f81dc0a4320 0x7f81dc0b05c0 secure :-1 s=READY pgs=501 cs=0 l=1 rev1=1 crypto rx=0x7f81e00047d0 tx=0x7f81e0044040 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-03-24T01:55:15.751 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T01:55:15.750+0000 7f81f6000640 1 --2- 172.21.15.118:0/1693767697 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7f81dc0a4320 0x7f81dc0b05c0 secure :-1 s=READY pgs=501 cs=0 l=1 rev1=1 crypto rx=0x7f81e00047d0 tx=0x7f81e0044040 comp rx=0 tx=0).stop 2024-03-24T01:55:15.751 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T01:55:15.750+0000 7f81f6000640 1 -- 172.21.15.118:0/1693767697 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7f81dc0a4320 msgr2=0x7f81dc0b05c0 unknown :-1 s=STATE_CLOSED l=1).tick see no progress in more than 10000000 us during connecting to v2:172.21.15.118:6800/2112912943, fault. 2024-03-24T01:55:15.751 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T01:55:15.750+0000 7f81ef400640 1 -- 172.21.15.118:0/1693767697 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7f81dc0a4320 msgr2=0x7f81dc0b05c0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-03-24T01:55:15.751 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T01:55:15.750+0000 7f81ef400640 1 --2- 172.21.15.118:0/1693767697 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7f81dc0a4320 0x7f81dc0b05c0 unknown :-1 s=CLOSED pgs=501 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T01:55:15.751 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T01:55:15.750+0000 7f81ef400640 1 --2- 172.21.15.118:0/1693767697 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7f81dc0a3bd0 0x7f81dc0b2b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T01:55:15.751 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T01:55:15.751+0000 7f81f6000640 1 --2- 172.21.15.118:0/1693767697 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7f81dc0a3bd0 0x7f81dc0b2b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:55:15.752 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T01:55:15.752+0000 7f81f6000640 1 --2- 172.21.15.118:0/1693767697 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7f81dc0a3bd0 0x7f81dc0b2b10 secure :-1 s=READY pgs=520 cs=0 l=1 rev1=1 crypto rx=0x7f81e0010870 tx=0x7f81e0044040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T01:55:15.765 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T01:55:15.765+0000 7f81f6000640 1 -- 172.21.15.118:0/659686424 >> [v2:172.21.15.189:6838/4074484168,v1:172.21.15.189:6839/4074484168] conn(0x5622651b71e0 msgr2=0x562265215110 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).tick idle (900000825) for more than 900000000 us, fault. 2024-03-24T01:55:15.765 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T01:55:15.765+0000 7f81f6000640 1 --2- 172.21.15.118:0/659686424 >> [v2:172.21.15.189:6838/4074484168,v1:172.21.15.189:6839/4074484168] conn(0x5622651b71e0 0x562265215110 unknown :-1 s=READY pgs=561 cs=4 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault initiating reconnect 2024-03-24T01:55:15.765 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T01:55:15.765+0000 7f81f6000640 1 --2- 172.21.15.118:0/659686424 >> [v2:172.21.15.189:6838/4074484168,v1:172.21.15.189:6839/4074484168] conn(0x5622651b71e0 0x562265215110 unknown :-1 s=BANNER_CONNECTING pgs=561 cs=5 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T01:55:15.766 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T01:55:15.766+0000 7f81f6000640 1 --2- 172.21.15.118:0/659686424 >> [v2:172.21.15.189:6838/4074484168,v1:172.21.15.189:6839/4074484168] conn(0x5622651b71e0 0x562265215110 crc :-1 s=READY pgs=561 cs=5 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=76f3d534289740f0 server_cookie=b31a2335a59552de in_seq=0 out_seq=0 2024-03-24T01:55:15.766 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T01:55:15.766+0000 7f81e4c00640 10 client.27094 ms_handle_connect on v2:172.21.15.189:6838/4074484168 2024-03-24T01:55:16.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:55:16 smithi118 ceph-mon[31535]: pgmap v3531: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T01:55:16.447 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:55:16 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 5411.691972 secs 2024-03-24T01:55:16.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:55:16 smithi189 ceph-mon[36187]: pgmap v3531: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T01:55:16.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:55:16 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 5411.691972 secs 2024-03-24T01:55:16.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:55:16 smithi193 ceph-mon[36878]: pgmap v3531: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T01:55:16.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:55:16 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 5411.691972 secs 2024-03-24T01:55:16.601 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T01:55:16.601+0000 7f81ee000640 1 -- 172.21.15.118:0/1693767697 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f81c80036c0 con 0x7f81f81052d0 2024-03-24T01:55:16.602 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T01:55:16.601+0000 7f81db400640 1 -- 172.21.15.118:0/659686424 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f81c4006460 con 0x5622651a13f0 2024-03-24T01:55:16.602 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T01:55:16.602+0000 7f81ef400640 1 -- 172.21.15.118:0/1693767697 <== mon.2 v2:172.21.15.189:3300/0 45 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (secure 0 0 0) 0x7f81c80036c0 con 0x7f81f81052d0 2024-03-24T01:55:16.602 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T01:55:16.602+0000 7f81e4c00640 1 -- 172.21.15.118:0/659686424 <== mon.1 v2:172.21.15.193:3300/0 45 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 409+0+0 (secure 0 0 0) 0x7f81c4006460 con 0x5622651a13f0 2024-03-24T01:55:18.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:55:18 smithi118 ceph-mon[31535]: pgmap v3532: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:55:18.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:55:18 smithi189 ceph-mon[36187]: pgmap v3532: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:55:18.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:55:18 smithi193 ceph-mon[36878]: pgmap v3532: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:55:20.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:55:20 smithi118 ceph-mon[31535]: pgmap v3533: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-24T01:55:20.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:55:20 smithi189 ceph-mon[36187]: pgmap v3533: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-24T01:55:20.462 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:55:20 smithi193 ceph-mon[36878]: pgmap v3533: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-24T01:55:21.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:55:21 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 5416.692015 secs 2024-03-24T01:55:21.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:55:21 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 5416.692015 secs 2024-03-24T01:55:21.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:55:21 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 5416.692015 secs 2024-03-24T01:55:22.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:55:22 smithi189 ceph-mon[36187]: pgmap v3534: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:55:22.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:55:22 smithi193 ceph-mon[36878]: pgmap v3534: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:55:22.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:55:22 smithi118 ceph-mon[31535]: pgmap v3534: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:55:24.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:55:24 smithi193 ceph-mon[36878]: pgmap v3535: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T01:55:24.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:55:24 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T01:55:24.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:55:24 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-24T01:55:24.462 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:55:24 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-24T01:55:24.462 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:55:24 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-24T01:55:24.462 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:55:24 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-03-24T01:55:24.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:55:24 smithi118 ceph-mon[31535]: pgmap v3535: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T01:55:24.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:55:24 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T01:55:24.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:55:24 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-24T01:55:24.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:55:24 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-24T01:55:24.697 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:55:24 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-24T01:55:24.697 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:55:24 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-03-24T01:55:24.704 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:55:24 smithi189 ceph-mon[36187]: pgmap v3535: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T01:55:24.704 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:55:24 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T01:55:24.704 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:55:24 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-24T01:55:24.704 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:55:24 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-24T01:55:24.704 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:55:24 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-24T01:55:24.704 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:55:24 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-03-24T01:55:25.697 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:55:25 smithi118 ceph-mon[31535]: Adjusting osd_memory_target on smithi189 to 174.1M 2024-03-24T01:55:25.697 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:55:25 smithi118 ceph-mon[31535]: Unable to set osd_memory_target on smithi189 to 182635622: error parsing value: Value '182635622' is below minimum 939524096 2024-03-24T01:55:25.697 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:55:25 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-24T01:55:25.697 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:55:25 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-24T01:55:25.697 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:55:25 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-24T01:55:25.697 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:55:25 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-03-24T01:55:25.697 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:55:25 smithi118 ceph-mon[31535]: Adjusting osd_memory_target on smithi118 to 176.7M 2024-03-24T01:55:25.697 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:55:25 smithi118 ceph-mon[31535]: Unable to set osd_memory_target on smithi118 to 185365196: error parsing value: Value '185365196' is below minimum 939524096 2024-03-24T01:55:25.698 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:55:25 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T01:55:25.698 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:55:25 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T01:55:25.698 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:55:25 smithi118 ceph-mon[31535]: pgmap v3536: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 6 op/s 2024-03-24T01:55:25.698 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:55:25 smithi118 ceph-mon[31535]: pgmap v3537: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-03-24T01:55:25.698 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:55:25 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T01:55:25.704 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:55:25 smithi189 ceph-mon[36187]: Adjusting osd_memory_target on smithi189 to 174.1M 2024-03-24T01:55:25.704 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:55:25 smithi189 ceph-mon[36187]: Unable to set osd_memory_target on smithi189 to 182635622: error parsing value: Value '182635622' is below minimum 939524096 2024-03-24T01:55:25.705 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:55:25 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-24T01:55:25.705 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:55:25 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-24T01:55:25.705 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:55:25 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-24T01:55:25.705 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:55:25 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-03-24T01:55:25.705 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:55:25 smithi189 ceph-mon[36187]: Adjusting osd_memory_target on smithi118 to 176.7M 2024-03-24T01:55:25.705 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:55:25 smithi189 ceph-mon[36187]: Unable to set osd_memory_target on smithi118 to 185365196: error parsing value: Value '185365196' is below minimum 939524096 2024-03-24T01:55:25.705 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:55:25 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T01:55:25.705 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:55:25 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T01:55:25.705 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:55:25 smithi189 ceph-mon[36187]: pgmap v3536: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 6 op/s 2024-03-24T01:55:25.705 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:55:25 smithi189 ceph-mon[36187]: pgmap v3537: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-03-24T01:55:25.705 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:55:25 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T01:55:25.711 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:55:25 smithi193 ceph-mon[36878]: Adjusting osd_memory_target on smithi189 to 174.1M 2024-03-24T01:55:25.711 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:55:25 smithi193 ceph-mon[36878]: Unable to set osd_memory_target on smithi189 to 182635622: error parsing value: Value '182635622' is below minimum 939524096 2024-03-24T01:55:25.711 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:55:25 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-24T01:55:25.711 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:55:25 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-24T01:55:25.711 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:55:25 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-24T01:55:25.711 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:55:25 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-03-24T01:55:25.712 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:55:25 smithi193 ceph-mon[36878]: Adjusting osd_memory_target on smithi118 to 176.7M 2024-03-24T01:55:25.712 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:55:25 smithi193 ceph-mon[36878]: Unable to set osd_memory_target on smithi118 to 185365196: error parsing value: Value '185365196' is below minimum 939524096 2024-03-24T01:55:25.712 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:55:25 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T01:55:25.712 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:55:25 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T01:55:25.712 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:55:25 smithi193 ceph-mon[36878]: pgmap v3536: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 6 op/s 2024-03-24T01:55:25.712 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:55:25 smithi193 ceph-mon[36878]: pgmap v3537: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-03-24T01:55:25.712 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:55:25 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T01:55:26.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:55:26 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 5421.692098 secs 2024-03-24T01:55:26.704 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:55:26 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 5421.692098 secs 2024-03-24T01:55:26.711 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:55:26 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 5421.692098 secs 2024-03-24T01:55:27.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:55:27 smithi118 ceph-mon[31535]: pgmap v3538: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-03-24T01:55:27.704 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:55:27 smithi189 ceph-mon[36187]: pgmap v3538: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-03-24T01:55:27.711 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:55:27 smithi193 ceph-mon[36878]: pgmap v3538: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-03-24T01:55:29.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:55:29 smithi118 ceph-mon[31535]: pgmap v3539: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 4 op/s 2024-03-24T01:55:29.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:55:29 smithi189 ceph-mon[36187]: pgmap v3539: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 4 op/s 2024-03-24T01:55:29.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:55:29 smithi193 ceph-mon[36878]: pgmap v3539: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 4 op/s 2024-03-24T01:55:31.947 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:55:31 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 5426.692158 secs 2024-03-24T01:55:31.947 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:55:31 smithi118 ceph-mon[31535]: pgmap v3540: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-03-24T01:55:31.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:55:31 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 5426.692158 secs 2024-03-24T01:55:31.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:55:31 smithi189 ceph-mon[36187]: pgmap v3540: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-03-24T01:55:31.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:55:31 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 5426.692158 secs 2024-03-24T01:55:31.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:55:31 smithi193 ceph-mon[36878]: pgmap v3540: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-03-24T01:55:33.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:55:33 smithi118 ceph-mon[31535]: pgmap v3541: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 5 op/s 2024-03-24T01:55:33.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:55:33 smithi189 ceph-mon[36187]: pgmap v3541: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 5 op/s 2024-03-24T01:55:33.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:55:33 smithi193 ceph-mon[36878]: pgmap v3541: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 5 op/s 2024-03-24T01:55:35.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:55:35 smithi118 ceph-mon[31535]: pgmap v3542: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 5 op/s 2024-03-24T01:55:35.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:55:35 smithi189 ceph-mon[36187]: pgmap v3542: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 5 op/s 2024-03-24T01:55:35.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:55:35 smithi193 ceph-mon[36878]: pgmap v3542: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 5 op/s 2024-03-24T01:55:36.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:55:36 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 5431.692219 secs 2024-03-24T01:55:36.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:55:36 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 5431.692219 secs 2024-03-24T01:55:36.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:55:36 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 5431.692219 secs 2024-03-24T01:55:37.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:55:37 smithi118 ceph-mon[31535]: pgmap v3543: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:55:37.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:55:37 smithi189 ceph-mon[36187]: pgmap v3543: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:55:37.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:55:37 smithi193 ceph-mon[36878]: pgmap v3543: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:55:39.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:55:39 smithi118 ceph-mon[31535]: pgmap v3544: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T01:55:39.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:55:39 smithi189 ceph-mon[36187]: pgmap v3544: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T01:55:39.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:55:39 smithi193 ceph-mon[36878]: pgmap v3544: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T01:55:41.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:55:41 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 5436.692280 secs 2024-03-24T01:55:41.947 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:55:41 smithi118 ceph-mon[31535]: pgmap v3545: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T01:55:41.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:55:41 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 5436.692280 secs 2024-03-24T01:55:41.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:55:41 smithi189 ceph-mon[36187]: pgmap v3545: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T01:55:41.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:55:41 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 5436.692280 secs 2024-03-24T01:55:41.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:55:41 smithi193 ceph-mon[36878]: pgmap v3545: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T01:55:43.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:55:43 smithi118 ceph-mon[31535]: pgmap v3546: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T01:55:43.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:55:43 smithi189 ceph-mon[36187]: pgmap v3546: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T01:55:43.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:55:43 smithi193 ceph-mon[36878]: pgmap v3546: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T01:55:45.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:55:45 smithi189 ceph-mon[36187]: pgmap v3547: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-03-24T01:55:45.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:55:45 smithi193 ceph-mon[36878]: pgmap v3547: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-03-24T01:55:46.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:55:45 smithi118 ceph-mon[31535]: pgmap v3547: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-03-24T01:55:46.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:55:46 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 5441.692294 secs 2024-03-24T01:55:47.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:55:46 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 5441.692294 secs 2024-03-24T01:55:47.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:55:46 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 5441.692294 secs 2024-03-24T01:55:47.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:55:47 smithi193 ceph-mon[36878]: pgmap v3548: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:55:48.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:55:47 smithi118 ceph-mon[31535]: pgmap v3548: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:55:48.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:55:47 smithi189 ceph-mon[36187]: pgmap v3548: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:55:50.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:55:49 smithi118 ceph-mon[31535]: pgmap v3549: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T01:55:50.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:55:49 smithi189 ceph-mon[36187]: pgmap v3549: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T01:55:50.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:55:49 smithi193 ceph-mon[36878]: pgmap v3549: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T01:55:51.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:55:50 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 5446.692364 secs 2024-03-24T01:55:51.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:55:50 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 5446.692364 secs 2024-03-24T01:55:51.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:55:50 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 5446.692364 secs 2024-03-24T01:55:52.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:55:51 smithi118 ceph-mon[31535]: pgmap v3550: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T01:55:52.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:55:51 smithi189 ceph-mon[36187]: pgmap v3550: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T01:55:52.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:55:51 smithi193 ceph-mon[36878]: pgmap v3550: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T01:55:54.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:55:53 smithi118 ceph-mon[31535]: pgmap v3551: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T01:55:54.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:55:53 smithi189 ceph-mon[36187]: pgmap v3551: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T01:55:54.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:55:53 smithi193 ceph-mon[36878]: pgmap v3551: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T01:55:56.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:55:55 smithi118 ceph-mon[31535]: pgmap v3552: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-03-24T01:55:56.197 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:55:55 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 5451.692424 secs 2024-03-24T01:55:56.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:55:55 smithi189 ceph-mon[36187]: pgmap v3552: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-03-24T01:55:56.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:55:55 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 5451.692424 secs 2024-03-24T01:55:56.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:55:55 smithi193 ceph-mon[36878]: pgmap v3552: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-03-24T01:55:56.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:55:55 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 5451.692424 secs 2024-03-24T01:55:58.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:55:57 smithi118 ceph-mon[31535]: pgmap v3553: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:55:58.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:55:57 smithi189 ceph-mon[36187]: pgmap v3553: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:55:58.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:55:57 smithi193 ceph-mon[36878]: pgmap v3553: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:56:00.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:55:59 smithi118 ceph-mon[31535]: pgmap v3554: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T01:56:00.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:55:59 smithi189 ceph-mon[36187]: pgmap v3554: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T01:56:00.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:55:59 smithi193 ceph-mon[36878]: pgmap v3554: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T01:56:01.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:56:00 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 5456.692484 secs 2024-03-24T01:56:01.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:56:00 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 5456.692484 secs 2024-03-24T01:56:01.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:56:00 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 5456.692484 secs 2024-03-24T01:56:01.609 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T01:56:01.609+0000 7f81ee000640 1 -- 172.21.15.118:0/1693767697 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- auth(proto 2 132 bytes epoch 0) v1 -- 0x7f81c80036c0 con 0x7f81f81052d0 2024-03-24T01:56:01.609 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T01:56:01.609+0000 7f81db400640 1 -- 172.21.15.118:0/659686424 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- auth(proto 2 132 bytes epoch 0) v1 -- 0x7f81c4006460 con 0x5622651a13f0 2024-03-24T01:56:01.609 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T01:56:01.609+0000 7f81ef400640 1 -- 172.21.15.118:0/1693767697 <== mon.2 v2:172.21.15.189:3300/0 46 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 233+0+0 (secure 0 0 0) 0x7f81c80036c0 con 0x7f81f81052d0 2024-03-24T01:56:01.610 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T01:56:01.609+0000 7f81e4c00640 1 -- 172.21.15.118:0/659686424 <== mon.1 v2:172.21.15.193:3300/0 46 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 233+0+0 (secure 0 0 0) 0x7f81c4006460 con 0x5622651a13f0 2024-03-24T01:56:02.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:56:01 smithi118 ceph-mon[31535]: pgmap v3555: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T01:56:02.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:56:01 smithi189 ceph-mon[36187]: pgmap v3555: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T01:56:02.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:56:01 smithi193 ceph-mon[36878]: pgmap v3555: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T01:56:04.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:56:03 smithi118 ceph-mon[31535]: pgmap v3556: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T01:56:04.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:56:03 smithi189 ceph-mon[36187]: pgmap v3556: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T01:56:04.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:56:03 smithi193 ceph-mon[36878]: pgmap v3556: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T01:56:06.197 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:56:05 smithi118 ceph-mon[31535]: pgmap v3557: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-03-24T01:56:06.197 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:56:05 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 5461.692576 secs 2024-03-24T01:56:06.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:56:05 smithi189 ceph-mon[36187]: pgmap v3557: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-03-24T01:56:06.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:56:05 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 5461.692576 secs 2024-03-24T01:56:06.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:56:05 smithi193 ceph-mon[36878]: pgmap v3557: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-03-24T01:56:06.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:56:05 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 5461.692576 secs 2024-03-24T01:56:08.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:56:07 smithi118 ceph-mon[31535]: pgmap v3558: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:56:08.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:56:07 smithi189 ceph-mon[36187]: pgmap v3558: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:56:08.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:56:07 smithi193 ceph-mon[36878]: pgmap v3558: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:56:10.197 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:56:09 smithi118 ceph-mon[31535]: pgmap v3559: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T01:56:10.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:56:09 smithi189 ceph-mon[36187]: pgmap v3559: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T01:56:10.214 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:56:09 smithi193 ceph-mon[36878]: pgmap v3559: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T01:56:11.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:56:10 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 5466.692665 secs 2024-03-24T01:56:11.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:56:10 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 5466.692665 secs 2024-03-24T01:56:11.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:56:10 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 5466.692665 secs 2024-03-24T01:56:12.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:56:11 smithi118 ceph-mon[31535]: pgmap v3560: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T01:56:12.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:56:11 smithi189 ceph-mon[36187]: pgmap v3560: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T01:56:12.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:56:11 smithi193 ceph-mon[36878]: pgmap v3560: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T01:56:14.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:56:13 smithi118 ceph-mon[31535]: pgmap v3561: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T01:56:14.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:56:13 smithi189 ceph-mon[36187]: pgmap v3561: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T01:56:14.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:56:13 smithi193 ceph-mon[36878]: pgmap v3561: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T01:56:16.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:56:15 smithi118 ceph-mon[31535]: pgmap v3562: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-03-24T01:56:16.197 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:56:15 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 5471.692754 secs 2024-03-24T01:56:16.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:56:15 smithi189 ceph-mon[36187]: pgmap v3562: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-03-24T01:56:16.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:56:15 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 5471.692754 secs 2024-03-24T01:56:16.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:56:15 smithi193 ceph-mon[36878]: pgmap v3562: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-03-24T01:56:16.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:56:15 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 5471.692754 secs 2024-03-24T01:56:18.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:56:17 smithi118 ceph-mon[31535]: pgmap v3563: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:56:18.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:56:17 smithi189 ceph-mon[36187]: pgmap v3563: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:56:18.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:56:17 smithi193 ceph-mon[36878]: pgmap v3563: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:56:20.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:56:19 smithi118 ceph-mon[31535]: pgmap v3564: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T01:56:20.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:56:19 smithi189 ceph-mon[36187]: pgmap v3564: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T01:56:20.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:56:19 smithi193 ceph-mon[36878]: pgmap v3564: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T01:56:21.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:56:20 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 5476.692835 secs 2024-03-24T01:56:21.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:56:20 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 5476.692835 secs 2024-03-24T01:56:21.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:56:20 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 5476.692835 secs 2024-03-24T01:56:22.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:56:21 smithi118 ceph-mon[31535]: pgmap v3565: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T01:56:22.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:56:21 smithi189 ceph-mon[36187]: pgmap v3565: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T01:56:22.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:56:21 smithi193 ceph-mon[36878]: pgmap v3565: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T01:56:24.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:56:23 smithi118 ceph-mon[31535]: pgmap v3566: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T01:56:24.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:56:23 smithi189 ceph-mon[36187]: pgmap v3566: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T01:56:24.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:56:23 smithi193 ceph-mon[36878]: pgmap v3566: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T01:56:25.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:56:24 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T01:56:25.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:56:24 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T01:56:25.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:56:24 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T01:56:26.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:56:25 smithi118 ceph-mon[31535]: pgmap v3567: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-03-24T01:56:26.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:56:25 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 5481.692923 secs 2024-03-24T01:56:26.197 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:56:25 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-24T01:56:26.197 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:56:25 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-24T01:56:26.197 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:56:25 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-03-24T01:56:26.197 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:56:25 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-03-24T01:56:26.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:56:25 smithi189 ceph-mon[36187]: pgmap v3567: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-03-24T01:56:26.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:56:25 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 5481.692923 secs 2024-03-24T01:56:26.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:56:25 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-24T01:56:26.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:56:25 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-24T01:56:26.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:56:25 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-03-24T01:56:26.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:56:25 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-03-24T01:56:26.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:56:25 smithi193 ceph-mon[36878]: pgmap v3567: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-03-24T01:56:26.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:56:25 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 5481.692923 secs 2024-03-24T01:56:26.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:56:25 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-24T01:56:26.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:56:25 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-24T01:56:26.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:56:25 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-03-24T01:56:26.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:56:25 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-03-24T01:56:27.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:56:26 smithi118 ceph-mon[31535]: Adjusting osd_memory_target on smithi193 to 174.1M 2024-03-24T01:56:27.197 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:56:26 smithi118 ceph-mon[31535]: Unable to set osd_memory_target on smithi193 to 182639923: error parsing value: Value '182639923' is below minimum 939524096 2024-03-24T01:56:27.197 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:56:26 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T01:56:27.197 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:56:26 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T01:56:27.197 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:56:26 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T01:56:27.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:56:26 smithi189 ceph-mon[36187]: Adjusting osd_memory_target on smithi193 to 174.1M 2024-03-24T01:56:27.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:56:26 smithi189 ceph-mon[36187]: Unable to set osd_memory_target on smithi193 to 182639923: error parsing value: Value '182639923' is below minimum 939524096 2024-03-24T01:56:27.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:56:26 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T01:56:27.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:56:26 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T01:56:27.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:56:26 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T01:56:27.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:56:26 smithi193 ceph-mon[36878]: Adjusting osd_memory_target on smithi193 to 174.1M 2024-03-24T01:56:27.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:56:26 smithi193 ceph-mon[36878]: Unable to set osd_memory_target on smithi193 to 182639923: error parsing value: Value '182639923' is below minimum 939524096 2024-03-24T01:56:27.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:56:26 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T01:56:27.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:56:26 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T01:56:27.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:56:26 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T01:56:28.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:56:27 smithi118 ceph-mon[31535]: pgmap v3568: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 5 op/s 2024-03-24T01:56:28.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:56:27 smithi118 ceph-mon[31535]: pgmap v3569: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-03-24T01:56:28.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:56:27 smithi189 ceph-mon[36187]: pgmap v3568: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 5 op/s 2024-03-24T01:56:28.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:56:27 smithi189 ceph-mon[36187]: pgmap v3569: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-03-24T01:56:28.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:56:27 smithi193 ceph-mon[36878]: pgmap v3568: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 5 op/s 2024-03-24T01:56:28.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:56:27 smithi193 ceph-mon[36878]: pgmap v3569: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-03-24T01:56:30.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:56:29 smithi118 ceph-mon[31535]: pgmap v3570: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 6 op/s 2024-03-24T01:56:30.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:56:29 smithi189 ceph-mon[36187]: pgmap v3570: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 6 op/s 2024-03-24T01:56:30.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:56:29 smithi193 ceph-mon[36878]: pgmap v3570: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 6 op/s 2024-03-24T01:56:31.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:56:30 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 5486.693008 secs 2024-03-24T01:56:31.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:56:30 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 5486.693008 secs 2024-03-24T01:56:31.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:56:30 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 5486.693008 secs 2024-03-24T01:56:32.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:56:31 smithi189 ceph-mon[36187]: pgmap v3571: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 6 op/s 2024-03-24T01:56:32.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:56:31 smithi193 ceph-mon[36878]: pgmap v3571: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 6 op/s 2024-03-24T01:56:32.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:56:31 smithi118 ceph-mon[31535]: pgmap v3571: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 6 op/s 2024-03-24T01:56:34.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:56:33 smithi193 ceph-mon[36878]: pgmap v3572: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-03-24T01:56:34.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:56:33 smithi118 ceph-mon[31535]: pgmap v3572: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-03-24T01:56:34.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:56:33 smithi189 ceph-mon[36187]: pgmap v3572: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-03-24T01:56:36.250 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:56:35 smithi118 ceph-mon[31535]: pgmap v3573: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 6 op/s 2024-03-24T01:56:36.250 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:56:35 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 5491.693091 secs 2024-03-24T01:56:36.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:56:35 smithi189 ceph-mon[36187]: pgmap v3573: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 6 op/s 2024-03-24T01:56:36.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:56:35 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 5491.693091 secs 2024-03-24T01:56:36.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:56:35 smithi193 ceph-mon[36878]: pgmap v3573: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 6 op/s 2024-03-24T01:56:36.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:56:35 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 5491.693091 secs 2024-03-24T01:56:38.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:56:37 smithi118 ceph-mon[31535]: pgmap v3574: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T01:56:38.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:56:37 smithi189 ceph-mon[36187]: pgmap v3574: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T01:56:38.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:56:37 smithi193 ceph-mon[36878]: pgmap v3574: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T01:56:40.249 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:56:39 smithi118 ceph-mon[31535]: pgmap v3575: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-03-24T01:56:40.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:56:39 smithi189 ceph-mon[36187]: pgmap v3575: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-03-24T01:56:40.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:56:39 smithi193 ceph-mon[36878]: pgmap v3575: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-03-24T01:56:41.355 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:56:40 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 5496.693179 secs 2024-03-24T01:56:41.361 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:56:40 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 5496.693179 secs 2024-03-24T01:56:41.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:56:40 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 5496.693179 secs 2024-03-24T01:56:42.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:56:41 smithi118 ceph-mon[31535]: pgmap v3576: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T01:56:42.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:56:41 smithi189 ceph-mon[36187]: pgmap v3576: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T01:56:42.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:56:41 smithi193 ceph-mon[36878]: pgmap v3576: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T01:56:44.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:56:43 smithi118 ceph-mon[31535]: pgmap v3577: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:56:44.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:56:43 smithi189 ceph-mon[36187]: pgmap v3577: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:56:44.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:56:43 smithi193 ceph-mon[36878]: pgmap v3577: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:56:46.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:56:45 smithi118 ceph-mon[31535]: pgmap v3578: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-24T01:56:46.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:56:45 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 5501.693269 secs 2024-03-24T01:56:46.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:56:45 smithi189 ceph-mon[36187]: pgmap v3578: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-24T01:56:46.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:56:45 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 5501.693269 secs 2024-03-24T01:56:46.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:56:45 smithi193 ceph-mon[36878]: pgmap v3578: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-24T01:56:46.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:56:45 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 5501.693269 secs 2024-03-24T01:56:48.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:56:47 smithi118 ceph-mon[31535]: pgmap v3579: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:56:48.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:56:47 smithi189 ceph-mon[36187]: pgmap v3579: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:56:48.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:56:47 smithi193 ceph-mon[36878]: pgmap v3579: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:56:50.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:56:50 smithi118 ceph-mon[31535]: pgmap v3580: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T01:56:50.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:56:50 smithi189 ceph-mon[36187]: pgmap v3580: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T01:56:50.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:56:50 smithi193 ceph-mon[36878]: pgmap v3580: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T01:56:51.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:56:51 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 5506.693330 secs 2024-03-24T01:56:51.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:56:51 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 5506.693330 secs 2024-03-24T01:56:51.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:56:51 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 5506.693330 secs 2024-03-24T01:56:52.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:56:52 smithi118 ceph-mon[31535]: pgmap v3581: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T01:56:52.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:56:52 smithi189 ceph-mon[36187]: pgmap v3581: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T01:56:52.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:56:52 smithi193 ceph-mon[36878]: pgmap v3581: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T01:56:54.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:56:54 smithi118 ceph-mon[31535]: pgmap v3582: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:56:54.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:56:54 smithi189 ceph-mon[36187]: pgmap v3582: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:56:54.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:56:54 smithi193 ceph-mon[36878]: pgmap v3582: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:56:56.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:56:56 smithi118 ceph-mon[31535]: pgmap v3583: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-24T01:56:56.447 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:56:56 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 5511.693394 secs 2024-03-24T01:56:56.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:56:56 smithi189 ceph-mon[36187]: pgmap v3583: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-24T01:56:56.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:56:56 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 5511.693394 secs 2024-03-24T01:56:56.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:56:56 smithi193 ceph-mon[36878]: pgmap v3583: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-24T01:56:56.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:56:56 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 5511.693394 secs 2024-03-24T01:56:58.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:56:58 smithi118 ceph-mon[31535]: pgmap v3584: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:56:58.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:56:58 smithi189 ceph-mon[36187]: pgmap v3584: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:56:58.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:56:58 smithi193 ceph-mon[36878]: pgmap v3584: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:57:00.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:57:00 smithi118 ceph-mon[31535]: pgmap v3585: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T01:57:00.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:57:00 smithi189 ceph-mon[36187]: pgmap v3585: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T01:57:00.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:57:00 smithi193 ceph-mon[36878]: pgmap v3585: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T01:57:01.447 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:57:01 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 5516.693484 secs 2024-03-24T01:57:01.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:57:01 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 5516.693484 secs 2024-03-24T01:57:01.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:57:01 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 5516.693484 secs 2024-03-24T01:57:02.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:57:02 smithi118 ceph-mon[31535]: pgmap v3586: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T01:57:02.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:57:02 smithi189 ceph-mon[36187]: pgmap v3586: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T01:57:02.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:57:02 smithi193 ceph-mon[36878]: pgmap v3586: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T01:57:03.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:57:03 smithi118 ceph-mon[31535]: pgmap v3587: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:57:03.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:57:03 smithi189 ceph-mon[36187]: pgmap v3587: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:57:03.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:57:03 smithi193 ceph-mon[36878]: pgmap v3587: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:57:05.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:57:05 smithi118 ceph-mon[31535]: pgmap v3588: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-24T01:57:05.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:57:05 smithi189 ceph-mon[36187]: pgmap v3588: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-24T01:57:05.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:57:05 smithi193 ceph-mon[36878]: pgmap v3588: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-24T01:57:06.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:57:06 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 5521.693526 secs 2024-03-24T01:57:06.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:57:06 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 5521.693526 secs 2024-03-24T01:57:06.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:57:06 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 5521.693526 secs 2024-03-24T01:57:07.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:57:07 smithi118 ceph-mon[31535]: pgmap v3589: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:57:07.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:57:07 smithi189 ceph-mon[36187]: pgmap v3589: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:57:07.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:57:07 smithi193 ceph-mon[36878]: pgmap v3589: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:57:09.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:57:09 smithi189 ceph-mon[36187]: pgmap v3590: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T01:57:09.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:57:09 smithi193 ceph-mon[36878]: pgmap v3590: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T01:57:09.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:57:09 smithi118 ceph-mon[31535]: pgmap v3590: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T01:57:11.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:57:11 smithi189 ceph-mon[36187]: pgmap v3591: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T01:57:11.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:57:11 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 5526.693588 secs 2024-03-24T01:57:11.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:57:11 smithi193 ceph-mon[36878]: pgmap v3591: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T01:57:11.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:57:11 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 5526.693588 secs 2024-03-24T01:57:11.500 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:57:11 smithi118 ceph-mon[31535]: pgmap v3591: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T01:57:11.500 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:57:11 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 5526.693588 secs 2024-03-24T01:57:13.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:57:13 smithi193 ceph-mon[36878]: pgmap v3592: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:57:13.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:57:13 smithi118 ceph-mon[31535]: pgmap v3592: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:57:13.704 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:57:13 smithi189 ceph-mon[36187]: pgmap v3592: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:57:15.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:57:15 smithi118 ceph-mon[31535]: pgmap v3593: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-24T01:57:15.704 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:57:15 smithi189 ceph-mon[36187]: pgmap v3593: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-24T01:57:15.711 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:57:15 smithi193 ceph-mon[36878]: pgmap v3593: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-24T01:57:16.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:57:16 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 5531.693680 secs 2024-03-24T01:57:16.704 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:57:16 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 5531.693680 secs 2024-03-24T01:57:16.711 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:57:16 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 5531.693680 secs 2024-03-24T01:57:17.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:57:17 smithi118 ceph-mon[31535]: pgmap v3594: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:57:17.704 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:57:17 smithi189 ceph-mon[36187]: pgmap v3594: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:57:17.711 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:57:17 smithi193 ceph-mon[36878]: pgmap v3594: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:57:19.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:57:19 smithi118 ceph-mon[31535]: pgmap v3595: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T01:57:19.704 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:57:19 smithi189 ceph-mon[36187]: pgmap v3595: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T01:57:19.711 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:57:19 smithi193 ceph-mon[36878]: pgmap v3595: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T01:57:21.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:57:21 smithi118 ceph-mon[31535]: pgmap v3596: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T01:57:21.697 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:57:21 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 5536.693768 secs 2024-03-24T01:57:21.704 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:57:21 smithi189 ceph-mon[36187]: pgmap v3596: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T01:57:21.704 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:57:21 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 5536.693768 secs 2024-03-24T01:57:21.711 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:57:21 smithi193 ceph-mon[36878]: pgmap v3596: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T01:57:21.711 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:57:21 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 5536.693768 secs 2024-03-24T01:57:23.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:57:23 smithi118 ceph-mon[31535]: pgmap v3597: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:57:23.704 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:57:23 smithi189 ceph-mon[36187]: pgmap v3597: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:57:23.711 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:57:23 smithi193 ceph-mon[36878]: pgmap v3597: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:57:25.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:57:25 smithi118 ceph-mon[31535]: pgmap v3598: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-24T01:57:25.704 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:57:25 smithi189 ceph-mon[36187]: pgmap v3598: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-24T01:57:25.711 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:57:25 smithi193 ceph-mon[36878]: pgmap v3598: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-24T01:57:26.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:57:26 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 5541.693860 secs 2024-03-24T01:57:26.704 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:57:26 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 5541.693860 secs 2024-03-24T01:57:26.711 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:57:26 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 5541.693860 secs 2024-03-24T01:57:27.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:57:27 smithi118 ceph-mon[31535]: pgmap v3599: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:57:27.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:57:27 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T01:57:27.704 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:57:27 smithi189 ceph-mon[36187]: pgmap v3599: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:57:27.704 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:57:27 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T01:57:27.711 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:57:27 smithi193 ceph-mon[36878]: pgmap v3599: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:57:27.711 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:57:27 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T01:57:28.697 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:57:28 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T01:57:28.697 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:57:28 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T01:57:28.697 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:57:28 smithi118 ceph-mon[31535]: pgmap v3600: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 5 op/s 2024-03-24T01:57:28.697 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:57:28 smithi118 ceph-mon[31535]: pgmap v3601: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 5 op/s 2024-03-24T01:57:28.697 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:57:28 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T01:57:28.704 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:57:28 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T01:57:28.704 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:57:28 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T01:57:28.704 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:57:28 smithi189 ceph-mon[36187]: pgmap v3600: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 5 op/s 2024-03-24T01:57:28.704 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:57:28 smithi189 ceph-mon[36187]: pgmap v3601: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 5 op/s 2024-03-24T01:57:28.704 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:57:28 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T01:57:28.711 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:57:28 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T01:57:28.711 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:57:28 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T01:57:28.711 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:57:28 smithi193 ceph-mon[36878]: pgmap v3600: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 5 op/s 2024-03-24T01:57:28.711 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:57:28 smithi193 ceph-mon[36878]: pgmap v3601: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 5 op/s 2024-03-24T01:57:28.711 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:57:28 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T01:57:31.197 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:57:30 smithi118 ceph-mon[31535]: pgmap v3602: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 6 op/s 2024-03-24T01:57:31.197 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:57:30 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 5546.693948 secs 2024-03-24T01:57:31.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:57:30 smithi189 ceph-mon[36187]: pgmap v3602: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 6 op/s 2024-03-24T01:57:31.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:57:30 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 5546.693948 secs 2024-03-24T01:57:31.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:57:30 smithi193 ceph-mon[36878]: pgmap v3602: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 6 op/s 2024-03-24T01:57:31.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:57:30 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 5546.693948 secs 2024-03-24T01:57:33.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:57:32 smithi118 ceph-mon[31535]: pgmap v3603: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 6 op/s 2024-03-24T01:57:33.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:57:32 smithi189 ceph-mon[36187]: pgmap v3603: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 6 op/s 2024-03-24T01:57:33.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:57:32 smithi193 ceph-mon[36878]: pgmap v3603: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 6 op/s 2024-03-24T01:57:35.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:57:34 smithi118 ceph-mon[31535]: pgmap v3604: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 4 op/s 2024-03-24T01:57:35.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:57:34 smithi189 ceph-mon[36187]: pgmap v3604: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 4 op/s 2024-03-24T01:57:35.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:57:34 smithi193 ceph-mon[36878]: pgmap v3604: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 4 op/s 2024-03-24T01:57:36.098 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:57:35 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 5551.694038 secs 2024-03-24T01:57:36.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:57:35 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 5551.694038 secs 2024-03-24T01:57:36.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:57:35 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 5551.694038 secs 2024-03-24T01:57:37.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:57:36 smithi118 ceph-mon[31535]: pgmap v3605: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 6 op/s 2024-03-24T01:57:37.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:57:36 smithi189 ceph-mon[36187]: pgmap v3605: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 6 op/s 2024-03-24T01:57:37.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:57:36 smithi193 ceph-mon[36878]: pgmap v3605: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 6 op/s 2024-03-24T01:57:39.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:57:38 smithi118 ceph-mon[31535]: pgmap v3606: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 5 op/s 2024-03-24T01:57:39.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:57:38 smithi189 ceph-mon[36187]: pgmap v3606: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 5 op/s 2024-03-24T01:57:39.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:57:38 smithi193 ceph-mon[36878]: pgmap v3606: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 5 op/s 2024-03-24T01:57:41.013 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:57:40 smithi193 ceph-mon[36878]: pgmap v3607: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-03-24T01:57:41.014 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:57:40 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 5556.694069 secs 2024-03-24T01:57:41.197 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:57:40 smithi118 ceph-mon[31535]: pgmap v3607: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-03-24T01:57:41.197 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:57:40 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 5556.694069 secs 2024-03-24T01:57:41.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:57:40 smithi189 ceph-mon[36187]: pgmap v3607: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-03-24T01:57:41.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:57:40 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 5556.694069 secs 2024-03-24T01:57:43.064 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:57:42 smithi118 ceph-mon[31535]: pgmap v3608: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:57:43.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:57:42 smithi189 ceph-mon[36187]: pgmap v3608: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:57:43.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:57:42 smithi193 ceph-mon[36878]: pgmap v3608: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:57:45.113 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:57:44 smithi118 ceph-mon[31535]: pgmap v3609: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T01:57:45.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:57:44 smithi189 ceph-mon[36187]: pgmap v3609: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T01:57:45.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:57:44 smithi193 ceph-mon[36878]: pgmap v3609: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T01:57:46.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:57:45 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 5561.694131 secs 2024-03-24T01:57:46.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:57:45 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 5561.694131 secs 2024-03-24T01:57:46.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:57:45 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 5561.694131 secs 2024-03-24T01:57:47.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:57:46 smithi118 ceph-mon[31535]: pgmap v3610: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T01:57:47.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:57:46 smithi189 ceph-mon[36187]: pgmap v3610: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T01:57:47.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:57:46 smithi193 ceph-mon[36878]: pgmap v3610: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T01:57:49.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:57:48 smithi118 ceph-mon[31535]: pgmap v3611: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 5 op/s 2024-03-24T01:57:49.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:57:48 smithi189 ceph-mon[36187]: pgmap v3611: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 5 op/s 2024-03-24T01:57:49.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:57:48 smithi193 ceph-mon[36878]: pgmap v3611: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 5 op/s 2024-03-24T01:57:51.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:57:50 smithi118 ceph-mon[31535]: pgmap v3612: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-03-24T01:57:51.197 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:57:50 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 5566.694215 secs 2024-03-24T01:57:51.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:57:50 smithi189 ceph-mon[36187]: pgmap v3612: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-03-24T01:57:51.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:57:50 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 5566.694215 secs 2024-03-24T01:57:51.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:57:50 smithi193 ceph-mon[36878]: pgmap v3612: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-03-24T01:57:51.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:57:50 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 5566.694215 secs 2024-03-24T01:57:53.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:57:52 smithi118 ceph-mon[31535]: pgmap v3613: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:57:53.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:57:52 smithi189 ceph-mon[36187]: pgmap v3613: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:57:53.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:57:52 smithi193 ceph-mon[36878]: pgmap v3613: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:57:55.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:57:54 smithi118 ceph-mon[31535]: pgmap v3614: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T01:57:55.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:57:54 smithi189 ceph-mon[36187]: pgmap v3614: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T01:57:55.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:57:54 smithi193 ceph-mon[36878]: pgmap v3614: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T01:57:56.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:57:55 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 5571.694302 secs 2024-03-24T01:57:56.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:57:55 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 5571.694302 secs 2024-03-24T01:57:56.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:57:55 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 5571.694302 secs 2024-03-24T01:57:57.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:57:56 smithi118 ceph-mon[31535]: pgmap v3615: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T01:57:57.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:57:56 smithi189 ceph-mon[36187]: pgmap v3615: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T01:57:57.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:57:56 smithi193 ceph-mon[36878]: pgmap v3615: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T01:57:59.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:57:58 smithi118 ceph-mon[31535]: pgmap v3616: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 5 op/s 2024-03-24T01:57:59.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:57:58 smithi189 ceph-mon[36187]: pgmap v3616: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 5 op/s 2024-03-24T01:57:59.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:57:58 smithi193 ceph-mon[36878]: pgmap v3616: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 5 op/s 2024-03-24T01:58:01.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:58:00 smithi118 ceph-mon[31535]: pgmap v3617: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T01:58:01.197 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:58:00 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 5576.694404 secs 2024-03-24T01:58:01.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:58:00 smithi189 ceph-mon[36187]: pgmap v3617: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T01:58:01.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:58:00 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 5576.694404 secs 2024-03-24T01:58:01.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:58:00 smithi193 ceph-mon[36878]: pgmap v3617: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T01:58:01.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:58:00 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 5576.694404 secs 2024-03-24T01:58:03.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:58:02 smithi118 ceph-mon[31535]: pgmap v3618: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:58:03.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:58:02 smithi189 ceph-mon[36187]: pgmap v3618: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:58:03.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:58:02 smithi193 ceph-mon[36878]: pgmap v3618: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:58:05.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:58:04 smithi118 ceph-mon[31535]: pgmap v3619: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T01:58:05.280 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:58:04 smithi193 ceph-mon[36878]: pgmap v3619: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T01:58:05.280 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:58:04 smithi189 ceph-mon[36187]: pgmap v3619: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T01:58:06.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:58:05 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 5581.694495 secs 2024-03-24T01:58:06.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:58:05 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 5581.694495 secs 2024-03-24T01:58:06.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:58:05 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 5581.694495 secs 2024-03-24T01:58:07.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:58:06 smithi118 ceph-mon[31535]: pgmap v3620: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T01:58:07.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:58:06 smithi189 ceph-mon[36187]: pgmap v3620: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T01:58:07.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:58:06 smithi193 ceph-mon[36878]: pgmap v3620: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T01:58:09.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:58:08 smithi189 ceph-mon[36187]: pgmap v3621: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 5 op/s 2024-03-24T01:58:09.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:58:08 smithi193 ceph-mon[36878]: pgmap v3621: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 5 op/s 2024-03-24T01:58:09.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:58:08 smithi118 ceph-mon[31535]: pgmap v3621: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 5 op/s 2024-03-24T01:58:11.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:58:10 smithi193 ceph-mon[36878]: pgmap v3622: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T01:58:11.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:58:10 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 5586.694558 secs 2024-03-24T01:58:11.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:58:10 smithi118 ceph-mon[31535]: pgmap v3622: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T01:58:11.447 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:58:10 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 5586.694558 secs 2024-03-24T01:58:11.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:58:10 smithi189 ceph-mon[36187]: pgmap v3622: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T01:58:11.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:58:10 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 5586.694558 secs 2024-03-24T01:58:13.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:58:12 smithi193 ceph-mon[36878]: pgmap v3623: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:58:13.273 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:58:12 smithi118 ceph-mon[31535]: pgmap v3623: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:58:13.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:58:12 smithi189 ceph-mon[36187]: pgmap v3623: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:58:15.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:58:14 smithi118 ceph-mon[31535]: pgmap v3624: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T01:58:15.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:58:14 smithi189 ceph-mon[36187]: pgmap v3624: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T01:58:15.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:58:14 smithi193 ceph-mon[36878]: pgmap v3624: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T01:58:16.344 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:58:15 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 5591.694646 secs 2024-03-24T01:58:16.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:58:15 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 5591.694646 secs 2024-03-24T01:58:16.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:58:15 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 5591.694646 secs 2024-03-24T01:58:16.632 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T01:58:16.631+0000 7f81db400640 1 -- 172.21.15.118:0/659686424 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f81c4006460 con 0x5622651a13f0 2024-03-24T01:58:16.632 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T01:58:16.631+0000 7f81ee000640 1 -- 172.21.15.118:0/1693767697 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f81c80036c0 con 0x7f81f81052d0 2024-03-24T01:58:16.632 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T01:58:16.632+0000 7f81ef400640 1 -- 172.21.15.118:0/1693767697 <== mon.2 v2:172.21.15.189:3300/0 47 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (secure 0 0 0) 0x7f81c80036c0 con 0x7f81f81052d0 2024-03-24T01:58:16.632 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T01:58:16.632+0000 7f81e4c00640 1 -- 172.21.15.118:0/659686424 <== mon.1 v2:172.21.15.193:3300/0 47 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 409+0+0 (secure 0 0 0) 0x7f81c4006460 con 0x5622651a13f0 2024-03-24T01:58:17.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:58:16 smithi118 ceph-mon[31535]: pgmap v3625: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T01:58:17.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:58:16 smithi189 ceph-mon[36187]: pgmap v3625: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T01:58:17.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:58:16 smithi193 ceph-mon[36878]: pgmap v3625: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T01:58:19.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:58:18 smithi118 ceph-mon[31535]: pgmap v3626: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 5 op/s 2024-03-24T01:58:19.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:58:18 smithi189 ceph-mon[36187]: pgmap v3626: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 5 op/s 2024-03-24T01:58:19.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:58:18 smithi193 ceph-mon[36878]: pgmap v3626: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 5 op/s 2024-03-24T01:58:21.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:58:20 smithi118 ceph-mon[31535]: pgmap v3627: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T01:58:21.447 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:58:20 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 5596.694731 secs 2024-03-24T01:58:21.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:58:20 smithi189 ceph-mon[36187]: pgmap v3627: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T01:58:21.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:58:20 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 5596.694731 secs 2024-03-24T01:58:21.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:58:20 smithi193 ceph-mon[36878]: pgmap v3627: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T01:58:21.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:58:20 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 5596.694731 secs 2024-03-24T01:58:23.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:58:22 smithi118 ceph-mon[31535]: pgmap v3628: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:58:23.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:58:22 smithi189 ceph-mon[36187]: pgmap v3628: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:58:23.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:58:22 smithi193 ceph-mon[36878]: pgmap v3628: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:58:25.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:58:24 smithi118 ceph-mon[31535]: pgmap v3629: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T01:58:25.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:58:24 smithi189 ceph-mon[36187]: pgmap v3629: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T01:58:25.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:58:24 smithi193 ceph-mon[36878]: pgmap v3629: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T01:58:26.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:58:26 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 5601.694820 secs 2024-03-24T01:58:26.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:58:26 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 5601.694820 secs 2024-03-24T01:58:26.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:58:26 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 5601.694820 secs 2024-03-24T01:58:27.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:58:27 smithi118 ceph-mon[31535]: pgmap v3630: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T01:58:27.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:58:27 smithi189 ceph-mon[36187]: pgmap v3630: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T01:58:27.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:58:27 smithi193 ceph-mon[36878]: pgmap v3630: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T01:58:28.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:58:28 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T01:58:28.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:58:28 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T01:58:28.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:58:28 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T01:58:29.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:58:29 smithi118 ceph-mon[31535]: pgmap v3631: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 5 op/s 2024-03-24T01:58:29.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:58:29 smithi189 ceph-mon[36187]: pgmap v3631: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 5 op/s 2024-03-24T01:58:29.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:58:29 smithi193 ceph-mon[36878]: pgmap v3631: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 5 op/s 2024-03-24T01:58:30.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:58:30 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T01:58:30.447 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:58:30 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T01:58:30.447 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:58:30 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T01:58:30.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:58:30 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T01:58:30.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:58:30 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T01:58:30.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:58:30 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T01:58:30.462 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:58:30 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T01:58:30.462 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:58:30 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T01:58:30.462 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:58:30 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T01:58:31.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:58:31 smithi118 ceph-mon[31535]: pgmap v3632: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 6 op/s 2024-03-24T01:58:31.447 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:58:31 smithi118 ceph-mon[31535]: pgmap v3633: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 6 op/s 2024-03-24T01:58:31.447 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:58:31 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 5606.694863 secs 2024-03-24T01:58:31.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:58:31 smithi189 ceph-mon[36187]: pgmap v3632: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 6 op/s 2024-03-24T01:58:31.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:58:31 smithi189 ceph-mon[36187]: pgmap v3633: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 6 op/s 2024-03-24T01:58:31.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:58:31 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 5606.694863 secs 2024-03-24T01:58:31.462 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:58:31 smithi193 ceph-mon[36878]: pgmap v3632: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 6 op/s 2024-03-24T01:58:31.462 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:58:31 smithi193 ceph-mon[36878]: pgmap v3633: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 6 op/s 2024-03-24T01:58:31.462 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:58:31 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 5606.694863 secs 2024-03-24T01:58:33.447 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:58:33 smithi118 ceph-mon[31535]: pgmap v3634: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 6 op/s 2024-03-24T01:58:33.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:58:33 smithi189 ceph-mon[36187]: pgmap v3634: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 6 op/s 2024-03-24T01:58:33.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:58:33 smithi193 ceph-mon[36878]: pgmap v3634: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 6 op/s 2024-03-24T01:58:35.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:58:35 smithi118 ceph-mon[31535]: pgmap v3635: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 6 op/s 2024-03-24T01:58:35.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:58:35 smithi189 ceph-mon[36187]: pgmap v3635: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 6 op/s 2024-03-24T01:58:35.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:58:35 smithi193 ceph-mon[36878]: pgmap v3635: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 6 op/s 2024-03-24T01:58:36.348 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:58:36 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 5611.694922 secs 2024-03-24T01:58:36.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:58:36 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 5611.694922 secs 2024-03-24T01:58:36.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:58:36 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 5611.694922 secs 2024-03-24T01:58:37.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:58:37 smithi118 ceph-mon[31535]: pgmap v3636: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 6 op/s 2024-03-24T01:58:37.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:58:37 smithi189 ceph-mon[36187]: pgmap v3636: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 6 op/s 2024-03-24T01:58:37.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:58:37 smithi193 ceph-mon[36878]: pgmap v3636: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 6 op/s 2024-03-24T01:58:39.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:58:39 smithi118 ceph-mon[31535]: pgmap v3637: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 5 op/s 2024-03-24T01:58:39.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:58:39 smithi189 ceph-mon[36187]: pgmap v3637: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 5 op/s 2024-03-24T01:58:39.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:58:39 smithi193 ceph-mon[36878]: pgmap v3637: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 5 op/s 2024-03-24T01:58:41.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:58:41 smithi118 ceph-mon[31535]: pgmap v3638: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T01:58:41.447 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:58:41 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 5616.695008 secs 2024-03-24T01:58:41.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:58:41 smithi189 ceph-mon[36187]: pgmap v3638: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T01:58:41.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:58:41 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 5616.695008 secs 2024-03-24T01:58:41.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:58:41 smithi193 ceph-mon[36878]: pgmap v3638: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T01:58:41.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:58:41 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 5616.695008 secs 2024-03-24T01:58:43.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:58:43 smithi118 ceph-mon[31535]: pgmap v3639: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-03-24T01:58:43.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:58:43 smithi189 ceph-mon[36187]: pgmap v3639: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-03-24T01:58:43.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:58:43 smithi193 ceph-mon[36878]: pgmap v3639: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-03-24T01:58:45.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:58:45 smithi118 ceph-mon[31535]: pgmap v3640: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-03-24T01:58:45.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:58:45 smithi189 ceph-mon[36187]: pgmap v3640: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-03-24T01:58:45.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:58:45 smithi193 ceph-mon[36878]: pgmap v3640: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-03-24T01:58:46.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:58:46 smithi118 ceph-mon[31535]: pgmap v3641: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 6 op/s 2024-03-24T01:58:46.447 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:58:46 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 5621.695109 secs 2024-03-24T01:58:46.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:58:46 smithi189 ceph-mon[36187]: pgmap v3641: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 6 op/s 2024-03-24T01:58:46.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:58:46 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 5621.695109 secs 2024-03-24T01:58:46.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:58:46 smithi193 ceph-mon[36878]: pgmap v3641: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 6 op/s 2024-03-24T01:58:46.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:58:46 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 5621.695109 secs 2024-03-24T01:58:48.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:58:48 smithi118 ceph-mon[31535]: pgmap v3642: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-03-24T01:58:48.705 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:58:48 smithi189 ceph-mon[36187]: pgmap v3642: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-03-24T01:58:48.711 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:58:48 smithi193 ceph-mon[36878]: pgmap v3642: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-03-24T01:58:50.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:58:50 smithi118 ceph-mon[31535]: pgmap v3643: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-24T01:58:50.704 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:58:50 smithi189 ceph-mon[36187]: pgmap v3643: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-24T01:58:50.711 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:58:50 smithi193 ceph-mon[36878]: pgmap v3643: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-24T01:58:51.697 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:58:51 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 5626.695169 secs 2024-03-24T01:58:51.704 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:58:51 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 5626.695169 secs 2024-03-24T01:58:51.711 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:58:51 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 5626.695169 secs 2024-03-24T01:58:52.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:58:52 smithi118 ceph-mon[31535]: pgmap v3644: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:58:52.704 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:58:52 smithi189 ceph-mon[36187]: pgmap v3644: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:58:52.711 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:58:52 smithi193 ceph-mon[36878]: pgmap v3644: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:58:54.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:58:54 smithi118 ceph-mon[31535]: pgmap v3645: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T01:58:54.704 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:58:54 smithi189 ceph-mon[36187]: pgmap v3645: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T01:58:54.711 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:58:54 smithi193 ceph-mon[36878]: pgmap v3645: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T01:58:56.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:58:56 smithi118 ceph-mon[31535]: pgmap v3646: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T01:58:56.697 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:58:56 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 5631.695207 secs 2024-03-24T01:58:56.704 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:58:56 smithi189 ceph-mon[36187]: pgmap v3646: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T01:58:56.704 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:58:56 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 5631.695207 secs 2024-03-24T01:58:56.711 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:58:56 smithi193 ceph-mon[36878]: pgmap v3646: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T01:58:56.711 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:58:56 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 5631.695207 secs 2024-03-24T01:58:58.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:58:58 smithi118 ceph-mon[31535]: pgmap v3647: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:58:58.704 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:58:58 smithi189 ceph-mon[36187]: pgmap v3647: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:58:58.711 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:58:58 smithi193 ceph-mon[36878]: pgmap v3647: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:59:00.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:59:00 smithi118 ceph-mon[31535]: pgmap v3648: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-24T01:59:00.704 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:59:00 smithi189 ceph-mon[36187]: pgmap v3648: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-24T01:59:00.711 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:59:00 smithi193 ceph-mon[36878]: pgmap v3648: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-24T01:59:01.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:59:01 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 5636.695295 secs 2024-03-24T01:59:01.704 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:59:01 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 5636.695295 secs 2024-03-24T01:59:01.711 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:59:01 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 5636.695295 secs 2024-03-24T01:59:02.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:59:02 smithi118 ceph-mon[31535]: pgmap v3649: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:59:02.704 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:59:02 smithi189 ceph-mon[36187]: pgmap v3649: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:59:02.711 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:59:02 smithi193 ceph-mon[36878]: pgmap v3649: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:59:04.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:59:04 smithi118 ceph-mon[31535]: pgmap v3650: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T01:59:04.704 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:59:04 smithi189 ceph-mon[36187]: pgmap v3650: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T01:59:04.711 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:59:04 smithi193 ceph-mon[36878]: pgmap v3650: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T01:59:06.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:59:06 smithi118 ceph-mon[31535]: pgmap v3651: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T01:59:06.697 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:59:06 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 5641.695328 secs 2024-03-24T01:59:06.704 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:59:06 smithi189 ceph-mon[36187]: pgmap v3651: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T01:59:06.704 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:59:06 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 5641.695328 secs 2024-03-24T01:59:06.711 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:59:06 smithi193 ceph-mon[36878]: pgmap v3651: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T01:59:06.711 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:59:06 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 5641.695328 secs 2024-03-24T01:59:08.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:59:08 smithi118 ceph-mon[31535]: pgmap v3652: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:59:08.704 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:59:08 smithi189 ceph-mon[36187]: pgmap v3652: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:59:08.711 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:59:08 smithi193 ceph-mon[36878]: pgmap v3652: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:59:10.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:59:10 smithi118 ceph-mon[31535]: pgmap v3653: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-24T01:59:10.704 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:59:10 smithi189 ceph-mon[36187]: pgmap v3653: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-24T01:59:10.711 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:59:10 smithi193 ceph-mon[36878]: pgmap v3653: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-24T01:59:11.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:59:11 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 5646.695416 secs 2024-03-24T01:59:11.704 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:59:11 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 5646.695416 secs 2024-03-24T01:59:11.711 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:59:11 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 5646.695416 secs 2024-03-24T01:59:12.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:59:12 smithi118 ceph-mon[31535]: pgmap v3654: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:59:12.704 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:59:12 smithi189 ceph-mon[36187]: pgmap v3654: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:59:12.711 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:59:12 smithi193 ceph-mon[36878]: pgmap v3654: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:59:14.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:59:14 smithi118 ceph-mon[31535]: pgmap v3655: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T01:59:14.704 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:59:14 smithi189 ceph-mon[36187]: pgmap v3655: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T01:59:14.711 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:59:14 smithi193 ceph-mon[36878]: pgmap v3655: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T01:59:16.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:59:16 smithi118 ceph-mon[31535]: pgmap v3656: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T01:59:16.697 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:59:16 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 5651.695503 secs 2024-03-24T01:59:16.704 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:59:16 smithi189 ceph-mon[36187]: pgmap v3656: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T01:59:16.704 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:59:16 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 5651.695503 secs 2024-03-24T01:59:16.711 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:59:16 smithi193 ceph-mon[36878]: pgmap v3656: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T01:59:16.711 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:59:16 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 5651.695503 secs 2024-03-24T01:59:18.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:59:18 smithi118 ceph-mon[31535]: pgmap v3657: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:59:18.704 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:59:18 smithi189 ceph-mon[36187]: pgmap v3657: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:59:18.711 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:59:18 smithi193 ceph-mon[36878]: pgmap v3657: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:59:20.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:59:20 smithi118 ceph-mon[31535]: pgmap v3658: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-24T01:59:20.704 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:59:20 smithi189 ceph-mon[36187]: pgmap v3658: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-24T01:59:20.711 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:59:20 smithi193 ceph-mon[36878]: pgmap v3658: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-24T01:59:21.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:59:21 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 5656.695589 secs 2024-03-24T01:59:21.704 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:59:21 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 5656.695589 secs 2024-03-24T01:59:21.711 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:59:21 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 5656.695589 secs 2024-03-24T01:59:22.697 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:59:22 smithi118 ceph-mon[31535]: pgmap v3659: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:59:22.704 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:59:22 smithi189 ceph-mon[36187]: pgmap v3659: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:59:22.711 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:59:22 smithi193 ceph-mon[36878]: pgmap v3659: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:59:24.697 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:59:24 smithi118 ceph-mon[31535]: pgmap v3660: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T01:59:24.704 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:59:24 smithi189 ceph-mon[36187]: pgmap v3660: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T01:59:24.711 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:59:24 smithi193 ceph-mon[36878]: pgmap v3660: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T01:59:26.697 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:59:26 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 5661.695671 secs 2024-03-24T01:59:26.697 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:59:26 smithi118 ceph-mon[31535]: pgmap v3661: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T01:59:26.704 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:59:26 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 5661.695671 secs 2024-03-24T01:59:26.704 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:59:26 smithi189 ceph-mon[36187]: pgmap v3661: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T01:59:26.711 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:59:26 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 5661.695671 secs 2024-03-24T01:59:26.711 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:59:26 smithi193 ceph-mon[36878]: pgmap v3661: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T01:59:28.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:59:28 smithi118 ceph-mon[31535]: pgmap v3662: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:59:28.704 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:59:28 smithi189 ceph-mon[36187]: pgmap v3662: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:59:28.711 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:59:28 smithi193 ceph-mon[36878]: pgmap v3662: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:59:30.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:59:30 smithi118 ceph-mon[31535]: pgmap v3663: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-24T01:59:30.697 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:59:30 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T01:59:30.697 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:59:30 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T01:59:30.697 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:59:30 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T01:59:30.697 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:59:30 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T01:59:30.697 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:59:30 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T01:59:30.704 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:59:30 smithi189 ceph-mon[36187]: pgmap v3663: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-24T01:59:30.704 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:59:30 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T01:59:30.704 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:59:30 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T01:59:30.704 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:59:30 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T01:59:30.704 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:59:30 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T01:59:30.704 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:59:30 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T01:59:30.711 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:59:30 smithi193 ceph-mon[36878]: pgmap v3663: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-24T01:59:30.711 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:59:30 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T01:59:30.711 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:59:30 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T01:59:30.712 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:59:30 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T01:59:30.712 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:59:30 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T01:59:30.712 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:59:30 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T01:59:31.697 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:59:31 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 5666.695766 secs 2024-03-24T01:59:31.704 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:59:31 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 5666.695766 secs 2024-03-24T01:59:31.711 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:59:31 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 5666.695766 secs 2024-03-24T01:59:32.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:59:32 smithi118 ceph-mon[31535]: pgmap v3664: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:59:32.947 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:59:32 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T01:59:32.947 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:59:32 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T01:59:32.947 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:59:32 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T01:59:32.947 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:59:32 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T01:59:32.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:59:32 smithi189 ceph-mon[36187]: pgmap v3664: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:59:32.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:59:32 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T01:59:32.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:59:32 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T01:59:32.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:59:32 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T01:59:32.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:59:32 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T01:59:32.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:59:32 smithi193 ceph-mon[36878]: pgmap v3664: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:59:32.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:59:32 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T01:59:32.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:59:32 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T01:59:32.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:59:32 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T01:59:32.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:59:32 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T01:59:34.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:59:34 smithi118 ceph-mon[31535]: pgmap v3665: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T01:59:34.947 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:59:34 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T01:59:34.947 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:59:34 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T01:59:34.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:59:34 smithi189 ceph-mon[36187]: pgmap v3665: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T01:59:34.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:59:34 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T01:59:34.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:59:34 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T01:59:34.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:59:34 smithi193 ceph-mon[36878]: pgmap v3665: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T01:59:34.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:59:34 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T01:59:34.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:59:34 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T01:59:35.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:59:35 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T01:59:35.947 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:59:35 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T01:59:35.947 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:59:35 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T01:59:35.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:59:35 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T01:59:35.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:59:35 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T01:59:35.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:59:35 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T01:59:35.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:59:35 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T01:59:35.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:59:35 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T01:59:35.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:59:35 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T01:59:36.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:59:36 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 5671.695855 secs 2024-03-24T01:59:36.947 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:59:36 smithi118 ceph-mon[31535]: pgmap v3666: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T01:59:36.947 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:59:36 smithi118 ceph-mon[31535]: pgmap v3667: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T01:59:36.947 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:59:36 smithi118 ceph-mon[31535]: pgmap v3668: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-03-24T01:59:36.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:59:36 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 5671.695855 secs 2024-03-24T01:59:36.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:59:36 smithi189 ceph-mon[36187]: pgmap v3666: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T01:59:36.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:59:36 smithi189 ceph-mon[36187]: pgmap v3667: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T01:59:36.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:59:36 smithi189 ceph-mon[36187]: pgmap v3668: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-03-24T01:59:36.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:59:36 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 5671.695855 secs 2024-03-24T01:59:36.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:59:36 smithi193 ceph-mon[36878]: pgmap v3666: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T01:59:36.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:59:36 smithi193 ceph-mon[36878]: pgmap v3667: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T01:59:36.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:59:36 smithi193 ceph-mon[36878]: pgmap v3668: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-03-24T01:59:38.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:59:38 smithi118 ceph-mon[31535]: pgmap v3669: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-03-24T01:59:38.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:59:38 smithi189 ceph-mon[36187]: pgmap v3669: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-03-24T01:59:38.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:59:38 smithi193 ceph-mon[36878]: pgmap v3669: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-03-24T01:59:40.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:59:40 smithi118 ceph-mon[31535]: pgmap v3670: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-03-24T01:59:40.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:59:40 smithi189 ceph-mon[36187]: pgmap v3670: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-03-24T01:59:40.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:59:40 smithi193 ceph-mon[36878]: pgmap v3670: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-03-24T01:59:41.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:59:41 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 5676.695915 secs 2024-03-24T01:59:41.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:59:41 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 5676.695915 secs 2024-03-24T01:59:41.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:59:41 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 5676.695915 secs 2024-03-24T01:59:42.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:59:42 smithi118 ceph-mon[31535]: pgmap v3671: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 5 op/s 2024-03-24T01:59:42.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:59:42 smithi189 ceph-mon[36187]: pgmap v3671: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 5 op/s 2024-03-24T01:59:42.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:59:42 smithi193 ceph-mon[36878]: pgmap v3671: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 5 op/s 2024-03-24T01:59:44.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:59:44 smithi118 ceph-mon[31535]: pgmap v3672: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 5 op/s 2024-03-24T01:59:44.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:59:44 smithi189 ceph-mon[36187]: pgmap v3672: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 5 op/s 2024-03-24T01:59:44.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:59:44 smithi193 ceph-mon[36878]: pgmap v3672: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 5 op/s 2024-03-24T01:59:46.949 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:59:46 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 5681.696001 secs 2024-03-24T01:59:46.949 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:59:46 smithi118 ceph-mon[31535]: pgmap v3673: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 5 op/s 2024-03-24T01:59:46.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:59:46 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 5681.696001 secs 2024-03-24T01:59:46.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:59:46 smithi189 ceph-mon[36187]: pgmap v3673: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 5 op/s 2024-03-24T01:59:46.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:59:46 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 5681.696001 secs 2024-03-24T01:59:46.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:59:46 smithi193 ceph-mon[36878]: pgmap v3673: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 5 op/s 2024-03-24T01:59:48.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:59:48 smithi189 ceph-mon[36187]: pgmap v3674: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T01:59:48.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:59:48 smithi193 ceph-mon[36878]: pgmap v3674: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T01:59:49.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:59:48 smithi118 ceph-mon[31535]: pgmap v3674: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T01:59:50.962 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:59:50 smithi193 ceph-mon[36878]: pgmap v3675: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-03-24T01:59:51.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:59:50 smithi118 ceph-mon[31535]: pgmap v3675: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-03-24T01:59:51.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:59:50 smithi189 ceph-mon[36187]: pgmap v3675: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-03-24T01:59:52.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:59:51 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 5686.696088 secs 2024-03-24T01:59:52.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:59:51 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 5686.696088 secs 2024-03-24T01:59:52.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:59:51 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 5686.696088 secs 2024-03-24T01:59:52.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:59:52 smithi193 ceph-mon[36878]: pgmap v3676: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:59:53.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:59:52 smithi118 ceph-mon[31535]: pgmap v3676: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:59:53.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:59:52 smithi189 ceph-mon[36187]: pgmap v3676: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T01:59:55.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:59:54 smithi118 ceph-mon[31535]: pgmap v3677: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T01:59:55.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:59:54 smithi189 ceph-mon[36187]: pgmap v3677: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T01:59:55.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:59:54 smithi193 ceph-mon[36878]: pgmap v3677: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T01:59:57.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:59:56 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 5691.696176 secs 2024-03-24T01:59:57.197 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:59:56 smithi118 ceph-mon[31535]: pgmap v3678: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T01:59:57.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:59:56 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 5691.696176 secs 2024-03-24T01:59:57.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:59:56 smithi189 ceph-mon[36187]: pgmap v3678: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T01:59:57.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:59:56 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 5691.696176 secs 2024-03-24T01:59:57.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:59:56 smithi193 ceph-mon[36878]: pgmap v3678: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T01:59:59.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 01:59:58 smithi118 ceph-mon[31535]: pgmap v3679: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T01:59:59.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 01:59:58 smithi189 ceph-mon[36187]: pgmap v3679: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T01:59:59.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 01:59:58 smithi193 ceph-mon[36878]: pgmap v3679: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T02:00:01.197 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:00:00 smithi118 ceph-mon[31535]: pgmap v3680: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-03-24T02:00:01.197 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:00:00 smithi118 ceph-mon[31535]: overall HEALTH_WARN 6 failed cephadm daemon(s); 1 MDSs report slow requests 2024-03-24T02:00:01.197 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:00:00 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 5696.696265 secs 2024-03-24T02:00:01.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:00:00 smithi189 ceph-mon[36187]: pgmap v3680: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-03-24T02:00:01.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:00:00 smithi189 ceph-mon[36187]: overall HEALTH_WARN 6 failed cephadm daemon(s); 1 MDSs report slow requests 2024-03-24T02:00:01.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:00:00 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 5696.696265 secs 2024-03-24T02:00:01.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:00:00 smithi193 ceph-mon[36878]: pgmap v3680: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-03-24T02:00:01.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:00:00 smithi193 ceph-mon[36878]: overall HEALTH_WARN 6 failed cephadm daemon(s); 1 MDSs report slow requests 2024-03-24T02:00:01.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:00:00 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 5696.696265 secs 2024-03-24T02:00:03.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:00:02 smithi118 ceph-mon[31535]: pgmap v3681: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-03-24T02:00:03.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:00:02 smithi189 ceph-mon[36187]: pgmap v3681: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-03-24T02:00:03.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:00:02 smithi193 ceph-mon[36878]: pgmap v3681: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-03-24T02:00:05.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:00:04 smithi118 ceph-mon[31535]: pgmap v3682: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-03-24T02:00:05.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:00:04 smithi189 ceph-mon[36187]: pgmap v3682: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-03-24T02:00:05.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:00:04 smithi193 ceph-mon[36878]: pgmap v3682: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-03-24T02:00:06.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:00:06 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 5701.696347 secs 2024-03-24T02:00:06.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:00:06 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 5701.696347 secs 2024-03-24T02:00:06.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:00:06 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 5701.696347 secs 2024-03-24T02:00:07.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:00:07 smithi118 ceph-mon[31535]: pgmap v3683: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 6 op/s 2024-03-24T02:00:07.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:00:07 smithi189 ceph-mon[36187]: pgmap v3683: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 6 op/s 2024-03-24T02:00:07.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:00:07 smithi193 ceph-mon[36878]: pgmap v3683: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 6 op/s 2024-03-24T02:00:09.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:00:09 smithi118 ceph-mon[31535]: pgmap v3684: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 5 op/s 2024-03-24T02:00:09.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:00:09 smithi189 ceph-mon[36187]: pgmap v3684: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 5 op/s 2024-03-24T02:00:09.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:00:09 smithi193 ceph-mon[36878]: pgmap v3684: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 5 op/s 2024-03-24T02:00:11.374 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:00:11 smithi189 ceph-mon[36187]: pgmap v3685: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-03-24T02:00:11.374 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:00:11 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 5706.696436 secs 2024-03-24T02:00:11.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:00:11 smithi118 ceph-mon[31535]: pgmap v3685: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-03-24T02:00:11.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:00:11 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 5706.696436 secs 2024-03-24T02:00:11.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:00:11 smithi193 ceph-mon[36878]: pgmap v3685: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-03-24T02:00:11.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:00:11 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 5706.696436 secs 2024-03-24T02:00:13.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:00:13 smithi118 ceph-mon[31535]: pgmap v3686: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:00:13.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:00:13 smithi189 ceph-mon[36187]: pgmap v3686: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:00:13.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:00:13 smithi193 ceph-mon[36878]: pgmap v3686: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:00:15.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:00:15 smithi118 ceph-mon[31535]: pgmap v3687: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T02:00:15.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:00:15 smithi189 ceph-mon[36187]: pgmap v3687: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T02:00:15.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:00:15 smithi193 ceph-mon[36878]: pgmap v3687: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T02:00:16.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:00:16 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 5711.696526 secs 2024-03-24T02:00:16.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:00:16 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 5711.696526 secs 2024-03-24T02:00:16.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:00:16 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 5711.696526 secs 2024-03-24T02:00:17.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:00:17 smithi118 ceph-mon[31535]: pgmap v3688: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T02:00:17.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:00:17 smithi189 ceph-mon[36187]: pgmap v3688: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T02:00:17.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:00:17 smithi193 ceph-mon[36878]: pgmap v3688: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T02:00:19.447 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:00:19 smithi118 ceph-mon[31535]: pgmap v3689: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T02:00:19.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:00:19 smithi189 ceph-mon[36187]: pgmap v3689: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T02:00:19.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:00:19 smithi193 ceph-mon[36878]: pgmap v3689: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T02:00:21.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:00:21 smithi118 ceph-mon[31535]: pgmap v3690: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-03-24T02:00:21.447 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:00:21 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 5716.696616 secs 2024-03-24T02:00:21.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:00:21 smithi189 ceph-mon[36187]: pgmap v3690: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-03-24T02:00:21.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:00:21 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 5716.696616 secs 2024-03-24T02:00:21.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:00:21 smithi193 ceph-mon[36878]: pgmap v3690: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-03-24T02:00:21.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:00:21 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 5716.696616 secs 2024-03-24T02:00:23.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:00:23 smithi118 ceph-mon[31535]: pgmap v3691: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:00:23.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:00:23 smithi189 ceph-mon[36187]: pgmap v3691: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:00:23.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:00:23 smithi193 ceph-mon[36878]: pgmap v3691: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:00:25.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:00:25 smithi118 ceph-mon[31535]: pgmap v3692: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T02:00:25.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:00:25 smithi189 ceph-mon[36187]: pgmap v3692: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T02:00:25.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:00:25 smithi193 ceph-mon[36878]: pgmap v3692: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T02:00:26.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:00:26 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 5721.696706 secs 2024-03-24T02:00:26.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:00:26 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 5721.696706 secs 2024-03-24T02:00:26.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:00:26 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 5721.696706 secs 2024-03-24T02:00:27.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:00:27 smithi118 ceph-mon[31535]: pgmap v3693: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T02:00:27.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:00:27 smithi189 ceph-mon[36187]: pgmap v3693: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T02:00:27.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:00:27 smithi193 ceph-mon[36878]: pgmap v3693: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T02:00:29.450 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:00:29 smithi118 ceph-mon[31535]: pgmap v3694: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T02:00:29.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:00:29 smithi189 ceph-mon[36187]: pgmap v3694: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T02:00:29.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:00:29 smithi193 ceph-mon[36878]: pgmap v3694: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T02:00:31.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:00:31 smithi118 ceph-mon[31535]: pgmap v3695: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-03-24T02:00:31.447 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:00:31 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 5726.696794 secs 2024-03-24T02:00:31.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:00:31 smithi189 ceph-mon[36187]: pgmap v3695: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-03-24T02:00:31.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:00:31 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 5726.696794 secs 2024-03-24T02:00:31.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:00:31 smithi193 ceph-mon[36878]: pgmap v3695: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-03-24T02:00:31.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:00:31 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 5726.696794 secs 2024-03-24T02:00:33.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:00:33 smithi118 ceph-mon[31535]: pgmap v3696: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:00:33.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:00:33 smithi189 ceph-mon[36187]: pgmap v3696: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:00:33.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:00:33 smithi193 ceph-mon[36878]: pgmap v3696: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:00:34.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:00:34 smithi118 ceph-mon[31535]: pgmap v3697: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T02:00:34.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:00:34 smithi189 ceph-mon[36187]: pgmap v3697: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T02:00:34.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:00:34 smithi193 ceph-mon[36878]: pgmap v3697: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T02:00:36.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:00:36 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 5731.696882 secs 2024-03-24T02:00:36.947 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:00:36 smithi118 ceph-mon[31535]: pgmap v3698: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T02:00:36.947 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:00:36 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T02:00:36.947 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:00:36 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T02:00:36.947 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:00:36 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T02:00:36.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:00:36 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 5731.696882 secs 2024-03-24T02:00:36.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:00:36 smithi189 ceph-mon[36187]: pgmap v3698: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T02:00:36.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:00:36 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T02:00:36.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:00:36 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T02:00:36.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:00:36 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T02:00:36.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:00:36 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 5731.696882 secs 2024-03-24T02:00:36.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:00:36 smithi193 ceph-mon[36878]: pgmap v3698: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T02:00:36.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:00:36 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T02:00:36.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:00:36 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T02:00:36.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:00:36 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T02:00:37.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:00:37 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T02:00:37.947 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:00:37 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T02:00:37.947 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:00:37 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T02:00:37.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:00:37 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T02:00:37.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:00:37 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T02:00:37.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:00:37 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T02:00:37.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:00:37 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T02:00:37.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:00:37 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T02:00:37.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:00:37 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T02:00:38.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:00:38 smithi118 ceph-mon[31535]: pgmap v3699: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 5 op/s 2024-03-24T02:00:38.947 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:00:38 smithi118 ceph-mon[31535]: pgmap v3700: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 5 op/s 2024-03-24T02:00:38.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:00:38 smithi189 ceph-mon[36187]: pgmap v3699: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 5 op/s 2024-03-24T02:00:38.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:00:38 smithi189 ceph-mon[36187]: pgmap v3700: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 5 op/s 2024-03-24T02:00:38.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:00:38 smithi193 ceph-mon[36878]: pgmap v3699: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 5 op/s 2024-03-24T02:00:38.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:00:38 smithi193 ceph-mon[36878]: pgmap v3700: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 5 op/s 2024-03-24T02:00:40.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:00:40 smithi118 ceph-mon[31535]: pgmap v3701: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 6 op/s 2024-03-24T02:00:40.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:00:40 smithi189 ceph-mon[36187]: pgmap v3701: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 6 op/s 2024-03-24T02:00:40.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:00:40 smithi193 ceph-mon[36878]: pgmap v3701: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 6 op/s 2024-03-24T02:00:41.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:00:41 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 5736.696970 secs 2024-03-24T02:00:41.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:00:41 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 5736.696970 secs 2024-03-24T02:00:41.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:00:41 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 5736.696970 secs 2024-03-24T02:00:42.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:00:42 smithi118 ceph-mon[31535]: pgmap v3702: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 6 op/s 2024-03-24T02:00:42.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:00:42 smithi189 ceph-mon[36187]: pgmap v3702: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 6 op/s 2024-03-24T02:00:42.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:00:42 smithi193 ceph-mon[36878]: pgmap v3702: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 6 op/s 2024-03-24T02:00:44.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:00:44 smithi118 ceph-mon[31535]: pgmap v3703: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 6 op/s 2024-03-24T02:00:44.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:00:44 smithi189 ceph-mon[36187]: pgmap v3703: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 6 op/s 2024-03-24T02:00:44.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:00:44 smithi193 ceph-mon[36878]: pgmap v3703: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 6 op/s 2024-03-24T02:00:46.947 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:00:46 smithi118 ceph-mon[31535]: pgmap v3704: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 6 op/s 2024-03-24T02:00:46.947 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:00:46 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 5741.697062 secs 2024-03-24T02:00:46.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:00:46 smithi189 ceph-mon[36187]: pgmap v3704: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 6 op/s 2024-03-24T02:00:46.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:00:46 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 5741.697062 secs 2024-03-24T02:00:46.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:00:46 smithi193 ceph-mon[36878]: pgmap v3704: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 6 op/s 2024-03-24T02:00:46.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:00:46 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 5741.697062 secs 2024-03-24T02:00:48.947 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:00:48 smithi118 ceph-mon[31535]: pgmap v3705: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 5 op/s 2024-03-24T02:00:48.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:00:48 smithi189 ceph-mon[36187]: pgmap v3705: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 5 op/s 2024-03-24T02:00:48.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:00:48 smithi193 ceph-mon[36878]: pgmap v3705: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 5 op/s 2024-03-24T02:00:50.947 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:00:50 smithi118 ceph-mon[31535]: pgmap v3706: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-24T02:00:50.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:00:50 smithi189 ceph-mon[36187]: pgmap v3706: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-24T02:00:50.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:00:50 smithi193 ceph-mon[36878]: pgmap v3706: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-24T02:00:51.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:00:51 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 5746.697144 secs 2024-03-24T02:00:51.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:00:51 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 5746.697144 secs 2024-03-24T02:00:51.992 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:00:51 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 5746.697144 secs 2024-03-24T02:00:52.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:00:52 smithi189 ceph-mon[36187]: pgmap v3707: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:00:52.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:00:52 smithi193 ceph-mon[36878]: pgmap v3707: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:00:53.197 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:00:52 smithi118 ceph-mon[31535]: pgmap v3707: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:00:54.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:00:54 smithi193 ceph-mon[36878]: pgmap v3708: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T02:00:55.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:00:54 smithi118 ceph-mon[31535]: pgmap v3708: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T02:00:55.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:00:54 smithi189 ceph-mon[36187]: pgmap v3708: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T02:00:56.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:00:56 smithi193 ceph-mon[36878]: pgmap v3709: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T02:00:56.962 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:00:56 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 5751.697205 secs 2024-03-24T02:00:57.197 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:00:56 smithi118 ceph-mon[31535]: pgmap v3709: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T02:00:57.197 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:00:56 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 5751.697205 secs 2024-03-24T02:00:57.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:00:56 smithi189 ceph-mon[36187]: pgmap v3709: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T02:00:57.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:00:56 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 5751.697205 secs 2024-03-24T02:00:59.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:00:58 smithi118 ceph-mon[31535]: pgmap v3710: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:00:59.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:00:58 smithi189 ceph-mon[36187]: pgmap v3710: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:00:59.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:00:58 smithi193 ceph-mon[36878]: pgmap v3710: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:01:00.947 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:01:00 smithi118 ceph-mon[31535]: pgmap v3711: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-24T02:01:01.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:01:00 smithi189 ceph-mon[36187]: pgmap v3711: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-24T02:01:01.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:01:00 smithi193 ceph-mon[36878]: pgmap v3711: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-24T02:01:02.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:01:01 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 5756.697266 secs 2024-03-24T02:01:02.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:01:01 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 5756.697266 secs 2024-03-24T02:01:02.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:01:01 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 5756.697266 secs 2024-03-24T02:01:03.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:01:02 smithi118 ceph-mon[31535]: pgmap v3712: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:01:03.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:01:02 smithi189 ceph-mon[36187]: pgmap v3712: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:01:03.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:01:02 smithi193 ceph-mon[36878]: pgmap v3712: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:01:05.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:01:04 smithi118 ceph-mon[31535]: pgmap v3713: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T02:01:05.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:01:04 smithi189 ceph-mon[36187]: pgmap v3713: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T02:01:05.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:01:04 smithi193 ceph-mon[36878]: pgmap v3713: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T02:01:06.099 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:01:05 smithi118 ceph-mon[31535]: 3 slow requests, 3 included below; oldest blocked for > 5761.697357 secs 2024-03-24T02:01:06.099 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:01:05 smithi118 ceph-mon[31535]: slow request 5761.697357 seconds old, received at 2024-03-24T00:25:03.572253+0000: peer_request:client.15159:154789 currently acquired locks 2024-03-24T02:01:06.099 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:01:05 smithi118 ceph-mon[31535]: slow request 5761.646110 seconds old, received at 2024-03-24T00:25:03.623500+0000: peer_request:client.15159:158090 currently acquired locks 2024-03-24T02:01:06.099 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:01:05 smithi118 ceph-mon[31535]: slow request 5761.639431 seconds old, received at 2024-03-24T00:25:03.630178+0000: client_request(client.15159:173196 rename #0x100000010bf/comment-15.xml #0x100000010bf/comment-15.xml.tmp 2024-03-24T00:25:03.629366+0000 caller_uid=1000, caller_gid=1301{6,36,1000,1301,}) currently failed to xlock, waiting 2024-03-24T02:01:06.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:01:05 smithi189 ceph-mon[36187]: 3 slow requests, 3 included below; oldest blocked for > 5761.697357 secs 2024-03-24T02:01:06.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:01:05 smithi189 ceph-mon[36187]: slow request 5761.697357 seconds old, received at 2024-03-24T00:25:03.572253+0000: peer_request:client.15159:154789 currently acquired locks 2024-03-24T02:01:06.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:01:05 smithi189 ceph-mon[36187]: slow request 5761.646110 seconds old, received at 2024-03-24T00:25:03.623500+0000: peer_request:client.15159:158090 currently acquired locks 2024-03-24T02:01:06.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:01:05 smithi189 ceph-mon[36187]: slow request 5761.639431 seconds old, received at 2024-03-24T00:25:03.630178+0000: client_request(client.15159:173196 rename #0x100000010bf/comment-15.xml #0x100000010bf/comment-15.xml.tmp 2024-03-24T00:25:03.629366+0000 caller_uid=1000, caller_gid=1301{6,36,1000,1301,}) currently failed to xlock, waiting 2024-03-24T02:01:06.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:01:05 smithi193 ceph-mon[36878]: 3 slow requests, 3 included below; oldest blocked for > 5761.697357 secs 2024-03-24T02:01:06.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:01:05 smithi193 ceph-mon[36878]: slow request 5761.697357 seconds old, received at 2024-03-24T00:25:03.572253+0000: peer_request:client.15159:154789 currently acquired locks 2024-03-24T02:01:06.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:01:05 smithi193 ceph-mon[36878]: slow request 5761.646110 seconds old, received at 2024-03-24T00:25:03.623500+0000: peer_request:client.15159:158090 currently acquired locks 2024-03-24T02:01:06.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:01:05 smithi193 ceph-mon[36878]: slow request 5761.639431 seconds old, received at 2024-03-24T00:25:03.630178+0000: client_request(client.15159:173196 rename #0x100000010bf/comment-15.xml #0x100000010bf/comment-15.xml.tmp 2024-03-24T00:25:03.629366+0000 caller_uid=1000, caller_gid=1301{6,36,1000,1301,}) currently failed to xlock, waiting 2024-03-24T02:01:07.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:01:06 smithi118 ceph-mon[31535]: pgmap v3714: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T02:01:07.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:01:06 smithi189 ceph-mon[36187]: pgmap v3714: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T02:01:07.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:01:06 smithi193 ceph-mon[36878]: pgmap v3714: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T02:01:09.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:01:08 smithi118 ceph-mon[31535]: pgmap v3715: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:01:09.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:01:08 smithi189 ceph-mon[36187]: pgmap v3715: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:01:09.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:01:08 smithi193 ceph-mon[36878]: pgmap v3715: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:01:11.012 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:01:10 smithi193 ceph-mon[36878]: pgmap v3716: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-24T02:01:11.012 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:01:10 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 5766.697439 secs 2024-03-24T02:01:11.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:01:10 smithi118 ceph-mon[31535]: pgmap v3716: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-24T02:01:11.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:01:10 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 5766.697439 secs 2024-03-24T02:01:11.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:01:10 smithi189 ceph-mon[36187]: pgmap v3716: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-24T02:01:11.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:01:10 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 5766.697439 secs 2024-03-24T02:01:13.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:01:12 smithi118 ceph-mon[31535]: pgmap v3717: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:01:13.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:01:12 smithi189 ceph-mon[36187]: pgmap v3717: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:01:13.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:01:12 smithi193 ceph-mon[36878]: pgmap v3717: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:01:15.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:01:14 smithi118 ceph-mon[31535]: pgmap v3718: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T02:01:15.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:01:14 smithi189 ceph-mon[36187]: pgmap v3718: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T02:01:15.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:01:14 smithi193 ceph-mon[36878]: pgmap v3718: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T02:01:16.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:01:15 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 5771.697478 secs 2024-03-24T02:01:16.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:01:15 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 5771.697478 secs 2024-03-24T02:01:16.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:01:15 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 5771.697478 secs 2024-03-24T02:01:16.661 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T02:01:16.659+0000 7f81db400640 1 -- 172.21.15.118:0/659686424 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f81c4006460 con 0x5622651a13f0 2024-03-24T02:01:16.661 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T02:01:16.660+0000 7f81ee000640 1 -- 172.21.15.118:0/1693767697 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f81c80036c0 con 0x7f81f81052d0 2024-03-24T02:01:16.661 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T02:01:16.660+0000 7f81e4c00640 1 -- 172.21.15.118:0/659686424 <== mon.1 v2:172.21.15.193:3300/0 48 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 409+0+0 (secure 0 0 0) 0x7f81c4006460 con 0x5622651a13f0 2024-03-24T02:01:16.661 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T02:01:16.660+0000 7f81ef400640 1 -- 172.21.15.118:0/1693767697 <== mon.2 v2:172.21.15.189:3300/0 48 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (secure 0 0 0) 0x7f81c80036c0 con 0x7f81f81052d0 2024-03-24T02:01:17.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:01:16 smithi118 ceph-mon[31535]: pgmap v3719: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T02:01:17.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:01:16 smithi189 ceph-mon[36187]: pgmap v3719: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T02:01:17.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:01:16 smithi193 ceph-mon[36878]: pgmap v3719: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T02:01:19.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:01:18 smithi118 ceph-mon[31535]: pgmap v3720: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:01:19.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:01:18 smithi189 ceph-mon[36187]: pgmap v3720: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:01:19.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:01:18 smithi193 ceph-mon[36878]: pgmap v3720: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:01:21.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:01:20 smithi118 ceph-mon[31535]: pgmap v3721: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-24T02:01:21.197 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:01:20 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 5776.697568 secs 2024-03-24T02:01:21.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:01:20 smithi189 ceph-mon[36187]: pgmap v3721: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-24T02:01:21.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:01:20 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 5776.697568 secs 2024-03-24T02:01:21.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:01:20 smithi193 ceph-mon[36878]: pgmap v3721: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-24T02:01:21.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:01:20 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 5776.697568 secs 2024-03-24T02:01:23.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:01:22 smithi118 ceph-mon[31535]: pgmap v3722: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:01:23.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:01:22 smithi189 ceph-mon[36187]: pgmap v3722: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:01:23.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:01:22 smithi193 ceph-mon[36878]: pgmap v3722: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:01:25.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:01:24 smithi118 ceph-mon[31535]: pgmap v3723: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T02:01:25.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:01:24 smithi189 ceph-mon[36187]: pgmap v3723: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T02:01:25.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:01:24 smithi193 ceph-mon[36878]: pgmap v3723: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T02:01:26.197 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:01:25 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 5781.697653 secs 2024-03-24T02:01:26.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:01:25 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 5781.697653 secs 2024-03-24T02:01:26.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:01:25 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 5781.697653 secs 2024-03-24T02:01:27.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:01:26 smithi118 ceph-mon[31535]: pgmap v3724: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T02:01:27.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:01:26 smithi189 ceph-mon[36187]: pgmap v3724: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T02:01:27.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:01:26 smithi193 ceph-mon[36878]: pgmap v3724: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T02:01:29.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:01:28 smithi118 ceph-mon[31535]: pgmap v3725: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:01:29.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:01:28 smithi189 ceph-mon[36187]: pgmap v3725: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:01:29.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:01:28 smithi193 ceph-mon[36878]: pgmap v3725: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:01:31.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:01:30 smithi118 ceph-mon[31535]: pgmap v3726: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-24T02:01:31.197 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:01:30 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 5786.697743 secs 2024-03-24T02:01:31.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:01:30 smithi189 ceph-mon[36187]: pgmap v3726: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-24T02:01:31.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:01:30 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 5786.697743 secs 2024-03-24T02:01:31.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:01:30 smithi193 ceph-mon[36878]: pgmap v3726: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-24T02:01:31.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:01:30 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 5786.697743 secs 2024-03-24T02:01:33.197 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:01:32 smithi118 ceph-mon[31535]: pgmap v3727: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:01:33.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:01:32 smithi189 ceph-mon[36187]: pgmap v3727: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:01:33.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:01:32 smithi193 ceph-mon[36878]: pgmap v3727: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:01:35.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:01:34 smithi118 ceph-mon[31535]: pgmap v3728: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T02:01:35.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:01:34 smithi189 ceph-mon[36187]: pgmap v3728: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T02:01:35.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:01:34 smithi193 ceph-mon[36878]: pgmap v3728: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T02:01:36.098 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:01:35 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 5791.697804 secs 2024-03-24T02:01:36.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:01:35 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 5791.697804 secs 2024-03-24T02:01:36.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:01:35 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 5791.697804 secs 2024-03-24T02:01:37.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:01:36 smithi118 ceph-mon[31535]: pgmap v3729: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T02:01:37.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:01:36 smithi189 ceph-mon[36187]: pgmap v3729: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T02:01:37.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:01:36 smithi193 ceph-mon[36878]: pgmap v3729: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T02:01:38.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:01:37 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T02:01:38.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:01:37 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T02:01:38.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:01:37 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T02:01:39.104 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:01:38 smithi118 ceph-mon[31535]: pgmap v3730: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:01:39.105 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:01:38 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T02:01:39.105 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:01:38 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T02:01:39.105 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:01:38 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T02:01:39.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:01:38 smithi189 ceph-mon[36187]: pgmap v3730: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:01:39.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:01:38 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T02:01:39.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:01:38 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T02:01:39.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:01:38 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T02:01:39.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:01:38 smithi193 ceph-mon[36878]: pgmap v3730: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:01:39.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:01:38 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T02:01:39.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:01:38 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T02:01:39.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:01:38 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T02:01:40.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:01:39 smithi118 ceph-mon[31535]: pgmap v3731: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 6 op/s 2024-03-24T02:01:40.197 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:01:39 smithi118 ceph-mon[31535]: pgmap v3732: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 4 op/s 2024-03-24T02:01:40.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:01:39 smithi189 ceph-mon[36187]: pgmap v3731: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 6 op/s 2024-03-24T02:01:40.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:01:39 smithi189 ceph-mon[36187]: pgmap v3732: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 4 op/s 2024-03-24T02:01:40.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:01:39 smithi193 ceph-mon[36878]: pgmap v3731: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 6 op/s 2024-03-24T02:01:40.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:01:39 smithi193 ceph-mon[36878]: pgmap v3732: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 4 op/s 2024-03-24T02:01:41.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:01:41 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 5796.697847 secs 2024-03-24T02:01:41.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:01:41 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 5796.697847 secs 2024-03-24T02:01:41.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:01:41 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 5796.697847 secs 2024-03-24T02:01:42.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:01:42 smithi118 ceph-mon[31535]: pgmap v3733: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 6 op/s 2024-03-24T02:01:42.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:01:42 smithi189 ceph-mon[36187]: pgmap v3733: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 6 op/s 2024-03-24T02:01:42.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:01:42 smithi193 ceph-mon[36878]: pgmap v3733: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 6 op/s 2024-03-24T02:01:44.447 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:01:44 smithi118 ceph-mon[31535]: pgmap v3734: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 5 op/s 2024-03-24T02:01:44.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:01:44 smithi189 ceph-mon[36187]: pgmap v3734: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 5 op/s 2024-03-24T02:01:44.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:01:44 smithi193 ceph-mon[36878]: pgmap v3734: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 5 op/s 2024-03-24T02:01:46.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:01:46 smithi118 ceph-mon[31535]: pgmap v3735: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 6 op/s 2024-03-24T02:01:46.447 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:01:46 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 5801.697934 secs 2024-03-24T02:01:46.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:01:46 smithi189 ceph-mon[36187]: pgmap v3735: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 6 op/s 2024-03-24T02:01:46.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:01:46 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 5801.697934 secs 2024-03-24T02:01:46.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:01:46 smithi193 ceph-mon[36878]: pgmap v3735: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 6 op/s 2024-03-24T02:01:46.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:01:46 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 5801.697934 secs 2024-03-24T02:01:48.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:01:48 smithi118 ceph-mon[31535]: pgmap v3736: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 6 op/s 2024-03-24T02:01:48.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:01:48 smithi189 ceph-mon[36187]: pgmap v3736: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 6 op/s 2024-03-24T02:01:48.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:01:48 smithi193 ceph-mon[36878]: pgmap v3736: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 6 op/s 2024-03-24T02:01:50.447 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:01:50 smithi118 ceph-mon[31535]: pgmap v3737: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 5 op/s 2024-03-24T02:01:50.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:01:50 smithi189 ceph-mon[36187]: pgmap v3737: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 5 op/s 2024-03-24T02:01:50.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:01:50 smithi193 ceph-mon[36878]: pgmap v3737: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 5 op/s 2024-03-24T02:01:51.197 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:01:51 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 5806.698021 secs 2024-03-24T02:01:51.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:01:51 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 5806.698021 secs 2024-03-24T02:01:51.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:01:51 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 5806.698021 secs 2024-03-24T02:01:52.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:01:52 smithi118 ceph-mon[31535]: pgmap v3738: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T02:01:52.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:01:52 smithi189 ceph-mon[36187]: pgmap v3738: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T02:01:52.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:01:52 smithi193 ceph-mon[36878]: pgmap v3738: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T02:01:53.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:01:53 smithi118 ceph-mon[31535]: pgmap v3739: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 5 op/s 2024-03-24T02:01:53.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:01:53 smithi189 ceph-mon[36187]: pgmap v3739: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 5 op/s 2024-03-24T02:01:53.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:01:53 smithi193 ceph-mon[36878]: pgmap v3739: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 5 op/s 2024-03-24T02:01:56.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:01:55 smithi118 ceph-mon[31535]: pgmap v3740: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T02:01:56.197 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:01:55 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 5811.698108 secs 2024-03-24T02:01:56.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:01:55 smithi189 ceph-mon[36187]: pgmap v3740: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T02:01:56.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:01:55 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 5811.698108 secs 2024-03-24T02:01:56.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:01:55 smithi193 ceph-mon[36878]: pgmap v3740: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T02:01:56.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:01:55 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 5811.698108 secs 2024-03-24T02:01:58.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:01:57 smithi118 ceph-mon[31535]: pgmap v3741: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:01:58.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:01:57 smithi189 ceph-mon[36187]: pgmap v3741: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:01:58.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:01:57 smithi193 ceph-mon[36878]: pgmap v3741: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:02:00.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:01:59 smithi118 ceph-mon[31535]: pgmap v3742: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T02:02:00.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:01:59 smithi189 ceph-mon[36187]: pgmap v3742: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T02:02:00.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:01:59 smithi193 ceph-mon[36878]: pgmap v3742: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T02:02:01.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:02:00 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 5816.698167 secs 2024-03-24T02:02:01.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:02:00 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 5816.698167 secs 2024-03-24T02:02:01.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:02:00 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 5816.698167 secs 2024-03-24T02:02:02.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:02:01 smithi118 ceph-mon[31535]: pgmap v3743: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T02:02:02.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:02:01 smithi189 ceph-mon[36187]: pgmap v3743: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T02:02:02.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:02:01 smithi193 ceph-mon[36878]: pgmap v3743: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T02:02:04.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:02:03 smithi118 ceph-mon[31535]: pgmap v3744: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 5 op/s 2024-03-24T02:02:04.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:02:03 smithi189 ceph-mon[36187]: pgmap v3744: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 5 op/s 2024-03-24T02:02:04.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:02:03 smithi193 ceph-mon[36878]: pgmap v3744: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 5 op/s 2024-03-24T02:02:06.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:02:05 smithi118 ceph-mon[31535]: pgmap v3745: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T02:02:06.197 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:02:05 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 5821.698256 secs 2024-03-24T02:02:06.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:02:05 smithi189 ceph-mon[36187]: pgmap v3745: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T02:02:06.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:02:05 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 5821.698256 secs 2024-03-24T02:02:06.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:02:05 smithi193 ceph-mon[36878]: pgmap v3745: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T02:02:06.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:02:05 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 5821.698256 secs 2024-03-24T02:02:08.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:02:07 smithi118 ceph-mon[31535]: pgmap v3746: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:02:08.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:02:07 smithi189 ceph-mon[36187]: pgmap v3746: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:02:08.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:02:07 smithi193 ceph-mon[36878]: pgmap v3746: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:02:10.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:02:09 smithi118 ceph-mon[31535]: pgmap v3747: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T02:02:10.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:02:09 smithi189 ceph-mon[36187]: pgmap v3747: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T02:02:10.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:02:09 smithi193 ceph-mon[36878]: pgmap v3747: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T02:02:11.105 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:02:10 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 5826.698362 secs 2024-03-24T02:02:11.112 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:02:10 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 5826.698362 secs 2024-03-24T02:02:11.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:02:10 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 5826.698362 secs 2024-03-24T02:02:12.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:02:11 smithi118 ceph-mon[31535]: pgmap v3748: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T02:02:12.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:02:11 smithi189 ceph-mon[36187]: pgmap v3748: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T02:02:12.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:02:11 smithi193 ceph-mon[36878]: pgmap v3748: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T02:02:14.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:02:13 smithi118 ceph-mon[31535]: pgmap v3749: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 5 op/s 2024-03-24T02:02:14.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:02:13 smithi189 ceph-mon[36187]: pgmap v3749: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 5 op/s 2024-03-24T02:02:14.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:02:13 smithi193 ceph-mon[36878]: pgmap v3749: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 5 op/s 2024-03-24T02:02:16.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:02:15 smithi118 ceph-mon[31535]: pgmap v3750: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T02:02:16.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:02:15 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 5831.698451 secs 2024-03-24T02:02:16.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:02:15 smithi189 ceph-mon[36187]: pgmap v3750: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T02:02:16.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:02:15 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 5831.698451 secs 2024-03-24T02:02:16.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:02:15 smithi193 ceph-mon[36878]: pgmap v3750: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T02:02:16.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:02:15 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 5831.698451 secs 2024-03-24T02:02:18.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:02:17 smithi118 ceph-mon[31535]: pgmap v3751: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:02:18.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:02:17 smithi189 ceph-mon[36187]: pgmap v3751: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:02:18.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:02:17 smithi193 ceph-mon[36878]: pgmap v3751: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:02:20.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:02:19 smithi118 ceph-mon[31535]: pgmap v3752: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T02:02:20.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:02:19 smithi189 ceph-mon[36187]: pgmap v3752: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T02:02:20.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:02:19 smithi193 ceph-mon[36878]: pgmap v3752: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T02:02:21.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:02:20 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 5836.698511 secs 2024-03-24T02:02:21.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:02:20 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 5836.698511 secs 2024-03-24T02:02:21.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:02:20 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 5836.698511 secs 2024-03-24T02:02:22.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:02:21 smithi118 ceph-mon[31535]: pgmap v3753: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T02:02:22.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:02:21 smithi189 ceph-mon[36187]: pgmap v3753: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T02:02:22.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:02:21 smithi193 ceph-mon[36878]: pgmap v3753: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T02:02:24.197 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:02:23 smithi118 ceph-mon[31535]: pgmap v3754: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 5 op/s 2024-03-24T02:02:24.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:02:23 smithi189 ceph-mon[36187]: pgmap v3754: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 5 op/s 2024-03-24T02:02:24.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:02:23 smithi193 ceph-mon[36878]: pgmap v3754: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 5 op/s 2024-03-24T02:02:26.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:02:25 smithi118 ceph-mon[31535]: pgmap v3755: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T02:02:26.197 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:02:25 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 5841.698571 secs 2024-03-24T02:02:26.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:02:25 smithi189 ceph-mon[36187]: pgmap v3755: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T02:02:26.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:02:25 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 5841.698571 secs 2024-03-24T02:02:26.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:02:25 smithi193 ceph-mon[36878]: pgmap v3755: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T02:02:26.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:02:25 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 5841.698571 secs 2024-03-24T02:02:28.197 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:02:27 smithi118 ceph-mon[31535]: pgmap v3756: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:02:28.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:02:27 smithi189 ceph-mon[36187]: pgmap v3756: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:02:28.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:02:27 smithi193 ceph-mon[36878]: pgmap v3756: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:02:30.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:02:29 smithi118 ceph-mon[31535]: pgmap v3757: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T02:02:30.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:02:29 smithi189 ceph-mon[36187]: pgmap v3757: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T02:02:30.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:02:29 smithi193 ceph-mon[36878]: pgmap v3757: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T02:02:31.197 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:02:30 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 5846.698655 secs 2024-03-24T02:02:31.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:02:30 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 5846.698655 secs 2024-03-24T02:02:31.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:02:30 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 5846.698655 secs 2024-03-24T02:02:32.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:02:31 smithi118 ceph-mon[31535]: pgmap v3758: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T02:02:32.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:02:31 smithi189 ceph-mon[36187]: pgmap v3758: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T02:02:32.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:02:31 smithi193 ceph-mon[36878]: pgmap v3758: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T02:02:34.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:02:33 smithi118 ceph-mon[31535]: pgmap v3759: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 5 op/s 2024-03-24T02:02:34.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:02:33 smithi189 ceph-mon[36187]: pgmap v3759: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 5 op/s 2024-03-24T02:02:34.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:02:33 smithi193 ceph-mon[36878]: pgmap v3759: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 5 op/s 2024-03-24T02:02:36.197 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:02:35 smithi118 ceph-mon[31535]: pgmap v3760: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T02:02:36.197 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:02:35 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 5851.698744 secs 2024-03-24T02:02:36.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:02:35 smithi189 ceph-mon[36187]: pgmap v3760: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T02:02:36.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:02:35 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 5851.698744 secs 2024-03-24T02:02:36.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:02:35 smithi193 ceph-mon[36878]: pgmap v3760: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T02:02:36.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:02:35 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 5851.698744 secs 2024-03-24T02:02:38.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:02:37 smithi118 ceph-mon[31535]: pgmap v3761: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:02:38.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:02:37 smithi189 ceph-mon[36187]: pgmap v3761: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:02:38.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:02:37 smithi193 ceph-mon[36878]: pgmap v3761: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:02:40.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:02:39 smithi118 ceph-mon[31535]: pgmap v3762: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T02:02:40.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:02:39 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T02:02:40.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:02:39 smithi189 ceph-mon[36187]: pgmap v3762: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T02:02:40.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:02:39 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T02:02:40.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:02:39 smithi193 ceph-mon[36878]: pgmap v3762: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T02:02:40.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:02:39 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T02:02:41.197 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:02:40 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T02:02:41.197 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:02:40 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T02:02:41.197 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:02:40 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T02:02:41.197 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:02:40 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 5856.698788 secs 2024-03-24T02:02:41.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:02:40 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T02:02:41.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:02:40 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T02:02:41.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:02:40 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T02:02:41.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:02:40 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 5856.698788 secs 2024-03-24T02:02:41.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:02:40 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T02:02:41.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:02:40 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T02:02:41.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:02:40 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T02:02:41.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:02:40 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 5856.698788 secs 2024-03-24T02:02:42.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:02:41 smithi118 ceph-mon[31535]: pgmap v3763: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 6 op/s 2024-03-24T02:02:42.197 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:02:41 smithi118 ceph-mon[31535]: pgmap v3764: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-03-24T02:02:42.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:02:41 smithi189 ceph-mon[36187]: pgmap v3763: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 6 op/s 2024-03-24T02:02:42.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:02:41 smithi189 ceph-mon[36187]: pgmap v3764: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-03-24T02:02:42.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:02:41 smithi193 ceph-mon[36878]: pgmap v3763: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 6 op/s 2024-03-24T02:02:42.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:02:41 smithi193 ceph-mon[36878]: pgmap v3764: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-03-24T02:02:44.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:02:43 smithi189 ceph-mon[36187]: pgmap v3765: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 5 op/s 2024-03-24T02:02:44.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:02:43 smithi193 ceph-mon[36878]: pgmap v3765: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 5 op/s 2024-03-24T02:02:44.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:02:43 smithi118 ceph-mon[31535]: pgmap v3765: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 5 op/s 2024-03-24T02:02:46.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:02:45 smithi189 ceph-mon[36187]: pgmap v3766: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-03-24T02:02:46.205 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:02:45 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 5861.698847 secs 2024-03-24T02:02:46.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:02:45 smithi193 ceph-mon[36878]: pgmap v3766: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-03-24T02:02:46.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:02:45 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 5861.698847 secs 2024-03-24T02:02:46.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:02:45 smithi118 ceph-mon[31535]: pgmap v3766: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-03-24T02:02:46.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:02:45 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 5861.698847 secs 2024-03-24T02:02:48.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:02:47 smithi193 ceph-mon[36878]: pgmap v3767: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-03-24T02:02:48.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:02:47 smithi118 ceph-mon[31535]: pgmap v3767: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-03-24T02:02:48.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:02:47 smithi189 ceph-mon[36187]: pgmap v3767: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-03-24T02:02:50.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:02:49 smithi193 ceph-mon[36878]: pgmap v3768: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-03-24T02:02:50.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:02:49 smithi118 ceph-mon[31535]: pgmap v3768: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-03-24T02:02:50.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:02:49 smithi189 ceph-mon[36187]: pgmap v3768: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-03-24T02:02:51.447 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:02:50 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 5866.698936 secs 2024-03-24T02:02:51.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:02:50 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 5866.698936 secs 2024-03-24T02:02:51.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:02:50 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 5866.698936 secs 2024-03-24T02:02:52.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:02:51 smithi118 ceph-mon[31535]: pgmap v3769: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 5 op/s 2024-03-24T02:02:52.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:02:51 smithi189 ceph-mon[36187]: pgmap v3769: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 5 op/s 2024-03-24T02:02:52.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:02:51 smithi193 ceph-mon[36878]: pgmap v3769: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 5 op/s 2024-03-24T02:02:54.447 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:02:53 smithi118 ceph-mon[31535]: pgmap v3770: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:02:54.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:02:53 smithi189 ceph-mon[36187]: pgmap v3770: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:02:54.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:02:53 smithi193 ceph-mon[36878]: pgmap v3770: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:02:56.447 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:02:55 smithi118 ceph-mon[31535]: pgmap v3771: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-24T02:02:56.447 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:02:55 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 5871.699025 secs 2024-03-24T02:02:56.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:02:55 smithi189 ceph-mon[36187]: pgmap v3771: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-24T02:02:56.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:02:55 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 5871.699025 secs 2024-03-24T02:02:56.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:02:55 smithi193 ceph-mon[36878]: pgmap v3771: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-24T02:02:56.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:02:55 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 5871.699025 secs 2024-03-24T02:02:58.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:02:57 smithi118 ceph-mon[31535]: pgmap v3772: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:02:58.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:02:57 smithi189 ceph-mon[36187]: pgmap v3772: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:02:58.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:02:57 smithi193 ceph-mon[36878]: pgmap v3772: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:03:00.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:02:59 smithi118 ceph-mon[31535]: pgmap v3773: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T02:03:00.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:02:59 smithi189 ceph-mon[36187]: pgmap v3773: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T02:03:00.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:02:59 smithi193 ceph-mon[36878]: pgmap v3773: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T02:03:01.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:03:00 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 5876.699116 secs 2024-03-24T02:03:01.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:03:00 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 5876.699116 secs 2024-03-24T02:03:01.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:03:00 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 5876.699116 secs 2024-03-24T02:03:02.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:03:01 smithi118 ceph-mon[31535]: pgmap v3774: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T02:03:02.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:03:01 smithi189 ceph-mon[36187]: pgmap v3774: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T02:03:02.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:03:01 smithi193 ceph-mon[36878]: pgmap v3774: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T02:03:04.447 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:03:04 smithi118 ceph-mon[31535]: pgmap v3775: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:03:04.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:03:04 smithi189 ceph-mon[36187]: pgmap v3775: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:03:04.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:03:04 smithi193 ceph-mon[36878]: pgmap v3775: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:03:06.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:03:06 smithi118 ceph-mon[31535]: pgmap v3776: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-24T02:03:06.447 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:03:06 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 5881.699178 secs 2024-03-24T02:03:06.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:03:06 smithi189 ceph-mon[36187]: pgmap v3776: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-24T02:03:06.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:03:06 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 5881.699178 secs 2024-03-24T02:03:06.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:03:06 smithi193 ceph-mon[36878]: pgmap v3776: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-24T02:03:06.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:03:06 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 5881.699178 secs 2024-03-24T02:03:08.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:03:08 smithi118 ceph-mon[31535]: pgmap v3777: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:03:08.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:03:08 smithi189 ceph-mon[36187]: pgmap v3777: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:03:08.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:03:08 smithi193 ceph-mon[36878]: pgmap v3777: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:03:10.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:03:10 smithi118 ceph-mon[31535]: pgmap v3778: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T02:03:10.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:03:10 smithi189 ceph-mon[36187]: pgmap v3778: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T02:03:10.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:03:10 smithi193 ceph-mon[36878]: pgmap v3778: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T02:03:11.355 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:03:11 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 5886.699265 secs 2024-03-24T02:03:11.362 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:03:11 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 5886.699265 secs 2024-03-24T02:03:11.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:03:11 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 5886.699265 secs 2024-03-24T02:03:12.280 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:03:12 smithi118 ceph-mon[31535]: pgmap v3779: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T02:03:12.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:03:12 smithi189 ceph-mon[36187]: pgmap v3779: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T02:03:12.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:03:12 smithi193 ceph-mon[36878]: pgmap v3779: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T02:03:14.447 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:03:14 smithi118 ceph-mon[31535]: pgmap v3780: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:03:14.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:03:14 smithi189 ceph-mon[36187]: pgmap v3780: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:03:14.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:03:14 smithi193 ceph-mon[36878]: pgmap v3780: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:03:16.447 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:03:16 smithi118 ceph-mon[31535]: pgmap v3781: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-24T02:03:16.447 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:03:16 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 5891.699369 secs 2024-03-24T02:03:16.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:03:16 smithi189 ceph-mon[36187]: pgmap v3781: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-24T02:03:16.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:03:16 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 5891.699369 secs 2024-03-24T02:03:16.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:03:16 smithi193 ceph-mon[36878]: pgmap v3781: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-24T02:03:16.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:03:16 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 5891.699369 secs 2024-03-24T02:03:18.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:03:18 smithi118 ceph-mon[31535]: pgmap v3782: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:03:18.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:03:18 smithi189 ceph-mon[36187]: pgmap v3782: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:03:18.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:03:18 smithi193 ceph-mon[36878]: pgmap v3782: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:03:20.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:03:20 smithi118 ceph-mon[31535]: pgmap v3783: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T02:03:20.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:03:20 smithi189 ceph-mon[36187]: pgmap v3783: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T02:03:20.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:03:20 smithi193 ceph-mon[36878]: pgmap v3783: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T02:03:21.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:03:21 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 5896.699427 secs 2024-03-24T02:03:21.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:03:21 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 5896.699427 secs 2024-03-24T02:03:21.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:03:21 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 5896.699427 secs 2024-03-24T02:03:22.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:03:22 smithi118 ceph-mon[31535]: pgmap v3784: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T02:03:22.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:03:22 smithi189 ceph-mon[36187]: pgmap v3784: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T02:03:22.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:03:22 smithi193 ceph-mon[36878]: pgmap v3784: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T02:03:24.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:03:24 smithi118 ceph-mon[31535]: pgmap v3785: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:03:24.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:03:24 smithi189 ceph-mon[36187]: pgmap v3785: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:03:24.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:03:24 smithi193 ceph-mon[36878]: pgmap v3785: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:03:26.447 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:03:26 smithi118 ceph-mon[31535]: pgmap v3786: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-24T02:03:26.447 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:03:26 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 5901.699488 secs 2024-03-24T02:03:26.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:03:26 smithi189 ceph-mon[36187]: pgmap v3786: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-24T02:03:26.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:03:26 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 5901.699488 secs 2024-03-24T02:03:26.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:03:26 smithi193 ceph-mon[36878]: pgmap v3786: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-24T02:03:26.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:03:26 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 5901.699488 secs 2024-03-24T02:03:28.447 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:03:28 smithi118 ceph-mon[31535]: pgmap v3787: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:03:28.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:03:28 smithi189 ceph-mon[36187]: pgmap v3787: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:03:28.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:03:28 smithi193 ceph-mon[36878]: pgmap v3787: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:03:30.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:03:30 smithi118 ceph-mon[31535]: pgmap v3788: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T02:03:30.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:03:30 smithi189 ceph-mon[36187]: pgmap v3788: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T02:03:30.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:03:30 smithi193 ceph-mon[36878]: pgmap v3788: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T02:03:31.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:03:31 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 5906.699547 secs 2024-03-24T02:03:31.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:03:31 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 5906.699547 secs 2024-03-24T02:03:31.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:03:31 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 5906.699547 secs 2024-03-24T02:03:32.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:03:32 smithi118 ceph-mon[31535]: pgmap v3789: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T02:03:32.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:03:32 smithi189 ceph-mon[36187]: pgmap v3789: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T02:03:32.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:03:32 smithi193 ceph-mon[36878]: pgmap v3789: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T02:03:34.447 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:03:34 smithi118 ceph-mon[31535]: pgmap v3790: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:03:34.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:03:34 smithi189 ceph-mon[36187]: pgmap v3790: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:03:34.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:03:34 smithi193 ceph-mon[36878]: pgmap v3790: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:03:36.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:03:36 smithi118 ceph-mon[31535]: pgmap v3791: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-24T02:03:36.447 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:03:36 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 5911.699603 secs 2024-03-24T02:03:36.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:03:36 smithi189 ceph-mon[36187]: pgmap v3791: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-24T02:03:36.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:03:36 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 5911.699603 secs 2024-03-24T02:03:36.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:03:36 smithi193 ceph-mon[36878]: pgmap v3791: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-24T02:03:36.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:03:36 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 5911.699603 secs 2024-03-24T02:03:38.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:03:38 smithi118 ceph-mon[31535]: pgmap v3792: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:03:38.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:03:38 smithi189 ceph-mon[36187]: pgmap v3792: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:03:38.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:03:38 smithi193 ceph-mon[36878]: pgmap v3792: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:03:40.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:03:40 smithi118 ceph-mon[31535]: pgmap v3793: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T02:03:40.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:03:40 smithi189 ceph-mon[36187]: pgmap v3793: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T02:03:40.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:03:40 smithi193 ceph-mon[36878]: pgmap v3793: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T02:03:41.363 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:03:41 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 5916.699690 secs 2024-03-24T02:03:41.363 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:03:41 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T02:03:41.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:03:41 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 5916.699690 secs 2024-03-24T02:03:41.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:03:41 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T02:03:41.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:03:41 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 5916.699690 secs 2024-03-24T02:03:41.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:03:41 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T02:03:42.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:03:42 smithi118 ceph-mon[31535]: pgmap v3794: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T02:03:42.447 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:03:42 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T02:03:42.447 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:03:42 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T02:03:42.447 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:03:42 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T02:03:42.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:03:42 smithi189 ceph-mon[36187]: pgmap v3794: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T02:03:42.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:03:42 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T02:03:42.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:03:42 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T02:03:42.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:03:42 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T02:03:42.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:03:42 smithi193 ceph-mon[36878]: pgmap v3794: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T02:03:42.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:03:42 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T02:03:42.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:03:42 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T02:03:42.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:03:42 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T02:03:43.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:03:43 smithi118 ceph-mon[31535]: pgmap v3795: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 5 op/s 2024-03-24T02:03:43.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:03:43 smithi118 ceph-mon[31535]: pgmap v3796: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-03-24T02:03:43.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:03:43 smithi189 ceph-mon[36187]: pgmap v3795: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 5 op/s 2024-03-24T02:03:43.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:03:43 smithi189 ceph-mon[36187]: pgmap v3796: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-03-24T02:03:43.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:03:43 smithi193 ceph-mon[36878]: pgmap v3795: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 5 op/s 2024-03-24T02:03:43.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:03:43 smithi193 ceph-mon[36878]: pgmap v3796: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-03-24T02:03:44.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:03:44 smithi118 ceph-mon[31535]: pgmap v3797: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 4 op/s 2024-03-24T02:03:44.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:03:44 smithi189 ceph-mon[36187]: pgmap v3797: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 4 op/s 2024-03-24T02:03:44.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:03:44 smithi193 ceph-mon[36878]: pgmap v3797: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 4 op/s 2024-03-24T02:03:47.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:03:46 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 5921.699748 secs 2024-03-24T02:03:47.197 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:03:46 smithi118 ceph-mon[31535]: pgmap v3798: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-03-24T02:03:47.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:03:46 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 5921.699748 secs 2024-03-24T02:03:47.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:03:46 smithi189 ceph-mon[36187]: pgmap v3798: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-03-24T02:03:47.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:03:46 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 5921.699748 secs 2024-03-24T02:03:47.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:03:46 smithi193 ceph-mon[36878]: pgmap v3798: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-03-24T02:03:49.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:03:48 smithi118 ceph-mon[31535]: pgmap v3799: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 5 op/s 2024-03-24T02:03:49.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:03:48 smithi189 ceph-mon[36187]: pgmap v3799: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 5 op/s 2024-03-24T02:03:49.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:03:48 smithi193 ceph-mon[36878]: pgmap v3799: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 5 op/s 2024-03-24T02:03:51.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:03:50 smithi118 ceph-mon[31535]: pgmap v3800: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-03-24T02:03:51.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:03:50 smithi189 ceph-mon[36187]: pgmap v3800: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-03-24T02:03:51.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:03:50 smithi193 ceph-mon[36878]: pgmap v3800: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-03-24T02:03:52.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:03:51 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 5926.699810 secs 2024-03-24T02:03:52.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:03:51 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 5926.699810 secs 2024-03-24T02:03:52.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:03:51 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 5926.699810 secs 2024-03-24T02:03:53.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:03:52 smithi118 ceph-mon[31535]: pgmap v3801: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 5 op/s 2024-03-24T02:03:53.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:03:52 smithi189 ceph-mon[36187]: pgmap v3801: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 5 op/s 2024-03-24T02:03:53.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:03:52 smithi193 ceph-mon[36878]: pgmap v3801: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 5 op/s 2024-03-24T02:03:55.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:03:54 smithi118 ceph-mon[31535]: pgmap v3802: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T02:03:55.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:03:54 smithi189 ceph-mon[36187]: pgmap v3802: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T02:03:55.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:03:54 smithi193 ceph-mon[36878]: pgmap v3802: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T02:03:56.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:03:55 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 5931.699870 secs 2024-03-24T02:03:56.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:03:55 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 5931.699870 secs 2024-03-24T02:03:56.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:03:55 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 5931.699870 secs 2024-03-24T02:03:57.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:03:56 smithi118 ceph-mon[31535]: pgmap v3803: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T02:03:57.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:03:56 smithi189 ceph-mon[36187]: pgmap v3803: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T02:03:57.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:03:56 smithi193 ceph-mon[36878]: pgmap v3803: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T02:03:59.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:03:58 smithi118 ceph-mon[31535]: pgmap v3804: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T02:03:59.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:03:58 smithi189 ceph-mon[36187]: pgmap v3804: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T02:03:59.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:03:58 smithi193 ceph-mon[36878]: pgmap v3804: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T02:04:01.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:04:00 smithi118 ceph-mon[31535]: pgmap v3805: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-03-24T02:04:01.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:04:00 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 5936.699961 secs 2024-03-24T02:04:01.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:04:00 smithi189 ceph-mon[36187]: pgmap v3805: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-03-24T02:04:01.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:04:00 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 5936.699961 secs 2024-03-24T02:04:01.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:04:00 smithi193 ceph-mon[36878]: pgmap v3805: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-03-24T02:04:01.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:04:00 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 5936.699961 secs 2024-03-24T02:04:03.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:04:02 smithi118 ceph-mon[31535]: pgmap v3806: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:04:03.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:04:02 smithi189 ceph-mon[36187]: pgmap v3806: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:04:03.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:04:02 smithi193 ceph-mon[36878]: pgmap v3806: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:04:05.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:04:04 smithi118 ceph-mon[31535]: pgmap v3807: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T02:04:05.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:04:04 smithi189 ceph-mon[36187]: pgmap v3807: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T02:04:05.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:04:04 smithi193 ceph-mon[36878]: pgmap v3807: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T02:04:06.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:04:06 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 5941.700045 secs 2024-03-24T02:04:06.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:04:06 smithi118 ceph-mon[31535]: pgmap v3808: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T02:04:06.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:04:06 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 5941.700045 secs 2024-03-24T02:04:06.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:04:06 smithi189 ceph-mon[36187]: pgmap v3808: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T02:04:06.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:04:06 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 5941.700045 secs 2024-03-24T02:04:06.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:04:06 smithi193 ceph-mon[36878]: pgmap v3808: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T02:04:09.197 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:04:08 smithi118 ceph-mon[31535]: pgmap v3809: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T02:04:09.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:04:08 smithi189 ceph-mon[36187]: pgmap v3809: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T02:04:09.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:04:08 smithi193 ceph-mon[36878]: pgmap v3809: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T02:04:10.999 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:04:10 smithi118 ceph-mon[31535]: pgmap v3810: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-03-24T02:04:11.006 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:04:10 smithi189 ceph-mon[36187]: pgmap v3810: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-03-24T02:04:11.012 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:04:10 smithi193 ceph-mon[36878]: pgmap v3810: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-03-24T02:04:12.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:04:11 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 5946.700067 secs 2024-03-24T02:04:12.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:04:11 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 5946.700067 secs 2024-03-24T02:04:12.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:04:11 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 5946.700067 secs 2024-03-24T02:04:13.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:04:12 smithi118 ceph-mon[31535]: pgmap v3811: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:04:13.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:04:12 smithi189 ceph-mon[36187]: pgmap v3811: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:04:13.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:04:12 smithi193 ceph-mon[36878]: pgmap v3811: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:04:15.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:04:14 smithi118 ceph-mon[31535]: pgmap v3812: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T02:04:15.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:04:14 smithi189 ceph-mon[36187]: pgmap v3812: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T02:04:15.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:04:14 smithi193 ceph-mon[36878]: pgmap v3812: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T02:04:16.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:04:15 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 5951.700154 secs 2024-03-24T02:04:16.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:04:15 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 5951.700154 secs 2024-03-24T02:04:16.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:04:15 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 5951.700154 secs 2024-03-24T02:04:16.690 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T02:04:16.689+0000 7f81ee000640 1 -- 172.21.15.118:0/1693767697 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- auth(proto 2 132 bytes epoch 0) v1 -- 0x7f81c80036c0 con 0x7f81f81052d0 2024-03-24T02:04:16.690 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T02:04:16.689+0000 7f81db400640 1 -- 172.21.15.118:0/659686424 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- auth(proto 2 132 bytes epoch 0) v1 -- 0x7f81c4006460 con 0x5622651a13f0 2024-03-24T02:04:16.691 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T02:04:16.689+0000 7f81ef400640 1 -- 172.21.15.118:0/1693767697 <== mon.2 v2:172.21.15.189:3300/0 49 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 783+0+0 (secure 0 0 0) 0x7f81c80036c0 con 0x7f81f81052d0 2024-03-24T02:04:16.691 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T02:04:16.689+0000 7f81e4c00640 1 -- 172.21.15.118:0/659686424 <== mon.1 v2:172.21.15.193:3300/0 49 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 612+0+0 (secure 0 0 0) 0x7f81c4006460 con 0x5622651a13f0 2024-03-24T02:04:16.691 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T02:04:16.689+0000 7f81e4c00640 1 -- 172.21.15.118:0/659686424 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f81c4006290 con 0x5622651a13f0 2024-03-24T02:04:16.691 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T02:04:16.689+0000 7f81ef400640 1 -- 172.21.15.118:0/1693767697 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f81c80065b0 con 0x7f81f81052d0 2024-03-24T02:04:16.691 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T02:04:16.690+0000 7f81e4c00640 1 -- 172.21.15.118:0/659686424 <== mon.1 v2:172.21.15.193:3300/0 50 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 409+0+0 (secure 0 0 0) 0x7f81c4006290 con 0x5622651a13f0 2024-03-24T02:04:16.691 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T02:04:16.690+0000 7f81ef400640 1 -- 172.21.15.118:0/1693767697 <== mon.2 v2:172.21.15.189:3300/0 50 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (secure 0 0 0) 0x7f81c80065b0 con 0x7f81f81052d0 2024-03-24T02:04:17.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:04:16 smithi118 ceph-mon[31535]: pgmap v3813: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T02:04:17.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:04:16 smithi189 ceph-mon[36187]: pgmap v3813: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T02:04:17.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:04:16 smithi193 ceph-mon[36878]: pgmap v3813: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T02:04:19.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:04:18 smithi118 ceph-mon[31535]: pgmap v3814: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T02:04:19.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:04:18 smithi189 ceph-mon[36187]: pgmap v3814: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T02:04:19.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:04:18 smithi193 ceph-mon[36878]: pgmap v3814: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T02:04:21.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:04:20 smithi118 ceph-mon[31535]: pgmap v3815: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-03-24T02:04:21.197 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:04:20 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 5956.700213 secs 2024-03-24T02:04:21.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:04:20 smithi189 ceph-mon[36187]: pgmap v3815: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-03-24T02:04:21.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:04:20 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 5956.700213 secs 2024-03-24T02:04:21.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:04:20 smithi193 ceph-mon[36878]: pgmap v3815: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-03-24T02:04:21.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:04:20 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 5956.700213 secs 2024-03-24T02:04:23.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:04:22 smithi118 ceph-mon[31535]: pgmap v3816: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:04:23.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:04:22 smithi189 ceph-mon[36187]: pgmap v3816: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:04:23.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:04:22 smithi193 ceph-mon[36878]: pgmap v3816: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:04:25.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:04:24 smithi118 ceph-mon[31535]: pgmap v3817: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T02:04:25.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:04:24 smithi189 ceph-mon[36187]: pgmap v3817: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T02:04:25.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:04:24 smithi193 ceph-mon[36878]: pgmap v3817: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T02:04:26.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:04:25 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 5961.700295 secs 2024-03-24T02:04:26.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:04:25 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 5961.700295 secs 2024-03-24T02:04:26.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:04:25 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 5961.700295 secs 2024-03-24T02:04:27.098 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:04:26 smithi118 ceph-mon[31535]: pgmap v3818: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T02:04:27.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:04:26 smithi189 ceph-mon[36187]: pgmap v3818: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T02:04:27.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:04:26 smithi193 ceph-mon[36878]: pgmap v3818: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T02:04:29.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:04:28 smithi118 ceph-mon[31535]: pgmap v3819: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T02:04:29.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:04:28 smithi189 ceph-mon[36187]: pgmap v3819: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T02:04:29.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:04:28 smithi193 ceph-mon[36878]: pgmap v3819: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T02:04:31.197 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:04:30 smithi118 ceph-mon[31535]: pgmap v3820: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-03-24T02:04:31.197 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:04:30 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 5966.700400 secs 2024-03-24T02:04:31.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:04:30 smithi189 ceph-mon[36187]: pgmap v3820: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-03-24T02:04:31.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:04:30 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 5966.700400 secs 2024-03-24T02:04:31.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:04:30 smithi193 ceph-mon[36878]: pgmap v3820: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-03-24T02:04:31.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:04:30 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 5966.700400 secs 2024-03-24T02:04:33.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:04:32 smithi118 ceph-mon[31535]: pgmap v3821: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:04:33.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:04:32 smithi189 ceph-mon[36187]: pgmap v3821: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:04:33.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:04:32 smithi193 ceph-mon[36878]: pgmap v3821: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:04:35.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:04:34 smithi118 ceph-mon[31535]: pgmap v3822: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T02:04:35.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:04:34 smithi189 ceph-mon[36187]: pgmap v3822: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T02:04:35.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:04:34 smithi193 ceph-mon[36878]: pgmap v3822: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T02:04:36.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:04:35 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 5971.700485 secs 2024-03-24T02:04:36.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:04:35 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 5971.700485 secs 2024-03-24T02:04:36.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:04:35 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 5971.700485 secs 2024-03-24T02:04:37.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:04:36 smithi118 ceph-mon[31535]: pgmap v3823: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T02:04:37.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:04:36 smithi189 ceph-mon[36187]: pgmap v3823: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T02:04:37.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:04:36 smithi193 ceph-mon[36878]: pgmap v3823: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T02:04:39.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:04:38 smithi118 ceph-mon[31535]: pgmap v3824: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T02:04:39.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:04:38 smithi189 ceph-mon[36187]: pgmap v3824: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T02:04:39.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:04:38 smithi193 ceph-mon[36878]: pgmap v3824: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T02:04:41.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:04:40 smithi118 ceph-mon[31535]: pgmap v3825: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-03-24T02:04:41.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:04:40 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 5976.700547 secs 2024-03-24T02:04:41.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:04:40 smithi189 ceph-mon[36187]: pgmap v3825: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-03-24T02:04:41.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:04:40 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 5976.700547 secs 2024-03-24T02:04:41.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:04:40 smithi193 ceph-mon[36878]: pgmap v3825: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-03-24T02:04:41.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:04:40 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 5976.700547 secs 2024-03-24T02:04:43.098 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:04:42 smithi118 ceph-mon[31535]: pgmap v3826: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:04:43.098 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:04:42 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T02:04:43.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:04:42 smithi189 ceph-mon[36187]: pgmap v3826: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:04:43.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:04:42 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T02:04:43.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:04:42 smithi193 ceph-mon[36878]: pgmap v3826: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:04:43.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:04:42 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T02:04:44.197 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:04:43 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T02:04:44.260 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:04:43 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T02:04:44.260 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:04:43 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T02:04:44.261 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:04:43 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T02:04:44.261 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:04:43 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T02:04:44.261 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:04:43 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T02:04:44.261 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:04:43 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T02:04:44.261 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:04:43 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T02:04:44.261 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:04:43 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T02:04:45.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:04:44 smithi118 ceph-mon[31535]: pgmap v3827: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 6 op/s 2024-03-24T02:04:45.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:04:44 smithi118 ceph-mon[31535]: pgmap v3828: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-03-24T02:04:45.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:04:44 smithi189 ceph-mon[36187]: pgmap v3827: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 6 op/s 2024-03-24T02:04:45.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:04:44 smithi189 ceph-mon[36187]: pgmap v3828: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-03-24T02:04:45.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:04:44 smithi193 ceph-mon[36878]: pgmap v3827: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 6 op/s 2024-03-24T02:04:45.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:04:44 smithi193 ceph-mon[36878]: pgmap v3828: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-03-24T02:04:46.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:04:45 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 5981.700607 secs 2024-03-24T02:04:46.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:04:45 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 5981.700607 secs 2024-03-24T02:04:46.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:04:45 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 5981.700607 secs 2024-03-24T02:04:47.197 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:04:46 smithi118 ceph-mon[31535]: pgmap v3829: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-03-24T02:04:47.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:04:46 smithi189 ceph-mon[36187]: pgmap v3829: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-03-24T02:04:47.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:04:46 smithi193 ceph-mon[36878]: pgmap v3829: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-03-24T02:04:49.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:04:48 smithi118 ceph-mon[31535]: pgmap v3830: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 5 op/s 2024-03-24T02:04:49.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:04:48 smithi189 ceph-mon[36187]: pgmap v3830: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 5 op/s 2024-03-24T02:04:49.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:04:48 smithi193 ceph-mon[36878]: pgmap v3830: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 5 op/s 2024-03-24T02:04:51.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:04:50 smithi118 ceph-mon[31535]: pgmap v3831: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-03-24T02:04:51.197 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:04:50 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 5986.700665 secs 2024-03-24T02:04:51.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:04:50 smithi189 ceph-mon[36187]: pgmap v3831: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-03-24T02:04:51.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:04:50 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 5986.700665 secs 2024-03-24T02:04:51.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:04:50 smithi193 ceph-mon[36878]: pgmap v3831: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-03-24T02:04:51.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:04:50 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 5986.700665 secs 2024-03-24T02:04:53.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:04:52 smithi118 ceph-mon[31535]: pgmap v3832: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-03-24T02:04:53.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:04:52 smithi189 ceph-mon[36187]: pgmap v3832: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-03-24T02:04:53.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:04:52 smithi193 ceph-mon[36878]: pgmap v3832: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-03-24T02:04:55.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:04:54 smithi118 ceph-mon[31535]: pgmap v3833: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 5 op/s 2024-03-24T02:04:55.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:04:54 smithi189 ceph-mon[36187]: pgmap v3833: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 5 op/s 2024-03-24T02:04:55.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:04:54 smithi193 ceph-mon[36878]: pgmap v3833: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 5 op/s 2024-03-24T02:04:56.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:04:55 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 5991.700753 secs 2024-03-24T02:04:56.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:04:55 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 5991.700753 secs 2024-03-24T02:04:56.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:04:55 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 5991.700753 secs 2024-03-24T02:04:57.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:04:56 smithi118 ceph-mon[31535]: pgmap v3834: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T02:04:57.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:04:56 smithi189 ceph-mon[36187]: pgmap v3834: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T02:04:57.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:04:56 smithi193 ceph-mon[36878]: pgmap v3834: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T02:04:59.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:04:58 smithi118 ceph-mon[31535]: pgmap v3835: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:04:59.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:04:58 smithi189 ceph-mon[36187]: pgmap v3835: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:04:59.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:04:58 smithi193 ceph-mon[36878]: pgmap v3835: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:05:01.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:05:00 smithi118 ceph-mon[31535]: pgmap v3836: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-24T02:05:01.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:05:00 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 5996.700808 secs 2024-03-24T02:05:01.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:05:00 smithi189 ceph-mon[36187]: pgmap v3836: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-24T02:05:01.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:05:00 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 5996.700808 secs 2024-03-24T02:05:01.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:05:00 smithi193 ceph-mon[36878]: pgmap v3836: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-24T02:05:01.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:05:00 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 5996.700808 secs 2024-03-24T02:05:03.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:05:02 smithi118 ceph-mon[31535]: pgmap v3837: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:05:03.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:05:02 smithi189 ceph-mon[36187]: pgmap v3837: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:05:03.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:05:02 smithi193 ceph-mon[36878]: pgmap v3837: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:05:05.197 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:05:04 smithi118 ceph-mon[31535]: pgmap v3838: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-03-24T02:05:05.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:05:04 smithi189 ceph-mon[36187]: pgmap v3838: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-03-24T02:05:05.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:05:04 smithi193 ceph-mon[36878]: pgmap v3838: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-03-24T02:05:06.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:05:06 smithi118 ceph-mon[31535]: pgmap v3839: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 6 op/s 2024-03-24T02:05:06.447 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:05:06 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 6001.700897 secs 2024-03-24T02:05:06.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:05:06 smithi189 ceph-mon[36187]: pgmap v3839: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 6 op/s 2024-03-24T02:05:06.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:05:06 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 6001.700897 secs 2024-03-24T02:05:06.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:05:06 smithi193 ceph-mon[36878]: pgmap v3839: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 6 op/s 2024-03-24T02:05:06.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:05:06 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 6001.700897 secs 2024-03-24T02:05:08.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:05:08 smithi118 ceph-mon[31535]: pgmap v3840: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-03-24T02:05:08.704 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:05:08 smithi189 ceph-mon[36187]: pgmap v3840: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-03-24T02:05:08.711 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:05:08 smithi193 ceph-mon[36878]: pgmap v3840: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-03-24T02:05:10.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:05:10 smithi118 ceph-mon[31535]: pgmap v3841: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-24T02:05:10.704 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:05:10 smithi189 ceph-mon[36187]: pgmap v3841: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-24T02:05:10.711 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:05:10 smithi193 ceph-mon[36878]: pgmap v3841: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-24T02:05:11.605 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:05:11 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 6006.700988 secs 2024-03-24T02:05:11.611 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:05:11 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 6006.700988 secs 2024-03-24T02:05:11.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:05:11 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 6006.700988 secs 2024-03-24T02:05:12.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:05:12 smithi118 ceph-mon[31535]: pgmap v3842: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-03-24T02:05:12.704 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:05:12 smithi189 ceph-mon[36187]: pgmap v3842: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-03-24T02:05:12.711 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:05:12 smithi193 ceph-mon[36878]: pgmap v3842: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-03-24T02:05:14.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:05:14 smithi118 ceph-mon[31535]: pgmap v3843: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-03-24T02:05:14.704 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:05:14 smithi189 ceph-mon[36187]: pgmap v3843: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-03-24T02:05:14.711 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:05:14 smithi193 ceph-mon[36878]: pgmap v3843: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-03-24T02:05:16.697 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:05:16 smithi118 ceph-mon[31535]: pgmap v3844: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T02:05:16.697 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:05:16 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 6011.701078 secs 2024-03-24T02:05:16.704 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:05:16 smithi189 ceph-mon[36187]: pgmap v3844: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T02:05:16.704 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:05:16 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 6011.701078 secs 2024-03-24T02:05:16.711 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:05:16 smithi193 ceph-mon[36878]: pgmap v3844: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T02:05:16.711 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:05:16 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 6011.701078 secs 2024-03-24T02:05:18.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:05:18 smithi118 ceph-mon[31535]: pgmap v3845: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:05:18.704 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:05:18 smithi189 ceph-mon[36187]: pgmap v3845: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:05:18.711 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:05:18 smithi193 ceph-mon[36878]: pgmap v3845: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:05:20.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:05:20 smithi118 ceph-mon[31535]: pgmap v3846: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-24T02:05:20.705 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:05:20 smithi189 ceph-mon[36187]: pgmap v3846: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-24T02:05:20.711 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:05:20 smithi193 ceph-mon[36878]: pgmap v3846: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-24T02:05:21.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:05:21 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 6016.701163 secs 2024-03-24T02:05:21.704 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:05:21 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 6016.701163 secs 2024-03-24T02:05:21.711 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:05:21 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 6016.701163 secs 2024-03-24T02:05:22.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:05:22 smithi118 ceph-mon[31535]: pgmap v3847: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:05:22.704 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:05:22 smithi189 ceph-mon[36187]: pgmap v3847: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:05:22.711 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:05:22 smithi193 ceph-mon[36878]: pgmap v3847: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:05:24.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:05:24 smithi118 ceph-mon[31535]: pgmap v3848: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T02:05:24.704 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:05:24 smithi189 ceph-mon[36187]: pgmap v3848: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T02:05:24.711 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:05:24 smithi193 ceph-mon[36878]: pgmap v3848: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T02:05:26.697 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:05:26 smithi118 ceph-mon[31535]: pgmap v3849: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T02:05:26.697 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:05:26 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 6021.701218 secs 2024-03-24T02:05:26.704 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:05:26 smithi189 ceph-mon[36187]: pgmap v3849: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T02:05:26.704 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:05:26 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 6021.701218 secs 2024-03-24T02:05:26.711 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:05:26 smithi193 ceph-mon[36878]: pgmap v3849: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T02:05:26.711 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:05:26 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 6021.701218 secs 2024-03-24T02:05:28.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:05:28 smithi118 ceph-mon[31535]: pgmap v3850: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:05:28.704 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:05:28 smithi189 ceph-mon[36187]: pgmap v3850: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:05:28.711 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:05:28 smithi193 ceph-mon[36878]: pgmap v3850: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:05:30.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:05:30 smithi118 ceph-mon[31535]: pgmap v3851: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-24T02:05:30.704 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:05:30 smithi189 ceph-mon[36187]: pgmap v3851: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-24T02:05:30.711 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:05:30 smithi193 ceph-mon[36878]: pgmap v3851: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-24T02:05:31.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:05:31 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 6026.701280 secs 2024-03-24T02:05:31.704 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:05:31 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 6026.701280 secs 2024-03-24T02:05:31.711 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:05:31 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 6026.701280 secs 2024-03-24T02:05:32.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:05:32 smithi118 ceph-mon[31535]: pgmap v3852: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:05:32.704 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:05:32 smithi189 ceph-mon[36187]: pgmap v3852: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:05:32.711 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:05:32 smithi193 ceph-mon[36878]: pgmap v3852: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:05:34.697 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:05:34 smithi118 ceph-mon[31535]: pgmap v3853: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T02:05:34.704 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:05:34 smithi189 ceph-mon[36187]: pgmap v3853: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T02:05:34.711 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:05:34 smithi193 ceph-mon[36878]: pgmap v3853: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T02:05:36.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:05:36 smithi118 ceph-mon[31535]: pgmap v3854: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T02:05:36.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:05:36 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 6031.701366 secs 2024-03-24T02:05:36.704 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:05:36 smithi189 ceph-mon[36187]: pgmap v3854: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T02:05:36.704 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:05:36 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 6031.701366 secs 2024-03-24T02:05:36.711 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:05:36 smithi193 ceph-mon[36878]: pgmap v3854: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T02:05:36.711 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:05:36 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 6031.701366 secs 2024-03-24T02:05:38.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:05:38 smithi118 ceph-mon[31535]: pgmap v3855: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:05:38.704 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:05:38 smithi189 ceph-mon[36187]: pgmap v3855: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:05:38.711 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:05:38 smithi193 ceph-mon[36878]: pgmap v3855: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:05:40.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:05:40 smithi118 ceph-mon[31535]: pgmap v3856: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-24T02:05:40.704 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:05:40 smithi189 ceph-mon[36187]: pgmap v3856: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-24T02:05:40.711 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:05:40 smithi193 ceph-mon[36878]: pgmap v3856: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-24T02:05:41.612 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:05:41 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 6036.701425 secs 2024-03-24T02:05:41.697 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:05:41 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 6036.701425 secs 2024-03-24T02:05:41.704 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:05:41 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 6036.701425 secs 2024-03-24T02:05:42.697 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:05:42 smithi118 ceph-mon[31535]: pgmap v3857: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:05:42.704 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:05:42 smithi189 ceph-mon[36187]: pgmap v3857: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:05:42.711 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:05:42 smithi193 ceph-mon[36878]: pgmap v3857: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:05:44.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:05:44 smithi118 ceph-mon[31535]: pgmap v3858: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T02:05:44.697 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:05:44 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T02:05:44.697 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:05:44 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-24T02:05:44.697 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:05:44 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-24T02:05:44.697 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:05:44 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-24T02:05:44.697 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:05:44 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-03-24T02:05:44.704 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:05:44 smithi189 ceph-mon[36187]: pgmap v3858: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T02:05:44.704 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:05:44 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T02:05:44.704 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:05:44 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-24T02:05:44.704 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:05:44 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-24T02:05:44.704 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:05:44 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-24T02:05:44.705 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:05:44 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-03-24T02:05:44.711 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:05:44 smithi193 ceph-mon[36878]: pgmap v3858: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T02:05:44.711 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:05:44 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T02:05:44.711 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:05:44 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-24T02:05:44.711 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:05:44 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-24T02:05:44.711 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:05:44 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-24T02:05:44.711 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:05:44 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-03-24T02:05:45.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:05:45 smithi118 ceph-mon[31535]: Adjusting osd_memory_target on smithi189 to 174.1M 2024-03-24T02:05:45.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:05:45 smithi118 ceph-mon[31535]: Unable to set osd_memory_target on smithi189 to 182635622: error parsing value: Value '182635622' is below minimum 939524096 2024-03-24T02:05:45.704 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:05:45 smithi189 ceph-mon[36187]: Adjusting osd_memory_target on smithi189 to 174.1M 2024-03-24T02:05:45.704 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:05:45 smithi189 ceph-mon[36187]: Unable to set osd_memory_target on smithi189 to 182635622: error parsing value: Value '182635622' is below minimum 939524096 2024-03-24T02:05:45.711 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:05:45 smithi193 ceph-mon[36878]: Adjusting osd_memory_target on smithi189 to 174.1M 2024-03-24T02:05:45.711 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:05:45 smithi193 ceph-mon[36878]: Unable to set osd_memory_target on smithi189 to 182635622: error parsing value: Value '182635622' is below minimum 939524096 2024-03-24T02:05:46.947 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:05:46 smithi118 ceph-mon[31535]: pgmap v3859: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T02:05:46.947 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:05:46 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 6041.701513 secs 2024-03-24T02:05:46.947 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:05:46 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T02:05:46.947 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:05:46 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T02:05:46.947 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:05:46 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-24T02:05:46.947 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:05:46 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-24T02:05:46.947 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:05:46 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-24T02:05:46.947 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:05:46 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-03-24T02:05:46.948 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:05:46 smithi118 ceph-mon[31535]: Adjusting osd_memory_target on smithi118 to 176.7M 2024-03-24T02:05:46.948 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:05:46 smithi118 ceph-mon[31535]: Unable to set osd_memory_target on smithi118 to 185365196: error parsing value: Value '185365196' is below minimum 939524096 2024-03-24T02:05:46.948 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:05:46 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T02:05:46.948 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:05:46 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T02:05:46.948 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:05:46 smithi118 ceph-mon[31535]: pgmap v3860: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T02:05:46.948 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:05:46 smithi118 ceph-mon[31535]: pgmap v3861: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-03-24T02:05:46.948 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:05:46 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T02:05:46.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:05:46 smithi189 ceph-mon[36187]: pgmap v3859: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T02:05:46.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:05:46 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 6041.701513 secs 2024-03-24T02:05:46.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:05:46 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T02:05:46.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:05:46 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T02:05:46.955 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:05:46 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-24T02:05:46.955 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:05:46 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-24T02:05:46.955 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:05:46 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-24T02:05:46.955 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:05:46 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-03-24T02:05:46.955 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:05:46 smithi189 ceph-mon[36187]: Adjusting osd_memory_target on smithi118 to 176.7M 2024-03-24T02:05:46.955 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:05:46 smithi189 ceph-mon[36187]: Unable to set osd_memory_target on smithi118 to 185365196: error parsing value: Value '185365196' is below minimum 939524096 2024-03-24T02:05:46.955 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:05:46 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T02:05:46.955 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:05:46 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T02:05:46.955 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:05:46 smithi189 ceph-mon[36187]: pgmap v3860: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T02:05:46.955 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:05:46 smithi189 ceph-mon[36187]: pgmap v3861: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-03-24T02:05:46.955 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:05:46 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T02:05:46.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:05:46 smithi193 ceph-mon[36878]: pgmap v3859: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T02:05:46.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:05:46 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 6041.701513 secs 2024-03-24T02:05:46.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:05:46 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T02:05:46.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:05:46 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T02:05:46.962 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:05:46 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-24T02:05:46.962 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:05:46 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-24T02:05:46.962 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:05:46 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-24T02:05:46.962 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:05:46 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-03-24T02:05:46.962 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:05:46 smithi193 ceph-mon[36878]: Adjusting osd_memory_target on smithi118 to 176.7M 2024-03-24T02:05:46.962 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:05:46 smithi193 ceph-mon[36878]: Unable to set osd_memory_target on smithi118 to 185365196: error parsing value: Value '185365196' is below minimum 939524096 2024-03-24T02:05:46.962 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:05:46 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T02:05:46.962 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:05:46 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T02:05:46.962 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:05:46 smithi193 ceph-mon[36878]: pgmap v3860: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T02:05:46.962 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:05:46 smithi193 ceph-mon[36878]: pgmap v3861: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-03-24T02:05:46.962 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:05:46 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T02:05:48.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:05:48 smithi118 ceph-mon[31535]: pgmap v3862: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-03-24T02:05:48.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:05:48 smithi189 ceph-mon[36187]: pgmap v3862: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-03-24T02:05:48.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:05:48 smithi193 ceph-mon[36878]: pgmap v3862: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-03-24T02:05:50.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:05:50 smithi118 ceph-mon[31535]: pgmap v3863: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-03-24T02:05:50.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:05:50 smithi189 ceph-mon[36187]: pgmap v3863: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-03-24T02:05:50.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:05:50 smithi193 ceph-mon[36878]: pgmap v3863: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-03-24T02:05:51.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:05:51 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 6046.701597 secs 2024-03-24T02:05:51.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:05:51 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 6046.701597 secs 2024-03-24T02:05:51.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:05:51 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 6046.701597 secs 2024-03-24T02:05:52.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:05:52 smithi118 ceph-mon[31535]: pgmap v3864: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 5 op/s 2024-03-24T02:05:52.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:05:52 smithi189 ceph-mon[36187]: pgmap v3864: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 5 op/s 2024-03-24T02:05:52.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:05:52 smithi193 ceph-mon[36878]: pgmap v3864: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 5 op/s 2024-03-24T02:05:54.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:05:54 smithi118 ceph-mon[31535]: pgmap v3865: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 5 op/s 2024-03-24T02:05:54.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:05:54 smithi189 ceph-mon[36187]: pgmap v3865: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 5 op/s 2024-03-24T02:05:54.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:05:54 smithi193 ceph-mon[36878]: pgmap v3865: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 5 op/s 2024-03-24T02:05:56.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:05:56 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 6051.701684 secs 2024-03-24T02:05:56.947 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:05:56 smithi118 ceph-mon[31535]: pgmap v3866: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 5 op/s 2024-03-24T02:05:56.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:05:56 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 6051.701684 secs 2024-03-24T02:05:56.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:05:56 smithi189 ceph-mon[36187]: pgmap v3866: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 5 op/s 2024-03-24T02:05:56.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:05:56 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 6051.701684 secs 2024-03-24T02:05:56.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:05:56 smithi193 ceph-mon[36878]: pgmap v3866: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 5 op/s 2024-03-24T02:05:58.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:05:58 smithi118 ceph-mon[31535]: pgmap v3867: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T02:05:58.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:05:58 smithi189 ceph-mon[36187]: pgmap v3867: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T02:05:58.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:05:58 smithi193 ceph-mon[36878]: pgmap v3867: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T02:06:00.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:06:00 smithi118 ceph-mon[31535]: pgmap v3868: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-03-24T02:06:00.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:06:00 smithi189 ceph-mon[36187]: pgmap v3868: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-03-24T02:06:00.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:06:00 smithi193 ceph-mon[36878]: pgmap v3868: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-03-24T02:06:01.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:06:01 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 6056.701770 secs 2024-03-24T02:06:01.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:06:01 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 6056.701770 secs 2024-03-24T02:06:01.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:06:01 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 6056.701770 secs 2024-03-24T02:06:02.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:06:02 smithi118 ceph-mon[31535]: pgmap v3869: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:06:02.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:06:02 smithi189 ceph-mon[36187]: pgmap v3869: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:06:02.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:06:02 smithi193 ceph-mon[36878]: pgmap v3869: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:06:04.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:06:04 smithi118 ceph-mon[31535]: pgmap v3870: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T02:06:04.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:06:04 smithi189 ceph-mon[36187]: pgmap v3870: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T02:06:04.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:06:04 smithi193 ceph-mon[36878]: pgmap v3870: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T02:06:06.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:06:06 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 6061.701854 secs 2024-03-24T02:06:06.947 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:06:06 smithi118 ceph-mon[31535]: pgmap v3871: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T02:06:06.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:06:06 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 6061.701854 secs 2024-03-24T02:06:06.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:06:06 smithi189 ceph-mon[36187]: pgmap v3871: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T02:06:06.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:06:06 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 6061.701854 secs 2024-03-24T02:06:06.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:06:06 smithi193 ceph-mon[36878]: pgmap v3871: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T02:06:08.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:06:08 smithi118 ceph-mon[31535]: pgmap v3872: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T02:06:08.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:06:08 smithi189 ceph-mon[36187]: pgmap v3872: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T02:06:08.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:06:08 smithi193 ceph-mon[36878]: pgmap v3872: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T02:06:10.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:06:10 smithi118 ceph-mon[31535]: pgmap v3873: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-03-24T02:06:10.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:06:10 smithi189 ceph-mon[36187]: pgmap v3873: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-03-24T02:06:10.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:06:10 smithi193 ceph-mon[36878]: pgmap v3873: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-03-24T02:06:11.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:06:11 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 6066.701905 secs 2024-03-24T02:06:11.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:06:11 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 6066.701905 secs 2024-03-24T02:06:11.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:06:11 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 6066.701905 secs 2024-03-24T02:06:12.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:06:12 smithi118 ceph-mon[31535]: pgmap v3874: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:06:12.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:06:12 smithi189 ceph-mon[36187]: pgmap v3874: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:06:12.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:06:12 smithi193 ceph-mon[36878]: pgmap v3874: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:06:14.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:06:14 smithi118 ceph-mon[31535]: pgmap v3875: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T02:06:14.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:06:14 smithi189 ceph-mon[36187]: pgmap v3875: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T02:06:14.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:06:14 smithi193 ceph-mon[36878]: pgmap v3875: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T02:06:16.947 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:06:16 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 6071.701920 secs 2024-03-24T02:06:16.947 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:06:16 smithi118 ceph-mon[31535]: pgmap v3876: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T02:06:16.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:06:16 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 6071.701920 secs 2024-03-24T02:06:16.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:06:16 smithi189 ceph-mon[36187]: pgmap v3876: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T02:06:16.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:06:16 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 6071.701920 secs 2024-03-24T02:06:16.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:06:16 smithi193 ceph-mon[36878]: pgmap v3876: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T02:06:18.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:06:18 smithi118 ceph-mon[31535]: pgmap v3877: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T02:06:18.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:06:18 smithi189 ceph-mon[36187]: pgmap v3877: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T02:06:18.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:06:18 smithi193 ceph-mon[36878]: pgmap v3877: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T02:06:20.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:06:20 smithi118 ceph-mon[31535]: pgmap v3878: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-03-24T02:06:20.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:06:20 smithi189 ceph-mon[36187]: pgmap v3878: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-03-24T02:06:20.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:06:20 smithi193 ceph-mon[36878]: pgmap v3878: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-03-24T02:06:21.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:06:21 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 6076.702002 secs 2024-03-24T02:06:21.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:06:21 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 6076.702002 secs 2024-03-24T02:06:21.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:06:21 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 6076.702002 secs 2024-03-24T02:06:22.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:06:22 smithi118 ceph-mon[31535]: pgmap v3879: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:06:22.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:06:22 smithi189 ceph-mon[36187]: pgmap v3879: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:06:22.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:06:22 smithi193 ceph-mon[36878]: pgmap v3879: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:06:24.949 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:06:24 smithi118 ceph-mon[31535]: pgmap v3880: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T02:06:24.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:06:24 smithi189 ceph-mon[36187]: pgmap v3880: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T02:06:24.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:06:24 smithi193 ceph-mon[36878]: pgmap v3880: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T02:06:26.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:06:26 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 6081.702088 secs 2024-03-24T02:06:26.947 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:06:26 smithi118 ceph-mon[31535]: pgmap v3881: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T02:06:26.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:06:26 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 6081.702088 secs 2024-03-24T02:06:26.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:06:26 smithi189 ceph-mon[36187]: pgmap v3881: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T02:06:26.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:06:26 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 6081.702088 secs 2024-03-24T02:06:26.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:06:26 smithi193 ceph-mon[36878]: pgmap v3881: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T02:06:28.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:06:28 smithi118 ceph-mon[31535]: pgmap v3882: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T02:06:28.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:06:28 smithi189 ceph-mon[36187]: pgmap v3882: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T02:06:28.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:06:28 smithi193 ceph-mon[36878]: pgmap v3882: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T02:06:30.947 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:06:30 smithi118 ceph-mon[31535]: pgmap v3883: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-03-24T02:06:30.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:06:30 smithi189 ceph-mon[36187]: pgmap v3883: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-03-24T02:06:30.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:06:30 smithi193 ceph-mon[36878]: pgmap v3883: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-03-24T02:06:31.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:06:31 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 6086.702147 secs 2024-03-24T02:06:31.960 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:06:31 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 6086.702147 secs 2024-03-24T02:06:31.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:06:31 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 6086.702147 secs 2024-03-24T02:06:32.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:06:32 smithi193 ceph-mon[36878]: pgmap v3884: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:06:33.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:06:32 smithi118 ceph-mon[31535]: pgmap v3884: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:06:33.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:06:32 smithi189 ceph-mon[36187]: pgmap v3884: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:06:34.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:06:34 smithi193 ceph-mon[36878]: pgmap v3885: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T02:06:35.032 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:06:34 smithi118 ceph-mon[31535]: pgmap v3885: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T02:06:35.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:06:34 smithi189 ceph-mon[36187]: pgmap v3885: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T02:06:37.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:06:36 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 6091.702208 secs 2024-03-24T02:06:37.197 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:06:36 smithi118 ceph-mon[31535]: pgmap v3886: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T02:06:37.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:06:36 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 6091.702208 secs 2024-03-24T02:06:37.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:06:36 smithi189 ceph-mon[36187]: pgmap v3886: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T02:06:37.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:06:36 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 6091.702208 secs 2024-03-24T02:06:37.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:06:36 smithi193 ceph-mon[36878]: pgmap v3886: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T02:06:39.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:06:38 smithi118 ceph-mon[31535]: pgmap v3887: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T02:06:39.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:06:38 smithi189 ceph-mon[36187]: pgmap v3887: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T02:06:39.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:06:38 smithi193 ceph-mon[36878]: pgmap v3887: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T02:06:41.006 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:06:40 smithi189 ceph-mon[36187]: pgmap v3888: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-03-24T02:06:41.013 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:06:40 smithi193 ceph-mon[36878]: pgmap v3888: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-03-24T02:06:41.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:06:40 smithi118 ceph-mon[31535]: pgmap v3888: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-03-24T02:06:42.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:06:41 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 6096.702292 secs 2024-03-24T02:06:42.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:06:41 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 6096.702292 secs 2024-03-24T02:06:42.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:06:41 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 6096.702292 secs 2024-03-24T02:06:43.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:06:42 smithi118 ceph-mon[31535]: pgmap v3889: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:06:43.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:06:42 smithi189 ceph-mon[36187]: pgmap v3889: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:06:43.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:06:42 smithi193 ceph-mon[36878]: pgmap v3889: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:06:44.999 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:06:44 smithi118 ceph-mon[31535]: pgmap v3890: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T02:06:45.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:06:44 smithi189 ceph-mon[36187]: pgmap v3890: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T02:06:45.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:06:44 smithi193 ceph-mon[36878]: pgmap v3890: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T02:06:47.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:06:46 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 6101.702397 secs 2024-03-24T02:06:47.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:06:46 smithi118 ceph-mon[31535]: pgmap v3891: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T02:06:47.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:06:46 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T02:06:47.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:06:46 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 6101.702397 secs 2024-03-24T02:06:47.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:06:46 smithi189 ceph-mon[36187]: pgmap v3891: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T02:06:47.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:06:46 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T02:06:47.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:06:46 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 6101.702397 secs 2024-03-24T02:06:47.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:06:46 smithi193 ceph-mon[36878]: pgmap v3891: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T02:06:47.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:06:46 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T02:06:48.447 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:06:48 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T02:06:48.448 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:06:48 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T02:06:48.448 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:06:48 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T02:06:48.448 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:06:48 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-24T02:06:48.448 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:06:48 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T02:06:48.448 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:06:48 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-24T02:06:48.448 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:06:48 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-03-24T02:06:48.448 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:06:48 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-03-24T02:06:48.448 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:06:48 smithi118 ceph-mon[31535]: Adjusting osd_memory_target on smithi193 to 174.1M 2024-03-24T02:06:48.449 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:06:48 smithi118 ceph-mon[31535]: Unable to set osd_memory_target on smithi193 to 182639923: error parsing value: Value '182639923' is below minimum 939524096 2024-03-24T02:06:48.449 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:06:48 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T02:06:48.449 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:06:48 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T02:06:48.449 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:06:48 smithi118 ceph-mon[31535]: pgmap v3892: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 5 op/s 2024-03-24T02:06:48.449 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:06:48 smithi118 ceph-mon[31535]: pgmap v3893: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 5 op/s 2024-03-24T02:06:48.449 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:06:48 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T02:06:48.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:06:48 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T02:06:48.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:06:48 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T02:06:48.455 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:06:48 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T02:06:48.455 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:06:48 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-24T02:06:48.455 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:06:48 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T02:06:48.455 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:06:48 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-24T02:06:48.455 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:06:48 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-03-24T02:06:48.455 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:06:48 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-03-24T02:06:48.455 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:06:48 smithi189 ceph-mon[36187]: Adjusting osd_memory_target on smithi193 to 174.1M 2024-03-24T02:06:48.455 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:06:48 smithi189 ceph-mon[36187]: Unable to set osd_memory_target on smithi193 to 182639923: error parsing value: Value '182639923' is below minimum 939524096 2024-03-24T02:06:48.455 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:06:48 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T02:06:48.455 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:06:48 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T02:06:48.455 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:06:48 smithi189 ceph-mon[36187]: pgmap v3892: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 5 op/s 2024-03-24T02:06:48.455 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:06:48 smithi189 ceph-mon[36187]: pgmap v3893: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 5 op/s 2024-03-24T02:06:48.456 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:06:48 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T02:06:48.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:06:48 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T02:06:48.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:06:48 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T02:06:48.462 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:06:48 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T02:06:48.462 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:06:48 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-24T02:06:48.462 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:06:48 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T02:06:48.462 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:06:48 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-24T02:06:48.462 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:06:48 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-03-24T02:06:48.462 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:06:48 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-03-24T02:06:48.462 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:06:48 smithi193 ceph-mon[36878]: Adjusting osd_memory_target on smithi193 to 174.1M 2024-03-24T02:06:48.462 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:06:48 smithi193 ceph-mon[36878]: Unable to set osd_memory_target on smithi193 to 182639923: error parsing value: Value '182639923' is below minimum 939524096 2024-03-24T02:06:48.462 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:06:48 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T02:06:48.462 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:06:48 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T02:06:48.463 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:06:48 smithi193 ceph-mon[36878]: pgmap v3892: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 5 op/s 2024-03-24T02:06:48.463 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:06:48 smithi193 ceph-mon[36878]: pgmap v3893: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 5 op/s 2024-03-24T02:06:48.463 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:06:48 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T02:06:49.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:06:49 smithi118 ceph-mon[31535]: pgmap v3894: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 6 op/s 2024-03-24T02:06:49.704 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:06:49 smithi189 ceph-mon[36187]: pgmap v3894: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 6 op/s 2024-03-24T02:06:49.711 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:06:49 smithi193 ceph-mon[36878]: pgmap v3894: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 6 op/s 2024-03-24T02:06:51.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:06:50 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 6106.702487 secs 2024-03-24T02:06:51.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:06:50 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 6106.702487 secs 2024-03-24T02:06:51.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:06:50 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 6106.702487 secs 2024-03-24T02:06:52.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:06:51 smithi118 ceph-mon[31535]: pgmap v3895: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 6 op/s 2024-03-24T02:06:52.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:06:51 smithi189 ceph-mon[36187]: pgmap v3895: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 6 op/s 2024-03-24T02:06:52.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:06:51 smithi193 ceph-mon[36878]: pgmap v3895: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 6 op/s 2024-03-24T02:06:53.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:06:53 smithi118 ceph-mon[31535]: pgmap v3896: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 6 op/s 2024-03-24T02:06:53.704 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:06:53 smithi189 ceph-mon[36187]: pgmap v3896: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 6 op/s 2024-03-24T02:06:53.711 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:06:53 smithi193 ceph-mon[36878]: pgmap v3896: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 6 op/s 2024-03-24T02:06:55.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:06:55 smithi118 ceph-mon[31535]: pgmap v3897: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 6 op/s 2024-03-24T02:06:55.704 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:06:55 smithi189 ceph-mon[36187]: pgmap v3897: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 6 op/s 2024-03-24T02:06:55.711 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:06:55 smithi193 ceph-mon[36878]: pgmap v3897: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 6 op/s 2024-03-24T02:06:56.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:06:56 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 6111.702549 secs 2024-03-24T02:06:56.704 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:06:56 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 6111.702549 secs 2024-03-24T02:06:56.711 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:06:56 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 6111.702549 secs 2024-03-24T02:06:57.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:06:57 smithi118 ceph-mon[31535]: pgmap v3898: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 5 op/s 2024-03-24T02:06:57.704 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:06:57 smithi189 ceph-mon[36187]: pgmap v3898: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 5 op/s 2024-03-24T02:06:57.711 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:06:57 smithi193 ceph-mon[36878]: pgmap v3898: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 5 op/s 2024-03-24T02:06:59.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:06:59 smithi118 ceph-mon[31535]: pgmap v3899: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-24T02:06:59.704 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:06:59 smithi189 ceph-mon[36187]: pgmap v3899: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-24T02:06:59.711 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:06:59 smithi193 ceph-mon[36878]: pgmap v3899: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-24T02:07:01.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:07:00 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 6116.702608 secs 2024-03-24T02:07:01.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:07:00 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 6116.702608 secs 2024-03-24T02:07:01.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:07:00 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 6116.702608 secs 2024-03-24T02:07:02.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:07:01 smithi118 ceph-mon[31535]: pgmap v3900: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:07:02.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:07:01 smithi189 ceph-mon[36187]: pgmap v3900: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:07:02.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:07:01 smithi193 ceph-mon[36878]: pgmap v3900: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:07:03.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:07:03 smithi118 ceph-mon[31535]: pgmap v3901: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T02:07:03.704 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:07:03 smithi189 ceph-mon[36187]: pgmap v3901: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T02:07:03.711 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:07:03 smithi193 ceph-mon[36878]: pgmap v3901: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T02:07:05.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:07:05 smithi118 ceph-mon[31535]: pgmap v3902: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T02:07:05.704 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:07:05 smithi189 ceph-mon[36187]: pgmap v3902: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T02:07:05.711 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:07:05 smithi193 ceph-mon[36878]: pgmap v3902: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T02:07:06.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:07:06 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 6121.702693 secs 2024-03-24T02:07:06.704 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:07:06 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 6121.702693 secs 2024-03-24T02:07:06.711 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:07:06 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 6121.702693 secs 2024-03-24T02:07:08.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:07:08 smithi118 ceph-mon[31535]: pgmap v3903: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:07:08.704 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:07:08 smithi189 ceph-mon[36187]: pgmap v3903: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:07:08.711 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:07:08 smithi193 ceph-mon[36878]: pgmap v3903: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:07:10.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:07:10 smithi118 ceph-mon[31535]: pgmap v3904: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-24T02:07:10.704 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:07:10 smithi189 ceph-mon[36187]: pgmap v3904: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-24T02:07:10.711 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:07:10 smithi193 ceph-mon[36878]: pgmap v3904: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-24T02:07:11.605 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:07:11 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 6126.702783 secs 2024-03-24T02:07:11.612 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:07:11 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 6126.702783 secs 2024-03-24T02:07:11.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:07:11 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 6126.702783 secs 2024-03-24T02:07:12.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:07:12 smithi118 ceph-mon[31535]: pgmap v3905: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:07:12.704 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:07:12 smithi189 ceph-mon[36187]: pgmap v3905: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:07:12.711 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:07:12 smithi193 ceph-mon[36878]: pgmap v3905: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:07:14.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:07:14 smithi118 ceph-mon[31535]: pgmap v3906: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T02:07:14.704 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:07:14 smithi189 ceph-mon[36187]: pgmap v3906: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T02:07:14.711 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:07:14 smithi193 ceph-mon[36878]: pgmap v3906: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T02:07:16.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:07:16 smithi118 ceph-mon[31535]: pgmap v3907: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T02:07:16.697 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:07:16 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 6131.702842 secs 2024-03-24T02:07:16.704 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:07:16 smithi189 ceph-mon[36187]: pgmap v3907: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T02:07:16.704 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:07:16 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 6131.702842 secs 2024-03-24T02:07:16.711 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:07:16 smithi193 ceph-mon[36878]: pgmap v3907: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T02:07:16.711 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:07:16 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 6131.702842 secs 2024-03-24T02:07:16.718 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T02:07:16.718+0000 7f81ee000640 1 -- 172.21.15.118:0/1693767697 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f81c80065b0 con 0x7f81f81052d0 2024-03-24T02:07:16.718 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T02:07:16.718+0000 7f81db400640 1 -- 172.21.15.118:0/659686424 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f81c4006290 con 0x5622651a13f0 2024-03-24T02:07:16.719 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T02:07:16.718+0000 7f81e4c00640 1 -- 172.21.15.118:0/659686424 <== mon.1 v2:172.21.15.193:3300/0 51 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 409+0+0 (secure 0 0 0) 0x7f81c4006290 con 0x5622651a13f0 2024-03-24T02:07:16.719 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T02:07:16.718+0000 7f81ef400640 1 -- 172.21.15.118:0/1693767697 <== mon.2 v2:172.21.15.189:3300/0 51 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (secure 0 0 0) 0x7f81c80065b0 con 0x7f81f81052d0 2024-03-24T02:07:18.697 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:07:18 smithi118 ceph-mon[31535]: pgmap v3908: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:07:18.705 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:07:18 smithi189 ceph-mon[36187]: pgmap v3908: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:07:18.711 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:07:18 smithi193 ceph-mon[36878]: pgmap v3908: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:07:20.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:07:20 smithi118 ceph-mon[31535]: pgmap v3909: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-24T02:07:20.704 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:07:20 smithi189 ceph-mon[36187]: pgmap v3909: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-24T02:07:20.711 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:07:20 smithi193 ceph-mon[36878]: pgmap v3909: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-24T02:07:21.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:07:21 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 6136.702884 secs 2024-03-24T02:07:21.704 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:07:21 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 6136.702884 secs 2024-03-24T02:07:21.711 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:07:21 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 6136.702884 secs 2024-03-24T02:07:22.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:07:22 smithi118 ceph-mon[31535]: pgmap v3910: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:07:22.704 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:07:22 smithi189 ceph-mon[36187]: pgmap v3910: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:07:22.711 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:07:22 smithi193 ceph-mon[36878]: pgmap v3910: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:07:24.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:07:24 smithi118 ceph-mon[31535]: pgmap v3911: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-03-24T02:07:24.704 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:07:24 smithi189 ceph-mon[36187]: pgmap v3911: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-03-24T02:07:24.711 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:07:24 smithi193 ceph-mon[36878]: pgmap v3911: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-03-24T02:07:26.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:07:26 smithi118 ceph-mon[31535]: pgmap v3912: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 6 op/s 2024-03-24T02:07:26.697 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:07:26 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 6141.702940 secs 2024-03-24T02:07:26.704 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:07:26 smithi189 ceph-mon[36187]: pgmap v3912: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 6 op/s 2024-03-24T02:07:26.704 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:07:26 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 6141.702940 secs 2024-03-24T02:07:26.711 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:07:26 smithi193 ceph-mon[36878]: pgmap v3912: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 6 op/s 2024-03-24T02:07:26.711 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:07:26 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 6141.702940 secs 2024-03-24T02:07:28.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:07:28 smithi118 ceph-mon[31535]: pgmap v3913: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-03-24T02:07:28.704 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:07:28 smithi189 ceph-mon[36187]: pgmap v3913: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-03-24T02:07:28.711 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:07:28 smithi193 ceph-mon[36878]: pgmap v3913: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-03-24T02:07:30.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:07:30 smithi118 ceph-mon[31535]: pgmap v3914: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-24T02:07:30.704 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:07:30 smithi189 ceph-mon[36187]: pgmap v3914: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-24T02:07:30.711 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:07:30 smithi193 ceph-mon[36878]: pgmap v3914: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-24T02:07:31.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:07:31 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 6146.703022 secs 2024-03-24T02:07:31.704 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:07:31 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 6146.703022 secs 2024-03-24T02:07:31.711 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:07:31 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 6146.703022 secs 2024-03-24T02:07:32.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:07:32 smithi118 ceph-mon[31535]: pgmap v3915: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-03-24T02:07:32.704 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:07:32 smithi189 ceph-mon[36187]: pgmap v3915: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-03-24T02:07:32.711 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:07:32 smithi193 ceph-mon[36878]: pgmap v3915: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-03-24T02:07:34.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:07:34 smithi118 ceph-mon[31535]: pgmap v3916: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-03-24T02:07:34.704 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:07:34 smithi189 ceph-mon[36187]: pgmap v3916: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-03-24T02:07:34.711 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:07:34 smithi193 ceph-mon[36878]: pgmap v3916: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-03-24T02:07:36.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:07:36 smithi118 ceph-mon[31535]: pgmap v3917: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T02:07:36.697 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:07:36 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 6151.703083 secs 2024-03-24T02:07:36.704 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:07:36 smithi189 ceph-mon[36187]: pgmap v3917: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T02:07:36.704 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:07:36 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 6151.703083 secs 2024-03-24T02:07:36.711 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:07:36 smithi193 ceph-mon[36878]: pgmap v3917: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T02:07:36.711 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:07:36 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 6151.703083 secs 2024-03-24T02:07:38.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:07:38 smithi118 ceph-mon[31535]: pgmap v3918: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:07:38.704 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:07:38 smithi189 ceph-mon[36187]: pgmap v3918: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:07:38.711 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:07:38 smithi193 ceph-mon[36878]: pgmap v3918: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:07:40.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:07:40 smithi118 ceph-mon[31535]: pgmap v3919: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-24T02:07:40.704 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:07:40 smithi189 ceph-mon[36187]: pgmap v3919: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-24T02:07:40.711 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:07:40 smithi193 ceph-mon[36878]: pgmap v3919: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-24T02:07:41.605 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:07:41 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 6156.703168 secs 2024-03-24T02:07:41.612 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:07:41 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 6156.703168 secs 2024-03-24T02:07:41.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:07:41 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 6156.703168 secs 2024-03-24T02:07:42.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:07:42 smithi118 ceph-mon[31535]: pgmap v3920: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:07:42.704 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:07:42 smithi189 ceph-mon[36187]: pgmap v3920: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:07:42.711 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:07:42 smithi193 ceph-mon[36878]: pgmap v3920: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:07:44.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:07:44 smithi118 ceph-mon[31535]: pgmap v3921: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T02:07:44.704 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:07:44 smithi189 ceph-mon[36187]: pgmap v3921: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T02:07:44.711 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:07:44 smithi193 ceph-mon[36878]: pgmap v3921: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T02:07:46.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:07:46 smithi118 ceph-mon[31535]: pgmap v3922: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T02:07:46.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:07:46 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 6161.703255 secs 2024-03-24T02:07:46.704 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:07:46 smithi189 ceph-mon[36187]: pgmap v3922: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T02:07:46.704 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:07:46 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 6161.703255 secs 2024-03-24T02:07:46.711 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:07:46 smithi193 ceph-mon[36878]: pgmap v3922: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T02:07:46.711 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:07:46 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 6161.703255 secs 2024-03-24T02:07:48.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:07:48 smithi118 ceph-mon[31535]: pgmap v3923: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:07:48.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:07:48 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T02:07:48.704 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:07:48 smithi189 ceph-mon[36187]: pgmap v3923: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:07:48.704 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:07:48 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T02:07:48.711 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:07:48 smithi193 ceph-mon[36878]: pgmap v3923: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:07:48.711 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:07:48 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T02:07:49.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:07:49 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T02:07:49.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:07:49 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T02:07:49.697 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:07:49 smithi118 ceph-mon[31535]: pgmap v3924: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 6 op/s 2024-03-24T02:07:49.697 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:07:49 smithi118 ceph-mon[31535]: pgmap v3925: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 4 op/s 2024-03-24T02:07:49.697 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:07:49 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T02:07:49.704 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:07:49 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T02:07:49.704 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:07:49 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T02:07:49.704 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:07:49 smithi189 ceph-mon[36187]: pgmap v3924: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 6 op/s 2024-03-24T02:07:49.704 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:07:49 smithi189 ceph-mon[36187]: pgmap v3925: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 4 op/s 2024-03-24T02:07:49.704 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:07:49 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T02:07:49.711 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:07:49 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T02:07:49.711 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:07:49 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T02:07:49.711 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:07:49 smithi193 ceph-mon[36878]: pgmap v3924: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 6 op/s 2024-03-24T02:07:49.711 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:07:49 smithi193 ceph-mon[36878]: pgmap v3925: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 4 op/s 2024-03-24T02:07:49.711 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:07:49 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T02:07:52.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:07:51 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 6166.703318 secs 2024-03-24T02:07:52.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:07:51 smithi118 ceph-mon[31535]: pgmap v3926: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-03-24T02:07:52.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:07:51 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 6166.703318 secs 2024-03-24T02:07:52.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:07:51 smithi189 ceph-mon[36187]: pgmap v3926: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-03-24T02:07:52.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:07:51 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 6166.703318 secs 2024-03-24T02:07:52.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:07:51 smithi193 ceph-mon[36878]: pgmap v3926: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-03-24T02:07:54.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:07:53 smithi118 ceph-mon[31535]: pgmap v3927: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 5 op/s 2024-03-24T02:07:54.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:07:53 smithi189 ceph-mon[36187]: pgmap v3927: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 5 op/s 2024-03-24T02:07:54.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:07:53 smithi193 ceph-mon[36878]: pgmap v3927: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 5 op/s 2024-03-24T02:07:56.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:07:55 smithi118 ceph-mon[31535]: pgmap v3928: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-03-24T02:07:56.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:07:55 smithi189 ceph-mon[36187]: pgmap v3928: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-03-24T02:07:56.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:07:55 smithi193 ceph-mon[36878]: pgmap v3928: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-03-24T02:07:57.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:07:56 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 6171.703378 secs 2024-03-24T02:07:57.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:07:56 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 6171.703378 secs 2024-03-24T02:07:57.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:07:56 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 6171.703378 secs 2024-03-24T02:07:58.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:07:57 smithi118 ceph-mon[31535]: pgmap v3929: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-03-24T02:07:58.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:07:57 smithi189 ceph-mon[36187]: pgmap v3929: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-03-24T02:07:58.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:07:57 smithi193 ceph-mon[36878]: pgmap v3929: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-03-24T02:08:00.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:07:59 smithi118 ceph-mon[31535]: pgmap v3930: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 5 op/s 2024-03-24T02:08:00.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:07:59 smithi189 ceph-mon[36187]: pgmap v3930: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 5 op/s 2024-03-24T02:08:00.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:07:59 smithi193 ceph-mon[36878]: pgmap v3930: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 5 op/s 2024-03-24T02:08:01.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:08:00 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 6176.703436 secs 2024-03-24T02:08:01.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:08:00 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 6176.703436 secs 2024-03-24T02:08:01.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:08:00 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 6176.703436 secs 2024-03-24T02:08:02.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:08:01 smithi118 ceph-mon[31535]: pgmap v3931: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T02:08:02.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:08:01 smithi189 ceph-mon[36187]: pgmap v3931: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T02:08:02.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:08:01 smithi193 ceph-mon[36878]: pgmap v3931: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T02:08:04.197 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:08:03 smithi118 ceph-mon[31535]: pgmap v3932: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T02:08:04.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:08:03 smithi189 ceph-mon[36187]: pgmap v3932: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T02:08:04.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:08:03 smithi193 ceph-mon[36878]: pgmap v3932: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T02:08:06.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:08:05 smithi118 ceph-mon[31535]: pgmap v3933: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-03-24T02:08:06.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:08:05 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 6181.703521 secs 2024-03-24T02:08:06.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:08:05 smithi189 ceph-mon[36187]: pgmap v3933: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-03-24T02:08:06.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:08:05 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 6181.703521 secs 2024-03-24T02:08:06.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:08:05 smithi193 ceph-mon[36878]: pgmap v3933: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-03-24T02:08:06.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:08:05 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 6181.703521 secs 2024-03-24T02:08:08.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:08:07 smithi118 ceph-mon[31535]: pgmap v3934: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:08:08.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:08:07 smithi189 ceph-mon[36187]: pgmap v3934: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:08:08.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:08:07 smithi193 ceph-mon[36878]: pgmap v3934: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:08:10.197 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:08:09 smithi118 ceph-mon[31535]: pgmap v3935: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T02:08:10.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:08:09 smithi189 ceph-mon[36187]: pgmap v3935: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T02:08:10.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:08:09 smithi193 ceph-mon[36878]: pgmap v3935: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T02:08:11.105 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:08:10 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 6186.703611 secs 2024-03-24T02:08:11.112 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:08:10 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 6186.703611 secs 2024-03-24T02:08:11.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:08:10 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 6186.703611 secs 2024-03-24T02:08:12.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:08:11 smithi118 ceph-mon[31535]: pgmap v3936: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T02:08:12.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:08:11 smithi189 ceph-mon[36187]: pgmap v3936: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T02:08:12.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:08:11 smithi193 ceph-mon[36878]: pgmap v3936: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T02:08:14.197 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:08:13 smithi118 ceph-mon[31535]: pgmap v3937: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T02:08:14.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:08:13 smithi189 ceph-mon[36187]: pgmap v3937: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T02:08:14.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:08:13 smithi193 ceph-mon[36878]: pgmap v3937: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T02:08:16.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:08:15 smithi118 ceph-mon[31535]: pgmap v3938: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-03-24T02:08:16.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:08:15 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 6191.703695 secs 2024-03-24T02:08:16.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:08:15 smithi189 ceph-mon[36187]: pgmap v3938: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-03-24T02:08:16.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:08:15 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 6191.703695 secs 2024-03-24T02:08:16.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:08:15 smithi193 ceph-mon[36878]: pgmap v3938: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-03-24T02:08:16.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:08:15 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 6191.703695 secs 2024-03-24T02:08:18.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:08:17 smithi118 ceph-mon[31535]: pgmap v3939: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:08:18.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:08:17 smithi189 ceph-mon[36187]: pgmap v3939: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:08:18.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:08:17 smithi193 ceph-mon[36878]: pgmap v3939: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:08:20.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:08:19 smithi118 ceph-mon[31535]: pgmap v3940: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T02:08:20.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:08:19 smithi189 ceph-mon[36187]: pgmap v3940: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T02:08:20.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:08:19 smithi193 ceph-mon[36878]: pgmap v3940: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T02:08:21.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:08:20 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 6196.703755 secs 2024-03-24T02:08:21.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:08:20 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 6196.703755 secs 2024-03-24T02:08:21.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:08:20 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 6196.703755 secs 2024-03-24T02:08:22.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:08:21 smithi118 ceph-mon[31535]: pgmap v3941: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T02:08:22.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:08:21 smithi189 ceph-mon[36187]: pgmap v3941: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T02:08:22.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:08:21 smithi193 ceph-mon[36878]: pgmap v3941: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T02:08:24.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:08:23 smithi118 ceph-mon[31535]: pgmap v3942: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T02:08:24.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:08:23 smithi189 ceph-mon[36187]: pgmap v3942: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T02:08:24.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:08:23 smithi193 ceph-mon[36878]: pgmap v3942: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T02:08:26.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:08:25 smithi118 ceph-mon[31535]: pgmap v3943: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-03-24T02:08:26.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:08:25 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 6201.703842 secs 2024-03-24T02:08:26.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:08:25 smithi189 ceph-mon[36187]: pgmap v3943: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-03-24T02:08:26.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:08:25 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 6201.703842 secs 2024-03-24T02:08:26.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:08:25 smithi193 ceph-mon[36878]: pgmap v3943: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-03-24T02:08:26.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:08:25 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 6201.703842 secs 2024-03-24T02:08:28.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:08:27 smithi118 ceph-mon[31535]: pgmap v3944: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:08:28.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:08:27 smithi189 ceph-mon[36187]: pgmap v3944: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:08:28.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:08:27 smithi193 ceph-mon[36878]: pgmap v3944: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:08:30.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:08:29 smithi118 ceph-mon[31535]: pgmap v3945: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T02:08:30.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:08:29 smithi189 ceph-mon[36187]: pgmap v3945: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T02:08:30.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:08:29 smithi193 ceph-mon[36878]: pgmap v3945: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T02:08:31.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:08:30 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 6206.703930 secs 2024-03-24T02:08:31.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:08:30 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 6206.703930 secs 2024-03-24T02:08:31.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:08:30 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 6206.703930 secs 2024-03-24T02:08:32.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:08:31 smithi118 ceph-mon[31535]: pgmap v3946: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T02:08:32.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:08:31 smithi189 ceph-mon[36187]: pgmap v3946: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T02:08:32.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:08:31 smithi193 ceph-mon[36878]: pgmap v3946: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T02:08:34.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:08:33 smithi118 ceph-mon[31535]: pgmap v3947: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T02:08:34.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:08:33 smithi189 ceph-mon[36187]: pgmap v3947: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T02:08:34.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:08:33 smithi193 ceph-mon[36878]: pgmap v3947: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T02:08:36.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:08:35 smithi118 ceph-mon[31535]: pgmap v3948: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-03-24T02:08:36.197 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:08:35 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 6211.703991 secs 2024-03-24T02:08:36.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:08:35 smithi189 ceph-mon[36187]: pgmap v3948: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-03-24T02:08:36.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:08:35 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 6211.703991 secs 2024-03-24T02:08:36.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:08:35 smithi193 ceph-mon[36878]: pgmap v3948: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-03-24T02:08:36.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:08:35 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 6211.703991 secs 2024-03-24T02:08:38.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:08:37 smithi118 ceph-mon[31535]: pgmap v3949: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:08:38.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:08:37 smithi189 ceph-mon[36187]: pgmap v3949: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:08:38.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:08:37 smithi193 ceph-mon[36878]: pgmap v3949: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:08:40.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:08:39 smithi118 ceph-mon[31535]: pgmap v3950: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T02:08:40.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:08:39 smithi189 ceph-mon[36187]: pgmap v3950: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T02:08:40.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:08:39 smithi193 ceph-mon[36878]: pgmap v3950: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T02:08:41.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:08:40 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 6216.704080 secs 2024-03-24T02:08:41.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:08:40 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 6216.704080 secs 2024-03-24T02:08:41.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:08:40 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 6216.704080 secs 2024-03-24T02:08:42.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:08:41 smithi118 ceph-mon[31535]: pgmap v3951: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T02:08:42.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:08:41 smithi189 ceph-mon[36187]: pgmap v3951: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T02:08:42.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:08:41 smithi193 ceph-mon[36878]: pgmap v3951: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T02:08:44.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:08:43 smithi118 ceph-mon[31535]: pgmap v3952: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T02:08:44.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:08:43 smithi189 ceph-mon[36187]: pgmap v3952: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T02:08:44.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:08:43 smithi193 ceph-mon[36878]: pgmap v3952: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T02:08:46.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:08:45 smithi189 ceph-mon[36187]: pgmap v3953: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-03-24T02:08:46.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:08:45 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 6221.704144 secs 2024-03-24T02:08:46.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:08:45 smithi193 ceph-mon[36878]: pgmap v3953: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-03-24T02:08:46.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:08:45 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 6221.704144 secs 2024-03-24T02:08:46.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:08:45 smithi118 ceph-mon[31535]: pgmap v3953: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-03-24T02:08:46.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:08:45 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 6221.704144 secs 2024-03-24T02:08:48.206 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:08:47 smithi189 ceph-mon[36187]: pgmap v3954: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:08:48.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:08:47 smithi193 ceph-mon[36878]: pgmap v3954: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:08:48.249 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:08:47 smithi118 ceph-mon[31535]: pgmap v3954: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:08:49.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:08:48 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T02:08:49.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:08:48 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T02:08:49.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:08:48 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T02:08:50.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:08:49 smithi118 ceph-mon[31535]: pgmap v3955: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T02:08:50.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:08:49 smithi189 ceph-mon[36187]: pgmap v3955: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T02:08:50.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:08:49 smithi193 ceph-mon[36878]: pgmap v3955: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T02:08:51.224 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:08:50 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T02:08:51.225 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:08:50 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T02:08:51.225 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:08:50 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T02:08:51.225 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:08:50 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 6226.704181 secs 2024-03-24T02:08:51.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:08:50 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T02:08:51.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:08:50 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T02:08:51.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:08:50 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T02:08:51.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:08:50 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 6226.704181 secs 2024-03-24T02:08:51.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:08:50 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T02:08:51.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:08:50 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T02:08:51.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:08:50 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T02:08:51.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:08:50 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 6226.704181 secs 2024-03-24T02:08:52.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:08:51 smithi118 ceph-mon[31535]: pgmap v3956: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 6 op/s 2024-03-24T02:08:52.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:08:51 smithi118 ceph-mon[31535]: pgmap v3957: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 6 op/s 2024-03-24T02:08:52.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:08:51 smithi189 ceph-mon[36187]: pgmap v3956: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 6 op/s 2024-03-24T02:08:52.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:08:51 smithi189 ceph-mon[36187]: pgmap v3957: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 6 op/s 2024-03-24T02:08:52.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:08:51 smithi193 ceph-mon[36878]: pgmap v3956: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 6 op/s 2024-03-24T02:08:52.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:08:51 smithi193 ceph-mon[36878]: pgmap v3957: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 6 op/s 2024-03-24T02:08:54.296 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:08:53 smithi118 ceph-mon[31535]: pgmap v3958: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 5 op/s 2024-03-24T02:08:54.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:08:53 smithi189 ceph-mon[36187]: pgmap v3958: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 5 op/s 2024-03-24T02:08:54.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:08:53 smithi193 ceph-mon[36878]: pgmap v3958: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 5 op/s 2024-03-24T02:08:56.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:08:55 smithi118 ceph-mon[31535]: pgmap v3959: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 6 op/s 2024-03-24T02:08:56.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:08:55 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 6231.704242 secs 2024-03-24T02:08:56.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:08:55 smithi189 ceph-mon[36187]: pgmap v3959: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 6 op/s 2024-03-24T02:08:56.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:08:55 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 6231.704242 secs 2024-03-24T02:08:56.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:08:55 smithi193 ceph-mon[36878]: pgmap v3959: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 6 op/s 2024-03-24T02:08:56.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:08:55 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 6231.704242 secs 2024-03-24T02:08:58.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:08:57 smithi118 ceph-mon[31535]: pgmap v3960: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 6 op/s 2024-03-24T02:08:58.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:08:57 smithi189 ceph-mon[36187]: pgmap v3960: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 6 op/s 2024-03-24T02:08:58.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:08:57 smithi193 ceph-mon[36878]: pgmap v3960: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 6 op/s 2024-03-24T02:09:00.449 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:08:59 smithi118 ceph-mon[31535]: pgmap v3961: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 6 op/s 2024-03-24T02:09:00.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:08:59 smithi189 ceph-mon[36187]: pgmap v3961: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 6 op/s 2024-03-24T02:09:00.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:08:59 smithi193 ceph-mon[36878]: pgmap v3961: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 6 op/s 2024-03-24T02:09:01.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:09:01 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 6236.704305 secs 2024-03-24T02:09:01.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:09:01 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 6236.704305 secs 2024-03-24T02:09:01.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:09:01 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 6236.704305 secs 2024-03-24T02:09:02.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:09:02 smithi118 ceph-mon[31535]: pgmap v3962: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 5 op/s 2024-03-24T02:09:02.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:09:02 smithi189 ceph-mon[36187]: pgmap v3962: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 5 op/s 2024-03-24T02:09:02.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:09:02 smithi193 ceph-mon[36878]: pgmap v3962: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 5 op/s 2024-03-24T02:09:04.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:09:04 smithi118 ceph-mon[31535]: pgmap v3963: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:09:04.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:09:04 smithi189 ceph-mon[36187]: pgmap v3963: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:09:04.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:09:04 smithi193 ceph-mon[36878]: pgmap v3963: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:09:06.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:09:06 smithi118 ceph-mon[31535]: pgmap v3964: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-24T02:09:06.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:09:06 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 6241.704318 secs 2024-03-24T02:09:06.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:09:06 smithi189 ceph-mon[36187]: pgmap v3964: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-24T02:09:06.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:09:06 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 6241.704318 secs 2024-03-24T02:09:06.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:09:06 smithi193 ceph-mon[36878]: pgmap v3964: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-24T02:09:06.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:09:06 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 6241.704318 secs 2024-03-24T02:09:08.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:09:08 smithi118 ceph-mon[31535]: pgmap v3965: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:09:08.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:09:08 smithi189 ceph-mon[36187]: pgmap v3965: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:09:08.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:09:08 smithi193 ceph-mon[36878]: pgmap v3965: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:09:10.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:09:10 smithi118 ceph-mon[31535]: pgmap v3966: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T02:09:10.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:09:10 smithi189 ceph-mon[36187]: pgmap v3966: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T02:09:10.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:09:10 smithi193 ceph-mon[36878]: pgmap v3966: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T02:09:11.355 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:09:11 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 6246.704380 secs 2024-03-24T02:09:11.362 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:09:11 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 6246.704380 secs 2024-03-24T02:09:11.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:09:11 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 6246.704380 secs 2024-03-24T02:09:12.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:09:12 smithi118 ceph-mon[31535]: pgmap v3967: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T02:09:12.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:09:12 smithi189 ceph-mon[36187]: pgmap v3967: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T02:09:12.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:09:12 smithi193 ceph-mon[36878]: pgmap v3967: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T02:09:14.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:09:14 smithi118 ceph-mon[31535]: pgmap v3968: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:09:14.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:09:14 smithi189 ceph-mon[36187]: pgmap v3968: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:09:14.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:09:14 smithi193 ceph-mon[36878]: pgmap v3968: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:09:16.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:09:16 smithi118 ceph-mon[31535]: pgmap v3969: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-24T02:09:16.447 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:09:16 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 6251.704418 secs 2024-03-24T02:09:16.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:09:16 smithi189 ceph-mon[36187]: pgmap v3969: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-24T02:09:16.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:09:16 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 6251.704418 secs 2024-03-24T02:09:16.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:09:16 smithi193 ceph-mon[36878]: pgmap v3969: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-24T02:09:16.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:09:16 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 6251.704418 secs 2024-03-24T02:09:18.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:09:18 smithi118 ceph-mon[31535]: pgmap v3970: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:09:18.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:09:18 smithi189 ceph-mon[36187]: pgmap v3970: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:09:18.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:09:18 smithi193 ceph-mon[36878]: pgmap v3970: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:09:20.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:09:20 smithi118 ceph-mon[31535]: pgmap v3971: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T02:09:20.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:09:20 smithi189 ceph-mon[36187]: pgmap v3971: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T02:09:20.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:09:20 smithi193 ceph-mon[36878]: pgmap v3971: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T02:09:21.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:09:21 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 6256.704504 secs 2024-03-24T02:09:21.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:09:21 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 6256.704504 secs 2024-03-24T02:09:21.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:09:21 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 6256.704504 secs 2024-03-24T02:09:22.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:09:22 smithi118 ceph-mon[31535]: pgmap v3972: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T02:09:22.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:09:22 smithi189 ceph-mon[36187]: pgmap v3972: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T02:09:22.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:09:22 smithi193 ceph-mon[36878]: pgmap v3972: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T02:09:24.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:09:24 smithi118 ceph-mon[31535]: pgmap v3973: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:09:24.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:09:24 smithi189 ceph-mon[36187]: pgmap v3973: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:09:24.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:09:24 smithi193 ceph-mon[36878]: pgmap v3973: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:09:26.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:09:26 smithi118 ceph-mon[31535]: pgmap v3974: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-24T02:09:26.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:09:26 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 6261.704587 secs 2024-03-24T02:09:26.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:09:26 smithi189 ceph-mon[36187]: pgmap v3974: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-24T02:09:26.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:09:26 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 6261.704587 secs 2024-03-24T02:09:26.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:09:26 smithi193 ceph-mon[36878]: pgmap v3974: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-24T02:09:26.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:09:26 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 6261.704587 secs 2024-03-24T02:09:28.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:09:28 smithi118 ceph-mon[31535]: pgmap v3975: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:09:28.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:09:28 smithi189 ceph-mon[36187]: pgmap v3975: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:09:28.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:09:28 smithi193 ceph-mon[36878]: pgmap v3975: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:09:30.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:09:30 smithi118 ceph-mon[31535]: pgmap v3976: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T02:09:30.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:09:30 smithi189 ceph-mon[36187]: pgmap v3976: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T02:09:30.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:09:30 smithi193 ceph-mon[36878]: pgmap v3976: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T02:09:31.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:09:31 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 6266.704631 secs 2024-03-24T02:09:31.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:09:31 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 6266.704631 secs 2024-03-24T02:09:31.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:09:31 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 6266.704631 secs 2024-03-24T02:09:32.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:09:32 smithi118 ceph-mon[31535]: pgmap v3977: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T02:09:32.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:09:32 smithi189 ceph-mon[36187]: pgmap v3977: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T02:09:32.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:09:32 smithi193 ceph-mon[36878]: pgmap v3977: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T02:09:34.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:09:34 smithi118 ceph-mon[31535]: pgmap v3978: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:09:34.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:09:34 smithi189 ceph-mon[36187]: pgmap v3978: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:09:34.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:09:34 smithi193 ceph-mon[36878]: pgmap v3978: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:09:36.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:09:36 smithi118 ceph-mon[31535]: pgmap v3979: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-24T02:09:36.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:09:36 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 6271.704716 secs 2024-03-24T02:09:36.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:09:36 smithi189 ceph-mon[36187]: pgmap v3979: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-24T02:09:36.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:09:36 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 6271.704716 secs 2024-03-24T02:09:36.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:09:36 smithi193 ceph-mon[36878]: pgmap v3979: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-24T02:09:36.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:09:36 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 6271.704716 secs 2024-03-24T02:09:38.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:09:38 smithi118 ceph-mon[31535]: pgmap v3980: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:09:38.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:09:38 smithi189 ceph-mon[36187]: pgmap v3980: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:09:38.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:09:38 smithi193 ceph-mon[36878]: pgmap v3980: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:09:40.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:09:40 smithi118 ceph-mon[31535]: pgmap v3981: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T02:09:40.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:09:40 smithi189 ceph-mon[36187]: pgmap v3981: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T02:09:40.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:09:40 smithi193 ceph-mon[36878]: pgmap v3981: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T02:09:41.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:09:41 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 6276.704733 secs 2024-03-24T02:09:41.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:09:41 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 6276.704733 secs 2024-03-24T02:09:41.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:09:41 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 6276.704733 secs 2024-03-24T02:09:42.447 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:09:42 smithi118 ceph-mon[31535]: pgmap v3982: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 6 op/s 2024-03-24T02:09:42.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:09:42 smithi189 ceph-mon[36187]: pgmap v3982: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 6 op/s 2024-03-24T02:09:42.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:09:42 smithi193 ceph-mon[36878]: pgmap v3982: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 6 op/s 2024-03-24T02:09:44.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:09:44 smithi118 ceph-mon[31535]: pgmap v3983: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-03-24T02:09:44.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:09:44 smithi189 ceph-mon[36187]: pgmap v3983: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-03-24T02:09:44.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:09:44 smithi193 ceph-mon[36878]: pgmap v3983: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-03-24T02:09:46.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:09:46 smithi118 ceph-mon[31535]: pgmap v3984: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-24T02:09:46.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:09:46 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 6281.704822 secs 2024-03-24T02:09:46.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:09:46 smithi189 ceph-mon[36187]: pgmap v3984: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-24T02:09:46.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:09:46 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 6281.704822 secs 2024-03-24T02:09:46.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:09:46 smithi193 ceph-mon[36878]: pgmap v3984: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-24T02:09:46.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:09:46 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 6281.704822 secs 2024-03-24T02:09:48.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:09:48 smithi118 ceph-mon[31535]: pgmap v3985: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-03-24T02:09:48.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:09:48 smithi189 ceph-mon[36187]: pgmap v3985: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-03-24T02:09:48.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:09:48 smithi193 ceph-mon[36878]: pgmap v3985: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-03-24T02:09:50.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:09:50 smithi118 ceph-mon[31535]: pgmap v3986: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 5 op/s 2024-03-24T02:09:50.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:09:50 smithi189 ceph-mon[36187]: pgmap v3986: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 5 op/s 2024-03-24T02:09:50.460 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:09:50 smithi193 ceph-mon[36878]: pgmap v3986: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 5 op/s 2024-03-24T02:09:51.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:09:51 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 6286.704910 secs 2024-03-24T02:09:51.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:09:51 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T02:09:51.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:09:51 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T02:09:51.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:09:51 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T02:09:51.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:09:51 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 6286.704910 secs 2024-03-24T02:09:51.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:09:51 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T02:09:51.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:09:51 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T02:09:51.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:09:51 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T02:09:51.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:09:51 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 6286.704910 secs 2024-03-24T02:09:51.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:09:51 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T02:09:51.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:09:51 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T02:09:51.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:09:51 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T02:09:52.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:09:52 smithi118 ceph-mon[31535]: pgmap v3987: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 5 op/s 2024-03-24T02:09:52.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:09:52 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T02:09:52.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:09:52 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T02:09:52.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:09:52 smithi189 ceph-mon[36187]: pgmap v3987: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 5 op/s 2024-03-24T02:09:52.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:09:52 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T02:09:52.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:09:52 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T02:09:52.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:09:52 smithi193 ceph-mon[36878]: pgmap v3987: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 5 op/s 2024-03-24T02:09:52.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:09:52 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T02:09:52.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:09:52 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T02:09:54.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:09:53 smithi118 ceph-mon[31535]: pgmap v3988: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:09:54.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:09:53 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T02:09:54.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:09:53 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T02:09:54.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:09:53 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T02:09:54.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:09:53 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T02:09:54.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:09:53 smithi189 ceph-mon[36187]: pgmap v3988: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:09:54.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:09:53 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T02:09:54.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:09:53 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T02:09:54.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:09:53 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T02:09:54.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:09:53 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T02:09:54.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:09:53 smithi193 ceph-mon[36878]: pgmap v3988: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:09:54.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:09:53 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T02:09:54.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:09:53 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T02:09:54.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:09:53 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T02:09:54.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:09:53 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T02:09:56.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:09:55 smithi118 ceph-mon[31535]: pgmap v3989: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-24T02:09:56.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:09:55 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T02:09:56.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:09:55 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T02:09:56.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:09:55 smithi189 ceph-mon[36187]: pgmap v3989: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-24T02:09:56.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:09:55 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T02:09:56.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:09:55 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T02:09:56.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:09:55 smithi193 ceph-mon[36878]: pgmap v3989: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-24T02:09:56.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:09:55 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T02:09:56.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:09:55 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T02:09:57.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:09:56 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 6291.705001 secs 2024-03-24T02:09:57.197 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:09:56 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T02:09:57.197 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:09:56 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T02:09:57.197 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:09:56 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T02:09:57.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:09:56 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 6291.705001 secs 2024-03-24T02:09:57.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:09:56 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T02:09:57.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:09:56 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T02:09:57.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:09:56 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T02:09:57.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:09:56 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 6291.705001 secs 2024-03-24T02:09:57.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:09:56 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T02:09:57.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:09:56 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T02:09:57.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:09:56 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T02:09:58.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:09:57 smithi118 ceph-mon[31535]: pgmap v3990: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:09:58.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:09:57 smithi118 ceph-mon[31535]: pgmap v3991: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T02:09:58.197 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:09:57 smithi118 ceph-mon[31535]: pgmap v3992: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 5 op/s 2024-03-24T02:09:58.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:09:57 smithi189 ceph-mon[36187]: pgmap v3990: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:09:58.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:09:57 smithi189 ceph-mon[36187]: pgmap v3991: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T02:09:58.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:09:57 smithi189 ceph-mon[36187]: pgmap v3992: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 5 op/s 2024-03-24T02:09:58.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:09:57 smithi193 ceph-mon[36878]: pgmap v3990: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:09:58.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:09:57 smithi193 ceph-mon[36878]: pgmap v3991: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T02:09:58.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:09:57 smithi193 ceph-mon[36878]: pgmap v3992: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 5 op/s 2024-03-24T02:10:00.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:09:59 smithi118 ceph-mon[31535]: pgmap v3993: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 5 op/s 2024-03-24T02:10:00.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:09:59 smithi189 ceph-mon[36187]: pgmap v3993: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 5 op/s 2024-03-24T02:10:00.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:09:59 smithi193 ceph-mon[36878]: pgmap v3993: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 5 op/s 2024-03-24T02:10:01.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:10:00 smithi118 ceph-mon[31535]: overall HEALTH_WARN 6 failed cephadm daemon(s); 1 MDSs report slow requests 2024-03-24T02:10:01.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:10:00 smithi189 ceph-mon[36187]: overall HEALTH_WARN 6 failed cephadm daemon(s); 1 MDSs report slow requests 2024-03-24T02:10:01.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:10:00 smithi193 ceph-mon[36878]: overall HEALTH_WARN 6 failed cephadm daemon(s); 1 MDSs report slow requests 2024-03-24T02:10:02.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:10:01 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 6296.705101 secs 2024-03-24T02:10:02.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:10:01 smithi118 ceph-mon[31535]: pgmap v3994: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-03-24T02:10:02.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:10:01 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 6296.705101 secs 2024-03-24T02:10:02.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:10:01 smithi189 ceph-mon[36187]: pgmap v3994: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-03-24T02:10:02.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:10:01 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 6296.705101 secs 2024-03-24T02:10:02.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:10:01 smithi193 ceph-mon[36878]: pgmap v3994: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-03-24T02:10:04.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:10:03 smithi118 ceph-mon[31535]: pgmap v3995: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-03-24T02:10:04.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:10:03 smithi189 ceph-mon[36187]: pgmap v3995: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-03-24T02:10:04.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:10:03 smithi193 ceph-mon[36878]: pgmap v3995: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-03-24T02:10:06.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:10:05 smithi118 ceph-mon[31535]: pgmap v3996: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-03-24T02:10:06.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:10:05 smithi189 ceph-mon[36187]: pgmap v3996: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-03-24T02:10:06.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:10:05 smithi193 ceph-mon[36878]: pgmap v3996: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-03-24T02:10:07.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:10:06 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 6301.705161 secs 2024-03-24T02:10:07.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:10:06 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 6301.705161 secs 2024-03-24T02:10:07.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:10:06 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 6301.705161 secs 2024-03-24T02:10:08.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:10:07 smithi118 ceph-mon[31535]: pgmap v3997: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 5 op/s 2024-03-24T02:10:08.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:10:07 smithi189 ceph-mon[36187]: pgmap v3997: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 5 op/s 2024-03-24T02:10:08.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:10:07 smithi193 ceph-mon[36878]: pgmap v3997: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 5 op/s 2024-03-24T02:10:10.072 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:10:09 smithi118 ceph-mon[31535]: pgmap v3998: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T02:10:10.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:10:09 smithi189 ceph-mon[36187]: pgmap v3998: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T02:10:10.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:10:09 smithi193 ceph-mon[36878]: pgmap v3998: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T02:10:11.105 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:10:10 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 6306.705242 secs 2024-03-24T02:10:11.112 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:10:10 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 6306.705242 secs 2024-03-24T02:10:11.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:10:10 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 6306.705242 secs 2024-03-24T02:10:12.117 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:10:11 smithi189 ceph-mon[36187]: pgmap v3999: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T02:10:12.121 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:10:11 smithi118 ceph-mon[31535]: pgmap v3999: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T02:10:12.128 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:10:11 smithi193 ceph-mon[36878]: pgmap v3999: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T02:10:14.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:10:13 smithi118 ceph-mon[31535]: pgmap v4000: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T02:10:14.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:10:13 smithi189 ceph-mon[36187]: pgmap v4000: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T02:10:14.213 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:10:13 smithi193 ceph-mon[36878]: pgmap v4000: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T02:10:15.755 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T02:10:15.754+0000 7f81f6000640 1 -- 172.21.15.118:0/1693767697 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7f81dc0a3bd0 msgr2=0x7f81dc0b2b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 12 2024-03-24T02:10:15.755 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T02:10:15.754+0000 7f81f6000640 1 -- 172.21.15.118:0/1693767697 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7f81dc0a3bd0 msgr2=0x7f81dc0b2b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-03-24T02:10:15.755 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T02:10:15.754+0000 7f81f6000640 1 --2- 172.21.15.118:0/1693767697 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7f81dc0a3bd0 0x7f81dc0b2b10 secure :-1 s=READY pgs=520 cs=0 l=1 rev1=1 crypto rx=0x7f81e0010870 tx=0x7f81e0044040 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-03-24T02:10:15.755 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T02:10:15.755+0000 7f81f6000640 1 --2- 172.21.15.118:0/1693767697 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7f81dc0a3bd0 0x7f81dc0b2b10 secure :-1 s=READY pgs=520 cs=0 l=1 rev1=1 crypto rx=0x7f81e0010870 tx=0x7f81e0044040 comp rx=0 tx=0).stop 2024-03-24T02:10:15.756 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T02:10:15.755+0000 7f81f6000640 1 -- 172.21.15.118:0/1693767697 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7f81dc0a3bd0 msgr2=0x7f81dc0b2b10 unknown :-1 s=STATE_CLOSED l=1).tick see no progress in more than 10000000 us during connecting to v2:172.21.15.118:6800/2112912943, fault. 2024-03-24T02:10:15.756 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T02:10:15.755+0000 7f81ef400640 1 -- 172.21.15.118:0/1693767697 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7f81dc0a3bd0 msgr2=0x7f81dc0b2b10 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-03-24T02:10:15.756 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T02:10:15.755+0000 7f81ef400640 1 --2- 172.21.15.118:0/1693767697 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7f81dc0a3bd0 0x7f81dc0b2b10 unknown :-1 s=CLOSED pgs=520 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T02:10:15.756 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T02:10:15.755+0000 7f81ef400640 1 --2- 172.21.15.118:0/1693767697 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7f81dc0a4320 0x7f81dc0b05c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T02:10:15.756 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T02:10:15.756+0000 7f81f6000640 1 --2- 172.21.15.118:0/1693767697 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7f81dc0a4320 0x7f81dc0b05c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T02:10:15.757 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T02:10:15.757+0000 7f81f6000640 1 --2- 172.21.15.118:0/1693767697 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7f81dc0a4320 0x7f81dc0b05c0 secure :-1 s=READY pgs=542 cs=0 l=1 rev1=1 crypto rx=0x7f81e0010990 tx=0x7f81e0067040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T02:10:15.766 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T02:10:15.767+0000 7f81f6000640 1 -- 172.21.15.118:0/659686424 >> [v2:172.21.15.189:6838/4074484168,v1:172.21.15.189:6839/4074484168] conn(0x5622651b71e0 msgr2=0x562265215110 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).tick idle (900000839) for more than 900000000 us, fault. 2024-03-24T02:10:15.767 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T02:10:15.767+0000 7f81f6000640 1 --2- 172.21.15.118:0/659686424 >> [v2:172.21.15.189:6838/4074484168,v1:172.21.15.189:6839/4074484168] conn(0x5622651b71e0 0x562265215110 unknown :-1 s=READY pgs=561 cs=5 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault initiating reconnect 2024-03-24T02:10:15.767 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T02:10:15.767+0000 7f81f6000640 1 --2- 172.21.15.118:0/659686424 >> [v2:172.21.15.189:6838/4074484168,v1:172.21.15.189:6839/4074484168] conn(0x5622651b71e0 0x562265215110 unknown :-1 s=BANNER_CONNECTING pgs=561 cs=6 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T02:10:15.768 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T02:10:15.768+0000 7f81f6000640 1 --2- 172.21.15.118:0/659686424 >> [v2:172.21.15.189:6838/4074484168,v1:172.21.15.189:6839/4074484168] conn(0x5622651b71e0 0x562265215110 crc :-1 s=READY pgs=561 cs=6 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=76f3d534289740f0 server_cookie=b31a2335a59552de in_seq=0 out_seq=0 2024-03-24T02:10:15.768 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T02:10:15.768+0000 7f81e4c00640 10 client.27094 ms_handle_connect on v2:172.21.15.189:6838/4074484168 2024-03-24T02:10:16.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:10:15 smithi118 ceph-mon[31535]: pgmap v4001: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-03-24T02:10:16.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:10:15 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 6311.705334 secs 2024-03-24T02:10:16.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:10:15 smithi189 ceph-mon[36187]: pgmap v4001: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-03-24T02:10:16.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:10:15 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 6311.705334 secs 2024-03-24T02:10:16.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:10:15 smithi193 ceph-mon[36878]: pgmap v4001: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-03-24T02:10:16.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:10:15 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 6311.705334 secs 2024-03-24T02:10:16.747 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T02:10:16.748+0000 7f81db400640 1 -- 172.21.15.118:0/659686424 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f81c4006290 con 0x5622651a13f0 2024-03-24T02:10:16.748 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T02:10:16.748+0000 7f81ee000640 1 -- 172.21.15.118:0/1693767697 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f81c8006590 con 0x7f81f81052d0 2024-03-24T02:10:16.748 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T02:10:16.748+0000 7f81ef400640 1 -- 172.21.15.118:0/1693767697 <== mon.2 v2:172.21.15.189:3300/0 52 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (secure 0 0 0) 0x7f81c8006590 con 0x7f81f81052d0 2024-03-24T02:10:16.748 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T02:10:16.748+0000 7f81e4c00640 1 -- 172.21.15.118:0/659686424 <== mon.1 v2:172.21.15.193:3300/0 52 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 409+0+0 (secure 0 0 0) 0x7f81c4006290 con 0x5622651a13f0 2024-03-24T02:10:18.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:10:17 smithi118 ceph-mon[31535]: pgmap v4002: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:10:18.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:10:17 smithi189 ceph-mon[36187]: pgmap v4002: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:10:18.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:10:17 smithi193 ceph-mon[36878]: pgmap v4002: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:10:20.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:10:19 smithi118 ceph-mon[31535]: pgmap v4003: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T02:10:20.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:10:19 smithi189 ceph-mon[36187]: pgmap v4003: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T02:10:20.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:10:19 smithi193 ceph-mon[36878]: pgmap v4003: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T02:10:21.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:10:20 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 6316.705423 secs 2024-03-24T02:10:21.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:10:20 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 6316.705423 secs 2024-03-24T02:10:21.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:10:20 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 6316.705423 secs 2024-03-24T02:10:22.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:10:21 smithi118 ceph-mon[31535]: pgmap v4004: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T02:10:22.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:10:21 smithi189 ceph-mon[36187]: pgmap v4004: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T02:10:22.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:10:21 smithi193 ceph-mon[36878]: pgmap v4004: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T02:10:24.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:10:23 smithi118 ceph-mon[31535]: pgmap v4005: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T02:10:24.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:10:23 smithi189 ceph-mon[36187]: pgmap v4005: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T02:10:24.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:10:23 smithi193 ceph-mon[36878]: pgmap v4005: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T02:10:26.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:10:25 smithi118 ceph-mon[31535]: pgmap v4006: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-03-24T02:10:26.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:10:25 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 6321.705513 secs 2024-03-24T02:10:26.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:10:25 smithi189 ceph-mon[36187]: pgmap v4006: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-03-24T02:10:26.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:10:25 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 6321.705513 secs 2024-03-24T02:10:26.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:10:25 smithi193 ceph-mon[36878]: pgmap v4006: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-03-24T02:10:26.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:10:25 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 6321.705513 secs 2024-03-24T02:10:28.197 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:10:27 smithi118 ceph-mon[31535]: pgmap v4007: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:10:28.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:10:27 smithi189 ceph-mon[36187]: pgmap v4007: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:10:28.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:10:27 smithi193 ceph-mon[36878]: pgmap v4007: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:10:30.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:10:29 smithi118 ceph-mon[31535]: pgmap v4008: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T02:10:30.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:10:29 smithi189 ceph-mon[36187]: pgmap v4008: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T02:10:30.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:10:29 smithi193 ceph-mon[36878]: pgmap v4008: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T02:10:31.197 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:10:30 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 6326.705602 secs 2024-03-24T02:10:31.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:10:30 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 6326.705602 secs 2024-03-24T02:10:31.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:10:30 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 6326.705602 secs 2024-03-24T02:10:32.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:10:31 smithi118 ceph-mon[31535]: pgmap v4009: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T02:10:32.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:10:31 smithi189 ceph-mon[36187]: pgmap v4009: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T02:10:32.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:10:31 smithi193 ceph-mon[36878]: pgmap v4009: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T02:10:34.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:10:33 smithi118 ceph-mon[31535]: pgmap v4010: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T02:10:34.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:10:33 smithi189 ceph-mon[36187]: pgmap v4010: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T02:10:34.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:10:33 smithi193 ceph-mon[36878]: pgmap v4010: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T02:10:36.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:10:35 smithi118 ceph-mon[31535]: pgmap v4011: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-03-24T02:10:36.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:10:35 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 6331.705689 secs 2024-03-24T02:10:36.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:10:35 smithi189 ceph-mon[36187]: pgmap v4011: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-03-24T02:10:36.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:10:35 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 6331.705689 secs 2024-03-24T02:10:36.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:10:35 smithi193 ceph-mon[36878]: pgmap v4011: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-03-24T02:10:36.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:10:35 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 6331.705689 secs 2024-03-24T02:10:38.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:10:37 smithi118 ceph-mon[31535]: pgmap v4012: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:10:38.203 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:10:37 smithi189 ceph-mon[36187]: pgmap v4012: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:10:38.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:10:37 smithi193 ceph-mon[36878]: pgmap v4012: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:10:40.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:10:39 smithi118 ceph-mon[31535]: pgmap v4013: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T02:10:40.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:10:39 smithi189 ceph-mon[36187]: pgmap v4013: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T02:10:40.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:10:39 smithi193 ceph-mon[36878]: pgmap v4013: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T02:10:41.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:10:40 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 6336.705774 secs 2024-03-24T02:10:41.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:10:40 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 6336.705774 secs 2024-03-24T02:10:41.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:10:40 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 6336.705774 secs 2024-03-24T02:10:42.066 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:10:41 smithi118 ceph-mon[31535]: pgmap v4014: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T02:10:42.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:10:41 smithi189 ceph-mon[36187]: pgmap v4014: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T02:10:42.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:10:41 smithi193 ceph-mon[36878]: pgmap v4014: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T02:10:44.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:10:43 smithi118 ceph-mon[31535]: pgmap v4015: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T02:10:44.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:10:43 smithi189 ceph-mon[36187]: pgmap v4015: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T02:10:44.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:10:43 smithi193 ceph-mon[36878]: pgmap v4015: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T02:10:46.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:10:45 smithi118 ceph-mon[31535]: pgmap v4016: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-03-24T02:10:46.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:10:45 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 6341.705832 secs 2024-03-24T02:10:46.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:10:45 smithi189 ceph-mon[36187]: pgmap v4016: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-03-24T02:10:46.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:10:45 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 6341.705832 secs 2024-03-24T02:10:46.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:10:45 smithi193 ceph-mon[36878]: pgmap v4016: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-03-24T02:10:46.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:10:45 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 6341.705832 secs 2024-03-24T02:10:48.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:10:47 smithi118 ceph-mon[31535]: pgmap v4017: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:10:48.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:10:47 smithi189 ceph-mon[36187]: pgmap v4017: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:10:48.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:10:47 smithi193 ceph-mon[36878]: pgmap v4017: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:10:50.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:10:49 smithi118 ceph-mon[31535]: pgmap v4018: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T02:10:50.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:10:49 smithi189 ceph-mon[36187]: pgmap v4018: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T02:10:50.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:10:49 smithi193 ceph-mon[36878]: pgmap v4018: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T02:10:51.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:10:50 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 6346.705919 secs 2024-03-24T02:10:51.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:10:50 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 6346.705919 secs 2024-03-24T02:10:51.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:10:50 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 6346.705919 secs 2024-03-24T02:10:52.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:10:51 smithi118 ceph-mon[31535]: pgmap v4019: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T02:10:52.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:10:51 smithi189 ceph-mon[36187]: pgmap v4019: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T02:10:52.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:10:51 smithi193 ceph-mon[36878]: pgmap v4019: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T02:10:54.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:10:53 smithi118 ceph-mon[31535]: pgmap v4020: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T02:10:54.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:10:53 smithi189 ceph-mon[36187]: pgmap v4020: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T02:10:54.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:10:53 smithi193 ceph-mon[36878]: pgmap v4020: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T02:10:56.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:10:55 smithi118 ceph-mon[31535]: pgmap v4021: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-03-24T02:10:56.197 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:10:55 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 6351.706014 secs 2024-03-24T02:10:56.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:10:55 smithi189 ceph-mon[36187]: pgmap v4021: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-03-24T02:10:56.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:10:55 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 6351.706014 secs 2024-03-24T02:10:56.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:10:55 smithi193 ceph-mon[36878]: pgmap v4021: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-03-24T02:10:56.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:10:55 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 6351.706014 secs 2024-03-24T02:10:57.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:10:56 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T02:10:57.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:10:56 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T02:10:57.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:10:56 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T02:10:58.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:10:57 smithi118 ceph-mon[31535]: pgmap v4022: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:10:58.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:10:57 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T02:10:58.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:10:57 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T02:10:58.203 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:10:57 smithi189 ceph-mon[36187]: pgmap v4022: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:10:58.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:10:57 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T02:10:58.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:10:57 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T02:10:58.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:10:57 smithi193 ceph-mon[36878]: pgmap v4022: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:10:58.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:10:57 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T02:10:58.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:10:57 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T02:10:59.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:10:58 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T02:10:59.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:10:58 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T02:10:59.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:10:58 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T02:10:59.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:10:58 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T02:10:59.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:10:58 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T02:10:59.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:10:58 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T02:10:59.224 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:10:58 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T02:10:59.224 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:10:58 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T02:10:59.224 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:10:58 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T02:11:00.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:10:59 smithi193 ceph-mon[36878]: pgmap v4023: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 6 op/s 2024-03-24T02:11:00.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:10:59 smithi193 ceph-mon[36878]: pgmap v4024: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 6 op/s 2024-03-24T02:11:00.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:10:59 smithi118 ceph-mon[31535]: pgmap v4023: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 6 op/s 2024-03-24T02:11:00.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:10:59 smithi118 ceph-mon[31535]: pgmap v4024: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 6 op/s 2024-03-24T02:11:00.453 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:10:59 smithi189 ceph-mon[36187]: pgmap v4023: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 6 op/s 2024-03-24T02:11:00.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:10:59 smithi189 ceph-mon[36187]: pgmap v4024: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 6 op/s 2024-03-24T02:11:01.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:11:00 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 6356.706103 secs 2024-03-24T02:11:01.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:11:00 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 6356.706103 secs 2024-03-24T02:11:01.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:11:00 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 6356.706103 secs 2024-03-24T02:11:02.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:11:01 smithi118 ceph-mon[31535]: pgmap v4025: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 6 op/s 2024-03-24T02:11:02.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:11:01 smithi189 ceph-mon[36187]: pgmap v4025: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 6 op/s 2024-03-24T02:11:02.460 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:11:01 smithi193 ceph-mon[36878]: pgmap v4025: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 6 op/s 2024-03-24T02:11:04.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:11:03 smithi118 ceph-mon[31535]: pgmap v4026: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 5 op/s 2024-03-24T02:11:04.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:11:03 smithi189 ceph-mon[36187]: pgmap v4026: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 5 op/s 2024-03-24T02:11:04.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:11:03 smithi193 ceph-mon[36878]: pgmap v4026: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 5 op/s 2024-03-24T02:11:06.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:11:05 smithi118 ceph-mon[31535]: pgmap v4027: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 6 op/s 2024-03-24T02:11:06.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:11:05 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 6361.706184 secs 2024-03-24T02:11:06.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:11:05 smithi189 ceph-mon[36187]: pgmap v4027: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 6 op/s 2024-03-24T02:11:06.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:11:05 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 6361.706184 secs 2024-03-24T02:11:06.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:11:05 smithi193 ceph-mon[36878]: pgmap v4027: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 6 op/s 2024-03-24T02:11:06.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:11:05 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 6361.706184 secs 2024-03-24T02:11:08.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:11:07 smithi118 ceph-mon[31535]: pgmap v4028: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 6 op/s 2024-03-24T02:11:08.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:11:07 smithi189 ceph-mon[36187]: pgmap v4028: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 6 op/s 2024-03-24T02:11:08.460 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:11:07 smithi193 ceph-mon[36878]: pgmap v4028: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 6 op/s 2024-03-24T02:11:10.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:11:09 smithi118 ceph-mon[31535]: pgmap v4029: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 5 op/s 2024-03-24T02:11:10.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:11:09 smithi189 ceph-mon[36187]: pgmap v4029: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 5 op/s 2024-03-24T02:11:10.460 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:11:09 smithi193 ceph-mon[36878]: pgmap v4029: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 5 op/s 2024-03-24T02:11:11.355 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:11:10 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 6366.706271 secs 2024-03-24T02:11:11.362 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:11:10 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 6366.706271 secs 2024-03-24T02:11:11.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:11:10 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 6366.706271 secs 2024-03-24T02:11:12.348 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:11:11 smithi118 ceph-mon[31535]: pgmap v4030: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T02:11:12.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:11:11 smithi189 ceph-mon[36187]: pgmap v4030: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T02:11:12.460 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:11:11 smithi193 ceph-mon[36878]: pgmap v4030: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T02:11:14.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:11:14 smithi118 ceph-mon[31535]: pgmap v4031: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:11:14.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:11:14 smithi189 ceph-mon[36187]: pgmap v4031: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:11:14.460 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:11:14 smithi193 ceph-mon[36878]: pgmap v4031: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:11:16.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:11:16 smithi118 ceph-mon[31535]: pgmap v4032: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-24T02:11:16.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:11:16 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 6371.706332 secs 2024-03-24T02:11:16.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:11:16 smithi189 ceph-mon[36187]: pgmap v4032: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-24T02:11:16.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:11:16 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 6371.706332 secs 2024-03-24T02:11:16.460 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:11:16 smithi193 ceph-mon[36878]: pgmap v4032: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-24T02:11:16.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:11:16 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 6371.706332 secs 2024-03-24T02:11:18.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:11:18 smithi118 ceph-mon[31535]: pgmap v4033: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:11:18.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:11:18 smithi189 ceph-mon[36187]: pgmap v4033: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:11:18.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:11:18 smithi193 ceph-mon[36878]: pgmap v4033: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:11:20.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:11:20 smithi118 ceph-mon[31535]: pgmap v4034: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T02:11:20.453 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:11:20 smithi189 ceph-mon[36187]: pgmap v4034: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T02:11:20.460 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:11:20 smithi193 ceph-mon[36878]: pgmap v4034: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T02:11:21.348 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:11:21 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 6376.706417 secs 2024-03-24T02:11:21.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:11:21 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 6376.706417 secs 2024-03-24T02:11:21.460 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:11:21 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 6376.706417 secs 2024-03-24T02:11:22.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:11:22 smithi118 ceph-mon[31535]: pgmap v4035: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T02:11:22.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:11:22 smithi189 ceph-mon[36187]: pgmap v4035: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T02:11:22.460 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:11:22 smithi193 ceph-mon[36878]: pgmap v4035: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T02:11:24.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:11:24 smithi118 ceph-mon[31535]: pgmap v4036: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:11:24.453 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:11:24 smithi189 ceph-mon[36187]: pgmap v4036: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:11:24.460 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:11:24 smithi193 ceph-mon[36878]: pgmap v4036: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:11:26.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:11:26 smithi118 ceph-mon[31535]: pgmap v4037: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-24T02:11:26.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:11:26 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 6381.706507 secs 2024-03-24T02:11:26.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:11:26 smithi189 ceph-mon[36187]: pgmap v4037: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-24T02:11:26.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:11:26 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 6381.706507 secs 2024-03-24T02:11:26.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:11:26 smithi193 ceph-mon[36878]: pgmap v4037: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-24T02:11:26.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:11:26 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 6381.706507 secs 2024-03-24T02:11:28.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:11:28 smithi118 ceph-mon[31535]: pgmap v4038: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:11:28.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:11:28 smithi189 ceph-mon[36187]: pgmap v4038: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:11:28.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:11:28 smithi193 ceph-mon[36878]: pgmap v4038: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:11:30.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:11:30 smithi118 ceph-mon[31535]: pgmap v4039: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T02:11:30.453 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:11:30 smithi189 ceph-mon[36187]: pgmap v4039: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T02:11:30.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:11:30 smithi193 ceph-mon[36878]: pgmap v4039: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T02:11:31.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:11:31 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 6386.706565 secs 2024-03-24T02:11:31.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:11:31 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 6386.706565 secs 2024-03-24T02:11:31.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:11:31 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 6386.706565 secs 2024-03-24T02:11:32.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:11:32 smithi118 ceph-mon[31535]: pgmap v4040: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T02:11:32.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:11:32 smithi189 ceph-mon[36187]: pgmap v4040: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T02:11:32.460 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:11:32 smithi193 ceph-mon[36878]: pgmap v4040: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T02:11:34.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:11:34 smithi118 ceph-mon[31535]: pgmap v4041: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:11:34.453 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:11:34 smithi189 ceph-mon[36187]: pgmap v4041: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:11:34.460 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:11:34 smithi193 ceph-mon[36878]: pgmap v4041: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:11:36.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:11:36 smithi118 ceph-mon[31535]: pgmap v4042: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-24T02:11:36.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:11:36 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 6391.706653 secs 2024-03-24T02:11:36.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:11:36 smithi189 ceph-mon[36187]: pgmap v4042: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-24T02:11:36.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:11:36 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 6391.706653 secs 2024-03-24T02:11:36.460 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:11:36 smithi193 ceph-mon[36878]: pgmap v4042: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-24T02:11:36.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:11:36 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 6391.706653 secs 2024-03-24T02:11:38.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:11:38 smithi118 ceph-mon[31535]: pgmap v4043: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:11:38.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:11:38 smithi189 ceph-mon[36187]: pgmap v4043: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:11:38.460 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:11:38 smithi193 ceph-mon[36878]: pgmap v4043: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:11:40.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:11:40 smithi118 ceph-mon[31535]: pgmap v4044: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T02:11:40.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:11:40 smithi189 ceph-mon[36187]: pgmap v4044: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T02:11:40.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:11:40 smithi193 ceph-mon[36878]: pgmap v4044: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T02:11:41.354 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:11:41 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 6396.706721 secs 2024-03-24T02:11:41.362 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:11:41 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 6396.706721 secs 2024-03-24T02:11:41.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:11:41 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 6396.706721 secs 2024-03-24T02:11:42.348 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:11:42 smithi118 ceph-mon[31535]: pgmap v4045: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T02:11:42.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:11:42 smithi189 ceph-mon[36187]: pgmap v4045: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T02:11:42.460 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:11:42 smithi193 ceph-mon[36878]: pgmap v4045: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T02:11:44.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:11:44 smithi118 ceph-mon[31535]: pgmap v4046: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:11:44.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:11:44 smithi189 ceph-mon[36187]: pgmap v4046: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:11:44.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:11:44 smithi193 ceph-mon[36878]: pgmap v4046: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:11:46.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:11:46 smithi118 ceph-mon[31535]: pgmap v4047: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-24T02:11:46.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:11:46 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 6401.706807 secs 2024-03-24T02:11:46.453 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:11:46 smithi189 ceph-mon[36187]: pgmap v4047: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-24T02:11:46.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:11:46 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 6401.706807 secs 2024-03-24T02:11:46.460 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:11:46 smithi193 ceph-mon[36878]: pgmap v4047: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-24T02:11:46.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:11:46 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 6401.706807 secs 2024-03-24T02:11:48.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:11:48 smithi118 ceph-mon[31535]: pgmap v4048: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:11:48.453 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:11:48 smithi189 ceph-mon[36187]: pgmap v4048: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:11:48.460 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:11:48 smithi193 ceph-mon[36878]: pgmap v4048: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:11:50.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:11:50 smithi118 ceph-mon[31535]: pgmap v4049: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T02:11:50.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:11:50 smithi189 ceph-mon[36187]: pgmap v4049: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T02:11:50.460 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:11:50 smithi193 ceph-mon[36878]: pgmap v4049: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T02:11:51.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:11:51 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 6406.706819 secs 2024-03-24T02:11:51.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:11:51 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 6406.706819 secs 2024-03-24T02:11:51.460 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:11:51 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 6406.706819 secs 2024-03-24T02:11:52.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:11:52 smithi118 ceph-mon[31535]: pgmap v4050: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T02:11:52.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:11:52 smithi189 ceph-mon[36187]: pgmap v4050: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T02:11:52.460 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:11:52 smithi193 ceph-mon[36878]: pgmap v4050: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T02:11:54.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:11:54 smithi118 ceph-mon[31535]: pgmap v4051: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:11:54.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:11:54 smithi189 ceph-mon[36187]: pgmap v4051: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:11:54.460 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:11:54 smithi193 ceph-mon[36878]: pgmap v4051: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:11:56.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:11:56 smithi118 ceph-mon[31535]: pgmap v4052: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-24T02:11:56.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:11:56 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 6411.706906 secs 2024-03-24T02:11:56.453 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:11:56 smithi189 ceph-mon[36187]: pgmap v4052: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-24T02:11:56.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:11:56 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 6411.706906 secs 2024-03-24T02:11:56.460 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:11:56 smithi193 ceph-mon[36878]: pgmap v4052: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-24T02:11:56.460 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:11:56 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 6411.706906 secs 2024-03-24T02:11:58.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:11:58 smithi118 ceph-mon[31535]: pgmap v4053: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:11:58.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:11:58 smithi189 ceph-mon[36187]: pgmap v4053: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:11:58.460 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:11:58 smithi193 ceph-mon[36878]: pgmap v4053: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:11:59.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:11:59 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T02:11:59.453 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:11:59 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T02:11:59.460 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:11:59 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T02:12:00.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:12:00 smithi118 ceph-mon[31535]: pgmap v4054: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T02:12:00.453 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:12:00 smithi189 ceph-mon[36187]: pgmap v4054: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T02:12:00.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:12:00 smithi193 ceph-mon[36878]: pgmap v4054: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T02:12:01.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:12:01 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 6416.706967 secs 2024-03-24T02:12:01.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:12:01 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 6416.706967 secs 2024-03-24T02:12:01.460 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:12:01 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 6416.706967 secs 2024-03-24T02:12:02.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:12:02 smithi118 ceph-mon[31535]: pgmap v4055: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T02:12:02.453 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:12:02 smithi189 ceph-mon[36187]: pgmap v4055: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T02:12:02.460 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:12:02 smithi193 ceph-mon[36878]: pgmap v4055: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T02:12:04.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:12:04 smithi118 ceph-mon[31535]: pgmap v4056: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:12:04.453 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:12:04 smithi189 ceph-mon[36187]: pgmap v4056: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:12:04.460 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:12:04 smithi193 ceph-mon[36878]: pgmap v4056: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:12:06.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:12:06 smithi118 ceph-mon[31535]: pgmap v4057: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-24T02:12:06.447 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:12:06 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 6421.707051 secs 2024-03-24T02:12:06.447 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:12:06 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T02:12:06.447 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:12:06 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T02:12:06.447 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:12:06 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T02:12:06.447 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:12:06 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T02:12:06.447 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:12:06 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T02:12:06.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:12:06 smithi189 ceph-mon[36187]: pgmap v4057: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-24T02:12:06.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:12:06 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 6421.707051 secs 2024-03-24T02:12:06.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:12:06 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T02:12:06.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:12:06 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T02:12:06.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:12:06 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T02:12:06.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:12:06 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T02:12:06.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:12:06 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T02:12:06.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:12:06 smithi193 ceph-mon[36878]: pgmap v4057: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-24T02:12:06.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:12:06 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 6421.707051 secs 2024-03-24T02:12:06.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:12:06 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T02:12:06.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:12:06 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T02:12:06.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:12:06 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T02:12:06.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:12:06 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T02:12:06.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:12:06 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T02:12:07.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:12:07 smithi118 ceph-mon[31535]: pgmap v4058: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 5 op/s 2024-03-24T02:12:07.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:12:07 smithi118 ceph-mon[31535]: pgmap v4059: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-03-24T02:12:07.453 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:12:07 smithi189 ceph-mon[36187]: pgmap v4058: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 5 op/s 2024-03-24T02:12:07.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:12:07 smithi189 ceph-mon[36187]: pgmap v4059: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-03-24T02:12:07.460 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:12:07 smithi193 ceph-mon[36878]: pgmap v4058: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 5 op/s 2024-03-24T02:12:07.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:12:07 smithi193 ceph-mon[36878]: pgmap v4059: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-03-24T02:12:09.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:12:09 smithi118 ceph-mon[31535]: pgmap v4060: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 5 op/s 2024-03-24T02:12:09.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:12:09 smithi189 ceph-mon[36187]: pgmap v4060: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 5 op/s 2024-03-24T02:12:09.460 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:12:09 smithi193 ceph-mon[36878]: pgmap v4060: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 5 op/s 2024-03-24T02:12:11.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:12:11 smithi118 ceph-mon[31535]: pgmap v4061: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-03-24T02:12:11.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:12:11 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 6426.707092 secs 2024-03-24T02:12:11.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:12:11 smithi189 ceph-mon[36187]: pgmap v4061: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-03-24T02:12:11.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:12:11 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 6426.707092 secs 2024-03-24T02:12:11.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:12:11 smithi193 ceph-mon[36878]: pgmap v4061: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-03-24T02:12:11.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:12:11 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 6426.707092 secs 2024-03-24T02:12:13.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:12:13 smithi118 ceph-mon[31535]: pgmap v4062: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-03-24T02:12:13.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:12:13 smithi189 ceph-mon[36187]: pgmap v4062: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-03-24T02:12:13.460 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:12:13 smithi193 ceph-mon[36878]: pgmap v4062: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-03-24T02:12:15.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:12:15 smithi118 ceph-mon[31535]: pgmap v4063: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 4 op/s 2024-03-24T02:12:15.453 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:12:15 smithi189 ceph-mon[36187]: pgmap v4063: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 4 op/s 2024-03-24T02:12:15.460 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:12:15 smithi193 ceph-mon[36878]: pgmap v4063: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 4 op/s 2024-03-24T02:12:16.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:12:16 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 6431.707129 secs 2024-03-24T02:12:16.453 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:12:16 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 6431.707129 secs 2024-03-24T02:12:16.460 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:12:16 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 6431.707129 secs 2024-03-24T02:12:17.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:12:17 smithi118 ceph-mon[31535]: pgmap v4064: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 5 op/s 2024-03-24T02:12:17.453 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:12:17 smithi189 ceph-mon[36187]: pgmap v4064: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 5 op/s 2024-03-24T02:12:17.460 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:12:17 smithi193 ceph-mon[36878]: pgmap v4064: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 5 op/s 2024-03-24T02:12:19.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:12:19 smithi189 ceph-mon[36187]: pgmap v4065: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T02:12:19.460 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:12:19 smithi193 ceph-mon[36878]: pgmap v4065: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T02:12:19.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:12:19 smithi118 ceph-mon[31535]: pgmap v4065: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T02:12:21.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:12:21 smithi189 ceph-mon[36187]: pgmap v4066: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-03-24T02:12:21.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:12:21 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 6436.707204 secs 2024-03-24T02:12:21.460 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:12:21 smithi193 ceph-mon[36878]: pgmap v4066: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-03-24T02:12:21.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:12:21 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 6436.707204 secs 2024-03-24T02:12:21.498 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:12:21 smithi118 ceph-mon[31535]: pgmap v4066: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-03-24T02:12:21.498 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:12:21 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 6436.707204 secs 2024-03-24T02:12:23.460 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:12:23 smithi193 ceph-mon[36878]: pgmap v4067: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:12:23.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:12:23 smithi118 ceph-mon[31535]: pgmap v4067: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:12:23.703 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:12:23 smithi189 ceph-mon[36187]: pgmap v4067: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:12:25.460 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:12:25 smithi193 ceph-mon[36878]: pgmap v4068: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T02:12:25.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:12:25 smithi118 ceph-mon[31535]: pgmap v4068: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T02:12:25.704 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:12:25 smithi189 ceph-mon[36187]: pgmap v4068: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T02:12:26.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:12:26 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 6441.707265 secs 2024-03-24T02:12:26.703 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:12:26 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 6441.707265 secs 2024-03-24T02:12:26.710 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:12:26 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 6441.707265 secs 2024-03-24T02:12:27.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:12:27 smithi118 ceph-mon[31535]: pgmap v4069: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T02:12:27.703 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:12:27 smithi189 ceph-mon[36187]: pgmap v4069: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T02:12:27.710 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:12:27 smithi193 ceph-mon[36878]: pgmap v4069: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T02:12:29.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:12:29 smithi118 ceph-mon[31535]: pgmap v4070: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T02:12:29.703 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:12:29 smithi189 ceph-mon[36187]: pgmap v4070: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T02:12:29.710 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:12:29 smithi193 ceph-mon[36878]: pgmap v4070: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T02:12:31.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:12:31 smithi118 ceph-mon[31535]: pgmap v4071: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-03-24T02:12:31.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:12:31 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 6446.707348 secs 2024-03-24T02:12:31.704 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:12:31 smithi189 ceph-mon[36187]: pgmap v4071: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-03-24T02:12:31.704 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:12:31 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 6446.707348 secs 2024-03-24T02:12:31.710 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:12:31 smithi193 ceph-mon[36878]: pgmap v4071: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-03-24T02:12:31.710 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:12:31 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 6446.707348 secs 2024-03-24T02:12:31.769 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T02:12:31.769+0000 7f81db400640 1 -- 172.21.15.118:0/659686424 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- auth(proto 2 132 bytes epoch 0) v1 -- 0x7f81c4006290 con 0x5622651a13f0 2024-03-24T02:12:31.770 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T02:12:31.769+0000 7f81ee000640 1 -- 172.21.15.118:0/1693767697 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- auth(proto 2 132 bytes epoch 0) v1 -- 0x7f81c8006590 con 0x7f81f81052d0 2024-03-24T02:12:31.770 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T02:12:31.770+0000 7f81ef400640 1 -- 172.21.15.118:0/1693767697 <== mon.2 v2:172.21.15.189:3300/0 53 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 233+0+0 (secure 0 0 0) 0x7f81c8006590 con 0x7f81f81052d0 2024-03-24T02:12:31.770 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T02:12:31.770+0000 7f81e4c00640 1 -- 172.21.15.118:0/659686424 <== mon.1 v2:172.21.15.193:3300/0 53 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 233+0+0 (secure 0 0 0) 0x7f81c4006290 con 0x5622651a13f0 2024-03-24T02:12:32.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:12:32 smithi118 ceph-mon[31535]: pgmap v4072: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:12:32.703 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:12:32 smithi189 ceph-mon[36187]: pgmap v4072: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:12:32.710 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:12:32 smithi193 ceph-mon[36878]: pgmap v4072: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:12:34.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:12:34 smithi118 ceph-mon[31535]: pgmap v4073: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T02:12:34.953 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:12:34 smithi189 ceph-mon[36187]: pgmap v4073: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T02:12:34.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:12:34 smithi193 ceph-mon[36878]: pgmap v4073: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T02:12:36.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:12:36 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 6451.707429 secs 2024-03-24T02:12:36.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:12:36 smithi118 ceph-mon[31535]: pgmap v4074: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T02:12:36.953 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:12:36 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 6451.707429 secs 2024-03-24T02:12:36.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:12:36 smithi189 ceph-mon[36187]: pgmap v4074: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T02:12:36.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:12:36 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 6451.707429 secs 2024-03-24T02:12:36.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:12:36 smithi193 ceph-mon[36878]: pgmap v4074: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T02:12:38.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:12:38 smithi118 ceph-mon[31535]: pgmap v4075: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T02:12:38.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:12:38 smithi189 ceph-mon[36187]: pgmap v4075: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T02:12:38.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:12:38 smithi193 ceph-mon[36878]: pgmap v4075: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T02:12:40.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:12:40 smithi118 ceph-mon[31535]: pgmap v4076: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-03-24T02:12:40.953 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:12:40 smithi189 ceph-mon[36187]: pgmap v4076: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-03-24T02:12:40.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:12:40 smithi193 ceph-mon[36878]: pgmap v4076: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-03-24T02:12:41.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:12:41 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 6456.707518 secs 2024-03-24T02:12:41.953 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:12:41 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 6456.707518 secs 2024-03-24T02:12:41.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:12:41 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 6456.707518 secs 2024-03-24T02:12:42.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:12:42 smithi118 ceph-mon[31535]: pgmap v4077: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:12:42.953 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:12:42 smithi189 ceph-mon[36187]: pgmap v4077: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:12:42.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:12:42 smithi193 ceph-mon[36878]: pgmap v4077: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:12:44.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:12:44 smithi118 ceph-mon[31535]: pgmap v4078: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T02:12:44.953 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:12:44 smithi189 ceph-mon[36187]: pgmap v4078: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T02:12:44.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:12:44 smithi193 ceph-mon[36878]: pgmap v4078: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T02:12:46.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:12:46 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 6461.707577 secs 2024-03-24T02:12:46.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:12:46 smithi118 ceph-mon[31535]: pgmap v4079: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T02:12:46.953 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:12:46 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 6461.707577 secs 2024-03-24T02:12:46.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:12:46 smithi189 ceph-mon[36187]: pgmap v4079: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T02:12:46.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:12:46 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 6461.707577 secs 2024-03-24T02:12:46.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:12:46 smithi193 ceph-mon[36878]: pgmap v4079: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T02:12:48.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:12:48 smithi118 ceph-mon[31535]: pgmap v4080: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T02:12:48.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:12:48 smithi189 ceph-mon[36187]: pgmap v4080: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T02:12:48.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:12:48 smithi193 ceph-mon[36878]: pgmap v4080: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T02:12:50.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:12:50 smithi118 ceph-mon[31535]: pgmap v4081: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-03-24T02:12:50.953 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:12:50 smithi189 ceph-mon[36187]: pgmap v4081: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-03-24T02:12:50.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:12:50 smithi193 ceph-mon[36878]: pgmap v4081: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-03-24T02:12:51.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:12:51 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 6466.707668 secs 2024-03-24T02:12:51.953 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:12:51 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 6466.707668 secs 2024-03-24T02:12:51.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:12:51 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 6466.707668 secs 2024-03-24T02:12:52.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:12:52 smithi118 ceph-mon[31535]: pgmap v4082: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:12:52.953 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:12:52 smithi189 ceph-mon[36187]: pgmap v4082: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:12:52.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:12:52 smithi193 ceph-mon[36878]: pgmap v4082: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:12:54.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:12:54 smithi118 ceph-mon[31535]: pgmap v4083: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T02:12:54.953 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:12:54 smithi189 ceph-mon[36187]: pgmap v4083: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T02:12:54.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:12:54 smithi193 ceph-mon[36878]: pgmap v4083: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T02:12:56.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:12:56 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 6471.707724 secs 2024-03-24T02:12:56.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:12:56 smithi118 ceph-mon[31535]: pgmap v4084: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T02:12:56.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:12:56 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 6471.707724 secs 2024-03-24T02:12:56.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:12:56 smithi189 ceph-mon[36187]: pgmap v4084: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T02:12:56.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:12:56 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 6471.707724 secs 2024-03-24T02:12:56.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:12:56 smithi193 ceph-mon[36878]: pgmap v4084: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T02:12:58.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:12:58 smithi189 ceph-mon[36187]: pgmap v4085: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T02:12:58.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:12:58 smithi193 ceph-mon[36878]: pgmap v4085: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T02:12:59.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:12:58 smithi118 ceph-mon[31535]: pgmap v4085: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T02:13:00.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:13:00 smithi189 ceph-mon[36187]: pgmap v4086: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-03-24T02:13:00.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:13:00 smithi193 ceph-mon[36878]: pgmap v4086: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-03-24T02:13:01.080 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:13:00 smithi118 ceph-mon[31535]: pgmap v4086: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-03-24T02:13:02.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:13:01 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 6476.707815 secs 2024-03-24T02:13:02.203 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:13:01 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 6476.707815 secs 2024-03-24T02:13:02.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:13:01 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 6476.707815 secs 2024-03-24T02:13:03.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:13:02 smithi118 ceph-mon[31535]: pgmap v4087: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:13:03.203 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:13:02 smithi189 ceph-mon[36187]: pgmap v4087: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:13:03.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:13:02 smithi193 ceph-mon[36878]: pgmap v4087: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:13:05.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:13:04 smithi118 ceph-mon[31535]: pgmap v4088: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T02:13:05.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:13:04 smithi189 ceph-mon[36187]: pgmap v4088: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T02:13:05.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:13:04 smithi193 ceph-mon[36878]: pgmap v4088: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T02:13:07.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:13:06 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 6481.707927 secs 2024-03-24T02:13:07.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:13:06 smithi118 ceph-mon[31535]: pgmap v4089: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T02:13:07.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:13:06 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T02:13:07.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:13:06 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 6481.707927 secs 2024-03-24T02:13:07.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:13:06 smithi189 ceph-mon[36187]: pgmap v4089: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T02:13:07.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:13:06 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T02:13:07.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:13:06 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 6481.707927 secs 2024-03-24T02:13:07.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:13:06 smithi193 ceph-mon[36878]: pgmap v4089: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T02:13:07.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:13:06 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T02:13:08.913 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:13:08 smithi189 ceph-mon[36187]: pgmap v4090: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T02:13:08.984 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:13:08 smithi193 ceph-mon[36878]: pgmap v4090: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T02:13:09.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:13:08 smithi118 ceph-mon[31535]: pgmap v4090: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T02:13:10.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:13:10 smithi193 ceph-mon[36878]: pgmap v4091: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-03-24T02:13:10.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:13:10 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T02:13:10.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:13:10 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T02:13:10.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:13:10 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T02:13:10.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:13:10 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T02:13:10.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:13:10 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T02:13:10.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:13:10 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T02:13:10.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:13:10 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T02:13:11.007 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:13:10 smithi189 ceph-mon[36187]: pgmap v4091: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-03-24T02:13:11.007 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:13:10 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T02:13:11.007 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:13:10 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T02:13:11.007 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:13:10 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T02:13:11.007 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:13:10 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T02:13:11.007 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:13:10 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T02:13:11.007 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:13:10 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T02:13:11.007 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:13:10 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T02:13:11.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:13:10 smithi118 ceph-mon[31535]: pgmap v4091: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-03-24T02:13:11.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:13:10 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T02:13:11.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:13:10 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T02:13:11.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:13:10 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T02:13:11.197 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:13:10 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T02:13:11.197 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:13:10 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T02:13:11.197 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:13:10 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T02:13:11.197 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:13:10 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T02:13:12.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:13:11 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 6486.707997 secs 2024-03-24T02:13:12.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:13:11 smithi118 ceph-mon[31535]: pgmap v4092: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 5 op/s 2024-03-24T02:13:12.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:13:11 smithi118 ceph-mon[31535]: pgmap v4093: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T02:13:12.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:13:11 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 6486.707997 secs 2024-03-24T02:13:12.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:13:11 smithi189 ceph-mon[36187]: pgmap v4092: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 5 op/s 2024-03-24T02:13:12.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:13:11 smithi189 ceph-mon[36187]: pgmap v4093: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T02:13:12.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:13:11 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 6486.707997 secs 2024-03-24T02:13:12.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:13:11 smithi193 ceph-mon[36878]: pgmap v4092: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 5 op/s 2024-03-24T02:13:12.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:13:11 smithi193 ceph-mon[36878]: pgmap v4093: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T02:13:14.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:13:13 smithi118 ceph-mon[31535]: pgmap v4094: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 5 op/s 2024-03-24T02:13:14.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:13:13 smithi189 ceph-mon[36187]: pgmap v4094: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 5 op/s 2024-03-24T02:13:14.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:13:13 smithi193 ceph-mon[36878]: pgmap v4094: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 5 op/s 2024-03-24T02:13:15.999 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:13:15 smithi118 ceph-mon[31535]: pgmap v4095: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T02:13:16.203 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:13:15 smithi189 ceph-mon[36187]: pgmap v4095: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T02:13:16.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:13:15 smithi193 ceph-mon[36878]: pgmap v4095: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T02:13:16.777 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T02:13:16.776+0000 7f81ee000640 1 -- 172.21.15.118:0/1693767697 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f81c8006590 con 0x7f81f81052d0 2024-03-24T02:13:16.777 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T02:13:16.776+0000 7f81db400640 1 -- 172.21.15.118:0/659686424 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f81c4006290 con 0x5622651a13f0 2024-03-24T02:13:16.778 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T02:13:16.776+0000 7f81e4c00640 1 -- 172.21.15.118:0/659686424 <== mon.1 v2:172.21.15.193:3300/0 54 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 409+0+0 (secure 0 0 0) 0x7f81c4006290 con 0x5622651a13f0 2024-03-24T02:13:16.778 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T02:13:16.776+0000 7f81ef400640 1 -- 172.21.15.118:0/1693767697 <== mon.2 v2:172.21.15.189:3300/0 54 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (secure 0 0 0) 0x7f81c8006590 con 0x7f81f81052d0 2024-03-24T02:13:17.027 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:13:16 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 6491.708059 secs 2024-03-24T02:13:17.203 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:13:16 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 6491.708059 secs 2024-03-24T02:13:17.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:13:16 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 6491.708059 secs 2024-03-24T02:13:17.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:13:17 smithi118 ceph-mon[31535]: pgmap v4096: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 6 op/s 2024-03-24T02:13:18.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:13:17 smithi189 ceph-mon[36187]: pgmap v4096: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 6 op/s 2024-03-24T02:13:18.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:13:17 smithi193 ceph-mon[36878]: pgmap v4096: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 6 op/s 2024-03-24T02:13:20.024 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:13:19 smithi118 ceph-mon[31535]: pgmap v4097: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 5 op/s 2024-03-24T02:13:20.203 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:13:19 smithi189 ceph-mon[36187]: pgmap v4097: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 5 op/s 2024-03-24T02:13:20.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:13:19 smithi193 ceph-mon[36878]: pgmap v4097: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 5 op/s 2024-03-24T02:13:22.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:13:21 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 6496.708115 secs 2024-03-24T02:13:22.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:13:21 smithi118 ceph-mon[31535]: pgmap v4098: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 5 op/s 2024-03-24T02:13:22.203 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:13:21 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 6496.708115 secs 2024-03-24T02:13:22.203 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:13:21 smithi189 ceph-mon[36187]: pgmap v4098: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 5 op/s 2024-03-24T02:13:22.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:13:21 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 6496.708115 secs 2024-03-24T02:13:22.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:13:21 smithi193 ceph-mon[36878]: pgmap v4098: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 5 op/s 2024-03-24T02:13:24.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:13:23 smithi118 ceph-mon[31535]: pgmap v4099: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T02:13:24.203 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:13:23 smithi189 ceph-mon[36187]: pgmap v4099: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T02:13:24.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:13:23 smithi193 ceph-mon[36878]: pgmap v4099: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T02:13:26.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:13:25 smithi118 ceph-mon[31535]: pgmap v4100: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-03-24T02:13:26.203 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:13:25 smithi189 ceph-mon[36187]: pgmap v4100: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-03-24T02:13:26.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:13:25 smithi193 ceph-mon[36878]: pgmap v4100: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-03-24T02:13:27.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:13:26 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 6501.708198 secs 2024-03-24T02:13:27.203 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:13:26 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 6501.708198 secs 2024-03-24T02:13:27.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:13:26 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 6501.708198 secs 2024-03-24T02:13:28.042 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:13:27 smithi118 ceph-mon[31535]: pgmap v4101: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:13:28.203 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:13:27 smithi189 ceph-mon[36187]: pgmap v4101: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:13:28.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:13:27 smithi193 ceph-mon[36878]: pgmap v4101: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:13:30.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:13:29 smithi118 ceph-mon[31535]: pgmap v4102: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-03-24T02:13:30.203 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:13:29 smithi189 ceph-mon[36187]: pgmap v4102: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-03-24T02:13:30.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:13:29 smithi193 ceph-mon[36878]: pgmap v4102: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-03-24T02:13:32.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:13:31 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 6506.708283 secs 2024-03-24T02:13:32.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:13:31 smithi118 ceph-mon[31535]: pgmap v4103: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 6 op/s 2024-03-24T02:13:32.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:13:31 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 6506.708283 secs 2024-03-24T02:13:32.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:13:31 smithi189 ceph-mon[36187]: pgmap v4103: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 6 op/s 2024-03-24T02:13:32.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:13:31 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 6506.708283 secs 2024-03-24T02:13:32.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:13:31 smithi193 ceph-mon[36878]: pgmap v4103: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 6 op/s 2024-03-24T02:13:34.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:13:33 smithi118 ceph-mon[31535]: pgmap v4104: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 5 op/s 2024-03-24T02:13:34.203 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:13:33 smithi189 ceph-mon[36187]: pgmap v4104: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 5 op/s 2024-03-24T02:13:34.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:13:33 smithi193 ceph-mon[36878]: pgmap v4104: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 5 op/s 2024-03-24T02:13:36.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:13:35 smithi118 ceph-mon[31535]: pgmap v4105: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-03-24T02:13:36.203 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:13:35 smithi189 ceph-mon[36187]: pgmap v4105: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-03-24T02:13:36.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:13:35 smithi193 ceph-mon[36878]: pgmap v4105: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-03-24T02:13:37.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:13:36 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 6511.708298 secs 2024-03-24T02:13:37.203 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:13:36 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 6511.708298 secs 2024-03-24T02:13:37.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:13:36 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 6511.708298 secs 2024-03-24T02:13:38.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:13:37 smithi118 ceph-mon[31535]: pgmap v4106: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-03-24T02:13:38.203 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:13:37 smithi189 ceph-mon[36187]: pgmap v4106: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-03-24T02:13:38.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:13:37 smithi193 ceph-mon[36878]: pgmap v4106: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-03-24T02:13:40.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:13:39 smithi118 ceph-mon[31535]: pgmap v4107: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-03-24T02:13:40.203 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:13:39 smithi189 ceph-mon[36187]: pgmap v4107: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-03-24T02:13:40.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:13:39 smithi193 ceph-mon[36878]: pgmap v4107: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-03-24T02:13:41.105 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:13:40 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 6516.708401 secs 2024-03-24T02:13:41.112 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:13:40 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 6516.708401 secs 2024-03-24T02:13:41.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:13:40 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 6516.708401 secs 2024-03-24T02:13:42.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:13:41 smithi118 ceph-mon[31535]: pgmap v4108: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T02:13:42.205 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:13:41 smithi189 ceph-mon[36187]: pgmap v4108: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T02:13:42.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:13:41 smithi193 ceph-mon[36878]: pgmap v4108: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T02:13:44.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:13:43 smithi118 ceph-mon[31535]: pgmap v4109: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T02:13:44.203 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:13:43 smithi189 ceph-mon[36187]: pgmap v4109: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T02:13:44.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:13:43 smithi193 ceph-mon[36878]: pgmap v4109: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T02:13:46.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:13:45 smithi118 ceph-mon[31535]: pgmap v4110: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-03-24T02:13:46.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:13:45 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 6521.708462 secs 2024-03-24T02:13:46.203 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:13:45 smithi189 ceph-mon[36187]: pgmap v4110: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-03-24T02:13:46.203 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:13:45 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 6521.708462 secs 2024-03-24T02:13:46.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:13:45 smithi193 ceph-mon[36878]: pgmap v4110: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-03-24T02:13:46.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:13:45 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 6521.708462 secs 2024-03-24T02:13:48.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:13:47 smithi118 ceph-mon[31535]: pgmap v4111: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:13:48.203 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:13:47 smithi189 ceph-mon[36187]: pgmap v4111: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:13:48.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:13:47 smithi193 ceph-mon[36878]: pgmap v4111: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:13:50.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:13:49 smithi118 ceph-mon[31535]: pgmap v4112: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T02:13:50.203 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:13:49 smithi189 ceph-mon[36187]: pgmap v4112: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T02:13:50.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:13:49 smithi193 ceph-mon[36878]: pgmap v4112: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T02:13:51.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:13:50 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 6526.708503 secs 2024-03-24T02:13:51.203 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:13:50 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 6526.708503 secs 2024-03-24T02:13:51.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:13:50 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 6526.708503 secs 2024-03-24T02:13:52.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:13:51 smithi118 ceph-mon[31535]: pgmap v4113: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T02:13:52.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:13:51 smithi189 ceph-mon[36187]: pgmap v4113: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T02:13:52.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:13:51 smithi193 ceph-mon[36878]: pgmap v4113: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T02:13:54.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:13:53 smithi118 ceph-mon[31535]: pgmap v4114: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T02:13:54.203 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:13:53 smithi189 ceph-mon[36187]: pgmap v4114: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T02:13:54.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:13:53 smithi193 ceph-mon[36878]: pgmap v4114: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T02:13:56.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:13:55 smithi118 ceph-mon[31535]: pgmap v4115: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-03-24T02:13:56.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:13:55 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 6531.708561 secs 2024-03-24T02:13:56.203 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:13:55 smithi189 ceph-mon[36187]: pgmap v4115: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-03-24T02:13:56.203 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:13:55 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 6531.708561 secs 2024-03-24T02:13:56.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:13:55 smithi193 ceph-mon[36878]: pgmap v4115: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-03-24T02:13:56.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:13:55 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 6531.708561 secs 2024-03-24T02:13:58.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:13:57 smithi118 ceph-mon[31535]: pgmap v4116: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:13:58.203 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:13:57 smithi189 ceph-mon[36187]: pgmap v4116: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:13:58.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:13:57 smithi193 ceph-mon[36878]: pgmap v4116: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:14:00.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:13:59 smithi118 ceph-mon[31535]: pgmap v4117: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T02:14:00.203 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:13:59 smithi189 ceph-mon[36187]: pgmap v4117: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T02:14:00.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:13:59 smithi193 ceph-mon[36878]: pgmap v4117: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T02:14:01.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:14:00 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 6536.708647 secs 2024-03-24T02:14:01.203 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:14:00 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 6536.708647 secs 2024-03-24T02:14:01.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:14:00 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 6536.708647 secs 2024-03-24T02:14:02.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:14:01 smithi118 ceph-mon[31535]: pgmap v4118: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T02:14:02.203 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:14:01 smithi189 ceph-mon[36187]: pgmap v4118: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T02:14:02.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:14:01 smithi193 ceph-mon[36878]: pgmap v4118: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T02:14:04.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:14:03 smithi118 ceph-mon[31535]: pgmap v4119: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T02:14:04.203 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:14:03 smithi189 ceph-mon[36187]: pgmap v4119: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T02:14:04.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:14:03 smithi193 ceph-mon[36878]: pgmap v4119: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T02:14:06.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:14:05 smithi118 ceph-mon[31535]: pgmap v4120: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-03-24T02:14:06.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:14:05 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 6541.708688 secs 2024-03-24T02:14:06.203 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:14:05 smithi189 ceph-mon[36187]: pgmap v4120: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-03-24T02:14:06.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:14:05 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 6541.708688 secs 2024-03-24T02:14:06.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:14:05 smithi193 ceph-mon[36878]: pgmap v4120: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-03-24T02:14:06.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:14:05 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 6541.708688 secs 2024-03-24T02:14:08.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:14:07 smithi118 ceph-mon[31535]: pgmap v4121: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:14:08.203 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:14:07 smithi189 ceph-mon[36187]: pgmap v4121: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:14:08.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:14:07 smithi193 ceph-mon[36878]: pgmap v4121: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:14:10.197 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:14:09 smithi118 ceph-mon[31535]: pgmap v4122: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T02:14:10.203 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:14:09 smithi189 ceph-mon[36187]: pgmap v4122: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T02:14:10.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:14:09 smithi193 ceph-mon[36878]: pgmap v4122: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T02:14:11.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:14:10 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 6546.708772 secs 2024-03-24T02:14:11.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:14:10 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T02:14:11.203 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:14:10 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 6546.708772 secs 2024-03-24T02:14:11.203 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:14:10 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T02:14:11.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:14:10 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 6546.708772 secs 2024-03-24T02:14:11.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:14:10 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T02:14:12.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:14:11 smithi118 ceph-mon[31535]: pgmap v4123: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T02:14:12.203 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:14:11 smithi189 ceph-mon[36187]: pgmap v4123: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T02:14:12.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:14:11 smithi193 ceph-mon[36878]: pgmap v4123: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T02:14:13.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:14:12 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T02:14:13.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:14:12 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T02:14:13.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:14:12 smithi118 ceph-mon[31535]: pgmap v4124: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 5 op/s 2024-03-24T02:14:13.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:14:12 smithi118 ceph-mon[31535]: pgmap v4125: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 5 op/s 2024-03-24T02:14:13.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:14:12 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T02:14:13.203 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:14:12 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T02:14:13.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:14:12 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T02:14:13.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:14:12 smithi189 ceph-mon[36187]: pgmap v4124: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 5 op/s 2024-03-24T02:14:13.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:14:12 smithi189 ceph-mon[36187]: pgmap v4125: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 5 op/s 2024-03-24T02:14:13.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:14:12 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T02:14:13.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:14:12 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T02:14:13.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:14:12 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T02:14:13.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:14:12 smithi193 ceph-mon[36878]: pgmap v4124: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 5 op/s 2024-03-24T02:14:13.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:14:12 smithi193 ceph-mon[36878]: pgmap v4125: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 5 op/s 2024-03-24T02:14:13.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:14:12 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T02:14:15.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:14:15 smithi118 ceph-mon[31535]: pgmap v4126: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-03-24T02:14:15.453 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:14:15 smithi189 ceph-mon[36187]: pgmap v4126: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-03-24T02:14:15.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:14:15 smithi193 ceph-mon[36878]: pgmap v4126: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-03-24T02:14:16.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:14:16 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 6551.708812 secs 2024-03-24T02:14:16.453 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:14:16 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 6551.708812 secs 2024-03-24T02:14:16.460 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:14:16 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 6551.708812 secs 2024-03-24T02:14:17.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:14:17 smithi118 ceph-mon[31535]: pgmap v4127: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-03-24T02:14:17.453 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:14:17 smithi189 ceph-mon[36187]: pgmap v4127: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-03-24T02:14:17.460 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:14:17 smithi193 ceph-mon[36878]: pgmap v4127: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-03-24T02:14:19.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:14:19 smithi118 ceph-mon[31535]: pgmap v4128: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-03-24T02:14:19.453 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:14:19 smithi189 ceph-mon[36187]: pgmap v4128: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-03-24T02:14:19.460 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:14:19 smithi193 ceph-mon[36878]: pgmap v4128: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-03-24T02:14:21.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:14:21 smithi118 ceph-mon[31535]: pgmap v4129: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-03-24T02:14:21.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:14:21 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 6556.708866 secs 2024-03-24T02:14:21.453 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:14:21 smithi189 ceph-mon[36187]: pgmap v4129: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-03-24T02:14:21.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:14:21 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 6556.708866 secs 2024-03-24T02:14:21.460 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:14:21 smithi193 ceph-mon[36878]: pgmap v4129: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-03-24T02:14:21.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:14:21 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 6556.708866 secs 2024-03-24T02:14:23.453 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:14:23 smithi189 ceph-mon[36187]: pgmap v4130: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 5 op/s 2024-03-24T02:14:23.460 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:14:23 smithi193 ceph-mon[36878]: pgmap v4130: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 5 op/s 2024-03-24T02:14:23.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:14:23 smithi118 ceph-mon[31535]: pgmap v4130: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 5 op/s 2024-03-24T02:14:25.453 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:14:25 smithi189 ceph-mon[36187]: pgmap v4131: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-24T02:14:25.460 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:14:25 smithi193 ceph-mon[36878]: pgmap v4131: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-24T02:14:25.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:14:25 smithi118 ceph-mon[31535]: pgmap v4131: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-24T02:14:26.460 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:14:26 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 6561.708952 secs 2024-03-24T02:14:26.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:14:26 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 6561.708952 secs 2024-03-24T02:14:26.703 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:14:26 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 6561.708952 secs 2024-03-24T02:14:27.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:14:27 smithi118 ceph-mon[31535]: pgmap v4132: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:14:27.703 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:14:27 smithi189 ceph-mon[36187]: pgmap v4132: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:14:27.710 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:14:27 smithi193 ceph-mon[36878]: pgmap v4132: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:14:29.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:14:29 smithi118 ceph-mon[31535]: pgmap v4133: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T02:14:29.703 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:14:29 smithi189 ceph-mon[36187]: pgmap v4133: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T02:14:29.710 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:14:29 smithi193 ceph-mon[36878]: pgmap v4133: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T02:14:31.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:14:31 smithi118 ceph-mon[31535]: pgmap v4134: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T02:14:31.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:14:31 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 6566.709013 secs 2024-03-24T02:14:31.704 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:14:31 smithi189 ceph-mon[36187]: pgmap v4134: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T02:14:31.704 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:14:31 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 6566.709013 secs 2024-03-24T02:14:31.710 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:14:31 smithi193 ceph-mon[36878]: pgmap v4134: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T02:14:31.710 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:14:31 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 6566.709013 secs 2024-03-24T02:14:33.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:14:33 smithi118 ceph-mon[31535]: pgmap v4135: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:14:33.703 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:14:33 smithi189 ceph-mon[36187]: pgmap v4135: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:14:33.711 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:14:33 smithi193 ceph-mon[36878]: pgmap v4135: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:14:35.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:14:35 smithi118 ceph-mon[31535]: pgmap v4136: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-24T02:14:35.703 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:14:35 smithi189 ceph-mon[36187]: pgmap v4136: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-24T02:14:35.710 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:14:35 smithi193 ceph-mon[36878]: pgmap v4136: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-24T02:14:36.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:14:36 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 6571.709070 secs 2024-03-24T02:14:36.703 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:14:36 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 6571.709070 secs 2024-03-24T02:14:36.710 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:14:36 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 6571.709070 secs 2024-03-24T02:14:37.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:14:37 smithi118 ceph-mon[31535]: pgmap v4137: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:14:37.703 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:14:37 smithi189 ceph-mon[36187]: pgmap v4137: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:14:37.710 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:14:37 smithi193 ceph-mon[36878]: pgmap v4137: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:14:39.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:14:39 smithi118 ceph-mon[31535]: pgmap v4138: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T02:14:39.703 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:14:39 smithi189 ceph-mon[36187]: pgmap v4138: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T02:14:39.710 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:14:39 smithi193 ceph-mon[36878]: pgmap v4138: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T02:14:41.504 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:14:41 smithi189 ceph-mon[36187]: pgmap v4139: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T02:14:41.505 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:14:41 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 6576.709132 secs 2024-03-24T02:14:41.512 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:14:41 smithi193 ceph-mon[36878]: pgmap v4139: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T02:14:41.513 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:14:41 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 6576.709132 secs 2024-03-24T02:14:41.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:14:41 smithi118 ceph-mon[31535]: pgmap v4139: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T02:14:41.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:14:41 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 6576.709132 secs 2024-03-24T02:14:43.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:14:43 smithi118 ceph-mon[31535]: pgmap v4140: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:14:43.704 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:14:43 smithi189 ceph-mon[36187]: pgmap v4140: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:14:43.710 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:14:43 smithi193 ceph-mon[36878]: pgmap v4140: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:14:45.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:14:45 smithi118 ceph-mon[31535]: pgmap v4141: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-24T02:14:45.703 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:14:45 smithi189 ceph-mon[36187]: pgmap v4141: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-24T02:14:45.710 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:14:45 smithi193 ceph-mon[36878]: pgmap v4141: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-24T02:14:46.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:14:46 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 6581.709193 secs 2024-03-24T02:14:46.703 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:14:46 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 6581.709193 secs 2024-03-24T02:14:46.710 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:14:46 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 6581.709193 secs 2024-03-24T02:14:47.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:14:47 smithi118 ceph-mon[31535]: pgmap v4142: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:14:47.703 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:14:47 smithi189 ceph-mon[36187]: pgmap v4142: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:14:47.710 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:14:47 smithi193 ceph-mon[36878]: pgmap v4142: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:14:48.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:14:48 smithi118 ceph-mon[31535]: pgmap v4143: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T02:14:48.703 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:14:48 smithi189 ceph-mon[36187]: pgmap v4143: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T02:14:48.710 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:14:48 smithi193 ceph-mon[36878]: pgmap v4143: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T02:14:51.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:14:51 smithi118 ceph-mon[31535]: pgmap v4144: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T02:14:51.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:14:51 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 6586.709250 secs 2024-03-24T02:14:51.453 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:14:51 smithi189 ceph-mon[36187]: pgmap v4144: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T02:14:51.453 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:14:51 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 6586.709250 secs 2024-03-24T02:14:51.460 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:14:51 smithi193 ceph-mon[36878]: pgmap v4144: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T02:14:51.460 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:14:51 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 6586.709250 secs 2024-03-24T02:14:53.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:14:53 smithi118 ceph-mon[31535]: pgmap v4145: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:14:53.453 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:14:53 smithi189 ceph-mon[36187]: pgmap v4145: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:14:53.460 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:14:53 smithi193 ceph-mon[36878]: pgmap v4145: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:14:55.453 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:14:55 smithi189 ceph-mon[36187]: pgmap v4146: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-24T02:14:55.460 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:14:55 smithi193 ceph-mon[36878]: pgmap v4146: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-24T02:14:55.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:14:55 smithi118 ceph-mon[31535]: pgmap v4146: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-24T02:14:56.460 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:14:56 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 6591.709312 secs 2024-03-24T02:14:56.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:14:56 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 6591.709312 secs 2024-03-24T02:14:56.703 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:14:56 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 6591.709312 secs 2024-03-24T02:14:57.460 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:14:57 smithi193 ceph-mon[36878]: pgmap v4147: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:14:57.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:14:57 smithi118 ceph-mon[31535]: pgmap v4147: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:14:57.703 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:14:57 smithi189 ceph-mon[36187]: pgmap v4147: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:14:59.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:14:59 smithi118 ceph-mon[31535]: pgmap v4148: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T02:14:59.703 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:14:59 smithi189 ceph-mon[36187]: pgmap v4148: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T02:14:59.710 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:14:59 smithi193 ceph-mon[36878]: pgmap v4148: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T02:15:01.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:15:01 smithi118 ceph-mon[31535]: pgmap v4149: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T02:15:01.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:15:01 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 6596.709374 secs 2024-03-24T02:15:01.703 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:15:01 smithi189 ceph-mon[36187]: pgmap v4149: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T02:15:01.703 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:15:01 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 6596.709374 secs 2024-03-24T02:15:01.710 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:15:01 smithi193 ceph-mon[36878]: pgmap v4149: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T02:15:01.710 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:15:01 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 6596.709374 secs 2024-03-24T02:15:03.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:15:03 smithi118 ceph-mon[31535]: pgmap v4150: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:15:03.703 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:15:03 smithi189 ceph-mon[36187]: pgmap v4150: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:15:03.710 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:15:03 smithi193 ceph-mon[36878]: pgmap v4150: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:15:05.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:15:05 smithi118 ceph-mon[31535]: pgmap v4151: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-24T02:15:05.703 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:15:05 smithi189 ceph-mon[36187]: pgmap v4151: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-24T02:15:05.710 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:15:05 smithi193 ceph-mon[36878]: pgmap v4151: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-24T02:15:06.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:15:06 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 6601.709461 secs 2024-03-24T02:15:06.703 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:15:06 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 6601.709461 secs 2024-03-24T02:15:06.710 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:15:06 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 6601.709461 secs 2024-03-24T02:15:07.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:15:07 smithi118 ceph-mon[31535]: pgmap v4152: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-03-24T02:15:07.703 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:15:07 smithi189 ceph-mon[36187]: pgmap v4152: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-03-24T02:15:07.710 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:15:07 smithi193 ceph-mon[36878]: pgmap v4152: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-03-24T02:15:09.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:15:09 smithi118 ceph-mon[31535]: pgmap v4153: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-03-24T02:15:09.704 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:15:09 smithi189 ceph-mon[36187]: pgmap v4153: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-03-24T02:15:09.710 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:15:09 smithi193 ceph-mon[36878]: pgmap v4153: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-03-24T02:15:11.506 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:15:11 smithi189 ceph-mon[36187]: pgmap v4154: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 6 op/s 2024-03-24T02:15:11.506 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:15:11 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 6606.709550 secs 2024-03-24T02:15:11.513 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:15:11 smithi193 ceph-mon[36878]: pgmap v4154: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 6 op/s 2024-03-24T02:15:11.513 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:15:11 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 6606.709550 secs 2024-03-24T02:15:11.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:15:11 smithi118 ceph-mon[31535]: pgmap v4154: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 6 op/s 2024-03-24T02:15:11.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:15:11 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 6606.709550 secs 2024-03-24T02:15:13.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:15:13 smithi118 ceph-mon[31535]: pgmap v4155: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-03-24T02:15:13.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:15:13 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T02:15:13.703 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:15:13 smithi189 ceph-mon[36187]: pgmap v4155: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-03-24T02:15:13.703 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:15:13 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T02:15:13.710 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:15:13 smithi193 ceph-mon[36878]: pgmap v4155: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-03-24T02:15:13.710 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:15:13 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T02:15:14.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:15:14 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T02:15:14.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:15:14 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T02:15:14.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:15:14 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T02:15:14.703 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:15:14 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T02:15:14.703 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:15:14 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T02:15:14.703 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:15:14 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T02:15:14.710 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:15:14 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T02:15:14.710 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:15:14 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T02:15:14.710 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:15:14 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T02:15:15.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:15:15 smithi118 ceph-mon[31535]: pgmap v4156: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 6 op/s 2024-03-24T02:15:15.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:15:15 smithi118 ceph-mon[31535]: pgmap v4157: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 4 op/s 2024-03-24T02:15:15.703 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:15:15 smithi189 ceph-mon[36187]: pgmap v4156: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 6 op/s 2024-03-24T02:15:15.703 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:15:15 smithi189 ceph-mon[36187]: pgmap v4157: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 4 op/s 2024-03-24T02:15:15.710 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:15:15 smithi193 ceph-mon[36878]: pgmap v4156: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 6 op/s 2024-03-24T02:15:15.710 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:15:15 smithi193 ceph-mon[36878]: pgmap v4157: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 4 op/s 2024-03-24T02:15:16.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:15:16 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 6611.709638 secs 2024-03-24T02:15:16.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:15:16 smithi118 ceph-mon[31535]: pgmap v4158: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 6 op/s 2024-03-24T02:15:16.703 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:15:16 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 6611.709638 secs 2024-03-24T02:15:16.703 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:15:16 smithi189 ceph-mon[36187]: pgmap v4158: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 6 op/s 2024-03-24T02:15:16.710 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:15:16 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 6611.709638 secs 2024-03-24T02:15:16.710 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:15:16 smithi193 ceph-mon[36878]: pgmap v4158: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 6 op/s 2024-03-24T02:15:19.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:15:18 smithi118 ceph-mon[31535]: pgmap v4159: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 5 op/s 2024-03-24T02:15:19.203 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:15:18 smithi189 ceph-mon[36187]: pgmap v4159: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 5 op/s 2024-03-24T02:15:19.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:15:18 smithi193 ceph-mon[36878]: pgmap v4159: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 5 op/s 2024-03-24T02:15:21.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:15:20 smithi118 ceph-mon[31535]: pgmap v4160: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 6 op/s 2024-03-24T02:15:21.203 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:15:20 smithi189 ceph-mon[36187]: pgmap v4160: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 6 op/s 2024-03-24T02:15:21.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:15:20 smithi193 ceph-mon[36878]: pgmap v4160: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 6 op/s 2024-03-24T02:15:22.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:15:21 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 6616.709727 secs 2024-03-24T02:15:22.203 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:15:21 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 6616.709727 secs 2024-03-24T02:15:22.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:15:21 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 6616.709727 secs 2024-03-24T02:15:23.098 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:15:22 smithi118 ceph-mon[31535]: pgmap v4161: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 6 op/s 2024-03-24T02:15:23.203 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:15:22 smithi189 ceph-mon[36187]: pgmap v4161: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 6 op/s 2024-03-24T02:15:23.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:15:22 smithi193 ceph-mon[36878]: pgmap v4161: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 6 op/s 2024-03-24T02:15:25.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:15:24 smithi118 ceph-mon[31535]: pgmap v4162: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 5 op/s 2024-03-24T02:15:25.203 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:15:24 smithi189 ceph-mon[36187]: pgmap v4162: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 5 op/s 2024-03-24T02:15:25.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:15:24 smithi193 ceph-mon[36878]: pgmap v4162: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 5 op/s 2024-03-24T02:15:27.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:15:26 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 6621.709815 secs 2024-03-24T02:15:27.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:15:26 smithi118 ceph-mon[31535]: pgmap v4163: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T02:15:27.203 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:15:26 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 6621.709815 secs 2024-03-24T02:15:27.203 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:15:26 smithi189 ceph-mon[36187]: pgmap v4163: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T02:15:27.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:15:26 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 6621.709815 secs 2024-03-24T02:15:27.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:15:26 smithi193 ceph-mon[36878]: pgmap v4163: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T02:15:29.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:15:28 smithi118 ceph-mon[31535]: pgmap v4164: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T02:15:29.203 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:15:28 smithi189 ceph-mon[36187]: pgmap v4164: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T02:15:29.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:15:28 smithi193 ceph-mon[36878]: pgmap v4164: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T02:15:31.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:15:30 smithi118 ceph-mon[31535]: pgmap v4165: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-03-24T02:15:31.203 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:15:30 smithi189 ceph-mon[36187]: pgmap v4165: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-03-24T02:15:31.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:15:30 smithi193 ceph-mon[36878]: pgmap v4165: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-03-24T02:15:32.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:15:31 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 6626.709905 secs 2024-03-24T02:15:32.203 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:15:31 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 6626.709905 secs 2024-03-24T02:15:32.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:15:31 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 6626.709905 secs 2024-03-24T02:15:33.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:15:32 smithi118 ceph-mon[31535]: pgmap v4166: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:15:33.203 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:15:32 smithi189 ceph-mon[36187]: pgmap v4166: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:15:33.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:15:32 smithi193 ceph-mon[36878]: pgmap v4166: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:15:35.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:15:34 smithi118 ceph-mon[31535]: pgmap v4167: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T02:15:35.203 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:15:34 smithi189 ceph-mon[36187]: pgmap v4167: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T02:15:35.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:15:34 smithi193 ceph-mon[36878]: pgmap v4167: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T02:15:37.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:15:36 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 6631.709994 secs 2024-03-24T02:15:37.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:15:36 smithi118 ceph-mon[31535]: pgmap v4168: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T02:15:37.203 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:15:36 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 6631.709994 secs 2024-03-24T02:15:37.203 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:15:36 smithi189 ceph-mon[36187]: pgmap v4168: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T02:15:37.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:15:36 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 6631.709994 secs 2024-03-24T02:15:37.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:15:36 smithi193 ceph-mon[36878]: pgmap v4168: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T02:15:39.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:15:38 smithi118 ceph-mon[31535]: pgmap v4169: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T02:15:39.203 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:15:38 smithi189 ceph-mon[36187]: pgmap v4169: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T02:15:39.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:15:38 smithi193 ceph-mon[36878]: pgmap v4169: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T02:15:41.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:15:40 smithi118 ceph-mon[31535]: pgmap v4170: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-03-24T02:15:41.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:15:40 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 6636.710082 secs 2024-03-24T02:15:41.203 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:15:40 smithi189 ceph-mon[36187]: pgmap v4170: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-03-24T02:15:41.203 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:15:40 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 6636.710082 secs 2024-03-24T02:15:41.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:15:40 smithi193 ceph-mon[36878]: pgmap v4170: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-03-24T02:15:41.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:15:40 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 6636.710082 secs 2024-03-24T02:15:43.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:15:42 smithi118 ceph-mon[31535]: pgmap v4171: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:15:43.203 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:15:42 smithi189 ceph-mon[36187]: pgmap v4171: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:15:43.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:15:42 smithi193 ceph-mon[36878]: pgmap v4171: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:15:45.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:15:44 smithi118 ceph-mon[31535]: pgmap v4172: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T02:15:45.203 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:15:44 smithi189 ceph-mon[36187]: pgmap v4172: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T02:15:45.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:15:44 smithi193 ceph-mon[36878]: pgmap v4172: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T02:15:46.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:15:45 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 6641.710166 secs 2024-03-24T02:15:46.203 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:15:45 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 6641.710166 secs 2024-03-24T02:15:46.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:15:45 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 6641.710166 secs 2024-03-24T02:15:47.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:15:46 smithi118 ceph-mon[31535]: pgmap v4173: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T02:15:47.203 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:15:46 smithi189 ceph-mon[36187]: pgmap v4173: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T02:15:47.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:15:46 smithi193 ceph-mon[36878]: pgmap v4173: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T02:15:49.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:15:48 smithi118 ceph-mon[31535]: pgmap v4174: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T02:15:49.203 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:15:48 smithi189 ceph-mon[36187]: pgmap v4174: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T02:15:49.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:15:48 smithi193 ceph-mon[36878]: pgmap v4174: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T02:15:51.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:15:50 smithi118 ceph-mon[31535]: pgmap v4175: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-03-24T02:15:51.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:15:50 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 6646.710250 secs 2024-03-24T02:15:51.203 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:15:50 smithi189 ceph-mon[36187]: pgmap v4175: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-03-24T02:15:51.203 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:15:50 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 6646.710250 secs 2024-03-24T02:15:51.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:15:50 smithi193 ceph-mon[36878]: pgmap v4175: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-03-24T02:15:51.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:15:50 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 6646.710250 secs 2024-03-24T02:15:53.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:15:52 smithi118 ceph-mon[31535]: pgmap v4176: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:15:53.203 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:15:52 smithi189 ceph-mon[36187]: pgmap v4176: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:15:53.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:15:52 smithi193 ceph-mon[36878]: pgmap v4176: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:15:55.160 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:15:54 smithi118 ceph-mon[31535]: pgmap v4177: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T02:15:55.203 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:15:54 smithi189 ceph-mon[36187]: pgmap v4177: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T02:15:55.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:15:54 smithi193 ceph-mon[36878]: pgmap v4177: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T02:15:56.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:15:55 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 6651.710310 secs 2024-03-24T02:15:56.203 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:15:55 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 6651.710310 secs 2024-03-24T02:15:56.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:15:55 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 6651.710310 secs 2024-03-24T02:15:57.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:15:56 smithi118 ceph-mon[31535]: pgmap v4178: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T02:15:57.203 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:15:56 smithi189 ceph-mon[36187]: pgmap v4178: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T02:15:57.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:15:56 smithi193 ceph-mon[36878]: pgmap v4178: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T02:15:59.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:15:58 smithi118 ceph-mon[31535]: pgmap v4179: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T02:15:59.203 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:15:58 smithi189 ceph-mon[36187]: pgmap v4179: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T02:15:59.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:15:58 smithi193 ceph-mon[36878]: pgmap v4179: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T02:16:01.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:16:00 smithi118 ceph-mon[31535]: pgmap v4180: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-03-24T02:16:01.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:16:00 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 6656.710326 secs 2024-03-24T02:16:01.203 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:16:00 smithi189 ceph-mon[36187]: pgmap v4180: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-03-24T02:16:01.203 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:16:00 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 6656.710326 secs 2024-03-24T02:16:01.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:16:00 smithi193 ceph-mon[36878]: pgmap v4180: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-03-24T02:16:01.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:16:00 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 6656.710326 secs 2024-03-24T02:16:03.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:16:02 smithi118 ceph-mon[31535]: pgmap v4181: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:16:03.203 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:16:02 smithi189 ceph-mon[36187]: pgmap v4181: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:16:03.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:16:02 smithi193 ceph-mon[36878]: pgmap v4181: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:16:05.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:16:04 smithi118 ceph-mon[31535]: pgmap v4182: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T02:16:05.203 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:16:04 smithi189 ceph-mon[36187]: pgmap v4182: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T02:16:05.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:16:04 smithi193 ceph-mon[36878]: pgmap v4182: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T02:16:06.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:16:06 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 6661.710409 secs 2024-03-24T02:16:06.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:16:06 smithi118 ceph-mon[31535]: pgmap v4183: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T02:16:06.953 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:16:06 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 6661.710409 secs 2024-03-24T02:16:06.953 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:16:06 smithi189 ceph-mon[36187]: pgmap v4183: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T02:16:06.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:16:06 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 6661.710409 secs 2024-03-24T02:16:06.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:16:06 smithi193 ceph-mon[36878]: pgmap v4183: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T02:16:09.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:16:08 smithi118 ceph-mon[31535]: pgmap v4184: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T02:16:09.203 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:16:08 smithi189 ceph-mon[36187]: pgmap v4184: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T02:16:09.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:16:08 smithi193 ceph-mon[36878]: pgmap v4184: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T02:16:11.041 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:16:10 smithi118 ceph-mon[31535]: pgmap v4185: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T02:16:11.203 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:16:10 smithi189 ceph-mon[36187]: pgmap v4185: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T02:16:11.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:16:10 smithi193 ceph-mon[36878]: pgmap v4185: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T02:16:12.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:16:11 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 6666.710490 secs 2024-03-24T02:16:12.203 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:16:11 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 6666.710490 secs 2024-03-24T02:16:12.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:16:11 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 6666.710490 secs 2024-03-24T02:16:13.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:16:12 smithi118 ceph-mon[31535]: pgmap v4186: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:16:13.203 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:16:12 smithi189 ceph-mon[36187]: pgmap v4186: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:16:13.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:16:12 smithi193 ceph-mon[36878]: pgmap v4186: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:16:15.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:16:14 smithi118 ceph-mon[31535]: pgmap v4187: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T02:16:15.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:16:14 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T02:16:15.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:16:14 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-24T02:16:15.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:16:14 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-24T02:16:15.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:16:14 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-24T02:16:15.197 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:16:14 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-03-24T02:16:15.203 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:16:14 smithi189 ceph-mon[36187]: pgmap v4187: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T02:16:15.203 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:16:14 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T02:16:15.203 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:16:14 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-24T02:16:15.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:16:14 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-24T02:16:15.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:16:14 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-24T02:16:15.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:16:14 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-03-24T02:16:15.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:16:14 smithi193 ceph-mon[36878]: pgmap v4187: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T02:16:15.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:16:14 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T02:16:15.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:16:14 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-24T02:16:15.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:16:14 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-24T02:16:15.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:16:14 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-24T02:16:15.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:16:14 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-03-24T02:16:16.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:16:15 smithi118 ceph-mon[31535]: Adjusting osd_memory_target on smithi189 to 174.1M 2024-03-24T02:16:16.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:16:15 smithi118 ceph-mon[31535]: Unable to set osd_memory_target on smithi189 to 182635622: error parsing value: Value '182635622' is below minimum 939524096 2024-03-24T02:16:16.197 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:16:15 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-24T02:16:16.197 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:16:15 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-24T02:16:16.197 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:16:15 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-24T02:16:16.197 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:16:15 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-03-24T02:16:16.197 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:16:15 smithi118 ceph-mon[31535]: Adjusting osd_memory_target on smithi118 to 176.7M 2024-03-24T02:16:16.197 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:16:15 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T02:16:16.197 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:16:15 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T02:16:16.197 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:16:15 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T02:16:16.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:16:15 smithi189 ceph-mon[36187]: Adjusting osd_memory_target on smithi189 to 174.1M 2024-03-24T02:16:16.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:16:15 smithi189 ceph-mon[36187]: Unable to set osd_memory_target on smithi189 to 182635622: error parsing value: Value '182635622' is below minimum 939524096 2024-03-24T02:16:16.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:16:15 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-24T02:16:16.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:16:15 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-24T02:16:16.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:16:15 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-24T02:16:16.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:16:15 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-03-24T02:16:16.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:16:15 smithi189 ceph-mon[36187]: Adjusting osd_memory_target on smithi118 to 176.7M 2024-03-24T02:16:16.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:16:15 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T02:16:16.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:16:15 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T02:16:16.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:16:15 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T02:16:16.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:16:15 smithi193 ceph-mon[36878]: Adjusting osd_memory_target on smithi189 to 174.1M 2024-03-24T02:16:16.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:16:15 smithi193 ceph-mon[36878]: Unable to set osd_memory_target on smithi189 to 182635622: error parsing value: Value '182635622' is below minimum 939524096 2024-03-24T02:16:16.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:16:15 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-24T02:16:16.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:16:15 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-24T02:16:16.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:16:15 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-24T02:16:16.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:16:15 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-03-24T02:16:16.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:16:15 smithi193 ceph-mon[36878]: Adjusting osd_memory_target on smithi118 to 176.7M 2024-03-24T02:16:16.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:16:15 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T02:16:16.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:16:15 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T02:16:16.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:16:15 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T02:16:16.806 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T02:16:16.806+0000 7f81ee000640 1 -- 172.21.15.118:0/1693767697 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f81c8006590 con 0x7f81f81052d0 2024-03-24T02:16:16.806 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T02:16:16.806+0000 7f81db400640 1 -- 172.21.15.118:0/659686424 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f81c4006290 con 0x5622651a13f0 2024-03-24T02:16:16.807 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T02:16:16.806+0000 7f81ef400640 1 -- 172.21.15.118:0/1693767697 <== mon.2 v2:172.21.15.189:3300/0 55 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (secure 0 0 0) 0x7f81c8006590 con 0x7f81f81052d0 2024-03-24T02:16:16.807 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T02:16:16.806+0000 7f81e4c00640 1 -- 172.21.15.118:0/659686424 <== mon.1 v2:172.21.15.193:3300/0 55 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 409+0+0 (secure 0 0 0) 0x7f81c4006290 con 0x5622651a13f0 2024-03-24T02:16:17.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:16:16 smithi118 ceph-mon[31535]: Unable to set osd_memory_target on smithi118 to 185365196: error parsing value: Value '185365196' is below minimum 939524096 2024-03-24T02:16:17.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:16:16 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 6671.710529 secs 2024-03-24T02:16:17.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:16:16 smithi118 ceph-mon[31535]: pgmap v4188: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 6 op/s 2024-03-24T02:16:17.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:16:16 smithi118 ceph-mon[31535]: pgmap v4189: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 6 op/s 2024-03-24T02:16:17.203 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:16:16 smithi189 ceph-mon[36187]: Unable to set osd_memory_target on smithi118 to 185365196: error parsing value: Value '185365196' is below minimum 939524096 2024-03-24T02:16:17.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:16:16 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 6671.710529 secs 2024-03-24T02:16:17.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:16:16 smithi189 ceph-mon[36187]: pgmap v4188: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 6 op/s 2024-03-24T02:16:17.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:16:16 smithi189 ceph-mon[36187]: pgmap v4189: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 6 op/s 2024-03-24T02:16:17.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:16:16 smithi193 ceph-mon[36878]: Unable to set osd_memory_target on smithi118 to 185365196: error parsing value: Value '185365196' is below minimum 939524096 2024-03-24T02:16:17.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:16:16 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 6671.710529 secs 2024-03-24T02:16:17.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:16:16 smithi193 ceph-mon[36878]: pgmap v4188: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 6 op/s 2024-03-24T02:16:17.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:16:16 smithi193 ceph-mon[36878]: pgmap v4189: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 6 op/s 2024-03-24T02:16:19.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:16:18 smithi118 ceph-mon[31535]: pgmap v4190: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 5 op/s 2024-03-24T02:16:19.203 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:16:18 smithi189 ceph-mon[36187]: pgmap v4190: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 5 op/s 2024-03-24T02:16:19.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:16:18 smithi193 ceph-mon[36878]: pgmap v4190: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 5 op/s 2024-03-24T02:16:21.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:16:20 smithi118 ceph-mon[31535]: pgmap v4191: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 6 op/s 2024-03-24T02:16:21.203 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:16:20 smithi189 ceph-mon[36187]: pgmap v4191: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 6 op/s 2024-03-24T02:16:21.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:16:20 smithi193 ceph-mon[36878]: pgmap v4191: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 6 op/s 2024-03-24T02:16:22.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:16:21 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 6676.710618 secs 2024-03-24T02:16:22.203 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:16:21 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 6676.710618 secs 2024-03-24T02:16:22.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:16:21 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 6676.710618 secs 2024-03-24T02:16:23.098 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:16:22 smithi118 ceph-mon[31535]: pgmap v4192: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 6 op/s 2024-03-24T02:16:23.203 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:16:22 smithi189 ceph-mon[36187]: pgmap v4192: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 6 op/s 2024-03-24T02:16:23.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:16:22 smithi193 ceph-mon[36878]: pgmap v4192: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 6 op/s 2024-03-24T02:16:25.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:16:24 smithi118 ceph-mon[31535]: pgmap v4193: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 6 op/s 2024-03-24T02:16:25.203 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:16:24 smithi189 ceph-mon[36187]: pgmap v4193: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 6 op/s 2024-03-24T02:16:25.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:16:24 smithi193 ceph-mon[36878]: pgmap v4193: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 6 op/s 2024-03-24T02:16:26.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:16:25 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 6681.710679 secs 2024-03-24T02:16:26.203 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:16:25 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 6681.710679 secs 2024-03-24T02:16:26.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:16:25 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 6681.710679 secs 2024-03-24T02:16:27.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:16:26 smithi118 ceph-mon[31535]: pgmap v4194: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 5 op/s 2024-03-24T02:16:27.203 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:16:26 smithi189 ceph-mon[36187]: pgmap v4194: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 5 op/s 2024-03-24T02:16:27.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:16:26 smithi193 ceph-mon[36878]: pgmap v4194: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 5 op/s 2024-03-24T02:16:29.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:16:28 smithi118 ceph-mon[31535]: pgmap v4195: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:16:29.203 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:16:28 smithi189 ceph-mon[36187]: pgmap v4195: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:16:29.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:16:28 smithi193 ceph-mon[36878]: pgmap v4195: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:16:31.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:16:30 smithi118 ceph-mon[31535]: pgmap v4196: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-24T02:16:31.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:16:30 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 6686.710767 secs 2024-03-24T02:16:31.203 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:16:30 smithi189 ceph-mon[36187]: pgmap v4196: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-24T02:16:31.203 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:16:30 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 6686.710767 secs 2024-03-24T02:16:31.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:16:30 smithi193 ceph-mon[36878]: pgmap v4196: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-24T02:16:31.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:16:30 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 6686.710767 secs 2024-03-24T02:16:33.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:16:32 smithi118 ceph-mon[31535]: pgmap v4197: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:16:33.203 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:16:32 smithi189 ceph-mon[36187]: pgmap v4197: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:16:33.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:16:32 smithi193 ceph-mon[36878]: pgmap v4197: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:16:35.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:16:34 smithi118 ceph-mon[31535]: pgmap v4198: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T02:16:35.203 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:16:34 smithi189 ceph-mon[36187]: pgmap v4198: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T02:16:35.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:16:34 smithi193 ceph-mon[36878]: pgmap v4198: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T02:16:36.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:16:35 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 6691.710858 secs 2024-03-24T02:16:36.203 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:16:35 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 6691.710858 secs 2024-03-24T02:16:36.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:16:35 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 6691.710858 secs 2024-03-24T02:16:37.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:16:36 smithi118 ceph-mon[31535]: pgmap v4199: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T02:16:37.203 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:16:36 smithi189 ceph-mon[36187]: pgmap v4199: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T02:16:37.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:16:36 smithi193 ceph-mon[36878]: pgmap v4199: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T02:16:39.192 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:16:38 smithi118 ceph-mon[31535]: pgmap v4200: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:16:39.203 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:16:38 smithi189 ceph-mon[36187]: pgmap v4200: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:16:39.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:16:38 smithi193 ceph-mon[36878]: pgmap v4200: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:16:41.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:16:40 smithi118 ceph-mon[31535]: pgmap v4201: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-24T02:16:41.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:16:40 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 6696.710944 secs 2024-03-24T02:16:41.203 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:16:40 smithi189 ceph-mon[36187]: pgmap v4201: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-24T02:16:41.203 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:16:40 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 6696.710944 secs 2024-03-24T02:16:41.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:16:40 smithi193 ceph-mon[36878]: pgmap v4201: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-24T02:16:41.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:16:40 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 6696.710944 secs 2024-03-24T02:16:43.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:16:42 smithi118 ceph-mon[31535]: pgmap v4202: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:16:43.203 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:16:42 smithi189 ceph-mon[36187]: pgmap v4202: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:16:43.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:16:42 smithi193 ceph-mon[36878]: pgmap v4202: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:16:45.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:16:44 smithi118 ceph-mon[31535]: pgmap v4203: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T02:16:45.203 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:16:44 smithi189 ceph-mon[36187]: pgmap v4203: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T02:16:45.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:16:44 smithi193 ceph-mon[36878]: pgmap v4203: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T02:16:46.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:16:45 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 6701.711033 secs 2024-03-24T02:16:46.203 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:16:45 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 6701.711033 secs 2024-03-24T02:16:46.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:16:45 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 6701.711033 secs 2024-03-24T02:16:47.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:16:46 smithi118 ceph-mon[31535]: pgmap v4204: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T02:16:47.203 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:16:46 smithi189 ceph-mon[36187]: pgmap v4204: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T02:16:47.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:16:46 smithi193 ceph-mon[36878]: pgmap v4204: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T02:16:49.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:16:48 smithi118 ceph-mon[31535]: pgmap v4205: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:16:49.205 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:16:48 smithi189 ceph-mon[36187]: pgmap v4205: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:16:49.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:16:48 smithi193 ceph-mon[36878]: pgmap v4205: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:16:51.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:16:50 smithi118 ceph-mon[31535]: pgmap v4206: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-24T02:16:51.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:16:50 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 6706.711093 secs 2024-03-24T02:16:51.203 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:16:50 smithi189 ceph-mon[36187]: pgmap v4206: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-24T02:16:51.203 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:16:50 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 6706.711093 secs 2024-03-24T02:16:51.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:16:50 smithi193 ceph-mon[36878]: pgmap v4206: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-24T02:16:51.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:16:50 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 6706.711093 secs 2024-03-24T02:16:53.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:16:52 smithi118 ceph-mon[31535]: pgmap v4207: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:16:53.203 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:16:52 smithi189 ceph-mon[36187]: pgmap v4207: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:16:53.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:16:52 smithi193 ceph-mon[36878]: pgmap v4207: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:16:55.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:16:54 smithi118 ceph-mon[31535]: pgmap v4208: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T02:16:55.203 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:16:54 smithi189 ceph-mon[36187]: pgmap v4208: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T02:16:55.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:16:54 smithi193 ceph-mon[36878]: pgmap v4208: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T02:16:56.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:16:55 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 6711.711178 secs 2024-03-24T02:16:56.203 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:16:55 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 6711.711178 secs 2024-03-24T02:16:56.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:16:55 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 6711.711178 secs 2024-03-24T02:16:57.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:16:56 smithi118 ceph-mon[31535]: pgmap v4209: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T02:16:57.203 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:16:56 smithi189 ceph-mon[36187]: pgmap v4209: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T02:16:57.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:16:56 smithi193 ceph-mon[36878]: pgmap v4209: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T02:16:59.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:16:58 smithi118 ceph-mon[31535]: pgmap v4210: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:16:59.203 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:16:58 smithi189 ceph-mon[36187]: pgmap v4210: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:16:59.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:16:58 smithi193 ceph-mon[36878]: pgmap v4210: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:17:01.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:17:00 smithi118 ceph-mon[31535]: pgmap v4211: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-24T02:17:01.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:17:00 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 6716.711260 secs 2024-03-24T02:17:01.203 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:17:00 smithi189 ceph-mon[36187]: pgmap v4211: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-24T02:17:01.203 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:17:00 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 6716.711260 secs 2024-03-24T02:17:01.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:17:00 smithi193 ceph-mon[36878]: pgmap v4211: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-24T02:17:01.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:17:00 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 6716.711260 secs 2024-03-24T02:17:03.203 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:17:02 smithi189 ceph-mon[36187]: pgmap v4212: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:17:03.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:17:02 smithi193 ceph-mon[36878]: pgmap v4212: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:17:03.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:17:02 smithi118 ceph-mon[31535]: pgmap v4212: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:17:05.203 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:17:04 smithi189 ceph-mon[36187]: pgmap v4213: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T02:17:05.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:17:04 smithi193 ceph-mon[36878]: pgmap v4213: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T02:17:05.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:17:04 smithi118 ceph-mon[31535]: pgmap v4213: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T02:17:06.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:17:05 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 6721.711299 secs 2024-03-24T02:17:06.453 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:17:05 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 6721.711299 secs 2024-03-24T02:17:06.460 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:17:05 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 6721.711299 secs 2024-03-24T02:17:07.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:17:06 smithi118 ceph-mon[31535]: pgmap v4214: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T02:17:07.453 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:17:06 smithi189 ceph-mon[36187]: pgmap v4214: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T02:17:07.460 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:17:06 smithi193 ceph-mon[36878]: pgmap v4214: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T02:17:09.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:17:08 smithi118 ceph-mon[31535]: pgmap v4215: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:17:09.453 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:17:08 smithi189 ceph-mon[36187]: pgmap v4215: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:17:09.460 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:17:08 smithi193 ceph-mon[36878]: pgmap v4215: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:17:11.255 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:17:10 smithi189 ceph-mon[36187]: pgmap v4216: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-24T02:17:11.255 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:17:10 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 6726.711361 secs 2024-03-24T02:17:11.262 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:17:10 smithi193 ceph-mon[36878]: pgmap v4216: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-24T02:17:11.262 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:17:10 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 6726.711361 secs 2024-03-24T02:17:11.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:17:10 smithi118 ceph-mon[31535]: pgmap v4216: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-24T02:17:11.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:17:10 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 6726.711361 secs 2024-03-24T02:17:13.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:17:12 smithi118 ceph-mon[31535]: pgmap v4217: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:17:13.453 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:17:12 smithi189 ceph-mon[36187]: pgmap v4217: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:17:13.460 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:17:12 smithi193 ceph-mon[36878]: pgmap v4217: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:17:15.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:17:14 smithi118 ceph-mon[31535]: pgmap v4218: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T02:17:15.453 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:17:14 smithi189 ceph-mon[36187]: pgmap v4218: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T02:17:15.460 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:17:14 smithi193 ceph-mon[36878]: pgmap v4218: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T02:17:16.445 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:17:15 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 6731.711451 secs 2024-03-24T02:17:16.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:17:15 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T02:17:16.453 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:17:15 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 6731.711451 secs 2024-03-24T02:17:16.453 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:17:15 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T02:17:16.460 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:17:15 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 6731.711451 secs 2024-03-24T02:17:16.460 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:17:15 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T02:17:17.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:17:17 smithi118 ceph-mon[31535]: pgmap v4219: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T02:17:17.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:17:17 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-24T02:17:17.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:17:17 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-24T02:17:17.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:17:17 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-03-24T02:17:17.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:17:17 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-03-24T02:17:17.447 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:17:17 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T02:17:17.447 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:17:17 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T02:17:17.447 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:17:17 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T02:17:17.453 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:17:17 smithi189 ceph-mon[36187]: pgmap v4219: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T02:17:17.453 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:17:17 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-24T02:17:17.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:17:17 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-24T02:17:17.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:17:17 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-03-24T02:17:17.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:17:17 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-03-24T02:17:17.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:17:17 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T02:17:17.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:17:17 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T02:17:17.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:17:17 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T02:17:17.460 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:17:17 smithi193 ceph-mon[36878]: pgmap v4219: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T02:17:17.460 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:17:17 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-24T02:17:17.460 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:17:17 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-24T02:17:17.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:17:17 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-03-24T02:17:17.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:17:17 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-03-24T02:17:17.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:17:17 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T02:17:17.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:17:17 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T02:17:17.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:17:17 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T02:17:18.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:17:18 smithi118 ceph-mon[31535]: Adjusting osd_memory_target on smithi193 to 174.1M 2024-03-24T02:17:18.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:17:18 smithi118 ceph-mon[31535]: Unable to set osd_memory_target on smithi193 to 182639923: error parsing value: Value '182639923' is below minimum 939524096 2024-03-24T02:17:18.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:17:18 smithi118 ceph-mon[31535]: pgmap v4220: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 5 op/s 2024-03-24T02:17:18.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:17:18 smithi118 ceph-mon[31535]: pgmap v4221: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-03-24T02:17:18.453 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:17:18 smithi189 ceph-mon[36187]: Adjusting osd_memory_target on smithi193 to 174.1M 2024-03-24T02:17:18.453 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:17:18 smithi189 ceph-mon[36187]: Unable to set osd_memory_target on smithi193 to 182639923: error parsing value: Value '182639923' is below minimum 939524096 2024-03-24T02:17:18.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:17:18 smithi189 ceph-mon[36187]: pgmap v4220: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 5 op/s 2024-03-24T02:17:18.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:17:18 smithi189 ceph-mon[36187]: pgmap v4221: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-03-24T02:17:18.460 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:17:18 smithi193 ceph-mon[36878]: Adjusting osd_memory_target on smithi193 to 174.1M 2024-03-24T02:17:18.460 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:17:18 smithi193 ceph-mon[36878]: Unable to set osd_memory_target on smithi193 to 182639923: error parsing value: Value '182639923' is below minimum 939524096 2024-03-24T02:17:18.460 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:17:18 smithi193 ceph-mon[36878]: pgmap v4220: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 5 op/s 2024-03-24T02:17:18.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:17:18 smithi193 ceph-mon[36878]: pgmap v4221: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-03-24T02:17:20.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:17:20 smithi118 ceph-mon[31535]: pgmap v4222: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 4 op/s 2024-03-24T02:17:20.453 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:17:20 smithi189 ceph-mon[36187]: pgmap v4222: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 4 op/s 2024-03-24T02:17:20.460 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:17:20 smithi193 ceph-mon[36878]: pgmap v4222: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 4 op/s 2024-03-24T02:17:21.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:17:21 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 6736.711534 secs 2024-03-24T02:17:21.453 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:17:21 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 6736.711534 secs 2024-03-24T02:17:21.460 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:17:21 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 6736.711534 secs 2024-03-24T02:17:22.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:17:22 smithi118 ceph-mon[31535]: pgmap v4223: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-03-24T02:17:22.453 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:17:22 smithi189 ceph-mon[36187]: pgmap v4223: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-03-24T02:17:22.460 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:17:22 smithi193 ceph-mon[36878]: pgmap v4223: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-03-24T02:17:24.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:17:24 smithi118 ceph-mon[31535]: pgmap v4224: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 5 op/s 2024-03-24T02:17:24.453 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:17:24 smithi189 ceph-mon[36187]: pgmap v4224: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 5 op/s 2024-03-24T02:17:24.460 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:17:24 smithi193 ceph-mon[36878]: pgmap v4224: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 5 op/s 2024-03-24T02:17:26.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:17:26 smithi118 ceph-mon[31535]: pgmap v4225: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-03-24T02:17:26.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:17:26 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 6741.711617 secs 2024-03-24T02:17:26.453 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:17:26 smithi189 ceph-mon[36187]: pgmap v4225: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-03-24T02:17:26.453 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:17:26 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 6741.711617 secs 2024-03-24T02:17:26.460 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:17:26 smithi193 ceph-mon[36878]: pgmap v4225: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-03-24T02:17:26.460 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:17:26 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 6741.711617 secs 2024-03-24T02:17:28.344 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:17:28 smithi118 ceph-mon[31535]: pgmap v4226: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 5 op/s 2024-03-24T02:17:28.453 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:17:28 smithi189 ceph-mon[36187]: pgmap v4226: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 5 op/s 2024-03-24T02:17:28.460 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:17:28 smithi193 ceph-mon[36878]: pgmap v4226: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 5 op/s 2024-03-24T02:17:30.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:17:30 smithi118 ceph-mon[31535]: pgmap v4227: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T02:17:30.453 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:17:30 smithi189 ceph-mon[36187]: pgmap v4227: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T02:17:30.460 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:17:30 smithi193 ceph-mon[36878]: pgmap v4227: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T02:17:31.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:17:31 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 6746.711706 secs 2024-03-24T02:17:31.453 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:17:31 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 6746.711706 secs 2024-03-24T02:17:31.460 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:17:31 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 6746.711706 secs 2024-03-24T02:17:32.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:17:32 smithi118 ceph-mon[31535]: pgmap v4228: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T02:17:32.453 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:17:32 smithi189 ceph-mon[36187]: pgmap v4228: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T02:17:32.460 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:17:32 smithi193 ceph-mon[36878]: pgmap v4228: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T02:17:34.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:17:34 smithi118 ceph-mon[31535]: pgmap v4229: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T02:17:34.453 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:17:34 smithi189 ceph-mon[36187]: pgmap v4229: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T02:17:34.460 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:17:34 smithi193 ceph-mon[36878]: pgmap v4229: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T02:17:36.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:17:36 smithi118 ceph-mon[31535]: pgmap v4230: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-03-24T02:17:36.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:17:36 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 6751.711795 secs 2024-03-24T02:17:36.453 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:17:36 smithi189 ceph-mon[36187]: pgmap v4230: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-03-24T02:17:36.453 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:17:36 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 6751.711795 secs 2024-03-24T02:17:36.460 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:17:36 smithi193 ceph-mon[36878]: pgmap v4230: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-03-24T02:17:36.460 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:17:36 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 6751.711795 secs 2024-03-24T02:17:38.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:17:38 smithi118 ceph-mon[31535]: pgmap v4231: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:17:38.453 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:17:38 smithi189 ceph-mon[36187]: pgmap v4231: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:17:38.460 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:17:38 smithi193 ceph-mon[36878]: pgmap v4231: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:17:40.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:17:40 smithi118 ceph-mon[31535]: pgmap v4232: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T02:17:40.453 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:17:40 smithi189 ceph-mon[36187]: pgmap v4232: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T02:17:40.460 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:17:40 smithi193 ceph-mon[36878]: pgmap v4232: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T02:17:41.354 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:17:41 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 6756.711880 secs 2024-03-24T02:17:41.361 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:17:41 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 6756.711880 secs 2024-03-24T02:17:41.445 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:17:41 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 6756.711880 secs 2024-03-24T02:17:42.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:17:42 smithi118 ceph-mon[31535]: pgmap v4233: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T02:17:42.453 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:17:42 smithi189 ceph-mon[36187]: pgmap v4233: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T02:17:42.460 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:17:42 smithi193 ceph-mon[36878]: pgmap v4233: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T02:17:44.447 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:17:44 smithi118 ceph-mon[31535]: pgmap v4234: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T02:17:44.453 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:17:44 smithi189 ceph-mon[36187]: pgmap v4234: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T02:17:44.460 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:17:44 smithi193 ceph-mon[36878]: pgmap v4234: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T02:17:46.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:17:46 smithi118 ceph-mon[31535]: pgmap v4235: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-03-24T02:17:46.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:17:46 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 6761.711966 secs 2024-03-24T02:17:46.453 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:17:46 smithi189 ceph-mon[36187]: pgmap v4235: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-03-24T02:17:46.453 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:17:46 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 6761.711966 secs 2024-03-24T02:17:46.460 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:17:46 smithi193 ceph-mon[36878]: pgmap v4235: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-03-24T02:17:46.460 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:17:46 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 6761.711966 secs 2024-03-24T02:17:48.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:17:48 smithi118 ceph-mon[31535]: pgmap v4236: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:17:48.453 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:17:48 smithi189 ceph-mon[36187]: pgmap v4236: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:17:48.460 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:17:48 smithi193 ceph-mon[36878]: pgmap v4236: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:17:50.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:17:50 smithi118 ceph-mon[31535]: pgmap v4237: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T02:17:50.453 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:17:50 smithi189 ceph-mon[36187]: pgmap v4237: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T02:17:50.460 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:17:50 smithi193 ceph-mon[36878]: pgmap v4237: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T02:17:51.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:17:51 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 6766.712053 secs 2024-03-24T02:17:51.453 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:17:51 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 6766.712053 secs 2024-03-24T02:17:51.460 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:17:51 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 6766.712053 secs 2024-03-24T02:17:52.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:17:52 smithi118 ceph-mon[31535]: pgmap v4238: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T02:17:52.453 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:17:52 smithi189 ceph-mon[36187]: pgmap v4238: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T02:17:52.460 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:17:52 smithi193 ceph-mon[36878]: pgmap v4238: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T02:17:54.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:17:54 smithi118 ceph-mon[31535]: pgmap v4239: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T02:17:54.453 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:17:54 smithi189 ceph-mon[36187]: pgmap v4239: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T02:17:54.460 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:17:54 smithi193 ceph-mon[36878]: pgmap v4239: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T02:17:56.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:17:56 smithi118 ceph-mon[31535]: pgmap v4240: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-03-24T02:17:56.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:17:56 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 6771.712139 secs 2024-03-24T02:17:56.453 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:17:56 smithi189 ceph-mon[36187]: pgmap v4240: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-03-24T02:17:56.453 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:17:56 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 6771.712139 secs 2024-03-24T02:17:56.460 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:17:56 smithi193 ceph-mon[36878]: pgmap v4240: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-03-24T02:17:56.460 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:17:56 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 6771.712139 secs 2024-03-24T02:17:58.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:17:58 smithi118 ceph-mon[31535]: pgmap v4241: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:17:58.453 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:17:58 smithi189 ceph-mon[36187]: pgmap v4241: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:17:58.460 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:17:58 smithi193 ceph-mon[36878]: pgmap v4241: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:18:00.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:18:00 smithi118 ceph-mon[31535]: pgmap v4242: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T02:18:00.453 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:18:00 smithi189 ceph-mon[36187]: pgmap v4242: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T02:18:00.460 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:18:00 smithi193 ceph-mon[36878]: pgmap v4242: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T02:18:01.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:18:01 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 6776.712227 secs 2024-03-24T02:18:01.453 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:18:01 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 6776.712227 secs 2024-03-24T02:18:01.460 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:18:01 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 6776.712227 secs 2024-03-24T02:18:02.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:18:02 smithi118 ceph-mon[31535]: pgmap v4243: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T02:18:02.453 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:18:02 smithi189 ceph-mon[36187]: pgmap v4243: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T02:18:02.460 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:18:02 smithi193 ceph-mon[36878]: pgmap v4243: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T02:18:04.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:18:04 smithi118 ceph-mon[31535]: pgmap v4244: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T02:18:04.453 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:18:04 smithi189 ceph-mon[36187]: pgmap v4244: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T02:18:04.460 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:18:04 smithi193 ceph-mon[36878]: pgmap v4244: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T02:18:06.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:18:06 smithi118 ceph-mon[31535]: pgmap v4245: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-03-24T02:18:06.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:18:06 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 6781.712288 secs 2024-03-24T02:18:06.453 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:18:06 smithi189 ceph-mon[36187]: pgmap v4245: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-03-24T02:18:06.453 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:18:06 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 6781.712288 secs 2024-03-24T02:18:06.460 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:18:06 smithi193 ceph-mon[36878]: pgmap v4245: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-03-24T02:18:06.460 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:18:06 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 6781.712288 secs 2024-03-24T02:18:08.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:18:08 smithi118 ceph-mon[31535]: pgmap v4246: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:18:08.453 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:18:08 smithi189 ceph-mon[36187]: pgmap v4246: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:18:08.460 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:18:08 smithi193 ceph-mon[36878]: pgmap v4246: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:18:10.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:18:10 smithi118 ceph-mon[31535]: pgmap v4247: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T02:18:10.453 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:18:10 smithi189 ceph-mon[36187]: pgmap v4247: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T02:18:10.460 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:18:10 smithi193 ceph-mon[36878]: pgmap v4247: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T02:18:11.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:18:11 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 6786.712362 secs 2024-03-24T02:18:11.453 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:18:11 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 6786.712362 secs 2024-03-24T02:18:11.460 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:18:11 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 6786.712362 secs 2024-03-24T02:18:12.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:18:12 smithi118 ceph-mon[31535]: pgmap v4248: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T02:18:12.453 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:18:12 smithi189 ceph-mon[36187]: pgmap v4248: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T02:18:12.460 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:18:12 smithi193 ceph-mon[36878]: pgmap v4248: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T02:18:14.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:18:14 smithi118 ceph-mon[31535]: pgmap v4249: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T02:18:14.453 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:18:14 smithi189 ceph-mon[36187]: pgmap v4249: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T02:18:14.460 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:18:14 smithi193 ceph-mon[36878]: pgmap v4249: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T02:18:16.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:18:16 smithi118 ceph-mon[31535]: pgmap v4250: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-03-24T02:18:16.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:18:16 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 6791.712452 secs 2024-03-24T02:18:16.453 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:18:16 smithi189 ceph-mon[36187]: pgmap v4250: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-03-24T02:18:16.453 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:18:16 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 6791.712452 secs 2024-03-24T02:18:16.460 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:18:16 smithi193 ceph-mon[36878]: pgmap v4250: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-03-24T02:18:16.460 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:18:16 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 6791.712452 secs 2024-03-24T02:18:17.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:18:17 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T02:18:17.453 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:18:17 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T02:18:17.460 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:18:17 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T02:18:18.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:18:18 smithi118 ceph-mon[31535]: pgmap v4251: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:18:18.453 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:18:18 smithi189 ceph-mon[36187]: pgmap v4251: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:18:18.460 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:18:18 smithi193 ceph-mon[36878]: pgmap v4251: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:18:19.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:18:19 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T02:18:19.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:18:19 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T02:18:19.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:18:19 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T02:18:19.453 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:18:19 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T02:18:19.453 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:18:19 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T02:18:19.453 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:18:19 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T02:18:19.460 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:18:19 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T02:18:19.460 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:18:19 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T02:18:19.460 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:18:19 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T02:18:20.453 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:18:20 smithi189 ceph-mon[36187]: pgmap v4252: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 6 op/s 2024-03-24T02:18:20.453 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:18:20 smithi189 ceph-mon[36187]: pgmap v4253: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 6 op/s 2024-03-24T02:18:20.460 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:18:20 smithi193 ceph-mon[36878]: pgmap v4252: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 6 op/s 2024-03-24T02:18:20.460 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:18:20 smithi193 ceph-mon[36878]: pgmap v4253: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 6 op/s 2024-03-24T02:18:20.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:18:20 smithi118 ceph-mon[31535]: pgmap v4252: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 6 op/s 2024-03-24T02:18:20.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:18:20 smithi118 ceph-mon[31535]: pgmap v4253: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 6 op/s 2024-03-24T02:18:21.460 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:18:21 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 6796.712513 secs 2024-03-24T02:18:21.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:18:21 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 6796.712513 secs 2024-03-24T02:18:21.703 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:18:21 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 6796.712513 secs 2024-03-24T02:18:22.460 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:18:22 smithi193 ceph-mon[36878]: pgmap v4254: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 6 op/s 2024-03-24T02:18:22.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:18:22 smithi118 ceph-mon[31535]: pgmap v4254: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 6 op/s 2024-03-24T02:18:22.703 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:18:22 smithi189 ceph-mon[36187]: pgmap v4254: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 6 op/s 2024-03-24T02:18:24.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:18:24 smithi118 ceph-mon[31535]: pgmap v4255: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 5 op/s 2024-03-24T02:18:24.703 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:18:24 smithi189 ceph-mon[36187]: pgmap v4255: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 5 op/s 2024-03-24T02:18:24.710 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:18:24 smithi193 ceph-mon[36878]: pgmap v4255: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 5 op/s 2024-03-24T02:18:26.499 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:18:26 smithi118 ceph-mon[31535]: pgmap v4256: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 6 op/s 2024-03-24T02:18:26.499 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:18:26 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 6801.712570 secs 2024-03-24T02:18:26.703 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:18:26 smithi189 ceph-mon[36187]: pgmap v4256: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 6 op/s 2024-03-24T02:18:26.703 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:18:26 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 6801.712570 secs 2024-03-24T02:18:26.710 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:18:26 smithi193 ceph-mon[36878]: pgmap v4256: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 6 op/s 2024-03-24T02:18:26.710 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:18:26 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 6801.712570 secs 2024-03-24T02:18:28.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:18:28 smithi118 ceph-mon[31535]: pgmap v4257: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 6 op/s 2024-03-24T02:18:28.703 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:18:28 smithi189 ceph-mon[36187]: pgmap v4257: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 6 op/s 2024-03-24T02:18:28.710 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:18:28 smithi193 ceph-mon[36878]: pgmap v4257: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 6 op/s 2024-03-24T02:18:30.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:18:30 smithi118 ceph-mon[31535]: pgmap v4258: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 5 op/s 2024-03-24T02:18:30.703 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:18:30 smithi189 ceph-mon[36187]: pgmap v4258: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 5 op/s 2024-03-24T02:18:30.710 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:18:30 smithi193 ceph-mon[36878]: pgmap v4258: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 5 op/s 2024-03-24T02:18:31.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:18:31 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 6806.712632 secs 2024-03-24T02:18:31.703 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:18:31 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 6806.712632 secs 2024-03-24T02:18:31.710 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:18:31 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 6806.712632 secs 2024-03-24T02:18:32.697 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:18:32 smithi118 ceph-mon[31535]: pgmap v4259: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T02:18:32.703 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:18:32 smithi189 ceph-mon[36187]: pgmap v4259: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T02:18:32.710 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:18:32 smithi193 ceph-mon[36878]: pgmap v4259: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T02:18:34.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:18:34 smithi118 ceph-mon[31535]: pgmap v4260: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:18:34.703 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:18:34 smithi189 ceph-mon[36187]: pgmap v4260: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:18:34.710 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:18:34 smithi193 ceph-mon[36878]: pgmap v4260: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:18:36.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:18:36 smithi118 ceph-mon[31535]: pgmap v4261: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-24T02:18:36.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:18:36 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 6811.712722 secs 2024-03-24T02:18:36.703 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:18:36 smithi189 ceph-mon[36187]: pgmap v4261: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-24T02:18:36.703 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:18:36 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 6811.712722 secs 2024-03-24T02:18:36.710 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:18:36 smithi193 ceph-mon[36878]: pgmap v4261: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-24T02:18:36.710 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:18:36 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 6811.712722 secs 2024-03-24T02:18:38.695 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:18:38 smithi118 ceph-mon[31535]: pgmap v4262: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:18:38.703 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:18:38 smithi189 ceph-mon[36187]: pgmap v4262: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:18:38.710 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:18:38 smithi193 ceph-mon[36878]: pgmap v4262: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:18:40.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:18:40 smithi118 ceph-mon[31535]: pgmap v4263: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T02:18:40.703 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:18:40 smithi189 ceph-mon[36187]: pgmap v4263: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T02:18:40.710 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:18:40 smithi193 ceph-mon[36878]: pgmap v4263: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T02:18:41.604 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:18:41 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 6816.712811 secs 2024-03-24T02:18:41.611 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:18:41 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 6816.712811 secs 2024-03-24T02:18:41.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:18:41 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 6816.712811 secs 2024-03-24T02:18:42.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:18:42 smithi118 ceph-mon[31535]: pgmap v4264: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T02:18:42.703 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:18:42 smithi189 ceph-mon[36187]: pgmap v4264: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T02:18:42.710 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:18:42 smithi193 ceph-mon[36878]: pgmap v4264: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T02:18:44.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:18:44 smithi118 ceph-mon[31535]: pgmap v4265: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:18:44.703 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:18:44 smithi189 ceph-mon[36187]: pgmap v4265: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:18:44.710 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:18:44 smithi193 ceph-mon[36878]: pgmap v4265: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:18:46.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:18:46 smithi118 ceph-mon[31535]: pgmap v4266: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-24T02:18:46.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:18:46 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 6821.712871 secs 2024-03-24T02:18:46.703 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:18:46 smithi189 ceph-mon[36187]: pgmap v4266: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-24T02:18:46.703 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:18:46 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 6821.712871 secs 2024-03-24T02:18:46.710 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:18:46 smithi193 ceph-mon[36878]: pgmap v4266: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-24T02:18:46.710 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:18:46 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 6821.712871 secs 2024-03-24T02:18:48.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:18:48 smithi118 ceph-mon[31535]: pgmap v4267: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:18:48.703 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:18:48 smithi189 ceph-mon[36187]: pgmap v4267: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:18:48.710 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:18:48 smithi193 ceph-mon[36878]: pgmap v4267: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:18:50.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:18:50 smithi118 ceph-mon[31535]: pgmap v4268: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T02:18:50.703 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:18:50 smithi189 ceph-mon[36187]: pgmap v4268: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T02:18:50.710 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:18:50 smithi193 ceph-mon[36878]: pgmap v4268: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T02:18:51.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:18:51 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 6826.712912 secs 2024-03-24T02:18:51.703 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:18:51 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 6826.712912 secs 2024-03-24T02:18:51.710 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:18:51 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 6826.712912 secs 2024-03-24T02:18:52.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:18:52 smithi118 ceph-mon[31535]: pgmap v4269: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T02:18:52.703 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:18:52 smithi189 ceph-mon[36187]: pgmap v4269: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T02:18:52.710 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:18:52 smithi193 ceph-mon[36878]: pgmap v4269: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T02:18:54.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:18:54 smithi118 ceph-mon[31535]: pgmap v4270: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:18:54.703 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:18:54 smithi189 ceph-mon[36187]: pgmap v4270: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:18:54.710 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:18:54 smithi193 ceph-mon[36878]: pgmap v4270: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:18:55.695 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:18:55 smithi118 ceph-mon[31535]: pgmap v4271: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-24T02:18:55.703 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:18:55 smithi189 ceph-mon[36187]: pgmap v4271: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-24T02:18:55.710 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:18:55 smithi193 ceph-mon[36878]: pgmap v4271: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-24T02:18:56.597 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:18:56 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 6831.713001 secs 2024-03-24T02:18:56.703 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:18:56 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 6831.713001 secs 2024-03-24T02:18:56.710 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:18:56 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 6831.713001 secs 2024-03-24T02:18:57.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:18:57 smithi118 ceph-mon[31535]: pgmap v4272: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:18:57.703 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:18:57 smithi189 ceph-mon[36187]: pgmap v4272: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:18:57.710 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:18:57 smithi193 ceph-mon[36878]: pgmap v4272: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:18:59.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:18:59 smithi118 ceph-mon[31535]: pgmap v4273: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T02:18:59.953 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:18:59 smithi189 ceph-mon[36187]: pgmap v4273: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T02:18:59.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:18:59 smithi193 ceph-mon[36878]: pgmap v4273: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T02:19:01.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:19:01 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 6836.713090 secs 2024-03-24T02:19:01.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:19:01 smithi118 ceph-mon[31535]: pgmap v4274: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T02:19:01.953 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:19:01 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 6836.713090 secs 2024-03-24T02:19:01.953 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:19:01 smithi189 ceph-mon[36187]: pgmap v4274: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T02:19:01.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:19:01 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 6836.713090 secs 2024-03-24T02:19:01.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:19:01 smithi193 ceph-mon[36878]: pgmap v4274: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T02:19:03.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:19:03 smithi118 ceph-mon[31535]: pgmap v4275: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:19:03.953 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:19:03 smithi189 ceph-mon[36187]: pgmap v4275: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:19:03.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:19:03 smithi193 ceph-mon[36878]: pgmap v4275: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:19:05.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:19:05 smithi118 ceph-mon[31535]: pgmap v4276: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-24T02:19:05.953 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:19:05 smithi189 ceph-mon[36187]: pgmap v4276: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-24T02:19:05.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:19:05 smithi193 ceph-mon[36878]: pgmap v4276: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-24T02:19:06.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:19:06 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 6841.713148 secs 2024-03-24T02:19:06.953 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:19:06 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 6841.713148 secs 2024-03-24T02:19:06.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:19:06 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 6841.713148 secs 2024-03-24T02:19:07.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:19:07 smithi118 ceph-mon[31535]: pgmap v4277: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-03-24T02:19:07.953 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:19:07 smithi189 ceph-mon[36187]: pgmap v4277: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-03-24T02:19:07.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:19:07 smithi193 ceph-mon[36878]: pgmap v4277: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-03-24T02:19:09.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:19:09 smithi118 ceph-mon[31535]: pgmap v4278: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-03-24T02:19:09.953 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:19:09 smithi189 ceph-mon[36187]: pgmap v4278: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-03-24T02:19:09.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:19:09 smithi193 ceph-mon[36878]: pgmap v4278: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-03-24T02:19:11.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:19:11 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 6846.713188 secs 2024-03-24T02:19:11.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:19:11 smithi118 ceph-mon[31535]: pgmap v4279: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 6 op/s 2024-03-24T02:19:11.953 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:19:11 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 6846.713188 secs 2024-03-24T02:19:11.953 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:19:11 smithi189 ceph-mon[36187]: pgmap v4279: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 6 op/s 2024-03-24T02:19:11.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:19:11 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 6846.713188 secs 2024-03-24T02:19:11.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:19:11 smithi193 ceph-mon[36878]: pgmap v4279: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 6 op/s 2024-03-24T02:19:13.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:19:13 smithi118 ceph-mon[31535]: pgmap v4280: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-03-24T02:19:13.953 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:19:13 smithi189 ceph-mon[36187]: pgmap v4280: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-03-24T02:19:13.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:19:13 smithi193 ceph-mon[36878]: pgmap v4280: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-03-24T02:19:15.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:19:15 smithi118 ceph-mon[31535]: pgmap v4281: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-24T02:19:15.953 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:19:15 smithi189 ceph-mon[36187]: pgmap v4281: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-24T02:19:15.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:19:15 smithi193 ceph-mon[36878]: pgmap v4281: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-24T02:19:16.836 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T02:19:16.836+0000 7f81db400640 1 -- 172.21.15.118:0/659686424 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f81c4006290 con 0x5622651a13f0 2024-03-24T02:19:16.836 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T02:19:16.836+0000 7f81ee000640 1 -- 172.21.15.118:0/1693767697 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f81c8006590 con 0x7f81f81052d0 2024-03-24T02:19:16.836 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T02:19:16.837+0000 7f81ef400640 1 -- 172.21.15.118:0/1693767697 <== mon.2 v2:172.21.15.189:3300/0 56 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (secure 0 0 0) 0x7f81c8006590 con 0x7f81f81052d0 2024-03-24T02:19:16.836 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T02:19:16.837+0000 7f81e4c00640 1 -- 172.21.15.118:0/659686424 <== mon.1 v2:172.21.15.193:3300/0 56 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 409+0+0 (secure 0 0 0) 0x7f81c4006290 con 0x5622651a13f0 2024-03-24T02:19:16.888 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:19:16 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 6851.713276 secs 2024-03-24T02:19:16.953 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:19:16 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 6851.713276 secs 2024-03-24T02:19:16.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:19:16 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 6851.713276 secs 2024-03-24T02:19:17.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:19:17 smithi118 ceph-mon[31535]: pgmap v4282: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:19:17.953 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:19:17 smithi189 ceph-mon[36187]: pgmap v4282: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:19:17.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:19:17 smithi193 ceph-mon[36878]: pgmap v4282: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:19:19.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:19:19 smithi118 ceph-mon[31535]: pgmap v4283: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T02:19:19.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:19:19 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T02:19:19.953 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:19:19 smithi189 ceph-mon[36187]: pgmap v4283: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T02:19:19.953 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:19:19 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T02:19:19.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:19:19 smithi193 ceph-mon[36878]: pgmap v4283: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T02:19:19.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:19:19 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T02:19:20.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:19:20 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T02:19:20.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:19:20 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T02:19:20.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:19:20 smithi118 ceph-mon[31535]: pgmap v4284: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 6 op/s 2024-03-24T02:19:20.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:19:20 smithi118 ceph-mon[31535]: pgmap v4285: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-03-24T02:19:20.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:19:20 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T02:19:20.953 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:19:20 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T02:19:20.953 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:19:20 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T02:19:20.953 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:19:20 smithi189 ceph-mon[36187]: pgmap v4284: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 6 op/s 2024-03-24T02:19:20.953 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:19:20 smithi189 ceph-mon[36187]: pgmap v4285: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-03-24T02:19:20.953 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:19:20 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T02:19:20.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:19:20 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T02:19:20.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:19:20 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T02:19:20.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:19:20 smithi193 ceph-mon[36878]: pgmap v4284: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 6 op/s 2024-03-24T02:19:20.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:19:20 smithi193 ceph-mon[36878]: pgmap v4285: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-03-24T02:19:20.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:19:20 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T02:19:21.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:19:21 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 6856.713367 secs 2024-03-24T02:19:21.953 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:19:21 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 6856.713367 secs 2024-03-24T02:19:21.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:19:21 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 6856.713367 secs 2024-03-24T02:19:22.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:19:22 smithi118 ceph-mon[31535]: pgmap v4286: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-03-24T02:19:22.953 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:19:22 smithi189 ceph-mon[36187]: pgmap v4286: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-03-24T02:19:22.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:19:22 smithi193 ceph-mon[36878]: pgmap v4286: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-03-24T02:19:25.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:19:25 smithi118 ceph-mon[31535]: pgmap v4287: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 5 op/s 2024-03-24T02:19:25.453 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:19:25 smithi189 ceph-mon[36187]: pgmap v4287: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 5 op/s 2024-03-24T02:19:25.460 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:19:25 smithi193 ceph-mon[36878]: pgmap v4287: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 5 op/s 2024-03-24T02:19:26.348 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:19:26 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 6861.713410 secs 2024-03-24T02:19:26.453 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:19:26 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 6861.713410 secs 2024-03-24T02:19:26.460 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:19:26 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 6861.713410 secs 2024-03-24T02:19:27.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:19:27 smithi118 ceph-mon[31535]: pgmap v4288: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-03-24T02:19:27.453 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:19:27 smithi189 ceph-mon[36187]: pgmap v4288: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-03-24T02:19:27.460 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:19:27 smithi193 ceph-mon[36878]: pgmap v4288: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-03-24T02:19:29.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:19:29 smithi118 ceph-mon[31535]: pgmap v4289: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 5 op/s 2024-03-24T02:19:29.453 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:19:29 smithi189 ceph-mon[36187]: pgmap v4289: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 5 op/s 2024-03-24T02:19:29.460 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:19:29 smithi193 ceph-mon[36878]: pgmap v4289: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 5 op/s 2024-03-24T02:19:31.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:19:31 smithi118 ceph-mon[31535]: pgmap v4290: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 5 op/s 2024-03-24T02:19:31.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:19:31 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 6866.713470 secs 2024-03-24T02:19:31.453 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:19:31 smithi189 ceph-mon[36187]: pgmap v4290: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 5 op/s 2024-03-24T02:19:31.453 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:19:31 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 6866.713470 secs 2024-03-24T02:19:31.460 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:19:31 smithi193 ceph-mon[36878]: pgmap v4290: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 5 op/s 2024-03-24T02:19:31.460 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:19:31 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 6866.713470 secs 2024-03-24T02:19:33.445 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:19:33 smithi118 ceph-mon[31535]: pgmap v4291: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:19:33.453 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:19:33 smithi189 ceph-mon[36187]: pgmap v4291: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:19:33.460 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:19:33 smithi193 ceph-mon[36878]: pgmap v4291: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:19:35.445 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:19:35 smithi118 ceph-mon[31535]: pgmap v4292: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T02:19:35.453 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:19:35 smithi189 ceph-mon[36187]: pgmap v4292: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T02:19:35.460 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:19:35 smithi193 ceph-mon[36878]: pgmap v4292: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T02:19:36.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:19:36 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 6871.713558 secs 2024-03-24T02:19:36.453 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:19:36 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 6871.713558 secs 2024-03-24T02:19:36.460 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:19:36 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 6871.713558 secs 2024-03-24T02:19:37.445 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:19:37 smithi118 ceph-mon[31535]: pgmap v4293: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T02:19:37.453 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:19:37 smithi189 ceph-mon[36187]: pgmap v4293: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T02:19:37.460 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:19:37 smithi193 ceph-mon[36878]: pgmap v4293: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T02:19:39.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:19:39 smithi118 ceph-mon[31535]: pgmap v4294: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 5 op/s 2024-03-24T02:19:39.453 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:19:39 smithi189 ceph-mon[36187]: pgmap v4294: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 5 op/s 2024-03-24T02:19:39.460 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:19:39 smithi193 ceph-mon[36878]: pgmap v4294: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 5 op/s 2024-03-24T02:19:41.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:19:41 smithi118 ceph-mon[31535]: pgmap v4295: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T02:19:41.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:19:41 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 6876.713619 secs 2024-03-24T02:19:41.453 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:19:41 smithi189 ceph-mon[36187]: pgmap v4295: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T02:19:41.453 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:19:41 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 6876.713619 secs 2024-03-24T02:19:41.460 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:19:41 smithi193 ceph-mon[36878]: pgmap v4295: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T02:19:41.460 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:19:41 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 6876.713619 secs 2024-03-24T02:19:43.445 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:19:43 smithi118 ceph-mon[31535]: pgmap v4296: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:19:43.453 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:19:43 smithi189 ceph-mon[36187]: pgmap v4296: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:19:43.460 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:19:43 smithi193 ceph-mon[36878]: pgmap v4296: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:19:45.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:19:45 smithi118 ceph-mon[31535]: pgmap v4297: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T02:19:45.455 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:19:45 smithi189 ceph-mon[36187]: pgmap v4297: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T02:19:45.460 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:19:45 smithi193 ceph-mon[36878]: pgmap v4297: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T02:19:46.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:19:46 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 6881.713709 secs 2024-03-24T02:19:46.453 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:19:46 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 6881.713709 secs 2024-03-24T02:19:46.460 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:19:46 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 6881.713709 secs 2024-03-24T02:19:47.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:19:47 smithi118 ceph-mon[31535]: pgmap v4298: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T02:19:47.453 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:19:47 smithi189 ceph-mon[36187]: pgmap v4298: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T02:19:47.460 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:19:47 smithi193 ceph-mon[36878]: pgmap v4298: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T02:19:49.445 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:19:49 smithi118 ceph-mon[31535]: pgmap v4299: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 5 op/s 2024-03-24T02:19:49.453 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:19:49 smithi189 ceph-mon[36187]: pgmap v4299: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 5 op/s 2024-03-24T02:19:49.460 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:19:49 smithi193 ceph-mon[36878]: pgmap v4299: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 5 op/s 2024-03-24T02:19:51.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:19:51 smithi118 ceph-mon[31535]: pgmap v4300: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T02:19:51.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:19:51 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 6886.713768 secs 2024-03-24T02:19:51.453 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:19:51 smithi189 ceph-mon[36187]: pgmap v4300: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T02:19:51.453 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:19:51 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 6886.713768 secs 2024-03-24T02:19:51.460 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:19:51 smithi193 ceph-mon[36878]: pgmap v4300: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T02:19:51.460 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:19:51 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 6886.713768 secs 2024-03-24T02:19:53.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:19:53 smithi118 ceph-mon[31535]: pgmap v4301: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:19:53.453 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:19:53 smithi189 ceph-mon[36187]: pgmap v4301: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:19:53.460 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:19:53 smithi193 ceph-mon[36878]: pgmap v4301: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:19:55.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:19:55 smithi118 ceph-mon[31535]: pgmap v4302: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T02:19:55.453 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:19:55 smithi189 ceph-mon[36187]: pgmap v4302: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T02:19:55.460 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:19:55 smithi193 ceph-mon[36878]: pgmap v4302: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T02:19:56.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:19:56 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 6891.713828 secs 2024-03-24T02:19:56.453 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:19:56 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 6891.713828 secs 2024-03-24T02:19:56.460 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:19:56 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 6891.713828 secs 2024-03-24T02:19:57.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:19:57 smithi118 ceph-mon[31535]: pgmap v4303: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T02:19:57.453 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:19:57 smithi189 ceph-mon[36187]: pgmap v4303: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T02:19:57.460 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:19:57 smithi193 ceph-mon[36878]: pgmap v4303: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T02:19:59.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:19:59 smithi118 ceph-mon[31535]: pgmap v4304: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 5 op/s 2024-03-24T02:19:59.453 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:19:59 smithi189 ceph-mon[36187]: pgmap v4304: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 5 op/s 2024-03-24T02:19:59.460 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:19:59 smithi193 ceph-mon[36878]: pgmap v4304: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 5 op/s 2024-03-24T02:20:00.445 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:20:00 smithi118 ceph-mon[31535]: overall HEALTH_WARN 6 failed cephadm daemon(s); 1 MDSs report slow requests 2024-03-24T02:20:00.453 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:20:00 smithi189 ceph-mon[36187]: overall HEALTH_WARN 6 failed cephadm daemon(s); 1 MDSs report slow requests 2024-03-24T02:20:00.460 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:20:00 smithi193 ceph-mon[36878]: overall HEALTH_WARN 6 failed cephadm daemon(s); 1 MDSs report slow requests 2024-03-24T02:20:01.195 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:20:01 smithi118 ceph-mon[31535]: pgmap v4305: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T02:20:01.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:20:01 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 6896.713910 secs 2024-03-24T02:20:01.453 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:20:01 smithi189 ceph-mon[36187]: pgmap v4305: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T02:20:01.453 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:20:01 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 6896.713910 secs 2024-03-24T02:20:01.460 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:20:01 smithi193 ceph-mon[36878]: pgmap v4305: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T02:20:01.460 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:20:01 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 6896.713910 secs 2024-03-24T02:20:03.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:20:03 smithi118 ceph-mon[31535]: pgmap v4306: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:20:03.453 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:20:03 smithi189 ceph-mon[36187]: pgmap v4306: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:20:03.460 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:20:03 smithi193 ceph-mon[36878]: pgmap v4306: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:20:05.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:20:05 smithi118 ceph-mon[31535]: pgmap v4307: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T02:20:05.453 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:20:05 smithi189 ceph-mon[36187]: pgmap v4307: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T02:20:05.460 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:20:05 smithi193 ceph-mon[36878]: pgmap v4307: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T02:20:06.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:20:06 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 6901.714000 secs 2024-03-24T02:20:06.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:20:06 smithi193 ceph-mon[36878]: pgmap v4308: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T02:20:07.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:20:06 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 6901.714000 secs 2024-03-24T02:20:07.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:20:06 smithi118 ceph-mon[31535]: pgmap v4308: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T02:20:07.203 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:20:06 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 6901.714000 secs 2024-03-24T02:20:07.203 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:20:06 smithi189 ceph-mon[36187]: pgmap v4308: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T02:20:09.445 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:20:09 smithi118 ceph-mon[31535]: pgmap v4309: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 5 op/s 2024-03-24T02:20:09.453 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:20:09 smithi189 ceph-mon[36187]: pgmap v4309: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 5 op/s 2024-03-24T02:20:09.460 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:20:09 smithi193 ceph-mon[36878]: pgmap v4309: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 5 op/s 2024-03-24T02:20:11.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:20:11 smithi118 ceph-mon[31535]: pgmap v4310: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T02:20:11.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:20:11 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 6906.714090 secs 2024-03-24T02:20:11.453 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:20:11 smithi189 ceph-mon[36187]: pgmap v4310: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T02:20:11.453 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:20:11 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 6906.714090 secs 2024-03-24T02:20:11.460 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:20:11 smithi193 ceph-mon[36878]: pgmap v4310: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T02:20:11.460 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:20:11 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 6906.714090 secs 2024-03-24T02:20:13.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:20:13 smithi118 ceph-mon[31535]: pgmap v4311: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:20:13.453 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:20:13 smithi189 ceph-mon[36187]: pgmap v4311: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:20:13.460 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:20:13 smithi193 ceph-mon[36878]: pgmap v4311: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:20:15.445 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:20:15 smithi118 ceph-mon[31535]: pgmap v4312: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T02:20:15.453 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:20:15 smithi189 ceph-mon[36187]: pgmap v4312: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T02:20:15.460 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:20:15 smithi193 ceph-mon[36878]: pgmap v4312: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T02:20:16.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:20:16 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 6911.714174 secs 2024-03-24T02:20:16.453 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:20:16 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 6911.714174 secs 2024-03-24T02:20:16.460 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:20:16 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 6911.714174 secs 2024-03-24T02:20:17.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:20:17 smithi118 ceph-mon[31535]: pgmap v4313: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T02:20:17.453 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:20:17 smithi189 ceph-mon[36187]: pgmap v4313: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T02:20:17.460 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:20:17 smithi193 ceph-mon[36878]: pgmap v4313: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T02:20:19.352 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:20:19 smithi118 ceph-mon[31535]: pgmap v4314: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 5 op/s 2024-03-24T02:20:19.453 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:20:19 smithi189 ceph-mon[36187]: pgmap v4314: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 5 op/s 2024-03-24T02:20:19.460 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:20:19 smithi193 ceph-mon[36878]: pgmap v4314: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 5 op/s 2024-03-24T02:20:21.369 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:20:21 smithi193 ceph-mon[36878]: pgmap v4315: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T02:20:21.369 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:20:21 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T02:20:21.369 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:20:21 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 6916.714258 secs 2024-03-24T02:20:21.369 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:20:21 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T02:20:21.369 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:20:21 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T02:20:21.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:20:21 smithi118 ceph-mon[31535]: pgmap v4315: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T02:20:21.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:20:21 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T02:20:21.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:20:21 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 6916.714258 secs 2024-03-24T02:20:21.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:20:21 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T02:20:21.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:20:21 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T02:20:21.453 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:20:21 smithi189 ceph-mon[36187]: pgmap v4315: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T02:20:21.453 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:20:21 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T02:20:21.453 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:20:21 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 6916.714258 secs 2024-03-24T02:20:21.453 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:20:21 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T02:20:21.453 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:20:21 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T02:20:22.375 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:20:22 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T02:20:22.375 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:20:22 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T02:20:22.453 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:20:22 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T02:20:22.453 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:20:22 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T02:20:22.460 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:20:22 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T02:20:22.460 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:20:22 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T02:20:23.445 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:20:23 smithi118 ceph-mon[31535]: pgmap v4316: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:20:23.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:20:23 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T02:20:23.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:20:23 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T02:20:23.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:20:23 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T02:20:23.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:20:23 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T02:20:23.453 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:20:23 smithi189 ceph-mon[36187]: pgmap v4316: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:20:23.453 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:20:23 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T02:20:23.453 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:20:23 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T02:20:23.453 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:20:23 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T02:20:23.453 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:20:23 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T02:20:23.460 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:20:23 smithi193 ceph-mon[36878]: pgmap v4316: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:20:23.460 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:20:23 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T02:20:23.460 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:20:23 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T02:20:23.460 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:20:23 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T02:20:23.460 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:20:23 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T02:20:25.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:20:25 smithi118 ceph-mon[31535]: pgmap v4317: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T02:20:25.453 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:20:25 smithi189 ceph-mon[36187]: pgmap v4317: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T02:20:25.460 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:20:25 smithi193 ceph-mon[36878]: pgmap v4317: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T02:20:26.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:20:26 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T02:20:26.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:20:26 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T02:20:26.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:20:26 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 6921.714344 secs 2024-03-24T02:20:26.453 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:20:26 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T02:20:26.453 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:20:26 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T02:20:26.453 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:20:26 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 6921.714344 secs 2024-03-24T02:20:26.460 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:20:26 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T02:20:26.460 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:20:26 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T02:20:26.460 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:20:26 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 6921.714344 secs 2024-03-24T02:20:27.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:20:27 smithi118 ceph-mon[31535]: pgmap v4318: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T02:20:27.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:20:27 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T02:20:27.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:20:27 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T02:20:27.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:20:27 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T02:20:27.453 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:20:27 smithi189 ceph-mon[36187]: pgmap v4318: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T02:20:27.453 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:20:27 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T02:20:27.453 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:20:27 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T02:20:27.453 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:20:27 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T02:20:27.460 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:20:27 smithi193 ceph-mon[36878]: pgmap v4318: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T02:20:27.460 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:20:27 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T02:20:27.460 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:20:27 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T02:20:27.460 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:20:27 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T02:20:28.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:20:28 smithi118 ceph-mon[31535]: pgmap v4319: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T02:20:28.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:20:28 smithi118 ceph-mon[31535]: pgmap v4320: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 5 op/s 2024-03-24T02:20:28.453 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:20:28 smithi189 ceph-mon[36187]: pgmap v4319: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T02:20:28.453 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:20:28 smithi189 ceph-mon[36187]: pgmap v4320: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 5 op/s 2024-03-24T02:20:28.460 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:20:28 smithi193 ceph-mon[36878]: pgmap v4319: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T02:20:28.460 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:20:28 smithi193 ceph-mon[36878]: pgmap v4320: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 5 op/s 2024-03-24T02:20:30.445 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:20:30 smithi118 ceph-mon[31535]: pgmap v4321: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 5 op/s 2024-03-24T02:20:30.453 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:20:30 smithi189 ceph-mon[36187]: pgmap v4321: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 5 op/s 2024-03-24T02:20:30.460 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:20:30 smithi193 ceph-mon[36878]: pgmap v4321: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 5 op/s 2024-03-24T02:20:31.445 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:20:31 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 6926.714426 secs 2024-03-24T02:20:31.453 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:20:31 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 6926.714426 secs 2024-03-24T02:20:31.460 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:20:31 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 6926.714426 secs 2024-03-24T02:20:32.445 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:20:32 smithi118 ceph-mon[31535]: pgmap v4322: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 7 op/s 2024-03-24T02:20:32.453 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:20:32 smithi189 ceph-mon[36187]: pgmap v4322: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 7 op/s 2024-03-24T02:20:32.460 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:20:32 smithi193 ceph-mon[36878]: pgmap v4322: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 7 op/s 2024-03-24T02:20:34.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:20:34 smithi118 ceph-mon[31535]: pgmap v4323: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-03-24T02:20:34.453 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:20:34 smithi189 ceph-mon[36187]: pgmap v4323: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-03-24T02:20:34.460 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:20:34 smithi193 ceph-mon[36878]: pgmap v4323: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-03-24T02:20:36.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:20:36 smithi118 ceph-mon[31535]: pgmap v4324: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 7 op/s 2024-03-24T02:20:36.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:20:36 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 6931.714513 secs 2024-03-24T02:20:36.453 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:20:36 smithi189 ceph-mon[36187]: pgmap v4324: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 7 op/s 2024-03-24T02:20:36.453 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:20:36 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 6931.714513 secs 2024-03-24T02:20:36.460 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:20:36 smithi193 ceph-mon[36878]: pgmap v4324: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 7 op/s 2024-03-24T02:20:36.460 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:20:36 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 6931.714513 secs 2024-03-24T02:20:38.453 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:20:38 smithi189 ceph-mon[36187]: pgmap v4325: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 5 op/s 2024-03-24T02:20:38.460 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:20:38 smithi193 ceph-mon[36878]: pgmap v4325: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 5 op/s 2024-03-24T02:20:38.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:20:38 smithi118 ceph-mon[31535]: pgmap v4325: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 5 op/s 2024-03-24T02:20:40.453 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:20:40 smithi189 ceph-mon[36187]: pgmap v4326: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T02:20:40.460 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:20:40 smithi193 ceph-mon[36878]: pgmap v4326: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T02:20:40.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:20:40 smithi118 ceph-mon[31535]: pgmap v4326: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T02:20:41.460 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:20:41 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 6936.714598 secs 2024-03-24T02:20:41.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:20:41 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 6936.714598 secs 2024-03-24T02:20:41.703 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:20:41 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 6936.714598 secs 2024-03-24T02:20:42.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:20:42 smithi118 ceph-mon[31535]: pgmap v4327: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T02:20:42.703 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:20:42 smithi189 ceph-mon[36187]: pgmap v4327: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T02:20:42.710 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:20:42 smithi193 ceph-mon[36878]: pgmap v4327: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T02:20:44.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:20:44 smithi118 ceph-mon[31535]: pgmap v4328: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:20:44.703 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:20:44 smithi189 ceph-mon[36187]: pgmap v4328: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:20:44.710 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:20:44 smithi193 ceph-mon[36878]: pgmap v4328: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:20:46.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:20:46 smithi118 ceph-mon[31535]: pgmap v4329: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-24T02:20:46.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:20:46 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 6941.714686 secs 2024-03-24T02:20:46.703 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:20:46 smithi189 ceph-mon[36187]: pgmap v4329: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-24T02:20:46.703 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:20:46 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 6941.714686 secs 2024-03-24T02:20:46.710 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:20:46 smithi193 ceph-mon[36878]: pgmap v4329: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-24T02:20:46.710 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:20:46 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 6941.714686 secs 2024-03-24T02:20:46.850 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T02:20:46.851+0000 7f81db400640 1 -- 172.21.15.118:0/659686424 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- auth(proto 2 132 bytes epoch 0) v1 -- 0x7f81c4006290 con 0x5622651a13f0 2024-03-24T02:20:46.851 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T02:20:46.851+0000 7f81ee000640 1 -- 172.21.15.118:0/1693767697 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- auth(proto 2 132 bytes epoch 0) v1 -- 0x7f81c8006590 con 0x7f81f81052d0 2024-03-24T02:20:46.851 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T02:20:46.851+0000 7f81e4c00640 1 -- 172.21.15.118:0/659686424 <== mon.1 v2:172.21.15.193:3300/0 57 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 233+0+0 (secure 0 0 0) 0x7f81c4006290 con 0x5622651a13f0 2024-03-24T02:20:46.851 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T02:20:46.851+0000 7f81ef400640 1 -- 172.21.15.118:0/1693767697 <== mon.2 v2:172.21.15.189:3300/0 57 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 233+0+0 (secure 0 0 0) 0x7f81c8006590 con 0x7f81f81052d0 2024-03-24T02:20:48.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:20:48 smithi118 ceph-mon[31535]: pgmap v4330: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:20:48.703 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:20:48 smithi189 ceph-mon[36187]: pgmap v4330: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:20:48.710 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:20:48 smithi193 ceph-mon[36878]: pgmap v4330: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:20:50.695 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:20:50 smithi118 ceph-mon[31535]: pgmap v4331: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T02:20:50.703 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:20:50 smithi189 ceph-mon[36187]: pgmap v4331: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T02:20:50.710 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:20:50 smithi193 ceph-mon[36878]: pgmap v4331: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T02:20:51.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:20:51 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 6946.714744 secs 2024-03-24T02:20:51.703 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:20:51 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 6946.714744 secs 2024-03-24T02:20:51.710 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:20:51 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 6946.714744 secs 2024-03-24T02:20:52.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:20:52 smithi118 ceph-mon[31535]: pgmap v4332: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T02:20:52.703 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:20:52 smithi189 ceph-mon[36187]: pgmap v4332: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T02:20:52.710 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:20:52 smithi193 ceph-mon[36878]: pgmap v4332: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T02:20:54.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:20:54 smithi118 ceph-mon[31535]: pgmap v4333: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:20:54.703 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:20:54 smithi189 ceph-mon[36187]: pgmap v4333: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:20:54.710 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:20:54 smithi193 ceph-mon[36878]: pgmap v4333: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:20:56.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:20:56 smithi118 ceph-mon[31535]: pgmap v4334: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-24T02:20:56.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:20:56 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 6951.714829 secs 2024-03-24T02:20:56.703 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:20:56 smithi189 ceph-mon[36187]: pgmap v4334: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-24T02:20:56.703 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:20:56 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 6951.714829 secs 2024-03-24T02:20:56.710 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:20:56 smithi193 ceph-mon[36878]: pgmap v4334: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-24T02:20:56.710 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:20:56 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 6951.714829 secs 2024-03-24T02:20:58.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:20:58 smithi118 ceph-mon[31535]: pgmap v4335: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:20:58.703 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:20:58 smithi189 ceph-mon[36187]: pgmap v4335: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:20:58.710 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:20:58 smithi193 ceph-mon[36878]: pgmap v4335: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:21:00.695 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:21:00 smithi118 ceph-mon[31535]: pgmap v4336: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T02:21:00.703 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:21:00 smithi189 ceph-mon[36187]: pgmap v4336: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T02:21:00.710 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:21:00 smithi193 ceph-mon[36878]: pgmap v4336: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T02:21:01.695 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:21:01 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 6956.714892 secs 2024-03-24T02:21:01.703 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:21:01 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 6956.714892 secs 2024-03-24T02:21:01.710 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:21:01 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 6956.714892 secs 2024-03-24T02:21:02.695 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:21:02 smithi118 ceph-mon[31535]: pgmap v4337: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T02:21:02.703 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:21:02 smithi189 ceph-mon[36187]: pgmap v4337: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T02:21:02.710 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:21:02 smithi193 ceph-mon[36878]: pgmap v4337: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T02:21:04.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:21:04 smithi118 ceph-mon[31535]: pgmap v4338: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:21:04.703 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:21:04 smithi189 ceph-mon[36187]: pgmap v4338: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:21:04.710 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:21:04 smithi193 ceph-mon[36878]: pgmap v4338: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:21:06.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:21:06 smithi118 ceph-mon[31535]: pgmap v4339: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-24T02:21:06.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:21:06 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 6961.714955 secs 2024-03-24T02:21:06.703 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:21:06 smithi189 ceph-mon[36187]: pgmap v4339: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-24T02:21:06.703 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:21:06 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 6961.714955 secs 2024-03-24T02:21:06.710 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:21:06 smithi193 ceph-mon[36878]: pgmap v4339: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-24T02:21:06.710 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:21:06 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 6961.714955 secs 2024-03-24T02:21:08.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:21:08 smithi118 ceph-mon[31535]: pgmap v4340: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:21:08.703 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:21:08 smithi189 ceph-mon[36187]: pgmap v4340: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:21:08.710 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:21:08 smithi193 ceph-mon[36878]: pgmap v4340: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:21:10.697 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:21:10 smithi118 ceph-mon[31535]: pgmap v4341: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T02:21:10.703 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:21:10 smithi189 ceph-mon[36187]: pgmap v4341: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T02:21:10.710 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:21:10 smithi193 ceph-mon[36878]: pgmap v4341: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T02:21:11.604 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:21:11 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 6966.715010 secs 2024-03-24T02:21:11.611 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:21:11 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 6966.715010 secs 2024-03-24T02:21:11.695 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:21:11 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 6966.715010 secs 2024-03-24T02:21:12.695 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:21:12 smithi118 ceph-mon[31535]: pgmap v4342: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T02:21:12.703 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:21:12 smithi189 ceph-mon[36187]: pgmap v4342: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T02:21:12.710 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:21:12 smithi193 ceph-mon[36878]: pgmap v4342: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T02:21:14.695 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:21:14 smithi118 ceph-mon[31535]: pgmap v4343: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:21:14.703 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:21:14 smithi189 ceph-mon[36187]: pgmap v4343: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:21:14.710 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:21:14 smithi193 ceph-mon[36878]: pgmap v4343: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:21:16.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:21:16 smithi118 ceph-mon[31535]: pgmap v4344: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-24T02:21:16.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:21:16 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 6971.715092 secs 2024-03-24T02:21:16.703 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:21:16 smithi189 ceph-mon[36187]: pgmap v4344: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-24T02:21:16.703 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:21:16 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 6971.715092 secs 2024-03-24T02:21:16.710 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:21:16 smithi193 ceph-mon[36878]: pgmap v4344: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-24T02:21:16.710 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:21:16 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 6971.715092 secs 2024-03-24T02:21:18.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:21:18 smithi118 ceph-mon[31535]: pgmap v4345: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:21:18.703 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:21:18 smithi189 ceph-mon[36187]: pgmap v4345: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:21:18.710 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:21:18 smithi193 ceph-mon[36878]: pgmap v4345: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:21:20.695 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:21:20 smithi118 ceph-mon[31535]: pgmap v4346: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T02:21:20.703 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:21:20 smithi189 ceph-mon[36187]: pgmap v4346: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T02:21:20.710 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:21:20 smithi193 ceph-mon[36878]: pgmap v4346: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T02:21:21.695 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:21:21 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 6976.715153 secs 2024-03-24T02:21:21.703 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:21:21 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 6976.715153 secs 2024-03-24T02:21:21.710 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:21:21 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 6976.715153 secs 2024-03-24T02:21:22.695 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:21:22 smithi118 ceph-mon[31535]: pgmap v4347: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T02:21:22.703 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:21:22 smithi189 ceph-mon[36187]: pgmap v4347: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T02:21:22.710 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:21:22 smithi193 ceph-mon[36878]: pgmap v4347: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T02:21:24.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:21:24 smithi118 ceph-mon[31535]: pgmap v4348: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:21:24.703 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:21:24 smithi189 ceph-mon[36187]: pgmap v4348: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:21:24.710 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:21:24 smithi193 ceph-mon[36878]: pgmap v4348: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:21:26.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:21:26 smithi118 ceph-mon[31535]: pgmap v4349: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-24T02:21:26.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:21:26 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 6981.715216 secs 2024-03-24T02:21:26.703 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:21:26 smithi189 ceph-mon[36187]: pgmap v4349: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-24T02:21:26.703 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:21:26 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 6981.715216 secs 2024-03-24T02:21:26.710 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:21:26 smithi193 ceph-mon[36878]: pgmap v4349: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-24T02:21:26.710 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:21:26 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 6981.715216 secs 2024-03-24T02:21:27.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:21:27 smithi118 ceph-mon[31535]: pgmap v4350: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:21:27.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:21:27 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T02:21:27.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:21:27 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T02:21:27.703 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:21:27 smithi189 ceph-mon[36187]: pgmap v4350: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:21:27.703 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:21:27 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T02:21:27.703 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:21:27 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T02:21:27.710 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:21:27 smithi193 ceph-mon[36878]: pgmap v4350: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:21:27.710 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:21:27 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T02:21:27.710 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:21:27 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T02:21:28.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:21:28 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T02:21:28.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:21:28 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T02:21:28.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:21:28 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T02:21:28.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:21:28 smithi118 ceph-mon[31535]: pgmap v4351: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 6 op/s 2024-03-24T02:21:28.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:21:28 smithi118 ceph-mon[31535]: pgmap v4352: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 6 op/s 2024-03-24T02:21:28.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:21:28 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T02:21:28.703 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:21:28 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T02:21:28.703 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:21:28 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T02:21:28.703 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:21:28 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T02:21:28.703 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:21:28 smithi189 ceph-mon[36187]: pgmap v4351: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 6 op/s 2024-03-24T02:21:28.703 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:21:28 smithi189 ceph-mon[36187]: pgmap v4352: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 6 op/s 2024-03-24T02:21:28.703 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:21:28 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T02:21:28.710 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:21:28 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T02:21:28.710 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:21:28 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T02:21:28.710 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:21:28 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T02:21:28.710 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:21:28 smithi193 ceph-mon[36878]: pgmap v4351: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 6 op/s 2024-03-24T02:21:28.710 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:21:28 smithi193 ceph-mon[36878]: pgmap v4352: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 6 op/s 2024-03-24T02:21:28.710 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:21:28 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T02:21:31.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:21:31 smithi118 ceph-mon[31535]: pgmap v4353: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 6 op/s 2024-03-24T02:21:31.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:21:31 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 6986.715270 secs 2024-03-24T02:21:31.453 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:21:31 smithi189 ceph-mon[36187]: pgmap v4353: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 6 op/s 2024-03-24T02:21:31.453 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:21:31 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 6986.715270 secs 2024-03-24T02:21:31.460 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:21:31 smithi193 ceph-mon[36878]: pgmap v4353: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 6 op/s 2024-03-24T02:21:31.460 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:21:31 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 6986.715270 secs 2024-03-24T02:21:33.445 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:21:33 smithi118 ceph-mon[31535]: pgmap v4354: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 6 op/s 2024-03-24T02:21:33.453 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:21:33 smithi189 ceph-mon[36187]: pgmap v4354: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 6 op/s 2024-03-24T02:21:33.460 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:21:33 smithi193 ceph-mon[36878]: pgmap v4354: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 6 op/s 2024-03-24T02:21:35.453 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:21:35 smithi189 ceph-mon[36187]: pgmap v4355: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 6 op/s 2024-03-24T02:21:35.460 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:21:35 smithi193 ceph-mon[36878]: pgmap v4355: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 6 op/s 2024-03-24T02:21:35.695 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:21:35 smithi118 ceph-mon[31535]: pgmap v4355: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 6 op/s 2024-03-24T02:21:36.453 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:21:36 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 6991.715330 secs 2024-03-24T02:21:36.460 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:21:36 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 6991.715330 secs 2024-03-24T02:21:36.695 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:21:36 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 6991.715330 secs 2024-03-24T02:21:37.460 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:21:37 smithi193 ceph-mon[36878]: pgmap v4356: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 6 op/s 2024-03-24T02:21:37.695 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:21:37 smithi118 ceph-mon[31535]: pgmap v4356: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 6 op/s 2024-03-24T02:21:37.703 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:21:37 smithi189 ceph-mon[36187]: pgmap v4356: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 6 op/s 2024-03-24T02:21:39.695 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:21:39 smithi118 ceph-mon[31535]: pgmap v4357: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 5 op/s 2024-03-24T02:21:39.703 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:21:39 smithi189 ceph-mon[36187]: pgmap v4357: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 5 op/s 2024-03-24T02:21:39.710 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:21:39 smithi193 ceph-mon[36878]: pgmap v4357: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 5 op/s 2024-03-24T02:21:41.504 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:21:41 smithi189 ceph-mon[36187]: pgmap v4358: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T02:21:41.505 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:21:41 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 6996.715394 secs 2024-03-24T02:21:41.511 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:21:41 smithi193 ceph-mon[36878]: pgmap v4358: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T02:21:41.511 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:21:41 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 6996.715394 secs 2024-03-24T02:21:41.695 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:21:41 smithi118 ceph-mon[31535]: pgmap v4358: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T02:21:41.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:21:41 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 6996.715394 secs 2024-03-24T02:21:43.695 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:21:43 smithi118 ceph-mon[31535]: pgmap v4359: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:21:43.703 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:21:43 smithi189 ceph-mon[36187]: pgmap v4359: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:21:43.710 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:21:43 smithi193 ceph-mon[36878]: pgmap v4359: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:21:45.695 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:21:45 smithi118 ceph-mon[31535]: pgmap v4360: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-24T02:21:45.703 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:21:45 smithi189 ceph-mon[36187]: pgmap v4360: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-24T02:21:45.710 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:21:45 smithi193 ceph-mon[36878]: pgmap v4360: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-24T02:21:46.695 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:21:46 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 7001.715455 secs 2024-03-24T02:21:46.703 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:21:46 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 7001.715455 secs 2024-03-24T02:21:46.710 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:21:46 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 7001.715455 secs 2024-03-24T02:21:47.695 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:21:47 smithi118 ceph-mon[31535]: pgmap v4361: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:21:47.703 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:21:47 smithi189 ceph-mon[36187]: pgmap v4361: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:21:47.710 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:21:47 smithi193 ceph-mon[36878]: pgmap v4361: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:21:49.695 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:21:49 smithi118 ceph-mon[31535]: pgmap v4362: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T02:21:49.703 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:21:49 smithi189 ceph-mon[36187]: pgmap v4362: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T02:21:49.710 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:21:49 smithi193 ceph-mon[36878]: pgmap v4362: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T02:21:51.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:21:51 smithi118 ceph-mon[31535]: pgmap v4363: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T02:21:51.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:21:51 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 7006.715515 secs 2024-03-24T02:21:51.703 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:21:51 smithi189 ceph-mon[36187]: pgmap v4363: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T02:21:51.703 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:21:51 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 7006.715515 secs 2024-03-24T02:21:51.710 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:21:51 smithi193 ceph-mon[36878]: pgmap v4363: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T02:21:51.710 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:21:51 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 7006.715515 secs 2024-03-24T02:21:53.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:21:53 smithi118 ceph-mon[31535]: pgmap v4364: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:21:53.703 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:21:53 smithi189 ceph-mon[36187]: pgmap v4364: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:21:53.710 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:21:53 smithi193 ceph-mon[36878]: pgmap v4364: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:21:55.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:21:55 smithi118 ceph-mon[31535]: pgmap v4365: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-24T02:21:55.703 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:21:55 smithi189 ceph-mon[36187]: pgmap v4365: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-24T02:21:55.710 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:21:55 smithi193 ceph-mon[36878]: pgmap v4365: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-24T02:21:56.598 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:21:56 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 7011.715576 secs 2024-03-24T02:21:56.703 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:21:56 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 7011.715576 secs 2024-03-24T02:21:56.710 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:21:56 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 7011.715576 secs 2024-03-24T02:21:57.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:21:57 smithi118 ceph-mon[31535]: pgmap v4366: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:21:57.703 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:21:57 smithi189 ceph-mon[36187]: pgmap v4366: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:21:57.710 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:21:57 smithi193 ceph-mon[36878]: pgmap v4366: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:21:59.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:21:59 smithi118 ceph-mon[31535]: pgmap v4367: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T02:21:59.703 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:21:59 smithi189 ceph-mon[36187]: pgmap v4367: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T02:21:59.710 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:21:59 smithi193 ceph-mon[36878]: pgmap v4367: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T02:22:01.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:22:01 smithi118 ceph-mon[31535]: pgmap v4368: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T02:22:01.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:22:01 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 7016.715637 secs 2024-03-24T02:22:01.703 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:22:01 smithi189 ceph-mon[36187]: pgmap v4368: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T02:22:01.703 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:22:01 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 7016.715637 secs 2024-03-24T02:22:01.710 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:22:01 smithi193 ceph-mon[36878]: pgmap v4368: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T02:22:01.710 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:22:01 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 7016.715637 secs 2024-03-24T02:22:03.695 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:22:03 smithi118 ceph-mon[31535]: pgmap v4369: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:22:03.703 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:22:03 smithi189 ceph-mon[36187]: pgmap v4369: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:22:03.710 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:22:03 smithi193 ceph-mon[36878]: pgmap v4369: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:22:05.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:22:05 smithi118 ceph-mon[31535]: pgmap v4370: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-24T02:22:05.703 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:22:05 smithi189 ceph-mon[36187]: pgmap v4370: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-24T02:22:05.710 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:22:05 smithi193 ceph-mon[36878]: pgmap v4370: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-24T02:22:06.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:22:06 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 7021.715727 secs 2024-03-24T02:22:06.703 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:22:06 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 7021.715727 secs 2024-03-24T02:22:06.710 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:22:06 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 7021.715727 secs 2024-03-24T02:22:07.695 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:22:07 smithi118 ceph-mon[31535]: pgmap v4371: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:22:07.703 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:22:07 smithi189 ceph-mon[36187]: pgmap v4371: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:22:07.710 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:22:07 smithi193 ceph-mon[36878]: pgmap v4371: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:22:09.695 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:22:09 smithi118 ceph-mon[31535]: pgmap v4372: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-03-24T02:22:09.703 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:22:09 smithi189 ceph-mon[36187]: pgmap v4372: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-03-24T02:22:09.710 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:22:09 smithi193 ceph-mon[36878]: pgmap v4372: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-03-24T02:22:11.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:22:11 smithi118 ceph-mon[31535]: pgmap v4373: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 6 op/s 2024-03-24T02:22:11.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:22:11 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 7026.715811 secs 2024-03-24T02:22:11.703 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:22:11 smithi189 ceph-mon[36187]: pgmap v4373: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 6 op/s 2024-03-24T02:22:11.703 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:22:11 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 7026.715811 secs 2024-03-24T02:22:11.710 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:22:11 smithi193 ceph-mon[36878]: pgmap v4373: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 6 op/s 2024-03-24T02:22:11.710 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:22:11 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 7026.715811 secs 2024-03-24T02:22:13.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:22:13 smithi118 ceph-mon[31535]: pgmap v4374: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-03-24T02:22:13.703 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:22:13 smithi189 ceph-mon[36187]: pgmap v4374: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-03-24T02:22:13.710 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:22:13 smithi193 ceph-mon[36878]: pgmap v4374: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-03-24T02:22:15.695 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:22:15 smithi118 ceph-mon[31535]: pgmap v4375: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-24T02:22:15.703 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:22:15 smithi189 ceph-mon[36187]: pgmap v4375: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-24T02:22:15.710 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:22:15 smithi193 ceph-mon[36878]: pgmap v4375: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-24T02:22:16.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:22:16 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 7031.715869 secs 2024-03-24T02:22:16.703 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:22:16 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 7031.715869 secs 2024-03-24T02:22:16.710 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:22:16 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 7031.715869 secs 2024-03-24T02:22:16.865 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T02:22:16.866+0000 7f81db400640 1 -- 172.21.15.118:0/659686424 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f81c4006290 con 0x5622651a13f0 2024-03-24T02:22:16.865 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T02:22:16.866+0000 7f81ee000640 1 -- 172.21.15.118:0/1693767697 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f81c8006590 con 0x7f81f81052d0 2024-03-24T02:22:16.866 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T02:22:16.866+0000 7f81e4c00640 1 -- 172.21.15.118:0/659686424 <== mon.1 v2:172.21.15.193:3300/0 58 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 409+0+0 (secure 0 0 0) 0x7f81c4006290 con 0x5622651a13f0 2024-03-24T02:22:16.866 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T02:22:16.866+0000 7f81ef400640 1 -- 172.21.15.118:0/1693767697 <== mon.2 v2:172.21.15.189:3300/0 58 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (secure 0 0 0) 0x7f81c8006590 con 0x7f81f81052d0 2024-03-24T02:22:17.695 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:22:17 smithi118 ceph-mon[31535]: pgmap v4376: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-03-24T02:22:17.703 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:22:17 smithi189 ceph-mon[36187]: pgmap v4376: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-03-24T02:22:17.710 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:22:17 smithi193 ceph-mon[36878]: pgmap v4376: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-03-24T02:22:19.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:22:19 smithi118 ceph-mon[31535]: pgmap v4377: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-03-24T02:22:19.703 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:22:19 smithi189 ceph-mon[36187]: pgmap v4377: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-03-24T02:22:19.710 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:22:19 smithi193 ceph-mon[36878]: pgmap v4377: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-03-24T02:22:21.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:22:21 smithi118 ceph-mon[31535]: pgmap v4378: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T02:22:21.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:22:21 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 7036.715958 secs 2024-03-24T02:22:21.703 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:22:21 smithi189 ceph-mon[36187]: pgmap v4378: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T02:22:21.703 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:22:21 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 7036.715958 secs 2024-03-24T02:22:21.710 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:22:21 smithi193 ceph-mon[36878]: pgmap v4378: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T02:22:21.711 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:22:21 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 7036.715958 secs 2024-03-24T02:22:23.695 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:22:23 smithi118 ceph-mon[31535]: pgmap v4379: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:22:23.703 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:22:23 smithi189 ceph-mon[36187]: pgmap v4379: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:22:23.710 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:22:23 smithi193 ceph-mon[36878]: pgmap v4379: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:22:25.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:22:25 smithi118 ceph-mon[31535]: pgmap v4380: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-24T02:22:25.703 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:22:25 smithi189 ceph-mon[36187]: pgmap v4380: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-24T02:22:25.710 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:22:25 smithi193 ceph-mon[36878]: pgmap v4380: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-24T02:22:26.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:22:26 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 7041.716017 secs 2024-03-24T02:22:26.703 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:22:26 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 7041.716017 secs 2024-03-24T02:22:26.710 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:22:26 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 7041.716017 secs 2024-03-24T02:22:27.697 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:22:27 smithi118 ceph-mon[31535]: pgmap v4381: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-03-24T02:22:27.703 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:22:27 smithi189 ceph-mon[36187]: pgmap v4381: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-03-24T02:22:27.710 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:22:27 smithi193 ceph-mon[36878]: pgmap v4381: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-03-24T02:22:28.695 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:22:28 smithi118 ceph-mon[31535]: pgmap v4382: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-03-24T02:22:28.703 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:22:28 smithi189 ceph-mon[36187]: pgmap v4382: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-03-24T02:22:28.710 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:22:28 smithi193 ceph-mon[36878]: pgmap v4382: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-03-24T02:22:29.695 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:22:29 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T02:22:29.703 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:22:29 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T02:22:29.710 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:22:29 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T02:22:30.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:22:30 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T02:22:30.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:22:30 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T02:22:30.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:22:30 smithi118 ceph-mon[31535]: pgmap v4383: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 6 op/s 2024-03-24T02:22:30.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:22:30 smithi118 ceph-mon[31535]: pgmap v4384: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 6 op/s 2024-03-24T02:22:30.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:22:30 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T02:22:30.703 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:22:30 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T02:22:30.703 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:22:30 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T02:22:30.703 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:22:30 smithi189 ceph-mon[36187]: pgmap v4383: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 6 op/s 2024-03-24T02:22:30.703 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:22:30 smithi189 ceph-mon[36187]: pgmap v4384: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 6 op/s 2024-03-24T02:22:30.703 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:22:30 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T02:22:30.710 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:22:30 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T02:22:30.710 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:22:30 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T02:22:30.710 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:22:30 smithi193 ceph-mon[36878]: pgmap v4383: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 6 op/s 2024-03-24T02:22:30.710 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:22:30 smithi193 ceph-mon[36878]: pgmap v4384: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 6 op/s 2024-03-24T02:22:30.710 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:22:30 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T02:22:31.695 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:22:31 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 7046.716090 secs 2024-03-24T02:22:31.703 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:22:31 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 7046.716090 secs 2024-03-24T02:22:31.710 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:22:31 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 7046.716090 secs 2024-03-24T02:22:32.695 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:22:32 smithi118 ceph-mon[31535]: pgmap v4385: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 6 op/s 2024-03-24T02:22:32.703 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:22:32 smithi189 ceph-mon[36187]: pgmap v4385: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 6 op/s 2024-03-24T02:22:32.710 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:22:32 smithi193 ceph-mon[36878]: pgmap v4385: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 6 op/s 2024-03-24T02:22:34.953 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:22:34 smithi189 ceph-mon[36187]: pgmap v4386: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 4 op/s 2024-03-24T02:22:34.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:22:34 smithi193 ceph-mon[36878]: pgmap v4386: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 4 op/s 2024-03-24T02:22:35.195 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:22:34 smithi118 ceph-mon[31535]: pgmap v4386: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 4 op/s 2024-03-24T02:22:36.953 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:22:36 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 7051.716177 secs 2024-03-24T02:22:36.953 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:22:36 smithi189 ceph-mon[36187]: pgmap v4387: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-03-24T02:22:36.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:22:36 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 7051.716177 secs 2024-03-24T02:22:36.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:22:36 smithi193 ceph-mon[36878]: pgmap v4387: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-03-24T02:22:37.080 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:22:36 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 7051.716177 secs 2024-03-24T02:22:37.080 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:22:36 smithi118 ceph-mon[31535]: pgmap v4387: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-03-24T02:22:38.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:22:38 smithi193 ceph-mon[36878]: pgmap v4388: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 5 op/s 2024-03-24T02:22:39.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:22:38 smithi118 ceph-mon[31535]: pgmap v4388: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 5 op/s 2024-03-24T02:22:39.203 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:22:38 smithi189 ceph-mon[36187]: pgmap v4388: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 5 op/s 2024-03-24T02:22:41.005 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:22:40 smithi189 ceph-mon[36187]: pgmap v4389: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 5 op/s 2024-03-24T02:22:41.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:22:40 smithi118 ceph-mon[31535]: pgmap v4389: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 5 op/s 2024-03-24T02:22:41.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:22:40 smithi193 ceph-mon[36878]: pgmap v4389: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 5 op/s 2024-03-24T02:22:42.110 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:22:41 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 7056.716240 secs 2024-03-24T02:22:42.195 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:22:41 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 7056.716240 secs 2024-03-24T02:22:42.203 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:22:41 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 7056.716240 secs 2024-03-24T02:22:43.195 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:22:42 smithi118 ceph-mon[31535]: pgmap v4390: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:22:43.203 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:22:42 smithi189 ceph-mon[36187]: pgmap v4390: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:22:43.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:22:42 smithi193 ceph-mon[36878]: pgmap v4390: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:22:45.195 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:22:44 smithi118 ceph-mon[31535]: pgmap v4391: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T02:22:45.203 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:22:44 smithi189 ceph-mon[36187]: pgmap v4391: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T02:22:45.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:22:44 smithi193 ceph-mon[36878]: pgmap v4391: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T02:22:47.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:22:46 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 7061.716282 secs 2024-03-24T02:22:47.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:22:46 smithi118 ceph-mon[31535]: pgmap v4392: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T02:22:47.203 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:22:46 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 7061.716282 secs 2024-03-24T02:22:47.203 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:22:46 smithi189 ceph-mon[36187]: pgmap v4392: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T02:22:47.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:22:46 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 7061.716282 secs 2024-03-24T02:22:47.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:22:46 smithi193 ceph-mon[36878]: pgmap v4392: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T02:22:49.195 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:22:48 smithi118 ceph-mon[31535]: pgmap v4393: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T02:22:49.203 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:22:48 smithi189 ceph-mon[36187]: pgmap v4393: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T02:22:49.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:22:48 smithi193 ceph-mon[36878]: pgmap v4393: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T02:22:51.195 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:22:50 smithi118 ceph-mon[31535]: pgmap v4394: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-03-24T02:22:51.203 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:22:50 smithi189 ceph-mon[36187]: pgmap v4394: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-03-24T02:22:51.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:22:50 smithi193 ceph-mon[36878]: pgmap v4394: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-03-24T02:22:52.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:22:51 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 7066.716363 secs 2024-03-24T02:22:52.203 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:22:51 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 7066.716363 secs 2024-03-24T02:22:52.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:22:51 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 7066.716363 secs 2024-03-24T02:22:53.195 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:22:52 smithi118 ceph-mon[31535]: pgmap v4395: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:22:53.203 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:22:52 smithi189 ceph-mon[36187]: pgmap v4395: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:22:53.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:22:52 smithi193 ceph-mon[36878]: pgmap v4395: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:22:55.195 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:22:54 smithi118 ceph-mon[31535]: pgmap v4396: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T02:22:55.203 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:22:54 smithi189 ceph-mon[36187]: pgmap v4396: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T02:22:55.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:22:54 smithi193 ceph-mon[36878]: pgmap v4396: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T02:22:57.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:22:56 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 7071.716404 secs 2024-03-24T02:22:57.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:22:56 smithi118 ceph-mon[31535]: pgmap v4397: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T02:22:57.203 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:22:56 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 7071.716404 secs 2024-03-24T02:22:57.203 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:22:56 smithi189 ceph-mon[36187]: pgmap v4397: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T02:22:57.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:22:56 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 7071.716404 secs 2024-03-24T02:22:57.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:22:56 smithi193 ceph-mon[36878]: pgmap v4397: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T02:22:59.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:22:58 smithi118 ceph-mon[31535]: pgmap v4398: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T02:22:59.203 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:22:58 smithi189 ceph-mon[36187]: pgmap v4398: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T02:22:59.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:22:58 smithi193 ceph-mon[36878]: pgmap v4398: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T02:23:01.195 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:23:00 smithi118 ceph-mon[31535]: pgmap v4399: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-03-24T02:23:01.203 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:23:00 smithi189 ceph-mon[36187]: pgmap v4399: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-03-24T02:23:01.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:23:00 smithi193 ceph-mon[36878]: pgmap v4399: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-03-24T02:23:02.195 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:23:01 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 7076.716487 secs 2024-03-24T02:23:02.203 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:23:01 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 7076.716487 secs 2024-03-24T02:23:02.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:23:01 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 7076.716487 secs 2024-03-24T02:23:03.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:23:02 smithi118 ceph-mon[31535]: pgmap v4400: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:23:03.203 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:23:02 smithi189 ceph-mon[36187]: pgmap v4400: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:23:03.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:23:02 smithi193 ceph-mon[36878]: pgmap v4400: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:23:05.195 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:23:04 smithi118 ceph-mon[31535]: pgmap v4401: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T02:23:05.203 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:23:04 smithi189 ceph-mon[36187]: pgmap v4401: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T02:23:05.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:23:04 smithi193 ceph-mon[36878]: pgmap v4401: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T02:23:07.195 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:23:06 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 7081.716576 secs 2024-03-24T02:23:07.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:23:06 smithi118 ceph-mon[31535]: pgmap v4402: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T02:23:07.203 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:23:06 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 7081.716576 secs 2024-03-24T02:23:07.203 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:23:06 smithi189 ceph-mon[36187]: pgmap v4402: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T02:23:07.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:23:06 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 7081.716576 secs 2024-03-24T02:23:07.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:23:06 smithi193 ceph-mon[36878]: pgmap v4402: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T02:23:09.195 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:23:08 smithi118 ceph-mon[31535]: pgmap v4403: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T02:23:09.203 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:23:08 smithi189 ceph-mon[36187]: pgmap v4403: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T02:23:09.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:23:08 smithi193 ceph-mon[36878]: pgmap v4403: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T02:23:11.195 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:23:10 smithi118 ceph-mon[31535]: pgmap v4404: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-03-24T02:23:11.203 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:23:10 smithi189 ceph-mon[36187]: pgmap v4404: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-03-24T02:23:11.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:23:10 smithi193 ceph-mon[36878]: pgmap v4404: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-03-24T02:23:12.110 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:23:11 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 7086.716664 secs 2024-03-24T02:23:12.195 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:23:11 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 7086.716664 secs 2024-03-24T02:23:12.203 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:23:11 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 7086.716664 secs 2024-03-24T02:23:13.195 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:23:12 smithi118 ceph-mon[31535]: pgmap v4405: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:23:13.203 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:23:12 smithi189 ceph-mon[36187]: pgmap v4405: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:23:13.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:23:12 smithi193 ceph-mon[36878]: pgmap v4405: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:23:15.195 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:23:14 smithi118 ceph-mon[31535]: pgmap v4406: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T02:23:15.203 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:23:14 smithi189 ceph-mon[36187]: pgmap v4406: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T02:23:15.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:23:14 smithi193 ceph-mon[36878]: pgmap v4406: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T02:23:17.195 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:23:16 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 7091.716749 secs 2024-03-24T02:23:17.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:23:16 smithi118 ceph-mon[31535]: pgmap v4407: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T02:23:17.203 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:23:16 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 7091.716749 secs 2024-03-24T02:23:17.203 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:23:16 smithi189 ceph-mon[36187]: pgmap v4407: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T02:23:17.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:23:16 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 7091.716749 secs 2024-03-24T02:23:17.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:23:16 smithi193 ceph-mon[36878]: pgmap v4407: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T02:23:19.195 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:23:18 smithi118 ceph-mon[31535]: pgmap v4408: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T02:23:19.203 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:23:18 smithi189 ceph-mon[36187]: pgmap v4408: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T02:23:19.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:23:18 smithi193 ceph-mon[36878]: pgmap v4408: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T02:23:21.195 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:23:20 smithi118 ceph-mon[31535]: pgmap v4409: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T02:23:21.203 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:23:20 smithi189 ceph-mon[36187]: pgmap v4409: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T02:23:21.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:23:20 smithi193 ceph-mon[36878]: pgmap v4409: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T02:23:22.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:23:21 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 7096.716809 secs 2024-03-24T02:23:22.203 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:23:21 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 7096.716809 secs 2024-03-24T02:23:22.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:23:21 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 7096.716809 secs 2024-03-24T02:23:23.159 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:23:22 smithi118 ceph-mon[31535]: pgmap v4410: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:23:23.203 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:23:22 smithi189 ceph-mon[36187]: pgmap v4410: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:23:23.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:23:22 smithi193 ceph-mon[36878]: pgmap v4410: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:23:25.195 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:23:24 smithi118 ceph-mon[31535]: pgmap v4411: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T02:23:25.203 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:23:24 smithi189 ceph-mon[36187]: pgmap v4411: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T02:23:25.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:23:24 smithi193 ceph-mon[36878]: pgmap v4411: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T02:23:27.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:23:26 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 7101.716898 secs 2024-03-24T02:23:27.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:23:26 smithi118 ceph-mon[31535]: pgmap v4412: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T02:23:27.203 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:23:26 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 7101.716898 secs 2024-03-24T02:23:27.203 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:23:26 smithi189 ceph-mon[36187]: pgmap v4412: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T02:23:27.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:23:26 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 7101.716898 secs 2024-03-24T02:23:27.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:23:26 smithi193 ceph-mon[36878]: pgmap v4412: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T02:23:29.195 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:23:28 smithi118 ceph-mon[31535]: pgmap v4413: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T02:23:29.203 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:23:28 smithi189 ceph-mon[36187]: pgmap v4413: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T02:23:29.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:23:28 smithi193 ceph-mon[36878]: pgmap v4413: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T02:23:31.195 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:23:30 smithi118 ceph-mon[31535]: pgmap v4414: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-03-24T02:23:31.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:23:30 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T02:23:31.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:23:30 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 7106.716962 secs 2024-03-24T02:23:31.203 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:23:30 smithi189 ceph-mon[36187]: pgmap v4414: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-03-24T02:23:31.203 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:23:30 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T02:23:31.203 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:23:30 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 7106.716962 secs 2024-03-24T02:23:31.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:23:30 smithi193 ceph-mon[36878]: pgmap v4414: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-03-24T02:23:31.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:23:30 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T02:23:31.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:23:30 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 7106.716962 secs 2024-03-24T02:23:32.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:23:31 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T02:23:32.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:23:31 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T02:23:32.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:23:31 smithi118 ceph-mon[31535]: pgmap v4415: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 5 op/s 2024-03-24T02:23:32.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:23:31 smithi118 ceph-mon[31535]: pgmap v4416: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-03-24T02:23:32.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:23:31 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T02:23:32.203 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:23:31 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T02:23:32.203 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:23:31 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T02:23:32.203 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:23:31 smithi189 ceph-mon[36187]: pgmap v4415: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 5 op/s 2024-03-24T02:23:32.203 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:23:31 smithi189 ceph-mon[36187]: pgmap v4416: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-03-24T02:23:32.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:23:31 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T02:23:32.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:23:31 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T02:23:32.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:23:31 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T02:23:32.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:23:31 smithi193 ceph-mon[36878]: pgmap v4415: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 5 op/s 2024-03-24T02:23:32.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:23:31 smithi193 ceph-mon[36878]: pgmap v4416: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-03-24T02:23:32.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:23:31 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T02:23:34.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:23:34 smithi118 ceph-mon[31535]: pgmap v4417: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-03-24T02:23:34.703 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:23:34 smithi189 ceph-mon[36187]: pgmap v4417: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-03-24T02:23:34.710 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:23:34 smithi193 ceph-mon[36878]: pgmap v4417: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-03-24T02:23:36.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:23:36 smithi118 ceph-mon[31535]: pgmap v4418: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-03-24T02:23:36.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:23:36 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 7111.717019 secs 2024-03-24T02:23:36.703 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:23:36 smithi189 ceph-mon[36187]: pgmap v4418: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-03-24T02:23:36.703 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:23:36 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 7111.717019 secs 2024-03-24T02:23:36.710 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:23:36 smithi193 ceph-mon[36878]: pgmap v4418: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-03-24T02:23:36.710 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:23:36 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 7111.717019 secs 2024-03-24T02:23:38.695 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:23:38 smithi118 ceph-mon[31535]: pgmap v4419: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 5 op/s 2024-03-24T02:23:38.703 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:23:38 smithi189 ceph-mon[36187]: pgmap v4419: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 5 op/s 2024-03-24T02:23:38.710 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:23:38 smithi193 ceph-mon[36878]: pgmap v4419: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 5 op/s 2024-03-24T02:23:40.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:23:40 smithi118 ceph-mon[31535]: pgmap v4420: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-03-24T02:23:40.703 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:23:40 smithi189 ceph-mon[36187]: pgmap v4420: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-03-24T02:23:40.710 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:23:40 smithi193 ceph-mon[36878]: pgmap v4420: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-03-24T02:23:41.604 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:23:41 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 7116.717097 secs 2024-03-24T02:23:41.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:23:41 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 7116.717097 secs 2024-03-24T02:23:41.710 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:23:41 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 7116.717097 secs 2024-03-24T02:23:42.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:23:42 smithi118 ceph-mon[31535]: pgmap v4421: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 5 op/s 2024-03-24T02:23:42.611 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:23:42 smithi193 ceph-mon[36878]: pgmap v4421: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 5 op/s 2024-03-24T02:23:42.703 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:23:42 smithi189 ceph-mon[36187]: pgmap v4421: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 5 op/s 2024-03-24T02:23:44.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:23:44 smithi118 ceph-mon[31535]: pgmap v4422: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T02:23:44.703 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:23:44 smithi189 ceph-mon[36187]: pgmap v4422: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T02:23:44.710 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:23:44 smithi193 ceph-mon[36878]: pgmap v4422: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T02:23:46.695 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:23:46 smithi118 ceph-mon[31535]: pgmap v4423: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T02:23:46.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:23:46 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 7121.717158 secs 2024-03-24T02:23:46.703 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:23:46 smithi189 ceph-mon[36187]: pgmap v4423: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T02:23:46.703 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:23:46 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 7121.717158 secs 2024-03-24T02:23:46.710 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:23:46 smithi193 ceph-mon[36878]: pgmap v4423: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T02:23:46.710 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:23:46 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 7121.717158 secs 2024-03-24T02:23:48.695 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:23:48 smithi118 ceph-mon[31535]: pgmap v4424: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:23:48.703 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:23:48 smithi189 ceph-mon[36187]: pgmap v4424: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:23:48.710 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:23:48 smithi193 ceph-mon[36878]: pgmap v4424: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:23:50.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:23:50 smithi118 ceph-mon[31535]: pgmap v4425: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-24T02:23:50.703 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:23:50 smithi189 ceph-mon[36187]: pgmap v4425: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-24T02:23:50.710 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:23:50 smithi193 ceph-mon[36878]: pgmap v4425: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-24T02:23:51.695 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:23:51 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 7126.717171 secs 2024-03-24T02:23:51.703 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:23:51 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 7126.717171 secs 2024-03-24T02:23:51.710 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:23:51 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 7126.717171 secs 2024-03-24T02:23:52.695 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:23:52 smithi118 ceph-mon[31535]: pgmap v4426: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:23:52.703 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:23:52 smithi189 ceph-mon[36187]: pgmap v4426: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:23:52.710 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:23:52 smithi193 ceph-mon[36878]: pgmap v4426: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:23:54.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:23:54 smithi118 ceph-mon[31535]: pgmap v4427: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T02:23:54.703 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:23:54 smithi189 ceph-mon[36187]: pgmap v4427: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T02:23:54.710 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:23:54 smithi193 ceph-mon[36878]: pgmap v4427: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T02:23:56.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:23:56 smithi118 ceph-mon[31535]: pgmap v4428: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T02:23:56.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:23:56 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 7131.717187 secs 2024-03-24T02:23:56.703 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:23:56 smithi189 ceph-mon[36187]: pgmap v4428: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T02:23:56.703 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:23:56 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 7131.717187 secs 2024-03-24T02:23:56.710 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:23:56 smithi193 ceph-mon[36878]: pgmap v4428: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T02:23:56.710 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:23:56 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 7131.717187 secs 2024-03-24T02:23:58.695 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:23:58 smithi118 ceph-mon[31535]: pgmap v4429: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:23:58.703 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:23:58 smithi189 ceph-mon[36187]: pgmap v4429: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:23:58.710 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:23:58 smithi193 ceph-mon[36878]: pgmap v4429: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:24:00.695 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:24:00 smithi118 ceph-mon[31535]: pgmap v4430: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-24T02:24:00.703 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:24:00 smithi189 ceph-mon[36187]: pgmap v4430: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-24T02:24:00.710 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:24:00 smithi193 ceph-mon[36878]: pgmap v4430: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-24T02:24:01.695 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:24:01 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 7136.717271 secs 2024-03-24T02:24:01.703 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:24:01 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 7136.717271 secs 2024-03-24T02:24:01.710 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:24:01 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 7136.717271 secs 2024-03-24T02:24:02.695 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:24:02 smithi118 ceph-mon[31535]: pgmap v4431: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:24:02.703 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:24:02 smithi189 ceph-mon[36187]: pgmap v4431: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:24:02.710 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:24:02 smithi193 ceph-mon[36878]: pgmap v4431: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:24:04.695 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:24:04 smithi118 ceph-mon[31535]: pgmap v4432: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T02:24:04.703 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:24:04 smithi189 ceph-mon[36187]: pgmap v4432: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T02:24:04.710 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:24:04 smithi193 ceph-mon[36878]: pgmap v4432: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T02:24:06.695 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:24:06 smithi118 ceph-mon[31535]: pgmap v4433: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T02:24:06.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:24:06 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 7141.717333 secs 2024-03-24T02:24:06.703 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:24:06 smithi189 ceph-mon[36187]: pgmap v4433: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T02:24:06.703 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:24:06 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 7141.717333 secs 2024-03-24T02:24:06.710 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:24:06 smithi193 ceph-mon[36878]: pgmap v4433: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T02:24:06.710 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:24:06 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 7141.717333 secs 2024-03-24T02:24:08.695 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:24:08 smithi118 ceph-mon[31535]: pgmap v4434: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:24:08.703 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:24:08 smithi189 ceph-mon[36187]: pgmap v4434: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:24:08.710 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:24:08 smithi193 ceph-mon[36878]: pgmap v4434: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:24:10.695 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:24:10 smithi118 ceph-mon[31535]: pgmap v4435: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-24T02:24:10.703 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:24:10 smithi189 ceph-mon[36187]: pgmap v4435: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-24T02:24:10.710 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:24:10 smithi193 ceph-mon[36878]: pgmap v4435: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-24T02:24:11.604 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:24:11 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 7146.717395 secs 2024-03-24T02:24:11.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:24:11 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 7146.717395 secs 2024-03-24T02:24:11.710 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:24:11 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 7146.717395 secs 2024-03-24T02:24:12.610 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:24:12 smithi193 ceph-mon[36878]: pgmap v4436: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:24:12.695 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:24:12 smithi118 ceph-mon[31535]: pgmap v4436: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:24:12.703 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:24:12 smithi189 ceph-mon[36187]: pgmap v4436: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:24:14.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:24:14 smithi118 ceph-mon[31535]: pgmap v4437: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T02:24:14.703 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:24:14 smithi189 ceph-mon[36187]: pgmap v4437: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T02:24:14.710 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:24:14 smithi193 ceph-mon[36878]: pgmap v4437: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T02:24:16.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:24:16 smithi118 ceph-mon[31535]: pgmap v4438: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T02:24:16.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:24:16 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 7151.717484 secs 2024-03-24T02:24:16.703 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:24:16 smithi189 ceph-mon[36187]: pgmap v4438: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T02:24:16.703 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:24:16 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 7151.717484 secs 2024-03-24T02:24:16.710 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:24:16 smithi193 ceph-mon[36878]: pgmap v4438: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T02:24:16.710 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:24:16 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 7151.717484 secs 2024-03-24T02:24:18.695 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:24:18 smithi118 ceph-mon[31535]: pgmap v4439: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:24:18.703 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:24:18 smithi189 ceph-mon[36187]: pgmap v4439: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:24:18.710 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:24:18 smithi193 ceph-mon[36878]: pgmap v4439: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:24:20.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:24:20 smithi118 ceph-mon[31535]: pgmap v4440: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-24T02:24:20.703 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:24:20 smithi189 ceph-mon[36187]: pgmap v4440: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-24T02:24:20.710 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:24:20 smithi193 ceph-mon[36878]: pgmap v4440: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-24T02:24:21.695 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:24:21 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 7156.717568 secs 2024-03-24T02:24:21.703 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:24:21 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 7156.717568 secs 2024-03-24T02:24:21.710 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:24:21 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 7156.717568 secs 2024-03-24T02:24:22.695 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:24:22 smithi118 ceph-mon[31535]: pgmap v4441: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:24:22.703 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:24:22 smithi189 ceph-mon[36187]: pgmap v4441: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:24:22.710 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:24:22 smithi193 ceph-mon[36878]: pgmap v4441: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:24:24.695 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:24:24 smithi118 ceph-mon[31535]: pgmap v4442: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T02:24:24.703 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:24:24 smithi189 ceph-mon[36187]: pgmap v4442: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T02:24:24.710 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:24:24 smithi193 ceph-mon[36878]: pgmap v4442: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T02:24:26.695 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:24:26 smithi118 ceph-mon[31535]: pgmap v4443: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T02:24:26.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:24:26 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 7161.717656 secs 2024-03-24T02:24:26.703 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:24:26 smithi189 ceph-mon[36187]: pgmap v4443: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T02:24:26.703 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:24:26 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 7161.717656 secs 2024-03-24T02:24:26.710 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:24:26 smithi193 ceph-mon[36878]: pgmap v4443: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T02:24:26.710 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:24:26 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 7161.717656 secs 2024-03-24T02:24:28.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:24:28 smithi118 ceph-mon[31535]: pgmap v4444: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:24:28.703 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:24:28 smithi189 ceph-mon[36187]: pgmap v4444: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:24:28.710 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:24:28 smithi193 ceph-mon[36878]: pgmap v4444: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:24:30.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:24:30 smithi118 ceph-mon[31535]: pgmap v4445: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-24T02:24:30.703 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:24:30 smithi189 ceph-mon[36187]: pgmap v4445: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-24T02:24:30.710 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:24:30 smithi193 ceph-mon[36878]: pgmap v4445: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-24T02:24:31.695 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:24:31 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 7166.717716 secs 2024-03-24T02:24:31.703 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:24:31 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 7166.717716 secs 2024-03-24T02:24:31.710 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:24:31 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 7166.717716 secs 2024-03-24T02:24:32.695 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:24:32 smithi118 ceph-mon[31535]: pgmap v4446: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:24:32.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:24:32 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T02:24:32.703 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:24:32 smithi189 ceph-mon[36187]: pgmap v4446: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:24:32.703 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:24:32 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T02:24:32.710 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:24:32 smithi193 ceph-mon[36878]: pgmap v4446: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:24:32.710 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:24:32 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T02:24:33.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:24:33 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T02:24:33.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:24:33 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T02:24:33.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:24:33 smithi118 ceph-mon[31535]: pgmap v4447: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 5 op/s 2024-03-24T02:24:33.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:24:33 smithi118 ceph-mon[31535]: pgmap v4448: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 5 op/s 2024-03-24T02:24:33.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:24:33 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T02:24:33.703 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:24:33 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T02:24:33.703 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:24:33 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T02:24:33.703 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:24:33 smithi189 ceph-mon[36187]: pgmap v4447: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 5 op/s 2024-03-24T02:24:33.703 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:24:33 smithi189 ceph-mon[36187]: pgmap v4448: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 5 op/s 2024-03-24T02:24:33.703 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:24:33 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T02:24:33.710 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:24:33 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T02:24:33.710 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:24:33 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T02:24:33.710 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:24:33 smithi193 ceph-mon[36878]: pgmap v4447: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 5 op/s 2024-03-24T02:24:33.710 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:24:33 smithi193 ceph-mon[36878]: pgmap v4448: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 5 op/s 2024-03-24T02:24:33.710 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:24:33 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T02:24:36.195 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:24:35 smithi118 ceph-mon[31535]: pgmap v4449: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-03-24T02:24:36.203 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:24:35 smithi189 ceph-mon[36187]: pgmap v4449: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-03-24T02:24:36.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:24:35 smithi193 ceph-mon[36878]: pgmap v4449: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-03-24T02:24:37.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:24:36 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 7171.717778 secs 2024-03-24T02:24:37.203 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:24:36 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 7171.717778 secs 2024-03-24T02:24:37.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:24:36 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 7171.717778 secs 2024-03-24T02:24:38.195 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:24:37 smithi118 ceph-mon[31535]: pgmap v4450: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-03-24T02:24:38.203 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:24:37 smithi189 ceph-mon[36187]: pgmap v4450: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-03-24T02:24:38.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:24:37 smithi193 ceph-mon[36878]: pgmap v4450: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-03-24T02:24:40.195 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:24:39 smithi118 ceph-mon[31535]: pgmap v4451: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 4 op/s 2024-03-24T02:24:40.203 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:24:39 smithi189 ceph-mon[36187]: pgmap v4451: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 4 op/s 2024-03-24T02:24:40.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:24:39 smithi193 ceph-mon[36878]: pgmap v4451: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 4 op/s 2024-03-24T02:24:42.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:24:41 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 7176.717868 secs 2024-03-24T02:24:42.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:24:41 smithi118 ceph-mon[31535]: pgmap v4452: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-03-24T02:24:42.203 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:24:41 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 7176.717868 secs 2024-03-24T02:24:42.203 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:24:41 smithi189 ceph-mon[36187]: pgmap v4452: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-03-24T02:24:42.212 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:24:41 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 7176.717868 secs 2024-03-24T02:24:42.212 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:24:41 smithi193 ceph-mon[36878]: pgmap v4452: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-03-24T02:24:44.195 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:24:43 smithi118 ceph-mon[31535]: pgmap v4453: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 5 op/s 2024-03-24T02:24:44.203 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:24:43 smithi189 ceph-mon[36187]: pgmap v4453: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 5 op/s 2024-03-24T02:24:44.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:24:43 smithi193 ceph-mon[36878]: pgmap v4453: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 5 op/s 2024-03-24T02:24:46.195 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:24:45 smithi118 ceph-mon[31535]: pgmap v4454: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-03-24T02:24:46.203 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:24:45 smithi189 ceph-mon[36187]: pgmap v4454: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-03-24T02:24:46.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:24:45 smithi193 ceph-mon[36878]: pgmap v4454: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-03-24T02:24:47.195 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:24:46 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 7181.717956 secs 2024-03-24T02:24:47.203 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:24:46 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 7181.717956 secs 2024-03-24T02:24:47.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:24:46 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 7181.717956 secs 2024-03-24T02:24:48.195 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:24:47 smithi118 ceph-mon[31535]: pgmap v4455: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:24:48.203 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:24:47 smithi189 ceph-mon[36187]: pgmap v4455: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:24:48.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:24:47 smithi193 ceph-mon[36878]: pgmap v4455: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:24:50.195 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:24:49 smithi118 ceph-mon[31535]: pgmap v4456: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T02:24:50.203 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:24:49 smithi189 ceph-mon[36187]: pgmap v4456: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T02:24:50.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:24:49 smithi193 ceph-mon[36878]: pgmap v4456: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T02:24:52.195 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:24:51 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 7186.717999 secs 2024-03-24T02:24:52.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:24:51 smithi118 ceph-mon[31535]: pgmap v4457: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T02:24:52.203 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:24:51 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 7186.717999 secs 2024-03-24T02:24:52.203 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:24:51 smithi189 ceph-mon[36187]: pgmap v4457: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T02:24:52.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:24:51 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 7186.717999 secs 2024-03-24T02:24:52.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:24:51 smithi193 ceph-mon[36878]: pgmap v4457: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T02:24:54.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:24:53 smithi118 ceph-mon[31535]: pgmap v4458: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T02:24:54.203 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:24:53 smithi189 ceph-mon[36187]: pgmap v4458: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T02:24:54.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:24:53 smithi193 ceph-mon[36878]: pgmap v4458: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T02:24:56.195 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:24:55 smithi118 ceph-mon[31535]: pgmap v4459: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T02:24:56.203 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:24:55 smithi189 ceph-mon[36187]: pgmap v4459: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T02:24:56.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:24:55 smithi193 ceph-mon[36878]: pgmap v4459: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T02:24:57.195 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:24:56 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 7191.718083 secs 2024-03-24T02:24:57.203 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:24:56 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 7191.718083 secs 2024-03-24T02:24:57.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:24:56 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 7191.718083 secs 2024-03-24T02:24:58.098 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:24:57 smithi118 ceph-mon[31535]: pgmap v4460: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:24:58.203 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:24:57 smithi189 ceph-mon[36187]: pgmap v4460: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:24:58.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:24:57 smithi193 ceph-mon[36878]: pgmap v4460: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:25:00.195 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:24:59 smithi118 ceph-mon[31535]: pgmap v4461: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T02:25:00.203 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:24:59 smithi189 ceph-mon[36187]: pgmap v4461: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T02:25:00.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:24:59 smithi193 ceph-mon[36878]: pgmap v4461: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T02:25:02.195 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:25:01 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 7196.718142 secs 2024-03-24T02:25:02.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:25:01 smithi118 ceph-mon[31535]: pgmap v4462: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T02:25:02.203 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:25:01 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 7196.718142 secs 2024-03-24T02:25:02.203 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:25:01 smithi189 ceph-mon[36187]: pgmap v4462: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T02:25:02.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:25:01 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 7196.718142 secs 2024-03-24T02:25:02.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:25:01 smithi193 ceph-mon[36878]: pgmap v4462: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T02:25:04.195 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:25:03 smithi118 ceph-mon[31535]: pgmap v4463: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T02:25:04.203 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:25:03 smithi189 ceph-mon[36187]: pgmap v4463: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T02:25:04.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:25:03 smithi193 ceph-mon[36878]: pgmap v4463: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T02:25:06.195 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:25:05 smithi118 ceph-mon[31535]: pgmap v4464: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-03-24T02:25:06.203 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:25:05 smithi189 ceph-mon[36187]: pgmap v4464: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-03-24T02:25:06.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:25:05 smithi193 ceph-mon[36878]: pgmap v4464: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-03-24T02:25:07.195 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:25:06 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 7201.718204 secs 2024-03-24T02:25:07.203 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:25:06 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 7201.718204 secs 2024-03-24T02:25:07.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:25:06 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 7201.718204 secs 2024-03-24T02:25:08.195 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:25:07 smithi118 ceph-mon[31535]: pgmap v4465: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:25:08.203 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:25:07 smithi189 ceph-mon[36187]: pgmap v4465: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:25:08.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:25:07 smithi193 ceph-mon[36878]: pgmap v4465: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:25:10.195 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:25:09 smithi118 ceph-mon[31535]: pgmap v4466: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T02:25:10.203 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:25:09 smithi189 ceph-mon[36187]: pgmap v4466: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T02:25:10.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:25:09 smithi193 ceph-mon[36878]: pgmap v4466: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T02:25:12.195 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:25:11 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 7206.718289 secs 2024-03-24T02:25:12.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:25:11 smithi118 ceph-mon[31535]: pgmap v4467: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T02:25:12.203 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:25:11 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 7206.718289 secs 2024-03-24T02:25:12.203 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:25:11 smithi189 ceph-mon[36187]: pgmap v4467: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T02:25:12.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:25:11 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 7206.718289 secs 2024-03-24T02:25:12.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:25:11 smithi193 ceph-mon[36878]: pgmap v4467: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T02:25:14.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:25:13 smithi118 ceph-mon[31535]: pgmap v4468: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T02:25:14.203 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:25:13 smithi189 ceph-mon[36187]: pgmap v4468: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T02:25:14.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:25:13 smithi193 ceph-mon[36878]: pgmap v4468: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T02:25:15.758 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T02:25:15.758+0000 7f81f6000640 1 -- 172.21.15.118:0/1693767697 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7f81dc0a4320 msgr2=0x7f81dc0b05c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).tick idle (900000751) for more than 900000000 us, fault. 2024-03-24T02:25:15.758 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T02:25:15.758+0000 7f81f6000640 1 --2- 172.21.15.118:0/1693767697 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7f81dc0a4320 0x7f81dc0b05c0 secure :-1 s=READY pgs=542 cs=0 l=1 rev1=1 crypto rx=0x7f81e0010990 tx=0x7f81e0067040 comp rx=0 tx=0).stop 2024-03-24T02:25:15.758 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T02:25:15.758+0000 7f81ef400640 1 -- 172.21.15.118:0/1693767697 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7f81dc0a4320 msgr2=0x7f81dc0b05c0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-03-24T02:25:15.758 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T02:25:15.758+0000 7f81ef400640 1 --2- 172.21.15.118:0/1693767697 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7f81dc0a4320 0x7f81dc0b05c0 unknown :-1 s=CLOSED pgs=542 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T02:25:15.758 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T02:25:15.758+0000 7f81ef400640 1 --2- 172.21.15.118:0/1693767697 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7f81dc0a3bd0 0x7f81dc0b2b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T02:25:15.758 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T02:25:15.759+0000 7f81f6000640 1 --2- 172.21.15.118:0/1693767697 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7f81dc0a3bd0 0x7f81dc0b2b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T02:25:15.760 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T02:25:15.760+0000 7f81f6000640 1 --2- 172.21.15.118:0/1693767697 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7f81dc0a3bd0 0x7f81dc0b2b10 secure :-1 s=READY pgs=563 cs=0 l=1 rev1=1 crypto rx=0x7f81e00048d0 tx=0x7f81e0067040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T02:25:15.768 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T02:25:15.769+0000 7f81f6000640 1 -- 172.21.15.118:0/659686424 >> [v2:172.21.15.189:6838/4074484168,v1:172.21.15.189:6839/4074484168] conn(0x5622651b71e0 msgr2=0x562265215110 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).tick idle (900000751) for more than 900000000 us, fault. 2024-03-24T02:25:15.768 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T02:25:15.769+0000 7f81f6000640 1 --2- 172.21.15.118:0/659686424 >> [v2:172.21.15.189:6838/4074484168,v1:172.21.15.189:6839/4074484168] conn(0x5622651b71e0 0x562265215110 unknown :-1 s=READY pgs=561 cs=6 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault initiating reconnect 2024-03-24T02:25:15.768 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T02:25:15.769+0000 7f81f6000640 1 --2- 172.21.15.118:0/659686424 >> [v2:172.21.15.189:6838/4074484168,v1:172.21.15.189:6839/4074484168] conn(0x5622651b71e0 0x562265215110 unknown :-1 s=BANNER_CONNECTING pgs=561 cs=7 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T02:25:15.769 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T02:25:15.770+0000 7f81f6000640 1 --2- 172.21.15.118:0/659686424 >> [v2:172.21.15.189:6838/4074484168,v1:172.21.15.189:6839/4074484168] conn(0x5622651b71e0 0x562265215110 crc :-1 s=READY pgs=561 cs=7 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=76f3d534289740f0 server_cookie=b31a2335a59552de in_seq=0 out_seq=0 2024-03-24T02:25:15.769 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T02:25:15.770+0000 7f81e4c00640 10 client.27094 ms_handle_connect on v2:172.21.15.189:6838/4074484168 2024-03-24T02:25:16.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:25:15 smithi118 ceph-mon[31535]: pgmap v4469: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-03-24T02:25:16.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:25:15 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 7211.718396 secs 2024-03-24T02:25:16.203 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:25:15 smithi189 ceph-mon[36187]: pgmap v4469: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-03-24T02:25:16.203 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:25:15 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 7211.718396 secs 2024-03-24T02:25:16.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:25:15 smithi193 ceph-mon[36878]: pgmap v4469: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-03-24T02:25:16.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:25:15 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 7211.718396 secs 2024-03-24T02:25:16.895 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T02:25:16.895+0000 7f81ee000640 1 -- 172.21.15.118:0/1693767697 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f81c8006590 con 0x7f81f81052d0 2024-03-24T02:25:16.896 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T02:25:16.895+0000 7f81db400640 1 -- 172.21.15.118:0/659686424 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f81c4006290 con 0x5622651a13f0 2024-03-24T02:25:16.896 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T02:25:16.896+0000 7f81ef400640 1 -- 172.21.15.118:0/1693767697 <== mon.2 v2:172.21.15.189:3300/0 59 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (secure 0 0 0) 0x7f81c8006590 con 0x7f81f81052d0 2024-03-24T02:25:16.896 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T02:25:16.896+0000 7f81e4c00640 1 -- 172.21.15.118:0/659686424 <== mon.1 v2:172.21.15.193:3300/0 59 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 409+0+0 (secure 0 0 0) 0x7f81c4006290 con 0x5622651a13f0 2024-03-24T02:25:18.195 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:25:17 smithi118 ceph-mon[31535]: pgmap v4470: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:25:18.203 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:25:17 smithi189 ceph-mon[36187]: pgmap v4470: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:25:18.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:25:17 smithi193 ceph-mon[36878]: pgmap v4470: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:25:20.195 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:25:19 smithi118 ceph-mon[31535]: pgmap v4471: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T02:25:20.203 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:25:19 smithi189 ceph-mon[36187]: pgmap v4471: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T02:25:20.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:25:19 smithi193 ceph-mon[36878]: pgmap v4471: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T02:25:21.195 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:25:20 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 7216.718458 secs 2024-03-24T02:25:21.203 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:25:20 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 7216.718458 secs 2024-03-24T02:25:21.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:25:20 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 7216.718458 secs 2024-03-24T02:25:22.195 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:25:21 smithi118 ceph-mon[31535]: pgmap v4472: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T02:25:22.203 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:25:21 smithi189 ceph-mon[36187]: pgmap v4472: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T02:25:22.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:25:21 smithi193 ceph-mon[36878]: pgmap v4472: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T02:25:24.195 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:25:23 smithi118 ceph-mon[31535]: pgmap v4473: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T02:25:24.203 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:25:23 smithi189 ceph-mon[36187]: pgmap v4473: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T02:25:24.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:25:23 smithi193 ceph-mon[36878]: pgmap v4473: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T02:25:26.195 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:25:25 smithi118 ceph-mon[31535]: pgmap v4474: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-03-24T02:25:26.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:25:25 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 7221.718520 secs 2024-03-24T02:25:26.203 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:25:25 smithi189 ceph-mon[36187]: pgmap v4474: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-03-24T02:25:26.203 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:25:25 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 7221.718520 secs 2024-03-24T02:25:26.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:25:25 smithi193 ceph-mon[36878]: pgmap v4474: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-03-24T02:25:26.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:25:25 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 7221.718520 secs 2024-03-24T02:25:28.195 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:25:27 smithi118 ceph-mon[31535]: pgmap v4475: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:25:28.203 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:25:27 smithi189 ceph-mon[36187]: pgmap v4475: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:25:28.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:25:27 smithi193 ceph-mon[36878]: pgmap v4475: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:25:30.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:25:29 smithi118 ceph-mon[31535]: pgmap v4476: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T02:25:30.203 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:25:29 smithi189 ceph-mon[36187]: pgmap v4476: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T02:25:30.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:25:29 smithi193 ceph-mon[36878]: pgmap v4476: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T02:25:31.195 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:25:30 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 7226.718573 secs 2024-03-24T02:25:31.203 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:25:30 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 7226.718573 secs 2024-03-24T02:25:31.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:25:30 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 7226.718573 secs 2024-03-24T02:25:32.195 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:25:31 smithi118 ceph-mon[31535]: pgmap v4477: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T02:25:32.205 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:25:31 smithi189 ceph-mon[36187]: pgmap v4477: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T02:25:32.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:25:31 smithi193 ceph-mon[36878]: pgmap v4477: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T02:25:34.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:25:33 smithi118 ceph-mon[31535]: pgmap v4478: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T02:25:34.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:25:33 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T02:25:34.203 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:25:33 smithi189 ceph-mon[36187]: pgmap v4478: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T02:25:34.203 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:25:33 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T02:25:34.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:25:33 smithi193 ceph-mon[36878]: pgmap v4478: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T02:25:34.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:25:33 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T02:25:35.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:25:34 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T02:25:35.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:25:34 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T02:25:35.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:25:34 smithi118 ceph-mon[31535]: pgmap v4479: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 7 op/s 2024-03-24T02:25:35.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:25:34 smithi118 ceph-mon[31535]: pgmap v4480: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 6 op/s 2024-03-24T02:25:35.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:25:34 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T02:25:35.203 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:25:34 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T02:25:35.203 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:25:34 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T02:25:35.203 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:25:34 smithi189 ceph-mon[36187]: pgmap v4479: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 7 op/s 2024-03-24T02:25:35.203 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:25:34 smithi189 ceph-mon[36187]: pgmap v4480: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 6 op/s 2024-03-24T02:25:35.203 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:25:34 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T02:25:35.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:25:34 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T02:25:35.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:25:34 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T02:25:35.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:25:34 smithi193 ceph-mon[36878]: pgmap v4479: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 7 op/s 2024-03-24T02:25:35.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:25:34 smithi193 ceph-mon[36878]: pgmap v4480: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 6 op/s 2024-03-24T02:25:35.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:25:34 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T02:25:36.203 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:25:35 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 7231.718612 secs 2024-03-24T02:25:36.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:25:35 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 7231.718612 secs 2024-03-24T02:25:36.445 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:25:35 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 7231.718612 secs 2024-03-24T02:25:37.203 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:25:36 smithi189 ceph-mon[36187]: pgmap v4481: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 6 op/s 2024-03-24T02:25:37.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:25:36 smithi193 ceph-mon[36878]: pgmap v4481: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 6 op/s 2024-03-24T02:25:37.445 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:25:36 smithi118 ceph-mon[31535]: pgmap v4481: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 6 op/s 2024-03-24T02:25:39.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:25:39 smithi118 ceph-mon[31535]: pgmap v4482: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 6 op/s 2024-03-24T02:25:39.703 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:25:39 smithi189 ceph-mon[36187]: pgmap v4482: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 6 op/s 2024-03-24T02:25:39.710 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:25:39 smithi193 ceph-mon[36878]: pgmap v4482: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 6 op/s 2024-03-24T02:25:41.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:25:41 smithi118 ceph-mon[31535]: pgmap v4483: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 6 op/s 2024-03-24T02:25:41.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:25:41 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 7236.718626 secs 2024-03-24T02:25:41.703 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:25:41 smithi189 ceph-mon[36187]: pgmap v4483: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 6 op/s 2024-03-24T02:25:41.703 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:25:41 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 7236.718626 secs 2024-03-24T02:25:41.710 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:25:41 smithi193 ceph-mon[36878]: pgmap v4483: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 6 op/s 2024-03-24T02:25:41.710 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:25:41 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 7236.718626 secs 2024-03-24T02:25:43.695 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:25:43 smithi118 ceph-mon[31535]: pgmap v4484: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 5 op/s 2024-03-24T02:25:43.703 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:25:43 smithi189 ceph-mon[36187]: pgmap v4484: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 5 op/s 2024-03-24T02:25:43.710 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:25:43 smithi193 ceph-mon[36878]: pgmap v4484: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 5 op/s 2024-03-24T02:25:45.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:25:45 smithi118 ceph-mon[31535]: pgmap v4485: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 5 op/s 2024-03-24T02:25:45.703 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:25:45 smithi189 ceph-mon[36187]: pgmap v4485: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 5 op/s 2024-03-24T02:25:45.710 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:25:45 smithi193 ceph-mon[36878]: pgmap v4485: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 5 op/s 2024-03-24T02:25:46.695 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:25:46 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 7241.718685 secs 2024-03-24T02:25:46.703 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:25:46 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 7241.718685 secs 2024-03-24T02:25:46.710 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:25:46 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 7241.718685 secs 2024-03-24T02:25:47.695 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:25:47 smithi118 ceph-mon[31535]: pgmap v4486: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:25:47.703 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:25:47 smithi189 ceph-mon[36187]: pgmap v4486: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:25:47.710 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:25:47 smithi193 ceph-mon[36878]: pgmap v4486: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:25:49.695 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:25:49 smithi118 ceph-mon[31535]: pgmap v4487: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T02:25:49.703 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:25:49 smithi189 ceph-mon[36187]: pgmap v4487: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T02:25:49.710 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:25:49 smithi193 ceph-mon[36878]: pgmap v4487: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T02:25:51.695 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:25:51 smithi118 ceph-mon[31535]: pgmap v4488: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T02:25:51.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:25:51 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 7246.718769 secs 2024-03-24T02:25:51.703 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:25:51 smithi189 ceph-mon[36187]: pgmap v4488: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T02:25:51.703 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:25:51 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 7246.718769 secs 2024-03-24T02:25:51.710 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:25:51 smithi193 ceph-mon[36878]: pgmap v4488: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T02:25:51.710 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:25:51 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 7246.718769 secs 2024-03-24T02:25:53.695 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:25:53 smithi118 ceph-mon[31535]: pgmap v4489: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:25:53.703 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:25:53 smithi189 ceph-mon[36187]: pgmap v4489: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:25:53.710 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:25:53 smithi193 ceph-mon[36878]: pgmap v4489: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:25:55.695 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:25:55 smithi118 ceph-mon[31535]: pgmap v4490: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-24T02:25:55.703 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:25:55 smithi189 ceph-mon[36187]: pgmap v4490: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-24T02:25:55.710 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:25:55 smithi193 ceph-mon[36878]: pgmap v4490: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-24T02:25:56.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:25:56 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 7251.718854 secs 2024-03-24T02:25:56.703 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:25:56 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 7251.718854 secs 2024-03-24T02:25:56.710 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:25:56 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 7251.718854 secs 2024-03-24T02:25:57.695 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:25:57 smithi118 ceph-mon[31535]: pgmap v4491: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:25:57.703 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:25:57 smithi189 ceph-mon[36187]: pgmap v4491: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:25:57.710 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:25:57 smithi193 ceph-mon[36878]: pgmap v4491: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:25:59.695 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:25:59 smithi118 ceph-mon[31535]: pgmap v4492: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T02:25:59.703 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:25:59 smithi189 ceph-mon[36187]: pgmap v4492: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T02:25:59.710 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:25:59 smithi193 ceph-mon[36878]: pgmap v4492: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T02:26:01.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:26:01 smithi118 ceph-mon[31535]: pgmap v4493: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T02:26:01.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:26:01 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 7256.718937 secs 2024-03-24T02:26:01.703 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:26:01 smithi189 ceph-mon[36187]: pgmap v4493: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T02:26:01.703 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:26:01 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 7256.718937 secs 2024-03-24T02:26:01.710 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:26:01 smithi193 ceph-mon[36878]: pgmap v4493: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T02:26:01.710 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:26:01 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 7256.718937 secs 2024-03-24T02:26:03.695 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:26:03 smithi118 ceph-mon[31535]: pgmap v4494: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:26:03.703 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:26:03 smithi189 ceph-mon[36187]: pgmap v4494: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:26:03.710 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:26:03 smithi193 ceph-mon[36878]: pgmap v4494: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:26:05.695 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:26:05 smithi118 ceph-mon[31535]: pgmap v4495: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-24T02:26:05.703 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:26:05 smithi189 ceph-mon[36187]: pgmap v4495: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-24T02:26:05.710 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:26:05 smithi193 ceph-mon[36878]: pgmap v4495: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-24T02:26:06.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:26:06 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 7261.719022 secs 2024-03-24T02:26:06.703 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:26:06 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 7261.719022 secs 2024-03-24T02:26:06.710 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:26:06 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 7261.719022 secs 2024-03-24T02:26:07.695 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:26:07 smithi118 ceph-mon[31535]: pgmap v4496: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:26:07.703 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:26:07 smithi189 ceph-mon[36187]: pgmap v4496: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:26:07.710 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:26:07 smithi193 ceph-mon[36878]: pgmap v4496: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:26:08.695 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:26:08 smithi118 ceph-mon[31535]: pgmap v4497: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T02:26:08.703 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:26:08 smithi189 ceph-mon[36187]: pgmap v4497: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T02:26:08.710 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:26:08 smithi193 ceph-mon[36878]: pgmap v4497: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T02:26:11.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:26:11 smithi118 ceph-mon[31535]: pgmap v4498: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T02:26:11.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:26:11 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 7266.719034 secs 2024-03-24T02:26:11.703 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:26:11 smithi189 ceph-mon[36187]: pgmap v4498: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T02:26:11.703 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:26:11 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 7266.719034 secs 2024-03-24T02:26:11.710 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:26:11 smithi193 ceph-mon[36878]: pgmap v4498: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T02:26:11.710 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:26:11 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 7266.719034 secs 2024-03-24T02:26:13.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:26:13 smithi118 ceph-mon[31535]: pgmap v4499: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:26:13.703 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:26:13 smithi189 ceph-mon[36187]: pgmap v4499: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:26:13.710 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:26:13 smithi193 ceph-mon[36878]: pgmap v4499: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:26:15.445 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:26:15 smithi118 ceph-mon[31535]: pgmap v4500: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-24T02:26:15.703 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:26:15 smithi189 ceph-mon[36187]: pgmap v4500: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-24T02:26:15.710 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:26:15 smithi193 ceph-mon[36878]: pgmap v4500: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-24T02:26:16.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:26:16 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 7271.719119 secs 2024-03-24T02:26:16.703 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:26:16 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 7271.719119 secs 2024-03-24T02:26:16.710 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:26:16 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 7271.719119 secs 2024-03-24T02:26:17.695 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:26:17 smithi118 ceph-mon[31535]: pgmap v4501: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:26:17.703 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:26:17 smithi189 ceph-mon[36187]: pgmap v4501: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:26:17.710 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:26:17 smithi193 ceph-mon[36878]: pgmap v4501: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:26:19.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:26:19 smithi118 ceph-mon[31535]: pgmap v4502: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T02:26:19.703 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:26:19 smithi189 ceph-mon[36187]: pgmap v4502: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T02:26:19.710 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:26:19 smithi193 ceph-mon[36878]: pgmap v4502: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T02:26:21.695 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:26:21 smithi118 ceph-mon[31535]: pgmap v4503: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T02:26:21.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:26:21 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 7276.719180 secs 2024-03-24T02:26:21.703 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:26:21 smithi189 ceph-mon[36187]: pgmap v4503: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T02:26:21.703 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:26:21 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 7276.719180 secs 2024-03-24T02:26:21.710 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:26:21 smithi193 ceph-mon[36878]: pgmap v4503: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T02:26:21.710 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:26:21 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 7276.719180 secs 2024-03-24T02:26:23.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:26:23 smithi118 ceph-mon[31535]: pgmap v4504: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:26:23.703 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:26:23 smithi189 ceph-mon[36187]: pgmap v4504: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:26:23.710 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:26:23 smithi193 ceph-mon[36878]: pgmap v4504: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:26:25.695 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:26:25 smithi118 ceph-mon[31535]: pgmap v4505: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-24T02:26:25.703 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:26:25 smithi189 ceph-mon[36187]: pgmap v4505: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-24T02:26:25.710 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:26:25 smithi193 ceph-mon[36878]: pgmap v4505: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-24T02:26:26.695 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:26:26 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 7281.719242 secs 2024-03-24T02:26:26.703 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:26:26 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 7281.719242 secs 2024-03-24T02:26:26.710 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:26:26 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 7281.719242 secs 2024-03-24T02:26:27.695 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:26:27 smithi118 ceph-mon[31535]: pgmap v4506: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:26:27.703 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:26:27 smithi189 ceph-mon[36187]: pgmap v4506: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:26:27.710 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:26:27 smithi193 ceph-mon[36878]: pgmap v4506: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:26:29.695 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:26:29 smithi118 ceph-mon[31535]: pgmap v4507: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T02:26:29.703 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:26:29 smithi189 ceph-mon[36187]: pgmap v4507: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T02:26:29.710 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:26:29 smithi193 ceph-mon[36878]: pgmap v4507: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T02:26:31.697 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:26:31 smithi118 ceph-mon[31535]: pgmap v4508: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T02:26:31.697 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:26:31 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 7286.719302 secs 2024-03-24T02:26:31.703 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:26:31 smithi189 ceph-mon[36187]: pgmap v4508: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T02:26:31.703 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:26:31 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 7286.719302 secs 2024-03-24T02:26:31.710 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:26:31 smithi193 ceph-mon[36878]: pgmap v4508: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T02:26:31.710 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:26:31 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 7286.719302 secs 2024-03-24T02:26:33.695 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:26:33 smithi118 ceph-mon[31535]: pgmap v4509: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:26:33.703 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:26:33 smithi189 ceph-mon[36187]: pgmap v4509: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:26:33.710 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:26:33 smithi193 ceph-mon[36878]: pgmap v4509: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:26:35.695 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:26:35 smithi118 ceph-mon[31535]: pgmap v4510: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-24T02:26:35.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:26:35 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T02:26:35.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:26:35 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-24T02:26:35.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:26:35 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-24T02:26:35.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:26:35 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-24T02:26:35.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:26:35 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-03-24T02:26:35.703 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:26:35 smithi189 ceph-mon[36187]: pgmap v4510: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-24T02:26:35.703 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:26:35 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T02:26:35.703 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:26:35 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-24T02:26:35.703 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:26:35 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-24T02:26:35.704 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:26:35 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-24T02:26:35.704 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:26:35 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-03-24T02:26:35.710 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:26:35 smithi193 ceph-mon[36878]: pgmap v4510: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-24T02:26:35.710 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:26:35 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T02:26:35.710 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:26:35 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-24T02:26:35.710 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:26:35 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-24T02:26:35.711 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:26:35 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-24T02:26:35.711 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:26:35 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-03-24T02:26:36.697 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:26:36 smithi118 ceph-mon[31535]: Adjusting osd_memory_target on smithi189 to 174.1M 2024-03-24T02:26:36.697 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:26:36 smithi118 ceph-mon[31535]: Unable to set osd_memory_target on smithi189 to 182635622: error parsing value: Value '182635622' is below minimum 939524096 2024-03-24T02:26:36.697 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:26:36 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 7291.719388 secs 2024-03-24T02:26:36.697 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:26:36 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-24T02:26:36.697 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:26:36 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-24T02:26:36.697 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:26:36 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-24T02:26:36.697 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:26:36 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-03-24T02:26:36.697 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:26:36 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T02:26:36.697 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:26:36 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T02:26:36.698 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:26:36 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T02:26:36.703 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:26:36 smithi189 ceph-mon[36187]: Adjusting osd_memory_target on smithi189 to 174.1M 2024-03-24T02:26:36.703 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:26:36 smithi189 ceph-mon[36187]: Unable to set osd_memory_target on smithi189 to 182635622: error parsing value: Value '182635622' is below minimum 939524096 2024-03-24T02:26:36.703 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:26:36 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 7291.719388 secs 2024-03-24T02:26:36.704 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:26:36 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-24T02:26:36.704 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:26:36 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-24T02:26:36.704 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:26:36 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-24T02:26:36.704 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:26:36 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-03-24T02:26:36.704 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:26:36 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T02:26:36.704 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:26:36 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T02:26:36.704 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:26:36 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T02:26:36.710 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:26:36 smithi193 ceph-mon[36878]: Adjusting osd_memory_target on smithi189 to 174.1M 2024-03-24T02:26:36.710 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:26:36 smithi193 ceph-mon[36878]: Unable to set osd_memory_target on smithi189 to 182635622: error parsing value: Value '182635622' is below minimum 939524096 2024-03-24T02:26:36.710 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:26:36 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 7291.719388 secs 2024-03-24T02:26:36.710 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:26:36 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-24T02:26:36.711 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:26:36 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-24T02:26:36.711 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:26:36 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-24T02:26:36.711 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:26:36 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-03-24T02:26:36.711 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:26:36 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T02:26:36.711 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:26:36 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T02:26:36.711 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:26:36 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T02:26:37.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:26:37 smithi118 ceph-mon[31535]: Adjusting osd_memory_target on smithi118 to 176.7M 2024-03-24T02:26:37.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:26:37 smithi118 ceph-mon[31535]: Unable to set osd_memory_target on smithi118 to 185365196: error parsing value: Value '185365196' is below minimum 939524096 2024-03-24T02:26:37.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:26:37 smithi118 ceph-mon[31535]: pgmap v4511: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 5 op/s 2024-03-24T02:26:37.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:26:37 smithi118 ceph-mon[31535]: pgmap v4512: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-03-24T02:26:37.703 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:26:37 smithi189 ceph-mon[36187]: Adjusting osd_memory_target on smithi118 to 176.7M 2024-03-24T02:26:37.703 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:26:37 smithi189 ceph-mon[36187]: Unable to set osd_memory_target on smithi118 to 185365196: error parsing value: Value '185365196' is below minimum 939524096 2024-03-24T02:26:37.703 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:26:37 smithi189 ceph-mon[36187]: pgmap v4511: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 5 op/s 2024-03-24T02:26:37.703 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:26:37 smithi189 ceph-mon[36187]: pgmap v4512: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-03-24T02:26:37.710 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:26:37 smithi193 ceph-mon[36878]: Adjusting osd_memory_target on smithi118 to 176.7M 2024-03-24T02:26:37.710 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:26:37 smithi193 ceph-mon[36878]: Unable to set osd_memory_target on smithi118 to 185365196: error parsing value: Value '185365196' is below minimum 939524096 2024-03-24T02:26:37.710 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:26:37 smithi193 ceph-mon[36878]: pgmap v4511: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 5 op/s 2024-03-24T02:26:37.710 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:26:37 smithi193 ceph-mon[36878]: pgmap v4512: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-03-24T02:26:38.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:26:38 smithi118 ceph-mon[31535]: pgmap v4513: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 5 op/s 2024-03-24T02:26:38.953 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:26:38 smithi189 ceph-mon[36187]: pgmap v4513: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 5 op/s 2024-03-24T02:26:38.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:26:38 smithi193 ceph-mon[36878]: pgmap v4513: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 5 op/s 2024-03-24T02:26:41.197 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:26:40 smithi118 ceph-mon[31535]: pgmap v4514: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-03-24T02:26:41.203 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:26:40 smithi189 ceph-mon[36187]: pgmap v4514: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-03-24T02:26:41.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:26:40 smithi193 ceph-mon[36878]: pgmap v4514: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-03-24T02:26:42.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:26:41 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 7296.719443 secs 2024-03-24T02:26:42.203 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:26:41 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 7296.719443 secs 2024-03-24T02:26:42.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:26:41 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 7296.719443 secs 2024-03-24T02:26:43.114 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:26:42 smithi193 ceph-mon[36878]: pgmap v4515: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-03-24T02:26:43.195 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:26:42 smithi118 ceph-mon[31535]: pgmap v4515: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-03-24T02:26:43.203 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:26:42 smithi189 ceph-mon[36187]: pgmap v4515: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-03-24T02:26:45.195 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:26:44 smithi118 ceph-mon[31535]: pgmap v4516: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 4 op/s 2024-03-24T02:26:45.203 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:26:44 smithi189 ceph-mon[36187]: pgmap v4516: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 4 op/s 2024-03-24T02:26:45.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:26:44 smithi193 ceph-mon[36878]: pgmap v4516: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 4 op/s 2024-03-24T02:26:47.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:26:46 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 7301.719504 secs 2024-03-24T02:26:47.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:26:46 smithi118 ceph-mon[31535]: pgmap v4517: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 5 op/s 2024-03-24T02:26:47.203 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:26:46 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 7301.719504 secs 2024-03-24T02:26:47.203 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:26:46 smithi189 ceph-mon[36187]: pgmap v4517: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 5 op/s 2024-03-24T02:26:47.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:26:46 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 7301.719504 secs 2024-03-24T02:26:47.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:26:46 smithi193 ceph-mon[36878]: pgmap v4517: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 5 op/s 2024-03-24T02:26:49.195 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:26:48 smithi118 ceph-mon[31535]: pgmap v4518: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 5 op/s 2024-03-24T02:26:49.203 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:26:48 smithi189 ceph-mon[36187]: pgmap v4518: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 5 op/s 2024-03-24T02:26:49.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:26:48 smithi193 ceph-mon[36878]: pgmap v4518: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 5 op/s 2024-03-24T02:26:51.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:26:50 smithi118 ceph-mon[31535]: pgmap v4519: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T02:26:51.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:26:50 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 7306.719565 secs 2024-03-24T02:26:51.203 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:26:50 smithi189 ceph-mon[36187]: pgmap v4519: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T02:26:51.203 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:26:50 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 7306.719565 secs 2024-03-24T02:26:51.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:26:50 smithi193 ceph-mon[36878]: pgmap v4519: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T02:26:51.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:26:50 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 7306.719565 secs 2024-03-24T02:26:53.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:26:52 smithi118 ceph-mon[31535]: pgmap v4520: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:26:53.203 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:26:52 smithi189 ceph-mon[36187]: pgmap v4520: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:26:53.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:26:52 smithi193 ceph-mon[36878]: pgmap v4520: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:26:55.195 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:26:54 smithi118 ceph-mon[31535]: pgmap v4521: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T02:26:55.203 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:26:54 smithi189 ceph-mon[36187]: pgmap v4521: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T02:26:55.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:26:54 smithi193 ceph-mon[36878]: pgmap v4521: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T02:26:56.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:26:55 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 7311.719653 secs 2024-03-24T02:26:56.203 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:26:55 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 7311.719653 secs 2024-03-24T02:26:56.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:26:55 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 7311.719653 secs 2024-03-24T02:26:57.195 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:26:56 smithi118 ceph-mon[31535]: pgmap v4522: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T02:26:57.203 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:26:56 smithi189 ceph-mon[36187]: pgmap v4522: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T02:26:57.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:26:56 smithi193 ceph-mon[36878]: pgmap v4522: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T02:26:59.195 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:26:58 smithi118 ceph-mon[31535]: pgmap v4523: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 5 op/s 2024-03-24T02:26:59.203 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:26:58 smithi189 ceph-mon[36187]: pgmap v4523: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 5 op/s 2024-03-24T02:26:59.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:26:58 smithi193 ceph-mon[36878]: pgmap v4523: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 5 op/s 2024-03-24T02:27:01.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:27:00 smithi118 ceph-mon[31535]: pgmap v4524: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T02:27:01.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:27:00 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 7316.719742 secs 2024-03-24T02:27:01.203 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:27:00 smithi189 ceph-mon[36187]: pgmap v4524: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T02:27:01.203 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:27:00 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 7316.719742 secs 2024-03-24T02:27:01.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:27:00 smithi193 ceph-mon[36878]: pgmap v4524: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T02:27:01.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:27:00 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 7316.719742 secs 2024-03-24T02:27:03.195 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:27:02 smithi118 ceph-mon[31535]: pgmap v4525: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:27:03.203 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:27:02 smithi189 ceph-mon[36187]: pgmap v4525: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:27:03.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:27:02 smithi193 ceph-mon[36878]: pgmap v4525: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:27:05.195 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:27:04 smithi118 ceph-mon[31535]: pgmap v4526: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T02:27:05.203 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:27:04 smithi189 ceph-mon[36187]: pgmap v4526: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T02:27:05.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:27:04 smithi193 ceph-mon[36878]: pgmap v4526: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T02:27:06.195 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:27:05 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 7321.719831 secs 2024-03-24T02:27:06.203 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:27:05 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 7321.719831 secs 2024-03-24T02:27:06.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:27:05 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 7321.719831 secs 2024-03-24T02:27:07.195 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:27:06 smithi118 ceph-mon[31535]: pgmap v4527: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T02:27:07.203 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:27:06 smithi189 ceph-mon[36187]: pgmap v4527: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T02:27:07.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:27:06 smithi193 ceph-mon[36878]: pgmap v4527: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T02:27:09.203 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:27:08 smithi189 ceph-mon[36187]: pgmap v4528: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 5 op/s 2024-03-24T02:27:09.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:27:08 smithi193 ceph-mon[36878]: pgmap v4528: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 5 op/s 2024-03-24T02:27:09.445 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:27:08 smithi118 ceph-mon[31535]: pgmap v4528: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 5 op/s 2024-03-24T02:27:11.203 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:27:10 smithi189 ceph-mon[36187]: pgmap v4529: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T02:27:11.203 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:27:10 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 7326.719873 secs 2024-03-24T02:27:11.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:27:10 smithi193 ceph-mon[36878]: pgmap v4529: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T02:27:11.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:27:10 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 7326.719873 secs 2024-03-24T02:27:11.445 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:27:10 smithi118 ceph-mon[31535]: pgmap v4529: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T02:27:11.445 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:27:10 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 7326.719873 secs 2024-03-24T02:27:13.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:27:12 smithi193 ceph-mon[36878]: pgmap v4530: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:27:13.445 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:27:12 smithi118 ceph-mon[31535]: pgmap v4530: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:27:13.453 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:27:12 smithi189 ceph-mon[36187]: pgmap v4530: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:27:15.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:27:14 smithi193 ceph-mon[36878]: pgmap v4531: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T02:27:15.346 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:27:14 smithi189 ceph-mon[36187]: pgmap v4531: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T02:27:15.445 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:27:14 smithi118 ceph-mon[31535]: pgmap v4531: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T02:27:16.445 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:27:15 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 7331.719961 secs 2024-03-24T02:27:16.453 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:27:15 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 7331.719961 secs 2024-03-24T02:27:16.460 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:27:15 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 7331.719961 secs 2024-03-24T02:27:17.348 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:27:16 smithi118 ceph-mon[31535]: pgmap v4532: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T02:27:17.453 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:27:16 smithi189 ceph-mon[36187]: pgmap v4532: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T02:27:17.460 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:27:16 smithi193 ceph-mon[36878]: pgmap v4532: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T02:27:19.445 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:27:18 smithi118 ceph-mon[31535]: pgmap v4533: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 5 op/s 2024-03-24T02:27:19.453 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:27:18 smithi189 ceph-mon[36187]: pgmap v4533: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 5 op/s 2024-03-24T02:27:19.460 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:27:18 smithi193 ceph-mon[36878]: pgmap v4533: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 5 op/s 2024-03-24T02:27:21.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:27:20 smithi118 ceph-mon[31535]: pgmap v4534: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T02:27:21.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:27:20 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 7336.720049 secs 2024-03-24T02:27:21.453 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:27:20 smithi189 ceph-mon[36187]: pgmap v4534: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T02:27:21.453 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:27:20 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 7336.720049 secs 2024-03-24T02:27:21.460 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:27:20 smithi193 ceph-mon[36878]: pgmap v4534: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T02:27:21.460 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:27:20 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 7336.720049 secs 2024-03-24T02:27:23.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:27:22 smithi118 ceph-mon[31535]: pgmap v4535: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:27:23.453 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:27:22 smithi189 ceph-mon[36187]: pgmap v4535: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:27:23.460 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:27:22 smithi193 ceph-mon[36878]: pgmap v4535: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:27:25.203 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:27:24 smithi189 ceph-mon[36187]: pgmap v4536: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T02:27:25.445 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:27:24 smithi118 ceph-mon[31535]: pgmap v4536: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T02:27:25.460 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:27:24 smithi193 ceph-mon[36878]: pgmap v4536: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T02:27:26.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:27:25 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 7341.720092 secs 2024-03-24T02:27:26.453 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:27:25 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 7341.720092 secs 2024-03-24T02:27:26.460 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:27:25 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 7341.720092 secs 2024-03-24T02:27:27.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:27:26 smithi118 ceph-mon[31535]: pgmap v4537: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T02:27:27.453 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:27:26 smithi189 ceph-mon[36187]: pgmap v4537: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T02:27:27.460 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:27:26 smithi193 ceph-mon[36878]: pgmap v4537: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T02:27:29.445 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:27:29 smithi118 ceph-mon[31535]: pgmap v4538: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 5 op/s 2024-03-24T02:27:29.453 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:27:29 smithi189 ceph-mon[36187]: pgmap v4538: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 5 op/s 2024-03-24T02:27:29.460 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:27:29 smithi193 ceph-mon[36878]: pgmap v4538: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 5 op/s 2024-03-24T02:27:31.445 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:27:31 smithi118 ceph-mon[31535]: pgmap v4539: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T02:27:31.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:27:31 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 7346.720184 secs 2024-03-24T02:27:31.453 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:27:31 smithi189 ceph-mon[36187]: pgmap v4539: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T02:27:31.453 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:27:31 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 7346.720184 secs 2024-03-24T02:27:31.460 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:27:31 smithi193 ceph-mon[36878]: pgmap v4539: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T02:27:31.460 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:27:31 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 7346.720184 secs 2024-03-24T02:27:33.445 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:27:33 smithi118 ceph-mon[31535]: pgmap v4540: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:27:33.453 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:27:33 smithi189 ceph-mon[36187]: pgmap v4540: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:27:33.460 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:27:33 smithi193 ceph-mon[36878]: pgmap v4540: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:27:35.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:27:35 smithi118 ceph-mon[31535]: pgmap v4541: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T02:27:35.453 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:27:35 smithi189 ceph-mon[36187]: pgmap v4541: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T02:27:35.460 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:27:35 smithi193 ceph-mon[36878]: pgmap v4541: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T02:27:36.445 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:27:36 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 7351.720247 secs 2024-03-24T02:27:36.453 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:27:36 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 7351.720247 secs 2024-03-24T02:27:36.460 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:27:36 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 7351.720247 secs 2024-03-24T02:27:37.445 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:27:37 smithi118 ceph-mon[31535]: pgmap v4542: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T02:27:37.445 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:27:37 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T02:27:37.445 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:27:37 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-24T02:27:37.445 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:27:37 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-24T02:27:37.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:27:37 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-03-24T02:27:37.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:27:37 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-03-24T02:27:37.453 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:27:37 smithi189 ceph-mon[36187]: pgmap v4542: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T02:27:37.453 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:27:37 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T02:27:37.453 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:27:37 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-24T02:27:37.453 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:27:37 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-24T02:27:37.453 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:27:37 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-03-24T02:27:37.453 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:27:37 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-03-24T02:27:37.460 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:27:37 smithi193 ceph-mon[36878]: pgmap v4542: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T02:27:37.460 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:27:37 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T02:27:37.460 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:27:37 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-24T02:27:37.460 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:27:37 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-24T02:27:37.460 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:27:37 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-03-24T02:27:37.460 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:27:37 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-03-24T02:27:38.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:27:38 smithi118 ceph-mon[31535]: Adjusting osd_memory_target on smithi193 to 174.1M 2024-03-24T02:27:38.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:27:38 smithi118 ceph-mon[31535]: Unable to set osd_memory_target on smithi193 to 182639923: error parsing value: Value '182639923' is below minimum 939524096 2024-03-24T02:27:38.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:27:38 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T02:27:38.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:27:38 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T02:27:38.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:27:38 smithi118 ceph-mon[31535]: pgmap v4543: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 5 op/s 2024-03-24T02:27:38.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:27:38 smithi118 ceph-mon[31535]: pgmap v4544: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 5 op/s 2024-03-24T02:27:38.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:27:38 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T02:27:38.453 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:27:38 smithi189 ceph-mon[36187]: Adjusting osd_memory_target on smithi193 to 174.1M 2024-03-24T02:27:38.453 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:27:38 smithi189 ceph-mon[36187]: Unable to set osd_memory_target on smithi193 to 182639923: error parsing value: Value '182639923' is below minimum 939524096 2024-03-24T02:27:38.453 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:27:38 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T02:27:38.453 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:27:38 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T02:27:38.453 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:27:38 smithi189 ceph-mon[36187]: pgmap v4543: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 5 op/s 2024-03-24T02:27:38.453 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:27:38 smithi189 ceph-mon[36187]: pgmap v4544: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 5 op/s 2024-03-24T02:27:38.453 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:27:38 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T02:27:38.460 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:27:38 smithi193 ceph-mon[36878]: Adjusting osd_memory_target on smithi193 to 174.1M 2024-03-24T02:27:38.460 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:27:38 smithi193 ceph-mon[36878]: Unable to set osd_memory_target on smithi193 to 182639923: error parsing value: Value '182639923' is below minimum 939524096 2024-03-24T02:27:38.460 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:27:38 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T02:27:38.460 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:27:38 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T02:27:38.460 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:27:38 smithi193 ceph-mon[36878]: pgmap v4543: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 5 op/s 2024-03-24T02:27:38.460 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:27:38 smithi193 ceph-mon[36878]: pgmap v4544: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 5 op/s 2024-03-24T02:27:38.460 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:27:38 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T02:27:40.695 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:27:40 smithi118 ceph-mon[31535]: pgmap v4545: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 6 op/s 2024-03-24T02:27:40.703 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:27:40 smithi189 ceph-mon[36187]: pgmap v4545: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 6 op/s 2024-03-24T02:27:40.710 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:27:40 smithi193 ceph-mon[36878]: pgmap v4545: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 6 op/s 2024-03-24T02:27:41.695 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:27:41 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 7356.720308 secs 2024-03-24T02:27:41.703 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:27:41 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 7356.720308 secs 2024-03-24T02:27:41.710 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:27:41 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 7356.720308 secs 2024-03-24T02:27:42.695 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:27:42 smithi118 ceph-mon[31535]: pgmap v4546: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 6 op/s 2024-03-24T02:27:42.703 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:27:42 smithi189 ceph-mon[36187]: pgmap v4546: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 6 op/s 2024-03-24T02:27:42.710 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:27:42 smithi193 ceph-mon[36878]: pgmap v4546: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 6 op/s 2024-03-24T02:27:43.695 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:27:43 smithi118 ceph-mon[31535]: pgmap v4547: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 6 op/s 2024-03-24T02:27:43.703 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:27:43 smithi189 ceph-mon[36187]: pgmap v4547: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 6 op/s 2024-03-24T02:27:43.710 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:27:43 smithi193 ceph-mon[36878]: pgmap v4547: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 6 op/s 2024-03-24T02:27:46.695 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:27:46 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 7361.720324 secs 2024-03-24T02:27:46.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:27:46 smithi118 ceph-mon[31535]: pgmap v4548: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 6 op/s 2024-03-24T02:27:46.703 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:27:46 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 7361.720324 secs 2024-03-24T02:27:46.703 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:27:46 smithi189 ceph-mon[36187]: pgmap v4548: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 6 op/s 2024-03-24T02:27:46.710 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:27:46 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 7361.720324 secs 2024-03-24T02:27:46.710 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:27:46 smithi193 ceph-mon[36878]: pgmap v4548: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 6 op/s 2024-03-24T02:27:47.695 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:27:47 smithi118 ceph-mon[31535]: pgmap v4549: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T02:27:47.703 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:27:47 smithi189 ceph-mon[36187]: pgmap v4549: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T02:27:47.710 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:27:47 smithi193 ceph-mon[36878]: pgmap v4549: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T02:27:49.945 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:27:49 smithi118 ceph-mon[31535]: pgmap v4550: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-03-24T02:27:49.953 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:27:49 smithi189 ceph-mon[36187]: pgmap v4550: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-03-24T02:27:49.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:27:49 smithi193 ceph-mon[36878]: pgmap v4550: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-03-24T02:27:51.195 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:27:50 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 7366.720385 secs 2024-03-24T02:27:51.203 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:27:50 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 7366.720385 secs 2024-03-24T02:27:51.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:27:50 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 7366.720385 secs 2024-03-24T02:27:52.195 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:27:51 smithi118 ceph-mon[31535]: pgmap v4551: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:27:52.203 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:27:51 smithi189 ceph-mon[36187]: pgmap v4551: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:27:52.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:27:51 smithi193 ceph-mon[36878]: pgmap v4551: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:27:53.945 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:27:53 smithi118 ceph-mon[31535]: pgmap v4552: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T02:27:53.953 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:27:53 smithi189 ceph-mon[36187]: pgmap v4552: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T02:27:53.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:27:53 smithi193 ceph-mon[36878]: pgmap v4552: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T02:27:55.945 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:27:55 smithi118 ceph-mon[31535]: pgmap v4553: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T02:27:55.953 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:27:55 smithi189 ceph-mon[36187]: pgmap v4553: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T02:27:55.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:27:55 smithi193 ceph-mon[36878]: pgmap v4553: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T02:27:56.945 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:27:56 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 7371.720445 secs 2024-03-24T02:27:56.953 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:27:56 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 7371.720445 secs 2024-03-24T02:27:56.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:27:56 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 7371.720445 secs 2024-03-24T02:27:57.945 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:27:57 smithi118 ceph-mon[31535]: pgmap v4554: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T02:27:57.953 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:27:57 smithi189 ceph-mon[36187]: pgmap v4554: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T02:27:57.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:27:57 smithi193 ceph-mon[36878]: pgmap v4554: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T02:27:59.945 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:27:59 smithi118 ceph-mon[31535]: pgmap v4555: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-03-24T02:27:59.953 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:27:59 smithi189 ceph-mon[36187]: pgmap v4555: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-03-24T02:27:59.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:27:59 smithi193 ceph-mon[36878]: pgmap v4555: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-03-24T02:28:01.195 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:28:00 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 7376.720535 secs 2024-03-24T02:28:01.203 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:28:00 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 7376.720535 secs 2024-03-24T02:28:01.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:28:00 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 7376.720535 secs 2024-03-24T02:28:02.195 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:28:01 smithi118 ceph-mon[31535]: pgmap v4556: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:28:02.203 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:28:01 smithi189 ceph-mon[36187]: pgmap v4556: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:28:02.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:28:01 smithi193 ceph-mon[36878]: pgmap v4556: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:28:03.945 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:28:03 smithi118 ceph-mon[31535]: pgmap v4557: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T02:28:03.953 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:28:03 smithi189 ceph-mon[36187]: pgmap v4557: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T02:28:03.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:28:03 smithi193 ceph-mon[36878]: pgmap v4557: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T02:28:05.945 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:28:05 smithi118 ceph-mon[31535]: pgmap v4558: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T02:28:05.953 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:28:05 smithi189 ceph-mon[36187]: pgmap v4558: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T02:28:05.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:28:05 smithi193 ceph-mon[36878]: pgmap v4558: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T02:28:06.695 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:28:06 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 7381.720619 secs 2024-03-24T02:28:06.953 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:28:06 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 7381.720619 secs 2024-03-24T02:28:06.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:28:06 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 7381.720619 secs 2024-03-24T02:28:07.945 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:28:07 smithi118 ceph-mon[31535]: pgmap v4559: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T02:28:07.953 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:28:07 smithi189 ceph-mon[36187]: pgmap v4559: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T02:28:07.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:28:07 smithi193 ceph-mon[36878]: pgmap v4559: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T02:28:09.945 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:28:09 smithi118 ceph-mon[31535]: pgmap v4560: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-03-24T02:28:09.953 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:28:09 smithi189 ceph-mon[36187]: pgmap v4560: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-03-24T02:28:09.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:28:09 smithi193 ceph-mon[36878]: pgmap v4560: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-03-24T02:28:11.195 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:28:10 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 7386.720708 secs 2024-03-24T02:28:11.203 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:28:10 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 7386.720708 secs 2024-03-24T02:28:11.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:28:10 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 7386.720708 secs 2024-03-24T02:28:12.195 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:28:11 smithi118 ceph-mon[31535]: pgmap v4561: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:28:12.203 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:28:11 smithi189 ceph-mon[36187]: pgmap v4561: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:28:12.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:28:11 smithi193 ceph-mon[36878]: pgmap v4561: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:28:13.945 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:28:13 smithi118 ceph-mon[31535]: pgmap v4562: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-03-24T02:28:13.953 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:28:13 smithi189 ceph-mon[36187]: pgmap v4562: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-03-24T02:28:13.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:28:13 smithi193 ceph-mon[36878]: pgmap v4562: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-03-24T02:28:15.945 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:28:15 smithi118 ceph-mon[31535]: pgmap v4563: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 6 op/s 2024-03-24T02:28:15.953 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:28:15 smithi189 ceph-mon[36187]: pgmap v4563: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 6 op/s 2024-03-24T02:28:15.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:28:15 smithi193 ceph-mon[36878]: pgmap v4563: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 6 op/s 2024-03-24T02:28:16.925 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T02:28:16.925+0000 7f81ee000640 1 -- 172.21.15.118:0/1693767697 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f81c8006590 con 0x7f81f81052d0 2024-03-24T02:28:16.925 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T02:28:16.925+0000 7f81db400640 1 -- 172.21.15.118:0/659686424 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f81c4006290 con 0x5622651a13f0 2024-03-24T02:28:16.926 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T02:28:16.925+0000 7f81e4c00640 1 -- 172.21.15.118:0/659686424 <== mon.1 v2:172.21.15.193:3300/0 60 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 409+0+0 (secure 0 0 0) 0x7f81c4006290 con 0x5622651a13f0 2024-03-24T02:28:16.926 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T02:28:16.925+0000 7f81ef400640 1 -- 172.21.15.118:0/1693767697 <== mon.2 v2:172.21.15.189:3300/0 60 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (secure 0 0 0) 0x7f81c8006590 con 0x7f81f81052d0 2024-03-24T02:28:16.945 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:28:16 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 7391.720747 secs 2024-03-24T02:28:16.953 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:28:16 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 7391.720747 secs 2024-03-24T02:28:16.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:28:16 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 7391.720747 secs 2024-03-24T02:28:17.945 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:28:17 smithi118 ceph-mon[31535]: pgmap v4564: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 5 op/s 2024-03-24T02:28:17.953 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:28:17 smithi189 ceph-mon[36187]: pgmap v4564: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 5 op/s 2024-03-24T02:28:17.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:28:17 smithi193 ceph-mon[36878]: pgmap v4564: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 5 op/s 2024-03-24T02:28:19.945 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:28:19 smithi118 ceph-mon[31535]: pgmap v4565: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-03-24T02:28:19.953 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:28:19 smithi189 ceph-mon[36187]: pgmap v4565: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-03-24T02:28:19.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:28:19 smithi193 ceph-mon[36878]: pgmap v4565: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-03-24T02:28:21.195 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:28:20 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 7396.720806 secs 2024-03-24T02:28:21.203 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:28:20 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 7396.720806 secs 2024-03-24T02:28:21.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:28:20 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 7396.720806 secs 2024-03-24T02:28:22.195 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:28:21 smithi118 ceph-mon[31535]: pgmap v4566: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-03-24T02:28:22.203 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:28:21 smithi189 ceph-mon[36187]: pgmap v4566: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-03-24T02:28:22.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:28:21 smithi193 ceph-mon[36878]: pgmap v4566: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-03-24T02:28:23.945 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:28:23 smithi118 ceph-mon[31535]: pgmap v4567: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-03-24T02:28:23.953 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:28:23 smithi189 ceph-mon[36187]: pgmap v4567: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-03-24T02:28:23.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:28:23 smithi193 ceph-mon[36878]: pgmap v4567: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-03-24T02:28:25.945 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:28:25 smithi118 ceph-mon[31535]: pgmap v4568: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T02:28:25.953 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:28:25 smithi189 ceph-mon[36187]: pgmap v4568: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T02:28:25.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:28:25 smithi193 ceph-mon[36878]: pgmap v4568: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T02:28:26.945 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:28:26 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 7401.720867 secs 2024-03-24T02:28:26.953 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:28:26 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 7401.720867 secs 2024-03-24T02:28:26.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:28:26 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 7401.720867 secs 2024-03-24T02:28:27.945 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:28:27 smithi118 ceph-mon[31535]: pgmap v4569: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T02:28:27.953 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:28:27 smithi189 ceph-mon[36187]: pgmap v4569: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T02:28:27.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:28:27 smithi193 ceph-mon[36878]: pgmap v4569: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T02:28:29.945 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:28:29 smithi118 ceph-mon[31535]: pgmap v4570: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-03-24T02:28:29.953 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:28:29 smithi189 ceph-mon[36187]: pgmap v4570: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-03-24T02:28:29.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:28:29 smithi193 ceph-mon[36878]: pgmap v4570: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-03-24T02:28:31.195 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:28:30 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 7406.720930 secs 2024-03-24T02:28:31.203 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:28:30 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 7406.720930 secs 2024-03-24T02:28:31.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:28:30 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 7406.720930 secs 2024-03-24T02:28:32.195 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:28:31 smithi118 ceph-mon[31535]: pgmap v4571: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:28:32.203 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:28:31 smithi189 ceph-mon[36187]: pgmap v4571: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:28:32.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:28:31 smithi193 ceph-mon[36878]: pgmap v4571: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:28:33.945 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:28:33 smithi118 ceph-mon[31535]: pgmap v4572: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T02:28:33.953 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:28:33 smithi189 ceph-mon[36187]: pgmap v4572: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T02:28:33.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:28:33 smithi193 ceph-mon[36878]: pgmap v4572: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T02:28:35.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:28:35 smithi118 ceph-mon[31535]: pgmap v4573: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T02:28:35.953 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:28:35 smithi189 ceph-mon[36187]: pgmap v4573: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T02:28:35.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:28:35 smithi193 ceph-mon[36878]: pgmap v4573: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T02:28:36.945 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:28:36 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 7411.721019 secs 2024-03-24T02:28:36.953 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:28:36 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 7411.721019 secs 2024-03-24T02:28:36.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:28:36 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 7411.721019 secs 2024-03-24T02:28:37.945 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:28:37 smithi118 ceph-mon[31535]: pgmap v4574: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T02:28:37.953 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:28:37 smithi189 ceph-mon[36187]: pgmap v4574: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T02:28:37.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:28:37 smithi193 ceph-mon[36878]: pgmap v4574: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T02:28:38.945 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:28:38 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T02:28:38.953 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:28:38 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T02:28:38.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:28:38 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T02:28:39.945 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:28:39 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T02:28:39.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:28:39 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T02:28:39.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:28:39 smithi118 ceph-mon[31535]: pgmap v4575: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 6 op/s 2024-03-24T02:28:39.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:28:39 smithi118 ceph-mon[31535]: pgmap v4576: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 5 op/s 2024-03-24T02:28:39.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:28:39 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T02:28:39.953 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:28:39 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T02:28:39.953 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:28:39 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T02:28:39.953 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:28:39 smithi189 ceph-mon[36187]: pgmap v4575: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 6 op/s 2024-03-24T02:28:39.953 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:28:39 smithi189 ceph-mon[36187]: pgmap v4576: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 5 op/s 2024-03-24T02:28:39.953 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:28:39 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T02:28:39.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:28:39 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T02:28:39.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:28:39 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T02:28:39.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:28:39 smithi193 ceph-mon[36878]: pgmap v4575: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 6 op/s 2024-03-24T02:28:39.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:28:39 smithi193 ceph-mon[36878]: pgmap v4576: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 5 op/s 2024-03-24T02:28:39.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:28:39 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T02:28:41.195 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:28:40 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 7416.721092 secs 2024-03-24T02:28:41.203 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:28:40 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 7416.721092 secs 2024-03-24T02:28:41.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:28:40 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 7416.721092 secs 2024-03-24T02:28:42.195 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:28:41 smithi118 ceph-mon[31535]: pgmap v4577: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 6 op/s 2024-03-24T02:28:42.203 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:28:41 smithi189 ceph-mon[36187]: pgmap v4577: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 6 op/s 2024-03-24T02:28:42.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:28:41 smithi193 ceph-mon[36878]: pgmap v4577: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 6 op/s 2024-03-24T02:28:44.445 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:28:44 smithi118 ceph-mon[31535]: pgmap v4578: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 5 op/s 2024-03-24T02:28:44.453 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:28:44 smithi189 ceph-mon[36187]: pgmap v4578: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 5 op/s 2024-03-24T02:28:44.460 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:28:44 smithi193 ceph-mon[36878]: pgmap v4578: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 5 op/s 2024-03-24T02:28:46.445 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:28:46 smithi118 ceph-mon[31535]: pgmap v4579: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 6 op/s 2024-03-24T02:28:46.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:28:46 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 7421.721154 secs 2024-03-24T02:28:46.453 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:28:46 smithi189 ceph-mon[36187]: pgmap v4579: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 6 op/s 2024-03-24T02:28:46.453 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:28:46 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 7421.721154 secs 2024-03-24T02:28:46.460 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:28:46 smithi193 ceph-mon[36878]: pgmap v4579: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 6 op/s 2024-03-24T02:28:46.460 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:28:46 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 7421.721154 secs 2024-03-24T02:28:48.445 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:28:48 smithi118 ceph-mon[31535]: pgmap v4580: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 5 op/s 2024-03-24T02:28:48.453 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:28:48 smithi189 ceph-mon[36187]: pgmap v4580: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 5 op/s 2024-03-24T02:28:48.460 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:28:48 smithi193 ceph-mon[36878]: pgmap v4580: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 5 op/s 2024-03-24T02:28:50.329 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:28:50 smithi118 ceph-mon[31535]: pgmap v4581: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 5 op/s 2024-03-24T02:28:50.453 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:28:50 smithi189 ceph-mon[36187]: pgmap v4581: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 5 op/s 2024-03-24T02:28:50.460 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:28:50 smithi193 ceph-mon[36878]: pgmap v4581: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 5 op/s 2024-03-24T02:28:51.445 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:28:51 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 7426.721239 secs 2024-03-24T02:28:51.453 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:28:51 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 7426.721239 secs 2024-03-24T02:28:51.460 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:28:51 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 7426.721239 secs 2024-03-24T02:28:52.445 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:28:52 smithi118 ceph-mon[31535]: pgmap v4582: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T02:28:52.453 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:28:52 smithi189 ceph-mon[36187]: pgmap v4582: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T02:28:52.460 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:28:52 smithi193 ceph-mon[36878]: pgmap v4582: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T02:28:54.445 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:28:54 smithi118 ceph-mon[31535]: pgmap v4583: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 5 op/s 2024-03-24T02:28:54.453 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:28:54 smithi189 ceph-mon[36187]: pgmap v4583: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 5 op/s 2024-03-24T02:28:54.460 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:28:54 smithi193 ceph-mon[36878]: pgmap v4583: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 5 op/s 2024-03-24T02:28:56.445 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:28:56 smithi118 ceph-mon[31535]: pgmap v4584: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T02:28:56.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:28:56 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 7431.721322 secs 2024-03-24T02:28:56.453 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:28:56 smithi189 ceph-mon[36187]: pgmap v4584: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T02:28:56.453 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:28:56 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 7431.721322 secs 2024-03-24T02:28:56.460 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:28:56 smithi193 ceph-mon[36878]: pgmap v4584: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T02:28:56.460 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:28:56 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 7431.721322 secs 2024-03-24T02:28:58.445 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:28:58 smithi118 ceph-mon[31535]: pgmap v4585: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:28:58.453 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:28:58 smithi189 ceph-mon[36187]: pgmap v4585: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:28:58.462 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:28:58 smithi193 ceph-mon[36878]: pgmap v4585: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:29:00.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:29:00 smithi118 ceph-mon[31535]: pgmap v4586: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T02:29:00.453 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:29:00 smithi189 ceph-mon[36187]: pgmap v4586: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T02:29:00.460 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:29:00 smithi193 ceph-mon[36878]: pgmap v4586: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T02:29:01.445 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:29:01 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 7436.721382 secs 2024-03-24T02:29:01.453 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:29:01 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 7436.721382 secs 2024-03-24T02:29:01.460 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:29:01 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 7436.721382 secs 2024-03-24T02:29:01.933 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T02:29:01.933+0000 7f81ee000640 1 -- 172.21.15.118:0/1693767697 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- auth(proto 2 132 bytes epoch 0) v1 -- 0x7f81c8006590 con 0x7f81f81052d0 2024-03-24T02:29:01.933 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T02:29:01.933+0000 7f81db400640 1 -- 172.21.15.118:0/659686424 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- auth(proto 2 132 bytes epoch 0) v1 -- 0x7f81c4006290 con 0x5622651a13f0 2024-03-24T02:29:01.933 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T02:29:01.933+0000 7f81ef400640 1 -- 172.21.15.118:0/1693767697 <== mon.2 v2:172.21.15.189:3300/0 61 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 233+0+0 (secure 0 0 0) 0x7f81c8006590 con 0x7f81f81052d0 2024-03-24T02:29:01.933 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T02:29:01.933+0000 7f81e4c00640 1 -- 172.21.15.118:0/659686424 <== mon.1 v2:172.21.15.193:3300/0 61 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 233+0+0 (secure 0 0 0) 0x7f81c4006290 con 0x5622651a13f0 2024-03-24T02:29:02.445 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:29:02 smithi118 ceph-mon[31535]: pgmap v4587: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T02:29:02.453 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:29:02 smithi189 ceph-mon[36187]: pgmap v4587: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T02:29:02.460 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:29:02 smithi193 ceph-mon[36878]: pgmap v4587: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T02:29:04.445 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:29:04 smithi118 ceph-mon[31535]: pgmap v4588: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 5 op/s 2024-03-24T02:29:04.453 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:29:04 smithi189 ceph-mon[36187]: pgmap v4588: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 5 op/s 2024-03-24T02:29:04.460 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:29:04 smithi193 ceph-mon[36878]: pgmap v4588: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 5 op/s 2024-03-24T02:29:06.445 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:29:06 smithi118 ceph-mon[31535]: pgmap v4589: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T02:29:06.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:29:06 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 7441.721444 secs 2024-03-24T02:29:06.453 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:29:06 smithi189 ceph-mon[36187]: pgmap v4589: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T02:29:06.453 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:29:06 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 7441.721444 secs 2024-03-24T02:29:06.460 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:29:06 smithi193 ceph-mon[36878]: pgmap v4589: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T02:29:06.460 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:29:06 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 7441.721444 secs 2024-03-24T02:29:08.445 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:29:08 smithi118 ceph-mon[31535]: pgmap v4590: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:29:08.453 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:29:08 smithi189 ceph-mon[36187]: pgmap v4590: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:29:08.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:29:08 smithi193 ceph-mon[36878]: pgmap v4590: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:29:10.445 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:29:10 smithi118 ceph-mon[31535]: pgmap v4591: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T02:29:10.453 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:29:10 smithi189 ceph-mon[36187]: pgmap v4591: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T02:29:10.460 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:29:10 smithi193 ceph-mon[36878]: pgmap v4591: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T02:29:11.445 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:29:11 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 7446.721532 secs 2024-03-24T02:29:11.453 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:29:11 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 7446.721532 secs 2024-03-24T02:29:11.460 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:29:11 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 7446.721532 secs 2024-03-24T02:29:12.445 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:29:12 smithi118 ceph-mon[31535]: pgmap v4592: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T02:29:12.453 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:29:12 smithi189 ceph-mon[36187]: pgmap v4592: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T02:29:12.460 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:29:12 smithi193 ceph-mon[36878]: pgmap v4592: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T02:29:14.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:29:14 smithi118 ceph-mon[31535]: pgmap v4593: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 5 op/s 2024-03-24T02:29:14.453 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:29:14 smithi189 ceph-mon[36187]: pgmap v4593: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 5 op/s 2024-03-24T02:29:14.460 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:29:14 smithi193 ceph-mon[36878]: pgmap v4593: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 5 op/s 2024-03-24T02:29:16.445 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:29:16 smithi118 ceph-mon[31535]: pgmap v4594: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T02:29:16.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:29:16 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 7451.721618 secs 2024-03-24T02:29:16.453 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:29:16 smithi189 ceph-mon[36187]: pgmap v4594: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T02:29:16.453 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:29:16 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 7451.721618 secs 2024-03-24T02:29:16.460 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:29:16 smithi193 ceph-mon[36878]: pgmap v4594: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T02:29:16.460 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:29:16 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 7451.721618 secs 2024-03-24T02:29:18.445 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:29:18 smithi118 ceph-mon[31535]: pgmap v4595: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:29:18.453 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:29:18 smithi189 ceph-mon[36187]: pgmap v4595: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:29:18.460 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:29:18 smithi193 ceph-mon[36878]: pgmap v4595: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:29:20.445 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:29:20 smithi118 ceph-mon[31535]: pgmap v4596: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T02:29:20.453 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:29:20 smithi189 ceph-mon[36187]: pgmap v4596: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T02:29:20.460 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:29:20 smithi193 ceph-mon[36878]: pgmap v4596: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T02:29:21.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:29:21 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 7456.721706 secs 2024-03-24T02:29:21.453 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:29:21 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 7456.721706 secs 2024-03-24T02:29:21.460 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:29:21 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 7456.721706 secs 2024-03-24T02:29:22.347 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:29:22 smithi118 ceph-mon[31535]: pgmap v4597: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T02:29:22.453 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:29:22 smithi189 ceph-mon[36187]: pgmap v4597: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T02:29:22.460 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:29:22 smithi193 ceph-mon[36878]: pgmap v4597: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T02:29:24.445 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:29:24 smithi118 ceph-mon[31535]: pgmap v4598: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 5 op/s 2024-03-24T02:29:24.453 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:29:24 smithi189 ceph-mon[36187]: pgmap v4598: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 5 op/s 2024-03-24T02:29:24.460 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:29:24 smithi193 ceph-mon[36878]: pgmap v4598: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 5 op/s 2024-03-24T02:29:26.445 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:29:26 smithi118 ceph-mon[31535]: pgmap v4599: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T02:29:26.445 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:29:26 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 7461.721793 secs 2024-03-24T02:29:26.453 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:29:26 smithi189 ceph-mon[36187]: pgmap v4599: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T02:29:26.453 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:29:26 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 7461.721793 secs 2024-03-24T02:29:26.460 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:29:26 smithi193 ceph-mon[36878]: pgmap v4599: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T02:29:26.460 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:29:26 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 7461.721793 secs 2024-03-24T02:29:28.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:29:28 smithi118 ceph-mon[31535]: pgmap v4600: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:29:28.453 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:29:28 smithi189 ceph-mon[36187]: pgmap v4600: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:29:28.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:29:28 smithi193 ceph-mon[36878]: pgmap v4600: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:29:30.445 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:29:30 smithi118 ceph-mon[31535]: pgmap v4601: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T02:29:30.453 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:29:30 smithi189 ceph-mon[36187]: pgmap v4601: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T02:29:30.460 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:29:30 smithi193 ceph-mon[36878]: pgmap v4601: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T02:29:31.453 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:29:31 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 7466.721829 secs 2024-03-24T02:29:31.460 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:29:31 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 7466.721829 secs 2024-03-24T02:29:31.695 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:29:31 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 7466.721829 secs 2024-03-24T02:29:32.453 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:29:32 smithi189 ceph-mon[36187]: pgmap v4602: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T02:29:32.460 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:29:32 smithi193 ceph-mon[36878]: pgmap v4602: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T02:29:32.695 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:29:32 smithi118 ceph-mon[31535]: pgmap v4602: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T02:29:34.460 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:29:34 smithi193 ceph-mon[36878]: pgmap v4603: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 5 op/s 2024-03-24T02:29:34.498 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:29:34 smithi118 ceph-mon[31535]: pgmap v4603: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 5 op/s 2024-03-24T02:29:34.703 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:29:34 smithi189 ceph-mon[36187]: pgmap v4603: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 5 op/s 2024-03-24T02:29:36.460 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:29:36 smithi193 ceph-mon[36878]: pgmap v4604: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T02:29:36.460 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:29:36 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 7471.721869 secs 2024-03-24T02:29:36.695 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:29:36 smithi118 ceph-mon[31535]: pgmap v4604: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T02:29:36.695 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:29:36 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 7471.721869 secs 2024-03-24T02:29:36.703 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:29:36 smithi189 ceph-mon[36187]: pgmap v4604: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T02:29:36.703 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:29:36 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 7471.721869 secs 2024-03-24T02:29:38.695 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:29:38 smithi118 ceph-mon[31535]: pgmap v4605: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:29:38.703 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:29:38 smithi189 ceph-mon[36187]: pgmap v4605: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:29:38.710 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:29:38 smithi193 ceph-mon[36878]: pgmap v4605: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:29:40.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:29:40 smithi118 ceph-mon[31535]: pgmap v4606: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T02:29:40.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:29:40 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T02:29:40.703 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:29:40 smithi189 ceph-mon[36187]: pgmap v4606: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T02:29:40.703 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:29:40 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T02:29:40.710 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:29:40 smithi193 ceph-mon[36878]: pgmap v4606: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T02:29:40.710 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:29:40 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T02:29:41.695 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:29:41 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 7476.721882 secs 2024-03-24T02:29:41.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:29:41 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T02:29:41.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:29:41 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T02:29:41.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:29:41 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T02:29:41.703 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:29:41 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 7476.721882 secs 2024-03-24T02:29:41.703 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:29:41 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T02:29:41.703 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:29:41 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T02:29:41.703 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:29:41 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T02:29:41.710 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:29:41 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 7476.721882 secs 2024-03-24T02:29:41.710 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:29:41 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T02:29:41.710 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:29:41 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T02:29:41.710 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:29:41 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T02:29:42.695 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:29:42 smithi118 ceph-mon[31535]: pgmap v4607: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 6 op/s 2024-03-24T02:29:42.695 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:29:42 smithi118 ceph-mon[31535]: pgmap v4608: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-03-24T02:29:42.703 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:29:42 smithi189 ceph-mon[36187]: pgmap v4607: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 6 op/s 2024-03-24T02:29:42.703 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:29:42 smithi189 ceph-mon[36187]: pgmap v4608: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-03-24T02:29:42.710 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:29:42 smithi193 ceph-mon[36878]: pgmap v4607: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 6 op/s 2024-03-24T02:29:42.710 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:29:42 smithi193 ceph-mon[36878]: pgmap v4608: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-03-24T02:29:44.505 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:29:44 smithi189 ceph-mon[36187]: pgmap v4609: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 5 op/s 2024-03-24T02:29:44.512 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:29:44 smithi193 ceph-mon[36878]: pgmap v4609: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 5 op/s 2024-03-24T02:29:44.695 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:29:44 smithi118 ceph-mon[31535]: pgmap v4609: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 5 op/s 2024-03-24T02:29:46.695 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:29:46 smithi118 ceph-mon[31535]: pgmap v4610: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-03-24T02:29:46.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:29:46 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 7481.721938 secs 2024-03-24T02:29:46.703 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:29:46 smithi189 ceph-mon[36187]: pgmap v4610: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-03-24T02:29:46.703 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:29:46 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 7481.721938 secs 2024-03-24T02:29:46.710 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:29:46 smithi193 ceph-mon[36878]: pgmap v4610: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-03-24T02:29:46.710 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:29:46 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 7481.721938 secs 2024-03-24T02:29:48.695 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:29:48 smithi118 ceph-mon[31535]: pgmap v4611: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-03-24T02:29:48.703 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:29:48 smithi189 ceph-mon[36187]: pgmap v4611: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-03-24T02:29:48.710 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:29:48 smithi193 ceph-mon[36878]: pgmap v4611: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-03-24T02:29:50.695 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:29:50 smithi118 ceph-mon[31535]: pgmap v4612: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 5 op/s 2024-03-24T02:29:50.703 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:29:50 smithi189 ceph-mon[36187]: pgmap v4612: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 5 op/s 2024-03-24T02:29:50.710 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:29:50 smithi193 ceph-mon[36878]: pgmap v4612: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 5 op/s 2024-03-24T02:29:51.695 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:29:51 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 7486.721978 secs 2024-03-24T02:29:51.703 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:29:51 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 7486.721978 secs 2024-03-24T02:29:51.710 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:29:51 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 7486.721978 secs 2024-03-24T02:29:52.597 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:29:52 smithi118 ceph-mon[31535]: pgmap v4613: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 5 op/s 2024-03-24T02:29:52.703 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:29:52 smithi189 ceph-mon[36187]: pgmap v4613: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 5 op/s 2024-03-24T02:29:52.710 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:29:52 smithi193 ceph-mon[36878]: pgmap v4613: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 5 op/s 2024-03-24T02:29:54.695 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:29:54 smithi118 ceph-mon[31535]: pgmap v4614: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T02:29:54.703 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:29:54 smithi189 ceph-mon[36187]: pgmap v4614: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T02:29:54.710 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:29:54 smithi193 ceph-mon[36878]: pgmap v4614: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T02:29:56.695 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:29:56 smithi118 ceph-mon[31535]: pgmap v4615: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-03-24T02:29:56.695 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:29:56 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 7491.722021 secs 2024-03-24T02:29:56.703 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:29:56 smithi189 ceph-mon[36187]: pgmap v4615: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-03-24T02:29:56.703 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:29:56 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 7491.722021 secs 2024-03-24T02:29:56.710 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:29:56 smithi193 ceph-mon[36878]: pgmap v4615: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-03-24T02:29:56.710 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:29:56 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 7491.722021 secs 2024-03-24T02:29:58.695 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:29:58 smithi118 ceph-mon[31535]: pgmap v4616: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:29:58.703 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:29:58 smithi189 ceph-mon[36187]: pgmap v4616: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:29:58.710 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:29:58 smithi193 ceph-mon[36878]: pgmap v4616: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:30:00.695 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:30:00 smithi118 ceph-mon[31535]: pgmap v4617: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T02:30:00.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:30:00 smithi118 ceph-mon[31535]: overall HEALTH_WARN 6 failed cephadm daemon(s); 1 MDSs report slow requests 2024-03-24T02:30:00.703 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:30:00 smithi189 ceph-mon[36187]: pgmap v4617: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T02:30:00.703 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:30:00 smithi189 ceph-mon[36187]: overall HEALTH_WARN 6 failed cephadm daemon(s); 1 MDSs report slow requests 2024-03-24T02:30:00.710 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:30:00 smithi193 ceph-mon[36878]: pgmap v4617: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T02:30:00.710 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:30:00 smithi193 ceph-mon[36878]: overall HEALTH_WARN 6 failed cephadm daemon(s); 1 MDSs report slow requests 2024-03-24T02:30:01.695 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:30:01 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 7496.722092 secs 2024-03-24T02:30:01.703 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:30:01 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 7496.722092 secs 2024-03-24T02:30:01.710 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:30:01 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 7496.722092 secs 2024-03-24T02:30:02.695 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:30:02 smithi118 ceph-mon[31535]: pgmap v4618: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 6 op/s 2024-03-24T02:30:02.703 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:30:02 smithi189 ceph-mon[36187]: pgmap v4618: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 6 op/s 2024-03-24T02:30:02.710 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:30:02 smithi193 ceph-mon[36878]: pgmap v4618: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 6 op/s 2024-03-24T02:30:04.695 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:30:04 smithi118 ceph-mon[31535]: pgmap v4619: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 5 op/s 2024-03-24T02:30:04.703 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:30:04 smithi189 ceph-mon[36187]: pgmap v4619: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 5 op/s 2024-03-24T02:30:04.710 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:30:04 smithi193 ceph-mon[36878]: pgmap v4619: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 5 op/s 2024-03-24T02:30:06.695 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:30:06 smithi118 ceph-mon[31535]: pgmap v4620: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-03-24T02:30:06.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:30:06 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 7501.722131 secs 2024-03-24T02:30:06.703 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:30:06 smithi189 ceph-mon[36187]: pgmap v4620: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-03-24T02:30:06.703 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:30:06 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 7501.722131 secs 2024-03-24T02:30:06.710 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:30:06 smithi193 ceph-mon[36878]: pgmap v4620: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-03-24T02:30:06.710 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:30:06 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 7501.722131 secs 2024-03-24T02:30:08.445 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:30:08 smithi118 ceph-mon[31535]: pgmap v4621: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-03-24T02:30:08.703 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:30:08 smithi189 ceph-mon[36187]: pgmap v4621: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-03-24T02:30:08.710 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:30:08 smithi193 ceph-mon[36878]: pgmap v4621: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-03-24T02:30:10.695 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:30:10 smithi118 ceph-mon[31535]: pgmap v4622: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-03-24T02:30:10.703 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:30:10 smithi189 ceph-mon[36187]: pgmap v4622: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-03-24T02:30:10.710 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:30:10 smithi193 ceph-mon[36878]: pgmap v4622: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-03-24T02:30:11.695 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:30:11 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 7506.722191 secs 2024-03-24T02:30:11.703 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:30:11 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 7506.722191 secs 2024-03-24T02:30:11.710 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:30:11 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 7506.722191 secs 2024-03-24T02:30:12.695 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:30:12 smithi118 ceph-mon[31535]: pgmap v4623: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 6 op/s 2024-03-24T02:30:12.703 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:30:12 smithi189 ceph-mon[36187]: pgmap v4623: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 6 op/s 2024-03-24T02:30:12.710 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:30:12 smithi193 ceph-mon[36878]: pgmap v4623: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 6 op/s 2024-03-24T02:30:14.575 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:30:14 smithi193 ceph-mon[36878]: pgmap v4624: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T02:30:14.695 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:30:14 smithi118 ceph-mon[31535]: pgmap v4624: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T02:30:14.703 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:30:14 smithi189 ceph-mon[36187]: pgmap v4624: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T02:30:16.695 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:30:16 smithi118 ceph-mon[31535]: pgmap v4625: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-03-24T02:30:16.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:30:16 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 7511.722251 secs 2024-03-24T02:30:16.703 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:30:16 smithi189 ceph-mon[36187]: pgmap v4625: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-03-24T02:30:16.703 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:30:16 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 7511.722251 secs 2024-03-24T02:30:16.710 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:30:16 smithi193 ceph-mon[36878]: pgmap v4625: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-03-24T02:30:16.710 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:30:16 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 7511.722251 secs 2024-03-24T02:30:18.695 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:30:18 smithi118 ceph-mon[31535]: pgmap v4626: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:30:18.703 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:30:18 smithi189 ceph-mon[36187]: pgmap v4626: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:30:18.710 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:30:18 smithi193 ceph-mon[36878]: pgmap v4626: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:30:20.695 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:30:20 smithi118 ceph-mon[31535]: pgmap v4627: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T02:30:20.703 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:30:20 smithi189 ceph-mon[36187]: pgmap v4627: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T02:30:20.710 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:30:20 smithi193 ceph-mon[36878]: pgmap v4627: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T02:30:21.695 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:30:21 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 7516.722310 secs 2024-03-24T02:30:21.703 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:30:21 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 7516.722310 secs 2024-03-24T02:30:21.710 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:30:21 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 7516.722310 secs 2024-03-24T02:30:22.695 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:30:22 smithi118 ceph-mon[31535]: pgmap v4628: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T02:30:22.703 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:30:22 smithi189 ceph-mon[36187]: pgmap v4628: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T02:30:22.710 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:30:22 smithi193 ceph-mon[36878]: pgmap v4628: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T02:30:24.695 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:30:24 smithi118 ceph-mon[31535]: pgmap v4629: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T02:30:24.703 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:30:24 smithi189 ceph-mon[36187]: pgmap v4629: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T02:30:24.710 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:30:24 smithi193 ceph-mon[36878]: pgmap v4629: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T02:30:26.695 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:30:26 smithi118 ceph-mon[31535]: pgmap v4630: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T02:30:26.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:30:26 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 7521.722369 secs 2024-03-24T02:30:26.703 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:30:26 smithi189 ceph-mon[36187]: pgmap v4630: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T02:30:26.703 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:30:26 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 7521.722369 secs 2024-03-24T02:30:26.710 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:30:26 smithi193 ceph-mon[36878]: pgmap v4630: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T02:30:26.710 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:30:26 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 7521.722369 secs 2024-03-24T02:30:28.695 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:30:28 smithi118 ceph-mon[31535]: pgmap v4631: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:30:28.703 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:30:28 smithi189 ceph-mon[36187]: pgmap v4631: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:30:28.710 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:30:28 smithi193 ceph-mon[36878]: pgmap v4631: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:30:30.695 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:30:30 smithi118 ceph-mon[31535]: pgmap v4632: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T02:30:30.703 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:30:30 smithi189 ceph-mon[36187]: pgmap v4632: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T02:30:30.710 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:30:30 smithi193 ceph-mon[36878]: pgmap v4632: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T02:30:31.695 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:30:31 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 7526.722428 secs 2024-03-24T02:30:31.705 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:30:31 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 7526.722428 secs 2024-03-24T02:30:31.710 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:30:31 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 7526.722428 secs 2024-03-24T02:30:32.695 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:30:32 smithi118 ceph-mon[31535]: pgmap v4633: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T02:30:32.703 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:30:32 smithi189 ceph-mon[36187]: pgmap v4633: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T02:30:32.710 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:30:32 smithi193 ceph-mon[36878]: pgmap v4633: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T02:30:34.695 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:30:34 smithi118 ceph-mon[31535]: pgmap v4634: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T02:30:34.703 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:30:34 smithi189 ceph-mon[36187]: pgmap v4634: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T02:30:34.710 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:30:34 smithi193 ceph-mon[36878]: pgmap v4634: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T02:30:36.695 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:30:36 smithi118 ceph-mon[31535]: pgmap v4635: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-03-24T02:30:36.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:30:36 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 7531.722516 secs 2024-03-24T02:30:36.703 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:30:36 smithi189 ceph-mon[36187]: pgmap v4635: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-03-24T02:30:36.703 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:30:36 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 7531.722516 secs 2024-03-24T02:30:36.710 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:30:36 smithi193 ceph-mon[36878]: pgmap v4635: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-03-24T02:30:36.710 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:30:36 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 7531.722516 secs 2024-03-24T02:30:38.695 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:30:38 smithi118 ceph-mon[31535]: pgmap v4636: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:30:38.703 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:30:38 smithi189 ceph-mon[36187]: pgmap v4636: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:30:38.710 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:30:38 smithi193 ceph-mon[36878]: pgmap v4636: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:30:40.695 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:30:40 smithi118 ceph-mon[31535]: pgmap v4637: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T02:30:40.704 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:30:40 smithi189 ceph-mon[36187]: pgmap v4637: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T02:30:40.710 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:30:40 smithi193 ceph-mon[36878]: pgmap v4637: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T02:30:41.695 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:30:41 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 7536.722602 secs 2024-03-24T02:30:41.695 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:30:41 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T02:30:41.703 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:30:41 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 7536.722602 secs 2024-03-24T02:30:41.703 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:30:41 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T02:30:41.710 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:30:41 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 7536.722602 secs 2024-03-24T02:30:41.710 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:30:41 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T02:30:42.695 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:30:42 smithi118 ceph-mon[31535]: pgmap v4638: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T02:30:42.695 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:30:42 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T02:30:42.695 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:30:42 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T02:30:42.695 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:30:42 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T02:30:42.695 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:30:42 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T02:30:42.703 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:30:42 smithi189 ceph-mon[36187]: pgmap v4638: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T02:30:42.703 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:30:42 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T02:30:42.703 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:30:42 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T02:30:42.703 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:30:42 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T02:30:42.703 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:30:42 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T02:30:42.710 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:30:42 smithi193 ceph-mon[36878]: pgmap v4638: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T02:30:42.710 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:30:42 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T02:30:42.710 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:30:42 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T02:30:42.710 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:30:42 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T02:30:42.710 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:30:42 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T02:30:44.445 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:30:44 smithi118 ceph-mon[31535]: pgmap v4639: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T02:30:44.445 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:30:44 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T02:30:44.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:30:44 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T02:30:44.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:30:44 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T02:30:44.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:30:44 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T02:30:44.453 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:30:44 smithi189 ceph-mon[36187]: pgmap v4639: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T02:30:44.453 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:30:44 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T02:30:44.453 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:30:44 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T02:30:44.453 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:30:44 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T02:30:44.453 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:30:44 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T02:30:44.460 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:30:44 smithi193 ceph-mon[36878]: pgmap v4639: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T02:30:44.460 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:30:44 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T02:30:44.460 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:30:44 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T02:30:44.460 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:30:44 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T02:30:44.460 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:30:44 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T02:30:46.445 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:30:46 smithi118 ceph-mon[31535]: pgmap v4640: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-03-24T02:30:46.445 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:30:46 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 7541.722684 secs 2024-03-24T02:30:46.445 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:30:46 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T02:30:46.445 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:30:46 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T02:30:46.453 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:30:46 smithi189 ceph-mon[36187]: pgmap v4640: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-03-24T02:30:46.453 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:30:46 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 7541.722684 secs 2024-03-24T02:30:46.453 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:30:46 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T02:30:46.453 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:30:46 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T02:30:46.460 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:30:46 smithi193 ceph-mon[36878]: pgmap v4640: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-03-24T02:30:46.460 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:30:46 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 7541.722684 secs 2024-03-24T02:30:46.460 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:30:46 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T02:30:46.460 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:30:46 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T02:30:47.445 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:30:47 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T02:30:47.445 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:30:47 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T02:30:47.445 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:30:47 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T02:30:47.453 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:30:47 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T02:30:47.453 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:30:47 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T02:30:47.453 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:30:47 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T02:30:47.460 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:30:47 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T02:30:47.460 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:30:47 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T02:30:47.460 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:30:47 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T02:30:48.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:30:48 smithi118 ceph-mon[31535]: pgmap v4641: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:30:48.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:30:48 smithi118 ceph-mon[31535]: pgmap v4642: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T02:30:48.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:30:48 smithi118 ceph-mon[31535]: pgmap v4643: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 5 op/s 2024-03-24T02:30:48.453 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:30:48 smithi189 ceph-mon[36187]: pgmap v4641: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:30:48.453 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:30:48 smithi189 ceph-mon[36187]: pgmap v4642: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T02:30:48.453 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:30:48 smithi189 ceph-mon[36187]: pgmap v4643: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 5 op/s 2024-03-24T02:30:48.460 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:30:48 smithi193 ceph-mon[36878]: pgmap v4641: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:30:48.460 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:30:48 smithi193 ceph-mon[36878]: pgmap v4642: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T02:30:48.460 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:30:48 smithi193 ceph-mon[36878]: pgmap v4643: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 5 op/s 2024-03-24T02:30:50.445 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:30:50 smithi118 ceph-mon[31535]: pgmap v4644: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 5 op/s 2024-03-24T02:30:50.453 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:30:50 smithi189 ceph-mon[36187]: pgmap v4644: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 5 op/s 2024-03-24T02:30:50.460 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:30:50 smithi193 ceph-mon[36878]: pgmap v4644: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 5 op/s 2024-03-24T02:30:51.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:30:51 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 7546.722774 secs 2024-03-24T02:30:51.453 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:30:51 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 7546.722774 secs 2024-03-24T02:30:51.460 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:30:51 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 7546.722774 secs 2024-03-24T02:30:52.445 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:30:52 smithi118 ceph-mon[31535]: pgmap v4645: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 6 op/s 2024-03-24T02:30:52.453 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:30:52 smithi189 ceph-mon[36187]: pgmap v4645: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 6 op/s 2024-03-24T02:30:52.460 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:30:52 smithi193 ceph-mon[36878]: pgmap v4645: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 6 op/s 2024-03-24T02:30:54.445 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:30:54 smithi118 ceph-mon[31535]: pgmap v4646: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 4 op/s 2024-03-24T02:30:54.453 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:30:54 smithi189 ceph-mon[36187]: pgmap v4646: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 4 op/s 2024-03-24T02:30:54.460 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:30:54 smithi193 ceph-mon[36878]: pgmap v4646: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 4 op/s 2024-03-24T02:30:56.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:30:56 smithi118 ceph-mon[31535]: pgmap v4647: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-03-24T02:30:56.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:30:56 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 7551.722866 secs 2024-03-24T02:30:56.453 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:30:56 smithi189 ceph-mon[36187]: pgmap v4647: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-03-24T02:30:56.453 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:30:56 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 7551.722866 secs 2024-03-24T02:30:56.460 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:30:56 smithi193 ceph-mon[36878]: pgmap v4647: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-03-24T02:30:56.460 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:30:56 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 7551.722866 secs 2024-03-24T02:30:58.445 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:30:58 smithi118 ceph-mon[31535]: pgmap v4648: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 5 op/s 2024-03-24T02:30:58.453 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:30:58 smithi189 ceph-mon[36187]: pgmap v4648: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 5 op/s 2024-03-24T02:30:58.460 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:30:58 smithi193 ceph-mon[36878]: pgmap v4648: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 5 op/s 2024-03-24T02:31:00.445 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:31:00 smithi118 ceph-mon[31535]: pgmap v4649: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T02:31:00.453 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:31:00 smithi189 ceph-mon[36187]: pgmap v4649: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T02:31:00.460 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:31:00 smithi193 ceph-mon[36878]: pgmap v4649: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T02:31:01.445 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:31:01 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 7556.722925 secs 2024-03-24T02:31:01.453 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:31:01 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 7556.722925 secs 2024-03-24T02:31:01.460 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:31:01 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 7556.722925 secs 2024-03-24T02:31:02.445 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:31:02 smithi118 ceph-mon[31535]: pgmap v4650: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T02:31:02.453 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:31:02 smithi189 ceph-mon[36187]: pgmap v4650: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T02:31:02.460 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:31:02 smithi193 ceph-mon[36878]: pgmap v4650: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T02:31:04.445 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:31:04 smithi118 ceph-mon[31535]: pgmap v4651: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 5 op/s 2024-03-24T02:31:04.453 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:31:04 smithi189 ceph-mon[36187]: pgmap v4651: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 5 op/s 2024-03-24T02:31:04.460 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:31:04 smithi193 ceph-mon[36878]: pgmap v4651: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 5 op/s 2024-03-24T02:31:06.445 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:31:06 smithi118 ceph-mon[31535]: pgmap v4652: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T02:31:06.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:31:06 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 7561.723014 secs 2024-03-24T02:31:06.453 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:31:06 smithi189 ceph-mon[36187]: pgmap v4652: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T02:31:06.453 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:31:06 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 7561.723014 secs 2024-03-24T02:31:06.460 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:31:06 smithi193 ceph-mon[36878]: pgmap v4652: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T02:31:06.460 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:31:06 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 7561.723014 secs 2024-03-24T02:31:08.445 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:31:08 smithi118 ceph-mon[31535]: pgmap v4653: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:31:08.453 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:31:08 smithi189 ceph-mon[36187]: pgmap v4653: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:31:08.460 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:31:08 smithi193 ceph-mon[36878]: pgmap v4653: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:31:10.445 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:31:10 smithi118 ceph-mon[31535]: pgmap v4654: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T02:31:10.453 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:31:10 smithi189 ceph-mon[36187]: pgmap v4654: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T02:31:10.460 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:31:10 smithi193 ceph-mon[36878]: pgmap v4654: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T02:31:11.445 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:31:11 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 7566.723065 secs 2024-03-24T02:31:11.453 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:31:11 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 7566.723065 secs 2024-03-24T02:31:11.460 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:31:11 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 7566.723065 secs 2024-03-24T02:31:12.445 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:31:12 smithi118 ceph-mon[31535]: pgmap v4655: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T02:31:12.453 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:31:12 smithi189 ceph-mon[36187]: pgmap v4655: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T02:31:12.460 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:31:12 smithi193 ceph-mon[36878]: pgmap v4655: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T02:31:14.445 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:31:14 smithi118 ceph-mon[31535]: pgmap v4656: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 5 op/s 2024-03-24T02:31:14.453 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:31:14 smithi189 ceph-mon[36187]: pgmap v4656: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 5 op/s 2024-03-24T02:31:14.460 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:31:14 smithi193 ceph-mon[36878]: pgmap v4656: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 5 op/s 2024-03-24T02:31:16.445 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:31:16 smithi118 ceph-mon[31535]: pgmap v4657: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T02:31:16.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:31:16 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 7571.723147 secs 2024-03-24T02:31:16.453 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:31:16 smithi189 ceph-mon[36187]: pgmap v4657: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T02:31:16.453 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:31:16 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 7571.723147 secs 2024-03-24T02:31:16.460 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:31:16 smithi193 ceph-mon[36878]: pgmap v4657: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T02:31:16.460 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:31:16 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 7571.723147 secs 2024-03-24T02:31:16.955 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T02:31:16.955+0000 7f81ee000640 1 -- 172.21.15.118:0/1693767697 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f81c8006590 con 0x7f81f81052d0 2024-03-24T02:31:16.955 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T02:31:16.955+0000 7f81db400640 1 -- 172.21.15.118:0/659686424 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f81c4006290 con 0x5622651a13f0 2024-03-24T02:31:16.955 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T02:31:16.956+0000 7f81ef400640 1 -- 172.21.15.118:0/1693767697 <== mon.2 v2:172.21.15.189:3300/0 62 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (secure 0 0 0) 0x7f81c8006590 con 0x7f81f81052d0 2024-03-24T02:31:16.955 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T02:31:16.956+0000 7f81e4c00640 1 -- 172.21.15.118:0/659686424 <== mon.1 v2:172.21.15.193:3300/0 62 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 409+0+0 (secure 0 0 0) 0x7f81c4006290 con 0x5622651a13f0 2024-03-24T02:31:18.445 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:31:18 smithi118 ceph-mon[31535]: pgmap v4658: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:31:18.453 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:31:18 smithi189 ceph-mon[36187]: pgmap v4658: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:31:18.460 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:31:18 smithi193 ceph-mon[36878]: pgmap v4658: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:31:20.445 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:31:20 smithi118 ceph-mon[31535]: pgmap v4659: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T02:31:20.453 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:31:20 smithi189 ceph-mon[36187]: pgmap v4659: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T02:31:20.460 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:31:20 smithi193 ceph-mon[36878]: pgmap v4659: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T02:31:21.445 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:31:21 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 7576.723233 secs 2024-03-24T02:31:21.453 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:31:21 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 7576.723233 secs 2024-03-24T02:31:21.460 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:31:21 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 7576.723233 secs 2024-03-24T02:31:22.445 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:31:22 smithi118 ceph-mon[31535]: pgmap v4660: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T02:31:22.453 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:31:22 smithi189 ceph-mon[36187]: pgmap v4660: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T02:31:22.460 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:31:22 smithi193 ceph-mon[36878]: pgmap v4660: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T02:31:24.453 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:31:24 smithi189 ceph-mon[36187]: pgmap v4661: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 5 op/s 2024-03-24T02:31:24.460 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:31:24 smithi193 ceph-mon[36878]: pgmap v4661: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 5 op/s 2024-03-24T02:31:24.695 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:31:24 smithi118 ceph-mon[31535]: pgmap v4661: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 5 op/s 2024-03-24T02:31:26.445 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:31:26 smithi118 ceph-mon[31535]: pgmap v4662: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T02:31:26.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:31:26 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 7581.723315 secs 2024-03-24T02:31:26.460 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:31:26 smithi193 ceph-mon[36878]: pgmap v4662: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T02:31:26.460 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:31:26 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 7581.723315 secs 2024-03-24T02:31:26.703 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:31:26 smithi189 ceph-mon[36187]: pgmap v4662: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T02:31:26.703 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:31:26 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 7581.723315 secs 2024-03-24T02:31:28.460 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:31:28 smithi193 ceph-mon[36878]: pgmap v4663: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:31:28.695 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:31:28 smithi118 ceph-mon[31535]: pgmap v4663: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:31:28.703 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:31:28 smithi189 ceph-mon[36187]: pgmap v4663: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:31:30.445 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:31:30 smithi118 ceph-mon[31535]: pgmap v4664: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T02:31:30.703 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:31:30 smithi189 ceph-mon[36187]: pgmap v4664: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T02:31:30.710 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:31:30 smithi193 ceph-mon[36878]: pgmap v4664: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T02:31:31.695 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:31:31 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 7586.723402 secs 2024-03-24T02:31:31.703 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:31:31 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 7586.723402 secs 2024-03-24T02:31:31.710 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:31:31 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 7586.723402 secs 2024-03-24T02:31:32.695 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:31:32 smithi118 ceph-mon[31535]: pgmap v4665: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T02:31:32.703 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:31:32 smithi189 ceph-mon[36187]: pgmap v4665: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T02:31:32.710 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:31:32 smithi193 ceph-mon[36878]: pgmap v4665: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T02:31:34.497 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:31:34 smithi118 ceph-mon[31535]: pgmap v4666: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 5 op/s 2024-03-24T02:31:34.703 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:31:34 smithi189 ceph-mon[36187]: pgmap v4666: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 5 op/s 2024-03-24T02:31:34.710 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:31:34 smithi193 ceph-mon[36878]: pgmap v4666: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 5 op/s 2024-03-24T02:31:36.695 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:31:36 smithi118 ceph-mon[31535]: pgmap v4667: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T02:31:36.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:31:36 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 7591.723445 secs 2024-03-24T02:31:36.703 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:31:36 smithi189 ceph-mon[36187]: pgmap v4667: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T02:31:36.703 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:31:36 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 7591.723445 secs 2024-03-24T02:31:36.710 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:31:36 smithi193 ceph-mon[36878]: pgmap v4667: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T02:31:36.710 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:31:36 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 7591.723445 secs 2024-03-24T02:31:38.695 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:31:38 smithi118 ceph-mon[31535]: pgmap v4668: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:31:38.703 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:31:38 smithi189 ceph-mon[36187]: pgmap v4668: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:31:38.710 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:31:38 smithi193 ceph-mon[36878]: pgmap v4668: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:31:40.695 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:31:40 smithi118 ceph-mon[31535]: pgmap v4669: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T02:31:40.703 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:31:40 smithi189 ceph-mon[36187]: pgmap v4669: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T02:31:40.710 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:31:40 smithi193 ceph-mon[36878]: pgmap v4669: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T02:31:41.695 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:31:41 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 7596.723533 secs 2024-03-24T02:31:41.703 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:31:41 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 7596.723533 secs 2024-03-24T02:31:41.710 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:31:41 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 7596.723533 secs 2024-03-24T02:31:42.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:31:42 smithi118 ceph-mon[31535]: pgmap v4670: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T02:31:42.703 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:31:42 smithi189 ceph-mon[36187]: pgmap v4670: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T02:31:42.710 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:31:42 smithi193 ceph-mon[36878]: pgmap v4670: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T02:31:44.512 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:31:44 smithi193 ceph-mon[36878]: pgmap v4671: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 5 op/s 2024-03-24T02:31:44.695 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:31:44 smithi118 ceph-mon[31535]: pgmap v4671: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 5 op/s 2024-03-24T02:31:44.703 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:31:44 smithi189 ceph-mon[36187]: pgmap v4671: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 5 op/s 2024-03-24T02:31:46.695 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:31:46 smithi118 ceph-mon[31535]: pgmap v4672: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T02:31:46.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:31:46 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 7601.723618 secs 2024-03-24T02:31:46.703 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:31:46 smithi189 ceph-mon[36187]: pgmap v4672: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T02:31:46.703 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:31:46 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 7601.723618 secs 2024-03-24T02:31:46.710 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:31:46 smithi193 ceph-mon[36878]: pgmap v4672: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T02:31:46.710 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:31:46 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 7601.723618 secs 2024-03-24T02:31:47.695 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:31:47 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T02:31:47.703 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:31:47 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T02:31:47.710 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:31:47 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T02:31:48.695 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:31:48 smithi118 ceph-mon[31535]: pgmap v4673: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:31:48.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:31:48 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T02:31:48.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:31:48 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T02:31:48.703 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:31:48 smithi189 ceph-mon[36187]: pgmap v4673: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:31:48.703 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:31:48 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T02:31:48.703 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:31:48 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T02:31:48.710 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:31:48 smithi193 ceph-mon[36878]: pgmap v4673: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:31:48.710 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:31:48 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T02:31:48.710 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:31:48 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T02:31:49.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:31:49 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T02:31:49.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:31:49 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T02:31:49.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:31:49 smithi118 ceph-mon[31535]: pgmap v4674: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 6 op/s 2024-03-24T02:31:49.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:31:49 smithi118 ceph-mon[31535]: pgmap v4675: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 6 op/s 2024-03-24T02:31:49.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:31:49 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T02:31:49.703 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:31:49 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T02:31:49.703 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:31:49 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T02:31:49.703 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:31:49 smithi189 ceph-mon[36187]: pgmap v4674: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 6 op/s 2024-03-24T02:31:49.703 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:31:49 smithi189 ceph-mon[36187]: pgmap v4675: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 6 op/s 2024-03-24T02:31:49.703 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:31:49 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T02:31:49.710 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:31:49 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T02:31:49.710 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:31:49 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T02:31:49.710 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:31:49 smithi193 ceph-mon[36878]: pgmap v4674: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 6 op/s 2024-03-24T02:31:49.710 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:31:49 smithi193 ceph-mon[36878]: pgmap v4675: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 6 op/s 2024-03-24T02:31:49.710 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:31:49 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T02:31:50.945 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:31:50 smithi118 ceph-mon[31535]: pgmap v4676: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 6 op/s 2024-03-24T02:31:50.953 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:31:50 smithi189 ceph-mon[36187]: pgmap v4676: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 6 op/s 2024-03-24T02:31:50.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:31:50 smithi193 ceph-mon[36878]: pgmap v4676: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 6 op/s 2024-03-24T02:31:51.945 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:31:51 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 7606.723679 secs 2024-03-24T02:31:51.953 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:31:51 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 7606.723679 secs 2024-03-24T02:31:51.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:31:51 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 7606.723679 secs 2024-03-24T02:31:52.945 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:31:52 smithi118 ceph-mon[31535]: pgmap v4677: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 5 op/s 2024-03-24T02:31:52.953 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:31:52 smithi189 ceph-mon[36187]: pgmap v4677: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 5 op/s 2024-03-24T02:31:52.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:31:52 smithi193 ceph-mon[36878]: pgmap v4677: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 5 op/s 2024-03-24T02:31:54.945 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:31:54 smithi118 ceph-mon[31535]: pgmap v4678: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 6 op/s 2024-03-24T02:31:54.953 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:31:54 smithi189 ceph-mon[36187]: pgmap v4678: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 6 op/s 2024-03-24T02:31:54.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:31:54 smithi193 ceph-mon[36878]: pgmap v4678: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 6 op/s 2024-03-24T02:31:56.203 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:31:55 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 7611.723766 secs 2024-03-24T02:31:56.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:31:55 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 7611.723766 secs 2024-03-24T02:31:56.445 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:31:55 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 7611.723766 secs 2024-03-24T02:31:57.445 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:31:57 smithi118 ceph-mon[31535]: pgmap v4679: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 6 op/s 2024-03-24T02:31:57.453 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:31:57 smithi189 ceph-mon[36187]: pgmap v4679: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 6 op/s 2024-03-24T02:31:57.460 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:31:57 smithi193 ceph-mon[36878]: pgmap v4679: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 6 op/s 2024-03-24T02:31:58.945 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:31:58 smithi118 ceph-mon[31535]: pgmap v4680: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 5 op/s 2024-03-24T02:31:58.953 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:31:58 smithi189 ceph-mon[36187]: pgmap v4680: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 5 op/s 2024-03-24T02:31:58.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:31:58 smithi193 ceph-mon[36878]: pgmap v4680: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 5 op/s 2024-03-24T02:32:00.695 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:32:00 smithi118 ceph-mon[31535]: pgmap v4681: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T02:32:00.953 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:32:00 smithi189 ceph-mon[36187]: pgmap v4681: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T02:32:00.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:32:00 smithi193 ceph-mon[36878]: pgmap v4681: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T02:32:01.945 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:32:01 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 7616.723854 secs 2024-03-24T02:32:01.953 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:32:01 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 7616.723854 secs 2024-03-24T02:32:01.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:32:01 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 7616.723854 secs 2024-03-24T02:32:02.945 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:32:02 smithi118 ceph-mon[31535]: pgmap v4682: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T02:32:02.953 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:32:02 smithi189 ceph-mon[36187]: pgmap v4682: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T02:32:02.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:32:02 smithi193 ceph-mon[36878]: pgmap v4682: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T02:32:04.945 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:32:04 smithi118 ceph-mon[31535]: pgmap v4683: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-03-24T02:32:04.953 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:32:04 smithi189 ceph-mon[36187]: pgmap v4683: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-03-24T02:32:04.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:32:04 smithi193 ceph-mon[36878]: pgmap v4683: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-03-24T02:32:06.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:32:05 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 7621.723943 secs 2024-03-24T02:32:06.445 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:32:05 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 7621.723943 secs 2024-03-24T02:32:06.453 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:32:05 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 7621.723943 secs 2024-03-24T02:32:07.445 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:32:07 smithi118 ceph-mon[31535]: pgmap v4684: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:32:07.453 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:32:07 smithi189 ceph-mon[36187]: pgmap v4684: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:32:07.460 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:32:07 smithi193 ceph-mon[36878]: pgmap v4684: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:32:08.945 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:32:08 smithi118 ceph-mon[31535]: pgmap v4685: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T02:32:08.953 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:32:08 smithi189 ceph-mon[36187]: pgmap v4685: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T02:32:08.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:32:08 smithi193 ceph-mon[36878]: pgmap v4685: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T02:32:10.945 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:32:10 smithi118 ceph-mon[31535]: pgmap v4686: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T02:32:10.953 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:32:10 smithi189 ceph-mon[36187]: pgmap v4686: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T02:32:10.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:32:10 smithi193 ceph-mon[36878]: pgmap v4686: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T02:32:11.945 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:32:11 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 7626.723982 secs 2024-03-24T02:32:11.953 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:32:11 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 7626.723982 secs 2024-03-24T02:32:11.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:32:11 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 7626.723982 secs 2024-03-24T02:32:12.945 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:32:12 smithi118 ceph-mon[31535]: pgmap v4687: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T02:32:12.953 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:32:12 smithi189 ceph-mon[36187]: pgmap v4687: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T02:32:12.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:32:12 smithi193 ceph-mon[36878]: pgmap v4687: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T02:32:14.945 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:32:14 smithi118 ceph-mon[31535]: pgmap v4688: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-03-24T02:32:14.953 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:32:14 smithi189 ceph-mon[36187]: pgmap v4688: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-03-24T02:32:14.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:32:14 smithi193 ceph-mon[36878]: pgmap v4688: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-03-24T02:32:16.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:32:15 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 7631.724043 secs 2024-03-24T02:32:16.445 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:32:15 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 7631.724043 secs 2024-03-24T02:32:16.453 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:32:15 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 7631.724043 secs 2024-03-24T02:32:17.445 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:32:16 smithi118 ceph-mon[31535]: pgmap v4689: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:32:17.453 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:32:16 smithi189 ceph-mon[36187]: pgmap v4689: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:32:17.460 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:32:16 smithi193 ceph-mon[36878]: pgmap v4689: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:32:18.945 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:32:18 smithi118 ceph-mon[31535]: pgmap v4690: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T02:32:18.953 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:32:18 smithi189 ceph-mon[36187]: pgmap v4690: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T02:32:18.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:32:18 smithi193 ceph-mon[36878]: pgmap v4690: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T02:32:20.945 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:32:20 smithi118 ceph-mon[31535]: pgmap v4691: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T02:32:20.953 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:32:20 smithi189 ceph-mon[36187]: pgmap v4691: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T02:32:20.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:32:20 smithi193 ceph-mon[36878]: pgmap v4691: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T02:32:21.945 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:32:21 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 7636.724123 secs 2024-03-24T02:32:21.953 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:32:21 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 7636.724123 secs 2024-03-24T02:32:21.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:32:21 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 7636.724123 secs 2024-03-24T02:32:22.945 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:32:22 smithi118 ceph-mon[31535]: pgmap v4692: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T02:32:22.953 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:32:22 smithi189 ceph-mon[36187]: pgmap v4692: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T02:32:22.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:32:22 smithi193 ceph-mon[36878]: pgmap v4692: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T02:32:24.945 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:32:24 smithi118 ceph-mon[31535]: pgmap v4693: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-03-24T02:32:24.953 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:32:24 smithi189 ceph-mon[36187]: pgmap v4693: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-03-24T02:32:24.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:32:24 smithi193 ceph-mon[36878]: pgmap v4693: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-03-24T02:32:26.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:32:25 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 7641.724184 secs 2024-03-24T02:32:26.445 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:32:25 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 7641.724184 secs 2024-03-24T02:32:26.453 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:32:25 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 7641.724184 secs 2024-03-24T02:32:27.445 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:32:26 smithi118 ceph-mon[31535]: pgmap v4694: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:32:27.453 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:32:26 smithi189 ceph-mon[36187]: pgmap v4694: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:32:27.460 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:32:26 smithi193 ceph-mon[36878]: pgmap v4694: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:32:28.945 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:32:28 smithi118 ceph-mon[31535]: pgmap v4695: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T02:32:28.953 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:32:28 smithi189 ceph-mon[36187]: pgmap v4695: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T02:32:28.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:32:28 smithi193 ceph-mon[36878]: pgmap v4695: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T02:32:30.945 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:32:30 smithi118 ceph-mon[31535]: pgmap v4696: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T02:32:30.953 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:32:30 smithi189 ceph-mon[36187]: pgmap v4696: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T02:32:30.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:32:30 smithi193 ceph-mon[36878]: pgmap v4696: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T02:32:31.945 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:32:31 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 7646.724219 secs 2024-03-24T02:32:31.953 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:32:31 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 7646.724219 secs 2024-03-24T02:32:31.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:32:31 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 7646.724219 secs 2024-03-24T02:32:32.945 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:32:32 smithi118 ceph-mon[31535]: pgmap v4697: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T02:32:32.953 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:32:32 smithi189 ceph-mon[36187]: pgmap v4697: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T02:32:32.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:32:32 smithi193 ceph-mon[36878]: pgmap v4697: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T02:32:34.945 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:32:34 smithi118 ceph-mon[31535]: pgmap v4698: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-03-24T02:32:34.953 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:32:34 smithi189 ceph-mon[36187]: pgmap v4698: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-03-24T02:32:34.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:32:34 smithi193 ceph-mon[36878]: pgmap v4698: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-03-24T02:32:36.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:32:35 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 7651.724308 secs 2024-03-24T02:32:36.445 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:32:35 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 7651.724308 secs 2024-03-24T02:32:36.453 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:32:35 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 7651.724308 secs 2024-03-24T02:32:37.445 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:32:36 smithi118 ceph-mon[31535]: pgmap v4699: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:32:37.453 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:32:36 smithi189 ceph-mon[36187]: pgmap v4699: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:32:37.460 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:32:36 smithi193 ceph-mon[36878]: pgmap v4699: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:32:38.945 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:32:38 smithi118 ceph-mon[31535]: pgmap v4700: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T02:32:38.953 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:32:38 smithi189 ceph-mon[36187]: pgmap v4700: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T02:32:38.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:32:38 smithi193 ceph-mon[36878]: pgmap v4700: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T02:32:40.945 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:32:40 smithi118 ceph-mon[31535]: pgmap v4701: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T02:32:40.953 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:32:40 smithi189 ceph-mon[36187]: pgmap v4701: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T02:32:40.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:32:40 smithi193 ceph-mon[36878]: pgmap v4701: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T02:32:41.945 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:32:41 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 7656.724396 secs 2024-03-24T02:32:41.953 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:32:41 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 7656.724396 secs 2024-03-24T02:32:41.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:32:41 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 7656.724396 secs 2024-03-24T02:32:42.945 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:32:42 smithi118 ceph-mon[31535]: pgmap v4702: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T02:32:42.953 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:32:42 smithi189 ceph-mon[36187]: pgmap v4702: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T02:32:42.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:32:42 smithi193 ceph-mon[36878]: pgmap v4702: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T02:32:44.945 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:32:44 smithi118 ceph-mon[31535]: pgmap v4703: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-03-24T02:32:44.953 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:32:44 smithi189 ceph-mon[36187]: pgmap v4703: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-03-24T02:32:44.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:32:44 smithi193 ceph-mon[36878]: pgmap v4703: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-03-24T02:32:46.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:32:45 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 7661.724456 secs 2024-03-24T02:32:46.445 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:32:45 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 7661.724456 secs 2024-03-24T02:32:46.453 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:32:45 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 7661.724456 secs 2024-03-24T02:32:47.195 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:32:46 smithi118 ceph-mon[31535]: pgmap v4704: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:32:47.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:32:46 smithi193 ceph-mon[36878]: pgmap v4704: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:32:47.453 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:32:46 smithi189 ceph-mon[36187]: pgmap v4704: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:32:48.945 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:32:48 smithi118 ceph-mon[31535]: pgmap v4705: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T02:32:48.953 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:32:48 smithi189 ceph-mon[36187]: pgmap v4705: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T02:32:48.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:32:48 smithi193 ceph-mon[36878]: pgmap v4705: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T02:32:49.945 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:32:49 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T02:32:49.953 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:32:49 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T02:32:49.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:32:49 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T02:32:50.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:32:50 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T02:32:50.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:32:50 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T02:32:50.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:32:50 smithi118 ceph-mon[31535]: pgmap v4706: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 6 op/s 2024-03-24T02:32:50.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:32:50 smithi118 ceph-mon[31535]: pgmap v4707: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-03-24T02:32:50.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:32:50 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T02:32:50.953 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:32:50 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T02:32:50.953 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:32:50 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T02:32:50.953 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:32:50 smithi189 ceph-mon[36187]: pgmap v4706: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 6 op/s 2024-03-24T02:32:50.953 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:32:50 smithi189 ceph-mon[36187]: pgmap v4707: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-03-24T02:32:50.953 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:32:50 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T02:32:50.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:32:50 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T02:32:50.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:32:50 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T02:32:50.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:32:50 smithi193 ceph-mon[36878]: pgmap v4706: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 6 op/s 2024-03-24T02:32:50.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:32:50 smithi193 ceph-mon[36878]: pgmap v4707: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-03-24T02:32:50.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:32:50 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T02:32:51.945 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:32:51 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 7666.724546 secs 2024-03-24T02:32:51.953 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:32:51 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 7666.724546 secs 2024-03-24T02:32:51.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:32:51 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 7666.724546 secs 2024-03-24T02:32:52.945 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:32:52 smithi118 ceph-mon[31535]: pgmap v4708: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 5 op/s 2024-03-24T02:32:52.953 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:32:52 smithi189 ceph-mon[36187]: pgmap v4708: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 5 op/s 2024-03-24T02:32:52.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:32:52 smithi193 ceph-mon[36878]: pgmap v4708: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 5 op/s 2024-03-24T02:32:55.445 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:32:55 smithi118 ceph-mon[31535]: pgmap v4709: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 5 op/s 2024-03-24T02:32:55.453 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:32:55 smithi189 ceph-mon[36187]: pgmap v4709: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 5 op/s 2024-03-24T02:32:55.460 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:32:55 smithi193 ceph-mon[36878]: pgmap v4709: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 5 op/s 2024-03-24T02:32:56.445 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:32:56 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 7671.724629 secs 2024-03-24T02:32:56.453 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:32:56 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 7671.724629 secs 2024-03-24T02:32:56.460 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:32:56 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 7671.724629 secs 2024-03-24T02:32:57.445 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:32:57 smithi118 ceph-mon[31535]: pgmap v4710: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-03-24T02:32:57.453 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:32:57 smithi189 ceph-mon[36187]: pgmap v4710: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-03-24T02:32:57.460 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:32:57 smithi193 ceph-mon[36878]: pgmap v4710: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-03-24T02:32:59.445 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:32:59 smithi118 ceph-mon[31535]: pgmap v4711: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 5 op/s 2024-03-24T02:32:59.453 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:32:59 smithi189 ceph-mon[36187]: pgmap v4711: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 5 op/s 2024-03-24T02:32:59.460 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:32:59 smithi193 ceph-mon[36878]: pgmap v4711: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 5 op/s 2024-03-24T02:33:01.445 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:33:01 smithi118 ceph-mon[31535]: pgmap v4712: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 5 op/s 2024-03-24T02:33:01.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:33:01 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 7676.724717 secs 2024-03-24T02:33:01.453 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:33:01 smithi189 ceph-mon[36187]: pgmap v4712: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 5 op/s 2024-03-24T02:33:01.453 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:33:01 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 7676.724717 secs 2024-03-24T02:33:01.460 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:33:01 smithi193 ceph-mon[36878]: pgmap v4712: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 5 op/s 2024-03-24T02:33:01.460 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:33:01 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 7676.724717 secs 2024-03-24T02:33:03.445 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:33:03 smithi118 ceph-mon[31535]: pgmap v4713: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:33:03.453 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:33:03 smithi189 ceph-mon[36187]: pgmap v4713: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:33:03.460 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:33:03 smithi193 ceph-mon[36878]: pgmap v4713: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:33:05.445 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:33:05 smithi118 ceph-mon[31535]: pgmap v4714: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T02:33:05.453 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:33:05 smithi189 ceph-mon[36187]: pgmap v4714: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T02:33:05.460 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:33:05 smithi193 ceph-mon[36878]: pgmap v4714: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T02:33:06.445 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:33:06 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 7681.724805 secs 2024-03-24T02:33:06.453 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:33:06 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 7681.724805 secs 2024-03-24T02:33:06.460 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:33:06 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 7681.724805 secs 2024-03-24T02:33:07.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:33:07 smithi118 ceph-mon[31535]: pgmap v4715: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T02:33:07.453 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:33:07 smithi189 ceph-mon[36187]: pgmap v4715: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T02:33:07.460 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:33:07 smithi193 ceph-mon[36878]: pgmap v4715: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T02:33:09.445 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:33:09 smithi118 ceph-mon[31535]: pgmap v4716: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 5 op/s 2024-03-24T02:33:09.453 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:33:09 smithi189 ceph-mon[36187]: pgmap v4716: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 5 op/s 2024-03-24T02:33:09.460 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:33:09 smithi193 ceph-mon[36878]: pgmap v4716: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 5 op/s 2024-03-24T02:33:11.445 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:33:11 smithi118 ceph-mon[31535]: pgmap v4717: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T02:33:11.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:33:11 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 7686.724889 secs 2024-03-24T02:33:11.453 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:33:11 smithi189 ceph-mon[36187]: pgmap v4717: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T02:33:11.453 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:33:11 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 7686.724889 secs 2024-03-24T02:33:11.460 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:33:11 smithi193 ceph-mon[36878]: pgmap v4717: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T02:33:11.460 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:33:11 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 7686.724889 secs 2024-03-24T02:33:13.445 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:33:13 smithi118 ceph-mon[31535]: pgmap v4718: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:33:13.453 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:33:13 smithi189 ceph-mon[36187]: pgmap v4718: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:33:13.460 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:33:13 smithi193 ceph-mon[36878]: pgmap v4718: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:33:15.445 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:33:15 smithi118 ceph-mon[31535]: pgmap v4719: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T02:33:15.453 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:33:15 smithi189 ceph-mon[36187]: pgmap v4719: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T02:33:15.460 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:33:15 smithi193 ceph-mon[36878]: pgmap v4719: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T02:33:16.445 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:33:16 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 7691.724978 secs 2024-03-24T02:33:16.453 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:33:16 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 7691.724978 secs 2024-03-24T02:33:16.460 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:33:16 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 7691.724978 secs 2024-03-24T02:33:17.445 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:33:17 smithi118 ceph-mon[31535]: pgmap v4720: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T02:33:17.453 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:33:17 smithi189 ceph-mon[36187]: pgmap v4720: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T02:33:17.459 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:33:17 smithi193 ceph-mon[36878]: pgmap v4720: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T02:33:19.445 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:33:19 smithi118 ceph-mon[31535]: pgmap v4721: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 5 op/s 2024-03-24T02:33:19.453 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:33:19 smithi189 ceph-mon[36187]: pgmap v4721: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 5 op/s 2024-03-24T02:33:19.460 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:33:19 smithi193 ceph-mon[36878]: pgmap v4721: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 5 op/s 2024-03-24T02:33:21.445 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:33:21 smithi118 ceph-mon[31535]: pgmap v4722: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T02:33:21.445 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:33:21 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 7696.725038 secs 2024-03-24T02:33:21.453 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:33:21 smithi189 ceph-mon[36187]: pgmap v4722: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T02:33:21.453 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:33:21 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 7696.725038 secs 2024-03-24T02:33:21.459 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:33:21 smithi193 ceph-mon[36878]: pgmap v4722: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T02:33:21.460 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:33:21 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 7696.725038 secs 2024-03-24T02:33:23.445 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:33:23 smithi118 ceph-mon[31535]: pgmap v4723: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:33:23.453 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:33:23 smithi189 ceph-mon[36187]: pgmap v4723: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:33:23.460 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:33:23 smithi193 ceph-mon[36878]: pgmap v4723: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:33:25.445 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:33:25 smithi118 ceph-mon[31535]: pgmap v4724: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T02:33:25.453 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:33:25 smithi189 ceph-mon[36187]: pgmap v4724: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T02:33:25.460 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:33:25 smithi193 ceph-mon[36878]: pgmap v4724: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T02:33:26.445 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:33:26 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 7701.725099 secs 2024-03-24T02:33:26.453 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:33:26 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 7701.725099 secs 2024-03-24T02:33:26.460 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:33:26 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 7701.725099 secs 2024-03-24T02:33:27.445 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:33:27 smithi118 ceph-mon[31535]: pgmap v4725: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T02:33:27.453 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:33:27 smithi189 ceph-mon[36187]: pgmap v4725: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T02:33:27.459 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:33:27 smithi193 ceph-mon[36878]: pgmap v4725: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T02:33:29.445 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:33:29 smithi118 ceph-mon[31535]: pgmap v4726: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 5 op/s 2024-03-24T02:33:29.453 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:33:29 smithi189 ceph-mon[36187]: pgmap v4726: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 5 op/s 2024-03-24T02:33:29.460 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:33:29 smithi193 ceph-mon[36878]: pgmap v4726: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 5 op/s 2024-03-24T02:33:31.445 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:33:31 smithi118 ceph-mon[31535]: pgmap v4727: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T02:33:31.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:33:31 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 7706.725142 secs 2024-03-24T02:33:31.453 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:33:31 smithi189 ceph-mon[36187]: pgmap v4727: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T02:33:31.453 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:33:31 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 7706.725142 secs 2024-03-24T02:33:31.460 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:33:31 smithi193 ceph-mon[36878]: pgmap v4727: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T02:33:31.460 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:33:31 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 7706.725142 secs 2024-03-24T02:33:33.445 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:33:33 smithi118 ceph-mon[31535]: pgmap v4728: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:33:33.453 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:33:33 smithi189 ceph-mon[36187]: pgmap v4728: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:33:33.460 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:33:33 smithi193 ceph-mon[36878]: pgmap v4728: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:33:35.445 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:33:35 smithi118 ceph-mon[31535]: pgmap v4729: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T02:33:35.453 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:33:35 smithi189 ceph-mon[36187]: pgmap v4729: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T02:33:35.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:33:35 smithi193 ceph-mon[36878]: pgmap v4729: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T02:33:36.445 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:33:36 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 7711.725229 secs 2024-03-24T02:33:36.453 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:33:36 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 7711.725229 secs 2024-03-24T02:33:36.460 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:33:36 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 7711.725229 secs 2024-03-24T02:33:37.445 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:33:37 smithi118 ceph-mon[31535]: pgmap v4730: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T02:33:37.453 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:33:37 smithi189 ceph-mon[36187]: pgmap v4730: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T02:33:37.459 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:33:37 smithi193 ceph-mon[36878]: pgmap v4730: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T02:33:39.445 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:33:39 smithi118 ceph-mon[31535]: pgmap v4731: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 5 op/s 2024-03-24T02:33:39.453 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:33:39 smithi189 ceph-mon[36187]: pgmap v4731: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 5 op/s 2024-03-24T02:33:39.460 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:33:39 smithi193 ceph-mon[36878]: pgmap v4731: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 5 op/s 2024-03-24T02:33:41.445 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:33:41 smithi118 ceph-mon[31535]: pgmap v4732: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T02:33:41.445 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:33:41 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 7716.725289 secs 2024-03-24T02:33:41.453 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:33:41 smithi189 ceph-mon[36187]: pgmap v4732: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T02:33:41.453 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:33:41 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 7716.725289 secs 2024-03-24T02:33:41.460 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:33:41 smithi193 ceph-mon[36878]: pgmap v4732: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T02:33:41.460 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:33:41 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 7716.725289 secs 2024-03-24T02:33:43.445 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:33:43 smithi118 ceph-mon[31535]: pgmap v4733: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:33:43.453 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:33:43 smithi189 ceph-mon[36187]: pgmap v4733: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:33:43.460 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:33:43 smithi193 ceph-mon[36878]: pgmap v4733: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:33:45.445 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:33:45 smithi118 ceph-mon[31535]: pgmap v4734: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T02:33:45.453 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:33:45 smithi189 ceph-mon[36187]: pgmap v4734: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T02:33:45.460 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:33:45 smithi193 ceph-mon[36878]: pgmap v4734: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T02:33:46.445 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:33:46 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 7721.725367 secs 2024-03-24T02:33:46.453 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:33:46 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 7721.725367 secs 2024-03-24T02:33:46.460 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:33:46 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 7721.725367 secs 2024-03-24T02:33:47.445 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:33:47 smithi118 ceph-mon[31535]: pgmap v4735: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T02:33:47.453 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:33:47 smithi189 ceph-mon[36187]: pgmap v4735: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T02:33:47.460 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:33:47 smithi193 ceph-mon[36878]: pgmap v4735: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T02:33:49.445 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:33:49 smithi118 ceph-mon[31535]: pgmap v4736: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 5 op/s 2024-03-24T02:33:49.453 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:33:49 smithi189 ceph-mon[36187]: pgmap v4736: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 5 op/s 2024-03-24T02:33:49.459 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:33:49 smithi193 ceph-mon[36878]: pgmap v4736: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 5 op/s 2024-03-24T02:33:51.445 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:33:51 smithi118 ceph-mon[31535]: pgmap v4737: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T02:33:51.445 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:33:51 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T02:33:51.445 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:33:51 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 7726.725429 secs 2024-03-24T02:33:51.453 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:33:51 smithi189 ceph-mon[36187]: pgmap v4737: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T02:33:51.453 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:33:51 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T02:33:51.453 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:33:51 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 7726.725429 secs 2024-03-24T02:33:51.460 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:33:51 smithi193 ceph-mon[36878]: pgmap v4737: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T02:33:51.460 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:33:51 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T02:33:51.460 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:33:51 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 7726.725429 secs 2024-03-24T02:33:52.445 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:33:52 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T02:33:52.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:33:52 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T02:33:52.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:33:52 smithi118 ceph-mon[31535]: pgmap v4738: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 5 op/s 2024-03-24T02:33:52.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:33:52 smithi118 ceph-mon[31535]: pgmap v4739: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-03-24T02:33:52.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:33:52 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T02:33:52.453 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:33:52 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T02:33:52.453 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:33:52 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T02:33:52.453 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:33:52 smithi189 ceph-mon[36187]: pgmap v4738: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 5 op/s 2024-03-24T02:33:52.453 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:33:52 smithi189 ceph-mon[36187]: pgmap v4739: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-03-24T02:33:52.453 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:33:52 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T02:33:52.460 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:33:52 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T02:33:52.460 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:33:52 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T02:33:52.460 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:33:52 smithi193 ceph-mon[36878]: pgmap v4738: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 5 op/s 2024-03-24T02:33:52.460 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:33:52 smithi193 ceph-mon[36878]: pgmap v4739: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-03-24T02:33:52.460 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:33:52 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T02:33:53.945 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:33:53 smithi118 ceph-mon[31535]: pgmap v4740: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 5 op/s 2024-03-24T02:33:53.953 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:33:53 smithi189 ceph-mon[36187]: pgmap v4740: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 5 op/s 2024-03-24T02:33:53.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:33:53 smithi193 ceph-mon[36878]: pgmap v4740: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 5 op/s 2024-03-24T02:33:55.945 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:33:55 smithi118 ceph-mon[31535]: pgmap v4741: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-03-24T02:33:55.953 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:33:55 smithi189 ceph-mon[36187]: pgmap v4741: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-03-24T02:33:55.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:33:55 smithi193 ceph-mon[36878]: pgmap v4741: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-03-24T02:33:56.945 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:33:56 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 7731.725488 secs 2024-03-24T02:33:56.953 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:33:56 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 7731.725488 secs 2024-03-24T02:33:56.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:33:56 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 7731.725488 secs 2024-03-24T02:33:57.945 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:33:57 smithi118 ceph-mon[31535]: pgmap v4742: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 5 op/s 2024-03-24T02:33:57.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:33:57 smithi189 ceph-mon[36187]: pgmap v4742: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 5 op/s 2024-03-24T02:33:57.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:33:57 smithi193 ceph-mon[36878]: pgmap v4742: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 5 op/s 2024-03-24T02:33:59.945 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:33:59 smithi118 ceph-mon[31535]: pgmap v4743: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-03-24T02:33:59.953 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:33:59 smithi189 ceph-mon[36187]: pgmap v4743: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-03-24T02:33:59.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:33:59 smithi193 ceph-mon[36878]: pgmap v4743: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-03-24T02:34:01.445 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:34:00 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 7736.725546 secs 2024-03-24T02:34:01.453 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:34:00 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 7736.725546 secs 2024-03-24T02:34:01.460 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:34:00 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 7736.725546 secs 2024-03-24T02:34:02.445 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:34:01 smithi118 ceph-mon[31535]: pgmap v4744: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 5 op/s 2024-03-24T02:34:02.453 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:34:01 smithi189 ceph-mon[36187]: pgmap v4744: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 5 op/s 2024-03-24T02:34:02.459 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:34:01 smithi193 ceph-mon[36878]: pgmap v4744: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 5 op/s 2024-03-24T02:34:03.945 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:34:03 smithi118 ceph-mon[31535]: pgmap v4745: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T02:34:03.953 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:34:03 smithi189 ceph-mon[36187]: pgmap v4745: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T02:34:03.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:34:03 smithi193 ceph-mon[36878]: pgmap v4745: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T02:34:05.945 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:34:05 smithi118 ceph-mon[31535]: pgmap v4746: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T02:34:05.953 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:34:05 smithi189 ceph-mon[36187]: pgmap v4746: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T02:34:05.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:34:05 smithi193 ceph-mon[36878]: pgmap v4746: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T02:34:06.945 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:34:06 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 7741.725605 secs 2024-03-24T02:34:06.953 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:34:06 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 7741.725605 secs 2024-03-24T02:34:06.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:34:06 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 7741.725605 secs 2024-03-24T02:34:07.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:34:07 smithi118 ceph-mon[31535]: pgmap v4747: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T02:34:07.953 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:34:07 smithi189 ceph-mon[36187]: pgmap v4747: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T02:34:07.959 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:34:07 smithi193 ceph-mon[36878]: pgmap v4747: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T02:34:09.945 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:34:09 smithi118 ceph-mon[31535]: pgmap v4748: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-03-24T02:34:09.953 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:34:09 smithi189 ceph-mon[36187]: pgmap v4748: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-03-24T02:34:09.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:34:09 smithi193 ceph-mon[36878]: pgmap v4748: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-03-24T02:34:11.445 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:34:10 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 7746.725661 secs 2024-03-24T02:34:11.453 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:34:10 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 7746.725661 secs 2024-03-24T02:34:11.460 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:34:10 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 7746.725661 secs 2024-03-24T02:34:12.445 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:34:11 smithi118 ceph-mon[31535]: pgmap v4749: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:34:12.453 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:34:11 smithi189 ceph-mon[36187]: pgmap v4749: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:34:12.460 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:34:11 smithi193 ceph-mon[36878]: pgmap v4749: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:34:13.945 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:34:13 smithi118 ceph-mon[31535]: pgmap v4750: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T02:34:13.953 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:34:13 smithi189 ceph-mon[36187]: pgmap v4750: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T02:34:13.959 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:34:13 smithi193 ceph-mon[36878]: pgmap v4750: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T02:34:15.945 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:34:15 smithi118 ceph-mon[31535]: pgmap v4751: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T02:34:15.953 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:34:15 smithi189 ceph-mon[36187]: pgmap v4751: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T02:34:15.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:34:15 smithi193 ceph-mon[36878]: pgmap v4751: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T02:34:16.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:34:16 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 7751.725745 secs 2024-03-24T02:34:16.953 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:34:16 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 7751.725745 secs 2024-03-24T02:34:16.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:34:16 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 7751.725745 secs 2024-03-24T02:34:16.985 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T02:34:16.985+0000 7f81ee000640 1 -- 172.21.15.118:0/1693767697 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f81c8006590 con 0x7f81f81052d0 2024-03-24T02:34:16.985 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T02:34:16.985+0000 7f81db400640 1 -- 172.21.15.118:0/659686424 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f81c4006290 con 0x5622651a13f0 2024-03-24T02:34:16.985 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T02:34:16.986+0000 7f81ef400640 1 -- 172.21.15.118:0/1693767697 <== mon.2 v2:172.21.15.189:3300/0 63 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (secure 0 0 0) 0x7f81c8006590 con 0x7f81f81052d0 2024-03-24T02:34:16.985 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T02:34:16.986+0000 7f81e4c00640 1 -- 172.21.15.118:0/659686424 <== mon.1 v2:172.21.15.193:3300/0 63 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 409+0+0 (secure 0 0 0) 0x7f81c4006290 con 0x5622651a13f0 2024-03-24T02:34:17.945 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:34:17 smithi118 ceph-mon[31535]: pgmap v4752: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T02:34:17.953 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:34:17 smithi189 ceph-mon[36187]: pgmap v4752: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T02:34:17.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:34:17 smithi193 ceph-mon[36878]: pgmap v4752: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T02:34:19.945 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:34:19 smithi118 ceph-mon[31535]: pgmap v4753: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-03-24T02:34:19.953 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:34:19 smithi189 ceph-mon[36187]: pgmap v4753: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-03-24T02:34:19.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:34:19 smithi193 ceph-mon[36878]: pgmap v4753: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-03-24T02:34:21.445 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:34:20 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 7756.725825 secs 2024-03-24T02:34:21.453 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:34:20 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 7756.725825 secs 2024-03-24T02:34:21.460 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:34:20 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 7756.725825 secs 2024-03-24T02:34:22.445 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:34:21 smithi118 ceph-mon[31535]: pgmap v4754: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:34:22.453 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:34:21 smithi189 ceph-mon[36187]: pgmap v4754: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:34:22.460 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:34:21 smithi193 ceph-mon[36878]: pgmap v4754: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:34:23.945 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:34:23 smithi118 ceph-mon[31535]: pgmap v4755: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T02:34:23.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:34:23 smithi189 ceph-mon[36187]: pgmap v4755: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T02:34:23.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:34:23 smithi193 ceph-mon[36878]: pgmap v4755: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T02:34:25.945 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:34:25 smithi118 ceph-mon[31535]: pgmap v4756: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T02:34:25.953 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:34:25 smithi189 ceph-mon[36187]: pgmap v4756: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T02:34:25.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:34:25 smithi193 ceph-mon[36878]: pgmap v4756: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T02:34:26.945 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:34:26 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 7761.725865 secs 2024-03-24T02:34:26.953 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:34:26 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 7761.725865 secs 2024-03-24T02:34:26.959 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:34:26 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 7761.725865 secs 2024-03-24T02:34:27.945 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:34:27 smithi118 ceph-mon[31535]: pgmap v4757: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T02:34:27.952 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:34:27 smithi189 ceph-mon[36187]: pgmap v4757: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T02:34:27.959 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:34:27 smithi193 ceph-mon[36878]: pgmap v4757: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T02:34:29.945 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:34:29 smithi118 ceph-mon[31535]: pgmap v4758: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-03-24T02:34:29.953 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:34:29 smithi189 ceph-mon[36187]: pgmap v4758: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-03-24T02:34:29.959 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:34:29 smithi193 ceph-mon[36878]: pgmap v4758: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-03-24T02:34:31.445 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:34:30 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 7766.725908 secs 2024-03-24T02:34:31.453 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:34:30 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 7766.725908 secs 2024-03-24T02:34:31.460 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:34:30 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 7766.725908 secs 2024-03-24T02:34:32.345 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:34:31 smithi118 ceph-mon[31535]: pgmap v4759: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:34:32.453 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:34:31 smithi189 ceph-mon[36187]: pgmap v4759: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:34:32.459 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:34:31 smithi193 ceph-mon[36878]: pgmap v4759: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:34:33.945 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:34:33 smithi118 ceph-mon[31535]: pgmap v4760: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T02:34:33.953 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:34:33 smithi189 ceph-mon[36187]: pgmap v4760: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T02:34:33.959 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:34:33 smithi193 ceph-mon[36878]: pgmap v4760: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T02:34:35.927 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:34:35 smithi118 ceph-mon[31535]: pgmap v4761: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T02:34:35.953 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:34:35 smithi189 ceph-mon[36187]: pgmap v4761: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T02:34:35.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:34:35 smithi193 ceph-mon[36878]: pgmap v4761: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T02:34:36.945 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:34:36 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 7771.725993 secs 2024-03-24T02:34:36.953 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:34:36 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 7771.725993 secs 2024-03-24T02:34:36.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:34:36 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 7771.725993 secs 2024-03-24T02:34:37.945 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:34:37 smithi118 ceph-mon[31535]: pgmap v4762: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T02:34:37.952 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:34:37 smithi189 ceph-mon[36187]: pgmap v4762: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T02:34:37.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:34:37 smithi193 ceph-mon[36878]: pgmap v4762: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T02:34:39.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:34:39 smithi118 ceph-mon[31535]: pgmap v4763: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-03-24T02:34:39.953 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:34:39 smithi189 ceph-mon[36187]: pgmap v4763: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-03-24T02:34:39.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:34:39 smithi193 ceph-mon[36878]: pgmap v4763: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-03-24T02:34:41.445 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:34:40 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 7776.726078 secs 2024-03-24T02:34:41.453 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:34:40 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 7776.726078 secs 2024-03-24T02:34:41.460 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:34:40 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 7776.726078 secs 2024-03-24T02:34:42.445 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:34:41 smithi118 ceph-mon[31535]: pgmap v4764: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:34:42.453 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:34:41 smithi189 ceph-mon[36187]: pgmap v4764: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:34:42.460 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:34:41 smithi193 ceph-mon[36878]: pgmap v4764: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:34:43.945 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:34:43 smithi118 ceph-mon[31535]: pgmap v4765: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T02:34:43.953 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:34:43 smithi189 ceph-mon[36187]: pgmap v4765: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T02:34:43.959 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:34:43 smithi193 ceph-mon[36878]: pgmap v4765: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T02:34:45.945 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:34:45 smithi118 ceph-mon[31535]: pgmap v4766: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T02:34:45.953 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:34:45 smithi189 ceph-mon[36187]: pgmap v4766: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T02:34:45.959 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:34:45 smithi193 ceph-mon[36878]: pgmap v4766: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T02:34:46.945 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:34:46 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 7781.726162 secs 2024-03-24T02:34:46.953 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:34:46 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 7781.726162 secs 2024-03-24T02:34:46.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:34:46 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 7781.726162 secs 2024-03-24T02:34:47.945 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:34:47 smithi118 ceph-mon[31535]: pgmap v4767: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T02:34:47.953 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:34:47 smithi189 ceph-mon[36187]: pgmap v4767: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T02:34:47.959 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:34:47 smithi193 ceph-mon[36878]: pgmap v4767: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T02:34:49.945 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:34:49 smithi118 ceph-mon[31535]: pgmap v4768: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-03-24T02:34:49.953 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:34:49 smithi189 ceph-mon[36187]: pgmap v4768: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-03-24T02:34:49.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:34:49 smithi193 ceph-mon[36878]: pgmap v4768: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-03-24T02:34:51.287 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:34:50 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 7786.726223 secs 2024-03-24T02:34:51.453 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:34:50 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 7786.726223 secs 2024-03-24T02:34:51.459 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:34:50 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 7786.726223 secs 2024-03-24T02:34:52.695 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:34:52 smithi118 ceph-mon[31535]: pgmap v4769: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:34:52.695 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:34:52 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T02:34:52.703 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:34:52 smithi189 ceph-mon[36187]: pgmap v4769: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:34:52.703 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:34:52 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T02:34:52.710 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:34:52 smithi193 ceph-mon[36878]: pgmap v4769: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:34:52.710 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:34:52 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T02:34:53.695 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:34:53 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T02:34:53.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:34:53 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T02:34:53.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:34:53 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T02:34:53.703 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:34:53 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T02:34:53.703 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:34:53 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T02:34:53.703 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:34:53 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T02:34:53.710 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:34:53 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T02:34:53.710 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:34:53 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T02:34:53.710 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:34:53 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T02:34:54.695 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:34:54 smithi118 ceph-mon[31535]: pgmap v4770: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 6 op/s 2024-03-24T02:34:54.695 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:34:54 smithi118 ceph-mon[31535]: pgmap v4771: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 5 op/s 2024-03-24T02:34:54.703 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:34:54 smithi189 ceph-mon[36187]: pgmap v4770: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 6 op/s 2024-03-24T02:34:54.703 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:34:54 smithi189 ceph-mon[36187]: pgmap v4771: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 5 op/s 2024-03-24T02:34:54.710 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:34:54 smithi193 ceph-mon[36878]: pgmap v4770: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 6 op/s 2024-03-24T02:34:54.710 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:34:54 smithi193 ceph-mon[36878]: pgmap v4771: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 5 op/s 2024-03-24T02:34:56.695 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:34:56 smithi118 ceph-mon[31535]: pgmap v4772: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 6 op/s 2024-03-24T02:34:56.695 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:34:56 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 7791.726311 secs 2024-03-24T02:34:56.703 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:34:56 smithi189 ceph-mon[36187]: pgmap v4772: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 6 op/s 2024-03-24T02:34:56.703 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:34:56 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 7791.726311 secs 2024-03-24T02:34:56.710 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:34:56 smithi193 ceph-mon[36878]: pgmap v4772: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 6 op/s 2024-03-24T02:34:56.710 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:34:56 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 7791.726311 secs 2024-03-24T02:34:58.695 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:34:58 smithi118 ceph-mon[31535]: pgmap v4773: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 5 op/s 2024-03-24T02:34:58.702 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:34:58 smithi189 ceph-mon[36187]: pgmap v4773: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 5 op/s 2024-03-24T02:34:58.710 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:34:58 smithi193 ceph-mon[36878]: pgmap v4773: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 5 op/s 2024-03-24T02:35:00.695 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:35:00 smithi118 ceph-mon[31535]: pgmap v4774: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 5 op/s 2024-03-24T02:35:00.702 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:35:00 smithi189 ceph-mon[36187]: pgmap v4774: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 5 op/s 2024-03-24T02:35:00.709 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:35:00 smithi193 ceph-mon[36878]: pgmap v4774: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 5 op/s 2024-03-24T02:35:01.695 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:35:01 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 7796.726370 secs 2024-03-24T02:35:01.703 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:35:01 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 7796.726370 secs 2024-03-24T02:35:01.710 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:35:01 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 7796.726370 secs 2024-03-24T02:35:02.695 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:35:02 smithi118 ceph-mon[31535]: pgmap v4775: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 6 op/s 2024-03-24T02:35:02.702 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:35:02 smithi189 ceph-mon[36187]: pgmap v4775: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 6 op/s 2024-03-24T02:35:02.709 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:35:02 smithi193 ceph-mon[36878]: pgmap v4775: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 6 op/s 2024-03-24T02:35:04.445 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:35:04 smithi118 ceph-mon[31535]: pgmap v4776: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T02:35:04.703 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:35:04 smithi189 ceph-mon[36187]: pgmap v4776: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T02:35:04.709 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:35:04 smithi193 ceph-mon[36878]: pgmap v4776: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T02:35:06.695 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:35:06 smithi118 ceph-mon[31535]: pgmap v4777: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 6 op/s 2024-03-24T02:35:06.695 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:35:06 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 7801.726458 secs 2024-03-24T02:35:06.702 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:35:06 smithi189 ceph-mon[36187]: pgmap v4777: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 6 op/s 2024-03-24T02:35:06.703 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:35:06 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 7801.726458 secs 2024-03-24T02:35:06.709 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:35:06 smithi193 ceph-mon[36878]: pgmap v4777: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 6 op/s 2024-03-24T02:35:06.710 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:35:06 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 7801.726458 secs 2024-03-24T02:35:08.695 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:35:08 smithi118 ceph-mon[31535]: pgmap v4778: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-03-24T02:35:08.703 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:35:08 smithi189 ceph-mon[36187]: pgmap v4778: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-03-24T02:35:08.710 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:35:08 smithi193 ceph-mon[36878]: pgmap v4778: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-03-24T02:35:10.695 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:35:10 smithi118 ceph-mon[31535]: pgmap v4779: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-03-24T02:35:10.703 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:35:10 smithi189 ceph-mon[36187]: pgmap v4779: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-03-24T02:35:10.710 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:35:10 smithi193 ceph-mon[36878]: pgmap v4779: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-03-24T02:35:11.695 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:35:11 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 7806.726501 secs 2024-03-24T02:35:11.703 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:35:11 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 7806.726501 secs 2024-03-24T02:35:11.709 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:35:11 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 7806.726501 secs 2024-03-24T02:35:12.695 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:35:12 smithi118 ceph-mon[31535]: pgmap v4780: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 5 op/s 2024-03-24T02:35:12.703 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:35:12 smithi189 ceph-mon[36187]: pgmap v4780: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 5 op/s 2024-03-24T02:35:12.710 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:35:12 smithi193 ceph-mon[36878]: pgmap v4780: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 5 op/s 2024-03-24T02:35:14.695 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:35:14 smithi118 ceph-mon[31535]: pgmap v4781: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 5 op/s 2024-03-24T02:35:14.703 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:35:14 smithi189 ceph-mon[36187]: pgmap v4781: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 5 op/s 2024-03-24T02:35:14.709 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:35:14 smithi193 ceph-mon[36878]: pgmap v4781: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 5 op/s 2024-03-24T02:35:16.695 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:35:16 smithi118 ceph-mon[31535]: pgmap v4782: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T02:35:16.695 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:35:16 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 7811.726586 secs 2024-03-24T02:35:16.703 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:35:16 smithi189 ceph-mon[36187]: pgmap v4782: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T02:35:16.703 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:35:16 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 7811.726586 secs 2024-03-24T02:35:16.710 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:35:16 smithi193 ceph-mon[36878]: pgmap v4782: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T02:35:16.710 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:35:16 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 7811.726586 secs 2024-03-24T02:35:18.695 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:35:18 smithi118 ceph-mon[31535]: pgmap v4783: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:35:18.703 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:35:18 smithi189 ceph-mon[36187]: pgmap v4783: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:35:18.710 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:35:18 smithi193 ceph-mon[36878]: pgmap v4783: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:35:20.695 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:35:20 smithi118 ceph-mon[31535]: pgmap v4784: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T02:35:20.703 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:35:20 smithi189 ceph-mon[36187]: pgmap v4784: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T02:35:20.710 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:35:20 smithi193 ceph-mon[36878]: pgmap v4784: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T02:35:21.695 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:35:21 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 7816.726646 secs 2024-03-24T02:35:21.703 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:35:21 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 7816.726646 secs 2024-03-24T02:35:21.710 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:35:21 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 7816.726646 secs 2024-03-24T02:35:22.695 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:35:22 smithi118 ceph-mon[31535]: pgmap v4785: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T02:35:22.703 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:35:22 smithi189 ceph-mon[36187]: pgmap v4785: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T02:35:22.710 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:35:22 smithi193 ceph-mon[36878]: pgmap v4785: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T02:35:24.695 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:35:24 smithi118 ceph-mon[31535]: pgmap v4786: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 5 op/s 2024-03-24T02:35:24.703 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:35:24 smithi189 ceph-mon[36187]: pgmap v4786: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 5 op/s 2024-03-24T02:35:24.710 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:35:24 smithi193 ceph-mon[36878]: pgmap v4786: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 5 op/s 2024-03-24T02:35:26.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:35:26 smithi118 ceph-mon[31535]: pgmap v4787: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T02:35:26.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:35:26 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 7821.726733 secs 2024-03-24T02:35:26.703 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:35:26 smithi189 ceph-mon[36187]: pgmap v4787: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T02:35:26.703 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:35:26 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 7821.726733 secs 2024-03-24T02:35:26.710 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:35:26 smithi193 ceph-mon[36878]: pgmap v4787: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T02:35:26.710 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:35:26 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 7821.726733 secs 2024-03-24T02:35:28.695 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:35:28 smithi118 ceph-mon[31535]: pgmap v4788: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:35:28.703 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:35:28 smithi189 ceph-mon[36187]: pgmap v4788: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:35:28.710 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:35:28 smithi193 ceph-mon[36878]: pgmap v4788: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:35:30.695 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:35:30 smithi118 ceph-mon[31535]: pgmap v4789: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T02:35:30.703 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:35:30 smithi189 ceph-mon[36187]: pgmap v4789: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T02:35:30.710 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:35:30 smithi193 ceph-mon[36878]: pgmap v4789: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T02:35:31.695 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:35:31 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 7826.726822 secs 2024-03-24T02:35:31.703 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:35:31 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 7826.726822 secs 2024-03-24T02:35:31.709 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:35:31 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 7826.726822 secs 2024-03-24T02:35:32.695 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:35:32 smithi118 ceph-mon[31535]: pgmap v4790: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T02:35:32.703 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:35:32 smithi189 ceph-mon[36187]: pgmap v4790: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T02:35:32.709 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:35:32 smithi193 ceph-mon[36878]: pgmap v4790: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T02:35:34.695 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:35:34 smithi118 ceph-mon[31535]: pgmap v4791: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 5 op/s 2024-03-24T02:35:34.703 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:35:34 smithi189 ceph-mon[36187]: pgmap v4791: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 5 op/s 2024-03-24T02:35:34.710 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:35:34 smithi193 ceph-mon[36878]: pgmap v4791: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 5 op/s 2024-03-24T02:35:36.695 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:35:36 smithi118 ceph-mon[31535]: pgmap v4792: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T02:35:36.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:35:36 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 7831.726861 secs 2024-03-24T02:35:36.703 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:35:36 smithi189 ceph-mon[36187]: pgmap v4792: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T02:35:36.703 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:35:36 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 7831.726861 secs 2024-03-24T02:35:36.710 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:35:36 smithi193 ceph-mon[36878]: pgmap v4792: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T02:35:36.710 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:35:36 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 7831.726861 secs 2024-03-24T02:35:38.695 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:35:38 smithi118 ceph-mon[31535]: pgmap v4793: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:35:38.703 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:35:38 smithi189 ceph-mon[36187]: pgmap v4793: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:35:38.710 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:35:38 smithi193 ceph-mon[36878]: pgmap v4793: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:35:40.695 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:35:40 smithi118 ceph-mon[31535]: pgmap v4794: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T02:35:40.703 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:35:40 smithi189 ceph-mon[36187]: pgmap v4794: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T02:35:40.710 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:35:40 smithi193 ceph-mon[36878]: pgmap v4794: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T02:35:41.695 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:35:41 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 7836.726903 secs 2024-03-24T02:35:41.703 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:35:41 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 7836.726903 secs 2024-03-24T02:35:41.710 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:35:41 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 7836.726903 secs 2024-03-24T02:35:42.695 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:35:42 smithi118 ceph-mon[31535]: pgmap v4795: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T02:35:42.703 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:35:42 smithi189 ceph-mon[36187]: pgmap v4795: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T02:35:42.710 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:35:42 smithi193 ceph-mon[36878]: pgmap v4795: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T02:35:44.695 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:35:44 smithi118 ceph-mon[31535]: pgmap v4796: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 5 op/s 2024-03-24T02:35:44.703 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:35:44 smithi189 ceph-mon[36187]: pgmap v4796: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 5 op/s 2024-03-24T02:35:44.709 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:35:44 smithi193 ceph-mon[36878]: pgmap v4796: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 5 op/s 2024-03-24T02:35:46.695 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:35:46 smithi118 ceph-mon[31535]: pgmap v4797: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T02:35:46.695 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:35:46 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 7841.726989 secs 2024-03-24T02:35:46.703 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:35:46 smithi189 ceph-mon[36187]: pgmap v4797: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T02:35:46.703 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:35:46 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 7841.726989 secs 2024-03-24T02:35:46.710 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:35:46 smithi193 ceph-mon[36878]: pgmap v4797: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T02:35:46.710 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:35:46 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 7841.726989 secs 2024-03-24T02:35:48.695 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:35:48 smithi118 ceph-mon[31535]: pgmap v4798: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:35:48.706 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:35:48 smithi189 ceph-mon[36187]: pgmap v4798: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:35:48.710 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:35:48 smithi193 ceph-mon[36878]: pgmap v4798: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:35:50.695 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:35:50 smithi118 ceph-mon[31535]: pgmap v4799: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T02:35:50.703 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:35:50 smithi189 ceph-mon[36187]: pgmap v4799: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T02:35:50.709 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:35:50 smithi193 ceph-mon[36878]: pgmap v4799: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T02:35:51.445 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:35:51 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 7846.727091 secs 2024-03-24T02:35:51.703 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:35:51 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 7846.727091 secs 2024-03-24T02:35:51.709 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:35:51 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 7846.727091 secs 2024-03-24T02:35:52.695 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:35:52 smithi118 ceph-mon[31535]: pgmap v4800: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T02:35:52.703 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:35:52 smithi189 ceph-mon[36187]: pgmap v4800: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T02:35:52.710 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:35:52 smithi193 ceph-mon[36878]: pgmap v4800: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T02:35:53.695 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:35:53 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T02:35:53.703 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:35:53 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T02:35:53.710 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:35:53 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T02:35:54.695 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:35:54 smithi118 ceph-mon[31535]: pgmap v4801: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 5 op/s 2024-03-24T02:35:54.702 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:35:54 smithi189 ceph-mon[36187]: pgmap v4801: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 5 op/s 2024-03-24T02:35:54.709 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:35:54 smithi193 ceph-mon[36878]: pgmap v4801: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 5 op/s 2024-03-24T02:35:56.695 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:35:56 smithi118 ceph-mon[31535]: pgmap v4802: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T02:35:56.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:35:56 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 7851.727174 secs 2024-03-24T02:35:56.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:35:56 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T02:35:56.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:35:56 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T02:35:56.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:35:56 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T02:35:56.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:35:56 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T02:35:56.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:35:56 smithi118 ceph-mon[31535]: pgmap v4803: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T02:35:56.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:35:56 smithi118 ceph-mon[31535]: pgmap v4804: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-03-24T02:35:56.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:35:56 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T02:35:56.703 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:35:56 smithi189 ceph-mon[36187]: pgmap v4802: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T02:35:56.703 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:35:56 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 7851.727174 secs 2024-03-24T02:35:56.703 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:35:56 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T02:35:56.703 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:35:56 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T02:35:56.703 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:35:56 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T02:35:56.703 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:35:56 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T02:35:56.703 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:35:56 smithi189 ceph-mon[36187]: pgmap v4803: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T02:35:56.703 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:35:56 smithi189 ceph-mon[36187]: pgmap v4804: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-03-24T02:35:56.703 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:35:56 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T02:35:56.710 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:35:56 smithi193 ceph-mon[36878]: pgmap v4802: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T02:35:56.710 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:35:56 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 7851.727174 secs 2024-03-24T02:35:56.710 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:35:56 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T02:35:56.710 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:35:56 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T02:35:56.710 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:35:56 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T02:35:56.710 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:35:56 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T02:35:56.710 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:35:56 smithi193 ceph-mon[36878]: pgmap v4803: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T02:35:56.710 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:35:56 smithi193 ceph-mon[36878]: pgmap v4804: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-03-24T02:35:56.710 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:35:56 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T02:35:58.695 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:35:58 smithi118 ceph-mon[31535]: pgmap v4805: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-03-24T02:35:58.703 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:35:58 smithi189 ceph-mon[36187]: pgmap v4805: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-03-24T02:35:58.709 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:35:58 smithi193 ceph-mon[36878]: pgmap v4805: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-03-24T02:36:00.695 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:36:00 smithi118 ceph-mon[31535]: pgmap v4806: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-03-24T02:36:00.703 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:36:00 smithi189 ceph-mon[36187]: pgmap v4806: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-03-24T02:36:00.709 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:36:00 smithi193 ceph-mon[36878]: pgmap v4806: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-03-24T02:36:01.695 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:36:01 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 7856.727260 secs 2024-03-24T02:36:01.703 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:36:01 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 7856.727260 secs 2024-03-24T02:36:01.709 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:36:01 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 7856.727260 secs 2024-03-24T02:36:02.695 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:36:02 smithi118 ceph-mon[31535]: pgmap v4807: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 6 op/s 2024-03-24T02:36:02.703 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:36:02 smithi189 ceph-mon[36187]: pgmap v4807: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 6 op/s 2024-03-24T02:36:02.709 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:36:02 smithi193 ceph-mon[36878]: pgmap v4807: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 6 op/s 2024-03-24T02:36:04.695 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:36:04 smithi118 ceph-mon[31535]: pgmap v4808: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 5 op/s 2024-03-24T02:36:04.703 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:36:04 smithi189 ceph-mon[36187]: pgmap v4808: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 5 op/s 2024-03-24T02:36:04.709 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:36:04 smithi193 ceph-mon[36878]: pgmap v4808: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 5 op/s 2024-03-24T02:36:06.695 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:36:06 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 7861.727343 secs 2024-03-24T02:36:06.695 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:36:06 smithi118 ceph-mon[31535]: pgmap v4809: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 5 op/s 2024-03-24T02:36:06.703 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:36:06 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 7861.727343 secs 2024-03-24T02:36:06.703 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:36:06 smithi189 ceph-mon[36187]: pgmap v4809: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 5 op/s 2024-03-24T02:36:06.709 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:36:06 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 7861.727343 secs 2024-03-24T02:36:06.710 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:36:06 smithi193 ceph-mon[36878]: pgmap v4809: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 5 op/s 2024-03-24T02:36:08.695 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:36:08 smithi118 ceph-mon[31535]: pgmap v4810: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:36:08.703 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:36:08 smithi189 ceph-mon[36187]: pgmap v4810: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:36:08.709 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:36:08 smithi193 ceph-mon[36878]: pgmap v4810: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:36:10.695 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:36:10 smithi118 ceph-mon[31535]: pgmap v4811: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-24T02:36:10.703 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:36:10 smithi189 ceph-mon[36187]: pgmap v4811: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-24T02:36:10.709 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:36:10 smithi193 ceph-mon[36878]: pgmap v4811: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-24T02:36:11.695 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:36:11 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 7866.727434 secs 2024-03-24T02:36:11.703 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:36:11 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 7866.727434 secs 2024-03-24T02:36:11.710 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:36:11 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 7866.727434 secs 2024-03-24T02:36:12.710 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:36:12 smithi193 ceph-mon[36878]: pgmap v4812: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:36:12.945 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:36:12 smithi118 ceph-mon[31535]: pgmap v4812: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:36:12.953 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:36:12 smithi189 ceph-mon[36187]: pgmap v4812: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:36:14.945 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:36:14 smithi118 ceph-mon[31535]: pgmap v4813: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T02:36:14.953 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:36:14 smithi189 ceph-mon[36187]: pgmap v4813: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T02:36:14.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:36:14 smithi193 ceph-mon[36878]: pgmap v4813: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T02:36:16.754 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:36:16 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 7871.727522 secs 2024-03-24T02:36:16.755 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:36:16 smithi189 ceph-mon[36187]: pgmap v4814: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T02:36:16.762 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:36:16 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 7871.727522 secs 2024-03-24T02:36:16.762 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:36:16 smithi193 ceph-mon[36878]: pgmap v4814: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T02:36:16.945 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:36:16 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 7871.727522 secs 2024-03-24T02:36:16.945 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:36:16 smithi118 ceph-mon[31535]: pgmap v4814: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T02:36:18.945 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:36:18 smithi118 ceph-mon[31535]: pgmap v4815: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:36:18.953 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:36:18 smithi189 ceph-mon[36187]: pgmap v4815: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:36:18.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:36:18 smithi193 ceph-mon[36878]: pgmap v4815: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:36:20.945 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:36:20 smithi118 ceph-mon[31535]: pgmap v4816: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-24T02:36:20.953 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:36:20 smithi189 ceph-mon[36187]: pgmap v4816: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-24T02:36:20.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:36:20 smithi193 ceph-mon[36878]: pgmap v4816: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-24T02:36:21.911 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:36:21 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 7876.727606 secs 2024-03-24T02:36:21.953 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:36:21 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 7876.727606 secs 2024-03-24T02:36:21.959 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:36:21 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 7876.727606 secs 2024-03-24T02:36:22.945 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:36:22 smithi118 ceph-mon[31535]: pgmap v4817: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:36:22.953 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:36:22 smithi189 ceph-mon[36187]: pgmap v4817: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:36:22.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:36:22 smithi193 ceph-mon[36878]: pgmap v4817: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:36:24.945 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:36:24 smithi118 ceph-mon[31535]: pgmap v4818: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T02:36:24.953 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:36:24 smithi189 ceph-mon[36187]: pgmap v4818: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T02:36:24.959 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:36:24 smithi193 ceph-mon[36878]: pgmap v4818: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T02:36:26.945 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:36:26 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 7881.727659 secs 2024-03-24T02:36:26.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:36:26 smithi118 ceph-mon[31535]: pgmap v4819: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T02:36:26.953 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:36:26 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 7881.727659 secs 2024-03-24T02:36:26.953 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:36:26 smithi189 ceph-mon[36187]: pgmap v4819: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T02:36:26.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:36:26 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 7881.727659 secs 2024-03-24T02:36:26.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:36:26 smithi193 ceph-mon[36878]: pgmap v4819: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T02:36:28.945 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:36:28 smithi118 ceph-mon[31535]: pgmap v4820: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:36:28.953 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:36:28 smithi189 ceph-mon[36187]: pgmap v4820: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:36:28.959 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:36:28 smithi193 ceph-mon[36878]: pgmap v4820: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:36:30.945 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:36:30 smithi118 ceph-mon[31535]: pgmap v4821: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-24T02:36:30.953 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:36:30 smithi189 ceph-mon[36187]: pgmap v4821: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-24T02:36:30.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:36:30 smithi193 ceph-mon[36878]: pgmap v4821: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-24T02:36:31.945 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:36:31 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 7886.727746 secs 2024-03-24T02:36:31.953 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:36:31 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 7886.727746 secs 2024-03-24T02:36:31.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:36:31 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 7886.727746 secs 2024-03-24T02:36:32.945 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:36:32 smithi118 ceph-mon[31535]: pgmap v4822: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:36:32.953 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:36:32 smithi189 ceph-mon[36187]: pgmap v4822: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:36:32.959 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:36:32 smithi193 ceph-mon[36878]: pgmap v4822: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:36:34.945 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:36:34 smithi118 ceph-mon[31535]: pgmap v4823: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T02:36:34.953 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:36:34 smithi189 ceph-mon[36187]: pgmap v4823: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T02:36:34.959 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:36:34 smithi193 ceph-mon[36878]: pgmap v4823: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T02:36:36.945 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:36:36 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 7891.727801 secs 2024-03-24T02:36:36.945 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:36:36 smithi118 ceph-mon[31535]: pgmap v4824: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T02:36:36.953 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:36:36 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 7891.727801 secs 2024-03-24T02:36:36.953 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:36:36 smithi189 ceph-mon[36187]: pgmap v4824: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T02:36:36.959 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:36:36 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 7891.727801 secs 2024-03-24T02:36:36.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:36:36 smithi193 ceph-mon[36878]: pgmap v4824: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T02:36:38.945 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:36:38 smithi118 ceph-mon[31535]: pgmap v4825: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:36:38.953 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:36:38 smithi189 ceph-mon[36187]: pgmap v4825: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:36:38.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:36:38 smithi193 ceph-mon[36878]: pgmap v4825: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:36:40.695 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:36:40 smithi118 ceph-mon[31535]: pgmap v4826: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-24T02:36:40.953 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:36:40 smithi189 ceph-mon[36187]: pgmap v4826: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-24T02:36:40.959 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:36:40 smithi193 ceph-mon[36878]: pgmap v4826: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-24T02:36:41.945 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:36:41 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 7896.727887 secs 2024-03-24T02:36:41.953 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:36:41 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 7896.727887 secs 2024-03-24T02:36:41.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:36:41 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 7896.727887 secs 2024-03-24T02:36:42.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:36:42 smithi118 ceph-mon[31535]: pgmap v4827: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:36:42.953 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:36:42 smithi189 ceph-mon[36187]: pgmap v4827: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:36:42.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:36:42 smithi193 ceph-mon[36878]: pgmap v4827: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:36:43.945 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:36:43 smithi118 ceph-mon[31535]: pgmap v4828: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-03-24T02:36:43.952 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:36:43 smithi189 ceph-mon[36187]: pgmap v4828: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-03-24T02:36:43.959 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:36:43 smithi193 ceph-mon[36878]: pgmap v4828: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-03-24T02:36:46.695 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:36:46 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 7901.727973 secs 2024-03-24T02:36:46.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:36:46 smithi118 ceph-mon[31535]: pgmap v4829: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 6 op/s 2024-03-24T02:36:46.703 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:36:46 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 7901.727973 secs 2024-03-24T02:36:46.703 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:36:46 smithi189 ceph-mon[36187]: pgmap v4829: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 6 op/s 2024-03-24T02:36:46.710 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:36:46 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 7901.727973 secs 2024-03-24T02:36:46.710 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:36:46 smithi193 ceph-mon[36878]: pgmap v4829: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 6 op/s 2024-03-24T02:36:48.695 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:36:48 smithi118 ceph-mon[31535]: pgmap v4830: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-03-24T02:36:48.702 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:36:48 smithi189 ceph-mon[36187]: pgmap v4830: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-03-24T02:36:48.710 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:36:48 smithi193 ceph-mon[36878]: pgmap v4830: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-03-24T02:36:50.695 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:36:50 smithi118 ceph-mon[31535]: pgmap v4831: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-24T02:36:50.703 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:36:50 smithi189 ceph-mon[36187]: pgmap v4831: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-24T02:36:50.709 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:36:50 smithi193 ceph-mon[36878]: pgmap v4831: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-24T02:36:51.695 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:36:51 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 7906.728068 secs 2024-03-24T02:36:51.703 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:36:51 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 7906.728068 secs 2024-03-24T02:36:51.710 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:36:51 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 7906.728068 secs 2024-03-24T02:36:52.695 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:36:52 smithi118 ceph-mon[31535]: pgmap v4832: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-03-24T02:36:52.703 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:36:52 smithi189 ceph-mon[36187]: pgmap v4832: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-03-24T02:36:52.709 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:36:52 smithi193 ceph-mon[36878]: pgmap v4832: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-03-24T02:36:54.703 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:36:54 smithi189 ceph-mon[36187]: pgmap v4833: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-03-24T02:36:54.709 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:36:54 smithi193 ceph-mon[36878]: pgmap v4833: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-03-24T02:36:54.945 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:36:54 smithi118 ceph-mon[31535]: pgmap v4833: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-03-24T02:36:56.695 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:36:56 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 7911.728158 secs 2024-03-24T02:36:56.695 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:36:56 smithi118 ceph-mon[31535]: pgmap v4834: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T02:36:56.695 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:36:56 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T02:36:56.703 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:36:56 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 7911.728158 secs 2024-03-24T02:36:56.703 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:36:56 smithi189 ceph-mon[36187]: pgmap v4834: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T02:36:56.703 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:36:56 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T02:36:56.710 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:36:56 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 7911.728158 secs 2024-03-24T02:36:56.710 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:36:56 smithi193 ceph-mon[36878]: pgmap v4834: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T02:36:56.710 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:36:56 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T02:36:57.710 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:36:57 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-24T02:36:57.710 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:36:57 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-24T02:36:57.710 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:36:57 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-24T02:36:57.710 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:36:57 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-03-24T02:36:57.710 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:36:57 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T02:36:57.711 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:36:57 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T02:36:57.711 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:36:57 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T02:36:57.711 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:36:57 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T02:36:57.711 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:36:57 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-24T02:36:57.711 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:36:57 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-24T02:36:57.711 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:36:57 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-24T02:36:57.711 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:36:57 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-03-24T02:36:57.711 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:36:57 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T02:36:57.711 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:36:57 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T02:36:57.711 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:36:57 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T02:36:57.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:36:57 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-24T02:36:57.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:36:57 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-24T02:36:57.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:36:57 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-24T02:36:57.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:36:57 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-03-24T02:36:57.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:36:57 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T02:36:57.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:36:57 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T02:36:57.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:36:57 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T02:36:57.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:36:57 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T02:36:57.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:36:57 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-24T02:36:57.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:36:57 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-24T02:36:57.947 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:36:57 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-24T02:36:57.947 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:36:57 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-03-24T02:36:57.947 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:36:57 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T02:36:57.947 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:36:57 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T02:36:57.947 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:36:57 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T02:36:57.953 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:36:57 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-24T02:36:57.953 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:36:57 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-24T02:36:57.953 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:36:57 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-24T02:36:57.953 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:36:57 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-03-24T02:36:57.953 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:36:57 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T02:36:57.953 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:36:57 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T02:36:57.953 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:36:57 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T02:36:57.953 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:36:57 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T02:36:57.953 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:36:57 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-24T02:36:57.953 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:36:57 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-24T02:36:57.953 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:36:57 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-24T02:36:57.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:36:57 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-03-24T02:36:57.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:36:57 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T02:36:57.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:36:57 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T02:36:57.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:36:57 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T02:36:58.710 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:36:58 smithi193 ceph-mon[36878]: Adjusting osd_memory_target on smithi118 to 176.7M 2024-03-24T02:36:58.710 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:36:58 smithi193 ceph-mon[36878]: Unable to set osd_memory_target on smithi118 to 185365196: error parsing value: Value '185365196' is below minimum 939524096 2024-03-24T02:36:58.710 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:36:58 smithi193 ceph-mon[36878]: Adjusting osd_memory_target on smithi189 to 174.1M 2024-03-24T02:36:58.710 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:36:58 smithi193 ceph-mon[36878]: Unable to set osd_memory_target on smithi189 to 182635622: error parsing value: Value '182635622' is below minimum 939524096 2024-03-24T02:36:58.710 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:36:58 smithi193 ceph-mon[36878]: pgmap v4835: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 5 op/s 2024-03-24T02:36:58.945 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:36:58 smithi118 ceph-mon[31535]: Adjusting osd_memory_target on smithi118 to 176.7M 2024-03-24T02:36:58.945 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:36:58 smithi118 ceph-mon[31535]: Unable to set osd_memory_target on smithi118 to 185365196: error parsing value: Value '185365196' is below minimum 939524096 2024-03-24T02:36:58.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:36:58 smithi118 ceph-mon[31535]: Adjusting osd_memory_target on smithi189 to 174.1M 2024-03-24T02:36:58.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:36:58 smithi118 ceph-mon[31535]: Unable to set osd_memory_target on smithi189 to 182635622: error parsing value: Value '182635622' is below minimum 939524096 2024-03-24T02:36:58.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:36:58 smithi118 ceph-mon[31535]: pgmap v4835: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 5 op/s 2024-03-24T02:36:58.953 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:36:58 smithi189 ceph-mon[36187]: Adjusting osd_memory_target on smithi118 to 176.7M 2024-03-24T02:36:58.953 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:36:58 smithi189 ceph-mon[36187]: Unable to set osd_memory_target on smithi118 to 185365196: error parsing value: Value '185365196' is below minimum 939524096 2024-03-24T02:36:58.953 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:36:58 smithi189 ceph-mon[36187]: Adjusting osd_memory_target on smithi189 to 174.1M 2024-03-24T02:36:58.953 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:36:58 smithi189 ceph-mon[36187]: Unable to set osd_memory_target on smithi189 to 182635622: error parsing value: Value '182635622' is below minimum 939524096 2024-03-24T02:36:58.953 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:36:58 smithi189 ceph-mon[36187]: pgmap v4835: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 5 op/s 2024-03-24T02:37:00.945 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:37:00 smithi118 ceph-mon[31535]: pgmap v4836: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 6 op/s 2024-03-24T02:37:00.953 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:37:00 smithi189 ceph-mon[36187]: pgmap v4836: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 6 op/s 2024-03-24T02:37:00.959 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:37:00 smithi193 ceph-mon[36878]: pgmap v4836: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 6 op/s 2024-03-24T02:37:01.945 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:37:01 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 7916.728244 secs 2024-03-24T02:37:01.953 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:37:01 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 7916.728244 secs 2024-03-24T02:37:01.959 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:37:01 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 7916.728244 secs 2024-03-24T02:37:02.945 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:37:02 smithi118 ceph-mon[31535]: pgmap v4837: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 5 op/s 2024-03-24T02:37:02.953 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:37:02 smithi189 ceph-mon[36187]: pgmap v4837: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 5 op/s 2024-03-24T02:37:02.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:37:02 smithi193 ceph-mon[36878]: pgmap v4837: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 5 op/s 2024-03-24T02:37:04.945 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:37:04 smithi118 ceph-mon[31535]: pgmap v4838: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 6 op/s 2024-03-24T02:37:04.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:37:04 smithi189 ceph-mon[36187]: pgmap v4838: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 6 op/s 2024-03-24T02:37:04.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:37:04 smithi193 ceph-mon[36878]: pgmap v4838: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 6 op/s 2024-03-24T02:37:06.748 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:37:06 smithi118 ceph-mon[31535]: pgmap v4839: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 6 op/s 2024-03-24T02:37:06.748 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:37:06 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 7921.728305 secs 2024-03-24T02:37:06.953 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:37:06 smithi189 ceph-mon[36187]: pgmap v4839: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 6 op/s 2024-03-24T02:37:06.953 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:37:06 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 7921.728305 secs 2024-03-24T02:37:06.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:37:06 smithi193 ceph-mon[36878]: pgmap v4839: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 6 op/s 2024-03-24T02:37:06.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:37:06 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 7921.728305 secs 2024-03-24T02:37:08.945 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:37:08 smithi118 ceph-mon[31535]: pgmap v4840: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 5 op/s 2024-03-24T02:37:08.953 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:37:08 smithi189 ceph-mon[36187]: pgmap v4840: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 5 op/s 2024-03-24T02:37:08.959 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:37:08 smithi193 ceph-mon[36878]: pgmap v4840: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 5 op/s 2024-03-24T02:37:10.945 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:37:10 smithi118 ceph-mon[31535]: pgmap v4841: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T02:37:10.952 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:37:10 smithi189 ceph-mon[36187]: pgmap v4841: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T02:37:10.959 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:37:10 smithi193 ceph-mon[36878]: pgmap v4841: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T02:37:11.945 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:37:11 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 7926.728365 secs 2024-03-24T02:37:11.953 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:37:11 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 7926.728365 secs 2024-03-24T02:37:11.959 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:37:11 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 7926.728365 secs 2024-03-24T02:37:12.945 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:37:12 smithi118 ceph-mon[31535]: pgmap v4842: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T02:37:12.953 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:37:12 smithi189 ceph-mon[36187]: pgmap v4842: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T02:37:12.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:37:12 smithi193 ceph-mon[36878]: pgmap v4842: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T02:37:14.945 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:37:14 smithi118 ceph-mon[31535]: pgmap v4843: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 5 op/s 2024-03-24T02:37:14.953 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:37:14 smithi189 ceph-mon[36187]: pgmap v4843: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 5 op/s 2024-03-24T02:37:14.959 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:37:14 smithi193 ceph-mon[36878]: pgmap v4843: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 5 op/s 2024-03-24T02:37:16.762 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:37:16 smithi193 ceph-mon[36878]: pgmap v4844: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T02:37:16.762 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:37:16 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 7931.728423 secs 2024-03-24T02:37:16.945 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:37:16 smithi118 ceph-mon[31535]: pgmap v4844: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T02:37:16.945 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:37:16 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 7931.728423 secs 2024-03-24T02:37:16.953 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:37:16 smithi189 ceph-mon[36187]: pgmap v4844: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T02:37:16.953 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:37:16 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 7931.728423 secs 2024-03-24T02:37:17.014 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T02:37:17.014+0000 7f81db400640 1 -- 172.21.15.118:0/659686424 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- auth(proto 2 132 bytes epoch 0) v1 -- 0x7f81c4006290 con 0x5622651a13f0 2024-03-24T02:37:17.014 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T02:37:17.014+0000 7f81ee000640 1 -- 172.21.15.118:0/1693767697 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- auth(proto 2 132 bytes epoch 0) v1 -- 0x7f81c8006590 con 0x7f81f81052d0 2024-03-24T02:37:17.015 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T02:37:17.015+0000 7f81ef400640 1 -- 172.21.15.118:0/1693767697 <== mon.2 v2:172.21.15.189:3300/0 64 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 783+0+0 (secure 0 0 0) 0x7f81c8006590 con 0x7f81f81052d0 2024-03-24T02:37:17.015 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T02:37:17.015+0000 7f81e4c00640 1 -- 172.21.15.118:0/659686424 <== mon.1 v2:172.21.15.193:3300/0 64 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 612+0+0 (secure 0 0 0) 0x7f81c4006290 con 0x5622651a13f0 2024-03-24T02:37:17.015 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T02:37:17.015+0000 7f81ef400640 1 -- 172.21.15.118:0/1693767697 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f81c80036c0 con 0x7f81f81052d0 2024-03-24T02:37:17.015 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T02:37:17.015+0000 7f81e4c00640 1 -- 172.21.15.118:0/659686424 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f81c4006460 con 0x5622651a13f0 2024-03-24T02:37:17.015 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T02:37:17.015+0000 7f81ef400640 1 -- 172.21.15.118:0/1693767697 <== mon.2 v2:172.21.15.189:3300/0 65 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (secure 0 0 0) 0x7f81c80036c0 con 0x7f81f81052d0 2024-03-24T02:37:17.015 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T02:37:17.015+0000 7f81e4c00640 1 -- 172.21.15.118:0/659686424 <== mon.1 v2:172.21.15.193:3300/0 65 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 409+0+0 (secure 0 0 0) 0x7f81c4006460 con 0x5622651a13f0 2024-03-24T02:37:18.945 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:37:18 smithi118 ceph-mon[31535]: pgmap v4845: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:37:18.953 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:37:18 smithi189 ceph-mon[36187]: pgmap v4845: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:37:18.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:37:18 smithi193 ceph-mon[36878]: pgmap v4845: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:37:20.945 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:37:20 smithi118 ceph-mon[31535]: pgmap v4846: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T02:37:20.953 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:37:20 smithi189 ceph-mon[36187]: pgmap v4846: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T02:37:20.959 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:37:20 smithi193 ceph-mon[36878]: pgmap v4846: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T02:37:21.945 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:37:21 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 7936.728512 secs 2024-03-24T02:37:21.953 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:37:21 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 7936.728512 secs 2024-03-24T02:37:21.959 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:37:21 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 7936.728512 secs 2024-03-24T02:37:22.945 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:37:22 smithi118 ceph-mon[31535]: pgmap v4847: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T02:37:22.953 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:37:22 smithi189 ceph-mon[36187]: pgmap v4847: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T02:37:22.959 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:37:22 smithi193 ceph-mon[36878]: pgmap v4847: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T02:37:24.945 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:37:24 smithi118 ceph-mon[31535]: pgmap v4848: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 5 op/s 2024-03-24T02:37:24.953 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:37:24 smithi189 ceph-mon[36187]: pgmap v4848: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 5 op/s 2024-03-24T02:37:24.959 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:37:24 smithi193 ceph-mon[36878]: pgmap v4848: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 5 op/s 2024-03-24T02:37:26.935 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:37:26 smithi118 ceph-mon[31535]: pgmap v4849: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T02:37:26.936 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:37:26 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 7941.728594 secs 2024-03-24T02:37:26.953 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:37:26 smithi189 ceph-mon[36187]: pgmap v4849: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T02:37:26.953 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:37:26 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 7941.728594 secs 2024-03-24T02:37:26.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:37:26 smithi193 ceph-mon[36878]: pgmap v4849: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T02:37:26.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:37:26 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 7941.728594 secs 2024-03-24T02:37:28.945 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:37:28 smithi118 ceph-mon[31535]: pgmap v4850: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:37:28.953 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:37:28 smithi189 ceph-mon[36187]: pgmap v4850: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:37:28.959 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:37:28 smithi193 ceph-mon[36878]: pgmap v4850: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:37:29.945 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:37:29 smithi118 ceph-mon[31535]: pgmap v4851: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T02:37:29.952 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:37:29 smithi189 ceph-mon[36187]: pgmap v4851: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T02:37:29.959 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:37:29 smithi193 ceph-mon[36878]: pgmap v4851: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T02:37:31.445 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:37:30 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 7946.728683 secs 2024-03-24T02:37:31.453 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:37:30 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 7946.728683 secs 2024-03-24T02:37:31.460 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:37:30 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 7946.728683 secs 2024-03-24T02:37:32.445 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:37:31 smithi118 ceph-mon[31535]: pgmap v4852: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T02:37:32.453 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:37:31 smithi189 ceph-mon[36187]: pgmap v4852: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T02:37:32.460 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:37:31 smithi193 ceph-mon[36878]: pgmap v4852: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T02:37:34.445 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:37:33 smithi118 ceph-mon[31535]: pgmap v4853: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 5 op/s 2024-03-24T02:37:34.453 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:37:33 smithi189 ceph-mon[36187]: pgmap v4853: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 5 op/s 2024-03-24T02:37:34.459 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:37:33 smithi193 ceph-mon[36878]: pgmap v4853: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 5 op/s 2024-03-24T02:37:36.445 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:37:36 smithi118 ceph-mon[31535]: pgmap v4854: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T02:37:36.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:37:36 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 7951.728745 secs 2024-03-24T02:37:36.453 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:37:36 smithi189 ceph-mon[36187]: pgmap v4854: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T02:37:36.453 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:37:36 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 7951.728745 secs 2024-03-24T02:37:36.459 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:37:36 smithi193 ceph-mon[36878]: pgmap v4854: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T02:37:36.460 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:37:36 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 7951.728745 secs 2024-03-24T02:37:38.445 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:37:38 smithi118 ceph-mon[31535]: pgmap v4855: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:37:38.453 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:37:38 smithi189 ceph-mon[36187]: pgmap v4855: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:37:38.460 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:37:38 smithi193 ceph-mon[36878]: pgmap v4855: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:37:40.445 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:37:40 smithi118 ceph-mon[31535]: pgmap v4856: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T02:37:40.453 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:37:40 smithi189 ceph-mon[36187]: pgmap v4856: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T02:37:40.460 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:37:40 smithi193 ceph-mon[36878]: pgmap v4856: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T02:37:41.447 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:37:41 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 7956.728805 secs 2024-03-24T02:37:41.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:37:41 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 7956.728805 secs 2024-03-24T02:37:41.460 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:37:41 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 7956.728805 secs 2024-03-24T02:37:42.295 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:37:42 smithi118 ceph-mon[31535]: pgmap v4857: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T02:37:42.453 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:37:42 smithi189 ceph-mon[36187]: pgmap v4857: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T02:37:42.460 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:37:42 smithi193 ceph-mon[36878]: pgmap v4857: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T02:37:44.445 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:37:44 smithi118 ceph-mon[31535]: pgmap v4858: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 5 op/s 2024-03-24T02:37:44.453 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:37:44 smithi189 ceph-mon[36187]: pgmap v4858: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 5 op/s 2024-03-24T02:37:44.460 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:37:44 smithi193 ceph-mon[36878]: pgmap v4858: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 5 op/s 2024-03-24T02:37:46.445 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:37:46 smithi118 ceph-mon[31535]: pgmap v4859: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T02:37:46.445 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:37:46 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 7961.728868 secs 2024-03-24T02:37:46.453 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:37:46 smithi189 ceph-mon[36187]: pgmap v4859: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T02:37:46.453 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:37:46 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 7961.728868 secs 2024-03-24T02:37:46.460 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:37:46 smithi193 ceph-mon[36878]: pgmap v4859: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T02:37:46.460 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:37:46 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 7961.728868 secs 2024-03-24T02:37:48.445 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:37:48 smithi118 ceph-mon[31535]: pgmap v4860: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:37:48.453 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:37:48 smithi189 ceph-mon[36187]: pgmap v4860: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:37:48.460 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:37:48 smithi193 ceph-mon[36878]: pgmap v4860: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:37:50.445 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:37:50 smithi118 ceph-mon[31535]: pgmap v4861: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T02:37:50.453 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:37:50 smithi189 ceph-mon[36187]: pgmap v4861: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T02:37:50.459 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:37:50 smithi193 ceph-mon[36878]: pgmap v4861: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T02:37:51.445 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:37:51 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 7966.728958 secs 2024-03-24T02:37:51.453 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:37:51 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 7966.728958 secs 2024-03-24T02:37:51.460 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:37:51 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 7966.728958 secs 2024-03-24T02:37:52.445 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:37:52 smithi118 ceph-mon[31535]: pgmap v4862: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T02:37:52.453 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:37:52 smithi189 ceph-mon[36187]: pgmap v4862: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T02:37:52.460 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:37:52 smithi193 ceph-mon[36878]: pgmap v4862: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T02:37:54.445 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:37:54 smithi118 ceph-mon[31535]: pgmap v4863: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 5 op/s 2024-03-24T02:37:54.453 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:37:54 smithi189 ceph-mon[36187]: pgmap v4863: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 5 op/s 2024-03-24T02:37:54.460 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:37:54 smithi193 ceph-mon[36878]: pgmap v4863: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 5 op/s 2024-03-24T02:37:56.445 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:37:56 smithi118 ceph-mon[31535]: pgmap v4864: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T02:37:56.445 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:37:56 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 7971.729047 secs 2024-03-24T02:37:56.453 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:37:56 smithi189 ceph-mon[36187]: pgmap v4864: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T02:37:56.453 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:37:56 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 7971.729047 secs 2024-03-24T02:37:56.459 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:37:56 smithi193 ceph-mon[36878]: pgmap v4864: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T02:37:56.460 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:37:56 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 7971.729047 secs 2024-03-24T02:37:58.445 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:37:58 smithi118 ceph-mon[31535]: pgmap v4865: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:37:58.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:37:58 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T02:37:58.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:37:58 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-24T02:37:58.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:37:58 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-24T02:37:58.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:37:58 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-03-24T02:37:58.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:37:58 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-03-24T02:37:58.453 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:37:58 smithi189 ceph-mon[36187]: pgmap v4865: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:37:58.453 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:37:58 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T02:37:58.453 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:37:58 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-24T02:37:58.453 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:37:58 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-24T02:37:58.453 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:37:58 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-03-24T02:37:58.453 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:37:58 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-03-24T02:37:58.460 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:37:58 smithi193 ceph-mon[36878]: pgmap v4865: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:37:58.460 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:37:58 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T02:37:58.460 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:37:58 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-24T02:37:58.460 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:37:58 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-24T02:37:58.460 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:37:58 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-03-24T02:37:58.460 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:37:58 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-03-24T02:37:59.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:37:59 smithi118 ceph-mon[31535]: Adjusting osd_memory_target on smithi193 to 174.1M 2024-03-24T02:37:59.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:37:59 smithi118 ceph-mon[31535]: Unable to set osd_memory_target on smithi193 to 182639923: error parsing value: Value '182639923' is below minimum 939524096 2024-03-24T02:37:59.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:37:59 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T02:37:59.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:37:59 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T02:37:59.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:37:59 smithi118 ceph-mon[31535]: pgmap v4866: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 6 op/s 2024-03-24T02:37:59.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:37:59 smithi118 ceph-mon[31535]: pgmap v4867: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 5 op/s 2024-03-24T02:37:59.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:37:59 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T02:37:59.453 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:37:59 smithi189 ceph-mon[36187]: Adjusting osd_memory_target on smithi193 to 174.1M 2024-03-24T02:37:59.453 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:37:59 smithi189 ceph-mon[36187]: Unable to set osd_memory_target on smithi193 to 182639923: error parsing value: Value '182639923' is below minimum 939524096 2024-03-24T02:37:59.453 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:37:59 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T02:37:59.453 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:37:59 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T02:37:59.453 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:37:59 smithi189 ceph-mon[36187]: pgmap v4866: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 6 op/s 2024-03-24T02:37:59.453 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:37:59 smithi189 ceph-mon[36187]: pgmap v4867: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 5 op/s 2024-03-24T02:37:59.453 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:37:59 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T02:37:59.460 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:37:59 smithi193 ceph-mon[36878]: Adjusting osd_memory_target on smithi193 to 174.1M 2024-03-24T02:37:59.460 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:37:59 smithi193 ceph-mon[36878]: Unable to set osd_memory_target on smithi193 to 182639923: error parsing value: Value '182639923' is below minimum 939524096 2024-03-24T02:37:59.460 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:37:59 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T02:37:59.460 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:37:59 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T02:37:59.460 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:37:59 smithi193 ceph-mon[36878]: pgmap v4866: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 6 op/s 2024-03-24T02:37:59.460 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:37:59 smithi193 ceph-mon[36878]: pgmap v4867: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 5 op/s 2024-03-24T02:37:59.460 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:37:59 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T02:38:00.945 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:38:00 smithi118 ceph-mon[31535]: pgmap v4868: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 6 op/s 2024-03-24T02:38:00.953 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:38:00 smithi189 ceph-mon[36187]: pgmap v4868: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 6 op/s 2024-03-24T02:38:00.959 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:38:00 smithi193 ceph-mon[36878]: pgmap v4868: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 6 op/s 2024-03-24T02:38:01.945 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:38:01 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 7976.729106 secs 2024-03-24T02:38:01.953 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:38:01 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 7976.729106 secs 2024-03-24T02:38:01.959 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:38:01 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 7976.729106 secs 2024-03-24T02:38:02.945 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:38:02 smithi118 ceph-mon[31535]: pgmap v4869: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 5 op/s 2024-03-24T02:38:02.953 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:38:02 smithi189 ceph-mon[36187]: pgmap v4869: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 5 op/s 2024-03-24T02:38:02.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:38:02 smithi193 ceph-mon[36878]: pgmap v4869: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 5 op/s 2024-03-24T02:38:04.945 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:38:04 smithi118 ceph-mon[31535]: pgmap v4870: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 6 op/s 2024-03-24T02:38:04.952 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:38:04 smithi189 ceph-mon[36187]: pgmap v4870: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 6 op/s 2024-03-24T02:38:04.959 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:38:04 smithi193 ceph-mon[36878]: pgmap v4870: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 6 op/s 2024-03-24T02:38:06.445 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:38:05 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 7981.729166 secs 2024-03-24T02:38:06.453 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:38:05 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 7981.729166 secs 2024-03-24T02:38:06.460 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:38:05 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 7981.729166 secs 2024-03-24T02:38:07.695 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:38:07 smithi118 ceph-mon[31535]: pgmap v4871: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 6 op/s 2024-03-24T02:38:07.703 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:38:07 smithi189 ceph-mon[36187]: pgmap v4871: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 6 op/s 2024-03-24T02:38:07.709 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:38:07 smithi193 ceph-mon[36878]: pgmap v4871: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 6 op/s 2024-03-24T02:38:08.945 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:38:08 smithi118 ceph-mon[31535]: pgmap v4872: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 5 op/s 2024-03-24T02:38:08.953 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:38:08 smithi189 ceph-mon[36187]: pgmap v4872: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 5 op/s 2024-03-24T02:38:08.959 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:38:08 smithi193 ceph-mon[36878]: pgmap v4872: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 5 op/s 2024-03-24T02:38:10.945 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:38:10 smithi118 ceph-mon[31535]: pgmap v4873: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T02:38:10.953 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:38:10 smithi189 ceph-mon[36187]: pgmap v4873: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T02:38:10.959 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:38:10 smithi193 ceph-mon[36878]: pgmap v4873: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T02:38:11.945 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:38:11 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 7986.729225 secs 2024-03-24T02:38:11.953 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:38:11 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 7986.729225 secs 2024-03-24T02:38:11.959 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:38:11 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 7986.729225 secs 2024-03-24T02:38:12.945 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:38:12 smithi118 ceph-mon[31535]: pgmap v4874: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T02:38:12.953 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:38:12 smithi189 ceph-mon[36187]: pgmap v4874: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T02:38:12.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:38:12 smithi193 ceph-mon[36878]: pgmap v4874: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T02:38:14.945 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:38:14 smithi118 ceph-mon[31535]: pgmap v4875: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-03-24T02:38:14.953 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:38:14 smithi189 ceph-mon[36187]: pgmap v4875: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-03-24T02:38:14.959 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:38:14 smithi193 ceph-mon[36878]: pgmap v4875: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-03-24T02:38:16.445 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:38:15 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 7991.729314 secs 2024-03-24T02:38:16.453 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:38:15 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 7991.729314 secs 2024-03-24T02:38:16.459 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:38:15 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 7991.729314 secs 2024-03-24T02:38:17.445 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:38:16 smithi118 ceph-mon[31535]: pgmap v4876: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:38:17.453 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:38:16 smithi189 ceph-mon[36187]: pgmap v4876: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:38:17.460 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:38:16 smithi193 ceph-mon[36878]: pgmap v4876: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:38:18.945 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:38:18 smithi118 ceph-mon[31535]: pgmap v4877: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T02:38:18.953 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:38:18 smithi189 ceph-mon[36187]: pgmap v4877: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T02:38:18.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:38:18 smithi193 ceph-mon[36878]: pgmap v4877: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T02:38:20.945 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:38:20 smithi118 ceph-mon[31535]: pgmap v4878: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T02:38:20.953 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:38:20 smithi189 ceph-mon[36187]: pgmap v4878: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T02:38:20.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:38:20 smithi193 ceph-mon[36878]: pgmap v4878: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T02:38:21.945 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:38:21 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 7996.729402 secs 2024-03-24T02:38:21.953 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:38:21 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 7996.729402 secs 2024-03-24T02:38:21.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:38:21 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 7996.729402 secs 2024-03-24T02:38:22.945 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:38:22 smithi118 ceph-mon[31535]: pgmap v4879: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T02:38:22.953 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:38:22 smithi189 ceph-mon[36187]: pgmap v4879: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T02:38:22.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:38:22 smithi193 ceph-mon[36878]: pgmap v4879: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T02:38:24.945 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:38:24 smithi118 ceph-mon[31535]: pgmap v4880: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-03-24T02:38:24.953 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:38:24 smithi189 ceph-mon[36187]: pgmap v4880: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-03-24T02:38:24.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:38:24 smithi193 ceph-mon[36878]: pgmap v4880: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-03-24T02:38:26.328 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:38:25 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 8001.729446 secs 2024-03-24T02:38:26.453 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:38:25 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 8001.729446 secs 2024-03-24T02:38:26.459 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:38:25 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 8001.729446 secs 2024-03-24T02:38:27.695 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:38:27 smithi118 ceph-mon[31535]: pgmap v4881: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:38:27.703 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:38:27 smithi189 ceph-mon[36187]: pgmap v4881: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:38:27.710 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:38:27 smithi193 ceph-mon[36878]: pgmap v4881: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:38:28.887 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:38:28 smithi118 ceph-mon[31535]: pgmap v4882: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T02:38:28.953 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:38:28 smithi189 ceph-mon[36187]: pgmap v4882: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T02:38:28.959 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:38:28 smithi193 ceph-mon[36878]: pgmap v4882: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T02:38:30.945 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:38:30 smithi118 ceph-mon[31535]: pgmap v4883: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T02:38:30.953 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:38:30 smithi189 ceph-mon[36187]: pgmap v4883: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T02:38:30.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:38:30 smithi193 ceph-mon[36878]: pgmap v4883: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T02:38:31.945 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:38:31 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 8006.729508 secs 2024-03-24T02:38:31.953 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:38:31 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 8006.729508 secs 2024-03-24T02:38:31.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:38:31 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 8006.729508 secs 2024-03-24T02:38:32.945 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:38:32 smithi118 ceph-mon[31535]: pgmap v4884: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T02:38:32.953 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:38:32 smithi189 ceph-mon[36187]: pgmap v4884: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T02:38:32.959 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:38:32 smithi193 ceph-mon[36878]: pgmap v4884: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T02:38:34.945 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:38:34 smithi118 ceph-mon[31535]: pgmap v4885: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-03-24T02:38:34.953 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:38:34 smithi189 ceph-mon[36187]: pgmap v4885: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-03-24T02:38:34.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:38:34 smithi193 ceph-mon[36878]: pgmap v4885: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-03-24T02:38:36.445 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:38:35 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 8011.729569 secs 2024-03-24T02:38:36.453 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:38:35 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 8011.729569 secs 2024-03-24T02:38:36.460 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:38:35 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 8011.729569 secs 2024-03-24T02:38:37.445 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:38:37 smithi118 ceph-mon[31535]: pgmap v4886: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:38:37.453 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:38:37 smithi189 ceph-mon[36187]: pgmap v4886: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:38:37.460 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:38:37 smithi193 ceph-mon[36878]: pgmap v4886: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:38:38.945 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:38:38 smithi118 ceph-mon[31535]: pgmap v4887: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T02:38:38.953 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:38:38 smithi189 ceph-mon[36187]: pgmap v4887: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T02:38:38.959 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:38:38 smithi193 ceph-mon[36878]: pgmap v4887: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T02:38:40.945 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:38:40 smithi118 ceph-mon[31535]: pgmap v4888: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T02:38:40.953 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:38:40 smithi189 ceph-mon[36187]: pgmap v4888: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T02:38:40.959 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:38:40 smithi193 ceph-mon[36878]: pgmap v4888: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T02:38:41.945 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:38:41 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 8016.729657 secs 2024-03-24T02:38:41.953 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:38:41 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 8016.729657 secs 2024-03-24T02:38:41.959 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:38:41 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 8016.729657 secs 2024-03-24T02:38:42.945 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:38:42 smithi118 ceph-mon[31535]: pgmap v4889: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T02:38:42.953 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:38:42 smithi189 ceph-mon[36187]: pgmap v4889: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T02:38:42.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:38:42 smithi193 ceph-mon[36878]: pgmap v4889: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T02:38:44.945 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:38:44 smithi118 ceph-mon[31535]: pgmap v4890: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-03-24T02:38:44.953 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:38:44 smithi189 ceph-mon[36187]: pgmap v4890: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-03-24T02:38:44.959 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:38:44 smithi193 ceph-mon[36878]: pgmap v4890: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-03-24T02:38:46.445 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:38:45 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 8021.729717 secs 2024-03-24T02:38:46.453 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:38:45 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 8021.729717 secs 2024-03-24T02:38:46.459 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:38:45 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 8021.729717 secs 2024-03-24T02:38:47.445 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:38:47 smithi118 ceph-mon[31535]: pgmap v4891: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:38:47.453 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:38:47 smithi189 ceph-mon[36187]: pgmap v4891: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:38:47.459 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:38:47 smithi193 ceph-mon[36878]: pgmap v4891: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:38:48.945 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:38:48 smithi118 ceph-mon[31535]: pgmap v4892: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T02:38:48.953 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:38:48 smithi189 ceph-mon[36187]: pgmap v4892: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T02:38:48.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:38:48 smithi193 ceph-mon[36878]: pgmap v4892: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T02:38:50.945 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:38:50 smithi118 ceph-mon[31535]: pgmap v4893: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T02:38:50.952 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:38:50 smithi189 ceph-mon[36187]: pgmap v4893: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T02:38:50.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:38:50 smithi193 ceph-mon[36878]: pgmap v4893: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T02:38:51.945 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:38:51 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 8026.729768 secs 2024-03-24T02:38:51.953 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:38:51 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 8026.729768 secs 2024-03-24T02:38:51.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:38:51 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 8026.729768 secs 2024-03-24T02:38:52.945 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:38:52 smithi118 ceph-mon[31535]: pgmap v4894: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T02:38:52.953 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:38:52 smithi189 ceph-mon[36187]: pgmap v4894: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T02:38:52.959 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:38:52 smithi193 ceph-mon[36878]: pgmap v4894: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T02:38:54.945 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:38:54 smithi118 ceph-mon[31535]: pgmap v4895: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-03-24T02:38:54.953 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:38:54 smithi189 ceph-mon[36187]: pgmap v4895: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-03-24T02:38:54.959 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:38:54 smithi193 ceph-mon[36878]: pgmap v4895: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-03-24T02:38:56.445 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:38:55 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 8031.729825 secs 2024-03-24T02:38:56.453 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:38:55 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 8031.729825 secs 2024-03-24T02:38:56.460 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:38:55 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 8031.729825 secs 2024-03-24T02:38:57.445 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:38:57 smithi118 ceph-mon[31535]: pgmap v4896: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:38:57.452 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:38:57 smithi189 ceph-mon[36187]: pgmap v4896: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:38:57.459 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:38:57 smithi193 ceph-mon[36878]: pgmap v4896: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:38:58.945 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:38:58 smithi118 ceph-mon[31535]: pgmap v4897: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T02:38:58.953 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:38:58 smithi189 ceph-mon[36187]: pgmap v4897: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T02:38:58.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:38:58 smithi193 ceph-mon[36878]: pgmap v4897: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T02:38:59.945 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:38:59 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T02:38:59.952 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:38:59 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T02:38:59.959 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:38:59 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T02:39:00.945 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:39:00 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T02:39:00.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:39:00 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T02:39:00.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:39:00 smithi118 ceph-mon[31535]: pgmap v4898: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 6 op/s 2024-03-24T02:39:00.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:39:00 smithi118 ceph-mon[31535]: pgmap v4899: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 6 op/s 2024-03-24T02:39:00.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:39:00 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T02:39:00.953 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:39:00 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T02:39:00.953 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:39:00 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T02:39:00.953 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:39:00 smithi189 ceph-mon[36187]: pgmap v4898: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 6 op/s 2024-03-24T02:39:00.953 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:39:00 smithi189 ceph-mon[36187]: pgmap v4899: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 6 op/s 2024-03-24T02:39:00.953 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:39:00 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T02:39:00.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:39:00 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T02:39:00.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:39:00 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T02:39:00.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:39:00 smithi193 ceph-mon[36878]: pgmap v4898: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 6 op/s 2024-03-24T02:39:00.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:39:00 smithi193 ceph-mon[36878]: pgmap v4899: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 6 op/s 2024-03-24T02:39:00.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:39:00 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T02:39:01.945 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:39:01 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 8036.729885 secs 2024-03-24T02:39:01.953 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:39:01 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 8036.729885 secs 2024-03-24T02:39:01.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:39:01 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 8036.729885 secs 2024-03-24T02:39:02.945 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:39:02 smithi118 ceph-mon[31535]: pgmap v4900: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 5 op/s 2024-03-24T02:39:02.953 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:39:02 smithi189 ceph-mon[36187]: pgmap v4900: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 5 op/s 2024-03-24T02:39:02.959 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:39:02 smithi193 ceph-mon[36878]: pgmap v4900: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 5 op/s 2024-03-24T02:39:05.445 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:39:05 smithi118 ceph-mon[31535]: pgmap v4901: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 5 op/s 2024-03-24T02:39:05.453 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:39:05 smithi189 ceph-mon[36187]: pgmap v4901: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 5 op/s 2024-03-24T02:39:05.459 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:39:05 smithi193 ceph-mon[36878]: pgmap v4901: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 5 op/s 2024-03-24T02:39:06.445 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:39:06 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 8041.729946 secs 2024-03-24T02:39:06.453 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:39:06 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 8041.729946 secs 2024-03-24T02:39:06.460 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:39:06 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 8041.729946 secs 2024-03-24T02:39:07.445 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:39:07 smithi118 ceph-mon[31535]: pgmap v4902: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 6 op/s 2024-03-24T02:39:07.452 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:39:07 smithi189 ceph-mon[36187]: pgmap v4902: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 6 op/s 2024-03-24T02:39:07.460 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:39:07 smithi193 ceph-mon[36878]: pgmap v4902: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 6 op/s 2024-03-24T02:39:09.445 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:39:09 smithi118 ceph-mon[31535]: pgmap v4903: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 5 op/s 2024-03-24T02:39:09.453 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:39:09 smithi189 ceph-mon[36187]: pgmap v4903: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 5 op/s 2024-03-24T02:39:09.459 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:39:09 smithi193 ceph-mon[36878]: pgmap v4903: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 5 op/s 2024-03-24T02:39:11.445 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:39:11 smithi118 ceph-mon[31535]: pgmap v4904: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 5 op/s 2024-03-24T02:39:11.445 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:39:11 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 8046.730006 secs 2024-03-24T02:39:11.453 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:39:11 smithi189 ceph-mon[36187]: pgmap v4904: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 5 op/s 2024-03-24T02:39:11.453 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:39:11 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 8046.730006 secs 2024-03-24T02:39:11.460 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:39:11 smithi193 ceph-mon[36878]: pgmap v4904: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 5 op/s 2024-03-24T02:39:11.460 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:39:11 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 8046.730006 secs 2024-03-24T02:39:13.445 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:39:13 smithi118 ceph-mon[31535]: pgmap v4905: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:39:13.453 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:39:13 smithi189 ceph-mon[36187]: pgmap v4905: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:39:13.460 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:39:13 smithi193 ceph-mon[36878]: pgmap v4905: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:39:15.445 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:39:15 smithi118 ceph-mon[31535]: pgmap v4906: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T02:39:15.453 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:39:15 smithi189 ceph-mon[36187]: pgmap v4906: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T02:39:15.459 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:39:15 smithi193 ceph-mon[36878]: pgmap v4906: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T02:39:16.445 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:39:16 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 8051.730049 secs 2024-03-24T02:39:16.453 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:39:16 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 8051.730049 secs 2024-03-24T02:39:16.460 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:39:16 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 8051.730049 secs 2024-03-24T02:39:17.445 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:39:17 smithi118 ceph-mon[31535]: pgmap v4907: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T02:39:17.453 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:39:17 smithi189 ceph-mon[36187]: pgmap v4907: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T02:39:17.460 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:39:17 smithi193 ceph-mon[36878]: pgmap v4907: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T02:39:19.445 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:39:19 smithi118 ceph-mon[31535]: pgmap v4908: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 5 op/s 2024-03-24T02:39:19.452 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:39:19 smithi189 ceph-mon[36187]: pgmap v4908: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 5 op/s 2024-03-24T02:39:19.459 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:39:19 smithi193 ceph-mon[36878]: pgmap v4908: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 5 op/s 2024-03-24T02:39:21.445 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:39:21 smithi118 ceph-mon[31535]: pgmap v4909: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T02:39:21.445 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:39:21 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 8056.730135 secs 2024-03-24T02:39:21.453 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:39:21 smithi189 ceph-mon[36187]: pgmap v4909: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T02:39:21.453 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:39:21 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 8056.730135 secs 2024-03-24T02:39:21.460 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:39:21 smithi193 ceph-mon[36878]: pgmap v4909: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T02:39:21.460 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:39:21 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 8056.730135 secs 2024-03-24T02:39:23.453 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:39:23 smithi189 ceph-mon[36187]: pgmap v4910: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:39:23.459 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:39:23 smithi193 ceph-mon[36878]: pgmap v4910: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:39:23.695 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:39:23 smithi118 ceph-mon[31535]: pgmap v4910: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:39:25.453 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:39:25 smithi189 ceph-mon[36187]: pgmap v4911: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T02:39:25.460 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:39:25 smithi193 ceph-mon[36878]: pgmap v4911: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T02:39:25.695 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:39:25 smithi118 ceph-mon[31535]: pgmap v4911: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T02:39:26.695 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:39:26 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 8061.730223 secs 2024-03-24T02:39:26.703 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:39:26 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 8061.730223 secs 2024-03-24T02:39:26.710 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:39:26 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 8061.730223 secs 2024-03-24T02:39:27.695 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:39:27 smithi118 ceph-mon[31535]: pgmap v4912: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T02:39:27.703 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:39:27 smithi189 ceph-mon[36187]: pgmap v4912: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T02:39:27.710 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:39:27 smithi193 ceph-mon[36878]: pgmap v4912: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T02:39:29.695 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:39:29 smithi118 ceph-mon[31535]: pgmap v4913: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 5 op/s 2024-03-24T02:39:29.703 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:39:29 smithi189 ceph-mon[36187]: pgmap v4913: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 5 op/s 2024-03-24T02:39:29.710 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:39:29 smithi193 ceph-mon[36878]: pgmap v4913: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 5 op/s 2024-03-24T02:39:31.695 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:39:31 smithi118 ceph-mon[31535]: pgmap v4914: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T02:39:31.695 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:39:31 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 8066.730309 secs 2024-03-24T02:39:31.703 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:39:31 smithi189 ceph-mon[36187]: pgmap v4914: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T02:39:31.703 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:39:31 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 8066.730309 secs 2024-03-24T02:39:31.709 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:39:31 smithi193 ceph-mon[36878]: pgmap v4914: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T02:39:31.710 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:39:31 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 8066.730309 secs 2024-03-24T02:39:33.695 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:39:33 smithi118 ceph-mon[31535]: pgmap v4915: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:39:33.703 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:39:33 smithi189 ceph-mon[36187]: pgmap v4915: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:39:33.710 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:39:33 smithi193 ceph-mon[36878]: pgmap v4915: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:39:35.695 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:39:35 smithi118 ceph-mon[31535]: pgmap v4916: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T02:39:35.703 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:39:35 smithi189 ceph-mon[36187]: pgmap v4916: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T02:39:35.710 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:39:35 smithi193 ceph-mon[36878]: pgmap v4916: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T02:39:36.597 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:39:36 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 8071.730368 secs 2024-03-24T02:39:36.703 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:39:36 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 8071.730368 secs 2024-03-24T02:39:36.709 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:39:36 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 8071.730368 secs 2024-03-24T02:39:37.695 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:39:37 smithi118 ceph-mon[31535]: pgmap v4917: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T02:39:37.702 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:39:37 smithi189 ceph-mon[36187]: pgmap v4917: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T02:39:37.709 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:39:37 smithi193 ceph-mon[36878]: pgmap v4917: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T02:39:39.695 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:39:39 smithi118 ceph-mon[31535]: pgmap v4918: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 5 op/s 2024-03-24T02:39:39.703 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:39:39 smithi189 ceph-mon[36187]: pgmap v4918: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 5 op/s 2024-03-24T02:39:39.710 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:39:39 smithi193 ceph-mon[36878]: pgmap v4918: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 5 op/s 2024-03-24T02:39:41.695 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:39:41 smithi118 ceph-mon[31535]: pgmap v4919: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T02:39:41.695 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:39:41 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 8076.730463 secs 2024-03-24T02:39:41.703 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:39:41 smithi189 ceph-mon[36187]: pgmap v4919: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T02:39:41.703 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:39:41 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 8076.730463 secs 2024-03-24T02:39:41.709 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:39:41 smithi193 ceph-mon[36878]: pgmap v4919: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T02:39:41.710 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:39:41 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 8076.730463 secs 2024-03-24T02:39:43.695 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:39:43 smithi118 ceph-mon[31535]: pgmap v4920: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:39:43.703 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:39:43 smithi189 ceph-mon[36187]: pgmap v4920: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:39:43.710 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:39:43 smithi193 ceph-mon[36878]: pgmap v4920: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:39:45.695 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:39:45 smithi118 ceph-mon[31535]: pgmap v4921: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T02:39:45.702 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:39:45 smithi189 ceph-mon[36187]: pgmap v4921: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T02:39:45.709 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:39:45 smithi193 ceph-mon[36878]: pgmap v4921: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T02:39:46.603 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:39:46 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 8081.730551 secs 2024-03-24T02:39:46.611 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:39:46 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 8081.730551 secs 2024-03-24T02:39:46.695 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:39:46 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 8081.730551 secs 2024-03-24T02:39:47.695 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:39:47 smithi118 ceph-mon[31535]: pgmap v4922: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T02:39:47.703 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:39:47 smithi189 ceph-mon[36187]: pgmap v4922: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T02:39:47.709 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:39:47 smithi193 ceph-mon[36878]: pgmap v4922: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T02:39:49.695 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:39:49 smithi118 ceph-mon[31535]: pgmap v4923: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T02:39:49.703 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:39:49 smithi189 ceph-mon[36187]: pgmap v4923: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T02:39:49.710 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:39:49 smithi193 ceph-mon[36878]: pgmap v4923: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T02:39:51.695 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:39:51 smithi118 ceph-mon[31535]: pgmap v4924: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T02:39:51.695 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:39:51 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 8086.730640 secs 2024-03-24T02:39:51.702 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:39:51 smithi189 ceph-mon[36187]: pgmap v4924: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T02:39:51.703 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:39:51 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 8086.730640 secs 2024-03-24T02:39:51.709 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:39:51 smithi193 ceph-mon[36878]: pgmap v4924: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T02:39:51.710 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:39:51 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 8086.730640 secs 2024-03-24T02:39:53.695 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:39:53 smithi118 ceph-mon[31535]: pgmap v4925: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:39:53.702 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:39:53 smithi189 ceph-mon[36187]: pgmap v4925: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:39:53.709 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:39:53 smithi193 ceph-mon[36878]: pgmap v4925: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:39:55.695 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:39:55 smithi118 ceph-mon[31535]: pgmap v4926: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T02:39:55.703 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:39:55 smithi189 ceph-mon[36187]: pgmap v4926: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T02:39:55.709 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:39:55 smithi193 ceph-mon[36878]: pgmap v4926: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T02:39:56.695 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:39:56 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 8091.730703 secs 2024-03-24T02:39:56.703 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:39:56 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 8091.730703 secs 2024-03-24T02:39:56.709 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:39:56 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 8091.730703 secs 2024-03-24T02:39:57.695 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:39:57 smithi118 ceph-mon[31535]: pgmap v4927: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T02:39:57.703 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:39:57 smithi189 ceph-mon[36187]: pgmap v4927: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T02:39:57.709 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:39:57 smithi193 ceph-mon[36878]: pgmap v4927: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T02:39:59.695 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:39:59 smithi118 ceph-mon[31535]: pgmap v4928: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T02:39:59.703 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:39:59 smithi189 ceph-mon[36187]: pgmap v4928: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T02:39:59.709 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:39:59 smithi193 ceph-mon[36878]: pgmap v4928: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T02:40:00.695 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:40:00 smithi118 ceph-mon[31535]: overall HEALTH_WARN 6 failed cephadm daemon(s); 1 MDSs report slow requests 2024-03-24T02:40:00.703 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:40:00 smithi189 ceph-mon[36187]: overall HEALTH_WARN 6 failed cephadm daemon(s); 1 MDSs report slow requests 2024-03-24T02:40:00.709 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:40:00 smithi193 ceph-mon[36878]: overall HEALTH_WARN 6 failed cephadm daemon(s); 1 MDSs report slow requests 2024-03-24T02:40:01.695 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:40:01 smithi118 ceph-mon[31535]: pgmap v4929: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T02:40:01.695 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:40:01 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 8096.730791 secs 2024-03-24T02:40:01.695 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:40:01 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T02:40:01.703 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:40:01 smithi189 ceph-mon[36187]: pgmap v4929: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T02:40:01.703 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:40:01 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 8096.730791 secs 2024-03-24T02:40:01.703 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:40:01 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T02:40:01.709 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:40:01 smithi193 ceph-mon[36878]: pgmap v4929: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T02:40:01.710 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:40:01 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 8096.730791 secs 2024-03-24T02:40:01.710 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:40:01 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T02:40:02.695 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:40:02 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T02:40:02.695 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:40:02 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T02:40:02.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:40:02 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T02:40:02.702 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:40:02 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T02:40:02.703 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:40:02 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T02:40:02.703 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:40:02 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T02:40:02.709 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:40:02 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T02:40:02.710 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:40:02 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T02:40:02.710 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:40:02 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T02:40:03.445 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:40:03 smithi118 ceph-mon[31535]: pgmap v4930: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 5 op/s 2024-03-24T02:40:03.445 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:40:03 smithi118 ceph-mon[31535]: pgmap v4931: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 6 op/s 2024-03-24T02:40:03.703 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:40:03 smithi189 ceph-mon[36187]: pgmap v4930: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 5 op/s 2024-03-24T02:40:03.703 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:40:03 smithi189 ceph-mon[36187]: pgmap v4931: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 6 op/s 2024-03-24T02:40:03.709 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:40:03 smithi193 ceph-mon[36878]: pgmap v4930: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 5 op/s 2024-03-24T02:40:03.710 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:40:03 smithi193 ceph-mon[36878]: pgmap v4931: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 6 op/s 2024-03-24T02:40:05.695 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:40:05 smithi118 ceph-mon[31535]: pgmap v4932: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 5 op/s 2024-03-24T02:40:05.702 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:40:05 smithi189 ceph-mon[36187]: pgmap v4932: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 5 op/s 2024-03-24T02:40:05.709 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:40:05 smithi193 ceph-mon[36878]: pgmap v4932: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 5 op/s 2024-03-24T02:40:06.695 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:40:06 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 8101.730879 secs 2024-03-24T02:40:06.703 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:40:06 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 8101.730879 secs 2024-03-24T02:40:06.709 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:40:06 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 8101.730879 secs 2024-03-24T02:40:07.695 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:40:07 smithi118 ceph-mon[31535]: pgmap v4933: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 6 op/s 2024-03-24T02:40:07.703 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:40:07 smithi189 ceph-mon[36187]: pgmap v4933: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 6 op/s 2024-03-24T02:40:07.709 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:40:07 smithi193 ceph-mon[36878]: pgmap v4933: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 6 op/s 2024-03-24T02:40:09.695 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:40:09 smithi118 ceph-mon[31535]: pgmap v4934: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 5 op/s 2024-03-24T02:40:09.702 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:40:09 smithi189 ceph-mon[36187]: pgmap v4934: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 5 op/s 2024-03-24T02:40:09.709 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:40:09 smithi193 ceph-mon[36878]: pgmap v4934: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 5 op/s 2024-03-24T02:40:11.695 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:40:11 smithi118 ceph-mon[31535]: pgmap v4935: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 6 op/s 2024-03-24T02:40:11.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:40:11 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 8106.730964 secs 2024-03-24T02:40:11.703 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:40:11 smithi189 ceph-mon[36187]: pgmap v4935: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 6 op/s 2024-03-24T02:40:11.703 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:40:11 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 8106.730964 secs 2024-03-24T02:40:11.709 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:40:11 smithi193 ceph-mon[36878]: pgmap v4935: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 6 op/s 2024-03-24T02:40:11.710 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:40:11 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 8106.730964 secs 2024-03-24T02:40:13.695 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:40:13 smithi118 ceph-mon[31535]: pgmap v4936: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 5 op/s 2024-03-24T02:40:13.703 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:40:13 smithi189 ceph-mon[36187]: pgmap v4936: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 5 op/s 2024-03-24T02:40:13.710 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:40:13 smithi193 ceph-mon[36878]: pgmap v4936: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 5 op/s 2024-03-24T02:40:15.695 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:40:15 smithi118 ceph-mon[31535]: pgmap v4937: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T02:40:15.702 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:40:15 smithi189 ceph-mon[36187]: pgmap v4937: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T02:40:15.709 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:40:15 smithi193 ceph-mon[36878]: pgmap v4937: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T02:40:15.762 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T02:40:15.762+0000 7f81f6000640 1 -- 172.21.15.118:0/1693767697 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7f81dc0a3bd0 msgr2=0x7f81dc0b2b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 12 2024-03-24T02:40:15.762 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T02:40:15.762+0000 7f81f6000640 1 -- 172.21.15.118:0/1693767697 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7f81dc0a3bd0 msgr2=0x7f81dc0b2b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-03-24T02:40:15.762 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T02:40:15.762+0000 7f81f6000640 1 --2- 172.21.15.118:0/1693767697 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7f81dc0a3bd0 0x7f81dc0b2b10 secure :-1 s=READY pgs=563 cs=0 l=1 rev1=1 crypto rx=0x7f81e00048d0 tx=0x7f81e0067040 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-03-24T02:40:15.762 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T02:40:15.762+0000 7f81f6000640 1 --2- 172.21.15.118:0/1693767697 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7f81dc0a3bd0 0x7f81dc0b2b10 secure :-1 s=READY pgs=563 cs=0 l=1 rev1=1 crypto rx=0x7f81e00048d0 tx=0x7f81e0067040 comp rx=0 tx=0).stop 2024-03-24T02:40:15.762 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T02:40:15.762+0000 7f81f6000640 1 -- 172.21.15.118:0/1693767697 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7f81dc0a3bd0 msgr2=0x7f81dc0b2b10 unknown :-1 s=STATE_CLOSED l=1).tick see no progress in more than 10000000 us during connecting to v2:172.21.15.118:6800/2112912943, fault. 2024-03-24T02:40:15.762 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T02:40:15.762+0000 7f81ef400640 1 -- 172.21.15.118:0/1693767697 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7f81dc0a3bd0 msgr2=0x7f81dc0b2b10 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-03-24T02:40:15.762 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T02:40:15.762+0000 7f81ef400640 1 --2- 172.21.15.118:0/1693767697 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7f81dc0a3bd0 0x7f81dc0b2b10 unknown :-1 s=CLOSED pgs=563 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T02:40:15.762 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T02:40:15.762+0000 7f81ef400640 1 --2- 172.21.15.118:0/1693767697 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7f81dc0a3fb0 0x7f81dc0b05c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T02:40:15.763 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T02:40:15.763+0000 7f81f6000640 1 --2- 172.21.15.118:0/1693767697 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7f81dc0a3fb0 0x7f81dc0b05c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T02:40:15.764 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T02:40:15.764+0000 7f81f6000640 1 --2- 172.21.15.118:0/1693767697 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7f81dc0a3fb0 0x7f81dc0b05c0 secure :-1 s=READY pgs=586 cs=0 l=1 rev1=1 crypto rx=0x7f81e0001eb0 tx=0x7f81e0047000 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T02:40:15.770 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T02:40:15.770+0000 7f81f6000640 1 -- 172.21.15.118:0/659686424 >> [v2:172.21.15.189:6838/4074484168,v1:172.21.15.189:6839/4074484168] conn(0x5622651b71e0 msgr2=0x562265215110 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).tick idle (900000744) for more than 900000000 us, fault. 2024-03-24T02:40:15.770 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T02:40:15.770+0000 7f81f6000640 1 --2- 172.21.15.118:0/659686424 >> [v2:172.21.15.189:6838/4074484168,v1:172.21.15.189:6839/4074484168] conn(0x5622651b71e0 0x562265215110 unknown :-1 s=READY pgs=561 cs=7 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault initiating reconnect 2024-03-24T02:40:15.770 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T02:40:15.770+0000 7f81f6000640 1 --2- 172.21.15.118:0/659686424 >> [v2:172.21.15.189:6838/4074484168,v1:172.21.15.189:6839/4074484168] conn(0x5622651b71e0 0x562265215110 unknown :-1 s=BANNER_CONNECTING pgs=561 cs=8 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T02:40:15.771 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T02:40:15.771+0000 7f81f6000640 1 --2- 172.21.15.118:0/659686424 >> [v2:172.21.15.189:6838/4074484168,v1:172.21.15.189:6839/4074484168] conn(0x5622651b71e0 0x562265215110 crc :-1 s=READY pgs=561 cs=8 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=76f3d534289740f0 server_cookie=b31a2335a59552de in_seq=0 out_seq=0 2024-03-24T02:40:15.771 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T02:40:15.771+0000 7f81e4c00640 10 client.27094 ms_handle_connect on v2:172.21.15.189:6838/4074484168 2024-03-24T02:40:16.695 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:40:16 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 8111.731025 secs 2024-03-24T02:40:16.703 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:40:16 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 8111.731025 secs 2024-03-24T02:40:16.709 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:40:16 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 8111.731025 secs 2024-03-24T02:40:17.044 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T02:40:17.044+0000 7f81ee000640 1 -- 172.21.15.118:0/1693767697 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f81c80036c0 con 0x7f81f81052d0 2024-03-24T02:40:17.044 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T02:40:17.044+0000 7f81db400640 1 -- 172.21.15.118:0/659686424 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f81c4006460 con 0x5622651a13f0 2024-03-24T02:40:17.044 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T02:40:17.044+0000 7f81e4c00640 1 -- 172.21.15.118:0/659686424 <== mon.1 v2:172.21.15.193:3300/0 66 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 409+0+0 (secure 0 0 0) 0x7f81c4006460 con 0x5622651a13f0 2024-03-24T02:40:17.045 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T02:40:17.044+0000 7f81ef400640 1 -- 172.21.15.118:0/1693767697 <== mon.2 v2:172.21.15.189:3300/0 66 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (secure 0 0 0) 0x7f81c80036c0 con 0x7f81f81052d0 2024-03-24T02:40:17.695 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:40:17 smithi118 ceph-mon[31535]: pgmap v4938: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T02:40:17.703 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:40:17 smithi189 ceph-mon[36187]: pgmap v4938: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T02:40:17.709 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:40:17 smithi193 ceph-mon[36878]: pgmap v4938: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T02:40:19.695 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:40:19 smithi118 ceph-mon[31535]: pgmap v4939: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T02:40:19.703 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:40:19 smithi189 ceph-mon[36187]: pgmap v4939: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T02:40:19.710 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:40:19 smithi193 ceph-mon[36878]: pgmap v4939: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T02:40:21.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:40:21 smithi118 ceph-mon[31535]: pgmap v4940: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-03-24T02:40:21.697 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:40:21 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 8116.731115 secs 2024-03-24T02:40:21.703 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:40:21 smithi189 ceph-mon[36187]: pgmap v4940: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-03-24T02:40:21.703 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:40:21 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 8116.731115 secs 2024-03-24T02:40:21.709 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:40:21 smithi193 ceph-mon[36878]: pgmap v4940: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-03-24T02:40:21.710 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:40:21 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 8116.731115 secs 2024-03-24T02:40:23.695 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:40:23 smithi118 ceph-mon[31535]: pgmap v4941: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:40:23.705 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:40:23 smithi189 ceph-mon[36187]: pgmap v4941: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:40:23.709 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:40:23 smithi193 ceph-mon[36878]: pgmap v4941: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:40:25.695 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:40:25 smithi118 ceph-mon[31535]: pgmap v4942: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T02:40:25.703 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:40:25 smithi189 ceph-mon[36187]: pgmap v4942: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T02:40:25.710 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:40:25 smithi193 ceph-mon[36878]: pgmap v4942: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T02:40:26.703 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:40:26 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 8121.731197 secs 2024-03-24T02:40:26.709 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:40:26 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 8121.731197 secs 2024-03-24T02:40:26.945 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:40:26 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 8121.731197 secs 2024-03-24T02:40:27.703 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:40:27 smithi189 ceph-mon[36187]: pgmap v4943: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T02:40:27.709 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:40:27 smithi193 ceph-mon[36878]: pgmap v4943: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T02:40:27.945 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:40:27 smithi118 ceph-mon[31535]: pgmap v4943: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T02:40:29.710 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:40:29 smithi193 ceph-mon[36878]: pgmap v4944: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T02:40:29.945 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:40:29 smithi118 ceph-mon[31535]: pgmap v4944: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T02:40:29.953 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:40:29 smithi189 ceph-mon[36187]: pgmap v4944: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T02:40:31.710 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:40:31 smithi193 ceph-mon[36878]: pgmap v4945: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-03-24T02:40:31.710 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:40:31 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 8126.731212 secs 2024-03-24T02:40:31.945 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:40:31 smithi118 ceph-mon[31535]: pgmap v4945: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-03-24T02:40:31.945 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:40:31 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 8126.731212 secs 2024-03-24T02:40:31.953 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:40:31 smithi189 ceph-mon[36187]: pgmap v4945: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-03-24T02:40:31.953 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:40:31 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 8126.731212 secs 2024-03-24T02:40:33.945 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:40:33 smithi118 ceph-mon[31535]: pgmap v4946: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:40:33.953 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:40:33 smithi189 ceph-mon[36187]: pgmap v4946: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:40:33.959 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:40:33 smithi193 ceph-mon[36878]: pgmap v4946: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:40:35.945 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:40:35 smithi118 ceph-mon[31535]: pgmap v4947: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T02:40:35.953 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:40:35 smithi189 ceph-mon[36187]: pgmap v4947: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T02:40:35.959 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:40:35 smithi193 ceph-mon[36878]: pgmap v4947: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T02:40:36.847 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:40:36 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 8131.731296 secs 2024-03-24T02:40:36.953 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:40:36 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 8131.731296 secs 2024-03-24T02:40:36.959 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:40:36 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 8131.731296 secs 2024-03-24T02:40:37.945 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:40:37 smithi118 ceph-mon[31535]: pgmap v4948: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T02:40:37.953 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:40:37 smithi189 ceph-mon[36187]: pgmap v4948: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T02:40:37.959 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:40:37 smithi193 ceph-mon[36878]: pgmap v4948: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T02:40:39.945 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:40:39 smithi118 ceph-mon[31535]: pgmap v4949: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T02:40:39.953 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:40:39 smithi189 ceph-mon[36187]: pgmap v4949: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T02:40:39.959 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:40:39 smithi193 ceph-mon[36878]: pgmap v4949: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T02:40:41.945 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:40:41 smithi118 ceph-mon[31535]: pgmap v4950: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-03-24T02:40:41.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:40:41 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 8136.731367 secs 2024-03-24T02:40:41.953 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:40:41 smithi189 ceph-mon[36187]: pgmap v4950: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-03-24T02:40:41.953 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:40:41 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 8136.731367 secs 2024-03-24T02:40:41.959 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:40:41 smithi193 ceph-mon[36878]: pgmap v4950: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-03-24T02:40:41.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:40:41 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 8136.731367 secs 2024-03-24T02:40:43.945 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:40:43 smithi118 ceph-mon[31535]: pgmap v4951: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:40:43.952 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:40:43 smithi189 ceph-mon[36187]: pgmap v4951: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:40:43.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:40:43 smithi193 ceph-mon[36878]: pgmap v4951: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:40:45.945 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:40:45 smithi118 ceph-mon[31535]: pgmap v4952: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T02:40:45.952 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:40:45 smithi189 ceph-mon[36187]: pgmap v4952: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T02:40:45.959 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:40:45 smithi193 ceph-mon[36878]: pgmap v4952: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T02:40:46.854 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:40:46 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 8141.731428 secs 2024-03-24T02:40:46.860 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:40:46 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 8141.731428 secs 2024-03-24T02:40:46.945 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:40:46 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 8141.731428 secs 2024-03-24T02:40:47.945 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:40:47 smithi118 ceph-mon[31535]: pgmap v4953: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T02:40:47.953 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:40:47 smithi189 ceph-mon[36187]: pgmap v4953: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T02:40:47.959 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:40:47 smithi193 ceph-mon[36878]: pgmap v4953: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T02:40:49.945 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:40:49 smithi118 ceph-mon[31535]: pgmap v4954: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T02:40:49.952 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:40:49 smithi189 ceph-mon[36187]: pgmap v4954: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T02:40:49.959 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:40:49 smithi193 ceph-mon[36878]: pgmap v4954: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T02:40:51.945 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:40:51 smithi118 ceph-mon[31535]: pgmap v4955: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-03-24T02:40:51.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:40:51 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 8146.731515 secs 2024-03-24T02:40:51.953 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:40:51 smithi189 ceph-mon[36187]: pgmap v4955: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-03-24T02:40:51.953 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:40:51 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 8146.731515 secs 2024-03-24T02:40:51.959 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:40:51 smithi193 ceph-mon[36878]: pgmap v4955: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-03-24T02:40:51.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:40:51 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 8146.731515 secs 2024-03-24T02:40:53.945 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:40:53 smithi118 ceph-mon[31535]: pgmap v4956: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:40:53.952 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:40:53 smithi189 ceph-mon[36187]: pgmap v4956: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:40:53.959 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:40:53 smithi193 ceph-mon[36878]: pgmap v4956: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:40:55.945 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:40:55 smithi118 ceph-mon[31535]: pgmap v4957: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T02:40:55.953 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:40:55 smithi189 ceph-mon[36187]: pgmap v4957: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T02:40:55.959 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:40:55 smithi193 ceph-mon[36878]: pgmap v4957: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T02:40:56.945 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:40:56 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 8151.731603 secs 2024-03-24T02:40:56.953 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:40:56 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 8151.731603 secs 2024-03-24T02:40:56.959 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:40:56 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 8151.731603 secs 2024-03-24T02:40:57.945 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:40:57 smithi118 ceph-mon[31535]: pgmap v4958: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 6 op/s 2024-03-24T02:40:57.952 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:40:57 smithi189 ceph-mon[36187]: pgmap v4958: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 6 op/s 2024-03-24T02:40:57.959 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:40:57 smithi193 ceph-mon[36878]: pgmap v4958: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 6 op/s 2024-03-24T02:40:59.945 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:40:59 smithi118 ceph-mon[31535]: pgmap v4959: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 5 op/s 2024-03-24T02:40:59.952 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:40:59 smithi189 ceph-mon[36187]: pgmap v4959: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 5 op/s 2024-03-24T02:40:59.959 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:40:59 smithi193 ceph-mon[36878]: pgmap v4959: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 5 op/s 2024-03-24T02:41:01.945 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:41:01 smithi118 ceph-mon[31535]: pgmap v4960: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-03-24T02:41:01.945 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:41:01 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 8156.731644 secs 2024-03-24T02:41:01.953 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:41:01 smithi189 ceph-mon[36187]: pgmap v4960: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-03-24T02:41:01.953 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:41:01 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 8156.731644 secs 2024-03-24T02:41:01.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:41:01 smithi193 ceph-mon[36878]: pgmap v4960: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-03-24T02:41:01.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:41:01 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 8156.731644 secs 2024-03-24T02:41:02.830 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:41:02 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T02:41:02.830 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:41:02 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T02:41:02.830 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:41:02 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T02:41:02.945 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:41:02 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T02:41:02.945 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:41:02 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T02:41:02.945 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:41:02 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T02:41:02.953 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:41:02 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T02:41:02.953 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:41:02 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T02:41:02.953 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:41:02 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T02:41:03.745 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:41:03 smithi189 ceph-mon[36187]: pgmap v4961: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-03-24T02:41:03.745 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:41:03 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T02:41:03.745 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:41:03 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T02:41:03.945 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:41:03 smithi118 ceph-mon[31535]: pgmap v4961: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-03-24T02:41:03.945 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:41:03 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T02:41:03.945 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:41:03 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T02:41:03.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:41:03 smithi193 ceph-mon[36878]: pgmap v4961: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-03-24T02:41:03.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:41:03 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T02:41:03.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:41:03 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T02:41:05.445 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:41:05 smithi118 ceph-mon[31535]: pgmap v4962: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-03-24T02:41:05.445 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:41:05 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T02:41:05.445 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:41:05 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T02:41:05.445 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:41:05 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T02:41:05.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:41:05 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T02:41:05.453 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:41:05 smithi189 ceph-mon[36187]: pgmap v4962: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-03-24T02:41:05.453 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:41:05 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T02:41:05.453 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:41:05 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T02:41:05.453 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:41:05 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T02:41:05.453 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:41:05 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T02:41:05.460 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:41:05 smithi193 ceph-mon[36878]: pgmap v4962: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-03-24T02:41:05.460 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:41:05 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T02:41:05.460 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:41:05 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T02:41:05.460 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:41:05 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T02:41:05.460 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:41:05 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T02:41:06.347 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:41:06 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 8161.731738 secs 2024-03-24T02:41:06.453 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:41:06 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 8161.731738 secs 2024-03-24T02:41:06.459 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:41:06 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 8161.731738 secs 2024-03-24T02:41:07.390 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:41:07 smithi118 ceph-mon[31535]: pgmap v4963: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 6 op/s 2024-03-24T02:41:07.390 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:41:07 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T02:41:07.390 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:41:07 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T02:41:07.453 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:41:07 smithi189 ceph-mon[36187]: pgmap v4963: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 6 op/s 2024-03-24T02:41:07.453 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:41:07 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T02:41:07.453 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:41:07 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T02:41:07.459 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:41:07 smithi193 ceph-mon[36878]: pgmap v4963: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 6 op/s 2024-03-24T02:41:07.460 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:41:07 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T02:41:07.460 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:41:07 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T02:41:08.445 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:41:08 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T02:41:08.445 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:41:08 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T02:41:08.445 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:41:08 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T02:41:08.452 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:41:08 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T02:41:08.453 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:41:08 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T02:41:08.453 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:41:08 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T02:41:08.460 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:41:08 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T02:41:08.460 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:41:08 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T02:41:08.460 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:41:08 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T02:41:09.445 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:41:09 smithi118 ceph-mon[31535]: pgmap v4964: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T02:41:09.445 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:41:09 smithi118 ceph-mon[31535]: pgmap v4965: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 6 op/s 2024-03-24T02:41:09.445 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:41:09 smithi118 ceph-mon[31535]: pgmap v4966: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 5 op/s 2024-03-24T02:41:09.453 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:41:09 smithi189 ceph-mon[36187]: pgmap v4964: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T02:41:09.453 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:41:09 smithi189 ceph-mon[36187]: pgmap v4965: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 6 op/s 2024-03-24T02:41:09.453 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:41:09 smithi189 ceph-mon[36187]: pgmap v4966: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 5 op/s 2024-03-24T02:41:09.460 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:41:09 smithi193 ceph-mon[36878]: pgmap v4964: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T02:41:09.460 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:41:09 smithi193 ceph-mon[36878]: pgmap v4965: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 6 op/s 2024-03-24T02:41:09.460 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:41:09 smithi193 ceph-mon[36878]: pgmap v4966: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 5 op/s 2024-03-24T02:41:11.445 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:41:11 smithi118 ceph-mon[31535]: pgmap v4967: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-03-24T02:41:11.445 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:41:11 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 8166.731776 secs 2024-03-24T02:41:11.453 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:41:11 smithi189 ceph-mon[36187]: pgmap v4967: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-03-24T02:41:11.453 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:41:11 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 8166.731776 secs 2024-03-24T02:41:11.459 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:41:11 smithi193 ceph-mon[36878]: pgmap v4967: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-03-24T02:41:11.460 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:41:11 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 8166.731776 secs 2024-03-24T02:41:13.445 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:41:13 smithi118 ceph-mon[31535]: pgmap v4968: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 5 op/s 2024-03-24T02:41:13.452 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:41:13 smithi189 ceph-mon[36187]: pgmap v4968: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 5 op/s 2024-03-24T02:41:13.459 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:41:13 smithi193 ceph-mon[36878]: pgmap v4968: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 5 op/s 2024-03-24T02:41:15.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:41:15 smithi118 ceph-mon[31535]: pgmap v4969: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 5 op/s 2024-03-24T02:41:15.453 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:41:15 smithi189 ceph-mon[36187]: pgmap v4969: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 5 op/s 2024-03-24T02:41:15.459 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:41:15 smithi193 ceph-mon[36878]: pgmap v4969: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 5 op/s 2024-03-24T02:41:16.445 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:41:16 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 8171.731817 secs 2024-03-24T02:41:16.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:41:16 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 8171.731817 secs 2024-03-24T02:41:16.459 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:41:16 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 8171.731817 secs 2024-03-24T02:41:17.445 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:41:17 smithi118 ceph-mon[31535]: pgmap v4970: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 6 op/s 2024-03-24T02:41:17.453 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:41:17 smithi189 ceph-mon[36187]: pgmap v4970: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 6 op/s 2024-03-24T02:41:17.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:41:17 smithi193 ceph-mon[36878]: pgmap v4970: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 6 op/s 2024-03-24T02:41:19.445 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:41:19 smithi118 ceph-mon[31535]: pgmap v4971: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 5 op/s 2024-03-24T02:41:19.453 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:41:19 smithi189 ceph-mon[36187]: pgmap v4971: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 5 op/s 2024-03-24T02:41:19.460 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:41:19 smithi193 ceph-mon[36878]: pgmap v4971: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 5 op/s 2024-03-24T02:41:21.445 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:41:21 smithi118 ceph-mon[31535]: pgmap v4972: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T02:41:21.445 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:41:21 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 8176.731876 secs 2024-03-24T02:41:21.453 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:41:21 smithi189 ceph-mon[36187]: pgmap v4972: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T02:41:21.453 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:41:21 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 8176.731876 secs 2024-03-24T02:41:21.459 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:41:21 smithi193 ceph-mon[36878]: pgmap v4972: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T02:41:21.459 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:41:21 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 8176.731876 secs 2024-03-24T02:41:23.445 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:41:23 smithi118 ceph-mon[31535]: pgmap v4973: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:41:23.453 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:41:23 smithi189 ceph-mon[36187]: pgmap v4973: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:41:23.459 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:41:23 smithi193 ceph-mon[36878]: pgmap v4973: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:41:25.445 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:41:25 smithi118 ceph-mon[31535]: pgmap v4974: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T02:41:25.452 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:41:25 smithi189 ceph-mon[36187]: pgmap v4974: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T02:41:25.459 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:41:25 smithi193 ceph-mon[36878]: pgmap v4974: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T02:41:26.445 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:41:26 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 8181.731936 secs 2024-03-24T02:41:26.452 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:41:26 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 8181.731936 secs 2024-03-24T02:41:26.459 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:41:26 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 8181.731936 secs 2024-03-24T02:41:27.445 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:41:27 smithi118 ceph-mon[31535]: pgmap v4975: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T02:41:27.452 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:41:27 smithi189 ceph-mon[36187]: pgmap v4975: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T02:41:27.459 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:41:27 smithi193 ceph-mon[36878]: pgmap v4975: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T02:41:29.453 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:41:29 smithi189 ceph-mon[36187]: pgmap v4976: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T02:41:29.459 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:41:29 smithi193 ceph-mon[36878]: pgmap v4976: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T02:41:29.695 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:41:29 smithi118 ceph-mon[31535]: pgmap v4976: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T02:41:31.453 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:41:31 smithi189 ceph-mon[36187]: pgmap v4977: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T02:41:31.453 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:41:31 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 8186.732018 secs 2024-03-24T02:41:31.459 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:41:31 smithi193 ceph-mon[36878]: pgmap v4977: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T02:41:31.460 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:41:31 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 8186.732018 secs 2024-03-24T02:41:31.695 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:41:31 smithi118 ceph-mon[31535]: pgmap v4977: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T02:41:31.695 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:41:31 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 8186.732018 secs 2024-03-24T02:41:33.460 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:41:33 smithi193 ceph-mon[36878]: pgmap v4978: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:41:33.695 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:41:33 smithi118 ceph-mon[31535]: pgmap v4978: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:41:33.703 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:41:33 smithi189 ceph-mon[36187]: pgmap v4978: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:41:35.695 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:41:35 smithi118 ceph-mon[31535]: pgmap v4979: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T02:41:35.702 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:41:35 smithi189 ceph-mon[36187]: pgmap v4979: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T02:41:35.709 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:41:35 smithi193 ceph-mon[36878]: pgmap v4979: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T02:41:36.597 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:41:36 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 8191.732086 secs 2024-03-24T02:41:36.703 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:41:36 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 8191.732086 secs 2024-03-24T02:41:36.709 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:41:36 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 8191.732086 secs 2024-03-24T02:41:37.695 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:41:37 smithi118 ceph-mon[31535]: pgmap v4980: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T02:41:37.702 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:41:37 smithi189 ceph-mon[36187]: pgmap v4980: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T02:41:37.709 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:41:37 smithi193 ceph-mon[36878]: pgmap v4980: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T02:41:39.695 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:41:39 smithi118 ceph-mon[31535]: pgmap v4981: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T02:41:39.703 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:41:39 smithi189 ceph-mon[36187]: pgmap v4981: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T02:41:39.709 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:41:39 smithi193 ceph-mon[36878]: pgmap v4981: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T02:41:41.695 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:41:41 smithi118 ceph-mon[31535]: pgmap v4982: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T02:41:41.695 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:41:41 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 8196.732163 secs 2024-03-24T02:41:41.703 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:41:41 smithi189 ceph-mon[36187]: pgmap v4982: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T02:41:41.703 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:41:41 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 8196.732163 secs 2024-03-24T02:41:41.709 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:41:41 smithi193 ceph-mon[36878]: pgmap v4982: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T02:41:41.710 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:41:41 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 8196.732163 secs 2024-03-24T02:41:43.695 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:41:43 smithi118 ceph-mon[31535]: pgmap v4983: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:41:43.703 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:41:43 smithi189 ceph-mon[36187]: pgmap v4983: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:41:43.709 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:41:43 smithi193 ceph-mon[36878]: pgmap v4983: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:41:45.695 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:41:45 smithi118 ceph-mon[31535]: pgmap v4984: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-24T02:41:45.703 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:41:45 smithi189 ceph-mon[36187]: pgmap v4984: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-24T02:41:45.710 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:41:45 smithi193 ceph-mon[36878]: pgmap v4984: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-24T02:41:46.603 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:41:46 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 8201.732246 secs 2024-03-24T02:41:46.611 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:41:46 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 8201.732246 secs 2024-03-24T02:41:46.695 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:41:46 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 8201.732246 secs 2024-03-24T02:41:47.695 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:41:47 smithi118 ceph-mon[31535]: pgmap v4985: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T02:41:47.703 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:41:47 smithi189 ceph-mon[36187]: pgmap v4985: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T02:41:47.710 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:41:47 smithi193 ceph-mon[36878]: pgmap v4985: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T02:41:49.445 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:41:49 smithi118 ceph-mon[31535]: pgmap v4986: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T02:41:49.703 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:41:49 smithi189 ceph-mon[36187]: pgmap v4986: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T02:41:49.709 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:41:49 smithi193 ceph-mon[36878]: pgmap v4986: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T02:41:51.695 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:41:51 smithi118 ceph-mon[31535]: pgmap v4987: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T02:41:51.695 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:41:51 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 8206.732310 secs 2024-03-24T02:41:51.702 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:41:51 smithi189 ceph-mon[36187]: pgmap v4987: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T02:41:51.703 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:41:51 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 8206.732310 secs 2024-03-24T02:41:51.709 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:41:51 smithi193 ceph-mon[36878]: pgmap v4987: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T02:41:51.710 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:41:51 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 8206.732310 secs 2024-03-24T02:41:53.695 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:41:53 smithi118 ceph-mon[31535]: pgmap v4988: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:41:53.965 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:41:53 smithi193 ceph-mon[36878]: pgmap v4988: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:41:53.965 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:41:53 smithi189 ceph-mon[36187]: pgmap v4988: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:41:55.695 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:41:55 smithi118 ceph-mon[31535]: pgmap v4989: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-24T02:41:55.703 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:41:55 smithi189 ceph-mon[36187]: pgmap v4989: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-24T02:41:55.709 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:41:55 smithi193 ceph-mon[36878]: pgmap v4989: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-24T02:41:56.695 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:41:56 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 8211.732371 secs 2024-03-24T02:41:56.703 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:41:56 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 8211.732371 secs 2024-03-24T02:41:56.709 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:41:56 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 8211.732371 secs 2024-03-24T02:41:57.695 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:41:57 smithi118 ceph-mon[31535]: pgmap v4990: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:41:57.702 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:41:57 smithi189 ceph-mon[36187]: pgmap v4990: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:41:57.709 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:41:57 smithi193 ceph-mon[36878]: pgmap v4990: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:41:59.695 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:41:59 smithi118 ceph-mon[31535]: pgmap v4991: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T02:41:59.702 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:41:59 smithi189 ceph-mon[36187]: pgmap v4991: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T02:41:59.709 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:41:59 smithi193 ceph-mon[36878]: pgmap v4991: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T02:42:01.695 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:42:01 smithi118 ceph-mon[31535]: pgmap v4992: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T02:42:01.695 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:42:01 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 8216.732474 secs 2024-03-24T02:42:01.702 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:42:01 smithi189 ceph-mon[36187]: pgmap v4992: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T02:42:01.703 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:42:01 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 8216.732474 secs 2024-03-24T02:42:01.709 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:42:01 smithi193 ceph-mon[36878]: pgmap v4992: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T02:42:01.710 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:42:01 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 8216.732474 secs 2024-03-24T02:42:03.695 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:42:03 smithi118 ceph-mon[31535]: pgmap v4993: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:42:03.702 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:42:03 smithi189 ceph-mon[36187]: pgmap v4993: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:42:03.709 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:42:03 smithi193 ceph-mon[36878]: pgmap v4993: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:42:05.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:42:05 smithi118 ceph-mon[31535]: pgmap v4994: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-24T02:42:05.702 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:42:05 smithi189 ceph-mon[36187]: pgmap v4994: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-24T02:42:05.709 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:42:05 smithi193 ceph-mon[36878]: pgmap v4994: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-24T02:42:06.598 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:42:06 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 8221.732534 secs 2024-03-24T02:42:06.703 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:42:06 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 8221.732534 secs 2024-03-24T02:42:06.709 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:42:06 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 8221.732534 secs 2024-03-24T02:42:07.695 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:42:07 smithi118 ceph-mon[31535]: pgmap v4995: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:42:07.702 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:42:07 smithi189 ceph-mon[36187]: pgmap v4995: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:42:07.709 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:42:07 smithi193 ceph-mon[36878]: pgmap v4995: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:42:08.695 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:42:08 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T02:42:08.702 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:42:08 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T02:42:08.709 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:42:08 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T02:42:09.695 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:42:09 smithi118 ceph-mon[31535]: pgmap v4996: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T02:42:09.695 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:42:09 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T02:42:09.695 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:42:09 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T02:42:09.702 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:42:09 smithi189 ceph-mon[36187]: pgmap v4996: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T02:42:09.703 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:42:09 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T02:42:09.703 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:42:09 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T02:42:09.709 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:42:09 smithi193 ceph-mon[36878]: pgmap v4996: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T02:42:09.710 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:42:09 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T02:42:09.710 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:42:09 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T02:42:11.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:42:11 smithi118 ceph-mon[31535]: pgmap v4997: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T02:42:11.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:42:11 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 8226.732619 secs 2024-03-24T02:42:11.702 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:42:11 smithi189 ceph-mon[36187]: pgmap v4997: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T02:42:11.703 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:42:11 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 8226.732619 secs 2024-03-24T02:42:11.709 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:42:11 smithi193 ceph-mon[36878]: pgmap v4997: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T02:42:11.710 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:42:11 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 8226.732619 secs 2024-03-24T02:42:13.695 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:42:13 smithi118 ceph-mon[31535]: pgmap v4998: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:42:13.702 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:42:13 smithi189 ceph-mon[36187]: pgmap v4998: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:42:13.709 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:42:13 smithi193 ceph-mon[36878]: pgmap v4998: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:42:15.445 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:42:15 smithi118 ceph-mon[31535]: pgmap v4999: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-24T02:42:15.445 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:42:15 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T02:42:15.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:42:15 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T02:42:15.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:42:15 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T02:42:15.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:42:15 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T02:42:15.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:42:15 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T02:42:15.703 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:42:15 smithi189 ceph-mon[36187]: pgmap v4999: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-24T02:42:15.703 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:42:15 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T02:42:15.703 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:42:15 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T02:42:15.703 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:42:15 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T02:42:15.703 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:42:15 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T02:42:15.703 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:42:15 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T02:42:15.709 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:42:15 smithi193 ceph-mon[36878]: pgmap v4999: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-24T02:42:15.710 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:42:15 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T02:42:15.710 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:42:15 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T02:42:15.710 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:42:15 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T02:42:15.710 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:42:15 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T02:42:15.710 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:42:15 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T02:42:16.603 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:42:16 smithi189 ceph-mon[36187]: pgmap v5000: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 5 op/s 2024-03-24T02:42:16.604 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:42:16 smithi189 ceph-mon[36187]: pgmap v5001: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T02:42:16.604 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:42:16 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 8231.732660 secs 2024-03-24T02:42:16.611 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:42:16 smithi193 ceph-mon[36878]: pgmap v5000: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 5 op/s 2024-03-24T02:42:16.611 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:42:16 smithi193 ceph-mon[36878]: pgmap v5001: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T02:42:16.611 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:42:16 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 8231.732660 secs 2024-03-24T02:42:16.695 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:42:16 smithi118 ceph-mon[31535]: pgmap v5000: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 5 op/s 2024-03-24T02:42:16.695 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:42:16 smithi118 ceph-mon[31535]: pgmap v5001: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T02:42:16.695 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:42:16 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 8231.732660 secs 2024-03-24T02:42:18.695 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:42:18 smithi118 ceph-mon[31535]: pgmap v5002: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-03-24T02:42:18.702 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:42:18 smithi189 ceph-mon[36187]: pgmap v5002: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-03-24T02:42:18.710 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:42:18 smithi193 ceph-mon[36878]: pgmap v5002: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 4 op/s 2024-03-24T02:42:20.626 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:42:20 smithi118 ceph-mon[31535]: pgmap v5003: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T02:42:20.703 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:42:20 smithi189 ceph-mon[36187]: pgmap v5003: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T02:42:20.709 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:42:20 smithi193 ceph-mon[36878]: pgmap v5003: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T02:42:21.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:42:21 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 8236.732674 secs 2024-03-24T02:42:21.703 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:42:21 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 8236.732674 secs 2024-03-24T02:42:21.709 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:42:21 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 8236.732674 secs 2024-03-24T02:42:22.445 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:42:22 smithi118 ceph-mon[31535]: pgmap v5004: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T02:42:22.703 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:42:22 smithi189 ceph-mon[36187]: pgmap v5004: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T02:42:22.710 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:42:22 smithi193 ceph-mon[36878]: pgmap v5004: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T02:42:24.695 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:42:24 smithi118 ceph-mon[31535]: pgmap v5005: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 4 op/s 2024-03-24T02:42:24.716 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:42:24 smithi193 ceph-mon[36878]: pgmap v5005: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 4 op/s 2024-03-24T02:42:24.716 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:42:24 smithi189 ceph-mon[36187]: pgmap v5005: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 4 op/s 2024-03-24T02:42:26.695 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:42:26 smithi118 ceph-mon[31535]: pgmap v5006: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 5 op/s 2024-03-24T02:42:26.695 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:42:26 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 8241.732734 secs 2024-03-24T02:42:26.703 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:42:26 smithi189 ceph-mon[36187]: pgmap v5006: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 5 op/s 2024-03-24T02:42:26.703 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:42:26 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 8241.732734 secs 2024-03-24T02:42:26.709 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:42:26 smithi193 ceph-mon[36878]: pgmap v5006: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 5 op/s 2024-03-24T02:42:26.710 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:42:26 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 8241.732734 secs 2024-03-24T02:42:28.695 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:42:28 smithi118 ceph-mon[31535]: pgmap v5007: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:42:28.702 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:42:28 smithi189 ceph-mon[36187]: pgmap v5007: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:42:28.709 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:42:28 smithi193 ceph-mon[36878]: pgmap v5007: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:42:30.695 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:42:30 smithi118 ceph-mon[31535]: pgmap v5008: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-24T02:42:30.702 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:42:30 smithi189 ceph-mon[36187]: pgmap v5008: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-24T02:42:30.710 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:42:30 smithi193 ceph-mon[36878]: pgmap v5008: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-24T02:42:31.652 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:42:31 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 8246.732823 secs 2024-03-24T02:42:31.703 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:42:31 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 8246.732823 secs 2024-03-24T02:42:31.709 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:42:31 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 8246.732823 secs 2024-03-24T02:42:32.695 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:42:32 smithi118 ceph-mon[31535]: pgmap v5009: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-03-24T02:42:32.703 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:42:32 smithi189 ceph-mon[36187]: pgmap v5009: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-03-24T02:42:32.709 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:42:32 smithi193 ceph-mon[36878]: pgmap v5009: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-03-24T02:42:34.695 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:42:34 smithi118 ceph-mon[31535]: pgmap v5010: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-03-24T02:42:34.703 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:42:34 smithi189 ceph-mon[36187]: pgmap v5010: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-03-24T02:42:34.710 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:42:34 smithi193 ceph-mon[36878]: pgmap v5010: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-03-24T02:42:36.695 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:42:36 smithi118 ceph-mon[31535]: pgmap v5011: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 6 op/s 2024-03-24T02:42:36.695 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:42:36 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 8251.732864 secs 2024-03-24T02:42:36.703 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:42:36 smithi189 ceph-mon[36187]: pgmap v5011: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 6 op/s 2024-03-24T02:42:36.703 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:42:36 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 8251.732864 secs 2024-03-24T02:42:36.709 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:42:36 smithi193 ceph-mon[36878]: pgmap v5011: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 6 op/s 2024-03-24T02:42:36.710 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:42:36 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 8251.732864 secs 2024-03-24T02:42:38.695 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:42:38 smithi118 ceph-mon[31535]: pgmap v5012: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-03-24T02:42:38.702 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:42:38 smithi189 ceph-mon[36187]: pgmap v5012: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-03-24T02:42:38.709 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:42:38 smithi193 ceph-mon[36878]: pgmap v5012: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-03-24T02:42:40.695 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:42:40 smithi118 ceph-mon[31535]: pgmap v5013: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-24T02:42:40.703 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:42:40 smithi189 ceph-mon[36187]: pgmap v5013: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-24T02:42:40.709 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:42:40 smithi193 ceph-mon[36878]: pgmap v5013: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-24T02:42:41.695 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:42:41 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 8256.732907 secs 2024-03-24T02:42:41.702 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:42:41 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 8256.732907 secs 2024-03-24T02:42:41.709 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:42:41 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 8256.732907 secs 2024-03-24T02:42:42.695 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:42:42 smithi118 ceph-mon[31535]: pgmap v5014: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:42:42.702 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:42:42 smithi189 ceph-mon[36187]: pgmap v5014: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:42:42.709 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:42:42 smithi193 ceph-mon[36878]: pgmap v5014: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:42:44.695 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:42:44 smithi118 ceph-mon[31535]: pgmap v5015: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T02:42:44.703 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:42:44 smithi189 ceph-mon[36187]: pgmap v5015: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T02:42:44.709 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:42:44 smithi193 ceph-mon[36878]: pgmap v5015: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T02:42:46.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:42:46 smithi118 ceph-mon[31535]: pgmap v5016: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T02:42:46.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:42:46 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 8261.732969 secs 2024-03-24T02:42:46.703 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:42:46 smithi189 ceph-mon[36187]: pgmap v5016: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T02:42:46.703 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:42:46 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 8261.732969 secs 2024-03-24T02:42:46.709 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:42:46 smithi193 ceph-mon[36878]: pgmap v5016: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T02:42:46.710 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:42:46 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 8261.732969 secs 2024-03-24T02:42:48.695 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:42:48 smithi118 ceph-mon[31535]: pgmap v5017: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:42:48.702 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:42:48 smithi189 ceph-mon[36187]: pgmap v5017: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:42:48.709 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:42:48 smithi193 ceph-mon[36878]: pgmap v5017: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:42:50.695 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:42:50 smithi118 ceph-mon[31535]: pgmap v5018: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-24T02:42:50.702 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:42:50 smithi189 ceph-mon[36187]: pgmap v5018: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-24T02:42:50.709 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:42:50 smithi193 ceph-mon[36878]: pgmap v5018: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-24T02:42:51.703 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:42:51 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 8266.733030 secs 2024-03-24T02:42:51.709 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:42:51 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 8266.733030 secs 2024-03-24T02:42:51.945 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:42:51 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 8266.733030 secs 2024-03-24T02:42:52.703 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:42:52 smithi189 ceph-mon[36187]: pgmap v5019: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:42:52.709 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:42:52 smithi193 ceph-mon[36878]: pgmap v5019: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:42:52.945 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:42:52 smithi118 ceph-mon[31535]: pgmap v5019: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:42:54.703 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:42:54 smithi189 ceph-mon[36187]: pgmap v5020: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T02:42:54.709 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:42:54 smithi193 ceph-mon[36878]: pgmap v5020: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T02:42:54.945 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:42:54 smithi118 ceph-mon[31535]: pgmap v5020: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T02:42:56.710 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:42:56 smithi193 ceph-mon[36878]: pgmap v5021: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T02:42:56.710 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:42:56 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 8271.733089 secs 2024-03-24T02:42:56.945 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:42:56 smithi118 ceph-mon[31535]: pgmap v5021: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T02:42:56.945 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:42:56 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 8271.733089 secs 2024-03-24T02:42:56.953 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:42:56 smithi189 ceph-mon[36187]: pgmap v5021: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T02:42:56.953 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:42:56 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 8271.733089 secs 2024-03-24T02:42:58.945 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:42:58 smithi118 ceph-mon[31535]: pgmap v5022: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:42:58.953 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:42:58 smithi189 ceph-mon[36187]: pgmap v5022: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:42:58.959 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:42:58 smithi193 ceph-mon[36878]: pgmap v5022: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:43:00.945 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:43:00 smithi118 ceph-mon[31535]: pgmap v5023: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-24T02:43:00.953 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:43:00 smithi189 ceph-mon[36187]: pgmap v5023: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-24T02:43:00.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:43:00 smithi193 ceph-mon[36878]: pgmap v5023: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-24T02:43:01.945 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:43:01 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 8276.733150 secs 2024-03-24T02:43:01.953 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:43:01 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 8276.733150 secs 2024-03-24T02:43:01.959 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:43:01 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 8276.733150 secs 2024-03-24T02:43:02.945 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:43:02 smithi118 ceph-mon[31535]: pgmap v5024: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:43:02.953 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:43:02 smithi189 ceph-mon[36187]: pgmap v5024: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:43:02.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:43:02 smithi193 ceph-mon[36878]: pgmap v5024: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:43:04.945 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:43:04 smithi118 ceph-mon[31535]: pgmap v5025: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T02:43:04.953 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:43:04 smithi189 ceph-mon[36187]: pgmap v5025: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T02:43:04.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:43:04 smithi193 ceph-mon[36878]: pgmap v5025: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T02:43:06.945 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:43:06 smithi118 ceph-mon[31535]: pgmap v5026: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T02:43:06.945 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:43:06 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 8281.733211 secs 2024-03-24T02:43:06.953 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:43:06 smithi189 ceph-mon[36187]: pgmap v5026: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T02:43:06.953 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:43:06 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 8281.733211 secs 2024-03-24T02:43:06.959 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:43:06 smithi193 ceph-mon[36878]: pgmap v5026: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T02:43:06.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:43:06 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 8281.733211 secs 2024-03-24T02:43:08.945 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:43:08 smithi118 ceph-mon[31535]: pgmap v5027: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:43:08.953 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:43:08 smithi189 ceph-mon[36187]: pgmap v5027: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:43:08.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:43:08 smithi193 ceph-mon[36878]: pgmap v5027: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:43:10.945 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:43:10 smithi118 ceph-mon[31535]: pgmap v5028: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-24T02:43:10.953 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:43:10 smithi189 ceph-mon[36187]: pgmap v5028: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-24T02:43:10.959 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:43:10 smithi193 ceph-mon[36878]: pgmap v5028: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-24T02:43:11.945 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:43:11 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 8286.733270 secs 2024-03-24T02:43:11.953 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:43:11 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 8286.733270 secs 2024-03-24T02:43:11.959 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:43:11 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 8286.733270 secs 2024-03-24T02:43:12.945 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:43:12 smithi118 ceph-mon[31535]: pgmap v5029: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:43:12.953 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:43:12 smithi189 ceph-mon[36187]: pgmap v5029: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:43:12.959 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:43:12 smithi193 ceph-mon[36878]: pgmap v5029: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:43:14.945 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:43:14 smithi118 ceph-mon[31535]: pgmap v5030: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T02:43:14.953 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:43:14 smithi189 ceph-mon[36187]: pgmap v5030: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T02:43:14.959 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:43:14 smithi193 ceph-mon[36878]: pgmap v5030: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T02:43:15.945 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:43:15 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T02:43:15.953 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:43:15 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T02:43:15.959 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:43:15 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T02:43:16.854 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:43:16 smithi189 ceph-mon[36187]: pgmap v5031: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T02:43:16.854 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:43:16 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 8291.733347 secs 2024-03-24T02:43:16.860 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:43:16 smithi193 ceph-mon[36878]: pgmap v5031: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T02:43:16.860 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:43:16 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 8291.733347 secs 2024-03-24T02:43:16.945 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:43:16 smithi118 ceph-mon[31535]: pgmap v5031: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T02:43:16.945 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:43:16 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 8291.733347 secs 2024-03-24T02:43:17.073 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T02:43:17.073+0000 7f81ee000640 1 -- 172.21.15.118:0/1693767697 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f81c80036c0 con 0x7f81f81052d0 2024-03-24T02:43:17.073 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T02:43:17.073+0000 7f81db400640 1 -- 172.21.15.118:0/659686424 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f81c4006460 con 0x5622651a13f0 2024-03-24T02:43:17.073 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T02:43:17.074+0000 7f81e4c00640 1 -- 172.21.15.118:0/659686424 <== mon.1 v2:172.21.15.193:3300/0 67 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 409+0+0 (secure 0 0 0) 0x7f81c4006460 con 0x5622651a13f0 2024-03-24T02:43:17.073 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T02:43:17.074+0000 7f81ef400640 1 -- 172.21.15.118:0/1693767697 <== mon.2 v2:172.21.15.189:3300/0 67 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (secure 0 0 0) 0x7f81c80036c0 con 0x7f81f81052d0 2024-03-24T02:43:18.710 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:43:18 smithi193 ceph-mon[36878]: pgmap v5032: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:43:18.945 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:43:18 smithi118 ceph-mon[31535]: pgmap v5032: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:43:18.953 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:43:18 smithi189 ceph-mon[36187]: pgmap v5032: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:43:20.703 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:43:20 smithi189 ceph-mon[36187]: pgmap v5033: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-24T02:43:20.703 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:43:20 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T02:43:20.703 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:43:20 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T02:43:20.703 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:43:20 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T02:43:20.704 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:43:20 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T02:43:20.704 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:43:20 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T02:43:20.704 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:43:20 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T02:43:20.704 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:43:20 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T02:43:20.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:43:20 smithi118 ceph-mon[31535]: pgmap v5033: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-24T02:43:20.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:43:20 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T02:43:20.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:43:20 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T02:43:20.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:43:20 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T02:43:20.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:43:20 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T02:43:20.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:43:20 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T02:43:20.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:43:20 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T02:43:20.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:43:20 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T02:43:20.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:43:20 smithi193 ceph-mon[36878]: pgmap v5033: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-24T02:43:20.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:43:20 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T02:43:20.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:43:20 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T02:43:20.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:43:20 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T02:43:20.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:43:20 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T02:43:20.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:43:20 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T02:43:20.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:43:20 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T02:43:20.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:43:20 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T02:43:21.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:43:21 smithi118 ceph-mon[31535]: pgmap v5034: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 5 op/s 2024-03-24T02:43:21.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:43:21 smithi118 ceph-mon[31535]: pgmap v5035: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T02:43:21.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:43:21 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 8296.733416 secs 2024-03-24T02:43:21.953 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:43:21 smithi189 ceph-mon[36187]: pgmap v5034: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 5 op/s 2024-03-24T02:43:21.953 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:43:21 smithi189 ceph-mon[36187]: pgmap v5035: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T02:43:21.953 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:43:21 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 8296.733416 secs 2024-03-24T02:43:21.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:43:21 smithi193 ceph-mon[36878]: pgmap v5034: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 5 op/s 2024-03-24T02:43:21.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:43:21 smithi193 ceph-mon[36878]: pgmap v5035: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T02:43:21.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:43:21 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 8296.733416 secs 2024-03-24T02:43:23.945 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:43:23 smithi118 ceph-mon[31535]: pgmap v5036: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 5 op/s 2024-03-24T02:43:23.953 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:43:23 smithi189 ceph-mon[36187]: pgmap v5036: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 5 op/s 2024-03-24T02:43:23.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:43:23 smithi193 ceph-mon[36878]: pgmap v5036: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.2 KiB/s rd, 5 op/s 2024-03-24T02:43:25.695 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:43:25 smithi118 ceph-mon[31535]: pgmap v5037: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T02:43:25.953 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:43:25 smithi189 ceph-mon[36187]: pgmap v5037: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T02:43:25.959 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:43:25 smithi193 ceph-mon[36878]: pgmap v5037: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T02:43:26.945 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:43:26 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 8301.733474 secs 2024-03-24T02:43:26.953 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:43:26 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 8301.733474 secs 2024-03-24T02:43:26.959 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:43:26 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 8301.733474 secs 2024-03-24T02:43:27.945 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:43:27 smithi118 ceph-mon[31535]: pgmap v5038: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T02:43:27.953 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:43:27 smithi189 ceph-mon[36187]: pgmap v5038: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T02:43:27.959 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:43:27 smithi193 ceph-mon[36878]: pgmap v5038: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T02:43:29.945 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:43:29 smithi118 ceph-mon[31535]: pgmap v5039: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 5 op/s 2024-03-24T02:43:29.953 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:43:29 smithi189 ceph-mon[36187]: pgmap v5039: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 5 op/s 2024-03-24T02:43:29.959 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:43:29 smithi193 ceph-mon[36878]: pgmap v5039: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 5 op/s 2024-03-24T02:43:31.945 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:43:31 smithi118 ceph-mon[31535]: pgmap v5040: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 5 op/s 2024-03-24T02:43:31.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:43:31 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 8306.733512 secs 2024-03-24T02:43:31.953 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:43:31 smithi189 ceph-mon[36187]: pgmap v5040: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 5 op/s 2024-03-24T02:43:31.953 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:43:31 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 8306.733512 secs 2024-03-24T02:43:31.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:43:31 smithi193 ceph-mon[36878]: pgmap v5040: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 5 op/s 2024-03-24T02:43:31.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:43:31 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 8306.733512 secs 2024-03-24T02:43:33.945 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:43:33 smithi118 ceph-mon[31535]: pgmap v5041: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:43:33.953 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:43:33 smithi189 ceph-mon[36187]: pgmap v5041: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:43:33.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:43:33 smithi193 ceph-mon[36878]: pgmap v5041: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:43:35.945 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:43:35 smithi118 ceph-mon[31535]: pgmap v5042: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-24T02:43:35.953 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:43:35 smithi189 ceph-mon[36187]: pgmap v5042: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-24T02:43:35.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:43:35 smithi193 ceph-mon[36878]: pgmap v5042: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-24T02:43:36.847 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:43:36 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 8311.733525 secs 2024-03-24T02:43:36.953 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:43:36 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 8311.733525 secs 2024-03-24T02:43:36.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:43:36 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 8311.733525 secs 2024-03-24T02:43:37.945 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:43:37 smithi118 ceph-mon[31535]: pgmap v5043: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:43:37.953 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:43:37 smithi189 ceph-mon[36187]: pgmap v5043: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:43:37.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:43:37 smithi193 ceph-mon[36878]: pgmap v5043: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:43:39.945 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:43:39 smithi118 ceph-mon[31535]: pgmap v5044: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T02:43:39.953 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:43:39 smithi189 ceph-mon[36187]: pgmap v5044: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T02:43:39.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:43:39 smithi193 ceph-mon[36878]: pgmap v5044: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T02:43:41.945 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:43:41 smithi118 ceph-mon[31535]: pgmap v5045: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T02:43:41.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:43:41 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 8316.733582 secs 2024-03-24T02:43:41.953 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:43:41 smithi189 ceph-mon[36187]: pgmap v5045: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T02:43:41.953 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:43:41 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 8316.733582 secs 2024-03-24T02:43:41.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:43:41 smithi193 ceph-mon[36878]: pgmap v5045: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T02:43:41.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:43:41 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 8316.733582 secs 2024-03-24T02:43:43.945 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:43:43 smithi118 ceph-mon[31535]: pgmap v5046: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:43:43.953 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:43:43 smithi189 ceph-mon[36187]: pgmap v5046: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:43:43.959 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:43:43 smithi193 ceph-mon[36878]: pgmap v5046: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:43:45.945 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:43:45 smithi118 ceph-mon[31535]: pgmap v5047: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-24T02:43:45.953 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:43:45 smithi189 ceph-mon[36187]: pgmap v5047: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-24T02:43:45.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:43:45 smithi193 ceph-mon[36878]: pgmap v5047: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-24T02:43:46.945 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:43:46 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 8321.733646 secs 2024-03-24T02:43:46.953 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:43:46 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 8321.733646 secs 2024-03-24T02:43:46.959 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:43:46 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 8321.733646 secs 2024-03-24T02:43:47.945 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:43:47 smithi118 ceph-mon[31535]: pgmap v5048: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:43:47.953 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:43:47 smithi189 ceph-mon[36187]: pgmap v5048: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:43:47.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:43:47 smithi193 ceph-mon[36878]: pgmap v5048: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:43:48.945 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:43:48 smithi118 ceph-mon[31535]: pgmap v5049: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T02:43:48.953 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:43:48 smithi189 ceph-mon[36187]: pgmap v5049: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T02:43:48.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:43:48 smithi193 ceph-mon[36878]: pgmap v5049: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T02:43:51.695 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:43:51 smithi118 ceph-mon[31535]: pgmap v5050: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T02:43:51.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:43:51 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 8326.733735 secs 2024-03-24T02:43:51.703 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:43:51 smithi189 ceph-mon[36187]: pgmap v5050: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T02:43:51.703 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:43:51 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 8326.733735 secs 2024-03-24T02:43:51.710 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:43:51 smithi193 ceph-mon[36878]: pgmap v5050: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T02:43:51.710 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:43:51 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 8326.733735 secs 2024-03-24T02:43:53.695 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:43:53 smithi118 ceph-mon[31535]: pgmap v5051: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:43:53.703 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:43:53 smithi189 ceph-mon[36187]: pgmap v5051: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:43:53.710 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:43:53 smithi193 ceph-mon[36878]: pgmap v5051: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:43:55.695 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:43:55 smithi118 ceph-mon[31535]: pgmap v5052: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-24T02:43:55.702 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:43:55 smithi189 ceph-mon[36187]: pgmap v5052: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-24T02:43:55.710 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:43:55 smithi193 ceph-mon[36878]: pgmap v5052: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-24T02:43:56.695 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:43:56 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 8331.733822 secs 2024-03-24T02:43:56.703 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:43:56 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 8331.733822 secs 2024-03-24T02:43:56.709 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:43:56 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 8331.733822 secs 2024-03-24T02:43:57.695 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:43:57 smithi118 ceph-mon[31535]: pgmap v5053: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:43:57.703 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:43:57 smithi189 ceph-mon[36187]: pgmap v5053: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:43:57.710 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:43:57 smithi193 ceph-mon[36878]: pgmap v5053: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:43:59.695 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:43:59 smithi118 ceph-mon[31535]: pgmap v5054: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T02:43:59.703 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:43:59 smithi189 ceph-mon[36187]: pgmap v5054: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T02:43:59.709 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:43:59 smithi193 ceph-mon[36878]: pgmap v5054: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T02:44:01.695 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:44:01 smithi118 ceph-mon[31535]: pgmap v5055: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T02:44:01.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:44:01 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 8336.733907 secs 2024-03-24T02:44:01.703 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:44:01 smithi189 ceph-mon[36187]: pgmap v5055: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T02:44:01.703 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:44:01 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 8336.733907 secs 2024-03-24T02:44:01.710 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:44:01 smithi193 ceph-mon[36878]: pgmap v5055: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T02:44:01.710 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:44:01 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 8336.733907 secs 2024-03-24T02:44:03.695 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:44:03 smithi118 ceph-mon[31535]: pgmap v5056: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:44:03.703 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:44:03 smithi189 ceph-mon[36187]: pgmap v5056: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:44:03.710 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:44:03 smithi193 ceph-mon[36878]: pgmap v5056: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:44:05.695 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:44:05 smithi118 ceph-mon[31535]: pgmap v5057: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-24T02:44:05.703 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:44:05 smithi189 ceph-mon[36187]: pgmap v5057: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-24T02:44:05.709 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:44:05 smithi193 ceph-mon[36878]: pgmap v5057: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-24T02:44:06.695 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:44:06 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 8341.733994 secs 2024-03-24T02:44:06.703 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:44:06 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 8341.733994 secs 2024-03-24T02:44:06.710 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:44:06 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 8341.733994 secs 2024-03-24T02:44:07.945 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:44:07 smithi118 ceph-mon[31535]: pgmap v5058: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:44:07.953 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:44:07 smithi189 ceph-mon[36187]: pgmap v5058: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:44:07.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:44:07 smithi193 ceph-mon[36878]: pgmap v5058: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:44:09.945 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:44:09 smithi118 ceph-mon[31535]: pgmap v5059: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T02:44:09.953 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:44:09 smithi189 ceph-mon[36187]: pgmap v5059: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T02:44:09.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:44:09 smithi193 ceph-mon[36878]: pgmap v5059: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T02:44:11.927 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:44:11 smithi118 ceph-mon[31535]: pgmap v5060: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T02:44:11.928 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:44:11 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 8346.734096 secs 2024-03-24T02:44:11.953 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:44:11 smithi189 ceph-mon[36187]: pgmap v5060: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T02:44:11.953 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:44:11 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 8346.734096 secs 2024-03-24T02:44:11.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:44:11 smithi193 ceph-mon[36878]: pgmap v5060: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T02:44:11.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:44:11 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 8346.734096 secs 2024-03-24T02:44:13.945 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:44:13 smithi118 ceph-mon[31535]: pgmap v5061: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:44:13.953 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:44:13 smithi189 ceph-mon[36187]: pgmap v5061: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:44:13.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:44:13 smithi193 ceph-mon[36878]: pgmap v5061: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:44:15.945 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:44:15 smithi118 ceph-mon[31535]: pgmap v5062: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-24T02:44:15.953 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:44:15 smithi189 ceph-mon[36187]: pgmap v5062: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-24T02:44:15.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:44:15 smithi193 ceph-mon[36878]: pgmap v5062: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-24T02:44:16.854 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:44:16 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 8351.734159 secs 2024-03-24T02:44:16.860 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:44:16 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 8351.734159 secs 2024-03-24T02:44:16.945 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:44:16 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 8351.734159 secs 2024-03-24T02:44:17.945 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:44:17 smithi118 ceph-mon[31535]: pgmap v5063: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:44:17.953 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:44:17 smithi189 ceph-mon[36187]: pgmap v5063: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:44:17.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:44:17 smithi193 ceph-mon[36878]: pgmap v5063: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:44:19.945 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:44:19 smithi118 ceph-mon[31535]: pgmap v5064: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T02:44:19.953 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:44:19 smithi189 ceph-mon[36187]: pgmap v5064: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T02:44:19.959 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:44:19 smithi193 ceph-mon[36878]: pgmap v5064: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T02:44:20.945 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:44:20 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T02:44:20.953 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:44:20 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T02:44:20.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:44:20 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T02:44:21.945 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:44:21 smithi118 ceph-mon[31535]: pgmap v5065: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T02:44:21.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:44:21 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 8356.734220 secs 2024-03-24T02:44:21.953 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:44:21 smithi189 ceph-mon[36187]: pgmap v5065: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T02:44:21.953 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:44:21 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 8356.734220 secs 2024-03-24T02:44:21.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:44:21 smithi193 ceph-mon[36878]: pgmap v5065: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T02:44:21.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:44:21 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 8356.734220 secs 2024-03-24T02:44:22.945 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:44:22 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T02:44:22.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:44:22 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T02:44:22.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:44:22 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T02:44:22.953 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:44:22 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T02:44:22.953 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:44:22 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T02:44:22.953 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:44:22 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T02:44:22.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:44:22 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T02:44:22.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:44:22 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T02:44:22.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:44:22 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T02:44:23.945 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:44:23 smithi118 ceph-mon[31535]: pgmap v5066: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 5 op/s 2024-03-24T02:44:23.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:44:23 smithi118 ceph-mon[31535]: pgmap v5067: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 6 op/s 2024-03-24T02:44:23.953 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:44:23 smithi189 ceph-mon[36187]: pgmap v5066: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 5 op/s 2024-03-24T02:44:23.953 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:44:23 smithi189 ceph-mon[36187]: pgmap v5067: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 6 op/s 2024-03-24T02:44:23.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:44:23 smithi193 ceph-mon[36878]: pgmap v5066: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 5 op/s 2024-03-24T02:44:23.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:44:23 smithi193 ceph-mon[36878]: pgmap v5067: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 6 op/s 2024-03-24T02:44:25.945 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:44:25 smithi118 ceph-mon[31535]: pgmap v5068: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 4 op/s 2024-03-24T02:44:25.953 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:44:25 smithi189 ceph-mon[36187]: pgmap v5068: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 4 op/s 2024-03-24T02:44:25.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:44:25 smithi193 ceph-mon[36878]: pgmap v5068: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 4 op/s 2024-03-24T02:44:26.945 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:44:26 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 8361.734235 secs 2024-03-24T02:44:26.953 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:44:26 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 8361.734235 secs 2024-03-24T02:44:26.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:44:26 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 8361.734235 secs 2024-03-24T02:44:27.945 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:44:27 smithi118 ceph-mon[31535]: pgmap v5069: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 6 op/s 2024-03-24T02:44:27.953 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:44:27 smithi189 ceph-mon[36187]: pgmap v5069: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 6 op/s 2024-03-24T02:44:27.959 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:44:27 smithi193 ceph-mon[36878]: pgmap v5069: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 6 op/s 2024-03-24T02:44:29.945 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:44:29 smithi118 ceph-mon[31535]: pgmap v5070: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 5 op/s 2024-03-24T02:44:29.953 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:44:29 smithi189 ceph-mon[36187]: pgmap v5070: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 5 op/s 2024-03-24T02:44:29.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:44:29 smithi193 ceph-mon[36878]: pgmap v5070: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 5 op/s 2024-03-24T02:44:31.945 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:44:31 smithi118 ceph-mon[31535]: pgmap v5071: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 6 op/s 2024-03-24T02:44:31.945 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:44:31 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 8366.734299 secs 2024-03-24T02:44:31.953 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:44:31 smithi189 ceph-mon[36187]: pgmap v5071: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 6 op/s 2024-03-24T02:44:31.953 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:44:31 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 8366.734299 secs 2024-03-24T02:44:31.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:44:31 smithi193 ceph-mon[36878]: pgmap v5071: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 6 op/s 2024-03-24T02:44:31.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:44:31 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 8366.734299 secs 2024-03-24T02:44:33.945 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:44:33 smithi118 ceph-mon[31535]: pgmap v5072: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 5 op/s 2024-03-24T02:44:33.953 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:44:33 smithi189 ceph-mon[36187]: pgmap v5072: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 5 op/s 2024-03-24T02:44:33.959 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:44:33 smithi193 ceph-mon[36878]: pgmap v5072: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 5 op/s 2024-03-24T02:44:35.945 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:44:35 smithi118 ceph-mon[31535]: pgmap v5073: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T02:44:35.953 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:44:35 smithi189 ceph-mon[36187]: pgmap v5073: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T02:44:35.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:44:35 smithi193 ceph-mon[36878]: pgmap v5073: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T02:44:36.945 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:44:36 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 8371.734316 secs 2024-03-24T02:44:36.953 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:44:36 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 8371.734316 secs 2024-03-24T02:44:36.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:44:36 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 8371.734316 secs 2024-03-24T02:44:37.945 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:44:37 smithi118 ceph-mon[31535]: pgmap v5074: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T02:44:37.953 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:44:37 smithi189 ceph-mon[36187]: pgmap v5074: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T02:44:37.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:44:37 smithi193 ceph-mon[36878]: pgmap v5074: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T02:44:38.945 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:44:38 smithi118 ceph-mon[31535]: pgmap v5075: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T02:44:38.953 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:44:38 smithi189 ceph-mon[36187]: pgmap v5075: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T02:44:38.959 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:44:38 smithi193 ceph-mon[36878]: pgmap v5075: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T02:44:41.195 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:44:40 smithi118 ceph-mon[31535]: pgmap v5076: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-03-24T02:44:41.203 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:44:40 smithi189 ceph-mon[36187]: pgmap v5076: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-03-24T02:44:41.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:44:40 smithi193 ceph-mon[36878]: pgmap v5076: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-03-24T02:44:42.195 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:44:41 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 8376.734404 secs 2024-03-24T02:44:42.203 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:44:41 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 8376.734404 secs 2024-03-24T02:44:42.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:44:41 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 8376.734404 secs 2024-03-24T02:44:43.161 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:44:42 smithi118 ceph-mon[31535]: pgmap v5077: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:44:43.203 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:44:42 smithi189 ceph-mon[36187]: pgmap v5077: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:44:43.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:44:42 smithi193 ceph-mon[36878]: pgmap v5077: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:44:45.195 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:44:44 smithi118 ceph-mon[31535]: pgmap v5078: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T02:44:45.203 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:44:44 smithi189 ceph-mon[36187]: pgmap v5078: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T02:44:45.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:44:44 smithi193 ceph-mon[36878]: pgmap v5078: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T02:44:47.195 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:44:46 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 8381.734468 secs 2024-03-24T02:44:47.195 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:44:46 smithi118 ceph-mon[31535]: pgmap v5079: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T02:44:47.203 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:44:46 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 8381.734468 secs 2024-03-24T02:44:47.203 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:44:46 smithi189 ceph-mon[36187]: pgmap v5079: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T02:44:47.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:44:46 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 8381.734468 secs 2024-03-24T02:44:47.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:44:46 smithi193 ceph-mon[36878]: pgmap v5079: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T02:44:49.195 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:44:48 smithi118 ceph-mon[31535]: pgmap v5080: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T02:44:49.203 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:44:48 smithi189 ceph-mon[36187]: pgmap v5080: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T02:44:49.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:44:48 smithi193 ceph-mon[36878]: pgmap v5080: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T02:44:51.195 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:44:50 smithi118 ceph-mon[31535]: pgmap v5081: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-03-24T02:44:51.203 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:44:50 smithi189 ceph-mon[36187]: pgmap v5081: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-03-24T02:44:51.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:44:50 smithi193 ceph-mon[36878]: pgmap v5081: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-03-24T02:44:52.195 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:44:51 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 8386.734560 secs 2024-03-24T02:44:52.203 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:44:51 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 8386.734560 secs 2024-03-24T02:44:52.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:44:51 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 8386.734560 secs 2024-03-24T02:44:53.195 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:44:52 smithi118 ceph-mon[31535]: pgmap v5082: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:44:53.203 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:44:52 smithi189 ceph-mon[36187]: pgmap v5082: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:44:53.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:44:52 smithi193 ceph-mon[36878]: pgmap v5082: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:44:55.195 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:44:54 smithi118 ceph-mon[31535]: pgmap v5083: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T02:44:55.203 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:44:54 smithi189 ceph-mon[36187]: pgmap v5083: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T02:44:55.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:44:54 smithi193 ceph-mon[36878]: pgmap v5083: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T02:44:57.195 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:44:56 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 8391.734646 secs 2024-03-24T02:44:57.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:44:56 smithi118 ceph-mon[31535]: pgmap v5084: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T02:44:57.203 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:44:56 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 8391.734646 secs 2024-03-24T02:44:57.203 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:44:56 smithi189 ceph-mon[36187]: pgmap v5084: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T02:44:57.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:44:56 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 8391.734646 secs 2024-03-24T02:44:57.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:44:56 smithi193 ceph-mon[36878]: pgmap v5084: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T02:44:59.195 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:44:58 smithi118 ceph-mon[31535]: pgmap v5085: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T02:44:59.203 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:44:58 smithi189 ceph-mon[36187]: pgmap v5085: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T02:44:59.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:44:58 smithi193 ceph-mon[36878]: pgmap v5085: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T02:45:01.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:45:00 smithi118 ceph-mon[31535]: pgmap v5086: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-03-24T02:45:01.203 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:45:00 smithi189 ceph-mon[36187]: pgmap v5086: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-03-24T02:45:01.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:45:00 smithi193 ceph-mon[36878]: pgmap v5086: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-03-24T02:45:02.195 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:45:01 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 8396.734734 secs 2024-03-24T02:45:02.203 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:45:01 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 8396.734734 secs 2024-03-24T02:45:02.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:45:01 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 8396.734734 secs 2024-03-24T02:45:03.945 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:45:03 smithi118 ceph-mon[31535]: pgmap v5087: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:45:03.953 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:45:03 smithi189 ceph-mon[36187]: pgmap v5087: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:45:03.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:45:03 smithi193 ceph-mon[36878]: pgmap v5087: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:45:05.945 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:45:05 smithi118 ceph-mon[31535]: pgmap v5088: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T02:45:05.953 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:45:05 smithi189 ceph-mon[36187]: pgmap v5088: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T02:45:05.959 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:45:05 smithi193 ceph-mon[36878]: pgmap v5088: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T02:45:06.847 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:45:06 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 8401.734821 secs 2024-03-24T02:45:06.953 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:45:06 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 8401.734821 secs 2024-03-24T02:45:06.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:45:06 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 8401.734821 secs 2024-03-24T02:45:07.945 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:45:07 smithi118 ceph-mon[31535]: pgmap v5089: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T02:45:07.953 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:45:07 smithi189 ceph-mon[36187]: pgmap v5089: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T02:45:07.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:45:07 smithi193 ceph-mon[36878]: pgmap v5089: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T02:45:09.945 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:45:09 smithi118 ceph-mon[31535]: pgmap v5090: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T02:45:09.953 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:45:09 smithi189 ceph-mon[36187]: pgmap v5090: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T02:45:09.959 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:45:09 smithi193 ceph-mon[36878]: pgmap v5090: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T02:45:11.945 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:45:11 smithi118 ceph-mon[31535]: pgmap v5091: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-03-24T02:45:11.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:45:11 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 8406.734906 secs 2024-03-24T02:45:11.953 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:45:11 smithi189 ceph-mon[36187]: pgmap v5091: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-03-24T02:45:11.953 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:45:11 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 8406.734906 secs 2024-03-24T02:45:11.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:45:11 smithi193 ceph-mon[36878]: pgmap v5091: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-03-24T02:45:11.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:45:11 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 8406.734906 secs 2024-03-24T02:45:13.880 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:45:13 smithi118 ceph-mon[31535]: pgmap v5092: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:45:13.953 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:45:13 smithi189 ceph-mon[36187]: pgmap v5092: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:45:13.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:45:13 smithi193 ceph-mon[36878]: pgmap v5092: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:45:15.945 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:45:15 smithi118 ceph-mon[31535]: pgmap v5093: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-03-24T02:45:15.953 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:45:15 smithi189 ceph-mon[36187]: pgmap v5093: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-03-24T02:45:15.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:45:15 smithi193 ceph-mon[36878]: pgmap v5093: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-03-24T02:45:16.945 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:45:16 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 8411.734962 secs 2024-03-24T02:45:16.953 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:45:16 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 8411.734962 secs 2024-03-24T02:45:16.959 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:45:16 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 8411.734962 secs 2024-03-24T02:45:17.945 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:45:17 smithi118 ceph-mon[31535]: pgmap v5094: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 6 op/s 2024-03-24T02:45:17.953 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:45:17 smithi189 ceph-mon[36187]: pgmap v5094: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 6 op/s 2024-03-24T02:45:17.959 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:45:17 smithi193 ceph-mon[36878]: pgmap v5094: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 6 op/s 2024-03-24T02:45:19.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:45:19 smithi118 ceph-mon[31535]: pgmap v5095: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 5 op/s 2024-03-24T02:45:19.953 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:45:19 smithi189 ceph-mon[36187]: pgmap v5095: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 5 op/s 2024-03-24T02:45:19.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:45:19 smithi193 ceph-mon[36878]: pgmap v5095: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 5 op/s 2024-03-24T02:45:21.945 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:45:21 smithi118 ceph-mon[31535]: pgmap v5096: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-03-24T02:45:21.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:45:21 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 8416.735004 secs 2024-03-24T02:45:21.953 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:45:21 smithi189 ceph-mon[36187]: pgmap v5096: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-03-24T02:45:21.953 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:45:21 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 8416.735004 secs 2024-03-24T02:45:21.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:45:21 smithi193 ceph-mon[36878]: pgmap v5096: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-03-24T02:45:21.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:45:21 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 8416.735004 secs 2024-03-24T02:45:22.945 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:45:22 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T02:45:22.953 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:45:22 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T02:45:22.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:45:22 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T02:45:23.945 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:45:23 smithi118 ceph-mon[31535]: pgmap v5097: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-03-24T02:45:23.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:45:23 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T02:45:23.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:45:23 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T02:45:23.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:45:23 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T02:45:23.953 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:45:23 smithi189 ceph-mon[36187]: pgmap v5097: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-03-24T02:45:23.953 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:45:23 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T02:45:23.953 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:45:23 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T02:45:23.953 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:45:23 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T02:45:23.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:45:23 smithi193 ceph-mon[36878]: pgmap v5097: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-03-24T02:45:23.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:45:23 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T02:45:23.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:45:23 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T02:45:23.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:45:23 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T02:45:24.945 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:45:24 smithi118 ceph-mon[31535]: pgmap v5098: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 6 op/s 2024-03-24T02:45:24.945 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:45:24 smithi118 ceph-mon[31535]: pgmap v5099: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 6 op/s 2024-03-24T02:45:24.953 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:45:24 smithi189 ceph-mon[36187]: pgmap v5098: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 6 op/s 2024-03-24T02:45:24.953 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:45:24 smithi189 ceph-mon[36187]: pgmap v5099: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 6 op/s 2024-03-24T02:45:24.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:45:24 smithi193 ceph-mon[36878]: pgmap v5098: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 6 op/s 2024-03-24T02:45:24.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:45:24 smithi193 ceph-mon[36878]: pgmap v5099: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 6 op/s 2024-03-24T02:45:25.953 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:45:25 smithi189 ceph-mon[36187]: pgmap v5100: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 6 op/s 2024-03-24T02:45:25.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:45:25 smithi193 ceph-mon[36878]: pgmap v5100: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 6 op/s 2024-03-24T02:45:26.195 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:45:25 smithi118 ceph-mon[31535]: pgmap v5100: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 6 op/s 2024-03-24T02:45:26.953 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:45:26 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 8421.735093 secs 2024-03-24T02:45:26.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:45:26 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 8421.735093 secs 2024-03-24T02:45:27.191 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:45:26 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 8421.735093 secs 2024-03-24T02:45:28.195 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:45:27 smithi118 ceph-mon[31535]: pgmap v5101: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 5 op/s 2024-03-24T02:45:28.203 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:45:27 smithi189 ceph-mon[36187]: pgmap v5101: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 5 op/s 2024-03-24T02:45:28.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:45:27 smithi193 ceph-mon[36878]: pgmap v5101: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 5 op/s 2024-03-24T02:45:30.695 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:45:30 smithi118 ceph-mon[31535]: pgmap v5102: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 6 op/s 2024-03-24T02:45:30.703 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:45:30 smithi189 ceph-mon[36187]: pgmap v5102: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 6 op/s 2024-03-24T02:45:30.710 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:45:30 smithi193 ceph-mon[36878]: pgmap v5102: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 6 op/s 2024-03-24T02:45:31.703 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:45:31 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 8426.735153 secs 2024-03-24T02:45:31.710 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:45:31 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 8426.735153 secs 2024-03-24T02:45:31.945 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:45:31 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 8426.735153 secs 2024-03-24T02:45:32.096 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T02:45:32.096+0000 7f81ee000640 1 -- 172.21.15.118:0/1693767697 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- auth(proto 2 132 bytes epoch 0) v1 -- 0x7f81c80036c0 con 0x7f81f81052d0 2024-03-24T02:45:32.096 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T02:45:32.096+0000 7f81db400640 1 -- 172.21.15.118:0/659686424 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- auth(proto 2 132 bytes epoch 0) v1 -- 0x7f81c4006460 con 0x5622651a13f0 2024-03-24T02:45:32.096 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T02:45:32.097+0000 7f81e4c00640 1 -- 172.21.15.118:0/659686424 <== mon.1 v2:172.21.15.193:3300/0 68 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 233+0+0 (secure 0 0 0) 0x7f81c4006460 con 0x5622651a13f0 2024-03-24T02:45:32.096 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T02:45:32.097+0000 7f81ef400640 1 -- 172.21.15.118:0/1693767697 <== mon.2 v2:172.21.15.189:3300/0 68 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 233+0+0 (secure 0 0 0) 0x7f81c80036c0 con 0x7f81f81052d0 2024-03-24T02:45:32.703 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:45:32 smithi189 ceph-mon[36187]: pgmap v5103: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 6 op/s 2024-03-24T02:45:32.710 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:45:32 smithi193 ceph-mon[36878]: pgmap v5103: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 6 op/s 2024-03-24T02:45:32.945 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:45:32 smithi118 ceph-mon[31535]: pgmap v5103: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 6 op/s 2024-03-24T02:45:34.710 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:45:34 smithi193 ceph-mon[36878]: pgmap v5104: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 5 op/s 2024-03-24T02:45:34.945 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:45:34 smithi118 ceph-mon[31535]: pgmap v5104: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 5 op/s 2024-03-24T02:45:34.953 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:45:34 smithi189 ceph-mon[36187]: pgmap v5104: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 5 op/s 2024-03-24T02:45:36.748 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:45:36 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 8431.735214 secs 2024-03-24T02:45:36.748 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:45:36 smithi118 ceph-mon[31535]: pgmap v5105: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T02:45:36.953 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:45:36 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 8431.735214 secs 2024-03-24T02:45:36.953 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:45:36 smithi189 ceph-mon[36187]: pgmap v5105: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T02:45:36.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:45:36 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 8431.735214 secs 2024-03-24T02:45:36.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:45:36 smithi193 ceph-mon[36878]: pgmap v5105: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T02:45:38.945 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:45:38 smithi118 ceph-mon[31535]: pgmap v5106: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:45:38.953 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:45:38 smithi189 ceph-mon[36187]: pgmap v5106: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:45:38.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:45:38 smithi193 ceph-mon[36878]: pgmap v5106: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:45:40.945 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:45:40 smithi118 ceph-mon[31535]: pgmap v5107: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-24T02:45:40.953 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:45:40 smithi189 ceph-mon[36187]: pgmap v5107: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-24T02:45:40.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:45:40 smithi193 ceph-mon[36878]: pgmap v5107: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-24T02:45:41.945 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:45:41 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 8436.735276 secs 2024-03-24T02:45:41.953 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:45:41 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 8436.735276 secs 2024-03-24T02:45:41.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:45:41 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 8436.735276 secs 2024-03-24T02:45:42.945 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:45:42 smithi118 ceph-mon[31535]: pgmap v5108: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:45:42.953 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:45:42 smithi189 ceph-mon[36187]: pgmap v5108: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:45:42.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:45:42 smithi193 ceph-mon[36878]: pgmap v5108: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:45:44.945 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:45:44 smithi118 ceph-mon[31535]: pgmap v5109: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T02:45:44.953 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:45:44 smithi189 ceph-mon[36187]: pgmap v5109: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T02:45:44.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:45:44 smithi193 ceph-mon[36878]: pgmap v5109: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T02:45:46.945 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:45:46 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 8441.735364 secs 2024-03-24T02:45:46.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:45:46 smithi118 ceph-mon[31535]: pgmap v5110: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T02:45:46.953 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:45:46 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 8441.735364 secs 2024-03-24T02:45:46.953 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:45:46 smithi189 ceph-mon[36187]: pgmap v5110: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T02:45:46.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:45:46 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 8441.735364 secs 2024-03-24T02:45:46.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:45:46 smithi193 ceph-mon[36878]: pgmap v5110: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T02:45:48.754 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:45:48 smithi189 ceph-mon[36187]: pgmap v5111: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:45:48.762 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:45:48 smithi193 ceph-mon[36878]: pgmap v5111: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:45:48.945 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:45:48 smithi118 ceph-mon[31535]: pgmap v5111: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:45:50.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:45:50 smithi118 ceph-mon[31535]: pgmap v5112: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-24T02:45:50.953 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:45:50 smithi189 ceph-mon[36187]: pgmap v5112: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-24T02:45:50.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:45:50 smithi193 ceph-mon[36878]: pgmap v5112: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-24T02:45:51.945 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:45:51 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 8446.735421 secs 2024-03-24T02:45:51.953 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:45:51 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 8446.735421 secs 2024-03-24T02:45:51.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:45:51 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 8446.735421 secs 2024-03-24T02:45:52.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:45:52 smithi118 ceph-mon[31535]: pgmap v5113: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:45:52.953 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:45:52 smithi189 ceph-mon[36187]: pgmap v5113: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:45:52.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:45:52 smithi193 ceph-mon[36878]: pgmap v5113: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:45:54.945 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:45:54 smithi118 ceph-mon[31535]: pgmap v5114: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T02:45:54.953 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:45:54 smithi189 ceph-mon[36187]: pgmap v5114: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T02:45:54.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:45:54 smithi193 ceph-mon[36878]: pgmap v5114: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T02:45:56.945 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:45:56 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 8451.735508 secs 2024-03-24T02:45:56.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:45:56 smithi118 ceph-mon[31535]: pgmap v5115: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T02:45:56.953 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:45:56 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 8451.735508 secs 2024-03-24T02:45:56.953 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:45:56 smithi189 ceph-mon[36187]: pgmap v5115: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T02:45:56.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:45:56 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 8451.735508 secs 2024-03-24T02:45:56.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:45:56 smithi193 ceph-mon[36878]: pgmap v5115: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T02:45:58.945 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:45:58 smithi118 ceph-mon[31535]: pgmap v5116: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:45:58.953 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:45:58 smithi189 ceph-mon[36187]: pgmap v5116: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:45:58.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:45:58 smithi193 ceph-mon[36878]: pgmap v5116: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:46:00.945 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:46:00 smithi118 ceph-mon[31535]: pgmap v5117: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-24T02:46:00.953 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:46:00 smithi189 ceph-mon[36187]: pgmap v5117: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-24T02:46:00.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:46:00 smithi193 ceph-mon[36878]: pgmap v5117: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-24T02:46:01.945 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:46:01 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 8456.735549 secs 2024-03-24T02:46:01.953 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:46:01 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 8456.735549 secs 2024-03-24T02:46:01.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:46:01 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 8456.735549 secs 2024-03-24T02:46:02.945 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:46:02 smithi118 ceph-mon[31535]: pgmap v5118: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:46:02.953 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:46:02 smithi189 ceph-mon[36187]: pgmap v5118: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:46:02.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:46:02 smithi193 ceph-mon[36878]: pgmap v5118: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:46:04.945 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:46:04 smithi118 ceph-mon[31535]: pgmap v5119: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T02:46:04.953 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:46:04 smithi189 ceph-mon[36187]: pgmap v5119: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T02:46:04.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:46:04 smithi193 ceph-mon[36878]: pgmap v5119: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T02:46:06.945 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:46:06 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 8461.735632 secs 2024-03-24T02:46:06.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:46:06 smithi118 ceph-mon[31535]: pgmap v5120: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T02:46:06.953 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:46:06 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 8461.735632 secs 2024-03-24T02:46:06.953 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:46:06 smithi189 ceph-mon[36187]: pgmap v5120: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T02:46:06.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:46:06 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 8461.735632 secs 2024-03-24T02:46:06.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:46:06 smithi193 ceph-mon[36878]: pgmap v5120: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T02:46:08.945 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:46:08 smithi118 ceph-mon[31535]: pgmap v5121: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:46:08.953 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:46:08 smithi189 ceph-mon[36187]: pgmap v5121: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:46:08.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:46:08 smithi193 ceph-mon[36878]: pgmap v5121: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:46:10.945 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:46:10 smithi118 ceph-mon[31535]: pgmap v5122: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-24T02:46:10.953 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:46:10 smithi189 ceph-mon[36187]: pgmap v5122: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-24T02:46:10.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:46:10 smithi193 ceph-mon[36878]: pgmap v5122: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-24T02:46:11.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:46:11 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 8466.735717 secs 2024-03-24T02:46:11.953 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:46:11 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 8466.735717 secs 2024-03-24T02:46:11.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:46:11 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 8466.735717 secs 2024-03-24T02:46:12.945 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:46:12 smithi118 ceph-mon[31535]: pgmap v5123: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:46:12.953 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:46:12 smithi189 ceph-mon[36187]: pgmap v5123: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:46:12.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:46:12 smithi193 ceph-mon[36878]: pgmap v5123: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:46:14.945 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:46:14 smithi118 ceph-mon[31535]: pgmap v5124: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T02:46:14.953 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:46:14 smithi189 ceph-mon[36187]: pgmap v5124: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T02:46:14.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:46:14 smithi193 ceph-mon[36878]: pgmap v5124: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T02:46:16.695 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:46:16 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 8471.735803 secs 2024-03-24T02:46:16.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:46:16 smithi118 ceph-mon[31535]: pgmap v5125: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T02:46:16.953 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:46:16 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 8471.735803 secs 2024-03-24T02:46:16.953 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:46:16 smithi189 ceph-mon[36187]: pgmap v5125: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T02:46:16.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:46:16 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 8471.735803 secs 2024-03-24T02:46:16.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:46:16 smithi193 ceph-mon[36878]: pgmap v5125: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T02:46:17.103 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T02:46:17.104+0000 7f81ee000640 1 -- 172.21.15.118:0/1693767697 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f81c80036c0 con 0x7f81f81052d0 2024-03-24T02:46:17.103 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T02:46:17.104+0000 7f81db400640 1 -- 172.21.15.118:0/659686424 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f81c4006460 con 0x5622651a13f0 2024-03-24T02:46:17.104 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T02:46:17.104+0000 7f81ef400640 1 -- 172.21.15.118:0/1693767697 <== mon.2 v2:172.21.15.189:3300/0 69 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (secure 0 0 0) 0x7f81c80036c0 con 0x7f81f81052d0 2024-03-24T02:46:17.104 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T02:46:17.104+0000 7f81e4c00640 1 -- 172.21.15.118:0/659686424 <== mon.1 v2:172.21.15.193:3300/0 69 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 409+0+0 (secure 0 0 0) 0x7f81c4006460 con 0x5622651a13f0 2024-03-24T02:46:18.945 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:46:18 smithi118 ceph-mon[31535]: pgmap v5126: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:46:18.953 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:46:18 smithi189 ceph-mon[36187]: pgmap v5126: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:46:18.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:46:18 smithi193 ceph-mon[36878]: pgmap v5126: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:46:20.945 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:46:20 smithi118 ceph-mon[31535]: pgmap v5127: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-24T02:46:20.953 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:46:20 smithi189 ceph-mon[36187]: pgmap v5127: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-24T02:46:20.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:46:20 smithi193 ceph-mon[36878]: pgmap v5127: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-24T02:46:21.945 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:46:21 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 8476.735865 secs 2024-03-24T02:46:21.953 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:46:21 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 8476.735865 secs 2024-03-24T02:46:21.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:46:21 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 8476.735865 secs 2024-03-24T02:46:22.945 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:46:22 smithi118 ceph-mon[31535]: pgmap v5128: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:46:22.953 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:46:22 smithi189 ceph-mon[36187]: pgmap v5128: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:46:22.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:46:22 smithi193 ceph-mon[36878]: pgmap v5128: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:46:24.945 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:46:24 smithi118 ceph-mon[31535]: pgmap v5129: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T02:46:24.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:46:24 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T02:46:24.953 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:46:24 smithi189 ceph-mon[36187]: pgmap v5129: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T02:46:24.953 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:46:24 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T02:46:24.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:46:24 smithi193 ceph-mon[36878]: pgmap v5129: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T02:46:24.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:46:24 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T02:46:25.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:46:25 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T02:46:25.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:46:25 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T02:46:25.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:46:25 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T02:46:25.953 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:46:25 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T02:46:25.953 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:46:25 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T02:46:25.953 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:46:25 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T02:46:25.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:46:25 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T02:46:25.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:46:25 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T02:46:25.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:46:25 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T02:46:26.945 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:46:26 smithi118 ceph-mon[31535]: pgmap v5130: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 6 op/s 2024-03-24T02:46:26.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:46:26 smithi118 ceph-mon[31535]: pgmap v5131: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 6 op/s 2024-03-24T02:46:26.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:46:26 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 8481.735951 secs 2024-03-24T02:46:26.953 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:46:26 smithi189 ceph-mon[36187]: pgmap v5130: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 6 op/s 2024-03-24T02:46:26.953 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:46:26 smithi189 ceph-mon[36187]: pgmap v5131: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 6 op/s 2024-03-24T02:46:26.953 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:46:26 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 8481.735951 secs 2024-03-24T02:46:26.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:46:26 smithi193 ceph-mon[36878]: pgmap v5130: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 6 op/s 2024-03-24T02:46:26.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:46:26 smithi193 ceph-mon[36878]: pgmap v5131: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 6 op/s 2024-03-24T02:46:26.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:46:26 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 8481.735951 secs 2024-03-24T02:46:28.945 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:46:28 smithi118 ceph-mon[31535]: pgmap v5132: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 6 op/s 2024-03-24T02:46:28.953 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:46:28 smithi189 ceph-mon[36187]: pgmap v5132: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 6 op/s 2024-03-24T02:46:28.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:46:28 smithi193 ceph-mon[36878]: pgmap v5132: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 6 op/s 2024-03-24T02:46:30.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:46:30 smithi118 ceph-mon[31535]: pgmap v5133: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 5 op/s 2024-03-24T02:46:30.953 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:46:30 smithi189 ceph-mon[36187]: pgmap v5133: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 5 op/s 2024-03-24T02:46:30.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:46:30 smithi193 ceph-mon[36878]: pgmap v5133: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 5 op/s 2024-03-24T02:46:31.945 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:46:31 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 8486.736012 secs 2024-03-24T02:46:31.953 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:46:31 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 8486.736012 secs 2024-03-24T02:46:31.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:46:31 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 8486.736012 secs 2024-03-24T02:46:32.945 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:46:32 smithi118 ceph-mon[31535]: pgmap v5134: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 6 op/s 2024-03-24T02:46:32.953 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:46:32 smithi189 ceph-mon[36187]: pgmap v5134: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 6 op/s 2024-03-24T02:46:32.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:46:32 smithi193 ceph-mon[36878]: pgmap v5134: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 6 op/s 2024-03-24T02:46:34.945 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:46:34 smithi118 ceph-mon[31535]: pgmap v5135: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 5 op/s 2024-03-24T02:46:34.953 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:46:34 smithi189 ceph-mon[36187]: pgmap v5135: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 5 op/s 2024-03-24T02:46:34.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:46:34 smithi193 ceph-mon[36878]: pgmap v5135: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 5 op/s 2024-03-24T02:46:36.945 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:46:36 smithi118 ceph-mon[31535]: pgmap v5136: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 5 op/s 2024-03-24T02:46:36.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:46:36 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 8491.736101 secs 2024-03-24T02:46:36.953 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:46:36 smithi189 ceph-mon[36187]: pgmap v5136: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 5 op/s 2024-03-24T02:46:36.953 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:46:36 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 8491.736101 secs 2024-03-24T02:46:36.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:46:36 smithi193 ceph-mon[36878]: pgmap v5136: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 5 op/s 2024-03-24T02:46:36.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:46:36 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 8491.736101 secs 2024-03-24T02:46:38.945 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:46:38 smithi118 ceph-mon[31535]: pgmap v5137: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:46:38.953 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:46:38 smithi189 ceph-mon[36187]: pgmap v5137: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:46:38.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:46:38 smithi193 ceph-mon[36878]: pgmap v5137: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:46:40.195 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:46:39 smithi118 ceph-mon[31535]: pgmap v5138: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T02:46:40.203 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:46:39 smithi189 ceph-mon[36187]: pgmap v5138: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T02:46:40.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:46:39 smithi193 ceph-mon[36878]: pgmap v5138: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T02:46:41.445 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:46:41 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 8496.736140 secs 2024-03-24T02:46:41.453 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:46:41 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 8496.736140 secs 2024-03-24T02:46:41.460 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:46:41 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 8496.736140 secs 2024-03-24T02:46:42.445 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:46:42 smithi118 ceph-mon[31535]: pgmap v5139: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T02:46:42.453 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:46:42 smithi189 ceph-mon[36187]: pgmap v5139: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T02:46:42.460 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:46:42 smithi193 ceph-mon[36878]: pgmap v5139: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T02:46:44.445 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:46:44 smithi118 ceph-mon[31535]: pgmap v5140: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 5 op/s 2024-03-24T02:46:44.453 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:46:44 smithi189 ceph-mon[36187]: pgmap v5140: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 5 op/s 2024-03-24T02:46:44.460 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:46:44 smithi193 ceph-mon[36878]: pgmap v5140: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 5 op/s 2024-03-24T02:46:46.445 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:46:46 smithi118 ceph-mon[31535]: pgmap v5141: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T02:46:46.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:46:46 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 8501.736224 secs 2024-03-24T02:46:46.453 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:46:46 smithi189 ceph-mon[36187]: pgmap v5141: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T02:46:46.453 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:46:46 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 8501.736224 secs 2024-03-24T02:46:46.460 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:46:46 smithi193 ceph-mon[36878]: pgmap v5141: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T02:46:46.460 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:46:46 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 8501.736224 secs 2024-03-24T02:46:48.445 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:46:48 smithi118 ceph-mon[31535]: pgmap v5142: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:46:48.453 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:46:48 smithi189 ceph-mon[36187]: pgmap v5142: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:46:48.460 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:46:48 smithi193 ceph-mon[36878]: pgmap v5142: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:46:50.445 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:46:50 smithi118 ceph-mon[31535]: pgmap v5143: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T02:46:50.453 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:46:50 smithi189 ceph-mon[36187]: pgmap v5143: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T02:46:50.460 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:46:50 smithi193 ceph-mon[36878]: pgmap v5143: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T02:46:51.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:46:51 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 8506.736288 secs 2024-03-24T02:46:51.453 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:46:51 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 8506.736288 secs 2024-03-24T02:46:51.460 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:46:51 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 8506.736288 secs 2024-03-24T02:46:52.445 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:46:52 smithi118 ceph-mon[31535]: pgmap v5144: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T02:46:52.453 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:46:52 smithi189 ceph-mon[36187]: pgmap v5144: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T02:46:52.460 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:46:52 smithi193 ceph-mon[36878]: pgmap v5144: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T02:46:54.445 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:46:54 smithi118 ceph-mon[31535]: pgmap v5145: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T02:46:54.453 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:46:54 smithi189 ceph-mon[36187]: pgmap v5145: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T02:46:54.460 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:46:54 smithi193 ceph-mon[36878]: pgmap v5145: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T02:46:56.445 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:46:56 smithi118 ceph-mon[31535]: pgmap v5146: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T02:46:56.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:46:56 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 8511.736348 secs 2024-03-24T02:46:56.453 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:46:56 smithi189 ceph-mon[36187]: pgmap v5146: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T02:46:56.453 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:46:56 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 8511.736348 secs 2024-03-24T02:46:56.460 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:46:56 smithi193 ceph-mon[36878]: pgmap v5146: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T02:46:56.460 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:46:56 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 8511.736348 secs 2024-03-24T02:46:58.445 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:46:58 smithi118 ceph-mon[31535]: pgmap v5147: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:46:58.453 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:46:58 smithi189 ceph-mon[36187]: pgmap v5147: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:46:58.460 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:46:58 smithi193 ceph-mon[36878]: pgmap v5147: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:47:00.195 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:47:00 smithi118 ceph-mon[31535]: pgmap v5148: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T02:47:00.453 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:47:00 smithi189 ceph-mon[36187]: pgmap v5148: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T02:47:00.460 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:47:00 smithi193 ceph-mon[36878]: pgmap v5148: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T02:47:01.445 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:47:01 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 8516.736405 secs 2024-03-24T02:47:01.453 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:47:01 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 8516.736405 secs 2024-03-24T02:47:01.460 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:47:01 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 8516.736405 secs 2024-03-24T02:47:02.445 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:47:02 smithi118 ceph-mon[31535]: pgmap v5149: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T02:47:02.453 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:47:02 smithi189 ceph-mon[36187]: pgmap v5149: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T02:47:02.460 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:47:02 smithi193 ceph-mon[36878]: pgmap v5149: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T02:47:04.445 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:47:04 smithi118 ceph-mon[31535]: pgmap v5150: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T02:47:04.453 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:47:04 smithi189 ceph-mon[36187]: pgmap v5150: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T02:47:04.460 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:47:04 smithi193 ceph-mon[36878]: pgmap v5150: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T02:47:06.445 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:47:06 smithi118 ceph-mon[31535]: pgmap v5151: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T02:47:06.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:47:06 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 8521.736494 secs 2024-03-24T02:47:06.453 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:47:06 smithi189 ceph-mon[36187]: pgmap v5151: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T02:47:06.453 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:47:06 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 8521.736494 secs 2024-03-24T02:47:06.460 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:47:06 smithi193 ceph-mon[36878]: pgmap v5151: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T02:47:06.460 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:47:06 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 8521.736494 secs 2024-03-24T02:47:08.445 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:47:08 smithi118 ceph-mon[31535]: pgmap v5152: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:47:08.453 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:47:08 smithi189 ceph-mon[36187]: pgmap v5152: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:47:08.460 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:47:08 smithi193 ceph-mon[36878]: pgmap v5152: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:47:10.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:47:10 smithi118 ceph-mon[31535]: pgmap v5153: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T02:47:10.453 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:47:10 smithi189 ceph-mon[36187]: pgmap v5153: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T02:47:10.460 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:47:10 smithi193 ceph-mon[36878]: pgmap v5153: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T02:47:11.445 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:47:11 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 8526.736555 secs 2024-03-24T02:47:11.453 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:47:11 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 8526.736555 secs 2024-03-24T02:47:11.460 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:47:11 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 8526.736555 secs 2024-03-24T02:47:12.445 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:47:12 smithi118 ceph-mon[31535]: pgmap v5154: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T02:47:12.453 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:47:12 smithi189 ceph-mon[36187]: pgmap v5154: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T02:47:12.460 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:47:12 smithi193 ceph-mon[36878]: pgmap v5154: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T02:47:14.445 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:47:14 smithi118 ceph-mon[31535]: pgmap v5155: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T02:47:14.453 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:47:14 smithi189 ceph-mon[36187]: pgmap v5155: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T02:47:14.460 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:47:14 smithi193 ceph-mon[36878]: pgmap v5155: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T02:47:16.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:47:16 smithi118 ceph-mon[31535]: pgmap v5156: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T02:47:16.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:47:16 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 8531.736616 secs 2024-03-24T02:47:16.453 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:47:16 smithi189 ceph-mon[36187]: pgmap v5156: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T02:47:16.453 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:47:16 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 8531.736616 secs 2024-03-24T02:47:16.460 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:47:16 smithi193 ceph-mon[36878]: pgmap v5156: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T02:47:16.460 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:47:16 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 8531.736616 secs 2024-03-24T02:47:18.445 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:47:18 smithi118 ceph-mon[31535]: pgmap v5157: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:47:18.453 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:47:18 smithi189 ceph-mon[36187]: pgmap v5157: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:47:18.460 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:47:18 smithi193 ceph-mon[36878]: pgmap v5157: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:47:20.445 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:47:20 smithi118 ceph-mon[31535]: pgmap v5158: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T02:47:20.453 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:47:20 smithi189 ceph-mon[36187]: pgmap v5158: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T02:47:20.460 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:47:20 smithi193 ceph-mon[36878]: pgmap v5158: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T02:47:21.445 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:47:21 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 8536.736704 secs 2024-03-24T02:47:21.453 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:47:21 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 8536.736704 secs 2024-03-24T02:47:21.460 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:47:21 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 8536.736704 secs 2024-03-24T02:47:22.445 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:47:22 smithi118 ceph-mon[31535]: pgmap v5159: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T02:47:22.453 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:47:22 smithi189 ceph-mon[36187]: pgmap v5159: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T02:47:22.460 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:47:22 smithi193 ceph-mon[36878]: pgmap v5159: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T02:47:24.445 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:47:24 smithi118 ceph-mon[31535]: pgmap v5160: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T02:47:24.453 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:47:24 smithi189 ceph-mon[36187]: pgmap v5160: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T02:47:24.460 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:47:24 smithi193 ceph-mon[36878]: pgmap v5160: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T02:47:26.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:47:26 smithi118 ceph-mon[31535]: pgmap v5161: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T02:47:26.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:47:26 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T02:47:26.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:47:26 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 8541.736774 secs 2024-03-24T02:47:26.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:47:26 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-24T02:47:26.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:47:26 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-24T02:47:26.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:47:26 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-24T02:47:26.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:47:26 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-03-24T02:47:26.453 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:47:26 smithi189 ceph-mon[36187]: pgmap v5161: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T02:47:26.453 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:47:26 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T02:47:26.453 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:47:26 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 8541.736774 secs 2024-03-24T02:47:26.453 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:47:26 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-24T02:47:26.453 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:47:26 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-24T02:47:26.453 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:47:26 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-24T02:47:26.453 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:47:26 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-03-24T02:47:26.460 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:47:26 smithi193 ceph-mon[36878]: pgmap v5161: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T02:47:26.460 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:47:26 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T02:47:26.460 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:47:26 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 8541.736774 secs 2024-03-24T02:47:26.460 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:47:26 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-24T02:47:26.460 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:47:26 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-24T02:47:26.460 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:47:26 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-24T02:47:26.460 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:47:26 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-03-24T02:47:27.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:47:27 smithi118 ceph-mon[31535]: Adjusting osd_memory_target on smithi189 to 174.1M 2024-03-24T02:47:27.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:47:27 smithi118 ceph-mon[31535]: Unable to set osd_memory_target on smithi189 to 182635622: error parsing value: Value '182635622' is below minimum 939524096 2024-03-24T02:47:27.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:47:27 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-24T02:47:27.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:47:27 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-24T02:47:27.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:47:27 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-24T02:47:27.447 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:47:27 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-03-24T02:47:27.447 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:47:27 smithi118 ceph-mon[31535]: Adjusting osd_memory_target on smithi118 to 176.7M 2024-03-24T02:47:27.447 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:47:27 smithi118 ceph-mon[31535]: Unable to set osd_memory_target on smithi118 to 185365196: error parsing value: Value '185365196' is below minimum 939524096 2024-03-24T02:47:27.447 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:47:27 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T02:47:27.447 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:47:27 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T02:47:27.447 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:47:27 smithi118 ceph-mon[31535]: pgmap v5162: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 5 op/s 2024-03-24T02:47:27.447 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:47:27 smithi118 ceph-mon[31535]: pgmap v5163: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 6 op/s 2024-03-24T02:47:27.447 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:47:27 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T02:47:27.453 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:47:27 smithi189 ceph-mon[36187]: Adjusting osd_memory_target on smithi189 to 174.1M 2024-03-24T02:47:27.453 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:47:27 smithi189 ceph-mon[36187]: Unable to set osd_memory_target on smithi189 to 182635622: error parsing value: Value '182635622' is below minimum 939524096 2024-03-24T02:47:27.453 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:47:27 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-24T02:47:27.453 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:47:27 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-24T02:47:27.453 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:47:27 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-24T02:47:27.453 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:47:27 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-03-24T02:47:27.453 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:47:27 smithi189 ceph-mon[36187]: Adjusting osd_memory_target on smithi118 to 176.7M 2024-03-24T02:47:27.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:47:27 smithi189 ceph-mon[36187]: Unable to set osd_memory_target on smithi118 to 185365196: error parsing value: Value '185365196' is below minimum 939524096 2024-03-24T02:47:27.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:47:27 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T02:47:27.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:47:27 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T02:47:27.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:47:27 smithi189 ceph-mon[36187]: pgmap v5162: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 5 op/s 2024-03-24T02:47:27.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:47:27 smithi189 ceph-mon[36187]: pgmap v5163: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 6 op/s 2024-03-24T02:47:27.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:47:27 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T02:47:27.460 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:47:27 smithi193 ceph-mon[36878]: Adjusting osd_memory_target on smithi189 to 174.1M 2024-03-24T02:47:27.460 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:47:27 smithi193 ceph-mon[36878]: Unable to set osd_memory_target on smithi189 to 182635622: error parsing value: Value '182635622' is below minimum 939524096 2024-03-24T02:47:27.460 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:47:27 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-24T02:47:27.460 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:47:27 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-24T02:47:27.460 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:47:27 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-24T02:47:27.460 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:47:27 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-03-24T02:47:27.460 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:47:27 smithi193 ceph-mon[36878]: Adjusting osd_memory_target on smithi118 to 176.7M 2024-03-24T02:47:27.460 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:47:27 smithi193 ceph-mon[36878]: Unable to set osd_memory_target on smithi118 to 185365196: error parsing value: Value '185365196' is below minimum 939524096 2024-03-24T02:47:27.460 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:47:27 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T02:47:27.460 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:47:27 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T02:47:27.460 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:47:27 smithi193 ceph-mon[36878]: pgmap v5162: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 5 op/s 2024-03-24T02:47:27.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:47:27 smithi193 ceph-mon[36878]: pgmap v5163: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 6 op/s 2024-03-24T02:47:27.461 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:47:27 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T02:47:29.945 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:47:29 smithi118 ceph-mon[31535]: pgmap v5164: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 5 op/s 2024-03-24T02:47:29.953 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:47:29 smithi189 ceph-mon[36187]: pgmap v5164: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 5 op/s 2024-03-24T02:47:29.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:47:29 smithi193 ceph-mon[36878]: pgmap v5164: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 5 op/s 2024-03-24T02:47:31.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:47:31 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 8546.736852 secs 2024-03-24T02:47:31.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:47:31 smithi118 ceph-mon[31535]: pgmap v5165: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 6 op/s 2024-03-24T02:47:31.953 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:47:31 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 8546.736852 secs 2024-03-24T02:47:31.953 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:47:31 smithi189 ceph-mon[36187]: pgmap v5165: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 6 op/s 2024-03-24T02:47:31.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:47:31 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 8546.736852 secs 2024-03-24T02:47:31.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:47:31 smithi193 ceph-mon[36878]: pgmap v5165: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 6 op/s 2024-03-24T02:47:33.945 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:47:33 smithi118 ceph-mon[31535]: pgmap v5166: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 5 op/s 2024-03-24T02:47:33.953 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:47:33 smithi189 ceph-mon[36187]: pgmap v5166: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 5 op/s 2024-03-24T02:47:33.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:47:33 smithi193 ceph-mon[36878]: pgmap v5166: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 5 op/s 2024-03-24T02:47:35.945 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:47:35 smithi118 ceph-mon[31535]: pgmap v5167: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 6 op/s 2024-03-24T02:47:35.953 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:47:35 smithi189 ceph-mon[36187]: pgmap v5167: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 6 op/s 2024-03-24T02:47:35.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:47:35 smithi193 ceph-mon[36878]: pgmap v5167: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 6 op/s 2024-03-24T02:47:36.945 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:47:36 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 8551.736911 secs 2024-03-24T02:47:36.953 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:47:36 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 8551.736911 secs 2024-03-24T02:47:36.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:47:36 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 8551.736911 secs 2024-03-24T02:47:37.945 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:47:37 smithi118 ceph-mon[31535]: pgmap v5168: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 5 op/s 2024-03-24T02:47:37.953 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:47:37 smithi189 ceph-mon[36187]: pgmap v5168: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 5 op/s 2024-03-24T02:47:37.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:47:37 smithi193 ceph-mon[36878]: pgmap v5168: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 5 op/s 2024-03-24T02:47:39.945 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:47:39 smithi118 ceph-mon[31535]: pgmap v5169: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T02:47:39.953 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:47:39 smithi189 ceph-mon[36187]: pgmap v5169: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T02:47:39.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:47:39 smithi193 ceph-mon[36878]: pgmap v5169: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T02:47:41.945 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:47:41 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 8556.736992 secs 2024-03-24T02:47:41.945 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:47:41 smithi118 ceph-mon[31535]: pgmap v5170: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T02:47:41.953 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:47:41 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 8556.736992 secs 2024-03-24T02:47:41.953 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:47:41 smithi189 ceph-mon[36187]: pgmap v5170: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T02:47:41.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:47:41 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 8556.736992 secs 2024-03-24T02:47:41.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:47:41 smithi193 ceph-mon[36878]: pgmap v5170: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T02:47:43.945 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:47:43 smithi118 ceph-mon[31535]: pgmap v5171: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T02:47:43.953 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:47:43 smithi189 ceph-mon[36187]: pgmap v5171: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T02:47:43.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:47:43 smithi193 ceph-mon[36878]: pgmap v5171: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T02:47:45.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:47:45 smithi118 ceph-mon[31535]: pgmap v5172: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-03-24T02:47:45.953 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:47:45 smithi189 ceph-mon[36187]: pgmap v5172: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-03-24T02:47:45.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:47:45 smithi193 ceph-mon[36878]: pgmap v5172: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-03-24T02:47:46.945 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:47:46 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 8561.737051 secs 2024-03-24T02:47:46.953 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:47:46 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 8561.737051 secs 2024-03-24T02:47:46.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:47:46 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 8561.737051 secs 2024-03-24T02:47:47.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:47:47 smithi118 ceph-mon[31535]: pgmap v5173: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:47:47.953 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:47:47 smithi189 ceph-mon[36187]: pgmap v5173: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:47:47.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:47:47 smithi193 ceph-mon[36878]: pgmap v5173: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:47:49.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:47:48 smithi118 ceph-mon[31535]: pgmap v5174: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T02:47:49.203 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:47:48 smithi189 ceph-mon[36187]: pgmap v5174: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T02:47:49.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:47:48 smithi193 ceph-mon[36878]: pgmap v5174: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T02:47:51.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:47:51 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 8566.737112 secs 2024-03-24T02:47:51.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:47:51 smithi118 ceph-mon[31535]: pgmap v5175: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T02:47:51.953 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:47:51 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 8566.737112 secs 2024-03-24T02:47:51.953 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:47:51 smithi189 ceph-mon[36187]: pgmap v5175: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T02:47:51.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:47:51 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 8566.737112 secs 2024-03-24T02:47:51.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:47:51 smithi193 ceph-mon[36878]: pgmap v5175: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T02:47:53.945 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:47:53 smithi118 ceph-mon[31535]: pgmap v5176: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T02:47:53.953 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:47:53 smithi189 ceph-mon[36187]: pgmap v5176: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T02:47:53.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:47:53 smithi193 ceph-mon[36878]: pgmap v5176: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T02:47:55.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:47:55 smithi118 ceph-mon[31535]: pgmap v5177: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-03-24T02:47:55.953 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:47:55 smithi189 ceph-mon[36187]: pgmap v5177: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-03-24T02:47:55.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:47:55 smithi193 ceph-mon[36878]: pgmap v5177: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-03-24T02:47:56.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:47:56 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 8571.737194 secs 2024-03-24T02:47:56.953 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:47:56 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 8571.737194 secs 2024-03-24T02:47:56.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:47:56 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 8571.737194 secs 2024-03-24T02:47:57.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:47:57 smithi118 ceph-mon[31535]: pgmap v5178: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:47:57.953 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:47:57 smithi189 ceph-mon[36187]: pgmap v5178: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:47:57.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:47:57 smithi193 ceph-mon[36878]: pgmap v5178: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:47:59.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:47:59 smithi118 ceph-mon[31535]: pgmap v5179: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T02:47:59.953 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:47:59 smithi189 ceph-mon[36187]: pgmap v5179: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T02:47:59.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:47:59 smithi193 ceph-mon[36878]: pgmap v5179: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T02:48:01.945 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:48:01 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 8576.737275 secs 2024-03-24T02:48:01.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:48:01 smithi118 ceph-mon[31535]: pgmap v5180: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T02:48:01.953 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:48:01 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 8576.737275 secs 2024-03-24T02:48:01.953 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:48:01 smithi189 ceph-mon[36187]: pgmap v5180: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T02:48:01.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:48:01 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 8576.737275 secs 2024-03-24T02:48:01.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:48:01 smithi193 ceph-mon[36878]: pgmap v5180: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T02:48:03.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:48:03 smithi118 ceph-mon[31535]: pgmap v5181: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T02:48:03.953 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:48:03 smithi189 ceph-mon[36187]: pgmap v5181: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T02:48:03.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:48:03 smithi193 ceph-mon[36878]: pgmap v5181: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T02:48:05.945 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:48:05 smithi118 ceph-mon[31535]: pgmap v5182: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-03-24T02:48:05.953 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:48:05 smithi189 ceph-mon[36187]: pgmap v5182: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-03-24T02:48:05.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:48:05 smithi193 ceph-mon[36878]: pgmap v5182: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-03-24T02:48:06.945 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:48:06 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 8581.737356 secs 2024-03-24T02:48:06.953 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:48:06 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 8581.737356 secs 2024-03-24T02:48:06.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:48:06 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 8581.737356 secs 2024-03-24T02:48:07.945 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:48:07 smithi118 ceph-mon[31535]: pgmap v5183: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:48:07.953 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:48:07 smithi189 ceph-mon[36187]: pgmap v5183: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:48:07.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:48:07 smithi193 ceph-mon[36878]: pgmap v5183: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:48:08.945 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:48:08 smithi118 ceph-mon[31535]: pgmap v5184: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T02:48:08.953 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:48:08 smithi189 ceph-mon[36187]: pgmap v5184: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T02:48:08.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:48:08 smithi193 ceph-mon[36878]: pgmap v5184: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T02:48:11.945 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:48:11 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 8586.737439 secs 2024-03-24T02:48:11.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:48:11 smithi118 ceph-mon[31535]: pgmap v5185: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T02:48:11.953 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:48:11 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 8586.737439 secs 2024-03-24T02:48:11.953 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:48:11 smithi189 ceph-mon[36187]: pgmap v5185: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T02:48:11.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:48:11 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 8586.737439 secs 2024-03-24T02:48:11.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:48:11 smithi193 ceph-mon[36878]: pgmap v5185: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T02:48:13.945 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:48:13 smithi118 ceph-mon[31535]: pgmap v5186: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T02:48:13.953 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:48:13 smithi189 ceph-mon[36187]: pgmap v5186: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T02:48:13.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:48:13 smithi193 ceph-mon[36878]: pgmap v5186: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T02:48:15.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:48:15 smithi118 ceph-mon[31535]: pgmap v5187: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T02:48:15.953 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:48:15 smithi189 ceph-mon[36187]: pgmap v5187: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T02:48:15.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:48:15 smithi193 ceph-mon[36878]: pgmap v5187: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T02:48:16.945 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:48:16 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 8591.737473 secs 2024-03-24T02:48:16.953 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:48:16 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 8591.737473 secs 2024-03-24T02:48:16.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:48:16 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 8591.737473 secs 2024-03-24T02:48:17.945 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:48:17 smithi118 ceph-mon[31535]: pgmap v5188: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:48:17.953 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:48:17 smithi189 ceph-mon[36187]: pgmap v5188: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:48:17.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:48:17 smithi193 ceph-mon[36878]: pgmap v5188: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:48:19.945 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:48:19 smithi118 ceph-mon[31535]: pgmap v5189: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T02:48:19.953 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:48:19 smithi189 ceph-mon[36187]: pgmap v5189: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T02:48:19.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:48:19 smithi193 ceph-mon[36878]: pgmap v5189: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T02:48:21.945 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:48:21 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 8596.737520 secs 2024-03-24T02:48:21.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:48:21 smithi118 ceph-mon[31535]: pgmap v5190: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T02:48:21.953 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:48:21 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 8596.737520 secs 2024-03-24T02:48:21.953 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:48:21 smithi189 ceph-mon[36187]: pgmap v5190: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T02:48:21.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:48:21 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 8596.737520 secs 2024-03-24T02:48:21.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:48:21 smithi193 ceph-mon[36878]: pgmap v5190: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T02:48:23.945 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:48:23 smithi118 ceph-mon[31535]: pgmap v5191: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T02:48:23.953 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:48:23 smithi189 ceph-mon[36187]: pgmap v5191: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T02:48:23.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:48:23 smithi193 ceph-mon[36878]: pgmap v5191: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T02:48:25.945 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:48:25 smithi118 ceph-mon[31535]: pgmap v5192: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T02:48:25.953 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:48:25 smithi189 ceph-mon[36187]: pgmap v5192: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T02:48:25.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:48:25 smithi193 ceph-mon[36878]: pgmap v5192: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T02:48:26.945 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:48:26 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 8601.737606 secs 2024-03-24T02:48:26.953 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:48:26 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 8601.737606 secs 2024-03-24T02:48:26.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:48:26 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 8601.737606 secs 2024-03-24T02:48:27.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:48:27 smithi118 ceph-mon[31535]: pgmap v5193: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:48:27.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:48:27 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T02:48:27.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:48:27 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-24T02:48:27.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:48:27 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-24T02:48:27.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:48:27 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-03-24T02:48:27.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:48:27 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-03-24T02:48:27.953 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:48:27 smithi189 ceph-mon[36187]: pgmap v5193: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:48:27.953 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:48:27 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T02:48:27.953 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:48:27 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-24T02:48:27.953 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:48:27 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-24T02:48:27.953 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:48:27 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-03-24T02:48:27.953 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:48:27 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-03-24T02:48:27.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:48:27 smithi193 ceph-mon[36878]: pgmap v5193: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:48:27.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:48:27 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T02:48:27.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:48:27 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-24T02:48:27.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:48:27 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-24T02:48:27.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:48:27 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-03-24T02:48:27.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:48:27 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-03-24T02:48:28.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:48:28 smithi118 ceph-mon[31535]: Adjusting osd_memory_target on smithi193 to 174.1M 2024-03-24T02:48:28.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:48:28 smithi118 ceph-mon[31535]: Unable to set osd_memory_target on smithi193 to 182639923: error parsing value: Value '182639923' is below minimum 939524096 2024-03-24T02:48:28.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:48:28 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T02:48:28.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:48:28 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T02:48:28.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:48:28 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T02:48:28.953 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:48:28 smithi189 ceph-mon[36187]: Adjusting osd_memory_target on smithi193 to 174.1M 2024-03-24T02:48:28.953 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:48:28 smithi189 ceph-mon[36187]: Unable to set osd_memory_target on smithi193 to 182639923: error parsing value: Value '182639923' is below minimum 939524096 2024-03-24T02:48:28.953 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:48:28 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T02:48:28.953 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:48:28 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T02:48:28.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:48:28 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T02:48:28.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:48:28 smithi193 ceph-mon[36878]: Adjusting osd_memory_target on smithi193 to 174.1M 2024-03-24T02:48:28.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:48:28 smithi193 ceph-mon[36878]: Unable to set osd_memory_target on smithi193 to 182639923: error parsing value: Value '182639923' is below minimum 939524096 2024-03-24T02:48:28.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:48:28 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T02:48:28.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:48:28 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T02:48:28.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:48:28 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T02:48:29.945 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:48:29 smithi118 ceph-mon[31535]: pgmap v5194: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 6 op/s 2024-03-24T02:48:29.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:48:29 smithi118 ceph-mon[31535]: pgmap v5195: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-03-24T02:48:29.953 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:48:29 smithi189 ceph-mon[36187]: pgmap v5194: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 6 op/s 2024-03-24T02:48:29.953 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:48:29 smithi189 ceph-mon[36187]: pgmap v5195: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-03-24T02:48:29.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:48:29 smithi193 ceph-mon[36878]: pgmap v5194: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 6 op/s 2024-03-24T02:48:29.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:48:29 smithi193 ceph-mon[36878]: pgmap v5195: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-03-24T02:48:30.953 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:48:30 smithi189 ceph-mon[36187]: pgmap v5196: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-03-24T02:48:30.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:48:30 smithi193 ceph-mon[36878]: pgmap v5196: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-03-24T02:48:31.195 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:48:30 smithi118 ceph-mon[31535]: pgmap v5196: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-03-24T02:48:31.953 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:48:31 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 8606.737693 secs 2024-03-24T02:48:31.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:48:31 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 8606.737693 secs 2024-03-24T02:48:32.195 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:48:31 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 8606.737693 secs 2024-03-24T02:48:33.195 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:48:32 smithi118 ceph-mon[31535]: pgmap v5197: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 5 op/s 2024-03-24T02:48:33.203 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:48:32 smithi189 ceph-mon[36187]: pgmap v5197: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 5 op/s 2024-03-24T02:48:33.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:48:32 smithi193 ceph-mon[36878]: pgmap v5197: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 5 op/s 2024-03-24T02:48:35.445 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:48:35 smithi118 ceph-mon[31535]: pgmap v5198: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-03-24T02:48:35.453 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:48:35 smithi189 ceph-mon[36187]: pgmap v5198: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-03-24T02:48:35.460 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:48:35 smithi193 ceph-mon[36878]: pgmap v5198: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-03-24T02:48:36.445 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:48:36 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 8611.737755 secs 2024-03-24T02:48:36.453 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:48:36 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 8611.737755 secs 2024-03-24T02:48:36.460 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:48:36 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 8611.737755 secs 2024-03-24T02:48:37.445 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:48:37 smithi118 ceph-mon[31535]: pgmap v5199: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-03-24T02:48:37.453 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:48:37 smithi189 ceph-mon[36187]: pgmap v5199: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-03-24T02:48:37.460 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:48:37 smithi193 ceph-mon[36878]: pgmap v5199: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-03-24T02:48:39.445 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:48:39 smithi118 ceph-mon[31535]: pgmap v5200: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 5 op/s 2024-03-24T02:48:39.453 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:48:39 smithi189 ceph-mon[36187]: pgmap v5200: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 5 op/s 2024-03-24T02:48:39.460 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:48:39 smithi193 ceph-mon[36878]: pgmap v5200: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 5 op/s 2024-03-24T02:48:41.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:48:41 smithi118 ceph-mon[31535]: pgmap v5201: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T02:48:41.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:48:41 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 8616.737849 secs 2024-03-24T02:48:41.453 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:48:41 smithi189 ceph-mon[36187]: pgmap v5201: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T02:48:41.453 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:48:41 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 8616.737849 secs 2024-03-24T02:48:41.460 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:48:41 smithi193 ceph-mon[36878]: pgmap v5201: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T02:48:41.460 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:48:41 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 8616.737849 secs 2024-03-24T02:48:43.445 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:48:43 smithi118 ceph-mon[31535]: pgmap v5202: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:48:43.453 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:48:43 smithi189 ceph-mon[36187]: pgmap v5202: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:48:43.460 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:48:43 smithi193 ceph-mon[36878]: pgmap v5202: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:48:45.445 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:48:45 smithi118 ceph-mon[31535]: pgmap v5203: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-24T02:48:45.453 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:48:45 smithi189 ceph-mon[36187]: pgmap v5203: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-24T02:48:45.460 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:48:45 smithi193 ceph-mon[36878]: pgmap v5203: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-24T02:48:46.445 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:48:46 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 8621.737890 secs 2024-03-24T02:48:46.453 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:48:46 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 8621.737890 secs 2024-03-24T02:48:46.460 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:48:46 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 8621.737890 secs 2024-03-24T02:48:47.445 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:48:47 smithi118 ceph-mon[31535]: pgmap v5204: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:48:47.453 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:48:47 smithi189 ceph-mon[36187]: pgmap v5204: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:48:47.460 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:48:47 smithi193 ceph-mon[36878]: pgmap v5204: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:48:49.445 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:48:49 smithi118 ceph-mon[31535]: pgmap v5205: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T02:48:49.453 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:48:49 smithi189 ceph-mon[36187]: pgmap v5205: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T02:48:49.460 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:48:49 smithi193 ceph-mon[36878]: pgmap v5205: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T02:48:51.445 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:48:51 smithi118 ceph-mon[31535]: pgmap v5206: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T02:48:51.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:48:51 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 8626.737931 secs 2024-03-24T02:48:51.453 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:48:51 smithi189 ceph-mon[36187]: pgmap v5206: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T02:48:51.453 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:48:51 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 8626.737931 secs 2024-03-24T02:48:51.460 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:48:51 smithi193 ceph-mon[36878]: pgmap v5206: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T02:48:51.460 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:48:51 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 8626.737931 secs 2024-03-24T02:48:53.445 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:48:53 smithi118 ceph-mon[31535]: pgmap v5207: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:48:53.453 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:48:53 smithi189 ceph-mon[36187]: pgmap v5207: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:48:53.460 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:48:53 smithi193 ceph-mon[36878]: pgmap v5207: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:48:55.453 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:48:55 smithi189 ceph-mon[36187]: pgmap v5208: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-24T02:48:55.460 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:48:55 smithi193 ceph-mon[36878]: pgmap v5208: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-24T02:48:55.695 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:48:55 smithi118 ceph-mon[31535]: pgmap v5208: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-24T02:48:56.695 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:48:56 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 8631.737994 secs 2024-03-24T02:48:56.703 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:48:56 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 8631.737994 secs 2024-03-24T02:48:56.710 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:48:56 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 8631.737994 secs 2024-03-24T02:48:57.460 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:48:57 smithi193 ceph-mon[36878]: pgmap v5209: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:48:57.695 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:48:57 smithi118 ceph-mon[31535]: pgmap v5209: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:48:57.703 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:48:57 smithi189 ceph-mon[36187]: pgmap v5209: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:48:59.695 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:48:59 smithi118 ceph-mon[31535]: pgmap v5210: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T02:48:59.703 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:48:59 smithi189 ceph-mon[36187]: pgmap v5210: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T02:48:59.710 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:48:59 smithi193 ceph-mon[36878]: pgmap v5210: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T02:49:01.695 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:49:01 smithi118 ceph-mon[31535]: pgmap v5211: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T02:49:01.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:49:01 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 8636.738078 secs 2024-03-24T02:49:01.703 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:49:01 smithi189 ceph-mon[36187]: pgmap v5211: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T02:49:01.703 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:49:01 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 8636.738078 secs 2024-03-24T02:49:01.710 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:49:01 smithi193 ceph-mon[36878]: pgmap v5211: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T02:49:01.710 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:49:01 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 8636.738078 secs 2024-03-24T02:49:03.695 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:49:03 smithi118 ceph-mon[31535]: pgmap v5212: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:49:03.703 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:49:03 smithi189 ceph-mon[36187]: pgmap v5212: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:49:03.710 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:49:03 smithi193 ceph-mon[36878]: pgmap v5212: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:49:05.695 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:49:05 smithi118 ceph-mon[31535]: pgmap v5213: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-24T02:49:05.703 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:49:05 smithi189 ceph-mon[36187]: pgmap v5213: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-24T02:49:05.710 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:49:05 smithi193 ceph-mon[36878]: pgmap v5213: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-24T02:49:06.597 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:49:06 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 8641.738095 secs 2024-03-24T02:49:06.703 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:49:06 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 8641.738095 secs 2024-03-24T02:49:06.710 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:49:06 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 8641.738095 secs 2024-03-24T02:49:07.695 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:49:07 smithi118 ceph-mon[31535]: pgmap v5214: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:49:07.703 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:49:07 smithi189 ceph-mon[36187]: pgmap v5214: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:49:07.710 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:49:07 smithi193 ceph-mon[36878]: pgmap v5214: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:49:09.695 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:49:09 smithi118 ceph-mon[31535]: pgmap v5215: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T02:49:09.703 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:49:09 smithi189 ceph-mon[36187]: pgmap v5215: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T02:49:09.710 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:49:09 smithi193 ceph-mon[36878]: pgmap v5215: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T02:49:11.695 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:49:11 smithi118 ceph-mon[31535]: pgmap v5216: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T02:49:11.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:49:11 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 8646.738158 secs 2024-03-24T02:49:11.703 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:49:11 smithi189 ceph-mon[36187]: pgmap v5216: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T02:49:11.703 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:49:11 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 8646.738158 secs 2024-03-24T02:49:11.710 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:49:11 smithi193 ceph-mon[36878]: pgmap v5216: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T02:49:11.710 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:49:11 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 8646.738158 secs 2024-03-24T02:49:13.695 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:49:13 smithi118 ceph-mon[31535]: pgmap v5217: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:49:13.703 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:49:13 smithi189 ceph-mon[36187]: pgmap v5217: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:49:13.710 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:49:13 smithi193 ceph-mon[36878]: pgmap v5217: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:49:15.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:49:15 smithi118 ceph-mon[31535]: pgmap v5218: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-24T02:49:15.703 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:49:15 smithi189 ceph-mon[36187]: pgmap v5218: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-24T02:49:15.710 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:49:15 smithi193 ceph-mon[36878]: pgmap v5218: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-24T02:49:16.695 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:49:16 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 8651.738215 secs 2024-03-24T02:49:16.703 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:49:16 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 8651.738215 secs 2024-03-24T02:49:16.710 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:49:16 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 8651.738215 secs 2024-03-24T02:49:17.134 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T02:49:17.135+0000 7f81ee000640 1 -- 172.21.15.118:0/1693767697 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f81c80036c0 con 0x7f81f81052d0 2024-03-24T02:49:17.134 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T02:49:17.135+0000 7f81db400640 1 -- 172.21.15.118:0/659686424 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f81c4006460 con 0x5622651a13f0 2024-03-24T02:49:17.134 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T02:49:17.135+0000 7f81ef400640 1 -- 172.21.15.118:0/1693767697 <== mon.2 v2:172.21.15.189:3300/0 70 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (secure 0 0 0) 0x7f81c80036c0 con 0x7f81f81052d0 2024-03-24T02:49:17.134 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T02:49:17.135+0000 7f81e4c00640 1 -- 172.21.15.118:0/659686424 <== mon.1 v2:172.21.15.193:3300/0 70 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 409+0+0 (secure 0 0 0) 0x7f81c4006460 con 0x5622651a13f0 2024-03-24T02:49:17.695 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:49:17 smithi118 ceph-mon[31535]: pgmap v5219: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:49:17.703 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:49:17 smithi189 ceph-mon[36187]: pgmap v5219: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:49:17.710 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:49:17 smithi193 ceph-mon[36878]: pgmap v5219: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:49:19.697 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:49:19 smithi118 ceph-mon[31535]: pgmap v5220: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T02:49:19.716 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:49:19 smithi193 ceph-mon[36878]: pgmap v5220: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T02:49:19.716 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:49:19 smithi189 ceph-mon[36187]: pgmap v5220: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T02:49:21.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:49:21 smithi118 ceph-mon[31535]: pgmap v5221: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T02:49:21.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:49:21 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 8656.738301 secs 2024-03-24T02:49:21.703 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:49:21 smithi189 ceph-mon[36187]: pgmap v5221: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T02:49:21.703 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:49:21 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 8656.738301 secs 2024-03-24T02:49:21.710 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:49:21 smithi193 ceph-mon[36878]: pgmap v5221: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T02:49:21.710 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:49:21 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 8656.738301 secs 2024-03-24T02:49:23.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:49:23 smithi118 ceph-mon[31535]: pgmap v5222: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:49:23.703 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:49:23 smithi189 ceph-mon[36187]: pgmap v5222: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:49:23.710 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:49:23 smithi193 ceph-mon[36878]: pgmap v5222: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:49:25.698 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:49:25 smithi118 ceph-mon[31535]: pgmap v5223: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-24T02:49:25.703 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:49:25 smithi189 ceph-mon[36187]: pgmap v5223: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-24T02:49:25.710 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:49:25 smithi193 ceph-mon[36878]: pgmap v5223: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-24T02:49:26.695 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:49:26 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 8661.738396 secs 2024-03-24T02:49:26.703 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:49:26 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 8661.738396 secs 2024-03-24T02:49:26.710 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:49:26 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 8661.738396 secs 2024-03-24T02:49:27.695 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:49:27 smithi118 ceph-mon[31535]: pgmap v5224: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:49:27.703 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:49:27 smithi189 ceph-mon[36187]: pgmap v5224: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:49:27.710 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:49:27 smithi193 ceph-mon[36878]: pgmap v5224: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:49:29.695 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:49:29 smithi118 ceph-mon[31535]: pgmap v5225: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T02:49:29.695 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:49:29 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T02:49:29.703 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:49:29 smithi189 ceph-mon[36187]: pgmap v5225: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T02:49:29.703 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:49:29 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T02:49:29.710 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:49:29 smithi193 ceph-mon[36878]: pgmap v5225: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T02:49:29.710 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:49:29 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T02:49:30.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:49:30 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T02:49:30.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:49:30 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T02:49:30.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:49:30 smithi118 ceph-mon[31535]: pgmap v5226: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 6 op/s 2024-03-24T02:49:30.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:49:30 smithi118 ceph-mon[31535]: pgmap v5227: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 6 op/s 2024-03-24T02:49:30.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:49:30 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T02:49:30.703 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:49:30 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T02:49:30.703 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:49:30 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T02:49:30.703 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:49:30 smithi189 ceph-mon[36187]: pgmap v5226: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 6 op/s 2024-03-24T02:49:30.703 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:49:30 smithi189 ceph-mon[36187]: pgmap v5227: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 6 op/s 2024-03-24T02:49:30.703 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:49:30 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T02:49:30.710 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:49:30 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T02:49:30.710 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:49:30 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T02:49:30.710 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:49:30 smithi193 ceph-mon[36878]: pgmap v5226: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 6 op/s 2024-03-24T02:49:30.710 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:49:30 smithi193 ceph-mon[36878]: pgmap v5227: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 6 op/s 2024-03-24T02:49:30.710 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:49:30 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T02:49:31.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:49:31 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 8666.738480 secs 2024-03-24T02:49:31.703 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:49:31 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 8666.738480 secs 2024-03-24T02:49:31.710 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:49:31 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 8666.738480 secs 2024-03-24T02:49:32.695 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:49:32 smithi118 ceph-mon[31535]: pgmap v5228: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 6 op/s 2024-03-24T02:49:32.703 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:49:32 smithi189 ceph-mon[36187]: pgmap v5228: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 6 op/s 2024-03-24T02:49:32.710 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:49:32 smithi193 ceph-mon[36878]: pgmap v5228: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 6 op/s 2024-03-24T02:49:34.195 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:49:33 smithi118 ceph-mon[31535]: pgmap v5229: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 4 op/s 2024-03-24T02:49:34.203 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:49:33 smithi189 ceph-mon[36187]: pgmap v5229: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 4 op/s 2024-03-24T02:49:34.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:49:33 smithi193 ceph-mon[36878]: pgmap v5229: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 4 op/s 2024-03-24T02:49:36.024 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:49:35 smithi118 ceph-mon[31535]: pgmap v5230: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 6 op/s 2024-03-24T02:49:36.203 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:49:35 smithi189 ceph-mon[36187]: pgmap v5230: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 6 op/s 2024-03-24T02:49:36.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:49:35 smithi193 ceph-mon[36878]: pgmap v5230: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 6 op/s 2024-03-24T02:49:37.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:49:36 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 8671.738541 secs 2024-03-24T02:49:37.203 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:49:36 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 8671.738541 secs 2024-03-24T02:49:37.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:49:36 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 8671.738541 secs 2024-03-24T02:49:38.195 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:49:37 smithi118 ceph-mon[31535]: pgmap v5231: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 5 op/s 2024-03-24T02:49:38.203 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:49:37 smithi189 ceph-mon[36187]: pgmap v5231: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 5 op/s 2024-03-24T02:49:38.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:49:37 smithi193 ceph-mon[36878]: pgmap v5231: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 5 op/s 2024-03-24T02:49:40.195 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:49:39 smithi118 ceph-mon[31535]: pgmap v5232: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 5 op/s 2024-03-24T02:49:40.203 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:49:39 smithi189 ceph-mon[36187]: pgmap v5232: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 5 op/s 2024-03-24T02:49:40.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:49:39 smithi193 ceph-mon[36878]: pgmap v5232: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 5 op/s 2024-03-24T02:49:41.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:49:41 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 8676.738599 secs 2024-03-24T02:49:41.453 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:49:41 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 8676.738599 secs 2024-03-24T02:49:41.460 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:49:41 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 8676.738599 secs 2024-03-24T02:49:42.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:49:42 smithi118 ceph-mon[31535]: pgmap v5233: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:49:42.453 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:49:42 smithi189 ceph-mon[36187]: pgmap v5233: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:49:42.462 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:49:42 smithi193 ceph-mon[36878]: pgmap v5233: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:49:44.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:49:43 smithi118 ceph-mon[31535]: pgmap v5234: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T02:49:44.203 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:49:43 smithi189 ceph-mon[36187]: pgmap v5234: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T02:49:44.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:49:43 smithi193 ceph-mon[36878]: pgmap v5234: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T02:49:46.195 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:49:45 smithi118 ceph-mon[31535]: pgmap v5235: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T02:49:46.203 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:49:45 smithi189 ceph-mon[36187]: pgmap v5235: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T02:49:46.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:49:45 smithi193 ceph-mon[36878]: pgmap v5235: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T02:49:47.195 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:49:46 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 8681.738686 secs 2024-03-24T02:49:47.203 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:49:46 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 8681.738686 secs 2024-03-24T02:49:47.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:49:46 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 8681.738686 secs 2024-03-24T02:49:48.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:49:47 smithi118 ceph-mon[31535]: pgmap v5236: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T02:49:48.203 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:49:47 smithi189 ceph-mon[36187]: pgmap v5236: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T02:49:48.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:49:47 smithi193 ceph-mon[36878]: pgmap v5236: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T02:49:50.005 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:49:49 smithi189 ceph-mon[36187]: pgmap v5237: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-03-24T02:49:50.011 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:49:49 smithi193 ceph-mon[36878]: pgmap v5237: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-03-24T02:49:50.195 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:49:49 smithi118 ceph-mon[31535]: pgmap v5237: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-03-24T02:49:51.195 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:49:51 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 8686.738775 secs 2024-03-24T02:49:51.453 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:49:51 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 8686.738775 secs 2024-03-24T02:49:51.460 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:49:51 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 8686.738775 secs 2024-03-24T02:49:52.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:49:52 smithi118 ceph-mon[31535]: pgmap v5238: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-03-24T02:49:52.453 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:49:52 smithi189 ceph-mon[36187]: pgmap v5238: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-03-24T02:49:52.460 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:49:52 smithi193 ceph-mon[36878]: pgmap v5238: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-03-24T02:49:54.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:49:53 smithi118 ceph-mon[31535]: pgmap v5239: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-03-24T02:49:54.203 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:49:53 smithi189 ceph-mon[36187]: pgmap v5239: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-03-24T02:49:54.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:49:53 smithi193 ceph-mon[36878]: pgmap v5239: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-03-24T02:49:56.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:49:55 smithi118 ceph-mon[31535]: pgmap v5240: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 6 op/s 2024-03-24T02:49:56.203 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:49:55 smithi189 ceph-mon[36187]: pgmap v5240: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 6 op/s 2024-03-24T02:49:56.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:49:55 smithi193 ceph-mon[36878]: pgmap v5240: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 6 op/s 2024-03-24T02:49:57.195 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:49:56 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 8691.738816 secs 2024-03-24T02:49:57.203 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:49:56 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 8691.738816 secs 2024-03-24T02:49:57.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:49:56 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 8691.738816 secs 2024-03-24T02:49:58.195 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:49:57 smithi118 ceph-mon[31535]: pgmap v5241: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 5 op/s 2024-03-24T02:49:58.203 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:49:57 smithi189 ceph-mon[36187]: pgmap v5241: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 5 op/s 2024-03-24T02:49:58.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:49:57 smithi193 ceph-mon[36878]: pgmap v5241: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 5 op/s 2024-03-24T02:50:00.195 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:49:59 smithi118 ceph-mon[31535]: pgmap v5242: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-03-24T02:50:00.203 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:49:59 smithi189 ceph-mon[36187]: pgmap v5242: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-03-24T02:50:00.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:49:59 smithi193 ceph-mon[36878]: pgmap v5242: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-03-24T02:50:01.195 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:50:00 smithi118 ceph-mon[31535]: overall HEALTH_WARN 6 failed cephadm daemon(s); 1 MDSs report slow requests 2024-03-24T02:50:01.203 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:50:00 smithi189 ceph-mon[36187]: overall HEALTH_WARN 6 failed cephadm daemon(s); 1 MDSs report slow requests 2024-03-24T02:50:01.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:50:00 smithi193 ceph-mon[36878]: overall HEALTH_WARN 6 failed cephadm daemon(s); 1 MDSs report slow requests 2024-03-24T02:50:02.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:50:01 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 8696.738906 secs 2024-03-24T02:50:02.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:50:01 smithi118 ceph-mon[31535]: pgmap v5243: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:50:02.203 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:50:01 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 8696.738906 secs 2024-03-24T02:50:02.203 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:50:01 smithi189 ceph-mon[36187]: pgmap v5243: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:50:02.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:50:01 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 8696.738906 secs 2024-03-24T02:50:02.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:50:01 smithi193 ceph-mon[36878]: pgmap v5243: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:50:04.195 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:50:03 smithi118 ceph-mon[31535]: pgmap v5244: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T02:50:04.203 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:50:03 smithi189 ceph-mon[36187]: pgmap v5244: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T02:50:04.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:50:03 smithi193 ceph-mon[36878]: pgmap v5244: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T02:50:06.195 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:50:05 smithi118 ceph-mon[31535]: pgmap v5245: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T02:50:06.203 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:50:05 smithi189 ceph-mon[36187]: pgmap v5245: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T02:50:06.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:50:05 smithi193 ceph-mon[36878]: pgmap v5245: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T02:50:07.195 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:50:06 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 8701.738947 secs 2024-03-24T02:50:07.203 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:50:06 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 8701.738947 secs 2024-03-24T02:50:07.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:50:06 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 8701.738947 secs 2024-03-24T02:50:08.195 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:50:07 smithi118 ceph-mon[31535]: pgmap v5246: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T02:50:08.203 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:50:07 smithi189 ceph-mon[36187]: pgmap v5246: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T02:50:08.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:50:07 smithi193 ceph-mon[36878]: pgmap v5246: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T02:50:10.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:50:09 smithi118 ceph-mon[31535]: pgmap v5247: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-03-24T02:50:10.203 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:50:09 smithi189 ceph-mon[36187]: pgmap v5247: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-03-24T02:50:10.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:50:09 smithi193 ceph-mon[36878]: pgmap v5247: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-03-24T02:50:11.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:50:11 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 8706.739008 secs 2024-03-24T02:50:11.453 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:50:11 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 8706.739008 secs 2024-03-24T02:50:11.460 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:50:11 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 8706.739008 secs 2024-03-24T02:50:12.445 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:50:12 smithi118 ceph-mon[31535]: pgmap v5248: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:50:12.453 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:50:12 smithi189 ceph-mon[36187]: pgmap v5248: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:50:12.460 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:50:12 smithi193 ceph-mon[36878]: pgmap v5248: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:50:14.195 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:50:13 smithi118 ceph-mon[31535]: pgmap v5249: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T02:50:14.203 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:50:13 smithi189 ceph-mon[36187]: pgmap v5249: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T02:50:14.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:50:13 smithi193 ceph-mon[36878]: pgmap v5249: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T02:50:16.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:50:15 smithi118 ceph-mon[31535]: pgmap v5250: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T02:50:16.203 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:50:15 smithi189 ceph-mon[36187]: pgmap v5250: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T02:50:16.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:50:15 smithi193 ceph-mon[36878]: pgmap v5250: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T02:50:17.195 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:50:16 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 8711.739098 secs 2024-03-24T02:50:17.203 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:50:16 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 8711.739098 secs 2024-03-24T02:50:17.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:50:16 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 8711.739098 secs 2024-03-24T02:50:18.195 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:50:17 smithi118 ceph-mon[31535]: pgmap v5251: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T02:50:18.203 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:50:17 smithi189 ceph-mon[36187]: pgmap v5251: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T02:50:18.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:50:17 smithi193 ceph-mon[36878]: pgmap v5251: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T02:50:20.011 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:50:19 smithi193 ceph-mon[36878]: pgmap v5252: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-03-24T02:50:20.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:50:19 smithi118 ceph-mon[31535]: pgmap v5252: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-03-24T02:50:20.203 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:50:19 smithi189 ceph-mon[36187]: pgmap v5252: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-03-24T02:50:21.445 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:50:21 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 8716.739186 secs 2024-03-24T02:50:21.453 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:50:21 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 8716.739186 secs 2024-03-24T02:50:21.460 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:50:21 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 8716.739186 secs 2024-03-24T02:50:22.445 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:50:22 smithi118 ceph-mon[31535]: pgmap v5253: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:50:22.453 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:50:22 smithi189 ceph-mon[36187]: pgmap v5253: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:50:22.460 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:50:22 smithi193 ceph-mon[36878]: pgmap v5253: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:50:24.195 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:50:23 smithi118 ceph-mon[31535]: pgmap v5254: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T02:50:24.203 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:50:23 smithi189 ceph-mon[36187]: pgmap v5254: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T02:50:24.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:50:23 smithi193 ceph-mon[36878]: pgmap v5254: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T02:50:26.195 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:50:25 smithi118 ceph-mon[31535]: pgmap v5255: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T02:50:26.203 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:50:25 smithi189 ceph-mon[36187]: pgmap v5255: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T02:50:26.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:50:25 smithi193 ceph-mon[36878]: pgmap v5255: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T02:50:27.195 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:50:26 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 8721.739244 secs 2024-03-24T02:50:27.203 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:50:26 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 8721.739244 secs 2024-03-24T02:50:27.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:50:26 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 8721.739244 secs 2024-03-24T02:50:28.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:50:27 smithi118 ceph-mon[31535]: pgmap v5256: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T02:50:28.203 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:50:27 smithi189 ceph-mon[36187]: pgmap v5256: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T02:50:28.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:50:27 smithi193 ceph-mon[36878]: pgmap v5256: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T02:50:30.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:50:29 smithi118 ceph-mon[31535]: pgmap v5257: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T02:50:30.203 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:50:29 smithi189 ceph-mon[36187]: pgmap v5257: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T02:50:30.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:50:29 smithi193 ceph-mon[36878]: pgmap v5257: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T02:50:31.195 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:50:30 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T02:50:31.203 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:50:30 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T02:50:31.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:50:30 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T02:50:32.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:50:31 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 8726.739306 secs 2024-03-24T02:50:32.197 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:50:31 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T02:50:32.197 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:50:31 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T02:50:32.197 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:50:31 smithi118 ceph-mon[31535]: pgmap v5258: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 5 op/s 2024-03-24T02:50:32.197 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:50:31 smithi118 ceph-mon[31535]: pgmap v5259: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-03-24T02:50:32.197 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:50:31 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T02:50:32.203 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:50:31 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 8726.739306 secs 2024-03-24T02:50:32.203 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:50:31 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T02:50:32.203 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:50:31 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T02:50:32.203 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:50:31 smithi189 ceph-mon[36187]: pgmap v5258: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 5 op/s 2024-03-24T02:50:32.203 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:50:31 smithi189 ceph-mon[36187]: pgmap v5259: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-03-24T02:50:32.203 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:50:31 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T02:50:32.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:50:31 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 8726.739306 secs 2024-03-24T02:50:32.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:50:31 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T02:50:32.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:50:31 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T02:50:32.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:50:31 smithi193 ceph-mon[36878]: pgmap v5258: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 5 op/s 2024-03-24T02:50:32.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:50:31 smithi193 ceph-mon[36878]: pgmap v5259: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-03-24T02:50:32.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:50:31 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T02:50:34.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:50:34 smithi118 ceph-mon[31535]: pgmap v5260: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-03-24T02:50:34.703 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:50:34 smithi189 ceph-mon[36187]: pgmap v5260: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-03-24T02:50:34.710 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:50:34 smithi193 ceph-mon[36878]: pgmap v5260: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-03-24T02:50:36.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:50:36 smithi118 ceph-mon[31535]: pgmap v5261: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 6 op/s 2024-03-24T02:50:36.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:50:36 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 8731.739322 secs 2024-03-24T02:50:36.703 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:50:36 smithi189 ceph-mon[36187]: pgmap v5261: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 6 op/s 2024-03-24T02:50:36.703 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:50:36 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 8731.739322 secs 2024-03-24T02:50:36.710 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:50:36 smithi193 ceph-mon[36878]: pgmap v5261: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 6 op/s 2024-03-24T02:50:36.710 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:50:36 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 8731.739322 secs 2024-03-24T02:50:38.695 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:50:38 smithi118 ceph-mon[31535]: pgmap v5262: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-03-24T02:50:38.703 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:50:38 smithi189 ceph-mon[36187]: pgmap v5262: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-03-24T02:50:38.710 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:50:38 smithi193 ceph-mon[36878]: pgmap v5262: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-03-24T02:50:40.695 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:50:40 smithi118 ceph-mon[31535]: pgmap v5263: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 6 op/s 2024-03-24T02:50:40.703 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:50:40 smithi189 ceph-mon[36187]: pgmap v5263: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 6 op/s 2024-03-24T02:50:40.710 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:50:40 smithi193 ceph-mon[36878]: pgmap v5263: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 6 op/s 2024-03-24T02:50:41.695 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:50:41 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 8736.739382 secs 2024-03-24T02:50:41.703 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:50:41 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 8736.739382 secs 2024-03-24T02:50:41.710 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:50:41 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 8736.739382 secs 2024-03-24T02:50:42.445 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:50:42 smithi118 ceph-mon[31535]: pgmap v5264: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T02:50:42.703 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:50:42 smithi189 ceph-mon[36187]: pgmap v5264: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T02:50:42.710 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:50:42 smithi193 ceph-mon[36878]: pgmap v5264: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T02:50:44.695 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:50:44 smithi118 ceph-mon[31535]: pgmap v5265: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-03-24T02:50:44.703 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:50:44 smithi189 ceph-mon[36187]: pgmap v5265: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-03-24T02:50:44.710 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:50:44 smithi193 ceph-mon[36878]: pgmap v5265: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-03-24T02:50:46.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:50:46 smithi118 ceph-mon[31535]: pgmap v5266: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 6 op/s 2024-03-24T02:50:46.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:50:46 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 8741.739470 secs 2024-03-24T02:50:46.703 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:50:46 smithi189 ceph-mon[36187]: pgmap v5266: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 6 op/s 2024-03-24T02:50:46.703 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:50:46 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 8741.739470 secs 2024-03-24T02:50:46.710 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:50:46 smithi193 ceph-mon[36878]: pgmap v5266: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 6 op/s 2024-03-24T02:50:46.710 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:50:46 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 8741.739470 secs 2024-03-24T02:50:48.695 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:50:48 smithi118 ceph-mon[31535]: pgmap v5267: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:50:48.703 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:50:48 smithi189 ceph-mon[36187]: pgmap v5267: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:50:48.710 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:50:48 smithi193 ceph-mon[36878]: pgmap v5267: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:50:50.695 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:50:50 smithi118 ceph-mon[31535]: pgmap v5268: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-24T02:50:50.703 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:50:50 smithi189 ceph-mon[36187]: pgmap v5268: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-24T02:50:50.710 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:50:50 smithi193 ceph-mon[36878]: pgmap v5268: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-24T02:50:51.697 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:50:51 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 8746.739507 secs 2024-03-24T02:50:51.703 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:50:51 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 8746.739507 secs 2024-03-24T02:50:51.710 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:50:51 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 8746.739507 secs 2024-03-24T02:50:52.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:50:52 smithi118 ceph-mon[31535]: pgmap v5269: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:50:52.703 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:50:52 smithi189 ceph-mon[36187]: pgmap v5269: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:50:52.710 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:50:52 smithi193 ceph-mon[36878]: pgmap v5269: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:50:54.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:50:54 smithi118 ceph-mon[31535]: pgmap v5270: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T02:50:54.703 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:50:54 smithi189 ceph-mon[36187]: pgmap v5270: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T02:50:54.710 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:50:54 smithi193 ceph-mon[36878]: pgmap v5270: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T02:50:56.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:50:56 smithi118 ceph-mon[31535]: pgmap v5271: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T02:50:56.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:50:56 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 8751.739566 secs 2024-03-24T02:50:56.703 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:50:56 smithi189 ceph-mon[36187]: pgmap v5271: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T02:50:56.703 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:50:56 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 8751.739566 secs 2024-03-24T02:50:56.710 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:50:56 smithi193 ceph-mon[36878]: pgmap v5271: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T02:50:56.710 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:50:56 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 8751.739566 secs 2024-03-24T02:50:58.695 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:50:58 smithi118 ceph-mon[31535]: pgmap v5272: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:50:58.703 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:50:58 smithi189 ceph-mon[36187]: pgmap v5272: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:50:58.710 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:50:58 smithi193 ceph-mon[36878]: pgmap v5272: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:51:00.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:51:00 smithi118 ceph-mon[31535]: pgmap v5273: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-24T02:51:00.703 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:51:00 smithi189 ceph-mon[36187]: pgmap v5273: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-24T02:51:00.710 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:51:00 smithi193 ceph-mon[36878]: pgmap v5273: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-24T02:51:01.695 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:51:01 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 8756.739654 secs 2024-03-24T02:51:01.705 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:51:01 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 8756.739654 secs 2024-03-24T02:51:01.710 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:51:01 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 8756.739654 secs 2024-03-24T02:51:02.695 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:51:02 smithi118 ceph-mon[31535]: pgmap v5274: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:51:02.703 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:51:02 smithi189 ceph-mon[36187]: pgmap v5274: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:51:02.710 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:51:02 smithi193 ceph-mon[36878]: pgmap v5274: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:51:04.695 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:51:04 smithi118 ceph-mon[31535]: pgmap v5275: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T02:51:04.703 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:51:04 smithi189 ceph-mon[36187]: pgmap v5275: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T02:51:04.710 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:51:04 smithi193 ceph-mon[36878]: pgmap v5275: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T02:51:06.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:51:06 smithi118 ceph-mon[31535]: pgmap v5276: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T02:51:06.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:51:06 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 8761.739740 secs 2024-03-24T02:51:06.703 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:51:06 smithi189 ceph-mon[36187]: pgmap v5276: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T02:51:06.703 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:51:06 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 8761.739740 secs 2024-03-24T02:51:06.710 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:51:06 smithi193 ceph-mon[36878]: pgmap v5276: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T02:51:06.710 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:51:06 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 8761.739740 secs 2024-03-24T02:51:08.695 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:51:08 smithi118 ceph-mon[31535]: pgmap v5277: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:51:08.703 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:51:08 smithi189 ceph-mon[36187]: pgmap v5277: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:51:08.710 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:51:08 smithi193 ceph-mon[36878]: pgmap v5277: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:51:10.695 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:51:10 smithi118 ceph-mon[31535]: pgmap v5278: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-24T02:51:10.703 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:51:10 smithi189 ceph-mon[36187]: pgmap v5278: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-24T02:51:10.710 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:51:10 smithi193 ceph-mon[36878]: pgmap v5278: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-24T02:51:11.695 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:51:11 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 8766.739829 secs 2024-03-24T02:51:11.703 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:51:11 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 8766.739829 secs 2024-03-24T02:51:11.710 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:51:11 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 8766.739829 secs 2024-03-24T02:51:12.695 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:51:12 smithi118 ceph-mon[31535]: pgmap v5279: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:51:12.703 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:51:12 smithi189 ceph-mon[36187]: pgmap v5279: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:51:12.710 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:51:12 smithi193 ceph-mon[36878]: pgmap v5279: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:51:14.695 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:51:14 smithi118 ceph-mon[31535]: pgmap v5280: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T02:51:14.703 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:51:14 smithi189 ceph-mon[36187]: pgmap v5280: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T02:51:14.710 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:51:14 smithi193 ceph-mon[36878]: pgmap v5280: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T02:51:16.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:51:16 smithi118 ceph-mon[31535]: pgmap v5281: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T02:51:16.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:51:16 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 8771.739919 secs 2024-03-24T02:51:16.703 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:51:16 smithi189 ceph-mon[36187]: pgmap v5281: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T02:51:16.703 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:51:16 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 8771.739919 secs 2024-03-24T02:51:16.710 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:51:16 smithi193 ceph-mon[36878]: pgmap v5281: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T02:51:16.710 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:51:16 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 8771.739919 secs 2024-03-24T02:51:18.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:51:18 smithi118 ceph-mon[31535]: pgmap v5282: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:51:18.703 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:51:18 smithi189 ceph-mon[36187]: pgmap v5282: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:51:18.710 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:51:18 smithi193 ceph-mon[36878]: pgmap v5282: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:51:20.695 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:51:20 smithi118 ceph-mon[31535]: pgmap v5283: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-24T02:51:20.703 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:51:20 smithi189 ceph-mon[36187]: pgmap v5283: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-24T02:51:20.710 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:51:20 smithi193 ceph-mon[36878]: pgmap v5283: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-24T02:51:21.695 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:51:21 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 8776.740008 secs 2024-03-24T02:51:21.703 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:51:21 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 8776.740008 secs 2024-03-24T02:51:21.710 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:51:21 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 8776.740008 secs 2024-03-24T02:51:22.695 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:51:22 smithi118 ceph-mon[31535]: pgmap v5284: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:51:22.703 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:51:22 smithi189 ceph-mon[36187]: pgmap v5284: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:51:22.710 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:51:22 smithi193 ceph-mon[36878]: pgmap v5284: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:51:24.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:51:24 smithi118 ceph-mon[31535]: pgmap v5285: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T02:51:24.703 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:51:24 smithi189 ceph-mon[36187]: pgmap v5285: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T02:51:24.710 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:51:24 smithi193 ceph-mon[36878]: pgmap v5285: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T02:51:26.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:51:26 smithi118 ceph-mon[31535]: pgmap v5286: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T02:51:26.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:51:26 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 8781.740043 secs 2024-03-24T02:51:26.703 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:51:26 smithi189 ceph-mon[36187]: pgmap v5286: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T02:51:26.703 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:51:26 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 8781.740043 secs 2024-03-24T02:51:26.710 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:51:26 smithi193 ceph-mon[36878]: pgmap v5286: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T02:51:26.710 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:51:26 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 8781.740043 secs 2024-03-24T02:51:28.152 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:51:27 smithi118 ceph-mon[31535]: pgmap v5287: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:51:28.203 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:51:27 smithi189 ceph-mon[36187]: pgmap v5287: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:51:28.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:51:27 smithi193 ceph-mon[36878]: pgmap v5287: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:51:30.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:51:30 smithi118 ceph-mon[31535]: pgmap v5288: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-24T02:51:30.703 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:51:30 smithi189 ceph-mon[36187]: pgmap v5288: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-24T02:51:30.710 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:51:30 smithi193 ceph-mon[36878]: pgmap v5288: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-24T02:51:31.695 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:51:31 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 8786.740110 secs 2024-03-24T02:51:31.703 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:51:31 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 8786.740110 secs 2024-03-24T02:51:31.710 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:51:31 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 8786.740110 secs 2024-03-24T02:51:32.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:51:32 smithi118 ceph-mon[31535]: pgmap v5289: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:51:32.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:51:32 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T02:51:32.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:51:32 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T02:51:32.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:51:32 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T02:51:32.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:51:32 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T02:51:32.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:51:32 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T02:51:32.706 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:51:32 smithi189 ceph-mon[36187]: pgmap v5289: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:51:32.706 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:51:32 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T02:51:32.706 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:51:32 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T02:51:32.706 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:51:32 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T02:51:32.706 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:51:32 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T02:51:32.706 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:51:32 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T02:51:32.710 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:51:32 smithi193 ceph-mon[36878]: pgmap v5289: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:51:32.710 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:51:32 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T02:51:32.710 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:51:32 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T02:51:32.710 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:51:32 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T02:51:32.710 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:51:32 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T02:51:32.710 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:51:32 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T02:51:34.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:51:34 smithi118 ceph-mon[31535]: pgmap v5290: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T02:51:34.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:51:34 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T02:51:34.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:51:34 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T02:51:34.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:51:34 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T02:51:34.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:51:34 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T02:51:34.953 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:51:34 smithi189 ceph-mon[36187]: pgmap v5290: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T02:51:34.953 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:51:34 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T02:51:34.953 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:51:34 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T02:51:34.953 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:51:34 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T02:51:34.953 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:51:34 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T02:51:34.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:51:34 smithi193 ceph-mon[36878]: pgmap v5290: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T02:51:34.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:51:34 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T02:51:34.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:51:34 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T02:51:34.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:51:34 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T02:51:34.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:51:34 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T02:51:36.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:51:36 smithi118 ceph-mon[31535]: pgmap v5291: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T02:51:36.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:51:36 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 8791.740173 secs 2024-03-24T02:51:36.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:51:36 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T02:51:36.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:51:36 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T02:51:36.953 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:51:36 smithi189 ceph-mon[36187]: pgmap v5291: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T02:51:36.953 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:51:36 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 8791.740173 secs 2024-03-24T02:51:36.953 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:51:36 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T02:51:36.953 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:51:36 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T02:51:36.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:51:36 smithi193 ceph-mon[36878]: pgmap v5291: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T02:51:36.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:51:36 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 8791.740173 secs 2024-03-24T02:51:36.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:51:36 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T02:51:36.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:51:36 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T02:51:38.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:51:37 smithi118 ceph-mon[31535]: pgmap v5292: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:51:38.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:51:37 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T02:51:38.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:51:37 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T02:51:38.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:51:37 smithi118 ceph-mon[31535]: pgmap v5293: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T02:51:38.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:51:37 smithi118 ceph-mon[31535]: pgmap v5294: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 3 op/s 2024-03-24T02:51:38.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:51:37 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T02:51:38.203 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:51:37 smithi189 ceph-mon[36187]: pgmap v5292: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:51:38.203 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:51:37 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T02:51:38.203 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:51:37 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T02:51:38.203 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:51:37 smithi189 ceph-mon[36187]: pgmap v5293: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T02:51:38.203 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:51:37 smithi189 ceph-mon[36187]: pgmap v5294: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 3 op/s 2024-03-24T02:51:38.203 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:51:37 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T02:51:38.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:51:37 smithi193 ceph-mon[36878]: pgmap v5292: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:51:38.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:51:37 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T02:51:38.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:51:37 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T02:51:38.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:51:37 smithi193 ceph-mon[36878]: pgmap v5293: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T02:51:38.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:51:37 smithi193 ceph-mon[36878]: pgmap v5294: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 1.8 KiB/s rd, 3 op/s 2024-03-24T02:51:38.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:51:37 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T02:51:40.945 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:51:40 smithi118 ceph-mon[31535]: pgmap v5295: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-03-24T02:51:40.953 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:51:40 smithi189 ceph-mon[36187]: pgmap v5295: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-03-24T02:51:40.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:51:40 smithi193 ceph-mon[36878]: pgmap v5295: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-03-24T02:51:41.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:51:41 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 8796.740186 secs 2024-03-24T02:51:41.953 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:51:41 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 8796.740186 secs 2024-03-24T02:51:41.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:51:41 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 8796.740186 secs 2024-03-24T02:51:42.945 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:51:42 smithi118 ceph-mon[31535]: pgmap v5296: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 5 op/s 2024-03-24T02:51:42.953 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:51:42 smithi189 ceph-mon[36187]: pgmap v5296: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 5 op/s 2024-03-24T02:51:42.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:51:42 smithi193 ceph-mon[36878]: pgmap v5296: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 5 op/s 2024-03-24T02:51:44.945 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:51:44 smithi118 ceph-mon[31535]: pgmap v5297: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 5 op/s 2024-03-24T02:51:44.953 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:51:44 smithi189 ceph-mon[36187]: pgmap v5297: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 5 op/s 2024-03-24T02:51:44.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:51:44 smithi193 ceph-mon[36878]: pgmap v5297: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 5 op/s 2024-03-24T02:51:46.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:51:46 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 8801.740269 secs 2024-03-24T02:51:46.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:51:46 smithi118 ceph-mon[31535]: pgmap v5298: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-03-24T02:51:46.953 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:51:46 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 8801.740269 secs 2024-03-24T02:51:46.953 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:51:46 smithi189 ceph-mon[36187]: pgmap v5298: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-03-24T02:51:46.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:51:46 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 8801.740269 secs 2024-03-24T02:51:46.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:51:46 smithi193 ceph-mon[36878]: pgmap v5298: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-03-24T02:51:48.945 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:51:48 smithi118 ceph-mon[31535]: pgmap v5299: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 5 op/s 2024-03-24T02:51:48.953 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:51:48 smithi189 ceph-mon[36187]: pgmap v5299: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 5 op/s 2024-03-24T02:51:48.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:51:48 smithi193 ceph-mon[36878]: pgmap v5299: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 5 op/s 2024-03-24T02:51:50.945 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:51:50 smithi118 ceph-mon[31535]: pgmap v5300: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-24T02:51:50.953 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:51:50 smithi189 ceph-mon[36187]: pgmap v5300: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-24T02:51:50.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:51:50 smithi193 ceph-mon[36878]: pgmap v5300: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-24T02:51:51.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:51:51 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 8806.740330 secs 2024-03-24T02:51:51.953 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:51:51 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 8806.740330 secs 2024-03-24T02:51:51.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:51:51 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 8806.740330 secs 2024-03-24T02:51:52.945 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:51:52 smithi118 ceph-mon[31535]: pgmap v5301: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:51:52.953 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:51:52 smithi189 ceph-mon[36187]: pgmap v5301: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:51:52.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:51:52 smithi193 ceph-mon[36878]: pgmap v5301: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:51:54.945 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:51:54 smithi118 ceph-mon[31535]: pgmap v5302: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T02:51:54.953 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:51:54 smithi189 ceph-mon[36187]: pgmap v5302: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T02:51:54.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:51:54 smithi193 ceph-mon[36878]: pgmap v5302: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T02:51:56.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:51:56 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 8811.740414 secs 2024-03-24T02:51:56.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:51:56 smithi118 ceph-mon[31535]: pgmap v5303: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T02:51:56.953 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:51:56 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 8811.740414 secs 2024-03-24T02:51:56.953 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:51:56 smithi189 ceph-mon[36187]: pgmap v5303: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T02:51:56.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:51:56 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 8811.740414 secs 2024-03-24T02:51:56.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:51:56 smithi193 ceph-mon[36878]: pgmap v5303: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T02:51:58.873 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:51:58 smithi118 ceph-mon[31535]: pgmap v5304: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:51:58.953 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:51:58 smithi189 ceph-mon[36187]: pgmap v5304: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:51:58.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:51:58 smithi193 ceph-mon[36878]: pgmap v5304: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:52:00.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:52:00 smithi118 ceph-mon[31535]: pgmap v5305: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-24T02:52:00.953 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:52:00 smithi189 ceph-mon[36187]: pgmap v5305: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-24T02:52:00.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:52:00 smithi193 ceph-mon[36878]: pgmap v5305: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-24T02:52:01.945 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:52:01 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 8816.740501 secs 2024-03-24T02:52:01.953 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:52:01 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 8816.740501 secs 2024-03-24T02:52:01.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:52:01 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 8816.740501 secs 2024-03-24T02:52:02.945 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:52:02 smithi118 ceph-mon[31535]: pgmap v5306: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:52:02.953 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:52:02 smithi189 ceph-mon[36187]: pgmap v5306: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:52:02.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:52:02 smithi193 ceph-mon[36878]: pgmap v5306: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:52:04.945 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:52:04 smithi118 ceph-mon[31535]: pgmap v5307: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T02:52:04.953 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:52:04 smithi189 ceph-mon[36187]: pgmap v5307: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T02:52:04.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:52:04 smithi193 ceph-mon[36878]: pgmap v5307: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T02:52:06.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:52:06 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 8821.740540 secs 2024-03-24T02:52:06.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:52:06 smithi118 ceph-mon[31535]: pgmap v5308: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T02:52:06.953 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:52:06 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 8821.740540 secs 2024-03-24T02:52:06.953 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:52:06 smithi189 ceph-mon[36187]: pgmap v5308: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T02:52:06.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:52:06 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 8821.740540 secs 2024-03-24T02:52:06.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:52:06 smithi193 ceph-mon[36878]: pgmap v5308: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T02:52:08.195 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:52:07 smithi118 ceph-mon[31535]: pgmap v5309: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:52:08.203 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:52:07 smithi189 ceph-mon[36187]: pgmap v5309: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:52:08.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:52:07 smithi193 ceph-mon[36878]: pgmap v5309: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:52:10.900 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:52:10 smithi118 ceph-mon[31535]: pgmap v5310: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-24T02:52:10.953 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:52:10 smithi189 ceph-mon[36187]: pgmap v5310: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-24T02:52:10.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:52:10 smithi193 ceph-mon[36878]: pgmap v5310: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-24T02:52:11.945 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:52:11 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 8826.740554 secs 2024-03-24T02:52:11.953 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:52:11 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 8826.740554 secs 2024-03-24T02:52:11.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:52:11 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 8826.740554 secs 2024-03-24T02:52:12.945 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:52:12 smithi118 ceph-mon[31535]: pgmap v5311: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:52:12.953 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:52:12 smithi189 ceph-mon[36187]: pgmap v5311: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:52:12.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:52:12 smithi193 ceph-mon[36878]: pgmap v5311: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:52:14.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:52:14 smithi118 ceph-mon[31535]: pgmap v5312: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T02:52:14.953 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:52:14 smithi189 ceph-mon[36187]: pgmap v5312: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T02:52:14.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:52:14 smithi193 ceph-mon[36878]: pgmap v5312: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T02:52:16.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:52:16 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 8831.740640 secs 2024-03-24T02:52:16.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:52:16 smithi118 ceph-mon[31535]: pgmap v5313: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T02:52:16.953 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:52:16 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 8831.740640 secs 2024-03-24T02:52:16.953 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:52:16 smithi189 ceph-mon[36187]: pgmap v5313: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T02:52:16.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:52:16 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 8831.740640 secs 2024-03-24T02:52:16.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:52:16 smithi193 ceph-mon[36878]: pgmap v5313: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T02:52:17.163 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T02:52:17.162+0000 7f81ee000640 1 -- 172.21.15.118:0/1693767697 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f81c80036c0 con 0x7f81f81052d0 2024-03-24T02:52:17.163 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T02:52:17.163+0000 7f81ef400640 1 -- 172.21.15.118:0/1693767697 <== mon.2 v2:172.21.15.189:3300/0 71 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (secure 0 0 0) 0x7f81c80036c0 con 0x7f81f81052d0 2024-03-24T02:52:17.167 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T02:52:17.166+0000 7f81db400640 1 -- 172.21.15.118:0/659686424 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f81c4006460 con 0x5622651a13f0 2024-03-24T02:52:17.167 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T02:52:17.167+0000 7f81e4c00640 1 -- 172.21.15.118:0/659686424 <== mon.1 v2:172.21.15.193:3300/0 71 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 409+0+0 (secure 0 0 0) 0x7f81c4006460 con 0x5622651a13f0 2024-03-24T02:52:18.945 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:52:18 smithi118 ceph-mon[31535]: pgmap v5314: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:52:18.953 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:52:18 smithi189 ceph-mon[36187]: pgmap v5314: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:52:18.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:52:18 smithi193 ceph-mon[36878]: pgmap v5314: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:52:20.945 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:52:20 smithi118 ceph-mon[31535]: pgmap v5315: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-24T02:52:20.953 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:52:20 smithi189 ceph-mon[36187]: pgmap v5315: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-24T02:52:20.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:52:20 smithi193 ceph-mon[36878]: pgmap v5315: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-24T02:52:21.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:52:21 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 8836.740725 secs 2024-03-24T02:52:21.953 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:52:21 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 8836.740725 secs 2024-03-24T02:52:21.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:52:21 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 8836.740725 secs 2024-03-24T02:52:22.945 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:52:22 smithi118 ceph-mon[31535]: pgmap v5316: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:52:22.953 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:52:22 smithi189 ceph-mon[36187]: pgmap v5316: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:52:22.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:52:22 smithi193 ceph-mon[36878]: pgmap v5316: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:52:24.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:52:24 smithi118 ceph-mon[31535]: pgmap v5317: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T02:52:24.953 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:52:24 smithi189 ceph-mon[36187]: pgmap v5317: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T02:52:24.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:52:24 smithi193 ceph-mon[36878]: pgmap v5317: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T02:52:26.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:52:26 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 8841.740813 secs 2024-03-24T02:52:26.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:52:26 smithi118 ceph-mon[31535]: pgmap v5318: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T02:52:26.953 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:52:26 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 8841.740813 secs 2024-03-24T02:52:26.953 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:52:26 smithi189 ceph-mon[36187]: pgmap v5318: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T02:52:26.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:52:26 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 8841.740813 secs 2024-03-24T02:52:26.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:52:26 smithi193 ceph-mon[36878]: pgmap v5318: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T02:52:28.945 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:52:28 smithi118 ceph-mon[31535]: pgmap v5319: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:52:28.953 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:52:28 smithi189 ceph-mon[36187]: pgmap v5319: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:52:28.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:52:28 smithi193 ceph-mon[36878]: pgmap v5319: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:52:30.945 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:52:30 smithi118 ceph-mon[31535]: pgmap v5320: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-24T02:52:30.953 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:52:30 smithi189 ceph-mon[36187]: pgmap v5320: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-24T02:52:30.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:52:30 smithi193 ceph-mon[36878]: pgmap v5320: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-24T02:52:31.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:52:31 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 8846.740853 secs 2024-03-24T02:52:31.953 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:52:31 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 8846.740853 secs 2024-03-24T02:52:31.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:52:31 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 8846.740853 secs 2024-03-24T02:52:32.945 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:52:32 smithi118 ceph-mon[31535]: pgmap v5321: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:52:32.953 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:52:32 smithi189 ceph-mon[36187]: pgmap v5321: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:52:32.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:52:32 smithi193 ceph-mon[36878]: pgmap v5321: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:52:34.945 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:52:34 smithi118 ceph-mon[31535]: pgmap v5322: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T02:52:34.953 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:52:34 smithi189 ceph-mon[36187]: pgmap v5322: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T02:52:34.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:52:34 smithi193 ceph-mon[36878]: pgmap v5322: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T02:52:36.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:52:36 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 8851.740912 secs 2024-03-24T02:52:36.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:52:36 smithi118 ceph-mon[31535]: pgmap v5323: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T02:52:36.953 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:52:36 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 8851.740912 secs 2024-03-24T02:52:36.953 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:52:36 smithi189 ceph-mon[36187]: pgmap v5323: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T02:52:36.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:52:36 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 8851.740912 secs 2024-03-24T02:52:36.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:52:36 smithi193 ceph-mon[36878]: pgmap v5323: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T02:52:38.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:52:38 smithi118 ceph-mon[31535]: pgmap v5324: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:52:38.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:52:38 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T02:52:38.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:52:38 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T02:52:38.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:52:38 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T02:52:38.953 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:52:38 smithi189 ceph-mon[36187]: pgmap v5324: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:52:38.953 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:52:38 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T02:52:38.953 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:52:38 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T02:52:38.953 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:52:38 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T02:52:38.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:52:38 smithi193 ceph-mon[36878]: pgmap v5324: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:52:38.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:52:38 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T02:52:38.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:52:38 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T02:52:38.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:52:38 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T02:52:39.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:52:39 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T02:52:39.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:52:39 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T02:52:39.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:52:39 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T02:52:39.953 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:52:39 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T02:52:39.953 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:52:39 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T02:52:39.953 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:52:39 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T02:52:39.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:52:39 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T02:52:39.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:52:39 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T02:52:39.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:52:39 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T02:52:40.848 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:52:40 smithi118 ceph-mon[31535]: pgmap v5325: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 7 op/s 2024-03-24T02:52:40.848 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:52:40 smithi118 ceph-mon[31535]: pgmap v5326: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 6 op/s 2024-03-24T02:52:40.953 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:52:40 smithi189 ceph-mon[36187]: pgmap v5325: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 7 op/s 2024-03-24T02:52:40.953 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:52:40 smithi189 ceph-mon[36187]: pgmap v5326: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 6 op/s 2024-03-24T02:52:40.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:52:40 smithi193 ceph-mon[36878]: pgmap v5325: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.9 KiB/s rd, 7 op/s 2024-03-24T02:52:40.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:52:40 smithi193 ceph-mon[36878]: pgmap v5326: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 6 op/s 2024-03-24T02:52:41.880 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:52:41 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 8856.741001 secs 2024-03-24T02:52:41.953 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:52:41 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 8856.741001 secs 2024-03-24T02:52:41.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:52:41 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 8856.741001 secs 2024-03-24T02:52:42.945 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:52:42 smithi118 ceph-mon[31535]: pgmap v5327: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 6 op/s 2024-03-24T02:52:42.953 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:52:42 smithi189 ceph-mon[36187]: pgmap v5327: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 6 op/s 2024-03-24T02:52:42.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:52:42 smithi193 ceph-mon[36878]: pgmap v5327: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 6 op/s 2024-03-24T02:52:44.945 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:52:44 smithi118 ceph-mon[31535]: pgmap v5328: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 6 op/s 2024-03-24T02:52:44.953 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:52:44 smithi189 ceph-mon[36187]: pgmap v5328: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 6 op/s 2024-03-24T02:52:44.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:52:44 smithi193 ceph-mon[36878]: pgmap v5328: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 6 op/s 2024-03-24T02:52:46.945 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:52:46 smithi118 ceph-mon[31535]: pgmap v5329: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 6 op/s 2024-03-24T02:52:46.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:52:46 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 8861.741087 secs 2024-03-24T02:52:46.953 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:52:46 smithi189 ceph-mon[36187]: pgmap v5329: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 6 op/s 2024-03-24T02:52:46.953 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:52:46 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 8861.741087 secs 2024-03-24T02:52:46.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:52:46 smithi193 ceph-mon[36878]: pgmap v5329: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 6 op/s 2024-03-24T02:52:46.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:52:46 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 8861.741087 secs 2024-03-24T02:52:48.945 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:52:48 smithi118 ceph-mon[31535]: pgmap v5330: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 6 op/s 2024-03-24T02:52:48.953 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:52:48 smithi189 ceph-mon[36187]: pgmap v5330: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 6 op/s 2024-03-24T02:52:48.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:52:48 smithi193 ceph-mon[36878]: pgmap v5330: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 6 op/s 2024-03-24T02:52:50.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:52:50 smithi118 ceph-mon[31535]: pgmap v5331: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 5 op/s 2024-03-24T02:52:50.953 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:52:50 smithi189 ceph-mon[36187]: pgmap v5331: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 5 op/s 2024-03-24T02:52:50.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:52:50 smithi193 ceph-mon[36878]: pgmap v5331: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 5 op/s 2024-03-24T02:52:51.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:52:51 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 8866.741144 secs 2024-03-24T02:52:51.953 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:52:51 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 8866.741144 secs 2024-03-24T02:52:51.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:52:51 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 8866.741144 secs 2024-03-24T02:52:52.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:52:52 smithi118 ceph-mon[31535]: pgmap v5332: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:52:52.953 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:52:52 smithi189 ceph-mon[36187]: pgmap v5332: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:52:52.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:52:52 smithi193 ceph-mon[36878]: pgmap v5332: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:52:54.945 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:52:54 smithi118 ceph-mon[31535]: pgmap v5333: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T02:52:54.953 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:52:54 smithi189 ceph-mon[36187]: pgmap v5333: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T02:52:54.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:52:54 smithi193 ceph-mon[36878]: pgmap v5333: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T02:52:56.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:52:56 smithi118 ceph-mon[31535]: pgmap v5334: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T02:52:56.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:52:56 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 8871.741203 secs 2024-03-24T02:52:56.953 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:52:56 smithi189 ceph-mon[36187]: pgmap v5334: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T02:52:56.953 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:52:56 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 8871.741203 secs 2024-03-24T02:52:56.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:52:56 smithi193 ceph-mon[36878]: pgmap v5334: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T02:52:56.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:52:56 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 8871.741203 secs 2024-03-24T02:52:58.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:52:58 smithi118 ceph-mon[31535]: pgmap v5335: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:52:58.953 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:52:58 smithi189 ceph-mon[36187]: pgmap v5335: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:52:58.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:52:58 smithi193 ceph-mon[36878]: pgmap v5335: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:53:00.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:53:00 smithi118 ceph-mon[31535]: pgmap v5336: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-24T02:53:00.953 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:53:00 smithi189 ceph-mon[36187]: pgmap v5336: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-24T02:53:00.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:53:00 smithi193 ceph-mon[36878]: pgmap v5336: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-24T02:53:01.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:53:01 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 8876.741263 secs 2024-03-24T02:53:01.953 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:53:01 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 8876.741263 secs 2024-03-24T02:53:01.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:53:01 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 8876.741263 secs 2024-03-24T02:53:02.945 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:53:02 smithi118 ceph-mon[31535]: pgmap v5337: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:53:02.953 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:53:02 smithi189 ceph-mon[36187]: pgmap v5337: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:53:02.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:53:02 smithi193 ceph-mon[36878]: pgmap v5337: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:53:04.945 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:53:04 smithi118 ceph-mon[31535]: pgmap v5338: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T02:53:04.953 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:53:04 smithi189 ceph-mon[36187]: pgmap v5338: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T02:53:04.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:53:04 smithi193 ceph-mon[36878]: pgmap v5338: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T02:53:06.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:53:06 smithi118 ceph-mon[31535]: pgmap v5339: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T02:53:06.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:53:06 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 8881.741325 secs 2024-03-24T02:53:06.953 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:53:06 smithi189 ceph-mon[36187]: pgmap v5339: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T02:53:06.953 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:53:06 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 8881.741325 secs 2024-03-24T02:53:06.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:53:06 smithi193 ceph-mon[36878]: pgmap v5339: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T02:53:06.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:53:06 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 8881.741325 secs 2024-03-24T02:53:08.945 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:53:08 smithi118 ceph-mon[31535]: pgmap v5340: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:53:08.953 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:53:08 smithi189 ceph-mon[36187]: pgmap v5340: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:53:08.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:53:08 smithi193 ceph-mon[36878]: pgmap v5340: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:53:10.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:53:10 smithi118 ceph-mon[31535]: pgmap v5341: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-24T02:53:10.953 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:53:10 smithi189 ceph-mon[36187]: pgmap v5341: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-24T02:53:10.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:53:10 smithi193 ceph-mon[36878]: pgmap v5341: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-24T02:53:11.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:53:11 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 8886.741387 secs 2024-03-24T02:53:11.953 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:53:11 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 8886.741387 secs 2024-03-24T02:53:11.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:53:11 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 8886.741387 secs 2024-03-24T02:53:12.945 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:53:12 smithi118 ceph-mon[31535]: pgmap v5342: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:53:12.953 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:53:12 smithi189 ceph-mon[36187]: pgmap v5342: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:53:12.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:53:12 smithi193 ceph-mon[36878]: pgmap v5342: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:53:14.953 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:53:14 smithi189 ceph-mon[36187]: pgmap v5343: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T02:53:14.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:53:14 smithi193 ceph-mon[36878]: pgmap v5343: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T02:53:15.195 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:53:14 smithi118 ceph-mon[31535]: pgmap v5343: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T02:53:16.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:53:16 smithi193 ceph-mon[36878]: pgmap v5344: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T02:53:16.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:53:16 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 8891.741447 secs 2024-03-24T02:53:17.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:53:16 smithi118 ceph-mon[31535]: pgmap v5344: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T02:53:17.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:53:16 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 8891.741447 secs 2024-03-24T02:53:17.203 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:53:16 smithi189 ceph-mon[36187]: pgmap v5344: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T02:53:17.203 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:53:16 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 8891.741447 secs 2024-03-24T02:53:18.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:53:18 smithi193 ceph-mon[36878]: pgmap v5345: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:53:19.195 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:53:18 smithi118 ceph-mon[31535]: pgmap v5345: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:53:19.203 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:53:18 smithi189 ceph-mon[36187]: pgmap v5345: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:53:21.005 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:53:20 smithi189 ceph-mon[36187]: pgmap v5346: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-24T02:53:21.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:53:20 smithi118 ceph-mon[31535]: pgmap v5346: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-24T02:53:21.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:53:20 smithi193 ceph-mon[36878]: pgmap v5346: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-24T02:53:22.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:53:21 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 8896.741509 secs 2024-03-24T02:53:22.203 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:53:21 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 8896.741509 secs 2024-03-24T02:53:22.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:53:21 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 8896.741509 secs 2024-03-24T02:53:23.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:53:22 smithi118 ceph-mon[31535]: pgmap v5347: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:53:23.203 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:53:22 smithi189 ceph-mon[36187]: pgmap v5347: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:53:23.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:53:22 smithi193 ceph-mon[36878]: pgmap v5347: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:53:25.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:53:24 smithi118 ceph-mon[31535]: pgmap v5348: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T02:53:25.203 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:53:24 smithi189 ceph-mon[36187]: pgmap v5348: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T02:53:25.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:53:24 smithi193 ceph-mon[36878]: pgmap v5348: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T02:53:27.195 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:53:26 smithi118 ceph-mon[31535]: pgmap v5349: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T02:53:27.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:53:26 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 8901.741598 secs 2024-03-24T02:53:27.203 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:53:26 smithi189 ceph-mon[36187]: pgmap v5349: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T02:53:27.203 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:53:26 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 8901.741598 secs 2024-03-24T02:53:27.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:53:26 smithi193 ceph-mon[36878]: pgmap v5349: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T02:53:27.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:53:26 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 8901.741598 secs 2024-03-24T02:53:28.991 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:53:28 smithi118 ceph-mon[31535]: pgmap v5350: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:53:29.203 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:53:28 smithi189 ceph-mon[36187]: pgmap v5350: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:53:29.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:53:28 smithi193 ceph-mon[36878]: pgmap v5350: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:53:31.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:53:30 smithi118 ceph-mon[31535]: pgmap v5351: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-24T02:53:31.203 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:53:30 smithi189 ceph-mon[36187]: pgmap v5351: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-24T02:53:31.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:53:30 smithi193 ceph-mon[36878]: pgmap v5351: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-24T02:53:32.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:53:31 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 8906.741686 secs 2024-03-24T02:53:32.203 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:53:31 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 8906.741686 secs 2024-03-24T02:53:32.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:53:31 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 8906.741686 secs 2024-03-24T02:53:33.080 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:53:32 smithi118 ceph-mon[31535]: pgmap v5352: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:53:33.203 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:53:32 smithi189 ceph-mon[36187]: pgmap v5352: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:53:33.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:53:32 smithi193 ceph-mon[36878]: pgmap v5352: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:53:35.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:53:34 smithi118 ceph-mon[31535]: pgmap v5353: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T02:53:35.203 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:53:34 smithi189 ceph-mon[36187]: pgmap v5353: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T02:53:35.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:53:34 smithi193 ceph-mon[36878]: pgmap v5353: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T02:53:36.195 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:53:35 smithi118 ceph-mon[31535]: pgmap v5354: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T02:53:36.203 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:53:35 smithi189 ceph-mon[36187]: pgmap v5354: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T02:53:36.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:53:35 smithi193 ceph-mon[36878]: pgmap v5354: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T02:53:37.195 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:53:36 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 8911.741746 secs 2024-03-24T02:53:37.203 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:53:36 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 8911.741746 secs 2024-03-24T02:53:37.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:53:36 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 8911.741746 secs 2024-03-24T02:53:38.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:53:37 smithi118 ceph-mon[31535]: pgmap v5355: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:53:38.203 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:53:37 smithi189 ceph-mon[36187]: pgmap v5355: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:53:38.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:53:37 smithi193 ceph-mon[36878]: pgmap v5355: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:53:40.445 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:53:40 smithi118 ceph-mon[31535]: pgmap v5356: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-24T02:53:40.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:53:40 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T02:53:40.453 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:53:40 smithi189 ceph-mon[36187]: pgmap v5356: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-24T02:53:40.453 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:53:40 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T02:53:40.460 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:53:40 smithi193 ceph-mon[36878]: pgmap v5356: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-24T02:53:40.460 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:53:40 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T02:53:41.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:53:41 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 8916.741809 secs 2024-03-24T02:53:41.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:53:41 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T02:53:41.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:53:41 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T02:53:41.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:53:41 smithi118 ceph-mon[31535]: pgmap v5357: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 5 op/s 2024-03-24T02:53:41.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:53:41 smithi118 ceph-mon[31535]: pgmap v5358: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-03-24T02:53:41.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:53:41 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T02:53:41.453 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:53:41 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 8916.741809 secs 2024-03-24T02:53:41.453 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:53:41 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T02:53:41.453 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:53:41 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T02:53:41.453 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:53:41 smithi189 ceph-mon[36187]: pgmap v5357: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 5 op/s 2024-03-24T02:53:41.453 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:53:41 smithi189 ceph-mon[36187]: pgmap v5358: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-03-24T02:53:41.453 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:53:41 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T02:53:41.460 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:53:41 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 8916.741809 secs 2024-03-24T02:53:41.460 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:53:41 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T02:53:41.460 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:53:41 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T02:53:41.460 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:53:41 smithi193 ceph-mon[36878]: pgmap v5357: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 5 op/s 2024-03-24T02:53:41.460 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:53:41 smithi193 ceph-mon[36878]: pgmap v5358: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-03-24T02:53:41.460 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:53:41 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T02:53:43.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:53:43 smithi118 ceph-mon[31535]: pgmap v5359: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-24T02:53:43.953 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:53:43 smithi189 ceph-mon[36187]: pgmap v5359: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-24T02:53:43.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:53:43 smithi193 ceph-mon[36878]: pgmap v5359: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 4 op/s 2024-03-24T02:53:45.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:53:45 smithi118 ceph-mon[31535]: pgmap v5360: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-03-24T02:53:45.953 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:53:45 smithi189 ceph-mon[36187]: pgmap v5360: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-03-24T02:53:45.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:53:45 smithi193 ceph-mon[36878]: pgmap v5360: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-03-24T02:53:46.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:53:46 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 8921.741894 secs 2024-03-24T02:53:46.953 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:53:46 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 8921.741894 secs 2024-03-24T02:53:46.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:53:46 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 8921.741894 secs 2024-03-24T02:53:47.177 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T02:53:47.177+0000 7f81ee000640 1 -- 172.21.15.118:0/1693767697 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- auth(proto 2 132 bytes epoch 0) v1 -- 0x7f81c80036c0 con 0x7f81f81052d0 2024-03-24T02:53:47.178 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T02:53:47.178+0000 7f81ef400640 1 -- 172.21.15.118:0/1693767697 <== mon.2 v2:172.21.15.189:3300/0 72 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 233+0+0 (secure 0 0 0) 0x7f81c80036c0 con 0x7f81f81052d0 2024-03-24T02:53:47.180 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T02:53:47.181+0000 7f81db400640 1 -- 172.21.15.118:0/659686424 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- auth(proto 2 132 bytes epoch 0) v1 -- 0x7f81c4006460 con 0x5622651a13f0 2024-03-24T02:53:47.181 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T02:53:47.181+0000 7f81e4c00640 1 -- 172.21.15.118:0/659686424 <== mon.1 v2:172.21.15.193:3300/0 72 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 233+0+0 (secure 0 0 0) 0x7f81c4006460 con 0x5622651a13f0 2024-03-24T02:53:47.928 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:53:47 smithi118 ceph-mon[31535]: pgmap v5361: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-03-24T02:53:47.953 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:53:47 smithi189 ceph-mon[36187]: pgmap v5361: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-03-24T02:53:47.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:53:47 smithi193 ceph-mon[36878]: pgmap v5361: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-03-24T02:53:49.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:53:49 smithi118 ceph-mon[31535]: pgmap v5362: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 4 op/s 2024-03-24T02:53:49.953 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:53:49 smithi189 ceph-mon[36187]: pgmap v5362: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 4 op/s 2024-03-24T02:53:49.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:53:49 smithi193 ceph-mon[36878]: pgmap v5362: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 4 op/s 2024-03-24T02:53:51.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:53:51 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 8926.741982 secs 2024-03-24T02:53:51.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:53:51 smithi118 ceph-mon[31535]: pgmap v5363: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 5 op/s 2024-03-24T02:53:51.953 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:53:51 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 8926.741982 secs 2024-03-24T02:53:51.953 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:53:51 smithi189 ceph-mon[36187]: pgmap v5363: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 5 op/s 2024-03-24T02:53:51.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:53:51 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 8926.741982 secs 2024-03-24T02:53:51.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:53:51 smithi193 ceph-mon[36878]: pgmap v5363: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 5 op/s 2024-03-24T02:53:53.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:53:53 smithi118 ceph-mon[31535]: pgmap v5364: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:53:53.953 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:53:53 smithi189 ceph-mon[36187]: pgmap v5364: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:53:53.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:53:53 smithi193 ceph-mon[36878]: pgmap v5364: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:53:55.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:53:55 smithi118 ceph-mon[31535]: pgmap v5365: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-24T02:53:55.953 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:53:55 smithi189 ceph-mon[36187]: pgmap v5365: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-24T02:53:55.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:53:55 smithi193 ceph-mon[36878]: pgmap v5365: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-24T02:53:56.945 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:53:56 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 8931.742068 secs 2024-03-24T02:53:56.953 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:53:56 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 8931.742068 secs 2024-03-24T02:53:56.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:53:56 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 8931.742068 secs 2024-03-24T02:53:57.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:53:57 smithi118 ceph-mon[31535]: pgmap v5366: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-03-24T02:53:57.953 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:53:57 smithi189 ceph-mon[36187]: pgmap v5366: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-03-24T02:53:57.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:53:57 smithi193 ceph-mon[36878]: pgmap v5366: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-03-24T02:53:59.945 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:53:59 smithi118 ceph-mon[31535]: pgmap v5367: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-03-24T02:53:59.953 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:53:59 smithi189 ceph-mon[36187]: pgmap v5367: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-03-24T02:53:59.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:53:59 smithi193 ceph-mon[36878]: pgmap v5367: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-03-24T02:54:01.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:54:01 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 8936.742085 secs 2024-03-24T02:54:01.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:54:01 smithi118 ceph-mon[31535]: pgmap v5368: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 6 op/s 2024-03-24T02:54:01.953 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:54:01 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 8936.742085 secs 2024-03-24T02:54:01.953 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:54:01 smithi189 ceph-mon[36187]: pgmap v5368: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 6 op/s 2024-03-24T02:54:01.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:54:01 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 8936.742085 secs 2024-03-24T02:54:01.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:54:01 smithi193 ceph-mon[36878]: pgmap v5368: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 6 op/s 2024-03-24T02:54:03.953 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:54:03 smithi189 ceph-mon[36187]: pgmap v5369: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 5 op/s 2024-03-24T02:54:03.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:54:03 smithi193 ceph-mon[36878]: pgmap v5369: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 5 op/s 2024-03-24T02:54:04.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:54:03 smithi118 ceph-mon[31535]: pgmap v5369: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 5 op/s 2024-03-24T02:54:05.953 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:54:05 smithi189 ceph-mon[36187]: pgmap v5370: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-24T02:54:05.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:54:05 smithi193 ceph-mon[36878]: pgmap v5370: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-24T02:54:06.195 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:54:05 smithi118 ceph-mon[31535]: pgmap v5370: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-24T02:54:06.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:54:06 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 8941.742172 secs 2024-03-24T02:54:07.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:54:06 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 8941.742172 secs 2024-03-24T02:54:07.203 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:54:06 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 8941.742172 secs 2024-03-24T02:54:08.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:54:07 smithi118 ceph-mon[31535]: pgmap v5371: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:54:08.203 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:54:07 smithi189 ceph-mon[36187]: pgmap v5371: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:54:08.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:54:07 smithi193 ceph-mon[36878]: pgmap v5371: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:54:10.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:54:09 smithi118 ceph-mon[31535]: pgmap v5372: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T02:54:10.203 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:54:09 smithi189 ceph-mon[36187]: pgmap v5372: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T02:54:10.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:54:09 smithi193 ceph-mon[36878]: pgmap v5372: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T02:54:11.999 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:54:11 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 8946.742230 secs 2024-03-24T02:54:11.999 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:54:11 smithi118 ceph-mon[31535]: pgmap v5373: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T02:54:12.203 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:54:11 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 8946.742230 secs 2024-03-24T02:54:12.203 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:54:11 smithi189 ceph-mon[36187]: pgmap v5373: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T02:54:12.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:54:11 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 8946.742230 secs 2024-03-24T02:54:12.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:54:11 smithi193 ceph-mon[36878]: pgmap v5373: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T02:54:14.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:54:13 smithi118 ceph-mon[31535]: pgmap v5374: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T02:54:14.203 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:54:13 smithi189 ceph-mon[36187]: pgmap v5374: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T02:54:14.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:54:13 smithi193 ceph-mon[36878]: pgmap v5374: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T02:54:16.088 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:54:15 smithi118 ceph-mon[31535]: pgmap v5375: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-03-24T02:54:16.203 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:54:15 smithi189 ceph-mon[36187]: pgmap v5375: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-03-24T02:54:16.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:54:15 smithi193 ceph-mon[36878]: pgmap v5375: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-03-24T02:54:17.195 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:54:16 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 8951.742290 secs 2024-03-24T02:54:17.203 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:54:16 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 8951.742290 secs 2024-03-24T02:54:17.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:54:16 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 8951.742290 secs 2024-03-24T02:54:18.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:54:17 smithi118 ceph-mon[31535]: pgmap v5376: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:54:18.203 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:54:17 smithi189 ceph-mon[36187]: pgmap v5376: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:54:18.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:54:17 smithi193 ceph-mon[36878]: pgmap v5376: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:54:20.195 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:54:19 smithi118 ceph-mon[31535]: pgmap v5377: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T02:54:20.203 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:54:19 smithi189 ceph-mon[36187]: pgmap v5377: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T02:54:20.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:54:19 smithi193 ceph-mon[36878]: pgmap v5377: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T02:54:22.005 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:54:21 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 8956.742377 secs 2024-03-24T02:54:22.005 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:54:21 smithi189 ceph-mon[36187]: pgmap v5378: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T02:54:22.012 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:54:21 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 8956.742377 secs 2024-03-24T02:54:22.012 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:54:21 smithi193 ceph-mon[36878]: pgmap v5378: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T02:54:22.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:54:21 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 8956.742377 secs 2024-03-24T02:54:22.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:54:21 smithi118 ceph-mon[31535]: pgmap v5378: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T02:54:24.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:54:23 smithi118 ceph-mon[31535]: pgmap v5379: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T02:54:24.203 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:54:23 smithi189 ceph-mon[36187]: pgmap v5379: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T02:54:24.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:54:23 smithi193 ceph-mon[36878]: pgmap v5379: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T02:54:25.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:54:24 smithi118 ceph-mon[31535]: pgmap v5380: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-03-24T02:54:25.203 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:54:24 smithi189 ceph-mon[36187]: pgmap v5380: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-03-24T02:54:25.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:54:24 smithi193 ceph-mon[36878]: pgmap v5380: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-03-24T02:54:26.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:54:26 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 8961.742468 secs 2024-03-24T02:54:26.453 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:54:26 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 8961.742468 secs 2024-03-24T02:54:26.460 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:54:26 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 8961.742468 secs 2024-03-24T02:54:27.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:54:27 smithi118 ceph-mon[31535]: pgmap v5381: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:54:27.453 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:54:27 smithi189 ceph-mon[36187]: pgmap v5381: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:54:27.460 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:54:27 smithi193 ceph-mon[36878]: pgmap v5381: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:54:29.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:54:29 smithi118 ceph-mon[31535]: pgmap v5382: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T02:54:29.953 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:54:29 smithi189 ceph-mon[36187]: pgmap v5382: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T02:54:29.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:54:29 smithi193 ceph-mon[36878]: pgmap v5382: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T02:54:31.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:54:31 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 8966.742526 secs 2024-03-24T02:54:31.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:54:31 smithi118 ceph-mon[31535]: pgmap v5383: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T02:54:31.953 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:54:31 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 8966.742526 secs 2024-03-24T02:54:31.953 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:54:31 smithi189 ceph-mon[36187]: pgmap v5383: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T02:54:31.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:54:31 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 8966.742526 secs 2024-03-24T02:54:31.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:54:31 smithi193 ceph-mon[36878]: pgmap v5383: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T02:54:33.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:54:33 smithi118 ceph-mon[31535]: pgmap v5384: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T02:54:33.953 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:54:33 smithi189 ceph-mon[36187]: pgmap v5384: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T02:54:33.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:54:33 smithi193 ceph-mon[36878]: pgmap v5384: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T02:54:35.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:54:35 smithi118 ceph-mon[31535]: pgmap v5385: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-03-24T02:54:35.953 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:54:35 smithi189 ceph-mon[36187]: pgmap v5385: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-03-24T02:54:35.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:54:35 smithi193 ceph-mon[36878]: pgmap v5385: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-03-24T02:54:36.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:54:36 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 8971.742615 secs 2024-03-24T02:54:36.953 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:54:36 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 8971.742615 secs 2024-03-24T02:54:36.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:54:36 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 8971.742615 secs 2024-03-24T02:54:37.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:54:37 smithi118 ceph-mon[31535]: pgmap v5386: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:54:37.953 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:54:37 smithi189 ceph-mon[36187]: pgmap v5386: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:54:37.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:54:37 smithi193 ceph-mon[36878]: pgmap v5386: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:54:39.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:54:39 smithi118 ceph-mon[31535]: pgmap v5387: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T02:54:39.953 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:54:39 smithi189 ceph-mon[36187]: pgmap v5387: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T02:54:39.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:54:39 smithi193 ceph-mon[36878]: pgmap v5387: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T02:54:41.945 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:54:41 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 8976.742696 secs 2024-03-24T02:54:41.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:54:41 smithi118 ceph-mon[31535]: pgmap v5388: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T02:54:41.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:54:41 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T02:54:41.953 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:54:41 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 8976.742696 secs 2024-03-24T02:54:41.953 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:54:41 smithi189 ceph-mon[36187]: pgmap v5388: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T02:54:41.953 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:54:41 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T02:54:41.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:54:41 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 8976.742696 secs 2024-03-24T02:54:41.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:54:41 smithi193 ceph-mon[36878]: pgmap v5388: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T02:54:41.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:54:41 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T02:54:42.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:54:42 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T02:54:42.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:54:42 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T02:54:42.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:54:42 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T02:54:43.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:54:42 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T02:54:43.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:54:42 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T02:54:43.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:54:42 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T02:54:43.203 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:54:42 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T02:54:43.203 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:54:42 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T02:54:43.203 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:54:42 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T02:54:43.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:54:43 smithi193 ceph-mon[36878]: pgmap v5389: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 5 op/s 2024-03-24T02:54:43.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:54:43 smithi193 ceph-mon[36878]: pgmap v5390: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 5 op/s 2024-03-24T02:54:44.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:54:43 smithi118 ceph-mon[31535]: pgmap v5389: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 5 op/s 2024-03-24T02:54:44.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:54:43 smithi118 ceph-mon[31535]: pgmap v5390: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 5 op/s 2024-03-24T02:54:44.203 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:54:43 smithi189 ceph-mon[36187]: pgmap v5389: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 5 op/s 2024-03-24T02:54:44.203 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:54:43 smithi189 ceph-mon[36187]: pgmap v5390: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 5 op/s 2024-03-24T02:54:46.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:54:45 smithi118 ceph-mon[31535]: pgmap v5391: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-03-24T02:54:46.203 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:54:45 smithi189 ceph-mon[36187]: pgmap v5391: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-03-24T02:54:46.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:54:45 smithi193 ceph-mon[36878]: pgmap v5391: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-03-24T02:54:47.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:54:46 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 8981.742780 secs 2024-03-24T02:54:47.203 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:54:46 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 8981.742780 secs 2024-03-24T02:54:47.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:54:46 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 8981.742780 secs 2024-03-24T02:54:48.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:54:47 smithi118 ceph-mon[31535]: pgmap v5392: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-03-24T02:54:48.203 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:54:47 smithi189 ceph-mon[36187]: pgmap v5392: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-03-24T02:54:48.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:54:47 smithi193 ceph-mon[36878]: pgmap v5392: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-03-24T02:54:50.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:54:49 smithi118 ceph-mon[31535]: pgmap v5393: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-03-24T02:54:50.203 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:54:49 smithi189 ceph-mon[36187]: pgmap v5393: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-03-24T02:54:50.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:54:49 smithi193 ceph-mon[36878]: pgmap v5393: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-03-24T02:54:52.005 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:54:51 smithi189 ceph-mon[36187]: pgmap v5394: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-03-24T02:54:52.005 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:54:51 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 8986.742867 secs 2024-03-24T02:54:52.011 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:54:51 smithi193 ceph-mon[36878]: pgmap v5394: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-03-24T02:54:52.011 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:54:51 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 8986.742867 secs 2024-03-24T02:54:52.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:54:51 smithi118 ceph-mon[31535]: pgmap v5394: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-03-24T02:54:52.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:54:51 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 8986.742867 secs 2024-03-24T02:54:54.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:54:53 smithi118 ceph-mon[31535]: pgmap v5395: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 5 op/s 2024-03-24T02:54:54.203 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:54:53 smithi189 ceph-mon[36187]: pgmap v5395: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 5 op/s 2024-03-24T02:54:54.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:54:53 smithi193 ceph-mon[36878]: pgmap v5395: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 5 op/s 2024-03-24T02:54:56.195 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:54:55 smithi118 ceph-mon[31535]: pgmap v5396: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-24T02:54:56.203 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:54:55 smithi189 ceph-mon[36187]: pgmap v5396: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-24T02:54:56.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:54:55 smithi193 ceph-mon[36878]: pgmap v5396: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-24T02:54:57.195 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:54:56 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 8991.742950 secs 2024-03-24T02:54:57.203 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:54:56 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 8991.742950 secs 2024-03-24T02:54:57.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:54:56 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 8991.742950 secs 2024-03-24T02:54:58.195 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:54:57 smithi118 ceph-mon[31535]: pgmap v5397: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:54:58.205 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:54:57 smithi189 ceph-mon[36187]: pgmap v5397: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:54:58.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:54:57 smithi193 ceph-mon[36878]: pgmap v5397: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:55:00.195 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:54:59 smithi118 ceph-mon[31535]: pgmap v5398: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T02:55:00.203 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:54:59 smithi189 ceph-mon[36187]: pgmap v5398: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T02:55:00.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:54:59 smithi193 ceph-mon[36878]: pgmap v5398: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T02:55:02.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:55:01 smithi118 ceph-mon[31535]: pgmap v5399: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T02:55:02.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:55:01 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 8996.743063 secs 2024-03-24T02:55:02.203 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:55:01 smithi189 ceph-mon[36187]: pgmap v5399: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T02:55:02.203 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:55:01 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 8996.743063 secs 2024-03-24T02:55:02.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:55:01 smithi193 ceph-mon[36878]: pgmap v5399: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T02:55:02.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:55:01 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 8996.743063 secs 2024-03-24T02:55:04.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:55:03 smithi118 ceph-mon[31535]: pgmap v5400: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:55:04.203 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:55:03 smithi189 ceph-mon[36187]: pgmap v5400: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:55:04.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:55:03 smithi193 ceph-mon[36878]: pgmap v5400: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:55:06.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:55:05 smithi118 ceph-mon[31535]: pgmap v5401: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-24T02:55:06.203 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:55:05 smithi189 ceph-mon[36187]: pgmap v5401: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-24T02:55:06.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:55:05 smithi193 ceph-mon[36878]: pgmap v5401: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-24T02:55:07.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:55:06 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 9001.743097 secs 2024-03-24T02:55:07.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:55:06 smithi118 ceph-mon[31535]: pgmap v5402: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:55:07.203 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:55:06 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 9001.743097 secs 2024-03-24T02:55:07.203 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:55:06 smithi189 ceph-mon[36187]: pgmap v5402: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:55:07.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:55:06 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 9001.743097 secs 2024-03-24T02:55:07.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:55:06 smithi193 ceph-mon[36878]: pgmap v5402: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:55:09.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:55:09 smithi118 ceph-mon[31535]: pgmap v5403: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T02:55:09.453 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:55:09 smithi189 ceph-mon[36187]: pgmap v5403: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T02:55:09.460 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:55:09 smithi193 ceph-mon[36878]: pgmap v5403: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T02:55:11.445 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:55:11 smithi118 ceph-mon[31535]: pgmap v5404: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T02:55:11.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:55:11 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 9006.743111 secs 2024-03-24T02:55:11.453 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:55:11 smithi189 ceph-mon[36187]: pgmap v5404: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T02:55:11.453 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:55:11 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 9006.743111 secs 2024-03-24T02:55:11.460 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:55:11 smithi193 ceph-mon[36878]: pgmap v5404: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T02:55:11.460 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:55:11 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 9006.743111 secs 2024-03-24T02:55:13.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:55:13 smithi118 ceph-mon[31535]: pgmap v5405: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:55:13.453 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:55:13 smithi189 ceph-mon[36187]: pgmap v5405: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:55:13.460 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:55:13 smithi193 ceph-mon[36878]: pgmap v5405: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:55:15.445 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:55:15 smithi118 ceph-mon[31535]: pgmap v5406: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-24T02:55:15.453 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:55:15 smithi189 ceph-mon[36187]: pgmap v5406: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-24T02:55:15.460 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:55:15 smithi193 ceph-mon[36878]: pgmap v5406: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-24T02:55:15.766 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T02:55:15.766+0000 7f81f6000640 1 -- 172.21.15.118:0/1693767697 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7f81dc0a3fb0 msgr2=0x7f81dc0b05c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 12 2024-03-24T02:55:15.766 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T02:55:15.766+0000 7f81f6000640 1 -- 172.21.15.118:0/1693767697 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7f81dc0a3fb0 msgr2=0x7f81dc0b05c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-03-24T02:55:15.766 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T02:55:15.766+0000 7f81f6000640 1 --2- 172.21.15.118:0/1693767697 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7f81dc0a3fb0 0x7f81dc0b05c0 secure :-1 s=READY pgs=586 cs=0 l=1 rev1=1 crypto rx=0x7f81e0001eb0 tx=0x7f81e0047000 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-03-24T02:55:15.766 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T02:55:15.766+0000 7f81f6000640 1 --2- 172.21.15.118:0/1693767697 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7f81dc0a3fb0 0x7f81dc0b05c0 secure :-1 s=READY pgs=586 cs=0 l=1 rev1=1 crypto rx=0x7f81e0001eb0 tx=0x7f81e0047000 comp rx=0 tx=0).stop 2024-03-24T02:55:15.766 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T02:55:15.766+0000 7f81f6000640 1 -- 172.21.15.118:0/1693767697 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7f81dc0a3fb0 msgr2=0x7f81dc0b05c0 unknown :-1 s=STATE_CLOSED l=1).tick see no progress in more than 10000000 us during connecting to v2:172.21.15.118:6800/2112912943, fault. 2024-03-24T02:55:15.766 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T02:55:15.766+0000 7f81ef400640 1 -- 172.21.15.118:0/1693767697 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7f81dc0a3fb0 msgr2=0x7f81dc0b05c0 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-03-24T02:55:15.766 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T02:55:15.766+0000 7f81ef400640 1 --2- 172.21.15.118:0/1693767697 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7f81dc0a3fb0 0x7f81dc0b05c0 unknown :-1 s=CLOSED pgs=586 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T02:55:15.766 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T02:55:15.766+0000 7f81ef400640 1 --2- 172.21.15.118:0/1693767697 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7f81dc0a3bd0 0x7f81dc0b2b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T02:55:15.767 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T02:55:15.767+0000 7f81f6000640 1 --2- 172.21.15.118:0/1693767697 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7f81dc0a3bd0 0x7f81dc0b2b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T02:55:15.767 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T02:55:15.768+0000 7f81f6000640 1 --2- 172.21.15.118:0/1693767697 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7f81dc0a3bd0 0x7f81dc0b2b10 secure :-1 s=READY pgs=606 cs=0 l=1 rev1=1 crypto rx=0x7f81e0010870 tx=0x7f81e0024040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T02:55:15.771 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T02:55:15.772+0000 7f81f6000640 1 -- 172.21.15.118:0/659686424 >> [v2:172.21.15.189:6838/4074484168,v1:172.21.15.189:6839/4074484168] conn(0x5622651b71e0 msgr2=0x562265215110 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).tick idle (900000855) for more than 900000000 us, fault. 2024-03-24T02:55:15.771 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T02:55:15.772+0000 7f81f6000640 1 --2- 172.21.15.118:0/659686424 >> [v2:172.21.15.189:6838/4074484168,v1:172.21.15.189:6839/4074484168] conn(0x5622651b71e0 0x562265215110 unknown :-1 s=READY pgs=561 cs=8 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault initiating reconnect 2024-03-24T02:55:15.771 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T02:55:15.772+0000 7f81f6000640 1 --2- 172.21.15.118:0/659686424 >> [v2:172.21.15.189:6838/4074484168,v1:172.21.15.189:6839/4074484168] conn(0x5622651b71e0 0x562265215110 unknown :-1 s=BANNER_CONNECTING pgs=561 cs=9 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T02:55:15.772 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T02:55:15.773+0000 7f81f6000640 1 --2- 172.21.15.118:0/659686424 >> [v2:172.21.15.189:6838/4074484168,v1:172.21.15.189:6839/4074484168] conn(0x5622651b71e0 0x562265215110 crc :-1 s=READY pgs=561 cs=9 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=76f3d534289740f0 server_cookie=b31a2335a59552de in_seq=0 out_seq=0 2024-03-24T02:55:15.772 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T02:55:15.773+0000 7f81e4c00640 10 client.27094 ms_handle_connect on v2:172.21.15.189:6838/4074484168 2024-03-24T02:55:16.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:55:16 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 9011.743173 secs 2024-03-24T02:55:16.453 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:55:16 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 9011.743173 secs 2024-03-24T02:55:16.460 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:55:16 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 9011.743173 secs 2024-03-24T02:55:17.193 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T02:55:17.193+0000 7f81ee000640 1 -- 172.21.15.118:0/1693767697 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f81c80036c0 con 0x7f81f81052d0 2024-03-24T02:55:17.193 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T02:55:17.194+0000 7f81db400640 1 -- 172.21.15.118:0/659686424 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f81c4006460 con 0x5622651a13f0 2024-03-24T02:55:17.199 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T02:55:17.199+0000 7f81e4c00640 1 -- 172.21.15.118:0/659686424 <== mon.1 v2:172.21.15.193:3300/0 73 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 409+0+0 (secure 0 0 0) 0x7f81c4006460 con 0x5622651a13f0 2024-03-24T02:55:17.199 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T02:55:17.199+0000 7f81ef400640 1 -- 172.21.15.118:0/1693767697 <== mon.2 v2:172.21.15.189:3300/0 73 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (secure 0 0 0) 0x7f81c80036c0 con 0x7f81f81052d0 2024-03-24T02:55:17.445 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:55:17 smithi118 ceph-mon[31535]: pgmap v5407: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:55:17.453 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:55:17 smithi189 ceph-mon[36187]: pgmap v5407: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:55:17.460 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:55:17 smithi193 ceph-mon[36878]: pgmap v5407: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:55:19.445 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:55:19 smithi118 ceph-mon[31535]: pgmap v5408: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T02:55:19.453 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:55:19 smithi189 ceph-mon[36187]: pgmap v5408: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T02:55:19.460 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:55:19 smithi193 ceph-mon[36878]: pgmap v5408: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T02:55:21.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:55:21 smithi118 ceph-mon[31535]: pgmap v5409: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T02:55:21.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:55:21 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 9016.743234 secs 2024-03-24T02:55:21.453 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:55:21 smithi189 ceph-mon[36187]: pgmap v5409: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T02:55:21.453 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:55:21 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 9016.743234 secs 2024-03-24T02:55:21.460 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:55:21 smithi193 ceph-mon[36878]: pgmap v5409: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T02:55:21.460 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:55:21 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 9016.743234 secs 2024-03-24T02:55:23.460 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:55:23 smithi193 ceph-mon[36878]: pgmap v5410: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:55:23.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:55:23 smithi118 ceph-mon[31535]: pgmap v5410: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:55:23.703 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:55:23 smithi189 ceph-mon[36187]: pgmap v5410: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:55:25.460 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:55:25 smithi193 ceph-mon[36878]: pgmap v5411: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-24T02:55:25.695 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:55:25 smithi118 ceph-mon[31535]: pgmap v5411: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-24T02:55:25.703 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:55:25 smithi189 ceph-mon[36187]: pgmap v5411: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-24T02:55:26.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:55:26 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 9021.743297 secs 2024-03-24T02:55:26.703 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:55:26 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 9021.743297 secs 2024-03-24T02:55:26.710 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:55:26 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 9021.743297 secs 2024-03-24T02:55:27.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:55:27 smithi118 ceph-mon[31535]: pgmap v5412: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:55:27.703 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:55:27 smithi189 ceph-mon[36187]: pgmap v5412: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:55:27.710 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:55:27 smithi193 ceph-mon[36878]: pgmap v5412: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:55:29.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:55:29 smithi118 ceph-mon[31535]: pgmap v5413: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T02:55:29.703 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:55:29 smithi189 ceph-mon[36187]: pgmap v5413: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T02:55:29.710 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:55:29 smithi193 ceph-mon[36878]: pgmap v5413: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T02:55:31.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:55:31 smithi118 ceph-mon[31535]: pgmap v5414: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T02:55:31.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:55:31 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 9026.743352 secs 2024-03-24T02:55:31.703 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:55:31 smithi189 ceph-mon[36187]: pgmap v5414: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T02:55:31.703 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:55:31 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 9026.743352 secs 2024-03-24T02:55:31.710 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:55:31 smithi193 ceph-mon[36878]: pgmap v5414: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T02:55:31.710 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:55:31 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 9026.743352 secs 2024-03-24T02:55:33.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:55:33 smithi118 ceph-mon[31535]: pgmap v5415: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:55:33.703 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:55:33 smithi189 ceph-mon[36187]: pgmap v5415: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:55:33.710 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:55:33 smithi193 ceph-mon[36878]: pgmap v5415: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:55:35.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:55:35 smithi118 ceph-mon[31535]: pgmap v5416: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-24T02:55:35.703 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:55:35 smithi189 ceph-mon[36187]: pgmap v5416: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-24T02:55:35.710 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:55:35 smithi193 ceph-mon[36878]: pgmap v5416: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-24T02:55:36.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:55:36 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 9031.743437 secs 2024-03-24T02:55:36.703 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:55:36 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 9031.743437 secs 2024-03-24T02:55:36.710 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:55:36 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 9031.743437 secs 2024-03-24T02:55:37.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:55:37 smithi118 ceph-mon[31535]: pgmap v5417: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:55:37.703 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:55:37 smithi189 ceph-mon[36187]: pgmap v5417: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:55:37.710 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:55:37 smithi193 ceph-mon[36878]: pgmap v5417: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:55:39.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:55:39 smithi118 ceph-mon[31535]: pgmap v5418: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T02:55:39.703 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:55:39 smithi189 ceph-mon[36187]: pgmap v5418: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T02:55:39.710 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:55:39 smithi193 ceph-mon[36878]: pgmap v5418: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T02:55:41.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:55:41 smithi118 ceph-mon[31535]: pgmap v5419: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T02:55:41.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:55:41 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 9036.743532 secs 2024-03-24T02:55:41.703 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:55:41 smithi189 ceph-mon[36187]: pgmap v5419: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T02:55:41.703 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:55:41 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 9036.743532 secs 2024-03-24T02:55:41.710 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:55:41 smithi193 ceph-mon[36878]: pgmap v5419: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T02:55:41.710 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:55:41 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 9036.743532 secs 2024-03-24T02:55:43.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:55:43 smithi118 ceph-mon[31535]: pgmap v5420: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:55:43.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:55:43 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T02:55:43.703 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:55:43 smithi189 ceph-mon[36187]: pgmap v5420: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:55:43.703 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:55:43 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T02:55:43.710 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:55:43 smithi193 ceph-mon[36878]: pgmap v5420: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:55:43.710 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:55:43 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T02:55:44.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:55:44 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T02:55:44.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:55:44 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T02:55:44.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:55:44 smithi118 ceph-mon[31535]: pgmap v5421: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 6 op/s 2024-03-24T02:55:44.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:55:44 smithi118 ceph-mon[31535]: pgmap v5422: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 4 op/s 2024-03-24T02:55:44.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:55:44 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T02:55:44.705 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:55:44 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T02:55:44.705 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:55:44 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T02:55:44.706 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:55:44 smithi189 ceph-mon[36187]: pgmap v5421: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 6 op/s 2024-03-24T02:55:44.706 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:55:44 smithi189 ceph-mon[36187]: pgmap v5422: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 4 op/s 2024-03-24T02:55:44.706 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:55:44 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T02:55:44.710 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:55:44 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T02:55:44.710 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:55:44 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T02:55:44.710 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:55:44 smithi193 ceph-mon[36878]: pgmap v5421: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 6 op/s 2024-03-24T02:55:44.710 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:55:44 smithi193 ceph-mon[36878]: pgmap v5422: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 4 op/s 2024-03-24T02:55:44.710 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:55:44 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T02:55:47.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:55:46 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 9041.743593 secs 2024-03-24T02:55:47.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:55:46 smithi118 ceph-mon[31535]: pgmap v5423: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 6 op/s 2024-03-24T02:55:47.203 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:55:46 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 9041.743593 secs 2024-03-24T02:55:47.203 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:55:46 smithi189 ceph-mon[36187]: pgmap v5423: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 6 op/s 2024-03-24T02:55:47.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:55:46 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 9041.743593 secs 2024-03-24T02:55:47.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:55:46 smithi193 ceph-mon[36878]: pgmap v5423: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 6 op/s 2024-03-24T02:55:49.195 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:55:48 smithi118 ceph-mon[31535]: pgmap v5424: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 5 op/s 2024-03-24T02:55:49.203 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:55:48 smithi189 ceph-mon[36187]: pgmap v5424: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 5 op/s 2024-03-24T02:55:49.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:55:48 smithi193 ceph-mon[36878]: pgmap v5424: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 5 op/s 2024-03-24T02:55:51.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:55:50 smithi118 ceph-mon[31535]: pgmap v5425: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 6 op/s 2024-03-24T02:55:51.203 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:55:50 smithi189 ceph-mon[36187]: pgmap v5425: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 6 op/s 2024-03-24T02:55:51.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:55:50 smithi193 ceph-mon[36878]: pgmap v5425: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 6 op/s 2024-03-24T02:55:52.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:55:51 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 9046.743671 secs 2024-03-24T02:55:52.203 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:55:51 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 9046.743671 secs 2024-03-24T02:55:52.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:55:51 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 9046.743671 secs 2024-03-24T02:55:53.104 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:55:52 smithi189 ceph-mon[36187]: pgmap v5426: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 6 op/s 2024-03-24T02:55:53.111 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:55:52 smithi193 ceph-mon[36878]: pgmap v5426: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 6 op/s 2024-03-24T02:55:53.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:55:52 smithi118 ceph-mon[31535]: pgmap v5426: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 6 op/s 2024-03-24T02:55:55.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:55:54 smithi118 ceph-mon[31535]: pgmap v5427: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 5 op/s 2024-03-24T02:55:55.203 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:55:54 smithi189 ceph-mon[36187]: pgmap v5427: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 5 op/s 2024-03-24T02:55:55.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:55:54 smithi193 ceph-mon[36878]: pgmap v5427: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 5 op/s 2024-03-24T02:55:57.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:55:56 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 9051.743732 secs 2024-03-24T02:55:57.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:55:56 smithi118 ceph-mon[31535]: pgmap v5428: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T02:55:57.203 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:55:56 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 9051.743732 secs 2024-03-24T02:55:57.203 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:55:56 smithi189 ceph-mon[36187]: pgmap v5428: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T02:55:57.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:55:56 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 9051.743732 secs 2024-03-24T02:55:57.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:55:56 smithi193 ceph-mon[36878]: pgmap v5428: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T02:55:59.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:55:58 smithi118 ceph-mon[31535]: pgmap v5429: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T02:55:59.203 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:55:58 smithi189 ceph-mon[36187]: pgmap v5429: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T02:55:59.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:55:58 smithi193 ceph-mon[36878]: pgmap v5429: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T02:56:01.195 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:56:00 smithi118 ceph-mon[31535]: pgmap v5430: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-03-24T02:56:01.203 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:56:00 smithi189 ceph-mon[36187]: pgmap v5430: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-03-24T02:56:01.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:56:00 smithi193 ceph-mon[36878]: pgmap v5430: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-03-24T02:56:02.195 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:56:01 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 9056.743785 secs 2024-03-24T02:56:02.203 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:56:01 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 9056.743785 secs 2024-03-24T02:56:02.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:56:01 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 9056.743785 secs 2024-03-24T02:56:03.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:56:02 smithi118 ceph-mon[31535]: pgmap v5431: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:56:03.203 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:56:02 smithi189 ceph-mon[36187]: pgmap v5431: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:56:03.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:56:02 smithi193 ceph-mon[36878]: pgmap v5431: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:56:05.144 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:56:04 smithi118 ceph-mon[31535]: pgmap v5432: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T02:56:05.203 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:56:04 smithi189 ceph-mon[36187]: pgmap v5432: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T02:56:05.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:56:04 smithi193 ceph-mon[36878]: pgmap v5432: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T02:56:07.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:56:06 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 9061.743844 secs 2024-03-24T02:56:07.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:56:06 smithi118 ceph-mon[31535]: pgmap v5433: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T02:56:07.203 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:56:06 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 9061.743844 secs 2024-03-24T02:56:07.203 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:56:06 smithi189 ceph-mon[36187]: pgmap v5433: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T02:56:07.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:56:06 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 9061.743844 secs 2024-03-24T02:56:07.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:56:06 smithi193 ceph-mon[36878]: pgmap v5433: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T02:56:08.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:56:07 smithi118 ceph-mon[31535]: pgmap v5434: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T02:56:08.453 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:56:07 smithi189 ceph-mon[36187]: pgmap v5434: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T02:56:08.460 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:56:07 smithi193 ceph-mon[36878]: pgmap v5434: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T02:56:11.195 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:56:10 smithi118 ceph-mon[31535]: pgmap v5435: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-03-24T02:56:11.203 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:56:10 smithi189 ceph-mon[36187]: pgmap v5435: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-03-24T02:56:11.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:56:10 smithi193 ceph-mon[36878]: pgmap v5435: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-03-24T02:56:12.195 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:56:11 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 9066.743905 secs 2024-03-24T02:56:12.203 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:56:11 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 9066.743905 secs 2024-03-24T02:56:12.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:56:11 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 9066.743905 secs 2024-03-24T02:56:13.097 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:56:12 smithi118 ceph-mon[31535]: pgmap v5436: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:56:13.203 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:56:12 smithi189 ceph-mon[36187]: pgmap v5436: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:56:13.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:56:12 smithi193 ceph-mon[36878]: pgmap v5436: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:56:15.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:56:14 smithi118 ceph-mon[31535]: pgmap v5437: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-03-24T02:56:15.203 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:56:14 smithi189 ceph-mon[36187]: pgmap v5437: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-03-24T02:56:15.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:56:14 smithi193 ceph-mon[36878]: pgmap v5437: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-03-24T02:56:17.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:56:16 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 9071.743992 secs 2024-03-24T02:56:17.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:56:16 smithi118 ceph-mon[31535]: pgmap v5438: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 6 op/s 2024-03-24T02:56:17.203 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:56:16 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 9071.743992 secs 2024-03-24T02:56:17.203 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:56:16 smithi189 ceph-mon[36187]: pgmap v5438: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 6 op/s 2024-03-24T02:56:17.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:56:16 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 9071.743992 secs 2024-03-24T02:56:17.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:56:16 smithi193 ceph-mon[36878]: pgmap v5438: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 6 op/s 2024-03-24T02:56:19.195 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:56:18 smithi118 ceph-mon[31535]: pgmap v5439: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 5 op/s 2024-03-24T02:56:19.203 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:56:18 smithi189 ceph-mon[36187]: pgmap v5439: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 5 op/s 2024-03-24T02:56:19.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:56:18 smithi193 ceph-mon[36878]: pgmap v5439: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 5 op/s 2024-03-24T02:56:21.024 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:56:20 smithi118 ceph-mon[31535]: pgmap v5440: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-03-24T02:56:21.203 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:56:20 smithi189 ceph-mon[36187]: pgmap v5440: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-03-24T02:56:21.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:56:20 smithi193 ceph-mon[36878]: pgmap v5440: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-03-24T02:56:22.195 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:56:21 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 9076.744033 secs 2024-03-24T02:56:22.203 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:56:21 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 9076.744033 secs 2024-03-24T02:56:22.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:56:21 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 9076.744033 secs 2024-03-24T02:56:23.104 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:56:22 smithi189 ceph-mon[36187]: pgmap v5441: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-03-24T02:56:23.111 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:56:22 smithi193 ceph-mon[36878]: pgmap v5441: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-03-24T02:56:23.195 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:56:22 smithi118 ceph-mon[31535]: pgmap v5441: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-03-24T02:56:25.195 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:56:24 smithi118 ceph-mon[31535]: pgmap v5442: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-03-24T02:56:25.203 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:56:24 smithi189 ceph-mon[36187]: pgmap v5442: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-03-24T02:56:25.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:56:24 smithi193 ceph-mon[36878]: pgmap v5442: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-03-24T02:56:26.195 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:56:25 smithi118 ceph-mon[31535]: pgmap v5443: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T02:56:26.203 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:56:25 smithi189 ceph-mon[36187]: pgmap v5443: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T02:56:26.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:56:25 smithi193 ceph-mon[36878]: pgmap v5443: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T02:56:27.195 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:56:26 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 9081.744088 secs 2024-03-24T02:56:27.203 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:56:26 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 9081.744088 secs 2024-03-24T02:56:27.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:56:26 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 9081.744088 secs 2024-03-24T02:56:28.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:56:27 smithi118 ceph-mon[31535]: pgmap v5444: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 5 op/s 2024-03-24T02:56:28.203 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:56:27 smithi189 ceph-mon[36187]: pgmap v5444: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 5 op/s 2024-03-24T02:56:28.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:56:27 smithi193 ceph-mon[36878]: pgmap v5444: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 5 op/s 2024-03-24T02:56:31.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:56:30 smithi118 ceph-mon[31535]: pgmap v5445: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T02:56:31.203 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:56:30 smithi189 ceph-mon[36187]: pgmap v5445: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T02:56:31.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:56:30 smithi193 ceph-mon[36878]: pgmap v5445: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T02:56:32.195 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:56:31 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 9086.744150 secs 2024-03-24T02:56:32.203 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:56:31 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 9086.744150 secs 2024-03-24T02:56:32.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:56:31 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 9086.744150 secs 2024-03-24T02:56:33.195 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:56:32 smithi118 ceph-mon[31535]: pgmap v5446: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:56:33.203 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:56:32 smithi189 ceph-mon[36187]: pgmap v5446: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:56:33.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:56:32 smithi193 ceph-mon[36878]: pgmap v5446: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:56:35.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:56:34 smithi118 ceph-mon[31535]: pgmap v5447: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T02:56:35.203 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:56:34 smithi189 ceph-mon[36187]: pgmap v5447: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T02:56:35.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:56:34 smithi193 ceph-mon[36878]: pgmap v5447: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T02:56:37.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:56:36 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 9091.744164 secs 2024-03-24T02:56:37.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:56:36 smithi118 ceph-mon[31535]: pgmap v5448: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T02:56:37.203 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:56:36 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 9091.744164 secs 2024-03-24T02:56:37.203 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:56:36 smithi189 ceph-mon[36187]: pgmap v5448: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T02:56:37.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:56:36 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 9091.744164 secs 2024-03-24T02:56:37.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:56:36 smithi193 ceph-mon[36878]: pgmap v5448: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T02:56:39.195 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:56:38 smithi118 ceph-mon[31535]: pgmap v5449: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 5 op/s 2024-03-24T02:56:39.203 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:56:38 smithi189 ceph-mon[36187]: pgmap v5449: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 5 op/s 2024-03-24T02:56:39.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:56:38 smithi193 ceph-mon[36878]: pgmap v5449: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 5 op/s 2024-03-24T02:56:41.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:56:40 smithi118 ceph-mon[31535]: pgmap v5450: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T02:56:41.203 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:56:40 smithi189 ceph-mon[36187]: pgmap v5450: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T02:56:41.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:56:40 smithi193 ceph-mon[36878]: pgmap v5450: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T02:56:42.195 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:56:41 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 9096.744224 secs 2024-03-24T02:56:42.203 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:56:41 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 9096.744224 secs 2024-03-24T02:56:42.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:56:41 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 9096.744224 secs 2024-03-24T02:56:43.098 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:56:42 smithi118 ceph-mon[31535]: pgmap v5451: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:56:43.203 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:56:42 smithi189 ceph-mon[36187]: pgmap v5451: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:56:43.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:56:42 smithi193 ceph-mon[36878]: pgmap v5451: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:56:44.195 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:56:43 smithi118 ceph-mon[31535]: pgmap v5452: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T02:56:44.203 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:56:43 smithi189 ceph-mon[36187]: pgmap v5452: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T02:56:44.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:56:43 smithi193 ceph-mon[36878]: pgmap v5452: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T02:56:45.195 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:56:44 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T02:56:45.203 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:56:44 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T02:56:45.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:56:44 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T02:56:46.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:56:45 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T02:56:46.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:56:45 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T02:56:46.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:56:45 smithi118 ceph-mon[31535]: pgmap v5453: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 6 op/s 2024-03-24T02:56:46.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:56:45 smithi118 ceph-mon[31535]: pgmap v5454: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 6 op/s 2024-03-24T02:56:46.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:56:45 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T02:56:46.203 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:56:45 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T02:56:46.203 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:56:45 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T02:56:46.203 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:56:45 smithi189 ceph-mon[36187]: pgmap v5453: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 6 op/s 2024-03-24T02:56:46.203 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:56:45 smithi189 ceph-mon[36187]: pgmap v5454: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 6 op/s 2024-03-24T02:56:46.203 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:56:45 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T02:56:46.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:56:45 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T02:56:46.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:56:45 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T02:56:46.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:56:45 smithi193 ceph-mon[36878]: pgmap v5453: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 6 op/s 2024-03-24T02:56:46.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:56:45 smithi193 ceph-mon[36878]: pgmap v5454: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 6 op/s 2024-03-24T02:56:46.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:56:45 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T02:56:47.195 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:56:46 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 9101.744286 secs 2024-03-24T02:56:47.203 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:56:46 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 9101.744286 secs 2024-03-24T02:56:47.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:56:46 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 9101.744286 secs 2024-03-24T02:56:48.195 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:56:47 smithi118 ceph-mon[31535]: pgmap v5455: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 5 op/s 2024-03-24T02:56:48.203 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:56:47 smithi189 ceph-mon[36187]: pgmap v5455: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 5 op/s 2024-03-24T02:56:48.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:56:47 smithi193 ceph-mon[36878]: pgmap v5455: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.1 KiB/s rd, 5 op/s 2024-03-24T02:56:50.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:56:50 smithi118 ceph-mon[31535]: pgmap v5456: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 6 op/s 2024-03-24T02:56:50.703 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:56:50 smithi189 ceph-mon[36187]: pgmap v5456: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 6 op/s 2024-03-24T02:56:50.710 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:56:50 smithi193 ceph-mon[36878]: pgmap v5456: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 6 op/s 2024-03-24T02:56:51.597 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:56:51 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 9106.744364 secs 2024-03-24T02:56:51.703 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:56:51 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 9106.744364 secs 2024-03-24T02:56:51.710 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:56:51 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 9106.744364 secs 2024-03-24T02:56:52.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:56:52 smithi118 ceph-mon[31535]: pgmap v5457: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 6 op/s 2024-03-24T02:56:52.703 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:56:52 smithi189 ceph-mon[36187]: pgmap v5457: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 6 op/s 2024-03-24T02:56:52.710 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:56:52 smithi193 ceph-mon[36878]: pgmap v5457: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 6 op/s 2024-03-24T02:56:54.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:56:54 smithi118 ceph-mon[31535]: pgmap v5458: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 6 op/s 2024-03-24T02:56:54.703 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:56:54 smithi189 ceph-mon[36187]: pgmap v5458: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 6 op/s 2024-03-24T02:56:54.710 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:56:54 smithi193 ceph-mon[36878]: pgmap v5458: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 6 op/s 2024-03-24T02:56:56.695 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:56:56 smithi118 ceph-mon[31535]: pgmap v5459: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 5 op/s 2024-03-24T02:56:56.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:56:56 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 9111.744425 secs 2024-03-24T02:56:56.703 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:56:56 smithi189 ceph-mon[36187]: pgmap v5459: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 5 op/s 2024-03-24T02:56:56.703 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:56:56 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 9111.744425 secs 2024-03-24T02:56:56.710 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:56:56 smithi193 ceph-mon[36878]: pgmap v5459: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 5 op/s 2024-03-24T02:56:56.710 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:56:56 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 9111.744425 secs 2024-03-24T02:56:58.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:56:58 smithi118 ceph-mon[31535]: pgmap v5460: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:56:58.703 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:56:58 smithi189 ceph-mon[36187]: pgmap v5460: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:56:58.710 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:56:58 smithi193 ceph-mon[36878]: pgmap v5460: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:57:00.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:57:00 smithi118 ceph-mon[31535]: pgmap v5461: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-24T02:57:00.703 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:57:00 smithi189 ceph-mon[36187]: pgmap v5461: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-24T02:57:00.710 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:57:00 smithi193 ceph-mon[36878]: pgmap v5461: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-24T02:57:01.695 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:57:01 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 9116.744488 secs 2024-03-24T02:57:01.703 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:57:01 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 9116.744488 secs 2024-03-24T02:57:01.710 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:57:01 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 9116.744488 secs 2024-03-24T02:57:02.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:57:02 smithi118 ceph-mon[31535]: pgmap v5462: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:57:02.703 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:57:02 smithi189 ceph-mon[36187]: pgmap v5462: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:57:02.710 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:57:02 smithi193 ceph-mon[36878]: pgmap v5462: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:57:04.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:57:04 smithi118 ceph-mon[31535]: pgmap v5463: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T02:57:04.703 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:57:04 smithi189 ceph-mon[36187]: pgmap v5463: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T02:57:04.710 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:57:04 smithi193 ceph-mon[36878]: pgmap v5463: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T02:57:06.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:57:06 smithi118 ceph-mon[31535]: pgmap v5464: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T02:57:06.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:57:06 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 9121.744523 secs 2024-03-24T02:57:06.705 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:57:06 smithi189 ceph-mon[36187]: pgmap v5464: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T02:57:06.705 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:57:06 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 9121.744523 secs 2024-03-24T02:57:06.710 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:57:06 smithi193 ceph-mon[36878]: pgmap v5464: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T02:57:06.710 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:57:06 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 9121.744523 secs 2024-03-24T02:57:08.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:57:08 smithi118 ceph-mon[31535]: pgmap v5465: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:57:08.703 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:57:08 smithi189 ceph-mon[36187]: pgmap v5465: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:57:08.710 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:57:08 smithi193 ceph-mon[36878]: pgmap v5465: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:57:10.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:57:10 smithi118 ceph-mon[31535]: pgmap v5466: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-24T02:57:10.703 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:57:10 smithi189 ceph-mon[36187]: pgmap v5466: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-24T02:57:10.710 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:57:10 smithi193 ceph-mon[36878]: pgmap v5466: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-24T02:57:11.695 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:57:11 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 9126.744609 secs 2024-03-24T02:57:11.703 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:57:11 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 9126.744609 secs 2024-03-24T02:57:11.710 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:57:11 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 9126.744609 secs 2024-03-24T02:57:12.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:57:12 smithi118 ceph-mon[31535]: pgmap v5467: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:57:12.703 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:57:12 smithi189 ceph-mon[36187]: pgmap v5467: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:57:12.710 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:57:12 smithi193 ceph-mon[36878]: pgmap v5467: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:57:14.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:57:14 smithi118 ceph-mon[31535]: pgmap v5468: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T02:57:14.703 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:57:14 smithi189 ceph-mon[36187]: pgmap v5468: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T02:57:14.710 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:57:14 smithi193 ceph-mon[36878]: pgmap v5468: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T02:57:16.695 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:57:16 smithi118 ceph-mon[31535]: pgmap v5469: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T02:57:16.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:57:16 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 9131.744622 secs 2024-03-24T02:57:16.703 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:57:16 smithi189 ceph-mon[36187]: pgmap v5469: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T02:57:16.703 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:57:16 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 9131.744622 secs 2024-03-24T02:57:16.710 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:57:16 smithi193 ceph-mon[36878]: pgmap v5469: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T02:57:16.710 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:57:16 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 9131.744622 secs 2024-03-24T02:57:18.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:57:18 smithi118 ceph-mon[31535]: pgmap v5470: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:57:18.703 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:57:18 smithi189 ceph-mon[36187]: pgmap v5470: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:57:18.710 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:57:18 smithi193 ceph-mon[36878]: pgmap v5470: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:57:20.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:57:20 smithi118 ceph-mon[31535]: pgmap v5471: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-24T02:57:20.703 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:57:20 smithi189 ceph-mon[36187]: pgmap v5471: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-24T02:57:20.710 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:57:20 smithi193 ceph-mon[36878]: pgmap v5471: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-24T02:57:21.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:57:21 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 9136.744678 secs 2024-03-24T02:57:21.703 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:57:21 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 9136.744678 secs 2024-03-24T02:57:21.710 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:57:21 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 9136.744678 secs 2024-03-24T02:57:22.695 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:57:22 smithi118 ceph-mon[31535]: pgmap v5472: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:57:22.703 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:57:22 smithi189 ceph-mon[36187]: pgmap v5472: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:57:22.710 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:57:22 smithi193 ceph-mon[36878]: pgmap v5472: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:57:24.695 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:57:24 smithi118 ceph-mon[31535]: pgmap v5473: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T02:57:24.703 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:57:24 smithi189 ceph-mon[36187]: pgmap v5473: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T02:57:24.710 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:57:24 smithi193 ceph-mon[36878]: pgmap v5473: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T02:57:26.695 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:57:26 smithi118 ceph-mon[31535]: pgmap v5474: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T02:57:26.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:57:26 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 9141.744750 secs 2024-03-24T02:57:26.703 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:57:26 smithi189 ceph-mon[36187]: pgmap v5474: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T02:57:26.703 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:57:26 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 9141.744750 secs 2024-03-24T02:57:26.710 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:57:26 smithi193 ceph-mon[36878]: pgmap v5474: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T02:57:26.710 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:57:26 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 9141.744750 secs 2024-03-24T02:57:28.695 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:57:28 smithi118 ceph-mon[31535]: pgmap v5475: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:57:28.703 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:57:28 smithi189 ceph-mon[36187]: pgmap v5475: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:57:28.710 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:57:28 smithi193 ceph-mon[36878]: pgmap v5475: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:57:30.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:57:30 smithi118 ceph-mon[31535]: pgmap v5476: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-24T02:57:30.703 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:57:30 smithi189 ceph-mon[36187]: pgmap v5476: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-24T02:57:30.710 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:57:30 smithi193 ceph-mon[36878]: pgmap v5476: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-24T02:57:31.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:57:31 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 9146.744837 secs 2024-03-24T02:57:31.703 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:57:31 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 9146.744837 secs 2024-03-24T02:57:31.710 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:57:31 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 9146.744837 secs 2024-03-24T02:57:32.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:57:32 smithi118 ceph-mon[31535]: pgmap v5477: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:57:32.703 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:57:32 smithi189 ceph-mon[36187]: pgmap v5477: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:57:32.710 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:57:32 smithi193 ceph-mon[36878]: pgmap v5477: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:57:34.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:57:34 smithi118 ceph-mon[31535]: pgmap v5478: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T02:57:34.703 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:57:34 smithi189 ceph-mon[36187]: pgmap v5478: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T02:57:34.710 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:57:34 smithi193 ceph-mon[36878]: pgmap v5478: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T02:57:36.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:57:36 smithi118 ceph-mon[31535]: pgmap v5479: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T02:57:36.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:57:36 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 9151.744892 secs 2024-03-24T02:57:36.703 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:57:36 smithi189 ceph-mon[36187]: pgmap v5479: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T02:57:36.703 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:57:36 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 9151.744892 secs 2024-03-24T02:57:36.710 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:57:36 smithi193 ceph-mon[36878]: pgmap v5479: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T02:57:36.710 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:57:36 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 9151.744892 secs 2024-03-24T02:57:38.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:57:38 smithi118 ceph-mon[31535]: pgmap v5480: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:57:38.703 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:57:38 smithi189 ceph-mon[36187]: pgmap v5480: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:57:38.710 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:57:38 smithi193 ceph-mon[36878]: pgmap v5480: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:57:40.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:57:40 smithi118 ceph-mon[31535]: pgmap v5481: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-24T02:57:40.703 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:57:40 smithi189 ceph-mon[36187]: pgmap v5481: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-24T02:57:40.710 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:57:40 smithi193 ceph-mon[36878]: pgmap v5481: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-24T02:57:41.703 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:57:41 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 9156.744908 secs 2024-03-24T02:57:41.710 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:57:41 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 9156.744908 secs 2024-03-24T02:57:41.945 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:57:41 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 9156.744908 secs 2024-03-24T02:57:42.710 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:57:42 smithi193 ceph-mon[36878]: pgmap v5482: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:57:42.945 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:57:42 smithi118 ceph-mon[31535]: pgmap v5482: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:57:42.953 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:57:42 smithi189 ceph-mon[36187]: pgmap v5482: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:57:44.710 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:57:44 smithi193 ceph-mon[36878]: pgmap v5483: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T02:57:44.748 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:57:44 smithi118 ceph-mon[31535]: pgmap v5483: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T02:57:44.953 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:57:44 smithi189 ceph-mon[36187]: pgmap v5483: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T02:57:46.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:57:46 smithi118 ceph-mon[31535]: pgmap v5484: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T02:57:46.949 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:57:46 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 9161.744966 secs 2024-03-24T02:57:46.950 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:57:46 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T02:57:46.950 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:57:46 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-24T02:57:46.950 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:57:46 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-24T02:57:46.950 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:57:46 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-24T02:57:46.950 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:57:46 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-03-24T02:57:46.953 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:57:46 smithi189 ceph-mon[36187]: pgmap v5484: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T02:57:46.953 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:57:46 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 9161.744966 secs 2024-03-24T02:57:46.953 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:57:46 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T02:57:46.953 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:57:46 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-24T02:57:46.953 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:57:46 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-24T02:57:46.953 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:57:46 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-24T02:57:46.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:57:46 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-03-24T02:57:46.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:57:46 smithi193 ceph-mon[36878]: pgmap v5484: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T02:57:46.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:57:46 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 9161.744966 secs 2024-03-24T02:57:46.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:57:46 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T02:57:46.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:57:46 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-24T02:57:46.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:57:46 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-24T02:57:46.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:57:46 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-24T02:57:46.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:57:46 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-03-24T02:57:47.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:57:47 smithi118 ceph-mon[31535]: Adjusting osd_memory_target on smithi189 to 174.1M 2024-03-24T02:57:47.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:57:47 smithi118 ceph-mon[31535]: Unable to set osd_memory_target on smithi189 to 182635622: error parsing value: Value '182635622' is below minimum 939524096 2024-03-24T02:57:47.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:57:47 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-24T02:57:47.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:57:47 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-24T02:57:47.947 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:57:47 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-24T02:57:47.947 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:57:47 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-03-24T02:57:47.947 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:57:47 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T02:57:47.947 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:57:47 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T02:57:47.947 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:57:47 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T02:57:47.953 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:57:47 smithi189 ceph-mon[36187]: Adjusting osd_memory_target on smithi189 to 174.1M 2024-03-24T02:57:47.953 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:57:47 smithi189 ceph-mon[36187]: Unable to set osd_memory_target on smithi189 to 182635622: error parsing value: Value '182635622' is below minimum 939524096 2024-03-24T02:57:47.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:57:47 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-24T02:57:47.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:57:47 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-24T02:57:47.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:57:47 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-24T02:57:47.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:57:47 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-03-24T02:57:47.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:57:47 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T02:57:47.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:57:47 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T02:57:47.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:57:47 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T02:57:47.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:57:47 smithi193 ceph-mon[36878]: Adjusting osd_memory_target on smithi189 to 174.1M 2024-03-24T02:57:47.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:57:47 smithi193 ceph-mon[36878]: Unable to set osd_memory_target on smithi189 to 182635622: error parsing value: Value '182635622' is below minimum 939524096 2024-03-24T02:57:47.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:57:47 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-24T02:57:47.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:57:47 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-24T02:57:47.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:57:47 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-24T02:57:47.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:57:47 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-03-24T02:57:47.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:57:47 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T02:57:47.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:57:47 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T02:57:47.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:57:47 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T02:57:48.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:57:48 smithi118 ceph-mon[31535]: Adjusting osd_memory_target on smithi118 to 176.7M 2024-03-24T02:57:48.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:57:48 smithi118 ceph-mon[31535]: Unable to set osd_memory_target on smithi118 to 185365196: error parsing value: Value '185365196' is below minimum 939524096 2024-03-24T02:57:48.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:57:48 smithi118 ceph-mon[31535]: pgmap v5485: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 5 op/s 2024-03-24T02:57:48.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:57:48 smithi118 ceph-mon[31535]: pgmap v5486: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 6 op/s 2024-03-24T02:57:48.953 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:57:48 smithi189 ceph-mon[36187]: Adjusting osd_memory_target on smithi118 to 176.7M 2024-03-24T02:57:48.953 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:57:48 smithi189 ceph-mon[36187]: Unable to set osd_memory_target on smithi118 to 185365196: error parsing value: Value '185365196' is below minimum 939524096 2024-03-24T02:57:48.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:57:48 smithi189 ceph-mon[36187]: pgmap v5485: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 5 op/s 2024-03-24T02:57:48.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:57:48 smithi189 ceph-mon[36187]: pgmap v5486: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 6 op/s 2024-03-24T02:57:48.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:57:48 smithi193 ceph-mon[36878]: Adjusting osd_memory_target on smithi118 to 176.7M 2024-03-24T02:57:48.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:57:48 smithi193 ceph-mon[36878]: Unable to set osd_memory_target on smithi118 to 185365196: error parsing value: Value '185365196' is below minimum 939524096 2024-03-24T02:57:48.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:57:48 smithi193 ceph-mon[36878]: pgmap v5485: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 5 op/s 2024-03-24T02:57:48.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:57:48 smithi193 ceph-mon[36878]: pgmap v5486: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 6 op/s 2024-03-24T02:57:50.945 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:57:50 smithi118 ceph-mon[31535]: pgmap v5487: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 4 op/s 2024-03-24T02:57:50.953 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:57:50 smithi189 ceph-mon[36187]: pgmap v5487: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 4 op/s 2024-03-24T02:57:50.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:57:50 smithi193 ceph-mon[36878]: pgmap v5487: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 4 op/s 2024-03-24T02:57:51.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:57:51 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 9166.745026 secs 2024-03-24T02:57:51.953 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:57:51 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 9166.745026 secs 2024-03-24T02:57:51.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:57:51 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 9166.745026 secs 2024-03-24T02:57:52.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:57:52 smithi118 ceph-mon[31535]: pgmap v5488: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 6 op/s 2024-03-24T02:57:52.953 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:57:52 smithi189 ceph-mon[36187]: pgmap v5488: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 6 op/s 2024-03-24T02:57:52.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:57:52 smithi193 ceph-mon[36878]: pgmap v5488: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 6 op/s 2024-03-24T02:57:54.754 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:57:54 smithi189 ceph-mon[36187]: pgmap v5489: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 5 op/s 2024-03-24T02:57:54.761 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:57:54 smithi193 ceph-mon[36878]: pgmap v5489: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 5 op/s 2024-03-24T02:57:54.945 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:57:54 smithi118 ceph-mon[31535]: pgmap v5489: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 5 op/s 2024-03-24T02:57:56.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:57:56 smithi118 ceph-mon[31535]: pgmap v5490: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 6 op/s 2024-03-24T02:57:56.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:57:56 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 9171.745095 secs 2024-03-24T02:57:56.953 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:57:56 smithi189 ceph-mon[36187]: pgmap v5490: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 6 op/s 2024-03-24T02:57:56.953 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:57:56 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 9171.745095 secs 2024-03-24T02:57:56.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:57:56 smithi193 ceph-mon[36878]: pgmap v5490: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 6 op/s 2024-03-24T02:57:56.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:57:56 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 9171.745095 secs 2024-03-24T02:57:58.945 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:57:58 smithi118 ceph-mon[31535]: pgmap v5491: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 5 op/s 2024-03-24T02:57:58.953 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:57:58 smithi189 ceph-mon[36187]: pgmap v5491: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 5 op/s 2024-03-24T02:57:58.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:57:58 smithi193 ceph-mon[36878]: pgmap v5491: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 5 op/s 2024-03-24T02:58:00.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:58:00 smithi118 ceph-mon[31535]: pgmap v5492: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T02:58:00.953 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:58:00 smithi189 ceph-mon[36187]: pgmap v5492: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T02:58:00.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:58:00 smithi193 ceph-mon[36878]: pgmap v5492: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T02:58:01.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:58:01 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 9176.745137 secs 2024-03-24T02:58:01.953 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:58:01 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 9176.745137 secs 2024-03-24T02:58:01.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:58:01 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 9176.745137 secs 2024-03-24T02:58:02.945 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:58:02 smithi118 ceph-mon[31535]: pgmap v5493: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T02:58:02.953 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:58:02 smithi189 ceph-mon[36187]: pgmap v5493: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T02:58:02.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:58:02 smithi193 ceph-mon[36878]: pgmap v5493: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T02:58:04.945 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:58:04 smithi118 ceph-mon[31535]: pgmap v5494: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 5 op/s 2024-03-24T02:58:04.953 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:58:04 smithi189 ceph-mon[36187]: pgmap v5494: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 5 op/s 2024-03-24T02:58:04.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:58:04 smithi193 ceph-mon[36878]: pgmap v5494: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 5 op/s 2024-03-24T02:58:06.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:58:06 smithi118 ceph-mon[31535]: pgmap v5495: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T02:58:06.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:58:06 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 9181.745196 secs 2024-03-24T02:58:06.953 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:58:06 smithi189 ceph-mon[36187]: pgmap v5495: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T02:58:06.953 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:58:06 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 9181.745196 secs 2024-03-24T02:58:06.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:58:06 smithi193 ceph-mon[36878]: pgmap v5495: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T02:58:06.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:58:06 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 9181.745196 secs 2024-03-24T02:58:08.945 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:58:08 smithi118 ceph-mon[31535]: pgmap v5496: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:58:08.953 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:58:08 smithi189 ceph-mon[36187]: pgmap v5496: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:58:08.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:58:08 smithi193 ceph-mon[36878]: pgmap v5496: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:58:10.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:58:10 smithi118 ceph-mon[31535]: pgmap v5497: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T02:58:10.953 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:58:10 smithi189 ceph-mon[36187]: pgmap v5497: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T02:58:10.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:58:10 smithi193 ceph-mon[36878]: pgmap v5497: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T02:58:11.945 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:58:11 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 9186.745257 secs 2024-03-24T02:58:11.953 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:58:11 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 9186.745257 secs 2024-03-24T02:58:11.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:58:11 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 9186.745257 secs 2024-03-24T02:58:12.945 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:58:12 smithi118 ceph-mon[31535]: pgmap v5498: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T02:58:12.953 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:58:12 smithi189 ceph-mon[36187]: pgmap v5498: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T02:58:12.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:58:12 smithi193 ceph-mon[36878]: pgmap v5498: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T02:58:14.945 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:58:14 smithi118 ceph-mon[31535]: pgmap v5499: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 5 op/s 2024-03-24T02:58:14.953 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:58:14 smithi189 ceph-mon[36187]: pgmap v5499: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 5 op/s 2024-03-24T02:58:14.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:58:14 smithi193 ceph-mon[36878]: pgmap v5499: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 5 op/s 2024-03-24T02:58:16.945 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:58:16 smithi118 ceph-mon[31535]: pgmap v5500: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T02:58:16.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:58:16 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 9191.745332 secs 2024-03-24T02:58:16.953 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:58:16 smithi189 ceph-mon[36187]: pgmap v5500: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T02:58:16.953 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:58:16 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 9191.745332 secs 2024-03-24T02:58:16.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:58:16 smithi193 ceph-mon[36878]: pgmap v5500: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T02:58:16.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:58:16 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 9191.745332 secs 2024-03-24T02:58:17.222 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T02:58:17.223+0000 7f81ee000640 1 -- 172.21.15.118:0/1693767697 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f81c80036c0 con 0x7f81f81052d0 2024-03-24T02:58:17.223 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T02:58:17.223+0000 7f81db400640 1 -- 172.21.15.118:0/659686424 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f81c4006460 con 0x5622651a13f0 2024-03-24T02:58:17.223 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T02:58:17.223+0000 7f81e4c00640 1 -- 172.21.15.118:0/659686424 <== mon.1 v2:172.21.15.193:3300/0 74 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 409+0+0 (secure 0 0 0) 0x7f81c4006460 con 0x5622651a13f0 2024-03-24T02:58:17.223 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T02:58:17.223+0000 7f81ef400640 1 -- 172.21.15.118:0/1693767697 <== mon.2 v2:172.21.15.189:3300/0 74 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (secure 0 0 0) 0x7f81c80036c0 con 0x7f81f81052d0 2024-03-24T02:58:18.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:58:18 smithi118 ceph-mon[31535]: pgmap v5501: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:58:18.953 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:58:18 smithi189 ceph-mon[36187]: pgmap v5501: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:58:18.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:58:18 smithi193 ceph-mon[36878]: pgmap v5501: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:58:20.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:58:20 smithi118 ceph-mon[31535]: pgmap v5502: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T02:58:20.953 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:58:20 smithi189 ceph-mon[36187]: pgmap v5502: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T02:58:20.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:58:20 smithi193 ceph-mon[36878]: pgmap v5502: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T02:58:21.945 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:58:21 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 9196.745390 secs 2024-03-24T02:58:21.953 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:58:21 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 9196.745390 secs 2024-03-24T02:58:21.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:58:21 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 9196.745390 secs 2024-03-24T02:58:22.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:58:22 smithi118 ceph-mon[31535]: pgmap v5503: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T02:58:22.953 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:58:22 smithi189 ceph-mon[36187]: pgmap v5503: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T02:58:22.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:58:22 smithi193 ceph-mon[36878]: pgmap v5503: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T02:58:24.920 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:58:24 smithi118 ceph-mon[31535]: pgmap v5504: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 5 op/s 2024-03-24T02:58:24.953 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:58:24 smithi189 ceph-mon[36187]: pgmap v5504: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 5 op/s 2024-03-24T02:58:24.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:58:24 smithi193 ceph-mon[36878]: pgmap v5504: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 5 op/s 2024-03-24T02:58:26.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:58:26 smithi118 ceph-mon[31535]: pgmap v5505: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T02:58:26.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:58:26 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 9201.745456 secs 2024-03-24T02:58:26.953 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:58:26 smithi189 ceph-mon[36187]: pgmap v5505: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T02:58:26.953 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:58:26 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 9201.745456 secs 2024-03-24T02:58:26.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:58:26 smithi193 ceph-mon[36878]: pgmap v5505: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T02:58:26.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:58:26 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 9201.745456 secs 2024-03-24T02:58:28.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:58:28 smithi118 ceph-mon[31535]: pgmap v5506: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:58:28.953 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:58:28 smithi189 ceph-mon[36187]: pgmap v5506: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:58:28.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:58:28 smithi193 ceph-mon[36878]: pgmap v5506: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:58:30.945 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:58:30 smithi118 ceph-mon[31535]: pgmap v5507: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T02:58:30.953 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:58:30 smithi189 ceph-mon[36187]: pgmap v5507: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T02:58:30.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:58:30 smithi193 ceph-mon[36878]: pgmap v5507: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T02:58:31.945 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:58:31 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 9206.745496 secs 2024-03-24T02:58:31.953 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:58:31 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 9206.745496 secs 2024-03-24T02:58:31.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:58:31 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 9206.745496 secs 2024-03-24T02:58:32.945 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:58:32 smithi118 ceph-mon[31535]: pgmap v5508: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T02:58:32.953 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:58:32 smithi189 ceph-mon[36187]: pgmap v5508: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T02:58:32.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:58:32 smithi193 ceph-mon[36878]: pgmap v5508: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T02:58:34.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:58:34 smithi118 ceph-mon[31535]: pgmap v5509: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 5 op/s 2024-03-24T02:58:34.953 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:58:34 smithi189 ceph-mon[36187]: pgmap v5509: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 5 op/s 2024-03-24T02:58:34.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:58:34 smithi193 ceph-mon[36878]: pgmap v5509: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 5 op/s 2024-03-24T02:58:36.945 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:58:36 smithi118 ceph-mon[31535]: pgmap v5510: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T02:58:36.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:58:36 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 9211.745557 secs 2024-03-24T02:58:36.953 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:58:36 smithi189 ceph-mon[36187]: pgmap v5510: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T02:58:36.953 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:58:36 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 9211.745557 secs 2024-03-24T02:58:36.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:58:36 smithi193 ceph-mon[36878]: pgmap v5510: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T02:58:36.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:58:36 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 9211.745557 secs 2024-03-24T02:58:38.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:58:38 smithi118 ceph-mon[31535]: pgmap v5511: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:58:38.953 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:58:38 smithi189 ceph-mon[36187]: pgmap v5511: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:58:38.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:58:38 smithi193 ceph-mon[36878]: pgmap v5511: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:58:40.945 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:58:40 smithi118 ceph-mon[31535]: pgmap v5512: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T02:58:40.953 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:58:40 smithi189 ceph-mon[36187]: pgmap v5512: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T02:58:40.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:58:40 smithi193 ceph-mon[36878]: pgmap v5512: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T02:58:41.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:58:41 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 9216.745644 secs 2024-03-24T02:58:41.953 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:58:41 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 9216.745644 secs 2024-03-24T02:58:41.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:58:41 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 9216.745644 secs 2024-03-24T02:58:42.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:58:42 smithi118 ceph-mon[31535]: pgmap v5513: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T02:58:42.953 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:58:42 smithi189 ceph-mon[36187]: pgmap v5513: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T02:58:42.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:58:42 smithi193 ceph-mon[36878]: pgmap v5513: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T02:58:44.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:58:44 smithi118 ceph-mon[31535]: pgmap v5514: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 5 op/s 2024-03-24T02:58:44.953 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:58:44 smithi189 ceph-mon[36187]: pgmap v5514: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 5 op/s 2024-03-24T02:58:44.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:58:44 smithi193 ceph-mon[36878]: pgmap v5514: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 5 op/s 2024-03-24T02:58:46.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:58:46 smithi118 ceph-mon[31535]: pgmap v5515: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T02:58:46.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:58:46 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 9221.745728 secs 2024-03-24T02:58:46.953 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:58:46 smithi189 ceph-mon[36187]: pgmap v5515: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T02:58:46.953 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:58:46 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 9221.745728 secs 2024-03-24T02:58:46.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:58:46 smithi193 ceph-mon[36878]: pgmap v5515: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T02:58:46.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:58:46 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 9221.745728 secs 2024-03-24T02:58:47.945 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:58:47 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T02:58:47.953 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:58:47 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T02:58:47.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:58:47 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T02:58:48.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:58:48 smithi118 ceph-mon[31535]: pgmap v5516: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:58:48.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:58:48 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-24T02:58:48.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:58:48 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-24T02:58:48.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:58:48 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-03-24T02:58:48.947 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:58:48 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-03-24T02:58:48.947 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:58:48 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T02:58:48.947 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:58:48 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T02:58:48.947 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:58:48 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T02:58:48.953 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:58:48 smithi189 ceph-mon[36187]: pgmap v5516: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:58:48.953 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:58:48 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-24T02:58:48.953 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:58:48 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-24T02:58:48.953 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:58:48 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-03-24T02:58:48.953 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:58:48 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-03-24T02:58:48.953 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:58:48 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T02:58:48.953 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:58:48 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T02:58:48.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:58:48 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T02:58:48.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:58:48 smithi193 ceph-mon[36878]: pgmap v5516: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:58:48.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:58:48 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-24T02:58:48.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:58:48 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-24T02:58:48.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:58:48 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-03-24T02:58:48.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:58:48 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-03-24T02:58:48.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:58:48 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T02:58:48.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:58:48 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T02:58:48.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:58:48 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T02:58:49.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:58:49 smithi118 ceph-mon[31535]: Adjusting osd_memory_target on smithi193 to 174.1M 2024-03-24T02:58:49.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:58:49 smithi118 ceph-mon[31535]: Unable to set osd_memory_target on smithi193 to 182639923: error parsing value: Value '182639923' is below minimum 939524096 2024-03-24T02:58:49.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:58:49 smithi118 ceph-mon[31535]: pgmap v5517: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 6 op/s 2024-03-24T02:58:49.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:58:49 smithi118 ceph-mon[31535]: pgmap v5518: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 6 op/s 2024-03-24T02:58:49.953 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:58:49 smithi189 ceph-mon[36187]: Adjusting osd_memory_target on smithi193 to 174.1M 2024-03-24T02:58:49.953 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:58:49 smithi189 ceph-mon[36187]: Unable to set osd_memory_target on smithi193 to 182639923: error parsing value: Value '182639923' is below minimum 939524096 2024-03-24T02:58:49.953 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:58:49 smithi189 ceph-mon[36187]: pgmap v5517: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 6 op/s 2024-03-24T02:58:49.953 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:58:49 smithi189 ceph-mon[36187]: pgmap v5518: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 6 op/s 2024-03-24T02:58:49.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:58:49 smithi193 ceph-mon[36878]: Adjusting osd_memory_target on smithi193 to 174.1M 2024-03-24T02:58:49.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:58:49 smithi193 ceph-mon[36878]: Unable to set osd_memory_target on smithi193 to 182639923: error parsing value: Value '182639923' is below minimum 939524096 2024-03-24T02:58:49.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:58:49 smithi193 ceph-mon[36878]: pgmap v5517: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 6 op/s 2024-03-24T02:58:49.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:58:49 smithi193 ceph-mon[36878]: pgmap v5518: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 6 op/s 2024-03-24T02:58:51.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:58:51 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 9226.745741 secs 2024-03-24T02:58:51.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:58:51 smithi118 ceph-mon[31535]: pgmap v5519: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 6 op/s 2024-03-24T02:58:51.953 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:58:51 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 9226.745741 secs 2024-03-24T02:58:51.953 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:58:51 smithi189 ceph-mon[36187]: pgmap v5519: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 6 op/s 2024-03-24T02:58:51.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:58:51 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 9226.745741 secs 2024-03-24T02:58:51.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:58:51 smithi193 ceph-mon[36878]: pgmap v5519: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 6 op/s 2024-03-24T02:58:53.945 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:58:53 smithi118 ceph-mon[31535]: pgmap v5520: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 5 op/s 2024-03-24T02:58:53.953 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:58:53 smithi189 ceph-mon[36187]: pgmap v5520: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 5 op/s 2024-03-24T02:58:53.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:58:53 smithi193 ceph-mon[36878]: pgmap v5520: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 5 op/s 2024-03-24T02:58:55.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:58:55 smithi118 ceph-mon[31535]: pgmap v5521: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 6 op/s 2024-03-24T02:58:55.953 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:58:55 smithi189 ceph-mon[36187]: pgmap v5521: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 6 op/s 2024-03-24T02:58:55.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:58:55 smithi193 ceph-mon[36878]: pgmap v5521: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 6 op/s 2024-03-24T02:58:56.953 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:58:56 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 9231.745828 secs 2024-03-24T02:58:56.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:58:56 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 9231.745828 secs 2024-03-24T02:58:57.195 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:58:56 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 9231.745828 secs 2024-03-24T02:58:57.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:58:57 smithi193 ceph-mon[36878]: pgmap v5522: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 6 op/s 2024-03-24T02:58:58.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:58:57 smithi118 ceph-mon[31535]: pgmap v5522: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 6 op/s 2024-03-24T02:58:58.203 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:58:57 smithi189 ceph-mon[36187]: pgmap v5522: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 6 op/s 2024-03-24T02:58:59.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:58:59 smithi193 ceph-mon[36878]: pgmap v5523: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 5 op/s 2024-03-24T02:59:00.195 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:58:59 smithi118 ceph-mon[31535]: pgmap v5523: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 5 op/s 2024-03-24T02:59:00.203 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:58:59 smithi189 ceph-mon[36187]: pgmap v5523: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 5 op/s 2024-03-24T02:59:02.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:59:01 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 9236.745912 secs 2024-03-24T02:59:02.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:59:01 smithi118 ceph-mon[31535]: pgmap v5524: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T02:59:02.203 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:59:01 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 9236.745912 secs 2024-03-24T02:59:02.203 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:59:01 smithi189 ceph-mon[36187]: pgmap v5524: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T02:59:02.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:59:01 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 9236.745912 secs 2024-03-24T02:59:02.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:59:01 smithi193 ceph-mon[36878]: pgmap v5524: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T02:59:04.195 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:59:03 smithi118 ceph-mon[31535]: pgmap v5525: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T02:59:04.203 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:59:03 smithi189 ceph-mon[36187]: pgmap v5525: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T02:59:04.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:59:03 smithi193 ceph-mon[36878]: pgmap v5525: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T02:59:06.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:59:05 smithi118 ceph-mon[31535]: pgmap v5526: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-03-24T02:59:06.203 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:59:05 smithi189 ceph-mon[36187]: pgmap v5526: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-03-24T02:59:06.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:59:05 smithi193 ceph-mon[36878]: pgmap v5526: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-03-24T02:59:07.195 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:59:06 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 9241.745998 secs 2024-03-24T02:59:07.203 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:59:06 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 9241.745998 secs 2024-03-24T02:59:07.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:59:06 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 9241.745998 secs 2024-03-24T02:59:08.195 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:59:07 smithi118 ceph-mon[31535]: pgmap v5527: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:59:08.203 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:59:07 smithi189 ceph-mon[36187]: pgmap v5527: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:59:08.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:59:07 smithi193 ceph-mon[36878]: pgmap v5527: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:59:10.195 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:59:09 smithi118 ceph-mon[31535]: pgmap v5528: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T02:59:10.203 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:59:09 smithi189 ceph-mon[36187]: pgmap v5528: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T02:59:10.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:59:09 smithi193 ceph-mon[36878]: pgmap v5528: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T02:59:12.024 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:59:11 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 9246.746083 secs 2024-03-24T02:59:12.024 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:59:11 smithi118 ceph-mon[31535]: pgmap v5529: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T02:59:12.203 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:59:11 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 9246.746083 secs 2024-03-24T02:59:12.203 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:59:11 smithi189 ceph-mon[36187]: pgmap v5529: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T02:59:12.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:59:11 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 9246.746083 secs 2024-03-24T02:59:12.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:59:11 smithi193 ceph-mon[36878]: pgmap v5529: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T02:59:14.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:59:13 smithi118 ceph-mon[31535]: pgmap v5530: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T02:59:14.203 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:59:13 smithi189 ceph-mon[36187]: pgmap v5530: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T02:59:14.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:59:13 smithi193 ceph-mon[36878]: pgmap v5530: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T02:59:16.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:59:15 smithi118 ceph-mon[31535]: pgmap v5531: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-03-24T02:59:16.203 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:59:15 smithi189 ceph-mon[36187]: pgmap v5531: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-03-24T02:59:16.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:59:15 smithi193 ceph-mon[36878]: pgmap v5531: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-03-24T02:59:17.195 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:59:16 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 9251.746170 secs 2024-03-24T02:59:17.203 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:59:16 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 9251.746170 secs 2024-03-24T02:59:17.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:59:16 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 9251.746170 secs 2024-03-24T02:59:18.195 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:59:17 smithi118 ceph-mon[31535]: pgmap v5532: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:59:18.203 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:59:17 smithi189 ceph-mon[36187]: pgmap v5532: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:59:18.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:59:17 smithi193 ceph-mon[36878]: pgmap v5532: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:59:20.195 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:59:19 smithi118 ceph-mon[31535]: pgmap v5533: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T02:59:20.203 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:59:19 smithi189 ceph-mon[36187]: pgmap v5533: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T02:59:20.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:59:19 smithi193 ceph-mon[36878]: pgmap v5533: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T02:59:22.195 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:59:21 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 9256.746229 secs 2024-03-24T02:59:22.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:59:21 smithi118 ceph-mon[31535]: pgmap v5534: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T02:59:22.203 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:59:21 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 9256.746229 secs 2024-03-24T02:59:22.203 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:59:21 smithi189 ceph-mon[36187]: pgmap v5534: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T02:59:22.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:59:21 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 9256.746229 secs 2024-03-24T02:59:22.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:59:21 smithi193 ceph-mon[36878]: pgmap v5534: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T02:59:24.195 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:59:23 smithi118 ceph-mon[31535]: pgmap v5535: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T02:59:24.203 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:59:23 smithi189 ceph-mon[36187]: pgmap v5535: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T02:59:24.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:59:23 smithi193 ceph-mon[36878]: pgmap v5535: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T02:59:26.195 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:59:25 smithi118 ceph-mon[31535]: pgmap v5536: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-03-24T02:59:26.203 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:59:25 smithi189 ceph-mon[36187]: pgmap v5536: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-03-24T02:59:26.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:59:25 smithi193 ceph-mon[36878]: pgmap v5536: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-03-24T02:59:27.098 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:59:26 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 9261.746288 secs 2024-03-24T02:59:27.203 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:59:26 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 9261.746288 secs 2024-03-24T02:59:27.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:59:26 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 9261.746288 secs 2024-03-24T02:59:28.195 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:59:27 smithi118 ceph-mon[31535]: pgmap v5537: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:59:28.203 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:59:27 smithi189 ceph-mon[36187]: pgmap v5537: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:59:28.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:59:27 smithi193 ceph-mon[36878]: pgmap v5537: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:59:30.195 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:59:29 smithi118 ceph-mon[31535]: pgmap v5538: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T02:59:30.203 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:59:29 smithi189 ceph-mon[36187]: pgmap v5538: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T02:59:30.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:59:29 smithi193 ceph-mon[36878]: pgmap v5538: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T02:59:32.195 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:59:31 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 9266.746375 secs 2024-03-24T02:59:32.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:59:31 smithi118 ceph-mon[31535]: pgmap v5539: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T02:59:32.203 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:59:31 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 9266.746375 secs 2024-03-24T02:59:32.203 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:59:31 smithi189 ceph-mon[36187]: pgmap v5539: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T02:59:32.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:59:31 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 9266.746375 secs 2024-03-24T02:59:32.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:59:31 smithi193 ceph-mon[36878]: pgmap v5539: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T02:59:34.195 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:59:33 smithi118 ceph-mon[31535]: pgmap v5540: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T02:59:34.203 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:59:33 smithi189 ceph-mon[36187]: pgmap v5540: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T02:59:34.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:59:33 smithi193 ceph-mon[36878]: pgmap v5540: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T02:59:36.195 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:59:35 smithi118 ceph-mon[31535]: pgmap v5541: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-03-24T02:59:36.203 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:59:35 smithi189 ceph-mon[36187]: pgmap v5541: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-03-24T02:59:36.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:59:35 smithi193 ceph-mon[36878]: pgmap v5541: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-03-24T02:59:37.195 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:59:36 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 9271.746462 secs 2024-03-24T02:59:37.203 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:59:36 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 9271.746462 secs 2024-03-24T02:59:37.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:59:36 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 9271.746462 secs 2024-03-24T02:59:38.195 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:59:37 smithi118 ceph-mon[31535]: pgmap v5542: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:59:38.203 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:59:37 smithi189 ceph-mon[36187]: pgmap v5542: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:59:38.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:59:37 smithi193 ceph-mon[36878]: pgmap v5542: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:59:40.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:59:39 smithi118 ceph-mon[31535]: pgmap v5543: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T02:59:40.203 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:59:39 smithi189 ceph-mon[36187]: pgmap v5543: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T02:59:40.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:59:39 smithi193 ceph-mon[36878]: pgmap v5543: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T02:59:41.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:59:40 smithi118 ceph-mon[31535]: pgmap v5544: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T02:59:41.203 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:59:40 smithi189 ceph-mon[36187]: pgmap v5544: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T02:59:41.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:59:40 smithi193 ceph-mon[36878]: pgmap v5544: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T02:59:42.195 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:59:41 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 9276.746521 secs 2024-03-24T02:59:42.203 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:59:41 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 9276.746521 secs 2024-03-24T02:59:42.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:59:41 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 9276.746521 secs 2024-03-24T02:59:43.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:59:42 smithi118 ceph-mon[31535]: pgmap v5545: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T02:59:43.203 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:59:42 smithi189 ceph-mon[36187]: pgmap v5545: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T02:59:43.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:59:42 smithi193 ceph-mon[36878]: pgmap v5545: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T02:59:45.899 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:59:45 smithi118 ceph-mon[31535]: pgmap v5546: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-03-24T02:59:45.953 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:59:45 smithi189 ceph-mon[36187]: pgmap v5546: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-03-24T02:59:45.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:59:45 smithi193 ceph-mon[36878]: pgmap v5546: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-03-24T02:59:46.695 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:59:46 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 9281.746607 secs 2024-03-24T02:59:46.953 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:59:46 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 9281.746607 secs 2024-03-24T02:59:46.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:59:46 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 9281.746607 secs 2024-03-24T02:59:47.945 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:59:47 smithi118 ceph-mon[31535]: pgmap v5547: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:59:47.953 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:59:47 smithi189 ceph-mon[36187]: pgmap v5547: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:59:47.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:59:47 smithi193 ceph-mon[36878]: pgmap v5547: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T02:59:49.945 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:59:49 smithi118 ceph-mon[31535]: pgmap v5548: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T02:59:49.945 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:59:49 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T02:59:49.953 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:59:49 smithi189 ceph-mon[36187]: pgmap v5548: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T02:59:49.953 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:59:49 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T02:59:49.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:59:49 smithi193 ceph-mon[36878]: pgmap v5548: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T02:59:49.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:59:49 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T02:59:50.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:59:50 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T02:59:50.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:59:50 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T02:59:50.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:59:50 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T02:59:50.953 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:59:50 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T02:59:50.953 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:59:50 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T02:59:50.953 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:59:50 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T02:59:50.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:59:50 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T02:59:50.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:59:50 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T02:59:50.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:59:50 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T02:59:51.945 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:59:51 smithi118 ceph-mon[31535]: pgmap v5549: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 6 op/s 2024-03-24T02:59:51.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:59:51 smithi118 ceph-mon[31535]: pgmap v5550: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-03-24T02:59:51.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:59:51 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 9286.746669 secs 2024-03-24T02:59:51.953 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:59:51 smithi189 ceph-mon[36187]: pgmap v5549: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 6 op/s 2024-03-24T02:59:51.953 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:59:51 smithi189 ceph-mon[36187]: pgmap v5550: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-03-24T02:59:51.953 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:59:51 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 9286.746669 secs 2024-03-24T02:59:51.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:59:51 smithi193 ceph-mon[36878]: pgmap v5549: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 6 op/s 2024-03-24T02:59:51.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:59:51 smithi193 ceph-mon[36878]: pgmap v5550: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-03-24T02:59:51.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:59:51 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 9286.746669 secs 2024-03-24T02:59:53.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:59:53 smithi118 ceph-mon[31535]: pgmap v5551: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 5 op/s 2024-03-24T02:59:53.953 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:59:53 smithi189 ceph-mon[36187]: pgmap v5551: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 5 op/s 2024-03-24T02:59:53.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:59:53 smithi193 ceph-mon[36878]: pgmap v5551: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 5 op/s 2024-03-24T02:59:55.945 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:59:55 smithi118 ceph-mon[31535]: pgmap v5552: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 4 op/s 2024-03-24T02:59:55.953 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:59:55 smithi189 ceph-mon[36187]: pgmap v5552: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 4 op/s 2024-03-24T02:59:55.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:59:55 smithi193 ceph-mon[36878]: pgmap v5552: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 4 op/s 2024-03-24T02:59:56.854 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:59:56 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 9291.746726 secs 2024-03-24T02:59:56.861 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:59:56 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 9291.746726 secs 2024-03-24T02:59:56.945 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:59:56 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 9291.746726 secs 2024-03-24T02:59:57.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:59:57 smithi118 ceph-mon[31535]: pgmap v5553: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-03-24T02:59:57.953 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:59:57 smithi189 ceph-mon[36187]: pgmap v5553: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-03-24T02:59:57.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:59:57 smithi193 ceph-mon[36878]: pgmap v5553: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-03-24T02:59:59.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 02:59:59 smithi118 ceph-mon[31535]: pgmap v5554: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 5 op/s 2024-03-24T02:59:59.953 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 02:59:59 smithi189 ceph-mon[36187]: pgmap v5554: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 5 op/s 2024-03-24T02:59:59.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 02:59:59 smithi193 ceph-mon[36878]: pgmap v5554: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 5 op/s 2024-03-24T03:00:00.945 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 03:00:00 smithi118 ceph-mon[31535]: overall HEALTH_WARN 6 failed cephadm daemon(s); 1 MDSs report slow requests 2024-03-24T03:00:00.953 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 03:00:00 smithi189 ceph-mon[36187]: overall HEALTH_WARN 6 failed cephadm daemon(s); 1 MDSs report slow requests 2024-03-24T03:00:00.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 03:00:00 smithi193 ceph-mon[36878]: overall HEALTH_WARN 6 failed cephadm daemon(s); 1 MDSs report slow requests 2024-03-24T03:00:01.945 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 03:00:01 smithi118 ceph-mon[31535]: pgmap v5555: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 5 op/s 2024-03-24T03:00:01.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 03:00:01 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 9296.746812 secs 2024-03-24T03:00:01.953 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 03:00:01 smithi189 ceph-mon[36187]: pgmap v5555: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 5 op/s 2024-03-24T03:00:01.953 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 03:00:01 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 9296.746812 secs 2024-03-24T03:00:01.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 03:00:01 smithi193 ceph-mon[36878]: pgmap v5555: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 5 op/s 2024-03-24T03:00:01.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 03:00:01 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 9296.746812 secs 2024-03-24T03:00:03.945 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 03:00:03 smithi118 ceph-mon[31535]: pgmap v5556: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:00:03.953 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 03:00:03 smithi189 ceph-mon[36187]: pgmap v5556: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:00:03.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 03:00:03 smithi193 ceph-mon[36878]: pgmap v5556: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:00:05.945 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 03:00:05 smithi118 ceph-mon[31535]: pgmap v5557: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-03-24T03:00:05.953 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 03:00:05 smithi189 ceph-mon[36187]: pgmap v5557: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-03-24T03:00:05.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 03:00:05 smithi193 ceph-mon[36878]: pgmap v5557: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-03-24T03:00:06.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 03:00:06 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 9301.746899 secs 2024-03-24T03:00:06.953 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 03:00:06 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 9301.746899 secs 2024-03-24T03:00:06.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 03:00:06 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 9301.746899 secs 2024-03-24T03:00:07.945 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 03:00:07 smithi118 ceph-mon[31535]: pgmap v5558: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 6 op/s 2024-03-24T03:00:07.953 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 03:00:07 smithi189 ceph-mon[36187]: pgmap v5558: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 6 op/s 2024-03-24T03:00:07.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 03:00:07 smithi193 ceph-mon[36878]: pgmap v5558: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 6 op/s 2024-03-24T03:00:09.945 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 03:00:09 smithi118 ceph-mon[31535]: pgmap v5559: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 5 op/s 2024-03-24T03:00:09.953 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 03:00:09 smithi189 ceph-mon[36187]: pgmap v5559: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 5 op/s 2024-03-24T03:00:09.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 03:00:09 smithi193 ceph-mon[36878]: pgmap v5559: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 5 op/s 2024-03-24T03:00:11.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 03:00:11 smithi118 ceph-mon[31535]: pgmap v5560: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 6 op/s 2024-03-24T03:00:11.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 03:00:11 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 9306.746939 secs 2024-03-24T03:00:11.953 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 03:00:11 smithi189 ceph-mon[36187]: pgmap v5560: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 6 op/s 2024-03-24T03:00:11.953 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 03:00:11 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 9306.746939 secs 2024-03-24T03:00:11.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 03:00:11 smithi193 ceph-mon[36878]: pgmap v5560: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 6 op/s 2024-03-24T03:00:11.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 03:00:11 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 9306.746939 secs 2024-03-24T03:00:13.945 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 03:00:13 smithi118 ceph-mon[31535]: pgmap v5561: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-03-24T03:00:13.953 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 03:00:13 smithi189 ceph-mon[36187]: pgmap v5561: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-03-24T03:00:13.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 03:00:13 smithi193 ceph-mon[36878]: pgmap v5561: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-03-24T03:00:15.945 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 03:00:15 smithi118 ceph-mon[31535]: pgmap v5562: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-03-24T03:00:15.953 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 03:00:15 smithi189 ceph-mon[36187]: pgmap v5562: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-03-24T03:00:15.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 03:00:15 smithi193 ceph-mon[36878]: pgmap v5562: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-03-24T03:00:16.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 03:00:16 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 9311.746994 secs 2024-03-24T03:00:16.953 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 03:00:16 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 9311.746994 secs 2024-03-24T03:00:16.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 03:00:16 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 9311.746994 secs 2024-03-24T03:00:17.945 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 03:00:17 smithi118 ceph-mon[31535]: pgmap v5563: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T03:00:17.953 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 03:00:17 smithi189 ceph-mon[36187]: pgmap v5563: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T03:00:17.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 03:00:17 smithi193 ceph-mon[36878]: pgmap v5563: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T03:00:19.945 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 03:00:19 smithi118 ceph-mon[31535]: pgmap v5564: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 5 op/s 2024-03-24T03:00:19.953 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 03:00:19 smithi189 ceph-mon[36187]: pgmap v5564: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 5 op/s 2024-03-24T03:00:19.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 03:00:19 smithi193 ceph-mon[36878]: pgmap v5564: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 5 op/s 2024-03-24T03:00:21.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 03:00:21 smithi118 ceph-mon[31535]: pgmap v5565: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T03:00:21.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 03:00:21 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 9316.747058 secs 2024-03-24T03:00:21.953 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 03:00:21 smithi189 ceph-mon[36187]: pgmap v5565: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T03:00:21.953 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 03:00:21 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 9316.747058 secs 2024-03-24T03:00:21.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 03:00:21 smithi193 ceph-mon[36878]: pgmap v5565: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T03:00:21.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 03:00:21 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 9316.747058 secs 2024-03-24T03:00:23.945 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 03:00:23 smithi118 ceph-mon[31535]: pgmap v5566: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:00:23.953 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 03:00:23 smithi189 ceph-mon[36187]: pgmap v5566: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:00:23.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 03:00:23 smithi193 ceph-mon[36878]: pgmap v5566: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:00:25.945 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 03:00:25 smithi118 ceph-mon[31535]: pgmap v5567: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T03:00:25.953 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 03:00:25 smithi189 ceph-mon[36187]: pgmap v5567: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T03:00:25.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 03:00:25 smithi193 ceph-mon[36878]: pgmap v5567: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T03:00:26.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 03:00:26 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 9321.747112 secs 2024-03-24T03:00:26.953 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 03:00:26 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 9321.747112 secs 2024-03-24T03:00:26.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 03:00:26 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 9321.747112 secs 2024-03-24T03:00:27.945 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 03:00:27 smithi118 ceph-mon[31535]: pgmap v5568: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T03:00:27.953 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 03:00:27 smithi189 ceph-mon[36187]: pgmap v5568: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T03:00:27.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 03:00:27 smithi193 ceph-mon[36878]: pgmap v5568: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T03:00:29.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 03:00:29 smithi118 ceph-mon[31535]: pgmap v5569: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 5 op/s 2024-03-24T03:00:29.953 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 03:00:29 smithi189 ceph-mon[36187]: pgmap v5569: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 5 op/s 2024-03-24T03:00:29.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 03:00:29 smithi193 ceph-mon[36878]: pgmap v5569: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 5 op/s 2024-03-24T03:00:31.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 03:00:31 smithi118 ceph-mon[31535]: pgmap v5570: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T03:00:31.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 03:00:31 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 9326.747194 secs 2024-03-24T03:00:31.953 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 03:00:31 smithi189 ceph-mon[36187]: pgmap v5570: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T03:00:31.953 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 03:00:31 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 9326.747194 secs 2024-03-24T03:00:31.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 03:00:31 smithi193 ceph-mon[36878]: pgmap v5570: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T03:00:31.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 03:00:31 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 9326.747194 secs 2024-03-24T03:00:33.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 03:00:33 smithi118 ceph-mon[31535]: pgmap v5571: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:00:33.953 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 03:00:33 smithi189 ceph-mon[36187]: pgmap v5571: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:00:33.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 03:00:33 smithi193 ceph-mon[36878]: pgmap v5571: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:00:35.945 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 03:00:35 smithi118 ceph-mon[31535]: pgmap v5572: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T03:00:35.953 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 03:00:35 smithi189 ceph-mon[36187]: pgmap v5572: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T03:00:35.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 03:00:35 smithi193 ceph-mon[36878]: pgmap v5572: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T03:00:36.945 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 03:00:36 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 9331.747278 secs 2024-03-24T03:00:36.953 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 03:00:36 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 9331.747278 secs 2024-03-24T03:00:36.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 03:00:36 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 9331.747278 secs 2024-03-24T03:00:37.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 03:00:37 smithi118 ceph-mon[31535]: pgmap v5573: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T03:00:37.953 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 03:00:37 smithi189 ceph-mon[36187]: pgmap v5573: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T03:00:37.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 03:00:37 smithi193 ceph-mon[36878]: pgmap v5573: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T03:00:39.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 03:00:39 smithi118 ceph-mon[31535]: pgmap v5574: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 5 op/s 2024-03-24T03:00:39.953 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 03:00:39 smithi189 ceph-mon[36187]: pgmap v5574: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 5 op/s 2024-03-24T03:00:39.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 03:00:39 smithi193 ceph-mon[36878]: pgmap v5574: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 5 op/s 2024-03-24T03:00:41.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 03:00:41 smithi118 ceph-mon[31535]: pgmap v5575: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T03:00:41.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 03:00:41 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 9336.747292 secs 2024-03-24T03:00:41.953 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 03:00:41 smithi189 ceph-mon[36187]: pgmap v5575: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T03:00:41.953 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 03:00:41 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 9336.747292 secs 2024-03-24T03:00:41.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 03:00:41 smithi193 ceph-mon[36878]: pgmap v5575: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T03:00:41.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 03:00:41 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 9336.747292 secs 2024-03-24T03:00:43.945 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 03:00:43 smithi118 ceph-mon[31535]: pgmap v5576: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:00:43.953 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 03:00:43 smithi189 ceph-mon[36187]: pgmap v5576: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:00:43.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 03:00:43 smithi193 ceph-mon[36878]: pgmap v5576: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:00:45.953 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 03:00:45 smithi189 ceph-mon[36187]: pgmap v5577: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T03:00:45.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 03:00:45 smithi193 ceph-mon[36878]: pgmap v5577: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T03:00:46.195 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 03:00:45 smithi118 ceph-mon[31535]: pgmap v5577: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T03:00:47.195 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 03:00:46 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 9341.747379 secs 2024-03-24T03:00:47.203 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 03:00:46 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 9341.747379 secs 2024-03-24T03:00:47.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 03:00:46 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 9341.747379 secs 2024-03-24T03:00:47.892 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 03:00:47 smithi118 ceph-mon[31535]: pgmap v5578: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T03:00:48.203 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 03:00:47 smithi189 ceph-mon[36187]: pgmap v5578: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T03:00:48.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 03:00:47 smithi193 ceph-mon[36878]: pgmap v5578: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T03:00:50.195 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 03:00:49 smithi118 ceph-mon[31535]: pgmap v5579: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 5 op/s 2024-03-24T03:00:50.203 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 03:00:49 smithi189 ceph-mon[36187]: pgmap v5579: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 5 op/s 2024-03-24T03:00:50.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 03:00:49 smithi193 ceph-mon[36878]: pgmap v5579: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 5 op/s 2024-03-24T03:00:51.195 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 03:00:50 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T03:00:51.203 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 03:00:50 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T03:00:51.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 03:00:50 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T03:00:52.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 03:00:51 smithi118 ceph-mon[31535]: pgmap v5580: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T03:00:52.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 03:00:51 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 9346.747447 secs 2024-03-24T03:00:52.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 03:00:51 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T03:00:52.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 03:00:51 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T03:00:52.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 03:00:51 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T03:00:52.203 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 03:00:51 smithi189 ceph-mon[36187]: pgmap v5580: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T03:00:52.203 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 03:00:51 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 9346.747447 secs 2024-03-24T03:00:52.203 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 03:00:51 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T03:00:52.203 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 03:00:51 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T03:00:52.203 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 03:00:51 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T03:00:52.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 03:00:51 smithi193 ceph-mon[36878]: pgmap v5580: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T03:00:52.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 03:00:51 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 9346.747447 secs 2024-03-24T03:00:52.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 03:00:51 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T03:00:52.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 03:00:51 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T03:00:52.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 03:00:51 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T03:00:53.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 03:00:52 smithi118 ceph-mon[31535]: pgmap v5581: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 5 op/s 2024-03-24T03:00:53.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 03:00:52 smithi118 ceph-mon[31535]: pgmap v5582: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 6 op/s 2024-03-24T03:00:53.203 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 03:00:52 smithi189 ceph-mon[36187]: pgmap v5581: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 5 op/s 2024-03-24T03:00:53.203 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 03:00:52 smithi189 ceph-mon[36187]: pgmap v5582: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 6 op/s 2024-03-24T03:00:53.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 03:00:52 smithi193 ceph-mon[36878]: pgmap v5581: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 5 op/s 2024-03-24T03:00:53.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 03:00:52 smithi193 ceph-mon[36878]: pgmap v5582: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 6 op/s 2024-03-24T03:00:55.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 03:00:54 smithi118 ceph-mon[31535]: pgmap v5583: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 6 op/s 2024-03-24T03:00:55.203 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 03:00:54 smithi189 ceph-mon[36187]: pgmap v5583: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 6 op/s 2024-03-24T03:00:55.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 03:00:54 smithi193 ceph-mon[36878]: pgmap v5583: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 6 op/s 2024-03-24T03:00:57.012 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 03:00:56 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 9351.747506 secs 2024-03-24T03:00:57.012 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 03:00:56 smithi193 ceph-mon[36878]: pgmap v5584: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 6 op/s 2024-03-24T03:00:57.195 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 03:00:56 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 9351.747506 secs 2024-03-24T03:00:57.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 03:00:56 smithi118 ceph-mon[31535]: pgmap v5584: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 6 op/s 2024-03-24T03:00:57.203 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 03:00:56 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 9351.747506 secs 2024-03-24T03:00:57.203 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 03:00:56 smithi189 ceph-mon[36187]: pgmap v5584: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 6 op/s 2024-03-24T03:00:59.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 03:00:58 smithi118 ceph-mon[31535]: pgmap v5585: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 5 op/s 2024-03-24T03:00:59.203 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 03:00:58 smithi189 ceph-mon[36187]: pgmap v5585: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 5 op/s 2024-03-24T03:00:59.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 03:00:58 smithi193 ceph-mon[36878]: pgmap v5585: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 5 op/s 2024-03-24T03:01:01.079 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 03:01:00 smithi118 ceph-mon[31535]: pgmap v5586: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 6 op/s 2024-03-24T03:01:01.203 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 03:01:00 smithi189 ceph-mon[36187]: pgmap v5586: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 6 op/s 2024-03-24T03:01:01.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 03:01:00 smithi193 ceph-mon[36878]: pgmap v5586: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 6 op/s 2024-03-24T03:01:02.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 03:01:01 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 9356.747566 secs 2024-03-24T03:01:02.203 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 03:01:01 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 9356.747566 secs 2024-03-24T03:01:02.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 03:01:01 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 9356.747566 secs 2024-03-24T03:01:03.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 03:01:02 smithi118 ceph-mon[31535]: pgmap v5587: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 5 op/s 2024-03-24T03:01:03.203 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 03:01:02 smithi189 ceph-mon[36187]: pgmap v5587: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 5 op/s 2024-03-24T03:01:03.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 03:01:02 smithi193 ceph-mon[36878]: pgmap v5587: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 5 op/s 2024-03-24T03:01:05.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 03:01:04 smithi118 ceph-mon[31535]: pgmap v5588: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T03:01:05.203 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 03:01:04 smithi189 ceph-mon[36187]: pgmap v5588: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T03:01:05.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 03:01:04 smithi193 ceph-mon[36878]: pgmap v5588: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T03:01:07.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 03:01:06 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 9361.747621 secs 2024-03-24T03:01:07.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 03:01:06 smithi118 ceph-mon[31535]: pgmap v5589: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T03:01:07.203 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 03:01:06 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 9361.747621 secs 2024-03-24T03:01:07.203 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 03:01:06 smithi189 ceph-mon[36187]: pgmap v5589: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T03:01:07.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 03:01:06 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 9361.747621 secs 2024-03-24T03:01:07.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 03:01:06 smithi193 ceph-mon[36878]: pgmap v5589: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T03:01:09.195 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 03:01:08 smithi118 ceph-mon[31535]: pgmap v5590: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T03:01:09.203 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 03:01:08 smithi189 ceph-mon[36187]: pgmap v5590: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T03:01:09.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 03:01:08 smithi193 ceph-mon[36878]: pgmap v5590: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T03:01:11.195 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 03:01:10 smithi118 ceph-mon[31535]: pgmap v5591: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-03-24T03:01:11.203 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 03:01:10 smithi189 ceph-mon[36187]: pgmap v5591: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-03-24T03:01:11.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 03:01:10 smithi193 ceph-mon[36878]: pgmap v5591: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-03-24T03:01:12.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 03:01:11 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 9366.747693 secs 2024-03-24T03:01:12.203 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 03:01:11 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 9366.747693 secs 2024-03-24T03:01:12.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 03:01:11 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 9366.747693 secs 2024-03-24T03:01:13.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 03:01:12 smithi118 ceph-mon[31535]: pgmap v5592: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:01:13.203 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 03:01:12 smithi189 ceph-mon[36187]: pgmap v5592: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:01:13.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 03:01:12 smithi193 ceph-mon[36878]: pgmap v5592: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:01:15.197 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 03:01:14 smithi118 ceph-mon[31535]: pgmap v5593: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T03:01:15.203 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 03:01:14 smithi189 ceph-mon[36187]: pgmap v5593: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T03:01:15.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 03:01:14 smithi193 ceph-mon[36878]: pgmap v5593: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T03:01:17.097 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 03:01:16 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 9371.747735 secs 2024-03-24T03:01:17.097 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 03:01:16 smithi118 ceph-mon[31535]: pgmap v5594: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 6 op/s 2024-03-24T03:01:17.203 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 03:01:16 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 9371.747735 secs 2024-03-24T03:01:17.203 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 03:01:16 smithi189 ceph-mon[36187]: pgmap v5594: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 6 op/s 2024-03-24T03:01:17.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 03:01:16 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 9371.747735 secs 2024-03-24T03:01:17.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 03:01:16 smithi193 ceph-mon[36878]: pgmap v5594: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 6 op/s 2024-03-24T03:01:17.252 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T03:01:17.252+0000 7f81db400640 1 -- 172.21.15.118:0/659686424 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f81c4006460 con 0x5622651a13f0 2024-03-24T03:01:17.252 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T03:01:17.253+0000 7f81e4c00640 1 -- 172.21.15.118:0/659686424 <== mon.1 v2:172.21.15.193:3300/0 75 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 409+0+0 (secure 0 0 0) 0x7f81c4006460 con 0x5622651a13f0 2024-03-24T03:01:17.252 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T03:01:17.253+0000 7f81ee000640 1 -- 172.21.15.118:0/1693767697 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f81c80036c0 con 0x7f81f81052d0 2024-03-24T03:01:17.253 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T03:01:17.253+0000 7f81ef400640 1 -- 172.21.15.118:0/1693767697 <== mon.2 v2:172.21.15.189:3300/0 75 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (secure 0 0 0) 0x7f81c80036c0 con 0x7f81f81052d0 2024-03-24T03:01:19.195 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 03:01:18 smithi118 ceph-mon[31535]: pgmap v5595: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 5 op/s 2024-03-24T03:01:19.203 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 03:01:18 smithi189 ceph-mon[36187]: pgmap v5595: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 5 op/s 2024-03-24T03:01:19.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 03:01:18 smithi193 ceph-mon[36878]: pgmap v5595: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 5 op/s 2024-03-24T03:01:21.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 03:01:20 smithi118 ceph-mon[31535]: pgmap v5596: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-03-24T03:01:21.203 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 03:01:20 smithi189 ceph-mon[36187]: pgmap v5596: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-03-24T03:01:21.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 03:01:20 smithi193 ceph-mon[36878]: pgmap v5596: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-03-24T03:01:22.195 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 03:01:21 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 9376.747792 secs 2024-03-24T03:01:22.203 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 03:01:21 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 9376.747792 secs 2024-03-24T03:01:22.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 03:01:21 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 9376.747792 secs 2024-03-24T03:01:23.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 03:01:22 smithi118 ceph-mon[31535]: pgmap v5597: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-03-24T03:01:23.203 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 03:01:22 smithi189 ceph-mon[36187]: pgmap v5597: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-03-24T03:01:23.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 03:01:22 smithi193 ceph-mon[36878]: pgmap v5597: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-03-24T03:01:25.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 03:01:24 smithi118 ceph-mon[31535]: pgmap v5598: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-03-24T03:01:25.203 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 03:01:24 smithi189 ceph-mon[36187]: pgmap v5598: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-03-24T03:01:25.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 03:01:24 smithi193 ceph-mon[36878]: pgmap v5598: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-03-24T03:01:27.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 03:01:26 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 9381.747880 secs 2024-03-24T03:01:27.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 03:01:26 smithi118 ceph-mon[31535]: pgmap v5599: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 6 op/s 2024-03-24T03:01:27.203 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 03:01:26 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 9381.747880 secs 2024-03-24T03:01:27.203 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 03:01:26 smithi189 ceph-mon[36187]: pgmap v5599: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 6 op/s 2024-03-24T03:01:27.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 03:01:26 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 9381.747880 secs 2024-03-24T03:01:27.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 03:01:26 smithi193 ceph-mon[36878]: pgmap v5599: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 6 op/s 2024-03-24T03:01:28.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 03:01:27 smithi118 ceph-mon[31535]: pgmap v5600: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T03:01:28.203 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 03:01:27 smithi189 ceph-mon[36187]: pgmap v5600: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T03:01:28.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 03:01:27 smithi193 ceph-mon[36878]: pgmap v5600: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T03:01:30.945 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 03:01:30 smithi118 ceph-mon[31535]: pgmap v5601: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-03-24T03:01:30.953 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 03:01:30 smithi189 ceph-mon[36187]: pgmap v5601: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-03-24T03:01:30.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 03:01:30 smithi193 ceph-mon[36878]: pgmap v5601: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-03-24T03:01:31.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 03:01:31 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 9386.747939 secs 2024-03-24T03:01:31.953 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 03:01:31 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 9386.747939 secs 2024-03-24T03:01:31.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 03:01:31 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 9386.747939 secs 2024-03-24T03:01:32.945 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 03:01:32 smithi118 ceph-mon[31535]: pgmap v5602: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:01:32.953 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 03:01:32 smithi189 ceph-mon[36187]: pgmap v5602: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:01:32.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 03:01:32 smithi193 ceph-mon[36878]: pgmap v5602: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:01:34.872 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 03:01:34 smithi118 ceph-mon[31535]: pgmap v5603: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T03:01:34.953 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 03:01:34 smithi189 ceph-mon[36187]: pgmap v5603: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T03:01:34.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 03:01:34 smithi193 ceph-mon[36878]: pgmap v5603: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T03:01:36.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 03:01:36 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 9391.747997 secs 2024-03-24T03:01:36.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 03:01:36 smithi118 ceph-mon[31535]: pgmap v5604: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T03:01:36.953 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 03:01:36 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 9391.747997 secs 2024-03-24T03:01:36.953 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 03:01:36 smithi189 ceph-mon[36187]: pgmap v5604: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T03:01:36.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 03:01:36 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 9391.747997 secs 2024-03-24T03:01:36.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 03:01:36 smithi193 ceph-mon[36878]: pgmap v5604: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T03:01:38.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 03:01:38 smithi118 ceph-mon[31535]: pgmap v5605: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T03:01:38.953 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 03:01:38 smithi189 ceph-mon[36187]: pgmap v5605: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T03:01:38.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 03:01:38 smithi193 ceph-mon[36878]: pgmap v5605: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T03:01:40.945 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 03:01:40 smithi118 ceph-mon[31535]: pgmap v5606: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-03-24T03:01:40.953 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 03:01:40 smithi189 ceph-mon[36187]: pgmap v5606: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-03-24T03:01:40.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 03:01:40 smithi193 ceph-mon[36878]: pgmap v5606: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-03-24T03:01:41.945 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 03:01:41 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 9396.748079 secs 2024-03-24T03:01:41.953 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 03:01:41 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 9396.748079 secs 2024-03-24T03:01:41.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 03:01:41 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 9396.748079 secs 2024-03-24T03:01:42.945 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 03:01:42 smithi118 ceph-mon[31535]: pgmap v5607: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:01:42.953 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 03:01:42 smithi189 ceph-mon[36187]: pgmap v5607: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:01:42.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 03:01:42 smithi193 ceph-mon[36878]: pgmap v5607: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:01:44.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 03:01:44 smithi118 ceph-mon[31535]: pgmap v5608: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T03:01:44.953 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 03:01:44 smithi189 ceph-mon[36187]: pgmap v5608: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T03:01:44.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 03:01:44 smithi193 ceph-mon[36878]: pgmap v5608: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T03:01:46.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 03:01:46 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 9401.748118 secs 2024-03-24T03:01:46.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 03:01:46 smithi118 ceph-mon[31535]: pgmap v5609: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T03:01:46.953 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 03:01:46 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 9401.748118 secs 2024-03-24T03:01:46.953 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 03:01:46 smithi189 ceph-mon[36187]: pgmap v5609: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T03:01:46.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 03:01:46 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 9401.748118 secs 2024-03-24T03:01:46.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 03:01:46 smithi193 ceph-mon[36878]: pgmap v5609: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T03:01:48.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 03:01:48 smithi118 ceph-mon[31535]: pgmap v5610: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T03:01:48.953 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 03:01:48 smithi189 ceph-mon[36187]: pgmap v5610: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T03:01:48.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 03:01:48 smithi193 ceph-mon[36878]: pgmap v5610: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T03:01:50.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 03:01:50 smithi118 ceph-mon[31535]: pgmap v5611: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-03-24T03:01:50.953 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 03:01:50 smithi189 ceph-mon[36187]: pgmap v5611: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-03-24T03:01:50.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 03:01:50 smithi193 ceph-mon[36878]: pgmap v5611: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-03-24T03:01:51.945 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 03:01:51 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 9406.748177 secs 2024-03-24T03:01:51.953 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 03:01:51 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 9406.748177 secs 2024-03-24T03:01:51.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 03:01:51 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 9406.748177 secs 2024-03-24T03:01:52.945 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 03:01:52 smithi118 ceph-mon[31535]: pgmap v5612: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:01:52.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 03:01:52 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T03:01:52.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 03:01:52 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T03:01:52.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 03:01:52 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T03:01:52.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 03:01:52 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T03:01:52.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 03:01:52 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T03:01:52.953 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 03:01:52 smithi189 ceph-mon[36187]: pgmap v5612: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:01:52.953 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 03:01:52 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T03:01:52.953 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 03:01:52 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T03:01:52.953 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 03:01:52 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T03:01:52.953 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 03:01:52 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T03:01:52.953 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 03:01:52 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T03:01:52.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 03:01:52 smithi193 ceph-mon[36878]: pgmap v5612: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:01:52.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 03:01:52 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T03:01:52.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 03:01:52 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T03:01:52.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 03:01:52 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T03:01:52.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 03:01:52 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T03:01:52.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 03:01:52 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T03:01:54.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 03:01:54 smithi118 ceph-mon[31535]: pgmap v5613: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T03:01:54.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 03:01:54 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T03:01:54.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 03:01:54 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T03:01:54.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 03:01:54 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T03:01:54.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 03:01:54 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T03:01:54.953 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 03:01:54 smithi189 ceph-mon[36187]: pgmap v5613: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T03:01:54.953 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 03:01:54 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T03:01:54.953 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 03:01:54 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T03:01:54.953 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 03:01:54 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T03:01:54.953 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 03:01:54 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T03:01:54.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 03:01:54 smithi193 ceph-mon[36878]: pgmap v5613: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T03:01:54.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 03:01:54 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T03:01:54.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 03:01:54 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T03:01:54.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 03:01:54 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T03:01:54.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 03:01:54 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T03:01:56.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 03:01:56 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 9411.748237 secs 2024-03-24T03:01:56.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 03:01:56 smithi118 ceph-mon[31535]: pgmap v5614: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T03:01:56.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 03:01:56 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T03:01:56.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 03:01:56 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T03:01:56.953 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 03:01:56 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 9411.748237 secs 2024-03-24T03:01:56.953 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 03:01:56 smithi189 ceph-mon[36187]: pgmap v5614: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T03:01:56.953 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 03:01:56 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T03:01:56.953 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 03:01:56 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T03:01:56.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 03:01:56 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 9411.748237 secs 2024-03-24T03:01:56.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 03:01:56 smithi193 ceph-mon[36878]: pgmap v5614: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T03:01:56.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 03:01:56 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T03:01:56.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 03:01:56 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T03:01:58.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 03:01:58 smithi118 ceph-mon[31535]: pgmap v5615: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T03:01:58.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 03:01:58 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T03:01:58.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 03:01:58 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T03:01:58.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 03:01:58 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T03:01:58.953 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 03:01:58 smithi189 ceph-mon[36187]: pgmap v5615: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T03:01:58.953 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 03:01:58 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T03:01:58.953 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 03:01:58 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T03:01:58.953 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 03:01:58 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T03:01:58.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 03:01:58 smithi193 ceph-mon[36878]: pgmap v5615: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T03:01:58.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 03:01:58 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T03:01:58.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 03:01:58 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T03:01:58.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 03:01:58 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T03:01:59.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 03:01:59 smithi118 ceph-mon[31535]: pgmap v5616: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 6 op/s 2024-03-24T03:01:59.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 03:01:59 smithi118 ceph-mon[31535]: pgmap v5617: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 4 op/s 2024-03-24T03:01:59.953 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 03:01:59 smithi189 ceph-mon[36187]: pgmap v5616: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 6 op/s 2024-03-24T03:01:59.953 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 03:01:59 smithi189 ceph-mon[36187]: pgmap v5617: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 4 op/s 2024-03-24T03:01:59.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 03:01:59 smithi193 ceph-mon[36878]: pgmap v5616: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 6 op/s 2024-03-24T03:01:59.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 03:01:59 smithi193 ceph-mon[36878]: pgmap v5617: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 4 op/s 2024-03-24T03:02:01.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 03:02:01 smithi118 ceph-mon[31535]: pgmap v5618: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-03-24T03:02:01.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 03:02:01 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 9416.748301 secs 2024-03-24T03:02:01.953 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 03:02:01 smithi189 ceph-mon[36187]: pgmap v5618: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-03-24T03:02:01.953 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 03:02:01 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 9416.748301 secs 2024-03-24T03:02:01.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 03:02:01 smithi193 ceph-mon[36878]: pgmap v5618: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-03-24T03:02:01.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 03:02:01 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 9416.748301 secs 2024-03-24T03:02:02.259 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T03:02:02.260+0000 7f81db400640 1 -- 172.21.15.118:0/659686424 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- auth(proto 2 132 bytes epoch 0) v1 -- 0x7f81c4006460 con 0x5622651a13f0 2024-03-24T03:02:02.260 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T03:02:02.260+0000 7f81ee000640 1 -- 172.21.15.118:0/1693767697 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- auth(proto 2 132 bytes epoch 0) v1 -- 0x7f81c80036c0 con 0x7f81f81052d0 2024-03-24T03:02:02.260 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T03:02:02.260+0000 7f81ef400640 1 -- 172.21.15.118:0/1693767697 <== mon.2 v2:172.21.15.189:3300/0 76 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 233+0+0 (secure 0 0 0) 0x7f81c80036c0 con 0x7f81f81052d0 2024-03-24T03:02:02.260 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T03:02:02.260+0000 7f81e4c00640 1 -- 172.21.15.118:0/659686424 <== mon.1 v2:172.21.15.193:3300/0 76 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 233+0+0 (secure 0 0 0) 0x7f81c4006460 con 0x5622651a13f0 2024-03-24T03:02:03.825 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 03:02:03 smithi118 ceph-mon[31535]: pgmap v5619: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 5 op/s 2024-03-24T03:02:03.953 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 03:02:03 smithi189 ceph-mon[36187]: pgmap v5619: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 5 op/s 2024-03-24T03:02:03.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 03:02:03 smithi193 ceph-mon[36878]: pgmap v5619: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 5 op/s 2024-03-24T03:02:05.945 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 03:02:05 smithi118 ceph-mon[31535]: pgmap v5620: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 5 op/s 2024-03-24T03:02:05.953 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 03:02:05 smithi189 ceph-mon[36187]: pgmap v5620: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 5 op/s 2024-03-24T03:02:05.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 03:02:05 smithi193 ceph-mon[36878]: pgmap v5620: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 5 op/s 2024-03-24T03:02:06.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 03:02:06 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 9421.748377 secs 2024-03-24T03:02:06.953 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 03:02:06 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 9421.748377 secs 2024-03-24T03:02:06.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 03:02:06 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 9421.748377 secs 2024-03-24T03:02:07.945 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 03:02:07 smithi118 ceph-mon[31535]: pgmap v5621: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 6 op/s 2024-03-24T03:02:07.953 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 03:02:07 smithi189 ceph-mon[36187]: pgmap v5621: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 6 op/s 2024-03-24T03:02:07.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 03:02:07 smithi193 ceph-mon[36878]: pgmap v5621: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 6 op/s 2024-03-24T03:02:09.945 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 03:02:09 smithi118 ceph-mon[31535]: pgmap v5622: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 5 op/s 2024-03-24T03:02:09.953 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 03:02:09 smithi189 ceph-mon[36187]: pgmap v5622: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 5 op/s 2024-03-24T03:02:09.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 03:02:09 smithi193 ceph-mon[36878]: pgmap v5622: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 5 op/s 2024-03-24T03:02:11.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 03:02:11 smithi118 ceph-mon[31535]: pgmap v5623: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T03:02:11.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 03:02:11 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 9426.748467 secs 2024-03-24T03:02:11.953 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 03:02:11 smithi189 ceph-mon[36187]: pgmap v5623: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T03:02:11.953 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 03:02:11 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 9426.748467 secs 2024-03-24T03:02:11.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 03:02:11 smithi193 ceph-mon[36878]: pgmap v5623: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T03:02:11.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 03:02:11 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 9426.748467 secs 2024-03-24T03:02:13.945 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 03:02:13 smithi118 ceph-mon[31535]: pgmap v5624: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:02:13.953 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 03:02:13 smithi189 ceph-mon[36187]: pgmap v5624: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:02:13.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 03:02:13 smithi193 ceph-mon[36878]: pgmap v5624: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:02:15.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 03:02:15 smithi118 ceph-mon[31535]: pgmap v5625: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T03:02:15.953 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 03:02:15 smithi189 ceph-mon[36187]: pgmap v5625: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T03:02:15.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 03:02:15 smithi193 ceph-mon[36878]: pgmap v5625: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T03:02:16.945 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 03:02:16 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 9431.748554 secs 2024-03-24T03:02:16.953 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 03:02:16 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 9431.748554 secs 2024-03-24T03:02:16.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 03:02:16 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 9431.748554 secs 2024-03-24T03:02:17.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 03:02:17 smithi118 ceph-mon[31535]: pgmap v5626: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T03:02:17.953 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 03:02:17 smithi189 ceph-mon[36187]: pgmap v5626: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T03:02:17.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 03:02:17 smithi193 ceph-mon[36878]: pgmap v5626: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T03:02:19.945 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 03:02:19 smithi118 ceph-mon[31535]: pgmap v5627: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 5 op/s 2024-03-24T03:02:19.953 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 03:02:19 smithi189 ceph-mon[36187]: pgmap v5627: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 5 op/s 2024-03-24T03:02:19.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 03:02:19 smithi193 ceph-mon[36878]: pgmap v5627: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 5 op/s 2024-03-24T03:02:21.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 03:02:21 smithi118 ceph-mon[31535]: pgmap v5628: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T03:02:21.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 03:02:21 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 9436.748644 secs 2024-03-24T03:02:21.953 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 03:02:21 smithi189 ceph-mon[36187]: pgmap v5628: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T03:02:21.953 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 03:02:21 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 9436.748644 secs 2024-03-24T03:02:21.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 03:02:21 smithi193 ceph-mon[36878]: pgmap v5628: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T03:02:21.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 03:02:21 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 9436.748644 secs 2024-03-24T03:02:23.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 03:02:23 smithi118 ceph-mon[31535]: pgmap v5629: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:02:23.953 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 03:02:23 smithi189 ceph-mon[36187]: pgmap v5629: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:02:23.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 03:02:23 smithi193 ceph-mon[36878]: pgmap v5629: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:02:25.953 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 03:02:25 smithi189 ceph-mon[36187]: pgmap v5630: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T03:02:25.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 03:02:25 smithi193 ceph-mon[36878]: pgmap v5630: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T03:02:26.195 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 03:02:25 smithi118 ceph-mon[31535]: pgmap v5630: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T03:02:26.953 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 03:02:26 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 9441.748732 secs 2024-03-24T03:02:26.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 03:02:26 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 9441.748732 secs 2024-03-24T03:02:27.195 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 03:02:26 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 9441.748732 secs 2024-03-24T03:02:28.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 03:02:27 smithi118 ceph-mon[31535]: pgmap v5631: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T03:02:28.203 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 03:02:27 smithi189 ceph-mon[36187]: pgmap v5631: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T03:02:28.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 03:02:27 smithi193 ceph-mon[36878]: pgmap v5631: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T03:02:30.005 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 03:02:29 smithi189 ceph-mon[36187]: pgmap v5632: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 5 op/s 2024-03-24T03:02:30.012 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 03:02:29 smithi193 ceph-mon[36878]: pgmap v5632: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 5 op/s 2024-03-24T03:02:30.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 03:02:29 smithi118 ceph-mon[31535]: pgmap v5632: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 5 op/s 2024-03-24T03:02:32.195 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 03:02:31 smithi118 ceph-mon[31535]: pgmap v5633: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T03:02:32.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 03:02:31 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 9446.748746 secs 2024-03-24T03:02:32.203 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 03:02:31 smithi189 ceph-mon[36187]: pgmap v5633: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T03:02:32.203 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 03:02:31 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 9446.748746 secs 2024-03-24T03:02:32.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 03:02:31 smithi193 ceph-mon[36878]: pgmap v5633: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T03:02:32.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 03:02:31 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 9446.748746 secs 2024-03-24T03:02:34.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 03:02:33 smithi118 ceph-mon[31535]: pgmap v5634: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:02:34.203 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 03:02:33 smithi189 ceph-mon[36187]: pgmap v5634: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:02:34.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 03:02:33 smithi193 ceph-mon[36878]: pgmap v5634: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:02:36.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 03:02:35 smithi118 ceph-mon[31535]: pgmap v5635: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T03:02:36.203 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 03:02:35 smithi189 ceph-mon[36187]: pgmap v5635: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T03:02:36.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 03:02:35 smithi193 ceph-mon[36878]: pgmap v5635: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T03:02:37.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 03:02:36 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 9451.748833 secs 2024-03-24T03:02:37.203 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 03:02:36 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 9451.748833 secs 2024-03-24T03:02:37.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 03:02:36 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 9451.748833 secs 2024-03-24T03:02:38.195 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 03:02:37 smithi118 ceph-mon[31535]: pgmap v5636: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T03:02:38.203 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 03:02:37 smithi189 ceph-mon[36187]: pgmap v5636: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T03:02:38.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 03:02:37 smithi193 ceph-mon[36878]: pgmap v5636: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T03:02:40.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 03:02:39 smithi118 ceph-mon[31535]: pgmap v5637: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 5 op/s 2024-03-24T03:02:40.203 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 03:02:39 smithi189 ceph-mon[36187]: pgmap v5637: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 5 op/s 2024-03-24T03:02:40.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 03:02:39 smithi193 ceph-mon[36878]: pgmap v5637: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 5 op/s 2024-03-24T03:02:42.195 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 03:02:41 smithi118 ceph-mon[31535]: pgmap v5638: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 6 op/s 2024-03-24T03:02:42.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 03:02:41 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 9456.748873 secs 2024-03-24T03:02:42.203 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 03:02:41 smithi189 ceph-mon[36187]: pgmap v5638: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 6 op/s 2024-03-24T03:02:42.203 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 03:02:41 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 9456.748873 secs 2024-03-24T03:02:42.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 03:02:41 smithi193 ceph-mon[36878]: pgmap v5638: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 6 op/s 2024-03-24T03:02:42.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 03:02:41 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 9456.748873 secs 2024-03-24T03:02:44.195 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 03:02:43 smithi118 ceph-mon[31535]: pgmap v5639: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-03-24T03:02:44.203 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 03:02:43 smithi189 ceph-mon[36187]: pgmap v5639: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-03-24T03:02:44.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 03:02:43 smithi193 ceph-mon[36878]: pgmap v5639: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-03-24T03:02:46.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 03:02:45 smithi118 ceph-mon[31535]: pgmap v5640: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-03-24T03:02:46.203 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 03:02:45 smithi189 ceph-mon[36187]: pgmap v5640: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-03-24T03:02:46.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 03:02:45 smithi193 ceph-mon[36878]: pgmap v5640: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-03-24T03:02:47.195 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 03:02:46 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 9461.748958 secs 2024-03-24T03:02:47.203 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 03:02:46 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 9461.748958 secs 2024-03-24T03:02:47.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 03:02:46 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 9461.748958 secs 2024-03-24T03:02:48.195 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 03:02:47 smithi118 ceph-mon[31535]: pgmap v5641: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 6 op/s 2024-03-24T03:02:48.203 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 03:02:47 smithi189 ceph-mon[36187]: pgmap v5641: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 6 op/s 2024-03-24T03:02:48.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 03:02:47 smithi193 ceph-mon[36878]: pgmap v5641: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 6 op/s 2024-03-24T03:02:50.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 03:02:49 smithi118 ceph-mon[31535]: pgmap v5642: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 5 op/s 2024-03-24T03:02:50.203 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 03:02:49 smithi189 ceph-mon[36187]: pgmap v5642: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 5 op/s 2024-03-24T03:02:50.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 03:02:49 smithi193 ceph-mon[36878]: pgmap v5642: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 5 op/s 2024-03-24T03:02:52.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 03:02:51 smithi118 ceph-mon[31535]: pgmap v5643: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T03:02:52.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 03:02:51 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 9466.749049 secs 2024-03-24T03:02:52.203 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 03:02:51 smithi189 ceph-mon[36187]: pgmap v5643: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T03:02:52.203 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 03:02:51 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 9466.749049 secs 2024-03-24T03:02:52.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 03:02:51 smithi193 ceph-mon[36878]: pgmap v5643: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T03:02:52.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 03:02:51 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 9466.749049 secs 2024-03-24T03:02:54.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 03:02:53 smithi118 ceph-mon[31535]: pgmap v5644: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:02:54.203 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 03:02:53 smithi189 ceph-mon[36187]: pgmap v5644: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:02:54.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 03:02:53 smithi193 ceph-mon[36878]: pgmap v5644: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:02:56.195 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 03:02:55 smithi118 ceph-mon[31535]: pgmap v5645: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T03:02:56.203 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 03:02:55 smithi189 ceph-mon[36187]: pgmap v5645: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T03:02:56.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 03:02:55 smithi193 ceph-mon[36878]: pgmap v5645: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T03:02:57.195 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 03:02:56 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 9471.749096 secs 2024-03-24T03:02:57.203 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 03:02:56 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 9471.749096 secs 2024-03-24T03:02:57.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 03:02:56 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 9471.749096 secs 2024-03-24T03:02:58.195 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 03:02:57 smithi118 ceph-mon[31535]: pgmap v5646: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T03:02:58.203 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 03:02:57 smithi189 ceph-mon[36187]: pgmap v5646: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T03:02:58.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 03:02:57 smithi193 ceph-mon[36878]: pgmap v5646: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T03:02:59.195 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 03:02:58 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T03:02:59.203 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 03:02:58 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T03:02:59.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 03:02:58 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T03:03:00.104 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 03:02:59 smithi189 ceph-mon[36187]: pgmap v5647: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 5 op/s 2024-03-24T03:03:00.104 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 03:02:59 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T03:03:00.104 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 03:02:59 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T03:03:00.105 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 03:02:59 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T03:03:00.105 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 03:02:59 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T03:03:00.105 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 03:02:59 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T03:03:00.111 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 03:02:59 smithi193 ceph-mon[36878]: pgmap v5647: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 5 op/s 2024-03-24T03:03:00.111 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 03:02:59 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T03:03:00.111 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 03:02:59 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T03:03:00.111 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 03:02:59 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T03:03:00.111 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 03:02:59 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T03:03:00.112 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 03:02:59 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T03:03:00.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 03:02:59 smithi118 ceph-mon[31535]: pgmap v5647: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 5 op/s 2024-03-24T03:03:00.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 03:02:59 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T03:03:00.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 03:02:59 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T03:03:00.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 03:02:59 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T03:03:00.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 03:02:59 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T03:03:00.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 03:02:59 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T03:03:01.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 03:03:00 smithi118 ceph-mon[31535]: pgmap v5648: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 6 op/s 2024-03-24T03:03:01.203 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 03:03:00 smithi189 ceph-mon[36187]: pgmap v5648: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 6 op/s 2024-03-24T03:03:01.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 03:03:00 smithi193 ceph-mon[36878]: pgmap v5648: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 6 op/s 2024-03-24T03:03:02.195 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 03:03:01 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 9476.749183 secs 2024-03-24T03:03:02.203 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 03:03:01 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 9476.749183 secs 2024-03-24T03:03:02.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 03:03:01 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 9476.749183 secs 2024-03-24T03:03:03.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 03:03:02 smithi118 ceph-mon[31535]: pgmap v5649: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 5 op/s 2024-03-24T03:03:03.203 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 03:03:02 smithi189 ceph-mon[36187]: pgmap v5649: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 5 op/s 2024-03-24T03:03:03.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 03:03:02 smithi193 ceph-mon[36878]: pgmap v5649: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 5 op/s 2024-03-24T03:03:05.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 03:03:04 smithi118 ceph-mon[31535]: pgmap v5650: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 6 op/s 2024-03-24T03:03:05.203 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 03:03:04 smithi189 ceph-mon[36187]: pgmap v5650: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 6 op/s 2024-03-24T03:03:05.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 03:03:04 smithi193 ceph-mon[36878]: pgmap v5650: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 6 op/s 2024-03-24T03:03:07.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 03:03:06 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 9481.749268 secs 2024-03-24T03:03:07.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 03:03:06 smithi118 ceph-mon[31535]: pgmap v5651: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 6 op/s 2024-03-24T03:03:07.203 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 03:03:06 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 9481.749268 secs 2024-03-24T03:03:07.203 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 03:03:06 smithi189 ceph-mon[36187]: pgmap v5651: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 6 op/s 2024-03-24T03:03:07.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 03:03:06 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 9481.749268 secs 2024-03-24T03:03:07.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 03:03:06 smithi193 ceph-mon[36878]: pgmap v5651: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 6 op/s 2024-03-24T03:03:09.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 03:03:08 smithi118 ceph-mon[31535]: pgmap v5652: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 5 op/s 2024-03-24T03:03:09.203 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 03:03:08 smithi189 ceph-mon[36187]: pgmap v5652: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 5 op/s 2024-03-24T03:03:09.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 03:03:08 smithi193 ceph-mon[36878]: pgmap v5652: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 5 op/s 2024-03-24T03:03:10.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 03:03:09 smithi118 ceph-mon[31535]: pgmap v5653: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 6 op/s 2024-03-24T03:03:10.203 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 03:03:09 smithi189 ceph-mon[36187]: pgmap v5653: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 6 op/s 2024-03-24T03:03:10.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 03:03:09 smithi193 ceph-mon[36878]: pgmap v5653: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 6 op/s 2024-03-24T03:03:11.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 03:03:11 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 9486.749331 secs 2024-03-24T03:03:11.453 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 03:03:11 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 9486.749331 secs 2024-03-24T03:03:11.460 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 03:03:11 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 9486.749331 secs 2024-03-24T03:03:12.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 03:03:12 smithi118 ceph-mon[31535]: pgmap v5654: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:03:12.453 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 03:03:12 smithi189 ceph-mon[36187]: pgmap v5654: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:03:12.460 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 03:03:12 smithi193 ceph-mon[36878]: pgmap v5654: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:03:14.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 03:03:14 smithi118 ceph-mon[31535]: pgmap v5655: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T03:03:14.953 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 03:03:14 smithi189 ceph-mon[36187]: pgmap v5655: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T03:03:14.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 03:03:14 smithi193 ceph-mon[36878]: pgmap v5655: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T03:03:16.945 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 03:03:16 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 9491.749390 secs 2024-03-24T03:03:16.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 03:03:16 smithi118 ceph-mon[31535]: pgmap v5656: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T03:03:16.953 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 03:03:16 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 9491.749390 secs 2024-03-24T03:03:16.953 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 03:03:16 smithi189 ceph-mon[36187]: pgmap v5656: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T03:03:16.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 03:03:16 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 9491.749390 secs 2024-03-24T03:03:16.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 03:03:16 smithi193 ceph-mon[36878]: pgmap v5656: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T03:03:18.945 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 03:03:18 smithi118 ceph-mon[31535]: pgmap v5657: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T03:03:18.953 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 03:03:18 smithi189 ceph-mon[36187]: pgmap v5657: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T03:03:18.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 03:03:18 smithi193 ceph-mon[36878]: pgmap v5657: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T03:03:20.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 03:03:20 smithi118 ceph-mon[31535]: pgmap v5658: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-03-24T03:03:20.953 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 03:03:20 smithi189 ceph-mon[36187]: pgmap v5658: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-03-24T03:03:20.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 03:03:20 smithi193 ceph-mon[36878]: pgmap v5658: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-03-24T03:03:21.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 03:03:21 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 9496.749479 secs 2024-03-24T03:03:21.953 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 03:03:21 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 9496.749479 secs 2024-03-24T03:03:21.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 03:03:21 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 9496.749479 secs 2024-03-24T03:03:22.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 03:03:22 smithi118 ceph-mon[31535]: pgmap v5659: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:03:22.953 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 03:03:22 smithi189 ceph-mon[36187]: pgmap v5659: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:03:22.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 03:03:22 smithi193 ceph-mon[36878]: pgmap v5659: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:03:24.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 03:03:24 smithi118 ceph-mon[31535]: pgmap v5660: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T03:03:24.953 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 03:03:24 smithi189 ceph-mon[36187]: pgmap v5660: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T03:03:24.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 03:03:24 smithi193 ceph-mon[36878]: pgmap v5660: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T03:03:26.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 03:03:26 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 9501.749569 secs 2024-03-24T03:03:26.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 03:03:26 smithi118 ceph-mon[31535]: pgmap v5661: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T03:03:26.953 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 03:03:26 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 9501.749569 secs 2024-03-24T03:03:26.953 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 03:03:26 smithi189 ceph-mon[36187]: pgmap v5661: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T03:03:26.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 03:03:26 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 9501.749569 secs 2024-03-24T03:03:26.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 03:03:26 smithi193 ceph-mon[36878]: pgmap v5661: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T03:03:28.945 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 03:03:28 smithi118 ceph-mon[31535]: pgmap v5662: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T03:03:28.953 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 03:03:28 smithi189 ceph-mon[36187]: pgmap v5662: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T03:03:28.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 03:03:28 smithi193 ceph-mon[36878]: pgmap v5662: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T03:03:30.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 03:03:30 smithi118 ceph-mon[31535]: pgmap v5663: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-03-24T03:03:30.953 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 03:03:30 smithi189 ceph-mon[36187]: pgmap v5663: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-03-24T03:03:30.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 03:03:30 smithi193 ceph-mon[36878]: pgmap v5663: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-03-24T03:03:31.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 03:03:31 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 9506.749657 secs 2024-03-24T03:03:31.953 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 03:03:31 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 9506.749657 secs 2024-03-24T03:03:31.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 03:03:31 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 9506.749657 secs 2024-03-24T03:03:32.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 03:03:32 smithi118 ceph-mon[31535]: pgmap v5664: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:03:32.953 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 03:03:32 smithi189 ceph-mon[36187]: pgmap v5664: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:03:32.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 03:03:32 smithi193 ceph-mon[36878]: pgmap v5664: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:03:34.945 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 03:03:34 smithi118 ceph-mon[31535]: pgmap v5665: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T03:03:34.953 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 03:03:34 smithi189 ceph-mon[36187]: pgmap v5665: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T03:03:34.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 03:03:34 smithi193 ceph-mon[36878]: pgmap v5665: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T03:03:36.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 03:03:36 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 9511.749740 secs 2024-03-24T03:03:36.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 03:03:36 smithi118 ceph-mon[31535]: pgmap v5666: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T03:03:36.953 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 03:03:36 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 9511.749740 secs 2024-03-24T03:03:36.953 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 03:03:36 smithi189 ceph-mon[36187]: pgmap v5666: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T03:03:36.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 03:03:36 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 9511.749740 secs 2024-03-24T03:03:36.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 03:03:36 smithi193 ceph-mon[36878]: pgmap v5666: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T03:03:38.945 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 03:03:38 smithi118 ceph-mon[31535]: pgmap v5667: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T03:03:38.953 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 03:03:38 smithi189 ceph-mon[36187]: pgmap v5667: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T03:03:38.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 03:03:38 smithi193 ceph-mon[36878]: pgmap v5667: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T03:03:40.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 03:03:40 smithi118 ceph-mon[31535]: pgmap v5668: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-03-24T03:03:40.953 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 03:03:40 smithi189 ceph-mon[36187]: pgmap v5668: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-03-24T03:03:40.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 03:03:40 smithi193 ceph-mon[36878]: pgmap v5668: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-03-24T03:03:41.945 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 03:03:41 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 9516.749754 secs 2024-03-24T03:03:41.953 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 03:03:41 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 9516.749754 secs 2024-03-24T03:03:41.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 03:03:41 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 9516.749754 secs 2024-03-24T03:03:42.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 03:03:42 smithi118 ceph-mon[31535]: pgmap v5669: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:03:42.953 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 03:03:42 smithi189 ceph-mon[36187]: pgmap v5669: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:03:42.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 03:03:42 smithi193 ceph-mon[36878]: pgmap v5669: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:03:44.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 03:03:44 smithi118 ceph-mon[31535]: pgmap v5670: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T03:03:44.953 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 03:03:44 smithi189 ceph-mon[36187]: pgmap v5670: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T03:03:44.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 03:03:44 smithi193 ceph-mon[36878]: pgmap v5670: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T03:03:46.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 03:03:46 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 9521.749794 secs 2024-03-24T03:03:46.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 03:03:46 smithi118 ceph-mon[31535]: pgmap v5671: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T03:03:46.953 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 03:03:46 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 9521.749794 secs 2024-03-24T03:03:46.953 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 03:03:46 smithi189 ceph-mon[36187]: pgmap v5671: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T03:03:46.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 03:03:46 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 9521.749794 secs 2024-03-24T03:03:46.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 03:03:46 smithi193 ceph-mon[36878]: pgmap v5671: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T03:03:48.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 03:03:48 smithi118 ceph-mon[31535]: pgmap v5672: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T03:03:48.953 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 03:03:48 smithi189 ceph-mon[36187]: pgmap v5672: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T03:03:48.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 03:03:48 smithi193 ceph-mon[36878]: pgmap v5672: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T03:03:50.945 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 03:03:50 smithi118 ceph-mon[31535]: pgmap v5673: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-03-24T03:03:50.953 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 03:03:50 smithi189 ceph-mon[36187]: pgmap v5673: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-03-24T03:03:50.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 03:03:50 smithi193 ceph-mon[36878]: pgmap v5673: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-03-24T03:03:51.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 03:03:51 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 9526.749883 secs 2024-03-24T03:03:51.953 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 03:03:51 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 9526.749883 secs 2024-03-24T03:03:51.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 03:03:51 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 9526.749883 secs 2024-03-24T03:03:52.945 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 03:03:52 smithi118 ceph-mon[31535]: pgmap v5674: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:03:52.953 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 03:03:52 smithi189 ceph-mon[36187]: pgmap v5674: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:03:52.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 03:03:52 smithi193 ceph-mon[36878]: pgmap v5674: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:03:54.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 03:03:54 smithi118 ceph-mon[31535]: pgmap v5675: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T03:03:54.953 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 03:03:54 smithi189 ceph-mon[36187]: pgmap v5675: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T03:03:54.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 03:03:54 smithi193 ceph-mon[36878]: pgmap v5675: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T03:03:56.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 03:03:56 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 9531.749970 secs 2024-03-24T03:03:56.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 03:03:56 smithi118 ceph-mon[31535]: pgmap v5676: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T03:03:56.953 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 03:03:56 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 9531.749970 secs 2024-03-24T03:03:56.953 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 03:03:56 smithi189 ceph-mon[36187]: pgmap v5676: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T03:03:56.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 03:03:56 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 9531.749970 secs 2024-03-24T03:03:56.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 03:03:56 smithi193 ceph-mon[36878]: pgmap v5676: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T03:03:58.945 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 03:03:58 smithi118 ceph-mon[31535]: pgmap v5677: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T03:03:58.953 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 03:03:58 smithi189 ceph-mon[36187]: pgmap v5677: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T03:03:58.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 03:03:58 smithi193 ceph-mon[36878]: pgmap v5677: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T03:04:00.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 03:04:00 smithi118 ceph-mon[31535]: pgmap v5678: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-03-24T03:04:00.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 03:04:00 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T03:04:00.953 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 03:04:00 smithi189 ceph-mon[36187]: pgmap v5678: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-03-24T03:04:00.953 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 03:04:00 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T03:04:00.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 03:04:00 smithi193 ceph-mon[36878]: pgmap v5678: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-03-24T03:04:00.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 03:04:00 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T03:04:01.953 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 03:04:01 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 9536.750066 secs 2024-03-24T03:04:01.953 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 03:04:01 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T03:04:01.953 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 03:04:01 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T03:04:01.953 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 03:04:01 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T03:04:01.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 03:04:01 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 9536.750066 secs 2024-03-24T03:04:01.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 03:04:01 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T03:04:01.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 03:04:01 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T03:04:01.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 03:04:01 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T03:04:02.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 03:04:01 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 9536.750066 secs 2024-03-24T03:04:02.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 03:04:01 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T03:04:02.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 03:04:01 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T03:04:02.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 03:04:01 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T03:04:02.953 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 03:04:02 smithi189 ceph-mon[36187]: pgmap v5679: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 5 op/s 2024-03-24T03:04:02.953 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 03:04:02 smithi189 ceph-mon[36187]: pgmap v5680: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-03-24T03:04:02.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 03:04:02 smithi193 ceph-mon[36878]: pgmap v5679: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 5 op/s 2024-03-24T03:04:02.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 03:04:02 smithi193 ceph-mon[36878]: pgmap v5680: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-03-24T03:04:03.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 03:04:02 smithi118 ceph-mon[31535]: pgmap v5679: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 5 op/s 2024-03-24T03:04:03.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 03:04:02 smithi118 ceph-mon[31535]: pgmap v5680: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-03-24T03:04:04.953 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 03:04:04 smithi189 ceph-mon[36187]: pgmap v5681: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 5 op/s 2024-03-24T03:04:04.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 03:04:04 smithi193 ceph-mon[36878]: pgmap v5681: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 5 op/s 2024-03-24T03:04:05.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 03:04:04 smithi118 ceph-mon[31535]: pgmap v5681: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 5 op/s 2024-03-24T03:04:06.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 03:04:06 smithi193 ceph-mon[36878]: pgmap v5682: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-03-24T03:04:06.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 03:04:06 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 9541.750150 secs 2024-03-24T03:04:07.195 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 03:04:06 smithi118 ceph-mon[31535]: pgmap v5682: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-03-24T03:04:07.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 03:04:06 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 9541.750150 secs 2024-03-24T03:04:07.203 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 03:04:06 smithi189 ceph-mon[36187]: pgmap v5682: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-03-24T03:04:07.203 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 03:04:06 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 9541.750150 secs 2024-03-24T03:04:09.195 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 03:04:08 smithi118 ceph-mon[31535]: pgmap v5683: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 5 op/s 2024-03-24T03:04:09.203 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 03:04:08 smithi189 ceph-mon[36187]: pgmap v5683: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 5 op/s 2024-03-24T03:04:09.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 03:04:08 smithi193 ceph-mon[36878]: pgmap v5683: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 5 op/s 2024-03-24T03:04:11.031 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 03:04:10 smithi118 ceph-mon[31535]: pgmap v5684: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 5 op/s 2024-03-24T03:04:11.203 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 03:04:10 smithi189 ceph-mon[36187]: pgmap v5684: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 5 op/s 2024-03-24T03:04:11.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 03:04:10 smithi193 ceph-mon[36878]: pgmap v5684: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 5 op/s 2024-03-24T03:04:12.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 03:04:11 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 9546.750209 secs 2024-03-24T03:04:12.203 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 03:04:11 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 9546.750209 secs 2024-03-24T03:04:12.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 03:04:11 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 9546.750209 secs 2024-03-24T03:04:13.195 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 03:04:12 smithi118 ceph-mon[31535]: pgmap v5685: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 5 op/s 2024-03-24T03:04:13.203 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 03:04:12 smithi189 ceph-mon[36187]: pgmap v5685: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 5 op/s 2024-03-24T03:04:13.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 03:04:12 smithi193 ceph-mon[36878]: pgmap v5685: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 5 op/s 2024-03-24T03:04:15.195 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 03:04:14 smithi118 ceph-mon[31535]: pgmap v5686: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 5 op/s 2024-03-24T03:04:15.203 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 03:04:14 smithi189 ceph-mon[36187]: pgmap v5686: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 5 op/s 2024-03-24T03:04:15.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 03:04:14 smithi193 ceph-mon[36878]: pgmap v5686: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 5 op/s 2024-03-24T03:04:17.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 03:04:16 smithi118 ceph-mon[31535]: pgmap v5687: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T03:04:17.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 03:04:16 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 9551.750270 secs 2024-03-24T03:04:17.203 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 03:04:16 smithi189 ceph-mon[36187]: pgmap v5687: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T03:04:17.203 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 03:04:16 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 9551.750270 secs 2024-03-24T03:04:17.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 03:04:16 smithi193 ceph-mon[36878]: pgmap v5687: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T03:04:17.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 03:04:16 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 9551.750270 secs 2024-03-24T03:04:17.281 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T03:04:17.281+0000 7f81ee000640 1 -- 172.21.15.118:0/1693767697 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f81c80036c0 con 0x7f81f81052d0 2024-03-24T03:04:17.281 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T03:04:17.281+0000 7f81db400640 1 -- 172.21.15.118:0/659686424 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f81c4006460 con 0x5622651a13f0 2024-03-24T03:04:17.282 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T03:04:17.281+0000 7f81ef400640 1 -- 172.21.15.118:0/1693767697 <== mon.2 v2:172.21.15.189:3300/0 77 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (secure 0 0 0) 0x7f81c80036c0 con 0x7f81f81052d0 2024-03-24T03:04:17.282 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T03:04:17.281+0000 7f81e4c00640 1 -- 172.21.15.118:0/659686424 <== mon.1 v2:172.21.15.193:3300/0 77 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 409+0+0 (secure 0 0 0) 0x7f81c4006460 con 0x5622651a13f0 2024-03-24T03:04:19.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 03:04:18 smithi118 ceph-mon[31535]: pgmap v5688: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:04:19.203 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 03:04:18 smithi189 ceph-mon[36187]: pgmap v5688: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:04:19.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 03:04:18 smithi193 ceph-mon[36878]: pgmap v5688: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:04:20.998 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 03:04:20 smithi118 ceph-mon[31535]: pgmap v5689: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T03:04:21.203 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 03:04:20 smithi189 ceph-mon[36187]: pgmap v5689: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T03:04:21.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 03:04:20 smithi193 ceph-mon[36878]: pgmap v5689: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T03:04:22.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 03:04:21 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 9556.750357 secs 2024-03-24T03:04:22.203 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 03:04:21 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 9556.750357 secs 2024-03-24T03:04:22.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 03:04:21 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 9556.750357 secs 2024-03-24T03:04:23.195 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 03:04:22 smithi118 ceph-mon[31535]: pgmap v5690: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T03:04:23.203 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 03:04:22 smithi189 ceph-mon[36187]: pgmap v5690: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T03:04:23.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 03:04:22 smithi193 ceph-mon[36878]: pgmap v5690: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T03:04:25.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 03:04:24 smithi118 ceph-mon[31535]: pgmap v5691: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 5 op/s 2024-03-24T03:04:25.203 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 03:04:24 smithi189 ceph-mon[36187]: pgmap v5691: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 5 op/s 2024-03-24T03:04:25.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 03:04:24 smithi193 ceph-mon[36878]: pgmap v5691: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 5 op/s 2024-03-24T03:04:27.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 03:04:26 smithi118 ceph-mon[31535]: pgmap v5692: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T03:04:27.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 03:04:26 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 9561.750419 secs 2024-03-24T03:04:27.203 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 03:04:26 smithi189 ceph-mon[36187]: pgmap v5692: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T03:04:27.203 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 03:04:26 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 9561.750419 secs 2024-03-24T03:04:27.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 03:04:26 smithi193 ceph-mon[36878]: pgmap v5692: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T03:04:27.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 03:04:26 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 9561.750419 secs 2024-03-24T03:04:29.195 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 03:04:28 smithi118 ceph-mon[31535]: pgmap v5693: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:04:29.203 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 03:04:28 smithi189 ceph-mon[36187]: pgmap v5693: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:04:29.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 03:04:28 smithi193 ceph-mon[36878]: pgmap v5693: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:04:31.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 03:04:30 smithi118 ceph-mon[31535]: pgmap v5694: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T03:04:31.203 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 03:04:30 smithi189 ceph-mon[36187]: pgmap v5694: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T03:04:31.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 03:04:30 smithi193 ceph-mon[36878]: pgmap v5694: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T03:04:32.195 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 03:04:31 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 9566.750435 secs 2024-03-24T03:04:32.203 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 03:04:31 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 9566.750435 secs 2024-03-24T03:04:32.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 03:04:31 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 9566.750435 secs 2024-03-24T03:04:33.195 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 03:04:32 smithi118 ceph-mon[31535]: pgmap v5695: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T03:04:33.203 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 03:04:32 smithi189 ceph-mon[36187]: pgmap v5695: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T03:04:33.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 03:04:32 smithi193 ceph-mon[36878]: pgmap v5695: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T03:04:35.195 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 03:04:34 smithi118 ceph-mon[31535]: pgmap v5696: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 5 op/s 2024-03-24T03:04:35.203 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 03:04:34 smithi189 ceph-mon[36187]: pgmap v5696: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 5 op/s 2024-03-24T03:04:35.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 03:04:34 smithi193 ceph-mon[36878]: pgmap v5696: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 5 op/s 2024-03-24T03:04:37.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 03:04:36 smithi118 ceph-mon[31535]: pgmap v5697: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T03:04:37.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 03:04:36 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 9571.750475 secs 2024-03-24T03:04:37.203 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 03:04:36 smithi189 ceph-mon[36187]: pgmap v5697: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T03:04:37.203 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 03:04:36 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 9571.750475 secs 2024-03-24T03:04:37.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 03:04:36 smithi193 ceph-mon[36878]: pgmap v5697: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T03:04:37.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 03:04:36 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 9571.750475 secs 2024-03-24T03:04:39.192 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 03:04:38 smithi118 ceph-mon[31535]: pgmap v5698: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:04:39.203 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 03:04:38 smithi189 ceph-mon[36187]: pgmap v5698: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:04:39.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 03:04:38 smithi193 ceph-mon[36878]: pgmap v5698: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:04:41.195 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 03:04:40 smithi118 ceph-mon[31535]: pgmap v5699: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T03:04:41.203 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 03:04:40 smithi189 ceph-mon[36187]: pgmap v5699: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T03:04:41.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 03:04:40 smithi193 ceph-mon[36878]: pgmap v5699: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T03:04:42.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 03:04:41 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 9576.750562 secs 2024-03-24T03:04:42.203 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 03:04:41 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 9576.750562 secs 2024-03-24T03:04:42.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 03:04:41 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 9576.750562 secs 2024-03-24T03:04:43.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 03:04:42 smithi118 ceph-mon[31535]: pgmap v5700: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T03:04:43.203 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 03:04:42 smithi189 ceph-mon[36187]: pgmap v5700: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T03:04:43.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 03:04:42 smithi193 ceph-mon[36878]: pgmap v5700: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T03:04:45.195 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 03:04:44 smithi118 ceph-mon[31535]: pgmap v5701: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 5 op/s 2024-03-24T03:04:45.203 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 03:04:44 smithi189 ceph-mon[36187]: pgmap v5701: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 5 op/s 2024-03-24T03:04:45.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 03:04:44 smithi193 ceph-mon[36878]: pgmap v5701: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 5 op/s 2024-03-24T03:04:47.195 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 03:04:46 smithi118 ceph-mon[31535]: pgmap v5702: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T03:04:47.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 03:04:46 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 9581.750603 secs 2024-03-24T03:04:47.203 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 03:04:46 smithi189 ceph-mon[36187]: pgmap v5702: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T03:04:47.203 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 03:04:46 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 9581.750603 secs 2024-03-24T03:04:47.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 03:04:46 smithi193 ceph-mon[36878]: pgmap v5702: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T03:04:47.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 03:04:46 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 9581.750603 secs 2024-03-24T03:04:49.195 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 03:04:48 smithi118 ceph-mon[31535]: pgmap v5703: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:04:49.203 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 03:04:48 smithi189 ceph-mon[36187]: pgmap v5703: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:04:49.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 03:04:48 smithi193 ceph-mon[36878]: pgmap v5703: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:04:51.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 03:04:50 smithi118 ceph-mon[31535]: pgmap v5704: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T03:04:51.203 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 03:04:50 smithi189 ceph-mon[36187]: pgmap v5704: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T03:04:51.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 03:04:50 smithi193 ceph-mon[36878]: pgmap v5704: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T03:04:52.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 03:04:51 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 9586.750693 secs 2024-03-24T03:04:52.203 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 03:04:51 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 9586.750693 secs 2024-03-24T03:04:52.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 03:04:51 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 9586.750693 secs 2024-03-24T03:04:53.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 03:04:52 smithi118 ceph-mon[31535]: pgmap v5705: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T03:04:53.203 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 03:04:52 smithi189 ceph-mon[36187]: pgmap v5705: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T03:04:53.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 03:04:52 smithi193 ceph-mon[36878]: pgmap v5705: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T03:04:55.195 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 03:04:54 smithi118 ceph-mon[31535]: pgmap v5706: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 5 op/s 2024-03-24T03:04:55.203 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 03:04:54 smithi189 ceph-mon[36187]: pgmap v5706: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 5 op/s 2024-03-24T03:04:55.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 03:04:54 smithi193 ceph-mon[36878]: pgmap v5706: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 5 op/s 2024-03-24T03:04:57.120 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 03:04:56 smithi118 ceph-mon[31535]: pgmap v5707: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T03:04:57.120 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 03:04:56 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 9591.750735 secs 2024-03-24T03:04:57.203 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 03:04:56 smithi189 ceph-mon[36187]: pgmap v5707: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T03:04:57.203 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 03:04:56 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 9591.750735 secs 2024-03-24T03:04:57.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 03:04:56 smithi193 ceph-mon[36878]: pgmap v5707: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T03:04:57.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 03:04:56 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 9591.750735 secs 2024-03-24T03:04:59.195 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 03:04:58 smithi118 ceph-mon[31535]: pgmap v5708: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:04:59.203 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 03:04:58 smithi189 ceph-mon[36187]: pgmap v5708: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:04:59.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 03:04:58 smithi193 ceph-mon[36878]: pgmap v5708: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:05:01.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 03:05:00 smithi118 ceph-mon[31535]: pgmap v5709: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T03:05:01.203 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 03:05:00 smithi189 ceph-mon[36187]: pgmap v5709: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T03:05:01.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 03:05:00 smithi193 ceph-mon[36878]: pgmap v5709: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T03:05:02.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 03:05:01 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 9596.750822 secs 2024-03-24T03:05:02.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 03:05:01 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T03:05:02.203 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 03:05:01 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 9596.750822 secs 2024-03-24T03:05:02.203 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 03:05:01 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T03:05:02.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 03:05:01 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 9596.750822 secs 2024-03-24T03:05:02.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 03:05:01 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T03:05:03.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 03:05:02 smithi118 ceph-mon[31535]: pgmap v5710: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T03:05:03.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 03:05:02 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T03:05:03.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 03:05:02 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T03:05:03.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 03:05:02 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T03:05:03.203 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 03:05:02 smithi189 ceph-mon[36187]: pgmap v5710: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T03:05:03.203 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 03:05:02 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T03:05:03.203 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 03:05:02 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T03:05:03.203 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 03:05:02 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T03:05:03.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 03:05:02 smithi193 ceph-mon[36878]: pgmap v5710: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T03:05:03.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 03:05:02 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T03:05:03.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 03:05:02 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T03:05:03.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 03:05:02 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T03:05:04.195 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 03:05:03 smithi118 ceph-mon[31535]: pgmap v5711: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 5 op/s 2024-03-24T03:05:04.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 03:05:03 smithi118 ceph-mon[31535]: pgmap v5712: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 5 op/s 2024-03-24T03:05:04.203 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 03:05:03 smithi189 ceph-mon[36187]: pgmap v5711: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 5 op/s 2024-03-24T03:05:04.203 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 03:05:03 smithi189 ceph-mon[36187]: pgmap v5712: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 5 op/s 2024-03-24T03:05:04.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 03:05:03 smithi193 ceph-mon[36878]: pgmap v5711: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 5 op/s 2024-03-24T03:05:04.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 03:05:03 smithi193 ceph-mon[36878]: pgmap v5712: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 5 op/s 2024-03-24T03:05:06.195 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 03:05:05 smithi118 ceph-mon[31535]: pgmap v5713: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-03-24T03:05:06.203 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 03:05:05 smithi189 ceph-mon[36187]: pgmap v5713: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-03-24T03:05:06.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 03:05:05 smithi193 ceph-mon[36878]: pgmap v5713: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-03-24T03:05:07.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 03:05:06 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 9601.750884 secs 2024-03-24T03:05:07.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 03:05:06 smithi118 ceph-mon[31535]: pgmap v5714: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-03-24T03:05:07.203 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 03:05:06 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 9601.750884 secs 2024-03-24T03:05:07.203 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 03:05:06 smithi189 ceph-mon[36187]: pgmap v5714: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-03-24T03:05:07.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 03:05:06 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 9601.750884 secs 2024-03-24T03:05:07.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 03:05:06 smithi193 ceph-mon[36878]: pgmap v5714: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-03-24T03:05:09.914 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 03:05:09 smithi118 ceph-mon[31535]: pgmap v5715: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 5 op/s 2024-03-24T03:05:09.953 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 03:05:09 smithi189 ceph-mon[36187]: pgmap v5715: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 5 op/s 2024-03-24T03:05:09.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 03:05:09 smithi193 ceph-mon[36878]: pgmap v5715: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 5 op/s 2024-03-24T03:05:11.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 03:05:11 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 9606.750974 secs 2024-03-24T03:05:11.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 03:05:11 smithi118 ceph-mon[31535]: pgmap v5716: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-03-24T03:05:11.953 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 03:05:11 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 9606.750974 secs 2024-03-24T03:05:11.953 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 03:05:11 smithi189 ceph-mon[36187]: pgmap v5716: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-03-24T03:05:11.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 03:05:11 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 9606.750974 secs 2024-03-24T03:05:11.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 03:05:11 smithi193 ceph-mon[36878]: pgmap v5716: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-03-24T03:05:13.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 03:05:13 smithi118 ceph-mon[31535]: pgmap v5717: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 5 op/s 2024-03-24T03:05:13.703 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 03:05:13 smithi189 ceph-mon[36187]: pgmap v5717: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 5 op/s 2024-03-24T03:05:13.710 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 03:05:13 smithi193 ceph-mon[36878]: pgmap v5717: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 5 op/s 2024-03-24T03:05:15.945 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 03:05:15 smithi118 ceph-mon[31535]: pgmap v5718: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T03:05:15.953 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 03:05:15 smithi189 ceph-mon[36187]: pgmap v5718: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T03:05:15.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 03:05:15 smithi193 ceph-mon[36878]: pgmap v5718: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T03:05:16.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 03:05:16 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 9611.751033 secs 2024-03-24T03:05:16.953 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 03:05:16 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 9611.751033 secs 2024-03-24T03:05:16.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 03:05:16 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 9611.751033 secs 2024-03-24T03:05:17.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 03:05:17 smithi118 ceph-mon[31535]: pgmap v5719: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:05:17.953 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 03:05:17 smithi189 ceph-mon[36187]: pgmap v5719: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:05:17.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 03:05:17 smithi193 ceph-mon[36878]: pgmap v5719: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:05:19.945 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 03:05:19 smithi118 ceph-mon[31535]: pgmap v5720: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T03:05:19.953 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 03:05:19 smithi189 ceph-mon[36187]: pgmap v5720: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T03:05:19.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 03:05:19 smithi193 ceph-mon[36878]: pgmap v5720: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T03:05:21.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 03:05:21 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 9616.751120 secs 2024-03-24T03:05:21.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 03:05:21 smithi118 ceph-mon[31535]: pgmap v5721: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T03:05:21.953 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 03:05:21 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 9616.751120 secs 2024-03-24T03:05:21.953 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 03:05:21 smithi189 ceph-mon[36187]: pgmap v5721: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T03:05:21.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 03:05:21 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 9616.751120 secs 2024-03-24T03:05:21.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 03:05:21 smithi193 ceph-mon[36878]: pgmap v5721: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T03:05:23.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 03:05:23 smithi118 ceph-mon[31535]: pgmap v5722: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T03:05:23.953 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 03:05:23 smithi189 ceph-mon[36187]: pgmap v5722: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T03:05:23.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 03:05:23 smithi193 ceph-mon[36878]: pgmap v5722: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T03:05:25.945 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 03:05:25 smithi118 ceph-mon[31535]: pgmap v5723: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-03-24T03:05:25.953 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 03:05:25 smithi189 ceph-mon[36187]: pgmap v5723: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-03-24T03:05:25.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 03:05:25 smithi193 ceph-mon[36878]: pgmap v5723: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-03-24T03:05:26.945 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 03:05:26 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 9621.751204 secs 2024-03-24T03:05:26.953 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 03:05:26 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 9621.751204 secs 2024-03-24T03:05:26.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 03:05:26 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 9621.751204 secs 2024-03-24T03:05:27.945 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 03:05:27 smithi118 ceph-mon[31535]: pgmap v5724: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:05:27.953 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 03:05:27 smithi189 ceph-mon[36187]: pgmap v5724: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:05:27.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 03:05:27 smithi193 ceph-mon[36878]: pgmap v5724: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:05:29.945 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 03:05:29 smithi118 ceph-mon[31535]: pgmap v5725: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T03:05:29.953 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 03:05:29 smithi189 ceph-mon[36187]: pgmap v5725: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T03:05:29.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 03:05:29 smithi193 ceph-mon[36878]: pgmap v5725: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T03:05:31.945 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 03:05:31 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 9626.751287 secs 2024-03-24T03:05:31.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 03:05:31 smithi118 ceph-mon[31535]: pgmap v5726: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T03:05:31.953 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 03:05:31 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 9626.751287 secs 2024-03-24T03:05:31.953 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 03:05:31 smithi189 ceph-mon[36187]: pgmap v5726: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T03:05:31.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 03:05:31 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 9626.751287 secs 2024-03-24T03:05:31.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 03:05:31 smithi193 ceph-mon[36878]: pgmap v5726: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T03:05:33.945 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 03:05:33 smithi118 ceph-mon[31535]: pgmap v5727: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T03:05:33.953 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 03:05:33 smithi189 ceph-mon[36187]: pgmap v5727: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T03:05:33.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 03:05:33 smithi193 ceph-mon[36878]: pgmap v5727: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T03:05:35.945 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 03:05:35 smithi118 ceph-mon[31535]: pgmap v5728: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-03-24T03:05:35.953 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 03:05:35 smithi189 ceph-mon[36187]: pgmap v5728: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-03-24T03:05:35.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 03:05:35 smithi193 ceph-mon[36878]: pgmap v5728: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-03-24T03:05:36.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 03:05:36 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 9631.751400 secs 2024-03-24T03:05:36.953 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 03:05:36 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 9631.751400 secs 2024-03-24T03:05:36.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 03:05:36 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 9631.751400 secs 2024-03-24T03:05:37.945 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 03:05:37 smithi118 ceph-mon[31535]: pgmap v5729: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:05:37.953 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 03:05:37 smithi189 ceph-mon[36187]: pgmap v5729: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:05:37.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 03:05:37 smithi193 ceph-mon[36878]: pgmap v5729: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:05:39.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 03:05:39 smithi118 ceph-mon[31535]: pgmap v5730: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T03:05:39.953 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 03:05:39 smithi189 ceph-mon[36187]: pgmap v5730: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T03:05:39.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 03:05:39 smithi193 ceph-mon[36878]: pgmap v5730: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T03:05:41.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 03:05:41 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 9636.751488 secs 2024-03-24T03:05:41.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 03:05:41 smithi118 ceph-mon[31535]: pgmap v5731: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T03:05:41.953 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 03:05:41 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 9636.751488 secs 2024-03-24T03:05:41.953 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 03:05:41 smithi189 ceph-mon[36187]: pgmap v5731: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T03:05:41.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 03:05:41 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 9636.751488 secs 2024-03-24T03:05:41.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 03:05:41 smithi193 ceph-mon[36878]: pgmap v5731: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T03:05:43.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 03:05:43 smithi118 ceph-mon[31535]: pgmap v5732: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T03:05:43.953 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 03:05:43 smithi189 ceph-mon[36187]: pgmap v5732: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T03:05:43.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 03:05:43 smithi193 ceph-mon[36878]: pgmap v5732: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T03:05:45.945 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 03:05:45 smithi118 ceph-mon[31535]: pgmap v5733: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-03-24T03:05:45.953 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 03:05:45 smithi189 ceph-mon[36187]: pgmap v5733: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-03-24T03:05:45.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 03:05:45 smithi193 ceph-mon[36878]: pgmap v5733: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-03-24T03:05:46.953 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 03:05:46 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 9641.751550 secs 2024-03-24T03:05:46.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 03:05:46 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 9641.751550 secs 2024-03-24T03:05:47.195 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 03:05:46 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 9641.751550 secs 2024-03-24T03:05:47.953 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 03:05:47 smithi189 ceph-mon[36187]: pgmap v5734: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:05:47.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 03:05:47 smithi193 ceph-mon[36878]: pgmap v5734: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:05:48.195 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 03:05:47 smithi118 ceph-mon[31535]: pgmap v5734: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:05:49.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 03:05:49 smithi193 ceph-mon[36878]: pgmap v5735: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T03:05:50.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 03:05:49 smithi118 ceph-mon[31535]: pgmap v5735: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T03:05:50.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 03:05:49 smithi189 ceph-mon[36187]: pgmap v5735: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T03:05:52.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 03:05:51 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 9646.751639 secs 2024-03-24T03:05:52.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 03:05:51 smithi118 ceph-mon[31535]: pgmap v5736: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T03:05:52.203 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 03:05:51 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 9646.751639 secs 2024-03-24T03:05:52.203 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 03:05:51 smithi189 ceph-mon[36187]: pgmap v5736: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T03:05:52.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 03:05:51 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 9646.751639 secs 2024-03-24T03:05:52.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 03:05:51 smithi193 ceph-mon[36878]: pgmap v5736: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T03:05:53.998 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 03:05:53 smithi118 ceph-mon[31535]: pgmap v5737: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T03:05:54.203 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 03:05:53 smithi189 ceph-mon[36187]: pgmap v5737: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T03:05:54.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 03:05:53 smithi193 ceph-mon[36878]: pgmap v5737: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T03:05:56.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 03:05:55 smithi118 ceph-mon[31535]: pgmap v5738: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-03-24T03:05:56.203 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 03:05:55 smithi189 ceph-mon[36187]: pgmap v5738: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-03-24T03:05:56.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 03:05:55 smithi193 ceph-mon[36878]: pgmap v5738: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-03-24T03:05:57.015 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 03:05:56 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 9651.751727 secs 2024-03-24T03:05:57.203 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 03:05:56 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 9651.751727 secs 2024-03-24T03:05:57.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 03:05:56 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 9651.751727 secs 2024-03-24T03:05:58.195 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 03:05:57 smithi118 ceph-mon[31535]: pgmap v5739: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:05:58.203 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 03:05:57 smithi189 ceph-mon[36187]: pgmap v5739: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:05:58.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 03:05:57 smithi193 ceph-mon[36878]: pgmap v5739: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:06:00.088 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 03:05:59 smithi118 ceph-mon[31535]: pgmap v5740: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T03:06:00.203 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 03:05:59 smithi189 ceph-mon[36187]: pgmap v5740: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T03:06:00.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 03:05:59 smithi193 ceph-mon[36878]: pgmap v5740: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T03:06:02.005 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 03:06:01 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 9656.751787 secs 2024-03-24T03:06:02.005 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 03:06:01 smithi189 ceph-mon[36187]: pgmap v5741: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T03:06:02.011 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 03:06:01 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 9656.751787 secs 2024-03-24T03:06:02.011 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 03:06:01 smithi193 ceph-mon[36878]: pgmap v5741: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T03:06:02.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 03:06:01 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 9656.751787 secs 2024-03-24T03:06:02.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 03:06:01 smithi118 ceph-mon[31535]: pgmap v5741: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T03:06:04.195 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 03:06:03 smithi118 ceph-mon[31535]: pgmap v5742: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T03:06:04.195 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 03:06:03 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T03:06:04.203 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 03:06:03 smithi189 ceph-mon[36187]: pgmap v5742: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T03:06:04.203 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 03:06:03 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T03:06:04.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 03:06:03 smithi193 ceph-mon[36878]: pgmap v5742: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T03:06:04.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 03:06:03 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T03:06:06.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 03:06:05 smithi118 ceph-mon[31535]: pgmap v5743: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-03-24T03:06:06.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 03:06:05 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T03:06:06.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 03:06:05 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T03:06:06.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 03:06:05 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T03:06:06.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 03:06:05 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T03:06:06.197 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 03:06:05 smithi118 ceph-mon[31535]: pgmap v5744: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T03:06:06.197 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 03:06:05 smithi118 ceph-mon[31535]: pgmap v5745: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-03-24T03:06:06.197 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 03:06:05 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T03:06:06.203 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 03:06:05 smithi189 ceph-mon[36187]: pgmap v5743: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-03-24T03:06:06.203 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 03:06:05 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T03:06:06.203 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 03:06:05 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T03:06:06.203 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 03:06:05 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T03:06:06.203 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 03:06:05 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T03:06:06.203 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 03:06:05 smithi189 ceph-mon[36187]: pgmap v5744: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T03:06:06.203 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 03:06:05 smithi189 ceph-mon[36187]: pgmap v5745: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-03-24T03:06:06.203 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 03:06:05 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T03:06:06.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 03:06:05 smithi193 ceph-mon[36878]: pgmap v5743: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-03-24T03:06:06.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 03:06:05 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T03:06:06.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 03:06:05 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T03:06:06.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 03:06:05 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T03:06:06.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 03:06:05 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T03:06:06.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 03:06:05 smithi193 ceph-mon[36878]: pgmap v5744: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T03:06:06.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 03:06:05 smithi193 ceph-mon[36878]: pgmap v5745: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.6 KiB/s rd, 7 op/s 2024-03-24T03:06:06.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 03:06:05 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T03:06:07.195 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 03:06:06 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 9661.751873 secs 2024-03-24T03:06:07.203 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 03:06:06 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 9661.751873 secs 2024-03-24T03:06:07.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 03:06:06 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 9661.751873 secs 2024-03-24T03:06:08.195 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 03:06:07 smithi118 ceph-mon[31535]: pgmap v5746: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 5 op/s 2024-03-24T03:06:08.203 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 03:06:07 smithi189 ceph-mon[36187]: pgmap v5746: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 5 op/s 2024-03-24T03:06:08.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 03:06:07 smithi193 ceph-mon[36878]: pgmap v5746: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 5 op/s 2024-03-24T03:06:09.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 03:06:08 smithi193 ceph-mon[36878]: pgmap v5747: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 5 op/s 2024-03-24T03:06:09.445 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 03:06:08 smithi118 ceph-mon[31535]: pgmap v5747: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 5 op/s 2024-03-24T03:06:09.453 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 03:06:08 smithi189 ceph-mon[36187]: pgmap v5747: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 5 op/s 2024-03-24T03:06:11.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 03:06:10 smithi193 ceph-mon[36878]: pgmap v5748: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 6 op/s 2024-03-24T03:06:11.445 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 03:06:10 smithi118 ceph-mon[31535]: pgmap v5748: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 6 op/s 2024-03-24T03:06:11.453 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 03:06:10 smithi189 ceph-mon[36187]: pgmap v5748: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 6 op/s 2024-03-24T03:06:12.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 03:06:11 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 9666.751975 secs 2024-03-24T03:06:12.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 03:06:11 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 9666.751975 secs 2024-03-24T03:06:12.453 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 03:06:11 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 9666.751975 secs 2024-03-24T03:06:13.445 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 03:06:12 smithi118 ceph-mon[31535]: pgmap v5749: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 4 op/s 2024-03-24T03:06:13.453 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 03:06:12 smithi189 ceph-mon[36187]: pgmap v5749: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 4 op/s 2024-03-24T03:06:13.460 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 03:06:12 smithi193 ceph-mon[36878]: pgmap v5749: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 4 op/s 2024-03-24T03:06:15.195 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 03:06:14 smithi118 ceph-mon[31535]: pgmap v5750: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 5 op/s 2024-03-24T03:06:15.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 03:06:14 smithi193 ceph-mon[36878]: pgmap v5750: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 5 op/s 2024-03-24T03:06:15.453 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 03:06:14 smithi189 ceph-mon[36187]: pgmap v5750: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 5 op/s 2024-03-24T03:06:16.445 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 03:06:16 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 9671.752036 secs 2024-03-24T03:06:16.453 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 03:06:16 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 9671.752036 secs 2024-03-24T03:06:16.460 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 03:06:16 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 9671.752036 secs 2024-03-24T03:06:17.445 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 03:06:17 smithi118 ceph-mon[31535]: pgmap v5751: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:06:17.453 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 03:06:17 smithi189 ceph-mon[36187]: pgmap v5751: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:06:17.460 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 03:06:17 smithi193 ceph-mon[36878]: pgmap v5751: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:06:19.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 03:06:18 smithi193 ceph-mon[36878]: pgmap v5752: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T03:06:19.445 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 03:06:18 smithi118 ceph-mon[31535]: pgmap v5752: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T03:06:19.453 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 03:06:18 smithi189 ceph-mon[36187]: pgmap v5752: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T03:06:21.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 03:06:20 smithi193 ceph-mon[36878]: pgmap v5753: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T03:06:21.445 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 03:06:20 smithi118 ceph-mon[31535]: pgmap v5753: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T03:06:21.453 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 03:06:20 smithi189 ceph-mon[36187]: pgmap v5753: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T03:06:22.445 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 03:06:21 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 9676.752121 secs 2024-03-24T03:06:22.453 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 03:06:21 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 9676.752121 secs 2024-03-24T03:06:22.460 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 03:06:21 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 9676.752121 secs 2024-03-24T03:06:23.347 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 03:06:22 smithi118 ceph-mon[31535]: pgmap v5754: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 5 op/s 2024-03-24T03:06:23.453 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 03:06:22 smithi189 ceph-mon[36187]: pgmap v5754: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 5 op/s 2024-03-24T03:06:23.460 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 03:06:22 smithi193 ceph-mon[36878]: pgmap v5754: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 5 op/s 2024-03-24T03:06:25.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 03:06:24 smithi193 ceph-mon[36878]: pgmap v5755: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T03:06:25.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 03:06:24 smithi118 ceph-mon[31535]: pgmap v5755: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T03:06:25.453 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 03:06:24 smithi189 ceph-mon[36187]: pgmap v5755: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T03:06:26.445 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 03:06:26 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 9681.752184 secs 2024-03-24T03:06:26.453 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 03:06:26 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 9681.752184 secs 2024-03-24T03:06:26.460 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 03:06:26 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 9681.752184 secs 2024-03-24T03:06:27.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 03:06:27 smithi118 ceph-mon[31535]: pgmap v5756: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:06:27.453 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 03:06:27 smithi189 ceph-mon[36187]: pgmap v5756: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:06:27.460 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 03:06:27 smithi193 ceph-mon[36878]: pgmap v5756: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:06:29.445 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 03:06:28 smithi118 ceph-mon[31535]: pgmap v5757: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T03:06:29.453 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 03:06:28 smithi189 ceph-mon[36187]: pgmap v5757: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T03:06:29.460 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 03:06:28 smithi193 ceph-mon[36878]: pgmap v5757: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T03:06:31.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 03:06:30 smithi118 ceph-mon[31535]: pgmap v5758: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T03:06:31.453 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 03:06:30 smithi189 ceph-mon[36187]: pgmap v5758: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T03:06:31.460 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 03:06:30 smithi193 ceph-mon[36878]: pgmap v5758: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T03:06:32.354 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 03:06:31 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 9686.752200 secs 2024-03-24T03:06:32.361 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 03:06:31 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 9686.752200 secs 2024-03-24T03:06:32.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 03:06:31 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 9686.752200 secs 2024-03-24T03:06:33.445 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 03:06:32 smithi118 ceph-mon[31535]: pgmap v5759: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 5 op/s 2024-03-24T03:06:33.453 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 03:06:32 smithi189 ceph-mon[36187]: pgmap v5759: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 5 op/s 2024-03-24T03:06:33.460 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 03:06:32 smithi193 ceph-mon[36878]: pgmap v5759: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 5 op/s 2024-03-24T03:06:35.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 03:06:34 smithi118 ceph-mon[31535]: pgmap v5760: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T03:06:35.453 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 03:06:34 smithi189 ceph-mon[36187]: pgmap v5760: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T03:06:35.460 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 03:06:34 smithi193 ceph-mon[36878]: pgmap v5760: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T03:06:36.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 03:06:36 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 9691.752263 secs 2024-03-24T03:06:36.453 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 03:06:36 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 9691.752263 secs 2024-03-24T03:06:36.460 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 03:06:36 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 9691.752263 secs 2024-03-24T03:06:37.445 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 03:06:37 smithi118 ceph-mon[31535]: pgmap v5761: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:06:37.453 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 03:06:37 smithi189 ceph-mon[36187]: pgmap v5761: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:06:37.460 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 03:06:37 smithi193 ceph-mon[36878]: pgmap v5761: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:06:39.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 03:06:38 smithi118 ceph-mon[31535]: pgmap v5762: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T03:06:39.453 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 03:06:38 smithi189 ceph-mon[36187]: pgmap v5762: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T03:06:39.460 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 03:06:38 smithi193 ceph-mon[36878]: pgmap v5762: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T03:06:41.445 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 03:06:40 smithi118 ceph-mon[31535]: pgmap v5763: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T03:06:41.453 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 03:06:40 smithi189 ceph-mon[36187]: pgmap v5763: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T03:06:41.460 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 03:06:40 smithi193 ceph-mon[36878]: pgmap v5763: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T03:06:42.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 03:06:41 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 9696.752352 secs 2024-03-24T03:06:42.453 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 03:06:41 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 9696.752352 secs 2024-03-24T03:06:42.460 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 03:06:41 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 9696.752352 secs 2024-03-24T03:06:43.348 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 03:06:42 smithi118 ceph-mon[31535]: pgmap v5764: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 5 op/s 2024-03-24T03:06:43.453 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 03:06:42 smithi189 ceph-mon[36187]: pgmap v5764: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 5 op/s 2024-03-24T03:06:43.460 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 03:06:42 smithi193 ceph-mon[36878]: pgmap v5764: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 5 op/s 2024-03-24T03:06:45.445 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 03:06:44 smithi118 ceph-mon[31535]: pgmap v5765: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T03:06:45.453 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 03:06:44 smithi189 ceph-mon[36187]: pgmap v5765: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T03:06:45.460 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 03:06:44 smithi193 ceph-mon[36878]: pgmap v5765: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T03:06:46.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 03:06:46 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 9701.752417 secs 2024-03-24T03:06:46.453 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 03:06:46 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 9701.752417 secs 2024-03-24T03:06:46.460 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 03:06:46 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 9701.752417 secs 2024-03-24T03:06:47.445 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 03:06:47 smithi118 ceph-mon[31535]: pgmap v5766: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:06:47.453 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 03:06:47 smithi189 ceph-mon[36187]: pgmap v5766: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:06:47.460 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 03:06:47 smithi193 ceph-mon[36878]: pgmap v5766: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:06:49.445 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 03:06:48 smithi118 ceph-mon[31535]: pgmap v5767: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T03:06:49.453 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 03:06:48 smithi189 ceph-mon[36187]: pgmap v5767: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T03:06:49.460 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 03:06:48 smithi193 ceph-mon[36878]: pgmap v5767: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T03:06:51.445 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 03:06:50 smithi118 ceph-mon[31535]: pgmap v5768: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T03:06:51.453 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 03:06:50 smithi189 ceph-mon[36187]: pgmap v5768: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T03:06:51.460 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 03:06:50 smithi193 ceph-mon[36878]: pgmap v5768: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T03:06:52.445 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 03:06:51 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 9706.752479 secs 2024-03-24T03:06:52.453 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 03:06:51 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 9706.752479 secs 2024-03-24T03:06:52.460 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 03:06:51 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 9706.752479 secs 2024-03-24T03:06:53.347 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 03:06:52 smithi118 ceph-mon[31535]: pgmap v5769: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 5 op/s 2024-03-24T03:06:53.453 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 03:06:52 smithi189 ceph-mon[36187]: pgmap v5769: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 5 op/s 2024-03-24T03:06:53.460 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 03:06:52 smithi193 ceph-mon[36878]: pgmap v5769: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 5 op/s 2024-03-24T03:06:55.445 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 03:06:54 smithi118 ceph-mon[31535]: pgmap v5770: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T03:06:55.453 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 03:06:54 smithi189 ceph-mon[36187]: pgmap v5770: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T03:06:55.460 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 03:06:54 smithi193 ceph-mon[36878]: pgmap v5770: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T03:06:56.445 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 03:06:56 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 9711.752541 secs 2024-03-24T03:06:56.453 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 03:06:56 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 9711.752541 secs 2024-03-24T03:06:56.460 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 03:06:56 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 9711.752541 secs 2024-03-24T03:06:57.445 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 03:06:57 smithi118 ceph-mon[31535]: pgmap v5771: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:06:57.453 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 03:06:57 smithi189 ceph-mon[36187]: pgmap v5771: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:06:57.460 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 03:06:57 smithi193 ceph-mon[36878]: pgmap v5771: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:06:59.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 03:06:58 smithi118 ceph-mon[31535]: pgmap v5772: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T03:06:59.453 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 03:06:58 smithi189 ceph-mon[36187]: pgmap v5772: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T03:06:59.460 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 03:06:58 smithi193 ceph-mon[36878]: pgmap v5772: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T03:07:01.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 03:07:00 smithi118 ceph-mon[31535]: pgmap v5773: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T03:07:01.453 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 03:07:00 smithi189 ceph-mon[36187]: pgmap v5773: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T03:07:01.460 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 03:07:00 smithi193 ceph-mon[36878]: pgmap v5773: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T03:07:02.354 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 03:07:01 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 9716.752601 secs 2024-03-24T03:07:02.361 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 03:07:01 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 9716.752601 secs 2024-03-24T03:07:02.445 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 03:07:01 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 9716.752601 secs 2024-03-24T03:07:03.445 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 03:07:02 smithi118 ceph-mon[31535]: pgmap v5774: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 5 op/s 2024-03-24T03:07:03.453 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 03:07:02 smithi189 ceph-mon[36187]: pgmap v5774: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 5 op/s 2024-03-24T03:07:03.460 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 03:07:02 smithi193 ceph-mon[36878]: pgmap v5774: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 5 op/s 2024-03-24T03:07:05.445 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 03:07:04 smithi118 ceph-mon[31535]: pgmap v5775: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T03:07:05.453 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 03:07:04 smithi189 ceph-mon[36187]: pgmap v5775: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T03:07:05.460 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 03:07:04 smithi193 ceph-mon[36878]: pgmap v5775: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T03:07:06.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 03:07:05 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T03:07:06.453 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 03:07:05 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T03:07:06.460 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 03:07:05 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T03:07:07.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 03:07:06 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 9721.752669 secs 2024-03-24T03:07:07.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 03:07:06 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T03:07:07.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 03:07:06 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T03:07:07.447 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 03:07:06 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T03:07:07.447 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 03:07:06 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T03:07:07.447 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 03:07:06 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T03:07:07.447 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 03:07:06 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T03:07:07.447 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 03:07:06 smithi118 ceph-mon[31535]: pgmap v5776: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 5 op/s 2024-03-24T03:07:07.447 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 03:07:06 smithi118 ceph-mon[31535]: pgmap v5777: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 6 op/s 2024-03-24T03:07:07.447 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 03:07:06 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T03:07:07.453 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 03:07:06 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 9721.752669 secs 2024-03-24T03:07:07.453 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 03:07:06 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T03:07:07.453 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 03:07:06 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T03:07:07.453 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 03:07:06 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T03:07:07.453 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 03:07:06 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T03:07:07.453 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 03:07:06 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T03:07:07.453 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 03:07:06 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T03:07:07.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 03:07:06 smithi189 ceph-mon[36187]: pgmap v5776: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 5 op/s 2024-03-24T03:07:07.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 03:07:06 smithi189 ceph-mon[36187]: pgmap v5777: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 6 op/s 2024-03-24T03:07:07.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 03:07:06 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T03:07:07.460 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 03:07:06 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 9721.752669 secs 2024-03-24T03:07:07.460 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 03:07:06 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T03:07:07.460 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 03:07:06 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T03:07:07.460 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 03:07:06 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T03:07:07.460 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 03:07:06 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T03:07:07.460 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 03:07:06 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T03:07:07.460 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 03:07:06 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T03:07:07.460 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 03:07:06 smithi193 ceph-mon[36878]: pgmap v5776: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 5 op/s 2024-03-24T03:07:07.460 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 03:07:06 smithi193 ceph-mon[36878]: pgmap v5777: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 6 op/s 2024-03-24T03:07:07.460 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 03:07:06 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T03:07:09.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 03:07:09 smithi118 ceph-mon[31535]: pgmap v5778: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 6 op/s 2024-03-24T03:07:09.953 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 03:07:09 smithi189 ceph-mon[36187]: pgmap v5778: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 6 op/s 2024-03-24T03:07:09.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 03:07:09 smithi193 ceph-mon[36878]: pgmap v5778: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 6 op/s 2024-03-24T03:07:11.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 03:07:11 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 9726.752757 secs 2024-03-24T03:07:11.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 03:07:11 smithi118 ceph-mon[31535]: pgmap v5779: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 6 op/s 2024-03-24T03:07:11.953 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 03:07:11 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 9726.752757 secs 2024-03-24T03:07:11.953 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 03:07:11 smithi189 ceph-mon[36187]: pgmap v5779: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 6 op/s 2024-03-24T03:07:11.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 03:07:11 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 9726.752757 secs 2024-03-24T03:07:11.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 03:07:11 smithi193 ceph-mon[36878]: pgmap v5779: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 6 op/s 2024-03-24T03:07:13.945 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 03:07:13 smithi118 ceph-mon[31535]: pgmap v5780: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 5 op/s 2024-03-24T03:07:13.953 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 03:07:13 smithi189 ceph-mon[36187]: pgmap v5780: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 5 op/s 2024-03-24T03:07:13.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 03:07:13 smithi193 ceph-mon[36878]: pgmap v5780: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 5 op/s 2024-03-24T03:07:15.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 03:07:15 smithi118 ceph-mon[31535]: pgmap v5781: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 6 op/s 2024-03-24T03:07:15.953 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 03:07:15 smithi189 ceph-mon[36187]: pgmap v5781: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 6 op/s 2024-03-24T03:07:15.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 03:07:15 smithi193 ceph-mon[36878]: pgmap v5781: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 6 op/s 2024-03-24T03:07:16.888 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 03:07:16 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 9731.752795 secs 2024-03-24T03:07:16.953 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 03:07:16 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 9731.752795 secs 2024-03-24T03:07:16.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 03:07:16 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 9731.752795 secs 2024-03-24T03:07:17.311 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T03:07:17.312+0000 7f81ee000640 1 -- 172.21.15.118:0/1693767697 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f81c80036c0 con 0x7f81f81052d0 2024-03-24T03:07:17.312 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T03:07:17.312+0000 7f81db400640 1 -- 172.21.15.118:0/659686424 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f81c4006460 con 0x5622651a13f0 2024-03-24T03:07:17.312 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T03:07:17.313+0000 7f81ef400640 1 -- 172.21.15.118:0/1693767697 <== mon.2 v2:172.21.15.189:3300/0 78 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (secure 0 0 0) 0x7f81c80036c0 con 0x7f81f81052d0 2024-03-24T03:07:17.312 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T03:07:17.313+0000 7f81e4c00640 1 -- 172.21.15.118:0/659686424 <== mon.1 v2:172.21.15.193:3300/0 78 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 409+0+0 (secure 0 0 0) 0x7f81c4006460 con 0x5622651a13f0 2024-03-24T03:07:17.945 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 03:07:17 smithi118 ceph-mon[31535]: pgmap v5782: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 5 op/s 2024-03-24T03:07:17.953 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 03:07:17 smithi189 ceph-mon[36187]: pgmap v5782: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 5 op/s 2024-03-24T03:07:17.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 03:07:17 smithi193 ceph-mon[36878]: pgmap v5782: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 5 op/s 2024-03-24T03:07:19.945 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 03:07:19 smithi118 ceph-mon[31535]: pgmap v5783: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T03:07:19.953 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 03:07:19 smithi189 ceph-mon[36187]: pgmap v5783: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T03:07:19.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 03:07:19 smithi193 ceph-mon[36878]: pgmap v5783: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T03:07:21.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 03:07:21 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 9736.752883 secs 2024-03-24T03:07:21.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 03:07:21 smithi118 ceph-mon[31535]: pgmap v5784: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T03:07:21.953 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 03:07:21 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 9736.752883 secs 2024-03-24T03:07:21.953 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 03:07:21 smithi189 ceph-mon[36187]: pgmap v5784: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T03:07:21.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 03:07:21 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 9736.752883 secs 2024-03-24T03:07:21.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 03:07:21 smithi193 ceph-mon[36878]: pgmap v5784: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T03:07:23.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 03:07:23 smithi118 ceph-mon[31535]: pgmap v5785: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T03:07:23.953 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 03:07:23 smithi189 ceph-mon[36187]: pgmap v5785: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T03:07:23.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 03:07:23 smithi193 ceph-mon[36878]: pgmap v5785: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T03:07:25.945 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 03:07:25 smithi118 ceph-mon[31535]: pgmap v5786: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-03-24T03:07:25.953 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 03:07:25 smithi189 ceph-mon[36187]: pgmap v5786: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-03-24T03:07:25.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 03:07:25 smithi193 ceph-mon[36878]: pgmap v5786: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-03-24T03:07:26.945 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 03:07:26 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 9741.752945 secs 2024-03-24T03:07:26.953 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 03:07:26 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 9741.752945 secs 2024-03-24T03:07:26.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 03:07:26 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 9741.752945 secs 2024-03-24T03:07:27.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 03:07:27 smithi118 ceph-mon[31535]: pgmap v5787: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:07:27.953 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 03:07:27 smithi189 ceph-mon[36187]: pgmap v5787: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:07:27.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 03:07:27 smithi193 ceph-mon[36878]: pgmap v5787: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:07:29.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 03:07:29 smithi118 ceph-mon[31535]: pgmap v5788: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T03:07:29.953 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 03:07:29 smithi189 ceph-mon[36187]: pgmap v5788: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T03:07:29.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 03:07:29 smithi193 ceph-mon[36878]: pgmap v5788: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T03:07:31.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 03:07:31 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 9746.753005 secs 2024-03-24T03:07:31.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 03:07:31 smithi118 ceph-mon[31535]: pgmap v5789: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T03:07:31.953 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 03:07:31 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 9746.753005 secs 2024-03-24T03:07:31.953 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 03:07:31 smithi189 ceph-mon[36187]: pgmap v5789: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T03:07:31.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 03:07:31 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 9746.753005 secs 2024-03-24T03:07:31.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 03:07:31 smithi193 ceph-mon[36878]: pgmap v5789: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T03:07:33.945 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 03:07:33 smithi118 ceph-mon[31535]: pgmap v5790: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T03:07:33.953 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 03:07:33 smithi189 ceph-mon[36187]: pgmap v5790: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T03:07:33.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 03:07:33 smithi193 ceph-mon[36878]: pgmap v5790: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T03:07:35.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 03:07:35 smithi118 ceph-mon[31535]: pgmap v5791: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-03-24T03:07:35.953 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 03:07:35 smithi189 ceph-mon[36187]: pgmap v5791: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-03-24T03:07:35.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 03:07:35 smithi193 ceph-mon[36878]: pgmap v5791: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-03-24T03:07:36.945 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 03:07:36 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 9751.753083 secs 2024-03-24T03:07:36.953 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 03:07:36 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 9751.753083 secs 2024-03-24T03:07:36.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 03:07:36 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 9751.753083 secs 2024-03-24T03:07:37.945 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 03:07:37 smithi118 ceph-mon[31535]: pgmap v5792: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:07:37.953 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 03:07:37 smithi189 ceph-mon[36187]: pgmap v5792: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:07:37.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 03:07:37 smithi193 ceph-mon[36878]: pgmap v5792: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:07:39.945 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 03:07:39 smithi118 ceph-mon[31535]: pgmap v5793: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T03:07:39.953 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 03:07:39 smithi189 ceph-mon[36187]: pgmap v5793: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T03:07:39.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 03:07:39 smithi193 ceph-mon[36878]: pgmap v5793: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T03:07:41.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 03:07:41 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 9756.753119 secs 2024-03-24T03:07:41.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 03:07:41 smithi118 ceph-mon[31535]: pgmap v5794: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T03:07:41.953 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 03:07:41 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 9756.753119 secs 2024-03-24T03:07:41.953 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 03:07:41 smithi189 ceph-mon[36187]: pgmap v5794: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T03:07:41.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 03:07:41 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 9756.753119 secs 2024-03-24T03:07:41.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 03:07:41 smithi193 ceph-mon[36878]: pgmap v5794: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T03:07:43.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 03:07:43 smithi118 ceph-mon[31535]: pgmap v5795: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T03:07:43.953 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 03:07:43 smithi189 ceph-mon[36187]: pgmap v5795: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T03:07:43.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 03:07:43 smithi193 ceph-mon[36878]: pgmap v5795: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T03:07:45.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 03:07:45 smithi118 ceph-mon[31535]: pgmap v5796: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-03-24T03:07:45.953 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 03:07:45 smithi189 ceph-mon[36187]: pgmap v5796: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-03-24T03:07:45.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 03:07:45 smithi193 ceph-mon[36878]: pgmap v5796: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-03-24T03:07:46.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 03:07:46 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 9761.753196 secs 2024-03-24T03:07:46.953 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 03:07:46 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 9761.753196 secs 2024-03-24T03:07:46.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 03:07:46 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 9761.753196 secs 2024-03-24T03:07:47.945 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 03:07:47 smithi118 ceph-mon[31535]: pgmap v5797: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:07:47.953 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 03:07:47 smithi189 ceph-mon[36187]: pgmap v5797: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:07:47.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 03:07:47 smithi193 ceph-mon[36878]: pgmap v5797: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:07:49.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 03:07:49 smithi118 ceph-mon[31535]: pgmap v5798: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T03:07:49.953 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 03:07:49 smithi189 ceph-mon[36187]: pgmap v5798: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T03:07:49.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 03:07:49 smithi193 ceph-mon[36878]: pgmap v5798: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T03:07:51.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 03:07:51 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 9766.753282 secs 2024-03-24T03:07:51.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 03:07:51 smithi118 ceph-mon[31535]: pgmap v5799: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T03:07:51.953 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 03:07:51 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 9766.753282 secs 2024-03-24T03:07:51.953 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 03:07:51 smithi189 ceph-mon[36187]: pgmap v5799: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T03:07:51.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 03:07:51 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 9766.753282 secs 2024-03-24T03:07:51.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 03:07:51 smithi193 ceph-mon[36878]: pgmap v5799: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T03:07:53.945 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 03:07:53 smithi118 ceph-mon[31535]: pgmap v5800: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T03:07:53.953 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 03:07:53 smithi189 ceph-mon[36187]: pgmap v5800: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T03:07:53.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 03:07:53 smithi193 ceph-mon[36878]: pgmap v5800: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T03:07:55.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 03:07:55 smithi118 ceph-mon[31535]: pgmap v5801: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-03-24T03:07:55.953 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 03:07:55 smithi189 ceph-mon[36187]: pgmap v5801: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-03-24T03:07:55.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 03:07:55 smithi193 ceph-mon[36878]: pgmap v5801: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-03-24T03:07:56.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 03:07:56 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 9771.753371 secs 2024-03-24T03:07:56.953 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 03:07:56 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 9771.753371 secs 2024-03-24T03:07:56.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 03:07:56 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 9771.753371 secs 2024-03-24T03:07:57.945 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 03:07:57 smithi118 ceph-mon[31535]: pgmap v5802: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:07:57.953 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 03:07:57 smithi189 ceph-mon[36187]: pgmap v5802: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:07:57.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 03:07:57 smithi193 ceph-mon[36878]: pgmap v5802: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:07:59.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 03:07:59 smithi118 ceph-mon[31535]: pgmap v5803: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T03:07:59.953 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 03:07:59 smithi189 ceph-mon[36187]: pgmap v5803: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T03:07:59.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 03:07:59 smithi193 ceph-mon[36878]: pgmap v5803: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T03:08:01.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 03:08:01 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 9776.753461 secs 2024-03-24T03:08:01.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 03:08:01 smithi118 ceph-mon[31535]: pgmap v5804: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T03:08:01.953 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 03:08:01 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 9776.753461 secs 2024-03-24T03:08:01.953 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 03:08:01 smithi189 ceph-mon[36187]: pgmap v5804: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T03:08:01.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 03:08:01 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 9776.753461 secs 2024-03-24T03:08:01.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 03:08:01 smithi193 ceph-mon[36878]: pgmap v5804: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T03:08:03.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 03:08:03 smithi118 ceph-mon[31535]: pgmap v5805: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T03:08:03.703 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 03:08:03 smithi189 ceph-mon[36187]: pgmap v5805: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T03:08:03.710 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 03:08:03 smithi193 ceph-mon[36878]: pgmap v5805: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T03:08:05.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 03:08:05 smithi118 ceph-mon[31535]: pgmap v5806: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-03-24T03:08:05.953 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 03:08:05 smithi189 ceph-mon[36187]: pgmap v5806: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-03-24T03:08:05.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 03:08:05 smithi193 ceph-mon[36878]: pgmap v5806: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-03-24T03:08:06.945 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 03:08:06 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 9781.753521 secs 2024-03-24T03:08:06.953 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 03:08:06 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 9781.753521 secs 2024-03-24T03:08:06.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 03:08:06 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 9781.753521 secs 2024-03-24T03:08:07.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 03:08:07 smithi118 ceph-mon[31535]: pgmap v5807: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:08:07.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 03:08:07 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T03:08:07.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 03:08:07 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-24T03:08:07.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 03:08:07 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-24T03:08:07.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 03:08:07 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-24T03:08:07.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 03:08:07 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-03-24T03:08:07.953 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 03:08:07 smithi189 ceph-mon[36187]: pgmap v5807: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:08:07.953 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 03:08:07 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T03:08:07.953 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 03:08:07 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-24T03:08:07.953 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 03:08:07 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-24T03:08:07.953 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 03:08:07 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-24T03:08:07.953 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 03:08:07 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-03-24T03:08:07.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 03:08:07 smithi193 ceph-mon[36878]: pgmap v5807: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:08:07.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 03:08:07 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T03:08:07.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 03:08:07 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-24T03:08:07.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 03:08:07 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-24T03:08:07.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 03:08:07 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-24T03:08:07.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 03:08:07 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-03-24T03:08:08.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 03:08:08 smithi118 ceph-mon[31535]: Adjusting osd_memory_target on smithi189 to 174.1M 2024-03-24T03:08:08.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 03:08:08 smithi118 ceph-mon[31535]: Unable to set osd_memory_target on smithi189 to 182635622: error parsing value: Value '182635622' is below minimum 939524096 2024-03-24T03:08:08.947 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 03:08:08 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-24T03:08:08.947 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 03:08:08 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-24T03:08:08.947 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 03:08:08 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-24T03:08:08.947 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 03:08:08 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-03-24T03:08:08.947 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 03:08:08 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T03:08:08.947 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 03:08:08 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T03:08:08.947 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 03:08:08 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T03:08:08.953 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 03:08:08 smithi189 ceph-mon[36187]: Adjusting osd_memory_target on smithi189 to 174.1M 2024-03-24T03:08:08.953 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 03:08:08 smithi189 ceph-mon[36187]: Unable to set osd_memory_target on smithi189 to 182635622: error parsing value: Value '182635622' is below minimum 939524096 2024-03-24T03:08:08.953 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 03:08:08 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-24T03:08:08.953 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 03:08:08 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-24T03:08:08.953 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 03:08:08 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-24T03:08:08.953 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 03:08:08 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-03-24T03:08:08.953 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 03:08:08 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T03:08:08.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 03:08:08 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T03:08:08.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 03:08:08 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T03:08:08.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 03:08:08 smithi193 ceph-mon[36878]: Adjusting osd_memory_target on smithi189 to 174.1M 2024-03-24T03:08:08.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 03:08:08 smithi193 ceph-mon[36878]: Unable to set osd_memory_target on smithi189 to 182635622: error parsing value: Value '182635622' is below minimum 939524096 2024-03-24T03:08:08.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 03:08:08 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-24T03:08:08.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 03:08:08 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-24T03:08:08.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 03:08:08 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-24T03:08:08.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 03:08:08 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-03-24T03:08:08.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 03:08:08 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T03:08:08.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 03:08:08 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T03:08:08.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 03:08:08 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T03:08:09.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 03:08:09 smithi118 ceph-mon[31535]: Adjusting osd_memory_target on smithi118 to 176.7M 2024-03-24T03:08:09.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 03:08:09 smithi118 ceph-mon[31535]: Unable to set osd_memory_target on smithi118 to 185365196: error parsing value: Value '185365196' is below minimum 939524096 2024-03-24T03:08:09.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 03:08:09 smithi118 ceph-mon[31535]: pgmap v5808: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 6 op/s 2024-03-24T03:08:09.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 03:08:09 smithi118 ceph-mon[31535]: pgmap v5809: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 5 op/s 2024-03-24T03:08:09.953 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 03:08:09 smithi189 ceph-mon[36187]: Adjusting osd_memory_target on smithi118 to 176.7M 2024-03-24T03:08:09.953 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 03:08:09 smithi189 ceph-mon[36187]: Unable to set osd_memory_target on smithi118 to 185365196: error parsing value: Value '185365196' is below minimum 939524096 2024-03-24T03:08:09.953 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 03:08:09 smithi189 ceph-mon[36187]: pgmap v5808: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 6 op/s 2024-03-24T03:08:09.953 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 03:08:09 smithi189 ceph-mon[36187]: pgmap v5809: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 5 op/s 2024-03-24T03:08:09.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 03:08:09 smithi193 ceph-mon[36878]: Adjusting osd_memory_target on smithi118 to 176.7M 2024-03-24T03:08:09.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 03:08:09 smithi193 ceph-mon[36878]: Unable to set osd_memory_target on smithi118 to 185365196: error parsing value: Value '185365196' is below minimum 939524096 2024-03-24T03:08:09.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 03:08:09 smithi193 ceph-mon[36878]: pgmap v5808: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 6 op/s 2024-03-24T03:08:09.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 03:08:09 smithi193 ceph-mon[36878]: pgmap v5809: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 5 op/s 2024-03-24T03:08:11.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 03:08:11 smithi118 ceph-mon[31535]: pgmap v5810: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-03-24T03:08:11.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 03:08:11 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 9786.753581 secs 2024-03-24T03:08:11.953 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 03:08:11 smithi189 ceph-mon[36187]: pgmap v5810: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-03-24T03:08:11.953 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 03:08:11 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 9786.753581 secs 2024-03-24T03:08:11.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 03:08:11 smithi193 ceph-mon[36878]: pgmap v5810: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-03-24T03:08:11.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 03:08:11 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 9786.753581 secs 2024-03-24T03:08:13.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 03:08:13 smithi118 ceph-mon[31535]: pgmap v5811: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 5 op/s 2024-03-24T03:08:13.953 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 03:08:13 smithi189 ceph-mon[36187]: pgmap v5811: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 5 op/s 2024-03-24T03:08:13.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 03:08:13 smithi193 ceph-mon[36878]: pgmap v5811: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 5 op/s 2024-03-24T03:08:15.945 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 03:08:15 smithi118 ceph-mon[31535]: pgmap v5812: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 5 op/s 2024-03-24T03:08:15.953 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 03:08:15 smithi189 ceph-mon[36187]: pgmap v5812: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 5 op/s 2024-03-24T03:08:15.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 03:08:15 smithi193 ceph-mon[36878]: pgmap v5812: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 5 op/s 2024-03-24T03:08:16.945 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 03:08:16 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 9791.753669 secs 2024-03-24T03:08:16.953 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 03:08:16 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 9791.753669 secs 2024-03-24T03:08:16.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 03:08:16 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 9791.753669 secs 2024-03-24T03:08:17.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 03:08:17 smithi118 ceph-mon[31535]: pgmap v5813: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-03-24T03:08:17.953 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 03:08:17 smithi189 ceph-mon[36187]: pgmap v5813: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-03-24T03:08:17.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 03:08:17 smithi193 ceph-mon[36878]: pgmap v5813: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-03-24T03:08:19.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 03:08:19 smithi118 ceph-mon[31535]: pgmap v5814: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 5 op/s 2024-03-24T03:08:19.953 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 03:08:19 smithi189 ceph-mon[36187]: pgmap v5814: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 5 op/s 2024-03-24T03:08:19.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 03:08:19 smithi193 ceph-mon[36878]: pgmap v5814: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 5 op/s 2024-03-24T03:08:21.945 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 03:08:21 smithi118 ceph-mon[31535]: pgmap v5815: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T03:08:21.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 03:08:21 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 9796.753729 secs 2024-03-24T03:08:21.953 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 03:08:21 smithi189 ceph-mon[36187]: pgmap v5815: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T03:08:21.953 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 03:08:21 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 9796.753729 secs 2024-03-24T03:08:21.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 03:08:21 smithi193 ceph-mon[36878]: pgmap v5815: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T03:08:21.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 03:08:21 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 9796.753729 secs 2024-03-24T03:08:23.945 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 03:08:23 smithi118 ceph-mon[31535]: pgmap v5816: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:08:23.953 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 03:08:23 smithi189 ceph-mon[36187]: pgmap v5816: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:08:23.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 03:08:23 smithi193 ceph-mon[36878]: pgmap v5816: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:08:25.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 03:08:25 smithi118 ceph-mon[31535]: pgmap v5817: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T03:08:25.953 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 03:08:25 smithi189 ceph-mon[36187]: pgmap v5817: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T03:08:25.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 03:08:25 smithi193 ceph-mon[36878]: pgmap v5817: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T03:08:26.945 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 03:08:26 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 9801.753787 secs 2024-03-24T03:08:26.953 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 03:08:26 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 9801.753787 secs 2024-03-24T03:08:26.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 03:08:26 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 9801.753787 secs 2024-03-24T03:08:27.945 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 03:08:27 smithi118 ceph-mon[31535]: pgmap v5818: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T03:08:27.953 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 03:08:27 smithi189 ceph-mon[36187]: pgmap v5818: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T03:08:27.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 03:08:27 smithi193 ceph-mon[36878]: pgmap v5818: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T03:08:29.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 03:08:29 smithi118 ceph-mon[31535]: pgmap v5819: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 5 op/s 2024-03-24T03:08:29.953 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 03:08:29 smithi189 ceph-mon[36187]: pgmap v5819: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 5 op/s 2024-03-24T03:08:29.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 03:08:29 smithi193 ceph-mon[36878]: pgmap v5819: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 5 op/s 2024-03-24T03:08:31.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 03:08:31 smithi118 ceph-mon[31535]: pgmap v5820: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T03:08:31.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 03:08:31 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 9806.753847 secs 2024-03-24T03:08:31.953 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 03:08:31 smithi189 ceph-mon[36187]: pgmap v5820: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T03:08:31.953 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 03:08:31 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 9806.753847 secs 2024-03-24T03:08:31.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 03:08:31 smithi193 ceph-mon[36878]: pgmap v5820: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T03:08:31.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 03:08:31 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 9806.753847 secs 2024-03-24T03:08:33.945 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 03:08:33 smithi118 ceph-mon[31535]: pgmap v5821: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:08:33.953 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 03:08:33 smithi189 ceph-mon[36187]: pgmap v5821: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:08:33.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 03:08:33 smithi193 ceph-mon[36878]: pgmap v5821: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:08:35.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 03:08:35 smithi118 ceph-mon[31535]: pgmap v5822: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-03-24T03:08:35.953 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 03:08:35 smithi189 ceph-mon[36187]: pgmap v5822: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-03-24T03:08:35.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 03:08:35 smithi193 ceph-mon[36878]: pgmap v5822: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-03-24T03:08:36.945 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 03:08:36 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 9811.753907 secs 2024-03-24T03:08:36.953 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 03:08:36 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 9811.753907 secs 2024-03-24T03:08:36.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 03:08:36 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 9811.753907 secs 2024-03-24T03:08:37.945 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 03:08:37 smithi118 ceph-mon[31535]: pgmap v5823: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 5 op/s 2024-03-24T03:08:37.953 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 03:08:37 smithi189 ceph-mon[36187]: pgmap v5823: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 5 op/s 2024-03-24T03:08:37.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 03:08:37 smithi193 ceph-mon[36878]: pgmap v5823: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 5 op/s 2024-03-24T03:08:39.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 03:08:39 smithi118 ceph-mon[31535]: pgmap v5824: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 5 op/s 2024-03-24T03:08:39.953 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 03:08:39 smithi189 ceph-mon[36187]: pgmap v5824: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 5 op/s 2024-03-24T03:08:39.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 03:08:39 smithi193 ceph-mon[36878]: pgmap v5824: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 5 op/s 2024-03-24T03:08:41.945 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 03:08:41 smithi118 ceph-mon[31535]: pgmap v5825: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 6 op/s 2024-03-24T03:08:41.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 03:08:41 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 9816.753970 secs 2024-03-24T03:08:41.953 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 03:08:41 smithi189 ceph-mon[36187]: pgmap v5825: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 6 op/s 2024-03-24T03:08:41.953 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 03:08:41 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 9816.753970 secs 2024-03-24T03:08:41.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 03:08:41 smithi193 ceph-mon[36878]: pgmap v5825: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 6 op/s 2024-03-24T03:08:41.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 03:08:41 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 9816.753970 secs 2024-03-24T03:08:43.945 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 03:08:43 smithi118 ceph-mon[31535]: pgmap v5826: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-03-24T03:08:43.953 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 03:08:43 smithi189 ceph-mon[36187]: pgmap v5826: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-03-24T03:08:43.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 03:08:43 smithi193 ceph-mon[36878]: pgmap v5826: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-03-24T03:08:45.945 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 03:08:45 smithi118 ceph-mon[31535]: pgmap v5827: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-03-24T03:08:45.953 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 03:08:45 smithi189 ceph-mon[36187]: pgmap v5827: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-03-24T03:08:45.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 03:08:45 smithi193 ceph-mon[36878]: pgmap v5827: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-03-24T03:08:46.945 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 03:08:46 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 9821.754024 secs 2024-03-24T03:08:46.953 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 03:08:46 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 9821.754024 secs 2024-03-24T03:08:46.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 03:08:46 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 9821.754024 secs 2024-03-24T03:08:47.945 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 03:08:47 smithi118 ceph-mon[31535]: pgmap v5828: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T03:08:47.953 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 03:08:47 smithi189 ceph-mon[36187]: pgmap v5828: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T03:08:47.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 03:08:47 smithi193 ceph-mon[36878]: pgmap v5828: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T03:08:49.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 03:08:49 smithi118 ceph-mon[31535]: pgmap v5829: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 5 op/s 2024-03-24T03:08:49.953 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 03:08:49 smithi189 ceph-mon[36187]: pgmap v5829: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 5 op/s 2024-03-24T03:08:49.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 03:08:49 smithi193 ceph-mon[36878]: pgmap v5829: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 5 op/s 2024-03-24T03:08:51.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 03:08:51 smithi118 ceph-mon[31535]: pgmap v5830: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T03:08:51.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 03:08:51 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 9826.754066 secs 2024-03-24T03:08:51.953 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 03:08:51 smithi189 ceph-mon[36187]: pgmap v5830: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T03:08:51.953 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 03:08:51 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 9826.754066 secs 2024-03-24T03:08:51.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 03:08:51 smithi193 ceph-mon[36878]: pgmap v5830: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T03:08:51.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 03:08:51 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 9826.754066 secs 2024-03-24T03:08:53.953 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 03:08:53 smithi189 ceph-mon[36187]: pgmap v5831: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:08:53.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 03:08:53 smithi193 ceph-mon[36878]: pgmap v5831: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:08:54.195 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 03:08:53 smithi118 ceph-mon[31535]: pgmap v5831: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:08:55.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 03:08:55 smithi189 ceph-mon[36187]: pgmap v5832: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T03:08:55.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 03:08:55 smithi193 ceph-mon[36878]: pgmap v5832: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T03:08:56.195 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 03:08:55 smithi118 ceph-mon[31535]: pgmap v5832: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T03:08:56.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 03:08:56 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 9831.754125 secs 2024-03-24T03:08:57.195 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 03:08:56 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 9831.754125 secs 2024-03-24T03:08:57.203 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 03:08:56 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 9831.754125 secs 2024-03-24T03:08:58.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 03:08:57 smithi118 ceph-mon[31535]: pgmap v5833: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T03:08:58.203 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 03:08:57 smithi189 ceph-mon[36187]: pgmap v5833: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T03:08:58.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 03:08:57 smithi193 ceph-mon[36878]: pgmap v5833: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T03:09:00.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 03:08:59 smithi118 ceph-mon[31535]: pgmap v5834: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 5 op/s 2024-03-24T03:09:00.203 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 03:08:59 smithi189 ceph-mon[36187]: pgmap v5834: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 5 op/s 2024-03-24T03:09:00.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 03:08:59 smithi193 ceph-mon[36878]: pgmap v5834: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 5 op/s 2024-03-24T03:09:02.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 03:09:01 smithi118 ceph-mon[31535]: pgmap v5835: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T03:09:02.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 03:09:01 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 9836.754208 secs 2024-03-24T03:09:02.203 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 03:09:01 smithi189 ceph-mon[36187]: pgmap v5835: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T03:09:02.203 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 03:09:01 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 9836.754208 secs 2024-03-24T03:09:02.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 03:09:01 smithi193 ceph-mon[36878]: pgmap v5835: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T03:09:02.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 03:09:01 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 9836.754208 secs 2024-03-24T03:09:04.004 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 03:09:03 smithi189 ceph-mon[36187]: pgmap v5836: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:09:04.012 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 03:09:03 smithi193 ceph-mon[36878]: pgmap v5836: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:09:04.195 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 03:09:03 smithi118 ceph-mon[31535]: pgmap v5836: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:09:06.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 03:09:05 smithi118 ceph-mon[31535]: pgmap v5837: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T03:09:06.203 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 03:09:05 smithi189 ceph-mon[36187]: pgmap v5837: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T03:09:06.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 03:09:05 smithi193 ceph-mon[36878]: pgmap v5837: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T03:09:07.195 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 03:09:06 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 9841.754246 secs 2024-03-24T03:09:07.203 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 03:09:06 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 9841.754246 secs 2024-03-24T03:09:07.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 03:09:06 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 9841.754246 secs 2024-03-24T03:09:08.195 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 03:09:07 smithi118 ceph-mon[31535]: pgmap v5838: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T03:09:08.203 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 03:09:07 smithi189 ceph-mon[36187]: pgmap v5838: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T03:09:08.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 03:09:07 smithi193 ceph-mon[36878]: pgmap v5838: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T03:09:09.195 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 03:09:08 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T03:09:09.203 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 03:09:08 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T03:09:09.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 03:09:08 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T03:09:10.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 03:09:09 smithi118 ceph-mon[31535]: pgmap v5839: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 5 op/s 2024-03-24T03:09:10.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 03:09:09 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-24T03:09:10.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 03:09:09 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-24T03:09:10.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 03:09:09 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-03-24T03:09:10.197 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 03:09:09 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-03-24T03:09:10.197 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 03:09:09 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T03:09:10.197 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 03:09:09 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T03:09:10.197 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 03:09:09 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T03:09:10.203 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 03:09:09 smithi189 ceph-mon[36187]: pgmap v5839: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 5 op/s 2024-03-24T03:09:10.203 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 03:09:09 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-24T03:09:10.203 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 03:09:09 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-24T03:09:10.203 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 03:09:09 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-03-24T03:09:10.203 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 03:09:09 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-03-24T03:09:10.203 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 03:09:09 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T03:09:10.203 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 03:09:09 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T03:09:10.203 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 03:09:09 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T03:09:10.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 03:09:09 smithi193 ceph-mon[36878]: pgmap v5839: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 5 op/s 2024-03-24T03:09:10.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 03:09:09 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-24T03:09:10.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 03:09:09 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-24T03:09:10.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 03:09:09 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-03-24T03:09:10.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 03:09:09 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-03-24T03:09:10.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 03:09:09 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T03:09:10.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 03:09:09 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T03:09:10.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 03:09:09 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T03:09:11.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 03:09:10 smithi118 ceph-mon[31535]: Adjusting osd_memory_target on smithi193 to 174.1M 2024-03-24T03:09:11.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 03:09:10 smithi118 ceph-mon[31535]: Unable to set osd_memory_target on smithi193 to 182639923: error parsing value: Value '182639923' is below minimum 939524096 2024-03-24T03:09:11.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 03:09:10 smithi118 ceph-mon[31535]: pgmap v5840: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 6 op/s 2024-03-24T03:09:11.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 03:09:10 smithi118 ceph-mon[31535]: pgmap v5841: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 6 op/s 2024-03-24T03:09:11.203 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 03:09:10 smithi189 ceph-mon[36187]: Adjusting osd_memory_target on smithi193 to 174.1M 2024-03-24T03:09:11.203 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 03:09:10 smithi189 ceph-mon[36187]: Unable to set osd_memory_target on smithi193 to 182639923: error parsing value: Value '182639923' is below minimum 939524096 2024-03-24T03:09:11.203 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 03:09:10 smithi189 ceph-mon[36187]: pgmap v5840: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 6 op/s 2024-03-24T03:09:11.203 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 03:09:10 smithi189 ceph-mon[36187]: pgmap v5841: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 6 op/s 2024-03-24T03:09:11.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 03:09:10 smithi193 ceph-mon[36878]: Adjusting osd_memory_target on smithi193 to 174.1M 2024-03-24T03:09:11.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 03:09:10 smithi193 ceph-mon[36878]: Unable to set osd_memory_target on smithi193 to 182639923: error parsing value: Value '182639923' is below minimum 939524096 2024-03-24T03:09:11.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 03:09:10 smithi193 ceph-mon[36878]: pgmap v5840: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 6 op/s 2024-03-24T03:09:11.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 03:09:10 smithi193 ceph-mon[36878]: pgmap v5841: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 6 op/s 2024-03-24T03:09:12.195 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 03:09:11 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 9846.754316 secs 2024-03-24T03:09:12.203 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 03:09:11 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 9846.754316 secs 2024-03-24T03:09:12.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 03:09:11 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 9846.754316 secs 2024-03-24T03:09:13.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 03:09:12 smithi118 ceph-mon[31535]: pgmap v5842: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 6 op/s 2024-03-24T03:09:13.203 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 03:09:12 smithi189 ceph-mon[36187]: pgmap v5842: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 6 op/s 2024-03-24T03:09:13.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 03:09:12 smithi193 ceph-mon[36878]: pgmap v5842: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 6 op/s 2024-03-24T03:09:15.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 03:09:14 smithi118 ceph-mon[31535]: pgmap v5843: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 5 op/s 2024-03-24T03:09:15.203 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 03:09:14 smithi189 ceph-mon[36187]: pgmap v5843: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 5 op/s 2024-03-24T03:09:15.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 03:09:14 smithi193 ceph-mon[36878]: pgmap v5843: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 5 op/s 2024-03-24T03:09:17.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 03:09:16 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 9851.754398 secs 2024-03-24T03:09:17.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 03:09:16 smithi118 ceph-mon[31535]: pgmap v5844: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 6 op/s 2024-03-24T03:09:17.203 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 03:09:16 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 9851.754398 secs 2024-03-24T03:09:17.203 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 03:09:16 smithi189 ceph-mon[36187]: pgmap v5844: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 6 op/s 2024-03-24T03:09:17.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 03:09:16 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 9851.754398 secs 2024-03-24T03:09:17.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 03:09:16 smithi193 ceph-mon[36878]: pgmap v5844: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 6 op/s 2024-03-24T03:09:19.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 03:09:18 smithi118 ceph-mon[31535]: pgmap v5845: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 5 op/s 2024-03-24T03:09:19.203 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 03:09:18 smithi189 ceph-mon[36187]: pgmap v5845: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 5 op/s 2024-03-24T03:09:19.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 03:09:18 smithi193 ceph-mon[36878]: pgmap v5845: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 5 op/s 2024-03-24T03:09:21.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 03:09:20 smithi118 ceph-mon[31535]: pgmap v5846: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 5 op/s 2024-03-24T03:09:21.203 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 03:09:20 smithi189 ceph-mon[36187]: pgmap v5846: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 5 op/s 2024-03-24T03:09:21.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 03:09:20 smithi193 ceph-mon[36878]: pgmap v5846: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 5 op/s 2024-03-24T03:09:22.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 03:09:21 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 9856.754458 secs 2024-03-24T03:09:22.203 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 03:09:21 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 9856.754458 secs 2024-03-24T03:09:22.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 03:09:21 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 9856.754458 secs 2024-03-24T03:09:23.195 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 03:09:22 smithi118 ceph-mon[31535]: pgmap v5847: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:09:23.203 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 03:09:22 smithi189 ceph-mon[36187]: pgmap v5847: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:09:23.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 03:09:22 smithi193 ceph-mon[36878]: pgmap v5847: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:09:25.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 03:09:24 smithi118 ceph-mon[31535]: pgmap v5848: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T03:09:25.203 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 03:09:24 smithi189 ceph-mon[36187]: pgmap v5848: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T03:09:25.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 03:09:24 smithi193 ceph-mon[36878]: pgmap v5848: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T03:09:27.195 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 03:09:26 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 9861.754520 secs 2024-03-24T03:09:27.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 03:09:26 smithi118 ceph-mon[31535]: pgmap v5849: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T03:09:27.203 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 03:09:26 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 9861.754520 secs 2024-03-24T03:09:27.203 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 03:09:26 smithi189 ceph-mon[36187]: pgmap v5849: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T03:09:27.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 03:09:26 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 9861.754520 secs 2024-03-24T03:09:27.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 03:09:26 smithi193 ceph-mon[36878]: pgmap v5849: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T03:09:29.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 03:09:28 smithi118 ceph-mon[31535]: pgmap v5850: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T03:09:29.203 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 03:09:28 smithi189 ceph-mon[36187]: pgmap v5850: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T03:09:29.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 03:09:28 smithi193 ceph-mon[36878]: pgmap v5850: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T03:09:31.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 03:09:30 smithi118 ceph-mon[31535]: pgmap v5851: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-03-24T03:09:31.203 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 03:09:30 smithi189 ceph-mon[36187]: pgmap v5851: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-03-24T03:09:31.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 03:09:30 smithi193 ceph-mon[36878]: pgmap v5851: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-03-24T03:09:32.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 03:09:31 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 9866.754579 secs 2024-03-24T03:09:32.203 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 03:09:31 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 9866.754579 secs 2024-03-24T03:09:32.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 03:09:31 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 9866.754579 secs 2024-03-24T03:09:33.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 03:09:32 smithi118 ceph-mon[31535]: pgmap v5852: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:09:33.203 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 03:09:32 smithi189 ceph-mon[36187]: pgmap v5852: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:09:33.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 03:09:32 smithi193 ceph-mon[36878]: pgmap v5852: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:09:35.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 03:09:34 smithi118 ceph-mon[31535]: pgmap v5853: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T03:09:35.203 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 03:09:34 smithi189 ceph-mon[36187]: pgmap v5853: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T03:09:35.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 03:09:34 smithi193 ceph-mon[36878]: pgmap v5853: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T03:09:37.195 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 03:09:36 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 9871.754590 secs 2024-03-24T03:09:37.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 03:09:36 smithi118 ceph-mon[31535]: pgmap v5854: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T03:09:37.203 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 03:09:36 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 9871.754590 secs 2024-03-24T03:09:37.203 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 03:09:36 smithi189 ceph-mon[36187]: pgmap v5854: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T03:09:37.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 03:09:36 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 9871.754590 secs 2024-03-24T03:09:37.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 03:09:36 smithi193 ceph-mon[36878]: pgmap v5854: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T03:09:39.195 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 03:09:38 smithi118 ceph-mon[31535]: pgmap v5855: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T03:09:39.203 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 03:09:38 smithi189 ceph-mon[36187]: pgmap v5855: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T03:09:39.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 03:09:38 smithi193 ceph-mon[36878]: pgmap v5855: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T03:09:41.195 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 03:09:40 smithi118 ceph-mon[31535]: pgmap v5856: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-03-24T03:09:41.203 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 03:09:40 smithi189 ceph-mon[36187]: pgmap v5856: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-03-24T03:09:41.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 03:09:40 smithi193 ceph-mon[36878]: pgmap v5856: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-03-24T03:09:42.195 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 03:09:41 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 9876.754649 secs 2024-03-24T03:09:42.203 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 03:09:41 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 9876.754649 secs 2024-03-24T03:09:42.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 03:09:41 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 9876.754649 secs 2024-03-24T03:09:43.195 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 03:09:42 smithi118 ceph-mon[31535]: pgmap v5857: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:09:43.205 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 03:09:42 smithi189 ceph-mon[36187]: pgmap v5857: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:09:43.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 03:09:42 smithi193 ceph-mon[36878]: pgmap v5857: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:09:45.195 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 03:09:44 smithi118 ceph-mon[31535]: pgmap v5858: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T03:09:45.203 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 03:09:44 smithi189 ceph-mon[36187]: pgmap v5858: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T03:09:45.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 03:09:44 smithi193 ceph-mon[36878]: pgmap v5858: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T03:09:47.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 03:09:46 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 9881.754711 secs 2024-03-24T03:09:47.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 03:09:46 smithi118 ceph-mon[31535]: pgmap v5859: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T03:09:47.203 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 03:09:46 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 9881.754711 secs 2024-03-24T03:09:47.203 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 03:09:46 smithi189 ceph-mon[36187]: pgmap v5859: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T03:09:47.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 03:09:46 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 9881.754711 secs 2024-03-24T03:09:47.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 03:09:46 smithi193 ceph-mon[36878]: pgmap v5859: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T03:09:49.195 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 03:09:48 smithi118 ceph-mon[31535]: pgmap v5860: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T03:09:49.203 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 03:09:48 smithi189 ceph-mon[36187]: pgmap v5860: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T03:09:49.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 03:09:48 smithi193 ceph-mon[36878]: pgmap v5860: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T03:09:51.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 03:09:50 smithi118 ceph-mon[31535]: pgmap v5861: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-03-24T03:09:51.203 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 03:09:50 smithi189 ceph-mon[36187]: pgmap v5861: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-03-24T03:09:51.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 03:09:50 smithi193 ceph-mon[36878]: pgmap v5861: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-03-24T03:09:52.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 03:09:51 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 9886.754772 secs 2024-03-24T03:09:52.203 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 03:09:51 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 9886.754772 secs 2024-03-24T03:09:52.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 03:09:51 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 9886.754772 secs 2024-03-24T03:09:53.195 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 03:09:52 smithi118 ceph-mon[31535]: pgmap v5862: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:09:53.203 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 03:09:52 smithi189 ceph-mon[36187]: pgmap v5862: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:09:53.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 03:09:52 smithi193 ceph-mon[36878]: pgmap v5862: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:09:55.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 03:09:54 smithi118 ceph-mon[31535]: pgmap v5863: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T03:09:55.203 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 03:09:54 smithi189 ceph-mon[36187]: pgmap v5863: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T03:09:55.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 03:09:54 smithi193 ceph-mon[36878]: pgmap v5863: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T03:09:57.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 03:09:56 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 9891.754856 secs 2024-03-24T03:09:57.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 03:09:56 smithi118 ceph-mon[31535]: pgmap v5864: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T03:09:57.203 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 03:09:56 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 9891.754856 secs 2024-03-24T03:09:57.203 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 03:09:56 smithi189 ceph-mon[36187]: pgmap v5864: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T03:09:57.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 03:09:56 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 9891.754856 secs 2024-03-24T03:09:57.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 03:09:56 smithi193 ceph-mon[36878]: pgmap v5864: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T03:09:59.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 03:09:58 smithi118 ceph-mon[31535]: pgmap v5865: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T03:09:59.203 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 03:09:58 smithi189 ceph-mon[36187]: pgmap v5865: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T03:09:59.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 03:09:58 smithi193 ceph-mon[36878]: pgmap v5865: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T03:10:01.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 03:10:00 smithi118 ceph-mon[31535]: pgmap v5866: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-03-24T03:10:01.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 03:10:00 smithi118 ceph-mon[31535]: overall HEALTH_WARN 6 failed cephadm daemon(s); 1 MDSs report slow requests 2024-03-24T03:10:01.203 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 03:10:00 smithi189 ceph-mon[36187]: pgmap v5866: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-03-24T03:10:01.203 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 03:10:00 smithi189 ceph-mon[36187]: overall HEALTH_WARN 6 failed cephadm daemon(s); 1 MDSs report slow requests 2024-03-24T03:10:01.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 03:10:00 smithi193 ceph-mon[36878]: pgmap v5866: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-03-24T03:10:01.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 03:10:00 smithi193 ceph-mon[36878]: overall HEALTH_WARN 6 failed cephadm daemon(s); 1 MDSs report slow requests 2024-03-24T03:10:02.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 03:10:01 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 9896.754947 secs 2024-03-24T03:10:02.203 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 03:10:01 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 9896.754947 secs 2024-03-24T03:10:02.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 03:10:01 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 9896.754947 secs 2024-03-24T03:10:03.195 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 03:10:02 smithi118 ceph-mon[31535]: pgmap v5867: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:10:03.203 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 03:10:02 smithi189 ceph-mon[36187]: pgmap v5867: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:10:03.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 03:10:02 smithi193 ceph-mon[36878]: pgmap v5867: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:10:05.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 03:10:04 smithi118 ceph-mon[31535]: pgmap v5868: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T03:10:05.203 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 03:10:04 smithi189 ceph-mon[36187]: pgmap v5868: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T03:10:05.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 03:10:04 smithi193 ceph-mon[36878]: pgmap v5868: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T03:10:07.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 03:10:06 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 9901.755008 secs 2024-03-24T03:10:07.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 03:10:06 smithi118 ceph-mon[31535]: pgmap v5869: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T03:10:07.203 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 03:10:06 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 9901.755008 secs 2024-03-24T03:10:07.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 03:10:06 smithi189 ceph-mon[36187]: pgmap v5869: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T03:10:07.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 03:10:06 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 9901.755008 secs 2024-03-24T03:10:07.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 03:10:06 smithi193 ceph-mon[36878]: pgmap v5869: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T03:10:09.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 03:10:08 smithi118 ceph-mon[31535]: pgmap v5870: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T03:10:09.203 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 03:10:08 smithi189 ceph-mon[36187]: pgmap v5870: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T03:10:09.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 03:10:08 smithi193 ceph-mon[36878]: pgmap v5870: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T03:10:10.203 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 03:10:09 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T03:10:10.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 03:10:09 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T03:10:10.445 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 03:10:09 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T03:10:11.203 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 03:10:10 smithi189 ceph-mon[36187]: pgmap v5871: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-03-24T03:10:11.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 03:10:10 smithi193 ceph-mon[36878]: pgmap v5871: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-03-24T03:10:11.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 03:10:10 smithi118 ceph-mon[31535]: pgmap v5871: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-03-24T03:10:12.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 03:10:11 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 9906.755090 secs 2024-03-24T03:10:12.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 03:10:11 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T03:10:12.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 03:10:11 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T03:10:12.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 03:10:11 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T03:10:12.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 03:10:11 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 9906.755090 secs 2024-03-24T03:10:12.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 03:10:11 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T03:10:12.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 03:10:11 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T03:10:12.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 03:10:11 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T03:10:12.453 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 03:10:11 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 9906.755090 secs 2024-03-24T03:10:12.453 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 03:10:11 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T03:10:12.453 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 03:10:11 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T03:10:12.453 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 03:10:11 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T03:10:13.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 03:10:12 smithi118 ceph-mon[31535]: pgmap v5872: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-03-24T03:10:13.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 03:10:12 smithi118 ceph-mon[31535]: pgmap v5873: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 6 op/s 2024-03-24T03:10:13.453 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 03:10:12 smithi189 ceph-mon[36187]: pgmap v5872: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-03-24T03:10:13.453 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 03:10:12 smithi189 ceph-mon[36187]: pgmap v5873: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 6 op/s 2024-03-24T03:10:13.460 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 03:10:12 smithi193 ceph-mon[36878]: pgmap v5872: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-03-24T03:10:13.460 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 03:10:12 smithi193 ceph-mon[36878]: pgmap v5873: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 6 op/s 2024-03-24T03:10:14.445 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 03:10:13 smithi118 ceph-mon[31535]: pgmap v5874: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 6 op/s 2024-03-24T03:10:14.453 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 03:10:13 smithi189 ceph-mon[36187]: pgmap v5874: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 6 op/s 2024-03-24T03:10:14.460 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 03:10:13 smithi193 ceph-mon[36878]: pgmap v5874: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 6 op/s 2024-03-24T03:10:15.770 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T03:10:15.770+0000 7f81f6000640 1 -- 172.21.15.118:0/1693767697 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7f81dc0a3bd0 msgr2=0x7f81dc0b2b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 12 2024-03-24T03:10:15.770 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T03:10:15.770+0000 7f81f6000640 1 -- 172.21.15.118:0/1693767697 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7f81dc0a3bd0 msgr2=0x7f81dc0b2b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-03-24T03:10:15.770 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T03:10:15.770+0000 7f81f6000640 1 --2- 172.21.15.118:0/1693767697 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7f81dc0a3bd0 0x7f81dc0b2b10 secure :-1 s=READY pgs=606 cs=0 l=1 rev1=1 crypto rx=0x7f81e0010870 tx=0x7f81e0024040 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 2024-03-24T03:10:15.770 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T03:10:15.770+0000 7f81f6000640 1 --2- 172.21.15.118:0/1693767697 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7f81dc0a3bd0 0x7f81dc0b2b10 secure :-1 s=READY pgs=606 cs=0 l=1 rev1=1 crypto rx=0x7f81e0010870 tx=0x7f81e0024040 comp rx=0 tx=0).stop 2024-03-24T03:10:15.771 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T03:10:15.770+0000 7f81f6000640 1 -- 172.21.15.118:0/1693767697 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7f81dc0a3bd0 msgr2=0x7f81dc0b2b10 unknown :-1 s=STATE_CLOSED l=1).tick see no progress in more than 10000000 us during connecting to v2:172.21.15.118:6800/2112912943, fault. 2024-03-24T03:10:15.771 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T03:10:15.770+0000 7f81ef400640 1 -- 172.21.15.118:0/1693767697 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7f81dc0a3bd0 msgr2=0x7f81dc0b2b10 unknown :-1 s=STATE_CLOSED l=1).mark_down 2024-03-24T03:10:15.771 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T03:10:15.770+0000 7f81ef400640 1 --2- 172.21.15.118:0/1693767697 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7f81dc0a3bd0 0x7f81dc0b2b10 unknown :-1 s=CLOSED pgs=606 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T03:10:15.771 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T03:10:15.770+0000 7f81ef400640 1 --2- 172.21.15.118:0/1693767697 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7f81dc0a3fb0 0x7f81dc0b05c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T03:10:15.772 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T03:10:15.771+0000 7f81f6000640 1 --2- 172.21.15.118:0/1693767697 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7f81dc0a3fb0 0x7f81dc0b05c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T03:10:15.772 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T03:10:15.772+0000 7f81f6000640 1 --2- 172.21.15.118:0/1693767697 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7f81dc0a3fb0 0x7f81dc0b05c0 secure :-1 s=READY pgs=628 cs=0 l=1 rev1=1 crypto rx=0x7f81e0013040 tx=0x7f81e004e040 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T03:10:15.773 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T03:10:15.774+0000 7f81f6000640 1 -- 172.21.15.118:0/659686424 >> [v2:172.21.15.189:6838/4074484168,v1:172.21.15.189:6839/4074484168] conn(0x5622651b71e0 msgr2=0x562265215110 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=0).tick idle (900000776) for more than 900000000 us, fault. 2024-03-24T03:10:15.773 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T03:10:15.774+0000 7f81f6000640 1 --2- 172.21.15.118:0/659686424 >> [v2:172.21.15.189:6838/4074484168,v1:172.21.15.189:6839/4074484168] conn(0x5622651b71e0 0x562265215110 unknown :-1 s=READY pgs=561 cs=9 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault initiating reconnect 2024-03-24T03:10:15.774 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T03:10:15.774+0000 7f81f6000640 1 --2- 172.21.15.118:0/659686424 >> [v2:172.21.15.189:6838/4074484168,v1:172.21.15.189:6839/4074484168] conn(0x5622651b71e0 0x562265215110 unknown :-1 s=BANNER_CONNECTING pgs=561 cs=10 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T03:10:15.774 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T03:10:15.775+0000 7f81f6000640 1 --2- 172.21.15.118:0/659686424 >> [v2:172.21.15.189:6838/4074484168,v1:172.21.15.189:6839/4074484168] conn(0x5622651b71e0 0x562265215110 crc :-1 s=READY pgs=561 cs=10 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=mds.0 client_cookie=76f3d534289740f0 server_cookie=b31a2335a59552de in_seq=0 out_seq=0 2024-03-24T03:10:15.775 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T03:10:15.775+0000 7f81e4c00640 10 client.27094 ms_handle_connect on v2:172.21.15.189:6838/4074484168 2024-03-24T03:10:16.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 03:10:16 smithi118 ceph-mon[31535]: pgmap v5875: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 6 op/s 2024-03-24T03:10:16.453 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 03:10:16 smithi189 ceph-mon[36187]: pgmap v5875: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 6 op/s 2024-03-24T03:10:16.460 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 03:10:16 smithi193 ceph-mon[36878]: pgmap v5875: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 6 op/s 2024-03-24T03:10:17.341 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T03:10:17.341+0000 7f81ee000640 1 -- 172.21.15.118:0/1693767697 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- auth(proto 2 132 bytes epoch 0) v1 -- 0x7f81c80036c0 con 0x7f81f81052d0 2024-03-24T03:10:17.342 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T03:10:17.342+0000 7f81ef400640 1 -- 172.21.15.118:0/1693767697 <== mon.2 v2:172.21.15.189:3300/0 79 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 783+0+0 (secure 0 0 0) 0x7f81c80036c0 con 0x7f81f81052d0 2024-03-24T03:10:17.342 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T03:10:17.342+0000 7f81ef400640 1 -- 172.21.15.118:0/1693767697 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f81c8006590 con 0x7f81f81052d0 2024-03-24T03:10:17.342 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T03:10:17.342+0000 7f81ef400640 1 -- 172.21.15.118:0/1693767697 <== mon.2 v2:172.21.15.189:3300/0 80 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (secure 0 0 0) 0x7f81c8006590 con 0x7f81f81052d0 2024-03-24T03:10:17.344 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T03:10:17.344+0000 7f81db400640 1 -- 172.21.15.118:0/659686424 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- auth(proto 2 132 bytes epoch 0) v1 -- 0x7f81c4006460 con 0x5622651a13f0 2024-03-24T03:10:17.344 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T03:10:17.345+0000 7f81e4c00640 1 -- 172.21.15.118:0/659686424 <== mon.1 v2:172.21.15.193:3300/0 79 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 612+0+0 (secure 0 0 0) 0x7f81c4006460 con 0x5622651a13f0 2024-03-24T03:10:17.344 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T03:10:17.345+0000 7f81e4c00640 1 -- 172.21.15.118:0/659686424 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f81c4006290 con 0x5622651a13f0 2024-03-24T03:10:17.345 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T03:10:17.345+0000 7f81e4c00640 1 -- 172.21.15.118:0/659686424 <== mon.1 v2:172.21.15.193:3300/0 80 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 409+0+0 (secure 0 0 0) 0x7f81c4006290 con 0x5622651a13f0 2024-03-24T03:10:17.445 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 03:10:17 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 9911.755151 secs 2024-03-24T03:10:17.453 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 03:10:17 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 9911.755151 secs 2024-03-24T03:10:17.460 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 03:10:17 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 9911.755151 secs 2024-03-24T03:10:18.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 03:10:18 smithi118 ceph-mon[31535]: pgmap v5876: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-03-24T03:10:18.453 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 03:10:18 smithi189 ceph-mon[36187]: pgmap v5876: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-03-24T03:10:18.460 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 03:10:18 smithi193 ceph-mon[36878]: pgmap v5876: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 5 op/s 2024-03-24T03:10:20.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 03:10:20 smithi118 ceph-mon[31535]: pgmap v5877: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 5 op/s 2024-03-24T03:10:20.453 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 03:10:20 smithi189 ceph-mon[36187]: pgmap v5877: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 5 op/s 2024-03-24T03:10:20.460 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 03:10:20 smithi193 ceph-mon[36878]: pgmap v5877: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 5 op/s 2024-03-24T03:10:22.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 03:10:22 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 9916.755210 secs 2024-03-24T03:10:22.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 03:10:22 smithi118 ceph-mon[31535]: pgmap v5878: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 5 op/s 2024-03-24T03:10:22.453 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 03:10:22 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 9916.755210 secs 2024-03-24T03:10:22.453 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 03:10:22 smithi189 ceph-mon[36187]: pgmap v5878: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 5 op/s 2024-03-24T03:10:22.460 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 03:10:22 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 9916.755210 secs 2024-03-24T03:10:22.460 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 03:10:22 smithi193 ceph-mon[36878]: pgmap v5878: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 5 op/s 2024-03-24T03:10:24.445 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 03:10:24 smithi118 ceph-mon[31535]: pgmap v5879: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T03:10:24.453 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 03:10:24 smithi189 ceph-mon[36187]: pgmap v5879: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T03:10:24.460 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 03:10:24 smithi193 ceph-mon[36878]: pgmap v5879: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T03:10:26.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 03:10:26 smithi118 ceph-mon[31535]: pgmap v5880: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T03:10:26.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 03:10:26 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 9921.755271 secs 2024-03-24T03:10:26.453 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 03:10:26 smithi189 ceph-mon[36187]: pgmap v5880: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T03:10:26.453 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 03:10:26 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 9921.755271 secs 2024-03-24T03:10:26.460 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 03:10:26 smithi193 ceph-mon[36878]: pgmap v5880: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T03:10:26.460 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 03:10:26 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 9921.755271 secs 2024-03-24T03:10:28.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 03:10:28 smithi118 ceph-mon[31535]: pgmap v5881: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:10:28.453 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 03:10:28 smithi189 ceph-mon[36187]: pgmap v5881: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:10:28.460 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 03:10:28 smithi193 ceph-mon[36878]: pgmap v5881: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:10:30.445 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 03:10:30 smithi118 ceph-mon[31535]: pgmap v5882: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T03:10:30.453 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 03:10:30 smithi189 ceph-mon[36187]: pgmap v5882: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T03:10:30.460 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 03:10:30 smithi193 ceph-mon[36878]: pgmap v5882: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T03:10:31.445 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 03:10:31 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 9926.755334 secs 2024-03-24T03:10:31.453 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 03:10:31 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 9926.755334 secs 2024-03-24T03:10:31.460 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 03:10:31 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 9926.755334 secs 2024-03-24T03:10:32.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 03:10:32 smithi118 ceph-mon[31535]: pgmap v5883: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T03:10:32.453 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 03:10:32 smithi189 ceph-mon[36187]: pgmap v5883: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T03:10:32.460 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 03:10:32 smithi193 ceph-mon[36878]: pgmap v5883: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T03:10:34.445 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 03:10:34 smithi118 ceph-mon[31535]: pgmap v5884: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T03:10:34.453 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 03:10:34 smithi189 ceph-mon[36187]: pgmap v5884: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T03:10:34.460 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 03:10:34 smithi193 ceph-mon[36878]: pgmap v5884: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T03:10:36.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 03:10:36 smithi118 ceph-mon[31535]: pgmap v5885: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T03:10:36.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 03:10:36 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 9931.755421 secs 2024-03-24T03:10:36.453 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 03:10:36 smithi189 ceph-mon[36187]: pgmap v5885: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T03:10:36.453 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 03:10:36 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 9931.755421 secs 2024-03-24T03:10:36.460 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 03:10:36 smithi193 ceph-mon[36878]: pgmap v5885: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T03:10:36.460 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 03:10:36 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 9931.755421 secs 2024-03-24T03:10:38.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 03:10:38 smithi118 ceph-mon[31535]: pgmap v5886: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:10:38.453 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 03:10:38 smithi189 ceph-mon[36187]: pgmap v5886: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:10:38.460 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 03:10:38 smithi193 ceph-mon[36878]: pgmap v5886: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:10:40.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 03:10:40 smithi118 ceph-mon[31535]: pgmap v5887: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T03:10:40.453 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 03:10:40 smithi189 ceph-mon[36187]: pgmap v5887: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T03:10:40.460 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 03:10:40 smithi193 ceph-mon[36878]: pgmap v5887: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T03:10:41.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 03:10:41 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 9936.755482 secs 2024-03-24T03:10:41.453 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 03:10:41 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 9936.755482 secs 2024-03-24T03:10:41.460 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 03:10:41 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 9936.755482 secs 2024-03-24T03:10:42.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 03:10:42 smithi118 ceph-mon[31535]: pgmap v5888: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T03:10:42.453 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 03:10:42 smithi189 ceph-mon[36187]: pgmap v5888: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T03:10:42.460 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 03:10:42 smithi193 ceph-mon[36878]: pgmap v5888: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T03:10:44.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 03:10:44 smithi118 ceph-mon[31535]: pgmap v5889: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T03:10:44.453 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 03:10:44 smithi189 ceph-mon[36187]: pgmap v5889: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T03:10:44.460 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 03:10:44 smithi193 ceph-mon[36878]: pgmap v5889: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T03:10:46.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 03:10:46 smithi118 ceph-mon[31535]: pgmap v5890: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T03:10:46.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 03:10:46 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 9941.755522 secs 2024-03-24T03:10:46.453 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 03:10:46 smithi189 ceph-mon[36187]: pgmap v5890: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T03:10:46.453 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 03:10:46 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 9941.755522 secs 2024-03-24T03:10:46.460 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 03:10:46 smithi193 ceph-mon[36878]: pgmap v5890: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T03:10:46.460 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 03:10:46 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 9941.755522 secs 2024-03-24T03:10:48.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 03:10:48 smithi118 ceph-mon[31535]: pgmap v5891: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:10:48.453 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 03:10:48 smithi189 ceph-mon[36187]: pgmap v5891: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:10:48.460 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 03:10:48 smithi193 ceph-mon[36878]: pgmap v5891: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:10:50.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 03:10:50 smithi118 ceph-mon[31535]: pgmap v5892: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T03:10:50.453 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 03:10:50 smithi189 ceph-mon[36187]: pgmap v5892: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T03:10:50.460 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 03:10:50 smithi193 ceph-mon[36878]: pgmap v5892: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T03:10:51.453 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 03:10:51 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 9946.755583 secs 2024-03-24T03:10:51.460 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 03:10:51 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 9946.755583 secs 2024-03-24T03:10:51.695 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 03:10:51 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 9946.755583 secs 2024-03-24T03:10:52.453 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 03:10:52 smithi189 ceph-mon[36187]: pgmap v5893: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T03:10:52.460 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 03:10:52 smithi193 ceph-mon[36878]: pgmap v5893: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T03:10:52.695 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 03:10:52 smithi118 ceph-mon[31535]: pgmap v5893: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T03:10:54.460 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 03:10:54 smithi193 ceph-mon[36878]: pgmap v5894: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T03:10:54.539 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 03:10:54 smithi118 ceph-mon[31535]: pgmap v5894: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T03:10:54.703 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 03:10:54 smithi189 ceph-mon[36187]: pgmap v5894: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T03:10:56.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 03:10:56 smithi118 ceph-mon[31535]: pgmap v5895: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T03:10:56.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 03:10:56 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 9951.755645 secs 2024-03-24T03:10:56.460 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 03:10:56 smithi193 ceph-mon[36878]: pgmap v5895: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T03:10:56.460 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 03:10:56 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 9951.755645 secs 2024-03-24T03:10:56.703 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 03:10:56 smithi189 ceph-mon[36187]: pgmap v5895: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T03:10:56.703 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 03:10:56 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 9951.755645 secs 2024-03-24T03:10:58.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 03:10:58 smithi118 ceph-mon[31535]: pgmap v5896: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:10:58.703 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 03:10:58 smithi189 ceph-mon[36187]: pgmap v5896: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:10:58.710 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 03:10:58 smithi193 ceph-mon[36878]: pgmap v5896: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:11:00.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 03:11:00 smithi118 ceph-mon[31535]: pgmap v5897: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T03:11:00.703 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 03:11:00 smithi189 ceph-mon[36187]: pgmap v5897: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T03:11:00.710 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 03:11:00 smithi193 ceph-mon[36878]: pgmap v5897: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T03:11:01.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 03:11:01 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 9956.755735 secs 2024-03-24T03:11:01.703 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 03:11:01 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 9956.755735 secs 2024-03-24T03:11:01.710 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 03:11:01 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 9956.755735 secs 2024-03-24T03:11:02.695 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 03:11:02 smithi118 ceph-mon[31535]: pgmap v5898: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T03:11:02.703 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 03:11:02 smithi189 ceph-mon[36187]: pgmap v5898: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T03:11:02.710 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 03:11:02 smithi193 ceph-mon[36878]: pgmap v5898: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T03:11:04.695 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 03:11:04 smithi118 ceph-mon[31535]: pgmap v5899: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T03:11:04.703 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 03:11:04 smithi189 ceph-mon[36187]: pgmap v5899: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T03:11:04.710 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 03:11:04 smithi193 ceph-mon[36878]: pgmap v5899: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T03:11:06.505 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 03:11:06 smithi189 ceph-mon[36187]: pgmap v5900: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T03:11:06.505 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 03:11:06 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 9961.755796 secs 2024-03-24T03:11:06.512 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 03:11:06 smithi193 ceph-mon[36878]: pgmap v5900: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T03:11:06.512 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 03:11:06 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 9961.755796 secs 2024-03-24T03:11:06.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 03:11:06 smithi118 ceph-mon[31535]: pgmap v5900: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T03:11:06.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 03:11:06 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 9961.755796 secs 2024-03-24T03:11:08.945 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 03:11:08 smithi118 ceph-mon[31535]: pgmap v5901: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:11:08.953 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 03:11:08 smithi189 ceph-mon[36187]: pgmap v5901: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:11:08.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 03:11:08 smithi193 ceph-mon[36878]: pgmap v5901: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:11:10.872 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 03:11:10 smithi118 ceph-mon[31535]: pgmap v5902: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T03:11:10.953 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 03:11:10 smithi189 ceph-mon[36187]: pgmap v5902: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T03:11:10.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 03:11:10 smithi193 ceph-mon[36878]: pgmap v5902: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T03:11:11.945 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 03:11:11 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 9966.755881 secs 2024-03-24T03:11:11.945 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 03:11:11 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T03:11:11.953 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 03:11:11 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 9966.755881 secs 2024-03-24T03:11:11.953 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 03:11:11 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T03:11:11.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 03:11:11 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 9966.755881 secs 2024-03-24T03:11:11.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 03:11:11 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T03:11:12.945 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 03:11:12 smithi118 ceph-mon[31535]: pgmap v5903: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T03:11:12.953 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 03:11:12 smithi189 ceph-mon[36187]: pgmap v5903: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T03:11:12.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 03:11:12 smithi193 ceph-mon[36878]: pgmap v5903: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T03:11:13.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 03:11:13 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T03:11:13.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 03:11:13 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T03:11:13.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 03:11:13 smithi118 ceph-mon[31535]: pgmap v5904: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 5 op/s 2024-03-24T03:11:13.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 03:11:13 smithi118 ceph-mon[31535]: pgmap v5905: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 5 op/s 2024-03-24T03:11:13.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 03:11:13 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T03:11:13.953 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 03:11:13 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T03:11:13.953 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 03:11:13 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T03:11:13.953 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 03:11:13 smithi189 ceph-mon[36187]: pgmap v5904: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 5 op/s 2024-03-24T03:11:13.953 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 03:11:13 smithi189 ceph-mon[36187]: pgmap v5905: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 5 op/s 2024-03-24T03:11:13.953 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 03:11:13 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T03:11:13.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 03:11:13 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T03:11:13.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 03:11:13 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T03:11:13.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 03:11:13 smithi193 ceph-mon[36878]: pgmap v5904: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 5 op/s 2024-03-24T03:11:13.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 03:11:13 smithi193 ceph-mon[36878]: pgmap v5905: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 5 op/s 2024-03-24T03:11:13.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 03:11:13 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T03:11:15.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 03:11:15 smithi118 ceph-mon[31535]: pgmap v5906: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-03-24T03:11:15.953 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 03:11:15 smithi189 ceph-mon[36187]: pgmap v5906: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-03-24T03:11:15.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 03:11:15 smithi193 ceph-mon[36878]: pgmap v5906: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-03-24T03:11:16.945 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 03:11:16 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 9971.755970 secs 2024-03-24T03:11:16.953 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 03:11:16 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 9971.755970 secs 2024-03-24T03:11:16.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 03:11:16 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 9971.755970 secs 2024-03-24T03:11:17.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 03:11:17 smithi118 ceph-mon[31535]: pgmap v5907: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-03-24T03:11:17.953 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 03:11:17 smithi189 ceph-mon[36187]: pgmap v5907: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-03-24T03:11:17.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 03:11:17 smithi193 ceph-mon[36878]: pgmap v5907: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-03-24T03:11:19.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 03:11:19 smithi118 ceph-mon[31535]: pgmap v5908: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 5 op/s 2024-03-24T03:11:19.953 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 03:11:19 smithi189 ceph-mon[36187]: pgmap v5908: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 5 op/s 2024-03-24T03:11:19.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 03:11:19 smithi193 ceph-mon[36878]: pgmap v5908: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 5 op/s 2024-03-24T03:11:21.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 03:11:21 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 9976.756066 secs 2024-03-24T03:11:21.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 03:11:21 smithi118 ceph-mon[31535]: pgmap v5909: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-03-24T03:11:21.953 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 03:11:21 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 9976.756066 secs 2024-03-24T03:11:21.953 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 03:11:21 smithi189 ceph-mon[36187]: pgmap v5909: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-03-24T03:11:21.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 03:11:21 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 9976.756066 secs 2024-03-24T03:11:21.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 03:11:21 smithi193 ceph-mon[36878]: pgmap v5909: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-03-24T03:11:23.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 03:11:23 smithi118 ceph-mon[31535]: pgmap v5910: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 5 op/s 2024-03-24T03:11:23.953 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 03:11:23 smithi189 ceph-mon[36187]: pgmap v5910: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 5 op/s 2024-03-24T03:11:23.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 03:11:23 smithi193 ceph-mon[36878]: pgmap v5910: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 5 op/s 2024-03-24T03:11:25.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 03:11:25 smithi118 ceph-mon[31535]: pgmap v5911: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T03:11:25.953 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 03:11:25 smithi189 ceph-mon[36187]: pgmap v5911: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T03:11:25.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 03:11:25 smithi193 ceph-mon[36878]: pgmap v5911: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T03:11:26.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 03:11:26 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 9981.756151 secs 2024-03-24T03:11:26.953 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 03:11:26 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 9981.756151 secs 2024-03-24T03:11:26.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 03:11:26 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 9981.756151 secs 2024-03-24T03:11:27.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 03:11:27 smithi118 ceph-mon[31535]: pgmap v5912: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:11:27.953 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 03:11:27 smithi189 ceph-mon[36187]: pgmap v5912: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:11:27.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 03:11:27 smithi193 ceph-mon[36878]: pgmap v5912: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:11:29.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 03:11:29 smithi118 ceph-mon[31535]: pgmap v5913: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T03:11:29.953 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 03:11:29 smithi189 ceph-mon[36187]: pgmap v5913: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T03:11:29.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 03:11:29 smithi193 ceph-mon[36878]: pgmap v5913: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T03:11:31.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 03:11:31 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 9986.756235 secs 2024-03-24T03:11:31.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 03:11:31 smithi118 ceph-mon[31535]: pgmap v5914: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T03:11:31.953 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 03:11:31 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 9986.756235 secs 2024-03-24T03:11:31.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 03:11:31 smithi189 ceph-mon[36187]: pgmap v5914: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T03:11:31.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 03:11:31 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 9986.756235 secs 2024-03-24T03:11:31.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 03:11:31 smithi193 ceph-mon[36878]: pgmap v5914: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T03:11:33.945 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 03:11:33 smithi118 ceph-mon[31535]: pgmap v5915: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T03:11:33.953 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 03:11:33 smithi189 ceph-mon[36187]: pgmap v5915: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T03:11:33.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 03:11:33 smithi193 ceph-mon[36878]: pgmap v5915: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T03:11:35.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 03:11:35 smithi118 ceph-mon[31535]: pgmap v5916: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T03:11:35.953 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 03:11:35 smithi189 ceph-mon[36187]: pgmap v5916: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T03:11:35.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 03:11:35 smithi193 ceph-mon[36878]: pgmap v5916: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T03:11:36.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 03:11:36 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 9991.756295 secs 2024-03-24T03:11:36.953 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 03:11:36 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 9991.756295 secs 2024-03-24T03:11:36.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 03:11:36 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 9991.756295 secs 2024-03-24T03:11:37.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 03:11:37 smithi118 ceph-mon[31535]: pgmap v5917: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:11:37.953 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 03:11:37 smithi189 ceph-mon[36187]: pgmap v5917: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:11:37.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 03:11:37 smithi193 ceph-mon[36878]: pgmap v5917: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:11:39.945 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 03:11:39 smithi118 ceph-mon[31535]: pgmap v5918: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T03:11:39.953 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 03:11:39 smithi189 ceph-mon[36187]: pgmap v5918: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T03:11:39.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 03:11:39 smithi193 ceph-mon[36878]: pgmap v5918: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T03:11:41.945 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 03:11:41 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 9996.756334 secs 2024-03-24T03:11:41.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 03:11:41 smithi118 ceph-mon[31535]: pgmap v5919: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 6 op/s 2024-03-24T03:11:41.953 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 03:11:41 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 9996.756334 secs 2024-03-24T03:11:41.953 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 03:11:41 smithi189 ceph-mon[36187]: pgmap v5919: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 6 op/s 2024-03-24T03:11:41.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 03:11:41 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 9996.756334 secs 2024-03-24T03:11:41.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 03:11:41 smithi193 ceph-mon[36878]: pgmap v5919: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 6 op/s 2024-03-24T03:11:43.953 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 03:11:43 smithi189 ceph-mon[36187]: pgmap v5920: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 5 op/s 2024-03-24T03:11:43.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 03:11:43 smithi193 ceph-mon[36878]: pgmap v5920: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 5 op/s 2024-03-24T03:11:44.195 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 03:11:43 smithi118 ceph-mon[31535]: pgmap v5920: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 5 op/s 2024-03-24T03:11:45.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 03:11:45 smithi193 ceph-mon[36878]: pgmap v5921: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-03-24T03:11:46.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 03:11:45 smithi118 ceph-mon[31535]: pgmap v5921: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-03-24T03:11:46.203 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 03:11:45 smithi189 ceph-mon[36187]: pgmap v5921: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-03-24T03:11:47.195 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 03:11:46 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 10001.756433 secs 2024-03-24T03:11:47.203 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 03:11:46 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 10001.756433 secs 2024-03-24T03:11:47.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 03:11:46 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 10001.756433 secs 2024-03-24T03:11:48.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 03:11:47 smithi118 ceph-mon[31535]: pgmap v5922: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-03-24T03:11:48.203 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 03:11:47 smithi189 ceph-mon[36187]: pgmap v5922: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-03-24T03:11:48.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 03:11:47 smithi193 ceph-mon[36878]: pgmap v5922: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-03-24T03:11:50.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 03:11:49 smithi118 ceph-mon[31535]: pgmap v5923: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 5 op/s 2024-03-24T03:11:50.203 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 03:11:49 smithi189 ceph-mon[36187]: pgmap v5923: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 5 op/s 2024-03-24T03:11:50.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 03:11:49 smithi193 ceph-mon[36878]: pgmap v5923: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 5 op/s 2024-03-24T03:11:52.195 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 03:11:51 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 10006.756524 secs 2024-03-24T03:11:52.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 03:11:51 smithi118 ceph-mon[31535]: pgmap v5924: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 5 op/s 2024-03-24T03:11:52.203 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 03:11:51 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 10006.756524 secs 2024-03-24T03:11:52.203 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 03:11:51 smithi189 ceph-mon[36187]: pgmap v5924: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 5 op/s 2024-03-24T03:11:52.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 03:11:51 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 10006.756524 secs 2024-03-24T03:11:52.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 03:11:51 smithi193 ceph-mon[36878]: pgmap v5924: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 5 op/s 2024-03-24T03:11:53.881 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 03:11:53 smithi118 ceph-mon[31535]: pgmap v5925: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T03:11:53.953 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 03:11:53 smithi189 ceph-mon[36187]: pgmap v5925: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T03:11:53.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 03:11:53 smithi193 ceph-mon[36878]: pgmap v5925: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T03:11:55.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 03:11:55 smithi118 ceph-mon[31535]: pgmap v5926: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T03:11:55.953 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 03:11:55 smithi189 ceph-mon[36187]: pgmap v5926: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T03:11:55.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 03:11:55 smithi193 ceph-mon[36878]: pgmap v5926: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T03:11:56.945 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 03:11:56 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 10011.756583 secs 2024-03-24T03:11:56.953 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 03:11:56 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 10011.756583 secs 2024-03-24T03:11:56.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 03:11:56 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 10011.756583 secs 2024-03-24T03:11:57.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 03:11:57 smithi118 ceph-mon[31535]: pgmap v5927: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:11:57.953 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 03:11:57 smithi189 ceph-mon[36187]: pgmap v5927: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:11:57.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 03:11:57 smithi193 ceph-mon[36878]: pgmap v5927: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:11:59.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 03:11:59 smithi118 ceph-mon[31535]: pgmap v5928: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T03:11:59.953 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 03:11:59 smithi189 ceph-mon[36187]: pgmap v5928: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T03:11:59.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 03:11:59 smithi193 ceph-mon[36878]: pgmap v5928: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T03:12:01.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 03:12:01 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 10016.756625 secs 2024-03-24T03:12:01.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 03:12:01 smithi118 ceph-mon[31535]: pgmap v5929: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T03:12:01.953 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 03:12:01 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 10016.756625 secs 2024-03-24T03:12:01.953 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 03:12:01 smithi189 ceph-mon[36187]: pgmap v5929: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T03:12:01.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 03:12:01 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 10016.756625 secs 2024-03-24T03:12:01.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 03:12:01 smithi193 ceph-mon[36878]: pgmap v5929: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T03:12:03.945 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 03:12:03 smithi118 ceph-mon[31535]: pgmap v5930: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T03:12:03.953 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 03:12:03 smithi189 ceph-mon[36187]: pgmap v5930: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T03:12:03.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 03:12:03 smithi193 ceph-mon[36878]: pgmap v5930: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T03:12:05.945 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 03:12:05 smithi118 ceph-mon[31535]: pgmap v5931: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-03-24T03:12:05.953 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 03:12:05 smithi189 ceph-mon[36187]: pgmap v5931: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-03-24T03:12:05.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 03:12:05 smithi193 ceph-mon[36878]: pgmap v5931: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-03-24T03:12:06.945 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 03:12:06 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 10021.756713 secs 2024-03-24T03:12:06.953 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 03:12:06 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 10021.756713 secs 2024-03-24T03:12:06.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 03:12:06 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 10021.756713 secs 2024-03-24T03:12:07.945 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 03:12:07 smithi118 ceph-mon[31535]: pgmap v5932: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:12:07.953 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 03:12:07 smithi189 ceph-mon[36187]: pgmap v5932: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:12:07.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 03:12:07 smithi193 ceph-mon[36878]: pgmap v5932: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:12:09.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 03:12:09 smithi118 ceph-mon[31535]: pgmap v5933: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T03:12:09.953 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 03:12:09 smithi189 ceph-mon[36187]: pgmap v5933: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T03:12:09.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 03:12:09 smithi193 ceph-mon[36878]: pgmap v5933: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T03:12:11.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 03:12:11 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 10026.756804 secs 2024-03-24T03:12:11.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 03:12:11 smithi118 ceph-mon[31535]: pgmap v5934: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T03:12:11.953 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 03:12:11 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 10026.756804 secs 2024-03-24T03:12:11.953 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 03:12:11 smithi189 ceph-mon[36187]: pgmap v5934: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T03:12:11.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 03:12:11 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 10026.756804 secs 2024-03-24T03:12:11.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 03:12:11 smithi193 ceph-mon[36878]: pgmap v5934: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T03:12:13.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 03:12:13 smithi118 ceph-mon[31535]: pgmap v5935: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T03:12:13.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 03:12:13 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T03:12:13.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 03:12:13 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T03:12:13.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 03:12:13 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T03:12:13.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 03:12:13 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T03:12:13.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 03:12:13 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T03:12:13.953 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 03:12:13 smithi189 ceph-mon[36187]: pgmap v5935: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T03:12:13.953 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 03:12:13 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T03:12:13.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 03:12:13 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T03:12:13.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 03:12:13 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T03:12:13.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 03:12:13 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T03:12:13.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 03:12:13 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T03:12:13.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 03:12:13 smithi193 ceph-mon[36878]: pgmap v5935: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T03:12:13.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 03:12:13 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T03:12:13.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 03:12:13 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T03:12:13.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 03:12:13 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T03:12:13.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 03:12:13 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T03:12:13.961 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 03:12:13 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T03:12:15.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 03:12:15 smithi118 ceph-mon[31535]: pgmap v5936: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-03-24T03:12:15.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 03:12:15 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T03:12:15.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 03:12:15 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T03:12:15.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 03:12:15 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T03:12:15.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 03:12:15 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T03:12:15.953 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 03:12:15 smithi189 ceph-mon[36187]: pgmap v5936: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-03-24T03:12:15.953 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 03:12:15 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T03:12:15.953 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 03:12:15 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T03:12:15.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 03:12:15 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T03:12:15.954 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 03:12:15 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T03:12:15.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 03:12:15 smithi193 ceph-mon[36878]: pgmap v5936: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-03-24T03:12:15.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 03:12:15 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T03:12:15.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 03:12:15 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T03:12:15.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 03:12:15 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T03:12:15.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 03:12:15 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T03:12:16.945 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 03:12:16 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 10031.756844 secs 2024-03-24T03:12:16.953 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 03:12:16 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 10031.756844 secs 2024-03-24T03:12:16.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 03:12:16 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 10031.756844 secs 2024-03-24T03:12:17.945 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 03:12:17 smithi118 ceph-mon[31535]: pgmap v5937: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:12:17.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 03:12:17 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T03:12:17.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 03:12:17 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T03:12:17.953 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 03:12:17 smithi189 ceph-mon[36187]: pgmap v5937: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:12:17.953 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 03:12:17 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T03:12:17.953 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 03:12:17 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T03:12:17.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 03:12:17 smithi193 ceph-mon[36878]: pgmap v5937: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:12:17.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 03:12:17 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T03:12:17.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 03:12:17 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T03:12:19.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 03:12:19 smithi118 ceph-mon[31535]: pgmap v5938: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T03:12:19.953 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 03:12:19 smithi189 ceph-mon[36187]: pgmap v5938: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T03:12:19.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 03:12:19 smithi193 ceph-mon[36878]: pgmap v5938: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T03:12:21.953 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 03:12:21 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 10036.756931 secs 2024-03-24T03:12:21.953 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 03:12:21 smithi189 ceph-mon[36187]: pgmap v5939: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T03:12:21.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 03:12:21 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 10036.756931 secs 2024-03-24T03:12:21.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 03:12:21 smithi193 ceph-mon[36878]: pgmap v5939: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T03:12:22.195 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 03:12:21 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 10036.756931 secs 2024-03-24T03:12:22.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 03:12:21 smithi118 ceph-mon[31535]: pgmap v5939: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T03:12:23.953 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 03:12:23 smithi189 ceph-mon[36187]: pgmap v5940: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T03:12:23.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 03:12:23 smithi193 ceph-mon[36878]: pgmap v5940: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T03:12:24.145 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 03:12:23 smithi118 ceph-mon[31535]: pgmap v5940: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T03:12:25.415 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 03:12:25 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T03:12:25.415 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 03:12:25 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T03:12:25.415 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 03:12:25 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T03:12:25.415 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 03:12:25 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T03:12:25.415 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 03:12:25 smithi118 ceph-mon[31535]: pgmap v5941: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 7 op/s 2024-03-24T03:12:25.415 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 03:12:25 smithi118 ceph-mon[31535]: pgmap v5942: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 6 op/s 2024-03-24T03:12:25.415 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 03:12:25 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T03:12:25.703 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 03:12:25 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T03:12:25.703 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 03:12:25 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T03:12:25.703 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 03:12:25 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T03:12:25.703 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 03:12:25 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T03:12:25.703 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 03:12:25 smithi189 ceph-mon[36187]: pgmap v5941: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 7 op/s 2024-03-24T03:12:25.703 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 03:12:25 smithi189 ceph-mon[36187]: pgmap v5942: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 6 op/s 2024-03-24T03:12:25.704 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 03:12:25 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T03:12:25.710 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 03:12:25 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T03:12:25.710 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 03:12:25 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T03:12:25.710 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 03:12:25 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T03:12:25.710 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 03:12:25 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T03:12:25.710 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 03:12:25 smithi193 ceph-mon[36878]: pgmap v5941: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 7 op/s 2024-03-24T03:12:25.710 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 03:12:25 smithi193 ceph-mon[36878]: pgmap v5942: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 6 op/s 2024-03-24T03:12:25.710 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 03:12:25 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T03:12:26.445 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 03:12:26 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 10041.757020 secs 2024-03-24T03:12:26.703 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 03:12:26 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 10041.757020 secs 2024-03-24T03:12:26.710 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 03:12:26 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 10041.757020 secs 2024-03-24T03:12:27.695 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 03:12:27 smithi118 ceph-mon[31535]: pgmap v5943: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 6 op/s 2024-03-24T03:12:27.703 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 03:12:27 smithi189 ceph-mon[36187]: pgmap v5943: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 6 op/s 2024-03-24T03:12:27.710 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 03:12:27 smithi193 ceph-mon[36878]: pgmap v5943: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 6 op/s 2024-03-24T03:12:29.695 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 03:12:29 smithi118 ceph-mon[31535]: pgmap v5944: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 6 op/s 2024-03-24T03:12:29.703 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 03:12:29 smithi189 ceph-mon[36187]: pgmap v5944: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 6 op/s 2024-03-24T03:12:29.710 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 03:12:29 smithi193 ceph-mon[36878]: pgmap v5944: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 6 op/s 2024-03-24T03:12:31.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 03:12:31 smithi118 ceph-mon[31535]: pgmap v5945: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 6 op/s 2024-03-24T03:12:31.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 03:12:31 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 10046.757071 secs 2024-03-24T03:12:31.703 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 03:12:31 smithi189 ceph-mon[36187]: pgmap v5945: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 6 op/s 2024-03-24T03:12:31.703 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 03:12:31 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 10046.757071 secs 2024-03-24T03:12:31.710 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 03:12:31 smithi193 ceph-mon[36878]: pgmap v5945: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 6 op/s 2024-03-24T03:12:31.710 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 03:12:31 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 10046.757071 secs 2024-03-24T03:12:33.697 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 03:12:33 smithi118 ceph-mon[31535]: pgmap v5946: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 5 op/s 2024-03-24T03:12:33.703 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 03:12:33 smithi189 ceph-mon[36187]: pgmap v5946: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 5 op/s 2024-03-24T03:12:33.710 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 03:12:33 smithi193 ceph-mon[36878]: pgmap v5946: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 5 op/s 2024-03-24T03:12:35.695 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 03:12:35 smithi118 ceph-mon[31535]: pgmap v5947: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 5 op/s 2024-03-24T03:12:35.703 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 03:12:35 smithi189 ceph-mon[36187]: pgmap v5947: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 5 op/s 2024-03-24T03:12:35.710 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 03:12:35 smithi193 ceph-mon[36878]: pgmap v5947: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 5 op/s 2024-03-24T03:12:36.604 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 03:12:36 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 10051.757158 secs 2024-03-24T03:12:36.611 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 03:12:36 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 10051.757158 secs 2024-03-24T03:12:36.695 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 03:12:36 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 10051.757158 secs 2024-03-24T03:12:37.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 03:12:37 smithi118 ceph-mon[31535]: pgmap v5948: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:12:37.617 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 03:12:37 smithi189 ceph-mon[36187]: pgmap v5948: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:12:37.631 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 03:12:37 smithi193 ceph-mon[36878]: pgmap v5948: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:12:39.695 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 03:12:39 smithi118 ceph-mon[31535]: pgmap v5949: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T03:12:39.703 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 03:12:39 smithi189 ceph-mon[36187]: pgmap v5949: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T03:12:39.710 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 03:12:39 smithi193 ceph-mon[36878]: pgmap v5949: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T03:12:41.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 03:12:41 smithi118 ceph-mon[31535]: pgmap v5950: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T03:12:41.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 03:12:41 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 10056.757196 secs 2024-03-24T03:12:41.703 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 03:12:41 smithi189 ceph-mon[36187]: pgmap v5950: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T03:12:41.703 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 03:12:41 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 10056.757196 secs 2024-03-24T03:12:41.710 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 03:12:41 smithi193 ceph-mon[36878]: pgmap v5950: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T03:12:41.710 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 03:12:41 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 10056.757196 secs 2024-03-24T03:12:43.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 03:12:43 smithi118 ceph-mon[31535]: pgmap v5951: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:12:43.703 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 03:12:43 smithi189 ceph-mon[36187]: pgmap v5951: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:12:43.710 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 03:12:43 smithi193 ceph-mon[36878]: pgmap v5951: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:12:45.695 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 03:12:45 smithi118 ceph-mon[31535]: pgmap v5952: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-24T03:12:45.703 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 03:12:45 smithi189 ceph-mon[36187]: pgmap v5952: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-24T03:12:45.710 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 03:12:45 smithi193 ceph-mon[36878]: pgmap v5952: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-24T03:12:46.695 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 03:12:46 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 10061.757280 secs 2024-03-24T03:12:46.703 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 03:12:46 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 10061.757280 secs 2024-03-24T03:12:46.710 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 03:12:46 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 10061.757280 secs 2024-03-24T03:12:47.695 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 03:12:47 smithi118 ceph-mon[31535]: pgmap v5953: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:12:47.703 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 03:12:47 smithi189 ceph-mon[36187]: pgmap v5953: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:12:47.710 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 03:12:47 smithi193 ceph-mon[36878]: pgmap v5953: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:12:49.695 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 03:12:49 smithi118 ceph-mon[31535]: pgmap v5954: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T03:12:49.703 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 03:12:49 smithi189 ceph-mon[36187]: pgmap v5954: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T03:12:49.710 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 03:12:49 smithi193 ceph-mon[36878]: pgmap v5954: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T03:12:51.695 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 03:12:51 smithi118 ceph-mon[31535]: pgmap v5955: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T03:12:51.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 03:12:51 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 10066.757363 secs 2024-03-24T03:12:51.703 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 03:12:51 smithi189 ceph-mon[36187]: pgmap v5955: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T03:12:51.703 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 03:12:51 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 10066.757363 secs 2024-03-24T03:12:51.710 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 03:12:51 smithi193 ceph-mon[36878]: pgmap v5955: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T03:12:51.710 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 03:12:51 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 10066.757363 secs 2024-03-24T03:12:53.695 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 03:12:53 smithi118 ceph-mon[31535]: pgmap v5956: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:12:53.703 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 03:12:53 smithi189 ceph-mon[36187]: pgmap v5956: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:12:53.710 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 03:12:53 smithi193 ceph-mon[36878]: pgmap v5956: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:12:55.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 03:12:55 smithi118 ceph-mon[31535]: pgmap v5957: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-24T03:12:55.703 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 03:12:55 smithi189 ceph-mon[36187]: pgmap v5957: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-24T03:12:55.710 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 03:12:55 smithi193 ceph-mon[36878]: pgmap v5957: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-24T03:12:56.601 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 03:12:56 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 10071.757424 secs 2024-03-24T03:12:56.703 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 03:12:56 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 10071.757424 secs 2024-03-24T03:12:56.710 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 03:12:56 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 10071.757424 secs 2024-03-24T03:12:57.445 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 03:12:57 smithi118 ceph-mon[31535]: pgmap v5958: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:12:57.703 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 03:12:57 smithi189 ceph-mon[36187]: pgmap v5958: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:12:57.710 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 03:12:57 smithi193 ceph-mon[36878]: pgmap v5958: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:12:59.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 03:12:59 smithi118 ceph-mon[31535]: pgmap v5959: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T03:12:59.703 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 03:12:59 smithi189 ceph-mon[36187]: pgmap v5959: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T03:12:59.710 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 03:12:59 smithi193 ceph-mon[36878]: pgmap v5959: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T03:13:01.695 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 03:13:01 smithi118 ceph-mon[31535]: pgmap v5960: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T03:13:01.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 03:13:01 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 10076.757510 secs 2024-03-24T03:13:01.703 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 03:13:01 smithi189 ceph-mon[36187]: pgmap v5960: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T03:13:01.703 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 03:13:01 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 10076.757510 secs 2024-03-24T03:13:01.710 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 03:13:01 smithi193 ceph-mon[36878]: pgmap v5960: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T03:13:01.710 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 03:13:01 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 10076.757510 secs 2024-03-24T03:13:03.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 03:13:03 smithi118 ceph-mon[31535]: pgmap v5961: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:13:03.705 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 03:13:03 smithi189 ceph-mon[36187]: pgmap v5961: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:13:03.710 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 03:13:03 smithi193 ceph-mon[36878]: pgmap v5961: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:13:05.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 03:13:05 smithi118 ceph-mon[31535]: pgmap v5962: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-24T03:13:05.703 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 03:13:05 smithi189 ceph-mon[36187]: pgmap v5962: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-24T03:13:05.710 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 03:13:05 smithi193 ceph-mon[36878]: pgmap v5962: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-24T03:13:06.695 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 03:13:06 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 10081.757597 secs 2024-03-24T03:13:06.703 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 03:13:06 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 10081.757597 secs 2024-03-24T03:13:06.710 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 03:13:06 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 10081.757597 secs 2024-03-24T03:13:07.695 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 03:13:07 smithi118 ceph-mon[31535]: pgmap v5963: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:13:07.703 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 03:13:07 smithi189 ceph-mon[36187]: pgmap v5963: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:13:07.710 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 03:13:07 smithi193 ceph-mon[36878]: pgmap v5963: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:13:09.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 03:13:09 smithi118 ceph-mon[31535]: pgmap v5964: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T03:13:09.703 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 03:13:09 smithi189 ceph-mon[36187]: pgmap v5964: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T03:13:09.710 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 03:13:09 smithi193 ceph-mon[36878]: pgmap v5964: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T03:13:11.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 03:13:11 smithi118 ceph-mon[31535]: pgmap v5965: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T03:13:11.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 03:13:11 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 10086.757644 secs 2024-03-24T03:13:11.703 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 03:13:11 smithi189 ceph-mon[36187]: pgmap v5965: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T03:13:11.703 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 03:13:11 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 10086.757644 secs 2024-03-24T03:13:11.710 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 03:13:11 smithi193 ceph-mon[36878]: pgmap v5965: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T03:13:11.710 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 03:13:11 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 10086.757644 secs 2024-03-24T03:13:13.703 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 03:13:13 smithi189 ceph-mon[36187]: pgmap v5966: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:13:13.710 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 03:13:13 smithi193 ceph-mon[36878]: pgmap v5966: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:13:13.945 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 03:13:13 smithi118 ceph-mon[31535]: pgmap v5966: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:13:15.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 03:13:15 smithi118 ceph-mon[31535]: pgmap v5967: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-24T03:13:15.703 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 03:13:15 smithi189 ceph-mon[36187]: pgmap v5967: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-24T03:13:15.710 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 03:13:15 smithi193 ceph-mon[36878]: pgmap v5967: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-24T03:13:16.710 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 03:13:16 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 10091.757731 secs 2024-03-24T03:13:16.945 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 03:13:16 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 10091.757731 secs 2024-03-24T03:13:16.953 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 03:13:16 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 10091.757731 secs 2024-03-24T03:13:17.371 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T03:13:17.370+0000 7f81ee000640 1 -- 172.21.15.118:0/1693767697 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f81c8006590 con 0x7f81f81052d0 2024-03-24T03:13:17.371 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T03:13:17.371+0000 7f81db400640 1 -- 172.21.15.118:0/659686424 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f81c4006290 con 0x5622651a13f0 2024-03-24T03:13:17.371 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T03:13:17.371+0000 7f81ef400640 1 -- 172.21.15.118:0/1693767697 <== mon.2 v2:172.21.15.189:3300/0 81 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (secure 0 0 0) 0x7f81c8006590 con 0x7f81f81052d0 2024-03-24T03:13:17.371 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T03:13:17.371+0000 7f81e4c00640 1 -- 172.21.15.118:0/659686424 <== mon.1 v2:172.21.15.193:3300/0 81 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 409+0+0 (secure 0 0 0) 0x7f81c4006290 con 0x5622651a13f0 2024-03-24T03:13:17.945 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 03:13:17 smithi118 ceph-mon[31535]: pgmap v5968: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:13:17.953 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 03:13:17 smithi189 ceph-mon[36187]: pgmap v5968: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:13:17.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 03:13:17 smithi193 ceph-mon[36878]: pgmap v5968: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:13:19.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 03:13:19 smithi118 ceph-mon[31535]: pgmap v5969: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T03:13:19.953 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 03:13:19 smithi189 ceph-mon[36187]: pgmap v5969: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T03:13:19.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 03:13:19 smithi193 ceph-mon[36878]: pgmap v5969: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T03:13:21.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 03:13:21 smithi118 ceph-mon[31535]: pgmap v5970: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T03:13:21.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 03:13:21 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 10096.757811 secs 2024-03-24T03:13:21.953 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 03:13:21 smithi189 ceph-mon[36187]: pgmap v5970: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T03:13:21.953 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 03:13:21 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 10096.757811 secs 2024-03-24T03:13:21.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 03:13:21 smithi193 ceph-mon[36878]: pgmap v5970: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T03:13:21.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 03:13:21 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 10096.757811 secs 2024-03-24T03:13:23.945 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 03:13:23 smithi118 ceph-mon[31535]: pgmap v5971: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:13:23.953 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 03:13:23 smithi189 ceph-mon[36187]: pgmap v5971: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:13:23.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 03:13:23 smithi193 ceph-mon[36878]: pgmap v5971: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:13:25.748 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 03:13:25 smithi118 ceph-mon[31535]: pgmap v5972: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-24T03:13:25.748 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 03:13:25 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T03:13:25.748 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 03:13:25 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T03:13:25.748 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 03:13:25 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T03:13:25.953 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 03:13:25 smithi189 ceph-mon[36187]: pgmap v5972: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-24T03:13:25.953 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 03:13:25 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T03:13:25.953 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 03:13:25 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T03:13:25.953 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 03:13:25 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T03:13:25.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 03:13:25 smithi193 ceph-mon[36878]: pgmap v5972: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-24T03:13:25.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 03:13:25 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T03:13:25.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 03:13:25 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T03:13:25.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 03:13:25 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T03:13:26.703 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 03:13:26 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 10101.757827 secs 2024-03-24T03:13:26.841 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 03:13:26 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 10101.757827 secs 2024-03-24T03:13:26.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 03:13:26 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 10101.757827 secs 2024-03-24T03:13:27.586 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 03:13:27 smithi189 ceph-mon[36187]: pgmap v5973: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:13:27.923 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 03:13:27 smithi193 ceph-mon[36878]: pgmap v5973: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:13:27.945 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 03:13:27 smithi118 ceph-mon[31535]: pgmap v5973: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:13:29.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 03:13:29 smithi118 ceph-mon[31535]: pgmap v5974: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T03:13:29.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 03:13:29 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T03:13:29.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 03:13:29 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T03:13:29.953 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 03:13:29 smithi189 ceph-mon[36187]: pgmap v5974: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T03:13:29.953 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 03:13:29 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T03:13:29.953 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 03:13:29 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T03:13:29.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 03:13:29 smithi193 ceph-mon[36878]: pgmap v5974: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T03:13:29.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 03:13:29 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T03:13:29.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 03:13:29 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T03:13:30.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 03:13:30 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T03:13:30.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 03:13:30 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T03:13:30.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 03:13:30 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T03:13:30.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 03:13:30 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T03:13:30.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 03:13:30 smithi118 ceph-mon[31535]: pgmap v5975: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 6 op/s 2024-03-24T03:13:30.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 03:13:30 smithi118 ceph-mon[31535]: pgmap v5976: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-03-24T03:13:30.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 03:13:30 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T03:13:30.953 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 03:13:30 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T03:13:30.953 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 03:13:30 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T03:13:30.953 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 03:13:30 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T03:13:30.953 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 03:13:30 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T03:13:30.953 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 03:13:30 smithi189 ceph-mon[36187]: pgmap v5975: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 6 op/s 2024-03-24T03:13:30.953 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 03:13:30 smithi189 ceph-mon[36187]: pgmap v5976: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-03-24T03:13:30.953 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 03:13:30 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T03:13:30.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 03:13:30 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T03:13:30.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 03:13:30 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T03:13:30.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 03:13:30 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T03:13:30.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 03:13:30 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T03:13:30.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 03:13:30 smithi193 ceph-mon[36878]: pgmap v5975: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 6 op/s 2024-03-24T03:13:30.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 03:13:30 smithi193 ceph-mon[36878]: pgmap v5976: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-03-24T03:13:30.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 03:13:30 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T03:13:31.945 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 03:13:31 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 10106.757889 secs 2024-03-24T03:13:31.953 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 03:13:31 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 10106.757889 secs 2024-03-24T03:13:31.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 03:13:31 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 10106.757889 secs 2024-03-24T03:13:32.945 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 03:13:32 smithi118 ceph-mon[31535]: pgmap v5977: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-03-24T03:13:32.953 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 03:13:32 smithi189 ceph-mon[36187]: pgmap v5977: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-03-24T03:13:32.960 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 03:13:32 smithi193 ceph-mon[36878]: pgmap v5977: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-03-24T03:13:35.195 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 03:13:34 smithi118 ceph-mon[31535]: pgmap v5978: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 4 op/s 2024-03-24T03:13:35.203 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 03:13:34 smithi189 ceph-mon[36187]: pgmap v5978: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 4 op/s 2024-03-24T03:13:35.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 03:13:34 smithi193 ceph-mon[36878]: pgmap v5978: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 4 op/s 2024-03-24T03:13:37.005 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 03:13:36 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 10111.757974 secs 2024-03-24T03:13:37.005 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 03:13:36 smithi189 ceph-mon[36187]: pgmap v5979: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-03-24T03:13:37.012 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 03:13:36 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 10111.757974 secs 2024-03-24T03:13:37.012 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 03:13:36 smithi193 ceph-mon[36878]: pgmap v5979: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-03-24T03:13:37.195 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 03:13:36 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 10111.757974 secs 2024-03-24T03:13:37.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 03:13:36 smithi118 ceph-mon[31535]: pgmap v5979: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-03-24T03:13:39.057 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 03:13:38 smithi189 ceph-mon[36187]: pgmap v5980: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 5 op/s 2024-03-24T03:13:39.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 03:13:38 smithi118 ceph-mon[31535]: pgmap v5980: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 5 op/s 2024-03-24T03:13:39.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 03:13:38 smithi193 ceph-mon[36878]: pgmap v5980: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 5 op/s 2024-03-24T03:13:41.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 03:13:40 smithi118 ceph-mon[31535]: pgmap v5981: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 5 op/s 2024-03-24T03:13:41.203 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 03:13:40 smithi189 ceph-mon[36187]: pgmap v5981: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 5 op/s 2024-03-24T03:13:41.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 03:13:40 smithi193 ceph-mon[36878]: pgmap v5981: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 5 op/s 2024-03-24T03:13:42.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 03:13:41 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 10116.758035 secs 2024-03-24T03:13:42.203 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 03:13:41 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 10116.758035 secs 2024-03-24T03:13:42.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 03:13:41 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 10116.758035 secs 2024-03-24T03:13:43.195 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 03:13:42 smithi118 ceph-mon[31535]: pgmap v5982: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:13:43.203 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 03:13:42 smithi189 ceph-mon[36187]: pgmap v5982: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:13:43.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 03:13:42 smithi193 ceph-mon[36878]: pgmap v5982: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:13:45.195 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 03:13:44 smithi118 ceph-mon[31535]: pgmap v5983: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T03:13:45.203 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 03:13:44 smithi189 ceph-mon[36187]: pgmap v5983: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T03:13:45.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 03:13:44 smithi193 ceph-mon[36878]: pgmap v5983: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T03:13:47.032 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 03:13:46 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 10121.758123 secs 2024-03-24T03:13:47.032 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 03:13:46 smithi118 ceph-mon[31535]: pgmap v5984: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T03:13:47.203 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 03:13:46 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 10121.758123 secs 2024-03-24T03:13:47.203 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 03:13:46 smithi189 ceph-mon[36187]: pgmap v5984: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T03:13:47.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 03:13:46 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 10121.758123 secs 2024-03-24T03:13:47.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 03:13:46 smithi193 ceph-mon[36878]: pgmap v5984: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T03:13:49.195 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 03:13:48 smithi118 ceph-mon[31535]: pgmap v5985: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T03:13:49.203 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 03:13:48 smithi189 ceph-mon[36187]: pgmap v5985: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T03:13:49.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 03:13:48 smithi193 ceph-mon[36878]: pgmap v5985: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T03:13:51.195 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 03:13:50 smithi118 ceph-mon[31535]: pgmap v5986: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-03-24T03:13:51.203 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 03:13:50 smithi189 ceph-mon[36187]: pgmap v5986: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-03-24T03:13:51.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 03:13:50 smithi193 ceph-mon[36878]: pgmap v5986: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-03-24T03:13:52.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 03:13:51 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 10126.758209 secs 2024-03-24T03:13:52.203 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 03:13:51 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 10126.758209 secs 2024-03-24T03:13:52.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 03:13:51 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 10126.758209 secs 2024-03-24T03:13:53.195 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 03:13:52 smithi118 ceph-mon[31535]: pgmap v5987: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:13:53.203 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 03:13:52 smithi189 ceph-mon[36187]: pgmap v5987: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:13:53.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 03:13:52 smithi193 ceph-mon[36878]: pgmap v5987: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:13:55.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 03:13:54 smithi118 ceph-mon[31535]: pgmap v5988: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T03:13:55.203 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 03:13:54 smithi189 ceph-mon[36187]: pgmap v5988: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T03:13:55.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 03:13:54 smithi193 ceph-mon[36878]: pgmap v5988: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T03:13:57.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 03:13:56 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 10131.758271 secs 2024-03-24T03:13:57.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 03:13:56 smithi118 ceph-mon[31535]: pgmap v5989: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T03:13:57.203 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 03:13:56 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 10131.758271 secs 2024-03-24T03:13:57.203 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 03:13:56 smithi189 ceph-mon[36187]: pgmap v5989: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T03:13:57.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 03:13:56 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 10131.758271 secs 2024-03-24T03:13:57.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 03:13:56 smithi193 ceph-mon[36878]: pgmap v5989: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T03:13:59.195 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 03:13:58 smithi118 ceph-mon[31535]: pgmap v5990: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T03:13:59.203 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 03:13:58 smithi189 ceph-mon[36187]: pgmap v5990: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T03:13:59.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 03:13:58 smithi193 ceph-mon[36878]: pgmap v5990: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T03:14:01.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 03:14:00 smithi118 ceph-mon[31535]: pgmap v5991: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-03-24T03:14:01.203 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 03:14:00 smithi189 ceph-mon[36187]: pgmap v5991: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-03-24T03:14:01.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 03:14:00 smithi193 ceph-mon[36878]: pgmap v5991: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-03-24T03:14:02.195 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 03:14:01 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 10136.758341 secs 2024-03-24T03:14:02.203 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 03:14:01 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 10136.758341 secs 2024-03-24T03:14:02.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 03:14:01 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 10136.758341 secs 2024-03-24T03:14:03.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 03:14:02 smithi118 ceph-mon[31535]: pgmap v5992: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:14:03.203 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 03:14:02 smithi189 ceph-mon[36187]: pgmap v5992: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:14:03.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 03:14:02 smithi193 ceph-mon[36878]: pgmap v5992: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:14:05.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 03:14:04 smithi118 ceph-mon[31535]: pgmap v5993: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T03:14:05.203 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 03:14:04 smithi189 ceph-mon[36187]: pgmap v5993: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T03:14:05.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 03:14:04 smithi193 ceph-mon[36878]: pgmap v5993: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T03:14:07.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 03:14:06 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 10141.758401 secs 2024-03-24T03:14:07.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 03:14:06 smithi118 ceph-mon[31535]: pgmap v5994: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T03:14:07.203 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 03:14:06 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 10141.758401 secs 2024-03-24T03:14:07.203 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 03:14:06 smithi189 ceph-mon[36187]: pgmap v5994: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T03:14:07.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 03:14:06 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 10141.758401 secs 2024-03-24T03:14:07.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 03:14:06 smithi193 ceph-mon[36878]: pgmap v5994: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T03:14:09.195 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 03:14:08 smithi118 ceph-mon[31535]: pgmap v5995: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T03:14:09.203 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 03:14:08 smithi189 ceph-mon[36187]: pgmap v5995: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T03:14:09.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 03:14:08 smithi193 ceph-mon[36878]: pgmap v5995: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T03:14:11.195 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 03:14:10 smithi118 ceph-mon[31535]: pgmap v5996: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-03-24T03:14:11.203 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 03:14:10 smithi189 ceph-mon[36187]: pgmap v5996: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-03-24T03:14:11.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 03:14:10 smithi193 ceph-mon[36878]: pgmap v5996: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-03-24T03:14:12.195 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 03:14:11 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 10146.758488 secs 2024-03-24T03:14:12.203 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 03:14:11 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 10146.758488 secs 2024-03-24T03:14:12.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 03:14:11 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 10146.758488 secs 2024-03-24T03:14:13.195 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 03:14:12 smithi118 ceph-mon[31535]: pgmap v5997: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:14:13.203 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 03:14:12 smithi189 ceph-mon[36187]: pgmap v5997: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:14:13.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 03:14:12 smithi193 ceph-mon[36878]: pgmap v5997: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:14:15.192 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 03:14:14 smithi118 ceph-mon[31535]: pgmap v5998: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T03:14:15.203 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 03:14:14 smithi189 ceph-mon[36187]: pgmap v5998: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T03:14:15.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 03:14:14 smithi193 ceph-mon[36878]: pgmap v5998: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T03:14:17.195 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 03:14:16 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 10151.758577 secs 2024-03-24T03:14:17.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 03:14:16 smithi118 ceph-mon[31535]: pgmap v5999: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T03:14:17.203 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 03:14:16 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 10151.758577 secs 2024-03-24T03:14:17.203 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 03:14:16 smithi189 ceph-mon[36187]: pgmap v5999: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T03:14:17.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 03:14:16 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 10151.758577 secs 2024-03-24T03:14:17.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 03:14:16 smithi193 ceph-mon[36878]: pgmap v5999: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T03:14:19.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 03:14:18 smithi118 ceph-mon[31535]: pgmap v6000: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T03:14:19.203 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 03:14:18 smithi189 ceph-mon[36187]: pgmap v6000: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T03:14:19.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 03:14:18 smithi193 ceph-mon[36878]: pgmap v6000: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T03:14:21.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 03:14:20 smithi118 ceph-mon[31535]: pgmap v6001: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-03-24T03:14:21.203 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 03:14:20 smithi189 ceph-mon[36187]: pgmap v6001: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-03-24T03:14:21.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 03:14:20 smithi193 ceph-mon[36878]: pgmap v6001: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-03-24T03:14:22.195 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 03:14:21 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 10156.758639 secs 2024-03-24T03:14:22.203 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 03:14:21 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 10156.758639 secs 2024-03-24T03:14:22.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 03:14:21 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 10156.758639 secs 2024-03-24T03:14:23.195 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 03:14:22 smithi118 ceph-mon[31535]: pgmap v6002: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:14:23.203 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 03:14:22 smithi189 ceph-mon[36187]: pgmap v6002: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:14:23.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 03:14:22 smithi193 ceph-mon[36878]: pgmap v6002: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:14:25.195 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 03:14:24 smithi118 ceph-mon[31535]: pgmap v6003: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T03:14:25.203 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 03:14:24 smithi189 ceph-mon[36187]: pgmap v6003: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T03:14:25.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 03:14:24 smithi193 ceph-mon[36878]: pgmap v6003: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T03:14:27.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 03:14:26 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 10161.758724 secs 2024-03-24T03:14:27.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 03:14:26 smithi118 ceph-mon[31535]: pgmap v6004: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T03:14:27.203 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 03:14:26 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 10161.758724 secs 2024-03-24T03:14:27.203 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 03:14:26 smithi189 ceph-mon[36187]: pgmap v6004: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T03:14:27.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 03:14:26 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 10161.758724 secs 2024-03-24T03:14:27.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 03:14:26 smithi193 ceph-mon[36878]: pgmap v6004: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T03:14:29.195 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 03:14:28 smithi118 ceph-mon[31535]: pgmap v6005: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T03:14:29.203 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 03:14:28 smithi189 ceph-mon[36187]: pgmap v6005: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T03:14:29.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 03:14:28 smithi193 ceph-mon[36878]: pgmap v6005: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T03:14:31.195 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 03:14:30 smithi118 ceph-mon[31535]: pgmap v6006: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-03-24T03:14:31.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 03:14:30 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T03:14:31.203 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 03:14:30 smithi189 ceph-mon[36187]: pgmap v6006: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-03-24T03:14:31.203 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 03:14:30 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T03:14:31.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 03:14:30 smithi193 ceph-mon[36878]: pgmap v6006: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-03-24T03:14:31.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 03:14:30 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T03:14:32.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 03:14:31 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 10166.758794 secs 2024-03-24T03:14:32.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 03:14:31 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T03:14:32.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 03:14:31 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T03:14:32.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 03:14:31 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T03:14:32.203 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 03:14:31 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 10166.758794 secs 2024-03-24T03:14:32.203 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 03:14:31 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T03:14:32.203 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 03:14:31 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T03:14:32.203 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 03:14:31 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T03:14:32.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 03:14:31 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 10166.758794 secs 2024-03-24T03:14:32.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 03:14:31 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T03:14:32.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 03:14:31 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T03:14:32.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 03:14:31 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T03:14:33.195 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 03:14:32 smithi118 ceph-mon[31535]: pgmap v6007: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 5 op/s 2024-03-24T03:14:33.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 03:14:32 smithi118 ceph-mon[31535]: pgmap v6008: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-03-24T03:14:33.203 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 03:14:32 smithi189 ceph-mon[36187]: pgmap v6007: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 5 op/s 2024-03-24T03:14:33.203 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 03:14:32 smithi189 ceph-mon[36187]: pgmap v6008: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-03-24T03:14:33.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 03:14:32 smithi193 ceph-mon[36878]: pgmap v6007: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 5 op/s 2024-03-24T03:14:33.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 03:14:32 smithi193 ceph-mon[36878]: pgmap v6008: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-03-24T03:14:35.195 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 03:14:34 smithi118 ceph-mon[31535]: pgmap v6009: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-03-24T03:14:35.203 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 03:14:34 smithi189 ceph-mon[36187]: pgmap v6009: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-03-24T03:14:35.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 03:14:34 smithi193 ceph-mon[36878]: pgmap v6009: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-03-24T03:14:37.195 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 03:14:36 smithi118 ceph-mon[31535]: pgmap v6010: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-03-24T03:14:37.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 03:14:36 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 10171.758883 secs 2024-03-24T03:14:37.203 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 03:14:36 smithi189 ceph-mon[36187]: pgmap v6010: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-03-24T03:14:37.203 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 03:14:36 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 10171.758883 secs 2024-03-24T03:14:37.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 03:14:36 smithi193 ceph-mon[36878]: pgmap v6010: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-03-24T03:14:37.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 03:14:36 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 10171.758883 secs 2024-03-24T03:14:39.195 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 03:14:38 smithi118 ceph-mon[31535]: pgmap v6011: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 5 op/s 2024-03-24T03:14:39.203 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 03:14:38 smithi189 ceph-mon[36187]: pgmap v6011: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 5 op/s 2024-03-24T03:14:39.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 03:14:38 smithi193 ceph-mon[36878]: pgmap v6011: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 5 op/s 2024-03-24T03:14:41.195 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 03:14:40 smithi118 ceph-mon[31535]: pgmap v6012: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-03-24T03:14:41.203 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 03:14:40 smithi189 ceph-mon[36187]: pgmap v6012: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-03-24T03:14:41.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 03:14:40 smithi193 ceph-mon[36878]: pgmap v6012: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-03-24T03:14:42.195 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 03:14:41 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 10176.758970 secs 2024-03-24T03:14:42.203 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 03:14:41 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 10176.758970 secs 2024-03-24T03:14:42.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 03:14:41 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 10176.758970 secs 2024-03-24T03:14:43.198 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 03:14:42 smithi118 ceph-mon[31535]: pgmap v6013: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 5 op/s 2024-03-24T03:14:43.203 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 03:14:42 smithi189 ceph-mon[36187]: pgmap v6013: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 5 op/s 2024-03-24T03:14:43.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 03:14:42 smithi193 ceph-mon[36878]: pgmap v6013: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 5 op/s 2024-03-24T03:14:45.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 03:14:44 smithi118 ceph-mon[31535]: pgmap v6014: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T03:14:45.203 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 03:14:44 smithi189 ceph-mon[36187]: pgmap v6014: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T03:14:45.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 03:14:44 smithi193 ceph-mon[36878]: pgmap v6014: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T03:14:47.195 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 03:14:46 smithi118 ceph-mon[31535]: pgmap v6015: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T03:14:47.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 03:14:46 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 10181.759062 secs 2024-03-24T03:14:47.203 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 03:14:46 smithi189 ceph-mon[36187]: pgmap v6015: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T03:14:47.203 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 03:14:46 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 10181.759062 secs 2024-03-24T03:14:47.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 03:14:46 smithi193 ceph-mon[36878]: pgmap v6015: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T03:14:47.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 03:14:46 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 10181.759062 secs 2024-03-24T03:14:49.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 03:14:48 smithi118 ceph-mon[31535]: pgmap v6016: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:14:49.203 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 03:14:48 smithi189 ceph-mon[36187]: pgmap v6016: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:14:49.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 03:14:48 smithi193 ceph-mon[36878]: pgmap v6016: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:14:51.195 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 03:14:50 smithi118 ceph-mon[31535]: pgmap v6017: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-24T03:14:51.203 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 03:14:50 smithi189 ceph-mon[36187]: pgmap v6017: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-24T03:14:51.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 03:14:50 smithi193 ceph-mon[36878]: pgmap v6017: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-24T03:14:52.203 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 03:14:51 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 10186.759139 secs 2024-03-24T03:14:52.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 03:14:51 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 10186.759139 secs 2024-03-24T03:14:52.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 03:14:51 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 10186.759139 secs 2024-03-24T03:14:53.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 03:14:52 smithi193 ceph-mon[36878]: pgmap v6018: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:14:53.445 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 03:14:52 smithi118 ceph-mon[31535]: pgmap v6018: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:14:53.453 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 03:14:52 smithi189 ceph-mon[36187]: pgmap v6018: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:14:55.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 03:14:54 smithi193 ceph-mon[36878]: pgmap v6019: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T03:14:55.445 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 03:14:54 smithi118 ceph-mon[31535]: pgmap v6019: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T03:14:55.453 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 03:14:54 smithi189 ceph-mon[36187]: pgmap v6019: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T03:14:57.445 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 03:14:56 smithi118 ceph-mon[31535]: pgmap v6020: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T03:14:57.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 03:14:56 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 10191.759202 secs 2024-03-24T03:14:57.453 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 03:14:56 smithi189 ceph-mon[36187]: pgmap v6020: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T03:14:57.453 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 03:14:56 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 10191.759202 secs 2024-03-24T03:14:57.460 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 03:14:56 smithi193 ceph-mon[36878]: pgmap v6020: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T03:14:57.460 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 03:14:56 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 10191.759202 secs 2024-03-24T03:14:59.249 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 03:14:58 smithi118 ceph-mon[31535]: pgmap v6021: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:14:59.453 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 03:14:58 smithi189 ceph-mon[36187]: pgmap v6021: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:14:59.460 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 03:14:58 smithi193 ceph-mon[36878]: pgmap v6021: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:15:01.272 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 03:15:00 smithi118 ceph-mon[31535]: pgmap v6022: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-24T03:15:01.453 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 03:15:00 smithi189 ceph-mon[36187]: pgmap v6022: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-24T03:15:01.460 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 03:15:00 smithi193 ceph-mon[36878]: pgmap v6022: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-24T03:15:02.445 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 03:15:01 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 10196.759263 secs 2024-03-24T03:15:02.453 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 03:15:01 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 10196.759263 secs 2024-03-24T03:15:02.460 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 03:15:01 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 10196.759263 secs 2024-03-24T03:15:03.445 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 03:15:02 smithi118 ceph-mon[31535]: pgmap v6023: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:15:03.453 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 03:15:02 smithi189 ceph-mon[36187]: pgmap v6023: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:15:03.460 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 03:15:02 smithi193 ceph-mon[36878]: pgmap v6023: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:15:05.445 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 03:15:04 smithi118 ceph-mon[31535]: pgmap v6024: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T03:15:05.453 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 03:15:04 smithi189 ceph-mon[36187]: pgmap v6024: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T03:15:05.460 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 03:15:04 smithi193 ceph-mon[36878]: pgmap v6024: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T03:15:07.255 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 03:15:07 smithi189 ceph-mon[36187]: pgmap v6025: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T03:15:07.255 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 03:15:07 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 10201.759349 secs 2024-03-24T03:15:07.261 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 03:15:07 smithi193 ceph-mon[36878]: pgmap v6025: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T03:15:07.261 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 03:15:07 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 10201.759349 secs 2024-03-24T03:15:07.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 03:15:07 smithi118 ceph-mon[31535]: pgmap v6025: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T03:15:07.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 03:15:07 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 10201.759349 secs 2024-03-24T03:15:09.445 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 03:15:09 smithi118 ceph-mon[31535]: pgmap v6026: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:15:09.453 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 03:15:09 smithi189 ceph-mon[36187]: pgmap v6026: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:15:09.460 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 03:15:09 smithi193 ceph-mon[36878]: pgmap v6026: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:15:11.445 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 03:15:11 smithi118 ceph-mon[31535]: pgmap v6027: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-24T03:15:11.453 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 03:15:11 smithi189 ceph-mon[36187]: pgmap v6027: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-24T03:15:11.460 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 03:15:11 smithi193 ceph-mon[36878]: pgmap v6027: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-24T03:15:12.445 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 03:15:12 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 10206.759410 secs 2024-03-24T03:15:12.453 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 03:15:12 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 10206.759410 secs 2024-03-24T03:15:12.460 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 03:15:12 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 10206.759410 secs 2024-03-24T03:15:13.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 03:15:13 smithi118 ceph-mon[31535]: pgmap v6028: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:15:13.453 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 03:15:13 smithi189 ceph-mon[36187]: pgmap v6028: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:15:13.460 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 03:15:13 smithi193 ceph-mon[36878]: pgmap v6028: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:15:14.445 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 03:15:14 smithi118 ceph-mon[31535]: pgmap v6029: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T03:15:14.453 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 03:15:14 smithi189 ceph-mon[36187]: pgmap v6029: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T03:15:14.460 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 03:15:14 smithi193 ceph-mon[36878]: pgmap v6029: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T03:15:16.697 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 03:15:16 smithi118 ceph-mon[31535]: pgmap v6030: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T03:15:16.697 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 03:15:16 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 10211.759499 secs 2024-03-24T03:15:16.703 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 03:15:16 smithi189 ceph-mon[36187]: pgmap v6030: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T03:15:16.703 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 03:15:16 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 10211.759499 secs 2024-03-24T03:15:16.710 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 03:15:16 smithi193 ceph-mon[36878]: pgmap v6030: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T03:15:16.710 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 03:15:16 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 10211.759499 secs 2024-03-24T03:15:18.695 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 03:15:18 smithi118 ceph-mon[31535]: pgmap v6031: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:15:18.703 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 03:15:18 smithi189 ceph-mon[36187]: pgmap v6031: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:15:18.710 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 03:15:18 smithi193 ceph-mon[36878]: pgmap v6031: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:15:20.695 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 03:15:20 smithi118 ceph-mon[31535]: pgmap v6032: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-24T03:15:20.703 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 03:15:20 smithi189 ceph-mon[36187]: pgmap v6032: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-24T03:15:20.710 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 03:15:20 smithi193 ceph-mon[36878]: pgmap v6032: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-24T03:15:21.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 03:15:21 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 10216.759587 secs 2024-03-24T03:15:21.703 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 03:15:21 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 10216.759587 secs 2024-03-24T03:15:21.710 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 03:15:21 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 10216.759587 secs 2024-03-24T03:15:22.695 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 03:15:22 smithi118 ceph-mon[31535]: pgmap v6033: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:15:22.703 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 03:15:22 smithi189 ceph-mon[36187]: pgmap v6033: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:15:22.710 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 03:15:22 smithi193 ceph-mon[36878]: pgmap v6033: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:15:24.695 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 03:15:24 smithi118 ceph-mon[31535]: pgmap v6034: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T03:15:24.703 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 03:15:24 smithi189 ceph-mon[36187]: pgmap v6034: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T03:15:24.710 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 03:15:24 smithi193 ceph-mon[36878]: pgmap v6034: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T03:15:26.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 03:15:26 smithi118 ceph-mon[31535]: pgmap v6035: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T03:15:26.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 03:15:26 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 10221.759678 secs 2024-03-24T03:15:26.703 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 03:15:26 smithi189 ceph-mon[36187]: pgmap v6035: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T03:15:26.703 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 03:15:26 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 10221.759678 secs 2024-03-24T03:15:26.710 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 03:15:26 smithi193 ceph-mon[36878]: pgmap v6035: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T03:15:26.710 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 03:15:26 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 10221.759678 secs 2024-03-24T03:15:28.695 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 03:15:28 smithi118 ceph-mon[31535]: pgmap v6036: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:15:28.703 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 03:15:28 smithi189 ceph-mon[36187]: pgmap v6036: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:15:28.710 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 03:15:28 smithi193 ceph-mon[36878]: pgmap v6036: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:15:30.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 03:15:30 smithi118 ceph-mon[31535]: pgmap v6037: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-24T03:15:30.703 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 03:15:30 smithi189 ceph-mon[36187]: pgmap v6037: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-24T03:15:30.710 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 03:15:30 smithi193 ceph-mon[36878]: pgmap v6037: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-24T03:15:31.695 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 03:15:31 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 10226.759770 secs 2024-03-24T03:15:31.703 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 03:15:31 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 10226.759770 secs 2024-03-24T03:15:31.710 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 03:15:31 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 10226.759770 secs 2024-03-24T03:15:32.695 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 03:15:32 smithi118 ceph-mon[31535]: pgmap v6038: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:15:32.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 03:15:32 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T03:15:32.703 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 03:15:32 smithi189 ceph-mon[36187]: pgmap v6038: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:15:32.703 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 03:15:32 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T03:15:32.710 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 03:15:32 smithi193 ceph-mon[36878]: pgmap v6038: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:15:32.710 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 03:15:32 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T03:15:33.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 03:15:33 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T03:15:33.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 03:15:33 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T03:15:33.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 03:15:33 smithi118 ceph-mon[31535]: pgmap v6039: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 5 op/s 2024-03-24T03:15:33.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 03:15:33 smithi118 ceph-mon[31535]: pgmap v6040: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 5 op/s 2024-03-24T03:15:33.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 03:15:33 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T03:15:33.703 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 03:15:33 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T03:15:33.703 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 03:15:33 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T03:15:33.703 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 03:15:33 smithi189 ceph-mon[36187]: pgmap v6039: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 5 op/s 2024-03-24T03:15:33.703 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 03:15:33 smithi189 ceph-mon[36187]: pgmap v6040: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 5 op/s 2024-03-24T03:15:33.703 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 03:15:33 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T03:15:33.710 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 03:15:33 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T03:15:33.710 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 03:15:33 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T03:15:33.710 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 03:15:33 smithi193 ceph-mon[36878]: pgmap v6039: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 5 op/s 2024-03-24T03:15:33.710 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 03:15:33 smithi193 ceph-mon[36878]: pgmap v6040: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 5 op/s 2024-03-24T03:15:33.710 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 03:15:33 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T03:15:36.087 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 03:15:35 smithi118 ceph-mon[31535]: pgmap v6041: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-03-24T03:15:36.203 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 03:15:35 smithi189 ceph-mon[36187]: pgmap v6041: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-03-24T03:15:36.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 03:15:35 smithi193 ceph-mon[36878]: pgmap v6041: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-03-24T03:15:37.104 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 03:15:36 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 10231.759856 secs 2024-03-24T03:15:37.110 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 03:15:36 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 10231.759856 secs 2024-03-24T03:15:37.195 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 03:15:36 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 10231.759856 secs 2024-03-24T03:15:38.195 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 03:15:37 smithi118 ceph-mon[31535]: pgmap v6042: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-03-24T03:15:38.203 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 03:15:37 smithi189 ceph-mon[36187]: pgmap v6042: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-03-24T03:15:38.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 03:15:37 smithi193 ceph-mon[36878]: pgmap v6042: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-03-24T03:15:40.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 03:15:39 smithi118 ceph-mon[31535]: pgmap v6043: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 4 op/s 2024-03-24T03:15:40.203 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 03:15:39 smithi189 ceph-mon[36187]: pgmap v6043: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 4 op/s 2024-03-24T03:15:40.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 03:15:39 smithi193 ceph-mon[36878]: pgmap v6043: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 4 op/s 2024-03-24T03:15:42.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 03:15:41 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 10236.759945 secs 2024-03-24T03:15:42.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 03:15:41 smithi118 ceph-mon[31535]: pgmap v6044: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-03-24T03:15:42.203 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 03:15:41 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 10236.759945 secs 2024-03-24T03:15:42.203 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 03:15:41 smithi189 ceph-mon[36187]: pgmap v6044: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-03-24T03:15:42.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 03:15:41 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 10236.759945 secs 2024-03-24T03:15:42.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 03:15:41 smithi193 ceph-mon[36878]: pgmap v6044: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-03-24T03:15:44.195 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 03:15:43 smithi118 ceph-mon[31535]: pgmap v6045: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 5 op/s 2024-03-24T03:15:44.203 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 03:15:43 smithi189 ceph-mon[36187]: pgmap v6045: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 5 op/s 2024-03-24T03:15:44.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 03:15:43 smithi193 ceph-mon[36878]: pgmap v6045: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 5 op/s 2024-03-24T03:15:46.195 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 03:15:45 smithi118 ceph-mon[31535]: pgmap v6046: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-03-24T03:15:46.203 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 03:15:45 smithi189 ceph-mon[36187]: pgmap v6046: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-03-24T03:15:46.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 03:15:45 smithi193 ceph-mon[36878]: pgmap v6046: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-03-24T03:15:47.195 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 03:15:46 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 10241.759987 secs 2024-03-24T03:15:47.203 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 03:15:46 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 10241.759987 secs 2024-03-24T03:15:47.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 03:15:46 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 10241.759987 secs 2024-03-24T03:15:48.195 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 03:15:47 smithi118 ceph-mon[31535]: pgmap v6047: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:15:48.203 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 03:15:47 smithi189 ceph-mon[36187]: pgmap v6047: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:15:48.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 03:15:47 smithi193 ceph-mon[36878]: pgmap v6047: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:15:50.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 03:15:49 smithi118 ceph-mon[31535]: pgmap v6048: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-03-24T03:15:50.203 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 03:15:49 smithi189 ceph-mon[36187]: pgmap v6048: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-03-24T03:15:50.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 03:15:49 smithi193 ceph-mon[36878]: pgmap v6048: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-03-24T03:15:52.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 03:15:51 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 10246.760078 secs 2024-03-24T03:15:52.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 03:15:51 smithi118 ceph-mon[31535]: pgmap v6049: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 6 op/s 2024-03-24T03:15:52.203 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 03:15:51 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 10246.760078 secs 2024-03-24T03:15:52.203 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 03:15:51 smithi189 ceph-mon[36187]: pgmap v6049: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 6 op/s 2024-03-24T03:15:52.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 03:15:51 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 10246.760078 secs 2024-03-24T03:15:52.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 03:15:51 smithi193 ceph-mon[36878]: pgmap v6049: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 6 op/s 2024-03-24T03:15:54.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 03:15:53 smithi118 ceph-mon[31535]: pgmap v6050: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 5 op/s 2024-03-24T03:15:54.203 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 03:15:53 smithi189 ceph-mon[36187]: pgmap v6050: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 5 op/s 2024-03-24T03:15:54.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 03:15:53 smithi193 ceph-mon[36878]: pgmap v6050: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 5 op/s 2024-03-24T03:15:56.195 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 03:15:55 smithi118 ceph-mon[31535]: pgmap v6051: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-03-24T03:15:56.203 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 03:15:55 smithi189 ceph-mon[36187]: pgmap v6051: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-03-24T03:15:56.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 03:15:55 smithi193 ceph-mon[36878]: pgmap v6051: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-03-24T03:15:57.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 03:15:56 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 10251.760163 secs 2024-03-24T03:15:57.203 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 03:15:56 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 10251.760163 secs 2024-03-24T03:15:57.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 03:15:56 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 10251.760163 secs 2024-03-24T03:15:58.195 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 03:15:57 smithi118 ceph-mon[31535]: pgmap v6052: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-03-24T03:15:58.203 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 03:15:57 smithi189 ceph-mon[36187]: pgmap v6052: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-03-24T03:15:58.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 03:15:57 smithi193 ceph-mon[36878]: pgmap v6052: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.4 KiB/s rd, 4 op/s 2024-03-24T03:16:00.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 03:15:59 smithi118 ceph-mon[31535]: pgmap v6053: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-03-24T03:16:00.203 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 03:15:59 smithi189 ceph-mon[36187]: pgmap v6053: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-03-24T03:16:00.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 03:15:59 smithi193 ceph-mon[36878]: pgmap v6053: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-03-24T03:16:02.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 03:16:01 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 10256.760227 secs 2024-03-24T03:16:02.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 03:16:01 smithi118 ceph-mon[31535]: pgmap v6054: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T03:16:02.203 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 03:16:01 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 10256.760227 secs 2024-03-24T03:16:02.203 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 03:16:01 smithi189 ceph-mon[36187]: pgmap v6054: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T03:16:02.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 03:16:01 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 10256.760227 secs 2024-03-24T03:16:02.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 03:16:01 smithi193 ceph-mon[36878]: pgmap v6054: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T03:16:04.195 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 03:16:03 smithi118 ceph-mon[31535]: pgmap v6055: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T03:16:04.203 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 03:16:03 smithi189 ceph-mon[36187]: pgmap v6055: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T03:16:04.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 03:16:03 smithi193 ceph-mon[36878]: pgmap v6055: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T03:16:06.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 03:16:05 smithi118 ceph-mon[31535]: pgmap v6056: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-03-24T03:16:06.203 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 03:16:05 smithi189 ceph-mon[36187]: pgmap v6056: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-03-24T03:16:06.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 03:16:05 smithi193 ceph-mon[36878]: pgmap v6056: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-03-24T03:16:07.104 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 03:16:06 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 10261.760313 secs 2024-03-24T03:16:07.111 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 03:16:06 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 10261.760313 secs 2024-03-24T03:16:07.195 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 03:16:06 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 10261.760313 secs 2024-03-24T03:16:08.195 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 03:16:07 smithi118 ceph-mon[31535]: pgmap v6057: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:16:08.203 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 03:16:07 smithi189 ceph-mon[36187]: pgmap v6057: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:16:08.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 03:16:07 smithi193 ceph-mon[36878]: pgmap v6057: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:16:10.195 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 03:16:09 smithi118 ceph-mon[31535]: pgmap v6058: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T03:16:10.203 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 03:16:09 smithi189 ceph-mon[36187]: pgmap v6058: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T03:16:10.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 03:16:09 smithi193 ceph-mon[36878]: pgmap v6058: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T03:16:12.195 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 03:16:11 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 10266.760376 secs 2024-03-24T03:16:12.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 03:16:11 smithi118 ceph-mon[31535]: pgmap v6059: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T03:16:12.203 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 03:16:11 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 10266.760376 secs 2024-03-24T03:16:12.203 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 03:16:11 smithi189 ceph-mon[36187]: pgmap v6059: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T03:16:12.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 03:16:11 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 10266.760376 secs 2024-03-24T03:16:12.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 03:16:11 smithi193 ceph-mon[36878]: pgmap v6059: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T03:16:14.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 03:16:13 smithi118 ceph-mon[31535]: pgmap v6060: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T03:16:14.203 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 03:16:13 smithi189 ceph-mon[36187]: pgmap v6060: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T03:16:14.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 03:16:13 smithi193 ceph-mon[36878]: pgmap v6060: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T03:16:16.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 03:16:15 smithi118 ceph-mon[31535]: pgmap v6061: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-03-24T03:16:16.203 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 03:16:15 smithi189 ceph-mon[36187]: pgmap v6061: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-03-24T03:16:16.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 03:16:15 smithi193 ceph-mon[36878]: pgmap v6061: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-03-24T03:16:17.195 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 03:16:16 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 10271.760435 secs 2024-03-24T03:16:17.203 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 03:16:16 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 10271.760435 secs 2024-03-24T03:16:17.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 03:16:16 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 10271.760435 secs 2024-03-24T03:16:17.399 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T03:16:17.399+0000 7f81ee000640 1 -- 172.21.15.118:0/1693767697 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f81c8006590 con 0x7f81f81052d0 2024-03-24T03:16:17.399 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T03:16:17.399+0000 7f81db400640 1 -- 172.21.15.118:0/659686424 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- auth(proto 2 165 bytes epoch 0) v1 -- 0x7f81c4006290 con 0x5622651a13f0 2024-03-24T03:16:17.399 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T03:16:17.399+0000 7f81ef400640 1 -- 172.21.15.118:0/1693767697 <== mon.2 v2:172.21.15.189:3300/0 82 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 580+0+0 (secure 0 0 0) 0x7f81c8006590 con 0x7f81f81052d0 2024-03-24T03:16:17.400 INFO:teuthology.task.background_exec.ubuntu@smithi118.front.sepia.ceph.com.smithi118.stderr:2024-03-24T03:16:17.399+0000 7f81e4c00640 1 -- 172.21.15.118:0/659686424 <== mon.1 v2:172.21.15.193:3300/0 82 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 409+0+0 (secure 0 0 0) 0x7f81c4006290 con 0x5622651a13f0 2024-03-24T03:16:18.195 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 03:16:17 smithi118 ceph-mon[31535]: pgmap v6062: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:16:18.203 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 03:16:17 smithi189 ceph-mon[36187]: pgmap v6062: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:16:18.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 03:16:17 smithi193 ceph-mon[36878]: pgmap v6062: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:16:20.195 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 03:16:19 smithi118 ceph-mon[31535]: pgmap v6063: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T03:16:20.203 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 03:16:19 smithi189 ceph-mon[36187]: pgmap v6063: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T03:16:20.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 03:16:19 smithi193 ceph-mon[36878]: pgmap v6063: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T03:16:22.168 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 03:16:21 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 10276.760475 secs 2024-03-24T03:16:22.168 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 03:16:21 smithi118 ceph-mon[31535]: pgmap v6064: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T03:16:22.203 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 03:16:21 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 10276.760475 secs 2024-03-24T03:16:22.203 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 03:16:21 smithi189 ceph-mon[36187]: pgmap v6064: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T03:16:22.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 03:16:21 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 10276.760475 secs 2024-03-24T03:16:22.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 03:16:21 smithi193 ceph-mon[36878]: pgmap v6064: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T03:16:24.195 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 03:16:23 smithi118 ceph-mon[31535]: pgmap v6065: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T03:16:24.203 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 03:16:23 smithi189 ceph-mon[36187]: pgmap v6065: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T03:16:24.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 03:16:23 smithi193 ceph-mon[36878]: pgmap v6065: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T03:16:26.195 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 03:16:25 smithi118 ceph-mon[31535]: pgmap v6066: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-03-24T03:16:26.203 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 03:16:25 smithi189 ceph-mon[36187]: pgmap v6066: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-03-24T03:16:26.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 03:16:25 smithi193 ceph-mon[36878]: pgmap v6066: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-03-24T03:16:27.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 03:16:26 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 10281.760544 secs 2024-03-24T03:16:27.203 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 03:16:26 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 10281.760544 secs 2024-03-24T03:16:27.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 03:16:26 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 10281.760544 secs 2024-03-24T03:16:28.197 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 03:16:27 smithi118 ceph-mon[31535]: pgmap v6067: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:16:28.203 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 03:16:27 smithi189 ceph-mon[36187]: pgmap v6067: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:16:28.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 03:16:27 smithi193 ceph-mon[36878]: pgmap v6067: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:16:30.195 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 03:16:29 smithi118 ceph-mon[31535]: pgmap v6068: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T03:16:30.203 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 03:16:29 smithi189 ceph-mon[36187]: pgmap v6068: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T03:16:30.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 03:16:29 smithi193 ceph-mon[36878]: pgmap v6068: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T03:16:32.195 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 03:16:31 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 10286.760606 secs 2024-03-24T03:16:32.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 03:16:31 smithi118 ceph-mon[31535]: pgmap v6069: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T03:16:32.203 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 03:16:31 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 10286.760606 secs 2024-03-24T03:16:32.203 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 03:16:31 smithi189 ceph-mon[36187]: pgmap v6069: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T03:16:32.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 03:16:31 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 10286.760606 secs 2024-03-24T03:16:32.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 03:16:31 smithi193 ceph-mon[36878]: pgmap v6069: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T03:16:34.195 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 03:16:33 smithi118 ceph-mon[31535]: pgmap v6070: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T03:16:34.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 03:16:33 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T03:16:34.203 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 03:16:33 smithi189 ceph-mon[36187]: pgmap v6070: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T03:16:34.203 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 03:16:33 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T03:16:34.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 03:16:33 smithi193 ceph-mon[36878]: pgmap v6070: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T03:16:34.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 03:16:33 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T03:16:35.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 03:16:34 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T03:16:35.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 03:16:34 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T03:16:35.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 03:16:34 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T03:16:35.203 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 03:16:34 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T03:16:35.203 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 03:16:34 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T03:16:35.203 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 03:16:34 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T03:16:35.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 03:16:34 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T03:16:35.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 03:16:34 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T03:16:35.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 03:16:34 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T03:16:36.195 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 03:16:35 smithi118 ceph-mon[31535]: pgmap v6071: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 7 op/s 2024-03-24T03:16:36.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 03:16:35 smithi118 ceph-mon[31535]: pgmap v6072: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-03-24T03:16:36.203 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 03:16:35 smithi189 ceph-mon[36187]: pgmap v6071: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 7 op/s 2024-03-24T03:16:36.203 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 03:16:35 smithi189 ceph-mon[36187]: pgmap v6072: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-03-24T03:16:36.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 03:16:35 smithi193 ceph-mon[36878]: pgmap v6071: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 7 op/s 2024-03-24T03:16:36.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 03:16:35 smithi193 ceph-mon[36878]: pgmap v6072: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-03-24T03:16:37.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 03:16:36 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 10291.760694 secs 2024-03-24T03:16:37.445 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 03:16:36 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 10291.760694 secs 2024-03-24T03:16:37.453 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 03:16:36 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 10291.760694 secs 2024-03-24T03:16:38.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 03:16:37 smithi193 ceph-mon[36878]: pgmap v6073: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-03-24T03:16:38.445 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 03:16:37 smithi118 ceph-mon[31535]: pgmap v6073: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-03-24T03:16:38.453 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 03:16:37 smithi189 ceph-mon[36187]: pgmap v6073: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-03-24T03:16:40.445 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 03:16:39 smithi118 ceph-mon[31535]: pgmap v6074: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-03-24T03:16:40.453 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 03:16:39 smithi189 ceph-mon[36187]: pgmap v6074: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-03-24T03:16:40.460 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 03:16:39 smithi193 ceph-mon[36878]: pgmap v6074: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-03-24T03:16:42.445 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 03:16:41 smithi118 ceph-mon[31535]: pgmap v6075: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-03-24T03:16:42.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 03:16:41 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 10296.760786 secs 2024-03-24T03:16:42.453 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 03:16:41 smithi189 ceph-mon[36187]: pgmap v6075: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-03-24T03:16:42.453 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 03:16:41 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 10296.760786 secs 2024-03-24T03:16:42.460 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 03:16:41 smithi193 ceph-mon[36878]: pgmap v6075: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-03-24T03:16:42.460 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 03:16:41 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 10296.760786 secs 2024-03-24T03:16:44.445 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 03:16:43 smithi118 ceph-mon[31535]: pgmap v6076: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 5 op/s 2024-03-24T03:16:44.453 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 03:16:43 smithi189 ceph-mon[36187]: pgmap v6076: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 5 op/s 2024-03-24T03:16:44.460 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 03:16:43 smithi193 ceph-mon[36878]: pgmap v6076: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 5 op/s 2024-03-24T03:16:46.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 03:16:45 smithi118 ceph-mon[31535]: pgmap v6077: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 5 op/s 2024-03-24T03:16:46.453 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 03:16:45 smithi189 ceph-mon[36187]: pgmap v6077: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 5 op/s 2024-03-24T03:16:46.460 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 03:16:45 smithi193 ceph-mon[36878]: pgmap v6077: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.0 KiB/s rd, 5 op/s 2024-03-24T03:16:47.445 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 03:16:46 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 10301.760843 secs 2024-03-24T03:16:47.453 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 03:16:46 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 10301.760843 secs 2024-03-24T03:16:47.460 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 03:16:46 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 10301.760843 secs 2024-03-24T03:16:48.445 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 03:16:47 smithi118 ceph-mon[31535]: pgmap v6078: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:16:48.453 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 03:16:47 smithi189 ceph-mon[36187]: pgmap v6078: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:16:48.460 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 03:16:47 smithi193 ceph-mon[36878]: pgmap v6078: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:16:50.328 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 03:16:49 smithi118 ceph-mon[31535]: pgmap v6079: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T03:16:50.453 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 03:16:49 smithi189 ceph-mon[36187]: pgmap v6079: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T03:16:50.460 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 03:16:49 smithi193 ceph-mon[36878]: pgmap v6079: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T03:16:52.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 03:16:51 smithi118 ceph-mon[31535]: pgmap v6080: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T03:16:52.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 03:16:51 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 10306.760931 secs 2024-03-24T03:16:52.453 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 03:16:51 smithi189 ceph-mon[36187]: pgmap v6080: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T03:16:52.453 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 03:16:51 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 10306.760931 secs 2024-03-24T03:16:52.460 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 03:16:51 smithi193 ceph-mon[36878]: pgmap v6080: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T03:16:52.460 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 03:16:51 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 10306.760931 secs 2024-03-24T03:16:54.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 03:16:54 smithi118 ceph-mon[31535]: pgmap v6081: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:16:54.453 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 03:16:54 smithi189 ceph-mon[36187]: pgmap v6081: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:16:54.460 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 03:16:54 smithi193 ceph-mon[36878]: pgmap v6081: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:16:56.445 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 03:16:56 smithi118 ceph-mon[31535]: pgmap v6082: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-24T03:16:56.453 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 03:16:56 smithi189 ceph-mon[36187]: pgmap v6082: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-24T03:16:56.460 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 03:16:56 smithi193 ceph-mon[36878]: pgmap v6082: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-24T03:16:57.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 03:16:57 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 10311.760945 secs 2024-03-24T03:16:57.453 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 03:16:57 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 10311.760945 secs 2024-03-24T03:16:57.460 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 03:16:57 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 10311.760945 secs 2024-03-24T03:16:58.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 03:16:58 smithi118 ceph-mon[31535]: pgmap v6083: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:16:58.453 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 03:16:58 smithi189 ceph-mon[36187]: pgmap v6083: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:16:58.460 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 03:16:58 smithi193 ceph-mon[36878]: pgmap v6083: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:17:00.445 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 03:17:00 smithi118 ceph-mon[31535]: pgmap v6084: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T03:17:00.453 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 03:17:00 smithi189 ceph-mon[36187]: pgmap v6084: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T03:17:00.460 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 03:17:00 smithi193 ceph-mon[36878]: pgmap v6084: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T03:17:02.445 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 03:17:02 smithi118 ceph-mon[31535]: pgmap v6085: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T03:17:02.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 03:17:02 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 10316.761005 secs 2024-03-24T03:17:02.453 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 03:17:02 smithi189 ceph-mon[36187]: pgmap v6085: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T03:17:02.453 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 03:17:02 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 10316.761005 secs 2024-03-24T03:17:02.460 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 03:17:02 smithi193 ceph-mon[36878]: pgmap v6085: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T03:17:02.460 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 03:17:02 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 10316.761005 secs 2024-03-24T03:17:04.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 03:17:04 smithi118 ceph-mon[31535]: pgmap v6086: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:17:04.453 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 03:17:04 smithi189 ceph-mon[36187]: pgmap v6086: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:17:04.460 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 03:17:04 smithi193 ceph-mon[36878]: pgmap v6086: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:17:06.445 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 03:17:06 smithi118 ceph-mon[31535]: pgmap v6087: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-24T03:17:06.453 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 03:17:06 smithi189 ceph-mon[36187]: pgmap v6087: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-24T03:17:06.460 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 03:17:06 smithi193 ceph-mon[36878]: pgmap v6087: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-24T03:17:07.355 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 03:17:07 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 10321.761085 secs 2024-03-24T03:17:07.361 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 03:17:07 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 10321.761085 secs 2024-03-24T03:17:07.445 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 03:17:07 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 10321.761085 secs 2024-03-24T03:17:08.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 03:17:08 smithi118 ceph-mon[31535]: pgmap v6088: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:17:08.453 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 03:17:08 smithi189 ceph-mon[36187]: pgmap v6088: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:17:08.460 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 03:17:08 smithi193 ceph-mon[36878]: pgmap v6088: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:17:09.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 03:17:09 smithi118 ceph-mon[31535]: pgmap v6089: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T03:17:09.453 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 03:17:09 smithi189 ceph-mon[36187]: pgmap v6089: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T03:17:09.460 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 03:17:09 smithi193 ceph-mon[36878]: pgmap v6089: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T03:17:11.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 03:17:11 smithi118 ceph-mon[31535]: pgmap v6090: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T03:17:11.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 03:17:11 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 10326.761143 secs 2024-03-24T03:17:11.703 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 03:17:11 smithi189 ceph-mon[36187]: pgmap v6090: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T03:17:11.703 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 03:17:11 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 10326.761143 secs 2024-03-24T03:17:11.710 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 03:17:11 smithi193 ceph-mon[36878]: pgmap v6090: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T03:17:11.710 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 03:17:11 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 10326.761143 secs 2024-03-24T03:17:13.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 03:17:13 smithi118 ceph-mon[31535]: pgmap v6091: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:17:13.703 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 03:17:13 smithi189 ceph-mon[36187]: pgmap v6091: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:17:13.710 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 03:17:13 smithi193 ceph-mon[36878]: pgmap v6091: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:17:15.695 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 03:17:15 smithi118 ceph-mon[31535]: pgmap v6092: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-24T03:17:15.703 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 03:17:15 smithi189 ceph-mon[36187]: pgmap v6092: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-24T03:17:15.710 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 03:17:15 smithi193 ceph-mon[36878]: pgmap v6092: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-24T03:17:16.695 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 03:17:16 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 10331.761203 secs 2024-03-24T03:17:16.703 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 03:17:16 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 10331.761203 secs 2024-03-24T03:17:16.710 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 03:17:16 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 10331.761203 secs 2024-03-24T03:17:17.695 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 03:17:17 smithi118 ceph-mon[31535]: pgmap v6093: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:17:17.703 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 03:17:17 smithi189 ceph-mon[36187]: pgmap v6093: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:17:17.710 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 03:17:17 smithi193 ceph-mon[36878]: pgmap v6093: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:17:18.810 DEBUG:teuthology.orchestra.run:got remote process result: 124 2024-03-24T03:17:18.821 INFO:tasks.workunit:Stopping ['suites/blogbench.sh'] on client.0... 2024-03-24T03:17:18.822 DEBUG:teuthology.orchestra.run.smithi118:> sudo rm -rf -- /home/ubuntu/cephtest/workunits.list.client.0 /home/ubuntu/cephtest/clone.client.0 2024-03-24T03:17:19.695 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 03:17:19 smithi118 ceph-mon[31535]: pgmap v6094: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T03:17:19.703 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 03:17:19 smithi189 ceph-mon[36187]: pgmap v6094: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T03:17:19.710 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 03:17:19 smithi193 ceph-mon[36878]: pgmap v6094: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T03:17:19.834 ERROR:teuthology.run_tasks:Saw exception from tasks. Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/run_tasks.py", line 105, in run_tasks manager = run_one_task(taskname, ctx=ctx, config=config) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/run_tasks.py", line 83, in run_one_task return task(**kwargs) File "/home/teuthworker/src/git.ceph.com_ceph-c_3be9a5c9bc793e11e2800a8c0c696e8b46742033/qa/tasks/workunit.py", line 145, in task _spawn_on_all_clients(ctx, refspec, all_tasks, config.get('env'), File "/home/teuthworker/src/git.ceph.com_ceph-c_3be9a5c9bc793e11e2800a8c0c696e8b46742033/qa/tasks/workunit.py", line 295, in _spawn_on_all_clients p.spawn(_run_tests, ctx, refspec, role, [unit], env, File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/parallel.py", line 84, in __exit__ for result in self: File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/parallel.py", line 98, in __next__ resurrect_traceback(result) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/parallel.py", line 30, in resurrect_traceback raise exc.exc_info[1] File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/parallel.py", line 23, in capture_traceback return func(*args, **kwargs) File "/home/teuthworker/src/git.ceph.com_ceph-c_3be9a5c9bc793e11e2800a8c0c696e8b46742033/qa/tasks/workunit.py", line 434, in _run_tests remote.run( File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed (workunit test suites/blogbench.sh) on smithi118 with status 124: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=3be9a5c9bc793e11e2800a8c0c696e8b46742033 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/suites/blogbench.sh' 2024-03-24T03:17:20.072 ERROR:teuthology.util.sentry: Sentry event: https://sentry.ceph.com/organizations/ceph/?query=461bca6cf1cc4ccd8e0b74511b1ce964 Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/run_tasks.py", line 105, in run_tasks manager = run_one_task(taskname, ctx=ctx, config=config) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/run_tasks.py", line 83, in run_one_task return task(**kwargs) File "/home/teuthworker/src/git.ceph.com_ceph-c_3be9a5c9bc793e11e2800a8c0c696e8b46742033/qa/tasks/workunit.py", line 145, in task _spawn_on_all_clients(ctx, refspec, all_tasks, config.get('env'), File "/home/teuthworker/src/git.ceph.com_ceph-c_3be9a5c9bc793e11e2800a8c0c696e8b46742033/qa/tasks/workunit.py", line 295, in _spawn_on_all_clients p.spawn(_run_tests, ctx, refspec, role, [unit], env, File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/parallel.py", line 84, in __exit__ for result in self: File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/parallel.py", line 98, in __next__ resurrect_traceback(result) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/parallel.py", line 30, in resurrect_traceback raise exc.exc_info[1] File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/parallel.py", line 23, in capture_traceback return func(*args, **kwargs) File "/home/teuthworker/src/git.ceph.com_ceph-c_3be9a5c9bc793e11e2800a8c0c696e8b46742033/qa/tasks/workunit.py", line 434, in _run_tests remote.run( File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed (workunit test suites/blogbench.sh) on smithi118 with status 124: 'mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=3be9a5c9bc793e11e2800a8c0c696e8b46742033 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/suites/blogbench.sh' 2024-03-24T03:17:20.074 DEBUG:teuthology.run_tasks:Unwinding manager background_exec 2024-03-24T03:17:20.084 INFO:teuthology.task.background_exec:Stopping background command on ubuntu@smithi118.front.sepia.ceph.com 2024-03-24T03:17:20.110 DEBUG:teuthology.run_tasks:Unwinding manager fwd_scrub 2024-03-24T03:17:20.125 INFO:tasks.fwd_scrub:joining ForwardScrubbers 2024-03-24T03:17:20.125 ERROR:teuthology.run_tasks:Manager failed: fwd_scrub Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_3be9a5c9bc793e11e2800a8c0c696e8b46742033/qa/tasks/fwd_scrub.py", line 164, in task stop_all_fwd_scrubbers(ctx.ceph[config['cluster']].thrashers) File "/home/teuthworker/src/git.ceph.com_ceph-c_3be9a5c9bc793e11e2800a8c0c696e8b46742033/qa/tasks/fwd_scrub.py", line 99, in stop_all_fwd_scrubbers raise RuntimeError(f"error during scrub thrashing: {thrasher.exception}") RuntimeError: error during scrub thrashing: 2024-03-24T03:17:20.131 DEBUG:teuthology.run_tasks:Unwinding manager check-counter 2024-03-24T03:17:20.140 DEBUG:teuthology.orchestra.run.smithi118:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-24T03:17:20.372 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T03:17:20.372+0000 7f1d0f600640 1 -- 172.21.15.118:0/3800254348 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f1d081038e0 msgr2=0x7f1d08103d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T03:17:20.372 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T03:17:20.372+0000 7f1d0f600640 1 --2- 172.21.15.118:0/3800254348 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f1d081038e0 0x7f1d08103d00 secure :-1 s=READY pgs=475 cs=0 l=1 rev1=1 crypto rx=0x7f1cfc009900 tx=0x7f1cfc02bc50 comp rx=0 tx=0).stop 2024-03-24T03:17:20.372 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T03:17:20.372+0000 7f1d0f600640 1 -- 172.21.15.118:0/3800254348 shutdown_connections 2024-03-24T03:17:20.372 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T03:17:20.372+0000 7f1d0f600640 1 --2- 172.21.15.118:0/3800254348 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1d08105490 0x7f1d0810bd20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T03:17:20.373 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T03:17:20.372+0000 7f1d0f600640 1 --2- 172.21.15.118:0/3800254348 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f1d08104ad0 0x7f1d08104f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T03:17:20.373 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T03:17:20.372+0000 7f1d0f600640 1 --2- 172.21.15.118:0/3800254348 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f1d081038e0 0x7f1d08103d00 unknown :-1 s=CLOSED pgs=475 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T03:17:20.373 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T03:17:20.372+0000 7f1d0f600640 1 -- 172.21.15.118:0/3800254348 >> 172.21.15.118:0/3800254348 conn(0x7f1d080fee90 msgr2=0x7f1d081012b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T03:17:20.373 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T03:17:20.372+0000 7f1d0f600640 1 -- 172.21.15.118:0/3800254348 shutdown_connections 2024-03-24T03:17:20.373 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T03:17:20.372+0000 7f1d0f600640 1 -- 172.21.15.118:0/3800254348 wait complete. 2024-03-24T03:17:20.375 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T03:17:20.373+0000 7f1d0f600640 1 Processor -- start 2024-03-24T03:17:20.375 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T03:17:20.373+0000 7f1d0f600640 1 -- start start 2024-03-24T03:17:20.375 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T03:17:20.373+0000 7f1d0f600640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f1d081038e0 0x7f1d08199bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T03:17:20.375 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T03:17:20.373+0000 7f1d0f600640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f1d08104ad0 0x7f1d0819a100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T03:17:20.376 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T03:17:20.374+0000 7f1d0f600640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1d08105490 0x7f1d081a1170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T03:17:20.376 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T03:17:20.374+0000 7f1d0f600640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1d0810f5b0 con 0x7f1d08104ad0 2024-03-24T03:17:20.376 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T03:17:20.374+0000 7f1d0f600640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1d0810f430 con 0x7f1d08105490 2024-03-24T03:17:20.376 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T03:17:20.374+0000 7f1d0f600640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1d0810f730 con 0x7f1d081038e0 2024-03-24T03:17:20.376 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T03:17:20.374+0000 7f1d0cc00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f1d081038e0 0x7f1d08199bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T03:17:20.376 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T03:17:20.374+0000 7f1d0d600640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1d08105490 0x7f1d081a1170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T03:17:20.376 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T03:17:20.374+0000 7f1d0cc00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f1d081038e0 0x7f1d08199bc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.118:47720/0 (socket says 172.21.15.118:47720) 2024-03-24T03:17:20.376 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T03:17:20.374+0000 7f1d0cc00640 1 -- 172.21.15.118:0/3671200811 learned_addr learned my addr 172.21.15.118:0/3671200811 (peer_addr_for_me v2:172.21.15.118:0/0) 2024-03-24T03:17:20.377 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T03:17:20.374+0000 7f1d07e00640 1 --2- 172.21.15.118:0/3671200811 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f1d08104ad0 0x7f1d0819a100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T03:17:20.377 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T03:17:20.374+0000 7f1d0cc00640 1 -- 172.21.15.118:0/3671200811 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1d08105490 msgr2=0x7f1d081a1170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T03:17:20.377 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T03:17:20.374+0000 7f1d0cc00640 1 --2- 172.21.15.118:0/3671200811 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1d08105490 0x7f1d081a1170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T03:17:20.377 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T03:17:20.374+0000 7f1d0cc00640 1 -- 172.21.15.118:0/3671200811 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f1d08104ad0 msgr2=0x7f1d0819a100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T03:17:20.377 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T03:17:20.374+0000 7f1d0cc00640 1 --2- 172.21.15.118:0/3671200811 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f1d08104ad0 0x7f1d0819a100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T03:17:20.377 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T03:17:20.374+0000 7f1d0cc00640 1 -- 172.21.15.118:0/3671200811 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1d081a17e0 con 0x7f1d081038e0 2024-03-24T03:17:20.377 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T03:17:20.374+0000 7f1d07e00640 1 --2- 172.21.15.118:0/3671200811 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f1d08104ad0 0x7f1d0819a100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T03:17:20.377 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T03:17:20.374+0000 7f1d0cc00640 1 --2- 172.21.15.118:0/3671200811 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f1d081038e0 0x7f1d08199bc0 secure :-1 s=READY pgs=476 cs=0 l=1 rev1=1 crypto rx=0x7f1cfc02c260 tx=0x7f1cfc02c670 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T03:17:20.377 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T03:17:20.375+0000 7f1d05600640 1 -- 172.21.15.118:0/3671200811 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1cfc041070 con 0x7f1d081038e0 2024-03-24T03:17:20.377 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T03:17:20.375+0000 7f1d05600640 1 -- 172.21.15.118:0/3671200811 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f1cfc004450 con 0x7f1d081038e0 2024-03-24T03:17:20.378 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T03:17:20.375+0000 7f1d05600640 1 -- 172.21.15.118:0/3671200811 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f1cfc034440 con 0x7f1d081038e0 2024-03-24T03:17:20.378 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T03:17:20.375+0000 7f1d0f600640 1 -- 172.21.15.118:0/3671200811 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1d081a1a70 con 0x7f1d081038e0 2024-03-24T03:17:20.378 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T03:17:20.375+0000 7f1d0f600640 1 -- 172.21.15.118:0/3671200811 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1d081a1f70 con 0x7f1d081038e0 2024-03-24T03:17:20.378 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T03:17:20.376+0000 7f1d0f600640 1 -- 172.21.15.118:0/3671200811 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1d0810ac00 con 0x7f1d081038e0 2024-03-24T03:17:20.379 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T03:17:20.377+0000 7f1d05600640 1 -- 172.21.15.118:0/3671200811 <== mon.1 v2:172.21.15.193:3300/0 4 ==== mgrmap(e 20) v1 ==== 137291+0+0 (secure 0 0 0) 0x7f1cfc044070 con 0x7f1d081038e0 2024-03-24T03:17:20.379 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T03:17:20.379+0000 7f1d05600640 1 --2- 172.21.15.118:0/3671200811 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7f1cf00a2880 0x7f1cf00a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T03:17:20.379 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T03:17:20.379+0000 7f1d05600640 1 -- 172.21.15.118:0/3671200811 <== mon.1 v2:172.21.15.193:3300/0 5 ==== osd_map(124..124 src has 1..124) v4 ==== 11201+0+0 (secure 0 0 0) 0x7f1cfc0f1110 con 0x7f1d081038e0 2024-03-24T03:17:20.379 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T03:17:20.379+0000 7f1d07e00640 1 --2- 172.21.15.118:0/3671200811 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7f1cf00a2880 0x7f1cf00a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T03:17:20.379 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T03:17:20.379+0000 7f1d07e00640 1 --2- 172.21.15.118:0/3671200811 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7f1cf00a2880 0x7f1cf00a4d40 secure :-1 s=READY pgs=636 cs=0 l=1 rev1=1 crypto rx=0x7f1d0819b0d0 tx=0x7f1cf4005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T03:17:20.382 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T03:17:20.383+0000 7f1d05600640 1 -- 172.21.15.118:0/3671200811 <== mon.1 v2:172.21.15.193:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f1cfc0bd5b0 con 0x7f1d081038e0 2024-03-24T03:17:20.582 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T03:17:20.583+0000 7f1d0f600640 1 -- 172.21.15.118:0/3671200811 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f1d080622a0 con 0x7f1d081038e0 2024-03-24T03:17:20.584 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T03:17:20.585+0000 7f1d05600640 1 -- 172.21.15.118:0/3671200811 <== mon.1 v2:172.21.15.193:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v124) v1 ==== 74+0+27305 (secure 0 0 0) 0x7f1cfc0bceb0 con 0x7f1d081038e0 2024-03-24T03:17:20.585 INFO:teuthology.orchestra.run.smithi118.stdout: 2024-03-24T03:17:20.585 INFO:teuthology.orchestra.run.smithi118.stdout:{"epoch":124,"fsid":"d79a5420-e971-11ee-95cd-87774f69a715","created":"2024-03-24T00:03:58.648524+0000","modified":"2024-03-24T00:24:59.499441+0000","last_up_change":"2024-03-24T00:12:03.809000+0000","last_in_change":"2024-03-24T00:11:44.389344+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-24T00:07:50.190965+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-24T00:12:59.993723+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-24T00:13:01.257347+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"86","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"c9590426-3901-4f68-a5ef-d57aa8fbac92","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6802","nonce":557690061},{"type":"v1","addr":"172.21.15.118:6803","nonce":557690061}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6804","nonce":557690061},{"type":"v1","addr":"172.21.15.118:6805","nonce":557690061}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6808","nonce":557690061},{"type":"v1","addr":"172.21.15.118:6809","nonce":557690061}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6806","nonce":557690061},{"type":"v1","addr":"172.21.15.118:6807","nonce":557690061}]},"public_addr":"172.21.15.118:6803/557690061","cluster_addr":"172.21.15.118:6805/557690061","heartbeat_back_addr":"172.21.15.118:6809/557690061","heartbeat_front_addr":"172.21.15.118:6807/557690061","state":["exists","up"]},{"osd":1,"uuid":"fc7f827e-ecf6-4661-aad0-7a22951892e0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6800","nonce":4203029485},{"type":"v1","addr":"172.21.15.189:6801","nonce":4203029485}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6802","nonce":4203029485},{"type":"v1","addr":"172.21.15.189:6803","nonce":4203029485}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6806","nonce":4203029485},{"type":"v1","addr":"172.21.15.189:6807","nonce":4203029485}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6804","nonce":4203029485},{"type":"v1","addr":"172.21.15.189:6805","nonce":4203029485}]},"public_addr":"172.21.15.189:6801/4203029485","cluster_addr":"172.21.15.189:6803/4203029485","heartbeat_back_addr":"172.21.15.189:6807/4203029485","heartbeat_front_addr":"172.21.15.189:6805/4203029485","state":["exists","up"]},{"osd":2,"uuid":"e73ccfe9-0e5b-4734-a154-a2f0d829fcfb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6800","nonce":3699076971},{"type":"v1","addr":"172.21.15.193:6801","nonce":3699076971}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6802","nonce":3699076971},{"type":"v1","addr":"172.21.15.193:6803","nonce":3699076971}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6806","nonce":3699076971},{"type":"v1","addr":"172.21.15.193:6807","nonce":3699076971}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6804","nonce":3699076971},{"type":"v1","addr":"172.21.15.193:6805","nonce":3699076971}]},"public_addr":"172.21.15.193:6801/3699076971","cluster_addr":"172.21.15.193:6803/3699076971","heartbeat_back_addr":"172.21.15.193:6807/3699076971","heartbeat_front_addr":"172.21.15.193:6805/3699076971","state":["exists","up"]},{"osd":3,"uuid":"63fed43b-43ce-4031-8957-7fe94725c1ed","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6810","nonce":4059617010},{"type":"v1","addr":"172.21.15.118:6811","nonce":4059617010}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6812","nonce":4059617010},{"type":"v1","addr":"172.21.15.118:6813","nonce":4059617010}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6816","nonce":4059617010},{"type":"v1","addr":"172.21.15.118:6817","nonce":4059617010}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6814","nonce":4059617010},{"type":"v1","addr":"172.21.15.118:6815","nonce":4059617010}]},"public_addr":"172.21.15.118:6811/4059617010","cluster_addr":"172.21.15.118:6813/4059617010","heartbeat_back_addr":"172.21.15.118:6817/4059617010","heartbeat_front_addr":"172.21.15.118:6815/4059617010","state":["exists","up"]},{"osd":4,"uuid":"57377f66-c3b4-43c1-8e8e-bc873672906d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":34,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6808","nonce":4075247068},{"type":"v1","addr":"172.21.15.189:6809","nonce":4075247068}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6810","nonce":4075247068},{"type":"v1","addr":"172.21.15.189:6811","nonce":4075247068}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6814","nonce":4075247068},{"type":"v1","addr":"172.21.15.189:6815","nonce":4075247068}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6812","nonce":4075247068},{"type":"v1","addr":"172.21.15.189:6813","nonce":4075247068}]},"public_addr":"172.21.15.189:6809/4075247068","cluster_addr":"172.21.15.189:6811/4075247068","heartbeat_back_addr":"172.21.15.189:6815/4075247068","heartbeat_front_addr":"172.21.15.189:6813/4075247068","state":["exists","up"]},{"osd":5,"uuid":"91970976-2193-4224-81c0-dc96cb022c85","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":39,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6808","nonce":2920363402},{"type":"v1","addr":"172.21.15.193:6809","nonce":2920363402}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6810","nonce":2920363402},{"type":"v1","addr":"172.21.15.193:6811","nonce":2920363402}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6814","nonce":2920363402},{"type":"v1","addr":"172.21.15.193:6815","nonce":2920363402}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6812","nonce":2920363402},{"type":"v1","addr":"172.21.15.193:6813","nonce":2920363402}]},"public_addr":"172.21.15.193:6809/2920363402","cluster_addr":"172.21.15.193:6811/2920363402","heartbeat_back_addr":"172.21.15.193:6815/2920363402","heartbeat_front_addr":"172.21.15.193:6813/2920363402","state":["exists","up"]},{"osd":6,"uuid":"0b77b196-8674-41c6-ad3d-726b5d01c8fd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":47,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6818","nonce":431354576},{"type":"v1","addr":"172.21.15.118:6819","nonce":431354576}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6820","nonce":431354576},{"type":"v1","addr":"172.21.15.118:6821","nonce":431354576}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6824","nonce":431354576},{"type":"v1","addr":"172.21.15.118:6825","nonce":431354576}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6822","nonce":431354576},{"type":"v1","addr":"172.21.15.118:6823","nonce":431354576}]},"public_addr":"172.21.15.118:6819/431354576","cluster_addr":"172.21.15.118:6821/431354576","heartbeat_back_addr":"172.21.15.118:6825/431354576","heartbeat_front_addr":"172.21.15.118:6823/431354576","state":["exists","up"]},{"osd":7,"uuid":"1fbbd38f-4e8e-4d4c-8e98-66f6862bc4bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":52,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6816","nonce":3667611284},{"type":"v1","addr":"172.21.15.189:6817","nonce":3667611284}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6818","nonce":3667611284},{"type":"v1","addr":"172.21.15.189:6819","nonce":3667611284}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6822","nonce":3667611284},{"type":"v1","addr":"172.21.15.189:6823","nonce":3667611284}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6820","nonce":3667611284},{"type":"v1","addr":"172.21.15.189:6821","nonce":3667611284}]},"public_addr":"172.21.15.189:6817/3667611284","cluster_addr":"172.21.15.189:6819/3667611284","heartbeat_back_addr":"172.21.15.189:6823/3667611284","heartbeat_front_addr":"172.21.15.189:6821/3667611284","state":["exists","up"]},{"osd":8,"uuid":"6462c2b5-e4b3-4b0b-bcfa-4de2c89e93e0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":60,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6816","nonce":3853800910},{"type":"v1","addr":"172.21.15.193:6817","nonce":3853800910}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6818","nonce":3853800910},{"type":"v1","addr":"172.21.15.193:6819","nonce":3853800910}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6822","nonce":3853800910},{"type":"v1","addr":"172.21.15.193:6823","nonce":3853800910}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6820","nonce":3853800910},{"type":"v1","addr":"172.21.15.193:6821","nonce":3853800910}]},"public_addr":"172.21.15.193:6817/3853800910","cluster_addr":"172.21.15.193:6819/3853800910","heartbeat_back_addr":"172.21.15.193:6823/3853800910","heartbeat_front_addr":"172.21.15.193:6821/3853800910","state":["exists","up"]},{"osd":9,"uuid":"54be277f-3596-422f-9de5-9005a8571c6f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":68,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6826","nonce":3248293936},{"type":"v1","addr":"172.21.15.118:6827","nonce":3248293936}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6828","nonce":3248293936},{"type":"v1","addr":"172.21.15.118:6829","nonce":3248293936}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6832","nonce":3248293936},{"type":"v1","addr":"172.21.15.118:6833","nonce":3248293936}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6830","nonce":3248293936},{"type":"v1","addr":"172.21.15.118:6831","nonce":3248293936}]},"public_addr":"172.21.15.118:6827/3248293936","cluster_addr":"172.21.15.118:6829/3248293936","heartbeat_back_addr":"172.21.15.118:6833/3248293936","heartbeat_front_addr":"172.21.15.118:6831/3248293936","state":["exists","up"]},{"osd":10,"uuid":"7ba69d69-ac41-44ce-bc39-c049a8ca5900","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":73,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6824","nonce":2112195760},{"type":"v1","addr":"172.21.15.189:6825","nonce":2112195760}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6826","nonce":2112195760},{"type":"v1","addr":"172.21.15.189:6827","nonce":2112195760}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6830","nonce":2112195760},{"type":"v1","addr":"172.21.15.189:6831","nonce":2112195760}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6828","nonce":2112195760},{"type":"v1","addr":"172.21.15.189:6829","nonce":2112195760}]},"public_addr":"172.21.15.189:6825/2112195760","cluster_addr":"172.21.15.189:6827/2112195760","heartbeat_back_addr":"172.21.15.189:6831/2112195760","heartbeat_front_addr":"172.21.15.189:6829/2112195760","state":["exists","up"]},{"osd":11,"uuid":"186f0b18-6b22-4eae-83b7-b21bbcce0817","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":81,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6824","nonce":3880515510},{"type":"v1","addr":"172.21.15.193:6825","nonce":3880515510}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6826","nonce":3880515510},{"type":"v1","addr":"172.21.15.193:6827","nonce":3880515510}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6830","nonce":3880515510},{"type":"v1","addr":"172.21.15.193:6831","nonce":3880515510}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6828","nonce":3880515510},{"type":"v1","addr":"172.21.15.193:6829","nonce":3880515510}]},"public_addr":"172.21.15.193:6825/3880515510","cluster_addr":"172.21.15.193:6827/3880515510","heartbeat_back_addr":"172.21.15.193:6831/3880515510","heartbeat_front_addr":"172.21.15.193:6829/3880515510","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:06:58.105543+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:07:23.795866+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:07:47.426050+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:08:19.427073+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:08:45.816082+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:09:11.081523+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:09:44.485077+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:10:11.338972+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:10:36.234024+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:11:09.730432+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:11:37.059083+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:12:02.371947+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.a","mappings":[{"from":2,"to":11}]},{"pgid":"2.14","mappings":[{"from":2,"to":4}]},{"pgid":"2.1d","mappings":[{"from":8,"to":11}]},{"pgid":"2.1f","mappings":[{"from":8,"to":1}]},{"pgid":"3.11","mappings":[{"from":4,"to":8},{"from":5,"to":9}]},{"pgid":"3.13","mappings":[{"from":4,"to":9}]},{"pgid":"3.14","mappings":[{"from":4,"to":6}]},{"pgid":"3.31","mappings":[{"from":4,"to":7}]},{"pgid":"3.38","mappings":[{"from":3,"to":9}]},{"pgid":"3.39","mappings":[{"from":5,"to":8}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.118:6837/179741564":"2024-03-25T00:24:41.735441+0000","172.21.15.118:6837/2587227307":"2024-03-25T00:24:58.665829+0000","172.21.15.118:6836/179741564":"2024-03-25T00:24:41.735441+0000","172.21.15.118:6835/1095918599":"2024-03-25T00:24:41.638747+0000","172.21.15.118:6834/1095918599":"2024-03-25T00:24:41.638747+0000","172.21.15.118:6835/3766636479":"2024-03-25T00:24:24.394949+0000","172.21.15.118:6834/3766636479":"2024-03-25T00:24:24.394949+0000","172.21.15.118:6836/1531167856":"2024-03-25T00:24:23.981049+0000","172.21.15.189:6836/2642988219":"2024-03-25T00:24:06.626820+0000","172.21.15.118:6835/2587469815":"2024-03-25T00:24:00.509562+0000","172.21.15.118:6834/2587469815":"2024-03-25T00:24:00.509562+0000","172.21.15.118:6839/282013400":"2024-03-25T00:23:44.005279+0000","172.21.15.118:6838/282013400":"2024-03-25T00:23:44.005279+0000","172.21.15.189:6836/635914573":"2024-03-25T00:23:24.718454+0000","172.21.15.118:6838/1710625324":"2024-03-25T00:23:08.950792+0000","172.21.15.193:6833/2424695757":"2024-03-25T00:22:51.575907+0000","172.21.15.118:6839/3600700815":"2024-03-25T00:22:26.228593+0000","172.21.15.118:6838/3600700815":"2024-03-25T00:22:26.228593+0000","172.21.15.118:6836/2587227307":"2024-03-25T00:24:58.665829+0000","172.21.15.189:6837/2350260652":"2024-03-25T00:22:09.715485+0000","172.21.15.118:6838/979323377":"2024-03-25T00:21:53.776281+0000","172.21.15.118:6835/456491655":"2024-03-25T00:24:59.499390+0000","172.21.15.193:6833/1023428412":"2024-03-25T00:20:03.007142+0000","172.21.15.193:6835/1699655293":"2024-03-25T00:20:48.266150+0000","172.21.15.193:6834/2055310541":"2024-03-25T00:19:47.282228+0000","172.21.15.118:6801/389249272":"2024-03-25T00:05:05.458796+0000","172.21.15.193:6834/3744499911":"2024-03-25T00:20:18.239231+0000","172.21.15.118:6836/3985972391":"2024-03-25T00:23:42.934077+0000","172.21.15.193:6832/3097400619":"2024-03-25T00:20:33.239209+0000","172.21.15.118:6837/3985972391":"2024-03-25T00:23:42.934077+0000","172.21.15.118:6835/3194741373":"2024-03-25T00:23:25.503800+0000","172.21.15.193:6833/4276594827":"2024-03-25T00:21:38.394987+0000","172.21.15.118:6836/1374462956":"2024-03-25T00:23:08.677886+0000","172.21.15.118:6800/389249272":"2024-03-25T00:05:05.458796+0000","172.21.15.118:6834/456491655":"2024-03-25T00:24:59.499390+0000","172.21.15.118:0/2101854694":"2024-03-25T00:05:05.458796+0000","172.21.15.193:6835/2814329894":"2024-03-25T00:21:53.247861+0000","172.21.15.118:6837/1531167856":"2024-03-25T00:24:23.981049+0000","172.21.15.189:6837/75274228":"2024-03-25T00:22:44.728301+0000","172.21.15.193:6835/2264220953":"2024-03-25T00:22:27.928314+0000","172.21.15.118:0/630779359":"2024-03-25T00:05:05.458796+0000","172.21.15.118:6834/3194741373":"2024-03-25T00:23:25.503800+0000","172.21.15.118:6837/1374462956":"2024-03-25T00:23:08.677886+0000","172.21.15.193:6832/2424695757":"2024-03-25T00:22:51.575907+0000","172.21.15.193:6832/1023428412":"2024-03-25T00:20:03.007142+0000","172.21.15.189:6836/75274228":"2024-03-25T00:22:44.728301+0000","172.21.15.189:6837/596160018":"2024-03-25T00:21:38.304407+0000","172.21.15.118:0/368712314":"2024-03-25T00:04:31.283725+0000","172.21.15.189:6837/635914573":"2024-03-25T00:23:24.718454+0000","172.21.15.193:6835/2055310541":"2024-03-25T00:19:47.282228+0000","172.21.15.118:0/2004775597":"2024-03-25T00:04:31.283725+0000","172.21.15.193:6835/3744499911":"2024-03-25T00:20:18.239231+0000","172.21.15.193:6832/2276762432":"2024-03-25T00:22:09.817146+0000","172.21.15.118:6800/1463147955":"2024-03-25T00:04:31.283725+0000","172.21.15.189:6837/2642988219":"2024-03-25T00:24:06.626820+0000","172.21.15.118:6801/1463147955":"2024-03-25T00:04:31.283725+0000","172.21.15.118:6839/3285788503":"2024-03-25T00:20:49.251252+0000","172.21.15.193:6835/799001823":"2024-03-25T00:21:21.268101+0000","172.21.15.193:6834/1699655293":"2024-03-25T00:20:48.266150+0000","172.21.15.118:6839/1710625324":"2024-03-25T00:23:08.950792+0000","172.21.15.118:0/1003670455":"2024-03-25T00:05:05.458796+0000","172.21.15.118:6838/3285788503":"2024-03-25T00:20:49.251252+0000","172.21.15.193:6834/2264220953":"2024-03-25T00:22:27.928314+0000","172.21.15.193:6832/1233062505":"2024-03-25T00:21:04.998288+0000","172.21.15.193:6833/1233062505":"2024-03-25T00:21:04.998288+0000","172.21.15.189:6836/596160018":"2024-03-25T00:21:38.304407+0000","172.21.15.118:0/2158921029":"2024-03-25T00:04:31.283725+0000","172.21.15.193:6834/2814329894":"2024-03-25T00:21:53.247861+0000","172.21.15.189:6836/2350260652":"2024-03-25T00:22:09.715485+0000","172.21.15.193:6833/3097400619":"2024-03-25T00:20:33.239209+0000","172.21.15.189:6837/3450187137":"2024-03-25T00:21:05.241581+0000","172.21.15.193:6834/799001823":"2024-03-25T00:21:21.268101+0000","172.21.15.118:6839/979323377":"2024-03-25T00:21:53.776281+0000","172.21.15.189:6836/3450187137":"2024-03-25T00:21:05.241581+0000","172.21.15.118:6838/3808689299":"2024-03-25T00:21:21.735792+0000","172.21.15.118:6839/3808689299":"2024-03-25T00:21:21.735792+0000","172.21.15.193:6833/2276762432":"2024-03-25T00:22:09.817146+0000","172.21.15.193:6832/4276594827":"2024-03-25T00:21:38.394987+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-24T03:17:20.587 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T03:17:20.587+0000 7f1d0f600640 1 -- 172.21.15.118:0/3671200811 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7f1cf00a2880 msgr2=0x7f1cf00a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T03:17:20.587 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T03:17:20.587+0000 7f1d0f600640 1 --2- 172.21.15.118:0/3671200811 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7f1cf00a2880 0x7f1cf00a4d40 secure :-1 s=READY pgs=636 cs=0 l=1 rev1=1 crypto rx=0x7f1d0819b0d0 tx=0x7f1cf4005f90 comp rx=0 tx=0).stop 2024-03-24T03:17:20.587 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T03:17:20.587+0000 7f1d0f600640 1 -- 172.21.15.118:0/3671200811 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f1d081038e0 msgr2=0x7f1d08199bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T03:17:20.587 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T03:17:20.587+0000 7f1d0f600640 1 --2- 172.21.15.118:0/3671200811 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f1d081038e0 0x7f1d08199bc0 secure :-1 s=READY pgs=476 cs=0 l=1 rev1=1 crypto rx=0x7f1cfc02c260 tx=0x7f1cfc02c670 comp rx=0 tx=0).stop 2024-03-24T03:17:20.588 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T03:17:20.588+0000 7f1d0f600640 1 -- 172.21.15.118:0/3671200811 shutdown_connections 2024-03-24T03:17:20.588 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T03:17:20.588+0000 7f1d0f600640 1 --2- 172.21.15.118:0/3671200811 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7f1cf00a2880 0x7f1cf00a4d40 unknown :-1 s=CLOSED pgs=636 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T03:17:20.588 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T03:17:20.588+0000 7f1d0f600640 1 --2- 172.21.15.118:0/3671200811 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f1d08105490 0x7f1d081a1170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T03:17:20.588 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T03:17:20.588+0000 7f1d0f600640 1 --2- 172.21.15.118:0/3671200811 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f1d08104ad0 0x7f1d0819a100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T03:17:20.588 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T03:17:20.588+0000 7f1d0f600640 1 --2- 172.21.15.118:0/3671200811 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f1d081038e0 0x7f1d08199bc0 unknown :-1 s=CLOSED pgs=476 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T03:17:20.588 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T03:17:20.588+0000 7f1d0f600640 1 -- 172.21.15.118:0/3671200811 >> 172.21.15.118:0/3671200811 conn(0x7f1d080fee90 msgr2=0x7f1d08107dd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T03:17:20.588 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T03:17:20.588+0000 7f1d0f600640 1 -- 172.21.15.118:0/3671200811 shutdown_connections 2024-03-24T03:17:20.588 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T03:17:20.588+0000 7f1d0f600640 1 -- 172.21.15.118:0/3671200811 wait complete. 2024-03-24T03:17:20.606 INFO:tasks.check_counter.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-24T00:07:50.190965+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-24T00:12:59.993723+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '85', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.8799999952316284, 'score_stable': 1.8799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.8799999952316284, 'raw_score_stable': 1.8799999952316284, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-03-24T00:13:01.257347+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '86', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-24T03:17:20.606 DEBUG:teuthology.orchestra.run.smithi118:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-24T03:17:20.823 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T03:17:20.823+0000 7fa603600640 1 -- 172.21.15.118:0/2972883838 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa5fc102130 msgr2=0x7fa5fc104520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T03:17:20.823 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T03:17:20.823+0000 7fa603600640 1 --2- 172.21.15.118:0/2972883838 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa5fc102130 0x7fa5fc104520 secure :-1 s=READY pgs=471 cs=0 l=1 rev1=1 crypto rx=0x7fa5f0009900 tx=0x7fa5f002bc30 comp rx=0 tx=0).stop 2024-03-24T03:17:20.823 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T03:17:20.823+0000 7fa603600640 1 -- 172.21.15.118:0/2972883838 shutdown_connections 2024-03-24T03:17:20.823 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T03:17:20.823+0000 7fa603600640 1 --2- 172.21.15.118:0/2972883838 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fa5fc107390 0x7fa5fc1097e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T03:17:20.823 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T03:17:20.823+0000 7fa603600640 1 --2- 172.21.15.118:0/2972883838 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fa5fc104a60 0x7fa5fc106e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T03:17:20.824 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T03:17:20.823+0000 7fa603600640 1 --2- 172.21.15.118:0/2972883838 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa5fc102130 0x7fa5fc104520 unknown :-1 s=CLOSED pgs=471 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T03:17:20.824 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T03:17:20.823+0000 7fa603600640 1 -- 172.21.15.118:0/2972883838 >> 172.21.15.118:0/2972883838 conn(0x7fa5fc0fbb40 msgr2=0x7fa5fc0fdf60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T03:17:20.824 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T03:17:20.823+0000 7fa603600640 1 -- 172.21.15.118:0/2972883838 shutdown_connections 2024-03-24T03:17:20.824 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T03:17:20.823+0000 7fa603600640 1 -- 172.21.15.118:0/2972883838 wait complete. 2024-03-24T03:17:20.826 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T03:17:20.825+0000 7fa603600640 1 Processor -- start 2024-03-24T03:17:20.826 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T03:17:20.825+0000 7fa603600640 1 -- start start 2024-03-24T03:17:20.826 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T03:17:20.825+0000 7fa603600640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa5fc102130 0x7fa5fc197a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T03:17:20.826 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T03:17:20.825+0000 7fa603600640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fa5fc104a60 0x7fa5fc197f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T03:17:20.827 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T03:17:20.825+0000 7fa603600640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fa5fc107390 0x7fa5fc19efd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T03:17:20.827 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T03:17:20.825+0000 7fa603600640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa5fc10d3a0 con 0x7fa5fc104a60 2024-03-24T03:17:20.827 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T03:17:20.825+0000 7fa603600640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa5fc10d220 con 0x7fa5fc102130 2024-03-24T03:17:20.827 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T03:17:20.825+0000 7fa603600640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa5fc10d520 con 0x7fa5fc107390 2024-03-24T03:17:20.827 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T03:17:20.826+0000 7fa600c00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa5fc102130 0x7fa5fc197a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T03:17:20.827 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T03:17:20.826+0000 7fa600c00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa5fc102130 0x7fa5fc197a20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.118:32968/0 (socket says 172.21.15.118:32968) 2024-03-24T03:17:20.827 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T03:17:20.826+0000 7fa600c00640 1 -- 172.21.15.118:0/2037400246 learned_addr learned my addr 172.21.15.118:0/2037400246 (peer_addr_for_me v2:172.21.15.118:0/0) 2024-03-24T03:17:20.827 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T03:17:20.826+0000 7fa5fbe00640 1 --2- 172.21.15.118:0/2037400246 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fa5fc104a60 0x7fa5fc197f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T03:17:20.827 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T03:17:20.826+0000 7fa601600640 1 --2- 172.21.15.118:0/2037400246 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fa5fc107390 0x7fa5fc19efd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T03:17:20.828 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T03:17:20.826+0000 7fa600c00640 1 -- 172.21.15.118:0/2037400246 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fa5fc107390 msgr2=0x7fa5fc19efd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T03:17:20.828 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T03:17:20.826+0000 7fa600c00640 1 --2- 172.21.15.118:0/2037400246 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fa5fc107390 0x7fa5fc19efd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T03:17:20.828 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T03:17:20.826+0000 7fa600c00640 1 -- 172.21.15.118:0/2037400246 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fa5fc104a60 msgr2=0x7fa5fc197f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T03:17:20.828 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T03:17:20.826+0000 7fa600c00640 1 --2- 172.21.15.118:0/2037400246 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fa5fc104a60 0x7fa5fc197f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T03:17:20.828 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T03:17:20.826+0000 7fa600c00640 1 -- 172.21.15.118:0/2037400246 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa5fc19f640 con 0x7fa5fc102130 2024-03-24T03:17:20.828 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T03:17:20.826+0000 7fa600c00640 1 --2- 172.21.15.118:0/2037400246 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa5fc102130 0x7fa5fc197a20 secure :-1 s=READY pgs=472 cs=0 l=1 rev1=1 crypto rx=0x7fa5f0000c00 tx=0x7fa5f002c650 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T03:17:20.828 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T03:17:20.827+0000 7fa5f9600640 1 -- 172.21.15.118:0/2037400246 <== mon.2 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa5f0043020 con 0x7fa5fc102130 2024-03-24T03:17:20.828 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T03:17:20.827+0000 7fa5f9600640 1 -- 172.21.15.118:0/2037400246 <== mon.2 v2:172.21.15.189:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fa5f00054e0 con 0x7fa5fc102130 2024-03-24T03:17:20.829 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T03:17:20.827+0000 7fa5f9600640 1 -- 172.21.15.118:0/2037400246 <== mon.2 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa5f0003c40 con 0x7fa5fc102130 2024-03-24T03:17:20.829 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T03:17:20.827+0000 7fa603600640 1 -- 172.21.15.118:0/2037400246 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa5fc19f8d0 con 0x7fa5fc102130 2024-03-24T03:17:20.829 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T03:17:20.827+0000 7fa603600640 1 -- 172.21.15.118:0/2037400246 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa5fc19fe70 con 0x7fa5fc102130 2024-03-24T03:17:20.829 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T03:17:20.828+0000 7fa603600640 1 -- 172.21.15.118:0/2037400246 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa5fc0fd630 con 0x7fa5fc102130 2024-03-24T03:17:20.829 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T03:17:20.830+0000 7fa5f9600640 1 -- 172.21.15.118:0/2037400246 <== mon.2 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 20) v1 ==== 137291+0+0 (secure 0 0 0) 0x7fa5f0046020 con 0x7fa5fc102130 2024-03-24T03:17:20.830 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T03:17:20.831+0000 7fa5f9600640 1 --2- 172.21.15.118:0/2037400246 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7fa5e40a2880 0x7fa5e40a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T03:17:20.830 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T03:17:20.831+0000 7fa5f9600640 1 -- 172.21.15.118:0/2037400246 <== mon.2 v2:172.21.15.189:3300/0 5 ==== osd_map(124..124 src has 1..124) v4 ==== 11201+0+0 (secure 0 0 0) 0x7fa5f00f1620 con 0x7fa5fc102130 2024-03-24T03:17:20.831 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T03:17:20.831+0000 7fa5fbe00640 1 --2- 172.21.15.118:0/2037400246 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7fa5e40a2880 0x7fa5e40a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T03:17:20.831 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T03:17:20.832+0000 7fa5fbe00640 1 --2- 172.21.15.118:0/2037400246 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7fa5e40a2880 0x7fa5e40a4d40 secure :-1 s=READY pgs=637 cs=0 l=1 rev1=1 crypto rx=0x7fa5fc198f30 tx=0x7fa5e8005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T03:17:20.834 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T03:17:20.835+0000 7fa5f9600640 1 -- 172.21.15.118:0/2037400246 <== mon.2 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7fa5f004d020 con 0x7fa5fc102130 2024-03-24T03:17:21.035 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T03:17:21.035+0000 7fa603600640 1 -- 172.21.15.118:0/2037400246 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fa5fc0622a0 con 0x7fa5fc102130 2024-03-24T03:17:21.035 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T03:17:21.036+0000 7fa5f9600640 1 -- 172.21.15.118:0/2037400246 <== mon.2 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v124) v1 ==== 93+0+10 (secure 0 0 0) 0x7fa5f00bdac0 con 0x7fa5fc102130 2024-03-24T03:17:21.035 INFO:teuthology.orchestra.run.smithi118.stdout:pg_num: 1 2024-03-24T03:17:21.038 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T03:17:21.038+0000 7fa603600640 1 -- 172.21.15.118:0/2037400246 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7fa5e40a2880 msgr2=0x7fa5e40a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T03:17:21.038 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T03:17:21.038+0000 7fa603600640 1 --2- 172.21.15.118:0/2037400246 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7fa5e40a2880 0x7fa5e40a4d40 secure :-1 s=READY pgs=637 cs=0 l=1 rev1=1 crypto rx=0x7fa5fc198f30 tx=0x7fa5e8005f90 comp rx=0 tx=0).stop 2024-03-24T03:17:21.038 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T03:17:21.038+0000 7fa603600640 1 -- 172.21.15.118:0/2037400246 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa5fc102130 msgr2=0x7fa5fc197a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T03:17:21.038 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T03:17:21.038+0000 7fa603600640 1 --2- 172.21.15.118:0/2037400246 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa5fc102130 0x7fa5fc197a20 secure :-1 s=READY pgs=472 cs=0 l=1 rev1=1 crypto rx=0x7fa5f0000c00 tx=0x7fa5f002c650 comp rx=0 tx=0).stop 2024-03-24T03:17:21.039 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T03:17:21.038+0000 7fa603600640 1 -- 172.21.15.118:0/2037400246 shutdown_connections 2024-03-24T03:17:21.039 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T03:17:21.038+0000 7fa603600640 1 --2- 172.21.15.118:0/2037400246 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7fa5e40a2880 0x7fa5e40a4d40 unknown :-1 s=CLOSED pgs=637 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T03:17:21.039 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T03:17:21.038+0000 7fa603600640 1 --2- 172.21.15.118:0/2037400246 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fa5fc107390 0x7fa5fc19efd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T03:17:21.039 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T03:17:21.038+0000 7fa603600640 1 --2- 172.21.15.118:0/2037400246 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fa5fc104a60 0x7fa5fc197f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T03:17:21.039 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T03:17:21.038+0000 7fa603600640 1 --2- 172.21.15.118:0/2037400246 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa5fc102130 0x7fa5fc197a20 unknown :-1 s=CLOSED pgs=472 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T03:17:21.039 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T03:17:21.038+0000 7fa603600640 1 -- 172.21.15.118:0/2037400246 >> 172.21.15.118:0/2037400246 conn(0x7fa5fc0fbb40 msgr2=0x7fa5fc102b90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T03:17:21.039 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T03:17:21.039+0000 7fa603600640 1 -- 172.21.15.118:0/2037400246 shutdown_connections 2024-03-24T03:17:21.039 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T03:17:21.039+0000 7fa603600640 1 -- 172.21.15.118:0/2037400246 wait complete. 2024-03-24T03:17:21.056 DEBUG:teuthology.orchestra.run.smithi118:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-24T03:17:21.270 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T03:17:21.270+0000 7f721b600640 1 -- 172.21.15.118:0/127535208 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f7214108db0 msgr2=0x7f72141091d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T03:17:21.270 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T03:17:21.270+0000 7f721b600640 1 --2- 172.21.15.118:0/127535208 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f7214108db0 0x7f72141091d0 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7f7208009900 tx=0x7f720802bc50 comp rx=0 tx=0).stop 2024-03-24T03:17:21.270 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T03:17:21.270+0000 7f721b600640 1 -- 172.21.15.118:0/127535208 shutdown_connections 2024-03-24T03:17:21.271 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T03:17:21.270+0000 7f721b600640 1 --2- 172.21.15.118:0/127535208 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f721410a960 0x7f72141111f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T03:17:21.271 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T03:17:21.270+0000 7f721b600640 1 --2- 172.21.15.118:0/127535208 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f7214109fa0 0x7f721410a420 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T03:17:21.271 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T03:17:21.270+0000 7f721b600640 1 --2- 172.21.15.118:0/127535208 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f7214108db0 0x7f72141091d0 unknown :-1 s=CLOSED pgs=236 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T03:17:21.271 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T03:17:21.270+0000 7f721b600640 1 -- 172.21.15.118:0/127535208 >> 172.21.15.118:0/127535208 conn(0x7f721406f530 msgr2=0x7f7214071990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T03:17:21.271 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T03:17:21.271+0000 7f721b600640 1 -- 172.21.15.118:0/127535208 shutdown_connections 2024-03-24T03:17:21.271 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T03:17:21.271+0000 7f721b600640 1 -- 172.21.15.118:0/127535208 wait complete. 2024-03-24T03:17:21.273 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T03:17:21.271+0000 7f721b600640 1 Processor -- start 2024-03-24T03:17:21.273 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T03:17:21.272+0000 7f721b600640 1 -- start start 2024-03-24T03:17:21.273 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T03:17:21.272+0000 7f721b600640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f7214108db0 0x7f72141a2470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T03:17:21.274 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T03:17:21.272+0000 7f721b600640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7214109fa0 0x7f72141a29b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T03:17:21.274 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T03:17:21.272+0000 7f721b600640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f721410a960 0x7f72141a9a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T03:17:21.274 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T03:17:21.272+0000 7f721b600640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7214115bb0 con 0x7f721410a960 2024-03-24T03:17:21.274 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T03:17:21.272+0000 7f721b600640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7214115a30 con 0x7f7214109fa0 2024-03-24T03:17:21.274 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T03:17:21.272+0000 7f721b600640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7214115d30 con 0x7f7214108db0 2024-03-24T03:17:21.274 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T03:17:21.272+0000 7f7218c00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f7214108db0 0x7f72141a2470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T03:17:21.274 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T03:17:21.272+0000 7f7213e00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7214109fa0 0x7f72141a29b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T03:17:21.274 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T03:17:21.272+0000 7f7218c00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f7214108db0 0x7f72141a2470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.118:47764/0 (socket says 172.21.15.118:47764) 2024-03-24T03:17:21.275 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T03:17:21.272+0000 7f7219600640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f721410a960 0x7f72141a9a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T03:17:21.275 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T03:17:21.272+0000 7f7218c00640 1 -- 172.21.15.118:0/520427555 learned_addr learned my addr 172.21.15.118:0/520427555 (peer_addr_for_me v2:172.21.15.118:0/0) 2024-03-24T03:17:21.275 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T03:17:21.273+0000 7f7213e00640 1 -- 172.21.15.118:0/520427555 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f7214108db0 msgr2=0x7f72141a2470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T03:17:21.275 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T03:17:21.273+0000 7f7213e00640 1 --2- 172.21.15.118:0/520427555 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f7214108db0 0x7f72141a2470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T03:17:21.275 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T03:17:21.273+0000 7f7213e00640 1 -- 172.21.15.118:0/520427555 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f721410a960 msgr2=0x7f72141a9a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T03:17:21.275 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T03:17:21.273+0000 7f7213e00640 1 --2- 172.21.15.118:0/520427555 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f721410a960 0x7f72141a9a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T03:17:21.275 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T03:17:21.273+0000 7f7213e00640 1 -- 172.21.15.118:0/520427555 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f72141aa090 con 0x7f7214109fa0 2024-03-24T03:17:21.275 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T03:17:21.273+0000 7f7218c00640 1 --2- 172.21.15.118:0/520427555 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f7214108db0 0x7f72141a2470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T03:17:21.275 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T03:17:21.273+0000 7f7213e00640 1 --2- 172.21.15.118:0/520427555 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7214109fa0 0x7f72141a29b0 secure :-1 s=READY pgs=473 cs=0 l=1 rev1=1 crypto rx=0x7f720000e9f0 tx=0x7f720000edf0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T03:17:21.276 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T03:17:21.273+0000 7f7211600640 1 -- 172.21.15.118:0/520427555 <== mon.2 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f720000ccc0 con 0x7f7214109fa0 2024-03-24T03:17:21.276 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T03:17:21.273+0000 7f721b600640 1 -- 172.21.15.118:0/520427555 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f72141aa380 con 0x7f7214109fa0 2024-03-24T03:17:21.276 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T03:17:21.273+0000 7f7211600640 1 -- 172.21.15.118:0/520427555 <== mon.2 v2:172.21.15.189:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f7200010910 con 0x7f7214109fa0 2024-03-24T03:17:21.276 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T03:17:21.273+0000 7f7211600640 1 -- 172.21.15.118:0/520427555 <== mon.2 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f7200018790 con 0x7f7214109fa0 2024-03-24T03:17:21.276 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T03:17:21.273+0000 7f721b600640 1 -- 172.21.15.118:0/520427555 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f72141aa920 con 0x7f7214109fa0 2024-03-24T03:17:21.276 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T03:17:21.274+0000 7f721b600640 1 -- 172.21.15.118:0/520427555 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f72141aab80 con 0x7f7214109fa0 2024-03-24T03:17:21.277 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T03:17:21.277+0000 7f7211600640 1 -- 172.21.15.118:0/520427555 <== mon.2 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 20) v1 ==== 137291+0+0 (secure 0 0 0) 0x7f7200010ac0 con 0x7f7214109fa0 2024-03-24T03:17:21.278 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T03:17:21.278+0000 7f7211600640 1 --2- 172.21.15.118:0/520427555 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7f71fc0a2880 0x7f71fc0a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T03:17:21.278 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T03:17:21.278+0000 7f7211600640 1 -- 172.21.15.118:0/520427555 <== mon.2 v2:172.21.15.189:3300/0 5 ==== osd_map(124..124 src has 1..124) v4 ==== 11201+0+0 (secure 0 0 0) 0x7f72000cf930 con 0x7f7214109fa0 2024-03-24T03:17:21.278 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T03:17:21.279+0000 7f7218c00640 1 --2- 172.21.15.118:0/520427555 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7f71fc0a2880 0x7f71fc0a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T03:17:21.278 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T03:17:21.279+0000 7f7218c00640 1 --2- 172.21.15.118:0/520427555 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7f71fc0a2880 0x7f71fc0a4d40 secure :-1 s=READY pgs=638 cs=0 l=1 rev1=1 crypto rx=0x7f720802c900 tx=0x7f7208000bc0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T03:17:21.286 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T03:17:21.287+0000 7f7211600640 1 -- 172.21.15.118:0/520427555 <== mon.2 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f720009bdd0 con 0x7f7214109fa0 2024-03-24T03:17:21.484 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T03:17:21.484+0000 7f721b600640 1 -- 172.21.15.118:0/520427555 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f72141aad50 con 0x7f7214109fa0 2024-03-24T03:17:21.485 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T03:17:21.485+0000 7f7211600640 1 -- 172.21.15.118:0/520427555 <== mon.2 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v124) v1 ==== 104+0+11 (secure 0 0 0) 0x7f720009b6d0 con 0x7f7214109fa0 2024-03-24T03:17:21.485 INFO:teuthology.orchestra.run.smithi118.stdout:pg_num: 32 2024-03-24T03:17:21.488 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T03:17:21.488+0000 7f721b600640 1 -- 172.21.15.118:0/520427555 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7f71fc0a2880 msgr2=0x7f71fc0a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T03:17:21.488 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T03:17:21.488+0000 7f721b600640 1 --2- 172.21.15.118:0/520427555 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7f71fc0a2880 0x7f71fc0a4d40 secure :-1 s=READY pgs=638 cs=0 l=1 rev1=1 crypto rx=0x7f720802c900 tx=0x7f7208000bc0 comp rx=0 tx=0).stop 2024-03-24T03:17:21.488 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T03:17:21.488+0000 7f721b600640 1 -- 172.21.15.118:0/520427555 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7214109fa0 msgr2=0x7f72141a29b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T03:17:21.488 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T03:17:21.488+0000 7f721b600640 1 --2- 172.21.15.118:0/520427555 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7214109fa0 0x7f72141a29b0 secure :-1 s=READY pgs=473 cs=0 l=1 rev1=1 crypto rx=0x7f720000e9f0 tx=0x7f720000edf0 comp rx=0 tx=0).stop 2024-03-24T03:17:21.488 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T03:17:21.488+0000 7f721b600640 1 -- 172.21.15.118:0/520427555 shutdown_connections 2024-03-24T03:17:21.488 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T03:17:21.488+0000 7f721b600640 1 --2- 172.21.15.118:0/520427555 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7f71fc0a2880 0x7f71fc0a4d40 unknown :-1 s=CLOSED pgs=638 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T03:17:21.489 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T03:17:21.488+0000 7f721b600640 1 --2- 172.21.15.118:0/520427555 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f721410a960 0x7f72141a9a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T03:17:21.489 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T03:17:21.488+0000 7f721b600640 1 --2- 172.21.15.118:0/520427555 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f7214109fa0 0x7f72141a29b0 unknown :-1 s=CLOSED pgs=473 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T03:17:21.489 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T03:17:21.488+0000 7f721b600640 1 --2- 172.21.15.118:0/520427555 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f7214108db0 0x7f72141a2470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T03:17:21.489 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T03:17:21.488+0000 7f721b600640 1 -- 172.21.15.118:0/520427555 >> 172.21.15.118:0/520427555 conn(0x7f721406f530 msgr2=0x7f7214111bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T03:17:21.489 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T03:17:21.488+0000 7f721b600640 1 -- 172.21.15.118:0/520427555 shutdown_connections 2024-03-24T03:17:21.489 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T03:17:21.488+0000 7f721b600640 1 -- 172.21.15.118:0/520427555 wait complete. 2024-03-24T03:17:21.508 DEBUG:teuthology.orchestra.run.smithi118:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-24T03:17:21.695 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 03:17:21 smithi118 ceph-mon[31535]: pgmap v6095: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T03:17:21.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 03:17:21 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 10336.761290 secs 2024-03-24T03:17:21.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 03:17:21 smithi118 ceph-mon[31535]: from='client.? 172.21.15.118:0/3671200811' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-24T03:17:21.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 03:17:21 smithi118 ceph-mon[31535]: from='client.? 172.21.15.118:0/2037400246' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-24T03:17:21.703 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 03:17:21 smithi189 ceph-mon[36187]: pgmap v6095: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T03:17:21.703 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 03:17:21 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 10336.761290 secs 2024-03-24T03:17:21.703 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 03:17:21 smithi189 ceph-mon[36187]: from='client.? 172.21.15.118:0/3671200811' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-24T03:17:21.704 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 03:17:21 smithi189 ceph-mon[36187]: from='client.? 172.21.15.118:0/2037400246' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-24T03:17:21.710 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 03:17:21 smithi193 ceph-mon[36878]: pgmap v6095: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T03:17:21.710 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 03:17:21 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 10336.761290 secs 2024-03-24T03:17:21.710 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 03:17:21 smithi193 ceph-mon[36878]: from='client.? 172.21.15.118:0/3671200811' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-24T03:17:21.710 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 03:17:21 smithi193 ceph-mon[36878]: from='client.? 172.21.15.118:0/2037400246' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-24T03:17:21.727 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T03:17:21.727+0000 7fdd31c00640 1 -- 172.21.15.118:0/320104599 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fdd2c0756e0 msgr2=0x7fdd2c073b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T03:17:21.727 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T03:17:21.727+0000 7fdd31c00640 1 --2- 172.21.15.118:0/320104599 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fdd2c0756e0 0x7fdd2c073b30 secure :-1 s=READY pgs=474 cs=0 l=1 rev1=1 crypto rx=0x7fdd1c009900 tx=0x7fdd1c02bc40 comp rx=0 tx=0).stop 2024-03-24T03:17:21.727 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T03:17:21.727+0000 7fdd31c00640 1 -- 172.21.15.118:0/320104599 shutdown_connections 2024-03-24T03:17:21.727 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T03:17:21.727+0000 7fdd31c00640 1 --2- 172.21.15.118:0/320104599 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fdd2c109440 0x7fdd2c10b8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T03:17:21.728 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T03:17:21.727+0000 7fdd31c00640 1 --2- 172.21.15.118:0/320104599 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fdd2c074100 0x7fdd2c074580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T03:17:21.728 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T03:17:21.728+0000 7fdd31c00640 1 --2- 172.21.15.118:0/320104599 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fdd2c0756e0 0x7fdd2c073b30 unknown :-1 s=CLOSED pgs=474 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T03:17:21.728 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T03:17:21.728+0000 7fdd31c00640 1 -- 172.21.15.118:0/320104599 >> 172.21.15.118:0/320104599 conn(0x7fdd2c0fcb10 msgr2=0x7fdd2c0fef30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T03:17:21.728 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T03:17:21.728+0000 7fdd31c00640 1 -- 172.21.15.118:0/320104599 shutdown_connections 2024-03-24T03:17:21.728 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T03:17:21.728+0000 7fdd31c00640 1 -- 172.21.15.118:0/320104599 wait complete. 2024-03-24T03:17:21.730 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T03:17:21.729+0000 7fdd31c00640 1 Processor -- start 2024-03-24T03:17:21.730 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T03:17:21.729+0000 7fdd31c00640 1 -- start start 2024-03-24T03:17:21.730 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T03:17:21.729+0000 7fdd31c00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fdd2c074100 0x7fdd2c199a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T03:17:21.731 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T03:17:21.729+0000 7fdd31c00640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fdd2c0756e0 0x7fdd2c199fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T03:17:21.731 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T03:17:21.729+0000 7fdd31c00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fdd2c109440 0x7fdd2c1a1010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T03:17:21.731 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T03:17:21.729+0000 7fdd31c00640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdd2c10f410 con 0x7fdd2c0756e0 2024-03-24T03:17:21.731 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T03:17:21.729+0000 7fdd31c00640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdd2c10f290 con 0x7fdd2c109440 2024-03-24T03:17:21.731 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T03:17:21.729+0000 7fdd31c00640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdd2c10f590 con 0x7fdd2c074100 2024-03-24T03:17:21.731 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T03:17:21.729+0000 7fdd2be00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fdd2c074100 0x7fdd2c199a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T03:17:21.731 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T03:17:21.729+0000 7fdd2b400640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fdd2c0756e0 0x7fdd2c199fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T03:17:21.731 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T03:17:21.730+0000 7fdd2be00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fdd2c074100 0x7fdd2c199a60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.118:47786/0 (socket says 172.21.15.118:47786) 2024-03-24T03:17:21.732 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T03:17:21.730+0000 7fdd31200640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fdd2c109440 0x7fdd2c1a1010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T03:17:21.732 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T03:17:21.730+0000 7fdd2be00640 1 -- 172.21.15.118:0/278904405 learned_addr learned my addr 172.21.15.118:0/278904405 (peer_addr_for_me v2:172.21.15.118:0/0) 2024-03-24T03:17:21.732 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T03:17:21.730+0000 7fdd2be00640 1 -- 172.21.15.118:0/278904405 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fdd2c109440 msgr2=0x7fdd2c1a1010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T03:17:21.732 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T03:17:21.730+0000 7fdd2be00640 1 --2- 172.21.15.118:0/278904405 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fdd2c109440 0x7fdd2c1a1010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T03:17:21.732 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T03:17:21.730+0000 7fdd2be00640 1 -- 172.21.15.118:0/278904405 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fdd2c0756e0 msgr2=0x7fdd2c199fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T03:17:21.732 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T03:17:21.730+0000 7fdd2be00640 1 --2- 172.21.15.118:0/278904405 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fdd2c0756e0 0x7fdd2c199fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T03:17:21.732 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T03:17:21.730+0000 7fdd2be00640 1 -- 172.21.15.118:0/278904405 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdd2c1a1680 con 0x7fdd2c074100 2024-03-24T03:17:21.732 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T03:17:21.730+0000 7fdd31200640 1 --2- 172.21.15.118:0/278904405 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fdd2c109440 0x7fdd2c1a1010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T03:17:21.733 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T03:17:21.730+0000 7fdd2be00640 1 --2- 172.21.15.118:0/278904405 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fdd2c074100 0x7fdd2c199a60 secure :-1 s=READY pgs=477 cs=0 l=1 rev1=1 crypto rx=0x7fdd1c02c8f0 tx=0x7fdd1c02c660 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T03:17:21.733 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T03:17:21.731+0000 7fdd28c00640 1 -- 172.21.15.118:0/278904405 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdd1c041070 con 0x7fdd2c074100 2024-03-24T03:17:21.733 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T03:17:21.731+0000 7fdd28c00640 1 -- 172.21.15.118:0/278904405 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fdd1c004450 con 0x7fdd2c074100 2024-03-24T03:17:21.733 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T03:17:21.731+0000 7fdd28c00640 1 -- 172.21.15.118:0/278904405 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fdd1c034440 con 0x7fdd2c074100 2024-03-24T03:17:21.733 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T03:17:21.731+0000 7fdd31c00640 1 -- 172.21.15.118:0/278904405 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdd2c1a1910 con 0x7fdd2c074100 2024-03-24T03:17:21.733 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T03:17:21.731+0000 7fdd31c00640 1 -- 172.21.15.118:0/278904405 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdd2c1a1e50 con 0x7fdd2c074100 2024-03-24T03:17:21.734 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T03:17:21.732+0000 7fdd31c00640 1 -- 172.21.15.118:0/278904405 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdd2c108170 con 0x7fdd2c074100 2024-03-24T03:17:21.734 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T03:17:21.734+0000 7fdd28c00640 1 -- 172.21.15.118:0/278904405 <== mon.1 v2:172.21.15.193:3300/0 4 ==== mgrmap(e 20) v1 ==== 137291+0+0 (secure 0 0 0) 0x7fdd1c044070 con 0x7fdd2c074100 2024-03-24T03:17:21.738 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T03:17:21.735+0000 7fdd28c00640 1 --2- 172.21.15.118:0/278904405 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7fdd100a2880 0x7fdd100a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T03:17:21.738 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T03:17:21.735+0000 7fdd28c00640 1 -- 172.21.15.118:0/278904405 <== mon.1 v2:172.21.15.193:3300/0 5 ==== osd_map(124..124 src has 1..124) v4 ==== 11201+0+0 (secure 0 0 0) 0x7fdd1c0f1480 con 0x7fdd2c074100 2024-03-24T03:17:21.739 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T03:17:21.735+0000 7fdd2b400640 1 --2- 172.21.15.118:0/278904405 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7fdd100a2880 0x7fdd100a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T03:17:21.739 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T03:17:21.736+0000 7fdd2b400640 1 --2- 172.21.15.118:0/278904405 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7fdd100a2880 0x7fdd100a4d40 secure :-1 s=READY pgs=639 cs=0 l=1 rev1=1 crypto rx=0x7fdd2c19af70 tx=0x7fdd18005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T03:17:21.739 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T03:17:21.738+0000 7fdd28c00640 1 -- 172.21.15.118:0/278904405 <== mon.1 v2:172.21.15.193:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7fdd1c0bd2d0 con 0x7fdd2c074100 2024-03-24T03:17:21.937 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T03:17:21.938+0000 7fdd31c00640 1 -- 172.21.15.118:0/278904405 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7fdd2c0622a0 con 0x7fdd2c074100 2024-03-24T03:17:21.938 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T03:17:21.938+0000 7fdd28c00640 1 -- 172.21.15.118:0/278904405 <== mon.1 v2:172.21.15.193:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v124) v1 ==== 100+0+11 (secure 0 0 0) 0x7fdd1c0c1370 con 0x7fdd2c074100 2024-03-24T03:17:21.938 INFO:teuthology.orchestra.run.smithi118.stdout:pg_num: 64 2024-03-24T03:17:21.942 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T03:17:21.942+0000 7fdd31c00640 1 -- 172.21.15.118:0/278904405 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7fdd100a2880 msgr2=0x7fdd100a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T03:17:21.942 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T03:17:21.942+0000 7fdd31c00640 1 --2- 172.21.15.118:0/278904405 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7fdd100a2880 0x7fdd100a4d40 secure :-1 s=READY pgs=639 cs=0 l=1 rev1=1 crypto rx=0x7fdd2c19af70 tx=0x7fdd18005f90 comp rx=0 tx=0).stop 2024-03-24T03:17:21.942 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T03:17:21.942+0000 7fdd31c00640 1 -- 172.21.15.118:0/278904405 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fdd2c074100 msgr2=0x7fdd2c199a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T03:17:21.942 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T03:17:21.942+0000 7fdd31c00640 1 --2- 172.21.15.118:0/278904405 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fdd2c074100 0x7fdd2c199a60 secure :-1 s=READY pgs=477 cs=0 l=1 rev1=1 crypto rx=0x7fdd1c02c8f0 tx=0x7fdd1c02c660 comp rx=0 tx=0).stop 2024-03-24T03:17:21.943 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T03:17:21.942+0000 7fdd31c00640 1 -- 172.21.15.118:0/278904405 shutdown_connections 2024-03-24T03:17:21.943 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T03:17:21.942+0000 7fdd31c00640 1 --2- 172.21.15.118:0/278904405 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7fdd100a2880 0x7fdd100a4d40 unknown :-1 s=CLOSED pgs=639 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T03:17:21.943 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T03:17:21.942+0000 7fdd31c00640 1 --2- 172.21.15.118:0/278904405 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fdd2c109440 0x7fdd2c1a1010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T03:17:21.943 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T03:17:21.942+0000 7fdd31c00640 1 --2- 172.21.15.118:0/278904405 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fdd2c0756e0 0x7fdd2c199fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T03:17:21.943 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T03:17:21.942+0000 7fdd31c00640 1 --2- 172.21.15.118:0/278904405 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fdd2c074100 0x7fdd2c199a60 unknown :-1 s=CLOSED pgs=477 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T03:17:21.943 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T03:17:21.942+0000 7fdd31c00640 1 -- 172.21.15.118:0/278904405 >> 172.21.15.118:0/278904405 conn(0x7fdd2c0fcb10 msgr2=0x7fdd2c105340 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T03:17:21.943 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T03:17:21.942+0000 7fdd31c00640 1 -- 172.21.15.118:0/278904405 shutdown_connections 2024-03-24T03:17:21.943 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T03:17:21.942+0000 7fdd31c00640 1 -- 172.21.15.118:0/278904405 wait complete. 2024-03-24T03:17:21.960 DEBUG:teuthology.orchestra.run.smithi118:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr dump --format=json-pretty 2024-03-24T03:17:22.164 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T03:17:22.164+0000 7f929d800640 1 -- 172.21.15.118:0/3391957088 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9298104ad0 msgr2=0x7f9298104f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T03:17:22.164 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T03:17:22.164+0000 7f929d800640 1 --2- 172.21.15.118:0/3391957088 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9298104ad0 0x7f9298104f50 secure :-1 s=READY pgs=475 cs=0 l=1 rev1=1 crypto rx=0x7f9280009900 tx=0x7f928002bc30 comp rx=0 tx=0).stop 2024-03-24T03:17:22.165 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T03:17:22.165+0000 7f929d800640 1 -- 172.21.15.118:0/3391957088 shutdown_connections 2024-03-24T03:17:22.165 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T03:17:22.165+0000 7f929d800640 1 --2- 172.21.15.118:0/3391957088 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f9298105490 0x7f929810bd20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T03:17:22.165 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T03:17:22.165+0000 7f929d800640 1 --2- 172.21.15.118:0/3391957088 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9298104ad0 0x7f9298104f50 unknown :-1 s=CLOSED pgs=475 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T03:17:22.165 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T03:17:22.165+0000 7f929d800640 1 --2- 172.21.15.118:0/3391957088 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f92981038e0 0x7f9298103d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T03:17:22.165 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T03:17:22.165+0000 7f929d800640 1 -- 172.21.15.118:0/3391957088 >> 172.21.15.118:0/3391957088 conn(0x7f92980fee50 msgr2=0x7f9298101290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T03:17:22.165 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T03:17:22.165+0000 7f929d800640 1 -- 172.21.15.118:0/3391957088 shutdown_connections 2024-03-24T03:17:22.165 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T03:17:22.165+0000 7f929d800640 1 -- 172.21.15.118:0/3391957088 wait complete. 2024-03-24T03:17:22.167 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T03:17:22.166+0000 7f929d800640 1 Processor -- start 2024-03-24T03:17:22.168 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T03:17:22.166+0000 7f929d800640 1 -- start start 2024-03-24T03:17:22.168 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T03:17:22.166+0000 7f929d800640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f92981038e0 0x7f9298199c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T03:17:22.168 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T03:17:22.166+0000 7f929d800640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f9298104ad0 0x7f929819a190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T03:17:22.168 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T03:17:22.166+0000 7f929d800640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f9298105490 0x7f92981a11c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T03:17:22.168 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T03:17:22.166+0000 7f929d800640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f929810f590 con 0x7f9298104ad0 2024-03-24T03:17:22.168 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T03:17:22.166+0000 7f929d800640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f929810f410 con 0x7f92981038e0 2024-03-24T03:17:22.169 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T03:17:22.166+0000 7f929d800640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f929810f710 con 0x7f9298105490 2024-03-24T03:17:22.169 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T03:17:22.166+0000 7f9296a00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f92981038e0 0x7f9298199c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T03:17:22.169 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T03:17:22.166+0000 7f928e000640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f9298104ad0 0x7f929819a190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T03:17:22.169 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T03:17:22.166+0000 7f9296a00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f92981038e0 0x7f9298199c50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.118:33016/0 (socket says 172.21.15.118:33016) 2024-03-24T03:17:22.169 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T03:17:22.166+0000 7f9296a00640 1 -- 172.21.15.118:0/2706433962 learned_addr learned my addr 172.21.15.118:0/2706433962 (peer_addr_for_me v2:172.21.15.118:0/0) 2024-03-24T03:17:22.169 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T03:17:22.166+0000 7f9297400640 1 --2- 172.21.15.118:0/2706433962 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f9298105490 0x7f92981a11c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T03:17:22.169 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T03:17:22.167+0000 7f9296a00640 1 -- 172.21.15.118:0/2706433962 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f9298105490 msgr2=0x7f92981a11c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T03:17:22.169 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T03:17:22.167+0000 7f9296a00640 1 --2- 172.21.15.118:0/2706433962 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f9298105490 0x7f92981a11c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T03:17:22.170 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T03:17:22.167+0000 7f9296a00640 1 -- 172.21.15.118:0/2706433962 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f9298104ad0 msgr2=0x7f929819a190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T03:17:22.170 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T03:17:22.167+0000 7f9296a00640 1 --2- 172.21.15.118:0/2706433962 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f9298104ad0 0x7f929819a190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T03:17:22.170 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T03:17:22.167+0000 7f9296a00640 1 -- 172.21.15.118:0/2706433962 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f92981a1860 con 0x7f92981038e0 2024-03-24T03:17:22.170 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T03:17:22.167+0000 7f9296a00640 1 --2- 172.21.15.118:0/2706433962 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f92981038e0 0x7f9298199c50 secure :-1 s=READY pgs=476 cs=0 l=1 rev1=1 crypto rx=0x7f928800ebc0 tx=0x7f928800c6a0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T03:17:22.170 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T03:17:22.167+0000 7f928fe00640 1 -- 172.21.15.118:0/2706433962 <== mon.2 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9288003b70 con 0x7f92981038e0 2024-03-24T03:17:22.170 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T03:17:22.167+0000 7f928fe00640 1 -- 172.21.15.118:0/2706433962 <== mon.2 v2:172.21.15.189:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f9288003d10 con 0x7f92981038e0 2024-03-24T03:17:22.170 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T03:17:22.167+0000 7f928fe00640 1 -- 172.21.15.118:0/2706433962 <== mon.2 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f9288019980 con 0x7f92981038e0 2024-03-24T03:17:22.170 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T03:17:22.167+0000 7f929d800640 1 -- 172.21.15.118:0/2706433962 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f92981a1b50 con 0x7f92981038e0 2024-03-24T03:17:22.171 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T03:17:22.167+0000 7f929d800640 1 -- 172.21.15.118:0/2706433962 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f92981a2050 con 0x7f92981038e0 2024-03-24T03:17:22.171 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T03:17:22.168+0000 7f929d800640 1 -- 172.21.15.118:0/2706433962 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f929810ac80 con 0x7f92981038e0 2024-03-24T03:17:22.171 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T03:17:22.170+0000 7f928fe00640 1 -- 172.21.15.118:0/2706433962 <== mon.2 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 20) v1 ==== 137291+0+0 (secure 0 0 0) 0x7f9288019b20 con 0x7f92981038e0 2024-03-24T03:17:22.172 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T03:17:22.172+0000 7f928fe00640 1 --2- 172.21.15.118:0/2706433962 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7f927c0a2880 0x7f927c0a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T03:17:22.172 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T03:17:22.172+0000 7f928fe00640 1 -- 172.21.15.118:0/2706433962 <== mon.2 v2:172.21.15.189:3300/0 5 ==== osd_map(124..124 src has 1..124) v4 ==== 11201+0+0 (secure 0 0 0) 0x7f92880cfe20 con 0x7f92981038e0 2024-03-24T03:17:22.172 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T03:17:22.172+0000 7f928e000640 1 --2- 172.21.15.118:0/2706433962 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7f927c0a2880 0x7f927c0a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T03:17:22.172 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T03:17:22.173+0000 7f928e000640 1 --2- 172.21.15.118:0/2706433962 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7f927c0a2880 0x7f927c0a4d40 secure :-1 s=READY pgs=640 cs=0 l=1 rev1=1 crypto rx=0x7f928002c240 tx=0x7f928002df90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T03:17:22.175 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T03:17:22.176+0000 7f928fe00640 1 -- 172.21.15.118:0/2706433962 <== mon.2 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f928809c2c0 con 0x7f92981038e0 2024-03-24T03:17:22.426 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T03:17:22.427+0000 7f929d800640 1 -- 172.21.15.118:0/2706433962 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "mgr dump", "format": "json-pretty"} v 0) v1 -- 0x7f929802d070 con 0x7f92981038e0 2024-03-24T03:17:22.437 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T03:17:22.437+0000 7f928fe00640 1 -- 172.21.15.118:0/2706433962 <== mon.2 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{"prefix": "mgr dump", "format": "json-pretty"}]=0 v20) v1 ==== 81+0+739838 (secure 0 0 0) 0x7f928809bbc0 con 0x7f92981038e0 2024-03-24T03:17:22.448 INFO:teuthology.orchestra.run.smithi118.stdout: 2024-03-24T03:17:22.448 INFO:teuthology.orchestra.run.smithi118.stdout:{ 2024-03-24T03:17:22.448 INFO:teuthology.orchestra.run.smithi118.stdout: "epoch": 20, 2024-03-24T03:17:22.449 INFO:teuthology.orchestra.run.smithi118.stdout: "flags": 0, 2024-03-24T03:17:22.449 INFO:teuthology.orchestra.run.smithi118.stdout: "active_gid": 14152, 2024-03-24T03:17:22.449 INFO:teuthology.orchestra.run.smithi118.stdout: "active_name": "x", 2024-03-24T03:17:22.449 INFO:teuthology.orchestra.run.smithi118.stdout: "active_addrs": { 2024-03-24T03:17:22.449 INFO:teuthology.orchestra.run.smithi118.stdout: "addrvec": [ 2024-03-24T03:17:22.449 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-24T03:17:22.449 INFO:teuthology.orchestra.run.smithi118.stdout: "type": "v2", 2024-03-24T03:17:22.449 INFO:teuthology.orchestra.run.smithi118.stdout: "addr": "172.21.15.118:6800", 2024-03-24T03:17:22.449 INFO:teuthology.orchestra.run.smithi118.stdout: "nonce": 2112912943 2024-03-24T03:17:22.450 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-24T03:17:22.450 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-24T03:17:22.450 INFO:teuthology.orchestra.run.smithi118.stdout: "type": "v1", 2024-03-24T03:17:22.450 INFO:teuthology.orchestra.run.smithi118.stdout: "addr": "172.21.15.118:6801", 2024-03-24T03:17:22.450 INFO:teuthology.orchestra.run.smithi118.stdout: "nonce": 2112912943 2024-03-24T03:17:22.450 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-03-24T03:17:22.451 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-03-24T03:17:22.451 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-24T03:17:22.451 INFO:teuthology.orchestra.run.smithi118.stdout: "active_addr": "172.21.15.118:6801/2112912943", 2024-03-24T03:17:22.451 INFO:teuthology.orchestra.run.smithi118.stdout: "active_change": "2024-03-24T00:05:05.459020+0000", 2024-03-24T03:17:22.451 INFO:teuthology.orchestra.run.smithi118.stdout: "active_mgr_features": 4540701547738038271, 2024-03-24T03:17:22.451 INFO:teuthology.orchestra.run.smithi118.stdout: "available": true, 2024-03-24T03:17:22.451 INFO:teuthology.orchestra.run.smithi118.stdout: "standbys": [ 2024-03-24T03:17:22.451 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-24T03:17:22.451 INFO:teuthology.orchestra.run.smithi118.stdout: "gid": 24112, 2024-03-24T03:17:22.452 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "z", 2024-03-24T03:17:22.452 INFO:teuthology.orchestra.run.smithi118.stdout: "mgr_features": 4540701547738038271, 2024-03-24T03:17:22.452 INFO:teuthology.orchestra.run.smithi118.stdout: "available_modules": [ 2024-03-24T03:17:22.452 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-24T03:17:22.452 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "alerts", 2024-03-24T03:17:22.452 INFO:teuthology.orchestra.run.smithi118.stdout: "can_run": true, 2024-03-24T03:17:22.452 INFO:teuthology.orchestra.run.smithi118.stdout: "error_string": "", 2024-03-24T03:17:22.453 INFO:teuthology.orchestra.run.smithi118.stdout: "module_options": { 2024-03-24T03:17:22.453 INFO:teuthology.orchestra.run.smithi118.stdout: "interval": { 2024-03-24T03:17:22.453 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "interval", 2024-03-24T03:17:22.453 INFO:teuthology.orchestra.run.smithi118.stdout: "type": "secs", 2024-03-24T03:17:22.453 INFO:teuthology.orchestra.run.smithi118.stdout: "level": "advanced", 2024-03-24T03:17:22.453 INFO:teuthology.orchestra.run.smithi118.stdout: "flags": 1, 2024-03-24T03:17:22.453 INFO:teuthology.orchestra.run.smithi118.stdout: "default_value": "60", 2024-03-24T03:17:22.453 INFO:teuthology.orchestra.run.smithi118.stdout: "min": "", 2024-03-24T03:17:22.453 INFO:teuthology.orchestra.run.smithi118.stdout: "max": "", 2024-03-24T03:17:22.454 INFO:teuthology.orchestra.run.smithi118.stdout: "enum_allowed": [], 2024-03-24T03:17:22.454 INFO:teuthology.orchestra.run.smithi118.stdout: "desc": "How frequently to reexamine health status", 2024-03-24T03:17:22.454 INFO:teuthology.orchestra.run.smithi118.stdout: "long_desc": "", 2024-03-24T03:17:22.454 INFO:teuthology.orchestra.run.smithi118.stdout: "tags": [], 2024-03-24T03:17:22.454 INFO:teuthology.orchestra.run.smithi118.stdout: "see_also": [] 2024-03-24T03:17:22.454 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-24T03:17:22.454 INFO:teuthology.orchestra.run.smithi118.stdout: "log_level": { 2024-03-24T03:17:22.454 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "log_level", 2024-03-24T03:17:22.455 INFO:teuthology.orchestra.run.smithi118.stdout: "type": "str", 2024-03-24T03:17:22.455 INFO:teuthology.orchestra.run.smithi118.stdout: "level": "advanced", 2024-03-24T03:17:22.455 INFO:teuthology.orchestra.run.smithi118.stdout: "flags": 1, 2024-03-24T03:17:22.455 INFO:teuthology.orchestra.run.smithi118.stdout: "default_value": "", 2024-03-24T03:17:22.455 INFO:teuthology.orchestra.run.smithi118.stdout: "min": "", 2024-03-24T03:17:22.455 INFO:teuthology.orchestra.run.smithi118.stdout: "max": "", 2024-03-24T03:17:22.455 INFO:teuthology.orchestra.run.smithi118.stdout: "enum_allowed": [ 2024-03-24T03:17:22.456 INFO:teuthology.orchestra.run.smithi118.stdout: "", 2024-03-24T03:17:22.456 INFO:teuthology.orchestra.run.smithi118.stdout: "critical", 2024-03-24T03:17:22.456 INFO:teuthology.orchestra.run.smithi118.stdout: "debug", 2024-03-24T03:17:22.456 INFO:teuthology.orchestra.run.smithi118.stdout: "error", 2024-03-24T03:17:22.456 INFO:teuthology.orchestra.run.smithi118.stdout: "info", 2024-03-24T03:17:22.456 INFO:teuthology.orchestra.run.smithi118.stdout: "warning" 2024-03-24T03:17:22.456 INFO:teuthology.orchestra.run.smithi118.stdout: ], 2024-03-24T03:17:22.456 INFO:teuthology.orchestra.run.smithi118.stdout: "desc": "", 2024-03-24T03:17:22.457 INFO:teuthology.orchestra.run.smithi118.stdout: "long_desc": "", 2024-03-24T03:17:22.457 INFO:teuthology.orchestra.run.smithi118.stdout: "tags": [], 2024-03-24T03:17:22.457 INFO:teuthology.orchestra.run.smithi118.stdout: "see_also": [] 2024-03-24T03:17:22.457 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-24T03:17:22.457 INFO:teuthology.orchestra.run.smithi118.stdout: "log_to_cluster": { 2024-03-24T03:17:22.457 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "log_to_cluster", 2024-03-24T03:17:22.457 INFO:teuthology.orchestra.run.smithi118.stdout: "type": "bool", 2024-03-24T03:17:22.458 INFO:teuthology.orchestra.run.smithi118.stdout: "level": "advanced", 2024-03-24T03:17:22.458 INFO:teuthology.orchestra.run.smithi118.stdout: "flags": 1, 2024-03-24T03:17:22.458 INFO:teuthology.orchestra.run.smithi118.stdout: "default_value": "False", 2024-03-24T03:17:22.458 INFO:teuthology.orchestra.run.smithi118.stdout: "min": "", 2024-03-24T03:17:22.458 INFO:teuthology.orchestra.run.smithi118.stdout: "max": "", 2024-03-24T03:17:22.458 INFO:teuthology.orchestra.run.smithi118.stdout: "enum_allowed": [], 2024-03-24T03:17:22.458 INFO:teuthology.orchestra.run.smithi118.stdout: "desc": "", 2024-03-24T03:17:22.459 INFO:teuthology.orchestra.run.smithi118.stdout: "long_desc": "", 2024-03-24T03:17:22.459 INFO:teuthology.orchestra.run.smithi118.stdout: "tags": [], 2024-03-24T03:17:22.459 INFO:teuthology.orchestra.run.smithi118.stdout: "see_also": [] 2024-03-24T03:17:22.459 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-24T03:17:22.459 INFO:teuthology.orchestra.run.smithi118.stdout: "log_to_cluster_level": { 2024-03-24T03:17:22.459 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "log_to_cluster_level", 2024-03-24T03:17:22.459 INFO:teuthology.orchestra.run.smithi118.stdout: "type": "str", 2024-03-24T03:17:22.459 INFO:teuthology.orchestra.run.smithi118.stdout: "level": "advanced", 2024-03-24T03:17:22.459 INFO:teuthology.orchestra.run.smithi118.stdout: "flags": 1, 2024-03-24T03:17:22.460 INFO:teuthology.orchestra.run.smithi118.stdout: "default_value": "info", 2024-03-24T03:17:22.460 INFO:teuthology.orchestra.run.smithi118.stdout: "min": "", 2024-03-24T03:17:22.460 INFO:teuthology.orchestra.run.smithi118.stdout: "max": "", 2024-03-24T03:17:22.460 INFO:teuthology.orchestra.run.smithi118.stdout: "enum_allowed": [ 2024-03-24T03:17:22.460 INFO:teuthology.orchestra.run.smithi118.stdout: "", 2024-03-24T03:17:22.460 INFO:teuthology.orchestra.run.smithi118.stdout: "critical", 2024-03-24T03:17:22.460 INFO:teuthology.orchestra.run.smithi118.stdout: "debug", 2024-03-24T03:17:22.460 INFO:teuthology.orchestra.run.smithi118.stdout: "error", 2024-03-24T03:17:22.460 INFO:teuthology.orchestra.run.smithi118.stdout: "info", 2024-03-24T03:17:22.461 INFO:teuthology.orchestra.run.smithi118.stdout: "warning" 2024-03-24T03:17:22.461 INFO:teuthology.orchestra.run.smithi118.stdout: ], 2024-03-24T03:17:22.461 INFO:teuthology.orchestra.run.smithi118.stdout: "desc": "", 2024-03-24T03:17:22.461 INFO:teuthology.orchestra.run.smithi118.stdout: "long_desc": "", 2024-03-24T03:17:22.461 INFO:teuthology.orchestra.run.smithi118.stdout: "tags": [], 2024-03-24T03:17:22.461 INFO:teuthology.orchestra.run.smithi118.stdout: "see_also": [] 2024-03-24T03:17:22.461 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-24T03:17:22.461 INFO:teuthology.orchestra.run.smithi118.stdout: "log_to_file": { 2024-03-24T03:17:22.462 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "log_to_file", 2024-03-24T03:17:22.462 INFO:teuthology.orchestra.run.smithi118.stdout: "type": "bool", 2024-03-24T03:17:22.462 INFO:teuthology.orchestra.run.smithi118.stdout: "level": "advanced", 2024-03-24T03:17:22.462 INFO:teuthology.orchestra.run.smithi118.stdout: "flags": 1, 2024-03-24T03:17:22.462 INFO:teuthology.orchestra.run.smithi118.stdout: "default_value": "False", 2024-03-24T03:17:22.462 INFO:teuthology.orchestra.run.smithi118.stdout: "min": "", 2024-03-24T03:17:22.462 INFO:teuthology.orchestra.run.smithi118.stdout: "max": "", 2024-03-24T03:17:22.462 INFO:teuthology.orchestra.run.smithi118.stdout: "enum_allowed": [], 2024-03-24T03:17:22.463 INFO:teuthology.orchestra.run.smithi118.stdout: "desc": "", 2024-03-24T03:17:22.463 INFO:teuthology.orchestra.run.smithi118.stdout: "long_desc": "", 2024-03-24T03:17:22.463 INFO:teuthology.orchestra.run.smithi118.stdout: "tags": [], 2024-03-24T03:17:22.463 INFO:teuthology.orchestra.run.smithi118.stdout: "see_also": [] 2024-03-24T03:17:22.463 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-24T03:17:22.463 INFO:teuthology.orchestra.run.smithi118.stdout: "smtp_destination": { 2024-03-24T03:17:22.463 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "smtp_destination", 2024-03-24T03:17:22.463 INFO:teuthology.orchestra.run.smithi118.stdout: "type": "str", 2024-03-24T03:17:22.463 INFO:teuthology.orchestra.run.smithi118.stdout: "level": "advanced", 2024-03-24T03:17:22.463 INFO:teuthology.orchestra.run.smithi118.stdout: "flags": 1, 2024-03-24T03:17:22.464 INFO:teuthology.orchestra.run.smithi118.stdout: "default_value": "", 2024-03-24T03:17:22.464 INFO:teuthology.orchestra.run.smithi118.stdout: "min": "", 2024-03-24T03:17:22.464 INFO:teuthology.orchestra.run.smithi118.stdout: "max": "", 2024-03-24T03:17:22.464 INFO:teuthology.orchestra.run.smithi118.stdout: "enum_allowed": [], 2024-03-24T03:17:22.464 INFO:teuthology.orchestra.run.smithi118.stdout: "desc": "Email address to send alerts to", 2024-03-24T03:17:22.464 INFO:teuthology.orchestra.run.smithi118.stdout: "long_desc": "", 2024-03-24T03:17:22.464 INFO:teuthology.orchestra.run.smithi118.stdout: "tags": [], 2024-03-24T03:17:22.464 INFO:teuthology.orchestra.run.smithi118.stdout: "see_also": [] 2024-03-24T03:17:22.465 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-24T03:17:22.465 INFO:teuthology.orchestra.run.smithi118.stdout: "smtp_from_name": { 2024-03-24T03:17:22.465 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "smtp_from_name", 2024-03-24T03:17:22.465 INFO:teuthology.orchestra.run.smithi118.stdout: "type": "str", 2024-03-24T03:17:22.465 INFO:teuthology.orchestra.run.smithi118.stdout: "level": "advanced", 2024-03-24T03:17:22.465 INFO:teuthology.orchestra.run.smithi118.stdout: "flags": 1, 2024-03-24T03:17:22.465 INFO:teuthology.orchestra.run.smithi118.stdout: "default_value": "Ceph", 2024-03-24T03:17:22.465 INFO:teuthology.orchestra.run.smithi118.stdout: "min": "", 2024-03-24T03:17:22.465 INFO:teuthology.orchestra.run.smithi118.stdout: "max": "", 2024-03-24T03:17:22.466 INFO:teuthology.orchestra.run.smithi118.stdout: "enum_allowed": [], 2024-03-24T03:17:22.466 INFO:teuthology.orchestra.run.smithi118.stdout: "desc": "Email From: name", 2024-03-24T03:17:22.466 INFO:teuthology.orchestra.run.smithi118.stdout: "long_desc": "", 2024-03-24T03:17:22.466 INFO:teuthology.orchestra.run.smithi118.stdout: "tags": [], 2024-03-24T03:17:22.466 INFO:teuthology.orchestra.run.smithi118.stdout: "see_also": [] 2024-03-24T03:17:22.466 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-24T03:17:22.467 INFO:teuthology.orchestra.run.smithi118.stdout: "smtp_host": { 2024-03-24T03:17:22.467 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "smtp_host", 2024-03-24T03:17:22.467 INFO:teuthology.orchestra.run.smithi118.stdout: "type": "str", 2024-03-24T03:17:22.467 INFO:teuthology.orchestra.run.smithi118.stdout: "level": "advanced", 2024-03-24T03:17:22.467 INFO:teuthology.orchestra.run.smithi118.stdout: "flags": 1, 2024-03-24T03:17:22.467 INFO:teuthology.orchestra.run.smithi118.stdout: "default_value": "", 2024-03-24T03:17:22.467 INFO:teuthology.orchestra.run.smithi118.stdout: "min": "", 2024-03-24T03:17:22.467 INFO:teuthology.orchestra.run.smithi118.stdout: "max": "", 2024-03-24T03:17:22.467 INFO:teuthology.orchestra.run.smithi118.stdout: "enum_allowed": [], 2024-03-24T03:17:22.468 INFO:teuthology.orchestra.run.smithi118.stdout: "desc": "SMTP server", 2024-03-24T03:17:22.468 INFO:teuthology.orchestra.run.smithi118.stdout: "long_desc": "", 2024-03-24T03:17:22.468 INFO:teuthology.orchestra.run.smithi118.stdout: "tags": [], 2024-03-24T03:17:22.468 INFO:teuthology.orchestra.run.smithi118.stdout: "see_also": [] 2024-03-24T03:17:22.468 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-24T03:17:22.468 INFO:teuthology.orchestra.run.smithi118.stdout: "smtp_password": { 2024-03-24T03:17:22.468 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "smtp_password", 2024-03-24T03:17:22.468 INFO:teuthology.orchestra.run.smithi118.stdout: "type": "str", 2024-03-24T03:17:22.468 INFO:teuthology.orchestra.run.smithi118.stdout: "level": "advanced", 2024-03-24T03:17:22.469 INFO:teuthology.orchestra.run.smithi118.stdout: "flags": 1, 2024-03-24T03:17:22.469 INFO:teuthology.orchestra.run.smithi118.stdout: "default_value": "", 2024-03-24T03:17:22.469 INFO:teuthology.orchestra.run.smithi118.stdout: "min": "", 2024-03-24T03:17:22.469 INFO:teuthology.orchestra.run.smithi118.stdout: "max": "", 2024-03-24T03:17:22.469 INFO:teuthology.orchestra.run.smithi118.stdout: "enum_allowed": [], 2024-03-24T03:17:22.469 INFO:teuthology.orchestra.run.smithi118.stdout: "desc": "Password to authenticate with", 2024-03-24T03:17:22.469 INFO:teuthology.orchestra.run.smithi118.stdout: "long_desc": "", 2024-03-24T03:17:22.469 INFO:teuthology.orchestra.run.smithi118.stdout: "tags": [], 2024-03-24T03:17:22.469 INFO:teuthology.orchestra.run.smithi118.stdout: "see_also": [] 2024-03-24T03:17:22.470 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-24T03:17:22.470 INFO:teuthology.orchestra.run.smithi118.stdout: "smtp_port": { 2024-03-24T03:17:22.470 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "smtp_port", 2024-03-24T03:17:22.470 INFO:teuthology.orchestra.run.smithi118.stdout: "type": "int", 2024-03-24T03:17:22.470 INFO:teuthology.orchestra.run.smithi118.stdout: "level": "advanced", 2024-03-24T03:17:22.470 INFO:teuthology.orchestra.run.smithi118.stdout: "flags": 1, 2024-03-24T03:17:22.470 INFO:teuthology.orchestra.run.smithi118.stdout: "default_value": "465", 2024-03-24T03:17:22.471 INFO:teuthology.orchestra.run.smithi118.stdout: "min": "", 2024-03-24T03:17:22.471 INFO:teuthology.orchestra.run.smithi118.stdout: "max": "", 2024-03-24T03:17:22.471 INFO:teuthology.orchestra.run.smithi118.stdout: "enum_allowed": [], 2024-03-24T03:17:22.471 INFO:teuthology.orchestra.run.smithi118.stdout: "desc": "SMTP port", 2024-03-24T03:17:22.471 INFO:teuthology.orchestra.run.smithi118.stdout: "long_desc": "", 2024-03-24T03:17:22.471 INFO:teuthology.orchestra.run.smithi118.stdout: "tags": [], 2024-03-24T03:17:22.471 INFO:teuthology.orchestra.run.smithi118.stdout: "see_also": [] 2024-03-24T03:17:22.471 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-24T03:17:22.471 INFO:teuthology.orchestra.run.smithi118.stdout: "smtp_sender": { 2024-03-24T03:17:22.471 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "smtp_sender", 2024-03-24T03:17:22.472 INFO:teuthology.orchestra.run.smithi118.stdout: "type": "str", 2024-03-24T03:17:22.472 INFO:teuthology.orchestra.run.smithi118.stdout: "level": "advanced", 2024-03-24T03:17:22.472 INFO:teuthology.orchestra.run.smithi118.stdout: "flags": 1, 2024-03-24T03:17:22.472 INFO:teuthology.orchestra.run.smithi118.stdout: "default_value": "", 2024-03-24T03:17:22.472 INFO:teuthology.orchestra.run.smithi118.stdout: "min": "", 2024-03-24T03:17:22.472 INFO:teuthology.orchestra.run.smithi118.stdout: "max": "", 2024-03-24T03:17:22.472 INFO:teuthology.orchestra.run.smithi118.stdout: "enum_allowed": [], 2024-03-24T03:17:22.472 INFO:teuthology.orchestra.run.smithi118.stdout: "desc": "SMTP envelope sender", 2024-03-24T03:17:22.473 INFO:teuthology.orchestra.run.smithi118.stdout: "long_desc": "", 2024-03-24T03:17:22.473 INFO:teuthology.orchestra.run.smithi118.stdout: "tags": [], 2024-03-24T03:17:22.473 INFO:teuthology.orchestra.run.smithi118.stdout: "see_also": [] 2024-03-24T03:17:22.473 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-24T03:17:22.473 INFO:teuthology.orchestra.run.smithi118.stdout: "smtp_ssl": { 2024-03-24T03:17:22.473 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "smtp_ssl", 2024-03-24T03:17:22.473 INFO:teuthology.orchestra.run.smithi118.stdout: "type": "bool", 2024-03-24T03:17:22.473 INFO:teuthology.orchestra.run.smithi118.stdout: "level": "advanced", 2024-03-24T03:17:22.473 INFO:teuthology.orchestra.run.smithi118.stdout: "flags": 1, 2024-03-24T03:17:22.474 INFO:teuthology.orchestra.run.smithi118.stdout: "default_value": "True", 2024-03-24T03:17:22.474 INFO:teuthology.orchestra.run.smithi118.stdout: "min": "", 2024-03-24T03:17:22.474 INFO:teuthology.orchestra.run.smithi118.stdout: "max": "", 2024-03-24T03:17:22.474 INFO:teuthology.orchestra.run.smithi118.stdout: "enum_allowed": [], 2024-03-24T03:17:22.474 INFO:teuthology.orchestra.run.smithi118.stdout: "desc": "Use SSL to connect to SMTP server", 2024-03-24T03:17:22.474 INFO:teuthology.orchestra.run.smithi118.stdout: "long_desc": "", 2024-03-24T03:17:22.474 INFO:teuthology.orchestra.run.smithi118.stdout: "tags": [], 2024-03-24T03:17:22.474 INFO:teuthology.orchestra.run.smithi118.stdout: "see_also": [] 2024-03-24T03:17:22.475 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-24T03:17:22.475 INFO:teuthology.orchestra.run.smithi118.stdout: "smtp_user": { 2024-03-24T03:17:22.475 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "smtp_user", 2024-03-24T03:17:22.475 INFO:teuthology.orchestra.run.smithi118.stdout: "type": "str", 2024-03-24T03:17:22.476 INFO:teuthology.orchestra.run.smithi118.stdout: "level": "advanced", 2024-03-24T03:17:22.476 INFO:teuthology.orchestra.run.smithi118.stdout: "flags": 1, 2024-03-24T03:17:22.476 INFO:teuthology.orchestra.run.smithi118.stdout: "default_value": "", 2024-03-24T03:17:22.476 INFO:teuthology.orchestra.run.smithi118.stdout: "min": "", 2024-03-24T03:17:22.476 INFO:teuthology.orchestra.run.smithi118.stdout: "max": "", 2024-03-24T03:17:22.477 INFO:teuthology.orchestra.run.smithi118.stdout: "enum_allowed": [], 2024-03-24T03:17:22.477 INFO:teuthology.orchestra.run.smithi118.stdout: "desc": "User to authenticate as", 2024-03-24T03:17:22.477 INFO:teuthology.orchestra.run.smithi118.stdout: "long_desc": "", 2024-03-24T03:17:22.477 INFO:teuthology.orchestra.run.smithi118.stdout: "tags": [], 2024-03-24T03:17:22.477 INFO:teuthology.orchestra.run.smithi118.stdout: "see_also": [] 2024-03-24T03:17:22.477 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-03-24T03:17:22.477 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-03-24T03:17:22.477 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-24T03:17:22.477 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-24T03:17:22.477 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "balancer", 2024-03-24T03:17:22.478 INFO:teuthology.orchestra.run.smithi118.stdout: "can_run": true, 2024-03-24T03:17:22.478 INFO:teuthology.orchestra.run.smithi118.stdout: "error_string": "", 2024-03-24T03:17:22.478 INFO:teuthology.orchestra.run.smithi118.stdout: "module_options": { 2024-03-24T03:17:22.478 INFO:teuthology.orchestra.run.smithi118.stdout: "active": { 2024-03-24T03:17:22.478 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "active", 2024-03-24T03:17:22.478 INFO:teuthology.orchestra.run.smithi118.stdout: "type": "bool", 2024-03-24T03:17:22.478 INFO:teuthology.orchestra.run.smithi118.stdout: "level": "advanced", 2024-03-24T03:17:22.478 INFO:teuthology.orchestra.run.smithi118.stdout: "flags": 1, 2024-03-24T03:17:22.479 INFO:teuthology.orchestra.run.smithi118.stdout: "default_value": "True", 2024-03-24T03:17:22.479 INFO:teuthology.orchestra.run.smithi118.stdout: "min": "", 2024-03-24T03:17:22.479 INFO:teuthology.orchestra.run.smithi118.stdout: "max": "", 2024-03-24T03:17:22.479 INFO:teuthology.orchestra.run.smithi118.stdout: "enum_allowed": [], 2024-03-24T03:17:22.479 INFO:teuthology.orchestra.run.smithi118.stdout: "desc": "automatically balance PGs across cluster", 2024-03-24T03:17:22.479 INFO:teuthology.orchestra.run.smithi118.stdout: "long_desc": "", 2024-03-24T03:17:22.479 INFO:teuthology.orchestra.run.smithi118.stdout: "tags": [], 2024-03-24T03:17:22.479 INFO:teuthology.orchestra.run.smithi118.stdout: "see_also": [] 2024-03-24T03:17:22.479 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-24T03:17:22.480 INFO:teuthology.orchestra.run.smithi118.stdout: "begin_time": { 2024-03-24T03:17:22.480 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "begin_time", 2024-03-24T03:17:22.480 INFO:teuthology.orchestra.run.smithi118.stdout: "type": "str", 2024-03-24T03:17:22.480 INFO:teuthology.orchestra.run.smithi118.stdout: "level": "advanced", 2024-03-24T03:17:22.480 INFO:teuthology.orchestra.run.smithi118.stdout: "flags": 1, 2024-03-24T03:17:22.480 INFO:teuthology.orchestra.run.smithi118.stdout: "default_value": "0000", 2024-03-24T03:17:22.480 INFO:teuthology.orchestra.run.smithi118.stdout: "min": "", 2024-03-24T03:17:22.480 INFO:teuthology.orchestra.run.smithi118.stdout: "max": "", 2024-03-24T03:17:22.480 INFO:teuthology.orchestra.run.smithi118.stdout: "enum_allowed": [], 2024-03-24T03:17:22.480 INFO:teuthology.orchestra.run.smithi118.stdout: "desc": "beginning time of day to automatically balance", 2024-03-24T03:17:22.481 INFO:teuthology.orchestra.run.smithi118.stdout: "long_desc": "This is a time of day in the format HHMM.", 2024-03-24T03:17:22.481 INFO:teuthology.orchestra.run.smithi118.stdout: "tags": [], 2024-03-24T03:17:22.481 INFO:teuthology.orchestra.run.smithi118.stdout: "see_also": [] 2024-03-24T03:17:22.481 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-24T03:17:22.481 INFO:teuthology.orchestra.run.smithi118.stdout: "begin_weekday": { 2024-03-24T03:17:22.481 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "begin_weekday", 2024-03-24T03:17:22.481 INFO:teuthology.orchestra.run.smithi118.stdout: "type": "uint", 2024-03-24T03:17:22.481 INFO:teuthology.orchestra.run.smithi118.stdout: "level": "advanced", 2024-03-24T03:17:22.481 INFO:teuthology.orchestra.run.smithi118.stdout: "flags": 1, 2024-03-24T03:17:22.482 INFO:teuthology.orchestra.run.smithi118.stdout: "default_value": "0", 2024-03-24T03:17:22.482 INFO:teuthology.orchestra.run.smithi118.stdout: "min": "0", 2024-03-24T03:17:22.482 INFO:teuthology.orchestra.run.smithi118.stdout: "max": "6", 2024-03-24T03:17:22.482 INFO:teuthology.orchestra.run.smithi118.stdout: "enum_allowed": [], 2024-03-24T03:17:22.482 INFO:teuthology.orchestra.run.smithi118.stdout: "desc": "Restrict automatic balancing to this day of the week or later", 2024-03-24T03:17:22.482 INFO:teuthology.orchestra.run.smithi118.stdout: "long_desc": "0 = Sunday, 1 = Monday, etc.", 2024-03-24T03:17:22.482 INFO:teuthology.orchestra.run.smithi118.stdout: "tags": [], 2024-03-24T03:17:22.482 INFO:teuthology.orchestra.run.smithi118.stdout: "see_also": [] 2024-03-24T03:17:22.483 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-24T03:17:22.483 INFO:teuthology.orchestra.run.smithi118.stdout: "crush_compat_max_iterations": { 2024-03-24T03:17:22.483 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "crush_compat_max_iterations", 2024-03-24T03:17:22.483 INFO:teuthology.orchestra.run.smithi118.stdout: "type": "uint", 2024-03-24T03:17:22.483 INFO:teuthology.orchestra.run.smithi118.stdout: "level": "advanced", 2024-03-24T03:17:22.483 INFO:teuthology.orchestra.run.smithi118.stdout: "flags": 1, 2024-03-24T03:17:22.483 INFO:teuthology.orchestra.run.smithi118.stdout: "default_value": "25", 2024-03-24T03:17:22.483 INFO:teuthology.orchestra.run.smithi118.stdout: "min": "1", 2024-03-24T03:17:22.483 INFO:teuthology.orchestra.run.smithi118.stdout: "max": "250", 2024-03-24T03:17:22.484 INFO:teuthology.orchestra.run.smithi118.stdout: "enum_allowed": [], 2024-03-24T03:17:22.484 INFO:teuthology.orchestra.run.smithi118.stdout: "desc": "maximum number of iterations to attempt optimization", 2024-03-24T03:17:22.484 INFO:teuthology.orchestra.run.smithi118.stdout: "long_desc": "", 2024-03-24T03:17:22.484 INFO:teuthology.orchestra.run.smithi118.stdout: "tags": [], 2024-03-24T03:17:22.484 INFO:teuthology.orchestra.run.smithi118.stdout: "see_also": [] 2024-03-24T03:17:22.484 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-24T03:17:22.484 INFO:teuthology.orchestra.run.smithi118.stdout: "crush_compat_metrics": { 2024-03-24T03:17:22.484 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "crush_compat_metrics", 2024-03-24T03:17:22.485 INFO:teuthology.orchestra.run.smithi118.stdout: "type": "str", 2024-03-24T03:17:22.485 INFO:teuthology.orchestra.run.smithi118.stdout: "level": "advanced", 2024-03-24T03:17:22.485 INFO:teuthology.orchestra.run.smithi118.stdout: "flags": 1, 2024-03-24T03:17:22.485 INFO:teuthology.orchestra.run.smithi118.stdout: "default_value": "pgs,objects,bytes", 2024-03-24T03:17:22.485 INFO:teuthology.orchestra.run.smithi118.stdout: "min": "", 2024-03-24T03:17:22.485 INFO:teuthology.orchestra.run.smithi118.stdout: "max": "", 2024-03-24T03:17:22.485 INFO:teuthology.orchestra.run.smithi118.stdout: "enum_allowed": [], 2024-03-24T03:17:22.485 INFO:teuthology.orchestra.run.smithi118.stdout: "desc": "metrics with which to calculate OSD utilization", 2024-03-24T03:17:22.485 INFO:teuthology.orchestra.run.smithi118.stdout: "long_desc": "Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.", 2024-03-24T03:17:22.486 INFO:teuthology.orchestra.run.smithi118.stdout: "tags": [], 2024-03-24T03:17:22.486 INFO:teuthology.orchestra.run.smithi118.stdout: "see_also": [] 2024-03-24T03:17:22.486 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-24T03:17:22.486 INFO:teuthology.orchestra.run.smithi118.stdout: "crush_compat_step": { 2024-03-24T03:17:22.486 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "crush_compat_step", 2024-03-24T03:17:22.486 INFO:teuthology.orchestra.run.smithi118.stdout: "type": "float", 2024-03-24T03:17:22.486 INFO:teuthology.orchestra.run.smithi118.stdout: "level": "advanced", 2024-03-24T03:17:22.487 INFO:teuthology.orchestra.run.smithi118.stdout: "flags": 1, 2024-03-24T03:17:22.487 INFO:teuthology.orchestra.run.smithi118.stdout: "default_value": "0.5", 2024-03-24T03:17:22.487 INFO:teuthology.orchestra.run.smithi118.stdout: "min": "0.001", 2024-03-24T03:17:22.487 INFO:teuthology.orchestra.run.smithi118.stdout: "max": "0.999", 2024-03-24T03:17:22.487 INFO:teuthology.orchestra.run.smithi118.stdout: "enum_allowed": [], 2024-03-24T03:17:22.487 INFO:teuthology.orchestra.run.smithi118.stdout: "desc": "aggressiveness of optimization", 2024-03-24T03:17:22.487 INFO:teuthology.orchestra.run.smithi118.stdout: "long_desc": ".99 is very aggressive, .01 is less aggressive", 2024-03-24T03:17:22.487 INFO:teuthology.orchestra.run.smithi118.stdout: "tags": [], 2024-03-24T03:17:22.488 INFO:teuthology.orchestra.run.smithi118.stdout: "see_also": [] 2024-03-24T03:17:22.488 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-24T03:17:22.488 INFO:teuthology.orchestra.run.smithi118.stdout: "end_time": { 2024-03-24T03:17:22.488 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "end_time", 2024-03-24T03:17:22.488 INFO:teuthology.orchestra.run.smithi118.stdout: "type": "str", 2024-03-24T03:17:22.488 INFO:teuthology.orchestra.run.smithi118.stdout: "level": "advanced", 2024-03-24T03:17:22.488 INFO:teuthology.orchestra.run.smithi118.stdout: "flags": 1, 2024-03-24T03:17:22.488 INFO:teuthology.orchestra.run.smithi118.stdout: "default_value": "2359", 2024-03-24T03:17:22.488 INFO:teuthology.orchestra.run.smithi118.stdout: "min": "", 2024-03-24T03:17:22.489 INFO:teuthology.orchestra.run.smithi118.stdout: "max": "", 2024-03-24T03:17:22.489 INFO:teuthology.orchestra.run.smithi118.stdout: "enum_allowed": [], 2024-03-24T03:17:22.489 INFO:teuthology.orchestra.run.smithi118.stdout: "desc": "ending time of day to automatically balance", 2024-03-24T03:17:22.489 INFO:teuthology.orchestra.run.smithi118.stdout: "long_desc": "This is a time of day in the format HHMM.", 2024-03-24T03:17:22.489 INFO:teuthology.orchestra.run.smithi118.stdout: "tags": [], 2024-03-24T03:17:22.489 INFO:teuthology.orchestra.run.smithi118.stdout: "see_also": [] 2024-03-24T03:17:22.489 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-24T03:17:22.489 INFO:teuthology.orchestra.run.smithi118.stdout: "end_weekday": { 2024-03-24T03:17:22.489 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "end_weekday", 2024-03-24T03:17:22.489 INFO:teuthology.orchestra.run.smithi118.stdout: "type": "uint", 2024-03-24T03:17:22.490 INFO:teuthology.orchestra.run.smithi118.stdout: "level": "advanced", 2024-03-24T03:17:22.490 INFO:teuthology.orchestra.run.smithi118.stdout: "flags": 1, 2024-03-24T03:17:22.490 INFO:teuthology.orchestra.run.smithi118.stdout: "default_value": "0", 2024-03-24T03:17:22.490 INFO:teuthology.orchestra.run.smithi118.stdout: "min": "0", 2024-03-24T03:17:22.490 INFO:teuthology.orchestra.run.smithi118.stdout: "max": "6", 2024-03-24T03:17:22.490 INFO:teuthology.orchestra.run.smithi118.stdout: "enum_allowed": [], 2024-03-24T03:17:22.490 INFO:teuthology.orchestra.run.smithi118.stdout: "desc": "Restrict automatic balancing to days of the week earlier than this", 2024-03-24T03:17:22.490 INFO:teuthology.orchestra.run.smithi118.stdout: "long_desc": "0 = Sunday, 1 = Monday, etc.", 2024-03-24T03:17:22.491 INFO:teuthology.orchestra.run.smithi118.stdout: "tags": [], 2024-03-24T03:17:22.491 INFO:teuthology.orchestra.run.smithi118.stdout: "see_also": [] 2024-03-24T03:17:22.491 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-24T03:17:22.491 INFO:teuthology.orchestra.run.smithi118.stdout: "log_level": { 2024-03-24T03:17:22.491 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "log_level", 2024-03-24T03:17:22.491 INFO:teuthology.orchestra.run.smithi118.stdout: "type": "str", 2024-03-24T03:17:22.491 INFO:teuthology.orchestra.run.smithi118.stdout: "level": "advanced", 2024-03-24T03:17:22.491 INFO:teuthology.orchestra.run.smithi118.stdout: "flags": 1, 2024-03-24T03:17:22.491 INFO:teuthology.orchestra.run.smithi118.stdout: "default_value": "", 2024-03-24T03:17:22.492 INFO:teuthology.orchestra.run.smithi118.stdout: "min": "", 2024-03-24T03:17:22.492 INFO:teuthology.orchestra.run.smithi118.stdout: "max": "", 2024-03-24T03:17:22.492 INFO:teuthology.orchestra.run.smithi118.stdout: "enum_allowed": [ 2024-03-24T03:17:22.492 INFO:teuthology.orchestra.run.smithi118.stdout: "", 2024-03-24T03:17:22.492 INFO:teuthology.orchestra.run.smithi118.stdout: "critical", 2024-03-24T03:17:22.492 INFO:teuthology.orchestra.run.smithi118.stdout: "debug", 2024-03-24T03:17:22.492 INFO:teuthology.orchestra.run.smithi118.stdout: "error", 2024-03-24T03:17:22.492 INFO:teuthology.orchestra.run.smithi118.stdout: "info", 2024-03-24T03:17:22.492 INFO:teuthology.orchestra.run.smithi118.stdout: "warning" 2024-03-24T03:17:22.493 INFO:teuthology.orchestra.run.smithi118.stdout: ], 2024-03-24T03:17:22.493 INFO:teuthology.orchestra.run.smithi118.stdout: "desc": "", 2024-03-24T03:17:22.493 INFO:teuthology.orchestra.run.smithi118.stdout: "long_desc": "", 2024-03-24T03:17:22.493 INFO:teuthology.orchestra.run.smithi118.stdout: "tags": [], 2024-03-24T03:17:22.493 INFO:teuthology.orchestra.run.smithi118.stdout: "see_also": [] 2024-03-24T03:17:22.493 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-24T03:17:22.493 INFO:teuthology.orchestra.run.smithi118.stdout: "log_to_cluster": { 2024-03-24T03:17:22.493 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "log_to_cluster", 2024-03-24T03:17:22.493 INFO:teuthology.orchestra.run.smithi118.stdout: "type": "bool", 2024-03-24T03:17:22.494 INFO:teuthology.orchestra.run.smithi118.stdout: "level": "advanced", 2024-03-24T03:17:22.494 INFO:teuthology.orchestra.run.smithi118.stdout: "flags": 1, 2024-03-24T03:17:22.494 INFO:teuthology.orchestra.run.smithi118.stdout: "default_value": "False", 2024-03-24T03:17:22.494 INFO:teuthology.orchestra.run.smithi118.stdout: "min": "", 2024-03-24T03:17:22.494 INFO:teuthology.orchestra.run.smithi118.stdout: "max": "", 2024-03-24T03:17:22.494 INFO:teuthology.orchestra.run.smithi118.stdout: "enum_allowed": [], 2024-03-24T03:17:22.494 INFO:teuthology.orchestra.run.smithi118.stdout: "desc": "", 2024-03-24T03:17:22.494 INFO:teuthology.orchestra.run.smithi118.stdout: "long_desc": "", 2024-03-24T03:17:22.495 INFO:teuthology.orchestra.run.smithi118.stdout: "tags": [], 2024-03-24T03:17:22.495 INFO:teuthology.orchestra.run.smithi118.stdout: "see_also": [] 2024-03-24T03:17:22.495 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-24T03:17:22.495 INFO:teuthology.orchestra.run.smithi118.stdout: "log_to_cluster_level": { 2024-03-24T03:17:22.495 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "log_to_cluster_level", 2024-03-24T03:17:22.495 INFO:teuthology.orchestra.run.smithi118.stdout: "type": "str", 2024-03-24T03:17:22.495 INFO:teuthology.orchestra.run.smithi118.stdout: "level": "advanced", 2024-03-24T03:17:22.495 INFO:teuthology.orchestra.run.smithi118.stdout: "flags": 1, 2024-03-24T03:17:22.495 INFO:teuthology.orchestra.run.smithi118.stdout: "default_value": "info", 2024-03-24T03:17:22.496 INFO:teuthology.orchestra.run.smithi118.stdout: "min": "", 2024-03-24T03:17:22.496 INFO:teuthology.orchestra.run.smithi118.stdout: "max": "", 2024-03-24T03:17:22.496 INFO:teuthology.orchestra.run.smithi118.stdout: "enum_allowed": [ 2024-03-24T03:17:22.496 INFO:teuthology.orchestra.run.smithi118.stdout: "", 2024-03-24T03:17:22.496 INFO:teuthology.orchestra.run.smithi118.stdout: "critical", 2024-03-24T03:17:22.496 INFO:teuthology.orchestra.run.smithi118.stdout: "debug", 2024-03-24T03:17:22.496 INFO:teuthology.orchestra.run.smithi118.stdout: "error", 2024-03-24T03:17:22.496 INFO:teuthology.orchestra.run.smithi118.stdout: "info", 2024-03-24T03:17:22.496 INFO:teuthology.orchestra.run.smithi118.stdout: "warning" 2024-03-24T03:17:22.497 INFO:teuthology.orchestra.run.smithi118.stdout: ], 2024-03-24T03:17:22.497 INFO:teuthology.orchestra.run.smithi118.stdout: "desc": "", 2024-03-24T03:17:22.497 INFO:teuthology.orchestra.run.smithi118.stdout: "long_desc": "", 2024-03-24T03:17:22.497 INFO:teuthology.orchestra.run.smithi118.stdout: "tags": [], 2024-03-24T03:17:22.497 INFO:teuthology.orchestra.run.smithi118.stdout: "see_also": [] 2024-03-24T03:17:22.497 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-24T03:17:22.498 INFO:teuthology.orchestra.run.smithi118.stdout: "log_to_file": { 2024-03-24T03:17:22.498 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "log_to_file", 2024-03-24T03:17:22.498 INFO:teuthology.orchestra.run.smithi118.stdout: "type": "bool", 2024-03-24T03:17:22.498 INFO:teuthology.orchestra.run.smithi118.stdout: "level": "advanced", 2024-03-24T03:17:22.498 INFO:teuthology.orchestra.run.smithi118.stdout: "flags": 1, 2024-03-24T03:17:22.498 INFO:teuthology.orchestra.run.smithi118.stdout: "default_value": "False", 2024-03-24T03:17:22.498 INFO:teuthology.orchestra.run.smithi118.stdout: "min": "", 2024-03-24T03:17:22.498 INFO:teuthology.orchestra.run.smithi118.stdout: "max": "", 2024-03-24T03:17:22.498 INFO:teuthology.orchestra.run.smithi118.stdout: "enum_allowed": [], 2024-03-24T03:17:22.498 INFO:teuthology.orchestra.run.smithi118.stdout: "desc": "", 2024-03-24T03:17:22.499 INFO:teuthology.orchestra.run.smithi118.stdout: "long_desc": "", 2024-03-24T03:17:22.499 INFO:teuthology.orchestra.run.smithi118.stdout: "tags": [], 2024-03-24T03:17:22.499 INFO:teuthology.orchestra.run.smithi118.stdout: "see_also": [] 2024-03-24T03:17:22.499 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-24T03:17:22.499 INFO:teuthology.orchestra.run.smithi118.stdout: "min_score": { 2024-03-24T03:17:22.499 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "min_score", 2024-03-24T03:17:22.499 INFO:teuthology.orchestra.run.smithi118.stdout: "type": "float", 2024-03-24T03:17:22.499 INFO:teuthology.orchestra.run.smithi118.stdout: "level": "advanced", 2024-03-24T03:17:22.499 INFO:teuthology.orchestra.run.smithi118.stdout: "flags": 1, 2024-03-24T03:17:22.500 INFO:teuthology.orchestra.run.smithi118.stdout: "default_value": "0", 2024-03-24T03:17:22.500 INFO:teuthology.orchestra.run.smithi118.stdout: "min": "", 2024-03-24T03:17:22.500 INFO:teuthology.orchestra.run.smithi118.stdout: "max": "", 2024-03-24T03:17:22.500 INFO:teuthology.orchestra.run.smithi118.stdout: "enum_allowed": [], 2024-03-24T03:17:22.500 INFO:teuthology.orchestra.run.smithi118.stdout: "desc": "minimum score, below which no optimization is attempted", 2024-03-24T03:17:22.500 INFO:teuthology.orchestra.run.smithi118.stdout: "long_desc": "", 2024-03-24T03:17:22.500 INFO:teuthology.orchestra.run.smithi118.stdout: "tags": [], 2024-03-24T03:17:22.500 INFO:teuthology.orchestra.run.smithi118.stdout: "see_also": [] 2024-03-24T03:17:22.500 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-24T03:17:22.501 INFO:teuthology.orchestra.run.smithi118.stdout: "mode": { 2024-03-24T03:17:22.501 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "mode", 2024-03-24T03:17:22.501 INFO:teuthology.orchestra.run.smithi118.stdout: "type": "str", 2024-03-24T03:17:22.501 INFO:teuthology.orchestra.run.smithi118.stdout: "level": "advanced", 2024-03-24T03:17:22.501 INFO:teuthology.orchestra.run.smithi118.stdout: "flags": 1, 2024-03-24T03:17:22.501 INFO:teuthology.orchestra.run.smithi118.stdout: "default_value": "upmap", 2024-03-24T03:17:22.501 INFO:teuthology.orchestra.run.smithi118.stdout: "min": "", 2024-03-24T03:17:22.501 INFO:teuthology.orchestra.run.smithi118.stdout: "max": "", 2024-03-24T03:17:22.501 INFO:teuthology.orchestra.run.smithi118.stdout: "enum_allowed": [ 2024-03-24T03:17:22.502 INFO:teuthology.orchestra.run.smithi118.stdout: "crush-compat", 2024-03-24T03:17:22.502 INFO:teuthology.orchestra.run.smithi118.stdout: "none", 2024-03-24T03:17:22.502 INFO:teuthology.orchestra.run.smithi118.stdout: "read", 2024-03-24T03:17:22.502 INFO:teuthology.orchestra.run.smithi118.stdout: "upmap", 2024-03-24T03:17:22.502 INFO:teuthology.orchestra.run.smithi118.stdout: "upmap-read" 2024-03-24T03:17:22.502 INFO:teuthology.orchestra.run.smithi118.stdout: ], 2024-03-24T03:17:22.502 INFO:teuthology.orchestra.run.smithi118.stdout: "desc": "Balancer mode", 2024-03-24T03:17:22.502 INFO:teuthology.orchestra.run.smithi118.stdout: "long_desc": "", 2024-03-24T03:17:22.503 INFO:teuthology.orchestra.run.smithi118.stdout: "tags": [], 2024-03-24T03:17:22.503 INFO:teuthology.orchestra.run.smithi118.stdout: "see_also": [] 2024-03-24T03:17:22.503 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-24T03:17:22.503 INFO:teuthology.orchestra.run.smithi118.stdout: "pool_ids": { 2024-03-24T03:17:22.503 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "pool_ids", 2024-03-24T03:17:22.503 INFO:teuthology.orchestra.run.smithi118.stdout: "type": "str", 2024-03-24T03:17:22.503 INFO:teuthology.orchestra.run.smithi118.stdout: "level": "advanced", 2024-03-24T03:17:22.503 INFO:teuthology.orchestra.run.smithi118.stdout: "flags": 1, 2024-03-24T03:17:22.504 INFO:teuthology.orchestra.run.smithi118.stdout: "default_value": "", 2024-03-24T03:17:22.504 INFO:teuthology.orchestra.run.smithi118.stdout: "min": "", 2024-03-24T03:17:22.504 INFO:teuthology.orchestra.run.smithi118.stdout: "max": "", 2024-03-24T03:17:22.504 INFO:teuthology.orchestra.run.smithi118.stdout: "enum_allowed": [], 2024-03-24T03:17:22.504 INFO:teuthology.orchestra.run.smithi118.stdout: "desc": "pools which the automatic balancing will be limited to", 2024-03-24T03:17:22.504 INFO:teuthology.orchestra.run.smithi118.stdout: "long_desc": "", 2024-03-24T03:17:22.504 INFO:teuthology.orchestra.run.smithi118.stdout: "tags": [], 2024-03-24T03:17:22.504 INFO:teuthology.orchestra.run.smithi118.stdout: "see_also": [] 2024-03-24T03:17:22.505 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-24T03:17:22.505 INFO:teuthology.orchestra.run.smithi118.stdout: "sleep_interval": { 2024-03-24T03:17:22.505 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "sleep_interval", 2024-03-24T03:17:22.505 INFO:teuthology.orchestra.run.smithi118.stdout: "type": "secs", 2024-03-24T03:17:22.505 INFO:teuthology.orchestra.run.smithi118.stdout: "level": "advanced", 2024-03-24T03:17:22.505 INFO:teuthology.orchestra.run.smithi118.stdout: "flags": 1, 2024-03-24T03:17:22.505 INFO:teuthology.orchestra.run.smithi118.stdout: "default_value": "60", 2024-03-24T03:17:22.505 INFO:teuthology.orchestra.run.smithi118.stdout: "min": "", 2024-03-24T03:17:22.505 INFO:teuthology.orchestra.run.smithi118.stdout: "max": "", 2024-03-24T03:17:22.506 INFO:teuthology.orchestra.run.smithi118.stdout: "enum_allowed": [], 2024-03-24T03:17:22.506 INFO:teuthology.orchestra.run.smithi118.stdout: "desc": "how frequently to wake up and attempt optimization", 2024-03-24T03:17:22.506 INFO:teuthology.orchestra.run.smithi118.stdout: "long_desc": "", 2024-03-24T03:17:22.506 INFO:teuthology.orchestra.run.smithi118.stdout: "tags": [], 2024-03-24T03:17:22.506 INFO:teuthology.orchestra.run.smithi118.stdout: "see_also": [] 2024-03-24T03:17:22.506 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-24T03:17:22.506 INFO:teuthology.orchestra.run.smithi118.stdout: "upmap_max_deviation": { 2024-03-24T03:17:22.506 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "upmap_max_deviation", 2024-03-24T03:17:22.506 INFO:teuthology.orchestra.run.smithi118.stdout: "type": "int", 2024-03-24T03:17:22.507 INFO:teuthology.orchestra.run.smithi118.stdout: "level": "advanced", 2024-03-24T03:17:22.507 INFO:teuthology.orchestra.run.smithi118.stdout: "flags": 1, 2024-03-24T03:17:22.507 INFO:teuthology.orchestra.run.smithi118.stdout: "default_value": "5", 2024-03-24T03:17:22.507 INFO:teuthology.orchestra.run.smithi118.stdout: "min": "1", 2024-03-24T03:17:22.507 INFO:teuthology.orchestra.run.smithi118.stdout: "max": "", 2024-03-24T03:17:22.507 INFO:teuthology.orchestra.run.smithi118.stdout: "enum_allowed": [], 2024-03-24T03:17:22.507 INFO:teuthology.orchestra.run.smithi118.stdout: "desc": "deviation below which no optimization is attempted", 2024-03-24T03:17:22.508 INFO:teuthology.orchestra.run.smithi118.stdout: "long_desc": "If the number of PGs are within this count then no optimization is attempted", 2024-03-24T03:17:22.508 INFO:teuthology.orchestra.run.smithi118.stdout: "tags": [], 2024-03-24T03:17:22.508 INFO:teuthology.orchestra.run.smithi118.stdout: "see_also": [] 2024-03-24T03:17:22.508 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-24T03:17:22.508 INFO:teuthology.orchestra.run.smithi118.stdout: "upmap_max_optimizations": { 2024-03-24T03:17:22.508 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "upmap_max_optimizations", 2024-03-24T03:17:22.508 INFO:teuthology.orchestra.run.smithi118.stdout: "type": "uint", 2024-03-24T03:17:22.508 INFO:teuthology.orchestra.run.smithi118.stdout: "level": "advanced", 2024-03-24T03:17:22.508 INFO:teuthology.orchestra.run.smithi118.stdout: "flags": 1, 2024-03-24T03:17:22.509 INFO:teuthology.orchestra.run.smithi118.stdout: "default_value": "10", 2024-03-24T03:17:22.509 INFO:teuthology.orchestra.run.smithi118.stdout: "min": "", 2024-03-24T03:17:22.509 INFO:teuthology.orchestra.run.smithi118.stdout: "max": "", 2024-03-24T03:17:22.509 INFO:teuthology.orchestra.run.smithi118.stdout: "enum_allowed": [], 2024-03-24T03:17:22.509 INFO:teuthology.orchestra.run.smithi118.stdout: "desc": "maximum upmap optimizations to make per attempt", 2024-03-24T03:17:22.509 INFO:teuthology.orchestra.run.smithi118.stdout: "long_desc": "", 2024-03-24T03:17:22.509 INFO:teuthology.orchestra.run.smithi118.stdout: "tags": [], 2024-03-24T03:17:22.509 INFO:teuthology.orchestra.run.smithi118.stdout: "see_also": [] 2024-03-24T03:17:22.509 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-03-24T03:17:22.510 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-03-24T03:17:22.510 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-24T03:17:22.510 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-24T03:17:22.510 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "cephadm", 2024-03-24T03:17:22.510 INFO:teuthology.orchestra.run.smithi118.stdout: "can_run": true, 2024-03-24T03:17:22.510 INFO:teuthology.orchestra.run.smithi118.stdout: "error_string": "", 2024-03-24T03:17:22.510 INFO:teuthology.orchestra.run.smithi118.stdout: "module_options": { 2024-03-24T03:17:22.510 INFO:teuthology.orchestra.run.smithi118.stdout: "agent_down_multiplier": { 2024-03-24T03:17:22.510 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "agent_down_multiplier", 2024-03-24T03:17:22.511 INFO:teuthology.orchestra.run.smithi118.stdout: "type": "float", 2024-03-24T03:17:22.511 INFO:teuthology.orchestra.run.smithi118.stdout: "level": "advanced", 2024-03-24T03:17:22.511 INFO:teuthology.orchestra.run.smithi118.stdout: "flags": 0, 2024-03-24T03:17:22.511 INFO:teuthology.orchestra.run.smithi118.stdout: "default_value": "3.0", 2024-03-24T03:17:22.511 INFO:teuthology.orchestra.run.smithi118.stdout: "min": "", 2024-03-24T03:17:22.511 INFO:teuthology.orchestra.run.smithi118.stdout: "max": "", 2024-03-24T03:17:22.511 INFO:teuthology.orchestra.run.smithi118.stdout: "enum_allowed": [], 2024-03-24T03:17:22.511 INFO:teuthology.orchestra.run.smithi118.stdout: "desc": "Multiplied by agent refresh rate to calculate how long agent must not report before being marked down", 2024-03-24T03:17:22.512 INFO:teuthology.orchestra.run.smithi118.stdout: "long_desc": "", 2024-03-24T03:17:22.512 INFO:teuthology.orchestra.run.smithi118.stdout: "tags": [], 2024-03-24T03:17:22.512 INFO:teuthology.orchestra.run.smithi118.stdout: "see_also": [] 2024-03-24T03:17:22.512 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-24T03:17:22.512 INFO:teuthology.orchestra.run.smithi118.stdout: "agent_refresh_rate": { 2024-03-24T03:17:22.512 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "agent_refresh_rate", 2024-03-24T03:17:22.512 INFO:teuthology.orchestra.run.smithi118.stdout: "type": "secs", 2024-03-24T03:17:22.512 INFO:teuthology.orchestra.run.smithi118.stdout: "level": "advanced", 2024-03-24T03:17:22.512 INFO:teuthology.orchestra.run.smithi118.stdout: "flags": 0, 2024-03-24T03:17:22.513 INFO:teuthology.orchestra.run.smithi118.stdout: "default_value": "20", 2024-03-24T03:17:22.513 INFO:teuthology.orchestra.run.smithi118.stdout: "min": "", 2024-03-24T03:17:22.513 INFO:teuthology.orchestra.run.smithi118.stdout: "max": "", 2024-03-24T03:17:22.513 INFO:teuthology.orchestra.run.smithi118.stdout: "enum_allowed": [], 2024-03-24T03:17:22.513 INFO:teuthology.orchestra.run.smithi118.stdout: "desc": "How often agent on each host will try to gather and send metadata", 2024-03-24T03:17:22.513 INFO:teuthology.orchestra.run.smithi118.stdout: "long_desc": "", 2024-03-24T03:17:22.513 INFO:teuthology.orchestra.run.smithi118.stdout: "tags": [], 2024-03-24T03:17:22.513 INFO:teuthology.orchestra.run.smithi118.stdout: "see_also": [] 2024-03-24T03:17:22.513 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-24T03:17:22.513 INFO:teuthology.orchestra.run.smithi118.stdout: "agent_starting_port": { 2024-03-24T03:17:22.514 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "agent_starting_port", 2024-03-24T03:17:22.514 INFO:teuthology.orchestra.run.smithi118.stdout: "type": "int", 2024-03-24T03:17:22.514 INFO:teuthology.orchestra.run.smithi118.stdout: "level": "advanced", 2024-03-24T03:17:22.514 INFO:teuthology.orchestra.run.smithi118.stdout: "flags": 0, 2024-03-24T03:17:22.514 INFO:teuthology.orchestra.run.smithi118.stdout: "default_value": "4721", 2024-03-24T03:17:22.514 INFO:teuthology.orchestra.run.smithi118.stdout: "min": "", 2024-03-24T03:17:22.514 INFO:teuthology.orchestra.run.smithi118.stdout: "max": "", 2024-03-24T03:17:22.514 INFO:teuthology.orchestra.run.smithi118.stdout: "enum_allowed": [], 2024-03-24T03:17:22.515 INFO:teuthology.orchestra.run.smithi118.stdout: "desc": "First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)", 2024-03-24T03:17:22.515 INFO:teuthology.orchestra.run.smithi118.stdout: "long_desc": "", 2024-03-24T03:17:22.515 INFO:teuthology.orchestra.run.smithi118.stdout: "tags": [], 2024-03-24T03:17:22.515 INFO:teuthology.orchestra.run.smithi118.stdout: "see_also": [] 2024-03-24T03:17:22.515 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-24T03:17:22.515 INFO:teuthology.orchestra.run.smithi118.stdout: "allow_ptrace": { 2024-03-24T03:17:22.515 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "allow_ptrace", 2024-03-24T03:17:22.515 INFO:teuthology.orchestra.run.smithi118.stdout: "type": "bool", 2024-03-24T03:17:22.516 INFO:teuthology.orchestra.run.smithi118.stdout: "level": "advanced", 2024-03-24T03:17:22.516 INFO:teuthology.orchestra.run.smithi118.stdout: "flags": 0, 2024-03-24T03:17:22.516 INFO:teuthology.orchestra.run.smithi118.stdout: "default_value": "False", 2024-03-24T03:17:22.516 INFO:teuthology.orchestra.run.smithi118.stdout: "min": "", 2024-03-24T03:17:22.516 INFO:teuthology.orchestra.run.smithi118.stdout: "max": "", 2024-03-24T03:17:22.516 INFO:teuthology.orchestra.run.smithi118.stdout: "enum_allowed": [], 2024-03-24T03:17:22.516 INFO:teuthology.orchestra.run.smithi118.stdout: "desc": "allow SYS_PTRACE capability on ceph containers", 2024-03-24T03:17:22.516 INFO:teuthology.orchestra.run.smithi118.stdout: "long_desc": "The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.", 2024-03-24T03:17:22.516 INFO:teuthology.orchestra.run.smithi118.stdout: "tags": [], 2024-03-24T03:17:22.517 INFO:teuthology.orchestra.run.smithi118.stdout: "see_also": [] 2024-03-24T03:17:22.517 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-24T03:17:22.517 INFO:teuthology.orchestra.run.smithi118.stdout: "autotune_interval": { 2024-03-24T03:17:22.517 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "autotune_interval", 2024-03-24T03:17:22.517 INFO:teuthology.orchestra.run.smithi118.stdout: "type": "secs", 2024-03-24T03:17:22.517 INFO:teuthology.orchestra.run.smithi118.stdout: "level": "advanced", 2024-03-24T03:17:22.517 INFO:teuthology.orchestra.run.smithi118.stdout: "flags": 0, 2024-03-24T03:17:22.517 INFO:teuthology.orchestra.run.smithi118.stdout: "default_value": "600", 2024-03-24T03:17:22.517 INFO:teuthology.orchestra.run.smithi118.stdout: "min": "", 2024-03-24T03:17:22.518 INFO:teuthology.orchestra.run.smithi118.stdout: "max": "", 2024-03-24T03:17:22.518 INFO:teuthology.orchestra.run.smithi118.stdout: "enum_allowed": [], 2024-03-24T03:17:22.518 INFO:teuthology.orchestra.run.smithi118.stdout: "desc": "how frequently to autotune daemon memory", 2024-03-24T03:17:22.518 INFO:teuthology.orchestra.run.smithi118.stdout: "long_desc": "", 2024-03-24T03:17:22.518 INFO:teuthology.orchestra.run.smithi118.stdout: "tags": [], 2024-03-24T03:17:22.519 INFO:teuthology.orchestra.run.smithi118.stdout: "see_also": [] 2024-03-24T03:17:22.519 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-24T03:17:22.519 INFO:teuthology.orchestra.run.smithi118.stdout: "autotune_memory_target_ratio": { 2024-03-24T03:17:22.519 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "autotune_memory_target_ratio", 2024-03-24T03:17:22.519 INFO:teuthology.orchestra.run.smithi118.stdout: "type": "float", 2024-03-24T03:17:22.519 INFO:teuthology.orchestra.run.smithi118.stdout: "level": "advanced", 2024-03-24T03:17:22.519 INFO:teuthology.orchestra.run.smithi118.stdout: "flags": 0, 2024-03-24T03:17:22.519 INFO:teuthology.orchestra.run.smithi118.stdout: "default_value": "0.7", 2024-03-24T03:17:22.520 INFO:teuthology.orchestra.run.smithi118.stdout: "min": "", 2024-03-24T03:17:22.520 INFO:teuthology.orchestra.run.smithi118.stdout: "max": "", 2024-03-24T03:17:22.520 INFO:teuthology.orchestra.run.smithi118.stdout: "enum_allowed": [], 2024-03-24T03:17:22.520 INFO:teuthology.orchestra.run.smithi118.stdout: "desc": "ratio of total system memory to divide amongst autotuned daemons", 2024-03-24T03:17:22.520 INFO:teuthology.orchestra.run.smithi118.stdout: "long_desc": "", 2024-03-24T03:17:22.520 INFO:teuthology.orchestra.run.smithi118.stdout: "tags": [], 2024-03-24T03:17:22.520 INFO:teuthology.orchestra.run.smithi118.stdout: "see_also": [] 2024-03-24T03:17:22.520 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-24T03:17:22.521 INFO:teuthology.orchestra.run.smithi118.stdout: "cephadm_log_destination": { 2024-03-24T03:17:22.521 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "cephadm_log_destination", 2024-03-24T03:17:22.521 INFO:teuthology.orchestra.run.smithi118.stdout: "type": "str", 2024-03-24T03:17:22.521 INFO:teuthology.orchestra.run.smithi118.stdout: "level": "advanced", 2024-03-24T03:17:22.521 INFO:teuthology.orchestra.run.smithi118.stdout: "flags": 0, 2024-03-24T03:17:22.521 INFO:teuthology.orchestra.run.smithi118.stdout: "default_value": "", 2024-03-24T03:17:22.521 INFO:teuthology.orchestra.run.smithi118.stdout: "min": "", 2024-03-24T03:17:22.521 INFO:teuthology.orchestra.run.smithi118.stdout: "max": "", 2024-03-24T03:17:22.521 INFO:teuthology.orchestra.run.smithi118.stdout: "enum_allowed": [ 2024-03-24T03:17:22.522 INFO:teuthology.orchestra.run.smithi118.stdout: "file", 2024-03-24T03:17:22.522 INFO:teuthology.orchestra.run.smithi118.stdout: "file,syslog", 2024-03-24T03:17:22.522 INFO:teuthology.orchestra.run.smithi118.stdout: "syslog" 2024-03-24T03:17:22.522 INFO:teuthology.orchestra.run.smithi118.stdout: ], 2024-03-24T03:17:22.522 INFO:teuthology.orchestra.run.smithi118.stdout: "desc": "Destination for cephadm command's persistent logging", 2024-03-24T03:17:22.522 INFO:teuthology.orchestra.run.smithi118.stdout: "long_desc": "", 2024-03-24T03:17:22.522 INFO:teuthology.orchestra.run.smithi118.stdout: "tags": [], 2024-03-24T03:17:22.522 INFO:teuthology.orchestra.run.smithi118.stdout: "see_also": [] 2024-03-24T03:17:22.523 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-24T03:17:22.523 INFO:teuthology.orchestra.run.smithi118.stdout: "cgroups_split": { 2024-03-24T03:17:22.523 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "cgroups_split", 2024-03-24T03:17:22.523 INFO:teuthology.orchestra.run.smithi118.stdout: "type": "bool", 2024-03-24T03:17:22.523 INFO:teuthology.orchestra.run.smithi118.stdout: "level": "advanced", 2024-03-24T03:17:22.523 INFO:teuthology.orchestra.run.smithi118.stdout: "flags": 0, 2024-03-24T03:17:22.523 INFO:teuthology.orchestra.run.smithi118.stdout: "default_value": "True", 2024-03-24T03:17:22.523 INFO:teuthology.orchestra.run.smithi118.stdout: "min": "", 2024-03-24T03:17:22.523 INFO:teuthology.orchestra.run.smithi118.stdout: "max": "", 2024-03-24T03:17:22.524 INFO:teuthology.orchestra.run.smithi118.stdout: "enum_allowed": [], 2024-03-24T03:17:22.524 INFO:teuthology.orchestra.run.smithi118.stdout: "desc": "Pass --cgroups=split when cephadm creates containers (currently podman only)", 2024-03-24T03:17:22.524 INFO:teuthology.orchestra.run.smithi118.stdout: "long_desc": "", 2024-03-24T03:17:22.524 INFO:teuthology.orchestra.run.smithi118.stdout: "tags": [], 2024-03-24T03:17:22.524 INFO:teuthology.orchestra.run.smithi118.stdout: "see_also": [] 2024-03-24T03:17:22.524 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-24T03:17:22.524 INFO:teuthology.orchestra.run.smithi118.stdout: "config_checks_enabled": { 2024-03-24T03:17:22.524 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "config_checks_enabled", 2024-03-24T03:17:22.524 INFO:teuthology.orchestra.run.smithi118.stdout: "type": "bool", 2024-03-24T03:17:22.525 INFO:teuthology.orchestra.run.smithi118.stdout: "level": "advanced", 2024-03-24T03:17:22.525 INFO:teuthology.orchestra.run.smithi118.stdout: "flags": 0, 2024-03-24T03:17:22.525 INFO:teuthology.orchestra.run.smithi118.stdout: "default_value": "False", 2024-03-24T03:17:22.525 INFO:teuthology.orchestra.run.smithi118.stdout: "min": "", 2024-03-24T03:17:22.525 INFO:teuthology.orchestra.run.smithi118.stdout: "max": "", 2024-03-24T03:17:22.525 INFO:teuthology.orchestra.run.smithi118.stdout: "enum_allowed": [], 2024-03-24T03:17:22.525 INFO:teuthology.orchestra.run.smithi118.stdout: "desc": "Enable or disable the cephadm configuration analysis", 2024-03-24T03:17:22.525 INFO:teuthology.orchestra.run.smithi118.stdout: "long_desc": "", 2024-03-24T03:17:22.525 INFO:teuthology.orchestra.run.smithi118.stdout: "tags": [], 2024-03-24T03:17:22.526 INFO:teuthology.orchestra.run.smithi118.stdout: "see_also": [] 2024-03-24T03:17:22.526 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-24T03:17:22.526 INFO:teuthology.orchestra.run.smithi118.stdout: "config_dashboard": { 2024-03-24T03:17:22.526 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "config_dashboard", 2024-03-24T03:17:22.526 INFO:teuthology.orchestra.run.smithi118.stdout: "type": "bool", 2024-03-24T03:17:22.526 INFO:teuthology.orchestra.run.smithi118.stdout: "level": "advanced", 2024-03-24T03:17:22.526 INFO:teuthology.orchestra.run.smithi118.stdout: "flags": 0, 2024-03-24T03:17:22.526 INFO:teuthology.orchestra.run.smithi118.stdout: "default_value": "True", 2024-03-24T03:17:22.526 INFO:teuthology.orchestra.run.smithi118.stdout: "min": "", 2024-03-24T03:17:22.527 INFO:teuthology.orchestra.run.smithi118.stdout: "max": "", 2024-03-24T03:17:22.527 INFO:teuthology.orchestra.run.smithi118.stdout: "enum_allowed": [], 2024-03-24T03:17:22.527 INFO:teuthology.orchestra.run.smithi118.stdout: "desc": "manage configs like API endpoints in Dashboard.", 2024-03-24T03:17:22.527 INFO:teuthology.orchestra.run.smithi118.stdout: "long_desc": "", 2024-03-24T03:17:22.527 INFO:teuthology.orchestra.run.smithi118.stdout: "tags": [], 2024-03-24T03:17:22.527 INFO:teuthology.orchestra.run.smithi118.stdout: "see_also": [] 2024-03-24T03:17:22.527 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-24T03:17:22.527 INFO:teuthology.orchestra.run.smithi118.stdout: "container_image_alertmanager": { 2024-03-24T03:17:22.527 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "container_image_alertmanager", 2024-03-24T03:17:22.528 INFO:teuthology.orchestra.run.smithi118.stdout: "type": "str", 2024-03-24T03:17:22.528 INFO:teuthology.orchestra.run.smithi118.stdout: "level": "advanced", 2024-03-24T03:17:22.528 INFO:teuthology.orchestra.run.smithi118.stdout: "flags": 0, 2024-03-24T03:17:22.528 INFO:teuthology.orchestra.run.smithi118.stdout: "default_value": "quay.io/prometheus/alertmanager:v0.25.0", 2024-03-24T03:17:22.528 INFO:teuthology.orchestra.run.smithi118.stdout: "min": "", 2024-03-24T03:17:22.528 INFO:teuthology.orchestra.run.smithi118.stdout: "max": "", 2024-03-24T03:17:22.528 INFO:teuthology.orchestra.run.smithi118.stdout: "enum_allowed": [], 2024-03-24T03:17:22.528 INFO:teuthology.orchestra.run.smithi118.stdout: "desc": "Prometheus container image", 2024-03-24T03:17:22.528 INFO:teuthology.orchestra.run.smithi118.stdout: "long_desc": "", 2024-03-24T03:17:22.529 INFO:teuthology.orchestra.run.smithi118.stdout: "tags": [], 2024-03-24T03:17:22.529 INFO:teuthology.orchestra.run.smithi118.stdout: "see_also": [] 2024-03-24T03:17:22.529 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-24T03:17:22.529 INFO:teuthology.orchestra.run.smithi118.stdout: "container_image_base": { 2024-03-24T03:17:22.529 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "container_image_base", 2024-03-24T03:17:22.529 INFO:teuthology.orchestra.run.smithi118.stdout: "type": "str", 2024-03-24T03:17:22.529 INFO:teuthology.orchestra.run.smithi118.stdout: "level": "advanced", 2024-03-24T03:17:22.530 INFO:teuthology.orchestra.run.smithi118.stdout: "flags": 1, 2024-03-24T03:17:22.530 INFO:teuthology.orchestra.run.smithi118.stdout: "default_value": "quay.io/ceph/ceph", 2024-03-24T03:17:22.530 INFO:teuthology.orchestra.run.smithi118.stdout: "min": "", 2024-03-24T03:17:22.530 INFO:teuthology.orchestra.run.smithi118.stdout: "max": "", 2024-03-24T03:17:22.530 INFO:teuthology.orchestra.run.smithi118.stdout: "enum_allowed": [], 2024-03-24T03:17:22.530 INFO:teuthology.orchestra.run.smithi118.stdout: "desc": "Container image name, without the tag", 2024-03-24T03:17:22.530 INFO:teuthology.orchestra.run.smithi118.stdout: "long_desc": "", 2024-03-24T03:17:22.530 INFO:teuthology.orchestra.run.smithi118.stdout: "tags": [], 2024-03-24T03:17:22.530 INFO:teuthology.orchestra.run.smithi118.stdout: "see_also": [] 2024-03-24T03:17:22.531 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-24T03:17:22.531 INFO:teuthology.orchestra.run.smithi118.stdout: "container_image_elasticsearch": { 2024-03-24T03:17:22.531 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "container_image_elasticsearch", 2024-03-24T03:17:22.531 INFO:teuthology.orchestra.run.smithi118.stdout: "type": "str", 2024-03-24T03:17:22.531 INFO:teuthology.orchestra.run.smithi118.stdout: "level": "advanced", 2024-03-24T03:17:22.531 INFO:teuthology.orchestra.run.smithi118.stdout: "flags": 0, 2024-03-24T03:17:22.531 INFO:teuthology.orchestra.run.smithi118.stdout: "default_value": "quay.io/omrizeneva/elasticsearch:6.8.23", 2024-03-24T03:17:22.531 INFO:teuthology.orchestra.run.smithi118.stdout: "min": "", 2024-03-24T03:17:22.531 INFO:teuthology.orchestra.run.smithi118.stdout: "max": "", 2024-03-24T03:17:22.531 INFO:teuthology.orchestra.run.smithi118.stdout: "enum_allowed": [], 2024-03-24T03:17:22.532 INFO:teuthology.orchestra.run.smithi118.stdout: "desc": "elasticsearch container image", 2024-03-24T03:17:22.532 INFO:teuthology.orchestra.run.smithi118.stdout: "long_desc": "", 2024-03-24T03:17:22.532 INFO:teuthology.orchestra.run.smithi118.stdout: "tags": [], 2024-03-24T03:17:22.532 INFO:teuthology.orchestra.run.smithi118.stdout: "see_also": [] 2024-03-24T03:17:22.532 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-24T03:17:22.532 INFO:teuthology.orchestra.run.smithi118.stdout: "container_image_grafana": { 2024-03-24T03:17:22.532 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "container_image_grafana", 2024-03-24T03:17:22.532 INFO:teuthology.orchestra.run.smithi118.stdout: "type": "str", 2024-03-24T03:17:22.533 INFO:teuthology.orchestra.run.smithi118.stdout: "level": "advanced", 2024-03-24T03:17:22.533 INFO:teuthology.orchestra.run.smithi118.stdout: "flags": 0, 2024-03-24T03:17:22.533 INFO:teuthology.orchestra.run.smithi118.stdout: "default_value": "quay.io/ceph/ceph-grafana:9.4.12", 2024-03-24T03:17:22.533 INFO:teuthology.orchestra.run.smithi118.stdout: "min": "", 2024-03-24T03:17:22.533 INFO:teuthology.orchestra.run.smithi118.stdout: "max": "", 2024-03-24T03:17:22.533 INFO:teuthology.orchestra.run.smithi118.stdout: "enum_allowed": [], 2024-03-24T03:17:22.533 INFO:teuthology.orchestra.run.smithi118.stdout: "desc": "Prometheus container image", 2024-03-24T03:17:22.533 INFO:teuthology.orchestra.run.smithi118.stdout: "long_desc": "", 2024-03-24T03:17:22.533 INFO:teuthology.orchestra.run.smithi118.stdout: "tags": [], 2024-03-24T03:17:22.534 INFO:teuthology.orchestra.run.smithi118.stdout: "see_also": [] 2024-03-24T03:17:22.534 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-24T03:17:22.534 INFO:teuthology.orchestra.run.smithi118.stdout: "container_image_haproxy": { 2024-03-24T03:17:22.534 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "container_image_haproxy", 2024-03-24T03:17:22.534 INFO:teuthology.orchestra.run.smithi118.stdout: "type": "str", 2024-03-24T03:17:22.534 INFO:teuthology.orchestra.run.smithi118.stdout: "level": "advanced", 2024-03-24T03:17:22.534 INFO:teuthology.orchestra.run.smithi118.stdout: "flags": 0, 2024-03-24T03:17:22.534 INFO:teuthology.orchestra.run.smithi118.stdout: "default_value": "quay.io/ceph/haproxy:2.3", 2024-03-24T03:17:22.535 INFO:teuthology.orchestra.run.smithi118.stdout: "min": "", 2024-03-24T03:17:22.535 INFO:teuthology.orchestra.run.smithi118.stdout: "max": "", 2024-03-24T03:17:22.535 INFO:teuthology.orchestra.run.smithi118.stdout: "enum_allowed": [], 2024-03-24T03:17:22.535 INFO:teuthology.orchestra.run.smithi118.stdout: "desc": "HAproxy container image", 2024-03-24T03:17:22.535 INFO:teuthology.orchestra.run.smithi118.stdout: "long_desc": "", 2024-03-24T03:17:22.535 INFO:teuthology.orchestra.run.smithi118.stdout: "tags": [], 2024-03-24T03:17:22.535 INFO:teuthology.orchestra.run.smithi118.stdout: "see_also": [] 2024-03-24T03:17:22.535 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-24T03:17:22.535 INFO:teuthology.orchestra.run.smithi118.stdout: "container_image_jaeger_agent": { 2024-03-24T03:17:22.536 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "container_image_jaeger_agent", 2024-03-24T03:17:22.536 INFO:teuthology.orchestra.run.smithi118.stdout: "type": "str", 2024-03-24T03:17:22.536 INFO:teuthology.orchestra.run.smithi118.stdout: "level": "advanced", 2024-03-24T03:17:22.536 INFO:teuthology.orchestra.run.smithi118.stdout: "flags": 0, 2024-03-24T03:17:22.536 INFO:teuthology.orchestra.run.smithi118.stdout: "default_value": "quay.io/jaegertracing/jaeger-agent:1.29", 2024-03-24T03:17:22.536 INFO:teuthology.orchestra.run.smithi118.stdout: "min": "", 2024-03-24T03:17:22.536 INFO:teuthology.orchestra.run.smithi118.stdout: "max": "", 2024-03-24T03:17:22.536 INFO:teuthology.orchestra.run.smithi118.stdout: "enum_allowed": [], 2024-03-24T03:17:22.536 INFO:teuthology.orchestra.run.smithi118.stdout: "desc": "Jaeger agent container image", 2024-03-24T03:17:22.537 INFO:teuthology.orchestra.run.smithi118.stdout: "long_desc": "", 2024-03-24T03:17:22.537 INFO:teuthology.orchestra.run.smithi118.stdout: "tags": [], 2024-03-24T03:17:22.537 INFO:teuthology.orchestra.run.smithi118.stdout: "see_also": [] 2024-03-24T03:17:22.537 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-24T03:17:22.537 INFO:teuthology.orchestra.run.smithi118.stdout: "container_image_jaeger_collector": { 2024-03-24T03:17:22.537 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "container_image_jaeger_collector", 2024-03-24T03:17:22.537 INFO:teuthology.orchestra.run.smithi118.stdout: "type": "str", 2024-03-24T03:17:22.537 INFO:teuthology.orchestra.run.smithi118.stdout: "level": "advanced", 2024-03-24T03:17:22.537 INFO:teuthology.orchestra.run.smithi118.stdout: "flags": 0, 2024-03-24T03:17:22.538 INFO:teuthology.orchestra.run.smithi118.stdout: "default_value": "quay.io/jaegertracing/jaeger-collector:1.29", 2024-03-24T03:17:22.538 INFO:teuthology.orchestra.run.smithi118.stdout: "min": "", 2024-03-24T03:17:22.538 INFO:teuthology.orchestra.run.smithi118.stdout: "max": "", 2024-03-24T03:17:22.538 INFO:teuthology.orchestra.run.smithi118.stdout: "enum_allowed": [], 2024-03-24T03:17:22.538 INFO:teuthology.orchestra.run.smithi118.stdout: "desc": "Jaeger collector container image", 2024-03-24T03:17:22.538 INFO:teuthology.orchestra.run.smithi118.stdout: "long_desc": "", 2024-03-24T03:17:22.538 INFO:teuthology.orchestra.run.smithi118.stdout: "tags": [], 2024-03-24T03:17:22.538 INFO:teuthology.orchestra.run.smithi118.stdout: "see_also": [] 2024-03-24T03:17:22.539 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-24T03:17:22.539 INFO:teuthology.orchestra.run.smithi118.stdout: "container_image_jaeger_query": { 2024-03-24T03:17:22.539 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "container_image_jaeger_query", 2024-03-24T03:17:22.539 INFO:teuthology.orchestra.run.smithi118.stdout: "type": "str", 2024-03-24T03:17:22.539 INFO:teuthology.orchestra.run.smithi118.stdout: "level": "advanced", 2024-03-24T03:17:22.539 INFO:teuthology.orchestra.run.smithi118.stdout: "flags": 0, 2024-03-24T03:17:22.539 INFO:teuthology.orchestra.run.smithi118.stdout: "default_value": "quay.io/jaegertracing/jaeger-query:1.29", 2024-03-24T03:17:22.540 INFO:teuthology.orchestra.run.smithi118.stdout: "min": "", 2024-03-24T03:17:22.540 INFO:teuthology.orchestra.run.smithi118.stdout: "max": "", 2024-03-24T03:17:22.540 INFO:teuthology.orchestra.run.smithi118.stdout: "enum_allowed": [], 2024-03-24T03:17:22.540 INFO:teuthology.orchestra.run.smithi118.stdout: "desc": "Jaeger query container image", 2024-03-24T03:17:22.540 INFO:teuthology.orchestra.run.smithi118.stdout: "long_desc": "", 2024-03-24T03:17:22.540 INFO:teuthology.orchestra.run.smithi118.stdout: "tags": [], 2024-03-24T03:17:22.540 INFO:teuthology.orchestra.run.smithi118.stdout: "see_also": [] 2024-03-24T03:17:22.540 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-24T03:17:22.540 INFO:teuthology.orchestra.run.smithi118.stdout: "container_image_keepalived": { 2024-03-24T03:17:22.540 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "container_image_keepalived", 2024-03-24T03:17:22.541 INFO:teuthology.orchestra.run.smithi118.stdout: "type": "str", 2024-03-24T03:17:22.541 INFO:teuthology.orchestra.run.smithi118.stdout: "level": "advanced", 2024-03-24T03:17:22.541 INFO:teuthology.orchestra.run.smithi118.stdout: "flags": 0, 2024-03-24T03:17:22.541 INFO:teuthology.orchestra.run.smithi118.stdout: "default_value": "quay.io/ceph/keepalived:2.2.4", 2024-03-24T03:17:22.541 INFO:teuthology.orchestra.run.smithi118.stdout: "min": "", 2024-03-24T03:17:22.541 INFO:teuthology.orchestra.run.smithi118.stdout: "max": "", 2024-03-24T03:17:22.541 INFO:teuthology.orchestra.run.smithi118.stdout: "enum_allowed": [], 2024-03-24T03:17:22.541 INFO:teuthology.orchestra.run.smithi118.stdout: "desc": "Keepalived container image", 2024-03-24T03:17:22.541 INFO:teuthology.orchestra.run.smithi118.stdout: "long_desc": "", 2024-03-24T03:17:22.542 INFO:teuthology.orchestra.run.smithi118.stdout: "tags": [], 2024-03-24T03:17:22.542 INFO:teuthology.orchestra.run.smithi118.stdout: "see_also": [] 2024-03-24T03:17:22.542 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-24T03:17:22.542 INFO:teuthology.orchestra.run.smithi118.stdout: "container_image_loki": { 2024-03-24T03:17:22.542 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "container_image_loki", 2024-03-24T03:17:22.542 INFO:teuthology.orchestra.run.smithi118.stdout: "type": "str", 2024-03-24T03:17:22.542 INFO:teuthology.orchestra.run.smithi118.stdout: "level": "advanced", 2024-03-24T03:17:22.542 INFO:teuthology.orchestra.run.smithi118.stdout: "flags": 0, 2024-03-24T03:17:22.542 INFO:teuthology.orchestra.run.smithi118.stdout: "default_value": "docker.io/grafana/loki:2.4.0", 2024-03-24T03:17:22.543 INFO:teuthology.orchestra.run.smithi118.stdout: "min": "", 2024-03-24T03:17:22.543 INFO:teuthology.orchestra.run.smithi118.stdout: "max": "", 2024-03-24T03:17:22.543 INFO:teuthology.orchestra.run.smithi118.stdout: "enum_allowed": [], 2024-03-24T03:17:22.543 INFO:teuthology.orchestra.run.smithi118.stdout: "desc": "Loki container image", 2024-03-24T03:17:22.543 INFO:teuthology.orchestra.run.smithi118.stdout: "long_desc": "", 2024-03-24T03:17:22.543 INFO:teuthology.orchestra.run.smithi118.stdout: "tags": [], 2024-03-24T03:17:22.543 INFO:teuthology.orchestra.run.smithi118.stdout: "see_also": [] 2024-03-24T03:17:22.543 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-24T03:17:22.543 INFO:teuthology.orchestra.run.smithi118.stdout: "container_image_node_exporter": { 2024-03-24T03:17:22.544 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "container_image_node_exporter", 2024-03-24T03:17:22.544 INFO:teuthology.orchestra.run.smithi118.stdout: "type": "str", 2024-03-24T03:17:22.544 INFO:teuthology.orchestra.run.smithi118.stdout: "level": "advanced", 2024-03-24T03:17:22.544 INFO:teuthology.orchestra.run.smithi118.stdout: "flags": 0, 2024-03-24T03:17:22.544 INFO:teuthology.orchestra.run.smithi118.stdout: "default_value": "quay.io/prometheus/node-exporter:v1.5.0", 2024-03-24T03:17:22.544 INFO:teuthology.orchestra.run.smithi118.stdout: "min": "", 2024-03-24T03:17:22.544 INFO:teuthology.orchestra.run.smithi118.stdout: "max": "", 2024-03-24T03:17:22.544 INFO:teuthology.orchestra.run.smithi118.stdout: "enum_allowed": [], 2024-03-24T03:17:22.545 INFO:teuthology.orchestra.run.smithi118.stdout: "desc": "Prometheus container image", 2024-03-24T03:17:22.545 INFO:teuthology.orchestra.run.smithi118.stdout: "long_desc": "", 2024-03-24T03:17:22.545 INFO:teuthology.orchestra.run.smithi118.stdout: "tags": [], 2024-03-24T03:17:22.545 INFO:teuthology.orchestra.run.smithi118.stdout: "see_also": [] 2024-03-24T03:17:22.545 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-24T03:17:22.545 INFO:teuthology.orchestra.run.smithi118.stdout: "container_image_nvmeof": { 2024-03-24T03:17:22.545 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "container_image_nvmeof", 2024-03-24T03:17:22.545 INFO:teuthology.orchestra.run.smithi118.stdout: "type": "str", 2024-03-24T03:17:22.545 INFO:teuthology.orchestra.run.smithi118.stdout: "level": "advanced", 2024-03-24T03:17:22.545 INFO:teuthology.orchestra.run.smithi118.stdout: "flags": 0, 2024-03-24T03:17:22.546 INFO:teuthology.orchestra.run.smithi118.stdout: "default_value": "quay.io/ceph/nvmeof:1.0.0", 2024-03-24T03:17:22.546 INFO:teuthology.orchestra.run.smithi118.stdout: "min": "", 2024-03-24T03:17:22.546 INFO:teuthology.orchestra.run.smithi118.stdout: "max": "", 2024-03-24T03:17:22.546 INFO:teuthology.orchestra.run.smithi118.stdout: "enum_allowed": [], 2024-03-24T03:17:22.546 INFO:teuthology.orchestra.run.smithi118.stdout: "desc": "Nvme-of container image", 2024-03-24T03:17:22.546 INFO:teuthology.orchestra.run.smithi118.stdout: "long_desc": "", 2024-03-24T03:17:22.546 INFO:teuthology.orchestra.run.smithi118.stdout: "tags": [], 2024-03-24T03:17:22.546 INFO:teuthology.orchestra.run.smithi118.stdout: "see_also": [] 2024-03-24T03:17:22.546 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-24T03:17:22.547 INFO:teuthology.orchestra.run.smithi118.stdout: "container_image_prometheus": { 2024-03-24T03:17:22.547 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "container_image_prometheus", 2024-03-24T03:17:22.547 INFO:teuthology.orchestra.run.smithi118.stdout: "type": "str", 2024-03-24T03:17:22.547 INFO:teuthology.orchestra.run.smithi118.stdout: "level": "advanced", 2024-03-24T03:17:22.547 INFO:teuthology.orchestra.run.smithi118.stdout: "flags": 0, 2024-03-24T03:17:22.547 INFO:teuthology.orchestra.run.smithi118.stdout: "default_value": "quay.io/prometheus/prometheus:v2.43.0", 2024-03-24T03:17:22.547 INFO:teuthology.orchestra.run.smithi118.stdout: "min": "", 2024-03-24T03:17:22.547 INFO:teuthology.orchestra.run.smithi118.stdout: "max": "", 2024-03-24T03:17:22.547 INFO:teuthology.orchestra.run.smithi118.stdout: "enum_allowed": [], 2024-03-24T03:17:22.548 INFO:teuthology.orchestra.run.smithi118.stdout: "desc": "Prometheus container image", 2024-03-24T03:17:22.548 INFO:teuthology.orchestra.run.smithi118.stdout: "long_desc": "", 2024-03-24T03:17:22.548 INFO:teuthology.orchestra.run.smithi118.stdout: "tags": [], 2024-03-24T03:17:22.548 INFO:teuthology.orchestra.run.smithi118.stdout: "see_also": [] 2024-03-24T03:17:22.548 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-24T03:17:22.548 INFO:teuthology.orchestra.run.smithi118.stdout: "container_image_promtail": { 2024-03-24T03:17:22.548 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "container_image_promtail", 2024-03-24T03:17:22.548 INFO:teuthology.orchestra.run.smithi118.stdout: "type": "str", 2024-03-24T03:17:22.548 INFO:teuthology.orchestra.run.smithi118.stdout: "level": "advanced", 2024-03-24T03:17:22.549 INFO:teuthology.orchestra.run.smithi118.stdout: "flags": 0, 2024-03-24T03:17:22.549 INFO:teuthology.orchestra.run.smithi118.stdout: "default_value": "docker.io/grafana/promtail:2.4.0", 2024-03-24T03:17:22.549 INFO:teuthology.orchestra.run.smithi118.stdout: "min": "", 2024-03-24T03:17:22.549 INFO:teuthology.orchestra.run.smithi118.stdout: "max": "", 2024-03-24T03:17:22.549 INFO:teuthology.orchestra.run.smithi118.stdout: "enum_allowed": [], 2024-03-24T03:17:22.549 INFO:teuthology.orchestra.run.smithi118.stdout: "desc": "Promtail container image", 2024-03-24T03:17:22.549 INFO:teuthology.orchestra.run.smithi118.stdout: "long_desc": "", 2024-03-24T03:17:22.549 INFO:teuthology.orchestra.run.smithi118.stdout: "tags": [], 2024-03-24T03:17:22.550 INFO:teuthology.orchestra.run.smithi118.stdout: "see_also": [] 2024-03-24T03:17:22.550 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-24T03:17:22.550 INFO:teuthology.orchestra.run.smithi118.stdout: "container_image_snmp_gateway": { 2024-03-24T03:17:22.550 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "container_image_snmp_gateway", 2024-03-24T03:17:22.550 INFO:teuthology.orchestra.run.smithi118.stdout: "type": "str", 2024-03-24T03:17:22.550 INFO:teuthology.orchestra.run.smithi118.stdout: "level": "advanced", 2024-03-24T03:17:22.550 INFO:teuthology.orchestra.run.smithi118.stdout: "flags": 0, 2024-03-24T03:17:22.550 INFO:teuthology.orchestra.run.smithi118.stdout: "default_value": "docker.io/maxwo/snmp-notifier:v1.2.1", 2024-03-24T03:17:22.551 INFO:teuthology.orchestra.run.smithi118.stdout: "min": "", 2024-03-24T03:17:22.551 INFO:teuthology.orchestra.run.smithi118.stdout: "max": "", 2024-03-24T03:17:22.551 INFO:teuthology.orchestra.run.smithi118.stdout: "enum_allowed": [], 2024-03-24T03:17:22.551 INFO:teuthology.orchestra.run.smithi118.stdout: "desc": "SNMP Gateway container image", 2024-03-24T03:17:22.551 INFO:teuthology.orchestra.run.smithi118.stdout: "long_desc": "", 2024-03-24T03:17:22.551 INFO:teuthology.orchestra.run.smithi118.stdout: "tags": [], 2024-03-24T03:17:22.551 INFO:teuthology.orchestra.run.smithi118.stdout: "see_also": [] 2024-03-24T03:17:22.551 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-24T03:17:22.552 INFO:teuthology.orchestra.run.smithi118.stdout: "container_init": { 2024-03-24T03:17:22.552 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "container_init", 2024-03-24T03:17:22.552 INFO:teuthology.orchestra.run.smithi118.stdout: "type": "bool", 2024-03-24T03:17:22.552 INFO:teuthology.orchestra.run.smithi118.stdout: "level": "advanced", 2024-03-24T03:17:22.552 INFO:teuthology.orchestra.run.smithi118.stdout: "flags": 0, 2024-03-24T03:17:22.552 INFO:teuthology.orchestra.run.smithi118.stdout: "default_value": "True", 2024-03-24T03:17:22.552 INFO:teuthology.orchestra.run.smithi118.stdout: "min": "", 2024-03-24T03:17:22.552 INFO:teuthology.orchestra.run.smithi118.stdout: "max": "", 2024-03-24T03:17:22.552 INFO:teuthology.orchestra.run.smithi118.stdout: "enum_allowed": [], 2024-03-24T03:17:22.553 INFO:teuthology.orchestra.run.smithi118.stdout: "desc": "Run podman/docker with `--init`", 2024-03-24T03:17:22.553 INFO:teuthology.orchestra.run.smithi118.stdout: "long_desc": "", 2024-03-24T03:17:22.553 INFO:teuthology.orchestra.run.smithi118.stdout: "tags": [], 2024-03-24T03:17:22.553 INFO:teuthology.orchestra.run.smithi118.stdout: "see_also": [] 2024-03-24T03:17:22.553 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-24T03:17:22.553 INFO:teuthology.orchestra.run.smithi118.stdout: "daemon_cache_timeout": { 2024-03-24T03:17:22.553 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "daemon_cache_timeout", 2024-03-24T03:17:22.553 INFO:teuthology.orchestra.run.smithi118.stdout: "type": "secs", 2024-03-24T03:17:22.554 INFO:teuthology.orchestra.run.smithi118.stdout: "level": "advanced", 2024-03-24T03:17:22.554 INFO:teuthology.orchestra.run.smithi118.stdout: "flags": 0, 2024-03-24T03:17:22.554 INFO:teuthology.orchestra.run.smithi118.stdout: "default_value": "600", 2024-03-24T03:17:22.554 INFO:teuthology.orchestra.run.smithi118.stdout: "min": "", 2024-03-24T03:17:22.554 INFO:teuthology.orchestra.run.smithi118.stdout: "max": "", 2024-03-24T03:17:22.554 INFO:teuthology.orchestra.run.smithi118.stdout: "enum_allowed": [], 2024-03-24T03:17:22.554 INFO:teuthology.orchestra.run.smithi118.stdout: "desc": "seconds to cache service (daemon) inventory", 2024-03-24T03:17:22.555 INFO:teuthology.orchestra.run.smithi118.stdout: "long_desc": "", 2024-03-24T03:17:22.555 INFO:teuthology.orchestra.run.smithi118.stdout: "tags": [], 2024-03-24T03:17:22.555 INFO:teuthology.orchestra.run.smithi118.stdout: "see_also": [] 2024-03-24T03:17:22.555 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-24T03:17:22.555 INFO:teuthology.orchestra.run.smithi118.stdout: "default_cephadm_command_timeout": { 2024-03-24T03:17:22.555 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "default_cephadm_command_timeout", 2024-03-24T03:17:22.555 INFO:teuthology.orchestra.run.smithi118.stdout: "type": "int", 2024-03-24T03:17:22.555 INFO:teuthology.orchestra.run.smithi118.stdout: "level": "advanced", 2024-03-24T03:17:22.555 INFO:teuthology.orchestra.run.smithi118.stdout: "flags": 0, 2024-03-24T03:17:22.556 INFO:teuthology.orchestra.run.smithi118.stdout: "default_value": "900", 2024-03-24T03:17:22.556 INFO:teuthology.orchestra.run.smithi118.stdout: "min": "", 2024-03-24T03:17:22.556 INFO:teuthology.orchestra.run.smithi118.stdout: "max": "", 2024-03-24T03:17:22.556 INFO:teuthology.orchestra.run.smithi118.stdout: "enum_allowed": [], 2024-03-24T03:17:22.556 INFO:teuthology.orchestra.run.smithi118.stdout: "desc": "Default timeout applied to cephadm commands run directly on the host (in seconds)", 2024-03-24T03:17:22.556 INFO:teuthology.orchestra.run.smithi118.stdout: "long_desc": "", 2024-03-24T03:17:22.556 INFO:teuthology.orchestra.run.smithi118.stdout: "tags": [], 2024-03-24T03:17:22.556 INFO:teuthology.orchestra.run.smithi118.stdout: "see_also": [] 2024-03-24T03:17:22.556 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-24T03:17:22.557 INFO:teuthology.orchestra.run.smithi118.stdout: "default_registry": { 2024-03-24T03:17:22.557 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "default_registry", 2024-03-24T03:17:22.557 INFO:teuthology.orchestra.run.smithi118.stdout: "type": "str", 2024-03-24T03:17:22.557 INFO:teuthology.orchestra.run.smithi118.stdout: "level": "advanced", 2024-03-24T03:17:22.557 INFO:teuthology.orchestra.run.smithi118.stdout: "flags": 0, 2024-03-24T03:17:22.557 INFO:teuthology.orchestra.run.smithi118.stdout: "default_value": "docker.io", 2024-03-24T03:17:22.557 INFO:teuthology.orchestra.run.smithi118.stdout: "min": "", 2024-03-24T03:17:22.557 INFO:teuthology.orchestra.run.smithi118.stdout: "max": "", 2024-03-24T03:17:22.557 INFO:teuthology.orchestra.run.smithi118.stdout: "enum_allowed": [], 2024-03-24T03:17:22.558 INFO:teuthology.orchestra.run.smithi118.stdout: "desc": "Search-registry to which we should normalize unqualified image names. This is not the default registry", 2024-03-24T03:17:22.558 INFO:teuthology.orchestra.run.smithi118.stdout: "long_desc": "", 2024-03-24T03:17:22.558 INFO:teuthology.orchestra.run.smithi118.stdout: "tags": [], 2024-03-24T03:17:22.558 INFO:teuthology.orchestra.run.smithi118.stdout: "see_also": [] 2024-03-24T03:17:22.558 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-24T03:17:22.559 INFO:teuthology.orchestra.run.smithi118.stdout: "device_cache_timeout": { 2024-03-24T03:17:22.559 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "device_cache_timeout", 2024-03-24T03:17:22.559 INFO:teuthology.orchestra.run.smithi118.stdout: "type": "secs", 2024-03-24T03:17:22.559 INFO:teuthology.orchestra.run.smithi118.stdout: "level": "advanced", 2024-03-24T03:17:22.559 INFO:teuthology.orchestra.run.smithi118.stdout: "flags": 0, 2024-03-24T03:17:22.559 INFO:teuthology.orchestra.run.smithi118.stdout: "default_value": "1800", 2024-03-24T03:17:22.559 INFO:teuthology.orchestra.run.smithi118.stdout: "min": "", 2024-03-24T03:17:22.559 INFO:teuthology.orchestra.run.smithi118.stdout: "max": "", 2024-03-24T03:17:22.559 INFO:teuthology.orchestra.run.smithi118.stdout: "enum_allowed": [], 2024-03-24T03:17:22.560 INFO:teuthology.orchestra.run.smithi118.stdout: "desc": "seconds to cache device inventory", 2024-03-24T03:17:22.560 INFO:teuthology.orchestra.run.smithi118.stdout: "long_desc": "", 2024-03-24T03:17:22.560 INFO:teuthology.orchestra.run.smithi118.stdout: "tags": [], 2024-03-24T03:17:22.560 INFO:teuthology.orchestra.run.smithi118.stdout: "see_also": [] 2024-03-24T03:17:22.560 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-24T03:17:22.560 INFO:teuthology.orchestra.run.smithi118.stdout: "device_enhanced_scan": { 2024-03-24T03:17:22.560 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "device_enhanced_scan", 2024-03-24T03:17:22.561 INFO:teuthology.orchestra.run.smithi118.stdout: "type": "bool", 2024-03-24T03:17:22.561 INFO:teuthology.orchestra.run.smithi118.stdout: "level": "advanced", 2024-03-24T03:17:22.561 INFO:teuthology.orchestra.run.smithi118.stdout: "flags": 0, 2024-03-24T03:17:22.561 INFO:teuthology.orchestra.run.smithi118.stdout: "default_value": "False", 2024-03-24T03:17:22.561 INFO:teuthology.orchestra.run.smithi118.stdout: "min": "", 2024-03-24T03:17:22.561 INFO:teuthology.orchestra.run.smithi118.stdout: "max": "", 2024-03-24T03:17:22.561 INFO:teuthology.orchestra.run.smithi118.stdout: "enum_allowed": [], 2024-03-24T03:17:22.561 INFO:teuthology.orchestra.run.smithi118.stdout: "desc": "Use libstoragemgmt during device scans", 2024-03-24T03:17:22.562 INFO:teuthology.orchestra.run.smithi118.stdout: "long_desc": "", 2024-03-24T03:17:22.562 INFO:teuthology.orchestra.run.smithi118.stdout: "tags": [], 2024-03-24T03:17:22.562 INFO:teuthology.orchestra.run.smithi118.stdout: "see_also": [] 2024-03-24T03:17:22.562 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-24T03:17:22.562 INFO:teuthology.orchestra.run.smithi118.stdout: "facts_cache_timeout": { 2024-03-24T03:17:22.562 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "facts_cache_timeout", 2024-03-24T03:17:22.562 INFO:teuthology.orchestra.run.smithi118.stdout: "type": "secs", 2024-03-24T03:17:22.562 INFO:teuthology.orchestra.run.smithi118.stdout: "level": "advanced", 2024-03-24T03:17:22.562 INFO:teuthology.orchestra.run.smithi118.stdout: "flags": 0, 2024-03-24T03:17:22.563 INFO:teuthology.orchestra.run.smithi118.stdout: "default_value": "60", 2024-03-24T03:17:22.563 INFO:teuthology.orchestra.run.smithi118.stdout: "min": "", 2024-03-24T03:17:22.563 INFO:teuthology.orchestra.run.smithi118.stdout: "max": "", 2024-03-24T03:17:22.563 INFO:teuthology.orchestra.run.smithi118.stdout: "enum_allowed": [], 2024-03-24T03:17:22.563 INFO:teuthology.orchestra.run.smithi118.stdout: "desc": "seconds to cache host facts data", 2024-03-24T03:17:22.563 INFO:teuthology.orchestra.run.smithi118.stdout: "long_desc": "", 2024-03-24T03:17:22.563 INFO:teuthology.orchestra.run.smithi118.stdout: "tags": [], 2024-03-24T03:17:22.563 INFO:teuthology.orchestra.run.smithi118.stdout: "see_also": [] 2024-03-24T03:17:22.563 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-24T03:17:22.564 INFO:teuthology.orchestra.run.smithi118.stdout: "host_check_interval": { 2024-03-24T03:17:22.564 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "host_check_interval", 2024-03-24T03:17:22.564 INFO:teuthology.orchestra.run.smithi118.stdout: "type": "secs", 2024-03-24T03:17:22.564 INFO:teuthology.orchestra.run.smithi118.stdout: "level": "advanced", 2024-03-24T03:17:22.564 INFO:teuthology.orchestra.run.smithi118.stdout: "flags": 0, 2024-03-24T03:17:22.564 INFO:teuthology.orchestra.run.smithi118.stdout: "default_value": "600", 2024-03-24T03:17:22.564 INFO:teuthology.orchestra.run.smithi118.stdout: "min": "", 2024-03-24T03:17:22.564 INFO:teuthology.orchestra.run.smithi118.stdout: "max": "", 2024-03-24T03:17:22.565 INFO:teuthology.orchestra.run.smithi118.stdout: "enum_allowed": [], 2024-03-24T03:17:22.565 INFO:teuthology.orchestra.run.smithi118.stdout: "desc": "how frequently to perform a host check", 2024-03-24T03:17:22.565 INFO:teuthology.orchestra.run.smithi118.stdout: "long_desc": "", 2024-03-24T03:17:22.565 INFO:teuthology.orchestra.run.smithi118.stdout: "tags": [], 2024-03-24T03:17:22.565 INFO:teuthology.orchestra.run.smithi118.stdout: "see_also": [] 2024-03-24T03:17:22.565 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-24T03:17:22.565 INFO:teuthology.orchestra.run.smithi118.stdout: "hw_monitoring": { 2024-03-24T03:17:22.565 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "hw_monitoring", 2024-03-24T03:17:22.565 INFO:teuthology.orchestra.run.smithi118.stdout: "type": "bool", 2024-03-24T03:17:22.565 INFO:teuthology.orchestra.run.smithi118.stdout: "level": "advanced", 2024-03-24T03:17:22.566 INFO:teuthology.orchestra.run.smithi118.stdout: "flags": 0, 2024-03-24T03:17:22.566 INFO:teuthology.orchestra.run.smithi118.stdout: "default_value": "False", 2024-03-24T03:17:22.566 INFO:teuthology.orchestra.run.smithi118.stdout: "min": "", 2024-03-24T03:17:22.566 INFO:teuthology.orchestra.run.smithi118.stdout: "max": "", 2024-03-24T03:17:22.566 INFO:teuthology.orchestra.run.smithi118.stdout: "enum_allowed": [], 2024-03-24T03:17:22.566 INFO:teuthology.orchestra.run.smithi118.stdout: "desc": "Deploy hw monitoring daemon on every host.", 2024-03-24T03:17:22.566 INFO:teuthology.orchestra.run.smithi118.stdout: "long_desc": "", 2024-03-24T03:17:22.566 INFO:teuthology.orchestra.run.smithi118.stdout: "tags": [], 2024-03-24T03:17:22.566 INFO:teuthology.orchestra.run.smithi118.stdout: "see_also": [] 2024-03-24T03:17:22.567 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-24T03:17:22.567 INFO:teuthology.orchestra.run.smithi118.stdout: "inventory_list_all": { 2024-03-24T03:17:22.567 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "inventory_list_all", 2024-03-24T03:17:22.567 INFO:teuthology.orchestra.run.smithi118.stdout: "type": "bool", 2024-03-24T03:17:22.567 INFO:teuthology.orchestra.run.smithi118.stdout: "level": "advanced", 2024-03-24T03:17:22.567 INFO:teuthology.orchestra.run.smithi118.stdout: "flags": 0, 2024-03-24T03:17:22.567 INFO:teuthology.orchestra.run.smithi118.stdout: "default_value": "False", 2024-03-24T03:17:22.567 INFO:teuthology.orchestra.run.smithi118.stdout: "min": "", 2024-03-24T03:17:22.568 INFO:teuthology.orchestra.run.smithi118.stdout: "max": "", 2024-03-24T03:17:22.568 INFO:teuthology.orchestra.run.smithi118.stdout: "enum_allowed": [], 2024-03-24T03:17:22.568 INFO:teuthology.orchestra.run.smithi118.stdout: "desc": "Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)", 2024-03-24T03:17:22.568 INFO:teuthology.orchestra.run.smithi118.stdout: "long_desc": "", 2024-03-24T03:17:22.568 INFO:teuthology.orchestra.run.smithi118.stdout: "tags": [], 2024-03-24T03:17:22.568 INFO:teuthology.orchestra.run.smithi118.stdout: "see_also": [] 2024-03-24T03:17:22.568 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-24T03:17:22.568 INFO:teuthology.orchestra.run.smithi118.stdout: "log_level": { 2024-03-24T03:17:22.569 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "log_level", 2024-03-24T03:17:22.569 INFO:teuthology.orchestra.run.smithi118.stdout: "type": "str", 2024-03-24T03:17:22.569 INFO:teuthology.orchestra.run.smithi118.stdout: "level": "advanced", 2024-03-24T03:17:22.569 INFO:teuthology.orchestra.run.smithi118.stdout: "flags": 1, 2024-03-24T03:17:22.569 INFO:teuthology.orchestra.run.smithi118.stdout: "default_value": "", 2024-03-24T03:17:22.569 INFO:teuthology.orchestra.run.smithi118.stdout: "min": "", 2024-03-24T03:17:22.569 INFO:teuthology.orchestra.run.smithi118.stdout: "max": "", 2024-03-24T03:17:22.569 INFO:teuthology.orchestra.run.smithi118.stdout: "enum_allowed": [ 2024-03-24T03:17:22.569 INFO:teuthology.orchestra.run.smithi118.stdout: "", 2024-03-24T03:17:22.569 INFO:teuthology.orchestra.run.smithi118.stdout: "critical", 2024-03-24T03:17:22.570 INFO:teuthology.orchestra.run.smithi118.stdout: "debug", 2024-03-24T03:17:22.570 INFO:teuthology.orchestra.run.smithi118.stdout: "error", 2024-03-24T03:17:22.570 INFO:teuthology.orchestra.run.smithi118.stdout: "info", 2024-03-24T03:17:22.570 INFO:teuthology.orchestra.run.smithi118.stdout: "warning" 2024-03-24T03:17:22.570 INFO:teuthology.orchestra.run.smithi118.stdout: ], 2024-03-24T03:17:22.570 INFO:teuthology.orchestra.run.smithi118.stdout: "desc": "", 2024-03-24T03:17:22.571 INFO:teuthology.orchestra.run.smithi118.stdout: "long_desc": "", 2024-03-24T03:17:22.571 INFO:teuthology.orchestra.run.smithi118.stdout: "tags": [], 2024-03-24T03:17:22.571 INFO:teuthology.orchestra.run.smithi118.stdout: "see_also": [] 2024-03-24T03:17:22.571 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-24T03:17:22.571 INFO:teuthology.orchestra.run.smithi118.stdout: "log_refresh_metadata": { 2024-03-24T03:17:22.571 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "log_refresh_metadata", 2024-03-24T03:17:22.571 INFO:teuthology.orchestra.run.smithi118.stdout: "type": "bool", 2024-03-24T03:17:22.571 INFO:teuthology.orchestra.run.smithi118.stdout: "level": "advanced", 2024-03-24T03:17:22.572 INFO:teuthology.orchestra.run.smithi118.stdout: "flags": 0, 2024-03-24T03:17:22.572 INFO:teuthology.orchestra.run.smithi118.stdout: "default_value": "False", 2024-03-24T03:17:22.572 INFO:teuthology.orchestra.run.smithi118.stdout: "min": "", 2024-03-24T03:17:22.572 INFO:teuthology.orchestra.run.smithi118.stdout: "max": "", 2024-03-24T03:17:22.572 INFO:teuthology.orchestra.run.smithi118.stdout: "enum_allowed": [], 2024-03-24T03:17:22.572 INFO:teuthology.orchestra.run.smithi118.stdout: "desc": "Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level", 2024-03-24T03:17:22.572 INFO:teuthology.orchestra.run.smithi118.stdout: "long_desc": "", 2024-03-24T03:17:22.572 INFO:teuthology.orchestra.run.smithi118.stdout: "tags": [], 2024-03-24T03:17:22.572 INFO:teuthology.orchestra.run.smithi118.stdout: "see_also": [] 2024-03-24T03:17:22.573 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-24T03:17:22.573 INFO:teuthology.orchestra.run.smithi118.stdout: "log_to_cluster": { 2024-03-24T03:17:22.573 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "log_to_cluster", 2024-03-24T03:17:22.573 INFO:teuthology.orchestra.run.smithi118.stdout: "type": "bool", 2024-03-24T03:17:22.573 INFO:teuthology.orchestra.run.smithi118.stdout: "level": "advanced", 2024-03-24T03:17:22.573 INFO:teuthology.orchestra.run.smithi118.stdout: "flags": 0, 2024-03-24T03:17:22.573 INFO:teuthology.orchestra.run.smithi118.stdout: "default_value": "True", 2024-03-24T03:17:22.573 INFO:teuthology.orchestra.run.smithi118.stdout: "min": "", 2024-03-24T03:17:22.573 INFO:teuthology.orchestra.run.smithi118.stdout: "max": "", 2024-03-24T03:17:22.574 INFO:teuthology.orchestra.run.smithi118.stdout: "enum_allowed": [], 2024-03-24T03:17:22.574 INFO:teuthology.orchestra.run.smithi118.stdout: "desc": "log to the \"cephadm\" cluster log channel\"", 2024-03-24T03:17:22.574 INFO:teuthology.orchestra.run.smithi118.stdout: "long_desc": "", 2024-03-24T03:17:22.574 INFO:teuthology.orchestra.run.smithi118.stdout: "tags": [], 2024-03-24T03:17:22.574 INFO:teuthology.orchestra.run.smithi118.stdout: "see_also": [] 2024-03-24T03:17:22.574 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-24T03:17:22.574 INFO:teuthology.orchestra.run.smithi118.stdout: "log_to_cluster_level": { 2024-03-24T03:17:22.574 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "log_to_cluster_level", 2024-03-24T03:17:22.574 INFO:teuthology.orchestra.run.smithi118.stdout: "type": "str", 2024-03-24T03:17:22.575 INFO:teuthology.orchestra.run.smithi118.stdout: "level": "advanced", 2024-03-24T03:17:22.575 INFO:teuthology.orchestra.run.smithi118.stdout: "flags": 1, 2024-03-24T03:17:22.575 INFO:teuthology.orchestra.run.smithi118.stdout: "default_value": "info", 2024-03-24T03:17:22.575 INFO:teuthology.orchestra.run.smithi118.stdout: "min": "", 2024-03-24T03:17:22.575 INFO:teuthology.orchestra.run.smithi118.stdout: "max": "", 2024-03-24T03:17:22.575 INFO:teuthology.orchestra.run.smithi118.stdout: "enum_allowed": [ 2024-03-24T03:17:22.575 INFO:teuthology.orchestra.run.smithi118.stdout: "", 2024-03-24T03:17:22.575 INFO:teuthology.orchestra.run.smithi118.stdout: "critical", 2024-03-24T03:17:22.576 INFO:teuthology.orchestra.run.smithi118.stdout: "debug", 2024-03-24T03:17:22.576 INFO:teuthology.orchestra.run.smithi118.stdout: "error", 2024-03-24T03:17:22.576 INFO:teuthology.orchestra.run.smithi118.stdout: "info", 2024-03-24T03:17:22.576 INFO:teuthology.orchestra.run.smithi118.stdout: "warning" 2024-03-24T03:17:22.576 INFO:teuthology.orchestra.run.smithi118.stdout: ], 2024-03-24T03:17:22.576 INFO:teuthology.orchestra.run.smithi118.stdout: "desc": "", 2024-03-24T03:17:22.576 INFO:teuthology.orchestra.run.smithi118.stdout: "long_desc": "", 2024-03-24T03:17:22.576 INFO:teuthology.orchestra.run.smithi118.stdout: "tags": [], 2024-03-24T03:17:22.576 INFO:teuthology.orchestra.run.smithi118.stdout: "see_also": [] 2024-03-24T03:17:22.576 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-24T03:17:22.577 INFO:teuthology.orchestra.run.smithi118.stdout: "log_to_file": { 2024-03-24T03:17:22.577 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "log_to_file", 2024-03-24T03:17:22.577 INFO:teuthology.orchestra.run.smithi118.stdout: "type": "bool", 2024-03-24T03:17:22.577 INFO:teuthology.orchestra.run.smithi118.stdout: "level": "advanced", 2024-03-24T03:17:22.577 INFO:teuthology.orchestra.run.smithi118.stdout: "flags": 1, 2024-03-24T03:17:22.577 INFO:teuthology.orchestra.run.smithi118.stdout: "default_value": "False", 2024-03-24T03:17:22.577 INFO:teuthology.orchestra.run.smithi118.stdout: "min": "", 2024-03-24T03:17:22.577 INFO:teuthology.orchestra.run.smithi118.stdout: "max": "", 2024-03-24T03:17:22.577 INFO:teuthology.orchestra.run.smithi118.stdout: "enum_allowed": [], 2024-03-24T03:17:22.577 INFO:teuthology.orchestra.run.smithi118.stdout: "desc": "", 2024-03-24T03:17:22.578 INFO:teuthology.orchestra.run.smithi118.stdout: "long_desc": "", 2024-03-24T03:17:22.578 INFO:teuthology.orchestra.run.smithi118.stdout: "tags": [], 2024-03-24T03:17:22.578 INFO:teuthology.orchestra.run.smithi118.stdout: "see_also": [] 2024-03-24T03:17:22.578 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-24T03:17:22.578 INFO:teuthology.orchestra.run.smithi118.stdout: "manage_etc_ceph_ceph_conf": { 2024-03-24T03:17:22.578 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "manage_etc_ceph_ceph_conf", 2024-03-24T03:17:22.578 INFO:teuthology.orchestra.run.smithi118.stdout: "type": "bool", 2024-03-24T03:17:22.578 INFO:teuthology.orchestra.run.smithi118.stdout: "level": "advanced", 2024-03-24T03:17:22.579 INFO:teuthology.orchestra.run.smithi118.stdout: "flags": 0, 2024-03-24T03:17:22.579 INFO:teuthology.orchestra.run.smithi118.stdout: "default_value": "False", 2024-03-24T03:17:22.579 INFO:teuthology.orchestra.run.smithi118.stdout: "min": "", 2024-03-24T03:17:22.579 INFO:teuthology.orchestra.run.smithi118.stdout: "max": "", 2024-03-24T03:17:22.579 INFO:teuthology.orchestra.run.smithi118.stdout: "enum_allowed": [], 2024-03-24T03:17:22.579 INFO:teuthology.orchestra.run.smithi118.stdout: "desc": "Manage and own /etc/ceph/ceph.conf on the hosts.", 2024-03-24T03:17:22.579 INFO:teuthology.orchestra.run.smithi118.stdout: "long_desc": "", 2024-03-24T03:17:22.579 INFO:teuthology.orchestra.run.smithi118.stdout: "tags": [], 2024-03-24T03:17:22.579 INFO:teuthology.orchestra.run.smithi118.stdout: "see_also": [] 2024-03-24T03:17:22.579 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-24T03:17:22.580 INFO:teuthology.orchestra.run.smithi118.stdout: "manage_etc_ceph_ceph_conf_hosts": { 2024-03-24T03:17:22.580 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "manage_etc_ceph_ceph_conf_hosts", 2024-03-24T03:17:22.580 INFO:teuthology.orchestra.run.smithi118.stdout: "type": "str", 2024-03-24T03:17:22.580 INFO:teuthology.orchestra.run.smithi118.stdout: "level": "advanced", 2024-03-24T03:17:22.580 INFO:teuthology.orchestra.run.smithi118.stdout: "flags": 0, 2024-03-24T03:17:22.580 INFO:teuthology.orchestra.run.smithi118.stdout: "default_value": "*", 2024-03-24T03:17:22.580 INFO:teuthology.orchestra.run.smithi118.stdout: "min": "", 2024-03-24T03:17:22.580 INFO:teuthology.orchestra.run.smithi118.stdout: "max": "", 2024-03-24T03:17:22.581 INFO:teuthology.orchestra.run.smithi118.stdout: "enum_allowed": [], 2024-03-24T03:17:22.581 INFO:teuthology.orchestra.run.smithi118.stdout: "desc": "PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf", 2024-03-24T03:17:22.581 INFO:teuthology.orchestra.run.smithi118.stdout: "long_desc": "", 2024-03-24T03:17:22.581 INFO:teuthology.orchestra.run.smithi118.stdout: "tags": [], 2024-03-24T03:17:22.581 INFO:teuthology.orchestra.run.smithi118.stdout: "see_also": [] 2024-03-24T03:17:22.581 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-24T03:17:22.581 INFO:teuthology.orchestra.run.smithi118.stdout: "max_count_per_host": { 2024-03-24T03:17:22.582 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "max_count_per_host", 2024-03-24T03:17:22.582 INFO:teuthology.orchestra.run.smithi118.stdout: "type": "int", 2024-03-24T03:17:22.582 INFO:teuthology.orchestra.run.smithi118.stdout: "level": "advanced", 2024-03-24T03:17:22.582 INFO:teuthology.orchestra.run.smithi118.stdout: "flags": 0, 2024-03-24T03:17:22.582 INFO:teuthology.orchestra.run.smithi118.stdout: "default_value": "10", 2024-03-24T03:17:22.582 INFO:teuthology.orchestra.run.smithi118.stdout: "min": "", 2024-03-24T03:17:22.582 INFO:teuthology.orchestra.run.smithi118.stdout: "max": "", 2024-03-24T03:17:22.582 INFO:teuthology.orchestra.run.smithi118.stdout: "enum_allowed": [], 2024-03-24T03:17:22.582 INFO:teuthology.orchestra.run.smithi118.stdout: "desc": "max number of daemons per service per host", 2024-03-24T03:17:22.582 INFO:teuthology.orchestra.run.smithi118.stdout: "long_desc": "", 2024-03-24T03:17:22.583 INFO:teuthology.orchestra.run.smithi118.stdout: "tags": [], 2024-03-24T03:17:22.583 INFO:teuthology.orchestra.run.smithi118.stdout: "see_also": [] 2024-03-24T03:17:22.583 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-24T03:17:22.583 INFO:teuthology.orchestra.run.smithi118.stdout: "max_osd_draining_count": { 2024-03-24T03:17:22.583 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "max_osd_draining_count", 2024-03-24T03:17:22.583 INFO:teuthology.orchestra.run.smithi118.stdout: "type": "int", 2024-03-24T03:17:22.583 INFO:teuthology.orchestra.run.smithi118.stdout: "level": "advanced", 2024-03-24T03:17:22.583 INFO:teuthology.orchestra.run.smithi118.stdout: "flags": 0, 2024-03-24T03:17:22.584 INFO:teuthology.orchestra.run.smithi118.stdout: "default_value": "10", 2024-03-24T03:17:22.584 INFO:teuthology.orchestra.run.smithi118.stdout: "min": "", 2024-03-24T03:17:22.584 INFO:teuthology.orchestra.run.smithi118.stdout: "max": "", 2024-03-24T03:17:22.584 INFO:teuthology.orchestra.run.smithi118.stdout: "enum_allowed": [], 2024-03-24T03:17:22.584 INFO:teuthology.orchestra.run.smithi118.stdout: "desc": "max number of osds that will be drained simultaneously when osds are removed", 2024-03-24T03:17:22.585 INFO:teuthology.orchestra.run.smithi118.stdout: "long_desc": "", 2024-03-24T03:17:22.585 INFO:teuthology.orchestra.run.smithi118.stdout: "tags": [], 2024-03-24T03:17:22.585 INFO:teuthology.orchestra.run.smithi118.stdout: "see_also": [] 2024-03-24T03:17:22.585 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-24T03:17:22.585 INFO:teuthology.orchestra.run.smithi118.stdout: "migration_current": { 2024-03-24T03:17:22.586 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "migration_current", 2024-03-24T03:17:22.586 INFO:teuthology.orchestra.run.smithi118.stdout: "type": "int", 2024-03-24T03:17:22.586 INFO:teuthology.orchestra.run.smithi118.stdout: "level": "advanced", 2024-03-24T03:17:22.586 INFO:teuthology.orchestra.run.smithi118.stdout: "flags": 0, 2024-03-24T03:17:22.587 INFO:teuthology.orchestra.run.smithi118.stdout: "default_value": "", 2024-03-24T03:17:22.587 INFO:teuthology.orchestra.run.smithi118.stdout: "min": "", 2024-03-24T03:17:22.587 INFO:teuthology.orchestra.run.smithi118.stdout: "max": "", 2024-03-24T03:17:22.587 INFO:teuthology.orchestra.run.smithi118.stdout: "enum_allowed": [], 2024-03-24T03:17:22.588 INFO:teuthology.orchestra.run.smithi118.stdout: "desc": "internal - do not modify", 2024-03-24T03:17:22.588 INFO:teuthology.orchestra.run.smithi118.stdout: "long_desc": "", 2024-03-24T03:17:22.588 INFO:teuthology.orchestra.run.smithi118.stdout: "tags": [], 2024-03-24T03:17:22.588 INFO:teuthology.orchestra.run.smithi118.stdout: "see_also": [] 2024-03-24T03:17:22.588 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-24T03:17:22.589 INFO:teuthology.orchestra.run.smithi118.stdout: "mode": { 2024-03-24T03:17:22.589 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "mode", 2024-03-24T03:17:22.589 INFO:teuthology.orchestra.run.smithi118.stdout: "type": "str", 2024-03-24T03:17:22.589 INFO:teuthology.orchestra.run.smithi118.stdout: "level": "advanced", 2024-03-24T03:17:22.590 INFO:teuthology.orchestra.run.smithi118.stdout: "flags": 0, 2024-03-24T03:17:22.590 INFO:teuthology.orchestra.run.smithi118.stdout: "default_value": "root", 2024-03-24T03:17:22.590 INFO:teuthology.orchestra.run.smithi118.stdout: "min": "", 2024-03-24T03:17:22.590 INFO:teuthology.orchestra.run.smithi118.stdout: "max": "", 2024-03-24T03:17:22.591 INFO:teuthology.orchestra.run.smithi118.stdout: "enum_allowed": [ 2024-03-24T03:17:22.591 INFO:teuthology.orchestra.run.smithi118.stdout: "cephadm-package", 2024-03-24T03:17:22.591 INFO:teuthology.orchestra.run.smithi118.stdout: "root" 2024-03-24T03:17:22.591 INFO:teuthology.orchestra.run.smithi118.stdout: ], 2024-03-24T03:17:22.592 INFO:teuthology.orchestra.run.smithi118.stdout: "desc": "mode for remote execution of cephadm", 2024-03-24T03:17:22.592 INFO:teuthology.orchestra.run.smithi118.stdout: "long_desc": "", 2024-03-24T03:17:22.592 INFO:teuthology.orchestra.run.smithi118.stdout: "tags": [], 2024-03-24T03:17:22.592 INFO:teuthology.orchestra.run.smithi118.stdout: "see_also": [] 2024-03-24T03:17:22.593 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-24T03:17:22.593 INFO:teuthology.orchestra.run.smithi118.stdout: "oob_default_addr": { 2024-03-24T03:17:22.593 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "oob_default_addr", 2024-03-24T03:17:22.593 INFO:teuthology.orchestra.run.smithi118.stdout: "type": "str", 2024-03-24T03:17:22.593 INFO:teuthology.orchestra.run.smithi118.stdout: "level": "advanced", 2024-03-24T03:17:22.594 INFO:teuthology.orchestra.run.smithi118.stdout: "flags": 0, 2024-03-24T03:17:22.594 INFO:teuthology.orchestra.run.smithi118.stdout: "default_value": "169.254.1.1", 2024-03-24T03:17:22.594 INFO:teuthology.orchestra.run.smithi118.stdout: "min": "", 2024-03-24T03:17:22.594 INFO:teuthology.orchestra.run.smithi118.stdout: "max": "", 2024-03-24T03:17:22.595 INFO:teuthology.orchestra.run.smithi118.stdout: "enum_allowed": [], 2024-03-24T03:17:22.595 INFO:teuthology.orchestra.run.smithi118.stdout: "desc": "Default address for RedFish API (oob management).", 2024-03-24T03:17:22.595 INFO:teuthology.orchestra.run.smithi118.stdout: "long_desc": "", 2024-03-24T03:17:22.595 INFO:teuthology.orchestra.run.smithi118.stdout: "tags": [], 2024-03-24T03:17:22.595 INFO:teuthology.orchestra.run.smithi118.stdout: "see_also": [] 2024-03-24T03:17:22.595 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-24T03:17:22.595 INFO:teuthology.orchestra.run.smithi118.stdout: "prometheus_alerts_path": { 2024-03-24T03:17:22.595 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "prometheus_alerts_path", 2024-03-24T03:17:22.595 INFO:teuthology.orchestra.run.smithi118.stdout: "type": "str", 2024-03-24T03:17:22.596 INFO:teuthology.orchestra.run.smithi118.stdout: "level": "advanced", 2024-03-24T03:17:22.596 INFO:teuthology.orchestra.run.smithi118.stdout: "flags": 0, 2024-03-24T03:17:22.596 INFO:teuthology.orchestra.run.smithi118.stdout: "default_value": "/etc/prometheus/ceph/ceph_default_alerts.yml", 2024-03-24T03:17:22.596 INFO:teuthology.orchestra.run.smithi118.stdout: "min": "", 2024-03-24T03:17:22.596 INFO:teuthology.orchestra.run.smithi118.stdout: "max": "", 2024-03-24T03:17:22.596 INFO:teuthology.orchestra.run.smithi118.stdout: "enum_allowed": [], 2024-03-24T03:17:22.596 INFO:teuthology.orchestra.run.smithi118.stdout: "desc": "location of alerts to include in prometheus deployments", 2024-03-24T03:17:22.596 INFO:teuthology.orchestra.run.smithi118.stdout: "long_desc": "", 2024-03-24T03:17:22.596 INFO:teuthology.orchestra.run.smithi118.stdout: "tags": [], 2024-03-24T03:17:22.597 INFO:teuthology.orchestra.run.smithi118.stdout: "see_also": [] 2024-03-24T03:17:22.597 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-24T03:17:22.597 INFO:teuthology.orchestra.run.smithi118.stdout: "registry_insecure": { 2024-03-24T03:17:22.597 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "registry_insecure", 2024-03-24T03:17:22.597 INFO:teuthology.orchestra.run.smithi118.stdout: "type": "bool", 2024-03-24T03:17:22.597 INFO:teuthology.orchestra.run.smithi118.stdout: "level": "advanced", 2024-03-24T03:17:22.597 INFO:teuthology.orchestra.run.smithi118.stdout: "flags": 0, 2024-03-24T03:17:22.597 INFO:teuthology.orchestra.run.smithi118.stdout: "default_value": "False", 2024-03-24T03:17:22.597 INFO:teuthology.orchestra.run.smithi118.stdout: "min": "", 2024-03-24T03:17:22.598 INFO:teuthology.orchestra.run.smithi118.stdout: "max": "", 2024-03-24T03:17:22.598 INFO:teuthology.orchestra.run.smithi118.stdout: "enum_allowed": [], 2024-03-24T03:17:22.598 INFO:teuthology.orchestra.run.smithi118.stdout: "desc": "Registry is to be considered insecure (no TLS available). Only for development purposes.", 2024-03-24T03:17:22.598 INFO:teuthology.orchestra.run.smithi118.stdout: "long_desc": "", 2024-03-24T03:17:22.598 INFO:teuthology.orchestra.run.smithi118.stdout: "tags": [], 2024-03-24T03:17:22.598 INFO:teuthology.orchestra.run.smithi118.stdout: "see_also": [] 2024-03-24T03:17:22.598 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-24T03:17:22.598 INFO:teuthology.orchestra.run.smithi118.stdout: "registry_password": { 2024-03-24T03:17:22.599 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "registry_password", 2024-03-24T03:17:22.599 INFO:teuthology.orchestra.run.smithi118.stdout: "type": "str", 2024-03-24T03:17:22.599 INFO:teuthology.orchestra.run.smithi118.stdout: "level": "advanced", 2024-03-24T03:17:22.599 INFO:teuthology.orchestra.run.smithi118.stdout: "flags": 0, 2024-03-24T03:17:22.599 INFO:teuthology.orchestra.run.smithi118.stdout: "default_value": "", 2024-03-24T03:17:22.599 INFO:teuthology.orchestra.run.smithi118.stdout: "min": "", 2024-03-24T03:17:22.599 INFO:teuthology.orchestra.run.smithi118.stdout: "max": "", 2024-03-24T03:17:22.599 INFO:teuthology.orchestra.run.smithi118.stdout: "enum_allowed": [], 2024-03-24T03:17:22.599 INFO:teuthology.orchestra.run.smithi118.stdout: "desc": "Custom repository password. Only used for logging into a registry.", 2024-03-24T03:17:22.600 INFO:teuthology.orchestra.run.smithi118.stdout: "long_desc": "", 2024-03-24T03:17:22.600 INFO:teuthology.orchestra.run.smithi118.stdout: "tags": [], 2024-03-24T03:17:22.600 INFO:teuthology.orchestra.run.smithi118.stdout: "see_also": [] 2024-03-24T03:17:22.600 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-24T03:17:22.600 INFO:teuthology.orchestra.run.smithi118.stdout: "registry_url": { 2024-03-24T03:17:22.600 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "registry_url", 2024-03-24T03:17:22.600 INFO:teuthology.orchestra.run.smithi118.stdout: "type": "str", 2024-03-24T03:17:22.600 INFO:teuthology.orchestra.run.smithi118.stdout: "level": "advanced", 2024-03-24T03:17:22.600 INFO:teuthology.orchestra.run.smithi118.stdout: "flags": 0, 2024-03-24T03:17:22.601 INFO:teuthology.orchestra.run.smithi118.stdout: "default_value": "", 2024-03-24T03:17:22.601 INFO:teuthology.orchestra.run.smithi118.stdout: "min": "", 2024-03-24T03:17:22.601 INFO:teuthology.orchestra.run.smithi118.stdout: "max": "", 2024-03-24T03:17:22.601 INFO:teuthology.orchestra.run.smithi118.stdout: "enum_allowed": [], 2024-03-24T03:17:22.601 INFO:teuthology.orchestra.run.smithi118.stdout: "desc": "Registry url for login purposes. This is not the default registry", 2024-03-24T03:17:22.601 INFO:teuthology.orchestra.run.smithi118.stdout: "long_desc": "", 2024-03-24T03:17:22.601 INFO:teuthology.orchestra.run.smithi118.stdout: "tags": [], 2024-03-24T03:17:22.601 INFO:teuthology.orchestra.run.smithi118.stdout: "see_also": [] 2024-03-24T03:17:22.602 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-24T03:17:22.602 INFO:teuthology.orchestra.run.smithi118.stdout: "registry_username": { 2024-03-24T03:17:22.602 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "registry_username", 2024-03-24T03:17:22.602 INFO:teuthology.orchestra.run.smithi118.stdout: "type": "str", 2024-03-24T03:17:22.602 INFO:teuthology.orchestra.run.smithi118.stdout: "level": "advanced", 2024-03-24T03:17:22.602 INFO:teuthology.orchestra.run.smithi118.stdout: "flags": 0, 2024-03-24T03:17:22.602 INFO:teuthology.orchestra.run.smithi118.stdout: "default_value": "", 2024-03-24T03:17:22.602 INFO:teuthology.orchestra.run.smithi118.stdout: "min": "", 2024-03-24T03:17:22.603 INFO:teuthology.orchestra.run.smithi118.stdout: "max": "", 2024-03-24T03:17:22.603 INFO:teuthology.orchestra.run.smithi118.stdout: "enum_allowed": [], 2024-03-24T03:17:22.603 INFO:teuthology.orchestra.run.smithi118.stdout: "desc": "Custom repository username. Only used for logging into a registry.", 2024-03-24T03:17:22.603 INFO:teuthology.orchestra.run.smithi118.stdout: "long_desc": "", 2024-03-24T03:17:22.603 INFO:teuthology.orchestra.run.smithi118.stdout: "tags": [], 2024-03-24T03:17:22.603 INFO:teuthology.orchestra.run.smithi118.stdout: "see_also": [] 2024-03-24T03:17:22.603 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-24T03:17:22.603 INFO:teuthology.orchestra.run.smithi118.stdout: "secure_monitoring_stack": { 2024-03-24T03:17:22.603 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "secure_monitoring_stack", 2024-03-24T03:17:22.604 INFO:teuthology.orchestra.run.smithi118.stdout: "type": "bool", 2024-03-24T03:17:22.604 INFO:teuthology.orchestra.run.smithi118.stdout: "level": "advanced", 2024-03-24T03:17:22.604 INFO:teuthology.orchestra.run.smithi118.stdout: "flags": 0, 2024-03-24T03:17:22.604 INFO:teuthology.orchestra.run.smithi118.stdout: "default_value": "False", 2024-03-24T03:17:22.604 INFO:teuthology.orchestra.run.smithi118.stdout: "min": "", 2024-03-24T03:17:22.604 INFO:teuthology.orchestra.run.smithi118.stdout: "max": "", 2024-03-24T03:17:22.604 INFO:teuthology.orchestra.run.smithi118.stdout: "enum_allowed": [], 2024-03-24T03:17:22.605 INFO:teuthology.orchestra.run.smithi118.stdout: "desc": "Enable TLS security for all the monitoring stack daemons", 2024-03-24T03:17:22.605 INFO:teuthology.orchestra.run.smithi118.stdout: "long_desc": "", 2024-03-24T03:17:22.605 INFO:teuthology.orchestra.run.smithi118.stdout: "tags": [], 2024-03-24T03:17:22.605 INFO:teuthology.orchestra.run.smithi118.stdout: "see_also": [] 2024-03-24T03:17:22.605 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-24T03:17:22.605 INFO:teuthology.orchestra.run.smithi118.stdout: "service_discovery_port": { 2024-03-24T03:17:22.605 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "service_discovery_port", 2024-03-24T03:17:22.605 INFO:teuthology.orchestra.run.smithi118.stdout: "type": "int", 2024-03-24T03:17:22.605 INFO:teuthology.orchestra.run.smithi118.stdout: "level": "advanced", 2024-03-24T03:17:22.605 INFO:teuthology.orchestra.run.smithi118.stdout: "flags": 0, 2024-03-24T03:17:22.606 INFO:teuthology.orchestra.run.smithi118.stdout: "default_value": "8765", 2024-03-24T03:17:22.606 INFO:teuthology.orchestra.run.smithi118.stdout: "min": "", 2024-03-24T03:17:22.606 INFO:teuthology.orchestra.run.smithi118.stdout: "max": "", 2024-03-24T03:17:22.606 INFO:teuthology.orchestra.run.smithi118.stdout: "enum_allowed": [], 2024-03-24T03:17:22.606 INFO:teuthology.orchestra.run.smithi118.stdout: "desc": "cephadm service discovery port", 2024-03-24T03:17:22.606 INFO:teuthology.orchestra.run.smithi118.stdout: "long_desc": "", 2024-03-24T03:17:22.606 INFO:teuthology.orchestra.run.smithi118.stdout: "tags": [], 2024-03-24T03:17:22.606 INFO:teuthology.orchestra.run.smithi118.stdout: "see_also": [] 2024-03-24T03:17:22.607 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-24T03:17:22.607 INFO:teuthology.orchestra.run.smithi118.stdout: "ssh_config_file": { 2024-03-24T03:17:22.607 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "ssh_config_file", 2024-03-24T03:17:22.607 INFO:teuthology.orchestra.run.smithi118.stdout: "type": "str", 2024-03-24T03:17:22.607 INFO:teuthology.orchestra.run.smithi118.stdout: "level": "advanced", 2024-03-24T03:17:22.607 INFO:teuthology.orchestra.run.smithi118.stdout: "flags": 0, 2024-03-24T03:17:22.607 INFO:teuthology.orchestra.run.smithi118.stdout: "default_value": "", 2024-03-24T03:17:22.607 INFO:teuthology.orchestra.run.smithi118.stdout: "min": "", 2024-03-24T03:17:22.607 INFO:teuthology.orchestra.run.smithi118.stdout: "max": "", 2024-03-24T03:17:22.608 INFO:teuthology.orchestra.run.smithi118.stdout: "enum_allowed": [], 2024-03-24T03:17:22.608 INFO:teuthology.orchestra.run.smithi118.stdout: "desc": "customized SSH config file to connect to managed hosts", 2024-03-24T03:17:22.608 INFO:teuthology.orchestra.run.smithi118.stdout: "long_desc": "", 2024-03-24T03:17:22.608 INFO:teuthology.orchestra.run.smithi118.stdout: "tags": [], 2024-03-24T03:17:22.608 INFO:teuthology.orchestra.run.smithi118.stdout: "see_also": [] 2024-03-24T03:17:22.608 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-24T03:17:22.608 INFO:teuthology.orchestra.run.smithi118.stdout: "use_agent": { 2024-03-24T03:17:22.608 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "use_agent", 2024-03-24T03:17:22.609 INFO:teuthology.orchestra.run.smithi118.stdout: "type": "bool", 2024-03-24T03:17:22.609 INFO:teuthology.orchestra.run.smithi118.stdout: "level": "advanced", 2024-03-24T03:17:22.609 INFO:teuthology.orchestra.run.smithi118.stdout: "flags": 0, 2024-03-24T03:17:22.609 INFO:teuthology.orchestra.run.smithi118.stdout: "default_value": "False", 2024-03-24T03:17:22.609 INFO:teuthology.orchestra.run.smithi118.stdout: "min": "", 2024-03-24T03:17:22.609 INFO:teuthology.orchestra.run.smithi118.stdout: "max": "", 2024-03-24T03:17:22.609 INFO:teuthology.orchestra.run.smithi118.stdout: "enum_allowed": [], 2024-03-24T03:17:22.609 INFO:teuthology.orchestra.run.smithi118.stdout: "desc": "Use cephadm agent on each host to gather and send metadata", 2024-03-24T03:17:22.609 INFO:teuthology.orchestra.run.smithi118.stdout: "long_desc": "", 2024-03-24T03:17:22.610 INFO:teuthology.orchestra.run.smithi118.stdout: "tags": [], 2024-03-24T03:17:22.610 INFO:teuthology.orchestra.run.smithi118.stdout: "see_also": [] 2024-03-24T03:17:22.610 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-24T03:17:22.610 INFO:teuthology.orchestra.run.smithi118.stdout: "use_repo_digest": { 2024-03-24T03:17:22.610 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "use_repo_digest", 2024-03-24T03:17:22.610 INFO:teuthology.orchestra.run.smithi118.stdout: "type": "bool", 2024-03-24T03:17:22.610 INFO:teuthology.orchestra.run.smithi118.stdout: "level": "advanced", 2024-03-24T03:17:22.611 INFO:teuthology.orchestra.run.smithi118.stdout: "flags": 0, 2024-03-24T03:17:22.611 INFO:teuthology.orchestra.run.smithi118.stdout: "default_value": "True", 2024-03-24T03:17:22.611 INFO:teuthology.orchestra.run.smithi118.stdout: "min": "", 2024-03-24T03:17:22.611 INFO:teuthology.orchestra.run.smithi118.stdout: "max": "", 2024-03-24T03:17:22.611 INFO:teuthology.orchestra.run.smithi118.stdout: "enum_allowed": [], 2024-03-24T03:17:22.611 INFO:teuthology.orchestra.run.smithi118.stdout: "desc": "Automatically convert image tags to image digest. Make sure all daemons use the same image", 2024-03-24T03:17:22.611 INFO:teuthology.orchestra.run.smithi118.stdout: "long_desc": "", 2024-03-24T03:17:22.611 INFO:teuthology.orchestra.run.smithi118.stdout: "tags": [], 2024-03-24T03:17:22.611 INFO:teuthology.orchestra.run.smithi118.stdout: "see_also": [] 2024-03-24T03:17:22.612 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-24T03:17:22.612 INFO:teuthology.orchestra.run.smithi118.stdout: "warn_on_failed_host_check": { 2024-03-24T03:17:22.612 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "warn_on_failed_host_check", 2024-03-24T03:17:22.612 INFO:teuthology.orchestra.run.smithi118.stdout: "type": "bool", 2024-03-24T03:17:22.612 INFO:teuthology.orchestra.run.smithi118.stdout: "level": "advanced", 2024-03-24T03:17:22.612 INFO:teuthology.orchestra.run.smithi118.stdout: "flags": 0, 2024-03-24T03:17:22.612 INFO:teuthology.orchestra.run.smithi118.stdout: "default_value": "True", 2024-03-24T03:17:22.612 INFO:teuthology.orchestra.run.smithi118.stdout: "min": "", 2024-03-24T03:17:22.613 INFO:teuthology.orchestra.run.smithi118.stdout: "max": "", 2024-03-24T03:17:22.613 INFO:teuthology.orchestra.run.smithi118.stdout: "enum_allowed": [], 2024-03-24T03:17:22.613 INFO:teuthology.orchestra.run.smithi118.stdout: "desc": "raise a health warning if the host check fails", 2024-03-24T03:17:22.613 INFO:teuthology.orchestra.run.smithi118.stdout: "long_desc": "", 2024-03-24T03:17:22.613 INFO:teuthology.orchestra.run.smithi118.stdout: "tags": [], 2024-03-24T03:17:22.613 INFO:teuthology.orchestra.run.smithi118.stdout: "see_also": [] 2024-03-24T03:17:22.613 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-24T03:17:22.614 INFO:teuthology.orchestra.run.smithi118.stdout: "warn_on_stray_daemons": { 2024-03-24T03:17:22.614 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "warn_on_stray_daemons", 2024-03-24T03:17:22.614 INFO:teuthology.orchestra.run.smithi118.stdout: "type": "bool", 2024-03-24T03:17:22.614 INFO:teuthology.orchestra.run.smithi118.stdout: "level": "advanced", 2024-03-24T03:17:22.614 INFO:teuthology.orchestra.run.smithi118.stdout: "flags": 0, 2024-03-24T03:17:22.614 INFO:teuthology.orchestra.run.smithi118.stdout: "default_value": "True", 2024-03-24T03:17:22.614 INFO:teuthology.orchestra.run.smithi118.stdout: "min": "", 2024-03-24T03:17:22.615 INFO:teuthology.orchestra.run.smithi118.stdout: "max": "", 2024-03-24T03:17:22.615 INFO:teuthology.orchestra.run.smithi118.stdout: "enum_allowed": [], 2024-03-24T03:17:22.615 INFO:teuthology.orchestra.run.smithi118.stdout: "desc": "raise a health warning if daemons are detected that are not managed by cephadm", 2024-03-24T03:17:22.615 INFO:teuthology.orchestra.run.smithi118.stdout: "long_desc": "", 2024-03-24T03:17:22.615 INFO:teuthology.orchestra.run.smithi118.stdout: "tags": [], 2024-03-24T03:17:22.615 INFO:teuthology.orchestra.run.smithi118.stdout: "see_also": [] 2024-03-24T03:17:22.615 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-24T03:17:22.615 INFO:teuthology.orchestra.run.smithi118.stdout: "warn_on_stray_hosts": { 2024-03-24T03:17:22.616 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "warn_on_stray_hosts", 2024-03-24T03:17:22.616 INFO:teuthology.orchestra.run.smithi118.stdout: "type": "bool", 2024-03-24T03:17:22.616 INFO:teuthology.orchestra.run.smithi118.stdout: "level": "advanced", 2024-03-24T03:17:22.616 INFO:teuthology.orchestra.run.smithi118.stdout: "flags": 0, 2024-03-24T03:17:22.616 INFO:teuthology.orchestra.run.smithi118.stdout: "default_value": "True", 2024-03-24T03:17:22.616 INFO:teuthology.orchestra.run.smithi118.stdout: "min": "", 2024-03-24T03:17:22.616 INFO:teuthology.orchestra.run.smithi118.stdout: "max": "", 2024-03-24T03:17:22.616 INFO:teuthology.orchestra.run.smithi118.stdout: "enum_allowed": [], 2024-03-24T03:17:22.617 INFO:teuthology.orchestra.run.smithi118.stdout: "desc": "raise a health warning if daemons are detected on a host that is not managed by cephadm", 2024-03-24T03:17:22.617 INFO:teuthology.orchestra.run.smithi118.stdout: "long_desc": "", 2024-03-24T03:17:22.617 INFO:teuthology.orchestra.run.smithi118.stdout: "tags": [], 2024-03-24T03:17:22.617 INFO:teuthology.orchestra.run.smithi118.stdout: "see_also": [] 2024-03-24T03:17:22.617 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-03-24T03:17:22.617 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-03-24T03:17:22.617 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-24T03:17:22.617 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-24T03:17:22.618 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "crash", 2024-03-24T03:17:22.618 INFO:teuthology.orchestra.run.smithi118.stdout: "can_run": true, 2024-03-24T03:17:22.618 INFO:teuthology.orchestra.run.smithi118.stdout: "error_string": "", 2024-03-24T03:17:22.618 INFO:teuthology.orchestra.run.smithi118.stdout: "module_options": { 2024-03-24T03:17:22.618 INFO:teuthology.orchestra.run.smithi118.stdout: "log_level": { 2024-03-24T03:17:22.618 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "log_level", 2024-03-24T03:17:22.618 INFO:teuthology.orchestra.run.smithi118.stdout: "type": "str", 2024-03-24T03:17:22.618 INFO:teuthology.orchestra.run.smithi118.stdout: "level": "advanced", 2024-03-24T03:17:22.619 INFO:teuthology.orchestra.run.smithi118.stdout: "flags": 1, 2024-03-24T03:17:22.619 INFO:teuthology.orchestra.run.smithi118.stdout: "default_value": "", 2024-03-24T03:17:22.619 INFO:teuthology.orchestra.run.smithi118.stdout: "min": "", 2024-03-24T03:17:22.619 INFO:teuthology.orchestra.run.smithi118.stdout: "max": "", 2024-03-24T03:17:22.619 INFO:teuthology.orchestra.run.smithi118.stdout: "enum_allowed": [ 2024-03-24T03:17:22.619 INFO:teuthology.orchestra.run.smithi118.stdout: "", 2024-03-24T03:17:22.619 INFO:teuthology.orchestra.run.smithi118.stdout: "critical", 2024-03-24T03:17:22.619 INFO:teuthology.orchestra.run.smithi118.stdout: "debug", 2024-03-24T03:17:22.620 INFO:teuthology.orchestra.run.smithi118.stdout: "error", 2024-03-24T03:17:22.620 INFO:teuthology.orchestra.run.smithi118.stdout: "info", 2024-03-24T03:17:22.620 INFO:teuthology.orchestra.run.smithi118.stdout: "warning" 2024-03-24T03:17:22.620 INFO:teuthology.orchestra.run.smithi118.stdout: ], 2024-03-24T03:17:22.620 INFO:teuthology.orchestra.run.smithi118.stdout: "desc": "", 2024-03-24T03:17:22.620 INFO:teuthology.orchestra.run.smithi118.stdout: "long_desc": "", 2024-03-24T03:17:22.620 INFO:teuthology.orchestra.run.smithi118.stdout: "tags": [], 2024-03-24T03:17:22.620 INFO:teuthology.orchestra.run.smithi118.stdout: "see_also": [] 2024-03-24T03:17:22.620 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-24T03:17:22.621 INFO:teuthology.orchestra.run.smithi118.stdout: "log_to_cluster": { 2024-03-24T03:17:22.621 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "log_to_cluster", 2024-03-24T03:17:22.621 INFO:teuthology.orchestra.run.smithi118.stdout: "type": "bool", 2024-03-24T03:17:22.621 INFO:teuthology.orchestra.run.smithi118.stdout: "level": "advanced", 2024-03-24T03:17:22.621 INFO:teuthology.orchestra.run.smithi118.stdout: "flags": 1, 2024-03-24T03:17:22.621 INFO:teuthology.orchestra.run.smithi118.stdout: "default_value": "False", 2024-03-24T03:17:22.621 INFO:teuthology.orchestra.run.smithi118.stdout: "min": "", 2024-03-24T03:17:22.622 INFO:teuthology.orchestra.run.smithi118.stdout: "max": "", 2024-03-24T03:17:22.622 INFO:teuthology.orchestra.run.smithi118.stdout: "enum_allowed": [], 2024-03-24T03:17:22.622 INFO:teuthology.orchestra.run.smithi118.stdout: "desc": "", 2024-03-24T03:17:22.622 INFO:teuthology.orchestra.run.smithi118.stdout: "long_desc": "", 2024-03-24T03:17:22.622 INFO:teuthology.orchestra.run.smithi118.stdout: "tags": [], 2024-03-24T03:17:22.622 INFO:teuthology.orchestra.run.smithi118.stdout: "see_also": [] 2024-03-24T03:17:22.622 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-24T03:17:22.622 INFO:teuthology.orchestra.run.smithi118.stdout: "log_to_cluster_level": { 2024-03-24T03:17:22.623 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "log_to_cluster_level", 2024-03-24T03:17:22.623 INFO:teuthology.orchestra.run.smithi118.stdout: "type": "str", 2024-03-24T03:17:22.623 INFO:teuthology.orchestra.run.smithi118.stdout: "level": "advanced", 2024-03-24T03:17:22.623 INFO:teuthology.orchestra.run.smithi118.stdout: "flags": 1, 2024-03-24T03:17:22.623 INFO:teuthology.orchestra.run.smithi118.stdout: "default_value": "info", 2024-03-24T03:17:22.623 INFO:teuthology.orchestra.run.smithi118.stdout: "min": "", 2024-03-24T03:17:22.623 INFO:teuthology.orchestra.run.smithi118.stdout: "max": "", 2024-03-24T03:17:22.623 INFO:teuthology.orchestra.run.smithi118.stdout: "enum_allowed": [ 2024-03-24T03:17:22.623 INFO:teuthology.orchestra.run.smithi118.stdout: "", 2024-03-24T03:17:22.624 INFO:teuthology.orchestra.run.smithi118.stdout: "critical", 2024-03-24T03:17:22.624 INFO:teuthology.orchestra.run.smithi118.stdout: "debug", 2024-03-24T03:17:22.624 INFO:teuthology.orchestra.run.smithi118.stdout: "error", 2024-03-24T03:17:22.624 INFO:teuthology.orchestra.run.smithi118.stdout: "info", 2024-03-24T03:17:22.624 INFO:teuthology.orchestra.run.smithi118.stdout: "warning" 2024-03-24T03:17:22.624 INFO:teuthology.orchestra.run.smithi118.stdout: ], 2024-03-24T03:17:22.624 INFO:teuthology.orchestra.run.smithi118.stdout: "desc": "", 2024-03-24T03:17:22.624 INFO:teuthology.orchestra.run.smithi118.stdout: "long_desc": "", 2024-03-24T03:17:22.625 INFO:teuthology.orchestra.run.smithi118.stdout: "tags": [], 2024-03-24T03:17:22.625 INFO:teuthology.orchestra.run.smithi118.stdout: "see_also": [] 2024-03-24T03:17:22.625 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-24T03:17:22.625 INFO:teuthology.orchestra.run.smithi118.stdout: "log_to_file": { 2024-03-24T03:17:22.625 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "log_to_file", 2024-03-24T03:17:22.625 INFO:teuthology.orchestra.run.smithi118.stdout: "type": "bool", 2024-03-24T03:17:22.625 INFO:teuthology.orchestra.run.smithi118.stdout: "level": "advanced", 2024-03-24T03:17:22.625 INFO:teuthology.orchestra.run.smithi118.stdout: "flags": 1, 2024-03-24T03:17:22.625 INFO:teuthology.orchestra.run.smithi118.stdout: "default_value": "False", 2024-03-24T03:17:22.626 INFO:teuthology.orchestra.run.smithi118.stdout: "min": "", 2024-03-24T03:17:22.626 INFO:teuthology.orchestra.run.smithi118.stdout: "max": "", 2024-03-24T03:17:22.626 INFO:teuthology.orchestra.run.smithi118.stdout: "enum_allowed": [], 2024-03-24T03:17:22.626 INFO:teuthology.orchestra.run.smithi118.stdout: "desc": "", 2024-03-24T03:17:22.626 INFO:teuthology.orchestra.run.smithi118.stdout: "long_desc": "", 2024-03-24T03:17:22.626 INFO:teuthology.orchestra.run.smithi118.stdout: "tags": [], 2024-03-24T03:17:22.626 INFO:teuthology.orchestra.run.smithi118.stdout: "see_also": [] 2024-03-24T03:17:22.626 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-24T03:17:22.627 INFO:teuthology.orchestra.run.smithi118.stdout: "retain_interval": { 2024-03-24T03:17:22.627 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "retain_interval", 2024-03-24T03:17:22.627 INFO:teuthology.orchestra.run.smithi118.stdout: "type": "secs", 2024-03-24T03:17:22.627 INFO:teuthology.orchestra.run.smithi118.stdout: "level": "advanced", 2024-03-24T03:17:22.627 INFO:teuthology.orchestra.run.smithi118.stdout: "flags": 1, 2024-03-24T03:17:22.627 INFO:teuthology.orchestra.run.smithi118.stdout: "default_value": "31536000", 2024-03-24T03:17:22.627 INFO:teuthology.orchestra.run.smithi118.stdout: "min": "", 2024-03-24T03:17:22.627 INFO:teuthology.orchestra.run.smithi118.stdout: "max": "", 2024-03-24T03:17:22.628 INFO:teuthology.orchestra.run.smithi118.stdout: "enum_allowed": [], 2024-03-24T03:17:22.628 INFO:teuthology.orchestra.run.smithi118.stdout: "desc": "how long to retain crashes before pruning them", 2024-03-24T03:17:22.628 INFO:teuthology.orchestra.run.smithi118.stdout: "long_desc": "", 2024-03-24T03:17:22.628 INFO:teuthology.orchestra.run.smithi118.stdout: "tags": [], 2024-03-24T03:17:22.628 INFO:teuthology.orchestra.run.smithi118.stdout: "see_also": [] 2024-03-24T03:17:22.628 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-24T03:17:22.628 INFO:teuthology.orchestra.run.smithi118.stdout: "warn_recent_interval": { 2024-03-24T03:17:22.628 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "warn_recent_interval", 2024-03-24T03:17:22.628 INFO:teuthology.orchestra.run.smithi118.stdout: "type": "secs", 2024-03-24T03:17:22.629 INFO:teuthology.orchestra.run.smithi118.stdout: "level": "advanced", 2024-03-24T03:17:22.629 INFO:teuthology.orchestra.run.smithi118.stdout: "flags": 1, 2024-03-24T03:17:22.629 INFO:teuthology.orchestra.run.smithi118.stdout: "default_value": "1209600", 2024-03-24T03:17:22.629 INFO:teuthology.orchestra.run.smithi118.stdout: "min": "", 2024-03-24T03:17:22.629 INFO:teuthology.orchestra.run.smithi118.stdout: "max": "", 2024-03-24T03:17:22.629 INFO:teuthology.orchestra.run.smithi118.stdout: "enum_allowed": [], 2024-03-24T03:17:22.629 INFO:teuthology.orchestra.run.smithi118.stdout: "desc": "time interval in which to warn about recent crashes", 2024-03-24T03:17:22.629 INFO:teuthology.orchestra.run.smithi118.stdout: "long_desc": "", 2024-03-24T03:17:22.630 INFO:teuthology.orchestra.run.smithi118.stdout: "tags": [], 2024-03-24T03:17:22.630 INFO:teuthology.orchestra.run.smithi118.stdout: "see_also": [] 2024-03-24T03:17:22.630 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-03-24T03:17:22.630 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-03-24T03:17:22.630 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-24T03:17:22.630 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-24T03:17:22.630 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "dashboard", 2024-03-24T03:17:22.631 INFO:teuthology.orchestra.run.smithi118.stdout: "can_run": true, 2024-03-24T03:17:22.631 INFO:teuthology.orchestra.run.smithi118.stdout: "error_string": "", 2024-03-24T03:17:22.631 INFO:teuthology.orchestra.run.smithi118.stdout: "module_options": { 2024-03-24T03:17:22.631 INFO:teuthology.orchestra.run.smithi118.stdout: "ACCOUNT_LOCKOUT_ATTEMPTS": { 2024-03-24T03:17:22.631 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "ACCOUNT_LOCKOUT_ATTEMPTS", 2024-03-24T03:17:22.631 INFO:teuthology.orchestra.run.smithi118.stdout: "type": "int", 2024-03-24T03:17:22.631 INFO:teuthology.orchestra.run.smithi118.stdout: "level": "advanced", 2024-03-24T03:17:22.631 INFO:teuthology.orchestra.run.smithi118.stdout: "flags": 0, 2024-03-24T03:17:22.632 INFO:teuthology.orchestra.run.smithi118.stdout: "default_value": "10", 2024-03-24T03:17:22.632 INFO:teuthology.orchestra.run.smithi118.stdout: "min": "", 2024-03-24T03:17:22.632 INFO:teuthology.orchestra.run.smithi118.stdout: "max": "", 2024-03-24T03:17:22.632 INFO:teuthology.orchestra.run.smithi118.stdout: "enum_allowed": [], 2024-03-24T03:17:22.632 INFO:teuthology.orchestra.run.smithi118.stdout: "desc": "", 2024-03-24T03:17:22.632 INFO:teuthology.orchestra.run.smithi118.stdout: "long_desc": "", 2024-03-24T03:17:22.632 INFO:teuthology.orchestra.run.smithi118.stdout: "tags": [], 2024-03-24T03:17:22.633 INFO:teuthology.orchestra.run.smithi118.stdout: "see_also": [] 2024-03-24T03:17:22.633 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-24T03:17:22.633 INFO:teuthology.orchestra.run.smithi118.stdout: "ALERTMANAGER_API_HOST": { 2024-03-24T03:17:22.633 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "ALERTMANAGER_API_HOST", 2024-03-24T03:17:22.633 INFO:teuthology.orchestra.run.smithi118.stdout: "type": "str", 2024-03-24T03:17:22.633 INFO:teuthology.orchestra.run.smithi118.stdout: "level": "advanced", 2024-03-24T03:17:22.633 INFO:teuthology.orchestra.run.smithi118.stdout: "flags": 0, 2024-03-24T03:17:22.633 INFO:teuthology.orchestra.run.smithi118.stdout: "default_value": "", 2024-03-24T03:17:22.633 INFO:teuthology.orchestra.run.smithi118.stdout: "min": "", 2024-03-24T03:17:22.634 INFO:teuthology.orchestra.run.smithi118.stdout: "max": "", 2024-03-24T03:17:22.634 INFO:teuthology.orchestra.run.smithi118.stdout: "enum_allowed": [], 2024-03-24T03:17:22.634 INFO:teuthology.orchestra.run.smithi118.stdout: "desc": "", 2024-03-24T03:17:22.634 INFO:teuthology.orchestra.run.smithi118.stdout: "long_desc": "", 2024-03-24T03:17:22.634 INFO:teuthology.orchestra.run.smithi118.stdout: "tags": [], 2024-03-24T03:17:22.634 INFO:teuthology.orchestra.run.smithi118.stdout: "see_also": [] 2024-03-24T03:17:22.634 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-24T03:17:22.635 INFO:teuthology.orchestra.run.smithi118.stdout: "ALERTMANAGER_API_SSL_VERIFY": { 2024-03-24T03:17:22.635 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "ALERTMANAGER_API_SSL_VERIFY", 2024-03-24T03:17:22.635 INFO:teuthology.orchestra.run.smithi118.stdout: "type": "bool", 2024-03-24T03:17:22.635 INFO:teuthology.orchestra.run.smithi118.stdout: "level": "advanced", 2024-03-24T03:17:22.635 INFO:teuthology.orchestra.run.smithi118.stdout: "flags": 0, 2024-03-24T03:17:22.635 INFO:teuthology.orchestra.run.smithi118.stdout: "default_value": "True", 2024-03-24T03:17:22.635 INFO:teuthology.orchestra.run.smithi118.stdout: "min": "", 2024-03-24T03:17:22.635 INFO:teuthology.orchestra.run.smithi118.stdout: "max": "", 2024-03-24T03:17:22.635 INFO:teuthology.orchestra.run.smithi118.stdout: "enum_allowed": [], 2024-03-24T03:17:22.635 INFO:teuthology.orchestra.run.smithi118.stdout: "desc": "", 2024-03-24T03:17:22.636 INFO:teuthology.orchestra.run.smithi118.stdout: "long_desc": "", 2024-03-24T03:17:22.636 INFO:teuthology.orchestra.run.smithi118.stdout: "tags": [], 2024-03-24T03:17:22.636 INFO:teuthology.orchestra.run.smithi118.stdout: "see_also": [] 2024-03-24T03:17:22.636 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-24T03:17:22.636 INFO:teuthology.orchestra.run.smithi118.stdout: "AUDIT_API_ENABLED": { 2024-03-24T03:17:22.636 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "AUDIT_API_ENABLED", 2024-03-24T03:17:22.636 INFO:teuthology.orchestra.run.smithi118.stdout: "type": "bool", 2024-03-24T03:17:22.636 INFO:teuthology.orchestra.run.smithi118.stdout: "level": "advanced", 2024-03-24T03:17:22.637 INFO:teuthology.orchestra.run.smithi118.stdout: "flags": 0, 2024-03-24T03:17:22.637 INFO:teuthology.orchestra.run.smithi118.stdout: "default_value": "False", 2024-03-24T03:17:22.637 INFO:teuthology.orchestra.run.smithi118.stdout: "min": "", 2024-03-24T03:17:22.637 INFO:teuthology.orchestra.run.smithi118.stdout: "max": "", 2024-03-24T03:17:22.637 INFO:teuthology.orchestra.run.smithi118.stdout: "enum_allowed": [], 2024-03-24T03:17:22.637 INFO:teuthology.orchestra.run.smithi118.stdout: "desc": "", 2024-03-24T03:17:22.637 INFO:teuthology.orchestra.run.smithi118.stdout: "long_desc": "", 2024-03-24T03:17:22.637 INFO:teuthology.orchestra.run.smithi118.stdout: "tags": [], 2024-03-24T03:17:22.637 INFO:teuthology.orchestra.run.smithi118.stdout: "see_also": [] 2024-03-24T03:17:22.638 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-24T03:17:22.638 INFO:teuthology.orchestra.run.smithi118.stdout: "AUDIT_API_LOG_PAYLOAD": { 2024-03-24T03:17:22.638 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "AUDIT_API_LOG_PAYLOAD", 2024-03-24T03:17:22.638 INFO:teuthology.orchestra.run.smithi118.stdout: "type": "bool", 2024-03-24T03:17:22.638 INFO:teuthology.orchestra.run.smithi118.stdout: "level": "advanced", 2024-03-24T03:17:22.638 INFO:teuthology.orchestra.run.smithi118.stdout: "flags": 0, 2024-03-24T03:17:22.638 INFO:teuthology.orchestra.run.smithi118.stdout: "default_value": "True", 2024-03-24T03:17:22.638 INFO:teuthology.orchestra.run.smithi118.stdout: "min": "", 2024-03-24T03:17:22.639 INFO:teuthology.orchestra.run.smithi118.stdout: "max": "", 2024-03-24T03:17:22.639 INFO:teuthology.orchestra.run.smithi118.stdout: "enum_allowed": [], 2024-03-24T03:17:22.639 INFO:teuthology.orchestra.run.smithi118.stdout: "desc": "", 2024-03-24T03:17:22.639 INFO:teuthology.orchestra.run.smithi118.stdout: "long_desc": "", 2024-03-24T03:17:22.639 INFO:teuthology.orchestra.run.smithi118.stdout: "tags": [], 2024-03-24T03:17:22.639 INFO:teuthology.orchestra.run.smithi118.stdout: "see_also": [] 2024-03-24T03:17:22.639 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-24T03:17:22.639 INFO:teuthology.orchestra.run.smithi118.stdout: "ENABLE_BROWSABLE_API": { 2024-03-24T03:17:22.639 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "ENABLE_BROWSABLE_API", 2024-03-24T03:17:22.640 INFO:teuthology.orchestra.run.smithi118.stdout: "type": "bool", 2024-03-24T03:17:22.640 INFO:teuthology.orchestra.run.smithi118.stdout: "level": "advanced", 2024-03-24T03:17:22.640 INFO:teuthology.orchestra.run.smithi118.stdout: "flags": 0, 2024-03-24T03:17:22.640 INFO:teuthology.orchestra.run.smithi118.stdout: "default_value": "True", 2024-03-24T03:17:22.640 INFO:teuthology.orchestra.run.smithi118.stdout: "min": "", 2024-03-24T03:17:22.640 INFO:teuthology.orchestra.run.smithi118.stdout: "max": "", 2024-03-24T03:17:22.640 INFO:teuthology.orchestra.run.smithi118.stdout: "enum_allowed": [], 2024-03-24T03:17:22.640 INFO:teuthology.orchestra.run.smithi118.stdout: "desc": "", 2024-03-24T03:17:22.640 INFO:teuthology.orchestra.run.smithi118.stdout: "long_desc": "", 2024-03-24T03:17:22.641 INFO:teuthology.orchestra.run.smithi118.stdout: "tags": [], 2024-03-24T03:17:22.641 INFO:teuthology.orchestra.run.smithi118.stdout: "see_also": [] 2024-03-24T03:17:22.641 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-24T03:17:22.641 INFO:teuthology.orchestra.run.smithi118.stdout: "FEATURE_TOGGLE_CEPHFS": { 2024-03-24T03:17:22.641 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "FEATURE_TOGGLE_CEPHFS", 2024-03-24T03:17:22.641 INFO:teuthology.orchestra.run.smithi118.stdout: "type": "bool", 2024-03-24T03:17:22.641 INFO:teuthology.orchestra.run.smithi118.stdout: "level": "advanced", 2024-03-24T03:17:22.641 INFO:teuthology.orchestra.run.smithi118.stdout: "flags": 0, 2024-03-24T03:17:22.641 INFO:teuthology.orchestra.run.smithi118.stdout: "default_value": "True", 2024-03-24T03:17:22.642 INFO:teuthology.orchestra.run.smithi118.stdout: "min": "", 2024-03-24T03:17:22.642 INFO:teuthology.orchestra.run.smithi118.stdout: "max": "", 2024-03-24T03:17:22.642 INFO:teuthology.orchestra.run.smithi118.stdout: "enum_allowed": [], 2024-03-24T03:17:22.642 INFO:teuthology.orchestra.run.smithi118.stdout: "desc": "", 2024-03-24T03:17:22.642 INFO:teuthology.orchestra.run.smithi118.stdout: "long_desc": "", 2024-03-24T03:17:22.642 INFO:teuthology.orchestra.run.smithi118.stdout: "tags": [], 2024-03-24T03:17:22.642 INFO:teuthology.orchestra.run.smithi118.stdout: "see_also": [] 2024-03-24T03:17:22.642 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-24T03:17:22.642 INFO:teuthology.orchestra.run.smithi118.stdout: "FEATURE_TOGGLE_DASHBOARD": { 2024-03-24T03:17:22.643 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "FEATURE_TOGGLE_DASHBOARD", 2024-03-24T03:17:22.643 INFO:teuthology.orchestra.run.smithi118.stdout: "type": "bool", 2024-03-24T03:17:22.643 INFO:teuthology.orchestra.run.smithi118.stdout: "level": "advanced", 2024-03-24T03:17:22.643 INFO:teuthology.orchestra.run.smithi118.stdout: "flags": 0, 2024-03-24T03:17:22.643 INFO:teuthology.orchestra.run.smithi118.stdout: "default_value": "True", 2024-03-24T03:17:22.643 INFO:teuthology.orchestra.run.smithi118.stdout: "min": "", 2024-03-24T03:17:22.643 INFO:teuthology.orchestra.run.smithi118.stdout: "max": "", 2024-03-24T03:17:22.643 INFO:teuthology.orchestra.run.smithi118.stdout: "enum_allowed": [], 2024-03-24T03:17:22.643 INFO:teuthology.orchestra.run.smithi118.stdout: "desc": "", 2024-03-24T03:17:22.644 INFO:teuthology.orchestra.run.smithi118.stdout: "long_desc": "", 2024-03-24T03:17:22.644 INFO:teuthology.orchestra.run.smithi118.stdout: "tags": [], 2024-03-24T03:17:22.644 INFO:teuthology.orchestra.run.smithi118.stdout: "see_also": [] 2024-03-24T03:17:22.644 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-24T03:17:22.644 INFO:teuthology.orchestra.run.smithi118.stdout: "FEATURE_TOGGLE_ISCSI": { 2024-03-24T03:17:22.644 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "FEATURE_TOGGLE_ISCSI", 2024-03-24T03:17:22.644 INFO:teuthology.orchestra.run.smithi118.stdout: "type": "bool", 2024-03-24T03:17:22.644 INFO:teuthology.orchestra.run.smithi118.stdout: "level": "advanced", 2024-03-24T03:17:22.645 INFO:teuthology.orchestra.run.smithi118.stdout: "flags": 0, 2024-03-24T03:17:22.645 INFO:teuthology.orchestra.run.smithi118.stdout: "default_value": "True", 2024-03-24T03:17:22.645 INFO:teuthology.orchestra.run.smithi118.stdout: "min": "", 2024-03-24T03:17:22.645 INFO:teuthology.orchestra.run.smithi118.stdout: "max": "", 2024-03-24T03:17:22.645 INFO:teuthology.orchestra.run.smithi118.stdout: "enum_allowed": [], 2024-03-24T03:17:22.645 INFO:teuthology.orchestra.run.smithi118.stdout: "desc": "", 2024-03-24T03:17:22.645 INFO:teuthology.orchestra.run.smithi118.stdout: "long_desc": "", 2024-03-24T03:17:22.645 INFO:teuthology.orchestra.run.smithi118.stdout: "tags": [], 2024-03-24T03:17:22.645 INFO:teuthology.orchestra.run.smithi118.stdout: "see_also": [] 2024-03-24T03:17:22.646 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-24T03:17:22.646 INFO:teuthology.orchestra.run.smithi118.stdout: "FEATURE_TOGGLE_MIRRORING": { 2024-03-24T03:17:22.646 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "FEATURE_TOGGLE_MIRRORING", 2024-03-24T03:17:22.646 INFO:teuthology.orchestra.run.smithi118.stdout: "type": "bool", 2024-03-24T03:17:22.646 INFO:teuthology.orchestra.run.smithi118.stdout: "level": "advanced", 2024-03-24T03:17:22.646 INFO:teuthology.orchestra.run.smithi118.stdout: "flags": 0, 2024-03-24T03:17:22.646 INFO:teuthology.orchestra.run.smithi118.stdout: "default_value": "True", 2024-03-24T03:17:22.646 INFO:teuthology.orchestra.run.smithi118.stdout: "min": "", 2024-03-24T03:17:22.646 INFO:teuthology.orchestra.run.smithi118.stdout: "max": "", 2024-03-24T03:17:22.647 INFO:teuthology.orchestra.run.smithi118.stdout: "enum_allowed": [], 2024-03-24T03:17:22.647 INFO:teuthology.orchestra.run.smithi118.stdout: "desc": "", 2024-03-24T03:17:22.647 INFO:teuthology.orchestra.run.smithi118.stdout: "long_desc": "", 2024-03-24T03:17:22.647 INFO:teuthology.orchestra.run.smithi118.stdout: "tags": [], 2024-03-24T03:17:22.647 INFO:teuthology.orchestra.run.smithi118.stdout: "see_also": [] 2024-03-24T03:17:22.647 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-24T03:17:22.647 INFO:teuthology.orchestra.run.smithi118.stdout: "FEATURE_TOGGLE_NFS": { 2024-03-24T03:17:22.647 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "FEATURE_TOGGLE_NFS", 2024-03-24T03:17:22.647 INFO:teuthology.orchestra.run.smithi118.stdout: "type": "bool", 2024-03-24T03:17:22.648 INFO:teuthology.orchestra.run.smithi118.stdout: "level": "advanced", 2024-03-24T03:17:22.648 INFO:teuthology.orchestra.run.smithi118.stdout: "flags": 0, 2024-03-24T03:17:22.648 INFO:teuthology.orchestra.run.smithi118.stdout: "default_value": "True", 2024-03-24T03:17:22.648 INFO:teuthology.orchestra.run.smithi118.stdout: "min": "", 2024-03-24T03:17:22.648 INFO:teuthology.orchestra.run.smithi118.stdout: "max": "", 2024-03-24T03:17:22.648 INFO:teuthology.orchestra.run.smithi118.stdout: "enum_allowed": [], 2024-03-24T03:17:22.648 INFO:teuthology.orchestra.run.smithi118.stdout: "desc": "", 2024-03-24T03:17:22.648 INFO:teuthology.orchestra.run.smithi118.stdout: "long_desc": "", 2024-03-24T03:17:22.648 INFO:teuthology.orchestra.run.smithi118.stdout: "tags": [], 2024-03-24T03:17:22.648 INFO:teuthology.orchestra.run.smithi118.stdout: "see_also": [] 2024-03-24T03:17:22.649 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-24T03:17:22.649 INFO:teuthology.orchestra.run.smithi118.stdout: "FEATURE_TOGGLE_RBD": { 2024-03-24T03:17:22.649 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "FEATURE_TOGGLE_RBD", 2024-03-24T03:17:22.649 INFO:teuthology.orchestra.run.smithi118.stdout: "type": "bool", 2024-03-24T03:17:22.649 INFO:teuthology.orchestra.run.smithi118.stdout: "level": "advanced", 2024-03-24T03:17:22.649 INFO:teuthology.orchestra.run.smithi118.stdout: "flags": 0, 2024-03-24T03:17:22.649 INFO:teuthology.orchestra.run.smithi118.stdout: "default_value": "True", 2024-03-24T03:17:22.649 INFO:teuthology.orchestra.run.smithi118.stdout: "min": "", 2024-03-24T03:17:22.649 INFO:teuthology.orchestra.run.smithi118.stdout: "max": "", 2024-03-24T03:17:22.650 INFO:teuthology.orchestra.run.smithi118.stdout: "enum_allowed": [], 2024-03-24T03:17:22.650 INFO:teuthology.orchestra.run.smithi118.stdout: "desc": "", 2024-03-24T03:17:22.650 INFO:teuthology.orchestra.run.smithi118.stdout: "long_desc": "", 2024-03-24T03:17:22.650 INFO:teuthology.orchestra.run.smithi118.stdout: "tags": [], 2024-03-24T03:17:22.650 INFO:teuthology.orchestra.run.smithi118.stdout: "see_also": [] 2024-03-24T03:17:22.650 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-24T03:17:22.650 INFO:teuthology.orchestra.run.smithi118.stdout: "FEATURE_TOGGLE_RGW": { 2024-03-24T03:17:22.650 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "FEATURE_TOGGLE_RGW", 2024-03-24T03:17:22.650 INFO:teuthology.orchestra.run.smithi118.stdout: "type": "bool", 2024-03-24T03:17:22.651 INFO:teuthology.orchestra.run.smithi118.stdout: "level": "advanced", 2024-03-24T03:17:22.651 INFO:teuthology.orchestra.run.smithi118.stdout: "flags": 0, 2024-03-24T03:17:22.651 INFO:teuthology.orchestra.run.smithi118.stdout: "default_value": "True", 2024-03-24T03:17:22.651 INFO:teuthology.orchestra.run.smithi118.stdout: "min": "", 2024-03-24T03:17:22.651 INFO:teuthology.orchestra.run.smithi118.stdout: "max": "", 2024-03-24T03:17:22.651 INFO:teuthology.orchestra.run.smithi118.stdout: "enum_allowed": [], 2024-03-24T03:17:22.651 INFO:teuthology.orchestra.run.smithi118.stdout: "desc": "", 2024-03-24T03:17:22.651 INFO:teuthology.orchestra.run.smithi118.stdout: "long_desc": "", 2024-03-24T03:17:22.652 INFO:teuthology.orchestra.run.smithi118.stdout: "tags": [], 2024-03-24T03:17:22.652 INFO:teuthology.orchestra.run.smithi118.stdout: "see_also": [] 2024-03-24T03:17:22.652 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-24T03:17:22.652 INFO:teuthology.orchestra.run.smithi118.stdout: "GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE": { 2024-03-24T03:17:22.652 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE", 2024-03-24T03:17:22.652 INFO:teuthology.orchestra.run.smithi118.stdout: "type": "str", 2024-03-24T03:17:22.652 INFO:teuthology.orchestra.run.smithi118.stdout: "level": "advanced", 2024-03-24T03:17:22.652 INFO:teuthology.orchestra.run.smithi118.stdout: "flags": 0, 2024-03-24T03:17:22.652 INFO:teuthology.orchestra.run.smithi118.stdout: "default_value": "", 2024-03-24T03:17:22.653 INFO:teuthology.orchestra.run.smithi118.stdout: "min": "", 2024-03-24T03:17:22.653 INFO:teuthology.orchestra.run.smithi118.stdout: "max": "", 2024-03-24T03:17:22.653 INFO:teuthology.orchestra.run.smithi118.stdout: "enum_allowed": [], 2024-03-24T03:17:22.653 INFO:teuthology.orchestra.run.smithi118.stdout: "desc": "", 2024-03-24T03:17:22.653 INFO:teuthology.orchestra.run.smithi118.stdout: "long_desc": "", 2024-03-24T03:17:22.653 INFO:teuthology.orchestra.run.smithi118.stdout: "tags": [], 2024-03-24T03:17:22.653 INFO:teuthology.orchestra.run.smithi118.stdout: "see_also": [] 2024-03-24T03:17:22.654 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-24T03:17:22.654 INFO:teuthology.orchestra.run.smithi118.stdout: "GRAFANA_API_PASSWORD": { 2024-03-24T03:17:22.654 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "GRAFANA_API_PASSWORD", 2024-03-24T03:17:22.654 INFO:teuthology.orchestra.run.smithi118.stdout: "type": "str", 2024-03-24T03:17:22.654 INFO:teuthology.orchestra.run.smithi118.stdout: "level": "advanced", 2024-03-24T03:17:22.654 INFO:teuthology.orchestra.run.smithi118.stdout: "flags": 0, 2024-03-24T03:17:22.654 INFO:teuthology.orchestra.run.smithi118.stdout: "default_value": "admin", 2024-03-24T03:17:22.655 INFO:teuthology.orchestra.run.smithi118.stdout: "min": "", 2024-03-24T03:17:22.655 INFO:teuthology.orchestra.run.smithi118.stdout: "max": "", 2024-03-24T03:17:22.655 INFO:teuthology.orchestra.run.smithi118.stdout: "enum_allowed": [], 2024-03-24T03:17:22.655 INFO:teuthology.orchestra.run.smithi118.stdout: "desc": "", 2024-03-24T03:17:22.655 INFO:teuthology.orchestra.run.smithi118.stdout: "long_desc": "", 2024-03-24T03:17:22.655 INFO:teuthology.orchestra.run.smithi118.stdout: "tags": [], 2024-03-24T03:17:22.655 INFO:teuthology.orchestra.run.smithi118.stdout: "see_also": [] 2024-03-24T03:17:22.655 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-24T03:17:22.655 INFO:teuthology.orchestra.run.smithi118.stdout: "GRAFANA_API_SSL_VERIFY": { 2024-03-24T03:17:22.656 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "GRAFANA_API_SSL_VERIFY", 2024-03-24T03:17:22.656 INFO:teuthology.orchestra.run.smithi118.stdout: "type": "bool", 2024-03-24T03:17:22.656 INFO:teuthology.orchestra.run.smithi118.stdout: "level": "advanced", 2024-03-24T03:17:22.656 INFO:teuthology.orchestra.run.smithi118.stdout: "flags": 0, 2024-03-24T03:17:22.656 INFO:teuthology.orchestra.run.smithi118.stdout: "default_value": "True", 2024-03-24T03:17:22.656 INFO:teuthology.orchestra.run.smithi118.stdout: "min": "", 2024-03-24T03:17:22.656 INFO:teuthology.orchestra.run.smithi118.stdout: "max": "", 2024-03-24T03:17:22.656 INFO:teuthology.orchestra.run.smithi118.stdout: "enum_allowed": [], 2024-03-24T03:17:22.657 INFO:teuthology.orchestra.run.smithi118.stdout: "desc": "", 2024-03-24T03:17:22.657 INFO:teuthology.orchestra.run.smithi118.stdout: "long_desc": "", 2024-03-24T03:17:22.657 INFO:teuthology.orchestra.run.smithi118.stdout: "tags": [], 2024-03-24T03:17:22.657 INFO:teuthology.orchestra.run.smithi118.stdout: "see_also": [] 2024-03-24T03:17:22.657 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-24T03:17:22.657 INFO:teuthology.orchestra.run.smithi118.stdout: "GRAFANA_API_URL": { 2024-03-24T03:17:22.657 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "GRAFANA_API_URL", 2024-03-24T03:17:22.657 INFO:teuthology.orchestra.run.smithi118.stdout: "type": "str", 2024-03-24T03:17:22.657 INFO:teuthology.orchestra.run.smithi118.stdout: "level": "advanced", 2024-03-24T03:17:22.658 INFO:teuthology.orchestra.run.smithi118.stdout: "flags": 0, 2024-03-24T03:17:22.658 INFO:teuthology.orchestra.run.smithi118.stdout: "default_value": "", 2024-03-24T03:17:22.658 INFO:teuthology.orchestra.run.smithi118.stdout: "min": "", 2024-03-24T03:17:22.658 INFO:teuthology.orchestra.run.smithi118.stdout: "max": "", 2024-03-24T03:17:22.658 INFO:teuthology.orchestra.run.smithi118.stdout: "enum_allowed": [], 2024-03-24T03:17:22.658 INFO:teuthology.orchestra.run.smithi118.stdout: "desc": "", 2024-03-24T03:17:22.659 INFO:teuthology.orchestra.run.smithi118.stdout: "long_desc": "", 2024-03-24T03:17:22.659 INFO:teuthology.orchestra.run.smithi118.stdout: "tags": [], 2024-03-24T03:17:22.659 INFO:teuthology.orchestra.run.smithi118.stdout: "see_also": [] 2024-03-24T03:17:22.659 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-24T03:17:22.659 INFO:teuthology.orchestra.run.smithi118.stdout: "GRAFANA_API_USERNAME": { 2024-03-24T03:17:22.659 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "GRAFANA_API_USERNAME", 2024-03-24T03:17:22.659 INFO:teuthology.orchestra.run.smithi118.stdout: "type": "str", 2024-03-24T03:17:22.659 INFO:teuthology.orchestra.run.smithi118.stdout: "level": "advanced", 2024-03-24T03:17:22.660 INFO:teuthology.orchestra.run.smithi118.stdout: "flags": 0, 2024-03-24T03:17:22.660 INFO:teuthology.orchestra.run.smithi118.stdout: "default_value": "admin", 2024-03-24T03:17:22.660 INFO:teuthology.orchestra.run.smithi118.stdout: "min": "", 2024-03-24T03:17:22.660 INFO:teuthology.orchestra.run.smithi118.stdout: "max": "", 2024-03-24T03:17:22.660 INFO:teuthology.orchestra.run.smithi118.stdout: "enum_allowed": [], 2024-03-24T03:17:22.660 INFO:teuthology.orchestra.run.smithi118.stdout: "desc": "", 2024-03-24T03:17:22.660 INFO:teuthology.orchestra.run.smithi118.stdout: "long_desc": "", 2024-03-24T03:17:22.660 INFO:teuthology.orchestra.run.smithi118.stdout: "tags": [], 2024-03-24T03:17:22.660 INFO:teuthology.orchestra.run.smithi118.stdout: "see_also": [] 2024-03-24T03:17:22.661 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-24T03:17:22.661 INFO:teuthology.orchestra.run.smithi118.stdout: "GRAFANA_FRONTEND_API_URL": { 2024-03-24T03:17:22.661 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "GRAFANA_FRONTEND_API_URL", 2024-03-24T03:17:22.661 INFO:teuthology.orchestra.run.smithi118.stdout: "type": "str", 2024-03-24T03:17:22.661 INFO:teuthology.orchestra.run.smithi118.stdout: "level": "advanced", 2024-03-24T03:17:22.661 INFO:teuthology.orchestra.run.smithi118.stdout: "flags": 0, 2024-03-24T03:17:22.661 INFO:teuthology.orchestra.run.smithi118.stdout: "default_value": "", 2024-03-24T03:17:22.661 INFO:teuthology.orchestra.run.smithi118.stdout: "min": "", 2024-03-24T03:17:22.662 INFO:teuthology.orchestra.run.smithi118.stdout: "max": "", 2024-03-24T03:17:22.662 INFO:teuthology.orchestra.run.smithi118.stdout: "enum_allowed": [], 2024-03-24T03:17:22.662 INFO:teuthology.orchestra.run.smithi118.stdout: "desc": "", 2024-03-24T03:17:22.662 INFO:teuthology.orchestra.run.smithi118.stdout: "long_desc": "", 2024-03-24T03:17:22.662 INFO:teuthology.orchestra.run.smithi118.stdout: "tags": [], 2024-03-24T03:17:22.662 INFO:teuthology.orchestra.run.smithi118.stdout: "see_also": [] 2024-03-24T03:17:22.662 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-24T03:17:22.662 INFO:teuthology.orchestra.run.smithi118.stdout: "GRAFANA_UPDATE_DASHBOARDS": { 2024-03-24T03:17:22.663 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "GRAFANA_UPDATE_DASHBOARDS", 2024-03-24T03:17:22.663 INFO:teuthology.orchestra.run.smithi118.stdout: "type": "bool", 2024-03-24T03:17:22.663 INFO:teuthology.orchestra.run.smithi118.stdout: "level": "advanced", 2024-03-24T03:17:22.663 INFO:teuthology.orchestra.run.smithi118.stdout: "flags": 0, 2024-03-24T03:17:22.663 INFO:teuthology.orchestra.run.smithi118.stdout: "default_value": "False", 2024-03-24T03:17:22.663 INFO:teuthology.orchestra.run.smithi118.stdout: "min": "", 2024-03-24T03:17:22.663 INFO:teuthology.orchestra.run.smithi118.stdout: "max": "", 2024-03-24T03:17:22.663 INFO:teuthology.orchestra.run.smithi118.stdout: "enum_allowed": [], 2024-03-24T03:17:22.663 INFO:teuthology.orchestra.run.smithi118.stdout: "desc": "", 2024-03-24T03:17:22.664 INFO:teuthology.orchestra.run.smithi118.stdout: "long_desc": "", 2024-03-24T03:17:22.664 INFO:teuthology.orchestra.run.smithi118.stdout: "tags": [], 2024-03-24T03:17:22.664 INFO:teuthology.orchestra.run.smithi118.stdout: "see_also": [] 2024-03-24T03:17:22.664 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-24T03:17:22.664 INFO:teuthology.orchestra.run.smithi118.stdout: "ISCSI_API_SSL_VERIFICATION": { 2024-03-24T03:17:22.664 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "ISCSI_API_SSL_VERIFICATION", 2024-03-24T03:17:22.664 INFO:teuthology.orchestra.run.smithi118.stdout: "type": "bool", 2024-03-24T03:17:22.664 INFO:teuthology.orchestra.run.smithi118.stdout: "level": "advanced", 2024-03-24T03:17:22.664 INFO:teuthology.orchestra.run.smithi118.stdout: "flags": 0, 2024-03-24T03:17:22.665 INFO:teuthology.orchestra.run.smithi118.stdout: "default_value": "True", 2024-03-24T03:17:22.665 INFO:teuthology.orchestra.run.smithi118.stdout: "min": "", 2024-03-24T03:17:22.665 INFO:teuthology.orchestra.run.smithi118.stdout: "max": "", 2024-03-24T03:17:22.665 INFO:teuthology.orchestra.run.smithi118.stdout: "enum_allowed": [], 2024-03-24T03:17:22.665 INFO:teuthology.orchestra.run.smithi118.stdout: "desc": "", 2024-03-24T03:17:22.665 INFO:teuthology.orchestra.run.smithi118.stdout: "long_desc": "", 2024-03-24T03:17:22.665 INFO:teuthology.orchestra.run.smithi118.stdout: "tags": [], 2024-03-24T03:17:22.665 INFO:teuthology.orchestra.run.smithi118.stdout: "see_also": [] 2024-03-24T03:17:22.666 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-24T03:17:22.666 INFO:teuthology.orchestra.run.smithi118.stdout: "ISSUE_TRACKER_API_KEY": { 2024-03-24T03:17:22.666 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "ISSUE_TRACKER_API_KEY", 2024-03-24T03:17:22.666 INFO:teuthology.orchestra.run.smithi118.stdout: "type": "str", 2024-03-24T03:17:22.666 INFO:teuthology.orchestra.run.smithi118.stdout: "level": "advanced", 2024-03-24T03:17:22.666 INFO:teuthology.orchestra.run.smithi118.stdout: "flags": 0, 2024-03-24T03:17:22.666 INFO:teuthology.orchestra.run.smithi118.stdout: "default_value": "", 2024-03-24T03:17:22.666 INFO:teuthology.orchestra.run.smithi118.stdout: "min": "", 2024-03-24T03:17:22.667 INFO:teuthology.orchestra.run.smithi118.stdout: "max": "", 2024-03-24T03:17:22.667 INFO:teuthology.orchestra.run.smithi118.stdout: "enum_allowed": [], 2024-03-24T03:17:22.667 INFO:teuthology.orchestra.run.smithi118.stdout: "desc": "", 2024-03-24T03:17:22.667 INFO:teuthology.orchestra.run.smithi118.stdout: "long_desc": "", 2024-03-24T03:17:22.667 INFO:teuthology.orchestra.run.smithi118.stdout: "tags": [], 2024-03-24T03:17:22.667 INFO:teuthology.orchestra.run.smithi118.stdout: "see_also": [] 2024-03-24T03:17:22.667 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-24T03:17:22.667 INFO:teuthology.orchestra.run.smithi118.stdout: "MULTICLUSTER_CONFIG": { 2024-03-24T03:17:22.668 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "MULTICLUSTER_CONFIG", 2024-03-24T03:17:22.668 INFO:teuthology.orchestra.run.smithi118.stdout: "type": "str", 2024-03-24T03:17:22.668 INFO:teuthology.orchestra.run.smithi118.stdout: "level": "advanced", 2024-03-24T03:17:22.668 INFO:teuthology.orchestra.run.smithi118.stdout: "flags": 0, 2024-03-24T03:17:22.668 INFO:teuthology.orchestra.run.smithi118.stdout: "default_value": "{}", 2024-03-24T03:17:22.668 INFO:teuthology.orchestra.run.smithi118.stdout: "min": "", 2024-03-24T03:17:22.668 INFO:teuthology.orchestra.run.smithi118.stdout: "max": "", 2024-03-24T03:17:22.668 INFO:teuthology.orchestra.run.smithi118.stdout: "enum_allowed": [], 2024-03-24T03:17:22.668 INFO:teuthology.orchestra.run.smithi118.stdout: "desc": "", 2024-03-24T03:17:22.669 INFO:teuthology.orchestra.run.smithi118.stdout: "long_desc": "", 2024-03-24T03:17:22.669 INFO:teuthology.orchestra.run.smithi118.stdout: "tags": [], 2024-03-24T03:17:22.669 INFO:teuthology.orchestra.run.smithi118.stdout: "see_also": [] 2024-03-24T03:17:22.669 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-24T03:17:22.669 INFO:teuthology.orchestra.run.smithi118.stdout: "PROMETHEUS_API_HOST": { 2024-03-24T03:17:22.669 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "PROMETHEUS_API_HOST", 2024-03-24T03:17:22.669 INFO:teuthology.orchestra.run.smithi118.stdout: "type": "str", 2024-03-24T03:17:22.669 INFO:teuthology.orchestra.run.smithi118.stdout: "level": "advanced", 2024-03-24T03:17:22.670 INFO:teuthology.orchestra.run.smithi118.stdout: "flags": 0, 2024-03-24T03:17:22.670 INFO:teuthology.orchestra.run.smithi118.stdout: "default_value": "", 2024-03-24T03:17:22.670 INFO:teuthology.orchestra.run.smithi118.stdout: "min": "", 2024-03-24T03:17:22.670 INFO:teuthology.orchestra.run.smithi118.stdout: "max": "", 2024-03-24T03:17:22.670 INFO:teuthology.orchestra.run.smithi118.stdout: "enum_allowed": [], 2024-03-24T03:17:22.670 INFO:teuthology.orchestra.run.smithi118.stdout: "desc": "", 2024-03-24T03:17:22.670 INFO:teuthology.orchestra.run.smithi118.stdout: "long_desc": "", 2024-03-24T03:17:22.670 INFO:teuthology.orchestra.run.smithi118.stdout: "tags": [], 2024-03-24T03:17:22.670 INFO:teuthology.orchestra.run.smithi118.stdout: "see_also": [] 2024-03-24T03:17:22.671 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-24T03:17:22.671 INFO:teuthology.orchestra.run.smithi118.stdout: "PROMETHEUS_API_SSL_VERIFY": { 2024-03-24T03:17:22.671 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "PROMETHEUS_API_SSL_VERIFY", 2024-03-24T03:17:22.671 INFO:teuthology.orchestra.run.smithi118.stdout: "type": "bool", 2024-03-24T03:17:22.671 INFO:teuthology.orchestra.run.smithi118.stdout: "level": "advanced", 2024-03-24T03:17:22.671 INFO:teuthology.orchestra.run.smithi118.stdout: "flags": 0, 2024-03-24T03:17:22.671 INFO:teuthology.orchestra.run.smithi118.stdout: "default_value": "True", 2024-03-24T03:17:22.671 INFO:teuthology.orchestra.run.smithi118.stdout: "min": "", 2024-03-24T03:17:22.672 INFO:teuthology.orchestra.run.smithi118.stdout: "max": "", 2024-03-24T03:17:22.672 INFO:teuthology.orchestra.run.smithi118.stdout: "enum_allowed": [], 2024-03-24T03:17:22.672 INFO:teuthology.orchestra.run.smithi118.stdout: "desc": "", 2024-03-24T03:17:22.672 INFO:teuthology.orchestra.run.smithi118.stdout: "long_desc": "", 2024-03-24T03:17:22.672 INFO:teuthology.orchestra.run.smithi118.stdout: "tags": [], 2024-03-24T03:17:22.672 INFO:teuthology.orchestra.run.smithi118.stdout: "see_also": [] 2024-03-24T03:17:22.672 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-24T03:17:22.672 INFO:teuthology.orchestra.run.smithi118.stdout: "PWD_POLICY_CHECK_COMPLEXITY_ENABLED": { 2024-03-24T03:17:22.673 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "PWD_POLICY_CHECK_COMPLEXITY_ENABLED", 2024-03-24T03:17:22.673 INFO:teuthology.orchestra.run.smithi118.stdout: "type": "bool", 2024-03-24T03:17:22.673 INFO:teuthology.orchestra.run.smithi118.stdout: "level": "advanced", 2024-03-24T03:17:22.673 INFO:teuthology.orchestra.run.smithi118.stdout: "flags": 0, 2024-03-24T03:17:22.673 INFO:teuthology.orchestra.run.smithi118.stdout: "default_value": "False", 2024-03-24T03:17:22.673 INFO:teuthology.orchestra.run.smithi118.stdout: "min": "", 2024-03-24T03:17:22.673 INFO:teuthology.orchestra.run.smithi118.stdout: "max": "", 2024-03-24T03:17:22.673 INFO:teuthology.orchestra.run.smithi118.stdout: "enum_allowed": [], 2024-03-24T03:17:22.674 INFO:teuthology.orchestra.run.smithi118.stdout: "desc": "", 2024-03-24T03:17:22.674 INFO:teuthology.orchestra.run.smithi118.stdout: "long_desc": "", 2024-03-24T03:17:22.674 INFO:teuthology.orchestra.run.smithi118.stdout: "tags": [], 2024-03-24T03:17:22.674 INFO:teuthology.orchestra.run.smithi118.stdout: "see_also": [] 2024-03-24T03:17:22.674 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-24T03:17:22.674 INFO:teuthology.orchestra.run.smithi118.stdout: "PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED": { 2024-03-24T03:17:22.674 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED", 2024-03-24T03:17:22.674 INFO:teuthology.orchestra.run.smithi118.stdout: "type": "bool", 2024-03-24T03:17:22.674 INFO:teuthology.orchestra.run.smithi118.stdout: "level": "advanced", 2024-03-24T03:17:22.675 INFO:teuthology.orchestra.run.smithi118.stdout: "flags": 0, 2024-03-24T03:17:22.675 INFO:teuthology.orchestra.run.smithi118.stdout: "default_value": "False", 2024-03-24T03:17:22.675 INFO:teuthology.orchestra.run.smithi118.stdout: "min": "", 2024-03-24T03:17:22.675 INFO:teuthology.orchestra.run.smithi118.stdout: "max": "", 2024-03-24T03:17:22.675 INFO:teuthology.orchestra.run.smithi118.stdout: "enum_allowed": [], 2024-03-24T03:17:22.675 INFO:teuthology.orchestra.run.smithi118.stdout: "desc": "", 2024-03-24T03:17:22.675 INFO:teuthology.orchestra.run.smithi118.stdout: "long_desc": "", 2024-03-24T03:17:22.675 INFO:teuthology.orchestra.run.smithi118.stdout: "tags": [], 2024-03-24T03:17:22.675 INFO:teuthology.orchestra.run.smithi118.stdout: "see_also": [] 2024-03-24T03:17:22.676 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-24T03:17:22.676 INFO:teuthology.orchestra.run.smithi118.stdout: "PWD_POLICY_CHECK_LENGTH_ENABLED": { 2024-03-24T03:17:22.676 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "PWD_POLICY_CHECK_LENGTH_ENABLED", 2024-03-24T03:17:22.676 INFO:teuthology.orchestra.run.smithi118.stdout: "type": "bool", 2024-03-24T03:17:22.676 INFO:teuthology.orchestra.run.smithi118.stdout: "level": "advanced", 2024-03-24T03:17:22.676 INFO:teuthology.orchestra.run.smithi118.stdout: "flags": 0, 2024-03-24T03:17:22.676 INFO:teuthology.orchestra.run.smithi118.stdout: "default_value": "True", 2024-03-24T03:17:22.676 INFO:teuthology.orchestra.run.smithi118.stdout: "min": "", 2024-03-24T03:17:22.677 INFO:teuthology.orchestra.run.smithi118.stdout: "max": "", 2024-03-24T03:17:22.677 INFO:teuthology.orchestra.run.smithi118.stdout: "enum_allowed": [], 2024-03-24T03:17:22.677 INFO:teuthology.orchestra.run.smithi118.stdout: "desc": "", 2024-03-24T03:17:22.677 INFO:teuthology.orchestra.run.smithi118.stdout: "long_desc": "", 2024-03-24T03:17:22.677 INFO:teuthology.orchestra.run.smithi118.stdout: "tags": [], 2024-03-24T03:17:22.677 INFO:teuthology.orchestra.run.smithi118.stdout: "see_also": [] 2024-03-24T03:17:22.677 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-24T03:17:22.677 INFO:teuthology.orchestra.run.smithi118.stdout: "PWD_POLICY_CHECK_OLDPWD_ENABLED": { 2024-03-24T03:17:22.677 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "PWD_POLICY_CHECK_OLDPWD_ENABLED", 2024-03-24T03:17:22.678 INFO:teuthology.orchestra.run.smithi118.stdout: "type": "bool", 2024-03-24T03:17:22.678 INFO:teuthology.orchestra.run.smithi118.stdout: "level": "advanced", 2024-03-24T03:17:22.678 INFO:teuthology.orchestra.run.smithi118.stdout: "flags": 0, 2024-03-24T03:17:22.678 INFO:teuthology.orchestra.run.smithi118.stdout: "default_value": "True", 2024-03-24T03:17:22.678 INFO:teuthology.orchestra.run.smithi118.stdout: "min": "", 2024-03-24T03:17:22.678 INFO:teuthology.orchestra.run.smithi118.stdout: "max": "", 2024-03-24T03:17:22.678 INFO:teuthology.orchestra.run.smithi118.stdout: "enum_allowed": [], 2024-03-24T03:17:22.678 INFO:teuthology.orchestra.run.smithi118.stdout: "desc": "", 2024-03-24T03:17:22.678 INFO:teuthology.orchestra.run.smithi118.stdout: "long_desc": "", 2024-03-24T03:17:22.679 INFO:teuthology.orchestra.run.smithi118.stdout: "tags": [], 2024-03-24T03:17:22.679 INFO:teuthology.orchestra.run.smithi118.stdout: "see_also": [] 2024-03-24T03:17:22.679 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-24T03:17:22.679 INFO:teuthology.orchestra.run.smithi118.stdout: "PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED": { 2024-03-24T03:17:22.679 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED", 2024-03-24T03:17:22.679 INFO:teuthology.orchestra.run.smithi118.stdout: "type": "bool", 2024-03-24T03:17:22.679 INFO:teuthology.orchestra.run.smithi118.stdout: "level": "advanced", 2024-03-24T03:17:22.679 INFO:teuthology.orchestra.run.smithi118.stdout: "flags": 0, 2024-03-24T03:17:22.679 INFO:teuthology.orchestra.run.smithi118.stdout: "default_value": "False", 2024-03-24T03:17:22.679 INFO:teuthology.orchestra.run.smithi118.stdout: "min": "", 2024-03-24T03:17:22.680 INFO:teuthology.orchestra.run.smithi118.stdout: "max": "", 2024-03-24T03:17:22.680 INFO:teuthology.orchestra.run.smithi118.stdout: "enum_allowed": [], 2024-03-24T03:17:22.680 INFO:teuthology.orchestra.run.smithi118.stdout: "desc": "", 2024-03-24T03:17:22.680 INFO:teuthology.orchestra.run.smithi118.stdout: "long_desc": "", 2024-03-24T03:17:22.680 INFO:teuthology.orchestra.run.smithi118.stdout: "tags": [], 2024-03-24T03:17:22.680 INFO:teuthology.orchestra.run.smithi118.stdout: "see_also": [] 2024-03-24T03:17:22.680 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-24T03:17:22.680 INFO:teuthology.orchestra.run.smithi118.stdout: "PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED": { 2024-03-24T03:17:22.681 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED", 2024-03-24T03:17:22.681 INFO:teuthology.orchestra.run.smithi118.stdout: "type": "bool", 2024-03-24T03:17:22.681 INFO:teuthology.orchestra.run.smithi118.stdout: "level": "advanced", 2024-03-24T03:17:22.681 INFO:teuthology.orchestra.run.smithi118.stdout: "flags": 0, 2024-03-24T03:17:22.681 INFO:teuthology.orchestra.run.smithi118.stdout: "default_value": "False", 2024-03-24T03:17:22.681 INFO:teuthology.orchestra.run.smithi118.stdout: "min": "", 2024-03-24T03:17:22.681 INFO:teuthology.orchestra.run.smithi118.stdout: "max": "", 2024-03-24T03:17:22.681 INFO:teuthology.orchestra.run.smithi118.stdout: "enum_allowed": [], 2024-03-24T03:17:22.681 INFO:teuthology.orchestra.run.smithi118.stdout: "desc": "", 2024-03-24T03:17:22.682 INFO:teuthology.orchestra.run.smithi118.stdout: "long_desc": "", 2024-03-24T03:17:22.682 INFO:teuthology.orchestra.run.smithi118.stdout: "tags": [], 2024-03-24T03:17:22.682 INFO:teuthology.orchestra.run.smithi118.stdout: "see_also": [] 2024-03-24T03:17:22.682 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-24T03:17:22.682 INFO:teuthology.orchestra.run.smithi118.stdout: "PWD_POLICY_CHECK_USERNAME_ENABLED": { 2024-03-24T03:17:22.682 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "PWD_POLICY_CHECK_USERNAME_ENABLED", 2024-03-24T03:17:22.682 INFO:teuthology.orchestra.run.smithi118.stdout: "type": "bool", 2024-03-24T03:17:22.682 INFO:teuthology.orchestra.run.smithi118.stdout: "level": "advanced", 2024-03-24T03:17:22.682 INFO:teuthology.orchestra.run.smithi118.stdout: "flags": 0, 2024-03-24T03:17:22.683 INFO:teuthology.orchestra.run.smithi118.stdout: "default_value": "False", 2024-03-24T03:17:22.683 INFO:teuthology.orchestra.run.smithi118.stdout: "min": "", 2024-03-24T03:17:22.683 INFO:teuthology.orchestra.run.smithi118.stdout: "max": "", 2024-03-24T03:17:22.683 INFO:teuthology.orchestra.run.smithi118.stdout: "enum_allowed": [], 2024-03-24T03:17:22.683 INFO:teuthology.orchestra.run.smithi118.stdout: "desc": "", 2024-03-24T03:17:22.683 INFO:teuthology.orchestra.run.smithi118.stdout: "long_desc": "", 2024-03-24T03:17:22.683 INFO:teuthology.orchestra.run.smithi118.stdout: "tags": [], 2024-03-24T03:17:22.683 INFO:teuthology.orchestra.run.smithi118.stdout: "see_also": [] 2024-03-24T03:17:22.684 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-24T03:17:22.684 INFO:teuthology.orchestra.run.smithi118.stdout: "PWD_POLICY_ENABLED": { 2024-03-24T03:17:22.684 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "PWD_POLICY_ENABLED", 2024-03-24T03:17:22.684 INFO:teuthology.orchestra.run.smithi118.stdout: "type": "bool", 2024-03-24T03:17:22.684 INFO:teuthology.orchestra.run.smithi118.stdout: "level": "advanced", 2024-03-24T03:17:22.685 INFO:teuthology.orchestra.run.smithi118.stdout: "flags": 0, 2024-03-24T03:17:22.685 INFO:teuthology.orchestra.run.smithi118.stdout: "default_value": "True", 2024-03-24T03:17:22.685 INFO:teuthology.orchestra.run.smithi118.stdout: "min": "", 2024-03-24T03:17:22.685 INFO:teuthology.orchestra.run.smithi118.stdout: "max": "", 2024-03-24T03:17:22.685 INFO:teuthology.orchestra.run.smithi118.stdout: "enum_allowed": [], 2024-03-24T03:17:22.685 INFO:teuthology.orchestra.run.smithi118.stdout: "desc": "", 2024-03-24T03:17:22.685 INFO:teuthology.orchestra.run.smithi118.stdout: "long_desc": "", 2024-03-24T03:17:22.685 INFO:teuthology.orchestra.run.smithi118.stdout: "tags": [], 2024-03-24T03:17:22.686 INFO:teuthology.orchestra.run.smithi118.stdout: "see_also": [] 2024-03-24T03:17:22.686 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-24T03:17:22.686 INFO:teuthology.orchestra.run.smithi118.stdout: "PWD_POLICY_EXCLUSION_LIST": { 2024-03-24T03:17:22.686 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "PWD_POLICY_EXCLUSION_LIST", 2024-03-24T03:17:22.686 INFO:teuthology.orchestra.run.smithi118.stdout: "type": "str", 2024-03-24T03:17:22.686 INFO:teuthology.orchestra.run.smithi118.stdout: "level": "advanced", 2024-03-24T03:17:22.686 INFO:teuthology.orchestra.run.smithi118.stdout: "flags": 0, 2024-03-24T03:17:22.686 INFO:teuthology.orchestra.run.smithi118.stdout: "default_value": "osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps", 2024-03-24T03:17:22.687 INFO:teuthology.orchestra.run.smithi118.stdout: "min": "", 2024-03-24T03:17:22.687 INFO:teuthology.orchestra.run.smithi118.stdout: "max": "", 2024-03-24T03:17:22.687 INFO:teuthology.orchestra.run.smithi118.stdout: "enum_allowed": [], 2024-03-24T03:17:22.687 INFO:teuthology.orchestra.run.smithi118.stdout: "desc": "", 2024-03-24T03:17:22.687 INFO:teuthology.orchestra.run.smithi118.stdout: "long_desc": "", 2024-03-24T03:17:22.687 INFO:teuthology.orchestra.run.smithi118.stdout: "tags": [], 2024-03-24T03:17:22.687 INFO:teuthology.orchestra.run.smithi118.stdout: "see_also": [] 2024-03-24T03:17:22.687 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-24T03:17:22.687 INFO:teuthology.orchestra.run.smithi118.stdout: "PWD_POLICY_MIN_COMPLEXITY": { 2024-03-24T03:17:22.688 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "PWD_POLICY_MIN_COMPLEXITY", 2024-03-24T03:17:22.688 INFO:teuthology.orchestra.run.smithi118.stdout: "type": "int", 2024-03-24T03:17:22.688 INFO:teuthology.orchestra.run.smithi118.stdout: "level": "advanced", 2024-03-24T03:17:22.688 INFO:teuthology.orchestra.run.smithi118.stdout: "flags": 0, 2024-03-24T03:17:22.688 INFO:teuthology.orchestra.run.smithi118.stdout: "default_value": "10", 2024-03-24T03:17:22.688 INFO:teuthology.orchestra.run.smithi118.stdout: "min": "", 2024-03-24T03:17:22.688 INFO:teuthology.orchestra.run.smithi118.stdout: "max": "", 2024-03-24T03:17:22.688 INFO:teuthology.orchestra.run.smithi118.stdout: "enum_allowed": [], 2024-03-24T03:17:22.689 INFO:teuthology.orchestra.run.smithi118.stdout: "desc": "", 2024-03-24T03:17:22.689 INFO:teuthology.orchestra.run.smithi118.stdout: "long_desc": "", 2024-03-24T03:17:22.689 INFO:teuthology.orchestra.run.smithi118.stdout: "tags": [], 2024-03-24T03:17:22.689 INFO:teuthology.orchestra.run.smithi118.stdout: "see_also": [] 2024-03-24T03:17:22.689 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-24T03:17:22.689 INFO:teuthology.orchestra.run.smithi118.stdout: "PWD_POLICY_MIN_LENGTH": { 2024-03-24T03:17:22.689 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "PWD_POLICY_MIN_LENGTH", 2024-03-24T03:17:22.689 INFO:teuthology.orchestra.run.smithi118.stdout: "type": "int", 2024-03-24T03:17:22.690 INFO:teuthology.orchestra.run.smithi118.stdout: "level": "advanced", 2024-03-24T03:17:22.690 INFO:teuthology.orchestra.run.smithi118.stdout: "flags": 0, 2024-03-24T03:17:22.690 INFO:teuthology.orchestra.run.smithi118.stdout: "default_value": "8", 2024-03-24T03:17:22.690 INFO:teuthology.orchestra.run.smithi118.stdout: "min": "", 2024-03-24T03:17:22.690 INFO:teuthology.orchestra.run.smithi118.stdout: "max": "", 2024-03-24T03:17:22.690 INFO:teuthology.orchestra.run.smithi118.stdout: "enum_allowed": [], 2024-03-24T03:17:22.690 INFO:teuthology.orchestra.run.smithi118.stdout: "desc": "", 2024-03-24T03:17:22.690 INFO:teuthology.orchestra.run.smithi118.stdout: "long_desc": "", 2024-03-24T03:17:22.691 INFO:teuthology.orchestra.run.smithi118.stdout: "tags": [], 2024-03-24T03:17:22.691 INFO:teuthology.orchestra.run.smithi118.stdout: "see_also": [] 2024-03-24T03:17:22.691 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-24T03:17:22.691 INFO:teuthology.orchestra.run.smithi118.stdout: "REST_REQUESTS_TIMEOUT": { 2024-03-24T03:17:22.691 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "REST_REQUESTS_TIMEOUT", 2024-03-24T03:17:22.691 INFO:teuthology.orchestra.run.smithi118.stdout: "type": "int", 2024-03-24T03:17:22.691 INFO:teuthology.orchestra.run.smithi118.stdout: "level": "advanced", 2024-03-24T03:17:22.692 INFO:teuthology.orchestra.run.smithi118.stdout: "flags": 0, 2024-03-24T03:17:22.692 INFO:teuthology.orchestra.run.smithi118.stdout: "default_value": "45", 2024-03-24T03:17:22.692 INFO:teuthology.orchestra.run.smithi118.stdout: "min": "", 2024-03-24T03:17:22.692 INFO:teuthology.orchestra.run.smithi118.stdout: "max": "", 2024-03-24T03:17:22.692 INFO:teuthology.orchestra.run.smithi118.stdout: "enum_allowed": [], 2024-03-24T03:17:22.692 INFO:teuthology.orchestra.run.smithi118.stdout: "desc": "", 2024-03-24T03:17:22.692 INFO:teuthology.orchestra.run.smithi118.stdout: "long_desc": "", 2024-03-24T03:17:22.692 INFO:teuthology.orchestra.run.smithi118.stdout: "tags": [], 2024-03-24T03:17:22.692 INFO:teuthology.orchestra.run.smithi118.stdout: "see_also": [] 2024-03-24T03:17:22.693 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-24T03:17:22.693 INFO:teuthology.orchestra.run.smithi118.stdout: "RGW_API_ACCESS_KEY": { 2024-03-24T03:17:22.693 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "RGW_API_ACCESS_KEY", 2024-03-24T03:17:22.693 INFO:teuthology.orchestra.run.smithi118.stdout: "type": "str", 2024-03-24T03:17:22.693 INFO:teuthology.orchestra.run.smithi118.stdout: "level": "advanced", 2024-03-24T03:17:22.693 INFO:teuthology.orchestra.run.smithi118.stdout: "flags": 0, 2024-03-24T03:17:22.693 INFO:teuthology.orchestra.run.smithi118.stdout: "default_value": "", 2024-03-24T03:17:22.693 INFO:teuthology.orchestra.run.smithi118.stdout: "min": "", 2024-03-24T03:17:22.694 INFO:teuthology.orchestra.run.smithi118.stdout: "max": "", 2024-03-24T03:17:22.694 INFO:teuthology.orchestra.run.smithi118.stdout: "enum_allowed": [], 2024-03-24T03:17:22.694 INFO:teuthology.orchestra.run.smithi118.stdout: "desc": "", 2024-03-24T03:17:22.694 INFO:teuthology.orchestra.run.smithi118.stdout: "long_desc": "", 2024-03-24T03:17:22.694 INFO:teuthology.orchestra.run.smithi118.stdout: "tags": [], 2024-03-24T03:17:22.694 INFO:teuthology.orchestra.run.smithi118.stdout: "see_also": [] 2024-03-24T03:17:22.694 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-24T03:17:22.694 INFO:teuthology.orchestra.run.smithi118.stdout: "RGW_API_ADMIN_RESOURCE": { 2024-03-24T03:17:22.695 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "RGW_API_ADMIN_RESOURCE", 2024-03-24T03:17:22.695 INFO:teuthology.orchestra.run.smithi118.stdout: "type": "str", 2024-03-24T03:17:22.695 INFO:teuthology.orchestra.run.smithi118.stdout: "level": "advanced", 2024-03-24T03:17:22.695 INFO:teuthology.orchestra.run.smithi118.stdout: "flags": 0, 2024-03-24T03:17:22.695 INFO:teuthology.orchestra.run.smithi118.stdout: "default_value": "admin", 2024-03-24T03:17:22.695 INFO:teuthology.orchestra.run.smithi118.stdout: "min": "", 2024-03-24T03:17:22.695 INFO:teuthology.orchestra.run.smithi118.stdout: "max": "", 2024-03-24T03:17:22.695 INFO:teuthology.orchestra.run.smithi118.stdout: "enum_allowed": [], 2024-03-24T03:17:22.696 INFO:teuthology.orchestra.run.smithi118.stdout: "desc": "", 2024-03-24T03:17:22.696 INFO:teuthology.orchestra.run.smithi118.stdout: "long_desc": "", 2024-03-24T03:17:22.696 INFO:teuthology.orchestra.run.smithi118.stdout: "tags": [], 2024-03-24T03:17:22.696 INFO:teuthology.orchestra.run.smithi118.stdout: "see_also": [] 2024-03-24T03:17:22.696 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-24T03:17:22.696 INFO:teuthology.orchestra.run.smithi118.stdout: "RGW_API_SECRET_KEY": { 2024-03-24T03:17:22.696 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "RGW_API_SECRET_KEY", 2024-03-24T03:17:22.697 INFO:teuthology.orchestra.run.smithi118.stdout: "type": "str", 2024-03-24T03:17:22.697 INFO:teuthology.orchestra.run.smithi118.stdout: "level": "advanced", 2024-03-24T03:17:22.697 INFO:teuthology.orchestra.run.smithi118.stdout: "flags": 0, 2024-03-24T03:17:22.697 INFO:teuthology.orchestra.run.smithi118.stdout: "default_value": "", 2024-03-24T03:17:22.698 INFO:teuthology.orchestra.run.smithi118.stdout: "min": "", 2024-03-24T03:17:22.698 INFO:teuthology.orchestra.run.smithi118.stdout: "max": "", 2024-03-24T03:17:22.698 INFO:teuthology.orchestra.run.smithi118.stdout: "enum_allowed": [], 2024-03-24T03:17:22.698 INFO:teuthology.orchestra.run.smithi118.stdout: "desc": "", 2024-03-24T03:17:22.698 INFO:teuthology.orchestra.run.smithi118.stdout: "long_desc": "", 2024-03-24T03:17:22.698 INFO:teuthology.orchestra.run.smithi118.stdout: "tags": [], 2024-03-24T03:17:22.698 INFO:teuthology.orchestra.run.smithi118.stdout: "see_also": [] 2024-03-24T03:17:22.698 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-24T03:17:22.699 INFO:teuthology.orchestra.run.smithi118.stdout: "RGW_API_SSL_VERIFY": { 2024-03-24T03:17:22.699 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "RGW_API_SSL_VERIFY", 2024-03-24T03:17:22.699 INFO:teuthology.orchestra.run.smithi118.stdout: "type": "bool", 2024-03-24T03:17:22.699 INFO:teuthology.orchestra.run.smithi118.stdout: "level": "advanced", 2024-03-24T03:17:22.699 INFO:teuthology.orchestra.run.smithi118.stdout: "flags": 0, 2024-03-24T03:17:22.699 INFO:teuthology.orchestra.run.smithi118.stdout: "default_value": "True", 2024-03-24T03:17:22.699 INFO:teuthology.orchestra.run.smithi118.stdout: "min": "", 2024-03-24T03:17:22.699 INFO:teuthology.orchestra.run.smithi118.stdout: "max": "", 2024-03-24T03:17:22.699 INFO:teuthology.orchestra.run.smithi118.stdout: "enum_allowed": [], 2024-03-24T03:17:22.700 INFO:teuthology.orchestra.run.smithi118.stdout: "desc": "", 2024-03-24T03:17:22.700 INFO:teuthology.orchestra.run.smithi118.stdout: "long_desc": "", 2024-03-24T03:17:22.700 INFO:teuthology.orchestra.run.smithi118.stdout: "tags": [], 2024-03-24T03:17:22.700 INFO:teuthology.orchestra.run.smithi118.stdout: "see_also": [] 2024-03-24T03:17:22.700 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-24T03:17:22.700 INFO:teuthology.orchestra.run.smithi118.stdout: "UNSAFE_TLS_v1_2": { 2024-03-24T03:17:22.700 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "UNSAFE_TLS_v1_2", 2024-03-24T03:17:22.700 INFO:teuthology.orchestra.run.smithi118.stdout: "type": "bool", 2024-03-24T03:17:22.700 INFO:teuthology.orchestra.run.smithi118.stdout: "level": "advanced", 2024-03-24T03:17:22.701 INFO:teuthology.orchestra.run.smithi118.stdout: "flags": 0, 2024-03-24T03:17:22.701 INFO:teuthology.orchestra.run.smithi118.stdout: "default_value": "False", 2024-03-24T03:17:22.701 INFO:teuthology.orchestra.run.smithi118.stdout: "min": "", 2024-03-24T03:17:22.701 INFO:teuthology.orchestra.run.smithi118.stdout: "max": "", 2024-03-24T03:17:22.701 INFO:teuthology.orchestra.run.smithi118.stdout: "enum_allowed": [], 2024-03-24T03:17:22.701 INFO:teuthology.orchestra.run.smithi118.stdout: "desc": "", 2024-03-24T03:17:22.701 INFO:teuthology.orchestra.run.smithi118.stdout: "long_desc": "", 2024-03-24T03:17:22.701 INFO:teuthology.orchestra.run.smithi118.stdout: "tags": [], 2024-03-24T03:17:22.701 INFO:teuthology.orchestra.run.smithi118.stdout: "see_also": [] 2024-03-24T03:17:22.702 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-24T03:17:22.702 INFO:teuthology.orchestra.run.smithi118.stdout: "USER_PWD_EXPIRATION_SPAN": { 2024-03-24T03:17:22.702 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "USER_PWD_EXPIRATION_SPAN", 2024-03-24T03:17:22.702 INFO:teuthology.orchestra.run.smithi118.stdout: "type": "int", 2024-03-24T03:17:22.702 INFO:teuthology.orchestra.run.smithi118.stdout: "level": "advanced", 2024-03-24T03:17:22.702 INFO:teuthology.orchestra.run.smithi118.stdout: "flags": 0, 2024-03-24T03:17:22.702 INFO:teuthology.orchestra.run.smithi118.stdout: "default_value": "0", 2024-03-24T03:17:22.702 INFO:teuthology.orchestra.run.smithi118.stdout: "min": "", 2024-03-24T03:17:22.703 INFO:teuthology.orchestra.run.smithi118.stdout: "max": "", 2024-03-24T03:17:22.703 INFO:teuthology.orchestra.run.smithi118.stdout: "enum_allowed": [], 2024-03-24T03:17:22.703 INFO:teuthology.orchestra.run.smithi118.stdout: "desc": "", 2024-03-24T03:17:22.703 INFO:teuthology.orchestra.run.smithi118.stdout: "long_desc": "", 2024-03-24T03:17:22.703 INFO:teuthology.orchestra.run.smithi118.stdout: "tags": [], 2024-03-24T03:17:22.703 INFO:teuthology.orchestra.run.smithi118.stdout: "see_also": [] 2024-03-24T03:17:22.703 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-24T03:17:22.703 INFO:teuthology.orchestra.run.smithi118.stdout: "USER_PWD_EXPIRATION_WARNING_1": { 2024-03-24T03:17:22.704 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "USER_PWD_EXPIRATION_WARNING_1", 2024-03-24T03:17:22.704 INFO:teuthology.orchestra.run.smithi118.stdout: "type": "int", 2024-03-24T03:17:22.704 INFO:teuthology.orchestra.run.smithi118.stdout: "level": "advanced", 2024-03-24T03:17:22.704 INFO:teuthology.orchestra.run.smithi118.stdout: "flags": 0, 2024-03-24T03:17:22.704 INFO:teuthology.orchestra.run.smithi118.stdout: "default_value": "10", 2024-03-24T03:17:22.704 INFO:teuthology.orchestra.run.smithi118.stdout: "min": "", 2024-03-24T03:17:22.704 INFO:teuthology.orchestra.run.smithi118.stdout: "max": "", 2024-03-24T03:17:22.705 INFO:teuthology.orchestra.run.smithi118.stdout: "enum_allowed": [], 2024-03-24T03:17:22.705 INFO:teuthology.orchestra.run.smithi118.stdout: "desc": "", 2024-03-24T03:17:22.705 INFO:teuthology.orchestra.run.smithi118.stdout: "long_desc": "", 2024-03-24T03:17:22.705 INFO:teuthology.orchestra.run.smithi118.stdout: "tags": [], 2024-03-24T03:17:22.705 INFO:teuthology.orchestra.run.smithi118.stdout: "see_also": [] 2024-03-24T03:17:22.705 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-24T03:17:22.705 INFO:teuthology.orchestra.run.smithi118.stdout: "USER_PWD_EXPIRATION_WARNING_2": { 2024-03-24T03:17:22.705 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "USER_PWD_EXPIRATION_WARNING_2", 2024-03-24T03:17:22.705 INFO:teuthology.orchestra.run.smithi118.stdout: "type": "int", 2024-03-24T03:17:22.705 INFO:teuthology.orchestra.run.smithi118.stdout: "level": "advanced", 2024-03-24T03:17:22.706 INFO:teuthology.orchestra.run.smithi118.stdout: "flags": 0, 2024-03-24T03:17:22.706 INFO:teuthology.orchestra.run.smithi118.stdout: "default_value": "5", 2024-03-24T03:17:22.706 INFO:teuthology.orchestra.run.smithi118.stdout: "min": "", 2024-03-24T03:17:22.706 INFO:teuthology.orchestra.run.smithi118.stdout: "max": "", 2024-03-24T03:17:22.706 INFO:teuthology.orchestra.run.smithi118.stdout: "enum_allowed": [], 2024-03-24T03:17:22.706 INFO:teuthology.orchestra.run.smithi118.stdout: "desc": "", 2024-03-24T03:17:22.706 INFO:teuthology.orchestra.run.smithi118.stdout: "long_desc": "", 2024-03-24T03:17:22.706 INFO:teuthology.orchestra.run.smithi118.stdout: "tags": [], 2024-03-24T03:17:22.707 INFO:teuthology.orchestra.run.smithi118.stdout: "see_also": [] 2024-03-24T03:17:22.707 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-24T03:17:22.707 INFO:teuthology.orchestra.run.smithi118.stdout: "cross_origin_url": { 2024-03-24T03:17:22.707 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "cross_origin_url", 2024-03-24T03:17:22.707 INFO:teuthology.orchestra.run.smithi118.stdout: "type": "str", 2024-03-24T03:17:22.707 INFO:teuthology.orchestra.run.smithi118.stdout: "level": "advanced", 2024-03-24T03:17:22.707 INFO:teuthology.orchestra.run.smithi118.stdout: "flags": 0, 2024-03-24T03:17:22.707 INFO:teuthology.orchestra.run.smithi118.stdout: "default_value": "", 2024-03-24T03:17:22.707 INFO:teuthology.orchestra.run.smithi118.stdout: "min": "", 2024-03-24T03:17:22.707 INFO:teuthology.orchestra.run.smithi118.stdout: "max": "", 2024-03-24T03:17:22.708 INFO:teuthology.orchestra.run.smithi118.stdout: "enum_allowed": [], 2024-03-24T03:17:22.708 INFO:teuthology.orchestra.run.smithi118.stdout: "desc": "", 2024-03-24T03:17:22.708 INFO:teuthology.orchestra.run.smithi118.stdout: "long_desc": "", 2024-03-24T03:17:22.708 INFO:teuthology.orchestra.run.smithi118.stdout: "tags": [], 2024-03-24T03:17:22.708 INFO:teuthology.orchestra.run.smithi118.stdout: "see_also": [] 2024-03-24T03:17:22.708 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-24T03:17:22.708 INFO:teuthology.orchestra.run.smithi118.stdout: "crt_file": { 2024-03-24T03:17:22.708 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "crt_file", 2024-03-24T03:17:22.708 INFO:teuthology.orchestra.run.smithi118.stdout: "type": "str", 2024-03-24T03:17:22.709 INFO:teuthology.orchestra.run.smithi118.stdout: "level": "advanced", 2024-03-24T03:17:22.709 INFO:teuthology.orchestra.run.smithi118.stdout: "flags": 0, 2024-03-24T03:17:22.709 INFO:teuthology.orchestra.run.smithi118.stdout: "default_value": "", 2024-03-24T03:17:22.709 INFO:teuthology.orchestra.run.smithi118.stdout: "min": "", 2024-03-24T03:17:22.709 INFO:teuthology.orchestra.run.smithi118.stdout: "max": "", 2024-03-24T03:17:22.709 INFO:teuthology.orchestra.run.smithi118.stdout: "enum_allowed": [], 2024-03-24T03:17:22.709 INFO:teuthology.orchestra.run.smithi118.stdout: "desc": "", 2024-03-24T03:17:22.709 INFO:teuthology.orchestra.run.smithi118.stdout: "long_desc": "", 2024-03-24T03:17:22.709 INFO:teuthology.orchestra.run.smithi118.stdout: "tags": [], 2024-03-24T03:17:22.709 INFO:teuthology.orchestra.run.smithi118.stdout: "see_also": [] 2024-03-24T03:17:22.710 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-24T03:17:22.710 INFO:teuthology.orchestra.run.smithi118.stdout: "debug": { 2024-03-24T03:17:22.710 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "debug", 2024-03-24T03:17:22.710 INFO:teuthology.orchestra.run.smithi118.stdout: "type": "bool", 2024-03-24T03:17:22.710 INFO:teuthology.orchestra.run.smithi118.stdout: "level": "advanced", 2024-03-24T03:17:22.710 INFO:teuthology.orchestra.run.smithi118.stdout: "flags": 0, 2024-03-24T03:17:22.710 INFO:teuthology.orchestra.run.smithi118.stdout: "default_value": "False", 2024-03-24T03:17:22.710 INFO:teuthology.orchestra.run.smithi118.stdout: "min": "", 2024-03-24T03:17:22.711 INFO:teuthology.orchestra.run.smithi118.stdout: "max": "", 2024-03-24T03:17:22.711 INFO:teuthology.orchestra.run.smithi118.stdout: "enum_allowed": [], 2024-03-24T03:17:22.711 INFO:teuthology.orchestra.run.smithi118.stdout: "desc": "Enable/disable debug options", 2024-03-24T03:17:22.711 INFO:teuthology.orchestra.run.smithi118.stdout: "long_desc": "", 2024-03-24T03:17:22.711 INFO:teuthology.orchestra.run.smithi118.stdout: "tags": [], 2024-03-24T03:17:22.711 INFO:teuthology.orchestra.run.smithi118.stdout: "see_also": [] 2024-03-24T03:17:22.711 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-24T03:17:22.711 INFO:teuthology.orchestra.run.smithi118.stdout: "jwt_token_ttl": { 2024-03-24T03:17:22.712 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "jwt_token_ttl", 2024-03-24T03:17:22.712 INFO:teuthology.orchestra.run.smithi118.stdout: "type": "int", 2024-03-24T03:17:22.712 INFO:teuthology.orchestra.run.smithi118.stdout: "level": "advanced", 2024-03-24T03:17:22.712 INFO:teuthology.orchestra.run.smithi118.stdout: "flags": 0, 2024-03-24T03:17:22.712 INFO:teuthology.orchestra.run.smithi118.stdout: "default_value": "28800", 2024-03-24T03:17:22.712 INFO:teuthology.orchestra.run.smithi118.stdout: "min": "", 2024-03-24T03:17:22.712 INFO:teuthology.orchestra.run.smithi118.stdout: "max": "", 2024-03-24T03:17:22.712 INFO:teuthology.orchestra.run.smithi118.stdout: "enum_allowed": [], 2024-03-24T03:17:22.712 INFO:teuthology.orchestra.run.smithi118.stdout: "desc": "", 2024-03-24T03:17:22.713 INFO:teuthology.orchestra.run.smithi118.stdout: "long_desc": "", 2024-03-24T03:17:22.713 INFO:teuthology.orchestra.run.smithi118.stdout: "tags": [], 2024-03-24T03:17:22.713 INFO:teuthology.orchestra.run.smithi118.stdout: "see_also": [] 2024-03-24T03:17:22.713 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-24T03:17:22.713 INFO:teuthology.orchestra.run.smithi118.stdout: "key_file": { 2024-03-24T03:17:22.713 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "key_file", 2024-03-24T03:17:22.713 INFO:teuthology.orchestra.run.smithi118.stdout: "type": "str", 2024-03-24T03:17:22.713 INFO:teuthology.orchestra.run.smithi118.stdout: "level": "advanced", 2024-03-24T03:17:22.714 INFO:teuthology.orchestra.run.smithi118.stdout: "flags": 0, 2024-03-24T03:17:22.714 INFO:teuthology.orchestra.run.smithi118.stdout: "default_value": "", 2024-03-24T03:17:22.714 INFO:teuthology.orchestra.run.smithi118.stdout: "min": "", 2024-03-24T03:17:22.714 INFO:teuthology.orchestra.run.smithi118.stdout: "max": "", 2024-03-24T03:17:22.714 INFO:teuthology.orchestra.run.smithi118.stdout: "enum_allowed": [], 2024-03-24T03:17:22.714 INFO:teuthology.orchestra.run.smithi118.stdout: "desc": "", 2024-03-24T03:17:22.714 INFO:teuthology.orchestra.run.smithi118.stdout: "long_desc": "", 2024-03-24T03:17:22.714 INFO:teuthology.orchestra.run.smithi118.stdout: "tags": [], 2024-03-24T03:17:22.715 INFO:teuthology.orchestra.run.smithi118.stdout: "see_also": [] 2024-03-24T03:17:22.715 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-24T03:17:22.715 INFO:teuthology.orchestra.run.smithi118.stdout: "log_level": { 2024-03-24T03:17:22.715 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "log_level", 2024-03-24T03:17:22.715 INFO:teuthology.orchestra.run.smithi118.stdout: "type": "str", 2024-03-24T03:17:22.715 INFO:teuthology.orchestra.run.smithi118.stdout: "level": "advanced", 2024-03-24T03:17:22.715 INFO:teuthology.orchestra.run.smithi118.stdout: "flags": 1, 2024-03-24T03:17:22.715 INFO:teuthology.orchestra.run.smithi118.stdout: "default_value": "", 2024-03-24T03:17:22.716 INFO:teuthology.orchestra.run.smithi118.stdout: "min": "", 2024-03-24T03:17:22.716 INFO:teuthology.orchestra.run.smithi118.stdout: "max": "", 2024-03-24T03:17:22.716 INFO:teuthology.orchestra.run.smithi118.stdout: "enum_allowed": [ 2024-03-24T03:17:22.716 INFO:teuthology.orchestra.run.smithi118.stdout: "", 2024-03-24T03:17:22.716 INFO:teuthology.orchestra.run.smithi118.stdout: "critical", 2024-03-24T03:17:22.716 INFO:teuthology.orchestra.run.smithi118.stdout: "debug", 2024-03-24T03:17:22.716 INFO:teuthology.orchestra.run.smithi118.stdout: "error", 2024-03-24T03:17:22.716 INFO:teuthology.orchestra.run.smithi118.stdout: "info", 2024-03-24T03:17:22.717 INFO:teuthology.orchestra.run.smithi118.stdout: "warning" 2024-03-24T03:17:22.717 INFO:teuthology.orchestra.run.smithi118.stdout: ], 2024-03-24T03:17:22.717 INFO:teuthology.orchestra.run.smithi118.stdout: "desc": "", 2024-03-24T03:17:22.717 INFO:teuthology.orchestra.run.smithi118.stdout: "long_desc": "", 2024-03-24T03:17:22.717 INFO:teuthology.orchestra.run.smithi118.stdout: "tags": [], 2024-03-24T03:17:22.717 INFO:teuthology.orchestra.run.smithi118.stdout: "see_also": [] 2024-03-24T03:17:22.717 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-24T03:17:22.717 INFO:teuthology.orchestra.run.smithi118.stdout: "log_to_cluster": { 2024-03-24T03:17:22.718 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "log_to_cluster", 2024-03-24T03:17:22.718 INFO:teuthology.orchestra.run.smithi118.stdout: "type": "bool", 2024-03-24T03:17:22.718 INFO:teuthology.orchestra.run.smithi118.stdout: "level": "advanced", 2024-03-24T03:17:22.718 INFO:teuthology.orchestra.run.smithi118.stdout: "flags": 1, 2024-03-24T03:17:22.718 INFO:teuthology.orchestra.run.smithi118.stdout: "default_value": "False", 2024-03-24T03:17:22.718 INFO:teuthology.orchestra.run.smithi118.stdout: "min": "", 2024-03-24T03:17:22.718 INFO:teuthology.orchestra.run.smithi118.stdout: "max": "", 2024-03-24T03:17:22.718 INFO:teuthology.orchestra.run.smithi118.stdout: "enum_allowed": [], 2024-03-24T03:17:22.718 INFO:teuthology.orchestra.run.smithi118.stdout: "desc": "", 2024-03-24T03:17:22.719 INFO:teuthology.orchestra.run.smithi118.stdout: "long_desc": "", 2024-03-24T03:17:22.719 INFO:teuthology.orchestra.run.smithi118.stdout: "tags": [], 2024-03-24T03:17:22.719 INFO:teuthology.orchestra.run.smithi118.stdout: "see_also": [] 2024-03-24T03:17:22.719 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-24T03:17:22.719 INFO:teuthology.orchestra.run.smithi118.stdout: "log_to_cluster_level": { 2024-03-24T03:17:22.719 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "log_to_cluster_level", 2024-03-24T03:17:22.719 INFO:teuthology.orchestra.run.smithi118.stdout: "type": "str", 2024-03-24T03:17:22.719 INFO:teuthology.orchestra.run.smithi118.stdout: "level": "advanced", 2024-03-24T03:17:22.719 INFO:teuthology.orchestra.run.smithi118.stdout: "flags": 1, 2024-03-24T03:17:22.720 INFO:teuthology.orchestra.run.smithi118.stdout: "default_value": "info", 2024-03-24T03:17:22.720 INFO:teuthology.orchestra.run.smithi118.stdout: "min": "", 2024-03-24T03:17:22.720 INFO:teuthology.orchestra.run.smithi118.stdout: "max": "", 2024-03-24T03:17:22.720 INFO:teuthology.orchestra.run.smithi118.stdout: "enum_allowed": [ 2024-03-24T03:17:22.720 INFO:teuthology.orchestra.run.smithi118.stdout: "", 2024-03-24T03:17:22.720 INFO:teuthology.orchestra.run.smithi118.stdout: "critical", 2024-03-24T03:17:22.720 INFO:teuthology.orchestra.run.smithi118.stdout: "debug", 2024-03-24T03:17:22.720 INFO:teuthology.orchestra.run.smithi118.stdout: "error", 2024-03-24T03:17:22.720 INFO:teuthology.orchestra.run.smithi118.stdout: "info", 2024-03-24T03:17:22.721 INFO:teuthology.orchestra.run.smithi118.stdout: "warning" 2024-03-24T03:17:22.721 INFO:teuthology.orchestra.run.smithi118.stdout: ], 2024-03-24T03:17:22.721 INFO:teuthology.orchestra.run.smithi118.stdout: "desc": "", 2024-03-24T03:17:22.721 INFO:teuthology.orchestra.run.smithi118.stdout: "long_desc": "", 2024-03-24T03:17:22.721 INFO:teuthology.orchestra.run.smithi118.stdout: "tags": [], 2024-03-24T03:17:22.721 INFO:teuthology.orchestra.run.smithi118.stdout: "see_also": [] 2024-03-24T03:17:22.721 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-24T03:17:22.721 INFO:teuthology.orchestra.run.smithi118.stdout: "log_to_file": { 2024-03-24T03:17:22.721 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "log_to_file", 2024-03-24T03:17:22.722 INFO:teuthology.orchestra.run.smithi118.stdout: "type": "bool", 2024-03-24T03:17:22.722 INFO:teuthology.orchestra.run.smithi118.stdout: "level": "advanced", 2024-03-24T03:17:22.722 INFO:teuthology.orchestra.run.smithi118.stdout: "flags": 1, 2024-03-24T03:17:22.722 INFO:teuthology.orchestra.run.smithi118.stdout: "default_value": "False", 2024-03-24T03:17:22.722 INFO:teuthology.orchestra.run.smithi118.stdout: "min": "", 2024-03-24T03:17:22.722 INFO:teuthology.orchestra.run.smithi118.stdout: "max": "", 2024-03-24T03:17:22.722 INFO:teuthology.orchestra.run.smithi118.stdout: "enum_allowed": [], 2024-03-24T03:17:22.723 INFO:teuthology.orchestra.run.smithi118.stdout: "desc": "", 2024-03-24T03:17:22.723 INFO:teuthology.orchestra.run.smithi118.stdout: "long_desc": "", 2024-03-24T03:17:22.723 INFO:teuthology.orchestra.run.smithi118.stdout: "tags": [], 2024-03-24T03:17:22.723 INFO:teuthology.orchestra.run.smithi118.stdout: "see_also": [] 2024-03-24T03:17:22.723 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-24T03:17:22.723 INFO:teuthology.orchestra.run.smithi118.stdout: "motd": { 2024-03-24T03:17:22.723 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "motd", 2024-03-24T03:17:22.723 INFO:teuthology.orchestra.run.smithi118.stdout: "type": "str", 2024-03-24T03:17:22.723 INFO:teuthology.orchestra.run.smithi118.stdout: "level": "advanced", 2024-03-24T03:17:22.723 INFO:teuthology.orchestra.run.smithi118.stdout: "flags": 0, 2024-03-24T03:17:22.724 INFO:teuthology.orchestra.run.smithi118.stdout: "default_value": "", 2024-03-24T03:17:22.724 INFO:teuthology.orchestra.run.smithi118.stdout: "min": "", 2024-03-24T03:17:22.724 INFO:teuthology.orchestra.run.smithi118.stdout: "max": "", 2024-03-24T03:17:22.724 INFO:teuthology.orchestra.run.smithi118.stdout: "enum_allowed": [], 2024-03-24T03:17:22.724 INFO:teuthology.orchestra.run.smithi118.stdout: "desc": "The message of the day", 2024-03-24T03:17:22.724 INFO:teuthology.orchestra.run.smithi118.stdout: "long_desc": "", 2024-03-24T03:17:22.724 INFO:teuthology.orchestra.run.smithi118.stdout: "tags": [], 2024-03-24T03:17:22.724 INFO:teuthology.orchestra.run.smithi118.stdout: "see_also": [] 2024-03-24T03:17:22.725 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-24T03:17:22.725 INFO:teuthology.orchestra.run.smithi118.stdout: "redirect_resolve_ip_addr": { 2024-03-24T03:17:22.725 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "redirect_resolve_ip_addr", 2024-03-24T03:17:22.725 INFO:teuthology.orchestra.run.smithi118.stdout: "type": "bool", 2024-03-24T03:17:22.725 INFO:teuthology.orchestra.run.smithi118.stdout: "level": "advanced", 2024-03-24T03:17:22.725 INFO:teuthology.orchestra.run.smithi118.stdout: "flags": 0, 2024-03-24T03:17:22.725 INFO:teuthology.orchestra.run.smithi118.stdout: "default_value": "False", 2024-03-24T03:17:22.725 INFO:teuthology.orchestra.run.smithi118.stdout: "min": "", 2024-03-24T03:17:22.725 INFO:teuthology.orchestra.run.smithi118.stdout: "max": "", 2024-03-24T03:17:22.725 INFO:teuthology.orchestra.run.smithi118.stdout: "enum_allowed": [], 2024-03-24T03:17:22.726 INFO:teuthology.orchestra.run.smithi118.stdout: "desc": "", 2024-03-24T03:17:22.726 INFO:teuthology.orchestra.run.smithi118.stdout: "long_desc": "", 2024-03-24T03:17:22.726 INFO:teuthology.orchestra.run.smithi118.stdout: "tags": [], 2024-03-24T03:17:22.726 INFO:teuthology.orchestra.run.smithi118.stdout: "see_also": [] 2024-03-24T03:17:22.726 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-24T03:17:22.726 INFO:teuthology.orchestra.run.smithi118.stdout: "server_addr": { 2024-03-24T03:17:22.726 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "server_addr", 2024-03-24T03:17:22.726 INFO:teuthology.orchestra.run.smithi118.stdout: "type": "str", 2024-03-24T03:17:22.727 INFO:teuthology.orchestra.run.smithi118.stdout: "level": "advanced", 2024-03-24T03:17:22.727 INFO:teuthology.orchestra.run.smithi118.stdout: "flags": 0, 2024-03-24T03:17:22.727 INFO:teuthology.orchestra.run.smithi118.stdout: "default_value": "::", 2024-03-24T03:17:22.727 INFO:teuthology.orchestra.run.smithi118.stdout: "min": "", 2024-03-24T03:17:22.727 INFO:teuthology.orchestra.run.smithi118.stdout: "max": "", 2024-03-24T03:17:22.727 INFO:teuthology.orchestra.run.smithi118.stdout: "enum_allowed": [], 2024-03-24T03:17:22.727 INFO:teuthology.orchestra.run.smithi118.stdout: "desc": "", 2024-03-24T03:17:22.727 INFO:teuthology.orchestra.run.smithi118.stdout: "long_desc": "", 2024-03-24T03:17:22.727 INFO:teuthology.orchestra.run.smithi118.stdout: "tags": [], 2024-03-24T03:17:22.727 INFO:teuthology.orchestra.run.smithi118.stdout: "see_also": [] 2024-03-24T03:17:22.728 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-24T03:17:22.728 INFO:teuthology.orchestra.run.smithi118.stdout: "server_port": { 2024-03-24T03:17:22.728 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "server_port", 2024-03-24T03:17:22.728 INFO:teuthology.orchestra.run.smithi118.stdout: "type": "int", 2024-03-24T03:17:22.728 INFO:teuthology.orchestra.run.smithi118.stdout: "level": "advanced", 2024-03-24T03:17:22.728 INFO:teuthology.orchestra.run.smithi118.stdout: "flags": 0, 2024-03-24T03:17:22.728 INFO:teuthology.orchestra.run.smithi118.stdout: "default_value": "8080", 2024-03-24T03:17:22.728 INFO:teuthology.orchestra.run.smithi118.stdout: "min": "", 2024-03-24T03:17:22.728 INFO:teuthology.orchestra.run.smithi118.stdout: "max": "", 2024-03-24T03:17:22.729 INFO:teuthology.orchestra.run.smithi118.stdout: "enum_allowed": [], 2024-03-24T03:17:22.729 INFO:teuthology.orchestra.run.smithi118.stdout: "desc": "", 2024-03-24T03:17:22.729 INFO:teuthology.orchestra.run.smithi118.stdout: "long_desc": "", 2024-03-24T03:17:22.729 INFO:teuthology.orchestra.run.smithi118.stdout: "tags": [], 2024-03-24T03:17:22.729 INFO:teuthology.orchestra.run.smithi118.stdout: "see_also": [] 2024-03-24T03:17:22.729 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-24T03:17:22.729 INFO:teuthology.orchestra.run.smithi118.stdout: "ssl": { 2024-03-24T03:17:22.729 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "ssl", 2024-03-24T03:17:22.729 INFO:teuthology.orchestra.run.smithi118.stdout: "type": "bool", 2024-03-24T03:17:22.729 INFO:teuthology.orchestra.run.smithi118.stdout: "level": "advanced", 2024-03-24T03:17:22.730 INFO:teuthology.orchestra.run.smithi118.stdout: "flags": 0, 2024-03-24T03:17:22.730 INFO:teuthology.orchestra.run.smithi118.stdout: "default_value": "True", 2024-03-24T03:17:22.730 INFO:teuthology.orchestra.run.smithi118.stdout: "min": "", 2024-03-24T03:17:22.730 INFO:teuthology.orchestra.run.smithi118.stdout: "max": "", 2024-03-24T03:17:22.730 INFO:teuthology.orchestra.run.smithi118.stdout: "enum_allowed": [], 2024-03-24T03:17:22.730 INFO:teuthology.orchestra.run.smithi118.stdout: "desc": "", 2024-03-24T03:17:22.730 INFO:teuthology.orchestra.run.smithi118.stdout: "long_desc": "", 2024-03-24T03:17:22.730 INFO:teuthology.orchestra.run.smithi118.stdout: "tags": [], 2024-03-24T03:17:22.730 INFO:teuthology.orchestra.run.smithi118.stdout: "see_also": [] 2024-03-24T03:17:22.731 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-24T03:17:22.731 INFO:teuthology.orchestra.run.smithi118.stdout: "ssl_server_port": { 2024-03-24T03:17:22.731 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "ssl_server_port", 2024-03-24T03:17:22.731 INFO:teuthology.orchestra.run.smithi118.stdout: "type": "int", 2024-03-24T03:17:22.731 INFO:teuthology.orchestra.run.smithi118.stdout: "level": "advanced", 2024-03-24T03:17:22.731 INFO:teuthology.orchestra.run.smithi118.stdout: "flags": 0, 2024-03-24T03:17:22.731 INFO:teuthology.orchestra.run.smithi118.stdout: "default_value": "8443", 2024-03-24T03:17:22.731 INFO:teuthology.orchestra.run.smithi118.stdout: "min": "", 2024-03-24T03:17:22.731 INFO:teuthology.orchestra.run.smithi118.stdout: "max": "", 2024-03-24T03:17:22.732 INFO:teuthology.orchestra.run.smithi118.stdout: "enum_allowed": [], 2024-03-24T03:17:22.732 INFO:teuthology.orchestra.run.smithi118.stdout: "desc": "", 2024-03-24T03:17:22.732 INFO:teuthology.orchestra.run.smithi118.stdout: "long_desc": "", 2024-03-24T03:17:22.732 INFO:teuthology.orchestra.run.smithi118.stdout: "tags": [], 2024-03-24T03:17:22.732 INFO:teuthology.orchestra.run.smithi118.stdout: "see_also": [] 2024-03-24T03:17:22.732 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-24T03:17:22.732 INFO:teuthology.orchestra.run.smithi118.stdout: "standby_behaviour": { 2024-03-24T03:17:22.732 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "standby_behaviour", 2024-03-24T03:17:22.733 INFO:teuthology.orchestra.run.smithi118.stdout: "type": "str", 2024-03-24T03:17:22.733 INFO:teuthology.orchestra.run.smithi118.stdout: "level": "advanced", 2024-03-24T03:17:22.733 INFO:teuthology.orchestra.run.smithi118.stdout: "flags": 0, 2024-03-24T03:17:22.733 INFO:teuthology.orchestra.run.smithi118.stdout: "default_value": "redirect", 2024-03-24T03:17:22.733 INFO:teuthology.orchestra.run.smithi118.stdout: "min": "", 2024-03-24T03:17:22.733 INFO:teuthology.orchestra.run.smithi118.stdout: "max": "", 2024-03-24T03:17:22.733 INFO:teuthology.orchestra.run.smithi118.stdout: "enum_allowed": [ 2024-03-24T03:17:22.733 INFO:teuthology.orchestra.run.smithi118.stdout: "error", 2024-03-24T03:17:22.734 INFO:teuthology.orchestra.run.smithi118.stdout: "redirect" 2024-03-24T03:17:22.734 INFO:teuthology.orchestra.run.smithi118.stdout: ], 2024-03-24T03:17:22.734 INFO:teuthology.orchestra.run.smithi118.stdout: "desc": "", 2024-03-24T03:17:22.734 INFO:teuthology.orchestra.run.smithi118.stdout: "long_desc": "", 2024-03-24T03:17:22.734 INFO:teuthology.orchestra.run.smithi118.stdout: "tags": [], 2024-03-24T03:17:22.734 INFO:teuthology.orchestra.run.smithi118.stdout: "see_also": [] 2024-03-24T03:17:22.734 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-24T03:17:22.734 INFO:teuthology.orchestra.run.smithi118.stdout: "standby_error_status_code": { 2024-03-24T03:17:22.735 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "standby_error_status_code", 2024-03-24T03:17:22.735 INFO:teuthology.orchestra.run.smithi118.stdout: "type": "int", 2024-03-24T03:17:22.735 INFO:teuthology.orchestra.run.smithi118.stdout: "level": "advanced", 2024-03-24T03:17:22.735 INFO:teuthology.orchestra.run.smithi118.stdout: "flags": 0, 2024-03-24T03:17:22.735 INFO:teuthology.orchestra.run.smithi118.stdout: "default_value": "500", 2024-03-24T03:17:22.735 INFO:teuthology.orchestra.run.smithi118.stdout: "min": "400", 2024-03-24T03:17:22.735 INFO:teuthology.orchestra.run.smithi118.stdout: "max": "599", 2024-03-24T03:17:22.735 INFO:teuthology.orchestra.run.smithi118.stdout: "enum_allowed": [], 2024-03-24T03:17:22.735 INFO:teuthology.orchestra.run.smithi118.stdout: "desc": "", 2024-03-24T03:17:22.736 INFO:teuthology.orchestra.run.smithi118.stdout: "long_desc": "", 2024-03-24T03:17:22.736 INFO:teuthology.orchestra.run.smithi118.stdout: "tags": [], 2024-03-24T03:17:22.736 INFO:teuthology.orchestra.run.smithi118.stdout: "see_also": [] 2024-03-24T03:17:22.736 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-24T03:17:22.736 INFO:teuthology.orchestra.run.smithi118.stdout: "url_prefix": { 2024-03-24T03:17:22.736 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "url_prefix", 2024-03-24T03:17:22.736 INFO:teuthology.orchestra.run.smithi118.stdout: "type": "str", 2024-03-24T03:17:22.736 INFO:teuthology.orchestra.run.smithi118.stdout: "level": "advanced", 2024-03-24T03:17:22.736 INFO:teuthology.orchestra.run.smithi118.stdout: "flags": 0, 2024-03-24T03:17:22.737 INFO:teuthology.orchestra.run.smithi118.stdout: "default_value": "", 2024-03-24T03:17:22.737 INFO:teuthology.orchestra.run.smithi118.stdout: "min": "", 2024-03-24T03:17:22.737 INFO:teuthology.orchestra.run.smithi118.stdout: "max": "", 2024-03-24T03:17:22.737 INFO:teuthology.orchestra.run.smithi118.stdout: "enum_allowed": [], 2024-03-24T03:17:22.737 INFO:teuthology.orchestra.run.smithi118.stdout: "desc": "", 2024-03-24T03:17:22.737 INFO:teuthology.orchestra.run.smithi118.stdout: "long_desc": "", 2024-03-24T03:17:22.737 INFO:teuthology.orchestra.run.smithi118.stdout: "tags": [], 2024-03-24T03:17:22.737 INFO:teuthology.orchestra.run.smithi118.stdout: "see_also": [] 2024-03-24T03:17:22.737 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-03-24T03:17:22.738 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-03-24T03:17:22.738 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-24T03:17:22.738 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-24T03:17:22.738 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "devicehealth", 2024-03-24T03:17:22.738 INFO:teuthology.orchestra.run.smithi118.stdout: "can_run": true, 2024-03-24T03:17:22.738 INFO:teuthology.orchestra.run.smithi118.stdout: "error_string": "", 2024-03-24T03:17:22.738 INFO:teuthology.orchestra.run.smithi118.stdout: "module_options": { 2024-03-24T03:17:22.738 INFO:teuthology.orchestra.run.smithi118.stdout: "enable_monitoring": { 2024-03-24T03:17:22.739 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "enable_monitoring", 2024-03-24T03:17:22.739 INFO:teuthology.orchestra.run.smithi118.stdout: "type": "bool", 2024-03-24T03:17:22.739 INFO:teuthology.orchestra.run.smithi118.stdout: "level": "advanced", 2024-03-24T03:17:22.739 INFO:teuthology.orchestra.run.smithi118.stdout: "flags": 1, 2024-03-24T03:17:22.739 INFO:teuthology.orchestra.run.smithi118.stdout: "default_value": "True", 2024-03-24T03:17:22.739 INFO:teuthology.orchestra.run.smithi118.stdout: "min": "", 2024-03-24T03:17:22.739 INFO:teuthology.orchestra.run.smithi118.stdout: "max": "", 2024-03-24T03:17:22.739 INFO:teuthology.orchestra.run.smithi118.stdout: "enum_allowed": [], 2024-03-24T03:17:22.740 INFO:teuthology.orchestra.run.smithi118.stdout: "desc": "monitor device health metrics", 2024-03-24T03:17:22.740 INFO:teuthology.orchestra.run.smithi118.stdout: "long_desc": "", 2024-03-24T03:17:22.740 INFO:teuthology.orchestra.run.smithi118.stdout: "tags": [], 2024-03-24T03:17:22.740 INFO:teuthology.orchestra.run.smithi118.stdout: "see_also": [] 2024-03-24T03:17:22.740 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-24T03:17:22.740 INFO:teuthology.orchestra.run.smithi118.stdout: "log_level": { 2024-03-24T03:17:22.740 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "log_level", 2024-03-24T03:17:22.740 INFO:teuthology.orchestra.run.smithi118.stdout: "type": "str", 2024-03-24T03:17:22.740 INFO:teuthology.orchestra.run.smithi118.stdout: "level": "advanced", 2024-03-24T03:17:22.741 INFO:teuthology.orchestra.run.smithi118.stdout: "flags": 1, 2024-03-24T03:17:22.741 INFO:teuthology.orchestra.run.smithi118.stdout: "default_value": "", 2024-03-24T03:17:22.741 INFO:teuthology.orchestra.run.smithi118.stdout: "min": "", 2024-03-24T03:17:22.741 INFO:teuthology.orchestra.run.smithi118.stdout: "max": "", 2024-03-24T03:17:22.741 INFO:teuthology.orchestra.run.smithi118.stdout: "enum_allowed": [ 2024-03-24T03:17:22.741 INFO:teuthology.orchestra.run.smithi118.stdout: "", 2024-03-24T03:17:22.741 INFO:teuthology.orchestra.run.smithi118.stdout: "critical", 2024-03-24T03:17:22.741 INFO:teuthology.orchestra.run.smithi118.stdout: "debug", 2024-03-24T03:17:22.741 INFO:teuthology.orchestra.run.smithi118.stdout: "error", 2024-03-24T03:17:22.742 INFO:teuthology.orchestra.run.smithi118.stdout: "info", 2024-03-24T03:17:22.742 INFO:teuthology.orchestra.run.smithi118.stdout: "warning" 2024-03-24T03:17:22.742 INFO:teuthology.orchestra.run.smithi118.stdout: ], 2024-03-24T03:17:22.742 INFO:teuthology.orchestra.run.smithi118.stdout: "desc": "", 2024-03-24T03:17:22.742 INFO:teuthology.orchestra.run.smithi118.stdout: "long_desc": "", 2024-03-24T03:17:22.742 INFO:teuthology.orchestra.run.smithi118.stdout: "tags": [], 2024-03-24T03:17:22.742 INFO:teuthology.orchestra.run.smithi118.stdout: "see_also": [] 2024-03-24T03:17:22.742 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-24T03:17:22.743 INFO:teuthology.orchestra.run.smithi118.stdout: "log_to_cluster": { 2024-03-24T03:17:22.743 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "log_to_cluster", 2024-03-24T03:17:22.743 INFO:teuthology.orchestra.run.smithi118.stdout: "type": "bool", 2024-03-24T03:17:22.743 INFO:teuthology.orchestra.run.smithi118.stdout: "level": "advanced", 2024-03-24T03:17:22.743 INFO:teuthology.orchestra.run.smithi118.stdout: "flags": 1, 2024-03-24T03:17:22.743 INFO:teuthology.orchestra.run.smithi118.stdout: "default_value": "False", 2024-03-24T03:17:22.743 INFO:teuthology.orchestra.run.smithi118.stdout: "min": "", 2024-03-24T03:17:22.743 INFO:teuthology.orchestra.run.smithi118.stdout: "max": "", 2024-03-24T03:17:22.743 INFO:teuthology.orchestra.run.smithi118.stdout: "enum_allowed": [], 2024-03-24T03:17:22.744 INFO:teuthology.orchestra.run.smithi118.stdout: "desc": "", 2024-03-24T03:17:22.744 INFO:teuthology.orchestra.run.smithi118.stdout: "long_desc": "", 2024-03-24T03:17:22.744 INFO:teuthology.orchestra.run.smithi118.stdout: "tags": [], 2024-03-24T03:17:22.744 INFO:teuthology.orchestra.run.smithi118.stdout: "see_also": [] 2024-03-24T03:17:22.744 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-24T03:17:22.744 INFO:teuthology.orchestra.run.smithi118.stdout: "log_to_cluster_level": { 2024-03-24T03:17:22.744 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "log_to_cluster_level", 2024-03-24T03:17:22.744 INFO:teuthology.orchestra.run.smithi118.stdout: "type": "str", 2024-03-24T03:17:22.744 INFO:teuthology.orchestra.run.smithi118.stdout: "level": "advanced", 2024-03-24T03:17:22.745 INFO:teuthology.orchestra.run.smithi118.stdout: "flags": 1, 2024-03-24T03:17:22.745 INFO:teuthology.orchestra.run.smithi118.stdout: "default_value": "info", 2024-03-24T03:17:22.745 INFO:teuthology.orchestra.run.smithi118.stdout: "min": "", 2024-03-24T03:17:22.745 INFO:teuthology.orchestra.run.smithi118.stdout: "max": "", 2024-03-24T03:17:22.745 INFO:teuthology.orchestra.run.smithi118.stdout: "enum_allowed": [ 2024-03-24T03:17:22.745 INFO:teuthology.orchestra.run.smithi118.stdout: "", 2024-03-24T03:17:22.745 INFO:teuthology.orchestra.run.smithi118.stdout: "critical", 2024-03-24T03:17:22.745 INFO:teuthology.orchestra.run.smithi118.stdout: "debug", 2024-03-24T03:17:22.745 INFO:teuthology.orchestra.run.smithi118.stdout: "error", 2024-03-24T03:17:22.746 INFO:teuthology.orchestra.run.smithi118.stdout: "info", 2024-03-24T03:17:22.746 INFO:teuthology.orchestra.run.smithi118.stdout: "warning" 2024-03-24T03:17:22.746 INFO:teuthology.orchestra.run.smithi118.stdout: ], 2024-03-24T03:17:22.746 INFO:teuthology.orchestra.run.smithi118.stdout: "desc": "", 2024-03-24T03:17:22.746 INFO:teuthology.orchestra.run.smithi118.stdout: "long_desc": "", 2024-03-24T03:17:22.746 INFO:teuthology.orchestra.run.smithi118.stdout: "tags": [], 2024-03-24T03:17:22.746 INFO:teuthology.orchestra.run.smithi118.stdout: "see_also": [] 2024-03-24T03:17:22.746 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-24T03:17:22.747 INFO:teuthology.orchestra.run.smithi118.stdout: "log_to_file": { 2024-03-24T03:17:22.747 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "log_to_file", 2024-03-24T03:17:22.747 INFO:teuthology.orchestra.run.smithi118.stdout: "type": "bool", 2024-03-24T03:17:22.747 INFO:teuthology.orchestra.run.smithi118.stdout: "level": "advanced", 2024-03-24T03:17:22.747 INFO:teuthology.orchestra.run.smithi118.stdout: "flags": 1, 2024-03-24T03:17:22.747 INFO:teuthology.orchestra.run.smithi118.stdout: "default_value": "False", 2024-03-24T03:17:22.747 INFO:teuthology.orchestra.run.smithi118.stdout: "min": "", 2024-03-24T03:17:22.747 INFO:teuthology.orchestra.run.smithi118.stdout: "max": "", 2024-03-24T03:17:22.747 INFO:teuthology.orchestra.run.smithi118.stdout: "enum_allowed": [], 2024-03-24T03:17:22.747 INFO:teuthology.orchestra.run.smithi118.stdout: "desc": "", 2024-03-24T03:17:22.748 INFO:teuthology.orchestra.run.smithi118.stdout: "long_desc": "", 2024-03-24T03:17:22.748 INFO:teuthology.orchestra.run.smithi118.stdout: "tags": [], 2024-03-24T03:17:22.748 INFO:teuthology.orchestra.run.smithi118.stdout: "see_also": [] 2024-03-24T03:17:22.748 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-24T03:17:22.748 INFO:teuthology.orchestra.run.smithi118.stdout: "mark_out_threshold": { 2024-03-24T03:17:22.748 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "mark_out_threshold", 2024-03-24T03:17:22.748 INFO:teuthology.orchestra.run.smithi118.stdout: "type": "secs", 2024-03-24T03:17:22.748 INFO:teuthology.orchestra.run.smithi118.stdout: "level": "advanced", 2024-03-24T03:17:22.748 INFO:teuthology.orchestra.run.smithi118.stdout: "flags": 1, 2024-03-24T03:17:22.748 INFO:teuthology.orchestra.run.smithi118.stdout: "default_value": "2419200", 2024-03-24T03:17:22.749 INFO:teuthology.orchestra.run.smithi118.stdout: "min": "", 2024-03-24T03:17:22.749 INFO:teuthology.orchestra.run.smithi118.stdout: "max": "", 2024-03-24T03:17:22.749 INFO:teuthology.orchestra.run.smithi118.stdout: "enum_allowed": [], 2024-03-24T03:17:22.749 INFO:teuthology.orchestra.run.smithi118.stdout: "desc": "automatically mark OSD if it may fail before this long", 2024-03-24T03:17:22.749 INFO:teuthology.orchestra.run.smithi118.stdout: "long_desc": "", 2024-03-24T03:17:22.749 INFO:teuthology.orchestra.run.smithi118.stdout: "tags": [], 2024-03-24T03:17:22.749 INFO:teuthology.orchestra.run.smithi118.stdout: "see_also": [] 2024-03-24T03:17:22.749 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-24T03:17:22.749 INFO:teuthology.orchestra.run.smithi118.stdout: "pool_name": { 2024-03-24T03:17:22.750 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "pool_name", 2024-03-24T03:17:22.750 INFO:teuthology.orchestra.run.smithi118.stdout: "type": "str", 2024-03-24T03:17:22.750 INFO:teuthology.orchestra.run.smithi118.stdout: "level": "advanced", 2024-03-24T03:17:22.750 INFO:teuthology.orchestra.run.smithi118.stdout: "flags": 1, 2024-03-24T03:17:22.750 INFO:teuthology.orchestra.run.smithi118.stdout: "default_value": "device_health_metrics", 2024-03-24T03:17:22.750 INFO:teuthology.orchestra.run.smithi118.stdout: "min": "", 2024-03-24T03:17:22.750 INFO:teuthology.orchestra.run.smithi118.stdout: "max": "", 2024-03-24T03:17:22.750 INFO:teuthology.orchestra.run.smithi118.stdout: "enum_allowed": [], 2024-03-24T03:17:22.751 INFO:teuthology.orchestra.run.smithi118.stdout: "desc": "name of pool in which to store device health metrics", 2024-03-24T03:17:22.751 INFO:teuthology.orchestra.run.smithi118.stdout: "long_desc": "", 2024-03-24T03:17:22.751 INFO:teuthology.orchestra.run.smithi118.stdout: "tags": [], 2024-03-24T03:17:22.751 INFO:teuthology.orchestra.run.smithi118.stdout: "see_also": [] 2024-03-24T03:17:22.751 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-24T03:17:22.751 INFO:teuthology.orchestra.run.smithi118.stdout: "retention_period": { 2024-03-24T03:17:22.751 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "retention_period", 2024-03-24T03:17:22.751 INFO:teuthology.orchestra.run.smithi118.stdout: "type": "secs", 2024-03-24T03:17:22.751 INFO:teuthology.orchestra.run.smithi118.stdout: "level": "advanced", 2024-03-24T03:17:22.752 INFO:teuthology.orchestra.run.smithi118.stdout: "flags": 1, 2024-03-24T03:17:22.752 INFO:teuthology.orchestra.run.smithi118.stdout: "default_value": "15552000", 2024-03-24T03:17:22.752 INFO:teuthology.orchestra.run.smithi118.stdout: "min": "", 2024-03-24T03:17:22.752 INFO:teuthology.orchestra.run.smithi118.stdout: "max": "", 2024-03-24T03:17:22.752 INFO:teuthology.orchestra.run.smithi118.stdout: "enum_allowed": [], 2024-03-24T03:17:22.752 INFO:teuthology.orchestra.run.smithi118.stdout: "desc": "how long to retain device health metrics", 2024-03-24T03:17:22.752 INFO:teuthology.orchestra.run.smithi118.stdout: "long_desc": "", 2024-03-24T03:17:22.752 INFO:teuthology.orchestra.run.smithi118.stdout: "tags": [], 2024-03-24T03:17:22.752 INFO:teuthology.orchestra.run.smithi118.stdout: "see_also": [] 2024-03-24T03:17:22.753 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-24T03:17:22.753 INFO:teuthology.orchestra.run.smithi118.stdout: "scrape_frequency": { 2024-03-24T03:17:22.753 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "scrape_frequency", 2024-03-24T03:17:22.753 INFO:teuthology.orchestra.run.smithi118.stdout: "type": "secs", 2024-03-24T03:17:22.753 INFO:teuthology.orchestra.run.smithi118.stdout: "level": "advanced", 2024-03-24T03:17:22.753 INFO:teuthology.orchestra.run.smithi118.stdout: "flags": 1, 2024-03-24T03:17:22.753 INFO:teuthology.orchestra.run.smithi118.stdout: "default_value": "86400", 2024-03-24T03:17:22.754 INFO:teuthology.orchestra.run.smithi118.stdout: "min": "", 2024-03-24T03:17:22.754 INFO:teuthology.orchestra.run.smithi118.stdout: "max": "", 2024-03-24T03:17:22.754 INFO:teuthology.orchestra.run.smithi118.stdout: "enum_allowed": [], 2024-03-24T03:17:22.754 INFO:teuthology.orchestra.run.smithi118.stdout: "desc": "how frequently to scrape device health metrics", 2024-03-24T03:17:22.754 INFO:teuthology.orchestra.run.smithi118.stdout: "long_desc": "", 2024-03-24T03:17:22.757 INFO:teuthology.orchestra.run.smithi118.stdout: "tags": [], 2024-03-24T03:17:22.757 INFO:teuthology.orchestra.run.smithi118.stdout: "see_also": [] 2024-03-24T03:17:22.757 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-24T03:17:22.757 INFO:teuthology.orchestra.run.smithi118.stdout: "self_heal": { 2024-03-24T03:17:22.757 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "self_heal", 2024-03-24T03:17:22.758 INFO:teuthology.orchestra.run.smithi118.stdout: "type": "bool", 2024-03-24T03:17:22.758 INFO:teuthology.orchestra.run.smithi118.stdout: "level": "advanced", 2024-03-24T03:17:22.758 INFO:teuthology.orchestra.run.smithi118.stdout: "flags": 1, 2024-03-24T03:17:22.758 INFO:teuthology.orchestra.run.smithi118.stdout: "default_value": "True", 2024-03-24T03:17:22.758 INFO:teuthology.orchestra.run.smithi118.stdout: "min": "", 2024-03-24T03:17:22.758 INFO:teuthology.orchestra.run.smithi118.stdout: "max": "", 2024-03-24T03:17:22.758 INFO:teuthology.orchestra.run.smithi118.stdout: "enum_allowed": [], 2024-03-24T03:17:22.758 INFO:teuthology.orchestra.run.smithi118.stdout: "desc": "preemptively heal cluster around devices that may fail", 2024-03-24T03:17:22.759 INFO:teuthology.orchestra.run.smithi118.stdout: "long_desc": "", 2024-03-24T03:17:22.759 INFO:teuthology.orchestra.run.smithi118.stdout: "tags": [], 2024-03-24T03:17:22.759 INFO:teuthology.orchestra.run.smithi118.stdout: "see_also": [] 2024-03-24T03:17:22.759 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-24T03:17:22.759 INFO:teuthology.orchestra.run.smithi118.stdout: "sleep_interval": { 2024-03-24T03:17:22.759 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "sleep_interval", 2024-03-24T03:17:22.759 INFO:teuthology.orchestra.run.smithi118.stdout: "type": "secs", 2024-03-24T03:17:22.759 INFO:teuthology.orchestra.run.smithi118.stdout: "level": "advanced", 2024-03-24T03:17:22.760 INFO:teuthology.orchestra.run.smithi118.stdout: "flags": 1, 2024-03-24T03:17:22.760 INFO:teuthology.orchestra.run.smithi118.stdout: "default_value": "600", 2024-03-24T03:17:22.760 INFO:teuthology.orchestra.run.smithi118.stdout: "min": "", 2024-03-24T03:17:22.760 INFO:teuthology.orchestra.run.smithi118.stdout: "max": "", 2024-03-24T03:17:22.760 INFO:teuthology.orchestra.run.smithi118.stdout: "enum_allowed": [], 2024-03-24T03:17:22.760 INFO:teuthology.orchestra.run.smithi118.stdout: "desc": "how frequently to wake up and check device health", 2024-03-24T03:17:22.760 INFO:teuthology.orchestra.run.smithi118.stdout: "long_desc": "", 2024-03-24T03:17:22.760 INFO:teuthology.orchestra.run.smithi118.stdout: "tags": [], 2024-03-24T03:17:22.760 INFO:teuthology.orchestra.run.smithi118.stdout: "see_also": [] 2024-03-24T03:17:22.761 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-24T03:17:22.761 INFO:teuthology.orchestra.run.smithi118.stdout: "warn_threshold": { 2024-03-24T03:17:22.761 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "warn_threshold", 2024-03-24T03:17:22.761 INFO:teuthology.orchestra.run.smithi118.stdout: "type": "secs", 2024-03-24T03:17:22.761 INFO:teuthology.orchestra.run.smithi118.stdout: "level": "advanced", 2024-03-24T03:17:22.761 INFO:teuthology.orchestra.run.smithi118.stdout: "flags": 1, 2024-03-24T03:17:22.761 INFO:teuthology.orchestra.run.smithi118.stdout: "default_value": "7257600", 2024-03-24T03:17:22.761 INFO:teuthology.orchestra.run.smithi118.stdout: "min": "", 2024-03-24T03:17:22.761 INFO:teuthology.orchestra.run.smithi118.stdout: "max": "", 2024-03-24T03:17:22.762 INFO:teuthology.orchestra.run.smithi118.stdout: "enum_allowed": [], 2024-03-24T03:17:22.762 INFO:teuthology.orchestra.run.smithi118.stdout: "desc": "raise health warning if OSD may fail before this long", 2024-03-24T03:17:22.762 INFO:teuthology.orchestra.run.smithi118.stdout: "long_desc": "", 2024-03-24T03:17:22.762 INFO:teuthology.orchestra.run.smithi118.stdout: "tags": [], 2024-03-24T03:17:22.762 INFO:teuthology.orchestra.run.smithi118.stdout: "see_also": [] 2024-03-24T03:17:22.762 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-03-24T03:17:22.762 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-03-24T03:17:22.762 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-24T03:17:22.763 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-24T03:17:22.763 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "diskprediction_local", 2024-03-24T03:17:22.763 INFO:teuthology.orchestra.run.smithi118.stdout: "can_run": true, 2024-03-24T03:17:22.763 INFO:teuthology.orchestra.run.smithi118.stdout: "error_string": "", 2024-03-24T03:17:22.763 INFO:teuthology.orchestra.run.smithi118.stdout: "module_options": { 2024-03-24T03:17:22.763 INFO:teuthology.orchestra.run.smithi118.stdout: "log_level": { 2024-03-24T03:17:22.763 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "log_level", 2024-03-24T03:17:22.763 INFO:teuthology.orchestra.run.smithi118.stdout: "type": "str", 2024-03-24T03:17:22.764 INFO:teuthology.orchestra.run.smithi118.stdout: "level": "advanced", 2024-03-24T03:17:22.764 INFO:teuthology.orchestra.run.smithi118.stdout: "flags": 1, 2024-03-24T03:17:22.764 INFO:teuthology.orchestra.run.smithi118.stdout: "default_value": "", 2024-03-24T03:17:22.764 INFO:teuthology.orchestra.run.smithi118.stdout: "min": "", 2024-03-24T03:17:22.764 INFO:teuthology.orchestra.run.smithi118.stdout: "max": "", 2024-03-24T03:17:22.764 INFO:teuthology.orchestra.run.smithi118.stdout: "enum_allowed": [ 2024-03-24T03:17:22.764 INFO:teuthology.orchestra.run.smithi118.stdout: "", 2024-03-24T03:17:22.764 INFO:teuthology.orchestra.run.smithi118.stdout: "critical", 2024-03-24T03:17:22.764 INFO:teuthology.orchestra.run.smithi118.stdout: "debug", 2024-03-24T03:17:22.765 INFO:teuthology.orchestra.run.smithi118.stdout: "error", 2024-03-24T03:17:22.765 INFO:teuthology.orchestra.run.smithi118.stdout: "info", 2024-03-24T03:17:22.765 INFO:teuthology.orchestra.run.smithi118.stdout: "warning" 2024-03-24T03:17:22.765 INFO:teuthology.orchestra.run.smithi118.stdout: ], 2024-03-24T03:17:22.765 INFO:teuthology.orchestra.run.smithi118.stdout: "desc": "", 2024-03-24T03:17:22.765 INFO:teuthology.orchestra.run.smithi118.stdout: "long_desc": "", 2024-03-24T03:17:22.765 INFO:teuthology.orchestra.run.smithi118.stdout: "tags": [], 2024-03-24T03:17:22.765 INFO:teuthology.orchestra.run.smithi118.stdout: "see_also": [] 2024-03-24T03:17:22.765 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-24T03:17:22.766 INFO:teuthology.orchestra.run.smithi118.stdout: "log_to_cluster": { 2024-03-24T03:17:22.766 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "log_to_cluster", 2024-03-24T03:17:22.766 INFO:teuthology.orchestra.run.smithi118.stdout: "type": "bool", 2024-03-24T03:17:22.766 INFO:teuthology.orchestra.run.smithi118.stdout: "level": "advanced", 2024-03-24T03:17:22.766 INFO:teuthology.orchestra.run.smithi118.stdout: "flags": 1, 2024-03-24T03:17:22.766 INFO:teuthology.orchestra.run.smithi118.stdout: "default_value": "False", 2024-03-24T03:17:22.766 INFO:teuthology.orchestra.run.smithi118.stdout: "min": "", 2024-03-24T03:17:22.766 INFO:teuthology.orchestra.run.smithi118.stdout: "max": "", 2024-03-24T03:17:22.767 INFO:teuthology.orchestra.run.smithi118.stdout: "enum_allowed": [], 2024-03-24T03:17:22.767 INFO:teuthology.orchestra.run.smithi118.stdout: "desc": "", 2024-03-24T03:17:22.767 INFO:teuthology.orchestra.run.smithi118.stdout: "long_desc": "", 2024-03-24T03:17:22.767 INFO:teuthology.orchestra.run.smithi118.stdout: "tags": [], 2024-03-24T03:17:22.767 INFO:teuthology.orchestra.run.smithi118.stdout: "see_also": [] 2024-03-24T03:17:22.767 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-24T03:17:22.767 INFO:teuthology.orchestra.run.smithi118.stdout: "log_to_cluster_level": { 2024-03-24T03:17:22.767 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "log_to_cluster_level", 2024-03-24T03:17:22.767 INFO:teuthology.orchestra.run.smithi118.stdout: "type": "str", 2024-03-24T03:17:22.767 INFO:teuthology.orchestra.run.smithi118.stdout: "level": "advanced", 2024-03-24T03:17:22.768 INFO:teuthology.orchestra.run.smithi118.stdout: "flags": 1, 2024-03-24T03:17:22.768 INFO:teuthology.orchestra.run.smithi118.stdout: "default_value": "info", 2024-03-24T03:17:22.768 INFO:teuthology.orchestra.run.smithi118.stdout: "min": "", 2024-03-24T03:17:22.768 INFO:teuthology.orchestra.run.smithi118.stdout: "max": "", 2024-03-24T03:17:22.768 INFO:teuthology.orchestra.run.smithi118.stdout: "enum_allowed": [ 2024-03-24T03:17:22.768 INFO:teuthology.orchestra.run.smithi118.stdout: "", 2024-03-24T03:17:22.768 INFO:teuthology.orchestra.run.smithi118.stdout: "critical", 2024-03-24T03:17:22.768 INFO:teuthology.orchestra.run.smithi118.stdout: "debug", 2024-03-24T03:17:22.768 INFO:teuthology.orchestra.run.smithi118.stdout: "error", 2024-03-24T03:17:22.769 INFO:teuthology.orchestra.run.smithi118.stdout: "info", 2024-03-24T03:17:22.769 INFO:teuthology.orchestra.run.smithi118.stdout: "warning" 2024-03-24T03:17:22.769 INFO:teuthology.orchestra.run.smithi118.stdout: ], 2024-03-24T03:17:22.769 INFO:teuthology.orchestra.run.smithi118.stdout: "desc": "", 2024-03-24T03:17:22.769 INFO:teuthology.orchestra.run.smithi118.stdout: "long_desc": "", 2024-03-24T03:17:22.769 INFO:teuthology.orchestra.run.smithi118.stdout: "tags": [], 2024-03-24T03:17:22.769 INFO:teuthology.orchestra.run.smithi118.stdout: "see_also": [] 2024-03-24T03:17:22.769 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-24T03:17:22.769 INFO:teuthology.orchestra.run.smithi118.stdout: "log_to_file": { 2024-03-24T03:17:22.769 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "log_to_file", 2024-03-24T03:17:22.770 INFO:teuthology.orchestra.run.smithi118.stdout: "type": "bool", 2024-03-24T03:17:22.770 INFO:teuthology.orchestra.run.smithi118.stdout: "level": "advanced", 2024-03-24T03:17:22.770 INFO:teuthology.orchestra.run.smithi118.stdout: "flags": 1, 2024-03-24T03:17:22.770 INFO:teuthology.orchestra.run.smithi118.stdout: "default_value": "False", 2024-03-24T03:17:22.770 INFO:teuthology.orchestra.run.smithi118.stdout: "min": "", 2024-03-24T03:17:22.770 INFO:teuthology.orchestra.run.smithi118.stdout: "max": "", 2024-03-24T03:17:22.770 INFO:teuthology.orchestra.run.smithi118.stdout: "enum_allowed": [], 2024-03-24T03:17:22.770 INFO:teuthology.orchestra.run.smithi118.stdout: "desc": "", 2024-03-24T03:17:22.770 INFO:teuthology.orchestra.run.smithi118.stdout: "long_desc": "", 2024-03-24T03:17:22.771 INFO:teuthology.orchestra.run.smithi118.stdout: "tags": [], 2024-03-24T03:17:22.771 INFO:teuthology.orchestra.run.smithi118.stdout: "see_also": [] 2024-03-24T03:17:22.771 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-24T03:17:22.771 INFO:teuthology.orchestra.run.smithi118.stdout: "predict_interval": { 2024-03-24T03:17:22.771 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "predict_interval", 2024-03-24T03:17:22.771 INFO:teuthology.orchestra.run.smithi118.stdout: "type": "str", 2024-03-24T03:17:22.771 INFO:teuthology.orchestra.run.smithi118.stdout: "level": "advanced", 2024-03-24T03:17:22.771 INFO:teuthology.orchestra.run.smithi118.stdout: "flags": 0, 2024-03-24T03:17:22.771 INFO:teuthology.orchestra.run.smithi118.stdout: "default_value": "86400", 2024-03-24T03:17:22.772 INFO:teuthology.orchestra.run.smithi118.stdout: "min": "", 2024-03-24T03:17:22.772 INFO:teuthology.orchestra.run.smithi118.stdout: "max": "", 2024-03-24T03:17:22.772 INFO:teuthology.orchestra.run.smithi118.stdout: "enum_allowed": [], 2024-03-24T03:17:22.772 INFO:teuthology.orchestra.run.smithi118.stdout: "desc": "", 2024-03-24T03:17:22.772 INFO:teuthology.orchestra.run.smithi118.stdout: "long_desc": "", 2024-03-24T03:17:22.772 INFO:teuthology.orchestra.run.smithi118.stdout: "tags": [], 2024-03-24T03:17:22.772 INFO:teuthology.orchestra.run.smithi118.stdout: "see_also": [] 2024-03-24T03:17:22.772 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-24T03:17:22.773 INFO:teuthology.orchestra.run.smithi118.stdout: "predictor_model": { 2024-03-24T03:17:22.773 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "predictor_model", 2024-03-24T03:17:22.773 INFO:teuthology.orchestra.run.smithi118.stdout: "type": "str", 2024-03-24T03:17:22.773 INFO:teuthology.orchestra.run.smithi118.stdout: "level": "advanced", 2024-03-24T03:17:22.773 INFO:teuthology.orchestra.run.smithi118.stdout: "flags": 0, 2024-03-24T03:17:22.773 INFO:teuthology.orchestra.run.smithi118.stdout: "default_value": "prophetstor", 2024-03-24T03:17:22.773 INFO:teuthology.orchestra.run.smithi118.stdout: "min": "", 2024-03-24T03:17:22.773 INFO:teuthology.orchestra.run.smithi118.stdout: "max": "", 2024-03-24T03:17:22.773 INFO:teuthology.orchestra.run.smithi118.stdout: "enum_allowed": [], 2024-03-24T03:17:22.774 INFO:teuthology.orchestra.run.smithi118.stdout: "desc": "", 2024-03-24T03:17:22.774 INFO:teuthology.orchestra.run.smithi118.stdout: "long_desc": "", 2024-03-24T03:17:22.774 INFO:teuthology.orchestra.run.smithi118.stdout: "tags": [], 2024-03-24T03:17:22.774 INFO:teuthology.orchestra.run.smithi118.stdout: "see_also": [] 2024-03-24T03:17:22.774 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-24T03:17:22.774 INFO:teuthology.orchestra.run.smithi118.stdout: "sleep_interval": { 2024-03-24T03:17:22.774 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "sleep_interval", 2024-03-24T03:17:22.774 INFO:teuthology.orchestra.run.smithi118.stdout: "type": "str", 2024-03-24T03:17:22.775 INFO:teuthology.orchestra.run.smithi118.stdout: "level": "advanced", 2024-03-24T03:17:22.775 INFO:teuthology.orchestra.run.smithi118.stdout: "flags": 0, 2024-03-24T03:17:22.775 INFO:teuthology.orchestra.run.smithi118.stdout: "default_value": "600", 2024-03-24T03:17:22.775 INFO:teuthology.orchestra.run.smithi118.stdout: "min": "", 2024-03-24T03:17:22.775 INFO:teuthology.orchestra.run.smithi118.stdout: "max": "", 2024-03-24T03:17:22.775 INFO:teuthology.orchestra.run.smithi118.stdout: "enum_allowed": [], 2024-03-24T03:17:22.775 INFO:teuthology.orchestra.run.smithi118.stdout: "desc": "", 2024-03-24T03:17:22.775 INFO:teuthology.orchestra.run.smithi118.stdout: "long_desc": "", 2024-03-24T03:17:22.775 INFO:teuthology.orchestra.run.smithi118.stdout: "tags": [], 2024-03-24T03:17:22.776 INFO:teuthology.orchestra.run.smithi118.stdout: "see_also": [] 2024-03-24T03:17:22.776 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-03-24T03:17:22.776 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-03-24T03:17:22.776 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-24T03:17:22.776 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-24T03:17:22.776 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "influx", 2024-03-24T03:17:22.776 INFO:teuthology.orchestra.run.smithi118.stdout: "can_run": false, 2024-03-24T03:17:22.776 INFO:teuthology.orchestra.run.smithi118.stdout: "error_string": "influxdb python module not found", 2024-03-24T03:17:22.776 INFO:teuthology.orchestra.run.smithi118.stdout: "module_options": { 2024-03-24T03:17:22.776 INFO:teuthology.orchestra.run.smithi118.stdout: "batch_size": { 2024-03-24T03:17:22.777 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "batch_size", 2024-03-24T03:17:22.777 INFO:teuthology.orchestra.run.smithi118.stdout: "type": "int", 2024-03-24T03:17:22.777 INFO:teuthology.orchestra.run.smithi118.stdout: "level": "advanced", 2024-03-24T03:17:22.777 INFO:teuthology.orchestra.run.smithi118.stdout: "flags": 0, 2024-03-24T03:17:22.777 INFO:teuthology.orchestra.run.smithi118.stdout: "default_value": "5000", 2024-03-24T03:17:22.777 INFO:teuthology.orchestra.run.smithi118.stdout: "min": "", 2024-03-24T03:17:22.777 INFO:teuthology.orchestra.run.smithi118.stdout: "max": "", 2024-03-24T03:17:22.777 INFO:teuthology.orchestra.run.smithi118.stdout: "enum_allowed": [], 2024-03-24T03:17:22.777 INFO:teuthology.orchestra.run.smithi118.stdout: "desc": "How big batches of data points should be when sending to InfluxDB.", 2024-03-24T03:17:22.778 INFO:teuthology.orchestra.run.smithi118.stdout: "long_desc": "", 2024-03-24T03:17:22.778 INFO:teuthology.orchestra.run.smithi118.stdout: "tags": [], 2024-03-24T03:17:22.778 INFO:teuthology.orchestra.run.smithi118.stdout: "see_also": [] 2024-03-24T03:17:22.778 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-24T03:17:22.778 INFO:teuthology.orchestra.run.smithi118.stdout: "database": { 2024-03-24T03:17:22.778 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "database", 2024-03-24T03:17:22.778 INFO:teuthology.orchestra.run.smithi118.stdout: "type": "str", 2024-03-24T03:17:22.778 INFO:teuthology.orchestra.run.smithi118.stdout: "level": "advanced", 2024-03-24T03:17:22.778 INFO:teuthology.orchestra.run.smithi118.stdout: "flags": 0, 2024-03-24T03:17:22.779 INFO:teuthology.orchestra.run.smithi118.stdout: "default_value": "ceph", 2024-03-24T03:17:22.779 INFO:teuthology.orchestra.run.smithi118.stdout: "min": "", 2024-03-24T03:17:22.779 INFO:teuthology.orchestra.run.smithi118.stdout: "max": "", 2024-03-24T03:17:22.779 INFO:teuthology.orchestra.run.smithi118.stdout: "enum_allowed": [], 2024-03-24T03:17:22.779 INFO:teuthology.orchestra.run.smithi118.stdout: "desc": "InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.", 2024-03-24T03:17:22.779 INFO:teuthology.orchestra.run.smithi118.stdout: "long_desc": "", 2024-03-24T03:17:22.779 INFO:teuthology.orchestra.run.smithi118.stdout: "tags": [], 2024-03-24T03:17:22.779 INFO:teuthology.orchestra.run.smithi118.stdout: "see_also": [] 2024-03-24T03:17:22.779 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-24T03:17:22.780 INFO:teuthology.orchestra.run.smithi118.stdout: "hostname": { 2024-03-24T03:17:22.780 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "hostname", 2024-03-24T03:17:22.780 INFO:teuthology.orchestra.run.smithi118.stdout: "type": "str", 2024-03-24T03:17:22.780 INFO:teuthology.orchestra.run.smithi118.stdout: "level": "advanced", 2024-03-24T03:17:22.780 INFO:teuthology.orchestra.run.smithi118.stdout: "flags": 0, 2024-03-24T03:17:22.780 INFO:teuthology.orchestra.run.smithi118.stdout: "default_value": "", 2024-03-24T03:17:22.780 INFO:teuthology.orchestra.run.smithi118.stdout: "min": "", 2024-03-24T03:17:22.780 INFO:teuthology.orchestra.run.smithi118.stdout: "max": "", 2024-03-24T03:17:22.780 INFO:teuthology.orchestra.run.smithi118.stdout: "enum_allowed": [], 2024-03-24T03:17:22.781 INFO:teuthology.orchestra.run.smithi118.stdout: "desc": "InfluxDB server hostname", 2024-03-24T03:17:22.781 INFO:teuthology.orchestra.run.smithi118.stdout: "long_desc": "", 2024-03-24T03:17:22.781 INFO:teuthology.orchestra.run.smithi118.stdout: "tags": [], 2024-03-24T03:17:22.781 INFO:teuthology.orchestra.run.smithi118.stdout: "see_also": [] 2024-03-24T03:17:22.781 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-24T03:17:22.781 INFO:teuthology.orchestra.run.smithi118.stdout: "interval": { 2024-03-24T03:17:22.781 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "interval", 2024-03-24T03:17:22.781 INFO:teuthology.orchestra.run.smithi118.stdout: "type": "secs", 2024-03-24T03:17:22.781 INFO:teuthology.orchestra.run.smithi118.stdout: "level": "advanced", 2024-03-24T03:17:22.782 INFO:teuthology.orchestra.run.smithi118.stdout: "flags": 0, 2024-03-24T03:17:22.782 INFO:teuthology.orchestra.run.smithi118.stdout: "default_value": "30", 2024-03-24T03:17:22.782 INFO:teuthology.orchestra.run.smithi118.stdout: "min": "5", 2024-03-24T03:17:22.782 INFO:teuthology.orchestra.run.smithi118.stdout: "max": "", 2024-03-24T03:17:22.782 INFO:teuthology.orchestra.run.smithi118.stdout: "enum_allowed": [], 2024-03-24T03:17:22.782 INFO:teuthology.orchestra.run.smithi118.stdout: "desc": "Time between reports to InfluxDB. Default 30 seconds.", 2024-03-24T03:17:22.782 INFO:teuthology.orchestra.run.smithi118.stdout: "long_desc": "", 2024-03-24T03:17:22.782 INFO:teuthology.orchestra.run.smithi118.stdout: "tags": [], 2024-03-24T03:17:22.782 INFO:teuthology.orchestra.run.smithi118.stdout: "see_also": [] 2024-03-24T03:17:22.783 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-24T03:17:22.783 INFO:teuthology.orchestra.run.smithi118.stdout: "log_level": { 2024-03-24T03:17:22.783 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "log_level", 2024-03-24T03:17:22.783 INFO:teuthology.orchestra.run.smithi118.stdout: "type": "str", 2024-03-24T03:17:22.783 INFO:teuthology.orchestra.run.smithi118.stdout: "level": "advanced", 2024-03-24T03:17:22.783 INFO:teuthology.orchestra.run.smithi118.stdout: "flags": 1, 2024-03-24T03:17:22.783 INFO:teuthology.orchestra.run.smithi118.stdout: "default_value": "", 2024-03-24T03:17:22.783 INFO:teuthology.orchestra.run.smithi118.stdout: "min": "", 2024-03-24T03:17:22.783 INFO:teuthology.orchestra.run.smithi118.stdout: "max": "", 2024-03-24T03:17:22.784 INFO:teuthology.orchestra.run.smithi118.stdout: "enum_allowed": [ 2024-03-24T03:17:22.784 INFO:teuthology.orchestra.run.smithi118.stdout: "", 2024-03-24T03:17:22.784 INFO:teuthology.orchestra.run.smithi118.stdout: "critical", 2024-03-24T03:17:22.784 INFO:teuthology.orchestra.run.smithi118.stdout: "debug", 2024-03-24T03:17:22.784 INFO:teuthology.orchestra.run.smithi118.stdout: "error", 2024-03-24T03:17:22.784 INFO:teuthology.orchestra.run.smithi118.stdout: "info", 2024-03-24T03:17:22.784 INFO:teuthology.orchestra.run.smithi118.stdout: "warning" 2024-03-24T03:17:22.784 INFO:teuthology.orchestra.run.smithi118.stdout: ], 2024-03-24T03:17:22.785 INFO:teuthology.orchestra.run.smithi118.stdout: "desc": "", 2024-03-24T03:17:22.785 INFO:teuthology.orchestra.run.smithi118.stdout: "long_desc": "", 2024-03-24T03:17:22.785 INFO:teuthology.orchestra.run.smithi118.stdout: "tags": [], 2024-03-24T03:17:22.785 INFO:teuthology.orchestra.run.smithi118.stdout: "see_also": [] 2024-03-24T03:17:22.785 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-24T03:17:22.785 INFO:teuthology.orchestra.run.smithi118.stdout: "log_to_cluster": { 2024-03-24T03:17:22.785 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "log_to_cluster", 2024-03-24T03:17:22.785 INFO:teuthology.orchestra.run.smithi118.stdout: "type": "bool", 2024-03-24T03:17:22.786 INFO:teuthology.orchestra.run.smithi118.stdout: "level": "advanced", 2024-03-24T03:17:22.786 INFO:teuthology.orchestra.run.smithi118.stdout: "flags": 1, 2024-03-24T03:17:22.786 INFO:teuthology.orchestra.run.smithi118.stdout: "default_value": "False", 2024-03-24T03:17:22.786 INFO:teuthology.orchestra.run.smithi118.stdout: "min": "", 2024-03-24T03:17:22.786 INFO:teuthology.orchestra.run.smithi118.stdout: "max": "", 2024-03-24T03:17:22.786 INFO:teuthology.orchestra.run.smithi118.stdout: "enum_allowed": [], 2024-03-24T03:17:22.786 INFO:teuthology.orchestra.run.smithi118.stdout: "desc": "", 2024-03-24T03:17:22.786 INFO:teuthology.orchestra.run.smithi118.stdout: "long_desc": "", 2024-03-24T03:17:22.786 INFO:teuthology.orchestra.run.smithi118.stdout: "tags": [], 2024-03-24T03:17:22.787 INFO:teuthology.orchestra.run.smithi118.stdout: "see_also": [] 2024-03-24T03:17:22.787 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-24T03:17:22.787 INFO:teuthology.orchestra.run.smithi118.stdout: "log_to_cluster_level": { 2024-03-24T03:17:22.787 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "log_to_cluster_level", 2024-03-24T03:17:22.787 INFO:teuthology.orchestra.run.smithi118.stdout: "type": "str", 2024-03-24T03:17:22.787 INFO:teuthology.orchestra.run.smithi118.stdout: "level": "advanced", 2024-03-24T03:17:22.787 INFO:teuthology.orchestra.run.smithi118.stdout: "flags": 1, 2024-03-24T03:17:22.787 INFO:teuthology.orchestra.run.smithi118.stdout: "default_value": "info", 2024-03-24T03:17:22.787 INFO:teuthology.orchestra.run.smithi118.stdout: "min": "", 2024-03-24T03:17:22.788 INFO:teuthology.orchestra.run.smithi118.stdout: "max": "", 2024-03-24T03:17:22.788 INFO:teuthology.orchestra.run.smithi118.stdout: "enum_allowed": [ 2024-03-24T03:17:22.788 INFO:teuthology.orchestra.run.smithi118.stdout: "", 2024-03-24T03:17:22.788 INFO:teuthology.orchestra.run.smithi118.stdout: "critical", 2024-03-24T03:17:22.788 INFO:teuthology.orchestra.run.smithi118.stdout: "debug", 2024-03-24T03:17:22.788 INFO:teuthology.orchestra.run.smithi118.stdout: "error", 2024-03-24T03:17:22.788 INFO:teuthology.orchestra.run.smithi118.stdout: "info", 2024-03-24T03:17:22.788 INFO:teuthology.orchestra.run.smithi118.stdout: "warning" 2024-03-24T03:17:22.788 INFO:teuthology.orchestra.run.smithi118.stdout: ], 2024-03-24T03:17:22.789 INFO:teuthology.orchestra.run.smithi118.stdout: "desc": "", 2024-03-24T03:17:22.789 INFO:teuthology.orchestra.run.smithi118.stdout: "long_desc": "", 2024-03-24T03:17:22.789 INFO:teuthology.orchestra.run.smithi118.stdout: "tags": [], 2024-03-24T03:17:22.789 INFO:teuthology.orchestra.run.smithi118.stdout: "see_also": [] 2024-03-24T03:17:22.789 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-24T03:17:22.789 INFO:teuthology.orchestra.run.smithi118.stdout: "log_to_file": { 2024-03-24T03:17:22.789 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "log_to_file", 2024-03-24T03:17:22.789 INFO:teuthology.orchestra.run.smithi118.stdout: "type": "bool", 2024-03-24T03:17:22.789 INFO:teuthology.orchestra.run.smithi118.stdout: "level": "advanced", 2024-03-24T03:17:22.789 INFO:teuthology.orchestra.run.smithi118.stdout: "flags": 1, 2024-03-24T03:17:22.790 INFO:teuthology.orchestra.run.smithi118.stdout: "default_value": "False", 2024-03-24T03:17:22.790 INFO:teuthology.orchestra.run.smithi118.stdout: "min": "", 2024-03-24T03:17:22.790 INFO:teuthology.orchestra.run.smithi118.stdout: "max": "", 2024-03-24T03:17:22.790 INFO:teuthology.orchestra.run.smithi118.stdout: "enum_allowed": [], 2024-03-24T03:17:22.790 INFO:teuthology.orchestra.run.smithi118.stdout: "desc": "", 2024-03-24T03:17:22.790 INFO:teuthology.orchestra.run.smithi118.stdout: "long_desc": "", 2024-03-24T03:17:22.790 INFO:teuthology.orchestra.run.smithi118.stdout: "tags": [], 2024-03-24T03:17:22.790 INFO:teuthology.orchestra.run.smithi118.stdout: "see_also": [] 2024-03-24T03:17:22.790 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-24T03:17:22.791 INFO:teuthology.orchestra.run.smithi118.stdout: "password": { 2024-03-24T03:17:22.791 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "password", 2024-03-24T03:17:22.791 INFO:teuthology.orchestra.run.smithi118.stdout: "type": "str", 2024-03-24T03:17:22.791 INFO:teuthology.orchestra.run.smithi118.stdout: "level": "advanced", 2024-03-24T03:17:22.791 INFO:teuthology.orchestra.run.smithi118.stdout: "flags": 0, 2024-03-24T03:17:22.791 INFO:teuthology.orchestra.run.smithi118.stdout: "default_value": "", 2024-03-24T03:17:22.791 INFO:teuthology.orchestra.run.smithi118.stdout: "min": "", 2024-03-24T03:17:22.791 INFO:teuthology.orchestra.run.smithi118.stdout: "max": "", 2024-03-24T03:17:22.791 INFO:teuthology.orchestra.run.smithi118.stdout: "enum_allowed": [], 2024-03-24T03:17:22.792 INFO:teuthology.orchestra.run.smithi118.stdout: "desc": "password of InfluxDB server user", 2024-03-24T03:17:22.792 INFO:teuthology.orchestra.run.smithi118.stdout: "long_desc": "", 2024-03-24T03:17:22.792 INFO:teuthology.orchestra.run.smithi118.stdout: "tags": [], 2024-03-24T03:17:22.792 INFO:teuthology.orchestra.run.smithi118.stdout: "see_also": [] 2024-03-24T03:17:22.792 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-24T03:17:22.792 INFO:teuthology.orchestra.run.smithi118.stdout: "port": { 2024-03-24T03:17:22.792 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "port", 2024-03-24T03:17:22.793 INFO:teuthology.orchestra.run.smithi118.stdout: "type": "int", 2024-03-24T03:17:22.793 INFO:teuthology.orchestra.run.smithi118.stdout: "level": "advanced", 2024-03-24T03:17:22.793 INFO:teuthology.orchestra.run.smithi118.stdout: "flags": 0, 2024-03-24T03:17:22.793 INFO:teuthology.orchestra.run.smithi118.stdout: "default_value": "8086", 2024-03-24T03:17:22.793 INFO:teuthology.orchestra.run.smithi118.stdout: "min": "", 2024-03-24T03:17:22.793 INFO:teuthology.orchestra.run.smithi118.stdout: "max": "", 2024-03-24T03:17:22.793 INFO:teuthology.orchestra.run.smithi118.stdout: "enum_allowed": [], 2024-03-24T03:17:22.793 INFO:teuthology.orchestra.run.smithi118.stdout: "desc": "InfluxDB server port", 2024-03-24T03:17:22.793 INFO:teuthology.orchestra.run.smithi118.stdout: "long_desc": "", 2024-03-24T03:17:22.794 INFO:teuthology.orchestra.run.smithi118.stdout: "tags": [], 2024-03-24T03:17:22.794 INFO:teuthology.orchestra.run.smithi118.stdout: "see_also": [] 2024-03-24T03:17:22.794 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-24T03:17:22.794 INFO:teuthology.orchestra.run.smithi118.stdout: "ssl": { 2024-03-24T03:17:22.794 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "ssl", 2024-03-24T03:17:22.794 INFO:teuthology.orchestra.run.smithi118.stdout: "type": "str", 2024-03-24T03:17:22.794 INFO:teuthology.orchestra.run.smithi118.stdout: "level": "advanced", 2024-03-24T03:17:22.794 INFO:teuthology.orchestra.run.smithi118.stdout: "flags": 0, 2024-03-24T03:17:22.795 INFO:teuthology.orchestra.run.smithi118.stdout: "default_value": "false", 2024-03-24T03:17:22.795 INFO:teuthology.orchestra.run.smithi118.stdout: "min": "", 2024-03-24T03:17:22.795 INFO:teuthology.orchestra.run.smithi118.stdout: "max": "", 2024-03-24T03:17:22.795 INFO:teuthology.orchestra.run.smithi118.stdout: "enum_allowed": [], 2024-03-24T03:17:22.795 INFO:teuthology.orchestra.run.smithi118.stdout: "desc": "Use https connection for InfluxDB server. Use \"true\" or \"false\".", 2024-03-24T03:17:22.795 INFO:teuthology.orchestra.run.smithi118.stdout: "long_desc": "", 2024-03-24T03:17:22.795 INFO:teuthology.orchestra.run.smithi118.stdout: "tags": [], 2024-03-24T03:17:22.795 INFO:teuthology.orchestra.run.smithi118.stdout: "see_also": [] 2024-03-24T03:17:22.795 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-24T03:17:22.795 INFO:teuthology.orchestra.run.smithi118.stdout: "threads": { 2024-03-24T03:17:22.796 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "threads", 2024-03-24T03:17:22.796 INFO:teuthology.orchestra.run.smithi118.stdout: "type": "int", 2024-03-24T03:17:22.796 INFO:teuthology.orchestra.run.smithi118.stdout: "level": "advanced", 2024-03-24T03:17:22.796 INFO:teuthology.orchestra.run.smithi118.stdout: "flags": 0, 2024-03-24T03:17:22.796 INFO:teuthology.orchestra.run.smithi118.stdout: "default_value": "5", 2024-03-24T03:17:22.796 INFO:teuthology.orchestra.run.smithi118.stdout: "min": "1", 2024-03-24T03:17:22.796 INFO:teuthology.orchestra.run.smithi118.stdout: "max": "32", 2024-03-24T03:17:22.796 INFO:teuthology.orchestra.run.smithi118.stdout: "enum_allowed": [], 2024-03-24T03:17:22.796 INFO:teuthology.orchestra.run.smithi118.stdout: "desc": "How many worker threads should be spawned for sending data to InfluxDB.", 2024-03-24T03:17:22.797 INFO:teuthology.orchestra.run.smithi118.stdout: "long_desc": "", 2024-03-24T03:17:22.797 INFO:teuthology.orchestra.run.smithi118.stdout: "tags": [], 2024-03-24T03:17:22.797 INFO:teuthology.orchestra.run.smithi118.stdout: "see_also": [] 2024-03-24T03:17:22.797 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-24T03:17:22.797 INFO:teuthology.orchestra.run.smithi118.stdout: "username": { 2024-03-24T03:17:22.797 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "username", 2024-03-24T03:17:22.797 INFO:teuthology.orchestra.run.smithi118.stdout: "type": "str", 2024-03-24T03:17:22.797 INFO:teuthology.orchestra.run.smithi118.stdout: "level": "advanced", 2024-03-24T03:17:22.798 INFO:teuthology.orchestra.run.smithi118.stdout: "flags": 0, 2024-03-24T03:17:22.798 INFO:teuthology.orchestra.run.smithi118.stdout: "default_value": "", 2024-03-24T03:17:22.798 INFO:teuthology.orchestra.run.smithi118.stdout: "min": "", 2024-03-24T03:17:22.798 INFO:teuthology.orchestra.run.smithi118.stdout: "max": "", 2024-03-24T03:17:22.798 INFO:teuthology.orchestra.run.smithi118.stdout: "enum_allowed": [], 2024-03-24T03:17:22.798 INFO:teuthology.orchestra.run.smithi118.stdout: "desc": "username of InfluxDB server user", 2024-03-24T03:17:22.798 INFO:teuthology.orchestra.run.smithi118.stdout: "long_desc": "", 2024-03-24T03:17:22.798 INFO:teuthology.orchestra.run.smithi118.stdout: "tags": [], 2024-03-24T03:17:22.798 INFO:teuthology.orchestra.run.smithi118.stdout: "see_also": [] 2024-03-24T03:17:22.799 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-24T03:17:22.799 INFO:teuthology.orchestra.run.smithi118.stdout: "verify_ssl": { 2024-03-24T03:17:22.799 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "verify_ssl", 2024-03-24T03:17:22.799 INFO:teuthology.orchestra.run.smithi118.stdout: "type": "str", 2024-03-24T03:17:22.799 INFO:teuthology.orchestra.run.smithi118.stdout: "level": "advanced", 2024-03-24T03:17:22.799 INFO:teuthology.orchestra.run.smithi118.stdout: "flags": 0, 2024-03-24T03:17:22.799 INFO:teuthology.orchestra.run.smithi118.stdout: "default_value": "true", 2024-03-24T03:17:22.799 INFO:teuthology.orchestra.run.smithi118.stdout: "min": "", 2024-03-24T03:17:22.800 INFO:teuthology.orchestra.run.smithi118.stdout: "max": "", 2024-03-24T03:17:22.800 INFO:teuthology.orchestra.run.smithi118.stdout: "enum_allowed": [], 2024-03-24T03:17:22.800 INFO:teuthology.orchestra.run.smithi118.stdout: "desc": "Verify https cert for InfluxDB server. Use \"true\" or \"false\".", 2024-03-24T03:17:22.800 INFO:teuthology.orchestra.run.smithi118.stdout: "long_desc": "", 2024-03-24T03:17:22.800 INFO:teuthology.orchestra.run.smithi118.stdout: "tags": [], 2024-03-24T03:17:22.800 INFO:teuthology.orchestra.run.smithi118.stdout: "see_also": [] 2024-03-24T03:17:22.800 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-03-24T03:17:22.800 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-03-24T03:17:22.800 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-24T03:17:22.801 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-24T03:17:22.801 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "insights", 2024-03-24T03:17:22.801 INFO:teuthology.orchestra.run.smithi118.stdout: "can_run": true, 2024-03-24T03:17:22.801 INFO:teuthology.orchestra.run.smithi118.stdout: "error_string": "", 2024-03-24T03:17:22.801 INFO:teuthology.orchestra.run.smithi118.stdout: "module_options": { 2024-03-24T03:17:22.801 INFO:teuthology.orchestra.run.smithi118.stdout: "log_level": { 2024-03-24T03:17:22.801 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "log_level", 2024-03-24T03:17:22.801 INFO:teuthology.orchestra.run.smithi118.stdout: "type": "str", 2024-03-24T03:17:22.801 INFO:teuthology.orchestra.run.smithi118.stdout: "level": "advanced", 2024-03-24T03:17:22.802 INFO:teuthology.orchestra.run.smithi118.stdout: "flags": 1, 2024-03-24T03:17:22.802 INFO:teuthology.orchestra.run.smithi118.stdout: "default_value": "", 2024-03-24T03:17:22.802 INFO:teuthology.orchestra.run.smithi118.stdout: "min": "", 2024-03-24T03:17:22.802 INFO:teuthology.orchestra.run.smithi118.stdout: "max": "", 2024-03-24T03:17:22.802 INFO:teuthology.orchestra.run.smithi118.stdout: "enum_allowed": [ 2024-03-24T03:17:22.802 INFO:teuthology.orchestra.run.smithi118.stdout: "", 2024-03-24T03:17:22.802 INFO:teuthology.orchestra.run.smithi118.stdout: "critical", 2024-03-24T03:17:22.802 INFO:teuthology.orchestra.run.smithi118.stdout: "debug", 2024-03-24T03:17:22.802 INFO:teuthology.orchestra.run.smithi118.stdout: "error", 2024-03-24T03:17:22.803 INFO:teuthology.orchestra.run.smithi118.stdout: "info", 2024-03-24T03:17:22.803 INFO:teuthology.orchestra.run.smithi118.stdout: "warning" 2024-03-24T03:17:22.803 INFO:teuthology.orchestra.run.smithi118.stdout: ], 2024-03-24T03:17:22.803 INFO:teuthology.orchestra.run.smithi118.stdout: "desc": "", 2024-03-24T03:17:22.803 INFO:teuthology.orchestra.run.smithi118.stdout: "long_desc": "", 2024-03-24T03:17:22.803 INFO:teuthology.orchestra.run.smithi118.stdout: "tags": [], 2024-03-24T03:17:22.803 INFO:teuthology.orchestra.run.smithi118.stdout: "see_also": [] 2024-03-24T03:17:22.803 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-24T03:17:22.804 INFO:teuthology.orchestra.run.smithi118.stdout: "log_to_cluster": { 2024-03-24T03:17:22.804 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "log_to_cluster", 2024-03-24T03:17:22.804 INFO:teuthology.orchestra.run.smithi118.stdout: "type": "bool", 2024-03-24T03:17:22.804 INFO:teuthology.orchestra.run.smithi118.stdout: "level": "advanced", 2024-03-24T03:17:22.804 INFO:teuthology.orchestra.run.smithi118.stdout: "flags": 1, 2024-03-24T03:17:22.804 INFO:teuthology.orchestra.run.smithi118.stdout: "default_value": "False", 2024-03-24T03:17:22.804 INFO:teuthology.orchestra.run.smithi118.stdout: "min": "", 2024-03-24T03:17:22.805 INFO:teuthology.orchestra.run.smithi118.stdout: "max": "", 2024-03-24T03:17:22.805 INFO:teuthology.orchestra.run.smithi118.stdout: "enum_allowed": [], 2024-03-24T03:17:22.805 INFO:teuthology.orchestra.run.smithi118.stdout: "desc": "", 2024-03-24T03:17:22.805 INFO:teuthology.orchestra.run.smithi118.stdout: "long_desc": "", 2024-03-24T03:17:22.805 INFO:teuthology.orchestra.run.smithi118.stdout: "tags": [], 2024-03-24T03:17:22.805 INFO:teuthology.orchestra.run.smithi118.stdout: "see_also": [] 2024-03-24T03:17:22.805 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-24T03:17:22.805 INFO:teuthology.orchestra.run.smithi118.stdout: "log_to_cluster_level": { 2024-03-24T03:17:22.805 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "log_to_cluster_level", 2024-03-24T03:17:22.806 INFO:teuthology.orchestra.run.smithi118.stdout: "type": "str", 2024-03-24T03:17:22.806 INFO:teuthology.orchestra.run.smithi118.stdout: "level": "advanced", 2024-03-24T03:17:22.806 INFO:teuthology.orchestra.run.smithi118.stdout: "flags": 1, 2024-03-24T03:17:22.806 INFO:teuthology.orchestra.run.smithi118.stdout: "default_value": "info", 2024-03-24T03:17:22.806 INFO:teuthology.orchestra.run.smithi118.stdout: "min": "", 2024-03-24T03:17:22.806 INFO:teuthology.orchestra.run.smithi118.stdout: "max": "", 2024-03-24T03:17:22.806 INFO:teuthology.orchestra.run.smithi118.stdout: "enum_allowed": [ 2024-03-24T03:17:22.806 INFO:teuthology.orchestra.run.smithi118.stdout: "", 2024-03-24T03:17:22.807 INFO:teuthology.orchestra.run.smithi118.stdout: "critical", 2024-03-24T03:17:22.807 INFO:teuthology.orchestra.run.smithi118.stdout: "debug", 2024-03-24T03:17:22.807 INFO:teuthology.orchestra.run.smithi118.stdout: "error", 2024-03-24T03:17:22.807 INFO:teuthology.orchestra.run.smithi118.stdout: "info", 2024-03-24T03:17:22.807 INFO:teuthology.orchestra.run.smithi118.stdout: "warning" 2024-03-24T03:17:22.807 INFO:teuthology.orchestra.run.smithi118.stdout: ], 2024-03-24T03:17:22.807 INFO:teuthology.orchestra.run.smithi118.stdout: "desc": "", 2024-03-24T03:17:22.807 INFO:teuthology.orchestra.run.smithi118.stdout: "long_desc": "", 2024-03-24T03:17:22.807 INFO:teuthology.orchestra.run.smithi118.stdout: "tags": [], 2024-03-24T03:17:22.808 INFO:teuthology.orchestra.run.smithi118.stdout: "see_also": [] 2024-03-24T03:17:22.808 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-24T03:17:22.808 INFO:teuthology.orchestra.run.smithi118.stdout: "log_to_file": { 2024-03-24T03:17:22.808 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "log_to_file", 2024-03-24T03:17:22.808 INFO:teuthology.orchestra.run.smithi118.stdout: "type": "bool", 2024-03-24T03:17:22.808 INFO:teuthology.orchestra.run.smithi118.stdout: "level": "advanced", 2024-03-24T03:17:22.808 INFO:teuthology.orchestra.run.smithi118.stdout: "flags": 1, 2024-03-24T03:17:22.808 INFO:teuthology.orchestra.run.smithi118.stdout: "default_value": "False", 2024-03-24T03:17:22.808 INFO:teuthology.orchestra.run.smithi118.stdout: "min": "", 2024-03-24T03:17:22.808 INFO:teuthology.orchestra.run.smithi118.stdout: "max": "", 2024-03-24T03:17:22.809 INFO:teuthology.orchestra.run.smithi118.stdout: "enum_allowed": [], 2024-03-24T03:17:22.809 INFO:teuthology.orchestra.run.smithi118.stdout: "desc": "", 2024-03-24T03:17:22.809 INFO:teuthology.orchestra.run.smithi118.stdout: "long_desc": "", 2024-03-24T03:17:22.809 INFO:teuthology.orchestra.run.smithi118.stdout: "tags": [], 2024-03-24T03:17:22.809 INFO:teuthology.orchestra.run.smithi118.stdout: "see_also": [] 2024-03-24T03:17:22.809 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-03-24T03:17:22.809 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-03-24T03:17:22.809 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-24T03:17:22.809 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-24T03:17:22.810 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "iostat", 2024-03-24T03:17:22.810 INFO:teuthology.orchestra.run.smithi118.stdout: "can_run": true, 2024-03-24T03:17:22.810 INFO:teuthology.orchestra.run.smithi118.stdout: "error_string": "", 2024-03-24T03:17:22.810 INFO:teuthology.orchestra.run.smithi118.stdout: "module_options": { 2024-03-24T03:17:22.810 INFO:teuthology.orchestra.run.smithi118.stdout: "log_level": { 2024-03-24T03:17:22.810 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "log_level", 2024-03-24T03:17:22.810 INFO:teuthology.orchestra.run.smithi118.stdout: "type": "str", 2024-03-24T03:17:22.810 INFO:teuthology.orchestra.run.smithi118.stdout: "level": "advanced", 2024-03-24T03:17:22.810 INFO:teuthology.orchestra.run.smithi118.stdout: "flags": 1, 2024-03-24T03:17:22.811 INFO:teuthology.orchestra.run.smithi118.stdout: "default_value": "", 2024-03-24T03:17:22.811 INFO:teuthology.orchestra.run.smithi118.stdout: "min": "", 2024-03-24T03:17:22.811 INFO:teuthology.orchestra.run.smithi118.stdout: "max": "", 2024-03-24T03:17:22.811 INFO:teuthology.orchestra.run.smithi118.stdout: "enum_allowed": [ 2024-03-24T03:17:22.812 INFO:teuthology.orchestra.run.smithi118.stdout: "", 2024-03-24T03:17:22.812 INFO:teuthology.orchestra.run.smithi118.stdout: "critical", 2024-03-24T03:17:22.812 INFO:teuthology.orchestra.run.smithi118.stdout: "debug", 2024-03-24T03:17:22.812 INFO:teuthology.orchestra.run.smithi118.stdout: "error", 2024-03-24T03:17:22.812 INFO:teuthology.orchestra.run.smithi118.stdout: "info", 2024-03-24T03:17:22.812 INFO:teuthology.orchestra.run.smithi118.stdout: "warning" 2024-03-24T03:17:22.812 INFO:teuthology.orchestra.run.smithi118.stdout: ], 2024-03-24T03:17:22.812 INFO:teuthology.orchestra.run.smithi118.stdout: "desc": "", 2024-03-24T03:17:22.812 INFO:teuthology.orchestra.run.smithi118.stdout: "long_desc": "", 2024-03-24T03:17:22.813 INFO:teuthology.orchestra.run.smithi118.stdout: "tags": [], 2024-03-24T03:17:22.813 INFO:teuthology.orchestra.run.smithi118.stdout: "see_also": [] 2024-03-24T03:17:22.813 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-24T03:17:22.813 INFO:teuthology.orchestra.run.smithi118.stdout: "log_to_cluster": { 2024-03-24T03:17:22.813 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "log_to_cluster", 2024-03-24T03:17:22.813 INFO:teuthology.orchestra.run.smithi118.stdout: "type": "bool", 2024-03-24T03:17:22.813 INFO:teuthology.orchestra.run.smithi118.stdout: "level": "advanced", 2024-03-24T03:17:22.813 INFO:teuthology.orchestra.run.smithi118.stdout: "flags": 1, 2024-03-24T03:17:22.813 INFO:teuthology.orchestra.run.smithi118.stdout: "default_value": "False", 2024-03-24T03:17:22.814 INFO:teuthology.orchestra.run.smithi118.stdout: "min": "", 2024-03-24T03:17:22.814 INFO:teuthology.orchestra.run.smithi118.stdout: "max": "", 2024-03-24T03:17:22.814 INFO:teuthology.orchestra.run.smithi118.stdout: "enum_allowed": [], 2024-03-24T03:17:22.814 INFO:teuthology.orchestra.run.smithi118.stdout: "desc": "", 2024-03-24T03:17:22.814 INFO:teuthology.orchestra.run.smithi118.stdout: "long_desc": "", 2024-03-24T03:17:22.814 INFO:teuthology.orchestra.run.smithi118.stdout: "tags": [], 2024-03-24T03:17:22.814 INFO:teuthology.orchestra.run.smithi118.stdout: "see_also": [] 2024-03-24T03:17:22.814 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-24T03:17:22.815 INFO:teuthology.orchestra.run.smithi118.stdout: "log_to_cluster_level": { 2024-03-24T03:17:22.815 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "log_to_cluster_level", 2024-03-24T03:17:22.815 INFO:teuthology.orchestra.run.smithi118.stdout: "type": "str", 2024-03-24T03:17:22.815 INFO:teuthology.orchestra.run.smithi118.stdout: "level": "advanced", 2024-03-24T03:17:22.815 INFO:teuthology.orchestra.run.smithi118.stdout: "flags": 1, 2024-03-24T03:17:22.815 INFO:teuthology.orchestra.run.smithi118.stdout: "default_value": "info", 2024-03-24T03:17:22.815 INFO:teuthology.orchestra.run.smithi118.stdout: "min": "", 2024-03-24T03:17:22.815 INFO:teuthology.orchestra.run.smithi118.stdout: "max": "", 2024-03-24T03:17:22.815 INFO:teuthology.orchestra.run.smithi118.stdout: "enum_allowed": [ 2024-03-24T03:17:22.816 INFO:teuthology.orchestra.run.smithi118.stdout: "", 2024-03-24T03:17:22.816 INFO:teuthology.orchestra.run.smithi118.stdout: "critical", 2024-03-24T03:17:22.816 INFO:teuthology.orchestra.run.smithi118.stdout: "debug", 2024-03-24T03:17:22.816 INFO:teuthology.orchestra.run.smithi118.stdout: "error", 2024-03-24T03:17:22.816 INFO:teuthology.orchestra.run.smithi118.stdout: "info", 2024-03-24T03:17:22.816 INFO:teuthology.orchestra.run.smithi118.stdout: "warning" 2024-03-24T03:17:22.816 INFO:teuthology.orchestra.run.smithi118.stdout: ], 2024-03-24T03:17:22.816 INFO:teuthology.orchestra.run.smithi118.stdout: "desc": "", 2024-03-24T03:17:22.816 INFO:teuthology.orchestra.run.smithi118.stdout: "long_desc": "", 2024-03-24T03:17:22.816 INFO:teuthology.orchestra.run.smithi118.stdout: "tags": [], 2024-03-24T03:17:22.817 INFO:teuthology.orchestra.run.smithi118.stdout: "see_also": [] 2024-03-24T03:17:22.817 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-24T03:17:22.817 INFO:teuthology.orchestra.run.smithi118.stdout: "log_to_file": { 2024-03-24T03:17:22.817 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "log_to_file", 2024-03-24T03:17:22.817 INFO:teuthology.orchestra.run.smithi118.stdout: "type": "bool", 2024-03-24T03:17:22.817 INFO:teuthology.orchestra.run.smithi118.stdout: "level": "advanced", 2024-03-24T03:17:22.817 INFO:teuthology.orchestra.run.smithi118.stdout: "flags": 1, 2024-03-24T03:17:22.817 INFO:teuthology.orchestra.run.smithi118.stdout: "default_value": "False", 2024-03-24T03:17:22.817 INFO:teuthology.orchestra.run.smithi118.stdout: "min": "", 2024-03-24T03:17:22.818 INFO:teuthology.orchestra.run.smithi118.stdout: "max": "", 2024-03-24T03:17:22.818 INFO:teuthology.orchestra.run.smithi118.stdout: "enum_allowed": [], 2024-03-24T03:17:22.818 INFO:teuthology.orchestra.run.smithi118.stdout: "desc": "", 2024-03-24T03:17:22.818 INFO:teuthology.orchestra.run.smithi118.stdout: "long_desc": "", 2024-03-24T03:17:22.818 INFO:teuthology.orchestra.run.smithi118.stdout: "tags": [], 2024-03-24T03:17:22.818 INFO:teuthology.orchestra.run.smithi118.stdout: "see_also": [] 2024-03-24T03:17:22.818 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-03-24T03:17:22.818 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-03-24T03:17:22.819 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-24T03:17:22.819 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-24T03:17:22.819 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "k8sevents", 2024-03-24T03:17:22.819 INFO:teuthology.orchestra.run.smithi118.stdout: "can_run": true, 2024-03-24T03:17:22.819 INFO:teuthology.orchestra.run.smithi118.stdout: "error_string": "", 2024-03-24T03:17:22.819 INFO:teuthology.orchestra.run.smithi118.stdout: "module_options": { 2024-03-24T03:17:22.819 INFO:teuthology.orchestra.run.smithi118.stdout: "ceph_event_retention_days": { 2024-03-24T03:17:22.819 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "ceph_event_retention_days", 2024-03-24T03:17:22.819 INFO:teuthology.orchestra.run.smithi118.stdout: "type": "int", 2024-03-24T03:17:22.819 INFO:teuthology.orchestra.run.smithi118.stdout: "level": "advanced", 2024-03-24T03:17:22.820 INFO:teuthology.orchestra.run.smithi118.stdout: "flags": 0, 2024-03-24T03:17:22.820 INFO:teuthology.orchestra.run.smithi118.stdout: "default_value": "7", 2024-03-24T03:17:22.820 INFO:teuthology.orchestra.run.smithi118.stdout: "min": "", 2024-03-24T03:17:22.820 INFO:teuthology.orchestra.run.smithi118.stdout: "max": "", 2024-03-24T03:17:22.820 INFO:teuthology.orchestra.run.smithi118.stdout: "enum_allowed": [], 2024-03-24T03:17:22.820 INFO:teuthology.orchestra.run.smithi118.stdout: "desc": "Days to hold ceph event information within local cache", 2024-03-24T03:17:22.820 INFO:teuthology.orchestra.run.smithi118.stdout: "long_desc": "", 2024-03-24T03:17:22.820 INFO:teuthology.orchestra.run.smithi118.stdout: "tags": [], 2024-03-24T03:17:22.820 INFO:teuthology.orchestra.run.smithi118.stdout: "see_also": [] 2024-03-24T03:17:22.821 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-24T03:17:22.821 INFO:teuthology.orchestra.run.smithi118.stdout: "config_check_secs": { 2024-03-24T03:17:22.821 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "config_check_secs", 2024-03-24T03:17:22.821 INFO:teuthology.orchestra.run.smithi118.stdout: "type": "int", 2024-03-24T03:17:22.821 INFO:teuthology.orchestra.run.smithi118.stdout: "level": "advanced", 2024-03-24T03:17:22.821 INFO:teuthology.orchestra.run.smithi118.stdout: "flags": 0, 2024-03-24T03:17:22.821 INFO:teuthology.orchestra.run.smithi118.stdout: "default_value": "10", 2024-03-24T03:17:22.822 INFO:teuthology.orchestra.run.smithi118.stdout: "min": "10", 2024-03-24T03:17:22.822 INFO:teuthology.orchestra.run.smithi118.stdout: "max": "", 2024-03-24T03:17:22.822 INFO:teuthology.orchestra.run.smithi118.stdout: "enum_allowed": [], 2024-03-24T03:17:22.822 INFO:teuthology.orchestra.run.smithi118.stdout: "desc": "interval (secs) to check for cluster configuration changes", 2024-03-24T03:17:22.822 INFO:teuthology.orchestra.run.smithi118.stdout: "long_desc": "", 2024-03-24T03:17:22.822 INFO:teuthology.orchestra.run.smithi118.stdout: "tags": [], 2024-03-24T03:17:22.822 INFO:teuthology.orchestra.run.smithi118.stdout: "see_also": [] 2024-03-24T03:17:22.823 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-24T03:17:22.823 INFO:teuthology.orchestra.run.smithi118.stdout: "log_level": { 2024-03-24T03:17:22.823 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "log_level", 2024-03-24T03:17:22.823 INFO:teuthology.orchestra.run.smithi118.stdout: "type": "str", 2024-03-24T03:17:22.823 INFO:teuthology.orchestra.run.smithi118.stdout: "level": "advanced", 2024-03-24T03:17:22.823 INFO:teuthology.orchestra.run.smithi118.stdout: "flags": 1, 2024-03-24T03:17:22.823 INFO:teuthology.orchestra.run.smithi118.stdout: "default_value": "", 2024-03-24T03:17:22.823 INFO:teuthology.orchestra.run.smithi118.stdout: "min": "", 2024-03-24T03:17:22.824 INFO:teuthology.orchestra.run.smithi118.stdout: "max": "", 2024-03-24T03:17:22.824 INFO:teuthology.orchestra.run.smithi118.stdout: "enum_allowed": [ 2024-03-24T03:17:22.824 INFO:teuthology.orchestra.run.smithi118.stdout: "", 2024-03-24T03:17:22.824 INFO:teuthology.orchestra.run.smithi118.stdout: "critical", 2024-03-24T03:17:22.824 INFO:teuthology.orchestra.run.smithi118.stdout: "debug", 2024-03-24T03:17:22.824 INFO:teuthology.orchestra.run.smithi118.stdout: "error", 2024-03-24T03:17:22.824 INFO:teuthology.orchestra.run.smithi118.stdout: "info", 2024-03-24T03:17:22.824 INFO:teuthology.orchestra.run.smithi118.stdout: "warning" 2024-03-24T03:17:22.824 INFO:teuthology.orchestra.run.smithi118.stdout: ], 2024-03-24T03:17:22.825 INFO:teuthology.orchestra.run.smithi118.stdout: "desc": "", 2024-03-24T03:17:22.825 INFO:teuthology.orchestra.run.smithi118.stdout: "long_desc": "", 2024-03-24T03:17:22.825 INFO:teuthology.orchestra.run.smithi118.stdout: "tags": [], 2024-03-24T03:17:22.825 INFO:teuthology.orchestra.run.smithi118.stdout: "see_also": [] 2024-03-24T03:17:22.825 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-24T03:17:22.825 INFO:teuthology.orchestra.run.smithi118.stdout: "log_to_cluster": { 2024-03-24T03:17:22.825 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "log_to_cluster", 2024-03-24T03:17:22.825 INFO:teuthology.orchestra.run.smithi118.stdout: "type": "bool", 2024-03-24T03:17:22.825 INFO:teuthology.orchestra.run.smithi118.stdout: "level": "advanced", 2024-03-24T03:17:22.826 INFO:teuthology.orchestra.run.smithi118.stdout: "flags": 1, 2024-03-24T03:17:22.826 INFO:teuthology.orchestra.run.smithi118.stdout: "default_value": "False", 2024-03-24T03:17:22.826 INFO:teuthology.orchestra.run.smithi118.stdout: "min": "", 2024-03-24T03:17:22.826 INFO:teuthology.orchestra.run.smithi118.stdout: "max": "", 2024-03-24T03:17:22.826 INFO:teuthology.orchestra.run.smithi118.stdout: "enum_allowed": [], 2024-03-24T03:17:22.826 INFO:teuthology.orchestra.run.smithi118.stdout: "desc": "", 2024-03-24T03:17:22.826 INFO:teuthology.orchestra.run.smithi118.stdout: "long_desc": "", 2024-03-24T03:17:22.826 INFO:teuthology.orchestra.run.smithi118.stdout: "tags": [], 2024-03-24T03:17:22.827 INFO:teuthology.orchestra.run.smithi118.stdout: "see_also": [] 2024-03-24T03:17:22.827 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-24T03:17:22.827 INFO:teuthology.orchestra.run.smithi118.stdout: "log_to_cluster_level": { 2024-03-24T03:17:22.827 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "log_to_cluster_level", 2024-03-24T03:17:22.827 INFO:teuthology.orchestra.run.smithi118.stdout: "type": "str", 2024-03-24T03:17:22.827 INFO:teuthology.orchestra.run.smithi118.stdout: "level": "advanced", 2024-03-24T03:17:22.827 INFO:teuthology.orchestra.run.smithi118.stdout: "flags": 1, 2024-03-24T03:17:22.827 INFO:teuthology.orchestra.run.smithi118.stdout: "default_value": "info", 2024-03-24T03:17:22.827 INFO:teuthology.orchestra.run.smithi118.stdout: "min": "", 2024-03-24T03:17:22.827 INFO:teuthology.orchestra.run.smithi118.stdout: "max": "", 2024-03-24T03:17:22.828 INFO:teuthology.orchestra.run.smithi118.stdout: "enum_allowed": [ 2024-03-24T03:17:22.828 INFO:teuthology.orchestra.run.smithi118.stdout: "", 2024-03-24T03:17:22.828 INFO:teuthology.orchestra.run.smithi118.stdout: "critical", 2024-03-24T03:17:22.828 INFO:teuthology.orchestra.run.smithi118.stdout: "debug", 2024-03-24T03:17:22.828 INFO:teuthology.orchestra.run.smithi118.stdout: "error", 2024-03-24T03:17:22.828 INFO:teuthology.orchestra.run.smithi118.stdout: "info", 2024-03-24T03:17:22.828 INFO:teuthology.orchestra.run.smithi118.stdout: "warning" 2024-03-24T03:17:22.828 INFO:teuthology.orchestra.run.smithi118.stdout: ], 2024-03-24T03:17:22.828 INFO:teuthology.orchestra.run.smithi118.stdout: "desc": "", 2024-03-24T03:17:22.829 INFO:teuthology.orchestra.run.smithi118.stdout: "long_desc": "", 2024-03-24T03:17:22.829 INFO:teuthology.orchestra.run.smithi118.stdout: "tags": [], 2024-03-24T03:17:22.829 INFO:teuthology.orchestra.run.smithi118.stdout: "see_also": [] 2024-03-24T03:17:22.829 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-24T03:17:22.829 INFO:teuthology.orchestra.run.smithi118.stdout: "log_to_file": { 2024-03-24T03:17:22.829 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "log_to_file", 2024-03-24T03:17:22.829 INFO:teuthology.orchestra.run.smithi118.stdout: "type": "bool", 2024-03-24T03:17:22.829 INFO:teuthology.orchestra.run.smithi118.stdout: "level": "advanced", 2024-03-24T03:17:22.829 INFO:teuthology.orchestra.run.smithi118.stdout: "flags": 1, 2024-03-24T03:17:22.830 INFO:teuthology.orchestra.run.smithi118.stdout: "default_value": "False", 2024-03-24T03:17:22.830 INFO:teuthology.orchestra.run.smithi118.stdout: "min": "", 2024-03-24T03:17:22.830 INFO:teuthology.orchestra.run.smithi118.stdout: "max": "", 2024-03-24T03:17:22.830 INFO:teuthology.orchestra.run.smithi118.stdout: "enum_allowed": [], 2024-03-24T03:17:22.830 INFO:teuthology.orchestra.run.smithi118.stdout: "desc": "", 2024-03-24T03:17:22.830 INFO:teuthology.orchestra.run.smithi118.stdout: "long_desc": "", 2024-03-24T03:17:22.830 INFO:teuthology.orchestra.run.smithi118.stdout: "tags": [], 2024-03-24T03:17:22.830 INFO:teuthology.orchestra.run.smithi118.stdout: "see_also": [] 2024-03-24T03:17:22.830 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-03-24T03:17:22.831 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-03-24T03:17:22.831 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-24T03:17:22.831 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-24T03:17:22.831 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "localpool", 2024-03-24T03:17:22.831 INFO:teuthology.orchestra.run.smithi118.stdout: "can_run": true, 2024-03-24T03:17:22.831 INFO:teuthology.orchestra.run.smithi118.stdout: "error_string": "", 2024-03-24T03:17:22.831 INFO:teuthology.orchestra.run.smithi118.stdout: "module_options": { 2024-03-24T03:17:22.831 INFO:teuthology.orchestra.run.smithi118.stdout: "failure_domain": { 2024-03-24T03:17:22.832 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "failure_domain", 2024-03-24T03:17:22.832 INFO:teuthology.orchestra.run.smithi118.stdout: "type": "str", 2024-03-24T03:17:22.832 INFO:teuthology.orchestra.run.smithi118.stdout: "level": "advanced", 2024-03-24T03:17:22.832 INFO:teuthology.orchestra.run.smithi118.stdout: "flags": 1, 2024-03-24T03:17:22.832 INFO:teuthology.orchestra.run.smithi118.stdout: "default_value": "host", 2024-03-24T03:17:22.832 INFO:teuthology.orchestra.run.smithi118.stdout: "min": "", 2024-03-24T03:17:22.832 INFO:teuthology.orchestra.run.smithi118.stdout: "max": "", 2024-03-24T03:17:22.833 INFO:teuthology.orchestra.run.smithi118.stdout: "enum_allowed": [], 2024-03-24T03:17:22.833 INFO:teuthology.orchestra.run.smithi118.stdout: "desc": "failure domain for any created local pool", 2024-03-24T03:17:22.833 INFO:teuthology.orchestra.run.smithi118.stdout: "long_desc": "what failure domain we should separate data replicas across.", 2024-03-24T03:17:22.833 INFO:teuthology.orchestra.run.smithi118.stdout: "tags": [], 2024-03-24T03:17:22.833 INFO:teuthology.orchestra.run.smithi118.stdout: "see_also": [] 2024-03-24T03:17:22.833 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-24T03:17:22.833 INFO:teuthology.orchestra.run.smithi118.stdout: "log_level": { 2024-03-24T03:17:22.833 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "log_level", 2024-03-24T03:17:22.833 INFO:teuthology.orchestra.run.smithi118.stdout: "type": "str", 2024-03-24T03:17:22.834 INFO:teuthology.orchestra.run.smithi118.stdout: "level": "advanced", 2024-03-24T03:17:22.834 INFO:teuthology.orchestra.run.smithi118.stdout: "flags": 1, 2024-03-24T03:17:22.834 INFO:teuthology.orchestra.run.smithi118.stdout: "default_value": "", 2024-03-24T03:17:22.834 INFO:teuthology.orchestra.run.smithi118.stdout: "min": "", 2024-03-24T03:17:22.834 INFO:teuthology.orchestra.run.smithi118.stdout: "max": "", 2024-03-24T03:17:22.834 INFO:teuthology.orchestra.run.smithi118.stdout: "enum_allowed": [ 2024-03-24T03:17:22.834 INFO:teuthology.orchestra.run.smithi118.stdout: "", 2024-03-24T03:17:22.834 INFO:teuthology.orchestra.run.smithi118.stdout: "critical", 2024-03-24T03:17:22.835 INFO:teuthology.orchestra.run.smithi118.stdout: "debug", 2024-03-24T03:17:22.835 INFO:teuthology.orchestra.run.smithi118.stdout: "error", 2024-03-24T03:17:22.835 INFO:teuthology.orchestra.run.smithi118.stdout: "info", 2024-03-24T03:17:22.835 INFO:teuthology.orchestra.run.smithi118.stdout: "warning" 2024-03-24T03:17:22.835 INFO:teuthology.orchestra.run.smithi118.stdout: ], 2024-03-24T03:17:22.835 INFO:teuthology.orchestra.run.smithi118.stdout: "desc": "", 2024-03-24T03:17:22.835 INFO:teuthology.orchestra.run.smithi118.stdout: "long_desc": "", 2024-03-24T03:17:22.835 INFO:teuthology.orchestra.run.smithi118.stdout: "tags": [], 2024-03-24T03:17:22.835 INFO:teuthology.orchestra.run.smithi118.stdout: "see_also": [] 2024-03-24T03:17:22.836 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-24T03:17:22.836 INFO:teuthology.orchestra.run.smithi118.stdout: "log_to_cluster": { 2024-03-24T03:17:22.836 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "log_to_cluster", 2024-03-24T03:17:22.836 INFO:teuthology.orchestra.run.smithi118.stdout: "type": "bool", 2024-03-24T03:17:22.836 INFO:teuthology.orchestra.run.smithi118.stdout: "level": "advanced", 2024-03-24T03:17:22.836 INFO:teuthology.orchestra.run.smithi118.stdout: "flags": 1, 2024-03-24T03:17:22.836 INFO:teuthology.orchestra.run.smithi118.stdout: "default_value": "False", 2024-03-24T03:17:22.836 INFO:teuthology.orchestra.run.smithi118.stdout: "min": "", 2024-03-24T03:17:22.836 INFO:teuthology.orchestra.run.smithi118.stdout: "max": "", 2024-03-24T03:17:22.837 INFO:teuthology.orchestra.run.smithi118.stdout: "enum_allowed": [], 2024-03-24T03:17:22.837 INFO:teuthology.orchestra.run.smithi118.stdout: "desc": "", 2024-03-24T03:17:22.837 INFO:teuthology.orchestra.run.smithi118.stdout: "long_desc": "", 2024-03-24T03:17:22.837 INFO:teuthology.orchestra.run.smithi118.stdout: "tags": [], 2024-03-24T03:17:22.837 INFO:teuthology.orchestra.run.smithi118.stdout: "see_also": [] 2024-03-24T03:17:22.837 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-24T03:17:22.837 INFO:teuthology.orchestra.run.smithi118.stdout: "log_to_cluster_level": { 2024-03-24T03:17:22.837 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "log_to_cluster_level", 2024-03-24T03:17:22.837 INFO:teuthology.orchestra.run.smithi118.stdout: "type": "str", 2024-03-24T03:17:22.838 INFO:teuthology.orchestra.run.smithi118.stdout: "level": "advanced", 2024-03-24T03:17:22.838 INFO:teuthology.orchestra.run.smithi118.stdout: "flags": 1, 2024-03-24T03:17:22.838 INFO:teuthology.orchestra.run.smithi118.stdout: "default_value": "info", 2024-03-24T03:17:22.838 INFO:teuthology.orchestra.run.smithi118.stdout: "min": "", 2024-03-24T03:17:22.838 INFO:teuthology.orchestra.run.smithi118.stdout: "max": "", 2024-03-24T03:17:22.838 INFO:teuthology.orchestra.run.smithi118.stdout: "enum_allowed": [ 2024-03-24T03:17:22.838 INFO:teuthology.orchestra.run.smithi118.stdout: "", 2024-03-24T03:17:22.838 INFO:teuthology.orchestra.run.smithi118.stdout: "critical", 2024-03-24T03:17:22.838 INFO:teuthology.orchestra.run.smithi118.stdout: "debug", 2024-03-24T03:17:22.839 INFO:teuthology.orchestra.run.smithi118.stdout: "error", 2024-03-24T03:17:22.839 INFO:teuthology.orchestra.run.smithi118.stdout: "info", 2024-03-24T03:17:22.839 INFO:teuthology.orchestra.run.smithi118.stdout: "warning" 2024-03-24T03:17:22.839 INFO:teuthology.orchestra.run.smithi118.stdout: ], 2024-03-24T03:17:22.839 INFO:teuthology.orchestra.run.smithi118.stdout: "desc": "", 2024-03-24T03:17:22.839 INFO:teuthology.orchestra.run.smithi118.stdout: "long_desc": "", 2024-03-24T03:17:22.839 INFO:teuthology.orchestra.run.smithi118.stdout: "tags": [], 2024-03-24T03:17:22.839 INFO:teuthology.orchestra.run.smithi118.stdout: "see_also": [] 2024-03-24T03:17:22.839 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-24T03:17:22.840 INFO:teuthology.orchestra.run.smithi118.stdout: "log_to_file": { 2024-03-24T03:17:22.840 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "log_to_file", 2024-03-24T03:17:22.840 INFO:teuthology.orchestra.run.smithi118.stdout: "type": "bool", 2024-03-24T03:17:22.840 INFO:teuthology.orchestra.run.smithi118.stdout: "level": "advanced", 2024-03-24T03:17:22.840 INFO:teuthology.orchestra.run.smithi118.stdout: "flags": 1, 2024-03-24T03:17:22.840 INFO:teuthology.orchestra.run.smithi118.stdout: "default_value": "False", 2024-03-24T03:17:22.840 INFO:teuthology.orchestra.run.smithi118.stdout: "min": "", 2024-03-24T03:17:22.840 INFO:teuthology.orchestra.run.smithi118.stdout: "max": "", 2024-03-24T03:17:22.840 INFO:teuthology.orchestra.run.smithi118.stdout: "enum_allowed": [], 2024-03-24T03:17:22.841 INFO:teuthology.orchestra.run.smithi118.stdout: "desc": "", 2024-03-24T03:17:22.841 INFO:teuthology.orchestra.run.smithi118.stdout: "long_desc": "", 2024-03-24T03:17:22.841 INFO:teuthology.orchestra.run.smithi118.stdout: "tags": [], 2024-03-24T03:17:22.841 INFO:teuthology.orchestra.run.smithi118.stdout: "see_also": [] 2024-03-24T03:17:22.841 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-24T03:17:22.841 INFO:teuthology.orchestra.run.smithi118.stdout: "min_size": { 2024-03-24T03:17:22.841 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "min_size", 2024-03-24T03:17:22.841 INFO:teuthology.orchestra.run.smithi118.stdout: "type": "int", 2024-03-24T03:17:22.841 INFO:teuthology.orchestra.run.smithi118.stdout: "level": "advanced", 2024-03-24T03:17:22.842 INFO:teuthology.orchestra.run.smithi118.stdout: "flags": 1, 2024-03-24T03:17:22.842 INFO:teuthology.orchestra.run.smithi118.stdout: "default_value": "", 2024-03-24T03:17:22.842 INFO:teuthology.orchestra.run.smithi118.stdout: "min": "", 2024-03-24T03:17:22.842 INFO:teuthology.orchestra.run.smithi118.stdout: "max": "", 2024-03-24T03:17:22.842 INFO:teuthology.orchestra.run.smithi118.stdout: "enum_allowed": [], 2024-03-24T03:17:22.842 INFO:teuthology.orchestra.run.smithi118.stdout: "desc": "default min_size for any created local pool", 2024-03-24T03:17:22.842 INFO:teuthology.orchestra.run.smithi118.stdout: "long_desc": "value to set min_size to (unchanged from Ceph's default if this option is not set)", 2024-03-24T03:17:22.842 INFO:teuthology.orchestra.run.smithi118.stdout: "tags": [], 2024-03-24T03:17:22.843 INFO:teuthology.orchestra.run.smithi118.stdout: "see_also": [] 2024-03-24T03:17:22.843 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-24T03:17:22.843 INFO:teuthology.orchestra.run.smithi118.stdout: "num_rep": { 2024-03-24T03:17:22.843 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "num_rep", 2024-03-24T03:17:22.843 INFO:teuthology.orchestra.run.smithi118.stdout: "type": "int", 2024-03-24T03:17:22.843 INFO:teuthology.orchestra.run.smithi118.stdout: "level": "advanced", 2024-03-24T03:17:22.843 INFO:teuthology.orchestra.run.smithi118.stdout: "flags": 1, 2024-03-24T03:17:22.843 INFO:teuthology.orchestra.run.smithi118.stdout: "default_value": "3", 2024-03-24T03:17:22.843 INFO:teuthology.orchestra.run.smithi118.stdout: "min": "", 2024-03-24T03:17:22.844 INFO:teuthology.orchestra.run.smithi118.stdout: "max": "", 2024-03-24T03:17:22.844 INFO:teuthology.orchestra.run.smithi118.stdout: "enum_allowed": [], 2024-03-24T03:17:22.844 INFO:teuthology.orchestra.run.smithi118.stdout: "desc": "default replica count for any created local pool", 2024-03-24T03:17:22.844 INFO:teuthology.orchestra.run.smithi118.stdout: "long_desc": "", 2024-03-24T03:17:22.844 INFO:teuthology.orchestra.run.smithi118.stdout: "tags": [], 2024-03-24T03:17:22.844 INFO:teuthology.orchestra.run.smithi118.stdout: "see_also": [] 2024-03-24T03:17:22.844 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-24T03:17:22.844 INFO:teuthology.orchestra.run.smithi118.stdout: "pg_num": { 2024-03-24T03:17:22.845 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "pg_num", 2024-03-24T03:17:22.845 INFO:teuthology.orchestra.run.smithi118.stdout: "type": "int", 2024-03-24T03:17:22.845 INFO:teuthology.orchestra.run.smithi118.stdout: "level": "advanced", 2024-03-24T03:17:22.845 INFO:teuthology.orchestra.run.smithi118.stdout: "flags": 1, 2024-03-24T03:17:22.845 INFO:teuthology.orchestra.run.smithi118.stdout: "default_value": "128", 2024-03-24T03:17:22.845 INFO:teuthology.orchestra.run.smithi118.stdout: "min": "", 2024-03-24T03:17:22.845 INFO:teuthology.orchestra.run.smithi118.stdout: "max": "", 2024-03-24T03:17:22.845 INFO:teuthology.orchestra.run.smithi118.stdout: "enum_allowed": [], 2024-03-24T03:17:22.845 INFO:teuthology.orchestra.run.smithi118.stdout: "desc": "default pg_num for any created local pool", 2024-03-24T03:17:22.845 INFO:teuthology.orchestra.run.smithi118.stdout: "long_desc": "", 2024-03-24T03:17:22.846 INFO:teuthology.orchestra.run.smithi118.stdout: "tags": [], 2024-03-24T03:17:22.846 INFO:teuthology.orchestra.run.smithi118.stdout: "see_also": [] 2024-03-24T03:17:22.846 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-24T03:17:22.846 INFO:teuthology.orchestra.run.smithi118.stdout: "prefix": { 2024-03-24T03:17:22.846 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "prefix", 2024-03-24T03:17:22.846 INFO:teuthology.orchestra.run.smithi118.stdout: "type": "str", 2024-03-24T03:17:22.846 INFO:teuthology.orchestra.run.smithi118.stdout: "level": "advanced", 2024-03-24T03:17:22.846 INFO:teuthology.orchestra.run.smithi118.stdout: "flags": 1, 2024-03-24T03:17:22.847 INFO:teuthology.orchestra.run.smithi118.stdout: "default_value": "", 2024-03-24T03:17:22.847 INFO:teuthology.orchestra.run.smithi118.stdout: "min": "", 2024-03-24T03:17:22.847 INFO:teuthology.orchestra.run.smithi118.stdout: "max": "", 2024-03-24T03:17:22.847 INFO:teuthology.orchestra.run.smithi118.stdout: "enum_allowed": [], 2024-03-24T03:17:22.847 INFO:teuthology.orchestra.run.smithi118.stdout: "desc": "name prefix for any created local pool", 2024-03-24T03:17:22.847 INFO:teuthology.orchestra.run.smithi118.stdout: "long_desc": "", 2024-03-24T03:17:22.847 INFO:teuthology.orchestra.run.smithi118.stdout: "tags": [], 2024-03-24T03:17:22.847 INFO:teuthology.orchestra.run.smithi118.stdout: "see_also": [] 2024-03-24T03:17:22.847 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-24T03:17:22.848 INFO:teuthology.orchestra.run.smithi118.stdout: "subtree": { 2024-03-24T03:17:22.848 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "subtree", 2024-03-24T03:17:22.848 INFO:teuthology.orchestra.run.smithi118.stdout: "type": "str", 2024-03-24T03:17:22.848 INFO:teuthology.orchestra.run.smithi118.stdout: "level": "advanced", 2024-03-24T03:17:22.848 INFO:teuthology.orchestra.run.smithi118.stdout: "flags": 1, 2024-03-24T03:17:22.848 INFO:teuthology.orchestra.run.smithi118.stdout: "default_value": "rack", 2024-03-24T03:17:22.848 INFO:teuthology.orchestra.run.smithi118.stdout: "min": "", 2024-03-24T03:17:22.848 INFO:teuthology.orchestra.run.smithi118.stdout: "max": "", 2024-03-24T03:17:22.848 INFO:teuthology.orchestra.run.smithi118.stdout: "enum_allowed": [], 2024-03-24T03:17:22.849 INFO:teuthology.orchestra.run.smithi118.stdout: "desc": "CRUSH level for which to create a local pool", 2024-03-24T03:17:22.849 INFO:teuthology.orchestra.run.smithi118.stdout: "long_desc": "which CRUSH subtree type the module should create a pool for.", 2024-03-24T03:17:22.849 INFO:teuthology.orchestra.run.smithi118.stdout: "tags": [], 2024-03-24T03:17:22.849 INFO:teuthology.orchestra.run.smithi118.stdout: "see_also": [] 2024-03-24T03:17:22.849 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-03-24T03:17:22.849 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-03-24T03:17:22.849 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-24T03:17:22.849 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-24T03:17:22.849 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "mds_autoscaler", 2024-03-24T03:17:22.849 INFO:teuthology.orchestra.run.smithi118.stdout: "can_run": true, 2024-03-24T03:17:22.850 INFO:teuthology.orchestra.run.smithi118.stdout: "error_string": "", 2024-03-24T03:17:22.850 INFO:teuthology.orchestra.run.smithi118.stdout: "module_options": { 2024-03-24T03:17:22.850 INFO:teuthology.orchestra.run.smithi118.stdout: "log_level": { 2024-03-24T03:17:22.850 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "log_level", 2024-03-24T03:17:22.850 INFO:teuthology.orchestra.run.smithi118.stdout: "type": "str", 2024-03-24T03:17:22.850 INFO:teuthology.orchestra.run.smithi118.stdout: "level": "advanced", 2024-03-24T03:17:22.850 INFO:teuthology.orchestra.run.smithi118.stdout: "flags": 1, 2024-03-24T03:17:22.850 INFO:teuthology.orchestra.run.smithi118.stdout: "default_value": "", 2024-03-24T03:17:22.851 INFO:teuthology.orchestra.run.smithi118.stdout: "min": "", 2024-03-24T03:17:22.851 INFO:teuthology.orchestra.run.smithi118.stdout: "max": "", 2024-03-24T03:17:22.851 INFO:teuthology.orchestra.run.smithi118.stdout: "enum_allowed": [ 2024-03-24T03:17:22.851 INFO:teuthology.orchestra.run.smithi118.stdout: "", 2024-03-24T03:17:22.851 INFO:teuthology.orchestra.run.smithi118.stdout: "critical", 2024-03-24T03:17:22.851 INFO:teuthology.orchestra.run.smithi118.stdout: "debug", 2024-03-24T03:17:22.851 INFO:teuthology.orchestra.run.smithi118.stdout: "error", 2024-03-24T03:17:22.851 INFO:teuthology.orchestra.run.smithi118.stdout: "info", 2024-03-24T03:17:22.851 INFO:teuthology.orchestra.run.smithi118.stdout: "warning" 2024-03-24T03:17:22.852 INFO:teuthology.orchestra.run.smithi118.stdout: ], 2024-03-24T03:17:22.852 INFO:teuthology.orchestra.run.smithi118.stdout: "desc": "", 2024-03-24T03:17:22.852 INFO:teuthology.orchestra.run.smithi118.stdout: "long_desc": "", 2024-03-24T03:17:22.852 INFO:teuthology.orchestra.run.smithi118.stdout: "tags": [], 2024-03-24T03:17:22.852 INFO:teuthology.orchestra.run.smithi118.stdout: "see_also": [] 2024-03-24T03:17:22.852 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-24T03:17:22.852 INFO:teuthology.orchestra.run.smithi118.stdout: "log_to_cluster": { 2024-03-24T03:17:22.852 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "log_to_cluster", 2024-03-24T03:17:22.852 INFO:teuthology.orchestra.run.smithi118.stdout: "type": "bool", 2024-03-24T03:17:22.852 INFO:teuthology.orchestra.run.smithi118.stdout: "level": "advanced", 2024-03-24T03:17:22.853 INFO:teuthology.orchestra.run.smithi118.stdout: "flags": 1, 2024-03-24T03:17:22.853 INFO:teuthology.orchestra.run.smithi118.stdout: "default_value": "False", 2024-03-24T03:17:22.853 INFO:teuthology.orchestra.run.smithi118.stdout: "min": "", 2024-03-24T03:17:22.853 INFO:teuthology.orchestra.run.smithi118.stdout: "max": "", 2024-03-24T03:17:22.853 INFO:teuthology.orchestra.run.smithi118.stdout: "enum_allowed": [], 2024-03-24T03:17:22.853 INFO:teuthology.orchestra.run.smithi118.stdout: "desc": "", 2024-03-24T03:17:22.853 INFO:teuthology.orchestra.run.smithi118.stdout: "long_desc": "", 2024-03-24T03:17:22.854 INFO:teuthology.orchestra.run.smithi118.stdout: "tags": [], 2024-03-24T03:17:22.854 INFO:teuthology.orchestra.run.smithi118.stdout: "see_also": [] 2024-03-24T03:17:22.854 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-24T03:17:22.854 INFO:teuthology.orchestra.run.smithi118.stdout: "log_to_cluster_level": { 2024-03-24T03:17:22.854 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "log_to_cluster_level", 2024-03-24T03:17:22.854 INFO:teuthology.orchestra.run.smithi118.stdout: "type": "str", 2024-03-24T03:17:22.854 INFO:teuthology.orchestra.run.smithi118.stdout: "level": "advanced", 2024-03-24T03:17:22.854 INFO:teuthology.orchestra.run.smithi118.stdout: "flags": 1, 2024-03-24T03:17:22.855 INFO:teuthology.orchestra.run.smithi118.stdout: "default_value": "info", 2024-03-24T03:17:22.855 INFO:teuthology.orchestra.run.smithi118.stdout: "min": "", 2024-03-24T03:17:22.855 INFO:teuthology.orchestra.run.smithi118.stdout: "max": "", 2024-03-24T03:17:22.855 INFO:teuthology.orchestra.run.smithi118.stdout: "enum_allowed": [ 2024-03-24T03:17:22.855 INFO:teuthology.orchestra.run.smithi118.stdout: "", 2024-03-24T03:17:22.855 INFO:teuthology.orchestra.run.smithi118.stdout: "critical", 2024-03-24T03:17:22.855 INFO:teuthology.orchestra.run.smithi118.stdout: "debug", 2024-03-24T03:17:22.855 INFO:teuthology.orchestra.run.smithi118.stdout: "error", 2024-03-24T03:17:22.855 INFO:teuthology.orchestra.run.smithi118.stdout: "info", 2024-03-24T03:17:22.856 INFO:teuthology.orchestra.run.smithi118.stdout: "warning" 2024-03-24T03:17:22.856 INFO:teuthology.orchestra.run.smithi118.stdout: ], 2024-03-24T03:17:22.856 INFO:teuthology.orchestra.run.smithi118.stdout: "desc": "", 2024-03-24T03:17:22.856 INFO:teuthology.orchestra.run.smithi118.stdout: "long_desc": "", 2024-03-24T03:17:22.856 INFO:teuthology.orchestra.run.smithi118.stdout: "tags": [], 2024-03-24T03:17:22.856 INFO:teuthology.orchestra.run.smithi118.stdout: "see_also": [] 2024-03-24T03:17:22.856 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-24T03:17:22.856 INFO:teuthology.orchestra.run.smithi118.stdout: "log_to_file": { 2024-03-24T03:17:22.856 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "log_to_file", 2024-03-24T03:17:22.857 INFO:teuthology.orchestra.run.smithi118.stdout: "type": "bool", 2024-03-24T03:17:22.857 INFO:teuthology.orchestra.run.smithi118.stdout: "level": "advanced", 2024-03-24T03:17:22.857 INFO:teuthology.orchestra.run.smithi118.stdout: "flags": 1, 2024-03-24T03:17:22.857 INFO:teuthology.orchestra.run.smithi118.stdout: "default_value": "False", 2024-03-24T03:17:22.857 INFO:teuthology.orchestra.run.smithi118.stdout: "min": "", 2024-03-24T03:17:22.857 INFO:teuthology.orchestra.run.smithi118.stdout: "max": "", 2024-03-24T03:17:22.857 INFO:teuthology.orchestra.run.smithi118.stdout: "enum_allowed": [], 2024-03-24T03:17:22.857 INFO:teuthology.orchestra.run.smithi118.stdout: "desc": "", 2024-03-24T03:17:22.857 INFO:teuthology.orchestra.run.smithi118.stdout: "long_desc": "", 2024-03-24T03:17:22.858 INFO:teuthology.orchestra.run.smithi118.stdout: "tags": [], 2024-03-24T03:17:22.858 INFO:teuthology.orchestra.run.smithi118.stdout: "see_also": [] 2024-03-24T03:17:22.858 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-03-24T03:17:22.858 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-03-24T03:17:22.858 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-24T03:17:22.858 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-24T03:17:22.858 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "mirroring", 2024-03-24T03:17:22.858 INFO:teuthology.orchestra.run.smithi118.stdout: "can_run": true, 2024-03-24T03:17:22.859 INFO:teuthology.orchestra.run.smithi118.stdout: "error_string": "", 2024-03-24T03:17:22.859 INFO:teuthology.orchestra.run.smithi118.stdout: "module_options": { 2024-03-24T03:17:22.859 INFO:teuthology.orchestra.run.smithi118.stdout: "log_level": { 2024-03-24T03:17:22.859 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "log_level", 2024-03-24T03:17:22.859 INFO:teuthology.orchestra.run.smithi118.stdout: "type": "str", 2024-03-24T03:17:22.859 INFO:teuthology.orchestra.run.smithi118.stdout: "level": "advanced", 2024-03-24T03:17:22.859 INFO:teuthology.orchestra.run.smithi118.stdout: "flags": 1, 2024-03-24T03:17:22.859 INFO:teuthology.orchestra.run.smithi118.stdout: "default_value": "", 2024-03-24T03:17:22.859 INFO:teuthology.orchestra.run.smithi118.stdout: "min": "", 2024-03-24T03:17:22.860 INFO:teuthology.orchestra.run.smithi118.stdout: "max": "", 2024-03-24T03:17:22.860 INFO:teuthology.orchestra.run.smithi118.stdout: "enum_allowed": [ 2024-03-24T03:17:22.860 INFO:teuthology.orchestra.run.smithi118.stdout: "", 2024-03-24T03:17:22.860 INFO:teuthology.orchestra.run.smithi118.stdout: "critical", 2024-03-24T03:17:22.860 INFO:teuthology.orchestra.run.smithi118.stdout: "debug", 2024-03-24T03:17:22.860 INFO:teuthology.orchestra.run.smithi118.stdout: "error", 2024-03-24T03:17:22.860 INFO:teuthology.orchestra.run.smithi118.stdout: "info", 2024-03-24T03:17:22.860 INFO:teuthology.orchestra.run.smithi118.stdout: "warning" 2024-03-24T03:17:22.861 INFO:teuthology.orchestra.run.smithi118.stdout: ], 2024-03-24T03:17:22.861 INFO:teuthology.orchestra.run.smithi118.stdout: "desc": "", 2024-03-24T03:17:22.861 INFO:teuthology.orchestra.run.smithi118.stdout: "long_desc": "", 2024-03-24T03:17:22.861 INFO:teuthology.orchestra.run.smithi118.stdout: "tags": [], 2024-03-24T03:17:22.861 INFO:teuthology.orchestra.run.smithi118.stdout: "see_also": [] 2024-03-24T03:17:22.861 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-24T03:17:22.861 INFO:teuthology.orchestra.run.smithi118.stdout: "log_to_cluster": { 2024-03-24T03:17:22.861 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "log_to_cluster", 2024-03-24T03:17:22.861 INFO:teuthology.orchestra.run.smithi118.stdout: "type": "bool", 2024-03-24T03:17:22.862 INFO:teuthology.orchestra.run.smithi118.stdout: "level": "advanced", 2024-03-24T03:17:22.862 INFO:teuthology.orchestra.run.smithi118.stdout: "flags": 1, 2024-03-24T03:17:22.862 INFO:teuthology.orchestra.run.smithi118.stdout: "default_value": "False", 2024-03-24T03:17:22.862 INFO:teuthology.orchestra.run.smithi118.stdout: "min": "", 2024-03-24T03:17:22.862 INFO:teuthology.orchestra.run.smithi118.stdout: "max": "", 2024-03-24T03:17:22.862 INFO:teuthology.orchestra.run.smithi118.stdout: "enum_allowed": [], 2024-03-24T03:17:22.862 INFO:teuthology.orchestra.run.smithi118.stdout: "desc": "", 2024-03-24T03:17:22.862 INFO:teuthology.orchestra.run.smithi118.stdout: "long_desc": "", 2024-03-24T03:17:22.862 INFO:teuthology.orchestra.run.smithi118.stdout: "tags": [], 2024-03-24T03:17:22.863 INFO:teuthology.orchestra.run.smithi118.stdout: "see_also": [] 2024-03-24T03:17:22.863 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-24T03:17:22.863 INFO:teuthology.orchestra.run.smithi118.stdout: "log_to_cluster_level": { 2024-03-24T03:17:22.863 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "log_to_cluster_level", 2024-03-24T03:17:22.863 INFO:teuthology.orchestra.run.smithi118.stdout: "type": "str", 2024-03-24T03:17:22.863 INFO:teuthology.orchestra.run.smithi118.stdout: "level": "advanced", 2024-03-24T03:17:22.863 INFO:teuthology.orchestra.run.smithi118.stdout: "flags": 1, 2024-03-24T03:17:22.863 INFO:teuthology.orchestra.run.smithi118.stdout: "default_value": "info", 2024-03-24T03:17:22.864 INFO:teuthology.orchestra.run.smithi118.stdout: "min": "", 2024-03-24T03:17:22.864 INFO:teuthology.orchestra.run.smithi118.stdout: "max": "", 2024-03-24T03:17:22.864 INFO:teuthology.orchestra.run.smithi118.stdout: "enum_allowed": [ 2024-03-24T03:17:22.864 INFO:teuthology.orchestra.run.smithi118.stdout: "", 2024-03-24T03:17:22.864 INFO:teuthology.orchestra.run.smithi118.stdout: "critical", 2024-03-24T03:17:22.864 INFO:teuthology.orchestra.run.smithi118.stdout: "debug", 2024-03-24T03:17:22.864 INFO:teuthology.orchestra.run.smithi118.stdout: "error", 2024-03-24T03:17:22.864 INFO:teuthology.orchestra.run.smithi118.stdout: "info", 2024-03-24T03:17:22.864 INFO:teuthology.orchestra.run.smithi118.stdout: "warning" 2024-03-24T03:17:22.865 INFO:teuthology.orchestra.run.smithi118.stdout: ], 2024-03-24T03:17:22.865 INFO:teuthology.orchestra.run.smithi118.stdout: "desc": "", 2024-03-24T03:17:22.865 INFO:teuthology.orchestra.run.smithi118.stdout: "long_desc": "", 2024-03-24T03:17:22.865 INFO:teuthology.orchestra.run.smithi118.stdout: "tags": [], 2024-03-24T03:17:22.865 INFO:teuthology.orchestra.run.smithi118.stdout: "see_also": [] 2024-03-24T03:17:22.865 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-24T03:17:22.865 INFO:teuthology.orchestra.run.smithi118.stdout: "log_to_file": { 2024-03-24T03:17:22.865 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "log_to_file", 2024-03-24T03:17:22.865 INFO:teuthology.orchestra.run.smithi118.stdout: "type": "bool", 2024-03-24T03:17:22.865 INFO:teuthology.orchestra.run.smithi118.stdout: "level": "advanced", 2024-03-24T03:17:22.866 INFO:teuthology.orchestra.run.smithi118.stdout: "flags": 1, 2024-03-24T03:17:22.866 INFO:teuthology.orchestra.run.smithi118.stdout: "default_value": "False", 2024-03-24T03:17:22.866 INFO:teuthology.orchestra.run.smithi118.stdout: "min": "", 2024-03-24T03:17:22.866 INFO:teuthology.orchestra.run.smithi118.stdout: "max": "", 2024-03-24T03:17:22.866 INFO:teuthology.orchestra.run.smithi118.stdout: "enum_allowed": [], 2024-03-24T03:17:22.866 INFO:teuthology.orchestra.run.smithi118.stdout: "desc": "", 2024-03-24T03:17:22.866 INFO:teuthology.orchestra.run.smithi118.stdout: "long_desc": "", 2024-03-24T03:17:22.867 INFO:teuthology.orchestra.run.smithi118.stdout: "tags": [], 2024-03-24T03:17:22.867 INFO:teuthology.orchestra.run.smithi118.stdout: "see_also": [] 2024-03-24T03:17:22.867 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-03-24T03:17:22.867 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-03-24T03:17:22.867 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-24T03:17:22.867 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-24T03:17:22.867 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "nfs", 2024-03-24T03:17:22.867 INFO:teuthology.orchestra.run.smithi118.stdout: "can_run": true, 2024-03-24T03:17:22.867 INFO:teuthology.orchestra.run.smithi118.stdout: "error_string": "", 2024-03-24T03:17:22.867 INFO:teuthology.orchestra.run.smithi118.stdout: "module_options": { 2024-03-24T03:17:22.868 INFO:teuthology.orchestra.run.smithi118.stdout: "log_level": { 2024-03-24T03:17:22.868 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "log_level", 2024-03-24T03:17:22.868 INFO:teuthology.orchestra.run.smithi118.stdout: "type": "str", 2024-03-24T03:17:22.868 INFO:teuthology.orchestra.run.smithi118.stdout: "level": "advanced", 2024-03-24T03:17:22.868 INFO:teuthology.orchestra.run.smithi118.stdout: "flags": 1, 2024-03-24T03:17:22.868 INFO:teuthology.orchestra.run.smithi118.stdout: "default_value": "", 2024-03-24T03:17:22.868 INFO:teuthology.orchestra.run.smithi118.stdout: "min": "", 2024-03-24T03:17:22.868 INFO:teuthology.orchestra.run.smithi118.stdout: "max": "", 2024-03-24T03:17:22.868 INFO:teuthology.orchestra.run.smithi118.stdout: "enum_allowed": [ 2024-03-24T03:17:22.869 INFO:teuthology.orchestra.run.smithi118.stdout: "", 2024-03-24T03:17:22.869 INFO:teuthology.orchestra.run.smithi118.stdout: "critical", 2024-03-24T03:17:22.869 INFO:teuthology.orchestra.run.smithi118.stdout: "debug", 2024-03-24T03:17:22.869 INFO:teuthology.orchestra.run.smithi118.stdout: "error", 2024-03-24T03:17:22.869 INFO:teuthology.orchestra.run.smithi118.stdout: "info", 2024-03-24T03:17:22.869 INFO:teuthology.orchestra.run.smithi118.stdout: "warning" 2024-03-24T03:17:22.869 INFO:teuthology.orchestra.run.smithi118.stdout: ], 2024-03-24T03:17:22.869 INFO:teuthology.orchestra.run.smithi118.stdout: "desc": "", 2024-03-24T03:17:22.869 INFO:teuthology.orchestra.run.smithi118.stdout: "long_desc": "", 2024-03-24T03:17:22.869 INFO:teuthology.orchestra.run.smithi118.stdout: "tags": [], 2024-03-24T03:17:22.870 INFO:teuthology.orchestra.run.smithi118.stdout: "see_also": [] 2024-03-24T03:17:22.870 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-24T03:17:22.870 INFO:teuthology.orchestra.run.smithi118.stdout: "log_to_cluster": { 2024-03-24T03:17:22.870 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "log_to_cluster", 2024-03-24T03:17:22.870 INFO:teuthology.orchestra.run.smithi118.stdout: "type": "bool", 2024-03-24T03:17:22.870 INFO:teuthology.orchestra.run.smithi118.stdout: "level": "advanced", 2024-03-24T03:17:22.870 INFO:teuthology.orchestra.run.smithi118.stdout: "flags": 1, 2024-03-24T03:17:22.870 INFO:teuthology.orchestra.run.smithi118.stdout: "default_value": "False", 2024-03-24T03:17:22.870 INFO:teuthology.orchestra.run.smithi118.stdout: "min": "", 2024-03-24T03:17:22.871 INFO:teuthology.orchestra.run.smithi118.stdout: "max": "", 2024-03-24T03:17:22.871 INFO:teuthology.orchestra.run.smithi118.stdout: "enum_allowed": [], 2024-03-24T03:17:22.871 INFO:teuthology.orchestra.run.smithi118.stdout: "desc": "", 2024-03-24T03:17:22.871 INFO:teuthology.orchestra.run.smithi118.stdout: "long_desc": "", 2024-03-24T03:17:22.871 INFO:teuthology.orchestra.run.smithi118.stdout: "tags": [], 2024-03-24T03:17:22.871 INFO:teuthology.orchestra.run.smithi118.stdout: "see_also": [] 2024-03-24T03:17:22.871 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-24T03:17:22.871 INFO:teuthology.orchestra.run.smithi118.stdout: "log_to_cluster_level": { 2024-03-24T03:17:22.871 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "log_to_cluster_level", 2024-03-24T03:17:22.872 INFO:teuthology.orchestra.run.smithi118.stdout: "type": "str", 2024-03-24T03:17:22.872 INFO:teuthology.orchestra.run.smithi118.stdout: "level": "advanced", 2024-03-24T03:17:22.872 INFO:teuthology.orchestra.run.smithi118.stdout: "flags": 1, 2024-03-24T03:17:22.872 INFO:teuthology.orchestra.run.smithi118.stdout: "default_value": "info", 2024-03-24T03:17:22.872 INFO:teuthology.orchestra.run.smithi118.stdout: "min": "", 2024-03-24T03:17:22.872 INFO:teuthology.orchestra.run.smithi118.stdout: "max": "", 2024-03-24T03:17:22.872 INFO:teuthology.orchestra.run.smithi118.stdout: "enum_allowed": [ 2024-03-24T03:17:22.872 INFO:teuthology.orchestra.run.smithi118.stdout: "", 2024-03-24T03:17:22.872 INFO:teuthology.orchestra.run.smithi118.stdout: "critical", 2024-03-24T03:17:22.873 INFO:teuthology.orchestra.run.smithi118.stdout: "debug", 2024-03-24T03:17:22.873 INFO:teuthology.orchestra.run.smithi118.stdout: "error", 2024-03-24T03:17:22.873 INFO:teuthology.orchestra.run.smithi118.stdout: "info", 2024-03-24T03:17:22.873 INFO:teuthology.orchestra.run.smithi118.stdout: "warning" 2024-03-24T03:17:22.873 INFO:teuthology.orchestra.run.smithi118.stdout: ], 2024-03-24T03:17:22.873 INFO:teuthology.orchestra.run.smithi118.stdout: "desc": "", 2024-03-24T03:17:22.873 INFO:teuthology.orchestra.run.smithi118.stdout: "long_desc": "", 2024-03-24T03:17:22.873 INFO:teuthology.orchestra.run.smithi118.stdout: "tags": [], 2024-03-24T03:17:22.874 INFO:teuthology.orchestra.run.smithi118.stdout: "see_also": [] 2024-03-24T03:17:22.874 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-24T03:17:22.874 INFO:teuthology.orchestra.run.smithi118.stdout: "log_to_file": { 2024-03-24T03:17:22.874 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "log_to_file", 2024-03-24T03:17:22.874 INFO:teuthology.orchestra.run.smithi118.stdout: "type": "bool", 2024-03-24T03:17:22.874 INFO:teuthology.orchestra.run.smithi118.stdout: "level": "advanced", 2024-03-24T03:17:22.874 INFO:teuthology.orchestra.run.smithi118.stdout: "flags": 1, 2024-03-24T03:17:22.874 INFO:teuthology.orchestra.run.smithi118.stdout: "default_value": "False", 2024-03-24T03:17:22.875 INFO:teuthology.orchestra.run.smithi118.stdout: "min": "", 2024-03-24T03:17:22.875 INFO:teuthology.orchestra.run.smithi118.stdout: "max": "", 2024-03-24T03:17:22.875 INFO:teuthology.orchestra.run.smithi118.stdout: "enum_allowed": [], 2024-03-24T03:17:22.875 INFO:teuthology.orchestra.run.smithi118.stdout: "desc": "", 2024-03-24T03:17:22.875 INFO:teuthology.orchestra.run.smithi118.stdout: "long_desc": "", 2024-03-24T03:17:22.875 INFO:teuthology.orchestra.run.smithi118.stdout: "tags": [], 2024-03-24T03:17:22.875 INFO:teuthology.orchestra.run.smithi118.stdout: "see_also": [] 2024-03-24T03:17:22.875 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-03-24T03:17:22.875 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-03-24T03:17:22.876 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-24T03:17:22.876 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-24T03:17:22.876 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "orchestrator", 2024-03-24T03:17:22.876 INFO:teuthology.orchestra.run.smithi118.stdout: "can_run": true, 2024-03-24T03:17:22.876 INFO:teuthology.orchestra.run.smithi118.stdout: "error_string": "", 2024-03-24T03:17:22.876 INFO:teuthology.orchestra.run.smithi118.stdout: "module_options": { 2024-03-24T03:17:22.876 INFO:teuthology.orchestra.run.smithi118.stdout: "fail_fs": { 2024-03-24T03:17:22.876 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "fail_fs", 2024-03-24T03:17:22.876 INFO:teuthology.orchestra.run.smithi118.stdout: "type": "bool", 2024-03-24T03:17:22.877 INFO:teuthology.orchestra.run.smithi118.stdout: "level": "advanced", 2024-03-24T03:17:22.877 INFO:teuthology.orchestra.run.smithi118.stdout: "flags": 0, 2024-03-24T03:17:22.877 INFO:teuthology.orchestra.run.smithi118.stdout: "default_value": "False", 2024-03-24T03:17:22.877 INFO:teuthology.orchestra.run.smithi118.stdout: "min": "", 2024-03-24T03:17:22.877 INFO:teuthology.orchestra.run.smithi118.stdout: "max": "", 2024-03-24T03:17:22.877 INFO:teuthology.orchestra.run.smithi118.stdout: "enum_allowed": [], 2024-03-24T03:17:22.877 INFO:teuthology.orchestra.run.smithi118.stdout: "desc": "Fail filesystem for rapid multi-rank mds upgrade", 2024-03-24T03:17:22.877 INFO:teuthology.orchestra.run.smithi118.stdout: "long_desc": "", 2024-03-24T03:17:22.877 INFO:teuthology.orchestra.run.smithi118.stdout: "tags": [], 2024-03-24T03:17:22.878 INFO:teuthology.orchestra.run.smithi118.stdout: "see_also": [] 2024-03-24T03:17:22.878 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-24T03:17:22.878 INFO:teuthology.orchestra.run.smithi118.stdout: "log_level": { 2024-03-24T03:17:22.878 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "log_level", 2024-03-24T03:17:22.878 INFO:teuthology.orchestra.run.smithi118.stdout: "type": "str", 2024-03-24T03:17:22.878 INFO:teuthology.orchestra.run.smithi118.stdout: "level": "advanced", 2024-03-24T03:17:22.878 INFO:teuthology.orchestra.run.smithi118.stdout: "flags": 1, 2024-03-24T03:17:22.878 INFO:teuthology.orchestra.run.smithi118.stdout: "default_value": "", 2024-03-24T03:17:22.878 INFO:teuthology.orchestra.run.smithi118.stdout: "min": "", 2024-03-24T03:17:22.879 INFO:teuthology.orchestra.run.smithi118.stdout: "max": "", 2024-03-24T03:17:22.879 INFO:teuthology.orchestra.run.smithi118.stdout: "enum_allowed": [ 2024-03-24T03:17:22.879 INFO:teuthology.orchestra.run.smithi118.stdout: "", 2024-03-24T03:17:22.879 INFO:teuthology.orchestra.run.smithi118.stdout: "critical", 2024-03-24T03:17:22.879 INFO:teuthology.orchestra.run.smithi118.stdout: "debug", 2024-03-24T03:17:22.879 INFO:teuthology.orchestra.run.smithi118.stdout: "error", 2024-03-24T03:17:22.879 INFO:teuthology.orchestra.run.smithi118.stdout: "info", 2024-03-24T03:17:22.879 INFO:teuthology.orchestra.run.smithi118.stdout: "warning" 2024-03-24T03:17:22.879 INFO:teuthology.orchestra.run.smithi118.stdout: ], 2024-03-24T03:17:22.879 INFO:teuthology.orchestra.run.smithi118.stdout: "desc": "", 2024-03-24T03:17:22.880 INFO:teuthology.orchestra.run.smithi118.stdout: "long_desc": "", 2024-03-24T03:17:22.880 INFO:teuthology.orchestra.run.smithi118.stdout: "tags": [], 2024-03-24T03:17:22.880 INFO:teuthology.orchestra.run.smithi118.stdout: "see_also": [] 2024-03-24T03:17:22.880 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-24T03:17:22.880 INFO:teuthology.orchestra.run.smithi118.stdout: "log_to_cluster": { 2024-03-24T03:17:22.880 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "log_to_cluster", 2024-03-24T03:17:22.880 INFO:teuthology.orchestra.run.smithi118.stdout: "type": "bool", 2024-03-24T03:17:22.880 INFO:teuthology.orchestra.run.smithi118.stdout: "level": "advanced", 2024-03-24T03:17:22.880 INFO:teuthology.orchestra.run.smithi118.stdout: "flags": 1, 2024-03-24T03:17:22.881 INFO:teuthology.orchestra.run.smithi118.stdout: "default_value": "False", 2024-03-24T03:17:22.881 INFO:teuthology.orchestra.run.smithi118.stdout: "min": "", 2024-03-24T03:17:22.881 INFO:teuthology.orchestra.run.smithi118.stdout: "max": "", 2024-03-24T03:17:22.881 INFO:teuthology.orchestra.run.smithi118.stdout: "enum_allowed": [], 2024-03-24T03:17:22.881 INFO:teuthology.orchestra.run.smithi118.stdout: "desc": "", 2024-03-24T03:17:22.881 INFO:teuthology.orchestra.run.smithi118.stdout: "long_desc": "", 2024-03-24T03:17:22.881 INFO:teuthology.orchestra.run.smithi118.stdout: "tags": [], 2024-03-24T03:17:22.881 INFO:teuthology.orchestra.run.smithi118.stdout: "see_also": [] 2024-03-24T03:17:22.881 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-24T03:17:22.882 INFO:teuthology.orchestra.run.smithi118.stdout: "log_to_cluster_level": { 2024-03-24T03:17:22.882 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "log_to_cluster_level", 2024-03-24T03:17:22.882 INFO:teuthology.orchestra.run.smithi118.stdout: "type": "str", 2024-03-24T03:17:22.882 INFO:teuthology.orchestra.run.smithi118.stdout: "level": "advanced", 2024-03-24T03:17:22.882 INFO:teuthology.orchestra.run.smithi118.stdout: "flags": 1, 2024-03-24T03:17:22.882 INFO:teuthology.orchestra.run.smithi118.stdout: "default_value": "info", 2024-03-24T03:17:22.882 INFO:teuthology.orchestra.run.smithi118.stdout: "min": "", 2024-03-24T03:17:22.882 INFO:teuthology.orchestra.run.smithi118.stdout: "max": "", 2024-03-24T03:17:22.882 INFO:teuthology.orchestra.run.smithi118.stdout: "enum_allowed": [ 2024-03-24T03:17:22.883 INFO:teuthology.orchestra.run.smithi118.stdout: "", 2024-03-24T03:17:22.883 INFO:teuthology.orchestra.run.smithi118.stdout: "critical", 2024-03-24T03:17:22.883 INFO:teuthology.orchestra.run.smithi118.stdout: "debug", 2024-03-24T03:17:22.883 INFO:teuthology.orchestra.run.smithi118.stdout: "error", 2024-03-24T03:17:22.883 INFO:teuthology.orchestra.run.smithi118.stdout: "info", 2024-03-24T03:17:22.883 INFO:teuthology.orchestra.run.smithi118.stdout: "warning" 2024-03-24T03:17:22.883 INFO:teuthology.orchestra.run.smithi118.stdout: ], 2024-03-24T03:17:22.883 INFO:teuthology.orchestra.run.smithi118.stdout: "desc": "", 2024-03-24T03:17:22.883 INFO:teuthology.orchestra.run.smithi118.stdout: "long_desc": "", 2024-03-24T03:17:22.884 INFO:teuthology.orchestra.run.smithi118.stdout: "tags": [], 2024-03-24T03:17:22.884 INFO:teuthology.orchestra.run.smithi118.stdout: "see_also": [] 2024-03-24T03:17:22.884 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-24T03:17:22.884 INFO:teuthology.orchestra.run.smithi118.stdout: "log_to_file": { 2024-03-24T03:17:22.884 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "log_to_file", 2024-03-24T03:17:22.884 INFO:teuthology.orchestra.run.smithi118.stdout: "type": "bool", 2024-03-24T03:17:22.884 INFO:teuthology.orchestra.run.smithi118.stdout: "level": "advanced", 2024-03-24T03:17:22.884 INFO:teuthology.orchestra.run.smithi118.stdout: "flags": 1, 2024-03-24T03:17:22.884 INFO:teuthology.orchestra.run.smithi118.stdout: "default_value": "False", 2024-03-24T03:17:22.885 INFO:teuthology.orchestra.run.smithi118.stdout: "min": "", 2024-03-24T03:17:22.885 INFO:teuthology.orchestra.run.smithi118.stdout: "max": "", 2024-03-24T03:17:22.885 INFO:teuthology.orchestra.run.smithi118.stdout: "enum_allowed": [], 2024-03-24T03:17:22.885 INFO:teuthology.orchestra.run.smithi118.stdout: "desc": "", 2024-03-24T03:17:22.885 INFO:teuthology.orchestra.run.smithi118.stdout: "long_desc": "", 2024-03-24T03:17:22.885 INFO:teuthology.orchestra.run.smithi118.stdout: "tags": [], 2024-03-24T03:17:22.885 INFO:teuthology.orchestra.run.smithi118.stdout: "see_also": [] 2024-03-24T03:17:22.885 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-24T03:17:22.885 INFO:teuthology.orchestra.run.smithi118.stdout: "orchestrator": { 2024-03-24T03:17:22.886 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "orchestrator", 2024-03-24T03:17:22.886 INFO:teuthology.orchestra.run.smithi118.stdout: "type": "str", 2024-03-24T03:17:22.886 INFO:teuthology.orchestra.run.smithi118.stdout: "level": "advanced", 2024-03-24T03:17:22.886 INFO:teuthology.orchestra.run.smithi118.stdout: "flags": 1, 2024-03-24T03:17:22.886 INFO:teuthology.orchestra.run.smithi118.stdout: "default_value": "", 2024-03-24T03:17:22.886 INFO:teuthology.orchestra.run.smithi118.stdout: "min": "", 2024-03-24T03:17:22.886 INFO:teuthology.orchestra.run.smithi118.stdout: "max": "", 2024-03-24T03:17:22.886 INFO:teuthology.orchestra.run.smithi118.stdout: "enum_allowed": [ 2024-03-24T03:17:22.887 INFO:teuthology.orchestra.run.smithi118.stdout: "cephadm", 2024-03-24T03:17:22.887 INFO:teuthology.orchestra.run.smithi118.stdout: "rook", 2024-03-24T03:17:22.887 INFO:teuthology.orchestra.run.smithi118.stdout: "test_orchestrator" 2024-03-24T03:17:22.887 INFO:teuthology.orchestra.run.smithi118.stdout: ], 2024-03-24T03:17:22.887 INFO:teuthology.orchestra.run.smithi118.stdout: "desc": "Orchestrator backend", 2024-03-24T03:17:22.887 INFO:teuthology.orchestra.run.smithi118.stdout: "long_desc": "", 2024-03-24T03:17:22.887 INFO:teuthology.orchestra.run.smithi118.stdout: "tags": [], 2024-03-24T03:17:22.887 INFO:teuthology.orchestra.run.smithi118.stdout: "see_also": [] 2024-03-24T03:17:22.887 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-03-24T03:17:22.888 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-03-24T03:17:22.888 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-24T03:17:22.888 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-24T03:17:22.888 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "osd_perf_query", 2024-03-24T03:17:22.888 INFO:teuthology.orchestra.run.smithi118.stdout: "can_run": true, 2024-03-24T03:17:22.888 INFO:teuthology.orchestra.run.smithi118.stdout: "error_string": "", 2024-03-24T03:17:22.888 INFO:teuthology.orchestra.run.smithi118.stdout: "module_options": { 2024-03-24T03:17:22.888 INFO:teuthology.orchestra.run.smithi118.stdout: "log_level": { 2024-03-24T03:17:22.889 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "log_level", 2024-03-24T03:17:22.889 INFO:teuthology.orchestra.run.smithi118.stdout: "type": "str", 2024-03-24T03:17:22.889 INFO:teuthology.orchestra.run.smithi118.stdout: "level": "advanced", 2024-03-24T03:17:22.889 INFO:teuthology.orchestra.run.smithi118.stdout: "flags": 1, 2024-03-24T03:17:22.889 INFO:teuthology.orchestra.run.smithi118.stdout: "default_value": "", 2024-03-24T03:17:22.889 INFO:teuthology.orchestra.run.smithi118.stdout: "min": "", 2024-03-24T03:17:22.889 INFO:teuthology.orchestra.run.smithi118.stdout: "max": "", 2024-03-24T03:17:22.889 INFO:teuthology.orchestra.run.smithi118.stdout: "enum_allowed": [ 2024-03-24T03:17:22.889 INFO:teuthology.orchestra.run.smithi118.stdout: "", 2024-03-24T03:17:22.889 INFO:teuthology.orchestra.run.smithi118.stdout: "critical", 2024-03-24T03:17:22.890 INFO:teuthology.orchestra.run.smithi118.stdout: "debug", 2024-03-24T03:17:22.890 INFO:teuthology.orchestra.run.smithi118.stdout: "error", 2024-03-24T03:17:22.890 INFO:teuthology.orchestra.run.smithi118.stdout: "info", 2024-03-24T03:17:22.890 INFO:teuthology.orchestra.run.smithi118.stdout: "warning" 2024-03-24T03:17:22.890 INFO:teuthology.orchestra.run.smithi118.stdout: ], 2024-03-24T03:17:22.890 INFO:teuthology.orchestra.run.smithi118.stdout: "desc": "", 2024-03-24T03:17:22.890 INFO:teuthology.orchestra.run.smithi118.stdout: "long_desc": "", 2024-03-24T03:17:22.890 INFO:teuthology.orchestra.run.smithi118.stdout: "tags": [], 2024-03-24T03:17:22.890 INFO:teuthology.orchestra.run.smithi118.stdout: "see_also": [] 2024-03-24T03:17:22.891 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-24T03:17:22.891 INFO:teuthology.orchestra.run.smithi118.stdout: "log_to_cluster": { 2024-03-24T03:17:22.891 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "log_to_cluster", 2024-03-24T03:17:22.891 INFO:teuthology.orchestra.run.smithi118.stdout: "type": "bool", 2024-03-24T03:17:22.891 INFO:teuthology.orchestra.run.smithi118.stdout: "level": "advanced", 2024-03-24T03:17:22.891 INFO:teuthology.orchestra.run.smithi118.stdout: "flags": 1, 2024-03-24T03:17:22.891 INFO:teuthology.orchestra.run.smithi118.stdout: "default_value": "False", 2024-03-24T03:17:22.891 INFO:teuthology.orchestra.run.smithi118.stdout: "min": "", 2024-03-24T03:17:22.891 INFO:teuthology.orchestra.run.smithi118.stdout: "max": "", 2024-03-24T03:17:22.892 INFO:teuthology.orchestra.run.smithi118.stdout: "enum_allowed": [], 2024-03-24T03:17:22.892 INFO:teuthology.orchestra.run.smithi118.stdout: "desc": "", 2024-03-24T03:17:22.892 INFO:teuthology.orchestra.run.smithi118.stdout: "long_desc": "", 2024-03-24T03:17:22.892 INFO:teuthology.orchestra.run.smithi118.stdout: "tags": [], 2024-03-24T03:17:22.892 INFO:teuthology.orchestra.run.smithi118.stdout: "see_also": [] 2024-03-24T03:17:22.892 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-24T03:17:22.892 INFO:teuthology.orchestra.run.smithi118.stdout: "log_to_cluster_level": { 2024-03-24T03:17:22.892 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "log_to_cluster_level", 2024-03-24T03:17:22.892 INFO:teuthology.orchestra.run.smithi118.stdout: "type": "str", 2024-03-24T03:17:22.893 INFO:teuthology.orchestra.run.smithi118.stdout: "level": "advanced", 2024-03-24T03:17:22.893 INFO:teuthology.orchestra.run.smithi118.stdout: "flags": 1, 2024-03-24T03:17:22.893 INFO:teuthology.orchestra.run.smithi118.stdout: "default_value": "info", 2024-03-24T03:17:22.893 INFO:teuthology.orchestra.run.smithi118.stdout: "min": "", 2024-03-24T03:17:22.893 INFO:teuthology.orchestra.run.smithi118.stdout: "max": "", 2024-03-24T03:17:22.893 INFO:teuthology.orchestra.run.smithi118.stdout: "enum_allowed": [ 2024-03-24T03:17:22.893 INFO:teuthology.orchestra.run.smithi118.stdout: "", 2024-03-24T03:17:22.893 INFO:teuthology.orchestra.run.smithi118.stdout: "critical", 2024-03-24T03:17:22.894 INFO:teuthology.orchestra.run.smithi118.stdout: "debug", 2024-03-24T03:17:22.894 INFO:teuthology.orchestra.run.smithi118.stdout: "error", 2024-03-24T03:17:22.894 INFO:teuthology.orchestra.run.smithi118.stdout: "info", 2024-03-24T03:17:22.894 INFO:teuthology.orchestra.run.smithi118.stdout: "warning" 2024-03-24T03:17:22.894 INFO:teuthology.orchestra.run.smithi118.stdout: ], 2024-03-24T03:17:22.894 INFO:teuthology.orchestra.run.smithi118.stdout: "desc": "", 2024-03-24T03:17:22.894 INFO:teuthology.orchestra.run.smithi118.stdout: "long_desc": "", 2024-03-24T03:17:22.894 INFO:teuthology.orchestra.run.smithi118.stdout: "tags": [], 2024-03-24T03:17:22.894 INFO:teuthology.orchestra.run.smithi118.stdout: "see_also": [] 2024-03-24T03:17:22.895 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-24T03:17:22.895 INFO:teuthology.orchestra.run.smithi118.stdout: "log_to_file": { 2024-03-24T03:17:22.895 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "log_to_file", 2024-03-24T03:17:22.895 INFO:teuthology.orchestra.run.smithi118.stdout: "type": "bool", 2024-03-24T03:17:22.895 INFO:teuthology.orchestra.run.smithi118.stdout: "level": "advanced", 2024-03-24T03:17:22.895 INFO:teuthology.orchestra.run.smithi118.stdout: "flags": 1, 2024-03-24T03:17:22.895 INFO:teuthology.orchestra.run.smithi118.stdout: "default_value": "False", 2024-03-24T03:17:22.895 INFO:teuthology.orchestra.run.smithi118.stdout: "min": "", 2024-03-24T03:17:22.895 INFO:teuthology.orchestra.run.smithi118.stdout: "max": "", 2024-03-24T03:17:22.896 INFO:teuthology.orchestra.run.smithi118.stdout: "enum_allowed": [], 2024-03-24T03:17:22.896 INFO:teuthology.orchestra.run.smithi118.stdout: "desc": "", 2024-03-24T03:17:22.896 INFO:teuthology.orchestra.run.smithi118.stdout: "long_desc": "", 2024-03-24T03:17:22.896 INFO:teuthology.orchestra.run.smithi118.stdout: "tags": [], 2024-03-24T03:17:22.896 INFO:teuthology.orchestra.run.smithi118.stdout: "see_also": [] 2024-03-24T03:17:22.896 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-03-24T03:17:22.896 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-03-24T03:17:22.896 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-24T03:17:22.896 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-24T03:17:22.897 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "osd_support", 2024-03-24T03:17:22.897 INFO:teuthology.orchestra.run.smithi118.stdout: "can_run": true, 2024-03-24T03:17:22.897 INFO:teuthology.orchestra.run.smithi118.stdout: "error_string": "", 2024-03-24T03:17:22.897 INFO:teuthology.orchestra.run.smithi118.stdout: "module_options": { 2024-03-24T03:17:22.897 INFO:teuthology.orchestra.run.smithi118.stdout: "log_level": { 2024-03-24T03:17:22.897 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "log_level", 2024-03-24T03:17:22.897 INFO:teuthology.orchestra.run.smithi118.stdout: "type": "str", 2024-03-24T03:17:22.897 INFO:teuthology.orchestra.run.smithi118.stdout: "level": "advanced", 2024-03-24T03:17:22.897 INFO:teuthology.orchestra.run.smithi118.stdout: "flags": 1, 2024-03-24T03:17:22.898 INFO:teuthology.orchestra.run.smithi118.stdout: "default_value": "", 2024-03-24T03:17:22.898 INFO:teuthology.orchestra.run.smithi118.stdout: "min": "", 2024-03-24T03:17:22.898 INFO:teuthology.orchestra.run.smithi118.stdout: "max": "", 2024-03-24T03:17:22.898 INFO:teuthology.orchestra.run.smithi118.stdout: "enum_allowed": [ 2024-03-24T03:17:22.898 INFO:teuthology.orchestra.run.smithi118.stdout: "", 2024-03-24T03:17:22.898 INFO:teuthology.orchestra.run.smithi118.stdout: "critical", 2024-03-24T03:17:22.898 INFO:teuthology.orchestra.run.smithi118.stdout: "debug", 2024-03-24T03:17:22.898 INFO:teuthology.orchestra.run.smithi118.stdout: "error", 2024-03-24T03:17:22.898 INFO:teuthology.orchestra.run.smithi118.stdout: "info", 2024-03-24T03:17:22.899 INFO:teuthology.orchestra.run.smithi118.stdout: "warning" 2024-03-24T03:17:22.899 INFO:teuthology.orchestra.run.smithi118.stdout: ], 2024-03-24T03:17:22.899 INFO:teuthology.orchestra.run.smithi118.stdout: "desc": "", 2024-03-24T03:17:22.899 INFO:teuthology.orchestra.run.smithi118.stdout: "long_desc": "", 2024-03-24T03:17:22.899 INFO:teuthology.orchestra.run.smithi118.stdout: "tags": [], 2024-03-24T03:17:22.899 INFO:teuthology.orchestra.run.smithi118.stdout: "see_also": [] 2024-03-24T03:17:22.899 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-24T03:17:22.899 INFO:teuthology.orchestra.run.smithi118.stdout: "log_to_cluster": { 2024-03-24T03:17:22.899 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "log_to_cluster", 2024-03-24T03:17:22.899 INFO:teuthology.orchestra.run.smithi118.stdout: "type": "bool", 2024-03-24T03:17:22.900 INFO:teuthology.orchestra.run.smithi118.stdout: "level": "advanced", 2024-03-24T03:17:22.900 INFO:teuthology.orchestra.run.smithi118.stdout: "flags": 1, 2024-03-24T03:17:22.900 INFO:teuthology.orchestra.run.smithi118.stdout: "default_value": "False", 2024-03-24T03:17:22.900 INFO:teuthology.orchestra.run.smithi118.stdout: "min": "", 2024-03-24T03:17:22.900 INFO:teuthology.orchestra.run.smithi118.stdout: "max": "", 2024-03-24T03:17:22.900 INFO:teuthology.orchestra.run.smithi118.stdout: "enum_allowed": [], 2024-03-24T03:17:22.900 INFO:teuthology.orchestra.run.smithi118.stdout: "desc": "", 2024-03-24T03:17:22.900 INFO:teuthology.orchestra.run.smithi118.stdout: "long_desc": "", 2024-03-24T03:17:22.900 INFO:teuthology.orchestra.run.smithi118.stdout: "tags": [], 2024-03-24T03:17:22.901 INFO:teuthology.orchestra.run.smithi118.stdout: "see_also": [] 2024-03-24T03:17:22.901 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-24T03:17:22.901 INFO:teuthology.orchestra.run.smithi118.stdout: "log_to_cluster_level": { 2024-03-24T03:17:22.901 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "log_to_cluster_level", 2024-03-24T03:17:22.901 INFO:teuthology.orchestra.run.smithi118.stdout: "type": "str", 2024-03-24T03:17:22.901 INFO:teuthology.orchestra.run.smithi118.stdout: "level": "advanced", 2024-03-24T03:17:22.901 INFO:teuthology.orchestra.run.smithi118.stdout: "flags": 1, 2024-03-24T03:17:22.901 INFO:teuthology.orchestra.run.smithi118.stdout: "default_value": "info", 2024-03-24T03:17:22.901 INFO:teuthology.orchestra.run.smithi118.stdout: "min": "", 2024-03-24T03:17:22.902 INFO:teuthology.orchestra.run.smithi118.stdout: "max": "", 2024-03-24T03:17:22.902 INFO:teuthology.orchestra.run.smithi118.stdout: "enum_allowed": [ 2024-03-24T03:17:22.902 INFO:teuthology.orchestra.run.smithi118.stdout: "", 2024-03-24T03:17:22.902 INFO:teuthology.orchestra.run.smithi118.stdout: "critical", 2024-03-24T03:17:22.902 INFO:teuthology.orchestra.run.smithi118.stdout: "debug", 2024-03-24T03:17:22.902 INFO:teuthology.orchestra.run.smithi118.stdout: "error", 2024-03-24T03:17:22.902 INFO:teuthology.orchestra.run.smithi118.stdout: "info", 2024-03-24T03:17:22.902 INFO:teuthology.orchestra.run.smithi118.stdout: "warning" 2024-03-24T03:17:22.902 INFO:teuthology.orchestra.run.smithi118.stdout: ], 2024-03-24T03:17:22.903 INFO:teuthology.orchestra.run.smithi118.stdout: "desc": "", 2024-03-24T03:17:22.903 INFO:teuthology.orchestra.run.smithi118.stdout: "long_desc": "", 2024-03-24T03:17:22.903 INFO:teuthology.orchestra.run.smithi118.stdout: "tags": [], 2024-03-24T03:17:22.903 INFO:teuthology.orchestra.run.smithi118.stdout: "see_also": [] 2024-03-24T03:17:22.903 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-24T03:17:22.903 INFO:teuthology.orchestra.run.smithi118.stdout: "log_to_file": { 2024-03-24T03:17:22.903 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "log_to_file", 2024-03-24T03:17:22.903 INFO:teuthology.orchestra.run.smithi118.stdout: "type": "bool", 2024-03-24T03:17:22.903 INFO:teuthology.orchestra.run.smithi118.stdout: "level": "advanced", 2024-03-24T03:17:22.904 INFO:teuthology.orchestra.run.smithi118.stdout: "flags": 1, 2024-03-24T03:17:22.904 INFO:teuthology.orchestra.run.smithi118.stdout: "default_value": "False", 2024-03-24T03:17:22.904 INFO:teuthology.orchestra.run.smithi118.stdout: "min": "", 2024-03-24T03:17:22.904 INFO:teuthology.orchestra.run.smithi118.stdout: "max": "", 2024-03-24T03:17:22.904 INFO:teuthology.orchestra.run.smithi118.stdout: "enum_allowed": [], 2024-03-24T03:17:22.904 INFO:teuthology.orchestra.run.smithi118.stdout: "desc": "", 2024-03-24T03:17:22.904 INFO:teuthology.orchestra.run.smithi118.stdout: "long_desc": "", 2024-03-24T03:17:22.904 INFO:teuthology.orchestra.run.smithi118.stdout: "tags": [], 2024-03-24T03:17:22.905 INFO:teuthology.orchestra.run.smithi118.stdout: "see_also": [] 2024-03-24T03:17:22.905 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-03-24T03:17:22.905 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-03-24T03:17:22.905 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-24T03:17:22.905 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-24T03:17:22.905 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "pg_autoscaler", 2024-03-24T03:17:22.905 INFO:teuthology.orchestra.run.smithi118.stdout: "can_run": true, 2024-03-24T03:17:22.905 INFO:teuthology.orchestra.run.smithi118.stdout: "error_string": "", 2024-03-24T03:17:22.905 INFO:teuthology.orchestra.run.smithi118.stdout: "module_options": { 2024-03-24T03:17:22.906 INFO:teuthology.orchestra.run.smithi118.stdout: "log_level": { 2024-03-24T03:17:22.906 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "log_level", 2024-03-24T03:17:22.906 INFO:teuthology.orchestra.run.smithi118.stdout: "type": "str", 2024-03-24T03:17:22.906 INFO:teuthology.orchestra.run.smithi118.stdout: "level": "advanced", 2024-03-24T03:17:22.906 INFO:teuthology.orchestra.run.smithi118.stdout: "flags": 1, 2024-03-24T03:17:22.906 INFO:teuthology.orchestra.run.smithi118.stdout: "default_value": "", 2024-03-24T03:17:22.906 INFO:teuthology.orchestra.run.smithi118.stdout: "min": "", 2024-03-24T03:17:22.906 INFO:teuthology.orchestra.run.smithi118.stdout: "max": "", 2024-03-24T03:17:22.907 INFO:teuthology.orchestra.run.smithi118.stdout: "enum_allowed": [ 2024-03-24T03:17:22.907 INFO:teuthology.orchestra.run.smithi118.stdout: "", 2024-03-24T03:17:22.907 INFO:teuthology.orchestra.run.smithi118.stdout: "critical", 2024-03-24T03:17:22.907 INFO:teuthology.orchestra.run.smithi118.stdout: "debug", 2024-03-24T03:17:22.907 INFO:teuthology.orchestra.run.smithi118.stdout: "error", 2024-03-24T03:17:22.907 INFO:teuthology.orchestra.run.smithi118.stdout: "info", 2024-03-24T03:17:22.907 INFO:teuthology.orchestra.run.smithi118.stdout: "warning" 2024-03-24T03:17:22.907 INFO:teuthology.orchestra.run.smithi118.stdout: ], 2024-03-24T03:17:22.907 INFO:teuthology.orchestra.run.smithi118.stdout: "desc": "", 2024-03-24T03:17:22.908 INFO:teuthology.orchestra.run.smithi118.stdout: "long_desc": "", 2024-03-24T03:17:22.908 INFO:teuthology.orchestra.run.smithi118.stdout: "tags": [], 2024-03-24T03:17:22.908 INFO:teuthology.orchestra.run.smithi118.stdout: "see_also": [] 2024-03-24T03:17:22.908 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-24T03:17:22.908 INFO:teuthology.orchestra.run.smithi118.stdout: "log_to_cluster": { 2024-03-24T03:17:22.908 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "log_to_cluster", 2024-03-24T03:17:22.908 INFO:teuthology.orchestra.run.smithi118.stdout: "type": "bool", 2024-03-24T03:17:22.908 INFO:teuthology.orchestra.run.smithi118.stdout: "level": "advanced", 2024-03-24T03:17:22.908 INFO:teuthology.orchestra.run.smithi118.stdout: "flags": 1, 2024-03-24T03:17:22.908 INFO:teuthology.orchestra.run.smithi118.stdout: "default_value": "False", 2024-03-24T03:17:22.909 INFO:teuthology.orchestra.run.smithi118.stdout: "min": "", 2024-03-24T03:17:22.909 INFO:teuthology.orchestra.run.smithi118.stdout: "max": "", 2024-03-24T03:17:22.909 INFO:teuthology.orchestra.run.smithi118.stdout: "enum_allowed": [], 2024-03-24T03:17:22.909 INFO:teuthology.orchestra.run.smithi118.stdout: "desc": "", 2024-03-24T03:17:22.909 INFO:teuthology.orchestra.run.smithi118.stdout: "long_desc": "", 2024-03-24T03:17:22.909 INFO:teuthology.orchestra.run.smithi118.stdout: "tags": [], 2024-03-24T03:17:22.909 INFO:teuthology.orchestra.run.smithi118.stdout: "see_also": [] 2024-03-24T03:17:22.909 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-24T03:17:22.909 INFO:teuthology.orchestra.run.smithi118.stdout: "log_to_cluster_level": { 2024-03-24T03:17:22.910 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "log_to_cluster_level", 2024-03-24T03:17:22.910 INFO:teuthology.orchestra.run.smithi118.stdout: "type": "str", 2024-03-24T03:17:22.910 INFO:teuthology.orchestra.run.smithi118.stdout: "level": "advanced", 2024-03-24T03:17:22.910 INFO:teuthology.orchestra.run.smithi118.stdout: "flags": 1, 2024-03-24T03:17:22.910 INFO:teuthology.orchestra.run.smithi118.stdout: "default_value": "info", 2024-03-24T03:17:22.910 INFO:teuthology.orchestra.run.smithi118.stdout: "min": "", 2024-03-24T03:17:22.910 INFO:teuthology.orchestra.run.smithi118.stdout: "max": "", 2024-03-24T03:17:22.910 INFO:teuthology.orchestra.run.smithi118.stdout: "enum_allowed": [ 2024-03-24T03:17:22.911 INFO:teuthology.orchestra.run.smithi118.stdout: "", 2024-03-24T03:17:22.911 INFO:teuthology.orchestra.run.smithi118.stdout: "critical", 2024-03-24T03:17:22.911 INFO:teuthology.orchestra.run.smithi118.stdout: "debug", 2024-03-24T03:17:22.911 INFO:teuthology.orchestra.run.smithi118.stdout: "error", 2024-03-24T03:17:22.911 INFO:teuthology.orchestra.run.smithi118.stdout: "info", 2024-03-24T03:17:22.911 INFO:teuthology.orchestra.run.smithi118.stdout: "warning" 2024-03-24T03:17:22.911 INFO:teuthology.orchestra.run.smithi118.stdout: ], 2024-03-24T03:17:22.911 INFO:teuthology.orchestra.run.smithi118.stdout: "desc": "", 2024-03-24T03:17:22.911 INFO:teuthology.orchestra.run.smithi118.stdout: "long_desc": "", 2024-03-24T03:17:22.912 INFO:teuthology.orchestra.run.smithi118.stdout: "tags": [], 2024-03-24T03:17:22.912 INFO:teuthology.orchestra.run.smithi118.stdout: "see_also": [] 2024-03-24T03:17:22.912 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-24T03:17:22.912 INFO:teuthology.orchestra.run.smithi118.stdout: "log_to_file": { 2024-03-24T03:17:22.912 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "log_to_file", 2024-03-24T03:17:22.912 INFO:teuthology.orchestra.run.smithi118.stdout: "type": "bool", 2024-03-24T03:17:22.912 INFO:teuthology.orchestra.run.smithi118.stdout: "level": "advanced", 2024-03-24T03:17:22.912 INFO:teuthology.orchestra.run.smithi118.stdout: "flags": 1, 2024-03-24T03:17:22.913 INFO:teuthology.orchestra.run.smithi118.stdout: "default_value": "False", 2024-03-24T03:17:22.913 INFO:teuthology.orchestra.run.smithi118.stdout: "min": "", 2024-03-24T03:17:22.913 INFO:teuthology.orchestra.run.smithi118.stdout: "max": "", 2024-03-24T03:17:22.913 INFO:teuthology.orchestra.run.smithi118.stdout: "enum_allowed": [], 2024-03-24T03:17:22.913 INFO:teuthology.orchestra.run.smithi118.stdout: "desc": "", 2024-03-24T03:17:22.913 INFO:teuthology.orchestra.run.smithi118.stdout: "long_desc": "", 2024-03-24T03:17:22.913 INFO:teuthology.orchestra.run.smithi118.stdout: "tags": [], 2024-03-24T03:17:22.913 INFO:teuthology.orchestra.run.smithi118.stdout: "see_also": [] 2024-03-24T03:17:22.913 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-24T03:17:22.914 INFO:teuthology.orchestra.run.smithi118.stdout: "sleep_interval": { 2024-03-24T03:17:22.914 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "sleep_interval", 2024-03-24T03:17:22.914 INFO:teuthology.orchestra.run.smithi118.stdout: "type": "secs", 2024-03-24T03:17:22.914 INFO:teuthology.orchestra.run.smithi118.stdout: "level": "advanced", 2024-03-24T03:17:22.914 INFO:teuthology.orchestra.run.smithi118.stdout: "flags": 0, 2024-03-24T03:17:22.914 INFO:teuthology.orchestra.run.smithi118.stdout: "default_value": "60", 2024-03-24T03:17:22.914 INFO:teuthology.orchestra.run.smithi118.stdout: "min": "", 2024-03-24T03:17:22.914 INFO:teuthology.orchestra.run.smithi118.stdout: "max": "", 2024-03-24T03:17:22.915 INFO:teuthology.orchestra.run.smithi118.stdout: "enum_allowed": [], 2024-03-24T03:17:22.915 INFO:teuthology.orchestra.run.smithi118.stdout: "desc": "", 2024-03-24T03:17:22.915 INFO:teuthology.orchestra.run.smithi118.stdout: "long_desc": "", 2024-03-24T03:17:22.915 INFO:teuthology.orchestra.run.smithi118.stdout: "tags": [], 2024-03-24T03:17:22.915 INFO:teuthology.orchestra.run.smithi118.stdout: "see_also": [] 2024-03-24T03:17:22.915 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-24T03:17:22.915 INFO:teuthology.orchestra.run.smithi118.stdout: "threshold": { 2024-03-24T03:17:22.915 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "threshold", 2024-03-24T03:17:22.915 INFO:teuthology.orchestra.run.smithi118.stdout: "type": "float", 2024-03-24T03:17:22.916 INFO:teuthology.orchestra.run.smithi118.stdout: "level": "advanced", 2024-03-24T03:17:22.916 INFO:teuthology.orchestra.run.smithi118.stdout: "flags": 0, 2024-03-24T03:17:22.916 INFO:teuthology.orchestra.run.smithi118.stdout: "default_value": "3.0", 2024-03-24T03:17:22.916 INFO:teuthology.orchestra.run.smithi118.stdout: "min": "1.0", 2024-03-24T03:17:22.916 INFO:teuthology.orchestra.run.smithi118.stdout: "max": "", 2024-03-24T03:17:22.916 INFO:teuthology.orchestra.run.smithi118.stdout: "enum_allowed": [], 2024-03-24T03:17:22.916 INFO:teuthology.orchestra.run.smithi118.stdout: "desc": "scaling threshold", 2024-03-24T03:17:22.916 INFO:teuthology.orchestra.run.smithi118.stdout: "long_desc": "The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0", 2024-03-24T03:17:22.916 INFO:teuthology.orchestra.run.smithi118.stdout: "tags": [], 2024-03-24T03:17:22.916 INFO:teuthology.orchestra.run.smithi118.stdout: "see_also": [] 2024-03-24T03:17:22.917 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-03-24T03:17:22.917 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-03-24T03:17:22.917 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-24T03:17:22.917 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-24T03:17:22.917 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "progress", 2024-03-24T03:17:22.917 INFO:teuthology.orchestra.run.smithi118.stdout: "can_run": true, 2024-03-24T03:17:22.917 INFO:teuthology.orchestra.run.smithi118.stdout: "error_string": "", 2024-03-24T03:17:22.917 INFO:teuthology.orchestra.run.smithi118.stdout: "module_options": { 2024-03-24T03:17:22.917 INFO:teuthology.orchestra.run.smithi118.stdout: "allow_pg_recovery_event": { 2024-03-24T03:17:22.918 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "allow_pg_recovery_event", 2024-03-24T03:17:22.918 INFO:teuthology.orchestra.run.smithi118.stdout: "type": "bool", 2024-03-24T03:17:22.918 INFO:teuthology.orchestra.run.smithi118.stdout: "level": "advanced", 2024-03-24T03:17:22.918 INFO:teuthology.orchestra.run.smithi118.stdout: "flags": 1, 2024-03-24T03:17:22.918 INFO:teuthology.orchestra.run.smithi118.stdout: "default_value": "False", 2024-03-24T03:17:22.918 INFO:teuthology.orchestra.run.smithi118.stdout: "min": "", 2024-03-24T03:17:22.918 INFO:teuthology.orchestra.run.smithi118.stdout: "max": "", 2024-03-24T03:17:22.918 INFO:teuthology.orchestra.run.smithi118.stdout: "enum_allowed": [], 2024-03-24T03:17:22.919 INFO:teuthology.orchestra.run.smithi118.stdout: "desc": "allow the module to show pg recovery progress", 2024-03-24T03:17:22.919 INFO:teuthology.orchestra.run.smithi118.stdout: "long_desc": "", 2024-03-24T03:17:22.919 INFO:teuthology.orchestra.run.smithi118.stdout: "tags": [], 2024-03-24T03:17:22.919 INFO:teuthology.orchestra.run.smithi118.stdout: "see_also": [] 2024-03-24T03:17:22.919 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-24T03:17:22.919 INFO:teuthology.orchestra.run.smithi118.stdout: "enabled": { 2024-03-24T03:17:22.919 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "enabled", 2024-03-24T03:17:22.919 INFO:teuthology.orchestra.run.smithi118.stdout: "type": "bool", 2024-03-24T03:17:22.919 INFO:teuthology.orchestra.run.smithi118.stdout: "level": "advanced", 2024-03-24T03:17:22.920 INFO:teuthology.orchestra.run.smithi118.stdout: "flags": 0, 2024-03-24T03:17:22.920 INFO:teuthology.orchestra.run.smithi118.stdout: "default_value": "True", 2024-03-24T03:17:22.920 INFO:teuthology.orchestra.run.smithi118.stdout: "min": "", 2024-03-24T03:17:22.920 INFO:teuthology.orchestra.run.smithi118.stdout: "max": "", 2024-03-24T03:17:22.920 INFO:teuthology.orchestra.run.smithi118.stdout: "enum_allowed": [], 2024-03-24T03:17:22.920 INFO:teuthology.orchestra.run.smithi118.stdout: "desc": "", 2024-03-24T03:17:22.920 INFO:teuthology.orchestra.run.smithi118.stdout: "long_desc": "", 2024-03-24T03:17:22.920 INFO:teuthology.orchestra.run.smithi118.stdout: "tags": [], 2024-03-24T03:17:22.920 INFO:teuthology.orchestra.run.smithi118.stdout: "see_also": [] 2024-03-24T03:17:22.921 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-24T03:17:22.921 INFO:teuthology.orchestra.run.smithi118.stdout: "log_level": { 2024-03-24T03:17:22.921 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "log_level", 2024-03-24T03:17:22.921 INFO:teuthology.orchestra.run.smithi118.stdout: "type": "str", 2024-03-24T03:17:22.921 INFO:teuthology.orchestra.run.smithi118.stdout: "level": "advanced", 2024-03-24T03:17:22.921 INFO:teuthology.orchestra.run.smithi118.stdout: "flags": 1, 2024-03-24T03:17:22.921 INFO:teuthology.orchestra.run.smithi118.stdout: "default_value": "", 2024-03-24T03:17:22.921 INFO:teuthology.orchestra.run.smithi118.stdout: "min": "", 2024-03-24T03:17:22.921 INFO:teuthology.orchestra.run.smithi118.stdout: "max": "", 2024-03-24T03:17:22.922 INFO:teuthology.orchestra.run.smithi118.stdout: "enum_allowed": [ 2024-03-24T03:17:22.922 INFO:teuthology.orchestra.run.smithi118.stdout: "", 2024-03-24T03:17:22.922 INFO:teuthology.orchestra.run.smithi118.stdout: "critical", 2024-03-24T03:17:22.922 INFO:teuthology.orchestra.run.smithi118.stdout: "debug", 2024-03-24T03:17:22.922 INFO:teuthology.orchestra.run.smithi118.stdout: "error", 2024-03-24T03:17:22.922 INFO:teuthology.orchestra.run.smithi118.stdout: "info", 2024-03-24T03:17:22.922 INFO:teuthology.orchestra.run.smithi118.stdout: "warning" 2024-03-24T03:17:22.922 INFO:teuthology.orchestra.run.smithi118.stdout: ], 2024-03-24T03:17:22.923 INFO:teuthology.orchestra.run.smithi118.stdout: "desc": "", 2024-03-24T03:17:22.923 INFO:teuthology.orchestra.run.smithi118.stdout: "long_desc": "", 2024-03-24T03:17:22.923 INFO:teuthology.orchestra.run.smithi118.stdout: "tags": [], 2024-03-24T03:17:22.923 INFO:teuthology.orchestra.run.smithi118.stdout: "see_also": [] 2024-03-24T03:17:22.923 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-24T03:17:22.923 INFO:teuthology.orchestra.run.smithi118.stdout: "log_to_cluster": { 2024-03-24T03:17:22.923 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "log_to_cluster", 2024-03-24T03:17:22.923 INFO:teuthology.orchestra.run.smithi118.stdout: "type": "bool", 2024-03-24T03:17:22.924 INFO:teuthology.orchestra.run.smithi118.stdout: "level": "advanced", 2024-03-24T03:17:22.924 INFO:teuthology.orchestra.run.smithi118.stdout: "flags": 1, 2024-03-24T03:17:22.924 INFO:teuthology.orchestra.run.smithi118.stdout: "default_value": "False", 2024-03-24T03:17:22.924 INFO:teuthology.orchestra.run.smithi118.stdout: "min": "", 2024-03-24T03:17:22.924 INFO:teuthology.orchestra.run.smithi118.stdout: "max": "", 2024-03-24T03:17:22.924 INFO:teuthology.orchestra.run.smithi118.stdout: "enum_allowed": [], 2024-03-24T03:17:22.924 INFO:teuthology.orchestra.run.smithi118.stdout: "desc": "", 2024-03-24T03:17:22.924 INFO:teuthology.orchestra.run.smithi118.stdout: "long_desc": "", 2024-03-24T03:17:22.924 INFO:teuthology.orchestra.run.smithi118.stdout: "tags": [], 2024-03-24T03:17:22.925 INFO:teuthology.orchestra.run.smithi118.stdout: "see_also": [] 2024-03-24T03:17:22.925 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-24T03:17:22.925 INFO:teuthology.orchestra.run.smithi118.stdout: "log_to_cluster_level": { 2024-03-24T03:17:22.925 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "log_to_cluster_level", 2024-03-24T03:17:22.925 INFO:teuthology.orchestra.run.smithi118.stdout: "type": "str", 2024-03-24T03:17:22.925 INFO:teuthology.orchestra.run.smithi118.stdout: "level": "advanced", 2024-03-24T03:17:22.925 INFO:teuthology.orchestra.run.smithi118.stdout: "flags": 1, 2024-03-24T03:17:22.925 INFO:teuthology.orchestra.run.smithi118.stdout: "default_value": "info", 2024-03-24T03:17:22.925 INFO:teuthology.orchestra.run.smithi118.stdout: "min": "", 2024-03-24T03:17:22.926 INFO:teuthology.orchestra.run.smithi118.stdout: "max": "", 2024-03-24T03:17:22.926 INFO:teuthology.orchestra.run.smithi118.stdout: "enum_allowed": [ 2024-03-24T03:17:22.926 INFO:teuthology.orchestra.run.smithi118.stdout: "", 2024-03-24T03:17:22.926 INFO:teuthology.orchestra.run.smithi118.stdout: "critical", 2024-03-24T03:17:22.926 INFO:teuthology.orchestra.run.smithi118.stdout: "debug", 2024-03-24T03:17:22.926 INFO:teuthology.orchestra.run.smithi118.stdout: "error", 2024-03-24T03:17:22.926 INFO:teuthology.orchestra.run.smithi118.stdout: "info", 2024-03-24T03:17:22.926 INFO:teuthology.orchestra.run.smithi118.stdout: "warning" 2024-03-24T03:17:22.926 INFO:teuthology.orchestra.run.smithi118.stdout: ], 2024-03-24T03:17:22.927 INFO:teuthology.orchestra.run.smithi118.stdout: "desc": "", 2024-03-24T03:17:22.927 INFO:teuthology.orchestra.run.smithi118.stdout: "long_desc": "", 2024-03-24T03:17:22.927 INFO:teuthology.orchestra.run.smithi118.stdout: "tags": [], 2024-03-24T03:17:22.927 INFO:teuthology.orchestra.run.smithi118.stdout: "see_also": [] 2024-03-24T03:17:22.927 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-24T03:17:22.927 INFO:teuthology.orchestra.run.smithi118.stdout: "log_to_file": { 2024-03-24T03:17:22.927 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "log_to_file", 2024-03-24T03:17:22.927 INFO:teuthology.orchestra.run.smithi118.stdout: "type": "bool", 2024-03-24T03:17:22.927 INFO:teuthology.orchestra.run.smithi118.stdout: "level": "advanced", 2024-03-24T03:17:22.927 INFO:teuthology.orchestra.run.smithi118.stdout: "flags": 1, 2024-03-24T03:17:22.928 INFO:teuthology.orchestra.run.smithi118.stdout: "default_value": "False", 2024-03-24T03:17:22.928 INFO:teuthology.orchestra.run.smithi118.stdout: "min": "", 2024-03-24T03:17:22.928 INFO:teuthology.orchestra.run.smithi118.stdout: "max": "", 2024-03-24T03:17:22.928 INFO:teuthology.orchestra.run.smithi118.stdout: "enum_allowed": [], 2024-03-24T03:17:22.928 INFO:teuthology.orchestra.run.smithi118.stdout: "desc": "", 2024-03-24T03:17:22.928 INFO:teuthology.orchestra.run.smithi118.stdout: "long_desc": "", 2024-03-24T03:17:22.928 INFO:teuthology.orchestra.run.smithi118.stdout: "tags": [], 2024-03-24T03:17:22.928 INFO:teuthology.orchestra.run.smithi118.stdout: "see_also": [] 2024-03-24T03:17:22.928 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-24T03:17:22.929 INFO:teuthology.orchestra.run.smithi118.stdout: "max_completed_events": { 2024-03-24T03:17:22.929 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "max_completed_events", 2024-03-24T03:17:22.929 INFO:teuthology.orchestra.run.smithi118.stdout: "type": "int", 2024-03-24T03:17:22.929 INFO:teuthology.orchestra.run.smithi118.stdout: "level": "advanced", 2024-03-24T03:17:22.929 INFO:teuthology.orchestra.run.smithi118.stdout: "flags": 1, 2024-03-24T03:17:22.929 INFO:teuthology.orchestra.run.smithi118.stdout: "default_value": "50", 2024-03-24T03:17:22.929 INFO:teuthology.orchestra.run.smithi118.stdout: "min": "", 2024-03-24T03:17:22.929 INFO:teuthology.orchestra.run.smithi118.stdout: "max": "", 2024-03-24T03:17:22.929 INFO:teuthology.orchestra.run.smithi118.stdout: "enum_allowed": [], 2024-03-24T03:17:22.930 INFO:teuthology.orchestra.run.smithi118.stdout: "desc": "number of past completed events to remember", 2024-03-24T03:17:22.930 INFO:teuthology.orchestra.run.smithi118.stdout: "long_desc": "", 2024-03-24T03:17:22.930 INFO:teuthology.orchestra.run.smithi118.stdout: "tags": [], 2024-03-24T03:17:22.930 INFO:teuthology.orchestra.run.smithi118.stdout: "see_also": [] 2024-03-24T03:17:22.930 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-24T03:17:22.930 INFO:teuthology.orchestra.run.smithi118.stdout: "sleep_interval": { 2024-03-24T03:17:22.930 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "sleep_interval", 2024-03-24T03:17:22.930 INFO:teuthology.orchestra.run.smithi118.stdout: "type": "secs", 2024-03-24T03:17:22.930 INFO:teuthology.orchestra.run.smithi118.stdout: "level": "advanced", 2024-03-24T03:17:22.931 INFO:teuthology.orchestra.run.smithi118.stdout: "flags": 1, 2024-03-24T03:17:22.931 INFO:teuthology.orchestra.run.smithi118.stdout: "default_value": "5", 2024-03-24T03:17:22.931 INFO:teuthology.orchestra.run.smithi118.stdout: "min": "", 2024-03-24T03:17:22.931 INFO:teuthology.orchestra.run.smithi118.stdout: "max": "", 2024-03-24T03:17:22.931 INFO:teuthology.orchestra.run.smithi118.stdout: "enum_allowed": [], 2024-03-24T03:17:22.931 INFO:teuthology.orchestra.run.smithi118.stdout: "desc": "how long the module is going to sleep", 2024-03-24T03:17:22.931 INFO:teuthology.orchestra.run.smithi118.stdout: "long_desc": "", 2024-03-24T03:17:22.931 INFO:teuthology.orchestra.run.smithi118.stdout: "tags": [], 2024-03-24T03:17:22.931 INFO:teuthology.orchestra.run.smithi118.stdout: "see_also": [] 2024-03-24T03:17:22.932 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-03-24T03:17:22.932 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-03-24T03:17:22.932 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-24T03:17:22.932 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-24T03:17:22.932 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "prometheus", 2024-03-24T03:17:22.932 INFO:teuthology.orchestra.run.smithi118.stdout: "can_run": true, 2024-03-24T03:17:22.932 INFO:teuthology.orchestra.run.smithi118.stdout: "error_string": "", 2024-03-24T03:17:22.932 INFO:teuthology.orchestra.run.smithi118.stdout: "module_options": { 2024-03-24T03:17:22.932 INFO:teuthology.orchestra.run.smithi118.stdout: "cache": { 2024-03-24T03:17:22.933 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "cache", 2024-03-24T03:17:22.933 INFO:teuthology.orchestra.run.smithi118.stdout: "type": "bool", 2024-03-24T03:17:22.933 INFO:teuthology.orchestra.run.smithi118.stdout: "level": "advanced", 2024-03-24T03:17:22.933 INFO:teuthology.orchestra.run.smithi118.stdout: "flags": 0, 2024-03-24T03:17:22.933 INFO:teuthology.orchestra.run.smithi118.stdout: "default_value": "True", 2024-03-24T03:17:22.933 INFO:teuthology.orchestra.run.smithi118.stdout: "min": "", 2024-03-24T03:17:22.933 INFO:teuthology.orchestra.run.smithi118.stdout: "max": "", 2024-03-24T03:17:22.933 INFO:teuthology.orchestra.run.smithi118.stdout: "enum_allowed": [], 2024-03-24T03:17:22.933 INFO:teuthology.orchestra.run.smithi118.stdout: "desc": "", 2024-03-24T03:17:22.934 INFO:teuthology.orchestra.run.smithi118.stdout: "long_desc": "", 2024-03-24T03:17:22.934 INFO:teuthology.orchestra.run.smithi118.stdout: "tags": [], 2024-03-24T03:17:22.934 INFO:teuthology.orchestra.run.smithi118.stdout: "see_also": [] 2024-03-24T03:17:22.934 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-24T03:17:22.934 INFO:teuthology.orchestra.run.smithi118.stdout: "exclude_perf_counters": { 2024-03-24T03:17:22.934 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "exclude_perf_counters", 2024-03-24T03:17:22.934 INFO:teuthology.orchestra.run.smithi118.stdout: "type": "bool", 2024-03-24T03:17:22.934 INFO:teuthology.orchestra.run.smithi118.stdout: "level": "advanced", 2024-03-24T03:17:22.935 INFO:teuthology.orchestra.run.smithi118.stdout: "flags": 1, 2024-03-24T03:17:22.935 INFO:teuthology.orchestra.run.smithi118.stdout: "default_value": "True", 2024-03-24T03:17:22.935 INFO:teuthology.orchestra.run.smithi118.stdout: "min": "", 2024-03-24T03:17:22.935 INFO:teuthology.orchestra.run.smithi118.stdout: "max": "", 2024-03-24T03:17:22.935 INFO:teuthology.orchestra.run.smithi118.stdout: "enum_allowed": [], 2024-03-24T03:17:22.935 INFO:teuthology.orchestra.run.smithi118.stdout: "desc": "Do not include perf-counters in the metrics output", 2024-03-24T03:17:22.935 INFO:teuthology.orchestra.run.smithi118.stdout: "long_desc": "Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.", 2024-03-24T03:17:22.935 INFO:teuthology.orchestra.run.smithi118.stdout: "tags": [], 2024-03-24T03:17:22.935 INFO:teuthology.orchestra.run.smithi118.stdout: "see_also": [] 2024-03-24T03:17:22.935 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-24T03:17:22.936 INFO:teuthology.orchestra.run.smithi118.stdout: "log_level": { 2024-03-24T03:17:22.936 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "log_level", 2024-03-24T03:17:22.936 INFO:teuthology.orchestra.run.smithi118.stdout: "type": "str", 2024-03-24T03:17:22.936 INFO:teuthology.orchestra.run.smithi118.stdout: "level": "advanced", 2024-03-24T03:17:22.936 INFO:teuthology.orchestra.run.smithi118.stdout: "flags": 1, 2024-03-24T03:17:22.936 INFO:teuthology.orchestra.run.smithi118.stdout: "default_value": "", 2024-03-24T03:17:22.936 INFO:teuthology.orchestra.run.smithi118.stdout: "min": "", 2024-03-24T03:17:22.936 INFO:teuthology.orchestra.run.smithi118.stdout: "max": "", 2024-03-24T03:17:22.937 INFO:teuthology.orchestra.run.smithi118.stdout: "enum_allowed": [ 2024-03-24T03:17:22.937 INFO:teuthology.orchestra.run.smithi118.stdout: "", 2024-03-24T03:17:22.937 INFO:teuthology.orchestra.run.smithi118.stdout: "critical", 2024-03-24T03:17:22.937 INFO:teuthology.orchestra.run.smithi118.stdout: "debug", 2024-03-24T03:17:22.937 INFO:teuthology.orchestra.run.smithi118.stdout: "error", 2024-03-24T03:17:22.937 INFO:teuthology.orchestra.run.smithi118.stdout: "info", 2024-03-24T03:17:22.937 INFO:teuthology.orchestra.run.smithi118.stdout: "warning" 2024-03-24T03:17:22.937 INFO:teuthology.orchestra.run.smithi118.stdout: ], 2024-03-24T03:17:22.937 INFO:teuthology.orchestra.run.smithi118.stdout: "desc": "", 2024-03-24T03:17:22.937 INFO:teuthology.orchestra.run.smithi118.stdout: "long_desc": "", 2024-03-24T03:17:22.938 INFO:teuthology.orchestra.run.smithi118.stdout: "tags": [], 2024-03-24T03:17:22.938 INFO:teuthology.orchestra.run.smithi118.stdout: "see_also": [] 2024-03-24T03:17:22.938 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-24T03:17:22.938 INFO:teuthology.orchestra.run.smithi118.stdout: "log_to_cluster": { 2024-03-24T03:17:22.938 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "log_to_cluster", 2024-03-24T03:17:22.938 INFO:teuthology.orchestra.run.smithi118.stdout: "type": "bool", 2024-03-24T03:17:22.938 INFO:teuthology.orchestra.run.smithi118.stdout: "level": "advanced", 2024-03-24T03:17:22.938 INFO:teuthology.orchestra.run.smithi118.stdout: "flags": 1, 2024-03-24T03:17:22.939 INFO:teuthology.orchestra.run.smithi118.stdout: "default_value": "False", 2024-03-24T03:17:22.939 INFO:teuthology.orchestra.run.smithi118.stdout: "min": "", 2024-03-24T03:17:22.939 INFO:teuthology.orchestra.run.smithi118.stdout: "max": "", 2024-03-24T03:17:22.939 INFO:teuthology.orchestra.run.smithi118.stdout: "enum_allowed": [], 2024-03-24T03:17:22.939 INFO:teuthology.orchestra.run.smithi118.stdout: "desc": "", 2024-03-24T03:17:22.939 INFO:teuthology.orchestra.run.smithi118.stdout: "long_desc": "", 2024-03-24T03:17:22.939 INFO:teuthology.orchestra.run.smithi118.stdout: "tags": [], 2024-03-24T03:17:22.939 INFO:teuthology.orchestra.run.smithi118.stdout: "see_also": [] 2024-03-24T03:17:22.939 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-24T03:17:22.940 INFO:teuthology.orchestra.run.smithi118.stdout: "log_to_cluster_level": { 2024-03-24T03:17:22.940 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "log_to_cluster_level", 2024-03-24T03:17:22.940 INFO:teuthology.orchestra.run.smithi118.stdout: "type": "str", 2024-03-24T03:17:22.940 INFO:teuthology.orchestra.run.smithi118.stdout: "level": "advanced", 2024-03-24T03:17:22.940 INFO:teuthology.orchestra.run.smithi118.stdout: "flags": 1, 2024-03-24T03:17:22.940 INFO:teuthology.orchestra.run.smithi118.stdout: "default_value": "info", 2024-03-24T03:17:22.940 INFO:teuthology.orchestra.run.smithi118.stdout: "min": "", 2024-03-24T03:17:22.940 INFO:teuthology.orchestra.run.smithi118.stdout: "max": "", 2024-03-24T03:17:22.940 INFO:teuthology.orchestra.run.smithi118.stdout: "enum_allowed": [ 2024-03-24T03:17:22.941 INFO:teuthology.orchestra.run.smithi118.stdout: "", 2024-03-24T03:17:22.941 INFO:teuthology.orchestra.run.smithi118.stdout: "critical", 2024-03-24T03:17:22.941 INFO:teuthology.orchestra.run.smithi118.stdout: "debug", 2024-03-24T03:17:22.941 INFO:teuthology.orchestra.run.smithi118.stdout: "error", 2024-03-24T03:17:22.941 INFO:teuthology.orchestra.run.smithi118.stdout: "info", 2024-03-24T03:17:22.941 INFO:teuthology.orchestra.run.smithi118.stdout: "warning" 2024-03-24T03:17:22.941 INFO:teuthology.orchestra.run.smithi118.stdout: ], 2024-03-24T03:17:22.941 INFO:teuthology.orchestra.run.smithi118.stdout: "desc": "", 2024-03-24T03:17:22.941 INFO:teuthology.orchestra.run.smithi118.stdout: "long_desc": "", 2024-03-24T03:17:22.942 INFO:teuthology.orchestra.run.smithi118.stdout: "tags": [], 2024-03-24T03:17:22.942 INFO:teuthology.orchestra.run.smithi118.stdout: "see_also": [] 2024-03-24T03:17:22.942 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-24T03:17:22.942 INFO:teuthology.orchestra.run.smithi118.stdout: "log_to_file": { 2024-03-24T03:17:22.942 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "log_to_file", 2024-03-24T03:17:22.942 INFO:teuthology.orchestra.run.smithi118.stdout: "type": "bool", 2024-03-24T03:17:22.942 INFO:teuthology.orchestra.run.smithi118.stdout: "level": "advanced", 2024-03-24T03:17:22.942 INFO:teuthology.orchestra.run.smithi118.stdout: "flags": 1, 2024-03-24T03:17:22.942 INFO:teuthology.orchestra.run.smithi118.stdout: "default_value": "False", 2024-03-24T03:17:22.943 INFO:teuthology.orchestra.run.smithi118.stdout: "min": "", 2024-03-24T03:17:22.943 INFO:teuthology.orchestra.run.smithi118.stdout: "max": "", 2024-03-24T03:17:22.943 INFO:teuthology.orchestra.run.smithi118.stdout: "enum_allowed": [], 2024-03-24T03:17:22.943 INFO:teuthology.orchestra.run.smithi118.stdout: "desc": "", 2024-03-24T03:17:22.943 INFO:teuthology.orchestra.run.smithi118.stdout: "long_desc": "", 2024-03-24T03:17:22.943 INFO:teuthology.orchestra.run.smithi118.stdout: "tags": [], 2024-03-24T03:17:22.943 INFO:teuthology.orchestra.run.smithi118.stdout: "see_also": [] 2024-03-24T03:17:22.943 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-24T03:17:22.944 INFO:teuthology.orchestra.run.smithi118.stdout: "rbd_stats_pools": { 2024-03-24T03:17:22.944 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "rbd_stats_pools", 2024-03-24T03:17:22.944 INFO:teuthology.orchestra.run.smithi118.stdout: "type": "str", 2024-03-24T03:17:22.944 INFO:teuthology.orchestra.run.smithi118.stdout: "level": "advanced", 2024-03-24T03:17:22.944 INFO:teuthology.orchestra.run.smithi118.stdout: "flags": 0, 2024-03-24T03:17:22.944 INFO:teuthology.orchestra.run.smithi118.stdout: "default_value": "", 2024-03-24T03:17:22.944 INFO:teuthology.orchestra.run.smithi118.stdout: "min": "", 2024-03-24T03:17:22.944 INFO:teuthology.orchestra.run.smithi118.stdout: "max": "", 2024-03-24T03:17:22.944 INFO:teuthology.orchestra.run.smithi118.stdout: "enum_allowed": [], 2024-03-24T03:17:22.945 INFO:teuthology.orchestra.run.smithi118.stdout: "desc": "", 2024-03-24T03:17:22.945 INFO:teuthology.orchestra.run.smithi118.stdout: "long_desc": "", 2024-03-24T03:17:22.945 INFO:teuthology.orchestra.run.smithi118.stdout: "tags": [], 2024-03-24T03:17:22.945 INFO:teuthology.orchestra.run.smithi118.stdout: "see_also": [] 2024-03-24T03:17:22.945 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-24T03:17:22.945 INFO:teuthology.orchestra.run.smithi118.stdout: "rbd_stats_pools_refresh_interval": { 2024-03-24T03:17:22.945 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "rbd_stats_pools_refresh_interval", 2024-03-24T03:17:22.945 INFO:teuthology.orchestra.run.smithi118.stdout: "type": "int", 2024-03-24T03:17:22.945 INFO:teuthology.orchestra.run.smithi118.stdout: "level": "advanced", 2024-03-24T03:17:22.946 INFO:teuthology.orchestra.run.smithi118.stdout: "flags": 0, 2024-03-24T03:17:22.946 INFO:teuthology.orchestra.run.smithi118.stdout: "default_value": "300", 2024-03-24T03:17:22.946 INFO:teuthology.orchestra.run.smithi118.stdout: "min": "", 2024-03-24T03:17:22.946 INFO:teuthology.orchestra.run.smithi118.stdout: "max": "", 2024-03-24T03:17:22.946 INFO:teuthology.orchestra.run.smithi118.stdout: "enum_allowed": [], 2024-03-24T03:17:22.946 INFO:teuthology.orchestra.run.smithi118.stdout: "desc": "", 2024-03-24T03:17:22.946 INFO:teuthology.orchestra.run.smithi118.stdout: "long_desc": "", 2024-03-24T03:17:22.946 INFO:teuthology.orchestra.run.smithi118.stdout: "tags": [], 2024-03-24T03:17:22.946 INFO:teuthology.orchestra.run.smithi118.stdout: "see_also": [] 2024-03-24T03:17:22.947 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-24T03:17:22.947 INFO:teuthology.orchestra.run.smithi118.stdout: "scrape_interval": { 2024-03-24T03:17:22.947 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "scrape_interval", 2024-03-24T03:17:22.947 INFO:teuthology.orchestra.run.smithi118.stdout: "type": "float", 2024-03-24T03:17:22.947 INFO:teuthology.orchestra.run.smithi118.stdout: "level": "advanced", 2024-03-24T03:17:22.947 INFO:teuthology.orchestra.run.smithi118.stdout: "flags": 0, 2024-03-24T03:17:22.947 INFO:teuthology.orchestra.run.smithi118.stdout: "default_value": "15.0", 2024-03-24T03:17:22.947 INFO:teuthology.orchestra.run.smithi118.stdout: "min": "", 2024-03-24T03:17:22.947 INFO:teuthology.orchestra.run.smithi118.stdout: "max": "", 2024-03-24T03:17:22.948 INFO:teuthology.orchestra.run.smithi118.stdout: "enum_allowed": [], 2024-03-24T03:17:22.948 INFO:teuthology.orchestra.run.smithi118.stdout: "desc": "", 2024-03-24T03:17:22.948 INFO:teuthology.orchestra.run.smithi118.stdout: "long_desc": "", 2024-03-24T03:17:22.948 INFO:teuthology.orchestra.run.smithi118.stdout: "tags": [], 2024-03-24T03:17:22.948 INFO:teuthology.orchestra.run.smithi118.stdout: "see_also": [] 2024-03-24T03:17:22.948 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-24T03:17:22.948 INFO:teuthology.orchestra.run.smithi118.stdout: "server_addr": { 2024-03-24T03:17:22.948 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "server_addr", 2024-03-24T03:17:22.948 INFO:teuthology.orchestra.run.smithi118.stdout: "type": "str", 2024-03-24T03:17:22.948 INFO:teuthology.orchestra.run.smithi118.stdout: "level": "advanced", 2024-03-24T03:17:22.949 INFO:teuthology.orchestra.run.smithi118.stdout: "flags": 0, 2024-03-24T03:17:22.949 INFO:teuthology.orchestra.run.smithi118.stdout: "default_value": "::", 2024-03-24T03:17:22.949 INFO:teuthology.orchestra.run.smithi118.stdout: "min": "", 2024-03-24T03:17:22.949 INFO:teuthology.orchestra.run.smithi118.stdout: "max": "", 2024-03-24T03:17:22.949 INFO:teuthology.orchestra.run.smithi118.stdout: "enum_allowed": [], 2024-03-24T03:17:22.949 INFO:teuthology.orchestra.run.smithi118.stdout: "desc": "the IPv4 or IPv6 address on which the module listens for HTTP requests", 2024-03-24T03:17:22.949 INFO:teuthology.orchestra.run.smithi118.stdout: "long_desc": "", 2024-03-24T03:17:22.949 INFO:teuthology.orchestra.run.smithi118.stdout: "tags": [], 2024-03-24T03:17:22.949 INFO:teuthology.orchestra.run.smithi118.stdout: "see_also": [] 2024-03-24T03:17:22.950 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-24T03:17:22.950 INFO:teuthology.orchestra.run.smithi118.stdout: "server_port": { 2024-03-24T03:17:22.950 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "server_port", 2024-03-24T03:17:22.950 INFO:teuthology.orchestra.run.smithi118.stdout: "type": "int", 2024-03-24T03:17:22.950 INFO:teuthology.orchestra.run.smithi118.stdout: "level": "advanced", 2024-03-24T03:17:22.950 INFO:teuthology.orchestra.run.smithi118.stdout: "flags": 1, 2024-03-24T03:17:22.950 INFO:teuthology.orchestra.run.smithi118.stdout: "default_value": "9283", 2024-03-24T03:17:22.950 INFO:teuthology.orchestra.run.smithi118.stdout: "min": "", 2024-03-24T03:17:22.950 INFO:teuthology.orchestra.run.smithi118.stdout: "max": "", 2024-03-24T03:17:22.951 INFO:teuthology.orchestra.run.smithi118.stdout: "enum_allowed": [], 2024-03-24T03:17:22.951 INFO:teuthology.orchestra.run.smithi118.stdout: "desc": "the port on which the module listens for HTTP requests", 2024-03-24T03:17:22.951 INFO:teuthology.orchestra.run.smithi118.stdout: "long_desc": "", 2024-03-24T03:17:22.951 INFO:teuthology.orchestra.run.smithi118.stdout: "tags": [], 2024-03-24T03:17:22.951 INFO:teuthology.orchestra.run.smithi118.stdout: "see_also": [] 2024-03-24T03:17:22.951 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-24T03:17:22.951 INFO:teuthology.orchestra.run.smithi118.stdout: "stale_cache_strategy": { 2024-03-24T03:17:22.951 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "stale_cache_strategy", 2024-03-24T03:17:22.951 INFO:teuthology.orchestra.run.smithi118.stdout: "type": "str", 2024-03-24T03:17:22.952 INFO:teuthology.orchestra.run.smithi118.stdout: "level": "advanced", 2024-03-24T03:17:22.952 INFO:teuthology.orchestra.run.smithi118.stdout: "flags": 0, 2024-03-24T03:17:22.952 INFO:teuthology.orchestra.run.smithi118.stdout: "default_value": "log", 2024-03-24T03:17:22.952 INFO:teuthology.orchestra.run.smithi118.stdout: "min": "", 2024-03-24T03:17:22.952 INFO:teuthology.orchestra.run.smithi118.stdout: "max": "", 2024-03-24T03:17:22.952 INFO:teuthology.orchestra.run.smithi118.stdout: "enum_allowed": [], 2024-03-24T03:17:22.952 INFO:teuthology.orchestra.run.smithi118.stdout: "desc": "", 2024-03-24T03:17:22.952 INFO:teuthology.orchestra.run.smithi118.stdout: "long_desc": "", 2024-03-24T03:17:22.952 INFO:teuthology.orchestra.run.smithi118.stdout: "tags": [], 2024-03-24T03:17:22.953 INFO:teuthology.orchestra.run.smithi118.stdout: "see_also": [] 2024-03-24T03:17:22.953 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-24T03:17:22.953 INFO:teuthology.orchestra.run.smithi118.stdout: "standby_behaviour": { 2024-03-24T03:17:22.953 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "standby_behaviour", 2024-03-24T03:17:22.953 INFO:teuthology.orchestra.run.smithi118.stdout: "type": "str", 2024-03-24T03:17:22.953 INFO:teuthology.orchestra.run.smithi118.stdout: "level": "advanced", 2024-03-24T03:17:22.953 INFO:teuthology.orchestra.run.smithi118.stdout: "flags": 1, 2024-03-24T03:17:22.953 INFO:teuthology.orchestra.run.smithi118.stdout: "default_value": "default", 2024-03-24T03:17:22.954 INFO:teuthology.orchestra.run.smithi118.stdout: "min": "", 2024-03-24T03:17:22.954 INFO:teuthology.orchestra.run.smithi118.stdout: "max": "", 2024-03-24T03:17:22.954 INFO:teuthology.orchestra.run.smithi118.stdout: "enum_allowed": [ 2024-03-24T03:17:22.954 INFO:teuthology.orchestra.run.smithi118.stdout: "default", 2024-03-24T03:17:22.954 INFO:teuthology.orchestra.run.smithi118.stdout: "error" 2024-03-24T03:17:22.954 INFO:teuthology.orchestra.run.smithi118.stdout: ], 2024-03-24T03:17:22.954 INFO:teuthology.orchestra.run.smithi118.stdout: "desc": "", 2024-03-24T03:17:22.955 INFO:teuthology.orchestra.run.smithi118.stdout: "long_desc": "", 2024-03-24T03:17:22.955 INFO:teuthology.orchestra.run.smithi118.stdout: "tags": [], 2024-03-24T03:17:22.955 INFO:teuthology.orchestra.run.smithi118.stdout: "see_also": [] 2024-03-24T03:17:22.955 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-24T03:17:22.955 INFO:teuthology.orchestra.run.smithi118.stdout: "standby_error_status_code": { 2024-03-24T03:17:22.955 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "standby_error_status_code", 2024-03-24T03:17:22.955 INFO:teuthology.orchestra.run.smithi118.stdout: "type": "int", 2024-03-24T03:17:22.955 INFO:teuthology.orchestra.run.smithi118.stdout: "level": "advanced", 2024-03-24T03:17:22.955 INFO:teuthology.orchestra.run.smithi118.stdout: "flags": 1, 2024-03-24T03:17:22.956 INFO:teuthology.orchestra.run.smithi118.stdout: "default_value": "500", 2024-03-24T03:17:22.956 INFO:teuthology.orchestra.run.smithi118.stdout: "min": "400", 2024-03-24T03:17:22.956 INFO:teuthology.orchestra.run.smithi118.stdout: "max": "599", 2024-03-24T03:17:22.956 INFO:teuthology.orchestra.run.smithi118.stdout: "enum_allowed": [], 2024-03-24T03:17:22.956 INFO:teuthology.orchestra.run.smithi118.stdout: "desc": "", 2024-03-24T03:17:22.956 INFO:teuthology.orchestra.run.smithi118.stdout: "long_desc": "", 2024-03-24T03:17:22.956 INFO:teuthology.orchestra.run.smithi118.stdout: "tags": [], 2024-03-24T03:17:22.956 INFO:teuthology.orchestra.run.smithi118.stdout: "see_also": [] 2024-03-24T03:17:22.956 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-03-24T03:17:22.957 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-03-24T03:17:22.957 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-24T03:17:22.957 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-24T03:17:22.957 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "rbd_support", 2024-03-24T03:17:22.957 INFO:teuthology.orchestra.run.smithi118.stdout: "can_run": true, 2024-03-24T03:17:22.957 INFO:teuthology.orchestra.run.smithi118.stdout: "error_string": "", 2024-03-24T03:17:22.957 INFO:teuthology.orchestra.run.smithi118.stdout: "module_options": { 2024-03-24T03:17:22.957 INFO:teuthology.orchestra.run.smithi118.stdout: "log_level": { 2024-03-24T03:17:22.957 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "log_level", 2024-03-24T03:17:22.958 INFO:teuthology.orchestra.run.smithi118.stdout: "type": "str", 2024-03-24T03:17:22.958 INFO:teuthology.orchestra.run.smithi118.stdout: "level": "advanced", 2024-03-24T03:17:22.958 INFO:teuthology.orchestra.run.smithi118.stdout: "flags": 1, 2024-03-24T03:17:22.958 INFO:teuthology.orchestra.run.smithi118.stdout: "default_value": "", 2024-03-24T03:17:22.958 INFO:teuthology.orchestra.run.smithi118.stdout: "min": "", 2024-03-24T03:17:22.958 INFO:teuthology.orchestra.run.smithi118.stdout: "max": "", 2024-03-24T03:17:22.958 INFO:teuthology.orchestra.run.smithi118.stdout: "enum_allowed": [ 2024-03-24T03:17:22.958 INFO:teuthology.orchestra.run.smithi118.stdout: "", 2024-03-24T03:17:22.959 INFO:teuthology.orchestra.run.smithi118.stdout: "critical", 2024-03-24T03:17:22.959 INFO:teuthology.orchestra.run.smithi118.stdout: "debug", 2024-03-24T03:17:22.959 INFO:teuthology.orchestra.run.smithi118.stdout: "error", 2024-03-24T03:17:22.959 INFO:teuthology.orchestra.run.smithi118.stdout: "info", 2024-03-24T03:17:22.959 INFO:teuthology.orchestra.run.smithi118.stdout: "warning" 2024-03-24T03:17:22.959 INFO:teuthology.orchestra.run.smithi118.stdout: ], 2024-03-24T03:17:22.959 INFO:teuthology.orchestra.run.smithi118.stdout: "desc": "", 2024-03-24T03:17:22.959 INFO:teuthology.orchestra.run.smithi118.stdout: "long_desc": "", 2024-03-24T03:17:22.959 INFO:teuthology.orchestra.run.smithi118.stdout: "tags": [], 2024-03-24T03:17:22.960 INFO:teuthology.orchestra.run.smithi118.stdout: "see_also": [] 2024-03-24T03:17:22.960 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-24T03:17:22.960 INFO:teuthology.orchestra.run.smithi118.stdout: "log_to_cluster": { 2024-03-24T03:17:22.960 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "log_to_cluster", 2024-03-24T03:17:22.960 INFO:teuthology.orchestra.run.smithi118.stdout: "type": "bool", 2024-03-24T03:17:22.960 INFO:teuthology.orchestra.run.smithi118.stdout: "level": "advanced", 2024-03-24T03:17:22.960 INFO:teuthology.orchestra.run.smithi118.stdout: "flags": 1, 2024-03-24T03:17:22.960 INFO:teuthology.orchestra.run.smithi118.stdout: "default_value": "False", 2024-03-24T03:17:22.961 INFO:teuthology.orchestra.run.smithi118.stdout: "min": "", 2024-03-24T03:17:22.961 INFO:teuthology.orchestra.run.smithi118.stdout: "max": "", 2024-03-24T03:17:22.961 INFO:teuthology.orchestra.run.smithi118.stdout: "enum_allowed": [], 2024-03-24T03:17:22.961 INFO:teuthology.orchestra.run.smithi118.stdout: "desc": "", 2024-03-24T03:17:22.961 INFO:teuthology.orchestra.run.smithi118.stdout: "long_desc": "", 2024-03-24T03:17:22.961 INFO:teuthology.orchestra.run.smithi118.stdout: "tags": [], 2024-03-24T03:17:22.961 INFO:teuthology.orchestra.run.smithi118.stdout: "see_also": [] 2024-03-24T03:17:22.961 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-24T03:17:22.962 INFO:teuthology.orchestra.run.smithi118.stdout: "log_to_cluster_level": { 2024-03-24T03:17:22.962 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "log_to_cluster_level", 2024-03-24T03:17:22.962 INFO:teuthology.orchestra.run.smithi118.stdout: "type": "str", 2024-03-24T03:17:22.962 INFO:teuthology.orchestra.run.smithi118.stdout: "level": "advanced", 2024-03-24T03:17:22.962 INFO:teuthology.orchestra.run.smithi118.stdout: "flags": 1, 2024-03-24T03:17:22.962 INFO:teuthology.orchestra.run.smithi118.stdout: "default_value": "info", 2024-03-24T03:17:22.962 INFO:teuthology.orchestra.run.smithi118.stdout: "min": "", 2024-03-24T03:17:22.962 INFO:teuthology.orchestra.run.smithi118.stdout: "max": "", 2024-03-24T03:17:22.963 INFO:teuthology.orchestra.run.smithi118.stdout: "enum_allowed": [ 2024-03-24T03:17:22.963 INFO:teuthology.orchestra.run.smithi118.stdout: "", 2024-03-24T03:17:22.963 INFO:teuthology.orchestra.run.smithi118.stdout: "critical", 2024-03-24T03:17:22.963 INFO:teuthology.orchestra.run.smithi118.stdout: "debug", 2024-03-24T03:17:22.963 INFO:teuthology.orchestra.run.smithi118.stdout: "error", 2024-03-24T03:17:22.963 INFO:teuthology.orchestra.run.smithi118.stdout: "info", 2024-03-24T03:17:22.963 INFO:teuthology.orchestra.run.smithi118.stdout: "warning" 2024-03-24T03:17:22.963 INFO:teuthology.orchestra.run.smithi118.stdout: ], 2024-03-24T03:17:22.964 INFO:teuthology.orchestra.run.smithi118.stdout: "desc": "", 2024-03-24T03:17:22.964 INFO:teuthology.orchestra.run.smithi118.stdout: "long_desc": "", 2024-03-24T03:17:22.964 INFO:teuthology.orchestra.run.smithi118.stdout: "tags": [], 2024-03-24T03:17:22.964 INFO:teuthology.orchestra.run.smithi118.stdout: "see_also": [] 2024-03-24T03:17:22.964 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-24T03:17:22.964 INFO:teuthology.orchestra.run.smithi118.stdout: "log_to_file": { 2024-03-24T03:17:22.964 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "log_to_file", 2024-03-24T03:17:22.965 INFO:teuthology.orchestra.run.smithi118.stdout: "type": "bool", 2024-03-24T03:17:22.965 INFO:teuthology.orchestra.run.smithi118.stdout: "level": "advanced", 2024-03-24T03:17:22.965 INFO:teuthology.orchestra.run.smithi118.stdout: "flags": 1, 2024-03-24T03:17:22.965 INFO:teuthology.orchestra.run.smithi118.stdout: "default_value": "False", 2024-03-24T03:17:22.965 INFO:teuthology.orchestra.run.smithi118.stdout: "min": "", 2024-03-24T03:17:22.965 INFO:teuthology.orchestra.run.smithi118.stdout: "max": "", 2024-03-24T03:17:22.965 INFO:teuthology.orchestra.run.smithi118.stdout: "enum_allowed": [], 2024-03-24T03:17:22.965 INFO:teuthology.orchestra.run.smithi118.stdout: "desc": "", 2024-03-24T03:17:22.966 INFO:teuthology.orchestra.run.smithi118.stdout: "long_desc": "", 2024-03-24T03:17:22.966 INFO:teuthology.orchestra.run.smithi118.stdout: "tags": [], 2024-03-24T03:17:22.966 INFO:teuthology.orchestra.run.smithi118.stdout: "see_also": [] 2024-03-24T03:17:22.966 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-24T03:17:22.966 INFO:teuthology.orchestra.run.smithi118.stdout: "max_concurrent_snap_create": { 2024-03-24T03:17:22.966 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "max_concurrent_snap_create", 2024-03-24T03:17:22.966 INFO:teuthology.orchestra.run.smithi118.stdout: "type": "int", 2024-03-24T03:17:22.966 INFO:teuthology.orchestra.run.smithi118.stdout: "level": "advanced", 2024-03-24T03:17:22.967 INFO:teuthology.orchestra.run.smithi118.stdout: "flags": 0, 2024-03-24T03:17:22.967 INFO:teuthology.orchestra.run.smithi118.stdout: "default_value": "10", 2024-03-24T03:17:22.967 INFO:teuthology.orchestra.run.smithi118.stdout: "min": "", 2024-03-24T03:17:22.967 INFO:teuthology.orchestra.run.smithi118.stdout: "max": "", 2024-03-24T03:17:22.967 INFO:teuthology.orchestra.run.smithi118.stdout: "enum_allowed": [], 2024-03-24T03:17:22.967 INFO:teuthology.orchestra.run.smithi118.stdout: "desc": "", 2024-03-24T03:17:22.967 INFO:teuthology.orchestra.run.smithi118.stdout: "long_desc": "", 2024-03-24T03:17:22.967 INFO:teuthology.orchestra.run.smithi118.stdout: "tags": [], 2024-03-24T03:17:22.967 INFO:teuthology.orchestra.run.smithi118.stdout: "see_also": [] 2024-03-24T03:17:22.968 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-24T03:17:22.968 INFO:teuthology.orchestra.run.smithi118.stdout: "mirror_snapshot_schedule": { 2024-03-24T03:17:22.968 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "mirror_snapshot_schedule", 2024-03-24T03:17:22.968 INFO:teuthology.orchestra.run.smithi118.stdout: "type": "str", 2024-03-24T03:17:22.968 INFO:teuthology.orchestra.run.smithi118.stdout: "level": "advanced", 2024-03-24T03:17:22.968 INFO:teuthology.orchestra.run.smithi118.stdout: "flags": 0, 2024-03-24T03:17:22.968 INFO:teuthology.orchestra.run.smithi118.stdout: "default_value": "", 2024-03-24T03:17:22.968 INFO:teuthology.orchestra.run.smithi118.stdout: "min": "", 2024-03-24T03:17:22.969 INFO:teuthology.orchestra.run.smithi118.stdout: "max": "", 2024-03-24T03:17:22.969 INFO:teuthology.orchestra.run.smithi118.stdout: "enum_allowed": [], 2024-03-24T03:17:22.969 INFO:teuthology.orchestra.run.smithi118.stdout: "desc": "", 2024-03-24T03:17:22.969 INFO:teuthology.orchestra.run.smithi118.stdout: "long_desc": "", 2024-03-24T03:17:22.969 INFO:teuthology.orchestra.run.smithi118.stdout: "tags": [], 2024-03-24T03:17:22.969 INFO:teuthology.orchestra.run.smithi118.stdout: "see_also": [] 2024-03-24T03:17:22.969 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-24T03:17:22.969 INFO:teuthology.orchestra.run.smithi118.stdout: "trash_purge_schedule": { 2024-03-24T03:17:22.969 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "trash_purge_schedule", 2024-03-24T03:17:22.970 INFO:teuthology.orchestra.run.smithi118.stdout: "type": "str", 2024-03-24T03:17:22.970 INFO:teuthology.orchestra.run.smithi118.stdout: "level": "advanced", 2024-03-24T03:17:22.970 INFO:teuthology.orchestra.run.smithi118.stdout: "flags": 0, 2024-03-24T03:17:22.970 INFO:teuthology.orchestra.run.smithi118.stdout: "default_value": "", 2024-03-24T03:17:22.970 INFO:teuthology.orchestra.run.smithi118.stdout: "min": "", 2024-03-24T03:17:22.970 INFO:teuthology.orchestra.run.smithi118.stdout: "max": "", 2024-03-24T03:17:22.970 INFO:teuthology.orchestra.run.smithi118.stdout: "enum_allowed": [], 2024-03-24T03:17:22.970 INFO:teuthology.orchestra.run.smithi118.stdout: "desc": "", 2024-03-24T03:17:22.971 INFO:teuthology.orchestra.run.smithi118.stdout: "long_desc": "", 2024-03-24T03:17:22.971 INFO:teuthology.orchestra.run.smithi118.stdout: "tags": [], 2024-03-24T03:17:22.971 INFO:teuthology.orchestra.run.smithi118.stdout: "see_also": [] 2024-03-24T03:17:22.971 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-03-24T03:17:22.971 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-03-24T03:17:22.971 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-24T03:17:22.971 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-24T03:17:22.971 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "restful", 2024-03-24T03:17:22.972 INFO:teuthology.orchestra.run.smithi118.stdout: "can_run": true, 2024-03-24T03:17:22.972 INFO:teuthology.orchestra.run.smithi118.stdout: "error_string": "", 2024-03-24T03:17:22.972 INFO:teuthology.orchestra.run.smithi118.stdout: "module_options": { 2024-03-24T03:17:22.972 INFO:teuthology.orchestra.run.smithi118.stdout: "enable_auth": { 2024-03-24T03:17:22.972 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "enable_auth", 2024-03-24T03:17:22.972 INFO:teuthology.orchestra.run.smithi118.stdout: "type": "bool", 2024-03-24T03:17:22.972 INFO:teuthology.orchestra.run.smithi118.stdout: "level": "advanced", 2024-03-24T03:17:22.972 INFO:teuthology.orchestra.run.smithi118.stdout: "flags": 0, 2024-03-24T03:17:22.972 INFO:teuthology.orchestra.run.smithi118.stdout: "default_value": "True", 2024-03-24T03:17:22.973 INFO:teuthology.orchestra.run.smithi118.stdout: "min": "", 2024-03-24T03:17:22.973 INFO:teuthology.orchestra.run.smithi118.stdout: "max": "", 2024-03-24T03:17:22.973 INFO:teuthology.orchestra.run.smithi118.stdout: "enum_allowed": [], 2024-03-24T03:17:22.973 INFO:teuthology.orchestra.run.smithi118.stdout: "desc": "", 2024-03-24T03:17:22.973 INFO:teuthology.orchestra.run.smithi118.stdout: "long_desc": "", 2024-03-24T03:17:22.973 INFO:teuthology.orchestra.run.smithi118.stdout: "tags": [], 2024-03-24T03:17:22.973 INFO:teuthology.orchestra.run.smithi118.stdout: "see_also": [] 2024-03-24T03:17:22.973 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-24T03:17:22.973 INFO:teuthology.orchestra.run.smithi118.stdout: "key_file": { 2024-03-24T03:17:22.974 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "key_file", 2024-03-24T03:17:22.974 INFO:teuthology.orchestra.run.smithi118.stdout: "type": "str", 2024-03-24T03:17:22.974 INFO:teuthology.orchestra.run.smithi118.stdout: "level": "advanced", 2024-03-24T03:17:22.974 INFO:teuthology.orchestra.run.smithi118.stdout: "flags": 0, 2024-03-24T03:17:22.974 INFO:teuthology.orchestra.run.smithi118.stdout: "default_value": "", 2024-03-24T03:17:22.974 INFO:teuthology.orchestra.run.smithi118.stdout: "min": "", 2024-03-24T03:17:22.975 INFO:teuthology.orchestra.run.smithi118.stdout: "max": "", 2024-03-24T03:17:22.975 INFO:teuthology.orchestra.run.smithi118.stdout: "enum_allowed": [], 2024-03-24T03:17:22.975 INFO:teuthology.orchestra.run.smithi118.stdout: "desc": "", 2024-03-24T03:17:22.975 INFO:teuthology.orchestra.run.smithi118.stdout: "long_desc": "", 2024-03-24T03:17:22.975 INFO:teuthology.orchestra.run.smithi118.stdout: "tags": [], 2024-03-24T03:17:22.975 INFO:teuthology.orchestra.run.smithi118.stdout: "see_also": [] 2024-03-24T03:17:22.975 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-24T03:17:22.975 INFO:teuthology.orchestra.run.smithi118.stdout: "log_level": { 2024-03-24T03:17:22.975 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "log_level", 2024-03-24T03:17:22.976 INFO:teuthology.orchestra.run.smithi118.stdout: "type": "str", 2024-03-24T03:17:22.976 INFO:teuthology.orchestra.run.smithi118.stdout: "level": "advanced", 2024-03-24T03:17:22.976 INFO:teuthology.orchestra.run.smithi118.stdout: "flags": 1, 2024-03-24T03:17:22.976 INFO:teuthology.orchestra.run.smithi118.stdout: "default_value": "", 2024-03-24T03:17:22.976 INFO:teuthology.orchestra.run.smithi118.stdout: "min": "", 2024-03-24T03:17:22.976 INFO:teuthology.orchestra.run.smithi118.stdout: "max": "", 2024-03-24T03:17:22.977 INFO:teuthology.orchestra.run.smithi118.stdout: "enum_allowed": [ 2024-03-24T03:17:22.977 INFO:teuthology.orchestra.run.smithi118.stdout: "", 2024-03-24T03:17:22.977 INFO:teuthology.orchestra.run.smithi118.stdout: "critical", 2024-03-24T03:17:22.977 INFO:teuthology.orchestra.run.smithi118.stdout: "debug", 2024-03-24T03:17:22.977 INFO:teuthology.orchestra.run.smithi118.stdout: "error", 2024-03-24T03:17:22.977 INFO:teuthology.orchestra.run.smithi118.stdout: "info", 2024-03-24T03:17:22.977 INFO:teuthology.orchestra.run.smithi118.stdout: "warning" 2024-03-24T03:17:22.977 INFO:teuthology.orchestra.run.smithi118.stdout: ], 2024-03-24T03:17:22.977 INFO:teuthology.orchestra.run.smithi118.stdout: "desc": "", 2024-03-24T03:17:22.978 INFO:teuthology.orchestra.run.smithi118.stdout: "long_desc": "", 2024-03-24T03:17:22.978 INFO:teuthology.orchestra.run.smithi118.stdout: "tags": [], 2024-03-24T03:17:22.978 INFO:teuthology.orchestra.run.smithi118.stdout: "see_also": [] 2024-03-24T03:17:22.978 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-24T03:17:22.978 INFO:teuthology.orchestra.run.smithi118.stdout: "log_to_cluster": { 2024-03-24T03:17:22.978 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "log_to_cluster", 2024-03-24T03:17:22.978 INFO:teuthology.orchestra.run.smithi118.stdout: "type": "bool", 2024-03-24T03:17:22.978 INFO:teuthology.orchestra.run.smithi118.stdout: "level": "advanced", 2024-03-24T03:17:22.979 INFO:teuthology.orchestra.run.smithi118.stdout: "flags": 1, 2024-03-24T03:17:22.979 INFO:teuthology.orchestra.run.smithi118.stdout: "default_value": "False", 2024-03-24T03:17:22.979 INFO:teuthology.orchestra.run.smithi118.stdout: "min": "", 2024-03-24T03:17:22.979 INFO:teuthology.orchestra.run.smithi118.stdout: "max": "", 2024-03-24T03:17:22.979 INFO:teuthology.orchestra.run.smithi118.stdout: "enum_allowed": [], 2024-03-24T03:17:22.979 INFO:teuthology.orchestra.run.smithi118.stdout: "desc": "", 2024-03-24T03:17:22.979 INFO:teuthology.orchestra.run.smithi118.stdout: "long_desc": "", 2024-03-24T03:17:22.979 INFO:teuthology.orchestra.run.smithi118.stdout: "tags": [], 2024-03-24T03:17:22.980 INFO:teuthology.orchestra.run.smithi118.stdout: "see_also": [] 2024-03-24T03:17:22.980 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-24T03:17:22.980 INFO:teuthology.orchestra.run.smithi118.stdout: "log_to_cluster_level": { 2024-03-24T03:17:22.980 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "log_to_cluster_level", 2024-03-24T03:17:22.980 INFO:teuthology.orchestra.run.smithi118.stdout: "type": "str", 2024-03-24T03:17:22.980 INFO:teuthology.orchestra.run.smithi118.stdout: "level": "advanced", 2024-03-24T03:17:22.980 INFO:teuthology.orchestra.run.smithi118.stdout: "flags": 1, 2024-03-24T03:17:22.980 INFO:teuthology.orchestra.run.smithi118.stdout: "default_value": "info", 2024-03-24T03:17:22.980 INFO:teuthology.orchestra.run.smithi118.stdout: "min": "", 2024-03-24T03:17:22.981 INFO:teuthology.orchestra.run.smithi118.stdout: "max": "", 2024-03-24T03:17:22.981 INFO:teuthology.orchestra.run.smithi118.stdout: "enum_allowed": [ 2024-03-24T03:17:22.981 INFO:teuthology.orchestra.run.smithi118.stdout: "", 2024-03-24T03:17:22.981 INFO:teuthology.orchestra.run.smithi118.stdout: "critical", 2024-03-24T03:17:22.981 INFO:teuthology.orchestra.run.smithi118.stdout: "debug", 2024-03-24T03:17:22.981 INFO:teuthology.orchestra.run.smithi118.stdout: "error", 2024-03-24T03:17:22.981 INFO:teuthology.orchestra.run.smithi118.stdout: "info", 2024-03-24T03:17:22.981 INFO:teuthology.orchestra.run.smithi118.stdout: "warning" 2024-03-24T03:17:22.981 INFO:teuthology.orchestra.run.smithi118.stdout: ], 2024-03-24T03:17:22.982 INFO:teuthology.orchestra.run.smithi118.stdout: "desc": "", 2024-03-24T03:17:22.982 INFO:teuthology.orchestra.run.smithi118.stdout: "long_desc": "", 2024-03-24T03:17:22.982 INFO:teuthology.orchestra.run.smithi118.stdout: "tags": [], 2024-03-24T03:17:22.982 INFO:teuthology.orchestra.run.smithi118.stdout: "see_also": [] 2024-03-24T03:17:22.982 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-24T03:17:22.982 INFO:teuthology.orchestra.run.smithi118.stdout: "log_to_file": { 2024-03-24T03:17:22.982 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "log_to_file", 2024-03-24T03:17:22.982 INFO:teuthology.orchestra.run.smithi118.stdout: "type": "bool", 2024-03-24T03:17:22.983 INFO:teuthology.orchestra.run.smithi118.stdout: "level": "advanced", 2024-03-24T03:17:22.983 INFO:teuthology.orchestra.run.smithi118.stdout: "flags": 1, 2024-03-24T03:17:22.983 INFO:teuthology.orchestra.run.smithi118.stdout: "default_value": "False", 2024-03-24T03:17:22.983 INFO:teuthology.orchestra.run.smithi118.stdout: "min": "", 2024-03-24T03:17:22.983 INFO:teuthology.orchestra.run.smithi118.stdout: "max": "", 2024-03-24T03:17:22.983 INFO:teuthology.orchestra.run.smithi118.stdout: "enum_allowed": [], 2024-03-24T03:17:22.983 INFO:teuthology.orchestra.run.smithi118.stdout: "desc": "", 2024-03-24T03:17:22.983 INFO:teuthology.orchestra.run.smithi118.stdout: "long_desc": "", 2024-03-24T03:17:22.983 INFO:teuthology.orchestra.run.smithi118.stdout: "tags": [], 2024-03-24T03:17:22.984 INFO:teuthology.orchestra.run.smithi118.stdout: "see_also": [] 2024-03-24T03:17:22.984 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-24T03:17:22.984 INFO:teuthology.orchestra.run.smithi118.stdout: "server_addr": { 2024-03-24T03:17:22.984 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "server_addr", 2024-03-24T03:17:22.984 INFO:teuthology.orchestra.run.smithi118.stdout: "type": "str", 2024-03-24T03:17:22.984 INFO:teuthology.orchestra.run.smithi118.stdout: "level": "advanced", 2024-03-24T03:17:22.984 INFO:teuthology.orchestra.run.smithi118.stdout: "flags": 0, 2024-03-24T03:17:22.984 INFO:teuthology.orchestra.run.smithi118.stdout: "default_value": "", 2024-03-24T03:17:22.985 INFO:teuthology.orchestra.run.smithi118.stdout: "min": "", 2024-03-24T03:17:22.985 INFO:teuthology.orchestra.run.smithi118.stdout: "max": "", 2024-03-24T03:17:22.985 INFO:teuthology.orchestra.run.smithi118.stdout: "enum_allowed": [], 2024-03-24T03:17:22.985 INFO:teuthology.orchestra.run.smithi118.stdout: "desc": "", 2024-03-24T03:17:22.985 INFO:teuthology.orchestra.run.smithi118.stdout: "long_desc": "", 2024-03-24T03:17:22.985 INFO:teuthology.orchestra.run.smithi118.stdout: "tags": [], 2024-03-24T03:17:22.985 INFO:teuthology.orchestra.run.smithi118.stdout: "see_also": [] 2024-03-24T03:17:22.985 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-24T03:17:22.986 INFO:teuthology.orchestra.run.smithi118.stdout: "server_port": { 2024-03-24T03:17:22.986 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "server_port", 2024-03-24T03:17:22.986 INFO:teuthology.orchestra.run.smithi118.stdout: "type": "str", 2024-03-24T03:17:22.986 INFO:teuthology.orchestra.run.smithi118.stdout: "level": "advanced", 2024-03-24T03:17:22.986 INFO:teuthology.orchestra.run.smithi118.stdout: "flags": 0, 2024-03-24T03:17:22.986 INFO:teuthology.orchestra.run.smithi118.stdout: "default_value": "", 2024-03-24T03:17:22.986 INFO:teuthology.orchestra.run.smithi118.stdout: "min": "", 2024-03-24T03:17:22.986 INFO:teuthology.orchestra.run.smithi118.stdout: "max": "", 2024-03-24T03:17:22.987 INFO:teuthology.orchestra.run.smithi118.stdout: "enum_allowed": [], 2024-03-24T03:17:22.987 INFO:teuthology.orchestra.run.smithi118.stdout: "desc": "", 2024-03-24T03:17:22.987 INFO:teuthology.orchestra.run.smithi118.stdout: "long_desc": "", 2024-03-24T03:17:22.987 INFO:teuthology.orchestra.run.smithi118.stdout: "tags": [], 2024-03-24T03:17:22.987 INFO:teuthology.orchestra.run.smithi118.stdout: "see_also": [] 2024-03-24T03:17:22.987 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-03-24T03:17:22.987 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-03-24T03:17:22.987 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-24T03:17:22.988 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-24T03:17:22.988 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "rgw", 2024-03-24T03:17:22.988 INFO:teuthology.orchestra.run.smithi118.stdout: "can_run": true, 2024-03-24T03:17:22.988 INFO:teuthology.orchestra.run.smithi118.stdout: "error_string": "", 2024-03-24T03:17:22.988 INFO:teuthology.orchestra.run.smithi118.stdout: "module_options": { 2024-03-24T03:17:22.988 INFO:teuthology.orchestra.run.smithi118.stdout: "log_level": { 2024-03-24T03:17:22.988 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "log_level", 2024-03-24T03:17:22.988 INFO:teuthology.orchestra.run.smithi118.stdout: "type": "str", 2024-03-24T03:17:22.989 INFO:teuthology.orchestra.run.smithi118.stdout: "level": "advanced", 2024-03-24T03:17:22.989 INFO:teuthology.orchestra.run.smithi118.stdout: "flags": 1, 2024-03-24T03:17:22.989 INFO:teuthology.orchestra.run.smithi118.stdout: "default_value": "", 2024-03-24T03:17:22.989 INFO:teuthology.orchestra.run.smithi118.stdout: "min": "", 2024-03-24T03:17:22.989 INFO:teuthology.orchestra.run.smithi118.stdout: "max": "", 2024-03-24T03:17:22.989 INFO:teuthology.orchestra.run.smithi118.stdout: "enum_allowed": [ 2024-03-24T03:17:22.989 INFO:teuthology.orchestra.run.smithi118.stdout: "", 2024-03-24T03:17:22.989 INFO:teuthology.orchestra.run.smithi118.stdout: "critical", 2024-03-24T03:17:22.990 INFO:teuthology.orchestra.run.smithi118.stdout: "debug", 2024-03-24T03:17:22.990 INFO:teuthology.orchestra.run.smithi118.stdout: "error", 2024-03-24T03:17:22.990 INFO:teuthology.orchestra.run.smithi118.stdout: "info", 2024-03-24T03:17:22.990 INFO:teuthology.orchestra.run.smithi118.stdout: "warning" 2024-03-24T03:17:22.990 INFO:teuthology.orchestra.run.smithi118.stdout: ], 2024-03-24T03:17:22.990 INFO:teuthology.orchestra.run.smithi118.stdout: "desc": "", 2024-03-24T03:17:22.990 INFO:teuthology.orchestra.run.smithi118.stdout: "long_desc": "", 2024-03-24T03:17:22.990 INFO:teuthology.orchestra.run.smithi118.stdout: "tags": [], 2024-03-24T03:17:22.990 INFO:teuthology.orchestra.run.smithi118.stdout: "see_also": [] 2024-03-24T03:17:22.991 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-24T03:17:22.991 INFO:teuthology.orchestra.run.smithi118.stdout: "log_to_cluster": { 2024-03-24T03:17:22.991 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "log_to_cluster", 2024-03-24T03:17:22.991 INFO:teuthology.orchestra.run.smithi118.stdout: "type": "bool", 2024-03-24T03:17:22.991 INFO:teuthology.orchestra.run.smithi118.stdout: "level": "advanced", 2024-03-24T03:17:22.991 INFO:teuthology.orchestra.run.smithi118.stdout: "flags": 1, 2024-03-24T03:17:22.991 INFO:teuthology.orchestra.run.smithi118.stdout: "default_value": "False", 2024-03-24T03:17:22.991 INFO:teuthology.orchestra.run.smithi118.stdout: "min": "", 2024-03-24T03:17:22.992 INFO:teuthology.orchestra.run.smithi118.stdout: "max": "", 2024-03-24T03:17:22.992 INFO:teuthology.orchestra.run.smithi118.stdout: "enum_allowed": [], 2024-03-24T03:17:22.992 INFO:teuthology.orchestra.run.smithi118.stdout: "desc": "", 2024-03-24T03:17:22.992 INFO:teuthology.orchestra.run.smithi118.stdout: "long_desc": "", 2024-03-24T03:17:22.992 INFO:teuthology.orchestra.run.smithi118.stdout: "tags": [], 2024-03-24T03:17:22.992 INFO:teuthology.orchestra.run.smithi118.stdout: "see_also": [] 2024-03-24T03:17:22.992 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-24T03:17:22.992 INFO:teuthology.orchestra.run.smithi118.stdout: "log_to_cluster_level": { 2024-03-24T03:17:22.992 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "log_to_cluster_level", 2024-03-24T03:17:22.993 INFO:teuthology.orchestra.run.smithi118.stdout: "type": "str", 2024-03-24T03:17:22.993 INFO:teuthology.orchestra.run.smithi118.stdout: "level": "advanced", 2024-03-24T03:17:22.993 INFO:teuthology.orchestra.run.smithi118.stdout: "flags": 1, 2024-03-24T03:17:22.993 INFO:teuthology.orchestra.run.smithi118.stdout: "default_value": "info", 2024-03-24T03:17:22.993 INFO:teuthology.orchestra.run.smithi118.stdout: "min": "", 2024-03-24T03:17:22.993 INFO:teuthology.orchestra.run.smithi118.stdout: "max": "", 2024-03-24T03:17:22.993 INFO:teuthology.orchestra.run.smithi118.stdout: "enum_allowed": [ 2024-03-24T03:17:22.993 INFO:teuthology.orchestra.run.smithi118.stdout: "", 2024-03-24T03:17:22.993 INFO:teuthology.orchestra.run.smithi118.stdout: "critical", 2024-03-24T03:17:22.994 INFO:teuthology.orchestra.run.smithi118.stdout: "debug", 2024-03-24T03:17:22.994 INFO:teuthology.orchestra.run.smithi118.stdout: "error", 2024-03-24T03:17:22.994 INFO:teuthology.orchestra.run.smithi118.stdout: "info", 2024-03-24T03:17:22.994 INFO:teuthology.orchestra.run.smithi118.stdout: "warning" 2024-03-24T03:17:22.994 INFO:teuthology.orchestra.run.smithi118.stdout: ], 2024-03-24T03:17:22.994 INFO:teuthology.orchestra.run.smithi118.stdout: "desc": "", 2024-03-24T03:17:22.994 INFO:teuthology.orchestra.run.smithi118.stdout: "long_desc": "", 2024-03-24T03:17:22.995 INFO:teuthology.orchestra.run.smithi118.stdout: "tags": [], 2024-03-24T03:17:22.995 INFO:teuthology.orchestra.run.smithi118.stdout: "see_also": [] 2024-03-24T03:17:22.995 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-24T03:17:22.995 INFO:teuthology.orchestra.run.smithi118.stdout: "log_to_file": { 2024-03-24T03:17:22.995 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "log_to_file", 2024-03-24T03:17:22.995 INFO:teuthology.orchestra.run.smithi118.stdout: "type": "bool", 2024-03-24T03:17:22.995 INFO:teuthology.orchestra.run.smithi118.stdout: "level": "advanced", 2024-03-24T03:17:22.995 INFO:teuthology.orchestra.run.smithi118.stdout: "flags": 1, 2024-03-24T03:17:22.995 INFO:teuthology.orchestra.run.smithi118.stdout: "default_value": "False", 2024-03-24T03:17:22.996 INFO:teuthology.orchestra.run.smithi118.stdout: "min": "", 2024-03-24T03:17:22.996 INFO:teuthology.orchestra.run.smithi118.stdout: "max": "", 2024-03-24T03:17:22.996 INFO:teuthology.orchestra.run.smithi118.stdout: "enum_allowed": [], 2024-03-24T03:17:22.996 INFO:teuthology.orchestra.run.smithi118.stdout: "desc": "", 2024-03-24T03:17:22.996 INFO:teuthology.orchestra.run.smithi118.stdout: "long_desc": "", 2024-03-24T03:17:22.996 INFO:teuthology.orchestra.run.smithi118.stdout: "tags": [], 2024-03-24T03:17:22.996 INFO:teuthology.orchestra.run.smithi118.stdout: "see_also": [] 2024-03-24T03:17:22.996 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-03-24T03:17:22.996 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-03-24T03:17:22.997 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-24T03:17:22.997 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-24T03:17:22.997 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "rook", 2024-03-24T03:17:22.997 INFO:teuthology.orchestra.run.smithi118.stdout: "can_run": true, 2024-03-24T03:17:22.997 INFO:teuthology.orchestra.run.smithi118.stdout: "error_string": "", 2024-03-24T03:17:22.997 INFO:teuthology.orchestra.run.smithi118.stdout: "module_options": { 2024-03-24T03:17:22.997 INFO:teuthology.orchestra.run.smithi118.stdout: "log_level": { 2024-03-24T03:17:22.997 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "log_level", 2024-03-24T03:17:22.997 INFO:teuthology.orchestra.run.smithi118.stdout: "type": "str", 2024-03-24T03:17:22.998 INFO:teuthology.orchestra.run.smithi118.stdout: "level": "advanced", 2024-03-24T03:17:22.998 INFO:teuthology.orchestra.run.smithi118.stdout: "flags": 1, 2024-03-24T03:17:22.998 INFO:teuthology.orchestra.run.smithi118.stdout: "default_value": "", 2024-03-24T03:17:22.998 INFO:teuthology.orchestra.run.smithi118.stdout: "min": "", 2024-03-24T03:17:22.998 INFO:teuthology.orchestra.run.smithi118.stdout: "max": "", 2024-03-24T03:17:22.998 INFO:teuthology.orchestra.run.smithi118.stdout: "enum_allowed": [ 2024-03-24T03:17:22.998 INFO:teuthology.orchestra.run.smithi118.stdout: "", 2024-03-24T03:17:22.998 INFO:teuthology.orchestra.run.smithi118.stdout: "critical", 2024-03-24T03:17:22.999 INFO:teuthology.orchestra.run.smithi118.stdout: "debug", 2024-03-24T03:17:22.999 INFO:teuthology.orchestra.run.smithi118.stdout: "error", 2024-03-24T03:17:22.999 INFO:teuthology.orchestra.run.smithi118.stdout: "info", 2024-03-24T03:17:22.999 INFO:teuthology.orchestra.run.smithi118.stdout: "warning" 2024-03-24T03:17:22.999 INFO:teuthology.orchestra.run.smithi118.stdout: ], 2024-03-24T03:17:22.999 INFO:teuthology.orchestra.run.smithi118.stdout: "desc": "", 2024-03-24T03:17:22.999 INFO:teuthology.orchestra.run.smithi118.stdout: "long_desc": "", 2024-03-24T03:17:22.999 INFO:teuthology.orchestra.run.smithi118.stdout: "tags": [], 2024-03-24T03:17:22.999 INFO:teuthology.orchestra.run.smithi118.stdout: "see_also": [] 2024-03-24T03:17:23.000 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-24T03:17:23.000 INFO:teuthology.orchestra.run.smithi118.stdout: "log_to_cluster": { 2024-03-24T03:17:23.000 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "log_to_cluster", 2024-03-24T03:17:23.000 INFO:teuthology.orchestra.run.smithi118.stdout: "type": "bool", 2024-03-24T03:17:23.000 INFO:teuthology.orchestra.run.smithi118.stdout: "level": "advanced", 2024-03-24T03:17:23.000 INFO:teuthology.orchestra.run.smithi118.stdout: "flags": 1, 2024-03-24T03:17:23.000 INFO:teuthology.orchestra.run.smithi118.stdout: "default_value": "False", 2024-03-24T03:17:23.000 INFO:teuthology.orchestra.run.smithi118.stdout: "min": "", 2024-03-24T03:17:23.001 INFO:teuthology.orchestra.run.smithi118.stdout: "max": "", 2024-03-24T03:17:23.001 INFO:teuthology.orchestra.run.smithi118.stdout: "enum_allowed": [], 2024-03-24T03:17:23.001 INFO:teuthology.orchestra.run.smithi118.stdout: "desc": "", 2024-03-24T03:17:23.001 INFO:teuthology.orchestra.run.smithi118.stdout: "long_desc": "", 2024-03-24T03:17:23.001 INFO:teuthology.orchestra.run.smithi118.stdout: "tags": [], 2024-03-24T03:17:23.001 INFO:teuthology.orchestra.run.smithi118.stdout: "see_also": [] 2024-03-24T03:17:23.001 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-24T03:17:23.001 INFO:teuthology.orchestra.run.smithi118.stdout: "log_to_cluster_level": { 2024-03-24T03:17:23.001 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "log_to_cluster_level", 2024-03-24T03:17:23.002 INFO:teuthology.orchestra.run.smithi118.stdout: "type": "str", 2024-03-24T03:17:23.002 INFO:teuthology.orchestra.run.smithi118.stdout: "level": "advanced", 2024-03-24T03:17:23.002 INFO:teuthology.orchestra.run.smithi118.stdout: "flags": 1, 2024-03-24T03:17:23.002 INFO:teuthology.orchestra.run.smithi118.stdout: "default_value": "info", 2024-03-24T03:17:23.002 INFO:teuthology.orchestra.run.smithi118.stdout: "min": "", 2024-03-24T03:17:23.002 INFO:teuthology.orchestra.run.smithi118.stdout: "max": "", 2024-03-24T03:17:23.002 INFO:teuthology.orchestra.run.smithi118.stdout: "enum_allowed": [ 2024-03-24T03:17:23.002 INFO:teuthology.orchestra.run.smithi118.stdout: "", 2024-03-24T03:17:23.002 INFO:teuthology.orchestra.run.smithi118.stdout: "critical", 2024-03-24T03:17:23.003 INFO:teuthology.orchestra.run.smithi118.stdout: "debug", 2024-03-24T03:17:23.003 INFO:teuthology.orchestra.run.smithi118.stdout: "error", 2024-03-24T03:17:23.003 INFO:teuthology.orchestra.run.smithi118.stdout: "info", 2024-03-24T03:17:23.003 INFO:teuthology.orchestra.run.smithi118.stdout: "warning" 2024-03-24T03:17:23.003 INFO:teuthology.orchestra.run.smithi118.stdout: ], 2024-03-24T03:17:23.003 INFO:teuthology.orchestra.run.smithi118.stdout: "desc": "", 2024-03-24T03:17:23.003 INFO:teuthology.orchestra.run.smithi118.stdout: "long_desc": "", 2024-03-24T03:17:23.003 INFO:teuthology.orchestra.run.smithi118.stdout: "tags": [], 2024-03-24T03:17:23.003 INFO:teuthology.orchestra.run.smithi118.stdout: "see_also": [] 2024-03-24T03:17:23.004 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-24T03:17:23.004 INFO:teuthology.orchestra.run.smithi118.stdout: "log_to_file": { 2024-03-24T03:17:23.004 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "log_to_file", 2024-03-24T03:17:23.004 INFO:teuthology.orchestra.run.smithi118.stdout: "type": "bool", 2024-03-24T03:17:23.004 INFO:teuthology.orchestra.run.smithi118.stdout: "level": "advanced", 2024-03-24T03:17:23.004 INFO:teuthology.orchestra.run.smithi118.stdout: "flags": 1, 2024-03-24T03:17:23.004 INFO:teuthology.orchestra.run.smithi118.stdout: "default_value": "False", 2024-03-24T03:17:23.004 INFO:teuthology.orchestra.run.smithi118.stdout: "min": "", 2024-03-24T03:17:23.004 INFO:teuthology.orchestra.run.smithi118.stdout: "max": "", 2024-03-24T03:17:23.005 INFO:teuthology.orchestra.run.smithi118.stdout: "enum_allowed": [], 2024-03-24T03:17:23.005 INFO:teuthology.orchestra.run.smithi118.stdout: "desc": "", 2024-03-24T03:17:23.005 INFO:teuthology.orchestra.run.smithi118.stdout: "long_desc": "", 2024-03-24T03:17:23.005 INFO:teuthology.orchestra.run.smithi118.stdout: "tags": [], 2024-03-24T03:17:23.005 INFO:teuthology.orchestra.run.smithi118.stdout: "see_also": [] 2024-03-24T03:17:23.005 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-24T03:17:23.005 INFO:teuthology.orchestra.run.smithi118.stdout: "storage_class": { 2024-03-24T03:17:23.005 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "storage_class", 2024-03-24T03:17:23.006 INFO:teuthology.orchestra.run.smithi118.stdout: "type": "str", 2024-03-24T03:17:23.006 INFO:teuthology.orchestra.run.smithi118.stdout: "level": "advanced", 2024-03-24T03:17:23.006 INFO:teuthology.orchestra.run.smithi118.stdout: "flags": 0, 2024-03-24T03:17:23.006 INFO:teuthology.orchestra.run.smithi118.stdout: "default_value": "local", 2024-03-24T03:17:23.006 INFO:teuthology.orchestra.run.smithi118.stdout: "min": "", 2024-03-24T03:17:23.006 INFO:teuthology.orchestra.run.smithi118.stdout: "max": "", 2024-03-24T03:17:23.006 INFO:teuthology.orchestra.run.smithi118.stdout: "enum_allowed": [], 2024-03-24T03:17:23.006 INFO:teuthology.orchestra.run.smithi118.stdout: "desc": "storage class name for LSO-discovered PVs", 2024-03-24T03:17:23.006 INFO:teuthology.orchestra.run.smithi118.stdout: "long_desc": "", 2024-03-24T03:17:23.009 INFO:teuthology.orchestra.run.smithi118.stdout: "tags": [], 2024-03-24T03:17:23.009 INFO:teuthology.orchestra.run.smithi118.stdout: "see_also": [] 2024-03-24T03:17:23.009 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-03-24T03:17:23.009 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-03-24T03:17:23.009 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-24T03:17:23.009 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-24T03:17:23.010 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "selftest", 2024-03-24T03:17:23.010 INFO:teuthology.orchestra.run.smithi118.stdout: "can_run": true, 2024-03-24T03:17:23.010 INFO:teuthology.orchestra.run.smithi118.stdout: "error_string": "", 2024-03-24T03:17:23.010 INFO:teuthology.orchestra.run.smithi118.stdout: "module_options": { 2024-03-24T03:17:23.010 INFO:teuthology.orchestra.run.smithi118.stdout: "log_level": { 2024-03-24T03:17:23.010 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "log_level", 2024-03-24T03:17:23.010 INFO:teuthology.orchestra.run.smithi118.stdout: "type": "str", 2024-03-24T03:17:23.010 INFO:teuthology.orchestra.run.smithi118.stdout: "level": "advanced", 2024-03-24T03:17:23.011 INFO:teuthology.orchestra.run.smithi118.stdout: "flags": 1, 2024-03-24T03:17:23.011 INFO:teuthology.orchestra.run.smithi118.stdout: "default_value": "", 2024-03-24T03:17:23.011 INFO:teuthology.orchestra.run.smithi118.stdout: "min": "", 2024-03-24T03:17:23.011 INFO:teuthology.orchestra.run.smithi118.stdout: "max": "", 2024-03-24T03:17:23.011 INFO:teuthology.orchestra.run.smithi118.stdout: "enum_allowed": [ 2024-03-24T03:17:23.011 INFO:teuthology.orchestra.run.smithi118.stdout: "", 2024-03-24T03:17:23.011 INFO:teuthology.orchestra.run.smithi118.stdout: "critical", 2024-03-24T03:17:23.011 INFO:teuthology.orchestra.run.smithi118.stdout: "debug", 2024-03-24T03:17:23.011 INFO:teuthology.orchestra.run.smithi118.stdout: "error", 2024-03-24T03:17:23.012 INFO:teuthology.orchestra.run.smithi118.stdout: "info", 2024-03-24T03:17:23.012 INFO:teuthology.orchestra.run.smithi118.stdout: "warning" 2024-03-24T03:17:23.012 INFO:teuthology.orchestra.run.smithi118.stdout: ], 2024-03-24T03:17:23.012 INFO:teuthology.orchestra.run.smithi118.stdout: "desc": "", 2024-03-24T03:17:23.012 INFO:teuthology.orchestra.run.smithi118.stdout: "long_desc": "", 2024-03-24T03:17:23.012 INFO:teuthology.orchestra.run.smithi118.stdout: "tags": [], 2024-03-24T03:17:23.012 INFO:teuthology.orchestra.run.smithi118.stdout: "see_also": [] 2024-03-24T03:17:23.012 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-24T03:17:23.012 INFO:teuthology.orchestra.run.smithi118.stdout: "log_to_cluster": { 2024-03-24T03:17:23.013 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "log_to_cluster", 2024-03-24T03:17:23.013 INFO:teuthology.orchestra.run.smithi118.stdout: "type": "bool", 2024-03-24T03:17:23.013 INFO:teuthology.orchestra.run.smithi118.stdout: "level": "advanced", 2024-03-24T03:17:23.013 INFO:teuthology.orchestra.run.smithi118.stdout: "flags": 1, 2024-03-24T03:17:23.013 INFO:teuthology.orchestra.run.smithi118.stdout: "default_value": "False", 2024-03-24T03:17:23.013 INFO:teuthology.orchestra.run.smithi118.stdout: "min": "", 2024-03-24T03:17:23.013 INFO:teuthology.orchestra.run.smithi118.stdout: "max": "", 2024-03-24T03:17:23.013 INFO:teuthology.orchestra.run.smithi118.stdout: "enum_allowed": [], 2024-03-24T03:17:23.014 INFO:teuthology.orchestra.run.smithi118.stdout: "desc": "", 2024-03-24T03:17:23.014 INFO:teuthology.orchestra.run.smithi118.stdout: "long_desc": "", 2024-03-24T03:17:23.014 INFO:teuthology.orchestra.run.smithi118.stdout: "tags": [], 2024-03-24T03:17:23.014 INFO:teuthology.orchestra.run.smithi118.stdout: "see_also": [] 2024-03-24T03:17:23.014 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-24T03:17:23.014 INFO:teuthology.orchestra.run.smithi118.stdout: "log_to_cluster_level": { 2024-03-24T03:17:23.014 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "log_to_cluster_level", 2024-03-24T03:17:23.015 INFO:teuthology.orchestra.run.smithi118.stdout: "type": "str", 2024-03-24T03:17:23.015 INFO:teuthology.orchestra.run.smithi118.stdout: "level": "advanced", 2024-03-24T03:17:23.015 INFO:teuthology.orchestra.run.smithi118.stdout: "flags": 1, 2024-03-24T03:17:23.015 INFO:teuthology.orchestra.run.smithi118.stdout: "default_value": "info", 2024-03-24T03:17:23.015 INFO:teuthology.orchestra.run.smithi118.stdout: "min": "", 2024-03-24T03:17:23.015 INFO:teuthology.orchestra.run.smithi118.stdout: "max": "", 2024-03-24T03:17:23.015 INFO:teuthology.orchestra.run.smithi118.stdout: "enum_allowed": [ 2024-03-24T03:17:23.015 INFO:teuthology.orchestra.run.smithi118.stdout: "", 2024-03-24T03:17:23.016 INFO:teuthology.orchestra.run.smithi118.stdout: "critical", 2024-03-24T03:17:23.016 INFO:teuthology.orchestra.run.smithi118.stdout: "debug", 2024-03-24T03:17:23.016 INFO:teuthology.orchestra.run.smithi118.stdout: "error", 2024-03-24T03:17:23.016 INFO:teuthology.orchestra.run.smithi118.stdout: "info", 2024-03-24T03:17:23.016 INFO:teuthology.orchestra.run.smithi118.stdout: "warning" 2024-03-24T03:17:23.016 INFO:teuthology.orchestra.run.smithi118.stdout: ], 2024-03-24T03:17:23.016 INFO:teuthology.orchestra.run.smithi118.stdout: "desc": "", 2024-03-24T03:17:23.016 INFO:teuthology.orchestra.run.smithi118.stdout: "long_desc": "", 2024-03-24T03:17:23.016 INFO:teuthology.orchestra.run.smithi118.stdout: "tags": [], 2024-03-24T03:17:23.016 INFO:teuthology.orchestra.run.smithi118.stdout: "see_also": [] 2024-03-24T03:17:23.017 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-24T03:17:23.017 INFO:teuthology.orchestra.run.smithi118.stdout: "log_to_file": { 2024-03-24T03:17:23.017 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "log_to_file", 2024-03-24T03:17:23.017 INFO:teuthology.orchestra.run.smithi118.stdout: "type": "bool", 2024-03-24T03:17:23.017 INFO:teuthology.orchestra.run.smithi118.stdout: "level": "advanced", 2024-03-24T03:17:23.017 INFO:teuthology.orchestra.run.smithi118.stdout: "flags": 1, 2024-03-24T03:17:23.017 INFO:teuthology.orchestra.run.smithi118.stdout: "default_value": "False", 2024-03-24T03:17:23.017 INFO:teuthology.orchestra.run.smithi118.stdout: "min": "", 2024-03-24T03:17:23.017 INFO:teuthology.orchestra.run.smithi118.stdout: "max": "", 2024-03-24T03:17:23.018 INFO:teuthology.orchestra.run.smithi118.stdout: "enum_allowed": [], 2024-03-24T03:17:23.018 INFO:teuthology.orchestra.run.smithi118.stdout: "desc": "", 2024-03-24T03:17:23.018 INFO:teuthology.orchestra.run.smithi118.stdout: "long_desc": "", 2024-03-24T03:17:23.018 INFO:teuthology.orchestra.run.smithi118.stdout: "tags": [], 2024-03-24T03:17:23.018 INFO:teuthology.orchestra.run.smithi118.stdout: "see_also": [] 2024-03-24T03:17:23.018 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-24T03:17:23.018 INFO:teuthology.orchestra.run.smithi118.stdout: "roption1": { 2024-03-24T03:17:23.018 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "roption1", 2024-03-24T03:17:23.019 INFO:teuthology.orchestra.run.smithi118.stdout: "type": "str", 2024-03-24T03:17:23.019 INFO:teuthology.orchestra.run.smithi118.stdout: "level": "advanced", 2024-03-24T03:17:23.019 INFO:teuthology.orchestra.run.smithi118.stdout: "flags": 0, 2024-03-24T03:17:23.019 INFO:teuthology.orchestra.run.smithi118.stdout: "default_value": "", 2024-03-24T03:17:23.019 INFO:teuthology.orchestra.run.smithi118.stdout: "min": "", 2024-03-24T03:17:23.019 INFO:teuthology.orchestra.run.smithi118.stdout: "max": "", 2024-03-24T03:17:23.019 INFO:teuthology.orchestra.run.smithi118.stdout: "enum_allowed": [], 2024-03-24T03:17:23.019 INFO:teuthology.orchestra.run.smithi118.stdout: "desc": "", 2024-03-24T03:17:23.020 INFO:teuthology.orchestra.run.smithi118.stdout: "long_desc": "", 2024-03-24T03:17:23.020 INFO:teuthology.orchestra.run.smithi118.stdout: "tags": [], 2024-03-24T03:17:23.020 INFO:teuthology.orchestra.run.smithi118.stdout: "see_also": [] 2024-03-24T03:17:23.020 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-24T03:17:23.020 INFO:teuthology.orchestra.run.smithi118.stdout: "roption2": { 2024-03-24T03:17:23.020 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "roption2", 2024-03-24T03:17:23.020 INFO:teuthology.orchestra.run.smithi118.stdout: "type": "str", 2024-03-24T03:17:23.020 INFO:teuthology.orchestra.run.smithi118.stdout: "level": "advanced", 2024-03-24T03:17:23.020 INFO:teuthology.orchestra.run.smithi118.stdout: "flags": 0, 2024-03-24T03:17:23.021 INFO:teuthology.orchestra.run.smithi118.stdout: "default_value": "xyz", 2024-03-24T03:17:23.021 INFO:teuthology.orchestra.run.smithi118.stdout: "min": "", 2024-03-24T03:17:23.021 INFO:teuthology.orchestra.run.smithi118.stdout: "max": "", 2024-03-24T03:17:23.021 INFO:teuthology.orchestra.run.smithi118.stdout: "enum_allowed": [], 2024-03-24T03:17:23.021 INFO:teuthology.orchestra.run.smithi118.stdout: "desc": "", 2024-03-24T03:17:23.021 INFO:teuthology.orchestra.run.smithi118.stdout: "long_desc": "", 2024-03-24T03:17:23.021 INFO:teuthology.orchestra.run.smithi118.stdout: "tags": [], 2024-03-24T03:17:23.021 INFO:teuthology.orchestra.run.smithi118.stdout: "see_also": [] 2024-03-24T03:17:23.021 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-24T03:17:23.022 INFO:teuthology.orchestra.run.smithi118.stdout: "rwoption1": { 2024-03-24T03:17:23.022 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "rwoption1", 2024-03-24T03:17:23.022 INFO:teuthology.orchestra.run.smithi118.stdout: "type": "str", 2024-03-24T03:17:23.022 INFO:teuthology.orchestra.run.smithi118.stdout: "level": "advanced", 2024-03-24T03:17:23.022 INFO:teuthology.orchestra.run.smithi118.stdout: "flags": 0, 2024-03-24T03:17:23.022 INFO:teuthology.orchestra.run.smithi118.stdout: "default_value": "", 2024-03-24T03:17:23.022 INFO:teuthology.orchestra.run.smithi118.stdout: "min": "", 2024-03-24T03:17:23.022 INFO:teuthology.orchestra.run.smithi118.stdout: "max": "", 2024-03-24T03:17:23.023 INFO:teuthology.orchestra.run.smithi118.stdout: "enum_allowed": [], 2024-03-24T03:17:23.023 INFO:teuthology.orchestra.run.smithi118.stdout: "desc": "", 2024-03-24T03:17:23.023 INFO:teuthology.orchestra.run.smithi118.stdout: "long_desc": "", 2024-03-24T03:17:23.023 INFO:teuthology.orchestra.run.smithi118.stdout: "tags": [], 2024-03-24T03:17:23.023 INFO:teuthology.orchestra.run.smithi118.stdout: "see_also": [] 2024-03-24T03:17:23.023 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-24T03:17:23.023 INFO:teuthology.orchestra.run.smithi118.stdout: "rwoption2": { 2024-03-24T03:17:23.023 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "rwoption2", 2024-03-24T03:17:23.023 INFO:teuthology.orchestra.run.smithi118.stdout: "type": "int", 2024-03-24T03:17:23.024 INFO:teuthology.orchestra.run.smithi118.stdout: "level": "advanced", 2024-03-24T03:17:23.024 INFO:teuthology.orchestra.run.smithi118.stdout: "flags": 0, 2024-03-24T03:17:23.024 INFO:teuthology.orchestra.run.smithi118.stdout: "default_value": "", 2024-03-24T03:17:23.024 INFO:teuthology.orchestra.run.smithi118.stdout: "min": "", 2024-03-24T03:17:23.024 INFO:teuthology.orchestra.run.smithi118.stdout: "max": "", 2024-03-24T03:17:23.024 INFO:teuthology.orchestra.run.smithi118.stdout: "enum_allowed": [], 2024-03-24T03:17:23.024 INFO:teuthology.orchestra.run.smithi118.stdout: "desc": "", 2024-03-24T03:17:23.025 INFO:teuthology.orchestra.run.smithi118.stdout: "long_desc": "", 2024-03-24T03:17:23.025 INFO:teuthology.orchestra.run.smithi118.stdout: "tags": [], 2024-03-24T03:17:23.025 INFO:teuthology.orchestra.run.smithi118.stdout: "see_also": [] 2024-03-24T03:17:23.025 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-24T03:17:23.025 INFO:teuthology.orchestra.run.smithi118.stdout: "rwoption3": { 2024-03-24T03:17:23.025 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "rwoption3", 2024-03-24T03:17:23.025 INFO:teuthology.orchestra.run.smithi118.stdout: "type": "float", 2024-03-24T03:17:23.025 INFO:teuthology.orchestra.run.smithi118.stdout: "level": "advanced", 2024-03-24T03:17:23.025 INFO:teuthology.orchestra.run.smithi118.stdout: "flags": 0, 2024-03-24T03:17:23.025 INFO:teuthology.orchestra.run.smithi118.stdout: "default_value": "", 2024-03-24T03:17:23.026 INFO:teuthology.orchestra.run.smithi118.stdout: "min": "", 2024-03-24T03:17:23.026 INFO:teuthology.orchestra.run.smithi118.stdout: "max": "", 2024-03-24T03:17:23.026 INFO:teuthology.orchestra.run.smithi118.stdout: "enum_allowed": [], 2024-03-24T03:17:23.026 INFO:teuthology.orchestra.run.smithi118.stdout: "desc": "", 2024-03-24T03:17:23.026 INFO:teuthology.orchestra.run.smithi118.stdout: "long_desc": "", 2024-03-24T03:17:23.026 INFO:teuthology.orchestra.run.smithi118.stdout: "tags": [], 2024-03-24T03:17:23.026 INFO:teuthology.orchestra.run.smithi118.stdout: "see_also": [] 2024-03-24T03:17:23.026 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-24T03:17:23.027 INFO:teuthology.orchestra.run.smithi118.stdout: "rwoption4": { 2024-03-24T03:17:23.027 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "rwoption4", 2024-03-24T03:17:23.027 INFO:teuthology.orchestra.run.smithi118.stdout: "type": "str", 2024-03-24T03:17:23.027 INFO:teuthology.orchestra.run.smithi118.stdout: "level": "advanced", 2024-03-24T03:17:23.027 INFO:teuthology.orchestra.run.smithi118.stdout: "flags": 0, 2024-03-24T03:17:23.027 INFO:teuthology.orchestra.run.smithi118.stdout: "default_value": "", 2024-03-24T03:17:23.027 INFO:teuthology.orchestra.run.smithi118.stdout: "min": "", 2024-03-24T03:17:23.027 INFO:teuthology.orchestra.run.smithi118.stdout: "max": "", 2024-03-24T03:17:23.027 INFO:teuthology.orchestra.run.smithi118.stdout: "enum_allowed": [], 2024-03-24T03:17:23.027 INFO:teuthology.orchestra.run.smithi118.stdout: "desc": "", 2024-03-24T03:17:23.028 INFO:teuthology.orchestra.run.smithi118.stdout: "long_desc": "", 2024-03-24T03:17:23.028 INFO:teuthology.orchestra.run.smithi118.stdout: "tags": [], 2024-03-24T03:17:23.028 INFO:teuthology.orchestra.run.smithi118.stdout: "see_also": [] 2024-03-24T03:17:23.028 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-24T03:17:23.028 INFO:teuthology.orchestra.run.smithi118.stdout: "rwoption5": { 2024-03-24T03:17:23.028 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "rwoption5", 2024-03-24T03:17:23.028 INFO:teuthology.orchestra.run.smithi118.stdout: "type": "bool", 2024-03-24T03:17:23.028 INFO:teuthology.orchestra.run.smithi118.stdout: "level": "advanced", 2024-03-24T03:17:23.028 INFO:teuthology.orchestra.run.smithi118.stdout: "flags": 0, 2024-03-24T03:17:23.029 INFO:teuthology.orchestra.run.smithi118.stdout: "default_value": "", 2024-03-24T03:17:23.029 INFO:teuthology.orchestra.run.smithi118.stdout: "min": "", 2024-03-24T03:17:23.029 INFO:teuthology.orchestra.run.smithi118.stdout: "max": "", 2024-03-24T03:17:23.029 INFO:teuthology.orchestra.run.smithi118.stdout: "enum_allowed": [], 2024-03-24T03:17:23.029 INFO:teuthology.orchestra.run.smithi118.stdout: "desc": "", 2024-03-24T03:17:23.029 INFO:teuthology.orchestra.run.smithi118.stdout: "long_desc": "", 2024-03-24T03:17:23.029 INFO:teuthology.orchestra.run.smithi118.stdout: "tags": [], 2024-03-24T03:17:23.029 INFO:teuthology.orchestra.run.smithi118.stdout: "see_also": [] 2024-03-24T03:17:23.029 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-24T03:17:23.029 INFO:teuthology.orchestra.run.smithi118.stdout: "rwoption6": { 2024-03-24T03:17:23.030 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "rwoption6", 2024-03-24T03:17:23.030 INFO:teuthology.orchestra.run.smithi118.stdout: "type": "bool", 2024-03-24T03:17:23.030 INFO:teuthology.orchestra.run.smithi118.stdout: "level": "advanced", 2024-03-24T03:17:23.030 INFO:teuthology.orchestra.run.smithi118.stdout: "flags": 0, 2024-03-24T03:17:23.030 INFO:teuthology.orchestra.run.smithi118.stdout: "default_value": "True", 2024-03-24T03:17:23.030 INFO:teuthology.orchestra.run.smithi118.stdout: "min": "", 2024-03-24T03:17:23.030 INFO:teuthology.orchestra.run.smithi118.stdout: "max": "", 2024-03-24T03:17:23.030 INFO:teuthology.orchestra.run.smithi118.stdout: "enum_allowed": [], 2024-03-24T03:17:23.031 INFO:teuthology.orchestra.run.smithi118.stdout: "desc": "", 2024-03-24T03:17:23.031 INFO:teuthology.orchestra.run.smithi118.stdout: "long_desc": "", 2024-03-24T03:17:23.031 INFO:teuthology.orchestra.run.smithi118.stdout: "tags": [], 2024-03-24T03:17:23.031 INFO:teuthology.orchestra.run.smithi118.stdout: "see_also": [] 2024-03-24T03:17:23.031 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-24T03:17:23.031 INFO:teuthology.orchestra.run.smithi118.stdout: "rwoption7": { 2024-03-24T03:17:23.031 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "rwoption7", 2024-03-24T03:17:23.031 INFO:teuthology.orchestra.run.smithi118.stdout: "type": "int", 2024-03-24T03:17:23.032 INFO:teuthology.orchestra.run.smithi118.stdout: "level": "advanced", 2024-03-24T03:17:23.032 INFO:teuthology.orchestra.run.smithi118.stdout: "flags": 0, 2024-03-24T03:17:23.032 INFO:teuthology.orchestra.run.smithi118.stdout: "default_value": "", 2024-03-24T03:17:23.032 INFO:teuthology.orchestra.run.smithi118.stdout: "min": "1", 2024-03-24T03:17:23.032 INFO:teuthology.orchestra.run.smithi118.stdout: "max": "42", 2024-03-24T03:17:23.032 INFO:teuthology.orchestra.run.smithi118.stdout: "enum_allowed": [], 2024-03-24T03:17:23.032 INFO:teuthology.orchestra.run.smithi118.stdout: "desc": "", 2024-03-24T03:17:23.033 INFO:teuthology.orchestra.run.smithi118.stdout: "long_desc": "", 2024-03-24T03:17:23.033 INFO:teuthology.orchestra.run.smithi118.stdout: "tags": [], 2024-03-24T03:17:23.033 INFO:teuthology.orchestra.run.smithi118.stdout: "see_also": [] 2024-03-24T03:17:23.033 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-24T03:17:23.033 INFO:teuthology.orchestra.run.smithi118.stdout: "testkey": { 2024-03-24T03:17:23.033 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "testkey", 2024-03-24T03:17:23.033 INFO:teuthology.orchestra.run.smithi118.stdout: "type": "str", 2024-03-24T03:17:23.033 INFO:teuthology.orchestra.run.smithi118.stdout: "level": "advanced", 2024-03-24T03:17:23.033 INFO:teuthology.orchestra.run.smithi118.stdout: "flags": 0, 2024-03-24T03:17:23.034 INFO:teuthology.orchestra.run.smithi118.stdout: "default_value": "", 2024-03-24T03:17:23.034 INFO:teuthology.orchestra.run.smithi118.stdout: "min": "", 2024-03-24T03:17:23.034 INFO:teuthology.orchestra.run.smithi118.stdout: "max": "", 2024-03-24T03:17:23.034 INFO:teuthology.orchestra.run.smithi118.stdout: "enum_allowed": [], 2024-03-24T03:17:23.034 INFO:teuthology.orchestra.run.smithi118.stdout: "desc": "", 2024-03-24T03:17:23.034 INFO:teuthology.orchestra.run.smithi118.stdout: "long_desc": "", 2024-03-24T03:17:23.034 INFO:teuthology.orchestra.run.smithi118.stdout: "tags": [], 2024-03-24T03:17:23.034 INFO:teuthology.orchestra.run.smithi118.stdout: "see_also": [] 2024-03-24T03:17:23.034 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-24T03:17:23.035 INFO:teuthology.orchestra.run.smithi118.stdout: "testlkey": { 2024-03-24T03:17:23.035 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "testlkey", 2024-03-24T03:17:23.035 INFO:teuthology.orchestra.run.smithi118.stdout: "type": "str", 2024-03-24T03:17:23.035 INFO:teuthology.orchestra.run.smithi118.stdout: "level": "advanced", 2024-03-24T03:17:23.035 INFO:teuthology.orchestra.run.smithi118.stdout: "flags": 0, 2024-03-24T03:17:23.035 INFO:teuthology.orchestra.run.smithi118.stdout: "default_value": "", 2024-03-24T03:17:23.035 INFO:teuthology.orchestra.run.smithi118.stdout: "min": "", 2024-03-24T03:17:23.035 INFO:teuthology.orchestra.run.smithi118.stdout: "max": "", 2024-03-24T03:17:23.035 INFO:teuthology.orchestra.run.smithi118.stdout: "enum_allowed": [], 2024-03-24T03:17:23.036 INFO:teuthology.orchestra.run.smithi118.stdout: "desc": "", 2024-03-24T03:17:23.036 INFO:teuthology.orchestra.run.smithi118.stdout: "long_desc": "", 2024-03-24T03:17:23.036 INFO:teuthology.orchestra.run.smithi118.stdout: "tags": [], 2024-03-24T03:17:23.036 INFO:teuthology.orchestra.run.smithi118.stdout: "see_also": [] 2024-03-24T03:17:23.036 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-24T03:17:23.036 INFO:teuthology.orchestra.run.smithi118.stdout: "testnewline": { 2024-03-24T03:17:23.036 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "testnewline", 2024-03-24T03:17:23.036 INFO:teuthology.orchestra.run.smithi118.stdout: "type": "str", 2024-03-24T03:17:23.037 INFO:teuthology.orchestra.run.smithi118.stdout: "level": "advanced", 2024-03-24T03:17:23.037 INFO:teuthology.orchestra.run.smithi118.stdout: "flags": 0, 2024-03-24T03:17:23.037 INFO:teuthology.orchestra.run.smithi118.stdout: "default_value": "", 2024-03-24T03:17:23.037 INFO:teuthology.orchestra.run.smithi118.stdout: "min": "", 2024-03-24T03:17:23.037 INFO:teuthology.orchestra.run.smithi118.stdout: "max": "", 2024-03-24T03:17:23.037 INFO:teuthology.orchestra.run.smithi118.stdout: "enum_allowed": [], 2024-03-24T03:17:23.037 INFO:teuthology.orchestra.run.smithi118.stdout: "desc": "", 2024-03-24T03:17:23.038 INFO:teuthology.orchestra.run.smithi118.stdout: "long_desc": "", 2024-03-24T03:17:23.038 INFO:teuthology.orchestra.run.smithi118.stdout: "tags": [], 2024-03-24T03:17:23.038 INFO:teuthology.orchestra.run.smithi118.stdout: "see_also": [] 2024-03-24T03:17:23.038 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-03-24T03:17:23.038 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-03-24T03:17:23.038 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-24T03:17:23.038 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-24T03:17:23.038 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "snap_schedule", 2024-03-24T03:17:23.038 INFO:teuthology.orchestra.run.smithi118.stdout: "can_run": true, 2024-03-24T03:17:23.039 INFO:teuthology.orchestra.run.smithi118.stdout: "error_string": "", 2024-03-24T03:17:23.039 INFO:teuthology.orchestra.run.smithi118.stdout: "module_options": { 2024-03-24T03:17:23.039 INFO:teuthology.orchestra.run.smithi118.stdout: "allow_m_granularity": { 2024-03-24T03:17:23.039 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "allow_m_granularity", 2024-03-24T03:17:23.039 INFO:teuthology.orchestra.run.smithi118.stdout: "type": "bool", 2024-03-24T03:17:23.039 INFO:teuthology.orchestra.run.smithi118.stdout: "level": "advanced", 2024-03-24T03:17:23.039 INFO:teuthology.orchestra.run.smithi118.stdout: "flags": 1, 2024-03-24T03:17:23.039 INFO:teuthology.orchestra.run.smithi118.stdout: "default_value": "False", 2024-03-24T03:17:23.039 INFO:teuthology.orchestra.run.smithi118.stdout: "min": "", 2024-03-24T03:17:23.040 INFO:teuthology.orchestra.run.smithi118.stdout: "max": "", 2024-03-24T03:17:23.040 INFO:teuthology.orchestra.run.smithi118.stdout: "enum_allowed": [], 2024-03-24T03:17:23.040 INFO:teuthology.orchestra.run.smithi118.stdout: "desc": "allow minute scheduled snapshots", 2024-03-24T03:17:23.040 INFO:teuthology.orchestra.run.smithi118.stdout: "long_desc": "", 2024-03-24T03:17:23.040 INFO:teuthology.orchestra.run.smithi118.stdout: "tags": [], 2024-03-24T03:17:23.040 INFO:teuthology.orchestra.run.smithi118.stdout: "see_also": [] 2024-03-24T03:17:23.040 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-24T03:17:23.040 INFO:teuthology.orchestra.run.smithi118.stdout: "dump_on_update": { 2024-03-24T03:17:23.040 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "dump_on_update", 2024-03-24T03:17:23.041 INFO:teuthology.orchestra.run.smithi118.stdout: "type": "bool", 2024-03-24T03:17:23.041 INFO:teuthology.orchestra.run.smithi118.stdout: "level": "advanced", 2024-03-24T03:17:23.041 INFO:teuthology.orchestra.run.smithi118.stdout: "flags": 1, 2024-03-24T03:17:23.041 INFO:teuthology.orchestra.run.smithi118.stdout: "default_value": "False", 2024-03-24T03:17:23.041 INFO:teuthology.orchestra.run.smithi118.stdout: "min": "", 2024-03-24T03:17:23.041 INFO:teuthology.orchestra.run.smithi118.stdout: "max": "", 2024-03-24T03:17:23.041 INFO:teuthology.orchestra.run.smithi118.stdout: "enum_allowed": [], 2024-03-24T03:17:23.041 INFO:teuthology.orchestra.run.smithi118.stdout: "desc": "dump database to debug log on update", 2024-03-24T03:17:23.042 INFO:teuthology.orchestra.run.smithi118.stdout: "long_desc": "", 2024-03-24T03:17:23.042 INFO:teuthology.orchestra.run.smithi118.stdout: "tags": [], 2024-03-24T03:17:23.042 INFO:teuthology.orchestra.run.smithi118.stdout: "see_also": [] 2024-03-24T03:17:23.042 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-24T03:17:23.042 INFO:teuthology.orchestra.run.smithi118.stdout: "log_level": { 2024-03-24T03:17:23.042 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "log_level", 2024-03-24T03:17:23.042 INFO:teuthology.orchestra.run.smithi118.stdout: "type": "str", 2024-03-24T03:17:23.042 INFO:teuthology.orchestra.run.smithi118.stdout: "level": "advanced", 2024-03-24T03:17:23.043 INFO:teuthology.orchestra.run.smithi118.stdout: "flags": 1, 2024-03-24T03:17:23.043 INFO:teuthology.orchestra.run.smithi118.stdout: "default_value": "", 2024-03-24T03:17:23.043 INFO:teuthology.orchestra.run.smithi118.stdout: "min": "", 2024-03-24T03:17:23.043 INFO:teuthology.orchestra.run.smithi118.stdout: "max": "", 2024-03-24T03:17:23.043 INFO:teuthology.orchestra.run.smithi118.stdout: "enum_allowed": [ 2024-03-24T03:17:23.043 INFO:teuthology.orchestra.run.smithi118.stdout: "", 2024-03-24T03:17:23.043 INFO:teuthology.orchestra.run.smithi118.stdout: "critical", 2024-03-24T03:17:23.043 INFO:teuthology.orchestra.run.smithi118.stdout: "debug", 2024-03-24T03:17:23.043 INFO:teuthology.orchestra.run.smithi118.stdout: "error", 2024-03-24T03:17:23.043 INFO:teuthology.orchestra.run.smithi118.stdout: "info", 2024-03-24T03:17:23.044 INFO:teuthology.orchestra.run.smithi118.stdout: "warning" 2024-03-24T03:17:23.044 INFO:teuthology.orchestra.run.smithi118.stdout: ], 2024-03-24T03:17:23.044 INFO:teuthology.orchestra.run.smithi118.stdout: "desc": "", 2024-03-24T03:17:23.044 INFO:teuthology.orchestra.run.smithi118.stdout: "long_desc": "", 2024-03-24T03:17:23.044 INFO:teuthology.orchestra.run.smithi118.stdout: "tags": [], 2024-03-24T03:17:23.044 INFO:teuthology.orchestra.run.smithi118.stdout: "see_also": [] 2024-03-24T03:17:23.044 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-24T03:17:23.044 INFO:teuthology.orchestra.run.smithi118.stdout: "log_to_cluster": { 2024-03-24T03:17:23.045 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "log_to_cluster", 2024-03-24T03:17:23.045 INFO:teuthology.orchestra.run.smithi118.stdout: "type": "bool", 2024-03-24T03:17:23.045 INFO:teuthology.orchestra.run.smithi118.stdout: "level": "advanced", 2024-03-24T03:17:23.045 INFO:teuthology.orchestra.run.smithi118.stdout: "flags": 1, 2024-03-24T03:17:23.045 INFO:teuthology.orchestra.run.smithi118.stdout: "default_value": "False", 2024-03-24T03:17:23.045 INFO:teuthology.orchestra.run.smithi118.stdout: "min": "", 2024-03-24T03:17:23.045 INFO:teuthology.orchestra.run.smithi118.stdout: "max": "", 2024-03-24T03:17:23.045 INFO:teuthology.orchestra.run.smithi118.stdout: "enum_allowed": [], 2024-03-24T03:17:23.045 INFO:teuthology.orchestra.run.smithi118.stdout: "desc": "", 2024-03-24T03:17:23.046 INFO:teuthology.orchestra.run.smithi118.stdout: "long_desc": "", 2024-03-24T03:17:23.046 INFO:teuthology.orchestra.run.smithi118.stdout: "tags": [], 2024-03-24T03:17:23.046 INFO:teuthology.orchestra.run.smithi118.stdout: "see_also": [] 2024-03-24T03:17:23.046 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-24T03:17:23.046 INFO:teuthology.orchestra.run.smithi118.stdout: "log_to_cluster_level": { 2024-03-24T03:17:23.046 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "log_to_cluster_level", 2024-03-24T03:17:23.046 INFO:teuthology.orchestra.run.smithi118.stdout: "type": "str", 2024-03-24T03:17:23.046 INFO:teuthology.orchestra.run.smithi118.stdout: "level": "advanced", 2024-03-24T03:17:23.046 INFO:teuthology.orchestra.run.smithi118.stdout: "flags": 1, 2024-03-24T03:17:23.047 INFO:teuthology.orchestra.run.smithi118.stdout: "default_value": "info", 2024-03-24T03:17:23.047 INFO:teuthology.orchestra.run.smithi118.stdout: "min": "", 2024-03-24T03:17:23.047 INFO:teuthology.orchestra.run.smithi118.stdout: "max": "", 2024-03-24T03:17:23.047 INFO:teuthology.orchestra.run.smithi118.stdout: "enum_allowed": [ 2024-03-24T03:17:23.047 INFO:teuthology.orchestra.run.smithi118.stdout: "", 2024-03-24T03:17:23.047 INFO:teuthology.orchestra.run.smithi118.stdout: "critical", 2024-03-24T03:17:23.047 INFO:teuthology.orchestra.run.smithi118.stdout: "debug", 2024-03-24T03:17:23.047 INFO:teuthology.orchestra.run.smithi118.stdout: "error", 2024-03-24T03:17:23.048 INFO:teuthology.orchestra.run.smithi118.stdout: "info", 2024-03-24T03:17:23.048 INFO:teuthology.orchestra.run.smithi118.stdout: "warning" 2024-03-24T03:17:23.048 INFO:teuthology.orchestra.run.smithi118.stdout: ], 2024-03-24T03:17:23.048 INFO:teuthology.orchestra.run.smithi118.stdout: "desc": "", 2024-03-24T03:17:23.048 INFO:teuthology.orchestra.run.smithi118.stdout: "long_desc": "", 2024-03-24T03:17:23.048 INFO:teuthology.orchestra.run.smithi118.stdout: "tags": [], 2024-03-24T03:17:23.048 INFO:teuthology.orchestra.run.smithi118.stdout: "see_also": [] 2024-03-24T03:17:23.048 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-24T03:17:23.048 INFO:teuthology.orchestra.run.smithi118.stdout: "log_to_file": { 2024-03-24T03:17:23.049 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "log_to_file", 2024-03-24T03:17:23.049 INFO:teuthology.orchestra.run.smithi118.stdout: "type": "bool", 2024-03-24T03:17:23.049 INFO:teuthology.orchestra.run.smithi118.stdout: "level": "advanced", 2024-03-24T03:17:23.049 INFO:teuthology.orchestra.run.smithi118.stdout: "flags": 1, 2024-03-24T03:17:23.049 INFO:teuthology.orchestra.run.smithi118.stdout: "default_value": "False", 2024-03-24T03:17:23.049 INFO:teuthology.orchestra.run.smithi118.stdout: "min": "", 2024-03-24T03:17:23.049 INFO:teuthology.orchestra.run.smithi118.stdout: "max": "", 2024-03-24T03:17:23.049 INFO:teuthology.orchestra.run.smithi118.stdout: "enum_allowed": [], 2024-03-24T03:17:23.049 INFO:teuthology.orchestra.run.smithi118.stdout: "desc": "", 2024-03-24T03:17:23.050 INFO:teuthology.orchestra.run.smithi118.stdout: "long_desc": "", 2024-03-24T03:17:23.050 INFO:teuthology.orchestra.run.smithi118.stdout: "tags": [], 2024-03-24T03:17:23.050 INFO:teuthology.orchestra.run.smithi118.stdout: "see_also": [] 2024-03-24T03:17:23.050 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-03-24T03:17:23.050 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-03-24T03:17:23.050 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-24T03:17:23.050 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-24T03:17:23.050 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "stats", 2024-03-24T03:17:23.050 INFO:teuthology.orchestra.run.smithi118.stdout: "can_run": true, 2024-03-24T03:17:23.051 INFO:teuthology.orchestra.run.smithi118.stdout: "error_string": "", 2024-03-24T03:17:23.051 INFO:teuthology.orchestra.run.smithi118.stdout: "module_options": { 2024-03-24T03:17:23.051 INFO:teuthology.orchestra.run.smithi118.stdout: "log_level": { 2024-03-24T03:17:23.051 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "log_level", 2024-03-24T03:17:23.051 INFO:teuthology.orchestra.run.smithi118.stdout: "type": "str", 2024-03-24T03:17:23.051 INFO:teuthology.orchestra.run.smithi118.stdout: "level": "advanced", 2024-03-24T03:17:23.051 INFO:teuthology.orchestra.run.smithi118.stdout: "flags": 1, 2024-03-24T03:17:23.051 INFO:teuthology.orchestra.run.smithi118.stdout: "default_value": "", 2024-03-24T03:17:23.052 INFO:teuthology.orchestra.run.smithi118.stdout: "min": "", 2024-03-24T03:17:23.052 INFO:teuthology.orchestra.run.smithi118.stdout: "max": "", 2024-03-24T03:17:23.052 INFO:teuthology.orchestra.run.smithi118.stdout: "enum_allowed": [ 2024-03-24T03:17:23.052 INFO:teuthology.orchestra.run.smithi118.stdout: "", 2024-03-24T03:17:23.052 INFO:teuthology.orchestra.run.smithi118.stdout: "critical", 2024-03-24T03:17:23.052 INFO:teuthology.orchestra.run.smithi118.stdout: "debug", 2024-03-24T03:17:23.052 INFO:teuthology.orchestra.run.smithi118.stdout: "error", 2024-03-24T03:17:23.052 INFO:teuthology.orchestra.run.smithi118.stdout: "info", 2024-03-24T03:17:23.052 INFO:teuthology.orchestra.run.smithi118.stdout: "warning" 2024-03-24T03:17:23.053 INFO:teuthology.orchestra.run.smithi118.stdout: ], 2024-03-24T03:17:23.053 INFO:teuthology.orchestra.run.smithi118.stdout: "desc": "", 2024-03-24T03:17:23.053 INFO:teuthology.orchestra.run.smithi118.stdout: "long_desc": "", 2024-03-24T03:17:23.053 INFO:teuthology.orchestra.run.smithi118.stdout: "tags": [], 2024-03-24T03:17:23.053 INFO:teuthology.orchestra.run.smithi118.stdout: "see_also": [] 2024-03-24T03:17:23.053 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-24T03:17:23.053 INFO:teuthology.orchestra.run.smithi118.stdout: "log_to_cluster": { 2024-03-24T03:17:23.053 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "log_to_cluster", 2024-03-24T03:17:23.053 INFO:teuthology.orchestra.run.smithi118.stdout: "type": "bool", 2024-03-24T03:17:23.054 INFO:teuthology.orchestra.run.smithi118.stdout: "level": "advanced", 2024-03-24T03:17:23.054 INFO:teuthology.orchestra.run.smithi118.stdout: "flags": 1, 2024-03-24T03:17:23.054 INFO:teuthology.orchestra.run.smithi118.stdout: "default_value": "False", 2024-03-24T03:17:23.054 INFO:teuthology.orchestra.run.smithi118.stdout: "min": "", 2024-03-24T03:17:23.054 INFO:teuthology.orchestra.run.smithi118.stdout: "max": "", 2024-03-24T03:17:23.054 INFO:teuthology.orchestra.run.smithi118.stdout: "enum_allowed": [], 2024-03-24T03:17:23.055 INFO:teuthology.orchestra.run.smithi118.stdout: "desc": "", 2024-03-24T03:17:23.055 INFO:teuthology.orchestra.run.smithi118.stdout: "long_desc": "", 2024-03-24T03:17:23.055 INFO:teuthology.orchestra.run.smithi118.stdout: "tags": [], 2024-03-24T03:17:23.055 INFO:teuthology.orchestra.run.smithi118.stdout: "see_also": [] 2024-03-24T03:17:23.055 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-24T03:17:23.055 INFO:teuthology.orchestra.run.smithi118.stdout: "log_to_cluster_level": { 2024-03-24T03:17:23.055 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "log_to_cluster_level", 2024-03-24T03:17:23.055 INFO:teuthology.orchestra.run.smithi118.stdout: "type": "str", 2024-03-24T03:17:23.055 INFO:teuthology.orchestra.run.smithi118.stdout: "level": "advanced", 2024-03-24T03:17:23.056 INFO:teuthology.orchestra.run.smithi118.stdout: "flags": 1, 2024-03-24T03:17:23.056 INFO:teuthology.orchestra.run.smithi118.stdout: "default_value": "info", 2024-03-24T03:17:23.056 INFO:teuthology.orchestra.run.smithi118.stdout: "min": "", 2024-03-24T03:17:23.056 INFO:teuthology.orchestra.run.smithi118.stdout: "max": "", 2024-03-24T03:17:23.056 INFO:teuthology.orchestra.run.smithi118.stdout: "enum_allowed": [ 2024-03-24T03:17:23.056 INFO:teuthology.orchestra.run.smithi118.stdout: "", 2024-03-24T03:17:23.056 INFO:teuthology.orchestra.run.smithi118.stdout: "critical", 2024-03-24T03:17:23.056 INFO:teuthology.orchestra.run.smithi118.stdout: "debug", 2024-03-24T03:17:23.056 INFO:teuthology.orchestra.run.smithi118.stdout: "error", 2024-03-24T03:17:23.057 INFO:teuthology.orchestra.run.smithi118.stdout: "info", 2024-03-24T03:17:23.057 INFO:teuthology.orchestra.run.smithi118.stdout: "warning" 2024-03-24T03:17:23.057 INFO:teuthology.orchestra.run.smithi118.stdout: ], 2024-03-24T03:17:23.057 INFO:teuthology.orchestra.run.smithi118.stdout: "desc": "", 2024-03-24T03:17:23.057 INFO:teuthology.orchestra.run.smithi118.stdout: "long_desc": "", 2024-03-24T03:17:23.057 INFO:teuthology.orchestra.run.smithi118.stdout: "tags": [], 2024-03-24T03:17:23.057 INFO:teuthology.orchestra.run.smithi118.stdout: "see_also": [] 2024-03-24T03:17:23.057 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-24T03:17:23.057 INFO:teuthology.orchestra.run.smithi118.stdout: "log_to_file": { 2024-03-24T03:17:23.058 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "log_to_file", 2024-03-24T03:17:23.058 INFO:teuthology.orchestra.run.smithi118.stdout: "type": "bool", 2024-03-24T03:17:23.058 INFO:teuthology.orchestra.run.smithi118.stdout: "level": "advanced", 2024-03-24T03:17:23.058 INFO:teuthology.orchestra.run.smithi118.stdout: "flags": 1, 2024-03-24T03:17:23.058 INFO:teuthology.orchestra.run.smithi118.stdout: "default_value": "False", 2024-03-24T03:17:23.058 INFO:teuthology.orchestra.run.smithi118.stdout: "min": "", 2024-03-24T03:17:23.058 INFO:teuthology.orchestra.run.smithi118.stdout: "max": "", 2024-03-24T03:17:23.058 INFO:teuthology.orchestra.run.smithi118.stdout: "enum_allowed": [], 2024-03-24T03:17:23.059 INFO:teuthology.orchestra.run.smithi118.stdout: "desc": "", 2024-03-24T03:17:23.059 INFO:teuthology.orchestra.run.smithi118.stdout: "long_desc": "", 2024-03-24T03:17:23.059 INFO:teuthology.orchestra.run.smithi118.stdout: "tags": [], 2024-03-24T03:17:23.059 INFO:teuthology.orchestra.run.smithi118.stdout: "see_also": [] 2024-03-24T03:17:23.059 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-03-24T03:17:23.059 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-03-24T03:17:23.059 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-24T03:17:23.059 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-24T03:17:23.060 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "status", 2024-03-24T03:17:23.060 INFO:teuthology.orchestra.run.smithi118.stdout: "can_run": true, 2024-03-24T03:17:23.060 INFO:teuthology.orchestra.run.smithi118.stdout: "error_string": "", 2024-03-24T03:17:23.060 INFO:teuthology.orchestra.run.smithi118.stdout: "module_options": { 2024-03-24T03:17:23.060 INFO:teuthology.orchestra.run.smithi118.stdout: "log_level": { 2024-03-24T03:17:23.060 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "log_level", 2024-03-24T03:17:23.060 INFO:teuthology.orchestra.run.smithi118.stdout: "type": "str", 2024-03-24T03:17:23.060 INFO:teuthology.orchestra.run.smithi118.stdout: "level": "advanced", 2024-03-24T03:17:23.060 INFO:teuthology.orchestra.run.smithi118.stdout: "flags": 1, 2024-03-24T03:17:23.060 INFO:teuthology.orchestra.run.smithi118.stdout: "default_value": "", 2024-03-24T03:17:23.061 INFO:teuthology.orchestra.run.smithi118.stdout: "min": "", 2024-03-24T03:17:23.061 INFO:teuthology.orchestra.run.smithi118.stdout: "max": "", 2024-03-24T03:17:23.061 INFO:teuthology.orchestra.run.smithi118.stdout: "enum_allowed": [ 2024-03-24T03:17:23.061 INFO:teuthology.orchestra.run.smithi118.stdout: "", 2024-03-24T03:17:23.061 INFO:teuthology.orchestra.run.smithi118.stdout: "critical", 2024-03-24T03:17:23.061 INFO:teuthology.orchestra.run.smithi118.stdout: "debug", 2024-03-24T03:17:23.061 INFO:teuthology.orchestra.run.smithi118.stdout: "error", 2024-03-24T03:17:23.062 INFO:teuthology.orchestra.run.smithi118.stdout: "info", 2024-03-24T03:17:23.062 INFO:teuthology.orchestra.run.smithi118.stdout: "warning" 2024-03-24T03:17:23.062 INFO:teuthology.orchestra.run.smithi118.stdout: ], 2024-03-24T03:17:23.062 INFO:teuthology.orchestra.run.smithi118.stdout: "desc": "", 2024-03-24T03:17:23.062 INFO:teuthology.orchestra.run.smithi118.stdout: "long_desc": "", 2024-03-24T03:17:23.062 INFO:teuthology.orchestra.run.smithi118.stdout: "tags": [], 2024-03-24T03:17:23.062 INFO:teuthology.orchestra.run.smithi118.stdout: "see_also": [] 2024-03-24T03:17:23.062 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-24T03:17:23.062 INFO:teuthology.orchestra.run.smithi118.stdout: "log_to_cluster": { 2024-03-24T03:17:23.063 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "log_to_cluster", 2024-03-24T03:17:23.063 INFO:teuthology.orchestra.run.smithi118.stdout: "type": "bool", 2024-03-24T03:17:23.063 INFO:teuthology.orchestra.run.smithi118.stdout: "level": "advanced", 2024-03-24T03:17:23.063 INFO:teuthology.orchestra.run.smithi118.stdout: "flags": 1, 2024-03-24T03:17:23.063 INFO:teuthology.orchestra.run.smithi118.stdout: "default_value": "False", 2024-03-24T03:17:23.063 INFO:teuthology.orchestra.run.smithi118.stdout: "min": "", 2024-03-24T03:17:23.063 INFO:teuthology.orchestra.run.smithi118.stdout: "max": "", 2024-03-24T03:17:23.063 INFO:teuthology.orchestra.run.smithi118.stdout: "enum_allowed": [], 2024-03-24T03:17:23.063 INFO:teuthology.orchestra.run.smithi118.stdout: "desc": "", 2024-03-24T03:17:23.064 INFO:teuthology.orchestra.run.smithi118.stdout: "long_desc": "", 2024-03-24T03:17:23.064 INFO:teuthology.orchestra.run.smithi118.stdout: "tags": [], 2024-03-24T03:17:23.064 INFO:teuthology.orchestra.run.smithi118.stdout: "see_also": [] 2024-03-24T03:17:23.064 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-24T03:17:23.064 INFO:teuthology.orchestra.run.smithi118.stdout: "log_to_cluster_level": { 2024-03-24T03:17:23.064 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "log_to_cluster_level", 2024-03-24T03:17:23.064 INFO:teuthology.orchestra.run.smithi118.stdout: "type": "str", 2024-03-24T03:17:23.064 INFO:teuthology.orchestra.run.smithi118.stdout: "level": "advanced", 2024-03-24T03:17:23.065 INFO:teuthology.orchestra.run.smithi118.stdout: "flags": 1, 2024-03-24T03:17:23.065 INFO:teuthology.orchestra.run.smithi118.stdout: "default_value": "info", 2024-03-24T03:17:23.065 INFO:teuthology.orchestra.run.smithi118.stdout: "min": "", 2024-03-24T03:17:23.065 INFO:teuthology.orchestra.run.smithi118.stdout: "max": "", 2024-03-24T03:17:23.065 INFO:teuthology.orchestra.run.smithi118.stdout: "enum_allowed": [ 2024-03-24T03:17:23.065 INFO:teuthology.orchestra.run.smithi118.stdout: "", 2024-03-24T03:17:23.065 INFO:teuthology.orchestra.run.smithi118.stdout: "critical", 2024-03-24T03:17:23.065 INFO:teuthology.orchestra.run.smithi118.stdout: "debug", 2024-03-24T03:17:23.065 INFO:teuthology.orchestra.run.smithi118.stdout: "error", 2024-03-24T03:17:23.066 INFO:teuthology.orchestra.run.smithi118.stdout: "info", 2024-03-24T03:17:23.066 INFO:teuthology.orchestra.run.smithi118.stdout: "warning" 2024-03-24T03:17:23.066 INFO:teuthology.orchestra.run.smithi118.stdout: ], 2024-03-24T03:17:23.066 INFO:teuthology.orchestra.run.smithi118.stdout: "desc": "", 2024-03-24T03:17:23.066 INFO:teuthology.orchestra.run.smithi118.stdout: "long_desc": "", 2024-03-24T03:17:23.066 INFO:teuthology.orchestra.run.smithi118.stdout: "tags": [], 2024-03-24T03:17:23.066 INFO:teuthology.orchestra.run.smithi118.stdout: "see_also": [] 2024-03-24T03:17:23.066 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-24T03:17:23.066 INFO:teuthology.orchestra.run.smithi118.stdout: "log_to_file": { 2024-03-24T03:17:23.067 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "log_to_file", 2024-03-24T03:17:23.067 INFO:teuthology.orchestra.run.smithi118.stdout: "type": "bool", 2024-03-24T03:17:23.067 INFO:teuthology.orchestra.run.smithi118.stdout: "level": "advanced", 2024-03-24T03:17:23.067 INFO:teuthology.orchestra.run.smithi118.stdout: "flags": 1, 2024-03-24T03:17:23.067 INFO:teuthology.orchestra.run.smithi118.stdout: "default_value": "False", 2024-03-24T03:17:23.067 INFO:teuthology.orchestra.run.smithi118.stdout: "min": "", 2024-03-24T03:17:23.067 INFO:teuthology.orchestra.run.smithi118.stdout: "max": "", 2024-03-24T03:17:23.067 INFO:teuthology.orchestra.run.smithi118.stdout: "enum_allowed": [], 2024-03-24T03:17:23.067 INFO:teuthology.orchestra.run.smithi118.stdout: "desc": "", 2024-03-24T03:17:23.067 INFO:teuthology.orchestra.run.smithi118.stdout: "long_desc": "", 2024-03-24T03:17:23.068 INFO:teuthology.orchestra.run.smithi118.stdout: "tags": [], 2024-03-24T03:17:23.068 INFO:teuthology.orchestra.run.smithi118.stdout: "see_also": [] 2024-03-24T03:17:23.068 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-03-24T03:17:23.068 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-03-24T03:17:23.068 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-24T03:17:23.068 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-24T03:17:23.068 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "telegraf", 2024-03-24T03:17:23.068 INFO:teuthology.orchestra.run.smithi118.stdout: "can_run": true, 2024-03-24T03:17:23.068 INFO:teuthology.orchestra.run.smithi118.stdout: "error_string": "", 2024-03-24T03:17:23.069 INFO:teuthology.orchestra.run.smithi118.stdout: "module_options": { 2024-03-24T03:17:23.069 INFO:teuthology.orchestra.run.smithi118.stdout: "address": { 2024-03-24T03:17:23.069 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "address", 2024-03-24T03:17:23.069 INFO:teuthology.orchestra.run.smithi118.stdout: "type": "str", 2024-03-24T03:17:23.069 INFO:teuthology.orchestra.run.smithi118.stdout: "level": "advanced", 2024-03-24T03:17:23.069 INFO:teuthology.orchestra.run.smithi118.stdout: "flags": 0, 2024-03-24T03:17:23.069 INFO:teuthology.orchestra.run.smithi118.stdout: "default_value": "unixgram:///tmp/telegraf.sock", 2024-03-24T03:17:23.069 INFO:teuthology.orchestra.run.smithi118.stdout: "min": "", 2024-03-24T03:17:23.069 INFO:teuthology.orchestra.run.smithi118.stdout: "max": "", 2024-03-24T03:17:23.069 INFO:teuthology.orchestra.run.smithi118.stdout: "enum_allowed": [], 2024-03-24T03:17:23.070 INFO:teuthology.orchestra.run.smithi118.stdout: "desc": "", 2024-03-24T03:17:23.070 INFO:teuthology.orchestra.run.smithi118.stdout: "long_desc": "", 2024-03-24T03:17:23.070 INFO:teuthology.orchestra.run.smithi118.stdout: "tags": [], 2024-03-24T03:17:23.070 INFO:teuthology.orchestra.run.smithi118.stdout: "see_also": [] 2024-03-24T03:17:23.070 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-24T03:17:23.070 INFO:teuthology.orchestra.run.smithi118.stdout: "interval": { 2024-03-24T03:17:23.070 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "interval", 2024-03-24T03:17:23.070 INFO:teuthology.orchestra.run.smithi118.stdout: "type": "secs", 2024-03-24T03:17:23.071 INFO:teuthology.orchestra.run.smithi118.stdout: "level": "advanced", 2024-03-24T03:17:23.071 INFO:teuthology.orchestra.run.smithi118.stdout: "flags": 0, 2024-03-24T03:17:23.071 INFO:teuthology.orchestra.run.smithi118.stdout: "default_value": "15", 2024-03-24T03:17:23.071 INFO:teuthology.orchestra.run.smithi118.stdout: "min": "", 2024-03-24T03:17:23.071 INFO:teuthology.orchestra.run.smithi118.stdout: "max": "", 2024-03-24T03:17:23.071 INFO:teuthology.orchestra.run.smithi118.stdout: "enum_allowed": [], 2024-03-24T03:17:23.071 INFO:teuthology.orchestra.run.smithi118.stdout: "desc": "", 2024-03-24T03:17:23.071 INFO:teuthology.orchestra.run.smithi118.stdout: "long_desc": "", 2024-03-24T03:17:23.071 INFO:teuthology.orchestra.run.smithi118.stdout: "tags": [], 2024-03-24T03:17:23.072 INFO:teuthology.orchestra.run.smithi118.stdout: "see_also": [] 2024-03-24T03:17:23.072 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-24T03:17:23.072 INFO:teuthology.orchestra.run.smithi118.stdout: "log_level": { 2024-03-24T03:17:23.072 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "log_level", 2024-03-24T03:17:23.072 INFO:teuthology.orchestra.run.smithi118.stdout: "type": "str", 2024-03-24T03:17:23.072 INFO:teuthology.orchestra.run.smithi118.stdout: "level": "advanced", 2024-03-24T03:17:23.072 INFO:teuthology.orchestra.run.smithi118.stdout: "flags": 1, 2024-03-24T03:17:23.072 INFO:teuthology.orchestra.run.smithi118.stdout: "default_value": "", 2024-03-24T03:17:23.072 INFO:teuthology.orchestra.run.smithi118.stdout: "min": "", 2024-03-24T03:17:23.072 INFO:teuthology.orchestra.run.smithi118.stdout: "max": "", 2024-03-24T03:17:23.073 INFO:teuthology.orchestra.run.smithi118.stdout: "enum_allowed": [ 2024-03-24T03:17:23.073 INFO:teuthology.orchestra.run.smithi118.stdout: "", 2024-03-24T03:17:23.073 INFO:teuthology.orchestra.run.smithi118.stdout: "critical", 2024-03-24T03:17:23.073 INFO:teuthology.orchestra.run.smithi118.stdout: "debug", 2024-03-24T03:17:23.073 INFO:teuthology.orchestra.run.smithi118.stdout: "error", 2024-03-24T03:17:23.073 INFO:teuthology.orchestra.run.smithi118.stdout: "info", 2024-03-24T03:17:23.073 INFO:teuthology.orchestra.run.smithi118.stdout: "warning" 2024-03-24T03:17:23.073 INFO:teuthology.orchestra.run.smithi118.stdout: ], 2024-03-24T03:17:23.073 INFO:teuthology.orchestra.run.smithi118.stdout: "desc": "", 2024-03-24T03:17:23.074 INFO:teuthology.orchestra.run.smithi118.stdout: "long_desc": "", 2024-03-24T03:17:23.074 INFO:teuthology.orchestra.run.smithi118.stdout: "tags": [], 2024-03-24T03:17:23.074 INFO:teuthology.orchestra.run.smithi118.stdout: "see_also": [] 2024-03-24T03:17:23.074 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-24T03:17:23.074 INFO:teuthology.orchestra.run.smithi118.stdout: "log_to_cluster": { 2024-03-24T03:17:23.074 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "log_to_cluster", 2024-03-24T03:17:23.074 INFO:teuthology.orchestra.run.smithi118.stdout: "type": "bool", 2024-03-24T03:17:23.075 INFO:teuthology.orchestra.run.smithi118.stdout: "level": "advanced", 2024-03-24T03:17:23.075 INFO:teuthology.orchestra.run.smithi118.stdout: "flags": 1, 2024-03-24T03:17:23.075 INFO:teuthology.orchestra.run.smithi118.stdout: "default_value": "False", 2024-03-24T03:17:23.075 INFO:teuthology.orchestra.run.smithi118.stdout: "min": "", 2024-03-24T03:17:23.075 INFO:teuthology.orchestra.run.smithi118.stdout: "max": "", 2024-03-24T03:17:23.075 INFO:teuthology.orchestra.run.smithi118.stdout: "enum_allowed": [], 2024-03-24T03:17:23.075 INFO:teuthology.orchestra.run.smithi118.stdout: "desc": "", 2024-03-24T03:17:23.075 INFO:teuthology.orchestra.run.smithi118.stdout: "long_desc": "", 2024-03-24T03:17:23.075 INFO:teuthology.orchestra.run.smithi118.stdout: "tags": [], 2024-03-24T03:17:23.076 INFO:teuthology.orchestra.run.smithi118.stdout: "see_also": [] 2024-03-24T03:17:23.076 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-24T03:17:23.076 INFO:teuthology.orchestra.run.smithi118.stdout: "log_to_cluster_level": { 2024-03-24T03:17:23.076 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "log_to_cluster_level", 2024-03-24T03:17:23.076 INFO:teuthology.orchestra.run.smithi118.stdout: "type": "str", 2024-03-24T03:17:23.076 INFO:teuthology.orchestra.run.smithi118.stdout: "level": "advanced", 2024-03-24T03:17:23.076 INFO:teuthology.orchestra.run.smithi118.stdout: "flags": 1, 2024-03-24T03:17:23.076 INFO:teuthology.orchestra.run.smithi118.stdout: "default_value": "info", 2024-03-24T03:17:23.076 INFO:teuthology.orchestra.run.smithi118.stdout: "min": "", 2024-03-24T03:17:23.077 INFO:teuthology.orchestra.run.smithi118.stdout: "max": "", 2024-03-24T03:17:23.077 INFO:teuthology.orchestra.run.smithi118.stdout: "enum_allowed": [ 2024-03-24T03:17:23.077 INFO:teuthology.orchestra.run.smithi118.stdout: "", 2024-03-24T03:17:23.077 INFO:teuthology.orchestra.run.smithi118.stdout: "critical", 2024-03-24T03:17:23.077 INFO:teuthology.orchestra.run.smithi118.stdout: "debug", 2024-03-24T03:17:23.077 INFO:teuthology.orchestra.run.smithi118.stdout: "error", 2024-03-24T03:17:23.077 INFO:teuthology.orchestra.run.smithi118.stdout: "info", 2024-03-24T03:17:23.077 INFO:teuthology.orchestra.run.smithi118.stdout: "warning" 2024-03-24T03:17:23.077 INFO:teuthology.orchestra.run.smithi118.stdout: ], 2024-03-24T03:17:23.077 INFO:teuthology.orchestra.run.smithi118.stdout: "desc": "", 2024-03-24T03:17:23.078 INFO:teuthology.orchestra.run.smithi118.stdout: "long_desc": "", 2024-03-24T03:17:23.078 INFO:teuthology.orchestra.run.smithi118.stdout: "tags": [], 2024-03-24T03:17:23.078 INFO:teuthology.orchestra.run.smithi118.stdout: "see_also": [] 2024-03-24T03:17:23.078 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-24T03:17:23.078 INFO:teuthology.orchestra.run.smithi118.stdout: "log_to_file": { 2024-03-24T03:17:23.078 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "log_to_file", 2024-03-24T03:17:23.078 INFO:teuthology.orchestra.run.smithi118.stdout: "type": "bool", 2024-03-24T03:17:23.078 INFO:teuthology.orchestra.run.smithi118.stdout: "level": "advanced", 2024-03-24T03:17:23.079 INFO:teuthology.orchestra.run.smithi118.stdout: "flags": 1, 2024-03-24T03:17:23.079 INFO:teuthology.orchestra.run.smithi118.stdout: "default_value": "False", 2024-03-24T03:17:23.079 INFO:teuthology.orchestra.run.smithi118.stdout: "min": "", 2024-03-24T03:17:23.079 INFO:teuthology.orchestra.run.smithi118.stdout: "max": "", 2024-03-24T03:17:23.079 INFO:teuthology.orchestra.run.smithi118.stdout: "enum_allowed": [], 2024-03-24T03:17:23.079 INFO:teuthology.orchestra.run.smithi118.stdout: "desc": "", 2024-03-24T03:17:23.079 INFO:teuthology.orchestra.run.smithi118.stdout: "long_desc": "", 2024-03-24T03:17:23.079 INFO:teuthology.orchestra.run.smithi118.stdout: "tags": [], 2024-03-24T03:17:23.079 INFO:teuthology.orchestra.run.smithi118.stdout: "see_also": [] 2024-03-24T03:17:23.079 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-03-24T03:17:23.080 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-03-24T03:17:23.080 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-24T03:17:23.080 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-24T03:17:23.080 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "telemetry", 2024-03-24T03:17:23.080 INFO:teuthology.orchestra.run.smithi118.stdout: "can_run": true, 2024-03-24T03:17:23.080 INFO:teuthology.orchestra.run.smithi118.stdout: "error_string": "", 2024-03-24T03:17:23.080 INFO:teuthology.orchestra.run.smithi118.stdout: "module_options": { 2024-03-24T03:17:23.080 INFO:teuthology.orchestra.run.smithi118.stdout: "channel_basic": { 2024-03-24T03:17:23.080 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "channel_basic", 2024-03-24T03:17:23.081 INFO:teuthology.orchestra.run.smithi118.stdout: "type": "bool", 2024-03-24T03:17:23.081 INFO:teuthology.orchestra.run.smithi118.stdout: "level": "advanced", 2024-03-24T03:17:23.081 INFO:teuthology.orchestra.run.smithi118.stdout: "flags": 0, 2024-03-24T03:17:23.081 INFO:teuthology.orchestra.run.smithi118.stdout: "default_value": "True", 2024-03-24T03:17:23.081 INFO:teuthology.orchestra.run.smithi118.stdout: "min": "", 2024-03-24T03:17:23.081 INFO:teuthology.orchestra.run.smithi118.stdout: "max": "", 2024-03-24T03:17:23.081 INFO:teuthology.orchestra.run.smithi118.stdout: "enum_allowed": [], 2024-03-24T03:17:23.081 INFO:teuthology.orchestra.run.smithi118.stdout: "desc": "Share basic cluster information (size, version)", 2024-03-24T03:17:23.081 INFO:teuthology.orchestra.run.smithi118.stdout: "long_desc": "", 2024-03-24T03:17:23.082 INFO:teuthology.orchestra.run.smithi118.stdout: "tags": [], 2024-03-24T03:17:23.082 INFO:teuthology.orchestra.run.smithi118.stdout: "see_also": [] 2024-03-24T03:17:23.082 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-24T03:17:23.082 INFO:teuthology.orchestra.run.smithi118.stdout: "channel_crash": { 2024-03-24T03:17:23.082 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "channel_crash", 2024-03-24T03:17:23.082 INFO:teuthology.orchestra.run.smithi118.stdout: "type": "bool", 2024-03-24T03:17:23.082 INFO:teuthology.orchestra.run.smithi118.stdout: "level": "advanced", 2024-03-24T03:17:23.082 INFO:teuthology.orchestra.run.smithi118.stdout: "flags": 0, 2024-03-24T03:17:23.082 INFO:teuthology.orchestra.run.smithi118.stdout: "default_value": "True", 2024-03-24T03:17:23.083 INFO:teuthology.orchestra.run.smithi118.stdout: "min": "", 2024-03-24T03:17:23.083 INFO:teuthology.orchestra.run.smithi118.stdout: "max": "", 2024-03-24T03:17:23.083 INFO:teuthology.orchestra.run.smithi118.stdout: "enum_allowed": [], 2024-03-24T03:17:23.083 INFO:teuthology.orchestra.run.smithi118.stdout: "desc": "Share metadata about Ceph daemon crashes (version, stack straces, etc)", 2024-03-24T03:17:23.083 INFO:teuthology.orchestra.run.smithi118.stdout: "long_desc": "", 2024-03-24T03:17:23.083 INFO:teuthology.orchestra.run.smithi118.stdout: "tags": [], 2024-03-24T03:17:23.083 INFO:teuthology.orchestra.run.smithi118.stdout: "see_also": [] 2024-03-24T03:17:23.083 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-24T03:17:23.083 INFO:teuthology.orchestra.run.smithi118.stdout: "channel_device": { 2024-03-24T03:17:23.084 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "channel_device", 2024-03-24T03:17:23.084 INFO:teuthology.orchestra.run.smithi118.stdout: "type": "bool", 2024-03-24T03:17:23.084 INFO:teuthology.orchestra.run.smithi118.stdout: "level": "advanced", 2024-03-24T03:17:23.084 INFO:teuthology.orchestra.run.smithi118.stdout: "flags": 0, 2024-03-24T03:17:23.084 INFO:teuthology.orchestra.run.smithi118.stdout: "default_value": "True", 2024-03-24T03:17:23.084 INFO:teuthology.orchestra.run.smithi118.stdout: "min": "", 2024-03-24T03:17:23.084 INFO:teuthology.orchestra.run.smithi118.stdout: "max": "", 2024-03-24T03:17:23.084 INFO:teuthology.orchestra.run.smithi118.stdout: "enum_allowed": [], 2024-03-24T03:17:23.085 INFO:teuthology.orchestra.run.smithi118.stdout: "desc": "Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)", 2024-03-24T03:17:23.085 INFO:teuthology.orchestra.run.smithi118.stdout: "long_desc": "", 2024-03-24T03:17:23.085 INFO:teuthology.orchestra.run.smithi118.stdout: "tags": [], 2024-03-24T03:17:23.085 INFO:teuthology.orchestra.run.smithi118.stdout: "see_also": [] 2024-03-24T03:17:23.085 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-24T03:17:23.085 INFO:teuthology.orchestra.run.smithi118.stdout: "channel_ident": { 2024-03-24T03:17:23.085 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "channel_ident", 2024-03-24T03:17:23.085 INFO:teuthology.orchestra.run.smithi118.stdout: "type": "bool", 2024-03-24T03:17:23.085 INFO:teuthology.orchestra.run.smithi118.stdout: "level": "advanced", 2024-03-24T03:17:23.086 INFO:teuthology.orchestra.run.smithi118.stdout: "flags": 0, 2024-03-24T03:17:23.086 INFO:teuthology.orchestra.run.smithi118.stdout: "default_value": "False", 2024-03-24T03:17:23.086 INFO:teuthology.orchestra.run.smithi118.stdout: "min": "", 2024-03-24T03:17:23.086 INFO:teuthology.orchestra.run.smithi118.stdout: "max": "", 2024-03-24T03:17:23.086 INFO:teuthology.orchestra.run.smithi118.stdout: "enum_allowed": [], 2024-03-24T03:17:23.086 INFO:teuthology.orchestra.run.smithi118.stdout: "desc": "Share a user-provided description and/or contact email for the cluster", 2024-03-24T03:17:23.086 INFO:teuthology.orchestra.run.smithi118.stdout: "long_desc": "", 2024-03-24T03:17:23.086 INFO:teuthology.orchestra.run.smithi118.stdout: "tags": [], 2024-03-24T03:17:23.086 INFO:teuthology.orchestra.run.smithi118.stdout: "see_also": [] 2024-03-24T03:17:23.087 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-24T03:17:23.087 INFO:teuthology.orchestra.run.smithi118.stdout: "channel_perf": { 2024-03-24T03:17:23.087 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "channel_perf", 2024-03-24T03:17:23.087 INFO:teuthology.orchestra.run.smithi118.stdout: "type": "bool", 2024-03-24T03:17:23.087 INFO:teuthology.orchestra.run.smithi118.stdout: "level": "advanced", 2024-03-24T03:17:23.087 INFO:teuthology.orchestra.run.smithi118.stdout: "flags": 0, 2024-03-24T03:17:23.087 INFO:teuthology.orchestra.run.smithi118.stdout: "default_value": "False", 2024-03-24T03:17:23.087 INFO:teuthology.orchestra.run.smithi118.stdout: "min": "", 2024-03-24T03:17:23.087 INFO:teuthology.orchestra.run.smithi118.stdout: "max": "", 2024-03-24T03:17:23.087 INFO:teuthology.orchestra.run.smithi118.stdout: "enum_allowed": [], 2024-03-24T03:17:23.088 INFO:teuthology.orchestra.run.smithi118.stdout: "desc": "Share various performance metrics of a cluster", 2024-03-24T03:17:23.088 INFO:teuthology.orchestra.run.smithi118.stdout: "long_desc": "", 2024-03-24T03:17:23.088 INFO:teuthology.orchestra.run.smithi118.stdout: "tags": [], 2024-03-24T03:17:23.088 INFO:teuthology.orchestra.run.smithi118.stdout: "see_also": [] 2024-03-24T03:17:23.088 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-24T03:17:23.088 INFO:teuthology.orchestra.run.smithi118.stdout: "contact": { 2024-03-24T03:17:23.088 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "contact", 2024-03-24T03:17:23.088 INFO:teuthology.orchestra.run.smithi118.stdout: "type": "str", 2024-03-24T03:17:23.089 INFO:teuthology.orchestra.run.smithi118.stdout: "level": "advanced", 2024-03-24T03:17:23.089 INFO:teuthology.orchestra.run.smithi118.stdout: "flags": 0, 2024-03-24T03:17:23.089 INFO:teuthology.orchestra.run.smithi118.stdout: "default_value": "", 2024-03-24T03:17:23.089 INFO:teuthology.orchestra.run.smithi118.stdout: "min": "", 2024-03-24T03:17:23.089 INFO:teuthology.orchestra.run.smithi118.stdout: "max": "", 2024-03-24T03:17:23.089 INFO:teuthology.orchestra.run.smithi118.stdout: "enum_allowed": [], 2024-03-24T03:17:23.089 INFO:teuthology.orchestra.run.smithi118.stdout: "desc": "", 2024-03-24T03:17:23.089 INFO:teuthology.orchestra.run.smithi118.stdout: "long_desc": "", 2024-03-24T03:17:23.089 INFO:teuthology.orchestra.run.smithi118.stdout: "tags": [], 2024-03-24T03:17:23.090 INFO:teuthology.orchestra.run.smithi118.stdout: "see_also": [] 2024-03-24T03:17:23.090 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-24T03:17:23.090 INFO:teuthology.orchestra.run.smithi118.stdout: "description": { 2024-03-24T03:17:23.090 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "description", 2024-03-24T03:17:23.090 INFO:teuthology.orchestra.run.smithi118.stdout: "type": "str", 2024-03-24T03:17:23.090 INFO:teuthology.orchestra.run.smithi118.stdout: "level": "advanced", 2024-03-24T03:17:23.090 INFO:teuthology.orchestra.run.smithi118.stdout: "flags": 0, 2024-03-24T03:17:23.090 INFO:teuthology.orchestra.run.smithi118.stdout: "default_value": "", 2024-03-24T03:17:23.090 INFO:teuthology.orchestra.run.smithi118.stdout: "min": "", 2024-03-24T03:17:23.091 INFO:teuthology.orchestra.run.smithi118.stdout: "max": "", 2024-03-24T03:17:23.091 INFO:teuthology.orchestra.run.smithi118.stdout: "enum_allowed": [], 2024-03-24T03:17:23.091 INFO:teuthology.orchestra.run.smithi118.stdout: "desc": "", 2024-03-24T03:17:23.091 INFO:teuthology.orchestra.run.smithi118.stdout: "long_desc": "", 2024-03-24T03:17:23.091 INFO:teuthology.orchestra.run.smithi118.stdout: "tags": [], 2024-03-24T03:17:23.091 INFO:teuthology.orchestra.run.smithi118.stdout: "see_also": [] 2024-03-24T03:17:23.091 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-24T03:17:23.091 INFO:teuthology.orchestra.run.smithi118.stdout: "device_url": { 2024-03-24T03:17:23.091 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "device_url", 2024-03-24T03:17:23.092 INFO:teuthology.orchestra.run.smithi118.stdout: "type": "str", 2024-03-24T03:17:23.092 INFO:teuthology.orchestra.run.smithi118.stdout: "level": "advanced", 2024-03-24T03:17:23.092 INFO:teuthology.orchestra.run.smithi118.stdout: "flags": 0, 2024-03-24T03:17:23.092 INFO:teuthology.orchestra.run.smithi118.stdout: "default_value": "https://telemetry.ceph.com/device", 2024-03-24T03:17:23.092 INFO:teuthology.orchestra.run.smithi118.stdout: "min": "", 2024-03-24T03:17:23.092 INFO:teuthology.orchestra.run.smithi118.stdout: "max": "", 2024-03-24T03:17:23.092 INFO:teuthology.orchestra.run.smithi118.stdout: "enum_allowed": [], 2024-03-24T03:17:23.092 INFO:teuthology.orchestra.run.smithi118.stdout: "desc": "", 2024-03-24T03:17:23.092 INFO:teuthology.orchestra.run.smithi118.stdout: "long_desc": "", 2024-03-24T03:17:23.093 INFO:teuthology.orchestra.run.smithi118.stdout: "tags": [], 2024-03-24T03:17:23.093 INFO:teuthology.orchestra.run.smithi118.stdout: "see_also": [] 2024-03-24T03:17:23.093 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-24T03:17:23.093 INFO:teuthology.orchestra.run.smithi118.stdout: "enabled": { 2024-03-24T03:17:23.093 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "enabled", 2024-03-24T03:17:23.093 INFO:teuthology.orchestra.run.smithi118.stdout: "type": "bool", 2024-03-24T03:17:23.093 INFO:teuthology.orchestra.run.smithi118.stdout: "level": "advanced", 2024-03-24T03:17:23.093 INFO:teuthology.orchestra.run.smithi118.stdout: "flags": 0, 2024-03-24T03:17:23.093 INFO:teuthology.orchestra.run.smithi118.stdout: "default_value": "False", 2024-03-24T03:17:23.094 INFO:teuthology.orchestra.run.smithi118.stdout: "min": "", 2024-03-24T03:17:23.094 INFO:teuthology.orchestra.run.smithi118.stdout: "max": "", 2024-03-24T03:17:23.094 INFO:teuthology.orchestra.run.smithi118.stdout: "enum_allowed": [], 2024-03-24T03:17:23.094 INFO:teuthology.orchestra.run.smithi118.stdout: "desc": "", 2024-03-24T03:17:23.094 INFO:teuthology.orchestra.run.smithi118.stdout: "long_desc": "", 2024-03-24T03:17:23.094 INFO:teuthology.orchestra.run.smithi118.stdout: "tags": [], 2024-03-24T03:17:23.094 INFO:teuthology.orchestra.run.smithi118.stdout: "see_also": [] 2024-03-24T03:17:23.095 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-24T03:17:23.095 INFO:teuthology.orchestra.run.smithi118.stdout: "interval": { 2024-03-24T03:17:23.095 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "interval", 2024-03-24T03:17:23.095 INFO:teuthology.orchestra.run.smithi118.stdout: "type": "int", 2024-03-24T03:17:23.095 INFO:teuthology.orchestra.run.smithi118.stdout: "level": "advanced", 2024-03-24T03:17:23.095 INFO:teuthology.orchestra.run.smithi118.stdout: "flags": 0, 2024-03-24T03:17:23.095 INFO:teuthology.orchestra.run.smithi118.stdout: "default_value": "24", 2024-03-24T03:17:23.095 INFO:teuthology.orchestra.run.smithi118.stdout: "min": "8", 2024-03-24T03:17:23.095 INFO:teuthology.orchestra.run.smithi118.stdout: "max": "", 2024-03-24T03:17:23.096 INFO:teuthology.orchestra.run.smithi118.stdout: "enum_allowed": [], 2024-03-24T03:17:23.096 INFO:teuthology.orchestra.run.smithi118.stdout: "desc": "", 2024-03-24T03:17:23.096 INFO:teuthology.orchestra.run.smithi118.stdout: "long_desc": "", 2024-03-24T03:17:23.096 INFO:teuthology.orchestra.run.smithi118.stdout: "tags": [], 2024-03-24T03:17:23.096 INFO:teuthology.orchestra.run.smithi118.stdout: "see_also": [] 2024-03-24T03:17:23.096 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-24T03:17:23.096 INFO:teuthology.orchestra.run.smithi118.stdout: "last_opt_revision": { 2024-03-24T03:17:23.096 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "last_opt_revision", 2024-03-24T03:17:23.096 INFO:teuthology.orchestra.run.smithi118.stdout: "type": "int", 2024-03-24T03:17:23.096 INFO:teuthology.orchestra.run.smithi118.stdout: "level": "advanced", 2024-03-24T03:17:23.097 INFO:teuthology.orchestra.run.smithi118.stdout: "flags": 0, 2024-03-24T03:17:23.097 INFO:teuthology.orchestra.run.smithi118.stdout: "default_value": "1", 2024-03-24T03:17:23.097 INFO:teuthology.orchestra.run.smithi118.stdout: "min": "", 2024-03-24T03:17:23.097 INFO:teuthology.orchestra.run.smithi118.stdout: "max": "", 2024-03-24T03:17:23.097 INFO:teuthology.orchestra.run.smithi118.stdout: "enum_allowed": [], 2024-03-24T03:17:23.097 INFO:teuthology.orchestra.run.smithi118.stdout: "desc": "", 2024-03-24T03:17:23.097 INFO:teuthology.orchestra.run.smithi118.stdout: "long_desc": "", 2024-03-24T03:17:23.097 INFO:teuthology.orchestra.run.smithi118.stdout: "tags": [], 2024-03-24T03:17:23.097 INFO:teuthology.orchestra.run.smithi118.stdout: "see_also": [] 2024-03-24T03:17:23.098 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-24T03:17:23.098 INFO:teuthology.orchestra.run.smithi118.stdout: "leaderboard": { 2024-03-24T03:17:23.098 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "leaderboard", 2024-03-24T03:17:23.098 INFO:teuthology.orchestra.run.smithi118.stdout: "type": "bool", 2024-03-24T03:17:23.098 INFO:teuthology.orchestra.run.smithi118.stdout: "level": "advanced", 2024-03-24T03:17:23.098 INFO:teuthology.orchestra.run.smithi118.stdout: "flags": 0, 2024-03-24T03:17:23.098 INFO:teuthology.orchestra.run.smithi118.stdout: "default_value": "False", 2024-03-24T03:17:23.098 INFO:teuthology.orchestra.run.smithi118.stdout: "min": "", 2024-03-24T03:17:23.098 INFO:teuthology.orchestra.run.smithi118.stdout: "max": "", 2024-03-24T03:17:23.099 INFO:teuthology.orchestra.run.smithi118.stdout: "enum_allowed": [], 2024-03-24T03:17:23.099 INFO:teuthology.orchestra.run.smithi118.stdout: "desc": "", 2024-03-24T03:17:23.099 INFO:teuthology.orchestra.run.smithi118.stdout: "long_desc": "", 2024-03-24T03:17:23.099 INFO:teuthology.orchestra.run.smithi118.stdout: "tags": [], 2024-03-24T03:17:23.099 INFO:teuthology.orchestra.run.smithi118.stdout: "see_also": [] 2024-03-24T03:17:23.099 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-24T03:17:23.099 INFO:teuthology.orchestra.run.smithi118.stdout: "leaderboard_description": { 2024-03-24T03:17:23.099 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "leaderboard_description", 2024-03-24T03:17:23.099 INFO:teuthology.orchestra.run.smithi118.stdout: "type": "str", 2024-03-24T03:17:23.100 INFO:teuthology.orchestra.run.smithi118.stdout: "level": "advanced", 2024-03-24T03:17:23.100 INFO:teuthology.orchestra.run.smithi118.stdout: "flags": 0, 2024-03-24T03:17:23.100 INFO:teuthology.orchestra.run.smithi118.stdout: "default_value": "", 2024-03-24T03:17:23.100 INFO:teuthology.orchestra.run.smithi118.stdout: "min": "", 2024-03-24T03:17:23.100 INFO:teuthology.orchestra.run.smithi118.stdout: "max": "", 2024-03-24T03:17:23.100 INFO:teuthology.orchestra.run.smithi118.stdout: "enum_allowed": [], 2024-03-24T03:17:23.100 INFO:teuthology.orchestra.run.smithi118.stdout: "desc": "", 2024-03-24T03:17:23.100 INFO:teuthology.orchestra.run.smithi118.stdout: "long_desc": "", 2024-03-24T03:17:23.100 INFO:teuthology.orchestra.run.smithi118.stdout: "tags": [], 2024-03-24T03:17:23.100 INFO:teuthology.orchestra.run.smithi118.stdout: "see_also": [] 2024-03-24T03:17:23.101 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-24T03:17:23.101 INFO:teuthology.orchestra.run.smithi118.stdout: "log_level": { 2024-03-24T03:17:23.101 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "log_level", 2024-03-24T03:17:23.101 INFO:teuthology.orchestra.run.smithi118.stdout: "type": "str", 2024-03-24T03:17:23.101 INFO:teuthology.orchestra.run.smithi118.stdout: "level": "advanced", 2024-03-24T03:17:23.101 INFO:teuthology.orchestra.run.smithi118.stdout: "flags": 1, 2024-03-24T03:17:23.101 INFO:teuthology.orchestra.run.smithi118.stdout: "default_value": "", 2024-03-24T03:17:23.101 INFO:teuthology.orchestra.run.smithi118.stdout: "min": "", 2024-03-24T03:17:23.101 INFO:teuthology.orchestra.run.smithi118.stdout: "max": "", 2024-03-24T03:17:23.102 INFO:teuthology.orchestra.run.smithi118.stdout: "enum_allowed": [ 2024-03-24T03:17:23.102 INFO:teuthology.orchestra.run.smithi118.stdout: "", 2024-03-24T03:17:23.102 INFO:teuthology.orchestra.run.smithi118.stdout: "critical", 2024-03-24T03:17:23.102 INFO:teuthology.orchestra.run.smithi118.stdout: "debug", 2024-03-24T03:17:23.102 INFO:teuthology.orchestra.run.smithi118.stdout: "error", 2024-03-24T03:17:23.102 INFO:teuthology.orchestra.run.smithi118.stdout: "info", 2024-03-24T03:17:23.102 INFO:teuthology.orchestra.run.smithi118.stdout: "warning" 2024-03-24T03:17:23.102 INFO:teuthology.orchestra.run.smithi118.stdout: ], 2024-03-24T03:17:23.103 INFO:teuthology.orchestra.run.smithi118.stdout: "desc": "", 2024-03-24T03:17:23.103 INFO:teuthology.orchestra.run.smithi118.stdout: "long_desc": "", 2024-03-24T03:17:23.103 INFO:teuthology.orchestra.run.smithi118.stdout: "tags": [], 2024-03-24T03:17:23.103 INFO:teuthology.orchestra.run.smithi118.stdout: "see_also": [] 2024-03-24T03:17:23.103 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-24T03:17:23.103 INFO:teuthology.orchestra.run.smithi118.stdout: "log_to_cluster": { 2024-03-24T03:17:23.103 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "log_to_cluster", 2024-03-24T03:17:23.103 INFO:teuthology.orchestra.run.smithi118.stdout: "type": "bool", 2024-03-24T03:17:23.103 INFO:teuthology.orchestra.run.smithi118.stdout: "level": "advanced", 2024-03-24T03:17:23.104 INFO:teuthology.orchestra.run.smithi118.stdout: "flags": 1, 2024-03-24T03:17:23.104 INFO:teuthology.orchestra.run.smithi118.stdout: "default_value": "False", 2024-03-24T03:17:23.104 INFO:teuthology.orchestra.run.smithi118.stdout: "min": "", 2024-03-24T03:17:23.104 INFO:teuthology.orchestra.run.smithi118.stdout: "max": "", 2024-03-24T03:17:23.104 INFO:teuthology.orchestra.run.smithi118.stdout: "enum_allowed": [], 2024-03-24T03:17:23.104 INFO:teuthology.orchestra.run.smithi118.stdout: "desc": "", 2024-03-24T03:17:23.104 INFO:teuthology.orchestra.run.smithi118.stdout: "long_desc": "", 2024-03-24T03:17:23.104 INFO:teuthology.orchestra.run.smithi118.stdout: "tags": [], 2024-03-24T03:17:23.105 INFO:teuthology.orchestra.run.smithi118.stdout: "see_also": [] 2024-03-24T03:17:23.105 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-24T03:17:23.105 INFO:teuthology.orchestra.run.smithi118.stdout: "log_to_cluster_level": { 2024-03-24T03:17:23.105 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "log_to_cluster_level", 2024-03-24T03:17:23.105 INFO:teuthology.orchestra.run.smithi118.stdout: "type": "str", 2024-03-24T03:17:23.105 INFO:teuthology.orchestra.run.smithi118.stdout: "level": "advanced", 2024-03-24T03:17:23.105 INFO:teuthology.orchestra.run.smithi118.stdout: "flags": 1, 2024-03-24T03:17:23.105 INFO:teuthology.orchestra.run.smithi118.stdout: "default_value": "info", 2024-03-24T03:17:23.105 INFO:teuthology.orchestra.run.smithi118.stdout: "min": "", 2024-03-24T03:17:23.106 INFO:teuthology.orchestra.run.smithi118.stdout: "max": "", 2024-03-24T03:17:23.106 INFO:teuthology.orchestra.run.smithi118.stdout: "enum_allowed": [ 2024-03-24T03:17:23.106 INFO:teuthology.orchestra.run.smithi118.stdout: "", 2024-03-24T03:17:23.106 INFO:teuthology.orchestra.run.smithi118.stdout: "critical", 2024-03-24T03:17:23.106 INFO:teuthology.orchestra.run.smithi118.stdout: "debug", 2024-03-24T03:17:23.106 INFO:teuthology.orchestra.run.smithi118.stdout: "error", 2024-03-24T03:17:23.106 INFO:teuthology.orchestra.run.smithi118.stdout: "info", 2024-03-24T03:17:23.106 INFO:teuthology.orchestra.run.smithi118.stdout: "warning" 2024-03-24T03:17:23.107 INFO:teuthology.orchestra.run.smithi118.stdout: ], 2024-03-24T03:17:23.107 INFO:teuthology.orchestra.run.smithi118.stdout: "desc": "", 2024-03-24T03:17:23.107 INFO:teuthology.orchestra.run.smithi118.stdout: "long_desc": "", 2024-03-24T03:17:23.107 INFO:teuthology.orchestra.run.smithi118.stdout: "tags": [], 2024-03-24T03:17:23.107 INFO:teuthology.orchestra.run.smithi118.stdout: "see_also": [] 2024-03-24T03:17:23.107 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-24T03:17:23.107 INFO:teuthology.orchestra.run.smithi118.stdout: "log_to_file": { 2024-03-24T03:17:23.107 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "log_to_file", 2024-03-24T03:17:23.107 INFO:teuthology.orchestra.run.smithi118.stdout: "type": "bool", 2024-03-24T03:17:23.107 INFO:teuthology.orchestra.run.smithi118.stdout: "level": "advanced", 2024-03-24T03:17:23.108 INFO:teuthology.orchestra.run.smithi118.stdout: "flags": 1, 2024-03-24T03:17:23.108 INFO:teuthology.orchestra.run.smithi118.stdout: "default_value": "False", 2024-03-24T03:17:23.108 INFO:teuthology.orchestra.run.smithi118.stdout: "min": "", 2024-03-24T03:17:23.108 INFO:teuthology.orchestra.run.smithi118.stdout: "max": "", 2024-03-24T03:17:23.108 INFO:teuthology.orchestra.run.smithi118.stdout: "enum_allowed": [], 2024-03-24T03:17:23.108 INFO:teuthology.orchestra.run.smithi118.stdout: "desc": "", 2024-03-24T03:17:23.108 INFO:teuthology.orchestra.run.smithi118.stdout: "long_desc": "", 2024-03-24T03:17:23.108 INFO:teuthology.orchestra.run.smithi118.stdout: "tags": [], 2024-03-24T03:17:23.108 INFO:teuthology.orchestra.run.smithi118.stdout: "see_also": [] 2024-03-24T03:17:23.109 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-24T03:17:23.109 INFO:teuthology.orchestra.run.smithi118.stdout: "organization": { 2024-03-24T03:17:23.109 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "organization", 2024-03-24T03:17:23.109 INFO:teuthology.orchestra.run.smithi118.stdout: "type": "str", 2024-03-24T03:17:23.109 INFO:teuthology.orchestra.run.smithi118.stdout: "level": "advanced", 2024-03-24T03:17:23.109 INFO:teuthology.orchestra.run.smithi118.stdout: "flags": 0, 2024-03-24T03:17:23.109 INFO:teuthology.orchestra.run.smithi118.stdout: "default_value": "", 2024-03-24T03:17:23.109 INFO:teuthology.orchestra.run.smithi118.stdout: "min": "", 2024-03-24T03:17:23.109 INFO:teuthology.orchestra.run.smithi118.stdout: "max": "", 2024-03-24T03:17:23.110 INFO:teuthology.orchestra.run.smithi118.stdout: "enum_allowed": [], 2024-03-24T03:17:23.110 INFO:teuthology.orchestra.run.smithi118.stdout: "desc": "", 2024-03-24T03:17:23.110 INFO:teuthology.orchestra.run.smithi118.stdout: "long_desc": "", 2024-03-24T03:17:23.110 INFO:teuthology.orchestra.run.smithi118.stdout: "tags": [], 2024-03-24T03:17:23.110 INFO:teuthology.orchestra.run.smithi118.stdout: "see_also": [] 2024-03-24T03:17:23.110 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-24T03:17:23.110 INFO:teuthology.orchestra.run.smithi118.stdout: "proxy": { 2024-03-24T03:17:23.110 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "proxy", 2024-03-24T03:17:23.110 INFO:teuthology.orchestra.run.smithi118.stdout: "type": "str", 2024-03-24T03:17:23.111 INFO:teuthology.orchestra.run.smithi118.stdout: "level": "advanced", 2024-03-24T03:17:23.111 INFO:teuthology.orchestra.run.smithi118.stdout: "flags": 0, 2024-03-24T03:17:23.111 INFO:teuthology.orchestra.run.smithi118.stdout: "default_value": "", 2024-03-24T03:17:23.111 INFO:teuthology.orchestra.run.smithi118.stdout: "min": "", 2024-03-24T03:17:23.111 INFO:teuthology.orchestra.run.smithi118.stdout: "max": "", 2024-03-24T03:17:23.111 INFO:teuthology.orchestra.run.smithi118.stdout: "enum_allowed": [], 2024-03-24T03:17:23.111 INFO:teuthology.orchestra.run.smithi118.stdout: "desc": "", 2024-03-24T03:17:23.111 INFO:teuthology.orchestra.run.smithi118.stdout: "long_desc": "", 2024-03-24T03:17:23.112 INFO:teuthology.orchestra.run.smithi118.stdout: "tags": [], 2024-03-24T03:17:23.112 INFO:teuthology.orchestra.run.smithi118.stdout: "see_also": [] 2024-03-24T03:17:23.112 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-24T03:17:23.112 INFO:teuthology.orchestra.run.smithi118.stdout: "url": { 2024-03-24T03:17:23.112 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "url", 2024-03-24T03:17:23.112 INFO:teuthology.orchestra.run.smithi118.stdout: "type": "str", 2024-03-24T03:17:23.112 INFO:teuthology.orchestra.run.smithi118.stdout: "level": "advanced", 2024-03-24T03:17:23.112 INFO:teuthology.orchestra.run.smithi118.stdout: "flags": 0, 2024-03-24T03:17:23.112 INFO:teuthology.orchestra.run.smithi118.stdout: "default_value": "https://telemetry.ceph.com/report", 2024-03-24T03:17:23.113 INFO:teuthology.orchestra.run.smithi118.stdout: "min": "", 2024-03-24T03:17:23.113 INFO:teuthology.orchestra.run.smithi118.stdout: "max": "", 2024-03-24T03:17:23.113 INFO:teuthology.orchestra.run.smithi118.stdout: "enum_allowed": [], 2024-03-24T03:17:23.113 INFO:teuthology.orchestra.run.smithi118.stdout: "desc": "", 2024-03-24T03:17:23.113 INFO:teuthology.orchestra.run.smithi118.stdout: "long_desc": "", 2024-03-24T03:17:23.113 INFO:teuthology.orchestra.run.smithi118.stdout: "tags": [], 2024-03-24T03:17:23.113 INFO:teuthology.orchestra.run.smithi118.stdout: "see_also": [] 2024-03-24T03:17:23.113 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-03-24T03:17:23.113 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-03-24T03:17:23.114 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-24T03:17:23.114 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-24T03:17:23.114 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "test_orchestrator", 2024-03-24T03:17:23.114 INFO:teuthology.orchestra.run.smithi118.stdout: "can_run": true, 2024-03-24T03:17:23.114 INFO:teuthology.orchestra.run.smithi118.stdout: "error_string": "", 2024-03-24T03:17:23.114 INFO:teuthology.orchestra.run.smithi118.stdout: "module_options": { 2024-03-24T03:17:23.114 INFO:teuthology.orchestra.run.smithi118.stdout: "log_level": { 2024-03-24T03:17:23.115 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "log_level", 2024-03-24T03:17:23.115 INFO:teuthology.orchestra.run.smithi118.stdout: "type": "str", 2024-03-24T03:17:23.115 INFO:teuthology.orchestra.run.smithi118.stdout: "level": "advanced", 2024-03-24T03:17:23.115 INFO:teuthology.orchestra.run.smithi118.stdout: "flags": 1, 2024-03-24T03:17:23.115 INFO:teuthology.orchestra.run.smithi118.stdout: "default_value": "", 2024-03-24T03:17:23.115 INFO:teuthology.orchestra.run.smithi118.stdout: "min": "", 2024-03-24T03:17:23.115 INFO:teuthology.orchestra.run.smithi118.stdout: "max": "", 2024-03-24T03:17:23.115 INFO:teuthology.orchestra.run.smithi118.stdout: "enum_allowed": [ 2024-03-24T03:17:23.115 INFO:teuthology.orchestra.run.smithi118.stdout: "", 2024-03-24T03:17:23.116 INFO:teuthology.orchestra.run.smithi118.stdout: "critical", 2024-03-24T03:17:23.116 INFO:teuthology.orchestra.run.smithi118.stdout: "debug", 2024-03-24T03:17:23.116 INFO:teuthology.orchestra.run.smithi118.stdout: "error", 2024-03-24T03:17:23.116 INFO:teuthology.orchestra.run.smithi118.stdout: "info", 2024-03-24T03:17:23.116 INFO:teuthology.orchestra.run.smithi118.stdout: "warning" 2024-03-24T03:17:23.116 INFO:teuthology.orchestra.run.smithi118.stdout: ], 2024-03-24T03:17:23.116 INFO:teuthology.orchestra.run.smithi118.stdout: "desc": "", 2024-03-24T03:17:23.116 INFO:teuthology.orchestra.run.smithi118.stdout: "long_desc": "", 2024-03-24T03:17:23.116 INFO:teuthology.orchestra.run.smithi118.stdout: "tags": [], 2024-03-24T03:17:23.117 INFO:teuthology.orchestra.run.smithi118.stdout: "see_also": [] 2024-03-24T03:17:23.117 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-24T03:17:23.117 INFO:teuthology.orchestra.run.smithi118.stdout: "log_to_cluster": { 2024-03-24T03:17:23.117 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "log_to_cluster", 2024-03-24T03:17:23.117 INFO:teuthology.orchestra.run.smithi118.stdout: "type": "bool", 2024-03-24T03:17:23.117 INFO:teuthology.orchestra.run.smithi118.stdout: "level": "advanced", 2024-03-24T03:17:23.117 INFO:teuthology.orchestra.run.smithi118.stdout: "flags": 1, 2024-03-24T03:17:23.117 INFO:teuthology.orchestra.run.smithi118.stdout: "default_value": "False", 2024-03-24T03:17:23.117 INFO:teuthology.orchestra.run.smithi118.stdout: "min": "", 2024-03-24T03:17:23.117 INFO:teuthology.orchestra.run.smithi118.stdout: "max": "", 2024-03-24T03:17:23.118 INFO:teuthology.orchestra.run.smithi118.stdout: "enum_allowed": [], 2024-03-24T03:17:23.118 INFO:teuthology.orchestra.run.smithi118.stdout: "desc": "", 2024-03-24T03:17:23.118 INFO:teuthology.orchestra.run.smithi118.stdout: "long_desc": "", 2024-03-24T03:17:23.118 INFO:teuthology.orchestra.run.smithi118.stdout: "tags": [], 2024-03-24T03:17:23.118 INFO:teuthology.orchestra.run.smithi118.stdout: "see_also": [] 2024-03-24T03:17:23.118 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-24T03:17:23.118 INFO:teuthology.orchestra.run.smithi118.stdout: "log_to_cluster_level": { 2024-03-24T03:17:23.118 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "log_to_cluster_level", 2024-03-24T03:17:23.119 INFO:teuthology.orchestra.run.smithi118.stdout: "type": "str", 2024-03-24T03:17:23.119 INFO:teuthology.orchestra.run.smithi118.stdout: "level": "advanced", 2024-03-24T03:17:23.119 INFO:teuthology.orchestra.run.smithi118.stdout: "flags": 1, 2024-03-24T03:17:23.119 INFO:teuthology.orchestra.run.smithi118.stdout: "default_value": "info", 2024-03-24T03:17:23.119 INFO:teuthology.orchestra.run.smithi118.stdout: "min": "", 2024-03-24T03:17:23.119 INFO:teuthology.orchestra.run.smithi118.stdout: "max": "", 2024-03-24T03:17:23.119 INFO:teuthology.orchestra.run.smithi118.stdout: "enum_allowed": [ 2024-03-24T03:17:23.119 INFO:teuthology.orchestra.run.smithi118.stdout: "", 2024-03-24T03:17:23.119 INFO:teuthology.orchestra.run.smithi118.stdout: "critical", 2024-03-24T03:17:23.120 INFO:teuthology.orchestra.run.smithi118.stdout: "debug", 2024-03-24T03:17:23.120 INFO:teuthology.orchestra.run.smithi118.stdout: "error", 2024-03-24T03:17:23.120 INFO:teuthology.orchestra.run.smithi118.stdout: "info", 2024-03-24T03:17:23.120 INFO:teuthology.orchestra.run.smithi118.stdout: "warning" 2024-03-24T03:17:23.120 INFO:teuthology.orchestra.run.smithi118.stdout: ], 2024-03-24T03:17:23.120 INFO:teuthology.orchestra.run.smithi118.stdout: "desc": "", 2024-03-24T03:17:23.120 INFO:teuthology.orchestra.run.smithi118.stdout: "long_desc": "", 2024-03-24T03:17:23.120 INFO:teuthology.orchestra.run.smithi118.stdout: "tags": [], 2024-03-24T03:17:23.120 INFO:teuthology.orchestra.run.smithi118.stdout: "see_also": [] 2024-03-24T03:17:23.121 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-24T03:17:23.121 INFO:teuthology.orchestra.run.smithi118.stdout: "log_to_file": { 2024-03-24T03:17:23.121 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "log_to_file", 2024-03-24T03:17:23.121 INFO:teuthology.orchestra.run.smithi118.stdout: "type": "bool", 2024-03-24T03:17:23.121 INFO:teuthology.orchestra.run.smithi118.stdout: "level": "advanced", 2024-03-24T03:17:23.121 INFO:teuthology.orchestra.run.smithi118.stdout: "flags": 1, 2024-03-24T03:17:23.121 INFO:teuthology.orchestra.run.smithi118.stdout: "default_value": "False", 2024-03-24T03:17:23.121 INFO:teuthology.orchestra.run.smithi118.stdout: "min": "", 2024-03-24T03:17:23.121 INFO:teuthology.orchestra.run.smithi118.stdout: "max": "", 2024-03-24T03:17:23.121 INFO:teuthology.orchestra.run.smithi118.stdout: "enum_allowed": [], 2024-03-24T03:17:23.122 INFO:teuthology.orchestra.run.smithi118.stdout: "desc": "", 2024-03-24T03:17:23.122 INFO:teuthology.orchestra.run.smithi118.stdout: "long_desc": "", 2024-03-24T03:17:23.122 INFO:teuthology.orchestra.run.smithi118.stdout: "tags": [], 2024-03-24T03:17:23.122 INFO:teuthology.orchestra.run.smithi118.stdout: "see_also": [] 2024-03-24T03:17:23.122 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-03-24T03:17:23.122 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-03-24T03:17:23.122 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-24T03:17:23.122 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-24T03:17:23.122 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "volumes", 2024-03-24T03:17:23.123 INFO:teuthology.orchestra.run.smithi118.stdout: "can_run": true, 2024-03-24T03:17:23.123 INFO:teuthology.orchestra.run.smithi118.stdout: "error_string": "", 2024-03-24T03:17:23.123 INFO:teuthology.orchestra.run.smithi118.stdout: "module_options": { 2024-03-24T03:17:23.123 INFO:teuthology.orchestra.run.smithi118.stdout: "log_level": { 2024-03-24T03:17:23.123 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "log_level", 2024-03-24T03:17:23.123 INFO:teuthology.orchestra.run.smithi118.stdout: "type": "str", 2024-03-24T03:17:23.123 INFO:teuthology.orchestra.run.smithi118.stdout: "level": "advanced", 2024-03-24T03:17:23.123 INFO:teuthology.orchestra.run.smithi118.stdout: "flags": 1, 2024-03-24T03:17:23.124 INFO:teuthology.orchestra.run.smithi118.stdout: "default_value": "", 2024-03-24T03:17:23.124 INFO:teuthology.orchestra.run.smithi118.stdout: "min": "", 2024-03-24T03:17:23.124 INFO:teuthology.orchestra.run.smithi118.stdout: "max": "", 2024-03-24T03:17:23.124 INFO:teuthology.orchestra.run.smithi118.stdout: "enum_allowed": [ 2024-03-24T03:17:23.124 INFO:teuthology.orchestra.run.smithi118.stdout: "", 2024-03-24T03:17:23.124 INFO:teuthology.orchestra.run.smithi118.stdout: "critical", 2024-03-24T03:17:23.124 INFO:teuthology.orchestra.run.smithi118.stdout: "debug", 2024-03-24T03:17:23.124 INFO:teuthology.orchestra.run.smithi118.stdout: "error", 2024-03-24T03:17:23.124 INFO:teuthology.orchestra.run.smithi118.stdout: "info", 2024-03-24T03:17:23.125 INFO:teuthology.orchestra.run.smithi118.stdout: "warning" 2024-03-24T03:17:23.125 INFO:teuthology.orchestra.run.smithi118.stdout: ], 2024-03-24T03:17:23.125 INFO:teuthology.orchestra.run.smithi118.stdout: "desc": "", 2024-03-24T03:17:23.125 INFO:teuthology.orchestra.run.smithi118.stdout: "long_desc": "", 2024-03-24T03:17:23.125 INFO:teuthology.orchestra.run.smithi118.stdout: "tags": [], 2024-03-24T03:17:23.125 INFO:teuthology.orchestra.run.smithi118.stdout: "see_also": [] 2024-03-24T03:17:23.125 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-24T03:17:23.125 INFO:teuthology.orchestra.run.smithi118.stdout: "log_to_cluster": { 2024-03-24T03:17:23.126 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "log_to_cluster", 2024-03-24T03:17:23.126 INFO:teuthology.orchestra.run.smithi118.stdout: "type": "bool", 2024-03-24T03:17:23.126 INFO:teuthology.orchestra.run.smithi118.stdout: "level": "advanced", 2024-03-24T03:17:23.126 INFO:teuthology.orchestra.run.smithi118.stdout: "flags": 1, 2024-03-24T03:17:23.126 INFO:teuthology.orchestra.run.smithi118.stdout: "default_value": "False", 2024-03-24T03:17:23.126 INFO:teuthology.orchestra.run.smithi118.stdout: "min": "", 2024-03-24T03:17:23.126 INFO:teuthology.orchestra.run.smithi118.stdout: "max": "", 2024-03-24T03:17:23.126 INFO:teuthology.orchestra.run.smithi118.stdout: "enum_allowed": [], 2024-03-24T03:17:23.126 INFO:teuthology.orchestra.run.smithi118.stdout: "desc": "", 2024-03-24T03:17:23.127 INFO:teuthology.orchestra.run.smithi118.stdout: "long_desc": "", 2024-03-24T03:17:23.127 INFO:teuthology.orchestra.run.smithi118.stdout: "tags": [], 2024-03-24T03:17:23.127 INFO:teuthology.orchestra.run.smithi118.stdout: "see_also": [] 2024-03-24T03:17:23.127 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-24T03:17:23.127 INFO:teuthology.orchestra.run.smithi118.stdout: "log_to_cluster_level": { 2024-03-24T03:17:23.127 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "log_to_cluster_level", 2024-03-24T03:17:23.127 INFO:teuthology.orchestra.run.smithi118.stdout: "type": "str", 2024-03-24T03:17:23.127 INFO:teuthology.orchestra.run.smithi118.stdout: "level": "advanced", 2024-03-24T03:17:23.127 INFO:teuthology.orchestra.run.smithi118.stdout: "flags": 1, 2024-03-24T03:17:23.128 INFO:teuthology.orchestra.run.smithi118.stdout: "default_value": "info", 2024-03-24T03:17:23.128 INFO:teuthology.orchestra.run.smithi118.stdout: "min": "", 2024-03-24T03:17:23.128 INFO:teuthology.orchestra.run.smithi118.stdout: "max": "", 2024-03-24T03:17:23.128 INFO:teuthology.orchestra.run.smithi118.stdout: "enum_allowed": [ 2024-03-24T03:17:23.128 INFO:teuthology.orchestra.run.smithi118.stdout: "", 2024-03-24T03:17:23.128 INFO:teuthology.orchestra.run.smithi118.stdout: "critical", 2024-03-24T03:17:23.128 INFO:teuthology.orchestra.run.smithi118.stdout: "debug", 2024-03-24T03:17:23.128 INFO:teuthology.orchestra.run.smithi118.stdout: "error", 2024-03-24T03:17:23.128 INFO:teuthology.orchestra.run.smithi118.stdout: "info", 2024-03-24T03:17:23.128 INFO:teuthology.orchestra.run.smithi118.stdout: "warning" 2024-03-24T03:17:23.129 INFO:teuthology.orchestra.run.smithi118.stdout: ], 2024-03-24T03:17:23.129 INFO:teuthology.orchestra.run.smithi118.stdout: "desc": "", 2024-03-24T03:17:23.129 INFO:teuthology.orchestra.run.smithi118.stdout: "long_desc": "", 2024-03-24T03:17:23.129 INFO:teuthology.orchestra.run.smithi118.stdout: "tags": [], 2024-03-24T03:17:23.129 INFO:teuthology.orchestra.run.smithi118.stdout: "see_also": [] 2024-03-24T03:17:23.129 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-24T03:17:23.129 INFO:teuthology.orchestra.run.smithi118.stdout: "log_to_file": { 2024-03-24T03:17:23.129 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "log_to_file", 2024-03-24T03:17:23.129 INFO:teuthology.orchestra.run.smithi118.stdout: "type": "bool", 2024-03-24T03:17:23.130 INFO:teuthology.orchestra.run.smithi118.stdout: "level": "advanced", 2024-03-24T03:17:23.130 INFO:teuthology.orchestra.run.smithi118.stdout: "flags": 1, 2024-03-24T03:17:23.130 INFO:teuthology.orchestra.run.smithi118.stdout: "default_value": "False", 2024-03-24T03:17:23.130 INFO:teuthology.orchestra.run.smithi118.stdout: "min": "", 2024-03-24T03:17:23.130 INFO:teuthology.orchestra.run.smithi118.stdout: "max": "", 2024-03-24T03:17:23.130 INFO:teuthology.orchestra.run.smithi118.stdout: "enum_allowed": [], 2024-03-24T03:17:23.130 INFO:teuthology.orchestra.run.smithi118.stdout: "desc": "", 2024-03-24T03:17:23.130 INFO:teuthology.orchestra.run.smithi118.stdout: "long_desc": "", 2024-03-24T03:17:23.130 INFO:teuthology.orchestra.run.smithi118.stdout: "tags": [], 2024-03-24T03:17:23.131 INFO:teuthology.orchestra.run.smithi118.stdout: "see_also": [] 2024-03-24T03:17:23.131 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-24T03:17:23.131 INFO:teuthology.orchestra.run.smithi118.stdout: "max_concurrent_clones": { 2024-03-24T03:17:23.131 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "max_concurrent_clones", 2024-03-24T03:17:23.131 INFO:teuthology.orchestra.run.smithi118.stdout: "type": "int", 2024-03-24T03:17:23.131 INFO:teuthology.orchestra.run.smithi118.stdout: "level": "advanced", 2024-03-24T03:17:23.131 INFO:teuthology.orchestra.run.smithi118.stdout: "flags": 0, 2024-03-24T03:17:23.131 INFO:teuthology.orchestra.run.smithi118.stdout: "default_value": "4", 2024-03-24T03:17:23.131 INFO:teuthology.orchestra.run.smithi118.stdout: "min": "", 2024-03-24T03:17:23.132 INFO:teuthology.orchestra.run.smithi118.stdout: "max": "", 2024-03-24T03:17:23.132 INFO:teuthology.orchestra.run.smithi118.stdout: "enum_allowed": [], 2024-03-24T03:17:23.132 INFO:teuthology.orchestra.run.smithi118.stdout: "desc": "Number of asynchronous cloner threads", 2024-03-24T03:17:23.132 INFO:teuthology.orchestra.run.smithi118.stdout: "long_desc": "", 2024-03-24T03:17:23.132 INFO:teuthology.orchestra.run.smithi118.stdout: "tags": [], 2024-03-24T03:17:23.132 INFO:teuthology.orchestra.run.smithi118.stdout: "see_also": [] 2024-03-24T03:17:23.132 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-24T03:17:23.132 INFO:teuthology.orchestra.run.smithi118.stdout: "periodic_async_work": { 2024-03-24T03:17:23.132 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "periodic_async_work", 2024-03-24T03:17:23.133 INFO:teuthology.orchestra.run.smithi118.stdout: "type": "bool", 2024-03-24T03:17:23.133 INFO:teuthology.orchestra.run.smithi118.stdout: "level": "advanced", 2024-03-24T03:17:23.133 INFO:teuthology.orchestra.run.smithi118.stdout: "flags": 0, 2024-03-24T03:17:23.133 INFO:teuthology.orchestra.run.smithi118.stdout: "default_value": "False", 2024-03-24T03:17:23.133 INFO:teuthology.orchestra.run.smithi118.stdout: "min": "", 2024-03-24T03:17:23.133 INFO:teuthology.orchestra.run.smithi118.stdout: "max": "", 2024-03-24T03:17:23.133 INFO:teuthology.orchestra.run.smithi118.stdout: "enum_allowed": [], 2024-03-24T03:17:23.133 INFO:teuthology.orchestra.run.smithi118.stdout: "desc": "Periodically check for async work", 2024-03-24T03:17:23.133 INFO:teuthology.orchestra.run.smithi118.stdout: "long_desc": "", 2024-03-24T03:17:23.134 INFO:teuthology.orchestra.run.smithi118.stdout: "tags": [], 2024-03-24T03:17:23.134 INFO:teuthology.orchestra.run.smithi118.stdout: "see_also": [] 2024-03-24T03:17:23.134 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-24T03:17:23.134 INFO:teuthology.orchestra.run.smithi118.stdout: "snapshot_clone_delay": { 2024-03-24T03:17:23.134 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "snapshot_clone_delay", 2024-03-24T03:17:23.134 INFO:teuthology.orchestra.run.smithi118.stdout: "type": "int", 2024-03-24T03:17:23.134 INFO:teuthology.orchestra.run.smithi118.stdout: "level": "advanced", 2024-03-24T03:17:23.134 INFO:teuthology.orchestra.run.smithi118.stdout: "flags": 0, 2024-03-24T03:17:23.135 INFO:teuthology.orchestra.run.smithi118.stdout: "default_value": "0", 2024-03-24T03:17:23.135 INFO:teuthology.orchestra.run.smithi118.stdout: "min": "", 2024-03-24T03:17:23.135 INFO:teuthology.orchestra.run.smithi118.stdout: "max": "", 2024-03-24T03:17:23.135 INFO:teuthology.orchestra.run.smithi118.stdout: "enum_allowed": [], 2024-03-24T03:17:23.135 INFO:teuthology.orchestra.run.smithi118.stdout: "desc": "Delay clone begin operation by snapshot_clone_delay seconds", 2024-03-24T03:17:23.135 INFO:teuthology.orchestra.run.smithi118.stdout: "long_desc": "", 2024-03-24T03:17:23.135 INFO:teuthology.orchestra.run.smithi118.stdout: "tags": [], 2024-03-24T03:17:23.135 INFO:teuthology.orchestra.run.smithi118.stdout: "see_also": [] 2024-03-24T03:17:23.135 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-24T03:17:23.136 INFO:teuthology.orchestra.run.smithi118.stdout: "snapshot_clone_no_wait": { 2024-03-24T03:17:23.136 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "snapshot_clone_no_wait", 2024-03-24T03:17:23.136 INFO:teuthology.orchestra.run.smithi118.stdout: "type": "bool", 2024-03-24T03:17:23.136 INFO:teuthology.orchestra.run.smithi118.stdout: "level": "advanced", 2024-03-24T03:17:23.136 INFO:teuthology.orchestra.run.smithi118.stdout: "flags": 0, 2024-03-24T03:17:23.136 INFO:teuthology.orchestra.run.smithi118.stdout: "default_value": "True", 2024-03-24T03:17:23.136 INFO:teuthology.orchestra.run.smithi118.stdout: "min": "", 2024-03-24T03:17:23.136 INFO:teuthology.orchestra.run.smithi118.stdout: "max": "", 2024-03-24T03:17:23.136 INFO:teuthology.orchestra.run.smithi118.stdout: "enum_allowed": [], 2024-03-24T03:17:23.136 INFO:teuthology.orchestra.run.smithi118.stdout: "desc": "Reject subvolume clone request when cloner threads are busy", 2024-03-24T03:17:23.137 INFO:teuthology.orchestra.run.smithi118.stdout: "long_desc": "", 2024-03-24T03:17:23.137 INFO:teuthology.orchestra.run.smithi118.stdout: "tags": [], 2024-03-24T03:17:23.137 INFO:teuthology.orchestra.run.smithi118.stdout: "see_also": [] 2024-03-24T03:17:23.137 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-03-24T03:17:23.137 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-03-24T03:17:23.137 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-24T03:17:23.137 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-24T03:17:23.137 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "zabbix", 2024-03-24T03:17:23.138 INFO:teuthology.orchestra.run.smithi118.stdout: "can_run": true, 2024-03-24T03:17:23.138 INFO:teuthology.orchestra.run.smithi118.stdout: "error_string": "", 2024-03-24T03:17:23.138 INFO:teuthology.orchestra.run.smithi118.stdout: "module_options": { 2024-03-24T03:17:23.138 INFO:teuthology.orchestra.run.smithi118.stdout: "discovery_interval": { 2024-03-24T03:17:23.138 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "discovery_interval", 2024-03-24T03:17:23.138 INFO:teuthology.orchestra.run.smithi118.stdout: "type": "uint", 2024-03-24T03:17:23.138 INFO:teuthology.orchestra.run.smithi118.stdout: "level": "advanced", 2024-03-24T03:17:23.138 INFO:teuthology.orchestra.run.smithi118.stdout: "flags": 0, 2024-03-24T03:17:23.138 INFO:teuthology.orchestra.run.smithi118.stdout: "default_value": "100", 2024-03-24T03:17:23.138 INFO:teuthology.orchestra.run.smithi118.stdout: "min": "", 2024-03-24T03:17:23.139 INFO:teuthology.orchestra.run.smithi118.stdout: "max": "", 2024-03-24T03:17:23.139 INFO:teuthology.orchestra.run.smithi118.stdout: "enum_allowed": [], 2024-03-24T03:17:23.139 INFO:teuthology.orchestra.run.smithi118.stdout: "desc": "", 2024-03-24T03:17:23.139 INFO:teuthology.orchestra.run.smithi118.stdout: "long_desc": "", 2024-03-24T03:17:23.139 INFO:teuthology.orchestra.run.smithi118.stdout: "tags": [], 2024-03-24T03:17:23.139 INFO:teuthology.orchestra.run.smithi118.stdout: "see_also": [] 2024-03-24T03:17:23.139 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-24T03:17:23.139 INFO:teuthology.orchestra.run.smithi118.stdout: "identifier": { 2024-03-24T03:17:23.139 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "identifier", 2024-03-24T03:17:23.140 INFO:teuthology.orchestra.run.smithi118.stdout: "type": "str", 2024-03-24T03:17:23.140 INFO:teuthology.orchestra.run.smithi118.stdout: "level": "advanced", 2024-03-24T03:17:23.140 INFO:teuthology.orchestra.run.smithi118.stdout: "flags": 0, 2024-03-24T03:17:23.140 INFO:teuthology.orchestra.run.smithi118.stdout: "default_value": "", 2024-03-24T03:17:23.140 INFO:teuthology.orchestra.run.smithi118.stdout: "min": "", 2024-03-24T03:17:23.140 INFO:teuthology.orchestra.run.smithi118.stdout: "max": "", 2024-03-24T03:17:23.140 INFO:teuthology.orchestra.run.smithi118.stdout: "enum_allowed": [], 2024-03-24T03:17:23.140 INFO:teuthology.orchestra.run.smithi118.stdout: "desc": "", 2024-03-24T03:17:23.140 INFO:teuthology.orchestra.run.smithi118.stdout: "long_desc": "", 2024-03-24T03:17:23.140 INFO:teuthology.orchestra.run.smithi118.stdout: "tags": [], 2024-03-24T03:17:23.141 INFO:teuthology.orchestra.run.smithi118.stdout: "see_also": [] 2024-03-24T03:17:23.141 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-24T03:17:23.141 INFO:teuthology.orchestra.run.smithi118.stdout: "interval": { 2024-03-24T03:17:23.141 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "interval", 2024-03-24T03:17:23.141 INFO:teuthology.orchestra.run.smithi118.stdout: "type": "secs", 2024-03-24T03:17:23.141 INFO:teuthology.orchestra.run.smithi118.stdout: "level": "advanced", 2024-03-24T03:17:23.141 INFO:teuthology.orchestra.run.smithi118.stdout: "flags": 0, 2024-03-24T03:17:23.141 INFO:teuthology.orchestra.run.smithi118.stdout: "default_value": "60", 2024-03-24T03:17:23.141 INFO:teuthology.orchestra.run.smithi118.stdout: "min": "", 2024-03-24T03:17:23.142 INFO:teuthology.orchestra.run.smithi118.stdout: "max": "", 2024-03-24T03:17:23.142 INFO:teuthology.orchestra.run.smithi118.stdout: "enum_allowed": [], 2024-03-24T03:17:23.142 INFO:teuthology.orchestra.run.smithi118.stdout: "desc": "", 2024-03-24T03:17:23.142 INFO:teuthology.orchestra.run.smithi118.stdout: "long_desc": "", 2024-03-24T03:17:23.142 INFO:teuthology.orchestra.run.smithi118.stdout: "tags": [], 2024-03-24T03:17:23.142 INFO:teuthology.orchestra.run.smithi118.stdout: "see_also": [] 2024-03-24T03:17:23.142 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-24T03:17:23.142 INFO:teuthology.orchestra.run.smithi118.stdout: "log_level": { 2024-03-24T03:17:23.143 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "log_level", 2024-03-24T03:17:23.143 INFO:teuthology.orchestra.run.smithi118.stdout: "type": "str", 2024-03-24T03:17:23.143 INFO:teuthology.orchestra.run.smithi118.stdout: "level": "advanced", 2024-03-24T03:17:23.143 INFO:teuthology.orchestra.run.smithi118.stdout: "flags": 1, 2024-03-24T03:17:23.143 INFO:teuthology.orchestra.run.smithi118.stdout: "default_value": "", 2024-03-24T03:17:23.143 INFO:teuthology.orchestra.run.smithi118.stdout: "min": "", 2024-03-24T03:17:23.143 INFO:teuthology.orchestra.run.smithi118.stdout: "max": "", 2024-03-24T03:17:23.143 INFO:teuthology.orchestra.run.smithi118.stdout: "enum_allowed": [ 2024-03-24T03:17:23.143 INFO:teuthology.orchestra.run.smithi118.stdout: "", 2024-03-24T03:17:23.144 INFO:teuthology.orchestra.run.smithi118.stdout: "critical", 2024-03-24T03:17:23.144 INFO:teuthology.orchestra.run.smithi118.stdout: "debug", 2024-03-24T03:17:23.144 INFO:teuthology.orchestra.run.smithi118.stdout: "error", 2024-03-24T03:17:23.144 INFO:teuthology.orchestra.run.smithi118.stdout: "info", 2024-03-24T03:17:23.144 INFO:teuthology.orchestra.run.smithi118.stdout: "warning" 2024-03-24T03:17:23.144 INFO:teuthology.orchestra.run.smithi118.stdout: ], 2024-03-24T03:17:23.144 INFO:teuthology.orchestra.run.smithi118.stdout: "desc": "", 2024-03-24T03:17:23.144 INFO:teuthology.orchestra.run.smithi118.stdout: "long_desc": "", 2024-03-24T03:17:23.145 INFO:teuthology.orchestra.run.smithi118.stdout: "tags": [], 2024-03-24T03:17:23.145 INFO:teuthology.orchestra.run.smithi118.stdout: "see_also": [] 2024-03-24T03:17:23.145 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-24T03:17:23.145 INFO:teuthology.orchestra.run.smithi118.stdout: "log_to_cluster": { 2024-03-24T03:17:23.145 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "log_to_cluster", 2024-03-24T03:17:23.145 INFO:teuthology.orchestra.run.smithi118.stdout: "type": "bool", 2024-03-24T03:17:23.145 INFO:teuthology.orchestra.run.smithi118.stdout: "level": "advanced", 2024-03-24T03:17:23.145 INFO:teuthology.orchestra.run.smithi118.stdout: "flags": 1, 2024-03-24T03:17:23.145 INFO:teuthology.orchestra.run.smithi118.stdout: "default_value": "False", 2024-03-24T03:17:23.145 INFO:teuthology.orchestra.run.smithi118.stdout: "min": "", 2024-03-24T03:17:23.146 INFO:teuthology.orchestra.run.smithi118.stdout: "max": "", 2024-03-24T03:17:23.146 INFO:teuthology.orchestra.run.smithi118.stdout: "enum_allowed": [], 2024-03-24T03:17:23.146 INFO:teuthology.orchestra.run.smithi118.stdout: "desc": "", 2024-03-24T03:17:23.146 INFO:teuthology.orchestra.run.smithi118.stdout: "long_desc": "", 2024-03-24T03:17:23.146 INFO:teuthology.orchestra.run.smithi118.stdout: "tags": [], 2024-03-24T03:17:23.146 INFO:teuthology.orchestra.run.smithi118.stdout: "see_also": [] 2024-03-24T03:17:23.146 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-24T03:17:23.146 INFO:teuthology.orchestra.run.smithi118.stdout: "log_to_cluster_level": { 2024-03-24T03:17:23.146 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "log_to_cluster_level", 2024-03-24T03:17:23.147 INFO:teuthology.orchestra.run.smithi118.stdout: "type": "str", 2024-03-24T03:17:23.147 INFO:teuthology.orchestra.run.smithi118.stdout: "level": "advanced", 2024-03-24T03:17:23.147 INFO:teuthology.orchestra.run.smithi118.stdout: "flags": 1, 2024-03-24T03:17:23.147 INFO:teuthology.orchestra.run.smithi118.stdout: "default_value": "info", 2024-03-24T03:17:23.147 INFO:teuthology.orchestra.run.smithi118.stdout: "min": "", 2024-03-24T03:17:23.147 INFO:teuthology.orchestra.run.smithi118.stdout: "max": "", 2024-03-24T03:17:23.147 INFO:teuthology.orchestra.run.smithi118.stdout: "enum_allowed": [ 2024-03-24T03:17:23.147 INFO:teuthology.orchestra.run.smithi118.stdout: "", 2024-03-24T03:17:23.147 INFO:teuthology.orchestra.run.smithi118.stdout: "critical", 2024-03-24T03:17:23.147 INFO:teuthology.orchestra.run.smithi118.stdout: "debug", 2024-03-24T03:17:23.148 INFO:teuthology.orchestra.run.smithi118.stdout: "error", 2024-03-24T03:17:23.148 INFO:teuthology.orchestra.run.smithi118.stdout: "info", 2024-03-24T03:17:23.148 INFO:teuthology.orchestra.run.smithi118.stdout: "warning" 2024-03-24T03:17:23.148 INFO:teuthology.orchestra.run.smithi118.stdout: ], 2024-03-24T03:17:23.148 INFO:teuthology.orchestra.run.smithi118.stdout: "desc": "", 2024-03-24T03:17:23.148 INFO:teuthology.orchestra.run.smithi118.stdout: "long_desc": "", 2024-03-24T03:17:23.148 INFO:teuthology.orchestra.run.smithi118.stdout: "tags": [], 2024-03-24T03:17:23.148 INFO:teuthology.orchestra.run.smithi118.stdout: "see_also": [] 2024-03-24T03:17:23.148 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-24T03:17:23.149 INFO:teuthology.orchestra.run.smithi118.stdout: "log_to_file": { 2024-03-24T03:17:23.149 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "log_to_file", 2024-03-24T03:17:23.149 INFO:teuthology.orchestra.run.smithi118.stdout: "type": "bool", 2024-03-24T03:17:23.149 INFO:teuthology.orchestra.run.smithi118.stdout: "level": "advanced", 2024-03-24T03:17:23.149 INFO:teuthology.orchestra.run.smithi118.stdout: "flags": 1, 2024-03-24T03:17:23.149 INFO:teuthology.orchestra.run.smithi118.stdout: "default_value": "False", 2024-03-24T03:17:23.149 INFO:teuthology.orchestra.run.smithi118.stdout: "min": "", 2024-03-24T03:17:23.149 INFO:teuthology.orchestra.run.smithi118.stdout: "max": "", 2024-03-24T03:17:23.149 INFO:teuthology.orchestra.run.smithi118.stdout: "enum_allowed": [], 2024-03-24T03:17:23.150 INFO:teuthology.orchestra.run.smithi118.stdout: "desc": "", 2024-03-24T03:17:23.150 INFO:teuthology.orchestra.run.smithi118.stdout: "long_desc": "", 2024-03-24T03:17:23.150 INFO:teuthology.orchestra.run.smithi118.stdout: "tags": [], 2024-03-24T03:17:23.150 INFO:teuthology.orchestra.run.smithi118.stdout: "see_also": [] 2024-03-24T03:17:23.150 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-24T03:17:23.150 INFO:teuthology.orchestra.run.smithi118.stdout: "zabbix_host": { 2024-03-24T03:17:23.150 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "zabbix_host", 2024-03-24T03:17:23.150 INFO:teuthology.orchestra.run.smithi118.stdout: "type": "str", 2024-03-24T03:17:23.150 INFO:teuthology.orchestra.run.smithi118.stdout: "level": "advanced", 2024-03-24T03:17:23.151 INFO:teuthology.orchestra.run.smithi118.stdout: "flags": 0, 2024-03-24T03:17:23.151 INFO:teuthology.orchestra.run.smithi118.stdout: "default_value": "", 2024-03-24T03:17:23.151 INFO:teuthology.orchestra.run.smithi118.stdout: "min": "", 2024-03-24T03:17:23.151 INFO:teuthology.orchestra.run.smithi118.stdout: "max": "", 2024-03-24T03:17:23.151 INFO:teuthology.orchestra.run.smithi118.stdout: "enum_allowed": [], 2024-03-24T03:17:23.151 INFO:teuthology.orchestra.run.smithi118.stdout: "desc": "", 2024-03-24T03:17:23.151 INFO:teuthology.orchestra.run.smithi118.stdout: "long_desc": "", 2024-03-24T03:17:23.151 INFO:teuthology.orchestra.run.smithi118.stdout: "tags": [], 2024-03-24T03:17:23.151 INFO:teuthology.orchestra.run.smithi118.stdout: "see_also": [] 2024-03-24T03:17:23.152 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-24T03:17:23.152 INFO:teuthology.orchestra.run.smithi118.stdout: "zabbix_port": { 2024-03-24T03:17:23.152 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "zabbix_port", 2024-03-24T03:17:23.152 INFO:teuthology.orchestra.run.smithi118.stdout: "type": "int", 2024-03-24T03:17:23.152 INFO:teuthology.orchestra.run.smithi118.stdout: "level": "advanced", 2024-03-24T03:17:23.152 INFO:teuthology.orchestra.run.smithi118.stdout: "flags": 0, 2024-03-24T03:17:23.152 INFO:teuthology.orchestra.run.smithi118.stdout: "default_value": "10051", 2024-03-24T03:17:23.152 INFO:teuthology.orchestra.run.smithi118.stdout: "min": "", 2024-03-24T03:17:23.152 INFO:teuthology.orchestra.run.smithi118.stdout: "max": "", 2024-03-24T03:17:23.152 INFO:teuthology.orchestra.run.smithi118.stdout: "enum_allowed": [], 2024-03-24T03:17:23.153 INFO:teuthology.orchestra.run.smithi118.stdout: "desc": "", 2024-03-24T03:17:23.153 INFO:teuthology.orchestra.run.smithi118.stdout: "long_desc": "", 2024-03-24T03:17:23.153 INFO:teuthology.orchestra.run.smithi118.stdout: "tags": [], 2024-03-24T03:17:23.153 INFO:teuthology.orchestra.run.smithi118.stdout: "see_also": [] 2024-03-24T03:17:23.153 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-24T03:17:23.153 INFO:teuthology.orchestra.run.smithi118.stdout: "zabbix_sender": { 2024-03-24T03:17:23.153 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "zabbix_sender", 2024-03-24T03:17:23.153 INFO:teuthology.orchestra.run.smithi118.stdout: "type": "str", 2024-03-24T03:17:23.153 INFO:teuthology.orchestra.run.smithi118.stdout: "level": "advanced", 2024-03-24T03:17:23.154 INFO:teuthology.orchestra.run.smithi118.stdout: "flags": 0, 2024-03-24T03:17:23.154 INFO:teuthology.orchestra.run.smithi118.stdout: "default_value": "/usr/bin/zabbix_sender", 2024-03-24T03:17:23.154 INFO:teuthology.orchestra.run.smithi118.stdout: "min": "", 2024-03-24T03:17:23.154 INFO:teuthology.orchestra.run.smithi118.stdout: "max": "", 2024-03-24T03:17:23.154 INFO:teuthology.orchestra.run.smithi118.stdout: "enum_allowed": [], 2024-03-24T03:17:23.154 INFO:teuthology.orchestra.run.smithi118.stdout: "desc": "", 2024-03-24T03:17:23.154 INFO:teuthology.orchestra.run.smithi118.stdout: "long_desc": "", 2024-03-24T03:17:23.155 INFO:teuthology.orchestra.run.smithi118.stdout: "tags": [], 2024-03-24T03:17:23.155 INFO:teuthology.orchestra.run.smithi118.stdout: "see_also": [] 2024-03-24T03:17:23.155 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-03-24T03:17:23.155 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-03-24T03:17:23.155 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-03-24T03:17:23.155 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-03-24T03:17:23.155 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-24T03:17:23.155 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-24T03:17:23.155 INFO:teuthology.orchestra.run.smithi118.stdout: "gid": 24116, 2024-03-24T03:17:23.156 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "y", 2024-03-24T03:17:23.156 INFO:teuthology.orchestra.run.smithi118.stdout: "mgr_features": 4540701547738038271, 2024-03-24T03:17:23.156 INFO:teuthology.orchestra.run.smithi118.stdout: "available_modules": [ 2024-03-24T03:17:23.156 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-24T03:17:23.156 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "alerts", 2024-03-24T03:17:23.156 INFO:teuthology.orchestra.run.smithi118.stdout: "can_run": true, 2024-03-24T03:17:23.156 INFO:teuthology.orchestra.run.smithi118.stdout: "error_string": "", 2024-03-24T03:17:23.156 INFO:teuthology.orchestra.run.smithi118.stdout: "module_options": { 2024-03-24T03:17:23.156 INFO:teuthology.orchestra.run.smithi118.stdout: "interval": { 2024-03-24T03:17:23.157 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "interval", 2024-03-24T03:17:23.157 INFO:teuthology.orchestra.run.smithi118.stdout: "type": "secs", 2024-03-24T03:17:23.157 INFO:teuthology.orchestra.run.smithi118.stdout: "level": "advanced", 2024-03-24T03:17:23.157 INFO:teuthology.orchestra.run.smithi118.stdout: "flags": 1, 2024-03-24T03:17:23.157 INFO:teuthology.orchestra.run.smithi118.stdout: "default_value": "60", 2024-03-24T03:17:23.157 INFO:teuthology.orchestra.run.smithi118.stdout: "min": "", 2024-03-24T03:17:23.157 INFO:teuthology.orchestra.run.smithi118.stdout: "max": "", 2024-03-24T03:17:23.157 INFO:teuthology.orchestra.run.smithi118.stdout: "enum_allowed": [], 2024-03-24T03:17:23.158 INFO:teuthology.orchestra.run.smithi118.stdout: "desc": "How frequently to reexamine health status", 2024-03-24T03:17:23.158 INFO:teuthology.orchestra.run.smithi118.stdout: "long_desc": "", 2024-03-24T03:17:23.158 INFO:teuthology.orchestra.run.smithi118.stdout: "tags": [], 2024-03-24T03:17:23.158 INFO:teuthology.orchestra.run.smithi118.stdout: "see_also": [] 2024-03-24T03:17:23.158 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-24T03:17:23.158 INFO:teuthology.orchestra.run.smithi118.stdout: "log_level": { 2024-03-24T03:17:23.158 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "log_level", 2024-03-24T03:17:23.158 INFO:teuthology.orchestra.run.smithi118.stdout: "type": "str", 2024-03-24T03:17:23.158 INFO:teuthology.orchestra.run.smithi118.stdout: "level": "advanced", 2024-03-24T03:17:23.159 INFO:teuthology.orchestra.run.smithi118.stdout: "flags": 1, 2024-03-24T03:17:23.159 INFO:teuthology.orchestra.run.smithi118.stdout: "default_value": "", 2024-03-24T03:17:23.159 INFO:teuthology.orchestra.run.smithi118.stdout: "min": "", 2024-03-24T03:17:23.159 INFO:teuthology.orchestra.run.smithi118.stdout: "max": "", 2024-03-24T03:17:23.159 INFO:teuthology.orchestra.run.smithi118.stdout: "enum_allowed": [ 2024-03-24T03:17:23.159 INFO:teuthology.orchestra.run.smithi118.stdout: "", 2024-03-24T03:17:23.159 INFO:teuthology.orchestra.run.smithi118.stdout: "critical", 2024-03-24T03:17:23.159 INFO:teuthology.orchestra.run.smithi118.stdout: "debug", 2024-03-24T03:17:23.159 INFO:teuthology.orchestra.run.smithi118.stdout: "error", 2024-03-24T03:17:23.160 INFO:teuthology.orchestra.run.smithi118.stdout: "info", 2024-03-24T03:17:23.160 INFO:teuthology.orchestra.run.smithi118.stdout: "warning" 2024-03-24T03:17:23.160 INFO:teuthology.orchestra.run.smithi118.stdout: ], 2024-03-24T03:17:23.160 INFO:teuthology.orchestra.run.smithi118.stdout: "desc": "", 2024-03-24T03:17:23.160 INFO:teuthology.orchestra.run.smithi118.stdout: "long_desc": "", 2024-03-24T03:17:23.160 INFO:teuthology.orchestra.run.smithi118.stdout: "tags": [], 2024-03-24T03:17:23.160 INFO:teuthology.orchestra.run.smithi118.stdout: "see_also": [] 2024-03-24T03:17:23.160 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-24T03:17:23.160 INFO:teuthology.orchestra.run.smithi118.stdout: "log_to_cluster": { 2024-03-24T03:17:23.161 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "log_to_cluster", 2024-03-24T03:17:23.161 INFO:teuthology.orchestra.run.smithi118.stdout: "type": "bool", 2024-03-24T03:17:23.161 INFO:teuthology.orchestra.run.smithi118.stdout: "level": "advanced", 2024-03-24T03:17:23.161 INFO:teuthology.orchestra.run.smithi118.stdout: "flags": 1, 2024-03-24T03:17:23.161 INFO:teuthology.orchestra.run.smithi118.stdout: "default_value": "False", 2024-03-24T03:17:23.161 INFO:teuthology.orchestra.run.smithi118.stdout: "min": "", 2024-03-24T03:17:23.161 INFO:teuthology.orchestra.run.smithi118.stdout: "max": "", 2024-03-24T03:17:23.161 INFO:teuthology.orchestra.run.smithi118.stdout: "enum_allowed": [], 2024-03-24T03:17:23.161 INFO:teuthology.orchestra.run.smithi118.stdout: "desc": "", 2024-03-24T03:17:23.162 INFO:teuthology.orchestra.run.smithi118.stdout: "long_desc": "", 2024-03-24T03:17:23.162 INFO:teuthology.orchestra.run.smithi118.stdout: "tags": [], 2024-03-24T03:17:23.162 INFO:teuthology.orchestra.run.smithi118.stdout: "see_also": [] 2024-03-24T03:17:23.162 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-24T03:17:23.162 INFO:teuthology.orchestra.run.smithi118.stdout: "log_to_cluster_level": { 2024-03-24T03:17:23.162 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "log_to_cluster_level", 2024-03-24T03:17:23.162 INFO:teuthology.orchestra.run.smithi118.stdout: "type": "str", 2024-03-24T03:17:23.163 INFO:teuthology.orchestra.run.smithi118.stdout: "level": "advanced", 2024-03-24T03:17:23.163 INFO:teuthology.orchestra.run.smithi118.stdout: "flags": 1, 2024-03-24T03:17:23.163 INFO:teuthology.orchestra.run.smithi118.stdout: "default_value": "info", 2024-03-24T03:17:23.163 INFO:teuthology.orchestra.run.smithi118.stdout: "min": "", 2024-03-24T03:17:23.163 INFO:teuthology.orchestra.run.smithi118.stdout: "max": "", 2024-03-24T03:17:23.163 INFO:teuthology.orchestra.run.smithi118.stdout: "enum_allowed": [ 2024-03-24T03:17:23.163 INFO:teuthology.orchestra.run.smithi118.stdout: "", 2024-03-24T03:17:23.163 INFO:teuthology.orchestra.run.smithi118.stdout: "critical", 2024-03-24T03:17:23.164 INFO:teuthology.orchestra.run.smithi118.stdout: "debug", 2024-03-24T03:17:23.164 INFO:teuthology.orchestra.run.smithi118.stdout: "error", 2024-03-24T03:17:23.164 INFO:teuthology.orchestra.run.smithi118.stdout: "info", 2024-03-24T03:17:23.164 INFO:teuthology.orchestra.run.smithi118.stdout: "warning" 2024-03-24T03:17:23.164 INFO:teuthology.orchestra.run.smithi118.stdout: ], 2024-03-24T03:17:23.164 INFO:teuthology.orchestra.run.smithi118.stdout: "desc": "", 2024-03-24T03:17:23.164 INFO:teuthology.orchestra.run.smithi118.stdout: "long_desc": "", 2024-03-24T03:17:23.164 INFO:teuthology.orchestra.run.smithi118.stdout: "tags": [], 2024-03-24T03:17:23.164 INFO:teuthology.orchestra.run.smithi118.stdout: "see_also": [] 2024-03-24T03:17:23.165 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-24T03:17:23.165 INFO:teuthology.orchestra.run.smithi118.stdout: "log_to_file": { 2024-03-24T03:17:23.165 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "log_to_file", 2024-03-24T03:17:23.165 INFO:teuthology.orchestra.run.smithi118.stdout: "type": "bool", 2024-03-24T03:17:23.165 INFO:teuthology.orchestra.run.smithi118.stdout: "level": "advanced", 2024-03-24T03:17:23.165 INFO:teuthology.orchestra.run.smithi118.stdout: "flags": 1, 2024-03-24T03:17:23.165 INFO:teuthology.orchestra.run.smithi118.stdout: "default_value": "False", 2024-03-24T03:17:23.165 INFO:teuthology.orchestra.run.smithi118.stdout: "min": "", 2024-03-24T03:17:23.165 INFO:teuthology.orchestra.run.smithi118.stdout: "max": "", 2024-03-24T03:17:23.166 INFO:teuthology.orchestra.run.smithi118.stdout: "enum_allowed": [], 2024-03-24T03:17:23.166 INFO:teuthology.orchestra.run.smithi118.stdout: "desc": "", 2024-03-24T03:17:23.166 INFO:teuthology.orchestra.run.smithi118.stdout: "long_desc": "", 2024-03-24T03:17:23.166 INFO:teuthology.orchestra.run.smithi118.stdout: "tags": [], 2024-03-24T03:17:23.166 INFO:teuthology.orchestra.run.smithi118.stdout: "see_also": [] 2024-03-24T03:17:23.166 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-24T03:17:23.166 INFO:teuthology.orchestra.run.smithi118.stdout: "smtp_destination": { 2024-03-24T03:17:23.166 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "smtp_destination", 2024-03-24T03:17:23.166 INFO:teuthology.orchestra.run.smithi118.stdout: "type": "str", 2024-03-24T03:17:23.167 INFO:teuthology.orchestra.run.smithi118.stdout: "level": "advanced", 2024-03-24T03:17:23.167 INFO:teuthology.orchestra.run.smithi118.stdout: "flags": 1, 2024-03-24T03:17:23.167 INFO:teuthology.orchestra.run.smithi118.stdout: "default_value": "", 2024-03-24T03:17:23.167 INFO:teuthology.orchestra.run.smithi118.stdout: "min": "", 2024-03-24T03:17:23.167 INFO:teuthology.orchestra.run.smithi118.stdout: "max": "", 2024-03-24T03:17:23.167 INFO:teuthology.orchestra.run.smithi118.stdout: "enum_allowed": [], 2024-03-24T03:17:23.167 INFO:teuthology.orchestra.run.smithi118.stdout: "desc": "Email address to send alerts to", 2024-03-24T03:17:23.167 INFO:teuthology.orchestra.run.smithi118.stdout: "long_desc": "", 2024-03-24T03:17:23.167 INFO:teuthology.orchestra.run.smithi118.stdout: "tags": [], 2024-03-24T03:17:23.168 INFO:teuthology.orchestra.run.smithi118.stdout: "see_also": [] 2024-03-24T03:17:23.168 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-24T03:17:23.168 INFO:teuthology.orchestra.run.smithi118.stdout: "smtp_from_name": { 2024-03-24T03:17:23.168 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "smtp_from_name", 2024-03-24T03:17:23.168 INFO:teuthology.orchestra.run.smithi118.stdout: "type": "str", 2024-03-24T03:17:23.168 INFO:teuthology.orchestra.run.smithi118.stdout: "level": "advanced", 2024-03-24T03:17:23.168 INFO:teuthology.orchestra.run.smithi118.stdout: "flags": 1, 2024-03-24T03:17:23.168 INFO:teuthology.orchestra.run.smithi118.stdout: "default_value": "Ceph", 2024-03-24T03:17:23.168 INFO:teuthology.orchestra.run.smithi118.stdout: "min": "", 2024-03-24T03:17:23.169 INFO:teuthology.orchestra.run.smithi118.stdout: "max": "", 2024-03-24T03:17:23.169 INFO:teuthology.orchestra.run.smithi118.stdout: "enum_allowed": [], 2024-03-24T03:17:23.169 INFO:teuthology.orchestra.run.smithi118.stdout: "desc": "Email From: name", 2024-03-24T03:17:23.169 INFO:teuthology.orchestra.run.smithi118.stdout: "long_desc": "", 2024-03-24T03:17:23.169 INFO:teuthology.orchestra.run.smithi118.stdout: "tags": [], 2024-03-24T03:17:23.169 INFO:teuthology.orchestra.run.smithi118.stdout: "see_also": [] 2024-03-24T03:17:23.169 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-24T03:17:23.169 INFO:teuthology.orchestra.run.smithi118.stdout: "smtp_host": { 2024-03-24T03:17:23.169 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "smtp_host", 2024-03-24T03:17:23.169 INFO:teuthology.orchestra.run.smithi118.stdout: "type": "str", 2024-03-24T03:17:23.170 INFO:teuthology.orchestra.run.smithi118.stdout: "level": "advanced", 2024-03-24T03:17:23.170 INFO:teuthology.orchestra.run.smithi118.stdout: "flags": 1, 2024-03-24T03:17:23.170 INFO:teuthology.orchestra.run.smithi118.stdout: "default_value": "", 2024-03-24T03:17:23.170 INFO:teuthology.orchestra.run.smithi118.stdout: "min": "", 2024-03-24T03:17:23.170 INFO:teuthology.orchestra.run.smithi118.stdout: "max": "", 2024-03-24T03:17:23.170 INFO:teuthology.orchestra.run.smithi118.stdout: "enum_allowed": [], 2024-03-24T03:17:23.170 INFO:teuthology.orchestra.run.smithi118.stdout: "desc": "SMTP server", 2024-03-24T03:17:23.170 INFO:teuthology.orchestra.run.smithi118.stdout: "long_desc": "", 2024-03-24T03:17:23.170 INFO:teuthology.orchestra.run.smithi118.stdout: "tags": [], 2024-03-24T03:17:23.171 INFO:teuthology.orchestra.run.smithi118.stdout: "see_also": [] 2024-03-24T03:17:23.171 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-24T03:17:23.171 INFO:teuthology.orchestra.run.smithi118.stdout: "smtp_password": { 2024-03-24T03:17:23.171 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "smtp_password", 2024-03-24T03:17:23.171 INFO:teuthology.orchestra.run.smithi118.stdout: "type": "str", 2024-03-24T03:17:23.171 INFO:teuthology.orchestra.run.smithi118.stdout: "level": "advanced", 2024-03-24T03:17:23.171 INFO:teuthology.orchestra.run.smithi118.stdout: "flags": 1, 2024-03-24T03:17:23.171 INFO:teuthology.orchestra.run.smithi118.stdout: "default_value": "", 2024-03-24T03:17:23.171 INFO:teuthology.orchestra.run.smithi118.stdout: "min": "", 2024-03-24T03:17:23.172 INFO:teuthology.orchestra.run.smithi118.stdout: "max": "", 2024-03-24T03:17:23.172 INFO:teuthology.orchestra.run.smithi118.stdout: "enum_allowed": [], 2024-03-24T03:17:23.172 INFO:teuthology.orchestra.run.smithi118.stdout: "desc": "Password to authenticate with", 2024-03-24T03:17:23.172 INFO:teuthology.orchestra.run.smithi118.stdout: "long_desc": "", 2024-03-24T03:17:23.172 INFO:teuthology.orchestra.run.smithi118.stdout: "tags": [], 2024-03-24T03:17:23.172 INFO:teuthology.orchestra.run.smithi118.stdout: "see_also": [] 2024-03-24T03:17:23.172 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-24T03:17:23.172 INFO:teuthology.orchestra.run.smithi118.stdout: "smtp_port": { 2024-03-24T03:17:23.173 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "smtp_port", 2024-03-24T03:17:23.173 INFO:teuthology.orchestra.run.smithi118.stdout: "type": "int", 2024-03-24T03:17:23.173 INFO:teuthology.orchestra.run.smithi118.stdout: "level": "advanced", 2024-03-24T03:17:23.173 INFO:teuthology.orchestra.run.smithi118.stdout: "flags": 1, 2024-03-24T03:17:23.173 INFO:teuthology.orchestra.run.smithi118.stdout: "default_value": "465", 2024-03-24T03:17:23.173 INFO:teuthology.orchestra.run.smithi118.stdout: "min": "", 2024-03-24T03:17:23.173 INFO:teuthology.orchestra.run.smithi118.stdout: "max": "", 2024-03-24T03:17:23.173 INFO:teuthology.orchestra.run.smithi118.stdout: "enum_allowed": [], 2024-03-24T03:17:23.173 INFO:teuthology.orchestra.run.smithi118.stdout: "desc": "SMTP port", 2024-03-24T03:17:23.174 INFO:teuthology.orchestra.run.smithi118.stdout: "long_desc": "", 2024-03-24T03:17:23.174 INFO:teuthology.orchestra.run.smithi118.stdout: "tags": [], 2024-03-24T03:17:23.174 INFO:teuthology.orchestra.run.smithi118.stdout: "see_also": [] 2024-03-24T03:17:23.174 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-24T03:17:23.174 INFO:teuthology.orchestra.run.smithi118.stdout: "smtp_sender": { 2024-03-24T03:17:23.174 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "smtp_sender", 2024-03-24T03:17:23.174 INFO:teuthology.orchestra.run.smithi118.stdout: "type": "str", 2024-03-24T03:17:23.174 INFO:teuthology.orchestra.run.smithi118.stdout: "level": "advanced", 2024-03-24T03:17:23.175 INFO:teuthology.orchestra.run.smithi118.stdout: "flags": 1, 2024-03-24T03:17:23.175 INFO:teuthology.orchestra.run.smithi118.stdout: "default_value": "", 2024-03-24T03:17:23.175 INFO:teuthology.orchestra.run.smithi118.stdout: "min": "", 2024-03-24T03:17:23.175 INFO:teuthology.orchestra.run.smithi118.stdout: "max": "", 2024-03-24T03:17:23.175 INFO:teuthology.orchestra.run.smithi118.stdout: "enum_allowed": [], 2024-03-24T03:17:23.175 INFO:teuthology.orchestra.run.smithi118.stdout: "desc": "SMTP envelope sender", 2024-03-24T03:17:23.175 INFO:teuthology.orchestra.run.smithi118.stdout: "long_desc": "", 2024-03-24T03:17:23.175 INFO:teuthology.orchestra.run.smithi118.stdout: "tags": [], 2024-03-24T03:17:23.176 INFO:teuthology.orchestra.run.smithi118.stdout: "see_also": [] 2024-03-24T03:17:23.176 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-24T03:17:23.176 INFO:teuthology.orchestra.run.smithi118.stdout: "smtp_ssl": { 2024-03-24T03:17:23.176 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "smtp_ssl", 2024-03-24T03:17:23.176 INFO:teuthology.orchestra.run.smithi118.stdout: "type": "bool", 2024-03-24T03:17:23.176 INFO:teuthology.orchestra.run.smithi118.stdout: "level": "advanced", 2024-03-24T03:17:23.176 INFO:teuthology.orchestra.run.smithi118.stdout: "flags": 1, 2024-03-24T03:17:23.176 INFO:teuthology.orchestra.run.smithi118.stdout: "default_value": "True", 2024-03-24T03:17:23.176 INFO:teuthology.orchestra.run.smithi118.stdout: "min": "", 2024-03-24T03:17:23.177 INFO:teuthology.orchestra.run.smithi118.stdout: "max": "", 2024-03-24T03:17:23.177 INFO:teuthology.orchestra.run.smithi118.stdout: "enum_allowed": [], 2024-03-24T03:17:23.177 INFO:teuthology.orchestra.run.smithi118.stdout: "desc": "Use SSL to connect to SMTP server", 2024-03-24T03:17:23.177 INFO:teuthology.orchestra.run.smithi118.stdout: "long_desc": "", 2024-03-24T03:17:23.177 INFO:teuthology.orchestra.run.smithi118.stdout: "tags": [], 2024-03-24T03:17:23.177 INFO:teuthology.orchestra.run.smithi118.stdout: "see_also": [] 2024-03-24T03:17:23.177 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-24T03:17:23.177 INFO:teuthology.orchestra.run.smithi118.stdout: "smtp_user": { 2024-03-24T03:17:23.177 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "smtp_user", 2024-03-24T03:17:23.178 INFO:teuthology.orchestra.run.smithi118.stdout: "type": "str", 2024-03-24T03:17:23.178 INFO:teuthology.orchestra.run.smithi118.stdout: "level": "advanced", 2024-03-24T03:17:23.178 INFO:teuthology.orchestra.run.smithi118.stdout: "flags": 1, 2024-03-24T03:17:23.178 INFO:teuthology.orchestra.run.smithi118.stdout: "default_value": "", 2024-03-24T03:17:23.178 INFO:teuthology.orchestra.run.smithi118.stdout: "min": "", 2024-03-24T03:17:23.178 INFO:teuthology.orchestra.run.smithi118.stdout: "max": "", 2024-03-24T03:17:23.178 INFO:teuthology.orchestra.run.smithi118.stdout: "enum_allowed": [], 2024-03-24T03:17:23.178 INFO:teuthology.orchestra.run.smithi118.stdout: "desc": "User to authenticate as", 2024-03-24T03:17:23.179 INFO:teuthology.orchestra.run.smithi118.stdout: "long_desc": "", 2024-03-24T03:17:23.179 INFO:teuthology.orchestra.run.smithi118.stdout: "tags": [], 2024-03-24T03:17:23.179 INFO:teuthology.orchestra.run.smithi118.stdout: "see_also": [] 2024-03-24T03:17:23.179 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-03-24T03:17:23.179 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-03-24T03:17:23.179 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-24T03:17:23.179 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-24T03:17:23.179 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "balancer", 2024-03-24T03:17:23.179 INFO:teuthology.orchestra.run.smithi118.stdout: "can_run": true, 2024-03-24T03:17:23.179 INFO:teuthology.orchestra.run.smithi118.stdout: "error_string": "", 2024-03-24T03:17:23.180 INFO:teuthology.orchestra.run.smithi118.stdout: "module_options": { 2024-03-24T03:17:23.180 INFO:teuthology.orchestra.run.smithi118.stdout: "active": { 2024-03-24T03:17:23.180 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "active", 2024-03-24T03:17:23.180 INFO:teuthology.orchestra.run.smithi118.stdout: "type": "bool", 2024-03-24T03:17:23.180 INFO:teuthology.orchestra.run.smithi118.stdout: "level": "advanced", 2024-03-24T03:17:23.180 INFO:teuthology.orchestra.run.smithi118.stdout: "flags": 1, 2024-03-24T03:17:23.180 INFO:teuthology.orchestra.run.smithi118.stdout: "default_value": "True", 2024-03-24T03:17:23.180 INFO:teuthology.orchestra.run.smithi118.stdout: "min": "", 2024-03-24T03:17:23.180 INFO:teuthology.orchestra.run.smithi118.stdout: "max": "", 2024-03-24T03:17:23.181 INFO:teuthology.orchestra.run.smithi118.stdout: "enum_allowed": [], 2024-03-24T03:17:23.181 INFO:teuthology.orchestra.run.smithi118.stdout: "desc": "automatically balance PGs across cluster", 2024-03-24T03:17:23.181 INFO:teuthology.orchestra.run.smithi118.stdout: "long_desc": "", 2024-03-24T03:17:23.181 INFO:teuthology.orchestra.run.smithi118.stdout: "tags": [], 2024-03-24T03:17:23.181 INFO:teuthology.orchestra.run.smithi118.stdout: "see_also": [] 2024-03-24T03:17:23.181 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-24T03:17:23.181 INFO:teuthology.orchestra.run.smithi118.stdout: "begin_time": { 2024-03-24T03:17:23.181 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "begin_time", 2024-03-24T03:17:23.181 INFO:teuthology.orchestra.run.smithi118.stdout: "type": "str", 2024-03-24T03:17:23.182 INFO:teuthology.orchestra.run.smithi118.stdout: "level": "advanced", 2024-03-24T03:17:23.182 INFO:teuthology.orchestra.run.smithi118.stdout: "flags": 1, 2024-03-24T03:17:23.182 INFO:teuthology.orchestra.run.smithi118.stdout: "default_value": "0000", 2024-03-24T03:17:23.182 INFO:teuthology.orchestra.run.smithi118.stdout: "min": "", 2024-03-24T03:17:23.182 INFO:teuthology.orchestra.run.smithi118.stdout: "max": "", 2024-03-24T03:17:23.182 INFO:teuthology.orchestra.run.smithi118.stdout: "enum_allowed": [], 2024-03-24T03:17:23.182 INFO:teuthology.orchestra.run.smithi118.stdout: "desc": "beginning time of day to automatically balance", 2024-03-24T03:17:23.182 INFO:teuthology.orchestra.run.smithi118.stdout: "long_desc": "This is a time of day in the format HHMM.", 2024-03-24T03:17:23.182 INFO:teuthology.orchestra.run.smithi118.stdout: "tags": [], 2024-03-24T03:17:23.183 INFO:teuthology.orchestra.run.smithi118.stdout: "see_also": [] 2024-03-24T03:17:23.183 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-24T03:17:23.183 INFO:teuthology.orchestra.run.smithi118.stdout: "begin_weekday": { 2024-03-24T03:17:23.183 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "begin_weekday", 2024-03-24T03:17:23.183 INFO:teuthology.orchestra.run.smithi118.stdout: "type": "uint", 2024-03-24T03:17:23.183 INFO:teuthology.orchestra.run.smithi118.stdout: "level": "advanced", 2024-03-24T03:17:23.183 INFO:teuthology.orchestra.run.smithi118.stdout: "flags": 1, 2024-03-24T03:17:23.183 INFO:teuthology.orchestra.run.smithi118.stdout: "default_value": "0", 2024-03-24T03:17:23.183 INFO:teuthology.orchestra.run.smithi118.stdout: "min": "0", 2024-03-24T03:17:23.184 INFO:teuthology.orchestra.run.smithi118.stdout: "max": "6", 2024-03-24T03:17:23.184 INFO:teuthology.orchestra.run.smithi118.stdout: "enum_allowed": [], 2024-03-24T03:17:23.184 INFO:teuthology.orchestra.run.smithi118.stdout: "desc": "Restrict automatic balancing to this day of the week or later", 2024-03-24T03:17:23.184 INFO:teuthology.orchestra.run.smithi118.stdout: "long_desc": "0 = Sunday, 1 = Monday, etc.", 2024-03-24T03:17:23.184 INFO:teuthology.orchestra.run.smithi118.stdout: "tags": [], 2024-03-24T03:17:23.184 INFO:teuthology.orchestra.run.smithi118.stdout: "see_also": [] 2024-03-24T03:17:23.184 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-24T03:17:23.184 INFO:teuthology.orchestra.run.smithi118.stdout: "crush_compat_max_iterations": { 2024-03-24T03:17:23.184 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "crush_compat_max_iterations", 2024-03-24T03:17:23.185 INFO:teuthology.orchestra.run.smithi118.stdout: "type": "uint", 2024-03-24T03:17:23.185 INFO:teuthology.orchestra.run.smithi118.stdout: "level": "advanced", 2024-03-24T03:17:23.185 INFO:teuthology.orchestra.run.smithi118.stdout: "flags": 1, 2024-03-24T03:17:23.185 INFO:teuthology.orchestra.run.smithi118.stdout: "default_value": "25", 2024-03-24T03:17:23.185 INFO:teuthology.orchestra.run.smithi118.stdout: "min": "1", 2024-03-24T03:17:23.185 INFO:teuthology.orchestra.run.smithi118.stdout: "max": "250", 2024-03-24T03:17:23.185 INFO:teuthology.orchestra.run.smithi118.stdout: "enum_allowed": [], 2024-03-24T03:17:23.185 INFO:teuthology.orchestra.run.smithi118.stdout: "desc": "maximum number of iterations to attempt optimization", 2024-03-24T03:17:23.186 INFO:teuthology.orchestra.run.smithi118.stdout: "long_desc": "", 2024-03-24T03:17:23.186 INFO:teuthology.orchestra.run.smithi118.stdout: "tags": [], 2024-03-24T03:17:23.186 INFO:teuthology.orchestra.run.smithi118.stdout: "see_also": [] 2024-03-24T03:17:23.186 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-24T03:17:23.186 INFO:teuthology.orchestra.run.smithi118.stdout: "crush_compat_metrics": { 2024-03-24T03:17:23.186 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "crush_compat_metrics", 2024-03-24T03:17:23.186 INFO:teuthology.orchestra.run.smithi118.stdout: "type": "str", 2024-03-24T03:17:23.186 INFO:teuthology.orchestra.run.smithi118.stdout: "level": "advanced", 2024-03-24T03:17:23.187 INFO:teuthology.orchestra.run.smithi118.stdout: "flags": 1, 2024-03-24T03:17:23.187 INFO:teuthology.orchestra.run.smithi118.stdout: "default_value": "pgs,objects,bytes", 2024-03-24T03:17:23.187 INFO:teuthology.orchestra.run.smithi118.stdout: "min": "", 2024-03-24T03:17:23.187 INFO:teuthology.orchestra.run.smithi118.stdout: "max": "", 2024-03-24T03:17:23.187 INFO:teuthology.orchestra.run.smithi118.stdout: "enum_allowed": [], 2024-03-24T03:17:23.187 INFO:teuthology.orchestra.run.smithi118.stdout: "desc": "metrics with which to calculate OSD utilization", 2024-03-24T03:17:23.187 INFO:teuthology.orchestra.run.smithi118.stdout: "long_desc": "Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.", 2024-03-24T03:17:23.187 INFO:teuthology.orchestra.run.smithi118.stdout: "tags": [], 2024-03-24T03:17:23.187 INFO:teuthology.orchestra.run.smithi118.stdout: "see_also": [] 2024-03-24T03:17:23.187 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-24T03:17:23.188 INFO:teuthology.orchestra.run.smithi118.stdout: "crush_compat_step": { 2024-03-24T03:17:23.188 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "crush_compat_step", 2024-03-24T03:17:23.188 INFO:teuthology.orchestra.run.smithi118.stdout: "type": "float", 2024-03-24T03:17:23.188 INFO:teuthology.orchestra.run.smithi118.stdout: "level": "advanced", 2024-03-24T03:17:23.188 INFO:teuthology.orchestra.run.smithi118.stdout: "flags": 1, 2024-03-24T03:17:23.188 INFO:teuthology.orchestra.run.smithi118.stdout: "default_value": "0.5", 2024-03-24T03:17:23.188 INFO:teuthology.orchestra.run.smithi118.stdout: "min": "0.001", 2024-03-24T03:17:23.188 INFO:teuthology.orchestra.run.smithi118.stdout: "max": "0.999", 2024-03-24T03:17:23.188 INFO:teuthology.orchestra.run.smithi118.stdout: "enum_allowed": [], 2024-03-24T03:17:23.189 INFO:teuthology.orchestra.run.smithi118.stdout: "desc": "aggressiveness of optimization", 2024-03-24T03:17:23.189 INFO:teuthology.orchestra.run.smithi118.stdout: "long_desc": ".99 is very aggressive, .01 is less aggressive", 2024-03-24T03:17:23.189 INFO:teuthology.orchestra.run.smithi118.stdout: "tags": [], 2024-03-24T03:17:23.189 INFO:teuthology.orchestra.run.smithi118.stdout: "see_also": [] 2024-03-24T03:17:23.189 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-24T03:17:23.189 INFO:teuthology.orchestra.run.smithi118.stdout: "end_time": { 2024-03-24T03:17:23.189 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "end_time", 2024-03-24T03:17:23.190 INFO:teuthology.orchestra.run.smithi118.stdout: "type": "str", 2024-03-24T03:17:23.190 INFO:teuthology.orchestra.run.smithi118.stdout: "level": "advanced", 2024-03-24T03:17:23.190 INFO:teuthology.orchestra.run.smithi118.stdout: "flags": 1, 2024-03-24T03:17:23.190 INFO:teuthology.orchestra.run.smithi118.stdout: "default_value": "2359", 2024-03-24T03:17:23.190 INFO:teuthology.orchestra.run.smithi118.stdout: "min": "", 2024-03-24T03:17:23.190 INFO:teuthology.orchestra.run.smithi118.stdout: "max": "", 2024-03-24T03:17:23.190 INFO:teuthology.orchestra.run.smithi118.stdout: "enum_allowed": [], 2024-03-24T03:17:23.190 INFO:teuthology.orchestra.run.smithi118.stdout: "desc": "ending time of day to automatically balance", 2024-03-24T03:17:23.191 INFO:teuthology.orchestra.run.smithi118.stdout: "long_desc": "This is a time of day in the format HHMM.", 2024-03-24T03:17:23.191 INFO:teuthology.orchestra.run.smithi118.stdout: "tags": [], 2024-03-24T03:17:23.191 INFO:teuthology.orchestra.run.smithi118.stdout: "see_also": [] 2024-03-24T03:17:23.191 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-24T03:17:23.191 INFO:teuthology.orchestra.run.smithi118.stdout: "end_weekday": { 2024-03-24T03:17:23.191 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "end_weekday", 2024-03-24T03:17:23.191 INFO:teuthology.orchestra.run.smithi118.stdout: "type": "uint", 2024-03-24T03:17:23.191 INFO:teuthology.orchestra.run.smithi118.stdout: "level": "advanced", 2024-03-24T03:17:23.191 INFO:teuthology.orchestra.run.smithi118.stdout: "flags": 1, 2024-03-24T03:17:23.192 INFO:teuthology.orchestra.run.smithi118.stdout: "default_value": "0", 2024-03-24T03:17:23.192 INFO:teuthology.orchestra.run.smithi118.stdout: "min": "0", 2024-03-24T03:17:23.192 INFO:teuthology.orchestra.run.smithi118.stdout: "max": "6", 2024-03-24T03:17:23.192 INFO:teuthology.orchestra.run.smithi118.stdout: "enum_allowed": [], 2024-03-24T03:17:23.192 INFO:teuthology.orchestra.run.smithi118.stdout: "desc": "Restrict automatic balancing to days of the week earlier than this", 2024-03-24T03:17:23.192 INFO:teuthology.orchestra.run.smithi118.stdout: "long_desc": "0 = Sunday, 1 = Monday, etc.", 2024-03-24T03:17:23.192 INFO:teuthology.orchestra.run.smithi118.stdout: "tags": [], 2024-03-24T03:17:23.192 INFO:teuthology.orchestra.run.smithi118.stdout: "see_also": [] 2024-03-24T03:17:23.192 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-24T03:17:23.193 INFO:teuthology.orchestra.run.smithi118.stdout: "log_level": { 2024-03-24T03:17:23.193 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "log_level", 2024-03-24T03:17:23.193 INFO:teuthology.orchestra.run.smithi118.stdout: "type": "str", 2024-03-24T03:17:23.193 INFO:teuthology.orchestra.run.smithi118.stdout: "level": "advanced", 2024-03-24T03:17:23.193 INFO:teuthology.orchestra.run.smithi118.stdout: "flags": 1, 2024-03-24T03:17:23.193 INFO:teuthology.orchestra.run.smithi118.stdout: "default_value": "", 2024-03-24T03:17:23.193 INFO:teuthology.orchestra.run.smithi118.stdout: "min": "", 2024-03-24T03:17:23.193 INFO:teuthology.orchestra.run.smithi118.stdout: "max": "", 2024-03-24T03:17:23.194 INFO:teuthology.orchestra.run.smithi118.stdout: "enum_allowed": [ 2024-03-24T03:17:23.194 INFO:teuthology.orchestra.run.smithi118.stdout: "", 2024-03-24T03:17:23.194 INFO:teuthology.orchestra.run.smithi118.stdout: "critical", 2024-03-24T03:17:23.194 INFO:teuthology.orchestra.run.smithi118.stdout: "debug", 2024-03-24T03:17:23.194 INFO:teuthology.orchestra.run.smithi118.stdout: "error", 2024-03-24T03:17:23.194 INFO:teuthology.orchestra.run.smithi118.stdout: "info", 2024-03-24T03:17:23.194 INFO:teuthology.orchestra.run.smithi118.stdout: "warning" 2024-03-24T03:17:23.194 INFO:teuthology.orchestra.run.smithi118.stdout: ], 2024-03-24T03:17:23.195 INFO:teuthology.orchestra.run.smithi118.stdout: "desc": "", 2024-03-24T03:17:23.195 INFO:teuthology.orchestra.run.smithi118.stdout: "long_desc": "", 2024-03-24T03:17:23.195 INFO:teuthology.orchestra.run.smithi118.stdout: "tags": [], 2024-03-24T03:17:23.195 INFO:teuthology.orchestra.run.smithi118.stdout: "see_also": [] 2024-03-24T03:17:23.195 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-24T03:17:23.195 INFO:teuthology.orchestra.run.smithi118.stdout: "log_to_cluster": { 2024-03-24T03:17:23.195 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "log_to_cluster", 2024-03-24T03:17:23.195 INFO:teuthology.orchestra.run.smithi118.stdout: "type": "bool", 2024-03-24T03:17:23.196 INFO:teuthology.orchestra.run.smithi118.stdout: "level": "advanced", 2024-03-24T03:17:23.196 INFO:teuthology.orchestra.run.smithi118.stdout: "flags": 1, 2024-03-24T03:17:23.196 INFO:teuthology.orchestra.run.smithi118.stdout: "default_value": "False", 2024-03-24T03:17:23.196 INFO:teuthology.orchestra.run.smithi118.stdout: "min": "", 2024-03-24T03:17:23.196 INFO:teuthology.orchestra.run.smithi118.stdout: "max": "", 2024-03-24T03:17:23.196 INFO:teuthology.orchestra.run.smithi118.stdout: "enum_allowed": [], 2024-03-24T03:17:23.196 INFO:teuthology.orchestra.run.smithi118.stdout: "desc": "", 2024-03-24T03:17:23.196 INFO:teuthology.orchestra.run.smithi118.stdout: "long_desc": "", 2024-03-24T03:17:23.196 INFO:teuthology.orchestra.run.smithi118.stdout: "tags": [], 2024-03-24T03:17:23.197 INFO:teuthology.orchestra.run.smithi118.stdout: "see_also": [] 2024-03-24T03:17:23.197 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-24T03:17:23.197 INFO:teuthology.orchestra.run.smithi118.stdout: "log_to_cluster_level": { 2024-03-24T03:17:23.197 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "log_to_cluster_level", 2024-03-24T03:17:23.197 INFO:teuthology.orchestra.run.smithi118.stdout: "type": "str", 2024-03-24T03:17:23.197 INFO:teuthology.orchestra.run.smithi118.stdout: "level": "advanced", 2024-03-24T03:17:23.197 INFO:teuthology.orchestra.run.smithi118.stdout: "flags": 1, 2024-03-24T03:17:23.197 INFO:teuthology.orchestra.run.smithi118.stdout: "default_value": "info", 2024-03-24T03:17:23.198 INFO:teuthology.orchestra.run.smithi118.stdout: "min": "", 2024-03-24T03:17:23.198 INFO:teuthology.orchestra.run.smithi118.stdout: "max": "", 2024-03-24T03:17:23.198 INFO:teuthology.orchestra.run.smithi118.stdout: "enum_allowed": [ 2024-03-24T03:17:23.198 INFO:teuthology.orchestra.run.smithi118.stdout: "", 2024-03-24T03:17:23.198 INFO:teuthology.orchestra.run.smithi118.stdout: "critical", 2024-03-24T03:17:23.198 INFO:teuthology.orchestra.run.smithi118.stdout: "debug", 2024-03-24T03:17:23.198 INFO:teuthology.orchestra.run.smithi118.stdout: "error", 2024-03-24T03:17:23.198 INFO:teuthology.orchestra.run.smithi118.stdout: "info", 2024-03-24T03:17:23.199 INFO:teuthology.orchestra.run.smithi118.stdout: "warning" 2024-03-24T03:17:23.199 INFO:teuthology.orchestra.run.smithi118.stdout: ], 2024-03-24T03:17:23.199 INFO:teuthology.orchestra.run.smithi118.stdout: "desc": "", 2024-03-24T03:17:23.199 INFO:teuthology.orchestra.run.smithi118.stdout: "long_desc": "", 2024-03-24T03:17:23.199 INFO:teuthology.orchestra.run.smithi118.stdout: "tags": [], 2024-03-24T03:17:23.199 INFO:teuthology.orchestra.run.smithi118.stdout: "see_also": [] 2024-03-24T03:17:23.199 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-24T03:17:23.199 INFO:teuthology.orchestra.run.smithi118.stdout: "log_to_file": { 2024-03-24T03:17:23.199 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "log_to_file", 2024-03-24T03:17:23.200 INFO:teuthology.orchestra.run.smithi118.stdout: "type": "bool", 2024-03-24T03:17:23.200 INFO:teuthology.orchestra.run.smithi118.stdout: "level": "advanced", 2024-03-24T03:17:23.200 INFO:teuthology.orchestra.run.smithi118.stdout: "flags": 1, 2024-03-24T03:17:23.200 INFO:teuthology.orchestra.run.smithi118.stdout: "default_value": "False", 2024-03-24T03:17:23.200 INFO:teuthology.orchestra.run.smithi118.stdout: "min": "", 2024-03-24T03:17:23.200 INFO:teuthology.orchestra.run.smithi118.stdout: "max": "", 2024-03-24T03:17:23.200 INFO:teuthology.orchestra.run.smithi118.stdout: "enum_allowed": [], 2024-03-24T03:17:23.200 INFO:teuthology.orchestra.run.smithi118.stdout: "desc": "", 2024-03-24T03:17:23.200 INFO:teuthology.orchestra.run.smithi118.stdout: "long_desc": "", 2024-03-24T03:17:23.201 INFO:teuthology.orchestra.run.smithi118.stdout: "tags": [], 2024-03-24T03:17:23.201 INFO:teuthology.orchestra.run.smithi118.stdout: "see_also": [] 2024-03-24T03:17:23.201 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-24T03:17:23.201 INFO:teuthology.orchestra.run.smithi118.stdout: "min_score": { 2024-03-24T03:17:23.201 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "min_score", 2024-03-24T03:17:23.201 INFO:teuthology.orchestra.run.smithi118.stdout: "type": "float", 2024-03-24T03:17:23.201 INFO:teuthology.orchestra.run.smithi118.stdout: "level": "advanced", 2024-03-24T03:17:23.201 INFO:teuthology.orchestra.run.smithi118.stdout: "flags": 1, 2024-03-24T03:17:23.202 INFO:teuthology.orchestra.run.smithi118.stdout: "default_value": "0", 2024-03-24T03:17:23.202 INFO:teuthology.orchestra.run.smithi118.stdout: "min": "", 2024-03-24T03:17:23.202 INFO:teuthology.orchestra.run.smithi118.stdout: "max": "", 2024-03-24T03:17:23.202 INFO:teuthology.orchestra.run.smithi118.stdout: "enum_allowed": [], 2024-03-24T03:17:23.202 INFO:teuthology.orchestra.run.smithi118.stdout: "desc": "minimum score, below which no optimization is attempted", 2024-03-24T03:17:23.202 INFO:teuthology.orchestra.run.smithi118.stdout: "long_desc": "", 2024-03-24T03:17:23.202 INFO:teuthology.orchestra.run.smithi118.stdout: "tags": [], 2024-03-24T03:17:23.202 INFO:teuthology.orchestra.run.smithi118.stdout: "see_also": [] 2024-03-24T03:17:23.203 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-24T03:17:23.203 INFO:teuthology.orchestra.run.smithi118.stdout: "mode": { 2024-03-24T03:17:23.203 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "mode", 2024-03-24T03:17:23.203 INFO:teuthology.orchestra.run.smithi118.stdout: "type": "str", 2024-03-24T03:17:23.203 INFO:teuthology.orchestra.run.smithi118.stdout: "level": "advanced", 2024-03-24T03:17:23.203 INFO:teuthology.orchestra.run.smithi118.stdout: "flags": 1, 2024-03-24T03:17:23.203 INFO:teuthology.orchestra.run.smithi118.stdout: "default_value": "upmap", 2024-03-24T03:17:23.203 INFO:teuthology.orchestra.run.smithi118.stdout: "min": "", 2024-03-24T03:17:23.203 INFO:teuthology.orchestra.run.smithi118.stdout: "max": "", 2024-03-24T03:17:23.204 INFO:teuthology.orchestra.run.smithi118.stdout: "enum_allowed": [ 2024-03-24T03:17:23.204 INFO:teuthology.orchestra.run.smithi118.stdout: "crush-compat", 2024-03-24T03:17:23.204 INFO:teuthology.orchestra.run.smithi118.stdout: "none", 2024-03-24T03:17:23.204 INFO:teuthology.orchestra.run.smithi118.stdout: "read", 2024-03-24T03:17:23.204 INFO:teuthology.orchestra.run.smithi118.stdout: "upmap", 2024-03-24T03:17:23.204 INFO:teuthology.orchestra.run.smithi118.stdout: "upmap-read" 2024-03-24T03:17:23.204 INFO:teuthology.orchestra.run.smithi118.stdout: ], 2024-03-24T03:17:23.204 INFO:teuthology.orchestra.run.smithi118.stdout: "desc": "Balancer mode", 2024-03-24T03:17:23.205 INFO:teuthology.orchestra.run.smithi118.stdout: "long_desc": "", 2024-03-24T03:17:23.205 INFO:teuthology.orchestra.run.smithi118.stdout: "tags": [], 2024-03-24T03:17:23.205 INFO:teuthology.orchestra.run.smithi118.stdout: "see_also": [] 2024-03-24T03:17:23.205 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-24T03:17:23.205 INFO:teuthology.orchestra.run.smithi118.stdout: "pool_ids": { 2024-03-24T03:17:23.205 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "pool_ids", 2024-03-24T03:17:23.205 INFO:teuthology.orchestra.run.smithi118.stdout: "type": "str", 2024-03-24T03:17:23.205 INFO:teuthology.orchestra.run.smithi118.stdout: "level": "advanced", 2024-03-24T03:17:23.206 INFO:teuthology.orchestra.run.smithi118.stdout: "flags": 1, 2024-03-24T03:17:23.206 INFO:teuthology.orchestra.run.smithi118.stdout: "default_value": "", 2024-03-24T03:17:23.206 INFO:teuthology.orchestra.run.smithi118.stdout: "min": "", 2024-03-24T03:17:23.206 INFO:teuthology.orchestra.run.smithi118.stdout: "max": "", 2024-03-24T03:17:23.206 INFO:teuthology.orchestra.run.smithi118.stdout: "enum_allowed": [], 2024-03-24T03:17:23.206 INFO:teuthology.orchestra.run.smithi118.stdout: "desc": "pools which the automatic balancing will be limited to", 2024-03-24T03:17:23.206 INFO:teuthology.orchestra.run.smithi118.stdout: "long_desc": "", 2024-03-24T03:17:23.207 INFO:teuthology.orchestra.run.smithi118.stdout: "tags": [], 2024-03-24T03:17:23.207 INFO:teuthology.orchestra.run.smithi118.stdout: "see_also": [] 2024-03-24T03:17:23.207 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-24T03:17:23.207 INFO:teuthology.orchestra.run.smithi118.stdout: "sleep_interval": { 2024-03-24T03:17:23.207 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "sleep_interval", 2024-03-24T03:17:23.207 INFO:teuthology.orchestra.run.smithi118.stdout: "type": "secs", 2024-03-24T03:17:23.207 INFO:teuthology.orchestra.run.smithi118.stdout: "level": "advanced", 2024-03-24T03:17:23.207 INFO:teuthology.orchestra.run.smithi118.stdout: "flags": 1, 2024-03-24T03:17:23.207 INFO:teuthology.orchestra.run.smithi118.stdout: "default_value": "60", 2024-03-24T03:17:23.208 INFO:teuthology.orchestra.run.smithi118.stdout: "min": "", 2024-03-24T03:17:23.208 INFO:teuthology.orchestra.run.smithi118.stdout: "max": "", 2024-03-24T03:17:23.208 INFO:teuthology.orchestra.run.smithi118.stdout: "enum_allowed": [], 2024-03-24T03:17:23.208 INFO:teuthology.orchestra.run.smithi118.stdout: "desc": "how frequently to wake up and attempt optimization", 2024-03-24T03:17:23.208 INFO:teuthology.orchestra.run.smithi118.stdout: "long_desc": "", 2024-03-24T03:17:23.208 INFO:teuthology.orchestra.run.smithi118.stdout: "tags": [], 2024-03-24T03:17:23.208 INFO:teuthology.orchestra.run.smithi118.stdout: "see_also": [] 2024-03-24T03:17:23.208 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-24T03:17:23.208 INFO:teuthology.orchestra.run.smithi118.stdout: "upmap_max_deviation": { 2024-03-24T03:17:23.209 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "upmap_max_deviation", 2024-03-24T03:17:23.209 INFO:teuthology.orchestra.run.smithi118.stdout: "type": "int", 2024-03-24T03:17:23.209 INFO:teuthology.orchestra.run.smithi118.stdout: "level": "advanced", 2024-03-24T03:17:23.209 INFO:teuthology.orchestra.run.smithi118.stdout: "flags": 1, 2024-03-24T03:17:23.209 INFO:teuthology.orchestra.run.smithi118.stdout: "default_value": "5", 2024-03-24T03:17:23.209 INFO:teuthology.orchestra.run.smithi118.stdout: "min": "1", 2024-03-24T03:17:23.209 INFO:teuthology.orchestra.run.smithi118.stdout: "max": "", 2024-03-24T03:17:23.209 INFO:teuthology.orchestra.run.smithi118.stdout: "enum_allowed": [], 2024-03-24T03:17:23.209 INFO:teuthology.orchestra.run.smithi118.stdout: "desc": "deviation below which no optimization is attempted", 2024-03-24T03:17:23.210 INFO:teuthology.orchestra.run.smithi118.stdout: "long_desc": "If the number of PGs are within this count then no optimization is attempted", 2024-03-24T03:17:23.210 INFO:teuthology.orchestra.run.smithi118.stdout: "tags": [], 2024-03-24T03:17:23.210 INFO:teuthology.orchestra.run.smithi118.stdout: "see_also": [] 2024-03-24T03:17:23.210 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-24T03:17:23.210 INFO:teuthology.orchestra.run.smithi118.stdout: "upmap_max_optimizations": { 2024-03-24T03:17:23.210 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "upmap_max_optimizations", 2024-03-24T03:17:23.210 INFO:teuthology.orchestra.run.smithi118.stdout: "type": "uint", 2024-03-24T03:17:23.210 INFO:teuthology.orchestra.run.smithi118.stdout: "level": "advanced", 2024-03-24T03:17:23.211 INFO:teuthology.orchestra.run.smithi118.stdout: "flags": 1, 2024-03-24T03:17:23.211 INFO:teuthology.orchestra.run.smithi118.stdout: "default_value": "10", 2024-03-24T03:17:23.211 INFO:teuthology.orchestra.run.smithi118.stdout: "min": "", 2024-03-24T03:17:23.211 INFO:teuthology.orchestra.run.smithi118.stdout: "max": "", 2024-03-24T03:17:23.211 INFO:teuthology.orchestra.run.smithi118.stdout: "enum_allowed": [], 2024-03-24T03:17:23.211 INFO:teuthology.orchestra.run.smithi118.stdout: "desc": "maximum upmap optimizations to make per attempt", 2024-03-24T03:17:23.211 INFO:teuthology.orchestra.run.smithi118.stdout: "long_desc": "", 2024-03-24T03:17:23.211 INFO:teuthology.orchestra.run.smithi118.stdout: "tags": [], 2024-03-24T03:17:23.211 INFO:teuthology.orchestra.run.smithi118.stdout: "see_also": [] 2024-03-24T03:17:23.212 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-03-24T03:17:23.212 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-03-24T03:17:23.212 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-24T03:17:23.212 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-24T03:17:23.212 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "cephadm", 2024-03-24T03:17:23.212 INFO:teuthology.orchestra.run.smithi118.stdout: "can_run": true, 2024-03-24T03:17:23.212 INFO:teuthology.orchestra.run.smithi118.stdout: "error_string": "", 2024-03-24T03:17:23.212 INFO:teuthology.orchestra.run.smithi118.stdout: "module_options": { 2024-03-24T03:17:23.213 INFO:teuthology.orchestra.run.smithi118.stdout: "agent_down_multiplier": { 2024-03-24T03:17:23.213 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "agent_down_multiplier", 2024-03-24T03:17:23.213 INFO:teuthology.orchestra.run.smithi118.stdout: "type": "float", 2024-03-24T03:17:23.213 INFO:teuthology.orchestra.run.smithi118.stdout: "level": "advanced", 2024-03-24T03:17:23.213 INFO:teuthology.orchestra.run.smithi118.stdout: "flags": 0, 2024-03-24T03:17:23.213 INFO:teuthology.orchestra.run.smithi118.stdout: "default_value": "3.0", 2024-03-24T03:17:23.213 INFO:teuthology.orchestra.run.smithi118.stdout: "min": "", 2024-03-24T03:17:23.213 INFO:teuthology.orchestra.run.smithi118.stdout: "max": "", 2024-03-24T03:17:23.213 INFO:teuthology.orchestra.run.smithi118.stdout: "enum_allowed": [], 2024-03-24T03:17:23.214 INFO:teuthology.orchestra.run.smithi118.stdout: "desc": "Multiplied by agent refresh rate to calculate how long agent must not report before being marked down", 2024-03-24T03:17:23.214 INFO:teuthology.orchestra.run.smithi118.stdout: "long_desc": "", 2024-03-24T03:17:23.214 INFO:teuthology.orchestra.run.smithi118.stdout: "tags": [], 2024-03-24T03:17:23.214 INFO:teuthology.orchestra.run.smithi118.stdout: "see_also": [] 2024-03-24T03:17:23.214 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-24T03:17:23.214 INFO:teuthology.orchestra.run.smithi118.stdout: "agent_refresh_rate": { 2024-03-24T03:17:23.214 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "agent_refresh_rate", 2024-03-24T03:17:23.215 INFO:teuthology.orchestra.run.smithi118.stdout: "type": "secs", 2024-03-24T03:17:23.215 INFO:teuthology.orchestra.run.smithi118.stdout: "level": "advanced", 2024-03-24T03:17:23.215 INFO:teuthology.orchestra.run.smithi118.stdout: "flags": 0, 2024-03-24T03:17:23.215 INFO:teuthology.orchestra.run.smithi118.stdout: "default_value": "20", 2024-03-24T03:17:23.215 INFO:teuthology.orchestra.run.smithi118.stdout: "min": "", 2024-03-24T03:17:23.215 INFO:teuthology.orchestra.run.smithi118.stdout: "max": "", 2024-03-24T03:17:23.215 INFO:teuthology.orchestra.run.smithi118.stdout: "enum_allowed": [], 2024-03-24T03:17:23.215 INFO:teuthology.orchestra.run.smithi118.stdout: "desc": "How often agent on each host will try to gather and send metadata", 2024-03-24T03:17:23.215 INFO:teuthology.orchestra.run.smithi118.stdout: "long_desc": "", 2024-03-24T03:17:23.216 INFO:teuthology.orchestra.run.smithi118.stdout: "tags": [], 2024-03-24T03:17:23.216 INFO:teuthology.orchestra.run.smithi118.stdout: "see_also": [] 2024-03-24T03:17:23.216 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-24T03:17:23.216 INFO:teuthology.orchestra.run.smithi118.stdout: "agent_starting_port": { 2024-03-24T03:17:23.216 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "agent_starting_port", 2024-03-24T03:17:23.216 INFO:teuthology.orchestra.run.smithi118.stdout: "type": "int", 2024-03-24T03:17:23.216 INFO:teuthology.orchestra.run.smithi118.stdout: "level": "advanced", 2024-03-24T03:17:23.216 INFO:teuthology.orchestra.run.smithi118.stdout: "flags": 0, 2024-03-24T03:17:23.217 INFO:teuthology.orchestra.run.smithi118.stdout: "default_value": "4721", 2024-03-24T03:17:23.217 INFO:teuthology.orchestra.run.smithi118.stdout: "min": "", 2024-03-24T03:17:23.217 INFO:teuthology.orchestra.run.smithi118.stdout: "max": "", 2024-03-24T03:17:23.217 INFO:teuthology.orchestra.run.smithi118.stdout: "enum_allowed": [], 2024-03-24T03:17:23.217 INFO:teuthology.orchestra.run.smithi118.stdout: "desc": "First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)", 2024-03-24T03:17:23.217 INFO:teuthology.orchestra.run.smithi118.stdout: "long_desc": "", 2024-03-24T03:17:23.217 INFO:teuthology.orchestra.run.smithi118.stdout: "tags": [], 2024-03-24T03:17:23.217 INFO:teuthology.orchestra.run.smithi118.stdout: "see_also": [] 2024-03-24T03:17:23.217 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-24T03:17:23.218 INFO:teuthology.orchestra.run.smithi118.stdout: "allow_ptrace": { 2024-03-24T03:17:23.218 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "allow_ptrace", 2024-03-24T03:17:23.218 INFO:teuthology.orchestra.run.smithi118.stdout: "type": "bool", 2024-03-24T03:17:23.218 INFO:teuthology.orchestra.run.smithi118.stdout: "level": "advanced", 2024-03-24T03:17:23.218 INFO:teuthology.orchestra.run.smithi118.stdout: "flags": 0, 2024-03-24T03:17:23.218 INFO:teuthology.orchestra.run.smithi118.stdout: "default_value": "False", 2024-03-24T03:17:23.218 INFO:teuthology.orchestra.run.smithi118.stdout: "min": "", 2024-03-24T03:17:23.218 INFO:teuthology.orchestra.run.smithi118.stdout: "max": "", 2024-03-24T03:17:23.219 INFO:teuthology.orchestra.run.smithi118.stdout: "enum_allowed": [], 2024-03-24T03:17:23.219 INFO:teuthology.orchestra.run.smithi118.stdout: "desc": "allow SYS_PTRACE capability on ceph containers", 2024-03-24T03:17:23.219 INFO:teuthology.orchestra.run.smithi118.stdout: "long_desc": "The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.", 2024-03-24T03:17:23.219 INFO:teuthology.orchestra.run.smithi118.stdout: "tags": [], 2024-03-24T03:17:23.219 INFO:teuthology.orchestra.run.smithi118.stdout: "see_also": [] 2024-03-24T03:17:23.219 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-24T03:17:23.219 INFO:teuthology.orchestra.run.smithi118.stdout: "autotune_interval": { 2024-03-24T03:17:23.219 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "autotune_interval", 2024-03-24T03:17:23.220 INFO:teuthology.orchestra.run.smithi118.stdout: "type": "secs", 2024-03-24T03:17:23.220 INFO:teuthology.orchestra.run.smithi118.stdout: "level": "advanced", 2024-03-24T03:17:23.220 INFO:teuthology.orchestra.run.smithi118.stdout: "flags": 0, 2024-03-24T03:17:23.220 INFO:teuthology.orchestra.run.smithi118.stdout: "default_value": "600", 2024-03-24T03:17:23.220 INFO:teuthology.orchestra.run.smithi118.stdout: "min": "", 2024-03-24T03:17:23.220 INFO:teuthology.orchestra.run.smithi118.stdout: "max": "", 2024-03-24T03:17:23.220 INFO:teuthology.orchestra.run.smithi118.stdout: "enum_allowed": [], 2024-03-24T03:17:23.220 INFO:teuthology.orchestra.run.smithi118.stdout: "desc": "how frequently to autotune daemon memory", 2024-03-24T03:17:23.220 INFO:teuthology.orchestra.run.smithi118.stdout: "long_desc": "", 2024-03-24T03:17:23.221 INFO:teuthology.orchestra.run.smithi118.stdout: "tags": [], 2024-03-24T03:17:23.221 INFO:teuthology.orchestra.run.smithi118.stdout: "see_also": [] 2024-03-24T03:17:23.221 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-24T03:17:23.221 INFO:teuthology.orchestra.run.smithi118.stdout: "autotune_memory_target_ratio": { 2024-03-24T03:17:23.221 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "autotune_memory_target_ratio", 2024-03-24T03:17:23.221 INFO:teuthology.orchestra.run.smithi118.stdout: "type": "float", 2024-03-24T03:17:23.221 INFO:teuthology.orchestra.run.smithi118.stdout: "level": "advanced", 2024-03-24T03:17:23.221 INFO:teuthology.orchestra.run.smithi118.stdout: "flags": 0, 2024-03-24T03:17:23.222 INFO:teuthology.orchestra.run.smithi118.stdout: "default_value": "0.7", 2024-03-24T03:17:23.222 INFO:teuthology.orchestra.run.smithi118.stdout: "min": "", 2024-03-24T03:17:23.222 INFO:teuthology.orchestra.run.smithi118.stdout: "max": "", 2024-03-24T03:17:23.222 INFO:teuthology.orchestra.run.smithi118.stdout: "enum_allowed": [], 2024-03-24T03:17:23.222 INFO:teuthology.orchestra.run.smithi118.stdout: "desc": "ratio of total system memory to divide amongst autotuned daemons", 2024-03-24T03:17:23.222 INFO:teuthology.orchestra.run.smithi118.stdout: "long_desc": "", 2024-03-24T03:17:23.222 INFO:teuthology.orchestra.run.smithi118.stdout: "tags": [], 2024-03-24T03:17:23.222 INFO:teuthology.orchestra.run.smithi118.stdout: "see_also": [] 2024-03-24T03:17:23.223 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-24T03:17:23.223 INFO:teuthology.orchestra.run.smithi118.stdout: "cephadm_log_destination": { 2024-03-24T03:17:23.223 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "cephadm_log_destination", 2024-03-24T03:17:23.223 INFO:teuthology.orchestra.run.smithi118.stdout: "type": "str", 2024-03-24T03:17:23.223 INFO:teuthology.orchestra.run.smithi118.stdout: "level": "advanced", 2024-03-24T03:17:23.223 INFO:teuthology.orchestra.run.smithi118.stdout: "flags": 0, 2024-03-24T03:17:23.223 INFO:teuthology.orchestra.run.smithi118.stdout: "default_value": "", 2024-03-24T03:17:23.223 INFO:teuthology.orchestra.run.smithi118.stdout: "min": "", 2024-03-24T03:17:23.223 INFO:teuthology.orchestra.run.smithi118.stdout: "max": "", 2024-03-24T03:17:23.224 INFO:teuthology.orchestra.run.smithi118.stdout: "enum_allowed": [ 2024-03-24T03:17:23.224 INFO:teuthology.orchestra.run.smithi118.stdout: "file", 2024-03-24T03:17:23.224 INFO:teuthology.orchestra.run.smithi118.stdout: "file,syslog", 2024-03-24T03:17:23.224 INFO:teuthology.orchestra.run.smithi118.stdout: "syslog" 2024-03-24T03:17:23.224 INFO:teuthology.orchestra.run.smithi118.stdout: ], 2024-03-24T03:17:23.224 INFO:teuthology.orchestra.run.smithi118.stdout: "desc": "Destination for cephadm command's persistent logging", 2024-03-24T03:17:23.224 INFO:teuthology.orchestra.run.smithi118.stdout: "long_desc": "", 2024-03-24T03:17:23.224 INFO:teuthology.orchestra.run.smithi118.stdout: "tags": [], 2024-03-24T03:17:23.225 INFO:teuthology.orchestra.run.smithi118.stdout: "see_also": [] 2024-03-24T03:17:23.225 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-24T03:17:23.225 INFO:teuthology.orchestra.run.smithi118.stdout: "cgroups_split": { 2024-03-24T03:17:23.225 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "cgroups_split", 2024-03-24T03:17:23.225 INFO:teuthology.orchestra.run.smithi118.stdout: "type": "bool", 2024-03-24T03:17:23.225 INFO:teuthology.orchestra.run.smithi118.stdout: "level": "advanced", 2024-03-24T03:17:23.225 INFO:teuthology.orchestra.run.smithi118.stdout: "flags": 0, 2024-03-24T03:17:23.225 INFO:teuthology.orchestra.run.smithi118.stdout: "default_value": "True", 2024-03-24T03:17:23.226 INFO:teuthology.orchestra.run.smithi118.stdout: "min": "", 2024-03-24T03:17:23.226 INFO:teuthology.orchestra.run.smithi118.stdout: "max": "", 2024-03-24T03:17:23.226 INFO:teuthology.orchestra.run.smithi118.stdout: "enum_allowed": [], 2024-03-24T03:17:23.226 INFO:teuthology.orchestra.run.smithi118.stdout: "desc": "Pass --cgroups=split when cephadm creates containers (currently podman only)", 2024-03-24T03:17:23.226 INFO:teuthology.orchestra.run.smithi118.stdout: "long_desc": "", 2024-03-24T03:17:23.226 INFO:teuthology.orchestra.run.smithi118.stdout: "tags": [], 2024-03-24T03:17:23.226 INFO:teuthology.orchestra.run.smithi118.stdout: "see_also": [] 2024-03-24T03:17:23.227 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-24T03:17:23.227 INFO:teuthology.orchestra.run.smithi118.stdout: "config_checks_enabled": { 2024-03-24T03:17:23.227 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "config_checks_enabled", 2024-03-24T03:17:23.227 INFO:teuthology.orchestra.run.smithi118.stdout: "type": "bool", 2024-03-24T03:17:23.227 INFO:teuthology.orchestra.run.smithi118.stdout: "level": "advanced", 2024-03-24T03:17:23.227 INFO:teuthology.orchestra.run.smithi118.stdout: "flags": 0, 2024-03-24T03:17:23.227 INFO:teuthology.orchestra.run.smithi118.stdout: "default_value": "False", 2024-03-24T03:17:23.227 INFO:teuthology.orchestra.run.smithi118.stdout: "min": "", 2024-03-24T03:17:23.227 INFO:teuthology.orchestra.run.smithi118.stdout: "max": "", 2024-03-24T03:17:23.228 INFO:teuthology.orchestra.run.smithi118.stdout: "enum_allowed": [], 2024-03-24T03:17:23.228 INFO:teuthology.orchestra.run.smithi118.stdout: "desc": "Enable or disable the cephadm configuration analysis", 2024-03-24T03:17:23.228 INFO:teuthology.orchestra.run.smithi118.stdout: "long_desc": "", 2024-03-24T03:17:23.228 INFO:teuthology.orchestra.run.smithi118.stdout: "tags": [], 2024-03-24T03:17:23.228 INFO:teuthology.orchestra.run.smithi118.stdout: "see_also": [] 2024-03-24T03:17:23.228 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-24T03:17:23.228 INFO:teuthology.orchestra.run.smithi118.stdout: "config_dashboard": { 2024-03-24T03:17:23.228 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "config_dashboard", 2024-03-24T03:17:23.228 INFO:teuthology.orchestra.run.smithi118.stdout: "type": "bool", 2024-03-24T03:17:23.229 INFO:teuthology.orchestra.run.smithi118.stdout: "level": "advanced", 2024-03-24T03:17:23.229 INFO:teuthology.orchestra.run.smithi118.stdout: "flags": 0, 2024-03-24T03:17:23.229 INFO:teuthology.orchestra.run.smithi118.stdout: "default_value": "True", 2024-03-24T03:17:23.229 INFO:teuthology.orchestra.run.smithi118.stdout: "min": "", 2024-03-24T03:17:23.229 INFO:teuthology.orchestra.run.smithi118.stdout: "max": "", 2024-03-24T03:17:23.229 INFO:teuthology.orchestra.run.smithi118.stdout: "enum_allowed": [], 2024-03-24T03:17:23.229 INFO:teuthology.orchestra.run.smithi118.stdout: "desc": "manage configs like API endpoints in Dashboard.", 2024-03-24T03:17:23.229 INFO:teuthology.orchestra.run.smithi118.stdout: "long_desc": "", 2024-03-24T03:17:23.229 INFO:teuthology.orchestra.run.smithi118.stdout: "tags": [], 2024-03-24T03:17:23.230 INFO:teuthology.orchestra.run.smithi118.stdout: "see_also": [] 2024-03-24T03:17:23.230 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-24T03:17:23.230 INFO:teuthology.orchestra.run.smithi118.stdout: "container_image_alertmanager": { 2024-03-24T03:17:23.230 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "container_image_alertmanager", 2024-03-24T03:17:23.230 INFO:teuthology.orchestra.run.smithi118.stdout: "type": "str", 2024-03-24T03:17:23.230 INFO:teuthology.orchestra.run.smithi118.stdout: "level": "advanced", 2024-03-24T03:17:23.230 INFO:teuthology.orchestra.run.smithi118.stdout: "flags": 0, 2024-03-24T03:17:23.231 INFO:teuthology.orchestra.run.smithi118.stdout: "default_value": "quay.io/prometheus/alertmanager:v0.25.0", 2024-03-24T03:17:23.231 INFO:teuthology.orchestra.run.smithi118.stdout: "min": "", 2024-03-24T03:17:23.231 INFO:teuthology.orchestra.run.smithi118.stdout: "max": "", 2024-03-24T03:17:23.231 INFO:teuthology.orchestra.run.smithi118.stdout: "enum_allowed": [], 2024-03-24T03:17:23.231 INFO:teuthology.orchestra.run.smithi118.stdout: "desc": "Prometheus container image", 2024-03-24T03:17:23.231 INFO:teuthology.orchestra.run.smithi118.stdout: "long_desc": "", 2024-03-24T03:17:23.231 INFO:teuthology.orchestra.run.smithi118.stdout: "tags": [], 2024-03-24T03:17:23.231 INFO:teuthology.orchestra.run.smithi118.stdout: "see_also": [] 2024-03-24T03:17:23.232 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-24T03:17:23.232 INFO:teuthology.orchestra.run.smithi118.stdout: "container_image_base": { 2024-03-24T03:17:23.232 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "container_image_base", 2024-03-24T03:17:23.232 INFO:teuthology.orchestra.run.smithi118.stdout: "type": "str", 2024-03-24T03:17:23.232 INFO:teuthology.orchestra.run.smithi118.stdout: "level": "advanced", 2024-03-24T03:17:23.232 INFO:teuthology.orchestra.run.smithi118.stdout: "flags": 1, 2024-03-24T03:17:23.232 INFO:teuthology.orchestra.run.smithi118.stdout: "default_value": "quay.io/ceph/ceph", 2024-03-24T03:17:23.233 INFO:teuthology.orchestra.run.smithi118.stdout: "min": "", 2024-03-24T03:17:23.233 INFO:teuthology.orchestra.run.smithi118.stdout: "max": "", 2024-03-24T03:17:23.233 INFO:teuthology.orchestra.run.smithi118.stdout: "enum_allowed": [], 2024-03-24T03:17:23.233 INFO:teuthology.orchestra.run.smithi118.stdout: "desc": "Container image name, without the tag", 2024-03-24T03:17:23.233 INFO:teuthology.orchestra.run.smithi118.stdout: "long_desc": "", 2024-03-24T03:17:23.233 INFO:teuthology.orchestra.run.smithi118.stdout: "tags": [], 2024-03-24T03:17:23.233 INFO:teuthology.orchestra.run.smithi118.stdout: "see_also": [] 2024-03-24T03:17:23.233 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-24T03:17:23.234 INFO:teuthology.orchestra.run.smithi118.stdout: "container_image_elasticsearch": { 2024-03-24T03:17:23.234 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "container_image_elasticsearch", 2024-03-24T03:17:23.234 INFO:teuthology.orchestra.run.smithi118.stdout: "type": "str", 2024-03-24T03:17:23.234 INFO:teuthology.orchestra.run.smithi118.stdout: "level": "advanced", 2024-03-24T03:17:23.234 INFO:teuthology.orchestra.run.smithi118.stdout: "flags": 0, 2024-03-24T03:17:23.234 INFO:teuthology.orchestra.run.smithi118.stdout: "default_value": "quay.io/omrizeneva/elasticsearch:6.8.23", 2024-03-24T03:17:23.234 INFO:teuthology.orchestra.run.smithi118.stdout: "min": "", 2024-03-24T03:17:23.234 INFO:teuthology.orchestra.run.smithi118.stdout: "max": "", 2024-03-24T03:17:23.235 INFO:teuthology.orchestra.run.smithi118.stdout: "enum_allowed": [], 2024-03-24T03:17:23.235 INFO:teuthology.orchestra.run.smithi118.stdout: "desc": "elasticsearch container image", 2024-03-24T03:17:23.235 INFO:teuthology.orchestra.run.smithi118.stdout: "long_desc": "", 2024-03-24T03:17:23.235 INFO:teuthology.orchestra.run.smithi118.stdout: "tags": [], 2024-03-24T03:17:23.235 INFO:teuthology.orchestra.run.smithi118.stdout: "see_also": [] 2024-03-24T03:17:23.235 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-24T03:17:23.235 INFO:teuthology.orchestra.run.smithi118.stdout: "container_image_grafana": { 2024-03-24T03:17:23.235 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "container_image_grafana", 2024-03-24T03:17:23.235 INFO:teuthology.orchestra.run.smithi118.stdout: "type": "str", 2024-03-24T03:17:23.236 INFO:teuthology.orchestra.run.smithi118.stdout: "level": "advanced", 2024-03-24T03:17:23.236 INFO:teuthology.orchestra.run.smithi118.stdout: "flags": 0, 2024-03-24T03:17:23.236 INFO:teuthology.orchestra.run.smithi118.stdout: "default_value": "quay.io/ceph/ceph-grafana:9.4.12", 2024-03-24T03:17:23.236 INFO:teuthology.orchestra.run.smithi118.stdout: "min": "", 2024-03-24T03:17:23.236 INFO:teuthology.orchestra.run.smithi118.stdout: "max": "", 2024-03-24T03:17:23.236 INFO:teuthology.orchestra.run.smithi118.stdout: "enum_allowed": [], 2024-03-24T03:17:23.236 INFO:teuthology.orchestra.run.smithi118.stdout: "desc": "Prometheus container image", 2024-03-24T03:17:23.236 INFO:teuthology.orchestra.run.smithi118.stdout: "long_desc": "", 2024-03-24T03:17:23.237 INFO:teuthology.orchestra.run.smithi118.stdout: "tags": [], 2024-03-24T03:17:23.237 INFO:teuthology.orchestra.run.smithi118.stdout: "see_also": [] 2024-03-24T03:17:23.237 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-24T03:17:23.237 INFO:teuthology.orchestra.run.smithi118.stdout: "container_image_haproxy": { 2024-03-24T03:17:23.237 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "container_image_haproxy", 2024-03-24T03:17:23.237 INFO:teuthology.orchestra.run.smithi118.stdout: "type": "str", 2024-03-24T03:17:23.237 INFO:teuthology.orchestra.run.smithi118.stdout: "level": "advanced", 2024-03-24T03:17:23.237 INFO:teuthology.orchestra.run.smithi118.stdout: "flags": 0, 2024-03-24T03:17:23.238 INFO:teuthology.orchestra.run.smithi118.stdout: "default_value": "quay.io/ceph/haproxy:2.3", 2024-03-24T03:17:23.238 INFO:teuthology.orchestra.run.smithi118.stdout: "min": "", 2024-03-24T03:17:23.238 INFO:teuthology.orchestra.run.smithi118.stdout: "max": "", 2024-03-24T03:17:23.238 INFO:teuthology.orchestra.run.smithi118.stdout: "enum_allowed": [], 2024-03-24T03:17:23.238 INFO:teuthology.orchestra.run.smithi118.stdout: "desc": "HAproxy container image", 2024-03-24T03:17:23.238 INFO:teuthology.orchestra.run.smithi118.stdout: "long_desc": "", 2024-03-24T03:17:23.238 INFO:teuthology.orchestra.run.smithi118.stdout: "tags": [], 2024-03-24T03:17:23.238 INFO:teuthology.orchestra.run.smithi118.stdout: "see_also": [] 2024-03-24T03:17:23.239 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-24T03:17:23.239 INFO:teuthology.orchestra.run.smithi118.stdout: "container_image_jaeger_agent": { 2024-03-24T03:17:23.239 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "container_image_jaeger_agent", 2024-03-24T03:17:23.239 INFO:teuthology.orchestra.run.smithi118.stdout: "type": "str", 2024-03-24T03:17:23.239 INFO:teuthology.orchestra.run.smithi118.stdout: "level": "advanced", 2024-03-24T03:17:23.239 INFO:teuthology.orchestra.run.smithi118.stdout: "flags": 0, 2024-03-24T03:17:23.239 INFO:teuthology.orchestra.run.smithi118.stdout: "default_value": "quay.io/jaegertracing/jaeger-agent:1.29", 2024-03-24T03:17:23.239 INFO:teuthology.orchestra.run.smithi118.stdout: "min": "", 2024-03-24T03:17:23.240 INFO:teuthology.orchestra.run.smithi118.stdout: "max": "", 2024-03-24T03:17:23.240 INFO:teuthology.orchestra.run.smithi118.stdout: "enum_allowed": [], 2024-03-24T03:17:23.240 INFO:teuthology.orchestra.run.smithi118.stdout: "desc": "Jaeger agent container image", 2024-03-24T03:17:23.240 INFO:teuthology.orchestra.run.smithi118.stdout: "long_desc": "", 2024-03-24T03:17:23.240 INFO:teuthology.orchestra.run.smithi118.stdout: "tags": [], 2024-03-24T03:17:23.240 INFO:teuthology.orchestra.run.smithi118.stdout: "see_also": [] 2024-03-24T03:17:23.240 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-24T03:17:23.240 INFO:teuthology.orchestra.run.smithi118.stdout: "container_image_jaeger_collector": { 2024-03-24T03:17:23.240 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "container_image_jaeger_collector", 2024-03-24T03:17:23.241 INFO:teuthology.orchestra.run.smithi118.stdout: "type": "str", 2024-03-24T03:17:23.241 INFO:teuthology.orchestra.run.smithi118.stdout: "level": "advanced", 2024-03-24T03:17:23.241 INFO:teuthology.orchestra.run.smithi118.stdout: "flags": 0, 2024-03-24T03:17:23.241 INFO:teuthology.orchestra.run.smithi118.stdout: "default_value": "quay.io/jaegertracing/jaeger-collector:1.29", 2024-03-24T03:17:23.241 INFO:teuthology.orchestra.run.smithi118.stdout: "min": "", 2024-03-24T03:17:23.241 INFO:teuthology.orchestra.run.smithi118.stdout: "max": "", 2024-03-24T03:17:23.241 INFO:teuthology.orchestra.run.smithi118.stdout: "enum_allowed": [], 2024-03-24T03:17:23.241 INFO:teuthology.orchestra.run.smithi118.stdout: "desc": "Jaeger collector container image", 2024-03-24T03:17:23.242 INFO:teuthology.orchestra.run.smithi118.stdout: "long_desc": "", 2024-03-24T03:17:23.242 INFO:teuthology.orchestra.run.smithi118.stdout: "tags": [], 2024-03-24T03:17:23.242 INFO:teuthology.orchestra.run.smithi118.stdout: "see_also": [] 2024-03-24T03:17:23.242 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-24T03:17:23.242 INFO:teuthology.orchestra.run.smithi118.stdout: "container_image_jaeger_query": { 2024-03-24T03:17:23.242 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "container_image_jaeger_query", 2024-03-24T03:17:23.242 INFO:teuthology.orchestra.run.smithi118.stdout: "type": "str", 2024-03-24T03:17:23.242 INFO:teuthology.orchestra.run.smithi118.stdout: "level": "advanced", 2024-03-24T03:17:23.243 INFO:teuthology.orchestra.run.smithi118.stdout: "flags": 0, 2024-03-24T03:17:23.243 INFO:teuthology.orchestra.run.smithi118.stdout: "default_value": "quay.io/jaegertracing/jaeger-query:1.29", 2024-03-24T03:17:23.243 INFO:teuthology.orchestra.run.smithi118.stdout: "min": "", 2024-03-24T03:17:23.243 INFO:teuthology.orchestra.run.smithi118.stdout: "max": "", 2024-03-24T03:17:23.243 INFO:teuthology.orchestra.run.smithi118.stdout: "enum_allowed": [], 2024-03-24T03:17:23.243 INFO:teuthology.orchestra.run.smithi118.stdout: "desc": "Jaeger query container image", 2024-03-24T03:17:23.243 INFO:teuthology.orchestra.run.smithi118.stdout: "long_desc": "", 2024-03-24T03:17:23.243 INFO:teuthology.orchestra.run.smithi118.stdout: "tags": [], 2024-03-24T03:17:23.243 INFO:teuthology.orchestra.run.smithi118.stdout: "see_also": [] 2024-03-24T03:17:23.244 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-24T03:17:23.244 INFO:teuthology.orchestra.run.smithi118.stdout: "container_image_keepalived": { 2024-03-24T03:17:23.244 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "container_image_keepalived", 2024-03-24T03:17:23.244 INFO:teuthology.orchestra.run.smithi118.stdout: "type": "str", 2024-03-24T03:17:23.244 INFO:teuthology.orchestra.run.smithi118.stdout: "level": "advanced", 2024-03-24T03:17:23.244 INFO:teuthology.orchestra.run.smithi118.stdout: "flags": 0, 2024-03-24T03:17:23.244 INFO:teuthology.orchestra.run.smithi118.stdout: "default_value": "quay.io/ceph/keepalived:2.2.4", 2024-03-24T03:17:23.244 INFO:teuthology.orchestra.run.smithi118.stdout: "min": "", 2024-03-24T03:17:23.245 INFO:teuthology.orchestra.run.smithi118.stdout: "max": "", 2024-03-24T03:17:23.245 INFO:teuthology.orchestra.run.smithi118.stdout: "enum_allowed": [], 2024-03-24T03:17:23.245 INFO:teuthology.orchestra.run.smithi118.stdout: "desc": "Keepalived container image", 2024-03-24T03:17:23.245 INFO:teuthology.orchestra.run.smithi118.stdout: "long_desc": "", 2024-03-24T03:17:23.245 INFO:teuthology.orchestra.run.smithi118.stdout: "tags": [], 2024-03-24T03:17:23.245 INFO:teuthology.orchestra.run.smithi118.stdout: "see_also": [] 2024-03-24T03:17:23.245 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-24T03:17:23.246 INFO:teuthology.orchestra.run.smithi118.stdout: "container_image_loki": { 2024-03-24T03:17:23.246 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "container_image_loki", 2024-03-24T03:17:23.246 INFO:teuthology.orchestra.run.smithi118.stdout: "type": "str", 2024-03-24T03:17:23.246 INFO:teuthology.orchestra.run.smithi118.stdout: "level": "advanced", 2024-03-24T03:17:23.246 INFO:teuthology.orchestra.run.smithi118.stdout: "flags": 0, 2024-03-24T03:17:23.246 INFO:teuthology.orchestra.run.smithi118.stdout: "default_value": "docker.io/grafana/loki:2.4.0", 2024-03-24T03:17:23.246 INFO:teuthology.orchestra.run.smithi118.stdout: "min": "", 2024-03-24T03:17:23.247 INFO:teuthology.orchestra.run.smithi118.stdout: "max": "", 2024-03-24T03:17:23.247 INFO:teuthology.orchestra.run.smithi118.stdout: "enum_allowed": [], 2024-03-24T03:17:23.247 INFO:teuthology.orchestra.run.smithi118.stdout: "desc": "Loki container image", 2024-03-24T03:17:23.247 INFO:teuthology.orchestra.run.smithi118.stdout: "long_desc": "", 2024-03-24T03:17:23.247 INFO:teuthology.orchestra.run.smithi118.stdout: "tags": [], 2024-03-24T03:17:23.247 INFO:teuthology.orchestra.run.smithi118.stdout: "see_also": [] 2024-03-24T03:17:23.247 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-24T03:17:23.247 INFO:teuthology.orchestra.run.smithi118.stdout: "container_image_node_exporter": { 2024-03-24T03:17:23.247 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "container_image_node_exporter", 2024-03-24T03:17:23.248 INFO:teuthology.orchestra.run.smithi118.stdout: "type": "str", 2024-03-24T03:17:23.248 INFO:teuthology.orchestra.run.smithi118.stdout: "level": "advanced", 2024-03-24T03:17:23.248 INFO:teuthology.orchestra.run.smithi118.stdout: "flags": 0, 2024-03-24T03:17:23.248 INFO:teuthology.orchestra.run.smithi118.stdout: "default_value": "quay.io/prometheus/node-exporter:v1.5.0", 2024-03-24T03:17:23.248 INFO:teuthology.orchestra.run.smithi118.stdout: "min": "", 2024-03-24T03:17:23.249 INFO:teuthology.orchestra.run.smithi118.stdout: "max": "", 2024-03-24T03:17:23.249 INFO:teuthology.orchestra.run.smithi118.stdout: "enum_allowed": [], 2024-03-24T03:17:23.249 INFO:teuthology.orchestra.run.smithi118.stdout: "desc": "Prometheus container image", 2024-03-24T03:17:23.249 INFO:teuthology.orchestra.run.smithi118.stdout: "long_desc": "", 2024-03-24T03:17:23.249 INFO:teuthology.orchestra.run.smithi118.stdout: "tags": [], 2024-03-24T03:17:23.249 INFO:teuthology.orchestra.run.smithi118.stdout: "see_also": [] 2024-03-24T03:17:23.250 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-24T03:17:23.250 INFO:teuthology.orchestra.run.smithi118.stdout: "container_image_nvmeof": { 2024-03-24T03:17:23.250 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "container_image_nvmeof", 2024-03-24T03:17:23.250 INFO:teuthology.orchestra.run.smithi118.stdout: "type": "str", 2024-03-24T03:17:23.250 INFO:teuthology.orchestra.run.smithi118.stdout: "level": "advanced", 2024-03-24T03:17:23.250 INFO:teuthology.orchestra.run.smithi118.stdout: "flags": 0, 2024-03-24T03:17:23.250 INFO:teuthology.orchestra.run.smithi118.stdout: "default_value": "quay.io/ceph/nvmeof:1.0.0", 2024-03-24T03:17:23.250 INFO:teuthology.orchestra.run.smithi118.stdout: "min": "", 2024-03-24T03:17:23.251 INFO:teuthology.orchestra.run.smithi118.stdout: "max": "", 2024-03-24T03:17:23.251 INFO:teuthology.orchestra.run.smithi118.stdout: "enum_allowed": [], 2024-03-24T03:17:23.251 INFO:teuthology.orchestra.run.smithi118.stdout: "desc": "Nvme-of container image", 2024-03-24T03:17:23.251 INFO:teuthology.orchestra.run.smithi118.stdout: "long_desc": "", 2024-03-24T03:17:23.251 INFO:teuthology.orchestra.run.smithi118.stdout: "tags": [], 2024-03-24T03:17:23.251 INFO:teuthology.orchestra.run.smithi118.stdout: "see_also": [] 2024-03-24T03:17:23.251 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-24T03:17:23.251 INFO:teuthology.orchestra.run.smithi118.stdout: "container_image_prometheus": { 2024-03-24T03:17:23.251 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "container_image_prometheus", 2024-03-24T03:17:23.252 INFO:teuthology.orchestra.run.smithi118.stdout: "type": "str", 2024-03-24T03:17:23.252 INFO:teuthology.orchestra.run.smithi118.stdout: "level": "advanced", 2024-03-24T03:17:23.252 INFO:teuthology.orchestra.run.smithi118.stdout: "flags": 0, 2024-03-24T03:17:23.252 INFO:teuthology.orchestra.run.smithi118.stdout: "default_value": "quay.io/prometheus/prometheus:v2.43.0", 2024-03-24T03:17:23.252 INFO:teuthology.orchestra.run.smithi118.stdout: "min": "", 2024-03-24T03:17:23.252 INFO:teuthology.orchestra.run.smithi118.stdout: "max": "", 2024-03-24T03:17:23.252 INFO:teuthology.orchestra.run.smithi118.stdout: "enum_allowed": [], 2024-03-24T03:17:23.252 INFO:teuthology.orchestra.run.smithi118.stdout: "desc": "Prometheus container image", 2024-03-24T03:17:23.253 INFO:teuthology.orchestra.run.smithi118.stdout: "long_desc": "", 2024-03-24T03:17:23.253 INFO:teuthology.orchestra.run.smithi118.stdout: "tags": [], 2024-03-24T03:17:23.253 INFO:teuthology.orchestra.run.smithi118.stdout: "see_also": [] 2024-03-24T03:17:23.253 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-24T03:17:23.253 INFO:teuthology.orchestra.run.smithi118.stdout: "container_image_promtail": { 2024-03-24T03:17:23.253 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "container_image_promtail", 2024-03-24T03:17:23.253 INFO:teuthology.orchestra.run.smithi118.stdout: "type": "str", 2024-03-24T03:17:23.253 INFO:teuthology.orchestra.run.smithi118.stdout: "level": "advanced", 2024-03-24T03:17:23.253 INFO:teuthology.orchestra.run.smithi118.stdout: "flags": 0, 2024-03-24T03:17:23.254 INFO:teuthology.orchestra.run.smithi118.stdout: "default_value": "docker.io/grafana/promtail:2.4.0", 2024-03-24T03:17:23.254 INFO:teuthology.orchestra.run.smithi118.stdout: "min": "", 2024-03-24T03:17:23.254 INFO:teuthology.orchestra.run.smithi118.stdout: "max": "", 2024-03-24T03:17:23.254 INFO:teuthology.orchestra.run.smithi118.stdout: "enum_allowed": [], 2024-03-24T03:17:23.254 INFO:teuthology.orchestra.run.smithi118.stdout: "desc": "Promtail container image", 2024-03-24T03:17:23.254 INFO:teuthology.orchestra.run.smithi118.stdout: "long_desc": "", 2024-03-24T03:17:23.255 INFO:teuthology.orchestra.run.smithi118.stdout: "tags": [], 2024-03-24T03:17:23.255 INFO:teuthology.orchestra.run.smithi118.stdout: "see_also": [] 2024-03-24T03:17:23.255 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-24T03:17:23.255 INFO:teuthology.orchestra.run.smithi118.stdout: "container_image_snmp_gateway": { 2024-03-24T03:17:23.255 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "container_image_snmp_gateway", 2024-03-24T03:17:23.255 INFO:teuthology.orchestra.run.smithi118.stdout: "type": "str", 2024-03-24T03:17:23.255 INFO:teuthology.orchestra.run.smithi118.stdout: "level": "advanced", 2024-03-24T03:17:23.255 INFO:teuthology.orchestra.run.smithi118.stdout: "flags": 0, 2024-03-24T03:17:23.255 INFO:teuthology.orchestra.run.smithi118.stdout: "default_value": "docker.io/maxwo/snmp-notifier:v1.2.1", 2024-03-24T03:17:23.256 INFO:teuthology.orchestra.run.smithi118.stdout: "min": "", 2024-03-24T03:17:23.256 INFO:teuthology.orchestra.run.smithi118.stdout: "max": "", 2024-03-24T03:17:23.256 INFO:teuthology.orchestra.run.smithi118.stdout: "enum_allowed": [], 2024-03-24T03:17:23.256 INFO:teuthology.orchestra.run.smithi118.stdout: "desc": "SNMP Gateway container image", 2024-03-24T03:17:23.256 INFO:teuthology.orchestra.run.smithi118.stdout: "long_desc": "", 2024-03-24T03:17:23.256 INFO:teuthology.orchestra.run.smithi118.stdout: "tags": [], 2024-03-24T03:17:23.256 INFO:teuthology.orchestra.run.smithi118.stdout: "see_also": [] 2024-03-24T03:17:23.256 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-24T03:17:23.257 INFO:teuthology.orchestra.run.smithi118.stdout: "container_init": { 2024-03-24T03:17:23.257 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "container_init", 2024-03-24T03:17:23.257 INFO:teuthology.orchestra.run.smithi118.stdout: "type": "bool", 2024-03-24T03:17:23.257 INFO:teuthology.orchestra.run.smithi118.stdout: "level": "advanced", 2024-03-24T03:17:23.257 INFO:teuthology.orchestra.run.smithi118.stdout: "flags": 0, 2024-03-24T03:17:23.257 INFO:teuthology.orchestra.run.smithi118.stdout: "default_value": "True", 2024-03-24T03:17:23.257 INFO:teuthology.orchestra.run.smithi118.stdout: "min": "", 2024-03-24T03:17:23.257 INFO:teuthology.orchestra.run.smithi118.stdout: "max": "", 2024-03-24T03:17:23.257 INFO:teuthology.orchestra.run.smithi118.stdout: "enum_allowed": [], 2024-03-24T03:17:23.258 INFO:teuthology.orchestra.run.smithi118.stdout: "desc": "Run podman/docker with `--init`", 2024-03-24T03:17:23.258 INFO:teuthology.orchestra.run.smithi118.stdout: "long_desc": "", 2024-03-24T03:17:23.258 INFO:teuthology.orchestra.run.smithi118.stdout: "tags": [], 2024-03-24T03:17:23.258 INFO:teuthology.orchestra.run.smithi118.stdout: "see_also": [] 2024-03-24T03:17:23.258 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-24T03:17:23.258 INFO:teuthology.orchestra.run.smithi118.stdout: "daemon_cache_timeout": { 2024-03-24T03:17:23.258 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "daemon_cache_timeout", 2024-03-24T03:17:23.258 INFO:teuthology.orchestra.run.smithi118.stdout: "type": "secs", 2024-03-24T03:17:23.259 INFO:teuthology.orchestra.run.smithi118.stdout: "level": "advanced", 2024-03-24T03:17:23.259 INFO:teuthology.orchestra.run.smithi118.stdout: "flags": 0, 2024-03-24T03:17:23.259 INFO:teuthology.orchestra.run.smithi118.stdout: "default_value": "600", 2024-03-24T03:17:23.259 INFO:teuthology.orchestra.run.smithi118.stdout: "min": "", 2024-03-24T03:17:23.259 INFO:teuthology.orchestra.run.smithi118.stdout: "max": "", 2024-03-24T03:17:23.259 INFO:teuthology.orchestra.run.smithi118.stdout: "enum_allowed": [], 2024-03-24T03:17:23.259 INFO:teuthology.orchestra.run.smithi118.stdout: "desc": "seconds to cache service (daemon) inventory", 2024-03-24T03:17:23.259 INFO:teuthology.orchestra.run.smithi118.stdout: "long_desc": "", 2024-03-24T03:17:23.259 INFO:teuthology.orchestra.run.smithi118.stdout: "tags": [], 2024-03-24T03:17:23.260 INFO:teuthology.orchestra.run.smithi118.stdout: "see_also": [] 2024-03-24T03:17:23.260 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-24T03:17:23.260 INFO:teuthology.orchestra.run.smithi118.stdout: "default_cephadm_command_timeout": { 2024-03-24T03:17:23.260 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "default_cephadm_command_timeout", 2024-03-24T03:17:23.260 INFO:teuthology.orchestra.run.smithi118.stdout: "type": "int", 2024-03-24T03:17:23.260 INFO:teuthology.orchestra.run.smithi118.stdout: "level": "advanced", 2024-03-24T03:17:23.260 INFO:teuthology.orchestra.run.smithi118.stdout: "flags": 0, 2024-03-24T03:17:23.260 INFO:teuthology.orchestra.run.smithi118.stdout: "default_value": "900", 2024-03-24T03:17:23.261 INFO:teuthology.orchestra.run.smithi118.stdout: "min": "", 2024-03-24T03:17:23.261 INFO:teuthology.orchestra.run.smithi118.stdout: "max": "", 2024-03-24T03:17:23.261 INFO:teuthology.orchestra.run.smithi118.stdout: "enum_allowed": [], 2024-03-24T03:17:23.261 INFO:teuthology.orchestra.run.smithi118.stdout: "desc": "Default timeout applied to cephadm commands run directly on the host (in seconds)", 2024-03-24T03:17:23.261 INFO:teuthology.orchestra.run.smithi118.stdout: "long_desc": "", 2024-03-24T03:17:23.261 INFO:teuthology.orchestra.run.smithi118.stdout: "tags": [], 2024-03-24T03:17:23.262 INFO:teuthology.orchestra.run.smithi118.stdout: "see_also": [] 2024-03-24T03:17:23.262 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-24T03:17:23.262 INFO:teuthology.orchestra.run.smithi118.stdout: "default_registry": { 2024-03-24T03:17:23.262 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "default_registry", 2024-03-24T03:17:23.262 INFO:teuthology.orchestra.run.smithi118.stdout: "type": "str", 2024-03-24T03:17:23.263 INFO:teuthology.orchestra.run.smithi118.stdout: "level": "advanced", 2024-03-24T03:17:23.263 INFO:teuthology.orchestra.run.smithi118.stdout: "flags": 0, 2024-03-24T03:17:23.263 INFO:teuthology.orchestra.run.smithi118.stdout: "default_value": "docker.io", 2024-03-24T03:17:23.263 INFO:teuthology.orchestra.run.smithi118.stdout: "min": "", 2024-03-24T03:17:23.263 INFO:teuthology.orchestra.run.smithi118.stdout: "max": "", 2024-03-24T03:17:23.263 INFO:teuthology.orchestra.run.smithi118.stdout: "enum_allowed": [], 2024-03-24T03:17:23.263 INFO:teuthology.orchestra.run.smithi118.stdout: "desc": "Search-registry to which we should normalize unqualified image names. This is not the default registry", 2024-03-24T03:17:23.263 INFO:teuthology.orchestra.run.smithi118.stdout: "long_desc": "", 2024-03-24T03:17:23.263 INFO:teuthology.orchestra.run.smithi118.stdout: "tags": [], 2024-03-24T03:17:23.264 INFO:teuthology.orchestra.run.smithi118.stdout: "see_also": [] 2024-03-24T03:17:23.264 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-24T03:17:23.264 INFO:teuthology.orchestra.run.smithi118.stdout: "device_cache_timeout": { 2024-03-24T03:17:23.264 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "device_cache_timeout", 2024-03-24T03:17:23.264 INFO:teuthology.orchestra.run.smithi118.stdout: "type": "secs", 2024-03-24T03:17:23.264 INFO:teuthology.orchestra.run.smithi118.stdout: "level": "advanced", 2024-03-24T03:17:23.264 INFO:teuthology.orchestra.run.smithi118.stdout: "flags": 0, 2024-03-24T03:17:23.264 INFO:teuthology.orchestra.run.smithi118.stdout: "default_value": "1800", 2024-03-24T03:17:23.265 INFO:teuthology.orchestra.run.smithi118.stdout: "min": "", 2024-03-24T03:17:23.265 INFO:teuthology.orchestra.run.smithi118.stdout: "max": "", 2024-03-24T03:17:23.265 INFO:teuthology.orchestra.run.smithi118.stdout: "enum_allowed": [], 2024-03-24T03:17:23.265 INFO:teuthology.orchestra.run.smithi118.stdout: "desc": "seconds to cache device inventory", 2024-03-24T03:17:23.265 INFO:teuthology.orchestra.run.smithi118.stdout: "long_desc": "", 2024-03-24T03:17:23.265 INFO:teuthology.orchestra.run.smithi118.stdout: "tags": [], 2024-03-24T03:17:23.265 INFO:teuthology.orchestra.run.smithi118.stdout: "see_also": [] 2024-03-24T03:17:23.265 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-24T03:17:23.266 INFO:teuthology.orchestra.run.smithi118.stdout: "device_enhanced_scan": { 2024-03-24T03:17:23.266 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "device_enhanced_scan", 2024-03-24T03:17:23.266 INFO:teuthology.orchestra.run.smithi118.stdout: "type": "bool", 2024-03-24T03:17:23.266 INFO:teuthology.orchestra.run.smithi118.stdout: "level": "advanced", 2024-03-24T03:17:23.266 INFO:teuthology.orchestra.run.smithi118.stdout: "flags": 0, 2024-03-24T03:17:23.266 INFO:teuthology.orchestra.run.smithi118.stdout: "default_value": "False", 2024-03-24T03:17:23.266 INFO:teuthology.orchestra.run.smithi118.stdout: "min": "", 2024-03-24T03:17:23.266 INFO:teuthology.orchestra.run.smithi118.stdout: "max": "", 2024-03-24T03:17:23.267 INFO:teuthology.orchestra.run.smithi118.stdout: "enum_allowed": [], 2024-03-24T03:17:23.267 INFO:teuthology.orchestra.run.smithi118.stdout: "desc": "Use libstoragemgmt during device scans", 2024-03-24T03:17:23.267 INFO:teuthology.orchestra.run.smithi118.stdout: "long_desc": "", 2024-03-24T03:17:23.267 INFO:teuthology.orchestra.run.smithi118.stdout: "tags": [], 2024-03-24T03:17:23.267 INFO:teuthology.orchestra.run.smithi118.stdout: "see_also": [] 2024-03-24T03:17:23.267 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-24T03:17:23.267 INFO:teuthology.orchestra.run.smithi118.stdout: "facts_cache_timeout": { 2024-03-24T03:17:23.267 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "facts_cache_timeout", 2024-03-24T03:17:23.267 INFO:teuthology.orchestra.run.smithi118.stdout: "type": "secs", 2024-03-24T03:17:23.268 INFO:teuthology.orchestra.run.smithi118.stdout: "level": "advanced", 2024-03-24T03:17:23.268 INFO:teuthology.orchestra.run.smithi118.stdout: "flags": 0, 2024-03-24T03:17:23.268 INFO:teuthology.orchestra.run.smithi118.stdout: "default_value": "60", 2024-03-24T03:17:23.268 INFO:teuthology.orchestra.run.smithi118.stdout: "min": "", 2024-03-24T03:17:23.268 INFO:teuthology.orchestra.run.smithi118.stdout: "max": "", 2024-03-24T03:17:23.268 INFO:teuthology.orchestra.run.smithi118.stdout: "enum_allowed": [], 2024-03-24T03:17:23.268 INFO:teuthology.orchestra.run.smithi118.stdout: "desc": "seconds to cache host facts data", 2024-03-24T03:17:23.268 INFO:teuthology.orchestra.run.smithi118.stdout: "long_desc": "", 2024-03-24T03:17:23.269 INFO:teuthology.orchestra.run.smithi118.stdout: "tags": [], 2024-03-24T03:17:23.269 INFO:teuthology.orchestra.run.smithi118.stdout: "see_also": [] 2024-03-24T03:17:23.269 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-24T03:17:23.269 INFO:teuthology.orchestra.run.smithi118.stdout: "host_check_interval": { 2024-03-24T03:17:23.269 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "host_check_interval", 2024-03-24T03:17:23.269 INFO:teuthology.orchestra.run.smithi118.stdout: "type": "secs", 2024-03-24T03:17:23.269 INFO:teuthology.orchestra.run.smithi118.stdout: "level": "advanced", 2024-03-24T03:17:23.269 INFO:teuthology.orchestra.run.smithi118.stdout: "flags": 0, 2024-03-24T03:17:23.269 INFO:teuthology.orchestra.run.smithi118.stdout: "default_value": "600", 2024-03-24T03:17:23.270 INFO:teuthology.orchestra.run.smithi118.stdout: "min": "", 2024-03-24T03:17:23.270 INFO:teuthology.orchestra.run.smithi118.stdout: "max": "", 2024-03-24T03:17:23.270 INFO:teuthology.orchestra.run.smithi118.stdout: "enum_allowed": [], 2024-03-24T03:17:23.270 INFO:teuthology.orchestra.run.smithi118.stdout: "desc": "how frequently to perform a host check", 2024-03-24T03:17:23.270 INFO:teuthology.orchestra.run.smithi118.stdout: "long_desc": "", 2024-03-24T03:17:23.270 INFO:teuthology.orchestra.run.smithi118.stdout: "tags": [], 2024-03-24T03:17:23.270 INFO:teuthology.orchestra.run.smithi118.stdout: "see_also": [] 2024-03-24T03:17:23.271 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-24T03:17:23.271 INFO:teuthology.orchestra.run.smithi118.stdout: "hw_monitoring": { 2024-03-24T03:17:23.271 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "hw_monitoring", 2024-03-24T03:17:23.271 INFO:teuthology.orchestra.run.smithi118.stdout: "type": "bool", 2024-03-24T03:17:23.271 INFO:teuthology.orchestra.run.smithi118.stdout: "level": "advanced", 2024-03-24T03:17:23.271 INFO:teuthology.orchestra.run.smithi118.stdout: "flags": 0, 2024-03-24T03:17:23.271 INFO:teuthology.orchestra.run.smithi118.stdout: "default_value": "False", 2024-03-24T03:17:23.271 INFO:teuthology.orchestra.run.smithi118.stdout: "min": "", 2024-03-24T03:17:23.272 INFO:teuthology.orchestra.run.smithi118.stdout: "max": "", 2024-03-24T03:17:23.272 INFO:teuthology.orchestra.run.smithi118.stdout: "enum_allowed": [], 2024-03-24T03:17:23.272 INFO:teuthology.orchestra.run.smithi118.stdout: "desc": "Deploy hw monitoring daemon on every host.", 2024-03-24T03:17:23.272 INFO:teuthology.orchestra.run.smithi118.stdout: "long_desc": "", 2024-03-24T03:17:23.272 INFO:teuthology.orchestra.run.smithi118.stdout: "tags": [], 2024-03-24T03:17:23.272 INFO:teuthology.orchestra.run.smithi118.stdout: "see_also": [] 2024-03-24T03:17:23.272 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-24T03:17:23.272 INFO:teuthology.orchestra.run.smithi118.stdout: "inventory_list_all": { 2024-03-24T03:17:23.272 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "inventory_list_all", 2024-03-24T03:17:23.273 INFO:teuthology.orchestra.run.smithi118.stdout: "type": "bool", 2024-03-24T03:17:23.273 INFO:teuthology.orchestra.run.smithi118.stdout: "level": "advanced", 2024-03-24T03:17:23.273 INFO:teuthology.orchestra.run.smithi118.stdout: "flags": 0, 2024-03-24T03:17:23.273 INFO:teuthology.orchestra.run.smithi118.stdout: "default_value": "False", 2024-03-24T03:17:23.273 INFO:teuthology.orchestra.run.smithi118.stdout: "min": "", 2024-03-24T03:17:23.273 INFO:teuthology.orchestra.run.smithi118.stdout: "max": "", 2024-03-24T03:17:23.273 INFO:teuthology.orchestra.run.smithi118.stdout: "enum_allowed": [], 2024-03-24T03:17:23.273 INFO:teuthology.orchestra.run.smithi118.stdout: "desc": "Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)", 2024-03-24T03:17:23.274 INFO:teuthology.orchestra.run.smithi118.stdout: "long_desc": "", 2024-03-24T03:17:23.274 INFO:teuthology.orchestra.run.smithi118.stdout: "tags": [], 2024-03-24T03:17:23.274 INFO:teuthology.orchestra.run.smithi118.stdout: "see_also": [] 2024-03-24T03:17:23.274 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-24T03:17:23.274 INFO:teuthology.orchestra.run.smithi118.stdout: "log_level": { 2024-03-24T03:17:23.274 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "log_level", 2024-03-24T03:17:23.274 INFO:teuthology.orchestra.run.smithi118.stdout: "type": "str", 2024-03-24T03:17:23.274 INFO:teuthology.orchestra.run.smithi118.stdout: "level": "advanced", 2024-03-24T03:17:23.275 INFO:teuthology.orchestra.run.smithi118.stdout: "flags": 1, 2024-03-24T03:17:23.275 INFO:teuthology.orchestra.run.smithi118.stdout: "default_value": "", 2024-03-24T03:17:23.275 INFO:teuthology.orchestra.run.smithi118.stdout: "min": "", 2024-03-24T03:17:23.275 INFO:teuthology.orchestra.run.smithi118.stdout: "max": "", 2024-03-24T03:17:23.275 INFO:teuthology.orchestra.run.smithi118.stdout: "enum_allowed": [ 2024-03-24T03:17:23.275 INFO:teuthology.orchestra.run.smithi118.stdout: "", 2024-03-24T03:17:23.275 INFO:teuthology.orchestra.run.smithi118.stdout: "critical", 2024-03-24T03:17:23.275 INFO:teuthology.orchestra.run.smithi118.stdout: "debug", 2024-03-24T03:17:23.275 INFO:teuthology.orchestra.run.smithi118.stdout: "error", 2024-03-24T03:17:23.276 INFO:teuthology.orchestra.run.smithi118.stdout: "info", 2024-03-24T03:17:23.276 INFO:teuthology.orchestra.run.smithi118.stdout: "warning" 2024-03-24T03:17:23.276 INFO:teuthology.orchestra.run.smithi118.stdout: ], 2024-03-24T03:17:23.276 INFO:teuthology.orchestra.run.smithi118.stdout: "desc": "", 2024-03-24T03:17:23.276 INFO:teuthology.orchestra.run.smithi118.stdout: "long_desc": "", 2024-03-24T03:17:23.276 INFO:teuthology.orchestra.run.smithi118.stdout: "tags": [], 2024-03-24T03:17:23.276 INFO:teuthology.orchestra.run.smithi118.stdout: "see_also": [] 2024-03-24T03:17:23.276 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-24T03:17:23.277 INFO:teuthology.orchestra.run.smithi118.stdout: "log_refresh_metadata": { 2024-03-24T03:17:23.277 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "log_refresh_metadata", 2024-03-24T03:17:23.277 INFO:teuthology.orchestra.run.smithi118.stdout: "type": "bool", 2024-03-24T03:17:23.277 INFO:teuthology.orchestra.run.smithi118.stdout: "level": "advanced", 2024-03-24T03:17:23.277 INFO:teuthology.orchestra.run.smithi118.stdout: "flags": 0, 2024-03-24T03:17:23.277 INFO:teuthology.orchestra.run.smithi118.stdout: "default_value": "False", 2024-03-24T03:17:23.277 INFO:teuthology.orchestra.run.smithi118.stdout: "min": "", 2024-03-24T03:17:23.277 INFO:teuthology.orchestra.run.smithi118.stdout: "max": "", 2024-03-24T03:17:23.277 INFO:teuthology.orchestra.run.smithi118.stdout: "enum_allowed": [], 2024-03-24T03:17:23.278 INFO:teuthology.orchestra.run.smithi118.stdout: "desc": "Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level", 2024-03-24T03:17:23.278 INFO:teuthology.orchestra.run.smithi118.stdout: "long_desc": "", 2024-03-24T03:17:23.278 INFO:teuthology.orchestra.run.smithi118.stdout: "tags": [], 2024-03-24T03:17:23.278 INFO:teuthology.orchestra.run.smithi118.stdout: "see_also": [] 2024-03-24T03:17:23.278 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-24T03:17:23.278 INFO:teuthology.orchestra.run.smithi118.stdout: "log_to_cluster": { 2024-03-24T03:17:23.278 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "log_to_cluster", 2024-03-24T03:17:23.278 INFO:teuthology.orchestra.run.smithi118.stdout: "type": "bool", 2024-03-24T03:17:23.279 INFO:teuthology.orchestra.run.smithi118.stdout: "level": "advanced", 2024-03-24T03:17:23.279 INFO:teuthology.orchestra.run.smithi118.stdout: "flags": 0, 2024-03-24T03:17:23.279 INFO:teuthology.orchestra.run.smithi118.stdout: "default_value": "True", 2024-03-24T03:17:23.279 INFO:teuthology.orchestra.run.smithi118.stdout: "min": "", 2024-03-24T03:17:23.279 INFO:teuthology.orchestra.run.smithi118.stdout: "max": "", 2024-03-24T03:17:23.279 INFO:teuthology.orchestra.run.smithi118.stdout: "enum_allowed": [], 2024-03-24T03:17:23.279 INFO:teuthology.orchestra.run.smithi118.stdout: "desc": "log to the \"cephadm\" cluster log channel\"", 2024-03-24T03:17:23.279 INFO:teuthology.orchestra.run.smithi118.stdout: "long_desc": "", 2024-03-24T03:17:23.279 INFO:teuthology.orchestra.run.smithi118.stdout: "tags": [], 2024-03-24T03:17:23.280 INFO:teuthology.orchestra.run.smithi118.stdout: "see_also": [] 2024-03-24T03:17:23.280 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-24T03:17:23.280 INFO:teuthology.orchestra.run.smithi118.stdout: "log_to_cluster_level": { 2024-03-24T03:17:23.280 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "log_to_cluster_level", 2024-03-24T03:17:23.280 INFO:teuthology.orchestra.run.smithi118.stdout: "type": "str", 2024-03-24T03:17:23.280 INFO:teuthology.orchestra.run.smithi118.stdout: "level": "advanced", 2024-03-24T03:17:23.280 INFO:teuthology.orchestra.run.smithi118.stdout: "flags": 1, 2024-03-24T03:17:23.280 INFO:teuthology.orchestra.run.smithi118.stdout: "default_value": "info", 2024-03-24T03:17:23.280 INFO:teuthology.orchestra.run.smithi118.stdout: "min": "", 2024-03-24T03:17:23.281 INFO:teuthology.orchestra.run.smithi118.stdout: "max": "", 2024-03-24T03:17:23.281 INFO:teuthology.orchestra.run.smithi118.stdout: "enum_allowed": [ 2024-03-24T03:17:23.281 INFO:teuthology.orchestra.run.smithi118.stdout: "", 2024-03-24T03:17:23.281 INFO:teuthology.orchestra.run.smithi118.stdout: "critical", 2024-03-24T03:17:23.281 INFO:teuthology.orchestra.run.smithi118.stdout: "debug", 2024-03-24T03:17:23.281 INFO:teuthology.orchestra.run.smithi118.stdout: "error", 2024-03-24T03:17:23.281 INFO:teuthology.orchestra.run.smithi118.stdout: "info", 2024-03-24T03:17:23.281 INFO:teuthology.orchestra.run.smithi118.stdout: "warning" 2024-03-24T03:17:23.282 INFO:teuthology.orchestra.run.smithi118.stdout: ], 2024-03-24T03:17:23.282 INFO:teuthology.orchestra.run.smithi118.stdout: "desc": "", 2024-03-24T03:17:23.282 INFO:teuthology.orchestra.run.smithi118.stdout: "long_desc": "", 2024-03-24T03:17:23.282 INFO:teuthology.orchestra.run.smithi118.stdout: "tags": [], 2024-03-24T03:17:23.282 INFO:teuthology.orchestra.run.smithi118.stdout: "see_also": [] 2024-03-24T03:17:23.282 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-24T03:17:23.282 INFO:teuthology.orchestra.run.smithi118.stdout: "log_to_file": { 2024-03-24T03:17:23.283 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "log_to_file", 2024-03-24T03:17:23.283 INFO:teuthology.orchestra.run.smithi118.stdout: "type": "bool", 2024-03-24T03:17:23.283 INFO:teuthology.orchestra.run.smithi118.stdout: "level": "advanced", 2024-03-24T03:17:23.283 INFO:teuthology.orchestra.run.smithi118.stdout: "flags": 1, 2024-03-24T03:17:23.283 INFO:teuthology.orchestra.run.smithi118.stdout: "default_value": "False", 2024-03-24T03:17:23.283 INFO:teuthology.orchestra.run.smithi118.stdout: "min": "", 2024-03-24T03:17:23.283 INFO:teuthology.orchestra.run.smithi118.stdout: "max": "", 2024-03-24T03:17:23.283 INFO:teuthology.orchestra.run.smithi118.stdout: "enum_allowed": [], 2024-03-24T03:17:23.283 INFO:teuthology.orchestra.run.smithi118.stdout: "desc": "", 2024-03-24T03:17:23.284 INFO:teuthology.orchestra.run.smithi118.stdout: "long_desc": "", 2024-03-24T03:17:23.284 INFO:teuthology.orchestra.run.smithi118.stdout: "tags": [], 2024-03-24T03:17:23.284 INFO:teuthology.orchestra.run.smithi118.stdout: "see_also": [] 2024-03-24T03:17:23.284 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-24T03:17:23.284 INFO:teuthology.orchestra.run.smithi118.stdout: "manage_etc_ceph_ceph_conf": { 2024-03-24T03:17:23.284 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "manage_etc_ceph_ceph_conf", 2024-03-24T03:17:23.284 INFO:teuthology.orchestra.run.smithi118.stdout: "type": "bool", 2024-03-24T03:17:23.284 INFO:teuthology.orchestra.run.smithi118.stdout: "level": "advanced", 2024-03-24T03:17:23.285 INFO:teuthology.orchestra.run.smithi118.stdout: "flags": 0, 2024-03-24T03:17:23.285 INFO:teuthology.orchestra.run.smithi118.stdout: "default_value": "False", 2024-03-24T03:17:23.285 INFO:teuthology.orchestra.run.smithi118.stdout: "min": "", 2024-03-24T03:17:23.285 INFO:teuthology.orchestra.run.smithi118.stdout: "max": "", 2024-03-24T03:17:23.285 INFO:teuthology.orchestra.run.smithi118.stdout: "enum_allowed": [], 2024-03-24T03:17:23.285 INFO:teuthology.orchestra.run.smithi118.stdout: "desc": "Manage and own /etc/ceph/ceph.conf on the hosts.", 2024-03-24T03:17:23.285 INFO:teuthology.orchestra.run.smithi118.stdout: "long_desc": "", 2024-03-24T03:17:23.285 INFO:teuthology.orchestra.run.smithi118.stdout: "tags": [], 2024-03-24T03:17:23.285 INFO:teuthology.orchestra.run.smithi118.stdout: "see_also": [] 2024-03-24T03:17:23.286 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-24T03:17:23.286 INFO:teuthology.orchestra.run.smithi118.stdout: "manage_etc_ceph_ceph_conf_hosts": { 2024-03-24T03:17:23.286 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "manage_etc_ceph_ceph_conf_hosts", 2024-03-24T03:17:23.286 INFO:teuthology.orchestra.run.smithi118.stdout: "type": "str", 2024-03-24T03:17:23.286 INFO:teuthology.orchestra.run.smithi118.stdout: "level": "advanced", 2024-03-24T03:17:23.286 INFO:teuthology.orchestra.run.smithi118.stdout: "flags": 0, 2024-03-24T03:17:23.286 INFO:teuthology.orchestra.run.smithi118.stdout: "default_value": "*", 2024-03-24T03:17:23.286 INFO:teuthology.orchestra.run.smithi118.stdout: "min": "", 2024-03-24T03:17:23.287 INFO:teuthology.orchestra.run.smithi118.stdout: "max": "", 2024-03-24T03:17:23.287 INFO:teuthology.orchestra.run.smithi118.stdout: "enum_allowed": [], 2024-03-24T03:17:23.287 INFO:teuthology.orchestra.run.smithi118.stdout: "desc": "PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf", 2024-03-24T03:17:23.287 INFO:teuthology.orchestra.run.smithi118.stdout: "long_desc": "", 2024-03-24T03:17:23.287 INFO:teuthology.orchestra.run.smithi118.stdout: "tags": [], 2024-03-24T03:17:23.287 INFO:teuthology.orchestra.run.smithi118.stdout: "see_also": [] 2024-03-24T03:17:23.287 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-24T03:17:23.287 INFO:teuthology.orchestra.run.smithi118.stdout: "max_count_per_host": { 2024-03-24T03:17:23.287 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "max_count_per_host", 2024-03-24T03:17:23.288 INFO:teuthology.orchestra.run.smithi118.stdout: "type": "int", 2024-03-24T03:17:23.288 INFO:teuthology.orchestra.run.smithi118.stdout: "level": "advanced", 2024-03-24T03:17:23.288 INFO:teuthology.orchestra.run.smithi118.stdout: "flags": 0, 2024-03-24T03:17:23.288 INFO:teuthology.orchestra.run.smithi118.stdout: "default_value": "10", 2024-03-24T03:17:23.288 INFO:teuthology.orchestra.run.smithi118.stdout: "min": "", 2024-03-24T03:17:23.288 INFO:teuthology.orchestra.run.smithi118.stdout: "max": "", 2024-03-24T03:17:23.288 INFO:teuthology.orchestra.run.smithi118.stdout: "enum_allowed": [], 2024-03-24T03:17:23.288 INFO:teuthology.orchestra.run.smithi118.stdout: "desc": "max number of daemons per service per host", 2024-03-24T03:17:23.288 INFO:teuthology.orchestra.run.smithi118.stdout: "long_desc": "", 2024-03-24T03:17:23.289 INFO:teuthology.orchestra.run.smithi118.stdout: "tags": [], 2024-03-24T03:17:23.289 INFO:teuthology.orchestra.run.smithi118.stdout: "see_also": [] 2024-03-24T03:17:23.289 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-24T03:17:23.289 INFO:teuthology.orchestra.run.smithi118.stdout: "max_osd_draining_count": { 2024-03-24T03:17:23.289 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "max_osd_draining_count", 2024-03-24T03:17:23.289 INFO:teuthology.orchestra.run.smithi118.stdout: "type": "int", 2024-03-24T03:17:23.289 INFO:teuthology.orchestra.run.smithi118.stdout: "level": "advanced", 2024-03-24T03:17:23.289 INFO:teuthology.orchestra.run.smithi118.stdout: "flags": 0, 2024-03-24T03:17:23.290 INFO:teuthology.orchestra.run.smithi118.stdout: "default_value": "10", 2024-03-24T03:17:23.290 INFO:teuthology.orchestra.run.smithi118.stdout: "min": "", 2024-03-24T03:17:23.290 INFO:teuthology.orchestra.run.smithi118.stdout: "max": "", 2024-03-24T03:17:23.290 INFO:teuthology.orchestra.run.smithi118.stdout: "enum_allowed": [], 2024-03-24T03:17:23.290 INFO:teuthology.orchestra.run.smithi118.stdout: "desc": "max number of osds that will be drained simultaneously when osds are removed", 2024-03-24T03:17:23.290 INFO:teuthology.orchestra.run.smithi118.stdout: "long_desc": "", 2024-03-24T03:17:23.290 INFO:teuthology.orchestra.run.smithi118.stdout: "tags": [], 2024-03-24T03:17:23.291 INFO:teuthology.orchestra.run.smithi118.stdout: "see_also": [] 2024-03-24T03:17:23.291 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-24T03:17:23.291 INFO:teuthology.orchestra.run.smithi118.stdout: "migration_current": { 2024-03-24T03:17:23.291 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "migration_current", 2024-03-24T03:17:23.291 INFO:teuthology.orchestra.run.smithi118.stdout: "type": "int", 2024-03-24T03:17:23.291 INFO:teuthology.orchestra.run.smithi118.stdout: "level": "advanced", 2024-03-24T03:17:23.291 INFO:teuthology.orchestra.run.smithi118.stdout: "flags": 0, 2024-03-24T03:17:23.291 INFO:teuthology.orchestra.run.smithi118.stdout: "default_value": "", 2024-03-24T03:17:23.291 INFO:teuthology.orchestra.run.smithi118.stdout: "min": "", 2024-03-24T03:17:23.292 INFO:teuthology.orchestra.run.smithi118.stdout: "max": "", 2024-03-24T03:17:23.292 INFO:teuthology.orchestra.run.smithi118.stdout: "enum_allowed": [], 2024-03-24T03:17:23.292 INFO:teuthology.orchestra.run.smithi118.stdout: "desc": "internal - do not modify", 2024-03-24T03:17:23.292 INFO:teuthology.orchestra.run.smithi118.stdout: "long_desc": "", 2024-03-24T03:17:23.292 INFO:teuthology.orchestra.run.smithi118.stdout: "tags": [], 2024-03-24T03:17:23.292 INFO:teuthology.orchestra.run.smithi118.stdout: "see_also": [] 2024-03-24T03:17:23.292 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-24T03:17:23.292 INFO:teuthology.orchestra.run.smithi118.stdout: "mode": { 2024-03-24T03:17:23.293 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "mode", 2024-03-24T03:17:23.293 INFO:teuthology.orchestra.run.smithi118.stdout: "type": "str", 2024-03-24T03:17:23.293 INFO:teuthology.orchestra.run.smithi118.stdout: "level": "advanced", 2024-03-24T03:17:23.293 INFO:teuthology.orchestra.run.smithi118.stdout: "flags": 0, 2024-03-24T03:17:23.293 INFO:teuthology.orchestra.run.smithi118.stdout: "default_value": "root", 2024-03-24T03:17:23.293 INFO:teuthology.orchestra.run.smithi118.stdout: "min": "", 2024-03-24T03:17:23.293 INFO:teuthology.orchestra.run.smithi118.stdout: "max": "", 2024-03-24T03:17:23.293 INFO:teuthology.orchestra.run.smithi118.stdout: "enum_allowed": [ 2024-03-24T03:17:23.293 INFO:teuthology.orchestra.run.smithi118.stdout: "cephadm-package", 2024-03-24T03:17:23.294 INFO:teuthology.orchestra.run.smithi118.stdout: "root" 2024-03-24T03:17:23.294 INFO:teuthology.orchestra.run.smithi118.stdout: ], 2024-03-24T03:17:23.294 INFO:teuthology.orchestra.run.smithi118.stdout: "desc": "mode for remote execution of cephadm", 2024-03-24T03:17:23.294 INFO:teuthology.orchestra.run.smithi118.stdout: "long_desc": "", 2024-03-24T03:17:23.294 INFO:teuthology.orchestra.run.smithi118.stdout: "tags": [], 2024-03-24T03:17:23.294 INFO:teuthology.orchestra.run.smithi118.stdout: "see_also": [] 2024-03-24T03:17:23.294 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-24T03:17:23.294 INFO:teuthology.orchestra.run.smithi118.stdout: "oob_default_addr": { 2024-03-24T03:17:23.295 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "oob_default_addr", 2024-03-24T03:17:23.295 INFO:teuthology.orchestra.run.smithi118.stdout: "type": "str", 2024-03-24T03:17:23.295 INFO:teuthology.orchestra.run.smithi118.stdout: "level": "advanced", 2024-03-24T03:17:23.295 INFO:teuthology.orchestra.run.smithi118.stdout: "flags": 0, 2024-03-24T03:17:23.295 INFO:teuthology.orchestra.run.smithi118.stdout: "default_value": "169.254.1.1", 2024-03-24T03:17:23.295 INFO:teuthology.orchestra.run.smithi118.stdout: "min": "", 2024-03-24T03:17:23.295 INFO:teuthology.orchestra.run.smithi118.stdout: "max": "", 2024-03-24T03:17:23.295 INFO:teuthology.orchestra.run.smithi118.stdout: "enum_allowed": [], 2024-03-24T03:17:23.296 INFO:teuthology.orchestra.run.smithi118.stdout: "desc": "Default address for RedFish API (oob management).", 2024-03-24T03:17:23.296 INFO:teuthology.orchestra.run.smithi118.stdout: "long_desc": "", 2024-03-24T03:17:23.296 INFO:teuthology.orchestra.run.smithi118.stdout: "tags": [], 2024-03-24T03:17:23.296 INFO:teuthology.orchestra.run.smithi118.stdout: "see_also": [] 2024-03-24T03:17:23.296 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-24T03:17:23.296 INFO:teuthology.orchestra.run.smithi118.stdout: "prometheus_alerts_path": { 2024-03-24T03:17:23.296 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "prometheus_alerts_path", 2024-03-24T03:17:23.296 INFO:teuthology.orchestra.run.smithi118.stdout: "type": "str", 2024-03-24T03:17:23.297 INFO:teuthology.orchestra.run.smithi118.stdout: "level": "advanced", 2024-03-24T03:17:23.297 INFO:teuthology.orchestra.run.smithi118.stdout: "flags": 0, 2024-03-24T03:17:23.297 INFO:teuthology.orchestra.run.smithi118.stdout: "default_value": "/etc/prometheus/ceph/ceph_default_alerts.yml", 2024-03-24T03:17:23.297 INFO:teuthology.orchestra.run.smithi118.stdout: "min": "", 2024-03-24T03:17:23.297 INFO:teuthology.orchestra.run.smithi118.stdout: "max": "", 2024-03-24T03:17:23.297 INFO:teuthology.orchestra.run.smithi118.stdout: "enum_allowed": [], 2024-03-24T03:17:23.297 INFO:teuthology.orchestra.run.smithi118.stdout: "desc": "location of alerts to include in prometheus deployments", 2024-03-24T03:17:23.297 INFO:teuthology.orchestra.run.smithi118.stdout: "long_desc": "", 2024-03-24T03:17:23.298 INFO:teuthology.orchestra.run.smithi118.stdout: "tags": [], 2024-03-24T03:17:23.298 INFO:teuthology.orchestra.run.smithi118.stdout: "see_also": [] 2024-03-24T03:17:23.298 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-24T03:17:23.298 INFO:teuthology.orchestra.run.smithi118.stdout: "registry_insecure": { 2024-03-24T03:17:23.298 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "registry_insecure", 2024-03-24T03:17:23.298 INFO:teuthology.orchestra.run.smithi118.stdout: "type": "bool", 2024-03-24T03:17:23.298 INFO:teuthology.orchestra.run.smithi118.stdout: "level": "advanced", 2024-03-24T03:17:23.298 INFO:teuthology.orchestra.run.smithi118.stdout: "flags": 0, 2024-03-24T03:17:23.298 INFO:teuthology.orchestra.run.smithi118.stdout: "default_value": "False", 2024-03-24T03:17:23.299 INFO:teuthology.orchestra.run.smithi118.stdout: "min": "", 2024-03-24T03:17:23.299 INFO:teuthology.orchestra.run.smithi118.stdout: "max": "", 2024-03-24T03:17:23.299 INFO:teuthology.orchestra.run.smithi118.stdout: "enum_allowed": [], 2024-03-24T03:17:23.299 INFO:teuthology.orchestra.run.smithi118.stdout: "desc": "Registry is to be considered insecure (no TLS available). Only for development purposes.", 2024-03-24T03:17:23.299 INFO:teuthology.orchestra.run.smithi118.stdout: "long_desc": "", 2024-03-24T03:17:23.299 INFO:teuthology.orchestra.run.smithi118.stdout: "tags": [], 2024-03-24T03:17:23.299 INFO:teuthology.orchestra.run.smithi118.stdout: "see_also": [] 2024-03-24T03:17:23.299 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-24T03:17:23.300 INFO:teuthology.orchestra.run.smithi118.stdout: "registry_password": { 2024-03-24T03:17:23.300 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "registry_password", 2024-03-24T03:17:23.300 INFO:teuthology.orchestra.run.smithi118.stdout: "type": "str", 2024-03-24T03:17:23.300 INFO:teuthology.orchestra.run.smithi118.stdout: "level": "advanced", 2024-03-24T03:17:23.300 INFO:teuthology.orchestra.run.smithi118.stdout: "flags": 0, 2024-03-24T03:17:23.300 INFO:teuthology.orchestra.run.smithi118.stdout: "default_value": "", 2024-03-24T03:17:23.300 INFO:teuthology.orchestra.run.smithi118.stdout: "min": "", 2024-03-24T03:17:23.300 INFO:teuthology.orchestra.run.smithi118.stdout: "max": "", 2024-03-24T03:17:23.300 INFO:teuthology.orchestra.run.smithi118.stdout: "enum_allowed": [], 2024-03-24T03:17:23.301 INFO:teuthology.orchestra.run.smithi118.stdout: "desc": "Custom repository password. Only used for logging into a registry.", 2024-03-24T03:17:23.301 INFO:teuthology.orchestra.run.smithi118.stdout: "long_desc": "", 2024-03-24T03:17:23.301 INFO:teuthology.orchestra.run.smithi118.stdout: "tags": [], 2024-03-24T03:17:23.301 INFO:teuthology.orchestra.run.smithi118.stdout: "see_also": [] 2024-03-24T03:17:23.301 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-24T03:17:23.301 INFO:teuthology.orchestra.run.smithi118.stdout: "registry_url": { 2024-03-24T03:17:23.301 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "registry_url", 2024-03-24T03:17:23.301 INFO:teuthology.orchestra.run.smithi118.stdout: "type": "str", 2024-03-24T03:17:23.301 INFO:teuthology.orchestra.run.smithi118.stdout: "level": "advanced", 2024-03-24T03:17:23.302 INFO:teuthology.orchestra.run.smithi118.stdout: "flags": 0, 2024-03-24T03:17:23.302 INFO:teuthology.orchestra.run.smithi118.stdout: "default_value": "", 2024-03-24T03:17:23.302 INFO:teuthology.orchestra.run.smithi118.stdout: "min": "", 2024-03-24T03:17:23.302 INFO:teuthology.orchestra.run.smithi118.stdout: "max": "", 2024-03-24T03:17:23.302 INFO:teuthology.orchestra.run.smithi118.stdout: "enum_allowed": [], 2024-03-24T03:17:23.302 INFO:teuthology.orchestra.run.smithi118.stdout: "desc": "Registry url for login purposes. This is not the default registry", 2024-03-24T03:17:23.302 INFO:teuthology.orchestra.run.smithi118.stdout: "long_desc": "", 2024-03-24T03:17:23.302 INFO:teuthology.orchestra.run.smithi118.stdout: "tags": [], 2024-03-24T03:17:23.303 INFO:teuthology.orchestra.run.smithi118.stdout: "see_also": [] 2024-03-24T03:17:23.303 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-24T03:17:23.303 INFO:teuthology.orchestra.run.smithi118.stdout: "registry_username": { 2024-03-24T03:17:23.303 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "registry_username", 2024-03-24T03:17:23.303 INFO:teuthology.orchestra.run.smithi118.stdout: "type": "str", 2024-03-24T03:17:23.303 INFO:teuthology.orchestra.run.smithi118.stdout: "level": "advanced", 2024-03-24T03:17:23.303 INFO:teuthology.orchestra.run.smithi118.stdout: "flags": 0, 2024-03-24T03:17:23.304 INFO:teuthology.orchestra.run.smithi118.stdout: "default_value": "", 2024-03-24T03:17:23.304 INFO:teuthology.orchestra.run.smithi118.stdout: "min": "", 2024-03-24T03:17:23.304 INFO:teuthology.orchestra.run.smithi118.stdout: "max": "", 2024-03-24T03:17:23.304 INFO:teuthology.orchestra.run.smithi118.stdout: "enum_allowed": [], 2024-03-24T03:17:23.304 INFO:teuthology.orchestra.run.smithi118.stdout: "desc": "Custom repository username. Only used for logging into a registry.", 2024-03-24T03:17:23.304 INFO:teuthology.orchestra.run.smithi118.stdout: "long_desc": "", 2024-03-24T03:17:23.304 INFO:teuthology.orchestra.run.smithi118.stdout: "tags": [], 2024-03-24T03:17:23.304 INFO:teuthology.orchestra.run.smithi118.stdout: "see_also": [] 2024-03-24T03:17:23.304 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-24T03:17:23.305 INFO:teuthology.orchestra.run.smithi118.stdout: "secure_monitoring_stack": { 2024-03-24T03:17:23.305 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "secure_monitoring_stack", 2024-03-24T03:17:23.305 INFO:teuthology.orchestra.run.smithi118.stdout: "type": "bool", 2024-03-24T03:17:23.305 INFO:teuthology.orchestra.run.smithi118.stdout: "level": "advanced", 2024-03-24T03:17:23.305 INFO:teuthology.orchestra.run.smithi118.stdout: "flags": 0, 2024-03-24T03:17:23.305 INFO:teuthology.orchestra.run.smithi118.stdout: "default_value": "False", 2024-03-24T03:17:23.305 INFO:teuthology.orchestra.run.smithi118.stdout: "min": "", 2024-03-24T03:17:23.305 INFO:teuthology.orchestra.run.smithi118.stdout: "max": "", 2024-03-24T03:17:23.305 INFO:teuthology.orchestra.run.smithi118.stdout: "enum_allowed": [], 2024-03-24T03:17:23.306 INFO:teuthology.orchestra.run.smithi118.stdout: "desc": "Enable TLS security for all the monitoring stack daemons", 2024-03-24T03:17:23.306 INFO:teuthology.orchestra.run.smithi118.stdout: "long_desc": "", 2024-03-24T03:17:23.306 INFO:teuthology.orchestra.run.smithi118.stdout: "tags": [], 2024-03-24T03:17:23.306 INFO:teuthology.orchestra.run.smithi118.stdout: "see_also": [] 2024-03-24T03:17:23.306 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-24T03:17:23.306 INFO:teuthology.orchestra.run.smithi118.stdout: "service_discovery_port": { 2024-03-24T03:17:23.307 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "service_discovery_port", 2024-03-24T03:17:23.307 INFO:teuthology.orchestra.run.smithi118.stdout: "type": "int", 2024-03-24T03:17:23.307 INFO:teuthology.orchestra.run.smithi118.stdout: "level": "advanced", 2024-03-24T03:17:23.307 INFO:teuthology.orchestra.run.smithi118.stdout: "flags": 0, 2024-03-24T03:17:23.307 INFO:teuthology.orchestra.run.smithi118.stdout: "default_value": "8765", 2024-03-24T03:17:23.307 INFO:teuthology.orchestra.run.smithi118.stdout: "min": "", 2024-03-24T03:17:23.307 INFO:teuthology.orchestra.run.smithi118.stdout: "max": "", 2024-03-24T03:17:23.307 INFO:teuthology.orchestra.run.smithi118.stdout: "enum_allowed": [], 2024-03-24T03:17:23.307 INFO:teuthology.orchestra.run.smithi118.stdout: "desc": "cephadm service discovery port", 2024-03-24T03:17:23.308 INFO:teuthology.orchestra.run.smithi118.stdout: "long_desc": "", 2024-03-24T03:17:23.308 INFO:teuthology.orchestra.run.smithi118.stdout: "tags": [], 2024-03-24T03:17:23.308 INFO:teuthology.orchestra.run.smithi118.stdout: "see_also": [] 2024-03-24T03:17:23.308 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-24T03:17:23.308 INFO:teuthology.orchestra.run.smithi118.stdout: "ssh_config_file": { 2024-03-24T03:17:23.308 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "ssh_config_file", 2024-03-24T03:17:23.308 INFO:teuthology.orchestra.run.smithi118.stdout: "type": "str", 2024-03-24T03:17:23.308 INFO:teuthology.orchestra.run.smithi118.stdout: "level": "advanced", 2024-03-24T03:17:23.308 INFO:teuthology.orchestra.run.smithi118.stdout: "flags": 0, 2024-03-24T03:17:23.309 INFO:teuthology.orchestra.run.smithi118.stdout: "default_value": "", 2024-03-24T03:17:23.309 INFO:teuthology.orchestra.run.smithi118.stdout: "min": "", 2024-03-24T03:17:23.309 INFO:teuthology.orchestra.run.smithi118.stdout: "max": "", 2024-03-24T03:17:23.309 INFO:teuthology.orchestra.run.smithi118.stdout: "enum_allowed": [], 2024-03-24T03:17:23.309 INFO:teuthology.orchestra.run.smithi118.stdout: "desc": "customized SSH config file to connect to managed hosts", 2024-03-24T03:17:23.309 INFO:teuthology.orchestra.run.smithi118.stdout: "long_desc": "", 2024-03-24T03:17:23.309 INFO:teuthology.orchestra.run.smithi118.stdout: "tags": [], 2024-03-24T03:17:23.309 INFO:teuthology.orchestra.run.smithi118.stdout: "see_also": [] 2024-03-24T03:17:23.309 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-24T03:17:23.310 INFO:teuthology.orchestra.run.smithi118.stdout: "use_agent": { 2024-03-24T03:17:23.310 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "use_agent", 2024-03-24T03:17:23.310 INFO:teuthology.orchestra.run.smithi118.stdout: "type": "bool", 2024-03-24T03:17:23.310 INFO:teuthology.orchestra.run.smithi118.stdout: "level": "advanced", 2024-03-24T03:17:23.310 INFO:teuthology.orchestra.run.smithi118.stdout: "flags": 0, 2024-03-24T03:17:23.310 INFO:teuthology.orchestra.run.smithi118.stdout: "default_value": "False", 2024-03-24T03:17:23.310 INFO:teuthology.orchestra.run.smithi118.stdout: "min": "", 2024-03-24T03:17:23.311 INFO:teuthology.orchestra.run.smithi118.stdout: "max": "", 2024-03-24T03:17:23.311 INFO:teuthology.orchestra.run.smithi118.stdout: "enum_allowed": [], 2024-03-24T03:17:23.311 INFO:teuthology.orchestra.run.smithi118.stdout: "desc": "Use cephadm agent on each host to gather and send metadata", 2024-03-24T03:17:23.311 INFO:teuthology.orchestra.run.smithi118.stdout: "long_desc": "", 2024-03-24T03:17:23.311 INFO:teuthology.orchestra.run.smithi118.stdout: "tags": [], 2024-03-24T03:17:23.312 INFO:teuthology.orchestra.run.smithi118.stdout: "see_also": [] 2024-03-24T03:17:23.312 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-24T03:17:23.312 INFO:teuthology.orchestra.run.smithi118.stdout: "use_repo_digest": { 2024-03-24T03:17:23.312 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "use_repo_digest", 2024-03-24T03:17:23.312 INFO:teuthology.orchestra.run.smithi118.stdout: "type": "bool", 2024-03-24T03:17:23.312 INFO:teuthology.orchestra.run.smithi118.stdout: "level": "advanced", 2024-03-24T03:17:23.312 INFO:teuthology.orchestra.run.smithi118.stdout: "flags": 0, 2024-03-24T03:17:23.313 INFO:teuthology.orchestra.run.smithi118.stdout: "default_value": "True", 2024-03-24T03:17:23.313 INFO:teuthology.orchestra.run.smithi118.stdout: "min": "", 2024-03-24T03:17:23.313 INFO:teuthology.orchestra.run.smithi118.stdout: "max": "", 2024-03-24T03:17:23.313 INFO:teuthology.orchestra.run.smithi118.stdout: "enum_allowed": [], 2024-03-24T03:17:23.313 INFO:teuthology.orchestra.run.smithi118.stdout: "desc": "Automatically convert image tags to image digest. Make sure all daemons use the same image", 2024-03-24T03:17:23.313 INFO:teuthology.orchestra.run.smithi118.stdout: "long_desc": "", 2024-03-24T03:17:23.313 INFO:teuthology.orchestra.run.smithi118.stdout: "tags": [], 2024-03-24T03:17:23.313 INFO:teuthology.orchestra.run.smithi118.stdout: "see_also": [] 2024-03-24T03:17:23.314 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-24T03:17:23.314 INFO:teuthology.orchestra.run.smithi118.stdout: "warn_on_failed_host_check": { 2024-03-24T03:17:23.314 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "warn_on_failed_host_check", 2024-03-24T03:17:23.314 INFO:teuthology.orchestra.run.smithi118.stdout: "type": "bool", 2024-03-24T03:17:23.314 INFO:teuthology.orchestra.run.smithi118.stdout: "level": "advanced", 2024-03-24T03:17:23.314 INFO:teuthology.orchestra.run.smithi118.stdout: "flags": 0, 2024-03-24T03:17:23.314 INFO:teuthology.orchestra.run.smithi118.stdout: "default_value": "True", 2024-03-24T03:17:23.314 INFO:teuthology.orchestra.run.smithi118.stdout: "min": "", 2024-03-24T03:17:23.315 INFO:teuthology.orchestra.run.smithi118.stdout: "max": "", 2024-03-24T03:17:23.315 INFO:teuthology.orchestra.run.smithi118.stdout: "enum_allowed": [], 2024-03-24T03:17:23.315 INFO:teuthology.orchestra.run.smithi118.stdout: "desc": "raise a health warning if the host check fails", 2024-03-24T03:17:23.315 INFO:teuthology.orchestra.run.smithi118.stdout: "long_desc": "", 2024-03-24T03:17:23.315 INFO:teuthology.orchestra.run.smithi118.stdout: "tags": [], 2024-03-24T03:17:23.315 INFO:teuthology.orchestra.run.smithi118.stdout: "see_also": [] 2024-03-24T03:17:23.315 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-24T03:17:23.316 INFO:teuthology.orchestra.run.smithi118.stdout: "warn_on_stray_daemons": { 2024-03-24T03:17:23.316 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "warn_on_stray_daemons", 2024-03-24T03:17:23.316 INFO:teuthology.orchestra.run.smithi118.stdout: "type": "bool", 2024-03-24T03:17:23.316 INFO:teuthology.orchestra.run.smithi118.stdout: "level": "advanced", 2024-03-24T03:17:23.316 INFO:teuthology.orchestra.run.smithi118.stdout: "flags": 0, 2024-03-24T03:17:23.316 INFO:teuthology.orchestra.run.smithi118.stdout: "default_value": "True", 2024-03-24T03:17:23.316 INFO:teuthology.orchestra.run.smithi118.stdout: "min": "", 2024-03-24T03:17:23.316 INFO:teuthology.orchestra.run.smithi118.stdout: "max": "", 2024-03-24T03:17:23.316 INFO:teuthology.orchestra.run.smithi118.stdout: "enum_allowed": [], 2024-03-24T03:17:23.317 INFO:teuthology.orchestra.run.smithi118.stdout: "desc": "raise a health warning if daemons are detected that are not managed by cephadm", 2024-03-24T03:17:23.317 INFO:teuthology.orchestra.run.smithi118.stdout: "long_desc": "", 2024-03-24T03:17:23.317 INFO:teuthology.orchestra.run.smithi118.stdout: "tags": [], 2024-03-24T03:17:23.317 INFO:teuthology.orchestra.run.smithi118.stdout: "see_also": [] 2024-03-24T03:17:23.317 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-24T03:17:23.317 INFO:teuthology.orchestra.run.smithi118.stdout: "warn_on_stray_hosts": { 2024-03-24T03:17:23.317 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "warn_on_stray_hosts", 2024-03-24T03:17:23.317 INFO:teuthology.orchestra.run.smithi118.stdout: "type": "bool", 2024-03-24T03:17:23.317 INFO:teuthology.orchestra.run.smithi118.stdout: "level": "advanced", 2024-03-24T03:17:23.318 INFO:teuthology.orchestra.run.smithi118.stdout: "flags": 0, 2024-03-24T03:17:23.318 INFO:teuthology.orchestra.run.smithi118.stdout: "default_value": "True", 2024-03-24T03:17:23.318 INFO:teuthology.orchestra.run.smithi118.stdout: "min": "", 2024-03-24T03:17:23.318 INFO:teuthology.orchestra.run.smithi118.stdout: "max": "", 2024-03-24T03:17:23.318 INFO:teuthology.orchestra.run.smithi118.stdout: "enum_allowed": [], 2024-03-24T03:17:23.318 INFO:teuthology.orchestra.run.smithi118.stdout: "desc": "raise a health warning if daemons are detected on a host that is not managed by cephadm", 2024-03-24T03:17:23.318 INFO:teuthology.orchestra.run.smithi118.stdout: "long_desc": "", 2024-03-24T03:17:23.318 INFO:teuthology.orchestra.run.smithi118.stdout: "tags": [], 2024-03-24T03:17:23.319 INFO:teuthology.orchestra.run.smithi118.stdout: "see_also": [] 2024-03-24T03:17:23.319 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-03-24T03:17:23.319 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-03-24T03:17:23.319 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-24T03:17:23.319 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-24T03:17:23.319 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "crash", 2024-03-24T03:17:23.319 INFO:teuthology.orchestra.run.smithi118.stdout: "can_run": true, 2024-03-24T03:17:23.319 INFO:teuthology.orchestra.run.smithi118.stdout: "error_string": "", 2024-03-24T03:17:23.320 INFO:teuthology.orchestra.run.smithi118.stdout: "module_options": { 2024-03-24T03:17:23.320 INFO:teuthology.orchestra.run.smithi118.stdout: "log_level": { 2024-03-24T03:17:23.320 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "log_level", 2024-03-24T03:17:23.320 INFO:teuthology.orchestra.run.smithi118.stdout: "type": "str", 2024-03-24T03:17:23.320 INFO:teuthology.orchestra.run.smithi118.stdout: "level": "advanced", 2024-03-24T03:17:23.320 INFO:teuthology.orchestra.run.smithi118.stdout: "flags": 1, 2024-03-24T03:17:23.320 INFO:teuthology.orchestra.run.smithi118.stdout: "default_value": "", 2024-03-24T03:17:23.320 INFO:teuthology.orchestra.run.smithi118.stdout: "min": "", 2024-03-24T03:17:23.320 INFO:teuthology.orchestra.run.smithi118.stdout: "max": "", 2024-03-24T03:17:23.321 INFO:teuthology.orchestra.run.smithi118.stdout: "enum_allowed": [ 2024-03-24T03:17:23.321 INFO:teuthology.orchestra.run.smithi118.stdout: "", 2024-03-24T03:17:23.321 INFO:teuthology.orchestra.run.smithi118.stdout: "critical", 2024-03-24T03:17:23.321 INFO:teuthology.orchestra.run.smithi118.stdout: "debug", 2024-03-24T03:17:23.321 INFO:teuthology.orchestra.run.smithi118.stdout: "error", 2024-03-24T03:17:23.321 INFO:teuthology.orchestra.run.smithi118.stdout: "info", 2024-03-24T03:17:23.321 INFO:teuthology.orchestra.run.smithi118.stdout: "warning" 2024-03-24T03:17:23.321 INFO:teuthology.orchestra.run.smithi118.stdout: ], 2024-03-24T03:17:23.321 INFO:teuthology.orchestra.run.smithi118.stdout: "desc": "", 2024-03-24T03:17:23.322 INFO:teuthology.orchestra.run.smithi118.stdout: "long_desc": "", 2024-03-24T03:17:23.322 INFO:teuthology.orchestra.run.smithi118.stdout: "tags": [], 2024-03-24T03:17:23.322 INFO:teuthology.orchestra.run.smithi118.stdout: "see_also": [] 2024-03-24T03:17:23.322 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-24T03:17:23.322 INFO:teuthology.orchestra.run.smithi118.stdout: "log_to_cluster": { 2024-03-24T03:17:23.322 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "log_to_cluster", 2024-03-24T03:17:23.322 INFO:teuthology.orchestra.run.smithi118.stdout: "type": "bool", 2024-03-24T03:17:23.322 INFO:teuthology.orchestra.run.smithi118.stdout: "level": "advanced", 2024-03-24T03:17:23.323 INFO:teuthology.orchestra.run.smithi118.stdout: "flags": 1, 2024-03-24T03:17:23.323 INFO:teuthology.orchestra.run.smithi118.stdout: "default_value": "False", 2024-03-24T03:17:23.323 INFO:teuthology.orchestra.run.smithi118.stdout: "min": "", 2024-03-24T03:17:23.323 INFO:teuthology.orchestra.run.smithi118.stdout: "max": "", 2024-03-24T03:17:23.323 INFO:teuthology.orchestra.run.smithi118.stdout: "enum_allowed": [], 2024-03-24T03:17:23.323 INFO:teuthology.orchestra.run.smithi118.stdout: "desc": "", 2024-03-24T03:17:23.323 INFO:teuthology.orchestra.run.smithi118.stdout: "long_desc": "", 2024-03-24T03:17:23.323 INFO:teuthology.orchestra.run.smithi118.stdout: "tags": [], 2024-03-24T03:17:23.324 INFO:teuthology.orchestra.run.smithi118.stdout: "see_also": [] 2024-03-24T03:17:23.324 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-24T03:17:23.324 INFO:teuthology.orchestra.run.smithi118.stdout: "log_to_cluster_level": { 2024-03-24T03:17:23.324 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "log_to_cluster_level", 2024-03-24T03:17:23.324 INFO:teuthology.orchestra.run.smithi118.stdout: "type": "str", 2024-03-24T03:17:23.324 INFO:teuthology.orchestra.run.smithi118.stdout: "level": "advanced", 2024-03-24T03:17:23.324 INFO:teuthology.orchestra.run.smithi118.stdout: "flags": 1, 2024-03-24T03:17:23.324 INFO:teuthology.orchestra.run.smithi118.stdout: "default_value": "info", 2024-03-24T03:17:23.324 INFO:teuthology.orchestra.run.smithi118.stdout: "min": "", 2024-03-24T03:17:23.325 INFO:teuthology.orchestra.run.smithi118.stdout: "max": "", 2024-03-24T03:17:23.325 INFO:teuthology.orchestra.run.smithi118.stdout: "enum_allowed": [ 2024-03-24T03:17:23.325 INFO:teuthology.orchestra.run.smithi118.stdout: "", 2024-03-24T03:17:23.325 INFO:teuthology.orchestra.run.smithi118.stdout: "critical", 2024-03-24T03:17:23.325 INFO:teuthology.orchestra.run.smithi118.stdout: "debug", 2024-03-24T03:17:23.325 INFO:teuthology.orchestra.run.smithi118.stdout: "error", 2024-03-24T03:17:23.325 INFO:teuthology.orchestra.run.smithi118.stdout: "info", 2024-03-24T03:17:23.325 INFO:teuthology.orchestra.run.smithi118.stdout: "warning" 2024-03-24T03:17:23.326 INFO:teuthology.orchestra.run.smithi118.stdout: ], 2024-03-24T03:17:23.326 INFO:teuthology.orchestra.run.smithi118.stdout: "desc": "", 2024-03-24T03:17:23.326 INFO:teuthology.orchestra.run.smithi118.stdout: "long_desc": "", 2024-03-24T03:17:23.326 INFO:teuthology.orchestra.run.smithi118.stdout: "tags": [], 2024-03-24T03:17:23.326 INFO:teuthology.orchestra.run.smithi118.stdout: "see_also": [] 2024-03-24T03:17:23.326 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-24T03:17:23.326 INFO:teuthology.orchestra.run.smithi118.stdout: "log_to_file": { 2024-03-24T03:17:23.326 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "log_to_file", 2024-03-24T03:17:23.327 INFO:teuthology.orchestra.run.smithi118.stdout: "type": "bool", 2024-03-24T03:17:23.327 INFO:teuthology.orchestra.run.smithi118.stdout: "level": "advanced", 2024-03-24T03:17:23.327 INFO:teuthology.orchestra.run.smithi118.stdout: "flags": 1, 2024-03-24T03:17:23.327 INFO:teuthology.orchestra.run.smithi118.stdout: "default_value": "False", 2024-03-24T03:17:23.327 INFO:teuthology.orchestra.run.smithi118.stdout: "min": "", 2024-03-24T03:17:23.327 INFO:teuthology.orchestra.run.smithi118.stdout: "max": "", 2024-03-24T03:17:23.327 INFO:teuthology.orchestra.run.smithi118.stdout: "enum_allowed": [], 2024-03-24T03:17:23.327 INFO:teuthology.orchestra.run.smithi118.stdout: "desc": "", 2024-03-24T03:17:23.327 INFO:teuthology.orchestra.run.smithi118.stdout: "long_desc": "", 2024-03-24T03:17:23.328 INFO:teuthology.orchestra.run.smithi118.stdout: "tags": [], 2024-03-24T03:17:23.328 INFO:teuthology.orchestra.run.smithi118.stdout: "see_also": [] 2024-03-24T03:17:23.328 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-24T03:17:23.328 INFO:teuthology.orchestra.run.smithi118.stdout: "retain_interval": { 2024-03-24T03:17:23.328 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "retain_interval", 2024-03-24T03:17:23.328 INFO:teuthology.orchestra.run.smithi118.stdout: "type": "secs", 2024-03-24T03:17:23.328 INFO:teuthology.orchestra.run.smithi118.stdout: "level": "advanced", 2024-03-24T03:17:23.328 INFO:teuthology.orchestra.run.smithi118.stdout: "flags": 1, 2024-03-24T03:17:23.328 INFO:teuthology.orchestra.run.smithi118.stdout: "default_value": "31536000", 2024-03-24T03:17:23.329 INFO:teuthology.orchestra.run.smithi118.stdout: "min": "", 2024-03-24T03:17:23.329 INFO:teuthology.orchestra.run.smithi118.stdout: "max": "", 2024-03-24T03:17:23.329 INFO:teuthology.orchestra.run.smithi118.stdout: "enum_allowed": [], 2024-03-24T03:17:23.329 INFO:teuthology.orchestra.run.smithi118.stdout: "desc": "how long to retain crashes before pruning them", 2024-03-24T03:17:23.329 INFO:teuthology.orchestra.run.smithi118.stdout: "long_desc": "", 2024-03-24T03:17:23.329 INFO:teuthology.orchestra.run.smithi118.stdout: "tags": [], 2024-03-24T03:17:23.329 INFO:teuthology.orchestra.run.smithi118.stdout: "see_also": [] 2024-03-24T03:17:23.329 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-24T03:17:23.329 INFO:teuthology.orchestra.run.smithi118.stdout: "warn_recent_interval": { 2024-03-24T03:17:23.330 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "warn_recent_interval", 2024-03-24T03:17:23.330 INFO:teuthology.orchestra.run.smithi118.stdout: "type": "secs", 2024-03-24T03:17:23.330 INFO:teuthology.orchestra.run.smithi118.stdout: "level": "advanced", 2024-03-24T03:17:23.330 INFO:teuthology.orchestra.run.smithi118.stdout: "flags": 1, 2024-03-24T03:17:23.330 INFO:teuthology.orchestra.run.smithi118.stdout: "default_value": "1209600", 2024-03-24T03:17:23.330 INFO:teuthology.orchestra.run.smithi118.stdout: "min": "", 2024-03-24T03:17:23.330 INFO:teuthology.orchestra.run.smithi118.stdout: "max": "", 2024-03-24T03:17:23.330 INFO:teuthology.orchestra.run.smithi118.stdout: "enum_allowed": [], 2024-03-24T03:17:23.331 INFO:teuthology.orchestra.run.smithi118.stdout: "desc": "time interval in which to warn about recent crashes", 2024-03-24T03:17:23.331 INFO:teuthology.orchestra.run.smithi118.stdout: "long_desc": "", 2024-03-24T03:17:23.331 INFO:teuthology.orchestra.run.smithi118.stdout: "tags": [], 2024-03-24T03:17:23.331 INFO:teuthology.orchestra.run.smithi118.stdout: "see_also": [] 2024-03-24T03:17:23.331 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-03-24T03:17:23.331 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-03-24T03:17:23.331 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-24T03:17:23.331 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-24T03:17:23.331 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "dashboard", 2024-03-24T03:17:23.332 INFO:teuthology.orchestra.run.smithi118.stdout: "can_run": true, 2024-03-24T03:17:23.332 INFO:teuthology.orchestra.run.smithi118.stdout: "error_string": "", 2024-03-24T03:17:23.332 INFO:teuthology.orchestra.run.smithi118.stdout: "module_options": { 2024-03-24T03:17:23.332 INFO:teuthology.orchestra.run.smithi118.stdout: "ACCOUNT_LOCKOUT_ATTEMPTS": { 2024-03-24T03:17:23.332 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "ACCOUNT_LOCKOUT_ATTEMPTS", 2024-03-24T03:17:23.332 INFO:teuthology.orchestra.run.smithi118.stdout: "type": "int", 2024-03-24T03:17:23.332 INFO:teuthology.orchestra.run.smithi118.stdout: "level": "advanced", 2024-03-24T03:17:23.332 INFO:teuthology.orchestra.run.smithi118.stdout: "flags": 0, 2024-03-24T03:17:23.333 INFO:teuthology.orchestra.run.smithi118.stdout: "default_value": "10", 2024-03-24T03:17:23.333 INFO:teuthology.orchestra.run.smithi118.stdout: "min": "", 2024-03-24T03:17:23.333 INFO:teuthology.orchestra.run.smithi118.stdout: "max": "", 2024-03-24T03:17:23.333 INFO:teuthology.orchestra.run.smithi118.stdout: "enum_allowed": [], 2024-03-24T03:17:23.333 INFO:teuthology.orchestra.run.smithi118.stdout: "desc": "", 2024-03-24T03:17:23.333 INFO:teuthology.orchestra.run.smithi118.stdout: "long_desc": "", 2024-03-24T03:17:23.333 INFO:teuthology.orchestra.run.smithi118.stdout: "tags": [], 2024-03-24T03:17:23.333 INFO:teuthology.orchestra.run.smithi118.stdout: "see_also": [] 2024-03-24T03:17:23.333 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-24T03:17:23.334 INFO:teuthology.orchestra.run.smithi118.stdout: "ALERTMANAGER_API_HOST": { 2024-03-24T03:17:23.334 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "ALERTMANAGER_API_HOST", 2024-03-24T03:17:23.334 INFO:teuthology.orchestra.run.smithi118.stdout: "type": "str", 2024-03-24T03:17:23.334 INFO:teuthology.orchestra.run.smithi118.stdout: "level": "advanced", 2024-03-24T03:17:23.334 INFO:teuthology.orchestra.run.smithi118.stdout: "flags": 0, 2024-03-24T03:17:23.334 INFO:teuthology.orchestra.run.smithi118.stdout: "default_value": "", 2024-03-24T03:17:23.334 INFO:teuthology.orchestra.run.smithi118.stdout: "min": "", 2024-03-24T03:17:23.334 INFO:teuthology.orchestra.run.smithi118.stdout: "max": "", 2024-03-24T03:17:23.335 INFO:teuthology.orchestra.run.smithi118.stdout: "enum_allowed": [], 2024-03-24T03:17:23.335 INFO:teuthology.orchestra.run.smithi118.stdout: "desc": "", 2024-03-24T03:17:23.335 INFO:teuthology.orchestra.run.smithi118.stdout: "long_desc": "", 2024-03-24T03:17:23.335 INFO:teuthology.orchestra.run.smithi118.stdout: "tags": [], 2024-03-24T03:17:23.335 INFO:teuthology.orchestra.run.smithi118.stdout: "see_also": [] 2024-03-24T03:17:23.335 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-24T03:17:23.335 INFO:teuthology.orchestra.run.smithi118.stdout: "ALERTMANAGER_API_SSL_VERIFY": { 2024-03-24T03:17:23.335 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "ALERTMANAGER_API_SSL_VERIFY", 2024-03-24T03:17:23.335 INFO:teuthology.orchestra.run.smithi118.stdout: "type": "bool", 2024-03-24T03:17:23.336 INFO:teuthology.orchestra.run.smithi118.stdout: "level": "advanced", 2024-03-24T03:17:23.336 INFO:teuthology.orchestra.run.smithi118.stdout: "flags": 0, 2024-03-24T03:17:23.336 INFO:teuthology.orchestra.run.smithi118.stdout: "default_value": "True", 2024-03-24T03:17:23.336 INFO:teuthology.orchestra.run.smithi118.stdout: "min": "", 2024-03-24T03:17:23.336 INFO:teuthology.orchestra.run.smithi118.stdout: "max": "", 2024-03-24T03:17:23.336 INFO:teuthology.orchestra.run.smithi118.stdout: "enum_allowed": [], 2024-03-24T03:17:23.336 INFO:teuthology.orchestra.run.smithi118.stdout: "desc": "", 2024-03-24T03:17:23.336 INFO:teuthology.orchestra.run.smithi118.stdout: "long_desc": "", 2024-03-24T03:17:23.336 INFO:teuthology.orchestra.run.smithi118.stdout: "tags": [], 2024-03-24T03:17:23.337 INFO:teuthology.orchestra.run.smithi118.stdout: "see_also": [] 2024-03-24T03:17:23.337 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-24T03:17:23.337 INFO:teuthology.orchestra.run.smithi118.stdout: "AUDIT_API_ENABLED": { 2024-03-24T03:17:23.337 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "AUDIT_API_ENABLED", 2024-03-24T03:17:23.337 INFO:teuthology.orchestra.run.smithi118.stdout: "type": "bool", 2024-03-24T03:17:23.337 INFO:teuthology.orchestra.run.smithi118.stdout: "level": "advanced", 2024-03-24T03:17:23.337 INFO:teuthology.orchestra.run.smithi118.stdout: "flags": 0, 2024-03-24T03:17:23.337 INFO:teuthology.orchestra.run.smithi118.stdout: "default_value": "False", 2024-03-24T03:17:23.338 INFO:teuthology.orchestra.run.smithi118.stdout: "min": "", 2024-03-24T03:17:23.338 INFO:teuthology.orchestra.run.smithi118.stdout: "max": "", 2024-03-24T03:17:23.338 INFO:teuthology.orchestra.run.smithi118.stdout: "enum_allowed": [], 2024-03-24T03:17:23.338 INFO:teuthology.orchestra.run.smithi118.stdout: "desc": "", 2024-03-24T03:17:23.338 INFO:teuthology.orchestra.run.smithi118.stdout: "long_desc": "", 2024-03-24T03:17:23.338 INFO:teuthology.orchestra.run.smithi118.stdout: "tags": [], 2024-03-24T03:17:23.338 INFO:teuthology.orchestra.run.smithi118.stdout: "see_also": [] 2024-03-24T03:17:23.338 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-24T03:17:23.339 INFO:teuthology.orchestra.run.smithi118.stdout: "AUDIT_API_LOG_PAYLOAD": { 2024-03-24T03:17:23.339 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "AUDIT_API_LOG_PAYLOAD", 2024-03-24T03:17:23.339 INFO:teuthology.orchestra.run.smithi118.stdout: "type": "bool", 2024-03-24T03:17:23.339 INFO:teuthology.orchestra.run.smithi118.stdout: "level": "advanced", 2024-03-24T03:17:23.339 INFO:teuthology.orchestra.run.smithi118.stdout: "flags": 0, 2024-03-24T03:17:23.339 INFO:teuthology.orchestra.run.smithi118.stdout: "default_value": "True", 2024-03-24T03:17:23.339 INFO:teuthology.orchestra.run.smithi118.stdout: "min": "", 2024-03-24T03:17:23.339 INFO:teuthology.orchestra.run.smithi118.stdout: "max": "", 2024-03-24T03:17:23.340 INFO:teuthology.orchestra.run.smithi118.stdout: "enum_allowed": [], 2024-03-24T03:17:23.340 INFO:teuthology.orchestra.run.smithi118.stdout: "desc": "", 2024-03-24T03:17:23.340 INFO:teuthology.orchestra.run.smithi118.stdout: "long_desc": "", 2024-03-24T03:17:23.340 INFO:teuthology.orchestra.run.smithi118.stdout: "tags": [], 2024-03-24T03:17:23.340 INFO:teuthology.orchestra.run.smithi118.stdout: "see_also": [] 2024-03-24T03:17:23.340 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-24T03:17:23.340 INFO:teuthology.orchestra.run.smithi118.stdout: "ENABLE_BROWSABLE_API": { 2024-03-24T03:17:23.340 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "ENABLE_BROWSABLE_API", 2024-03-24T03:17:23.340 INFO:teuthology.orchestra.run.smithi118.stdout: "type": "bool", 2024-03-24T03:17:23.341 INFO:teuthology.orchestra.run.smithi118.stdout: "level": "advanced", 2024-03-24T03:17:23.341 INFO:teuthology.orchestra.run.smithi118.stdout: "flags": 0, 2024-03-24T03:17:23.341 INFO:teuthology.orchestra.run.smithi118.stdout: "default_value": "True", 2024-03-24T03:17:23.341 INFO:teuthology.orchestra.run.smithi118.stdout: "min": "", 2024-03-24T03:17:23.341 INFO:teuthology.orchestra.run.smithi118.stdout: "max": "", 2024-03-24T03:17:23.341 INFO:teuthology.orchestra.run.smithi118.stdout: "enum_allowed": [], 2024-03-24T03:17:23.341 INFO:teuthology.orchestra.run.smithi118.stdout: "desc": "", 2024-03-24T03:17:23.341 INFO:teuthology.orchestra.run.smithi118.stdout: "long_desc": "", 2024-03-24T03:17:23.342 INFO:teuthology.orchestra.run.smithi118.stdout: "tags": [], 2024-03-24T03:17:23.342 INFO:teuthology.orchestra.run.smithi118.stdout: "see_also": [] 2024-03-24T03:17:23.342 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-24T03:17:23.342 INFO:teuthology.orchestra.run.smithi118.stdout: "FEATURE_TOGGLE_CEPHFS": { 2024-03-24T03:17:23.342 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "FEATURE_TOGGLE_CEPHFS", 2024-03-24T03:17:23.342 INFO:teuthology.orchestra.run.smithi118.stdout: "type": "bool", 2024-03-24T03:17:23.342 INFO:teuthology.orchestra.run.smithi118.stdout: "level": "advanced", 2024-03-24T03:17:23.342 INFO:teuthology.orchestra.run.smithi118.stdout: "flags": 0, 2024-03-24T03:17:23.342 INFO:teuthology.orchestra.run.smithi118.stdout: "default_value": "True", 2024-03-24T03:17:23.343 INFO:teuthology.orchestra.run.smithi118.stdout: "min": "", 2024-03-24T03:17:23.343 INFO:teuthology.orchestra.run.smithi118.stdout: "max": "", 2024-03-24T03:17:23.343 INFO:teuthology.orchestra.run.smithi118.stdout: "enum_allowed": [], 2024-03-24T03:17:23.343 INFO:teuthology.orchestra.run.smithi118.stdout: "desc": "", 2024-03-24T03:17:23.343 INFO:teuthology.orchestra.run.smithi118.stdout: "long_desc": "", 2024-03-24T03:17:23.343 INFO:teuthology.orchestra.run.smithi118.stdout: "tags": [], 2024-03-24T03:17:23.343 INFO:teuthology.orchestra.run.smithi118.stdout: "see_also": [] 2024-03-24T03:17:23.343 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-24T03:17:23.344 INFO:teuthology.orchestra.run.smithi118.stdout: "FEATURE_TOGGLE_DASHBOARD": { 2024-03-24T03:17:23.344 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "FEATURE_TOGGLE_DASHBOARD", 2024-03-24T03:17:23.344 INFO:teuthology.orchestra.run.smithi118.stdout: "type": "bool", 2024-03-24T03:17:23.344 INFO:teuthology.orchestra.run.smithi118.stdout: "level": "advanced", 2024-03-24T03:17:23.344 INFO:teuthology.orchestra.run.smithi118.stdout: "flags": 0, 2024-03-24T03:17:23.344 INFO:teuthology.orchestra.run.smithi118.stdout: "default_value": "True", 2024-03-24T03:17:23.344 INFO:teuthology.orchestra.run.smithi118.stdout: "min": "", 2024-03-24T03:17:23.344 INFO:teuthology.orchestra.run.smithi118.stdout: "max": "", 2024-03-24T03:17:23.345 INFO:teuthology.orchestra.run.smithi118.stdout: "enum_allowed": [], 2024-03-24T03:17:23.345 INFO:teuthology.orchestra.run.smithi118.stdout: "desc": "", 2024-03-24T03:17:23.345 INFO:teuthology.orchestra.run.smithi118.stdout: "long_desc": "", 2024-03-24T03:17:23.345 INFO:teuthology.orchestra.run.smithi118.stdout: "tags": [], 2024-03-24T03:17:23.345 INFO:teuthology.orchestra.run.smithi118.stdout: "see_also": [] 2024-03-24T03:17:23.345 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-24T03:17:23.345 INFO:teuthology.orchestra.run.smithi118.stdout: "FEATURE_TOGGLE_ISCSI": { 2024-03-24T03:17:23.345 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "FEATURE_TOGGLE_ISCSI", 2024-03-24T03:17:23.345 INFO:teuthology.orchestra.run.smithi118.stdout: "type": "bool", 2024-03-24T03:17:23.346 INFO:teuthology.orchestra.run.smithi118.stdout: "level": "advanced", 2024-03-24T03:17:23.346 INFO:teuthology.orchestra.run.smithi118.stdout: "flags": 0, 2024-03-24T03:17:23.346 INFO:teuthology.orchestra.run.smithi118.stdout: "default_value": "True", 2024-03-24T03:17:23.346 INFO:teuthology.orchestra.run.smithi118.stdout: "min": "", 2024-03-24T03:17:23.346 INFO:teuthology.orchestra.run.smithi118.stdout: "max": "", 2024-03-24T03:17:23.346 INFO:teuthology.orchestra.run.smithi118.stdout: "enum_allowed": [], 2024-03-24T03:17:23.346 INFO:teuthology.orchestra.run.smithi118.stdout: "desc": "", 2024-03-24T03:17:23.346 INFO:teuthology.orchestra.run.smithi118.stdout: "long_desc": "", 2024-03-24T03:17:23.347 INFO:teuthology.orchestra.run.smithi118.stdout: "tags": [], 2024-03-24T03:17:23.347 INFO:teuthology.orchestra.run.smithi118.stdout: "see_also": [] 2024-03-24T03:17:23.347 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-24T03:17:23.347 INFO:teuthology.orchestra.run.smithi118.stdout: "FEATURE_TOGGLE_MIRRORING": { 2024-03-24T03:17:23.347 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "FEATURE_TOGGLE_MIRRORING", 2024-03-24T03:17:23.347 INFO:teuthology.orchestra.run.smithi118.stdout: "type": "bool", 2024-03-24T03:17:23.347 INFO:teuthology.orchestra.run.smithi118.stdout: "level": "advanced", 2024-03-24T03:17:23.347 INFO:teuthology.orchestra.run.smithi118.stdout: "flags": 0, 2024-03-24T03:17:23.348 INFO:teuthology.orchestra.run.smithi118.stdout: "default_value": "True", 2024-03-24T03:17:23.348 INFO:teuthology.orchestra.run.smithi118.stdout: "min": "", 2024-03-24T03:17:23.348 INFO:teuthology.orchestra.run.smithi118.stdout: "max": "", 2024-03-24T03:17:23.348 INFO:teuthology.orchestra.run.smithi118.stdout: "enum_allowed": [], 2024-03-24T03:17:23.348 INFO:teuthology.orchestra.run.smithi118.stdout: "desc": "", 2024-03-24T03:17:23.348 INFO:teuthology.orchestra.run.smithi118.stdout: "long_desc": "", 2024-03-24T03:17:23.348 INFO:teuthology.orchestra.run.smithi118.stdout: "tags": [], 2024-03-24T03:17:23.348 INFO:teuthology.orchestra.run.smithi118.stdout: "see_also": [] 2024-03-24T03:17:23.348 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-24T03:17:23.349 INFO:teuthology.orchestra.run.smithi118.stdout: "FEATURE_TOGGLE_NFS": { 2024-03-24T03:17:23.349 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "FEATURE_TOGGLE_NFS", 2024-03-24T03:17:23.349 INFO:teuthology.orchestra.run.smithi118.stdout: "type": "bool", 2024-03-24T03:17:23.349 INFO:teuthology.orchestra.run.smithi118.stdout: "level": "advanced", 2024-03-24T03:17:23.349 INFO:teuthology.orchestra.run.smithi118.stdout: "flags": 0, 2024-03-24T03:17:23.349 INFO:teuthology.orchestra.run.smithi118.stdout: "default_value": "True", 2024-03-24T03:17:23.349 INFO:teuthology.orchestra.run.smithi118.stdout: "min": "", 2024-03-24T03:17:23.350 INFO:teuthology.orchestra.run.smithi118.stdout: "max": "", 2024-03-24T03:17:23.350 INFO:teuthology.orchestra.run.smithi118.stdout: "enum_allowed": [], 2024-03-24T03:17:23.350 INFO:teuthology.orchestra.run.smithi118.stdout: "desc": "", 2024-03-24T03:17:23.350 INFO:teuthology.orchestra.run.smithi118.stdout: "long_desc": "", 2024-03-24T03:17:23.350 INFO:teuthology.orchestra.run.smithi118.stdout: "tags": [], 2024-03-24T03:17:23.350 INFO:teuthology.orchestra.run.smithi118.stdout: "see_also": [] 2024-03-24T03:17:23.350 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-24T03:17:23.350 INFO:teuthology.orchestra.run.smithi118.stdout: "FEATURE_TOGGLE_RBD": { 2024-03-24T03:17:23.351 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "FEATURE_TOGGLE_RBD", 2024-03-24T03:17:23.351 INFO:teuthology.orchestra.run.smithi118.stdout: "type": "bool", 2024-03-24T03:17:23.351 INFO:teuthology.orchestra.run.smithi118.stdout: "level": "advanced", 2024-03-24T03:17:23.351 INFO:teuthology.orchestra.run.smithi118.stdout: "flags": 0, 2024-03-24T03:17:23.351 INFO:teuthology.orchestra.run.smithi118.stdout: "default_value": "True", 2024-03-24T03:17:23.351 INFO:teuthology.orchestra.run.smithi118.stdout: "min": "", 2024-03-24T03:17:23.351 INFO:teuthology.orchestra.run.smithi118.stdout: "max": "", 2024-03-24T03:17:23.351 INFO:teuthology.orchestra.run.smithi118.stdout: "enum_allowed": [], 2024-03-24T03:17:23.352 INFO:teuthology.orchestra.run.smithi118.stdout: "desc": "", 2024-03-24T03:17:23.352 INFO:teuthology.orchestra.run.smithi118.stdout: "long_desc": "", 2024-03-24T03:17:23.352 INFO:teuthology.orchestra.run.smithi118.stdout: "tags": [], 2024-03-24T03:17:23.352 INFO:teuthology.orchestra.run.smithi118.stdout: "see_also": [] 2024-03-24T03:17:23.352 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-24T03:17:23.352 INFO:teuthology.orchestra.run.smithi118.stdout: "FEATURE_TOGGLE_RGW": { 2024-03-24T03:17:23.352 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "FEATURE_TOGGLE_RGW", 2024-03-24T03:17:23.352 INFO:teuthology.orchestra.run.smithi118.stdout: "type": "bool", 2024-03-24T03:17:23.352 INFO:teuthology.orchestra.run.smithi118.stdout: "level": "advanced", 2024-03-24T03:17:23.353 INFO:teuthology.orchestra.run.smithi118.stdout: "flags": 0, 2024-03-24T03:17:23.353 INFO:teuthology.orchestra.run.smithi118.stdout: "default_value": "True", 2024-03-24T03:17:23.353 INFO:teuthology.orchestra.run.smithi118.stdout: "min": "", 2024-03-24T03:17:23.353 INFO:teuthology.orchestra.run.smithi118.stdout: "max": "", 2024-03-24T03:17:23.353 INFO:teuthology.orchestra.run.smithi118.stdout: "enum_allowed": [], 2024-03-24T03:17:23.353 INFO:teuthology.orchestra.run.smithi118.stdout: "desc": "", 2024-03-24T03:17:23.353 INFO:teuthology.orchestra.run.smithi118.stdout: "long_desc": "", 2024-03-24T03:17:23.353 INFO:teuthology.orchestra.run.smithi118.stdout: "tags": [], 2024-03-24T03:17:23.354 INFO:teuthology.orchestra.run.smithi118.stdout: "see_also": [] 2024-03-24T03:17:23.354 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-24T03:17:23.354 INFO:teuthology.orchestra.run.smithi118.stdout: "GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE": { 2024-03-24T03:17:23.354 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE", 2024-03-24T03:17:23.354 INFO:teuthology.orchestra.run.smithi118.stdout: "type": "str", 2024-03-24T03:17:23.354 INFO:teuthology.orchestra.run.smithi118.stdout: "level": "advanced", 2024-03-24T03:17:23.354 INFO:teuthology.orchestra.run.smithi118.stdout: "flags": 0, 2024-03-24T03:17:23.354 INFO:teuthology.orchestra.run.smithi118.stdout: "default_value": "", 2024-03-24T03:17:23.355 INFO:teuthology.orchestra.run.smithi118.stdout: "min": "", 2024-03-24T03:17:23.355 INFO:teuthology.orchestra.run.smithi118.stdout: "max": "", 2024-03-24T03:17:23.355 INFO:teuthology.orchestra.run.smithi118.stdout: "enum_allowed": [], 2024-03-24T03:17:23.355 INFO:teuthology.orchestra.run.smithi118.stdout: "desc": "", 2024-03-24T03:17:23.355 INFO:teuthology.orchestra.run.smithi118.stdout: "long_desc": "", 2024-03-24T03:17:23.355 INFO:teuthology.orchestra.run.smithi118.stdout: "tags": [], 2024-03-24T03:17:23.355 INFO:teuthology.orchestra.run.smithi118.stdout: "see_also": [] 2024-03-24T03:17:23.356 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-24T03:17:23.356 INFO:teuthology.orchestra.run.smithi118.stdout: "GRAFANA_API_PASSWORD": { 2024-03-24T03:17:23.356 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "GRAFANA_API_PASSWORD", 2024-03-24T03:17:23.356 INFO:teuthology.orchestra.run.smithi118.stdout: "type": "str", 2024-03-24T03:17:23.356 INFO:teuthology.orchestra.run.smithi118.stdout: "level": "advanced", 2024-03-24T03:17:23.356 INFO:teuthology.orchestra.run.smithi118.stdout: "flags": 0, 2024-03-24T03:17:23.356 INFO:teuthology.orchestra.run.smithi118.stdout: "default_value": "admin", 2024-03-24T03:17:23.356 INFO:teuthology.orchestra.run.smithi118.stdout: "min": "", 2024-03-24T03:17:23.356 INFO:teuthology.orchestra.run.smithi118.stdout: "max": "", 2024-03-24T03:17:23.357 INFO:teuthology.orchestra.run.smithi118.stdout: "enum_allowed": [], 2024-03-24T03:17:23.357 INFO:teuthology.orchestra.run.smithi118.stdout: "desc": "", 2024-03-24T03:17:23.357 INFO:teuthology.orchestra.run.smithi118.stdout: "long_desc": "", 2024-03-24T03:17:23.357 INFO:teuthology.orchestra.run.smithi118.stdout: "tags": [], 2024-03-24T03:17:23.357 INFO:teuthology.orchestra.run.smithi118.stdout: "see_also": [] 2024-03-24T03:17:23.357 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-24T03:17:23.357 INFO:teuthology.orchestra.run.smithi118.stdout: "GRAFANA_API_SSL_VERIFY": { 2024-03-24T03:17:23.357 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "GRAFANA_API_SSL_VERIFY", 2024-03-24T03:17:23.358 INFO:teuthology.orchestra.run.smithi118.stdout: "type": "bool", 2024-03-24T03:17:23.358 INFO:teuthology.orchestra.run.smithi118.stdout: "level": "advanced", 2024-03-24T03:17:23.358 INFO:teuthology.orchestra.run.smithi118.stdout: "flags": 0, 2024-03-24T03:17:23.358 INFO:teuthology.orchestra.run.smithi118.stdout: "default_value": "True", 2024-03-24T03:17:23.358 INFO:teuthology.orchestra.run.smithi118.stdout: "min": "", 2024-03-24T03:17:23.358 INFO:teuthology.orchestra.run.smithi118.stdout: "max": "", 2024-03-24T03:17:23.358 INFO:teuthology.orchestra.run.smithi118.stdout: "enum_allowed": [], 2024-03-24T03:17:23.359 INFO:teuthology.orchestra.run.smithi118.stdout: "desc": "", 2024-03-24T03:17:23.359 INFO:teuthology.orchestra.run.smithi118.stdout: "long_desc": "", 2024-03-24T03:17:23.359 INFO:teuthology.orchestra.run.smithi118.stdout: "tags": [], 2024-03-24T03:17:23.359 INFO:teuthology.orchestra.run.smithi118.stdout: "see_also": [] 2024-03-24T03:17:23.359 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-24T03:17:23.359 INFO:teuthology.orchestra.run.smithi118.stdout: "GRAFANA_API_URL": { 2024-03-24T03:17:23.359 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "GRAFANA_API_URL", 2024-03-24T03:17:23.359 INFO:teuthology.orchestra.run.smithi118.stdout: "type": "str", 2024-03-24T03:17:23.359 INFO:teuthology.orchestra.run.smithi118.stdout: "level": "advanced", 2024-03-24T03:17:23.360 INFO:teuthology.orchestra.run.smithi118.stdout: "flags": 0, 2024-03-24T03:17:23.360 INFO:teuthology.orchestra.run.smithi118.stdout: "default_value": "", 2024-03-24T03:17:23.360 INFO:teuthology.orchestra.run.smithi118.stdout: "min": "", 2024-03-24T03:17:23.360 INFO:teuthology.orchestra.run.smithi118.stdout: "max": "", 2024-03-24T03:17:23.360 INFO:teuthology.orchestra.run.smithi118.stdout: "enum_allowed": [], 2024-03-24T03:17:23.360 INFO:teuthology.orchestra.run.smithi118.stdout: "desc": "", 2024-03-24T03:17:23.360 INFO:teuthology.orchestra.run.smithi118.stdout: "long_desc": "", 2024-03-24T03:17:23.360 INFO:teuthology.orchestra.run.smithi118.stdout: "tags": [], 2024-03-24T03:17:23.360 INFO:teuthology.orchestra.run.smithi118.stdout: "see_also": [] 2024-03-24T03:17:23.361 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-24T03:17:23.361 INFO:teuthology.orchestra.run.smithi118.stdout: "GRAFANA_API_USERNAME": { 2024-03-24T03:17:23.361 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "GRAFANA_API_USERNAME", 2024-03-24T03:17:23.361 INFO:teuthology.orchestra.run.smithi118.stdout: "type": "str", 2024-03-24T03:17:23.361 INFO:teuthology.orchestra.run.smithi118.stdout: "level": "advanced", 2024-03-24T03:17:23.361 INFO:teuthology.orchestra.run.smithi118.stdout: "flags": 0, 2024-03-24T03:17:23.361 INFO:teuthology.orchestra.run.smithi118.stdout: "default_value": "admin", 2024-03-24T03:17:23.361 INFO:teuthology.orchestra.run.smithi118.stdout: "min": "", 2024-03-24T03:17:23.362 INFO:teuthology.orchestra.run.smithi118.stdout: "max": "", 2024-03-24T03:17:23.362 INFO:teuthology.orchestra.run.smithi118.stdout: "enum_allowed": [], 2024-03-24T03:17:23.362 INFO:teuthology.orchestra.run.smithi118.stdout: "desc": "", 2024-03-24T03:17:23.362 INFO:teuthology.orchestra.run.smithi118.stdout: "long_desc": "", 2024-03-24T03:17:23.362 INFO:teuthology.orchestra.run.smithi118.stdout: "tags": [], 2024-03-24T03:17:23.362 INFO:teuthology.orchestra.run.smithi118.stdout: "see_also": [] 2024-03-24T03:17:23.362 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-24T03:17:23.363 INFO:teuthology.orchestra.run.smithi118.stdout: "GRAFANA_FRONTEND_API_URL": { 2024-03-24T03:17:23.363 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "GRAFANA_FRONTEND_API_URL", 2024-03-24T03:17:23.363 INFO:teuthology.orchestra.run.smithi118.stdout: "type": "str", 2024-03-24T03:17:23.363 INFO:teuthology.orchestra.run.smithi118.stdout: "level": "advanced", 2024-03-24T03:17:23.363 INFO:teuthology.orchestra.run.smithi118.stdout: "flags": 0, 2024-03-24T03:17:23.363 INFO:teuthology.orchestra.run.smithi118.stdout: "default_value": "", 2024-03-24T03:17:23.363 INFO:teuthology.orchestra.run.smithi118.stdout: "min": "", 2024-03-24T03:17:23.363 INFO:teuthology.orchestra.run.smithi118.stdout: "max": "", 2024-03-24T03:17:23.363 INFO:teuthology.orchestra.run.smithi118.stdout: "enum_allowed": [], 2024-03-24T03:17:23.364 INFO:teuthology.orchestra.run.smithi118.stdout: "desc": "", 2024-03-24T03:17:23.364 INFO:teuthology.orchestra.run.smithi118.stdout: "long_desc": "", 2024-03-24T03:17:23.364 INFO:teuthology.orchestra.run.smithi118.stdout: "tags": [], 2024-03-24T03:17:23.364 INFO:teuthology.orchestra.run.smithi118.stdout: "see_also": [] 2024-03-24T03:17:23.364 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-24T03:17:23.364 INFO:teuthology.orchestra.run.smithi118.stdout: "GRAFANA_UPDATE_DASHBOARDS": { 2024-03-24T03:17:23.364 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "GRAFANA_UPDATE_DASHBOARDS", 2024-03-24T03:17:23.364 INFO:teuthology.orchestra.run.smithi118.stdout: "type": "bool", 2024-03-24T03:17:23.365 INFO:teuthology.orchestra.run.smithi118.stdout: "level": "advanced", 2024-03-24T03:17:23.365 INFO:teuthology.orchestra.run.smithi118.stdout: "flags": 0, 2024-03-24T03:17:23.365 INFO:teuthology.orchestra.run.smithi118.stdout: "default_value": "False", 2024-03-24T03:17:23.365 INFO:teuthology.orchestra.run.smithi118.stdout: "min": "", 2024-03-24T03:17:23.365 INFO:teuthology.orchestra.run.smithi118.stdout: "max": "", 2024-03-24T03:17:23.365 INFO:teuthology.orchestra.run.smithi118.stdout: "enum_allowed": [], 2024-03-24T03:17:23.366 INFO:teuthology.orchestra.run.smithi118.stdout: "desc": "", 2024-03-24T03:17:23.366 INFO:teuthology.orchestra.run.smithi118.stdout: "long_desc": "", 2024-03-24T03:17:23.366 INFO:teuthology.orchestra.run.smithi118.stdout: "tags": [], 2024-03-24T03:17:23.366 INFO:teuthology.orchestra.run.smithi118.stdout: "see_also": [] 2024-03-24T03:17:23.366 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-24T03:17:23.366 INFO:teuthology.orchestra.run.smithi118.stdout: "ISCSI_API_SSL_VERIFICATION": { 2024-03-24T03:17:23.366 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "ISCSI_API_SSL_VERIFICATION", 2024-03-24T03:17:23.366 INFO:teuthology.orchestra.run.smithi118.stdout: "type": "bool", 2024-03-24T03:17:23.367 INFO:teuthology.orchestra.run.smithi118.stdout: "level": "advanced", 2024-03-24T03:17:23.367 INFO:teuthology.orchestra.run.smithi118.stdout: "flags": 0, 2024-03-24T03:17:23.367 INFO:teuthology.orchestra.run.smithi118.stdout: "default_value": "True", 2024-03-24T03:17:23.367 INFO:teuthology.orchestra.run.smithi118.stdout: "min": "", 2024-03-24T03:17:23.367 INFO:teuthology.orchestra.run.smithi118.stdout: "max": "", 2024-03-24T03:17:23.367 INFO:teuthology.orchestra.run.smithi118.stdout: "enum_allowed": [], 2024-03-24T03:17:23.367 INFO:teuthology.orchestra.run.smithi118.stdout: "desc": "", 2024-03-24T03:17:23.367 INFO:teuthology.orchestra.run.smithi118.stdout: "long_desc": "", 2024-03-24T03:17:23.367 INFO:teuthology.orchestra.run.smithi118.stdout: "tags": [], 2024-03-24T03:17:23.368 INFO:teuthology.orchestra.run.smithi118.stdout: "see_also": [] 2024-03-24T03:17:23.368 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-24T03:17:23.368 INFO:teuthology.orchestra.run.smithi118.stdout: "ISSUE_TRACKER_API_KEY": { 2024-03-24T03:17:23.368 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "ISSUE_TRACKER_API_KEY", 2024-03-24T03:17:23.368 INFO:teuthology.orchestra.run.smithi118.stdout: "type": "str", 2024-03-24T03:17:23.368 INFO:teuthology.orchestra.run.smithi118.stdout: "level": "advanced", 2024-03-24T03:17:23.368 INFO:teuthology.orchestra.run.smithi118.stdout: "flags": 0, 2024-03-24T03:17:23.368 INFO:teuthology.orchestra.run.smithi118.stdout: "default_value": "", 2024-03-24T03:17:23.369 INFO:teuthology.orchestra.run.smithi118.stdout: "min": "", 2024-03-24T03:17:23.369 INFO:teuthology.orchestra.run.smithi118.stdout: "max": "", 2024-03-24T03:17:23.369 INFO:teuthology.orchestra.run.smithi118.stdout: "enum_allowed": [], 2024-03-24T03:17:23.369 INFO:teuthology.orchestra.run.smithi118.stdout: "desc": "", 2024-03-24T03:17:23.369 INFO:teuthology.orchestra.run.smithi118.stdout: "long_desc": "", 2024-03-24T03:17:23.369 INFO:teuthology.orchestra.run.smithi118.stdout: "tags": [], 2024-03-24T03:17:23.369 INFO:teuthology.orchestra.run.smithi118.stdout: "see_also": [] 2024-03-24T03:17:23.369 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-24T03:17:23.369 INFO:teuthology.orchestra.run.smithi118.stdout: "MULTICLUSTER_CONFIG": { 2024-03-24T03:17:23.370 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "MULTICLUSTER_CONFIG", 2024-03-24T03:17:23.370 INFO:teuthology.orchestra.run.smithi118.stdout: "type": "str", 2024-03-24T03:17:23.370 INFO:teuthology.orchestra.run.smithi118.stdout: "level": "advanced", 2024-03-24T03:17:23.370 INFO:teuthology.orchestra.run.smithi118.stdout: "flags": 0, 2024-03-24T03:17:23.370 INFO:teuthology.orchestra.run.smithi118.stdout: "default_value": "{}", 2024-03-24T03:17:23.370 INFO:teuthology.orchestra.run.smithi118.stdout: "min": "", 2024-03-24T03:17:23.370 INFO:teuthology.orchestra.run.smithi118.stdout: "max": "", 2024-03-24T03:17:23.371 INFO:teuthology.orchestra.run.smithi118.stdout: "enum_allowed": [], 2024-03-24T03:17:23.371 INFO:teuthology.orchestra.run.smithi118.stdout: "desc": "", 2024-03-24T03:17:23.371 INFO:teuthology.orchestra.run.smithi118.stdout: "long_desc": "", 2024-03-24T03:17:23.371 INFO:teuthology.orchestra.run.smithi118.stdout: "tags": [], 2024-03-24T03:17:23.371 INFO:teuthology.orchestra.run.smithi118.stdout: "see_also": [] 2024-03-24T03:17:23.371 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-24T03:17:23.371 INFO:teuthology.orchestra.run.smithi118.stdout: "PROMETHEUS_API_HOST": { 2024-03-24T03:17:23.371 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "PROMETHEUS_API_HOST", 2024-03-24T03:17:23.372 INFO:teuthology.orchestra.run.smithi118.stdout: "type": "str", 2024-03-24T03:17:23.372 INFO:teuthology.orchestra.run.smithi118.stdout: "level": "advanced", 2024-03-24T03:17:23.372 INFO:teuthology.orchestra.run.smithi118.stdout: "flags": 0, 2024-03-24T03:17:23.372 INFO:teuthology.orchestra.run.smithi118.stdout: "default_value": "", 2024-03-24T03:17:23.372 INFO:teuthology.orchestra.run.smithi118.stdout: "min": "", 2024-03-24T03:17:23.372 INFO:teuthology.orchestra.run.smithi118.stdout: "max": "", 2024-03-24T03:17:23.372 INFO:teuthology.orchestra.run.smithi118.stdout: "enum_allowed": [], 2024-03-24T03:17:23.372 INFO:teuthology.orchestra.run.smithi118.stdout: "desc": "", 2024-03-24T03:17:23.372 INFO:teuthology.orchestra.run.smithi118.stdout: "long_desc": "", 2024-03-24T03:17:23.373 INFO:teuthology.orchestra.run.smithi118.stdout: "tags": [], 2024-03-24T03:17:23.373 INFO:teuthology.orchestra.run.smithi118.stdout: "see_also": [] 2024-03-24T03:17:23.373 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-24T03:17:23.373 INFO:teuthology.orchestra.run.smithi118.stdout: "PROMETHEUS_API_SSL_VERIFY": { 2024-03-24T03:17:23.373 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "PROMETHEUS_API_SSL_VERIFY", 2024-03-24T03:17:23.373 INFO:teuthology.orchestra.run.smithi118.stdout: "type": "bool", 2024-03-24T03:17:23.373 INFO:teuthology.orchestra.run.smithi118.stdout: "level": "advanced", 2024-03-24T03:17:23.373 INFO:teuthology.orchestra.run.smithi118.stdout: "flags": 0, 2024-03-24T03:17:23.373 INFO:teuthology.orchestra.run.smithi118.stdout: "default_value": "True", 2024-03-24T03:17:23.374 INFO:teuthology.orchestra.run.smithi118.stdout: "min": "", 2024-03-24T03:17:23.374 INFO:teuthology.orchestra.run.smithi118.stdout: "max": "", 2024-03-24T03:17:23.374 INFO:teuthology.orchestra.run.smithi118.stdout: "enum_allowed": [], 2024-03-24T03:17:23.374 INFO:teuthology.orchestra.run.smithi118.stdout: "desc": "", 2024-03-24T03:17:23.374 INFO:teuthology.orchestra.run.smithi118.stdout: "long_desc": "", 2024-03-24T03:17:23.374 INFO:teuthology.orchestra.run.smithi118.stdout: "tags": [], 2024-03-24T03:17:23.374 INFO:teuthology.orchestra.run.smithi118.stdout: "see_also": [] 2024-03-24T03:17:23.375 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-24T03:17:23.375 INFO:teuthology.orchestra.run.smithi118.stdout: "PWD_POLICY_CHECK_COMPLEXITY_ENABLED": { 2024-03-24T03:17:23.375 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "PWD_POLICY_CHECK_COMPLEXITY_ENABLED", 2024-03-24T03:17:23.375 INFO:teuthology.orchestra.run.smithi118.stdout: "type": "bool", 2024-03-24T03:17:23.375 INFO:teuthology.orchestra.run.smithi118.stdout: "level": "advanced", 2024-03-24T03:17:23.375 INFO:teuthology.orchestra.run.smithi118.stdout: "flags": 0, 2024-03-24T03:17:23.375 INFO:teuthology.orchestra.run.smithi118.stdout: "default_value": "False", 2024-03-24T03:17:23.375 INFO:teuthology.orchestra.run.smithi118.stdout: "min": "", 2024-03-24T03:17:23.376 INFO:teuthology.orchestra.run.smithi118.stdout: "max": "", 2024-03-24T03:17:23.376 INFO:teuthology.orchestra.run.smithi118.stdout: "enum_allowed": [], 2024-03-24T03:17:23.376 INFO:teuthology.orchestra.run.smithi118.stdout: "desc": "", 2024-03-24T03:17:23.376 INFO:teuthology.orchestra.run.smithi118.stdout: "long_desc": "", 2024-03-24T03:17:23.376 INFO:teuthology.orchestra.run.smithi118.stdout: "tags": [], 2024-03-24T03:17:23.376 INFO:teuthology.orchestra.run.smithi118.stdout: "see_also": [] 2024-03-24T03:17:23.376 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-24T03:17:23.376 INFO:teuthology.orchestra.run.smithi118.stdout: "PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED": { 2024-03-24T03:17:23.376 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED", 2024-03-24T03:17:23.377 INFO:teuthology.orchestra.run.smithi118.stdout: "type": "bool", 2024-03-24T03:17:23.377 INFO:teuthology.orchestra.run.smithi118.stdout: "level": "advanced", 2024-03-24T03:17:23.377 INFO:teuthology.orchestra.run.smithi118.stdout: "flags": 0, 2024-03-24T03:17:23.377 INFO:teuthology.orchestra.run.smithi118.stdout: "default_value": "False", 2024-03-24T03:17:23.377 INFO:teuthology.orchestra.run.smithi118.stdout: "min": "", 2024-03-24T03:17:23.377 INFO:teuthology.orchestra.run.smithi118.stdout: "max": "", 2024-03-24T03:17:23.378 INFO:teuthology.orchestra.run.smithi118.stdout: "enum_allowed": [], 2024-03-24T03:17:23.378 INFO:teuthology.orchestra.run.smithi118.stdout: "desc": "", 2024-03-24T03:17:23.378 INFO:teuthology.orchestra.run.smithi118.stdout: "long_desc": "", 2024-03-24T03:17:23.378 INFO:teuthology.orchestra.run.smithi118.stdout: "tags": [], 2024-03-24T03:17:23.378 INFO:teuthology.orchestra.run.smithi118.stdout: "see_also": [] 2024-03-24T03:17:23.378 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-24T03:17:23.378 INFO:teuthology.orchestra.run.smithi118.stdout: "PWD_POLICY_CHECK_LENGTH_ENABLED": { 2024-03-24T03:17:23.378 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "PWD_POLICY_CHECK_LENGTH_ENABLED", 2024-03-24T03:17:23.379 INFO:teuthology.orchestra.run.smithi118.stdout: "type": "bool", 2024-03-24T03:17:23.379 INFO:teuthology.orchestra.run.smithi118.stdout: "level": "advanced", 2024-03-24T03:17:23.379 INFO:teuthology.orchestra.run.smithi118.stdout: "flags": 0, 2024-03-24T03:17:23.379 INFO:teuthology.orchestra.run.smithi118.stdout: "default_value": "True", 2024-03-24T03:17:23.379 INFO:teuthology.orchestra.run.smithi118.stdout: "min": "", 2024-03-24T03:17:23.379 INFO:teuthology.orchestra.run.smithi118.stdout: "max": "", 2024-03-24T03:17:23.379 INFO:teuthology.orchestra.run.smithi118.stdout: "enum_allowed": [], 2024-03-24T03:17:23.379 INFO:teuthology.orchestra.run.smithi118.stdout: "desc": "", 2024-03-24T03:17:23.379 INFO:teuthology.orchestra.run.smithi118.stdout: "long_desc": "", 2024-03-24T03:17:23.380 INFO:teuthology.orchestra.run.smithi118.stdout: "tags": [], 2024-03-24T03:17:23.380 INFO:teuthology.orchestra.run.smithi118.stdout: "see_also": [] 2024-03-24T03:17:23.380 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-24T03:17:23.380 INFO:teuthology.orchestra.run.smithi118.stdout: "PWD_POLICY_CHECK_OLDPWD_ENABLED": { 2024-03-24T03:17:23.380 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "PWD_POLICY_CHECK_OLDPWD_ENABLED", 2024-03-24T03:17:23.380 INFO:teuthology.orchestra.run.smithi118.stdout: "type": "bool", 2024-03-24T03:17:23.380 INFO:teuthology.orchestra.run.smithi118.stdout: "level": "advanced", 2024-03-24T03:17:23.380 INFO:teuthology.orchestra.run.smithi118.stdout: "flags": 0, 2024-03-24T03:17:23.380 INFO:teuthology.orchestra.run.smithi118.stdout: "default_value": "True", 2024-03-24T03:17:23.381 INFO:teuthology.orchestra.run.smithi118.stdout: "min": "", 2024-03-24T03:17:23.381 INFO:teuthology.orchestra.run.smithi118.stdout: "max": "", 2024-03-24T03:17:23.381 INFO:teuthology.orchestra.run.smithi118.stdout: "enum_allowed": [], 2024-03-24T03:17:23.381 INFO:teuthology.orchestra.run.smithi118.stdout: "desc": "", 2024-03-24T03:17:23.381 INFO:teuthology.orchestra.run.smithi118.stdout: "long_desc": "", 2024-03-24T03:17:23.381 INFO:teuthology.orchestra.run.smithi118.stdout: "tags": [], 2024-03-24T03:17:23.381 INFO:teuthology.orchestra.run.smithi118.stdout: "see_also": [] 2024-03-24T03:17:23.381 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-24T03:17:23.381 INFO:teuthology.orchestra.run.smithi118.stdout: "PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED": { 2024-03-24T03:17:23.382 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED", 2024-03-24T03:17:23.382 INFO:teuthology.orchestra.run.smithi118.stdout: "type": "bool", 2024-03-24T03:17:23.382 INFO:teuthology.orchestra.run.smithi118.stdout: "level": "advanced", 2024-03-24T03:17:23.382 INFO:teuthology.orchestra.run.smithi118.stdout: "flags": 0, 2024-03-24T03:17:23.382 INFO:teuthology.orchestra.run.smithi118.stdout: "default_value": "False", 2024-03-24T03:17:23.382 INFO:teuthology.orchestra.run.smithi118.stdout: "min": "", 2024-03-24T03:17:23.382 INFO:teuthology.orchestra.run.smithi118.stdout: "max": "", 2024-03-24T03:17:23.382 INFO:teuthology.orchestra.run.smithi118.stdout: "enum_allowed": [], 2024-03-24T03:17:23.383 INFO:teuthology.orchestra.run.smithi118.stdout: "desc": "", 2024-03-24T03:17:23.383 INFO:teuthology.orchestra.run.smithi118.stdout: "long_desc": "", 2024-03-24T03:17:23.383 INFO:teuthology.orchestra.run.smithi118.stdout: "tags": [], 2024-03-24T03:17:23.383 INFO:teuthology.orchestra.run.smithi118.stdout: "see_also": [] 2024-03-24T03:17:23.383 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-24T03:17:23.383 INFO:teuthology.orchestra.run.smithi118.stdout: "PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED": { 2024-03-24T03:17:23.383 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED", 2024-03-24T03:17:23.383 INFO:teuthology.orchestra.run.smithi118.stdout: "type": "bool", 2024-03-24T03:17:23.383 INFO:teuthology.orchestra.run.smithi118.stdout: "level": "advanced", 2024-03-24T03:17:23.384 INFO:teuthology.orchestra.run.smithi118.stdout: "flags": 0, 2024-03-24T03:17:23.384 INFO:teuthology.orchestra.run.smithi118.stdout: "default_value": "False", 2024-03-24T03:17:23.384 INFO:teuthology.orchestra.run.smithi118.stdout: "min": "", 2024-03-24T03:17:23.384 INFO:teuthology.orchestra.run.smithi118.stdout: "max": "", 2024-03-24T03:17:23.384 INFO:teuthology.orchestra.run.smithi118.stdout: "enum_allowed": [], 2024-03-24T03:17:23.384 INFO:teuthology.orchestra.run.smithi118.stdout: "desc": "", 2024-03-24T03:17:23.384 INFO:teuthology.orchestra.run.smithi118.stdout: "long_desc": "", 2024-03-24T03:17:23.384 INFO:teuthology.orchestra.run.smithi118.stdout: "tags": [], 2024-03-24T03:17:23.384 INFO:teuthology.orchestra.run.smithi118.stdout: "see_also": [] 2024-03-24T03:17:23.385 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-24T03:17:23.385 INFO:teuthology.orchestra.run.smithi118.stdout: "PWD_POLICY_CHECK_USERNAME_ENABLED": { 2024-03-24T03:17:23.385 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "PWD_POLICY_CHECK_USERNAME_ENABLED", 2024-03-24T03:17:23.385 INFO:teuthology.orchestra.run.smithi118.stdout: "type": "bool", 2024-03-24T03:17:23.385 INFO:teuthology.orchestra.run.smithi118.stdout: "level": "advanced", 2024-03-24T03:17:23.385 INFO:teuthology.orchestra.run.smithi118.stdout: "flags": 0, 2024-03-24T03:17:23.385 INFO:teuthology.orchestra.run.smithi118.stdout: "default_value": "False", 2024-03-24T03:17:23.385 INFO:teuthology.orchestra.run.smithi118.stdout: "min": "", 2024-03-24T03:17:23.386 INFO:teuthology.orchestra.run.smithi118.stdout: "max": "", 2024-03-24T03:17:23.386 INFO:teuthology.orchestra.run.smithi118.stdout: "enum_allowed": [], 2024-03-24T03:17:23.386 INFO:teuthology.orchestra.run.smithi118.stdout: "desc": "", 2024-03-24T03:17:23.386 INFO:teuthology.orchestra.run.smithi118.stdout: "long_desc": "", 2024-03-24T03:17:23.386 INFO:teuthology.orchestra.run.smithi118.stdout: "tags": [], 2024-03-24T03:17:23.386 INFO:teuthology.orchestra.run.smithi118.stdout: "see_also": [] 2024-03-24T03:17:23.386 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-24T03:17:23.386 INFO:teuthology.orchestra.run.smithi118.stdout: "PWD_POLICY_ENABLED": { 2024-03-24T03:17:23.387 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "PWD_POLICY_ENABLED", 2024-03-24T03:17:23.387 INFO:teuthology.orchestra.run.smithi118.stdout: "type": "bool", 2024-03-24T03:17:23.387 INFO:teuthology.orchestra.run.smithi118.stdout: "level": "advanced", 2024-03-24T03:17:23.387 INFO:teuthology.orchestra.run.smithi118.stdout: "flags": 0, 2024-03-24T03:17:23.387 INFO:teuthology.orchestra.run.smithi118.stdout: "default_value": "True", 2024-03-24T03:17:23.387 INFO:teuthology.orchestra.run.smithi118.stdout: "min": "", 2024-03-24T03:17:23.387 INFO:teuthology.orchestra.run.smithi118.stdout: "max": "", 2024-03-24T03:17:23.387 INFO:teuthology.orchestra.run.smithi118.stdout: "enum_allowed": [], 2024-03-24T03:17:23.387 INFO:teuthology.orchestra.run.smithi118.stdout: "desc": "", 2024-03-24T03:17:23.388 INFO:teuthology.orchestra.run.smithi118.stdout: "long_desc": "", 2024-03-24T03:17:23.388 INFO:teuthology.orchestra.run.smithi118.stdout: "tags": [], 2024-03-24T03:17:23.388 INFO:teuthology.orchestra.run.smithi118.stdout: "see_also": [] 2024-03-24T03:17:23.388 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-24T03:17:23.388 INFO:teuthology.orchestra.run.smithi118.stdout: "PWD_POLICY_EXCLUSION_LIST": { 2024-03-24T03:17:23.388 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "PWD_POLICY_EXCLUSION_LIST", 2024-03-24T03:17:23.388 INFO:teuthology.orchestra.run.smithi118.stdout: "type": "str", 2024-03-24T03:17:23.388 INFO:teuthology.orchestra.run.smithi118.stdout: "level": "advanced", 2024-03-24T03:17:23.389 INFO:teuthology.orchestra.run.smithi118.stdout: "flags": 0, 2024-03-24T03:17:23.389 INFO:teuthology.orchestra.run.smithi118.stdout: "default_value": "osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps", 2024-03-24T03:17:23.389 INFO:teuthology.orchestra.run.smithi118.stdout: "min": "", 2024-03-24T03:17:23.389 INFO:teuthology.orchestra.run.smithi118.stdout: "max": "", 2024-03-24T03:17:23.389 INFO:teuthology.orchestra.run.smithi118.stdout: "enum_allowed": [], 2024-03-24T03:17:23.389 INFO:teuthology.orchestra.run.smithi118.stdout: "desc": "", 2024-03-24T03:17:23.389 INFO:teuthology.orchestra.run.smithi118.stdout: "long_desc": "", 2024-03-24T03:17:23.389 INFO:teuthology.orchestra.run.smithi118.stdout: "tags": [], 2024-03-24T03:17:23.389 INFO:teuthology.orchestra.run.smithi118.stdout: "see_also": [] 2024-03-24T03:17:23.390 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-24T03:17:23.390 INFO:teuthology.orchestra.run.smithi118.stdout: "PWD_POLICY_MIN_COMPLEXITY": { 2024-03-24T03:17:23.390 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "PWD_POLICY_MIN_COMPLEXITY", 2024-03-24T03:17:23.390 INFO:teuthology.orchestra.run.smithi118.stdout: "type": "int", 2024-03-24T03:17:23.390 INFO:teuthology.orchestra.run.smithi118.stdout: "level": "advanced", 2024-03-24T03:17:23.390 INFO:teuthology.orchestra.run.smithi118.stdout: "flags": 0, 2024-03-24T03:17:23.390 INFO:teuthology.orchestra.run.smithi118.stdout: "default_value": "10", 2024-03-24T03:17:23.391 INFO:teuthology.orchestra.run.smithi118.stdout: "min": "", 2024-03-24T03:17:23.391 INFO:teuthology.orchestra.run.smithi118.stdout: "max": "", 2024-03-24T03:17:23.391 INFO:teuthology.orchestra.run.smithi118.stdout: "enum_allowed": [], 2024-03-24T03:17:23.391 INFO:teuthology.orchestra.run.smithi118.stdout: "desc": "", 2024-03-24T03:17:23.391 INFO:teuthology.orchestra.run.smithi118.stdout: "long_desc": "", 2024-03-24T03:17:23.391 INFO:teuthology.orchestra.run.smithi118.stdout: "tags": [], 2024-03-24T03:17:23.391 INFO:teuthology.orchestra.run.smithi118.stdout: "see_also": [] 2024-03-24T03:17:23.392 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-24T03:17:23.392 INFO:teuthology.orchestra.run.smithi118.stdout: "PWD_POLICY_MIN_LENGTH": { 2024-03-24T03:17:23.392 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "PWD_POLICY_MIN_LENGTH", 2024-03-24T03:17:23.392 INFO:teuthology.orchestra.run.smithi118.stdout: "type": "int", 2024-03-24T03:17:23.392 INFO:teuthology.orchestra.run.smithi118.stdout: "level": "advanced", 2024-03-24T03:17:23.392 INFO:teuthology.orchestra.run.smithi118.stdout: "flags": 0, 2024-03-24T03:17:23.392 INFO:teuthology.orchestra.run.smithi118.stdout: "default_value": "8", 2024-03-24T03:17:23.392 INFO:teuthology.orchestra.run.smithi118.stdout: "min": "", 2024-03-24T03:17:23.393 INFO:teuthology.orchestra.run.smithi118.stdout: "max": "", 2024-03-24T03:17:23.393 INFO:teuthology.orchestra.run.smithi118.stdout: "enum_allowed": [], 2024-03-24T03:17:23.393 INFO:teuthology.orchestra.run.smithi118.stdout: "desc": "", 2024-03-24T03:17:23.393 INFO:teuthology.orchestra.run.smithi118.stdout: "long_desc": "", 2024-03-24T03:17:23.393 INFO:teuthology.orchestra.run.smithi118.stdout: "tags": [], 2024-03-24T03:17:23.393 INFO:teuthology.orchestra.run.smithi118.stdout: "see_also": [] 2024-03-24T03:17:23.393 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-24T03:17:23.393 INFO:teuthology.orchestra.run.smithi118.stdout: "REST_REQUESTS_TIMEOUT": { 2024-03-24T03:17:23.394 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "REST_REQUESTS_TIMEOUT", 2024-03-24T03:17:23.394 INFO:teuthology.orchestra.run.smithi118.stdout: "type": "int", 2024-03-24T03:17:23.394 INFO:teuthology.orchestra.run.smithi118.stdout: "level": "advanced", 2024-03-24T03:17:23.394 INFO:teuthology.orchestra.run.smithi118.stdout: "flags": 0, 2024-03-24T03:17:23.394 INFO:teuthology.orchestra.run.smithi118.stdout: "default_value": "45", 2024-03-24T03:17:23.394 INFO:teuthology.orchestra.run.smithi118.stdout: "min": "", 2024-03-24T03:17:23.394 INFO:teuthology.orchestra.run.smithi118.stdout: "max": "", 2024-03-24T03:17:23.394 INFO:teuthology.orchestra.run.smithi118.stdout: "enum_allowed": [], 2024-03-24T03:17:23.395 INFO:teuthology.orchestra.run.smithi118.stdout: "desc": "", 2024-03-24T03:17:23.395 INFO:teuthology.orchestra.run.smithi118.stdout: "long_desc": "", 2024-03-24T03:17:23.395 INFO:teuthology.orchestra.run.smithi118.stdout: "tags": [], 2024-03-24T03:17:23.395 INFO:teuthology.orchestra.run.smithi118.stdout: "see_also": [] 2024-03-24T03:17:23.395 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-24T03:17:23.395 INFO:teuthology.orchestra.run.smithi118.stdout: "RGW_API_ACCESS_KEY": { 2024-03-24T03:17:23.395 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "RGW_API_ACCESS_KEY", 2024-03-24T03:17:23.395 INFO:teuthology.orchestra.run.smithi118.stdout: "type": "str", 2024-03-24T03:17:23.395 INFO:teuthology.orchestra.run.smithi118.stdout: "level": "advanced", 2024-03-24T03:17:23.396 INFO:teuthology.orchestra.run.smithi118.stdout: "flags": 0, 2024-03-24T03:17:23.396 INFO:teuthology.orchestra.run.smithi118.stdout: "default_value": "", 2024-03-24T03:17:23.396 INFO:teuthology.orchestra.run.smithi118.stdout: "min": "", 2024-03-24T03:17:23.396 INFO:teuthology.orchestra.run.smithi118.stdout: "max": "", 2024-03-24T03:17:23.396 INFO:teuthology.orchestra.run.smithi118.stdout: "enum_allowed": [], 2024-03-24T03:17:23.396 INFO:teuthology.orchestra.run.smithi118.stdout: "desc": "", 2024-03-24T03:17:23.396 INFO:teuthology.orchestra.run.smithi118.stdout: "long_desc": "", 2024-03-24T03:17:23.396 INFO:teuthology.orchestra.run.smithi118.stdout: "tags": [], 2024-03-24T03:17:23.396 INFO:teuthology.orchestra.run.smithi118.stdout: "see_also": [] 2024-03-24T03:17:23.397 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-24T03:17:23.397 INFO:teuthology.orchestra.run.smithi118.stdout: "RGW_API_ADMIN_RESOURCE": { 2024-03-24T03:17:23.397 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "RGW_API_ADMIN_RESOURCE", 2024-03-24T03:17:23.397 INFO:teuthology.orchestra.run.smithi118.stdout: "type": "str", 2024-03-24T03:17:23.397 INFO:teuthology.orchestra.run.smithi118.stdout: "level": "advanced", 2024-03-24T03:17:23.397 INFO:teuthology.orchestra.run.smithi118.stdout: "flags": 0, 2024-03-24T03:17:23.397 INFO:teuthology.orchestra.run.smithi118.stdout: "default_value": "admin", 2024-03-24T03:17:23.397 INFO:teuthology.orchestra.run.smithi118.stdout: "min": "", 2024-03-24T03:17:23.398 INFO:teuthology.orchestra.run.smithi118.stdout: "max": "", 2024-03-24T03:17:23.398 INFO:teuthology.orchestra.run.smithi118.stdout: "enum_allowed": [], 2024-03-24T03:17:23.398 INFO:teuthology.orchestra.run.smithi118.stdout: "desc": "", 2024-03-24T03:17:23.398 INFO:teuthology.orchestra.run.smithi118.stdout: "long_desc": "", 2024-03-24T03:17:23.398 INFO:teuthology.orchestra.run.smithi118.stdout: "tags": [], 2024-03-24T03:17:23.398 INFO:teuthology.orchestra.run.smithi118.stdout: "see_also": [] 2024-03-24T03:17:23.398 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-24T03:17:23.398 INFO:teuthology.orchestra.run.smithi118.stdout: "RGW_API_SECRET_KEY": { 2024-03-24T03:17:23.399 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "RGW_API_SECRET_KEY", 2024-03-24T03:17:23.399 INFO:teuthology.orchestra.run.smithi118.stdout: "type": "str", 2024-03-24T03:17:23.399 INFO:teuthology.orchestra.run.smithi118.stdout: "level": "advanced", 2024-03-24T03:17:23.399 INFO:teuthology.orchestra.run.smithi118.stdout: "flags": 0, 2024-03-24T03:17:23.399 INFO:teuthology.orchestra.run.smithi118.stdout: "default_value": "", 2024-03-24T03:17:23.399 INFO:teuthology.orchestra.run.smithi118.stdout: "min": "", 2024-03-24T03:17:23.399 INFO:teuthology.orchestra.run.smithi118.stdout: "max": "", 2024-03-24T03:17:23.399 INFO:teuthology.orchestra.run.smithi118.stdout: "enum_allowed": [], 2024-03-24T03:17:23.399 INFO:teuthology.orchestra.run.smithi118.stdout: "desc": "", 2024-03-24T03:17:23.400 INFO:teuthology.orchestra.run.smithi118.stdout: "long_desc": "", 2024-03-24T03:17:23.400 INFO:teuthology.orchestra.run.smithi118.stdout: "tags": [], 2024-03-24T03:17:23.400 INFO:teuthology.orchestra.run.smithi118.stdout: "see_also": [] 2024-03-24T03:17:23.400 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-24T03:17:23.400 INFO:teuthology.orchestra.run.smithi118.stdout: "RGW_API_SSL_VERIFY": { 2024-03-24T03:17:23.400 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "RGW_API_SSL_VERIFY", 2024-03-24T03:17:23.400 INFO:teuthology.orchestra.run.smithi118.stdout: "type": "bool", 2024-03-24T03:17:23.400 INFO:teuthology.orchestra.run.smithi118.stdout: "level": "advanced", 2024-03-24T03:17:23.401 INFO:teuthology.orchestra.run.smithi118.stdout: "flags": 0, 2024-03-24T03:17:23.401 INFO:teuthology.orchestra.run.smithi118.stdout: "default_value": "True", 2024-03-24T03:17:23.401 INFO:teuthology.orchestra.run.smithi118.stdout: "min": "", 2024-03-24T03:17:23.401 INFO:teuthology.orchestra.run.smithi118.stdout: "max": "", 2024-03-24T03:17:23.401 INFO:teuthology.orchestra.run.smithi118.stdout: "enum_allowed": [], 2024-03-24T03:17:23.401 INFO:teuthology.orchestra.run.smithi118.stdout: "desc": "", 2024-03-24T03:17:23.401 INFO:teuthology.orchestra.run.smithi118.stdout: "long_desc": "", 2024-03-24T03:17:23.401 INFO:teuthology.orchestra.run.smithi118.stdout: "tags": [], 2024-03-24T03:17:23.401 INFO:teuthology.orchestra.run.smithi118.stdout: "see_also": [] 2024-03-24T03:17:23.402 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-24T03:17:23.402 INFO:teuthology.orchestra.run.smithi118.stdout: "UNSAFE_TLS_v1_2": { 2024-03-24T03:17:23.402 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "UNSAFE_TLS_v1_2", 2024-03-24T03:17:23.402 INFO:teuthology.orchestra.run.smithi118.stdout: "type": "bool", 2024-03-24T03:17:23.402 INFO:teuthology.orchestra.run.smithi118.stdout: "level": "advanced", 2024-03-24T03:17:23.402 INFO:teuthology.orchestra.run.smithi118.stdout: "flags": 0, 2024-03-24T03:17:23.402 INFO:teuthology.orchestra.run.smithi118.stdout: "default_value": "False", 2024-03-24T03:17:23.402 INFO:teuthology.orchestra.run.smithi118.stdout: "min": "", 2024-03-24T03:17:23.403 INFO:teuthology.orchestra.run.smithi118.stdout: "max": "", 2024-03-24T03:17:23.403 INFO:teuthology.orchestra.run.smithi118.stdout: "enum_allowed": [], 2024-03-24T03:17:23.403 INFO:teuthology.orchestra.run.smithi118.stdout: "desc": "", 2024-03-24T03:17:23.403 INFO:teuthology.orchestra.run.smithi118.stdout: "long_desc": "", 2024-03-24T03:17:23.403 INFO:teuthology.orchestra.run.smithi118.stdout: "tags": [], 2024-03-24T03:17:23.403 INFO:teuthology.orchestra.run.smithi118.stdout: "see_also": [] 2024-03-24T03:17:23.403 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-24T03:17:23.403 INFO:teuthology.orchestra.run.smithi118.stdout: "USER_PWD_EXPIRATION_SPAN": { 2024-03-24T03:17:23.404 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "USER_PWD_EXPIRATION_SPAN", 2024-03-24T03:17:23.404 INFO:teuthology.orchestra.run.smithi118.stdout: "type": "int", 2024-03-24T03:17:23.404 INFO:teuthology.orchestra.run.smithi118.stdout: "level": "advanced", 2024-03-24T03:17:23.404 INFO:teuthology.orchestra.run.smithi118.stdout: "flags": 0, 2024-03-24T03:17:23.404 INFO:teuthology.orchestra.run.smithi118.stdout: "default_value": "0", 2024-03-24T03:17:23.404 INFO:teuthology.orchestra.run.smithi118.stdout: "min": "", 2024-03-24T03:17:23.404 INFO:teuthology.orchestra.run.smithi118.stdout: "max": "", 2024-03-24T03:17:23.404 INFO:teuthology.orchestra.run.smithi118.stdout: "enum_allowed": [], 2024-03-24T03:17:23.405 INFO:teuthology.orchestra.run.smithi118.stdout: "desc": "", 2024-03-24T03:17:23.405 INFO:teuthology.orchestra.run.smithi118.stdout: "long_desc": "", 2024-03-24T03:17:23.405 INFO:teuthology.orchestra.run.smithi118.stdout: "tags": [], 2024-03-24T03:17:23.405 INFO:teuthology.orchestra.run.smithi118.stdout: "see_also": [] 2024-03-24T03:17:23.405 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-24T03:17:23.405 INFO:teuthology.orchestra.run.smithi118.stdout: "USER_PWD_EXPIRATION_WARNING_1": { 2024-03-24T03:17:23.405 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "USER_PWD_EXPIRATION_WARNING_1", 2024-03-24T03:17:23.405 INFO:teuthology.orchestra.run.smithi118.stdout: "type": "int", 2024-03-24T03:17:23.406 INFO:teuthology.orchestra.run.smithi118.stdout: "level": "advanced", 2024-03-24T03:17:23.406 INFO:teuthology.orchestra.run.smithi118.stdout: "flags": 0, 2024-03-24T03:17:23.406 INFO:teuthology.orchestra.run.smithi118.stdout: "default_value": "10", 2024-03-24T03:17:23.406 INFO:teuthology.orchestra.run.smithi118.stdout: "min": "", 2024-03-24T03:17:23.406 INFO:teuthology.orchestra.run.smithi118.stdout: "max": "", 2024-03-24T03:17:23.406 INFO:teuthology.orchestra.run.smithi118.stdout: "enum_allowed": [], 2024-03-24T03:17:23.406 INFO:teuthology.orchestra.run.smithi118.stdout: "desc": "", 2024-03-24T03:17:23.406 INFO:teuthology.orchestra.run.smithi118.stdout: "long_desc": "", 2024-03-24T03:17:23.406 INFO:teuthology.orchestra.run.smithi118.stdout: "tags": [], 2024-03-24T03:17:23.407 INFO:teuthology.orchestra.run.smithi118.stdout: "see_also": [] 2024-03-24T03:17:23.407 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-24T03:17:23.407 INFO:teuthology.orchestra.run.smithi118.stdout: "USER_PWD_EXPIRATION_WARNING_2": { 2024-03-24T03:17:23.407 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "USER_PWD_EXPIRATION_WARNING_2", 2024-03-24T03:17:23.407 INFO:teuthology.orchestra.run.smithi118.stdout: "type": "int", 2024-03-24T03:17:23.407 INFO:teuthology.orchestra.run.smithi118.stdout: "level": "advanced", 2024-03-24T03:17:23.407 INFO:teuthology.orchestra.run.smithi118.stdout: "flags": 0, 2024-03-24T03:17:23.407 INFO:teuthology.orchestra.run.smithi118.stdout: "default_value": "5", 2024-03-24T03:17:23.407 INFO:teuthology.orchestra.run.smithi118.stdout: "min": "", 2024-03-24T03:17:23.408 INFO:teuthology.orchestra.run.smithi118.stdout: "max": "", 2024-03-24T03:17:23.408 INFO:teuthology.orchestra.run.smithi118.stdout: "enum_allowed": [], 2024-03-24T03:17:23.408 INFO:teuthology.orchestra.run.smithi118.stdout: "desc": "", 2024-03-24T03:17:23.408 INFO:teuthology.orchestra.run.smithi118.stdout: "long_desc": "", 2024-03-24T03:17:23.408 INFO:teuthology.orchestra.run.smithi118.stdout: "tags": [], 2024-03-24T03:17:23.408 INFO:teuthology.orchestra.run.smithi118.stdout: "see_also": [] 2024-03-24T03:17:23.408 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-24T03:17:23.408 INFO:teuthology.orchestra.run.smithi118.stdout: "cross_origin_url": { 2024-03-24T03:17:23.408 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "cross_origin_url", 2024-03-24T03:17:23.408 INFO:teuthology.orchestra.run.smithi118.stdout: "type": "str", 2024-03-24T03:17:23.409 INFO:teuthology.orchestra.run.smithi118.stdout: "level": "advanced", 2024-03-24T03:17:23.409 INFO:teuthology.orchestra.run.smithi118.stdout: "flags": 0, 2024-03-24T03:17:23.409 INFO:teuthology.orchestra.run.smithi118.stdout: "default_value": "", 2024-03-24T03:17:23.409 INFO:teuthology.orchestra.run.smithi118.stdout: "min": "", 2024-03-24T03:17:23.409 INFO:teuthology.orchestra.run.smithi118.stdout: "max": "", 2024-03-24T03:17:23.409 INFO:teuthology.orchestra.run.smithi118.stdout: "enum_allowed": [], 2024-03-24T03:17:23.409 INFO:teuthology.orchestra.run.smithi118.stdout: "desc": "", 2024-03-24T03:17:23.409 INFO:teuthology.orchestra.run.smithi118.stdout: "long_desc": "", 2024-03-24T03:17:23.409 INFO:teuthology.orchestra.run.smithi118.stdout: "tags": [], 2024-03-24T03:17:23.410 INFO:teuthology.orchestra.run.smithi118.stdout: "see_also": [] 2024-03-24T03:17:23.410 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-24T03:17:23.410 INFO:teuthology.orchestra.run.smithi118.stdout: "crt_file": { 2024-03-24T03:17:23.410 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "crt_file", 2024-03-24T03:17:23.410 INFO:teuthology.orchestra.run.smithi118.stdout: "type": "str", 2024-03-24T03:17:23.410 INFO:teuthology.orchestra.run.smithi118.stdout: "level": "advanced", 2024-03-24T03:17:23.410 INFO:teuthology.orchestra.run.smithi118.stdout: "flags": 0, 2024-03-24T03:17:23.410 INFO:teuthology.orchestra.run.smithi118.stdout: "default_value": "", 2024-03-24T03:17:23.411 INFO:teuthology.orchestra.run.smithi118.stdout: "min": "", 2024-03-24T03:17:23.411 INFO:teuthology.orchestra.run.smithi118.stdout: "max": "", 2024-03-24T03:17:23.411 INFO:teuthology.orchestra.run.smithi118.stdout: "enum_allowed": [], 2024-03-24T03:17:23.411 INFO:teuthology.orchestra.run.smithi118.stdout: "desc": "", 2024-03-24T03:17:23.411 INFO:teuthology.orchestra.run.smithi118.stdout: "long_desc": "", 2024-03-24T03:17:23.411 INFO:teuthology.orchestra.run.smithi118.stdout: "tags": [], 2024-03-24T03:17:23.411 INFO:teuthology.orchestra.run.smithi118.stdout: "see_also": [] 2024-03-24T03:17:23.411 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-24T03:17:23.412 INFO:teuthology.orchestra.run.smithi118.stdout: "debug": { 2024-03-24T03:17:23.412 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "debug", 2024-03-24T03:17:23.412 INFO:teuthology.orchestra.run.smithi118.stdout: "type": "bool", 2024-03-24T03:17:23.412 INFO:teuthology.orchestra.run.smithi118.stdout: "level": "advanced", 2024-03-24T03:17:23.412 INFO:teuthology.orchestra.run.smithi118.stdout: "flags": 0, 2024-03-24T03:17:23.412 INFO:teuthology.orchestra.run.smithi118.stdout: "default_value": "False", 2024-03-24T03:17:23.412 INFO:teuthology.orchestra.run.smithi118.stdout: "min": "", 2024-03-24T03:17:23.412 INFO:teuthology.orchestra.run.smithi118.stdout: "max": "", 2024-03-24T03:17:23.413 INFO:teuthology.orchestra.run.smithi118.stdout: "enum_allowed": [], 2024-03-24T03:17:23.413 INFO:teuthology.orchestra.run.smithi118.stdout: "desc": "Enable/disable debug options", 2024-03-24T03:17:23.413 INFO:teuthology.orchestra.run.smithi118.stdout: "long_desc": "", 2024-03-24T03:17:23.413 INFO:teuthology.orchestra.run.smithi118.stdout: "tags": [], 2024-03-24T03:17:23.413 INFO:teuthology.orchestra.run.smithi118.stdout: "see_also": [] 2024-03-24T03:17:23.413 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-24T03:17:23.413 INFO:teuthology.orchestra.run.smithi118.stdout: "jwt_token_ttl": { 2024-03-24T03:17:23.413 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "jwt_token_ttl", 2024-03-24T03:17:23.414 INFO:teuthology.orchestra.run.smithi118.stdout: "type": "int", 2024-03-24T03:17:23.414 INFO:teuthology.orchestra.run.smithi118.stdout: "level": "advanced", 2024-03-24T03:17:23.414 INFO:teuthology.orchestra.run.smithi118.stdout: "flags": 0, 2024-03-24T03:17:23.414 INFO:teuthology.orchestra.run.smithi118.stdout: "default_value": "28800", 2024-03-24T03:17:23.414 INFO:teuthology.orchestra.run.smithi118.stdout: "min": "", 2024-03-24T03:17:23.414 INFO:teuthology.orchestra.run.smithi118.stdout: "max": "", 2024-03-24T03:17:23.414 INFO:teuthology.orchestra.run.smithi118.stdout: "enum_allowed": [], 2024-03-24T03:17:23.414 INFO:teuthology.orchestra.run.smithi118.stdout: "desc": "", 2024-03-24T03:17:23.415 INFO:teuthology.orchestra.run.smithi118.stdout: "long_desc": "", 2024-03-24T03:17:23.415 INFO:teuthology.orchestra.run.smithi118.stdout: "tags": [], 2024-03-24T03:17:23.415 INFO:teuthology.orchestra.run.smithi118.stdout: "see_also": [] 2024-03-24T03:17:23.415 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-24T03:17:23.415 INFO:teuthology.orchestra.run.smithi118.stdout: "key_file": { 2024-03-24T03:17:23.415 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "key_file", 2024-03-24T03:17:23.415 INFO:teuthology.orchestra.run.smithi118.stdout: "type": "str", 2024-03-24T03:17:23.415 INFO:teuthology.orchestra.run.smithi118.stdout: "level": "advanced", 2024-03-24T03:17:23.416 INFO:teuthology.orchestra.run.smithi118.stdout: "flags": 0, 2024-03-24T03:17:23.416 INFO:teuthology.orchestra.run.smithi118.stdout: "default_value": "", 2024-03-24T03:17:23.416 INFO:teuthology.orchestra.run.smithi118.stdout: "min": "", 2024-03-24T03:17:23.416 INFO:teuthology.orchestra.run.smithi118.stdout: "max": "", 2024-03-24T03:17:23.416 INFO:teuthology.orchestra.run.smithi118.stdout: "enum_allowed": [], 2024-03-24T03:17:23.416 INFO:teuthology.orchestra.run.smithi118.stdout: "desc": "", 2024-03-24T03:17:23.416 INFO:teuthology.orchestra.run.smithi118.stdout: "long_desc": "", 2024-03-24T03:17:23.417 INFO:teuthology.orchestra.run.smithi118.stdout: "tags": [], 2024-03-24T03:17:23.417 INFO:teuthology.orchestra.run.smithi118.stdout: "see_also": [] 2024-03-24T03:17:23.417 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-24T03:17:23.417 INFO:teuthology.orchestra.run.smithi118.stdout: "log_level": { 2024-03-24T03:17:23.417 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "log_level", 2024-03-24T03:17:23.417 INFO:teuthology.orchestra.run.smithi118.stdout: "type": "str", 2024-03-24T03:17:23.417 INFO:teuthology.orchestra.run.smithi118.stdout: "level": "advanced", 2024-03-24T03:17:23.417 INFO:teuthology.orchestra.run.smithi118.stdout: "flags": 1, 2024-03-24T03:17:23.417 INFO:teuthology.orchestra.run.smithi118.stdout: "default_value": "", 2024-03-24T03:17:23.418 INFO:teuthology.orchestra.run.smithi118.stdout: "min": "", 2024-03-24T03:17:23.418 INFO:teuthology.orchestra.run.smithi118.stdout: "max": "", 2024-03-24T03:17:23.418 INFO:teuthology.orchestra.run.smithi118.stdout: "enum_allowed": [ 2024-03-24T03:17:23.418 INFO:teuthology.orchestra.run.smithi118.stdout: "", 2024-03-24T03:17:23.418 INFO:teuthology.orchestra.run.smithi118.stdout: "critical", 2024-03-24T03:17:23.418 INFO:teuthology.orchestra.run.smithi118.stdout: "debug", 2024-03-24T03:17:23.418 INFO:teuthology.orchestra.run.smithi118.stdout: "error", 2024-03-24T03:17:23.418 INFO:teuthology.orchestra.run.smithi118.stdout: "info", 2024-03-24T03:17:23.419 INFO:teuthology.orchestra.run.smithi118.stdout: "warning" 2024-03-24T03:17:23.419 INFO:teuthology.orchestra.run.smithi118.stdout: ], 2024-03-24T03:17:23.419 INFO:teuthology.orchestra.run.smithi118.stdout: "desc": "", 2024-03-24T03:17:23.419 INFO:teuthology.orchestra.run.smithi118.stdout: "long_desc": "", 2024-03-24T03:17:23.419 INFO:teuthology.orchestra.run.smithi118.stdout: "tags": [], 2024-03-24T03:17:23.419 INFO:teuthology.orchestra.run.smithi118.stdout: "see_also": [] 2024-03-24T03:17:23.419 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-24T03:17:23.419 INFO:teuthology.orchestra.run.smithi118.stdout: "log_to_cluster": { 2024-03-24T03:17:23.419 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "log_to_cluster", 2024-03-24T03:17:23.420 INFO:teuthology.orchestra.run.smithi118.stdout: "type": "bool", 2024-03-24T03:17:23.420 INFO:teuthology.orchestra.run.smithi118.stdout: "level": "advanced", 2024-03-24T03:17:23.420 INFO:teuthology.orchestra.run.smithi118.stdout: "flags": 1, 2024-03-24T03:17:23.420 INFO:teuthology.orchestra.run.smithi118.stdout: "default_value": "False", 2024-03-24T03:17:23.420 INFO:teuthology.orchestra.run.smithi118.stdout: "min": "", 2024-03-24T03:17:23.420 INFO:teuthology.orchestra.run.smithi118.stdout: "max": "", 2024-03-24T03:17:23.420 INFO:teuthology.orchestra.run.smithi118.stdout: "enum_allowed": [], 2024-03-24T03:17:23.420 INFO:teuthology.orchestra.run.smithi118.stdout: "desc": "", 2024-03-24T03:17:23.421 INFO:teuthology.orchestra.run.smithi118.stdout: "long_desc": "", 2024-03-24T03:17:23.421 INFO:teuthology.orchestra.run.smithi118.stdout: "tags": [], 2024-03-24T03:17:23.421 INFO:teuthology.orchestra.run.smithi118.stdout: "see_also": [] 2024-03-24T03:17:23.421 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-24T03:17:23.421 INFO:teuthology.orchestra.run.smithi118.stdout: "log_to_cluster_level": { 2024-03-24T03:17:23.421 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "log_to_cluster_level", 2024-03-24T03:17:23.421 INFO:teuthology.orchestra.run.smithi118.stdout: "type": "str", 2024-03-24T03:17:23.421 INFO:teuthology.orchestra.run.smithi118.stdout: "level": "advanced", 2024-03-24T03:17:23.421 INFO:teuthology.orchestra.run.smithi118.stdout: "flags": 1, 2024-03-24T03:17:23.422 INFO:teuthology.orchestra.run.smithi118.stdout: "default_value": "info", 2024-03-24T03:17:23.422 INFO:teuthology.orchestra.run.smithi118.stdout: "min": "", 2024-03-24T03:17:23.422 INFO:teuthology.orchestra.run.smithi118.stdout: "max": "", 2024-03-24T03:17:23.422 INFO:teuthology.orchestra.run.smithi118.stdout: "enum_allowed": [ 2024-03-24T03:17:23.422 INFO:teuthology.orchestra.run.smithi118.stdout: "", 2024-03-24T03:17:23.422 INFO:teuthology.orchestra.run.smithi118.stdout: "critical", 2024-03-24T03:17:23.422 INFO:teuthology.orchestra.run.smithi118.stdout: "debug", 2024-03-24T03:17:23.422 INFO:teuthology.orchestra.run.smithi118.stdout: "error", 2024-03-24T03:17:23.423 INFO:teuthology.orchestra.run.smithi118.stdout: "info", 2024-03-24T03:17:23.423 INFO:teuthology.orchestra.run.smithi118.stdout: "warning" 2024-03-24T03:17:23.423 INFO:teuthology.orchestra.run.smithi118.stdout: ], 2024-03-24T03:17:23.423 INFO:teuthology.orchestra.run.smithi118.stdout: "desc": "", 2024-03-24T03:17:23.424 INFO:teuthology.orchestra.run.smithi118.stdout: "long_desc": "", 2024-03-24T03:17:23.424 INFO:teuthology.orchestra.run.smithi118.stdout: "tags": [], 2024-03-24T03:17:23.424 INFO:teuthology.orchestra.run.smithi118.stdout: "see_also": [] 2024-03-24T03:17:23.424 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-24T03:17:23.424 INFO:teuthology.orchestra.run.smithi118.stdout: "log_to_file": { 2024-03-24T03:17:23.424 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "log_to_file", 2024-03-24T03:17:23.425 INFO:teuthology.orchestra.run.smithi118.stdout: "type": "bool", 2024-03-24T03:17:23.425 INFO:teuthology.orchestra.run.smithi118.stdout: "level": "advanced", 2024-03-24T03:17:23.425 INFO:teuthology.orchestra.run.smithi118.stdout: "flags": 1, 2024-03-24T03:17:23.425 INFO:teuthology.orchestra.run.smithi118.stdout: "default_value": "False", 2024-03-24T03:17:23.426 INFO:teuthology.orchestra.run.smithi118.stdout: "min": "", 2024-03-24T03:17:23.426 INFO:teuthology.orchestra.run.smithi118.stdout: "max": "", 2024-03-24T03:17:23.426 INFO:teuthology.orchestra.run.smithi118.stdout: "enum_allowed": [], 2024-03-24T03:17:23.426 INFO:teuthology.orchestra.run.smithi118.stdout: "desc": "", 2024-03-24T03:17:23.426 INFO:teuthology.orchestra.run.smithi118.stdout: "long_desc": "", 2024-03-24T03:17:23.426 INFO:teuthology.orchestra.run.smithi118.stdout: "tags": [], 2024-03-24T03:17:23.427 INFO:teuthology.orchestra.run.smithi118.stdout: "see_also": [] 2024-03-24T03:17:23.427 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-24T03:17:23.427 INFO:teuthology.orchestra.run.smithi118.stdout: "motd": { 2024-03-24T03:17:23.427 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "motd", 2024-03-24T03:17:23.427 INFO:teuthology.orchestra.run.smithi118.stdout: "type": "str", 2024-03-24T03:17:23.427 INFO:teuthology.orchestra.run.smithi118.stdout: "level": "advanced", 2024-03-24T03:17:23.428 INFO:teuthology.orchestra.run.smithi118.stdout: "flags": 0, 2024-03-24T03:17:23.428 INFO:teuthology.orchestra.run.smithi118.stdout: "default_value": "", 2024-03-24T03:17:23.428 INFO:teuthology.orchestra.run.smithi118.stdout: "min": "", 2024-03-24T03:17:23.428 INFO:teuthology.orchestra.run.smithi118.stdout: "max": "", 2024-03-24T03:17:23.428 INFO:teuthology.orchestra.run.smithi118.stdout: "enum_allowed": [], 2024-03-24T03:17:23.428 INFO:teuthology.orchestra.run.smithi118.stdout: "desc": "The message of the day", 2024-03-24T03:17:23.429 INFO:teuthology.orchestra.run.smithi118.stdout: "long_desc": "", 2024-03-24T03:17:23.429 INFO:teuthology.orchestra.run.smithi118.stdout: "tags": [], 2024-03-24T03:17:23.429 INFO:teuthology.orchestra.run.smithi118.stdout: "see_also": [] 2024-03-24T03:17:23.429 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-24T03:17:23.429 INFO:teuthology.orchestra.run.smithi118.stdout: "redirect_resolve_ip_addr": { 2024-03-24T03:17:23.429 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "redirect_resolve_ip_addr", 2024-03-24T03:17:23.430 INFO:teuthology.orchestra.run.smithi118.stdout: "type": "bool", 2024-03-24T03:17:23.430 INFO:teuthology.orchestra.run.smithi118.stdout: "level": "advanced", 2024-03-24T03:17:23.430 INFO:teuthology.orchestra.run.smithi118.stdout: "flags": 0, 2024-03-24T03:17:23.430 INFO:teuthology.orchestra.run.smithi118.stdout: "default_value": "False", 2024-03-24T03:17:23.430 INFO:teuthology.orchestra.run.smithi118.stdout: "min": "", 2024-03-24T03:17:23.431 INFO:teuthology.orchestra.run.smithi118.stdout: "max": "", 2024-03-24T03:17:23.431 INFO:teuthology.orchestra.run.smithi118.stdout: "enum_allowed": [], 2024-03-24T03:17:23.431 INFO:teuthology.orchestra.run.smithi118.stdout: "desc": "", 2024-03-24T03:17:23.431 INFO:teuthology.orchestra.run.smithi118.stdout: "long_desc": "", 2024-03-24T03:17:23.431 INFO:teuthology.orchestra.run.smithi118.stdout: "tags": [], 2024-03-24T03:17:23.431 INFO:teuthology.orchestra.run.smithi118.stdout: "see_also": [] 2024-03-24T03:17:23.432 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-24T03:17:23.432 INFO:teuthology.orchestra.run.smithi118.stdout: "server_addr": { 2024-03-24T03:17:23.432 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "server_addr", 2024-03-24T03:17:23.432 INFO:teuthology.orchestra.run.smithi118.stdout: "type": "str", 2024-03-24T03:17:23.432 INFO:teuthology.orchestra.run.smithi118.stdout: "level": "advanced", 2024-03-24T03:17:23.433 INFO:teuthology.orchestra.run.smithi118.stdout: "flags": 0, 2024-03-24T03:17:23.433 INFO:teuthology.orchestra.run.smithi118.stdout: "default_value": "::", 2024-03-24T03:17:23.433 INFO:teuthology.orchestra.run.smithi118.stdout: "min": "", 2024-03-24T03:17:23.433 INFO:teuthology.orchestra.run.smithi118.stdout: "max": "", 2024-03-24T03:17:23.433 INFO:teuthology.orchestra.run.smithi118.stdout: "enum_allowed": [], 2024-03-24T03:17:23.433 INFO:teuthology.orchestra.run.smithi118.stdout: "desc": "", 2024-03-24T03:17:23.433 INFO:teuthology.orchestra.run.smithi118.stdout: "long_desc": "", 2024-03-24T03:17:23.434 INFO:teuthology.orchestra.run.smithi118.stdout: "tags": [], 2024-03-24T03:17:23.434 INFO:teuthology.orchestra.run.smithi118.stdout: "see_also": [] 2024-03-24T03:17:23.434 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-24T03:17:23.434 INFO:teuthology.orchestra.run.smithi118.stdout: "server_port": { 2024-03-24T03:17:23.434 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "server_port", 2024-03-24T03:17:23.434 INFO:teuthology.orchestra.run.smithi118.stdout: "type": "int", 2024-03-24T03:17:23.434 INFO:teuthology.orchestra.run.smithi118.stdout: "level": "advanced", 2024-03-24T03:17:23.434 INFO:teuthology.orchestra.run.smithi118.stdout: "flags": 0, 2024-03-24T03:17:23.435 INFO:teuthology.orchestra.run.smithi118.stdout: "default_value": "8080", 2024-03-24T03:17:23.435 INFO:teuthology.orchestra.run.smithi118.stdout: "min": "", 2024-03-24T03:17:23.435 INFO:teuthology.orchestra.run.smithi118.stdout: "max": "", 2024-03-24T03:17:23.435 INFO:teuthology.orchestra.run.smithi118.stdout: "enum_allowed": [], 2024-03-24T03:17:23.435 INFO:teuthology.orchestra.run.smithi118.stdout: "desc": "", 2024-03-24T03:17:23.435 INFO:teuthology.orchestra.run.smithi118.stdout: "long_desc": "", 2024-03-24T03:17:23.435 INFO:teuthology.orchestra.run.smithi118.stdout: "tags": [], 2024-03-24T03:17:23.435 INFO:teuthology.orchestra.run.smithi118.stdout: "see_also": [] 2024-03-24T03:17:23.436 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-24T03:17:23.436 INFO:teuthology.orchestra.run.smithi118.stdout: "ssl": { 2024-03-24T03:17:23.436 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "ssl", 2024-03-24T03:17:23.436 INFO:teuthology.orchestra.run.smithi118.stdout: "type": "bool", 2024-03-24T03:17:23.436 INFO:teuthology.orchestra.run.smithi118.stdout: "level": "advanced", 2024-03-24T03:17:23.436 INFO:teuthology.orchestra.run.smithi118.stdout: "flags": 0, 2024-03-24T03:17:23.436 INFO:teuthology.orchestra.run.smithi118.stdout: "default_value": "True", 2024-03-24T03:17:23.436 INFO:teuthology.orchestra.run.smithi118.stdout: "min": "", 2024-03-24T03:17:23.436 INFO:teuthology.orchestra.run.smithi118.stdout: "max": "", 2024-03-24T03:17:23.437 INFO:teuthology.orchestra.run.smithi118.stdout: "enum_allowed": [], 2024-03-24T03:17:23.437 INFO:teuthology.orchestra.run.smithi118.stdout: "desc": "", 2024-03-24T03:17:23.437 INFO:teuthology.orchestra.run.smithi118.stdout: "long_desc": "", 2024-03-24T03:17:23.437 INFO:teuthology.orchestra.run.smithi118.stdout: "tags": [], 2024-03-24T03:17:23.437 INFO:teuthology.orchestra.run.smithi118.stdout: "see_also": [] 2024-03-24T03:17:23.437 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-24T03:17:23.437 INFO:teuthology.orchestra.run.smithi118.stdout: "ssl_server_port": { 2024-03-24T03:17:23.437 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "ssl_server_port", 2024-03-24T03:17:23.437 INFO:teuthology.orchestra.run.smithi118.stdout: "type": "int", 2024-03-24T03:17:23.438 INFO:teuthology.orchestra.run.smithi118.stdout: "level": "advanced", 2024-03-24T03:17:23.438 INFO:teuthology.orchestra.run.smithi118.stdout: "flags": 0, 2024-03-24T03:17:23.438 INFO:teuthology.orchestra.run.smithi118.stdout: "default_value": "8443", 2024-03-24T03:17:23.438 INFO:teuthology.orchestra.run.smithi118.stdout: "min": "", 2024-03-24T03:17:23.438 INFO:teuthology.orchestra.run.smithi118.stdout: "max": "", 2024-03-24T03:17:23.438 INFO:teuthology.orchestra.run.smithi118.stdout: "enum_allowed": [], 2024-03-24T03:17:23.438 INFO:teuthology.orchestra.run.smithi118.stdout: "desc": "", 2024-03-24T03:17:23.439 INFO:teuthology.orchestra.run.smithi118.stdout: "long_desc": "", 2024-03-24T03:17:23.439 INFO:teuthology.orchestra.run.smithi118.stdout: "tags": [], 2024-03-24T03:17:23.439 INFO:teuthology.orchestra.run.smithi118.stdout: "see_also": [] 2024-03-24T03:17:23.439 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-24T03:17:23.439 INFO:teuthology.orchestra.run.smithi118.stdout: "standby_behaviour": { 2024-03-24T03:17:23.439 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "standby_behaviour", 2024-03-24T03:17:23.439 INFO:teuthology.orchestra.run.smithi118.stdout: "type": "str", 2024-03-24T03:17:23.439 INFO:teuthology.orchestra.run.smithi118.stdout: "level": "advanced", 2024-03-24T03:17:23.439 INFO:teuthology.orchestra.run.smithi118.stdout: "flags": 0, 2024-03-24T03:17:23.440 INFO:teuthology.orchestra.run.smithi118.stdout: "default_value": "redirect", 2024-03-24T03:17:23.440 INFO:teuthology.orchestra.run.smithi118.stdout: "min": "", 2024-03-24T03:17:23.440 INFO:teuthology.orchestra.run.smithi118.stdout: "max": "", 2024-03-24T03:17:23.440 INFO:teuthology.orchestra.run.smithi118.stdout: "enum_allowed": [ 2024-03-24T03:17:23.440 INFO:teuthology.orchestra.run.smithi118.stdout: "error", 2024-03-24T03:17:23.440 INFO:teuthology.orchestra.run.smithi118.stdout: "redirect" 2024-03-24T03:17:23.440 INFO:teuthology.orchestra.run.smithi118.stdout: ], 2024-03-24T03:17:23.440 INFO:teuthology.orchestra.run.smithi118.stdout: "desc": "", 2024-03-24T03:17:23.440 INFO:teuthology.orchestra.run.smithi118.stdout: "long_desc": "", 2024-03-24T03:17:23.441 INFO:teuthology.orchestra.run.smithi118.stdout: "tags": [], 2024-03-24T03:17:23.441 INFO:teuthology.orchestra.run.smithi118.stdout: "see_also": [] 2024-03-24T03:17:23.441 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-24T03:17:23.441 INFO:teuthology.orchestra.run.smithi118.stdout: "standby_error_status_code": { 2024-03-24T03:17:23.441 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "standby_error_status_code", 2024-03-24T03:17:23.441 INFO:teuthology.orchestra.run.smithi118.stdout: "type": "int", 2024-03-24T03:17:23.441 INFO:teuthology.orchestra.run.smithi118.stdout: "level": "advanced", 2024-03-24T03:17:23.441 INFO:teuthology.orchestra.run.smithi118.stdout: "flags": 0, 2024-03-24T03:17:23.442 INFO:teuthology.orchestra.run.smithi118.stdout: "default_value": "500", 2024-03-24T03:17:23.442 INFO:teuthology.orchestra.run.smithi118.stdout: "min": "400", 2024-03-24T03:17:23.442 INFO:teuthology.orchestra.run.smithi118.stdout: "max": "599", 2024-03-24T03:17:23.442 INFO:teuthology.orchestra.run.smithi118.stdout: "enum_allowed": [], 2024-03-24T03:17:23.442 INFO:teuthology.orchestra.run.smithi118.stdout: "desc": "", 2024-03-24T03:17:23.442 INFO:teuthology.orchestra.run.smithi118.stdout: "long_desc": "", 2024-03-24T03:17:23.442 INFO:teuthology.orchestra.run.smithi118.stdout: "tags": [], 2024-03-24T03:17:23.443 INFO:teuthology.orchestra.run.smithi118.stdout: "see_also": [] 2024-03-24T03:17:23.443 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-24T03:17:23.443 INFO:teuthology.orchestra.run.smithi118.stdout: "url_prefix": { 2024-03-24T03:17:23.443 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "url_prefix", 2024-03-24T03:17:23.443 INFO:teuthology.orchestra.run.smithi118.stdout: "type": "str", 2024-03-24T03:17:23.443 INFO:teuthology.orchestra.run.smithi118.stdout: "level": "advanced", 2024-03-24T03:17:23.443 INFO:teuthology.orchestra.run.smithi118.stdout: "flags": 0, 2024-03-24T03:17:23.443 INFO:teuthology.orchestra.run.smithi118.stdout: "default_value": "", 2024-03-24T03:17:23.443 INFO:teuthology.orchestra.run.smithi118.stdout: "min": "", 2024-03-24T03:17:23.444 INFO:teuthology.orchestra.run.smithi118.stdout: "max": "", 2024-03-24T03:17:23.444 INFO:teuthology.orchestra.run.smithi118.stdout: "enum_allowed": [], 2024-03-24T03:17:23.444 INFO:teuthology.orchestra.run.smithi118.stdout: "desc": "", 2024-03-24T03:17:23.444 INFO:teuthology.orchestra.run.smithi118.stdout: "long_desc": "", 2024-03-24T03:17:23.444 INFO:teuthology.orchestra.run.smithi118.stdout: "tags": [], 2024-03-24T03:17:23.444 INFO:teuthology.orchestra.run.smithi118.stdout: "see_also": [] 2024-03-24T03:17:23.444 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-03-24T03:17:23.444 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-03-24T03:17:23.445 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-24T03:17:23.445 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-24T03:17:23.445 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "devicehealth", 2024-03-24T03:17:23.445 INFO:teuthology.orchestra.run.smithi118.stdout: "can_run": true, 2024-03-24T03:17:23.445 INFO:teuthology.orchestra.run.smithi118.stdout: "error_string": "", 2024-03-24T03:17:23.445 INFO:teuthology.orchestra.run.smithi118.stdout: "module_options": { 2024-03-24T03:17:23.445 INFO:teuthology.orchestra.run.smithi118.stdout: "enable_monitoring": { 2024-03-24T03:17:23.445 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "enable_monitoring", 2024-03-24T03:17:23.445 INFO:teuthology.orchestra.run.smithi118.stdout: "type": "bool", 2024-03-24T03:17:23.446 INFO:teuthology.orchestra.run.smithi118.stdout: "level": "advanced", 2024-03-24T03:17:23.446 INFO:teuthology.orchestra.run.smithi118.stdout: "flags": 1, 2024-03-24T03:17:23.446 INFO:teuthology.orchestra.run.smithi118.stdout: "default_value": "True", 2024-03-24T03:17:23.446 INFO:teuthology.orchestra.run.smithi118.stdout: "min": "", 2024-03-24T03:17:23.446 INFO:teuthology.orchestra.run.smithi118.stdout: "max": "", 2024-03-24T03:17:23.446 INFO:teuthology.orchestra.run.smithi118.stdout: "enum_allowed": [], 2024-03-24T03:17:23.446 INFO:teuthology.orchestra.run.smithi118.stdout: "desc": "monitor device health metrics", 2024-03-24T03:17:23.446 INFO:teuthology.orchestra.run.smithi118.stdout: "long_desc": "", 2024-03-24T03:17:23.447 INFO:teuthology.orchestra.run.smithi118.stdout: "tags": [], 2024-03-24T03:17:23.447 INFO:teuthology.orchestra.run.smithi118.stdout: "see_also": [] 2024-03-24T03:17:23.447 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-24T03:17:23.447 INFO:teuthology.orchestra.run.smithi118.stdout: "log_level": { 2024-03-24T03:17:23.447 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "log_level", 2024-03-24T03:17:23.447 INFO:teuthology.orchestra.run.smithi118.stdout: "type": "str", 2024-03-24T03:17:23.447 INFO:teuthology.orchestra.run.smithi118.stdout: "level": "advanced", 2024-03-24T03:17:23.447 INFO:teuthology.orchestra.run.smithi118.stdout: "flags": 1, 2024-03-24T03:17:23.447 INFO:teuthology.orchestra.run.smithi118.stdout: "default_value": "", 2024-03-24T03:17:23.448 INFO:teuthology.orchestra.run.smithi118.stdout: "min": "", 2024-03-24T03:17:23.448 INFO:teuthology.orchestra.run.smithi118.stdout: "max": "", 2024-03-24T03:17:23.448 INFO:teuthology.orchestra.run.smithi118.stdout: "enum_allowed": [ 2024-03-24T03:17:23.448 INFO:teuthology.orchestra.run.smithi118.stdout: "", 2024-03-24T03:17:23.448 INFO:teuthology.orchestra.run.smithi118.stdout: "critical", 2024-03-24T03:17:23.448 INFO:teuthology.orchestra.run.smithi118.stdout: "debug", 2024-03-24T03:17:23.448 INFO:teuthology.orchestra.run.smithi118.stdout: "error", 2024-03-24T03:17:23.448 INFO:teuthology.orchestra.run.smithi118.stdout: "info", 2024-03-24T03:17:23.449 INFO:teuthology.orchestra.run.smithi118.stdout: "warning" 2024-03-24T03:17:23.449 INFO:teuthology.orchestra.run.smithi118.stdout: ], 2024-03-24T03:17:23.449 INFO:teuthology.orchestra.run.smithi118.stdout: "desc": "", 2024-03-24T03:17:23.449 INFO:teuthology.orchestra.run.smithi118.stdout: "long_desc": "", 2024-03-24T03:17:23.449 INFO:teuthology.orchestra.run.smithi118.stdout: "tags": [], 2024-03-24T03:17:23.449 INFO:teuthology.orchestra.run.smithi118.stdout: "see_also": [] 2024-03-24T03:17:23.449 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-24T03:17:23.449 INFO:teuthology.orchestra.run.smithi118.stdout: "log_to_cluster": { 2024-03-24T03:17:23.449 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "log_to_cluster", 2024-03-24T03:17:23.450 INFO:teuthology.orchestra.run.smithi118.stdout: "type": "bool", 2024-03-24T03:17:23.450 INFO:teuthology.orchestra.run.smithi118.stdout: "level": "advanced", 2024-03-24T03:17:23.450 INFO:teuthology.orchestra.run.smithi118.stdout: "flags": 1, 2024-03-24T03:17:23.450 INFO:teuthology.orchestra.run.smithi118.stdout: "default_value": "False", 2024-03-24T03:17:23.450 INFO:teuthology.orchestra.run.smithi118.stdout: "min": "", 2024-03-24T03:17:23.450 INFO:teuthology.orchestra.run.smithi118.stdout: "max": "", 2024-03-24T03:17:23.450 INFO:teuthology.orchestra.run.smithi118.stdout: "enum_allowed": [], 2024-03-24T03:17:23.451 INFO:teuthology.orchestra.run.smithi118.stdout: "desc": "", 2024-03-24T03:17:23.451 INFO:teuthology.orchestra.run.smithi118.stdout: "long_desc": "", 2024-03-24T03:17:23.451 INFO:teuthology.orchestra.run.smithi118.stdout: "tags": [], 2024-03-24T03:17:23.451 INFO:teuthology.orchestra.run.smithi118.stdout: "see_also": [] 2024-03-24T03:17:23.451 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-24T03:17:23.451 INFO:teuthology.orchestra.run.smithi118.stdout: "log_to_cluster_level": { 2024-03-24T03:17:23.451 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "log_to_cluster_level", 2024-03-24T03:17:23.451 INFO:teuthology.orchestra.run.smithi118.stdout: "type": "str", 2024-03-24T03:17:23.452 INFO:teuthology.orchestra.run.smithi118.stdout: "level": "advanced", 2024-03-24T03:17:23.452 INFO:teuthology.orchestra.run.smithi118.stdout: "flags": 1, 2024-03-24T03:17:23.452 INFO:teuthology.orchestra.run.smithi118.stdout: "default_value": "info", 2024-03-24T03:17:23.452 INFO:teuthology.orchestra.run.smithi118.stdout: "min": "", 2024-03-24T03:17:23.452 INFO:teuthology.orchestra.run.smithi118.stdout: "max": "", 2024-03-24T03:17:23.452 INFO:teuthology.orchestra.run.smithi118.stdout: "enum_allowed": [ 2024-03-24T03:17:23.452 INFO:teuthology.orchestra.run.smithi118.stdout: "", 2024-03-24T03:17:23.452 INFO:teuthology.orchestra.run.smithi118.stdout: "critical", 2024-03-24T03:17:23.452 INFO:teuthology.orchestra.run.smithi118.stdout: "debug", 2024-03-24T03:17:23.453 INFO:teuthology.orchestra.run.smithi118.stdout: "error", 2024-03-24T03:17:23.453 INFO:teuthology.orchestra.run.smithi118.stdout: "info", 2024-03-24T03:17:23.453 INFO:teuthology.orchestra.run.smithi118.stdout: "warning" 2024-03-24T03:17:23.453 INFO:teuthology.orchestra.run.smithi118.stdout: ], 2024-03-24T03:17:23.453 INFO:teuthology.orchestra.run.smithi118.stdout: "desc": "", 2024-03-24T03:17:23.453 INFO:teuthology.orchestra.run.smithi118.stdout: "long_desc": "", 2024-03-24T03:17:23.453 INFO:teuthology.orchestra.run.smithi118.stdout: "tags": [], 2024-03-24T03:17:23.453 INFO:teuthology.orchestra.run.smithi118.stdout: "see_also": [] 2024-03-24T03:17:23.454 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-24T03:17:23.454 INFO:teuthology.orchestra.run.smithi118.stdout: "log_to_file": { 2024-03-24T03:17:23.454 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "log_to_file", 2024-03-24T03:17:23.454 INFO:teuthology.orchestra.run.smithi118.stdout: "type": "bool", 2024-03-24T03:17:23.454 INFO:teuthology.orchestra.run.smithi118.stdout: "level": "advanced", 2024-03-24T03:17:23.454 INFO:teuthology.orchestra.run.smithi118.stdout: "flags": 1, 2024-03-24T03:17:23.454 INFO:teuthology.orchestra.run.smithi118.stdout: "default_value": "False", 2024-03-24T03:17:23.454 INFO:teuthology.orchestra.run.smithi118.stdout: "min": "", 2024-03-24T03:17:23.455 INFO:teuthology.orchestra.run.smithi118.stdout: "max": "", 2024-03-24T03:17:23.455 INFO:teuthology.orchestra.run.smithi118.stdout: "enum_allowed": [], 2024-03-24T03:17:23.455 INFO:teuthology.orchestra.run.smithi118.stdout: "desc": "", 2024-03-24T03:17:23.455 INFO:teuthology.orchestra.run.smithi118.stdout: "long_desc": "", 2024-03-24T03:17:23.455 INFO:teuthology.orchestra.run.smithi118.stdout: "tags": [], 2024-03-24T03:17:23.455 INFO:teuthology.orchestra.run.smithi118.stdout: "see_also": [] 2024-03-24T03:17:23.455 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-24T03:17:23.455 INFO:teuthology.orchestra.run.smithi118.stdout: "mark_out_threshold": { 2024-03-24T03:17:23.456 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "mark_out_threshold", 2024-03-24T03:17:23.456 INFO:teuthology.orchestra.run.smithi118.stdout: "type": "secs", 2024-03-24T03:17:23.456 INFO:teuthology.orchestra.run.smithi118.stdout: "level": "advanced", 2024-03-24T03:17:23.456 INFO:teuthology.orchestra.run.smithi118.stdout: "flags": 1, 2024-03-24T03:17:23.456 INFO:teuthology.orchestra.run.smithi118.stdout: "default_value": "2419200", 2024-03-24T03:17:23.456 INFO:teuthology.orchestra.run.smithi118.stdout: "min": "", 2024-03-24T03:17:23.456 INFO:teuthology.orchestra.run.smithi118.stdout: "max": "", 2024-03-24T03:17:23.456 INFO:teuthology.orchestra.run.smithi118.stdout: "enum_allowed": [], 2024-03-24T03:17:23.457 INFO:teuthology.orchestra.run.smithi118.stdout: "desc": "automatically mark OSD if it may fail before this long", 2024-03-24T03:17:23.457 INFO:teuthology.orchestra.run.smithi118.stdout: "long_desc": "", 2024-03-24T03:17:23.457 INFO:teuthology.orchestra.run.smithi118.stdout: "tags": [], 2024-03-24T03:17:23.457 INFO:teuthology.orchestra.run.smithi118.stdout: "see_also": [] 2024-03-24T03:17:23.457 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-24T03:17:23.457 INFO:teuthology.orchestra.run.smithi118.stdout: "pool_name": { 2024-03-24T03:17:23.457 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "pool_name", 2024-03-24T03:17:23.457 INFO:teuthology.orchestra.run.smithi118.stdout: "type": "str", 2024-03-24T03:17:23.457 INFO:teuthology.orchestra.run.smithi118.stdout: "level": "advanced", 2024-03-24T03:17:23.458 INFO:teuthology.orchestra.run.smithi118.stdout: "flags": 1, 2024-03-24T03:17:23.458 INFO:teuthology.orchestra.run.smithi118.stdout: "default_value": "device_health_metrics", 2024-03-24T03:17:23.458 INFO:teuthology.orchestra.run.smithi118.stdout: "min": "", 2024-03-24T03:17:23.458 INFO:teuthology.orchestra.run.smithi118.stdout: "max": "", 2024-03-24T03:17:23.458 INFO:teuthology.orchestra.run.smithi118.stdout: "enum_allowed": [], 2024-03-24T03:17:23.458 INFO:teuthology.orchestra.run.smithi118.stdout: "desc": "name of pool in which to store device health metrics", 2024-03-24T03:17:23.458 INFO:teuthology.orchestra.run.smithi118.stdout: "long_desc": "", 2024-03-24T03:17:23.458 INFO:teuthology.orchestra.run.smithi118.stdout: "tags": [], 2024-03-24T03:17:23.459 INFO:teuthology.orchestra.run.smithi118.stdout: "see_also": [] 2024-03-24T03:17:23.459 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-24T03:17:23.459 INFO:teuthology.orchestra.run.smithi118.stdout: "retention_period": { 2024-03-24T03:17:23.459 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "retention_period", 2024-03-24T03:17:23.459 INFO:teuthology.orchestra.run.smithi118.stdout: "type": "secs", 2024-03-24T03:17:23.459 INFO:teuthology.orchestra.run.smithi118.stdout: "level": "advanced", 2024-03-24T03:17:23.459 INFO:teuthology.orchestra.run.smithi118.stdout: "flags": 1, 2024-03-24T03:17:23.459 INFO:teuthology.orchestra.run.smithi118.stdout: "default_value": "15552000", 2024-03-24T03:17:23.460 INFO:teuthology.orchestra.run.smithi118.stdout: "min": "", 2024-03-24T03:17:23.460 INFO:teuthology.orchestra.run.smithi118.stdout: "max": "", 2024-03-24T03:17:23.460 INFO:teuthology.orchestra.run.smithi118.stdout: "enum_allowed": [], 2024-03-24T03:17:23.460 INFO:teuthology.orchestra.run.smithi118.stdout: "desc": "how long to retain device health metrics", 2024-03-24T03:17:23.460 INFO:teuthology.orchestra.run.smithi118.stdout: "long_desc": "", 2024-03-24T03:17:23.460 INFO:teuthology.orchestra.run.smithi118.stdout: "tags": [], 2024-03-24T03:17:23.460 INFO:teuthology.orchestra.run.smithi118.stdout: "see_also": [] 2024-03-24T03:17:23.461 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-24T03:17:23.461 INFO:teuthology.orchestra.run.smithi118.stdout: "scrape_frequency": { 2024-03-24T03:17:23.461 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "scrape_frequency", 2024-03-24T03:17:23.461 INFO:teuthology.orchestra.run.smithi118.stdout: "type": "secs", 2024-03-24T03:17:23.461 INFO:teuthology.orchestra.run.smithi118.stdout: "level": "advanced", 2024-03-24T03:17:23.461 INFO:teuthology.orchestra.run.smithi118.stdout: "flags": 1, 2024-03-24T03:17:23.461 INFO:teuthology.orchestra.run.smithi118.stdout: "default_value": "86400", 2024-03-24T03:17:23.461 INFO:teuthology.orchestra.run.smithi118.stdout: "min": "", 2024-03-24T03:17:23.462 INFO:teuthology.orchestra.run.smithi118.stdout: "max": "", 2024-03-24T03:17:23.462 INFO:teuthology.orchestra.run.smithi118.stdout: "enum_allowed": [], 2024-03-24T03:17:23.462 INFO:teuthology.orchestra.run.smithi118.stdout: "desc": "how frequently to scrape device health metrics", 2024-03-24T03:17:23.462 INFO:teuthology.orchestra.run.smithi118.stdout: "long_desc": "", 2024-03-24T03:17:23.462 INFO:teuthology.orchestra.run.smithi118.stdout: "tags": [], 2024-03-24T03:17:23.462 INFO:teuthology.orchestra.run.smithi118.stdout: "see_also": [] 2024-03-24T03:17:23.462 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-24T03:17:23.462 INFO:teuthology.orchestra.run.smithi118.stdout: "self_heal": { 2024-03-24T03:17:23.463 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "self_heal", 2024-03-24T03:17:23.463 INFO:teuthology.orchestra.run.smithi118.stdout: "type": "bool", 2024-03-24T03:17:23.463 INFO:teuthology.orchestra.run.smithi118.stdout: "level": "advanced", 2024-03-24T03:17:23.463 INFO:teuthology.orchestra.run.smithi118.stdout: "flags": 1, 2024-03-24T03:17:23.463 INFO:teuthology.orchestra.run.smithi118.stdout: "default_value": "True", 2024-03-24T03:17:23.463 INFO:teuthology.orchestra.run.smithi118.stdout: "min": "", 2024-03-24T03:17:23.463 INFO:teuthology.orchestra.run.smithi118.stdout: "max": "", 2024-03-24T03:17:23.464 INFO:teuthology.orchestra.run.smithi118.stdout: "enum_allowed": [], 2024-03-24T03:17:23.464 INFO:teuthology.orchestra.run.smithi118.stdout: "desc": "preemptively heal cluster around devices that may fail", 2024-03-24T03:17:23.464 INFO:teuthology.orchestra.run.smithi118.stdout: "long_desc": "", 2024-03-24T03:17:23.464 INFO:teuthology.orchestra.run.smithi118.stdout: "tags": [], 2024-03-24T03:17:23.464 INFO:teuthology.orchestra.run.smithi118.stdout: "see_also": [] 2024-03-24T03:17:23.464 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-24T03:17:23.464 INFO:teuthology.orchestra.run.smithi118.stdout: "sleep_interval": { 2024-03-24T03:17:23.464 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "sleep_interval", 2024-03-24T03:17:23.465 INFO:teuthology.orchestra.run.smithi118.stdout: "type": "secs", 2024-03-24T03:17:23.465 INFO:teuthology.orchestra.run.smithi118.stdout: "level": "advanced", 2024-03-24T03:17:23.465 INFO:teuthology.orchestra.run.smithi118.stdout: "flags": 1, 2024-03-24T03:17:23.465 INFO:teuthology.orchestra.run.smithi118.stdout: "default_value": "600", 2024-03-24T03:17:23.465 INFO:teuthology.orchestra.run.smithi118.stdout: "min": "", 2024-03-24T03:17:23.465 INFO:teuthology.orchestra.run.smithi118.stdout: "max": "", 2024-03-24T03:17:23.465 INFO:teuthology.orchestra.run.smithi118.stdout: "enum_allowed": [], 2024-03-24T03:17:23.465 INFO:teuthology.orchestra.run.smithi118.stdout: "desc": "how frequently to wake up and check device health", 2024-03-24T03:17:23.465 INFO:teuthology.orchestra.run.smithi118.stdout: "long_desc": "", 2024-03-24T03:17:23.466 INFO:teuthology.orchestra.run.smithi118.stdout: "tags": [], 2024-03-24T03:17:23.466 INFO:teuthology.orchestra.run.smithi118.stdout: "see_also": [] 2024-03-24T03:17:23.466 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-24T03:17:23.466 INFO:teuthology.orchestra.run.smithi118.stdout: "warn_threshold": { 2024-03-24T03:17:23.466 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "warn_threshold", 2024-03-24T03:17:23.466 INFO:teuthology.orchestra.run.smithi118.stdout: "type": "secs", 2024-03-24T03:17:23.466 INFO:teuthology.orchestra.run.smithi118.stdout: "level": "advanced", 2024-03-24T03:17:23.466 INFO:teuthology.orchestra.run.smithi118.stdout: "flags": 1, 2024-03-24T03:17:23.467 INFO:teuthology.orchestra.run.smithi118.stdout: "default_value": "7257600", 2024-03-24T03:17:23.467 INFO:teuthology.orchestra.run.smithi118.stdout: "min": "", 2024-03-24T03:17:23.467 INFO:teuthology.orchestra.run.smithi118.stdout: "max": "", 2024-03-24T03:17:23.467 INFO:teuthology.orchestra.run.smithi118.stdout: "enum_allowed": [], 2024-03-24T03:17:23.467 INFO:teuthology.orchestra.run.smithi118.stdout: "desc": "raise health warning if OSD may fail before this long", 2024-03-24T03:17:23.467 INFO:teuthology.orchestra.run.smithi118.stdout: "long_desc": "", 2024-03-24T03:17:23.467 INFO:teuthology.orchestra.run.smithi118.stdout: "tags": [], 2024-03-24T03:17:23.467 INFO:teuthology.orchestra.run.smithi118.stdout: "see_also": [] 2024-03-24T03:17:23.467 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-03-24T03:17:23.468 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-03-24T03:17:23.468 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-24T03:17:23.468 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-24T03:17:23.468 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "diskprediction_local", 2024-03-24T03:17:23.468 INFO:teuthology.orchestra.run.smithi118.stdout: "can_run": true, 2024-03-24T03:17:23.468 INFO:teuthology.orchestra.run.smithi118.stdout: "error_string": "", 2024-03-24T03:17:23.468 INFO:teuthology.orchestra.run.smithi118.stdout: "module_options": { 2024-03-24T03:17:23.468 INFO:teuthology.orchestra.run.smithi118.stdout: "log_level": { 2024-03-24T03:17:23.468 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "log_level", 2024-03-24T03:17:23.469 INFO:teuthology.orchestra.run.smithi118.stdout: "type": "str", 2024-03-24T03:17:23.469 INFO:teuthology.orchestra.run.smithi118.stdout: "level": "advanced", 2024-03-24T03:17:23.469 INFO:teuthology.orchestra.run.smithi118.stdout: "flags": 1, 2024-03-24T03:17:23.469 INFO:teuthology.orchestra.run.smithi118.stdout: "default_value": "", 2024-03-24T03:17:23.469 INFO:teuthology.orchestra.run.smithi118.stdout: "min": "", 2024-03-24T03:17:23.469 INFO:teuthology.orchestra.run.smithi118.stdout: "max": "", 2024-03-24T03:17:23.469 INFO:teuthology.orchestra.run.smithi118.stdout: "enum_allowed": [ 2024-03-24T03:17:23.469 INFO:teuthology.orchestra.run.smithi118.stdout: "", 2024-03-24T03:17:23.469 INFO:teuthology.orchestra.run.smithi118.stdout: "critical", 2024-03-24T03:17:23.470 INFO:teuthology.orchestra.run.smithi118.stdout: "debug", 2024-03-24T03:17:23.470 INFO:teuthology.orchestra.run.smithi118.stdout: "error", 2024-03-24T03:17:23.470 INFO:teuthology.orchestra.run.smithi118.stdout: "info", 2024-03-24T03:17:23.470 INFO:teuthology.orchestra.run.smithi118.stdout: "warning" 2024-03-24T03:17:23.470 INFO:teuthology.orchestra.run.smithi118.stdout: ], 2024-03-24T03:17:23.470 INFO:teuthology.orchestra.run.smithi118.stdout: "desc": "", 2024-03-24T03:17:23.470 INFO:teuthology.orchestra.run.smithi118.stdout: "long_desc": "", 2024-03-24T03:17:23.470 INFO:teuthology.orchestra.run.smithi118.stdout: "tags": [], 2024-03-24T03:17:23.471 INFO:teuthology.orchestra.run.smithi118.stdout: "see_also": [] 2024-03-24T03:17:23.471 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-24T03:17:23.471 INFO:teuthology.orchestra.run.smithi118.stdout: "log_to_cluster": { 2024-03-24T03:17:23.471 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "log_to_cluster", 2024-03-24T03:17:23.471 INFO:teuthology.orchestra.run.smithi118.stdout: "type": "bool", 2024-03-24T03:17:23.471 INFO:teuthology.orchestra.run.smithi118.stdout: "level": "advanced", 2024-03-24T03:17:23.471 INFO:teuthology.orchestra.run.smithi118.stdout: "flags": 1, 2024-03-24T03:17:23.471 INFO:teuthology.orchestra.run.smithi118.stdout: "default_value": "False", 2024-03-24T03:17:23.472 INFO:teuthology.orchestra.run.smithi118.stdout: "min": "", 2024-03-24T03:17:23.472 INFO:teuthology.orchestra.run.smithi118.stdout: "max": "", 2024-03-24T03:17:23.472 INFO:teuthology.orchestra.run.smithi118.stdout: "enum_allowed": [], 2024-03-24T03:17:23.472 INFO:teuthology.orchestra.run.smithi118.stdout: "desc": "", 2024-03-24T03:17:23.472 INFO:teuthology.orchestra.run.smithi118.stdout: "long_desc": "", 2024-03-24T03:17:23.472 INFO:teuthology.orchestra.run.smithi118.stdout: "tags": [], 2024-03-24T03:17:23.472 INFO:teuthology.orchestra.run.smithi118.stdout: "see_also": [] 2024-03-24T03:17:23.472 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-24T03:17:23.473 INFO:teuthology.orchestra.run.smithi118.stdout: "log_to_cluster_level": { 2024-03-24T03:17:23.473 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "log_to_cluster_level", 2024-03-24T03:17:23.473 INFO:teuthology.orchestra.run.smithi118.stdout: "type": "str", 2024-03-24T03:17:23.473 INFO:teuthology.orchestra.run.smithi118.stdout: "level": "advanced", 2024-03-24T03:17:23.473 INFO:teuthology.orchestra.run.smithi118.stdout: "flags": 1, 2024-03-24T03:17:23.473 INFO:teuthology.orchestra.run.smithi118.stdout: "default_value": "info", 2024-03-24T03:17:23.473 INFO:teuthology.orchestra.run.smithi118.stdout: "min": "", 2024-03-24T03:17:23.473 INFO:teuthology.orchestra.run.smithi118.stdout: "max": "", 2024-03-24T03:17:23.474 INFO:teuthology.orchestra.run.smithi118.stdout: "enum_allowed": [ 2024-03-24T03:17:23.474 INFO:teuthology.orchestra.run.smithi118.stdout: "", 2024-03-24T03:17:23.474 INFO:teuthology.orchestra.run.smithi118.stdout: "critical", 2024-03-24T03:17:23.474 INFO:teuthology.orchestra.run.smithi118.stdout: "debug", 2024-03-24T03:17:23.474 INFO:teuthology.orchestra.run.smithi118.stdout: "error", 2024-03-24T03:17:23.474 INFO:teuthology.orchestra.run.smithi118.stdout: "info", 2024-03-24T03:17:23.474 INFO:teuthology.orchestra.run.smithi118.stdout: "warning" 2024-03-24T03:17:23.474 INFO:teuthology.orchestra.run.smithi118.stdout: ], 2024-03-24T03:17:23.475 INFO:teuthology.orchestra.run.smithi118.stdout: "desc": "", 2024-03-24T03:17:23.475 INFO:teuthology.orchestra.run.smithi118.stdout: "long_desc": "", 2024-03-24T03:17:23.475 INFO:teuthology.orchestra.run.smithi118.stdout: "tags": [], 2024-03-24T03:17:23.475 INFO:teuthology.orchestra.run.smithi118.stdout: "see_also": [] 2024-03-24T03:17:23.475 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-24T03:17:23.475 INFO:teuthology.orchestra.run.smithi118.stdout: "log_to_file": { 2024-03-24T03:17:23.475 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "log_to_file", 2024-03-24T03:17:23.475 INFO:teuthology.orchestra.run.smithi118.stdout: "type": "bool", 2024-03-24T03:17:23.475 INFO:teuthology.orchestra.run.smithi118.stdout: "level": "advanced", 2024-03-24T03:17:23.476 INFO:teuthology.orchestra.run.smithi118.stdout: "flags": 1, 2024-03-24T03:17:23.476 INFO:teuthology.orchestra.run.smithi118.stdout: "default_value": "False", 2024-03-24T03:17:23.476 INFO:teuthology.orchestra.run.smithi118.stdout: "min": "", 2024-03-24T03:17:23.476 INFO:teuthology.orchestra.run.smithi118.stdout: "max": "", 2024-03-24T03:17:23.476 INFO:teuthology.orchestra.run.smithi118.stdout: "enum_allowed": [], 2024-03-24T03:17:23.476 INFO:teuthology.orchestra.run.smithi118.stdout: "desc": "", 2024-03-24T03:17:23.476 INFO:teuthology.orchestra.run.smithi118.stdout: "long_desc": "", 2024-03-24T03:17:23.476 INFO:teuthology.orchestra.run.smithi118.stdout: "tags": [], 2024-03-24T03:17:23.477 INFO:teuthology.orchestra.run.smithi118.stdout: "see_also": [] 2024-03-24T03:17:23.477 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-24T03:17:23.477 INFO:teuthology.orchestra.run.smithi118.stdout: "predict_interval": { 2024-03-24T03:17:23.477 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "predict_interval", 2024-03-24T03:17:23.477 INFO:teuthology.orchestra.run.smithi118.stdout: "type": "str", 2024-03-24T03:17:23.477 INFO:teuthology.orchestra.run.smithi118.stdout: "level": "advanced", 2024-03-24T03:17:23.477 INFO:teuthology.orchestra.run.smithi118.stdout: "flags": 0, 2024-03-24T03:17:23.477 INFO:teuthology.orchestra.run.smithi118.stdout: "default_value": "86400", 2024-03-24T03:17:23.478 INFO:teuthology.orchestra.run.smithi118.stdout: "min": "", 2024-03-24T03:17:23.478 INFO:teuthology.orchestra.run.smithi118.stdout: "max": "", 2024-03-24T03:17:23.478 INFO:teuthology.orchestra.run.smithi118.stdout: "enum_allowed": [], 2024-03-24T03:17:23.478 INFO:teuthology.orchestra.run.smithi118.stdout: "desc": "", 2024-03-24T03:17:23.478 INFO:teuthology.orchestra.run.smithi118.stdout: "long_desc": "", 2024-03-24T03:17:23.478 INFO:teuthology.orchestra.run.smithi118.stdout: "tags": [], 2024-03-24T03:17:23.478 INFO:teuthology.orchestra.run.smithi118.stdout: "see_also": [] 2024-03-24T03:17:23.478 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-24T03:17:23.479 INFO:teuthology.orchestra.run.smithi118.stdout: "predictor_model": { 2024-03-24T03:17:23.479 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "predictor_model", 2024-03-24T03:17:23.479 INFO:teuthology.orchestra.run.smithi118.stdout: "type": "str", 2024-03-24T03:17:23.479 INFO:teuthology.orchestra.run.smithi118.stdout: "level": "advanced", 2024-03-24T03:17:23.479 INFO:teuthology.orchestra.run.smithi118.stdout: "flags": 0, 2024-03-24T03:17:23.479 INFO:teuthology.orchestra.run.smithi118.stdout: "default_value": "prophetstor", 2024-03-24T03:17:23.479 INFO:teuthology.orchestra.run.smithi118.stdout: "min": "", 2024-03-24T03:17:23.479 INFO:teuthology.orchestra.run.smithi118.stdout: "max": "", 2024-03-24T03:17:23.479 INFO:teuthology.orchestra.run.smithi118.stdout: "enum_allowed": [], 2024-03-24T03:17:23.480 INFO:teuthology.orchestra.run.smithi118.stdout: "desc": "", 2024-03-24T03:17:23.480 INFO:teuthology.orchestra.run.smithi118.stdout: "long_desc": "", 2024-03-24T03:17:23.480 INFO:teuthology.orchestra.run.smithi118.stdout: "tags": [], 2024-03-24T03:17:23.480 INFO:teuthology.orchestra.run.smithi118.stdout: "see_also": [] 2024-03-24T03:17:23.480 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-24T03:17:23.480 INFO:teuthology.orchestra.run.smithi118.stdout: "sleep_interval": { 2024-03-24T03:17:23.480 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "sleep_interval", 2024-03-24T03:17:23.480 INFO:teuthology.orchestra.run.smithi118.stdout: "type": "str", 2024-03-24T03:17:23.481 INFO:teuthology.orchestra.run.smithi118.stdout: "level": "advanced", 2024-03-24T03:17:23.481 INFO:teuthology.orchestra.run.smithi118.stdout: "flags": 0, 2024-03-24T03:17:23.481 INFO:teuthology.orchestra.run.smithi118.stdout: "default_value": "600", 2024-03-24T03:17:23.481 INFO:teuthology.orchestra.run.smithi118.stdout: "min": "", 2024-03-24T03:17:23.481 INFO:teuthology.orchestra.run.smithi118.stdout: "max": "", 2024-03-24T03:17:23.481 INFO:teuthology.orchestra.run.smithi118.stdout: "enum_allowed": [], 2024-03-24T03:17:23.481 INFO:teuthology.orchestra.run.smithi118.stdout: "desc": "", 2024-03-24T03:17:23.481 INFO:teuthology.orchestra.run.smithi118.stdout: "long_desc": "", 2024-03-24T03:17:23.481 INFO:teuthology.orchestra.run.smithi118.stdout: "tags": [], 2024-03-24T03:17:23.482 INFO:teuthology.orchestra.run.smithi118.stdout: "see_also": [] 2024-03-24T03:17:23.482 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-03-24T03:17:23.482 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-03-24T03:17:23.482 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-24T03:17:23.482 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-24T03:17:23.482 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "influx", 2024-03-24T03:17:23.482 INFO:teuthology.orchestra.run.smithi118.stdout: "can_run": false, 2024-03-24T03:17:23.483 INFO:teuthology.orchestra.run.smithi118.stdout: "error_string": "influxdb python module not found", 2024-03-24T03:17:23.483 INFO:teuthology.orchestra.run.smithi118.stdout: "module_options": { 2024-03-24T03:17:23.483 INFO:teuthology.orchestra.run.smithi118.stdout: "batch_size": { 2024-03-24T03:17:23.483 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "batch_size", 2024-03-24T03:17:23.483 INFO:teuthology.orchestra.run.smithi118.stdout: "type": "int", 2024-03-24T03:17:23.483 INFO:teuthology.orchestra.run.smithi118.stdout: "level": "advanced", 2024-03-24T03:17:23.484 INFO:teuthology.orchestra.run.smithi118.stdout: "flags": 0, 2024-03-24T03:17:23.484 INFO:teuthology.orchestra.run.smithi118.stdout: "default_value": "5000", 2024-03-24T03:17:23.484 INFO:teuthology.orchestra.run.smithi118.stdout: "min": "", 2024-03-24T03:17:23.484 INFO:teuthology.orchestra.run.smithi118.stdout: "max": "", 2024-03-24T03:17:23.484 INFO:teuthology.orchestra.run.smithi118.stdout: "enum_allowed": [], 2024-03-24T03:17:23.484 INFO:teuthology.orchestra.run.smithi118.stdout: "desc": "How big batches of data points should be when sending to InfluxDB.", 2024-03-24T03:17:23.485 INFO:teuthology.orchestra.run.smithi118.stdout: "long_desc": "", 2024-03-24T03:17:23.485 INFO:teuthology.orchestra.run.smithi118.stdout: "tags": [], 2024-03-24T03:17:23.485 INFO:teuthology.orchestra.run.smithi118.stdout: "see_also": [] 2024-03-24T03:17:23.485 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-24T03:17:23.485 INFO:teuthology.orchestra.run.smithi118.stdout: "database": { 2024-03-24T03:17:23.485 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "database", 2024-03-24T03:17:23.485 INFO:teuthology.orchestra.run.smithi118.stdout: "type": "str", 2024-03-24T03:17:23.486 INFO:teuthology.orchestra.run.smithi118.stdout: "level": "advanced", 2024-03-24T03:17:23.486 INFO:teuthology.orchestra.run.smithi118.stdout: "flags": 0, 2024-03-24T03:17:23.486 INFO:teuthology.orchestra.run.smithi118.stdout: "default_value": "ceph", 2024-03-24T03:17:23.486 INFO:teuthology.orchestra.run.smithi118.stdout: "min": "", 2024-03-24T03:17:23.486 INFO:teuthology.orchestra.run.smithi118.stdout: "max": "", 2024-03-24T03:17:23.487 INFO:teuthology.orchestra.run.smithi118.stdout: "enum_allowed": [], 2024-03-24T03:17:23.487 INFO:teuthology.orchestra.run.smithi118.stdout: "desc": "InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.", 2024-03-24T03:17:23.487 INFO:teuthology.orchestra.run.smithi118.stdout: "long_desc": "", 2024-03-24T03:17:23.487 INFO:teuthology.orchestra.run.smithi118.stdout: "tags": [], 2024-03-24T03:17:23.487 INFO:teuthology.orchestra.run.smithi118.stdout: "see_also": [] 2024-03-24T03:17:23.487 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-24T03:17:23.487 INFO:teuthology.orchestra.run.smithi118.stdout: "hostname": { 2024-03-24T03:17:23.488 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "hostname", 2024-03-24T03:17:23.488 INFO:teuthology.orchestra.run.smithi118.stdout: "type": "str", 2024-03-24T03:17:23.488 INFO:teuthology.orchestra.run.smithi118.stdout: "level": "advanced", 2024-03-24T03:17:23.488 INFO:teuthology.orchestra.run.smithi118.stdout: "flags": 0, 2024-03-24T03:17:23.488 INFO:teuthology.orchestra.run.smithi118.stdout: "default_value": "", 2024-03-24T03:17:23.488 INFO:teuthology.orchestra.run.smithi118.stdout: "min": "", 2024-03-24T03:17:23.489 INFO:teuthology.orchestra.run.smithi118.stdout: "max": "", 2024-03-24T03:17:23.489 INFO:teuthology.orchestra.run.smithi118.stdout: "enum_allowed": [], 2024-03-24T03:17:23.489 INFO:teuthology.orchestra.run.smithi118.stdout: "desc": "InfluxDB server hostname", 2024-03-24T03:17:23.489 INFO:teuthology.orchestra.run.smithi118.stdout: "long_desc": "", 2024-03-24T03:17:23.489 INFO:teuthology.orchestra.run.smithi118.stdout: "tags": [], 2024-03-24T03:17:23.489 INFO:teuthology.orchestra.run.smithi118.stdout: "see_also": [] 2024-03-24T03:17:23.489 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-24T03:17:23.490 INFO:teuthology.orchestra.run.smithi118.stdout: "interval": { 2024-03-24T03:17:23.490 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "interval", 2024-03-24T03:17:23.490 INFO:teuthology.orchestra.run.smithi118.stdout: "type": "secs", 2024-03-24T03:17:23.490 INFO:teuthology.orchestra.run.smithi118.stdout: "level": "advanced", 2024-03-24T03:17:23.490 INFO:teuthology.orchestra.run.smithi118.stdout: "flags": 0, 2024-03-24T03:17:23.490 INFO:teuthology.orchestra.run.smithi118.stdout: "default_value": "30", 2024-03-24T03:17:23.491 INFO:teuthology.orchestra.run.smithi118.stdout: "min": "5", 2024-03-24T03:17:23.491 INFO:teuthology.orchestra.run.smithi118.stdout: "max": "", 2024-03-24T03:17:23.491 INFO:teuthology.orchestra.run.smithi118.stdout: "enum_allowed": [], 2024-03-24T03:17:23.491 INFO:teuthology.orchestra.run.smithi118.stdout: "desc": "Time between reports to InfluxDB. Default 30 seconds.", 2024-03-24T03:17:23.491 INFO:teuthology.orchestra.run.smithi118.stdout: "long_desc": "", 2024-03-24T03:17:23.492 INFO:teuthology.orchestra.run.smithi118.stdout: "tags": [], 2024-03-24T03:17:23.492 INFO:teuthology.orchestra.run.smithi118.stdout: "see_also": [] 2024-03-24T03:17:23.492 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-24T03:17:23.492 INFO:teuthology.orchestra.run.smithi118.stdout: "log_level": { 2024-03-24T03:17:23.492 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "log_level", 2024-03-24T03:17:23.492 INFO:teuthology.orchestra.run.smithi118.stdout: "type": "str", 2024-03-24T03:17:23.493 INFO:teuthology.orchestra.run.smithi118.stdout: "level": "advanced", 2024-03-24T03:17:23.493 INFO:teuthology.orchestra.run.smithi118.stdout: "flags": 1, 2024-03-24T03:17:23.493 INFO:teuthology.orchestra.run.smithi118.stdout: "default_value": "", 2024-03-24T03:17:23.493 INFO:teuthology.orchestra.run.smithi118.stdout: "min": "", 2024-03-24T03:17:23.493 INFO:teuthology.orchestra.run.smithi118.stdout: "max": "", 2024-03-24T03:17:23.493 INFO:teuthology.orchestra.run.smithi118.stdout: "enum_allowed": [ 2024-03-24T03:17:23.493 INFO:teuthology.orchestra.run.smithi118.stdout: "", 2024-03-24T03:17:23.493 INFO:teuthology.orchestra.run.smithi118.stdout: "critical", 2024-03-24T03:17:23.493 INFO:teuthology.orchestra.run.smithi118.stdout: "debug", 2024-03-24T03:17:23.494 INFO:teuthology.orchestra.run.smithi118.stdout: "error", 2024-03-24T03:17:23.494 INFO:teuthology.orchestra.run.smithi118.stdout: "info", 2024-03-24T03:17:23.494 INFO:teuthology.orchestra.run.smithi118.stdout: "warning" 2024-03-24T03:17:23.494 INFO:teuthology.orchestra.run.smithi118.stdout: ], 2024-03-24T03:17:23.494 INFO:teuthology.orchestra.run.smithi118.stdout: "desc": "", 2024-03-24T03:17:23.494 INFO:teuthology.orchestra.run.smithi118.stdout: "long_desc": "", 2024-03-24T03:17:23.494 INFO:teuthology.orchestra.run.smithi118.stdout: "tags": [], 2024-03-24T03:17:23.494 INFO:teuthology.orchestra.run.smithi118.stdout: "see_also": [] 2024-03-24T03:17:23.495 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-24T03:17:23.495 INFO:teuthology.orchestra.run.smithi118.stdout: "log_to_cluster": { 2024-03-24T03:17:23.495 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "log_to_cluster", 2024-03-24T03:17:23.495 INFO:teuthology.orchestra.run.smithi118.stdout: "type": "bool", 2024-03-24T03:17:23.495 INFO:teuthology.orchestra.run.smithi118.stdout: "level": "advanced", 2024-03-24T03:17:23.495 INFO:teuthology.orchestra.run.smithi118.stdout: "flags": 1, 2024-03-24T03:17:23.495 INFO:teuthology.orchestra.run.smithi118.stdout: "default_value": "False", 2024-03-24T03:17:23.495 INFO:teuthology.orchestra.run.smithi118.stdout: "min": "", 2024-03-24T03:17:23.495 INFO:teuthology.orchestra.run.smithi118.stdout: "max": "", 2024-03-24T03:17:23.495 INFO:teuthology.orchestra.run.smithi118.stdout: "enum_allowed": [], 2024-03-24T03:17:23.496 INFO:teuthology.orchestra.run.smithi118.stdout: "desc": "", 2024-03-24T03:17:23.496 INFO:teuthology.orchestra.run.smithi118.stdout: "long_desc": "", 2024-03-24T03:17:23.496 INFO:teuthology.orchestra.run.smithi118.stdout: "tags": [], 2024-03-24T03:17:23.496 INFO:teuthology.orchestra.run.smithi118.stdout: "see_also": [] 2024-03-24T03:17:23.496 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-24T03:17:23.496 INFO:teuthology.orchestra.run.smithi118.stdout: "log_to_cluster_level": { 2024-03-24T03:17:23.496 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "log_to_cluster_level", 2024-03-24T03:17:23.496 INFO:teuthology.orchestra.run.smithi118.stdout: "type": "str", 2024-03-24T03:17:23.496 INFO:teuthology.orchestra.run.smithi118.stdout: "level": "advanced", 2024-03-24T03:17:23.497 INFO:teuthology.orchestra.run.smithi118.stdout: "flags": 1, 2024-03-24T03:17:23.497 INFO:teuthology.orchestra.run.smithi118.stdout: "default_value": "info", 2024-03-24T03:17:23.497 INFO:teuthology.orchestra.run.smithi118.stdout: "min": "", 2024-03-24T03:17:23.497 INFO:teuthology.orchestra.run.smithi118.stdout: "max": "", 2024-03-24T03:17:23.497 INFO:teuthology.orchestra.run.smithi118.stdout: "enum_allowed": [ 2024-03-24T03:17:23.497 INFO:teuthology.orchestra.run.smithi118.stdout: "", 2024-03-24T03:17:23.497 INFO:teuthology.orchestra.run.smithi118.stdout: "critical", 2024-03-24T03:17:23.497 INFO:teuthology.orchestra.run.smithi118.stdout: "debug", 2024-03-24T03:17:23.497 INFO:teuthology.orchestra.run.smithi118.stdout: "error", 2024-03-24T03:17:23.497 INFO:teuthology.orchestra.run.smithi118.stdout: "info", 2024-03-24T03:17:23.498 INFO:teuthology.orchestra.run.smithi118.stdout: "warning" 2024-03-24T03:17:23.498 INFO:teuthology.orchestra.run.smithi118.stdout: ], 2024-03-24T03:17:23.498 INFO:teuthology.orchestra.run.smithi118.stdout: "desc": "", 2024-03-24T03:17:23.498 INFO:teuthology.orchestra.run.smithi118.stdout: "long_desc": "", 2024-03-24T03:17:23.498 INFO:teuthology.orchestra.run.smithi118.stdout: "tags": [], 2024-03-24T03:17:23.498 INFO:teuthology.orchestra.run.smithi118.stdout: "see_also": [] 2024-03-24T03:17:23.498 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-24T03:17:23.499 INFO:teuthology.orchestra.run.smithi118.stdout: "log_to_file": { 2024-03-24T03:17:23.499 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "log_to_file", 2024-03-24T03:17:23.499 INFO:teuthology.orchestra.run.smithi118.stdout: "type": "bool", 2024-03-24T03:17:23.499 INFO:teuthology.orchestra.run.smithi118.stdout: "level": "advanced", 2024-03-24T03:17:23.499 INFO:teuthology.orchestra.run.smithi118.stdout: "flags": 1, 2024-03-24T03:17:23.499 INFO:teuthology.orchestra.run.smithi118.stdout: "default_value": "False", 2024-03-24T03:17:23.499 INFO:teuthology.orchestra.run.smithi118.stdout: "min": "", 2024-03-24T03:17:23.499 INFO:teuthology.orchestra.run.smithi118.stdout: "max": "", 2024-03-24T03:17:23.499 INFO:teuthology.orchestra.run.smithi118.stdout: "enum_allowed": [], 2024-03-24T03:17:23.500 INFO:teuthology.orchestra.run.smithi118.stdout: "desc": "", 2024-03-24T03:17:23.500 INFO:teuthology.orchestra.run.smithi118.stdout: "long_desc": "", 2024-03-24T03:17:23.500 INFO:teuthology.orchestra.run.smithi118.stdout: "tags": [], 2024-03-24T03:17:23.500 INFO:teuthology.orchestra.run.smithi118.stdout: "see_also": [] 2024-03-24T03:17:23.500 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-24T03:17:23.500 INFO:teuthology.orchestra.run.smithi118.stdout: "password": { 2024-03-24T03:17:23.500 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "password", 2024-03-24T03:17:23.500 INFO:teuthology.orchestra.run.smithi118.stdout: "type": "str", 2024-03-24T03:17:23.500 INFO:teuthology.orchestra.run.smithi118.stdout: "level": "advanced", 2024-03-24T03:17:23.501 INFO:teuthology.orchestra.run.smithi118.stdout: "flags": 0, 2024-03-24T03:17:23.501 INFO:teuthology.orchestra.run.smithi118.stdout: "default_value": "", 2024-03-24T03:17:23.501 INFO:teuthology.orchestra.run.smithi118.stdout: "min": "", 2024-03-24T03:17:23.501 INFO:teuthology.orchestra.run.smithi118.stdout: "max": "", 2024-03-24T03:17:23.501 INFO:teuthology.orchestra.run.smithi118.stdout: "enum_allowed": [], 2024-03-24T03:17:23.501 INFO:teuthology.orchestra.run.smithi118.stdout: "desc": "password of InfluxDB server user", 2024-03-24T03:17:23.501 INFO:teuthology.orchestra.run.smithi118.stdout: "long_desc": "", 2024-03-24T03:17:23.501 INFO:teuthology.orchestra.run.smithi118.stdout: "tags": [], 2024-03-24T03:17:23.501 INFO:teuthology.orchestra.run.smithi118.stdout: "see_also": [] 2024-03-24T03:17:23.501 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-24T03:17:23.502 INFO:teuthology.orchestra.run.smithi118.stdout: "port": { 2024-03-24T03:17:23.502 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "port", 2024-03-24T03:17:23.502 INFO:teuthology.orchestra.run.smithi118.stdout: "type": "int", 2024-03-24T03:17:23.502 INFO:teuthology.orchestra.run.smithi118.stdout: "level": "advanced", 2024-03-24T03:17:23.502 INFO:teuthology.orchestra.run.smithi118.stdout: "flags": 0, 2024-03-24T03:17:23.502 INFO:teuthology.orchestra.run.smithi118.stdout: "default_value": "8086", 2024-03-24T03:17:23.502 INFO:teuthology.orchestra.run.smithi118.stdout: "min": "", 2024-03-24T03:17:23.502 INFO:teuthology.orchestra.run.smithi118.stdout: "max": "", 2024-03-24T03:17:23.503 INFO:teuthology.orchestra.run.smithi118.stdout: "enum_allowed": [], 2024-03-24T03:17:23.503 INFO:teuthology.orchestra.run.smithi118.stdout: "desc": "InfluxDB server port", 2024-03-24T03:17:23.503 INFO:teuthology.orchestra.run.smithi118.stdout: "long_desc": "", 2024-03-24T03:17:23.503 INFO:teuthology.orchestra.run.smithi118.stdout: "tags": [], 2024-03-24T03:17:23.503 INFO:teuthology.orchestra.run.smithi118.stdout: "see_also": [] 2024-03-24T03:17:23.503 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-24T03:17:23.503 INFO:teuthology.orchestra.run.smithi118.stdout: "ssl": { 2024-03-24T03:17:23.503 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "ssl", 2024-03-24T03:17:23.503 INFO:teuthology.orchestra.run.smithi118.stdout: "type": "str", 2024-03-24T03:17:23.504 INFO:teuthology.orchestra.run.smithi118.stdout: "level": "advanced", 2024-03-24T03:17:23.504 INFO:teuthology.orchestra.run.smithi118.stdout: "flags": 0, 2024-03-24T03:17:23.504 INFO:teuthology.orchestra.run.smithi118.stdout: "default_value": "false", 2024-03-24T03:17:23.504 INFO:teuthology.orchestra.run.smithi118.stdout: "min": "", 2024-03-24T03:17:23.504 INFO:teuthology.orchestra.run.smithi118.stdout: "max": "", 2024-03-24T03:17:23.504 INFO:teuthology.orchestra.run.smithi118.stdout: "enum_allowed": [], 2024-03-24T03:17:23.504 INFO:teuthology.orchestra.run.smithi118.stdout: "desc": "Use https connection for InfluxDB server. Use \"true\" or \"false\".", 2024-03-24T03:17:23.504 INFO:teuthology.orchestra.run.smithi118.stdout: "long_desc": "", 2024-03-24T03:17:23.505 INFO:teuthology.orchestra.run.smithi118.stdout: "tags": [], 2024-03-24T03:17:23.505 INFO:teuthology.orchestra.run.smithi118.stdout: "see_also": [] 2024-03-24T03:17:23.505 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-24T03:17:23.505 INFO:teuthology.orchestra.run.smithi118.stdout: "threads": { 2024-03-24T03:17:23.505 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "threads", 2024-03-24T03:17:23.505 INFO:teuthology.orchestra.run.smithi118.stdout: "type": "int", 2024-03-24T03:17:23.505 INFO:teuthology.orchestra.run.smithi118.stdout: "level": "advanced", 2024-03-24T03:17:23.505 INFO:teuthology.orchestra.run.smithi118.stdout: "flags": 0, 2024-03-24T03:17:23.505 INFO:teuthology.orchestra.run.smithi118.stdout: "default_value": "5", 2024-03-24T03:17:23.506 INFO:teuthology.orchestra.run.smithi118.stdout: "min": "1", 2024-03-24T03:17:23.506 INFO:teuthology.orchestra.run.smithi118.stdout: "max": "32", 2024-03-24T03:17:23.506 INFO:teuthology.orchestra.run.smithi118.stdout: "enum_allowed": [], 2024-03-24T03:17:23.506 INFO:teuthology.orchestra.run.smithi118.stdout: "desc": "How many worker threads should be spawned for sending data to InfluxDB.", 2024-03-24T03:17:23.506 INFO:teuthology.orchestra.run.smithi118.stdout: "long_desc": "", 2024-03-24T03:17:23.506 INFO:teuthology.orchestra.run.smithi118.stdout: "tags": [], 2024-03-24T03:17:23.506 INFO:teuthology.orchestra.run.smithi118.stdout: "see_also": [] 2024-03-24T03:17:23.506 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-24T03:17:23.507 INFO:teuthology.orchestra.run.smithi118.stdout: "username": { 2024-03-24T03:17:23.507 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "username", 2024-03-24T03:17:23.507 INFO:teuthology.orchestra.run.smithi118.stdout: "type": "str", 2024-03-24T03:17:23.507 INFO:teuthology.orchestra.run.smithi118.stdout: "level": "advanced", 2024-03-24T03:17:23.507 INFO:teuthology.orchestra.run.smithi118.stdout: "flags": 0, 2024-03-24T03:17:23.507 INFO:teuthology.orchestra.run.smithi118.stdout: "default_value": "", 2024-03-24T03:17:23.507 INFO:teuthology.orchestra.run.smithi118.stdout: "min": "", 2024-03-24T03:17:23.507 INFO:teuthology.orchestra.run.smithi118.stdout: "max": "", 2024-03-24T03:17:23.507 INFO:teuthology.orchestra.run.smithi118.stdout: "enum_allowed": [], 2024-03-24T03:17:23.507 INFO:teuthology.orchestra.run.smithi118.stdout: "desc": "username of InfluxDB server user", 2024-03-24T03:17:23.508 INFO:teuthology.orchestra.run.smithi118.stdout: "long_desc": "", 2024-03-24T03:17:23.508 INFO:teuthology.orchestra.run.smithi118.stdout: "tags": [], 2024-03-24T03:17:23.508 INFO:teuthology.orchestra.run.smithi118.stdout: "see_also": [] 2024-03-24T03:17:23.508 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-24T03:17:23.508 INFO:teuthology.orchestra.run.smithi118.stdout: "verify_ssl": { 2024-03-24T03:17:23.508 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "verify_ssl", 2024-03-24T03:17:23.508 INFO:teuthology.orchestra.run.smithi118.stdout: "type": "str", 2024-03-24T03:17:23.508 INFO:teuthology.orchestra.run.smithi118.stdout: "level": "advanced", 2024-03-24T03:17:23.508 INFO:teuthology.orchestra.run.smithi118.stdout: "flags": 0, 2024-03-24T03:17:23.509 INFO:teuthology.orchestra.run.smithi118.stdout: "default_value": "true", 2024-03-24T03:17:23.509 INFO:teuthology.orchestra.run.smithi118.stdout: "min": "", 2024-03-24T03:17:23.509 INFO:teuthology.orchestra.run.smithi118.stdout: "max": "", 2024-03-24T03:17:23.509 INFO:teuthology.orchestra.run.smithi118.stdout: "enum_allowed": [], 2024-03-24T03:17:23.509 INFO:teuthology.orchestra.run.smithi118.stdout: "desc": "Verify https cert for InfluxDB server. Use \"true\" or \"false\".", 2024-03-24T03:17:23.509 INFO:teuthology.orchestra.run.smithi118.stdout: "long_desc": "", 2024-03-24T03:17:23.509 INFO:teuthology.orchestra.run.smithi118.stdout: "tags": [], 2024-03-24T03:17:23.509 INFO:teuthology.orchestra.run.smithi118.stdout: "see_also": [] 2024-03-24T03:17:23.509 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-03-24T03:17:23.509 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-03-24T03:17:23.510 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-24T03:17:23.510 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-24T03:17:23.510 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "insights", 2024-03-24T03:17:23.510 INFO:teuthology.orchestra.run.smithi118.stdout: "can_run": true, 2024-03-24T03:17:23.510 INFO:teuthology.orchestra.run.smithi118.stdout: "error_string": "", 2024-03-24T03:17:23.510 INFO:teuthology.orchestra.run.smithi118.stdout: "module_options": { 2024-03-24T03:17:23.510 INFO:teuthology.orchestra.run.smithi118.stdout: "log_level": { 2024-03-24T03:17:23.510 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "log_level", 2024-03-24T03:17:23.511 INFO:teuthology.orchestra.run.smithi118.stdout: "type": "str", 2024-03-24T03:17:23.511 INFO:teuthology.orchestra.run.smithi118.stdout: "level": "advanced", 2024-03-24T03:17:23.511 INFO:teuthology.orchestra.run.smithi118.stdout: "flags": 1, 2024-03-24T03:17:23.511 INFO:teuthology.orchestra.run.smithi118.stdout: "default_value": "", 2024-03-24T03:17:23.511 INFO:teuthology.orchestra.run.smithi118.stdout: "min": "", 2024-03-24T03:17:23.511 INFO:teuthology.orchestra.run.smithi118.stdout: "max": "", 2024-03-24T03:17:23.511 INFO:teuthology.orchestra.run.smithi118.stdout: "enum_allowed": [ 2024-03-24T03:17:23.511 INFO:teuthology.orchestra.run.smithi118.stdout: "", 2024-03-24T03:17:23.511 INFO:teuthology.orchestra.run.smithi118.stdout: "critical", 2024-03-24T03:17:23.511 INFO:teuthology.orchestra.run.smithi118.stdout: "debug", 2024-03-24T03:17:23.512 INFO:teuthology.orchestra.run.smithi118.stdout: "error", 2024-03-24T03:17:23.512 INFO:teuthology.orchestra.run.smithi118.stdout: "info", 2024-03-24T03:17:23.512 INFO:teuthology.orchestra.run.smithi118.stdout: "warning" 2024-03-24T03:17:23.512 INFO:teuthology.orchestra.run.smithi118.stdout: ], 2024-03-24T03:17:23.512 INFO:teuthology.orchestra.run.smithi118.stdout: "desc": "", 2024-03-24T03:17:23.512 INFO:teuthology.orchestra.run.smithi118.stdout: "long_desc": "", 2024-03-24T03:17:23.512 INFO:teuthology.orchestra.run.smithi118.stdout: "tags": [], 2024-03-24T03:17:23.512 INFO:teuthology.orchestra.run.smithi118.stdout: "see_also": [] 2024-03-24T03:17:23.513 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-24T03:17:23.513 INFO:teuthology.orchestra.run.smithi118.stdout: "log_to_cluster": { 2024-03-24T03:17:23.513 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "log_to_cluster", 2024-03-24T03:17:23.513 INFO:teuthology.orchestra.run.smithi118.stdout: "type": "bool", 2024-03-24T03:17:23.513 INFO:teuthology.orchestra.run.smithi118.stdout: "level": "advanced", 2024-03-24T03:17:23.513 INFO:teuthology.orchestra.run.smithi118.stdout: "flags": 1, 2024-03-24T03:17:23.513 INFO:teuthology.orchestra.run.smithi118.stdout: "default_value": "False", 2024-03-24T03:17:23.514 INFO:teuthology.orchestra.run.smithi118.stdout: "min": "", 2024-03-24T03:17:23.514 INFO:teuthology.orchestra.run.smithi118.stdout: "max": "", 2024-03-24T03:17:23.514 INFO:teuthology.orchestra.run.smithi118.stdout: "enum_allowed": [], 2024-03-24T03:17:23.514 INFO:teuthology.orchestra.run.smithi118.stdout: "desc": "", 2024-03-24T03:17:23.514 INFO:teuthology.orchestra.run.smithi118.stdout: "long_desc": "", 2024-03-24T03:17:23.514 INFO:teuthology.orchestra.run.smithi118.stdout: "tags": [], 2024-03-24T03:17:23.514 INFO:teuthology.orchestra.run.smithi118.stdout: "see_also": [] 2024-03-24T03:17:23.514 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-24T03:17:23.515 INFO:teuthology.orchestra.run.smithi118.stdout: "log_to_cluster_level": { 2024-03-24T03:17:23.515 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "log_to_cluster_level", 2024-03-24T03:17:23.515 INFO:teuthology.orchestra.run.smithi118.stdout: "type": "str", 2024-03-24T03:17:23.515 INFO:teuthology.orchestra.run.smithi118.stdout: "level": "advanced", 2024-03-24T03:17:23.515 INFO:teuthology.orchestra.run.smithi118.stdout: "flags": 1, 2024-03-24T03:17:23.515 INFO:teuthology.orchestra.run.smithi118.stdout: "default_value": "info", 2024-03-24T03:17:23.515 INFO:teuthology.orchestra.run.smithi118.stdout: "min": "", 2024-03-24T03:17:23.515 INFO:teuthology.orchestra.run.smithi118.stdout: "max": "", 2024-03-24T03:17:23.515 INFO:teuthology.orchestra.run.smithi118.stdout: "enum_allowed": [ 2024-03-24T03:17:23.516 INFO:teuthology.orchestra.run.smithi118.stdout: "", 2024-03-24T03:17:23.516 INFO:teuthology.orchestra.run.smithi118.stdout: "critical", 2024-03-24T03:17:23.516 INFO:teuthology.orchestra.run.smithi118.stdout: "debug", 2024-03-24T03:17:23.516 INFO:teuthology.orchestra.run.smithi118.stdout: "error", 2024-03-24T03:17:23.516 INFO:teuthology.orchestra.run.smithi118.stdout: "info", 2024-03-24T03:17:23.516 INFO:teuthology.orchestra.run.smithi118.stdout: "warning" 2024-03-24T03:17:23.516 INFO:teuthology.orchestra.run.smithi118.stdout: ], 2024-03-24T03:17:23.516 INFO:teuthology.orchestra.run.smithi118.stdout: "desc": "", 2024-03-24T03:17:23.517 INFO:teuthology.orchestra.run.smithi118.stdout: "long_desc": "", 2024-03-24T03:17:23.517 INFO:teuthology.orchestra.run.smithi118.stdout: "tags": [], 2024-03-24T03:17:23.517 INFO:teuthology.orchestra.run.smithi118.stdout: "see_also": [] 2024-03-24T03:17:23.517 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-24T03:17:23.517 INFO:teuthology.orchestra.run.smithi118.stdout: "log_to_file": { 2024-03-24T03:17:23.517 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "log_to_file", 2024-03-24T03:17:23.517 INFO:teuthology.orchestra.run.smithi118.stdout: "type": "bool", 2024-03-24T03:17:23.518 INFO:teuthology.orchestra.run.smithi118.stdout: "level": "advanced", 2024-03-24T03:17:23.518 INFO:teuthology.orchestra.run.smithi118.stdout: "flags": 1, 2024-03-24T03:17:23.518 INFO:teuthology.orchestra.run.smithi118.stdout: "default_value": "False", 2024-03-24T03:17:23.518 INFO:teuthology.orchestra.run.smithi118.stdout: "min": "", 2024-03-24T03:17:23.518 INFO:teuthology.orchestra.run.smithi118.stdout: "max": "", 2024-03-24T03:17:23.518 INFO:teuthology.orchestra.run.smithi118.stdout: "enum_allowed": [], 2024-03-24T03:17:23.518 INFO:teuthology.orchestra.run.smithi118.stdout: "desc": "", 2024-03-24T03:17:23.518 INFO:teuthology.orchestra.run.smithi118.stdout: "long_desc": "", 2024-03-24T03:17:23.519 INFO:teuthology.orchestra.run.smithi118.stdout: "tags": [], 2024-03-24T03:17:23.519 INFO:teuthology.orchestra.run.smithi118.stdout: "see_also": [] 2024-03-24T03:17:23.519 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-03-24T03:17:23.519 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-03-24T03:17:23.519 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-24T03:17:23.519 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-24T03:17:23.519 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "iostat", 2024-03-24T03:17:23.519 INFO:teuthology.orchestra.run.smithi118.stdout: "can_run": true, 2024-03-24T03:17:23.519 INFO:teuthology.orchestra.run.smithi118.stdout: "error_string": "", 2024-03-24T03:17:23.519 INFO:teuthology.orchestra.run.smithi118.stdout: "module_options": { 2024-03-24T03:17:23.520 INFO:teuthology.orchestra.run.smithi118.stdout: "log_level": { 2024-03-24T03:17:23.520 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "log_level", 2024-03-24T03:17:23.520 INFO:teuthology.orchestra.run.smithi118.stdout: "type": "str", 2024-03-24T03:17:23.520 INFO:teuthology.orchestra.run.smithi118.stdout: "level": "advanced", 2024-03-24T03:17:23.520 INFO:teuthology.orchestra.run.smithi118.stdout: "flags": 1, 2024-03-24T03:17:23.520 INFO:teuthology.orchestra.run.smithi118.stdout: "default_value": "", 2024-03-24T03:17:23.520 INFO:teuthology.orchestra.run.smithi118.stdout: "min": "", 2024-03-24T03:17:23.520 INFO:teuthology.orchestra.run.smithi118.stdout: "max": "", 2024-03-24T03:17:23.520 INFO:teuthology.orchestra.run.smithi118.stdout: "enum_allowed": [ 2024-03-24T03:17:23.521 INFO:teuthology.orchestra.run.smithi118.stdout: "", 2024-03-24T03:17:23.521 INFO:teuthology.orchestra.run.smithi118.stdout: "critical", 2024-03-24T03:17:23.521 INFO:teuthology.orchestra.run.smithi118.stdout: "debug", 2024-03-24T03:17:23.521 INFO:teuthology.orchestra.run.smithi118.stdout: "error", 2024-03-24T03:17:23.521 INFO:teuthology.orchestra.run.smithi118.stdout: "info", 2024-03-24T03:17:23.521 INFO:teuthology.orchestra.run.smithi118.stdout: "warning" 2024-03-24T03:17:23.521 INFO:teuthology.orchestra.run.smithi118.stdout: ], 2024-03-24T03:17:23.521 INFO:teuthology.orchestra.run.smithi118.stdout: "desc": "", 2024-03-24T03:17:23.522 INFO:teuthology.orchestra.run.smithi118.stdout: "long_desc": "", 2024-03-24T03:17:23.522 INFO:teuthology.orchestra.run.smithi118.stdout: "tags": [], 2024-03-24T03:17:23.522 INFO:teuthology.orchestra.run.smithi118.stdout: "see_also": [] 2024-03-24T03:17:23.522 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-24T03:17:23.522 INFO:teuthology.orchestra.run.smithi118.stdout: "log_to_cluster": { 2024-03-24T03:17:23.522 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "log_to_cluster", 2024-03-24T03:17:23.522 INFO:teuthology.orchestra.run.smithi118.stdout: "type": "bool", 2024-03-24T03:17:23.522 INFO:teuthology.orchestra.run.smithi118.stdout: "level": "advanced", 2024-03-24T03:17:23.522 INFO:teuthology.orchestra.run.smithi118.stdout: "flags": 1, 2024-03-24T03:17:23.523 INFO:teuthology.orchestra.run.smithi118.stdout: "default_value": "False", 2024-03-24T03:17:23.523 INFO:teuthology.orchestra.run.smithi118.stdout: "min": "", 2024-03-24T03:17:23.523 INFO:teuthology.orchestra.run.smithi118.stdout: "max": "", 2024-03-24T03:17:23.523 INFO:teuthology.orchestra.run.smithi118.stdout: "enum_allowed": [], 2024-03-24T03:17:23.523 INFO:teuthology.orchestra.run.smithi118.stdout: "desc": "", 2024-03-24T03:17:23.523 INFO:teuthology.orchestra.run.smithi118.stdout: "long_desc": "", 2024-03-24T03:17:23.523 INFO:teuthology.orchestra.run.smithi118.stdout: "tags": [], 2024-03-24T03:17:23.523 INFO:teuthology.orchestra.run.smithi118.stdout: "see_also": [] 2024-03-24T03:17:23.523 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-24T03:17:23.524 INFO:teuthology.orchestra.run.smithi118.stdout: "log_to_cluster_level": { 2024-03-24T03:17:23.524 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "log_to_cluster_level", 2024-03-24T03:17:23.524 INFO:teuthology.orchestra.run.smithi118.stdout: "type": "str", 2024-03-24T03:17:23.524 INFO:teuthology.orchestra.run.smithi118.stdout: "level": "advanced", 2024-03-24T03:17:23.524 INFO:teuthology.orchestra.run.smithi118.stdout: "flags": 1, 2024-03-24T03:17:23.524 INFO:teuthology.orchestra.run.smithi118.stdout: "default_value": "info", 2024-03-24T03:17:23.524 INFO:teuthology.orchestra.run.smithi118.stdout: "min": "", 2024-03-24T03:17:23.524 INFO:teuthology.orchestra.run.smithi118.stdout: "max": "", 2024-03-24T03:17:23.524 INFO:teuthology.orchestra.run.smithi118.stdout: "enum_allowed": [ 2024-03-24T03:17:23.525 INFO:teuthology.orchestra.run.smithi118.stdout: "", 2024-03-24T03:17:23.525 INFO:teuthology.orchestra.run.smithi118.stdout: "critical", 2024-03-24T03:17:23.525 INFO:teuthology.orchestra.run.smithi118.stdout: "debug", 2024-03-24T03:17:23.525 INFO:teuthology.orchestra.run.smithi118.stdout: "error", 2024-03-24T03:17:23.525 INFO:teuthology.orchestra.run.smithi118.stdout: "info", 2024-03-24T03:17:23.525 INFO:teuthology.orchestra.run.smithi118.stdout: "warning" 2024-03-24T03:17:23.525 INFO:teuthology.orchestra.run.smithi118.stdout: ], 2024-03-24T03:17:23.525 INFO:teuthology.orchestra.run.smithi118.stdout: "desc": "", 2024-03-24T03:17:23.525 INFO:teuthology.orchestra.run.smithi118.stdout: "long_desc": "", 2024-03-24T03:17:23.525 INFO:teuthology.orchestra.run.smithi118.stdout: "tags": [], 2024-03-24T03:17:23.526 INFO:teuthology.orchestra.run.smithi118.stdout: "see_also": [] 2024-03-24T03:17:23.526 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-24T03:17:23.526 INFO:teuthology.orchestra.run.smithi118.stdout: "log_to_file": { 2024-03-24T03:17:23.526 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "log_to_file", 2024-03-24T03:17:23.526 INFO:teuthology.orchestra.run.smithi118.stdout: "type": "bool", 2024-03-24T03:17:23.526 INFO:teuthology.orchestra.run.smithi118.stdout: "level": "advanced", 2024-03-24T03:17:23.526 INFO:teuthology.orchestra.run.smithi118.stdout: "flags": 1, 2024-03-24T03:17:23.526 INFO:teuthology.orchestra.run.smithi118.stdout: "default_value": "False", 2024-03-24T03:17:23.527 INFO:teuthology.orchestra.run.smithi118.stdout: "min": "", 2024-03-24T03:17:23.527 INFO:teuthology.orchestra.run.smithi118.stdout: "max": "", 2024-03-24T03:17:23.527 INFO:teuthology.orchestra.run.smithi118.stdout: "enum_allowed": [], 2024-03-24T03:17:23.527 INFO:teuthology.orchestra.run.smithi118.stdout: "desc": "", 2024-03-24T03:17:23.527 INFO:teuthology.orchestra.run.smithi118.stdout: "long_desc": "", 2024-03-24T03:17:23.527 INFO:teuthology.orchestra.run.smithi118.stdout: "tags": [], 2024-03-24T03:17:23.527 INFO:teuthology.orchestra.run.smithi118.stdout: "see_also": [] 2024-03-24T03:17:23.527 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-03-24T03:17:23.527 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-03-24T03:17:23.527 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-24T03:17:23.528 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-24T03:17:23.528 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "k8sevents", 2024-03-24T03:17:23.528 INFO:teuthology.orchestra.run.smithi118.stdout: "can_run": true, 2024-03-24T03:17:23.528 INFO:teuthology.orchestra.run.smithi118.stdout: "error_string": "", 2024-03-24T03:17:23.528 INFO:teuthology.orchestra.run.smithi118.stdout: "module_options": { 2024-03-24T03:17:23.528 INFO:teuthology.orchestra.run.smithi118.stdout: "ceph_event_retention_days": { 2024-03-24T03:17:23.528 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "ceph_event_retention_days", 2024-03-24T03:17:23.528 INFO:teuthology.orchestra.run.smithi118.stdout: "type": "int", 2024-03-24T03:17:23.528 INFO:teuthology.orchestra.run.smithi118.stdout: "level": "advanced", 2024-03-24T03:17:23.529 INFO:teuthology.orchestra.run.smithi118.stdout: "flags": 0, 2024-03-24T03:17:23.529 INFO:teuthology.orchestra.run.smithi118.stdout: "default_value": "7", 2024-03-24T03:17:23.529 INFO:teuthology.orchestra.run.smithi118.stdout: "min": "", 2024-03-24T03:17:23.529 INFO:teuthology.orchestra.run.smithi118.stdout: "max": "", 2024-03-24T03:17:23.529 INFO:teuthology.orchestra.run.smithi118.stdout: "enum_allowed": [], 2024-03-24T03:17:23.529 INFO:teuthology.orchestra.run.smithi118.stdout: "desc": "Days to hold ceph event information within local cache", 2024-03-24T03:17:23.529 INFO:teuthology.orchestra.run.smithi118.stdout: "long_desc": "", 2024-03-24T03:17:23.529 INFO:teuthology.orchestra.run.smithi118.stdout: "tags": [], 2024-03-24T03:17:23.529 INFO:teuthology.orchestra.run.smithi118.stdout: "see_also": [] 2024-03-24T03:17:23.529 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-24T03:17:23.530 INFO:teuthology.orchestra.run.smithi118.stdout: "config_check_secs": { 2024-03-24T03:17:23.530 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "config_check_secs", 2024-03-24T03:17:23.530 INFO:teuthology.orchestra.run.smithi118.stdout: "type": "int", 2024-03-24T03:17:23.530 INFO:teuthology.orchestra.run.smithi118.stdout: "level": "advanced", 2024-03-24T03:17:23.530 INFO:teuthology.orchestra.run.smithi118.stdout: "flags": 0, 2024-03-24T03:17:23.530 INFO:teuthology.orchestra.run.smithi118.stdout: "default_value": "10", 2024-03-24T03:17:23.530 INFO:teuthology.orchestra.run.smithi118.stdout: "min": "10", 2024-03-24T03:17:23.530 INFO:teuthology.orchestra.run.smithi118.stdout: "max": "", 2024-03-24T03:17:23.531 INFO:teuthology.orchestra.run.smithi118.stdout: "enum_allowed": [], 2024-03-24T03:17:23.531 INFO:teuthology.orchestra.run.smithi118.stdout: "desc": "interval (secs) to check for cluster configuration changes", 2024-03-24T03:17:23.531 INFO:teuthology.orchestra.run.smithi118.stdout: "long_desc": "", 2024-03-24T03:17:23.531 INFO:teuthology.orchestra.run.smithi118.stdout: "tags": [], 2024-03-24T03:17:23.531 INFO:teuthology.orchestra.run.smithi118.stdout: "see_also": [] 2024-03-24T03:17:23.531 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-24T03:17:23.531 INFO:teuthology.orchestra.run.smithi118.stdout: "log_level": { 2024-03-24T03:17:23.531 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "log_level", 2024-03-24T03:17:23.531 INFO:teuthology.orchestra.run.smithi118.stdout: "type": "str", 2024-03-24T03:17:23.531 INFO:teuthology.orchestra.run.smithi118.stdout: "level": "advanced", 2024-03-24T03:17:23.532 INFO:teuthology.orchestra.run.smithi118.stdout: "flags": 1, 2024-03-24T03:17:23.532 INFO:teuthology.orchestra.run.smithi118.stdout: "default_value": "", 2024-03-24T03:17:23.532 INFO:teuthology.orchestra.run.smithi118.stdout: "min": "", 2024-03-24T03:17:23.532 INFO:teuthology.orchestra.run.smithi118.stdout: "max": "", 2024-03-24T03:17:23.532 INFO:teuthology.orchestra.run.smithi118.stdout: "enum_allowed": [ 2024-03-24T03:17:23.532 INFO:teuthology.orchestra.run.smithi118.stdout: "", 2024-03-24T03:17:23.532 INFO:teuthology.orchestra.run.smithi118.stdout: "critical", 2024-03-24T03:17:23.532 INFO:teuthology.orchestra.run.smithi118.stdout: "debug", 2024-03-24T03:17:23.532 INFO:teuthology.orchestra.run.smithi118.stdout: "error", 2024-03-24T03:17:23.533 INFO:teuthology.orchestra.run.smithi118.stdout: "info", 2024-03-24T03:17:23.533 INFO:teuthology.orchestra.run.smithi118.stdout: "warning" 2024-03-24T03:17:23.533 INFO:teuthology.orchestra.run.smithi118.stdout: ], 2024-03-24T03:17:23.533 INFO:teuthology.orchestra.run.smithi118.stdout: "desc": "", 2024-03-24T03:17:23.533 INFO:teuthology.orchestra.run.smithi118.stdout: "long_desc": "", 2024-03-24T03:17:23.533 INFO:teuthology.orchestra.run.smithi118.stdout: "tags": [], 2024-03-24T03:17:23.533 INFO:teuthology.orchestra.run.smithi118.stdout: "see_also": [] 2024-03-24T03:17:23.533 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-24T03:17:23.534 INFO:teuthology.orchestra.run.smithi118.stdout: "log_to_cluster": { 2024-03-24T03:17:23.534 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "log_to_cluster", 2024-03-24T03:17:23.534 INFO:teuthology.orchestra.run.smithi118.stdout: "type": "bool", 2024-03-24T03:17:23.534 INFO:teuthology.orchestra.run.smithi118.stdout: "level": "advanced", 2024-03-24T03:17:23.534 INFO:teuthology.orchestra.run.smithi118.stdout: "flags": 1, 2024-03-24T03:17:23.534 INFO:teuthology.orchestra.run.smithi118.stdout: "default_value": "False", 2024-03-24T03:17:23.534 INFO:teuthology.orchestra.run.smithi118.stdout: "min": "", 2024-03-24T03:17:23.534 INFO:teuthology.orchestra.run.smithi118.stdout: "max": "", 2024-03-24T03:17:23.535 INFO:teuthology.orchestra.run.smithi118.stdout: "enum_allowed": [], 2024-03-24T03:17:23.535 INFO:teuthology.orchestra.run.smithi118.stdout: "desc": "", 2024-03-24T03:17:23.535 INFO:teuthology.orchestra.run.smithi118.stdout: "long_desc": "", 2024-03-24T03:17:23.535 INFO:teuthology.orchestra.run.smithi118.stdout: "tags": [], 2024-03-24T03:17:23.535 INFO:teuthology.orchestra.run.smithi118.stdout: "see_also": [] 2024-03-24T03:17:23.535 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-24T03:17:23.535 INFO:teuthology.orchestra.run.smithi118.stdout: "log_to_cluster_level": { 2024-03-24T03:17:23.535 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "log_to_cluster_level", 2024-03-24T03:17:23.535 INFO:teuthology.orchestra.run.smithi118.stdout: "type": "str", 2024-03-24T03:17:23.535 INFO:teuthology.orchestra.run.smithi118.stdout: "level": "advanced", 2024-03-24T03:17:23.536 INFO:teuthology.orchestra.run.smithi118.stdout: "flags": 1, 2024-03-24T03:17:23.536 INFO:teuthology.orchestra.run.smithi118.stdout: "default_value": "info", 2024-03-24T03:17:23.536 INFO:teuthology.orchestra.run.smithi118.stdout: "min": "", 2024-03-24T03:17:23.536 INFO:teuthology.orchestra.run.smithi118.stdout: "max": "", 2024-03-24T03:17:23.536 INFO:teuthology.orchestra.run.smithi118.stdout: "enum_allowed": [ 2024-03-24T03:17:23.536 INFO:teuthology.orchestra.run.smithi118.stdout: "", 2024-03-24T03:17:23.536 INFO:teuthology.orchestra.run.smithi118.stdout: "critical", 2024-03-24T03:17:23.536 INFO:teuthology.orchestra.run.smithi118.stdout: "debug", 2024-03-24T03:17:23.536 INFO:teuthology.orchestra.run.smithi118.stdout: "error", 2024-03-24T03:17:23.537 INFO:teuthology.orchestra.run.smithi118.stdout: "info", 2024-03-24T03:17:23.537 INFO:teuthology.orchestra.run.smithi118.stdout: "warning" 2024-03-24T03:17:23.537 INFO:teuthology.orchestra.run.smithi118.stdout: ], 2024-03-24T03:17:23.537 INFO:teuthology.orchestra.run.smithi118.stdout: "desc": "", 2024-03-24T03:17:23.537 INFO:teuthology.orchestra.run.smithi118.stdout: "long_desc": "", 2024-03-24T03:17:23.537 INFO:teuthology.orchestra.run.smithi118.stdout: "tags": [], 2024-03-24T03:17:23.537 INFO:teuthology.orchestra.run.smithi118.stdout: "see_also": [] 2024-03-24T03:17:23.537 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-24T03:17:23.537 INFO:teuthology.orchestra.run.smithi118.stdout: "log_to_file": { 2024-03-24T03:17:23.538 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "log_to_file", 2024-03-24T03:17:23.538 INFO:teuthology.orchestra.run.smithi118.stdout: "type": "bool", 2024-03-24T03:17:23.538 INFO:teuthology.orchestra.run.smithi118.stdout: "level": "advanced", 2024-03-24T03:17:23.538 INFO:teuthology.orchestra.run.smithi118.stdout: "flags": 1, 2024-03-24T03:17:23.538 INFO:teuthology.orchestra.run.smithi118.stdout: "default_value": "False", 2024-03-24T03:17:23.538 INFO:teuthology.orchestra.run.smithi118.stdout: "min": "", 2024-03-24T03:17:23.538 INFO:teuthology.orchestra.run.smithi118.stdout: "max": "", 2024-03-24T03:17:23.538 INFO:teuthology.orchestra.run.smithi118.stdout: "enum_allowed": [], 2024-03-24T03:17:23.538 INFO:teuthology.orchestra.run.smithi118.stdout: "desc": "", 2024-03-24T03:17:23.539 INFO:teuthology.orchestra.run.smithi118.stdout: "long_desc": "", 2024-03-24T03:17:23.539 INFO:teuthology.orchestra.run.smithi118.stdout: "tags": [], 2024-03-24T03:17:23.539 INFO:teuthology.orchestra.run.smithi118.stdout: "see_also": [] 2024-03-24T03:17:23.539 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-03-24T03:17:23.539 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-03-24T03:17:23.539 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-24T03:17:23.539 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-24T03:17:23.539 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "localpool", 2024-03-24T03:17:23.539 INFO:teuthology.orchestra.run.smithi118.stdout: "can_run": true, 2024-03-24T03:17:23.540 INFO:teuthology.orchestra.run.smithi118.stdout: "error_string": "", 2024-03-24T03:17:23.540 INFO:teuthology.orchestra.run.smithi118.stdout: "module_options": { 2024-03-24T03:17:23.540 INFO:teuthology.orchestra.run.smithi118.stdout: "failure_domain": { 2024-03-24T03:17:23.540 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "failure_domain", 2024-03-24T03:17:23.540 INFO:teuthology.orchestra.run.smithi118.stdout: "type": "str", 2024-03-24T03:17:23.540 INFO:teuthology.orchestra.run.smithi118.stdout: "level": "advanced", 2024-03-24T03:17:23.540 INFO:teuthology.orchestra.run.smithi118.stdout: "flags": 1, 2024-03-24T03:17:23.540 INFO:teuthology.orchestra.run.smithi118.stdout: "default_value": "host", 2024-03-24T03:17:23.540 INFO:teuthology.orchestra.run.smithi118.stdout: "min": "", 2024-03-24T03:17:23.540 INFO:teuthology.orchestra.run.smithi118.stdout: "max": "", 2024-03-24T03:17:23.541 INFO:teuthology.orchestra.run.smithi118.stdout: "enum_allowed": [], 2024-03-24T03:17:23.541 INFO:teuthology.orchestra.run.smithi118.stdout: "desc": "failure domain for any created local pool", 2024-03-24T03:17:23.541 INFO:teuthology.orchestra.run.smithi118.stdout: "long_desc": "what failure domain we should separate data replicas across.", 2024-03-24T03:17:23.541 INFO:teuthology.orchestra.run.smithi118.stdout: "tags": [], 2024-03-24T03:17:23.541 INFO:teuthology.orchestra.run.smithi118.stdout: "see_also": [] 2024-03-24T03:17:23.541 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-24T03:17:23.541 INFO:teuthology.orchestra.run.smithi118.stdout: "log_level": { 2024-03-24T03:17:23.541 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "log_level", 2024-03-24T03:17:23.541 INFO:teuthology.orchestra.run.smithi118.stdout: "type": "str", 2024-03-24T03:17:23.542 INFO:teuthology.orchestra.run.smithi118.stdout: "level": "advanced", 2024-03-24T03:17:23.542 INFO:teuthology.orchestra.run.smithi118.stdout: "flags": 1, 2024-03-24T03:17:23.542 INFO:teuthology.orchestra.run.smithi118.stdout: "default_value": "", 2024-03-24T03:17:23.542 INFO:teuthology.orchestra.run.smithi118.stdout: "min": "", 2024-03-24T03:17:23.542 INFO:teuthology.orchestra.run.smithi118.stdout: "max": "", 2024-03-24T03:17:23.542 INFO:teuthology.orchestra.run.smithi118.stdout: "enum_allowed": [ 2024-03-24T03:17:23.542 INFO:teuthology.orchestra.run.smithi118.stdout: "", 2024-03-24T03:17:23.542 INFO:teuthology.orchestra.run.smithi118.stdout: "critical", 2024-03-24T03:17:23.542 INFO:teuthology.orchestra.run.smithi118.stdout: "debug", 2024-03-24T03:17:23.543 INFO:teuthology.orchestra.run.smithi118.stdout: "error", 2024-03-24T03:17:23.543 INFO:teuthology.orchestra.run.smithi118.stdout: "info", 2024-03-24T03:17:23.543 INFO:teuthology.orchestra.run.smithi118.stdout: "warning" 2024-03-24T03:17:23.543 INFO:teuthology.orchestra.run.smithi118.stdout: ], 2024-03-24T03:17:23.543 INFO:teuthology.orchestra.run.smithi118.stdout: "desc": "", 2024-03-24T03:17:23.543 INFO:teuthology.orchestra.run.smithi118.stdout: "long_desc": "", 2024-03-24T03:17:23.543 INFO:teuthology.orchestra.run.smithi118.stdout: "tags": [], 2024-03-24T03:17:23.543 INFO:teuthology.orchestra.run.smithi118.stdout: "see_also": [] 2024-03-24T03:17:23.543 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-24T03:17:23.544 INFO:teuthology.orchestra.run.smithi118.stdout: "log_to_cluster": { 2024-03-24T03:17:23.544 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "log_to_cluster", 2024-03-24T03:17:23.544 INFO:teuthology.orchestra.run.smithi118.stdout: "type": "bool", 2024-03-24T03:17:23.544 INFO:teuthology.orchestra.run.smithi118.stdout: "level": "advanced", 2024-03-24T03:17:23.544 INFO:teuthology.orchestra.run.smithi118.stdout: "flags": 1, 2024-03-24T03:17:23.544 INFO:teuthology.orchestra.run.smithi118.stdout: "default_value": "False", 2024-03-24T03:17:23.544 INFO:teuthology.orchestra.run.smithi118.stdout: "min": "", 2024-03-24T03:17:23.544 INFO:teuthology.orchestra.run.smithi118.stdout: "max": "", 2024-03-24T03:17:23.545 INFO:teuthology.orchestra.run.smithi118.stdout: "enum_allowed": [], 2024-03-24T03:17:23.545 INFO:teuthology.orchestra.run.smithi118.stdout: "desc": "", 2024-03-24T03:17:23.545 INFO:teuthology.orchestra.run.smithi118.stdout: "long_desc": "", 2024-03-24T03:17:23.545 INFO:teuthology.orchestra.run.smithi118.stdout: "tags": [], 2024-03-24T03:17:23.545 INFO:teuthology.orchestra.run.smithi118.stdout: "see_also": [] 2024-03-24T03:17:23.545 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-24T03:17:23.545 INFO:teuthology.orchestra.run.smithi118.stdout: "log_to_cluster_level": { 2024-03-24T03:17:23.545 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "log_to_cluster_level", 2024-03-24T03:17:23.545 INFO:teuthology.orchestra.run.smithi118.stdout: "type": "str", 2024-03-24T03:17:23.545 INFO:teuthology.orchestra.run.smithi118.stdout: "level": "advanced", 2024-03-24T03:17:23.546 INFO:teuthology.orchestra.run.smithi118.stdout: "flags": 1, 2024-03-24T03:17:23.546 INFO:teuthology.orchestra.run.smithi118.stdout: "default_value": "info", 2024-03-24T03:17:23.546 INFO:teuthology.orchestra.run.smithi118.stdout: "min": "", 2024-03-24T03:17:23.546 INFO:teuthology.orchestra.run.smithi118.stdout: "max": "", 2024-03-24T03:17:23.546 INFO:teuthology.orchestra.run.smithi118.stdout: "enum_allowed": [ 2024-03-24T03:17:23.546 INFO:teuthology.orchestra.run.smithi118.stdout: "", 2024-03-24T03:17:23.546 INFO:teuthology.orchestra.run.smithi118.stdout: "critical", 2024-03-24T03:17:23.546 INFO:teuthology.orchestra.run.smithi118.stdout: "debug", 2024-03-24T03:17:23.546 INFO:teuthology.orchestra.run.smithi118.stdout: "error", 2024-03-24T03:17:23.547 INFO:teuthology.orchestra.run.smithi118.stdout: "info", 2024-03-24T03:17:23.547 INFO:teuthology.orchestra.run.smithi118.stdout: "warning" 2024-03-24T03:17:23.547 INFO:teuthology.orchestra.run.smithi118.stdout: ], 2024-03-24T03:17:23.547 INFO:teuthology.orchestra.run.smithi118.stdout: "desc": "", 2024-03-24T03:17:23.547 INFO:teuthology.orchestra.run.smithi118.stdout: "long_desc": "", 2024-03-24T03:17:23.547 INFO:teuthology.orchestra.run.smithi118.stdout: "tags": [], 2024-03-24T03:17:23.547 INFO:teuthology.orchestra.run.smithi118.stdout: "see_also": [] 2024-03-24T03:17:23.547 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-24T03:17:23.547 INFO:teuthology.orchestra.run.smithi118.stdout: "log_to_file": { 2024-03-24T03:17:23.548 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "log_to_file", 2024-03-24T03:17:23.548 INFO:teuthology.orchestra.run.smithi118.stdout: "type": "bool", 2024-03-24T03:17:23.548 INFO:teuthology.orchestra.run.smithi118.stdout: "level": "advanced", 2024-03-24T03:17:23.548 INFO:teuthology.orchestra.run.smithi118.stdout: "flags": 1, 2024-03-24T03:17:23.548 INFO:teuthology.orchestra.run.smithi118.stdout: "default_value": "False", 2024-03-24T03:17:23.548 INFO:teuthology.orchestra.run.smithi118.stdout: "min": "", 2024-03-24T03:17:23.548 INFO:teuthology.orchestra.run.smithi118.stdout: "max": "", 2024-03-24T03:17:23.548 INFO:teuthology.orchestra.run.smithi118.stdout: "enum_allowed": [], 2024-03-24T03:17:23.548 INFO:teuthology.orchestra.run.smithi118.stdout: "desc": "", 2024-03-24T03:17:23.549 INFO:teuthology.orchestra.run.smithi118.stdout: "long_desc": "", 2024-03-24T03:17:23.549 INFO:teuthology.orchestra.run.smithi118.stdout: "tags": [], 2024-03-24T03:17:23.549 INFO:teuthology.orchestra.run.smithi118.stdout: "see_also": [] 2024-03-24T03:17:23.549 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-24T03:17:23.549 INFO:teuthology.orchestra.run.smithi118.stdout: "min_size": { 2024-03-24T03:17:23.549 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "min_size", 2024-03-24T03:17:23.549 INFO:teuthology.orchestra.run.smithi118.stdout: "type": "int", 2024-03-24T03:17:23.549 INFO:teuthology.orchestra.run.smithi118.stdout: "level": "advanced", 2024-03-24T03:17:23.549 INFO:teuthology.orchestra.run.smithi118.stdout: "flags": 1, 2024-03-24T03:17:23.549 INFO:teuthology.orchestra.run.smithi118.stdout: "default_value": "", 2024-03-24T03:17:23.550 INFO:teuthology.orchestra.run.smithi118.stdout: "min": "", 2024-03-24T03:17:23.550 INFO:teuthology.orchestra.run.smithi118.stdout: "max": "", 2024-03-24T03:17:23.550 INFO:teuthology.orchestra.run.smithi118.stdout: "enum_allowed": [], 2024-03-24T03:17:23.550 INFO:teuthology.orchestra.run.smithi118.stdout: "desc": "default min_size for any created local pool", 2024-03-24T03:17:23.550 INFO:teuthology.orchestra.run.smithi118.stdout: "long_desc": "value to set min_size to (unchanged from Ceph's default if this option is not set)", 2024-03-24T03:17:23.550 INFO:teuthology.orchestra.run.smithi118.stdout: "tags": [], 2024-03-24T03:17:23.550 INFO:teuthology.orchestra.run.smithi118.stdout: "see_also": [] 2024-03-24T03:17:23.550 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-24T03:17:23.550 INFO:teuthology.orchestra.run.smithi118.stdout: "num_rep": { 2024-03-24T03:17:23.551 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "num_rep", 2024-03-24T03:17:23.551 INFO:teuthology.orchestra.run.smithi118.stdout: "type": "int", 2024-03-24T03:17:23.551 INFO:teuthology.orchestra.run.smithi118.stdout: "level": "advanced", 2024-03-24T03:17:23.551 INFO:teuthology.orchestra.run.smithi118.stdout: "flags": 1, 2024-03-24T03:17:23.551 INFO:teuthology.orchestra.run.smithi118.stdout: "default_value": "3", 2024-03-24T03:17:23.551 INFO:teuthology.orchestra.run.smithi118.stdout: "min": "", 2024-03-24T03:17:23.551 INFO:teuthology.orchestra.run.smithi118.stdout: "max": "", 2024-03-24T03:17:23.551 INFO:teuthology.orchestra.run.smithi118.stdout: "enum_allowed": [], 2024-03-24T03:17:23.552 INFO:teuthology.orchestra.run.smithi118.stdout: "desc": "default replica count for any created local pool", 2024-03-24T03:17:23.552 INFO:teuthology.orchestra.run.smithi118.stdout: "long_desc": "", 2024-03-24T03:17:23.552 INFO:teuthology.orchestra.run.smithi118.stdout: "tags": [], 2024-03-24T03:17:23.552 INFO:teuthology.orchestra.run.smithi118.stdout: "see_also": [] 2024-03-24T03:17:23.552 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-24T03:17:23.552 INFO:teuthology.orchestra.run.smithi118.stdout: "pg_num": { 2024-03-24T03:17:23.552 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "pg_num", 2024-03-24T03:17:23.552 INFO:teuthology.orchestra.run.smithi118.stdout: "type": "int", 2024-03-24T03:17:23.552 INFO:teuthology.orchestra.run.smithi118.stdout: "level": "advanced", 2024-03-24T03:17:23.553 INFO:teuthology.orchestra.run.smithi118.stdout: "flags": 1, 2024-03-24T03:17:23.553 INFO:teuthology.orchestra.run.smithi118.stdout: "default_value": "128", 2024-03-24T03:17:23.553 INFO:teuthology.orchestra.run.smithi118.stdout: "min": "", 2024-03-24T03:17:23.553 INFO:teuthology.orchestra.run.smithi118.stdout: "max": "", 2024-03-24T03:17:23.553 INFO:teuthology.orchestra.run.smithi118.stdout: "enum_allowed": [], 2024-03-24T03:17:23.553 INFO:teuthology.orchestra.run.smithi118.stdout: "desc": "default pg_num for any created local pool", 2024-03-24T03:17:23.553 INFO:teuthology.orchestra.run.smithi118.stdout: "long_desc": "", 2024-03-24T03:17:23.553 INFO:teuthology.orchestra.run.smithi118.stdout: "tags": [], 2024-03-24T03:17:23.554 INFO:teuthology.orchestra.run.smithi118.stdout: "see_also": [] 2024-03-24T03:17:23.554 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-24T03:17:23.554 INFO:teuthology.orchestra.run.smithi118.stdout: "prefix": { 2024-03-24T03:17:23.554 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "prefix", 2024-03-24T03:17:23.554 INFO:teuthology.orchestra.run.smithi118.stdout: "type": "str", 2024-03-24T03:17:23.554 INFO:teuthology.orchestra.run.smithi118.stdout: "level": "advanced", 2024-03-24T03:17:23.555 INFO:teuthology.orchestra.run.smithi118.stdout: "flags": 1, 2024-03-24T03:17:23.555 INFO:teuthology.orchestra.run.smithi118.stdout: "default_value": "", 2024-03-24T03:17:23.555 INFO:teuthology.orchestra.run.smithi118.stdout: "min": "", 2024-03-24T03:17:23.555 INFO:teuthology.orchestra.run.smithi118.stdout: "max": "", 2024-03-24T03:17:23.555 INFO:teuthology.orchestra.run.smithi118.stdout: "enum_allowed": [], 2024-03-24T03:17:23.555 INFO:teuthology.orchestra.run.smithi118.stdout: "desc": "name prefix for any created local pool", 2024-03-24T03:17:23.555 INFO:teuthology.orchestra.run.smithi118.stdout: "long_desc": "", 2024-03-24T03:17:23.555 INFO:teuthology.orchestra.run.smithi118.stdout: "tags": [], 2024-03-24T03:17:23.555 INFO:teuthology.orchestra.run.smithi118.stdout: "see_also": [] 2024-03-24T03:17:23.556 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-24T03:17:23.556 INFO:teuthology.orchestra.run.smithi118.stdout: "subtree": { 2024-03-24T03:17:23.556 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "subtree", 2024-03-24T03:17:23.556 INFO:teuthology.orchestra.run.smithi118.stdout: "type": "str", 2024-03-24T03:17:23.556 INFO:teuthology.orchestra.run.smithi118.stdout: "level": "advanced", 2024-03-24T03:17:23.556 INFO:teuthology.orchestra.run.smithi118.stdout: "flags": 1, 2024-03-24T03:17:23.556 INFO:teuthology.orchestra.run.smithi118.stdout: "default_value": "rack", 2024-03-24T03:17:23.556 INFO:teuthology.orchestra.run.smithi118.stdout: "min": "", 2024-03-24T03:17:23.556 INFO:teuthology.orchestra.run.smithi118.stdout: "max": "", 2024-03-24T03:17:23.557 INFO:teuthology.orchestra.run.smithi118.stdout: "enum_allowed": [], 2024-03-24T03:17:23.557 INFO:teuthology.orchestra.run.smithi118.stdout: "desc": "CRUSH level for which to create a local pool", 2024-03-24T03:17:23.557 INFO:teuthology.orchestra.run.smithi118.stdout: "long_desc": "which CRUSH subtree type the module should create a pool for.", 2024-03-24T03:17:23.557 INFO:teuthology.orchestra.run.smithi118.stdout: "tags": [], 2024-03-24T03:17:23.557 INFO:teuthology.orchestra.run.smithi118.stdout: "see_also": [] 2024-03-24T03:17:23.557 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-03-24T03:17:23.557 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-03-24T03:17:23.557 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-24T03:17:23.557 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-24T03:17:23.558 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "mds_autoscaler", 2024-03-24T03:17:23.558 INFO:teuthology.orchestra.run.smithi118.stdout: "can_run": true, 2024-03-24T03:17:23.558 INFO:teuthology.orchestra.run.smithi118.stdout: "error_string": "", 2024-03-24T03:17:23.558 INFO:teuthology.orchestra.run.smithi118.stdout: "module_options": { 2024-03-24T03:17:23.558 INFO:teuthology.orchestra.run.smithi118.stdout: "log_level": { 2024-03-24T03:17:23.558 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "log_level", 2024-03-24T03:17:23.558 INFO:teuthology.orchestra.run.smithi118.stdout: "type": "str", 2024-03-24T03:17:23.558 INFO:teuthology.orchestra.run.smithi118.stdout: "level": "advanced", 2024-03-24T03:17:23.559 INFO:teuthology.orchestra.run.smithi118.stdout: "flags": 1, 2024-03-24T03:17:23.559 INFO:teuthology.orchestra.run.smithi118.stdout: "default_value": "", 2024-03-24T03:17:23.559 INFO:teuthology.orchestra.run.smithi118.stdout: "min": "", 2024-03-24T03:17:23.559 INFO:teuthology.orchestra.run.smithi118.stdout: "max": "", 2024-03-24T03:17:23.559 INFO:teuthology.orchestra.run.smithi118.stdout: "enum_allowed": [ 2024-03-24T03:17:23.559 INFO:teuthology.orchestra.run.smithi118.stdout: "", 2024-03-24T03:17:23.559 INFO:teuthology.orchestra.run.smithi118.stdout: "critical", 2024-03-24T03:17:23.559 INFO:teuthology.orchestra.run.smithi118.stdout: "debug", 2024-03-24T03:17:23.559 INFO:teuthology.orchestra.run.smithi118.stdout: "error", 2024-03-24T03:17:23.560 INFO:teuthology.orchestra.run.smithi118.stdout: "info", 2024-03-24T03:17:23.560 INFO:teuthology.orchestra.run.smithi118.stdout: "warning" 2024-03-24T03:17:23.560 INFO:teuthology.orchestra.run.smithi118.stdout: ], 2024-03-24T03:17:23.560 INFO:teuthology.orchestra.run.smithi118.stdout: "desc": "", 2024-03-24T03:17:23.560 INFO:teuthology.orchestra.run.smithi118.stdout: "long_desc": "", 2024-03-24T03:17:23.560 INFO:teuthology.orchestra.run.smithi118.stdout: "tags": [], 2024-03-24T03:17:23.560 INFO:teuthology.orchestra.run.smithi118.stdout: "see_also": [] 2024-03-24T03:17:23.560 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-24T03:17:23.560 INFO:teuthology.orchestra.run.smithi118.stdout: "log_to_cluster": { 2024-03-24T03:17:23.561 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "log_to_cluster", 2024-03-24T03:17:23.561 INFO:teuthology.orchestra.run.smithi118.stdout: "type": "bool", 2024-03-24T03:17:23.561 INFO:teuthology.orchestra.run.smithi118.stdout: "level": "advanced", 2024-03-24T03:17:23.561 INFO:teuthology.orchestra.run.smithi118.stdout: "flags": 1, 2024-03-24T03:17:23.561 INFO:teuthology.orchestra.run.smithi118.stdout: "default_value": "False", 2024-03-24T03:17:23.561 INFO:teuthology.orchestra.run.smithi118.stdout: "min": "", 2024-03-24T03:17:23.561 INFO:teuthology.orchestra.run.smithi118.stdout: "max": "", 2024-03-24T03:17:23.561 INFO:teuthology.orchestra.run.smithi118.stdout: "enum_allowed": [], 2024-03-24T03:17:23.561 INFO:teuthology.orchestra.run.smithi118.stdout: "desc": "", 2024-03-24T03:17:23.562 INFO:teuthology.orchestra.run.smithi118.stdout: "long_desc": "", 2024-03-24T03:17:23.562 INFO:teuthology.orchestra.run.smithi118.stdout: "tags": [], 2024-03-24T03:17:23.562 INFO:teuthology.orchestra.run.smithi118.stdout: "see_also": [] 2024-03-24T03:17:23.562 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-24T03:17:23.562 INFO:teuthology.orchestra.run.smithi118.stdout: "log_to_cluster_level": { 2024-03-24T03:17:23.562 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "log_to_cluster_level", 2024-03-24T03:17:23.562 INFO:teuthology.orchestra.run.smithi118.stdout: "type": "str", 2024-03-24T03:17:23.562 INFO:teuthology.orchestra.run.smithi118.stdout: "level": "advanced", 2024-03-24T03:17:23.563 INFO:teuthology.orchestra.run.smithi118.stdout: "flags": 1, 2024-03-24T03:17:23.563 INFO:teuthology.orchestra.run.smithi118.stdout: "default_value": "info", 2024-03-24T03:17:23.563 INFO:teuthology.orchestra.run.smithi118.stdout: "min": "", 2024-03-24T03:17:23.563 INFO:teuthology.orchestra.run.smithi118.stdout: "max": "", 2024-03-24T03:17:23.563 INFO:teuthology.orchestra.run.smithi118.stdout: "enum_allowed": [ 2024-03-24T03:17:23.563 INFO:teuthology.orchestra.run.smithi118.stdout: "", 2024-03-24T03:17:23.563 INFO:teuthology.orchestra.run.smithi118.stdout: "critical", 2024-03-24T03:17:23.563 INFO:teuthology.orchestra.run.smithi118.stdout: "debug", 2024-03-24T03:17:23.563 INFO:teuthology.orchestra.run.smithi118.stdout: "error", 2024-03-24T03:17:23.563 INFO:teuthology.orchestra.run.smithi118.stdout: "info", 2024-03-24T03:17:23.564 INFO:teuthology.orchestra.run.smithi118.stdout: "warning" 2024-03-24T03:17:23.564 INFO:teuthology.orchestra.run.smithi118.stdout: ], 2024-03-24T03:17:23.564 INFO:teuthology.orchestra.run.smithi118.stdout: "desc": "", 2024-03-24T03:17:23.564 INFO:teuthology.orchestra.run.smithi118.stdout: "long_desc": "", 2024-03-24T03:17:23.564 INFO:teuthology.orchestra.run.smithi118.stdout: "tags": [], 2024-03-24T03:17:23.564 INFO:teuthology.orchestra.run.smithi118.stdout: "see_also": [] 2024-03-24T03:17:23.564 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-24T03:17:23.565 INFO:teuthology.orchestra.run.smithi118.stdout: "log_to_file": { 2024-03-24T03:17:23.565 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "log_to_file", 2024-03-24T03:17:23.565 INFO:teuthology.orchestra.run.smithi118.stdout: "type": "bool", 2024-03-24T03:17:23.565 INFO:teuthology.orchestra.run.smithi118.stdout: "level": "advanced", 2024-03-24T03:17:23.565 INFO:teuthology.orchestra.run.smithi118.stdout: "flags": 1, 2024-03-24T03:17:23.565 INFO:teuthology.orchestra.run.smithi118.stdout: "default_value": "False", 2024-03-24T03:17:23.565 INFO:teuthology.orchestra.run.smithi118.stdout: "min": "", 2024-03-24T03:17:23.565 INFO:teuthology.orchestra.run.smithi118.stdout: "max": "", 2024-03-24T03:17:23.565 INFO:teuthology.orchestra.run.smithi118.stdout: "enum_allowed": [], 2024-03-24T03:17:23.566 INFO:teuthology.orchestra.run.smithi118.stdout: "desc": "", 2024-03-24T03:17:23.566 INFO:teuthology.orchestra.run.smithi118.stdout: "long_desc": "", 2024-03-24T03:17:23.566 INFO:teuthology.orchestra.run.smithi118.stdout: "tags": [], 2024-03-24T03:17:23.566 INFO:teuthology.orchestra.run.smithi118.stdout: "see_also": [] 2024-03-24T03:17:23.566 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-03-24T03:17:23.566 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-03-24T03:17:23.566 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-24T03:17:23.566 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-24T03:17:23.566 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "mirroring", 2024-03-24T03:17:23.567 INFO:teuthology.orchestra.run.smithi118.stdout: "can_run": true, 2024-03-24T03:17:23.567 INFO:teuthology.orchestra.run.smithi118.stdout: "error_string": "", 2024-03-24T03:17:23.567 INFO:teuthology.orchestra.run.smithi118.stdout: "module_options": { 2024-03-24T03:17:23.567 INFO:teuthology.orchestra.run.smithi118.stdout: "log_level": { 2024-03-24T03:17:23.567 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "log_level", 2024-03-24T03:17:23.567 INFO:teuthology.orchestra.run.smithi118.stdout: "type": "str", 2024-03-24T03:17:23.567 INFO:teuthology.orchestra.run.smithi118.stdout: "level": "advanced", 2024-03-24T03:17:23.567 INFO:teuthology.orchestra.run.smithi118.stdout: "flags": 1, 2024-03-24T03:17:23.568 INFO:teuthology.orchestra.run.smithi118.stdout: "default_value": "", 2024-03-24T03:17:23.568 INFO:teuthology.orchestra.run.smithi118.stdout: "min": "", 2024-03-24T03:17:23.568 INFO:teuthology.orchestra.run.smithi118.stdout: "max": "", 2024-03-24T03:17:23.568 INFO:teuthology.orchestra.run.smithi118.stdout: "enum_allowed": [ 2024-03-24T03:17:23.568 INFO:teuthology.orchestra.run.smithi118.stdout: "", 2024-03-24T03:17:23.568 INFO:teuthology.orchestra.run.smithi118.stdout: "critical", 2024-03-24T03:17:23.568 INFO:teuthology.orchestra.run.smithi118.stdout: "debug", 2024-03-24T03:17:23.568 INFO:teuthology.orchestra.run.smithi118.stdout: "error", 2024-03-24T03:17:23.568 INFO:teuthology.orchestra.run.smithi118.stdout: "info", 2024-03-24T03:17:23.568 INFO:teuthology.orchestra.run.smithi118.stdout: "warning" 2024-03-24T03:17:23.569 INFO:teuthology.orchestra.run.smithi118.stdout: ], 2024-03-24T03:17:23.569 INFO:teuthology.orchestra.run.smithi118.stdout: "desc": "", 2024-03-24T03:17:23.569 INFO:teuthology.orchestra.run.smithi118.stdout: "long_desc": "", 2024-03-24T03:17:23.569 INFO:teuthology.orchestra.run.smithi118.stdout: "tags": [], 2024-03-24T03:17:23.569 INFO:teuthology.orchestra.run.smithi118.stdout: "see_also": [] 2024-03-24T03:17:23.569 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-24T03:17:23.569 INFO:teuthology.orchestra.run.smithi118.stdout: "log_to_cluster": { 2024-03-24T03:17:23.569 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "log_to_cluster", 2024-03-24T03:17:23.569 INFO:teuthology.orchestra.run.smithi118.stdout: "type": "bool", 2024-03-24T03:17:23.570 INFO:teuthology.orchestra.run.smithi118.stdout: "level": "advanced", 2024-03-24T03:17:23.570 INFO:teuthology.orchestra.run.smithi118.stdout: "flags": 1, 2024-03-24T03:17:23.570 INFO:teuthology.orchestra.run.smithi118.stdout: "default_value": "False", 2024-03-24T03:17:23.570 INFO:teuthology.orchestra.run.smithi118.stdout: "min": "", 2024-03-24T03:17:23.570 INFO:teuthology.orchestra.run.smithi118.stdout: "max": "", 2024-03-24T03:17:23.570 INFO:teuthology.orchestra.run.smithi118.stdout: "enum_allowed": [], 2024-03-24T03:17:23.570 INFO:teuthology.orchestra.run.smithi118.stdout: "desc": "", 2024-03-24T03:17:23.570 INFO:teuthology.orchestra.run.smithi118.stdout: "long_desc": "", 2024-03-24T03:17:23.571 INFO:teuthology.orchestra.run.smithi118.stdout: "tags": [], 2024-03-24T03:17:23.571 INFO:teuthology.orchestra.run.smithi118.stdout: "see_also": [] 2024-03-24T03:17:23.571 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-24T03:17:23.571 INFO:teuthology.orchestra.run.smithi118.stdout: "log_to_cluster_level": { 2024-03-24T03:17:23.571 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "log_to_cluster_level", 2024-03-24T03:17:23.571 INFO:teuthology.orchestra.run.smithi118.stdout: "type": "str", 2024-03-24T03:17:23.571 INFO:teuthology.orchestra.run.smithi118.stdout: "level": "advanced", 2024-03-24T03:17:23.571 INFO:teuthology.orchestra.run.smithi118.stdout: "flags": 1, 2024-03-24T03:17:23.571 INFO:teuthology.orchestra.run.smithi118.stdout: "default_value": "info", 2024-03-24T03:17:23.571 INFO:teuthology.orchestra.run.smithi118.stdout: "min": "", 2024-03-24T03:17:23.572 INFO:teuthology.orchestra.run.smithi118.stdout: "max": "", 2024-03-24T03:17:23.572 INFO:teuthology.orchestra.run.smithi118.stdout: "enum_allowed": [ 2024-03-24T03:17:23.572 INFO:teuthology.orchestra.run.smithi118.stdout: "", 2024-03-24T03:17:23.572 INFO:teuthology.orchestra.run.smithi118.stdout: "critical", 2024-03-24T03:17:23.572 INFO:teuthology.orchestra.run.smithi118.stdout: "debug", 2024-03-24T03:17:23.572 INFO:teuthology.orchestra.run.smithi118.stdout: "error", 2024-03-24T03:17:23.572 INFO:teuthology.orchestra.run.smithi118.stdout: "info", 2024-03-24T03:17:23.572 INFO:teuthology.orchestra.run.smithi118.stdout: "warning" 2024-03-24T03:17:23.572 INFO:teuthology.orchestra.run.smithi118.stdout: ], 2024-03-24T03:17:23.573 INFO:teuthology.orchestra.run.smithi118.stdout: "desc": "", 2024-03-24T03:17:23.573 INFO:teuthology.orchestra.run.smithi118.stdout: "long_desc": "", 2024-03-24T03:17:23.573 INFO:teuthology.orchestra.run.smithi118.stdout: "tags": [], 2024-03-24T03:17:23.573 INFO:teuthology.orchestra.run.smithi118.stdout: "see_also": [] 2024-03-24T03:17:23.573 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-24T03:17:23.573 INFO:teuthology.orchestra.run.smithi118.stdout: "log_to_file": { 2024-03-24T03:17:23.573 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "log_to_file", 2024-03-24T03:17:23.573 INFO:teuthology.orchestra.run.smithi118.stdout: "type": "bool", 2024-03-24T03:17:23.573 INFO:teuthology.orchestra.run.smithi118.stdout: "level": "advanced", 2024-03-24T03:17:23.574 INFO:teuthology.orchestra.run.smithi118.stdout: "flags": 1, 2024-03-24T03:17:23.574 INFO:teuthology.orchestra.run.smithi118.stdout: "default_value": "False", 2024-03-24T03:17:23.574 INFO:teuthology.orchestra.run.smithi118.stdout: "min": "", 2024-03-24T03:17:23.574 INFO:teuthology.orchestra.run.smithi118.stdout: "max": "", 2024-03-24T03:17:23.574 INFO:teuthology.orchestra.run.smithi118.stdout: "enum_allowed": [], 2024-03-24T03:17:23.574 INFO:teuthology.orchestra.run.smithi118.stdout: "desc": "", 2024-03-24T03:17:23.574 INFO:teuthology.orchestra.run.smithi118.stdout: "long_desc": "", 2024-03-24T03:17:23.574 INFO:teuthology.orchestra.run.smithi118.stdout: "tags": [], 2024-03-24T03:17:23.575 INFO:teuthology.orchestra.run.smithi118.stdout: "see_also": [] 2024-03-24T03:17:23.575 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-03-24T03:17:23.575 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-03-24T03:17:23.575 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-24T03:17:23.575 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-24T03:17:23.575 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "nfs", 2024-03-24T03:17:23.575 INFO:teuthology.orchestra.run.smithi118.stdout: "can_run": true, 2024-03-24T03:17:23.575 INFO:teuthology.orchestra.run.smithi118.stdout: "error_string": "", 2024-03-24T03:17:23.575 INFO:teuthology.orchestra.run.smithi118.stdout: "module_options": { 2024-03-24T03:17:23.575 INFO:teuthology.orchestra.run.smithi118.stdout: "log_level": { 2024-03-24T03:17:23.576 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "log_level", 2024-03-24T03:17:23.576 INFO:teuthology.orchestra.run.smithi118.stdout: "type": "str", 2024-03-24T03:17:23.576 INFO:teuthology.orchestra.run.smithi118.stdout: "level": "advanced", 2024-03-24T03:17:23.576 INFO:teuthology.orchestra.run.smithi118.stdout: "flags": 1, 2024-03-24T03:17:23.576 INFO:teuthology.orchestra.run.smithi118.stdout: "default_value": "", 2024-03-24T03:17:23.576 INFO:teuthology.orchestra.run.smithi118.stdout: "min": "", 2024-03-24T03:17:23.576 INFO:teuthology.orchestra.run.smithi118.stdout: "max": "", 2024-03-24T03:17:23.576 INFO:teuthology.orchestra.run.smithi118.stdout: "enum_allowed": [ 2024-03-24T03:17:23.576 INFO:teuthology.orchestra.run.smithi118.stdout: "", 2024-03-24T03:17:23.577 INFO:teuthology.orchestra.run.smithi118.stdout: "critical", 2024-03-24T03:17:23.577 INFO:teuthology.orchestra.run.smithi118.stdout: "debug", 2024-03-24T03:17:23.577 INFO:teuthology.orchestra.run.smithi118.stdout: "error", 2024-03-24T03:17:23.577 INFO:teuthology.orchestra.run.smithi118.stdout: "info", 2024-03-24T03:17:23.577 INFO:teuthology.orchestra.run.smithi118.stdout: "warning" 2024-03-24T03:17:23.577 INFO:teuthology.orchestra.run.smithi118.stdout: ], 2024-03-24T03:17:23.577 INFO:teuthology.orchestra.run.smithi118.stdout: "desc": "", 2024-03-24T03:17:23.577 INFO:teuthology.orchestra.run.smithi118.stdout: "long_desc": "", 2024-03-24T03:17:23.577 INFO:teuthology.orchestra.run.smithi118.stdout: "tags": [], 2024-03-24T03:17:23.578 INFO:teuthology.orchestra.run.smithi118.stdout: "see_also": [] 2024-03-24T03:17:23.578 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-24T03:17:23.578 INFO:teuthology.orchestra.run.smithi118.stdout: "log_to_cluster": { 2024-03-24T03:17:23.578 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "log_to_cluster", 2024-03-24T03:17:23.578 INFO:teuthology.orchestra.run.smithi118.stdout: "type": "bool", 2024-03-24T03:17:23.578 INFO:teuthology.orchestra.run.smithi118.stdout: "level": "advanced", 2024-03-24T03:17:23.578 INFO:teuthology.orchestra.run.smithi118.stdout: "flags": 1, 2024-03-24T03:17:23.578 INFO:teuthology.orchestra.run.smithi118.stdout: "default_value": "False", 2024-03-24T03:17:23.581 INFO:teuthology.orchestra.run.smithi118.stdout: "min": "", 2024-03-24T03:17:23.581 INFO:teuthology.orchestra.run.smithi118.stdout: "max": "", 2024-03-24T03:17:23.581 INFO:teuthology.orchestra.run.smithi118.stdout: "enum_allowed": [], 2024-03-24T03:17:23.581 INFO:teuthology.orchestra.run.smithi118.stdout: "desc": "", 2024-03-24T03:17:23.581 INFO:teuthology.orchestra.run.smithi118.stdout: "long_desc": "", 2024-03-24T03:17:23.581 INFO:teuthology.orchestra.run.smithi118.stdout: "tags": [], 2024-03-24T03:17:23.582 INFO:teuthology.orchestra.run.smithi118.stdout: "see_also": [] 2024-03-24T03:17:23.582 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-24T03:17:23.582 INFO:teuthology.orchestra.run.smithi118.stdout: "log_to_cluster_level": { 2024-03-24T03:17:23.582 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "log_to_cluster_level", 2024-03-24T03:17:23.582 INFO:teuthology.orchestra.run.smithi118.stdout: "type": "str", 2024-03-24T03:17:23.582 INFO:teuthology.orchestra.run.smithi118.stdout: "level": "advanced", 2024-03-24T03:17:23.582 INFO:teuthology.orchestra.run.smithi118.stdout: "flags": 1, 2024-03-24T03:17:23.582 INFO:teuthology.orchestra.run.smithi118.stdout: "default_value": "info", 2024-03-24T03:17:23.583 INFO:teuthology.orchestra.run.smithi118.stdout: "min": "", 2024-03-24T03:17:23.583 INFO:teuthology.orchestra.run.smithi118.stdout: "max": "", 2024-03-24T03:17:23.583 INFO:teuthology.orchestra.run.smithi118.stdout: "enum_allowed": [ 2024-03-24T03:17:23.583 INFO:teuthology.orchestra.run.smithi118.stdout: "", 2024-03-24T03:17:23.583 INFO:teuthology.orchestra.run.smithi118.stdout: "critical", 2024-03-24T03:17:23.583 INFO:teuthology.orchestra.run.smithi118.stdout: "debug", 2024-03-24T03:17:23.583 INFO:teuthology.orchestra.run.smithi118.stdout: "error", 2024-03-24T03:17:23.583 INFO:teuthology.orchestra.run.smithi118.stdout: "info", 2024-03-24T03:17:23.583 INFO:teuthology.orchestra.run.smithi118.stdout: "warning" 2024-03-24T03:17:23.584 INFO:teuthology.orchestra.run.smithi118.stdout: ], 2024-03-24T03:17:23.584 INFO:teuthology.orchestra.run.smithi118.stdout: "desc": "", 2024-03-24T03:17:23.584 INFO:teuthology.orchestra.run.smithi118.stdout: "long_desc": "", 2024-03-24T03:17:23.584 INFO:teuthology.orchestra.run.smithi118.stdout: "tags": [], 2024-03-24T03:17:23.584 INFO:teuthology.orchestra.run.smithi118.stdout: "see_also": [] 2024-03-24T03:17:23.584 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-24T03:17:23.584 INFO:teuthology.orchestra.run.smithi118.stdout: "log_to_file": { 2024-03-24T03:17:23.584 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "log_to_file", 2024-03-24T03:17:23.584 INFO:teuthology.orchestra.run.smithi118.stdout: "type": "bool", 2024-03-24T03:17:23.585 INFO:teuthology.orchestra.run.smithi118.stdout: "level": "advanced", 2024-03-24T03:17:23.585 INFO:teuthology.orchestra.run.smithi118.stdout: "flags": 1, 2024-03-24T03:17:23.585 INFO:teuthology.orchestra.run.smithi118.stdout: "default_value": "False", 2024-03-24T03:17:23.585 INFO:teuthology.orchestra.run.smithi118.stdout: "min": "", 2024-03-24T03:17:23.585 INFO:teuthology.orchestra.run.smithi118.stdout: "max": "", 2024-03-24T03:17:23.585 INFO:teuthology.orchestra.run.smithi118.stdout: "enum_allowed": [], 2024-03-24T03:17:23.585 INFO:teuthology.orchestra.run.smithi118.stdout: "desc": "", 2024-03-24T03:17:23.585 INFO:teuthology.orchestra.run.smithi118.stdout: "long_desc": "", 2024-03-24T03:17:23.585 INFO:teuthology.orchestra.run.smithi118.stdout: "tags": [], 2024-03-24T03:17:23.585 INFO:teuthology.orchestra.run.smithi118.stdout: "see_also": [] 2024-03-24T03:17:23.586 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-03-24T03:17:23.586 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-03-24T03:17:23.586 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-24T03:17:23.586 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-24T03:17:23.586 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "orchestrator", 2024-03-24T03:17:23.586 INFO:teuthology.orchestra.run.smithi118.stdout: "can_run": true, 2024-03-24T03:17:23.586 INFO:teuthology.orchestra.run.smithi118.stdout: "error_string": "", 2024-03-24T03:17:23.586 INFO:teuthology.orchestra.run.smithi118.stdout: "module_options": { 2024-03-24T03:17:23.586 INFO:teuthology.orchestra.run.smithi118.stdout: "fail_fs": { 2024-03-24T03:17:23.587 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "fail_fs", 2024-03-24T03:17:23.587 INFO:teuthology.orchestra.run.smithi118.stdout: "type": "bool", 2024-03-24T03:17:23.587 INFO:teuthology.orchestra.run.smithi118.stdout: "level": "advanced", 2024-03-24T03:17:23.587 INFO:teuthology.orchestra.run.smithi118.stdout: "flags": 0, 2024-03-24T03:17:23.587 INFO:teuthology.orchestra.run.smithi118.stdout: "default_value": "False", 2024-03-24T03:17:23.587 INFO:teuthology.orchestra.run.smithi118.stdout: "min": "", 2024-03-24T03:17:23.587 INFO:teuthology.orchestra.run.smithi118.stdout: "max": "", 2024-03-24T03:17:23.587 INFO:teuthology.orchestra.run.smithi118.stdout: "enum_allowed": [], 2024-03-24T03:17:23.587 INFO:teuthology.orchestra.run.smithi118.stdout: "desc": "Fail filesystem for rapid multi-rank mds upgrade", 2024-03-24T03:17:23.588 INFO:teuthology.orchestra.run.smithi118.stdout: "long_desc": "", 2024-03-24T03:17:23.588 INFO:teuthology.orchestra.run.smithi118.stdout: "tags": [], 2024-03-24T03:17:23.588 INFO:teuthology.orchestra.run.smithi118.stdout: "see_also": [] 2024-03-24T03:17:23.588 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-24T03:17:23.588 INFO:teuthology.orchestra.run.smithi118.stdout: "log_level": { 2024-03-24T03:17:23.588 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "log_level", 2024-03-24T03:17:23.588 INFO:teuthology.orchestra.run.smithi118.stdout: "type": "str", 2024-03-24T03:17:23.588 INFO:teuthology.orchestra.run.smithi118.stdout: "level": "advanced", 2024-03-24T03:17:23.588 INFO:teuthology.orchestra.run.smithi118.stdout: "flags": 1, 2024-03-24T03:17:23.588 INFO:teuthology.orchestra.run.smithi118.stdout: "default_value": "", 2024-03-24T03:17:23.589 INFO:teuthology.orchestra.run.smithi118.stdout: "min": "", 2024-03-24T03:17:23.589 INFO:teuthology.orchestra.run.smithi118.stdout: "max": "", 2024-03-24T03:17:23.589 INFO:teuthology.orchestra.run.smithi118.stdout: "enum_allowed": [ 2024-03-24T03:17:23.589 INFO:teuthology.orchestra.run.smithi118.stdout: "", 2024-03-24T03:17:23.589 INFO:teuthology.orchestra.run.smithi118.stdout: "critical", 2024-03-24T03:17:23.589 INFO:teuthology.orchestra.run.smithi118.stdout: "debug", 2024-03-24T03:17:23.589 INFO:teuthology.orchestra.run.smithi118.stdout: "error", 2024-03-24T03:17:23.589 INFO:teuthology.orchestra.run.smithi118.stdout: "info", 2024-03-24T03:17:23.589 INFO:teuthology.orchestra.run.smithi118.stdout: "warning" 2024-03-24T03:17:23.590 INFO:teuthology.orchestra.run.smithi118.stdout: ], 2024-03-24T03:17:23.590 INFO:teuthology.orchestra.run.smithi118.stdout: "desc": "", 2024-03-24T03:17:23.590 INFO:teuthology.orchestra.run.smithi118.stdout: "long_desc": "", 2024-03-24T03:17:23.590 INFO:teuthology.orchestra.run.smithi118.stdout: "tags": [], 2024-03-24T03:17:23.590 INFO:teuthology.orchestra.run.smithi118.stdout: "see_also": [] 2024-03-24T03:17:23.590 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-24T03:17:23.591 INFO:teuthology.orchestra.run.smithi118.stdout: "log_to_cluster": { 2024-03-24T03:17:23.591 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "log_to_cluster", 2024-03-24T03:17:23.591 INFO:teuthology.orchestra.run.smithi118.stdout: "type": "bool", 2024-03-24T03:17:23.591 INFO:teuthology.orchestra.run.smithi118.stdout: "level": "advanced", 2024-03-24T03:17:23.591 INFO:teuthology.orchestra.run.smithi118.stdout: "flags": 1, 2024-03-24T03:17:23.591 INFO:teuthology.orchestra.run.smithi118.stdout: "default_value": "False", 2024-03-24T03:17:23.592 INFO:teuthology.orchestra.run.smithi118.stdout: "min": "", 2024-03-24T03:17:23.592 INFO:teuthology.orchestra.run.smithi118.stdout: "max": "", 2024-03-24T03:17:23.592 INFO:teuthology.orchestra.run.smithi118.stdout: "enum_allowed": [], 2024-03-24T03:17:23.592 INFO:teuthology.orchestra.run.smithi118.stdout: "desc": "", 2024-03-24T03:17:23.592 INFO:teuthology.orchestra.run.smithi118.stdout: "long_desc": "", 2024-03-24T03:17:23.592 INFO:teuthology.orchestra.run.smithi118.stdout: "tags": [], 2024-03-24T03:17:23.592 INFO:teuthology.orchestra.run.smithi118.stdout: "see_also": [] 2024-03-24T03:17:23.593 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-24T03:17:23.593 INFO:teuthology.orchestra.run.smithi118.stdout: "log_to_cluster_level": { 2024-03-24T03:17:23.593 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "log_to_cluster_level", 2024-03-24T03:17:23.593 INFO:teuthology.orchestra.run.smithi118.stdout: "type": "str", 2024-03-24T03:17:23.593 INFO:teuthology.orchestra.run.smithi118.stdout: "level": "advanced", 2024-03-24T03:17:23.593 INFO:teuthology.orchestra.run.smithi118.stdout: "flags": 1, 2024-03-24T03:17:23.593 INFO:teuthology.orchestra.run.smithi118.stdout: "default_value": "info", 2024-03-24T03:17:23.593 INFO:teuthology.orchestra.run.smithi118.stdout: "min": "", 2024-03-24T03:17:23.593 INFO:teuthology.orchestra.run.smithi118.stdout: "max": "", 2024-03-24T03:17:23.594 INFO:teuthology.orchestra.run.smithi118.stdout: "enum_allowed": [ 2024-03-24T03:17:23.594 INFO:teuthology.orchestra.run.smithi118.stdout: "", 2024-03-24T03:17:23.594 INFO:teuthology.orchestra.run.smithi118.stdout: "critical", 2024-03-24T03:17:23.594 INFO:teuthology.orchestra.run.smithi118.stdout: "debug", 2024-03-24T03:17:23.594 INFO:teuthology.orchestra.run.smithi118.stdout: "error", 2024-03-24T03:17:23.594 INFO:teuthology.orchestra.run.smithi118.stdout: "info", 2024-03-24T03:17:23.594 INFO:teuthology.orchestra.run.smithi118.stdout: "warning" 2024-03-24T03:17:23.595 INFO:teuthology.orchestra.run.smithi118.stdout: ], 2024-03-24T03:17:23.595 INFO:teuthology.orchestra.run.smithi118.stdout: "desc": "", 2024-03-24T03:17:23.595 INFO:teuthology.orchestra.run.smithi118.stdout: "long_desc": "", 2024-03-24T03:17:23.595 INFO:teuthology.orchestra.run.smithi118.stdout: "tags": [], 2024-03-24T03:17:23.595 INFO:teuthology.orchestra.run.smithi118.stdout: "see_also": [] 2024-03-24T03:17:23.595 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-24T03:17:23.595 INFO:teuthology.orchestra.run.smithi118.stdout: "log_to_file": { 2024-03-24T03:17:23.595 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "log_to_file", 2024-03-24T03:17:23.595 INFO:teuthology.orchestra.run.smithi118.stdout: "type": "bool", 2024-03-24T03:17:23.596 INFO:teuthology.orchestra.run.smithi118.stdout: "level": "advanced", 2024-03-24T03:17:23.596 INFO:teuthology.orchestra.run.smithi118.stdout: "flags": 1, 2024-03-24T03:17:23.596 INFO:teuthology.orchestra.run.smithi118.stdout: "default_value": "False", 2024-03-24T03:17:23.596 INFO:teuthology.orchestra.run.smithi118.stdout: "min": "", 2024-03-24T03:17:23.596 INFO:teuthology.orchestra.run.smithi118.stdout: "max": "", 2024-03-24T03:17:23.596 INFO:teuthology.orchestra.run.smithi118.stdout: "enum_allowed": [], 2024-03-24T03:17:23.596 INFO:teuthology.orchestra.run.smithi118.stdout: "desc": "", 2024-03-24T03:17:23.596 INFO:teuthology.orchestra.run.smithi118.stdout: "long_desc": "", 2024-03-24T03:17:23.596 INFO:teuthology.orchestra.run.smithi118.stdout: "tags": [], 2024-03-24T03:17:23.597 INFO:teuthology.orchestra.run.smithi118.stdout: "see_also": [] 2024-03-24T03:17:23.597 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-24T03:17:23.597 INFO:teuthology.orchestra.run.smithi118.stdout: "orchestrator": { 2024-03-24T03:17:23.597 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "orchestrator", 2024-03-24T03:17:23.597 INFO:teuthology.orchestra.run.smithi118.stdout: "type": "str", 2024-03-24T03:17:23.597 INFO:teuthology.orchestra.run.smithi118.stdout: "level": "advanced", 2024-03-24T03:17:23.597 INFO:teuthology.orchestra.run.smithi118.stdout: "flags": 1, 2024-03-24T03:17:23.597 INFO:teuthology.orchestra.run.smithi118.stdout: "default_value": "", 2024-03-24T03:17:23.597 INFO:teuthology.orchestra.run.smithi118.stdout: "min": "", 2024-03-24T03:17:23.598 INFO:teuthology.orchestra.run.smithi118.stdout: "max": "", 2024-03-24T03:17:23.598 INFO:teuthology.orchestra.run.smithi118.stdout: "enum_allowed": [ 2024-03-24T03:17:23.598 INFO:teuthology.orchestra.run.smithi118.stdout: "cephadm", 2024-03-24T03:17:23.598 INFO:teuthology.orchestra.run.smithi118.stdout: "rook", 2024-03-24T03:17:23.598 INFO:teuthology.orchestra.run.smithi118.stdout: "test_orchestrator" 2024-03-24T03:17:23.598 INFO:teuthology.orchestra.run.smithi118.stdout: ], 2024-03-24T03:17:23.598 INFO:teuthology.orchestra.run.smithi118.stdout: "desc": "Orchestrator backend", 2024-03-24T03:17:23.598 INFO:teuthology.orchestra.run.smithi118.stdout: "long_desc": "", 2024-03-24T03:17:23.598 INFO:teuthology.orchestra.run.smithi118.stdout: "tags": [], 2024-03-24T03:17:23.599 INFO:teuthology.orchestra.run.smithi118.stdout: "see_also": [] 2024-03-24T03:17:23.599 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-03-24T03:17:23.599 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-03-24T03:17:23.599 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-24T03:17:23.599 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-24T03:17:23.599 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "osd_perf_query", 2024-03-24T03:17:23.599 INFO:teuthology.orchestra.run.smithi118.stdout: "can_run": true, 2024-03-24T03:17:23.599 INFO:teuthology.orchestra.run.smithi118.stdout: "error_string": "", 2024-03-24T03:17:23.599 INFO:teuthology.orchestra.run.smithi118.stdout: "module_options": { 2024-03-24T03:17:23.600 INFO:teuthology.orchestra.run.smithi118.stdout: "log_level": { 2024-03-24T03:17:23.600 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "log_level", 2024-03-24T03:17:23.600 INFO:teuthology.orchestra.run.smithi118.stdout: "type": "str", 2024-03-24T03:17:23.600 INFO:teuthology.orchestra.run.smithi118.stdout: "level": "advanced", 2024-03-24T03:17:23.600 INFO:teuthology.orchestra.run.smithi118.stdout: "flags": 1, 2024-03-24T03:17:23.600 INFO:teuthology.orchestra.run.smithi118.stdout: "default_value": "", 2024-03-24T03:17:23.600 INFO:teuthology.orchestra.run.smithi118.stdout: "min": "", 2024-03-24T03:17:23.600 INFO:teuthology.orchestra.run.smithi118.stdout: "max": "", 2024-03-24T03:17:23.600 INFO:teuthology.orchestra.run.smithi118.stdout: "enum_allowed": [ 2024-03-24T03:17:23.601 INFO:teuthology.orchestra.run.smithi118.stdout: "", 2024-03-24T03:17:23.601 INFO:teuthology.orchestra.run.smithi118.stdout: "critical", 2024-03-24T03:17:23.601 INFO:teuthology.orchestra.run.smithi118.stdout: "debug", 2024-03-24T03:17:23.601 INFO:teuthology.orchestra.run.smithi118.stdout: "error", 2024-03-24T03:17:23.601 INFO:teuthology.orchestra.run.smithi118.stdout: "info", 2024-03-24T03:17:23.601 INFO:teuthology.orchestra.run.smithi118.stdout: "warning" 2024-03-24T03:17:23.601 INFO:teuthology.orchestra.run.smithi118.stdout: ], 2024-03-24T03:17:23.601 INFO:teuthology.orchestra.run.smithi118.stdout: "desc": "", 2024-03-24T03:17:23.602 INFO:teuthology.orchestra.run.smithi118.stdout: "long_desc": "", 2024-03-24T03:17:23.602 INFO:teuthology.orchestra.run.smithi118.stdout: "tags": [], 2024-03-24T03:17:23.602 INFO:teuthology.orchestra.run.smithi118.stdout: "see_also": [] 2024-03-24T03:17:23.602 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-24T03:17:23.602 INFO:teuthology.orchestra.run.smithi118.stdout: "log_to_cluster": { 2024-03-24T03:17:23.602 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "log_to_cluster", 2024-03-24T03:17:23.602 INFO:teuthology.orchestra.run.smithi118.stdout: "type": "bool", 2024-03-24T03:17:23.602 INFO:teuthology.orchestra.run.smithi118.stdout: "level": "advanced", 2024-03-24T03:17:23.602 INFO:teuthology.orchestra.run.smithi118.stdout: "flags": 1, 2024-03-24T03:17:23.603 INFO:teuthology.orchestra.run.smithi118.stdout: "default_value": "False", 2024-03-24T03:17:23.603 INFO:teuthology.orchestra.run.smithi118.stdout: "min": "", 2024-03-24T03:17:23.603 INFO:teuthology.orchestra.run.smithi118.stdout: "max": "", 2024-03-24T03:17:23.603 INFO:teuthology.orchestra.run.smithi118.stdout: "enum_allowed": [], 2024-03-24T03:17:23.603 INFO:teuthology.orchestra.run.smithi118.stdout: "desc": "", 2024-03-24T03:17:23.603 INFO:teuthology.orchestra.run.smithi118.stdout: "long_desc": "", 2024-03-24T03:17:23.603 INFO:teuthology.orchestra.run.smithi118.stdout: "tags": [], 2024-03-24T03:17:23.603 INFO:teuthology.orchestra.run.smithi118.stdout: "see_also": [] 2024-03-24T03:17:23.603 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-24T03:17:23.604 INFO:teuthology.orchestra.run.smithi118.stdout: "log_to_cluster_level": { 2024-03-24T03:17:23.604 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "log_to_cluster_level", 2024-03-24T03:17:23.604 INFO:teuthology.orchestra.run.smithi118.stdout: "type": "str", 2024-03-24T03:17:23.604 INFO:teuthology.orchestra.run.smithi118.stdout: "level": "advanced", 2024-03-24T03:17:23.604 INFO:teuthology.orchestra.run.smithi118.stdout: "flags": 1, 2024-03-24T03:17:23.604 INFO:teuthology.orchestra.run.smithi118.stdout: "default_value": "info", 2024-03-24T03:17:23.604 INFO:teuthology.orchestra.run.smithi118.stdout: "min": "", 2024-03-24T03:17:23.604 INFO:teuthology.orchestra.run.smithi118.stdout: "max": "", 2024-03-24T03:17:23.605 INFO:teuthology.orchestra.run.smithi118.stdout: "enum_allowed": [ 2024-03-24T03:17:23.605 INFO:teuthology.orchestra.run.smithi118.stdout: "", 2024-03-24T03:17:23.605 INFO:teuthology.orchestra.run.smithi118.stdout: "critical", 2024-03-24T03:17:23.605 INFO:teuthology.orchestra.run.smithi118.stdout: "debug", 2024-03-24T03:17:23.605 INFO:teuthology.orchestra.run.smithi118.stdout: "error", 2024-03-24T03:17:23.605 INFO:teuthology.orchestra.run.smithi118.stdout: "info", 2024-03-24T03:17:23.605 INFO:teuthology.orchestra.run.smithi118.stdout: "warning" 2024-03-24T03:17:23.605 INFO:teuthology.orchestra.run.smithi118.stdout: ], 2024-03-24T03:17:23.605 INFO:teuthology.orchestra.run.smithi118.stdout: "desc": "", 2024-03-24T03:17:23.605 INFO:teuthology.orchestra.run.smithi118.stdout: "long_desc": "", 2024-03-24T03:17:23.606 INFO:teuthology.orchestra.run.smithi118.stdout: "tags": [], 2024-03-24T03:17:23.606 INFO:teuthology.orchestra.run.smithi118.stdout: "see_also": [] 2024-03-24T03:17:23.606 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-24T03:17:23.606 INFO:teuthology.orchestra.run.smithi118.stdout: "log_to_file": { 2024-03-24T03:17:23.606 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "log_to_file", 2024-03-24T03:17:23.606 INFO:teuthology.orchestra.run.smithi118.stdout: "type": "bool", 2024-03-24T03:17:23.606 INFO:teuthology.orchestra.run.smithi118.stdout: "level": "advanced", 2024-03-24T03:17:23.607 INFO:teuthology.orchestra.run.smithi118.stdout: "flags": 1, 2024-03-24T03:17:23.607 INFO:teuthology.orchestra.run.smithi118.stdout: "default_value": "False", 2024-03-24T03:17:23.607 INFO:teuthology.orchestra.run.smithi118.stdout: "min": "", 2024-03-24T03:17:23.607 INFO:teuthology.orchestra.run.smithi118.stdout: "max": "", 2024-03-24T03:17:23.607 INFO:teuthology.orchestra.run.smithi118.stdout: "enum_allowed": [], 2024-03-24T03:17:23.607 INFO:teuthology.orchestra.run.smithi118.stdout: "desc": "", 2024-03-24T03:17:23.607 INFO:teuthology.orchestra.run.smithi118.stdout: "long_desc": "", 2024-03-24T03:17:23.607 INFO:teuthology.orchestra.run.smithi118.stdout: "tags": [], 2024-03-24T03:17:23.607 INFO:teuthology.orchestra.run.smithi118.stdout: "see_also": [] 2024-03-24T03:17:23.607 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-03-24T03:17:23.608 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-03-24T03:17:23.608 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-24T03:17:23.608 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-24T03:17:23.608 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "osd_support", 2024-03-24T03:17:23.608 INFO:teuthology.orchestra.run.smithi118.stdout: "can_run": true, 2024-03-24T03:17:23.608 INFO:teuthology.orchestra.run.smithi118.stdout: "error_string": "", 2024-03-24T03:17:23.608 INFO:teuthology.orchestra.run.smithi118.stdout: "module_options": { 2024-03-24T03:17:23.608 INFO:teuthology.orchestra.run.smithi118.stdout: "log_level": { 2024-03-24T03:17:23.608 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "log_level", 2024-03-24T03:17:23.609 INFO:teuthology.orchestra.run.smithi118.stdout: "type": "str", 2024-03-24T03:17:23.609 INFO:teuthology.orchestra.run.smithi118.stdout: "level": "advanced", 2024-03-24T03:17:23.609 INFO:teuthology.orchestra.run.smithi118.stdout: "flags": 1, 2024-03-24T03:17:23.609 INFO:teuthology.orchestra.run.smithi118.stdout: "default_value": "", 2024-03-24T03:17:23.609 INFO:teuthology.orchestra.run.smithi118.stdout: "min": "", 2024-03-24T03:17:23.609 INFO:teuthology.orchestra.run.smithi118.stdout: "max": "", 2024-03-24T03:17:23.609 INFO:teuthology.orchestra.run.smithi118.stdout: "enum_allowed": [ 2024-03-24T03:17:23.609 INFO:teuthology.orchestra.run.smithi118.stdout: "", 2024-03-24T03:17:23.609 INFO:teuthology.orchestra.run.smithi118.stdout: "critical", 2024-03-24T03:17:23.610 INFO:teuthology.orchestra.run.smithi118.stdout: "debug", 2024-03-24T03:17:23.610 INFO:teuthology.orchestra.run.smithi118.stdout: "error", 2024-03-24T03:17:23.610 INFO:teuthology.orchestra.run.smithi118.stdout: "info", 2024-03-24T03:17:23.610 INFO:teuthology.orchestra.run.smithi118.stdout: "warning" 2024-03-24T03:17:23.610 INFO:teuthology.orchestra.run.smithi118.stdout: ], 2024-03-24T03:17:23.610 INFO:teuthology.orchestra.run.smithi118.stdout: "desc": "", 2024-03-24T03:17:23.610 INFO:teuthology.orchestra.run.smithi118.stdout: "long_desc": "", 2024-03-24T03:17:23.610 INFO:teuthology.orchestra.run.smithi118.stdout: "tags": [], 2024-03-24T03:17:23.611 INFO:teuthology.orchestra.run.smithi118.stdout: "see_also": [] 2024-03-24T03:17:23.611 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-24T03:17:23.611 INFO:teuthology.orchestra.run.smithi118.stdout: "log_to_cluster": { 2024-03-24T03:17:23.611 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "log_to_cluster", 2024-03-24T03:17:23.611 INFO:teuthology.orchestra.run.smithi118.stdout: "type": "bool", 2024-03-24T03:17:23.611 INFO:teuthology.orchestra.run.smithi118.stdout: "level": "advanced", 2024-03-24T03:17:23.611 INFO:teuthology.orchestra.run.smithi118.stdout: "flags": 1, 2024-03-24T03:17:23.611 INFO:teuthology.orchestra.run.smithi118.stdout: "default_value": "False", 2024-03-24T03:17:23.612 INFO:teuthology.orchestra.run.smithi118.stdout: "min": "", 2024-03-24T03:17:23.612 INFO:teuthology.orchestra.run.smithi118.stdout: "max": "", 2024-03-24T03:17:23.612 INFO:teuthology.orchestra.run.smithi118.stdout: "enum_allowed": [], 2024-03-24T03:17:23.612 INFO:teuthology.orchestra.run.smithi118.stdout: "desc": "", 2024-03-24T03:17:23.612 INFO:teuthology.orchestra.run.smithi118.stdout: "long_desc": "", 2024-03-24T03:17:23.612 INFO:teuthology.orchestra.run.smithi118.stdout: "tags": [], 2024-03-24T03:17:23.612 INFO:teuthology.orchestra.run.smithi118.stdout: "see_also": [] 2024-03-24T03:17:23.612 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-24T03:17:23.612 INFO:teuthology.orchestra.run.smithi118.stdout: "log_to_cluster_level": { 2024-03-24T03:17:23.613 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "log_to_cluster_level", 2024-03-24T03:17:23.613 INFO:teuthology.orchestra.run.smithi118.stdout: "type": "str", 2024-03-24T03:17:23.613 INFO:teuthology.orchestra.run.smithi118.stdout: "level": "advanced", 2024-03-24T03:17:23.613 INFO:teuthology.orchestra.run.smithi118.stdout: "flags": 1, 2024-03-24T03:17:23.613 INFO:teuthology.orchestra.run.smithi118.stdout: "default_value": "info", 2024-03-24T03:17:23.613 INFO:teuthology.orchestra.run.smithi118.stdout: "min": "", 2024-03-24T03:17:23.613 INFO:teuthology.orchestra.run.smithi118.stdout: "max": "", 2024-03-24T03:17:23.614 INFO:teuthology.orchestra.run.smithi118.stdout: "enum_allowed": [ 2024-03-24T03:17:23.614 INFO:teuthology.orchestra.run.smithi118.stdout: "", 2024-03-24T03:17:23.614 INFO:teuthology.orchestra.run.smithi118.stdout: "critical", 2024-03-24T03:17:23.614 INFO:teuthology.orchestra.run.smithi118.stdout: "debug", 2024-03-24T03:17:23.614 INFO:teuthology.orchestra.run.smithi118.stdout: "error", 2024-03-24T03:17:23.614 INFO:teuthology.orchestra.run.smithi118.stdout: "info", 2024-03-24T03:17:23.615 INFO:teuthology.orchestra.run.smithi118.stdout: "warning" 2024-03-24T03:17:23.615 INFO:teuthology.orchestra.run.smithi118.stdout: ], 2024-03-24T03:17:23.615 INFO:teuthology.orchestra.run.smithi118.stdout: "desc": "", 2024-03-24T03:17:23.615 INFO:teuthology.orchestra.run.smithi118.stdout: "long_desc": "", 2024-03-24T03:17:23.615 INFO:teuthology.orchestra.run.smithi118.stdout: "tags": [], 2024-03-24T03:17:23.615 INFO:teuthology.orchestra.run.smithi118.stdout: "see_also": [] 2024-03-24T03:17:23.615 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-24T03:17:23.615 INFO:teuthology.orchestra.run.smithi118.stdout: "log_to_file": { 2024-03-24T03:17:23.616 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "log_to_file", 2024-03-24T03:17:23.616 INFO:teuthology.orchestra.run.smithi118.stdout: "type": "bool", 2024-03-24T03:17:23.616 INFO:teuthology.orchestra.run.smithi118.stdout: "level": "advanced", 2024-03-24T03:17:23.616 INFO:teuthology.orchestra.run.smithi118.stdout: "flags": 1, 2024-03-24T03:17:23.616 INFO:teuthology.orchestra.run.smithi118.stdout: "default_value": "False", 2024-03-24T03:17:23.616 INFO:teuthology.orchestra.run.smithi118.stdout: "min": "", 2024-03-24T03:17:23.616 INFO:teuthology.orchestra.run.smithi118.stdout: "max": "", 2024-03-24T03:17:23.616 INFO:teuthology.orchestra.run.smithi118.stdout: "enum_allowed": [], 2024-03-24T03:17:23.616 INFO:teuthology.orchestra.run.smithi118.stdout: "desc": "", 2024-03-24T03:17:23.617 INFO:teuthology.orchestra.run.smithi118.stdout: "long_desc": "", 2024-03-24T03:17:23.617 INFO:teuthology.orchestra.run.smithi118.stdout: "tags": [], 2024-03-24T03:17:23.617 INFO:teuthology.orchestra.run.smithi118.stdout: "see_also": [] 2024-03-24T03:17:23.617 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-03-24T03:17:23.617 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-03-24T03:17:23.617 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-24T03:17:23.617 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-24T03:17:23.617 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "pg_autoscaler", 2024-03-24T03:17:23.617 INFO:teuthology.orchestra.run.smithi118.stdout: "can_run": true, 2024-03-24T03:17:23.618 INFO:teuthology.orchestra.run.smithi118.stdout: "error_string": "", 2024-03-24T03:17:23.618 INFO:teuthology.orchestra.run.smithi118.stdout: "module_options": { 2024-03-24T03:17:23.618 INFO:teuthology.orchestra.run.smithi118.stdout: "log_level": { 2024-03-24T03:17:23.618 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "log_level", 2024-03-24T03:17:23.618 INFO:teuthology.orchestra.run.smithi118.stdout: "type": "str", 2024-03-24T03:17:23.618 INFO:teuthology.orchestra.run.smithi118.stdout: "level": "advanced", 2024-03-24T03:17:23.618 INFO:teuthology.orchestra.run.smithi118.stdout: "flags": 1, 2024-03-24T03:17:23.619 INFO:teuthology.orchestra.run.smithi118.stdout: "default_value": "", 2024-03-24T03:17:23.619 INFO:teuthology.orchestra.run.smithi118.stdout: "min": "", 2024-03-24T03:17:23.619 INFO:teuthology.orchestra.run.smithi118.stdout: "max": "", 2024-03-24T03:17:23.619 INFO:teuthology.orchestra.run.smithi118.stdout: "enum_allowed": [ 2024-03-24T03:17:23.619 INFO:teuthology.orchestra.run.smithi118.stdout: "", 2024-03-24T03:17:23.619 INFO:teuthology.orchestra.run.smithi118.stdout: "critical", 2024-03-24T03:17:23.619 INFO:teuthology.orchestra.run.smithi118.stdout: "debug", 2024-03-24T03:17:23.619 INFO:teuthology.orchestra.run.smithi118.stdout: "error", 2024-03-24T03:17:23.619 INFO:teuthology.orchestra.run.smithi118.stdout: "info", 2024-03-24T03:17:23.620 INFO:teuthology.orchestra.run.smithi118.stdout: "warning" 2024-03-24T03:17:23.620 INFO:teuthology.orchestra.run.smithi118.stdout: ], 2024-03-24T03:17:23.620 INFO:teuthology.orchestra.run.smithi118.stdout: "desc": "", 2024-03-24T03:17:23.620 INFO:teuthology.orchestra.run.smithi118.stdout: "long_desc": "", 2024-03-24T03:17:23.620 INFO:teuthology.orchestra.run.smithi118.stdout: "tags": [], 2024-03-24T03:17:23.620 INFO:teuthology.orchestra.run.smithi118.stdout: "see_also": [] 2024-03-24T03:17:23.620 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-24T03:17:23.620 INFO:teuthology.orchestra.run.smithi118.stdout: "log_to_cluster": { 2024-03-24T03:17:23.620 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "log_to_cluster", 2024-03-24T03:17:23.620 INFO:teuthology.orchestra.run.smithi118.stdout: "type": "bool", 2024-03-24T03:17:23.621 INFO:teuthology.orchestra.run.smithi118.stdout: "level": "advanced", 2024-03-24T03:17:23.621 INFO:teuthology.orchestra.run.smithi118.stdout: "flags": 1, 2024-03-24T03:17:23.621 INFO:teuthology.orchestra.run.smithi118.stdout: "default_value": "False", 2024-03-24T03:17:23.621 INFO:teuthology.orchestra.run.smithi118.stdout: "min": "", 2024-03-24T03:17:23.621 INFO:teuthology.orchestra.run.smithi118.stdout: "max": "", 2024-03-24T03:17:23.621 INFO:teuthology.orchestra.run.smithi118.stdout: "enum_allowed": [], 2024-03-24T03:17:23.621 INFO:teuthology.orchestra.run.smithi118.stdout: "desc": "", 2024-03-24T03:17:23.621 INFO:teuthology.orchestra.run.smithi118.stdout: "long_desc": "", 2024-03-24T03:17:23.621 INFO:teuthology.orchestra.run.smithi118.stdout: "tags": [], 2024-03-24T03:17:23.622 INFO:teuthology.orchestra.run.smithi118.stdout: "see_also": [] 2024-03-24T03:17:23.622 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-24T03:17:23.622 INFO:teuthology.orchestra.run.smithi118.stdout: "log_to_cluster_level": { 2024-03-24T03:17:23.622 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "log_to_cluster_level", 2024-03-24T03:17:23.622 INFO:teuthology.orchestra.run.smithi118.stdout: "type": "str", 2024-03-24T03:17:23.622 INFO:teuthology.orchestra.run.smithi118.stdout: "level": "advanced", 2024-03-24T03:17:23.622 INFO:teuthology.orchestra.run.smithi118.stdout: "flags": 1, 2024-03-24T03:17:23.622 INFO:teuthology.orchestra.run.smithi118.stdout: "default_value": "info", 2024-03-24T03:17:23.622 INFO:teuthology.orchestra.run.smithi118.stdout: "min": "", 2024-03-24T03:17:23.623 INFO:teuthology.orchestra.run.smithi118.stdout: "max": "", 2024-03-24T03:17:23.623 INFO:teuthology.orchestra.run.smithi118.stdout: "enum_allowed": [ 2024-03-24T03:17:23.623 INFO:teuthology.orchestra.run.smithi118.stdout: "", 2024-03-24T03:17:23.623 INFO:teuthology.orchestra.run.smithi118.stdout: "critical", 2024-03-24T03:17:23.623 INFO:teuthology.orchestra.run.smithi118.stdout: "debug", 2024-03-24T03:17:23.623 INFO:teuthology.orchestra.run.smithi118.stdout: "error", 2024-03-24T03:17:23.623 INFO:teuthology.orchestra.run.smithi118.stdout: "info", 2024-03-24T03:17:23.623 INFO:teuthology.orchestra.run.smithi118.stdout: "warning" 2024-03-24T03:17:23.623 INFO:teuthology.orchestra.run.smithi118.stdout: ], 2024-03-24T03:17:23.624 INFO:teuthology.orchestra.run.smithi118.stdout: "desc": "", 2024-03-24T03:17:23.624 INFO:teuthology.orchestra.run.smithi118.stdout: "long_desc": "", 2024-03-24T03:17:23.624 INFO:teuthology.orchestra.run.smithi118.stdout: "tags": [], 2024-03-24T03:17:23.624 INFO:teuthology.orchestra.run.smithi118.stdout: "see_also": [] 2024-03-24T03:17:23.624 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-24T03:17:23.624 INFO:teuthology.orchestra.run.smithi118.stdout: "log_to_file": { 2024-03-24T03:17:23.624 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "log_to_file", 2024-03-24T03:17:23.624 INFO:teuthology.orchestra.run.smithi118.stdout: "type": "bool", 2024-03-24T03:17:23.625 INFO:teuthology.orchestra.run.smithi118.stdout: "level": "advanced", 2024-03-24T03:17:23.625 INFO:teuthology.orchestra.run.smithi118.stdout: "flags": 1, 2024-03-24T03:17:23.625 INFO:teuthology.orchestra.run.smithi118.stdout: "default_value": "False", 2024-03-24T03:17:23.625 INFO:teuthology.orchestra.run.smithi118.stdout: "min": "", 2024-03-24T03:17:23.625 INFO:teuthology.orchestra.run.smithi118.stdout: "max": "", 2024-03-24T03:17:23.625 INFO:teuthology.orchestra.run.smithi118.stdout: "enum_allowed": [], 2024-03-24T03:17:23.625 INFO:teuthology.orchestra.run.smithi118.stdout: "desc": "", 2024-03-24T03:17:23.625 INFO:teuthology.orchestra.run.smithi118.stdout: "long_desc": "", 2024-03-24T03:17:23.625 INFO:teuthology.orchestra.run.smithi118.stdout: "tags": [], 2024-03-24T03:17:23.625 INFO:teuthology.orchestra.run.smithi118.stdout: "see_also": [] 2024-03-24T03:17:23.626 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-24T03:17:23.626 INFO:teuthology.orchestra.run.smithi118.stdout: "sleep_interval": { 2024-03-24T03:17:23.626 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "sleep_interval", 2024-03-24T03:17:23.626 INFO:teuthology.orchestra.run.smithi118.stdout: "type": "secs", 2024-03-24T03:17:23.626 INFO:teuthology.orchestra.run.smithi118.stdout: "level": "advanced", 2024-03-24T03:17:23.626 INFO:teuthology.orchestra.run.smithi118.stdout: "flags": 0, 2024-03-24T03:17:23.626 INFO:teuthology.orchestra.run.smithi118.stdout: "default_value": "60", 2024-03-24T03:17:23.626 INFO:teuthology.orchestra.run.smithi118.stdout: "min": "", 2024-03-24T03:17:23.627 INFO:teuthology.orchestra.run.smithi118.stdout: "max": "", 2024-03-24T03:17:23.627 INFO:teuthology.orchestra.run.smithi118.stdout: "enum_allowed": [], 2024-03-24T03:17:23.627 INFO:teuthology.orchestra.run.smithi118.stdout: "desc": "", 2024-03-24T03:17:23.627 INFO:teuthology.orchestra.run.smithi118.stdout: "long_desc": "", 2024-03-24T03:17:23.627 INFO:teuthology.orchestra.run.smithi118.stdout: "tags": [], 2024-03-24T03:17:23.627 INFO:teuthology.orchestra.run.smithi118.stdout: "see_also": [] 2024-03-24T03:17:23.627 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-24T03:17:23.627 INFO:teuthology.orchestra.run.smithi118.stdout: "threshold": { 2024-03-24T03:17:23.627 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "threshold", 2024-03-24T03:17:23.628 INFO:teuthology.orchestra.run.smithi118.stdout: "type": "float", 2024-03-24T03:17:23.628 INFO:teuthology.orchestra.run.smithi118.stdout: "level": "advanced", 2024-03-24T03:17:23.628 INFO:teuthology.orchestra.run.smithi118.stdout: "flags": 0, 2024-03-24T03:17:23.628 INFO:teuthology.orchestra.run.smithi118.stdout: "default_value": "3.0", 2024-03-24T03:17:23.628 INFO:teuthology.orchestra.run.smithi118.stdout: "min": "1.0", 2024-03-24T03:17:23.628 INFO:teuthology.orchestra.run.smithi118.stdout: "max": "", 2024-03-24T03:17:23.628 INFO:teuthology.orchestra.run.smithi118.stdout: "enum_allowed": [], 2024-03-24T03:17:23.628 INFO:teuthology.orchestra.run.smithi118.stdout: "desc": "scaling threshold", 2024-03-24T03:17:23.628 INFO:teuthology.orchestra.run.smithi118.stdout: "long_desc": "The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0", 2024-03-24T03:17:23.628 INFO:teuthology.orchestra.run.smithi118.stdout: "tags": [], 2024-03-24T03:17:23.629 INFO:teuthology.orchestra.run.smithi118.stdout: "see_also": [] 2024-03-24T03:17:23.629 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-03-24T03:17:23.629 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-03-24T03:17:23.629 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-24T03:17:23.629 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-24T03:17:23.629 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "progress", 2024-03-24T03:17:23.629 INFO:teuthology.orchestra.run.smithi118.stdout: "can_run": true, 2024-03-24T03:17:23.629 INFO:teuthology.orchestra.run.smithi118.stdout: "error_string": "", 2024-03-24T03:17:23.629 INFO:teuthology.orchestra.run.smithi118.stdout: "module_options": { 2024-03-24T03:17:23.630 INFO:teuthology.orchestra.run.smithi118.stdout: "allow_pg_recovery_event": { 2024-03-24T03:17:23.630 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "allow_pg_recovery_event", 2024-03-24T03:17:23.630 INFO:teuthology.orchestra.run.smithi118.stdout: "type": "bool", 2024-03-24T03:17:23.630 INFO:teuthology.orchestra.run.smithi118.stdout: "level": "advanced", 2024-03-24T03:17:23.630 INFO:teuthology.orchestra.run.smithi118.stdout: "flags": 1, 2024-03-24T03:17:23.630 INFO:teuthology.orchestra.run.smithi118.stdout: "default_value": "False", 2024-03-24T03:17:23.630 INFO:teuthology.orchestra.run.smithi118.stdout: "min": "", 2024-03-24T03:17:23.630 INFO:teuthology.orchestra.run.smithi118.stdout: "max": "", 2024-03-24T03:17:23.631 INFO:teuthology.orchestra.run.smithi118.stdout: "enum_allowed": [], 2024-03-24T03:17:23.631 INFO:teuthology.orchestra.run.smithi118.stdout: "desc": "allow the module to show pg recovery progress", 2024-03-24T03:17:23.631 INFO:teuthology.orchestra.run.smithi118.stdout: "long_desc": "", 2024-03-24T03:17:23.631 INFO:teuthology.orchestra.run.smithi118.stdout: "tags": [], 2024-03-24T03:17:23.631 INFO:teuthology.orchestra.run.smithi118.stdout: "see_also": [] 2024-03-24T03:17:23.631 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-24T03:17:23.631 INFO:teuthology.orchestra.run.smithi118.stdout: "enabled": { 2024-03-24T03:17:23.631 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "enabled", 2024-03-24T03:17:23.631 INFO:teuthology.orchestra.run.smithi118.stdout: "type": "bool", 2024-03-24T03:17:23.632 INFO:teuthology.orchestra.run.smithi118.stdout: "level": "advanced", 2024-03-24T03:17:23.632 INFO:teuthology.orchestra.run.smithi118.stdout: "flags": 0, 2024-03-24T03:17:23.632 INFO:teuthology.orchestra.run.smithi118.stdout: "default_value": "True", 2024-03-24T03:17:23.632 INFO:teuthology.orchestra.run.smithi118.stdout: "min": "", 2024-03-24T03:17:23.632 INFO:teuthology.orchestra.run.smithi118.stdout: "max": "", 2024-03-24T03:17:23.632 INFO:teuthology.orchestra.run.smithi118.stdout: "enum_allowed": [], 2024-03-24T03:17:23.632 INFO:teuthology.orchestra.run.smithi118.stdout: "desc": "", 2024-03-24T03:17:23.632 INFO:teuthology.orchestra.run.smithi118.stdout: "long_desc": "", 2024-03-24T03:17:23.632 INFO:teuthology.orchestra.run.smithi118.stdout: "tags": [], 2024-03-24T03:17:23.633 INFO:teuthology.orchestra.run.smithi118.stdout: "see_also": [] 2024-03-24T03:17:23.633 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-24T03:17:23.633 INFO:teuthology.orchestra.run.smithi118.stdout: "log_level": { 2024-03-24T03:17:23.633 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "log_level", 2024-03-24T03:17:23.633 INFO:teuthology.orchestra.run.smithi118.stdout: "type": "str", 2024-03-24T03:17:23.633 INFO:teuthology.orchestra.run.smithi118.stdout: "level": "advanced", 2024-03-24T03:17:23.633 INFO:teuthology.orchestra.run.smithi118.stdout: "flags": 1, 2024-03-24T03:17:23.633 INFO:teuthology.orchestra.run.smithi118.stdout: "default_value": "", 2024-03-24T03:17:23.634 INFO:teuthology.orchestra.run.smithi118.stdout: "min": "", 2024-03-24T03:17:23.634 INFO:teuthology.orchestra.run.smithi118.stdout: "max": "", 2024-03-24T03:17:23.634 INFO:teuthology.orchestra.run.smithi118.stdout: "enum_allowed": [ 2024-03-24T03:17:23.634 INFO:teuthology.orchestra.run.smithi118.stdout: "", 2024-03-24T03:17:23.634 INFO:teuthology.orchestra.run.smithi118.stdout: "critical", 2024-03-24T03:17:23.634 INFO:teuthology.orchestra.run.smithi118.stdout: "debug", 2024-03-24T03:17:23.634 INFO:teuthology.orchestra.run.smithi118.stdout: "error", 2024-03-24T03:17:23.634 INFO:teuthology.orchestra.run.smithi118.stdout: "info", 2024-03-24T03:17:23.635 INFO:teuthology.orchestra.run.smithi118.stdout: "warning" 2024-03-24T03:17:23.635 INFO:teuthology.orchestra.run.smithi118.stdout: ], 2024-03-24T03:17:23.635 INFO:teuthology.orchestra.run.smithi118.stdout: "desc": "", 2024-03-24T03:17:23.635 INFO:teuthology.orchestra.run.smithi118.stdout: "long_desc": "", 2024-03-24T03:17:23.635 INFO:teuthology.orchestra.run.smithi118.stdout: "tags": [], 2024-03-24T03:17:23.635 INFO:teuthology.orchestra.run.smithi118.stdout: "see_also": [] 2024-03-24T03:17:23.635 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-24T03:17:23.635 INFO:teuthology.orchestra.run.smithi118.stdout: "log_to_cluster": { 2024-03-24T03:17:23.635 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "log_to_cluster", 2024-03-24T03:17:23.636 INFO:teuthology.orchestra.run.smithi118.stdout: "type": "bool", 2024-03-24T03:17:23.636 INFO:teuthology.orchestra.run.smithi118.stdout: "level": "advanced", 2024-03-24T03:17:23.636 INFO:teuthology.orchestra.run.smithi118.stdout: "flags": 1, 2024-03-24T03:17:23.636 INFO:teuthology.orchestra.run.smithi118.stdout: "default_value": "False", 2024-03-24T03:17:23.636 INFO:teuthology.orchestra.run.smithi118.stdout: "min": "", 2024-03-24T03:17:23.636 INFO:teuthology.orchestra.run.smithi118.stdout: "max": "", 2024-03-24T03:17:23.636 INFO:teuthology.orchestra.run.smithi118.stdout: "enum_allowed": [], 2024-03-24T03:17:23.636 INFO:teuthology.orchestra.run.smithi118.stdout: "desc": "", 2024-03-24T03:17:23.636 INFO:teuthology.orchestra.run.smithi118.stdout: "long_desc": "", 2024-03-24T03:17:23.637 INFO:teuthology.orchestra.run.smithi118.stdout: "tags": [], 2024-03-24T03:17:23.637 INFO:teuthology.orchestra.run.smithi118.stdout: "see_also": [] 2024-03-24T03:17:23.637 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-24T03:17:23.637 INFO:teuthology.orchestra.run.smithi118.stdout: "log_to_cluster_level": { 2024-03-24T03:17:23.637 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "log_to_cluster_level", 2024-03-24T03:17:23.637 INFO:teuthology.orchestra.run.smithi118.stdout: "type": "str", 2024-03-24T03:17:23.637 INFO:teuthology.orchestra.run.smithi118.stdout: "level": "advanced", 2024-03-24T03:17:23.637 INFO:teuthology.orchestra.run.smithi118.stdout: "flags": 1, 2024-03-24T03:17:23.638 INFO:teuthology.orchestra.run.smithi118.stdout: "default_value": "info", 2024-03-24T03:17:23.638 INFO:teuthology.orchestra.run.smithi118.stdout: "min": "", 2024-03-24T03:17:23.638 INFO:teuthology.orchestra.run.smithi118.stdout: "max": "", 2024-03-24T03:17:23.638 INFO:teuthology.orchestra.run.smithi118.stdout: "enum_allowed": [ 2024-03-24T03:17:23.638 INFO:teuthology.orchestra.run.smithi118.stdout: "", 2024-03-24T03:17:23.638 INFO:teuthology.orchestra.run.smithi118.stdout: "critical", 2024-03-24T03:17:23.638 INFO:teuthology.orchestra.run.smithi118.stdout: "debug", 2024-03-24T03:17:23.638 INFO:teuthology.orchestra.run.smithi118.stdout: "error", 2024-03-24T03:17:23.639 INFO:teuthology.orchestra.run.smithi118.stdout: "info", 2024-03-24T03:17:23.639 INFO:teuthology.orchestra.run.smithi118.stdout: "warning" 2024-03-24T03:17:23.639 INFO:teuthology.orchestra.run.smithi118.stdout: ], 2024-03-24T03:17:23.639 INFO:teuthology.orchestra.run.smithi118.stdout: "desc": "", 2024-03-24T03:17:23.639 INFO:teuthology.orchestra.run.smithi118.stdout: "long_desc": "", 2024-03-24T03:17:23.639 INFO:teuthology.orchestra.run.smithi118.stdout: "tags": [], 2024-03-24T03:17:23.639 INFO:teuthology.orchestra.run.smithi118.stdout: "see_also": [] 2024-03-24T03:17:23.639 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-24T03:17:23.639 INFO:teuthology.orchestra.run.smithi118.stdout: "log_to_file": { 2024-03-24T03:17:23.640 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "log_to_file", 2024-03-24T03:17:23.640 INFO:teuthology.orchestra.run.smithi118.stdout: "type": "bool", 2024-03-24T03:17:23.640 INFO:teuthology.orchestra.run.smithi118.stdout: "level": "advanced", 2024-03-24T03:17:23.640 INFO:teuthology.orchestra.run.smithi118.stdout: "flags": 1, 2024-03-24T03:17:23.640 INFO:teuthology.orchestra.run.smithi118.stdout: "default_value": "False", 2024-03-24T03:17:23.640 INFO:teuthology.orchestra.run.smithi118.stdout: "min": "", 2024-03-24T03:17:23.640 INFO:teuthology.orchestra.run.smithi118.stdout: "max": "", 2024-03-24T03:17:23.640 INFO:teuthology.orchestra.run.smithi118.stdout: "enum_allowed": [], 2024-03-24T03:17:23.640 INFO:teuthology.orchestra.run.smithi118.stdout: "desc": "", 2024-03-24T03:17:23.641 INFO:teuthology.orchestra.run.smithi118.stdout: "long_desc": "", 2024-03-24T03:17:23.641 INFO:teuthology.orchestra.run.smithi118.stdout: "tags": [], 2024-03-24T03:17:23.641 INFO:teuthology.orchestra.run.smithi118.stdout: "see_also": [] 2024-03-24T03:17:23.641 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-24T03:17:23.641 INFO:teuthology.orchestra.run.smithi118.stdout: "max_completed_events": { 2024-03-24T03:17:23.641 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "max_completed_events", 2024-03-24T03:17:23.641 INFO:teuthology.orchestra.run.smithi118.stdout: "type": "int", 2024-03-24T03:17:23.641 INFO:teuthology.orchestra.run.smithi118.stdout: "level": "advanced", 2024-03-24T03:17:23.642 INFO:teuthology.orchestra.run.smithi118.stdout: "flags": 1, 2024-03-24T03:17:23.642 INFO:teuthology.orchestra.run.smithi118.stdout: "default_value": "50", 2024-03-24T03:17:23.642 INFO:teuthology.orchestra.run.smithi118.stdout: "min": "", 2024-03-24T03:17:23.642 INFO:teuthology.orchestra.run.smithi118.stdout: "max": "", 2024-03-24T03:17:23.642 INFO:teuthology.orchestra.run.smithi118.stdout: "enum_allowed": [], 2024-03-24T03:17:23.642 INFO:teuthology.orchestra.run.smithi118.stdout: "desc": "number of past completed events to remember", 2024-03-24T03:17:23.642 INFO:teuthology.orchestra.run.smithi118.stdout: "long_desc": "", 2024-03-24T03:17:23.642 INFO:teuthology.orchestra.run.smithi118.stdout: "tags": [], 2024-03-24T03:17:23.642 INFO:teuthology.orchestra.run.smithi118.stdout: "see_also": [] 2024-03-24T03:17:23.643 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-24T03:17:23.643 INFO:teuthology.orchestra.run.smithi118.stdout: "sleep_interval": { 2024-03-24T03:17:23.643 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "sleep_interval", 2024-03-24T03:17:23.643 INFO:teuthology.orchestra.run.smithi118.stdout: "type": "secs", 2024-03-24T03:17:23.643 INFO:teuthology.orchestra.run.smithi118.stdout: "level": "advanced", 2024-03-24T03:17:23.643 INFO:teuthology.orchestra.run.smithi118.stdout: "flags": 1, 2024-03-24T03:17:23.643 INFO:teuthology.orchestra.run.smithi118.stdout: "default_value": "5", 2024-03-24T03:17:23.643 INFO:teuthology.orchestra.run.smithi118.stdout: "min": "", 2024-03-24T03:17:23.643 INFO:teuthology.orchestra.run.smithi118.stdout: "max": "", 2024-03-24T03:17:23.644 INFO:teuthology.orchestra.run.smithi118.stdout: "enum_allowed": [], 2024-03-24T03:17:23.644 INFO:teuthology.orchestra.run.smithi118.stdout: "desc": "how long the module is going to sleep", 2024-03-24T03:17:23.644 INFO:teuthology.orchestra.run.smithi118.stdout: "long_desc": "", 2024-03-24T03:17:23.644 INFO:teuthology.orchestra.run.smithi118.stdout: "tags": [], 2024-03-24T03:17:23.644 INFO:teuthology.orchestra.run.smithi118.stdout: "see_also": [] 2024-03-24T03:17:23.644 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-03-24T03:17:23.644 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-03-24T03:17:23.644 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-24T03:17:23.644 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-24T03:17:23.645 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "prometheus", 2024-03-24T03:17:23.645 INFO:teuthology.orchestra.run.smithi118.stdout: "can_run": true, 2024-03-24T03:17:23.645 INFO:teuthology.orchestra.run.smithi118.stdout: "error_string": "", 2024-03-24T03:17:23.645 INFO:teuthology.orchestra.run.smithi118.stdout: "module_options": { 2024-03-24T03:17:23.645 INFO:teuthology.orchestra.run.smithi118.stdout: "cache": { 2024-03-24T03:17:23.645 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "cache", 2024-03-24T03:17:23.645 INFO:teuthology.orchestra.run.smithi118.stdout: "type": "bool", 2024-03-24T03:17:23.645 INFO:teuthology.orchestra.run.smithi118.stdout: "level": "advanced", 2024-03-24T03:17:23.646 INFO:teuthology.orchestra.run.smithi118.stdout: "flags": 0, 2024-03-24T03:17:23.646 INFO:teuthology.orchestra.run.smithi118.stdout: "default_value": "True", 2024-03-24T03:17:23.646 INFO:teuthology.orchestra.run.smithi118.stdout: "min": "", 2024-03-24T03:17:23.646 INFO:teuthology.orchestra.run.smithi118.stdout: "max": "", 2024-03-24T03:17:23.646 INFO:teuthology.orchestra.run.smithi118.stdout: "enum_allowed": [], 2024-03-24T03:17:23.646 INFO:teuthology.orchestra.run.smithi118.stdout: "desc": "", 2024-03-24T03:17:23.646 INFO:teuthology.orchestra.run.smithi118.stdout: "long_desc": "", 2024-03-24T03:17:23.646 INFO:teuthology.orchestra.run.smithi118.stdout: "tags": [], 2024-03-24T03:17:23.647 INFO:teuthology.orchestra.run.smithi118.stdout: "see_also": [] 2024-03-24T03:17:23.647 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-24T03:17:23.647 INFO:teuthology.orchestra.run.smithi118.stdout: "exclude_perf_counters": { 2024-03-24T03:17:23.647 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "exclude_perf_counters", 2024-03-24T03:17:23.647 INFO:teuthology.orchestra.run.smithi118.stdout: "type": "bool", 2024-03-24T03:17:23.647 INFO:teuthology.orchestra.run.smithi118.stdout: "level": "advanced", 2024-03-24T03:17:23.647 INFO:teuthology.orchestra.run.smithi118.stdout: "flags": 1, 2024-03-24T03:17:23.647 INFO:teuthology.orchestra.run.smithi118.stdout: "default_value": "True", 2024-03-24T03:17:23.647 INFO:teuthology.orchestra.run.smithi118.stdout: "min": "", 2024-03-24T03:17:23.647 INFO:teuthology.orchestra.run.smithi118.stdout: "max": "", 2024-03-24T03:17:23.648 INFO:teuthology.orchestra.run.smithi118.stdout: "enum_allowed": [], 2024-03-24T03:17:23.648 INFO:teuthology.orchestra.run.smithi118.stdout: "desc": "Do not include perf-counters in the metrics output", 2024-03-24T03:17:23.648 INFO:teuthology.orchestra.run.smithi118.stdout: "long_desc": "Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.", 2024-03-24T03:17:23.648 INFO:teuthology.orchestra.run.smithi118.stdout: "tags": [], 2024-03-24T03:17:23.648 INFO:teuthology.orchestra.run.smithi118.stdout: "see_also": [] 2024-03-24T03:17:23.648 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-24T03:17:23.648 INFO:teuthology.orchestra.run.smithi118.stdout: "log_level": { 2024-03-24T03:17:23.648 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "log_level", 2024-03-24T03:17:23.649 INFO:teuthology.orchestra.run.smithi118.stdout: "type": "str", 2024-03-24T03:17:23.649 INFO:teuthology.orchestra.run.smithi118.stdout: "level": "advanced", 2024-03-24T03:17:23.649 INFO:teuthology.orchestra.run.smithi118.stdout: "flags": 1, 2024-03-24T03:17:23.649 INFO:teuthology.orchestra.run.smithi118.stdout: "default_value": "", 2024-03-24T03:17:23.649 INFO:teuthology.orchestra.run.smithi118.stdout: "min": "", 2024-03-24T03:17:23.649 INFO:teuthology.orchestra.run.smithi118.stdout: "max": "", 2024-03-24T03:17:23.649 INFO:teuthology.orchestra.run.smithi118.stdout: "enum_allowed": [ 2024-03-24T03:17:23.649 INFO:teuthology.orchestra.run.smithi118.stdout: "", 2024-03-24T03:17:23.649 INFO:teuthology.orchestra.run.smithi118.stdout: "critical", 2024-03-24T03:17:23.650 INFO:teuthology.orchestra.run.smithi118.stdout: "debug", 2024-03-24T03:17:23.650 INFO:teuthology.orchestra.run.smithi118.stdout: "error", 2024-03-24T03:17:23.650 INFO:teuthology.orchestra.run.smithi118.stdout: "info", 2024-03-24T03:17:23.650 INFO:teuthology.orchestra.run.smithi118.stdout: "warning" 2024-03-24T03:17:23.650 INFO:teuthology.orchestra.run.smithi118.stdout: ], 2024-03-24T03:17:23.650 INFO:teuthology.orchestra.run.smithi118.stdout: "desc": "", 2024-03-24T03:17:23.650 INFO:teuthology.orchestra.run.smithi118.stdout: "long_desc": "", 2024-03-24T03:17:23.650 INFO:teuthology.orchestra.run.smithi118.stdout: "tags": [], 2024-03-24T03:17:23.651 INFO:teuthology.orchestra.run.smithi118.stdout: "see_also": [] 2024-03-24T03:17:23.651 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-24T03:17:23.651 INFO:teuthology.orchestra.run.smithi118.stdout: "log_to_cluster": { 2024-03-24T03:17:23.651 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "log_to_cluster", 2024-03-24T03:17:23.651 INFO:teuthology.orchestra.run.smithi118.stdout: "type": "bool", 2024-03-24T03:17:23.651 INFO:teuthology.orchestra.run.smithi118.stdout: "level": "advanced", 2024-03-24T03:17:23.651 INFO:teuthology.orchestra.run.smithi118.stdout: "flags": 1, 2024-03-24T03:17:23.651 INFO:teuthology.orchestra.run.smithi118.stdout: "default_value": "False", 2024-03-24T03:17:23.652 INFO:teuthology.orchestra.run.smithi118.stdout: "min": "", 2024-03-24T03:17:23.652 INFO:teuthology.orchestra.run.smithi118.stdout: "max": "", 2024-03-24T03:17:23.652 INFO:teuthology.orchestra.run.smithi118.stdout: "enum_allowed": [], 2024-03-24T03:17:23.652 INFO:teuthology.orchestra.run.smithi118.stdout: "desc": "", 2024-03-24T03:17:23.652 INFO:teuthology.orchestra.run.smithi118.stdout: "long_desc": "", 2024-03-24T03:17:23.652 INFO:teuthology.orchestra.run.smithi118.stdout: "tags": [], 2024-03-24T03:17:23.652 INFO:teuthology.orchestra.run.smithi118.stdout: "see_also": [] 2024-03-24T03:17:23.652 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-24T03:17:23.653 INFO:teuthology.orchestra.run.smithi118.stdout: "log_to_cluster_level": { 2024-03-24T03:17:23.653 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "log_to_cluster_level", 2024-03-24T03:17:23.653 INFO:teuthology.orchestra.run.smithi118.stdout: "type": "str", 2024-03-24T03:17:23.653 INFO:teuthology.orchestra.run.smithi118.stdout: "level": "advanced", 2024-03-24T03:17:23.653 INFO:teuthology.orchestra.run.smithi118.stdout: "flags": 1, 2024-03-24T03:17:23.653 INFO:teuthology.orchestra.run.smithi118.stdout: "default_value": "info", 2024-03-24T03:17:23.653 INFO:teuthology.orchestra.run.smithi118.stdout: "min": "", 2024-03-24T03:17:23.653 INFO:teuthology.orchestra.run.smithi118.stdout: "max": "", 2024-03-24T03:17:23.654 INFO:teuthology.orchestra.run.smithi118.stdout: "enum_allowed": [ 2024-03-24T03:17:23.654 INFO:teuthology.orchestra.run.smithi118.stdout: "", 2024-03-24T03:17:23.654 INFO:teuthology.orchestra.run.smithi118.stdout: "critical", 2024-03-24T03:17:23.654 INFO:teuthology.orchestra.run.smithi118.stdout: "debug", 2024-03-24T03:17:23.654 INFO:teuthology.orchestra.run.smithi118.stdout: "error", 2024-03-24T03:17:23.654 INFO:teuthology.orchestra.run.smithi118.stdout: "info", 2024-03-24T03:17:23.654 INFO:teuthology.orchestra.run.smithi118.stdout: "warning" 2024-03-24T03:17:23.655 INFO:teuthology.orchestra.run.smithi118.stdout: ], 2024-03-24T03:17:23.655 INFO:teuthology.orchestra.run.smithi118.stdout: "desc": "", 2024-03-24T03:17:23.655 INFO:teuthology.orchestra.run.smithi118.stdout: "long_desc": "", 2024-03-24T03:17:23.655 INFO:teuthology.orchestra.run.smithi118.stdout: "tags": [], 2024-03-24T03:17:23.655 INFO:teuthology.orchestra.run.smithi118.stdout: "see_also": [] 2024-03-24T03:17:23.655 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-24T03:17:23.655 INFO:teuthology.orchestra.run.smithi118.stdout: "log_to_file": { 2024-03-24T03:17:23.655 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "log_to_file", 2024-03-24T03:17:23.655 INFO:teuthology.orchestra.run.smithi118.stdout: "type": "bool", 2024-03-24T03:17:23.656 INFO:teuthology.orchestra.run.smithi118.stdout: "level": "advanced", 2024-03-24T03:17:23.656 INFO:teuthology.orchestra.run.smithi118.stdout: "flags": 1, 2024-03-24T03:17:23.656 INFO:teuthology.orchestra.run.smithi118.stdout: "default_value": "False", 2024-03-24T03:17:23.656 INFO:teuthology.orchestra.run.smithi118.stdout: "min": "", 2024-03-24T03:17:23.656 INFO:teuthology.orchestra.run.smithi118.stdout: "max": "", 2024-03-24T03:17:23.656 INFO:teuthology.orchestra.run.smithi118.stdout: "enum_allowed": [], 2024-03-24T03:17:23.656 INFO:teuthology.orchestra.run.smithi118.stdout: "desc": "", 2024-03-24T03:17:23.656 INFO:teuthology.orchestra.run.smithi118.stdout: "long_desc": "", 2024-03-24T03:17:23.656 INFO:teuthology.orchestra.run.smithi118.stdout: "tags": [], 2024-03-24T03:17:23.657 INFO:teuthology.orchestra.run.smithi118.stdout: "see_also": [] 2024-03-24T03:17:23.657 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-24T03:17:23.657 INFO:teuthology.orchestra.run.smithi118.stdout: "rbd_stats_pools": { 2024-03-24T03:17:23.657 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "rbd_stats_pools", 2024-03-24T03:17:23.657 INFO:teuthology.orchestra.run.smithi118.stdout: "type": "str", 2024-03-24T03:17:23.657 INFO:teuthology.orchestra.run.smithi118.stdout: "level": "advanced", 2024-03-24T03:17:23.657 INFO:teuthology.orchestra.run.smithi118.stdout: "flags": 0, 2024-03-24T03:17:23.657 INFO:teuthology.orchestra.run.smithi118.stdout: "default_value": "", 2024-03-24T03:17:23.658 INFO:teuthology.orchestra.run.smithi118.stdout: "min": "", 2024-03-24T03:17:23.658 INFO:teuthology.orchestra.run.smithi118.stdout: "max": "", 2024-03-24T03:17:23.658 INFO:teuthology.orchestra.run.smithi118.stdout: "enum_allowed": [], 2024-03-24T03:17:23.658 INFO:teuthology.orchestra.run.smithi118.stdout: "desc": "", 2024-03-24T03:17:23.658 INFO:teuthology.orchestra.run.smithi118.stdout: "long_desc": "", 2024-03-24T03:17:23.658 INFO:teuthology.orchestra.run.smithi118.stdout: "tags": [], 2024-03-24T03:17:23.658 INFO:teuthology.orchestra.run.smithi118.stdout: "see_also": [] 2024-03-24T03:17:23.658 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-24T03:17:23.659 INFO:teuthology.orchestra.run.smithi118.stdout: "rbd_stats_pools_refresh_interval": { 2024-03-24T03:17:23.659 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "rbd_stats_pools_refresh_interval", 2024-03-24T03:17:23.659 INFO:teuthology.orchestra.run.smithi118.stdout: "type": "int", 2024-03-24T03:17:23.659 INFO:teuthology.orchestra.run.smithi118.stdout: "level": "advanced", 2024-03-24T03:17:23.659 INFO:teuthology.orchestra.run.smithi118.stdout: "flags": 0, 2024-03-24T03:17:23.659 INFO:teuthology.orchestra.run.smithi118.stdout: "default_value": "300", 2024-03-24T03:17:23.659 INFO:teuthology.orchestra.run.smithi118.stdout: "min": "", 2024-03-24T03:17:23.659 INFO:teuthology.orchestra.run.smithi118.stdout: "max": "", 2024-03-24T03:17:23.659 INFO:teuthology.orchestra.run.smithi118.stdout: "enum_allowed": [], 2024-03-24T03:17:23.660 INFO:teuthology.orchestra.run.smithi118.stdout: "desc": "", 2024-03-24T03:17:23.660 INFO:teuthology.orchestra.run.smithi118.stdout: "long_desc": "", 2024-03-24T03:17:23.660 INFO:teuthology.orchestra.run.smithi118.stdout: "tags": [], 2024-03-24T03:17:23.660 INFO:teuthology.orchestra.run.smithi118.stdout: "see_also": [] 2024-03-24T03:17:23.660 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-24T03:17:23.660 INFO:teuthology.orchestra.run.smithi118.stdout: "scrape_interval": { 2024-03-24T03:17:23.660 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "scrape_interval", 2024-03-24T03:17:23.660 INFO:teuthology.orchestra.run.smithi118.stdout: "type": "float", 2024-03-24T03:17:23.660 INFO:teuthology.orchestra.run.smithi118.stdout: "level": "advanced", 2024-03-24T03:17:23.661 INFO:teuthology.orchestra.run.smithi118.stdout: "flags": 0, 2024-03-24T03:17:23.661 INFO:teuthology.orchestra.run.smithi118.stdout: "default_value": "15.0", 2024-03-24T03:17:23.661 INFO:teuthology.orchestra.run.smithi118.stdout: "min": "", 2024-03-24T03:17:23.661 INFO:teuthology.orchestra.run.smithi118.stdout: "max": "", 2024-03-24T03:17:23.661 INFO:teuthology.orchestra.run.smithi118.stdout: "enum_allowed": [], 2024-03-24T03:17:23.661 INFO:teuthology.orchestra.run.smithi118.stdout: "desc": "", 2024-03-24T03:17:23.661 INFO:teuthology.orchestra.run.smithi118.stdout: "long_desc": "", 2024-03-24T03:17:23.661 INFO:teuthology.orchestra.run.smithi118.stdout: "tags": [], 2024-03-24T03:17:23.662 INFO:teuthology.orchestra.run.smithi118.stdout: "see_also": [] 2024-03-24T03:17:23.662 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-24T03:17:23.662 INFO:teuthology.orchestra.run.smithi118.stdout: "server_addr": { 2024-03-24T03:17:23.662 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "server_addr", 2024-03-24T03:17:23.662 INFO:teuthology.orchestra.run.smithi118.stdout: "type": "str", 2024-03-24T03:17:23.662 INFO:teuthology.orchestra.run.smithi118.stdout: "level": "advanced", 2024-03-24T03:17:23.662 INFO:teuthology.orchestra.run.smithi118.stdout: "flags": 0, 2024-03-24T03:17:23.662 INFO:teuthology.orchestra.run.smithi118.stdout: "default_value": "::", 2024-03-24T03:17:23.662 INFO:teuthology.orchestra.run.smithi118.stdout: "min": "", 2024-03-24T03:17:23.663 INFO:teuthology.orchestra.run.smithi118.stdout: "max": "", 2024-03-24T03:17:23.663 INFO:teuthology.orchestra.run.smithi118.stdout: "enum_allowed": [], 2024-03-24T03:17:23.663 INFO:teuthology.orchestra.run.smithi118.stdout: "desc": "the IPv4 or IPv6 address on which the module listens for HTTP requests", 2024-03-24T03:17:23.663 INFO:teuthology.orchestra.run.smithi118.stdout: "long_desc": "", 2024-03-24T03:17:23.663 INFO:teuthology.orchestra.run.smithi118.stdout: "tags": [], 2024-03-24T03:17:23.663 INFO:teuthology.orchestra.run.smithi118.stdout: "see_also": [] 2024-03-24T03:17:23.663 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-24T03:17:23.663 INFO:teuthology.orchestra.run.smithi118.stdout: "server_port": { 2024-03-24T03:17:23.664 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "server_port", 2024-03-24T03:17:23.664 INFO:teuthology.orchestra.run.smithi118.stdout: "type": "int", 2024-03-24T03:17:23.664 INFO:teuthology.orchestra.run.smithi118.stdout: "level": "advanced", 2024-03-24T03:17:23.664 INFO:teuthology.orchestra.run.smithi118.stdout: "flags": 1, 2024-03-24T03:17:23.664 INFO:teuthology.orchestra.run.smithi118.stdout: "default_value": "9283", 2024-03-24T03:17:23.665 INFO:teuthology.orchestra.run.smithi118.stdout: "min": "", 2024-03-24T03:17:23.665 INFO:teuthology.orchestra.run.smithi118.stdout: "max": "", 2024-03-24T03:17:23.665 INFO:teuthology.orchestra.run.smithi118.stdout: "enum_allowed": [], 2024-03-24T03:17:23.665 INFO:teuthology.orchestra.run.smithi118.stdout: "desc": "the port on which the module listens for HTTP requests", 2024-03-24T03:17:23.665 INFO:teuthology.orchestra.run.smithi118.stdout: "long_desc": "", 2024-03-24T03:17:23.666 INFO:teuthology.orchestra.run.smithi118.stdout: "tags": [], 2024-03-24T03:17:23.666 INFO:teuthology.orchestra.run.smithi118.stdout: "see_also": [] 2024-03-24T03:17:23.666 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-24T03:17:23.666 INFO:teuthology.orchestra.run.smithi118.stdout: "stale_cache_strategy": { 2024-03-24T03:17:23.667 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "stale_cache_strategy", 2024-03-24T03:17:23.667 INFO:teuthology.orchestra.run.smithi118.stdout: "type": "str", 2024-03-24T03:17:23.667 INFO:teuthology.orchestra.run.smithi118.stdout: "level": "advanced", 2024-03-24T03:17:23.667 INFO:teuthology.orchestra.run.smithi118.stdout: "flags": 0, 2024-03-24T03:17:23.668 INFO:teuthology.orchestra.run.smithi118.stdout: "default_value": "log", 2024-03-24T03:17:23.668 INFO:teuthology.orchestra.run.smithi118.stdout: "min": "", 2024-03-24T03:17:23.668 INFO:teuthology.orchestra.run.smithi118.stdout: "max": "", 2024-03-24T03:17:23.668 INFO:teuthology.orchestra.run.smithi118.stdout: "enum_allowed": [], 2024-03-24T03:17:23.668 INFO:teuthology.orchestra.run.smithi118.stdout: "desc": "", 2024-03-24T03:17:23.669 INFO:teuthology.orchestra.run.smithi118.stdout: "long_desc": "", 2024-03-24T03:17:23.669 INFO:teuthology.orchestra.run.smithi118.stdout: "tags": [], 2024-03-24T03:17:23.669 INFO:teuthology.orchestra.run.smithi118.stdout: "see_also": [] 2024-03-24T03:17:23.669 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-24T03:17:23.669 INFO:teuthology.orchestra.run.smithi118.stdout: "standby_behaviour": { 2024-03-24T03:17:23.670 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "standby_behaviour", 2024-03-24T03:17:23.670 INFO:teuthology.orchestra.run.smithi118.stdout: "type": "str", 2024-03-24T03:17:23.670 INFO:teuthology.orchestra.run.smithi118.stdout: "level": "advanced", 2024-03-24T03:17:23.670 INFO:teuthology.orchestra.run.smithi118.stdout: "flags": 1, 2024-03-24T03:17:23.671 INFO:teuthology.orchestra.run.smithi118.stdout: "default_value": "default", 2024-03-24T03:17:23.671 INFO:teuthology.orchestra.run.smithi118.stdout: "min": "", 2024-03-24T03:17:23.671 INFO:teuthology.orchestra.run.smithi118.stdout: "max": "", 2024-03-24T03:17:23.671 INFO:teuthology.orchestra.run.smithi118.stdout: "enum_allowed": [ 2024-03-24T03:17:23.672 INFO:teuthology.orchestra.run.smithi118.stdout: "default", 2024-03-24T03:17:23.672 INFO:teuthology.orchestra.run.smithi118.stdout: "error" 2024-03-24T03:17:23.672 INFO:teuthology.orchestra.run.smithi118.stdout: ], 2024-03-24T03:17:23.672 INFO:teuthology.orchestra.run.smithi118.stdout: "desc": "", 2024-03-24T03:17:23.672 INFO:teuthology.orchestra.run.smithi118.stdout: "long_desc": "", 2024-03-24T03:17:23.673 INFO:teuthology.orchestra.run.smithi118.stdout: "tags": [], 2024-03-24T03:17:23.673 INFO:teuthology.orchestra.run.smithi118.stdout: "see_also": [] 2024-03-24T03:17:23.673 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-24T03:17:23.673 INFO:teuthology.orchestra.run.smithi118.stdout: "standby_error_status_code": { 2024-03-24T03:17:23.674 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "standby_error_status_code", 2024-03-24T03:17:23.674 INFO:teuthology.orchestra.run.smithi118.stdout: "type": "int", 2024-03-24T03:17:23.674 INFO:teuthology.orchestra.run.smithi118.stdout: "level": "advanced", 2024-03-24T03:17:23.674 INFO:teuthology.orchestra.run.smithi118.stdout: "flags": 1, 2024-03-24T03:17:23.674 INFO:teuthology.orchestra.run.smithi118.stdout: "default_value": "500", 2024-03-24T03:17:23.674 INFO:teuthology.orchestra.run.smithi118.stdout: "min": "400", 2024-03-24T03:17:23.674 INFO:teuthology.orchestra.run.smithi118.stdout: "max": "599", 2024-03-24T03:17:23.675 INFO:teuthology.orchestra.run.smithi118.stdout: "enum_allowed": [], 2024-03-24T03:17:23.675 INFO:teuthology.orchestra.run.smithi118.stdout: "desc": "", 2024-03-24T03:17:23.675 INFO:teuthology.orchestra.run.smithi118.stdout: "long_desc": "", 2024-03-24T03:17:23.675 INFO:teuthology.orchestra.run.smithi118.stdout: "tags": [], 2024-03-24T03:17:23.675 INFO:teuthology.orchestra.run.smithi118.stdout: "see_also": [] 2024-03-24T03:17:23.675 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-03-24T03:17:23.675 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-03-24T03:17:23.675 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-24T03:17:23.675 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-24T03:17:23.676 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "rbd_support", 2024-03-24T03:17:23.676 INFO:teuthology.orchestra.run.smithi118.stdout: "can_run": true, 2024-03-24T03:17:23.676 INFO:teuthology.orchestra.run.smithi118.stdout: "error_string": "", 2024-03-24T03:17:23.676 INFO:teuthology.orchestra.run.smithi118.stdout: "module_options": { 2024-03-24T03:17:23.676 INFO:teuthology.orchestra.run.smithi118.stdout: "log_level": { 2024-03-24T03:17:23.676 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "log_level", 2024-03-24T03:17:23.676 INFO:teuthology.orchestra.run.smithi118.stdout: "type": "str", 2024-03-24T03:17:23.676 INFO:teuthology.orchestra.run.smithi118.stdout: "level": "advanced", 2024-03-24T03:17:23.676 INFO:teuthology.orchestra.run.smithi118.stdout: "flags": 1, 2024-03-24T03:17:23.677 INFO:teuthology.orchestra.run.smithi118.stdout: "default_value": "", 2024-03-24T03:17:23.677 INFO:teuthology.orchestra.run.smithi118.stdout: "min": "", 2024-03-24T03:17:23.677 INFO:teuthology.orchestra.run.smithi118.stdout: "max": "", 2024-03-24T03:17:23.677 INFO:teuthology.orchestra.run.smithi118.stdout: "enum_allowed": [ 2024-03-24T03:17:23.677 INFO:teuthology.orchestra.run.smithi118.stdout: "", 2024-03-24T03:17:23.677 INFO:teuthology.orchestra.run.smithi118.stdout: "critical", 2024-03-24T03:17:23.677 INFO:teuthology.orchestra.run.smithi118.stdout: "debug", 2024-03-24T03:17:23.677 INFO:teuthology.orchestra.run.smithi118.stdout: "error", 2024-03-24T03:17:23.677 INFO:teuthology.orchestra.run.smithi118.stdout: "info", 2024-03-24T03:17:23.678 INFO:teuthology.orchestra.run.smithi118.stdout: "warning" 2024-03-24T03:17:23.678 INFO:teuthology.orchestra.run.smithi118.stdout: ], 2024-03-24T03:17:23.678 INFO:teuthology.orchestra.run.smithi118.stdout: "desc": "", 2024-03-24T03:17:23.678 INFO:teuthology.orchestra.run.smithi118.stdout: "long_desc": "", 2024-03-24T03:17:23.678 INFO:teuthology.orchestra.run.smithi118.stdout: "tags": [], 2024-03-24T03:17:23.678 INFO:teuthology.orchestra.run.smithi118.stdout: "see_also": [] 2024-03-24T03:17:23.678 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-24T03:17:23.678 INFO:teuthology.orchestra.run.smithi118.stdout: "log_to_cluster": { 2024-03-24T03:17:23.679 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "log_to_cluster", 2024-03-24T03:17:23.679 INFO:teuthology.orchestra.run.smithi118.stdout: "type": "bool", 2024-03-24T03:17:23.679 INFO:teuthology.orchestra.run.smithi118.stdout: "level": "advanced", 2024-03-24T03:17:23.679 INFO:teuthology.orchestra.run.smithi118.stdout: "flags": 1, 2024-03-24T03:17:23.679 INFO:teuthology.orchestra.run.smithi118.stdout: "default_value": "False", 2024-03-24T03:17:23.679 INFO:teuthology.orchestra.run.smithi118.stdout: "min": "", 2024-03-24T03:17:23.679 INFO:teuthology.orchestra.run.smithi118.stdout: "max": "", 2024-03-24T03:17:23.679 INFO:teuthology.orchestra.run.smithi118.stdout: "enum_allowed": [], 2024-03-24T03:17:23.679 INFO:teuthology.orchestra.run.smithi118.stdout: "desc": "", 2024-03-24T03:17:23.680 INFO:teuthology.orchestra.run.smithi118.stdout: "long_desc": "", 2024-03-24T03:17:23.680 INFO:teuthology.orchestra.run.smithi118.stdout: "tags": [], 2024-03-24T03:17:23.680 INFO:teuthology.orchestra.run.smithi118.stdout: "see_also": [] 2024-03-24T03:17:23.680 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-24T03:17:23.680 INFO:teuthology.orchestra.run.smithi118.stdout: "log_to_cluster_level": { 2024-03-24T03:17:23.680 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "log_to_cluster_level", 2024-03-24T03:17:23.680 INFO:teuthology.orchestra.run.smithi118.stdout: "type": "str", 2024-03-24T03:17:23.680 INFO:teuthology.orchestra.run.smithi118.stdout: "level": "advanced", 2024-03-24T03:17:23.680 INFO:teuthology.orchestra.run.smithi118.stdout: "flags": 1, 2024-03-24T03:17:23.681 INFO:teuthology.orchestra.run.smithi118.stdout: "default_value": "info", 2024-03-24T03:17:23.681 INFO:teuthology.orchestra.run.smithi118.stdout: "min": "", 2024-03-24T03:17:23.681 INFO:teuthology.orchestra.run.smithi118.stdout: "max": "", 2024-03-24T03:17:23.681 INFO:teuthology.orchestra.run.smithi118.stdout: "enum_allowed": [ 2024-03-24T03:17:23.681 INFO:teuthology.orchestra.run.smithi118.stdout: "", 2024-03-24T03:17:23.681 INFO:teuthology.orchestra.run.smithi118.stdout: "critical", 2024-03-24T03:17:23.681 INFO:teuthology.orchestra.run.smithi118.stdout: "debug", 2024-03-24T03:17:23.681 INFO:teuthology.orchestra.run.smithi118.stdout: "error", 2024-03-24T03:17:23.681 INFO:teuthology.orchestra.run.smithi118.stdout: "info", 2024-03-24T03:17:23.682 INFO:teuthology.orchestra.run.smithi118.stdout: "warning" 2024-03-24T03:17:23.682 INFO:teuthology.orchestra.run.smithi118.stdout: ], 2024-03-24T03:17:23.682 INFO:teuthology.orchestra.run.smithi118.stdout: "desc": "", 2024-03-24T03:17:23.682 INFO:teuthology.orchestra.run.smithi118.stdout: "long_desc": "", 2024-03-24T03:17:23.682 INFO:teuthology.orchestra.run.smithi118.stdout: "tags": [], 2024-03-24T03:17:23.682 INFO:teuthology.orchestra.run.smithi118.stdout: "see_also": [] 2024-03-24T03:17:23.682 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-24T03:17:23.682 INFO:teuthology.orchestra.run.smithi118.stdout: "log_to_file": { 2024-03-24T03:17:23.683 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "log_to_file", 2024-03-24T03:17:23.683 INFO:teuthology.orchestra.run.smithi118.stdout: "type": "bool", 2024-03-24T03:17:23.683 INFO:teuthology.orchestra.run.smithi118.stdout: "level": "advanced", 2024-03-24T03:17:23.683 INFO:teuthology.orchestra.run.smithi118.stdout: "flags": 1, 2024-03-24T03:17:23.683 INFO:teuthology.orchestra.run.smithi118.stdout: "default_value": "False", 2024-03-24T03:17:23.683 INFO:teuthology.orchestra.run.smithi118.stdout: "min": "", 2024-03-24T03:17:23.683 INFO:teuthology.orchestra.run.smithi118.stdout: "max": "", 2024-03-24T03:17:23.683 INFO:teuthology.orchestra.run.smithi118.stdout: "enum_allowed": [], 2024-03-24T03:17:23.683 INFO:teuthology.orchestra.run.smithi118.stdout: "desc": "", 2024-03-24T03:17:23.684 INFO:teuthology.orchestra.run.smithi118.stdout: "long_desc": "", 2024-03-24T03:17:23.684 INFO:teuthology.orchestra.run.smithi118.stdout: "tags": [], 2024-03-24T03:17:23.684 INFO:teuthology.orchestra.run.smithi118.stdout: "see_also": [] 2024-03-24T03:17:23.684 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-24T03:17:23.684 INFO:teuthology.orchestra.run.smithi118.stdout: "max_concurrent_snap_create": { 2024-03-24T03:17:23.685 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "max_concurrent_snap_create", 2024-03-24T03:17:23.685 INFO:teuthology.orchestra.run.smithi118.stdout: "type": "int", 2024-03-24T03:17:23.685 INFO:teuthology.orchestra.run.smithi118.stdout: "level": "advanced", 2024-03-24T03:17:23.685 INFO:teuthology.orchestra.run.smithi118.stdout: "flags": 0, 2024-03-24T03:17:23.685 INFO:teuthology.orchestra.run.smithi118.stdout: "default_value": "10", 2024-03-24T03:17:23.685 INFO:teuthology.orchestra.run.smithi118.stdout: "min": "", 2024-03-24T03:17:23.686 INFO:teuthology.orchestra.run.smithi118.stdout: "max": "", 2024-03-24T03:17:23.686 INFO:teuthology.orchestra.run.smithi118.stdout: "enum_allowed": [], 2024-03-24T03:17:23.686 INFO:teuthology.orchestra.run.smithi118.stdout: "desc": "", 2024-03-24T03:17:23.686 INFO:teuthology.orchestra.run.smithi118.stdout: "long_desc": "", 2024-03-24T03:17:23.686 INFO:teuthology.orchestra.run.smithi118.stdout: "tags": [], 2024-03-24T03:17:23.687 INFO:teuthology.orchestra.run.smithi118.stdout: "see_also": [] 2024-03-24T03:17:23.687 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-24T03:17:23.687 INFO:teuthology.orchestra.run.smithi118.stdout: "mirror_snapshot_schedule": { 2024-03-24T03:17:23.687 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "mirror_snapshot_schedule", 2024-03-24T03:17:23.687 INFO:teuthology.orchestra.run.smithi118.stdout: "type": "str", 2024-03-24T03:17:23.688 INFO:teuthology.orchestra.run.smithi118.stdout: "level": "advanced", 2024-03-24T03:17:23.688 INFO:teuthology.orchestra.run.smithi118.stdout: "flags": 0, 2024-03-24T03:17:23.688 INFO:teuthology.orchestra.run.smithi118.stdout: "default_value": "", 2024-03-24T03:17:23.688 INFO:teuthology.orchestra.run.smithi118.stdout: "min": "", 2024-03-24T03:17:23.688 INFO:teuthology.orchestra.run.smithi118.stdout: "max": "", 2024-03-24T03:17:23.688 INFO:teuthology.orchestra.run.smithi118.stdout: "enum_allowed": [], 2024-03-24T03:17:23.689 INFO:teuthology.orchestra.run.smithi118.stdout: "desc": "", 2024-03-24T03:17:23.689 INFO:teuthology.orchestra.run.smithi118.stdout: "long_desc": "", 2024-03-24T03:17:23.689 INFO:teuthology.orchestra.run.smithi118.stdout: "tags": [], 2024-03-24T03:17:23.689 INFO:teuthology.orchestra.run.smithi118.stdout: "see_also": [] 2024-03-24T03:17:23.689 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-24T03:17:23.690 INFO:teuthology.orchestra.run.smithi118.stdout: "trash_purge_schedule": { 2024-03-24T03:17:23.690 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "trash_purge_schedule", 2024-03-24T03:17:23.690 INFO:teuthology.orchestra.run.smithi118.stdout: "type": "str", 2024-03-24T03:17:23.690 INFO:teuthology.orchestra.run.smithi118.stdout: "level": "advanced", 2024-03-24T03:17:23.690 INFO:teuthology.orchestra.run.smithi118.stdout: "flags": 0, 2024-03-24T03:17:23.691 INFO:teuthology.orchestra.run.smithi118.stdout: "default_value": "", 2024-03-24T03:17:23.691 INFO:teuthology.orchestra.run.smithi118.stdout: "min": "", 2024-03-24T03:17:23.691 INFO:teuthology.orchestra.run.smithi118.stdout: "max": "", 2024-03-24T03:17:23.691 INFO:teuthology.orchestra.run.smithi118.stdout: "enum_allowed": [], 2024-03-24T03:17:23.691 INFO:teuthology.orchestra.run.smithi118.stdout: "desc": "", 2024-03-24T03:17:23.692 INFO:teuthology.orchestra.run.smithi118.stdout: "long_desc": "", 2024-03-24T03:17:23.692 INFO:teuthology.orchestra.run.smithi118.stdout: "tags": [], 2024-03-24T03:17:23.692 INFO:teuthology.orchestra.run.smithi118.stdout: "see_also": [] 2024-03-24T03:17:23.692 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-03-24T03:17:23.692 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-03-24T03:17:23.692 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-24T03:17:23.693 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-24T03:17:23.693 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "restful", 2024-03-24T03:17:23.693 INFO:teuthology.orchestra.run.smithi118.stdout: "can_run": true, 2024-03-24T03:17:23.693 INFO:teuthology.orchestra.run.smithi118.stdout: "error_string": "", 2024-03-24T03:17:23.693 INFO:teuthology.orchestra.run.smithi118.stdout: "module_options": { 2024-03-24T03:17:23.694 INFO:teuthology.orchestra.run.smithi118.stdout: "enable_auth": { 2024-03-24T03:17:23.694 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "enable_auth", 2024-03-24T03:17:23.694 INFO:teuthology.orchestra.run.smithi118.stdout: "type": "bool", 2024-03-24T03:17:23.694 INFO:teuthology.orchestra.run.smithi118.stdout: "level": "advanced", 2024-03-24T03:17:23.694 INFO:teuthology.orchestra.run.smithi118.stdout: "flags": 0, 2024-03-24T03:17:23.695 INFO:teuthology.orchestra.run.smithi118.stdout: "default_value": "True", 2024-03-24T03:17:23.695 INFO:teuthology.orchestra.run.smithi118.stdout: "min": "", 2024-03-24T03:17:23.695 INFO:teuthology.orchestra.run.smithi118.stdout: "max": "", 2024-03-24T03:17:23.695 INFO:teuthology.orchestra.run.smithi118.stdout: "enum_allowed": [], 2024-03-24T03:17:23.695 INFO:teuthology.orchestra.run.smithi118.stdout: "desc": "", 2024-03-24T03:17:23.696 INFO:teuthology.orchestra.run.smithi118.stdout: "long_desc": "", 2024-03-24T03:17:23.696 INFO:teuthology.orchestra.run.smithi118.stdout: "tags": [], 2024-03-24T03:17:23.696 INFO:teuthology.orchestra.run.smithi118.stdout: "see_also": [] 2024-03-24T03:17:23.696 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-24T03:17:23.696 INFO:teuthology.orchestra.run.smithi118.stdout: "key_file": { 2024-03-24T03:17:23.697 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "key_file", 2024-03-24T03:17:23.697 INFO:teuthology.orchestra.run.smithi118.stdout: "type": "str", 2024-03-24T03:17:23.697 INFO:teuthology.orchestra.run.smithi118.stdout: "level": "advanced", 2024-03-24T03:17:23.697 INFO:teuthology.orchestra.run.smithi118.stdout: "flags": 0, 2024-03-24T03:17:23.697 INFO:teuthology.orchestra.run.smithi118.stdout: "default_value": "", 2024-03-24T03:17:23.697 INFO:teuthology.orchestra.run.smithi118.stdout: "min": "", 2024-03-24T03:17:23.698 INFO:teuthology.orchestra.run.smithi118.stdout: "max": "", 2024-03-24T03:17:23.698 INFO:teuthology.orchestra.run.smithi118.stdout: "enum_allowed": [], 2024-03-24T03:17:23.698 INFO:teuthology.orchestra.run.smithi118.stdout: "desc": "", 2024-03-24T03:17:23.698 INFO:teuthology.orchestra.run.smithi118.stdout: "long_desc": "", 2024-03-24T03:17:23.698 INFO:teuthology.orchestra.run.smithi118.stdout: "tags": [], 2024-03-24T03:17:23.699 INFO:teuthology.orchestra.run.smithi118.stdout: "see_also": [] 2024-03-24T03:17:23.699 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-24T03:17:23.699 INFO:teuthology.orchestra.run.smithi118.stdout: "log_level": { 2024-03-24T03:17:23.699 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "log_level", 2024-03-24T03:17:23.699 INFO:teuthology.orchestra.run.smithi118.stdout: "type": "str", 2024-03-24T03:17:23.699 INFO:teuthology.orchestra.run.smithi118.stdout: "level": "advanced", 2024-03-24T03:17:23.699 INFO:teuthology.orchestra.run.smithi118.stdout: "flags": 1, 2024-03-24T03:17:23.700 INFO:teuthology.orchestra.run.smithi118.stdout: "default_value": "", 2024-03-24T03:17:23.700 INFO:teuthology.orchestra.run.smithi118.stdout: "min": "", 2024-03-24T03:17:23.700 INFO:teuthology.orchestra.run.smithi118.stdout: "max": "", 2024-03-24T03:17:23.700 INFO:teuthology.orchestra.run.smithi118.stdout: "enum_allowed": [ 2024-03-24T03:17:23.700 INFO:teuthology.orchestra.run.smithi118.stdout: "", 2024-03-24T03:17:23.700 INFO:teuthology.orchestra.run.smithi118.stdout: "critical", 2024-03-24T03:17:23.700 INFO:teuthology.orchestra.run.smithi118.stdout: "debug", 2024-03-24T03:17:23.700 INFO:teuthology.orchestra.run.smithi118.stdout: "error", 2024-03-24T03:17:23.701 INFO:teuthology.orchestra.run.smithi118.stdout: "info", 2024-03-24T03:17:23.701 INFO:teuthology.orchestra.run.smithi118.stdout: "warning" 2024-03-24T03:17:23.701 INFO:teuthology.orchestra.run.smithi118.stdout: ], 2024-03-24T03:17:23.701 INFO:teuthology.orchestra.run.smithi118.stdout: "desc": "", 2024-03-24T03:17:23.701 INFO:teuthology.orchestra.run.smithi118.stdout: "long_desc": "", 2024-03-24T03:17:23.701 INFO:teuthology.orchestra.run.smithi118.stdout: "tags": [], 2024-03-24T03:17:23.701 INFO:teuthology.orchestra.run.smithi118.stdout: "see_also": [] 2024-03-24T03:17:23.702 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-24T03:17:23.702 INFO:teuthology.orchestra.run.smithi118.stdout: "log_to_cluster": { 2024-03-24T03:17:23.702 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "log_to_cluster", 2024-03-24T03:17:23.702 INFO:teuthology.orchestra.run.smithi118.stdout: "type": "bool", 2024-03-24T03:17:23.702 INFO:teuthology.orchestra.run.smithi118.stdout: "level": "advanced", 2024-03-24T03:17:23.702 INFO:teuthology.orchestra.run.smithi118.stdout: "flags": 1, 2024-03-24T03:17:23.702 INFO:teuthology.orchestra.run.smithi118.stdout: "default_value": "False", 2024-03-24T03:17:23.703 INFO:teuthology.orchestra.run.smithi118.stdout: "min": "", 2024-03-24T03:17:23.703 INFO:teuthology.orchestra.run.smithi118.stdout: "max": "", 2024-03-24T03:17:23.703 INFO:teuthology.orchestra.run.smithi118.stdout: "enum_allowed": [], 2024-03-24T03:17:23.703 INFO:teuthology.orchestra.run.smithi118.stdout: "desc": "", 2024-03-24T03:17:23.703 INFO:teuthology.orchestra.run.smithi118.stdout: "long_desc": "", 2024-03-24T03:17:23.703 INFO:teuthology.orchestra.run.smithi118.stdout: "tags": [], 2024-03-24T03:17:23.703 INFO:teuthology.orchestra.run.smithi118.stdout: "see_also": [] 2024-03-24T03:17:23.703 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-24T03:17:23.703 INFO:teuthology.orchestra.run.smithi118.stdout: "log_to_cluster_level": { 2024-03-24T03:17:23.704 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "log_to_cluster_level", 2024-03-24T03:17:23.704 INFO:teuthology.orchestra.run.smithi118.stdout: "type": "str", 2024-03-24T03:17:23.704 INFO:teuthology.orchestra.run.smithi118.stdout: "level": "advanced", 2024-03-24T03:17:23.704 INFO:teuthology.orchestra.run.smithi118.stdout: "flags": 1, 2024-03-24T03:17:23.704 INFO:teuthology.orchestra.run.smithi118.stdout: "default_value": "info", 2024-03-24T03:17:23.704 INFO:teuthology.orchestra.run.smithi118.stdout: "min": "", 2024-03-24T03:17:23.704 INFO:teuthology.orchestra.run.smithi118.stdout: "max": "", 2024-03-24T03:17:23.704 INFO:teuthology.orchestra.run.smithi118.stdout: "enum_allowed": [ 2024-03-24T03:17:23.705 INFO:teuthology.orchestra.run.smithi118.stdout: "", 2024-03-24T03:17:23.705 INFO:teuthology.orchestra.run.smithi118.stdout: "critical", 2024-03-24T03:17:23.705 INFO:teuthology.orchestra.run.smithi118.stdout: "debug", 2024-03-24T03:17:23.705 INFO:teuthology.orchestra.run.smithi118.stdout: "error", 2024-03-24T03:17:23.705 INFO:teuthology.orchestra.run.smithi118.stdout: "info", 2024-03-24T03:17:23.705 INFO:teuthology.orchestra.run.smithi118.stdout: "warning" 2024-03-24T03:17:23.705 INFO:teuthology.orchestra.run.smithi118.stdout: ], 2024-03-24T03:17:23.705 INFO:teuthology.orchestra.run.smithi118.stdout: "desc": "", 2024-03-24T03:17:23.705 INFO:teuthology.orchestra.run.smithi118.stdout: "long_desc": "", 2024-03-24T03:17:23.705 INFO:teuthology.orchestra.run.smithi118.stdout: "tags": [], 2024-03-24T03:17:23.706 INFO:teuthology.orchestra.run.smithi118.stdout: "see_also": [] 2024-03-24T03:17:23.706 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-24T03:17:23.706 INFO:teuthology.orchestra.run.smithi118.stdout: "log_to_file": { 2024-03-24T03:17:23.706 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "log_to_file", 2024-03-24T03:17:23.706 INFO:teuthology.orchestra.run.smithi118.stdout: "type": "bool", 2024-03-24T03:17:23.706 INFO:teuthology.orchestra.run.smithi118.stdout: "level": "advanced", 2024-03-24T03:17:23.706 INFO:teuthology.orchestra.run.smithi118.stdout: "flags": 1, 2024-03-24T03:17:23.707 INFO:teuthology.orchestra.run.smithi118.stdout: "default_value": "False", 2024-03-24T03:17:23.707 INFO:teuthology.orchestra.run.smithi118.stdout: "min": "", 2024-03-24T03:17:23.707 INFO:teuthology.orchestra.run.smithi118.stdout: "max": "", 2024-03-24T03:17:23.707 INFO:teuthology.orchestra.run.smithi118.stdout: "enum_allowed": [], 2024-03-24T03:17:23.707 INFO:teuthology.orchestra.run.smithi118.stdout: "desc": "", 2024-03-24T03:17:23.707 INFO:teuthology.orchestra.run.smithi118.stdout: "long_desc": "", 2024-03-24T03:17:23.707 INFO:teuthology.orchestra.run.smithi118.stdout: "tags": [], 2024-03-24T03:17:23.707 INFO:teuthology.orchestra.run.smithi118.stdout: "see_also": [] 2024-03-24T03:17:23.707 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-24T03:17:23.707 INFO:teuthology.orchestra.run.smithi118.stdout: "server_addr": { 2024-03-24T03:17:23.708 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "server_addr", 2024-03-24T03:17:23.708 INFO:teuthology.orchestra.run.smithi118.stdout: "type": "str", 2024-03-24T03:17:23.708 INFO:teuthology.orchestra.run.smithi118.stdout: "level": "advanced", 2024-03-24T03:17:23.708 INFO:teuthology.orchestra.run.smithi118.stdout: "flags": 0, 2024-03-24T03:17:23.708 INFO:teuthology.orchestra.run.smithi118.stdout: "default_value": "", 2024-03-24T03:17:23.708 INFO:teuthology.orchestra.run.smithi118.stdout: "min": "", 2024-03-24T03:17:23.708 INFO:teuthology.orchestra.run.smithi118.stdout: "max": "", 2024-03-24T03:17:23.708 INFO:teuthology.orchestra.run.smithi118.stdout: "enum_allowed": [], 2024-03-24T03:17:23.708 INFO:teuthology.orchestra.run.smithi118.stdout: "desc": "", 2024-03-24T03:17:23.709 INFO:teuthology.orchestra.run.smithi118.stdout: "long_desc": "", 2024-03-24T03:17:23.709 INFO:teuthology.orchestra.run.smithi118.stdout: "tags": [], 2024-03-24T03:17:23.709 INFO:teuthology.orchestra.run.smithi118.stdout: "see_also": [] 2024-03-24T03:17:23.709 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-24T03:17:23.709 INFO:teuthology.orchestra.run.smithi118.stdout: "server_port": { 2024-03-24T03:17:23.709 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "server_port", 2024-03-24T03:17:23.709 INFO:teuthology.orchestra.run.smithi118.stdout: "type": "str", 2024-03-24T03:17:23.709 INFO:teuthology.orchestra.run.smithi118.stdout: "level": "advanced", 2024-03-24T03:17:23.709 INFO:teuthology.orchestra.run.smithi118.stdout: "flags": 0, 2024-03-24T03:17:23.709 INFO:teuthology.orchestra.run.smithi118.stdout: "default_value": "", 2024-03-24T03:17:23.710 INFO:teuthology.orchestra.run.smithi118.stdout: "min": "", 2024-03-24T03:17:23.710 INFO:teuthology.orchestra.run.smithi118.stdout: "max": "", 2024-03-24T03:17:23.710 INFO:teuthology.orchestra.run.smithi118.stdout: "enum_allowed": [], 2024-03-24T03:17:23.710 INFO:teuthology.orchestra.run.smithi118.stdout: "desc": "", 2024-03-24T03:17:23.710 INFO:teuthology.orchestra.run.smithi118.stdout: "long_desc": "", 2024-03-24T03:17:23.710 INFO:teuthology.orchestra.run.smithi118.stdout: "tags": [], 2024-03-24T03:17:23.710 INFO:teuthology.orchestra.run.smithi118.stdout: "see_also": [] 2024-03-24T03:17:23.710 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-03-24T03:17:23.711 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-03-24T03:17:23.711 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-24T03:17:23.711 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-24T03:17:23.711 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "rgw", 2024-03-24T03:17:23.711 INFO:teuthology.orchestra.run.smithi118.stdout: "can_run": true, 2024-03-24T03:17:23.711 INFO:teuthology.orchestra.run.smithi118.stdout: "error_string": "", 2024-03-24T03:17:23.711 INFO:teuthology.orchestra.run.smithi118.stdout: "module_options": { 2024-03-24T03:17:23.711 INFO:teuthology.orchestra.run.smithi118.stdout: "log_level": { 2024-03-24T03:17:23.711 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "log_level", 2024-03-24T03:17:23.712 INFO:teuthology.orchestra.run.smithi118.stdout: "type": "str", 2024-03-24T03:17:23.712 INFO:teuthology.orchestra.run.smithi118.stdout: "level": "advanced", 2024-03-24T03:17:23.712 INFO:teuthology.orchestra.run.smithi118.stdout: "flags": 1, 2024-03-24T03:17:23.712 INFO:teuthology.orchestra.run.smithi118.stdout: "default_value": "", 2024-03-24T03:17:23.712 INFO:teuthology.orchestra.run.smithi118.stdout: "min": "", 2024-03-24T03:17:23.712 INFO:teuthology.orchestra.run.smithi118.stdout: "max": "", 2024-03-24T03:17:23.712 INFO:teuthology.orchestra.run.smithi118.stdout: "enum_allowed": [ 2024-03-24T03:17:23.712 INFO:teuthology.orchestra.run.smithi118.stdout: "", 2024-03-24T03:17:23.712 INFO:teuthology.orchestra.run.smithi118.stdout: "critical", 2024-03-24T03:17:23.713 INFO:teuthology.orchestra.run.smithi118.stdout: "debug", 2024-03-24T03:17:23.713 INFO:teuthology.orchestra.run.smithi118.stdout: "error", 2024-03-24T03:17:23.713 INFO:teuthology.orchestra.run.smithi118.stdout: "info", 2024-03-24T03:17:23.713 INFO:teuthology.orchestra.run.smithi118.stdout: "warning" 2024-03-24T03:17:23.713 INFO:teuthology.orchestra.run.smithi118.stdout: ], 2024-03-24T03:17:23.713 INFO:teuthology.orchestra.run.smithi118.stdout: "desc": "", 2024-03-24T03:17:23.713 INFO:teuthology.orchestra.run.smithi118.stdout: "long_desc": "", 2024-03-24T03:17:23.713 INFO:teuthology.orchestra.run.smithi118.stdout: "tags": [], 2024-03-24T03:17:23.713 INFO:teuthology.orchestra.run.smithi118.stdout: "see_also": [] 2024-03-24T03:17:23.714 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-24T03:17:23.714 INFO:teuthology.orchestra.run.smithi118.stdout: "log_to_cluster": { 2024-03-24T03:17:23.714 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "log_to_cluster", 2024-03-24T03:17:23.714 INFO:teuthology.orchestra.run.smithi118.stdout: "type": "bool", 2024-03-24T03:17:23.714 INFO:teuthology.orchestra.run.smithi118.stdout: "level": "advanced", 2024-03-24T03:17:23.714 INFO:teuthology.orchestra.run.smithi118.stdout: "flags": 1, 2024-03-24T03:17:23.715 INFO:teuthology.orchestra.run.smithi118.stdout: "default_value": "False", 2024-03-24T03:17:23.715 INFO:teuthology.orchestra.run.smithi118.stdout: "min": "", 2024-03-24T03:17:23.715 INFO:teuthology.orchestra.run.smithi118.stdout: "max": "", 2024-03-24T03:17:23.715 INFO:teuthology.orchestra.run.smithi118.stdout: "enum_allowed": [], 2024-03-24T03:17:23.715 INFO:teuthology.orchestra.run.smithi118.stdout: "desc": "", 2024-03-24T03:17:23.715 INFO:teuthology.orchestra.run.smithi118.stdout: "long_desc": "", 2024-03-24T03:17:23.715 INFO:teuthology.orchestra.run.smithi118.stdout: "tags": [], 2024-03-24T03:17:23.715 INFO:teuthology.orchestra.run.smithi118.stdout: "see_also": [] 2024-03-24T03:17:23.715 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-24T03:17:23.716 INFO:teuthology.orchestra.run.smithi118.stdout: "log_to_cluster_level": { 2024-03-24T03:17:23.716 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "log_to_cluster_level", 2024-03-24T03:17:23.716 INFO:teuthology.orchestra.run.smithi118.stdout: "type": "str", 2024-03-24T03:17:23.716 INFO:teuthology.orchestra.run.smithi118.stdout: "level": "advanced", 2024-03-24T03:17:23.716 INFO:teuthology.orchestra.run.smithi118.stdout: "flags": 1, 2024-03-24T03:17:23.716 INFO:teuthology.orchestra.run.smithi118.stdout: "default_value": "info", 2024-03-24T03:17:23.716 INFO:teuthology.orchestra.run.smithi118.stdout: "min": "", 2024-03-24T03:17:23.716 INFO:teuthology.orchestra.run.smithi118.stdout: "max": "", 2024-03-24T03:17:23.716 INFO:teuthology.orchestra.run.smithi118.stdout: "enum_allowed": [ 2024-03-24T03:17:23.716 INFO:teuthology.orchestra.run.smithi118.stdout: "", 2024-03-24T03:17:23.717 INFO:teuthology.orchestra.run.smithi118.stdout: "critical", 2024-03-24T03:17:23.717 INFO:teuthology.orchestra.run.smithi118.stdout: "debug", 2024-03-24T03:17:23.717 INFO:teuthology.orchestra.run.smithi118.stdout: "error", 2024-03-24T03:17:23.717 INFO:teuthology.orchestra.run.smithi118.stdout: "info", 2024-03-24T03:17:23.717 INFO:teuthology.orchestra.run.smithi118.stdout: "warning" 2024-03-24T03:17:23.717 INFO:teuthology.orchestra.run.smithi118.stdout: ], 2024-03-24T03:17:23.717 INFO:teuthology.orchestra.run.smithi118.stdout: "desc": "", 2024-03-24T03:17:23.717 INFO:teuthology.orchestra.run.smithi118.stdout: "long_desc": "", 2024-03-24T03:17:23.717 INFO:teuthology.orchestra.run.smithi118.stdout: "tags": [], 2024-03-24T03:17:23.718 INFO:teuthology.orchestra.run.smithi118.stdout: "see_also": [] 2024-03-24T03:17:23.718 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-24T03:17:23.718 INFO:teuthology.orchestra.run.smithi118.stdout: "log_to_file": { 2024-03-24T03:17:23.718 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "log_to_file", 2024-03-24T03:17:23.718 INFO:teuthology.orchestra.run.smithi118.stdout: "type": "bool", 2024-03-24T03:17:23.718 INFO:teuthology.orchestra.run.smithi118.stdout: "level": "advanced", 2024-03-24T03:17:23.718 INFO:teuthology.orchestra.run.smithi118.stdout: "flags": 1, 2024-03-24T03:17:23.718 INFO:teuthology.orchestra.run.smithi118.stdout: "default_value": "False", 2024-03-24T03:17:23.718 INFO:teuthology.orchestra.run.smithi118.stdout: "min": "", 2024-03-24T03:17:23.719 INFO:teuthology.orchestra.run.smithi118.stdout: "max": "", 2024-03-24T03:17:23.719 INFO:teuthology.orchestra.run.smithi118.stdout: "enum_allowed": [], 2024-03-24T03:17:23.719 INFO:teuthology.orchestra.run.smithi118.stdout: "desc": "", 2024-03-24T03:17:23.719 INFO:teuthology.orchestra.run.smithi118.stdout: "long_desc": "", 2024-03-24T03:17:23.719 INFO:teuthology.orchestra.run.smithi118.stdout: "tags": [], 2024-03-24T03:17:23.719 INFO:teuthology.orchestra.run.smithi118.stdout: "see_also": [] 2024-03-24T03:17:23.719 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-03-24T03:17:23.719 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-03-24T03:17:23.719 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-24T03:17:23.720 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-24T03:17:23.720 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "rook", 2024-03-24T03:17:23.720 INFO:teuthology.orchestra.run.smithi118.stdout: "can_run": true, 2024-03-24T03:17:23.720 INFO:teuthology.orchestra.run.smithi118.stdout: "error_string": "", 2024-03-24T03:17:23.720 INFO:teuthology.orchestra.run.smithi118.stdout: "module_options": { 2024-03-24T03:17:23.720 INFO:teuthology.orchestra.run.smithi118.stdout: "log_level": { 2024-03-24T03:17:23.720 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "log_level", 2024-03-24T03:17:23.720 INFO:teuthology.orchestra.run.smithi118.stdout: "type": "str", 2024-03-24T03:17:23.720 INFO:teuthology.orchestra.run.smithi118.stdout: "level": "advanced", 2024-03-24T03:17:23.721 INFO:teuthology.orchestra.run.smithi118.stdout: "flags": 1, 2024-03-24T03:17:23.721 INFO:teuthology.orchestra.run.smithi118.stdout: "default_value": "", 2024-03-24T03:17:23.721 INFO:teuthology.orchestra.run.smithi118.stdout: "min": "", 2024-03-24T03:17:23.721 INFO:teuthology.orchestra.run.smithi118.stdout: "max": "", 2024-03-24T03:17:23.721 INFO:teuthology.orchestra.run.smithi118.stdout: "enum_allowed": [ 2024-03-24T03:17:23.721 INFO:teuthology.orchestra.run.smithi118.stdout: "", 2024-03-24T03:17:23.721 INFO:teuthology.orchestra.run.smithi118.stdout: "critical", 2024-03-24T03:17:23.721 INFO:teuthology.orchestra.run.smithi118.stdout: "debug", 2024-03-24T03:17:23.721 INFO:teuthology.orchestra.run.smithi118.stdout: "error", 2024-03-24T03:17:23.722 INFO:teuthology.orchestra.run.smithi118.stdout: "info", 2024-03-24T03:17:23.722 INFO:teuthology.orchestra.run.smithi118.stdout: "warning" 2024-03-24T03:17:23.722 INFO:teuthology.orchestra.run.smithi118.stdout: ], 2024-03-24T03:17:23.722 INFO:teuthology.orchestra.run.smithi118.stdout: "desc": "", 2024-03-24T03:17:23.722 INFO:teuthology.orchestra.run.smithi118.stdout: "long_desc": "", 2024-03-24T03:17:23.722 INFO:teuthology.orchestra.run.smithi118.stdout: "tags": [], 2024-03-24T03:17:23.722 INFO:teuthology.orchestra.run.smithi118.stdout: "see_also": [] 2024-03-24T03:17:23.722 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-24T03:17:23.722 INFO:teuthology.orchestra.run.smithi118.stdout: "log_to_cluster": { 2024-03-24T03:17:23.723 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "log_to_cluster", 2024-03-24T03:17:23.723 INFO:teuthology.orchestra.run.smithi118.stdout: "type": "bool", 2024-03-24T03:17:23.723 INFO:teuthology.orchestra.run.smithi118.stdout: "level": "advanced", 2024-03-24T03:17:23.723 INFO:teuthology.orchestra.run.smithi118.stdout: "flags": 1, 2024-03-24T03:17:23.723 INFO:teuthology.orchestra.run.smithi118.stdout: "default_value": "False", 2024-03-24T03:17:23.723 INFO:teuthology.orchestra.run.smithi118.stdout: "min": "", 2024-03-24T03:17:23.723 INFO:teuthology.orchestra.run.smithi118.stdout: "max": "", 2024-03-24T03:17:23.723 INFO:teuthology.orchestra.run.smithi118.stdout: "enum_allowed": [], 2024-03-24T03:17:23.723 INFO:teuthology.orchestra.run.smithi118.stdout: "desc": "", 2024-03-24T03:17:23.724 INFO:teuthology.orchestra.run.smithi118.stdout: "long_desc": "", 2024-03-24T03:17:23.724 INFO:teuthology.orchestra.run.smithi118.stdout: "tags": [], 2024-03-24T03:17:23.724 INFO:teuthology.orchestra.run.smithi118.stdout: "see_also": [] 2024-03-24T03:17:23.724 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-24T03:17:23.724 INFO:teuthology.orchestra.run.smithi118.stdout: "log_to_cluster_level": { 2024-03-24T03:17:23.724 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "log_to_cluster_level", 2024-03-24T03:17:23.724 INFO:teuthology.orchestra.run.smithi118.stdout: "type": "str", 2024-03-24T03:17:23.724 INFO:teuthology.orchestra.run.smithi118.stdout: "level": "advanced", 2024-03-24T03:17:23.725 INFO:teuthology.orchestra.run.smithi118.stdout: "flags": 1, 2024-03-24T03:17:23.725 INFO:teuthology.orchestra.run.smithi118.stdout: "default_value": "info", 2024-03-24T03:17:23.725 INFO:teuthology.orchestra.run.smithi118.stdout: "min": "", 2024-03-24T03:17:23.725 INFO:teuthology.orchestra.run.smithi118.stdout: "max": "", 2024-03-24T03:17:23.725 INFO:teuthology.orchestra.run.smithi118.stdout: "enum_allowed": [ 2024-03-24T03:17:23.725 INFO:teuthology.orchestra.run.smithi118.stdout: "", 2024-03-24T03:17:23.725 INFO:teuthology.orchestra.run.smithi118.stdout: "critical", 2024-03-24T03:17:23.725 INFO:teuthology.orchestra.run.smithi118.stdout: "debug", 2024-03-24T03:17:23.725 INFO:teuthology.orchestra.run.smithi118.stdout: "error", 2024-03-24T03:17:23.726 INFO:teuthology.orchestra.run.smithi118.stdout: "info", 2024-03-24T03:17:23.726 INFO:teuthology.orchestra.run.smithi118.stdout: "warning" 2024-03-24T03:17:23.726 INFO:teuthology.orchestra.run.smithi118.stdout: ], 2024-03-24T03:17:23.726 INFO:teuthology.orchestra.run.smithi118.stdout: "desc": "", 2024-03-24T03:17:23.726 INFO:teuthology.orchestra.run.smithi118.stdout: "long_desc": "", 2024-03-24T03:17:23.726 INFO:teuthology.orchestra.run.smithi118.stdout: "tags": [], 2024-03-24T03:17:23.726 INFO:teuthology.orchestra.run.smithi118.stdout: "see_also": [] 2024-03-24T03:17:23.726 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-24T03:17:23.726 INFO:teuthology.orchestra.run.smithi118.stdout: "log_to_file": { 2024-03-24T03:17:23.727 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "log_to_file", 2024-03-24T03:17:23.727 INFO:teuthology.orchestra.run.smithi118.stdout: "type": "bool", 2024-03-24T03:17:23.727 INFO:teuthology.orchestra.run.smithi118.stdout: "level": "advanced", 2024-03-24T03:17:23.727 INFO:teuthology.orchestra.run.smithi118.stdout: "flags": 1, 2024-03-24T03:17:23.727 INFO:teuthology.orchestra.run.smithi118.stdout: "default_value": "False", 2024-03-24T03:17:23.727 INFO:teuthology.orchestra.run.smithi118.stdout: "min": "", 2024-03-24T03:17:23.727 INFO:teuthology.orchestra.run.smithi118.stdout: "max": "", 2024-03-24T03:17:23.727 INFO:teuthology.orchestra.run.smithi118.stdout: "enum_allowed": [], 2024-03-24T03:17:23.727 INFO:teuthology.orchestra.run.smithi118.stdout: "desc": "", 2024-03-24T03:17:23.727 INFO:teuthology.orchestra.run.smithi118.stdout: "long_desc": "", 2024-03-24T03:17:23.728 INFO:teuthology.orchestra.run.smithi118.stdout: "tags": [], 2024-03-24T03:17:23.728 INFO:teuthology.orchestra.run.smithi118.stdout: "see_also": [] 2024-03-24T03:17:23.728 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-24T03:17:23.728 INFO:teuthology.orchestra.run.smithi118.stdout: "storage_class": { 2024-03-24T03:17:23.728 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "storage_class", 2024-03-24T03:17:23.728 INFO:teuthology.orchestra.run.smithi118.stdout: "type": "str", 2024-03-24T03:17:23.728 INFO:teuthology.orchestra.run.smithi118.stdout: "level": "advanced", 2024-03-24T03:17:23.728 INFO:teuthology.orchestra.run.smithi118.stdout: "flags": 0, 2024-03-24T03:17:23.728 INFO:teuthology.orchestra.run.smithi118.stdout: "default_value": "local", 2024-03-24T03:17:23.729 INFO:teuthology.orchestra.run.smithi118.stdout: "min": "", 2024-03-24T03:17:23.729 INFO:teuthology.orchestra.run.smithi118.stdout: "max": "", 2024-03-24T03:17:23.729 INFO:teuthology.orchestra.run.smithi118.stdout: "enum_allowed": [], 2024-03-24T03:17:23.729 INFO:teuthology.orchestra.run.smithi118.stdout: "desc": "storage class name for LSO-discovered PVs", 2024-03-24T03:17:23.729 INFO:teuthology.orchestra.run.smithi118.stdout: "long_desc": "", 2024-03-24T03:17:23.729 INFO:teuthology.orchestra.run.smithi118.stdout: "tags": [], 2024-03-24T03:17:23.729 INFO:teuthology.orchestra.run.smithi118.stdout: "see_also": [] 2024-03-24T03:17:23.729 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-03-24T03:17:23.729 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-03-24T03:17:23.730 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-24T03:17:23.730 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-24T03:17:23.730 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "selftest", 2024-03-24T03:17:23.730 INFO:teuthology.orchestra.run.smithi118.stdout: "can_run": true, 2024-03-24T03:17:23.730 INFO:teuthology.orchestra.run.smithi118.stdout: "error_string": "", 2024-03-24T03:17:23.730 INFO:teuthology.orchestra.run.smithi118.stdout: "module_options": { 2024-03-24T03:17:23.730 INFO:teuthology.orchestra.run.smithi118.stdout: "log_level": { 2024-03-24T03:17:23.730 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "log_level", 2024-03-24T03:17:23.730 INFO:teuthology.orchestra.run.smithi118.stdout: "type": "str", 2024-03-24T03:17:23.731 INFO:teuthology.orchestra.run.smithi118.stdout: "level": "advanced", 2024-03-24T03:17:23.731 INFO:teuthology.orchestra.run.smithi118.stdout: "flags": 1, 2024-03-24T03:17:23.731 INFO:teuthology.orchestra.run.smithi118.stdout: "default_value": "", 2024-03-24T03:17:23.731 INFO:teuthology.orchestra.run.smithi118.stdout: "min": "", 2024-03-24T03:17:23.731 INFO:teuthology.orchestra.run.smithi118.stdout: "max": "", 2024-03-24T03:17:23.731 INFO:teuthology.orchestra.run.smithi118.stdout: "enum_allowed": [ 2024-03-24T03:17:23.731 INFO:teuthology.orchestra.run.smithi118.stdout: "", 2024-03-24T03:17:23.731 INFO:teuthology.orchestra.run.smithi118.stdout: "critical", 2024-03-24T03:17:23.731 INFO:teuthology.orchestra.run.smithi118.stdout: "debug", 2024-03-24T03:17:23.732 INFO:teuthology.orchestra.run.smithi118.stdout: "error", 2024-03-24T03:17:23.732 INFO:teuthology.orchestra.run.smithi118.stdout: "info", 2024-03-24T03:17:23.732 INFO:teuthology.orchestra.run.smithi118.stdout: "warning" 2024-03-24T03:17:23.732 INFO:teuthology.orchestra.run.smithi118.stdout: ], 2024-03-24T03:17:23.732 INFO:teuthology.orchestra.run.smithi118.stdout: "desc": "", 2024-03-24T03:17:23.732 INFO:teuthology.orchestra.run.smithi118.stdout: "long_desc": "", 2024-03-24T03:17:23.732 INFO:teuthology.orchestra.run.smithi118.stdout: "tags": [], 2024-03-24T03:17:23.732 INFO:teuthology.orchestra.run.smithi118.stdout: "see_also": [] 2024-03-24T03:17:23.732 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-24T03:17:23.733 INFO:teuthology.orchestra.run.smithi118.stdout: "log_to_cluster": { 2024-03-24T03:17:23.733 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "log_to_cluster", 2024-03-24T03:17:23.733 INFO:teuthology.orchestra.run.smithi118.stdout: "type": "bool", 2024-03-24T03:17:23.733 INFO:teuthology.orchestra.run.smithi118.stdout: "level": "advanced", 2024-03-24T03:17:23.733 INFO:teuthology.orchestra.run.smithi118.stdout: "flags": 1, 2024-03-24T03:17:23.733 INFO:teuthology.orchestra.run.smithi118.stdout: "default_value": "False", 2024-03-24T03:17:23.733 INFO:teuthology.orchestra.run.smithi118.stdout: "min": "", 2024-03-24T03:17:23.733 INFO:teuthology.orchestra.run.smithi118.stdout: "max": "", 2024-03-24T03:17:23.733 INFO:teuthology.orchestra.run.smithi118.stdout: "enum_allowed": [], 2024-03-24T03:17:23.734 INFO:teuthology.orchestra.run.smithi118.stdout: "desc": "", 2024-03-24T03:17:23.734 INFO:teuthology.orchestra.run.smithi118.stdout: "long_desc": "", 2024-03-24T03:17:23.734 INFO:teuthology.orchestra.run.smithi118.stdout: "tags": [], 2024-03-24T03:17:23.734 INFO:teuthology.orchestra.run.smithi118.stdout: "see_also": [] 2024-03-24T03:17:23.734 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-24T03:17:23.734 INFO:teuthology.orchestra.run.smithi118.stdout: "log_to_cluster_level": { 2024-03-24T03:17:23.734 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "log_to_cluster_level", 2024-03-24T03:17:23.734 INFO:teuthology.orchestra.run.smithi118.stdout: "type": "str", 2024-03-24T03:17:23.734 INFO:teuthology.orchestra.run.smithi118.stdout: "level": "advanced", 2024-03-24T03:17:23.735 INFO:teuthology.orchestra.run.smithi118.stdout: "flags": 1, 2024-03-24T03:17:23.735 INFO:teuthology.orchestra.run.smithi118.stdout: "default_value": "info", 2024-03-24T03:17:23.735 INFO:teuthology.orchestra.run.smithi118.stdout: "min": "", 2024-03-24T03:17:23.735 INFO:teuthology.orchestra.run.smithi118.stdout: "max": "", 2024-03-24T03:17:23.735 INFO:teuthology.orchestra.run.smithi118.stdout: "enum_allowed": [ 2024-03-24T03:17:23.735 INFO:teuthology.orchestra.run.smithi118.stdout: "", 2024-03-24T03:17:23.735 INFO:teuthology.orchestra.run.smithi118.stdout: "critical", 2024-03-24T03:17:23.735 INFO:teuthology.orchestra.run.smithi118.stdout: "debug", 2024-03-24T03:17:23.735 INFO:teuthology.orchestra.run.smithi118.stdout: "error", 2024-03-24T03:17:23.736 INFO:teuthology.orchestra.run.smithi118.stdout: "info", 2024-03-24T03:17:23.736 INFO:teuthology.orchestra.run.smithi118.stdout: "warning" 2024-03-24T03:17:23.736 INFO:teuthology.orchestra.run.smithi118.stdout: ], 2024-03-24T03:17:23.736 INFO:teuthology.orchestra.run.smithi118.stdout: "desc": "", 2024-03-24T03:17:23.736 INFO:teuthology.orchestra.run.smithi118.stdout: "long_desc": "", 2024-03-24T03:17:23.736 INFO:teuthology.orchestra.run.smithi118.stdout: "tags": [], 2024-03-24T03:17:23.736 INFO:teuthology.orchestra.run.smithi118.stdout: "see_also": [] 2024-03-24T03:17:23.736 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-24T03:17:23.736 INFO:teuthology.orchestra.run.smithi118.stdout: "log_to_file": { 2024-03-24T03:17:23.736 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "log_to_file", 2024-03-24T03:17:23.737 INFO:teuthology.orchestra.run.smithi118.stdout: "type": "bool", 2024-03-24T03:17:23.737 INFO:teuthology.orchestra.run.smithi118.stdout: "level": "advanced", 2024-03-24T03:17:23.737 INFO:teuthology.orchestra.run.smithi118.stdout: "flags": 1, 2024-03-24T03:17:23.737 INFO:teuthology.orchestra.run.smithi118.stdout: "default_value": "False", 2024-03-24T03:17:23.737 INFO:teuthology.orchestra.run.smithi118.stdout: "min": "", 2024-03-24T03:17:23.737 INFO:teuthology.orchestra.run.smithi118.stdout: "max": "", 2024-03-24T03:17:23.737 INFO:teuthology.orchestra.run.smithi118.stdout: "enum_allowed": [], 2024-03-24T03:17:23.737 INFO:teuthology.orchestra.run.smithi118.stdout: "desc": "", 2024-03-24T03:17:23.737 INFO:teuthology.orchestra.run.smithi118.stdout: "long_desc": "", 2024-03-24T03:17:23.738 INFO:teuthology.orchestra.run.smithi118.stdout: "tags": [], 2024-03-24T03:17:23.738 INFO:teuthology.orchestra.run.smithi118.stdout: "see_also": [] 2024-03-24T03:17:23.738 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-24T03:17:23.738 INFO:teuthology.orchestra.run.smithi118.stdout: "roption1": { 2024-03-24T03:17:23.738 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "roption1", 2024-03-24T03:17:23.738 INFO:teuthology.orchestra.run.smithi118.stdout: "type": "str", 2024-03-24T03:17:23.738 INFO:teuthology.orchestra.run.smithi118.stdout: "level": "advanced", 2024-03-24T03:17:23.738 INFO:teuthology.orchestra.run.smithi118.stdout: "flags": 0, 2024-03-24T03:17:23.739 INFO:teuthology.orchestra.run.smithi118.stdout: "default_value": "", 2024-03-24T03:17:23.739 INFO:teuthology.orchestra.run.smithi118.stdout: "min": "", 2024-03-24T03:17:23.739 INFO:teuthology.orchestra.run.smithi118.stdout: "max": "", 2024-03-24T03:17:23.739 INFO:teuthology.orchestra.run.smithi118.stdout: "enum_allowed": [], 2024-03-24T03:17:23.739 INFO:teuthology.orchestra.run.smithi118.stdout: "desc": "", 2024-03-24T03:17:23.739 INFO:teuthology.orchestra.run.smithi118.stdout: "long_desc": "", 2024-03-24T03:17:23.739 INFO:teuthology.orchestra.run.smithi118.stdout: "tags": [], 2024-03-24T03:17:23.739 INFO:teuthology.orchestra.run.smithi118.stdout: "see_also": [] 2024-03-24T03:17:23.739 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-24T03:17:23.740 INFO:teuthology.orchestra.run.smithi118.stdout: "roption2": { 2024-03-24T03:17:23.740 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "roption2", 2024-03-24T03:17:23.740 INFO:teuthology.orchestra.run.smithi118.stdout: "type": "str", 2024-03-24T03:17:23.740 INFO:teuthology.orchestra.run.smithi118.stdout: "level": "advanced", 2024-03-24T03:17:23.740 INFO:teuthology.orchestra.run.smithi118.stdout: "flags": 0, 2024-03-24T03:17:23.740 INFO:teuthology.orchestra.run.smithi118.stdout: "default_value": "xyz", 2024-03-24T03:17:23.740 INFO:teuthology.orchestra.run.smithi118.stdout: "min": "", 2024-03-24T03:17:23.740 INFO:teuthology.orchestra.run.smithi118.stdout: "max": "", 2024-03-24T03:17:23.740 INFO:teuthology.orchestra.run.smithi118.stdout: "enum_allowed": [], 2024-03-24T03:17:23.740 INFO:teuthology.orchestra.run.smithi118.stdout: "desc": "", 2024-03-24T03:17:23.741 INFO:teuthology.orchestra.run.smithi118.stdout: "long_desc": "", 2024-03-24T03:17:23.741 INFO:teuthology.orchestra.run.smithi118.stdout: "tags": [], 2024-03-24T03:17:23.741 INFO:teuthology.orchestra.run.smithi118.stdout: "see_also": [] 2024-03-24T03:17:23.741 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-24T03:17:23.741 INFO:teuthology.orchestra.run.smithi118.stdout: "rwoption1": { 2024-03-24T03:17:23.741 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "rwoption1", 2024-03-24T03:17:23.741 INFO:teuthology.orchestra.run.smithi118.stdout: "type": "str", 2024-03-24T03:17:23.741 INFO:teuthology.orchestra.run.smithi118.stdout: "level": "advanced", 2024-03-24T03:17:23.741 INFO:teuthology.orchestra.run.smithi118.stdout: "flags": 0, 2024-03-24T03:17:23.742 INFO:teuthology.orchestra.run.smithi118.stdout: "default_value": "", 2024-03-24T03:17:23.742 INFO:teuthology.orchestra.run.smithi118.stdout: "min": "", 2024-03-24T03:17:23.742 INFO:teuthology.orchestra.run.smithi118.stdout: "max": "", 2024-03-24T03:17:23.742 INFO:teuthology.orchestra.run.smithi118.stdout: "enum_allowed": [], 2024-03-24T03:17:23.742 INFO:teuthology.orchestra.run.smithi118.stdout: "desc": "", 2024-03-24T03:17:23.742 INFO:teuthology.orchestra.run.smithi118.stdout: "long_desc": "", 2024-03-24T03:17:23.742 INFO:teuthology.orchestra.run.smithi118.stdout: "tags": [], 2024-03-24T03:17:23.742 INFO:teuthology.orchestra.run.smithi118.stdout: "see_also": [] 2024-03-24T03:17:23.743 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-24T03:17:23.743 INFO:teuthology.orchestra.run.smithi118.stdout: "rwoption2": { 2024-03-24T03:17:23.743 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "rwoption2", 2024-03-24T03:17:23.743 INFO:teuthology.orchestra.run.smithi118.stdout: "type": "int", 2024-03-24T03:17:23.743 INFO:teuthology.orchestra.run.smithi118.stdout: "level": "advanced", 2024-03-24T03:17:23.743 INFO:teuthology.orchestra.run.smithi118.stdout: "flags": 0, 2024-03-24T03:17:23.743 INFO:teuthology.orchestra.run.smithi118.stdout: "default_value": "", 2024-03-24T03:17:23.743 INFO:teuthology.orchestra.run.smithi118.stdout: "min": "", 2024-03-24T03:17:23.743 INFO:teuthology.orchestra.run.smithi118.stdout: "max": "", 2024-03-24T03:17:23.744 INFO:teuthology.orchestra.run.smithi118.stdout: "enum_allowed": [], 2024-03-24T03:17:23.744 INFO:teuthology.orchestra.run.smithi118.stdout: "desc": "", 2024-03-24T03:17:23.744 INFO:teuthology.orchestra.run.smithi118.stdout: "long_desc": "", 2024-03-24T03:17:23.744 INFO:teuthology.orchestra.run.smithi118.stdout: "tags": [], 2024-03-24T03:17:23.744 INFO:teuthology.orchestra.run.smithi118.stdout: "see_also": [] 2024-03-24T03:17:23.744 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-24T03:17:23.744 INFO:teuthology.orchestra.run.smithi118.stdout: "rwoption3": { 2024-03-24T03:17:23.744 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "rwoption3", 2024-03-24T03:17:23.744 INFO:teuthology.orchestra.run.smithi118.stdout: "type": "float", 2024-03-24T03:17:23.745 INFO:teuthology.orchestra.run.smithi118.stdout: "level": "advanced", 2024-03-24T03:17:23.745 INFO:teuthology.orchestra.run.smithi118.stdout: "flags": 0, 2024-03-24T03:17:23.745 INFO:teuthology.orchestra.run.smithi118.stdout: "default_value": "", 2024-03-24T03:17:23.745 INFO:teuthology.orchestra.run.smithi118.stdout: "min": "", 2024-03-24T03:17:23.745 INFO:teuthology.orchestra.run.smithi118.stdout: "max": "", 2024-03-24T03:17:23.745 INFO:teuthology.orchestra.run.smithi118.stdout: "enum_allowed": [], 2024-03-24T03:17:23.745 INFO:teuthology.orchestra.run.smithi118.stdout: "desc": "", 2024-03-24T03:17:23.745 INFO:teuthology.orchestra.run.smithi118.stdout: "long_desc": "", 2024-03-24T03:17:23.745 INFO:teuthology.orchestra.run.smithi118.stdout: "tags": [], 2024-03-24T03:17:23.746 INFO:teuthology.orchestra.run.smithi118.stdout: "see_also": [] 2024-03-24T03:17:23.746 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-24T03:17:23.746 INFO:teuthology.orchestra.run.smithi118.stdout: "rwoption4": { 2024-03-24T03:17:23.746 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "rwoption4", 2024-03-24T03:17:23.746 INFO:teuthology.orchestra.run.smithi118.stdout: "type": "str", 2024-03-24T03:17:23.746 INFO:teuthology.orchestra.run.smithi118.stdout: "level": "advanced", 2024-03-24T03:17:23.746 INFO:teuthology.orchestra.run.smithi118.stdout: "flags": 0, 2024-03-24T03:17:23.746 INFO:teuthology.orchestra.run.smithi118.stdout: "default_value": "", 2024-03-24T03:17:23.746 INFO:teuthology.orchestra.run.smithi118.stdout: "min": "", 2024-03-24T03:17:23.747 INFO:teuthology.orchestra.run.smithi118.stdout: "max": "", 2024-03-24T03:17:23.747 INFO:teuthology.orchestra.run.smithi118.stdout: "enum_allowed": [], 2024-03-24T03:17:23.747 INFO:teuthology.orchestra.run.smithi118.stdout: "desc": "", 2024-03-24T03:17:23.747 INFO:teuthology.orchestra.run.smithi118.stdout: "long_desc": "", 2024-03-24T03:17:23.747 INFO:teuthology.orchestra.run.smithi118.stdout: "tags": [], 2024-03-24T03:17:23.747 INFO:teuthology.orchestra.run.smithi118.stdout: "see_also": [] 2024-03-24T03:17:23.747 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-24T03:17:23.747 INFO:teuthology.orchestra.run.smithi118.stdout: "rwoption5": { 2024-03-24T03:17:23.747 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "rwoption5", 2024-03-24T03:17:23.747 INFO:teuthology.orchestra.run.smithi118.stdout: "type": "bool", 2024-03-24T03:17:23.748 INFO:teuthology.orchestra.run.smithi118.stdout: "level": "advanced", 2024-03-24T03:17:23.748 INFO:teuthology.orchestra.run.smithi118.stdout: "flags": 0, 2024-03-24T03:17:23.748 INFO:teuthology.orchestra.run.smithi118.stdout: "default_value": "", 2024-03-24T03:17:23.748 INFO:teuthology.orchestra.run.smithi118.stdout: "min": "", 2024-03-24T03:17:23.748 INFO:teuthology.orchestra.run.smithi118.stdout: "max": "", 2024-03-24T03:17:23.748 INFO:teuthology.orchestra.run.smithi118.stdout: "enum_allowed": [], 2024-03-24T03:17:23.748 INFO:teuthology.orchestra.run.smithi118.stdout: "desc": "", 2024-03-24T03:17:23.748 INFO:teuthology.orchestra.run.smithi118.stdout: "long_desc": "", 2024-03-24T03:17:23.748 INFO:teuthology.orchestra.run.smithi118.stdout: "tags": [], 2024-03-24T03:17:23.749 INFO:teuthology.orchestra.run.smithi118.stdout: "see_also": [] 2024-03-24T03:17:23.749 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-24T03:17:23.749 INFO:teuthology.orchestra.run.smithi118.stdout: "rwoption6": { 2024-03-24T03:17:23.749 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "rwoption6", 2024-03-24T03:17:23.749 INFO:teuthology.orchestra.run.smithi118.stdout: "type": "bool", 2024-03-24T03:17:23.749 INFO:teuthology.orchestra.run.smithi118.stdout: "level": "advanced", 2024-03-24T03:17:23.749 INFO:teuthology.orchestra.run.smithi118.stdout: "flags": 0, 2024-03-24T03:17:23.749 INFO:teuthology.orchestra.run.smithi118.stdout: "default_value": "True", 2024-03-24T03:17:23.749 INFO:teuthology.orchestra.run.smithi118.stdout: "min": "", 2024-03-24T03:17:23.750 INFO:teuthology.orchestra.run.smithi118.stdout: "max": "", 2024-03-24T03:17:23.750 INFO:teuthology.orchestra.run.smithi118.stdout: "enum_allowed": [], 2024-03-24T03:17:23.750 INFO:teuthology.orchestra.run.smithi118.stdout: "desc": "", 2024-03-24T03:17:23.750 INFO:teuthology.orchestra.run.smithi118.stdout: "long_desc": "", 2024-03-24T03:17:23.750 INFO:teuthology.orchestra.run.smithi118.stdout: "tags": [], 2024-03-24T03:17:23.750 INFO:teuthology.orchestra.run.smithi118.stdout: "see_also": [] 2024-03-24T03:17:23.750 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-24T03:17:23.750 INFO:teuthology.orchestra.run.smithi118.stdout: "rwoption7": { 2024-03-24T03:17:23.750 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "rwoption7", 2024-03-24T03:17:23.751 INFO:teuthology.orchestra.run.smithi118.stdout: "type": "int", 2024-03-24T03:17:23.751 INFO:teuthology.orchestra.run.smithi118.stdout: "level": "advanced", 2024-03-24T03:17:23.751 INFO:teuthology.orchestra.run.smithi118.stdout: "flags": 0, 2024-03-24T03:17:23.751 INFO:teuthology.orchestra.run.smithi118.stdout: "default_value": "", 2024-03-24T03:17:23.751 INFO:teuthology.orchestra.run.smithi118.stdout: "min": "1", 2024-03-24T03:17:23.751 INFO:teuthology.orchestra.run.smithi118.stdout: "max": "42", 2024-03-24T03:17:23.751 INFO:teuthology.orchestra.run.smithi118.stdout: "enum_allowed": [], 2024-03-24T03:17:23.751 INFO:teuthology.orchestra.run.smithi118.stdout: "desc": "", 2024-03-24T03:17:23.751 INFO:teuthology.orchestra.run.smithi118.stdout: "long_desc": "", 2024-03-24T03:17:23.751 INFO:teuthology.orchestra.run.smithi118.stdout: "tags": [], 2024-03-24T03:17:23.752 INFO:teuthology.orchestra.run.smithi118.stdout: "see_also": [] 2024-03-24T03:17:23.752 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-24T03:17:23.752 INFO:teuthology.orchestra.run.smithi118.stdout: "testkey": { 2024-03-24T03:17:23.752 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "testkey", 2024-03-24T03:17:23.752 INFO:teuthology.orchestra.run.smithi118.stdout: "type": "str", 2024-03-24T03:17:23.752 INFO:teuthology.orchestra.run.smithi118.stdout: "level": "advanced", 2024-03-24T03:17:23.752 INFO:teuthology.orchestra.run.smithi118.stdout: "flags": 0, 2024-03-24T03:17:23.753 INFO:teuthology.orchestra.run.smithi118.stdout: "default_value": "", 2024-03-24T03:17:23.753 INFO:teuthology.orchestra.run.smithi118.stdout: "min": "", 2024-03-24T03:17:23.753 INFO:teuthology.orchestra.run.smithi118.stdout: "max": "", 2024-03-24T03:17:23.753 INFO:teuthology.orchestra.run.smithi118.stdout: "enum_allowed": [], 2024-03-24T03:17:23.753 INFO:teuthology.orchestra.run.smithi118.stdout: "desc": "", 2024-03-24T03:17:23.753 INFO:teuthology.orchestra.run.smithi118.stdout: "long_desc": "", 2024-03-24T03:17:23.753 INFO:teuthology.orchestra.run.smithi118.stdout: "tags": [], 2024-03-24T03:17:23.753 INFO:teuthology.orchestra.run.smithi118.stdout: "see_also": [] 2024-03-24T03:17:23.753 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-24T03:17:23.754 INFO:teuthology.orchestra.run.smithi118.stdout: "testlkey": { 2024-03-24T03:17:23.754 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "testlkey", 2024-03-24T03:17:23.754 INFO:teuthology.orchestra.run.smithi118.stdout: "type": "str", 2024-03-24T03:17:23.754 INFO:teuthology.orchestra.run.smithi118.stdout: "level": "advanced", 2024-03-24T03:17:23.754 INFO:teuthology.orchestra.run.smithi118.stdout: "flags": 0, 2024-03-24T03:17:23.754 INFO:teuthology.orchestra.run.smithi118.stdout: "default_value": "", 2024-03-24T03:17:23.754 INFO:teuthology.orchestra.run.smithi118.stdout: "min": "", 2024-03-24T03:17:23.755 INFO:teuthology.orchestra.run.smithi118.stdout: "max": "", 2024-03-24T03:17:23.755 INFO:teuthology.orchestra.run.smithi118.stdout: "enum_allowed": [], 2024-03-24T03:17:23.755 INFO:teuthology.orchestra.run.smithi118.stdout: "desc": "", 2024-03-24T03:17:23.755 INFO:teuthology.orchestra.run.smithi118.stdout: "long_desc": "", 2024-03-24T03:17:23.755 INFO:teuthology.orchestra.run.smithi118.stdout: "tags": [], 2024-03-24T03:17:23.755 INFO:teuthology.orchestra.run.smithi118.stdout: "see_also": [] 2024-03-24T03:17:23.755 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-24T03:17:23.755 INFO:teuthology.orchestra.run.smithi118.stdout: "testnewline": { 2024-03-24T03:17:23.756 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "testnewline", 2024-03-24T03:17:23.756 INFO:teuthology.orchestra.run.smithi118.stdout: "type": "str", 2024-03-24T03:17:23.756 INFO:teuthology.orchestra.run.smithi118.stdout: "level": "advanced", 2024-03-24T03:17:23.756 INFO:teuthology.orchestra.run.smithi118.stdout: "flags": 0, 2024-03-24T03:17:23.756 INFO:teuthology.orchestra.run.smithi118.stdout: "default_value": "", 2024-03-24T03:17:23.756 INFO:teuthology.orchestra.run.smithi118.stdout: "min": "", 2024-03-24T03:17:23.756 INFO:teuthology.orchestra.run.smithi118.stdout: "max": "", 2024-03-24T03:17:23.756 INFO:teuthology.orchestra.run.smithi118.stdout: "enum_allowed": [], 2024-03-24T03:17:23.756 INFO:teuthology.orchestra.run.smithi118.stdout: "desc": "", 2024-03-24T03:17:23.757 INFO:teuthology.orchestra.run.smithi118.stdout: "long_desc": "", 2024-03-24T03:17:23.757 INFO:teuthology.orchestra.run.smithi118.stdout: "tags": [], 2024-03-24T03:17:23.757 INFO:teuthology.orchestra.run.smithi118.stdout: "see_also": [] 2024-03-24T03:17:23.757 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-03-24T03:17:23.757 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-03-24T03:17:23.757 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-24T03:17:23.757 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-24T03:17:23.757 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "snap_schedule", 2024-03-24T03:17:23.757 INFO:teuthology.orchestra.run.smithi118.stdout: "can_run": true, 2024-03-24T03:17:23.757 INFO:teuthology.orchestra.run.smithi118.stdout: "error_string": "", 2024-03-24T03:17:23.758 INFO:teuthology.orchestra.run.smithi118.stdout: "module_options": { 2024-03-24T03:17:23.758 INFO:teuthology.orchestra.run.smithi118.stdout: "allow_m_granularity": { 2024-03-24T03:17:23.758 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "allow_m_granularity", 2024-03-24T03:17:23.758 INFO:teuthology.orchestra.run.smithi118.stdout: "type": "bool", 2024-03-24T03:17:23.758 INFO:teuthology.orchestra.run.smithi118.stdout: "level": "advanced", 2024-03-24T03:17:23.758 INFO:teuthology.orchestra.run.smithi118.stdout: "flags": 1, 2024-03-24T03:17:23.758 INFO:teuthology.orchestra.run.smithi118.stdout: "default_value": "False", 2024-03-24T03:17:23.758 INFO:teuthology.orchestra.run.smithi118.stdout: "min": "", 2024-03-24T03:17:23.759 INFO:teuthology.orchestra.run.smithi118.stdout: "max": "", 2024-03-24T03:17:23.759 INFO:teuthology.orchestra.run.smithi118.stdout: "enum_allowed": [], 2024-03-24T03:17:23.759 INFO:teuthology.orchestra.run.smithi118.stdout: "desc": "allow minute scheduled snapshots", 2024-03-24T03:17:23.759 INFO:teuthology.orchestra.run.smithi118.stdout: "long_desc": "", 2024-03-24T03:17:23.759 INFO:teuthology.orchestra.run.smithi118.stdout: "tags": [], 2024-03-24T03:17:23.759 INFO:teuthology.orchestra.run.smithi118.stdout: "see_also": [] 2024-03-24T03:17:23.759 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-24T03:17:23.759 INFO:teuthology.orchestra.run.smithi118.stdout: "dump_on_update": { 2024-03-24T03:17:23.759 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "dump_on_update", 2024-03-24T03:17:23.760 INFO:teuthology.orchestra.run.smithi118.stdout: "type": "bool", 2024-03-24T03:17:23.760 INFO:teuthology.orchestra.run.smithi118.stdout: "level": "advanced", 2024-03-24T03:17:23.760 INFO:teuthology.orchestra.run.smithi118.stdout: "flags": 1, 2024-03-24T03:17:23.760 INFO:teuthology.orchestra.run.smithi118.stdout: "default_value": "False", 2024-03-24T03:17:23.760 INFO:teuthology.orchestra.run.smithi118.stdout: "min": "", 2024-03-24T03:17:23.760 INFO:teuthology.orchestra.run.smithi118.stdout: "max": "", 2024-03-24T03:17:23.760 INFO:teuthology.orchestra.run.smithi118.stdout: "enum_allowed": [], 2024-03-24T03:17:23.760 INFO:teuthology.orchestra.run.smithi118.stdout: "desc": "dump database to debug log on update", 2024-03-24T03:17:23.761 INFO:teuthology.orchestra.run.smithi118.stdout: "long_desc": "", 2024-03-24T03:17:23.761 INFO:teuthology.orchestra.run.smithi118.stdout: "tags": [], 2024-03-24T03:17:23.761 INFO:teuthology.orchestra.run.smithi118.stdout: "see_also": [] 2024-03-24T03:17:23.761 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-24T03:17:23.761 INFO:teuthology.orchestra.run.smithi118.stdout: "log_level": { 2024-03-24T03:17:23.761 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "log_level", 2024-03-24T03:17:23.761 INFO:teuthology.orchestra.run.smithi118.stdout: "type": "str", 2024-03-24T03:17:23.761 INFO:teuthology.orchestra.run.smithi118.stdout: "level": "advanced", 2024-03-24T03:17:23.762 INFO:teuthology.orchestra.run.smithi118.stdout: "flags": 1, 2024-03-24T03:17:23.762 INFO:teuthology.orchestra.run.smithi118.stdout: "default_value": "", 2024-03-24T03:17:23.762 INFO:teuthology.orchestra.run.smithi118.stdout: "min": "", 2024-03-24T03:17:23.762 INFO:teuthology.orchestra.run.smithi118.stdout: "max": "", 2024-03-24T03:17:23.762 INFO:teuthology.orchestra.run.smithi118.stdout: "enum_allowed": [ 2024-03-24T03:17:23.762 INFO:teuthology.orchestra.run.smithi118.stdout: "", 2024-03-24T03:17:23.762 INFO:teuthology.orchestra.run.smithi118.stdout: "critical", 2024-03-24T03:17:23.762 INFO:teuthology.orchestra.run.smithi118.stdout: "debug", 2024-03-24T03:17:23.762 INFO:teuthology.orchestra.run.smithi118.stdout: "error", 2024-03-24T03:17:23.763 INFO:teuthology.orchestra.run.smithi118.stdout: "info", 2024-03-24T03:17:23.763 INFO:teuthology.orchestra.run.smithi118.stdout: "warning" 2024-03-24T03:17:23.763 INFO:teuthology.orchestra.run.smithi118.stdout: ], 2024-03-24T03:17:23.763 INFO:teuthology.orchestra.run.smithi118.stdout: "desc": "", 2024-03-24T03:17:23.763 INFO:teuthology.orchestra.run.smithi118.stdout: "long_desc": "", 2024-03-24T03:17:23.763 INFO:teuthology.orchestra.run.smithi118.stdout: "tags": [], 2024-03-24T03:17:23.763 INFO:teuthology.orchestra.run.smithi118.stdout: "see_also": [] 2024-03-24T03:17:23.763 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-24T03:17:23.763 INFO:teuthology.orchestra.run.smithi118.stdout: "log_to_cluster": { 2024-03-24T03:17:23.764 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "log_to_cluster", 2024-03-24T03:17:23.764 INFO:teuthology.orchestra.run.smithi118.stdout: "type": "bool", 2024-03-24T03:17:23.764 INFO:teuthology.orchestra.run.smithi118.stdout: "level": "advanced", 2024-03-24T03:17:23.764 INFO:teuthology.orchestra.run.smithi118.stdout: "flags": 1, 2024-03-24T03:17:23.764 INFO:teuthology.orchestra.run.smithi118.stdout: "default_value": "False", 2024-03-24T03:17:23.764 INFO:teuthology.orchestra.run.smithi118.stdout: "min": "", 2024-03-24T03:17:23.764 INFO:teuthology.orchestra.run.smithi118.stdout: "max": "", 2024-03-24T03:17:23.764 INFO:teuthology.orchestra.run.smithi118.stdout: "enum_allowed": [], 2024-03-24T03:17:23.765 INFO:teuthology.orchestra.run.smithi118.stdout: "desc": "", 2024-03-24T03:17:23.765 INFO:teuthology.orchestra.run.smithi118.stdout: "long_desc": "", 2024-03-24T03:17:23.765 INFO:teuthology.orchestra.run.smithi118.stdout: "tags": [], 2024-03-24T03:17:23.765 INFO:teuthology.orchestra.run.smithi118.stdout: "see_also": [] 2024-03-24T03:17:23.765 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-24T03:17:23.765 INFO:teuthology.orchestra.run.smithi118.stdout: "log_to_cluster_level": { 2024-03-24T03:17:23.765 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "log_to_cluster_level", 2024-03-24T03:17:23.765 INFO:teuthology.orchestra.run.smithi118.stdout: "type": "str", 2024-03-24T03:17:23.765 INFO:teuthology.orchestra.run.smithi118.stdout: "level": "advanced", 2024-03-24T03:17:23.765 INFO:teuthology.orchestra.run.smithi118.stdout: "flags": 1, 2024-03-24T03:17:23.766 INFO:teuthology.orchestra.run.smithi118.stdout: "default_value": "info", 2024-03-24T03:17:23.766 INFO:teuthology.orchestra.run.smithi118.stdout: "min": "", 2024-03-24T03:17:23.766 INFO:teuthology.orchestra.run.smithi118.stdout: "max": "", 2024-03-24T03:17:23.766 INFO:teuthology.orchestra.run.smithi118.stdout: "enum_allowed": [ 2024-03-24T03:17:23.766 INFO:teuthology.orchestra.run.smithi118.stdout: "", 2024-03-24T03:17:23.766 INFO:teuthology.orchestra.run.smithi118.stdout: "critical", 2024-03-24T03:17:23.766 INFO:teuthology.orchestra.run.smithi118.stdout: "debug", 2024-03-24T03:17:23.767 INFO:teuthology.orchestra.run.smithi118.stdout: "error", 2024-03-24T03:17:23.767 INFO:teuthology.orchestra.run.smithi118.stdout: "info", 2024-03-24T03:17:23.767 INFO:teuthology.orchestra.run.smithi118.stdout: "warning" 2024-03-24T03:17:23.767 INFO:teuthology.orchestra.run.smithi118.stdout: ], 2024-03-24T03:17:23.767 INFO:teuthology.orchestra.run.smithi118.stdout: "desc": "", 2024-03-24T03:17:23.767 INFO:teuthology.orchestra.run.smithi118.stdout: "long_desc": "", 2024-03-24T03:17:23.767 INFO:teuthology.orchestra.run.smithi118.stdout: "tags": [], 2024-03-24T03:17:23.767 INFO:teuthology.orchestra.run.smithi118.stdout: "see_also": [] 2024-03-24T03:17:23.767 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-24T03:17:23.767 INFO:teuthology.orchestra.run.smithi118.stdout: "log_to_file": { 2024-03-24T03:17:23.768 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "log_to_file", 2024-03-24T03:17:23.768 INFO:teuthology.orchestra.run.smithi118.stdout: "type": "bool", 2024-03-24T03:17:23.768 INFO:teuthology.orchestra.run.smithi118.stdout: "level": "advanced", 2024-03-24T03:17:23.768 INFO:teuthology.orchestra.run.smithi118.stdout: "flags": 1, 2024-03-24T03:17:23.768 INFO:teuthology.orchestra.run.smithi118.stdout: "default_value": "False", 2024-03-24T03:17:23.768 INFO:teuthology.orchestra.run.smithi118.stdout: "min": "", 2024-03-24T03:17:23.768 INFO:teuthology.orchestra.run.smithi118.stdout: "max": "", 2024-03-24T03:17:23.768 INFO:teuthology.orchestra.run.smithi118.stdout: "enum_allowed": [], 2024-03-24T03:17:23.768 INFO:teuthology.orchestra.run.smithi118.stdout: "desc": "", 2024-03-24T03:17:23.769 INFO:teuthology.orchestra.run.smithi118.stdout: "long_desc": "", 2024-03-24T03:17:23.769 INFO:teuthology.orchestra.run.smithi118.stdout: "tags": [], 2024-03-24T03:17:23.769 INFO:teuthology.orchestra.run.smithi118.stdout: "see_also": [] 2024-03-24T03:17:23.769 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-03-24T03:17:23.769 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-03-24T03:17:23.769 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-24T03:17:23.769 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-24T03:17:23.769 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "stats", 2024-03-24T03:17:23.769 INFO:teuthology.orchestra.run.smithi118.stdout: "can_run": true, 2024-03-24T03:17:23.770 INFO:teuthology.orchestra.run.smithi118.stdout: "error_string": "", 2024-03-24T03:17:23.770 INFO:teuthology.orchestra.run.smithi118.stdout: "module_options": { 2024-03-24T03:17:23.770 INFO:teuthology.orchestra.run.smithi118.stdout: "log_level": { 2024-03-24T03:17:23.770 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "log_level", 2024-03-24T03:17:23.770 INFO:teuthology.orchestra.run.smithi118.stdout: "type": "str", 2024-03-24T03:17:23.770 INFO:teuthology.orchestra.run.smithi118.stdout: "level": "advanced", 2024-03-24T03:17:23.770 INFO:teuthology.orchestra.run.smithi118.stdout: "flags": 1, 2024-03-24T03:17:23.770 INFO:teuthology.orchestra.run.smithi118.stdout: "default_value": "", 2024-03-24T03:17:23.770 INFO:teuthology.orchestra.run.smithi118.stdout: "min": "", 2024-03-24T03:17:23.771 INFO:teuthology.orchestra.run.smithi118.stdout: "max": "", 2024-03-24T03:17:23.771 INFO:teuthology.orchestra.run.smithi118.stdout: "enum_allowed": [ 2024-03-24T03:17:23.771 INFO:teuthology.orchestra.run.smithi118.stdout: "", 2024-03-24T03:17:23.771 INFO:teuthology.orchestra.run.smithi118.stdout: "critical", 2024-03-24T03:17:23.771 INFO:teuthology.orchestra.run.smithi118.stdout: "debug", 2024-03-24T03:17:23.771 INFO:teuthology.orchestra.run.smithi118.stdout: "error", 2024-03-24T03:17:23.771 INFO:teuthology.orchestra.run.smithi118.stdout: "info", 2024-03-24T03:17:23.771 INFO:teuthology.orchestra.run.smithi118.stdout: "warning" 2024-03-24T03:17:23.771 INFO:teuthology.orchestra.run.smithi118.stdout: ], 2024-03-24T03:17:23.772 INFO:teuthology.orchestra.run.smithi118.stdout: "desc": "", 2024-03-24T03:17:23.772 INFO:teuthology.orchestra.run.smithi118.stdout: "long_desc": "", 2024-03-24T03:17:23.772 INFO:teuthology.orchestra.run.smithi118.stdout: "tags": [], 2024-03-24T03:17:23.772 INFO:teuthology.orchestra.run.smithi118.stdout: "see_also": [] 2024-03-24T03:17:23.772 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-24T03:17:23.772 INFO:teuthology.orchestra.run.smithi118.stdout: "log_to_cluster": { 2024-03-24T03:17:23.772 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "log_to_cluster", 2024-03-24T03:17:23.772 INFO:teuthology.orchestra.run.smithi118.stdout: "type": "bool", 2024-03-24T03:17:23.772 INFO:teuthology.orchestra.run.smithi118.stdout: "level": "advanced", 2024-03-24T03:17:23.773 INFO:teuthology.orchestra.run.smithi118.stdout: "flags": 1, 2024-03-24T03:17:23.773 INFO:teuthology.orchestra.run.smithi118.stdout: "default_value": "False", 2024-03-24T03:17:23.773 INFO:teuthology.orchestra.run.smithi118.stdout: "min": "", 2024-03-24T03:17:23.773 INFO:teuthology.orchestra.run.smithi118.stdout: "max": "", 2024-03-24T03:17:23.773 INFO:teuthology.orchestra.run.smithi118.stdout: "enum_allowed": [], 2024-03-24T03:17:23.773 INFO:teuthology.orchestra.run.smithi118.stdout: "desc": "", 2024-03-24T03:17:23.773 INFO:teuthology.orchestra.run.smithi118.stdout: "long_desc": "", 2024-03-24T03:17:23.773 INFO:teuthology.orchestra.run.smithi118.stdout: "tags": [], 2024-03-24T03:17:23.773 INFO:teuthology.orchestra.run.smithi118.stdout: "see_also": [] 2024-03-24T03:17:23.774 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-24T03:17:23.774 INFO:teuthology.orchestra.run.smithi118.stdout: "log_to_cluster_level": { 2024-03-24T03:17:23.774 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "log_to_cluster_level", 2024-03-24T03:17:23.774 INFO:teuthology.orchestra.run.smithi118.stdout: "type": "str", 2024-03-24T03:17:23.774 INFO:teuthology.orchestra.run.smithi118.stdout: "level": "advanced", 2024-03-24T03:17:23.774 INFO:teuthology.orchestra.run.smithi118.stdout: "flags": 1, 2024-03-24T03:17:23.774 INFO:teuthology.orchestra.run.smithi118.stdout: "default_value": "info", 2024-03-24T03:17:23.774 INFO:teuthology.orchestra.run.smithi118.stdout: "min": "", 2024-03-24T03:17:23.775 INFO:teuthology.orchestra.run.smithi118.stdout: "max": "", 2024-03-24T03:17:23.775 INFO:teuthology.orchestra.run.smithi118.stdout: "enum_allowed": [ 2024-03-24T03:17:23.775 INFO:teuthology.orchestra.run.smithi118.stdout: "", 2024-03-24T03:17:23.775 INFO:teuthology.orchestra.run.smithi118.stdout: "critical", 2024-03-24T03:17:23.775 INFO:teuthology.orchestra.run.smithi118.stdout: "debug", 2024-03-24T03:17:23.775 INFO:teuthology.orchestra.run.smithi118.stdout: "error", 2024-03-24T03:17:23.775 INFO:teuthology.orchestra.run.smithi118.stdout: "info", 2024-03-24T03:17:23.775 INFO:teuthology.orchestra.run.smithi118.stdout: "warning" 2024-03-24T03:17:23.775 INFO:teuthology.orchestra.run.smithi118.stdout: ], 2024-03-24T03:17:23.776 INFO:teuthology.orchestra.run.smithi118.stdout: "desc": "", 2024-03-24T03:17:23.776 INFO:teuthology.orchestra.run.smithi118.stdout: "long_desc": "", 2024-03-24T03:17:23.776 INFO:teuthology.orchestra.run.smithi118.stdout: "tags": [], 2024-03-24T03:17:23.776 INFO:teuthology.orchestra.run.smithi118.stdout: "see_also": [] 2024-03-24T03:17:23.776 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-24T03:17:23.776 INFO:teuthology.orchestra.run.smithi118.stdout: "log_to_file": { 2024-03-24T03:17:23.776 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "log_to_file", 2024-03-24T03:17:23.776 INFO:teuthology.orchestra.run.smithi118.stdout: "type": "bool", 2024-03-24T03:17:23.776 INFO:teuthology.orchestra.run.smithi118.stdout: "level": "advanced", 2024-03-24T03:17:23.777 INFO:teuthology.orchestra.run.smithi118.stdout: "flags": 1, 2024-03-24T03:17:23.777 INFO:teuthology.orchestra.run.smithi118.stdout: "default_value": "False", 2024-03-24T03:17:23.777 INFO:teuthology.orchestra.run.smithi118.stdout: "min": "", 2024-03-24T03:17:23.777 INFO:teuthology.orchestra.run.smithi118.stdout: "max": "", 2024-03-24T03:17:23.777 INFO:teuthology.orchestra.run.smithi118.stdout: "enum_allowed": [], 2024-03-24T03:17:23.777 INFO:teuthology.orchestra.run.smithi118.stdout: "desc": "", 2024-03-24T03:17:23.777 INFO:teuthology.orchestra.run.smithi118.stdout: "long_desc": "", 2024-03-24T03:17:23.777 INFO:teuthology.orchestra.run.smithi118.stdout: "tags": [], 2024-03-24T03:17:23.777 INFO:teuthology.orchestra.run.smithi118.stdout: "see_also": [] 2024-03-24T03:17:23.777 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-03-24T03:17:23.778 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-03-24T03:17:23.778 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-24T03:17:23.778 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-24T03:17:23.778 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "status", 2024-03-24T03:17:23.778 INFO:teuthology.orchestra.run.smithi118.stdout: "can_run": true, 2024-03-24T03:17:23.778 INFO:teuthology.orchestra.run.smithi118.stdout: "error_string": "", 2024-03-24T03:17:23.778 INFO:teuthology.orchestra.run.smithi118.stdout: "module_options": { 2024-03-24T03:17:23.778 INFO:teuthology.orchestra.run.smithi118.stdout: "log_level": { 2024-03-24T03:17:23.779 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "log_level", 2024-03-24T03:17:23.779 INFO:teuthology.orchestra.run.smithi118.stdout: "type": "str", 2024-03-24T03:17:23.779 INFO:teuthology.orchestra.run.smithi118.stdout: "level": "advanced", 2024-03-24T03:17:23.779 INFO:teuthology.orchestra.run.smithi118.stdout: "flags": 1, 2024-03-24T03:17:23.779 INFO:teuthology.orchestra.run.smithi118.stdout: "default_value": "", 2024-03-24T03:17:23.779 INFO:teuthology.orchestra.run.smithi118.stdout: "min": "", 2024-03-24T03:17:23.779 INFO:teuthology.orchestra.run.smithi118.stdout: "max": "", 2024-03-24T03:17:23.779 INFO:teuthology.orchestra.run.smithi118.stdout: "enum_allowed": [ 2024-03-24T03:17:23.779 INFO:teuthology.orchestra.run.smithi118.stdout: "", 2024-03-24T03:17:23.779 INFO:teuthology.orchestra.run.smithi118.stdout: "critical", 2024-03-24T03:17:23.780 INFO:teuthology.orchestra.run.smithi118.stdout: "debug", 2024-03-24T03:17:23.780 INFO:teuthology.orchestra.run.smithi118.stdout: "error", 2024-03-24T03:17:23.780 INFO:teuthology.orchestra.run.smithi118.stdout: "info", 2024-03-24T03:17:23.780 INFO:teuthology.orchestra.run.smithi118.stdout: "warning" 2024-03-24T03:17:23.780 INFO:teuthology.orchestra.run.smithi118.stdout: ], 2024-03-24T03:17:23.780 INFO:teuthology.orchestra.run.smithi118.stdout: "desc": "", 2024-03-24T03:17:23.780 INFO:teuthology.orchestra.run.smithi118.stdout: "long_desc": "", 2024-03-24T03:17:23.780 INFO:teuthology.orchestra.run.smithi118.stdout: "tags": [], 2024-03-24T03:17:23.780 INFO:teuthology.orchestra.run.smithi118.stdout: "see_also": [] 2024-03-24T03:17:23.781 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-24T03:17:23.781 INFO:teuthology.orchestra.run.smithi118.stdout: "log_to_cluster": { 2024-03-24T03:17:23.781 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "log_to_cluster", 2024-03-24T03:17:23.781 INFO:teuthology.orchestra.run.smithi118.stdout: "type": "bool", 2024-03-24T03:17:23.781 INFO:teuthology.orchestra.run.smithi118.stdout: "level": "advanced", 2024-03-24T03:17:23.781 INFO:teuthology.orchestra.run.smithi118.stdout: "flags": 1, 2024-03-24T03:17:23.781 INFO:teuthology.orchestra.run.smithi118.stdout: "default_value": "False", 2024-03-24T03:17:23.781 INFO:teuthology.orchestra.run.smithi118.stdout: "min": "", 2024-03-24T03:17:23.781 INFO:teuthology.orchestra.run.smithi118.stdout: "max": "", 2024-03-24T03:17:23.781 INFO:teuthology.orchestra.run.smithi118.stdout: "enum_allowed": [], 2024-03-24T03:17:23.782 INFO:teuthology.orchestra.run.smithi118.stdout: "desc": "", 2024-03-24T03:17:23.782 INFO:teuthology.orchestra.run.smithi118.stdout: "long_desc": "", 2024-03-24T03:17:23.782 INFO:teuthology.orchestra.run.smithi118.stdout: "tags": [], 2024-03-24T03:17:23.782 INFO:teuthology.orchestra.run.smithi118.stdout: "see_also": [] 2024-03-24T03:17:23.782 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-24T03:17:23.782 INFO:teuthology.orchestra.run.smithi118.stdout: "log_to_cluster_level": { 2024-03-24T03:17:23.782 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "log_to_cluster_level", 2024-03-24T03:17:23.783 INFO:teuthology.orchestra.run.smithi118.stdout: "type": "str", 2024-03-24T03:17:23.783 INFO:teuthology.orchestra.run.smithi118.stdout: "level": "advanced", 2024-03-24T03:17:23.783 INFO:teuthology.orchestra.run.smithi118.stdout: "flags": 1, 2024-03-24T03:17:23.783 INFO:teuthology.orchestra.run.smithi118.stdout: "default_value": "info", 2024-03-24T03:17:23.783 INFO:teuthology.orchestra.run.smithi118.stdout: "min": "", 2024-03-24T03:17:23.783 INFO:teuthology.orchestra.run.smithi118.stdout: "max": "", 2024-03-24T03:17:23.785 INFO:teuthology.orchestra.run.smithi118.stdout: "enum_allowed": [ 2024-03-24T03:17:23.785 INFO:teuthology.orchestra.run.smithi118.stdout: "", 2024-03-24T03:17:23.785 INFO:teuthology.orchestra.run.smithi118.stdout: "critical", 2024-03-24T03:17:23.785 INFO:teuthology.orchestra.run.smithi118.stdout: "debug", 2024-03-24T03:17:23.786 INFO:teuthology.orchestra.run.smithi118.stdout: "error", 2024-03-24T03:17:23.786 INFO:teuthology.orchestra.run.smithi118.stdout: "info", 2024-03-24T03:17:23.786 INFO:teuthology.orchestra.run.smithi118.stdout: "warning" 2024-03-24T03:17:23.786 INFO:teuthology.orchestra.run.smithi118.stdout: ], 2024-03-24T03:17:23.786 INFO:teuthology.orchestra.run.smithi118.stdout: "desc": "", 2024-03-24T03:17:23.786 INFO:teuthology.orchestra.run.smithi118.stdout: "long_desc": "", 2024-03-24T03:17:23.786 INFO:teuthology.orchestra.run.smithi118.stdout: "tags": [], 2024-03-24T03:17:23.786 INFO:teuthology.orchestra.run.smithi118.stdout: "see_also": [] 2024-03-24T03:17:23.786 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-24T03:17:23.787 INFO:teuthology.orchestra.run.smithi118.stdout: "log_to_file": { 2024-03-24T03:17:23.787 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "log_to_file", 2024-03-24T03:17:23.787 INFO:teuthology.orchestra.run.smithi118.stdout: "type": "bool", 2024-03-24T03:17:23.787 INFO:teuthology.orchestra.run.smithi118.stdout: "level": "advanced", 2024-03-24T03:17:23.787 INFO:teuthology.orchestra.run.smithi118.stdout: "flags": 1, 2024-03-24T03:17:23.787 INFO:teuthology.orchestra.run.smithi118.stdout: "default_value": "False", 2024-03-24T03:17:23.787 INFO:teuthology.orchestra.run.smithi118.stdout: "min": "", 2024-03-24T03:17:23.787 INFO:teuthology.orchestra.run.smithi118.stdout: "max": "", 2024-03-24T03:17:23.787 INFO:teuthology.orchestra.run.smithi118.stdout: "enum_allowed": [], 2024-03-24T03:17:23.788 INFO:teuthology.orchestra.run.smithi118.stdout: "desc": "", 2024-03-24T03:17:23.788 INFO:teuthology.orchestra.run.smithi118.stdout: "long_desc": "", 2024-03-24T03:17:23.788 INFO:teuthology.orchestra.run.smithi118.stdout: "tags": [], 2024-03-24T03:17:23.788 INFO:teuthology.orchestra.run.smithi118.stdout: "see_also": [] 2024-03-24T03:17:23.788 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-03-24T03:17:23.788 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-03-24T03:17:23.788 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-24T03:17:23.788 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-24T03:17:23.788 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "telegraf", 2024-03-24T03:17:23.788 INFO:teuthology.orchestra.run.smithi118.stdout: "can_run": true, 2024-03-24T03:17:23.789 INFO:teuthology.orchestra.run.smithi118.stdout: "error_string": "", 2024-03-24T03:17:23.789 INFO:teuthology.orchestra.run.smithi118.stdout: "module_options": { 2024-03-24T03:17:23.789 INFO:teuthology.orchestra.run.smithi118.stdout: "address": { 2024-03-24T03:17:23.789 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "address", 2024-03-24T03:17:23.789 INFO:teuthology.orchestra.run.smithi118.stdout: "type": "str", 2024-03-24T03:17:23.789 INFO:teuthology.orchestra.run.smithi118.stdout: "level": "advanced", 2024-03-24T03:17:23.789 INFO:teuthology.orchestra.run.smithi118.stdout: "flags": 0, 2024-03-24T03:17:23.789 INFO:teuthology.orchestra.run.smithi118.stdout: "default_value": "unixgram:///tmp/telegraf.sock", 2024-03-24T03:17:23.789 INFO:teuthology.orchestra.run.smithi118.stdout: "min": "", 2024-03-24T03:17:23.789 INFO:teuthology.orchestra.run.smithi118.stdout: "max": "", 2024-03-24T03:17:23.790 INFO:teuthology.orchestra.run.smithi118.stdout: "enum_allowed": [], 2024-03-24T03:17:23.790 INFO:teuthology.orchestra.run.smithi118.stdout: "desc": "", 2024-03-24T03:17:23.790 INFO:teuthology.orchestra.run.smithi118.stdout: "long_desc": "", 2024-03-24T03:17:23.790 INFO:teuthology.orchestra.run.smithi118.stdout: "tags": [], 2024-03-24T03:17:23.790 INFO:teuthology.orchestra.run.smithi118.stdout: "see_also": [] 2024-03-24T03:17:23.790 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-24T03:17:23.790 INFO:teuthology.orchestra.run.smithi118.stdout: "interval": { 2024-03-24T03:17:23.790 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "interval", 2024-03-24T03:17:23.790 INFO:teuthology.orchestra.run.smithi118.stdout: "type": "secs", 2024-03-24T03:17:23.791 INFO:teuthology.orchestra.run.smithi118.stdout: "level": "advanced", 2024-03-24T03:17:23.791 INFO:teuthology.orchestra.run.smithi118.stdout: "flags": 0, 2024-03-24T03:17:23.791 INFO:teuthology.orchestra.run.smithi118.stdout: "default_value": "15", 2024-03-24T03:17:23.791 INFO:teuthology.orchestra.run.smithi118.stdout: "min": "", 2024-03-24T03:17:23.791 INFO:teuthology.orchestra.run.smithi118.stdout: "max": "", 2024-03-24T03:17:23.791 INFO:teuthology.orchestra.run.smithi118.stdout: "enum_allowed": [], 2024-03-24T03:17:23.791 INFO:teuthology.orchestra.run.smithi118.stdout: "desc": "", 2024-03-24T03:17:23.791 INFO:teuthology.orchestra.run.smithi118.stdout: "long_desc": "", 2024-03-24T03:17:23.791 INFO:teuthology.orchestra.run.smithi118.stdout: "tags": [], 2024-03-24T03:17:23.792 INFO:teuthology.orchestra.run.smithi118.stdout: "see_also": [] 2024-03-24T03:17:23.792 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-24T03:17:23.792 INFO:teuthology.orchestra.run.smithi118.stdout: "log_level": { 2024-03-24T03:17:23.792 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "log_level", 2024-03-24T03:17:23.792 INFO:teuthology.orchestra.run.smithi118.stdout: "type": "str", 2024-03-24T03:17:23.792 INFO:teuthology.orchestra.run.smithi118.stdout: "level": "advanced", 2024-03-24T03:17:23.792 INFO:teuthology.orchestra.run.smithi118.stdout: "flags": 1, 2024-03-24T03:17:23.792 INFO:teuthology.orchestra.run.smithi118.stdout: "default_value": "", 2024-03-24T03:17:23.792 INFO:teuthology.orchestra.run.smithi118.stdout: "min": "", 2024-03-24T03:17:23.793 INFO:teuthology.orchestra.run.smithi118.stdout: "max": "", 2024-03-24T03:17:23.793 INFO:teuthology.orchestra.run.smithi118.stdout: "enum_allowed": [ 2024-03-24T03:17:23.793 INFO:teuthology.orchestra.run.smithi118.stdout: "", 2024-03-24T03:17:23.793 INFO:teuthology.orchestra.run.smithi118.stdout: "critical", 2024-03-24T03:17:23.793 INFO:teuthology.orchestra.run.smithi118.stdout: "debug", 2024-03-24T03:17:23.793 INFO:teuthology.orchestra.run.smithi118.stdout: "error", 2024-03-24T03:17:23.793 INFO:teuthology.orchestra.run.smithi118.stdout: "info", 2024-03-24T03:17:23.793 INFO:teuthology.orchestra.run.smithi118.stdout: "warning" 2024-03-24T03:17:23.793 INFO:teuthology.orchestra.run.smithi118.stdout: ], 2024-03-24T03:17:23.794 INFO:teuthology.orchestra.run.smithi118.stdout: "desc": "", 2024-03-24T03:17:23.794 INFO:teuthology.orchestra.run.smithi118.stdout: "long_desc": "", 2024-03-24T03:17:23.794 INFO:teuthology.orchestra.run.smithi118.stdout: "tags": [], 2024-03-24T03:17:23.794 INFO:teuthology.orchestra.run.smithi118.stdout: "see_also": [] 2024-03-24T03:17:23.794 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-24T03:17:23.794 INFO:teuthology.orchestra.run.smithi118.stdout: "log_to_cluster": { 2024-03-24T03:17:23.794 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "log_to_cluster", 2024-03-24T03:17:23.794 INFO:teuthology.orchestra.run.smithi118.stdout: "type": "bool", 2024-03-24T03:17:23.795 INFO:teuthology.orchestra.run.smithi118.stdout: "level": "advanced", 2024-03-24T03:17:23.795 INFO:teuthology.orchestra.run.smithi118.stdout: "flags": 1, 2024-03-24T03:17:23.795 INFO:teuthology.orchestra.run.smithi118.stdout: "default_value": "False", 2024-03-24T03:17:23.795 INFO:teuthology.orchestra.run.smithi118.stdout: "min": "", 2024-03-24T03:17:23.795 INFO:teuthology.orchestra.run.smithi118.stdout: "max": "", 2024-03-24T03:17:23.795 INFO:teuthology.orchestra.run.smithi118.stdout: "enum_allowed": [], 2024-03-24T03:17:23.795 INFO:teuthology.orchestra.run.smithi118.stdout: "desc": "", 2024-03-24T03:17:23.795 INFO:teuthology.orchestra.run.smithi118.stdout: "long_desc": "", 2024-03-24T03:17:23.795 INFO:teuthology.orchestra.run.smithi118.stdout: "tags": [], 2024-03-24T03:17:23.795 INFO:teuthology.orchestra.run.smithi118.stdout: "see_also": [] 2024-03-24T03:17:23.796 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-24T03:17:23.796 INFO:teuthology.orchestra.run.smithi118.stdout: "log_to_cluster_level": { 2024-03-24T03:17:23.796 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "log_to_cluster_level", 2024-03-24T03:17:23.796 INFO:teuthology.orchestra.run.smithi118.stdout: "type": "str", 2024-03-24T03:17:23.796 INFO:teuthology.orchestra.run.smithi118.stdout: "level": "advanced", 2024-03-24T03:17:23.796 INFO:teuthology.orchestra.run.smithi118.stdout: "flags": 1, 2024-03-24T03:17:23.796 INFO:teuthology.orchestra.run.smithi118.stdout: "default_value": "info", 2024-03-24T03:17:23.796 INFO:teuthology.orchestra.run.smithi118.stdout: "min": "", 2024-03-24T03:17:23.796 INFO:teuthology.orchestra.run.smithi118.stdout: "max": "", 2024-03-24T03:17:23.797 INFO:teuthology.orchestra.run.smithi118.stdout: "enum_allowed": [ 2024-03-24T03:17:23.797 INFO:teuthology.orchestra.run.smithi118.stdout: "", 2024-03-24T03:17:23.797 INFO:teuthology.orchestra.run.smithi118.stdout: "critical", 2024-03-24T03:17:23.797 INFO:teuthology.orchestra.run.smithi118.stdout: "debug", 2024-03-24T03:17:23.797 INFO:teuthology.orchestra.run.smithi118.stdout: "error", 2024-03-24T03:17:23.797 INFO:teuthology.orchestra.run.smithi118.stdout: "info", 2024-03-24T03:17:23.797 INFO:teuthology.orchestra.run.smithi118.stdout: "warning" 2024-03-24T03:17:23.797 INFO:teuthology.orchestra.run.smithi118.stdout: ], 2024-03-24T03:17:23.797 INFO:teuthology.orchestra.run.smithi118.stdout: "desc": "", 2024-03-24T03:17:23.798 INFO:teuthology.orchestra.run.smithi118.stdout: "long_desc": "", 2024-03-24T03:17:23.798 INFO:teuthology.orchestra.run.smithi118.stdout: "tags": [], 2024-03-24T03:17:23.798 INFO:teuthology.orchestra.run.smithi118.stdout: "see_also": [] 2024-03-24T03:17:23.798 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-24T03:17:23.798 INFO:teuthology.orchestra.run.smithi118.stdout: "log_to_file": { 2024-03-24T03:17:23.798 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "log_to_file", 2024-03-24T03:17:23.798 INFO:teuthology.orchestra.run.smithi118.stdout: "type": "bool", 2024-03-24T03:17:23.798 INFO:teuthology.orchestra.run.smithi118.stdout: "level": "advanced", 2024-03-24T03:17:23.798 INFO:teuthology.orchestra.run.smithi118.stdout: "flags": 1, 2024-03-24T03:17:23.798 INFO:teuthology.orchestra.run.smithi118.stdout: "default_value": "False", 2024-03-24T03:17:23.799 INFO:teuthology.orchestra.run.smithi118.stdout: "min": "", 2024-03-24T03:17:23.799 INFO:teuthology.orchestra.run.smithi118.stdout: "max": "", 2024-03-24T03:17:23.799 INFO:teuthology.orchestra.run.smithi118.stdout: "enum_allowed": [], 2024-03-24T03:17:23.799 INFO:teuthology.orchestra.run.smithi118.stdout: "desc": "", 2024-03-24T03:17:23.799 INFO:teuthology.orchestra.run.smithi118.stdout: "long_desc": "", 2024-03-24T03:17:23.799 INFO:teuthology.orchestra.run.smithi118.stdout: "tags": [], 2024-03-24T03:17:23.799 INFO:teuthology.orchestra.run.smithi118.stdout: "see_also": [] 2024-03-24T03:17:23.799 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-03-24T03:17:23.799 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-03-24T03:17:23.800 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-24T03:17:23.800 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-24T03:17:23.800 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "telemetry", 2024-03-24T03:17:23.800 INFO:teuthology.orchestra.run.smithi118.stdout: "can_run": true, 2024-03-24T03:17:23.800 INFO:teuthology.orchestra.run.smithi118.stdout: "error_string": "", 2024-03-24T03:17:23.800 INFO:teuthology.orchestra.run.smithi118.stdout: "module_options": { 2024-03-24T03:17:23.800 INFO:teuthology.orchestra.run.smithi118.stdout: "channel_basic": { 2024-03-24T03:17:23.800 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "channel_basic", 2024-03-24T03:17:23.800 INFO:teuthology.orchestra.run.smithi118.stdout: "type": "bool", 2024-03-24T03:17:23.801 INFO:teuthology.orchestra.run.smithi118.stdout: "level": "advanced", 2024-03-24T03:17:23.801 INFO:teuthology.orchestra.run.smithi118.stdout: "flags": 0, 2024-03-24T03:17:23.801 INFO:teuthology.orchestra.run.smithi118.stdout: "default_value": "True", 2024-03-24T03:17:23.801 INFO:teuthology.orchestra.run.smithi118.stdout: "min": "", 2024-03-24T03:17:23.801 INFO:teuthology.orchestra.run.smithi118.stdout: "max": "", 2024-03-24T03:17:23.801 INFO:teuthology.orchestra.run.smithi118.stdout: "enum_allowed": [], 2024-03-24T03:17:23.801 INFO:teuthology.orchestra.run.smithi118.stdout: "desc": "Share basic cluster information (size, version)", 2024-03-24T03:17:23.801 INFO:teuthology.orchestra.run.smithi118.stdout: "long_desc": "", 2024-03-24T03:17:23.801 INFO:teuthology.orchestra.run.smithi118.stdout: "tags": [], 2024-03-24T03:17:23.802 INFO:teuthology.orchestra.run.smithi118.stdout: "see_also": [] 2024-03-24T03:17:23.802 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-24T03:17:23.802 INFO:teuthology.orchestra.run.smithi118.stdout: "channel_crash": { 2024-03-24T03:17:23.802 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "channel_crash", 2024-03-24T03:17:23.802 INFO:teuthology.orchestra.run.smithi118.stdout: "type": "bool", 2024-03-24T03:17:23.802 INFO:teuthology.orchestra.run.smithi118.stdout: "level": "advanced", 2024-03-24T03:17:23.802 INFO:teuthology.orchestra.run.smithi118.stdout: "flags": 0, 2024-03-24T03:17:23.802 INFO:teuthology.orchestra.run.smithi118.stdout: "default_value": "True", 2024-03-24T03:17:23.803 INFO:teuthology.orchestra.run.smithi118.stdout: "min": "", 2024-03-24T03:17:23.803 INFO:teuthology.orchestra.run.smithi118.stdout: "max": "", 2024-03-24T03:17:23.803 INFO:teuthology.orchestra.run.smithi118.stdout: "enum_allowed": [], 2024-03-24T03:17:23.803 INFO:teuthology.orchestra.run.smithi118.stdout: "desc": "Share metadata about Ceph daemon crashes (version, stack straces, etc)", 2024-03-24T03:17:23.803 INFO:teuthology.orchestra.run.smithi118.stdout: "long_desc": "", 2024-03-24T03:17:23.803 INFO:teuthology.orchestra.run.smithi118.stdout: "tags": [], 2024-03-24T03:17:23.803 INFO:teuthology.orchestra.run.smithi118.stdout: "see_also": [] 2024-03-24T03:17:23.803 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-24T03:17:23.803 INFO:teuthology.orchestra.run.smithi118.stdout: "channel_device": { 2024-03-24T03:17:23.804 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "channel_device", 2024-03-24T03:17:23.804 INFO:teuthology.orchestra.run.smithi118.stdout: "type": "bool", 2024-03-24T03:17:23.804 INFO:teuthology.orchestra.run.smithi118.stdout: "level": "advanced", 2024-03-24T03:17:23.804 INFO:teuthology.orchestra.run.smithi118.stdout: "flags": 0, 2024-03-24T03:17:23.804 INFO:teuthology.orchestra.run.smithi118.stdout: "default_value": "True", 2024-03-24T03:17:23.804 INFO:teuthology.orchestra.run.smithi118.stdout: "min": "", 2024-03-24T03:17:23.804 INFO:teuthology.orchestra.run.smithi118.stdout: "max": "", 2024-03-24T03:17:23.804 INFO:teuthology.orchestra.run.smithi118.stdout: "enum_allowed": [], 2024-03-24T03:17:23.804 INFO:teuthology.orchestra.run.smithi118.stdout: "desc": "Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)", 2024-03-24T03:17:23.805 INFO:teuthology.orchestra.run.smithi118.stdout: "long_desc": "", 2024-03-24T03:17:23.805 INFO:teuthology.orchestra.run.smithi118.stdout: "tags": [], 2024-03-24T03:17:23.805 INFO:teuthology.orchestra.run.smithi118.stdout: "see_also": [] 2024-03-24T03:17:23.805 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-24T03:17:23.805 INFO:teuthology.orchestra.run.smithi118.stdout: "channel_ident": { 2024-03-24T03:17:23.805 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "channel_ident", 2024-03-24T03:17:23.805 INFO:teuthology.orchestra.run.smithi118.stdout: "type": "bool", 2024-03-24T03:17:23.806 INFO:teuthology.orchestra.run.smithi118.stdout: "level": "advanced", 2024-03-24T03:17:23.806 INFO:teuthology.orchestra.run.smithi118.stdout: "flags": 0, 2024-03-24T03:17:23.806 INFO:teuthology.orchestra.run.smithi118.stdout: "default_value": "False", 2024-03-24T03:17:23.806 INFO:teuthology.orchestra.run.smithi118.stdout: "min": "", 2024-03-24T03:17:23.806 INFO:teuthology.orchestra.run.smithi118.stdout: "max": "", 2024-03-24T03:17:23.806 INFO:teuthology.orchestra.run.smithi118.stdout: "enum_allowed": [], 2024-03-24T03:17:23.806 INFO:teuthology.orchestra.run.smithi118.stdout: "desc": "Share a user-provided description and/or contact email for the cluster", 2024-03-24T03:17:23.806 INFO:teuthology.orchestra.run.smithi118.stdout: "long_desc": "", 2024-03-24T03:17:23.807 INFO:teuthology.orchestra.run.smithi118.stdout: "tags": [], 2024-03-24T03:17:23.807 INFO:teuthology.orchestra.run.smithi118.stdout: "see_also": [] 2024-03-24T03:17:23.807 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-24T03:17:23.807 INFO:teuthology.orchestra.run.smithi118.stdout: "channel_perf": { 2024-03-24T03:17:23.807 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "channel_perf", 2024-03-24T03:17:23.807 INFO:teuthology.orchestra.run.smithi118.stdout: "type": "bool", 2024-03-24T03:17:23.807 INFO:teuthology.orchestra.run.smithi118.stdout: "level": "advanced", 2024-03-24T03:17:23.807 INFO:teuthology.orchestra.run.smithi118.stdout: "flags": 0, 2024-03-24T03:17:23.807 INFO:teuthology.orchestra.run.smithi118.stdout: "default_value": "False", 2024-03-24T03:17:23.808 INFO:teuthology.orchestra.run.smithi118.stdout: "min": "", 2024-03-24T03:17:23.808 INFO:teuthology.orchestra.run.smithi118.stdout: "max": "", 2024-03-24T03:17:23.808 INFO:teuthology.orchestra.run.smithi118.stdout: "enum_allowed": [], 2024-03-24T03:17:23.808 INFO:teuthology.orchestra.run.smithi118.stdout: "desc": "Share various performance metrics of a cluster", 2024-03-24T03:17:23.808 INFO:teuthology.orchestra.run.smithi118.stdout: "long_desc": "", 2024-03-24T03:17:23.808 INFO:teuthology.orchestra.run.smithi118.stdout: "tags": [], 2024-03-24T03:17:23.808 INFO:teuthology.orchestra.run.smithi118.stdout: "see_also": [] 2024-03-24T03:17:23.808 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-24T03:17:23.808 INFO:teuthology.orchestra.run.smithi118.stdout: "contact": { 2024-03-24T03:17:23.809 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "contact", 2024-03-24T03:17:23.809 INFO:teuthology.orchestra.run.smithi118.stdout: "type": "str", 2024-03-24T03:17:23.809 INFO:teuthology.orchestra.run.smithi118.stdout: "level": "advanced", 2024-03-24T03:17:23.809 INFO:teuthology.orchestra.run.smithi118.stdout: "flags": 0, 2024-03-24T03:17:23.809 INFO:teuthology.orchestra.run.smithi118.stdout: "default_value": "", 2024-03-24T03:17:23.809 INFO:teuthology.orchestra.run.smithi118.stdout: "min": "", 2024-03-24T03:17:23.809 INFO:teuthology.orchestra.run.smithi118.stdout: "max": "", 2024-03-24T03:17:23.809 INFO:teuthology.orchestra.run.smithi118.stdout: "enum_allowed": [], 2024-03-24T03:17:23.809 INFO:teuthology.orchestra.run.smithi118.stdout: "desc": "", 2024-03-24T03:17:23.809 INFO:teuthology.orchestra.run.smithi118.stdout: "long_desc": "", 2024-03-24T03:17:23.810 INFO:teuthology.orchestra.run.smithi118.stdout: "tags": [], 2024-03-24T03:17:23.810 INFO:teuthology.orchestra.run.smithi118.stdout: "see_also": [] 2024-03-24T03:17:23.810 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-24T03:17:23.810 INFO:teuthology.orchestra.run.smithi118.stdout: "description": { 2024-03-24T03:17:23.810 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "description", 2024-03-24T03:17:23.810 INFO:teuthology.orchestra.run.smithi118.stdout: "type": "str", 2024-03-24T03:17:23.810 INFO:teuthology.orchestra.run.smithi118.stdout: "level": "advanced", 2024-03-24T03:17:23.810 INFO:teuthology.orchestra.run.smithi118.stdout: "flags": 0, 2024-03-24T03:17:23.811 INFO:teuthology.orchestra.run.smithi118.stdout: "default_value": "", 2024-03-24T03:17:23.811 INFO:teuthology.orchestra.run.smithi118.stdout: "min": "", 2024-03-24T03:17:23.811 INFO:teuthology.orchestra.run.smithi118.stdout: "max": "", 2024-03-24T03:17:23.811 INFO:teuthology.orchestra.run.smithi118.stdout: "enum_allowed": [], 2024-03-24T03:17:23.811 INFO:teuthology.orchestra.run.smithi118.stdout: "desc": "", 2024-03-24T03:17:23.811 INFO:teuthology.orchestra.run.smithi118.stdout: "long_desc": "", 2024-03-24T03:17:23.811 INFO:teuthology.orchestra.run.smithi118.stdout: "tags": [], 2024-03-24T03:17:23.811 INFO:teuthology.orchestra.run.smithi118.stdout: "see_also": [] 2024-03-24T03:17:23.811 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-24T03:17:23.812 INFO:teuthology.orchestra.run.smithi118.stdout: "device_url": { 2024-03-24T03:17:23.812 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "device_url", 2024-03-24T03:17:23.812 INFO:teuthology.orchestra.run.smithi118.stdout: "type": "str", 2024-03-24T03:17:23.812 INFO:teuthology.orchestra.run.smithi118.stdout: "level": "advanced", 2024-03-24T03:17:23.812 INFO:teuthology.orchestra.run.smithi118.stdout: "flags": 0, 2024-03-24T03:17:23.812 INFO:teuthology.orchestra.run.smithi118.stdout: "default_value": "https://telemetry.ceph.com/device", 2024-03-24T03:17:23.812 INFO:teuthology.orchestra.run.smithi118.stdout: "min": "", 2024-03-24T03:17:23.812 INFO:teuthology.orchestra.run.smithi118.stdout: "max": "", 2024-03-24T03:17:23.812 INFO:teuthology.orchestra.run.smithi118.stdout: "enum_allowed": [], 2024-03-24T03:17:23.813 INFO:teuthology.orchestra.run.smithi118.stdout: "desc": "", 2024-03-24T03:17:23.813 INFO:teuthology.orchestra.run.smithi118.stdout: "long_desc": "", 2024-03-24T03:17:23.813 INFO:teuthology.orchestra.run.smithi118.stdout: "tags": [], 2024-03-24T03:17:23.813 INFO:teuthology.orchestra.run.smithi118.stdout: "see_also": [] 2024-03-24T03:17:23.813 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-24T03:17:23.813 INFO:teuthology.orchestra.run.smithi118.stdout: "enabled": { 2024-03-24T03:17:23.813 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "enabled", 2024-03-24T03:17:23.813 INFO:teuthology.orchestra.run.smithi118.stdout: "type": "bool", 2024-03-24T03:17:23.813 INFO:teuthology.orchestra.run.smithi118.stdout: "level": "advanced", 2024-03-24T03:17:23.814 INFO:teuthology.orchestra.run.smithi118.stdout: "flags": 0, 2024-03-24T03:17:23.814 INFO:teuthology.orchestra.run.smithi118.stdout: "default_value": "False", 2024-03-24T03:17:23.814 INFO:teuthology.orchestra.run.smithi118.stdout: "min": "", 2024-03-24T03:17:23.814 INFO:teuthology.orchestra.run.smithi118.stdout: "max": "", 2024-03-24T03:17:23.814 INFO:teuthology.orchestra.run.smithi118.stdout: "enum_allowed": [], 2024-03-24T03:17:23.814 INFO:teuthology.orchestra.run.smithi118.stdout: "desc": "", 2024-03-24T03:17:23.814 INFO:teuthology.orchestra.run.smithi118.stdout: "long_desc": "", 2024-03-24T03:17:23.814 INFO:teuthology.orchestra.run.smithi118.stdout: "tags": [], 2024-03-24T03:17:23.815 INFO:teuthology.orchestra.run.smithi118.stdout: "see_also": [] 2024-03-24T03:17:23.815 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-24T03:17:23.815 INFO:teuthology.orchestra.run.smithi118.stdout: "interval": { 2024-03-24T03:17:23.815 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "interval", 2024-03-24T03:17:23.815 INFO:teuthology.orchestra.run.smithi118.stdout: "type": "int", 2024-03-24T03:17:23.815 INFO:teuthology.orchestra.run.smithi118.stdout: "level": "advanced", 2024-03-24T03:17:23.815 INFO:teuthology.orchestra.run.smithi118.stdout: "flags": 0, 2024-03-24T03:17:23.815 INFO:teuthology.orchestra.run.smithi118.stdout: "default_value": "24", 2024-03-24T03:17:23.816 INFO:teuthology.orchestra.run.smithi118.stdout: "min": "8", 2024-03-24T03:17:23.816 INFO:teuthology.orchestra.run.smithi118.stdout: "max": "", 2024-03-24T03:17:23.816 INFO:teuthology.orchestra.run.smithi118.stdout: "enum_allowed": [], 2024-03-24T03:17:23.816 INFO:teuthology.orchestra.run.smithi118.stdout: "desc": "", 2024-03-24T03:17:23.816 INFO:teuthology.orchestra.run.smithi118.stdout: "long_desc": "", 2024-03-24T03:17:23.816 INFO:teuthology.orchestra.run.smithi118.stdout: "tags": [], 2024-03-24T03:17:23.816 INFO:teuthology.orchestra.run.smithi118.stdout: "see_also": [] 2024-03-24T03:17:23.816 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-24T03:17:23.816 INFO:teuthology.orchestra.run.smithi118.stdout: "last_opt_revision": { 2024-03-24T03:17:23.817 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "last_opt_revision", 2024-03-24T03:17:23.817 INFO:teuthology.orchestra.run.smithi118.stdout: "type": "int", 2024-03-24T03:17:23.817 INFO:teuthology.orchestra.run.smithi118.stdout: "level": "advanced", 2024-03-24T03:17:23.817 INFO:teuthology.orchestra.run.smithi118.stdout: "flags": 0, 2024-03-24T03:17:23.817 INFO:teuthology.orchestra.run.smithi118.stdout: "default_value": "1", 2024-03-24T03:17:23.817 INFO:teuthology.orchestra.run.smithi118.stdout: "min": "", 2024-03-24T03:17:23.817 INFO:teuthology.orchestra.run.smithi118.stdout: "max": "", 2024-03-24T03:17:23.817 INFO:teuthology.orchestra.run.smithi118.stdout: "enum_allowed": [], 2024-03-24T03:17:23.817 INFO:teuthology.orchestra.run.smithi118.stdout: "desc": "", 2024-03-24T03:17:23.818 INFO:teuthology.orchestra.run.smithi118.stdout: "long_desc": "", 2024-03-24T03:17:23.818 INFO:teuthology.orchestra.run.smithi118.stdout: "tags": [], 2024-03-24T03:17:23.818 INFO:teuthology.orchestra.run.smithi118.stdout: "see_also": [] 2024-03-24T03:17:23.818 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-24T03:17:23.818 INFO:teuthology.orchestra.run.smithi118.stdout: "leaderboard": { 2024-03-24T03:17:23.818 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "leaderboard", 2024-03-24T03:17:23.818 INFO:teuthology.orchestra.run.smithi118.stdout: "type": "bool", 2024-03-24T03:17:23.818 INFO:teuthology.orchestra.run.smithi118.stdout: "level": "advanced", 2024-03-24T03:17:23.818 INFO:teuthology.orchestra.run.smithi118.stdout: "flags": 0, 2024-03-24T03:17:23.819 INFO:teuthology.orchestra.run.smithi118.stdout: "default_value": "False", 2024-03-24T03:17:23.819 INFO:teuthology.orchestra.run.smithi118.stdout: "min": "", 2024-03-24T03:17:23.819 INFO:teuthology.orchestra.run.smithi118.stdout: "max": "", 2024-03-24T03:17:23.819 INFO:teuthology.orchestra.run.smithi118.stdout: "enum_allowed": [], 2024-03-24T03:17:23.819 INFO:teuthology.orchestra.run.smithi118.stdout: "desc": "", 2024-03-24T03:17:23.819 INFO:teuthology.orchestra.run.smithi118.stdout: "long_desc": "", 2024-03-24T03:17:23.819 INFO:teuthology.orchestra.run.smithi118.stdout: "tags": [], 2024-03-24T03:17:23.819 INFO:teuthology.orchestra.run.smithi118.stdout: "see_also": [] 2024-03-24T03:17:23.819 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-24T03:17:23.820 INFO:teuthology.orchestra.run.smithi118.stdout: "leaderboard_description": { 2024-03-24T03:17:23.820 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "leaderboard_description", 2024-03-24T03:17:23.820 INFO:teuthology.orchestra.run.smithi118.stdout: "type": "str", 2024-03-24T03:17:23.820 INFO:teuthology.orchestra.run.smithi118.stdout: "level": "advanced", 2024-03-24T03:17:23.820 INFO:teuthology.orchestra.run.smithi118.stdout: "flags": 0, 2024-03-24T03:17:23.820 INFO:teuthology.orchestra.run.smithi118.stdout: "default_value": "", 2024-03-24T03:17:23.820 INFO:teuthology.orchestra.run.smithi118.stdout: "min": "", 2024-03-24T03:17:23.820 INFO:teuthology.orchestra.run.smithi118.stdout: "max": "", 2024-03-24T03:17:23.820 INFO:teuthology.orchestra.run.smithi118.stdout: "enum_allowed": [], 2024-03-24T03:17:23.821 INFO:teuthology.orchestra.run.smithi118.stdout: "desc": "", 2024-03-24T03:17:23.821 INFO:teuthology.orchestra.run.smithi118.stdout: "long_desc": "", 2024-03-24T03:17:23.821 INFO:teuthology.orchestra.run.smithi118.stdout: "tags": [], 2024-03-24T03:17:23.821 INFO:teuthology.orchestra.run.smithi118.stdout: "see_also": [] 2024-03-24T03:17:23.821 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-24T03:17:23.821 INFO:teuthology.orchestra.run.smithi118.stdout: "log_level": { 2024-03-24T03:17:23.821 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "log_level", 2024-03-24T03:17:23.821 INFO:teuthology.orchestra.run.smithi118.stdout: "type": "str", 2024-03-24T03:17:23.821 INFO:teuthology.orchestra.run.smithi118.stdout: "level": "advanced", 2024-03-24T03:17:23.821 INFO:teuthology.orchestra.run.smithi118.stdout: "flags": 1, 2024-03-24T03:17:23.822 INFO:teuthology.orchestra.run.smithi118.stdout: "default_value": "", 2024-03-24T03:17:23.822 INFO:teuthology.orchestra.run.smithi118.stdout: "min": "", 2024-03-24T03:17:23.822 INFO:teuthology.orchestra.run.smithi118.stdout: "max": "", 2024-03-24T03:17:23.822 INFO:teuthology.orchestra.run.smithi118.stdout: "enum_allowed": [ 2024-03-24T03:17:23.822 INFO:teuthology.orchestra.run.smithi118.stdout: "", 2024-03-24T03:17:23.822 INFO:teuthology.orchestra.run.smithi118.stdout: "critical", 2024-03-24T03:17:23.822 INFO:teuthology.orchestra.run.smithi118.stdout: "debug", 2024-03-24T03:17:23.822 INFO:teuthology.orchestra.run.smithi118.stdout: "error", 2024-03-24T03:17:23.823 INFO:teuthology.orchestra.run.smithi118.stdout: "info", 2024-03-24T03:17:23.823 INFO:teuthology.orchestra.run.smithi118.stdout: "warning" 2024-03-24T03:17:23.823 INFO:teuthology.orchestra.run.smithi118.stdout: ], 2024-03-24T03:17:23.823 INFO:teuthology.orchestra.run.smithi118.stdout: "desc": "", 2024-03-24T03:17:23.823 INFO:teuthology.orchestra.run.smithi118.stdout: "long_desc": "", 2024-03-24T03:17:23.823 INFO:teuthology.orchestra.run.smithi118.stdout: "tags": [], 2024-03-24T03:17:23.823 INFO:teuthology.orchestra.run.smithi118.stdout: "see_also": [] 2024-03-24T03:17:23.823 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-24T03:17:23.823 INFO:teuthology.orchestra.run.smithi118.stdout: "log_to_cluster": { 2024-03-24T03:17:23.824 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "log_to_cluster", 2024-03-24T03:17:23.824 INFO:teuthology.orchestra.run.smithi118.stdout: "type": "bool", 2024-03-24T03:17:23.824 INFO:teuthology.orchestra.run.smithi118.stdout: "level": "advanced", 2024-03-24T03:17:23.824 INFO:teuthology.orchestra.run.smithi118.stdout: "flags": 1, 2024-03-24T03:17:23.824 INFO:teuthology.orchestra.run.smithi118.stdout: "default_value": "False", 2024-03-24T03:17:23.824 INFO:teuthology.orchestra.run.smithi118.stdout: "min": "", 2024-03-24T03:17:23.824 INFO:teuthology.orchestra.run.smithi118.stdout: "max": "", 2024-03-24T03:17:23.824 INFO:teuthology.orchestra.run.smithi118.stdout: "enum_allowed": [], 2024-03-24T03:17:23.824 INFO:teuthology.orchestra.run.smithi118.stdout: "desc": "", 2024-03-24T03:17:23.825 INFO:teuthology.orchestra.run.smithi118.stdout: "long_desc": "", 2024-03-24T03:17:23.825 INFO:teuthology.orchestra.run.smithi118.stdout: "tags": [], 2024-03-24T03:17:23.825 INFO:teuthology.orchestra.run.smithi118.stdout: "see_also": [] 2024-03-24T03:17:23.825 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-24T03:17:23.825 INFO:teuthology.orchestra.run.smithi118.stdout: "log_to_cluster_level": { 2024-03-24T03:17:23.825 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "log_to_cluster_level", 2024-03-24T03:17:23.825 INFO:teuthology.orchestra.run.smithi118.stdout: "type": "str", 2024-03-24T03:17:23.825 INFO:teuthology.orchestra.run.smithi118.stdout: "level": "advanced", 2024-03-24T03:17:23.825 INFO:teuthology.orchestra.run.smithi118.stdout: "flags": 1, 2024-03-24T03:17:23.826 INFO:teuthology.orchestra.run.smithi118.stdout: "default_value": "info", 2024-03-24T03:17:23.826 INFO:teuthology.orchestra.run.smithi118.stdout: "min": "", 2024-03-24T03:17:23.826 INFO:teuthology.orchestra.run.smithi118.stdout: "max": "", 2024-03-24T03:17:23.826 INFO:teuthology.orchestra.run.smithi118.stdout: "enum_allowed": [ 2024-03-24T03:17:23.826 INFO:teuthology.orchestra.run.smithi118.stdout: "", 2024-03-24T03:17:23.826 INFO:teuthology.orchestra.run.smithi118.stdout: "critical", 2024-03-24T03:17:23.826 INFO:teuthology.orchestra.run.smithi118.stdout: "debug", 2024-03-24T03:17:23.826 INFO:teuthology.orchestra.run.smithi118.stdout: "error", 2024-03-24T03:17:23.826 INFO:teuthology.orchestra.run.smithi118.stdout: "info", 2024-03-24T03:17:23.827 INFO:teuthology.orchestra.run.smithi118.stdout: "warning" 2024-03-24T03:17:23.827 INFO:teuthology.orchestra.run.smithi118.stdout: ], 2024-03-24T03:17:23.827 INFO:teuthology.orchestra.run.smithi118.stdout: "desc": "", 2024-03-24T03:17:23.827 INFO:teuthology.orchestra.run.smithi118.stdout: "long_desc": "", 2024-03-24T03:17:23.827 INFO:teuthology.orchestra.run.smithi118.stdout: "tags": [], 2024-03-24T03:17:23.827 INFO:teuthology.orchestra.run.smithi118.stdout: "see_also": [] 2024-03-24T03:17:23.827 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-24T03:17:23.827 INFO:teuthology.orchestra.run.smithi118.stdout: "log_to_file": { 2024-03-24T03:17:23.827 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "log_to_file", 2024-03-24T03:17:23.827 INFO:teuthology.orchestra.run.smithi118.stdout: "type": "bool", 2024-03-24T03:17:23.828 INFO:teuthology.orchestra.run.smithi118.stdout: "level": "advanced", 2024-03-24T03:17:23.828 INFO:teuthology.orchestra.run.smithi118.stdout: "flags": 1, 2024-03-24T03:17:23.828 INFO:teuthology.orchestra.run.smithi118.stdout: "default_value": "False", 2024-03-24T03:17:23.828 INFO:teuthology.orchestra.run.smithi118.stdout: "min": "", 2024-03-24T03:17:23.828 INFO:teuthology.orchestra.run.smithi118.stdout: "max": "", 2024-03-24T03:17:23.828 INFO:teuthology.orchestra.run.smithi118.stdout: "enum_allowed": [], 2024-03-24T03:17:23.828 INFO:teuthology.orchestra.run.smithi118.stdout: "desc": "", 2024-03-24T03:17:23.828 INFO:teuthology.orchestra.run.smithi118.stdout: "long_desc": "", 2024-03-24T03:17:23.828 INFO:teuthology.orchestra.run.smithi118.stdout: "tags": [], 2024-03-24T03:17:23.829 INFO:teuthology.orchestra.run.smithi118.stdout: "see_also": [] 2024-03-24T03:17:23.829 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-24T03:17:23.829 INFO:teuthology.orchestra.run.smithi118.stdout: "organization": { 2024-03-24T03:17:23.829 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "organization", 2024-03-24T03:17:23.829 INFO:teuthology.orchestra.run.smithi118.stdout: "type": "str", 2024-03-24T03:17:23.829 INFO:teuthology.orchestra.run.smithi118.stdout: "level": "advanced", 2024-03-24T03:17:23.829 INFO:teuthology.orchestra.run.smithi118.stdout: "flags": 0, 2024-03-24T03:17:23.829 INFO:teuthology.orchestra.run.smithi118.stdout: "default_value": "", 2024-03-24T03:17:23.829 INFO:teuthology.orchestra.run.smithi118.stdout: "min": "", 2024-03-24T03:17:23.829 INFO:teuthology.orchestra.run.smithi118.stdout: "max": "", 2024-03-24T03:17:23.830 INFO:teuthology.orchestra.run.smithi118.stdout: "enum_allowed": [], 2024-03-24T03:17:23.830 INFO:teuthology.orchestra.run.smithi118.stdout: "desc": "", 2024-03-24T03:17:23.830 INFO:teuthology.orchestra.run.smithi118.stdout: "long_desc": "", 2024-03-24T03:17:23.830 INFO:teuthology.orchestra.run.smithi118.stdout: "tags": [], 2024-03-24T03:17:23.830 INFO:teuthology.orchestra.run.smithi118.stdout: "see_also": [] 2024-03-24T03:17:23.830 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-24T03:17:23.830 INFO:teuthology.orchestra.run.smithi118.stdout: "proxy": { 2024-03-24T03:17:23.830 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "proxy", 2024-03-24T03:17:23.830 INFO:teuthology.orchestra.run.smithi118.stdout: "type": "str", 2024-03-24T03:17:23.831 INFO:teuthology.orchestra.run.smithi118.stdout: "level": "advanced", 2024-03-24T03:17:23.831 INFO:teuthology.orchestra.run.smithi118.stdout: "flags": 0, 2024-03-24T03:17:23.831 INFO:teuthology.orchestra.run.smithi118.stdout: "default_value": "", 2024-03-24T03:17:23.831 INFO:teuthology.orchestra.run.smithi118.stdout: "min": "", 2024-03-24T03:17:23.831 INFO:teuthology.orchestra.run.smithi118.stdout: "max": "", 2024-03-24T03:17:23.831 INFO:teuthology.orchestra.run.smithi118.stdout: "enum_allowed": [], 2024-03-24T03:17:23.831 INFO:teuthology.orchestra.run.smithi118.stdout: "desc": "", 2024-03-24T03:17:23.831 INFO:teuthology.orchestra.run.smithi118.stdout: "long_desc": "", 2024-03-24T03:17:23.831 INFO:teuthology.orchestra.run.smithi118.stdout: "tags": [], 2024-03-24T03:17:23.832 INFO:teuthology.orchestra.run.smithi118.stdout: "see_also": [] 2024-03-24T03:17:23.832 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-24T03:17:23.832 INFO:teuthology.orchestra.run.smithi118.stdout: "url": { 2024-03-24T03:17:23.832 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "url", 2024-03-24T03:17:23.832 INFO:teuthology.orchestra.run.smithi118.stdout: "type": "str", 2024-03-24T03:17:23.832 INFO:teuthology.orchestra.run.smithi118.stdout: "level": "advanced", 2024-03-24T03:17:23.832 INFO:teuthology.orchestra.run.smithi118.stdout: "flags": 0, 2024-03-24T03:17:23.832 INFO:teuthology.orchestra.run.smithi118.stdout: "default_value": "https://telemetry.ceph.com/report", 2024-03-24T03:17:23.832 INFO:teuthology.orchestra.run.smithi118.stdout: "min": "", 2024-03-24T03:17:23.833 INFO:teuthology.orchestra.run.smithi118.stdout: "max": "", 2024-03-24T03:17:23.833 INFO:teuthology.orchestra.run.smithi118.stdout: "enum_allowed": [], 2024-03-24T03:17:23.833 INFO:teuthology.orchestra.run.smithi118.stdout: "desc": "", 2024-03-24T03:17:23.833 INFO:teuthology.orchestra.run.smithi118.stdout: "long_desc": "", 2024-03-24T03:17:23.833 INFO:teuthology.orchestra.run.smithi118.stdout: "tags": [], 2024-03-24T03:17:23.833 INFO:teuthology.orchestra.run.smithi118.stdout: "see_also": [] 2024-03-24T03:17:23.833 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-03-24T03:17:23.833 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-03-24T03:17:23.833 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-24T03:17:23.834 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-24T03:17:23.834 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "test_orchestrator", 2024-03-24T03:17:23.834 INFO:teuthology.orchestra.run.smithi118.stdout: "can_run": true, 2024-03-24T03:17:23.834 INFO:teuthology.orchestra.run.smithi118.stdout: "error_string": "", 2024-03-24T03:17:23.834 INFO:teuthology.orchestra.run.smithi118.stdout: "module_options": { 2024-03-24T03:17:23.834 INFO:teuthology.orchestra.run.smithi118.stdout: "log_level": { 2024-03-24T03:17:23.834 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "log_level", 2024-03-24T03:17:23.834 INFO:teuthology.orchestra.run.smithi118.stdout: "type": "str", 2024-03-24T03:17:23.834 INFO:teuthology.orchestra.run.smithi118.stdout: "level": "advanced", 2024-03-24T03:17:23.835 INFO:teuthology.orchestra.run.smithi118.stdout: "flags": 1, 2024-03-24T03:17:23.835 INFO:teuthology.orchestra.run.smithi118.stdout: "default_value": "", 2024-03-24T03:17:23.835 INFO:teuthology.orchestra.run.smithi118.stdout: "min": "", 2024-03-24T03:17:23.835 INFO:teuthology.orchestra.run.smithi118.stdout: "max": "", 2024-03-24T03:17:23.835 INFO:teuthology.orchestra.run.smithi118.stdout: "enum_allowed": [ 2024-03-24T03:17:23.835 INFO:teuthology.orchestra.run.smithi118.stdout: "", 2024-03-24T03:17:23.835 INFO:teuthology.orchestra.run.smithi118.stdout: "critical", 2024-03-24T03:17:23.835 INFO:teuthology.orchestra.run.smithi118.stdout: "debug", 2024-03-24T03:17:23.835 INFO:teuthology.orchestra.run.smithi118.stdout: "error", 2024-03-24T03:17:23.836 INFO:teuthology.orchestra.run.smithi118.stdout: "info", 2024-03-24T03:17:23.836 INFO:teuthology.orchestra.run.smithi118.stdout: "warning" 2024-03-24T03:17:23.836 INFO:teuthology.orchestra.run.smithi118.stdout: ], 2024-03-24T03:17:23.836 INFO:teuthology.orchestra.run.smithi118.stdout: "desc": "", 2024-03-24T03:17:23.836 INFO:teuthology.orchestra.run.smithi118.stdout: "long_desc": "", 2024-03-24T03:17:23.836 INFO:teuthology.orchestra.run.smithi118.stdout: "tags": [], 2024-03-24T03:17:23.836 INFO:teuthology.orchestra.run.smithi118.stdout: "see_also": [] 2024-03-24T03:17:23.836 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-24T03:17:23.836 INFO:teuthology.orchestra.run.smithi118.stdout: "log_to_cluster": { 2024-03-24T03:17:23.836 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "log_to_cluster", 2024-03-24T03:17:23.837 INFO:teuthology.orchestra.run.smithi118.stdout: "type": "bool", 2024-03-24T03:17:23.837 INFO:teuthology.orchestra.run.smithi118.stdout: "level": "advanced", 2024-03-24T03:17:23.837 INFO:teuthology.orchestra.run.smithi118.stdout: "flags": 1, 2024-03-24T03:17:23.837 INFO:teuthology.orchestra.run.smithi118.stdout: "default_value": "False", 2024-03-24T03:17:23.837 INFO:teuthology.orchestra.run.smithi118.stdout: "min": "", 2024-03-24T03:17:23.837 INFO:teuthology.orchestra.run.smithi118.stdout: "max": "", 2024-03-24T03:17:23.837 INFO:teuthology.orchestra.run.smithi118.stdout: "enum_allowed": [], 2024-03-24T03:17:23.837 INFO:teuthology.orchestra.run.smithi118.stdout: "desc": "", 2024-03-24T03:17:23.837 INFO:teuthology.orchestra.run.smithi118.stdout: "long_desc": "", 2024-03-24T03:17:23.838 INFO:teuthology.orchestra.run.smithi118.stdout: "tags": [], 2024-03-24T03:17:23.838 INFO:teuthology.orchestra.run.smithi118.stdout: "see_also": [] 2024-03-24T03:17:23.838 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-24T03:17:23.838 INFO:teuthology.orchestra.run.smithi118.stdout: "log_to_cluster_level": { 2024-03-24T03:17:23.838 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "log_to_cluster_level", 2024-03-24T03:17:23.838 INFO:teuthology.orchestra.run.smithi118.stdout: "type": "str", 2024-03-24T03:17:23.838 INFO:teuthology.orchestra.run.smithi118.stdout: "level": "advanced", 2024-03-24T03:17:23.838 INFO:teuthology.orchestra.run.smithi118.stdout: "flags": 1, 2024-03-24T03:17:23.839 INFO:teuthology.orchestra.run.smithi118.stdout: "default_value": "info", 2024-03-24T03:17:23.839 INFO:teuthology.orchestra.run.smithi118.stdout: "min": "", 2024-03-24T03:17:23.839 INFO:teuthology.orchestra.run.smithi118.stdout: "max": "", 2024-03-24T03:17:23.839 INFO:teuthology.orchestra.run.smithi118.stdout: "enum_allowed": [ 2024-03-24T03:17:23.839 INFO:teuthology.orchestra.run.smithi118.stdout: "", 2024-03-24T03:17:23.839 INFO:teuthology.orchestra.run.smithi118.stdout: "critical", 2024-03-24T03:17:23.839 INFO:teuthology.orchestra.run.smithi118.stdout: "debug", 2024-03-24T03:17:23.839 INFO:teuthology.orchestra.run.smithi118.stdout: "error", 2024-03-24T03:17:23.839 INFO:teuthology.orchestra.run.smithi118.stdout: "info", 2024-03-24T03:17:23.839 INFO:teuthology.orchestra.run.smithi118.stdout: "warning" 2024-03-24T03:17:23.840 INFO:teuthology.orchestra.run.smithi118.stdout: ], 2024-03-24T03:17:23.840 INFO:teuthology.orchestra.run.smithi118.stdout: "desc": "", 2024-03-24T03:17:23.840 INFO:teuthology.orchestra.run.smithi118.stdout: "long_desc": "", 2024-03-24T03:17:23.840 INFO:teuthology.orchestra.run.smithi118.stdout: "tags": [], 2024-03-24T03:17:23.840 INFO:teuthology.orchestra.run.smithi118.stdout: "see_also": [] 2024-03-24T03:17:23.840 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-24T03:17:23.840 INFO:teuthology.orchestra.run.smithi118.stdout: "log_to_file": { 2024-03-24T03:17:23.840 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "log_to_file", 2024-03-24T03:17:23.841 INFO:teuthology.orchestra.run.smithi118.stdout: "type": "bool", 2024-03-24T03:17:23.841 INFO:teuthology.orchestra.run.smithi118.stdout: "level": "advanced", 2024-03-24T03:17:23.841 INFO:teuthology.orchestra.run.smithi118.stdout: "flags": 1, 2024-03-24T03:17:23.841 INFO:teuthology.orchestra.run.smithi118.stdout: "default_value": "False", 2024-03-24T03:17:23.841 INFO:teuthology.orchestra.run.smithi118.stdout: "min": "", 2024-03-24T03:17:23.841 INFO:teuthology.orchestra.run.smithi118.stdout: "max": "", 2024-03-24T03:17:23.841 INFO:teuthology.orchestra.run.smithi118.stdout: "enum_allowed": [], 2024-03-24T03:17:23.841 INFO:teuthology.orchestra.run.smithi118.stdout: "desc": "", 2024-03-24T03:17:23.841 INFO:teuthology.orchestra.run.smithi118.stdout: "long_desc": "", 2024-03-24T03:17:23.842 INFO:teuthology.orchestra.run.smithi118.stdout: "tags": [], 2024-03-24T03:17:23.842 INFO:teuthology.orchestra.run.smithi118.stdout: "see_also": [] 2024-03-24T03:17:23.842 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-03-24T03:17:23.842 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-03-24T03:17:23.842 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-24T03:17:23.842 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-24T03:17:23.842 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "volumes", 2024-03-24T03:17:23.842 INFO:teuthology.orchestra.run.smithi118.stdout: "can_run": true, 2024-03-24T03:17:23.843 INFO:teuthology.orchestra.run.smithi118.stdout: "error_string": "", 2024-03-24T03:17:23.843 INFO:teuthology.orchestra.run.smithi118.stdout: "module_options": { 2024-03-24T03:17:23.843 INFO:teuthology.orchestra.run.smithi118.stdout: "log_level": { 2024-03-24T03:17:23.843 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "log_level", 2024-03-24T03:17:23.843 INFO:teuthology.orchestra.run.smithi118.stdout: "type": "str", 2024-03-24T03:17:23.843 INFO:teuthology.orchestra.run.smithi118.stdout: "level": "advanced", 2024-03-24T03:17:23.843 INFO:teuthology.orchestra.run.smithi118.stdout: "flags": 1, 2024-03-24T03:17:23.843 INFO:teuthology.orchestra.run.smithi118.stdout: "default_value": "", 2024-03-24T03:17:23.843 INFO:teuthology.orchestra.run.smithi118.stdout: "min": "", 2024-03-24T03:17:23.844 INFO:teuthology.orchestra.run.smithi118.stdout: "max": "", 2024-03-24T03:17:23.844 INFO:teuthology.orchestra.run.smithi118.stdout: "enum_allowed": [ 2024-03-24T03:17:23.844 INFO:teuthology.orchestra.run.smithi118.stdout: "", 2024-03-24T03:17:23.844 INFO:teuthology.orchestra.run.smithi118.stdout: "critical", 2024-03-24T03:17:23.844 INFO:teuthology.orchestra.run.smithi118.stdout: "debug", 2024-03-24T03:17:23.844 INFO:teuthology.orchestra.run.smithi118.stdout: "error", 2024-03-24T03:17:23.844 INFO:teuthology.orchestra.run.smithi118.stdout: "info", 2024-03-24T03:17:23.844 INFO:teuthology.orchestra.run.smithi118.stdout: "warning" 2024-03-24T03:17:23.844 INFO:teuthology.orchestra.run.smithi118.stdout: ], 2024-03-24T03:17:23.845 INFO:teuthology.orchestra.run.smithi118.stdout: "desc": "", 2024-03-24T03:17:23.845 INFO:teuthology.orchestra.run.smithi118.stdout: "long_desc": "", 2024-03-24T03:17:23.845 INFO:teuthology.orchestra.run.smithi118.stdout: "tags": [], 2024-03-24T03:17:23.845 INFO:teuthology.orchestra.run.smithi118.stdout: "see_also": [] 2024-03-24T03:17:23.845 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-24T03:17:23.845 INFO:teuthology.orchestra.run.smithi118.stdout: "log_to_cluster": { 2024-03-24T03:17:23.845 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "log_to_cluster", 2024-03-24T03:17:23.845 INFO:teuthology.orchestra.run.smithi118.stdout: "type": "bool", 2024-03-24T03:17:23.845 INFO:teuthology.orchestra.run.smithi118.stdout: "level": "advanced", 2024-03-24T03:17:23.846 INFO:teuthology.orchestra.run.smithi118.stdout: "flags": 1, 2024-03-24T03:17:23.846 INFO:teuthology.orchestra.run.smithi118.stdout: "default_value": "False", 2024-03-24T03:17:23.846 INFO:teuthology.orchestra.run.smithi118.stdout: "min": "", 2024-03-24T03:17:23.846 INFO:teuthology.orchestra.run.smithi118.stdout: "max": "", 2024-03-24T03:17:23.846 INFO:teuthology.orchestra.run.smithi118.stdout: "enum_allowed": [], 2024-03-24T03:17:23.846 INFO:teuthology.orchestra.run.smithi118.stdout: "desc": "", 2024-03-24T03:17:23.846 INFO:teuthology.orchestra.run.smithi118.stdout: "long_desc": "", 2024-03-24T03:17:23.846 INFO:teuthology.orchestra.run.smithi118.stdout: "tags": [], 2024-03-24T03:17:23.847 INFO:teuthology.orchestra.run.smithi118.stdout: "see_also": [] 2024-03-24T03:17:23.847 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-24T03:17:23.847 INFO:teuthology.orchestra.run.smithi118.stdout: "log_to_cluster_level": { 2024-03-24T03:17:23.847 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "log_to_cluster_level", 2024-03-24T03:17:23.847 INFO:teuthology.orchestra.run.smithi118.stdout: "type": "str", 2024-03-24T03:17:23.847 INFO:teuthology.orchestra.run.smithi118.stdout: "level": "advanced", 2024-03-24T03:17:23.847 INFO:teuthology.orchestra.run.smithi118.stdout: "flags": 1, 2024-03-24T03:17:23.847 INFO:teuthology.orchestra.run.smithi118.stdout: "default_value": "info", 2024-03-24T03:17:23.847 INFO:teuthology.orchestra.run.smithi118.stdout: "min": "", 2024-03-24T03:17:23.848 INFO:teuthology.orchestra.run.smithi118.stdout: "max": "", 2024-03-24T03:17:23.848 INFO:teuthology.orchestra.run.smithi118.stdout: "enum_allowed": [ 2024-03-24T03:17:23.848 INFO:teuthology.orchestra.run.smithi118.stdout: "", 2024-03-24T03:17:23.848 INFO:teuthology.orchestra.run.smithi118.stdout: "critical", 2024-03-24T03:17:23.848 INFO:teuthology.orchestra.run.smithi118.stdout: "debug", 2024-03-24T03:17:23.848 INFO:teuthology.orchestra.run.smithi118.stdout: "error", 2024-03-24T03:17:23.848 INFO:teuthology.orchestra.run.smithi118.stdout: "info", 2024-03-24T03:17:23.848 INFO:teuthology.orchestra.run.smithi118.stdout: "warning" 2024-03-24T03:17:23.848 INFO:teuthology.orchestra.run.smithi118.stdout: ], 2024-03-24T03:17:23.849 INFO:teuthology.orchestra.run.smithi118.stdout: "desc": "", 2024-03-24T03:17:23.849 INFO:teuthology.orchestra.run.smithi118.stdout: "long_desc": "", 2024-03-24T03:17:23.849 INFO:teuthology.orchestra.run.smithi118.stdout: "tags": [], 2024-03-24T03:17:23.849 INFO:teuthology.orchestra.run.smithi118.stdout: "see_also": [] 2024-03-24T03:17:23.849 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-24T03:17:23.849 INFO:teuthology.orchestra.run.smithi118.stdout: "log_to_file": { 2024-03-24T03:17:23.849 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "log_to_file", 2024-03-24T03:17:23.849 INFO:teuthology.orchestra.run.smithi118.stdout: "type": "bool", 2024-03-24T03:17:23.850 INFO:teuthology.orchestra.run.smithi118.stdout: "level": "advanced", 2024-03-24T03:17:23.850 INFO:teuthology.orchestra.run.smithi118.stdout: "flags": 1, 2024-03-24T03:17:23.850 INFO:teuthology.orchestra.run.smithi118.stdout: "default_value": "False", 2024-03-24T03:17:23.850 INFO:teuthology.orchestra.run.smithi118.stdout: "min": "", 2024-03-24T03:17:23.850 INFO:teuthology.orchestra.run.smithi118.stdout: "max": "", 2024-03-24T03:17:23.850 INFO:teuthology.orchestra.run.smithi118.stdout: "enum_allowed": [], 2024-03-24T03:17:23.850 INFO:teuthology.orchestra.run.smithi118.stdout: "desc": "", 2024-03-24T03:17:23.850 INFO:teuthology.orchestra.run.smithi118.stdout: "long_desc": "", 2024-03-24T03:17:23.850 INFO:teuthology.orchestra.run.smithi118.stdout: "tags": [], 2024-03-24T03:17:23.851 INFO:teuthology.orchestra.run.smithi118.stdout: "see_also": [] 2024-03-24T03:17:23.851 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-24T03:17:23.851 INFO:teuthology.orchestra.run.smithi118.stdout: "max_concurrent_clones": { 2024-03-24T03:17:23.851 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "max_concurrent_clones", 2024-03-24T03:17:23.851 INFO:teuthology.orchestra.run.smithi118.stdout: "type": "int", 2024-03-24T03:17:23.851 INFO:teuthology.orchestra.run.smithi118.stdout: "level": "advanced", 2024-03-24T03:17:23.851 INFO:teuthology.orchestra.run.smithi118.stdout: "flags": 0, 2024-03-24T03:17:23.851 INFO:teuthology.orchestra.run.smithi118.stdout: "default_value": "4", 2024-03-24T03:17:23.851 INFO:teuthology.orchestra.run.smithi118.stdout: "min": "", 2024-03-24T03:17:23.852 INFO:teuthology.orchestra.run.smithi118.stdout: "max": "", 2024-03-24T03:17:23.852 INFO:teuthology.orchestra.run.smithi118.stdout: "enum_allowed": [], 2024-03-24T03:17:23.852 INFO:teuthology.orchestra.run.smithi118.stdout: "desc": "Number of asynchronous cloner threads", 2024-03-24T03:17:23.852 INFO:teuthology.orchestra.run.smithi118.stdout: "long_desc": "", 2024-03-24T03:17:23.852 INFO:teuthology.orchestra.run.smithi118.stdout: "tags": [], 2024-03-24T03:17:23.852 INFO:teuthology.orchestra.run.smithi118.stdout: "see_also": [] 2024-03-24T03:17:23.852 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-24T03:17:23.852 INFO:teuthology.orchestra.run.smithi118.stdout: "periodic_async_work": { 2024-03-24T03:17:23.853 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "periodic_async_work", 2024-03-24T03:17:23.853 INFO:teuthology.orchestra.run.smithi118.stdout: "type": "bool", 2024-03-24T03:17:23.853 INFO:teuthology.orchestra.run.smithi118.stdout: "level": "advanced", 2024-03-24T03:17:23.853 INFO:teuthology.orchestra.run.smithi118.stdout: "flags": 0, 2024-03-24T03:17:23.853 INFO:teuthology.orchestra.run.smithi118.stdout: "default_value": "False", 2024-03-24T03:17:23.853 INFO:teuthology.orchestra.run.smithi118.stdout: "min": "", 2024-03-24T03:17:23.853 INFO:teuthology.orchestra.run.smithi118.stdout: "max": "", 2024-03-24T03:17:23.853 INFO:teuthology.orchestra.run.smithi118.stdout: "enum_allowed": [], 2024-03-24T03:17:23.854 INFO:teuthology.orchestra.run.smithi118.stdout: "desc": "Periodically check for async work", 2024-03-24T03:17:23.854 INFO:teuthology.orchestra.run.smithi118.stdout: "long_desc": "", 2024-03-24T03:17:23.854 INFO:teuthology.orchestra.run.smithi118.stdout: "tags": [], 2024-03-24T03:17:23.854 INFO:teuthology.orchestra.run.smithi118.stdout: "see_also": [] 2024-03-24T03:17:23.854 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-24T03:17:23.854 INFO:teuthology.orchestra.run.smithi118.stdout: "snapshot_clone_delay": { 2024-03-24T03:17:23.854 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "snapshot_clone_delay", 2024-03-24T03:17:23.854 INFO:teuthology.orchestra.run.smithi118.stdout: "type": "int", 2024-03-24T03:17:23.855 INFO:teuthology.orchestra.run.smithi118.stdout: "level": "advanced", 2024-03-24T03:17:23.855 INFO:teuthology.orchestra.run.smithi118.stdout: "flags": 0, 2024-03-24T03:17:23.855 INFO:teuthology.orchestra.run.smithi118.stdout: "default_value": "0", 2024-03-24T03:17:23.855 INFO:teuthology.orchestra.run.smithi118.stdout: "min": "", 2024-03-24T03:17:23.855 INFO:teuthology.orchestra.run.smithi118.stdout: "max": "", 2024-03-24T03:17:23.855 INFO:teuthology.orchestra.run.smithi118.stdout: "enum_allowed": [], 2024-03-24T03:17:23.855 INFO:teuthology.orchestra.run.smithi118.stdout: "desc": "Delay clone begin operation by snapshot_clone_delay seconds", 2024-03-24T03:17:23.856 INFO:teuthology.orchestra.run.smithi118.stdout: "long_desc": "", 2024-03-24T03:17:23.856 INFO:teuthology.orchestra.run.smithi118.stdout: "tags": [], 2024-03-24T03:17:23.856 INFO:teuthology.orchestra.run.smithi118.stdout: "see_also": [] 2024-03-24T03:17:23.856 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-24T03:17:23.856 INFO:teuthology.orchestra.run.smithi118.stdout: "snapshot_clone_no_wait": { 2024-03-24T03:17:23.856 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "snapshot_clone_no_wait", 2024-03-24T03:17:23.856 INFO:teuthology.orchestra.run.smithi118.stdout: "type": "bool", 2024-03-24T03:17:23.856 INFO:teuthology.orchestra.run.smithi118.stdout: "level": "advanced", 2024-03-24T03:17:23.856 INFO:teuthology.orchestra.run.smithi118.stdout: "flags": 0, 2024-03-24T03:17:23.857 INFO:teuthology.orchestra.run.smithi118.stdout: "default_value": "True", 2024-03-24T03:17:23.857 INFO:teuthology.orchestra.run.smithi118.stdout: "min": "", 2024-03-24T03:17:23.857 INFO:teuthology.orchestra.run.smithi118.stdout: "max": "", 2024-03-24T03:17:23.857 INFO:teuthology.orchestra.run.smithi118.stdout: "enum_allowed": [], 2024-03-24T03:17:23.857 INFO:teuthology.orchestra.run.smithi118.stdout: "desc": "Reject subvolume clone request when cloner threads are busy", 2024-03-24T03:17:23.857 INFO:teuthology.orchestra.run.smithi118.stdout: "long_desc": "", 2024-03-24T03:17:23.857 INFO:teuthology.orchestra.run.smithi118.stdout: "tags": [], 2024-03-24T03:17:23.857 INFO:teuthology.orchestra.run.smithi118.stdout: "see_also": [] 2024-03-24T03:17:23.858 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-03-24T03:17:23.858 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-03-24T03:17:23.858 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-24T03:17:23.858 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-24T03:17:23.858 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "zabbix", 2024-03-24T03:17:23.858 INFO:teuthology.orchestra.run.smithi118.stdout: "can_run": true, 2024-03-24T03:17:23.858 INFO:teuthology.orchestra.run.smithi118.stdout: "error_string": "", 2024-03-24T03:17:23.859 INFO:teuthology.orchestra.run.smithi118.stdout: "module_options": { 2024-03-24T03:17:23.859 INFO:teuthology.orchestra.run.smithi118.stdout: "discovery_interval": { 2024-03-24T03:17:23.859 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "discovery_interval", 2024-03-24T03:17:23.859 INFO:teuthology.orchestra.run.smithi118.stdout: "type": "uint", 2024-03-24T03:17:23.859 INFO:teuthology.orchestra.run.smithi118.stdout: "level": "advanced", 2024-03-24T03:17:23.859 INFO:teuthology.orchestra.run.smithi118.stdout: "flags": 0, 2024-03-24T03:17:23.859 INFO:teuthology.orchestra.run.smithi118.stdout: "default_value": "100", 2024-03-24T03:17:23.859 INFO:teuthology.orchestra.run.smithi118.stdout: "min": "", 2024-03-24T03:17:23.860 INFO:teuthology.orchestra.run.smithi118.stdout: "max": "", 2024-03-24T03:17:23.860 INFO:teuthology.orchestra.run.smithi118.stdout: "enum_allowed": [], 2024-03-24T03:17:23.860 INFO:teuthology.orchestra.run.smithi118.stdout: "desc": "", 2024-03-24T03:17:23.860 INFO:teuthology.orchestra.run.smithi118.stdout: "long_desc": "", 2024-03-24T03:17:23.860 INFO:teuthology.orchestra.run.smithi118.stdout: "tags": [], 2024-03-24T03:17:23.860 INFO:teuthology.orchestra.run.smithi118.stdout: "see_also": [] 2024-03-24T03:17:23.861 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-24T03:17:23.861 INFO:teuthology.orchestra.run.smithi118.stdout: "identifier": { 2024-03-24T03:17:23.861 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "identifier", 2024-03-24T03:17:23.861 INFO:teuthology.orchestra.run.smithi118.stdout: "type": "str", 2024-03-24T03:17:23.861 INFO:teuthology.orchestra.run.smithi118.stdout: "level": "advanced", 2024-03-24T03:17:23.862 INFO:teuthology.orchestra.run.smithi118.stdout: "flags": 0, 2024-03-24T03:17:23.862 INFO:teuthology.orchestra.run.smithi118.stdout: "default_value": "", 2024-03-24T03:17:23.862 INFO:teuthology.orchestra.run.smithi118.stdout: "min": "", 2024-03-24T03:17:23.862 INFO:teuthology.orchestra.run.smithi118.stdout: "max": "", 2024-03-24T03:17:23.863 INFO:teuthology.orchestra.run.smithi118.stdout: "enum_allowed": [], 2024-03-24T03:17:23.863 INFO:teuthology.orchestra.run.smithi118.stdout: "desc": "", 2024-03-24T03:17:23.863 INFO:teuthology.orchestra.run.smithi118.stdout: "long_desc": "", 2024-03-24T03:17:23.863 INFO:teuthology.orchestra.run.smithi118.stdout: "tags": [], 2024-03-24T03:17:23.864 INFO:teuthology.orchestra.run.smithi118.stdout: "see_also": [] 2024-03-24T03:17:23.864 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-24T03:17:23.864 INFO:teuthology.orchestra.run.smithi118.stdout: "interval": { 2024-03-24T03:17:23.864 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "interval", 2024-03-24T03:17:23.865 INFO:teuthology.orchestra.run.smithi118.stdout: "type": "secs", 2024-03-24T03:17:23.865 INFO:teuthology.orchestra.run.smithi118.stdout: "level": "advanced", 2024-03-24T03:17:23.865 INFO:teuthology.orchestra.run.smithi118.stdout: "flags": 0, 2024-03-24T03:17:23.865 INFO:teuthology.orchestra.run.smithi118.stdout: "default_value": "60", 2024-03-24T03:17:23.865 INFO:teuthology.orchestra.run.smithi118.stdout: "min": "", 2024-03-24T03:17:23.866 INFO:teuthology.orchestra.run.smithi118.stdout: "max": "", 2024-03-24T03:17:23.866 INFO:teuthology.orchestra.run.smithi118.stdout: "enum_allowed": [], 2024-03-24T03:17:23.866 INFO:teuthology.orchestra.run.smithi118.stdout: "desc": "", 2024-03-24T03:17:23.866 INFO:teuthology.orchestra.run.smithi118.stdout: "long_desc": "", 2024-03-24T03:17:23.867 INFO:teuthology.orchestra.run.smithi118.stdout: "tags": [], 2024-03-24T03:17:23.867 INFO:teuthology.orchestra.run.smithi118.stdout: "see_also": [] 2024-03-24T03:17:23.867 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-24T03:17:23.867 INFO:teuthology.orchestra.run.smithi118.stdout: "log_level": { 2024-03-24T03:17:23.867 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "log_level", 2024-03-24T03:17:23.868 INFO:teuthology.orchestra.run.smithi118.stdout: "type": "str", 2024-03-24T03:17:23.868 INFO:teuthology.orchestra.run.smithi118.stdout: "level": "advanced", 2024-03-24T03:17:23.868 INFO:teuthology.orchestra.run.smithi118.stdout: "flags": 1, 2024-03-24T03:17:23.868 INFO:teuthology.orchestra.run.smithi118.stdout: "default_value": "", 2024-03-24T03:17:23.869 INFO:teuthology.orchestra.run.smithi118.stdout: "min": "", 2024-03-24T03:17:23.869 INFO:teuthology.orchestra.run.smithi118.stdout: "max": "", 2024-03-24T03:17:23.869 INFO:teuthology.orchestra.run.smithi118.stdout: "enum_allowed": [ 2024-03-24T03:17:23.869 INFO:teuthology.orchestra.run.smithi118.stdout: "", 2024-03-24T03:17:23.870 INFO:teuthology.orchestra.run.smithi118.stdout: "critical", 2024-03-24T03:17:23.870 INFO:teuthology.orchestra.run.smithi118.stdout: "debug", 2024-03-24T03:17:23.870 INFO:teuthology.orchestra.run.smithi118.stdout: "error", 2024-03-24T03:17:23.870 INFO:teuthology.orchestra.run.smithi118.stdout: "info", 2024-03-24T03:17:23.870 INFO:teuthology.orchestra.run.smithi118.stdout: "warning" 2024-03-24T03:17:23.870 INFO:teuthology.orchestra.run.smithi118.stdout: ], 2024-03-24T03:17:23.871 INFO:teuthology.orchestra.run.smithi118.stdout: "desc": "", 2024-03-24T03:17:23.871 INFO:teuthology.orchestra.run.smithi118.stdout: "long_desc": "", 2024-03-24T03:17:23.871 INFO:teuthology.orchestra.run.smithi118.stdout: "tags": [], 2024-03-24T03:17:23.871 INFO:teuthology.orchestra.run.smithi118.stdout: "see_also": [] 2024-03-24T03:17:23.871 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-24T03:17:23.871 INFO:teuthology.orchestra.run.smithi118.stdout: "log_to_cluster": { 2024-03-24T03:17:23.871 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "log_to_cluster", 2024-03-24T03:17:23.871 INFO:teuthology.orchestra.run.smithi118.stdout: "type": "bool", 2024-03-24T03:17:23.871 INFO:teuthology.orchestra.run.smithi118.stdout: "level": "advanced", 2024-03-24T03:17:23.872 INFO:teuthology.orchestra.run.smithi118.stdout: "flags": 1, 2024-03-24T03:17:23.872 INFO:teuthology.orchestra.run.smithi118.stdout: "default_value": "False", 2024-03-24T03:17:23.872 INFO:teuthology.orchestra.run.smithi118.stdout: "min": "", 2024-03-24T03:17:23.872 INFO:teuthology.orchestra.run.smithi118.stdout: "max": "", 2024-03-24T03:17:23.872 INFO:teuthology.orchestra.run.smithi118.stdout: "enum_allowed": [], 2024-03-24T03:17:23.872 INFO:teuthology.orchestra.run.smithi118.stdout: "desc": "", 2024-03-24T03:17:23.872 INFO:teuthology.orchestra.run.smithi118.stdout: "long_desc": "", 2024-03-24T03:17:23.872 INFO:teuthology.orchestra.run.smithi118.stdout: "tags": [], 2024-03-24T03:17:23.873 INFO:teuthology.orchestra.run.smithi118.stdout: "see_also": [] 2024-03-24T03:17:23.873 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-24T03:17:23.873 INFO:teuthology.orchestra.run.smithi118.stdout: "log_to_cluster_level": { 2024-03-24T03:17:23.873 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "log_to_cluster_level", 2024-03-24T03:17:23.873 INFO:teuthology.orchestra.run.smithi118.stdout: "type": "str", 2024-03-24T03:17:23.873 INFO:teuthology.orchestra.run.smithi118.stdout: "level": "advanced", 2024-03-24T03:17:23.873 INFO:teuthology.orchestra.run.smithi118.stdout: "flags": 1, 2024-03-24T03:17:23.873 INFO:teuthology.orchestra.run.smithi118.stdout: "default_value": "info", 2024-03-24T03:17:23.873 INFO:teuthology.orchestra.run.smithi118.stdout: "min": "", 2024-03-24T03:17:23.874 INFO:teuthology.orchestra.run.smithi118.stdout: "max": "", 2024-03-24T03:17:23.874 INFO:teuthology.orchestra.run.smithi118.stdout: "enum_allowed": [ 2024-03-24T03:17:23.874 INFO:teuthology.orchestra.run.smithi118.stdout: "", 2024-03-24T03:17:23.874 INFO:teuthology.orchestra.run.smithi118.stdout: "critical", 2024-03-24T03:17:23.874 INFO:teuthology.orchestra.run.smithi118.stdout: "debug", 2024-03-24T03:17:23.874 INFO:teuthology.orchestra.run.smithi118.stdout: "error", 2024-03-24T03:17:23.874 INFO:teuthology.orchestra.run.smithi118.stdout: "info", 2024-03-24T03:17:23.874 INFO:teuthology.orchestra.run.smithi118.stdout: "warning" 2024-03-24T03:17:23.875 INFO:teuthology.orchestra.run.smithi118.stdout: ], 2024-03-24T03:17:23.875 INFO:teuthology.orchestra.run.smithi118.stdout: "desc": "", 2024-03-24T03:17:23.875 INFO:teuthology.orchestra.run.smithi118.stdout: "long_desc": "", 2024-03-24T03:17:23.875 INFO:teuthology.orchestra.run.smithi118.stdout: "tags": [], 2024-03-24T03:17:23.875 INFO:teuthology.orchestra.run.smithi118.stdout: "see_also": [] 2024-03-24T03:17:23.875 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-24T03:17:23.875 INFO:teuthology.orchestra.run.smithi118.stdout: "log_to_file": { 2024-03-24T03:17:23.875 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "log_to_file", 2024-03-24T03:17:23.875 INFO:teuthology.orchestra.run.smithi118.stdout: "type": "bool", 2024-03-24T03:17:23.876 INFO:teuthology.orchestra.run.smithi118.stdout: "level": "advanced", 2024-03-24T03:17:23.876 INFO:teuthology.orchestra.run.smithi118.stdout: "flags": 1, 2024-03-24T03:17:23.876 INFO:teuthology.orchestra.run.smithi118.stdout: "default_value": "False", 2024-03-24T03:17:23.876 INFO:teuthology.orchestra.run.smithi118.stdout: "min": "", 2024-03-24T03:17:23.876 INFO:teuthology.orchestra.run.smithi118.stdout: "max": "", 2024-03-24T03:17:23.876 INFO:teuthology.orchestra.run.smithi118.stdout: "enum_allowed": [], 2024-03-24T03:17:23.876 INFO:teuthology.orchestra.run.smithi118.stdout: "desc": "", 2024-03-24T03:17:23.876 INFO:teuthology.orchestra.run.smithi118.stdout: "long_desc": "", 2024-03-24T03:17:23.876 INFO:teuthology.orchestra.run.smithi118.stdout: "tags": [], 2024-03-24T03:17:23.877 INFO:teuthology.orchestra.run.smithi118.stdout: "see_also": [] 2024-03-24T03:17:23.877 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-24T03:17:23.877 INFO:teuthology.orchestra.run.smithi118.stdout: "zabbix_host": { 2024-03-24T03:17:23.877 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "zabbix_host", 2024-03-24T03:17:23.877 INFO:teuthology.orchestra.run.smithi118.stdout: "type": "str", 2024-03-24T03:17:23.877 INFO:teuthology.orchestra.run.smithi118.stdout: "level": "advanced", 2024-03-24T03:17:23.877 INFO:teuthology.orchestra.run.smithi118.stdout: "flags": 0, 2024-03-24T03:17:23.877 INFO:teuthology.orchestra.run.smithi118.stdout: "default_value": "", 2024-03-24T03:17:23.877 INFO:teuthology.orchestra.run.smithi118.stdout: "min": "", 2024-03-24T03:17:23.878 INFO:teuthology.orchestra.run.smithi118.stdout: "max": "", 2024-03-24T03:17:23.878 INFO:teuthology.orchestra.run.smithi118.stdout: "enum_allowed": [], 2024-03-24T03:17:23.878 INFO:teuthology.orchestra.run.smithi118.stdout: "desc": "", 2024-03-24T03:17:23.878 INFO:teuthology.orchestra.run.smithi118.stdout: "long_desc": "", 2024-03-24T03:17:23.878 INFO:teuthology.orchestra.run.smithi118.stdout: "tags": [], 2024-03-24T03:17:23.878 INFO:teuthology.orchestra.run.smithi118.stdout: "see_also": [] 2024-03-24T03:17:23.878 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-24T03:17:23.878 INFO:teuthology.orchestra.run.smithi118.stdout: "zabbix_port": { 2024-03-24T03:17:23.879 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "zabbix_port", 2024-03-24T03:17:23.879 INFO:teuthology.orchestra.run.smithi118.stdout: "type": "int", 2024-03-24T03:17:23.879 INFO:teuthology.orchestra.run.smithi118.stdout: "level": "advanced", 2024-03-24T03:17:23.879 INFO:teuthology.orchestra.run.smithi118.stdout: "flags": 0, 2024-03-24T03:17:23.879 INFO:teuthology.orchestra.run.smithi118.stdout: "default_value": "10051", 2024-03-24T03:17:23.879 INFO:teuthology.orchestra.run.smithi118.stdout: "min": "", 2024-03-24T03:17:23.879 INFO:teuthology.orchestra.run.smithi118.stdout: "max": "", 2024-03-24T03:17:23.879 INFO:teuthology.orchestra.run.smithi118.stdout: "enum_allowed": [], 2024-03-24T03:17:23.879 INFO:teuthology.orchestra.run.smithi118.stdout: "desc": "", 2024-03-24T03:17:23.880 INFO:teuthology.orchestra.run.smithi118.stdout: "long_desc": "", 2024-03-24T03:17:23.880 INFO:teuthology.orchestra.run.smithi118.stdout: "tags": [], 2024-03-24T03:17:23.880 INFO:teuthology.orchestra.run.smithi118.stdout: "see_also": [] 2024-03-24T03:17:23.880 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-24T03:17:23.880 INFO:teuthology.orchestra.run.smithi118.stdout: "zabbix_sender": { 2024-03-24T03:17:23.880 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "zabbix_sender", 2024-03-24T03:17:23.880 INFO:teuthology.orchestra.run.smithi118.stdout: "type": "str", 2024-03-24T03:17:23.880 INFO:teuthology.orchestra.run.smithi118.stdout: "level": "advanced", 2024-03-24T03:17:23.880 INFO:teuthology.orchestra.run.smithi118.stdout: "flags": 0, 2024-03-24T03:17:23.881 INFO:teuthology.orchestra.run.smithi118.stdout: "default_value": "/usr/bin/zabbix_sender", 2024-03-24T03:17:23.881 INFO:teuthology.orchestra.run.smithi118.stdout: "min": "", 2024-03-24T03:17:23.881 INFO:teuthology.orchestra.run.smithi118.stdout: "max": "", 2024-03-24T03:17:23.881 INFO:teuthology.orchestra.run.smithi118.stdout: "enum_allowed": [], 2024-03-24T03:17:23.881 INFO:teuthology.orchestra.run.smithi118.stdout: "desc": "", 2024-03-24T03:17:23.881 INFO:teuthology.orchestra.run.smithi118.stdout: "long_desc": "", 2024-03-24T03:17:23.881 INFO:teuthology.orchestra.run.smithi118.stdout: "tags": [], 2024-03-24T03:17:23.881 INFO:teuthology.orchestra.run.smithi118.stdout: "see_also": [] 2024-03-24T03:17:23.881 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-03-24T03:17:23.882 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-03-24T03:17:23.882 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-03-24T03:17:23.882 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-03-24T03:17:23.882 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-03-24T03:17:23.882 INFO:teuthology.orchestra.run.smithi118.stdout: ], 2024-03-24T03:17:23.882 INFO:teuthology.orchestra.run.smithi118.stdout: "modules": [ 2024-03-24T03:17:23.882 INFO:teuthology.orchestra.run.smithi118.stdout: "cephadm", 2024-03-24T03:17:23.882 INFO:teuthology.orchestra.run.smithi118.stdout: "dashboard", 2024-03-24T03:17:23.882 INFO:teuthology.orchestra.run.smithi118.stdout: "iostat", 2024-03-24T03:17:23.883 INFO:teuthology.orchestra.run.smithi118.stdout: "nfs", 2024-03-24T03:17:23.883 INFO:teuthology.orchestra.run.smithi118.stdout: "restful" 2024-03-24T03:17:23.883 INFO:teuthology.orchestra.run.smithi118.stdout: ], 2024-03-24T03:17:23.883 INFO:teuthology.orchestra.run.smithi118.stdout: "available_modules": [ 2024-03-24T03:17:23.883 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-24T03:17:23.883 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "alerts", 2024-03-24T03:17:23.883 INFO:teuthology.orchestra.run.smithi118.stdout: "can_run": true, 2024-03-24T03:17:23.883 INFO:teuthology.orchestra.run.smithi118.stdout: "error_string": "", 2024-03-24T03:17:23.884 INFO:teuthology.orchestra.run.smithi118.stdout: "module_options": { 2024-03-24T03:17:23.884 INFO:teuthology.orchestra.run.smithi118.stdout: "interval": { 2024-03-24T03:17:23.884 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "interval", 2024-03-24T03:17:23.884 INFO:teuthology.orchestra.run.smithi118.stdout: "type": "secs", 2024-03-24T03:17:23.884 INFO:teuthology.orchestra.run.smithi118.stdout: "level": "advanced", 2024-03-24T03:17:23.884 INFO:teuthology.orchestra.run.smithi118.stdout: "flags": 1, 2024-03-24T03:17:23.884 INFO:teuthology.orchestra.run.smithi118.stdout: "default_value": "60", 2024-03-24T03:17:23.884 INFO:teuthology.orchestra.run.smithi118.stdout: "min": "", 2024-03-24T03:17:23.885 INFO:teuthology.orchestra.run.smithi118.stdout: "max": "", 2024-03-24T03:17:23.885 INFO:teuthology.orchestra.run.smithi118.stdout: "enum_allowed": [], 2024-03-24T03:17:23.885 INFO:teuthology.orchestra.run.smithi118.stdout: "desc": "How frequently to reexamine health status", 2024-03-24T03:17:23.885 INFO:teuthology.orchestra.run.smithi118.stdout: "long_desc": "", 2024-03-24T03:17:23.885 INFO:teuthology.orchestra.run.smithi118.stdout: "tags": [], 2024-03-24T03:17:23.885 INFO:teuthology.orchestra.run.smithi118.stdout: "see_also": [] 2024-03-24T03:17:23.885 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-24T03:17:23.885 INFO:teuthology.orchestra.run.smithi118.stdout: "log_level": { 2024-03-24T03:17:23.885 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "log_level", 2024-03-24T03:17:23.886 INFO:teuthology.orchestra.run.smithi118.stdout: "type": "str", 2024-03-24T03:17:23.886 INFO:teuthology.orchestra.run.smithi118.stdout: "level": "advanced", 2024-03-24T03:17:23.886 INFO:teuthology.orchestra.run.smithi118.stdout: "flags": 1, 2024-03-24T03:17:23.886 INFO:teuthology.orchestra.run.smithi118.stdout: "default_value": "", 2024-03-24T03:17:23.886 INFO:teuthology.orchestra.run.smithi118.stdout: "min": "", 2024-03-24T03:17:23.886 INFO:teuthology.orchestra.run.smithi118.stdout: "max": "", 2024-03-24T03:17:23.886 INFO:teuthology.orchestra.run.smithi118.stdout: "enum_allowed": [ 2024-03-24T03:17:23.887 INFO:teuthology.orchestra.run.smithi118.stdout: "", 2024-03-24T03:17:23.887 INFO:teuthology.orchestra.run.smithi118.stdout: "critical", 2024-03-24T03:17:23.887 INFO:teuthology.orchestra.run.smithi118.stdout: "debug", 2024-03-24T03:17:23.887 INFO:teuthology.orchestra.run.smithi118.stdout: "error", 2024-03-24T03:17:23.887 INFO:teuthology.orchestra.run.smithi118.stdout: "info", 2024-03-24T03:17:23.887 INFO:teuthology.orchestra.run.smithi118.stdout: "warning" 2024-03-24T03:17:23.887 INFO:teuthology.orchestra.run.smithi118.stdout: ], 2024-03-24T03:17:23.888 INFO:teuthology.orchestra.run.smithi118.stdout: "desc": "", 2024-03-24T03:17:23.888 INFO:teuthology.orchestra.run.smithi118.stdout: "long_desc": "", 2024-03-24T03:17:23.888 INFO:teuthology.orchestra.run.smithi118.stdout: "tags": [], 2024-03-24T03:17:23.888 INFO:teuthology.orchestra.run.smithi118.stdout: "see_also": [] 2024-03-24T03:17:23.888 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-24T03:17:23.889 INFO:teuthology.orchestra.run.smithi118.stdout: "log_to_cluster": { 2024-03-24T03:17:23.889 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "log_to_cluster", 2024-03-24T03:17:23.889 INFO:teuthology.orchestra.run.smithi118.stdout: "type": "bool", 2024-03-24T03:17:23.889 INFO:teuthology.orchestra.run.smithi118.stdout: "level": "advanced", 2024-03-24T03:17:23.889 INFO:teuthology.orchestra.run.smithi118.stdout: "flags": 1, 2024-03-24T03:17:23.889 INFO:teuthology.orchestra.run.smithi118.stdout: "default_value": "False", 2024-03-24T03:17:23.890 INFO:teuthology.orchestra.run.smithi118.stdout: "min": "", 2024-03-24T03:17:23.890 INFO:teuthology.orchestra.run.smithi118.stdout: "max": "", 2024-03-24T03:17:23.890 INFO:teuthology.orchestra.run.smithi118.stdout: "enum_allowed": [], 2024-03-24T03:17:23.890 INFO:teuthology.orchestra.run.smithi118.stdout: "desc": "", 2024-03-24T03:17:23.890 INFO:teuthology.orchestra.run.smithi118.stdout: "long_desc": "", 2024-03-24T03:17:23.891 INFO:teuthology.orchestra.run.smithi118.stdout: "tags": [], 2024-03-24T03:17:23.891 INFO:teuthology.orchestra.run.smithi118.stdout: "see_also": [] 2024-03-24T03:17:23.891 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-24T03:17:23.891 INFO:teuthology.orchestra.run.smithi118.stdout: "log_to_cluster_level": { 2024-03-24T03:17:23.891 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "log_to_cluster_level", 2024-03-24T03:17:23.892 INFO:teuthology.orchestra.run.smithi118.stdout: "type": "str", 2024-03-24T03:17:23.892 INFO:teuthology.orchestra.run.smithi118.stdout: "level": "advanced", 2024-03-24T03:17:23.892 INFO:teuthology.orchestra.run.smithi118.stdout: "flags": 1, 2024-03-24T03:17:23.892 INFO:teuthology.orchestra.run.smithi118.stdout: "default_value": "info", 2024-03-24T03:17:23.892 INFO:teuthology.orchestra.run.smithi118.stdout: "min": "", 2024-03-24T03:17:23.893 INFO:teuthology.orchestra.run.smithi118.stdout: "max": "", 2024-03-24T03:17:23.893 INFO:teuthology.orchestra.run.smithi118.stdout: "enum_allowed": [ 2024-03-24T03:17:23.893 INFO:teuthology.orchestra.run.smithi118.stdout: "", 2024-03-24T03:17:23.893 INFO:teuthology.orchestra.run.smithi118.stdout: "critical", 2024-03-24T03:17:23.893 INFO:teuthology.orchestra.run.smithi118.stdout: "debug", 2024-03-24T03:17:23.894 INFO:teuthology.orchestra.run.smithi118.stdout: "error", 2024-03-24T03:17:23.894 INFO:teuthology.orchestra.run.smithi118.stdout: "info", 2024-03-24T03:17:23.894 INFO:teuthology.orchestra.run.smithi118.stdout: "warning" 2024-03-24T03:17:23.894 INFO:teuthology.orchestra.run.smithi118.stdout: ], 2024-03-24T03:17:23.894 INFO:teuthology.orchestra.run.smithi118.stdout: "desc": "", 2024-03-24T03:17:23.895 INFO:teuthology.orchestra.run.smithi118.stdout: "long_desc": "", 2024-03-24T03:17:23.895 INFO:teuthology.orchestra.run.smithi118.stdout: "tags": [], 2024-03-24T03:17:23.895 INFO:teuthology.orchestra.run.smithi118.stdout: "see_also": [] 2024-03-24T03:17:23.895 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-24T03:17:23.896 INFO:teuthology.orchestra.run.smithi118.stdout: "log_to_file": { 2024-03-24T03:17:23.896 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "log_to_file", 2024-03-24T03:17:23.896 INFO:teuthology.orchestra.run.smithi118.stdout: "type": "bool", 2024-03-24T03:17:23.896 INFO:teuthology.orchestra.run.smithi118.stdout: "level": "advanced", 2024-03-24T03:17:23.896 INFO:teuthology.orchestra.run.smithi118.stdout: "flags": 1, 2024-03-24T03:17:23.896 INFO:teuthology.orchestra.run.smithi118.stdout: "default_value": "False", 2024-03-24T03:17:23.897 INFO:teuthology.orchestra.run.smithi118.stdout: "min": "", 2024-03-24T03:17:23.897 INFO:teuthology.orchestra.run.smithi118.stdout: "max": "", 2024-03-24T03:17:23.897 INFO:teuthology.orchestra.run.smithi118.stdout: "enum_allowed": [], 2024-03-24T03:17:23.897 INFO:teuthology.orchestra.run.smithi118.stdout: "desc": "", 2024-03-24T03:17:23.897 INFO:teuthology.orchestra.run.smithi118.stdout: "long_desc": "", 2024-03-24T03:17:23.898 INFO:teuthology.orchestra.run.smithi118.stdout: "tags": [], 2024-03-24T03:17:23.898 INFO:teuthology.orchestra.run.smithi118.stdout: "see_also": [] 2024-03-24T03:17:23.898 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-24T03:17:23.898 INFO:teuthology.orchestra.run.smithi118.stdout: "smtp_destination": { 2024-03-24T03:17:23.898 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "smtp_destination", 2024-03-24T03:17:23.899 INFO:teuthology.orchestra.run.smithi118.stdout: "type": "str", 2024-03-24T03:17:23.899 INFO:teuthology.orchestra.run.smithi118.stdout: "level": "advanced", 2024-03-24T03:17:23.899 INFO:teuthology.orchestra.run.smithi118.stdout: "flags": 1, 2024-03-24T03:17:23.899 INFO:teuthology.orchestra.run.smithi118.stdout: "default_value": "", 2024-03-24T03:17:23.899 INFO:teuthology.orchestra.run.smithi118.stdout: "min": "", 2024-03-24T03:17:23.899 INFO:teuthology.orchestra.run.smithi118.stdout: "max": "", 2024-03-24T03:17:23.900 INFO:teuthology.orchestra.run.smithi118.stdout: "enum_allowed": [], 2024-03-24T03:17:23.900 INFO:teuthology.orchestra.run.smithi118.stdout: "desc": "Email address to send alerts to", 2024-03-24T03:17:23.900 INFO:teuthology.orchestra.run.smithi118.stdout: "long_desc": "", 2024-03-24T03:17:23.900 INFO:teuthology.orchestra.run.smithi118.stdout: "tags": [], 2024-03-24T03:17:23.900 INFO:teuthology.orchestra.run.smithi118.stdout: "see_also": [] 2024-03-24T03:17:23.901 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-24T03:17:23.901 INFO:teuthology.orchestra.run.smithi118.stdout: "smtp_from_name": { 2024-03-24T03:17:23.901 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "smtp_from_name", 2024-03-24T03:17:23.901 INFO:teuthology.orchestra.run.smithi118.stdout: "type": "str", 2024-03-24T03:17:23.901 INFO:teuthology.orchestra.run.smithi118.stdout: "level": "advanced", 2024-03-24T03:17:23.902 INFO:teuthology.orchestra.run.smithi118.stdout: "flags": 1, 2024-03-24T03:17:23.902 INFO:teuthology.orchestra.run.smithi118.stdout: "default_value": "Ceph", 2024-03-24T03:17:23.902 INFO:teuthology.orchestra.run.smithi118.stdout: "min": "", 2024-03-24T03:17:23.902 INFO:teuthology.orchestra.run.smithi118.stdout: "max": "", 2024-03-24T03:17:23.902 INFO:teuthology.orchestra.run.smithi118.stdout: "enum_allowed": [], 2024-03-24T03:17:23.902 INFO:teuthology.orchestra.run.smithi118.stdout: "desc": "Email From: name", 2024-03-24T03:17:23.903 INFO:teuthology.orchestra.run.smithi118.stdout: "long_desc": "", 2024-03-24T03:17:23.903 INFO:teuthology.orchestra.run.smithi118.stdout: "tags": [], 2024-03-24T03:17:23.903 INFO:teuthology.orchestra.run.smithi118.stdout: "see_also": [] 2024-03-24T03:17:23.903 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-24T03:17:23.903 INFO:teuthology.orchestra.run.smithi118.stdout: "smtp_host": { 2024-03-24T03:17:23.903 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "smtp_host", 2024-03-24T03:17:23.903 INFO:teuthology.orchestra.run.smithi118.stdout: "type": "str", 2024-03-24T03:17:23.903 INFO:teuthology.orchestra.run.smithi118.stdout: "level": "advanced", 2024-03-24T03:17:23.903 INFO:teuthology.orchestra.run.smithi118.stdout: "flags": 1, 2024-03-24T03:17:23.904 INFO:teuthology.orchestra.run.smithi118.stdout: "default_value": "", 2024-03-24T03:17:23.904 INFO:teuthology.orchestra.run.smithi118.stdout: "min": "", 2024-03-24T03:17:23.904 INFO:teuthology.orchestra.run.smithi118.stdout: "max": "", 2024-03-24T03:17:23.904 INFO:teuthology.orchestra.run.smithi118.stdout: "enum_allowed": [], 2024-03-24T03:17:23.904 INFO:teuthology.orchestra.run.smithi118.stdout: "desc": "SMTP server", 2024-03-24T03:17:23.904 INFO:teuthology.orchestra.run.smithi118.stdout: "long_desc": "", 2024-03-24T03:17:23.904 INFO:teuthology.orchestra.run.smithi118.stdout: "tags": [], 2024-03-24T03:17:23.905 INFO:teuthology.orchestra.run.smithi118.stdout: "see_also": [] 2024-03-24T03:17:23.905 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-24T03:17:23.905 INFO:teuthology.orchestra.run.smithi118.stdout: "smtp_password": { 2024-03-24T03:17:23.905 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "smtp_password", 2024-03-24T03:17:23.905 INFO:teuthology.orchestra.run.smithi118.stdout: "type": "str", 2024-03-24T03:17:23.905 INFO:teuthology.orchestra.run.smithi118.stdout: "level": "advanced", 2024-03-24T03:17:23.905 INFO:teuthology.orchestra.run.smithi118.stdout: "flags": 1, 2024-03-24T03:17:23.905 INFO:teuthology.orchestra.run.smithi118.stdout: "default_value": "", 2024-03-24T03:17:23.905 INFO:teuthology.orchestra.run.smithi118.stdout: "min": "", 2024-03-24T03:17:23.905 INFO:teuthology.orchestra.run.smithi118.stdout: "max": "", 2024-03-24T03:17:23.906 INFO:teuthology.orchestra.run.smithi118.stdout: "enum_allowed": [], 2024-03-24T03:17:23.906 INFO:teuthology.orchestra.run.smithi118.stdout: "desc": "Password to authenticate with", 2024-03-24T03:17:23.906 INFO:teuthology.orchestra.run.smithi118.stdout: "long_desc": "", 2024-03-24T03:17:23.906 INFO:teuthology.orchestra.run.smithi118.stdout: "tags": [], 2024-03-24T03:17:23.906 INFO:teuthology.orchestra.run.smithi118.stdout: "see_also": [] 2024-03-24T03:17:23.906 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-24T03:17:23.906 INFO:teuthology.orchestra.run.smithi118.stdout: "smtp_port": { 2024-03-24T03:17:23.906 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "smtp_port", 2024-03-24T03:17:23.907 INFO:teuthology.orchestra.run.smithi118.stdout: "type": "int", 2024-03-24T03:17:23.907 INFO:teuthology.orchestra.run.smithi118.stdout: "level": "advanced", 2024-03-24T03:17:23.907 INFO:teuthology.orchestra.run.smithi118.stdout: "flags": 1, 2024-03-24T03:17:23.907 INFO:teuthology.orchestra.run.smithi118.stdout: "default_value": "465", 2024-03-24T03:17:23.907 INFO:teuthology.orchestra.run.smithi118.stdout: "min": "", 2024-03-24T03:17:23.907 INFO:teuthology.orchestra.run.smithi118.stdout: "max": "", 2024-03-24T03:17:23.907 INFO:teuthology.orchestra.run.smithi118.stdout: "enum_allowed": [], 2024-03-24T03:17:23.907 INFO:teuthology.orchestra.run.smithi118.stdout: "desc": "SMTP port", 2024-03-24T03:17:23.908 INFO:teuthology.orchestra.run.smithi118.stdout: "long_desc": "", 2024-03-24T03:17:23.908 INFO:teuthology.orchestra.run.smithi118.stdout: "tags": [], 2024-03-24T03:17:23.908 INFO:teuthology.orchestra.run.smithi118.stdout: "see_also": [] 2024-03-24T03:17:23.908 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-24T03:17:23.908 INFO:teuthology.orchestra.run.smithi118.stdout: "smtp_sender": { 2024-03-24T03:17:23.908 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "smtp_sender", 2024-03-24T03:17:23.908 INFO:teuthology.orchestra.run.smithi118.stdout: "type": "str", 2024-03-24T03:17:23.908 INFO:teuthology.orchestra.run.smithi118.stdout: "level": "advanced", 2024-03-24T03:17:23.908 INFO:teuthology.orchestra.run.smithi118.stdout: "flags": 1, 2024-03-24T03:17:23.909 INFO:teuthology.orchestra.run.smithi118.stdout: "default_value": "", 2024-03-24T03:17:23.909 INFO:teuthology.orchestra.run.smithi118.stdout: "min": "", 2024-03-24T03:17:23.909 INFO:teuthology.orchestra.run.smithi118.stdout: "max": "", 2024-03-24T03:17:23.909 INFO:teuthology.orchestra.run.smithi118.stdout: "enum_allowed": [], 2024-03-24T03:17:23.909 INFO:teuthology.orchestra.run.smithi118.stdout: "desc": "SMTP envelope sender", 2024-03-24T03:17:23.909 INFO:teuthology.orchestra.run.smithi118.stdout: "long_desc": "", 2024-03-24T03:17:23.909 INFO:teuthology.orchestra.run.smithi118.stdout: "tags": [], 2024-03-24T03:17:23.909 INFO:teuthology.orchestra.run.smithi118.stdout: "see_also": [] 2024-03-24T03:17:23.909 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-24T03:17:23.910 INFO:teuthology.orchestra.run.smithi118.stdout: "smtp_ssl": { 2024-03-24T03:17:23.910 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "smtp_ssl", 2024-03-24T03:17:23.910 INFO:teuthology.orchestra.run.smithi118.stdout: "type": "bool", 2024-03-24T03:17:23.910 INFO:teuthology.orchestra.run.smithi118.stdout: "level": "advanced", 2024-03-24T03:17:23.910 INFO:teuthology.orchestra.run.smithi118.stdout: "flags": 1, 2024-03-24T03:17:23.910 INFO:teuthology.orchestra.run.smithi118.stdout: "default_value": "True", 2024-03-24T03:17:23.910 INFO:teuthology.orchestra.run.smithi118.stdout: "min": "", 2024-03-24T03:17:23.910 INFO:teuthology.orchestra.run.smithi118.stdout: "max": "", 2024-03-24T03:17:23.911 INFO:teuthology.orchestra.run.smithi118.stdout: "enum_allowed": [], 2024-03-24T03:17:23.911 INFO:teuthology.orchestra.run.smithi118.stdout: "desc": "Use SSL to connect to SMTP server", 2024-03-24T03:17:23.911 INFO:teuthology.orchestra.run.smithi118.stdout: "long_desc": "", 2024-03-24T03:17:23.911 INFO:teuthology.orchestra.run.smithi118.stdout: "tags": [], 2024-03-24T03:17:23.911 INFO:teuthology.orchestra.run.smithi118.stdout: "see_also": [] 2024-03-24T03:17:23.911 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-24T03:17:23.911 INFO:teuthology.orchestra.run.smithi118.stdout: "smtp_user": { 2024-03-24T03:17:23.911 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "smtp_user", 2024-03-24T03:17:23.911 INFO:teuthology.orchestra.run.smithi118.stdout: "type": "str", 2024-03-24T03:17:23.912 INFO:teuthology.orchestra.run.smithi118.stdout: "level": "advanced", 2024-03-24T03:17:23.912 INFO:teuthology.orchestra.run.smithi118.stdout: "flags": 1, 2024-03-24T03:17:23.912 INFO:teuthology.orchestra.run.smithi118.stdout: "default_value": "", 2024-03-24T03:17:23.912 INFO:teuthology.orchestra.run.smithi118.stdout: "min": "", 2024-03-24T03:17:23.912 INFO:teuthology.orchestra.run.smithi118.stdout: "max": "", 2024-03-24T03:17:23.912 INFO:teuthology.orchestra.run.smithi118.stdout: "enum_allowed": [], 2024-03-24T03:17:23.912 INFO:teuthology.orchestra.run.smithi118.stdout: "desc": "User to authenticate as", 2024-03-24T03:17:23.912 INFO:teuthology.orchestra.run.smithi118.stdout: "long_desc": "", 2024-03-24T03:17:23.912 INFO:teuthology.orchestra.run.smithi118.stdout: "tags": [], 2024-03-24T03:17:23.913 INFO:teuthology.orchestra.run.smithi118.stdout: "see_also": [] 2024-03-24T03:17:23.913 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-03-24T03:17:23.913 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-03-24T03:17:23.913 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-24T03:17:23.913 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-24T03:17:23.913 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "balancer", 2024-03-24T03:17:23.913 INFO:teuthology.orchestra.run.smithi118.stdout: "can_run": true, 2024-03-24T03:17:23.913 INFO:teuthology.orchestra.run.smithi118.stdout: "error_string": "", 2024-03-24T03:17:23.913 INFO:teuthology.orchestra.run.smithi118.stdout: "module_options": { 2024-03-24T03:17:23.914 INFO:teuthology.orchestra.run.smithi118.stdout: "active": { 2024-03-24T03:17:23.914 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "active", 2024-03-24T03:17:23.914 INFO:teuthology.orchestra.run.smithi118.stdout: "type": "bool", 2024-03-24T03:17:23.914 INFO:teuthology.orchestra.run.smithi118.stdout: "level": "advanced", 2024-03-24T03:17:23.914 INFO:teuthology.orchestra.run.smithi118.stdout: "flags": 1, 2024-03-24T03:17:23.914 INFO:teuthology.orchestra.run.smithi118.stdout: "default_value": "True", 2024-03-24T03:17:23.915 INFO:teuthology.orchestra.run.smithi118.stdout: "min": "", 2024-03-24T03:17:23.915 INFO:teuthology.orchestra.run.smithi118.stdout: "max": "", 2024-03-24T03:17:23.915 INFO:teuthology.orchestra.run.smithi118.stdout: "enum_allowed": [], 2024-03-24T03:17:23.915 INFO:teuthology.orchestra.run.smithi118.stdout: "desc": "automatically balance PGs across cluster", 2024-03-24T03:17:23.915 INFO:teuthology.orchestra.run.smithi118.stdout: "long_desc": "", 2024-03-24T03:17:23.915 INFO:teuthology.orchestra.run.smithi118.stdout: "tags": [], 2024-03-24T03:17:23.915 INFO:teuthology.orchestra.run.smithi118.stdout: "see_also": [] 2024-03-24T03:17:23.916 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-24T03:17:23.916 INFO:teuthology.orchestra.run.smithi118.stdout: "begin_time": { 2024-03-24T03:17:23.916 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "begin_time", 2024-03-24T03:17:23.916 INFO:teuthology.orchestra.run.smithi118.stdout: "type": "str", 2024-03-24T03:17:23.916 INFO:teuthology.orchestra.run.smithi118.stdout: "level": "advanced", 2024-03-24T03:17:23.916 INFO:teuthology.orchestra.run.smithi118.stdout: "flags": 1, 2024-03-24T03:17:23.916 INFO:teuthology.orchestra.run.smithi118.stdout: "default_value": "0000", 2024-03-24T03:17:23.916 INFO:teuthology.orchestra.run.smithi118.stdout: "min": "", 2024-03-24T03:17:23.917 INFO:teuthology.orchestra.run.smithi118.stdout: "max": "", 2024-03-24T03:17:23.917 INFO:teuthology.orchestra.run.smithi118.stdout: "enum_allowed": [], 2024-03-24T03:17:23.917 INFO:teuthology.orchestra.run.smithi118.stdout: "desc": "beginning time of day to automatically balance", 2024-03-24T03:17:23.917 INFO:teuthology.orchestra.run.smithi118.stdout: "long_desc": "This is a time of day in the format HHMM.", 2024-03-24T03:17:23.917 INFO:teuthology.orchestra.run.smithi118.stdout: "tags": [], 2024-03-24T03:17:23.917 INFO:teuthology.orchestra.run.smithi118.stdout: "see_also": [] 2024-03-24T03:17:23.917 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-24T03:17:23.917 INFO:teuthology.orchestra.run.smithi118.stdout: "begin_weekday": { 2024-03-24T03:17:23.918 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "begin_weekday", 2024-03-24T03:17:23.918 INFO:teuthology.orchestra.run.smithi118.stdout: "type": "uint", 2024-03-24T03:17:23.918 INFO:teuthology.orchestra.run.smithi118.stdout: "level": "advanced", 2024-03-24T03:17:23.918 INFO:teuthology.orchestra.run.smithi118.stdout: "flags": 1, 2024-03-24T03:17:23.918 INFO:teuthology.orchestra.run.smithi118.stdout: "default_value": "0", 2024-03-24T03:17:23.918 INFO:teuthology.orchestra.run.smithi118.stdout: "min": "0", 2024-03-24T03:17:23.918 INFO:teuthology.orchestra.run.smithi118.stdout: "max": "6", 2024-03-24T03:17:23.918 INFO:teuthology.orchestra.run.smithi118.stdout: "enum_allowed": [], 2024-03-24T03:17:23.919 INFO:teuthology.orchestra.run.smithi118.stdout: "desc": "Restrict automatic balancing to this day of the week or later", 2024-03-24T03:17:23.919 INFO:teuthology.orchestra.run.smithi118.stdout: "long_desc": "0 = Sunday, 1 = Monday, etc.", 2024-03-24T03:17:23.919 INFO:teuthology.orchestra.run.smithi118.stdout: "tags": [], 2024-03-24T03:17:23.919 INFO:teuthology.orchestra.run.smithi118.stdout: "see_also": [] 2024-03-24T03:17:23.919 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-24T03:17:23.919 INFO:teuthology.orchestra.run.smithi118.stdout: "crush_compat_max_iterations": { 2024-03-24T03:17:23.919 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "crush_compat_max_iterations", 2024-03-24T03:17:23.919 INFO:teuthology.orchestra.run.smithi118.stdout: "type": "uint", 2024-03-24T03:17:23.920 INFO:teuthology.orchestra.run.smithi118.stdout: "level": "advanced", 2024-03-24T03:17:23.920 INFO:teuthology.orchestra.run.smithi118.stdout: "flags": 1, 2024-03-24T03:17:23.920 INFO:teuthology.orchestra.run.smithi118.stdout: "default_value": "25", 2024-03-24T03:17:23.920 INFO:teuthology.orchestra.run.smithi118.stdout: "min": "1", 2024-03-24T03:17:23.920 INFO:teuthology.orchestra.run.smithi118.stdout: "max": "250", 2024-03-24T03:17:23.920 INFO:teuthology.orchestra.run.smithi118.stdout: "enum_allowed": [], 2024-03-24T03:17:23.920 INFO:teuthology.orchestra.run.smithi118.stdout: "desc": "maximum number of iterations to attempt optimization", 2024-03-24T03:17:23.920 INFO:teuthology.orchestra.run.smithi118.stdout: "long_desc": "", 2024-03-24T03:17:23.921 INFO:teuthology.orchestra.run.smithi118.stdout: "tags": [], 2024-03-24T03:17:23.921 INFO:teuthology.orchestra.run.smithi118.stdout: "see_also": [] 2024-03-24T03:17:23.921 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-24T03:17:23.921 INFO:teuthology.orchestra.run.smithi118.stdout: "crush_compat_metrics": { 2024-03-24T03:17:23.921 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "crush_compat_metrics", 2024-03-24T03:17:23.921 INFO:teuthology.orchestra.run.smithi118.stdout: "type": "str", 2024-03-24T03:17:23.921 INFO:teuthology.orchestra.run.smithi118.stdout: "level": "advanced", 2024-03-24T03:17:23.921 INFO:teuthology.orchestra.run.smithi118.stdout: "flags": 1, 2024-03-24T03:17:23.921 INFO:teuthology.orchestra.run.smithi118.stdout: "default_value": "pgs,objects,bytes", 2024-03-24T03:17:23.922 INFO:teuthology.orchestra.run.smithi118.stdout: "min": "", 2024-03-24T03:17:23.922 INFO:teuthology.orchestra.run.smithi118.stdout: "max": "", 2024-03-24T03:17:23.922 INFO:teuthology.orchestra.run.smithi118.stdout: "enum_allowed": [], 2024-03-24T03:17:23.922 INFO:teuthology.orchestra.run.smithi118.stdout: "desc": "metrics with which to calculate OSD utilization", 2024-03-24T03:17:23.922 INFO:teuthology.orchestra.run.smithi118.stdout: "long_desc": "Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.", 2024-03-24T03:17:23.922 INFO:teuthology.orchestra.run.smithi118.stdout: "tags": [], 2024-03-24T03:17:23.922 INFO:teuthology.orchestra.run.smithi118.stdout: "see_also": [] 2024-03-24T03:17:23.923 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-24T03:17:23.923 INFO:teuthology.orchestra.run.smithi118.stdout: "crush_compat_step": { 2024-03-24T03:17:23.923 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "crush_compat_step", 2024-03-24T03:17:23.923 INFO:teuthology.orchestra.run.smithi118.stdout: "type": "float", 2024-03-24T03:17:23.923 INFO:teuthology.orchestra.run.smithi118.stdout: "level": "advanced", 2024-03-24T03:17:23.923 INFO:teuthology.orchestra.run.smithi118.stdout: "flags": 1, 2024-03-24T03:17:23.923 INFO:teuthology.orchestra.run.smithi118.stdout: "default_value": "0.5", 2024-03-24T03:17:23.923 INFO:teuthology.orchestra.run.smithi118.stdout: "min": "0.001", 2024-03-24T03:17:23.923 INFO:teuthology.orchestra.run.smithi118.stdout: "max": "0.999", 2024-03-24T03:17:23.924 INFO:teuthology.orchestra.run.smithi118.stdout: "enum_allowed": [], 2024-03-24T03:17:23.924 INFO:teuthology.orchestra.run.smithi118.stdout: "desc": "aggressiveness of optimization", 2024-03-24T03:17:23.924 INFO:teuthology.orchestra.run.smithi118.stdout: "long_desc": ".99 is very aggressive, .01 is less aggressive", 2024-03-24T03:17:23.924 INFO:teuthology.orchestra.run.smithi118.stdout: "tags": [], 2024-03-24T03:17:23.924 INFO:teuthology.orchestra.run.smithi118.stdout: "see_also": [] 2024-03-24T03:17:23.924 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-24T03:17:23.924 INFO:teuthology.orchestra.run.smithi118.stdout: "end_time": { 2024-03-24T03:17:23.924 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "end_time", 2024-03-24T03:17:23.924 INFO:teuthology.orchestra.run.smithi118.stdout: "type": "str", 2024-03-24T03:17:23.925 INFO:teuthology.orchestra.run.smithi118.stdout: "level": "advanced", 2024-03-24T03:17:23.925 INFO:teuthology.orchestra.run.smithi118.stdout: "flags": 1, 2024-03-24T03:17:23.925 INFO:teuthology.orchestra.run.smithi118.stdout: "default_value": "2359", 2024-03-24T03:17:23.925 INFO:teuthology.orchestra.run.smithi118.stdout: "min": "", 2024-03-24T03:17:23.925 INFO:teuthology.orchestra.run.smithi118.stdout: "max": "", 2024-03-24T03:17:23.925 INFO:teuthology.orchestra.run.smithi118.stdout: "enum_allowed": [], 2024-03-24T03:17:23.925 INFO:teuthology.orchestra.run.smithi118.stdout: "desc": "ending time of day to automatically balance", 2024-03-24T03:17:23.925 INFO:teuthology.orchestra.run.smithi118.stdout: "long_desc": "This is a time of day in the format HHMM.", 2024-03-24T03:17:23.925 INFO:teuthology.orchestra.run.smithi118.stdout: "tags": [], 2024-03-24T03:17:23.926 INFO:teuthology.orchestra.run.smithi118.stdout: "see_also": [] 2024-03-24T03:17:23.926 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-24T03:17:23.926 INFO:teuthology.orchestra.run.smithi118.stdout: "end_weekday": { 2024-03-24T03:17:23.926 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "end_weekday", 2024-03-24T03:17:23.926 INFO:teuthology.orchestra.run.smithi118.stdout: "type": "uint", 2024-03-24T03:17:23.926 INFO:teuthology.orchestra.run.smithi118.stdout: "level": "advanced", 2024-03-24T03:17:23.926 INFO:teuthology.orchestra.run.smithi118.stdout: "flags": 1, 2024-03-24T03:17:23.926 INFO:teuthology.orchestra.run.smithi118.stdout: "default_value": "0", 2024-03-24T03:17:23.926 INFO:teuthology.orchestra.run.smithi118.stdout: "min": "0", 2024-03-24T03:17:23.927 INFO:teuthology.orchestra.run.smithi118.stdout: "max": "6", 2024-03-24T03:17:23.927 INFO:teuthology.orchestra.run.smithi118.stdout: "enum_allowed": [], 2024-03-24T03:17:23.927 INFO:teuthology.orchestra.run.smithi118.stdout: "desc": "Restrict automatic balancing to days of the week earlier than this", 2024-03-24T03:17:23.927 INFO:teuthology.orchestra.run.smithi118.stdout: "long_desc": "0 = Sunday, 1 = Monday, etc.", 2024-03-24T03:17:23.927 INFO:teuthology.orchestra.run.smithi118.stdout: "tags": [], 2024-03-24T03:17:23.927 INFO:teuthology.orchestra.run.smithi118.stdout: "see_also": [] 2024-03-24T03:17:23.927 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-24T03:17:23.927 INFO:teuthology.orchestra.run.smithi118.stdout: "log_level": { 2024-03-24T03:17:23.928 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "log_level", 2024-03-24T03:17:23.928 INFO:teuthology.orchestra.run.smithi118.stdout: "type": "str", 2024-03-24T03:17:23.928 INFO:teuthology.orchestra.run.smithi118.stdout: "level": "advanced", 2024-03-24T03:17:23.928 INFO:teuthology.orchestra.run.smithi118.stdout: "flags": 1, 2024-03-24T03:17:23.928 INFO:teuthology.orchestra.run.smithi118.stdout: "default_value": "", 2024-03-24T03:17:23.928 INFO:teuthology.orchestra.run.smithi118.stdout: "min": "", 2024-03-24T03:17:23.928 INFO:teuthology.orchestra.run.smithi118.stdout: "max": "", 2024-03-24T03:17:23.928 INFO:teuthology.orchestra.run.smithi118.stdout: "enum_allowed": [ 2024-03-24T03:17:23.928 INFO:teuthology.orchestra.run.smithi118.stdout: "", 2024-03-24T03:17:23.929 INFO:teuthology.orchestra.run.smithi118.stdout: "critical", 2024-03-24T03:17:23.929 INFO:teuthology.orchestra.run.smithi118.stdout: "debug", 2024-03-24T03:17:23.929 INFO:teuthology.orchestra.run.smithi118.stdout: "error", 2024-03-24T03:17:23.929 INFO:teuthology.orchestra.run.smithi118.stdout: "info", 2024-03-24T03:17:23.929 INFO:teuthology.orchestra.run.smithi118.stdout: "warning" 2024-03-24T03:17:23.929 INFO:teuthology.orchestra.run.smithi118.stdout: ], 2024-03-24T03:17:23.929 INFO:teuthology.orchestra.run.smithi118.stdout: "desc": "", 2024-03-24T03:17:23.929 INFO:teuthology.orchestra.run.smithi118.stdout: "long_desc": "", 2024-03-24T03:17:23.929 INFO:teuthology.orchestra.run.smithi118.stdout: "tags": [], 2024-03-24T03:17:23.929 INFO:teuthology.orchestra.run.smithi118.stdout: "see_also": [] 2024-03-24T03:17:23.930 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-24T03:17:23.930 INFO:teuthology.orchestra.run.smithi118.stdout: "log_to_cluster": { 2024-03-24T03:17:23.930 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "log_to_cluster", 2024-03-24T03:17:23.930 INFO:teuthology.orchestra.run.smithi118.stdout: "type": "bool", 2024-03-24T03:17:23.930 INFO:teuthology.orchestra.run.smithi118.stdout: "level": "advanced", 2024-03-24T03:17:23.930 INFO:teuthology.orchestra.run.smithi118.stdout: "flags": 1, 2024-03-24T03:17:23.930 INFO:teuthology.orchestra.run.smithi118.stdout: "default_value": "False", 2024-03-24T03:17:23.930 INFO:teuthology.orchestra.run.smithi118.stdout: "min": "", 2024-03-24T03:17:23.931 INFO:teuthology.orchestra.run.smithi118.stdout: "max": "", 2024-03-24T03:17:23.931 INFO:teuthology.orchestra.run.smithi118.stdout: "enum_allowed": [], 2024-03-24T03:17:23.931 INFO:teuthology.orchestra.run.smithi118.stdout: "desc": "", 2024-03-24T03:17:23.931 INFO:teuthology.orchestra.run.smithi118.stdout: "long_desc": "", 2024-03-24T03:17:23.931 INFO:teuthology.orchestra.run.smithi118.stdout: "tags": [], 2024-03-24T03:17:23.931 INFO:teuthology.orchestra.run.smithi118.stdout: "see_also": [] 2024-03-24T03:17:23.931 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-24T03:17:23.931 INFO:teuthology.orchestra.run.smithi118.stdout: "log_to_cluster_level": { 2024-03-24T03:17:23.931 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "log_to_cluster_level", 2024-03-24T03:17:23.932 INFO:teuthology.orchestra.run.smithi118.stdout: "type": "str", 2024-03-24T03:17:23.932 INFO:teuthology.orchestra.run.smithi118.stdout: "level": "advanced", 2024-03-24T03:17:23.932 INFO:teuthology.orchestra.run.smithi118.stdout: "flags": 1, 2024-03-24T03:17:23.932 INFO:teuthology.orchestra.run.smithi118.stdout: "default_value": "info", 2024-03-24T03:17:23.932 INFO:teuthology.orchestra.run.smithi118.stdout: "min": "", 2024-03-24T03:17:23.932 INFO:teuthology.orchestra.run.smithi118.stdout: "max": "", 2024-03-24T03:17:23.932 INFO:teuthology.orchestra.run.smithi118.stdout: "enum_allowed": [ 2024-03-24T03:17:23.932 INFO:teuthology.orchestra.run.smithi118.stdout: "", 2024-03-24T03:17:23.932 INFO:teuthology.orchestra.run.smithi118.stdout: "critical", 2024-03-24T03:17:23.933 INFO:teuthology.orchestra.run.smithi118.stdout: "debug", 2024-03-24T03:17:23.933 INFO:teuthology.orchestra.run.smithi118.stdout: "error", 2024-03-24T03:17:23.933 INFO:teuthology.orchestra.run.smithi118.stdout: "info", 2024-03-24T03:17:23.933 INFO:teuthology.orchestra.run.smithi118.stdout: "warning" 2024-03-24T03:17:23.933 INFO:teuthology.orchestra.run.smithi118.stdout: ], 2024-03-24T03:17:23.933 INFO:teuthology.orchestra.run.smithi118.stdout: "desc": "", 2024-03-24T03:17:23.933 INFO:teuthology.orchestra.run.smithi118.stdout: "long_desc": "", 2024-03-24T03:17:23.933 INFO:teuthology.orchestra.run.smithi118.stdout: "tags": [], 2024-03-24T03:17:23.933 INFO:teuthology.orchestra.run.smithi118.stdout: "see_also": [] 2024-03-24T03:17:23.934 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-24T03:17:23.934 INFO:teuthology.orchestra.run.smithi118.stdout: "log_to_file": { 2024-03-24T03:17:23.934 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "log_to_file", 2024-03-24T03:17:23.934 INFO:teuthology.orchestra.run.smithi118.stdout: "type": "bool", 2024-03-24T03:17:23.934 INFO:teuthology.orchestra.run.smithi118.stdout: "level": "advanced", 2024-03-24T03:17:23.934 INFO:teuthology.orchestra.run.smithi118.stdout: "flags": 1, 2024-03-24T03:17:23.934 INFO:teuthology.orchestra.run.smithi118.stdout: "default_value": "False", 2024-03-24T03:17:23.934 INFO:teuthology.orchestra.run.smithi118.stdout: "min": "", 2024-03-24T03:17:23.935 INFO:teuthology.orchestra.run.smithi118.stdout: "max": "", 2024-03-24T03:17:23.935 INFO:teuthology.orchestra.run.smithi118.stdout: "enum_allowed": [], 2024-03-24T03:17:23.935 INFO:teuthology.orchestra.run.smithi118.stdout: "desc": "", 2024-03-24T03:17:23.935 INFO:teuthology.orchestra.run.smithi118.stdout: "long_desc": "", 2024-03-24T03:17:23.935 INFO:teuthology.orchestra.run.smithi118.stdout: "tags": [], 2024-03-24T03:17:23.935 INFO:teuthology.orchestra.run.smithi118.stdout: "see_also": [] 2024-03-24T03:17:23.935 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-24T03:17:23.935 INFO:teuthology.orchestra.run.smithi118.stdout: "min_score": { 2024-03-24T03:17:23.935 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "min_score", 2024-03-24T03:17:23.936 INFO:teuthology.orchestra.run.smithi118.stdout: "type": "float", 2024-03-24T03:17:23.936 INFO:teuthology.orchestra.run.smithi118.stdout: "level": "advanced", 2024-03-24T03:17:23.936 INFO:teuthology.orchestra.run.smithi118.stdout: "flags": 1, 2024-03-24T03:17:23.936 INFO:teuthology.orchestra.run.smithi118.stdout: "default_value": "0", 2024-03-24T03:17:23.936 INFO:teuthology.orchestra.run.smithi118.stdout: "min": "", 2024-03-24T03:17:23.936 INFO:teuthology.orchestra.run.smithi118.stdout: "max": "", 2024-03-24T03:17:23.936 INFO:teuthology.orchestra.run.smithi118.stdout: "enum_allowed": [], 2024-03-24T03:17:23.936 INFO:teuthology.orchestra.run.smithi118.stdout: "desc": "minimum score, below which no optimization is attempted", 2024-03-24T03:17:23.936 INFO:teuthology.orchestra.run.smithi118.stdout: "long_desc": "", 2024-03-24T03:17:23.937 INFO:teuthology.orchestra.run.smithi118.stdout: "tags": [], 2024-03-24T03:17:23.937 INFO:teuthology.orchestra.run.smithi118.stdout: "see_also": [] 2024-03-24T03:17:23.937 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-24T03:17:23.937 INFO:teuthology.orchestra.run.smithi118.stdout: "mode": { 2024-03-24T03:17:23.937 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "mode", 2024-03-24T03:17:23.937 INFO:teuthology.orchestra.run.smithi118.stdout: "type": "str", 2024-03-24T03:17:23.937 INFO:teuthology.orchestra.run.smithi118.stdout: "level": "advanced", 2024-03-24T03:17:23.937 INFO:teuthology.orchestra.run.smithi118.stdout: "flags": 1, 2024-03-24T03:17:23.938 INFO:teuthology.orchestra.run.smithi118.stdout: "default_value": "upmap", 2024-03-24T03:17:23.938 INFO:teuthology.orchestra.run.smithi118.stdout: "min": "", 2024-03-24T03:17:23.938 INFO:teuthology.orchestra.run.smithi118.stdout: "max": "", 2024-03-24T03:17:23.938 INFO:teuthology.orchestra.run.smithi118.stdout: "enum_allowed": [ 2024-03-24T03:17:23.938 INFO:teuthology.orchestra.run.smithi118.stdout: "crush-compat", 2024-03-24T03:17:23.938 INFO:teuthology.orchestra.run.smithi118.stdout: "none", 2024-03-24T03:17:23.938 INFO:teuthology.orchestra.run.smithi118.stdout: "read", 2024-03-24T03:17:23.938 INFO:teuthology.orchestra.run.smithi118.stdout: "upmap", 2024-03-24T03:17:23.939 INFO:teuthology.orchestra.run.smithi118.stdout: "upmap-read" 2024-03-24T03:17:23.939 INFO:teuthology.orchestra.run.smithi118.stdout: ], 2024-03-24T03:17:23.939 INFO:teuthology.orchestra.run.smithi118.stdout: "desc": "Balancer mode", 2024-03-24T03:17:23.939 INFO:teuthology.orchestra.run.smithi118.stdout: "long_desc": "", 2024-03-24T03:17:23.939 INFO:teuthology.orchestra.run.smithi118.stdout: "tags": [], 2024-03-24T03:17:23.939 INFO:teuthology.orchestra.run.smithi118.stdout: "see_also": [] 2024-03-24T03:17:23.939 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-24T03:17:23.939 INFO:teuthology.orchestra.run.smithi118.stdout: "pool_ids": { 2024-03-24T03:17:23.940 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "pool_ids", 2024-03-24T03:17:23.940 INFO:teuthology.orchestra.run.smithi118.stdout: "type": "str", 2024-03-24T03:17:23.940 INFO:teuthology.orchestra.run.smithi118.stdout: "level": "advanced", 2024-03-24T03:17:23.940 INFO:teuthology.orchestra.run.smithi118.stdout: "flags": 1, 2024-03-24T03:17:23.940 INFO:teuthology.orchestra.run.smithi118.stdout: "default_value": "", 2024-03-24T03:17:23.940 INFO:teuthology.orchestra.run.smithi118.stdout: "min": "", 2024-03-24T03:17:23.940 INFO:teuthology.orchestra.run.smithi118.stdout: "max": "", 2024-03-24T03:17:23.940 INFO:teuthology.orchestra.run.smithi118.stdout: "enum_allowed": [], 2024-03-24T03:17:23.940 INFO:teuthology.orchestra.run.smithi118.stdout: "desc": "pools which the automatic balancing will be limited to", 2024-03-24T03:17:23.941 INFO:teuthology.orchestra.run.smithi118.stdout: "long_desc": "", 2024-03-24T03:17:23.941 INFO:teuthology.orchestra.run.smithi118.stdout: "tags": [], 2024-03-24T03:17:23.941 INFO:teuthology.orchestra.run.smithi118.stdout: "see_also": [] 2024-03-24T03:17:23.941 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-24T03:17:23.941 INFO:teuthology.orchestra.run.smithi118.stdout: "sleep_interval": { 2024-03-24T03:17:23.941 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "sleep_interval", 2024-03-24T03:17:23.941 INFO:teuthology.orchestra.run.smithi118.stdout: "type": "secs", 2024-03-24T03:17:23.941 INFO:teuthology.orchestra.run.smithi118.stdout: "level": "advanced", 2024-03-24T03:17:23.941 INFO:teuthology.orchestra.run.smithi118.stdout: "flags": 1, 2024-03-24T03:17:23.942 INFO:teuthology.orchestra.run.smithi118.stdout: "default_value": "60", 2024-03-24T03:17:23.942 INFO:teuthology.orchestra.run.smithi118.stdout: "min": "", 2024-03-24T03:17:23.942 INFO:teuthology.orchestra.run.smithi118.stdout: "max": "", 2024-03-24T03:17:23.942 INFO:teuthology.orchestra.run.smithi118.stdout: "enum_allowed": [], 2024-03-24T03:17:23.942 INFO:teuthology.orchestra.run.smithi118.stdout: "desc": "how frequently to wake up and attempt optimization", 2024-03-24T03:17:23.942 INFO:teuthology.orchestra.run.smithi118.stdout: "long_desc": "", 2024-03-24T03:17:23.942 INFO:teuthology.orchestra.run.smithi118.stdout: "tags": [], 2024-03-24T03:17:23.942 INFO:teuthology.orchestra.run.smithi118.stdout: "see_also": [] 2024-03-24T03:17:23.942 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-24T03:17:23.943 INFO:teuthology.orchestra.run.smithi118.stdout: "upmap_max_deviation": { 2024-03-24T03:17:23.943 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "upmap_max_deviation", 2024-03-24T03:17:23.943 INFO:teuthology.orchestra.run.smithi118.stdout: "type": "int", 2024-03-24T03:17:23.943 INFO:teuthology.orchestra.run.smithi118.stdout: "level": "advanced", 2024-03-24T03:17:23.943 INFO:teuthology.orchestra.run.smithi118.stdout: "flags": 1, 2024-03-24T03:17:23.943 INFO:teuthology.orchestra.run.smithi118.stdout: "default_value": "5", 2024-03-24T03:17:23.943 INFO:teuthology.orchestra.run.smithi118.stdout: "min": "1", 2024-03-24T03:17:23.943 INFO:teuthology.orchestra.run.smithi118.stdout: "max": "", 2024-03-24T03:17:23.943 INFO:teuthology.orchestra.run.smithi118.stdout: "enum_allowed": [], 2024-03-24T03:17:23.944 INFO:teuthology.orchestra.run.smithi118.stdout: "desc": "deviation below which no optimization is attempted", 2024-03-24T03:17:23.944 INFO:teuthology.orchestra.run.smithi118.stdout: "long_desc": "If the number of PGs are within this count then no optimization is attempted", 2024-03-24T03:17:23.944 INFO:teuthology.orchestra.run.smithi118.stdout: "tags": [], 2024-03-24T03:17:23.944 INFO:teuthology.orchestra.run.smithi118.stdout: "see_also": [] 2024-03-24T03:17:23.944 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-24T03:17:23.944 INFO:teuthology.orchestra.run.smithi118.stdout: "upmap_max_optimizations": { 2024-03-24T03:17:23.944 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "upmap_max_optimizations", 2024-03-24T03:17:23.944 INFO:teuthology.orchestra.run.smithi118.stdout: "type": "uint", 2024-03-24T03:17:23.944 INFO:teuthology.orchestra.run.smithi118.stdout: "level": "advanced", 2024-03-24T03:17:23.945 INFO:teuthology.orchestra.run.smithi118.stdout: "flags": 1, 2024-03-24T03:17:23.945 INFO:teuthology.orchestra.run.smithi118.stdout: "default_value": "10", 2024-03-24T03:17:23.945 INFO:teuthology.orchestra.run.smithi118.stdout: "min": "", 2024-03-24T03:17:23.945 INFO:teuthology.orchestra.run.smithi118.stdout: "max": "", 2024-03-24T03:17:23.945 INFO:teuthology.orchestra.run.smithi118.stdout: "enum_allowed": [], 2024-03-24T03:17:23.945 INFO:teuthology.orchestra.run.smithi118.stdout: "desc": "maximum upmap optimizations to make per attempt", 2024-03-24T03:17:23.945 INFO:teuthology.orchestra.run.smithi118.stdout: "long_desc": "", 2024-03-24T03:17:23.945 INFO:teuthology.orchestra.run.smithi118.stdout: "tags": [], 2024-03-24T03:17:23.945 INFO:teuthology.orchestra.run.smithi118.stdout: "see_also": [] 2024-03-24T03:17:23.945 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-03-24T03:17:23.946 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-03-24T03:17:23.946 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-24T03:17:23.946 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-24T03:17:23.946 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "cephadm", 2024-03-24T03:17:23.946 INFO:teuthology.orchestra.run.smithi118.stdout: "can_run": true, 2024-03-24T03:17:23.946 INFO:teuthology.orchestra.run.smithi118.stdout: "error_string": "", 2024-03-24T03:17:23.946 INFO:teuthology.orchestra.run.smithi118.stdout: "module_options": { 2024-03-24T03:17:23.946 INFO:teuthology.orchestra.run.smithi118.stdout: "agent_down_multiplier": { 2024-03-24T03:17:23.947 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "agent_down_multiplier", 2024-03-24T03:17:23.947 INFO:teuthology.orchestra.run.smithi118.stdout: "type": "float", 2024-03-24T03:17:23.947 INFO:teuthology.orchestra.run.smithi118.stdout: "level": "advanced", 2024-03-24T03:17:23.947 INFO:teuthology.orchestra.run.smithi118.stdout: "flags": 0, 2024-03-24T03:17:23.947 INFO:teuthology.orchestra.run.smithi118.stdout: "default_value": "3.0", 2024-03-24T03:17:23.947 INFO:teuthology.orchestra.run.smithi118.stdout: "min": "", 2024-03-24T03:17:23.947 INFO:teuthology.orchestra.run.smithi118.stdout: "max": "", 2024-03-24T03:17:23.947 INFO:teuthology.orchestra.run.smithi118.stdout: "enum_allowed": [], 2024-03-24T03:17:23.947 INFO:teuthology.orchestra.run.smithi118.stdout: "desc": "Multiplied by agent refresh rate to calculate how long agent must not report before being marked down", 2024-03-24T03:17:23.948 INFO:teuthology.orchestra.run.smithi118.stdout: "long_desc": "", 2024-03-24T03:17:23.948 INFO:teuthology.orchestra.run.smithi118.stdout: "tags": [], 2024-03-24T03:17:23.948 INFO:teuthology.orchestra.run.smithi118.stdout: "see_also": [] 2024-03-24T03:17:23.948 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-24T03:17:23.948 INFO:teuthology.orchestra.run.smithi118.stdout: "agent_refresh_rate": { 2024-03-24T03:17:23.948 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "agent_refresh_rate", 2024-03-24T03:17:23.948 INFO:teuthology.orchestra.run.smithi118.stdout: "type": "secs", 2024-03-24T03:17:23.948 INFO:teuthology.orchestra.run.smithi118.stdout: "level": "advanced", 2024-03-24T03:17:23.949 INFO:teuthology.orchestra.run.smithi118.stdout: "flags": 0, 2024-03-24T03:17:23.949 INFO:teuthology.orchestra.run.smithi118.stdout: "default_value": "20", 2024-03-24T03:17:23.949 INFO:teuthology.orchestra.run.smithi118.stdout: "min": "", 2024-03-24T03:17:23.949 INFO:teuthology.orchestra.run.smithi118.stdout: "max": "", 2024-03-24T03:17:23.949 INFO:teuthology.orchestra.run.smithi118.stdout: "enum_allowed": [], 2024-03-24T03:17:23.949 INFO:teuthology.orchestra.run.smithi118.stdout: "desc": "How often agent on each host will try to gather and send metadata", 2024-03-24T03:17:23.949 INFO:teuthology.orchestra.run.smithi118.stdout: "long_desc": "", 2024-03-24T03:17:23.949 INFO:teuthology.orchestra.run.smithi118.stdout: "tags": [], 2024-03-24T03:17:23.949 INFO:teuthology.orchestra.run.smithi118.stdout: "see_also": [] 2024-03-24T03:17:23.950 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-24T03:17:23.950 INFO:teuthology.orchestra.run.smithi118.stdout: "agent_starting_port": { 2024-03-24T03:17:23.950 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "agent_starting_port", 2024-03-24T03:17:23.950 INFO:teuthology.orchestra.run.smithi118.stdout: "type": "int", 2024-03-24T03:17:23.950 INFO:teuthology.orchestra.run.smithi118.stdout: "level": "advanced", 2024-03-24T03:17:23.950 INFO:teuthology.orchestra.run.smithi118.stdout: "flags": 0, 2024-03-24T03:17:23.950 INFO:teuthology.orchestra.run.smithi118.stdout: "default_value": "4721", 2024-03-24T03:17:23.950 INFO:teuthology.orchestra.run.smithi118.stdout: "min": "", 2024-03-24T03:17:23.951 INFO:teuthology.orchestra.run.smithi118.stdout: "max": "", 2024-03-24T03:17:23.951 INFO:teuthology.orchestra.run.smithi118.stdout: "enum_allowed": [], 2024-03-24T03:17:23.951 INFO:teuthology.orchestra.run.smithi118.stdout: "desc": "First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)", 2024-03-24T03:17:23.951 INFO:teuthology.orchestra.run.smithi118.stdout: "long_desc": "", 2024-03-24T03:17:23.951 INFO:teuthology.orchestra.run.smithi118.stdout: "tags": [], 2024-03-24T03:17:23.951 INFO:teuthology.orchestra.run.smithi118.stdout: "see_also": [] 2024-03-24T03:17:23.951 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-24T03:17:23.951 INFO:teuthology.orchestra.run.smithi118.stdout: "allow_ptrace": { 2024-03-24T03:17:23.951 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "allow_ptrace", 2024-03-24T03:17:23.952 INFO:teuthology.orchestra.run.smithi118.stdout: "type": "bool", 2024-03-24T03:17:23.952 INFO:teuthology.orchestra.run.smithi118.stdout: "level": "advanced", 2024-03-24T03:17:23.952 INFO:teuthology.orchestra.run.smithi118.stdout: "flags": 0, 2024-03-24T03:17:23.952 INFO:teuthology.orchestra.run.smithi118.stdout: "default_value": "False", 2024-03-24T03:17:23.952 INFO:teuthology.orchestra.run.smithi118.stdout: "min": "", 2024-03-24T03:17:23.952 INFO:teuthology.orchestra.run.smithi118.stdout: "max": "", 2024-03-24T03:17:23.952 INFO:teuthology.orchestra.run.smithi118.stdout: "enum_allowed": [], 2024-03-24T03:17:23.952 INFO:teuthology.orchestra.run.smithi118.stdout: "desc": "allow SYS_PTRACE capability on ceph containers", 2024-03-24T03:17:23.952 INFO:teuthology.orchestra.run.smithi118.stdout: "long_desc": "The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.", 2024-03-24T03:17:23.952 INFO:teuthology.orchestra.run.smithi118.stdout: "tags": [], 2024-03-24T03:17:23.953 INFO:teuthology.orchestra.run.smithi118.stdout: "see_also": [] 2024-03-24T03:17:23.953 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-24T03:17:23.953 INFO:teuthology.orchestra.run.smithi118.stdout: "autotune_interval": { 2024-03-24T03:17:23.953 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "autotune_interval", 2024-03-24T03:17:23.953 INFO:teuthology.orchestra.run.smithi118.stdout: "type": "secs", 2024-03-24T03:17:23.953 INFO:teuthology.orchestra.run.smithi118.stdout: "level": "advanced", 2024-03-24T03:17:23.953 INFO:teuthology.orchestra.run.smithi118.stdout: "flags": 0, 2024-03-24T03:17:23.953 INFO:teuthology.orchestra.run.smithi118.stdout: "default_value": "600", 2024-03-24T03:17:23.953 INFO:teuthology.orchestra.run.smithi118.stdout: "min": "", 2024-03-24T03:17:23.954 INFO:teuthology.orchestra.run.smithi118.stdout: "max": "", 2024-03-24T03:17:23.954 INFO:teuthology.orchestra.run.smithi118.stdout: "enum_allowed": [], 2024-03-24T03:17:23.954 INFO:teuthology.orchestra.run.smithi118.stdout: "desc": "how frequently to autotune daemon memory", 2024-03-24T03:17:23.954 INFO:teuthology.orchestra.run.smithi118.stdout: "long_desc": "", 2024-03-24T03:17:23.954 INFO:teuthology.orchestra.run.smithi118.stdout: "tags": [], 2024-03-24T03:17:23.954 INFO:teuthology.orchestra.run.smithi118.stdout: "see_also": [] 2024-03-24T03:17:23.954 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-24T03:17:23.954 INFO:teuthology.orchestra.run.smithi118.stdout: "autotune_memory_target_ratio": { 2024-03-24T03:17:23.955 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "autotune_memory_target_ratio", 2024-03-24T03:17:23.955 INFO:teuthology.orchestra.run.smithi118.stdout: "type": "float", 2024-03-24T03:17:23.955 INFO:teuthology.orchestra.run.smithi118.stdout: "level": "advanced", 2024-03-24T03:17:23.955 INFO:teuthology.orchestra.run.smithi118.stdout: "flags": 0, 2024-03-24T03:17:23.955 INFO:teuthology.orchestra.run.smithi118.stdout: "default_value": "0.7", 2024-03-24T03:17:23.955 INFO:teuthology.orchestra.run.smithi118.stdout: "min": "", 2024-03-24T03:17:23.955 INFO:teuthology.orchestra.run.smithi118.stdout: "max": "", 2024-03-24T03:17:23.955 INFO:teuthology.orchestra.run.smithi118.stdout: "enum_allowed": [], 2024-03-24T03:17:23.956 INFO:teuthology.orchestra.run.smithi118.stdout: "desc": "ratio of total system memory to divide amongst autotuned daemons", 2024-03-24T03:17:23.956 INFO:teuthology.orchestra.run.smithi118.stdout: "long_desc": "", 2024-03-24T03:17:23.956 INFO:teuthology.orchestra.run.smithi118.stdout: "tags": [], 2024-03-24T03:17:23.956 INFO:teuthology.orchestra.run.smithi118.stdout: "see_also": [] 2024-03-24T03:17:23.956 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-24T03:17:23.956 INFO:teuthology.orchestra.run.smithi118.stdout: "cephadm_log_destination": { 2024-03-24T03:17:23.956 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "cephadm_log_destination", 2024-03-24T03:17:23.956 INFO:teuthology.orchestra.run.smithi118.stdout: "type": "str", 2024-03-24T03:17:23.957 INFO:teuthology.orchestra.run.smithi118.stdout: "level": "advanced", 2024-03-24T03:17:23.957 INFO:teuthology.orchestra.run.smithi118.stdout: "flags": 0, 2024-03-24T03:17:23.957 INFO:teuthology.orchestra.run.smithi118.stdout: "default_value": "", 2024-03-24T03:17:23.957 INFO:teuthology.orchestra.run.smithi118.stdout: "min": "", 2024-03-24T03:17:23.957 INFO:teuthology.orchestra.run.smithi118.stdout: "max": "", 2024-03-24T03:17:23.957 INFO:teuthology.orchestra.run.smithi118.stdout: "enum_allowed": [ 2024-03-24T03:17:23.957 INFO:teuthology.orchestra.run.smithi118.stdout: "file", 2024-03-24T03:17:23.957 INFO:teuthology.orchestra.run.smithi118.stdout: "file,syslog", 2024-03-24T03:17:23.958 INFO:teuthology.orchestra.run.smithi118.stdout: "syslog" 2024-03-24T03:17:23.958 INFO:teuthology.orchestra.run.smithi118.stdout: ], 2024-03-24T03:17:23.958 INFO:teuthology.orchestra.run.smithi118.stdout: "desc": "Destination for cephadm command's persistent logging", 2024-03-24T03:17:23.958 INFO:teuthology.orchestra.run.smithi118.stdout: "long_desc": "", 2024-03-24T03:17:23.958 INFO:teuthology.orchestra.run.smithi118.stdout: "tags": [], 2024-03-24T03:17:23.958 INFO:teuthology.orchestra.run.smithi118.stdout: "see_also": [] 2024-03-24T03:17:23.958 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-24T03:17:23.958 INFO:teuthology.orchestra.run.smithi118.stdout: "cgroups_split": { 2024-03-24T03:17:23.959 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "cgroups_split", 2024-03-24T03:17:23.959 INFO:teuthology.orchestra.run.smithi118.stdout: "type": "bool", 2024-03-24T03:17:23.959 INFO:teuthology.orchestra.run.smithi118.stdout: "level": "advanced", 2024-03-24T03:17:23.959 INFO:teuthology.orchestra.run.smithi118.stdout: "flags": 0, 2024-03-24T03:17:23.959 INFO:teuthology.orchestra.run.smithi118.stdout: "default_value": "True", 2024-03-24T03:17:23.959 INFO:teuthology.orchestra.run.smithi118.stdout: "min": "", 2024-03-24T03:17:23.959 INFO:teuthology.orchestra.run.smithi118.stdout: "max": "", 2024-03-24T03:17:23.959 INFO:teuthology.orchestra.run.smithi118.stdout: "enum_allowed": [], 2024-03-24T03:17:23.960 INFO:teuthology.orchestra.run.smithi118.stdout: "desc": "Pass --cgroups=split when cephadm creates containers (currently podman only)", 2024-03-24T03:17:23.960 INFO:teuthology.orchestra.run.smithi118.stdout: "long_desc": "", 2024-03-24T03:17:23.960 INFO:teuthology.orchestra.run.smithi118.stdout: "tags": [], 2024-03-24T03:17:23.960 INFO:teuthology.orchestra.run.smithi118.stdout: "see_also": [] 2024-03-24T03:17:23.960 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-24T03:17:23.960 INFO:teuthology.orchestra.run.smithi118.stdout: "config_checks_enabled": { 2024-03-24T03:17:23.960 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "config_checks_enabled", 2024-03-24T03:17:23.960 INFO:teuthology.orchestra.run.smithi118.stdout: "type": "bool", 2024-03-24T03:17:23.960 INFO:teuthology.orchestra.run.smithi118.stdout: "level": "advanced", 2024-03-24T03:17:23.961 INFO:teuthology.orchestra.run.smithi118.stdout: "flags": 0, 2024-03-24T03:17:23.961 INFO:teuthology.orchestra.run.smithi118.stdout: "default_value": "False", 2024-03-24T03:17:23.961 INFO:teuthology.orchestra.run.smithi118.stdout: "min": "", 2024-03-24T03:17:23.961 INFO:teuthology.orchestra.run.smithi118.stdout: "max": "", 2024-03-24T03:17:23.961 INFO:teuthology.orchestra.run.smithi118.stdout: "enum_allowed": [], 2024-03-24T03:17:23.961 INFO:teuthology.orchestra.run.smithi118.stdout: "desc": "Enable or disable the cephadm configuration analysis", 2024-03-24T03:17:23.961 INFO:teuthology.orchestra.run.smithi118.stdout: "long_desc": "", 2024-03-24T03:17:23.961 INFO:teuthology.orchestra.run.smithi118.stdout: "tags": [], 2024-03-24T03:17:23.961 INFO:teuthology.orchestra.run.smithi118.stdout: "see_also": [] 2024-03-24T03:17:23.962 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-24T03:17:23.962 INFO:teuthology.orchestra.run.smithi118.stdout: "config_dashboard": { 2024-03-24T03:17:23.962 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "config_dashboard", 2024-03-24T03:17:23.962 INFO:teuthology.orchestra.run.smithi118.stdout: "type": "bool", 2024-03-24T03:17:23.962 INFO:teuthology.orchestra.run.smithi118.stdout: "level": "advanced", 2024-03-24T03:17:23.962 INFO:teuthology.orchestra.run.smithi118.stdout: "flags": 0, 2024-03-24T03:17:23.962 INFO:teuthology.orchestra.run.smithi118.stdout: "default_value": "True", 2024-03-24T03:17:23.962 INFO:teuthology.orchestra.run.smithi118.stdout: "min": "", 2024-03-24T03:17:23.962 INFO:teuthology.orchestra.run.smithi118.stdout: "max": "", 2024-03-24T03:17:23.963 INFO:teuthology.orchestra.run.smithi118.stdout: "enum_allowed": [], 2024-03-24T03:17:23.963 INFO:teuthology.orchestra.run.smithi118.stdout: "desc": "manage configs like API endpoints in Dashboard.", 2024-03-24T03:17:23.963 INFO:teuthology.orchestra.run.smithi118.stdout: "long_desc": "", 2024-03-24T03:17:23.963 INFO:teuthology.orchestra.run.smithi118.stdout: "tags": [], 2024-03-24T03:17:23.963 INFO:teuthology.orchestra.run.smithi118.stdout: "see_also": [] 2024-03-24T03:17:23.963 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-24T03:17:23.963 INFO:teuthology.orchestra.run.smithi118.stdout: "container_image_alertmanager": { 2024-03-24T03:17:23.963 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "container_image_alertmanager", 2024-03-24T03:17:23.963 INFO:teuthology.orchestra.run.smithi118.stdout: "type": "str", 2024-03-24T03:17:23.964 INFO:teuthology.orchestra.run.smithi118.stdout: "level": "advanced", 2024-03-24T03:17:23.964 INFO:teuthology.orchestra.run.smithi118.stdout: "flags": 0, 2024-03-24T03:17:23.964 INFO:teuthology.orchestra.run.smithi118.stdout: "default_value": "quay.io/prometheus/alertmanager:v0.25.0", 2024-03-24T03:17:23.964 INFO:teuthology.orchestra.run.smithi118.stdout: "min": "", 2024-03-24T03:17:23.964 INFO:teuthology.orchestra.run.smithi118.stdout: "max": "", 2024-03-24T03:17:23.964 INFO:teuthology.orchestra.run.smithi118.stdout: "enum_allowed": [], 2024-03-24T03:17:23.964 INFO:teuthology.orchestra.run.smithi118.stdout: "desc": "Prometheus container image", 2024-03-24T03:17:23.964 INFO:teuthology.orchestra.run.smithi118.stdout: "long_desc": "", 2024-03-24T03:17:23.964 INFO:teuthology.orchestra.run.smithi118.stdout: "tags": [], 2024-03-24T03:17:23.964 INFO:teuthology.orchestra.run.smithi118.stdout: "see_also": [] 2024-03-24T03:17:23.965 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-24T03:17:23.965 INFO:teuthology.orchestra.run.smithi118.stdout: "container_image_base": { 2024-03-24T03:17:23.965 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "container_image_base", 2024-03-24T03:17:23.965 INFO:teuthology.orchestra.run.smithi118.stdout: "type": "str", 2024-03-24T03:17:23.965 INFO:teuthology.orchestra.run.smithi118.stdout: "level": "advanced", 2024-03-24T03:17:23.965 INFO:teuthology.orchestra.run.smithi118.stdout: "flags": 1, 2024-03-24T03:17:23.965 INFO:teuthology.orchestra.run.smithi118.stdout: "default_value": "quay.io/ceph/ceph", 2024-03-24T03:17:23.965 INFO:teuthology.orchestra.run.smithi118.stdout: "min": "", 2024-03-24T03:17:23.965 INFO:teuthology.orchestra.run.smithi118.stdout: "max": "", 2024-03-24T03:17:23.966 INFO:teuthology.orchestra.run.smithi118.stdout: "enum_allowed": [], 2024-03-24T03:17:23.966 INFO:teuthology.orchestra.run.smithi118.stdout: "desc": "Container image name, without the tag", 2024-03-24T03:17:23.966 INFO:teuthology.orchestra.run.smithi118.stdout: "long_desc": "", 2024-03-24T03:17:23.966 INFO:teuthology.orchestra.run.smithi118.stdout: "tags": [], 2024-03-24T03:17:23.966 INFO:teuthology.orchestra.run.smithi118.stdout: "see_also": [] 2024-03-24T03:17:23.966 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-24T03:17:23.966 INFO:teuthology.orchestra.run.smithi118.stdout: "container_image_elasticsearch": { 2024-03-24T03:17:23.966 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "container_image_elasticsearch", 2024-03-24T03:17:23.966 INFO:teuthology.orchestra.run.smithi118.stdout: "type": "str", 2024-03-24T03:17:23.967 INFO:teuthology.orchestra.run.smithi118.stdout: "level": "advanced", 2024-03-24T03:17:23.967 INFO:teuthology.orchestra.run.smithi118.stdout: "flags": 0, 2024-03-24T03:17:23.967 INFO:teuthology.orchestra.run.smithi118.stdout: "default_value": "quay.io/omrizeneva/elasticsearch:6.8.23", 2024-03-24T03:17:23.967 INFO:teuthology.orchestra.run.smithi118.stdout: "min": "", 2024-03-24T03:17:23.967 INFO:teuthology.orchestra.run.smithi118.stdout: "max": "", 2024-03-24T03:17:23.967 INFO:teuthology.orchestra.run.smithi118.stdout: "enum_allowed": [], 2024-03-24T03:17:23.967 INFO:teuthology.orchestra.run.smithi118.stdout: "desc": "elasticsearch container image", 2024-03-24T03:17:23.967 INFO:teuthology.orchestra.run.smithi118.stdout: "long_desc": "", 2024-03-24T03:17:23.967 INFO:teuthology.orchestra.run.smithi118.stdout: "tags": [], 2024-03-24T03:17:23.968 INFO:teuthology.orchestra.run.smithi118.stdout: "see_also": [] 2024-03-24T03:17:23.968 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-24T03:17:23.968 INFO:teuthology.orchestra.run.smithi118.stdout: "container_image_grafana": { 2024-03-24T03:17:23.968 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "container_image_grafana", 2024-03-24T03:17:23.968 INFO:teuthology.orchestra.run.smithi118.stdout: "type": "str", 2024-03-24T03:17:23.968 INFO:teuthology.orchestra.run.smithi118.stdout: "level": "advanced", 2024-03-24T03:17:23.968 INFO:teuthology.orchestra.run.smithi118.stdout: "flags": 0, 2024-03-24T03:17:23.968 INFO:teuthology.orchestra.run.smithi118.stdout: "default_value": "quay.io/ceph/ceph-grafana:9.4.12", 2024-03-24T03:17:23.968 INFO:teuthology.orchestra.run.smithi118.stdout: "min": "", 2024-03-24T03:17:23.968 INFO:teuthology.orchestra.run.smithi118.stdout: "max": "", 2024-03-24T03:17:23.969 INFO:teuthology.orchestra.run.smithi118.stdout: "enum_allowed": [], 2024-03-24T03:17:23.969 INFO:teuthology.orchestra.run.smithi118.stdout: "desc": "Prometheus container image", 2024-03-24T03:17:23.969 INFO:teuthology.orchestra.run.smithi118.stdout: "long_desc": "", 2024-03-24T03:17:23.969 INFO:teuthology.orchestra.run.smithi118.stdout: "tags": [], 2024-03-24T03:17:23.969 INFO:teuthology.orchestra.run.smithi118.stdout: "see_also": [] 2024-03-24T03:17:23.969 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-24T03:17:23.969 INFO:teuthology.orchestra.run.smithi118.stdout: "container_image_haproxy": { 2024-03-24T03:17:23.969 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "container_image_haproxy", 2024-03-24T03:17:23.969 INFO:teuthology.orchestra.run.smithi118.stdout: "type": "str", 2024-03-24T03:17:23.970 INFO:teuthology.orchestra.run.smithi118.stdout: "level": "advanced", 2024-03-24T03:17:23.970 INFO:teuthology.orchestra.run.smithi118.stdout: "flags": 0, 2024-03-24T03:17:23.970 INFO:teuthology.orchestra.run.smithi118.stdout: "default_value": "quay.io/ceph/haproxy:2.3", 2024-03-24T03:17:23.970 INFO:teuthology.orchestra.run.smithi118.stdout: "min": "", 2024-03-24T03:17:23.970 INFO:teuthology.orchestra.run.smithi118.stdout: "max": "", 2024-03-24T03:17:23.970 INFO:teuthology.orchestra.run.smithi118.stdout: "enum_allowed": [], 2024-03-24T03:17:23.970 INFO:teuthology.orchestra.run.smithi118.stdout: "desc": "HAproxy container image", 2024-03-24T03:17:23.970 INFO:teuthology.orchestra.run.smithi118.stdout: "long_desc": "", 2024-03-24T03:17:23.971 INFO:teuthology.orchestra.run.smithi118.stdout: "tags": [], 2024-03-24T03:17:23.971 INFO:teuthology.orchestra.run.smithi118.stdout: "see_also": [] 2024-03-24T03:17:23.971 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-24T03:17:23.971 INFO:teuthology.orchestra.run.smithi118.stdout: "container_image_jaeger_agent": { 2024-03-24T03:17:23.971 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "container_image_jaeger_agent", 2024-03-24T03:17:23.971 INFO:teuthology.orchestra.run.smithi118.stdout: "type": "str", 2024-03-24T03:17:23.971 INFO:teuthology.orchestra.run.smithi118.stdout: "level": "advanced", 2024-03-24T03:17:23.971 INFO:teuthology.orchestra.run.smithi118.stdout: "flags": 0, 2024-03-24T03:17:23.971 INFO:teuthology.orchestra.run.smithi118.stdout: "default_value": "quay.io/jaegertracing/jaeger-agent:1.29", 2024-03-24T03:17:23.972 INFO:teuthology.orchestra.run.smithi118.stdout: "min": "", 2024-03-24T03:17:23.972 INFO:teuthology.orchestra.run.smithi118.stdout: "max": "", 2024-03-24T03:17:23.972 INFO:teuthology.orchestra.run.smithi118.stdout: "enum_allowed": [], 2024-03-24T03:17:23.972 INFO:teuthology.orchestra.run.smithi118.stdout: "desc": "Jaeger agent container image", 2024-03-24T03:17:23.972 INFO:teuthology.orchestra.run.smithi118.stdout: "long_desc": "", 2024-03-24T03:17:23.972 INFO:teuthology.orchestra.run.smithi118.stdout: "tags": [], 2024-03-24T03:17:23.972 INFO:teuthology.orchestra.run.smithi118.stdout: "see_also": [] 2024-03-24T03:17:23.972 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-24T03:17:23.972 INFO:teuthology.orchestra.run.smithi118.stdout: "container_image_jaeger_collector": { 2024-03-24T03:17:23.973 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "container_image_jaeger_collector", 2024-03-24T03:17:23.973 INFO:teuthology.orchestra.run.smithi118.stdout: "type": "str", 2024-03-24T03:17:23.973 INFO:teuthology.orchestra.run.smithi118.stdout: "level": "advanced", 2024-03-24T03:17:23.973 INFO:teuthology.orchestra.run.smithi118.stdout: "flags": 0, 2024-03-24T03:17:23.973 INFO:teuthology.orchestra.run.smithi118.stdout: "default_value": "quay.io/jaegertracing/jaeger-collector:1.29", 2024-03-24T03:17:23.973 INFO:teuthology.orchestra.run.smithi118.stdout: "min": "", 2024-03-24T03:17:23.973 INFO:teuthology.orchestra.run.smithi118.stdout: "max": "", 2024-03-24T03:17:23.973 INFO:teuthology.orchestra.run.smithi118.stdout: "enum_allowed": [], 2024-03-24T03:17:23.973 INFO:teuthology.orchestra.run.smithi118.stdout: "desc": "Jaeger collector container image", 2024-03-24T03:17:23.973 INFO:teuthology.orchestra.run.smithi118.stdout: "long_desc": "", 2024-03-24T03:17:23.974 INFO:teuthology.orchestra.run.smithi118.stdout: "tags": [], 2024-03-24T03:17:23.974 INFO:teuthology.orchestra.run.smithi118.stdout: "see_also": [] 2024-03-24T03:17:23.974 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-24T03:17:23.974 INFO:teuthology.orchestra.run.smithi118.stdout: "container_image_jaeger_query": { 2024-03-24T03:17:23.974 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "container_image_jaeger_query", 2024-03-24T03:17:23.974 INFO:teuthology.orchestra.run.smithi118.stdout: "type": "str", 2024-03-24T03:17:23.974 INFO:teuthology.orchestra.run.smithi118.stdout: "level": "advanced", 2024-03-24T03:17:23.974 INFO:teuthology.orchestra.run.smithi118.stdout: "flags": 0, 2024-03-24T03:17:23.974 INFO:teuthology.orchestra.run.smithi118.stdout: "default_value": "quay.io/jaegertracing/jaeger-query:1.29", 2024-03-24T03:17:23.975 INFO:teuthology.orchestra.run.smithi118.stdout: "min": "", 2024-03-24T03:17:23.975 INFO:teuthology.orchestra.run.smithi118.stdout: "max": "", 2024-03-24T03:17:23.975 INFO:teuthology.orchestra.run.smithi118.stdout: "enum_allowed": [], 2024-03-24T03:17:23.975 INFO:teuthology.orchestra.run.smithi118.stdout: "desc": "Jaeger query container image", 2024-03-24T03:17:23.975 INFO:teuthology.orchestra.run.smithi118.stdout: "long_desc": "", 2024-03-24T03:17:23.975 INFO:teuthology.orchestra.run.smithi118.stdout: "tags": [], 2024-03-24T03:17:23.975 INFO:teuthology.orchestra.run.smithi118.stdout: "see_also": [] 2024-03-24T03:17:23.975 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-24T03:17:23.976 INFO:teuthology.orchestra.run.smithi118.stdout: "container_image_keepalived": { 2024-03-24T03:17:23.976 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "container_image_keepalived", 2024-03-24T03:17:23.976 INFO:teuthology.orchestra.run.smithi118.stdout: "type": "str", 2024-03-24T03:17:23.976 INFO:teuthology.orchestra.run.smithi118.stdout: "level": "advanced", 2024-03-24T03:17:23.976 INFO:teuthology.orchestra.run.smithi118.stdout: "flags": 0, 2024-03-24T03:17:23.976 INFO:teuthology.orchestra.run.smithi118.stdout: "default_value": "quay.io/ceph/keepalived:2.2.4", 2024-03-24T03:17:23.976 INFO:teuthology.orchestra.run.smithi118.stdout: "min": "", 2024-03-24T03:17:23.976 INFO:teuthology.orchestra.run.smithi118.stdout: "max": "", 2024-03-24T03:17:23.976 INFO:teuthology.orchestra.run.smithi118.stdout: "enum_allowed": [], 2024-03-24T03:17:23.976 INFO:teuthology.orchestra.run.smithi118.stdout: "desc": "Keepalived container image", 2024-03-24T03:17:23.977 INFO:teuthology.orchestra.run.smithi118.stdout: "long_desc": "", 2024-03-24T03:17:23.977 INFO:teuthology.orchestra.run.smithi118.stdout: "tags": [], 2024-03-24T03:17:23.977 INFO:teuthology.orchestra.run.smithi118.stdout: "see_also": [] 2024-03-24T03:17:23.977 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-24T03:17:23.977 INFO:teuthology.orchestra.run.smithi118.stdout: "container_image_loki": { 2024-03-24T03:17:23.977 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "container_image_loki", 2024-03-24T03:17:23.977 INFO:teuthology.orchestra.run.smithi118.stdout: "type": "str", 2024-03-24T03:17:23.977 INFO:teuthology.orchestra.run.smithi118.stdout: "level": "advanced", 2024-03-24T03:17:23.977 INFO:teuthology.orchestra.run.smithi118.stdout: "flags": 0, 2024-03-24T03:17:23.978 INFO:teuthology.orchestra.run.smithi118.stdout: "default_value": "docker.io/grafana/loki:2.4.0", 2024-03-24T03:17:23.978 INFO:teuthology.orchestra.run.smithi118.stdout: "min": "", 2024-03-24T03:17:23.978 INFO:teuthology.orchestra.run.smithi118.stdout: "max": "", 2024-03-24T03:17:23.978 INFO:teuthology.orchestra.run.smithi118.stdout: "enum_allowed": [], 2024-03-24T03:17:23.978 INFO:teuthology.orchestra.run.smithi118.stdout: "desc": "Loki container image", 2024-03-24T03:17:23.978 INFO:teuthology.orchestra.run.smithi118.stdout: "long_desc": "", 2024-03-24T03:17:23.978 INFO:teuthology.orchestra.run.smithi118.stdout: "tags": [], 2024-03-24T03:17:23.978 INFO:teuthology.orchestra.run.smithi118.stdout: "see_also": [] 2024-03-24T03:17:23.978 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-24T03:17:23.979 INFO:teuthology.orchestra.run.smithi118.stdout: "container_image_node_exporter": { 2024-03-24T03:17:23.979 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "container_image_node_exporter", 2024-03-24T03:17:23.979 INFO:teuthology.orchestra.run.smithi118.stdout: "type": "str", 2024-03-24T03:17:23.979 INFO:teuthology.orchestra.run.smithi118.stdout: "level": "advanced", 2024-03-24T03:17:23.979 INFO:teuthology.orchestra.run.smithi118.stdout: "flags": 0, 2024-03-24T03:17:23.979 INFO:teuthology.orchestra.run.smithi118.stdout: "default_value": "quay.io/prometheus/node-exporter:v1.5.0", 2024-03-24T03:17:23.979 INFO:teuthology.orchestra.run.smithi118.stdout: "min": "", 2024-03-24T03:17:23.979 INFO:teuthology.orchestra.run.smithi118.stdout: "max": "", 2024-03-24T03:17:23.979 INFO:teuthology.orchestra.run.smithi118.stdout: "enum_allowed": [], 2024-03-24T03:17:23.980 INFO:teuthology.orchestra.run.smithi118.stdout: "desc": "Prometheus container image", 2024-03-24T03:17:23.980 INFO:teuthology.orchestra.run.smithi118.stdout: "long_desc": "", 2024-03-24T03:17:23.980 INFO:teuthology.orchestra.run.smithi118.stdout: "tags": [], 2024-03-24T03:17:23.980 INFO:teuthology.orchestra.run.smithi118.stdout: "see_also": [] 2024-03-24T03:17:23.980 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-24T03:17:23.980 INFO:teuthology.orchestra.run.smithi118.stdout: "container_image_nvmeof": { 2024-03-24T03:17:23.980 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "container_image_nvmeof", 2024-03-24T03:17:23.980 INFO:teuthology.orchestra.run.smithi118.stdout: "type": "str", 2024-03-24T03:17:23.980 INFO:teuthology.orchestra.run.smithi118.stdout: "level": "advanced", 2024-03-24T03:17:23.981 INFO:teuthology.orchestra.run.smithi118.stdout: "flags": 0, 2024-03-24T03:17:23.981 INFO:teuthology.orchestra.run.smithi118.stdout: "default_value": "quay.io/ceph/nvmeof:1.0.0", 2024-03-24T03:17:23.981 INFO:teuthology.orchestra.run.smithi118.stdout: "min": "", 2024-03-24T03:17:23.981 INFO:teuthology.orchestra.run.smithi118.stdout: "max": "", 2024-03-24T03:17:23.981 INFO:teuthology.orchestra.run.smithi118.stdout: "enum_allowed": [], 2024-03-24T03:17:23.981 INFO:teuthology.orchestra.run.smithi118.stdout: "desc": "Nvme-of container image", 2024-03-24T03:17:23.981 INFO:teuthology.orchestra.run.smithi118.stdout: "long_desc": "", 2024-03-24T03:17:23.981 INFO:teuthology.orchestra.run.smithi118.stdout: "tags": [], 2024-03-24T03:17:23.981 INFO:teuthology.orchestra.run.smithi118.stdout: "see_also": [] 2024-03-24T03:17:23.981 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-24T03:17:23.982 INFO:teuthology.orchestra.run.smithi118.stdout: "container_image_prometheus": { 2024-03-24T03:17:23.982 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "container_image_prometheus", 2024-03-24T03:17:23.982 INFO:teuthology.orchestra.run.smithi118.stdout: "type": "str", 2024-03-24T03:17:23.982 INFO:teuthology.orchestra.run.smithi118.stdout: "level": "advanced", 2024-03-24T03:17:23.982 INFO:teuthology.orchestra.run.smithi118.stdout: "flags": 0, 2024-03-24T03:17:23.982 INFO:teuthology.orchestra.run.smithi118.stdout: "default_value": "quay.io/prometheus/prometheus:v2.43.0", 2024-03-24T03:17:23.982 INFO:teuthology.orchestra.run.smithi118.stdout: "min": "", 2024-03-24T03:17:23.982 INFO:teuthology.orchestra.run.smithi118.stdout: "max": "", 2024-03-24T03:17:23.982 INFO:teuthology.orchestra.run.smithi118.stdout: "enum_allowed": [], 2024-03-24T03:17:23.983 INFO:teuthology.orchestra.run.smithi118.stdout: "desc": "Prometheus container image", 2024-03-24T03:17:23.983 INFO:teuthology.orchestra.run.smithi118.stdout: "long_desc": "", 2024-03-24T03:17:23.983 INFO:teuthology.orchestra.run.smithi118.stdout: "tags": [], 2024-03-24T03:17:23.983 INFO:teuthology.orchestra.run.smithi118.stdout: "see_also": [] 2024-03-24T03:17:23.983 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-24T03:17:23.983 INFO:teuthology.orchestra.run.smithi118.stdout: "container_image_promtail": { 2024-03-24T03:17:23.983 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "container_image_promtail", 2024-03-24T03:17:23.983 INFO:teuthology.orchestra.run.smithi118.stdout: "type": "str", 2024-03-24T03:17:23.983 INFO:teuthology.orchestra.run.smithi118.stdout: "level": "advanced", 2024-03-24T03:17:23.984 INFO:teuthology.orchestra.run.smithi118.stdout: "flags": 0, 2024-03-24T03:17:23.984 INFO:teuthology.orchestra.run.smithi118.stdout: "default_value": "docker.io/grafana/promtail:2.4.0", 2024-03-24T03:17:23.984 INFO:teuthology.orchestra.run.smithi118.stdout: "min": "", 2024-03-24T03:17:23.984 INFO:teuthology.orchestra.run.smithi118.stdout: "max": "", 2024-03-24T03:17:23.984 INFO:teuthology.orchestra.run.smithi118.stdout: "enum_allowed": [], 2024-03-24T03:17:23.984 INFO:teuthology.orchestra.run.smithi118.stdout: "desc": "Promtail container image", 2024-03-24T03:17:23.984 INFO:teuthology.orchestra.run.smithi118.stdout: "long_desc": "", 2024-03-24T03:17:23.984 INFO:teuthology.orchestra.run.smithi118.stdout: "tags": [], 2024-03-24T03:17:23.984 INFO:teuthology.orchestra.run.smithi118.stdout: "see_also": [] 2024-03-24T03:17:23.984 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-24T03:17:23.985 INFO:teuthology.orchestra.run.smithi118.stdout: "container_image_snmp_gateway": { 2024-03-24T03:17:23.985 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "container_image_snmp_gateway", 2024-03-24T03:17:23.985 INFO:teuthology.orchestra.run.smithi118.stdout: "type": "str", 2024-03-24T03:17:23.985 INFO:teuthology.orchestra.run.smithi118.stdout: "level": "advanced", 2024-03-24T03:17:23.985 INFO:teuthology.orchestra.run.smithi118.stdout: "flags": 0, 2024-03-24T03:17:23.985 INFO:teuthology.orchestra.run.smithi118.stdout: "default_value": "docker.io/maxwo/snmp-notifier:v1.2.1", 2024-03-24T03:17:23.985 INFO:teuthology.orchestra.run.smithi118.stdout: "min": "", 2024-03-24T03:17:23.985 INFO:teuthology.orchestra.run.smithi118.stdout: "max": "", 2024-03-24T03:17:23.985 INFO:teuthology.orchestra.run.smithi118.stdout: "enum_allowed": [], 2024-03-24T03:17:23.985 INFO:teuthology.orchestra.run.smithi118.stdout: "desc": "SNMP Gateway container image", 2024-03-24T03:17:23.986 INFO:teuthology.orchestra.run.smithi118.stdout: "long_desc": "", 2024-03-24T03:17:23.986 INFO:teuthology.orchestra.run.smithi118.stdout: "tags": [], 2024-03-24T03:17:23.986 INFO:teuthology.orchestra.run.smithi118.stdout: "see_also": [] 2024-03-24T03:17:23.986 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-24T03:17:23.986 INFO:teuthology.orchestra.run.smithi118.stdout: "container_init": { 2024-03-24T03:17:23.986 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "container_init", 2024-03-24T03:17:23.986 INFO:teuthology.orchestra.run.smithi118.stdout: "type": "bool", 2024-03-24T03:17:23.986 INFO:teuthology.orchestra.run.smithi118.stdout: "level": "advanced", 2024-03-24T03:17:23.986 INFO:teuthology.orchestra.run.smithi118.stdout: "flags": 0, 2024-03-24T03:17:23.987 INFO:teuthology.orchestra.run.smithi118.stdout: "default_value": "True", 2024-03-24T03:17:23.987 INFO:teuthology.orchestra.run.smithi118.stdout: "min": "", 2024-03-24T03:17:23.989 INFO:teuthology.orchestra.run.smithi118.stdout: "max": "", 2024-03-24T03:17:23.989 INFO:teuthology.orchestra.run.smithi118.stdout: "enum_allowed": [], 2024-03-24T03:17:23.989 INFO:teuthology.orchestra.run.smithi118.stdout: "desc": "Run podman/docker with `--init`", 2024-03-24T03:17:23.990 INFO:teuthology.orchestra.run.smithi118.stdout: "long_desc": "", 2024-03-24T03:17:23.990 INFO:teuthology.orchestra.run.smithi118.stdout: "tags": [], 2024-03-24T03:17:23.990 INFO:teuthology.orchestra.run.smithi118.stdout: "see_also": [] 2024-03-24T03:17:23.990 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-24T03:17:23.990 INFO:teuthology.orchestra.run.smithi118.stdout: "daemon_cache_timeout": { 2024-03-24T03:17:23.990 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "daemon_cache_timeout", 2024-03-24T03:17:23.990 INFO:teuthology.orchestra.run.smithi118.stdout: "type": "secs", 2024-03-24T03:17:23.990 INFO:teuthology.orchestra.run.smithi118.stdout: "level": "advanced", 2024-03-24T03:17:23.990 INFO:teuthology.orchestra.run.smithi118.stdout: "flags": 0, 2024-03-24T03:17:23.991 INFO:teuthology.orchestra.run.smithi118.stdout: "default_value": "600", 2024-03-24T03:17:23.991 INFO:teuthology.orchestra.run.smithi118.stdout: "min": "", 2024-03-24T03:17:23.991 INFO:teuthology.orchestra.run.smithi118.stdout: "max": "", 2024-03-24T03:17:23.991 INFO:teuthology.orchestra.run.smithi118.stdout: "enum_allowed": [], 2024-03-24T03:17:23.991 INFO:teuthology.orchestra.run.smithi118.stdout: "desc": "seconds to cache service (daemon) inventory", 2024-03-24T03:17:23.991 INFO:teuthology.orchestra.run.smithi118.stdout: "long_desc": "", 2024-03-24T03:17:23.991 INFO:teuthology.orchestra.run.smithi118.stdout: "tags": [], 2024-03-24T03:17:23.991 INFO:teuthology.orchestra.run.smithi118.stdout: "see_also": [] 2024-03-24T03:17:23.991 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-24T03:17:23.992 INFO:teuthology.orchestra.run.smithi118.stdout: "default_cephadm_command_timeout": { 2024-03-24T03:17:23.992 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "default_cephadm_command_timeout", 2024-03-24T03:17:23.992 INFO:teuthology.orchestra.run.smithi118.stdout: "type": "int", 2024-03-24T03:17:23.992 INFO:teuthology.orchestra.run.smithi118.stdout: "level": "advanced", 2024-03-24T03:17:23.992 INFO:teuthology.orchestra.run.smithi118.stdout: "flags": 0, 2024-03-24T03:17:23.992 INFO:teuthology.orchestra.run.smithi118.stdout: "default_value": "900", 2024-03-24T03:17:23.992 INFO:teuthology.orchestra.run.smithi118.stdout: "min": "", 2024-03-24T03:17:23.992 INFO:teuthology.orchestra.run.smithi118.stdout: "max": "", 2024-03-24T03:17:23.992 INFO:teuthology.orchestra.run.smithi118.stdout: "enum_allowed": [], 2024-03-24T03:17:23.993 INFO:teuthology.orchestra.run.smithi118.stdout: "desc": "Default timeout applied to cephadm commands run directly on the host (in seconds)", 2024-03-24T03:17:23.993 INFO:teuthology.orchestra.run.smithi118.stdout: "long_desc": "", 2024-03-24T03:17:23.993 INFO:teuthology.orchestra.run.smithi118.stdout: "tags": [], 2024-03-24T03:17:23.993 INFO:teuthology.orchestra.run.smithi118.stdout: "see_also": [] 2024-03-24T03:17:23.993 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-24T03:17:23.993 INFO:teuthology.orchestra.run.smithi118.stdout: "default_registry": { 2024-03-24T03:17:23.993 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "default_registry", 2024-03-24T03:17:23.993 INFO:teuthology.orchestra.run.smithi118.stdout: "type": "str", 2024-03-24T03:17:23.993 INFO:teuthology.orchestra.run.smithi118.stdout: "level": "advanced", 2024-03-24T03:17:23.994 INFO:teuthology.orchestra.run.smithi118.stdout: "flags": 0, 2024-03-24T03:17:23.994 INFO:teuthology.orchestra.run.smithi118.stdout: "default_value": "docker.io", 2024-03-24T03:17:23.994 INFO:teuthology.orchestra.run.smithi118.stdout: "min": "", 2024-03-24T03:17:23.994 INFO:teuthology.orchestra.run.smithi118.stdout: "max": "", 2024-03-24T03:17:23.994 INFO:teuthology.orchestra.run.smithi118.stdout: "enum_allowed": [], 2024-03-24T03:17:23.994 INFO:teuthology.orchestra.run.smithi118.stdout: "desc": "Search-registry to which we should normalize unqualified image names. This is not the default registry", 2024-03-24T03:17:23.994 INFO:teuthology.orchestra.run.smithi118.stdout: "long_desc": "", 2024-03-24T03:17:23.994 INFO:teuthology.orchestra.run.smithi118.stdout: "tags": [], 2024-03-24T03:17:23.994 INFO:teuthology.orchestra.run.smithi118.stdout: "see_also": [] 2024-03-24T03:17:23.995 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-24T03:17:23.995 INFO:teuthology.orchestra.run.smithi118.stdout: "device_cache_timeout": { 2024-03-24T03:17:23.995 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "device_cache_timeout", 2024-03-24T03:17:23.995 INFO:teuthology.orchestra.run.smithi118.stdout: "type": "secs", 2024-03-24T03:17:23.995 INFO:teuthology.orchestra.run.smithi118.stdout: "level": "advanced", 2024-03-24T03:17:23.995 INFO:teuthology.orchestra.run.smithi118.stdout: "flags": 0, 2024-03-24T03:17:23.995 INFO:teuthology.orchestra.run.smithi118.stdout: "default_value": "1800", 2024-03-24T03:17:23.995 INFO:teuthology.orchestra.run.smithi118.stdout: "min": "", 2024-03-24T03:17:23.995 INFO:teuthology.orchestra.run.smithi118.stdout: "max": "", 2024-03-24T03:17:23.996 INFO:teuthology.orchestra.run.smithi118.stdout: "enum_allowed": [], 2024-03-24T03:17:23.996 INFO:teuthology.orchestra.run.smithi118.stdout: "desc": "seconds to cache device inventory", 2024-03-24T03:17:23.996 INFO:teuthology.orchestra.run.smithi118.stdout: "long_desc": "", 2024-03-24T03:17:23.996 INFO:teuthology.orchestra.run.smithi118.stdout: "tags": [], 2024-03-24T03:17:23.996 INFO:teuthology.orchestra.run.smithi118.stdout: "see_also": [] 2024-03-24T03:17:23.996 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-24T03:17:23.996 INFO:teuthology.orchestra.run.smithi118.stdout: "device_enhanced_scan": { 2024-03-24T03:17:23.996 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "device_enhanced_scan", 2024-03-24T03:17:23.996 INFO:teuthology.orchestra.run.smithi118.stdout: "type": "bool", 2024-03-24T03:17:23.996 INFO:teuthology.orchestra.run.smithi118.stdout: "level": "advanced", 2024-03-24T03:17:23.997 INFO:teuthology.orchestra.run.smithi118.stdout: "flags": 0, 2024-03-24T03:17:23.997 INFO:teuthology.orchestra.run.smithi118.stdout: "default_value": "False", 2024-03-24T03:17:23.997 INFO:teuthology.orchestra.run.smithi118.stdout: "min": "", 2024-03-24T03:17:23.997 INFO:teuthology.orchestra.run.smithi118.stdout: "max": "", 2024-03-24T03:17:23.997 INFO:teuthology.orchestra.run.smithi118.stdout: "enum_allowed": [], 2024-03-24T03:17:23.997 INFO:teuthology.orchestra.run.smithi118.stdout: "desc": "Use libstoragemgmt during device scans", 2024-03-24T03:17:23.997 INFO:teuthology.orchestra.run.smithi118.stdout: "long_desc": "", 2024-03-24T03:17:23.997 INFO:teuthology.orchestra.run.smithi118.stdout: "tags": [], 2024-03-24T03:17:23.997 INFO:teuthology.orchestra.run.smithi118.stdout: "see_also": [] 2024-03-24T03:17:23.998 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-24T03:17:23.998 INFO:teuthology.orchestra.run.smithi118.stdout: "facts_cache_timeout": { 2024-03-24T03:17:23.998 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "facts_cache_timeout", 2024-03-24T03:17:23.998 INFO:teuthology.orchestra.run.smithi118.stdout: "type": "secs", 2024-03-24T03:17:23.998 INFO:teuthology.orchestra.run.smithi118.stdout: "level": "advanced", 2024-03-24T03:17:23.998 INFO:teuthology.orchestra.run.smithi118.stdout: "flags": 0, 2024-03-24T03:17:23.998 INFO:teuthology.orchestra.run.smithi118.stdout: "default_value": "60", 2024-03-24T03:17:23.998 INFO:teuthology.orchestra.run.smithi118.stdout: "min": "", 2024-03-24T03:17:23.998 INFO:teuthology.orchestra.run.smithi118.stdout: "max": "", 2024-03-24T03:17:23.999 INFO:teuthology.orchestra.run.smithi118.stdout: "enum_allowed": [], 2024-03-24T03:17:23.999 INFO:teuthology.orchestra.run.smithi118.stdout: "desc": "seconds to cache host facts data", 2024-03-24T03:17:23.999 INFO:teuthology.orchestra.run.smithi118.stdout: "long_desc": "", 2024-03-24T03:17:23.999 INFO:teuthology.orchestra.run.smithi118.stdout: "tags": [], 2024-03-24T03:17:23.999 INFO:teuthology.orchestra.run.smithi118.stdout: "see_also": [] 2024-03-24T03:17:23.999 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-24T03:17:23.999 INFO:teuthology.orchestra.run.smithi118.stdout: "host_check_interval": { 2024-03-24T03:17:23.999 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "host_check_interval", 2024-03-24T03:17:23.999 INFO:teuthology.orchestra.run.smithi118.stdout: "type": "secs", 2024-03-24T03:17:23.999 INFO:teuthology.orchestra.run.smithi118.stdout: "level": "advanced", 2024-03-24T03:17:24.000 INFO:teuthology.orchestra.run.smithi118.stdout: "flags": 0, 2024-03-24T03:17:24.000 INFO:teuthology.orchestra.run.smithi118.stdout: "default_value": "600", 2024-03-24T03:17:24.000 INFO:teuthology.orchestra.run.smithi118.stdout: "min": "", 2024-03-24T03:17:24.000 INFO:teuthology.orchestra.run.smithi118.stdout: "max": "", 2024-03-24T03:17:24.000 INFO:teuthology.orchestra.run.smithi118.stdout: "enum_allowed": [], 2024-03-24T03:17:24.000 INFO:teuthology.orchestra.run.smithi118.stdout: "desc": "how frequently to perform a host check", 2024-03-24T03:17:24.000 INFO:teuthology.orchestra.run.smithi118.stdout: "long_desc": "", 2024-03-24T03:17:24.000 INFO:teuthology.orchestra.run.smithi118.stdout: "tags": [], 2024-03-24T03:17:24.000 INFO:teuthology.orchestra.run.smithi118.stdout: "see_also": [] 2024-03-24T03:17:24.001 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-24T03:17:24.001 INFO:teuthology.orchestra.run.smithi118.stdout: "hw_monitoring": { 2024-03-24T03:17:24.001 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "hw_monitoring", 2024-03-24T03:17:24.001 INFO:teuthology.orchestra.run.smithi118.stdout: "type": "bool", 2024-03-24T03:17:24.001 INFO:teuthology.orchestra.run.smithi118.stdout: "level": "advanced", 2024-03-24T03:17:24.001 INFO:teuthology.orchestra.run.smithi118.stdout: "flags": 0, 2024-03-24T03:17:24.001 INFO:teuthology.orchestra.run.smithi118.stdout: "default_value": "False", 2024-03-24T03:17:24.001 INFO:teuthology.orchestra.run.smithi118.stdout: "min": "", 2024-03-24T03:17:24.001 INFO:teuthology.orchestra.run.smithi118.stdout: "max": "", 2024-03-24T03:17:24.001 INFO:teuthology.orchestra.run.smithi118.stdout: "enum_allowed": [], 2024-03-24T03:17:24.002 INFO:teuthology.orchestra.run.smithi118.stdout: "desc": "Deploy hw monitoring daemon on every host.", 2024-03-24T03:17:24.002 INFO:teuthology.orchestra.run.smithi118.stdout: "long_desc": "", 2024-03-24T03:17:24.002 INFO:teuthology.orchestra.run.smithi118.stdout: "tags": [], 2024-03-24T03:17:24.002 INFO:teuthology.orchestra.run.smithi118.stdout: "see_also": [] 2024-03-24T03:17:24.002 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-24T03:17:24.002 INFO:teuthology.orchestra.run.smithi118.stdout: "inventory_list_all": { 2024-03-24T03:17:24.002 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "inventory_list_all", 2024-03-24T03:17:24.002 INFO:teuthology.orchestra.run.smithi118.stdout: "type": "bool", 2024-03-24T03:17:24.002 INFO:teuthology.orchestra.run.smithi118.stdout: "level": "advanced", 2024-03-24T03:17:24.003 INFO:teuthology.orchestra.run.smithi118.stdout: "flags": 0, 2024-03-24T03:17:24.003 INFO:teuthology.orchestra.run.smithi118.stdout: "default_value": "False", 2024-03-24T03:17:24.003 INFO:teuthology.orchestra.run.smithi118.stdout: "min": "", 2024-03-24T03:17:24.003 INFO:teuthology.orchestra.run.smithi118.stdout: "max": "", 2024-03-24T03:17:24.003 INFO:teuthology.orchestra.run.smithi118.stdout: "enum_allowed": [], 2024-03-24T03:17:24.003 INFO:teuthology.orchestra.run.smithi118.stdout: "desc": "Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)", 2024-03-24T03:17:24.003 INFO:teuthology.orchestra.run.smithi118.stdout: "long_desc": "", 2024-03-24T03:17:24.003 INFO:teuthology.orchestra.run.smithi118.stdout: "tags": [], 2024-03-24T03:17:24.003 INFO:teuthology.orchestra.run.smithi118.stdout: "see_also": [] 2024-03-24T03:17:24.004 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-24T03:17:24.004 INFO:teuthology.orchestra.run.smithi118.stdout: "log_level": { 2024-03-24T03:17:24.004 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "log_level", 2024-03-24T03:17:24.004 INFO:teuthology.orchestra.run.smithi118.stdout: "type": "str", 2024-03-24T03:17:24.004 INFO:teuthology.orchestra.run.smithi118.stdout: "level": "advanced", 2024-03-24T03:17:24.004 INFO:teuthology.orchestra.run.smithi118.stdout: "flags": 1, 2024-03-24T03:17:24.004 INFO:teuthology.orchestra.run.smithi118.stdout: "default_value": "", 2024-03-24T03:17:24.004 INFO:teuthology.orchestra.run.smithi118.stdout: "min": "", 2024-03-24T03:17:24.004 INFO:teuthology.orchestra.run.smithi118.stdout: "max": "", 2024-03-24T03:17:24.005 INFO:teuthology.orchestra.run.smithi118.stdout: "enum_allowed": [ 2024-03-24T03:17:24.005 INFO:teuthology.orchestra.run.smithi118.stdout: "", 2024-03-24T03:17:24.005 INFO:teuthology.orchestra.run.smithi118.stdout: "critical", 2024-03-24T03:17:24.005 INFO:teuthology.orchestra.run.smithi118.stdout: "debug", 2024-03-24T03:17:24.005 INFO:teuthology.orchestra.run.smithi118.stdout: "error", 2024-03-24T03:17:24.005 INFO:teuthology.orchestra.run.smithi118.stdout: "info", 2024-03-24T03:17:24.005 INFO:teuthology.orchestra.run.smithi118.stdout: "warning" 2024-03-24T03:17:24.005 INFO:teuthology.orchestra.run.smithi118.stdout: ], 2024-03-24T03:17:24.005 INFO:teuthology.orchestra.run.smithi118.stdout: "desc": "", 2024-03-24T03:17:24.006 INFO:teuthology.orchestra.run.smithi118.stdout: "long_desc": "", 2024-03-24T03:17:24.006 INFO:teuthology.orchestra.run.smithi118.stdout: "tags": [], 2024-03-24T03:17:24.006 INFO:teuthology.orchestra.run.smithi118.stdout: "see_also": [] 2024-03-24T03:17:24.006 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-24T03:17:24.006 INFO:teuthology.orchestra.run.smithi118.stdout: "log_refresh_metadata": { 2024-03-24T03:17:24.006 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "log_refresh_metadata", 2024-03-24T03:17:24.006 INFO:teuthology.orchestra.run.smithi118.stdout: "type": "bool", 2024-03-24T03:17:24.006 INFO:teuthology.orchestra.run.smithi118.stdout: "level": "advanced", 2024-03-24T03:17:24.006 INFO:teuthology.orchestra.run.smithi118.stdout: "flags": 0, 2024-03-24T03:17:24.006 INFO:teuthology.orchestra.run.smithi118.stdout: "default_value": "False", 2024-03-24T03:17:24.007 INFO:teuthology.orchestra.run.smithi118.stdout: "min": "", 2024-03-24T03:17:24.007 INFO:teuthology.orchestra.run.smithi118.stdout: "max": "", 2024-03-24T03:17:24.007 INFO:teuthology.orchestra.run.smithi118.stdout: "enum_allowed": [], 2024-03-24T03:17:24.007 INFO:teuthology.orchestra.run.smithi118.stdout: "desc": "Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level", 2024-03-24T03:17:24.007 INFO:teuthology.orchestra.run.smithi118.stdout: "long_desc": "", 2024-03-24T03:17:24.007 INFO:teuthology.orchestra.run.smithi118.stdout: "tags": [], 2024-03-24T03:17:24.007 INFO:teuthology.orchestra.run.smithi118.stdout: "see_also": [] 2024-03-24T03:17:24.007 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-24T03:17:24.007 INFO:teuthology.orchestra.run.smithi118.stdout: "log_to_cluster": { 2024-03-24T03:17:24.008 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "log_to_cluster", 2024-03-24T03:17:24.008 INFO:teuthology.orchestra.run.smithi118.stdout: "type": "bool", 2024-03-24T03:17:24.008 INFO:teuthology.orchestra.run.smithi118.stdout: "level": "advanced", 2024-03-24T03:17:24.008 INFO:teuthology.orchestra.run.smithi118.stdout: "flags": 0, 2024-03-24T03:17:24.008 INFO:teuthology.orchestra.run.smithi118.stdout: "default_value": "True", 2024-03-24T03:17:24.008 INFO:teuthology.orchestra.run.smithi118.stdout: "min": "", 2024-03-24T03:17:24.008 INFO:teuthology.orchestra.run.smithi118.stdout: "max": "", 2024-03-24T03:17:24.008 INFO:teuthology.orchestra.run.smithi118.stdout: "enum_allowed": [], 2024-03-24T03:17:24.008 INFO:teuthology.orchestra.run.smithi118.stdout: "desc": "log to the \"cephadm\" cluster log channel\"", 2024-03-24T03:17:24.009 INFO:teuthology.orchestra.run.smithi118.stdout: "long_desc": "", 2024-03-24T03:17:24.009 INFO:teuthology.orchestra.run.smithi118.stdout: "tags": [], 2024-03-24T03:17:24.009 INFO:teuthology.orchestra.run.smithi118.stdout: "see_also": [] 2024-03-24T03:17:24.009 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-24T03:17:24.009 INFO:teuthology.orchestra.run.smithi118.stdout: "log_to_cluster_level": { 2024-03-24T03:17:24.009 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "log_to_cluster_level", 2024-03-24T03:17:24.009 INFO:teuthology.orchestra.run.smithi118.stdout: "type": "str", 2024-03-24T03:17:24.009 INFO:teuthology.orchestra.run.smithi118.stdout: "level": "advanced", 2024-03-24T03:17:24.009 INFO:teuthology.orchestra.run.smithi118.stdout: "flags": 1, 2024-03-24T03:17:24.010 INFO:teuthology.orchestra.run.smithi118.stdout: "default_value": "info", 2024-03-24T03:17:24.010 INFO:teuthology.orchestra.run.smithi118.stdout: "min": "", 2024-03-24T03:17:24.010 INFO:teuthology.orchestra.run.smithi118.stdout: "max": "", 2024-03-24T03:17:24.010 INFO:teuthology.orchestra.run.smithi118.stdout: "enum_allowed": [ 2024-03-24T03:17:24.010 INFO:teuthology.orchestra.run.smithi118.stdout: "", 2024-03-24T03:17:24.010 INFO:teuthology.orchestra.run.smithi118.stdout: "critical", 2024-03-24T03:17:24.010 INFO:teuthology.orchestra.run.smithi118.stdout: "debug", 2024-03-24T03:17:24.010 INFO:teuthology.orchestra.run.smithi118.stdout: "error", 2024-03-24T03:17:24.010 INFO:teuthology.orchestra.run.smithi118.stdout: "info", 2024-03-24T03:17:24.011 INFO:teuthology.orchestra.run.smithi118.stdout: "warning" 2024-03-24T03:17:24.011 INFO:teuthology.orchestra.run.smithi118.stdout: ], 2024-03-24T03:17:24.011 INFO:teuthology.orchestra.run.smithi118.stdout: "desc": "", 2024-03-24T03:17:24.011 INFO:teuthology.orchestra.run.smithi118.stdout: "long_desc": "", 2024-03-24T03:17:24.011 INFO:teuthology.orchestra.run.smithi118.stdout: "tags": [], 2024-03-24T03:17:24.011 INFO:teuthology.orchestra.run.smithi118.stdout: "see_also": [] 2024-03-24T03:17:24.011 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-24T03:17:24.011 INFO:teuthology.orchestra.run.smithi118.stdout: "log_to_file": { 2024-03-24T03:17:24.011 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "log_to_file", 2024-03-24T03:17:24.011 INFO:teuthology.orchestra.run.smithi118.stdout: "type": "bool", 2024-03-24T03:17:24.012 INFO:teuthology.orchestra.run.smithi118.stdout: "level": "advanced", 2024-03-24T03:17:24.012 INFO:teuthology.orchestra.run.smithi118.stdout: "flags": 1, 2024-03-24T03:17:24.012 INFO:teuthology.orchestra.run.smithi118.stdout: "default_value": "False", 2024-03-24T03:17:24.012 INFO:teuthology.orchestra.run.smithi118.stdout: "min": "", 2024-03-24T03:17:24.012 INFO:teuthology.orchestra.run.smithi118.stdout: "max": "", 2024-03-24T03:17:24.012 INFO:teuthology.orchestra.run.smithi118.stdout: "enum_allowed": [], 2024-03-24T03:17:24.012 INFO:teuthology.orchestra.run.smithi118.stdout: "desc": "", 2024-03-24T03:17:24.012 INFO:teuthology.orchestra.run.smithi118.stdout: "long_desc": "", 2024-03-24T03:17:24.012 INFO:teuthology.orchestra.run.smithi118.stdout: "tags": [], 2024-03-24T03:17:24.013 INFO:teuthology.orchestra.run.smithi118.stdout: "see_also": [] 2024-03-24T03:17:24.013 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-24T03:17:24.013 INFO:teuthology.orchestra.run.smithi118.stdout: "manage_etc_ceph_ceph_conf": { 2024-03-24T03:17:24.013 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "manage_etc_ceph_ceph_conf", 2024-03-24T03:17:24.013 INFO:teuthology.orchestra.run.smithi118.stdout: "type": "bool", 2024-03-24T03:17:24.013 INFO:teuthology.orchestra.run.smithi118.stdout: "level": "advanced", 2024-03-24T03:17:24.013 INFO:teuthology.orchestra.run.smithi118.stdout: "flags": 0, 2024-03-24T03:17:24.013 INFO:teuthology.orchestra.run.smithi118.stdout: "default_value": "False", 2024-03-24T03:17:24.013 INFO:teuthology.orchestra.run.smithi118.stdout: "min": "", 2024-03-24T03:17:24.013 INFO:teuthology.orchestra.run.smithi118.stdout: "max": "", 2024-03-24T03:17:24.014 INFO:teuthology.orchestra.run.smithi118.stdout: "enum_allowed": [], 2024-03-24T03:17:24.014 INFO:teuthology.orchestra.run.smithi118.stdout: "desc": "Manage and own /etc/ceph/ceph.conf on the hosts.", 2024-03-24T03:17:24.014 INFO:teuthology.orchestra.run.smithi118.stdout: "long_desc": "", 2024-03-24T03:17:24.014 INFO:teuthology.orchestra.run.smithi118.stdout: "tags": [], 2024-03-24T03:17:24.014 INFO:teuthology.orchestra.run.smithi118.stdout: "see_also": [] 2024-03-24T03:17:24.014 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-24T03:17:24.014 INFO:teuthology.orchestra.run.smithi118.stdout: "manage_etc_ceph_ceph_conf_hosts": { 2024-03-24T03:17:24.014 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "manage_etc_ceph_ceph_conf_hosts", 2024-03-24T03:17:24.015 INFO:teuthology.orchestra.run.smithi118.stdout: "type": "str", 2024-03-24T03:17:24.015 INFO:teuthology.orchestra.run.smithi118.stdout: "level": "advanced", 2024-03-24T03:17:24.015 INFO:teuthology.orchestra.run.smithi118.stdout: "flags": 0, 2024-03-24T03:17:24.015 INFO:teuthology.orchestra.run.smithi118.stdout: "default_value": "*", 2024-03-24T03:17:24.015 INFO:teuthology.orchestra.run.smithi118.stdout: "min": "", 2024-03-24T03:17:24.015 INFO:teuthology.orchestra.run.smithi118.stdout: "max": "", 2024-03-24T03:17:24.015 INFO:teuthology.orchestra.run.smithi118.stdout: "enum_allowed": [], 2024-03-24T03:17:24.015 INFO:teuthology.orchestra.run.smithi118.stdout: "desc": "PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf", 2024-03-24T03:17:24.016 INFO:teuthology.orchestra.run.smithi118.stdout: "long_desc": "", 2024-03-24T03:17:24.016 INFO:teuthology.orchestra.run.smithi118.stdout: "tags": [], 2024-03-24T03:17:24.016 INFO:teuthology.orchestra.run.smithi118.stdout: "see_also": [] 2024-03-24T03:17:24.016 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-24T03:17:24.016 INFO:teuthology.orchestra.run.smithi118.stdout: "max_count_per_host": { 2024-03-24T03:17:24.016 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "max_count_per_host", 2024-03-24T03:17:24.016 INFO:teuthology.orchestra.run.smithi118.stdout: "type": "int", 2024-03-24T03:17:24.016 INFO:teuthology.orchestra.run.smithi118.stdout: "level": "advanced", 2024-03-24T03:17:24.016 INFO:teuthology.orchestra.run.smithi118.stdout: "flags": 0, 2024-03-24T03:17:24.017 INFO:teuthology.orchestra.run.smithi118.stdout: "default_value": "10", 2024-03-24T03:17:24.017 INFO:teuthology.orchestra.run.smithi118.stdout: "min": "", 2024-03-24T03:17:24.017 INFO:teuthology.orchestra.run.smithi118.stdout: "max": "", 2024-03-24T03:17:24.017 INFO:teuthology.orchestra.run.smithi118.stdout: "enum_allowed": [], 2024-03-24T03:17:24.017 INFO:teuthology.orchestra.run.smithi118.stdout: "desc": "max number of daemons per service per host", 2024-03-24T03:17:24.017 INFO:teuthology.orchestra.run.smithi118.stdout: "long_desc": "", 2024-03-24T03:17:24.017 INFO:teuthology.orchestra.run.smithi118.stdout: "tags": [], 2024-03-24T03:17:24.017 INFO:teuthology.orchestra.run.smithi118.stdout: "see_also": [] 2024-03-24T03:17:24.017 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-24T03:17:24.017 INFO:teuthology.orchestra.run.smithi118.stdout: "max_osd_draining_count": { 2024-03-24T03:17:24.018 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "max_osd_draining_count", 2024-03-24T03:17:24.018 INFO:teuthology.orchestra.run.smithi118.stdout: "type": "int", 2024-03-24T03:17:24.018 INFO:teuthology.orchestra.run.smithi118.stdout: "level": "advanced", 2024-03-24T03:17:24.018 INFO:teuthology.orchestra.run.smithi118.stdout: "flags": 0, 2024-03-24T03:17:24.018 INFO:teuthology.orchestra.run.smithi118.stdout: "default_value": "10", 2024-03-24T03:17:24.018 INFO:teuthology.orchestra.run.smithi118.stdout: "min": "", 2024-03-24T03:17:24.018 INFO:teuthology.orchestra.run.smithi118.stdout: "max": "", 2024-03-24T03:17:24.018 INFO:teuthology.orchestra.run.smithi118.stdout: "enum_allowed": [], 2024-03-24T03:17:24.018 INFO:teuthology.orchestra.run.smithi118.stdout: "desc": "max number of osds that will be drained simultaneously when osds are removed", 2024-03-24T03:17:24.019 INFO:teuthology.orchestra.run.smithi118.stdout: "long_desc": "", 2024-03-24T03:17:24.019 INFO:teuthology.orchestra.run.smithi118.stdout: "tags": [], 2024-03-24T03:17:24.019 INFO:teuthology.orchestra.run.smithi118.stdout: "see_also": [] 2024-03-24T03:17:24.019 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-24T03:17:24.019 INFO:teuthology.orchestra.run.smithi118.stdout: "migration_current": { 2024-03-24T03:17:24.019 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "migration_current", 2024-03-24T03:17:24.019 INFO:teuthology.orchestra.run.smithi118.stdout: "type": "int", 2024-03-24T03:17:24.019 INFO:teuthology.orchestra.run.smithi118.stdout: "level": "advanced", 2024-03-24T03:17:24.019 INFO:teuthology.orchestra.run.smithi118.stdout: "flags": 0, 2024-03-24T03:17:24.020 INFO:teuthology.orchestra.run.smithi118.stdout: "default_value": "", 2024-03-24T03:17:24.020 INFO:teuthology.orchestra.run.smithi118.stdout: "min": "", 2024-03-24T03:17:24.020 INFO:teuthology.orchestra.run.smithi118.stdout: "max": "", 2024-03-24T03:17:24.020 INFO:teuthology.orchestra.run.smithi118.stdout: "enum_allowed": [], 2024-03-24T03:17:24.020 INFO:teuthology.orchestra.run.smithi118.stdout: "desc": "internal - do not modify", 2024-03-24T03:17:24.020 INFO:teuthology.orchestra.run.smithi118.stdout: "long_desc": "", 2024-03-24T03:17:24.020 INFO:teuthology.orchestra.run.smithi118.stdout: "tags": [], 2024-03-24T03:17:24.020 INFO:teuthology.orchestra.run.smithi118.stdout: "see_also": [] 2024-03-24T03:17:24.020 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-24T03:17:24.020 INFO:teuthology.orchestra.run.smithi118.stdout: "mode": { 2024-03-24T03:17:24.021 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "mode", 2024-03-24T03:17:24.021 INFO:teuthology.orchestra.run.smithi118.stdout: "type": "str", 2024-03-24T03:17:24.021 INFO:teuthology.orchestra.run.smithi118.stdout: "level": "advanced", 2024-03-24T03:17:24.021 INFO:teuthology.orchestra.run.smithi118.stdout: "flags": 0, 2024-03-24T03:17:24.021 INFO:teuthology.orchestra.run.smithi118.stdout: "default_value": "root", 2024-03-24T03:17:24.021 INFO:teuthology.orchestra.run.smithi118.stdout: "min": "", 2024-03-24T03:17:24.021 INFO:teuthology.orchestra.run.smithi118.stdout: "max": "", 2024-03-24T03:17:24.021 INFO:teuthology.orchestra.run.smithi118.stdout: "enum_allowed": [ 2024-03-24T03:17:24.021 INFO:teuthology.orchestra.run.smithi118.stdout: "cephadm-package", 2024-03-24T03:17:24.022 INFO:teuthology.orchestra.run.smithi118.stdout: "root" 2024-03-24T03:17:24.022 INFO:teuthology.orchestra.run.smithi118.stdout: ], 2024-03-24T03:17:24.022 INFO:teuthology.orchestra.run.smithi118.stdout: "desc": "mode for remote execution of cephadm", 2024-03-24T03:17:24.022 INFO:teuthology.orchestra.run.smithi118.stdout: "long_desc": "", 2024-03-24T03:17:24.022 INFO:teuthology.orchestra.run.smithi118.stdout: "tags": [], 2024-03-24T03:17:24.022 INFO:teuthology.orchestra.run.smithi118.stdout: "see_also": [] 2024-03-24T03:17:24.022 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-24T03:17:24.022 INFO:teuthology.orchestra.run.smithi118.stdout: "oob_default_addr": { 2024-03-24T03:17:24.022 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "oob_default_addr", 2024-03-24T03:17:24.023 INFO:teuthology.orchestra.run.smithi118.stdout: "type": "str", 2024-03-24T03:17:24.023 INFO:teuthology.orchestra.run.smithi118.stdout: "level": "advanced", 2024-03-24T03:17:24.023 INFO:teuthology.orchestra.run.smithi118.stdout: "flags": 0, 2024-03-24T03:17:24.023 INFO:teuthology.orchestra.run.smithi118.stdout: "default_value": "169.254.1.1", 2024-03-24T03:17:24.023 INFO:teuthology.orchestra.run.smithi118.stdout: "min": "", 2024-03-24T03:17:24.023 INFO:teuthology.orchestra.run.smithi118.stdout: "max": "", 2024-03-24T03:17:24.023 INFO:teuthology.orchestra.run.smithi118.stdout: "enum_allowed": [], 2024-03-24T03:17:24.023 INFO:teuthology.orchestra.run.smithi118.stdout: "desc": "Default address for RedFish API (oob management).", 2024-03-24T03:17:24.023 INFO:teuthology.orchestra.run.smithi118.stdout: "long_desc": "", 2024-03-24T03:17:24.024 INFO:teuthology.orchestra.run.smithi118.stdout: "tags": [], 2024-03-24T03:17:24.024 INFO:teuthology.orchestra.run.smithi118.stdout: "see_also": [] 2024-03-24T03:17:24.024 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-24T03:17:24.024 INFO:teuthology.orchestra.run.smithi118.stdout: "prometheus_alerts_path": { 2024-03-24T03:17:24.024 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "prometheus_alerts_path", 2024-03-24T03:17:24.024 INFO:teuthology.orchestra.run.smithi118.stdout: "type": "str", 2024-03-24T03:17:24.024 INFO:teuthology.orchestra.run.smithi118.stdout: "level": "advanced", 2024-03-24T03:17:24.024 INFO:teuthology.orchestra.run.smithi118.stdout: "flags": 0, 2024-03-24T03:17:24.024 INFO:teuthology.orchestra.run.smithi118.stdout: "default_value": "/etc/prometheus/ceph/ceph_default_alerts.yml", 2024-03-24T03:17:24.024 INFO:teuthology.orchestra.run.smithi118.stdout: "min": "", 2024-03-24T03:17:24.025 INFO:teuthology.orchestra.run.smithi118.stdout: "max": "", 2024-03-24T03:17:24.025 INFO:teuthology.orchestra.run.smithi118.stdout: "enum_allowed": [], 2024-03-24T03:17:24.025 INFO:teuthology.orchestra.run.smithi118.stdout: "desc": "location of alerts to include in prometheus deployments", 2024-03-24T03:17:24.025 INFO:teuthology.orchestra.run.smithi118.stdout: "long_desc": "", 2024-03-24T03:17:24.025 INFO:teuthology.orchestra.run.smithi118.stdout: "tags": [], 2024-03-24T03:17:24.025 INFO:teuthology.orchestra.run.smithi118.stdout: "see_also": [] 2024-03-24T03:17:24.025 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-24T03:17:24.025 INFO:teuthology.orchestra.run.smithi118.stdout: "registry_insecure": { 2024-03-24T03:17:24.025 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "registry_insecure", 2024-03-24T03:17:24.026 INFO:teuthology.orchestra.run.smithi118.stdout: "type": "bool", 2024-03-24T03:17:24.026 INFO:teuthology.orchestra.run.smithi118.stdout: "level": "advanced", 2024-03-24T03:17:24.026 INFO:teuthology.orchestra.run.smithi118.stdout: "flags": 0, 2024-03-24T03:17:24.026 INFO:teuthology.orchestra.run.smithi118.stdout: "default_value": "False", 2024-03-24T03:17:24.026 INFO:teuthology.orchestra.run.smithi118.stdout: "min": "", 2024-03-24T03:17:24.026 INFO:teuthology.orchestra.run.smithi118.stdout: "max": "", 2024-03-24T03:17:24.026 INFO:teuthology.orchestra.run.smithi118.stdout: "enum_allowed": [], 2024-03-24T03:17:24.026 INFO:teuthology.orchestra.run.smithi118.stdout: "desc": "Registry is to be considered insecure (no TLS available). Only for development purposes.", 2024-03-24T03:17:24.026 INFO:teuthology.orchestra.run.smithi118.stdout: "long_desc": "", 2024-03-24T03:17:24.027 INFO:teuthology.orchestra.run.smithi118.stdout: "tags": [], 2024-03-24T03:17:24.027 INFO:teuthology.orchestra.run.smithi118.stdout: "see_also": [] 2024-03-24T03:17:24.027 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-24T03:17:24.027 INFO:teuthology.orchestra.run.smithi118.stdout: "registry_password": { 2024-03-24T03:17:24.027 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "registry_password", 2024-03-24T03:17:24.027 INFO:teuthology.orchestra.run.smithi118.stdout: "type": "str", 2024-03-24T03:17:24.027 INFO:teuthology.orchestra.run.smithi118.stdout: "level": "advanced", 2024-03-24T03:17:24.027 INFO:teuthology.orchestra.run.smithi118.stdout: "flags": 0, 2024-03-24T03:17:24.027 INFO:teuthology.orchestra.run.smithi118.stdout: "default_value": "", 2024-03-24T03:17:24.027 INFO:teuthology.orchestra.run.smithi118.stdout: "min": "", 2024-03-24T03:17:24.028 INFO:teuthology.orchestra.run.smithi118.stdout: "max": "", 2024-03-24T03:17:24.028 INFO:teuthology.orchestra.run.smithi118.stdout: "enum_allowed": [], 2024-03-24T03:17:24.028 INFO:teuthology.orchestra.run.smithi118.stdout: "desc": "Custom repository password. Only used for logging into a registry.", 2024-03-24T03:17:24.028 INFO:teuthology.orchestra.run.smithi118.stdout: "long_desc": "", 2024-03-24T03:17:24.028 INFO:teuthology.orchestra.run.smithi118.stdout: "tags": [], 2024-03-24T03:17:24.028 INFO:teuthology.orchestra.run.smithi118.stdout: "see_also": [] 2024-03-24T03:17:24.028 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-24T03:17:24.028 INFO:teuthology.orchestra.run.smithi118.stdout: "registry_url": { 2024-03-24T03:17:24.028 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "registry_url", 2024-03-24T03:17:24.028 INFO:teuthology.orchestra.run.smithi118.stdout: "type": "str", 2024-03-24T03:17:24.029 INFO:teuthology.orchestra.run.smithi118.stdout: "level": "advanced", 2024-03-24T03:17:24.029 INFO:teuthology.orchestra.run.smithi118.stdout: "flags": 0, 2024-03-24T03:17:24.029 INFO:teuthology.orchestra.run.smithi118.stdout: "default_value": "", 2024-03-24T03:17:24.029 INFO:teuthology.orchestra.run.smithi118.stdout: "min": "", 2024-03-24T03:17:24.029 INFO:teuthology.orchestra.run.smithi118.stdout: "max": "", 2024-03-24T03:17:24.029 INFO:teuthology.orchestra.run.smithi118.stdout: "enum_allowed": [], 2024-03-24T03:17:24.029 INFO:teuthology.orchestra.run.smithi118.stdout: "desc": "Registry url for login purposes. This is not the default registry", 2024-03-24T03:17:24.029 INFO:teuthology.orchestra.run.smithi118.stdout: "long_desc": "", 2024-03-24T03:17:24.029 INFO:teuthology.orchestra.run.smithi118.stdout: "tags": [], 2024-03-24T03:17:24.030 INFO:teuthology.orchestra.run.smithi118.stdout: "see_also": [] 2024-03-24T03:17:24.030 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-24T03:17:24.030 INFO:teuthology.orchestra.run.smithi118.stdout: "registry_username": { 2024-03-24T03:17:24.030 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "registry_username", 2024-03-24T03:17:24.030 INFO:teuthology.orchestra.run.smithi118.stdout: "type": "str", 2024-03-24T03:17:24.030 INFO:teuthology.orchestra.run.smithi118.stdout: "level": "advanced", 2024-03-24T03:17:24.030 INFO:teuthology.orchestra.run.smithi118.stdout: "flags": 0, 2024-03-24T03:17:24.030 INFO:teuthology.orchestra.run.smithi118.stdout: "default_value": "", 2024-03-24T03:17:24.030 INFO:teuthology.orchestra.run.smithi118.stdout: "min": "", 2024-03-24T03:17:24.031 INFO:teuthology.orchestra.run.smithi118.stdout: "max": "", 2024-03-24T03:17:24.031 INFO:teuthology.orchestra.run.smithi118.stdout: "enum_allowed": [], 2024-03-24T03:17:24.031 INFO:teuthology.orchestra.run.smithi118.stdout: "desc": "Custom repository username. Only used for logging into a registry.", 2024-03-24T03:17:24.031 INFO:teuthology.orchestra.run.smithi118.stdout: "long_desc": "", 2024-03-24T03:17:24.031 INFO:teuthology.orchestra.run.smithi118.stdout: "tags": [], 2024-03-24T03:17:24.031 INFO:teuthology.orchestra.run.smithi118.stdout: "see_also": [] 2024-03-24T03:17:24.031 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-24T03:17:24.031 INFO:teuthology.orchestra.run.smithi118.stdout: "secure_monitoring_stack": { 2024-03-24T03:17:24.031 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "secure_monitoring_stack", 2024-03-24T03:17:24.032 INFO:teuthology.orchestra.run.smithi118.stdout: "type": "bool", 2024-03-24T03:17:24.032 INFO:teuthology.orchestra.run.smithi118.stdout: "level": "advanced", 2024-03-24T03:17:24.032 INFO:teuthology.orchestra.run.smithi118.stdout: "flags": 0, 2024-03-24T03:17:24.032 INFO:teuthology.orchestra.run.smithi118.stdout: "default_value": "False", 2024-03-24T03:17:24.032 INFO:teuthology.orchestra.run.smithi118.stdout: "min": "", 2024-03-24T03:17:24.032 INFO:teuthology.orchestra.run.smithi118.stdout: "max": "", 2024-03-24T03:17:24.032 INFO:teuthology.orchestra.run.smithi118.stdout: "enum_allowed": [], 2024-03-24T03:17:24.032 INFO:teuthology.orchestra.run.smithi118.stdout: "desc": "Enable TLS security for all the monitoring stack daemons", 2024-03-24T03:17:24.032 INFO:teuthology.orchestra.run.smithi118.stdout: "long_desc": "", 2024-03-24T03:17:24.032 INFO:teuthology.orchestra.run.smithi118.stdout: "tags": [], 2024-03-24T03:17:24.033 INFO:teuthology.orchestra.run.smithi118.stdout: "see_also": [] 2024-03-24T03:17:24.033 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-24T03:17:24.033 INFO:teuthology.orchestra.run.smithi118.stdout: "service_discovery_port": { 2024-03-24T03:17:24.033 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "service_discovery_port", 2024-03-24T03:17:24.033 INFO:teuthology.orchestra.run.smithi118.stdout: "type": "int", 2024-03-24T03:17:24.033 INFO:teuthology.orchestra.run.smithi118.stdout: "level": "advanced", 2024-03-24T03:17:24.033 INFO:teuthology.orchestra.run.smithi118.stdout: "flags": 0, 2024-03-24T03:17:24.033 INFO:teuthology.orchestra.run.smithi118.stdout: "default_value": "8765", 2024-03-24T03:17:24.033 INFO:teuthology.orchestra.run.smithi118.stdout: "min": "", 2024-03-24T03:17:24.034 INFO:teuthology.orchestra.run.smithi118.stdout: "max": "", 2024-03-24T03:17:24.034 INFO:teuthology.orchestra.run.smithi118.stdout: "enum_allowed": [], 2024-03-24T03:17:24.034 INFO:teuthology.orchestra.run.smithi118.stdout: "desc": "cephadm service discovery port", 2024-03-24T03:17:24.034 INFO:teuthology.orchestra.run.smithi118.stdout: "long_desc": "", 2024-03-24T03:17:24.034 INFO:teuthology.orchestra.run.smithi118.stdout: "tags": [], 2024-03-24T03:17:24.034 INFO:teuthology.orchestra.run.smithi118.stdout: "see_also": [] 2024-03-24T03:17:24.034 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-24T03:17:24.034 INFO:teuthology.orchestra.run.smithi118.stdout: "ssh_config_file": { 2024-03-24T03:17:24.034 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "ssh_config_file", 2024-03-24T03:17:24.035 INFO:teuthology.orchestra.run.smithi118.stdout: "type": "str", 2024-03-24T03:17:24.035 INFO:teuthology.orchestra.run.smithi118.stdout: "level": "advanced", 2024-03-24T03:17:24.035 INFO:teuthology.orchestra.run.smithi118.stdout: "flags": 0, 2024-03-24T03:17:24.035 INFO:teuthology.orchestra.run.smithi118.stdout: "default_value": "", 2024-03-24T03:17:24.035 INFO:teuthology.orchestra.run.smithi118.stdout: "min": "", 2024-03-24T03:17:24.035 INFO:teuthology.orchestra.run.smithi118.stdout: "max": "", 2024-03-24T03:17:24.035 INFO:teuthology.orchestra.run.smithi118.stdout: "enum_allowed": [], 2024-03-24T03:17:24.035 INFO:teuthology.orchestra.run.smithi118.stdout: "desc": "customized SSH config file to connect to managed hosts", 2024-03-24T03:17:24.035 INFO:teuthology.orchestra.run.smithi118.stdout: "long_desc": "", 2024-03-24T03:17:24.035 INFO:teuthology.orchestra.run.smithi118.stdout: "tags": [], 2024-03-24T03:17:24.036 INFO:teuthology.orchestra.run.smithi118.stdout: "see_also": [] 2024-03-24T03:17:24.036 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-24T03:17:24.036 INFO:teuthology.orchestra.run.smithi118.stdout: "use_agent": { 2024-03-24T03:17:24.036 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "use_agent", 2024-03-24T03:17:24.036 INFO:teuthology.orchestra.run.smithi118.stdout: "type": "bool", 2024-03-24T03:17:24.036 INFO:teuthology.orchestra.run.smithi118.stdout: "level": "advanced", 2024-03-24T03:17:24.036 INFO:teuthology.orchestra.run.smithi118.stdout: "flags": 0, 2024-03-24T03:17:24.036 INFO:teuthology.orchestra.run.smithi118.stdout: "default_value": "False", 2024-03-24T03:17:24.036 INFO:teuthology.orchestra.run.smithi118.stdout: "min": "", 2024-03-24T03:17:24.037 INFO:teuthology.orchestra.run.smithi118.stdout: "max": "", 2024-03-24T03:17:24.037 INFO:teuthology.orchestra.run.smithi118.stdout: "enum_allowed": [], 2024-03-24T03:17:24.037 INFO:teuthology.orchestra.run.smithi118.stdout: "desc": "Use cephadm agent on each host to gather and send metadata", 2024-03-24T03:17:24.037 INFO:teuthology.orchestra.run.smithi118.stdout: "long_desc": "", 2024-03-24T03:17:24.037 INFO:teuthology.orchestra.run.smithi118.stdout: "tags": [], 2024-03-24T03:17:24.037 INFO:teuthology.orchestra.run.smithi118.stdout: "see_also": [] 2024-03-24T03:17:24.037 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-24T03:17:24.037 INFO:teuthology.orchestra.run.smithi118.stdout: "use_repo_digest": { 2024-03-24T03:17:24.037 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "use_repo_digest", 2024-03-24T03:17:24.037 INFO:teuthology.orchestra.run.smithi118.stdout: "type": "bool", 2024-03-24T03:17:24.038 INFO:teuthology.orchestra.run.smithi118.stdout: "level": "advanced", 2024-03-24T03:17:24.038 INFO:teuthology.orchestra.run.smithi118.stdout: "flags": 0, 2024-03-24T03:17:24.038 INFO:teuthology.orchestra.run.smithi118.stdout: "default_value": "True", 2024-03-24T03:17:24.038 INFO:teuthology.orchestra.run.smithi118.stdout: "min": "", 2024-03-24T03:17:24.038 INFO:teuthology.orchestra.run.smithi118.stdout: "max": "", 2024-03-24T03:17:24.038 INFO:teuthology.orchestra.run.smithi118.stdout: "enum_allowed": [], 2024-03-24T03:17:24.038 INFO:teuthology.orchestra.run.smithi118.stdout: "desc": "Automatically convert image tags to image digest. Make sure all daemons use the same image", 2024-03-24T03:17:24.038 INFO:teuthology.orchestra.run.smithi118.stdout: "long_desc": "", 2024-03-24T03:17:24.038 INFO:teuthology.orchestra.run.smithi118.stdout: "tags": [], 2024-03-24T03:17:24.039 INFO:teuthology.orchestra.run.smithi118.stdout: "see_also": [] 2024-03-24T03:17:24.039 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-24T03:17:24.039 INFO:teuthology.orchestra.run.smithi118.stdout: "warn_on_failed_host_check": { 2024-03-24T03:17:24.039 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "warn_on_failed_host_check", 2024-03-24T03:17:24.039 INFO:teuthology.orchestra.run.smithi118.stdout: "type": "bool", 2024-03-24T03:17:24.039 INFO:teuthology.orchestra.run.smithi118.stdout: "level": "advanced", 2024-03-24T03:17:24.039 INFO:teuthology.orchestra.run.smithi118.stdout: "flags": 0, 2024-03-24T03:17:24.039 INFO:teuthology.orchestra.run.smithi118.stdout: "default_value": "True", 2024-03-24T03:17:24.039 INFO:teuthology.orchestra.run.smithi118.stdout: "min": "", 2024-03-24T03:17:24.039 INFO:teuthology.orchestra.run.smithi118.stdout: "max": "", 2024-03-24T03:17:24.040 INFO:teuthology.orchestra.run.smithi118.stdout: "enum_allowed": [], 2024-03-24T03:17:24.040 INFO:teuthology.orchestra.run.smithi118.stdout: "desc": "raise a health warning if the host check fails", 2024-03-24T03:17:24.040 INFO:teuthology.orchestra.run.smithi118.stdout: "long_desc": "", 2024-03-24T03:17:24.040 INFO:teuthology.orchestra.run.smithi118.stdout: "tags": [], 2024-03-24T03:17:24.040 INFO:teuthology.orchestra.run.smithi118.stdout: "see_also": [] 2024-03-24T03:17:24.040 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-24T03:17:24.040 INFO:teuthology.orchestra.run.smithi118.stdout: "warn_on_stray_daemons": { 2024-03-24T03:17:24.040 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "warn_on_stray_daemons", 2024-03-24T03:17:24.040 INFO:teuthology.orchestra.run.smithi118.stdout: "type": "bool", 2024-03-24T03:17:24.041 INFO:teuthology.orchestra.run.smithi118.stdout: "level": "advanced", 2024-03-24T03:17:24.041 INFO:teuthology.orchestra.run.smithi118.stdout: "flags": 0, 2024-03-24T03:17:24.041 INFO:teuthology.orchestra.run.smithi118.stdout: "default_value": "True", 2024-03-24T03:17:24.041 INFO:teuthology.orchestra.run.smithi118.stdout: "min": "", 2024-03-24T03:17:24.041 INFO:teuthology.orchestra.run.smithi118.stdout: "max": "", 2024-03-24T03:17:24.041 INFO:teuthology.orchestra.run.smithi118.stdout: "enum_allowed": [], 2024-03-24T03:17:24.041 INFO:teuthology.orchestra.run.smithi118.stdout: "desc": "raise a health warning if daemons are detected that are not managed by cephadm", 2024-03-24T03:17:24.041 INFO:teuthology.orchestra.run.smithi118.stdout: "long_desc": "", 2024-03-24T03:17:24.041 INFO:teuthology.orchestra.run.smithi118.stdout: "tags": [], 2024-03-24T03:17:24.041 INFO:teuthology.orchestra.run.smithi118.stdout: "see_also": [] 2024-03-24T03:17:24.042 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-24T03:17:24.042 INFO:teuthology.orchestra.run.smithi118.stdout: "warn_on_stray_hosts": { 2024-03-24T03:17:24.042 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "warn_on_stray_hosts", 2024-03-24T03:17:24.042 INFO:teuthology.orchestra.run.smithi118.stdout: "type": "bool", 2024-03-24T03:17:24.042 INFO:teuthology.orchestra.run.smithi118.stdout: "level": "advanced", 2024-03-24T03:17:24.042 INFO:teuthology.orchestra.run.smithi118.stdout: "flags": 0, 2024-03-24T03:17:24.042 INFO:teuthology.orchestra.run.smithi118.stdout: "default_value": "True", 2024-03-24T03:17:24.042 INFO:teuthology.orchestra.run.smithi118.stdout: "min": "", 2024-03-24T03:17:24.042 INFO:teuthology.orchestra.run.smithi118.stdout: "max": "", 2024-03-24T03:17:24.043 INFO:teuthology.orchestra.run.smithi118.stdout: "enum_allowed": [], 2024-03-24T03:17:24.043 INFO:teuthology.orchestra.run.smithi118.stdout: "desc": "raise a health warning if daemons are detected on a host that is not managed by cephadm", 2024-03-24T03:17:24.043 INFO:teuthology.orchestra.run.smithi118.stdout: "long_desc": "", 2024-03-24T03:17:24.043 INFO:teuthology.orchestra.run.smithi118.stdout: "tags": [], 2024-03-24T03:17:24.043 INFO:teuthology.orchestra.run.smithi118.stdout: "see_also": [] 2024-03-24T03:17:24.043 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-03-24T03:17:24.043 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-03-24T03:17:24.043 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-24T03:17:24.043 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-24T03:17:24.044 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "crash", 2024-03-24T03:17:24.044 INFO:teuthology.orchestra.run.smithi118.stdout: "can_run": true, 2024-03-24T03:17:24.044 INFO:teuthology.orchestra.run.smithi118.stdout: "error_string": "", 2024-03-24T03:17:24.044 INFO:teuthology.orchestra.run.smithi118.stdout: "module_options": { 2024-03-24T03:17:24.044 INFO:teuthology.orchestra.run.smithi118.stdout: "log_level": { 2024-03-24T03:17:24.044 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "log_level", 2024-03-24T03:17:24.044 INFO:teuthology.orchestra.run.smithi118.stdout: "type": "str", 2024-03-24T03:17:24.044 INFO:teuthology.orchestra.run.smithi118.stdout: "level": "advanced", 2024-03-24T03:17:24.044 INFO:teuthology.orchestra.run.smithi118.stdout: "flags": 1, 2024-03-24T03:17:24.045 INFO:teuthology.orchestra.run.smithi118.stdout: "default_value": "", 2024-03-24T03:17:24.045 INFO:teuthology.orchestra.run.smithi118.stdout: "min": "", 2024-03-24T03:17:24.045 INFO:teuthology.orchestra.run.smithi118.stdout: "max": "", 2024-03-24T03:17:24.045 INFO:teuthology.orchestra.run.smithi118.stdout: "enum_allowed": [ 2024-03-24T03:17:24.045 INFO:teuthology.orchestra.run.smithi118.stdout: "", 2024-03-24T03:17:24.045 INFO:teuthology.orchestra.run.smithi118.stdout: "critical", 2024-03-24T03:17:24.045 INFO:teuthology.orchestra.run.smithi118.stdout: "debug", 2024-03-24T03:17:24.045 INFO:teuthology.orchestra.run.smithi118.stdout: "error", 2024-03-24T03:17:24.045 INFO:teuthology.orchestra.run.smithi118.stdout: "info", 2024-03-24T03:17:24.046 INFO:teuthology.orchestra.run.smithi118.stdout: "warning" 2024-03-24T03:17:24.046 INFO:teuthology.orchestra.run.smithi118.stdout: ], 2024-03-24T03:17:24.046 INFO:teuthology.orchestra.run.smithi118.stdout: "desc": "", 2024-03-24T03:17:24.046 INFO:teuthology.orchestra.run.smithi118.stdout: "long_desc": "", 2024-03-24T03:17:24.046 INFO:teuthology.orchestra.run.smithi118.stdout: "tags": [], 2024-03-24T03:17:24.046 INFO:teuthology.orchestra.run.smithi118.stdout: "see_also": [] 2024-03-24T03:17:24.046 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-24T03:17:24.046 INFO:teuthology.orchestra.run.smithi118.stdout: "log_to_cluster": { 2024-03-24T03:17:24.047 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "log_to_cluster", 2024-03-24T03:17:24.047 INFO:teuthology.orchestra.run.smithi118.stdout: "type": "bool", 2024-03-24T03:17:24.047 INFO:teuthology.orchestra.run.smithi118.stdout: "level": "advanced", 2024-03-24T03:17:24.047 INFO:teuthology.orchestra.run.smithi118.stdout: "flags": 1, 2024-03-24T03:17:24.047 INFO:teuthology.orchestra.run.smithi118.stdout: "default_value": "False", 2024-03-24T03:17:24.047 INFO:teuthology.orchestra.run.smithi118.stdout: "min": "", 2024-03-24T03:17:24.047 INFO:teuthology.orchestra.run.smithi118.stdout: "max": "", 2024-03-24T03:17:24.047 INFO:teuthology.orchestra.run.smithi118.stdout: "enum_allowed": [], 2024-03-24T03:17:24.047 INFO:teuthology.orchestra.run.smithi118.stdout: "desc": "", 2024-03-24T03:17:24.047 INFO:teuthology.orchestra.run.smithi118.stdout: "long_desc": "", 2024-03-24T03:17:24.048 INFO:teuthology.orchestra.run.smithi118.stdout: "tags": [], 2024-03-24T03:17:24.048 INFO:teuthology.orchestra.run.smithi118.stdout: "see_also": [] 2024-03-24T03:17:24.048 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-24T03:17:24.048 INFO:teuthology.orchestra.run.smithi118.stdout: "log_to_cluster_level": { 2024-03-24T03:17:24.048 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "log_to_cluster_level", 2024-03-24T03:17:24.048 INFO:teuthology.orchestra.run.smithi118.stdout: "type": "str", 2024-03-24T03:17:24.048 INFO:teuthology.orchestra.run.smithi118.stdout: "level": "advanced", 2024-03-24T03:17:24.048 INFO:teuthology.orchestra.run.smithi118.stdout: "flags": 1, 2024-03-24T03:17:24.049 INFO:teuthology.orchestra.run.smithi118.stdout: "default_value": "info", 2024-03-24T03:17:24.049 INFO:teuthology.orchestra.run.smithi118.stdout: "min": "", 2024-03-24T03:17:24.049 INFO:teuthology.orchestra.run.smithi118.stdout: "max": "", 2024-03-24T03:17:24.049 INFO:teuthology.orchestra.run.smithi118.stdout: "enum_allowed": [ 2024-03-24T03:17:24.049 INFO:teuthology.orchestra.run.smithi118.stdout: "", 2024-03-24T03:17:24.049 INFO:teuthology.orchestra.run.smithi118.stdout: "critical", 2024-03-24T03:17:24.049 INFO:teuthology.orchestra.run.smithi118.stdout: "debug", 2024-03-24T03:17:24.049 INFO:teuthology.orchestra.run.smithi118.stdout: "error", 2024-03-24T03:17:24.049 INFO:teuthology.orchestra.run.smithi118.stdout: "info", 2024-03-24T03:17:24.050 INFO:teuthology.orchestra.run.smithi118.stdout: "warning" 2024-03-24T03:17:24.050 INFO:teuthology.orchestra.run.smithi118.stdout: ], 2024-03-24T03:17:24.050 INFO:teuthology.orchestra.run.smithi118.stdout: "desc": "", 2024-03-24T03:17:24.050 INFO:teuthology.orchestra.run.smithi118.stdout: "long_desc": "", 2024-03-24T03:17:24.050 INFO:teuthology.orchestra.run.smithi118.stdout: "tags": [], 2024-03-24T03:17:24.050 INFO:teuthology.orchestra.run.smithi118.stdout: "see_also": [] 2024-03-24T03:17:24.050 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-24T03:17:24.050 INFO:teuthology.orchestra.run.smithi118.stdout: "log_to_file": { 2024-03-24T03:17:24.050 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "log_to_file", 2024-03-24T03:17:24.051 INFO:teuthology.orchestra.run.smithi118.stdout: "type": "bool", 2024-03-24T03:17:24.051 INFO:teuthology.orchestra.run.smithi118.stdout: "level": "advanced", 2024-03-24T03:17:24.051 INFO:teuthology.orchestra.run.smithi118.stdout: "flags": 1, 2024-03-24T03:17:24.051 INFO:teuthology.orchestra.run.smithi118.stdout: "default_value": "False", 2024-03-24T03:17:24.051 INFO:teuthology.orchestra.run.smithi118.stdout: "min": "", 2024-03-24T03:17:24.051 INFO:teuthology.orchestra.run.smithi118.stdout: "max": "", 2024-03-24T03:17:24.051 INFO:teuthology.orchestra.run.smithi118.stdout: "enum_allowed": [], 2024-03-24T03:17:24.051 INFO:teuthology.orchestra.run.smithi118.stdout: "desc": "", 2024-03-24T03:17:24.051 INFO:teuthology.orchestra.run.smithi118.stdout: "long_desc": "", 2024-03-24T03:17:24.052 INFO:teuthology.orchestra.run.smithi118.stdout: "tags": [], 2024-03-24T03:17:24.052 INFO:teuthology.orchestra.run.smithi118.stdout: "see_also": [] 2024-03-24T03:17:24.052 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-24T03:17:24.052 INFO:teuthology.orchestra.run.smithi118.stdout: "retain_interval": { 2024-03-24T03:17:24.052 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "retain_interval", 2024-03-24T03:17:24.052 INFO:teuthology.orchestra.run.smithi118.stdout: "type": "secs", 2024-03-24T03:17:24.052 INFO:teuthology.orchestra.run.smithi118.stdout: "level": "advanced", 2024-03-24T03:17:24.052 INFO:teuthology.orchestra.run.smithi118.stdout: "flags": 1, 2024-03-24T03:17:24.052 INFO:teuthology.orchestra.run.smithi118.stdout: "default_value": "31536000", 2024-03-24T03:17:24.053 INFO:teuthology.orchestra.run.smithi118.stdout: "min": "", 2024-03-24T03:17:24.053 INFO:teuthology.orchestra.run.smithi118.stdout: "max": "", 2024-03-24T03:17:24.053 INFO:teuthology.orchestra.run.smithi118.stdout: "enum_allowed": [], 2024-03-24T03:17:24.053 INFO:teuthology.orchestra.run.smithi118.stdout: "desc": "how long to retain crashes before pruning them", 2024-03-24T03:17:24.053 INFO:teuthology.orchestra.run.smithi118.stdout: "long_desc": "", 2024-03-24T03:17:24.053 INFO:teuthology.orchestra.run.smithi118.stdout: "tags": [], 2024-03-24T03:17:24.053 INFO:teuthology.orchestra.run.smithi118.stdout: "see_also": [] 2024-03-24T03:17:24.053 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-24T03:17:24.053 INFO:teuthology.orchestra.run.smithi118.stdout: "warn_recent_interval": { 2024-03-24T03:17:24.054 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "warn_recent_interval", 2024-03-24T03:17:24.054 INFO:teuthology.orchestra.run.smithi118.stdout: "type": "secs", 2024-03-24T03:17:24.054 INFO:teuthology.orchestra.run.smithi118.stdout: "level": "advanced", 2024-03-24T03:17:24.054 INFO:teuthology.orchestra.run.smithi118.stdout: "flags": 1, 2024-03-24T03:17:24.054 INFO:teuthology.orchestra.run.smithi118.stdout: "default_value": "1209600", 2024-03-24T03:17:24.054 INFO:teuthology.orchestra.run.smithi118.stdout: "min": "", 2024-03-24T03:17:24.054 INFO:teuthology.orchestra.run.smithi118.stdout: "max": "", 2024-03-24T03:17:24.054 INFO:teuthology.orchestra.run.smithi118.stdout: "enum_allowed": [], 2024-03-24T03:17:24.055 INFO:teuthology.orchestra.run.smithi118.stdout: "desc": "time interval in which to warn about recent crashes", 2024-03-24T03:17:24.055 INFO:teuthology.orchestra.run.smithi118.stdout: "long_desc": "", 2024-03-24T03:17:24.055 INFO:teuthology.orchestra.run.smithi118.stdout: "tags": [], 2024-03-24T03:17:24.055 INFO:teuthology.orchestra.run.smithi118.stdout: "see_also": [] 2024-03-24T03:17:24.055 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-03-24T03:17:24.055 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-03-24T03:17:24.055 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-24T03:17:24.055 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-24T03:17:24.055 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "dashboard", 2024-03-24T03:17:24.055 INFO:teuthology.orchestra.run.smithi118.stdout: "can_run": true, 2024-03-24T03:17:24.056 INFO:teuthology.orchestra.run.smithi118.stdout: "error_string": "", 2024-03-24T03:17:24.056 INFO:teuthology.orchestra.run.smithi118.stdout: "module_options": { 2024-03-24T03:17:24.056 INFO:teuthology.orchestra.run.smithi118.stdout: "ACCOUNT_LOCKOUT_ATTEMPTS": { 2024-03-24T03:17:24.056 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "ACCOUNT_LOCKOUT_ATTEMPTS", 2024-03-24T03:17:24.056 INFO:teuthology.orchestra.run.smithi118.stdout: "type": "int", 2024-03-24T03:17:24.056 INFO:teuthology.orchestra.run.smithi118.stdout: "level": "advanced", 2024-03-24T03:17:24.056 INFO:teuthology.orchestra.run.smithi118.stdout: "flags": 0, 2024-03-24T03:17:24.056 INFO:teuthology.orchestra.run.smithi118.stdout: "default_value": "10", 2024-03-24T03:17:24.056 INFO:teuthology.orchestra.run.smithi118.stdout: "min": "", 2024-03-24T03:17:24.057 INFO:teuthology.orchestra.run.smithi118.stdout: "max": "", 2024-03-24T03:17:24.057 INFO:teuthology.orchestra.run.smithi118.stdout: "enum_allowed": [], 2024-03-24T03:17:24.057 INFO:teuthology.orchestra.run.smithi118.stdout: "desc": "", 2024-03-24T03:17:24.057 INFO:teuthology.orchestra.run.smithi118.stdout: "long_desc": "", 2024-03-24T03:17:24.057 INFO:teuthology.orchestra.run.smithi118.stdout: "tags": [], 2024-03-24T03:17:24.057 INFO:teuthology.orchestra.run.smithi118.stdout: "see_also": [] 2024-03-24T03:17:24.057 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-24T03:17:24.057 INFO:teuthology.orchestra.run.smithi118.stdout: "ALERTMANAGER_API_HOST": { 2024-03-24T03:17:24.057 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "ALERTMANAGER_API_HOST", 2024-03-24T03:17:24.058 INFO:teuthology.orchestra.run.smithi118.stdout: "type": "str", 2024-03-24T03:17:24.058 INFO:teuthology.orchestra.run.smithi118.stdout: "level": "advanced", 2024-03-24T03:17:24.058 INFO:teuthology.orchestra.run.smithi118.stdout: "flags": 0, 2024-03-24T03:17:24.058 INFO:teuthology.orchestra.run.smithi118.stdout: "default_value": "", 2024-03-24T03:17:24.058 INFO:teuthology.orchestra.run.smithi118.stdout: "min": "", 2024-03-24T03:17:24.058 INFO:teuthology.orchestra.run.smithi118.stdout: "max": "", 2024-03-24T03:17:24.058 INFO:teuthology.orchestra.run.smithi118.stdout: "enum_allowed": [], 2024-03-24T03:17:24.058 INFO:teuthology.orchestra.run.smithi118.stdout: "desc": "", 2024-03-24T03:17:24.059 INFO:teuthology.orchestra.run.smithi118.stdout: "long_desc": "", 2024-03-24T03:17:24.059 INFO:teuthology.orchestra.run.smithi118.stdout: "tags": [], 2024-03-24T03:17:24.059 INFO:teuthology.orchestra.run.smithi118.stdout: "see_also": [] 2024-03-24T03:17:24.059 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-24T03:17:24.059 INFO:teuthology.orchestra.run.smithi118.stdout: "ALERTMANAGER_API_SSL_VERIFY": { 2024-03-24T03:17:24.059 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "ALERTMANAGER_API_SSL_VERIFY", 2024-03-24T03:17:24.059 INFO:teuthology.orchestra.run.smithi118.stdout: "type": "bool", 2024-03-24T03:17:24.059 INFO:teuthology.orchestra.run.smithi118.stdout: "level": "advanced", 2024-03-24T03:17:24.059 INFO:teuthology.orchestra.run.smithi118.stdout: "flags": 0, 2024-03-24T03:17:24.060 INFO:teuthology.orchestra.run.smithi118.stdout: "default_value": "True", 2024-03-24T03:17:24.060 INFO:teuthology.orchestra.run.smithi118.stdout: "min": "", 2024-03-24T03:17:24.060 INFO:teuthology.orchestra.run.smithi118.stdout: "max": "", 2024-03-24T03:17:24.060 INFO:teuthology.orchestra.run.smithi118.stdout: "enum_allowed": [], 2024-03-24T03:17:24.060 INFO:teuthology.orchestra.run.smithi118.stdout: "desc": "", 2024-03-24T03:17:24.060 INFO:teuthology.orchestra.run.smithi118.stdout: "long_desc": "", 2024-03-24T03:17:24.060 INFO:teuthology.orchestra.run.smithi118.stdout: "tags": [], 2024-03-24T03:17:24.060 INFO:teuthology.orchestra.run.smithi118.stdout: "see_also": [] 2024-03-24T03:17:24.061 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-24T03:17:24.061 INFO:teuthology.orchestra.run.smithi118.stdout: "AUDIT_API_ENABLED": { 2024-03-24T03:17:24.061 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "AUDIT_API_ENABLED", 2024-03-24T03:17:24.061 INFO:teuthology.orchestra.run.smithi118.stdout: "type": "bool", 2024-03-24T03:17:24.061 INFO:teuthology.orchestra.run.smithi118.stdout: "level": "advanced", 2024-03-24T03:17:24.061 INFO:teuthology.orchestra.run.smithi118.stdout: "flags": 0, 2024-03-24T03:17:24.061 INFO:teuthology.orchestra.run.smithi118.stdout: "default_value": "False", 2024-03-24T03:17:24.061 INFO:teuthology.orchestra.run.smithi118.stdout: "min": "", 2024-03-24T03:17:24.061 INFO:teuthology.orchestra.run.smithi118.stdout: "max": "", 2024-03-24T03:17:24.062 INFO:teuthology.orchestra.run.smithi118.stdout: "enum_allowed": [], 2024-03-24T03:17:24.062 INFO:teuthology.orchestra.run.smithi118.stdout: "desc": "", 2024-03-24T03:17:24.062 INFO:teuthology.orchestra.run.smithi118.stdout: "long_desc": "", 2024-03-24T03:17:24.062 INFO:teuthology.orchestra.run.smithi118.stdout: "tags": [], 2024-03-24T03:17:24.062 INFO:teuthology.orchestra.run.smithi118.stdout: "see_also": [] 2024-03-24T03:17:24.062 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-24T03:17:24.062 INFO:teuthology.orchestra.run.smithi118.stdout: "AUDIT_API_LOG_PAYLOAD": { 2024-03-24T03:17:24.062 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "AUDIT_API_LOG_PAYLOAD", 2024-03-24T03:17:24.063 INFO:teuthology.orchestra.run.smithi118.stdout: "type": "bool", 2024-03-24T03:17:24.063 INFO:teuthology.orchestra.run.smithi118.stdout: "level": "advanced", 2024-03-24T03:17:24.063 INFO:teuthology.orchestra.run.smithi118.stdout: "flags": 0, 2024-03-24T03:17:24.063 INFO:teuthology.orchestra.run.smithi118.stdout: "default_value": "True", 2024-03-24T03:17:24.063 INFO:teuthology.orchestra.run.smithi118.stdout: "min": "", 2024-03-24T03:17:24.063 INFO:teuthology.orchestra.run.smithi118.stdout: "max": "", 2024-03-24T03:17:24.063 INFO:teuthology.orchestra.run.smithi118.stdout: "enum_allowed": [], 2024-03-24T03:17:24.063 INFO:teuthology.orchestra.run.smithi118.stdout: "desc": "", 2024-03-24T03:17:24.064 INFO:teuthology.orchestra.run.smithi118.stdout: "long_desc": "", 2024-03-24T03:17:24.064 INFO:teuthology.orchestra.run.smithi118.stdout: "tags": [], 2024-03-24T03:17:24.064 INFO:teuthology.orchestra.run.smithi118.stdout: "see_also": [] 2024-03-24T03:17:24.064 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-24T03:17:24.064 INFO:teuthology.orchestra.run.smithi118.stdout: "ENABLE_BROWSABLE_API": { 2024-03-24T03:17:24.064 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "ENABLE_BROWSABLE_API", 2024-03-24T03:17:24.064 INFO:teuthology.orchestra.run.smithi118.stdout: "type": "bool", 2024-03-24T03:17:24.064 INFO:teuthology.orchestra.run.smithi118.stdout: "level": "advanced", 2024-03-24T03:17:24.064 INFO:teuthology.orchestra.run.smithi118.stdout: "flags": 0, 2024-03-24T03:17:24.065 INFO:teuthology.orchestra.run.smithi118.stdout: "default_value": "True", 2024-03-24T03:17:24.065 INFO:teuthology.orchestra.run.smithi118.stdout: "min": "", 2024-03-24T03:17:24.065 INFO:teuthology.orchestra.run.smithi118.stdout: "max": "", 2024-03-24T03:17:24.065 INFO:teuthology.orchestra.run.smithi118.stdout: "enum_allowed": [], 2024-03-24T03:17:24.065 INFO:teuthology.orchestra.run.smithi118.stdout: "desc": "", 2024-03-24T03:17:24.065 INFO:teuthology.orchestra.run.smithi118.stdout: "long_desc": "", 2024-03-24T03:17:24.065 INFO:teuthology.orchestra.run.smithi118.stdout: "tags": [], 2024-03-24T03:17:24.065 INFO:teuthology.orchestra.run.smithi118.stdout: "see_also": [] 2024-03-24T03:17:24.065 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-24T03:17:24.066 INFO:teuthology.orchestra.run.smithi118.stdout: "FEATURE_TOGGLE_CEPHFS": { 2024-03-24T03:17:24.066 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "FEATURE_TOGGLE_CEPHFS", 2024-03-24T03:17:24.066 INFO:teuthology.orchestra.run.smithi118.stdout: "type": "bool", 2024-03-24T03:17:24.066 INFO:teuthology.orchestra.run.smithi118.stdout: "level": "advanced", 2024-03-24T03:17:24.066 INFO:teuthology.orchestra.run.smithi118.stdout: "flags": 0, 2024-03-24T03:17:24.066 INFO:teuthology.orchestra.run.smithi118.stdout: "default_value": "True", 2024-03-24T03:17:24.066 INFO:teuthology.orchestra.run.smithi118.stdout: "min": "", 2024-03-24T03:17:24.066 INFO:teuthology.orchestra.run.smithi118.stdout: "max": "", 2024-03-24T03:17:24.067 INFO:teuthology.orchestra.run.smithi118.stdout: "enum_allowed": [], 2024-03-24T03:17:24.067 INFO:teuthology.orchestra.run.smithi118.stdout: "desc": "", 2024-03-24T03:17:24.067 INFO:teuthology.orchestra.run.smithi118.stdout: "long_desc": "", 2024-03-24T03:17:24.067 INFO:teuthology.orchestra.run.smithi118.stdout: "tags": [], 2024-03-24T03:17:24.067 INFO:teuthology.orchestra.run.smithi118.stdout: "see_also": [] 2024-03-24T03:17:24.067 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-24T03:17:24.067 INFO:teuthology.orchestra.run.smithi118.stdout: "FEATURE_TOGGLE_DASHBOARD": { 2024-03-24T03:17:24.067 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "FEATURE_TOGGLE_DASHBOARD", 2024-03-24T03:17:24.067 INFO:teuthology.orchestra.run.smithi118.stdout: "type": "bool", 2024-03-24T03:17:24.068 INFO:teuthology.orchestra.run.smithi118.stdout: "level": "advanced", 2024-03-24T03:17:24.068 INFO:teuthology.orchestra.run.smithi118.stdout: "flags": 0, 2024-03-24T03:17:24.068 INFO:teuthology.orchestra.run.smithi118.stdout: "default_value": "True", 2024-03-24T03:17:24.068 INFO:teuthology.orchestra.run.smithi118.stdout: "min": "", 2024-03-24T03:17:24.068 INFO:teuthology.orchestra.run.smithi118.stdout: "max": "", 2024-03-24T03:17:24.068 INFO:teuthology.orchestra.run.smithi118.stdout: "enum_allowed": [], 2024-03-24T03:17:24.068 INFO:teuthology.orchestra.run.smithi118.stdout: "desc": "", 2024-03-24T03:17:24.068 INFO:teuthology.orchestra.run.smithi118.stdout: "long_desc": "", 2024-03-24T03:17:24.068 INFO:teuthology.orchestra.run.smithi118.stdout: "tags": [], 2024-03-24T03:17:24.068 INFO:teuthology.orchestra.run.smithi118.stdout: "see_also": [] 2024-03-24T03:17:24.069 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-24T03:17:24.069 INFO:teuthology.orchestra.run.smithi118.stdout: "FEATURE_TOGGLE_ISCSI": { 2024-03-24T03:17:24.069 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "FEATURE_TOGGLE_ISCSI", 2024-03-24T03:17:24.069 INFO:teuthology.orchestra.run.smithi118.stdout: "type": "bool", 2024-03-24T03:17:24.069 INFO:teuthology.orchestra.run.smithi118.stdout: "level": "advanced", 2024-03-24T03:17:24.069 INFO:teuthology.orchestra.run.smithi118.stdout: "flags": 0, 2024-03-24T03:17:24.069 INFO:teuthology.orchestra.run.smithi118.stdout: "default_value": "True", 2024-03-24T03:17:24.069 INFO:teuthology.orchestra.run.smithi118.stdout: "min": "", 2024-03-24T03:17:24.069 INFO:teuthology.orchestra.run.smithi118.stdout: "max": "", 2024-03-24T03:17:24.070 INFO:teuthology.orchestra.run.smithi118.stdout: "enum_allowed": [], 2024-03-24T03:17:24.070 INFO:teuthology.orchestra.run.smithi118.stdout: "desc": "", 2024-03-24T03:17:24.070 INFO:teuthology.orchestra.run.smithi118.stdout: "long_desc": "", 2024-03-24T03:17:24.070 INFO:teuthology.orchestra.run.smithi118.stdout: "tags": [], 2024-03-24T03:17:24.070 INFO:teuthology.orchestra.run.smithi118.stdout: "see_also": [] 2024-03-24T03:17:24.070 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-24T03:17:24.070 INFO:teuthology.orchestra.run.smithi118.stdout: "FEATURE_TOGGLE_MIRRORING": { 2024-03-24T03:17:24.070 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "FEATURE_TOGGLE_MIRRORING", 2024-03-24T03:17:24.070 INFO:teuthology.orchestra.run.smithi118.stdout: "type": "bool", 2024-03-24T03:17:24.071 INFO:teuthology.orchestra.run.smithi118.stdout: "level": "advanced", 2024-03-24T03:17:24.071 INFO:teuthology.orchestra.run.smithi118.stdout: "flags": 0, 2024-03-24T03:17:24.071 INFO:teuthology.orchestra.run.smithi118.stdout: "default_value": "True", 2024-03-24T03:17:24.071 INFO:teuthology.orchestra.run.smithi118.stdout: "min": "", 2024-03-24T03:17:24.071 INFO:teuthology.orchestra.run.smithi118.stdout: "max": "", 2024-03-24T03:17:24.071 INFO:teuthology.orchestra.run.smithi118.stdout: "enum_allowed": [], 2024-03-24T03:17:24.071 INFO:teuthology.orchestra.run.smithi118.stdout: "desc": "", 2024-03-24T03:17:24.071 INFO:teuthology.orchestra.run.smithi118.stdout: "long_desc": "", 2024-03-24T03:17:24.071 INFO:teuthology.orchestra.run.smithi118.stdout: "tags": [], 2024-03-24T03:17:24.072 INFO:teuthology.orchestra.run.smithi118.stdout: "see_also": [] 2024-03-24T03:17:24.072 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-24T03:17:24.072 INFO:teuthology.orchestra.run.smithi118.stdout: "FEATURE_TOGGLE_NFS": { 2024-03-24T03:17:24.072 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "FEATURE_TOGGLE_NFS", 2024-03-24T03:17:24.072 INFO:teuthology.orchestra.run.smithi118.stdout: "type": "bool", 2024-03-24T03:17:24.072 INFO:teuthology.orchestra.run.smithi118.stdout: "level": "advanced", 2024-03-24T03:17:24.072 INFO:teuthology.orchestra.run.smithi118.stdout: "flags": 0, 2024-03-24T03:17:24.072 INFO:teuthology.orchestra.run.smithi118.stdout: "default_value": "True", 2024-03-24T03:17:24.073 INFO:teuthology.orchestra.run.smithi118.stdout: "min": "", 2024-03-24T03:17:24.073 INFO:teuthology.orchestra.run.smithi118.stdout: "max": "", 2024-03-24T03:17:24.073 INFO:teuthology.orchestra.run.smithi118.stdout: "enum_allowed": [], 2024-03-24T03:17:24.073 INFO:teuthology.orchestra.run.smithi118.stdout: "desc": "", 2024-03-24T03:17:24.073 INFO:teuthology.orchestra.run.smithi118.stdout: "long_desc": "", 2024-03-24T03:17:24.073 INFO:teuthology.orchestra.run.smithi118.stdout: "tags": [], 2024-03-24T03:17:24.073 INFO:teuthology.orchestra.run.smithi118.stdout: "see_also": [] 2024-03-24T03:17:24.073 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-24T03:17:24.073 INFO:teuthology.orchestra.run.smithi118.stdout: "FEATURE_TOGGLE_RBD": { 2024-03-24T03:17:24.074 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "FEATURE_TOGGLE_RBD", 2024-03-24T03:17:24.074 INFO:teuthology.orchestra.run.smithi118.stdout: "type": "bool", 2024-03-24T03:17:24.074 INFO:teuthology.orchestra.run.smithi118.stdout: "level": "advanced", 2024-03-24T03:17:24.074 INFO:teuthology.orchestra.run.smithi118.stdout: "flags": 0, 2024-03-24T03:17:24.074 INFO:teuthology.orchestra.run.smithi118.stdout: "default_value": "True", 2024-03-24T03:17:24.074 INFO:teuthology.orchestra.run.smithi118.stdout: "min": "", 2024-03-24T03:17:24.074 INFO:teuthology.orchestra.run.smithi118.stdout: "max": "", 2024-03-24T03:17:24.074 INFO:teuthology.orchestra.run.smithi118.stdout: "enum_allowed": [], 2024-03-24T03:17:24.074 INFO:teuthology.orchestra.run.smithi118.stdout: "desc": "", 2024-03-24T03:17:24.075 INFO:teuthology.orchestra.run.smithi118.stdout: "long_desc": "", 2024-03-24T03:17:24.075 INFO:teuthology.orchestra.run.smithi118.stdout: "tags": [], 2024-03-24T03:17:24.075 INFO:teuthology.orchestra.run.smithi118.stdout: "see_also": [] 2024-03-24T03:17:24.075 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-24T03:17:24.075 INFO:teuthology.orchestra.run.smithi118.stdout: "FEATURE_TOGGLE_RGW": { 2024-03-24T03:17:24.075 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "FEATURE_TOGGLE_RGW", 2024-03-24T03:17:24.075 INFO:teuthology.orchestra.run.smithi118.stdout: "type": "bool", 2024-03-24T03:17:24.075 INFO:teuthology.orchestra.run.smithi118.stdout: "level": "advanced", 2024-03-24T03:17:24.075 INFO:teuthology.orchestra.run.smithi118.stdout: "flags": 0, 2024-03-24T03:17:24.076 INFO:teuthology.orchestra.run.smithi118.stdout: "default_value": "True", 2024-03-24T03:17:24.076 INFO:teuthology.orchestra.run.smithi118.stdout: "min": "", 2024-03-24T03:17:24.076 INFO:teuthology.orchestra.run.smithi118.stdout: "max": "", 2024-03-24T03:17:24.076 INFO:teuthology.orchestra.run.smithi118.stdout: "enum_allowed": [], 2024-03-24T03:17:24.076 INFO:teuthology.orchestra.run.smithi118.stdout: "desc": "", 2024-03-24T03:17:24.076 INFO:teuthology.orchestra.run.smithi118.stdout: "long_desc": "", 2024-03-24T03:17:24.076 INFO:teuthology.orchestra.run.smithi118.stdout: "tags": [], 2024-03-24T03:17:24.076 INFO:teuthology.orchestra.run.smithi118.stdout: "see_also": [] 2024-03-24T03:17:24.076 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-24T03:17:24.077 INFO:teuthology.orchestra.run.smithi118.stdout: "GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE": { 2024-03-24T03:17:24.077 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE", 2024-03-24T03:17:24.077 INFO:teuthology.orchestra.run.smithi118.stdout: "type": "str", 2024-03-24T03:17:24.077 INFO:teuthology.orchestra.run.smithi118.stdout: "level": "advanced", 2024-03-24T03:17:24.077 INFO:teuthology.orchestra.run.smithi118.stdout: "flags": 0, 2024-03-24T03:17:24.077 INFO:teuthology.orchestra.run.smithi118.stdout: "default_value": "", 2024-03-24T03:17:24.077 INFO:teuthology.orchestra.run.smithi118.stdout: "min": "", 2024-03-24T03:17:24.077 INFO:teuthology.orchestra.run.smithi118.stdout: "max": "", 2024-03-24T03:17:24.077 INFO:teuthology.orchestra.run.smithi118.stdout: "enum_allowed": [], 2024-03-24T03:17:24.078 INFO:teuthology.orchestra.run.smithi118.stdout: "desc": "", 2024-03-24T03:17:24.078 INFO:teuthology.orchestra.run.smithi118.stdout: "long_desc": "", 2024-03-24T03:17:24.078 INFO:teuthology.orchestra.run.smithi118.stdout: "tags": [], 2024-03-24T03:17:24.078 INFO:teuthology.orchestra.run.smithi118.stdout: "see_also": [] 2024-03-24T03:17:24.078 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-24T03:17:24.078 INFO:teuthology.orchestra.run.smithi118.stdout: "GRAFANA_API_PASSWORD": { 2024-03-24T03:17:24.078 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "GRAFANA_API_PASSWORD", 2024-03-24T03:17:24.078 INFO:teuthology.orchestra.run.smithi118.stdout: "type": "str", 2024-03-24T03:17:24.078 INFO:teuthology.orchestra.run.smithi118.stdout: "level": "advanced", 2024-03-24T03:17:24.079 INFO:teuthology.orchestra.run.smithi118.stdout: "flags": 0, 2024-03-24T03:17:24.079 INFO:teuthology.orchestra.run.smithi118.stdout: "default_value": "admin", 2024-03-24T03:17:24.079 INFO:teuthology.orchestra.run.smithi118.stdout: "min": "", 2024-03-24T03:17:24.079 INFO:teuthology.orchestra.run.smithi118.stdout: "max": "", 2024-03-24T03:17:24.079 INFO:teuthology.orchestra.run.smithi118.stdout: "enum_allowed": [], 2024-03-24T03:17:24.079 INFO:teuthology.orchestra.run.smithi118.stdout: "desc": "", 2024-03-24T03:17:24.079 INFO:teuthology.orchestra.run.smithi118.stdout: "long_desc": "", 2024-03-24T03:17:24.079 INFO:teuthology.orchestra.run.smithi118.stdout: "tags": [], 2024-03-24T03:17:24.080 INFO:teuthology.orchestra.run.smithi118.stdout: "see_also": [] 2024-03-24T03:17:24.080 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-24T03:17:24.080 INFO:teuthology.orchestra.run.smithi118.stdout: "GRAFANA_API_SSL_VERIFY": { 2024-03-24T03:17:24.080 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "GRAFANA_API_SSL_VERIFY", 2024-03-24T03:17:24.080 INFO:teuthology.orchestra.run.smithi118.stdout: "type": "bool", 2024-03-24T03:17:24.080 INFO:teuthology.orchestra.run.smithi118.stdout: "level": "advanced", 2024-03-24T03:17:24.080 INFO:teuthology.orchestra.run.smithi118.stdout: "flags": 0, 2024-03-24T03:17:24.080 INFO:teuthology.orchestra.run.smithi118.stdout: "default_value": "True", 2024-03-24T03:17:24.081 INFO:teuthology.orchestra.run.smithi118.stdout: "min": "", 2024-03-24T03:17:24.081 INFO:teuthology.orchestra.run.smithi118.stdout: "max": "", 2024-03-24T03:17:24.081 INFO:teuthology.orchestra.run.smithi118.stdout: "enum_allowed": [], 2024-03-24T03:17:24.081 INFO:teuthology.orchestra.run.smithi118.stdout: "desc": "", 2024-03-24T03:17:24.081 INFO:teuthology.orchestra.run.smithi118.stdout: "long_desc": "", 2024-03-24T03:17:24.081 INFO:teuthology.orchestra.run.smithi118.stdout: "tags": [], 2024-03-24T03:17:24.081 INFO:teuthology.orchestra.run.smithi118.stdout: "see_also": [] 2024-03-24T03:17:24.081 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-24T03:17:24.081 INFO:teuthology.orchestra.run.smithi118.stdout: "GRAFANA_API_URL": { 2024-03-24T03:17:24.081 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "GRAFANA_API_URL", 2024-03-24T03:17:24.082 INFO:teuthology.orchestra.run.smithi118.stdout: "type": "str", 2024-03-24T03:17:24.082 INFO:teuthology.orchestra.run.smithi118.stdout: "level": "advanced", 2024-03-24T03:17:24.082 INFO:teuthology.orchestra.run.smithi118.stdout: "flags": 0, 2024-03-24T03:17:24.082 INFO:teuthology.orchestra.run.smithi118.stdout: "default_value": "", 2024-03-24T03:17:24.082 INFO:teuthology.orchestra.run.smithi118.stdout: "min": "", 2024-03-24T03:17:24.082 INFO:teuthology.orchestra.run.smithi118.stdout: "max": "", 2024-03-24T03:17:24.082 INFO:teuthology.orchestra.run.smithi118.stdout: "enum_allowed": [], 2024-03-24T03:17:24.082 INFO:teuthology.orchestra.run.smithi118.stdout: "desc": "", 2024-03-24T03:17:24.082 INFO:teuthology.orchestra.run.smithi118.stdout: "long_desc": "", 2024-03-24T03:17:24.083 INFO:teuthology.orchestra.run.smithi118.stdout: "tags": [], 2024-03-24T03:17:24.083 INFO:teuthology.orchestra.run.smithi118.stdout: "see_also": [] 2024-03-24T03:17:24.083 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-24T03:17:24.083 INFO:teuthology.orchestra.run.smithi118.stdout: "GRAFANA_API_USERNAME": { 2024-03-24T03:17:24.083 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "GRAFANA_API_USERNAME", 2024-03-24T03:17:24.083 INFO:teuthology.orchestra.run.smithi118.stdout: "type": "str", 2024-03-24T03:17:24.083 INFO:teuthology.orchestra.run.smithi118.stdout: "level": "advanced", 2024-03-24T03:17:24.083 INFO:teuthology.orchestra.run.smithi118.stdout: "flags": 0, 2024-03-24T03:17:24.083 INFO:teuthology.orchestra.run.smithi118.stdout: "default_value": "admin", 2024-03-24T03:17:24.084 INFO:teuthology.orchestra.run.smithi118.stdout: "min": "", 2024-03-24T03:17:24.084 INFO:teuthology.orchestra.run.smithi118.stdout: "max": "", 2024-03-24T03:17:24.084 INFO:teuthology.orchestra.run.smithi118.stdout: "enum_allowed": [], 2024-03-24T03:17:24.084 INFO:teuthology.orchestra.run.smithi118.stdout: "desc": "", 2024-03-24T03:17:24.084 INFO:teuthology.orchestra.run.smithi118.stdout: "long_desc": "", 2024-03-24T03:17:24.084 INFO:teuthology.orchestra.run.smithi118.stdout: "tags": [], 2024-03-24T03:17:24.084 INFO:teuthology.orchestra.run.smithi118.stdout: "see_also": [] 2024-03-24T03:17:24.084 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-24T03:17:24.084 INFO:teuthology.orchestra.run.smithi118.stdout: "GRAFANA_FRONTEND_API_URL": { 2024-03-24T03:17:24.085 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "GRAFANA_FRONTEND_API_URL", 2024-03-24T03:17:24.085 INFO:teuthology.orchestra.run.smithi118.stdout: "type": "str", 2024-03-24T03:17:24.085 INFO:teuthology.orchestra.run.smithi118.stdout: "level": "advanced", 2024-03-24T03:17:24.085 INFO:teuthology.orchestra.run.smithi118.stdout: "flags": 0, 2024-03-24T03:17:24.085 INFO:teuthology.orchestra.run.smithi118.stdout: "default_value": "", 2024-03-24T03:17:24.085 INFO:teuthology.orchestra.run.smithi118.stdout: "min": "", 2024-03-24T03:17:24.085 INFO:teuthology.orchestra.run.smithi118.stdout: "max": "", 2024-03-24T03:17:24.085 INFO:teuthology.orchestra.run.smithi118.stdout: "enum_allowed": [], 2024-03-24T03:17:24.085 INFO:teuthology.orchestra.run.smithi118.stdout: "desc": "", 2024-03-24T03:17:24.086 INFO:teuthology.orchestra.run.smithi118.stdout: "long_desc": "", 2024-03-24T03:17:24.086 INFO:teuthology.orchestra.run.smithi118.stdout: "tags": [], 2024-03-24T03:17:24.086 INFO:teuthology.orchestra.run.smithi118.stdout: "see_also": [] 2024-03-24T03:17:24.086 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-24T03:17:24.086 INFO:teuthology.orchestra.run.smithi118.stdout: "GRAFANA_UPDATE_DASHBOARDS": { 2024-03-24T03:17:24.086 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "GRAFANA_UPDATE_DASHBOARDS", 2024-03-24T03:17:24.086 INFO:teuthology.orchestra.run.smithi118.stdout: "type": "bool", 2024-03-24T03:17:24.086 INFO:teuthology.orchestra.run.smithi118.stdout: "level": "advanced", 2024-03-24T03:17:24.086 INFO:teuthology.orchestra.run.smithi118.stdout: "flags": 0, 2024-03-24T03:17:24.087 INFO:teuthology.orchestra.run.smithi118.stdout: "default_value": "False", 2024-03-24T03:17:24.087 INFO:teuthology.orchestra.run.smithi118.stdout: "min": "", 2024-03-24T03:17:24.087 INFO:teuthology.orchestra.run.smithi118.stdout: "max": "", 2024-03-24T03:17:24.087 INFO:teuthology.orchestra.run.smithi118.stdout: "enum_allowed": [], 2024-03-24T03:17:24.087 INFO:teuthology.orchestra.run.smithi118.stdout: "desc": "", 2024-03-24T03:17:24.087 INFO:teuthology.orchestra.run.smithi118.stdout: "long_desc": "", 2024-03-24T03:17:24.087 INFO:teuthology.orchestra.run.smithi118.stdout: "tags": [], 2024-03-24T03:17:24.087 INFO:teuthology.orchestra.run.smithi118.stdout: "see_also": [] 2024-03-24T03:17:24.087 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-24T03:17:24.088 INFO:teuthology.orchestra.run.smithi118.stdout: "ISCSI_API_SSL_VERIFICATION": { 2024-03-24T03:17:24.088 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "ISCSI_API_SSL_VERIFICATION", 2024-03-24T03:17:24.088 INFO:teuthology.orchestra.run.smithi118.stdout: "type": "bool", 2024-03-24T03:17:24.088 INFO:teuthology.orchestra.run.smithi118.stdout: "level": "advanced", 2024-03-24T03:17:24.088 INFO:teuthology.orchestra.run.smithi118.stdout: "flags": 0, 2024-03-24T03:17:24.088 INFO:teuthology.orchestra.run.smithi118.stdout: "default_value": "True", 2024-03-24T03:17:24.088 INFO:teuthology.orchestra.run.smithi118.stdout: "min": "", 2024-03-24T03:17:24.088 INFO:teuthology.orchestra.run.smithi118.stdout: "max": "", 2024-03-24T03:17:24.088 INFO:teuthology.orchestra.run.smithi118.stdout: "enum_allowed": [], 2024-03-24T03:17:24.089 INFO:teuthology.orchestra.run.smithi118.stdout: "desc": "", 2024-03-24T03:17:24.089 INFO:teuthology.orchestra.run.smithi118.stdout: "long_desc": "", 2024-03-24T03:17:24.089 INFO:teuthology.orchestra.run.smithi118.stdout: "tags": [], 2024-03-24T03:17:24.089 INFO:teuthology.orchestra.run.smithi118.stdout: "see_also": [] 2024-03-24T03:17:24.089 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-24T03:17:24.089 INFO:teuthology.orchestra.run.smithi118.stdout: "ISSUE_TRACKER_API_KEY": { 2024-03-24T03:17:24.089 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "ISSUE_TRACKER_API_KEY", 2024-03-24T03:17:24.089 INFO:teuthology.orchestra.run.smithi118.stdout: "type": "str", 2024-03-24T03:17:24.089 INFO:teuthology.orchestra.run.smithi118.stdout: "level": "advanced", 2024-03-24T03:17:24.090 INFO:teuthology.orchestra.run.smithi118.stdout: "flags": 0, 2024-03-24T03:17:24.090 INFO:teuthology.orchestra.run.smithi118.stdout: "default_value": "", 2024-03-24T03:17:24.090 INFO:teuthology.orchestra.run.smithi118.stdout: "min": "", 2024-03-24T03:17:24.090 INFO:teuthology.orchestra.run.smithi118.stdout: "max": "", 2024-03-24T03:17:24.090 INFO:teuthology.orchestra.run.smithi118.stdout: "enum_allowed": [], 2024-03-24T03:17:24.090 INFO:teuthology.orchestra.run.smithi118.stdout: "desc": "", 2024-03-24T03:17:24.090 INFO:teuthology.orchestra.run.smithi118.stdout: "long_desc": "", 2024-03-24T03:17:24.090 INFO:teuthology.orchestra.run.smithi118.stdout: "tags": [], 2024-03-24T03:17:24.090 INFO:teuthology.orchestra.run.smithi118.stdout: "see_also": [] 2024-03-24T03:17:24.091 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-24T03:17:24.091 INFO:teuthology.orchestra.run.smithi118.stdout: "MULTICLUSTER_CONFIG": { 2024-03-24T03:17:24.091 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "MULTICLUSTER_CONFIG", 2024-03-24T03:17:24.091 INFO:teuthology.orchestra.run.smithi118.stdout: "type": "str", 2024-03-24T03:17:24.091 INFO:teuthology.orchestra.run.smithi118.stdout: "level": "advanced", 2024-03-24T03:17:24.091 INFO:teuthology.orchestra.run.smithi118.stdout: "flags": 0, 2024-03-24T03:17:24.091 INFO:teuthology.orchestra.run.smithi118.stdout: "default_value": "{}", 2024-03-24T03:17:24.091 INFO:teuthology.orchestra.run.smithi118.stdout: "min": "", 2024-03-24T03:17:24.092 INFO:teuthology.orchestra.run.smithi118.stdout: "max": "", 2024-03-24T03:17:24.092 INFO:teuthology.orchestra.run.smithi118.stdout: "enum_allowed": [], 2024-03-24T03:17:24.092 INFO:teuthology.orchestra.run.smithi118.stdout: "desc": "", 2024-03-24T03:17:24.092 INFO:teuthology.orchestra.run.smithi118.stdout: "long_desc": "", 2024-03-24T03:17:24.092 INFO:teuthology.orchestra.run.smithi118.stdout: "tags": [], 2024-03-24T03:17:24.092 INFO:teuthology.orchestra.run.smithi118.stdout: "see_also": [] 2024-03-24T03:17:24.092 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-24T03:17:24.092 INFO:teuthology.orchestra.run.smithi118.stdout: "PROMETHEUS_API_HOST": { 2024-03-24T03:17:24.093 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "PROMETHEUS_API_HOST", 2024-03-24T03:17:24.093 INFO:teuthology.orchestra.run.smithi118.stdout: "type": "str", 2024-03-24T03:17:24.093 INFO:teuthology.orchestra.run.smithi118.stdout: "level": "advanced", 2024-03-24T03:17:24.093 INFO:teuthology.orchestra.run.smithi118.stdout: "flags": 0, 2024-03-24T03:17:24.093 INFO:teuthology.orchestra.run.smithi118.stdout: "default_value": "", 2024-03-24T03:17:24.093 INFO:teuthology.orchestra.run.smithi118.stdout: "min": "", 2024-03-24T03:17:24.093 INFO:teuthology.orchestra.run.smithi118.stdout: "max": "", 2024-03-24T03:17:24.093 INFO:teuthology.orchestra.run.smithi118.stdout: "enum_allowed": [], 2024-03-24T03:17:24.093 INFO:teuthology.orchestra.run.smithi118.stdout: "desc": "", 2024-03-24T03:17:24.093 INFO:teuthology.orchestra.run.smithi118.stdout: "long_desc": "", 2024-03-24T03:17:24.094 INFO:teuthology.orchestra.run.smithi118.stdout: "tags": [], 2024-03-24T03:17:24.094 INFO:teuthology.orchestra.run.smithi118.stdout: "see_also": [] 2024-03-24T03:17:24.094 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-24T03:17:24.094 INFO:teuthology.orchestra.run.smithi118.stdout: "PROMETHEUS_API_SSL_VERIFY": { 2024-03-24T03:17:24.094 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "PROMETHEUS_API_SSL_VERIFY", 2024-03-24T03:17:24.094 INFO:teuthology.orchestra.run.smithi118.stdout: "type": "bool", 2024-03-24T03:17:24.094 INFO:teuthology.orchestra.run.smithi118.stdout: "level": "advanced", 2024-03-24T03:17:24.095 INFO:teuthology.orchestra.run.smithi118.stdout: "flags": 0, 2024-03-24T03:17:24.095 INFO:teuthology.orchestra.run.smithi118.stdout: "default_value": "True", 2024-03-24T03:17:24.095 INFO:teuthology.orchestra.run.smithi118.stdout: "min": "", 2024-03-24T03:17:24.095 INFO:teuthology.orchestra.run.smithi118.stdout: "max": "", 2024-03-24T03:17:24.095 INFO:teuthology.orchestra.run.smithi118.stdout: "enum_allowed": [], 2024-03-24T03:17:24.095 INFO:teuthology.orchestra.run.smithi118.stdout: "desc": "", 2024-03-24T03:17:24.095 INFO:teuthology.orchestra.run.smithi118.stdout: "long_desc": "", 2024-03-24T03:17:24.095 INFO:teuthology.orchestra.run.smithi118.stdout: "tags": [], 2024-03-24T03:17:24.096 INFO:teuthology.orchestra.run.smithi118.stdout: "see_also": [] 2024-03-24T03:17:24.096 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-24T03:17:24.096 INFO:teuthology.orchestra.run.smithi118.stdout: "PWD_POLICY_CHECK_COMPLEXITY_ENABLED": { 2024-03-24T03:17:24.096 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "PWD_POLICY_CHECK_COMPLEXITY_ENABLED", 2024-03-24T03:17:24.096 INFO:teuthology.orchestra.run.smithi118.stdout: "type": "bool", 2024-03-24T03:17:24.096 INFO:teuthology.orchestra.run.smithi118.stdout: "level": "advanced", 2024-03-24T03:17:24.096 INFO:teuthology.orchestra.run.smithi118.stdout: "flags": 0, 2024-03-24T03:17:24.096 INFO:teuthology.orchestra.run.smithi118.stdout: "default_value": "False", 2024-03-24T03:17:24.096 INFO:teuthology.orchestra.run.smithi118.stdout: "min": "", 2024-03-24T03:17:24.097 INFO:teuthology.orchestra.run.smithi118.stdout: "max": "", 2024-03-24T03:17:24.097 INFO:teuthology.orchestra.run.smithi118.stdout: "enum_allowed": [], 2024-03-24T03:17:24.097 INFO:teuthology.orchestra.run.smithi118.stdout: "desc": "", 2024-03-24T03:17:24.097 INFO:teuthology.orchestra.run.smithi118.stdout: "long_desc": "", 2024-03-24T03:17:24.097 INFO:teuthology.orchestra.run.smithi118.stdout: "tags": [], 2024-03-24T03:17:24.097 INFO:teuthology.orchestra.run.smithi118.stdout: "see_also": [] 2024-03-24T03:17:24.097 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-24T03:17:24.097 INFO:teuthology.orchestra.run.smithi118.stdout: "PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED": { 2024-03-24T03:17:24.097 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED", 2024-03-24T03:17:24.098 INFO:teuthology.orchestra.run.smithi118.stdout: "type": "bool", 2024-03-24T03:17:24.098 INFO:teuthology.orchestra.run.smithi118.stdout: "level": "advanced", 2024-03-24T03:17:24.098 INFO:teuthology.orchestra.run.smithi118.stdout: "flags": 0, 2024-03-24T03:17:24.098 INFO:teuthology.orchestra.run.smithi118.stdout: "default_value": "False", 2024-03-24T03:17:24.098 INFO:teuthology.orchestra.run.smithi118.stdout: "min": "", 2024-03-24T03:17:24.098 INFO:teuthology.orchestra.run.smithi118.stdout: "max": "", 2024-03-24T03:17:24.098 INFO:teuthology.orchestra.run.smithi118.stdout: "enum_allowed": [], 2024-03-24T03:17:24.098 INFO:teuthology.orchestra.run.smithi118.stdout: "desc": "", 2024-03-24T03:17:24.099 INFO:teuthology.orchestra.run.smithi118.stdout: "long_desc": "", 2024-03-24T03:17:24.099 INFO:teuthology.orchestra.run.smithi118.stdout: "tags": [], 2024-03-24T03:17:24.099 INFO:teuthology.orchestra.run.smithi118.stdout: "see_also": [] 2024-03-24T03:17:24.099 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-24T03:17:24.099 INFO:teuthology.orchestra.run.smithi118.stdout: "PWD_POLICY_CHECK_LENGTH_ENABLED": { 2024-03-24T03:17:24.099 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "PWD_POLICY_CHECK_LENGTH_ENABLED", 2024-03-24T03:17:24.099 INFO:teuthology.orchestra.run.smithi118.stdout: "type": "bool", 2024-03-24T03:17:24.099 INFO:teuthology.orchestra.run.smithi118.stdout: "level": "advanced", 2024-03-24T03:17:24.099 INFO:teuthology.orchestra.run.smithi118.stdout: "flags": 0, 2024-03-24T03:17:24.100 INFO:teuthology.orchestra.run.smithi118.stdout: "default_value": "True", 2024-03-24T03:17:24.100 INFO:teuthology.orchestra.run.smithi118.stdout: "min": "", 2024-03-24T03:17:24.100 INFO:teuthology.orchestra.run.smithi118.stdout: "max": "", 2024-03-24T03:17:24.100 INFO:teuthology.orchestra.run.smithi118.stdout: "enum_allowed": [], 2024-03-24T03:17:24.100 INFO:teuthology.orchestra.run.smithi118.stdout: "desc": "", 2024-03-24T03:17:24.100 INFO:teuthology.orchestra.run.smithi118.stdout: "long_desc": "", 2024-03-24T03:17:24.100 INFO:teuthology.orchestra.run.smithi118.stdout: "tags": [], 2024-03-24T03:17:24.100 INFO:teuthology.orchestra.run.smithi118.stdout: "see_also": [] 2024-03-24T03:17:24.100 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-24T03:17:24.101 INFO:teuthology.orchestra.run.smithi118.stdout: "PWD_POLICY_CHECK_OLDPWD_ENABLED": { 2024-03-24T03:17:24.101 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "PWD_POLICY_CHECK_OLDPWD_ENABLED", 2024-03-24T03:17:24.101 INFO:teuthology.orchestra.run.smithi118.stdout: "type": "bool", 2024-03-24T03:17:24.101 INFO:teuthology.orchestra.run.smithi118.stdout: "level": "advanced", 2024-03-24T03:17:24.101 INFO:teuthology.orchestra.run.smithi118.stdout: "flags": 0, 2024-03-24T03:17:24.101 INFO:teuthology.orchestra.run.smithi118.stdout: "default_value": "True", 2024-03-24T03:17:24.101 INFO:teuthology.orchestra.run.smithi118.stdout: "min": "", 2024-03-24T03:17:24.101 INFO:teuthology.orchestra.run.smithi118.stdout: "max": "", 2024-03-24T03:17:24.101 INFO:teuthology.orchestra.run.smithi118.stdout: "enum_allowed": [], 2024-03-24T03:17:24.102 INFO:teuthology.orchestra.run.smithi118.stdout: "desc": "", 2024-03-24T03:17:24.102 INFO:teuthology.orchestra.run.smithi118.stdout: "long_desc": "", 2024-03-24T03:17:24.102 INFO:teuthology.orchestra.run.smithi118.stdout: "tags": [], 2024-03-24T03:17:24.102 INFO:teuthology.orchestra.run.smithi118.stdout: "see_also": [] 2024-03-24T03:17:24.102 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-24T03:17:24.102 INFO:teuthology.orchestra.run.smithi118.stdout: "PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED": { 2024-03-24T03:17:24.102 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED", 2024-03-24T03:17:24.102 INFO:teuthology.orchestra.run.smithi118.stdout: "type": "bool", 2024-03-24T03:17:24.102 INFO:teuthology.orchestra.run.smithi118.stdout: "level": "advanced", 2024-03-24T03:17:24.103 INFO:teuthology.orchestra.run.smithi118.stdout: "flags": 0, 2024-03-24T03:17:24.103 INFO:teuthology.orchestra.run.smithi118.stdout: "default_value": "False", 2024-03-24T03:17:24.103 INFO:teuthology.orchestra.run.smithi118.stdout: "min": "", 2024-03-24T03:17:24.103 INFO:teuthology.orchestra.run.smithi118.stdout: "max": "", 2024-03-24T03:17:24.103 INFO:teuthology.orchestra.run.smithi118.stdout: "enum_allowed": [], 2024-03-24T03:17:24.103 INFO:teuthology.orchestra.run.smithi118.stdout: "desc": "", 2024-03-24T03:17:24.103 INFO:teuthology.orchestra.run.smithi118.stdout: "long_desc": "", 2024-03-24T03:17:24.103 INFO:teuthology.orchestra.run.smithi118.stdout: "tags": [], 2024-03-24T03:17:24.103 INFO:teuthology.orchestra.run.smithi118.stdout: "see_also": [] 2024-03-24T03:17:24.104 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-24T03:17:24.104 INFO:teuthology.orchestra.run.smithi118.stdout: "PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED": { 2024-03-24T03:17:24.104 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED", 2024-03-24T03:17:24.104 INFO:teuthology.orchestra.run.smithi118.stdout: "type": "bool", 2024-03-24T03:17:24.104 INFO:teuthology.orchestra.run.smithi118.stdout: "level": "advanced", 2024-03-24T03:17:24.104 INFO:teuthology.orchestra.run.smithi118.stdout: "flags": 0, 2024-03-24T03:17:24.104 INFO:teuthology.orchestra.run.smithi118.stdout: "default_value": "False", 2024-03-24T03:17:24.104 INFO:teuthology.orchestra.run.smithi118.stdout: "min": "", 2024-03-24T03:17:24.104 INFO:teuthology.orchestra.run.smithi118.stdout: "max": "", 2024-03-24T03:17:24.105 INFO:teuthology.orchestra.run.smithi118.stdout: "enum_allowed": [], 2024-03-24T03:17:24.105 INFO:teuthology.orchestra.run.smithi118.stdout: "desc": "", 2024-03-24T03:17:24.105 INFO:teuthology.orchestra.run.smithi118.stdout: "long_desc": "", 2024-03-24T03:17:24.105 INFO:teuthology.orchestra.run.smithi118.stdout: "tags": [], 2024-03-24T03:17:24.105 INFO:teuthology.orchestra.run.smithi118.stdout: "see_also": [] 2024-03-24T03:17:24.105 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-24T03:17:24.105 INFO:teuthology.orchestra.run.smithi118.stdout: "PWD_POLICY_CHECK_USERNAME_ENABLED": { 2024-03-24T03:17:24.105 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "PWD_POLICY_CHECK_USERNAME_ENABLED", 2024-03-24T03:17:24.105 INFO:teuthology.orchestra.run.smithi118.stdout: "type": "bool", 2024-03-24T03:17:24.106 INFO:teuthology.orchestra.run.smithi118.stdout: "level": "advanced", 2024-03-24T03:17:24.106 INFO:teuthology.orchestra.run.smithi118.stdout: "flags": 0, 2024-03-24T03:17:24.106 INFO:teuthology.orchestra.run.smithi118.stdout: "default_value": "False", 2024-03-24T03:17:24.106 INFO:teuthology.orchestra.run.smithi118.stdout: "min": "", 2024-03-24T03:17:24.106 INFO:teuthology.orchestra.run.smithi118.stdout: "max": "", 2024-03-24T03:17:24.106 INFO:teuthology.orchestra.run.smithi118.stdout: "enum_allowed": [], 2024-03-24T03:17:24.106 INFO:teuthology.orchestra.run.smithi118.stdout: "desc": "", 2024-03-24T03:17:24.106 INFO:teuthology.orchestra.run.smithi118.stdout: "long_desc": "", 2024-03-24T03:17:24.106 INFO:teuthology.orchestra.run.smithi118.stdout: "tags": [], 2024-03-24T03:17:24.107 INFO:teuthology.orchestra.run.smithi118.stdout: "see_also": [] 2024-03-24T03:17:24.107 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-24T03:17:24.107 INFO:teuthology.orchestra.run.smithi118.stdout: "PWD_POLICY_ENABLED": { 2024-03-24T03:17:24.107 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "PWD_POLICY_ENABLED", 2024-03-24T03:17:24.107 INFO:teuthology.orchestra.run.smithi118.stdout: "type": "bool", 2024-03-24T03:17:24.107 INFO:teuthology.orchestra.run.smithi118.stdout: "level": "advanced", 2024-03-24T03:17:24.107 INFO:teuthology.orchestra.run.smithi118.stdout: "flags": 0, 2024-03-24T03:17:24.107 INFO:teuthology.orchestra.run.smithi118.stdout: "default_value": "True", 2024-03-24T03:17:24.107 INFO:teuthology.orchestra.run.smithi118.stdout: "min": "", 2024-03-24T03:17:24.107 INFO:teuthology.orchestra.run.smithi118.stdout: "max": "", 2024-03-24T03:17:24.108 INFO:teuthology.orchestra.run.smithi118.stdout: "enum_allowed": [], 2024-03-24T03:17:24.108 INFO:teuthology.orchestra.run.smithi118.stdout: "desc": "", 2024-03-24T03:17:24.108 INFO:teuthology.orchestra.run.smithi118.stdout: "long_desc": "", 2024-03-24T03:17:24.108 INFO:teuthology.orchestra.run.smithi118.stdout: "tags": [], 2024-03-24T03:17:24.108 INFO:teuthology.orchestra.run.smithi118.stdout: "see_also": [] 2024-03-24T03:17:24.108 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-24T03:17:24.108 INFO:teuthology.orchestra.run.smithi118.stdout: "PWD_POLICY_EXCLUSION_LIST": { 2024-03-24T03:17:24.108 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "PWD_POLICY_EXCLUSION_LIST", 2024-03-24T03:17:24.108 INFO:teuthology.orchestra.run.smithi118.stdout: "type": "str", 2024-03-24T03:17:24.109 INFO:teuthology.orchestra.run.smithi118.stdout: "level": "advanced", 2024-03-24T03:17:24.109 INFO:teuthology.orchestra.run.smithi118.stdout: "flags": 0, 2024-03-24T03:17:24.109 INFO:teuthology.orchestra.run.smithi118.stdout: "default_value": "osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps", 2024-03-24T03:17:24.109 INFO:teuthology.orchestra.run.smithi118.stdout: "min": "", 2024-03-24T03:17:24.109 INFO:teuthology.orchestra.run.smithi118.stdout: "max": "", 2024-03-24T03:17:24.109 INFO:teuthology.orchestra.run.smithi118.stdout: "enum_allowed": [], 2024-03-24T03:17:24.109 INFO:teuthology.orchestra.run.smithi118.stdout: "desc": "", 2024-03-24T03:17:24.109 INFO:teuthology.orchestra.run.smithi118.stdout: "long_desc": "", 2024-03-24T03:17:24.109 INFO:teuthology.orchestra.run.smithi118.stdout: "tags": [], 2024-03-24T03:17:24.109 INFO:teuthology.orchestra.run.smithi118.stdout: "see_also": [] 2024-03-24T03:17:24.110 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-24T03:17:24.110 INFO:teuthology.orchestra.run.smithi118.stdout: "PWD_POLICY_MIN_COMPLEXITY": { 2024-03-24T03:17:24.110 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "PWD_POLICY_MIN_COMPLEXITY", 2024-03-24T03:17:24.110 INFO:teuthology.orchestra.run.smithi118.stdout: "type": "int", 2024-03-24T03:17:24.110 INFO:teuthology.orchestra.run.smithi118.stdout: "level": "advanced", 2024-03-24T03:17:24.110 INFO:teuthology.orchestra.run.smithi118.stdout: "flags": 0, 2024-03-24T03:17:24.110 INFO:teuthology.orchestra.run.smithi118.stdout: "default_value": "10", 2024-03-24T03:17:24.110 INFO:teuthology.orchestra.run.smithi118.stdout: "min": "", 2024-03-24T03:17:24.110 INFO:teuthology.orchestra.run.smithi118.stdout: "max": "", 2024-03-24T03:17:24.111 INFO:teuthology.orchestra.run.smithi118.stdout: "enum_allowed": [], 2024-03-24T03:17:24.111 INFO:teuthology.orchestra.run.smithi118.stdout: "desc": "", 2024-03-24T03:17:24.111 INFO:teuthology.orchestra.run.smithi118.stdout: "long_desc": "", 2024-03-24T03:17:24.111 INFO:teuthology.orchestra.run.smithi118.stdout: "tags": [], 2024-03-24T03:17:24.111 INFO:teuthology.orchestra.run.smithi118.stdout: "see_also": [] 2024-03-24T03:17:24.111 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-24T03:17:24.111 INFO:teuthology.orchestra.run.smithi118.stdout: "PWD_POLICY_MIN_LENGTH": { 2024-03-24T03:17:24.111 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "PWD_POLICY_MIN_LENGTH", 2024-03-24T03:17:24.112 INFO:teuthology.orchestra.run.smithi118.stdout: "type": "int", 2024-03-24T03:17:24.112 INFO:teuthology.orchestra.run.smithi118.stdout: "level": "advanced", 2024-03-24T03:17:24.112 INFO:teuthology.orchestra.run.smithi118.stdout: "flags": 0, 2024-03-24T03:17:24.112 INFO:teuthology.orchestra.run.smithi118.stdout: "default_value": "8", 2024-03-24T03:17:24.112 INFO:teuthology.orchestra.run.smithi118.stdout: "min": "", 2024-03-24T03:17:24.112 INFO:teuthology.orchestra.run.smithi118.stdout: "max": "", 2024-03-24T03:17:24.112 INFO:teuthology.orchestra.run.smithi118.stdout: "enum_allowed": [], 2024-03-24T03:17:24.112 INFO:teuthology.orchestra.run.smithi118.stdout: "desc": "", 2024-03-24T03:17:24.112 INFO:teuthology.orchestra.run.smithi118.stdout: "long_desc": "", 2024-03-24T03:17:24.112 INFO:teuthology.orchestra.run.smithi118.stdout: "tags": [], 2024-03-24T03:17:24.113 INFO:teuthology.orchestra.run.smithi118.stdout: "see_also": [] 2024-03-24T03:17:24.113 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-24T03:17:24.113 INFO:teuthology.orchestra.run.smithi118.stdout: "REST_REQUESTS_TIMEOUT": { 2024-03-24T03:17:24.113 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "REST_REQUESTS_TIMEOUT", 2024-03-24T03:17:24.113 INFO:teuthology.orchestra.run.smithi118.stdout: "type": "int", 2024-03-24T03:17:24.113 INFO:teuthology.orchestra.run.smithi118.stdout: "level": "advanced", 2024-03-24T03:17:24.113 INFO:teuthology.orchestra.run.smithi118.stdout: "flags": 0, 2024-03-24T03:17:24.113 INFO:teuthology.orchestra.run.smithi118.stdout: "default_value": "45", 2024-03-24T03:17:24.113 INFO:teuthology.orchestra.run.smithi118.stdout: "min": "", 2024-03-24T03:17:24.113 INFO:teuthology.orchestra.run.smithi118.stdout: "max": "", 2024-03-24T03:17:24.114 INFO:teuthology.orchestra.run.smithi118.stdout: "enum_allowed": [], 2024-03-24T03:17:24.114 INFO:teuthology.orchestra.run.smithi118.stdout: "desc": "", 2024-03-24T03:17:24.114 INFO:teuthology.orchestra.run.smithi118.stdout: "long_desc": "", 2024-03-24T03:17:24.114 INFO:teuthology.orchestra.run.smithi118.stdout: "tags": [], 2024-03-24T03:17:24.114 INFO:teuthology.orchestra.run.smithi118.stdout: "see_also": [] 2024-03-24T03:17:24.114 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-24T03:17:24.114 INFO:teuthology.orchestra.run.smithi118.stdout: "RGW_API_ACCESS_KEY": { 2024-03-24T03:17:24.114 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "RGW_API_ACCESS_KEY", 2024-03-24T03:17:24.115 INFO:teuthology.orchestra.run.smithi118.stdout: "type": "str", 2024-03-24T03:17:24.115 INFO:teuthology.orchestra.run.smithi118.stdout: "level": "advanced", 2024-03-24T03:17:24.115 INFO:teuthology.orchestra.run.smithi118.stdout: "flags": 0, 2024-03-24T03:17:24.115 INFO:teuthology.orchestra.run.smithi118.stdout: "default_value": "", 2024-03-24T03:17:24.115 INFO:teuthology.orchestra.run.smithi118.stdout: "min": "", 2024-03-24T03:17:24.115 INFO:teuthology.orchestra.run.smithi118.stdout: "max": "", 2024-03-24T03:17:24.115 INFO:teuthology.orchestra.run.smithi118.stdout: "enum_allowed": [], 2024-03-24T03:17:24.115 INFO:teuthology.orchestra.run.smithi118.stdout: "desc": "", 2024-03-24T03:17:24.115 INFO:teuthology.orchestra.run.smithi118.stdout: "long_desc": "", 2024-03-24T03:17:24.115 INFO:teuthology.orchestra.run.smithi118.stdout: "tags": [], 2024-03-24T03:17:24.116 INFO:teuthology.orchestra.run.smithi118.stdout: "see_also": [] 2024-03-24T03:17:24.116 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-24T03:17:24.116 INFO:teuthology.orchestra.run.smithi118.stdout: "RGW_API_ADMIN_RESOURCE": { 2024-03-24T03:17:24.116 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "RGW_API_ADMIN_RESOURCE", 2024-03-24T03:17:24.116 INFO:teuthology.orchestra.run.smithi118.stdout: "type": "str", 2024-03-24T03:17:24.116 INFO:teuthology.orchestra.run.smithi118.stdout: "level": "advanced", 2024-03-24T03:17:24.116 INFO:teuthology.orchestra.run.smithi118.stdout: "flags": 0, 2024-03-24T03:17:24.116 INFO:teuthology.orchestra.run.smithi118.stdout: "default_value": "admin", 2024-03-24T03:17:24.116 INFO:teuthology.orchestra.run.smithi118.stdout: "min": "", 2024-03-24T03:17:24.117 INFO:teuthology.orchestra.run.smithi118.stdout: "max": "", 2024-03-24T03:17:24.117 INFO:teuthology.orchestra.run.smithi118.stdout: "enum_allowed": [], 2024-03-24T03:17:24.117 INFO:teuthology.orchestra.run.smithi118.stdout: "desc": "", 2024-03-24T03:17:24.117 INFO:teuthology.orchestra.run.smithi118.stdout: "long_desc": "", 2024-03-24T03:17:24.117 INFO:teuthology.orchestra.run.smithi118.stdout: "tags": [], 2024-03-24T03:17:24.117 INFO:teuthology.orchestra.run.smithi118.stdout: "see_also": [] 2024-03-24T03:17:24.117 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-24T03:17:24.117 INFO:teuthology.orchestra.run.smithi118.stdout: "RGW_API_SECRET_KEY": { 2024-03-24T03:17:24.117 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "RGW_API_SECRET_KEY", 2024-03-24T03:17:24.118 INFO:teuthology.orchestra.run.smithi118.stdout: "type": "str", 2024-03-24T03:17:24.118 INFO:teuthology.orchestra.run.smithi118.stdout: "level": "advanced", 2024-03-24T03:17:24.118 INFO:teuthology.orchestra.run.smithi118.stdout: "flags": 0, 2024-03-24T03:17:24.118 INFO:teuthology.orchestra.run.smithi118.stdout: "default_value": "", 2024-03-24T03:17:24.118 INFO:teuthology.orchestra.run.smithi118.stdout: "min": "", 2024-03-24T03:17:24.118 INFO:teuthology.orchestra.run.smithi118.stdout: "max": "", 2024-03-24T03:17:24.118 INFO:teuthology.orchestra.run.smithi118.stdout: "enum_allowed": [], 2024-03-24T03:17:24.118 INFO:teuthology.orchestra.run.smithi118.stdout: "desc": "", 2024-03-24T03:17:24.118 INFO:teuthology.orchestra.run.smithi118.stdout: "long_desc": "", 2024-03-24T03:17:24.119 INFO:teuthology.orchestra.run.smithi118.stdout: "tags": [], 2024-03-24T03:17:24.119 INFO:teuthology.orchestra.run.smithi118.stdout: "see_also": [] 2024-03-24T03:17:24.119 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-24T03:17:24.119 INFO:teuthology.orchestra.run.smithi118.stdout: "RGW_API_SSL_VERIFY": { 2024-03-24T03:17:24.119 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "RGW_API_SSL_VERIFY", 2024-03-24T03:17:24.119 INFO:teuthology.orchestra.run.smithi118.stdout: "type": "bool", 2024-03-24T03:17:24.119 INFO:teuthology.orchestra.run.smithi118.stdout: "level": "advanced", 2024-03-24T03:17:24.119 INFO:teuthology.orchestra.run.smithi118.stdout: "flags": 0, 2024-03-24T03:17:24.119 INFO:teuthology.orchestra.run.smithi118.stdout: "default_value": "True", 2024-03-24T03:17:24.119 INFO:teuthology.orchestra.run.smithi118.stdout: "min": "", 2024-03-24T03:17:24.120 INFO:teuthology.orchestra.run.smithi118.stdout: "max": "", 2024-03-24T03:17:24.120 INFO:teuthology.orchestra.run.smithi118.stdout: "enum_allowed": [], 2024-03-24T03:17:24.120 INFO:teuthology.orchestra.run.smithi118.stdout: "desc": "", 2024-03-24T03:17:24.120 INFO:teuthology.orchestra.run.smithi118.stdout: "long_desc": "", 2024-03-24T03:17:24.120 INFO:teuthology.orchestra.run.smithi118.stdout: "tags": [], 2024-03-24T03:17:24.120 INFO:teuthology.orchestra.run.smithi118.stdout: "see_also": [] 2024-03-24T03:17:24.120 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-24T03:17:24.120 INFO:teuthology.orchestra.run.smithi118.stdout: "UNSAFE_TLS_v1_2": { 2024-03-24T03:17:24.120 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "UNSAFE_TLS_v1_2", 2024-03-24T03:17:24.121 INFO:teuthology.orchestra.run.smithi118.stdout: "type": "bool", 2024-03-24T03:17:24.121 INFO:teuthology.orchestra.run.smithi118.stdout: "level": "advanced", 2024-03-24T03:17:24.121 INFO:teuthology.orchestra.run.smithi118.stdout: "flags": 0, 2024-03-24T03:17:24.121 INFO:teuthology.orchestra.run.smithi118.stdout: "default_value": "False", 2024-03-24T03:17:24.121 INFO:teuthology.orchestra.run.smithi118.stdout: "min": "", 2024-03-24T03:17:24.121 INFO:teuthology.orchestra.run.smithi118.stdout: "max": "", 2024-03-24T03:17:24.121 INFO:teuthology.orchestra.run.smithi118.stdout: "enum_allowed": [], 2024-03-24T03:17:24.121 INFO:teuthology.orchestra.run.smithi118.stdout: "desc": "", 2024-03-24T03:17:24.121 INFO:teuthology.orchestra.run.smithi118.stdout: "long_desc": "", 2024-03-24T03:17:24.122 INFO:teuthology.orchestra.run.smithi118.stdout: "tags": [], 2024-03-24T03:17:24.122 INFO:teuthology.orchestra.run.smithi118.stdout: "see_also": [] 2024-03-24T03:17:24.122 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-24T03:17:24.122 INFO:teuthology.orchestra.run.smithi118.stdout: "USER_PWD_EXPIRATION_SPAN": { 2024-03-24T03:17:24.122 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "USER_PWD_EXPIRATION_SPAN", 2024-03-24T03:17:24.122 INFO:teuthology.orchestra.run.smithi118.stdout: "type": "int", 2024-03-24T03:17:24.122 INFO:teuthology.orchestra.run.smithi118.stdout: "level": "advanced", 2024-03-24T03:17:24.122 INFO:teuthology.orchestra.run.smithi118.stdout: "flags": 0, 2024-03-24T03:17:24.122 INFO:teuthology.orchestra.run.smithi118.stdout: "default_value": "0", 2024-03-24T03:17:24.122 INFO:teuthology.orchestra.run.smithi118.stdout: "min": "", 2024-03-24T03:17:24.123 INFO:teuthology.orchestra.run.smithi118.stdout: "max": "", 2024-03-24T03:17:24.123 INFO:teuthology.orchestra.run.smithi118.stdout: "enum_allowed": [], 2024-03-24T03:17:24.123 INFO:teuthology.orchestra.run.smithi118.stdout: "desc": "", 2024-03-24T03:17:24.123 INFO:teuthology.orchestra.run.smithi118.stdout: "long_desc": "", 2024-03-24T03:17:24.123 INFO:teuthology.orchestra.run.smithi118.stdout: "tags": [], 2024-03-24T03:17:24.123 INFO:teuthology.orchestra.run.smithi118.stdout: "see_also": [] 2024-03-24T03:17:24.123 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-24T03:17:24.123 INFO:teuthology.orchestra.run.smithi118.stdout: "USER_PWD_EXPIRATION_WARNING_1": { 2024-03-24T03:17:24.123 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "USER_PWD_EXPIRATION_WARNING_1", 2024-03-24T03:17:24.124 INFO:teuthology.orchestra.run.smithi118.stdout: "type": "int", 2024-03-24T03:17:24.124 INFO:teuthology.orchestra.run.smithi118.stdout: "level": "advanced", 2024-03-24T03:17:24.124 INFO:teuthology.orchestra.run.smithi118.stdout: "flags": 0, 2024-03-24T03:17:24.124 INFO:teuthology.orchestra.run.smithi118.stdout: "default_value": "10", 2024-03-24T03:17:24.124 INFO:teuthology.orchestra.run.smithi118.stdout: "min": "", 2024-03-24T03:17:24.124 INFO:teuthology.orchestra.run.smithi118.stdout: "max": "", 2024-03-24T03:17:24.124 INFO:teuthology.orchestra.run.smithi118.stdout: "enum_allowed": [], 2024-03-24T03:17:24.124 INFO:teuthology.orchestra.run.smithi118.stdout: "desc": "", 2024-03-24T03:17:24.124 INFO:teuthology.orchestra.run.smithi118.stdout: "long_desc": "", 2024-03-24T03:17:24.125 INFO:teuthology.orchestra.run.smithi118.stdout: "tags": [], 2024-03-24T03:17:24.125 INFO:teuthology.orchestra.run.smithi118.stdout: "see_also": [] 2024-03-24T03:17:24.125 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-24T03:17:24.125 INFO:teuthology.orchestra.run.smithi118.stdout: "USER_PWD_EXPIRATION_WARNING_2": { 2024-03-24T03:17:24.125 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "USER_PWD_EXPIRATION_WARNING_2", 2024-03-24T03:17:24.125 INFO:teuthology.orchestra.run.smithi118.stdout: "type": "int", 2024-03-24T03:17:24.125 INFO:teuthology.orchestra.run.smithi118.stdout: "level": "advanced", 2024-03-24T03:17:24.125 INFO:teuthology.orchestra.run.smithi118.stdout: "flags": 0, 2024-03-24T03:17:24.125 INFO:teuthology.orchestra.run.smithi118.stdout: "default_value": "5", 2024-03-24T03:17:24.126 INFO:teuthology.orchestra.run.smithi118.stdout: "min": "", 2024-03-24T03:17:24.126 INFO:teuthology.orchestra.run.smithi118.stdout: "max": "", 2024-03-24T03:17:24.126 INFO:teuthology.orchestra.run.smithi118.stdout: "enum_allowed": [], 2024-03-24T03:17:24.126 INFO:teuthology.orchestra.run.smithi118.stdout: "desc": "", 2024-03-24T03:17:24.126 INFO:teuthology.orchestra.run.smithi118.stdout: "long_desc": "", 2024-03-24T03:17:24.126 INFO:teuthology.orchestra.run.smithi118.stdout: "tags": [], 2024-03-24T03:17:24.126 INFO:teuthology.orchestra.run.smithi118.stdout: "see_also": [] 2024-03-24T03:17:24.126 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-24T03:17:24.126 INFO:teuthology.orchestra.run.smithi118.stdout: "cross_origin_url": { 2024-03-24T03:17:24.127 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "cross_origin_url", 2024-03-24T03:17:24.127 INFO:teuthology.orchestra.run.smithi118.stdout: "type": "str", 2024-03-24T03:17:24.127 INFO:teuthology.orchestra.run.smithi118.stdout: "level": "advanced", 2024-03-24T03:17:24.127 INFO:teuthology.orchestra.run.smithi118.stdout: "flags": 0, 2024-03-24T03:17:24.127 INFO:teuthology.orchestra.run.smithi118.stdout: "default_value": "", 2024-03-24T03:17:24.127 INFO:teuthology.orchestra.run.smithi118.stdout: "min": "", 2024-03-24T03:17:24.127 INFO:teuthology.orchestra.run.smithi118.stdout: "max": "", 2024-03-24T03:17:24.127 INFO:teuthology.orchestra.run.smithi118.stdout: "enum_allowed": [], 2024-03-24T03:17:24.127 INFO:teuthology.orchestra.run.smithi118.stdout: "desc": "", 2024-03-24T03:17:24.127 INFO:teuthology.orchestra.run.smithi118.stdout: "long_desc": "", 2024-03-24T03:17:24.128 INFO:teuthology.orchestra.run.smithi118.stdout: "tags": [], 2024-03-24T03:17:24.128 INFO:teuthology.orchestra.run.smithi118.stdout: "see_also": [] 2024-03-24T03:17:24.128 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-24T03:17:24.128 INFO:teuthology.orchestra.run.smithi118.stdout: "crt_file": { 2024-03-24T03:17:24.128 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "crt_file", 2024-03-24T03:17:24.128 INFO:teuthology.orchestra.run.smithi118.stdout: "type": "str", 2024-03-24T03:17:24.128 INFO:teuthology.orchestra.run.smithi118.stdout: "level": "advanced", 2024-03-24T03:17:24.128 INFO:teuthology.orchestra.run.smithi118.stdout: "flags": 0, 2024-03-24T03:17:24.128 INFO:teuthology.orchestra.run.smithi118.stdout: "default_value": "", 2024-03-24T03:17:24.128 INFO:teuthology.orchestra.run.smithi118.stdout: "min": "", 2024-03-24T03:17:24.129 INFO:teuthology.orchestra.run.smithi118.stdout: "max": "", 2024-03-24T03:17:24.129 INFO:teuthology.orchestra.run.smithi118.stdout: "enum_allowed": [], 2024-03-24T03:17:24.129 INFO:teuthology.orchestra.run.smithi118.stdout: "desc": "", 2024-03-24T03:17:24.129 INFO:teuthology.orchestra.run.smithi118.stdout: "long_desc": "", 2024-03-24T03:17:24.129 INFO:teuthology.orchestra.run.smithi118.stdout: "tags": [], 2024-03-24T03:17:24.129 INFO:teuthology.orchestra.run.smithi118.stdout: "see_also": [] 2024-03-24T03:17:24.129 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-24T03:17:24.129 INFO:teuthology.orchestra.run.smithi118.stdout: "debug": { 2024-03-24T03:17:24.129 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "debug", 2024-03-24T03:17:24.130 INFO:teuthology.orchestra.run.smithi118.stdout: "type": "bool", 2024-03-24T03:17:24.130 INFO:teuthology.orchestra.run.smithi118.stdout: "level": "advanced", 2024-03-24T03:17:24.130 INFO:teuthology.orchestra.run.smithi118.stdout: "flags": 0, 2024-03-24T03:17:24.130 INFO:teuthology.orchestra.run.smithi118.stdout: "default_value": "False", 2024-03-24T03:17:24.130 INFO:teuthology.orchestra.run.smithi118.stdout: "min": "", 2024-03-24T03:17:24.130 INFO:teuthology.orchestra.run.smithi118.stdout: "max": "", 2024-03-24T03:17:24.130 INFO:teuthology.orchestra.run.smithi118.stdout: "enum_allowed": [], 2024-03-24T03:17:24.130 INFO:teuthology.orchestra.run.smithi118.stdout: "desc": "Enable/disable debug options", 2024-03-24T03:17:24.130 INFO:teuthology.orchestra.run.smithi118.stdout: "long_desc": "", 2024-03-24T03:17:24.131 INFO:teuthology.orchestra.run.smithi118.stdout: "tags": [], 2024-03-24T03:17:24.131 INFO:teuthology.orchestra.run.smithi118.stdout: "see_also": [] 2024-03-24T03:17:24.131 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-24T03:17:24.131 INFO:teuthology.orchestra.run.smithi118.stdout: "jwt_token_ttl": { 2024-03-24T03:17:24.131 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "jwt_token_ttl", 2024-03-24T03:17:24.131 INFO:teuthology.orchestra.run.smithi118.stdout: "type": "int", 2024-03-24T03:17:24.131 INFO:teuthology.orchestra.run.smithi118.stdout: "level": "advanced", 2024-03-24T03:17:24.131 INFO:teuthology.orchestra.run.smithi118.stdout: "flags": 0, 2024-03-24T03:17:24.131 INFO:teuthology.orchestra.run.smithi118.stdout: "default_value": "28800", 2024-03-24T03:17:24.131 INFO:teuthology.orchestra.run.smithi118.stdout: "min": "", 2024-03-24T03:17:24.132 INFO:teuthology.orchestra.run.smithi118.stdout: "max": "", 2024-03-24T03:17:24.132 INFO:teuthology.orchestra.run.smithi118.stdout: "enum_allowed": [], 2024-03-24T03:17:24.132 INFO:teuthology.orchestra.run.smithi118.stdout: "desc": "", 2024-03-24T03:17:24.132 INFO:teuthology.orchestra.run.smithi118.stdout: "long_desc": "", 2024-03-24T03:17:24.132 INFO:teuthology.orchestra.run.smithi118.stdout: "tags": [], 2024-03-24T03:17:24.132 INFO:teuthology.orchestra.run.smithi118.stdout: "see_also": [] 2024-03-24T03:17:24.132 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-24T03:17:24.132 INFO:teuthology.orchestra.run.smithi118.stdout: "key_file": { 2024-03-24T03:17:24.132 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "key_file", 2024-03-24T03:17:24.133 INFO:teuthology.orchestra.run.smithi118.stdout: "type": "str", 2024-03-24T03:17:24.133 INFO:teuthology.orchestra.run.smithi118.stdout: "level": "advanced", 2024-03-24T03:17:24.133 INFO:teuthology.orchestra.run.smithi118.stdout: "flags": 0, 2024-03-24T03:17:24.133 INFO:teuthology.orchestra.run.smithi118.stdout: "default_value": "", 2024-03-24T03:17:24.133 INFO:teuthology.orchestra.run.smithi118.stdout: "min": "", 2024-03-24T03:17:24.133 INFO:teuthology.orchestra.run.smithi118.stdout: "max": "", 2024-03-24T03:17:24.133 INFO:teuthology.orchestra.run.smithi118.stdout: "enum_allowed": [], 2024-03-24T03:17:24.133 INFO:teuthology.orchestra.run.smithi118.stdout: "desc": "", 2024-03-24T03:17:24.133 INFO:teuthology.orchestra.run.smithi118.stdout: "long_desc": "", 2024-03-24T03:17:24.134 INFO:teuthology.orchestra.run.smithi118.stdout: "tags": [], 2024-03-24T03:17:24.134 INFO:teuthology.orchestra.run.smithi118.stdout: "see_also": [] 2024-03-24T03:17:24.134 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-24T03:17:24.134 INFO:teuthology.orchestra.run.smithi118.stdout: "log_level": { 2024-03-24T03:17:24.134 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "log_level", 2024-03-24T03:17:24.134 INFO:teuthology.orchestra.run.smithi118.stdout: "type": "str", 2024-03-24T03:17:24.134 INFO:teuthology.orchestra.run.smithi118.stdout: "level": "advanced", 2024-03-24T03:17:24.134 INFO:teuthology.orchestra.run.smithi118.stdout: "flags": 1, 2024-03-24T03:17:24.134 INFO:teuthology.orchestra.run.smithi118.stdout: "default_value": "", 2024-03-24T03:17:24.135 INFO:teuthology.orchestra.run.smithi118.stdout: "min": "", 2024-03-24T03:17:24.135 INFO:teuthology.orchestra.run.smithi118.stdout: "max": "", 2024-03-24T03:17:24.135 INFO:teuthology.orchestra.run.smithi118.stdout: "enum_allowed": [ 2024-03-24T03:17:24.135 INFO:teuthology.orchestra.run.smithi118.stdout: "", 2024-03-24T03:17:24.135 INFO:teuthology.orchestra.run.smithi118.stdout: "critical", 2024-03-24T03:17:24.135 INFO:teuthology.orchestra.run.smithi118.stdout: "debug", 2024-03-24T03:17:24.135 INFO:teuthology.orchestra.run.smithi118.stdout: "error", 2024-03-24T03:17:24.135 INFO:teuthology.orchestra.run.smithi118.stdout: "info", 2024-03-24T03:17:24.135 INFO:teuthology.orchestra.run.smithi118.stdout: "warning" 2024-03-24T03:17:24.136 INFO:teuthology.orchestra.run.smithi118.stdout: ], 2024-03-24T03:17:24.136 INFO:teuthology.orchestra.run.smithi118.stdout: "desc": "", 2024-03-24T03:17:24.136 INFO:teuthology.orchestra.run.smithi118.stdout: "long_desc": "", 2024-03-24T03:17:24.136 INFO:teuthology.orchestra.run.smithi118.stdout: "tags": [], 2024-03-24T03:17:24.136 INFO:teuthology.orchestra.run.smithi118.stdout: "see_also": [] 2024-03-24T03:17:24.136 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-24T03:17:24.136 INFO:teuthology.orchestra.run.smithi118.stdout: "log_to_cluster": { 2024-03-24T03:17:24.136 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "log_to_cluster", 2024-03-24T03:17:24.136 INFO:teuthology.orchestra.run.smithi118.stdout: "type": "bool", 2024-03-24T03:17:24.137 INFO:teuthology.orchestra.run.smithi118.stdout: "level": "advanced", 2024-03-24T03:17:24.137 INFO:teuthology.orchestra.run.smithi118.stdout: "flags": 1, 2024-03-24T03:17:24.137 INFO:teuthology.orchestra.run.smithi118.stdout: "default_value": "False", 2024-03-24T03:17:24.137 INFO:teuthology.orchestra.run.smithi118.stdout: "min": "", 2024-03-24T03:17:24.137 INFO:teuthology.orchestra.run.smithi118.stdout: "max": "", 2024-03-24T03:17:24.137 INFO:teuthology.orchestra.run.smithi118.stdout: "enum_allowed": [], 2024-03-24T03:17:24.137 INFO:teuthology.orchestra.run.smithi118.stdout: "desc": "", 2024-03-24T03:17:24.137 INFO:teuthology.orchestra.run.smithi118.stdout: "long_desc": "", 2024-03-24T03:17:24.137 INFO:teuthology.orchestra.run.smithi118.stdout: "tags": [], 2024-03-24T03:17:24.137 INFO:teuthology.orchestra.run.smithi118.stdout: "see_also": [] 2024-03-24T03:17:24.138 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-24T03:17:24.138 INFO:teuthology.orchestra.run.smithi118.stdout: "log_to_cluster_level": { 2024-03-24T03:17:24.138 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "log_to_cluster_level", 2024-03-24T03:17:24.138 INFO:teuthology.orchestra.run.smithi118.stdout: "type": "str", 2024-03-24T03:17:24.138 INFO:teuthology.orchestra.run.smithi118.stdout: "level": "advanced", 2024-03-24T03:17:24.138 INFO:teuthology.orchestra.run.smithi118.stdout: "flags": 1, 2024-03-24T03:17:24.138 INFO:teuthology.orchestra.run.smithi118.stdout: "default_value": "info", 2024-03-24T03:17:24.138 INFO:teuthology.orchestra.run.smithi118.stdout: "min": "", 2024-03-24T03:17:24.138 INFO:teuthology.orchestra.run.smithi118.stdout: "max": "", 2024-03-24T03:17:24.139 INFO:teuthology.orchestra.run.smithi118.stdout: "enum_allowed": [ 2024-03-24T03:17:24.139 INFO:teuthology.orchestra.run.smithi118.stdout: "", 2024-03-24T03:17:24.139 INFO:teuthology.orchestra.run.smithi118.stdout: "critical", 2024-03-24T03:17:24.139 INFO:teuthology.orchestra.run.smithi118.stdout: "debug", 2024-03-24T03:17:24.139 INFO:teuthology.orchestra.run.smithi118.stdout: "error", 2024-03-24T03:17:24.139 INFO:teuthology.orchestra.run.smithi118.stdout: "info", 2024-03-24T03:17:24.139 INFO:teuthology.orchestra.run.smithi118.stdout: "warning" 2024-03-24T03:17:24.139 INFO:teuthology.orchestra.run.smithi118.stdout: ], 2024-03-24T03:17:24.139 INFO:teuthology.orchestra.run.smithi118.stdout: "desc": "", 2024-03-24T03:17:24.139 INFO:teuthology.orchestra.run.smithi118.stdout: "long_desc": "", 2024-03-24T03:17:24.140 INFO:teuthology.orchestra.run.smithi118.stdout: "tags": [], 2024-03-24T03:17:24.140 INFO:teuthology.orchestra.run.smithi118.stdout: "see_also": [] 2024-03-24T03:17:24.140 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-24T03:17:24.140 INFO:teuthology.orchestra.run.smithi118.stdout: "log_to_file": { 2024-03-24T03:17:24.140 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "log_to_file", 2024-03-24T03:17:24.140 INFO:teuthology.orchestra.run.smithi118.stdout: "type": "bool", 2024-03-24T03:17:24.140 INFO:teuthology.orchestra.run.smithi118.stdout: "level": "advanced", 2024-03-24T03:17:24.140 INFO:teuthology.orchestra.run.smithi118.stdout: "flags": 1, 2024-03-24T03:17:24.140 INFO:teuthology.orchestra.run.smithi118.stdout: "default_value": "False", 2024-03-24T03:17:24.141 INFO:teuthology.orchestra.run.smithi118.stdout: "min": "", 2024-03-24T03:17:24.141 INFO:teuthology.orchestra.run.smithi118.stdout: "max": "", 2024-03-24T03:17:24.141 INFO:teuthology.orchestra.run.smithi118.stdout: "enum_allowed": [], 2024-03-24T03:17:24.141 INFO:teuthology.orchestra.run.smithi118.stdout: "desc": "", 2024-03-24T03:17:24.141 INFO:teuthology.orchestra.run.smithi118.stdout: "long_desc": "", 2024-03-24T03:17:24.141 INFO:teuthology.orchestra.run.smithi118.stdout: "tags": [], 2024-03-24T03:17:24.141 INFO:teuthology.orchestra.run.smithi118.stdout: "see_also": [] 2024-03-24T03:17:24.141 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-24T03:17:24.141 INFO:teuthology.orchestra.run.smithi118.stdout: "motd": { 2024-03-24T03:17:24.142 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "motd", 2024-03-24T03:17:24.142 INFO:teuthology.orchestra.run.smithi118.stdout: "type": "str", 2024-03-24T03:17:24.142 INFO:teuthology.orchestra.run.smithi118.stdout: "level": "advanced", 2024-03-24T03:17:24.142 INFO:teuthology.orchestra.run.smithi118.stdout: "flags": 0, 2024-03-24T03:17:24.142 INFO:teuthology.orchestra.run.smithi118.stdout: "default_value": "", 2024-03-24T03:17:24.142 INFO:teuthology.orchestra.run.smithi118.stdout: "min": "", 2024-03-24T03:17:24.142 INFO:teuthology.orchestra.run.smithi118.stdout: "max": "", 2024-03-24T03:17:24.142 INFO:teuthology.orchestra.run.smithi118.stdout: "enum_allowed": [], 2024-03-24T03:17:24.142 INFO:teuthology.orchestra.run.smithi118.stdout: "desc": "The message of the day", 2024-03-24T03:17:24.143 INFO:teuthology.orchestra.run.smithi118.stdout: "long_desc": "", 2024-03-24T03:17:24.143 INFO:teuthology.orchestra.run.smithi118.stdout: "tags": [], 2024-03-24T03:17:24.143 INFO:teuthology.orchestra.run.smithi118.stdout: "see_also": [] 2024-03-24T03:17:24.143 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-24T03:17:24.143 INFO:teuthology.orchestra.run.smithi118.stdout: "redirect_resolve_ip_addr": { 2024-03-24T03:17:24.143 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "redirect_resolve_ip_addr", 2024-03-24T03:17:24.143 INFO:teuthology.orchestra.run.smithi118.stdout: "type": "bool", 2024-03-24T03:17:24.143 INFO:teuthology.orchestra.run.smithi118.stdout: "level": "advanced", 2024-03-24T03:17:24.143 INFO:teuthology.orchestra.run.smithi118.stdout: "flags": 0, 2024-03-24T03:17:24.144 INFO:teuthology.orchestra.run.smithi118.stdout: "default_value": "False", 2024-03-24T03:17:24.144 INFO:teuthology.orchestra.run.smithi118.stdout: "min": "", 2024-03-24T03:17:24.144 INFO:teuthology.orchestra.run.smithi118.stdout: "max": "", 2024-03-24T03:17:24.144 INFO:teuthology.orchestra.run.smithi118.stdout: "enum_allowed": [], 2024-03-24T03:17:24.144 INFO:teuthology.orchestra.run.smithi118.stdout: "desc": "", 2024-03-24T03:17:24.144 INFO:teuthology.orchestra.run.smithi118.stdout: "long_desc": "", 2024-03-24T03:17:24.144 INFO:teuthology.orchestra.run.smithi118.stdout: "tags": [], 2024-03-24T03:17:24.144 INFO:teuthology.orchestra.run.smithi118.stdout: "see_also": [] 2024-03-24T03:17:24.144 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-24T03:17:24.145 INFO:teuthology.orchestra.run.smithi118.stdout: "server_addr": { 2024-03-24T03:17:24.145 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "server_addr", 2024-03-24T03:17:24.145 INFO:teuthology.orchestra.run.smithi118.stdout: "type": "str", 2024-03-24T03:17:24.145 INFO:teuthology.orchestra.run.smithi118.stdout: "level": "advanced", 2024-03-24T03:17:24.145 INFO:teuthology.orchestra.run.smithi118.stdout: "flags": 0, 2024-03-24T03:17:24.145 INFO:teuthology.orchestra.run.smithi118.stdout: "default_value": "::", 2024-03-24T03:17:24.145 INFO:teuthology.orchestra.run.smithi118.stdout: "min": "", 2024-03-24T03:17:24.145 INFO:teuthology.orchestra.run.smithi118.stdout: "max": "", 2024-03-24T03:17:24.145 INFO:teuthology.orchestra.run.smithi118.stdout: "enum_allowed": [], 2024-03-24T03:17:24.145 INFO:teuthology.orchestra.run.smithi118.stdout: "desc": "", 2024-03-24T03:17:24.146 INFO:teuthology.orchestra.run.smithi118.stdout: "long_desc": "", 2024-03-24T03:17:24.146 INFO:teuthology.orchestra.run.smithi118.stdout: "tags": [], 2024-03-24T03:17:24.146 INFO:teuthology.orchestra.run.smithi118.stdout: "see_also": [] 2024-03-24T03:17:24.146 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-24T03:17:24.146 INFO:teuthology.orchestra.run.smithi118.stdout: "server_port": { 2024-03-24T03:17:24.146 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "server_port", 2024-03-24T03:17:24.146 INFO:teuthology.orchestra.run.smithi118.stdout: "type": "int", 2024-03-24T03:17:24.146 INFO:teuthology.orchestra.run.smithi118.stdout: "level": "advanced", 2024-03-24T03:17:24.146 INFO:teuthology.orchestra.run.smithi118.stdout: "flags": 0, 2024-03-24T03:17:24.147 INFO:teuthology.orchestra.run.smithi118.stdout: "default_value": "8080", 2024-03-24T03:17:24.147 INFO:teuthology.orchestra.run.smithi118.stdout: "min": "", 2024-03-24T03:17:24.147 INFO:teuthology.orchestra.run.smithi118.stdout: "max": "", 2024-03-24T03:17:24.147 INFO:teuthology.orchestra.run.smithi118.stdout: "enum_allowed": [], 2024-03-24T03:17:24.147 INFO:teuthology.orchestra.run.smithi118.stdout: "desc": "", 2024-03-24T03:17:24.147 INFO:teuthology.orchestra.run.smithi118.stdout: "long_desc": "", 2024-03-24T03:17:24.147 INFO:teuthology.orchestra.run.smithi118.stdout: "tags": [], 2024-03-24T03:17:24.147 INFO:teuthology.orchestra.run.smithi118.stdout: "see_also": [] 2024-03-24T03:17:24.147 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-24T03:17:24.147 INFO:teuthology.orchestra.run.smithi118.stdout: "ssl": { 2024-03-24T03:17:24.148 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "ssl", 2024-03-24T03:17:24.148 INFO:teuthology.orchestra.run.smithi118.stdout: "type": "bool", 2024-03-24T03:17:24.148 INFO:teuthology.orchestra.run.smithi118.stdout: "level": "advanced", 2024-03-24T03:17:24.148 INFO:teuthology.orchestra.run.smithi118.stdout: "flags": 0, 2024-03-24T03:17:24.148 INFO:teuthology.orchestra.run.smithi118.stdout: "default_value": "True", 2024-03-24T03:17:24.148 INFO:teuthology.orchestra.run.smithi118.stdout: "min": "", 2024-03-24T03:17:24.148 INFO:teuthology.orchestra.run.smithi118.stdout: "max": "", 2024-03-24T03:17:24.148 INFO:teuthology.orchestra.run.smithi118.stdout: "enum_allowed": [], 2024-03-24T03:17:24.148 INFO:teuthology.orchestra.run.smithi118.stdout: "desc": "", 2024-03-24T03:17:24.148 INFO:teuthology.orchestra.run.smithi118.stdout: "long_desc": "", 2024-03-24T03:17:24.149 INFO:teuthology.orchestra.run.smithi118.stdout: "tags": [], 2024-03-24T03:17:24.149 INFO:teuthology.orchestra.run.smithi118.stdout: "see_also": [] 2024-03-24T03:17:24.149 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-24T03:17:24.149 INFO:teuthology.orchestra.run.smithi118.stdout: "ssl_server_port": { 2024-03-24T03:17:24.149 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "ssl_server_port", 2024-03-24T03:17:24.149 INFO:teuthology.orchestra.run.smithi118.stdout: "type": "int", 2024-03-24T03:17:24.149 INFO:teuthology.orchestra.run.smithi118.stdout: "level": "advanced", 2024-03-24T03:17:24.149 INFO:teuthology.orchestra.run.smithi118.stdout: "flags": 0, 2024-03-24T03:17:24.149 INFO:teuthology.orchestra.run.smithi118.stdout: "default_value": "8443", 2024-03-24T03:17:24.150 INFO:teuthology.orchestra.run.smithi118.stdout: "min": "", 2024-03-24T03:17:24.150 INFO:teuthology.orchestra.run.smithi118.stdout: "max": "", 2024-03-24T03:17:24.150 INFO:teuthology.orchestra.run.smithi118.stdout: "enum_allowed": [], 2024-03-24T03:17:24.150 INFO:teuthology.orchestra.run.smithi118.stdout: "desc": "", 2024-03-24T03:17:24.150 INFO:teuthology.orchestra.run.smithi118.stdout: "long_desc": "", 2024-03-24T03:17:24.150 INFO:teuthology.orchestra.run.smithi118.stdout: "tags": [], 2024-03-24T03:17:24.150 INFO:teuthology.orchestra.run.smithi118.stdout: "see_also": [] 2024-03-24T03:17:24.150 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-24T03:17:24.150 INFO:teuthology.orchestra.run.smithi118.stdout: "standby_behaviour": { 2024-03-24T03:17:24.150 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "standby_behaviour", 2024-03-24T03:17:24.151 INFO:teuthology.orchestra.run.smithi118.stdout: "type": "str", 2024-03-24T03:17:24.151 INFO:teuthology.orchestra.run.smithi118.stdout: "level": "advanced", 2024-03-24T03:17:24.151 INFO:teuthology.orchestra.run.smithi118.stdout: "flags": 0, 2024-03-24T03:17:24.151 INFO:teuthology.orchestra.run.smithi118.stdout: "default_value": "redirect", 2024-03-24T03:17:24.151 INFO:teuthology.orchestra.run.smithi118.stdout: "min": "", 2024-03-24T03:17:24.151 INFO:teuthology.orchestra.run.smithi118.stdout: "max": "", 2024-03-24T03:17:24.151 INFO:teuthology.orchestra.run.smithi118.stdout: "enum_allowed": [ 2024-03-24T03:17:24.151 INFO:teuthology.orchestra.run.smithi118.stdout: "error", 2024-03-24T03:17:24.151 INFO:teuthology.orchestra.run.smithi118.stdout: "redirect" 2024-03-24T03:17:24.152 INFO:teuthology.orchestra.run.smithi118.stdout: ], 2024-03-24T03:17:24.152 INFO:teuthology.orchestra.run.smithi118.stdout: "desc": "", 2024-03-24T03:17:24.152 INFO:teuthology.orchestra.run.smithi118.stdout: "long_desc": "", 2024-03-24T03:17:24.152 INFO:teuthology.orchestra.run.smithi118.stdout: "tags": [], 2024-03-24T03:17:24.152 INFO:teuthology.orchestra.run.smithi118.stdout: "see_also": [] 2024-03-24T03:17:24.152 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-24T03:17:24.152 INFO:teuthology.orchestra.run.smithi118.stdout: "standby_error_status_code": { 2024-03-24T03:17:24.152 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "standby_error_status_code", 2024-03-24T03:17:24.152 INFO:teuthology.orchestra.run.smithi118.stdout: "type": "int", 2024-03-24T03:17:24.153 INFO:teuthology.orchestra.run.smithi118.stdout: "level": "advanced", 2024-03-24T03:17:24.153 INFO:teuthology.orchestra.run.smithi118.stdout: "flags": 0, 2024-03-24T03:17:24.153 INFO:teuthology.orchestra.run.smithi118.stdout: "default_value": "500", 2024-03-24T03:17:24.153 INFO:teuthology.orchestra.run.smithi118.stdout: "min": "400", 2024-03-24T03:17:24.153 INFO:teuthology.orchestra.run.smithi118.stdout: "max": "599", 2024-03-24T03:17:24.153 INFO:teuthology.orchestra.run.smithi118.stdout: "enum_allowed": [], 2024-03-24T03:17:24.153 INFO:teuthology.orchestra.run.smithi118.stdout: "desc": "", 2024-03-24T03:17:24.153 INFO:teuthology.orchestra.run.smithi118.stdout: "long_desc": "", 2024-03-24T03:17:24.153 INFO:teuthology.orchestra.run.smithi118.stdout: "tags": [], 2024-03-24T03:17:24.153 INFO:teuthology.orchestra.run.smithi118.stdout: "see_also": [] 2024-03-24T03:17:24.154 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-24T03:17:24.154 INFO:teuthology.orchestra.run.smithi118.stdout: "url_prefix": { 2024-03-24T03:17:24.154 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "url_prefix", 2024-03-24T03:17:24.154 INFO:teuthology.orchestra.run.smithi118.stdout: "type": "str", 2024-03-24T03:17:24.154 INFO:teuthology.orchestra.run.smithi118.stdout: "level": "advanced", 2024-03-24T03:17:24.154 INFO:teuthology.orchestra.run.smithi118.stdout: "flags": 0, 2024-03-24T03:17:24.154 INFO:teuthology.orchestra.run.smithi118.stdout: "default_value": "", 2024-03-24T03:17:24.154 INFO:teuthology.orchestra.run.smithi118.stdout: "min": "", 2024-03-24T03:17:24.155 INFO:teuthology.orchestra.run.smithi118.stdout: "max": "", 2024-03-24T03:17:24.155 INFO:teuthology.orchestra.run.smithi118.stdout: "enum_allowed": [], 2024-03-24T03:17:24.155 INFO:teuthology.orchestra.run.smithi118.stdout: "desc": "", 2024-03-24T03:17:24.155 INFO:teuthology.orchestra.run.smithi118.stdout: "long_desc": "", 2024-03-24T03:17:24.155 INFO:teuthology.orchestra.run.smithi118.stdout: "tags": [], 2024-03-24T03:17:24.155 INFO:teuthology.orchestra.run.smithi118.stdout: "see_also": [] 2024-03-24T03:17:24.155 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-03-24T03:17:24.155 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-03-24T03:17:24.155 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-24T03:17:24.156 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-24T03:17:24.156 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "devicehealth", 2024-03-24T03:17:24.156 INFO:teuthology.orchestra.run.smithi118.stdout: "can_run": true, 2024-03-24T03:17:24.156 INFO:teuthology.orchestra.run.smithi118.stdout: "error_string": "", 2024-03-24T03:17:24.156 INFO:teuthology.orchestra.run.smithi118.stdout: "module_options": { 2024-03-24T03:17:24.156 INFO:teuthology.orchestra.run.smithi118.stdout: "enable_monitoring": { 2024-03-24T03:17:24.156 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "enable_monitoring", 2024-03-24T03:17:24.156 INFO:teuthology.orchestra.run.smithi118.stdout: "type": "bool", 2024-03-24T03:17:24.156 INFO:teuthology.orchestra.run.smithi118.stdout: "level": "advanced", 2024-03-24T03:17:24.156 INFO:teuthology.orchestra.run.smithi118.stdout: "flags": 1, 2024-03-24T03:17:24.157 INFO:teuthology.orchestra.run.smithi118.stdout: "default_value": "True", 2024-03-24T03:17:24.157 INFO:teuthology.orchestra.run.smithi118.stdout: "min": "", 2024-03-24T03:17:24.157 INFO:teuthology.orchestra.run.smithi118.stdout: "max": "", 2024-03-24T03:17:24.157 INFO:teuthology.orchestra.run.smithi118.stdout: "enum_allowed": [], 2024-03-24T03:17:24.157 INFO:teuthology.orchestra.run.smithi118.stdout: "desc": "monitor device health metrics", 2024-03-24T03:17:24.157 INFO:teuthology.orchestra.run.smithi118.stdout: "long_desc": "", 2024-03-24T03:17:24.157 INFO:teuthology.orchestra.run.smithi118.stdout: "tags": [], 2024-03-24T03:17:24.157 INFO:teuthology.orchestra.run.smithi118.stdout: "see_also": [] 2024-03-24T03:17:24.157 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-24T03:17:24.157 INFO:teuthology.orchestra.run.smithi118.stdout: "log_level": { 2024-03-24T03:17:24.158 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "log_level", 2024-03-24T03:17:24.158 INFO:teuthology.orchestra.run.smithi118.stdout: "type": "str", 2024-03-24T03:17:24.158 INFO:teuthology.orchestra.run.smithi118.stdout: "level": "advanced", 2024-03-24T03:17:24.158 INFO:teuthology.orchestra.run.smithi118.stdout: "flags": 1, 2024-03-24T03:17:24.158 INFO:teuthology.orchestra.run.smithi118.stdout: "default_value": "", 2024-03-24T03:17:24.158 INFO:teuthology.orchestra.run.smithi118.stdout: "min": "", 2024-03-24T03:17:24.158 INFO:teuthology.orchestra.run.smithi118.stdout: "max": "", 2024-03-24T03:17:24.158 INFO:teuthology.orchestra.run.smithi118.stdout: "enum_allowed": [ 2024-03-24T03:17:24.158 INFO:teuthology.orchestra.run.smithi118.stdout: "", 2024-03-24T03:17:24.159 INFO:teuthology.orchestra.run.smithi118.stdout: "critical", 2024-03-24T03:17:24.159 INFO:teuthology.orchestra.run.smithi118.stdout: "debug", 2024-03-24T03:17:24.159 INFO:teuthology.orchestra.run.smithi118.stdout: "error", 2024-03-24T03:17:24.159 INFO:teuthology.orchestra.run.smithi118.stdout: "info", 2024-03-24T03:17:24.159 INFO:teuthology.orchestra.run.smithi118.stdout: "warning" 2024-03-24T03:17:24.159 INFO:teuthology.orchestra.run.smithi118.stdout: ], 2024-03-24T03:17:24.159 INFO:teuthology.orchestra.run.smithi118.stdout: "desc": "", 2024-03-24T03:17:24.159 INFO:teuthology.orchestra.run.smithi118.stdout: "long_desc": "", 2024-03-24T03:17:24.159 INFO:teuthology.orchestra.run.smithi118.stdout: "tags": [], 2024-03-24T03:17:24.160 INFO:teuthology.orchestra.run.smithi118.stdout: "see_also": [] 2024-03-24T03:17:24.160 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-24T03:17:24.160 INFO:teuthology.orchestra.run.smithi118.stdout: "log_to_cluster": { 2024-03-24T03:17:24.160 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "log_to_cluster", 2024-03-24T03:17:24.160 INFO:teuthology.orchestra.run.smithi118.stdout: "type": "bool", 2024-03-24T03:17:24.160 INFO:teuthology.orchestra.run.smithi118.stdout: "level": "advanced", 2024-03-24T03:17:24.160 INFO:teuthology.orchestra.run.smithi118.stdout: "flags": 1, 2024-03-24T03:17:24.160 INFO:teuthology.orchestra.run.smithi118.stdout: "default_value": "False", 2024-03-24T03:17:24.160 INFO:teuthology.orchestra.run.smithi118.stdout: "min": "", 2024-03-24T03:17:24.160 INFO:teuthology.orchestra.run.smithi118.stdout: "max": "", 2024-03-24T03:17:24.161 INFO:teuthology.orchestra.run.smithi118.stdout: "enum_allowed": [], 2024-03-24T03:17:24.161 INFO:teuthology.orchestra.run.smithi118.stdout: "desc": "", 2024-03-24T03:17:24.161 INFO:teuthology.orchestra.run.smithi118.stdout: "long_desc": "", 2024-03-24T03:17:24.161 INFO:teuthology.orchestra.run.smithi118.stdout: "tags": [], 2024-03-24T03:17:24.161 INFO:teuthology.orchestra.run.smithi118.stdout: "see_also": [] 2024-03-24T03:17:24.161 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-24T03:17:24.161 INFO:teuthology.orchestra.run.smithi118.stdout: "log_to_cluster_level": { 2024-03-24T03:17:24.161 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "log_to_cluster_level", 2024-03-24T03:17:24.161 INFO:teuthology.orchestra.run.smithi118.stdout: "type": "str", 2024-03-24T03:17:24.162 INFO:teuthology.orchestra.run.smithi118.stdout: "level": "advanced", 2024-03-24T03:17:24.162 INFO:teuthology.orchestra.run.smithi118.stdout: "flags": 1, 2024-03-24T03:17:24.162 INFO:teuthology.orchestra.run.smithi118.stdout: "default_value": "info", 2024-03-24T03:17:24.162 INFO:teuthology.orchestra.run.smithi118.stdout: "min": "", 2024-03-24T03:17:24.162 INFO:teuthology.orchestra.run.smithi118.stdout: "max": "", 2024-03-24T03:17:24.162 INFO:teuthology.orchestra.run.smithi118.stdout: "enum_allowed": [ 2024-03-24T03:17:24.162 INFO:teuthology.orchestra.run.smithi118.stdout: "", 2024-03-24T03:17:24.162 INFO:teuthology.orchestra.run.smithi118.stdout: "critical", 2024-03-24T03:17:24.162 INFO:teuthology.orchestra.run.smithi118.stdout: "debug", 2024-03-24T03:17:24.163 INFO:teuthology.orchestra.run.smithi118.stdout: "error", 2024-03-24T03:17:24.163 INFO:teuthology.orchestra.run.smithi118.stdout: "info", 2024-03-24T03:17:24.163 INFO:teuthology.orchestra.run.smithi118.stdout: "warning" 2024-03-24T03:17:24.163 INFO:teuthology.orchestra.run.smithi118.stdout: ], 2024-03-24T03:17:24.163 INFO:teuthology.orchestra.run.smithi118.stdout: "desc": "", 2024-03-24T03:17:24.163 INFO:teuthology.orchestra.run.smithi118.stdout: "long_desc": "", 2024-03-24T03:17:24.163 INFO:teuthology.orchestra.run.smithi118.stdout: "tags": [], 2024-03-24T03:17:24.163 INFO:teuthology.orchestra.run.smithi118.stdout: "see_also": [] 2024-03-24T03:17:24.163 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-24T03:17:24.164 INFO:teuthology.orchestra.run.smithi118.stdout: "log_to_file": { 2024-03-24T03:17:24.164 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "log_to_file", 2024-03-24T03:17:24.164 INFO:teuthology.orchestra.run.smithi118.stdout: "type": "bool", 2024-03-24T03:17:24.164 INFO:teuthology.orchestra.run.smithi118.stdout: "level": "advanced", 2024-03-24T03:17:24.164 INFO:teuthology.orchestra.run.smithi118.stdout: "flags": 1, 2024-03-24T03:17:24.164 INFO:teuthology.orchestra.run.smithi118.stdout: "default_value": "False", 2024-03-24T03:17:24.164 INFO:teuthology.orchestra.run.smithi118.stdout: "min": "", 2024-03-24T03:17:24.164 INFO:teuthology.orchestra.run.smithi118.stdout: "max": "", 2024-03-24T03:17:24.165 INFO:teuthology.orchestra.run.smithi118.stdout: "enum_allowed": [], 2024-03-24T03:17:24.165 INFO:teuthology.orchestra.run.smithi118.stdout: "desc": "", 2024-03-24T03:17:24.165 INFO:teuthology.orchestra.run.smithi118.stdout: "long_desc": "", 2024-03-24T03:17:24.165 INFO:teuthology.orchestra.run.smithi118.stdout: "tags": [], 2024-03-24T03:17:24.165 INFO:teuthology.orchestra.run.smithi118.stdout: "see_also": [] 2024-03-24T03:17:24.165 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-24T03:17:24.165 INFO:teuthology.orchestra.run.smithi118.stdout: "mark_out_threshold": { 2024-03-24T03:17:24.165 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "mark_out_threshold", 2024-03-24T03:17:24.165 INFO:teuthology.orchestra.run.smithi118.stdout: "type": "secs", 2024-03-24T03:17:24.165 INFO:teuthology.orchestra.run.smithi118.stdout: "level": "advanced", 2024-03-24T03:17:24.166 INFO:teuthology.orchestra.run.smithi118.stdout: "flags": 1, 2024-03-24T03:17:24.166 INFO:teuthology.orchestra.run.smithi118.stdout: "default_value": "2419200", 2024-03-24T03:17:24.166 INFO:teuthology.orchestra.run.smithi118.stdout: "min": "", 2024-03-24T03:17:24.166 INFO:teuthology.orchestra.run.smithi118.stdout: "max": "", 2024-03-24T03:17:24.166 INFO:teuthology.orchestra.run.smithi118.stdout: "enum_allowed": [], 2024-03-24T03:17:24.166 INFO:teuthology.orchestra.run.smithi118.stdout: "desc": "automatically mark OSD if it may fail before this long", 2024-03-24T03:17:24.166 INFO:teuthology.orchestra.run.smithi118.stdout: "long_desc": "", 2024-03-24T03:17:24.166 INFO:teuthology.orchestra.run.smithi118.stdout: "tags": [], 2024-03-24T03:17:24.166 INFO:teuthology.orchestra.run.smithi118.stdout: "see_also": [] 2024-03-24T03:17:24.167 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-24T03:17:24.167 INFO:teuthology.orchestra.run.smithi118.stdout: "pool_name": { 2024-03-24T03:17:24.167 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "pool_name", 2024-03-24T03:17:24.167 INFO:teuthology.orchestra.run.smithi118.stdout: "type": "str", 2024-03-24T03:17:24.167 INFO:teuthology.orchestra.run.smithi118.stdout: "level": "advanced", 2024-03-24T03:17:24.167 INFO:teuthology.orchestra.run.smithi118.stdout: "flags": 1, 2024-03-24T03:17:24.167 INFO:teuthology.orchestra.run.smithi118.stdout: "default_value": "device_health_metrics", 2024-03-24T03:17:24.167 INFO:teuthology.orchestra.run.smithi118.stdout: "min": "", 2024-03-24T03:17:24.167 INFO:teuthology.orchestra.run.smithi118.stdout: "max": "", 2024-03-24T03:17:24.168 INFO:teuthology.orchestra.run.smithi118.stdout: "enum_allowed": [], 2024-03-24T03:17:24.168 INFO:teuthology.orchestra.run.smithi118.stdout: "desc": "name of pool in which to store device health metrics", 2024-03-24T03:17:24.168 INFO:teuthology.orchestra.run.smithi118.stdout: "long_desc": "", 2024-03-24T03:17:24.168 INFO:teuthology.orchestra.run.smithi118.stdout: "tags": [], 2024-03-24T03:17:24.168 INFO:teuthology.orchestra.run.smithi118.stdout: "see_also": [] 2024-03-24T03:17:24.168 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-24T03:17:24.168 INFO:teuthology.orchestra.run.smithi118.stdout: "retention_period": { 2024-03-24T03:17:24.168 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "retention_period", 2024-03-24T03:17:24.168 INFO:teuthology.orchestra.run.smithi118.stdout: "type": "secs", 2024-03-24T03:17:24.168 INFO:teuthology.orchestra.run.smithi118.stdout: "level": "advanced", 2024-03-24T03:17:24.169 INFO:teuthology.orchestra.run.smithi118.stdout: "flags": 1, 2024-03-24T03:17:24.169 INFO:teuthology.orchestra.run.smithi118.stdout: "default_value": "15552000", 2024-03-24T03:17:24.169 INFO:teuthology.orchestra.run.smithi118.stdout: "min": "", 2024-03-24T03:17:24.169 INFO:teuthology.orchestra.run.smithi118.stdout: "max": "", 2024-03-24T03:17:24.169 INFO:teuthology.orchestra.run.smithi118.stdout: "enum_allowed": [], 2024-03-24T03:17:24.169 INFO:teuthology.orchestra.run.smithi118.stdout: "desc": "how long to retain device health metrics", 2024-03-24T03:17:24.169 INFO:teuthology.orchestra.run.smithi118.stdout: "long_desc": "", 2024-03-24T03:17:24.169 INFO:teuthology.orchestra.run.smithi118.stdout: "tags": [], 2024-03-24T03:17:24.169 INFO:teuthology.orchestra.run.smithi118.stdout: "see_also": [] 2024-03-24T03:17:24.169 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-24T03:17:24.170 INFO:teuthology.orchestra.run.smithi118.stdout: "scrape_frequency": { 2024-03-24T03:17:24.170 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "scrape_frequency", 2024-03-24T03:17:24.170 INFO:teuthology.orchestra.run.smithi118.stdout: "type": "secs", 2024-03-24T03:17:24.170 INFO:teuthology.orchestra.run.smithi118.stdout: "level": "advanced", 2024-03-24T03:17:24.170 INFO:teuthology.orchestra.run.smithi118.stdout: "flags": 1, 2024-03-24T03:17:24.170 INFO:teuthology.orchestra.run.smithi118.stdout: "default_value": "86400", 2024-03-24T03:17:24.170 INFO:teuthology.orchestra.run.smithi118.stdout: "min": "", 2024-03-24T03:17:24.170 INFO:teuthology.orchestra.run.smithi118.stdout: "max": "", 2024-03-24T03:17:24.171 INFO:teuthology.orchestra.run.smithi118.stdout: "enum_allowed": [], 2024-03-24T03:17:24.171 INFO:teuthology.orchestra.run.smithi118.stdout: "desc": "how frequently to scrape device health metrics", 2024-03-24T03:17:24.171 INFO:teuthology.orchestra.run.smithi118.stdout: "long_desc": "", 2024-03-24T03:17:24.171 INFO:teuthology.orchestra.run.smithi118.stdout: "tags": [], 2024-03-24T03:17:24.171 INFO:teuthology.orchestra.run.smithi118.stdout: "see_also": [] 2024-03-24T03:17:24.171 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-24T03:17:24.171 INFO:teuthology.orchestra.run.smithi118.stdout: "self_heal": { 2024-03-24T03:17:24.171 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "self_heal", 2024-03-24T03:17:24.171 INFO:teuthology.orchestra.run.smithi118.stdout: "type": "bool", 2024-03-24T03:17:24.172 INFO:teuthology.orchestra.run.smithi118.stdout: "level": "advanced", 2024-03-24T03:17:24.172 INFO:teuthology.orchestra.run.smithi118.stdout: "flags": 1, 2024-03-24T03:17:24.172 INFO:teuthology.orchestra.run.smithi118.stdout: "default_value": "True", 2024-03-24T03:17:24.172 INFO:teuthology.orchestra.run.smithi118.stdout: "min": "", 2024-03-24T03:17:24.172 INFO:teuthology.orchestra.run.smithi118.stdout: "max": "", 2024-03-24T03:17:24.172 INFO:teuthology.orchestra.run.smithi118.stdout: "enum_allowed": [], 2024-03-24T03:17:24.172 INFO:teuthology.orchestra.run.smithi118.stdout: "desc": "preemptively heal cluster around devices that may fail", 2024-03-24T03:17:24.172 INFO:teuthology.orchestra.run.smithi118.stdout: "long_desc": "", 2024-03-24T03:17:24.172 INFO:teuthology.orchestra.run.smithi118.stdout: "tags": [], 2024-03-24T03:17:24.173 INFO:teuthology.orchestra.run.smithi118.stdout: "see_also": [] 2024-03-24T03:17:24.173 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-24T03:17:24.173 INFO:teuthology.orchestra.run.smithi118.stdout: "sleep_interval": { 2024-03-24T03:17:24.173 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "sleep_interval", 2024-03-24T03:17:24.173 INFO:teuthology.orchestra.run.smithi118.stdout: "type": "secs", 2024-03-24T03:17:24.173 INFO:teuthology.orchestra.run.smithi118.stdout: "level": "advanced", 2024-03-24T03:17:24.173 INFO:teuthology.orchestra.run.smithi118.stdout: "flags": 1, 2024-03-24T03:17:24.173 INFO:teuthology.orchestra.run.smithi118.stdout: "default_value": "600", 2024-03-24T03:17:24.173 INFO:teuthology.orchestra.run.smithi118.stdout: "min": "", 2024-03-24T03:17:24.173 INFO:teuthology.orchestra.run.smithi118.stdout: "max": "", 2024-03-24T03:17:24.174 INFO:teuthology.orchestra.run.smithi118.stdout: "enum_allowed": [], 2024-03-24T03:17:24.174 INFO:teuthology.orchestra.run.smithi118.stdout: "desc": "how frequently to wake up and check device health", 2024-03-24T03:17:24.174 INFO:teuthology.orchestra.run.smithi118.stdout: "long_desc": "", 2024-03-24T03:17:24.174 INFO:teuthology.orchestra.run.smithi118.stdout: "tags": [], 2024-03-24T03:17:24.174 INFO:teuthology.orchestra.run.smithi118.stdout: "see_also": [] 2024-03-24T03:17:24.174 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-24T03:17:24.174 INFO:teuthology.orchestra.run.smithi118.stdout: "warn_threshold": { 2024-03-24T03:17:24.174 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "warn_threshold", 2024-03-24T03:17:24.175 INFO:teuthology.orchestra.run.smithi118.stdout: "type": "secs", 2024-03-24T03:17:24.175 INFO:teuthology.orchestra.run.smithi118.stdout: "level": "advanced", 2024-03-24T03:17:24.175 INFO:teuthology.orchestra.run.smithi118.stdout: "flags": 1, 2024-03-24T03:17:24.175 INFO:teuthology.orchestra.run.smithi118.stdout: "default_value": "7257600", 2024-03-24T03:17:24.175 INFO:teuthology.orchestra.run.smithi118.stdout: "min": "", 2024-03-24T03:17:24.175 INFO:teuthology.orchestra.run.smithi118.stdout: "max": "", 2024-03-24T03:17:24.175 INFO:teuthology.orchestra.run.smithi118.stdout: "enum_allowed": [], 2024-03-24T03:17:24.175 INFO:teuthology.orchestra.run.smithi118.stdout: "desc": "raise health warning if OSD may fail before this long", 2024-03-24T03:17:24.175 INFO:teuthology.orchestra.run.smithi118.stdout: "long_desc": "", 2024-03-24T03:17:24.175 INFO:teuthology.orchestra.run.smithi118.stdout: "tags": [], 2024-03-24T03:17:24.176 INFO:teuthology.orchestra.run.smithi118.stdout: "see_also": [] 2024-03-24T03:17:24.176 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-03-24T03:17:24.176 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-03-24T03:17:24.176 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-24T03:17:24.176 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-24T03:17:24.176 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "diskprediction_local", 2024-03-24T03:17:24.176 INFO:teuthology.orchestra.run.smithi118.stdout: "can_run": true, 2024-03-24T03:17:24.176 INFO:teuthology.orchestra.run.smithi118.stdout: "error_string": "", 2024-03-24T03:17:24.176 INFO:teuthology.orchestra.run.smithi118.stdout: "module_options": { 2024-03-24T03:17:24.177 INFO:teuthology.orchestra.run.smithi118.stdout: "log_level": { 2024-03-24T03:17:24.177 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "log_level", 2024-03-24T03:17:24.177 INFO:teuthology.orchestra.run.smithi118.stdout: "type": "str", 2024-03-24T03:17:24.177 INFO:teuthology.orchestra.run.smithi118.stdout: "level": "advanced", 2024-03-24T03:17:24.177 INFO:teuthology.orchestra.run.smithi118.stdout: "flags": 1, 2024-03-24T03:17:24.177 INFO:teuthology.orchestra.run.smithi118.stdout: "default_value": "", 2024-03-24T03:17:24.177 INFO:teuthology.orchestra.run.smithi118.stdout: "min": "", 2024-03-24T03:17:24.177 INFO:teuthology.orchestra.run.smithi118.stdout: "max": "", 2024-03-24T03:17:24.177 INFO:teuthology.orchestra.run.smithi118.stdout: "enum_allowed": [ 2024-03-24T03:17:24.177 INFO:teuthology.orchestra.run.smithi118.stdout: "", 2024-03-24T03:17:24.178 INFO:teuthology.orchestra.run.smithi118.stdout: "critical", 2024-03-24T03:17:24.178 INFO:teuthology.orchestra.run.smithi118.stdout: "debug", 2024-03-24T03:17:24.178 INFO:teuthology.orchestra.run.smithi118.stdout: "error", 2024-03-24T03:17:24.178 INFO:teuthology.orchestra.run.smithi118.stdout: "info", 2024-03-24T03:17:24.178 INFO:teuthology.orchestra.run.smithi118.stdout: "warning" 2024-03-24T03:17:24.178 INFO:teuthology.orchestra.run.smithi118.stdout: ], 2024-03-24T03:17:24.178 INFO:teuthology.orchestra.run.smithi118.stdout: "desc": "", 2024-03-24T03:17:24.178 INFO:teuthology.orchestra.run.smithi118.stdout: "long_desc": "", 2024-03-24T03:17:24.178 INFO:teuthology.orchestra.run.smithi118.stdout: "tags": [], 2024-03-24T03:17:24.179 INFO:teuthology.orchestra.run.smithi118.stdout: "see_also": [] 2024-03-24T03:17:24.179 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-24T03:17:24.179 INFO:teuthology.orchestra.run.smithi118.stdout: "log_to_cluster": { 2024-03-24T03:17:24.179 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "log_to_cluster", 2024-03-24T03:17:24.179 INFO:teuthology.orchestra.run.smithi118.stdout: "type": "bool", 2024-03-24T03:17:24.179 INFO:teuthology.orchestra.run.smithi118.stdout: "level": "advanced", 2024-03-24T03:17:24.179 INFO:teuthology.orchestra.run.smithi118.stdout: "flags": 1, 2024-03-24T03:17:24.179 INFO:teuthology.orchestra.run.smithi118.stdout: "default_value": "False", 2024-03-24T03:17:24.179 INFO:teuthology.orchestra.run.smithi118.stdout: "min": "", 2024-03-24T03:17:24.179 INFO:teuthology.orchestra.run.smithi118.stdout: "max": "", 2024-03-24T03:17:24.180 INFO:teuthology.orchestra.run.smithi118.stdout: "enum_allowed": [], 2024-03-24T03:17:24.180 INFO:teuthology.orchestra.run.smithi118.stdout: "desc": "", 2024-03-24T03:17:24.180 INFO:teuthology.orchestra.run.smithi118.stdout: "long_desc": "", 2024-03-24T03:17:24.180 INFO:teuthology.orchestra.run.smithi118.stdout: "tags": [], 2024-03-24T03:17:24.180 INFO:teuthology.orchestra.run.smithi118.stdout: "see_also": [] 2024-03-24T03:17:24.180 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-24T03:17:24.180 INFO:teuthology.orchestra.run.smithi118.stdout: "log_to_cluster_level": { 2024-03-24T03:17:24.180 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "log_to_cluster_level", 2024-03-24T03:17:24.180 INFO:teuthology.orchestra.run.smithi118.stdout: "type": "str", 2024-03-24T03:17:24.181 INFO:teuthology.orchestra.run.smithi118.stdout: "level": "advanced", 2024-03-24T03:17:24.181 INFO:teuthology.orchestra.run.smithi118.stdout: "flags": 1, 2024-03-24T03:17:24.181 INFO:teuthology.orchestra.run.smithi118.stdout: "default_value": "info", 2024-03-24T03:17:24.181 INFO:teuthology.orchestra.run.smithi118.stdout: "min": "", 2024-03-24T03:17:24.181 INFO:teuthology.orchestra.run.smithi118.stdout: "max": "", 2024-03-24T03:17:24.181 INFO:teuthology.orchestra.run.smithi118.stdout: "enum_allowed": [ 2024-03-24T03:17:24.181 INFO:teuthology.orchestra.run.smithi118.stdout: "", 2024-03-24T03:17:24.181 INFO:teuthology.orchestra.run.smithi118.stdout: "critical", 2024-03-24T03:17:24.181 INFO:teuthology.orchestra.run.smithi118.stdout: "debug", 2024-03-24T03:17:24.181 INFO:teuthology.orchestra.run.smithi118.stdout: "error", 2024-03-24T03:17:24.182 INFO:teuthology.orchestra.run.smithi118.stdout: "info", 2024-03-24T03:17:24.182 INFO:teuthology.orchestra.run.smithi118.stdout: "warning" 2024-03-24T03:17:24.182 INFO:teuthology.orchestra.run.smithi118.stdout: ], 2024-03-24T03:17:24.182 INFO:teuthology.orchestra.run.smithi118.stdout: "desc": "", 2024-03-24T03:17:24.182 INFO:teuthology.orchestra.run.smithi118.stdout: "long_desc": "", 2024-03-24T03:17:24.182 INFO:teuthology.orchestra.run.smithi118.stdout: "tags": [], 2024-03-24T03:17:24.182 INFO:teuthology.orchestra.run.smithi118.stdout: "see_also": [] 2024-03-24T03:17:24.182 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-24T03:17:24.182 INFO:teuthology.orchestra.run.smithi118.stdout: "log_to_file": { 2024-03-24T03:17:24.183 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "log_to_file", 2024-03-24T03:17:24.183 INFO:teuthology.orchestra.run.smithi118.stdout: "type": "bool", 2024-03-24T03:17:24.183 INFO:teuthology.orchestra.run.smithi118.stdout: "level": "advanced", 2024-03-24T03:17:24.183 INFO:teuthology.orchestra.run.smithi118.stdout: "flags": 1, 2024-03-24T03:17:24.183 INFO:teuthology.orchestra.run.smithi118.stdout: "default_value": "False", 2024-03-24T03:17:24.183 INFO:teuthology.orchestra.run.smithi118.stdout: "min": "", 2024-03-24T03:17:24.183 INFO:teuthology.orchestra.run.smithi118.stdout: "max": "", 2024-03-24T03:17:24.183 INFO:teuthology.orchestra.run.smithi118.stdout: "enum_allowed": [], 2024-03-24T03:17:24.183 INFO:teuthology.orchestra.run.smithi118.stdout: "desc": "", 2024-03-24T03:17:24.184 INFO:teuthology.orchestra.run.smithi118.stdout: "long_desc": "", 2024-03-24T03:17:24.184 INFO:teuthology.orchestra.run.smithi118.stdout: "tags": [], 2024-03-24T03:17:24.184 INFO:teuthology.orchestra.run.smithi118.stdout: "see_also": [] 2024-03-24T03:17:24.184 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-24T03:17:24.184 INFO:teuthology.orchestra.run.smithi118.stdout: "predict_interval": { 2024-03-24T03:17:24.184 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "predict_interval", 2024-03-24T03:17:24.184 INFO:teuthology.orchestra.run.smithi118.stdout: "type": "str", 2024-03-24T03:17:24.184 INFO:teuthology.orchestra.run.smithi118.stdout: "level": "advanced", 2024-03-24T03:17:24.184 INFO:teuthology.orchestra.run.smithi118.stdout: "flags": 0, 2024-03-24T03:17:24.185 INFO:teuthology.orchestra.run.smithi118.stdout: "default_value": "86400", 2024-03-24T03:17:24.185 INFO:teuthology.orchestra.run.smithi118.stdout: "min": "", 2024-03-24T03:17:24.185 INFO:teuthology.orchestra.run.smithi118.stdout: "max": "", 2024-03-24T03:17:24.185 INFO:teuthology.orchestra.run.smithi118.stdout: "enum_allowed": [], 2024-03-24T03:17:24.185 INFO:teuthology.orchestra.run.smithi118.stdout: "desc": "", 2024-03-24T03:17:24.185 INFO:teuthology.orchestra.run.smithi118.stdout: "long_desc": "", 2024-03-24T03:17:24.185 INFO:teuthology.orchestra.run.smithi118.stdout: "tags": [], 2024-03-24T03:17:24.185 INFO:teuthology.orchestra.run.smithi118.stdout: "see_also": [] 2024-03-24T03:17:24.185 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-24T03:17:24.186 INFO:teuthology.orchestra.run.smithi118.stdout: "predictor_model": { 2024-03-24T03:17:24.186 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "predictor_model", 2024-03-24T03:17:24.186 INFO:teuthology.orchestra.run.smithi118.stdout: "type": "str", 2024-03-24T03:17:24.186 INFO:teuthology.orchestra.run.smithi118.stdout: "level": "advanced", 2024-03-24T03:17:24.186 INFO:teuthology.orchestra.run.smithi118.stdout: "flags": 0, 2024-03-24T03:17:24.186 INFO:teuthology.orchestra.run.smithi118.stdout: "default_value": "prophetstor", 2024-03-24T03:17:24.186 INFO:teuthology.orchestra.run.smithi118.stdout: "min": "", 2024-03-24T03:17:24.187 INFO:teuthology.orchestra.run.smithi118.stdout: "max": "", 2024-03-24T03:17:24.187 INFO:teuthology.orchestra.run.smithi118.stdout: "enum_allowed": [], 2024-03-24T03:17:24.187 INFO:teuthology.orchestra.run.smithi118.stdout: "desc": "", 2024-03-24T03:17:24.187 INFO:teuthology.orchestra.run.smithi118.stdout: "long_desc": "", 2024-03-24T03:17:24.187 INFO:teuthology.orchestra.run.smithi118.stdout: "tags": [], 2024-03-24T03:17:24.187 INFO:teuthology.orchestra.run.smithi118.stdout: "see_also": [] 2024-03-24T03:17:24.187 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-24T03:17:24.187 INFO:teuthology.orchestra.run.smithi118.stdout: "sleep_interval": { 2024-03-24T03:17:24.187 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "sleep_interval", 2024-03-24T03:17:24.188 INFO:teuthology.orchestra.run.smithi118.stdout: "type": "str", 2024-03-24T03:17:24.188 INFO:teuthology.orchestra.run.smithi118.stdout: "level": "advanced", 2024-03-24T03:17:24.188 INFO:teuthology.orchestra.run.smithi118.stdout: "flags": 0, 2024-03-24T03:17:24.188 INFO:teuthology.orchestra.run.smithi118.stdout: "default_value": "600", 2024-03-24T03:17:24.188 INFO:teuthology.orchestra.run.smithi118.stdout: "min": "", 2024-03-24T03:17:24.188 INFO:teuthology.orchestra.run.smithi118.stdout: "max": "", 2024-03-24T03:17:24.188 INFO:teuthology.orchestra.run.smithi118.stdout: "enum_allowed": [], 2024-03-24T03:17:24.188 INFO:teuthology.orchestra.run.smithi118.stdout: "desc": "", 2024-03-24T03:17:24.188 INFO:teuthology.orchestra.run.smithi118.stdout: "long_desc": "", 2024-03-24T03:17:24.189 INFO:teuthology.orchestra.run.smithi118.stdout: "tags": [], 2024-03-24T03:17:24.189 INFO:teuthology.orchestra.run.smithi118.stdout: "see_also": [] 2024-03-24T03:17:24.189 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-03-24T03:17:24.189 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-03-24T03:17:24.189 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-24T03:17:24.189 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-24T03:17:24.189 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "influx", 2024-03-24T03:17:24.189 INFO:teuthology.orchestra.run.smithi118.stdout: "can_run": false, 2024-03-24T03:17:24.189 INFO:teuthology.orchestra.run.smithi118.stdout: "error_string": "influxdb python module not found", 2024-03-24T03:17:24.189 INFO:teuthology.orchestra.run.smithi118.stdout: "module_options": { 2024-03-24T03:17:24.190 INFO:teuthology.orchestra.run.smithi118.stdout: "batch_size": { 2024-03-24T03:17:24.190 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "batch_size", 2024-03-24T03:17:24.190 INFO:teuthology.orchestra.run.smithi118.stdout: "type": "int", 2024-03-24T03:17:24.190 INFO:teuthology.orchestra.run.smithi118.stdout: "level": "advanced", 2024-03-24T03:17:24.190 INFO:teuthology.orchestra.run.smithi118.stdout: "flags": 0, 2024-03-24T03:17:24.190 INFO:teuthology.orchestra.run.smithi118.stdout: "default_value": "5000", 2024-03-24T03:17:24.193 INFO:teuthology.orchestra.run.smithi118.stdout: "min": "", 2024-03-24T03:17:24.193 INFO:teuthology.orchestra.run.smithi118.stdout: "max": "", 2024-03-24T03:17:24.193 INFO:teuthology.orchestra.run.smithi118.stdout: "enum_allowed": [], 2024-03-24T03:17:24.193 INFO:teuthology.orchestra.run.smithi118.stdout: "desc": "How big batches of data points should be when sending to InfluxDB.", 2024-03-24T03:17:24.193 INFO:teuthology.orchestra.run.smithi118.stdout: "long_desc": "", 2024-03-24T03:17:24.193 INFO:teuthology.orchestra.run.smithi118.stdout: "tags": [], 2024-03-24T03:17:24.194 INFO:teuthology.orchestra.run.smithi118.stdout: "see_also": [] 2024-03-24T03:17:24.194 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-24T03:17:24.194 INFO:teuthology.orchestra.run.smithi118.stdout: "database": { 2024-03-24T03:17:24.194 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "database", 2024-03-24T03:17:24.194 INFO:teuthology.orchestra.run.smithi118.stdout: "type": "str", 2024-03-24T03:17:24.194 INFO:teuthology.orchestra.run.smithi118.stdout: "level": "advanced", 2024-03-24T03:17:24.194 INFO:teuthology.orchestra.run.smithi118.stdout: "flags": 0, 2024-03-24T03:17:24.194 INFO:teuthology.orchestra.run.smithi118.stdout: "default_value": "ceph", 2024-03-24T03:17:24.195 INFO:teuthology.orchestra.run.smithi118.stdout: "min": "", 2024-03-24T03:17:24.195 INFO:teuthology.orchestra.run.smithi118.stdout: "max": "", 2024-03-24T03:17:24.195 INFO:teuthology.orchestra.run.smithi118.stdout: "enum_allowed": [], 2024-03-24T03:17:24.195 INFO:teuthology.orchestra.run.smithi118.stdout: "desc": "InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.", 2024-03-24T03:17:24.195 INFO:teuthology.orchestra.run.smithi118.stdout: "long_desc": "", 2024-03-24T03:17:24.195 INFO:teuthology.orchestra.run.smithi118.stdout: "tags": [], 2024-03-24T03:17:24.195 INFO:teuthology.orchestra.run.smithi118.stdout: "see_also": [] 2024-03-24T03:17:24.195 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-24T03:17:24.195 INFO:teuthology.orchestra.run.smithi118.stdout: "hostname": { 2024-03-24T03:17:24.196 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "hostname", 2024-03-24T03:17:24.196 INFO:teuthology.orchestra.run.smithi118.stdout: "type": "str", 2024-03-24T03:17:24.196 INFO:teuthology.orchestra.run.smithi118.stdout: "level": "advanced", 2024-03-24T03:17:24.196 INFO:teuthology.orchestra.run.smithi118.stdout: "flags": 0, 2024-03-24T03:17:24.196 INFO:teuthology.orchestra.run.smithi118.stdout: "default_value": "", 2024-03-24T03:17:24.196 INFO:teuthology.orchestra.run.smithi118.stdout: "min": "", 2024-03-24T03:17:24.196 INFO:teuthology.orchestra.run.smithi118.stdout: "max": "", 2024-03-24T03:17:24.196 INFO:teuthology.orchestra.run.smithi118.stdout: "enum_allowed": [], 2024-03-24T03:17:24.196 INFO:teuthology.orchestra.run.smithi118.stdout: "desc": "InfluxDB server hostname", 2024-03-24T03:17:24.197 INFO:teuthology.orchestra.run.smithi118.stdout: "long_desc": "", 2024-03-24T03:17:24.197 INFO:teuthology.orchestra.run.smithi118.stdout: "tags": [], 2024-03-24T03:17:24.197 INFO:teuthology.orchestra.run.smithi118.stdout: "see_also": [] 2024-03-24T03:17:24.197 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-24T03:17:24.197 INFO:teuthology.orchestra.run.smithi118.stdout: "interval": { 2024-03-24T03:17:24.197 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "interval", 2024-03-24T03:17:24.197 INFO:teuthology.orchestra.run.smithi118.stdout: "type": "secs", 2024-03-24T03:17:24.197 INFO:teuthology.orchestra.run.smithi118.stdout: "level": "advanced", 2024-03-24T03:17:24.197 INFO:teuthology.orchestra.run.smithi118.stdout: "flags": 0, 2024-03-24T03:17:24.198 INFO:teuthology.orchestra.run.smithi118.stdout: "default_value": "30", 2024-03-24T03:17:24.198 INFO:teuthology.orchestra.run.smithi118.stdout: "min": "5", 2024-03-24T03:17:24.198 INFO:teuthology.orchestra.run.smithi118.stdout: "max": "", 2024-03-24T03:17:24.198 INFO:teuthology.orchestra.run.smithi118.stdout: "enum_allowed": [], 2024-03-24T03:17:24.198 INFO:teuthology.orchestra.run.smithi118.stdout: "desc": "Time between reports to InfluxDB. Default 30 seconds.", 2024-03-24T03:17:24.198 INFO:teuthology.orchestra.run.smithi118.stdout: "long_desc": "", 2024-03-24T03:17:24.198 INFO:teuthology.orchestra.run.smithi118.stdout: "tags": [], 2024-03-24T03:17:24.198 INFO:teuthology.orchestra.run.smithi118.stdout: "see_also": [] 2024-03-24T03:17:24.198 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-24T03:17:24.199 INFO:teuthology.orchestra.run.smithi118.stdout: "log_level": { 2024-03-24T03:17:24.199 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "log_level", 2024-03-24T03:17:24.199 INFO:teuthology.orchestra.run.smithi118.stdout: "type": "str", 2024-03-24T03:17:24.199 INFO:teuthology.orchestra.run.smithi118.stdout: "level": "advanced", 2024-03-24T03:17:24.199 INFO:teuthology.orchestra.run.smithi118.stdout: "flags": 1, 2024-03-24T03:17:24.199 INFO:teuthology.orchestra.run.smithi118.stdout: "default_value": "", 2024-03-24T03:17:24.199 INFO:teuthology.orchestra.run.smithi118.stdout: "min": "", 2024-03-24T03:17:24.199 INFO:teuthology.orchestra.run.smithi118.stdout: "max": "", 2024-03-24T03:17:24.199 INFO:teuthology.orchestra.run.smithi118.stdout: "enum_allowed": [ 2024-03-24T03:17:24.200 INFO:teuthology.orchestra.run.smithi118.stdout: "", 2024-03-24T03:17:24.200 INFO:teuthology.orchestra.run.smithi118.stdout: "critical", 2024-03-24T03:17:24.200 INFO:teuthology.orchestra.run.smithi118.stdout: "debug", 2024-03-24T03:17:24.200 INFO:teuthology.orchestra.run.smithi118.stdout: "error", 2024-03-24T03:17:24.200 INFO:teuthology.orchestra.run.smithi118.stdout: "info", 2024-03-24T03:17:24.200 INFO:teuthology.orchestra.run.smithi118.stdout: "warning" 2024-03-24T03:17:24.200 INFO:teuthology.orchestra.run.smithi118.stdout: ], 2024-03-24T03:17:24.200 INFO:teuthology.orchestra.run.smithi118.stdout: "desc": "", 2024-03-24T03:17:24.200 INFO:teuthology.orchestra.run.smithi118.stdout: "long_desc": "", 2024-03-24T03:17:24.201 INFO:teuthology.orchestra.run.smithi118.stdout: "tags": [], 2024-03-24T03:17:24.201 INFO:teuthology.orchestra.run.smithi118.stdout: "see_also": [] 2024-03-24T03:17:24.201 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-24T03:17:24.201 INFO:teuthology.orchestra.run.smithi118.stdout: "log_to_cluster": { 2024-03-24T03:17:24.201 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "log_to_cluster", 2024-03-24T03:17:24.201 INFO:teuthology.orchestra.run.smithi118.stdout: "type": "bool", 2024-03-24T03:17:24.201 INFO:teuthology.orchestra.run.smithi118.stdout: "level": "advanced", 2024-03-24T03:17:24.201 INFO:teuthology.orchestra.run.smithi118.stdout: "flags": 1, 2024-03-24T03:17:24.201 INFO:teuthology.orchestra.run.smithi118.stdout: "default_value": "False", 2024-03-24T03:17:24.201 INFO:teuthology.orchestra.run.smithi118.stdout: "min": "", 2024-03-24T03:17:24.202 INFO:teuthology.orchestra.run.smithi118.stdout: "max": "", 2024-03-24T03:17:24.202 INFO:teuthology.orchestra.run.smithi118.stdout: "enum_allowed": [], 2024-03-24T03:17:24.202 INFO:teuthology.orchestra.run.smithi118.stdout: "desc": "", 2024-03-24T03:17:24.202 INFO:teuthology.orchestra.run.smithi118.stdout: "long_desc": "", 2024-03-24T03:17:24.202 INFO:teuthology.orchestra.run.smithi118.stdout: "tags": [], 2024-03-24T03:17:24.202 INFO:teuthology.orchestra.run.smithi118.stdout: "see_also": [] 2024-03-24T03:17:24.202 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-24T03:17:24.202 INFO:teuthology.orchestra.run.smithi118.stdout: "log_to_cluster_level": { 2024-03-24T03:17:24.203 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "log_to_cluster_level", 2024-03-24T03:17:24.203 INFO:teuthology.orchestra.run.smithi118.stdout: "type": "str", 2024-03-24T03:17:24.203 INFO:teuthology.orchestra.run.smithi118.stdout: "level": "advanced", 2024-03-24T03:17:24.203 INFO:teuthology.orchestra.run.smithi118.stdout: "flags": 1, 2024-03-24T03:17:24.203 INFO:teuthology.orchestra.run.smithi118.stdout: "default_value": "info", 2024-03-24T03:17:24.203 INFO:teuthology.orchestra.run.smithi118.stdout: "min": "", 2024-03-24T03:17:24.203 INFO:teuthology.orchestra.run.smithi118.stdout: "max": "", 2024-03-24T03:17:24.203 INFO:teuthology.orchestra.run.smithi118.stdout: "enum_allowed": [ 2024-03-24T03:17:24.203 INFO:teuthology.orchestra.run.smithi118.stdout: "", 2024-03-24T03:17:24.204 INFO:teuthology.orchestra.run.smithi118.stdout: "critical", 2024-03-24T03:17:24.204 INFO:teuthology.orchestra.run.smithi118.stdout: "debug", 2024-03-24T03:17:24.204 INFO:teuthology.orchestra.run.smithi118.stdout: "error", 2024-03-24T03:17:24.204 INFO:teuthology.orchestra.run.smithi118.stdout: "info", 2024-03-24T03:17:24.204 INFO:teuthology.orchestra.run.smithi118.stdout: "warning" 2024-03-24T03:17:24.204 INFO:teuthology.orchestra.run.smithi118.stdout: ], 2024-03-24T03:17:24.204 INFO:teuthology.orchestra.run.smithi118.stdout: "desc": "", 2024-03-24T03:17:24.204 INFO:teuthology.orchestra.run.smithi118.stdout: "long_desc": "", 2024-03-24T03:17:24.205 INFO:teuthology.orchestra.run.smithi118.stdout: "tags": [], 2024-03-24T03:17:24.205 INFO:teuthology.orchestra.run.smithi118.stdout: "see_also": [] 2024-03-24T03:17:24.205 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-24T03:17:24.205 INFO:teuthology.orchestra.run.smithi118.stdout: "log_to_file": { 2024-03-24T03:17:24.205 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "log_to_file", 2024-03-24T03:17:24.205 INFO:teuthology.orchestra.run.smithi118.stdout: "type": "bool", 2024-03-24T03:17:24.205 INFO:teuthology.orchestra.run.smithi118.stdout: "level": "advanced", 2024-03-24T03:17:24.205 INFO:teuthology.orchestra.run.smithi118.stdout: "flags": 1, 2024-03-24T03:17:24.205 INFO:teuthology.orchestra.run.smithi118.stdout: "default_value": "False", 2024-03-24T03:17:24.205 INFO:teuthology.orchestra.run.smithi118.stdout: "min": "", 2024-03-24T03:17:24.206 INFO:teuthology.orchestra.run.smithi118.stdout: "max": "", 2024-03-24T03:17:24.206 INFO:teuthology.orchestra.run.smithi118.stdout: "enum_allowed": [], 2024-03-24T03:17:24.206 INFO:teuthology.orchestra.run.smithi118.stdout: "desc": "", 2024-03-24T03:17:24.206 INFO:teuthology.orchestra.run.smithi118.stdout: "long_desc": "", 2024-03-24T03:17:24.206 INFO:teuthology.orchestra.run.smithi118.stdout: "tags": [], 2024-03-24T03:17:24.206 INFO:teuthology.orchestra.run.smithi118.stdout: "see_also": [] 2024-03-24T03:17:24.206 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-24T03:17:24.206 INFO:teuthology.orchestra.run.smithi118.stdout: "password": { 2024-03-24T03:17:24.207 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "password", 2024-03-24T03:17:24.207 INFO:teuthology.orchestra.run.smithi118.stdout: "type": "str", 2024-03-24T03:17:24.207 INFO:teuthology.orchestra.run.smithi118.stdout: "level": "advanced", 2024-03-24T03:17:24.207 INFO:teuthology.orchestra.run.smithi118.stdout: "flags": 0, 2024-03-24T03:17:24.207 INFO:teuthology.orchestra.run.smithi118.stdout: "default_value": "", 2024-03-24T03:17:24.207 INFO:teuthology.orchestra.run.smithi118.stdout: "min": "", 2024-03-24T03:17:24.207 INFO:teuthology.orchestra.run.smithi118.stdout: "max": "", 2024-03-24T03:17:24.207 INFO:teuthology.orchestra.run.smithi118.stdout: "enum_allowed": [], 2024-03-24T03:17:24.207 INFO:teuthology.orchestra.run.smithi118.stdout: "desc": "password of InfluxDB server user", 2024-03-24T03:17:24.207 INFO:teuthology.orchestra.run.smithi118.stdout: "long_desc": "", 2024-03-24T03:17:24.208 INFO:teuthology.orchestra.run.smithi118.stdout: "tags": [], 2024-03-24T03:17:24.208 INFO:teuthology.orchestra.run.smithi118.stdout: "see_also": [] 2024-03-24T03:17:24.208 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-24T03:17:24.208 INFO:teuthology.orchestra.run.smithi118.stdout: "port": { 2024-03-24T03:17:24.208 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "port", 2024-03-24T03:17:24.208 INFO:teuthology.orchestra.run.smithi118.stdout: "type": "int", 2024-03-24T03:17:24.208 INFO:teuthology.orchestra.run.smithi118.stdout: "level": "advanced", 2024-03-24T03:17:24.208 INFO:teuthology.orchestra.run.smithi118.stdout: "flags": 0, 2024-03-24T03:17:24.208 INFO:teuthology.orchestra.run.smithi118.stdout: "default_value": "8086", 2024-03-24T03:17:24.209 INFO:teuthology.orchestra.run.smithi118.stdout: "min": "", 2024-03-24T03:17:24.209 INFO:teuthology.orchestra.run.smithi118.stdout: "max": "", 2024-03-24T03:17:24.209 INFO:teuthology.orchestra.run.smithi118.stdout: "enum_allowed": [], 2024-03-24T03:17:24.209 INFO:teuthology.orchestra.run.smithi118.stdout: "desc": "InfluxDB server port", 2024-03-24T03:17:24.209 INFO:teuthology.orchestra.run.smithi118.stdout: "long_desc": "", 2024-03-24T03:17:24.209 INFO:teuthology.orchestra.run.smithi118.stdout: "tags": [], 2024-03-24T03:17:24.209 INFO:teuthology.orchestra.run.smithi118.stdout: "see_also": [] 2024-03-24T03:17:24.209 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-24T03:17:24.209 INFO:teuthology.orchestra.run.smithi118.stdout: "ssl": { 2024-03-24T03:17:24.209 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "ssl", 2024-03-24T03:17:24.210 INFO:teuthology.orchestra.run.smithi118.stdout: "type": "str", 2024-03-24T03:17:24.210 INFO:teuthology.orchestra.run.smithi118.stdout: "level": "advanced", 2024-03-24T03:17:24.210 INFO:teuthology.orchestra.run.smithi118.stdout: "flags": 0, 2024-03-24T03:17:24.210 INFO:teuthology.orchestra.run.smithi118.stdout: "default_value": "false", 2024-03-24T03:17:24.210 INFO:teuthology.orchestra.run.smithi118.stdout: "min": "", 2024-03-24T03:17:24.210 INFO:teuthology.orchestra.run.smithi118.stdout: "max": "", 2024-03-24T03:17:24.210 INFO:teuthology.orchestra.run.smithi118.stdout: "enum_allowed": [], 2024-03-24T03:17:24.210 INFO:teuthology.orchestra.run.smithi118.stdout: "desc": "Use https connection for InfluxDB server. Use \"true\" or \"false\".", 2024-03-24T03:17:24.210 INFO:teuthology.orchestra.run.smithi118.stdout: "long_desc": "", 2024-03-24T03:17:24.211 INFO:teuthology.orchestra.run.smithi118.stdout: "tags": [], 2024-03-24T03:17:24.211 INFO:teuthology.orchestra.run.smithi118.stdout: "see_also": [] 2024-03-24T03:17:24.211 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-24T03:17:24.211 INFO:teuthology.orchestra.run.smithi118.stdout: "threads": { 2024-03-24T03:17:24.211 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "threads", 2024-03-24T03:17:24.211 INFO:teuthology.orchestra.run.smithi118.stdout: "type": "int", 2024-03-24T03:17:24.211 INFO:teuthology.orchestra.run.smithi118.stdout: "level": "advanced", 2024-03-24T03:17:24.211 INFO:teuthology.orchestra.run.smithi118.stdout: "flags": 0, 2024-03-24T03:17:24.211 INFO:teuthology.orchestra.run.smithi118.stdout: "default_value": "5", 2024-03-24T03:17:24.212 INFO:teuthology.orchestra.run.smithi118.stdout: "min": "1", 2024-03-24T03:17:24.212 INFO:teuthology.orchestra.run.smithi118.stdout: "max": "32", 2024-03-24T03:17:24.212 INFO:teuthology.orchestra.run.smithi118.stdout: "enum_allowed": [], 2024-03-24T03:17:24.212 INFO:teuthology.orchestra.run.smithi118.stdout: "desc": "How many worker threads should be spawned for sending data to InfluxDB.", 2024-03-24T03:17:24.212 INFO:teuthology.orchestra.run.smithi118.stdout: "long_desc": "", 2024-03-24T03:17:24.212 INFO:teuthology.orchestra.run.smithi118.stdout: "tags": [], 2024-03-24T03:17:24.212 INFO:teuthology.orchestra.run.smithi118.stdout: "see_also": [] 2024-03-24T03:17:24.212 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-24T03:17:24.212 INFO:teuthology.orchestra.run.smithi118.stdout: "username": { 2024-03-24T03:17:24.212 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "username", 2024-03-24T03:17:24.213 INFO:teuthology.orchestra.run.smithi118.stdout: "type": "str", 2024-03-24T03:17:24.213 INFO:teuthology.orchestra.run.smithi118.stdout: "level": "advanced", 2024-03-24T03:17:24.213 INFO:teuthology.orchestra.run.smithi118.stdout: "flags": 0, 2024-03-24T03:17:24.213 INFO:teuthology.orchestra.run.smithi118.stdout: "default_value": "", 2024-03-24T03:17:24.213 INFO:teuthology.orchestra.run.smithi118.stdout: "min": "", 2024-03-24T03:17:24.213 INFO:teuthology.orchestra.run.smithi118.stdout: "max": "", 2024-03-24T03:17:24.213 INFO:teuthology.orchestra.run.smithi118.stdout: "enum_allowed": [], 2024-03-24T03:17:24.213 INFO:teuthology.orchestra.run.smithi118.stdout: "desc": "username of InfluxDB server user", 2024-03-24T03:17:24.213 INFO:teuthology.orchestra.run.smithi118.stdout: "long_desc": "", 2024-03-24T03:17:24.214 INFO:teuthology.orchestra.run.smithi118.stdout: "tags": [], 2024-03-24T03:17:24.214 INFO:teuthology.orchestra.run.smithi118.stdout: "see_also": [] 2024-03-24T03:17:24.214 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-24T03:17:24.214 INFO:teuthology.orchestra.run.smithi118.stdout: "verify_ssl": { 2024-03-24T03:17:24.214 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "verify_ssl", 2024-03-24T03:17:24.214 INFO:teuthology.orchestra.run.smithi118.stdout: "type": "str", 2024-03-24T03:17:24.214 INFO:teuthology.orchestra.run.smithi118.stdout: "level": "advanced", 2024-03-24T03:17:24.214 INFO:teuthology.orchestra.run.smithi118.stdout: "flags": 0, 2024-03-24T03:17:24.214 INFO:teuthology.orchestra.run.smithi118.stdout: "default_value": "true", 2024-03-24T03:17:24.215 INFO:teuthology.orchestra.run.smithi118.stdout: "min": "", 2024-03-24T03:17:24.215 INFO:teuthology.orchestra.run.smithi118.stdout: "max": "", 2024-03-24T03:17:24.215 INFO:teuthology.orchestra.run.smithi118.stdout: "enum_allowed": [], 2024-03-24T03:17:24.215 INFO:teuthology.orchestra.run.smithi118.stdout: "desc": "Verify https cert for InfluxDB server. Use \"true\" or \"false\".", 2024-03-24T03:17:24.215 INFO:teuthology.orchestra.run.smithi118.stdout: "long_desc": "", 2024-03-24T03:17:24.215 INFO:teuthology.orchestra.run.smithi118.stdout: "tags": [], 2024-03-24T03:17:24.215 INFO:teuthology.orchestra.run.smithi118.stdout: "see_also": [] 2024-03-24T03:17:24.215 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-03-24T03:17:24.215 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-03-24T03:17:24.216 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-24T03:17:24.216 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-24T03:17:24.216 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "insights", 2024-03-24T03:17:24.216 INFO:teuthology.orchestra.run.smithi118.stdout: "can_run": true, 2024-03-24T03:17:24.216 INFO:teuthology.orchestra.run.smithi118.stdout: "error_string": "", 2024-03-24T03:17:24.216 INFO:teuthology.orchestra.run.smithi118.stdout: "module_options": { 2024-03-24T03:17:24.216 INFO:teuthology.orchestra.run.smithi118.stdout: "log_level": { 2024-03-24T03:17:24.216 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "log_level", 2024-03-24T03:17:24.216 INFO:teuthology.orchestra.run.smithi118.stdout: "type": "str", 2024-03-24T03:17:24.217 INFO:teuthology.orchestra.run.smithi118.stdout: "level": "advanced", 2024-03-24T03:17:24.217 INFO:teuthology.orchestra.run.smithi118.stdout: "flags": 1, 2024-03-24T03:17:24.217 INFO:teuthology.orchestra.run.smithi118.stdout: "default_value": "", 2024-03-24T03:17:24.217 INFO:teuthology.orchestra.run.smithi118.stdout: "min": "", 2024-03-24T03:17:24.217 INFO:teuthology.orchestra.run.smithi118.stdout: "max": "", 2024-03-24T03:17:24.217 INFO:teuthology.orchestra.run.smithi118.stdout: "enum_allowed": [ 2024-03-24T03:17:24.217 INFO:teuthology.orchestra.run.smithi118.stdout: "", 2024-03-24T03:17:24.217 INFO:teuthology.orchestra.run.smithi118.stdout: "critical", 2024-03-24T03:17:24.217 INFO:teuthology.orchestra.run.smithi118.stdout: "debug", 2024-03-24T03:17:24.218 INFO:teuthology.orchestra.run.smithi118.stdout: "error", 2024-03-24T03:17:24.218 INFO:teuthology.orchestra.run.smithi118.stdout: "info", 2024-03-24T03:17:24.218 INFO:teuthology.orchestra.run.smithi118.stdout: "warning" 2024-03-24T03:17:24.218 INFO:teuthology.orchestra.run.smithi118.stdout: ], 2024-03-24T03:17:24.218 INFO:teuthology.orchestra.run.smithi118.stdout: "desc": "", 2024-03-24T03:17:24.218 INFO:teuthology.orchestra.run.smithi118.stdout: "long_desc": "", 2024-03-24T03:17:24.218 INFO:teuthology.orchestra.run.smithi118.stdout: "tags": [], 2024-03-24T03:17:24.218 INFO:teuthology.orchestra.run.smithi118.stdout: "see_also": [] 2024-03-24T03:17:24.218 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-24T03:17:24.219 INFO:teuthology.orchestra.run.smithi118.stdout: "log_to_cluster": { 2024-03-24T03:17:24.219 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "log_to_cluster", 2024-03-24T03:17:24.219 INFO:teuthology.orchestra.run.smithi118.stdout: "type": "bool", 2024-03-24T03:17:24.219 INFO:teuthology.orchestra.run.smithi118.stdout: "level": "advanced", 2024-03-24T03:17:24.219 INFO:teuthology.orchestra.run.smithi118.stdout: "flags": 1, 2024-03-24T03:17:24.219 INFO:teuthology.orchestra.run.smithi118.stdout: "default_value": "False", 2024-03-24T03:17:24.219 INFO:teuthology.orchestra.run.smithi118.stdout: "min": "", 2024-03-24T03:17:24.219 INFO:teuthology.orchestra.run.smithi118.stdout: "max": "", 2024-03-24T03:17:24.219 INFO:teuthology.orchestra.run.smithi118.stdout: "enum_allowed": [], 2024-03-24T03:17:24.219 INFO:teuthology.orchestra.run.smithi118.stdout: "desc": "", 2024-03-24T03:17:24.220 INFO:teuthology.orchestra.run.smithi118.stdout: "long_desc": "", 2024-03-24T03:17:24.220 INFO:teuthology.orchestra.run.smithi118.stdout: "tags": [], 2024-03-24T03:17:24.220 INFO:teuthology.orchestra.run.smithi118.stdout: "see_also": [] 2024-03-24T03:17:24.220 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-24T03:17:24.220 INFO:teuthology.orchestra.run.smithi118.stdout: "log_to_cluster_level": { 2024-03-24T03:17:24.220 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "log_to_cluster_level", 2024-03-24T03:17:24.220 INFO:teuthology.orchestra.run.smithi118.stdout: "type": "str", 2024-03-24T03:17:24.220 INFO:teuthology.orchestra.run.smithi118.stdout: "level": "advanced", 2024-03-24T03:17:24.220 INFO:teuthology.orchestra.run.smithi118.stdout: "flags": 1, 2024-03-24T03:17:24.221 INFO:teuthology.orchestra.run.smithi118.stdout: "default_value": "info", 2024-03-24T03:17:24.221 INFO:teuthology.orchestra.run.smithi118.stdout: "min": "", 2024-03-24T03:17:24.221 INFO:teuthology.orchestra.run.smithi118.stdout: "max": "", 2024-03-24T03:17:24.221 INFO:teuthology.orchestra.run.smithi118.stdout: "enum_allowed": [ 2024-03-24T03:17:24.221 INFO:teuthology.orchestra.run.smithi118.stdout: "", 2024-03-24T03:17:24.221 INFO:teuthology.orchestra.run.smithi118.stdout: "critical", 2024-03-24T03:17:24.221 INFO:teuthology.orchestra.run.smithi118.stdout: "debug", 2024-03-24T03:17:24.221 INFO:teuthology.orchestra.run.smithi118.stdout: "error", 2024-03-24T03:17:24.221 INFO:teuthology.orchestra.run.smithi118.stdout: "info", 2024-03-24T03:17:24.221 INFO:teuthology.orchestra.run.smithi118.stdout: "warning" 2024-03-24T03:17:24.222 INFO:teuthology.orchestra.run.smithi118.stdout: ], 2024-03-24T03:17:24.222 INFO:teuthology.orchestra.run.smithi118.stdout: "desc": "", 2024-03-24T03:17:24.222 INFO:teuthology.orchestra.run.smithi118.stdout: "long_desc": "", 2024-03-24T03:17:24.222 INFO:teuthology.orchestra.run.smithi118.stdout: "tags": [], 2024-03-24T03:17:24.222 INFO:teuthology.orchestra.run.smithi118.stdout: "see_also": [] 2024-03-24T03:17:24.222 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-24T03:17:24.222 INFO:teuthology.orchestra.run.smithi118.stdout: "log_to_file": { 2024-03-24T03:17:24.222 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "log_to_file", 2024-03-24T03:17:24.223 INFO:teuthology.orchestra.run.smithi118.stdout: "type": "bool", 2024-03-24T03:17:24.223 INFO:teuthology.orchestra.run.smithi118.stdout: "level": "advanced", 2024-03-24T03:17:24.223 INFO:teuthology.orchestra.run.smithi118.stdout: "flags": 1, 2024-03-24T03:17:24.223 INFO:teuthology.orchestra.run.smithi118.stdout: "default_value": "False", 2024-03-24T03:17:24.223 INFO:teuthology.orchestra.run.smithi118.stdout: "min": "", 2024-03-24T03:17:24.223 INFO:teuthology.orchestra.run.smithi118.stdout: "max": "", 2024-03-24T03:17:24.223 INFO:teuthology.orchestra.run.smithi118.stdout: "enum_allowed": [], 2024-03-24T03:17:24.223 INFO:teuthology.orchestra.run.smithi118.stdout: "desc": "", 2024-03-24T03:17:24.224 INFO:teuthology.orchestra.run.smithi118.stdout: "long_desc": "", 2024-03-24T03:17:24.224 INFO:teuthology.orchestra.run.smithi118.stdout: "tags": [], 2024-03-24T03:17:24.224 INFO:teuthology.orchestra.run.smithi118.stdout: "see_also": [] 2024-03-24T03:17:24.224 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-03-24T03:17:24.224 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-03-24T03:17:24.224 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-24T03:17:24.224 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-24T03:17:24.224 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "iostat", 2024-03-24T03:17:24.224 INFO:teuthology.orchestra.run.smithi118.stdout: "can_run": true, 2024-03-24T03:17:24.225 INFO:teuthology.orchestra.run.smithi118.stdout: "error_string": "", 2024-03-24T03:17:24.225 INFO:teuthology.orchestra.run.smithi118.stdout: "module_options": { 2024-03-24T03:17:24.225 INFO:teuthology.orchestra.run.smithi118.stdout: "log_level": { 2024-03-24T03:17:24.225 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "log_level", 2024-03-24T03:17:24.225 INFO:teuthology.orchestra.run.smithi118.stdout: "type": "str", 2024-03-24T03:17:24.225 INFO:teuthology.orchestra.run.smithi118.stdout: "level": "advanced", 2024-03-24T03:17:24.225 INFO:teuthology.orchestra.run.smithi118.stdout: "flags": 1, 2024-03-24T03:17:24.225 INFO:teuthology.orchestra.run.smithi118.stdout: "default_value": "", 2024-03-24T03:17:24.225 INFO:teuthology.orchestra.run.smithi118.stdout: "min": "", 2024-03-24T03:17:24.225 INFO:teuthology.orchestra.run.smithi118.stdout: "max": "", 2024-03-24T03:17:24.226 INFO:teuthology.orchestra.run.smithi118.stdout: "enum_allowed": [ 2024-03-24T03:17:24.226 INFO:teuthology.orchestra.run.smithi118.stdout: "", 2024-03-24T03:17:24.226 INFO:teuthology.orchestra.run.smithi118.stdout: "critical", 2024-03-24T03:17:24.226 INFO:teuthology.orchestra.run.smithi118.stdout: "debug", 2024-03-24T03:17:24.226 INFO:teuthology.orchestra.run.smithi118.stdout: "error", 2024-03-24T03:17:24.226 INFO:teuthology.orchestra.run.smithi118.stdout: "info", 2024-03-24T03:17:24.226 INFO:teuthology.orchestra.run.smithi118.stdout: "warning" 2024-03-24T03:17:24.226 INFO:teuthology.orchestra.run.smithi118.stdout: ], 2024-03-24T03:17:24.226 INFO:teuthology.orchestra.run.smithi118.stdout: "desc": "", 2024-03-24T03:17:24.227 INFO:teuthology.orchestra.run.smithi118.stdout: "long_desc": "", 2024-03-24T03:17:24.227 INFO:teuthology.orchestra.run.smithi118.stdout: "tags": [], 2024-03-24T03:17:24.227 INFO:teuthology.orchestra.run.smithi118.stdout: "see_also": [] 2024-03-24T03:17:24.227 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-24T03:17:24.227 INFO:teuthology.orchestra.run.smithi118.stdout: "log_to_cluster": { 2024-03-24T03:17:24.227 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "log_to_cluster", 2024-03-24T03:17:24.227 INFO:teuthology.orchestra.run.smithi118.stdout: "type": "bool", 2024-03-24T03:17:24.227 INFO:teuthology.orchestra.run.smithi118.stdout: "level": "advanced", 2024-03-24T03:17:24.227 INFO:teuthology.orchestra.run.smithi118.stdout: "flags": 1, 2024-03-24T03:17:24.227 INFO:teuthology.orchestra.run.smithi118.stdout: "default_value": "False", 2024-03-24T03:17:24.228 INFO:teuthology.orchestra.run.smithi118.stdout: "min": "", 2024-03-24T03:17:24.228 INFO:teuthology.orchestra.run.smithi118.stdout: "max": "", 2024-03-24T03:17:24.228 INFO:teuthology.orchestra.run.smithi118.stdout: "enum_allowed": [], 2024-03-24T03:17:24.228 INFO:teuthology.orchestra.run.smithi118.stdout: "desc": "", 2024-03-24T03:17:24.228 INFO:teuthology.orchestra.run.smithi118.stdout: "long_desc": "", 2024-03-24T03:17:24.228 INFO:teuthology.orchestra.run.smithi118.stdout: "tags": [], 2024-03-24T03:17:24.228 INFO:teuthology.orchestra.run.smithi118.stdout: "see_also": [] 2024-03-24T03:17:24.228 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-24T03:17:24.228 INFO:teuthology.orchestra.run.smithi118.stdout: "log_to_cluster_level": { 2024-03-24T03:17:24.228 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "log_to_cluster_level", 2024-03-24T03:17:24.229 INFO:teuthology.orchestra.run.smithi118.stdout: "type": "str", 2024-03-24T03:17:24.229 INFO:teuthology.orchestra.run.smithi118.stdout: "level": "advanced", 2024-03-24T03:17:24.229 INFO:teuthology.orchestra.run.smithi118.stdout: "flags": 1, 2024-03-24T03:17:24.229 INFO:teuthology.orchestra.run.smithi118.stdout: "default_value": "info", 2024-03-24T03:17:24.229 INFO:teuthology.orchestra.run.smithi118.stdout: "min": "", 2024-03-24T03:17:24.229 INFO:teuthology.orchestra.run.smithi118.stdout: "max": "", 2024-03-24T03:17:24.229 INFO:teuthology.orchestra.run.smithi118.stdout: "enum_allowed": [ 2024-03-24T03:17:24.229 INFO:teuthology.orchestra.run.smithi118.stdout: "", 2024-03-24T03:17:24.229 INFO:teuthology.orchestra.run.smithi118.stdout: "critical", 2024-03-24T03:17:24.230 INFO:teuthology.orchestra.run.smithi118.stdout: "debug", 2024-03-24T03:17:24.230 INFO:teuthology.orchestra.run.smithi118.stdout: "error", 2024-03-24T03:17:24.230 INFO:teuthology.orchestra.run.smithi118.stdout: "info", 2024-03-24T03:17:24.230 INFO:teuthology.orchestra.run.smithi118.stdout: "warning" 2024-03-24T03:17:24.230 INFO:teuthology.orchestra.run.smithi118.stdout: ], 2024-03-24T03:17:24.230 INFO:teuthology.orchestra.run.smithi118.stdout: "desc": "", 2024-03-24T03:17:24.230 INFO:teuthology.orchestra.run.smithi118.stdout: "long_desc": "", 2024-03-24T03:17:24.230 INFO:teuthology.orchestra.run.smithi118.stdout: "tags": [], 2024-03-24T03:17:24.230 INFO:teuthology.orchestra.run.smithi118.stdout: "see_also": [] 2024-03-24T03:17:24.230 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-24T03:17:24.231 INFO:teuthology.orchestra.run.smithi118.stdout: "log_to_file": { 2024-03-24T03:17:24.231 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "log_to_file", 2024-03-24T03:17:24.231 INFO:teuthology.orchestra.run.smithi118.stdout: "type": "bool", 2024-03-24T03:17:24.231 INFO:teuthology.orchestra.run.smithi118.stdout: "level": "advanced", 2024-03-24T03:17:24.231 INFO:teuthology.orchestra.run.smithi118.stdout: "flags": 1, 2024-03-24T03:17:24.231 INFO:teuthology.orchestra.run.smithi118.stdout: "default_value": "False", 2024-03-24T03:17:24.231 INFO:teuthology.orchestra.run.smithi118.stdout: "min": "", 2024-03-24T03:17:24.231 INFO:teuthology.orchestra.run.smithi118.stdout: "max": "", 2024-03-24T03:17:24.231 INFO:teuthology.orchestra.run.smithi118.stdout: "enum_allowed": [], 2024-03-24T03:17:24.232 INFO:teuthology.orchestra.run.smithi118.stdout: "desc": "", 2024-03-24T03:17:24.232 INFO:teuthology.orchestra.run.smithi118.stdout: "long_desc": "", 2024-03-24T03:17:24.232 INFO:teuthology.orchestra.run.smithi118.stdout: "tags": [], 2024-03-24T03:17:24.232 INFO:teuthology.orchestra.run.smithi118.stdout: "see_also": [] 2024-03-24T03:17:24.232 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-03-24T03:17:24.232 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-03-24T03:17:24.232 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-24T03:17:24.232 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-24T03:17:24.232 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "k8sevents", 2024-03-24T03:17:24.233 INFO:teuthology.orchestra.run.smithi118.stdout: "can_run": true, 2024-03-24T03:17:24.233 INFO:teuthology.orchestra.run.smithi118.stdout: "error_string": "", 2024-03-24T03:17:24.233 INFO:teuthology.orchestra.run.smithi118.stdout: "module_options": { 2024-03-24T03:17:24.233 INFO:teuthology.orchestra.run.smithi118.stdout: "ceph_event_retention_days": { 2024-03-24T03:17:24.233 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "ceph_event_retention_days", 2024-03-24T03:17:24.233 INFO:teuthology.orchestra.run.smithi118.stdout: "type": "int", 2024-03-24T03:17:24.233 INFO:teuthology.orchestra.run.smithi118.stdout: "level": "advanced", 2024-03-24T03:17:24.233 INFO:teuthology.orchestra.run.smithi118.stdout: "flags": 0, 2024-03-24T03:17:24.233 INFO:teuthology.orchestra.run.smithi118.stdout: "default_value": "7", 2024-03-24T03:17:24.233 INFO:teuthology.orchestra.run.smithi118.stdout: "min": "", 2024-03-24T03:17:24.234 INFO:teuthology.orchestra.run.smithi118.stdout: "max": "", 2024-03-24T03:17:24.234 INFO:teuthology.orchestra.run.smithi118.stdout: "enum_allowed": [], 2024-03-24T03:17:24.234 INFO:teuthology.orchestra.run.smithi118.stdout: "desc": "Days to hold ceph event information within local cache", 2024-03-24T03:17:24.234 INFO:teuthology.orchestra.run.smithi118.stdout: "long_desc": "", 2024-03-24T03:17:24.234 INFO:teuthology.orchestra.run.smithi118.stdout: "tags": [], 2024-03-24T03:17:24.234 INFO:teuthology.orchestra.run.smithi118.stdout: "see_also": [] 2024-03-24T03:17:24.234 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-24T03:17:24.234 INFO:teuthology.orchestra.run.smithi118.stdout: "config_check_secs": { 2024-03-24T03:17:24.235 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "config_check_secs", 2024-03-24T03:17:24.235 INFO:teuthology.orchestra.run.smithi118.stdout: "type": "int", 2024-03-24T03:17:24.235 INFO:teuthology.orchestra.run.smithi118.stdout: "level": "advanced", 2024-03-24T03:17:24.235 INFO:teuthology.orchestra.run.smithi118.stdout: "flags": 0, 2024-03-24T03:17:24.235 INFO:teuthology.orchestra.run.smithi118.stdout: "default_value": "10", 2024-03-24T03:17:24.235 INFO:teuthology.orchestra.run.smithi118.stdout: "min": "10", 2024-03-24T03:17:24.235 INFO:teuthology.orchestra.run.smithi118.stdout: "max": "", 2024-03-24T03:17:24.235 INFO:teuthology.orchestra.run.smithi118.stdout: "enum_allowed": [], 2024-03-24T03:17:24.235 INFO:teuthology.orchestra.run.smithi118.stdout: "desc": "interval (secs) to check for cluster configuration changes", 2024-03-24T03:17:24.235 INFO:teuthology.orchestra.run.smithi118.stdout: "long_desc": "", 2024-03-24T03:17:24.236 INFO:teuthology.orchestra.run.smithi118.stdout: "tags": [], 2024-03-24T03:17:24.236 INFO:teuthology.orchestra.run.smithi118.stdout: "see_also": [] 2024-03-24T03:17:24.236 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-24T03:17:24.236 INFO:teuthology.orchestra.run.smithi118.stdout: "log_level": { 2024-03-24T03:17:24.236 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "log_level", 2024-03-24T03:17:24.236 INFO:teuthology.orchestra.run.smithi118.stdout: "type": "str", 2024-03-24T03:17:24.236 INFO:teuthology.orchestra.run.smithi118.stdout: "level": "advanced", 2024-03-24T03:17:24.236 INFO:teuthology.orchestra.run.smithi118.stdout: "flags": 1, 2024-03-24T03:17:24.236 INFO:teuthology.orchestra.run.smithi118.stdout: "default_value": "", 2024-03-24T03:17:24.237 INFO:teuthology.orchestra.run.smithi118.stdout: "min": "", 2024-03-24T03:17:24.237 INFO:teuthology.orchestra.run.smithi118.stdout: "max": "", 2024-03-24T03:17:24.237 INFO:teuthology.orchestra.run.smithi118.stdout: "enum_allowed": [ 2024-03-24T03:17:24.237 INFO:teuthology.orchestra.run.smithi118.stdout: "", 2024-03-24T03:17:24.237 INFO:teuthology.orchestra.run.smithi118.stdout: "critical", 2024-03-24T03:17:24.237 INFO:teuthology.orchestra.run.smithi118.stdout: "debug", 2024-03-24T03:17:24.237 INFO:teuthology.orchestra.run.smithi118.stdout: "error", 2024-03-24T03:17:24.237 INFO:teuthology.orchestra.run.smithi118.stdout: "info", 2024-03-24T03:17:24.237 INFO:teuthology.orchestra.run.smithi118.stdout: "warning" 2024-03-24T03:17:24.237 INFO:teuthology.orchestra.run.smithi118.stdout: ], 2024-03-24T03:17:24.238 INFO:teuthology.orchestra.run.smithi118.stdout: "desc": "", 2024-03-24T03:17:24.238 INFO:teuthology.orchestra.run.smithi118.stdout: "long_desc": "", 2024-03-24T03:17:24.238 INFO:teuthology.orchestra.run.smithi118.stdout: "tags": [], 2024-03-24T03:17:24.238 INFO:teuthology.orchestra.run.smithi118.stdout: "see_also": [] 2024-03-24T03:17:24.238 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-24T03:17:24.238 INFO:teuthology.orchestra.run.smithi118.stdout: "log_to_cluster": { 2024-03-24T03:17:24.238 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "log_to_cluster", 2024-03-24T03:17:24.238 INFO:teuthology.orchestra.run.smithi118.stdout: "type": "bool", 2024-03-24T03:17:24.238 INFO:teuthology.orchestra.run.smithi118.stdout: "level": "advanced", 2024-03-24T03:17:24.238 INFO:teuthology.orchestra.run.smithi118.stdout: "flags": 1, 2024-03-24T03:17:24.239 INFO:teuthology.orchestra.run.smithi118.stdout: "default_value": "False", 2024-03-24T03:17:24.239 INFO:teuthology.orchestra.run.smithi118.stdout: "min": "", 2024-03-24T03:17:24.239 INFO:teuthology.orchestra.run.smithi118.stdout: "max": "", 2024-03-24T03:17:24.239 INFO:teuthology.orchestra.run.smithi118.stdout: "enum_allowed": [], 2024-03-24T03:17:24.239 INFO:teuthology.orchestra.run.smithi118.stdout: "desc": "", 2024-03-24T03:17:24.239 INFO:teuthology.orchestra.run.smithi118.stdout: "long_desc": "", 2024-03-24T03:17:24.239 INFO:teuthology.orchestra.run.smithi118.stdout: "tags": [], 2024-03-24T03:17:24.239 INFO:teuthology.orchestra.run.smithi118.stdout: "see_also": [] 2024-03-24T03:17:24.239 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-24T03:17:24.240 INFO:teuthology.orchestra.run.smithi118.stdout: "log_to_cluster_level": { 2024-03-24T03:17:24.240 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "log_to_cluster_level", 2024-03-24T03:17:24.240 INFO:teuthology.orchestra.run.smithi118.stdout: "type": "str", 2024-03-24T03:17:24.240 INFO:teuthology.orchestra.run.smithi118.stdout: "level": "advanced", 2024-03-24T03:17:24.240 INFO:teuthology.orchestra.run.smithi118.stdout: "flags": 1, 2024-03-24T03:17:24.240 INFO:teuthology.orchestra.run.smithi118.stdout: "default_value": "info", 2024-03-24T03:17:24.240 INFO:teuthology.orchestra.run.smithi118.stdout: "min": "", 2024-03-24T03:17:24.240 INFO:teuthology.orchestra.run.smithi118.stdout: "max": "", 2024-03-24T03:17:24.240 INFO:teuthology.orchestra.run.smithi118.stdout: "enum_allowed": [ 2024-03-24T03:17:24.240 INFO:teuthology.orchestra.run.smithi118.stdout: "", 2024-03-24T03:17:24.241 INFO:teuthology.orchestra.run.smithi118.stdout: "critical", 2024-03-24T03:17:24.241 INFO:teuthology.orchestra.run.smithi118.stdout: "debug", 2024-03-24T03:17:24.241 INFO:teuthology.orchestra.run.smithi118.stdout: "error", 2024-03-24T03:17:24.241 INFO:teuthology.orchestra.run.smithi118.stdout: "info", 2024-03-24T03:17:24.241 INFO:teuthology.orchestra.run.smithi118.stdout: "warning" 2024-03-24T03:17:24.241 INFO:teuthology.orchestra.run.smithi118.stdout: ], 2024-03-24T03:17:24.241 INFO:teuthology.orchestra.run.smithi118.stdout: "desc": "", 2024-03-24T03:17:24.241 INFO:teuthology.orchestra.run.smithi118.stdout: "long_desc": "", 2024-03-24T03:17:24.241 INFO:teuthology.orchestra.run.smithi118.stdout: "tags": [], 2024-03-24T03:17:24.242 INFO:teuthology.orchestra.run.smithi118.stdout: "see_also": [] 2024-03-24T03:17:24.242 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-24T03:17:24.242 INFO:teuthology.orchestra.run.smithi118.stdout: "log_to_file": { 2024-03-24T03:17:24.242 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "log_to_file", 2024-03-24T03:17:24.242 INFO:teuthology.orchestra.run.smithi118.stdout: "type": "bool", 2024-03-24T03:17:24.242 INFO:teuthology.orchestra.run.smithi118.stdout: "level": "advanced", 2024-03-24T03:17:24.242 INFO:teuthology.orchestra.run.smithi118.stdout: "flags": 1, 2024-03-24T03:17:24.242 INFO:teuthology.orchestra.run.smithi118.stdout: "default_value": "False", 2024-03-24T03:17:24.242 INFO:teuthology.orchestra.run.smithi118.stdout: "min": "", 2024-03-24T03:17:24.243 INFO:teuthology.orchestra.run.smithi118.stdout: "max": "", 2024-03-24T03:17:24.243 INFO:teuthology.orchestra.run.smithi118.stdout: "enum_allowed": [], 2024-03-24T03:17:24.243 INFO:teuthology.orchestra.run.smithi118.stdout: "desc": "", 2024-03-24T03:17:24.243 INFO:teuthology.orchestra.run.smithi118.stdout: "long_desc": "", 2024-03-24T03:17:24.243 INFO:teuthology.orchestra.run.smithi118.stdout: "tags": [], 2024-03-24T03:17:24.243 INFO:teuthology.orchestra.run.smithi118.stdout: "see_also": [] 2024-03-24T03:17:24.243 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-03-24T03:17:24.243 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-03-24T03:17:24.243 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-24T03:17:24.243 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-24T03:17:24.244 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "localpool", 2024-03-24T03:17:24.244 INFO:teuthology.orchestra.run.smithi118.stdout: "can_run": true, 2024-03-24T03:17:24.244 INFO:teuthology.orchestra.run.smithi118.stdout: "error_string": "", 2024-03-24T03:17:24.244 INFO:teuthology.orchestra.run.smithi118.stdout: "module_options": { 2024-03-24T03:17:24.244 INFO:teuthology.orchestra.run.smithi118.stdout: "failure_domain": { 2024-03-24T03:17:24.244 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "failure_domain", 2024-03-24T03:17:24.244 INFO:teuthology.orchestra.run.smithi118.stdout: "type": "str", 2024-03-24T03:17:24.244 INFO:teuthology.orchestra.run.smithi118.stdout: "level": "advanced", 2024-03-24T03:17:24.244 INFO:teuthology.orchestra.run.smithi118.stdout: "flags": 1, 2024-03-24T03:17:24.245 INFO:teuthology.orchestra.run.smithi118.stdout: "default_value": "host", 2024-03-24T03:17:24.245 INFO:teuthology.orchestra.run.smithi118.stdout: "min": "", 2024-03-24T03:17:24.245 INFO:teuthology.orchestra.run.smithi118.stdout: "max": "", 2024-03-24T03:17:24.245 INFO:teuthology.orchestra.run.smithi118.stdout: "enum_allowed": [], 2024-03-24T03:17:24.245 INFO:teuthology.orchestra.run.smithi118.stdout: "desc": "failure domain for any created local pool", 2024-03-24T03:17:24.245 INFO:teuthology.orchestra.run.smithi118.stdout: "long_desc": "what failure domain we should separate data replicas across.", 2024-03-24T03:17:24.245 INFO:teuthology.orchestra.run.smithi118.stdout: "tags": [], 2024-03-24T03:17:24.245 INFO:teuthology.orchestra.run.smithi118.stdout: "see_also": [] 2024-03-24T03:17:24.245 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-24T03:17:24.246 INFO:teuthology.orchestra.run.smithi118.stdout: "log_level": { 2024-03-24T03:17:24.246 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "log_level", 2024-03-24T03:17:24.246 INFO:teuthology.orchestra.run.smithi118.stdout: "type": "str", 2024-03-24T03:17:24.246 INFO:teuthology.orchestra.run.smithi118.stdout: "level": "advanced", 2024-03-24T03:17:24.246 INFO:teuthology.orchestra.run.smithi118.stdout: "flags": 1, 2024-03-24T03:17:24.246 INFO:teuthology.orchestra.run.smithi118.stdout: "default_value": "", 2024-03-24T03:17:24.246 INFO:teuthology.orchestra.run.smithi118.stdout: "min": "", 2024-03-24T03:17:24.246 INFO:teuthology.orchestra.run.smithi118.stdout: "max": "", 2024-03-24T03:17:24.246 INFO:teuthology.orchestra.run.smithi118.stdout: "enum_allowed": [ 2024-03-24T03:17:24.246 INFO:teuthology.orchestra.run.smithi118.stdout: "", 2024-03-24T03:17:24.247 INFO:teuthology.orchestra.run.smithi118.stdout: "critical", 2024-03-24T03:17:24.247 INFO:teuthology.orchestra.run.smithi118.stdout: "debug", 2024-03-24T03:17:24.247 INFO:teuthology.orchestra.run.smithi118.stdout: "error", 2024-03-24T03:17:24.247 INFO:teuthology.orchestra.run.smithi118.stdout: "info", 2024-03-24T03:17:24.247 INFO:teuthology.orchestra.run.smithi118.stdout: "warning" 2024-03-24T03:17:24.247 INFO:teuthology.orchestra.run.smithi118.stdout: ], 2024-03-24T03:17:24.247 INFO:teuthology.orchestra.run.smithi118.stdout: "desc": "", 2024-03-24T03:17:24.247 INFO:teuthology.orchestra.run.smithi118.stdout: "long_desc": "", 2024-03-24T03:17:24.247 INFO:teuthology.orchestra.run.smithi118.stdout: "tags": [], 2024-03-24T03:17:24.248 INFO:teuthology.orchestra.run.smithi118.stdout: "see_also": [] 2024-03-24T03:17:24.248 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-24T03:17:24.248 INFO:teuthology.orchestra.run.smithi118.stdout: "log_to_cluster": { 2024-03-24T03:17:24.248 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "log_to_cluster", 2024-03-24T03:17:24.248 INFO:teuthology.orchestra.run.smithi118.stdout: "type": "bool", 2024-03-24T03:17:24.248 INFO:teuthology.orchestra.run.smithi118.stdout: "level": "advanced", 2024-03-24T03:17:24.248 INFO:teuthology.orchestra.run.smithi118.stdout: "flags": 1, 2024-03-24T03:17:24.248 INFO:teuthology.orchestra.run.smithi118.stdout: "default_value": "False", 2024-03-24T03:17:24.248 INFO:teuthology.orchestra.run.smithi118.stdout: "min": "", 2024-03-24T03:17:24.248 INFO:teuthology.orchestra.run.smithi118.stdout: "max": "", 2024-03-24T03:17:24.249 INFO:teuthology.orchestra.run.smithi118.stdout: "enum_allowed": [], 2024-03-24T03:17:24.249 INFO:teuthology.orchestra.run.smithi118.stdout: "desc": "", 2024-03-24T03:17:24.249 INFO:teuthology.orchestra.run.smithi118.stdout: "long_desc": "", 2024-03-24T03:17:24.249 INFO:teuthology.orchestra.run.smithi118.stdout: "tags": [], 2024-03-24T03:17:24.249 INFO:teuthology.orchestra.run.smithi118.stdout: "see_also": [] 2024-03-24T03:17:24.249 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-24T03:17:24.249 INFO:teuthology.orchestra.run.smithi118.stdout: "log_to_cluster_level": { 2024-03-24T03:17:24.249 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "log_to_cluster_level", 2024-03-24T03:17:24.249 INFO:teuthology.orchestra.run.smithi118.stdout: "type": "str", 2024-03-24T03:17:24.250 INFO:teuthology.orchestra.run.smithi118.stdout: "level": "advanced", 2024-03-24T03:17:24.250 INFO:teuthology.orchestra.run.smithi118.stdout: "flags": 1, 2024-03-24T03:17:24.250 INFO:teuthology.orchestra.run.smithi118.stdout: "default_value": "info", 2024-03-24T03:17:24.250 INFO:teuthology.orchestra.run.smithi118.stdout: "min": "", 2024-03-24T03:17:24.250 INFO:teuthology.orchestra.run.smithi118.stdout: "max": "", 2024-03-24T03:17:24.250 INFO:teuthology.orchestra.run.smithi118.stdout: "enum_allowed": [ 2024-03-24T03:17:24.250 INFO:teuthology.orchestra.run.smithi118.stdout: "", 2024-03-24T03:17:24.250 INFO:teuthology.orchestra.run.smithi118.stdout: "critical", 2024-03-24T03:17:24.250 INFO:teuthology.orchestra.run.smithi118.stdout: "debug", 2024-03-24T03:17:24.251 INFO:teuthology.orchestra.run.smithi118.stdout: "error", 2024-03-24T03:17:24.251 INFO:teuthology.orchestra.run.smithi118.stdout: "info", 2024-03-24T03:17:24.251 INFO:teuthology.orchestra.run.smithi118.stdout: "warning" 2024-03-24T03:17:24.251 INFO:teuthology.orchestra.run.smithi118.stdout: ], 2024-03-24T03:17:24.251 INFO:teuthology.orchestra.run.smithi118.stdout: "desc": "", 2024-03-24T03:17:24.251 INFO:teuthology.orchestra.run.smithi118.stdout: "long_desc": "", 2024-03-24T03:17:24.251 INFO:teuthology.orchestra.run.smithi118.stdout: "tags": [], 2024-03-24T03:17:24.251 INFO:teuthology.orchestra.run.smithi118.stdout: "see_also": [] 2024-03-24T03:17:24.251 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-24T03:17:24.251 INFO:teuthology.orchestra.run.smithi118.stdout: "log_to_file": { 2024-03-24T03:17:24.252 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "log_to_file", 2024-03-24T03:17:24.252 INFO:teuthology.orchestra.run.smithi118.stdout: "type": "bool", 2024-03-24T03:17:24.252 INFO:teuthology.orchestra.run.smithi118.stdout: "level": "advanced", 2024-03-24T03:17:24.252 INFO:teuthology.orchestra.run.smithi118.stdout: "flags": 1, 2024-03-24T03:17:24.252 INFO:teuthology.orchestra.run.smithi118.stdout: "default_value": "False", 2024-03-24T03:17:24.252 INFO:teuthology.orchestra.run.smithi118.stdout: "min": "", 2024-03-24T03:17:24.252 INFO:teuthology.orchestra.run.smithi118.stdout: "max": "", 2024-03-24T03:17:24.252 INFO:teuthology.orchestra.run.smithi118.stdout: "enum_allowed": [], 2024-03-24T03:17:24.252 INFO:teuthology.orchestra.run.smithi118.stdout: "desc": "", 2024-03-24T03:17:24.253 INFO:teuthology.orchestra.run.smithi118.stdout: "long_desc": "", 2024-03-24T03:17:24.253 INFO:teuthology.orchestra.run.smithi118.stdout: "tags": [], 2024-03-24T03:17:24.253 INFO:teuthology.orchestra.run.smithi118.stdout: "see_also": [] 2024-03-24T03:17:24.253 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-24T03:17:24.253 INFO:teuthology.orchestra.run.smithi118.stdout: "min_size": { 2024-03-24T03:17:24.253 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "min_size", 2024-03-24T03:17:24.253 INFO:teuthology.orchestra.run.smithi118.stdout: "type": "int", 2024-03-24T03:17:24.253 INFO:teuthology.orchestra.run.smithi118.stdout: "level": "advanced", 2024-03-24T03:17:24.253 INFO:teuthology.orchestra.run.smithi118.stdout: "flags": 1, 2024-03-24T03:17:24.253 INFO:teuthology.orchestra.run.smithi118.stdout: "default_value": "", 2024-03-24T03:17:24.254 INFO:teuthology.orchestra.run.smithi118.stdout: "min": "", 2024-03-24T03:17:24.254 INFO:teuthology.orchestra.run.smithi118.stdout: "max": "", 2024-03-24T03:17:24.254 INFO:teuthology.orchestra.run.smithi118.stdout: "enum_allowed": [], 2024-03-24T03:17:24.254 INFO:teuthology.orchestra.run.smithi118.stdout: "desc": "default min_size for any created local pool", 2024-03-24T03:17:24.254 INFO:teuthology.orchestra.run.smithi118.stdout: "long_desc": "value to set min_size to (unchanged from Ceph's default if this option is not set)", 2024-03-24T03:17:24.254 INFO:teuthology.orchestra.run.smithi118.stdout: "tags": [], 2024-03-24T03:17:24.254 INFO:teuthology.orchestra.run.smithi118.stdout: "see_also": [] 2024-03-24T03:17:24.254 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-24T03:17:24.255 INFO:teuthology.orchestra.run.smithi118.stdout: "num_rep": { 2024-03-24T03:17:24.255 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "num_rep", 2024-03-24T03:17:24.255 INFO:teuthology.orchestra.run.smithi118.stdout: "type": "int", 2024-03-24T03:17:24.255 INFO:teuthology.orchestra.run.smithi118.stdout: "level": "advanced", 2024-03-24T03:17:24.255 INFO:teuthology.orchestra.run.smithi118.stdout: "flags": 1, 2024-03-24T03:17:24.255 INFO:teuthology.orchestra.run.smithi118.stdout: "default_value": "3", 2024-03-24T03:17:24.255 INFO:teuthology.orchestra.run.smithi118.stdout: "min": "", 2024-03-24T03:17:24.255 INFO:teuthology.orchestra.run.smithi118.stdout: "max": "", 2024-03-24T03:17:24.255 INFO:teuthology.orchestra.run.smithi118.stdout: "enum_allowed": [], 2024-03-24T03:17:24.255 INFO:teuthology.orchestra.run.smithi118.stdout: "desc": "default replica count for any created local pool", 2024-03-24T03:17:24.256 INFO:teuthology.orchestra.run.smithi118.stdout: "long_desc": "", 2024-03-24T03:17:24.256 INFO:teuthology.orchestra.run.smithi118.stdout: "tags": [], 2024-03-24T03:17:24.256 INFO:teuthology.orchestra.run.smithi118.stdout: "see_also": [] 2024-03-24T03:17:24.256 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-24T03:17:24.256 INFO:teuthology.orchestra.run.smithi118.stdout: "pg_num": { 2024-03-24T03:17:24.256 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "pg_num", 2024-03-24T03:17:24.256 INFO:teuthology.orchestra.run.smithi118.stdout: "type": "int", 2024-03-24T03:17:24.256 INFO:teuthology.orchestra.run.smithi118.stdout: "level": "advanced", 2024-03-24T03:17:24.256 INFO:teuthology.orchestra.run.smithi118.stdout: "flags": 1, 2024-03-24T03:17:24.257 INFO:teuthology.orchestra.run.smithi118.stdout: "default_value": "128", 2024-03-24T03:17:24.257 INFO:teuthology.orchestra.run.smithi118.stdout: "min": "", 2024-03-24T03:17:24.257 INFO:teuthology.orchestra.run.smithi118.stdout: "max": "", 2024-03-24T03:17:24.257 INFO:teuthology.orchestra.run.smithi118.stdout: "enum_allowed": [], 2024-03-24T03:17:24.257 INFO:teuthology.orchestra.run.smithi118.stdout: "desc": "default pg_num for any created local pool", 2024-03-24T03:17:24.257 INFO:teuthology.orchestra.run.smithi118.stdout: "long_desc": "", 2024-03-24T03:17:24.257 INFO:teuthology.orchestra.run.smithi118.stdout: "tags": [], 2024-03-24T03:17:24.257 INFO:teuthology.orchestra.run.smithi118.stdout: "see_also": [] 2024-03-24T03:17:24.257 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-24T03:17:24.258 INFO:teuthology.orchestra.run.smithi118.stdout: "prefix": { 2024-03-24T03:17:24.258 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "prefix", 2024-03-24T03:17:24.258 INFO:teuthology.orchestra.run.smithi118.stdout: "type": "str", 2024-03-24T03:17:24.258 INFO:teuthology.orchestra.run.smithi118.stdout: "level": "advanced", 2024-03-24T03:17:24.258 INFO:teuthology.orchestra.run.smithi118.stdout: "flags": 1, 2024-03-24T03:17:24.258 INFO:teuthology.orchestra.run.smithi118.stdout: "default_value": "", 2024-03-24T03:17:24.258 INFO:teuthology.orchestra.run.smithi118.stdout: "min": "", 2024-03-24T03:17:24.258 INFO:teuthology.orchestra.run.smithi118.stdout: "max": "", 2024-03-24T03:17:24.258 INFO:teuthology.orchestra.run.smithi118.stdout: "enum_allowed": [], 2024-03-24T03:17:24.259 INFO:teuthology.orchestra.run.smithi118.stdout: "desc": "name prefix for any created local pool", 2024-03-24T03:17:24.259 INFO:teuthology.orchestra.run.smithi118.stdout: "long_desc": "", 2024-03-24T03:17:24.259 INFO:teuthology.orchestra.run.smithi118.stdout: "tags": [], 2024-03-24T03:17:24.259 INFO:teuthology.orchestra.run.smithi118.stdout: "see_also": [] 2024-03-24T03:17:24.259 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-24T03:17:24.259 INFO:teuthology.orchestra.run.smithi118.stdout: "subtree": { 2024-03-24T03:17:24.259 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "subtree", 2024-03-24T03:17:24.259 INFO:teuthology.orchestra.run.smithi118.stdout: "type": "str", 2024-03-24T03:17:24.259 INFO:teuthology.orchestra.run.smithi118.stdout: "level": "advanced", 2024-03-24T03:17:24.260 INFO:teuthology.orchestra.run.smithi118.stdout: "flags": 1, 2024-03-24T03:17:24.260 INFO:teuthology.orchestra.run.smithi118.stdout: "default_value": "rack", 2024-03-24T03:17:24.260 INFO:teuthology.orchestra.run.smithi118.stdout: "min": "", 2024-03-24T03:17:24.260 INFO:teuthology.orchestra.run.smithi118.stdout: "max": "", 2024-03-24T03:17:24.260 INFO:teuthology.orchestra.run.smithi118.stdout: "enum_allowed": [], 2024-03-24T03:17:24.260 INFO:teuthology.orchestra.run.smithi118.stdout: "desc": "CRUSH level for which to create a local pool", 2024-03-24T03:17:24.260 INFO:teuthology.orchestra.run.smithi118.stdout: "long_desc": "which CRUSH subtree type the module should create a pool for.", 2024-03-24T03:17:24.260 INFO:teuthology.orchestra.run.smithi118.stdout: "tags": [], 2024-03-24T03:17:24.260 INFO:teuthology.orchestra.run.smithi118.stdout: "see_also": [] 2024-03-24T03:17:24.260 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-03-24T03:17:24.261 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-03-24T03:17:24.261 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-24T03:17:24.261 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-24T03:17:24.261 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "mds_autoscaler", 2024-03-24T03:17:24.261 INFO:teuthology.orchestra.run.smithi118.stdout: "can_run": true, 2024-03-24T03:17:24.261 INFO:teuthology.orchestra.run.smithi118.stdout: "error_string": "", 2024-03-24T03:17:24.261 INFO:teuthology.orchestra.run.smithi118.stdout: "module_options": { 2024-03-24T03:17:24.261 INFO:teuthology.orchestra.run.smithi118.stdout: "log_level": { 2024-03-24T03:17:24.261 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "log_level", 2024-03-24T03:17:24.262 INFO:teuthology.orchestra.run.smithi118.stdout: "type": "str", 2024-03-24T03:17:24.262 INFO:teuthology.orchestra.run.smithi118.stdout: "level": "advanced", 2024-03-24T03:17:24.262 INFO:teuthology.orchestra.run.smithi118.stdout: "flags": 1, 2024-03-24T03:17:24.262 INFO:teuthology.orchestra.run.smithi118.stdout: "default_value": "", 2024-03-24T03:17:24.262 INFO:teuthology.orchestra.run.smithi118.stdout: "min": "", 2024-03-24T03:17:24.262 INFO:teuthology.orchestra.run.smithi118.stdout: "max": "", 2024-03-24T03:17:24.262 INFO:teuthology.orchestra.run.smithi118.stdout: "enum_allowed": [ 2024-03-24T03:17:24.262 INFO:teuthology.orchestra.run.smithi118.stdout: "", 2024-03-24T03:17:24.263 INFO:teuthology.orchestra.run.smithi118.stdout: "critical", 2024-03-24T03:17:24.263 INFO:teuthology.orchestra.run.smithi118.stdout: "debug", 2024-03-24T03:17:24.263 INFO:teuthology.orchestra.run.smithi118.stdout: "error", 2024-03-24T03:17:24.263 INFO:teuthology.orchestra.run.smithi118.stdout: "info", 2024-03-24T03:17:24.263 INFO:teuthology.orchestra.run.smithi118.stdout: "warning" 2024-03-24T03:17:24.263 INFO:teuthology.orchestra.run.smithi118.stdout: ], 2024-03-24T03:17:24.263 INFO:teuthology.orchestra.run.smithi118.stdout: "desc": "", 2024-03-24T03:17:24.263 INFO:teuthology.orchestra.run.smithi118.stdout: "long_desc": "", 2024-03-24T03:17:24.263 INFO:teuthology.orchestra.run.smithi118.stdout: "tags": [], 2024-03-24T03:17:24.263 INFO:teuthology.orchestra.run.smithi118.stdout: "see_also": [] 2024-03-24T03:17:24.264 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-24T03:17:24.264 INFO:teuthology.orchestra.run.smithi118.stdout: "log_to_cluster": { 2024-03-24T03:17:24.264 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "log_to_cluster", 2024-03-24T03:17:24.264 INFO:teuthology.orchestra.run.smithi118.stdout: "type": "bool", 2024-03-24T03:17:24.264 INFO:teuthology.orchestra.run.smithi118.stdout: "level": "advanced", 2024-03-24T03:17:24.264 INFO:teuthology.orchestra.run.smithi118.stdout: "flags": 1, 2024-03-24T03:17:24.264 INFO:teuthology.orchestra.run.smithi118.stdout: "default_value": "False", 2024-03-24T03:17:24.264 INFO:teuthology.orchestra.run.smithi118.stdout: "min": "", 2024-03-24T03:17:24.265 INFO:teuthology.orchestra.run.smithi118.stdout: "max": "", 2024-03-24T03:17:24.265 INFO:teuthology.orchestra.run.smithi118.stdout: "enum_allowed": [], 2024-03-24T03:17:24.265 INFO:teuthology.orchestra.run.smithi118.stdout: "desc": "", 2024-03-24T03:17:24.265 INFO:teuthology.orchestra.run.smithi118.stdout: "long_desc": "", 2024-03-24T03:17:24.265 INFO:teuthology.orchestra.run.smithi118.stdout: "tags": [], 2024-03-24T03:17:24.265 INFO:teuthology.orchestra.run.smithi118.stdout: "see_also": [] 2024-03-24T03:17:24.265 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-24T03:17:24.265 INFO:teuthology.orchestra.run.smithi118.stdout: "log_to_cluster_level": { 2024-03-24T03:17:24.265 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "log_to_cluster_level", 2024-03-24T03:17:24.265 INFO:teuthology.orchestra.run.smithi118.stdout: "type": "str", 2024-03-24T03:17:24.266 INFO:teuthology.orchestra.run.smithi118.stdout: "level": "advanced", 2024-03-24T03:17:24.266 INFO:teuthology.orchestra.run.smithi118.stdout: "flags": 1, 2024-03-24T03:17:24.266 INFO:teuthology.orchestra.run.smithi118.stdout: "default_value": "info", 2024-03-24T03:17:24.266 INFO:teuthology.orchestra.run.smithi118.stdout: "min": "", 2024-03-24T03:17:24.266 INFO:teuthology.orchestra.run.smithi118.stdout: "max": "", 2024-03-24T03:17:24.266 INFO:teuthology.orchestra.run.smithi118.stdout: "enum_allowed": [ 2024-03-24T03:17:24.266 INFO:teuthology.orchestra.run.smithi118.stdout: "", 2024-03-24T03:17:24.266 INFO:teuthology.orchestra.run.smithi118.stdout: "critical", 2024-03-24T03:17:24.267 INFO:teuthology.orchestra.run.smithi118.stdout: "debug", 2024-03-24T03:17:24.267 INFO:teuthology.orchestra.run.smithi118.stdout: "error", 2024-03-24T03:17:24.267 INFO:teuthology.orchestra.run.smithi118.stdout: "info", 2024-03-24T03:17:24.267 INFO:teuthology.orchestra.run.smithi118.stdout: "warning" 2024-03-24T03:17:24.267 INFO:teuthology.orchestra.run.smithi118.stdout: ], 2024-03-24T03:17:24.267 INFO:teuthology.orchestra.run.smithi118.stdout: "desc": "", 2024-03-24T03:17:24.267 INFO:teuthology.orchestra.run.smithi118.stdout: "long_desc": "", 2024-03-24T03:17:24.267 INFO:teuthology.orchestra.run.smithi118.stdout: "tags": [], 2024-03-24T03:17:24.267 INFO:teuthology.orchestra.run.smithi118.stdout: "see_also": [] 2024-03-24T03:17:24.267 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-24T03:17:24.268 INFO:teuthology.orchestra.run.smithi118.stdout: "log_to_file": { 2024-03-24T03:17:24.268 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "log_to_file", 2024-03-24T03:17:24.268 INFO:teuthology.orchestra.run.smithi118.stdout: "type": "bool", 2024-03-24T03:17:24.268 INFO:teuthology.orchestra.run.smithi118.stdout: "level": "advanced", 2024-03-24T03:17:24.268 INFO:teuthology.orchestra.run.smithi118.stdout: "flags": 1, 2024-03-24T03:17:24.268 INFO:teuthology.orchestra.run.smithi118.stdout: "default_value": "False", 2024-03-24T03:17:24.268 INFO:teuthology.orchestra.run.smithi118.stdout: "min": "", 2024-03-24T03:17:24.268 INFO:teuthology.orchestra.run.smithi118.stdout: "max": "", 2024-03-24T03:17:24.268 INFO:teuthology.orchestra.run.smithi118.stdout: "enum_allowed": [], 2024-03-24T03:17:24.269 INFO:teuthology.orchestra.run.smithi118.stdout: "desc": "", 2024-03-24T03:17:24.269 INFO:teuthology.orchestra.run.smithi118.stdout: "long_desc": "", 2024-03-24T03:17:24.269 INFO:teuthology.orchestra.run.smithi118.stdout: "tags": [], 2024-03-24T03:17:24.269 INFO:teuthology.orchestra.run.smithi118.stdout: "see_also": [] 2024-03-24T03:17:24.269 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-03-24T03:17:24.269 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-03-24T03:17:24.269 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-24T03:17:24.269 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-24T03:17:24.269 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "mirroring", 2024-03-24T03:17:24.269 INFO:teuthology.orchestra.run.smithi118.stdout: "can_run": true, 2024-03-24T03:17:24.270 INFO:teuthology.orchestra.run.smithi118.stdout: "error_string": "", 2024-03-24T03:17:24.270 INFO:teuthology.orchestra.run.smithi118.stdout: "module_options": { 2024-03-24T03:17:24.270 INFO:teuthology.orchestra.run.smithi118.stdout: "log_level": { 2024-03-24T03:17:24.270 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "log_level", 2024-03-24T03:17:24.270 INFO:teuthology.orchestra.run.smithi118.stdout: "type": "str", 2024-03-24T03:17:24.270 INFO:teuthology.orchestra.run.smithi118.stdout: "level": "advanced", 2024-03-24T03:17:24.270 INFO:teuthology.orchestra.run.smithi118.stdout: "flags": 1, 2024-03-24T03:17:24.270 INFO:teuthology.orchestra.run.smithi118.stdout: "default_value": "", 2024-03-24T03:17:24.270 INFO:teuthology.orchestra.run.smithi118.stdout: "min": "", 2024-03-24T03:17:24.271 INFO:teuthology.orchestra.run.smithi118.stdout: "max": "", 2024-03-24T03:17:24.271 INFO:teuthology.orchestra.run.smithi118.stdout: "enum_allowed": [ 2024-03-24T03:17:24.271 INFO:teuthology.orchestra.run.smithi118.stdout: "", 2024-03-24T03:17:24.271 INFO:teuthology.orchestra.run.smithi118.stdout: "critical", 2024-03-24T03:17:24.271 INFO:teuthology.orchestra.run.smithi118.stdout: "debug", 2024-03-24T03:17:24.271 INFO:teuthology.orchestra.run.smithi118.stdout: "error", 2024-03-24T03:17:24.271 INFO:teuthology.orchestra.run.smithi118.stdout: "info", 2024-03-24T03:17:24.271 INFO:teuthology.orchestra.run.smithi118.stdout: "warning" 2024-03-24T03:17:24.271 INFO:teuthology.orchestra.run.smithi118.stdout: ], 2024-03-24T03:17:24.272 INFO:teuthology.orchestra.run.smithi118.stdout: "desc": "", 2024-03-24T03:17:24.272 INFO:teuthology.orchestra.run.smithi118.stdout: "long_desc": "", 2024-03-24T03:17:24.272 INFO:teuthology.orchestra.run.smithi118.stdout: "tags": [], 2024-03-24T03:17:24.272 INFO:teuthology.orchestra.run.smithi118.stdout: "see_also": [] 2024-03-24T03:17:24.272 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-24T03:17:24.272 INFO:teuthology.orchestra.run.smithi118.stdout: "log_to_cluster": { 2024-03-24T03:17:24.272 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "log_to_cluster", 2024-03-24T03:17:24.272 INFO:teuthology.orchestra.run.smithi118.stdout: "type": "bool", 2024-03-24T03:17:24.272 INFO:teuthology.orchestra.run.smithi118.stdout: "level": "advanced", 2024-03-24T03:17:24.273 INFO:teuthology.orchestra.run.smithi118.stdout: "flags": 1, 2024-03-24T03:17:24.273 INFO:teuthology.orchestra.run.smithi118.stdout: "default_value": "False", 2024-03-24T03:17:24.273 INFO:teuthology.orchestra.run.smithi118.stdout: "min": "", 2024-03-24T03:17:24.273 INFO:teuthology.orchestra.run.smithi118.stdout: "max": "", 2024-03-24T03:17:24.273 INFO:teuthology.orchestra.run.smithi118.stdout: "enum_allowed": [], 2024-03-24T03:17:24.273 INFO:teuthology.orchestra.run.smithi118.stdout: "desc": "", 2024-03-24T03:17:24.273 INFO:teuthology.orchestra.run.smithi118.stdout: "long_desc": "", 2024-03-24T03:17:24.273 INFO:teuthology.orchestra.run.smithi118.stdout: "tags": [], 2024-03-24T03:17:24.273 INFO:teuthology.orchestra.run.smithi118.stdout: "see_also": [] 2024-03-24T03:17:24.273 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-24T03:17:24.274 INFO:teuthology.orchestra.run.smithi118.stdout: "log_to_cluster_level": { 2024-03-24T03:17:24.274 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "log_to_cluster_level", 2024-03-24T03:17:24.274 INFO:teuthology.orchestra.run.smithi118.stdout: "type": "str", 2024-03-24T03:17:24.274 INFO:teuthology.orchestra.run.smithi118.stdout: "level": "advanced", 2024-03-24T03:17:24.274 INFO:teuthology.orchestra.run.smithi118.stdout: "flags": 1, 2024-03-24T03:17:24.274 INFO:teuthology.orchestra.run.smithi118.stdout: "default_value": "info", 2024-03-24T03:17:24.274 INFO:teuthology.orchestra.run.smithi118.stdout: "min": "", 2024-03-24T03:17:24.274 INFO:teuthology.orchestra.run.smithi118.stdout: "max": "", 2024-03-24T03:17:24.275 INFO:teuthology.orchestra.run.smithi118.stdout: "enum_allowed": [ 2024-03-24T03:17:24.275 INFO:teuthology.orchestra.run.smithi118.stdout: "", 2024-03-24T03:17:24.275 INFO:teuthology.orchestra.run.smithi118.stdout: "critical", 2024-03-24T03:17:24.275 INFO:teuthology.orchestra.run.smithi118.stdout: "debug", 2024-03-24T03:17:24.275 INFO:teuthology.orchestra.run.smithi118.stdout: "error", 2024-03-24T03:17:24.275 INFO:teuthology.orchestra.run.smithi118.stdout: "info", 2024-03-24T03:17:24.275 INFO:teuthology.orchestra.run.smithi118.stdout: "warning" 2024-03-24T03:17:24.275 INFO:teuthology.orchestra.run.smithi118.stdout: ], 2024-03-24T03:17:24.275 INFO:teuthology.orchestra.run.smithi118.stdout: "desc": "", 2024-03-24T03:17:24.275 INFO:teuthology.orchestra.run.smithi118.stdout: "long_desc": "", 2024-03-24T03:17:24.276 INFO:teuthology.orchestra.run.smithi118.stdout: "tags": [], 2024-03-24T03:17:24.276 INFO:teuthology.orchestra.run.smithi118.stdout: "see_also": [] 2024-03-24T03:17:24.276 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-24T03:17:24.276 INFO:teuthology.orchestra.run.smithi118.stdout: "log_to_file": { 2024-03-24T03:17:24.276 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "log_to_file", 2024-03-24T03:17:24.276 INFO:teuthology.orchestra.run.smithi118.stdout: "type": "bool", 2024-03-24T03:17:24.276 INFO:teuthology.orchestra.run.smithi118.stdout: "level": "advanced", 2024-03-24T03:17:24.276 INFO:teuthology.orchestra.run.smithi118.stdout: "flags": 1, 2024-03-24T03:17:24.276 INFO:teuthology.orchestra.run.smithi118.stdout: "default_value": "False", 2024-03-24T03:17:24.276 INFO:teuthology.orchestra.run.smithi118.stdout: "min": "", 2024-03-24T03:17:24.277 INFO:teuthology.orchestra.run.smithi118.stdout: "max": "", 2024-03-24T03:17:24.277 INFO:teuthology.orchestra.run.smithi118.stdout: "enum_allowed": [], 2024-03-24T03:17:24.277 INFO:teuthology.orchestra.run.smithi118.stdout: "desc": "", 2024-03-24T03:17:24.277 INFO:teuthology.orchestra.run.smithi118.stdout: "long_desc": "", 2024-03-24T03:17:24.277 INFO:teuthology.orchestra.run.smithi118.stdout: "tags": [], 2024-03-24T03:17:24.277 INFO:teuthology.orchestra.run.smithi118.stdout: "see_also": [] 2024-03-24T03:17:24.277 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-03-24T03:17:24.277 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-03-24T03:17:24.277 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-24T03:17:24.278 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-24T03:17:24.278 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "nfs", 2024-03-24T03:17:24.278 INFO:teuthology.orchestra.run.smithi118.stdout: "can_run": true, 2024-03-24T03:17:24.278 INFO:teuthology.orchestra.run.smithi118.stdout: "error_string": "", 2024-03-24T03:17:24.278 INFO:teuthology.orchestra.run.smithi118.stdout: "module_options": { 2024-03-24T03:17:24.278 INFO:teuthology.orchestra.run.smithi118.stdout: "log_level": { 2024-03-24T03:17:24.278 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "log_level", 2024-03-24T03:17:24.278 INFO:teuthology.orchestra.run.smithi118.stdout: "type": "str", 2024-03-24T03:17:24.278 INFO:teuthology.orchestra.run.smithi118.stdout: "level": "advanced", 2024-03-24T03:17:24.278 INFO:teuthology.orchestra.run.smithi118.stdout: "flags": 1, 2024-03-24T03:17:24.279 INFO:teuthology.orchestra.run.smithi118.stdout: "default_value": "", 2024-03-24T03:17:24.279 INFO:teuthology.orchestra.run.smithi118.stdout: "min": "", 2024-03-24T03:17:24.279 INFO:teuthology.orchestra.run.smithi118.stdout: "max": "", 2024-03-24T03:17:24.279 INFO:teuthology.orchestra.run.smithi118.stdout: "enum_allowed": [ 2024-03-24T03:17:24.279 INFO:teuthology.orchestra.run.smithi118.stdout: "", 2024-03-24T03:17:24.279 INFO:teuthology.orchestra.run.smithi118.stdout: "critical", 2024-03-24T03:17:24.279 INFO:teuthology.orchestra.run.smithi118.stdout: "debug", 2024-03-24T03:17:24.279 INFO:teuthology.orchestra.run.smithi118.stdout: "error", 2024-03-24T03:17:24.279 INFO:teuthology.orchestra.run.smithi118.stdout: "info", 2024-03-24T03:17:24.280 INFO:teuthology.orchestra.run.smithi118.stdout: "warning" 2024-03-24T03:17:24.280 INFO:teuthology.orchestra.run.smithi118.stdout: ], 2024-03-24T03:17:24.280 INFO:teuthology.orchestra.run.smithi118.stdout: "desc": "", 2024-03-24T03:17:24.280 INFO:teuthology.orchestra.run.smithi118.stdout: "long_desc": "", 2024-03-24T03:17:24.280 INFO:teuthology.orchestra.run.smithi118.stdout: "tags": [], 2024-03-24T03:17:24.280 INFO:teuthology.orchestra.run.smithi118.stdout: "see_also": [] 2024-03-24T03:17:24.280 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-24T03:17:24.280 INFO:teuthology.orchestra.run.smithi118.stdout: "log_to_cluster": { 2024-03-24T03:17:24.280 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "log_to_cluster", 2024-03-24T03:17:24.280 INFO:teuthology.orchestra.run.smithi118.stdout: "type": "bool", 2024-03-24T03:17:24.281 INFO:teuthology.orchestra.run.smithi118.stdout: "level": "advanced", 2024-03-24T03:17:24.281 INFO:teuthology.orchestra.run.smithi118.stdout: "flags": 1, 2024-03-24T03:17:24.281 INFO:teuthology.orchestra.run.smithi118.stdout: "default_value": "False", 2024-03-24T03:17:24.281 INFO:teuthology.orchestra.run.smithi118.stdout: "min": "", 2024-03-24T03:17:24.281 INFO:teuthology.orchestra.run.smithi118.stdout: "max": "", 2024-03-24T03:17:24.281 INFO:teuthology.orchestra.run.smithi118.stdout: "enum_allowed": [], 2024-03-24T03:17:24.281 INFO:teuthology.orchestra.run.smithi118.stdout: "desc": "", 2024-03-24T03:17:24.281 INFO:teuthology.orchestra.run.smithi118.stdout: "long_desc": "", 2024-03-24T03:17:24.281 INFO:teuthology.orchestra.run.smithi118.stdout: "tags": [], 2024-03-24T03:17:24.281 INFO:teuthology.orchestra.run.smithi118.stdout: "see_also": [] 2024-03-24T03:17:24.282 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-24T03:17:24.282 INFO:teuthology.orchestra.run.smithi118.stdout: "log_to_cluster_level": { 2024-03-24T03:17:24.282 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "log_to_cluster_level", 2024-03-24T03:17:24.282 INFO:teuthology.orchestra.run.smithi118.stdout: "type": "str", 2024-03-24T03:17:24.282 INFO:teuthology.orchestra.run.smithi118.stdout: "level": "advanced", 2024-03-24T03:17:24.282 INFO:teuthology.orchestra.run.smithi118.stdout: "flags": 1, 2024-03-24T03:17:24.282 INFO:teuthology.orchestra.run.smithi118.stdout: "default_value": "info", 2024-03-24T03:17:24.282 INFO:teuthology.orchestra.run.smithi118.stdout: "min": "", 2024-03-24T03:17:24.282 INFO:teuthology.orchestra.run.smithi118.stdout: "max": "", 2024-03-24T03:17:24.283 INFO:teuthology.orchestra.run.smithi118.stdout: "enum_allowed": [ 2024-03-24T03:17:24.283 INFO:teuthology.orchestra.run.smithi118.stdout: "", 2024-03-24T03:17:24.283 INFO:teuthology.orchestra.run.smithi118.stdout: "critical", 2024-03-24T03:17:24.283 INFO:teuthology.orchestra.run.smithi118.stdout: "debug", 2024-03-24T03:17:24.283 INFO:teuthology.orchestra.run.smithi118.stdout: "error", 2024-03-24T03:17:24.283 INFO:teuthology.orchestra.run.smithi118.stdout: "info", 2024-03-24T03:17:24.283 INFO:teuthology.orchestra.run.smithi118.stdout: "warning" 2024-03-24T03:17:24.283 INFO:teuthology.orchestra.run.smithi118.stdout: ], 2024-03-24T03:17:24.283 INFO:teuthology.orchestra.run.smithi118.stdout: "desc": "", 2024-03-24T03:17:24.284 INFO:teuthology.orchestra.run.smithi118.stdout: "long_desc": "", 2024-03-24T03:17:24.284 INFO:teuthology.orchestra.run.smithi118.stdout: "tags": [], 2024-03-24T03:17:24.284 INFO:teuthology.orchestra.run.smithi118.stdout: "see_also": [] 2024-03-24T03:17:24.284 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-24T03:17:24.284 INFO:teuthology.orchestra.run.smithi118.stdout: "log_to_file": { 2024-03-24T03:17:24.284 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "log_to_file", 2024-03-24T03:17:24.284 INFO:teuthology.orchestra.run.smithi118.stdout: "type": "bool", 2024-03-24T03:17:24.284 INFO:teuthology.orchestra.run.smithi118.stdout: "level": "advanced", 2024-03-24T03:17:24.284 INFO:teuthology.orchestra.run.smithi118.stdout: "flags": 1, 2024-03-24T03:17:24.285 INFO:teuthology.orchestra.run.smithi118.stdout: "default_value": "False", 2024-03-24T03:17:24.285 INFO:teuthology.orchestra.run.smithi118.stdout: "min": "", 2024-03-24T03:17:24.285 INFO:teuthology.orchestra.run.smithi118.stdout: "max": "", 2024-03-24T03:17:24.285 INFO:teuthology.orchestra.run.smithi118.stdout: "enum_allowed": [], 2024-03-24T03:17:24.285 INFO:teuthology.orchestra.run.smithi118.stdout: "desc": "", 2024-03-24T03:17:24.285 INFO:teuthology.orchestra.run.smithi118.stdout: "long_desc": "", 2024-03-24T03:17:24.285 INFO:teuthology.orchestra.run.smithi118.stdout: "tags": [], 2024-03-24T03:17:24.285 INFO:teuthology.orchestra.run.smithi118.stdout: "see_also": [] 2024-03-24T03:17:24.285 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-03-24T03:17:24.285 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-03-24T03:17:24.286 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-24T03:17:24.286 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-24T03:17:24.286 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "orchestrator", 2024-03-24T03:17:24.286 INFO:teuthology.orchestra.run.smithi118.stdout: "can_run": true, 2024-03-24T03:17:24.286 INFO:teuthology.orchestra.run.smithi118.stdout: "error_string": "", 2024-03-24T03:17:24.286 INFO:teuthology.orchestra.run.smithi118.stdout: "module_options": { 2024-03-24T03:17:24.286 INFO:teuthology.orchestra.run.smithi118.stdout: "fail_fs": { 2024-03-24T03:17:24.286 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "fail_fs", 2024-03-24T03:17:24.286 INFO:teuthology.orchestra.run.smithi118.stdout: "type": "bool", 2024-03-24T03:17:24.287 INFO:teuthology.orchestra.run.smithi118.stdout: "level": "advanced", 2024-03-24T03:17:24.287 INFO:teuthology.orchestra.run.smithi118.stdout: "flags": 0, 2024-03-24T03:17:24.287 INFO:teuthology.orchestra.run.smithi118.stdout: "default_value": "False", 2024-03-24T03:17:24.287 INFO:teuthology.orchestra.run.smithi118.stdout: "min": "", 2024-03-24T03:17:24.287 INFO:teuthology.orchestra.run.smithi118.stdout: "max": "", 2024-03-24T03:17:24.287 INFO:teuthology.orchestra.run.smithi118.stdout: "enum_allowed": [], 2024-03-24T03:17:24.287 INFO:teuthology.orchestra.run.smithi118.stdout: "desc": "Fail filesystem for rapid multi-rank mds upgrade", 2024-03-24T03:17:24.287 INFO:teuthology.orchestra.run.smithi118.stdout: "long_desc": "", 2024-03-24T03:17:24.287 INFO:teuthology.orchestra.run.smithi118.stdout: "tags": [], 2024-03-24T03:17:24.287 INFO:teuthology.orchestra.run.smithi118.stdout: "see_also": [] 2024-03-24T03:17:24.288 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-24T03:17:24.288 INFO:teuthology.orchestra.run.smithi118.stdout: "log_level": { 2024-03-24T03:17:24.288 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "log_level", 2024-03-24T03:17:24.288 INFO:teuthology.orchestra.run.smithi118.stdout: "type": "str", 2024-03-24T03:17:24.288 INFO:teuthology.orchestra.run.smithi118.stdout: "level": "advanced", 2024-03-24T03:17:24.288 INFO:teuthology.orchestra.run.smithi118.stdout: "flags": 1, 2024-03-24T03:17:24.288 INFO:teuthology.orchestra.run.smithi118.stdout: "default_value": "", 2024-03-24T03:17:24.288 INFO:teuthology.orchestra.run.smithi118.stdout: "min": "", 2024-03-24T03:17:24.288 INFO:teuthology.orchestra.run.smithi118.stdout: "max": "", 2024-03-24T03:17:24.289 INFO:teuthology.orchestra.run.smithi118.stdout: "enum_allowed": [ 2024-03-24T03:17:24.289 INFO:teuthology.orchestra.run.smithi118.stdout: "", 2024-03-24T03:17:24.289 INFO:teuthology.orchestra.run.smithi118.stdout: "critical", 2024-03-24T03:17:24.289 INFO:teuthology.orchestra.run.smithi118.stdout: "debug", 2024-03-24T03:17:24.289 INFO:teuthology.orchestra.run.smithi118.stdout: "error", 2024-03-24T03:17:24.289 INFO:teuthology.orchestra.run.smithi118.stdout: "info", 2024-03-24T03:17:24.289 INFO:teuthology.orchestra.run.smithi118.stdout: "warning" 2024-03-24T03:17:24.289 INFO:teuthology.orchestra.run.smithi118.stdout: ], 2024-03-24T03:17:24.289 INFO:teuthology.orchestra.run.smithi118.stdout: "desc": "", 2024-03-24T03:17:24.289 INFO:teuthology.orchestra.run.smithi118.stdout: "long_desc": "", 2024-03-24T03:17:24.290 INFO:teuthology.orchestra.run.smithi118.stdout: "tags": [], 2024-03-24T03:17:24.290 INFO:teuthology.orchestra.run.smithi118.stdout: "see_also": [] 2024-03-24T03:17:24.290 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-24T03:17:24.290 INFO:teuthology.orchestra.run.smithi118.stdout: "log_to_cluster": { 2024-03-24T03:17:24.290 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "log_to_cluster", 2024-03-24T03:17:24.290 INFO:teuthology.orchestra.run.smithi118.stdout: "type": "bool", 2024-03-24T03:17:24.290 INFO:teuthology.orchestra.run.smithi118.stdout: "level": "advanced", 2024-03-24T03:17:24.290 INFO:teuthology.orchestra.run.smithi118.stdout: "flags": 1, 2024-03-24T03:17:24.290 INFO:teuthology.orchestra.run.smithi118.stdout: "default_value": "False", 2024-03-24T03:17:24.291 INFO:teuthology.orchestra.run.smithi118.stdout: "min": "", 2024-03-24T03:17:24.291 INFO:teuthology.orchestra.run.smithi118.stdout: "max": "", 2024-03-24T03:17:24.291 INFO:teuthology.orchestra.run.smithi118.stdout: "enum_allowed": [], 2024-03-24T03:17:24.291 INFO:teuthology.orchestra.run.smithi118.stdout: "desc": "", 2024-03-24T03:17:24.291 INFO:teuthology.orchestra.run.smithi118.stdout: "long_desc": "", 2024-03-24T03:17:24.291 INFO:teuthology.orchestra.run.smithi118.stdout: "tags": [], 2024-03-24T03:17:24.291 INFO:teuthology.orchestra.run.smithi118.stdout: "see_also": [] 2024-03-24T03:17:24.291 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-24T03:17:24.291 INFO:teuthology.orchestra.run.smithi118.stdout: "log_to_cluster_level": { 2024-03-24T03:17:24.292 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "log_to_cluster_level", 2024-03-24T03:17:24.292 INFO:teuthology.orchestra.run.smithi118.stdout: "type": "str", 2024-03-24T03:17:24.292 INFO:teuthology.orchestra.run.smithi118.stdout: "level": "advanced", 2024-03-24T03:17:24.292 INFO:teuthology.orchestra.run.smithi118.stdout: "flags": 1, 2024-03-24T03:17:24.292 INFO:teuthology.orchestra.run.smithi118.stdout: "default_value": "info", 2024-03-24T03:17:24.292 INFO:teuthology.orchestra.run.smithi118.stdout: "min": "", 2024-03-24T03:17:24.292 INFO:teuthology.orchestra.run.smithi118.stdout: "max": "", 2024-03-24T03:17:24.292 INFO:teuthology.orchestra.run.smithi118.stdout: "enum_allowed": [ 2024-03-24T03:17:24.292 INFO:teuthology.orchestra.run.smithi118.stdout: "", 2024-03-24T03:17:24.293 INFO:teuthology.orchestra.run.smithi118.stdout: "critical", 2024-03-24T03:17:24.293 INFO:teuthology.orchestra.run.smithi118.stdout: "debug", 2024-03-24T03:17:24.293 INFO:teuthology.orchestra.run.smithi118.stdout: "error", 2024-03-24T03:17:24.293 INFO:teuthology.orchestra.run.smithi118.stdout: "info", 2024-03-24T03:17:24.293 INFO:teuthology.orchestra.run.smithi118.stdout: "warning" 2024-03-24T03:17:24.293 INFO:teuthology.orchestra.run.smithi118.stdout: ], 2024-03-24T03:17:24.293 INFO:teuthology.orchestra.run.smithi118.stdout: "desc": "", 2024-03-24T03:17:24.293 INFO:teuthology.orchestra.run.smithi118.stdout: "long_desc": "", 2024-03-24T03:17:24.293 INFO:teuthology.orchestra.run.smithi118.stdout: "tags": [], 2024-03-24T03:17:24.293 INFO:teuthology.orchestra.run.smithi118.stdout: "see_also": [] 2024-03-24T03:17:24.294 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-24T03:17:24.294 INFO:teuthology.orchestra.run.smithi118.stdout: "log_to_file": { 2024-03-24T03:17:24.294 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "log_to_file", 2024-03-24T03:17:24.294 INFO:teuthology.orchestra.run.smithi118.stdout: "type": "bool", 2024-03-24T03:17:24.294 INFO:teuthology.orchestra.run.smithi118.stdout: "level": "advanced", 2024-03-24T03:17:24.294 INFO:teuthology.orchestra.run.smithi118.stdout: "flags": 1, 2024-03-24T03:17:24.294 INFO:teuthology.orchestra.run.smithi118.stdout: "default_value": "False", 2024-03-24T03:17:24.294 INFO:teuthology.orchestra.run.smithi118.stdout: "min": "", 2024-03-24T03:17:24.295 INFO:teuthology.orchestra.run.smithi118.stdout: "max": "", 2024-03-24T03:17:24.295 INFO:teuthology.orchestra.run.smithi118.stdout: "enum_allowed": [], 2024-03-24T03:17:24.295 INFO:teuthology.orchestra.run.smithi118.stdout: "desc": "", 2024-03-24T03:17:24.295 INFO:teuthology.orchestra.run.smithi118.stdout: "long_desc": "", 2024-03-24T03:17:24.295 INFO:teuthology.orchestra.run.smithi118.stdout: "tags": [], 2024-03-24T03:17:24.295 INFO:teuthology.orchestra.run.smithi118.stdout: "see_also": [] 2024-03-24T03:17:24.295 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-24T03:17:24.295 INFO:teuthology.orchestra.run.smithi118.stdout: "orchestrator": { 2024-03-24T03:17:24.295 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "orchestrator", 2024-03-24T03:17:24.295 INFO:teuthology.orchestra.run.smithi118.stdout: "type": "str", 2024-03-24T03:17:24.296 INFO:teuthology.orchestra.run.smithi118.stdout: "level": "advanced", 2024-03-24T03:17:24.296 INFO:teuthology.orchestra.run.smithi118.stdout: "flags": 1, 2024-03-24T03:17:24.296 INFO:teuthology.orchestra.run.smithi118.stdout: "default_value": "", 2024-03-24T03:17:24.296 INFO:teuthology.orchestra.run.smithi118.stdout: "min": "", 2024-03-24T03:17:24.296 INFO:teuthology.orchestra.run.smithi118.stdout: "max": "", 2024-03-24T03:17:24.296 INFO:teuthology.orchestra.run.smithi118.stdout: "enum_allowed": [ 2024-03-24T03:17:24.296 INFO:teuthology.orchestra.run.smithi118.stdout: "cephadm", 2024-03-24T03:17:24.296 INFO:teuthology.orchestra.run.smithi118.stdout: "rook", 2024-03-24T03:17:24.297 INFO:teuthology.orchestra.run.smithi118.stdout: "test_orchestrator" 2024-03-24T03:17:24.297 INFO:teuthology.orchestra.run.smithi118.stdout: ], 2024-03-24T03:17:24.297 INFO:teuthology.orchestra.run.smithi118.stdout: "desc": "Orchestrator backend", 2024-03-24T03:17:24.297 INFO:teuthology.orchestra.run.smithi118.stdout: "long_desc": "", 2024-03-24T03:17:24.297 INFO:teuthology.orchestra.run.smithi118.stdout: "tags": [], 2024-03-24T03:17:24.297 INFO:teuthology.orchestra.run.smithi118.stdout: "see_also": [] 2024-03-24T03:17:24.297 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-03-24T03:17:24.297 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-03-24T03:17:24.297 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-24T03:17:24.298 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-24T03:17:24.298 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "osd_perf_query", 2024-03-24T03:17:24.298 INFO:teuthology.orchestra.run.smithi118.stdout: "can_run": true, 2024-03-24T03:17:24.298 INFO:teuthology.orchestra.run.smithi118.stdout: "error_string": "", 2024-03-24T03:17:24.298 INFO:teuthology.orchestra.run.smithi118.stdout: "module_options": { 2024-03-24T03:17:24.298 INFO:teuthology.orchestra.run.smithi118.stdout: "log_level": { 2024-03-24T03:17:24.298 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "log_level", 2024-03-24T03:17:24.298 INFO:teuthology.orchestra.run.smithi118.stdout: "type": "str", 2024-03-24T03:17:24.298 INFO:teuthology.orchestra.run.smithi118.stdout: "level": "advanced", 2024-03-24T03:17:24.299 INFO:teuthology.orchestra.run.smithi118.stdout: "flags": 1, 2024-03-24T03:17:24.299 INFO:teuthology.orchestra.run.smithi118.stdout: "default_value": "", 2024-03-24T03:17:24.299 INFO:teuthology.orchestra.run.smithi118.stdout: "min": "", 2024-03-24T03:17:24.299 INFO:teuthology.orchestra.run.smithi118.stdout: "max": "", 2024-03-24T03:17:24.299 INFO:teuthology.orchestra.run.smithi118.stdout: "enum_allowed": [ 2024-03-24T03:17:24.299 INFO:teuthology.orchestra.run.smithi118.stdout: "", 2024-03-24T03:17:24.299 INFO:teuthology.orchestra.run.smithi118.stdout: "critical", 2024-03-24T03:17:24.299 INFO:teuthology.orchestra.run.smithi118.stdout: "debug", 2024-03-24T03:17:24.299 INFO:teuthology.orchestra.run.smithi118.stdout: "error", 2024-03-24T03:17:24.300 INFO:teuthology.orchestra.run.smithi118.stdout: "info", 2024-03-24T03:17:24.300 INFO:teuthology.orchestra.run.smithi118.stdout: "warning" 2024-03-24T03:17:24.300 INFO:teuthology.orchestra.run.smithi118.stdout: ], 2024-03-24T03:17:24.300 INFO:teuthology.orchestra.run.smithi118.stdout: "desc": "", 2024-03-24T03:17:24.300 INFO:teuthology.orchestra.run.smithi118.stdout: "long_desc": "", 2024-03-24T03:17:24.300 INFO:teuthology.orchestra.run.smithi118.stdout: "tags": [], 2024-03-24T03:17:24.300 INFO:teuthology.orchestra.run.smithi118.stdout: "see_also": [] 2024-03-24T03:17:24.300 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-24T03:17:24.300 INFO:teuthology.orchestra.run.smithi118.stdout: "log_to_cluster": { 2024-03-24T03:17:24.301 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "log_to_cluster", 2024-03-24T03:17:24.301 INFO:teuthology.orchestra.run.smithi118.stdout: "type": "bool", 2024-03-24T03:17:24.301 INFO:teuthology.orchestra.run.smithi118.stdout: "level": "advanced", 2024-03-24T03:17:24.301 INFO:teuthology.orchestra.run.smithi118.stdout: "flags": 1, 2024-03-24T03:17:24.301 INFO:teuthology.orchestra.run.smithi118.stdout: "default_value": "False", 2024-03-24T03:17:24.301 INFO:teuthology.orchestra.run.smithi118.stdout: "min": "", 2024-03-24T03:17:24.301 INFO:teuthology.orchestra.run.smithi118.stdout: "max": "", 2024-03-24T03:17:24.301 INFO:teuthology.orchestra.run.smithi118.stdout: "enum_allowed": [], 2024-03-24T03:17:24.301 INFO:teuthology.orchestra.run.smithi118.stdout: "desc": "", 2024-03-24T03:17:24.302 INFO:teuthology.orchestra.run.smithi118.stdout: "long_desc": "", 2024-03-24T03:17:24.302 INFO:teuthology.orchestra.run.smithi118.stdout: "tags": [], 2024-03-24T03:17:24.302 INFO:teuthology.orchestra.run.smithi118.stdout: "see_also": [] 2024-03-24T03:17:24.302 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-24T03:17:24.302 INFO:teuthology.orchestra.run.smithi118.stdout: "log_to_cluster_level": { 2024-03-24T03:17:24.302 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "log_to_cluster_level", 2024-03-24T03:17:24.302 INFO:teuthology.orchestra.run.smithi118.stdout: "type": "str", 2024-03-24T03:17:24.302 INFO:teuthology.orchestra.run.smithi118.stdout: "level": "advanced", 2024-03-24T03:17:24.302 INFO:teuthology.orchestra.run.smithi118.stdout: "flags": 1, 2024-03-24T03:17:24.303 INFO:teuthology.orchestra.run.smithi118.stdout: "default_value": "info", 2024-03-24T03:17:24.303 INFO:teuthology.orchestra.run.smithi118.stdout: "min": "", 2024-03-24T03:17:24.303 INFO:teuthology.orchestra.run.smithi118.stdout: "max": "", 2024-03-24T03:17:24.303 INFO:teuthology.orchestra.run.smithi118.stdout: "enum_allowed": [ 2024-03-24T03:17:24.303 INFO:teuthology.orchestra.run.smithi118.stdout: "", 2024-03-24T03:17:24.303 INFO:teuthology.orchestra.run.smithi118.stdout: "critical", 2024-03-24T03:17:24.303 INFO:teuthology.orchestra.run.smithi118.stdout: "debug", 2024-03-24T03:17:24.303 INFO:teuthology.orchestra.run.smithi118.stdout: "error", 2024-03-24T03:17:24.303 INFO:teuthology.orchestra.run.smithi118.stdout: "info", 2024-03-24T03:17:24.304 INFO:teuthology.orchestra.run.smithi118.stdout: "warning" 2024-03-24T03:17:24.304 INFO:teuthology.orchestra.run.smithi118.stdout: ], 2024-03-24T03:17:24.304 INFO:teuthology.orchestra.run.smithi118.stdout: "desc": "", 2024-03-24T03:17:24.304 INFO:teuthology.orchestra.run.smithi118.stdout: "long_desc": "", 2024-03-24T03:17:24.304 INFO:teuthology.orchestra.run.smithi118.stdout: "tags": [], 2024-03-24T03:17:24.304 INFO:teuthology.orchestra.run.smithi118.stdout: "see_also": [] 2024-03-24T03:17:24.304 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-24T03:17:24.304 INFO:teuthology.orchestra.run.smithi118.stdout: "log_to_file": { 2024-03-24T03:17:24.304 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "log_to_file", 2024-03-24T03:17:24.305 INFO:teuthology.orchestra.run.smithi118.stdout: "type": "bool", 2024-03-24T03:17:24.305 INFO:teuthology.orchestra.run.smithi118.stdout: "level": "advanced", 2024-03-24T03:17:24.305 INFO:teuthology.orchestra.run.smithi118.stdout: "flags": 1, 2024-03-24T03:17:24.305 INFO:teuthology.orchestra.run.smithi118.stdout: "default_value": "False", 2024-03-24T03:17:24.305 INFO:teuthology.orchestra.run.smithi118.stdout: "min": "", 2024-03-24T03:17:24.305 INFO:teuthology.orchestra.run.smithi118.stdout: "max": "", 2024-03-24T03:17:24.305 INFO:teuthology.orchestra.run.smithi118.stdout: "enum_allowed": [], 2024-03-24T03:17:24.305 INFO:teuthology.orchestra.run.smithi118.stdout: "desc": "", 2024-03-24T03:17:24.306 INFO:teuthology.orchestra.run.smithi118.stdout: "long_desc": "", 2024-03-24T03:17:24.306 INFO:teuthology.orchestra.run.smithi118.stdout: "tags": [], 2024-03-24T03:17:24.306 INFO:teuthology.orchestra.run.smithi118.stdout: "see_also": [] 2024-03-24T03:17:24.306 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-03-24T03:17:24.306 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-03-24T03:17:24.306 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-24T03:17:24.306 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-24T03:17:24.306 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "osd_support", 2024-03-24T03:17:24.306 INFO:teuthology.orchestra.run.smithi118.stdout: "can_run": true, 2024-03-24T03:17:24.307 INFO:teuthology.orchestra.run.smithi118.stdout: "error_string": "", 2024-03-24T03:17:24.307 INFO:teuthology.orchestra.run.smithi118.stdout: "module_options": { 2024-03-24T03:17:24.307 INFO:teuthology.orchestra.run.smithi118.stdout: "log_level": { 2024-03-24T03:17:24.307 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "log_level", 2024-03-24T03:17:24.307 INFO:teuthology.orchestra.run.smithi118.stdout: "type": "str", 2024-03-24T03:17:24.307 INFO:teuthology.orchestra.run.smithi118.stdout: "level": "advanced", 2024-03-24T03:17:24.307 INFO:teuthology.orchestra.run.smithi118.stdout: "flags": 1, 2024-03-24T03:17:24.307 INFO:teuthology.orchestra.run.smithi118.stdout: "default_value": "", 2024-03-24T03:17:24.307 INFO:teuthology.orchestra.run.smithi118.stdout: "min": "", 2024-03-24T03:17:24.307 INFO:teuthology.orchestra.run.smithi118.stdout: "max": "", 2024-03-24T03:17:24.308 INFO:teuthology.orchestra.run.smithi118.stdout: "enum_allowed": [ 2024-03-24T03:17:24.308 INFO:teuthology.orchestra.run.smithi118.stdout: "", 2024-03-24T03:17:24.308 INFO:teuthology.orchestra.run.smithi118.stdout: "critical", 2024-03-24T03:17:24.308 INFO:teuthology.orchestra.run.smithi118.stdout: "debug", 2024-03-24T03:17:24.308 INFO:teuthology.orchestra.run.smithi118.stdout: "error", 2024-03-24T03:17:24.308 INFO:teuthology.orchestra.run.smithi118.stdout: "info", 2024-03-24T03:17:24.308 INFO:teuthology.orchestra.run.smithi118.stdout: "warning" 2024-03-24T03:17:24.308 INFO:teuthology.orchestra.run.smithi118.stdout: ], 2024-03-24T03:17:24.308 INFO:teuthology.orchestra.run.smithi118.stdout: "desc": "", 2024-03-24T03:17:24.309 INFO:teuthology.orchestra.run.smithi118.stdout: "long_desc": "", 2024-03-24T03:17:24.309 INFO:teuthology.orchestra.run.smithi118.stdout: "tags": [], 2024-03-24T03:17:24.309 INFO:teuthology.orchestra.run.smithi118.stdout: "see_also": [] 2024-03-24T03:17:24.309 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-24T03:17:24.309 INFO:teuthology.orchestra.run.smithi118.stdout: "log_to_cluster": { 2024-03-24T03:17:24.309 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "log_to_cluster", 2024-03-24T03:17:24.309 INFO:teuthology.orchestra.run.smithi118.stdout: "type": "bool", 2024-03-24T03:17:24.309 INFO:teuthology.orchestra.run.smithi118.stdout: "level": "advanced", 2024-03-24T03:17:24.309 INFO:teuthology.orchestra.run.smithi118.stdout: "flags": 1, 2024-03-24T03:17:24.310 INFO:teuthology.orchestra.run.smithi118.stdout: "default_value": "False", 2024-03-24T03:17:24.310 INFO:teuthology.orchestra.run.smithi118.stdout: "min": "", 2024-03-24T03:17:24.310 INFO:teuthology.orchestra.run.smithi118.stdout: "max": "", 2024-03-24T03:17:24.310 INFO:teuthology.orchestra.run.smithi118.stdout: "enum_allowed": [], 2024-03-24T03:17:24.310 INFO:teuthology.orchestra.run.smithi118.stdout: "desc": "", 2024-03-24T03:17:24.310 INFO:teuthology.orchestra.run.smithi118.stdout: "long_desc": "", 2024-03-24T03:17:24.310 INFO:teuthology.orchestra.run.smithi118.stdout: "tags": [], 2024-03-24T03:17:24.310 INFO:teuthology.orchestra.run.smithi118.stdout: "see_also": [] 2024-03-24T03:17:24.310 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-24T03:17:24.311 INFO:teuthology.orchestra.run.smithi118.stdout: "log_to_cluster_level": { 2024-03-24T03:17:24.311 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "log_to_cluster_level", 2024-03-24T03:17:24.311 INFO:teuthology.orchestra.run.smithi118.stdout: "type": "str", 2024-03-24T03:17:24.311 INFO:teuthology.orchestra.run.smithi118.stdout: "level": "advanced", 2024-03-24T03:17:24.311 INFO:teuthology.orchestra.run.smithi118.stdout: "flags": 1, 2024-03-24T03:17:24.311 INFO:teuthology.orchestra.run.smithi118.stdout: "default_value": "info", 2024-03-24T03:17:24.311 INFO:teuthology.orchestra.run.smithi118.stdout: "min": "", 2024-03-24T03:17:24.311 INFO:teuthology.orchestra.run.smithi118.stdout: "max": "", 2024-03-24T03:17:24.311 INFO:teuthology.orchestra.run.smithi118.stdout: "enum_allowed": [ 2024-03-24T03:17:24.312 INFO:teuthology.orchestra.run.smithi118.stdout: "", 2024-03-24T03:17:24.312 INFO:teuthology.orchestra.run.smithi118.stdout: "critical", 2024-03-24T03:17:24.312 INFO:teuthology.orchestra.run.smithi118.stdout: "debug", 2024-03-24T03:17:24.312 INFO:teuthology.orchestra.run.smithi118.stdout: "error", 2024-03-24T03:17:24.312 INFO:teuthology.orchestra.run.smithi118.stdout: "info", 2024-03-24T03:17:24.312 INFO:teuthology.orchestra.run.smithi118.stdout: "warning" 2024-03-24T03:17:24.312 INFO:teuthology.orchestra.run.smithi118.stdout: ], 2024-03-24T03:17:24.312 INFO:teuthology.orchestra.run.smithi118.stdout: "desc": "", 2024-03-24T03:17:24.312 INFO:teuthology.orchestra.run.smithi118.stdout: "long_desc": "", 2024-03-24T03:17:24.313 INFO:teuthology.orchestra.run.smithi118.stdout: "tags": [], 2024-03-24T03:17:24.313 INFO:teuthology.orchestra.run.smithi118.stdout: "see_also": [] 2024-03-24T03:17:24.313 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-24T03:17:24.313 INFO:teuthology.orchestra.run.smithi118.stdout: "log_to_file": { 2024-03-24T03:17:24.313 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "log_to_file", 2024-03-24T03:17:24.313 INFO:teuthology.orchestra.run.smithi118.stdout: "type": "bool", 2024-03-24T03:17:24.313 INFO:teuthology.orchestra.run.smithi118.stdout: "level": "advanced", 2024-03-24T03:17:24.313 INFO:teuthology.orchestra.run.smithi118.stdout: "flags": 1, 2024-03-24T03:17:24.313 INFO:teuthology.orchestra.run.smithi118.stdout: "default_value": "False", 2024-03-24T03:17:24.314 INFO:teuthology.orchestra.run.smithi118.stdout: "min": "", 2024-03-24T03:17:24.314 INFO:teuthology.orchestra.run.smithi118.stdout: "max": "", 2024-03-24T03:17:24.314 INFO:teuthology.orchestra.run.smithi118.stdout: "enum_allowed": [], 2024-03-24T03:17:24.314 INFO:teuthology.orchestra.run.smithi118.stdout: "desc": "", 2024-03-24T03:17:24.314 INFO:teuthology.orchestra.run.smithi118.stdout: "long_desc": "", 2024-03-24T03:17:24.314 INFO:teuthology.orchestra.run.smithi118.stdout: "tags": [], 2024-03-24T03:17:24.314 INFO:teuthology.orchestra.run.smithi118.stdout: "see_also": [] 2024-03-24T03:17:24.314 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-03-24T03:17:24.314 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-03-24T03:17:24.315 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-24T03:17:24.315 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-24T03:17:24.315 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "pg_autoscaler", 2024-03-24T03:17:24.315 INFO:teuthology.orchestra.run.smithi118.stdout: "can_run": true, 2024-03-24T03:17:24.315 INFO:teuthology.orchestra.run.smithi118.stdout: "error_string": "", 2024-03-24T03:17:24.315 INFO:teuthology.orchestra.run.smithi118.stdout: "module_options": { 2024-03-24T03:17:24.315 INFO:teuthology.orchestra.run.smithi118.stdout: "log_level": { 2024-03-24T03:17:24.315 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "log_level", 2024-03-24T03:17:24.315 INFO:teuthology.orchestra.run.smithi118.stdout: "type": "str", 2024-03-24T03:17:24.316 INFO:teuthology.orchestra.run.smithi118.stdout: "level": "advanced", 2024-03-24T03:17:24.316 INFO:teuthology.orchestra.run.smithi118.stdout: "flags": 1, 2024-03-24T03:17:24.316 INFO:teuthology.orchestra.run.smithi118.stdout: "default_value": "", 2024-03-24T03:17:24.316 INFO:teuthology.orchestra.run.smithi118.stdout: "min": "", 2024-03-24T03:17:24.316 INFO:teuthology.orchestra.run.smithi118.stdout: "max": "", 2024-03-24T03:17:24.316 INFO:teuthology.orchestra.run.smithi118.stdout: "enum_allowed": [ 2024-03-24T03:17:24.316 INFO:teuthology.orchestra.run.smithi118.stdout: "", 2024-03-24T03:17:24.316 INFO:teuthology.orchestra.run.smithi118.stdout: "critical", 2024-03-24T03:17:24.316 INFO:teuthology.orchestra.run.smithi118.stdout: "debug", 2024-03-24T03:17:24.317 INFO:teuthology.orchestra.run.smithi118.stdout: "error", 2024-03-24T03:17:24.317 INFO:teuthology.orchestra.run.smithi118.stdout: "info", 2024-03-24T03:17:24.317 INFO:teuthology.orchestra.run.smithi118.stdout: "warning" 2024-03-24T03:17:24.317 INFO:teuthology.orchestra.run.smithi118.stdout: ], 2024-03-24T03:17:24.317 INFO:teuthology.orchestra.run.smithi118.stdout: "desc": "", 2024-03-24T03:17:24.317 INFO:teuthology.orchestra.run.smithi118.stdout: "long_desc": "", 2024-03-24T03:17:24.317 INFO:teuthology.orchestra.run.smithi118.stdout: "tags": [], 2024-03-24T03:17:24.317 INFO:teuthology.orchestra.run.smithi118.stdout: "see_also": [] 2024-03-24T03:17:24.317 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-24T03:17:24.318 INFO:teuthology.orchestra.run.smithi118.stdout: "log_to_cluster": { 2024-03-24T03:17:24.318 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "log_to_cluster", 2024-03-24T03:17:24.318 INFO:teuthology.orchestra.run.smithi118.stdout: "type": "bool", 2024-03-24T03:17:24.318 INFO:teuthology.orchestra.run.smithi118.stdout: "level": "advanced", 2024-03-24T03:17:24.318 INFO:teuthology.orchestra.run.smithi118.stdout: "flags": 1, 2024-03-24T03:17:24.318 INFO:teuthology.orchestra.run.smithi118.stdout: "default_value": "False", 2024-03-24T03:17:24.318 INFO:teuthology.orchestra.run.smithi118.stdout: "min": "", 2024-03-24T03:17:24.318 INFO:teuthology.orchestra.run.smithi118.stdout: "max": "", 2024-03-24T03:17:24.319 INFO:teuthology.orchestra.run.smithi118.stdout: "enum_allowed": [], 2024-03-24T03:17:24.319 INFO:teuthology.orchestra.run.smithi118.stdout: "desc": "", 2024-03-24T03:17:24.319 INFO:teuthology.orchestra.run.smithi118.stdout: "long_desc": "", 2024-03-24T03:17:24.319 INFO:teuthology.orchestra.run.smithi118.stdout: "tags": [], 2024-03-24T03:17:24.319 INFO:teuthology.orchestra.run.smithi118.stdout: "see_also": [] 2024-03-24T03:17:24.319 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-24T03:17:24.319 INFO:teuthology.orchestra.run.smithi118.stdout: "log_to_cluster_level": { 2024-03-24T03:17:24.319 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "log_to_cluster_level", 2024-03-24T03:17:24.319 INFO:teuthology.orchestra.run.smithi118.stdout: "type": "str", 2024-03-24T03:17:24.320 INFO:teuthology.orchestra.run.smithi118.stdout: "level": "advanced", 2024-03-24T03:17:24.320 INFO:teuthology.orchestra.run.smithi118.stdout: "flags": 1, 2024-03-24T03:17:24.320 INFO:teuthology.orchestra.run.smithi118.stdout: "default_value": "info", 2024-03-24T03:17:24.320 INFO:teuthology.orchestra.run.smithi118.stdout: "min": "", 2024-03-24T03:17:24.320 INFO:teuthology.orchestra.run.smithi118.stdout: "max": "", 2024-03-24T03:17:24.320 INFO:teuthology.orchestra.run.smithi118.stdout: "enum_allowed": [ 2024-03-24T03:17:24.320 INFO:teuthology.orchestra.run.smithi118.stdout: "", 2024-03-24T03:17:24.320 INFO:teuthology.orchestra.run.smithi118.stdout: "critical", 2024-03-24T03:17:24.320 INFO:teuthology.orchestra.run.smithi118.stdout: "debug", 2024-03-24T03:17:24.321 INFO:teuthology.orchestra.run.smithi118.stdout: "error", 2024-03-24T03:17:24.321 INFO:teuthology.orchestra.run.smithi118.stdout: "info", 2024-03-24T03:17:24.321 INFO:teuthology.orchestra.run.smithi118.stdout: "warning" 2024-03-24T03:17:24.321 INFO:teuthology.orchestra.run.smithi118.stdout: ], 2024-03-24T03:17:24.321 INFO:teuthology.orchestra.run.smithi118.stdout: "desc": "", 2024-03-24T03:17:24.321 INFO:teuthology.orchestra.run.smithi118.stdout: "long_desc": "", 2024-03-24T03:17:24.321 INFO:teuthology.orchestra.run.smithi118.stdout: "tags": [], 2024-03-24T03:17:24.321 INFO:teuthology.orchestra.run.smithi118.stdout: "see_also": [] 2024-03-24T03:17:24.321 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-24T03:17:24.322 INFO:teuthology.orchestra.run.smithi118.stdout: "log_to_file": { 2024-03-24T03:17:24.322 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "log_to_file", 2024-03-24T03:17:24.322 INFO:teuthology.orchestra.run.smithi118.stdout: "type": "bool", 2024-03-24T03:17:24.322 INFO:teuthology.orchestra.run.smithi118.stdout: "level": "advanced", 2024-03-24T03:17:24.322 INFO:teuthology.orchestra.run.smithi118.stdout: "flags": 1, 2024-03-24T03:17:24.322 INFO:teuthology.orchestra.run.smithi118.stdout: "default_value": "False", 2024-03-24T03:17:24.322 INFO:teuthology.orchestra.run.smithi118.stdout: "min": "", 2024-03-24T03:17:24.323 INFO:teuthology.orchestra.run.smithi118.stdout: "max": "", 2024-03-24T03:17:24.323 INFO:teuthology.orchestra.run.smithi118.stdout: "enum_allowed": [], 2024-03-24T03:17:24.323 INFO:teuthology.orchestra.run.smithi118.stdout: "desc": "", 2024-03-24T03:17:24.323 INFO:teuthology.orchestra.run.smithi118.stdout: "long_desc": "", 2024-03-24T03:17:24.323 INFO:teuthology.orchestra.run.smithi118.stdout: "tags": [], 2024-03-24T03:17:24.323 INFO:teuthology.orchestra.run.smithi118.stdout: "see_also": [] 2024-03-24T03:17:24.323 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-24T03:17:24.323 INFO:teuthology.orchestra.run.smithi118.stdout: "sleep_interval": { 2024-03-24T03:17:24.323 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "sleep_interval", 2024-03-24T03:17:24.324 INFO:teuthology.orchestra.run.smithi118.stdout: "type": "secs", 2024-03-24T03:17:24.324 INFO:teuthology.orchestra.run.smithi118.stdout: "level": "advanced", 2024-03-24T03:17:24.324 INFO:teuthology.orchestra.run.smithi118.stdout: "flags": 0, 2024-03-24T03:17:24.324 INFO:teuthology.orchestra.run.smithi118.stdout: "default_value": "60", 2024-03-24T03:17:24.324 INFO:teuthology.orchestra.run.smithi118.stdout: "min": "", 2024-03-24T03:17:24.324 INFO:teuthology.orchestra.run.smithi118.stdout: "max": "", 2024-03-24T03:17:24.324 INFO:teuthology.orchestra.run.smithi118.stdout: "enum_allowed": [], 2024-03-24T03:17:24.324 INFO:teuthology.orchestra.run.smithi118.stdout: "desc": "", 2024-03-24T03:17:24.324 INFO:teuthology.orchestra.run.smithi118.stdout: "long_desc": "", 2024-03-24T03:17:24.325 INFO:teuthology.orchestra.run.smithi118.stdout: "tags": [], 2024-03-24T03:17:24.325 INFO:teuthology.orchestra.run.smithi118.stdout: "see_also": [] 2024-03-24T03:17:24.325 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-24T03:17:24.325 INFO:teuthology.orchestra.run.smithi118.stdout: "threshold": { 2024-03-24T03:17:24.325 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "threshold", 2024-03-24T03:17:24.325 INFO:teuthology.orchestra.run.smithi118.stdout: "type": "float", 2024-03-24T03:17:24.325 INFO:teuthology.orchestra.run.smithi118.stdout: "level": "advanced", 2024-03-24T03:17:24.325 INFO:teuthology.orchestra.run.smithi118.stdout: "flags": 0, 2024-03-24T03:17:24.325 INFO:teuthology.orchestra.run.smithi118.stdout: "default_value": "3.0", 2024-03-24T03:17:24.325 INFO:teuthology.orchestra.run.smithi118.stdout: "min": "1.0", 2024-03-24T03:17:24.326 INFO:teuthology.orchestra.run.smithi118.stdout: "max": "", 2024-03-24T03:17:24.326 INFO:teuthology.orchestra.run.smithi118.stdout: "enum_allowed": [], 2024-03-24T03:17:24.326 INFO:teuthology.orchestra.run.smithi118.stdout: "desc": "scaling threshold", 2024-03-24T03:17:24.326 INFO:teuthology.orchestra.run.smithi118.stdout: "long_desc": "The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0", 2024-03-24T03:17:24.326 INFO:teuthology.orchestra.run.smithi118.stdout: "tags": [], 2024-03-24T03:17:24.326 INFO:teuthology.orchestra.run.smithi118.stdout: "see_also": [] 2024-03-24T03:17:24.326 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-03-24T03:17:24.326 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-03-24T03:17:24.326 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-24T03:17:24.327 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-24T03:17:24.327 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "progress", 2024-03-24T03:17:24.327 INFO:teuthology.orchestra.run.smithi118.stdout: "can_run": true, 2024-03-24T03:17:24.327 INFO:teuthology.orchestra.run.smithi118.stdout: "error_string": "", 2024-03-24T03:17:24.327 INFO:teuthology.orchestra.run.smithi118.stdout: "module_options": { 2024-03-24T03:17:24.327 INFO:teuthology.orchestra.run.smithi118.stdout: "allow_pg_recovery_event": { 2024-03-24T03:17:24.327 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "allow_pg_recovery_event", 2024-03-24T03:17:24.327 INFO:teuthology.orchestra.run.smithi118.stdout: "type": "bool", 2024-03-24T03:17:24.327 INFO:teuthology.orchestra.run.smithi118.stdout: "level": "advanced", 2024-03-24T03:17:24.328 INFO:teuthology.orchestra.run.smithi118.stdout: "flags": 1, 2024-03-24T03:17:24.328 INFO:teuthology.orchestra.run.smithi118.stdout: "default_value": "False", 2024-03-24T03:17:24.328 INFO:teuthology.orchestra.run.smithi118.stdout: "min": "", 2024-03-24T03:17:24.328 INFO:teuthology.orchestra.run.smithi118.stdout: "max": "", 2024-03-24T03:17:24.328 INFO:teuthology.orchestra.run.smithi118.stdout: "enum_allowed": [], 2024-03-24T03:17:24.328 INFO:teuthology.orchestra.run.smithi118.stdout: "desc": "allow the module to show pg recovery progress", 2024-03-24T03:17:24.328 INFO:teuthology.orchestra.run.smithi118.stdout: "long_desc": "", 2024-03-24T03:17:24.328 INFO:teuthology.orchestra.run.smithi118.stdout: "tags": [], 2024-03-24T03:17:24.328 INFO:teuthology.orchestra.run.smithi118.stdout: "see_also": [] 2024-03-24T03:17:24.328 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-24T03:17:24.329 INFO:teuthology.orchestra.run.smithi118.stdout: "enabled": { 2024-03-24T03:17:24.329 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "enabled", 2024-03-24T03:17:24.329 INFO:teuthology.orchestra.run.smithi118.stdout: "type": "bool", 2024-03-24T03:17:24.329 INFO:teuthology.orchestra.run.smithi118.stdout: "level": "advanced", 2024-03-24T03:17:24.329 INFO:teuthology.orchestra.run.smithi118.stdout: "flags": 0, 2024-03-24T03:17:24.329 INFO:teuthology.orchestra.run.smithi118.stdout: "default_value": "True", 2024-03-24T03:17:24.329 INFO:teuthology.orchestra.run.smithi118.stdout: "min": "", 2024-03-24T03:17:24.329 INFO:teuthology.orchestra.run.smithi118.stdout: "max": "", 2024-03-24T03:17:24.329 INFO:teuthology.orchestra.run.smithi118.stdout: "enum_allowed": [], 2024-03-24T03:17:24.329 INFO:teuthology.orchestra.run.smithi118.stdout: "desc": "", 2024-03-24T03:17:24.330 INFO:teuthology.orchestra.run.smithi118.stdout: "long_desc": "", 2024-03-24T03:17:24.330 INFO:teuthology.orchestra.run.smithi118.stdout: "tags": [], 2024-03-24T03:17:24.330 INFO:teuthology.orchestra.run.smithi118.stdout: "see_also": [] 2024-03-24T03:17:24.330 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-24T03:17:24.330 INFO:teuthology.orchestra.run.smithi118.stdout: "log_level": { 2024-03-24T03:17:24.330 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "log_level", 2024-03-24T03:17:24.330 INFO:teuthology.orchestra.run.smithi118.stdout: "type": "str", 2024-03-24T03:17:24.330 INFO:teuthology.orchestra.run.smithi118.stdout: "level": "advanced", 2024-03-24T03:17:24.330 INFO:teuthology.orchestra.run.smithi118.stdout: "flags": 1, 2024-03-24T03:17:24.331 INFO:teuthology.orchestra.run.smithi118.stdout: "default_value": "", 2024-03-24T03:17:24.331 INFO:teuthology.orchestra.run.smithi118.stdout: "min": "", 2024-03-24T03:17:24.331 INFO:teuthology.orchestra.run.smithi118.stdout: "max": "", 2024-03-24T03:17:24.331 INFO:teuthology.orchestra.run.smithi118.stdout: "enum_allowed": [ 2024-03-24T03:17:24.331 INFO:teuthology.orchestra.run.smithi118.stdout: "", 2024-03-24T03:17:24.331 INFO:teuthology.orchestra.run.smithi118.stdout: "critical", 2024-03-24T03:17:24.331 INFO:teuthology.orchestra.run.smithi118.stdout: "debug", 2024-03-24T03:17:24.331 INFO:teuthology.orchestra.run.smithi118.stdout: "error", 2024-03-24T03:17:24.331 INFO:teuthology.orchestra.run.smithi118.stdout: "info", 2024-03-24T03:17:24.332 INFO:teuthology.orchestra.run.smithi118.stdout: "warning" 2024-03-24T03:17:24.332 INFO:teuthology.orchestra.run.smithi118.stdout: ], 2024-03-24T03:17:24.332 INFO:teuthology.orchestra.run.smithi118.stdout: "desc": "", 2024-03-24T03:17:24.332 INFO:teuthology.orchestra.run.smithi118.stdout: "long_desc": "", 2024-03-24T03:17:24.332 INFO:teuthology.orchestra.run.smithi118.stdout: "tags": [], 2024-03-24T03:17:24.332 INFO:teuthology.orchestra.run.smithi118.stdout: "see_also": [] 2024-03-24T03:17:24.332 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-24T03:17:24.332 INFO:teuthology.orchestra.run.smithi118.stdout: "log_to_cluster": { 2024-03-24T03:17:24.332 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "log_to_cluster", 2024-03-24T03:17:24.333 INFO:teuthology.orchestra.run.smithi118.stdout: "type": "bool", 2024-03-24T03:17:24.333 INFO:teuthology.orchestra.run.smithi118.stdout: "level": "advanced", 2024-03-24T03:17:24.333 INFO:teuthology.orchestra.run.smithi118.stdout: "flags": 1, 2024-03-24T03:17:24.333 INFO:teuthology.orchestra.run.smithi118.stdout: "default_value": "False", 2024-03-24T03:17:24.333 INFO:teuthology.orchestra.run.smithi118.stdout: "min": "", 2024-03-24T03:17:24.333 INFO:teuthology.orchestra.run.smithi118.stdout: "max": "", 2024-03-24T03:17:24.333 INFO:teuthology.orchestra.run.smithi118.stdout: "enum_allowed": [], 2024-03-24T03:17:24.333 INFO:teuthology.orchestra.run.smithi118.stdout: "desc": "", 2024-03-24T03:17:24.333 INFO:teuthology.orchestra.run.smithi118.stdout: "long_desc": "", 2024-03-24T03:17:24.333 INFO:teuthology.orchestra.run.smithi118.stdout: "tags": [], 2024-03-24T03:17:24.334 INFO:teuthology.orchestra.run.smithi118.stdout: "see_also": [] 2024-03-24T03:17:24.334 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-24T03:17:24.334 INFO:teuthology.orchestra.run.smithi118.stdout: "log_to_cluster_level": { 2024-03-24T03:17:24.334 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "log_to_cluster_level", 2024-03-24T03:17:24.334 INFO:teuthology.orchestra.run.smithi118.stdout: "type": "str", 2024-03-24T03:17:24.334 INFO:teuthology.orchestra.run.smithi118.stdout: "level": "advanced", 2024-03-24T03:17:24.334 INFO:teuthology.orchestra.run.smithi118.stdout: "flags": 1, 2024-03-24T03:17:24.334 INFO:teuthology.orchestra.run.smithi118.stdout: "default_value": "info", 2024-03-24T03:17:24.335 INFO:teuthology.orchestra.run.smithi118.stdout: "min": "", 2024-03-24T03:17:24.335 INFO:teuthology.orchestra.run.smithi118.stdout: "max": "", 2024-03-24T03:17:24.335 INFO:teuthology.orchestra.run.smithi118.stdout: "enum_allowed": [ 2024-03-24T03:17:24.335 INFO:teuthology.orchestra.run.smithi118.stdout: "", 2024-03-24T03:17:24.335 INFO:teuthology.orchestra.run.smithi118.stdout: "critical", 2024-03-24T03:17:24.335 INFO:teuthology.orchestra.run.smithi118.stdout: "debug", 2024-03-24T03:17:24.335 INFO:teuthology.orchestra.run.smithi118.stdout: "error", 2024-03-24T03:17:24.335 INFO:teuthology.orchestra.run.smithi118.stdout: "info", 2024-03-24T03:17:24.335 INFO:teuthology.orchestra.run.smithi118.stdout: "warning" 2024-03-24T03:17:24.335 INFO:teuthology.orchestra.run.smithi118.stdout: ], 2024-03-24T03:17:24.336 INFO:teuthology.orchestra.run.smithi118.stdout: "desc": "", 2024-03-24T03:17:24.336 INFO:teuthology.orchestra.run.smithi118.stdout: "long_desc": "", 2024-03-24T03:17:24.336 INFO:teuthology.orchestra.run.smithi118.stdout: "tags": [], 2024-03-24T03:17:24.336 INFO:teuthology.orchestra.run.smithi118.stdout: "see_also": [] 2024-03-24T03:17:24.336 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-24T03:17:24.336 INFO:teuthology.orchestra.run.smithi118.stdout: "log_to_file": { 2024-03-24T03:17:24.336 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "log_to_file", 2024-03-24T03:17:24.336 INFO:teuthology.orchestra.run.smithi118.stdout: "type": "bool", 2024-03-24T03:17:24.336 INFO:teuthology.orchestra.run.smithi118.stdout: "level": "advanced", 2024-03-24T03:17:24.337 INFO:teuthology.orchestra.run.smithi118.stdout: "flags": 1, 2024-03-24T03:17:24.337 INFO:teuthology.orchestra.run.smithi118.stdout: "default_value": "False", 2024-03-24T03:17:24.337 INFO:teuthology.orchestra.run.smithi118.stdout: "min": "", 2024-03-24T03:17:24.337 INFO:teuthology.orchestra.run.smithi118.stdout: "max": "", 2024-03-24T03:17:24.337 INFO:teuthology.orchestra.run.smithi118.stdout: "enum_allowed": [], 2024-03-24T03:17:24.337 INFO:teuthology.orchestra.run.smithi118.stdout: "desc": "", 2024-03-24T03:17:24.337 INFO:teuthology.orchestra.run.smithi118.stdout: "long_desc": "", 2024-03-24T03:17:24.337 INFO:teuthology.orchestra.run.smithi118.stdout: "tags": [], 2024-03-24T03:17:24.337 INFO:teuthology.orchestra.run.smithi118.stdout: "see_also": [] 2024-03-24T03:17:24.338 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-24T03:17:24.338 INFO:teuthology.orchestra.run.smithi118.stdout: "max_completed_events": { 2024-03-24T03:17:24.338 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "max_completed_events", 2024-03-24T03:17:24.338 INFO:teuthology.orchestra.run.smithi118.stdout: "type": "int", 2024-03-24T03:17:24.338 INFO:teuthology.orchestra.run.smithi118.stdout: "level": "advanced", 2024-03-24T03:17:24.338 INFO:teuthology.orchestra.run.smithi118.stdout: "flags": 1, 2024-03-24T03:17:24.338 INFO:teuthology.orchestra.run.smithi118.stdout: "default_value": "50", 2024-03-24T03:17:24.338 INFO:teuthology.orchestra.run.smithi118.stdout: "min": "", 2024-03-24T03:17:24.338 INFO:teuthology.orchestra.run.smithi118.stdout: "max": "", 2024-03-24T03:17:24.339 INFO:teuthology.orchestra.run.smithi118.stdout: "enum_allowed": [], 2024-03-24T03:17:24.339 INFO:teuthology.orchestra.run.smithi118.stdout: "desc": "number of past completed events to remember", 2024-03-24T03:17:24.339 INFO:teuthology.orchestra.run.smithi118.stdout: "long_desc": "", 2024-03-24T03:17:24.339 INFO:teuthology.orchestra.run.smithi118.stdout: "tags": [], 2024-03-24T03:17:24.339 INFO:teuthology.orchestra.run.smithi118.stdout: "see_also": [] 2024-03-24T03:17:24.339 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-24T03:17:24.339 INFO:teuthology.orchestra.run.smithi118.stdout: "sleep_interval": { 2024-03-24T03:17:24.339 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "sleep_interval", 2024-03-24T03:17:24.340 INFO:teuthology.orchestra.run.smithi118.stdout: "type": "secs", 2024-03-24T03:17:24.340 INFO:teuthology.orchestra.run.smithi118.stdout: "level": "advanced", 2024-03-24T03:17:24.340 INFO:teuthology.orchestra.run.smithi118.stdout: "flags": 1, 2024-03-24T03:17:24.340 INFO:teuthology.orchestra.run.smithi118.stdout: "default_value": "5", 2024-03-24T03:17:24.340 INFO:teuthology.orchestra.run.smithi118.stdout: "min": "", 2024-03-24T03:17:24.340 INFO:teuthology.orchestra.run.smithi118.stdout: "max": "", 2024-03-24T03:17:24.340 INFO:teuthology.orchestra.run.smithi118.stdout: "enum_allowed": [], 2024-03-24T03:17:24.340 INFO:teuthology.orchestra.run.smithi118.stdout: "desc": "how long the module is going to sleep", 2024-03-24T03:17:24.340 INFO:teuthology.orchestra.run.smithi118.stdout: "long_desc": "", 2024-03-24T03:17:24.341 INFO:teuthology.orchestra.run.smithi118.stdout: "tags": [], 2024-03-24T03:17:24.341 INFO:teuthology.orchestra.run.smithi118.stdout: "see_also": [] 2024-03-24T03:17:24.341 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-03-24T03:17:24.341 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-03-24T03:17:24.341 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-24T03:17:24.341 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-24T03:17:24.341 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "prometheus", 2024-03-24T03:17:24.341 INFO:teuthology.orchestra.run.smithi118.stdout: "can_run": true, 2024-03-24T03:17:24.342 INFO:teuthology.orchestra.run.smithi118.stdout: "error_string": "", 2024-03-24T03:17:24.342 INFO:teuthology.orchestra.run.smithi118.stdout: "module_options": { 2024-03-24T03:17:24.342 INFO:teuthology.orchestra.run.smithi118.stdout: "cache": { 2024-03-24T03:17:24.342 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "cache", 2024-03-24T03:17:24.342 INFO:teuthology.orchestra.run.smithi118.stdout: "type": "bool", 2024-03-24T03:17:24.342 INFO:teuthology.orchestra.run.smithi118.stdout: "level": "advanced", 2024-03-24T03:17:24.342 INFO:teuthology.orchestra.run.smithi118.stdout: "flags": 0, 2024-03-24T03:17:24.342 INFO:teuthology.orchestra.run.smithi118.stdout: "default_value": "True", 2024-03-24T03:17:24.342 INFO:teuthology.orchestra.run.smithi118.stdout: "min": "", 2024-03-24T03:17:24.343 INFO:teuthology.orchestra.run.smithi118.stdout: "max": "", 2024-03-24T03:17:24.343 INFO:teuthology.orchestra.run.smithi118.stdout: "enum_allowed": [], 2024-03-24T03:17:24.343 INFO:teuthology.orchestra.run.smithi118.stdout: "desc": "", 2024-03-24T03:17:24.343 INFO:teuthology.orchestra.run.smithi118.stdout: "long_desc": "", 2024-03-24T03:17:24.343 INFO:teuthology.orchestra.run.smithi118.stdout: "tags": [], 2024-03-24T03:17:24.343 INFO:teuthology.orchestra.run.smithi118.stdout: "see_also": [] 2024-03-24T03:17:24.343 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-24T03:17:24.343 INFO:teuthology.orchestra.run.smithi118.stdout: "exclude_perf_counters": { 2024-03-24T03:17:24.344 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "exclude_perf_counters", 2024-03-24T03:17:24.344 INFO:teuthology.orchestra.run.smithi118.stdout: "type": "bool", 2024-03-24T03:17:24.344 INFO:teuthology.orchestra.run.smithi118.stdout: "level": "advanced", 2024-03-24T03:17:24.344 INFO:teuthology.orchestra.run.smithi118.stdout: "flags": 1, 2024-03-24T03:17:24.344 INFO:teuthology.orchestra.run.smithi118.stdout: "default_value": "True", 2024-03-24T03:17:24.344 INFO:teuthology.orchestra.run.smithi118.stdout: "min": "", 2024-03-24T03:17:24.344 INFO:teuthology.orchestra.run.smithi118.stdout: "max": "", 2024-03-24T03:17:24.344 INFO:teuthology.orchestra.run.smithi118.stdout: "enum_allowed": [], 2024-03-24T03:17:24.345 INFO:teuthology.orchestra.run.smithi118.stdout: "desc": "Do not include perf-counters in the metrics output", 2024-03-24T03:17:24.345 INFO:teuthology.orchestra.run.smithi118.stdout: "long_desc": "Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.", 2024-03-24T03:17:24.345 INFO:teuthology.orchestra.run.smithi118.stdout: "tags": [], 2024-03-24T03:17:24.345 INFO:teuthology.orchestra.run.smithi118.stdout: "see_also": [] 2024-03-24T03:17:24.345 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-24T03:17:24.345 INFO:teuthology.orchestra.run.smithi118.stdout: "log_level": { 2024-03-24T03:17:24.345 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "log_level", 2024-03-24T03:17:24.345 INFO:teuthology.orchestra.run.smithi118.stdout: "type": "str", 2024-03-24T03:17:24.346 INFO:teuthology.orchestra.run.smithi118.stdout: "level": "advanced", 2024-03-24T03:17:24.346 INFO:teuthology.orchestra.run.smithi118.stdout: "flags": 1, 2024-03-24T03:17:24.346 INFO:teuthology.orchestra.run.smithi118.stdout: "default_value": "", 2024-03-24T03:17:24.346 INFO:teuthology.orchestra.run.smithi118.stdout: "min": "", 2024-03-24T03:17:24.346 INFO:teuthology.orchestra.run.smithi118.stdout: "max": "", 2024-03-24T03:17:24.346 INFO:teuthology.orchestra.run.smithi118.stdout: "enum_allowed": [ 2024-03-24T03:17:24.346 INFO:teuthology.orchestra.run.smithi118.stdout: "", 2024-03-24T03:17:24.346 INFO:teuthology.orchestra.run.smithi118.stdout: "critical", 2024-03-24T03:17:24.346 INFO:teuthology.orchestra.run.smithi118.stdout: "debug", 2024-03-24T03:17:24.347 INFO:teuthology.orchestra.run.smithi118.stdout: "error", 2024-03-24T03:17:24.347 INFO:teuthology.orchestra.run.smithi118.stdout: "info", 2024-03-24T03:17:24.347 INFO:teuthology.orchestra.run.smithi118.stdout: "warning" 2024-03-24T03:17:24.347 INFO:teuthology.orchestra.run.smithi118.stdout: ], 2024-03-24T03:17:24.347 INFO:teuthology.orchestra.run.smithi118.stdout: "desc": "", 2024-03-24T03:17:24.347 INFO:teuthology.orchestra.run.smithi118.stdout: "long_desc": "", 2024-03-24T03:17:24.347 INFO:teuthology.orchestra.run.smithi118.stdout: "tags": [], 2024-03-24T03:17:24.347 INFO:teuthology.orchestra.run.smithi118.stdout: "see_also": [] 2024-03-24T03:17:24.347 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-24T03:17:24.348 INFO:teuthology.orchestra.run.smithi118.stdout: "log_to_cluster": { 2024-03-24T03:17:24.348 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "log_to_cluster", 2024-03-24T03:17:24.348 INFO:teuthology.orchestra.run.smithi118.stdout: "type": "bool", 2024-03-24T03:17:24.348 INFO:teuthology.orchestra.run.smithi118.stdout: "level": "advanced", 2024-03-24T03:17:24.348 INFO:teuthology.orchestra.run.smithi118.stdout: "flags": 1, 2024-03-24T03:17:24.348 INFO:teuthology.orchestra.run.smithi118.stdout: "default_value": "False", 2024-03-24T03:17:24.348 INFO:teuthology.orchestra.run.smithi118.stdout: "min": "", 2024-03-24T03:17:24.348 INFO:teuthology.orchestra.run.smithi118.stdout: "max": "", 2024-03-24T03:17:24.348 INFO:teuthology.orchestra.run.smithi118.stdout: "enum_allowed": [], 2024-03-24T03:17:24.348 INFO:teuthology.orchestra.run.smithi118.stdout: "desc": "", 2024-03-24T03:17:24.349 INFO:teuthology.orchestra.run.smithi118.stdout: "long_desc": "", 2024-03-24T03:17:24.349 INFO:teuthology.orchestra.run.smithi118.stdout: "tags": [], 2024-03-24T03:17:24.349 INFO:teuthology.orchestra.run.smithi118.stdout: "see_also": [] 2024-03-24T03:17:24.349 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-24T03:17:24.349 INFO:teuthology.orchestra.run.smithi118.stdout: "log_to_cluster_level": { 2024-03-24T03:17:24.349 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "log_to_cluster_level", 2024-03-24T03:17:24.349 INFO:teuthology.orchestra.run.smithi118.stdout: "type": "str", 2024-03-24T03:17:24.349 INFO:teuthology.orchestra.run.smithi118.stdout: "level": "advanced", 2024-03-24T03:17:24.350 INFO:teuthology.orchestra.run.smithi118.stdout: "flags": 1, 2024-03-24T03:17:24.350 INFO:teuthology.orchestra.run.smithi118.stdout: "default_value": "info", 2024-03-24T03:17:24.350 INFO:teuthology.orchestra.run.smithi118.stdout: "min": "", 2024-03-24T03:17:24.350 INFO:teuthology.orchestra.run.smithi118.stdout: "max": "", 2024-03-24T03:17:24.350 INFO:teuthology.orchestra.run.smithi118.stdout: "enum_allowed": [ 2024-03-24T03:17:24.350 INFO:teuthology.orchestra.run.smithi118.stdout: "", 2024-03-24T03:17:24.350 INFO:teuthology.orchestra.run.smithi118.stdout: "critical", 2024-03-24T03:17:24.350 INFO:teuthology.orchestra.run.smithi118.stdout: "debug", 2024-03-24T03:17:24.351 INFO:teuthology.orchestra.run.smithi118.stdout: "error", 2024-03-24T03:17:24.351 INFO:teuthology.orchestra.run.smithi118.stdout: "info", 2024-03-24T03:17:24.351 INFO:teuthology.orchestra.run.smithi118.stdout: "warning" 2024-03-24T03:17:24.351 INFO:teuthology.orchestra.run.smithi118.stdout: ], 2024-03-24T03:17:24.351 INFO:teuthology.orchestra.run.smithi118.stdout: "desc": "", 2024-03-24T03:17:24.351 INFO:teuthology.orchestra.run.smithi118.stdout: "long_desc": "", 2024-03-24T03:17:24.351 INFO:teuthology.orchestra.run.smithi118.stdout: "tags": [], 2024-03-24T03:17:24.351 INFO:teuthology.orchestra.run.smithi118.stdout: "see_also": [] 2024-03-24T03:17:24.351 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-24T03:17:24.351 INFO:teuthology.orchestra.run.smithi118.stdout: "log_to_file": { 2024-03-24T03:17:24.352 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "log_to_file", 2024-03-24T03:17:24.352 INFO:teuthology.orchestra.run.smithi118.stdout: "type": "bool", 2024-03-24T03:17:24.352 INFO:teuthology.orchestra.run.smithi118.stdout: "level": "advanced", 2024-03-24T03:17:24.352 INFO:teuthology.orchestra.run.smithi118.stdout: "flags": 1, 2024-03-24T03:17:24.352 INFO:teuthology.orchestra.run.smithi118.stdout: "default_value": "False", 2024-03-24T03:17:24.352 INFO:teuthology.orchestra.run.smithi118.stdout: "min": "", 2024-03-24T03:17:24.352 INFO:teuthology.orchestra.run.smithi118.stdout: "max": "", 2024-03-24T03:17:24.352 INFO:teuthology.orchestra.run.smithi118.stdout: "enum_allowed": [], 2024-03-24T03:17:24.352 INFO:teuthology.orchestra.run.smithi118.stdout: "desc": "", 2024-03-24T03:17:24.353 INFO:teuthology.orchestra.run.smithi118.stdout: "long_desc": "", 2024-03-24T03:17:24.353 INFO:teuthology.orchestra.run.smithi118.stdout: "tags": [], 2024-03-24T03:17:24.353 INFO:teuthology.orchestra.run.smithi118.stdout: "see_also": [] 2024-03-24T03:17:24.353 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-24T03:17:24.353 INFO:teuthology.orchestra.run.smithi118.stdout: "rbd_stats_pools": { 2024-03-24T03:17:24.353 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "rbd_stats_pools", 2024-03-24T03:17:24.353 INFO:teuthology.orchestra.run.smithi118.stdout: "type": "str", 2024-03-24T03:17:24.353 INFO:teuthology.orchestra.run.smithi118.stdout: "level": "advanced", 2024-03-24T03:17:24.353 INFO:teuthology.orchestra.run.smithi118.stdout: "flags": 0, 2024-03-24T03:17:24.354 INFO:teuthology.orchestra.run.smithi118.stdout: "default_value": "", 2024-03-24T03:17:24.354 INFO:teuthology.orchestra.run.smithi118.stdout: "min": "", 2024-03-24T03:17:24.354 INFO:teuthology.orchestra.run.smithi118.stdout: "max": "", 2024-03-24T03:17:24.354 INFO:teuthology.orchestra.run.smithi118.stdout: "enum_allowed": [], 2024-03-24T03:17:24.354 INFO:teuthology.orchestra.run.smithi118.stdout: "desc": "", 2024-03-24T03:17:24.354 INFO:teuthology.orchestra.run.smithi118.stdout: "long_desc": "", 2024-03-24T03:17:24.354 INFO:teuthology.orchestra.run.smithi118.stdout: "tags": [], 2024-03-24T03:17:24.355 INFO:teuthology.orchestra.run.smithi118.stdout: "see_also": [] 2024-03-24T03:17:24.355 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-24T03:17:24.355 INFO:teuthology.orchestra.run.smithi118.stdout: "rbd_stats_pools_refresh_interval": { 2024-03-24T03:17:24.355 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "rbd_stats_pools_refresh_interval", 2024-03-24T03:17:24.355 INFO:teuthology.orchestra.run.smithi118.stdout: "type": "int", 2024-03-24T03:17:24.355 INFO:teuthology.orchestra.run.smithi118.stdout: "level": "advanced", 2024-03-24T03:17:24.355 INFO:teuthology.orchestra.run.smithi118.stdout: "flags": 0, 2024-03-24T03:17:24.355 INFO:teuthology.orchestra.run.smithi118.stdout: "default_value": "300", 2024-03-24T03:17:24.356 INFO:teuthology.orchestra.run.smithi118.stdout: "min": "", 2024-03-24T03:17:24.356 INFO:teuthology.orchestra.run.smithi118.stdout: "max": "", 2024-03-24T03:17:24.356 INFO:teuthology.orchestra.run.smithi118.stdout: "enum_allowed": [], 2024-03-24T03:17:24.356 INFO:teuthology.orchestra.run.smithi118.stdout: "desc": "", 2024-03-24T03:17:24.356 INFO:teuthology.orchestra.run.smithi118.stdout: "long_desc": "", 2024-03-24T03:17:24.356 INFO:teuthology.orchestra.run.smithi118.stdout: "tags": [], 2024-03-24T03:17:24.356 INFO:teuthology.orchestra.run.smithi118.stdout: "see_also": [] 2024-03-24T03:17:24.356 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-24T03:17:24.357 INFO:teuthology.orchestra.run.smithi118.stdout: "scrape_interval": { 2024-03-24T03:17:24.357 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "scrape_interval", 2024-03-24T03:17:24.357 INFO:teuthology.orchestra.run.smithi118.stdout: "type": "float", 2024-03-24T03:17:24.357 INFO:teuthology.orchestra.run.smithi118.stdout: "level": "advanced", 2024-03-24T03:17:24.357 INFO:teuthology.orchestra.run.smithi118.stdout: "flags": 0, 2024-03-24T03:17:24.357 INFO:teuthology.orchestra.run.smithi118.stdout: "default_value": "15.0", 2024-03-24T03:17:24.357 INFO:teuthology.orchestra.run.smithi118.stdout: "min": "", 2024-03-24T03:17:24.357 INFO:teuthology.orchestra.run.smithi118.stdout: "max": "", 2024-03-24T03:17:24.358 INFO:teuthology.orchestra.run.smithi118.stdout: "enum_allowed": [], 2024-03-24T03:17:24.358 INFO:teuthology.orchestra.run.smithi118.stdout: "desc": "", 2024-03-24T03:17:24.358 INFO:teuthology.orchestra.run.smithi118.stdout: "long_desc": "", 2024-03-24T03:17:24.358 INFO:teuthology.orchestra.run.smithi118.stdout: "tags": [], 2024-03-24T03:17:24.358 INFO:teuthology.orchestra.run.smithi118.stdout: "see_also": [] 2024-03-24T03:17:24.358 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-24T03:17:24.358 INFO:teuthology.orchestra.run.smithi118.stdout: "server_addr": { 2024-03-24T03:17:24.358 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "server_addr", 2024-03-24T03:17:24.358 INFO:teuthology.orchestra.run.smithi118.stdout: "type": "str", 2024-03-24T03:17:24.359 INFO:teuthology.orchestra.run.smithi118.stdout: "level": "advanced", 2024-03-24T03:17:24.359 INFO:teuthology.orchestra.run.smithi118.stdout: "flags": 0, 2024-03-24T03:17:24.359 INFO:teuthology.orchestra.run.smithi118.stdout: "default_value": "::", 2024-03-24T03:17:24.359 INFO:teuthology.orchestra.run.smithi118.stdout: "min": "", 2024-03-24T03:17:24.359 INFO:teuthology.orchestra.run.smithi118.stdout: "max": "", 2024-03-24T03:17:24.359 INFO:teuthology.orchestra.run.smithi118.stdout: "enum_allowed": [], 2024-03-24T03:17:24.359 INFO:teuthology.orchestra.run.smithi118.stdout: "desc": "the IPv4 or IPv6 address on which the module listens for HTTP requests", 2024-03-24T03:17:24.359 INFO:teuthology.orchestra.run.smithi118.stdout: "long_desc": "", 2024-03-24T03:17:24.360 INFO:teuthology.orchestra.run.smithi118.stdout: "tags": [], 2024-03-24T03:17:24.360 INFO:teuthology.orchestra.run.smithi118.stdout: "see_also": [] 2024-03-24T03:17:24.360 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-24T03:17:24.360 INFO:teuthology.orchestra.run.smithi118.stdout: "server_port": { 2024-03-24T03:17:24.360 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "server_port", 2024-03-24T03:17:24.360 INFO:teuthology.orchestra.run.smithi118.stdout: "type": "int", 2024-03-24T03:17:24.360 INFO:teuthology.orchestra.run.smithi118.stdout: "level": "advanced", 2024-03-24T03:17:24.360 INFO:teuthology.orchestra.run.smithi118.stdout: "flags": 1, 2024-03-24T03:17:24.360 INFO:teuthology.orchestra.run.smithi118.stdout: "default_value": "9283", 2024-03-24T03:17:24.361 INFO:teuthology.orchestra.run.smithi118.stdout: "min": "", 2024-03-24T03:17:24.361 INFO:teuthology.orchestra.run.smithi118.stdout: "max": "", 2024-03-24T03:17:24.361 INFO:teuthology.orchestra.run.smithi118.stdout: "enum_allowed": [], 2024-03-24T03:17:24.361 INFO:teuthology.orchestra.run.smithi118.stdout: "desc": "the port on which the module listens for HTTP requests", 2024-03-24T03:17:24.361 INFO:teuthology.orchestra.run.smithi118.stdout: "long_desc": "", 2024-03-24T03:17:24.361 INFO:teuthology.orchestra.run.smithi118.stdout: "tags": [], 2024-03-24T03:17:24.361 INFO:teuthology.orchestra.run.smithi118.stdout: "see_also": [] 2024-03-24T03:17:24.361 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-24T03:17:24.362 INFO:teuthology.orchestra.run.smithi118.stdout: "stale_cache_strategy": { 2024-03-24T03:17:24.362 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "stale_cache_strategy", 2024-03-24T03:17:24.362 INFO:teuthology.orchestra.run.smithi118.stdout: "type": "str", 2024-03-24T03:17:24.362 INFO:teuthology.orchestra.run.smithi118.stdout: "level": "advanced", 2024-03-24T03:17:24.362 INFO:teuthology.orchestra.run.smithi118.stdout: "flags": 0, 2024-03-24T03:17:24.362 INFO:teuthology.orchestra.run.smithi118.stdout: "default_value": "log", 2024-03-24T03:17:24.362 INFO:teuthology.orchestra.run.smithi118.stdout: "min": "", 2024-03-24T03:17:24.362 INFO:teuthology.orchestra.run.smithi118.stdout: "max": "", 2024-03-24T03:17:24.363 INFO:teuthology.orchestra.run.smithi118.stdout: "enum_allowed": [], 2024-03-24T03:17:24.363 INFO:teuthology.orchestra.run.smithi118.stdout: "desc": "", 2024-03-24T03:17:24.363 INFO:teuthology.orchestra.run.smithi118.stdout: "long_desc": "", 2024-03-24T03:17:24.363 INFO:teuthology.orchestra.run.smithi118.stdout: "tags": [], 2024-03-24T03:17:24.363 INFO:teuthology.orchestra.run.smithi118.stdout: "see_also": [] 2024-03-24T03:17:24.363 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-24T03:17:24.363 INFO:teuthology.orchestra.run.smithi118.stdout: "standby_behaviour": { 2024-03-24T03:17:24.363 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "standby_behaviour", 2024-03-24T03:17:24.364 INFO:teuthology.orchestra.run.smithi118.stdout: "type": "str", 2024-03-24T03:17:24.364 INFO:teuthology.orchestra.run.smithi118.stdout: "level": "advanced", 2024-03-24T03:17:24.364 INFO:teuthology.orchestra.run.smithi118.stdout: "flags": 1, 2024-03-24T03:17:24.364 INFO:teuthology.orchestra.run.smithi118.stdout: "default_value": "default", 2024-03-24T03:17:24.364 INFO:teuthology.orchestra.run.smithi118.stdout: "min": "", 2024-03-24T03:17:24.364 INFO:teuthology.orchestra.run.smithi118.stdout: "max": "", 2024-03-24T03:17:24.364 INFO:teuthology.orchestra.run.smithi118.stdout: "enum_allowed": [ 2024-03-24T03:17:24.364 INFO:teuthology.orchestra.run.smithi118.stdout: "default", 2024-03-24T03:17:24.364 INFO:teuthology.orchestra.run.smithi118.stdout: "error" 2024-03-24T03:17:24.365 INFO:teuthology.orchestra.run.smithi118.stdout: ], 2024-03-24T03:17:24.365 INFO:teuthology.orchestra.run.smithi118.stdout: "desc": "", 2024-03-24T03:17:24.365 INFO:teuthology.orchestra.run.smithi118.stdout: "long_desc": "", 2024-03-24T03:17:24.365 INFO:teuthology.orchestra.run.smithi118.stdout: "tags": [], 2024-03-24T03:17:24.365 INFO:teuthology.orchestra.run.smithi118.stdout: "see_also": [] 2024-03-24T03:17:24.365 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-24T03:17:24.365 INFO:teuthology.orchestra.run.smithi118.stdout: "standby_error_status_code": { 2024-03-24T03:17:24.365 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "standby_error_status_code", 2024-03-24T03:17:24.365 INFO:teuthology.orchestra.run.smithi118.stdout: "type": "int", 2024-03-24T03:17:24.366 INFO:teuthology.orchestra.run.smithi118.stdout: "level": "advanced", 2024-03-24T03:17:24.366 INFO:teuthology.orchestra.run.smithi118.stdout: "flags": 1, 2024-03-24T03:17:24.366 INFO:teuthology.orchestra.run.smithi118.stdout: "default_value": "500", 2024-03-24T03:17:24.366 INFO:teuthology.orchestra.run.smithi118.stdout: "min": "400", 2024-03-24T03:17:24.366 INFO:teuthology.orchestra.run.smithi118.stdout: "max": "599", 2024-03-24T03:17:24.366 INFO:teuthology.orchestra.run.smithi118.stdout: "enum_allowed": [], 2024-03-24T03:17:24.366 INFO:teuthology.orchestra.run.smithi118.stdout: "desc": "", 2024-03-24T03:17:24.366 INFO:teuthology.orchestra.run.smithi118.stdout: "long_desc": "", 2024-03-24T03:17:24.366 INFO:teuthology.orchestra.run.smithi118.stdout: "tags": [], 2024-03-24T03:17:24.367 INFO:teuthology.orchestra.run.smithi118.stdout: "see_also": [] 2024-03-24T03:17:24.367 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-03-24T03:17:24.367 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-03-24T03:17:24.367 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-24T03:17:24.367 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-24T03:17:24.367 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "rbd_support", 2024-03-24T03:17:24.367 INFO:teuthology.orchestra.run.smithi118.stdout: "can_run": true, 2024-03-24T03:17:24.367 INFO:teuthology.orchestra.run.smithi118.stdout: "error_string": "", 2024-03-24T03:17:24.368 INFO:teuthology.orchestra.run.smithi118.stdout: "module_options": { 2024-03-24T03:17:24.368 INFO:teuthology.orchestra.run.smithi118.stdout: "log_level": { 2024-03-24T03:17:24.368 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "log_level", 2024-03-24T03:17:24.368 INFO:teuthology.orchestra.run.smithi118.stdout: "type": "str", 2024-03-24T03:17:24.368 INFO:teuthology.orchestra.run.smithi118.stdout: "level": "advanced", 2024-03-24T03:17:24.368 INFO:teuthology.orchestra.run.smithi118.stdout: "flags": 1, 2024-03-24T03:17:24.368 INFO:teuthology.orchestra.run.smithi118.stdout: "default_value": "", 2024-03-24T03:17:24.368 INFO:teuthology.orchestra.run.smithi118.stdout: "min": "", 2024-03-24T03:17:24.368 INFO:teuthology.orchestra.run.smithi118.stdout: "max": "", 2024-03-24T03:17:24.369 INFO:teuthology.orchestra.run.smithi118.stdout: "enum_allowed": [ 2024-03-24T03:17:24.369 INFO:teuthology.orchestra.run.smithi118.stdout: "", 2024-03-24T03:17:24.369 INFO:teuthology.orchestra.run.smithi118.stdout: "critical", 2024-03-24T03:17:24.369 INFO:teuthology.orchestra.run.smithi118.stdout: "debug", 2024-03-24T03:17:24.369 INFO:teuthology.orchestra.run.smithi118.stdout: "error", 2024-03-24T03:17:24.369 INFO:teuthology.orchestra.run.smithi118.stdout: "info", 2024-03-24T03:17:24.369 INFO:teuthology.orchestra.run.smithi118.stdout: "warning" 2024-03-24T03:17:24.369 INFO:teuthology.orchestra.run.smithi118.stdout: ], 2024-03-24T03:17:24.369 INFO:teuthology.orchestra.run.smithi118.stdout: "desc": "", 2024-03-24T03:17:24.370 INFO:teuthology.orchestra.run.smithi118.stdout: "long_desc": "", 2024-03-24T03:17:24.370 INFO:teuthology.orchestra.run.smithi118.stdout: "tags": [], 2024-03-24T03:17:24.370 INFO:teuthology.orchestra.run.smithi118.stdout: "see_also": [] 2024-03-24T03:17:24.370 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-24T03:17:24.370 INFO:teuthology.orchestra.run.smithi118.stdout: "log_to_cluster": { 2024-03-24T03:17:24.370 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "log_to_cluster", 2024-03-24T03:17:24.370 INFO:teuthology.orchestra.run.smithi118.stdout: "type": "bool", 2024-03-24T03:17:24.370 INFO:teuthology.orchestra.run.smithi118.stdout: "level": "advanced", 2024-03-24T03:17:24.371 INFO:teuthology.orchestra.run.smithi118.stdout: "flags": 1, 2024-03-24T03:17:24.371 INFO:teuthology.orchestra.run.smithi118.stdout: "default_value": "False", 2024-03-24T03:17:24.371 INFO:teuthology.orchestra.run.smithi118.stdout: "min": "", 2024-03-24T03:17:24.371 INFO:teuthology.orchestra.run.smithi118.stdout: "max": "", 2024-03-24T03:17:24.371 INFO:teuthology.orchestra.run.smithi118.stdout: "enum_allowed": [], 2024-03-24T03:17:24.371 INFO:teuthology.orchestra.run.smithi118.stdout: "desc": "", 2024-03-24T03:17:24.371 INFO:teuthology.orchestra.run.smithi118.stdout: "long_desc": "", 2024-03-24T03:17:24.371 INFO:teuthology.orchestra.run.smithi118.stdout: "tags": [], 2024-03-24T03:17:24.372 INFO:teuthology.orchestra.run.smithi118.stdout: "see_also": [] 2024-03-24T03:17:24.372 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-24T03:17:24.372 INFO:teuthology.orchestra.run.smithi118.stdout: "log_to_cluster_level": { 2024-03-24T03:17:24.372 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "log_to_cluster_level", 2024-03-24T03:17:24.372 INFO:teuthology.orchestra.run.smithi118.stdout: "type": "str", 2024-03-24T03:17:24.372 INFO:teuthology.orchestra.run.smithi118.stdout: "level": "advanced", 2024-03-24T03:17:24.372 INFO:teuthology.orchestra.run.smithi118.stdout: "flags": 1, 2024-03-24T03:17:24.372 INFO:teuthology.orchestra.run.smithi118.stdout: "default_value": "info", 2024-03-24T03:17:24.372 INFO:teuthology.orchestra.run.smithi118.stdout: "min": "", 2024-03-24T03:17:24.373 INFO:teuthology.orchestra.run.smithi118.stdout: "max": "", 2024-03-24T03:17:24.373 INFO:teuthology.orchestra.run.smithi118.stdout: "enum_allowed": [ 2024-03-24T03:17:24.373 INFO:teuthology.orchestra.run.smithi118.stdout: "", 2024-03-24T03:17:24.373 INFO:teuthology.orchestra.run.smithi118.stdout: "critical", 2024-03-24T03:17:24.373 INFO:teuthology.orchestra.run.smithi118.stdout: "debug", 2024-03-24T03:17:24.373 INFO:teuthology.orchestra.run.smithi118.stdout: "error", 2024-03-24T03:17:24.373 INFO:teuthology.orchestra.run.smithi118.stdout: "info", 2024-03-24T03:17:24.373 INFO:teuthology.orchestra.run.smithi118.stdout: "warning" 2024-03-24T03:17:24.374 INFO:teuthology.orchestra.run.smithi118.stdout: ], 2024-03-24T03:17:24.374 INFO:teuthology.orchestra.run.smithi118.stdout: "desc": "", 2024-03-24T03:17:24.374 INFO:teuthology.orchestra.run.smithi118.stdout: "long_desc": "", 2024-03-24T03:17:24.374 INFO:teuthology.orchestra.run.smithi118.stdout: "tags": [], 2024-03-24T03:17:24.374 INFO:teuthology.orchestra.run.smithi118.stdout: "see_also": [] 2024-03-24T03:17:24.374 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-24T03:17:24.374 INFO:teuthology.orchestra.run.smithi118.stdout: "log_to_file": { 2024-03-24T03:17:24.374 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "log_to_file", 2024-03-24T03:17:24.374 INFO:teuthology.orchestra.run.smithi118.stdout: "type": "bool", 2024-03-24T03:17:24.375 INFO:teuthology.orchestra.run.smithi118.stdout: "level": "advanced", 2024-03-24T03:17:24.375 INFO:teuthology.orchestra.run.smithi118.stdout: "flags": 1, 2024-03-24T03:17:24.375 INFO:teuthology.orchestra.run.smithi118.stdout: "default_value": "False", 2024-03-24T03:17:24.375 INFO:teuthology.orchestra.run.smithi118.stdout: "min": "", 2024-03-24T03:17:24.375 INFO:teuthology.orchestra.run.smithi118.stdout: "max": "", 2024-03-24T03:17:24.375 INFO:teuthology.orchestra.run.smithi118.stdout: "enum_allowed": [], 2024-03-24T03:17:24.375 INFO:teuthology.orchestra.run.smithi118.stdout: "desc": "", 2024-03-24T03:17:24.375 INFO:teuthology.orchestra.run.smithi118.stdout: "long_desc": "", 2024-03-24T03:17:24.376 INFO:teuthology.orchestra.run.smithi118.stdout: "tags": [], 2024-03-24T03:17:24.376 INFO:teuthology.orchestra.run.smithi118.stdout: "see_also": [] 2024-03-24T03:17:24.376 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-24T03:17:24.376 INFO:teuthology.orchestra.run.smithi118.stdout: "max_concurrent_snap_create": { 2024-03-24T03:17:24.376 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "max_concurrent_snap_create", 2024-03-24T03:17:24.376 INFO:teuthology.orchestra.run.smithi118.stdout: "type": "int", 2024-03-24T03:17:24.376 INFO:teuthology.orchestra.run.smithi118.stdout: "level": "advanced", 2024-03-24T03:17:24.376 INFO:teuthology.orchestra.run.smithi118.stdout: "flags": 0, 2024-03-24T03:17:24.376 INFO:teuthology.orchestra.run.smithi118.stdout: "default_value": "10", 2024-03-24T03:17:24.377 INFO:teuthology.orchestra.run.smithi118.stdout: "min": "", 2024-03-24T03:17:24.377 INFO:teuthology.orchestra.run.smithi118.stdout: "max": "", 2024-03-24T03:17:24.377 INFO:teuthology.orchestra.run.smithi118.stdout: "enum_allowed": [], 2024-03-24T03:17:24.377 INFO:teuthology.orchestra.run.smithi118.stdout: "desc": "", 2024-03-24T03:17:24.377 INFO:teuthology.orchestra.run.smithi118.stdout: "long_desc": "", 2024-03-24T03:17:24.377 INFO:teuthology.orchestra.run.smithi118.stdout: "tags": [], 2024-03-24T03:17:24.377 INFO:teuthology.orchestra.run.smithi118.stdout: "see_also": [] 2024-03-24T03:17:24.377 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-24T03:17:24.377 INFO:teuthology.orchestra.run.smithi118.stdout: "mirror_snapshot_schedule": { 2024-03-24T03:17:24.378 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "mirror_snapshot_schedule", 2024-03-24T03:17:24.378 INFO:teuthology.orchestra.run.smithi118.stdout: "type": "str", 2024-03-24T03:17:24.378 INFO:teuthology.orchestra.run.smithi118.stdout: "level": "advanced", 2024-03-24T03:17:24.378 INFO:teuthology.orchestra.run.smithi118.stdout: "flags": 0, 2024-03-24T03:17:24.378 INFO:teuthology.orchestra.run.smithi118.stdout: "default_value": "", 2024-03-24T03:17:24.378 INFO:teuthology.orchestra.run.smithi118.stdout: "min": "", 2024-03-24T03:17:24.378 INFO:teuthology.orchestra.run.smithi118.stdout: "max": "", 2024-03-24T03:17:24.378 INFO:teuthology.orchestra.run.smithi118.stdout: "enum_allowed": [], 2024-03-24T03:17:24.379 INFO:teuthology.orchestra.run.smithi118.stdout: "desc": "", 2024-03-24T03:17:24.379 INFO:teuthology.orchestra.run.smithi118.stdout: "long_desc": "", 2024-03-24T03:17:24.379 INFO:teuthology.orchestra.run.smithi118.stdout: "tags": [], 2024-03-24T03:17:24.379 INFO:teuthology.orchestra.run.smithi118.stdout: "see_also": [] 2024-03-24T03:17:24.379 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-24T03:17:24.379 INFO:teuthology.orchestra.run.smithi118.stdout: "trash_purge_schedule": { 2024-03-24T03:17:24.379 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "trash_purge_schedule", 2024-03-24T03:17:24.379 INFO:teuthology.orchestra.run.smithi118.stdout: "type": "str", 2024-03-24T03:17:24.379 INFO:teuthology.orchestra.run.smithi118.stdout: "level": "advanced", 2024-03-24T03:17:24.380 INFO:teuthology.orchestra.run.smithi118.stdout: "flags": 0, 2024-03-24T03:17:24.380 INFO:teuthology.orchestra.run.smithi118.stdout: "default_value": "", 2024-03-24T03:17:24.380 INFO:teuthology.orchestra.run.smithi118.stdout: "min": "", 2024-03-24T03:17:24.380 INFO:teuthology.orchestra.run.smithi118.stdout: "max": "", 2024-03-24T03:17:24.380 INFO:teuthology.orchestra.run.smithi118.stdout: "enum_allowed": [], 2024-03-24T03:17:24.380 INFO:teuthology.orchestra.run.smithi118.stdout: "desc": "", 2024-03-24T03:17:24.380 INFO:teuthology.orchestra.run.smithi118.stdout: "long_desc": "", 2024-03-24T03:17:24.380 INFO:teuthology.orchestra.run.smithi118.stdout: "tags": [], 2024-03-24T03:17:24.380 INFO:teuthology.orchestra.run.smithi118.stdout: "see_also": [] 2024-03-24T03:17:24.380 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-03-24T03:17:24.381 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-03-24T03:17:24.381 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-24T03:17:24.381 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-24T03:17:24.381 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "restful", 2024-03-24T03:17:24.381 INFO:teuthology.orchestra.run.smithi118.stdout: "can_run": true, 2024-03-24T03:17:24.381 INFO:teuthology.orchestra.run.smithi118.stdout: "error_string": "", 2024-03-24T03:17:24.381 INFO:teuthology.orchestra.run.smithi118.stdout: "module_options": { 2024-03-24T03:17:24.381 INFO:teuthology.orchestra.run.smithi118.stdout: "enable_auth": { 2024-03-24T03:17:24.381 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "enable_auth", 2024-03-24T03:17:24.382 INFO:teuthology.orchestra.run.smithi118.stdout: "type": "bool", 2024-03-24T03:17:24.382 INFO:teuthology.orchestra.run.smithi118.stdout: "level": "advanced", 2024-03-24T03:17:24.382 INFO:teuthology.orchestra.run.smithi118.stdout: "flags": 0, 2024-03-24T03:17:24.382 INFO:teuthology.orchestra.run.smithi118.stdout: "default_value": "True", 2024-03-24T03:17:24.382 INFO:teuthology.orchestra.run.smithi118.stdout: "min": "", 2024-03-24T03:17:24.382 INFO:teuthology.orchestra.run.smithi118.stdout: "max": "", 2024-03-24T03:17:24.382 INFO:teuthology.orchestra.run.smithi118.stdout: "enum_allowed": [], 2024-03-24T03:17:24.382 INFO:teuthology.orchestra.run.smithi118.stdout: "desc": "", 2024-03-24T03:17:24.383 INFO:teuthology.orchestra.run.smithi118.stdout: "long_desc": "", 2024-03-24T03:17:24.383 INFO:teuthology.orchestra.run.smithi118.stdout: "tags": [], 2024-03-24T03:17:24.383 INFO:teuthology.orchestra.run.smithi118.stdout: "see_also": [] 2024-03-24T03:17:24.383 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-24T03:17:24.383 INFO:teuthology.orchestra.run.smithi118.stdout: "key_file": { 2024-03-24T03:17:24.383 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "key_file", 2024-03-24T03:17:24.383 INFO:teuthology.orchestra.run.smithi118.stdout: "type": "str", 2024-03-24T03:17:24.383 INFO:teuthology.orchestra.run.smithi118.stdout: "level": "advanced", 2024-03-24T03:17:24.383 INFO:teuthology.orchestra.run.smithi118.stdout: "flags": 0, 2024-03-24T03:17:24.384 INFO:teuthology.orchestra.run.smithi118.stdout: "default_value": "", 2024-03-24T03:17:24.384 INFO:teuthology.orchestra.run.smithi118.stdout: "min": "", 2024-03-24T03:17:24.384 INFO:teuthology.orchestra.run.smithi118.stdout: "max": "", 2024-03-24T03:17:24.384 INFO:teuthology.orchestra.run.smithi118.stdout: "enum_allowed": [], 2024-03-24T03:17:24.384 INFO:teuthology.orchestra.run.smithi118.stdout: "desc": "", 2024-03-24T03:17:24.384 INFO:teuthology.orchestra.run.smithi118.stdout: "long_desc": "", 2024-03-24T03:17:24.384 INFO:teuthology.orchestra.run.smithi118.stdout: "tags": [], 2024-03-24T03:17:24.384 INFO:teuthology.orchestra.run.smithi118.stdout: "see_also": [] 2024-03-24T03:17:24.385 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-24T03:17:24.385 INFO:teuthology.orchestra.run.smithi118.stdout: "log_level": { 2024-03-24T03:17:24.385 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "log_level", 2024-03-24T03:17:24.385 INFO:teuthology.orchestra.run.smithi118.stdout: "type": "str", 2024-03-24T03:17:24.385 INFO:teuthology.orchestra.run.smithi118.stdout: "level": "advanced", 2024-03-24T03:17:24.385 INFO:teuthology.orchestra.run.smithi118.stdout: "flags": 1, 2024-03-24T03:17:24.385 INFO:teuthology.orchestra.run.smithi118.stdout: "default_value": "", 2024-03-24T03:17:24.385 INFO:teuthology.orchestra.run.smithi118.stdout: "min": "", 2024-03-24T03:17:24.385 INFO:teuthology.orchestra.run.smithi118.stdout: "max": "", 2024-03-24T03:17:24.386 INFO:teuthology.orchestra.run.smithi118.stdout: "enum_allowed": [ 2024-03-24T03:17:24.386 INFO:teuthology.orchestra.run.smithi118.stdout: "", 2024-03-24T03:17:24.386 INFO:teuthology.orchestra.run.smithi118.stdout: "critical", 2024-03-24T03:17:24.386 INFO:teuthology.orchestra.run.smithi118.stdout: "debug", 2024-03-24T03:17:24.386 INFO:teuthology.orchestra.run.smithi118.stdout: "error", 2024-03-24T03:17:24.386 INFO:teuthology.orchestra.run.smithi118.stdout: "info", 2024-03-24T03:17:24.386 INFO:teuthology.orchestra.run.smithi118.stdout: "warning" 2024-03-24T03:17:24.386 INFO:teuthology.orchestra.run.smithi118.stdout: ], 2024-03-24T03:17:24.387 INFO:teuthology.orchestra.run.smithi118.stdout: "desc": "", 2024-03-24T03:17:24.387 INFO:teuthology.orchestra.run.smithi118.stdout: "long_desc": "", 2024-03-24T03:17:24.387 INFO:teuthology.orchestra.run.smithi118.stdout: "tags": [], 2024-03-24T03:17:24.387 INFO:teuthology.orchestra.run.smithi118.stdout: "see_also": [] 2024-03-24T03:17:24.387 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-24T03:17:24.387 INFO:teuthology.orchestra.run.smithi118.stdout: "log_to_cluster": { 2024-03-24T03:17:24.387 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "log_to_cluster", 2024-03-24T03:17:24.387 INFO:teuthology.orchestra.run.smithi118.stdout: "type": "bool", 2024-03-24T03:17:24.387 INFO:teuthology.orchestra.run.smithi118.stdout: "level": "advanced", 2024-03-24T03:17:24.388 INFO:teuthology.orchestra.run.smithi118.stdout: "flags": 1, 2024-03-24T03:17:24.388 INFO:teuthology.orchestra.run.smithi118.stdout: "default_value": "False", 2024-03-24T03:17:24.388 INFO:teuthology.orchestra.run.smithi118.stdout: "min": "", 2024-03-24T03:17:24.388 INFO:teuthology.orchestra.run.smithi118.stdout: "max": "", 2024-03-24T03:17:24.388 INFO:teuthology.orchestra.run.smithi118.stdout: "enum_allowed": [], 2024-03-24T03:17:24.388 INFO:teuthology.orchestra.run.smithi118.stdout: "desc": "", 2024-03-24T03:17:24.388 INFO:teuthology.orchestra.run.smithi118.stdout: "long_desc": "", 2024-03-24T03:17:24.388 INFO:teuthology.orchestra.run.smithi118.stdout: "tags": [], 2024-03-24T03:17:24.388 INFO:teuthology.orchestra.run.smithi118.stdout: "see_also": [] 2024-03-24T03:17:24.389 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-24T03:17:24.389 INFO:teuthology.orchestra.run.smithi118.stdout: "log_to_cluster_level": { 2024-03-24T03:17:24.389 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "log_to_cluster_level", 2024-03-24T03:17:24.389 INFO:teuthology.orchestra.run.smithi118.stdout: "type": "str", 2024-03-24T03:17:24.389 INFO:teuthology.orchestra.run.smithi118.stdout: "level": "advanced", 2024-03-24T03:17:24.389 INFO:teuthology.orchestra.run.smithi118.stdout: "flags": 1, 2024-03-24T03:17:24.389 INFO:teuthology.orchestra.run.smithi118.stdout: "default_value": "info", 2024-03-24T03:17:24.389 INFO:teuthology.orchestra.run.smithi118.stdout: "min": "", 2024-03-24T03:17:24.389 INFO:teuthology.orchestra.run.smithi118.stdout: "max": "", 2024-03-24T03:17:24.389 INFO:teuthology.orchestra.run.smithi118.stdout: "enum_allowed": [ 2024-03-24T03:17:24.390 INFO:teuthology.orchestra.run.smithi118.stdout: "", 2024-03-24T03:17:24.390 INFO:teuthology.orchestra.run.smithi118.stdout: "critical", 2024-03-24T03:17:24.390 INFO:teuthology.orchestra.run.smithi118.stdout: "debug", 2024-03-24T03:17:24.390 INFO:teuthology.orchestra.run.smithi118.stdout: "error", 2024-03-24T03:17:24.390 INFO:teuthology.orchestra.run.smithi118.stdout: "info", 2024-03-24T03:17:24.390 INFO:teuthology.orchestra.run.smithi118.stdout: "warning" 2024-03-24T03:17:24.390 INFO:teuthology.orchestra.run.smithi118.stdout: ], 2024-03-24T03:17:24.390 INFO:teuthology.orchestra.run.smithi118.stdout: "desc": "", 2024-03-24T03:17:24.390 INFO:teuthology.orchestra.run.smithi118.stdout: "long_desc": "", 2024-03-24T03:17:24.391 INFO:teuthology.orchestra.run.smithi118.stdout: "tags": [], 2024-03-24T03:17:24.391 INFO:teuthology.orchestra.run.smithi118.stdout: "see_also": [] 2024-03-24T03:17:24.391 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-24T03:17:24.391 INFO:teuthology.orchestra.run.smithi118.stdout: "log_to_file": { 2024-03-24T03:17:24.391 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "log_to_file", 2024-03-24T03:17:24.391 INFO:teuthology.orchestra.run.smithi118.stdout: "type": "bool", 2024-03-24T03:17:24.391 INFO:teuthology.orchestra.run.smithi118.stdout: "level": "advanced", 2024-03-24T03:17:24.391 INFO:teuthology.orchestra.run.smithi118.stdout: "flags": 1, 2024-03-24T03:17:24.392 INFO:teuthology.orchestra.run.smithi118.stdout: "default_value": "False", 2024-03-24T03:17:24.392 INFO:teuthology.orchestra.run.smithi118.stdout: "min": "", 2024-03-24T03:17:24.392 INFO:teuthology.orchestra.run.smithi118.stdout: "max": "", 2024-03-24T03:17:24.392 INFO:teuthology.orchestra.run.smithi118.stdout: "enum_allowed": [], 2024-03-24T03:17:24.392 INFO:teuthology.orchestra.run.smithi118.stdout: "desc": "", 2024-03-24T03:17:24.392 INFO:teuthology.orchestra.run.smithi118.stdout: "long_desc": "", 2024-03-24T03:17:24.392 INFO:teuthology.orchestra.run.smithi118.stdout: "tags": [], 2024-03-24T03:17:24.392 INFO:teuthology.orchestra.run.smithi118.stdout: "see_also": [] 2024-03-24T03:17:24.392 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-24T03:17:24.393 INFO:teuthology.orchestra.run.smithi118.stdout: "server_addr": { 2024-03-24T03:17:24.393 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "server_addr", 2024-03-24T03:17:24.393 INFO:teuthology.orchestra.run.smithi118.stdout: "type": "str", 2024-03-24T03:17:24.393 INFO:teuthology.orchestra.run.smithi118.stdout: "level": "advanced", 2024-03-24T03:17:24.393 INFO:teuthology.orchestra.run.smithi118.stdout: "flags": 0, 2024-03-24T03:17:24.393 INFO:teuthology.orchestra.run.smithi118.stdout: "default_value": "", 2024-03-24T03:17:24.393 INFO:teuthology.orchestra.run.smithi118.stdout: "min": "", 2024-03-24T03:17:24.393 INFO:teuthology.orchestra.run.smithi118.stdout: "max": "", 2024-03-24T03:17:24.393 INFO:teuthology.orchestra.run.smithi118.stdout: "enum_allowed": [], 2024-03-24T03:17:24.394 INFO:teuthology.orchestra.run.smithi118.stdout: "desc": "", 2024-03-24T03:17:24.394 INFO:teuthology.orchestra.run.smithi118.stdout: "long_desc": "", 2024-03-24T03:17:24.394 INFO:teuthology.orchestra.run.smithi118.stdout: "tags": [], 2024-03-24T03:17:24.394 INFO:teuthology.orchestra.run.smithi118.stdout: "see_also": [] 2024-03-24T03:17:24.394 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-24T03:17:24.394 INFO:teuthology.orchestra.run.smithi118.stdout: "server_port": { 2024-03-24T03:17:24.394 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "server_port", 2024-03-24T03:17:24.394 INFO:teuthology.orchestra.run.smithi118.stdout: "type": "str", 2024-03-24T03:17:24.394 INFO:teuthology.orchestra.run.smithi118.stdout: "level": "advanced", 2024-03-24T03:17:24.395 INFO:teuthology.orchestra.run.smithi118.stdout: "flags": 0, 2024-03-24T03:17:24.395 INFO:teuthology.orchestra.run.smithi118.stdout: "default_value": "", 2024-03-24T03:17:24.395 INFO:teuthology.orchestra.run.smithi118.stdout: "min": "", 2024-03-24T03:17:24.395 INFO:teuthology.orchestra.run.smithi118.stdout: "max": "", 2024-03-24T03:17:24.395 INFO:teuthology.orchestra.run.smithi118.stdout: "enum_allowed": [], 2024-03-24T03:17:24.395 INFO:teuthology.orchestra.run.smithi118.stdout: "desc": "", 2024-03-24T03:17:24.395 INFO:teuthology.orchestra.run.smithi118.stdout: "long_desc": "", 2024-03-24T03:17:24.395 INFO:teuthology.orchestra.run.smithi118.stdout: "tags": [], 2024-03-24T03:17:24.395 INFO:teuthology.orchestra.run.smithi118.stdout: "see_also": [] 2024-03-24T03:17:24.397 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-03-24T03:17:24.397 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-03-24T03:17:24.397 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-24T03:17:24.397 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-24T03:17:24.397 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "rgw", 2024-03-24T03:17:24.397 INFO:teuthology.orchestra.run.smithi118.stdout: "can_run": true, 2024-03-24T03:17:24.398 INFO:teuthology.orchestra.run.smithi118.stdout: "error_string": "", 2024-03-24T03:17:24.398 INFO:teuthology.orchestra.run.smithi118.stdout: "module_options": { 2024-03-24T03:17:24.398 INFO:teuthology.orchestra.run.smithi118.stdout: "log_level": { 2024-03-24T03:17:24.398 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "log_level", 2024-03-24T03:17:24.398 INFO:teuthology.orchestra.run.smithi118.stdout: "type": "str", 2024-03-24T03:17:24.398 INFO:teuthology.orchestra.run.smithi118.stdout: "level": "advanced", 2024-03-24T03:17:24.398 INFO:teuthology.orchestra.run.smithi118.stdout: "flags": 1, 2024-03-24T03:17:24.398 INFO:teuthology.orchestra.run.smithi118.stdout: "default_value": "", 2024-03-24T03:17:24.398 INFO:teuthology.orchestra.run.smithi118.stdout: "min": "", 2024-03-24T03:17:24.399 INFO:teuthology.orchestra.run.smithi118.stdout: "max": "", 2024-03-24T03:17:24.399 INFO:teuthology.orchestra.run.smithi118.stdout: "enum_allowed": [ 2024-03-24T03:17:24.399 INFO:teuthology.orchestra.run.smithi118.stdout: "", 2024-03-24T03:17:24.399 INFO:teuthology.orchestra.run.smithi118.stdout: "critical", 2024-03-24T03:17:24.399 INFO:teuthology.orchestra.run.smithi118.stdout: "debug", 2024-03-24T03:17:24.399 INFO:teuthology.orchestra.run.smithi118.stdout: "error", 2024-03-24T03:17:24.399 INFO:teuthology.orchestra.run.smithi118.stdout: "info", 2024-03-24T03:17:24.399 INFO:teuthology.orchestra.run.smithi118.stdout: "warning" 2024-03-24T03:17:24.399 INFO:teuthology.orchestra.run.smithi118.stdout: ], 2024-03-24T03:17:24.400 INFO:teuthology.orchestra.run.smithi118.stdout: "desc": "", 2024-03-24T03:17:24.400 INFO:teuthology.orchestra.run.smithi118.stdout: "long_desc": "", 2024-03-24T03:17:24.400 INFO:teuthology.orchestra.run.smithi118.stdout: "tags": [], 2024-03-24T03:17:24.400 INFO:teuthology.orchestra.run.smithi118.stdout: "see_also": [] 2024-03-24T03:17:24.400 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-24T03:17:24.400 INFO:teuthology.orchestra.run.smithi118.stdout: "log_to_cluster": { 2024-03-24T03:17:24.400 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "log_to_cluster", 2024-03-24T03:17:24.400 INFO:teuthology.orchestra.run.smithi118.stdout: "type": "bool", 2024-03-24T03:17:24.400 INFO:teuthology.orchestra.run.smithi118.stdout: "level": "advanced", 2024-03-24T03:17:24.401 INFO:teuthology.orchestra.run.smithi118.stdout: "flags": 1, 2024-03-24T03:17:24.401 INFO:teuthology.orchestra.run.smithi118.stdout: "default_value": "False", 2024-03-24T03:17:24.401 INFO:teuthology.orchestra.run.smithi118.stdout: "min": "", 2024-03-24T03:17:24.401 INFO:teuthology.orchestra.run.smithi118.stdout: "max": "", 2024-03-24T03:17:24.401 INFO:teuthology.orchestra.run.smithi118.stdout: "enum_allowed": [], 2024-03-24T03:17:24.401 INFO:teuthology.orchestra.run.smithi118.stdout: "desc": "", 2024-03-24T03:17:24.401 INFO:teuthology.orchestra.run.smithi118.stdout: "long_desc": "", 2024-03-24T03:17:24.401 INFO:teuthology.orchestra.run.smithi118.stdout: "tags": [], 2024-03-24T03:17:24.401 INFO:teuthology.orchestra.run.smithi118.stdout: "see_also": [] 2024-03-24T03:17:24.401 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-24T03:17:24.402 INFO:teuthology.orchestra.run.smithi118.stdout: "log_to_cluster_level": { 2024-03-24T03:17:24.402 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "log_to_cluster_level", 2024-03-24T03:17:24.402 INFO:teuthology.orchestra.run.smithi118.stdout: "type": "str", 2024-03-24T03:17:24.402 INFO:teuthology.orchestra.run.smithi118.stdout: "level": "advanced", 2024-03-24T03:17:24.402 INFO:teuthology.orchestra.run.smithi118.stdout: "flags": 1, 2024-03-24T03:17:24.402 INFO:teuthology.orchestra.run.smithi118.stdout: "default_value": "info", 2024-03-24T03:17:24.402 INFO:teuthology.orchestra.run.smithi118.stdout: "min": "", 2024-03-24T03:17:24.402 INFO:teuthology.orchestra.run.smithi118.stdout: "max": "", 2024-03-24T03:17:24.403 INFO:teuthology.orchestra.run.smithi118.stdout: "enum_allowed": [ 2024-03-24T03:17:24.403 INFO:teuthology.orchestra.run.smithi118.stdout: "", 2024-03-24T03:17:24.403 INFO:teuthology.orchestra.run.smithi118.stdout: "critical", 2024-03-24T03:17:24.403 INFO:teuthology.orchestra.run.smithi118.stdout: "debug", 2024-03-24T03:17:24.403 INFO:teuthology.orchestra.run.smithi118.stdout: "error", 2024-03-24T03:17:24.403 INFO:teuthology.orchestra.run.smithi118.stdout: "info", 2024-03-24T03:17:24.403 INFO:teuthology.orchestra.run.smithi118.stdout: "warning" 2024-03-24T03:17:24.403 INFO:teuthology.orchestra.run.smithi118.stdout: ], 2024-03-24T03:17:24.403 INFO:teuthology.orchestra.run.smithi118.stdout: "desc": "", 2024-03-24T03:17:24.404 INFO:teuthology.orchestra.run.smithi118.stdout: "long_desc": "", 2024-03-24T03:17:24.404 INFO:teuthology.orchestra.run.smithi118.stdout: "tags": [], 2024-03-24T03:17:24.404 INFO:teuthology.orchestra.run.smithi118.stdout: "see_also": [] 2024-03-24T03:17:24.404 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-24T03:17:24.404 INFO:teuthology.orchestra.run.smithi118.stdout: "log_to_file": { 2024-03-24T03:17:24.404 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "log_to_file", 2024-03-24T03:17:24.404 INFO:teuthology.orchestra.run.smithi118.stdout: "type": "bool", 2024-03-24T03:17:24.404 INFO:teuthology.orchestra.run.smithi118.stdout: "level": "advanced", 2024-03-24T03:17:24.404 INFO:teuthology.orchestra.run.smithi118.stdout: "flags": 1, 2024-03-24T03:17:24.405 INFO:teuthology.orchestra.run.smithi118.stdout: "default_value": "False", 2024-03-24T03:17:24.405 INFO:teuthology.orchestra.run.smithi118.stdout: "min": "", 2024-03-24T03:17:24.405 INFO:teuthology.orchestra.run.smithi118.stdout: "max": "", 2024-03-24T03:17:24.405 INFO:teuthology.orchestra.run.smithi118.stdout: "enum_allowed": [], 2024-03-24T03:17:24.405 INFO:teuthology.orchestra.run.smithi118.stdout: "desc": "", 2024-03-24T03:17:24.405 INFO:teuthology.orchestra.run.smithi118.stdout: "long_desc": "", 2024-03-24T03:17:24.405 INFO:teuthology.orchestra.run.smithi118.stdout: "tags": [], 2024-03-24T03:17:24.405 INFO:teuthology.orchestra.run.smithi118.stdout: "see_also": [] 2024-03-24T03:17:24.405 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-03-24T03:17:24.406 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-03-24T03:17:24.406 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-24T03:17:24.406 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-24T03:17:24.406 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "rook", 2024-03-24T03:17:24.406 INFO:teuthology.orchestra.run.smithi118.stdout: "can_run": true, 2024-03-24T03:17:24.406 INFO:teuthology.orchestra.run.smithi118.stdout: "error_string": "", 2024-03-24T03:17:24.406 INFO:teuthology.orchestra.run.smithi118.stdout: "module_options": { 2024-03-24T03:17:24.406 INFO:teuthology.orchestra.run.smithi118.stdout: "log_level": { 2024-03-24T03:17:24.407 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "log_level", 2024-03-24T03:17:24.407 INFO:teuthology.orchestra.run.smithi118.stdout: "type": "str", 2024-03-24T03:17:24.407 INFO:teuthology.orchestra.run.smithi118.stdout: "level": "advanced", 2024-03-24T03:17:24.407 INFO:teuthology.orchestra.run.smithi118.stdout: "flags": 1, 2024-03-24T03:17:24.407 INFO:teuthology.orchestra.run.smithi118.stdout: "default_value": "", 2024-03-24T03:17:24.407 INFO:teuthology.orchestra.run.smithi118.stdout: "min": "", 2024-03-24T03:17:24.407 INFO:teuthology.orchestra.run.smithi118.stdout: "max": "", 2024-03-24T03:17:24.407 INFO:teuthology.orchestra.run.smithi118.stdout: "enum_allowed": [ 2024-03-24T03:17:24.407 INFO:teuthology.orchestra.run.smithi118.stdout: "", 2024-03-24T03:17:24.407 INFO:teuthology.orchestra.run.smithi118.stdout: "critical", 2024-03-24T03:17:24.408 INFO:teuthology.orchestra.run.smithi118.stdout: "debug", 2024-03-24T03:17:24.408 INFO:teuthology.orchestra.run.smithi118.stdout: "error", 2024-03-24T03:17:24.408 INFO:teuthology.orchestra.run.smithi118.stdout: "info", 2024-03-24T03:17:24.408 INFO:teuthology.orchestra.run.smithi118.stdout: "warning" 2024-03-24T03:17:24.408 INFO:teuthology.orchestra.run.smithi118.stdout: ], 2024-03-24T03:17:24.408 INFO:teuthology.orchestra.run.smithi118.stdout: "desc": "", 2024-03-24T03:17:24.408 INFO:teuthology.orchestra.run.smithi118.stdout: "long_desc": "", 2024-03-24T03:17:24.408 INFO:teuthology.orchestra.run.smithi118.stdout: "tags": [], 2024-03-24T03:17:24.409 INFO:teuthology.orchestra.run.smithi118.stdout: "see_also": [] 2024-03-24T03:17:24.409 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-24T03:17:24.409 INFO:teuthology.orchestra.run.smithi118.stdout: "log_to_cluster": { 2024-03-24T03:17:24.409 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "log_to_cluster", 2024-03-24T03:17:24.409 INFO:teuthology.orchestra.run.smithi118.stdout: "type": "bool", 2024-03-24T03:17:24.409 INFO:teuthology.orchestra.run.smithi118.stdout: "level": "advanced", 2024-03-24T03:17:24.409 INFO:teuthology.orchestra.run.smithi118.stdout: "flags": 1, 2024-03-24T03:17:24.409 INFO:teuthology.orchestra.run.smithi118.stdout: "default_value": "False", 2024-03-24T03:17:24.409 INFO:teuthology.orchestra.run.smithi118.stdout: "min": "", 2024-03-24T03:17:24.409 INFO:teuthology.orchestra.run.smithi118.stdout: "max": "", 2024-03-24T03:17:24.410 INFO:teuthology.orchestra.run.smithi118.stdout: "enum_allowed": [], 2024-03-24T03:17:24.410 INFO:teuthology.orchestra.run.smithi118.stdout: "desc": "", 2024-03-24T03:17:24.410 INFO:teuthology.orchestra.run.smithi118.stdout: "long_desc": "", 2024-03-24T03:17:24.410 INFO:teuthology.orchestra.run.smithi118.stdout: "tags": [], 2024-03-24T03:17:24.410 INFO:teuthology.orchestra.run.smithi118.stdout: "see_also": [] 2024-03-24T03:17:24.410 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-24T03:17:24.410 INFO:teuthology.orchestra.run.smithi118.stdout: "log_to_cluster_level": { 2024-03-24T03:17:24.410 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "log_to_cluster_level", 2024-03-24T03:17:24.411 INFO:teuthology.orchestra.run.smithi118.stdout: "type": "str", 2024-03-24T03:17:24.411 INFO:teuthology.orchestra.run.smithi118.stdout: "level": "advanced", 2024-03-24T03:17:24.411 INFO:teuthology.orchestra.run.smithi118.stdout: "flags": 1, 2024-03-24T03:17:24.411 INFO:teuthology.orchestra.run.smithi118.stdout: "default_value": "info", 2024-03-24T03:17:24.411 INFO:teuthology.orchestra.run.smithi118.stdout: "min": "", 2024-03-24T03:17:24.411 INFO:teuthology.orchestra.run.smithi118.stdout: "max": "", 2024-03-24T03:17:24.411 INFO:teuthology.orchestra.run.smithi118.stdout: "enum_allowed": [ 2024-03-24T03:17:24.411 INFO:teuthology.orchestra.run.smithi118.stdout: "", 2024-03-24T03:17:24.411 INFO:teuthology.orchestra.run.smithi118.stdout: "critical", 2024-03-24T03:17:24.412 INFO:teuthology.orchestra.run.smithi118.stdout: "debug", 2024-03-24T03:17:24.412 INFO:teuthology.orchestra.run.smithi118.stdout: "error", 2024-03-24T03:17:24.412 INFO:teuthology.orchestra.run.smithi118.stdout: "info", 2024-03-24T03:17:24.412 INFO:teuthology.orchestra.run.smithi118.stdout: "warning" 2024-03-24T03:17:24.412 INFO:teuthology.orchestra.run.smithi118.stdout: ], 2024-03-24T03:17:24.412 INFO:teuthology.orchestra.run.smithi118.stdout: "desc": "", 2024-03-24T03:17:24.412 INFO:teuthology.orchestra.run.smithi118.stdout: "long_desc": "", 2024-03-24T03:17:24.412 INFO:teuthology.orchestra.run.smithi118.stdout: "tags": [], 2024-03-24T03:17:24.412 INFO:teuthology.orchestra.run.smithi118.stdout: "see_also": [] 2024-03-24T03:17:24.412 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-24T03:17:24.413 INFO:teuthology.orchestra.run.smithi118.stdout: "log_to_file": { 2024-03-24T03:17:24.413 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "log_to_file", 2024-03-24T03:17:24.413 INFO:teuthology.orchestra.run.smithi118.stdout: "type": "bool", 2024-03-24T03:17:24.413 INFO:teuthology.orchestra.run.smithi118.stdout: "level": "advanced", 2024-03-24T03:17:24.413 INFO:teuthology.orchestra.run.smithi118.stdout: "flags": 1, 2024-03-24T03:17:24.413 INFO:teuthology.orchestra.run.smithi118.stdout: "default_value": "False", 2024-03-24T03:17:24.413 INFO:teuthology.orchestra.run.smithi118.stdout: "min": "", 2024-03-24T03:17:24.413 INFO:teuthology.orchestra.run.smithi118.stdout: "max": "", 2024-03-24T03:17:24.413 INFO:teuthology.orchestra.run.smithi118.stdout: "enum_allowed": [], 2024-03-24T03:17:24.414 INFO:teuthology.orchestra.run.smithi118.stdout: "desc": "", 2024-03-24T03:17:24.414 INFO:teuthology.orchestra.run.smithi118.stdout: "long_desc": "", 2024-03-24T03:17:24.414 INFO:teuthology.orchestra.run.smithi118.stdout: "tags": [], 2024-03-24T03:17:24.414 INFO:teuthology.orchestra.run.smithi118.stdout: "see_also": [] 2024-03-24T03:17:24.414 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-24T03:17:24.414 INFO:teuthology.orchestra.run.smithi118.stdout: "storage_class": { 2024-03-24T03:17:24.414 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "storage_class", 2024-03-24T03:17:24.414 INFO:teuthology.orchestra.run.smithi118.stdout: "type": "str", 2024-03-24T03:17:24.415 INFO:teuthology.orchestra.run.smithi118.stdout: "level": "advanced", 2024-03-24T03:17:24.415 INFO:teuthology.orchestra.run.smithi118.stdout: "flags": 0, 2024-03-24T03:17:24.415 INFO:teuthology.orchestra.run.smithi118.stdout: "default_value": "local", 2024-03-24T03:17:24.415 INFO:teuthology.orchestra.run.smithi118.stdout: "min": "", 2024-03-24T03:17:24.415 INFO:teuthology.orchestra.run.smithi118.stdout: "max": "", 2024-03-24T03:17:24.415 INFO:teuthology.orchestra.run.smithi118.stdout: "enum_allowed": [], 2024-03-24T03:17:24.415 INFO:teuthology.orchestra.run.smithi118.stdout: "desc": "storage class name for LSO-discovered PVs", 2024-03-24T03:17:24.415 INFO:teuthology.orchestra.run.smithi118.stdout: "long_desc": "", 2024-03-24T03:17:24.416 INFO:teuthology.orchestra.run.smithi118.stdout: "tags": [], 2024-03-24T03:17:24.416 INFO:teuthology.orchestra.run.smithi118.stdout: "see_also": [] 2024-03-24T03:17:24.416 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-03-24T03:17:24.416 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-03-24T03:17:24.416 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-24T03:17:24.416 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-24T03:17:24.416 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "selftest", 2024-03-24T03:17:24.416 INFO:teuthology.orchestra.run.smithi118.stdout: "can_run": true, 2024-03-24T03:17:24.416 INFO:teuthology.orchestra.run.smithi118.stdout: "error_string": "", 2024-03-24T03:17:24.416 INFO:teuthology.orchestra.run.smithi118.stdout: "module_options": { 2024-03-24T03:17:24.417 INFO:teuthology.orchestra.run.smithi118.stdout: "log_level": { 2024-03-24T03:17:24.417 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "log_level", 2024-03-24T03:17:24.417 INFO:teuthology.orchestra.run.smithi118.stdout: "type": "str", 2024-03-24T03:17:24.417 INFO:teuthology.orchestra.run.smithi118.stdout: "level": "advanced", 2024-03-24T03:17:24.417 INFO:teuthology.orchestra.run.smithi118.stdout: "flags": 1, 2024-03-24T03:17:24.417 INFO:teuthology.orchestra.run.smithi118.stdout: "default_value": "", 2024-03-24T03:17:24.417 INFO:teuthology.orchestra.run.smithi118.stdout: "min": "", 2024-03-24T03:17:24.417 INFO:teuthology.orchestra.run.smithi118.stdout: "max": "", 2024-03-24T03:17:24.417 INFO:teuthology.orchestra.run.smithi118.stdout: "enum_allowed": [ 2024-03-24T03:17:24.418 INFO:teuthology.orchestra.run.smithi118.stdout: "", 2024-03-24T03:17:24.418 INFO:teuthology.orchestra.run.smithi118.stdout: "critical", 2024-03-24T03:17:24.418 INFO:teuthology.orchestra.run.smithi118.stdout: "debug", 2024-03-24T03:17:24.418 INFO:teuthology.orchestra.run.smithi118.stdout: "error", 2024-03-24T03:17:24.418 INFO:teuthology.orchestra.run.smithi118.stdout: "info", 2024-03-24T03:17:24.418 INFO:teuthology.orchestra.run.smithi118.stdout: "warning" 2024-03-24T03:17:24.418 INFO:teuthology.orchestra.run.smithi118.stdout: ], 2024-03-24T03:17:24.419 INFO:teuthology.orchestra.run.smithi118.stdout: "desc": "", 2024-03-24T03:17:24.419 INFO:teuthology.orchestra.run.smithi118.stdout: "long_desc": "", 2024-03-24T03:17:24.419 INFO:teuthology.orchestra.run.smithi118.stdout: "tags": [], 2024-03-24T03:17:24.419 INFO:teuthology.orchestra.run.smithi118.stdout: "see_also": [] 2024-03-24T03:17:24.419 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-24T03:17:24.419 INFO:teuthology.orchestra.run.smithi118.stdout: "log_to_cluster": { 2024-03-24T03:17:24.419 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "log_to_cluster", 2024-03-24T03:17:24.419 INFO:teuthology.orchestra.run.smithi118.stdout: "type": "bool", 2024-03-24T03:17:24.419 INFO:teuthology.orchestra.run.smithi118.stdout: "level": "advanced", 2024-03-24T03:17:24.419 INFO:teuthology.orchestra.run.smithi118.stdout: "flags": 1, 2024-03-24T03:17:24.420 INFO:teuthology.orchestra.run.smithi118.stdout: "default_value": "False", 2024-03-24T03:17:24.420 INFO:teuthology.orchestra.run.smithi118.stdout: "min": "", 2024-03-24T03:17:24.420 INFO:teuthology.orchestra.run.smithi118.stdout: "max": "", 2024-03-24T03:17:24.420 INFO:teuthology.orchestra.run.smithi118.stdout: "enum_allowed": [], 2024-03-24T03:17:24.420 INFO:teuthology.orchestra.run.smithi118.stdout: "desc": "", 2024-03-24T03:17:24.420 INFO:teuthology.orchestra.run.smithi118.stdout: "long_desc": "", 2024-03-24T03:17:24.420 INFO:teuthology.orchestra.run.smithi118.stdout: "tags": [], 2024-03-24T03:17:24.420 INFO:teuthology.orchestra.run.smithi118.stdout: "see_also": [] 2024-03-24T03:17:24.420 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-24T03:17:24.421 INFO:teuthology.orchestra.run.smithi118.stdout: "log_to_cluster_level": { 2024-03-24T03:17:24.421 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "log_to_cluster_level", 2024-03-24T03:17:24.421 INFO:teuthology.orchestra.run.smithi118.stdout: "type": "str", 2024-03-24T03:17:24.421 INFO:teuthology.orchestra.run.smithi118.stdout: "level": "advanced", 2024-03-24T03:17:24.421 INFO:teuthology.orchestra.run.smithi118.stdout: "flags": 1, 2024-03-24T03:17:24.421 INFO:teuthology.orchestra.run.smithi118.stdout: "default_value": "info", 2024-03-24T03:17:24.421 INFO:teuthology.orchestra.run.smithi118.stdout: "min": "", 2024-03-24T03:17:24.421 INFO:teuthology.orchestra.run.smithi118.stdout: "max": "", 2024-03-24T03:17:24.422 INFO:teuthology.orchestra.run.smithi118.stdout: "enum_allowed": [ 2024-03-24T03:17:24.422 INFO:teuthology.orchestra.run.smithi118.stdout: "", 2024-03-24T03:17:24.422 INFO:teuthology.orchestra.run.smithi118.stdout: "critical", 2024-03-24T03:17:24.422 INFO:teuthology.orchestra.run.smithi118.stdout: "debug", 2024-03-24T03:17:24.422 INFO:teuthology.orchestra.run.smithi118.stdout: "error", 2024-03-24T03:17:24.422 INFO:teuthology.orchestra.run.smithi118.stdout: "info", 2024-03-24T03:17:24.422 INFO:teuthology.orchestra.run.smithi118.stdout: "warning" 2024-03-24T03:17:24.422 INFO:teuthology.orchestra.run.smithi118.stdout: ], 2024-03-24T03:17:24.422 INFO:teuthology.orchestra.run.smithi118.stdout: "desc": "", 2024-03-24T03:17:24.423 INFO:teuthology.orchestra.run.smithi118.stdout: "long_desc": "", 2024-03-24T03:17:24.423 INFO:teuthology.orchestra.run.smithi118.stdout: "tags": [], 2024-03-24T03:17:24.423 INFO:teuthology.orchestra.run.smithi118.stdout: "see_also": [] 2024-03-24T03:17:24.423 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-24T03:17:24.423 INFO:teuthology.orchestra.run.smithi118.stdout: "log_to_file": { 2024-03-24T03:17:24.423 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "log_to_file", 2024-03-24T03:17:24.423 INFO:teuthology.orchestra.run.smithi118.stdout: "type": "bool", 2024-03-24T03:17:24.423 INFO:teuthology.orchestra.run.smithi118.stdout: "level": "advanced", 2024-03-24T03:17:24.423 INFO:teuthology.orchestra.run.smithi118.stdout: "flags": 1, 2024-03-24T03:17:24.424 INFO:teuthology.orchestra.run.smithi118.stdout: "default_value": "False", 2024-03-24T03:17:24.424 INFO:teuthology.orchestra.run.smithi118.stdout: "min": "", 2024-03-24T03:17:24.424 INFO:teuthology.orchestra.run.smithi118.stdout: "max": "", 2024-03-24T03:17:24.424 INFO:teuthology.orchestra.run.smithi118.stdout: "enum_allowed": [], 2024-03-24T03:17:24.424 INFO:teuthology.orchestra.run.smithi118.stdout: "desc": "", 2024-03-24T03:17:24.424 INFO:teuthology.orchestra.run.smithi118.stdout: "long_desc": "", 2024-03-24T03:17:24.424 INFO:teuthology.orchestra.run.smithi118.stdout: "tags": [], 2024-03-24T03:17:24.424 INFO:teuthology.orchestra.run.smithi118.stdout: "see_also": [] 2024-03-24T03:17:24.424 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-24T03:17:24.425 INFO:teuthology.orchestra.run.smithi118.stdout: "roption1": { 2024-03-24T03:17:24.425 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "roption1", 2024-03-24T03:17:24.425 INFO:teuthology.orchestra.run.smithi118.stdout: "type": "str", 2024-03-24T03:17:24.425 INFO:teuthology.orchestra.run.smithi118.stdout: "level": "advanced", 2024-03-24T03:17:24.425 INFO:teuthology.orchestra.run.smithi118.stdout: "flags": 0, 2024-03-24T03:17:24.425 INFO:teuthology.orchestra.run.smithi118.stdout: "default_value": "", 2024-03-24T03:17:24.425 INFO:teuthology.orchestra.run.smithi118.stdout: "min": "", 2024-03-24T03:17:24.425 INFO:teuthology.orchestra.run.smithi118.stdout: "max": "", 2024-03-24T03:17:24.425 INFO:teuthology.orchestra.run.smithi118.stdout: "enum_allowed": [], 2024-03-24T03:17:24.426 INFO:teuthology.orchestra.run.smithi118.stdout: "desc": "", 2024-03-24T03:17:24.426 INFO:teuthology.orchestra.run.smithi118.stdout: "long_desc": "", 2024-03-24T03:17:24.426 INFO:teuthology.orchestra.run.smithi118.stdout: "tags": [], 2024-03-24T03:17:24.426 INFO:teuthology.orchestra.run.smithi118.stdout: "see_also": [] 2024-03-24T03:17:24.426 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-24T03:17:24.426 INFO:teuthology.orchestra.run.smithi118.stdout: "roption2": { 2024-03-24T03:17:24.426 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "roption2", 2024-03-24T03:17:24.426 INFO:teuthology.orchestra.run.smithi118.stdout: "type": "str", 2024-03-24T03:17:24.427 INFO:teuthology.orchestra.run.smithi118.stdout: "level": "advanced", 2024-03-24T03:17:24.427 INFO:teuthology.orchestra.run.smithi118.stdout: "flags": 0, 2024-03-24T03:17:24.427 INFO:teuthology.orchestra.run.smithi118.stdout: "default_value": "xyz", 2024-03-24T03:17:24.427 INFO:teuthology.orchestra.run.smithi118.stdout: "min": "", 2024-03-24T03:17:24.427 INFO:teuthology.orchestra.run.smithi118.stdout: "max": "", 2024-03-24T03:17:24.427 INFO:teuthology.orchestra.run.smithi118.stdout: "enum_allowed": [], 2024-03-24T03:17:24.427 INFO:teuthology.orchestra.run.smithi118.stdout: "desc": "", 2024-03-24T03:17:24.427 INFO:teuthology.orchestra.run.smithi118.stdout: "long_desc": "", 2024-03-24T03:17:24.427 INFO:teuthology.orchestra.run.smithi118.stdout: "tags": [], 2024-03-24T03:17:24.427 INFO:teuthology.orchestra.run.smithi118.stdout: "see_also": [] 2024-03-24T03:17:24.428 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-24T03:17:24.428 INFO:teuthology.orchestra.run.smithi118.stdout: "rwoption1": { 2024-03-24T03:17:24.428 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "rwoption1", 2024-03-24T03:17:24.428 INFO:teuthology.orchestra.run.smithi118.stdout: "type": "str", 2024-03-24T03:17:24.428 INFO:teuthology.orchestra.run.smithi118.stdout: "level": "advanced", 2024-03-24T03:17:24.428 INFO:teuthology.orchestra.run.smithi118.stdout: "flags": 0, 2024-03-24T03:17:24.428 INFO:teuthology.orchestra.run.smithi118.stdout: "default_value": "", 2024-03-24T03:17:24.428 INFO:teuthology.orchestra.run.smithi118.stdout: "min": "", 2024-03-24T03:17:24.428 INFO:teuthology.orchestra.run.smithi118.stdout: "max": "", 2024-03-24T03:17:24.429 INFO:teuthology.orchestra.run.smithi118.stdout: "enum_allowed": [], 2024-03-24T03:17:24.429 INFO:teuthology.orchestra.run.smithi118.stdout: "desc": "", 2024-03-24T03:17:24.429 INFO:teuthology.orchestra.run.smithi118.stdout: "long_desc": "", 2024-03-24T03:17:24.429 INFO:teuthology.orchestra.run.smithi118.stdout: "tags": [], 2024-03-24T03:17:24.429 INFO:teuthology.orchestra.run.smithi118.stdout: "see_also": [] 2024-03-24T03:17:24.429 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-24T03:17:24.429 INFO:teuthology.orchestra.run.smithi118.stdout: "rwoption2": { 2024-03-24T03:17:24.429 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "rwoption2", 2024-03-24T03:17:24.429 INFO:teuthology.orchestra.run.smithi118.stdout: "type": "int", 2024-03-24T03:17:24.429 INFO:teuthology.orchestra.run.smithi118.stdout: "level": "advanced", 2024-03-24T03:17:24.430 INFO:teuthology.orchestra.run.smithi118.stdout: "flags": 0, 2024-03-24T03:17:24.430 INFO:teuthology.orchestra.run.smithi118.stdout: "default_value": "", 2024-03-24T03:17:24.430 INFO:teuthology.orchestra.run.smithi118.stdout: "min": "", 2024-03-24T03:17:24.430 INFO:teuthology.orchestra.run.smithi118.stdout: "max": "", 2024-03-24T03:17:24.430 INFO:teuthology.orchestra.run.smithi118.stdout: "enum_allowed": [], 2024-03-24T03:17:24.430 INFO:teuthology.orchestra.run.smithi118.stdout: "desc": "", 2024-03-24T03:17:24.430 INFO:teuthology.orchestra.run.smithi118.stdout: "long_desc": "", 2024-03-24T03:17:24.430 INFO:teuthology.orchestra.run.smithi118.stdout: "tags": [], 2024-03-24T03:17:24.430 INFO:teuthology.orchestra.run.smithi118.stdout: "see_also": [] 2024-03-24T03:17:24.431 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-24T03:17:24.431 INFO:teuthology.orchestra.run.smithi118.stdout: "rwoption3": { 2024-03-24T03:17:24.431 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "rwoption3", 2024-03-24T03:17:24.431 INFO:teuthology.orchestra.run.smithi118.stdout: "type": "float", 2024-03-24T03:17:24.431 INFO:teuthology.orchestra.run.smithi118.stdout: "level": "advanced", 2024-03-24T03:17:24.431 INFO:teuthology.orchestra.run.smithi118.stdout: "flags": 0, 2024-03-24T03:17:24.431 INFO:teuthology.orchestra.run.smithi118.stdout: "default_value": "", 2024-03-24T03:17:24.431 INFO:teuthology.orchestra.run.smithi118.stdout: "min": "", 2024-03-24T03:17:24.431 INFO:teuthology.orchestra.run.smithi118.stdout: "max": "", 2024-03-24T03:17:24.432 INFO:teuthology.orchestra.run.smithi118.stdout: "enum_allowed": [], 2024-03-24T03:17:24.432 INFO:teuthology.orchestra.run.smithi118.stdout: "desc": "", 2024-03-24T03:17:24.432 INFO:teuthology.orchestra.run.smithi118.stdout: "long_desc": "", 2024-03-24T03:17:24.432 INFO:teuthology.orchestra.run.smithi118.stdout: "tags": [], 2024-03-24T03:17:24.432 INFO:teuthology.orchestra.run.smithi118.stdout: "see_also": [] 2024-03-24T03:17:24.432 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-24T03:17:24.432 INFO:teuthology.orchestra.run.smithi118.stdout: "rwoption4": { 2024-03-24T03:17:24.433 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "rwoption4", 2024-03-24T03:17:24.433 INFO:teuthology.orchestra.run.smithi118.stdout: "type": "str", 2024-03-24T03:17:24.433 INFO:teuthology.orchestra.run.smithi118.stdout: "level": "advanced", 2024-03-24T03:17:24.433 INFO:teuthology.orchestra.run.smithi118.stdout: "flags": 0, 2024-03-24T03:17:24.433 INFO:teuthology.orchestra.run.smithi118.stdout: "default_value": "", 2024-03-24T03:17:24.433 INFO:teuthology.orchestra.run.smithi118.stdout: "min": "", 2024-03-24T03:17:24.434 INFO:teuthology.orchestra.run.smithi118.stdout: "max": "", 2024-03-24T03:17:24.434 INFO:teuthology.orchestra.run.smithi118.stdout: "enum_allowed": [], 2024-03-24T03:17:24.434 INFO:teuthology.orchestra.run.smithi118.stdout: "desc": "", 2024-03-24T03:17:24.434 INFO:teuthology.orchestra.run.smithi118.stdout: "long_desc": "", 2024-03-24T03:17:24.434 INFO:teuthology.orchestra.run.smithi118.stdout: "tags": [], 2024-03-24T03:17:24.435 INFO:teuthology.orchestra.run.smithi118.stdout: "see_also": [] 2024-03-24T03:17:24.435 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-24T03:17:24.435 INFO:teuthology.orchestra.run.smithi118.stdout: "rwoption5": { 2024-03-24T03:17:24.435 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "rwoption5", 2024-03-24T03:17:24.435 INFO:teuthology.orchestra.run.smithi118.stdout: "type": "bool", 2024-03-24T03:17:24.436 INFO:teuthology.orchestra.run.smithi118.stdout: "level": "advanced", 2024-03-24T03:17:24.436 INFO:teuthology.orchestra.run.smithi118.stdout: "flags": 0, 2024-03-24T03:17:24.436 INFO:teuthology.orchestra.run.smithi118.stdout: "default_value": "", 2024-03-24T03:17:24.436 INFO:teuthology.orchestra.run.smithi118.stdout: "min": "", 2024-03-24T03:17:24.436 INFO:teuthology.orchestra.run.smithi118.stdout: "max": "", 2024-03-24T03:17:24.436 INFO:teuthology.orchestra.run.smithi118.stdout: "enum_allowed": [], 2024-03-24T03:17:24.437 INFO:teuthology.orchestra.run.smithi118.stdout: "desc": "", 2024-03-24T03:17:24.437 INFO:teuthology.orchestra.run.smithi118.stdout: "long_desc": "", 2024-03-24T03:17:24.437 INFO:teuthology.orchestra.run.smithi118.stdout: "tags": [], 2024-03-24T03:17:24.437 INFO:teuthology.orchestra.run.smithi118.stdout: "see_also": [] 2024-03-24T03:17:24.437 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-24T03:17:24.437 INFO:teuthology.orchestra.run.smithi118.stdout: "rwoption6": { 2024-03-24T03:17:24.438 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "rwoption6", 2024-03-24T03:17:24.438 INFO:teuthology.orchestra.run.smithi118.stdout: "type": "bool", 2024-03-24T03:17:24.438 INFO:teuthology.orchestra.run.smithi118.stdout: "level": "advanced", 2024-03-24T03:17:24.438 INFO:teuthology.orchestra.run.smithi118.stdout: "flags": 0, 2024-03-24T03:17:24.438 INFO:teuthology.orchestra.run.smithi118.stdout: "default_value": "True", 2024-03-24T03:17:24.439 INFO:teuthology.orchestra.run.smithi118.stdout: "min": "", 2024-03-24T03:17:24.439 INFO:teuthology.orchestra.run.smithi118.stdout: "max": "", 2024-03-24T03:17:24.439 INFO:teuthology.orchestra.run.smithi118.stdout: "enum_allowed": [], 2024-03-24T03:17:24.439 INFO:teuthology.orchestra.run.smithi118.stdout: "desc": "", 2024-03-24T03:17:24.439 INFO:teuthology.orchestra.run.smithi118.stdout: "long_desc": "", 2024-03-24T03:17:24.439 INFO:teuthology.orchestra.run.smithi118.stdout: "tags": [], 2024-03-24T03:17:24.440 INFO:teuthology.orchestra.run.smithi118.stdout: "see_also": [] 2024-03-24T03:17:24.440 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-24T03:17:24.440 INFO:teuthology.orchestra.run.smithi118.stdout: "rwoption7": { 2024-03-24T03:17:24.440 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "rwoption7", 2024-03-24T03:17:24.440 INFO:teuthology.orchestra.run.smithi118.stdout: "type": "int", 2024-03-24T03:17:24.440 INFO:teuthology.orchestra.run.smithi118.stdout: "level": "advanced", 2024-03-24T03:17:24.440 INFO:teuthology.orchestra.run.smithi118.stdout: "flags": 0, 2024-03-24T03:17:24.440 INFO:teuthology.orchestra.run.smithi118.stdout: "default_value": "", 2024-03-24T03:17:24.440 INFO:teuthology.orchestra.run.smithi118.stdout: "min": "1", 2024-03-24T03:17:24.441 INFO:teuthology.orchestra.run.smithi118.stdout: "max": "42", 2024-03-24T03:17:24.441 INFO:teuthology.orchestra.run.smithi118.stdout: "enum_allowed": [], 2024-03-24T03:17:24.441 INFO:teuthology.orchestra.run.smithi118.stdout: "desc": "", 2024-03-24T03:17:24.441 INFO:teuthology.orchestra.run.smithi118.stdout: "long_desc": "", 2024-03-24T03:17:24.441 INFO:teuthology.orchestra.run.smithi118.stdout: "tags": [], 2024-03-24T03:17:24.441 INFO:teuthology.orchestra.run.smithi118.stdout: "see_also": [] 2024-03-24T03:17:24.441 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-24T03:17:24.442 INFO:teuthology.orchestra.run.smithi118.stdout: "testkey": { 2024-03-24T03:17:24.442 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "testkey", 2024-03-24T03:17:24.442 INFO:teuthology.orchestra.run.smithi118.stdout: "type": "str", 2024-03-24T03:17:24.442 INFO:teuthology.orchestra.run.smithi118.stdout: "level": "advanced", 2024-03-24T03:17:24.442 INFO:teuthology.orchestra.run.smithi118.stdout: "flags": 0, 2024-03-24T03:17:24.442 INFO:teuthology.orchestra.run.smithi118.stdout: "default_value": "", 2024-03-24T03:17:24.443 INFO:teuthology.orchestra.run.smithi118.stdout: "min": "", 2024-03-24T03:17:24.443 INFO:teuthology.orchestra.run.smithi118.stdout: "max": "", 2024-03-24T03:17:24.443 INFO:teuthology.orchestra.run.smithi118.stdout: "enum_allowed": [], 2024-03-24T03:17:24.443 INFO:teuthology.orchestra.run.smithi118.stdout: "desc": "", 2024-03-24T03:17:24.443 INFO:teuthology.orchestra.run.smithi118.stdout: "long_desc": "", 2024-03-24T03:17:24.443 INFO:teuthology.orchestra.run.smithi118.stdout: "tags": [], 2024-03-24T03:17:24.443 INFO:teuthology.orchestra.run.smithi118.stdout: "see_also": [] 2024-03-24T03:17:24.444 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-24T03:17:24.444 INFO:teuthology.orchestra.run.smithi118.stdout: "testlkey": { 2024-03-24T03:17:24.444 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "testlkey", 2024-03-24T03:17:24.444 INFO:teuthology.orchestra.run.smithi118.stdout: "type": "str", 2024-03-24T03:17:24.444 INFO:teuthology.orchestra.run.smithi118.stdout: "level": "advanced", 2024-03-24T03:17:24.444 INFO:teuthology.orchestra.run.smithi118.stdout: "flags": 0, 2024-03-24T03:17:24.444 INFO:teuthology.orchestra.run.smithi118.stdout: "default_value": "", 2024-03-24T03:17:24.444 INFO:teuthology.orchestra.run.smithi118.stdout: "min": "", 2024-03-24T03:17:24.444 INFO:teuthology.orchestra.run.smithi118.stdout: "max": "", 2024-03-24T03:17:24.445 INFO:teuthology.orchestra.run.smithi118.stdout: "enum_allowed": [], 2024-03-24T03:17:24.445 INFO:teuthology.orchestra.run.smithi118.stdout: "desc": "", 2024-03-24T03:17:24.445 INFO:teuthology.orchestra.run.smithi118.stdout: "long_desc": "", 2024-03-24T03:17:24.445 INFO:teuthology.orchestra.run.smithi118.stdout: "tags": [], 2024-03-24T03:17:24.445 INFO:teuthology.orchestra.run.smithi118.stdout: "see_also": [] 2024-03-24T03:17:24.445 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-24T03:17:24.445 INFO:teuthology.orchestra.run.smithi118.stdout: "testnewline": { 2024-03-24T03:17:24.445 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "testnewline", 2024-03-24T03:17:24.445 INFO:teuthology.orchestra.run.smithi118.stdout: "type": "str", 2024-03-24T03:17:24.446 INFO:teuthology.orchestra.run.smithi118.stdout: "level": "advanced", 2024-03-24T03:17:24.446 INFO:teuthology.orchestra.run.smithi118.stdout: "flags": 0, 2024-03-24T03:17:24.446 INFO:teuthology.orchestra.run.smithi118.stdout: "default_value": "", 2024-03-24T03:17:24.446 INFO:teuthology.orchestra.run.smithi118.stdout: "min": "", 2024-03-24T03:17:24.446 INFO:teuthology.orchestra.run.smithi118.stdout: "max": "", 2024-03-24T03:17:24.446 INFO:teuthology.orchestra.run.smithi118.stdout: "enum_allowed": [], 2024-03-24T03:17:24.446 INFO:teuthology.orchestra.run.smithi118.stdout: "desc": "", 2024-03-24T03:17:24.446 INFO:teuthology.orchestra.run.smithi118.stdout: "long_desc": "", 2024-03-24T03:17:24.446 INFO:teuthology.orchestra.run.smithi118.stdout: "tags": [], 2024-03-24T03:17:24.446 INFO:teuthology.orchestra.run.smithi118.stdout: "see_also": [] 2024-03-24T03:17:24.447 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-03-24T03:17:24.447 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-03-24T03:17:24.447 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-24T03:17:24.447 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-24T03:17:24.447 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "snap_schedule", 2024-03-24T03:17:24.447 INFO:teuthology.orchestra.run.smithi118.stdout: "can_run": true, 2024-03-24T03:17:24.447 INFO:teuthology.orchestra.run.smithi118.stdout: "error_string": "", 2024-03-24T03:17:24.447 INFO:teuthology.orchestra.run.smithi118.stdout: "module_options": { 2024-03-24T03:17:24.447 INFO:teuthology.orchestra.run.smithi118.stdout: "allow_m_granularity": { 2024-03-24T03:17:24.448 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "allow_m_granularity", 2024-03-24T03:17:24.448 INFO:teuthology.orchestra.run.smithi118.stdout: "type": "bool", 2024-03-24T03:17:24.448 INFO:teuthology.orchestra.run.smithi118.stdout: "level": "advanced", 2024-03-24T03:17:24.448 INFO:teuthology.orchestra.run.smithi118.stdout: "flags": 1, 2024-03-24T03:17:24.448 INFO:teuthology.orchestra.run.smithi118.stdout: "default_value": "False", 2024-03-24T03:17:24.448 INFO:teuthology.orchestra.run.smithi118.stdout: "min": "", 2024-03-24T03:17:24.448 INFO:teuthology.orchestra.run.smithi118.stdout: "max": "", 2024-03-24T03:17:24.448 INFO:teuthology.orchestra.run.smithi118.stdout: "enum_allowed": [], 2024-03-24T03:17:24.448 INFO:teuthology.orchestra.run.smithi118.stdout: "desc": "allow minute scheduled snapshots", 2024-03-24T03:17:24.448 INFO:teuthology.orchestra.run.smithi118.stdout: "long_desc": "", 2024-03-24T03:17:24.449 INFO:teuthology.orchestra.run.smithi118.stdout: "tags": [], 2024-03-24T03:17:24.449 INFO:teuthology.orchestra.run.smithi118.stdout: "see_also": [] 2024-03-24T03:17:24.449 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-24T03:17:24.449 INFO:teuthology.orchestra.run.smithi118.stdout: "dump_on_update": { 2024-03-24T03:17:24.449 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "dump_on_update", 2024-03-24T03:17:24.449 INFO:teuthology.orchestra.run.smithi118.stdout: "type": "bool", 2024-03-24T03:17:24.449 INFO:teuthology.orchestra.run.smithi118.stdout: "level": "advanced", 2024-03-24T03:17:24.449 INFO:teuthology.orchestra.run.smithi118.stdout: "flags": 1, 2024-03-24T03:17:24.449 INFO:teuthology.orchestra.run.smithi118.stdout: "default_value": "False", 2024-03-24T03:17:24.450 INFO:teuthology.orchestra.run.smithi118.stdout: "min": "", 2024-03-24T03:17:24.450 INFO:teuthology.orchestra.run.smithi118.stdout: "max": "", 2024-03-24T03:17:24.450 INFO:teuthology.orchestra.run.smithi118.stdout: "enum_allowed": [], 2024-03-24T03:17:24.450 INFO:teuthology.orchestra.run.smithi118.stdout: "desc": "dump database to debug log on update", 2024-03-24T03:17:24.450 INFO:teuthology.orchestra.run.smithi118.stdout: "long_desc": "", 2024-03-24T03:17:24.450 INFO:teuthology.orchestra.run.smithi118.stdout: "tags": [], 2024-03-24T03:17:24.450 INFO:teuthology.orchestra.run.smithi118.stdout: "see_also": [] 2024-03-24T03:17:24.450 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-24T03:17:24.450 INFO:teuthology.orchestra.run.smithi118.stdout: "log_level": { 2024-03-24T03:17:24.450 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "log_level", 2024-03-24T03:17:24.451 INFO:teuthology.orchestra.run.smithi118.stdout: "type": "str", 2024-03-24T03:17:24.451 INFO:teuthology.orchestra.run.smithi118.stdout: "level": "advanced", 2024-03-24T03:17:24.451 INFO:teuthology.orchestra.run.smithi118.stdout: "flags": 1, 2024-03-24T03:17:24.451 INFO:teuthology.orchestra.run.smithi118.stdout: "default_value": "", 2024-03-24T03:17:24.451 INFO:teuthology.orchestra.run.smithi118.stdout: "min": "", 2024-03-24T03:17:24.451 INFO:teuthology.orchestra.run.smithi118.stdout: "max": "", 2024-03-24T03:17:24.451 INFO:teuthology.orchestra.run.smithi118.stdout: "enum_allowed": [ 2024-03-24T03:17:24.451 INFO:teuthology.orchestra.run.smithi118.stdout: "", 2024-03-24T03:17:24.451 INFO:teuthology.orchestra.run.smithi118.stdout: "critical", 2024-03-24T03:17:24.452 INFO:teuthology.orchestra.run.smithi118.stdout: "debug", 2024-03-24T03:17:24.452 INFO:teuthology.orchestra.run.smithi118.stdout: "error", 2024-03-24T03:17:24.452 INFO:teuthology.orchestra.run.smithi118.stdout: "info", 2024-03-24T03:17:24.452 INFO:teuthology.orchestra.run.smithi118.stdout: "warning" 2024-03-24T03:17:24.452 INFO:teuthology.orchestra.run.smithi118.stdout: ], 2024-03-24T03:17:24.452 INFO:teuthology.orchestra.run.smithi118.stdout: "desc": "", 2024-03-24T03:17:24.452 INFO:teuthology.orchestra.run.smithi118.stdout: "long_desc": "", 2024-03-24T03:17:24.452 INFO:teuthology.orchestra.run.smithi118.stdout: "tags": [], 2024-03-24T03:17:24.452 INFO:teuthology.orchestra.run.smithi118.stdout: "see_also": [] 2024-03-24T03:17:24.452 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-24T03:17:24.453 INFO:teuthology.orchestra.run.smithi118.stdout: "log_to_cluster": { 2024-03-24T03:17:24.453 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "log_to_cluster", 2024-03-24T03:17:24.453 INFO:teuthology.orchestra.run.smithi118.stdout: "type": "bool", 2024-03-24T03:17:24.453 INFO:teuthology.orchestra.run.smithi118.stdout: "level": "advanced", 2024-03-24T03:17:24.453 INFO:teuthology.orchestra.run.smithi118.stdout: "flags": 1, 2024-03-24T03:17:24.453 INFO:teuthology.orchestra.run.smithi118.stdout: "default_value": "False", 2024-03-24T03:17:24.453 INFO:teuthology.orchestra.run.smithi118.stdout: "min": "", 2024-03-24T03:17:24.453 INFO:teuthology.orchestra.run.smithi118.stdout: "max": "", 2024-03-24T03:17:24.453 INFO:teuthology.orchestra.run.smithi118.stdout: "enum_allowed": [], 2024-03-24T03:17:24.454 INFO:teuthology.orchestra.run.smithi118.stdout: "desc": "", 2024-03-24T03:17:24.454 INFO:teuthology.orchestra.run.smithi118.stdout: "long_desc": "", 2024-03-24T03:17:24.454 INFO:teuthology.orchestra.run.smithi118.stdout: "tags": [], 2024-03-24T03:17:24.454 INFO:teuthology.orchestra.run.smithi118.stdout: "see_also": [] 2024-03-24T03:17:24.454 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-24T03:17:24.454 INFO:teuthology.orchestra.run.smithi118.stdout: "log_to_cluster_level": { 2024-03-24T03:17:24.454 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "log_to_cluster_level", 2024-03-24T03:17:24.454 INFO:teuthology.orchestra.run.smithi118.stdout: "type": "str", 2024-03-24T03:17:24.455 INFO:teuthology.orchestra.run.smithi118.stdout: "level": "advanced", 2024-03-24T03:17:24.455 INFO:teuthology.orchestra.run.smithi118.stdout: "flags": 1, 2024-03-24T03:17:24.455 INFO:teuthology.orchestra.run.smithi118.stdout: "default_value": "info", 2024-03-24T03:17:24.455 INFO:teuthology.orchestra.run.smithi118.stdout: "min": "", 2024-03-24T03:17:24.455 INFO:teuthology.orchestra.run.smithi118.stdout: "max": "", 2024-03-24T03:17:24.455 INFO:teuthology.orchestra.run.smithi118.stdout: "enum_allowed": [ 2024-03-24T03:17:24.455 INFO:teuthology.orchestra.run.smithi118.stdout: "", 2024-03-24T03:17:24.455 INFO:teuthology.orchestra.run.smithi118.stdout: "critical", 2024-03-24T03:17:24.455 INFO:teuthology.orchestra.run.smithi118.stdout: "debug", 2024-03-24T03:17:24.456 INFO:teuthology.orchestra.run.smithi118.stdout: "error", 2024-03-24T03:17:24.456 INFO:teuthology.orchestra.run.smithi118.stdout: "info", 2024-03-24T03:17:24.456 INFO:teuthology.orchestra.run.smithi118.stdout: "warning" 2024-03-24T03:17:24.456 INFO:teuthology.orchestra.run.smithi118.stdout: ], 2024-03-24T03:17:24.456 INFO:teuthology.orchestra.run.smithi118.stdout: "desc": "", 2024-03-24T03:17:24.456 INFO:teuthology.orchestra.run.smithi118.stdout: "long_desc": "", 2024-03-24T03:17:24.456 INFO:teuthology.orchestra.run.smithi118.stdout: "tags": [], 2024-03-24T03:17:24.456 INFO:teuthology.orchestra.run.smithi118.stdout: "see_also": [] 2024-03-24T03:17:24.456 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-24T03:17:24.457 INFO:teuthology.orchestra.run.smithi118.stdout: "log_to_file": { 2024-03-24T03:17:24.457 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "log_to_file", 2024-03-24T03:17:24.457 INFO:teuthology.orchestra.run.smithi118.stdout: "type": "bool", 2024-03-24T03:17:24.457 INFO:teuthology.orchestra.run.smithi118.stdout: "level": "advanced", 2024-03-24T03:17:24.457 INFO:teuthology.orchestra.run.smithi118.stdout: "flags": 1, 2024-03-24T03:17:24.457 INFO:teuthology.orchestra.run.smithi118.stdout: "default_value": "False", 2024-03-24T03:17:24.457 INFO:teuthology.orchestra.run.smithi118.stdout: "min": "", 2024-03-24T03:17:24.457 INFO:teuthology.orchestra.run.smithi118.stdout: "max": "", 2024-03-24T03:17:24.457 INFO:teuthology.orchestra.run.smithi118.stdout: "enum_allowed": [], 2024-03-24T03:17:24.457 INFO:teuthology.orchestra.run.smithi118.stdout: "desc": "", 2024-03-24T03:17:24.458 INFO:teuthology.orchestra.run.smithi118.stdout: "long_desc": "", 2024-03-24T03:17:24.458 INFO:teuthology.orchestra.run.smithi118.stdout: "tags": [], 2024-03-24T03:17:24.458 INFO:teuthology.orchestra.run.smithi118.stdout: "see_also": [] 2024-03-24T03:17:24.458 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-03-24T03:17:24.458 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-03-24T03:17:24.458 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-24T03:17:24.458 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-24T03:17:24.458 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "stats", 2024-03-24T03:17:24.458 INFO:teuthology.orchestra.run.smithi118.stdout: "can_run": true, 2024-03-24T03:17:24.459 INFO:teuthology.orchestra.run.smithi118.stdout: "error_string": "", 2024-03-24T03:17:24.459 INFO:teuthology.orchestra.run.smithi118.stdout: "module_options": { 2024-03-24T03:17:24.459 INFO:teuthology.orchestra.run.smithi118.stdout: "log_level": { 2024-03-24T03:17:24.459 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "log_level", 2024-03-24T03:17:24.459 INFO:teuthology.orchestra.run.smithi118.stdout: "type": "str", 2024-03-24T03:17:24.459 INFO:teuthology.orchestra.run.smithi118.stdout: "level": "advanced", 2024-03-24T03:17:24.459 INFO:teuthology.orchestra.run.smithi118.stdout: "flags": 1, 2024-03-24T03:17:24.459 INFO:teuthology.orchestra.run.smithi118.stdout: "default_value": "", 2024-03-24T03:17:24.459 INFO:teuthology.orchestra.run.smithi118.stdout: "min": "", 2024-03-24T03:17:24.459 INFO:teuthology.orchestra.run.smithi118.stdout: "max": "", 2024-03-24T03:17:24.460 INFO:teuthology.orchestra.run.smithi118.stdout: "enum_allowed": [ 2024-03-24T03:17:24.460 INFO:teuthology.orchestra.run.smithi118.stdout: "", 2024-03-24T03:17:24.460 INFO:teuthology.orchestra.run.smithi118.stdout: "critical", 2024-03-24T03:17:24.460 INFO:teuthology.orchestra.run.smithi118.stdout: "debug", 2024-03-24T03:17:24.460 INFO:teuthology.orchestra.run.smithi118.stdout: "error", 2024-03-24T03:17:24.460 INFO:teuthology.orchestra.run.smithi118.stdout: "info", 2024-03-24T03:17:24.460 INFO:teuthology.orchestra.run.smithi118.stdout: "warning" 2024-03-24T03:17:24.460 INFO:teuthology.orchestra.run.smithi118.stdout: ], 2024-03-24T03:17:24.460 INFO:teuthology.orchestra.run.smithi118.stdout: "desc": "", 2024-03-24T03:17:24.461 INFO:teuthology.orchestra.run.smithi118.stdout: "long_desc": "", 2024-03-24T03:17:24.461 INFO:teuthology.orchestra.run.smithi118.stdout: "tags": [], 2024-03-24T03:17:24.461 INFO:teuthology.orchestra.run.smithi118.stdout: "see_also": [] 2024-03-24T03:17:24.461 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-24T03:17:24.461 INFO:teuthology.orchestra.run.smithi118.stdout: "log_to_cluster": { 2024-03-24T03:17:24.461 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "log_to_cluster", 2024-03-24T03:17:24.461 INFO:teuthology.orchestra.run.smithi118.stdout: "type": "bool", 2024-03-24T03:17:24.461 INFO:teuthology.orchestra.run.smithi118.stdout: "level": "advanced", 2024-03-24T03:17:24.461 INFO:teuthology.orchestra.run.smithi118.stdout: "flags": 1, 2024-03-24T03:17:24.462 INFO:teuthology.orchestra.run.smithi118.stdout: "default_value": "False", 2024-03-24T03:17:24.462 INFO:teuthology.orchestra.run.smithi118.stdout: "min": "", 2024-03-24T03:17:24.462 INFO:teuthology.orchestra.run.smithi118.stdout: "max": "", 2024-03-24T03:17:24.462 INFO:teuthology.orchestra.run.smithi118.stdout: "enum_allowed": [], 2024-03-24T03:17:24.462 INFO:teuthology.orchestra.run.smithi118.stdout: "desc": "", 2024-03-24T03:17:24.462 INFO:teuthology.orchestra.run.smithi118.stdout: "long_desc": "", 2024-03-24T03:17:24.462 INFO:teuthology.orchestra.run.smithi118.stdout: "tags": [], 2024-03-24T03:17:24.462 INFO:teuthology.orchestra.run.smithi118.stdout: "see_also": [] 2024-03-24T03:17:24.462 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-24T03:17:24.463 INFO:teuthology.orchestra.run.smithi118.stdout: "log_to_cluster_level": { 2024-03-24T03:17:24.463 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "log_to_cluster_level", 2024-03-24T03:17:24.463 INFO:teuthology.orchestra.run.smithi118.stdout: "type": "str", 2024-03-24T03:17:24.463 INFO:teuthology.orchestra.run.smithi118.stdout: "level": "advanced", 2024-03-24T03:17:24.463 INFO:teuthology.orchestra.run.smithi118.stdout: "flags": 1, 2024-03-24T03:17:24.463 INFO:teuthology.orchestra.run.smithi118.stdout: "default_value": "info", 2024-03-24T03:17:24.463 INFO:teuthology.orchestra.run.smithi118.stdout: "min": "", 2024-03-24T03:17:24.463 INFO:teuthology.orchestra.run.smithi118.stdout: "max": "", 2024-03-24T03:17:24.463 INFO:teuthology.orchestra.run.smithi118.stdout: "enum_allowed": [ 2024-03-24T03:17:24.464 INFO:teuthology.orchestra.run.smithi118.stdout: "", 2024-03-24T03:17:24.464 INFO:teuthology.orchestra.run.smithi118.stdout: "critical", 2024-03-24T03:17:24.464 INFO:teuthology.orchestra.run.smithi118.stdout: "debug", 2024-03-24T03:17:24.464 INFO:teuthology.orchestra.run.smithi118.stdout: "error", 2024-03-24T03:17:24.464 INFO:teuthology.orchestra.run.smithi118.stdout: "info", 2024-03-24T03:17:24.464 INFO:teuthology.orchestra.run.smithi118.stdout: "warning" 2024-03-24T03:17:24.464 INFO:teuthology.orchestra.run.smithi118.stdout: ], 2024-03-24T03:17:24.464 INFO:teuthology.orchestra.run.smithi118.stdout: "desc": "", 2024-03-24T03:17:24.464 INFO:teuthology.orchestra.run.smithi118.stdout: "long_desc": "", 2024-03-24T03:17:24.465 INFO:teuthology.orchestra.run.smithi118.stdout: "tags": [], 2024-03-24T03:17:24.465 INFO:teuthology.orchestra.run.smithi118.stdout: "see_also": [] 2024-03-24T03:17:24.465 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-24T03:17:24.465 INFO:teuthology.orchestra.run.smithi118.stdout: "log_to_file": { 2024-03-24T03:17:24.465 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "log_to_file", 2024-03-24T03:17:24.465 INFO:teuthology.orchestra.run.smithi118.stdout: "type": "bool", 2024-03-24T03:17:24.465 INFO:teuthology.orchestra.run.smithi118.stdout: "level": "advanced", 2024-03-24T03:17:24.465 INFO:teuthology.orchestra.run.smithi118.stdout: "flags": 1, 2024-03-24T03:17:24.465 INFO:teuthology.orchestra.run.smithi118.stdout: "default_value": "False", 2024-03-24T03:17:24.466 INFO:teuthology.orchestra.run.smithi118.stdout: "min": "", 2024-03-24T03:17:24.466 INFO:teuthology.orchestra.run.smithi118.stdout: "max": "", 2024-03-24T03:17:24.466 INFO:teuthology.orchestra.run.smithi118.stdout: "enum_allowed": [], 2024-03-24T03:17:24.466 INFO:teuthology.orchestra.run.smithi118.stdout: "desc": "", 2024-03-24T03:17:24.466 INFO:teuthology.orchestra.run.smithi118.stdout: "long_desc": "", 2024-03-24T03:17:24.466 INFO:teuthology.orchestra.run.smithi118.stdout: "tags": [], 2024-03-24T03:17:24.466 INFO:teuthology.orchestra.run.smithi118.stdout: "see_also": [] 2024-03-24T03:17:24.466 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-03-24T03:17:24.466 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-03-24T03:17:24.466 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-24T03:17:24.467 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-24T03:17:24.467 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "status", 2024-03-24T03:17:24.467 INFO:teuthology.orchestra.run.smithi118.stdout: "can_run": true, 2024-03-24T03:17:24.467 INFO:teuthology.orchestra.run.smithi118.stdout: "error_string": "", 2024-03-24T03:17:24.467 INFO:teuthology.orchestra.run.smithi118.stdout: "module_options": { 2024-03-24T03:17:24.467 INFO:teuthology.orchestra.run.smithi118.stdout: "log_level": { 2024-03-24T03:17:24.467 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "log_level", 2024-03-24T03:17:24.467 INFO:teuthology.orchestra.run.smithi118.stdout: "type": "str", 2024-03-24T03:17:24.467 INFO:teuthology.orchestra.run.smithi118.stdout: "level": "advanced", 2024-03-24T03:17:24.468 INFO:teuthology.orchestra.run.smithi118.stdout: "flags": 1, 2024-03-24T03:17:24.468 INFO:teuthology.orchestra.run.smithi118.stdout: "default_value": "", 2024-03-24T03:17:24.468 INFO:teuthology.orchestra.run.smithi118.stdout: "min": "", 2024-03-24T03:17:24.468 INFO:teuthology.orchestra.run.smithi118.stdout: "max": "", 2024-03-24T03:17:24.468 INFO:teuthology.orchestra.run.smithi118.stdout: "enum_allowed": [ 2024-03-24T03:17:24.468 INFO:teuthology.orchestra.run.smithi118.stdout: "", 2024-03-24T03:17:24.468 INFO:teuthology.orchestra.run.smithi118.stdout: "critical", 2024-03-24T03:17:24.468 INFO:teuthology.orchestra.run.smithi118.stdout: "debug", 2024-03-24T03:17:24.468 INFO:teuthology.orchestra.run.smithi118.stdout: "error", 2024-03-24T03:17:24.468 INFO:teuthology.orchestra.run.smithi118.stdout: "info", 2024-03-24T03:17:24.469 INFO:teuthology.orchestra.run.smithi118.stdout: "warning" 2024-03-24T03:17:24.469 INFO:teuthology.orchestra.run.smithi118.stdout: ], 2024-03-24T03:17:24.469 INFO:teuthology.orchestra.run.smithi118.stdout: "desc": "", 2024-03-24T03:17:24.469 INFO:teuthology.orchestra.run.smithi118.stdout: "long_desc": "", 2024-03-24T03:17:24.469 INFO:teuthology.orchestra.run.smithi118.stdout: "tags": [], 2024-03-24T03:17:24.469 INFO:teuthology.orchestra.run.smithi118.stdout: "see_also": [] 2024-03-24T03:17:24.469 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-24T03:17:24.469 INFO:teuthology.orchestra.run.smithi118.stdout: "log_to_cluster": { 2024-03-24T03:17:24.469 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "log_to_cluster", 2024-03-24T03:17:24.470 INFO:teuthology.orchestra.run.smithi118.stdout: "type": "bool", 2024-03-24T03:17:24.470 INFO:teuthology.orchestra.run.smithi118.stdout: "level": "advanced", 2024-03-24T03:17:24.470 INFO:teuthology.orchestra.run.smithi118.stdout: "flags": 1, 2024-03-24T03:17:24.470 INFO:teuthology.orchestra.run.smithi118.stdout: "default_value": "False", 2024-03-24T03:17:24.470 INFO:teuthology.orchestra.run.smithi118.stdout: "min": "", 2024-03-24T03:17:24.470 INFO:teuthology.orchestra.run.smithi118.stdout: "max": "", 2024-03-24T03:17:24.470 INFO:teuthology.orchestra.run.smithi118.stdout: "enum_allowed": [], 2024-03-24T03:17:24.470 INFO:teuthology.orchestra.run.smithi118.stdout: "desc": "", 2024-03-24T03:17:24.470 INFO:teuthology.orchestra.run.smithi118.stdout: "long_desc": "", 2024-03-24T03:17:24.470 INFO:teuthology.orchestra.run.smithi118.stdout: "tags": [], 2024-03-24T03:17:24.471 INFO:teuthology.orchestra.run.smithi118.stdout: "see_also": [] 2024-03-24T03:17:24.471 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-24T03:17:24.471 INFO:teuthology.orchestra.run.smithi118.stdout: "log_to_cluster_level": { 2024-03-24T03:17:24.471 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "log_to_cluster_level", 2024-03-24T03:17:24.471 INFO:teuthology.orchestra.run.smithi118.stdout: "type": "str", 2024-03-24T03:17:24.471 INFO:teuthology.orchestra.run.smithi118.stdout: "level": "advanced", 2024-03-24T03:17:24.471 INFO:teuthology.orchestra.run.smithi118.stdout: "flags": 1, 2024-03-24T03:17:24.471 INFO:teuthology.orchestra.run.smithi118.stdout: "default_value": "info", 2024-03-24T03:17:24.471 INFO:teuthology.orchestra.run.smithi118.stdout: "min": "", 2024-03-24T03:17:24.472 INFO:teuthology.orchestra.run.smithi118.stdout: "max": "", 2024-03-24T03:17:24.472 INFO:teuthology.orchestra.run.smithi118.stdout: "enum_allowed": [ 2024-03-24T03:17:24.472 INFO:teuthology.orchestra.run.smithi118.stdout: "", 2024-03-24T03:17:24.472 INFO:teuthology.orchestra.run.smithi118.stdout: "critical", 2024-03-24T03:17:24.472 INFO:teuthology.orchestra.run.smithi118.stdout: "debug", 2024-03-24T03:17:24.472 INFO:teuthology.orchestra.run.smithi118.stdout: "error", 2024-03-24T03:17:24.472 INFO:teuthology.orchestra.run.smithi118.stdout: "info", 2024-03-24T03:17:24.472 INFO:teuthology.orchestra.run.smithi118.stdout: "warning" 2024-03-24T03:17:24.472 INFO:teuthology.orchestra.run.smithi118.stdout: ], 2024-03-24T03:17:24.472 INFO:teuthology.orchestra.run.smithi118.stdout: "desc": "", 2024-03-24T03:17:24.473 INFO:teuthology.orchestra.run.smithi118.stdout: "long_desc": "", 2024-03-24T03:17:24.473 INFO:teuthology.orchestra.run.smithi118.stdout: "tags": [], 2024-03-24T03:17:24.473 INFO:teuthology.orchestra.run.smithi118.stdout: "see_also": [] 2024-03-24T03:17:24.473 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-24T03:17:24.473 INFO:teuthology.orchestra.run.smithi118.stdout: "log_to_file": { 2024-03-24T03:17:24.473 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "log_to_file", 2024-03-24T03:17:24.473 INFO:teuthology.orchestra.run.smithi118.stdout: "type": "bool", 2024-03-24T03:17:24.473 INFO:teuthology.orchestra.run.smithi118.stdout: "level": "advanced", 2024-03-24T03:17:24.473 INFO:teuthology.orchestra.run.smithi118.stdout: "flags": 1, 2024-03-24T03:17:24.474 INFO:teuthology.orchestra.run.smithi118.stdout: "default_value": "False", 2024-03-24T03:17:24.474 INFO:teuthology.orchestra.run.smithi118.stdout: "min": "", 2024-03-24T03:17:24.474 INFO:teuthology.orchestra.run.smithi118.stdout: "max": "", 2024-03-24T03:17:24.474 INFO:teuthology.orchestra.run.smithi118.stdout: "enum_allowed": [], 2024-03-24T03:17:24.474 INFO:teuthology.orchestra.run.smithi118.stdout: "desc": "", 2024-03-24T03:17:24.474 INFO:teuthology.orchestra.run.smithi118.stdout: "long_desc": "", 2024-03-24T03:17:24.474 INFO:teuthology.orchestra.run.smithi118.stdout: "tags": [], 2024-03-24T03:17:24.474 INFO:teuthology.orchestra.run.smithi118.stdout: "see_also": [] 2024-03-24T03:17:24.474 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-03-24T03:17:24.475 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-03-24T03:17:24.475 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-24T03:17:24.475 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-24T03:17:24.475 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "telegraf", 2024-03-24T03:17:24.475 INFO:teuthology.orchestra.run.smithi118.stdout: "can_run": true, 2024-03-24T03:17:24.475 INFO:teuthology.orchestra.run.smithi118.stdout: "error_string": "", 2024-03-24T03:17:24.475 INFO:teuthology.orchestra.run.smithi118.stdout: "module_options": { 2024-03-24T03:17:24.475 INFO:teuthology.orchestra.run.smithi118.stdout: "address": { 2024-03-24T03:17:24.475 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "address", 2024-03-24T03:17:24.476 INFO:teuthology.orchestra.run.smithi118.stdout: "type": "str", 2024-03-24T03:17:24.476 INFO:teuthology.orchestra.run.smithi118.stdout: "level": "advanced", 2024-03-24T03:17:24.476 INFO:teuthology.orchestra.run.smithi118.stdout: "flags": 0, 2024-03-24T03:17:24.476 INFO:teuthology.orchestra.run.smithi118.stdout: "default_value": "unixgram:///tmp/telegraf.sock", 2024-03-24T03:17:24.476 INFO:teuthology.orchestra.run.smithi118.stdout: "min": "", 2024-03-24T03:17:24.476 INFO:teuthology.orchestra.run.smithi118.stdout: "max": "", 2024-03-24T03:17:24.476 INFO:teuthology.orchestra.run.smithi118.stdout: "enum_allowed": [], 2024-03-24T03:17:24.476 INFO:teuthology.orchestra.run.smithi118.stdout: "desc": "", 2024-03-24T03:17:24.476 INFO:teuthology.orchestra.run.smithi118.stdout: "long_desc": "", 2024-03-24T03:17:24.476 INFO:teuthology.orchestra.run.smithi118.stdout: "tags": [], 2024-03-24T03:17:24.477 INFO:teuthology.orchestra.run.smithi118.stdout: "see_also": [] 2024-03-24T03:17:24.477 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-24T03:17:24.477 INFO:teuthology.orchestra.run.smithi118.stdout: "interval": { 2024-03-24T03:17:24.477 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "interval", 2024-03-24T03:17:24.477 INFO:teuthology.orchestra.run.smithi118.stdout: "type": "secs", 2024-03-24T03:17:24.477 INFO:teuthology.orchestra.run.smithi118.stdout: "level": "advanced", 2024-03-24T03:17:24.477 INFO:teuthology.orchestra.run.smithi118.stdout: "flags": 0, 2024-03-24T03:17:24.477 INFO:teuthology.orchestra.run.smithi118.stdout: "default_value": "15", 2024-03-24T03:17:24.477 INFO:teuthology.orchestra.run.smithi118.stdout: "min": "", 2024-03-24T03:17:24.477 INFO:teuthology.orchestra.run.smithi118.stdout: "max": "", 2024-03-24T03:17:24.478 INFO:teuthology.orchestra.run.smithi118.stdout: "enum_allowed": [], 2024-03-24T03:17:24.478 INFO:teuthology.orchestra.run.smithi118.stdout: "desc": "", 2024-03-24T03:17:24.478 INFO:teuthology.orchestra.run.smithi118.stdout: "long_desc": "", 2024-03-24T03:17:24.478 INFO:teuthology.orchestra.run.smithi118.stdout: "tags": [], 2024-03-24T03:17:24.478 INFO:teuthology.orchestra.run.smithi118.stdout: "see_also": [] 2024-03-24T03:17:24.478 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-24T03:17:24.478 INFO:teuthology.orchestra.run.smithi118.stdout: "log_level": { 2024-03-24T03:17:24.478 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "log_level", 2024-03-24T03:17:24.478 INFO:teuthology.orchestra.run.smithi118.stdout: "type": "str", 2024-03-24T03:17:24.479 INFO:teuthology.orchestra.run.smithi118.stdout: "level": "advanced", 2024-03-24T03:17:24.479 INFO:teuthology.orchestra.run.smithi118.stdout: "flags": 1, 2024-03-24T03:17:24.479 INFO:teuthology.orchestra.run.smithi118.stdout: "default_value": "", 2024-03-24T03:17:24.479 INFO:teuthology.orchestra.run.smithi118.stdout: "min": "", 2024-03-24T03:17:24.479 INFO:teuthology.orchestra.run.smithi118.stdout: "max": "", 2024-03-24T03:17:24.479 INFO:teuthology.orchestra.run.smithi118.stdout: "enum_allowed": [ 2024-03-24T03:17:24.479 INFO:teuthology.orchestra.run.smithi118.stdout: "", 2024-03-24T03:17:24.479 INFO:teuthology.orchestra.run.smithi118.stdout: "critical", 2024-03-24T03:17:24.479 INFO:teuthology.orchestra.run.smithi118.stdout: "debug", 2024-03-24T03:17:24.479 INFO:teuthology.orchestra.run.smithi118.stdout: "error", 2024-03-24T03:17:24.480 INFO:teuthology.orchestra.run.smithi118.stdout: "info", 2024-03-24T03:17:24.480 INFO:teuthology.orchestra.run.smithi118.stdout: "warning" 2024-03-24T03:17:24.480 INFO:teuthology.orchestra.run.smithi118.stdout: ], 2024-03-24T03:17:24.480 INFO:teuthology.orchestra.run.smithi118.stdout: "desc": "", 2024-03-24T03:17:24.480 INFO:teuthology.orchestra.run.smithi118.stdout: "long_desc": "", 2024-03-24T03:17:24.480 INFO:teuthology.orchestra.run.smithi118.stdout: "tags": [], 2024-03-24T03:17:24.480 INFO:teuthology.orchestra.run.smithi118.stdout: "see_also": [] 2024-03-24T03:17:24.480 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-24T03:17:24.480 INFO:teuthology.orchestra.run.smithi118.stdout: "log_to_cluster": { 2024-03-24T03:17:24.481 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "log_to_cluster", 2024-03-24T03:17:24.481 INFO:teuthology.orchestra.run.smithi118.stdout: "type": "bool", 2024-03-24T03:17:24.481 INFO:teuthology.orchestra.run.smithi118.stdout: "level": "advanced", 2024-03-24T03:17:24.481 INFO:teuthology.orchestra.run.smithi118.stdout: "flags": 1, 2024-03-24T03:17:24.481 INFO:teuthology.orchestra.run.smithi118.stdout: "default_value": "False", 2024-03-24T03:17:24.481 INFO:teuthology.orchestra.run.smithi118.stdout: "min": "", 2024-03-24T03:17:24.481 INFO:teuthology.orchestra.run.smithi118.stdout: "max": "", 2024-03-24T03:17:24.481 INFO:teuthology.orchestra.run.smithi118.stdout: "enum_allowed": [], 2024-03-24T03:17:24.481 INFO:teuthology.orchestra.run.smithi118.stdout: "desc": "", 2024-03-24T03:17:24.481 INFO:teuthology.orchestra.run.smithi118.stdout: "long_desc": "", 2024-03-24T03:17:24.482 INFO:teuthology.orchestra.run.smithi118.stdout: "tags": [], 2024-03-24T03:17:24.482 INFO:teuthology.orchestra.run.smithi118.stdout: "see_also": [] 2024-03-24T03:17:24.482 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-24T03:17:24.482 INFO:teuthology.orchestra.run.smithi118.stdout: "log_to_cluster_level": { 2024-03-24T03:17:24.482 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "log_to_cluster_level", 2024-03-24T03:17:24.482 INFO:teuthology.orchestra.run.smithi118.stdout: "type": "str", 2024-03-24T03:17:24.482 INFO:teuthology.orchestra.run.smithi118.stdout: "level": "advanced", 2024-03-24T03:17:24.482 INFO:teuthology.orchestra.run.smithi118.stdout: "flags": 1, 2024-03-24T03:17:24.482 INFO:teuthology.orchestra.run.smithi118.stdout: "default_value": "info", 2024-03-24T03:17:24.483 INFO:teuthology.orchestra.run.smithi118.stdout: "min": "", 2024-03-24T03:17:24.483 INFO:teuthology.orchestra.run.smithi118.stdout: "max": "", 2024-03-24T03:17:24.483 INFO:teuthology.orchestra.run.smithi118.stdout: "enum_allowed": [ 2024-03-24T03:17:24.483 INFO:teuthology.orchestra.run.smithi118.stdout: "", 2024-03-24T03:17:24.483 INFO:teuthology.orchestra.run.smithi118.stdout: "critical", 2024-03-24T03:17:24.483 INFO:teuthology.orchestra.run.smithi118.stdout: "debug", 2024-03-24T03:17:24.483 INFO:teuthology.orchestra.run.smithi118.stdout: "error", 2024-03-24T03:17:24.483 INFO:teuthology.orchestra.run.smithi118.stdout: "info", 2024-03-24T03:17:24.483 INFO:teuthology.orchestra.run.smithi118.stdout: "warning" 2024-03-24T03:17:24.484 INFO:teuthology.orchestra.run.smithi118.stdout: ], 2024-03-24T03:17:24.484 INFO:teuthology.orchestra.run.smithi118.stdout: "desc": "", 2024-03-24T03:17:24.484 INFO:teuthology.orchestra.run.smithi118.stdout: "long_desc": "", 2024-03-24T03:17:24.484 INFO:teuthology.orchestra.run.smithi118.stdout: "tags": [], 2024-03-24T03:17:24.484 INFO:teuthology.orchestra.run.smithi118.stdout: "see_also": [] 2024-03-24T03:17:24.484 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-24T03:17:24.484 INFO:teuthology.orchestra.run.smithi118.stdout: "log_to_file": { 2024-03-24T03:17:24.484 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "log_to_file", 2024-03-24T03:17:24.484 INFO:teuthology.orchestra.run.smithi118.stdout: "type": "bool", 2024-03-24T03:17:24.485 INFO:teuthology.orchestra.run.smithi118.stdout: "level": "advanced", 2024-03-24T03:17:24.485 INFO:teuthology.orchestra.run.smithi118.stdout: "flags": 1, 2024-03-24T03:17:24.485 INFO:teuthology.orchestra.run.smithi118.stdout: "default_value": "False", 2024-03-24T03:17:24.485 INFO:teuthology.orchestra.run.smithi118.stdout: "min": "", 2024-03-24T03:17:24.485 INFO:teuthology.orchestra.run.smithi118.stdout: "max": "", 2024-03-24T03:17:24.485 INFO:teuthology.orchestra.run.smithi118.stdout: "enum_allowed": [], 2024-03-24T03:17:24.485 INFO:teuthology.orchestra.run.smithi118.stdout: "desc": "", 2024-03-24T03:17:24.485 INFO:teuthology.orchestra.run.smithi118.stdout: "long_desc": "", 2024-03-24T03:17:24.485 INFO:teuthology.orchestra.run.smithi118.stdout: "tags": [], 2024-03-24T03:17:24.485 INFO:teuthology.orchestra.run.smithi118.stdout: "see_also": [] 2024-03-24T03:17:24.486 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-03-24T03:17:24.486 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-03-24T03:17:24.486 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-24T03:17:24.486 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-24T03:17:24.486 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "telemetry", 2024-03-24T03:17:24.486 INFO:teuthology.orchestra.run.smithi118.stdout: "can_run": true, 2024-03-24T03:17:24.486 INFO:teuthology.orchestra.run.smithi118.stdout: "error_string": "", 2024-03-24T03:17:24.486 INFO:teuthology.orchestra.run.smithi118.stdout: "module_options": { 2024-03-24T03:17:24.486 INFO:teuthology.orchestra.run.smithi118.stdout: "channel_basic": { 2024-03-24T03:17:24.487 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "channel_basic", 2024-03-24T03:17:24.487 INFO:teuthology.orchestra.run.smithi118.stdout: "type": "bool", 2024-03-24T03:17:24.487 INFO:teuthology.orchestra.run.smithi118.stdout: "level": "advanced", 2024-03-24T03:17:24.487 INFO:teuthology.orchestra.run.smithi118.stdout: "flags": 0, 2024-03-24T03:17:24.487 INFO:teuthology.orchestra.run.smithi118.stdout: "default_value": "True", 2024-03-24T03:17:24.487 INFO:teuthology.orchestra.run.smithi118.stdout: "min": "", 2024-03-24T03:17:24.487 INFO:teuthology.orchestra.run.smithi118.stdout: "max": "", 2024-03-24T03:17:24.487 INFO:teuthology.orchestra.run.smithi118.stdout: "enum_allowed": [], 2024-03-24T03:17:24.487 INFO:teuthology.orchestra.run.smithi118.stdout: "desc": "Share basic cluster information (size, version)", 2024-03-24T03:17:24.487 INFO:teuthology.orchestra.run.smithi118.stdout: "long_desc": "", 2024-03-24T03:17:24.488 INFO:teuthology.orchestra.run.smithi118.stdout: "tags": [], 2024-03-24T03:17:24.488 INFO:teuthology.orchestra.run.smithi118.stdout: "see_also": [] 2024-03-24T03:17:24.488 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-24T03:17:24.488 INFO:teuthology.orchestra.run.smithi118.stdout: "channel_crash": { 2024-03-24T03:17:24.488 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "channel_crash", 2024-03-24T03:17:24.488 INFO:teuthology.orchestra.run.smithi118.stdout: "type": "bool", 2024-03-24T03:17:24.488 INFO:teuthology.orchestra.run.smithi118.stdout: "level": "advanced", 2024-03-24T03:17:24.488 INFO:teuthology.orchestra.run.smithi118.stdout: "flags": 0, 2024-03-24T03:17:24.488 INFO:teuthology.orchestra.run.smithi118.stdout: "default_value": "True", 2024-03-24T03:17:24.489 INFO:teuthology.orchestra.run.smithi118.stdout: "min": "", 2024-03-24T03:17:24.489 INFO:teuthology.orchestra.run.smithi118.stdout: "max": "", 2024-03-24T03:17:24.489 INFO:teuthology.orchestra.run.smithi118.stdout: "enum_allowed": [], 2024-03-24T03:17:24.489 INFO:teuthology.orchestra.run.smithi118.stdout: "desc": "Share metadata about Ceph daemon crashes (version, stack straces, etc)", 2024-03-24T03:17:24.489 INFO:teuthology.orchestra.run.smithi118.stdout: "long_desc": "", 2024-03-24T03:17:24.489 INFO:teuthology.orchestra.run.smithi118.stdout: "tags": [], 2024-03-24T03:17:24.489 INFO:teuthology.orchestra.run.smithi118.stdout: "see_also": [] 2024-03-24T03:17:24.489 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-24T03:17:24.489 INFO:teuthology.orchestra.run.smithi118.stdout: "channel_device": { 2024-03-24T03:17:24.489 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "channel_device", 2024-03-24T03:17:24.490 INFO:teuthology.orchestra.run.smithi118.stdout: "type": "bool", 2024-03-24T03:17:24.490 INFO:teuthology.orchestra.run.smithi118.stdout: "level": "advanced", 2024-03-24T03:17:24.490 INFO:teuthology.orchestra.run.smithi118.stdout: "flags": 0, 2024-03-24T03:17:24.490 INFO:teuthology.orchestra.run.smithi118.stdout: "default_value": "True", 2024-03-24T03:17:24.490 INFO:teuthology.orchestra.run.smithi118.stdout: "min": "", 2024-03-24T03:17:24.490 INFO:teuthology.orchestra.run.smithi118.stdout: "max": "", 2024-03-24T03:17:24.490 INFO:teuthology.orchestra.run.smithi118.stdout: "enum_allowed": [], 2024-03-24T03:17:24.490 INFO:teuthology.orchestra.run.smithi118.stdout: "desc": "Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)", 2024-03-24T03:17:24.490 INFO:teuthology.orchestra.run.smithi118.stdout: "long_desc": "", 2024-03-24T03:17:24.491 INFO:teuthology.orchestra.run.smithi118.stdout: "tags": [], 2024-03-24T03:17:24.491 INFO:teuthology.orchestra.run.smithi118.stdout: "see_also": [] 2024-03-24T03:17:24.491 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-24T03:17:24.491 INFO:teuthology.orchestra.run.smithi118.stdout: "channel_ident": { 2024-03-24T03:17:24.491 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "channel_ident", 2024-03-24T03:17:24.491 INFO:teuthology.orchestra.run.smithi118.stdout: "type": "bool", 2024-03-24T03:17:24.491 INFO:teuthology.orchestra.run.smithi118.stdout: "level": "advanced", 2024-03-24T03:17:24.491 INFO:teuthology.orchestra.run.smithi118.stdout: "flags": 0, 2024-03-24T03:17:24.491 INFO:teuthology.orchestra.run.smithi118.stdout: "default_value": "False", 2024-03-24T03:17:24.491 INFO:teuthology.orchestra.run.smithi118.stdout: "min": "", 2024-03-24T03:17:24.492 INFO:teuthology.orchestra.run.smithi118.stdout: "max": "", 2024-03-24T03:17:24.492 INFO:teuthology.orchestra.run.smithi118.stdout: "enum_allowed": [], 2024-03-24T03:17:24.492 INFO:teuthology.orchestra.run.smithi118.stdout: "desc": "Share a user-provided description and/or contact email for the cluster", 2024-03-24T03:17:24.492 INFO:teuthology.orchestra.run.smithi118.stdout: "long_desc": "", 2024-03-24T03:17:24.492 INFO:teuthology.orchestra.run.smithi118.stdout: "tags": [], 2024-03-24T03:17:24.492 INFO:teuthology.orchestra.run.smithi118.stdout: "see_also": [] 2024-03-24T03:17:24.492 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-24T03:17:24.492 INFO:teuthology.orchestra.run.smithi118.stdout: "channel_perf": { 2024-03-24T03:17:24.492 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "channel_perf", 2024-03-24T03:17:24.493 INFO:teuthology.orchestra.run.smithi118.stdout: "type": "bool", 2024-03-24T03:17:24.493 INFO:teuthology.orchestra.run.smithi118.stdout: "level": "advanced", 2024-03-24T03:17:24.493 INFO:teuthology.orchestra.run.smithi118.stdout: "flags": 0, 2024-03-24T03:17:24.493 INFO:teuthology.orchestra.run.smithi118.stdout: "default_value": "False", 2024-03-24T03:17:24.493 INFO:teuthology.orchestra.run.smithi118.stdout: "min": "", 2024-03-24T03:17:24.493 INFO:teuthology.orchestra.run.smithi118.stdout: "max": "", 2024-03-24T03:17:24.493 INFO:teuthology.orchestra.run.smithi118.stdout: "enum_allowed": [], 2024-03-24T03:17:24.493 INFO:teuthology.orchestra.run.smithi118.stdout: "desc": "Share various performance metrics of a cluster", 2024-03-24T03:17:24.493 INFO:teuthology.orchestra.run.smithi118.stdout: "long_desc": "", 2024-03-24T03:17:24.493 INFO:teuthology.orchestra.run.smithi118.stdout: "tags": [], 2024-03-24T03:17:24.494 INFO:teuthology.orchestra.run.smithi118.stdout: "see_also": [] 2024-03-24T03:17:24.494 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-24T03:17:24.494 INFO:teuthology.orchestra.run.smithi118.stdout: "contact": { 2024-03-24T03:17:24.494 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "contact", 2024-03-24T03:17:24.494 INFO:teuthology.orchestra.run.smithi118.stdout: "type": "str", 2024-03-24T03:17:24.494 INFO:teuthology.orchestra.run.smithi118.stdout: "level": "advanced", 2024-03-24T03:17:24.494 INFO:teuthology.orchestra.run.smithi118.stdout: "flags": 0, 2024-03-24T03:17:24.494 INFO:teuthology.orchestra.run.smithi118.stdout: "default_value": "", 2024-03-24T03:17:24.495 INFO:teuthology.orchestra.run.smithi118.stdout: "min": "", 2024-03-24T03:17:24.495 INFO:teuthology.orchestra.run.smithi118.stdout: "max": "", 2024-03-24T03:17:24.495 INFO:teuthology.orchestra.run.smithi118.stdout: "enum_allowed": [], 2024-03-24T03:17:24.495 INFO:teuthology.orchestra.run.smithi118.stdout: "desc": "", 2024-03-24T03:17:24.495 INFO:teuthology.orchestra.run.smithi118.stdout: "long_desc": "", 2024-03-24T03:17:24.495 INFO:teuthology.orchestra.run.smithi118.stdout: "tags": [], 2024-03-24T03:17:24.495 INFO:teuthology.orchestra.run.smithi118.stdout: "see_also": [] 2024-03-24T03:17:24.495 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-24T03:17:24.495 INFO:teuthology.orchestra.run.smithi118.stdout: "description": { 2024-03-24T03:17:24.495 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "description", 2024-03-24T03:17:24.496 INFO:teuthology.orchestra.run.smithi118.stdout: "type": "str", 2024-03-24T03:17:24.496 INFO:teuthology.orchestra.run.smithi118.stdout: "level": "advanced", 2024-03-24T03:17:24.496 INFO:teuthology.orchestra.run.smithi118.stdout: "flags": 0, 2024-03-24T03:17:24.496 INFO:teuthology.orchestra.run.smithi118.stdout: "default_value": "", 2024-03-24T03:17:24.496 INFO:teuthology.orchestra.run.smithi118.stdout: "min": "", 2024-03-24T03:17:24.496 INFO:teuthology.orchestra.run.smithi118.stdout: "max": "", 2024-03-24T03:17:24.496 INFO:teuthology.orchestra.run.smithi118.stdout: "enum_allowed": [], 2024-03-24T03:17:24.496 INFO:teuthology.orchestra.run.smithi118.stdout: "desc": "", 2024-03-24T03:17:24.496 INFO:teuthology.orchestra.run.smithi118.stdout: "long_desc": "", 2024-03-24T03:17:24.497 INFO:teuthology.orchestra.run.smithi118.stdout: "tags": [], 2024-03-24T03:17:24.497 INFO:teuthology.orchestra.run.smithi118.stdout: "see_also": [] 2024-03-24T03:17:24.497 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-24T03:17:24.497 INFO:teuthology.orchestra.run.smithi118.stdout: "device_url": { 2024-03-24T03:17:24.497 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "device_url", 2024-03-24T03:17:24.497 INFO:teuthology.orchestra.run.smithi118.stdout: "type": "str", 2024-03-24T03:17:24.497 INFO:teuthology.orchestra.run.smithi118.stdout: "level": "advanced", 2024-03-24T03:17:24.497 INFO:teuthology.orchestra.run.smithi118.stdout: "flags": 0, 2024-03-24T03:17:24.497 INFO:teuthology.orchestra.run.smithi118.stdout: "default_value": "https://telemetry.ceph.com/device", 2024-03-24T03:17:24.497 INFO:teuthology.orchestra.run.smithi118.stdout: "min": "", 2024-03-24T03:17:24.498 INFO:teuthology.orchestra.run.smithi118.stdout: "max": "", 2024-03-24T03:17:24.498 INFO:teuthology.orchestra.run.smithi118.stdout: "enum_allowed": [], 2024-03-24T03:17:24.498 INFO:teuthology.orchestra.run.smithi118.stdout: "desc": "", 2024-03-24T03:17:24.498 INFO:teuthology.orchestra.run.smithi118.stdout: "long_desc": "", 2024-03-24T03:17:24.498 INFO:teuthology.orchestra.run.smithi118.stdout: "tags": [], 2024-03-24T03:17:24.498 INFO:teuthology.orchestra.run.smithi118.stdout: "see_also": [] 2024-03-24T03:17:24.498 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-24T03:17:24.498 INFO:teuthology.orchestra.run.smithi118.stdout: "enabled": { 2024-03-24T03:17:24.498 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "enabled", 2024-03-24T03:17:24.499 INFO:teuthology.orchestra.run.smithi118.stdout: "type": "bool", 2024-03-24T03:17:24.499 INFO:teuthology.orchestra.run.smithi118.stdout: "level": "advanced", 2024-03-24T03:17:24.499 INFO:teuthology.orchestra.run.smithi118.stdout: "flags": 0, 2024-03-24T03:17:24.499 INFO:teuthology.orchestra.run.smithi118.stdout: "default_value": "False", 2024-03-24T03:17:24.499 INFO:teuthology.orchestra.run.smithi118.stdout: "min": "", 2024-03-24T03:17:24.499 INFO:teuthology.orchestra.run.smithi118.stdout: "max": "", 2024-03-24T03:17:24.499 INFO:teuthology.orchestra.run.smithi118.stdout: "enum_allowed": [], 2024-03-24T03:17:24.499 INFO:teuthology.orchestra.run.smithi118.stdout: "desc": "", 2024-03-24T03:17:24.499 INFO:teuthology.orchestra.run.smithi118.stdout: "long_desc": "", 2024-03-24T03:17:24.500 INFO:teuthology.orchestra.run.smithi118.stdout: "tags": [], 2024-03-24T03:17:24.500 INFO:teuthology.orchestra.run.smithi118.stdout: "see_also": [] 2024-03-24T03:17:24.500 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-24T03:17:24.500 INFO:teuthology.orchestra.run.smithi118.stdout: "interval": { 2024-03-24T03:17:24.500 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "interval", 2024-03-24T03:17:24.500 INFO:teuthology.orchestra.run.smithi118.stdout: "type": "int", 2024-03-24T03:17:24.500 INFO:teuthology.orchestra.run.smithi118.stdout: "level": "advanced", 2024-03-24T03:17:24.500 INFO:teuthology.orchestra.run.smithi118.stdout: "flags": 0, 2024-03-24T03:17:24.500 INFO:teuthology.orchestra.run.smithi118.stdout: "default_value": "24", 2024-03-24T03:17:24.501 INFO:teuthology.orchestra.run.smithi118.stdout: "min": "8", 2024-03-24T03:17:24.501 INFO:teuthology.orchestra.run.smithi118.stdout: "max": "", 2024-03-24T03:17:24.501 INFO:teuthology.orchestra.run.smithi118.stdout: "enum_allowed": [], 2024-03-24T03:17:24.501 INFO:teuthology.orchestra.run.smithi118.stdout: "desc": "", 2024-03-24T03:17:24.501 INFO:teuthology.orchestra.run.smithi118.stdout: "long_desc": "", 2024-03-24T03:17:24.501 INFO:teuthology.orchestra.run.smithi118.stdout: "tags": [], 2024-03-24T03:17:24.501 INFO:teuthology.orchestra.run.smithi118.stdout: "see_also": [] 2024-03-24T03:17:24.501 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-24T03:17:24.501 INFO:teuthology.orchestra.run.smithi118.stdout: "last_opt_revision": { 2024-03-24T03:17:24.502 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "last_opt_revision", 2024-03-24T03:17:24.502 INFO:teuthology.orchestra.run.smithi118.stdout: "type": "int", 2024-03-24T03:17:24.502 INFO:teuthology.orchestra.run.smithi118.stdout: "level": "advanced", 2024-03-24T03:17:24.502 INFO:teuthology.orchestra.run.smithi118.stdout: "flags": 0, 2024-03-24T03:17:24.502 INFO:teuthology.orchestra.run.smithi118.stdout: "default_value": "1", 2024-03-24T03:17:24.502 INFO:teuthology.orchestra.run.smithi118.stdout: "min": "", 2024-03-24T03:17:24.502 INFO:teuthology.orchestra.run.smithi118.stdout: "max": "", 2024-03-24T03:17:24.502 INFO:teuthology.orchestra.run.smithi118.stdout: "enum_allowed": [], 2024-03-24T03:17:24.502 INFO:teuthology.orchestra.run.smithi118.stdout: "desc": "", 2024-03-24T03:17:24.503 INFO:teuthology.orchestra.run.smithi118.stdout: "long_desc": "", 2024-03-24T03:17:24.503 INFO:teuthology.orchestra.run.smithi118.stdout: "tags": [], 2024-03-24T03:17:24.503 INFO:teuthology.orchestra.run.smithi118.stdout: "see_also": [] 2024-03-24T03:17:24.503 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-24T03:17:24.503 INFO:teuthology.orchestra.run.smithi118.stdout: "leaderboard": { 2024-03-24T03:17:24.503 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "leaderboard", 2024-03-24T03:17:24.503 INFO:teuthology.orchestra.run.smithi118.stdout: "type": "bool", 2024-03-24T03:17:24.503 INFO:teuthology.orchestra.run.smithi118.stdout: "level": "advanced", 2024-03-24T03:17:24.504 INFO:teuthology.orchestra.run.smithi118.stdout: "flags": 0, 2024-03-24T03:17:24.504 INFO:teuthology.orchestra.run.smithi118.stdout: "default_value": "False", 2024-03-24T03:17:24.504 INFO:teuthology.orchestra.run.smithi118.stdout: "min": "", 2024-03-24T03:17:24.504 INFO:teuthology.orchestra.run.smithi118.stdout: "max": "", 2024-03-24T03:17:24.504 INFO:teuthology.orchestra.run.smithi118.stdout: "enum_allowed": [], 2024-03-24T03:17:24.504 INFO:teuthology.orchestra.run.smithi118.stdout: "desc": "", 2024-03-24T03:17:24.504 INFO:teuthology.orchestra.run.smithi118.stdout: "long_desc": "", 2024-03-24T03:17:24.504 INFO:teuthology.orchestra.run.smithi118.stdout: "tags": [], 2024-03-24T03:17:24.504 INFO:teuthology.orchestra.run.smithi118.stdout: "see_also": [] 2024-03-24T03:17:24.505 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-24T03:17:24.505 INFO:teuthology.orchestra.run.smithi118.stdout: "leaderboard_description": { 2024-03-24T03:17:24.505 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "leaderboard_description", 2024-03-24T03:17:24.505 INFO:teuthology.orchestra.run.smithi118.stdout: "type": "str", 2024-03-24T03:17:24.505 INFO:teuthology.orchestra.run.smithi118.stdout: "level": "advanced", 2024-03-24T03:17:24.505 INFO:teuthology.orchestra.run.smithi118.stdout: "flags": 0, 2024-03-24T03:17:24.505 INFO:teuthology.orchestra.run.smithi118.stdout: "default_value": "", 2024-03-24T03:17:24.505 INFO:teuthology.orchestra.run.smithi118.stdout: "min": "", 2024-03-24T03:17:24.505 INFO:teuthology.orchestra.run.smithi118.stdout: "max": "", 2024-03-24T03:17:24.505 INFO:teuthology.orchestra.run.smithi118.stdout: "enum_allowed": [], 2024-03-24T03:17:24.506 INFO:teuthology.orchestra.run.smithi118.stdout: "desc": "", 2024-03-24T03:17:24.506 INFO:teuthology.orchestra.run.smithi118.stdout: "long_desc": "", 2024-03-24T03:17:24.506 INFO:teuthology.orchestra.run.smithi118.stdout: "tags": [], 2024-03-24T03:17:24.506 INFO:teuthology.orchestra.run.smithi118.stdout: "see_also": [] 2024-03-24T03:17:24.506 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-24T03:17:24.506 INFO:teuthology.orchestra.run.smithi118.stdout: "log_level": { 2024-03-24T03:17:24.506 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "log_level", 2024-03-24T03:17:24.506 INFO:teuthology.orchestra.run.smithi118.stdout: "type": "str", 2024-03-24T03:17:24.507 INFO:teuthology.orchestra.run.smithi118.stdout: "level": "advanced", 2024-03-24T03:17:24.507 INFO:teuthology.orchestra.run.smithi118.stdout: "flags": 1, 2024-03-24T03:17:24.507 INFO:teuthology.orchestra.run.smithi118.stdout: "default_value": "", 2024-03-24T03:17:24.507 INFO:teuthology.orchestra.run.smithi118.stdout: "min": "", 2024-03-24T03:17:24.507 INFO:teuthology.orchestra.run.smithi118.stdout: "max": "", 2024-03-24T03:17:24.507 INFO:teuthology.orchestra.run.smithi118.stdout: "enum_allowed": [ 2024-03-24T03:17:24.507 INFO:teuthology.orchestra.run.smithi118.stdout: "", 2024-03-24T03:17:24.507 INFO:teuthology.orchestra.run.smithi118.stdout: "critical", 2024-03-24T03:17:24.507 INFO:teuthology.orchestra.run.smithi118.stdout: "debug", 2024-03-24T03:17:24.507 INFO:teuthology.orchestra.run.smithi118.stdout: "error", 2024-03-24T03:17:24.508 INFO:teuthology.orchestra.run.smithi118.stdout: "info", 2024-03-24T03:17:24.508 INFO:teuthology.orchestra.run.smithi118.stdout: "warning" 2024-03-24T03:17:24.508 INFO:teuthology.orchestra.run.smithi118.stdout: ], 2024-03-24T03:17:24.508 INFO:teuthology.orchestra.run.smithi118.stdout: "desc": "", 2024-03-24T03:17:24.508 INFO:teuthology.orchestra.run.smithi118.stdout: "long_desc": "", 2024-03-24T03:17:24.508 INFO:teuthology.orchestra.run.smithi118.stdout: "tags": [], 2024-03-24T03:17:24.508 INFO:teuthology.orchestra.run.smithi118.stdout: "see_also": [] 2024-03-24T03:17:24.508 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-24T03:17:24.508 INFO:teuthology.orchestra.run.smithi118.stdout: "log_to_cluster": { 2024-03-24T03:17:24.509 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "log_to_cluster", 2024-03-24T03:17:24.509 INFO:teuthology.orchestra.run.smithi118.stdout: "type": "bool", 2024-03-24T03:17:24.509 INFO:teuthology.orchestra.run.smithi118.stdout: "level": "advanced", 2024-03-24T03:17:24.509 INFO:teuthology.orchestra.run.smithi118.stdout: "flags": 1, 2024-03-24T03:17:24.509 INFO:teuthology.orchestra.run.smithi118.stdout: "default_value": "False", 2024-03-24T03:17:24.509 INFO:teuthology.orchestra.run.smithi118.stdout: "min": "", 2024-03-24T03:17:24.509 INFO:teuthology.orchestra.run.smithi118.stdout: "max": "", 2024-03-24T03:17:24.509 INFO:teuthology.orchestra.run.smithi118.stdout: "enum_allowed": [], 2024-03-24T03:17:24.509 INFO:teuthology.orchestra.run.smithi118.stdout: "desc": "", 2024-03-24T03:17:24.510 INFO:teuthology.orchestra.run.smithi118.stdout: "long_desc": "", 2024-03-24T03:17:24.510 INFO:teuthology.orchestra.run.smithi118.stdout: "tags": [], 2024-03-24T03:17:24.510 INFO:teuthology.orchestra.run.smithi118.stdout: "see_also": [] 2024-03-24T03:17:24.510 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-24T03:17:24.510 INFO:teuthology.orchestra.run.smithi118.stdout: "log_to_cluster_level": { 2024-03-24T03:17:24.510 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "log_to_cluster_level", 2024-03-24T03:17:24.510 INFO:teuthology.orchestra.run.smithi118.stdout: "type": "str", 2024-03-24T03:17:24.510 INFO:teuthology.orchestra.run.smithi118.stdout: "level": "advanced", 2024-03-24T03:17:24.510 INFO:teuthology.orchestra.run.smithi118.stdout: "flags": 1, 2024-03-24T03:17:24.511 INFO:teuthology.orchestra.run.smithi118.stdout: "default_value": "info", 2024-03-24T03:17:24.511 INFO:teuthology.orchestra.run.smithi118.stdout: "min": "", 2024-03-24T03:17:24.511 INFO:teuthology.orchestra.run.smithi118.stdout: "max": "", 2024-03-24T03:17:24.511 INFO:teuthology.orchestra.run.smithi118.stdout: "enum_allowed": [ 2024-03-24T03:17:24.511 INFO:teuthology.orchestra.run.smithi118.stdout: "", 2024-03-24T03:17:24.511 INFO:teuthology.orchestra.run.smithi118.stdout: "critical", 2024-03-24T03:17:24.511 INFO:teuthology.orchestra.run.smithi118.stdout: "debug", 2024-03-24T03:17:24.511 INFO:teuthology.orchestra.run.smithi118.stdout: "error", 2024-03-24T03:17:24.511 INFO:teuthology.orchestra.run.smithi118.stdout: "info", 2024-03-24T03:17:24.512 INFO:teuthology.orchestra.run.smithi118.stdout: "warning" 2024-03-24T03:17:24.512 INFO:teuthology.orchestra.run.smithi118.stdout: ], 2024-03-24T03:17:24.512 INFO:teuthology.orchestra.run.smithi118.stdout: "desc": "", 2024-03-24T03:17:24.512 INFO:teuthology.orchestra.run.smithi118.stdout: "long_desc": "", 2024-03-24T03:17:24.512 INFO:teuthology.orchestra.run.smithi118.stdout: "tags": [], 2024-03-24T03:17:24.512 INFO:teuthology.orchestra.run.smithi118.stdout: "see_also": [] 2024-03-24T03:17:24.512 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-24T03:17:24.512 INFO:teuthology.orchestra.run.smithi118.stdout: "log_to_file": { 2024-03-24T03:17:24.513 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "log_to_file", 2024-03-24T03:17:24.513 INFO:teuthology.orchestra.run.smithi118.stdout: "type": "bool", 2024-03-24T03:17:24.513 INFO:teuthology.orchestra.run.smithi118.stdout: "level": "advanced", 2024-03-24T03:17:24.513 INFO:teuthology.orchestra.run.smithi118.stdout: "flags": 1, 2024-03-24T03:17:24.513 INFO:teuthology.orchestra.run.smithi118.stdout: "default_value": "False", 2024-03-24T03:17:24.513 INFO:teuthology.orchestra.run.smithi118.stdout: "min": "", 2024-03-24T03:17:24.513 INFO:teuthology.orchestra.run.smithi118.stdout: "max": "", 2024-03-24T03:17:24.513 INFO:teuthology.orchestra.run.smithi118.stdout: "enum_allowed": [], 2024-03-24T03:17:24.513 INFO:teuthology.orchestra.run.smithi118.stdout: "desc": "", 2024-03-24T03:17:24.514 INFO:teuthology.orchestra.run.smithi118.stdout: "long_desc": "", 2024-03-24T03:17:24.514 INFO:teuthology.orchestra.run.smithi118.stdout: "tags": [], 2024-03-24T03:17:24.514 INFO:teuthology.orchestra.run.smithi118.stdout: "see_also": [] 2024-03-24T03:17:24.514 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-24T03:17:24.514 INFO:teuthology.orchestra.run.smithi118.stdout: "organization": { 2024-03-24T03:17:24.514 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "organization", 2024-03-24T03:17:24.514 INFO:teuthology.orchestra.run.smithi118.stdout: "type": "str", 2024-03-24T03:17:24.514 INFO:teuthology.orchestra.run.smithi118.stdout: "level": "advanced", 2024-03-24T03:17:24.515 INFO:teuthology.orchestra.run.smithi118.stdout: "flags": 0, 2024-03-24T03:17:24.515 INFO:teuthology.orchestra.run.smithi118.stdout: "default_value": "", 2024-03-24T03:17:24.515 INFO:teuthology.orchestra.run.smithi118.stdout: "min": "", 2024-03-24T03:17:24.515 INFO:teuthology.orchestra.run.smithi118.stdout: "max": "", 2024-03-24T03:17:24.515 INFO:teuthology.orchestra.run.smithi118.stdout: "enum_allowed": [], 2024-03-24T03:17:24.515 INFO:teuthology.orchestra.run.smithi118.stdout: "desc": "", 2024-03-24T03:17:24.515 INFO:teuthology.orchestra.run.smithi118.stdout: "long_desc": "", 2024-03-24T03:17:24.515 INFO:teuthology.orchestra.run.smithi118.stdout: "tags": [], 2024-03-24T03:17:24.515 INFO:teuthology.orchestra.run.smithi118.stdout: "see_also": [] 2024-03-24T03:17:24.515 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-24T03:17:24.516 INFO:teuthology.orchestra.run.smithi118.stdout: "proxy": { 2024-03-24T03:17:24.516 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "proxy", 2024-03-24T03:17:24.516 INFO:teuthology.orchestra.run.smithi118.stdout: "type": "str", 2024-03-24T03:17:24.516 INFO:teuthology.orchestra.run.smithi118.stdout: "level": "advanced", 2024-03-24T03:17:24.516 INFO:teuthology.orchestra.run.smithi118.stdout: "flags": 0, 2024-03-24T03:17:24.516 INFO:teuthology.orchestra.run.smithi118.stdout: "default_value": "", 2024-03-24T03:17:24.516 INFO:teuthology.orchestra.run.smithi118.stdout: "min": "", 2024-03-24T03:17:24.516 INFO:teuthology.orchestra.run.smithi118.stdout: "max": "", 2024-03-24T03:17:24.516 INFO:teuthology.orchestra.run.smithi118.stdout: "enum_allowed": [], 2024-03-24T03:17:24.517 INFO:teuthology.orchestra.run.smithi118.stdout: "desc": "", 2024-03-24T03:17:24.517 INFO:teuthology.orchestra.run.smithi118.stdout: "long_desc": "", 2024-03-24T03:17:24.517 INFO:teuthology.orchestra.run.smithi118.stdout: "tags": [], 2024-03-24T03:17:24.517 INFO:teuthology.orchestra.run.smithi118.stdout: "see_also": [] 2024-03-24T03:17:24.517 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-24T03:17:24.517 INFO:teuthology.orchestra.run.smithi118.stdout: "url": { 2024-03-24T03:17:24.517 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "url", 2024-03-24T03:17:24.517 INFO:teuthology.orchestra.run.smithi118.stdout: "type": "str", 2024-03-24T03:17:24.517 INFO:teuthology.orchestra.run.smithi118.stdout: "level": "advanced", 2024-03-24T03:17:24.517 INFO:teuthology.orchestra.run.smithi118.stdout: "flags": 0, 2024-03-24T03:17:24.518 INFO:teuthology.orchestra.run.smithi118.stdout: "default_value": "https://telemetry.ceph.com/report", 2024-03-24T03:17:24.518 INFO:teuthology.orchestra.run.smithi118.stdout: "min": "", 2024-03-24T03:17:24.518 INFO:teuthology.orchestra.run.smithi118.stdout: "max": "", 2024-03-24T03:17:24.518 INFO:teuthology.orchestra.run.smithi118.stdout: "enum_allowed": [], 2024-03-24T03:17:24.518 INFO:teuthology.orchestra.run.smithi118.stdout: "desc": "", 2024-03-24T03:17:24.518 INFO:teuthology.orchestra.run.smithi118.stdout: "long_desc": "", 2024-03-24T03:17:24.518 INFO:teuthology.orchestra.run.smithi118.stdout: "tags": [], 2024-03-24T03:17:24.518 INFO:teuthology.orchestra.run.smithi118.stdout: "see_also": [] 2024-03-24T03:17:24.519 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-03-24T03:17:24.519 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-03-24T03:17:24.519 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-24T03:17:24.519 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-24T03:17:24.519 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "test_orchestrator", 2024-03-24T03:17:24.519 INFO:teuthology.orchestra.run.smithi118.stdout: "can_run": true, 2024-03-24T03:17:24.519 INFO:teuthology.orchestra.run.smithi118.stdout: "error_string": "", 2024-03-24T03:17:24.519 INFO:teuthology.orchestra.run.smithi118.stdout: "module_options": { 2024-03-24T03:17:24.519 INFO:teuthology.orchestra.run.smithi118.stdout: "log_level": { 2024-03-24T03:17:24.520 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "log_level", 2024-03-24T03:17:24.520 INFO:teuthology.orchestra.run.smithi118.stdout: "type": "str", 2024-03-24T03:17:24.520 INFO:teuthology.orchestra.run.smithi118.stdout: "level": "advanced", 2024-03-24T03:17:24.520 INFO:teuthology.orchestra.run.smithi118.stdout: "flags": 1, 2024-03-24T03:17:24.520 INFO:teuthology.orchestra.run.smithi118.stdout: "default_value": "", 2024-03-24T03:17:24.520 INFO:teuthology.orchestra.run.smithi118.stdout: "min": "", 2024-03-24T03:17:24.520 INFO:teuthology.orchestra.run.smithi118.stdout: "max": "", 2024-03-24T03:17:24.520 INFO:teuthology.orchestra.run.smithi118.stdout: "enum_allowed": [ 2024-03-24T03:17:24.520 INFO:teuthology.orchestra.run.smithi118.stdout: "", 2024-03-24T03:17:24.521 INFO:teuthology.orchestra.run.smithi118.stdout: "critical", 2024-03-24T03:17:24.521 INFO:teuthology.orchestra.run.smithi118.stdout: "debug", 2024-03-24T03:17:24.521 INFO:teuthology.orchestra.run.smithi118.stdout: "error", 2024-03-24T03:17:24.521 INFO:teuthology.orchestra.run.smithi118.stdout: "info", 2024-03-24T03:17:24.521 INFO:teuthology.orchestra.run.smithi118.stdout: "warning" 2024-03-24T03:17:24.521 INFO:teuthology.orchestra.run.smithi118.stdout: ], 2024-03-24T03:17:24.521 INFO:teuthology.orchestra.run.smithi118.stdout: "desc": "", 2024-03-24T03:17:24.521 INFO:teuthology.orchestra.run.smithi118.stdout: "long_desc": "", 2024-03-24T03:17:24.521 INFO:teuthology.orchestra.run.smithi118.stdout: "tags": [], 2024-03-24T03:17:24.521 INFO:teuthology.orchestra.run.smithi118.stdout: "see_also": [] 2024-03-24T03:17:24.522 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-24T03:17:24.522 INFO:teuthology.orchestra.run.smithi118.stdout: "log_to_cluster": { 2024-03-24T03:17:24.522 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "log_to_cluster", 2024-03-24T03:17:24.522 INFO:teuthology.orchestra.run.smithi118.stdout: "type": "bool", 2024-03-24T03:17:24.522 INFO:teuthology.orchestra.run.smithi118.stdout: "level": "advanced", 2024-03-24T03:17:24.522 INFO:teuthology.orchestra.run.smithi118.stdout: "flags": 1, 2024-03-24T03:17:24.522 INFO:teuthology.orchestra.run.smithi118.stdout: "default_value": "False", 2024-03-24T03:17:24.522 INFO:teuthology.orchestra.run.smithi118.stdout: "min": "", 2024-03-24T03:17:24.523 INFO:teuthology.orchestra.run.smithi118.stdout: "max": "", 2024-03-24T03:17:24.523 INFO:teuthology.orchestra.run.smithi118.stdout: "enum_allowed": [], 2024-03-24T03:17:24.523 INFO:teuthology.orchestra.run.smithi118.stdout: "desc": "", 2024-03-24T03:17:24.523 INFO:teuthology.orchestra.run.smithi118.stdout: "long_desc": "", 2024-03-24T03:17:24.523 INFO:teuthology.orchestra.run.smithi118.stdout: "tags": [], 2024-03-24T03:17:24.523 INFO:teuthology.orchestra.run.smithi118.stdout: "see_also": [] 2024-03-24T03:17:24.523 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-24T03:17:24.523 INFO:teuthology.orchestra.run.smithi118.stdout: "log_to_cluster_level": { 2024-03-24T03:17:24.523 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "log_to_cluster_level", 2024-03-24T03:17:24.524 INFO:teuthology.orchestra.run.smithi118.stdout: "type": "str", 2024-03-24T03:17:24.524 INFO:teuthology.orchestra.run.smithi118.stdout: "level": "advanced", 2024-03-24T03:17:24.524 INFO:teuthology.orchestra.run.smithi118.stdout: "flags": 1, 2024-03-24T03:17:24.524 INFO:teuthology.orchestra.run.smithi118.stdout: "default_value": "info", 2024-03-24T03:17:24.524 INFO:teuthology.orchestra.run.smithi118.stdout: "min": "", 2024-03-24T03:17:24.524 INFO:teuthology.orchestra.run.smithi118.stdout: "max": "", 2024-03-24T03:17:24.524 INFO:teuthology.orchestra.run.smithi118.stdout: "enum_allowed": [ 2024-03-24T03:17:24.524 INFO:teuthology.orchestra.run.smithi118.stdout: "", 2024-03-24T03:17:24.524 INFO:teuthology.orchestra.run.smithi118.stdout: "critical", 2024-03-24T03:17:24.525 INFO:teuthology.orchestra.run.smithi118.stdout: "debug", 2024-03-24T03:17:24.525 INFO:teuthology.orchestra.run.smithi118.stdout: "error", 2024-03-24T03:17:24.525 INFO:teuthology.orchestra.run.smithi118.stdout: "info", 2024-03-24T03:17:24.525 INFO:teuthology.orchestra.run.smithi118.stdout: "warning" 2024-03-24T03:17:24.525 INFO:teuthology.orchestra.run.smithi118.stdout: ], 2024-03-24T03:17:24.525 INFO:teuthology.orchestra.run.smithi118.stdout: "desc": "", 2024-03-24T03:17:24.525 INFO:teuthology.orchestra.run.smithi118.stdout: "long_desc": "", 2024-03-24T03:17:24.525 INFO:teuthology.orchestra.run.smithi118.stdout: "tags": [], 2024-03-24T03:17:24.525 INFO:teuthology.orchestra.run.smithi118.stdout: "see_also": [] 2024-03-24T03:17:24.526 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-24T03:17:24.526 INFO:teuthology.orchestra.run.smithi118.stdout: "log_to_file": { 2024-03-24T03:17:24.526 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "log_to_file", 2024-03-24T03:17:24.526 INFO:teuthology.orchestra.run.smithi118.stdout: "type": "bool", 2024-03-24T03:17:24.526 INFO:teuthology.orchestra.run.smithi118.stdout: "level": "advanced", 2024-03-24T03:17:24.526 INFO:teuthology.orchestra.run.smithi118.stdout: "flags": 1, 2024-03-24T03:17:24.526 INFO:teuthology.orchestra.run.smithi118.stdout: "default_value": "False", 2024-03-24T03:17:24.526 INFO:teuthology.orchestra.run.smithi118.stdout: "min": "", 2024-03-24T03:17:24.526 INFO:teuthology.orchestra.run.smithi118.stdout: "max": "", 2024-03-24T03:17:24.526 INFO:teuthology.orchestra.run.smithi118.stdout: "enum_allowed": [], 2024-03-24T03:17:24.527 INFO:teuthology.orchestra.run.smithi118.stdout: "desc": "", 2024-03-24T03:17:24.527 INFO:teuthology.orchestra.run.smithi118.stdout: "long_desc": "", 2024-03-24T03:17:24.527 INFO:teuthology.orchestra.run.smithi118.stdout: "tags": [], 2024-03-24T03:17:24.527 INFO:teuthology.orchestra.run.smithi118.stdout: "see_also": [] 2024-03-24T03:17:24.527 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-03-24T03:17:24.527 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-03-24T03:17:24.527 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-24T03:17:24.527 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-24T03:17:24.527 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "volumes", 2024-03-24T03:17:24.528 INFO:teuthology.orchestra.run.smithi118.stdout: "can_run": true, 2024-03-24T03:17:24.528 INFO:teuthology.orchestra.run.smithi118.stdout: "error_string": "", 2024-03-24T03:17:24.528 INFO:teuthology.orchestra.run.smithi118.stdout: "module_options": { 2024-03-24T03:17:24.528 INFO:teuthology.orchestra.run.smithi118.stdout: "log_level": { 2024-03-24T03:17:24.528 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "log_level", 2024-03-24T03:17:24.528 INFO:teuthology.orchestra.run.smithi118.stdout: "type": "str", 2024-03-24T03:17:24.528 INFO:teuthology.orchestra.run.smithi118.stdout: "level": "advanced", 2024-03-24T03:17:24.528 INFO:teuthology.orchestra.run.smithi118.stdout: "flags": 1, 2024-03-24T03:17:24.528 INFO:teuthology.orchestra.run.smithi118.stdout: "default_value": "", 2024-03-24T03:17:24.529 INFO:teuthology.orchestra.run.smithi118.stdout: "min": "", 2024-03-24T03:17:24.529 INFO:teuthology.orchestra.run.smithi118.stdout: "max": "", 2024-03-24T03:17:24.529 INFO:teuthology.orchestra.run.smithi118.stdout: "enum_allowed": [ 2024-03-24T03:17:24.529 INFO:teuthology.orchestra.run.smithi118.stdout: "", 2024-03-24T03:17:24.529 INFO:teuthology.orchestra.run.smithi118.stdout: "critical", 2024-03-24T03:17:24.529 INFO:teuthology.orchestra.run.smithi118.stdout: "debug", 2024-03-24T03:17:24.529 INFO:teuthology.orchestra.run.smithi118.stdout: "error", 2024-03-24T03:17:24.529 INFO:teuthology.orchestra.run.smithi118.stdout: "info", 2024-03-24T03:17:24.529 INFO:teuthology.orchestra.run.smithi118.stdout: "warning" 2024-03-24T03:17:24.530 INFO:teuthology.orchestra.run.smithi118.stdout: ], 2024-03-24T03:17:24.530 INFO:teuthology.orchestra.run.smithi118.stdout: "desc": "", 2024-03-24T03:17:24.530 INFO:teuthology.orchestra.run.smithi118.stdout: "long_desc": "", 2024-03-24T03:17:24.530 INFO:teuthology.orchestra.run.smithi118.stdout: "tags": [], 2024-03-24T03:17:24.530 INFO:teuthology.orchestra.run.smithi118.stdout: "see_also": [] 2024-03-24T03:17:24.530 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-24T03:17:24.530 INFO:teuthology.orchestra.run.smithi118.stdout: "log_to_cluster": { 2024-03-24T03:17:24.530 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "log_to_cluster", 2024-03-24T03:17:24.530 INFO:teuthology.orchestra.run.smithi118.stdout: "type": "bool", 2024-03-24T03:17:24.531 INFO:teuthology.orchestra.run.smithi118.stdout: "level": "advanced", 2024-03-24T03:17:24.531 INFO:teuthology.orchestra.run.smithi118.stdout: "flags": 1, 2024-03-24T03:17:24.531 INFO:teuthology.orchestra.run.smithi118.stdout: "default_value": "False", 2024-03-24T03:17:24.531 INFO:teuthology.orchestra.run.smithi118.stdout: "min": "", 2024-03-24T03:17:24.531 INFO:teuthology.orchestra.run.smithi118.stdout: "max": "", 2024-03-24T03:17:24.531 INFO:teuthology.orchestra.run.smithi118.stdout: "enum_allowed": [], 2024-03-24T03:17:24.531 INFO:teuthology.orchestra.run.smithi118.stdout: "desc": "", 2024-03-24T03:17:24.531 INFO:teuthology.orchestra.run.smithi118.stdout: "long_desc": "", 2024-03-24T03:17:24.531 INFO:teuthology.orchestra.run.smithi118.stdout: "tags": [], 2024-03-24T03:17:24.532 INFO:teuthology.orchestra.run.smithi118.stdout: "see_also": [] 2024-03-24T03:17:24.532 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-24T03:17:24.532 INFO:teuthology.orchestra.run.smithi118.stdout: "log_to_cluster_level": { 2024-03-24T03:17:24.532 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "log_to_cluster_level", 2024-03-24T03:17:24.532 INFO:teuthology.orchestra.run.smithi118.stdout: "type": "str", 2024-03-24T03:17:24.532 INFO:teuthology.orchestra.run.smithi118.stdout: "level": "advanced", 2024-03-24T03:17:24.532 INFO:teuthology.orchestra.run.smithi118.stdout: "flags": 1, 2024-03-24T03:17:24.532 INFO:teuthology.orchestra.run.smithi118.stdout: "default_value": "info", 2024-03-24T03:17:24.533 INFO:teuthology.orchestra.run.smithi118.stdout: "min": "", 2024-03-24T03:17:24.533 INFO:teuthology.orchestra.run.smithi118.stdout: "max": "", 2024-03-24T03:17:24.533 INFO:teuthology.orchestra.run.smithi118.stdout: "enum_allowed": [ 2024-03-24T03:17:24.533 INFO:teuthology.orchestra.run.smithi118.stdout: "", 2024-03-24T03:17:24.533 INFO:teuthology.orchestra.run.smithi118.stdout: "critical", 2024-03-24T03:17:24.533 INFO:teuthology.orchestra.run.smithi118.stdout: "debug", 2024-03-24T03:17:24.533 INFO:teuthology.orchestra.run.smithi118.stdout: "error", 2024-03-24T03:17:24.533 INFO:teuthology.orchestra.run.smithi118.stdout: "info", 2024-03-24T03:17:24.533 INFO:teuthology.orchestra.run.smithi118.stdout: "warning" 2024-03-24T03:17:24.534 INFO:teuthology.orchestra.run.smithi118.stdout: ], 2024-03-24T03:17:24.534 INFO:teuthology.orchestra.run.smithi118.stdout: "desc": "", 2024-03-24T03:17:24.534 INFO:teuthology.orchestra.run.smithi118.stdout: "long_desc": "", 2024-03-24T03:17:24.534 INFO:teuthology.orchestra.run.smithi118.stdout: "tags": [], 2024-03-24T03:17:24.534 INFO:teuthology.orchestra.run.smithi118.stdout: "see_also": [] 2024-03-24T03:17:24.534 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-24T03:17:24.534 INFO:teuthology.orchestra.run.smithi118.stdout: "log_to_file": { 2024-03-24T03:17:24.534 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "log_to_file", 2024-03-24T03:17:24.535 INFO:teuthology.orchestra.run.smithi118.stdout: "type": "bool", 2024-03-24T03:17:24.535 INFO:teuthology.orchestra.run.smithi118.stdout: "level": "advanced", 2024-03-24T03:17:24.535 INFO:teuthology.orchestra.run.smithi118.stdout: "flags": 1, 2024-03-24T03:17:24.535 INFO:teuthology.orchestra.run.smithi118.stdout: "default_value": "False", 2024-03-24T03:17:24.535 INFO:teuthology.orchestra.run.smithi118.stdout: "min": "", 2024-03-24T03:17:24.535 INFO:teuthology.orchestra.run.smithi118.stdout: "max": "", 2024-03-24T03:17:24.535 INFO:teuthology.orchestra.run.smithi118.stdout: "enum_allowed": [], 2024-03-24T03:17:24.535 INFO:teuthology.orchestra.run.smithi118.stdout: "desc": "", 2024-03-24T03:17:24.535 INFO:teuthology.orchestra.run.smithi118.stdout: "long_desc": "", 2024-03-24T03:17:24.536 INFO:teuthology.orchestra.run.smithi118.stdout: "tags": [], 2024-03-24T03:17:24.536 INFO:teuthology.orchestra.run.smithi118.stdout: "see_also": [] 2024-03-24T03:17:24.536 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-24T03:17:24.536 INFO:teuthology.orchestra.run.smithi118.stdout: "max_concurrent_clones": { 2024-03-24T03:17:24.536 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "max_concurrent_clones", 2024-03-24T03:17:24.536 INFO:teuthology.orchestra.run.smithi118.stdout: "type": "int", 2024-03-24T03:17:24.536 INFO:teuthology.orchestra.run.smithi118.stdout: "level": "advanced", 2024-03-24T03:17:24.536 INFO:teuthology.orchestra.run.smithi118.stdout: "flags": 0, 2024-03-24T03:17:24.536 INFO:teuthology.orchestra.run.smithi118.stdout: "default_value": "4", 2024-03-24T03:17:24.537 INFO:teuthology.orchestra.run.smithi118.stdout: "min": "", 2024-03-24T03:17:24.537 INFO:teuthology.orchestra.run.smithi118.stdout: "max": "", 2024-03-24T03:17:24.537 INFO:teuthology.orchestra.run.smithi118.stdout: "enum_allowed": [], 2024-03-24T03:17:24.537 INFO:teuthology.orchestra.run.smithi118.stdout: "desc": "Number of asynchronous cloner threads", 2024-03-24T03:17:24.537 INFO:teuthology.orchestra.run.smithi118.stdout: "long_desc": "", 2024-03-24T03:17:24.537 INFO:teuthology.orchestra.run.smithi118.stdout: "tags": [], 2024-03-24T03:17:24.537 INFO:teuthology.orchestra.run.smithi118.stdout: "see_also": [] 2024-03-24T03:17:24.537 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-24T03:17:24.537 INFO:teuthology.orchestra.run.smithi118.stdout: "periodic_async_work": { 2024-03-24T03:17:24.538 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "periodic_async_work", 2024-03-24T03:17:24.538 INFO:teuthology.orchestra.run.smithi118.stdout: "type": "bool", 2024-03-24T03:17:24.538 INFO:teuthology.orchestra.run.smithi118.stdout: "level": "advanced", 2024-03-24T03:17:24.538 INFO:teuthology.orchestra.run.smithi118.stdout: "flags": 0, 2024-03-24T03:17:24.538 INFO:teuthology.orchestra.run.smithi118.stdout: "default_value": "False", 2024-03-24T03:17:24.538 INFO:teuthology.orchestra.run.smithi118.stdout: "min": "", 2024-03-24T03:17:24.538 INFO:teuthology.orchestra.run.smithi118.stdout: "max": "", 2024-03-24T03:17:24.538 INFO:teuthology.orchestra.run.smithi118.stdout: "enum_allowed": [], 2024-03-24T03:17:24.538 INFO:teuthology.orchestra.run.smithi118.stdout: "desc": "Periodically check for async work", 2024-03-24T03:17:24.539 INFO:teuthology.orchestra.run.smithi118.stdout: "long_desc": "", 2024-03-24T03:17:24.539 INFO:teuthology.orchestra.run.smithi118.stdout: "tags": [], 2024-03-24T03:17:24.539 INFO:teuthology.orchestra.run.smithi118.stdout: "see_also": [] 2024-03-24T03:17:24.539 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-24T03:17:24.539 INFO:teuthology.orchestra.run.smithi118.stdout: "snapshot_clone_delay": { 2024-03-24T03:17:24.539 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "snapshot_clone_delay", 2024-03-24T03:17:24.539 INFO:teuthology.orchestra.run.smithi118.stdout: "type": "int", 2024-03-24T03:17:24.539 INFO:teuthology.orchestra.run.smithi118.stdout: "level": "advanced", 2024-03-24T03:17:24.539 INFO:teuthology.orchestra.run.smithi118.stdout: "flags": 0, 2024-03-24T03:17:24.540 INFO:teuthology.orchestra.run.smithi118.stdout: "default_value": "0", 2024-03-24T03:17:24.540 INFO:teuthology.orchestra.run.smithi118.stdout: "min": "", 2024-03-24T03:17:24.540 INFO:teuthology.orchestra.run.smithi118.stdout: "max": "", 2024-03-24T03:17:24.540 INFO:teuthology.orchestra.run.smithi118.stdout: "enum_allowed": [], 2024-03-24T03:17:24.540 INFO:teuthology.orchestra.run.smithi118.stdout: "desc": "Delay clone begin operation by snapshot_clone_delay seconds", 2024-03-24T03:17:24.540 INFO:teuthology.orchestra.run.smithi118.stdout: "long_desc": "", 2024-03-24T03:17:24.540 INFO:teuthology.orchestra.run.smithi118.stdout: "tags": [], 2024-03-24T03:17:24.540 INFO:teuthology.orchestra.run.smithi118.stdout: "see_also": [] 2024-03-24T03:17:24.540 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-24T03:17:24.541 INFO:teuthology.orchestra.run.smithi118.stdout: "snapshot_clone_no_wait": { 2024-03-24T03:17:24.541 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "snapshot_clone_no_wait", 2024-03-24T03:17:24.541 INFO:teuthology.orchestra.run.smithi118.stdout: "type": "bool", 2024-03-24T03:17:24.541 INFO:teuthology.orchestra.run.smithi118.stdout: "level": "advanced", 2024-03-24T03:17:24.541 INFO:teuthology.orchestra.run.smithi118.stdout: "flags": 0, 2024-03-24T03:17:24.541 INFO:teuthology.orchestra.run.smithi118.stdout: "default_value": "True", 2024-03-24T03:17:24.541 INFO:teuthology.orchestra.run.smithi118.stdout: "min": "", 2024-03-24T03:17:24.541 INFO:teuthology.orchestra.run.smithi118.stdout: "max": "", 2024-03-24T03:17:24.542 INFO:teuthology.orchestra.run.smithi118.stdout: "enum_allowed": [], 2024-03-24T03:17:24.542 INFO:teuthology.orchestra.run.smithi118.stdout: "desc": "Reject subvolume clone request when cloner threads are busy", 2024-03-24T03:17:24.542 INFO:teuthology.orchestra.run.smithi118.stdout: "long_desc": "", 2024-03-24T03:17:24.542 INFO:teuthology.orchestra.run.smithi118.stdout: "tags": [], 2024-03-24T03:17:24.542 INFO:teuthology.orchestra.run.smithi118.stdout: "see_also": [] 2024-03-24T03:17:24.542 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-03-24T03:17:24.542 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-03-24T03:17:24.542 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-24T03:17:24.542 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-24T03:17:24.543 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "zabbix", 2024-03-24T03:17:24.543 INFO:teuthology.orchestra.run.smithi118.stdout: "can_run": true, 2024-03-24T03:17:24.543 INFO:teuthology.orchestra.run.smithi118.stdout: "error_string": "", 2024-03-24T03:17:24.543 INFO:teuthology.orchestra.run.smithi118.stdout: "module_options": { 2024-03-24T03:17:24.543 INFO:teuthology.orchestra.run.smithi118.stdout: "discovery_interval": { 2024-03-24T03:17:24.543 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "discovery_interval", 2024-03-24T03:17:24.543 INFO:teuthology.orchestra.run.smithi118.stdout: "type": "uint", 2024-03-24T03:17:24.543 INFO:teuthology.orchestra.run.smithi118.stdout: "level": "advanced", 2024-03-24T03:17:24.543 INFO:teuthology.orchestra.run.smithi118.stdout: "flags": 0, 2024-03-24T03:17:24.544 INFO:teuthology.orchestra.run.smithi118.stdout: "default_value": "100", 2024-03-24T03:17:24.544 INFO:teuthology.orchestra.run.smithi118.stdout: "min": "", 2024-03-24T03:17:24.544 INFO:teuthology.orchestra.run.smithi118.stdout: "max": "", 2024-03-24T03:17:24.544 INFO:teuthology.orchestra.run.smithi118.stdout: "enum_allowed": [], 2024-03-24T03:17:24.544 INFO:teuthology.orchestra.run.smithi118.stdout: "desc": "", 2024-03-24T03:17:24.544 INFO:teuthology.orchestra.run.smithi118.stdout: "long_desc": "", 2024-03-24T03:17:24.544 INFO:teuthology.orchestra.run.smithi118.stdout: "tags": [], 2024-03-24T03:17:24.544 INFO:teuthology.orchestra.run.smithi118.stdout: "see_also": [] 2024-03-24T03:17:24.544 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-24T03:17:24.545 INFO:teuthology.orchestra.run.smithi118.stdout: "identifier": { 2024-03-24T03:17:24.545 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "identifier", 2024-03-24T03:17:24.545 INFO:teuthology.orchestra.run.smithi118.stdout: "type": "str", 2024-03-24T03:17:24.545 INFO:teuthology.orchestra.run.smithi118.stdout: "level": "advanced", 2024-03-24T03:17:24.545 INFO:teuthology.orchestra.run.smithi118.stdout: "flags": 0, 2024-03-24T03:17:24.545 INFO:teuthology.orchestra.run.smithi118.stdout: "default_value": "", 2024-03-24T03:17:24.545 INFO:teuthology.orchestra.run.smithi118.stdout: "min": "", 2024-03-24T03:17:24.545 INFO:teuthology.orchestra.run.smithi118.stdout: "max": "", 2024-03-24T03:17:24.545 INFO:teuthology.orchestra.run.smithi118.stdout: "enum_allowed": [], 2024-03-24T03:17:24.546 INFO:teuthology.orchestra.run.smithi118.stdout: "desc": "", 2024-03-24T03:17:24.546 INFO:teuthology.orchestra.run.smithi118.stdout: "long_desc": "", 2024-03-24T03:17:24.546 INFO:teuthology.orchestra.run.smithi118.stdout: "tags": [], 2024-03-24T03:17:24.546 INFO:teuthology.orchestra.run.smithi118.stdout: "see_also": [] 2024-03-24T03:17:24.546 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-24T03:17:24.546 INFO:teuthology.orchestra.run.smithi118.stdout: "interval": { 2024-03-24T03:17:24.546 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "interval", 2024-03-24T03:17:24.546 INFO:teuthology.orchestra.run.smithi118.stdout: "type": "secs", 2024-03-24T03:17:24.546 INFO:teuthology.orchestra.run.smithi118.stdout: "level": "advanced", 2024-03-24T03:17:24.547 INFO:teuthology.orchestra.run.smithi118.stdout: "flags": 0, 2024-03-24T03:17:24.547 INFO:teuthology.orchestra.run.smithi118.stdout: "default_value": "60", 2024-03-24T03:17:24.547 INFO:teuthology.orchestra.run.smithi118.stdout: "min": "", 2024-03-24T03:17:24.547 INFO:teuthology.orchestra.run.smithi118.stdout: "max": "", 2024-03-24T03:17:24.547 INFO:teuthology.orchestra.run.smithi118.stdout: "enum_allowed": [], 2024-03-24T03:17:24.547 INFO:teuthology.orchestra.run.smithi118.stdout: "desc": "", 2024-03-24T03:17:24.547 INFO:teuthology.orchestra.run.smithi118.stdout: "long_desc": "", 2024-03-24T03:17:24.547 INFO:teuthology.orchestra.run.smithi118.stdout: "tags": [], 2024-03-24T03:17:24.547 INFO:teuthology.orchestra.run.smithi118.stdout: "see_also": [] 2024-03-24T03:17:24.548 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-24T03:17:24.548 INFO:teuthology.orchestra.run.smithi118.stdout: "log_level": { 2024-03-24T03:17:24.548 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "log_level", 2024-03-24T03:17:24.548 INFO:teuthology.orchestra.run.smithi118.stdout: "type": "str", 2024-03-24T03:17:24.548 INFO:teuthology.orchestra.run.smithi118.stdout: "level": "advanced", 2024-03-24T03:17:24.548 INFO:teuthology.orchestra.run.smithi118.stdout: "flags": 1, 2024-03-24T03:17:24.548 INFO:teuthology.orchestra.run.smithi118.stdout: "default_value": "", 2024-03-24T03:17:24.548 INFO:teuthology.orchestra.run.smithi118.stdout: "min": "", 2024-03-24T03:17:24.548 INFO:teuthology.orchestra.run.smithi118.stdout: "max": "", 2024-03-24T03:17:24.549 INFO:teuthology.orchestra.run.smithi118.stdout: "enum_allowed": [ 2024-03-24T03:17:24.549 INFO:teuthology.orchestra.run.smithi118.stdout: "", 2024-03-24T03:17:24.549 INFO:teuthology.orchestra.run.smithi118.stdout: "critical", 2024-03-24T03:17:24.549 INFO:teuthology.orchestra.run.smithi118.stdout: "debug", 2024-03-24T03:17:24.549 INFO:teuthology.orchestra.run.smithi118.stdout: "error", 2024-03-24T03:17:24.549 INFO:teuthology.orchestra.run.smithi118.stdout: "info", 2024-03-24T03:17:24.549 INFO:teuthology.orchestra.run.smithi118.stdout: "warning" 2024-03-24T03:17:24.549 INFO:teuthology.orchestra.run.smithi118.stdout: ], 2024-03-24T03:17:24.549 INFO:teuthology.orchestra.run.smithi118.stdout: "desc": "", 2024-03-24T03:17:24.550 INFO:teuthology.orchestra.run.smithi118.stdout: "long_desc": "", 2024-03-24T03:17:24.550 INFO:teuthology.orchestra.run.smithi118.stdout: "tags": [], 2024-03-24T03:17:24.550 INFO:teuthology.orchestra.run.smithi118.stdout: "see_also": [] 2024-03-24T03:17:24.550 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-24T03:17:24.550 INFO:teuthology.orchestra.run.smithi118.stdout: "log_to_cluster": { 2024-03-24T03:17:24.550 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "log_to_cluster", 2024-03-24T03:17:24.550 INFO:teuthology.orchestra.run.smithi118.stdout: "type": "bool", 2024-03-24T03:17:24.550 INFO:teuthology.orchestra.run.smithi118.stdout: "level": "advanced", 2024-03-24T03:17:24.551 INFO:teuthology.orchestra.run.smithi118.stdout: "flags": 1, 2024-03-24T03:17:24.551 INFO:teuthology.orchestra.run.smithi118.stdout: "default_value": "False", 2024-03-24T03:17:24.551 INFO:teuthology.orchestra.run.smithi118.stdout: "min": "", 2024-03-24T03:17:24.551 INFO:teuthology.orchestra.run.smithi118.stdout: "max": "", 2024-03-24T03:17:24.551 INFO:teuthology.orchestra.run.smithi118.stdout: "enum_allowed": [], 2024-03-24T03:17:24.551 INFO:teuthology.orchestra.run.smithi118.stdout: "desc": "", 2024-03-24T03:17:24.551 INFO:teuthology.orchestra.run.smithi118.stdout: "long_desc": "", 2024-03-24T03:17:24.551 INFO:teuthology.orchestra.run.smithi118.stdout: "tags": [], 2024-03-24T03:17:24.551 INFO:teuthology.orchestra.run.smithi118.stdout: "see_also": [] 2024-03-24T03:17:24.551 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-24T03:17:24.552 INFO:teuthology.orchestra.run.smithi118.stdout: "log_to_cluster_level": { 2024-03-24T03:17:24.552 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "log_to_cluster_level", 2024-03-24T03:17:24.552 INFO:teuthology.orchestra.run.smithi118.stdout: "type": "str", 2024-03-24T03:17:24.552 INFO:teuthology.orchestra.run.smithi118.stdout: "level": "advanced", 2024-03-24T03:17:24.552 INFO:teuthology.orchestra.run.smithi118.stdout: "flags": 1, 2024-03-24T03:17:24.552 INFO:teuthology.orchestra.run.smithi118.stdout: "default_value": "info", 2024-03-24T03:17:24.552 INFO:teuthology.orchestra.run.smithi118.stdout: "min": "", 2024-03-24T03:17:24.552 INFO:teuthology.orchestra.run.smithi118.stdout: "max": "", 2024-03-24T03:17:24.552 INFO:teuthology.orchestra.run.smithi118.stdout: "enum_allowed": [ 2024-03-24T03:17:24.553 INFO:teuthology.orchestra.run.smithi118.stdout: "", 2024-03-24T03:17:24.553 INFO:teuthology.orchestra.run.smithi118.stdout: "critical", 2024-03-24T03:17:24.553 INFO:teuthology.orchestra.run.smithi118.stdout: "debug", 2024-03-24T03:17:24.553 INFO:teuthology.orchestra.run.smithi118.stdout: "error", 2024-03-24T03:17:24.553 INFO:teuthology.orchestra.run.smithi118.stdout: "info", 2024-03-24T03:17:24.553 INFO:teuthology.orchestra.run.smithi118.stdout: "warning" 2024-03-24T03:17:24.553 INFO:teuthology.orchestra.run.smithi118.stdout: ], 2024-03-24T03:17:24.553 INFO:teuthology.orchestra.run.smithi118.stdout: "desc": "", 2024-03-24T03:17:24.553 INFO:teuthology.orchestra.run.smithi118.stdout: "long_desc": "", 2024-03-24T03:17:24.554 INFO:teuthology.orchestra.run.smithi118.stdout: "tags": [], 2024-03-24T03:17:24.554 INFO:teuthology.orchestra.run.smithi118.stdout: "see_also": [] 2024-03-24T03:17:24.554 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-24T03:17:24.554 INFO:teuthology.orchestra.run.smithi118.stdout: "log_to_file": { 2024-03-24T03:17:24.554 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "log_to_file", 2024-03-24T03:17:24.554 INFO:teuthology.orchestra.run.smithi118.stdout: "type": "bool", 2024-03-24T03:17:24.554 INFO:teuthology.orchestra.run.smithi118.stdout: "level": "advanced", 2024-03-24T03:17:24.555 INFO:teuthology.orchestra.run.smithi118.stdout: "flags": 1, 2024-03-24T03:17:24.555 INFO:teuthology.orchestra.run.smithi118.stdout: "default_value": "False", 2024-03-24T03:17:24.555 INFO:teuthology.orchestra.run.smithi118.stdout: "min": "", 2024-03-24T03:17:24.555 INFO:teuthology.orchestra.run.smithi118.stdout: "max": "", 2024-03-24T03:17:24.555 INFO:teuthology.orchestra.run.smithi118.stdout: "enum_allowed": [], 2024-03-24T03:17:24.555 INFO:teuthology.orchestra.run.smithi118.stdout: "desc": "", 2024-03-24T03:17:24.555 INFO:teuthology.orchestra.run.smithi118.stdout: "long_desc": "", 2024-03-24T03:17:24.555 INFO:teuthology.orchestra.run.smithi118.stdout: "tags": [], 2024-03-24T03:17:24.555 INFO:teuthology.orchestra.run.smithi118.stdout: "see_also": [] 2024-03-24T03:17:24.556 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-24T03:17:24.556 INFO:teuthology.orchestra.run.smithi118.stdout: "zabbix_host": { 2024-03-24T03:17:24.556 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "zabbix_host", 2024-03-24T03:17:24.556 INFO:teuthology.orchestra.run.smithi118.stdout: "type": "str", 2024-03-24T03:17:24.556 INFO:teuthology.orchestra.run.smithi118.stdout: "level": "advanced", 2024-03-24T03:17:24.556 INFO:teuthology.orchestra.run.smithi118.stdout: "flags": 0, 2024-03-24T03:17:24.556 INFO:teuthology.orchestra.run.smithi118.stdout: "default_value": "", 2024-03-24T03:17:24.556 INFO:teuthology.orchestra.run.smithi118.stdout: "min": "", 2024-03-24T03:17:24.556 INFO:teuthology.orchestra.run.smithi118.stdout: "max": "", 2024-03-24T03:17:24.557 INFO:teuthology.orchestra.run.smithi118.stdout: "enum_allowed": [], 2024-03-24T03:17:24.557 INFO:teuthology.orchestra.run.smithi118.stdout: "desc": "", 2024-03-24T03:17:24.557 INFO:teuthology.orchestra.run.smithi118.stdout: "long_desc": "", 2024-03-24T03:17:24.557 INFO:teuthology.orchestra.run.smithi118.stdout: "tags": [], 2024-03-24T03:17:24.557 INFO:teuthology.orchestra.run.smithi118.stdout: "see_also": [] 2024-03-24T03:17:24.557 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-24T03:17:24.557 INFO:teuthology.orchestra.run.smithi118.stdout: "zabbix_port": { 2024-03-24T03:17:24.557 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "zabbix_port", 2024-03-24T03:17:24.557 INFO:teuthology.orchestra.run.smithi118.stdout: "type": "int", 2024-03-24T03:17:24.558 INFO:teuthology.orchestra.run.smithi118.stdout: "level": "advanced", 2024-03-24T03:17:24.558 INFO:teuthology.orchestra.run.smithi118.stdout: "flags": 0, 2024-03-24T03:17:24.558 INFO:teuthology.orchestra.run.smithi118.stdout: "default_value": "10051", 2024-03-24T03:17:24.558 INFO:teuthology.orchestra.run.smithi118.stdout: "min": "", 2024-03-24T03:17:24.558 INFO:teuthology.orchestra.run.smithi118.stdout: "max": "", 2024-03-24T03:17:24.558 INFO:teuthology.orchestra.run.smithi118.stdout: "enum_allowed": [], 2024-03-24T03:17:24.558 INFO:teuthology.orchestra.run.smithi118.stdout: "desc": "", 2024-03-24T03:17:24.558 INFO:teuthology.orchestra.run.smithi118.stdout: "long_desc": "", 2024-03-24T03:17:24.559 INFO:teuthology.orchestra.run.smithi118.stdout: "tags": [], 2024-03-24T03:17:24.559 INFO:teuthology.orchestra.run.smithi118.stdout: "see_also": [] 2024-03-24T03:17:24.559 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-24T03:17:24.559 INFO:teuthology.orchestra.run.smithi118.stdout: "zabbix_sender": { 2024-03-24T03:17:24.559 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "zabbix_sender", 2024-03-24T03:17:24.559 INFO:teuthology.orchestra.run.smithi118.stdout: "type": "str", 2024-03-24T03:17:24.559 INFO:teuthology.orchestra.run.smithi118.stdout: "level": "advanced", 2024-03-24T03:17:24.559 INFO:teuthology.orchestra.run.smithi118.stdout: "flags": 0, 2024-03-24T03:17:24.559 INFO:teuthology.orchestra.run.smithi118.stdout: "default_value": "/usr/bin/zabbix_sender", 2024-03-24T03:17:24.560 INFO:teuthology.orchestra.run.smithi118.stdout: "min": "", 2024-03-24T03:17:24.560 INFO:teuthology.orchestra.run.smithi118.stdout: "max": "", 2024-03-24T03:17:24.560 INFO:teuthology.orchestra.run.smithi118.stdout: "enum_allowed": [], 2024-03-24T03:17:24.560 INFO:teuthology.orchestra.run.smithi118.stdout: "desc": "", 2024-03-24T03:17:24.560 INFO:teuthology.orchestra.run.smithi118.stdout: "long_desc": "", 2024-03-24T03:17:24.560 INFO:teuthology.orchestra.run.smithi118.stdout: "tags": [], 2024-03-24T03:17:24.560 INFO:teuthology.orchestra.run.smithi118.stdout: "see_also": [] 2024-03-24T03:17:24.560 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-03-24T03:17:24.560 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-03-24T03:17:24.560 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-03-24T03:17:24.561 INFO:teuthology.orchestra.run.smithi118.stdout: ], 2024-03-24T03:17:24.561 INFO:teuthology.orchestra.run.smithi118.stdout: "services": { 2024-03-24T03:17:24.561 INFO:teuthology.orchestra.run.smithi118.stdout: "dashboard": "https://172.21.15.118:8443/" 2024-03-24T03:17:24.561 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-24T03:17:24.561 INFO:teuthology.orchestra.run.smithi118.stdout: "always_on_modules": { 2024-03-24T03:17:24.561 INFO:teuthology.orchestra.run.smithi118.stdout: "octopus": [ 2024-03-24T03:17:24.561 INFO:teuthology.orchestra.run.smithi118.stdout: "balancer", 2024-03-24T03:17:24.561 INFO:teuthology.orchestra.run.smithi118.stdout: "crash", 2024-03-24T03:17:24.562 INFO:teuthology.orchestra.run.smithi118.stdout: "devicehealth", 2024-03-24T03:17:24.562 INFO:teuthology.orchestra.run.smithi118.stdout: "orchestrator", 2024-03-24T03:17:24.562 INFO:teuthology.orchestra.run.smithi118.stdout: "pg_autoscaler", 2024-03-24T03:17:24.562 INFO:teuthology.orchestra.run.smithi118.stdout: "progress", 2024-03-24T03:17:24.562 INFO:teuthology.orchestra.run.smithi118.stdout: "rbd_support", 2024-03-24T03:17:24.562 INFO:teuthology.orchestra.run.smithi118.stdout: "status", 2024-03-24T03:17:24.562 INFO:teuthology.orchestra.run.smithi118.stdout: "telemetry", 2024-03-24T03:17:24.562 INFO:teuthology.orchestra.run.smithi118.stdout: "volumes" 2024-03-24T03:17:24.562 INFO:teuthology.orchestra.run.smithi118.stdout: ], 2024-03-24T03:17:24.563 INFO:teuthology.orchestra.run.smithi118.stdout: "pacific": [ 2024-03-24T03:17:24.563 INFO:teuthology.orchestra.run.smithi118.stdout: "balancer", 2024-03-24T03:17:24.563 INFO:teuthology.orchestra.run.smithi118.stdout: "crash", 2024-03-24T03:17:24.563 INFO:teuthology.orchestra.run.smithi118.stdout: "devicehealth", 2024-03-24T03:17:24.563 INFO:teuthology.orchestra.run.smithi118.stdout: "orchestrator", 2024-03-24T03:17:24.563 INFO:teuthology.orchestra.run.smithi118.stdout: "pg_autoscaler", 2024-03-24T03:17:24.563 INFO:teuthology.orchestra.run.smithi118.stdout: "progress", 2024-03-24T03:17:24.563 INFO:teuthology.orchestra.run.smithi118.stdout: "rbd_support", 2024-03-24T03:17:24.564 INFO:teuthology.orchestra.run.smithi118.stdout: "status", 2024-03-24T03:17:24.564 INFO:teuthology.orchestra.run.smithi118.stdout: "telemetry", 2024-03-24T03:17:24.564 INFO:teuthology.orchestra.run.smithi118.stdout: "volumes" 2024-03-24T03:17:24.564 INFO:teuthology.orchestra.run.smithi118.stdout: ], 2024-03-24T03:17:24.564 INFO:teuthology.orchestra.run.smithi118.stdout: "quincy": [ 2024-03-24T03:17:24.564 INFO:teuthology.orchestra.run.smithi118.stdout: "balancer", 2024-03-24T03:17:24.564 INFO:teuthology.orchestra.run.smithi118.stdout: "crash", 2024-03-24T03:17:24.564 INFO:teuthology.orchestra.run.smithi118.stdout: "devicehealth", 2024-03-24T03:17:24.565 INFO:teuthology.orchestra.run.smithi118.stdout: "orchestrator", 2024-03-24T03:17:24.565 INFO:teuthology.orchestra.run.smithi118.stdout: "pg_autoscaler", 2024-03-24T03:17:24.565 INFO:teuthology.orchestra.run.smithi118.stdout: "progress", 2024-03-24T03:17:24.565 INFO:teuthology.orchestra.run.smithi118.stdout: "rbd_support", 2024-03-24T03:17:24.565 INFO:teuthology.orchestra.run.smithi118.stdout: "status", 2024-03-24T03:17:24.565 INFO:teuthology.orchestra.run.smithi118.stdout: "telemetry", 2024-03-24T03:17:24.565 INFO:teuthology.orchestra.run.smithi118.stdout: "volumes" 2024-03-24T03:17:24.565 INFO:teuthology.orchestra.run.smithi118.stdout: ], 2024-03-24T03:17:24.565 INFO:teuthology.orchestra.run.smithi118.stdout: "reef": [ 2024-03-24T03:17:24.566 INFO:teuthology.orchestra.run.smithi118.stdout: "balancer", 2024-03-24T03:17:24.566 INFO:teuthology.orchestra.run.smithi118.stdout: "crash", 2024-03-24T03:17:24.566 INFO:teuthology.orchestra.run.smithi118.stdout: "devicehealth", 2024-03-24T03:17:24.566 INFO:teuthology.orchestra.run.smithi118.stdout: "orchestrator", 2024-03-24T03:17:24.566 INFO:teuthology.orchestra.run.smithi118.stdout: "pg_autoscaler", 2024-03-24T03:17:24.566 INFO:teuthology.orchestra.run.smithi118.stdout: "progress", 2024-03-24T03:17:24.566 INFO:teuthology.orchestra.run.smithi118.stdout: "rbd_support", 2024-03-24T03:17:24.566 INFO:teuthology.orchestra.run.smithi118.stdout: "status", 2024-03-24T03:17:24.567 INFO:teuthology.orchestra.run.smithi118.stdout: "telemetry", 2024-03-24T03:17:24.567 INFO:teuthology.orchestra.run.smithi118.stdout: "volumes" 2024-03-24T03:17:24.567 INFO:teuthology.orchestra.run.smithi118.stdout: ], 2024-03-24T03:17:24.567 INFO:teuthology.orchestra.run.smithi118.stdout: "squid": [ 2024-03-24T03:17:24.567 INFO:teuthology.orchestra.run.smithi118.stdout: "balancer", 2024-03-24T03:17:24.567 INFO:teuthology.orchestra.run.smithi118.stdout: "crash", 2024-03-24T03:17:24.567 INFO:teuthology.orchestra.run.smithi118.stdout: "devicehealth", 2024-03-24T03:17:24.567 INFO:teuthology.orchestra.run.smithi118.stdout: "orchestrator", 2024-03-24T03:17:24.567 INFO:teuthology.orchestra.run.smithi118.stdout: "pg_autoscaler", 2024-03-24T03:17:24.568 INFO:teuthology.orchestra.run.smithi118.stdout: "progress", 2024-03-24T03:17:24.568 INFO:teuthology.orchestra.run.smithi118.stdout: "rbd_support", 2024-03-24T03:17:24.568 INFO:teuthology.orchestra.run.smithi118.stdout: "status", 2024-03-24T03:17:24.568 INFO:teuthology.orchestra.run.smithi118.stdout: "telemetry", 2024-03-24T03:17:24.568 INFO:teuthology.orchestra.run.smithi118.stdout: "volumes" 2024-03-24T03:17:24.568 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-03-24T03:17:24.568 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-24T03:17:24.568 INFO:teuthology.orchestra.run.smithi118.stdout: "last_failure_osd_epoch": 3, 2024-03-24T03:17:24.569 INFO:teuthology.orchestra.run.smithi118.stdout: "active_clients": [ 2024-03-24T03:17:24.569 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-24T03:17:24.569 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "devicehealth", 2024-03-24T03:17:24.569 INFO:teuthology.orchestra.run.smithi118.stdout: "addrvec": [ 2024-03-24T03:17:24.569 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-24T03:17:24.569 INFO:teuthology.orchestra.run.smithi118.stdout: "type": "v2", 2024-03-24T03:17:24.569 INFO:teuthology.orchestra.run.smithi118.stdout: "addr": "172.21.15.118:0", 2024-03-24T03:17:24.569 INFO:teuthology.orchestra.run.smithi118.stdout: "nonce": 1128391920 2024-03-24T03:17:24.569 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-03-24T03:17:24.569 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-03-24T03:17:24.570 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-24T03:17:24.570 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-24T03:17:24.570 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "libcephsqlite", 2024-03-24T03:17:24.570 INFO:teuthology.orchestra.run.smithi118.stdout: "addrvec": [ 2024-03-24T03:17:24.570 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-24T03:17:24.570 INFO:teuthology.orchestra.run.smithi118.stdout: "type": "v2", 2024-03-24T03:17:24.570 INFO:teuthology.orchestra.run.smithi118.stdout: "addr": "172.21.15.118:0", 2024-03-24T03:17:24.570 INFO:teuthology.orchestra.run.smithi118.stdout: "nonce": 533985636 2024-03-24T03:17:24.571 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-03-24T03:17:24.571 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-03-24T03:17:24.571 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-24T03:17:24.571 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-24T03:17:24.571 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "rbd_support", 2024-03-24T03:17:24.571 INFO:teuthology.orchestra.run.smithi118.stdout: "addrvec": [ 2024-03-24T03:17:24.571 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-24T03:17:24.571 INFO:teuthology.orchestra.run.smithi118.stdout: "type": "v2", 2024-03-24T03:17:24.571 INFO:teuthology.orchestra.run.smithi118.stdout: "addr": "172.21.15.118:0", 2024-03-24T03:17:24.572 INFO:teuthology.orchestra.run.smithi118.stdout: "nonce": 1198929553 2024-03-24T03:17:24.572 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-03-24T03:17:24.572 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-03-24T03:17:24.572 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-24T03:17:24.572 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-24T03:17:24.572 INFO:teuthology.orchestra.run.smithi118.stdout: "name": "volumes", 2024-03-24T03:17:24.572 INFO:teuthology.orchestra.run.smithi118.stdout: "addrvec": [ 2024-03-24T03:17:24.572 INFO:teuthology.orchestra.run.smithi118.stdout: { 2024-03-24T03:17:24.572 INFO:teuthology.orchestra.run.smithi118.stdout: "type": "v2", 2024-03-24T03:17:24.572 INFO:teuthology.orchestra.run.smithi118.stdout: "addr": "172.21.15.118:0", 2024-03-24T03:17:24.573 INFO:teuthology.orchestra.run.smithi118.stdout: "nonce": 2229730490 2024-03-24T03:17:24.573 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-03-24T03:17:24.573 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-03-24T03:17:24.573 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-03-24T03:17:24.573 INFO:teuthology.orchestra.run.smithi118.stdout: ] 2024-03-24T03:17:24.573 INFO:teuthology.orchestra.run.smithi118.stdout:} 2024-03-24T03:17:24.575 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 03:17:22 smithi193 ceph-mon[36878]: from='client.? 172.21.15.118:0/520427555' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-24T03:17:24.575 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 03:17:22 smithi193 ceph-mon[36878]: from='client.? 172.21.15.118:0/278904405' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-24T03:17:24.575 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 03:17:23 smithi193 ceph-mon[36878]: pgmap v6096: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:17:24.575 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 03:17:23 smithi193 ceph-mon[36878]: from='client.? 172.21.15.118:0/2706433962' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json-pretty"}]: dispatch 2024-03-24T03:17:24.575 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 03:17:22 smithi189 ceph-mon[36187]: from='client.? 172.21.15.118:0/520427555' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-24T03:17:24.575 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 03:17:22 smithi189 ceph-mon[36187]: from='client.? 172.21.15.118:0/278904405' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-24T03:17:24.575 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 03:17:23 smithi189 ceph-mon[36187]: pgmap v6096: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:17:24.575 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 03:17:23 smithi189 ceph-mon[36187]: from='client.? 172.21.15.118:0/2706433962' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json-pretty"}]: dispatch 2024-03-24T03:17:24.576 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T03:17:22.450+0000 7f929d800640 1 -- 172.21.15.118:0/2706433962 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7f927c0a2880 msgr2=0x7f927c0a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T03:17:24.576 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T03:17:22.450+0000 7f929d800640 1 --2- 172.21.15.118:0/2706433962 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7f927c0a2880 0x7f927c0a4d40 secure :-1 s=READY pgs=640 cs=0 l=1 rev1=1 crypto rx=0x7f928002c240 tx=0x7f928002df90 comp rx=0 tx=0).stop 2024-03-24T03:17:24.576 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T03:17:22.450+0000 7f929d800640 1 -- 172.21.15.118:0/2706433962 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f92981038e0 msgr2=0x7f9298199c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T03:17:24.576 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T03:17:22.450+0000 7f929d800640 1 --2- 172.21.15.118:0/2706433962 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f92981038e0 0x7f9298199c50 secure :-1 s=READY pgs=476 cs=0 l=1 rev1=1 crypto rx=0x7f928800ebc0 tx=0x7f928800c6a0 comp rx=0 tx=0).stop 2024-03-24T03:17:24.577 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T03:17:22.450+0000 7f929d800640 1 -- 172.21.15.118:0/2706433962 shutdown_connections 2024-03-24T03:17:24.577 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T03:17:22.450+0000 7f929d800640 1 --2- 172.21.15.118:0/2706433962 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7f927c0a2880 0x7f927c0a4d40 unknown :-1 s=CLOSED pgs=640 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T03:17:24.577 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T03:17:22.450+0000 7f929d800640 1 --2- 172.21.15.118:0/2706433962 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f9298105490 0x7f92981a11c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T03:17:24.577 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T03:17:22.450+0000 7f929d800640 1 --2- 172.21.15.118:0/2706433962 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f9298104ad0 0x7f929819a190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T03:17:24.577 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T03:17:22.450+0000 7f929d800640 1 --2- 172.21.15.118:0/2706433962 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f92981038e0 0x7f9298199c50 unknown :-1 s=CLOSED pgs=476 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T03:17:24.577 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T03:17:22.450+0000 7f929d800640 1 -- 172.21.15.118:0/2706433962 >> 172.21.15.118:0/2706433962 conn(0x7f92980fee50 msgr2=0x7f9298107e50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T03:17:24.577 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T03:17:22.450+0000 7f929d800640 1 -- 172.21.15.118:0/2706433962 shutdown_connections 2024-03-24T03:17:24.577 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T03:17:22.450+0000 7f929d800640 1 -- 172.21.15.118:0/2706433962 wait complete. 2024-03-24T03:17:24.584 DEBUG:tasks.check_counter:Getting stats from a 2024-03-24T03:17:24.584 DEBUG:teuthology.orchestra.run.smithi118:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 shell --fsid d79a5420-e971-11ee-95cd-87774f69a715 -- ceph daemon mds.a perf dump 2024-03-24T03:17:24.585 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 03:17:22 smithi118 ceph-mon[31535]: from='client.? 172.21.15.118:0/520427555' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-24T03:17:24.585 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 03:17:22 smithi118 ceph-mon[31535]: from='client.? 172.21.15.118:0/278904405' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-24T03:17:24.585 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 03:17:23 smithi118 ceph-mon[31535]: pgmap v6096: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:17:24.585 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 03:17:23 smithi118 ceph-mon[31535]: from='client.? 172.21.15.118:0/2706433962' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json-pretty"}]: dispatch 2024-03-24T03:17:25.031 INFO:teuthology.orchestra.run.smithi118.stderr:Inferring config /var/lib/ceph/d79a5420-e971-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T03:17:25.445 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 03:17:25 smithi118 ceph-mon[31535]: pgmap v6097: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 1.4 KiB/s wr, 8 op/s 2024-03-24T03:17:25.703 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 03:17:25 smithi189 ceph-mon[36187]: pgmap v6097: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 1.4 KiB/s wr, 8 op/s 2024-03-24T03:17:25.710 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 03:17:25 smithi193 ceph-mon[36878]: pgmap v6097: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 1.4 KiB/s wr, 8 op/s 2024-03-24T03:17:26.695 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 03:17:26 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 10341.761395 secs 2024-03-24T03:17:26.703 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 03:17:26 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 10341.761395 secs 2024-03-24T03:17:26.710 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 03:17:26 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 10341.761395 secs 2024-03-24T03:17:27.358 INFO:teuthology.orchestra.run.smithi118.stdout:{ 2024-03-24T03:17:27.358 INFO:teuthology.orchestra.run.smithi118.stdout: "AsyncMessenger::Worker-0": { 2024-03-24T03:17:27.358 INFO:teuthology.orchestra.run.smithi118.stdout: "msgr_recv_messages": 11034, 2024-03-24T03:17:27.358 INFO:teuthology.orchestra.run.smithi118.stdout: "msgr_send_messages": 11035, 2024-03-24T03:17:27.358 INFO:teuthology.orchestra.run.smithi118.stdout: "msgr_recv_bytes": 3707459, 2024-03-24T03:17:27.358 INFO:teuthology.orchestra.run.smithi118.stdout: "msgr_send_bytes": 3508495, 2024-03-24T03:17:27.359 INFO:teuthology.orchestra.run.smithi118.stdout: "msgr_created_connections": 18, 2024-03-24T03:17:27.359 INFO:teuthology.orchestra.run.smithi118.stdout: "msgr_active_connections": 2, 2024-03-24T03:17:27.359 INFO:teuthology.orchestra.run.smithi118.stdout: "msgr_running_total_time": 2.831034767, 2024-03-24T03:17:27.359 INFO:teuthology.orchestra.run.smithi118.stdout: "msgr_running_send_time": 1.284001991, 2024-03-24T03:17:27.359 INFO:teuthology.orchestra.run.smithi118.stdout: "msgr_running_recv_time": 0.611035753, 2024-03-24T03:17:27.359 INFO:teuthology.orchestra.run.smithi118.stdout: "msgr_running_fast_dispatch_time": 0.659555874, 2024-03-24T03:17:27.359 INFO:teuthology.orchestra.run.smithi118.stdout: "msgr_send_messages_queue_lat": { 2024-03-24T03:17:27.359 INFO:teuthology.orchestra.run.smithi118.stdout: "avgcount": 11033, 2024-03-24T03:17:27.359 INFO:teuthology.orchestra.run.smithi118.stdout: "sum": 1.166660911, 2024-03-24T03:17:27.360 INFO:teuthology.orchestra.run.smithi118.stdout: "avgtime": 0.000105742 2024-03-24T03:17:27.360 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-24T03:17:27.360 INFO:teuthology.orchestra.run.smithi118.stdout: "msgr_handle_ack_lat": { 2024-03-24T03:17:27.360 INFO:teuthology.orchestra.run.smithi118.stdout: "avgcount": 0, 2024-03-24T03:17:27.360 INFO:teuthology.orchestra.run.smithi118.stdout: "sum": 0.000000000, 2024-03-24T03:17:27.360 INFO:teuthology.orchestra.run.smithi118.stdout: "avgtime": 0.000000000 2024-03-24T03:17:27.360 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-24T03:17:27.360 INFO:teuthology.orchestra.run.smithi118.stdout: "msgr_recv_encrypted_bytes": 0, 2024-03-24T03:17:27.360 INFO:teuthology.orchestra.run.smithi118.stdout: "msgr_send_encrypted_bytes": 0 2024-03-24T03:17:27.361 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-24T03:17:27.361 INFO:teuthology.orchestra.run.smithi118.stdout: "AsyncMessenger::Worker-1": { 2024-03-24T03:17:27.361 INFO:teuthology.orchestra.run.smithi118.stdout: "msgr_recv_messages": 25025, 2024-03-24T03:17:27.361 INFO:teuthology.orchestra.run.smithi118.stdout: "msgr_send_messages": 24857, 2024-03-24T03:17:27.361 INFO:teuthology.orchestra.run.smithi118.stdout: "msgr_recv_bytes": 8026518, 2024-03-24T03:17:27.361 INFO:teuthology.orchestra.run.smithi118.stdout: "msgr_send_bytes": 8738769, 2024-03-24T03:17:27.361 INFO:teuthology.orchestra.run.smithi118.stdout: "msgr_created_connections": 4, 2024-03-24T03:17:27.361 INFO:teuthology.orchestra.run.smithi118.stdout: "msgr_active_connections": 3, 2024-03-24T03:17:27.361 INFO:teuthology.orchestra.run.smithi118.stdout: "msgr_running_total_time": 4.461996823, 2024-03-24T03:17:27.361 INFO:teuthology.orchestra.run.smithi118.stdout: "msgr_running_send_time": 1.671501067, 2024-03-24T03:17:27.362 INFO:teuthology.orchestra.run.smithi118.stdout: "msgr_running_recv_time": 1.158470271, 2024-03-24T03:17:27.362 INFO:teuthology.orchestra.run.smithi118.stdout: "msgr_running_fast_dispatch_time": 1.266713480, 2024-03-24T03:17:27.362 INFO:teuthology.orchestra.run.smithi118.stdout: "msgr_send_messages_queue_lat": { 2024-03-24T03:17:27.362 INFO:teuthology.orchestra.run.smithi118.stdout: "avgcount": 24857, 2024-03-24T03:17:27.362 INFO:teuthology.orchestra.run.smithi118.stdout: "sum": 2.358786040, 2024-03-24T03:17:27.362 INFO:teuthology.orchestra.run.smithi118.stdout: "avgtime": 0.000094894 2024-03-24T03:17:27.362 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-24T03:17:27.362 INFO:teuthology.orchestra.run.smithi118.stdout: "msgr_handle_ack_lat": { 2024-03-24T03:17:27.363 INFO:teuthology.orchestra.run.smithi118.stdout: "avgcount": 0, 2024-03-24T03:17:27.363 INFO:teuthology.orchestra.run.smithi118.stdout: "sum": 0.000000000, 2024-03-24T03:17:27.363 INFO:teuthology.orchestra.run.smithi118.stdout: "avgtime": 0.000000000 2024-03-24T03:17:27.363 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-24T03:17:27.363 INFO:teuthology.orchestra.run.smithi118.stdout: "msgr_recv_encrypted_bytes": 2459680, 2024-03-24T03:17:27.363 INFO:teuthology.orchestra.run.smithi118.stdout: "msgr_send_encrypted_bytes": 1771968 2024-03-24T03:17:27.363 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-24T03:17:27.363 INFO:teuthology.orchestra.run.smithi118.stdout: "AsyncMessenger::Worker-2": { 2024-03-24T03:17:27.363 INFO:teuthology.orchestra.run.smithi118.stdout: "msgr_recv_messages": 15, 2024-03-24T03:17:27.363 INFO:teuthology.orchestra.run.smithi118.stdout: "msgr_send_messages": 2232, 2024-03-24T03:17:27.364 INFO:teuthology.orchestra.run.smithi118.stdout: "msgr_recv_bytes": 3268, 2024-03-24T03:17:27.364 INFO:teuthology.orchestra.run.smithi118.stdout: "msgr_send_bytes": 4974805, 2024-03-24T03:17:27.364 INFO:teuthology.orchestra.run.smithi118.stdout: "msgr_created_connections": 39, 2024-03-24T03:17:27.364 INFO:teuthology.orchestra.run.smithi118.stdout: "msgr_active_connections": 4, 2024-03-24T03:17:27.364 INFO:teuthology.orchestra.run.smithi118.stdout: "msgr_running_total_time": 0.445956073, 2024-03-24T03:17:27.364 INFO:teuthology.orchestra.run.smithi118.stdout: "msgr_running_send_time": 0.347670587, 2024-03-24T03:17:27.364 INFO:teuthology.orchestra.run.smithi118.stdout: "msgr_running_recv_time": 0.039898607, 2024-03-24T03:17:27.364 INFO:teuthology.orchestra.run.smithi118.stdout: "msgr_running_fast_dispatch_time": 0.000055826, 2024-03-24T03:17:27.364 INFO:teuthology.orchestra.run.smithi118.stdout: "msgr_send_messages_queue_lat": { 2024-03-24T03:17:27.365 INFO:teuthology.orchestra.run.smithi118.stdout: "avgcount": 2230, 2024-03-24T03:17:27.365 INFO:teuthology.orchestra.run.smithi118.stdout: "sum": 0.253254163, 2024-03-24T03:17:27.365 INFO:teuthology.orchestra.run.smithi118.stdout: "avgtime": 0.000113566 2024-03-24T03:17:27.365 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-24T03:17:27.365 INFO:teuthology.orchestra.run.smithi118.stdout: "msgr_handle_ack_lat": { 2024-03-24T03:17:27.365 INFO:teuthology.orchestra.run.smithi118.stdout: "avgcount": 0, 2024-03-24T03:17:27.365 INFO:teuthology.orchestra.run.smithi118.stdout: "sum": 0.000000000, 2024-03-24T03:17:27.365 INFO:teuthology.orchestra.run.smithi118.stdout: "avgtime": 0.000000000 2024-03-24T03:17:27.365 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-24T03:17:27.366 INFO:teuthology.orchestra.run.smithi118.stdout: "msgr_recv_encrypted_bytes": 1920, 2024-03-24T03:17:27.366 INFO:teuthology.orchestra.run.smithi118.stdout: "msgr_send_encrypted_bytes": 4973840 2024-03-24T03:17:27.366 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-24T03:17:27.366 INFO:teuthology.orchestra.run.smithi118.stdout: "cct": { 2024-03-24T03:17:27.366 INFO:teuthology.orchestra.run.smithi118.stdout: "total_workers": 1, 2024-03-24T03:17:27.366 INFO:teuthology.orchestra.run.smithi118.stdout: "unhealthy_workers": 0 2024-03-24T03:17:27.366 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-24T03:17:27.366 INFO:teuthology.orchestra.run.smithi118.stdout: "finisher-MDSRank": { 2024-03-24T03:17:27.366 INFO:teuthology.orchestra.run.smithi118.stdout: "queue_len": 0, 2024-03-24T03:17:27.367 INFO:teuthology.orchestra.run.smithi118.stdout: "complete_latency": { 2024-03-24T03:17:27.367 INFO:teuthology.orchestra.run.smithi118.stdout: "avgcount": 54644, 2024-03-24T03:17:27.367 INFO:teuthology.orchestra.run.smithi118.stdout: "sum": 2.805416289, 2024-03-24T03:17:27.367 INFO:teuthology.orchestra.run.smithi118.stdout: "avgtime": 0.000051339 2024-03-24T03:17:27.367 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-03-24T03:17:27.367 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-24T03:17:27.367 INFO:teuthology.orchestra.run.smithi118.stdout: "finisher-PurgeQueue": { 2024-03-24T03:17:27.367 INFO:teuthology.orchestra.run.smithi118.stdout: "queue_len": 0, 2024-03-24T03:17:27.367 INFO:teuthology.orchestra.run.smithi118.stdout: "complete_latency": { 2024-03-24T03:17:27.368 INFO:teuthology.orchestra.run.smithi118.stdout: "avgcount": 0, 2024-03-24T03:17:27.368 INFO:teuthology.orchestra.run.smithi118.stdout: "sum": 0.000000000, 2024-03-24T03:17:27.368 INFO:teuthology.orchestra.run.smithi118.stdout: "avgtime": 0.000000000 2024-03-24T03:17:27.368 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-03-24T03:17:27.368 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-24T03:17:27.368 INFO:teuthology.orchestra.run.smithi118.stdout: "mds": { 2024-03-24T03:17:27.368 INFO:teuthology.orchestra.run.smithi118.stdout: "request": 0, 2024-03-24T03:17:27.368 INFO:teuthology.orchestra.run.smithi118.stdout: "reply": 0, 2024-03-24T03:17:27.368 INFO:teuthology.orchestra.run.smithi118.stdout: "reply_latency": { 2024-03-24T03:17:27.368 INFO:teuthology.orchestra.run.smithi118.stdout: "avgcount": 0, 2024-03-24T03:17:27.369 INFO:teuthology.orchestra.run.smithi118.stdout: "sum": 0.000000000, 2024-03-24T03:17:27.369 INFO:teuthology.orchestra.run.smithi118.stdout: "avgtime": 0.000000000 2024-03-24T03:17:27.369 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-24T03:17:27.369 INFO:teuthology.orchestra.run.smithi118.stdout: "slow_reply": 0, 2024-03-24T03:17:27.369 INFO:teuthology.orchestra.run.smithi118.stdout: "forward": 0, 2024-03-24T03:17:27.369 INFO:teuthology.orchestra.run.smithi118.stdout: "dir_fetch_complete": 0, 2024-03-24T03:17:27.369 INFO:teuthology.orchestra.run.smithi118.stdout: "dir_fetch_keys": 0, 2024-03-24T03:17:27.369 INFO:teuthology.orchestra.run.smithi118.stdout: "dir_commit": 0, 2024-03-24T03:17:27.369 INFO:teuthology.orchestra.run.smithi118.stdout: "dir_split": 0, 2024-03-24T03:17:27.369 INFO:teuthology.orchestra.run.smithi118.stdout: "dir_merge": 0, 2024-03-24T03:17:27.370 INFO:teuthology.orchestra.run.smithi118.stdout: "inodes": 0, 2024-03-24T03:17:27.370 INFO:teuthology.orchestra.run.smithi118.stdout: "inodes_top": 0, 2024-03-24T03:17:27.370 INFO:teuthology.orchestra.run.smithi118.stdout: "inodes_bottom": 0, 2024-03-24T03:17:27.370 INFO:teuthology.orchestra.run.smithi118.stdout: "inodes_pin_tail": 0, 2024-03-24T03:17:27.370 INFO:teuthology.orchestra.run.smithi118.stdout: "inodes_pinned": 0, 2024-03-24T03:17:27.370 INFO:teuthology.orchestra.run.smithi118.stdout: "inodes_expired": 0, 2024-03-24T03:17:27.370 INFO:teuthology.orchestra.run.smithi118.stdout: "inodes_with_caps": 0, 2024-03-24T03:17:27.370 INFO:teuthology.orchestra.run.smithi118.stdout: "caps": 0, 2024-03-24T03:17:27.371 INFO:teuthology.orchestra.run.smithi118.stdout: "subtrees": 1, 2024-03-24T03:17:27.371 INFO:teuthology.orchestra.run.smithi118.stdout: "traverse": 0, 2024-03-24T03:17:27.371 INFO:teuthology.orchestra.run.smithi118.stdout: "traverse_hit": 0, 2024-03-24T03:17:27.371 INFO:teuthology.orchestra.run.smithi118.stdout: "traverse_forward": 0, 2024-03-24T03:17:27.371 INFO:teuthology.orchestra.run.smithi118.stdout: "traverse_discover": 0, 2024-03-24T03:17:27.371 INFO:teuthology.orchestra.run.smithi118.stdout: "traverse_dir_fetch": 0, 2024-03-24T03:17:27.371 INFO:teuthology.orchestra.run.smithi118.stdout: "traverse_remote_ino": 0, 2024-03-24T03:17:27.371 INFO:teuthology.orchestra.run.smithi118.stdout: "traverse_lock": 0, 2024-03-24T03:17:27.371 INFO:teuthology.orchestra.run.smithi118.stdout: "load_cent": 0, 2024-03-24T03:17:27.372 INFO:teuthology.orchestra.run.smithi118.stdout: "q": 0, 2024-03-24T03:17:27.372 INFO:teuthology.orchestra.run.smithi118.stdout: "exported": 0, 2024-03-24T03:17:27.372 INFO:teuthology.orchestra.run.smithi118.stdout: "exported_inodes": 0, 2024-03-24T03:17:27.372 INFO:teuthology.orchestra.run.smithi118.stdout: "imported": 0, 2024-03-24T03:17:27.372 INFO:teuthology.orchestra.run.smithi118.stdout: "imported_inodes": 0, 2024-03-24T03:17:27.372 INFO:teuthology.orchestra.run.smithi118.stdout: "openino_dir_fetch": 0, 2024-03-24T03:17:27.372 INFO:teuthology.orchestra.run.smithi118.stdout: "openino_backtrace_fetch": 0, 2024-03-24T03:17:27.372 INFO:teuthology.orchestra.run.smithi118.stdout: "openino_peer_discover": 0, 2024-03-24T03:17:27.372 INFO:teuthology.orchestra.run.smithi118.stdout: "root_rfiles": 0, 2024-03-24T03:17:27.373 INFO:teuthology.orchestra.run.smithi118.stdout: "root_rbytes": 0, 2024-03-24T03:17:27.373 INFO:teuthology.orchestra.run.smithi118.stdout: "root_rsnaps": 0, 2024-03-24T03:17:27.373 INFO:teuthology.orchestra.run.smithi118.stdout: "scrub_backtrace_fetch": 0, 2024-03-24T03:17:27.373 INFO:teuthology.orchestra.run.smithi118.stdout: "scrub_set_tag": 0, 2024-03-24T03:17:27.373 INFO:teuthology.orchestra.run.smithi118.stdout: "scrub_backtrace_repaired": 0, 2024-03-24T03:17:27.373 INFO:teuthology.orchestra.run.smithi118.stdout: "scrub_inotable_repaired": 0, 2024-03-24T03:17:27.373 INFO:teuthology.orchestra.run.smithi118.stdout: "scrub_dir_inodes": 0, 2024-03-24T03:17:27.373 INFO:teuthology.orchestra.run.smithi118.stdout: "scrub_dir_base_inodes": 0, 2024-03-24T03:17:27.373 INFO:teuthology.orchestra.run.smithi118.stdout: "scrub_dirfrag_rstats": 0, 2024-03-24T03:17:27.374 INFO:teuthology.orchestra.run.smithi118.stdout: "scrub_file_inodes": 0, 2024-03-24T03:17:27.374 INFO:teuthology.orchestra.run.smithi118.stdout: "handle_inode_file_caps": 0, 2024-03-24T03:17:27.374 INFO:teuthology.orchestra.run.smithi118.stdout: "ceph_cap_op_revoke": 0, 2024-03-24T03:17:27.374 INFO:teuthology.orchestra.run.smithi118.stdout: "ceph_cap_op_grant": 0, 2024-03-24T03:17:27.374 INFO:teuthology.orchestra.run.smithi118.stdout: "ceph_cap_op_trunc": 0, 2024-03-24T03:17:27.374 INFO:teuthology.orchestra.run.smithi118.stdout: "ceph_cap_op_flushsnap_ack": 0, 2024-03-24T03:17:27.374 INFO:teuthology.orchestra.run.smithi118.stdout: "ceph_cap_op_flush_ack": 0, 2024-03-24T03:17:27.374 INFO:teuthology.orchestra.run.smithi118.stdout: "handle_client_caps": 0, 2024-03-24T03:17:27.375 INFO:teuthology.orchestra.run.smithi118.stdout: "handle_client_caps_dirty": 0, 2024-03-24T03:17:27.375 INFO:teuthology.orchestra.run.smithi118.stdout: "handle_client_cap_release": 0, 2024-03-24T03:17:27.375 INFO:teuthology.orchestra.run.smithi118.stdout: "process_request_cap_release": 0 2024-03-24T03:17:27.375 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-24T03:17:27.375 INFO:teuthology.orchestra.run.smithi118.stdout: "mds_cache": { 2024-03-24T03:17:27.375 INFO:teuthology.orchestra.run.smithi118.stdout: "dir_update": 0, 2024-03-24T03:17:27.375 INFO:teuthology.orchestra.run.smithi118.stdout: "dir_update_receipt": 0, 2024-03-24T03:17:27.375 INFO:teuthology.orchestra.run.smithi118.stdout: "dir_try_discover": 0, 2024-03-24T03:17:27.375 INFO:teuthology.orchestra.run.smithi118.stdout: "dir_send_discover": 0, 2024-03-24T03:17:27.376 INFO:teuthology.orchestra.run.smithi118.stdout: "dir_handle_discover": 0, 2024-03-24T03:17:27.376 INFO:teuthology.orchestra.run.smithi118.stdout: "num_strays": 0, 2024-03-24T03:17:27.376 INFO:teuthology.orchestra.run.smithi118.stdout: "num_strays_delayed": 0, 2024-03-24T03:17:27.376 INFO:teuthology.orchestra.run.smithi118.stdout: "num_strays_enqueuing": 0, 2024-03-24T03:17:27.376 INFO:teuthology.orchestra.run.smithi118.stdout: "strays_created": 0, 2024-03-24T03:17:27.376 INFO:teuthology.orchestra.run.smithi118.stdout: "strays_enqueued": 0, 2024-03-24T03:17:27.376 INFO:teuthology.orchestra.run.smithi118.stdout: "strays_reintegrated": 0, 2024-03-24T03:17:27.376 INFO:teuthology.orchestra.run.smithi118.stdout: "strays_migrated": 0, 2024-03-24T03:17:27.376 INFO:teuthology.orchestra.run.smithi118.stdout: "num_recovering_processing": 0, 2024-03-24T03:17:27.377 INFO:teuthology.orchestra.run.smithi118.stdout: "num_recovering_enqueued": 0, 2024-03-24T03:17:27.377 INFO:teuthology.orchestra.run.smithi118.stdout: "num_recovering_prioritized": 0, 2024-03-24T03:17:27.377 INFO:teuthology.orchestra.run.smithi118.stdout: "recovery_started": 0, 2024-03-24T03:17:27.377 INFO:teuthology.orchestra.run.smithi118.stdout: "recovery_completed": 0, 2024-03-24T03:17:27.377 INFO:teuthology.orchestra.run.smithi118.stdout: "ireq_enqueue_scrub": 0, 2024-03-24T03:17:27.377 INFO:teuthology.orchestra.run.smithi118.stdout: "ireq_exportdir": 0, 2024-03-24T03:17:27.377 INFO:teuthology.orchestra.run.smithi118.stdout: "ireq_flush": 0, 2024-03-24T03:17:27.377 INFO:teuthology.orchestra.run.smithi118.stdout: "ireq_fragmentdir": 0, 2024-03-24T03:17:27.377 INFO:teuthology.orchestra.run.smithi118.stdout: "ireq_fragstats": 0, 2024-03-24T03:17:27.378 INFO:teuthology.orchestra.run.smithi118.stdout: "ireq_inodestats": 0 2024-03-24T03:17:27.378 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-24T03:17:27.378 INFO:teuthology.orchestra.run.smithi118.stdout: "mds_log": { 2024-03-24T03:17:27.378 INFO:teuthology.orchestra.run.smithi118.stdout: "evlrg": 0, 2024-03-24T03:17:27.378 INFO:teuthology.orchestra.run.smithi118.stdout: "evadd": 0, 2024-03-24T03:17:27.378 INFO:teuthology.orchestra.run.smithi118.stdout: "evex": 0, 2024-03-24T03:17:27.378 INFO:teuthology.orchestra.run.smithi118.stdout: "evtrm": 0, 2024-03-24T03:17:27.378 INFO:teuthology.orchestra.run.smithi118.stdout: "ev": 3, 2024-03-24T03:17:27.379 INFO:teuthology.orchestra.run.smithi118.stdout: "evexg": 0, 2024-03-24T03:17:27.379 INFO:teuthology.orchestra.run.smithi118.stdout: "evexd": 0, 2024-03-24T03:17:27.379 INFO:teuthology.orchestra.run.smithi118.stdout: "segadd": 0, 2024-03-24T03:17:27.379 INFO:teuthology.orchestra.run.smithi118.stdout: "segex": 0, 2024-03-24T03:17:27.379 INFO:teuthology.orchestra.run.smithi118.stdout: "segtrm": 0, 2024-03-24T03:17:27.379 INFO:teuthology.orchestra.run.smithi118.stdout: "seg": 2, 2024-03-24T03:17:27.379 INFO:teuthology.orchestra.run.smithi118.stdout: "segmjr": 2, 2024-03-24T03:17:27.379 INFO:teuthology.orchestra.run.smithi118.stdout: "segexg": 0, 2024-03-24T03:17:27.379 INFO:teuthology.orchestra.run.smithi118.stdout: "segexd": 0, 2024-03-24T03:17:27.380 INFO:teuthology.orchestra.run.smithi118.stdout: "expos": 4194304, 2024-03-24T03:17:27.380 INFO:teuthology.orchestra.run.smithi118.stdout: "wrpos": 4195198, 2024-03-24T03:17:27.380 INFO:teuthology.orchestra.run.smithi118.stdout: "rdpos": 4194892, 2024-03-24T03:17:27.380 INFO:teuthology.orchestra.run.smithi118.stdout: "jlat": { 2024-03-24T03:17:27.380 INFO:teuthology.orchestra.run.smithi118.stdout: "avgcount": 0, 2024-03-24T03:17:27.380 INFO:teuthology.orchestra.run.smithi118.stdout: "sum": 0.000000000, 2024-03-24T03:17:27.380 INFO:teuthology.orchestra.run.smithi118.stdout: "avgtime": 0.000000000 2024-03-24T03:17:27.380 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-24T03:17:27.380 INFO:teuthology.orchestra.run.smithi118.stdout: "replayed": 3 2024-03-24T03:17:27.380 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-24T03:17:27.381 INFO:teuthology.orchestra.run.smithi118.stdout: "mds_mem": { 2024-03-24T03:17:27.381 INFO:teuthology.orchestra.run.smithi118.stdout: "ino": 3, 2024-03-24T03:17:27.381 INFO:teuthology.orchestra.run.smithi118.stdout: "ino+": 3, 2024-03-24T03:17:27.381 INFO:teuthology.orchestra.run.smithi118.stdout: "ino-": 0, 2024-03-24T03:17:27.381 INFO:teuthology.orchestra.run.smithi118.stdout: "dir": 1, 2024-03-24T03:17:27.381 INFO:teuthology.orchestra.run.smithi118.stdout: "dir+": 1, 2024-03-24T03:17:27.381 INFO:teuthology.orchestra.run.smithi118.stdout: "dir-": 0, 2024-03-24T03:17:27.381 INFO:teuthology.orchestra.run.smithi118.stdout: "dn": 0, 2024-03-24T03:17:27.381 INFO:teuthology.orchestra.run.smithi118.stdout: "dn+": 0, 2024-03-24T03:17:27.381 INFO:teuthology.orchestra.run.smithi118.stdout: "dn-": 0, 2024-03-24T03:17:27.382 INFO:teuthology.orchestra.run.smithi118.stdout: "cap": 0, 2024-03-24T03:17:27.382 INFO:teuthology.orchestra.run.smithi118.stdout: "cap+": 0, 2024-03-24T03:17:27.382 INFO:teuthology.orchestra.run.smithi118.stdout: "cap-": 0, 2024-03-24T03:17:27.382 INFO:teuthology.orchestra.run.smithi118.stdout: "rss": 49408, 2024-03-24T03:17:27.382 INFO:teuthology.orchestra.run.smithi118.stdout: "heap": 215420 2024-03-24T03:17:27.382 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-24T03:17:27.382 INFO:teuthology.orchestra.run.smithi118.stdout: "mds_server": { 2024-03-24T03:17:27.382 INFO:teuthology.orchestra.run.smithi118.stdout: "dispatch_client_request": 0, 2024-03-24T03:17:27.383 INFO:teuthology.orchestra.run.smithi118.stdout: "dispatch_server_request": 0, 2024-03-24T03:17:27.383 INFO:teuthology.orchestra.run.smithi118.stdout: "handle_client_request": 0, 2024-03-24T03:17:27.383 INFO:teuthology.orchestra.run.smithi118.stdout: "handle_client_session": 0, 2024-03-24T03:17:27.383 INFO:teuthology.orchestra.run.smithi118.stdout: "handle_peer_request": 0, 2024-03-24T03:17:27.383 INFO:teuthology.orchestra.run.smithi118.stdout: "req_create_latency": { 2024-03-24T03:17:27.383 INFO:teuthology.orchestra.run.smithi118.stdout: "avgcount": 0, 2024-03-24T03:17:27.383 INFO:teuthology.orchestra.run.smithi118.stdout: "sum": 0.000000000, 2024-03-24T03:17:27.383 INFO:teuthology.orchestra.run.smithi118.stdout: "avgtime": 0.000000000 2024-03-24T03:17:27.384 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-24T03:17:27.384 INFO:teuthology.orchestra.run.smithi118.stdout: "req_getattr_latency": { 2024-03-24T03:17:27.384 INFO:teuthology.orchestra.run.smithi118.stdout: "avgcount": 0, 2024-03-24T03:17:27.384 INFO:teuthology.orchestra.run.smithi118.stdout: "sum": 0.000000000, 2024-03-24T03:17:27.384 INFO:teuthology.orchestra.run.smithi118.stdout: "avgtime": 0.000000000 2024-03-24T03:17:27.384 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-24T03:17:27.384 INFO:teuthology.orchestra.run.smithi118.stdout: "req_getfilelock_latency": { 2024-03-24T03:17:27.384 INFO:teuthology.orchestra.run.smithi118.stdout: "avgcount": 0, 2024-03-24T03:17:27.385 INFO:teuthology.orchestra.run.smithi118.stdout: "sum": 0.000000000, 2024-03-24T03:17:27.385 INFO:teuthology.orchestra.run.smithi118.stdout: "avgtime": 0.000000000 2024-03-24T03:17:27.385 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-24T03:17:27.385 INFO:teuthology.orchestra.run.smithi118.stdout: "req_link_latency": { 2024-03-24T03:17:27.385 INFO:teuthology.orchestra.run.smithi118.stdout: "avgcount": 0, 2024-03-24T03:17:27.385 INFO:teuthology.orchestra.run.smithi118.stdout: "sum": 0.000000000, 2024-03-24T03:17:27.386 INFO:teuthology.orchestra.run.smithi118.stdout: "avgtime": 0.000000000 2024-03-24T03:17:27.386 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-24T03:17:27.386 INFO:teuthology.orchestra.run.smithi118.stdout: "req_lookup_latency": { 2024-03-24T03:17:27.386 INFO:teuthology.orchestra.run.smithi118.stdout: "avgcount": 0, 2024-03-24T03:17:27.386 INFO:teuthology.orchestra.run.smithi118.stdout: "sum": 0.000000000, 2024-03-24T03:17:27.386 INFO:teuthology.orchestra.run.smithi118.stdout: "avgtime": 0.000000000 2024-03-24T03:17:27.386 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-24T03:17:27.387 INFO:teuthology.orchestra.run.smithi118.stdout: "req_lookuphash_latency": { 2024-03-24T03:17:27.387 INFO:teuthology.orchestra.run.smithi118.stdout: "avgcount": 0, 2024-03-24T03:17:27.387 INFO:teuthology.orchestra.run.smithi118.stdout: "sum": 0.000000000, 2024-03-24T03:17:27.387 INFO:teuthology.orchestra.run.smithi118.stdout: "avgtime": 0.000000000 2024-03-24T03:17:27.387 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-24T03:17:27.387 INFO:teuthology.orchestra.run.smithi118.stdout: "req_lookupino_latency": { 2024-03-24T03:17:27.387 INFO:teuthology.orchestra.run.smithi118.stdout: "avgcount": 0, 2024-03-24T03:17:27.387 INFO:teuthology.orchestra.run.smithi118.stdout: "sum": 0.000000000, 2024-03-24T03:17:27.387 INFO:teuthology.orchestra.run.smithi118.stdout: "avgtime": 0.000000000 2024-03-24T03:17:27.388 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-24T03:17:27.388 INFO:teuthology.orchestra.run.smithi118.stdout: "req_lookupname_latency": { 2024-03-24T03:17:27.388 INFO:teuthology.orchestra.run.smithi118.stdout: "avgcount": 0, 2024-03-24T03:17:27.388 INFO:teuthology.orchestra.run.smithi118.stdout: "sum": 0.000000000, 2024-03-24T03:17:27.388 INFO:teuthology.orchestra.run.smithi118.stdout: "avgtime": 0.000000000 2024-03-24T03:17:27.388 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-24T03:17:27.388 INFO:teuthology.orchestra.run.smithi118.stdout: "req_lookupparent_latency": { 2024-03-24T03:17:27.388 INFO:teuthology.orchestra.run.smithi118.stdout: "avgcount": 0, 2024-03-24T03:17:27.388 INFO:teuthology.orchestra.run.smithi118.stdout: "sum": 0.000000000, 2024-03-24T03:17:27.389 INFO:teuthology.orchestra.run.smithi118.stdout: "avgtime": 0.000000000 2024-03-24T03:17:27.389 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-24T03:17:27.389 INFO:teuthology.orchestra.run.smithi118.stdout: "req_lookupsnap_latency": { 2024-03-24T03:17:27.389 INFO:teuthology.orchestra.run.smithi118.stdout: "avgcount": 0, 2024-03-24T03:17:27.389 INFO:teuthology.orchestra.run.smithi118.stdout: "sum": 0.000000000, 2024-03-24T03:17:27.389 INFO:teuthology.orchestra.run.smithi118.stdout: "avgtime": 0.000000000 2024-03-24T03:17:27.389 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-24T03:17:27.389 INFO:teuthology.orchestra.run.smithi118.stdout: "req_lssnap_latency": { 2024-03-24T03:17:27.390 INFO:teuthology.orchestra.run.smithi118.stdout: "avgcount": 0, 2024-03-24T03:17:27.390 INFO:teuthology.orchestra.run.smithi118.stdout: "sum": 0.000000000, 2024-03-24T03:17:27.390 INFO:teuthology.orchestra.run.smithi118.stdout: "avgtime": 0.000000000 2024-03-24T03:17:27.390 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-24T03:17:27.390 INFO:teuthology.orchestra.run.smithi118.stdout: "req_mkdir_latency": { 2024-03-24T03:17:27.390 INFO:teuthology.orchestra.run.smithi118.stdout: "avgcount": 0, 2024-03-24T03:17:27.390 INFO:teuthology.orchestra.run.smithi118.stdout: "sum": 0.000000000, 2024-03-24T03:17:27.390 INFO:teuthology.orchestra.run.smithi118.stdout: "avgtime": 0.000000000 2024-03-24T03:17:27.391 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-24T03:17:27.391 INFO:teuthology.orchestra.run.smithi118.stdout: "req_mknod_latency": { 2024-03-24T03:17:27.391 INFO:teuthology.orchestra.run.smithi118.stdout: "avgcount": 0, 2024-03-24T03:17:27.391 INFO:teuthology.orchestra.run.smithi118.stdout: "sum": 0.000000000, 2024-03-24T03:17:27.391 INFO:teuthology.orchestra.run.smithi118.stdout: "avgtime": 0.000000000 2024-03-24T03:17:27.392 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-24T03:17:27.392 INFO:teuthology.orchestra.run.smithi118.stdout: "req_mksnap_latency": { 2024-03-24T03:17:27.392 INFO:teuthology.orchestra.run.smithi118.stdout: "avgcount": 0, 2024-03-24T03:17:27.392 INFO:teuthology.orchestra.run.smithi118.stdout: "sum": 0.000000000, 2024-03-24T03:17:27.392 INFO:teuthology.orchestra.run.smithi118.stdout: "avgtime": 0.000000000 2024-03-24T03:17:27.392 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-24T03:17:27.392 INFO:teuthology.orchestra.run.smithi118.stdout: "req_open_latency": { 2024-03-24T03:17:27.392 INFO:teuthology.orchestra.run.smithi118.stdout: "avgcount": 0, 2024-03-24T03:17:27.392 INFO:teuthology.orchestra.run.smithi118.stdout: "sum": 0.000000000, 2024-03-24T03:17:27.392 INFO:teuthology.orchestra.run.smithi118.stdout: "avgtime": 0.000000000 2024-03-24T03:17:27.393 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-24T03:17:27.393 INFO:teuthology.orchestra.run.smithi118.stdout: "req_readdir_latency": { 2024-03-24T03:17:27.393 INFO:teuthology.orchestra.run.smithi118.stdout: "avgcount": 0, 2024-03-24T03:17:27.393 INFO:teuthology.orchestra.run.smithi118.stdout: "sum": 0.000000000, 2024-03-24T03:17:27.393 INFO:teuthology.orchestra.run.smithi118.stdout: "avgtime": 0.000000000 2024-03-24T03:17:27.393 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-24T03:17:27.393 INFO:teuthology.orchestra.run.smithi118.stdout: "req_rename_latency": { 2024-03-24T03:17:27.393 INFO:teuthology.orchestra.run.smithi118.stdout: "avgcount": 0, 2024-03-24T03:17:27.393 INFO:teuthology.orchestra.run.smithi118.stdout: "sum": 0.000000000, 2024-03-24T03:17:27.394 INFO:teuthology.orchestra.run.smithi118.stdout: "avgtime": 0.000000000 2024-03-24T03:17:27.394 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-24T03:17:27.394 INFO:teuthology.orchestra.run.smithi118.stdout: "req_renamesnap_latency": { 2024-03-24T03:17:27.394 INFO:teuthology.orchestra.run.smithi118.stdout: "avgcount": 0, 2024-03-24T03:17:27.394 INFO:teuthology.orchestra.run.smithi118.stdout: "sum": 0.000000000, 2024-03-24T03:17:27.394 INFO:teuthology.orchestra.run.smithi118.stdout: "avgtime": 0.000000000 2024-03-24T03:17:27.394 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-24T03:17:27.394 INFO:teuthology.orchestra.run.smithi118.stdout: "req_snapdiff_latency": { 2024-03-24T03:17:27.394 INFO:teuthology.orchestra.run.smithi118.stdout: "avgcount": 0, 2024-03-24T03:17:27.395 INFO:teuthology.orchestra.run.smithi118.stdout: "sum": 0.000000000, 2024-03-24T03:17:27.395 INFO:teuthology.orchestra.run.smithi118.stdout: "avgtime": 0.000000000 2024-03-24T03:17:27.395 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-24T03:17:27.395 INFO:teuthology.orchestra.run.smithi118.stdout: "req_rmdir_latency": { 2024-03-24T03:17:27.395 INFO:teuthology.orchestra.run.smithi118.stdout: "avgcount": 0, 2024-03-24T03:17:27.395 INFO:teuthology.orchestra.run.smithi118.stdout: "sum": 0.000000000, 2024-03-24T03:17:27.395 INFO:teuthology.orchestra.run.smithi118.stdout: "avgtime": 0.000000000 2024-03-24T03:17:27.395 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-24T03:17:27.396 INFO:teuthology.orchestra.run.smithi118.stdout: "req_rmsnap_latency": { 2024-03-24T03:17:27.396 INFO:teuthology.orchestra.run.smithi118.stdout: "avgcount": 0, 2024-03-24T03:17:27.396 INFO:teuthology.orchestra.run.smithi118.stdout: "sum": 0.000000000, 2024-03-24T03:17:27.396 INFO:teuthology.orchestra.run.smithi118.stdout: "avgtime": 0.000000000 2024-03-24T03:17:27.396 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-24T03:17:27.396 INFO:teuthology.orchestra.run.smithi118.stdout: "req_rmxattr_latency": { 2024-03-24T03:17:27.396 INFO:teuthology.orchestra.run.smithi118.stdout: "avgcount": 0, 2024-03-24T03:17:27.396 INFO:teuthology.orchestra.run.smithi118.stdout: "sum": 0.000000000, 2024-03-24T03:17:27.396 INFO:teuthology.orchestra.run.smithi118.stdout: "avgtime": 0.000000000 2024-03-24T03:17:27.396 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-24T03:17:27.397 INFO:teuthology.orchestra.run.smithi118.stdout: "req_setattr_latency": { 2024-03-24T03:17:27.397 INFO:teuthology.orchestra.run.smithi118.stdout: "avgcount": 0, 2024-03-24T03:17:27.397 INFO:teuthology.orchestra.run.smithi118.stdout: "sum": 0.000000000, 2024-03-24T03:17:27.397 INFO:teuthology.orchestra.run.smithi118.stdout: "avgtime": 0.000000000 2024-03-24T03:17:27.397 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-24T03:17:27.397 INFO:teuthology.orchestra.run.smithi118.stdout: "req_setdirlayout_latency": { 2024-03-24T03:17:27.397 INFO:teuthology.orchestra.run.smithi118.stdout: "avgcount": 0, 2024-03-24T03:17:27.397 INFO:teuthology.orchestra.run.smithi118.stdout: "sum": 0.000000000, 2024-03-24T03:17:27.397 INFO:teuthology.orchestra.run.smithi118.stdout: "avgtime": 0.000000000 2024-03-24T03:17:27.397 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-24T03:17:27.398 INFO:teuthology.orchestra.run.smithi118.stdout: "req_setfilelock_latency": { 2024-03-24T03:17:27.398 INFO:teuthology.orchestra.run.smithi118.stdout: "avgcount": 0, 2024-03-24T03:17:27.398 INFO:teuthology.orchestra.run.smithi118.stdout: "sum": 0.000000000, 2024-03-24T03:17:27.398 INFO:teuthology.orchestra.run.smithi118.stdout: "avgtime": 0.000000000 2024-03-24T03:17:27.398 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-24T03:17:27.398 INFO:teuthology.orchestra.run.smithi118.stdout: "req_setlayout_latency": { 2024-03-24T03:17:27.398 INFO:teuthology.orchestra.run.smithi118.stdout: "avgcount": 0, 2024-03-24T03:17:27.398 INFO:teuthology.orchestra.run.smithi118.stdout: "sum": 0.000000000, 2024-03-24T03:17:27.399 INFO:teuthology.orchestra.run.smithi118.stdout: "avgtime": 0.000000000 2024-03-24T03:17:27.399 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-24T03:17:27.399 INFO:teuthology.orchestra.run.smithi118.stdout: "req_setxattr_latency": { 2024-03-24T03:17:27.399 INFO:teuthology.orchestra.run.smithi118.stdout: "avgcount": 0, 2024-03-24T03:17:27.399 INFO:teuthology.orchestra.run.smithi118.stdout: "sum": 0.000000000, 2024-03-24T03:17:27.399 INFO:teuthology.orchestra.run.smithi118.stdout: "avgtime": 0.000000000 2024-03-24T03:17:27.399 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-24T03:17:27.399 INFO:teuthology.orchestra.run.smithi118.stdout: "req_symlink_latency": { 2024-03-24T03:17:27.399 INFO:teuthology.orchestra.run.smithi118.stdout: "avgcount": 0, 2024-03-24T03:17:27.400 INFO:teuthology.orchestra.run.smithi118.stdout: "sum": 0.000000000, 2024-03-24T03:17:27.400 INFO:teuthology.orchestra.run.smithi118.stdout: "avgtime": 0.000000000 2024-03-24T03:17:27.400 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-24T03:17:27.400 INFO:teuthology.orchestra.run.smithi118.stdout: "req_unlink_latency": { 2024-03-24T03:17:27.400 INFO:teuthology.orchestra.run.smithi118.stdout: "avgcount": 0, 2024-03-24T03:17:27.400 INFO:teuthology.orchestra.run.smithi118.stdout: "sum": 0.000000000, 2024-03-24T03:17:27.400 INFO:teuthology.orchestra.run.smithi118.stdout: "avgtime": 0.000000000 2024-03-24T03:17:27.400 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-24T03:17:27.400 INFO:teuthology.orchestra.run.smithi118.stdout: "cap_revoke_eviction": 0, 2024-03-24T03:17:27.401 INFO:teuthology.orchestra.run.smithi118.stdout: "cap_acquisition_throttle": 0, 2024-03-24T03:17:27.401 INFO:teuthology.orchestra.run.smithi118.stdout: "req_getvxattr_latency": { 2024-03-24T03:17:27.401 INFO:teuthology.orchestra.run.smithi118.stdout: "avgcount": 0, 2024-03-24T03:17:27.401 INFO:teuthology.orchestra.run.smithi118.stdout: "sum": 0.000000000, 2024-03-24T03:17:27.401 INFO:teuthology.orchestra.run.smithi118.stdout: "avgtime": 0.000000000 2024-03-24T03:17:27.401 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-03-24T03:17:27.401 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-24T03:17:27.401 INFO:teuthology.orchestra.run.smithi118.stdout: "mds_sessions": { 2024-03-24T03:17:27.402 INFO:teuthology.orchestra.run.smithi118.stdout: "session_count": 1, 2024-03-24T03:17:27.402 INFO:teuthology.orchestra.run.smithi118.stdout: "session_add": 1, 2024-03-24T03:17:27.402 INFO:teuthology.orchestra.run.smithi118.stdout: "session_remove": 0, 2024-03-24T03:17:27.402 INFO:teuthology.orchestra.run.smithi118.stdout: "sessions_open": 1, 2024-03-24T03:17:27.402 INFO:teuthology.orchestra.run.smithi118.stdout: "sessions_stale": 0, 2024-03-24T03:17:27.402 INFO:teuthology.orchestra.run.smithi118.stdout: "total_load": 0, 2024-03-24T03:17:27.402 INFO:teuthology.orchestra.run.smithi118.stdout: "average_load": 0, 2024-03-24T03:17:27.402 INFO:teuthology.orchestra.run.smithi118.stdout: "avg_session_uptime": 12222, 2024-03-24T03:17:27.403 INFO:teuthology.orchestra.run.smithi118.stdout: "mdthresh_evicted": 0 2024-03-24T03:17:27.403 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-24T03:17:27.403 INFO:teuthology.orchestra.run.smithi118.stdout: "mempool": { 2024-03-24T03:17:27.403 INFO:teuthology.orchestra.run.smithi118.stdout: "bloom_filter_bytes": 0, 2024-03-24T03:17:27.403 INFO:teuthology.orchestra.run.smithi118.stdout: "bloom_filter_items": 0, 2024-03-24T03:17:27.403 INFO:teuthology.orchestra.run.smithi118.stdout: "bluestore_alloc_bytes": 0, 2024-03-24T03:17:27.403 INFO:teuthology.orchestra.run.smithi118.stdout: "bluestore_alloc_items": 0, 2024-03-24T03:17:27.403 INFO:teuthology.orchestra.run.smithi118.stdout: "bluestore_cache_data_bytes": 0, 2024-03-24T03:17:27.404 INFO:teuthology.orchestra.run.smithi118.stdout: "bluestore_cache_data_items": 0, 2024-03-24T03:17:27.404 INFO:teuthology.orchestra.run.smithi118.stdout: "bluestore_cache_onode_bytes": 0, 2024-03-24T03:17:27.404 INFO:teuthology.orchestra.run.smithi118.stdout: "bluestore_cache_onode_items": 0, 2024-03-24T03:17:27.404 INFO:teuthology.orchestra.run.smithi118.stdout: "bluestore_cache_meta_bytes": 0, 2024-03-24T03:17:27.404 INFO:teuthology.orchestra.run.smithi118.stdout: "bluestore_cache_meta_items": 0, 2024-03-24T03:17:27.404 INFO:teuthology.orchestra.run.smithi118.stdout: "bluestore_cache_other_bytes": 0, 2024-03-24T03:17:27.404 INFO:teuthology.orchestra.run.smithi118.stdout: "bluestore_cache_other_items": 0, 2024-03-24T03:17:27.404 INFO:teuthology.orchestra.run.smithi118.stdout: "bluestore_cache_buffer_bytes": 0, 2024-03-24T03:17:27.404 INFO:teuthology.orchestra.run.smithi118.stdout: "bluestore_cache_buffer_items": 0, 2024-03-24T03:17:27.405 INFO:teuthology.orchestra.run.smithi118.stdout: "bluestore_extent_bytes": 0, 2024-03-24T03:17:27.405 INFO:teuthology.orchestra.run.smithi118.stdout: "bluestore_extent_items": 0, 2024-03-24T03:17:27.405 INFO:teuthology.orchestra.run.smithi118.stdout: "bluestore_blob_bytes": 0, 2024-03-24T03:17:27.405 INFO:teuthology.orchestra.run.smithi118.stdout: "bluestore_blob_items": 0, 2024-03-24T03:17:27.405 INFO:teuthology.orchestra.run.smithi118.stdout: "bluestore_shared_blob_bytes": 0, 2024-03-24T03:17:27.405 INFO:teuthology.orchestra.run.smithi118.stdout: "bluestore_shared_blob_items": 0, 2024-03-24T03:17:27.405 INFO:teuthology.orchestra.run.smithi118.stdout: "bluestore_inline_bl_bytes": 0, 2024-03-24T03:17:27.405 INFO:teuthology.orchestra.run.smithi118.stdout: "bluestore_inline_bl_items": 0, 2024-03-24T03:17:27.405 INFO:teuthology.orchestra.run.smithi118.stdout: "bluestore_fsck_bytes": 0, 2024-03-24T03:17:27.405 INFO:teuthology.orchestra.run.smithi118.stdout: "bluestore_fsck_items": 0, 2024-03-24T03:17:27.406 INFO:teuthology.orchestra.run.smithi118.stdout: "bluestore_txc_bytes": 0, 2024-03-24T03:17:27.406 INFO:teuthology.orchestra.run.smithi118.stdout: "bluestore_txc_items": 0, 2024-03-24T03:17:27.406 INFO:teuthology.orchestra.run.smithi118.stdout: "bluestore_writing_deferred_bytes": 0, 2024-03-24T03:17:27.406 INFO:teuthology.orchestra.run.smithi118.stdout: "bluestore_writing_deferred_items": 0, 2024-03-24T03:17:27.406 INFO:teuthology.orchestra.run.smithi118.stdout: "bluestore_writing_bytes": 0, 2024-03-24T03:17:27.406 INFO:teuthology.orchestra.run.smithi118.stdout: "bluestore_writing_items": 0, 2024-03-24T03:17:27.406 INFO:teuthology.orchestra.run.smithi118.stdout: "bluefs_bytes": 0, 2024-03-24T03:17:27.406 INFO:teuthology.orchestra.run.smithi118.stdout: "bluefs_items": 0, 2024-03-24T03:17:27.406 INFO:teuthology.orchestra.run.smithi118.stdout: "bluefs_file_reader_bytes": 0, 2024-03-24T03:17:27.407 INFO:teuthology.orchestra.run.smithi118.stdout: "bluefs_file_reader_items": 0, 2024-03-24T03:17:27.407 INFO:teuthology.orchestra.run.smithi118.stdout: "bluefs_file_writer_bytes": 0, 2024-03-24T03:17:27.407 INFO:teuthology.orchestra.run.smithi118.stdout: "bluefs_file_writer_items": 0, 2024-03-24T03:17:27.407 INFO:teuthology.orchestra.run.smithi118.stdout: "buffer_anon_bytes": 199015, 2024-03-24T03:17:27.407 INFO:teuthology.orchestra.run.smithi118.stdout: "buffer_anon_items": 57, 2024-03-24T03:17:27.407 INFO:teuthology.orchestra.run.smithi118.stdout: "buffer_meta_bytes": 0, 2024-03-24T03:17:27.407 INFO:teuthology.orchestra.run.smithi118.stdout: "buffer_meta_items": 0, 2024-03-24T03:17:27.407 INFO:teuthology.orchestra.run.smithi118.stdout: "osd_bytes": 0, 2024-03-24T03:17:27.407 INFO:teuthology.orchestra.run.smithi118.stdout: "osd_items": 0, 2024-03-24T03:17:27.408 INFO:teuthology.orchestra.run.smithi118.stdout: "osd_mapbl_bytes": 0, 2024-03-24T03:17:27.408 INFO:teuthology.orchestra.run.smithi118.stdout: "osd_mapbl_items": 0, 2024-03-24T03:17:27.408 INFO:teuthology.orchestra.run.smithi118.stdout: "osd_pglog_bytes": 0, 2024-03-24T03:17:27.408 INFO:teuthology.orchestra.run.smithi118.stdout: "osd_pglog_items": 0, 2024-03-24T03:17:27.408 INFO:teuthology.orchestra.run.smithi118.stdout: "osdmap_bytes": 12576, 2024-03-24T03:17:27.408 INFO:teuthology.orchestra.run.smithi118.stdout: "osdmap_items": 293, 2024-03-24T03:17:27.408 INFO:teuthology.orchestra.run.smithi118.stdout: "osdmap_mapping_bytes": 0, 2024-03-24T03:17:27.408 INFO:teuthology.orchestra.run.smithi118.stdout: "osdmap_mapping_items": 0, 2024-03-24T03:17:27.408 INFO:teuthology.orchestra.run.smithi118.stdout: "pgmap_bytes": 0, 2024-03-24T03:17:27.409 INFO:teuthology.orchestra.run.smithi118.stdout: "pgmap_items": 0, 2024-03-24T03:17:27.409 INFO:teuthology.orchestra.run.smithi118.stdout: "mds_co_bytes": 8320, 2024-03-24T03:17:27.409 INFO:teuthology.orchestra.run.smithi118.stdout: "mds_co_items": 13, 2024-03-24T03:17:27.409 INFO:teuthology.orchestra.run.smithi118.stdout: "unittest_1_bytes": 0, 2024-03-24T03:17:27.409 INFO:teuthology.orchestra.run.smithi118.stdout: "unittest_1_items": 0, 2024-03-24T03:17:27.409 INFO:teuthology.orchestra.run.smithi118.stdout: "unittest_2_bytes": 0, 2024-03-24T03:17:27.409 INFO:teuthology.orchestra.run.smithi118.stdout: "unittest_2_items": 0 2024-03-24T03:17:27.409 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-24T03:17:27.409 INFO:teuthology.orchestra.run.smithi118.stdout: "objecter": { 2024-03-24T03:17:27.409 INFO:teuthology.orchestra.run.smithi118.stdout: "op_active": 0, 2024-03-24T03:17:27.410 INFO:teuthology.orchestra.run.smithi118.stdout: "op_laggy": 0, 2024-03-24T03:17:27.410 INFO:teuthology.orchestra.run.smithi118.stdout: "op_send": 32944, 2024-03-24T03:17:27.410 INFO:teuthology.orchestra.run.smithi118.stdout: "op_send_bytes": 21, 2024-03-24T03:17:27.410 INFO:teuthology.orchestra.run.smithi118.stdout: "op_resend": 3, 2024-03-24T03:17:27.410 INFO:teuthology.orchestra.run.smithi118.stdout: "op_reply": 32944, 2024-03-24T03:17:27.410 INFO:teuthology.orchestra.run.smithi118.stdout: "op_latency": { 2024-03-24T03:17:27.410 INFO:teuthology.orchestra.run.smithi118.stdout: "avgcount": 32944, 2024-03-24T03:17:27.410 INFO:teuthology.orchestra.run.smithi118.stdout: "sum": 26.893368039, 2024-03-24T03:17:27.410 INFO:teuthology.orchestra.run.smithi118.stdout: "avgtime": 0.000816335 2024-03-24T03:17:27.411 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-24T03:17:27.411 INFO:teuthology.orchestra.run.smithi118.stdout: "op_inflight": 0, 2024-03-24T03:17:27.411 INFO:teuthology.orchestra.run.smithi118.stdout: "oplen_avg": { 2024-03-24T03:17:27.411 INFO:teuthology.orchestra.run.smithi118.stdout: "avgcount": 32944, 2024-03-24T03:17:27.411 INFO:teuthology.orchestra.run.smithi118.stdout: "sum": 32945 2024-03-24T03:17:27.411 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-24T03:17:27.411 INFO:teuthology.orchestra.run.smithi118.stdout: "op": 32944, 2024-03-24T03:17:27.411 INFO:teuthology.orchestra.run.smithi118.stdout: "op_r": 32944, 2024-03-24T03:17:27.411 INFO:teuthology.orchestra.run.smithi118.stdout: "op_w": 0, 2024-03-24T03:17:27.412 INFO:teuthology.orchestra.run.smithi118.stdout: "op_rmw": 0, 2024-03-24T03:17:27.412 INFO:teuthology.orchestra.run.smithi118.stdout: "op_pg": 0, 2024-03-24T03:17:27.412 INFO:teuthology.orchestra.run.smithi118.stdout: "osdop_stat": 21905, 2024-03-24T03:17:27.412 INFO:teuthology.orchestra.run.smithi118.stdout: "osdop_create": 0, 2024-03-24T03:17:27.412 INFO:teuthology.orchestra.run.smithi118.stdout: "osdop_read": 11037, 2024-03-24T03:17:27.412 INFO:teuthology.orchestra.run.smithi118.stdout: "osdop_write": 0, 2024-03-24T03:17:27.412 INFO:teuthology.orchestra.run.smithi118.stdout: "osdop_writefull": 0, 2024-03-24T03:17:27.412 INFO:teuthology.orchestra.run.smithi118.stdout: "osdop_writesame": 0, 2024-03-24T03:17:27.412 INFO:teuthology.orchestra.run.smithi118.stdout: "osdop_append": 0, 2024-03-24T03:17:27.413 INFO:teuthology.orchestra.run.smithi118.stdout: "osdop_zero": 0, 2024-03-24T03:17:27.413 INFO:teuthology.orchestra.run.smithi118.stdout: "osdop_truncate": 0, 2024-03-24T03:17:27.413 INFO:teuthology.orchestra.run.smithi118.stdout: "osdop_delete": 0, 2024-03-24T03:17:27.413 INFO:teuthology.orchestra.run.smithi118.stdout: "osdop_mapext": 0, 2024-03-24T03:17:27.413 INFO:teuthology.orchestra.run.smithi118.stdout: "osdop_sparse_read": 0, 2024-03-24T03:17:27.413 INFO:teuthology.orchestra.run.smithi118.stdout: "osdop_clonerange": 0, 2024-03-24T03:17:27.413 INFO:teuthology.orchestra.run.smithi118.stdout: "osdop_getxattr": 1, 2024-03-24T03:17:27.413 INFO:teuthology.orchestra.run.smithi118.stdout: "osdop_setxattr": 0, 2024-03-24T03:17:27.413 INFO:teuthology.orchestra.run.smithi118.stdout: "osdop_cmpxattr": 0, 2024-03-24T03:17:27.414 INFO:teuthology.orchestra.run.smithi118.stdout: "osdop_rmxattr": 0, 2024-03-24T03:17:27.414 INFO:teuthology.orchestra.run.smithi118.stdout: "osdop_resetxattrs": 0, 2024-03-24T03:17:27.414 INFO:teuthology.orchestra.run.smithi118.stdout: "osdop_call": 0, 2024-03-24T03:17:27.414 INFO:teuthology.orchestra.run.smithi118.stdout: "osdop_watch": 0, 2024-03-24T03:17:27.414 INFO:teuthology.orchestra.run.smithi118.stdout: "osdop_notify": 0, 2024-03-24T03:17:27.414 INFO:teuthology.orchestra.run.smithi118.stdout: "osdop_src_cmpxattr": 0, 2024-03-24T03:17:27.414 INFO:teuthology.orchestra.run.smithi118.stdout: "osdop_pgls": 0, 2024-03-24T03:17:27.414 INFO:teuthology.orchestra.run.smithi118.stdout: "osdop_pgls_filter": 0, 2024-03-24T03:17:27.415 INFO:teuthology.orchestra.run.smithi118.stdout: "osdop_other": 0, 2024-03-24T03:17:27.415 INFO:teuthology.orchestra.run.smithi118.stdout: "linger_active": 0, 2024-03-24T03:17:27.415 INFO:teuthology.orchestra.run.smithi118.stdout: "linger_send": 0, 2024-03-24T03:17:27.415 INFO:teuthology.orchestra.run.smithi118.stdout: "linger_resend": 0, 2024-03-24T03:17:27.415 INFO:teuthology.orchestra.run.smithi118.stdout: "linger_ping": 0, 2024-03-24T03:17:27.415 INFO:teuthology.orchestra.run.smithi118.stdout: "poolop_active": 0, 2024-03-24T03:17:27.415 INFO:teuthology.orchestra.run.smithi118.stdout: "poolop_send": 0, 2024-03-24T03:17:27.415 INFO:teuthology.orchestra.run.smithi118.stdout: "poolop_resend": 0, 2024-03-24T03:17:27.415 INFO:teuthology.orchestra.run.smithi118.stdout: "poolstat_active": 0, 2024-03-24T03:17:27.416 INFO:teuthology.orchestra.run.smithi118.stdout: "poolstat_send": 0, 2024-03-24T03:17:27.416 INFO:teuthology.orchestra.run.smithi118.stdout: "poolstat_resend": 0, 2024-03-24T03:17:27.416 INFO:teuthology.orchestra.run.smithi118.stdout: "statfs_active": 0, 2024-03-24T03:17:27.416 INFO:teuthology.orchestra.run.smithi118.stdout: "statfs_send": 0, 2024-03-24T03:17:27.416 INFO:teuthology.orchestra.run.smithi118.stdout: "statfs_resend": 0, 2024-03-24T03:17:27.416 INFO:teuthology.orchestra.run.smithi118.stdout: "command_active": 0, 2024-03-24T03:17:27.416 INFO:teuthology.orchestra.run.smithi118.stdout: "command_send": 0, 2024-03-24T03:17:27.416 INFO:teuthology.orchestra.run.smithi118.stdout: "command_resend": 0, 2024-03-24T03:17:27.416 INFO:teuthology.orchestra.run.smithi118.stdout: "map_epoch": 124, 2024-03-24T03:17:27.417 INFO:teuthology.orchestra.run.smithi118.stdout: "map_full": 0, 2024-03-24T03:17:27.417 INFO:teuthology.orchestra.run.smithi118.stdout: "map_inc": 38, 2024-03-24T03:17:27.417 INFO:teuthology.orchestra.run.smithi118.stdout: "osd_sessions": 6, 2024-03-24T03:17:27.417 INFO:teuthology.orchestra.run.smithi118.stdout: "osd_session_open": 41, 2024-03-24T03:17:27.417 INFO:teuthology.orchestra.run.smithi118.stdout: "osd_session_close": 35, 2024-03-24T03:17:27.417 INFO:teuthology.orchestra.run.smithi118.stdout: "osd_laggy": 0, 2024-03-24T03:17:27.417 INFO:teuthology.orchestra.run.smithi118.stdout: "omap_wr": 0, 2024-03-24T03:17:27.417 INFO:teuthology.orchestra.run.smithi118.stdout: "omap_rd": 2, 2024-03-24T03:17:27.417 INFO:teuthology.orchestra.run.smithi118.stdout: "omap_del": 0 2024-03-24T03:17:27.417 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-24T03:17:27.418 INFO:teuthology.orchestra.run.smithi118.stdout: "oft": { 2024-03-24T03:17:27.418 INFO:teuthology.orchestra.run.smithi118.stdout: "omap_total_objs": 0, 2024-03-24T03:17:27.418 INFO:teuthology.orchestra.run.smithi118.stdout: "omap_total_kv_pairs": 0, 2024-03-24T03:17:27.418 INFO:teuthology.orchestra.run.smithi118.stdout: "omap_total_updates": 0, 2024-03-24T03:17:27.418 INFO:teuthology.orchestra.run.smithi118.stdout: "omap_total_removes": 0 2024-03-24T03:17:27.418 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-24T03:17:27.418 INFO:teuthology.orchestra.run.smithi118.stdout: "osd-slow-ops": { 2024-03-24T03:17:27.418 INFO:teuthology.orchestra.run.smithi118.stdout: "slow_ops_count": 0 2024-03-24T03:17:27.418 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-24T03:17:27.419 INFO:teuthology.orchestra.run.smithi118.stdout: "purge_queue": { 2024-03-24T03:17:27.419 INFO:teuthology.orchestra.run.smithi118.stdout: "pq_executing_ops": 0, 2024-03-24T03:17:27.419 INFO:teuthology.orchestra.run.smithi118.stdout: "pq_executing_ops_high_water": 0, 2024-03-24T03:17:27.419 INFO:teuthology.orchestra.run.smithi118.stdout: "pq_executing": 0, 2024-03-24T03:17:27.419 INFO:teuthology.orchestra.run.smithi118.stdout: "pq_executing_high_water": 0, 2024-03-24T03:17:27.419 INFO:teuthology.orchestra.run.smithi118.stdout: "pq_executed_ops": 0, 2024-03-24T03:17:27.419 INFO:teuthology.orchestra.run.smithi118.stdout: "pq_executed": 0, 2024-03-24T03:17:27.419 INFO:teuthology.orchestra.run.smithi118.stdout: "pq_item_in_journal": 0 2024-03-24T03:17:27.419 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-24T03:17:27.419 INFO:teuthology.orchestra.run.smithi118.stdout: "throttle-msgr_dispatch_throttler-mds": { 2024-03-24T03:17:27.420 INFO:teuthology.orchestra.run.smithi118.stdout: "val": 0, 2024-03-24T03:17:27.420 INFO:teuthology.orchestra.run.smithi118.stdout: "max": 104857600, 2024-03-24T03:17:27.420 INFO:teuthology.orchestra.run.smithi118.stdout: "get_started": 0, 2024-03-24T03:17:27.420 INFO:teuthology.orchestra.run.smithi118.stdout: "get": 36070, 2024-03-24T03:17:27.420 INFO:teuthology.orchestra.run.smithi118.stdout: "get_sum": 8327914, 2024-03-24T03:17:27.420 INFO:teuthology.orchestra.run.smithi118.stdout: "get_or_fail_fail": 0, 2024-03-24T03:17:27.420 INFO:teuthology.orchestra.run.smithi118.stdout: "get_or_fail_success": 36070, 2024-03-24T03:17:27.420 INFO:teuthology.orchestra.run.smithi118.stdout: "take": 0, 2024-03-24T03:17:27.420 INFO:teuthology.orchestra.run.smithi118.stdout: "take_sum": 0, 2024-03-24T03:17:27.421 INFO:teuthology.orchestra.run.smithi118.stdout: "put": 36070, 2024-03-24T03:17:27.421 INFO:teuthology.orchestra.run.smithi118.stdout: "put_sum": 8327914, 2024-03-24T03:17:27.421 INFO:teuthology.orchestra.run.smithi118.stdout: "wait": { 2024-03-24T03:17:27.421 INFO:teuthology.orchestra.run.smithi118.stdout: "avgcount": 0, 2024-03-24T03:17:27.421 INFO:teuthology.orchestra.run.smithi118.stdout: "sum": 0.000000000, 2024-03-24T03:17:27.421 INFO:teuthology.orchestra.run.smithi118.stdout: "avgtime": 0.000000000 2024-03-24T03:17:27.421 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-03-24T03:17:27.421 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-24T03:17:27.421 INFO:teuthology.orchestra.run.smithi118.stdout: "throttle-objecter_bytes": { 2024-03-24T03:17:27.422 INFO:teuthology.orchestra.run.smithi118.stdout: "val": 0, 2024-03-24T03:17:27.422 INFO:teuthology.orchestra.run.smithi118.stdout: "max": 104857600, 2024-03-24T03:17:27.422 INFO:teuthology.orchestra.run.smithi118.stdout: "get_started": 0, 2024-03-24T03:17:27.422 INFO:teuthology.orchestra.run.smithi118.stdout: "get": 0, 2024-03-24T03:17:27.422 INFO:teuthology.orchestra.run.smithi118.stdout: "get_sum": 0, 2024-03-24T03:17:27.422 INFO:teuthology.orchestra.run.smithi118.stdout: "get_or_fail_fail": 0, 2024-03-24T03:17:27.422 INFO:teuthology.orchestra.run.smithi118.stdout: "get_or_fail_success": 0, 2024-03-24T03:17:27.422 INFO:teuthology.orchestra.run.smithi118.stdout: "take": 32944, 2024-03-24T03:17:27.422 INFO:teuthology.orchestra.run.smithi118.stdout: "take_sum": 899, 2024-03-24T03:17:27.423 INFO:teuthology.orchestra.run.smithi118.stdout: "put": 32944, 2024-03-24T03:17:27.423 INFO:teuthology.orchestra.run.smithi118.stdout: "put_sum": 899, 2024-03-24T03:17:27.423 INFO:teuthology.orchestra.run.smithi118.stdout: "wait": { 2024-03-24T03:17:27.423 INFO:teuthology.orchestra.run.smithi118.stdout: "avgcount": 0, 2024-03-24T03:17:27.423 INFO:teuthology.orchestra.run.smithi118.stdout: "sum": 0.000000000, 2024-03-24T03:17:27.423 INFO:teuthology.orchestra.run.smithi118.stdout: "avgtime": 0.000000000 2024-03-24T03:17:27.423 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-03-24T03:17:27.423 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-24T03:17:27.423 INFO:teuthology.orchestra.run.smithi118.stdout: "throttle-objecter_ops": { 2024-03-24T03:17:27.424 INFO:teuthology.orchestra.run.smithi118.stdout: "val": 0, 2024-03-24T03:17:27.424 INFO:teuthology.orchestra.run.smithi118.stdout: "max": 1024, 2024-03-24T03:17:27.424 INFO:teuthology.orchestra.run.smithi118.stdout: "get_started": 0, 2024-03-24T03:17:27.424 INFO:teuthology.orchestra.run.smithi118.stdout: "get": 0, 2024-03-24T03:17:27.424 INFO:teuthology.orchestra.run.smithi118.stdout: "get_sum": 0, 2024-03-24T03:17:27.424 INFO:teuthology.orchestra.run.smithi118.stdout: "get_or_fail_fail": 0, 2024-03-24T03:17:27.424 INFO:teuthology.orchestra.run.smithi118.stdout: "get_or_fail_success": 0, 2024-03-24T03:17:27.424 INFO:teuthology.orchestra.run.smithi118.stdout: "take": 32944, 2024-03-24T03:17:27.424 INFO:teuthology.orchestra.run.smithi118.stdout: "take_sum": 32944, 2024-03-24T03:17:27.424 INFO:teuthology.orchestra.run.smithi118.stdout: "put": 32944, 2024-03-24T03:17:27.425 INFO:teuthology.orchestra.run.smithi118.stdout: "put_sum": 32944, 2024-03-24T03:17:27.425 INFO:teuthology.orchestra.run.smithi118.stdout: "wait": { 2024-03-24T03:17:27.425 INFO:teuthology.orchestra.run.smithi118.stdout: "avgcount": 0, 2024-03-24T03:17:27.425 INFO:teuthology.orchestra.run.smithi118.stdout: "sum": 0.000000000, 2024-03-24T03:17:27.425 INFO:teuthology.orchestra.run.smithi118.stdout: "avgtime": 0.000000000 2024-03-24T03:17:27.425 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-03-24T03:17:27.425 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-24T03:17:27.425 INFO:teuthology.orchestra.run.smithi118.stdout: "throttle-write_buf_throttle": { 2024-03-24T03:17:27.425 INFO:teuthology.orchestra.run.smithi118.stdout: "val": 0, 2024-03-24T03:17:27.426 INFO:teuthology.orchestra.run.smithi118.stdout: "max": 3758096384, 2024-03-24T03:17:27.426 INFO:teuthology.orchestra.run.smithi118.stdout: "get_started": 0, 2024-03-24T03:17:27.426 INFO:teuthology.orchestra.run.smithi118.stdout: "get": 0, 2024-03-24T03:17:27.426 INFO:teuthology.orchestra.run.smithi118.stdout: "get_sum": 0, 2024-03-24T03:17:27.426 INFO:teuthology.orchestra.run.smithi118.stdout: "get_or_fail_fail": 0, 2024-03-24T03:17:27.426 INFO:teuthology.orchestra.run.smithi118.stdout: "get_or_fail_success": 0, 2024-03-24T03:17:27.426 INFO:teuthology.orchestra.run.smithi118.stdout: "take": 0, 2024-03-24T03:17:27.426 INFO:teuthology.orchestra.run.smithi118.stdout: "take_sum": 0, 2024-03-24T03:17:27.427 INFO:teuthology.orchestra.run.smithi118.stdout: "put": 0, 2024-03-24T03:17:27.427 INFO:teuthology.orchestra.run.smithi118.stdout: "put_sum": 0, 2024-03-24T03:17:27.427 INFO:teuthology.orchestra.run.smithi118.stdout: "wait": { 2024-03-24T03:17:27.427 INFO:teuthology.orchestra.run.smithi118.stdout: "avgcount": 0, 2024-03-24T03:17:27.427 INFO:teuthology.orchestra.run.smithi118.stdout: "sum": 0.000000000, 2024-03-24T03:17:27.427 INFO:teuthology.orchestra.run.smithi118.stdout: "avgtime": 0.000000000 2024-03-24T03:17:27.427 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-03-24T03:17:27.427 INFO:teuthology.orchestra.run.smithi118.stdout: }, 2024-03-24T03:17:27.427 INFO:teuthology.orchestra.run.smithi118.stdout: "throttle-write_buf_throttle-0x5570ec830780": { 2024-03-24T03:17:27.427 INFO:teuthology.orchestra.run.smithi118.stdout: "val": 0, 2024-03-24T03:17:27.428 INFO:teuthology.orchestra.run.smithi118.stdout: "max": 3758096384, 2024-03-24T03:17:27.428 INFO:teuthology.orchestra.run.smithi118.stdout: "get_started": 0, 2024-03-24T03:17:27.428 INFO:teuthology.orchestra.run.smithi118.stdout: "get": 0, 2024-03-24T03:17:27.428 INFO:teuthology.orchestra.run.smithi118.stdout: "get_sum": 0, 2024-03-24T03:17:27.428 INFO:teuthology.orchestra.run.smithi118.stdout: "get_or_fail_fail": 0, 2024-03-24T03:17:27.428 INFO:teuthology.orchestra.run.smithi118.stdout: "get_or_fail_success": 0, 2024-03-24T03:17:27.428 INFO:teuthology.orchestra.run.smithi118.stdout: "take": 0, 2024-03-24T03:17:27.428 INFO:teuthology.orchestra.run.smithi118.stdout: "take_sum": 0, 2024-03-24T03:17:27.428 INFO:teuthology.orchestra.run.smithi118.stdout: "put": 0, 2024-03-24T03:17:27.428 INFO:teuthology.orchestra.run.smithi118.stdout: "put_sum": 0, 2024-03-24T03:17:27.429 INFO:teuthology.orchestra.run.smithi118.stdout: "wait": { 2024-03-24T03:17:27.429 INFO:teuthology.orchestra.run.smithi118.stdout: "avgcount": 0, 2024-03-24T03:17:27.429 INFO:teuthology.orchestra.run.smithi118.stdout: "sum": 0.000000000, 2024-03-24T03:17:27.429 INFO:teuthology.orchestra.run.smithi118.stdout: "avgtime": 0.000000000 2024-03-24T03:17:27.429 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-03-24T03:17:27.429 INFO:teuthology.orchestra.run.smithi118.stdout: } 2024-03-24T03:17:27.429 INFO:teuthology.orchestra.run.smithi118.stdout:} 2024-03-24T03:17:27.445 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 03:17:27 smithi118 ceph-mon[31535]: pgmap v6098: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 1.4 KiB/s wr, 5 op/s 2024-03-24T03:17:27.703 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 03:17:27 smithi189 ceph-mon[36187]: pgmap v6098: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 1.4 KiB/s wr, 5 op/s 2024-03-24T03:17:27.710 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 03:17:27 smithi193 ceph-mon[36878]: pgmap v6098: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 1.4 KiB/s wr, 5 op/s 2024-03-24T03:17:28.276 INFO:tasks.check_counter:Daemon mds.a mds.exported=0 2024-03-24T03:17:28.276 INFO:tasks.check_counter:Daemon mds.a mds.imported=0 2024-03-24T03:17:28.276 INFO:tasks.check_counter:Daemon mds.a mds_cache.dir_update=0 2024-03-24T03:17:28.276 INFO:tasks.check_counter:Daemon mds.a mds_cache.dir_update_receipt=0 2024-03-24T03:17:28.276 INFO:tasks.check_counter:Daemon mds.a mds.dir_split=0 2024-03-24T03:17:28.276 DEBUG:tasks.check_counter:Getting stats from d 2024-03-24T03:17:28.277 DEBUG:teuthology.orchestra.run.smithi118:> sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 shell --fsid d79a5420-e971-11ee-95cd-87774f69a715 -- ceph daemon mds.d perf dump 2024-03-24T03:17:28.731 INFO:teuthology.orchestra.run.smithi118.stderr:Inferring config /var/lib/ceph/d79a5420-e971-11ee-95cd-87774f69a715/mon.a/config 2024-03-24T03:17:29.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 03:17:29 smithi118 ceph-mon[31535]: pgmap v6099: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 1.4 KiB/s wr, 6 op/s 2024-03-24T03:17:29.703 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 03:17:29 smithi189 ceph-mon[36187]: pgmap v6099: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 1.4 KiB/s wr, 6 op/s 2024-03-24T03:17:29.710 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 03:17:29 smithi193 ceph-mon[36878]: pgmap v6099: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 4.6 KiB/s rd, 1.4 KiB/s wr, 6 op/s 2024-03-24T03:17:30.884 INFO:teuthology.orchestra.run.smithi118.stderr:admin_socket: exception getting command descriptions: [Errno 111] Connection refused 2024-03-24T03:17:31.591 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 03:17:31 smithi118 ceph-mon[31535]: pgmap v6100: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 1.4 KiB/s wr, 6 op/s 2024-03-24T03:17:31.591 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 03:17:31 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 10346.761483 secs 2024-03-24T03:17:31.643 DEBUG:teuthology.orchestra.run:got remote process result: 22 2024-03-24T03:17:31.643 ERROR:teuthology.run_tasks:Manager failed: check-counter Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/task/__init__.py", line 132, in __exit__ self.end() File "/home/teuthworker/src/git.ceph.com_ceph-c_3be9a5c9bc793e11e2800a8c0c696e8b46742033/qa/tasks/check_counter.py", line 84, in end proc = manager.admin_socket(daemon_type, daemon_id, ["perf", "dump"]) File "/home/teuthworker/src/git.ceph.com_ceph-c_3be9a5c9bc793e11e2800a8c0c696e8b46742033/qa/tasks/ceph_manager.py", line 1875, in admin_socket return shell( File "/home/teuthworker/src/git.ceph.com_ceph-c_3be9a5c9bc793e11e2800a8c0c696e8b46742033/qa/tasks/ceph_manager.py", line 43, in shell return remote.run( File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi118 with status 22: 'sudo /home/ubuntu/cephtest/cephadm --image quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033 shell --fsid d79a5420-e971-11ee-95cd-87774f69a715 -- ceph daemon mds.d perf dump' 2024-03-24T03:17:31.651 DEBUG:teuthology.run_tasks:Unwinding manager kclient 2024-03-24T03:17:31.661 INFO:tasks.kclient:Unmounting kernel clients... 2024-03-24T03:17:31.661 DEBUG:teuthology.orchestra.run.smithi118:> set -ex 2024-03-24T03:17:31.662 DEBUG:teuthology.orchestra.run.smithi118:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-24T03:17:31.703 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 03:17:31 smithi189 ceph-mon[36187]: pgmap v6100: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 1.4 KiB/s wr, 6 op/s 2024-03-24T03:17:31.703 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 03:17:31 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 10346.761483 secs 2024-03-24T03:17:31.710 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 03:17:31 smithi193 ceph-mon[36878]: pgmap v6100: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 4.2 KiB/s rd, 1.4 KiB/s wr, 6 op/s 2024-03-24T03:17:31.710 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 03:17:31 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 10346.761483 secs 2024-03-24T03:17:31.718 DEBUG:teuthology.orchestra.run.smithi118:> set -ex 2024-03-24T03:17:31.719 DEBUG:teuthology.orchestra.run.smithi118:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-24T03:17:31.768 DEBUG:teuthology.orchestra.run.smithi118:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-24T03:17:32.029 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T03:17:32.030+0000 7f49d6a00640 1 -- 172.21.15.118:0/652284198 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f49d0074100 msgr2=0x7f49d0074580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T03:17:32.030 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T03:17:32.030+0000 7f49d6a00640 1 --2- 172.21.15.118:0/652284198 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f49d0074100 0x7f49d0074580 secure :-1 s=READY pgs=477 cs=0 l=1 rev1=1 crypto rx=0x7f49bc009930 tx=0x7f49bc02bc20 comp rx=0 tx=0).stop 2024-03-24T03:17:32.030 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T03:17:32.030+0000 7f49d6a00640 1 -- 172.21.15.118:0/652284198 shutdown_connections 2024-03-24T03:17:32.030 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T03:17:32.030+0000 7f49d6a00640 1 --2- 172.21.15.118:0/652284198 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f49d010dc80 0x7f49d0110070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T03:17:32.030 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T03:17:32.030+0000 7f49d6a00640 1 --2- 172.21.15.118:0/652284198 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f49d0074100 0x7f49d0074580 unknown :-1 s=CLOSED pgs=477 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T03:17:32.030 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T03:17:32.030+0000 7f49d6a00640 1 --2- 172.21.15.118:0/652284198 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f49d00756e0 0x7f49d0073b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T03:17:32.030 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T03:17:32.030+0000 7f49d6a00640 1 -- 172.21.15.118:0/652284198 >> 172.21.15.118:0/652284198 conn(0x7f49d00fcc80 msgr2=0x7f49d00ff0a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T03:17:32.031 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T03:17:32.030+0000 7f49d6a00640 1 -- 172.21.15.118:0/652284198 shutdown_connections 2024-03-24T03:17:32.031 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T03:17:32.030+0000 7f49d6a00640 1 -- 172.21.15.118:0/652284198 wait complete. 2024-03-24T03:17:32.033 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T03:17:32.031+0000 7f49d6a00640 1 Processor -- start 2024-03-24T03:17:32.033 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T03:17:32.031+0000 7f49d6a00640 1 -- start start 2024-03-24T03:17:32.033 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T03:17:32.031+0000 7f49d6a00640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f49d0074100 0x7f49d019dfe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T03:17:32.033 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T03:17:32.031+0000 7f49d6a00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f49d00756e0 0x7f49d019e520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T03:17:32.034 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T03:17:32.031+0000 7f49d6a00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f49d010dc80 0x7f49d01a5590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T03:17:32.034 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T03:17:32.031+0000 7f49d6a00640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f49d0113950 con 0x7f49d0074100 2024-03-24T03:17:32.034 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T03:17:32.031+0000 7f49d6a00640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f49d01137d0 con 0x7f49d00756e0 2024-03-24T03:17:32.034 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T03:17:32.031+0000 7f49d6a00640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f49d0113ad0 con 0x7f49d010dc80 2024-03-24T03:17:32.034 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T03:17:32.032+0000 7f49cf400640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f49d00756e0 0x7f49d019e520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T03:17:32.034 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T03:17:32.032+0000 7f49cf400640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f49d00756e0 0x7f49d019e520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.118:41094/0 (socket says 172.21.15.118:41094) 2024-03-24T03:17:32.034 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T03:17:32.032+0000 7f49cf400640 1 -- 172.21.15.118:0/126458697 learned_addr learned my addr 172.21.15.118:0/126458697 (peer_addr_for_me v2:172.21.15.118:0/0) 2024-03-24T03:17:32.034 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T03:17:32.032+0000 7f49cfe00640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f49d0074100 0x7f49d019dfe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T03:17:32.035 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T03:17:32.032+0000 7f49d4a00640 1 --2- 172.21.15.118:0/126458697 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f49d010dc80 0x7f49d01a5590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T03:17:32.035 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T03:17:32.032+0000 7f49cf400640 1 -- 172.21.15.118:0/126458697 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f49d010dc80 msgr2=0x7f49d01a5590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T03:17:32.035 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T03:17:32.032+0000 7f49cf400640 1 --2- 172.21.15.118:0/126458697 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f49d010dc80 0x7f49d01a5590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T03:17:32.035 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T03:17:32.032+0000 7f49cf400640 1 -- 172.21.15.118:0/126458697 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f49d0074100 msgr2=0x7f49d019dfe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T03:17:32.035 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T03:17:32.032+0000 7f49cf400640 1 --2- 172.21.15.118:0/126458697 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f49d0074100 0x7f49d019dfe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T03:17:32.035 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T03:17:32.032+0000 7f49cf400640 1 -- 172.21.15.118:0/126458697 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f49d01a5c00 con 0x7f49d00756e0 2024-03-24T03:17:32.035 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T03:17:32.032+0000 7f49cf400640 1 --2- 172.21.15.118:0/126458697 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f49d00756e0 0x7f49d019e520 secure :-1 s=READY pgs=478 cs=0 l=1 rev1=1 crypto rx=0x7f49bc000c00 tx=0x7f49bc02c630 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T03:17:32.035 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T03:17:32.032+0000 7f49ccc00640 1 -- 172.21.15.118:0/126458697 <== mon.2 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f49bc043020 con 0x7f49d00756e0 2024-03-24T03:17:32.035 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T03:17:32.033+0000 7f49ccc00640 1 -- 172.21.15.118:0/126458697 <== mon.2 v2:172.21.15.189:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f49bc0054e0 con 0x7f49d00756e0 2024-03-24T03:17:32.036 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T03:17:32.033+0000 7f49d6a00640 1 -- 172.21.15.118:0/126458697 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f49d01a5e90 con 0x7f49d00756e0 2024-03-24T03:17:32.036 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T03:17:32.033+0000 7f49ccc00640 1 -- 172.21.15.118:0/126458697 <== mon.2 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f49bc003c10 con 0x7f49d00756e0 2024-03-24T03:17:32.036 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T03:17:32.033+0000 7f49d6a00640 1 -- 172.21.15.118:0/126458697 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f49d01a63a0 con 0x7f49d00756e0 2024-03-24T03:17:32.036 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T03:17:32.034+0000 7f49d6a00640 1 -- 172.21.15.118:0/126458697 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f49d0103e50 con 0x7f49d00756e0 2024-03-24T03:17:32.037 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T03:17:32.036+0000 7f49ccc00640 1 -- 172.21.15.118:0/126458697 <== mon.2 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 20) v1 ==== 137291+0+0 (secure 0 0 0) 0x7f49bc046020 con 0x7f49d00756e0 2024-03-24T03:17:32.037 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T03:17:32.037+0000 7f49ccc00640 1 --2- 172.21.15.118:0/126458697 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7f49b40a2880 0x7f49b40a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T03:17:32.037 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T03:17:32.037+0000 7f49ccc00640 1 -- 172.21.15.118:0/126458697 <== mon.2 v2:172.21.15.189:3300/0 5 ==== osd_map(124..124 src has 1..124) v4 ==== 11201+0+0 (secure 0 0 0) 0x7f49bc0f1990 con 0x7f49d00756e0 2024-03-24T03:17:32.037 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T03:17:32.037+0000 7f49cfe00640 1 --2- 172.21.15.118:0/126458697 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7f49b40a2880 0x7f49b40a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T03:17:32.038 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T03:17:32.038+0000 7f49cfe00640 1 --2- 172.21.15.118:0/126458697 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7f49b40a2880 0x7f49b40a4d40 secure :-1 s=READY pgs=641 cs=0 l=1 rev1=1 crypto rx=0x7f49c0009bd0 tx=0x7f49c0005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T03:17:32.038 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T03:17:32.039+0000 7f49ccc00640 1 -- 172.21.15.118:0/126458697 <== mon.2 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f49bc04d020 con 0x7f49d00756e0 2024-03-24T03:17:32.239 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T03:17:32.239+0000 7f49d6a00640 1 -- 172.21.15.118:0/126458697 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f49d002d270 con 0x7f49d00756e0 2024-03-24T03:17:32.242 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T03:17:32.243+0000 7f49ccc00640 1 -- 172.21.15.118:0/126458697 <== mon.2 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v124) v1 ==== 74+0+27305 (secure 0 0 0) 0x7f49bc0bdec0 con 0x7f49d00756e0 2024-03-24T03:17:32.243 INFO:teuthology.orchestra.run.smithi118.stdout: 2024-03-24T03:17:32.243 INFO:teuthology.orchestra.run.smithi118.stdout:{"epoch":124,"fsid":"d79a5420-e971-11ee-95cd-87774f69a715","created":"2024-03-24T00:03:58.648524+0000","modified":"2024-03-24T00:24:59.499441+0000","last_up_change":"2024-03-24T00:12:03.809000+0000","last_in_change":"2024-03-24T00:11:44.389344+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-24T00:07:50.190965+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-24T00:12:59.993723+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-24T00:13:01.257347+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"86","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"c9590426-3901-4f68-a5ef-d57aa8fbac92","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6802","nonce":557690061},{"type":"v1","addr":"172.21.15.118:6803","nonce":557690061}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6804","nonce":557690061},{"type":"v1","addr":"172.21.15.118:6805","nonce":557690061}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6808","nonce":557690061},{"type":"v1","addr":"172.21.15.118:6809","nonce":557690061}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6806","nonce":557690061},{"type":"v1","addr":"172.21.15.118:6807","nonce":557690061}]},"public_addr":"172.21.15.118:6803/557690061","cluster_addr":"172.21.15.118:6805/557690061","heartbeat_back_addr":"172.21.15.118:6809/557690061","heartbeat_front_addr":"172.21.15.118:6807/557690061","state":["exists","up"]},{"osd":1,"uuid":"fc7f827e-ecf6-4661-aad0-7a22951892e0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6800","nonce":4203029485},{"type":"v1","addr":"172.21.15.189:6801","nonce":4203029485}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6802","nonce":4203029485},{"type":"v1","addr":"172.21.15.189:6803","nonce":4203029485}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6806","nonce":4203029485},{"type":"v1","addr":"172.21.15.189:6807","nonce":4203029485}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6804","nonce":4203029485},{"type":"v1","addr":"172.21.15.189:6805","nonce":4203029485}]},"public_addr":"172.21.15.189:6801/4203029485","cluster_addr":"172.21.15.189:6803/4203029485","heartbeat_back_addr":"172.21.15.189:6807/4203029485","heartbeat_front_addr":"172.21.15.189:6805/4203029485","state":["exists","up"]},{"osd":2,"uuid":"e73ccfe9-0e5b-4734-a154-a2f0d829fcfb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6800","nonce":3699076971},{"type":"v1","addr":"172.21.15.193:6801","nonce":3699076971}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6802","nonce":3699076971},{"type":"v1","addr":"172.21.15.193:6803","nonce":3699076971}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6806","nonce":3699076971},{"type":"v1","addr":"172.21.15.193:6807","nonce":3699076971}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6804","nonce":3699076971},{"type":"v1","addr":"172.21.15.193:6805","nonce":3699076971}]},"public_addr":"172.21.15.193:6801/3699076971","cluster_addr":"172.21.15.193:6803/3699076971","heartbeat_back_addr":"172.21.15.193:6807/3699076971","heartbeat_front_addr":"172.21.15.193:6805/3699076971","state":["exists","up"]},{"osd":3,"uuid":"63fed43b-43ce-4031-8957-7fe94725c1ed","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6810","nonce":4059617010},{"type":"v1","addr":"172.21.15.118:6811","nonce":4059617010}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6812","nonce":4059617010},{"type":"v1","addr":"172.21.15.118:6813","nonce":4059617010}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6816","nonce":4059617010},{"type":"v1","addr":"172.21.15.118:6817","nonce":4059617010}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6814","nonce":4059617010},{"type":"v1","addr":"172.21.15.118:6815","nonce":4059617010}]},"public_addr":"172.21.15.118:6811/4059617010","cluster_addr":"172.21.15.118:6813/4059617010","heartbeat_back_addr":"172.21.15.118:6817/4059617010","heartbeat_front_addr":"172.21.15.118:6815/4059617010","state":["exists","up"]},{"osd":4,"uuid":"57377f66-c3b4-43c1-8e8e-bc873672906d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":34,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6808","nonce":4075247068},{"type":"v1","addr":"172.21.15.189:6809","nonce":4075247068}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6810","nonce":4075247068},{"type":"v1","addr":"172.21.15.189:6811","nonce":4075247068}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6814","nonce":4075247068},{"type":"v1","addr":"172.21.15.189:6815","nonce":4075247068}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6812","nonce":4075247068},{"type":"v1","addr":"172.21.15.189:6813","nonce":4075247068}]},"public_addr":"172.21.15.189:6809/4075247068","cluster_addr":"172.21.15.189:6811/4075247068","heartbeat_back_addr":"172.21.15.189:6815/4075247068","heartbeat_front_addr":"172.21.15.189:6813/4075247068","state":["exists","up"]},{"osd":5,"uuid":"91970976-2193-4224-81c0-dc96cb022c85","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":39,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6808","nonce":2920363402},{"type":"v1","addr":"172.21.15.193:6809","nonce":2920363402}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6810","nonce":2920363402},{"type":"v1","addr":"172.21.15.193:6811","nonce":2920363402}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6814","nonce":2920363402},{"type":"v1","addr":"172.21.15.193:6815","nonce":2920363402}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6812","nonce":2920363402},{"type":"v1","addr":"172.21.15.193:6813","nonce":2920363402}]},"public_addr":"172.21.15.193:6809/2920363402","cluster_addr":"172.21.15.193:6811/2920363402","heartbeat_back_addr":"172.21.15.193:6815/2920363402","heartbeat_front_addr":"172.21.15.193:6813/2920363402","state":["exists","up"]},{"osd":6,"uuid":"0b77b196-8674-41c6-ad3d-726b5d01c8fd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":47,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6818","nonce":431354576},{"type":"v1","addr":"172.21.15.118:6819","nonce":431354576}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6820","nonce":431354576},{"type":"v1","addr":"172.21.15.118:6821","nonce":431354576}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6824","nonce":431354576},{"type":"v1","addr":"172.21.15.118:6825","nonce":431354576}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6822","nonce":431354576},{"type":"v1","addr":"172.21.15.118:6823","nonce":431354576}]},"public_addr":"172.21.15.118:6819/431354576","cluster_addr":"172.21.15.118:6821/431354576","heartbeat_back_addr":"172.21.15.118:6825/431354576","heartbeat_front_addr":"172.21.15.118:6823/431354576","state":["exists","up"]},{"osd":7,"uuid":"1fbbd38f-4e8e-4d4c-8e98-66f6862bc4bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":52,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6816","nonce":3667611284},{"type":"v1","addr":"172.21.15.189:6817","nonce":3667611284}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6818","nonce":3667611284},{"type":"v1","addr":"172.21.15.189:6819","nonce":3667611284}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6822","nonce":3667611284},{"type":"v1","addr":"172.21.15.189:6823","nonce":3667611284}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6820","nonce":3667611284},{"type":"v1","addr":"172.21.15.189:6821","nonce":3667611284}]},"public_addr":"172.21.15.189:6817/3667611284","cluster_addr":"172.21.15.189:6819/3667611284","heartbeat_back_addr":"172.21.15.189:6823/3667611284","heartbeat_front_addr":"172.21.15.189:6821/3667611284","state":["exists","up"]},{"osd":8,"uuid":"6462c2b5-e4b3-4b0b-bcfa-4de2c89e93e0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":60,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6816","nonce":3853800910},{"type":"v1","addr":"172.21.15.193:6817","nonce":3853800910}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6818","nonce":3853800910},{"type":"v1","addr":"172.21.15.193:6819","nonce":3853800910}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6822","nonce":3853800910},{"type":"v1","addr":"172.21.15.193:6823","nonce":3853800910}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6820","nonce":3853800910},{"type":"v1","addr":"172.21.15.193:6821","nonce":3853800910}]},"public_addr":"172.21.15.193:6817/3853800910","cluster_addr":"172.21.15.193:6819/3853800910","heartbeat_back_addr":"172.21.15.193:6823/3853800910","heartbeat_front_addr":"172.21.15.193:6821/3853800910","state":["exists","up"]},{"osd":9,"uuid":"54be277f-3596-422f-9de5-9005a8571c6f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":68,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6826","nonce":3248293936},{"type":"v1","addr":"172.21.15.118:6827","nonce":3248293936}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6828","nonce":3248293936},{"type":"v1","addr":"172.21.15.118:6829","nonce":3248293936}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6832","nonce":3248293936},{"type":"v1","addr":"172.21.15.118:6833","nonce":3248293936}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6830","nonce":3248293936},{"type":"v1","addr":"172.21.15.118:6831","nonce":3248293936}]},"public_addr":"172.21.15.118:6827/3248293936","cluster_addr":"172.21.15.118:6829/3248293936","heartbeat_back_addr":"172.21.15.118:6833/3248293936","heartbeat_front_addr":"172.21.15.118:6831/3248293936","state":["exists","up"]},{"osd":10,"uuid":"7ba69d69-ac41-44ce-bc39-c049a8ca5900","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":73,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6824","nonce":2112195760},{"type":"v1","addr":"172.21.15.189:6825","nonce":2112195760}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6826","nonce":2112195760},{"type":"v1","addr":"172.21.15.189:6827","nonce":2112195760}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6830","nonce":2112195760},{"type":"v1","addr":"172.21.15.189:6831","nonce":2112195760}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6828","nonce":2112195760},{"type":"v1","addr":"172.21.15.189:6829","nonce":2112195760}]},"public_addr":"172.21.15.189:6825/2112195760","cluster_addr":"172.21.15.189:6827/2112195760","heartbeat_back_addr":"172.21.15.189:6831/2112195760","heartbeat_front_addr":"172.21.15.189:6829/2112195760","state":["exists","up"]},{"osd":11,"uuid":"186f0b18-6b22-4eae-83b7-b21bbcce0817","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":81,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6824","nonce":3880515510},{"type":"v1","addr":"172.21.15.193:6825","nonce":3880515510}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6826","nonce":3880515510},{"type":"v1","addr":"172.21.15.193:6827","nonce":3880515510}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6830","nonce":3880515510},{"type":"v1","addr":"172.21.15.193:6831","nonce":3880515510}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6828","nonce":3880515510},{"type":"v1","addr":"172.21.15.193:6829","nonce":3880515510}]},"public_addr":"172.21.15.193:6825/3880515510","cluster_addr":"172.21.15.193:6827/3880515510","heartbeat_back_addr":"172.21.15.193:6831/3880515510","heartbeat_front_addr":"172.21.15.193:6829/3880515510","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:06:58.105543+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:07:23.795866+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:07:47.426050+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:08:19.427073+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:08:45.816082+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:09:11.081523+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:09:44.485077+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:10:11.338972+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:10:36.234024+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:11:09.730432+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:11:37.059083+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:12:02.371947+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.a","mappings":[{"from":2,"to":11}]},{"pgid":"2.14","mappings":[{"from":2,"to":4}]},{"pgid":"2.1d","mappings":[{"from":8,"to":11}]},{"pgid":"2.1f","mappings":[{"from":8,"to":1}]},{"pgid":"3.11","mappings":[{"from":4,"to":8},{"from":5,"to":9}]},{"pgid":"3.13","mappings":[{"from":4,"to":9}]},{"pgid":"3.14","mappings":[{"from":4,"to":6}]},{"pgid":"3.31","mappings":[{"from":4,"to":7}]},{"pgid":"3.38","mappings":[{"from":3,"to":9}]},{"pgid":"3.39","mappings":[{"from":5,"to":8}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.118:6837/179741564":"2024-03-25T00:24:41.735441+0000","172.21.15.118:6837/2587227307":"2024-03-25T00:24:58.665829+0000","172.21.15.118:6836/179741564":"2024-03-25T00:24:41.735441+0000","172.21.15.118:6835/1095918599":"2024-03-25T00:24:41.638747+0000","172.21.15.118:6834/1095918599":"2024-03-25T00:24:41.638747+0000","172.21.15.118:6835/3766636479":"2024-03-25T00:24:24.394949+0000","172.21.15.118:6834/3766636479":"2024-03-25T00:24:24.394949+0000","172.21.15.118:6836/1531167856":"2024-03-25T00:24:23.981049+0000","172.21.15.189:6836/2642988219":"2024-03-25T00:24:06.626820+0000","172.21.15.118:6835/2587469815":"2024-03-25T00:24:00.509562+0000","172.21.15.118:6834/2587469815":"2024-03-25T00:24:00.509562+0000","172.21.15.118:6839/282013400":"2024-03-25T00:23:44.005279+0000","172.21.15.118:6838/282013400":"2024-03-25T00:23:44.005279+0000","172.21.15.189:6836/635914573":"2024-03-25T00:23:24.718454+0000","172.21.15.118:6838/1710625324":"2024-03-25T00:23:08.950792+0000","172.21.15.193:6833/2424695757":"2024-03-25T00:22:51.575907+0000","172.21.15.118:6839/3600700815":"2024-03-25T00:22:26.228593+0000","172.21.15.118:6838/3600700815":"2024-03-25T00:22:26.228593+0000","172.21.15.118:6836/2587227307":"2024-03-25T00:24:58.665829+0000","172.21.15.189:6837/2350260652":"2024-03-25T00:22:09.715485+0000","172.21.15.118:6838/979323377":"2024-03-25T00:21:53.776281+0000","172.21.15.118:6835/456491655":"2024-03-25T00:24:59.499390+0000","172.21.15.193:6833/1023428412":"2024-03-25T00:20:03.007142+0000","172.21.15.193:6835/1699655293":"2024-03-25T00:20:48.266150+0000","172.21.15.193:6834/2055310541":"2024-03-25T00:19:47.282228+0000","172.21.15.118:6801/389249272":"2024-03-25T00:05:05.458796+0000","172.21.15.193:6834/3744499911":"2024-03-25T00:20:18.239231+0000","172.21.15.118:6836/3985972391":"2024-03-25T00:23:42.934077+0000","172.21.15.193:6832/3097400619":"2024-03-25T00:20:33.239209+0000","172.21.15.118:6837/3985972391":"2024-03-25T00:23:42.934077+0000","172.21.15.118:6835/3194741373":"2024-03-25T00:23:25.503800+0000","172.21.15.193:6833/4276594827":"2024-03-25T00:21:38.394987+0000","172.21.15.118:6836/1374462956":"2024-03-25T00:23:08.677886+0000","172.21.15.118:6800/389249272":"2024-03-25T00:05:05.458796+0000","172.21.15.118:6834/456491655":"2024-03-25T00:24:59.499390+0000","172.21.15.118:0/2101854694":"2024-03-25T00:05:05.458796+0000","172.21.15.193:6835/2814329894":"2024-03-25T00:21:53.247861+0000","172.21.15.118:6837/1531167856":"2024-03-25T00:24:23.981049+0000","172.21.15.189:6837/75274228":"2024-03-25T00:22:44.728301+0000","172.21.15.193:6835/2264220953":"2024-03-25T00:22:27.928314+0000","172.21.15.118:0/630779359":"2024-03-25T00:05:05.458796+0000","172.21.15.118:6834/3194741373":"2024-03-25T00:23:25.503800+0000","172.21.15.118:6837/1374462956":"2024-03-25T00:23:08.677886+0000","172.21.15.193:6832/2424695757":"2024-03-25T00:22:51.575907+0000","172.21.15.193:6832/1023428412":"2024-03-25T00:20:03.007142+0000","172.21.15.189:6836/75274228":"2024-03-25T00:22:44.728301+0000","172.21.15.189:6837/596160018":"2024-03-25T00:21:38.304407+0000","172.21.15.118:0/368712314":"2024-03-25T00:04:31.283725+0000","172.21.15.189:6837/635914573":"2024-03-25T00:23:24.718454+0000","172.21.15.193:6835/2055310541":"2024-03-25T00:19:47.282228+0000","172.21.15.118:0/2004775597":"2024-03-25T00:04:31.283725+0000","172.21.15.193:6835/3744499911":"2024-03-25T00:20:18.239231+0000","172.21.15.193:6832/2276762432":"2024-03-25T00:22:09.817146+0000","172.21.15.118:6800/1463147955":"2024-03-25T00:04:31.283725+0000","172.21.15.189:6837/2642988219":"2024-03-25T00:24:06.626820+0000","172.21.15.118:6801/1463147955":"2024-03-25T00:04:31.283725+0000","172.21.15.118:6839/3285788503":"2024-03-25T00:20:49.251252+0000","172.21.15.193:6835/799001823":"2024-03-25T00:21:21.268101+0000","172.21.15.193:6834/1699655293":"2024-03-25T00:20:48.266150+0000","172.21.15.118:6839/1710625324":"2024-03-25T00:23:08.950792+0000","172.21.15.118:0/1003670455":"2024-03-25T00:05:05.458796+0000","172.21.15.118:6838/3285788503":"2024-03-25T00:20:49.251252+0000","172.21.15.193:6834/2264220953":"2024-03-25T00:22:27.928314+0000","172.21.15.193:6832/1233062505":"2024-03-25T00:21:04.998288+0000","172.21.15.193:6833/1233062505":"2024-03-25T00:21:04.998288+0000","172.21.15.189:6836/596160018":"2024-03-25T00:21:38.304407+0000","172.21.15.118:0/2158921029":"2024-03-25T00:04:31.283725+0000","172.21.15.193:6834/2814329894":"2024-03-25T00:21:53.247861+0000","172.21.15.189:6836/2350260652":"2024-03-25T00:22:09.715485+0000","172.21.15.193:6833/3097400619":"2024-03-25T00:20:33.239209+0000","172.21.15.189:6837/3450187137":"2024-03-25T00:21:05.241581+0000","172.21.15.193:6834/799001823":"2024-03-25T00:21:21.268101+0000","172.21.15.118:6839/979323377":"2024-03-25T00:21:53.776281+0000","172.21.15.189:6836/3450187137":"2024-03-25T00:21:05.241581+0000","172.21.15.118:6838/3808689299":"2024-03-25T00:21:21.735792+0000","172.21.15.118:6839/3808689299":"2024-03-25T00:21:21.735792+0000","172.21.15.193:6833/2276762432":"2024-03-25T00:22:09.817146+0000","172.21.15.193:6832/4276594827":"2024-03-25T00:21:38.394987+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-24T03:17:32.245 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T03:17:32.245+0000 7f49d6a00640 1 -- 172.21.15.118:0/126458697 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7f49b40a2880 msgr2=0x7f49b40a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T03:17:32.245 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T03:17:32.245+0000 7f49d6a00640 1 --2- 172.21.15.118:0/126458697 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7f49b40a2880 0x7f49b40a4d40 secure :-1 s=READY pgs=641 cs=0 l=1 rev1=1 crypto rx=0x7f49c0009bd0 tx=0x7f49c0005f90 comp rx=0 tx=0).stop 2024-03-24T03:17:32.245 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T03:17:32.245+0000 7f49d6a00640 1 -- 172.21.15.118:0/126458697 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f49d00756e0 msgr2=0x7f49d019e520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T03:17:32.246 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T03:17:32.245+0000 7f49d6a00640 1 --2- 172.21.15.118:0/126458697 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f49d00756e0 0x7f49d019e520 secure :-1 s=READY pgs=478 cs=0 l=1 rev1=1 crypto rx=0x7f49bc000c00 tx=0x7f49bc02c630 comp rx=0 tx=0).stop 2024-03-24T03:17:32.246 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T03:17:32.246+0000 7f49d6a00640 1 -- 172.21.15.118:0/126458697 shutdown_connections 2024-03-24T03:17:32.246 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T03:17:32.246+0000 7f49d6a00640 1 --2- 172.21.15.118:0/126458697 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7f49b40a2880 0x7f49b40a4d40 unknown :-1 s=CLOSED pgs=641 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T03:17:32.246 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T03:17:32.246+0000 7f49d6a00640 1 --2- 172.21.15.118:0/126458697 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f49d010dc80 0x7f49d01a5590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T03:17:32.246 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T03:17:32.246+0000 7f49d6a00640 1 --2- 172.21.15.118:0/126458697 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f49d00756e0 0x7f49d019e520 unknown :-1 s=CLOSED pgs=478 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T03:17:32.246 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T03:17:32.246+0000 7f49d6a00640 1 --2- 172.21.15.118:0/126458697 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f49d0074100 0x7f49d019dfe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T03:17:32.247 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T03:17:32.246+0000 7f49d6a00640 1 -- 172.21.15.118:0/126458697 >> 172.21.15.118:0/126458697 conn(0x7f49d00fcc80 msgr2=0x7f49d0101020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T03:17:32.247 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T03:17:32.246+0000 7f49d6a00640 1 -- 172.21.15.118:0/126458697 shutdown_connections 2024-03-24T03:17:32.247 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T03:17:32.246+0000 7f49d6a00640 1 -- 172.21.15.118:0/126458697 wait complete. 2024-03-24T03:17:32.264 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-24T00:07:50.190965+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 12, 'score_stable': 12, 'optimal_score': 0.25, 'raw_score_acting': 3, 'raw_score_stable': 3, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 2, 'pool_name': 'cephfs_metadata', 'create_time': '2024-03-24T00:12:59.993723+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '85', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.8799999952316284, 'score_stable': 1.8799999952316284, 'optimal_score': 1, 'raw_score_acting': 1.8799999952316284, 'raw_score_stable': 1.8799999952316284, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}, {'pool': 3, 'pool_name': 'cephfs_data', 'create_time': '2024-03-24T00:13:01.257347+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 64, 'pg_placement_num': 64, 'pg_placement_num_target': 64, 'pg_num_target': 64, 'pg_num_pending': 64, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '86', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 64, 'target_size_ratio': 0.8999999761581421}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}, 'read_balance': {'score_type': 'Fair distribution', 'score_acting': 1.5, 'score_stable': 1.5, 'optimal_score': 1, 'raw_score_acting': 1.5, 'raw_score_stable': 1.5, 'primary_affinity_weighted': 0.9999998807907104, 'average_primary_affinity': 1, 'average_primary_affinity_weighted': 1}}] 2024-03-24T03:17:32.265 DEBUG:teuthology.orchestra.run.smithi118:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-24T03:17:32.478 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T03:17:32.478+0000 7eff44a00640 1 -- 172.21.15.118:0/1672333418 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7eff401038e0 msgr2=0x7eff40103d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T03:17:32.478 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T03:17:32.478+0000 7eff44a00640 1 --2- 172.21.15.118:0/1672333418 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7eff401038e0 0x7eff40103d00 secure :-1 s=READY pgs=479 cs=0 l=1 rev1=1 crypto rx=0x7eff30009900 tx=0x7eff3002bc30 comp rx=0 tx=0).stop 2024-03-24T03:17:32.478 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T03:17:32.478+0000 7eff44a00640 1 -- 172.21.15.118:0/1672333418 shutdown_connections 2024-03-24T03:17:32.479 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T03:17:32.478+0000 7eff44a00640 1 --2- 172.21.15.118:0/1672333418 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7eff40105490 0x7eff4010bd20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T03:17:32.479 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T03:17:32.478+0000 7eff44a00640 1 --2- 172.21.15.118:0/1672333418 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7eff40104ad0 0x7eff40104f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T03:17:32.479 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T03:17:32.478+0000 7eff44a00640 1 --2- 172.21.15.118:0/1672333418 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7eff401038e0 0x7eff40103d00 unknown :-1 s=CLOSED pgs=479 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T03:17:32.479 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T03:17:32.478+0000 7eff44a00640 1 -- 172.21.15.118:0/1672333418 >> 172.21.15.118:0/1672333418 conn(0x7eff400fee50 msgr2=0x7eff40101290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T03:17:32.479 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T03:17:32.478+0000 7eff44a00640 1 -- 172.21.15.118:0/1672333418 shutdown_connections 2024-03-24T03:17:32.479 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T03:17:32.478+0000 7eff44a00640 1 -- 172.21.15.118:0/1672333418 wait complete. 2024-03-24T03:17:32.481 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T03:17:32.479+0000 7eff44a00640 1 Processor -- start 2024-03-24T03:17:32.481 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T03:17:32.479+0000 7eff44a00640 1 -- start start 2024-03-24T03:17:32.482 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T03:17:32.480+0000 7eff44a00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7eff401038e0 0x7eff40199c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T03:17:32.482 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T03:17:32.480+0000 7eff44a00640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7eff40104ad0 0x7eff4019a1b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T03:17:32.482 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T03:17:32.480+0000 7eff44a00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7eff40105490 0x7eff401a11e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T03:17:32.482 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T03:17:32.480+0000 7eff44a00640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7eff4010f580 con 0x7eff40104ad0 2024-03-24T03:17:32.482 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T03:17:32.480+0000 7eff44a00640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7eff4010f400 con 0x7eff40105490 2024-03-24T03:17:32.482 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T03:17:32.480+0000 7eff44a00640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7eff4010f700 con 0x7eff401038e0 2024-03-24T03:17:32.482 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T03:17:32.480+0000 7eff3de00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7eff401038e0 0x7eff40199c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T03:17:32.482 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T03:17:32.480+0000 7eff3de00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7eff401038e0 0x7eff40199c70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.118:34298/0 (socket says 172.21.15.118:34298) 2024-03-24T03:17:32.483 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T03:17:32.480+0000 7eff3de00640 1 -- 172.21.15.118:0/1843819182 learned_addr learned my addr 172.21.15.118:0/1843819182 (peer_addr_for_me v2:172.21.15.118:0/0) 2024-03-24T03:17:32.483 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T03:17:32.480+0000 7eff3e800640 1 --2- 172.21.15.118:0/1843819182 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7eff40105490 0x7eff401a11e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T03:17:32.483 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T03:17:32.480+0000 7eff35400640 1 --2- 172.21.15.118:0/1843819182 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7eff40104ad0 0x7eff4019a1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T03:17:32.483 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T03:17:32.480+0000 7eff3de00640 1 -- 172.21.15.118:0/1843819182 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7eff40105490 msgr2=0x7eff401a11e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T03:17:32.483 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T03:17:32.480+0000 7eff3de00640 1 --2- 172.21.15.118:0/1843819182 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7eff40105490 0x7eff401a11e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T03:17:32.483 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T03:17:32.480+0000 7eff3de00640 1 -- 172.21.15.118:0/1843819182 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7eff40104ad0 msgr2=0x7eff4019a1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T03:17:32.483 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T03:17:32.480+0000 7eff3de00640 1 --2- 172.21.15.118:0/1843819182 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7eff40104ad0 0x7eff4019a1b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T03:17:32.483 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T03:17:32.480+0000 7eff3de00640 1 -- 172.21.15.118:0/1843819182 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7eff401a1880 con 0x7eff401038e0 2024-03-24T03:17:32.483 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T03:17:32.480+0000 7eff35400640 1 --2- 172.21.15.118:0/1843819182 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7eff40104ad0 0x7eff4019a1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T03:17:32.484 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T03:17:32.481+0000 7eff3de00640 1 --2- 172.21.15.118:0/1843819182 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7eff401038e0 0x7eff40199c70 secure :-1 s=READY pgs=478 cs=0 l=1 rev1=1 crypto rx=0x7eff30000c00 tx=0x7eff30005dd0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T03:17:32.484 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T03:17:32.481+0000 7eff37400640 1 -- 172.21.15.118:0/1843819182 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7eff30043020 con 0x7eff401038e0 2024-03-24T03:17:32.484 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T03:17:32.481+0000 7eff37400640 1 -- 172.21.15.118:0/1843819182 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7eff300054e0 con 0x7eff401038e0 2024-03-24T03:17:32.484 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T03:17:32.481+0000 7eff37400640 1 -- 172.21.15.118:0/1843819182 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7eff30003e60 con 0x7eff401038e0 2024-03-24T03:17:32.484 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T03:17:32.481+0000 7eff44a00640 1 -- 172.21.15.118:0/1843819182 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7eff401a1b10 con 0x7eff401038e0 2024-03-24T03:17:32.484 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T03:17:32.481+0000 7eff44a00640 1 -- 172.21.15.118:0/1843819182 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7eff401a2060 con 0x7eff401038e0 2024-03-24T03:17:32.484 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T03:17:32.482+0000 7eff44a00640 1 -- 172.21.15.118:0/1843819182 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7eff40068490 con 0x7eff401038e0 2024-03-24T03:17:32.485 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T03:17:32.484+0000 7eff37400640 1 -- 172.21.15.118:0/1843819182 <== mon.1 v2:172.21.15.193:3300/0 4 ==== mgrmap(e 20) v1 ==== 137291+0+0 (secure 0 0 0) 0x7eff30046020 con 0x7eff401038e0 2024-03-24T03:17:32.485 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T03:17:32.485+0000 7eff37400640 1 --2- 172.21.15.118:0/1843819182 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7eff240a2880 0x7eff240a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T03:17:32.485 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T03:17:32.485+0000 7eff37400640 1 -- 172.21.15.118:0/1843819182 <== mon.1 v2:172.21.15.193:3300/0 5 ==== osd_map(124..124 src has 1..124) v4 ==== 11201+0+0 (secure 0 0 0) 0x7eff300f1850 con 0x7eff401038e0 2024-03-24T03:17:32.486 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T03:17:32.485+0000 7eff35400640 1 --2- 172.21.15.118:0/1843819182 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7eff240a2880 0x7eff240a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T03:17:32.486 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T03:17:32.486+0000 7eff35400640 1 --2- 172.21.15.118:0/1843819182 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7eff240a2880 0x7eff240a4d40 secure :-1 s=READY pgs=642 cs=0 l=1 rev1=1 crypto rx=0x7eff28005d00 tx=0x7eff28005c70 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T03:17:32.488 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T03:17:32.489+0000 7eff37400640 1 -- 172.21.15.118:0/1843819182 <== mon.1 v2:172.21.15.193:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7eff3004d020 con 0x7eff401038e0 2024-03-24T03:17:32.685 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T03:17:32.685+0000 7eff44a00640 1 -- 172.21.15.118:0/1843819182 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7eff400622a0 con 0x7eff401038e0 2024-03-24T03:17:32.685 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T03:17:32.686+0000 7eff37400640 1 -- 172.21.15.118:0/1843819182 <== mon.1 v2:172.21.15.193:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v124) v1 ==== 93+0+10 (secure 0 0 0) 0x7eff300bdcf0 con 0x7eff401038e0 2024-03-24T03:17:32.686 INFO:teuthology.orchestra.run.smithi118.stdout:pg_num: 1 2024-03-24T03:17:32.688 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T03:17:32.688+0000 7eff44a00640 1 -- 172.21.15.118:0/1843819182 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7eff240a2880 msgr2=0x7eff240a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T03:17:32.688 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T03:17:32.688+0000 7eff44a00640 1 --2- 172.21.15.118:0/1843819182 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7eff240a2880 0x7eff240a4d40 secure :-1 s=READY pgs=642 cs=0 l=1 rev1=1 crypto rx=0x7eff28005d00 tx=0x7eff28005c70 comp rx=0 tx=0).stop 2024-03-24T03:17:32.688 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T03:17:32.688+0000 7eff44a00640 1 -- 172.21.15.118:0/1843819182 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7eff401038e0 msgr2=0x7eff40199c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T03:17:32.689 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T03:17:32.688+0000 7eff44a00640 1 --2- 172.21.15.118:0/1843819182 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7eff401038e0 0x7eff40199c70 secure :-1 s=READY pgs=478 cs=0 l=1 rev1=1 crypto rx=0x7eff30000c00 tx=0x7eff30005dd0 comp rx=0 tx=0).stop 2024-03-24T03:17:32.689 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T03:17:32.689+0000 7eff44a00640 1 -- 172.21.15.118:0/1843819182 shutdown_connections 2024-03-24T03:17:32.689 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T03:17:32.689+0000 7eff44a00640 1 --2- 172.21.15.118:0/1843819182 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7eff240a2880 0x7eff240a4d40 unknown :-1 s=CLOSED pgs=642 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T03:17:32.689 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T03:17:32.689+0000 7eff44a00640 1 --2- 172.21.15.118:0/1843819182 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7eff40105490 0x7eff401a11e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T03:17:32.689 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T03:17:32.689+0000 7eff44a00640 1 --2- 172.21.15.118:0/1843819182 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7eff40104ad0 0x7eff4019a1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T03:17:32.689 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T03:17:32.689+0000 7eff44a00640 1 --2- 172.21.15.118:0/1843819182 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7eff401038e0 0x7eff40199c70 unknown :-1 s=CLOSED pgs=478 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T03:17:32.689 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T03:17:32.689+0000 7eff44a00640 1 -- 172.21.15.118:0/1843819182 >> 172.21.15.118:0/1843819182 conn(0x7eff400fee50 msgr2=0x7eff40107e70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T03:17:32.690 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T03:17:32.689+0000 7eff44a00640 1 -- 172.21.15.118:0/1843819182 shutdown_connections 2024-03-24T03:17:32.690 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T03:17:32.689+0000 7eff44a00640 1 -- 172.21.15.118:0/1843819182 wait complete. 2024-03-24T03:17:32.695 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 03:17:32 smithi118 ceph-mon[31535]: from='client.? 172.21.15.118:0/126458697' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-24T03:17:32.703 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 03:17:32 smithi189 ceph-mon[36187]: from='client.? 172.21.15.118:0/126458697' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-24T03:17:32.707 DEBUG:teuthology.orchestra.run.smithi118:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2024-03-24T03:17:32.710 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 03:17:32 smithi193 ceph-mon[36878]: from='client.? 172.21.15.118:0/126458697' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-24T03:17:32.927 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T03:17:32.927+0000 7f9815200640 1 -- 172.21.15.118:0/1438985567 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9810105490 msgr2=0x7f981010bd20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T03:17:32.928 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T03:17:32.928+0000 7f9815200640 1 --2- 172.21.15.118:0/1438985567 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9810105490 0x7f981010bd20 secure :-1 s=READY pgs=480 cs=0 l=1 rev1=1 crypto rx=0x7f9808009ca0 tx=0x7f980802c100 comp rx=0 tx=0).stop 2024-03-24T03:17:32.928 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T03:17:32.928+0000 7f9815200640 1 -- 172.21.15.118:0/1438985567 shutdown_connections 2024-03-24T03:17:32.928 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T03:17:32.928+0000 7f9815200640 1 --2- 172.21.15.118:0/1438985567 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9810105490 0x7f981010bd20 unknown :-1 s=CLOSED pgs=480 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T03:17:32.928 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T03:17:32.928+0000 7f9815200640 1 --2- 172.21.15.118:0/1438985567 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f9810104ad0 0x7f9810104f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T03:17:32.928 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T03:17:32.928+0000 7f9815200640 1 --2- 172.21.15.118:0/1438985567 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f98101038e0 0x7f9810103d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T03:17:32.928 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T03:17:32.928+0000 7f9815200640 1 -- 172.21.15.118:0/1438985567 >> 172.21.15.118:0/1438985567 conn(0x7f98100fee50 msgr2=0x7f9810101290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T03:17:32.928 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T03:17:32.928+0000 7f9815200640 1 -- 172.21.15.118:0/1438985567 shutdown_connections 2024-03-24T03:17:32.928 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T03:17:32.928+0000 7f9815200640 1 -- 172.21.15.118:0/1438985567 wait complete. 2024-03-24T03:17:32.931 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T03:17:32.929+0000 7f9815200640 1 Processor -- start 2024-03-24T03:17:32.931 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T03:17:32.929+0000 7f9815200640 1 -- start start 2024-03-24T03:17:32.931 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T03:17:32.929+0000 7f9815200640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f98101038e0 0x7f9810199b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T03:17:32.931 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T03:17:32.930+0000 7f9815200640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9810104ad0 0x7f981019a0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T03:17:32.931 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T03:17:32.930+0000 7f9815200640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f9810105490 0x7f98101a1140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T03:17:32.931 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T03:17:32.930+0000 7f9815200640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f981010f540 con 0x7f98101038e0 2024-03-24T03:17:32.931 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T03:17:32.930+0000 7f9815200640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f981010f3c0 con 0x7f9810104ad0 2024-03-24T03:17:32.931 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T03:17:32.930+0000 7f9815200640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f981010f6c0 con 0x7f9810105490 2024-03-24T03:17:32.932 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T03:17:32.930+0000 7f980de00640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f98101038e0 0x7f9810199b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T03:17:32.932 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T03:17:32.930+0000 7f980d400640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9810104ad0 0x7f981019a0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T03:17:32.932 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T03:17:32.930+0000 7f980d400640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9810104ad0 0x7f981019a0d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.118:43994/0 (socket says 172.21.15.118:43994) 2024-03-24T03:17:32.932 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T03:17:32.930+0000 7f980e800640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f9810105490 0x7f98101a1140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T03:17:32.932 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T03:17:32.930+0000 7f980de00640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f98101038e0 0x7f9810199b90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.118:3300/0 says I am v2:172.21.15.118:57128/0 (socket says 172.21.15.118:57128) 2024-03-24T03:17:32.932 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T03:17:32.930+0000 7f980d400640 1 -- 172.21.15.118:0/1498514777 learned_addr learned my addr 172.21.15.118:0/1498514777 (peer_addr_for_me v2:172.21.15.118:0/0) 2024-03-24T03:17:32.932 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T03:17:32.930+0000 7f980d400640 1 -- 172.21.15.118:0/1498514777 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f9810105490 msgr2=0x7f98101a1140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T03:17:32.932 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T03:17:32.930+0000 7f980d400640 1 --2- 172.21.15.118:0/1498514777 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f9810105490 0x7f98101a1140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T03:17:32.933 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T03:17:32.930+0000 7f980d400640 1 -- 172.21.15.118:0/1498514777 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f98101038e0 msgr2=0x7f9810199b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T03:17:32.933 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T03:17:32.930+0000 7f980d400640 1 --2- 172.21.15.118:0/1498514777 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f98101038e0 0x7f9810199b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T03:17:32.933 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T03:17:32.930+0000 7f980d400640 1 -- 172.21.15.118:0/1498514777 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f98101a17b0 con 0x7f9810104ad0 2024-03-24T03:17:32.933 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T03:17:32.930+0000 7f980e800640 1 --2- 172.21.15.118:0/1498514777 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f9810105490 0x7f98101a1140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T03:17:32.933 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T03:17:32.931+0000 7f980d400640 1 --2- 172.21.15.118:0/1498514777 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9810104ad0 0x7f981019a0d0 secure :-1 s=READY pgs=481 cs=0 l=1 rev1=1 crypto rx=0x7f97f800e9f0 tx=0x7f97f800eeb0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T03:17:32.933 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T03:17:32.931+0000 7f9806a00640 1 -- 172.21.15.118:0/1498514777 <== mon.2 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f97f800cd80 con 0x7f9810104ad0 2024-03-24T03:17:32.933 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T03:17:32.931+0000 7f9806a00640 1 -- 172.21.15.118:0/1498514777 <== mon.2 v2:172.21.15.189:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f97f8010910 con 0x7f9810104ad0 2024-03-24T03:17:32.933 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T03:17:32.931+0000 7f9815200640 1 -- 172.21.15.118:0/1498514777 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f98101a1aa0 con 0x7f9810104ad0 2024-03-24T03:17:32.933 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T03:17:32.931+0000 7f9806a00640 1 -- 172.21.15.118:0/1498514777 <== mon.2 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f97f8018990 con 0x7f9810104ad0 2024-03-24T03:17:32.934 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T03:17:32.931+0000 7f9815200640 1 -- 172.21.15.118:0/1498514777 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f98101a1ff0 con 0x7f9810104ad0 2024-03-24T03:17:32.935 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T03:17:32.932+0000 7f9815200640 1 -- 172.21.15.118:0/1498514777 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f981010ac00 con 0x7f9810104ad0 2024-03-24T03:17:32.935 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T03:17:32.934+0000 7f9806a00640 1 -- 172.21.15.118:0/1498514777 <== mon.2 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 20) v1 ==== 137291+0+0 (secure 0 0 0) 0x7f97f8010ac0 con 0x7f9810104ad0 2024-03-24T03:17:32.935 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T03:17:32.935+0000 7f9806a00640 1 --2- 172.21.15.118:0/1498514777 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7f97f00a2800 0x7f97f00a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T03:17:32.935 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T03:17:32.935+0000 7f9806a00640 1 -- 172.21.15.118:0/1498514777 <== mon.2 v2:172.21.15.189:3300/0 5 ==== osd_map(124..124 src has 1..124) v4 ==== 11201+0+0 (secure 0 0 0) 0x7f97f80cf9b0 con 0x7f9810104ad0 2024-03-24T03:17:32.935 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T03:17:32.935+0000 7f980de00640 1 --2- 172.21.15.118:0/1498514777 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7f97f00a2800 0x7f97f00a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T03:17:32.935 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T03:17:32.936+0000 7f980de00640 1 --2- 172.21.15.118:0/1498514777 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7f97f00a2800 0x7f97f00a4cc0 secure :-1 s=READY pgs=643 cs=0 l=1 rev1=1 crypto rx=0x7f9810104930 tx=0x7f980000b390 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T03:17:32.942 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T03:17:32.943+0000 7f9806a00640 1 -- 172.21.15.118:0/1498514777 <== mon.2 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f97f809b800 con 0x7f9810104ad0 2024-03-24T03:17:33.143 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T03:17:33.143+0000 7f9815200640 1 -- 172.21.15.118:0/1498514777 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f98100622a0 con 0x7f9810104ad0 2024-03-24T03:17:33.143 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T03:17:33.144+0000 7f9806a00640 1 -- 172.21.15.118:0/1498514777 <== mon.2 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v124) v1 ==== 104+0+11 (secure 0 0 0) 0x7f97f809f8a0 con 0x7f9810104ad0 2024-03-24T03:17:33.144 INFO:teuthology.orchestra.run.smithi118.stdout:pg_num: 32 2024-03-24T03:17:33.146 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T03:17:33.147+0000 7f9815200640 1 -- 172.21.15.118:0/1498514777 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7f97f00a2800 msgr2=0x7f97f00a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T03:17:33.147 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T03:17:33.147+0000 7f9815200640 1 --2- 172.21.15.118:0/1498514777 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7f97f00a2800 0x7f97f00a4cc0 secure :-1 s=READY pgs=643 cs=0 l=1 rev1=1 crypto rx=0x7f9810104930 tx=0x7f980000b390 comp rx=0 tx=0).stop 2024-03-24T03:17:33.147 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T03:17:33.147+0000 7f9815200640 1 -- 172.21.15.118:0/1498514777 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9810104ad0 msgr2=0x7f981019a0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T03:17:33.147 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T03:17:33.147+0000 7f9815200640 1 --2- 172.21.15.118:0/1498514777 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9810104ad0 0x7f981019a0d0 secure :-1 s=READY pgs=481 cs=0 l=1 rev1=1 crypto rx=0x7f97f800e9f0 tx=0x7f97f800eeb0 comp rx=0 tx=0).stop 2024-03-24T03:17:33.147 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T03:17:33.147+0000 7f9815200640 1 -- 172.21.15.118:0/1498514777 shutdown_connections 2024-03-24T03:17:33.147 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T03:17:33.147+0000 7f9815200640 1 --2- 172.21.15.118:0/1498514777 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7f97f00a2800 0x7f97f00a4cc0 unknown :-1 s=CLOSED pgs=643 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T03:17:33.147 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T03:17:33.147+0000 7f9815200640 1 --2- 172.21.15.118:0/1498514777 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f9810105490 0x7f98101a1140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T03:17:33.147 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T03:17:33.147+0000 7f9815200640 1 --2- 172.21.15.118:0/1498514777 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f9810104ad0 0x7f981019a0d0 unknown :-1 s=CLOSED pgs=481 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T03:17:33.147 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T03:17:33.147+0000 7f9815200640 1 --2- 172.21.15.118:0/1498514777 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f98101038e0 0x7f9810199b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T03:17:33.148 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T03:17:33.147+0000 7f9815200640 1 -- 172.21.15.118:0/1498514777 >> 172.21.15.118:0/1498514777 conn(0x7f98100fee50 msgr2=0x7f9810107dd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T03:17:33.148 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T03:17:33.147+0000 7f9815200640 1 -- 172.21.15.118:0/1498514777 shutdown_connections 2024-03-24T03:17:33.148 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T03:17:33.147+0000 7f9815200640 1 -- 172.21.15.118:0/1498514777 wait complete. 2024-03-24T03:17:33.166 DEBUG:teuthology.orchestra.run.smithi118:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2024-03-24T03:17:33.385 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T03:17:33.385+0000 7f5651200640 1 -- 172.21.15.118:0/2816527921 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f564c105490 msgr2=0x7f564c10bd20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T03:17:33.385 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T03:17:33.385+0000 7f5651200640 1 --2- 172.21.15.118:0/2816527921 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f564c105490 0x7f564c10bd20 secure :-1 s=READY pgs=479 cs=0 l=1 rev1=1 crypto rx=0x7f5644009be0 tx=0x7f564402bef0 comp rx=0 tx=0).stop 2024-03-24T03:17:33.385 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T03:17:33.385+0000 7f5651200640 1 -- 172.21.15.118:0/2816527921 shutdown_connections 2024-03-24T03:17:33.385 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T03:17:33.385+0000 7f5651200640 1 --2- 172.21.15.118:0/2816527921 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f564c105490 0x7f564c10bd20 unknown :-1 s=CLOSED pgs=479 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T03:17:33.385 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T03:17:33.385+0000 7f5651200640 1 --2- 172.21.15.118:0/2816527921 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f564c104ad0 0x7f564c104f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T03:17:33.386 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T03:17:33.385+0000 7f5651200640 1 --2- 172.21.15.118:0/2816527921 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f564c1038e0 0x7f564c103d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T03:17:33.386 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T03:17:33.385+0000 7f5651200640 1 -- 172.21.15.118:0/2816527921 >> 172.21.15.118:0/2816527921 conn(0x7f564c0fee50 msgr2=0x7f564c101290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T03:17:33.386 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T03:17:33.385+0000 7f5651200640 1 -- 172.21.15.118:0/2816527921 shutdown_connections 2024-03-24T03:17:33.386 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T03:17:33.385+0000 7f5651200640 1 -- 172.21.15.118:0/2816527921 wait complete. 2024-03-24T03:17:33.388 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T03:17:33.386+0000 7f5651200640 1 Processor -- start 2024-03-24T03:17:33.389 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T03:17:33.386+0000 7f5651200640 1 -- start start 2024-03-24T03:17:33.389 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T03:17:33.386+0000 7f5651200640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f564c1038e0 0x7f564c199b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T03:17:33.389 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T03:17:33.386+0000 7f5651200640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f564c104ad0 0x7f564c19a0b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T03:17:33.389 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T03:17:33.386+0000 7f5651200640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f564c105490 0x7f564c1a1120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T03:17:33.389 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T03:17:33.386+0000 7f5651200640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f564c10f590 con 0x7f564c104ad0 2024-03-24T03:17:33.389 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T03:17:33.386+0000 7f5651200640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f564c10f410 con 0x7f564c1038e0 2024-03-24T03:17:33.389 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T03:17:33.386+0000 7f5651200640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f564c10f710 con 0x7f564c105490 2024-03-24T03:17:33.390 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T03:17:33.387+0000 7f5649e00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f564c1038e0 0x7f564c199b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T03:17:33.390 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T03:17:33.387+0000 7f5641400640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f564c104ad0 0x7f564c19a0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T03:17:33.390 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T03:17:33.387+0000 7f5649e00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f564c1038e0 0x7f564c199b70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.118:44018/0 (socket says 172.21.15.118:44018) 2024-03-24T03:17:33.390 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T03:17:33.387+0000 7f5649e00640 1 -- 172.21.15.118:0/3666401955 learned_addr learned my addr 172.21.15.118:0/3666401955 (peer_addr_for_me v2:172.21.15.118:0/0) 2024-03-24T03:17:33.390 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T03:17:33.387+0000 7f564a800640 1 --2- 172.21.15.118:0/3666401955 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f564c105490 0x7f564c1a1120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T03:17:33.390 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T03:17:33.387+0000 7f5641400640 1 -- 172.21.15.118:0/3666401955 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f564c105490 msgr2=0x7f564c1a1120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T03:17:33.390 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T03:17:33.387+0000 7f5641400640 1 --2- 172.21.15.118:0/3666401955 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f564c105490 0x7f564c1a1120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T03:17:33.391 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T03:17:33.387+0000 7f5641400640 1 -- 172.21.15.118:0/3666401955 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f564c1038e0 msgr2=0x7f564c199b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T03:17:33.391 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T03:17:33.387+0000 7f5641400640 1 --2- 172.21.15.118:0/3666401955 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f564c1038e0 0x7f564c199b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T03:17:33.391 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T03:17:33.387+0000 7f5641400640 1 -- 172.21.15.118:0/3666401955 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f564c1a1790 con 0x7f564c104ad0 2024-03-24T03:17:33.391 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T03:17:33.387+0000 7f564a800640 1 --2- 172.21.15.118:0/3666401955 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f564c105490 0x7f564c1a1120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T03:17:33.391 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T03:17:33.387+0000 7f5649e00640 1 --2- 172.21.15.118:0/3666401955 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f564c1038e0 0x7f564c199b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T03:17:33.391 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T03:17:33.387+0000 7f5641400640 1 --2- 172.21.15.118:0/3666401955 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f564c104ad0 0x7f564c19a0b0 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7f5634005fd0 tx=0x7f563400ee50 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T03:17:33.391 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T03:17:33.388+0000 7f5643400640 1 -- 172.21.15.118:0/3666401955 <== mon.0 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f563400cd00 con 0x7f564c104ad0 2024-03-24T03:17:33.391 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T03:17:33.388+0000 7f5643400640 1 -- 172.21.15.118:0/3666401955 <== mon.0 v2:172.21.15.118:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f56340035f0 con 0x7f564c104ad0 2024-03-24T03:17:33.392 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T03:17:33.388+0000 7f5651200640 1 -- 172.21.15.118:0/3666401955 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f564c1a1a80 con 0x7f564c104ad0 2024-03-24T03:17:33.392 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T03:17:33.388+0000 7f5643400640 1 -- 172.21.15.118:0/3666401955 <== mon.0 v2:172.21.15.118:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f5634010690 con 0x7f564c104ad0 2024-03-24T03:17:33.392 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T03:17:33.388+0000 7f5651200640 1 -- 172.21.15.118:0/3666401955 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f564c1a1fd0 con 0x7f564c104ad0 2024-03-24T03:17:33.392 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T03:17:33.389+0000 7f5651200640 1 -- 172.21.15.118:0/3666401955 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f564c10aba0 con 0x7f564c104ad0 2024-03-24T03:17:33.393 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T03:17:33.391+0000 7f5643400640 1 -- 172.21.15.118:0/3666401955 <== mon.0 v2:172.21.15.118:3300/0 4 ==== mgrmap(e 20) v1 ==== 137291+0+0 (secure 0 0 0) 0x7f56340037a0 con 0x7f564c104ad0 2024-03-24T03:17:33.393 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T03:17:33.392+0000 7f5643400640 1 --2- 172.21.15.118:0/3666401955 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7f56300a2800 0x7f56300a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T03:17:33.393 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T03:17:33.392+0000 7f5643400640 1 -- 172.21.15.118:0/3666401955 <== mon.0 v2:172.21.15.118:3300/0 5 ==== osd_map(124..124 src has 1..124) v4 ==== 11201+0+0 (secure 0 0 0) 0x7f56340cf860 con 0x7f564c104ad0 2024-03-24T03:17:33.393 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T03:17:33.393+0000 7f5649e00640 1 --2- 172.21.15.118:0/3666401955 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7f56300a2800 0x7f56300a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T03:17:33.393 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T03:17:33.393+0000 7f5649e00640 1 --2- 172.21.15.118:0/3666401955 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7f56300a2800 0x7f56300a4cc0 secure :-1 s=READY pgs=644 cs=0 l=1 rev1=1 crypto rx=0x7f563c0058c0 tx=0x7f563c005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T03:17:33.397 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T03:17:33.397+0000 7f5643400640 1 -- 172.21.15.118:0/3666401955 <== mon.0 v2:172.21.15.118:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f563409bd60 con 0x7f564c104ad0 2024-03-24T03:17:33.445 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 03:17:33 smithi118 ceph-mon[31535]: pgmap v6101: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 1.4 KiB/s wr, 5 op/s 2024-03-24T03:17:33.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 03:17:33 smithi118 ceph-mon[31535]: from='client.? 172.21.15.118:0/1843819182' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-24T03:17:33.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 03:17:33 smithi118 ceph-mon[31535]: from='client.? 172.21.15.118:0/1498514777' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-24T03:17:33.598 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T03:17:33.598+0000 7f5651200640 1 -- 172.21.15.118:0/3666401955 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f564c0622a0 con 0x7f564c104ad0 2024-03-24T03:17:33.599 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T03:17:33.599+0000 7f5643400640 1 -- 172.21.15.118:0/3666401955 <== mon.0 v2:172.21.15.118:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v124) v1 ==== 100+0+11 (secure 0 0 0) 0x7f563409b660 con 0x7f564c104ad0 2024-03-24T03:17:33.599 INFO:teuthology.orchestra.run.smithi118.stdout:pg_num: 64 2024-03-24T03:17:33.602 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T03:17:33.602+0000 7f5651200640 1 -- 172.21.15.118:0/3666401955 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7f56300a2800 msgr2=0x7f56300a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T03:17:33.602 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T03:17:33.602+0000 7f5651200640 1 --2- 172.21.15.118:0/3666401955 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7f56300a2800 0x7f56300a4cc0 secure :-1 s=READY pgs=644 cs=0 l=1 rev1=1 crypto rx=0x7f563c0058c0 tx=0x7f563c005f90 comp rx=0 tx=0).stop 2024-03-24T03:17:33.603 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T03:17:33.602+0000 7f5651200640 1 -- 172.21.15.118:0/3666401955 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f564c104ad0 msgr2=0x7f564c19a0b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T03:17:33.603 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T03:17:33.602+0000 7f5651200640 1 --2- 172.21.15.118:0/3666401955 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f564c104ad0 0x7f564c19a0b0 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7f5634005fd0 tx=0x7f563400ee50 comp rx=0 tx=0).stop 2024-03-24T03:17:33.603 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T03:17:33.603+0000 7f5651200640 1 -- 172.21.15.118:0/3666401955 shutdown_connections 2024-03-24T03:17:33.603 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T03:17:33.603+0000 7f5651200640 1 --2- 172.21.15.118:0/3666401955 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7f56300a2800 0x7f56300a4cc0 unknown :-1 s=CLOSED pgs=644 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T03:17:33.603 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T03:17:33.603+0000 7f5651200640 1 --2- 172.21.15.118:0/3666401955 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f564c105490 0x7f564c1a1120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T03:17:33.603 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T03:17:33.603+0000 7f5651200640 1 --2- 172.21.15.118:0/3666401955 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f564c104ad0 0x7f564c19a0b0 unknown :-1 s=CLOSED pgs=237 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T03:17:33.603 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T03:17:33.603+0000 7f5651200640 1 --2- 172.21.15.118:0/3666401955 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f564c1038e0 0x7f564c199b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T03:17:33.603 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T03:17:33.603+0000 7f5651200640 1 -- 172.21.15.118:0/3666401955 >> 172.21.15.118:0/3666401955 conn(0x7f564c0fee50 msgr2=0x7f564c107d70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T03:17:33.604 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T03:17:33.603+0000 7f5651200640 1 -- 172.21.15.118:0/3666401955 shutdown_connections 2024-03-24T03:17:33.604 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T03:17:33.603+0000 7f5651200640 1 -- 172.21.15.118:0/3666401955 wait complete. 2024-03-24T03:17:33.622 DEBUG:teuthology.orchestra.run.smithi118:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blocklist ls 2024-03-24T03:17:33.703 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 03:17:33 smithi189 ceph-mon[36187]: pgmap v6101: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 1.4 KiB/s wr, 5 op/s 2024-03-24T03:17:33.703 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 03:17:33 smithi189 ceph-mon[36187]: from='client.? 172.21.15.118:0/1843819182' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-24T03:17:33.703 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 03:17:33 smithi189 ceph-mon[36187]: from='client.? 172.21.15.118:0/1498514777' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-24T03:17:33.710 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 03:17:33 smithi193 ceph-mon[36878]: pgmap v6101: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 1.4 KiB/s wr, 5 op/s 2024-03-24T03:17:33.710 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 03:17:33 smithi193 ceph-mon[36878]: from='client.? 172.21.15.118:0/1843819182' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-24T03:17:33.710 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 03:17:33 smithi193 ceph-mon[36878]: from='client.? 172.21.15.118:0/1498514777' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]: dispatch 2024-03-24T03:17:33.839 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T03:17:33.838+0000 7f369b200640 1 -- 172.21.15.118:0/924202489 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f369c0772a0 msgr2=0x7f369c077700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T03:17:33.839 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T03:17:33.838+0000 7f369b200640 1 --2- 172.21.15.118:0/924202489 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f369c0772a0 0x7f369c077700 secure :-1 s=READY pgs=480 cs=0 l=1 rev1=1 crypto rx=0x7f3684009900 tx=0x7f368402bc40 comp rx=0 tx=0).stop 2024-03-24T03:17:33.839 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T03:17:33.839+0000 7f369b200640 1 -- 172.21.15.118:0/924202489 shutdown_connections 2024-03-24T03:17:33.839 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T03:17:33.839+0000 7f369b200640 1 --2- 172.21.15.118:0/924202489 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f369c077c40 0x7f369c0784e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T03:17:33.839 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T03:17:33.839+0000 7f369b200640 1 --2- 172.21.15.118:0/924202489 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f369c0772a0 0x7f369c077700 unknown :-1 s=CLOSED pgs=480 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T03:17:33.839 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T03:17:33.839+0000 7f369b200640 1 --2- 172.21.15.118:0/924202489 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f369c076060 0x7f369c076480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T03:17:33.839 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T03:17:33.839+0000 7f369b200640 1 -- 172.21.15.118:0/924202489 >> 172.21.15.118:0/924202489 conn(0x7f369c0fed10 msgr2=0x7f369c101150 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T03:17:33.840 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T03:17:33.839+0000 7f369b200640 1 -- 172.21.15.118:0/924202489 shutdown_connections 2024-03-24T03:17:33.840 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T03:17:33.839+0000 7f369b200640 1 -- 172.21.15.118:0/924202489 wait complete. 2024-03-24T03:17:33.842 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T03:17:33.840+0000 7f369b200640 1 Processor -- start 2024-03-24T03:17:33.842 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T03:17:33.840+0000 7f369b200640 1 -- start start 2024-03-24T03:17:33.843 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T03:17:33.841+0000 7f369b200640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f369c076060 0x7f369c19dec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T03:17:33.843 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T03:17:33.841+0000 7f369b200640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f369c0772a0 0x7f369c19e400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T03:17:33.843 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T03:17:33.841+0000 7f369b200640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f369c077c40 0x7f369c1a5470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T03:17:33.843 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T03:17:33.841+0000 7f369b200640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f369c1137f0 con 0x7f369c0772a0 2024-03-24T03:17:33.843 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T03:17:33.841+0000 7f369b200640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f369c113670 con 0x7f369c076060 2024-03-24T03:17:33.843 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T03:17:33.841+0000 7f369b200640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f369c113970 con 0x7f369c077c40 2024-03-24T03:17:33.843 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T03:17:33.841+0000 7f3699e00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f369c076060 0x7f369c19dec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T03:17:33.844 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T03:17:33.841+0000 7f3699e00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f369c076060 0x7f369c19dec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.118:44040/0 (socket says 172.21.15.118:44040) 2024-03-24T03:17:33.844 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T03:17:33.841+0000 7f3699e00640 1 -- 172.21.15.118:0/1971912757 learned_addr learned my addr 172.21.15.118:0/1971912757 (peer_addr_for_me v2:172.21.15.118:0/0) 2024-03-24T03:17:33.844 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T03:17:33.841+0000 7f369a800640 1 --2- 172.21.15.118:0/1971912757 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f369c077c40 0x7f369c1a5470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T03:17:33.844 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T03:17:33.841+0000 7f3693e00640 1 --2- 172.21.15.118:0/1971912757 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f369c0772a0 0x7f369c19e400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T03:17:33.844 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T03:17:33.841+0000 7f3699e00640 1 -- 172.21.15.118:0/1971912757 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f369c077c40 msgr2=0x7f369c1a5470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T03:17:33.844 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T03:17:33.841+0000 7f3699e00640 1 --2- 172.21.15.118:0/1971912757 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f369c077c40 0x7f369c1a5470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T03:17:33.845 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T03:17:33.841+0000 7f3699e00640 1 -- 172.21.15.118:0/1971912757 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f369c0772a0 msgr2=0x7f369c19e400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T03:17:33.845 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T03:17:33.841+0000 7f3699e00640 1 --2- 172.21.15.118:0/1971912757 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f369c0772a0 0x7f369c19e400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T03:17:33.845 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T03:17:33.841+0000 7f3699e00640 1 -- 172.21.15.118:0/1971912757 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f369c1a5ae0 con 0x7f369c076060 2024-03-24T03:17:33.845 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T03:17:33.842+0000 7f3699e00640 1 --2- 172.21.15.118:0/1971912757 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f369c076060 0x7f369c19dec0 secure :-1 s=READY pgs=482 cs=0 l=1 rev1=1 crypto rx=0x7f368c00e9f0 tx=0x7f368c00edf0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T03:17:33.845 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T03:17:33.842+0000 7f3692a00640 1 -- 172.21.15.118:0/1971912757 <== mon.2 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f368c00ccc0 con 0x7f369c076060 2024-03-24T03:17:33.845 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T03:17:33.842+0000 7f3692a00640 1 -- 172.21.15.118:0/1971912757 <== mon.2 v2:172.21.15.189:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f368c010910 con 0x7f369c076060 2024-03-24T03:17:33.845 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T03:17:33.842+0000 7f369b200640 1 -- 172.21.15.118:0/1971912757 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f369c1a5dd0 con 0x7f369c076060 2024-03-24T03:17:33.846 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T03:17:33.842+0000 7f3692a00640 1 -- 172.21.15.118:0/1971912757 <== mon.2 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f368c0188a0 con 0x7f369c076060 2024-03-24T03:17:33.846 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T03:17:33.842+0000 7f369b200640 1 -- 172.21.15.118:0/1971912757 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f369c1a6230 con 0x7f369c076060 2024-03-24T03:17:33.846 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T03:17:33.843+0000 7f369b200640 1 -- 172.21.15.118:0/1971912757 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f369c110110 con 0x7f369c076060 2024-03-24T03:17:33.847 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T03:17:33.845+0000 7f3692a00640 1 -- 172.21.15.118:0/1971912757 <== mon.2 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 20) v1 ==== 137291+0+0 (secure 0 0 0) 0x7f368c010ac0 con 0x7f369c076060 2024-03-24T03:17:33.847 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T03:17:33.846+0000 7f3692a00640 1 --2- 172.21.15.118:0/1971912757 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7f36800a2800 0x7f36800a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T03:17:33.847 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T03:17:33.846+0000 7f3692a00640 1 -- 172.21.15.118:0/1971912757 <== mon.2 v2:172.21.15.189:3300/0 5 ==== osd_map(124..124 src has 1..124) v4 ==== 11201+0+0 (secure 0 0 0) 0x7f368c0cfb50 con 0x7f369c076060 2024-03-24T03:17:33.847 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T03:17:33.847+0000 7f3693e00640 1 --2- 172.21.15.118:0/1971912757 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7f36800a2800 0x7f36800a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T03:17:33.847 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T03:17:33.847+0000 7f3693e00640 1 --2- 172.21.15.118:0/1971912757 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7f36800a2800 0x7f36800a4cc0 secure :-1 s=READY pgs=645 cs=0 l=1 rev1=1 crypto rx=0x7f3684006010 tx=0x7f3684000bc0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T03:17:33.853 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T03:17:33.854+0000 7f3692a00640 1 -- 172.21.15.118:0/1971912757 <== mon.2 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f368c09bff0 con 0x7f369c076060 2024-03-24T03:17:34.073 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T03:17:34.073+0000 7f369b200640 1 -- 172.21.15.118:0/1971912757 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "osd blocklist ls"} v 0) v1 -- 0x7f369c062340 con 0x7f369c076060 2024-03-24T03:17:34.074 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T03:17:34.074+0000 7f3692a00640 1 -- 172.21.15.118:0/1971912757 <== mon.2 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{"prefix": "osd blocklist ls"}]=0 listed 78 entries v124) v1 ==== 81+0+4796 (secure 0 0 0) 0x7f368c09b8f0 con 0x7f369c076060 2024-03-24T03:17:34.074 INFO:teuthology.orchestra.run.smithi118.stdout:172.21.15.118:6837/179741564 2024-03-25T00:24:41.735441+0000 2024-03-24T03:17:34.074 INFO:teuthology.orchestra.run.smithi118.stdout:172.21.15.118:6837/2587227307 2024-03-25T00:24:58.665829+0000 2024-03-24T03:17:34.074 INFO:teuthology.orchestra.run.smithi118.stdout:172.21.15.118:6836/179741564 2024-03-25T00:24:41.735441+0000 2024-03-24T03:17:34.075 INFO:teuthology.orchestra.run.smithi118.stdout:172.21.15.118:6835/1095918599 2024-03-25T00:24:41.638747+0000 2024-03-24T03:17:34.075 INFO:teuthology.orchestra.run.smithi118.stdout:172.21.15.118:6834/1095918599 2024-03-25T00:24:41.638747+0000 2024-03-24T03:17:34.075 INFO:teuthology.orchestra.run.smithi118.stdout:172.21.15.118:6835/3766636479 2024-03-25T00:24:24.394949+0000 2024-03-24T03:17:34.075 INFO:teuthology.orchestra.run.smithi118.stdout:172.21.15.118:6834/3766636479 2024-03-25T00:24:24.394949+0000 2024-03-24T03:17:34.075 INFO:teuthology.orchestra.run.smithi118.stdout:172.21.15.118:6836/1531167856 2024-03-25T00:24:23.981049+0000 2024-03-24T03:17:34.075 INFO:teuthology.orchestra.run.smithi118.stdout:172.21.15.189:6836/2642988219 2024-03-25T00:24:06.626820+0000 2024-03-24T03:17:34.075 INFO:teuthology.orchestra.run.smithi118.stdout:172.21.15.118:6835/2587469815 2024-03-25T00:24:00.509562+0000 2024-03-24T03:17:34.075 INFO:teuthology.orchestra.run.smithi118.stdout:172.21.15.118:6834/2587469815 2024-03-25T00:24:00.509562+0000 2024-03-24T03:17:34.076 INFO:teuthology.orchestra.run.smithi118.stdout:172.21.15.118:6839/282013400 2024-03-25T00:23:44.005279+0000 2024-03-24T03:17:34.076 INFO:teuthology.orchestra.run.smithi118.stdout:172.21.15.118:6838/282013400 2024-03-25T00:23:44.005279+0000 2024-03-24T03:17:34.076 INFO:teuthology.orchestra.run.smithi118.stdout:172.21.15.189:6836/635914573 2024-03-25T00:23:24.718454+0000 2024-03-24T03:17:34.076 INFO:teuthology.orchestra.run.smithi118.stdout:172.21.15.118:6838/1710625324 2024-03-25T00:23:08.950792+0000 2024-03-24T03:17:34.076 INFO:teuthology.orchestra.run.smithi118.stdout:172.21.15.193:6833/2424695757 2024-03-25T00:22:51.575907+0000 2024-03-24T03:17:34.076 INFO:teuthology.orchestra.run.smithi118.stdout:172.21.15.118:6839/3600700815 2024-03-25T00:22:26.228593+0000 2024-03-24T03:17:34.076 INFO:teuthology.orchestra.run.smithi118.stdout:172.21.15.118:6838/3600700815 2024-03-25T00:22:26.228593+0000 2024-03-24T03:17:34.076 INFO:teuthology.orchestra.run.smithi118.stdout:172.21.15.118:6836/2587227307 2024-03-25T00:24:58.665829+0000 2024-03-24T03:17:34.076 INFO:teuthology.orchestra.run.smithi118.stdout:172.21.15.189:6837/2350260652 2024-03-25T00:22:09.715485+0000 2024-03-24T03:17:34.077 INFO:teuthology.orchestra.run.smithi118.stdout:172.21.15.118:6838/979323377 2024-03-25T00:21:53.776281+0000 2024-03-24T03:17:34.077 INFO:teuthology.orchestra.run.smithi118.stdout:172.21.15.118:6835/456491655 2024-03-25T00:24:59.499390+0000 2024-03-24T03:17:34.077 INFO:teuthology.orchestra.run.smithi118.stdout:172.21.15.193:6833/1023428412 2024-03-25T00:20:03.007142+0000 2024-03-24T03:17:34.077 INFO:teuthology.orchestra.run.smithi118.stdout:172.21.15.193:6835/1699655293 2024-03-25T00:20:48.266150+0000 2024-03-24T03:17:34.077 INFO:teuthology.orchestra.run.smithi118.stdout:172.21.15.193:6834/2055310541 2024-03-25T00:19:47.282228+0000 2024-03-24T03:17:34.077 INFO:teuthology.orchestra.run.smithi118.stdout:172.21.15.118:6801/389249272 2024-03-25T00:05:05.458796+0000 2024-03-24T03:17:34.077 INFO:teuthology.orchestra.run.smithi118.stdout:172.21.15.193:6834/3744499911 2024-03-25T00:20:18.239231+0000 2024-03-24T03:17:34.078 INFO:teuthology.orchestra.run.smithi118.stdout:172.21.15.118:6836/3985972391 2024-03-25T00:23:42.934077+0000 2024-03-24T03:17:34.078 INFO:teuthology.orchestra.run.smithi118.stdout:172.21.15.193:6832/3097400619 2024-03-25T00:20:33.239209+0000 2024-03-24T03:17:34.078 INFO:teuthology.orchestra.run.smithi118.stdout:172.21.15.118:6837/3985972391 2024-03-25T00:23:42.934077+0000 2024-03-24T03:17:34.078 INFO:teuthology.orchestra.run.smithi118.stdout:172.21.15.118:6835/3194741373 2024-03-25T00:23:25.503800+0000 2024-03-24T03:17:34.078 INFO:teuthology.orchestra.run.smithi118.stdout:172.21.15.193:6833/4276594827 2024-03-25T00:21:38.394987+0000 2024-03-24T03:17:34.078 INFO:teuthology.orchestra.run.smithi118.stdout:172.21.15.118:6836/1374462956 2024-03-25T00:23:08.677886+0000 2024-03-24T03:17:34.078 INFO:teuthology.orchestra.run.smithi118.stdout:172.21.15.118:6800/389249272 2024-03-25T00:05:05.458796+0000 2024-03-24T03:17:34.078 INFO:teuthology.orchestra.run.smithi118.stdout:172.21.15.118:6834/456491655 2024-03-25T00:24:59.499390+0000 2024-03-24T03:17:34.079 INFO:teuthology.orchestra.run.smithi118.stdout:172.21.15.118:0/2101854694 2024-03-25T00:05:05.458796+0000 2024-03-24T03:17:34.079 INFO:teuthology.orchestra.run.smithi118.stdout:172.21.15.193:6835/2814329894 2024-03-25T00:21:53.247861+0000 2024-03-24T03:17:34.079 INFO:teuthology.orchestra.run.smithi118.stdout:172.21.15.118:6837/1531167856 2024-03-25T00:24:23.981049+0000 2024-03-24T03:17:34.079 INFO:teuthology.orchestra.run.smithi118.stdout:172.21.15.189:6837/75274228 2024-03-25T00:22:44.728301+0000 2024-03-24T03:17:34.079 INFO:teuthology.orchestra.run.smithi118.stdout:172.21.15.193:6835/2264220953 2024-03-25T00:22:27.928314+0000 2024-03-24T03:17:34.079 INFO:teuthology.orchestra.run.smithi118.stdout:172.21.15.118:0/630779359 2024-03-25T00:05:05.458796+0000 2024-03-24T03:17:34.079 INFO:teuthology.orchestra.run.smithi118.stdout:172.21.15.118:6834/3194741373 2024-03-25T00:23:25.503800+0000 2024-03-24T03:17:34.079 INFO:teuthology.orchestra.run.smithi118.stdout:172.21.15.118:6837/1374462956 2024-03-25T00:23:08.677886+0000 2024-03-24T03:17:34.080 INFO:teuthology.orchestra.run.smithi118.stdout:172.21.15.193:6832/2424695757 2024-03-25T00:22:51.575907+0000 2024-03-24T03:17:34.080 INFO:teuthology.orchestra.run.smithi118.stdout:172.21.15.193:6832/1023428412 2024-03-25T00:20:03.007142+0000 2024-03-24T03:17:34.080 INFO:teuthology.orchestra.run.smithi118.stdout:172.21.15.189:6836/75274228 2024-03-25T00:22:44.728301+0000 2024-03-24T03:17:34.080 INFO:teuthology.orchestra.run.smithi118.stdout:172.21.15.189:6837/596160018 2024-03-25T00:21:38.304407+0000 2024-03-24T03:17:34.080 INFO:teuthology.orchestra.run.smithi118.stdout:172.21.15.118:0/368712314 2024-03-25T00:04:31.283725+0000 2024-03-24T03:17:34.080 INFO:teuthology.orchestra.run.smithi118.stdout:172.21.15.189:6837/635914573 2024-03-25T00:23:24.718454+0000 2024-03-24T03:17:34.080 INFO:teuthology.orchestra.run.smithi118.stdout:172.21.15.193:6835/2055310541 2024-03-25T00:19:47.282228+0000 2024-03-24T03:17:34.080 INFO:teuthology.orchestra.run.smithi118.stdout:172.21.15.118:0/2004775597 2024-03-25T00:04:31.283725+0000 2024-03-24T03:17:34.080 INFO:teuthology.orchestra.run.smithi118.stdout:172.21.15.193:6835/3744499911 2024-03-25T00:20:18.239231+0000 2024-03-24T03:17:34.081 INFO:teuthology.orchestra.run.smithi118.stdout:172.21.15.193:6832/2276762432 2024-03-25T00:22:09.817146+0000 2024-03-24T03:17:34.081 INFO:teuthology.orchestra.run.smithi118.stdout:172.21.15.118:6800/1463147955 2024-03-25T00:04:31.283725+0000 2024-03-24T03:17:34.081 INFO:teuthology.orchestra.run.smithi118.stdout:172.21.15.189:6837/2642988219 2024-03-25T00:24:06.626820+0000 2024-03-24T03:17:34.081 INFO:teuthology.orchestra.run.smithi118.stdout:172.21.15.118:6801/1463147955 2024-03-25T00:04:31.283725+0000 2024-03-24T03:17:34.081 INFO:teuthology.orchestra.run.smithi118.stdout:172.21.15.118:6839/3285788503 2024-03-25T00:20:49.251252+0000 2024-03-24T03:17:34.081 INFO:teuthology.orchestra.run.smithi118.stdout:172.21.15.193:6835/799001823 2024-03-25T00:21:21.268101+0000 2024-03-24T03:17:34.081 INFO:teuthology.orchestra.run.smithi118.stdout:172.21.15.193:6834/1699655293 2024-03-25T00:20:48.266150+0000 2024-03-24T03:17:34.081 INFO:teuthology.orchestra.run.smithi118.stdout:172.21.15.118:6839/1710625324 2024-03-25T00:23:08.950792+0000 2024-03-24T03:17:34.082 INFO:teuthology.orchestra.run.smithi118.stdout:172.21.15.118:0/1003670455 2024-03-25T00:05:05.458796+0000 2024-03-24T03:17:34.082 INFO:teuthology.orchestra.run.smithi118.stdout:172.21.15.118:6838/3285788503 2024-03-25T00:20:49.251252+0000 2024-03-24T03:17:34.082 INFO:teuthology.orchestra.run.smithi118.stdout:172.21.15.193:6834/2264220953 2024-03-25T00:22:27.928314+0000 2024-03-24T03:17:34.082 INFO:teuthology.orchestra.run.smithi118.stdout:172.21.15.193:6832/1233062505 2024-03-25T00:21:04.998288+0000 2024-03-24T03:17:34.082 INFO:teuthology.orchestra.run.smithi118.stdout:172.21.15.193:6833/1233062505 2024-03-25T00:21:04.998288+0000 2024-03-24T03:17:34.082 INFO:teuthology.orchestra.run.smithi118.stdout:172.21.15.189:6836/596160018 2024-03-25T00:21:38.304407+0000 2024-03-24T03:17:34.082 INFO:teuthology.orchestra.run.smithi118.stdout:172.21.15.118:0/2158921029 2024-03-25T00:04:31.283725+0000 2024-03-24T03:17:34.082 INFO:teuthology.orchestra.run.smithi118.stdout:172.21.15.193:6834/2814329894 2024-03-25T00:21:53.247861+0000 2024-03-24T03:17:34.083 INFO:teuthology.orchestra.run.smithi118.stdout:172.21.15.189:6836/2350260652 2024-03-25T00:22:09.715485+0000 2024-03-24T03:17:34.083 INFO:teuthology.orchestra.run.smithi118.stdout:172.21.15.193:6833/3097400619 2024-03-25T00:20:33.239209+0000 2024-03-24T03:17:34.083 INFO:teuthology.orchestra.run.smithi118.stdout:172.21.15.189:6837/3450187137 2024-03-25T00:21:05.241581+0000 2024-03-24T03:17:34.083 INFO:teuthology.orchestra.run.smithi118.stdout:172.21.15.193:6834/799001823 2024-03-25T00:21:21.268101+0000 2024-03-24T03:17:34.083 INFO:teuthology.orchestra.run.smithi118.stdout:172.21.15.118:6839/979323377 2024-03-25T00:21:53.776281+0000 2024-03-24T03:17:34.083 INFO:teuthology.orchestra.run.smithi118.stdout:172.21.15.189:6836/3450187137 2024-03-25T00:21:05.241581+0000 2024-03-24T03:17:34.083 INFO:teuthology.orchestra.run.smithi118.stdout:172.21.15.118:6838/3808689299 2024-03-25T00:21:21.735792+0000 2024-03-24T03:17:34.083 INFO:teuthology.orchestra.run.smithi118.stdout:172.21.15.118:6839/3808689299 2024-03-25T00:21:21.735792+0000 2024-03-24T03:17:34.083 INFO:teuthology.orchestra.run.smithi118.stdout:172.21.15.193:6833/2276762432 2024-03-25T00:22:09.817146+0000 2024-03-24T03:17:34.084 INFO:teuthology.orchestra.run.smithi118.stdout:172.21.15.193:6832/4276594827 2024-03-25T00:21:38.394987+0000 2024-03-24T03:17:34.084 INFO:teuthology.orchestra.run.smithi118.stderr:listed 78 entries 2024-03-24T03:17:34.084 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T03:17:34.078+0000 7f369b200640 1 -- 172.21.15.118:0/1971912757 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7f36800a2800 msgr2=0x7f36800a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T03:17:34.085 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T03:17:34.078+0000 7f369b200640 1 --2- 172.21.15.118:0/1971912757 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7f36800a2800 0x7f36800a4cc0 secure :-1 s=READY pgs=645 cs=0 l=1 rev1=1 crypto rx=0x7f3684006010 tx=0x7f3684000bc0 comp rx=0 tx=0).stop 2024-03-24T03:17:34.085 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T03:17:34.078+0000 7f369b200640 1 -- 172.21.15.118:0/1971912757 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f369c076060 msgr2=0x7f369c19dec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T03:17:34.085 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T03:17:34.078+0000 7f369b200640 1 --2- 172.21.15.118:0/1971912757 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f369c076060 0x7f369c19dec0 secure :-1 s=READY pgs=482 cs=0 l=1 rev1=1 crypto rx=0x7f368c00e9f0 tx=0x7f368c00edf0 comp rx=0 tx=0).stop 2024-03-24T03:17:34.085 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T03:17:34.078+0000 7f369b200640 1 -- 172.21.15.118:0/1971912757 shutdown_connections 2024-03-24T03:17:34.085 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T03:17:34.078+0000 7f369b200640 1 --2- 172.21.15.118:0/1971912757 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7f36800a2800 0x7f36800a4cc0 unknown :-1 s=CLOSED pgs=645 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T03:17:34.085 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T03:17:34.078+0000 7f369b200640 1 --2- 172.21.15.118:0/1971912757 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f369c077c40 0x7f369c1a5470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T03:17:34.085 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T03:17:34.078+0000 7f369b200640 1 --2- 172.21.15.118:0/1971912757 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f369c0772a0 0x7f369c19e400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T03:17:34.085 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T03:17:34.078+0000 7f369b200640 1 --2- 172.21.15.118:0/1971912757 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f369c076060 0x7f369c19dec0 unknown :-1 s=CLOSED pgs=482 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T03:17:34.086 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T03:17:34.078+0000 7f369b200640 1 -- 172.21.15.118:0/1971912757 >> 172.21.15.118:0/1971912757 conn(0x7f369c0fed10 msgr2=0x7f369c102070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T03:17:34.086 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T03:17:34.078+0000 7f369b200640 1 -- 172.21.15.118:0/1971912757 shutdown_connections 2024-03-24T03:17:34.086 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T03:17:34.078+0000 7f369b200640 1 -- 172.21.15.118:0/1971912757 wait complete. 2024-03-24T03:17:34.097 DEBUG:tasks.cephfs.kernel_mount:Unmounting client client.0... 2024-03-24T03:17:34.097 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T03:17:34.097 DEBUG:teuthology.orchestra.run.smithi118:> sudo umount /home/ubuntu/cephtest/mnt.0 2024-03-24T03:17:34.695 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 03:17:34 smithi118 ceph-mon[31535]: from='client.? 172.21.15.118:0/3666401955' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-24T03:17:34.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 03:17:34 smithi118 ceph-mon[31535]: from='client.? 172.21.15.118:0/1971912757' entity='client.admin' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-03-24T03:17:34.703 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 03:17:34 smithi189 ceph-mon[36187]: from='client.? 172.21.15.118:0/3666401955' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-24T03:17:34.703 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 03:17:34 smithi189 ceph-mon[36187]: from='client.? 172.21.15.118:0/1971912757' entity='client.admin' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-03-24T03:17:34.710 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 03:17:34 smithi193 ceph-mon[36878]: from='client.? 172.21.15.118:0/3666401955' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]: dispatch 2024-03-24T03:17:34.710 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 03:17:34 smithi193 ceph-mon[36878]: from='client.? 172.21.15.118:0/1971912757' entity='client.admin' cmd=[{"prefix": "osd blocklist ls"}]: dispatch 2024-03-24T03:17:35.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 03:17:35 smithi118 ceph-mon[31535]: pgmap v6102: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 1.4 KiB/s wr, 8 op/s 2024-03-24T03:17:35.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 03:17:35 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T03:17:35.703 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 03:17:35 smithi189 ceph-mon[36187]: pgmap v6102: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 1.4 KiB/s wr, 8 op/s 2024-03-24T03:17:35.703 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 03:17:35 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T03:17:35.710 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 03:17:35 smithi193 ceph-mon[36878]: pgmap v6102: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 5.0 KiB/s rd, 1.4 KiB/s wr, 8 op/s 2024-03-24T03:17:35.710 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 03:17:35 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T03:17:36.422 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 03:17:36 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 10351.761544 secs 2024-03-24T03:17:36.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 03:17:36 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T03:17:36.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 03:17:36 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T03:17:36.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 03:17:36 smithi118 ceph-mon[31535]: pgmap v6103: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 5 op/s 2024-03-24T03:17:36.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 03:17:36 smithi118 ceph-mon[31535]: pgmap v6104: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-03-24T03:17:36.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 03:17:36 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T03:17:36.703 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 03:17:36 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 10351.761544 secs 2024-03-24T03:17:36.703 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 03:17:36 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T03:17:36.703 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 03:17:36 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T03:17:36.703 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 03:17:36 smithi189 ceph-mon[36187]: pgmap v6103: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 5 op/s 2024-03-24T03:17:36.704 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 03:17:36 smithi189 ceph-mon[36187]: pgmap v6104: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-03-24T03:17:36.704 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 03:17:36 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T03:17:36.710 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 03:17:36 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 10351.761544 secs 2024-03-24T03:17:36.710 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 03:17:36 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T03:17:36.710 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 03:17:36 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T03:17:36.711 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 03:17:36 smithi193 ceph-mon[36878]: pgmap v6103: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 5 op/s 2024-03-24T03:17:36.711 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 03:17:36 smithi193 ceph-mon[36878]: pgmap v6104: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-03-24T03:17:36.711 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 03:17:36 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T03:17:39.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 03:17:38 smithi118 ceph-mon[31535]: pgmap v6105: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 5 op/s 2024-03-24T03:17:39.203 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 03:17:38 smithi189 ceph-mon[36187]: pgmap v6105: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 5 op/s 2024-03-24T03:17:39.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 03:17:38 smithi193 ceph-mon[36878]: pgmap v6105: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.7 KiB/s rd, 5 op/s 2024-03-24T03:17:41.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 03:17:40 smithi118 ceph-mon[31535]: pgmap v6106: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 6 op/s 2024-03-24T03:17:41.203 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 03:17:40 smithi189 ceph-mon[36187]: pgmap v6106: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 6 op/s 2024-03-24T03:17:41.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 03:17:40 smithi193 ceph-mon[36878]: pgmap v6106: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 6 op/s 2024-03-24T03:17:42.195 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 03:17:41 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 10356.761601 secs 2024-03-24T03:17:42.203 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 03:17:41 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 10356.761601 secs 2024-03-24T03:17:42.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 03:17:41 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 10356.761601 secs 2024-03-24T03:17:43.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 03:17:42 smithi118 ceph-mon[31535]: pgmap v6107: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 6 op/s 2024-03-24T03:17:43.203 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 03:17:42 smithi189 ceph-mon[36187]: pgmap v6107: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 6 op/s 2024-03-24T03:17:43.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 03:17:42 smithi193 ceph-mon[36878]: pgmap v6107: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.1 KiB/s rd, 6 op/s 2024-03-24T03:17:45.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 03:17:44 smithi118 ceph-mon[31535]: pgmap v6108: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:17:45.203 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 03:17:44 smithi189 ceph-mon[36187]: pgmap v6108: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:17:45.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 03:17:44 smithi193 ceph-mon[36878]: pgmap v6108: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:17:47.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 03:17:46 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 10361.761660 secs 2024-03-24T03:17:47.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 03:17:46 smithi118 ceph-mon[31535]: pgmap v6109: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T03:17:47.203 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 03:17:46 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 10361.761660 secs 2024-03-24T03:17:47.203 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 03:17:46 smithi189 ceph-mon[36187]: pgmap v6109: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T03:17:47.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 03:17:46 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 10361.761660 secs 2024-03-24T03:17:47.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 03:17:46 smithi193 ceph-mon[36878]: pgmap v6109: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T03:17:49.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 03:17:48 smithi118 ceph-mon[31535]: pgmap v6110: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 5 op/s 2024-03-24T03:17:49.203 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 03:17:48 smithi189 ceph-mon[36187]: pgmap v6110: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 5 op/s 2024-03-24T03:17:49.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 03:17:48 smithi193 ceph-mon[36878]: pgmap v6110: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.8 KiB/s rd, 5 op/s 2024-03-24T03:17:51.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 03:17:50 smithi118 ceph-mon[31535]: pgmap v6111: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-03-24T03:17:51.203 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 03:17:50 smithi189 ceph-mon[36187]: pgmap v6111: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-03-24T03:17:51.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 03:17:50 smithi193 ceph-mon[36878]: pgmap v6111: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-03-24T03:17:52.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 03:17:51 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 10366.761720 secs 2024-03-24T03:17:52.203 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 03:17:51 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 10366.761720 secs 2024-03-24T03:17:52.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 03:17:51 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 10366.761720 secs 2024-03-24T03:17:53.195 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 03:17:52 smithi118 ceph-mon[31535]: pgmap v6112: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:17:53.205 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 03:17:52 smithi189 ceph-mon[36187]: pgmap v6112: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:17:53.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 03:17:52 smithi193 ceph-mon[36878]: pgmap v6112: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:17:55.195 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 03:17:54 smithi118 ceph-mon[31535]: pgmap v6113: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T03:17:55.203 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 03:17:54 smithi189 ceph-mon[36187]: pgmap v6113: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T03:17:55.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 03:17:54 smithi193 ceph-mon[36878]: pgmap v6113: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T03:17:57.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 03:17:56 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 10371.761807 secs 2024-03-24T03:17:57.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 03:17:56 smithi118 ceph-mon[31535]: pgmap v6114: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T03:17:57.203 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 03:17:56 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 10371.761807 secs 2024-03-24T03:17:57.203 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 03:17:56 smithi189 ceph-mon[36187]: pgmap v6114: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T03:17:57.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 03:17:56 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 10371.761807 secs 2024-03-24T03:17:57.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 03:17:56 smithi193 ceph-mon[36878]: pgmap v6114: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T03:17:59.195 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 03:17:58 smithi118 ceph-mon[31535]: pgmap v6115: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T03:17:59.203 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 03:17:58 smithi189 ceph-mon[36187]: pgmap v6115: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T03:17:59.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 03:17:58 smithi193 ceph-mon[36878]: pgmap v6115: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T03:18:01.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 03:18:00 smithi118 ceph-mon[31535]: pgmap v6116: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-03-24T03:18:01.203 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 03:18:00 smithi189 ceph-mon[36187]: pgmap v6116: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-03-24T03:18:01.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 03:18:00 smithi193 ceph-mon[36878]: pgmap v6116: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-03-24T03:18:02.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 03:18:01 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 10376.761892 secs 2024-03-24T03:18:02.205 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 03:18:01 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 10376.761892 secs 2024-03-24T03:18:02.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 03:18:01 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 10376.761892 secs 2024-03-24T03:18:03.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 03:18:02 smithi118 ceph-mon[31535]: pgmap v6117: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:18:03.203 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 03:18:02 smithi189 ceph-mon[36187]: pgmap v6117: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:18:03.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 03:18:02 smithi193 ceph-mon[36878]: pgmap v6117: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:18:05.195 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 03:18:04 smithi118 ceph-mon[31535]: pgmap v6118: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T03:18:05.203 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 03:18:04 smithi189 ceph-mon[36187]: pgmap v6118: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T03:18:05.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 03:18:04 smithi193 ceph-mon[36878]: pgmap v6118: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T03:18:07.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 03:18:06 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 10381.761980 secs 2024-03-24T03:18:07.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 03:18:06 smithi118 ceph-mon[31535]: pgmap v6119: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T03:18:07.203 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 03:18:06 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 10381.761980 secs 2024-03-24T03:18:07.203 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 03:18:06 smithi189 ceph-mon[36187]: pgmap v6119: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T03:18:07.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 03:18:06 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 10381.761980 secs 2024-03-24T03:18:07.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 03:18:06 smithi193 ceph-mon[36878]: pgmap v6119: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T03:18:09.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 03:18:08 smithi118 ceph-mon[31535]: pgmap v6120: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T03:18:09.203 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 03:18:08 smithi189 ceph-mon[36187]: pgmap v6120: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T03:18:09.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 03:18:08 smithi193 ceph-mon[36878]: pgmap v6120: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T03:18:11.195 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 03:18:10 smithi118 ceph-mon[31535]: pgmap v6121: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-03-24T03:18:11.203 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 03:18:10 smithi189 ceph-mon[36187]: pgmap v6121: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-03-24T03:18:11.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 03:18:10 smithi193 ceph-mon[36878]: pgmap v6121: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-03-24T03:18:12.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 03:18:11 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 10386.762020 secs 2024-03-24T03:18:12.203 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 03:18:11 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 10386.762020 secs 2024-03-24T03:18:12.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 03:18:11 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 10386.762020 secs 2024-03-24T03:18:13.195 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 03:18:12 smithi118 ceph-mon[31535]: pgmap v6122: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:18:13.203 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 03:18:12 smithi189 ceph-mon[36187]: pgmap v6122: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:18:13.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 03:18:12 smithi193 ceph-mon[36878]: pgmap v6122: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:18:15.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 03:18:14 smithi118 ceph-mon[31535]: pgmap v6123: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T03:18:15.203 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 03:18:14 smithi189 ceph-mon[36187]: pgmap v6123: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T03:18:15.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 03:18:14 smithi193 ceph-mon[36878]: pgmap v6123: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T03:18:17.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 03:18:16 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 10391.762095 secs 2024-03-24T03:18:17.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 03:18:16 smithi118 ceph-mon[31535]: pgmap v6124: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T03:18:17.203 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 03:18:16 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 10391.762095 secs 2024-03-24T03:18:17.203 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 03:18:16 smithi189 ceph-mon[36187]: pgmap v6124: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T03:18:17.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 03:18:16 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 10391.762095 secs 2024-03-24T03:18:17.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 03:18:16 smithi193 ceph-mon[36878]: pgmap v6124: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T03:18:19.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 03:18:18 smithi118 ceph-mon[31535]: pgmap v6125: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T03:18:19.203 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 03:18:18 smithi189 ceph-mon[36187]: pgmap v6125: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T03:18:19.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 03:18:18 smithi193 ceph-mon[36878]: pgmap v6125: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T03:18:21.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 03:18:20 smithi118 ceph-mon[31535]: pgmap v6126: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-03-24T03:18:21.203 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 03:18:20 smithi189 ceph-mon[36187]: pgmap v6126: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-03-24T03:18:21.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 03:18:20 smithi193 ceph-mon[36878]: pgmap v6126: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-03-24T03:18:22.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 03:18:21 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 10396.762156 secs 2024-03-24T03:18:22.203 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 03:18:21 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 10396.762156 secs 2024-03-24T03:18:22.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 03:18:21 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 10396.762156 secs 2024-03-24T03:18:23.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 03:18:22 smithi118 ceph-mon[31535]: pgmap v6127: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:18:23.203 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 03:18:22 smithi189 ceph-mon[36187]: pgmap v6127: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:18:23.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 03:18:22 smithi193 ceph-mon[36878]: pgmap v6127: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:18:25.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 03:18:24 smithi118 ceph-mon[31535]: pgmap v6128: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T03:18:25.203 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 03:18:24 smithi189 ceph-mon[36187]: pgmap v6128: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T03:18:25.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 03:18:24 smithi193 ceph-mon[36878]: pgmap v6128: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T03:18:27.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 03:18:26 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 10401.762216 secs 2024-03-24T03:18:27.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 03:18:26 smithi118 ceph-mon[31535]: pgmap v6129: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T03:18:27.203 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 03:18:26 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 10401.762216 secs 2024-03-24T03:18:27.203 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 03:18:26 smithi189 ceph-mon[36187]: pgmap v6129: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T03:18:27.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 03:18:26 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 10401.762216 secs 2024-03-24T03:18:27.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 03:18:26 smithi193 ceph-mon[36878]: pgmap v6129: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T03:18:29.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 03:18:28 smithi118 ceph-mon[31535]: pgmap v6130: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T03:18:29.203 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 03:18:28 smithi189 ceph-mon[36187]: pgmap v6130: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T03:18:29.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 03:18:28 smithi193 ceph-mon[36878]: pgmap v6130: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 5 op/s 2024-03-24T03:18:31.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 03:18:30 smithi118 ceph-mon[31535]: pgmap v6131: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-03-24T03:18:31.203 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 03:18:30 smithi189 ceph-mon[36187]: pgmap v6131: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-03-24T03:18:31.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 03:18:30 smithi193 ceph-mon[36878]: pgmap v6131: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 7 op/s 2024-03-24T03:18:32.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 03:18:31 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 10406.762278 secs 2024-03-24T03:18:32.203 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 03:18:31 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 10406.762278 secs 2024-03-24T03:18:32.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 03:18:31 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 10406.762278 secs 2024-03-24T03:18:33.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 03:18:32 smithi118 ceph-mon[31535]: pgmap v6132: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:18:33.203 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 03:18:32 smithi189 ceph-mon[36187]: pgmap v6132: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:18:33.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 03:18:32 smithi193 ceph-mon[36878]: pgmap v6132: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:18:35.195 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 03:18:34 smithi118 ceph-mon[31535]: pgmap v6133: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T03:18:35.203 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 03:18:34 smithi189 ceph-mon[36187]: pgmap v6133: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T03:18:35.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 03:18:34 smithi193 ceph-mon[36878]: pgmap v6133: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T03:18:37.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 03:18:36 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 10411.762368 secs 2024-03-24T03:18:37.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 03:18:36 smithi118 ceph-mon[31535]: pgmap v6134: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T03:18:37.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 03:18:36 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T03:18:37.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 03:18:36 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-24T03:18:37.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 03:18:36 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-24T03:18:37.197 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 03:18:36 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-24T03:18:37.197 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 03:18:36 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-03-24T03:18:37.203 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 03:18:36 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 10411.762368 secs 2024-03-24T03:18:37.203 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 03:18:36 smithi189 ceph-mon[36187]: pgmap v6134: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T03:18:37.203 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 03:18:36 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T03:18:37.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 03:18:36 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-24T03:18:37.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 03:18:36 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-24T03:18:37.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 03:18:36 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-24T03:18:37.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 03:18:36 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-03-24T03:18:37.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 03:18:36 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 10411.762368 secs 2024-03-24T03:18:37.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 03:18:36 smithi193 ceph-mon[36878]: pgmap v6134: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T03:18:37.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 03:18:36 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T03:18:37.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 03:18:36 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-24T03:18:37.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 03:18:36 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-24T03:18:37.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 03:18:36 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.7", "name": "osd_memory_target"}]: dispatch 2024-03-24T03:18:37.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 03:18:36 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.10", "name": "osd_memory_target"}]: dispatch 2024-03-24T03:18:38.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 03:18:37 smithi118 ceph-mon[31535]: Adjusting osd_memory_target on smithi189 to 174.1M 2024-03-24T03:18:38.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 03:18:37 smithi118 ceph-mon[31535]: Unable to set osd_memory_target on smithi189 to 182635622: error parsing value: Value '182635622' is below minimum 939524096 2024-03-24T03:18:38.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 03:18:37 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-24T03:18:38.197 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 03:18:37 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-24T03:18:38.197 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 03:18:37 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-24T03:18:38.197 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 03:18:37 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-03-24T03:18:38.197 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 03:18:37 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T03:18:38.197 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 03:18:37 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T03:18:38.197 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 03:18:37 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T03:18:38.203 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 03:18:37 smithi189 ceph-mon[36187]: Adjusting osd_memory_target on smithi189 to 174.1M 2024-03-24T03:18:38.203 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 03:18:37 smithi189 ceph-mon[36187]: Unable to set osd_memory_target on smithi189 to 182635622: error parsing value: Value '182635622' is below minimum 939524096 2024-03-24T03:18:38.203 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 03:18:37 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-24T03:18:38.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 03:18:37 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-24T03:18:38.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 03:18:37 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-24T03:18:38.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 03:18:37 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-03-24T03:18:38.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 03:18:37 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T03:18:38.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 03:18:37 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T03:18:38.204 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 03:18:37 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T03:18:38.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 03:18:37 smithi193 ceph-mon[36878]: Adjusting osd_memory_target on smithi189 to 174.1M 2024-03-24T03:18:38.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 03:18:37 smithi193 ceph-mon[36878]: Unable to set osd_memory_target on smithi189 to 182635622: error parsing value: Value '182635622' is below minimum 939524096 2024-03-24T03:18:38.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 03:18:37 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-24T03:18:38.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 03:18:37 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-24T03:18:38.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 03:18:37 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.6", "name": "osd_memory_target"}]: dispatch 2024-03-24T03:18:38.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 03:18:37 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.9", "name": "osd_memory_target"}]: dispatch 2024-03-24T03:18:38.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 03:18:37 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T03:18:38.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 03:18:37 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T03:18:38.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 03:18:37 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T03:18:39.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 03:18:38 smithi118 ceph-mon[31535]: Adjusting osd_memory_target on smithi118 to 176.7M 2024-03-24T03:18:39.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 03:18:38 smithi118 ceph-mon[31535]: Unable to set osd_memory_target on smithi118 to 185365196: error parsing value: Value '185365196' is below minimum 939524096 2024-03-24T03:18:39.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 03:18:38 smithi118 ceph-mon[31535]: pgmap v6135: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 5 op/s 2024-03-24T03:18:39.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 03:18:38 smithi118 ceph-mon[31535]: pgmap v6136: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 5 op/s 2024-03-24T03:18:39.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 03:18:38 smithi193 ceph-mon[36878]: Adjusting osd_memory_target on smithi118 to 176.7M 2024-03-24T03:18:39.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 03:18:38 smithi193 ceph-mon[36878]: Unable to set osd_memory_target on smithi118 to 185365196: error parsing value: Value '185365196' is below minimum 939524096 2024-03-24T03:18:39.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 03:18:38 smithi193 ceph-mon[36878]: pgmap v6135: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 5 op/s 2024-03-24T03:18:39.211 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 03:18:38 smithi193 ceph-mon[36878]: pgmap v6136: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 5 op/s 2024-03-24T03:18:39.453 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 03:18:38 smithi189 ceph-mon[36187]: Adjusting osd_memory_target on smithi118 to 176.7M 2024-03-24T03:18:39.453 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 03:18:38 smithi189 ceph-mon[36187]: Unable to set osd_memory_target on smithi118 to 185365196: error parsing value: Value '185365196' is below minimum 939524096 2024-03-24T03:18:39.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 03:18:38 smithi189 ceph-mon[36187]: pgmap v6135: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 5 op/s 2024-03-24T03:18:39.454 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 03:18:38 smithi189 ceph-mon[36187]: pgmap v6136: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 5 op/s 2024-03-24T03:18:41.210 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 03:18:40 smithi193 ceph-mon[36878]: pgmap v6137: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 862 B/s wr, 7 op/s 2024-03-24T03:18:41.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 03:18:40 smithi118 ceph-mon[31535]: pgmap v6137: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 862 B/s wr, 7 op/s 2024-03-24T03:18:41.453 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 03:18:40 smithi189 ceph-mon[36187]: pgmap v6137: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 862 B/s wr, 7 op/s 2024-03-24T03:18:42.445 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 03:18:41 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 10416.762411 secs 2024-03-24T03:18:42.453 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 03:18:41 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 10416.762411 secs 2024-03-24T03:18:42.460 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 03:18:41 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 10416.762411 secs 2024-03-24T03:18:43.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 03:18:42 smithi118 ceph-mon[31535]: pgmap v6138: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 862 B/s wr, 7 op/s 2024-03-24T03:18:43.453 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 03:18:42 smithi189 ceph-mon[36187]: pgmap v6138: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 862 B/s wr, 7 op/s 2024-03-24T03:18:43.460 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 03:18:42 smithi193 ceph-mon[36878]: pgmap v6138: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 862 B/s wr, 7 op/s 2024-03-24T03:18:45.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 03:18:44 smithi118 ceph-mon[31535]: pgmap v6139: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 862 B/s wr, 8 op/s 2024-03-24T03:18:45.453 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 03:18:44 smithi189 ceph-mon[36187]: pgmap v6139: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 862 B/s wr, 8 op/s 2024-03-24T03:18:45.460 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 03:18:44 smithi193 ceph-mon[36878]: pgmap v6139: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 862 B/s wr, 8 op/s 2024-03-24T03:18:47.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 03:18:46 smithi118 ceph-mon[31535]: pgmap v6140: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 862 B/s wr, 8 op/s 2024-03-24T03:18:47.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 03:18:46 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 10421.762498 secs 2024-03-24T03:18:47.453 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 03:18:46 smithi189 ceph-mon[36187]: pgmap v6140: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 862 B/s wr, 8 op/s 2024-03-24T03:18:47.453 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 03:18:46 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 10421.762498 secs 2024-03-24T03:18:47.460 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 03:18:46 smithi193 ceph-mon[36878]: pgmap v6140: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 862 B/s wr, 8 op/s 2024-03-24T03:18:47.460 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 03:18:46 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 10421.762498 secs 2024-03-24T03:18:49.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 03:18:48 smithi118 ceph-mon[31535]: pgmap v6141: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 818 B/s wr, 7 op/s 2024-03-24T03:18:49.453 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 03:18:48 smithi189 ceph-mon[36187]: pgmap v6141: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 818 B/s wr, 7 op/s 2024-03-24T03:18:49.460 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 03:18:48 smithi193 ceph-mon[36878]: pgmap v6141: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 818 B/s wr, 7 op/s 2024-03-24T03:18:51.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 03:18:50 smithi118 ceph-mon[31535]: pgmap v6142: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 682 B/s wr, 8 op/s 2024-03-24T03:18:51.453 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 03:18:50 smithi189 ceph-mon[36187]: pgmap v6142: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 682 B/s wr, 8 op/s 2024-03-24T03:18:51.460 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 03:18:50 smithi193 ceph-mon[36878]: pgmap v6142: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 4.0 KiB/s rd, 682 B/s wr, 8 op/s 2024-03-24T03:18:52.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 03:18:51 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 10426.762583 secs 2024-03-24T03:18:52.453 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 03:18:51 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 10426.762583 secs 2024-03-24T03:18:52.460 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 03:18:51 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 10426.762583 secs 2024-03-24T03:18:53.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 03:18:52 smithi118 ceph-mon[31535]: pgmap v6143: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 0 B/s wr, 5 op/s 2024-03-24T03:18:53.453 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 03:18:52 smithi189 ceph-mon[36187]: pgmap v6143: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 0 B/s wr, 5 op/s 2024-03-24T03:18:53.460 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 03:18:52 smithi193 ceph-mon[36878]: pgmap v6143: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 0 B/s wr, 5 op/s 2024-03-24T03:18:55.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 03:18:55 smithi118 ceph-mon[31535]: pgmap v6144: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 0 B/s wr, 6 op/s 2024-03-24T03:18:55.453 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 03:18:55 smithi189 ceph-mon[36187]: pgmap v6144: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 0 B/s wr, 6 op/s 2024-03-24T03:18:55.460 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 03:18:55 smithi193 ceph-mon[36878]: pgmap v6144: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.4 KiB/s rd, 0 B/s wr, 6 op/s 2024-03-24T03:18:57.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 03:18:57 smithi118 ceph-mon[31535]: pgmap v6145: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T03:18:57.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 03:18:57 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 10431.762671 secs 2024-03-24T03:18:57.453 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 03:18:57 smithi189 ceph-mon[36187]: pgmap v6145: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T03:18:57.453 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 03:18:57 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 10431.762671 secs 2024-03-24T03:18:57.460 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 03:18:57 smithi193 ceph-mon[36878]: pgmap v6145: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T03:18:57.460 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 03:18:57 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 10431.762671 secs 2024-03-24T03:18:59.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 03:18:59 smithi118 ceph-mon[31535]: pgmap v6146: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:18:59.453 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 03:18:59 smithi189 ceph-mon[36187]: pgmap v6146: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:18:59.460 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 03:18:59 smithi193 ceph-mon[36878]: pgmap v6146: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:19:01.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 03:19:01 smithi118 ceph-mon[31535]: pgmap v6147: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-24T03:19:01.453 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 03:19:01 smithi189 ceph-mon[36187]: pgmap v6147: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-24T03:19:01.460 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 03:19:01 smithi193 ceph-mon[36878]: pgmap v6147: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-24T03:19:02.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 03:19:02 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 10436.762760 secs 2024-03-24T03:19:02.453 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 03:19:02 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 10436.762760 secs 2024-03-24T03:19:02.460 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 03:19:02 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 10436.762760 secs 2024-03-24T03:19:03.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 03:19:03 smithi118 ceph-mon[31535]: pgmap v6148: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:19:03.453 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 03:19:03 smithi189 ceph-mon[36187]: pgmap v6148: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:19:03.460 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 03:19:03 smithi193 ceph-mon[36878]: pgmap v6148: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:19:05.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 03:19:05 smithi118 ceph-mon[31535]: pgmap v6149: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T03:19:05.453 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 03:19:05 smithi189 ceph-mon[36187]: pgmap v6149: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T03:19:05.460 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 03:19:05 smithi193 ceph-mon[36878]: pgmap v6149: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T03:19:07.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 03:19:07 smithi118 ceph-mon[31535]: pgmap v6150: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T03:19:07.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 03:19:07 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 10441.762849 secs 2024-03-24T03:19:07.453 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 03:19:07 smithi189 ceph-mon[36187]: pgmap v6150: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T03:19:07.453 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 03:19:07 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 10441.762849 secs 2024-03-24T03:19:07.460 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 03:19:07 smithi193 ceph-mon[36878]: pgmap v6150: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T03:19:07.460 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 03:19:07 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 10441.762849 secs 2024-03-24T03:19:09.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 03:19:09 smithi118 ceph-mon[31535]: pgmap v6151: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:19:09.453 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 03:19:09 smithi189 ceph-mon[36187]: pgmap v6151: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:19:09.460 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 03:19:09 smithi193 ceph-mon[36878]: pgmap v6151: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:19:11.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 03:19:11 smithi118 ceph-mon[31535]: pgmap v6152: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-24T03:19:11.453 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 03:19:11 smithi189 ceph-mon[36187]: pgmap v6152: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-24T03:19:11.460 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 03:19:11 smithi193 ceph-mon[36878]: pgmap v6152: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-24T03:19:12.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 03:19:12 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 10446.762935 secs 2024-03-24T03:19:12.453 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 03:19:12 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 10446.762935 secs 2024-03-24T03:19:12.460 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 03:19:12 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 10446.762935 secs 2024-03-24T03:19:13.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 03:19:13 smithi118 ceph-mon[31535]: pgmap v6153: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:19:13.453 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 03:19:13 smithi189 ceph-mon[36187]: pgmap v6153: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:19:13.460 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 03:19:13 smithi193 ceph-mon[36878]: pgmap v6153: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:19:15.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 03:19:15 smithi118 ceph-mon[31535]: pgmap v6154: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T03:19:15.453 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 03:19:15 smithi189 ceph-mon[36187]: pgmap v6154: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T03:19:15.460 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 03:19:15 smithi193 ceph-mon[36878]: pgmap v6154: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T03:19:17.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 03:19:17 smithi118 ceph-mon[31535]: pgmap v6155: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T03:19:17.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 03:19:17 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 10451.763023 secs 2024-03-24T03:19:17.453 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 03:19:17 smithi189 ceph-mon[36187]: pgmap v6155: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T03:19:17.453 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 03:19:17 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 10451.763023 secs 2024-03-24T03:19:17.460 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 03:19:17 smithi193 ceph-mon[36878]: pgmap v6155: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T03:19:17.460 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 03:19:17 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 10451.763023 secs 2024-03-24T03:19:19.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 03:19:19 smithi118 ceph-mon[31535]: pgmap v6156: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:19:19.453 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 03:19:19 smithi189 ceph-mon[36187]: pgmap v6156: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:19:19.460 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 03:19:19 smithi193 ceph-mon[36878]: pgmap v6156: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:19:20.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 03:19:20 smithi118 ceph-mon[31535]: pgmap v6157: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-24T03:19:20.453 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 03:19:20 smithi189 ceph-mon[36187]: pgmap v6157: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-24T03:19:20.460 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 03:19:20 smithi193 ceph-mon[36878]: pgmap v6157: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-24T03:19:21.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 03:19:21 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 10456.763098 secs 2024-03-24T03:19:21.703 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 03:19:21 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 10456.763098 secs 2024-03-24T03:19:21.710 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 03:19:21 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 10456.763098 secs 2024-03-24T03:19:22.695 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 03:19:22 smithi118 ceph-mon[31535]: pgmap v6158: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:19:22.703 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 03:19:22 smithi189 ceph-mon[36187]: pgmap v6158: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:19:22.710 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 03:19:22 smithi193 ceph-mon[36878]: pgmap v6158: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:19:24.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 03:19:24 smithi118 ceph-mon[31535]: pgmap v6159: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T03:19:24.703 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 03:19:24 smithi189 ceph-mon[36187]: pgmap v6159: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T03:19:24.710 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 03:19:24 smithi193 ceph-mon[36878]: pgmap v6159: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T03:19:26.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 03:19:26 smithi118 ceph-mon[31535]: pgmap v6160: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T03:19:26.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 03:19:26 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 10461.763183 secs 2024-03-24T03:19:26.703 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 03:19:26 smithi189 ceph-mon[36187]: pgmap v6160: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T03:19:26.703 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 03:19:26 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 10461.763183 secs 2024-03-24T03:19:26.710 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 03:19:26 smithi193 ceph-mon[36878]: pgmap v6160: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T03:19:26.710 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 03:19:26 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 10461.763183 secs 2024-03-24T03:19:28.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 03:19:28 smithi118 ceph-mon[31535]: pgmap v6161: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:19:28.703 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 03:19:28 smithi189 ceph-mon[36187]: pgmap v6161: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:19:28.710 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 03:19:28 smithi193 ceph-mon[36878]: pgmap v6161: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:19:30.445 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 03:19:30 smithi118 ceph-mon[31535]: pgmap v6162: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-24T03:19:30.703 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 03:19:30 smithi189 ceph-mon[36187]: pgmap v6162: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-24T03:19:30.710 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 03:19:30 smithi193 ceph-mon[36878]: pgmap v6162: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.7 KiB/s rd, 7 op/s 2024-03-24T03:19:31.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 03:19:31 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 10466.763246 secs 2024-03-24T03:19:31.703 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 03:19:31 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 10466.763246 secs 2024-03-24T03:19:31.710 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 03:19:31 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 10466.763246 secs 2024-03-24T03:19:32.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 03:19:32 smithi118 ceph-mon[31535]: pgmap v6163: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:19:32.703 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 03:19:32 smithi189 ceph-mon[36187]: pgmap v6163: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:19:32.710 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 03:19:32 smithi193 ceph-mon[36878]: pgmap v6163: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:19:34.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 03:19:34 smithi118 ceph-mon[31535]: pgmap v6164: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T03:19:34.703 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 03:19:34 smithi189 ceph-mon[36187]: pgmap v6164: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T03:19:34.710 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 03:19:34 smithi193 ceph-mon[36878]: pgmap v6164: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.3 KiB/s rd, 6 op/s 2024-03-24T03:19:36.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 03:19:36 smithi118 ceph-mon[31535]: pgmap v6165: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T03:19:36.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 03:19:36 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 10471.763281 secs 2024-03-24T03:19:36.703 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 03:19:36 smithi189 ceph-mon[36187]: pgmap v6165: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T03:19:36.703 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 03:19:36 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 10471.763281 secs 2024-03-24T03:19:36.710 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 03:19:36 smithi193 ceph-mon[36878]: pgmap v6165: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.9 KiB/s rd, 6 op/s 2024-03-24T03:19:36.710 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 03:19:36 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 10471.763281 secs 2024-03-24T03:19:38.509 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 03:19:38 smithi118 ceph-mon[31535]: pgmap v6166: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:19:38.509 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 03:19:38 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T03:19:38.509 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 03:19:38 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-24T03:19:38.509 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 03:19:38 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-24T03:19:38.509 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 03:19:38 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-03-24T03:19:38.509 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 03:19:38 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-03-24T03:19:38.703 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 03:19:38 smithi189 ceph-mon[36187]: pgmap v6166: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:19:38.703 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 03:19:38 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T03:19:38.704 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 03:19:38 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-24T03:19:38.704 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 03:19:38 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-24T03:19:38.704 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 03:19:38 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-03-24T03:19:38.704 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 03:19:38 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-03-24T03:19:38.710 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 03:19:38 smithi193 ceph-mon[36878]: pgmap v6166: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:19:38.710 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 03:19:38 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T03:19:38.711 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 03:19:38 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-24T03:19:38.711 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 03:19:38 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-24T03:19:38.711 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 03:19:38 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.8", "name": "osd_memory_target"}]: dispatch 2024-03-24T03:19:38.711 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 03:19:38 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config rm", "who": "osd.11", "name": "osd_memory_target"}]: dispatch 2024-03-24T03:19:38.777 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi118.front.sepia.ceph.com 2024-03-24T03:19:38.777 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T03:19:38.777 DEBUG:teuthology.orchestra.run.smithi118:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2024-03-24T03:19:38.822 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T03:19:38.822 DEBUG:teuthology.orchestra.run.smithi118:> ip netns list 2024-03-24T03:19:38.915 INFO:teuthology.orchestra.run.smithi118.stdout:ceph-ns--home-ubuntu-cephtest-mnt.0 (id: 0) 2024-03-24T03:19:38.916 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T03:19:38.916 DEBUG:teuthology.orchestra.run.smithi118:> sudo ip netns delete ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-24T03:19:39.004 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T03:19:39.004 DEBUG:teuthology.orchestra.run.smithi118:> sudo ip link delete ceph-brx 2024-03-24T03:19:39.129 INFO:tasks.kclient:Unmounting kernel clients... 2024-03-24T03:19:39.130 DEBUG:teuthology.orchestra.run.smithi118:> set -ex 2024-03-24T03:19:39.130 DEBUG:teuthology.orchestra.run.smithi118:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-24T03:19:39.175 DEBUG:tasks.cephfs.mount:not mounted; /proc/self/mounts is: proc /proc proc rw,nosuid,nodev,noexec,relatime 0 0 sysfs /sys sysfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 devtmpfs /dev devtmpfs rw,seclabel,nosuid,size=4096k,nr_inodes=4053413,mode=755 0 0 securityfs /sys/kernel/security securityfs rw,nosuid,nodev,noexec,relatime 0 0 tmpfs /dev/shm tmpfs rw,seclabel,nosuid,nodev 0 0 devpts /dev/pts devpts rw,seclabel,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000 0 0 tmpfs /run tmpfs rw,seclabel,nosuid,nodev,size=6498340k,nr_inodes=819200,mode=755 0 0 cgroup2 /sys/fs/cgroup cgroup2 rw,seclabel,nosuid,nodev,noexec,relatime,nsdelegate,memory_recursiveprot 0 0 pstore /sys/fs/pstore pstore rw,seclabel,nosuid,nodev,noexec,relatime 0 0 bpf /sys/fs/bpf bpf rw,nosuid,nodev,noexec,relatime,mode=700 0 0 /dev/sda1 / ext4 rw,seclabel,relatime 0 0 selinuxfs /sys/fs/selinux selinuxfs rw,nosuid,noexec,relatime 0 0 systemd-1 /proc/sys/fs/binfmt_misc autofs rw,relatime,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct 0 0 mqueue /dev/mqueue mqueue rw,seclabel,nosuid,nodev,noexec,relatime 0 0 hugetlbfs /dev/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M 0 0 debugfs /sys/kernel/debug debugfs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 tracefs /sys/kernel/tracing tracefs rw,seclabel,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-tmpfiles-setup-dev.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 fusectl /sys/fs/fuse/connections fusectl rw,nosuid,nodev,noexec,relatime 0 0 configfs /sys/kernel/config configfs rw,nosuid,nodev,noexec,relatime 0 0 ramfs /run/credentials/systemd-sysctl.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 ramfs /run/credentials/systemd-tmpfiles-setup.service ramfs ro,seclabel,nosuid,nodev,noexec,relatime,mode=700 0 0 sunrpc /var/lib/nfs/rpc_pipefs rpc_pipefs rw,relatime 0 0 tmpfs /run/user/1000 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249168k,nr_inodes=812292,mode=700,uid=1000,gid=1301 0 0 /dev/mapper/vg_nvme-lv_5 /var/lib/ceph xfs rw,seclabel,relatime,attr2,inode64,logbufs=8,logbsize=32k,noquota 0 0 /dev/sda1 /var/lib/containers/storage/overlay ext4 rw,seclabel,relatime 0 0 overlay /var/lib/containers/storage/overlay/83efdf1c18da3e9868c4ee7b8a6f0e23ce847bb46f84e39fa4919070189eac1d/merged overlay rw,context="system_u:object_r:container_file_t:s0:c1022,c1023",relatime,lowerdir=/var/lib/containers/storage/overlay/l/GBCYM7Z4PDS3WDKG6VTSNQW6J2:/var/lib/containers/storage/overlay/l/S6EQFOPFAP3J54JB6AX7BW26AD,upperdir=/var/lib/containers/storage/overlay/83efdf1c18da3e9868c4ee7b8a6f0e23ce847bb46f84e39fa4919070189eac1d/diff,workdir=/var/lib/containers/storage/overlay/83efdf1c18da3e9868c4ee7b8a6f0e23ce847bb46f84e39fa4919070189eac1d/work,redirect_dir=on,uuid=on,metacopy=on,volatile 0 0 overlay /var/lib/containers/storage/overlay/ee311df782a40040a3d56dfe3df158ec45de7e308d06f8acdb5dabda37965481/merged overlay rw,context="system_u:object_r:container_file_t:s0:c1022,c1023",nodev,relatime,lowerdir=/var/lib/containers/storage/overlay/l/GBCYM7Z4PDS3WDKG6VTSNQW6J2:/var/lib/containers/storage/overlay/l/S6EQFOPFAP3J54JB6AX7BW26AD,upperdir=/var/lib/containers/storage/overlay/ee311df782a40040a3d56dfe3df158ec45de7e308d06f8acdb5dabda37965481/diff,workdir=/var/lib/containers/storage/overlay/ee311df782a40040a3d56dfe3df158ec45de7e308d06f8acdb5dabda37965481/work,redirect_dir=on,uuid=on,metacopy=on,volatile 0 0 tmpfs /run/user/0 tmpfs rw,seclabel,nosuid,nodev,relatime,size=3249168k,nr_inodes=812292,mode=700 0 0 binfmt_misc /proc/sys/fs/binfmt_misc binfmt_misc rw,nosuid,nodev,noexec,relatime 0 0 overlay /var/lib/containers/storage/overlay/27e56973c184af1439467a2d225a78c2010535bde0de0e97896ead519a35b98e/merged overlay rw,context="system_u:object_r:container_file_t:s0:c1022,c1023",relatime,lowerdir=/var/lib/containers/storage/overlay/l/GBCYM7Z4PDS3WDKG6VTSNQW6J2:/var/lib/containers/storage/overlay/l/S6EQFOPFAP3J54JB6AX7BW26AD,upperdir=/var/lib/containers/storage/overlay/27e56973c184af1439467a2d225a78c2010535bde0de0e97896ead519a35b98e/diff,workdir=/var/lib/containers/storage/overlay/27e56973c184af1439467a2d225a78c2010535bde0de0e97896ead519a35b98e/work,redirect_dir=on,uuid=on,metacopy=on,volatile 0 0 overlay /var/lib/containers/storage/overlay/29f2c8e161d2b360cd5dbd38681d8518ea1ee60a8e511ad705682b591b6b5329/merged overlay rw,context="system_u:object_r:container_file_t:s0:c1022,c1023",relatime,lowerdir=/var/lib/containers/storage/overlay/l/GBCYM7Z4PDS3WDKG6VTSNQW6J2:/var/lib/containers/storage/overlay/l/S6EQFOPFAP3J54JB6AX7BW26AD,upperdir=/var/lib/containers/storage/overlay/29f2c8e161d2b360cd5dbd38681d8518ea1ee60a8e511ad705682b591b6b5329/diff,workdir=/var/lib/containers/storage/overlay/29f2c8e161d2b360cd5dbd38681d8518ea1ee60a8e511ad705682b591b6b5329/work,redirect_dir=on,uuid=on,metacopy=on,volatile 0 0 overlay /var/lib/containers/storage/overlay/2d61e33cea4cb5e4d20c2aefda514d626064328ed98910c3d968ff5cca3fc68b/merged overlay rw,context="system_u:object_r:container_file_t:s0:c1022,c1023",relatime,lowerdir=/var/lib/containers/storage/overlay/l/GBCYM7Z4PDS3WDKG6VTSNQW6J2:/var/lib/containers/storage/overlay/l/S6EQFOPFAP3J54JB6AX7BW26AD,upperdir=/var/lib/containers/storage/overlay/2d61e33cea4cb5e4d20c2aefda514d626064328ed98910c3d968ff5cca3fc68b/diff,workdir=/var/lib/containers/storage/overlay/2d61e33cea4cb5e4d20c2aefda514d626064328ed98910c3d968ff5cca3fc68b/work,redirect_dir=on,uuid=on,metacopy=on,volatile 0 0 overlay /var/lib/containers/storage/overlay/8f25f73c547518d3285a36e3ec9af60637297f6229a252b31ddee60bde661f99/merged overlay rw,context="system_u:object_r:container_file_t:s0:c1022,c1023",relatime,lowerdir=/var/lib/containers/storage/overlay/l/GBCYM7Z4PDS3WDKG6VTSNQW6J2:/var/lib/containers/storage/overlay/l/S6EQFOPFAP3J54JB6AX7BW26AD,upperdir=/var/lib/containers/storage/overlay/8f25f73c547518d3285a36e3ec9af60637297f6229a252b31ddee60bde661f99/diff,workdir=/var/lib/containers/storage/overlay/8f25f73c547518d3285a36e3ec9af60637297f6229a252b31ddee60bde661f99/work,redirect_dir=on,uuid=on,metacopy=on,volatile 0 0 overlay /var/lib/containers/storage/overlay/4cba689008f401463cc9b429d33d35d00f2f226eff1b42894c016467b9848744/merged overlay rw,context="system_u:object_r:container_file_t:s0:c1022,c1023",nodev,relatime,lowerdir=/var/lib/containers/storage/overlay/l/GBCYM7Z4PDS3WDKG6VTSNQW6J2:/var/lib/containers/storage/overlay/l/S6EQFOPFAP3J54JB6AX7BW26AD,upperdir=/var/lib/containers/storage/overlay/4cba689008f401463cc9b429d33d35d00f2f226eff1b42894c016467b9848744/diff,workdir=/var/lib/containers/storage/overlay/4cba689008f401463cc9b429d33d35d00f2f226eff1b42894c016467b9848744/work,redirect_dir=on,uuid=on,metacopy=on,volatile 0 0 tmpfs /run/netns tmpfs rw,seclabel,nosuid,nodev,size=6498340k,nr_inodes=819200,mode=755 0 0 2024-03-24T03:19:39.175 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T03:19:39.175 DEBUG:teuthology.orchestra.run.smithi118:> ip netns list 2024-03-24T03:19:39.261 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T03:19:39.261 DEBUG:teuthology.orchestra.run.smithi118:> sudo ip link delete ceph-brx 2024-03-24T03:19:39.404 INFO:teuthology.orchestra.run.smithi118.stderr:Cannot find device "ceph-brx" 2024-03-24T03:19:39.408 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T03:19:39.409 DEBUG:teuthology.run_tasks:Unwinding manager cephadm 2024-03-24T03:19:39.420 INFO:tasks.cephadm:Teardown begin 2024-03-24T03:19:39.420 INFO:tasks.cephfs.filesystem:Destroying file system cephfs and related pools 2024-03-24T03:19:39.420 DEBUG:teuthology.orchestra.run.smithi118:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-24T03:19:39.604 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 03:19:39 smithi189 ceph-mon[36187]: Adjusting osd_memory_target on smithi193 to 174.1M 2024-03-24T03:19:39.604 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 03:19:39 smithi189 ceph-mon[36187]: Unable to set osd_memory_target on smithi193 to 182639923: error parsing value: Value '182639923' is below minimum 939524096 2024-03-24T03:19:39.605 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 03:19:39 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T03:19:39.605 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 03:19:39 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T03:19:39.605 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 03:19:39 smithi189 ceph-mon[36187]: pgmap v6167: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 6 op/s 2024-03-24T03:19:39.605 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 03:19:39 smithi189 ceph-mon[36187]: pgmap v6168: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 4 op/s 2024-03-24T03:19:39.605 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 03:19:39 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T03:19:39.610 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 03:19:39 smithi193 ceph-mon[36878]: Adjusting osd_memory_target on smithi193 to 174.1M 2024-03-24T03:19:39.611 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 03:19:39 smithi193 ceph-mon[36878]: Unable to set osd_memory_target on smithi193 to 182639923: error parsing value: Value '182639923' is below minimum 939524096 2024-03-24T03:19:39.611 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 03:19:39 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T03:19:39.611 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 03:19:39 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T03:19:39.611 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 03:19:39 smithi193 ceph-mon[36878]: pgmap v6167: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 6 op/s 2024-03-24T03:19:39.611 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 03:19:39 smithi193 ceph-mon[36878]: pgmap v6168: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 4 op/s 2024-03-24T03:19:39.611 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 03:19:39 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T03:19:39.631 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T03:19:39.631+0000 7f72f9400640 1 -- 172.21.15.118:0/3122357875 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f72f40740c0 msgr2=0x7f72f4074540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T03:19:39.631 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T03:19:39.631+0000 7f72f9400640 1 --2- 172.21.15.118:0/3122357875 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f72f40740c0 0x7f72f4074540 secure :-1 s=READY pgs=481 cs=0 l=1 rev1=1 crypto rx=0x7f72dc009900 tx=0x7f72dc02bc30 comp rx=0 tx=0).stop 2024-03-24T03:19:39.631 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T03:19:39.631+0000 7f72f9400640 1 -- 172.21.15.118:0/3122357875 shutdown_connections 2024-03-24T03:19:39.631 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T03:19:39.631+0000 7f72f9400640 1 --2- 172.21.15.118:0/3122357875 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f72f4109590 0x7f72f410ba00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T03:19:39.632 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T03:19:39.631+0000 7f72f9400640 1 --2- 172.21.15.118:0/3122357875 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f72f40740c0 0x7f72f4074540 unknown :-1 s=CLOSED pgs=481 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T03:19:39.632 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T03:19:39.631+0000 7f72f9400640 1 --2- 172.21.15.118:0/3122357875 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f72f40756a0 0x7f72f4073af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T03:19:39.632 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T03:19:39.631+0000 7f72f9400640 1 -- 172.21.15.118:0/3122357875 >> 172.21.15.118:0/3122357875 conn(0x7f72f40fcc50 msgr2=0x7f72f40ff090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T03:19:39.632 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T03:19:39.631+0000 7f72f9400640 1 -- 172.21.15.118:0/3122357875 shutdown_connections 2024-03-24T03:19:39.632 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T03:19:39.631+0000 7f72f9400640 1 -- 172.21.15.118:0/3122357875 wait complete. 2024-03-24T03:19:39.634 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T03:19:39.633+0000 7f72f9400640 1 Processor -- start 2024-03-24T03:19:39.634 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T03:19:39.633+0000 7f72f9400640 1 -- start start 2024-03-24T03:19:39.635 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T03:19:39.633+0000 7f72f9400640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f72f40740c0 0x7f72f4199bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T03:19:39.635 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T03:19:39.633+0000 7f72f9400640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f72f40756a0 0x7f72f419a100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T03:19:39.635 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T03:19:39.633+0000 7f72f9400640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f72f4109590 0x7f72f41a1170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T03:19:39.635 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T03:19:39.633+0000 7f72f9400640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f72f410f570 con 0x7f72f4109590 2024-03-24T03:19:39.635 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T03:19:39.633+0000 7f72f9400640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f72f410f3f0 con 0x7f72f40756a0 2024-03-24T03:19:39.635 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T03:19:39.633+0000 7f72f9400640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f72f410f6f0 con 0x7f72f40740c0 2024-03-24T03:19:39.635 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T03:19:39.633+0000 7f72f1e00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f72f40756a0 0x7f72f419a100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T03:19:39.635 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T03:19:39.633+0000 7f72f2800640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f72f40740c0 0x7f72f4199bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T03:19:39.636 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T03:19:39.633+0000 7f72f1e00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f72f40756a0 0x7f72f419a100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.118:37434/0 (socket says 172.21.15.118:37434) 2024-03-24T03:19:39.636 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T03:19:39.633+0000 7f72f2800640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f72f40740c0 0x7f72f4199bc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.118:47934/0 (socket says 172.21.15.118:47934) 2024-03-24T03:19:39.636 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T03:19:39.633+0000 7f72f1e00640 1 -- 172.21.15.118:0/2368098318 learned_addr learned my addr 172.21.15.118:0/2368098318 (peer_addr_for_me v2:172.21.15.118:0/0) 2024-03-24T03:19:39.636 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T03:19:39.634+0000 7f72f3200640 1 --2- 172.21.15.118:0/2368098318 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f72f4109590 0x7f72f41a1170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T03:19:39.636 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T03:19:39.634+0000 7f72f2800640 1 -- 172.21.15.118:0/2368098318 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f72f40756a0 msgr2=0x7f72f419a100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T03:19:39.636 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T03:19:39.634+0000 7f72f2800640 1 --2- 172.21.15.118:0/2368098318 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f72f40756a0 0x7f72f419a100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T03:19:39.636 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T03:19:39.634+0000 7f72f2800640 1 -- 172.21.15.118:0/2368098318 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f72f4109590 msgr2=0x7f72f41a1170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T03:19:39.636 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T03:19:39.634+0000 7f72f2800640 1 --2- 172.21.15.118:0/2368098318 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f72f4109590 0x7f72f41a1170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T03:19:39.637 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T03:19:39.634+0000 7f72f2800640 1 -- 172.21.15.118:0/2368098318 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f72f41a17e0 con 0x7f72f40740c0 2024-03-24T03:19:39.637 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T03:19:39.634+0000 7f72f3200640 1 --2- 172.21.15.118:0/2368098318 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f72f4109590 0x7f72f41a1170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T03:19:39.637 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T03:19:39.634+0000 7f72f2800640 1 --2- 172.21.15.118:0/2368098318 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f72f40740c0 0x7f72f4199bc0 secure :-1 s=READY pgs=482 cs=0 l=1 rev1=1 crypto rx=0x7f72e400e9f0 tx=0x7f72e400eeb0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T03:19:39.637 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T03:19:39.634+0000 7f72eb400640 1 -- 172.21.15.118:0/2368098318 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f72e400ccc0 con 0x7f72f40740c0 2024-03-24T03:19:39.637 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T03:19:39.634+0000 7f72eb400640 1 -- 172.21.15.118:0/2368098318 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f72e4010910 con 0x7f72f40740c0 2024-03-24T03:19:39.637 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T03:19:39.634+0000 7f72f9400640 1 -- 172.21.15.118:0/2368098318 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f72f41a1ad0 con 0x7f72f40740c0 2024-03-24T03:19:39.637 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T03:19:39.634+0000 7f72eb400640 1 -- 172.21.15.118:0/2368098318 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f72e4018960 con 0x7f72f40740c0 2024-03-24T03:19:39.637 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T03:19:39.635+0000 7f72f9400640 1 -- 172.21.15.118:0/2368098318 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f72f41a1ff0 con 0x7f72f40740c0 2024-03-24T03:19:39.638 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T03:19:39.636+0000 7f72f9400640 1 -- 172.21.15.118:0/2368098318 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f72f4108310 con 0x7f72f40740c0 2024-03-24T03:19:39.638 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T03:19:39.637+0000 7f72eb400640 1 -- 172.21.15.118:0/2368098318 <== mon.1 v2:172.21.15.193:3300/0 4 ==== mgrmap(e 20) v1 ==== 137291+0+0 (secure 0 0 0) 0x7f72e4010ac0 con 0x7f72f40740c0 2024-03-24T03:19:39.638 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T03:19:39.639+0000 7f72eb400640 1 --2- 172.21.15.118:0/2368098318 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7f72d80a2800 0x7f72d80a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T03:19:39.639 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T03:19:39.639+0000 7f72eb400640 1 -- 172.21.15.118:0/2368098318 <== mon.1 v2:172.21.15.193:3300/0 5 ==== osd_map(124..124 src has 1..124) v4 ==== 11201+0+0 (secure 0 0 0) 0x7f72e40cfac0 con 0x7f72f40740c0 2024-03-24T03:19:39.639 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T03:19:39.639+0000 7f72f1e00640 1 --2- 172.21.15.118:0/2368098318 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7f72d80a2800 0x7f72d80a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T03:19:39.639 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T03:19:39.640+0000 7f72f1e00640 1 --2- 172.21.15.118:0/2368098318 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7f72d80a2800 0x7f72d80a4cc0 secure :-1 s=READY pgs=646 cs=0 l=1 rev1=1 crypto rx=0x7f72dc02c8e0 tx=0x7f72dc02df90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T03:19:39.642 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T03:19:39.643+0000 7f72eb400640 1 -- 172.21.15.118:0/2368098318 <== mon.1 v2:172.21.15.193:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f72e409bf60 con 0x7f72f40740c0 2024-03-24T03:19:39.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 03:19:39 smithi118 ceph-mon[31535]: Adjusting osd_memory_target on smithi193 to 174.1M 2024-03-24T03:19:39.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 03:19:39 smithi118 ceph-mon[31535]: Unable to set osd_memory_target on smithi193 to 182639923: error parsing value: Value '182639923' is below minimum 939524096 2024-03-24T03:19:39.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 03:19:39 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T03:19:39.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 03:19:39 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T03:19:39.697 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 03:19:39 smithi118 ceph-mon[31535]: pgmap v6167: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.5 KiB/s rd, 6 op/s 2024-03-24T03:19:39.697 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 03:19:39 smithi118 ceph-mon[31535]: pgmap v6168: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.6 KiB/s rd, 4 op/s 2024-03-24T03:19:39.697 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 03:19:39 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' 2024-03-24T03:19:39.881 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T03:19:39.881+0000 7f72f9400640 1 -- 172.21.15.118:0/2368098318 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f72f402d070 con 0x7f72f40740c0 2024-03-24T03:19:39.882 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T03:19:39.882+0000 7f72eb400640 1 -- 172.21.15.118:0/2368098318 <== mon.1 v2:172.21.15.193:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 188 v188) v1 ==== 95+0+8388 (secure 0 0 0) 0x7f72e409b860 con 0x7f72f40740c0 2024-03-24T03:19:39.882 INFO:teuthology.orchestra.run.smithi118.stdout: 2024-03-24T03:19:39.882 INFO:teuthology.orchestra.run.smithi118.stdout:{"epoch":188,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":188,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-03-24T00:13:01.782474+0000","modified":"2024-03-24T00:28:06.371587+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":124,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24569,"mds_1":27076,"mds_2":24511,"mds_3":27070,"mds_4":24488},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14469":{"gid":14469,"name":"a","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.118:6841/3040946537","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6840","nonce":3040946537},{"type":"v1","addr":"172.21.15.118:6841","nonce":3040946537}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24440":{"gid":24440,"name":"k","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.189:6833/2740024078","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6832","nonce":2740024078},{"type":"v1","addr":"172.21.15.189:6833","nonce":2740024078}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24488":{"gid":24488,"name":"e","rank":4,"incarnation":25,"state":"up:active","state_seq":10,"addr":"172.21.15.189:6835/599473205","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6834","nonce":599473205},{"type":"v1","addr":"172.21.15.189:6835","nonce":599473205}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24511":{"gid":24511,"name":"c","rank":2,"incarnation":21,"state":"up:active","state_seq":8,"addr":"172.21.15.193:6837/2921520053","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6836","nonce":2921520053},{"type":"v1","addr":"172.21.15.193:6837","nonce":2921520053}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24532":{"gid":24532,"name":"i","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.193:6839/2126842603","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6838","nonce":2126842603},{"type":"v1","addr":"172.21.15.193:6839","nonce":2126842603}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24569":{"gid":24569,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":229,"addr":"172.21.15.189:6839/4074484168","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6838","nonce":4074484168},{"type":"v1","addr":"172.21.15.189:6839","nonce":4074484168}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_27070":{"gid":27070,"name":"j","rank":3,"incarnation":180,"state":"up:active","state_seq":5,"addr":"172.21.15.118:6835/395010504","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6834","nonce":395010504},{"type":"v1","addr":"172.21.15.118:6835","nonce":395010504}]},"join_fscid":-1,"laggy_since":"2024-03-24T00:25:21.665663+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_27076":{"gid":27076,"name":"d","rank":1,"incarnation":182,"state":"up:active","state_seq":5,"addr":"172.21.15.118:6837/3350294232","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6836","nonce":3350294232},{"type":"v1","addr":"172.21.15.118:6837","nonce":3350294232}]},"join_fscid":-1,"laggy_since":"2024-03-24T00:25:21.665677+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24569,"qdb_cluster":[24488,24511,24569,27070,27076]},"id":1}]} 2024-03-24T03:19:39.883 INFO:teuthology.orchestra.run.smithi118.stderr:dumped fsmap epoch 188 2024-03-24T03:19:39.885 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T03:19:39.885+0000 7f72f9400640 1 -- 172.21.15.118:0/2368098318 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7f72d80a2800 msgr2=0x7f72d80a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T03:19:39.885 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T03:19:39.885+0000 7f72f9400640 1 --2- 172.21.15.118:0/2368098318 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7f72d80a2800 0x7f72d80a4cc0 secure :-1 s=READY pgs=646 cs=0 l=1 rev1=1 crypto rx=0x7f72dc02c8e0 tx=0x7f72dc02df90 comp rx=0 tx=0).stop 2024-03-24T03:19:39.885 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T03:19:39.885+0000 7f72f9400640 1 -- 172.21.15.118:0/2368098318 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f72f40740c0 msgr2=0x7f72f4199bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T03:19:39.885 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T03:19:39.885+0000 7f72f9400640 1 --2- 172.21.15.118:0/2368098318 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f72f40740c0 0x7f72f4199bc0 secure :-1 s=READY pgs=482 cs=0 l=1 rev1=1 crypto rx=0x7f72e400e9f0 tx=0x7f72e400eeb0 comp rx=0 tx=0).stop 2024-03-24T03:19:39.886 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T03:19:39.885+0000 7f72f9400640 1 -- 172.21.15.118:0/2368098318 shutdown_connections 2024-03-24T03:19:39.886 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T03:19:39.885+0000 7f72f9400640 1 --2- 172.21.15.118:0/2368098318 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7f72d80a2800 0x7f72d80a4cc0 unknown :-1 s=CLOSED pgs=646 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T03:19:39.886 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T03:19:39.885+0000 7f72f9400640 1 --2- 172.21.15.118:0/2368098318 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f72f4109590 0x7f72f41a1170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T03:19:39.886 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T03:19:39.885+0000 7f72f9400640 1 --2- 172.21.15.118:0/2368098318 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f72f40756a0 0x7f72f419a100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T03:19:39.886 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T03:19:39.885+0000 7f72f9400640 1 --2- 172.21.15.118:0/2368098318 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f72f40740c0 0x7f72f4199bc0 unknown :-1 s=CLOSED pgs=482 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T03:19:39.886 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T03:19:39.885+0000 7f72f9400640 1 -- 172.21.15.118:0/2368098318 >> 172.21.15.118:0/2368098318 conn(0x7f72f40fcc50 msgr2=0x7f72f41054e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T03:19:39.886 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T03:19:39.885+0000 7f72f9400640 1 -- 172.21.15.118:0/2368098318 shutdown_connections 2024-03-24T03:19:39.887 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T03:19:39.885+0000 7f72f9400640 1 -- 172.21.15.118:0/2368098318 wait complete. 2024-03-24T03:19:39.903 DEBUG:teuthology.orchestra.run.smithi118:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-24T03:19:40.107 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T03:19:40.107+0000 7f05a8a00640 1 -- 172.21.15.118:0/123317483 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f05a4104ad0 msgr2=0x7f05a4104f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T03:19:40.108 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T03:19:40.107+0000 7f05a8a00640 1 --2- 172.21.15.118:0/123317483 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f05a4104ad0 0x7f05a4104f50 secure :-1 s=READY pgs=483 cs=0 l=1 rev1=1 crypto rx=0x7f0594009900 tx=0x7f059402bc30 comp rx=0 tx=0).stop 2024-03-24T03:19:40.108 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T03:19:40.108+0000 7f05a8a00640 1 -- 172.21.15.118:0/123317483 shutdown_connections 2024-03-24T03:19:40.108 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T03:19:40.108+0000 7f05a8a00640 1 --2- 172.21.15.118:0/123317483 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f05a4105490 0x7f05a410bd20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T03:19:40.108 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T03:19:40.108+0000 7f05a8a00640 1 --2- 172.21.15.118:0/123317483 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f05a4104ad0 0x7f05a4104f50 unknown :-1 s=CLOSED pgs=483 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T03:19:40.108 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T03:19:40.108+0000 7f05a8a00640 1 --2- 172.21.15.118:0/123317483 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f05a41038e0 0x7f05a4103d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T03:19:40.108 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T03:19:40.108+0000 7f05a8a00640 1 -- 172.21.15.118:0/123317483 >> 172.21.15.118:0/123317483 conn(0x7f05a40fee50 msgr2=0x7f05a4101290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T03:19:40.109 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T03:19:40.108+0000 7f05a8a00640 1 -- 172.21.15.118:0/123317483 shutdown_connections 2024-03-24T03:19:40.109 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T03:19:40.108+0000 7f05a8a00640 1 -- 172.21.15.118:0/123317483 wait complete. 2024-03-24T03:19:40.111 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T03:19:40.109+0000 7f05a8a00640 1 Processor -- start 2024-03-24T03:19:40.111 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T03:19:40.109+0000 7f05a8a00640 1 -- start start 2024-03-24T03:19:40.111 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T03:19:40.109+0000 7f05a8a00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f05a41038e0 0x7f05a4199c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T03:19:40.111 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T03:19:40.109+0000 7f05a8a00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f05a4104ad0 0x7f05a419a160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T03:19:40.112 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T03:19:40.109+0000 7f05a8a00640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f05a4105490 0x7f05a41a1190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T03:19:40.112 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T03:19:40.109+0000 7f05a8a00640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f05a410f590 con 0x7f05a4105490 2024-03-24T03:19:40.112 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T03:19:40.109+0000 7f05a8a00640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f05a410f410 con 0x7f05a4104ad0 2024-03-24T03:19:40.112 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T03:19:40.109+0000 7f05a8a00640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f05a410f710 con 0x7f05a41038e0 2024-03-24T03:19:40.112 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T03:19:40.109+0000 7f05a1e00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f05a41038e0 0x7f05a4199c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T03:19:40.112 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T03:19:40.109+0000 7f05a1e00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f05a41038e0 0x7f05a4199c20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.118:47946/0 (socket says 172.21.15.118:47946) 2024-03-24T03:19:40.112 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T03:19:40.109+0000 7f05a1400640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f05a4104ad0 0x7f05a419a160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T03:19:40.112 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T03:19:40.109+0000 7f05a1e00640 1 -- 172.21.15.118:0/4119823497 learned_addr learned my addr 172.21.15.118:0/4119823497 (peer_addr_for_me v2:172.21.15.118:0/0) 2024-03-24T03:19:40.113 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T03:19:40.109+0000 7f05a2800640 1 --2- 172.21.15.118:0/4119823497 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f05a4105490 0x7f05a41a1190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T03:19:40.113 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T03:19:40.110+0000 7f05a1400640 1 -- 172.21.15.118:0/4119823497 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f05a41038e0 msgr2=0x7f05a4199c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T03:19:40.113 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T03:19:40.110+0000 7f05a1400640 1 --2- 172.21.15.118:0/4119823497 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f05a41038e0 0x7f05a4199c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T03:19:40.113 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T03:19:40.110+0000 7f05a1400640 1 -- 172.21.15.118:0/4119823497 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f05a4105490 msgr2=0x7f05a41a1190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T03:19:40.113 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T03:19:40.110+0000 7f05a1400640 1 --2- 172.21.15.118:0/4119823497 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f05a4105490 0x7f05a41a1190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T03:19:40.113 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T03:19:40.110+0000 7f05a1400640 1 -- 172.21.15.118:0/4119823497 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f05a41a1830 con 0x7f05a4104ad0 2024-03-24T03:19:40.113 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T03:19:40.110+0000 7f05a1e00640 1 --2- 172.21.15.118:0/4119823497 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f05a41038e0 0x7f05a4199c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T03:19:40.113 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T03:19:40.110+0000 7f05a1400640 1 --2- 172.21.15.118:0/4119823497 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f05a4104ad0 0x7f05a419a160 secure :-1 s=READY pgs=484 cs=0 l=1 rev1=1 crypto rx=0x7f0594036040 tx=0x7f059402c6c0 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T03:19:40.114 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T03:19:40.110+0000 7f059aa00640 1 -- 172.21.15.118:0/4119823497 <== mon.2 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0594043020 con 0x7f05a4104ad0 2024-03-24T03:19:40.114 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T03:19:40.110+0000 7f059aa00640 1 -- 172.21.15.118:0/4119823497 <== mon.2 v2:172.21.15.189:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f05940054e0 con 0x7f05a4104ad0 2024-03-24T03:19:40.114 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T03:19:40.110+0000 7f059aa00640 1 -- 172.21.15.118:0/4119823497 <== mon.2 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f0594003ca0 con 0x7f05a4104ad0 2024-03-24T03:19:40.114 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T03:19:40.110+0000 7f05a8a00640 1 -- 172.21.15.118:0/4119823497 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f05a41a1ac0 con 0x7f05a4104ad0 2024-03-24T03:19:40.114 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T03:19:40.111+0000 7f05a8a00640 1 -- 172.21.15.118:0/4119823497 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f05a41a2030 con 0x7f05a4104ad0 2024-03-24T03:19:40.114 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T03:19:40.112+0000 7f05a8a00640 1 -- 172.21.15.118:0/4119823497 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f05a410ac80 con 0x7f05a4104ad0 2024-03-24T03:19:40.115 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T03:19:40.113+0000 7f059aa00640 1 -- 172.21.15.118:0/4119823497 <== mon.2 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 20) v1 ==== 137291+0+0 (secure 0 0 0) 0x7f0594046020 con 0x7f05a4104ad0 2024-03-24T03:19:40.115 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T03:19:40.115+0000 7f059aa00640 1 --2- 172.21.15.118:0/4119823497 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7f05880a2880 0x7f05880a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T03:19:40.115 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T03:19:40.115+0000 7f059aa00640 1 -- 172.21.15.118:0/4119823497 <== mon.2 v2:172.21.15.189:3300/0 5 ==== osd_map(124..124 src has 1..124) v4 ==== 11201+0+0 (secure 0 0 0) 0x7f05940f16c0 con 0x7f05a4104ad0 2024-03-24T03:19:40.115 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T03:19:40.115+0000 7f05a1e00640 1 --2- 172.21.15.118:0/4119823497 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7f05880a2880 0x7f05880a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T03:19:40.116 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T03:19:40.116+0000 7f05a1e00640 1 --2- 172.21.15.118:0/4119823497 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7f05880a2880 0x7f05880a4d40 secure :-1 s=READY pgs=647 cs=0 l=1 rev1=1 crypto rx=0x7f05a419b0f0 tx=0x7f059000b410 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T03:19:40.118 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T03:19:40.119+0000 7f059aa00640 1 -- 172.21.15.118:0/4119823497 <== mon.2 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f059404d020 con 0x7f05a4104ad0 2024-03-24T03:19:40.354 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T03:19:40.355+0000 7f05a8a00640 1 -- 172.21.15.118:0/4119823497 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f05a40622d0 con 0x7f05a4104ad0 2024-03-24T03:19:40.358 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T03:19:40.358+0000 7f059aa00640 1 -- 172.21.15.118:0/4119823497 <== mon.2 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 188 v188) v1 ==== 95+0+8388 (secure 0 0 0) 0x7f05940bdb60 con 0x7f05a4104ad0 2024-03-24T03:19:40.358 INFO:teuthology.orchestra.run.smithi118.stdout: 2024-03-24T03:19:40.358 INFO:teuthology.orchestra.run.smithi118.stdout:{"epoch":188,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[],"filesystems":[{"mdsmap":{"epoch":188,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true,"refuse_client_session":false,"refuse_standby_for_another_fs":false,"balance_automate":false},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-03-24T00:13:01.782474+0000","modified":"2024-03-24T00:28:06.371587+0000","tableserver":0,"root":0,"session_timeout":300,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"max_xattr_size":65536,"last_failure":0,"last_failure_osd_epoch":124,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}},"max_mds":5,"in":[0,1,2,3,4],"up":{"mds_0":24569,"mds_1":27076,"mds_2":24511,"mds_3":27070,"mds_4":24488},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14469":{"gid":14469,"name":"a","rank":4,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.118:6841/3040946537","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6840","nonce":3040946537},{"type":"v1","addr":"172.21.15.118:6841","nonce":3040946537}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24440":{"gid":24440,"name":"k","rank":2,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.189:6833/2740024078","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6832","nonce":2740024078},{"type":"v1","addr":"172.21.15.189:6833","nonce":2740024078}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24488":{"gid":24488,"name":"e","rank":4,"incarnation":25,"state":"up:active","state_seq":10,"addr":"172.21.15.189:6835/599473205","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6834","nonce":599473205},{"type":"v1","addr":"172.21.15.189:6835","nonce":599473205}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24511":{"gid":24511,"name":"c","rank":2,"incarnation":21,"state":"up:active","state_seq":8,"addr":"172.21.15.193:6837/2921520053","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6836","nonce":2921520053},{"type":"v1","addr":"172.21.15.193:6837","nonce":2921520053}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24532":{"gid":24532,"name":"i","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":1,"addr":"172.21.15.193:6839/2126842603","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6838","nonce":2126842603},{"type":"v1","addr":"172.21.15.193:6839","nonce":2126842603}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_24569":{"gid":24569,"name":"b","rank":0,"incarnation":14,"state":"up:active","state_seq":229,"addr":"172.21.15.189:6839/4074484168","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6838","nonce":4074484168},{"type":"v1","addr":"172.21.15.189:6839","nonce":4074484168}]},"join_fscid":-1,"export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_27070":{"gid":27070,"name":"j","rank":3,"incarnation":180,"state":"up:active","state_seq":5,"addr":"172.21.15.118:6835/395010504","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6834","nonce":395010504},{"type":"v1","addr":"172.21.15.118:6835","nonce":395010504}]},"join_fscid":-1,"laggy_since":"2024-03-24T00:25:21.665663+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}},"gid_27076":{"gid":27076,"name":"d","rank":1,"incarnation":182,"state":"up:active","state_seq":5,"addr":"172.21.15.118:6837/3350294232","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6836","nonce":3350294232},{"type":"v1","addr":"172.21.15.118:6837","nonce":3350294232}]},"join_fscid":-1,"laggy_since":"2024-03-24T00:25:21.665677+0000","export_targets":[],"features":4540701547738038271,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2","feature_11":"minor log segments"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","bal_rank_mask":"-1","standby_count_wanted":0,"qdb_leader":24569,"qdb_cluster":[24488,24511,24569,27070,27076]},"id":1}]} 2024-03-24T03:19:40.358 INFO:teuthology.orchestra.run.smithi118.stderr:dumped fsmap epoch 188 2024-03-24T03:19:40.360 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T03:19:40.360+0000 7f05a8a00640 1 -- 172.21.15.118:0/4119823497 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7f05880a2880 msgr2=0x7f05880a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T03:19:40.361 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T03:19:40.360+0000 7f05a8a00640 1 --2- 172.21.15.118:0/4119823497 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7f05880a2880 0x7f05880a4d40 secure :-1 s=READY pgs=647 cs=0 l=1 rev1=1 crypto rx=0x7f05a419b0f0 tx=0x7f059000b410 comp rx=0 tx=0).stop 2024-03-24T03:19:40.361 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T03:19:40.361+0000 7f05a8a00640 1 -- 172.21.15.118:0/4119823497 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f05a4104ad0 msgr2=0x7f05a419a160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T03:19:40.361 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T03:19:40.361+0000 7f05a8a00640 1 --2- 172.21.15.118:0/4119823497 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f05a4104ad0 0x7f05a419a160 secure :-1 s=READY pgs=484 cs=0 l=1 rev1=1 crypto rx=0x7f0594036040 tx=0x7f059402c6c0 comp rx=0 tx=0).stop 2024-03-24T03:19:40.361 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T03:19:40.361+0000 7f05a8a00640 1 -- 172.21.15.118:0/4119823497 shutdown_connections 2024-03-24T03:19:40.361 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T03:19:40.361+0000 7f05a8a00640 1 --2- 172.21.15.118:0/4119823497 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7f05880a2880 0x7f05880a4d40 unknown :-1 s=CLOSED pgs=647 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T03:19:40.361 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T03:19:40.361+0000 7f05a8a00640 1 --2- 172.21.15.118:0/4119823497 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f05a4105490 0x7f05a41a1190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T03:19:40.361 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T03:19:40.361+0000 7f05a8a00640 1 --2- 172.21.15.118:0/4119823497 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f05a4104ad0 0x7f05a419a160 unknown :-1 s=CLOSED pgs=484 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T03:19:40.361 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T03:19:40.361+0000 7f05a8a00640 1 --2- 172.21.15.118:0/4119823497 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f05a41038e0 0x7f05a4199c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T03:19:40.362 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T03:19:40.361+0000 7f05a8a00640 1 -- 172.21.15.118:0/4119823497 >> 172.21.15.118:0/4119823497 conn(0x7f05a40fee50 msgr2=0x7f05a4107e50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T03:19:40.362 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T03:19:40.361+0000 7f05a8a00640 1 -- 172.21.15.118:0/4119823497 shutdown_connections 2024-03-24T03:19:40.362 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T03:19:40.361+0000 7f05a8a00640 1 -- 172.21.15.118:0/4119823497 wait complete. 2024-03-24T03:19:40.378 DEBUG:teuthology.orchestra.run.smithi118:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-24T03:19:40.586 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T03:19:40.586+0000 7fd3a3600640 1 -- 172.21.15.118:0/1083742838 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd39c1014a0 msgr2=0x7fd39c107d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T03:19:40.586 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T03:19:40.586+0000 7fd3a3600640 1 --2- 172.21.15.118:0/1083742838 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd39c1014a0 0x7fd39c107d30 secure :-1 s=READY pgs=485 cs=0 l=1 rev1=1 crypto rx=0x7fd394009be0 tx=0x7fd39402bef0 comp rx=0 tx=0).stop 2024-03-24T03:19:40.587 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T03:19:40.587+0000 7fd3a3600640 1 -- 172.21.15.118:0/1083742838 shutdown_connections 2024-03-24T03:19:40.587 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T03:19:40.587+0000 7fd3a3600640 1 --2- 172.21.15.118:0/1083742838 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd39c1014a0 0x7fd39c107d30 unknown :-1 s=CLOSED pgs=485 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T03:19:40.587 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T03:19:40.587+0000 7fd3a3600640 1 --2- 172.21.15.118:0/1083742838 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fd39c100ae0 0x7fd39c100f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T03:19:40.587 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T03:19:40.587+0000 7fd3a3600640 1 --2- 172.21.15.118:0/1083742838 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fd39c0ff8f0 0x7fd39c0ffd10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T03:19:40.587 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T03:19:40.587+0000 7fd3a3600640 1 -- 172.21.15.118:0/1083742838 >> 172.21.15.118:0/1083742838 conn(0x7fd39c0faea0 msgr2=0x7fd39c0fd2c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T03:19:40.587 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T03:19:40.587+0000 7fd3a3600640 1 -- 172.21.15.118:0/1083742838 shutdown_connections 2024-03-24T03:19:40.587 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T03:19:40.587+0000 7fd3a3600640 1 -- 172.21.15.118:0/1083742838 wait complete. 2024-03-24T03:19:40.589 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T03:19:40.587+0000 7fd3a3600640 1 Processor -- start 2024-03-24T03:19:40.589 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T03:19:40.588+0000 7fd3a3600640 1 -- start start 2024-03-24T03:19:40.590 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T03:19:40.588+0000 7fd3a3600640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd39c0ff8f0 0x7fd39c103880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T03:19:40.590 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T03:19:40.588+0000 7fd3a3600640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fd39c100ae0 0x7fd39c101ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T03:19:40.590 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T03:19:40.588+0000 7fd3a3600640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fd39c1014a0 0x7fd39c102400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T03:19:40.590 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T03:19:40.588+0000 7fd3a3600640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd39c10b190 con 0x7fd39c100ae0 2024-03-24T03:19:40.590 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T03:19:40.588+0000 7fd3a3600640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd39c10b010 con 0x7fd39c0ff8f0 2024-03-24T03:19:40.590 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T03:19:40.588+0000 7fd3a3600640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd39c10b310 con 0x7fd39c1014a0 2024-03-24T03:19:40.590 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T03:19:40.588+0000 7fd3a0c00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd39c0ff8f0 0x7fd39c103880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T03:19:40.590 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T03:19:40.588+0000 7fd39be00640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fd39c100ae0 0x7fd39c101ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T03:19:40.591 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T03:19:40.588+0000 7fd3a1600640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fd39c1014a0 0x7fd39c102400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T03:19:40.591 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T03:19:40.588+0000 7fd3a0c00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd39c0ff8f0 0x7fd39c103880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.118:37474/0 (socket says 172.21.15.118:37474) 2024-03-24T03:19:40.591 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T03:19:40.588+0000 7fd3a0c00640 1 -- 172.21.15.118:0/1562286803 learned_addr learned my addr 172.21.15.118:0/1562286803 (peer_addr_for_me v2:172.21.15.118:0/0) 2024-03-24T03:19:40.591 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T03:19:40.589+0000 7fd3a1600640 1 -- 172.21.15.118:0/1562286803 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd39c0ff8f0 msgr2=0x7fd39c103880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T03:19:40.591 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T03:19:40.589+0000 7fd3a1600640 1 --2- 172.21.15.118:0/1562286803 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd39c0ff8f0 0x7fd39c103880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T03:19:40.591 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T03:19:40.589+0000 7fd3a1600640 1 -- 172.21.15.118:0/1562286803 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fd39c100ae0 msgr2=0x7fd39c101ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T03:19:40.591 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T03:19:40.589+0000 7fd3a1600640 1 --2- 172.21.15.118:0/1562286803 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fd39c100ae0 0x7fd39c101ec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T03:19:40.591 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T03:19:40.589+0000 7fd3a1600640 1 -- 172.21.15.118:0/1562286803 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd39c102c90 con 0x7fd39c1014a0 2024-03-24T03:19:40.592 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T03:19:40.589+0000 7fd3a1600640 1 --2- 172.21.15.118:0/1562286803 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fd39c1014a0 0x7fd39c102400 secure :-1 s=READY pgs=483 cs=0 l=1 rev1=1 crypto rx=0x7fd39402c430 tx=0x7fd3940337b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T03:19:40.592 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T03:19:40.589+0000 7fd399600640 1 -- 172.21.15.118:0/1562286803 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd394043020 con 0x7fd39c1014a0 2024-03-24T03:19:40.592 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T03:19:40.589+0000 7fd399600640 1 -- 172.21.15.118:0/1562286803 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fd394003b80 con 0x7fd39c1014a0 2024-03-24T03:19:40.592 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T03:19:40.589+0000 7fd3a3600640 1 -- 172.21.15.118:0/1562286803 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd39c1a1b70 con 0x7fd39c1014a0 2024-03-24T03:19:40.592 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T03:19:40.589+0000 7fd399600640 1 -- 172.21.15.118:0/1562286803 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fd394034690 con 0x7fd39c1014a0 2024-03-24T03:19:40.592 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T03:19:40.589+0000 7fd3a3600640 1 -- 172.21.15.118:0/1562286803 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd39c1a2110 con 0x7fd39c1014a0 2024-03-24T03:19:40.593 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T03:19:40.590+0000 7fd3a3600640 1 -- 172.21.15.118:0/1562286803 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd39c1a2340 con 0x7fd39c1014a0 2024-03-24T03:19:40.593 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T03:19:40.592+0000 7fd399600640 1 -- 172.21.15.118:0/1562286803 <== mon.1 v2:172.21.15.193:3300/0 4 ==== mgrmap(e 20) v1 ==== 137291+0+0 (secure 0 0 0) 0x7fd394046020 con 0x7fd39c1014a0 2024-03-24T03:19:40.593 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T03:19:40.594+0000 7fd399600640 1 --2- 172.21.15.118:0/1562286803 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7fd3800a2880 0x7fd3800a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T03:19:40.594 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T03:19:40.594+0000 7fd399600640 1 -- 172.21.15.118:0/1562286803 <== mon.1 v2:172.21.15.193:3300/0 5 ==== osd_map(124..124 src has 1..124) v4 ==== 11201+0+0 (secure 0 0 0) 0x7fd3940f1b40 con 0x7fd39c1014a0 2024-03-24T03:19:40.594 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T03:19:40.594+0000 7fd3a0c00640 1 --2- 172.21.15.118:0/1562286803 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7fd3800a2880 0x7fd3800a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T03:19:40.594 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T03:19:40.595+0000 7fd3a0c00640 1 --2- 172.21.15.118:0/1562286803 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7fd3800a2880 0x7fd3800a4d40 secure :-1 s=READY pgs=648 cs=0 l=1 rev1=1 crypto rx=0x7fd38c005fd0 tx=0x7fd38c00b4a0 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T03:19:40.597 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T03:19:40.598+0000 7fd399600640 1 -- 172.21.15.118:0/1562286803 <== mon.1 v2:172.21.15.193:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7fd39404d020 con 0x7fd39c1014a0 2024-03-24T03:19:40.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 03:19:40 smithi118 ceph-mon[31535]: from='client.? 172.21.15.118:0/2368098318' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-24T03:19:40.703 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 03:19:40 smithi189 ceph-mon[36187]: from='client.? 172.21.15.118:0/2368098318' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-24T03:19:40.710 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 03:19:40 smithi193 ceph-mon[36878]: from='client.? 172.21.15.118:0/2368098318' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-24T03:19:40.797 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T03:19:40.797+0000 7fd3a3600640 1 -- 172.21.15.118:0/1562286803 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fd39c1a26d0 con 0x7fd39c1014a0 2024-03-24T03:19:40.799 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T03:19:40.799+0000 7fd399600640 1 -- 172.21.15.118:0/1562286803 <== mon.1 v2:172.21.15.193:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v124) v1 ==== 74+0+27305 (secure 0 0 0) 0x7fd39c1a26d0 con 0x7fd39c1014a0 2024-03-24T03:19:40.799 INFO:teuthology.orchestra.run.smithi118.stdout: 2024-03-24T03:19:40.800 INFO:teuthology.orchestra.run.smithi118.stdout:{"epoch":124,"fsid":"d79a5420-e971-11ee-95cd-87774f69a715","created":"2024-03-24T00:03:58.648524+0000","modified":"2024-03-24T00:24:59.499441+0000","last_up_change":"2024-03-24T00:12:03.809000+0000","last_in_change":"2024-03-24T00:11:44.389344+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":26,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":12,"require_min_compat_client":"luminous","min_compat_client":"luminous","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-24T00:07:50.190965+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":12,"score_stable":12,"optimal_score":0.25,"raw_score_acting":3,"raw_score_stable":3,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"cephfs_metadata","create_time":"2024-03-24T00:12:59.993723+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"85","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":3,"pool_name":"cephfs_data","create_time":"2024-03-24T00:13:01.257347+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":64,"pg_placement_num":64,"pg_placement_num_target":64,"pg_num_target":64,"pg_num_pending":64,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"86","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":64,"target_size_ratio":0.89999997615814209},"application_metadata":{"cephfs":{"data":"cephfs"}},"read_balance":{"score_type":"Fair distribution","score_acting":1.5,"score_stable":1.5,"optimal_score":1,"raw_score_acting":1.5,"raw_score_stable":1.5,"primary_affinity_weighted":0.99999988079071045,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"c9590426-3901-4f68-a5ef-d57aa8fbac92","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6802","nonce":557690061},{"type":"v1","addr":"172.21.15.118:6803","nonce":557690061}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6804","nonce":557690061},{"type":"v1","addr":"172.21.15.118:6805","nonce":557690061}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6808","nonce":557690061},{"type":"v1","addr":"172.21.15.118:6809","nonce":557690061}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6806","nonce":557690061},{"type":"v1","addr":"172.21.15.118:6807","nonce":557690061}]},"public_addr":"172.21.15.118:6803/557690061","cluster_addr":"172.21.15.118:6805/557690061","heartbeat_back_addr":"172.21.15.118:6809/557690061","heartbeat_front_addr":"172.21.15.118:6807/557690061","state":["exists","up"]},{"osd":1,"uuid":"fc7f827e-ecf6-4661-aad0-7a22951892e0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6800","nonce":4203029485},{"type":"v1","addr":"172.21.15.189:6801","nonce":4203029485}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6802","nonce":4203029485},{"type":"v1","addr":"172.21.15.189:6803","nonce":4203029485}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6806","nonce":4203029485},{"type":"v1","addr":"172.21.15.189:6807","nonce":4203029485}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6804","nonce":4203029485},{"type":"v1","addr":"172.21.15.189:6805","nonce":4203029485}]},"public_addr":"172.21.15.189:6801/4203029485","cluster_addr":"172.21.15.189:6803/4203029485","heartbeat_back_addr":"172.21.15.189:6807/4203029485","heartbeat_front_addr":"172.21.15.189:6805/4203029485","state":["exists","up"]},{"osd":2,"uuid":"e73ccfe9-0e5b-4734-a154-a2f0d829fcfb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":18,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6800","nonce":3699076971},{"type":"v1","addr":"172.21.15.193:6801","nonce":3699076971}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6802","nonce":3699076971},{"type":"v1","addr":"172.21.15.193:6803","nonce":3699076971}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6806","nonce":3699076971},{"type":"v1","addr":"172.21.15.193:6807","nonce":3699076971}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6804","nonce":3699076971},{"type":"v1","addr":"172.21.15.193:6805","nonce":3699076971}]},"public_addr":"172.21.15.193:6801/3699076971","cluster_addr":"172.21.15.193:6803/3699076971","heartbeat_back_addr":"172.21.15.193:6807/3699076971","heartbeat_front_addr":"172.21.15.193:6805/3699076971","state":["exists","up"]},{"osd":3,"uuid":"63fed43b-43ce-4031-8957-7fe94725c1ed","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":26,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6810","nonce":4059617010},{"type":"v1","addr":"172.21.15.118:6811","nonce":4059617010}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6812","nonce":4059617010},{"type":"v1","addr":"172.21.15.118:6813","nonce":4059617010}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6816","nonce":4059617010},{"type":"v1","addr":"172.21.15.118:6817","nonce":4059617010}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6814","nonce":4059617010},{"type":"v1","addr":"172.21.15.118:6815","nonce":4059617010}]},"public_addr":"172.21.15.118:6811/4059617010","cluster_addr":"172.21.15.118:6813/4059617010","heartbeat_back_addr":"172.21.15.118:6817/4059617010","heartbeat_front_addr":"172.21.15.118:6815/4059617010","state":["exists","up"]},{"osd":4,"uuid":"57377f66-c3b4-43c1-8e8e-bc873672906d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":34,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6808","nonce":4075247068},{"type":"v1","addr":"172.21.15.189:6809","nonce":4075247068}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6810","nonce":4075247068},{"type":"v1","addr":"172.21.15.189:6811","nonce":4075247068}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6814","nonce":4075247068},{"type":"v1","addr":"172.21.15.189:6815","nonce":4075247068}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6812","nonce":4075247068},{"type":"v1","addr":"172.21.15.189:6813","nonce":4075247068}]},"public_addr":"172.21.15.189:6809/4075247068","cluster_addr":"172.21.15.189:6811/4075247068","heartbeat_back_addr":"172.21.15.189:6815/4075247068","heartbeat_front_addr":"172.21.15.189:6813/4075247068","state":["exists","up"]},{"osd":5,"uuid":"91970976-2193-4224-81c0-dc96cb022c85","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":39,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6808","nonce":2920363402},{"type":"v1","addr":"172.21.15.193:6809","nonce":2920363402}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6810","nonce":2920363402},{"type":"v1","addr":"172.21.15.193:6811","nonce":2920363402}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6814","nonce":2920363402},{"type":"v1","addr":"172.21.15.193:6815","nonce":2920363402}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6812","nonce":2920363402},{"type":"v1","addr":"172.21.15.193:6813","nonce":2920363402}]},"public_addr":"172.21.15.193:6809/2920363402","cluster_addr":"172.21.15.193:6811/2920363402","heartbeat_back_addr":"172.21.15.193:6815/2920363402","heartbeat_front_addr":"172.21.15.193:6813/2920363402","state":["exists","up"]},{"osd":6,"uuid":"0b77b196-8674-41c6-ad3d-726b5d01c8fd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":47,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6818","nonce":431354576},{"type":"v1","addr":"172.21.15.118:6819","nonce":431354576}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6820","nonce":431354576},{"type":"v1","addr":"172.21.15.118:6821","nonce":431354576}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6824","nonce":431354576},{"type":"v1","addr":"172.21.15.118:6825","nonce":431354576}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6822","nonce":431354576},{"type":"v1","addr":"172.21.15.118:6823","nonce":431354576}]},"public_addr":"172.21.15.118:6819/431354576","cluster_addr":"172.21.15.118:6821/431354576","heartbeat_back_addr":"172.21.15.118:6825/431354576","heartbeat_front_addr":"172.21.15.118:6823/431354576","state":["exists","up"]},{"osd":7,"uuid":"1fbbd38f-4e8e-4d4c-8e98-66f6862bc4bd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":52,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6816","nonce":3667611284},{"type":"v1","addr":"172.21.15.189:6817","nonce":3667611284}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6818","nonce":3667611284},{"type":"v1","addr":"172.21.15.189:6819","nonce":3667611284}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6822","nonce":3667611284},{"type":"v1","addr":"172.21.15.189:6823","nonce":3667611284}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6820","nonce":3667611284},{"type":"v1","addr":"172.21.15.189:6821","nonce":3667611284}]},"public_addr":"172.21.15.189:6817/3667611284","cluster_addr":"172.21.15.189:6819/3667611284","heartbeat_back_addr":"172.21.15.189:6823/3667611284","heartbeat_front_addr":"172.21.15.189:6821/3667611284","state":["exists","up"]},{"osd":8,"uuid":"6462c2b5-e4b3-4b0b-bcfa-4de2c89e93e0","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":60,"up_thru":84,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6816","nonce":3853800910},{"type":"v1","addr":"172.21.15.193:6817","nonce":3853800910}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6818","nonce":3853800910},{"type":"v1","addr":"172.21.15.193:6819","nonce":3853800910}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6822","nonce":3853800910},{"type":"v1","addr":"172.21.15.193:6823","nonce":3853800910}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6820","nonce":3853800910},{"type":"v1","addr":"172.21.15.193:6821","nonce":3853800910}]},"public_addr":"172.21.15.193:6817/3853800910","cluster_addr":"172.21.15.193:6819/3853800910","heartbeat_back_addr":"172.21.15.193:6823/3853800910","heartbeat_front_addr":"172.21.15.193:6821/3853800910","state":["exists","up"]},{"osd":9,"uuid":"54be277f-3596-422f-9de5-9005a8571c6f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":68,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6826","nonce":3248293936},{"type":"v1","addr":"172.21.15.118:6827","nonce":3248293936}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6828","nonce":3248293936},{"type":"v1","addr":"172.21.15.118:6829","nonce":3248293936}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6832","nonce":3248293936},{"type":"v1","addr":"172.21.15.118:6833","nonce":3248293936}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.118:6830","nonce":3248293936},{"type":"v1","addr":"172.21.15.118:6831","nonce":3248293936}]},"public_addr":"172.21.15.118:6827/3248293936","cluster_addr":"172.21.15.118:6829/3248293936","heartbeat_back_addr":"172.21.15.118:6833/3248293936","heartbeat_front_addr":"172.21.15.118:6831/3248293936","state":["exists","up"]},{"osd":10,"uuid":"7ba69d69-ac41-44ce-bc39-c049a8ca5900","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":73,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6824","nonce":2112195760},{"type":"v1","addr":"172.21.15.189:6825","nonce":2112195760}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6826","nonce":2112195760},{"type":"v1","addr":"172.21.15.189:6827","nonce":2112195760}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6830","nonce":2112195760},{"type":"v1","addr":"172.21.15.189:6831","nonce":2112195760}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.189:6828","nonce":2112195760},{"type":"v1","addr":"172.21.15.189:6829","nonce":2112195760}]},"public_addr":"172.21.15.189:6825/2112195760","cluster_addr":"172.21.15.189:6827/2112195760","heartbeat_back_addr":"172.21.15.189:6831/2112195760","heartbeat_front_addr":"172.21.15.189:6829/2112195760","state":["exists","up"]},{"osd":11,"uuid":"186f0b18-6b22-4eae-83b7-b21bbcce0817","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":81,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6824","nonce":3880515510},{"type":"v1","addr":"172.21.15.193:6825","nonce":3880515510}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6826","nonce":3880515510},{"type":"v1","addr":"172.21.15.193:6827","nonce":3880515510}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6830","nonce":3880515510},{"type":"v1","addr":"172.21.15.193:6831","nonce":3880515510}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.193:6828","nonce":3880515510},{"type":"v1","addr":"172.21.15.193:6829","nonce":3880515510}]},"public_addr":"172.21.15.193:6825/3880515510","cluster_addr":"172.21.15.193:6827/3880515510","heartbeat_back_addr":"172.21.15.193:6831/3880515510","heartbeat_front_addr":"172.21.15.193:6829/3880515510","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:06:58.105543+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:07:23.795866+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:07:47.426050+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:08:19.427073+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:08:45.816082+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:09:11.081523+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:09:44.485077+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:10:11.338972+0000","dead_epoch":0},{"osd":8,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:10:36.234024+0000","dead_epoch":0},{"osd":9,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:11:09.730432+0000","dead_epoch":0},{"osd":10,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:11:37.059083+0000","dead_epoch":0},{"osd":11,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T00:12:02.371947+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[{"pgid":"2.a","mappings":[{"from":2,"to":11}]},{"pgid":"2.14","mappings":[{"from":2,"to":4}]},{"pgid":"2.1d","mappings":[{"from":8,"to":11}]},{"pgid":"2.1f","mappings":[{"from":8,"to":1}]},{"pgid":"3.11","mappings":[{"from":4,"to":8},{"from":5,"to":9}]},{"pgid":"3.13","mappings":[{"from":4,"to":9}]},{"pgid":"3.14","mappings":[{"from":4,"to":6}]},{"pgid":"3.31","mappings":[{"from":4,"to":7}]},{"pgid":"3.38","mappings":[{"from":3,"to":9}]},{"pgid":"3.39","mappings":[{"from":5,"to":8}]}],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.118:6837/179741564":"2024-03-25T00:24:41.735441+0000","172.21.15.118:6837/2587227307":"2024-03-25T00:24:58.665829+0000","172.21.15.118:6836/179741564":"2024-03-25T00:24:41.735441+0000","172.21.15.118:6835/1095918599":"2024-03-25T00:24:41.638747+0000","172.21.15.118:6834/1095918599":"2024-03-25T00:24:41.638747+0000","172.21.15.118:6835/3766636479":"2024-03-25T00:24:24.394949+0000","172.21.15.118:6834/3766636479":"2024-03-25T00:24:24.394949+0000","172.21.15.118:6836/1531167856":"2024-03-25T00:24:23.981049+0000","172.21.15.189:6836/2642988219":"2024-03-25T00:24:06.626820+0000","172.21.15.118:6835/2587469815":"2024-03-25T00:24:00.509562+0000","172.21.15.118:6834/2587469815":"2024-03-25T00:24:00.509562+0000","172.21.15.118:6839/282013400":"2024-03-25T00:23:44.005279+0000","172.21.15.118:6838/282013400":"2024-03-25T00:23:44.005279+0000","172.21.15.189:6836/635914573":"2024-03-25T00:23:24.718454+0000","172.21.15.118:6838/1710625324":"2024-03-25T00:23:08.950792+0000","172.21.15.193:6833/2424695757":"2024-03-25T00:22:51.575907+0000","172.21.15.118:6839/3600700815":"2024-03-25T00:22:26.228593+0000","172.21.15.118:6838/3600700815":"2024-03-25T00:22:26.228593+0000","172.21.15.118:6836/2587227307":"2024-03-25T00:24:58.665829+0000","172.21.15.189:6837/2350260652":"2024-03-25T00:22:09.715485+0000","172.21.15.118:6838/979323377":"2024-03-25T00:21:53.776281+0000","172.21.15.118:6835/456491655":"2024-03-25T00:24:59.499390+0000","172.21.15.193:6833/1023428412":"2024-03-25T00:20:03.007142+0000","172.21.15.193:6835/1699655293":"2024-03-25T00:20:48.266150+0000","172.21.15.193:6834/2055310541":"2024-03-25T00:19:47.282228+0000","172.21.15.118:6801/389249272":"2024-03-25T00:05:05.458796+0000","172.21.15.193:6834/3744499911":"2024-03-25T00:20:18.239231+0000","172.21.15.118:6836/3985972391":"2024-03-25T00:23:42.934077+0000","172.21.15.193:6832/3097400619":"2024-03-25T00:20:33.239209+0000","172.21.15.118:6837/3985972391":"2024-03-25T00:23:42.934077+0000","172.21.15.118:6835/3194741373":"2024-03-25T00:23:25.503800+0000","172.21.15.193:6833/4276594827":"2024-03-25T00:21:38.394987+0000","172.21.15.118:6836/1374462956":"2024-03-25T00:23:08.677886+0000","172.21.15.118:6800/389249272":"2024-03-25T00:05:05.458796+0000","172.21.15.118:6834/456491655":"2024-03-25T00:24:59.499390+0000","172.21.15.118:0/2101854694":"2024-03-25T00:05:05.458796+0000","172.21.15.193:6835/2814329894":"2024-03-25T00:21:53.247861+0000","172.21.15.118:6837/1531167856":"2024-03-25T00:24:23.981049+0000","172.21.15.189:6837/75274228":"2024-03-25T00:22:44.728301+0000","172.21.15.193:6835/2264220953":"2024-03-25T00:22:27.928314+0000","172.21.15.118:0/630779359":"2024-03-25T00:05:05.458796+0000","172.21.15.118:6834/3194741373":"2024-03-25T00:23:25.503800+0000","172.21.15.118:6837/1374462956":"2024-03-25T00:23:08.677886+0000","172.21.15.193:6832/2424695757":"2024-03-25T00:22:51.575907+0000","172.21.15.193:6832/1023428412":"2024-03-25T00:20:03.007142+0000","172.21.15.189:6836/75274228":"2024-03-25T00:22:44.728301+0000","172.21.15.189:6837/596160018":"2024-03-25T00:21:38.304407+0000","172.21.15.118:0/368712314":"2024-03-25T00:04:31.283725+0000","172.21.15.189:6837/635914573":"2024-03-25T00:23:24.718454+0000","172.21.15.193:6835/2055310541":"2024-03-25T00:19:47.282228+0000","172.21.15.118:0/2004775597":"2024-03-25T00:04:31.283725+0000","172.21.15.193:6835/3744499911":"2024-03-25T00:20:18.239231+0000","172.21.15.193:6832/2276762432":"2024-03-25T00:22:09.817146+0000","172.21.15.118:6800/1463147955":"2024-03-25T00:04:31.283725+0000","172.21.15.189:6837/2642988219":"2024-03-25T00:24:06.626820+0000","172.21.15.118:6801/1463147955":"2024-03-25T00:04:31.283725+0000","172.21.15.118:6839/3285788503":"2024-03-25T00:20:49.251252+0000","172.21.15.193:6835/799001823":"2024-03-25T00:21:21.268101+0000","172.21.15.193:6834/1699655293":"2024-03-25T00:20:48.266150+0000","172.21.15.118:6839/1710625324":"2024-03-25T00:23:08.950792+0000","172.21.15.118:0/1003670455":"2024-03-25T00:05:05.458796+0000","172.21.15.118:6838/3285788503":"2024-03-25T00:20:49.251252+0000","172.21.15.193:6834/2264220953":"2024-03-25T00:22:27.928314+0000","172.21.15.193:6832/1233062505":"2024-03-25T00:21:04.998288+0000","172.21.15.193:6833/1233062505":"2024-03-25T00:21:04.998288+0000","172.21.15.189:6836/596160018":"2024-03-25T00:21:38.304407+0000","172.21.15.118:0/2158921029":"2024-03-25T00:04:31.283725+0000","172.21.15.193:6834/2814329894":"2024-03-25T00:21:53.247861+0000","172.21.15.189:6836/2350260652":"2024-03-25T00:22:09.715485+0000","172.21.15.193:6833/3097400619":"2024-03-25T00:20:33.239209+0000","172.21.15.189:6837/3450187137":"2024-03-25T00:21:05.241581+0000","172.21.15.193:6834/799001823":"2024-03-25T00:21:21.268101+0000","172.21.15.118:6839/979323377":"2024-03-25T00:21:53.776281+0000","172.21.15.189:6836/3450187137":"2024-03-25T00:21:05.241581+0000","172.21.15.118:6838/3808689299":"2024-03-25T00:21:21.735792+0000","172.21.15.118:6839/3808689299":"2024-03-25T00:21:21.735792+0000","172.21.15.193:6833/2276762432":"2024-03-25T00:22:09.817146+0000","172.21.15.193:6832/4276594827":"2024-03-25T00:21:38.394987+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-24T03:19:40.802 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T03:19:40.802+0000 7fd3a3600640 1 -- 172.21.15.118:0/1562286803 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7fd3800a2880 msgr2=0x7fd3800a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T03:19:40.802 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T03:19:40.802+0000 7fd3a3600640 1 --2- 172.21.15.118:0/1562286803 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7fd3800a2880 0x7fd3800a4d40 secure :-1 s=READY pgs=648 cs=0 l=1 rev1=1 crypto rx=0x7fd38c005fd0 tx=0x7fd38c00b4a0 comp rx=0 tx=0).stop 2024-03-24T03:19:40.802 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T03:19:40.802+0000 7fd3a3600640 1 -- 172.21.15.118:0/1562286803 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fd39c1014a0 msgr2=0x7fd39c102400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T03:19:40.802 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T03:19:40.802+0000 7fd3a3600640 1 --2- 172.21.15.118:0/1562286803 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fd39c1014a0 0x7fd39c102400 secure :-1 s=READY pgs=483 cs=0 l=1 rev1=1 crypto rx=0x7fd39402c430 tx=0x7fd3940337b0 comp rx=0 tx=0).stop 2024-03-24T03:19:40.802 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T03:19:40.802+0000 7fd3a3600640 1 -- 172.21.15.118:0/1562286803 shutdown_connections 2024-03-24T03:19:40.802 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T03:19:40.802+0000 7fd3a3600640 1 --2- 172.21.15.118:0/1562286803 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7fd3800a2880 0x7fd3800a4d40 unknown :-1 s=CLOSED pgs=648 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T03:19:40.803 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T03:19:40.802+0000 7fd3a3600640 1 --2- 172.21.15.118:0/1562286803 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fd39c1014a0 0x7fd39c102400 unknown :-1 s=CLOSED pgs=483 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T03:19:40.803 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T03:19:40.802+0000 7fd3a3600640 1 --2- 172.21.15.118:0/1562286803 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fd39c100ae0 0x7fd39c101ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T03:19:40.803 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T03:19:40.802+0000 7fd3a3600640 1 --2- 172.21.15.118:0/1562286803 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fd39c0ff8f0 0x7fd39c103880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T03:19:40.803 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T03:19:40.802+0000 7fd3a3600640 1 -- 172.21.15.118:0/1562286803 >> 172.21.15.118:0/1562286803 conn(0x7fd39c0faea0 msgr2=0x7fd39c105f20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T03:19:40.803 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T03:19:40.802+0000 7fd3a3600640 1 -- 172.21.15.118:0/1562286803 shutdown_connections 2024-03-24T03:19:40.803 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T03:19:40.802+0000 7fd3a3600640 1 -- 172.21.15.118:0/1562286803 wait complete. 2024-03-24T03:19:40.823 DEBUG:teuthology.orchestra.run.smithi118:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2024-03-24T03:19:41.038 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T03:19:41.038+0000 7fa3dcc00640 1 -- 172.21.15.118:0/1455168046 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa3d8105490 msgr2=0x7fa3d810bd20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T03:19:41.038 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T03:19:41.038+0000 7fa3dcc00640 1 --2- 172.21.15.118:0/1455168046 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa3d8105490 0x7fa3d810bd20 secure :-1 s=READY pgs=486 cs=0 l=1 rev1=1 crypto rx=0x7fa3d0009ca0 tx=0x7fa3d002c0f0 comp rx=0 tx=0).stop 2024-03-24T03:19:41.038 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T03:19:41.038+0000 7fa3dcc00640 1 -- 172.21.15.118:0/1455168046 shutdown_connections 2024-03-24T03:19:41.038 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T03:19:41.038+0000 7fa3dcc00640 1 --2- 172.21.15.118:0/1455168046 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa3d8105490 0x7fa3d810bd20 unknown :-1 s=CLOSED pgs=486 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T03:19:41.039 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T03:19:41.038+0000 7fa3dcc00640 1 --2- 172.21.15.118:0/1455168046 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fa3d8104ad0 0x7fa3d8104f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T03:19:41.039 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T03:19:41.038+0000 7fa3dcc00640 1 --2- 172.21.15.118:0/1455168046 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fa3d81038e0 0x7fa3d8103d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T03:19:41.039 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T03:19:41.038+0000 7fa3dcc00640 1 -- 172.21.15.118:0/1455168046 >> 172.21.15.118:0/1455168046 conn(0x7fa3d80fee50 msgr2=0x7fa3d8101290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T03:19:41.039 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T03:19:41.038+0000 7fa3dcc00640 1 -- 172.21.15.118:0/1455168046 shutdown_connections 2024-03-24T03:19:41.039 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T03:19:41.038+0000 7fa3dcc00640 1 -- 172.21.15.118:0/1455168046 wait complete. 2024-03-24T03:19:41.041 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T03:19:41.039+0000 7fa3dcc00640 1 Processor -- start 2024-03-24T03:19:41.041 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T03:19:41.039+0000 7fa3dcc00640 1 -- start start 2024-03-24T03:19:41.041 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T03:19:41.040+0000 7fa3dcc00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fa3d81038e0 0x7fa3d8199b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T03:19:41.041 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T03:19:41.040+0000 7fa3dcc00640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fa3d8104ad0 0x7fa3d819a0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T03:19:41.042 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T03:19:41.040+0000 7fa3dcc00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa3d8105490 0x7fa3d81a1130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T03:19:41.042 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T03:19:41.040+0000 7fa3dcc00640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa3d810f540 con 0x7fa3d8104ad0 2024-03-24T03:19:41.042 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T03:19:41.040+0000 7fa3dcc00640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa3d810f3c0 con 0x7fa3d8105490 2024-03-24T03:19:41.042 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T03:19:41.040+0000 7fa3dcc00640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa3d810f6c0 con 0x7fa3d81038e0 2024-03-24T03:19:41.042 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T03:19:41.040+0000 7fa3d5e00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fa3d81038e0 0x7fa3d8199b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T03:19:41.042 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T03:19:41.040+0000 7fa3d5e00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fa3d81038e0 0x7fa3d8199b80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.118:47972/0 (socket says 172.21.15.118:47972) 2024-03-24T03:19:41.042 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T03:19:41.040+0000 7fa3d5e00640 1 -- 172.21.15.118:0/3676738978 learned_addr learned my addr 172.21.15.118:0/3676738978 (peer_addr_for_me v2:172.21.15.118:0/0) 2024-03-24T03:19:41.042 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T03:19:41.040+0000 7fa3d5400640 1 --2- 172.21.15.118:0/3676738978 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fa3d8104ad0 0x7fa3d819a0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T03:19:41.042 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T03:19:41.040+0000 7fa3d6800640 1 --2- 172.21.15.118:0/3676738978 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa3d8105490 0x7fa3d81a1130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T03:19:41.043 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T03:19:41.040+0000 7fa3d5e00640 1 -- 172.21.15.118:0/3676738978 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa3d8105490 msgr2=0x7fa3d81a1130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T03:19:41.043 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T03:19:41.040+0000 7fa3d5e00640 1 --2- 172.21.15.118:0/3676738978 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa3d8105490 0x7fa3d81a1130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T03:19:41.043 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T03:19:41.040+0000 7fa3d5e00640 1 -- 172.21.15.118:0/3676738978 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fa3d8104ad0 msgr2=0x7fa3d819a0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T03:19:41.043 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T03:19:41.040+0000 7fa3d5e00640 1 --2- 172.21.15.118:0/3676738978 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fa3d8104ad0 0x7fa3d819a0c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T03:19:41.043 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T03:19:41.040+0000 7fa3d5e00640 1 -- 172.21.15.118:0/3676738978 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa3d81a17a0 con 0x7fa3d81038e0 2024-03-24T03:19:41.043 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T03:19:41.041+0000 7fa3d5e00640 1 --2- 172.21.15.118:0/3676738978 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fa3d81038e0 0x7fa3d8199b80 secure :-1 s=READY pgs=484 cs=0 l=1 rev1=1 crypto rx=0x7fa3c800e9c0 tx=0x7fa3c800ee80 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T03:19:41.043 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T03:19:41.041+0000 7fa3cea00640 1 -- 172.21.15.118:0/3676738978 <== mon.1 v2:172.21.15.193:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa3c800cd80 con 0x7fa3d81038e0 2024-03-24T03:19:41.043 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T03:19:41.041+0000 7fa3cea00640 1 -- 172.21.15.118:0/3676738978 <== mon.1 v2:172.21.15.193:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7fa3c80035f0 con 0x7fa3d81038e0 2024-03-24T03:19:41.044 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T03:19:41.041+0000 7fa3cea00640 1 -- 172.21.15.118:0/3676738978 <== mon.1 v2:172.21.15.193:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7fa3c8010690 con 0x7fa3d81038e0 2024-03-24T03:19:41.044 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T03:19:41.041+0000 7fa3dcc00640 1 -- 172.21.15.118:0/3676738978 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa3d81a1a90 con 0x7fa3d81038e0 2024-03-24T03:19:41.044 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T03:19:41.041+0000 7fa3dcc00640 1 -- 172.21.15.118:0/3676738978 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa3d81a1f90 con 0x7fa3d81038e0 2024-03-24T03:19:41.044 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T03:19:41.042+0000 7fa3dcc00640 1 -- 172.21.15.118:0/3676738978 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa3d8068490 con 0x7fa3d81038e0 2024-03-24T03:19:41.045 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T03:19:41.044+0000 7fa3cea00640 1 -- 172.21.15.118:0/3676738978 <== mon.1 v2:172.21.15.193:3300/0 4 ==== mgrmap(e 20) v1 ==== 137291+0+0 (secure 0 0 0) 0x7fa3c80037a0 con 0x7fa3d81038e0 2024-03-24T03:19:41.045 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T03:19:41.045+0000 7fa3cea00640 1 --2- 172.21.15.118:0/3676738978 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7fa3bc0a2800 0x7fa3bc0a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T03:19:41.045 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T03:19:41.045+0000 7fa3cea00640 1 -- 172.21.15.118:0/3676738978 <== mon.1 v2:172.21.15.193:3300/0 5 ==== osd_map(124..124 src has 1..124) v4 ==== 11201+0+0 (secure 0 0 0) 0x7fa3c80cf9b0 con 0x7fa3d81038e0 2024-03-24T03:19:41.045 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T03:19:41.046+0000 7fa3d5400640 1 --2- 172.21.15.118:0/3676738978 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7fa3bc0a2800 0x7fa3bc0a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T03:19:41.046 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T03:19:41.046+0000 7fa3d5400640 1 --2- 172.21.15.118:0/3676738978 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7fa3bc0a2800 0x7fa3bc0a4cc0 secure :-1 s=READY pgs=649 cs=0 l=1 rev1=1 crypto rx=0x7fa3d819b090 tx=0x7fa3c0005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T03:19:41.049 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T03:19:41.049+0000 7fa3cea00640 1 -- 172.21.15.118:0/3676738978 <== mon.1 v2:172.21.15.193:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7fa3c809beb0 con 0x7fa3d81038e0 2024-03-24T03:19:41.287 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T03:19:41.288+0000 7fa3dcc00640 1 -- 172.21.15.118:0/3676738978 --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) v1 -- 0x7fa3d810ac00 con 0x7fa3d81038e0 2024-03-24T03:19:41.297 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T03:19:41.297+0000 7fa3cea00640 1 -- 172.21.15.118:0/3676738978 <== mon.1 v2:172.21.15.193:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v189) v1 ==== 161+0+0 (secure 0 0 0) 0x7fa3c809b7b0 con 0x7fa3d81038e0 2024-03-24T03:19:41.297 INFO:teuthology.orchestra.run.smithi118.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2024-03-24T03:19:41.300 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T03:19:41.301+0000 7fa3dcc00640 1 -- 172.21.15.118:0/3676738978 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7fa3bc0a2800 msgr2=0x7fa3bc0a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T03:19:41.300 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T03:19:41.301+0000 7fa3dcc00640 1 --2- 172.21.15.118:0/3676738978 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7fa3bc0a2800 0x7fa3bc0a4cc0 secure :-1 s=READY pgs=649 cs=0 l=1 rev1=1 crypto rx=0x7fa3d819b090 tx=0x7fa3c0005f90 comp rx=0 tx=0).stop 2024-03-24T03:19:41.301 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T03:19:41.301+0000 7fa3dcc00640 1 -- 172.21.15.118:0/3676738978 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fa3d81038e0 msgr2=0x7fa3d8199b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T03:19:41.301 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T03:19:41.301+0000 7fa3dcc00640 1 --2- 172.21.15.118:0/3676738978 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fa3d81038e0 0x7fa3d8199b80 secure :-1 s=READY pgs=484 cs=0 l=1 rev1=1 crypto rx=0x7fa3c800e9c0 tx=0x7fa3c800ee80 comp rx=0 tx=0).stop 2024-03-24T03:19:41.301 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T03:19:41.301+0000 7fa3dcc00640 1 -- 172.21.15.118:0/3676738978 shutdown_connections 2024-03-24T03:19:41.301 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T03:19:41.301+0000 7fa3dcc00640 1 --2- 172.21.15.118:0/3676738978 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7fa3bc0a2800 0x7fa3bc0a4cc0 unknown :-1 s=CLOSED pgs=649 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T03:19:41.301 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T03:19:41.301+0000 7fa3dcc00640 1 --2- 172.21.15.118:0/3676738978 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7fa3d8105490 0x7fa3d81a1130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T03:19:41.301 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T03:19:41.301+0000 7fa3dcc00640 1 --2- 172.21.15.118:0/3676738978 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7fa3d8104ad0 0x7fa3d819a0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T03:19:41.301 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T03:19:41.301+0000 7fa3dcc00640 1 --2- 172.21.15.118:0/3676738978 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7fa3d81038e0 0x7fa3d8199b80 unknown :-1 s=CLOSED pgs=484 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T03:19:41.301 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T03:19:41.301+0000 7fa3dcc00640 1 -- 172.21.15.118:0/3676738978 >> 172.21.15.118:0/3676738978 conn(0x7fa3d80fee50 msgr2=0x7fa3d8107dd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T03:19:41.301 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T03:19:41.301+0000 7fa3dcc00640 1 -- 172.21.15.118:0/3676738978 shutdown_connections 2024-03-24T03:19:41.302 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T03:19:41.301+0000 7fa3dcc00640 1 -- 172.21.15.118:0/3676738978 wait complete. 2024-03-24T03:19:41.321 DEBUG:teuthology.orchestra.run.smithi118:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs rm cephfs --yes-i-really-mean-it 2024-03-24T03:19:41.507 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T03:19:41.508+0000 7f45fa800640 1 -- 172.21.15.118:0/1830120499 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f45f4102770 msgr2=0x7f45f4104be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T03:19:41.508 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T03:19:41.508+0000 7f45fa800640 1 --2- 172.21.15.118:0/1830120499 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f45f4102770 0x7f45f4104be0 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7f45e0009900 tx=0x7f45e002bc30 comp rx=0 tx=0).stop 2024-03-24T03:19:41.508 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T03:19:41.508+0000 7f45fa800640 1 -- 172.21.15.118:0/1830120499 shutdown_connections 2024-03-24T03:19:41.508 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T03:19:41.508+0000 7f45fa800640 1 --2- 172.21.15.118:0/1830120499 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f45f4105120 0x7f45f41055a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T03:19:41.508 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T03:19:41.508+0000 7f45fa800640 1 --2- 172.21.15.118:0/1830120499 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f45f4102770 0x7f45f4104be0 unknown :-1 s=CLOSED pgs=239 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T03:19:41.508 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T03:19:41.508+0000 7f45fa800640 1 --2- 172.21.15.118:0/1830120499 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f45f40ffe00 0x7f45f4102230 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T03:19:41.508 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T03:19:41.508+0000 7f45fa800640 1 -- 172.21.15.118:0/1830120499 >> 172.21.15.118:0/1830120499 conn(0x7f45f40fba20 msgr2=0x7f45f40fde40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T03:19:41.508 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T03:19:41.508+0000 7f45fa800640 1 -- 172.21.15.118:0/1830120499 shutdown_connections 2024-03-24T03:19:41.508 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T03:19:41.508+0000 7f45fa800640 1 -- 172.21.15.118:0/1830120499 wait complete. 2024-03-24T03:19:41.510 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T03:19:41.509+0000 7f45fa800640 1 Processor -- start 2024-03-24T03:19:41.511 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T03:19:41.509+0000 7f45fa800640 1 -- start start 2024-03-24T03:19:41.511 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T03:19:41.509+0000 7f45fa800640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f45f40ffe00 0x7f45f419de70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T03:19:41.511 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T03:19:41.509+0000 7f45fa800640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f45f4102770 0x7f45f419e3b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T03:19:41.511 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T03:19:41.509+0000 7f45fa800640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f45f4105120 0x7f45f41a5420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T03:19:41.511 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T03:19:41.509+0000 7f45fa800640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f45f4113830 con 0x7f45f40ffe00 2024-03-24T03:19:41.511 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T03:19:41.509+0000 7f45fa800640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f45f41136b0 con 0x7f45f4102770 2024-03-24T03:19:41.511 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T03:19:41.509+0000 7f45fa800640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f45f41139b0 con 0x7f45f4105120 2024-03-24T03:19:41.512 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T03:19:41.510+0000 7f45f9400640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f45f40ffe00 0x7f45f419de70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T03:19:41.512 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T03:19:41.510+0000 7f45f8a00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f45f4102770 0x7f45f419e3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T03:19:41.512 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T03:19:41.510+0000 7f45f9e00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f45f4105120 0x7f45f41a5420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T03:19:41.512 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T03:19:41.510+0000 7f45f8a00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f45f4102770 0x7f45f419e3b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.118:37528/0 (socket says 172.21.15.118:37528) 2024-03-24T03:19:41.512 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T03:19:41.510+0000 7f45f8a00640 1 -- 172.21.15.118:0/1686438014 learned_addr learned my addr 172.21.15.118:0/1686438014 (peer_addr_for_me v2:172.21.15.118:0/0) 2024-03-24T03:19:41.512 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T03:19:41.510+0000 7f45f8a00640 1 -- 172.21.15.118:0/1686438014 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f45f4105120 msgr2=0x7f45f41a5420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T03:19:41.512 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T03:19:41.510+0000 7f45f8a00640 1 --2- 172.21.15.118:0/1686438014 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f45f4105120 0x7f45f41a5420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T03:19:41.512 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T03:19:41.510+0000 7f45f8a00640 1 -- 172.21.15.118:0/1686438014 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f45f40ffe00 msgr2=0x7f45f419de70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T03:19:41.512 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T03:19:41.510+0000 7f45f8a00640 1 --2- 172.21.15.118:0/1686438014 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f45f40ffe00 0x7f45f419de70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T03:19:41.513 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T03:19:41.510+0000 7f45f8a00640 1 -- 172.21.15.118:0/1686438014 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f45f41a5a90 con 0x7f45f4102770 2024-03-24T03:19:41.513 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T03:19:41.510+0000 7f45f9400640 1 --2- 172.21.15.118:0/1686438014 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f45f40ffe00 0x7f45f419de70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T03:19:41.513 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T03:19:41.510+0000 7f45f8a00640 1 --2- 172.21.15.118:0/1686438014 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f45f4102770 0x7f45f419e3b0 secure :-1 s=READY pgs=492 cs=0 l=1 rev1=1 crypto rx=0x7f45e0000c00 tx=0x7f45e002c650 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T03:19:41.513 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T03:19:41.511+0000 7f45f2000640 1 -- 172.21.15.118:0/1686438014 <== mon.2 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f45e0043020 con 0x7f45f4102770 2024-03-24T03:19:41.513 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T03:19:41.511+0000 7f45f2000640 1 -- 172.21.15.118:0/1686438014 <== mon.2 v2:172.21.15.189:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f45e00054e0 con 0x7f45f4102770 2024-03-24T03:19:41.513 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T03:19:41.511+0000 7f45f2000640 1 -- 172.21.15.118:0/1686438014 <== mon.2 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f45e0003bd0 con 0x7f45f4102770 2024-03-24T03:19:41.513 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T03:19:41.511+0000 7f45fa800640 1 -- 172.21.15.118:0/1686438014 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f45f41a5d20 con 0x7f45f4102770 2024-03-24T03:19:41.513 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T03:19:41.511+0000 7f45fa800640 1 -- 172.21.15.118:0/1686438014 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f45f41a6260 con 0x7f45f4102770 2024-03-24T03:19:41.514 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T03:19:41.512+0000 7f45fa800640 1 -- 172.21.15.118:0/1686438014 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f45f40fd510 con 0x7f45f4102770 2024-03-24T03:19:41.514 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T03:19:41.515+0000 7f45f2000640 1 -- 172.21.15.118:0/1686438014 <== mon.2 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 20) v1 ==== 137291+0+0 (secure 0 0 0) 0x7f45e0046020 con 0x7f45f4102770 2024-03-24T03:19:41.515 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T03:19:41.516+0000 7f45f2000640 1 --2- 172.21.15.118:0/1686438014 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7f45d80a2880 0x7f45d80a4d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T03:19:41.516 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T03:19:41.516+0000 7f45f2000640 1 -- 172.21.15.118:0/1686438014 <== mon.2 v2:172.21.15.189:3300/0 5 ==== osd_map(125..125 src has 1..125) v4 ==== 11631+0+0 (secure 0 0 0) 0x7f45e00f17e0 con 0x7f45f4102770 2024-03-24T03:19:41.516 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T03:19:41.516+0000 7f45f9400640 1 --2- 172.21.15.118:0/1686438014 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7f45d80a2880 0x7f45d80a4d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T03:19:41.516 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T03:19:41.517+0000 7f45f9400640 1 --2- 172.21.15.118:0/1686438014 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7f45d80a2880 0x7f45d80a4d40 secure :-1 s=READY pgs=650 cs=0 l=1 rev1=1 crypto rx=0x7f45e40095f0 tx=0x7f45e4005f90 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T03:19:41.519 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T03:19:41.520+0000 7f45f2000640 1 -- 172.21.15.118:0/1686438014 <== mon.2 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f45e004d020 con 0x7f45f4102770 2024-03-24T03:19:41.696 INFO:journalctl@ceph.mds.a.smithi118.stdout:Mar 24 03:19:41 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-a[73967]: ignoring --setuser ceph since I am not root 2024-03-24T03:19:41.696 INFO:journalctl@ceph.mds.a.smithi118.stdout:Mar 24 03:19:41 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-a[73967]: ignoring --setgroup ceph since I am not root 2024-03-24T03:19:41.696 INFO:journalctl@ceph.mds.a.smithi118.stdout:Mar 24 03:19:41 smithi118 ceph-mds[73988]: -- 172.21.15.118:0/4148345677 <== mon.2 v2:172.21.15.189:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x55b47aa37a40 con 0x55b47aa0ac00 2024-03-24T03:19:41.696 INFO:journalctl@ceph.mds.a.smithi118.stdout:Mar 24 03:19:41 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-a[73967]: starting mds.a at 2024-03-24T03:19:41.697 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 03:19:41 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mon-a[31531]: 2024-03-24T03:19:41.290+0000 7f2d67e00700 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-03-24T03:19:41.697 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 03:19:41 smithi118 ceph-mon[31535]: 3 slow requests, 0 included below; oldest blocked for > 10476.763297 secs 2024-03-24T03:19:41.697 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 03:19:41 smithi118 ceph-mon[31535]: from='client.? 172.21.15.118:0/4119823497' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-24T03:19:41.697 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 03:19:41 smithi118 ceph-mon[31535]: pgmap v6169: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-03-24T03:19:41.697 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 03:19:41 smithi118 ceph-mon[31535]: from='client.? 172.21.15.118:0/1562286803' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-24T03:19:41.697 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 03:19:41 smithi118 ceph-mon[31535]: from='client.? 172.21.15.118:0/3676738978' entity='client.admin' cmd=[{"prefix": "fs fail", "fs_name": "cephfs"}]: dispatch 2024-03-24T03:19:41.697 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 03:19:41 smithi118 ceph-mon[31535]: from='client.? ' entity='client.admin' cmd=[{"prefix": "fs fail", "fs_name": "cephfs"}]: dispatch 2024-03-24T03:19:41.697 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 03:19:41 smithi118 ceph-mon[31535]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-03-24T03:19:41.698 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 03:19:41 smithi118 ceph-mon[31535]: Health check failed: 1 filesystem has a failed mds daemon (FS_WITH_FAILED_MDS) 2024-03-24T03:19:41.698 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 03:19:41 smithi118 ceph-mon[31535]: Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-03-24T03:19:41.698 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 03:19:41 smithi118 ceph-mon[31535]: Health check cleared: MDS_SLOW_REQUEST (was: 1 MDSs report slow requests) 2024-03-24T03:19:41.698 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 03:19:41 smithi118 ceph-mon[31535]: from='client.? ' entity='client.admin' cmd='[{"prefix": "fs fail", "fs_name": "cephfs"}]': finished 2024-03-24T03:19:41.698 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 03:19:41 smithi118 ceph-mon[31535]: osdmap e125: 12 total, 12 up, 12 in 2024-03-24T03:19:41.698 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 03:19:41 smithi118 ceph-mon[31535]: fsmap cephfs:0/5, 5 failed 2024-03-24T03:19:41.703 INFO:journalctl@ceph.mds.e.smithi189.stdout:Mar 24 03:19:41 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-e[66361]: ignoring --setuser ceph since I am not root 2024-03-24T03:19:41.703 INFO:journalctl@ceph.mds.e.smithi189.stdout:Mar 24 03:19:41 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-e[66361]: ignoring --setgroup ceph since I am not root 2024-03-24T03:19:41.703 INFO:journalctl@ceph.mds.e.smithi189.stdout:Mar 24 03:19:41 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-e[66361]: starting mds.e at 2024-03-24T03:19:41.704 INFO:journalctl@ceph.mds.k.smithi189.stdout:Mar 24 03:19:41 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-k[66038]: ignoring --setuser ceph since I am not root 2024-03-24T03:19:41.704 INFO:journalctl@ceph.mds.k.smithi189.stdout:Mar 24 03:19:41 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-k[66038]: ignoring --setgroup ceph since I am not root 2024-03-24T03:19:41.704 INFO:journalctl@ceph.mds.k.smithi189.stdout:Mar 24 03:19:41 smithi189 ceph-mds[66042]: -- 172.21.15.189:0/776075971 <== mon.1 v2:172.21.15.193:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x55a1c93afa40 con 0x55a1c9383000 2024-03-24T03:19:41.704 INFO:journalctl@ceph.mds.k.smithi189.stdout:Mar 24 03:19:41 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-k[66038]: starting mds.k at 2024-03-24T03:19:41.705 INFO:journalctl@ceph.mds.b.smithi189.stdout:Mar 24 03:19:41 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-b[66975]: ignoring --setuser ceph since I am not root 2024-03-24T03:19:41.705 INFO:journalctl@ceph.mds.b.smithi189.stdout:Mar 24 03:19:41 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-b[66975]: ignoring --setgroup ceph since I am not root 2024-03-24T03:19:41.705 INFO:journalctl@ceph.mds.b.smithi189.stdout:Mar 24 03:19:41 smithi189 ceph-mds[66979]: -- 172.21.15.189:0/1383522568 <== mon.2 v2:172.21.15.189:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x562d86f8ba40 con 0x562d86f5e800 2024-03-24T03:19:41.705 INFO:journalctl@ceph.mds.b.smithi189.stdout:Mar 24 03:19:41 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-b[66975]: starting mds.b at 2024-03-24T03:19:41.705 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 03:19:41 smithi189 ceph-mon[36187]: 3 slow requests, 0 included below; oldest blocked for > 10476.763297 secs 2024-03-24T03:19:41.705 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 03:19:41 smithi189 ceph-mon[36187]: from='client.? 172.21.15.118:0/4119823497' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-24T03:19:41.705 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 03:19:41 smithi189 ceph-mon[36187]: pgmap v6169: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-03-24T03:19:41.706 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 03:19:41 smithi189 ceph-mon[36187]: from='client.? 172.21.15.118:0/1562286803' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-24T03:19:41.706 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 03:19:41 smithi189 ceph-mon[36187]: from='client.? 172.21.15.118:0/3676738978' entity='client.admin' cmd=[{"prefix": "fs fail", "fs_name": "cephfs"}]: dispatch 2024-03-24T03:19:41.706 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 03:19:41 smithi189 ceph-mon[36187]: from='client.? ' entity='client.admin' cmd=[{"prefix": "fs fail", "fs_name": "cephfs"}]: dispatch 2024-03-24T03:19:41.706 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 03:19:41 smithi189 ceph-mon[36187]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-03-24T03:19:41.706 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 03:19:41 smithi189 ceph-mon[36187]: Health check failed: 1 filesystem has a failed mds daemon (FS_WITH_FAILED_MDS) 2024-03-24T03:19:41.706 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 03:19:41 smithi189 ceph-mon[36187]: Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-03-24T03:19:41.706 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 03:19:41 smithi189 ceph-mon[36187]: Health check cleared: MDS_SLOW_REQUEST (was: 1 MDSs report slow requests) 2024-03-24T03:19:41.706 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 03:19:41 smithi189 ceph-mon[36187]: from='client.? ' entity='client.admin' cmd='[{"prefix": "fs fail", "fs_name": "cephfs"}]': finished 2024-03-24T03:19:41.707 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 03:19:41 smithi189 ceph-mon[36187]: osdmap e125: 12 total, 12 up, 12 in 2024-03-24T03:19:41.707 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 03:19:41 smithi189 ceph-mon[36187]: fsmap cephfs:0/5, 5 failed 2024-03-24T03:19:41.710 INFO:journalctl@ceph.mds.c.smithi193.stdout:Mar 24 03:19:41 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-c[67774]: ignoring --setuser ceph since I am not root 2024-03-24T03:19:41.710 INFO:journalctl@ceph.mds.c.smithi193.stdout:Mar 24 03:19:41 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-c[67774]: ignoring --setgroup ceph since I am not root 2024-03-24T03:19:41.711 INFO:journalctl@ceph.mds.c.smithi193.stdout:Mar 24 03:19:41 smithi193 ceph-mds[67778]: -- 172.21.15.193:0/663375659 <== mon.1 v2:172.21.15.193:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x55e578211a40 con 0x55e5781e4800 2024-03-24T03:19:41.711 INFO:journalctl@ceph.mds.c.smithi193.stdout:Mar 24 03:19:41 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-c[67774]: starting mds.c at 2024-03-24T03:19:41.711 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 03:19:41 smithi193 ceph-mon[36878]: 3 slow requests, 0 included below; oldest blocked for > 10476.763297 secs 2024-03-24T03:19:41.711 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 03:19:41 smithi193 ceph-mon[36878]: from='client.? 172.21.15.118:0/4119823497' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-24T03:19:41.711 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 03:19:41 smithi193 ceph-mon[36878]: pgmap v6169: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 3.2 KiB/s rd, 6 op/s 2024-03-24T03:19:41.711 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 03:19:41 smithi193 ceph-mon[36878]: from='client.? 172.21.15.118:0/1562286803' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-24T03:19:41.711 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 03:19:41 smithi193 ceph-mon[36878]: from='client.? 172.21.15.118:0/3676738978' entity='client.admin' cmd=[{"prefix": "fs fail", "fs_name": "cephfs"}]: dispatch 2024-03-24T03:19:41.712 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 03:19:41 smithi193 ceph-mon[36878]: from='client.? ' entity='client.admin' cmd=[{"prefix": "fs fail", "fs_name": "cephfs"}]: dispatch 2024-03-24T03:19:41.712 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 03:19:41 smithi193 ceph-mon[36878]: Health check failed: 1 filesystem is degraded (FS_DEGRADED) 2024-03-24T03:19:41.712 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 03:19:41 smithi193 ceph-mon[36878]: Health check failed: 1 filesystem has a failed mds daemon (FS_WITH_FAILED_MDS) 2024-03-24T03:19:41.712 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 03:19:41 smithi193 ceph-mon[36878]: Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-03-24T03:19:41.712 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 03:19:41 smithi193 ceph-mon[36878]: Health check cleared: MDS_SLOW_REQUEST (was: 1 MDSs report slow requests) 2024-03-24T03:19:41.712 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 03:19:41 smithi193 ceph-mon[36878]: from='client.? ' entity='client.admin' cmd='[{"prefix": "fs fail", "fs_name": "cephfs"}]': finished 2024-03-24T03:19:41.712 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 03:19:41 smithi193 ceph-mon[36878]: osdmap e125: 12 total, 12 up, 12 in 2024-03-24T03:19:41.712 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 03:19:41 smithi193 ceph-mon[36878]: fsmap cephfs:0/5, 5 failed 2024-03-24T03:19:41.713 INFO:journalctl@ceph.mds.i.smithi193.stdout:Mar 24 03:19:41 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-i[68088]: ignoring --setuser ceph since I am not root 2024-03-24T03:19:41.713 INFO:journalctl@ceph.mds.i.smithi193.stdout:Mar 24 03:19:41 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-i[68088]: ignoring --setgroup ceph since I am not root 2024-03-24T03:19:41.713 INFO:journalctl@ceph.mds.i.smithi193.stdout:Mar 24 03:19:41 smithi193 ceph-mds[68092]: -- 172.21.15.193:0/4189859909 <== mon.1 v2:172.21.15.193:3300/0 4 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 194+0+0 (secure 0 0 0) 0x55e77eaefa40 con 0x55e77eac2c00 2024-03-24T03:19:41.713 INFO:journalctl@ceph.mds.i.smithi193.stdout:Mar 24 03:19:41 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-i[68088]: starting mds.i at 2024-03-24T03:19:41.758 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T03:19:41.758+0000 7f45fa800640 1 -- 172.21.15.118:0/1686438014 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true} v 0) v1 -- 0x7f45f40622a0 con 0x7f45f4102770 2024-03-24T03:19:42.299 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T03:19:42.300+0000 7f45f2000640 1 -- 172.21.15.118:0/1686438014 <== mon.2 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]=0 v190) v1 ==== 104+0+0 (secure 0 0 0) 0x7f45e00bdae0 con 0x7f45f4102770 2024-03-24T03:19:42.304 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T03:19:42.304+0000 7f45fa800640 1 -- 172.21.15.118:0/1686438014 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7f45d80a2880 msgr2=0x7f45d80a4d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T03:19:42.304 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T03:19:42.304+0000 7f45fa800640 1 --2- 172.21.15.118:0/1686438014 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7f45d80a2880 0x7f45d80a4d40 secure :-1 s=READY pgs=650 cs=0 l=1 rev1=1 crypto rx=0x7f45e40095f0 tx=0x7f45e4005f90 comp rx=0 tx=0).stop 2024-03-24T03:19:42.304 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T03:19:42.304+0000 7f45fa800640 1 -- 172.21.15.118:0/1686438014 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f45f4102770 msgr2=0x7f45f419e3b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T03:19:42.304 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T03:19:42.304+0000 7f45fa800640 1 --2- 172.21.15.118:0/1686438014 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f45f4102770 0x7f45f419e3b0 secure :-1 s=READY pgs=492 cs=0 l=1 rev1=1 crypto rx=0x7f45e0000c00 tx=0x7f45e002c650 comp rx=0 tx=0).stop 2024-03-24T03:19:42.304 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T03:19:42.304+0000 7f45fa800640 1 -- 172.21.15.118:0/1686438014 shutdown_connections 2024-03-24T03:19:42.304 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T03:19:42.304+0000 7f45fa800640 1 --2- 172.21.15.118:0/1686438014 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7f45d80a2880 0x7f45d80a4d40 unknown :-1 s=CLOSED pgs=650 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T03:19:42.304 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T03:19:42.304+0000 7f45fa800640 1 --2- 172.21.15.118:0/1686438014 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f45f4105120 0x7f45f41a5420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T03:19:42.305 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T03:19:42.304+0000 7f45fa800640 1 --2- 172.21.15.118:0/1686438014 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f45f4102770 0x7f45f419e3b0 unknown :-1 s=CLOSED pgs=492 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T03:19:42.305 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T03:19:42.304+0000 7f45fa800640 1 --2- 172.21.15.118:0/1686438014 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f45f40ffe00 0x7f45f419de70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T03:19:42.305 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T03:19:42.304+0000 7f45fa800640 1 -- 172.21.15.118:0/1686438014 >> 172.21.15.118:0/1686438014 conn(0x7f45f40fba20 msgr2=0x7f45f4100780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T03:19:42.305 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T03:19:42.305+0000 7f45fa800640 1 -- 172.21.15.118:0/1686438014 shutdown_connections 2024-03-24T03:19:42.306 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T03:19:42.305+0000 7f45fa800640 1 -- 172.21.15.118:0/1686438014 wait complete. 2024-03-24T03:19:42.326 DEBUG:teuthology.orchestra.run.smithi118:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_metadata cephfs_metadata --yes-i-really-really-mean-it 2024-03-24T03:19:42.539 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T03:19:42.539+0000 7f3393800640 1 -- 172.21.15.118:0/57224536 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f338c1038e0 msgr2=0x7f338c103d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T03:19:42.539 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T03:19:42.539+0000 7f3393800640 1 --2- 172.21.15.118:0/57224536 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f338c1038e0 0x7f338c103d00 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7f3380009900 tx=0x7f338002bc30 comp rx=0 tx=0).stop 2024-03-24T03:19:42.539 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T03:19:42.539+0000 7f3393800640 1 -- 172.21.15.118:0/57224536 shutdown_connections 2024-03-24T03:19:42.539 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T03:19:42.539+0000 7f3393800640 1 --2- 172.21.15.118:0/57224536 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f338c105490 0x7f338c10bd20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T03:19:42.539 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T03:19:42.539+0000 7f3393800640 1 --2- 172.21.15.118:0/57224536 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f338c104ad0 0x7f338c104f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T03:19:42.540 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T03:19:42.539+0000 7f3393800640 1 --2- 172.21.15.118:0/57224536 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f338c1038e0 0x7f338c103d00 unknown :-1 s=CLOSED pgs=240 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T03:19:42.540 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T03:19:42.539+0000 7f3393800640 1 -- 172.21.15.118:0/57224536 >> 172.21.15.118:0/57224536 conn(0x7f338c0fee50 msgr2=0x7f338c101290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T03:19:42.540 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T03:19:42.539+0000 7f3393800640 1 -- 172.21.15.118:0/57224536 shutdown_connections 2024-03-24T03:19:42.540 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T03:19:42.539+0000 7f3393800640 1 -- 172.21.15.118:0/57224536 wait complete. 2024-03-24T03:19:42.542 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T03:19:42.540+0000 7f3393800640 1 Processor -- start 2024-03-24T03:19:42.542 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T03:19:42.540+0000 7f3393800640 1 -- start start 2024-03-24T03:19:42.542 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T03:19:42.540+0000 7f3393800640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f338c1038e0 0x7f338c199c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T03:19:42.543 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T03:19:42.541+0000 7f3393800640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f338c104ad0 0x7f338c19a1b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T03:19:42.543 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T03:19:42.541+0000 7f3393800640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f338c105490 0x7f338c1a11e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T03:19:42.543 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T03:19:42.541+0000 7f3393800640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f338c10f530 con 0x7f338c105490 2024-03-24T03:19:42.543 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T03:19:42.541+0000 7f3393800640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f338c10f3b0 con 0x7f338c1038e0 2024-03-24T03:19:42.543 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T03:19:42.541+0000 7f3393800640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f338c10f6b0 con 0x7f338c104ad0 2024-03-24T03:19:42.543 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T03:19:42.541+0000 7f3391800640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f338c105490 0x7f338c1a11e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T03:19:42.543 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T03:19:42.541+0000 7f3390e00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f338c1038e0 0x7f338c199c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T03:19:42.543 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T03:19:42.541+0000 7f3390e00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f338c1038e0 0x7f338c199c70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.118:37546/0 (socket says 172.21.15.118:37546) 2024-03-24T03:19:42.543 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T03:19:42.541+0000 7f3390e00640 1 -- 172.21.15.118:0/3018692129 learned_addr learned my addr 172.21.15.118:0/3018692129 (peer_addr_for_me v2:172.21.15.118:0/0) 2024-03-24T03:19:42.544 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T03:19:42.541+0000 7f3391800640 1 --2- 172.21.15.118:0/3018692129 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f338c105490 0x7f338c1a11e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.118:3300/0 says I am v2:172.21.15.118:37088/0 (socket says 172.21.15.118:37088) 2024-03-24T03:19:42.544 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T03:19:42.541+0000 7f338be00640 1 --2- 172.21.15.118:0/3018692129 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f338c104ad0 0x7f338c19a1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T03:19:42.544 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T03:19:42.541+0000 7f3391800640 1 -- 172.21.15.118:0/3018692129 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f338c104ad0 msgr2=0x7f338c19a1b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T03:19:42.544 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T03:19:42.541+0000 7f3391800640 1 --2- 172.21.15.118:0/3018692129 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f338c104ad0 0x7f338c19a1b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T03:19:42.544 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T03:19:42.541+0000 7f3391800640 1 -- 172.21.15.118:0/3018692129 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f338c1038e0 msgr2=0x7f338c199c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T03:19:42.544 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T03:19:42.541+0000 7f3391800640 1 --2- 172.21.15.118:0/3018692129 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f338c1038e0 0x7f338c199c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T03:19:42.544 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T03:19:42.541+0000 7f3391800640 1 -- 172.21.15.118:0/3018692129 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f338c1a1880 con 0x7f338c105490 2024-03-24T03:19:42.544 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T03:19:42.542+0000 7f3391800640 1 --2- 172.21.15.118:0/3018692129 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f338c105490 0x7f338c1a11e0 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7f338400ec50 tx=0x7f338400c6a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T03:19:42.544 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T03:19:42.542+0000 7f3389600640 1 -- 172.21.15.118:0/3018692129 <== mon.0 v2:172.21.15.118:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3384003b70 con 0x7f338c105490 2024-03-24T03:19:42.545 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T03:19:42.542+0000 7f3389600640 1 -- 172.21.15.118:0/3018692129 <== mon.0 v2:172.21.15.118:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7f3384003d10 con 0x7f338c105490 2024-03-24T03:19:42.545 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T03:19:42.542+0000 7f3393800640 1 -- 172.21.15.118:0/3018692129 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f338c1a1b70 con 0x7f338c105490 2024-03-24T03:19:42.545 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T03:19:42.542+0000 7f3389600640 1 -- 172.21.15.118:0/3018692129 <== mon.0 v2:172.21.15.118:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7f3384010890 con 0x7f338c105490 2024-03-24T03:19:42.545 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T03:19:42.542+0000 7f3393800640 1 -- 172.21.15.118:0/3018692129 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f338c1a2070 con 0x7f338c105490 2024-03-24T03:19:42.545 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T03:19:42.543+0000 7f3393800640 1 -- 172.21.15.118:0/3018692129 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f338c10acd0 con 0x7f338c105490 2024-03-24T03:19:42.546 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T03:19:42.545+0000 7f3389600640 1 -- 172.21.15.118:0/3018692129 <== mon.0 v2:172.21.15.118:3300/0 4 ==== mgrmap(e 20) v1 ==== 137291+0+0 (secure 0 0 0) 0x7f3384003fc0 con 0x7f338c105490 2024-03-24T03:19:42.547 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T03:19:42.547+0000 7f3389600640 1 --2- 172.21.15.118:0/3018692129 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7f33740a2800 0x7f33740a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T03:19:42.547 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T03:19:42.547+0000 7f3389600640 1 -- 172.21.15.118:0/3018692129 <== mon.0 v2:172.21.15.118:3300/0 5 ==== osd_map(125..125 src has 1..125) v4 ==== 11631+0+0 (secure 0 0 0) 0x7f33840cfdb0 con 0x7f338c105490 2024-03-24T03:19:42.547 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T03:19:42.548+0000 7f3390e00640 1 --2- 172.21.15.118:0/3018692129 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7f33740a2800 0x7f33740a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T03:19:42.547 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T03:19:42.548+0000 7f3390e00640 1 --2- 172.21.15.118:0/3018692129 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7f33740a2800 0x7f33740a4cc0 secure :-1 s=READY pgs=657 cs=0 l=1 rev1=1 crypto rx=0x7f3380006010 tx=0x7f3380005b50 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T03:19:42.552 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T03:19:42.552+0000 7f3389600640 1 -- 172.21.15.118:0/3018692129 <== mon.0 v2:172.21.15.118:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7f33840d4070 con 0x7f338c105490 2024-03-24T03:19:42.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 03:19:42 smithi118 ceph-mon[31535]: from='client.? 172.21.15.118:0/1686438014' entity='client.admin' cmd=[{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]: dispatch 2024-03-24T03:19:43.001 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 03:19:42 smithi118 ceph-mon[31535]: from='client.? ' entity='client.admin' cmd=[{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]: dispatch 2024-03-24T03:19:43.002 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 03:19:42 smithi118 ceph-mon[31535]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-03-24T03:19:43.002 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 03:19:42 smithi118 ceph-mon[31535]: Health check cleared: FS_WITH_FAILED_MDS (was: 1 filesystem has a failed mds daemon) 2024-03-24T03:19:43.002 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 03:19:42 smithi118 ceph-mon[31535]: Health check cleared: MDS_ALL_DOWN (was: 1 filesystem is offline) 2024-03-24T03:19:43.002 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 03:19:42 smithi118 ceph-mon[31535]: from='client.? ' entity='client.admin' cmd='[{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]': finished 2024-03-24T03:19:43.002 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 03:19:42 smithi118 ceph-mon[31535]: mds.? [v2:172.21.15.193:6834/2621986617,v1:172.21.15.193:6835/2621986617] up:boot 2024-03-24T03:19:43.002 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 03:19:42 smithi118 ceph-mon[31535]: mds.? [v2:172.21.15.193:6832/2631057528,v1:172.21.15.193:6833/2631057528] up:boot 2024-03-24T03:19:43.002 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 03:19:42 smithi118 ceph-mon[31535]: mds.? [v2:172.21.15.189:6832/2532452865,v1:172.21.15.189:6833/2532452865] up:boot 2024-03-24T03:19:43.002 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 03:19:42 smithi118 ceph-mon[31535]: mds.? [v2:172.21.15.189:6836/2662490215,v1:172.21.15.189:6837/2662490215] up:boot 2024-03-24T03:19:43.003 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 03:19:42 smithi118 ceph-mon[31535]: mds.? [v2:172.21.15.189:6834/3873243855,v1:172.21.15.189:6835/3873243855] up:boot 2024-03-24T03:19:43.003 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 03:19:42 smithi118 ceph-mon[31535]: mds.? [v2:172.21.15.118:6834/3867866433,v1:172.21.15.118:6835/3867866433] up:boot 2024-03-24T03:19:43.003 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 03:19:42 smithi118 ceph-mon[31535]: fsmap 6 up:standby 2024-03-24T03:19:43.003 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 03:19:42 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "a"}]: dispatch 2024-03-24T03:19:43.003 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 03:19:42 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2024-03-24T03:19:43.003 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 03:19:42 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2024-03-24T03:19:43.003 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 03:19:42 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-03-24T03:19:43.003 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 03:19:42 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "k"}]: dispatch 2024-03-24T03:19:43.004 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 03:19:42 smithi118 ceph-mon[31535]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2024-03-24T03:19:43.004 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 03:19:42 smithi193 ceph-mon[36878]: from='client.? 172.21.15.118:0/1686438014' entity='client.admin' cmd=[{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]: dispatch 2024-03-24T03:19:43.005 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 03:19:42 smithi193 ceph-mon[36878]: from='client.? ' entity='client.admin' cmd=[{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]: dispatch 2024-03-24T03:19:43.005 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 03:19:42 smithi193 ceph-mon[36878]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-03-24T03:19:43.005 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 03:19:42 smithi193 ceph-mon[36878]: Health check cleared: FS_WITH_FAILED_MDS (was: 1 filesystem has a failed mds daemon) 2024-03-24T03:19:43.005 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 03:19:42 smithi193 ceph-mon[36878]: Health check cleared: MDS_ALL_DOWN (was: 1 filesystem is offline) 2024-03-24T03:19:43.005 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 03:19:42 smithi193 ceph-mon[36878]: from='client.? ' entity='client.admin' cmd='[{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]': finished 2024-03-24T03:19:43.005 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 03:19:42 smithi193 ceph-mon[36878]: mds.? [v2:172.21.15.193:6834/2621986617,v1:172.21.15.193:6835/2621986617] up:boot 2024-03-24T03:19:43.005 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 03:19:42 smithi193 ceph-mon[36878]: mds.? [v2:172.21.15.193:6832/2631057528,v1:172.21.15.193:6833/2631057528] up:boot 2024-03-24T03:19:43.006 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 03:19:42 smithi193 ceph-mon[36878]: mds.? [v2:172.21.15.189:6832/2532452865,v1:172.21.15.189:6833/2532452865] up:boot 2024-03-24T03:19:43.006 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 03:19:42 smithi193 ceph-mon[36878]: mds.? [v2:172.21.15.189:6836/2662490215,v1:172.21.15.189:6837/2662490215] up:boot 2024-03-24T03:19:43.006 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 03:19:42 smithi193 ceph-mon[36878]: mds.? [v2:172.21.15.189:6834/3873243855,v1:172.21.15.189:6835/3873243855] up:boot 2024-03-24T03:19:43.006 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 03:19:42 smithi193 ceph-mon[36878]: mds.? [v2:172.21.15.118:6834/3867866433,v1:172.21.15.118:6835/3867866433] up:boot 2024-03-24T03:19:43.006 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 03:19:42 smithi193 ceph-mon[36878]: fsmap 6 up:standby 2024-03-24T03:19:43.006 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 03:19:42 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "a"}]: dispatch 2024-03-24T03:19:43.006 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 03:19:42 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2024-03-24T03:19:43.006 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 03:19:42 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2024-03-24T03:19:43.006 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 03:19:42 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-03-24T03:19:43.007 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 03:19:42 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "k"}]: dispatch 2024-03-24T03:19:43.007 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 03:19:42 smithi193 ceph-mon[36878]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2024-03-24T03:19:43.007 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 03:19:42 smithi189 ceph-mon[36187]: from='client.? 172.21.15.118:0/1686438014' entity='client.admin' cmd=[{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]: dispatch 2024-03-24T03:19:43.007 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 03:19:42 smithi189 ceph-mon[36187]: from='client.? ' entity='client.admin' cmd=[{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]: dispatch 2024-03-24T03:19:43.007 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 03:19:42 smithi189 ceph-mon[36187]: Health check cleared: FS_DEGRADED (was: 1 filesystem is degraded) 2024-03-24T03:19:43.007 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 03:19:42 smithi189 ceph-mon[36187]: Health check cleared: FS_WITH_FAILED_MDS (was: 1 filesystem has a failed mds daemon) 2024-03-24T03:19:43.007 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 03:19:42 smithi189 ceph-mon[36187]: Health check cleared: MDS_ALL_DOWN (was: 1 filesystem is offline) 2024-03-24T03:19:43.007 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 03:19:42 smithi189 ceph-mon[36187]: from='client.? ' entity='client.admin' cmd='[{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]': finished 2024-03-24T03:19:43.008 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 03:19:42 smithi189 ceph-mon[36187]: mds.? [v2:172.21.15.193:6834/2621986617,v1:172.21.15.193:6835/2621986617] up:boot 2024-03-24T03:19:43.008 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 03:19:42 smithi189 ceph-mon[36187]: mds.? [v2:172.21.15.193:6832/2631057528,v1:172.21.15.193:6833/2631057528] up:boot 2024-03-24T03:19:43.008 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 03:19:42 smithi189 ceph-mon[36187]: mds.? [v2:172.21.15.189:6832/2532452865,v1:172.21.15.189:6833/2532452865] up:boot 2024-03-24T03:19:43.008 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 03:19:42 smithi189 ceph-mon[36187]: mds.? [v2:172.21.15.189:6836/2662490215,v1:172.21.15.189:6837/2662490215] up:boot 2024-03-24T03:19:43.008 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 03:19:42 smithi189 ceph-mon[36187]: mds.? [v2:172.21.15.189:6834/3873243855,v1:172.21.15.189:6835/3873243855] up:boot 2024-03-24T03:19:43.008 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 03:19:42 smithi189 ceph-mon[36187]: mds.? [v2:172.21.15.118:6834/3867866433,v1:172.21.15.118:6835/3867866433] up:boot 2024-03-24T03:19:43.008 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 03:19:42 smithi189 ceph-mon[36187]: fsmap 6 up:standby 2024-03-24T03:19:43.008 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 03:19:42 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "a"}]: dispatch 2024-03-24T03:19:43.008 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 03:19:42 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "i"}]: dispatch 2024-03-24T03:19:43.009 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 03:19:42 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "b"}]: dispatch 2024-03-24T03:19:43.009 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 03:19:42 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "c"}]: dispatch 2024-03-24T03:19:43.009 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 03:19:42 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "k"}]: dispatch 2024-03-24T03:19:43.009 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 03:19:42 smithi189 ceph-mon[36187]: from='mgr.14152 172.21.15.118:0/4106045650' entity='mgr.x' cmd=[{"prefix": "mds metadata", "who": "e"}]: dispatch 2024-03-24T03:19:43.009 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T03:19:42.744+0000 7f3393800640 1 -- 172.21.15.118:0/3018692129 --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7f338c02d070 con 0x7f338c105490 2024-03-24T03:19:43.372 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T03:19:43.371+0000 7f3389600640 1 -- 172.21.15.118:0/3018692129 <== mon.0 v2:172.21.15.118:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_metadata' removed v126) v1 ==== 181+0+0 (secure 0 0 0) 0x7f338409c0b0 con 0x7f338c105490 2024-03-24T03:19:43.372 INFO:teuthology.orchestra.run.smithi118.stderr:pool 'cephfs_metadata' removed 2024-03-24T03:19:43.379 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T03:19:43.378+0000 7f3393800640 1 -- 172.21.15.118:0/3018692129 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7f33740a2800 msgr2=0x7f33740a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T03:19:43.379 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T03:19:43.378+0000 7f3393800640 1 --2- 172.21.15.118:0/3018692129 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7f33740a2800 0x7f33740a4cc0 secure :-1 s=READY pgs=657 cs=0 l=1 rev1=1 crypto rx=0x7f3380006010 tx=0x7f3380005b50 comp rx=0 tx=0).stop 2024-03-24T03:19:43.379 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T03:19:43.378+0000 7f3393800640 1 -- 172.21.15.118:0/3018692129 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f338c105490 msgr2=0x7f338c1a11e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T03:19:43.380 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T03:19:43.378+0000 7f3393800640 1 --2- 172.21.15.118:0/3018692129 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f338c105490 0x7f338c1a11e0 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7f338400ec50 tx=0x7f338400c6a0 comp rx=0 tx=0).stop 2024-03-24T03:19:43.380 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T03:19:43.379+0000 7f3393800640 1 -- 172.21.15.118:0/3018692129 shutdown_connections 2024-03-24T03:19:43.380 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T03:19:43.379+0000 7f3393800640 1 --2- 172.21.15.118:0/3018692129 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7f33740a2800 0x7f33740a4cc0 unknown :-1 s=CLOSED pgs=657 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T03:19:43.380 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T03:19:43.379+0000 7f3393800640 1 --2- 172.21.15.118:0/3018692129 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7f338c105490 0x7f338c1a11e0 unknown :-1 s=CLOSED pgs=241 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T03:19:43.380 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T03:19:43.379+0000 7f3393800640 1 --2- 172.21.15.118:0/3018692129 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7f338c104ad0 0x7f338c19a1b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T03:19:43.381 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T03:19:43.379+0000 7f3393800640 1 --2- 172.21.15.118:0/3018692129 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7f338c1038e0 0x7f338c199c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T03:19:43.381 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T03:19:43.379+0000 7f3393800640 1 -- 172.21.15.118:0/3018692129 >> 172.21.15.118:0/3018692129 conn(0x7f338c0fee50 msgr2=0x7f338c107ea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T03:19:43.381 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T03:19:43.379+0000 7f3393800640 1 -- 172.21.15.118:0/3018692129 shutdown_connections 2024-03-24T03:19:43.381 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T03:19:43.379+0000 7f3393800640 1 -- 172.21.15.118:0/3018692129 wait complete. 2024-03-24T03:19:43.404 DEBUG:teuthology.orchestra.run.smithi118:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool rm cephfs_data cephfs_data --yes-i-really-really-mean-it 2024-03-24T03:19:43.613 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T03:19:43.612+0000 7ff0fe800640 1 -- 172.21.15.118:0/3652223077 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7ff0f8108db0 msgr2=0x7ff0f8109230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T03:19:43.614 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T03:19:43.612+0000 7ff0fe800640 1 --2- 172.21.15.118:0/3652223077 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7ff0f8108db0 0x7ff0f8109230 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7ff0e0009900 tx=0x7ff0e002bc30 comp rx=0 tx=0).stop 2024-03-24T03:19:43.614 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T03:19:43.613+0000 7ff0fe800640 1 -- 172.21.15.118:0/3652223077 shutdown_connections 2024-03-24T03:19:43.614 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T03:19:43.613+0000 7ff0fe800640 1 --2- 172.21.15.118:0/3652223077 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff0f8109770 0x7ff0f8110000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T03:19:43.614 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T03:19:43.613+0000 7ff0fe800640 1 --2- 172.21.15.118:0/3652223077 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7ff0f8108db0 0x7ff0f8109230 unknown :-1 s=CLOSED pgs=242 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T03:19:43.614 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T03:19:43.613+0000 7ff0fe800640 1 --2- 172.21.15.118:0/3652223077 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7ff0f8107bc0 0x7ff0f8107fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T03:19:43.614 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T03:19:43.613+0000 7ff0fe800640 1 -- 172.21.15.118:0/3652223077 >> 172.21.15.118:0/3652223077 conn(0x7ff0f8076b70 msgr2=0x7ff0f8078f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T03:19:43.615 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T03:19:43.613+0000 7ff0fe800640 1 -- 172.21.15.118:0/3652223077 shutdown_connections 2024-03-24T03:19:43.615 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T03:19:43.613+0000 7ff0fe800640 1 -- 172.21.15.118:0/3652223077 wait complete. 2024-03-24T03:19:43.617 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T03:19:43.614+0000 7ff0fe800640 1 Processor -- start 2024-03-24T03:19:43.617 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T03:19:43.614+0000 7ff0fe800640 1 -- start start 2024-03-24T03:19:43.617 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T03:19:43.614+0000 7ff0fe800640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7ff0f8107bc0 0x7ff0f819df90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T03:19:43.617 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T03:19:43.614+0000 7ff0fe800640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7ff0f8108db0 0x7ff0f819e4d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T03:19:43.617 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T03:19:43.614+0000 7ff0fe800640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff0f8109770 0x7ff0f81a54a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T03:19:43.617 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T03:19:43.614+0000 7ff0fe800640 1 -- --> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff0f8113890 con 0x7ff0f8107bc0 2024-03-24T03:19:43.617 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T03:19:43.614+0000 7ff0fe800640 1 -- --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff0f8113710 con 0x7ff0f8109770 2024-03-24T03:19:43.617 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T03:19:43.614+0000 7ff0fe800640 1 -- --> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff0f8113a10 con 0x7ff0f8108db0 2024-03-24T03:19:43.617 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T03:19:43.614+0000 7ff0fd400640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7ff0f8107bc0 0x7ff0f819df90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T03:19:43.618 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T03:19:43.614+0000 7ff0fca00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7ff0f8108db0 0x7ff0f819e4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T03:19:43.618 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T03:19:43.614+0000 7ff0fde00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff0f8109770 0x7ff0f81a54a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T03:19:43.618 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T03:19:43.614+0000 7ff0fca00640 1 --2- >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7ff0f8108db0 0x7ff0f819e4d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.193:3300/0 says I am v2:172.21.15.118:49726/0 (socket says 172.21.15.118:49726) 2024-03-24T03:19:43.618 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T03:19:43.614+0000 7ff0fde00640 1 --2- >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff0f8109770 0x7ff0f81a54a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.189:3300/0 says I am v2:172.21.15.118:44352/0 (socket says 172.21.15.118:44352) 2024-03-24T03:19:43.618 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T03:19:43.614+0000 7ff0fca00640 1 -- 172.21.15.118:0/1101792489 learned_addr learned my addr 172.21.15.118:0/1101792489 (peer_addr_for_me v2:172.21.15.118:0/0) 2024-03-24T03:19:43.618 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T03:19:43.614+0000 7ff0fd400640 1 --2- >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7ff0f8107bc0 0x7ff0f819df90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.118:3300/0 says I am v2:172.21.15.118:34190/0 (socket says 172.21.15.118:34190) 2024-03-24T03:19:43.618 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T03:19:43.615+0000 7ff0fde00640 1 -- 172.21.15.118:0/1101792489 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7ff0f8108db0 msgr2=0x7ff0f819e4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T03:19:43.618 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T03:19:43.615+0000 7ff0fde00640 1 --2- 172.21.15.118:0/1101792489 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7ff0f8108db0 0x7ff0f819e4d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T03:19:43.618 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T03:19:43.615+0000 7ff0fde00640 1 -- 172.21.15.118:0/1101792489 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7ff0f8107bc0 msgr2=0x7ff0f819df90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T03:19:43.619 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T03:19:43.615+0000 7ff0fde00640 1 --2- 172.21.15.118:0/1101792489 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7ff0f8107bc0 0x7ff0f819df90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T03:19:43.619 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T03:19:43.615+0000 7ff0fde00640 1 -- 172.21.15.118:0/1101792489 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff0f81a5b10 con 0x7ff0f8109770 2024-03-24T03:19:43.619 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T03:19:43.615+0000 7ff0fde00640 1 --2- 172.21.15.118:0/1101792489 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff0f8109770 0x7ff0f81a54a0 secure :-1 s=READY pgs=493 cs=0 l=1 rev1=1 crypto rx=0x7ff0f000e970 tx=0x7ff0f000ee30 comp rx=0 tx=0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T03:19:43.619 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T03:19:43.615+0000 7ff0f6000640 1 -- 172.21.15.118:0/1101792489 <== mon.2 v2:172.21.15.189:3300/0 1 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff0f000ccc0 con 0x7ff0f8109770 2024-03-24T03:19:43.619 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T03:19:43.615+0000 7ff0fe800640 1 -- 172.21.15.118:0/1101792489 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff0f81a5e00 con 0x7ff0f8109770 2024-03-24T03:19:43.619 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T03:19:43.615+0000 7ff0f6000640 1 -- 172.21.15.118:0/1101792489 <== mon.2 v2:172.21.15.189:3300/0 2 ==== config(27 keys) v1 ==== 1140+0+0 (secure 0 0 0) 0x7ff0f00035f0 con 0x7ff0f8109770 2024-03-24T03:19:43.619 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T03:19:43.615+0000 7ff0f6000640 1 -- 172.21.15.118:0/1101792489 <== mon.2 v2:172.21.15.189:3300/0 3 ==== mon_map magic: 0 v1 ==== 413+0+0 (secure 0 0 0) 0x7ff0f000ce60 con 0x7ff0f8109770 2024-03-24T03:19:43.619 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T03:19:43.615+0000 7ff0fe800640 1 -- 172.21.15.118:0/1101792489 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff0f81a62c0 con 0x7ff0f8109770 2024-03-24T03:19:43.620 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T03:19:43.616+0000 7ff0fe800640 1 -- 172.21.15.118:0/1101792489 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff0f810ef00 con 0x7ff0f8109770 2024-03-24T03:19:43.620 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T03:19:43.618+0000 7ff0f6000640 1 -- 172.21.15.118:0/1101792489 <== mon.2 v2:172.21.15.189:3300/0 4 ==== mgrmap(e 20) v1 ==== 137291+0+0 (secure 0 0 0) 0x7ff0f00037a0 con 0x7ff0f8109770 2024-03-24T03:19:43.620 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T03:19:43.620+0000 7ff0f6000640 1 --2- 172.21.15.118:0/1101792489 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7ff0dc0a2800 0x7ff0dc0a4cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T03:19:43.620 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T03:19:43.620+0000 7ff0f6000640 1 -- 172.21.15.118:0/1101792489 <== mon.2 v2:172.21.15.189:3300/0 5 ==== osd_map(126..126 src has 1..126) v4 ==== 11077+0+0 (secure 0 0 0) 0x7ff0f00cf890 con 0x7ff0f8109770 2024-03-24T03:19:43.621 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T03:19:43.620+0000 7ff0fd400640 1 --2- 172.21.15.118:0/1101792489 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7ff0dc0a2800 0x7ff0dc0a4cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T03:19:43.621 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T03:19:43.621+0000 7ff0fd400640 1 --2- 172.21.15.118:0/1101792489 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7ff0dc0a2800 0x7ff0dc0a4cc0 secure :-1 s=READY pgs=663 cs=0 l=1 rev1=1 crypto rx=0x7ff0f8108c10 tx=0x7ff0e8009500 comp rx=0 tx=0).ready entity=mgr.14152 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T03:19:43.622 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T03:19:43.622+0000 7ff0f6000640 1 -- 172.21.15.118:0/1101792489 <== mon.2 v2:172.21.15.189:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+186149 (secure 0 0 0) 0x7ff0f009be40 con 0x7ff0f8109770 2024-03-24T03:19:43.695 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 03:19:43 smithi118 ceph-mon[31535]: pgmap v6171: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:19:43.695 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 03:19:43 smithi118 ceph-mon[31535]: from='client.? 172.21.15.118:0/3018692129' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]: dispatch 2024-03-24T03:19:43.703 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 03:19:43 smithi189 ceph-mon[36187]: pgmap v6171: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:19:43.703 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 03:19:43 smithi189 ceph-mon[36187]: from='client.? 172.21.15.118:0/3018692129' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]: dispatch 2024-03-24T03:19:43.710 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 03:19:43 smithi193 ceph-mon[36878]: pgmap v6171: 97 pgs: 97 active+clean; 147 MiB data, 882 MiB used, 1.0 TiB / 1.0 TiB avail; 2.5 KiB/s rd, 4 op/s 2024-03-24T03:19:43.710 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 03:19:43 smithi193 ceph-mon[36878]: from='client.? 172.21.15.118:0/3018692129' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]: dispatch 2024-03-24T03:19:43.807 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T03:19:43.806+0000 7ff0fe800640 1 -- 172.21.15.118:0/1101792489 --> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] -- mon_command({"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7ff0f802d070 con 0x7ff0f8109770 2024-03-24T03:19:44.393 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T03:19:44.392+0000 7ff0f6000640 1 -- 172.21.15.118:0/1101792489 <== mon.2 v2:172.21.15.189:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data' removed v127) v1 ==== 169+0+0 (secure 0 0 0) 0x7ff0f00bb020 con 0x7ff0f8109770 2024-03-24T03:19:44.393 INFO:teuthology.orchestra.run.smithi118.stderr:pool 'cephfs_data' removed 2024-03-24T03:19:44.397 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T03:19:44.397+0000 7ff0fe800640 1 -- 172.21.15.118:0/1101792489 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7ff0dc0a2800 msgr2=0x7ff0dc0a4cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T03:19:44.398 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T03:19:44.397+0000 7ff0fe800640 1 --2- 172.21.15.118:0/1101792489 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7ff0dc0a2800 0x7ff0dc0a4cc0 secure :-1 s=READY pgs=663 cs=0 l=1 rev1=1 crypto rx=0x7ff0f8108c10 tx=0x7ff0e8009500 comp rx=0 tx=0).stop 2024-03-24T03:19:44.398 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T03:19:44.397+0000 7ff0fe800640 1 -- 172.21.15.118:0/1101792489 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff0f8109770 msgr2=0x7ff0f81a54a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T03:19:44.398 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T03:19:44.397+0000 7ff0fe800640 1 --2- 172.21.15.118:0/1101792489 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff0f8109770 0x7ff0f81a54a0 secure :-1 s=READY pgs=493 cs=0 l=1 rev1=1 crypto rx=0x7ff0f000e970 tx=0x7ff0f000ee30 comp rx=0 tx=0).stop 2024-03-24T03:19:44.398 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T03:19:44.397+0000 7ff0fe800640 1 -- 172.21.15.118:0/1101792489 shutdown_connections 2024-03-24T03:19:44.398 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T03:19:44.397+0000 7ff0fe800640 1 --2- 172.21.15.118:0/1101792489 >> [v2:172.21.15.118:6800/2112912943,v1:172.21.15.118:6801/2112912943] conn(0x7ff0dc0a2800 0x7ff0dc0a4cc0 unknown :-1 s=CLOSED pgs=663 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T03:19:44.398 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T03:19:44.397+0000 7ff0fe800640 1 --2- 172.21.15.118:0/1101792489 >> [v2:172.21.15.189:3300/0,v1:172.21.15.189:6789/0] conn(0x7ff0f8109770 0x7ff0f81a54a0 unknown :-1 s=CLOSED pgs=493 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T03:19:44.398 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T03:19:44.397+0000 7ff0fe800640 1 --2- 172.21.15.118:0/1101792489 >> [v2:172.21.15.193:3300/0,v1:172.21.15.193:6789/0] conn(0x7ff0f8108db0 0x7ff0f819e4d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T03:19:44.399 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T03:19:44.397+0000 7ff0fe800640 1 --2- 172.21.15.118:0/1101792489 >> [v2:172.21.15.118:3300/0,v1:172.21.15.118:6789/0] conn(0x7ff0f8107bc0 0x7ff0f819df90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T03:19:44.399 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T03:19:44.397+0000 7ff0fe800640 1 -- 172.21.15.118:0/1101792489 >> 172.21.15.118:0/1101792489 conn(0x7ff0f8076b70 msgr2=0x7ff0f810c0d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T03:19:44.399 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T03:19:44.397+0000 7ff0fe800640 1 -- 172.21.15.118:0/1101792489 shutdown_connections 2024-03-24T03:19:44.399 INFO:teuthology.orchestra.run.smithi118.stderr:2024-03-24T03:19:44.397+0000 7ff0fe800640 1 -- 172.21.15.118:0/1101792489 wait complete. 2024-03-24T03:19:44.423 DEBUG:teuthology.orchestra.run.smithi118:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-03-24T03:19:44.507 DEBUG:teuthology.orchestra.run.smithi189:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-03-24T03:19:44.542 DEBUG:teuthology.orchestra.run.smithi193:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-03-24T03:19:44.575 INFO:tasks.cephadm:Cleaning up testdir ceph.* files... 2024-03-24T03:19:44.576 DEBUG:teuthology.orchestra.run.smithi118:> rm -f /home/ubuntu/cephtest/seed.ceph.conf /home/ubuntu/cephtest/ceph.pub 2024-03-24T03:19:44.629 DEBUG:teuthology.orchestra.run.smithi189:> rm -f /home/ubuntu/cephtest/seed.ceph.conf /home/ubuntu/cephtest/ceph.pub 2024-03-24T03:19:44.645 DEBUG:teuthology.orchestra.run.smithi193:> rm -f /home/ubuntu/cephtest/seed.ceph.conf /home/ubuntu/cephtest/ceph.pub 2024-03-24T03:19:44.662 INFO:tasks.cephadm:Stopping all daemons... 2024-03-24T03:19:44.662 INFO:tasks.cephadm.mon.a:Stopping mon.a... 2024-03-24T03:19:44.662 DEBUG:teuthology.orchestra.run.smithi118:> sudo systemctl stop ceph-d79a5420-e971-11ee-95cd-87774f69a715@mon.a 2024-03-24T03:19:44.695 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 03:19:44 smithi118 ceph-mon[31535]: from='client.? 172.21.15.118:0/3018692129' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]': finished 2024-03-24T03:19:44.695 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 03:19:44 smithi118 ceph-mon[31535]: osdmap e126: 12 total, 12 up, 12 in 2024-03-24T03:19:44.695 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 03:19:44 smithi118 ceph-mon[31535]: from='client.? 172.21.15.118:0/1101792489' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]: dispatch 2024-03-24T03:19:44.696 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 03:19:44 smithi118 ceph-mon[31535]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]: dispatch 2024-03-24T03:19:44.703 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 03:19:44 smithi189 ceph-mon[36187]: from='client.? 172.21.15.118:0/3018692129' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]': finished 2024-03-24T03:19:44.704 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 03:19:44 smithi189 ceph-mon[36187]: osdmap e126: 12 total, 12 up, 12 in 2024-03-24T03:19:44.704 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 03:19:44 smithi189 ceph-mon[36187]: from='client.? 172.21.15.118:0/1101792489' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]: dispatch 2024-03-24T03:19:44.704 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 03:19:44 smithi189 ceph-mon[36187]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]: dispatch 2024-03-24T03:19:44.710 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 03:19:44 smithi193 ceph-mon[36878]: from='client.? 172.21.15.118:0/3018692129' entity='client.admin' cmd='[{"prefix": "osd pool rm", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]': finished 2024-03-24T03:19:44.710 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 03:19:44 smithi193 ceph-mon[36878]: osdmap e126: 12 total, 12 up, 12 in 2024-03-24T03:19:44.710 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 03:19:44 smithi193 ceph-mon[36878]: from='client.? 172.21.15.118:0/1101792489' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]: dispatch 2024-03-24T03:19:44.711 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 03:19:44 smithi193 ceph-mon[36878]: from='client.? ' entity='client.admin' cmd=[{"prefix": "osd pool rm", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]: dispatch 2024-03-24T03:19:45.196 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 03:19:44 smithi118 systemd[1]: Stopping Ceph mon.a for d79a5420-e971-11ee-95cd-87774f69a715... 2024-03-24T03:19:45.639 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 03:19:45 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mon-a[31531]: 2024-03-24T03:19:45.271+0000 7f2d6ec00700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.a -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-03-24T03:19:45.639 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 03:19:45 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mon-a[31531]: 2024-03-24T03:19:45.271+0000 7f2d6ec00700 -1 mon.a@0(leader) e3 *** Got Signal Terminated *** 2024-03-24T03:19:45.640 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 03:19:45 smithi118 podman[145573]: 2024-03-24 03:19:45.335405278 +0000 UTC m=+0.268123671 container died 9a3b56cb7cc8e0313c3963fe3db6b16bb1ecf7d71ab784c3a43ea706ab1ec49b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033, name=ceph-d79a5420-e971-11ee-95cd-87774f69a715-mon-a, org.label-schema.license=GPLv2, ceph=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240319, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD) 2024-03-24T03:19:45.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 03:19:45 smithi118 podman[145573]: 2024-03-24 03:19:45.76554505 +0000 UTC m=+0.698263449 container cleanup 9a3b56cb7cc8e0313c3963fe3db6b16bb1ecf7d71ab784c3a43ea706ab1ec49b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033, name=ceph-d79a5420-e971-11ee-95cd-87774f69a715-mon-a, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.license=GPLv2, GIT_CLEAN=True, org.label-schema.build-date=20240319, GIT_BRANCH=HEAD, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux ) 2024-03-24T03:19:45.946 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 03:19:45 smithi118 bash[145573]: ceph-d79a5420-e971-11ee-95cd-87774f69a715-mon-a 2024-03-24T03:19:46.446 INFO:journalctl@ceph.mon.a.smithi118.stdout:Mar 24 03:19:46 smithi118 podman[145587]: 2024-03-24 03:19:46.109352276 +0000 UTC m=+0.758923929 container remove 9a3b56cb7cc8e0313c3963fe3db6b16bb1ecf7d71ab784c3a43ea706ab1ec49b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033, name=ceph-d79a5420-e971-11ee-95cd-87774f69a715-mon-a, maintainer=Guillaume Abrioux , GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.schema-version=1.0, ceph=True, org.label-schema.build-date=20240319, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5) 2024-03-24T03:19:46.537 DEBUG:teuthology.orchestra.run.smithi118:> sudo pkill -f 'journalctl -f -n 0 -u ceph-d79a5420-e971-11ee-95cd-87774f69a715@mon.a.service' 2024-03-24T03:19:46.724 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-24T03:19:46.724 INFO:tasks.cephadm.mon.a:Stopped mon.a 2024-03-24T03:19:46.724 INFO:tasks.cephadm.mon.c:Stopping mon.b... 2024-03-24T03:19:46.725 DEBUG:teuthology.orchestra.run.smithi189:> sudo systemctl stop ceph-d79a5420-e971-11ee-95cd-87774f69a715@mon.b 2024-03-24T03:19:47.049 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 03:19:46 smithi189 systemd[1]: Stopping Ceph mon.b for d79a5420-e971-11ee-95cd-87774f69a715... 2024-03-24T03:19:47.375 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 03:19:47 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mon-b[36183]: 2024-03-24T03:19:47.049+0000 7f90bff0f700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.b -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-03-24T03:19:47.376 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 03:19:47 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mon-b[36183]: 2024-03-24T03:19:47.049+0000 7f90bff0f700 -1 mon.b@2(peon) e3 *** Got Signal Terminated *** 2024-03-24T03:19:47.376 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 03:19:47 smithi189 podman[116767]: 2024-03-24 03:19:47.070685395 +0000 UTC m=+0.156176114 container died 332ea823179e88f4b6faf6793d78c9576c10138bb235aa5cd94334b2af6c7389 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-d79a5420-e971-11ee-95cd-87774f69a715-mon-b, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, org.label-schema.build-date=20240319, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, io.buildah.version=1.33.5, ceph=True) 2024-03-24T03:19:47.703 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 03:19:47 smithi189 podman[116767]: 2024-03-24 03:19:47.576199162 +0000 UTC m=+0.661689891 container cleanup 332ea823179e88f4b6faf6793d78c9576c10138bb235aa5cd94334b2af6c7389 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-d79a5420-e971-11ee-95cd-87774f69a715-mon-b, GIT_CLEAN=True, ceph=True, org.label-schema.build-date=20240319, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, io.buildah.version=1.33.5) 2024-03-24T03:19:47.704 INFO:journalctl@ceph.mon.b.smithi189.stdout:Mar 24 03:19:47 smithi189 bash[116767]: ceph-d79a5420-e971-11ee-95cd-87774f69a715-mon-b 2024-03-24T03:19:47.986 DEBUG:teuthology.orchestra.run.smithi189:> sudo pkill -f 'journalctl -f -n 0 -u ceph-d79a5420-e971-11ee-95cd-87774f69a715@mon.b.service' 2024-03-24T03:19:48.033 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-24T03:19:48.034 INFO:tasks.cephadm.mon.c:Stopped mon.b 2024-03-24T03:19:48.034 INFO:tasks.cephadm.mon.c:Stopping mon.c... 2024-03-24T03:19:48.034 DEBUG:teuthology.orchestra.run.smithi193:> sudo systemctl stop ceph-d79a5420-e971-11ee-95cd-87774f69a715@mon.c 2024-03-24T03:19:48.344 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 03:19:48 smithi193 systemd[1]: Stopping Ceph mon.c for d79a5420-e971-11ee-95cd-87774f69a715... 2024-03-24T03:19:48.710 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 03:19:48 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mon-c[36874]: 2024-03-24T03:19:48.344+0000 7f447b43b700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.c -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-03-24T03:19:48.711 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 03:19:48 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mon-c[36874]: 2024-03-24T03:19:48.344+0000 7f447b43b700 -1 mon.c@1(peon) e3 *** Got Signal Terminated *** 2024-03-24T03:19:48.711 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 03:19:48 smithi193 podman[120051]: 2024-03-24 03:19:48.416233179 +0000 UTC m=+0.182356735 container died 0284545a88c562a09f5395625948e2d9c1fc6f6487ee318a6e852bc19f0211ae (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-d79a5420-e971-11ee-95cd-87774f69a715-mon-c, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.license=GPLv2, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.build-date=20240319, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD) 2024-03-24T03:19:49.129 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 03:19:48 smithi193 podman[120051]: 2024-03-24 03:19:48.928874409 +0000 UTC m=+0.694997992 container cleanup 0284545a88c562a09f5395625948e2d9c1fc6f6487ee318a6e852bc19f0211ae (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-d79a5420-e971-11ee-95cd-87774f69a715-mon-c, org.label-schema.vendor=CentOS, GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240319, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD) 2024-03-24T03:19:49.130 INFO:journalctl@ceph.mon.c.smithi193.stdout:Mar 24 03:19:48 smithi193 bash[120051]: ceph-d79a5420-e971-11ee-95cd-87774f69a715-mon-c 2024-03-24T03:19:49.337 DEBUG:teuthology.orchestra.run.smithi193:> sudo pkill -f 'journalctl -f -n 0 -u ceph-d79a5420-e971-11ee-95cd-87774f69a715@mon.c.service' 2024-03-24T03:19:49.382 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-24T03:19:49.383 INFO:tasks.cephadm.mon.c:Stopped mon.c 2024-03-24T03:19:49.383 INFO:tasks.cephadm.mgr.x:Stopping mgr.x... 2024-03-24T03:19:49.383 DEBUG:teuthology.orchestra.run.smithi118:> sudo systemctl stop ceph-d79a5420-e971-11ee-95cd-87774f69a715@mgr.x 2024-03-24T03:19:49.946 INFO:journalctl@ceph.mgr.x.smithi118.stdout:Mar 24 03:19:49 smithi118 systemd[1]: Stopping Ceph mgr.x for d79a5420-e971-11ee-95cd-87774f69a715... 2024-03-24T03:19:50.402 INFO:journalctl@ceph.mgr.x.smithi118.stdout:Mar 24 03:19:50 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mgr-x[31762]: 2024-03-24T03:19:50.001+0000 7f634b800700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mgr -n mgr.x -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-24T03:19:50.402 INFO:journalctl@ceph.mgr.x.smithi118.stdout:Mar 24 03:19:50 smithi118 podman[145707]: 2024-03-24 03:19:50.09546011 +0000 UTC m=+0.306261672 container died 16a004216cba58ac648fa39a41748df811122aec56543cff33c3b23f58bcd392 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033, name=ceph-d79a5420-e971-11ee-95cd-87774f69a715-mgr-x, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240319, GIT_CLEAN=True, ceph=True, io.buildah.version=1.33.5, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux ) 2024-03-24T03:19:51.195 INFO:journalctl@ceph.mgr.x.smithi118.stdout:Mar 24 03:19:50 smithi118 podman[145707]: 2024-03-24 03:19:50.771935901 +0000 UTC m=+0.982737503 container cleanup 16a004216cba58ac648fa39a41748df811122aec56543cff33c3b23f58bcd392 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033, name=ceph-d79a5420-e971-11ee-95cd-87774f69a715-mgr-x, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, org.label-schema.build-date=20240319, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, maintainer=Guillaume Abrioux ) 2024-03-24T03:19:51.196 INFO:journalctl@ceph.mgr.x.smithi118.stdout:Mar 24 03:19:50 smithi118 bash[145707]: ceph-d79a5420-e971-11ee-95cd-87774f69a715-mgr-x 2024-03-24T03:19:51.196 INFO:journalctl@ceph.mgr.x.smithi118.stdout:Mar 24 03:19:50 smithi118 podman[145721]: 2024-03-24 03:19:50.956799103 +0000 UTC m=+0.847543897 container remove 16a004216cba58ac648fa39a41748df811122aec56543cff33c3b23f58bcd392 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph:3be9a5c9bc793e11e2800a8c0c696e8b46742033, name=ceph-d79a5420-e971-11ee-95cd-87774f69a715-mgr-x, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, org.label-schema.build-date=20240319, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, ceph=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30) 2024-03-24T03:19:51.438 DEBUG:teuthology.orchestra.run.smithi118:> sudo pkill -f 'journalctl -f -n 0 -u ceph-d79a5420-e971-11ee-95cd-87774f69a715@mgr.x.service' 2024-03-24T03:19:51.557 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-24T03:19:51.557 INFO:tasks.cephadm.mgr.x:Stopped mgr.x 2024-03-24T03:19:51.558 INFO:tasks.cephadm.mgr.y:Stopping mgr.y... 2024-03-24T03:19:51.558 DEBUG:teuthology.orchestra.run.smithi189:> sudo systemctl stop ceph-d79a5420-e971-11ee-95cd-87774f69a715@mgr.y 2024-03-24T03:19:51.953 INFO:journalctl@ceph.mgr.y.smithi189.stdout:Mar 24 03:19:51 smithi189 systemd[1]: Stopping Ceph mgr.y for d79a5420-e971-11ee-95cd-87774f69a715... 2024-03-24T03:19:52.306 INFO:journalctl@ceph.mgr.y.smithi189.stdout:Mar 24 03:19:51 smithi189 podman[116893]: 2024-03-24 03:19:51.984714565 +0000 UTC m=+0.224703540 container died 93099988c06e582719019354c097b6472802e97fa46a70b36b7d77cbad9480b3 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-d79a5420-e971-11ee-95cd-87774f69a715-mgr-y, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, org.label-schema.build-date=20240319, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, GIT_CLEAN=True, ceph=True, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5) 2024-03-24T03:19:52.649 INFO:journalctl@ceph.mgr.y.smithi189.stdout:Mar 24 03:19:52 smithi189 podman[116893]: 2024-03-24 03:19:52.3981361 +0000 UTC m=+0.638125074 container cleanup 93099988c06e582719019354c097b6472802e97fa46a70b36b7d77cbad9480b3 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-d79a5420-e971-11ee-95cd-87774f69a715-mgr-y, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, ceph=True, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240319, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30) 2024-03-24T03:19:52.649 INFO:journalctl@ceph.mgr.y.smithi189.stdout:Mar 24 03:19:52 smithi189 bash[116893]: ceph-d79a5420-e971-11ee-95cd-87774f69a715-mgr-y 2024-03-24T03:19:52.813 DEBUG:teuthology.orchestra.run.smithi189:> sudo pkill -f 'journalctl -f -n 0 -u ceph-d79a5420-e971-11ee-95cd-87774f69a715@mgr.y.service' 2024-03-24T03:19:52.849 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-24T03:19:52.850 INFO:tasks.cephadm.mgr.y:Stopped mgr.y 2024-03-24T03:19:52.850 INFO:tasks.cephadm.mgr.z:Stopping mgr.z... 2024-03-24T03:19:52.850 DEBUG:teuthology.orchestra.run.smithi193:> sudo systemctl stop ceph-d79a5420-e971-11ee-95cd-87774f69a715@mgr.z 2024-03-24T03:19:53.178 INFO:journalctl@ceph.mgr.z.smithi193.stdout:Mar 24 03:19:52 smithi193 systemd[1]: Stopping Ceph mgr.z for d79a5420-e971-11ee-95cd-87774f69a715... 2024-03-24T03:19:53.460 INFO:journalctl@ceph.mgr.z.smithi193.stdout:Mar 24 03:19:53 smithi193 podman[120176]: 2024-03-24 03:19:53.179836327 +0000 UTC m=+0.137769968 container died df9c7647eda091a2686a260871afc3ace7bcfccd6aa82915bada036e7b01a781 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-d79a5420-e971-11ee-95cd-87774f69a715-mgr-z, org.label-schema.build-date=20240319, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, GIT_BRANCH=HEAD, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5) 2024-03-24T03:19:53.863 INFO:journalctl@ceph.mgr.z.smithi193.stdout:Mar 24 03:19:53 smithi193 podman[120176]: 2024-03-24 03:19:53.584545109 +0000 UTC m=+0.542478755 container cleanup df9c7647eda091a2686a260871afc3ace7bcfccd6aa82915bada036e7b01a781 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-d79a5420-e971-11ee-95cd-87774f69a715-mgr-z, GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240319, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True) 2024-03-24T03:19:53.864 INFO:journalctl@ceph.mgr.z.smithi193.stdout:Mar 24 03:19:53 smithi193 bash[120176]: ceph-d79a5420-e971-11ee-95cd-87774f69a715-mgr-z 2024-03-24T03:19:53.993 DEBUG:teuthology.orchestra.run.smithi193:> sudo pkill -f 'journalctl -f -n 0 -u ceph-d79a5420-e971-11ee-95cd-87774f69a715@mgr.z.service' 2024-03-24T03:19:54.029 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-24T03:19:54.029 INFO:tasks.cephadm.mgr.z:Stopped mgr.z 2024-03-24T03:19:54.030 INFO:tasks.cephadm.osd.0:Stopping osd.0... 2024-03-24T03:19:54.030 DEBUG:teuthology.orchestra.run.smithi118:> sudo systemctl stop ceph-d79a5420-e971-11ee-95cd-87774f69a715@osd.0 2024-03-24T03:19:54.445 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 24 03:19:54 smithi118 systemd[1]: Stopping Ceph osd.0 for d79a5420-e971-11ee-95cd-87774f69a715... 2024-03-24T03:19:54.946 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 24 03:19:54 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-osd-0[46346]: 2024-03-24T03:19:54.632+0000 7faec7400700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.0 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-24T03:19:54.947 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 24 03:19:54 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-osd-0[46346]: 2024-03-24T03:19:54.632+0000 7faec7400700 -1 osd.0 127 *** Got signal Terminated *** 2024-03-24T03:19:54.947 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 24 03:19:54 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-osd-0[46346]: 2024-03-24T03:19:54.632+0000 7faec7400700 -1 osd.0 127 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-24T03:20:00.301 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 24 03:20:00 smithi118 podman[145826]: 2024-03-24 03:20:00.054834013 +0000 UTC m=+5.570978311 container died 796fc494a23829de97ac23d7c97fb4b4f47a3d66c1b4b153d249e92661f26a5e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-d79a5420-e971-11ee-95cd-87774f69a715-osd-0, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, ceph=True, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240319, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, GIT_CLEAN=True) 2024-03-24T03:20:00.696 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 24 03:20:00 smithi118 podman[145826]: 2024-03-24 03:20:00.477051358 +0000 UTC m=+5.993195694 container cleanup 796fc494a23829de97ac23d7c97fb4b4f47a3d66c1b4b153d249e92661f26a5e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-d79a5420-e971-11ee-95cd-87774f69a715-osd-0, maintainer=Guillaume Abrioux , GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, ceph=True, org.label-schema.build-date=20240319, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0) 2024-03-24T03:20:00.696 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 24 03:20:00 smithi118 bash[145826]: ceph-d79a5420-e971-11ee-95cd-87774f69a715-osd-0 2024-03-24T03:20:01.196 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 24 03:20:00 smithi118 podman[145848]: 2024-03-24 03:20:00.778717146 +0000 UTC m=+0.710173997 container remove 796fc494a23829de97ac23d7c97fb4b4f47a3d66c1b4b153d249e92661f26a5e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-d79a5420-e971-11ee-95cd-87774f69a715-osd-0, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, org.label-schema.build-date=20240319, CEPH_POINT_RELEASE=, ceph=True, io.buildah.version=1.33.5, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD) 2024-03-24T03:20:01.887 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 24 03:20:01 smithi118 podman[145936]: 2024-03-24 03:20:01.49550095 +0000 UTC m=+0.034075157 image pull c664afe56bb01db4f9afa9532c20598e201e664f2e9ba2be50db0d52674a26d2 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc 2024-03-24T03:20:01.887 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 24 03:20:01 smithi118 podman[145936]: 2024-03-24 03:20:01.621357767 +0000 UTC m=+0.159932010 container create 260ba9f84fa64269c11d63ff4d23e36c5f8c2d4d103df05aadaab26ccac3db75 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-d79a5420-e971-11ee-95cd-87774f69a715-osd-0-deactivate, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240319, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, ceph=True, org.label-schema.license=GPLv2) 2024-03-24T03:20:01.888 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 24 03:20:01 smithi118 podman[145936]: 2024-03-24 03:20:01.887146847 +0000 UTC m=+0.425721096 container init 260ba9f84fa64269c11d63ff4d23e36c5f8c2d4d103df05aadaab26ccac3db75 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-d79a5420-e971-11ee-95cd-87774f69a715-osd-0-deactivate, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, org.label-schema.build-date=20240319, GIT_BRANCH=HEAD) 2024-03-24T03:20:02.196 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 24 03:20:01 smithi118 podman[145936]: 2024-03-24 03:20:01.897862286 +0000 UTC m=+0.436436499 container start 260ba9f84fa64269c11d63ff4d23e36c5f8c2d4d103df05aadaab26ccac3db75 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-d79a5420-e971-11ee-95cd-87774f69a715-osd-0-deactivate, io.buildah.version=1.33.5, ceph=True, org.label-schema.license=GPLv2, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, GIT_BRANCH=HEAD, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240319, CEPH_POINT_RELEASE=) 2024-03-24T03:20:02.196 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 24 03:20:01 smithi118 podman[145936]: 2024-03-24 03:20:01.983036664 +0000 UTC m=+0.521610964 container attach 260ba9f84fa64269c11d63ff4d23e36c5f8c2d4d103df05aadaab26ccac3db75 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-d79a5420-e971-11ee-95cd-87774f69a715-osd-0-deactivate, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, ceph=True, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.build-date=20240319) 2024-03-24T03:20:02.677 INFO:journalctl@ceph.osd.0.smithi118.stdout:Mar 24 03:20:02 smithi118 podman[145962]: 2024-03-24 03:20:02.371457292 +0000 UTC m=+0.031598212 container died 260ba9f84fa64269c11d63ff4d23e36c5f8c2d4d103df05aadaab26ccac3db75 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-d79a5420-e971-11ee-95cd-87774f69a715-osd-0-deactivate, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240319, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, org.label-schema.license=GPLv2) 2024-03-24T03:20:03.061 DEBUG:teuthology.orchestra.run.smithi118:> sudo pkill -f 'journalctl -f -n 0 -u ceph-d79a5420-e971-11ee-95cd-87774f69a715@osd.0.service' 2024-03-24T03:20:03.179 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-24T03:20:03.179 INFO:tasks.cephadm.osd.0:Stopped osd.0 2024-03-24T03:20:03.180 INFO:tasks.cephadm.osd.1:Stopping osd.1... 2024-03-24T03:20:03.180 DEBUG:teuthology.orchestra.run.smithi189:> sudo systemctl stop ceph-d79a5420-e971-11ee-95cd-87774f69a715@osd.1 2024-03-24T03:20:03.519 INFO:journalctl@ceph.osd.1.smithi189.stdout:Mar 24 03:20:03 smithi189 systemd[1]: Stopping Ceph osd.1 for d79a5420-e971-11ee-95cd-87774f69a715... 2024-03-24T03:20:03.953 INFO:journalctl@ceph.osd.1.smithi189.stdout:Mar 24 03:20:03 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-osd-1[40812]: 2024-03-24T03:20:03.519+0000 7f747f2e0700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.1 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-24T03:20:03.954 INFO:journalctl@ceph.osd.1.smithi189.stdout:Mar 24 03:20:03 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-osd-1[40812]: 2024-03-24T03:20:03.519+0000 7f747f2e0700 -1 osd.1 127 *** Got signal Terminated *** 2024-03-24T03:20:03.954 INFO:journalctl@ceph.osd.1.smithi189.stdout:Mar 24 03:20:03 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-osd-1[40812]: 2024-03-24T03:20:03.519+0000 7f747f2e0700 -1 osd.1 127 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-24T03:20:09.344 INFO:journalctl@ceph.osd.1.smithi189.stdout:Mar 24 03:20:09 smithi189 podman[117018]: 2024-03-24 03:20:09.037111064 +0000 UTC m=+5.655614744 container died 5b082a48fda491e9a6d3c106c1a72ede5bc67761d079eb0a2f7f54b3b987e154 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-d79a5420-e971-11ee-95cd-87774f69a715-osd-1, GIT_BRANCH=HEAD, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240319, CEPH_POINT_RELEASE=, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, org.label-schema.vendor=CentOS) 2024-03-24T03:20:09.687 INFO:journalctl@ceph.osd.1.smithi189.stdout:Mar 24 03:20:09 smithi189 podman[117018]: 2024-03-24 03:20:09.453180437 +0000 UTC m=+6.071684108 container cleanup 5b082a48fda491e9a6d3c106c1a72ede5bc67761d079eb0a2f7f54b3b987e154 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-d79a5420-e971-11ee-95cd-87774f69a715-osd-1, org.label-schema.build-date=20240319, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30) 2024-03-24T03:20:09.687 INFO:journalctl@ceph.osd.1.smithi189.stdout:Mar 24 03:20:09 smithi189 bash[117018]: ceph-d79a5420-e971-11ee-95cd-87774f69a715-osd-1 2024-03-24T03:20:09.953 INFO:journalctl@ceph.osd.1.smithi189.stdout:Mar 24 03:20:09 smithi189 podman[117036]: 2024-03-24 03:20:09.688192886 +0000 UTC m=+0.649298784 container remove 5b082a48fda491e9a6d3c106c1a72ede5bc67761d079eb0a2f7f54b3b987e154 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-d79a5420-e971-11ee-95cd-87774f69a715-osd-1, maintainer=Guillaume Abrioux , GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_CLEAN=True, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240319, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, ceph=True, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS) 2024-03-24T03:20:10.432 INFO:journalctl@ceph.osd.1.smithi189.stdout:Mar 24 03:20:10 smithi189 podman[117123]: 2024-03-24 03:20:10.020278964 +0000 UTC m=+0.018406924 image pull c664afe56bb01db4f9afa9532c20598e201e664f2e9ba2be50db0d52674a26d2 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc 2024-03-24T03:20:10.433 INFO:journalctl@ceph.osd.1.smithi189.stdout:Mar 24 03:20:10 smithi189 podman[117123]: 2024-03-24 03:20:10.122426806 +0000 UTC m=+0.120554766 container create 959a0273f12b4585322254e9f6978e790cd9ecefe11f3d49b381c4dd8741018f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-d79a5420-e971-11ee-95cd-87774f69a715-osd-1-deactivate, ceph=True, org.label-schema.build-date=20240319, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0) 2024-03-24T03:20:10.433 INFO:journalctl@ceph.osd.1.smithi189.stdout:Mar 24 03:20:10 smithi189 podman[117123]: 2024-03-24 03:20:10.341786422 +0000 UTC m=+0.339914380 container init 959a0273f12b4585322254e9f6978e790cd9ecefe11f3d49b381c4dd8741018f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-d79a5420-e971-11ee-95cd-87774f69a715-osd-1-deactivate, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, ceph=True, GIT_BRANCH=HEAD, GIT_CLEAN=True, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, org.label-schema.build-date=20240319, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30) 2024-03-24T03:20:10.433 INFO:journalctl@ceph.osd.1.smithi189.stdout:Mar 24 03:20:10 smithi189 podman[117123]: 2024-03-24 03:20:10.345835101 +0000 UTC m=+0.343963060 container start 959a0273f12b4585322254e9f6978e790cd9ecefe11f3d49b381c4dd8741018f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-d79a5420-e971-11ee-95cd-87774f69a715-osd-1-deactivate, org.label-schema.vendor=CentOS, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, ceph=True, org.label-schema.build-date=20240319, GIT_CLEAN=True, org.label-schema.schema-version=1.0) 2024-03-24T03:20:10.433 INFO:journalctl@ceph.osd.1.smithi189.stdout:Mar 24 03:20:10 smithi189 podman[117123]: 2024-03-24 03:20:10.433448156 +0000 UTC m=+0.431576134 container attach 959a0273f12b4585322254e9f6978e790cd9ecefe11f3d49b381c4dd8741018f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-d79a5420-e971-11ee-95cd-87774f69a715-osd-1-deactivate, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, GIT_CLEAN=True, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, ceph=True, org.label-schema.build-date=20240319, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS) 2024-03-24T03:20:10.704 INFO:journalctl@ceph.osd.1.smithi189.stdout:Mar 24 03:20:10 smithi189 podman[117123]: 2024-03-24 03:20:10.639483714 +0000 UTC m=+0.637611660 container died 959a0273f12b4585322254e9f6978e790cd9ecefe11f3d49b381c4dd8741018f (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-d79a5420-e971-11ee-95cd-87774f69a715-osd-1-deactivate, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240319, ceph=True, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.license=GPLv2) 2024-03-24T03:20:11.329 DEBUG:teuthology.orchestra.run.smithi189:> sudo pkill -f 'journalctl -f -n 0 -u ceph-d79a5420-e971-11ee-95cd-87774f69a715@osd.1.service' 2024-03-24T03:20:11.371 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-24T03:20:11.371 INFO:tasks.cephadm.osd.1:Stopped osd.1 2024-03-24T03:20:11.372 INFO:tasks.cephadm.osd.2:Stopping osd.2... 2024-03-24T03:20:11.372 DEBUG:teuthology.orchestra.run.smithi193:> sudo systemctl stop ceph-d79a5420-e971-11ee-95cd-87774f69a715@osd.2 2024-03-24T03:20:11.710 INFO:journalctl@ceph.osd.2.smithi193.stdout:Mar 24 03:20:11 smithi193 systemd[1]: Stopping Ceph osd.2 for d79a5420-e971-11ee-95cd-87774f69a715... 2024-03-24T03:20:12.210 INFO:journalctl@ceph.osd.2.smithi193.stdout:Mar 24 03:20:11 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-osd-2[42239]: 2024-03-24T03:20:11.762+0000 7fdccb501700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.2 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-24T03:20:12.211 INFO:journalctl@ceph.osd.2.smithi193.stdout:Mar 24 03:20:11 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-osd-2[42239]: 2024-03-24T03:20:11.762+0000 7fdccb501700 -1 osd.2 127 *** Got signal Terminated *** 2024-03-24T03:20:12.211 INFO:journalctl@ceph.osd.2.smithi193.stdout:Mar 24 03:20:11 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-osd-2[42239]: 2024-03-24T03:20:11.762+0000 7fdccb501700 -1 osd.2 127 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-24T03:20:17.556 INFO:journalctl@ceph.osd.2.smithi193.stdout:Mar 24 03:20:17 smithi193 podman[120303]: 2024-03-24 03:20:17.306087667 +0000 UTC m=+5.742284111 container died c5b2b368e96bdc607b25e567a8e2e8526f6ef6ee4e5ce3a366091181acff8046 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-d79a5420-e971-11ee-95cd-87774f69a715-osd-2, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_CLEAN=True, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240319, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True) 2024-03-24T03:20:17.889 INFO:journalctl@ceph.osd.2.smithi193.stdout:Mar 24 03:20:17 smithi193 podman[120303]: 2024-03-24 03:20:17.663545711 +0000 UTC m=+6.099742141 container cleanup c5b2b368e96bdc607b25e567a8e2e8526f6ef6ee4e5ce3a366091181acff8046 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-d79a5420-e971-11ee-95cd-87774f69a715-osd-2, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.build-date=20240319, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, GIT_CLEAN=True, org.label-schema.vendor=CentOS) 2024-03-24T03:20:17.889 INFO:journalctl@ceph.osd.2.smithi193.stdout:Mar 24 03:20:17 smithi193 bash[120303]: ceph-d79a5420-e971-11ee-95cd-87774f69a715-osd-2 2024-03-24T03:20:18.210 INFO:journalctl@ceph.osd.2.smithi193.stdout:Mar 24 03:20:17 smithi193 podman[120318]: 2024-03-24 03:20:17.890102349 +0000 UTC m=+0.579130586 container remove c5b2b368e96bdc607b25e567a8e2e8526f6ef6ee4e5ce3a366091181acff8046 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-d79a5420-e971-11ee-95cd-87774f69a715-osd-2, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240319, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True) 2024-03-24T03:20:18.609 INFO:journalctl@ceph.osd.2.smithi193.stdout:Mar 24 03:20:18 smithi193 podman[120411]: 2024-03-24 03:20:18.228248768 +0000 UTC m=+0.018169697 image pull c664afe56bb01db4f9afa9532c20598e201e664f2e9ba2be50db0d52674a26d2 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc 2024-03-24T03:20:18.610 INFO:journalctl@ceph.osd.2.smithi193.stdout:Mar 24 03:20:18 smithi193 podman[120411]: 2024-03-24 03:20:18.341051162 +0000 UTC m=+0.130972080 container create 20e0918e61ceee6228d7cdb26ee98a5b1239eb2ded275cbc1f1067d4ccbff70e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-d79a5420-e971-11ee-95cd-87774f69a715-osd-2-deactivate, ceph=True, GIT_BRANCH=HEAD, org.label-schema.build-date=20240319, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True) 2024-03-24T03:20:18.610 INFO:journalctl@ceph.osd.2.smithi193.stdout:Mar 24 03:20:18 smithi193 podman[120411]: 2024-03-24 03:20:18.610606459 +0000 UTC m=+0.400527388 container init 20e0918e61ceee6228d7cdb26ee98a5b1239eb2ded275cbc1f1067d4ccbff70e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-d79a5420-e971-11ee-95cd-87774f69a715-osd-2-deactivate, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.build-date=20240319, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, GIT_CLEAN=True, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c) 2024-03-24T03:20:18.898 INFO:journalctl@ceph.osd.2.smithi193.stdout:Mar 24 03:20:18 smithi193 podman[120411]: 2024-03-24 03:20:18.613851957 +0000 UTC m=+0.403772874 container start 20e0918e61ceee6228d7cdb26ee98a5b1239eb2ded275cbc1f1067d4ccbff70e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-d79a5420-e971-11ee-95cd-87774f69a715-osd-2-deactivate, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, GIT_CLEAN=True, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240319, org.label-schema.license=GPLv2) 2024-03-24T03:20:18.899 INFO:journalctl@ceph.osd.2.smithi193.stdout:Mar 24 03:20:18 smithi193 podman[120411]: 2024-03-24 03:20:18.694202572 +0000 UTC m=+0.484123495 container attach 20e0918e61ceee6228d7cdb26ee98a5b1239eb2ded275cbc1f1067d4ccbff70e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-d79a5420-e971-11ee-95cd-87774f69a715-osd-2-deactivate, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, ceph=True, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, GIT_CLEAN=True, org.label-schema.license=GPLv2, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240319) 2024-03-24T03:20:19.210 INFO:journalctl@ceph.osd.2.smithi193.stdout:Mar 24 03:20:18 smithi193 podman[120411]: 2024-03-24 03:20:18.899915427 +0000 UTC m=+0.689836360 container died 20e0918e61ceee6228d7cdb26ee98a5b1239eb2ded275cbc1f1067d4ccbff70e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-d79a5420-e971-11ee-95cd-87774f69a715-osd-2-deactivate, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, ceph=True, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240319, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_CLEAN=True, GIT_BRANCH=HEAD) 2024-03-24T03:20:19.599 DEBUG:teuthology.orchestra.run.smithi193:> sudo pkill -f 'journalctl -f -n 0 -u ceph-d79a5420-e971-11ee-95cd-87774f69a715@osd.2.service' 2024-03-24T03:20:19.639 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-24T03:20:19.639 INFO:tasks.cephadm.osd.2:Stopped osd.2 2024-03-24T03:20:19.639 INFO:tasks.cephadm.osd.3:Stopping osd.3... 2024-03-24T03:20:19.639 DEBUG:teuthology.orchestra.run.smithi118:> sudo systemctl stop ceph-d79a5420-e971-11ee-95cd-87774f69a715@osd.3 2024-03-24T03:20:20.196 INFO:journalctl@ceph.osd.3.smithi118.stdout:Mar 24 03:20:19 smithi118 systemd[1]: Stopping Ceph osd.3 for d79a5420-e971-11ee-95cd-87774f69a715... 2024-03-24T03:20:20.696 INFO:journalctl@ceph.osd.3.smithi118.stdout:Mar 24 03:20:20 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-osd-3[53301]: 2024-03-24T03:20:20.262+0000 7fca0f200700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.3 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-24T03:20:20.696 INFO:journalctl@ceph.osd.3.smithi118.stdout:Mar 24 03:20:20 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-osd-3[53301]: 2024-03-24T03:20:20.262+0000 7fca0f200700 -1 osd.3 127 *** Got signal Terminated *** 2024-03-24T03:20:20.696 INFO:journalctl@ceph.osd.3.smithi118.stdout:Mar 24 03:20:20 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-osd-3[53301]: 2024-03-24T03:20:20.262+0000 7fca0f200700 -1 osd.3 127 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-24T03:20:21.211 INFO:journalctl@ceph.osd.11.smithi193.stdout:Mar 24 03:20:20 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-osd-11[62452]: 2024-03-24T03:20:20.744+0000 7f0edd521700 -1 osd.11 127 heartbeat_check: no reply from 172.21.15.118:6806 osd.0 since back 2024-03-24T03:19:59.095638+0000 front 2024-03-24T03:19:59.095627+0000 (oldest deadline 2024-03-24T03:20:20.195530+0000) 2024-03-24T03:20:21.892 INFO:journalctl@ceph.osd.9.smithi118.stdout:Mar 24 03:20:21 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-osd-9[67027]: 2024-03-24T03:20:21.671+0000 7f69f7400700 -1 osd.9 127 heartbeat_check: no reply from 172.21.15.118:6806 osd.0 since back 2024-03-24T03:19:57.002010+0000 front 2024-03-24T03:19:57.002333+0000 (oldest deadline 2024-03-24T03:20:21.101732+0000) 2024-03-24T03:20:21.961 INFO:journalctl@ceph.osd.11.smithi193.stdout:Mar 24 03:20:21 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-osd-11[62452]: 2024-03-24T03:20:21.699+0000 7f0edd521700 -1 osd.11 127 heartbeat_check: no reply from 172.21.15.118:6806 osd.0 since back 2024-03-24T03:19:59.095638+0000 front 2024-03-24T03:19:59.095627+0000 (oldest deadline 2024-03-24T03:20:20.195530+0000) 2024-03-24T03:20:21.961 INFO:journalctl@ceph.osd.5.smithi193.stdout:Mar 24 03:20:21 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-osd-5[49201]: 2024-03-24T03:20:21.777+0000 7f48090ef700 -1 osd.5 127 heartbeat_check: no reply from 172.21.15.118:6806 osd.0 since back 2024-03-24T03:19:56.517618+0000 front 2024-03-24T03:19:56.517703+0000 (oldest deadline 2024-03-24T03:20:21.718373+0000) 2024-03-24T03:20:22.676 INFO:journalctl@ceph.osd.6.smithi118.stdout:Mar 24 03:20:22 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-osd-6[59976]: 2024-03-24T03:20:22.399+0000 7fbc40a00700 -1 osd.6 127 heartbeat_check: no reply from 172.21.15.118:6806 osd.0 since back 2024-03-24T03:19:56.910256+0000 front 2024-03-24T03:19:56.910419+0000 (oldest deadline 2024-03-24T03:20:21.610387+0000) 2024-03-24T03:20:22.946 INFO:journalctl@ceph.osd.9.smithi118.stdout:Mar 24 03:20:22 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-osd-9[67027]: 2024-03-24T03:20:22.674+0000 7f69f7400700 -1 osd.9 127 heartbeat_check: no reply from 172.21.15.118:6806 osd.0 since back 2024-03-24T03:19:57.002010+0000 front 2024-03-24T03:19:57.002333+0000 (oldest deadline 2024-03-24T03:20:21.101732+0000) 2024-03-24T03:20:23.203 INFO:journalctl@ceph.osd.10.smithi189.stdout:Mar 24 03:20:22 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-osd-10[61294]: 2024-03-24T03:20:22.948+0000 7f804e31f700 -1 osd.10 127 heartbeat_check: no reply from 172.21.15.118:6806 osd.0 since back 2024-03-24T03:19:56.942815+0000 front 2024-03-24T03:19:56.942951+0000 (oldest deadline 2024-03-24T03:20:22.842714+0000) 2024-03-24T03:20:23.210 INFO:journalctl@ceph.osd.5.smithi193.stdout:Mar 24 03:20:22 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-osd-5[49201]: 2024-03-24T03:20:22.736+0000 7f48090ef700 -1 osd.5 127 heartbeat_check: no reply from 172.21.15.118:6806 osd.0 since back 2024-03-24T03:19:56.517618+0000 front 2024-03-24T03:19:56.517703+0000 (oldest deadline 2024-03-24T03:20:21.718373+0000) 2024-03-24T03:20:23.210 INFO:journalctl@ceph.osd.11.smithi193.stdout:Mar 24 03:20:22 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-osd-11[62452]: 2024-03-24T03:20:22.721+0000 7f0edd521700 -1 osd.11 127 heartbeat_check: no reply from 172.21.15.118:6806 osd.0 since back 2024-03-24T03:19:59.095638+0000 front 2024-03-24T03:19:59.095627+0000 (oldest deadline 2024-03-24T03:20:20.195530+0000) 2024-03-24T03:20:23.696 INFO:journalctl@ceph.osd.9.smithi118.stdout:Mar 24 03:20:23 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-osd-9[67027]: 2024-03-24T03:20:23.626+0000 7f69f7400700 -1 osd.9 127 heartbeat_check: no reply from 172.21.15.118:6806 osd.0 since back 2024-03-24T03:19:57.002010+0000 front 2024-03-24T03:19:57.002333+0000 (oldest deadline 2024-03-24T03:20:21.101732+0000) 2024-03-24T03:20:23.696 INFO:journalctl@ceph.osd.6.smithi118.stdout:Mar 24 03:20:23 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-osd-6[59976]: 2024-03-24T03:20:23.408+0000 7fbc40a00700 -1 osd.6 127 heartbeat_check: no reply from 172.21.15.118:6806 osd.0 since back 2024-03-24T03:19:56.910256+0000 front 2024-03-24T03:19:56.910419+0000 (oldest deadline 2024-03-24T03:20:21.610387+0000) 2024-03-24T03:20:24.210 INFO:journalctl@ceph.osd.5.smithi193.stdout:Mar 24 03:20:23 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-osd-5[49201]: 2024-03-24T03:20:23.734+0000 7f48090ef700 -1 osd.5 127 heartbeat_check: no reply from 172.21.15.118:6806 osd.0 since back 2024-03-24T03:19:56.517618+0000 front 2024-03-24T03:19:56.517703+0000 (oldest deadline 2024-03-24T03:20:21.718373+0000) 2024-03-24T03:20:24.211 INFO:journalctl@ceph.osd.11.smithi193.stdout:Mar 24 03:20:23 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-osd-11[62452]: 2024-03-24T03:20:23.743+0000 7f0edd521700 -1 osd.11 127 heartbeat_check: no reply from 172.21.15.118:6806 osd.0 since back 2024-03-24T03:19:59.095638+0000 front 2024-03-24T03:19:59.095627+0000 (oldest deadline 2024-03-24T03:20:20.195530+0000) 2024-03-24T03:20:24.453 INFO:journalctl@ceph.osd.10.smithi189.stdout:Mar 24 03:20:23 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-osd-10[61294]: 2024-03-24T03:20:23.977+0000 7f804e31f700 -1 osd.10 127 heartbeat_check: no reply from 172.21.15.118:6806 osd.0 since back 2024-03-24T03:19:56.942815+0000 front 2024-03-24T03:19:56.942951+0000 (oldest deadline 2024-03-24T03:20:22.842714+0000) 2024-03-24T03:20:24.946 INFO:journalctl@ceph.osd.6.smithi118.stdout:Mar 24 03:20:24 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-osd-6[59976]: 2024-03-24T03:20:24.446+0000 7fbc40a00700 -1 osd.6 127 heartbeat_check: no reply from 172.21.15.118:6806 osd.0 since back 2024-03-24T03:19:56.910256+0000 front 2024-03-24T03:19:56.910419+0000 (oldest deadline 2024-03-24T03:20:21.610387+0000) 2024-03-24T03:20:24.946 INFO:journalctl@ceph.osd.9.smithi118.stdout:Mar 24 03:20:24 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-osd-9[67027]: 2024-03-24T03:20:24.663+0000 7f69f7400700 -1 osd.9 127 heartbeat_check: no reply from 172.21.15.118:6806 osd.0 since back 2024-03-24T03:19:57.002010+0000 front 2024-03-24T03:19:57.002333+0000 (oldest deadline 2024-03-24T03:20:21.101732+0000) 2024-03-24T03:20:24.960 INFO:journalctl@ceph.osd.5.smithi193.stdout:Mar 24 03:20:24 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-osd-5[49201]: 2024-03-24T03:20:24.698+0000 7f48090ef700 -1 osd.5 127 heartbeat_check: no reply from 172.21.15.118:6806 osd.0 since back 2024-03-24T03:19:56.517618+0000 front 2024-03-24T03:19:56.517703+0000 (oldest deadline 2024-03-24T03:20:21.718373+0000) 2024-03-24T03:20:24.961 INFO:journalctl@ceph.osd.11.smithi193.stdout:Mar 24 03:20:24 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-osd-11[62452]: 2024-03-24T03:20:24.775+0000 7f0edd521700 -1 osd.11 127 heartbeat_check: no reply from 172.21.15.118:6806 osd.0 since back 2024-03-24T03:19:59.095638+0000 front 2024-03-24T03:19:59.095627+0000 (oldest deadline 2024-03-24T03:20:20.195530+0000) 2024-03-24T03:20:25.203 INFO:journalctl@ceph.osd.10.smithi189.stdout:Mar 24 03:20:24 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-osd-10[61294]: 2024-03-24T03:20:24.940+0000 7f804e31f700 -1 osd.10 127 heartbeat_check: no reply from 172.21.15.118:6806 osd.0 since back 2024-03-24T03:19:56.942815+0000 front 2024-03-24T03:19:56.942951+0000 (oldest deadline 2024-03-24T03:20:22.842714+0000) 2024-03-24T03:20:25.815 INFO:journalctl@ceph.osd.6.smithi118.stdout:Mar 24 03:20:25 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-osd-6[59976]: 2024-03-24T03:20:25.484+0000 7fbc40a00700 -1 osd.6 127 heartbeat_check: no reply from 172.21.15.118:6806 osd.0 since back 2024-03-24T03:19:56.910256+0000 front 2024-03-24T03:19:56.910419+0000 (oldest deadline 2024-03-24T03:20:21.610387+0000) 2024-03-24T03:20:25.815 INFO:journalctl@ceph.osd.9.smithi118.stdout:Mar 24 03:20:25 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-osd-9[67027]: 2024-03-24T03:20:25.650+0000 7f69f7400700 -1 osd.9 127 heartbeat_check: no reply from 172.21.15.118:6806 osd.0 since back 2024-03-24T03:19:57.002010+0000 front 2024-03-24T03:19:57.002333+0000 (oldest deadline 2024-03-24T03:20:21.101732+0000) 2024-03-24T03:20:25.960 INFO:journalctl@ceph.osd.5.smithi193.stdout:Mar 24 03:20:25 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-osd-5[49201]: 2024-03-24T03:20:25.697+0000 7f48090ef700 -1 osd.5 127 heartbeat_check: no reply from 172.21.15.118:6806 osd.0 since back 2024-03-24T03:19:56.517618+0000 front 2024-03-24T03:19:56.517703+0000 (oldest deadline 2024-03-24T03:20:21.718373+0000) 2024-03-24T03:20:25.961 INFO:journalctl@ceph.osd.11.smithi193.stdout:Mar 24 03:20:25 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-osd-11[62452]: 2024-03-24T03:20:25.741+0000 7f0edd521700 -1 osd.11 127 heartbeat_check: no reply from 172.21.15.118:6806 osd.0 since back 2024-03-24T03:19:59.095638+0000 front 2024-03-24T03:19:59.095627+0000 (oldest deadline 2024-03-24T03:20:20.195530+0000) 2024-03-24T03:20:26.110 INFO:journalctl@ceph.osd.3.smithi118.stdout:Mar 24 03:20:25 smithi118 podman[146041]: 2024-03-24 03:20:25.815541417 +0000 UTC m=+5.702620584 container died 622b727a5611d0f336cd0bbdfe15e9ce6f16cf288cd8e4c668cdb342b9e2bbf7 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-d79a5420-e971-11ee-95cd-87774f69a715-osd-3, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, io.buildah.version=1.33.5, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, ceph=True, org.label-schema.build-date=20240319, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, GIT_CLEAN=True, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-24T03:20:26.204 INFO:journalctl@ceph.osd.10.smithi189.stdout:Mar 24 03:20:25 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-osd-10[61294]: 2024-03-24T03:20:25.931+0000 7f804e31f700 -1 osd.10 127 heartbeat_check: no reply from 172.21.15.118:6806 osd.0 since back 2024-03-24T03:19:56.942815+0000 front 2024-03-24T03:19:56.942951+0000 (oldest deadline 2024-03-24T03:20:22.842714+0000) 2024-03-24T03:20:26.446 INFO:journalctl@ceph.osd.3.smithi118.stdout:Mar 24 03:20:26 smithi118 podman[146041]: 2024-03-24 03:20:26.247355033 +0000 UTC m=+6.134434241 container cleanup 622b727a5611d0f336cd0bbdfe15e9ce6f16cf288cd8e4c668cdb342b9e2bbf7 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-d79a5420-e971-11ee-95cd-87774f69a715-osd-3, org.label-schema.vendor=CentOS, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, io.buildah.version=1.33.5, GIT_CLEAN=True, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, org.label-schema.build-date=20240319, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c) 2024-03-24T03:20:26.446 INFO:journalctl@ceph.osd.3.smithi118.stdout:Mar 24 03:20:26 smithi118 bash[146041]: ceph-d79a5420-e971-11ee-95cd-87774f69a715-osd-3 2024-03-24T03:20:26.946 INFO:journalctl@ceph.osd.6.smithi118.stdout:Mar 24 03:20:26 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-osd-6[59976]: 2024-03-24T03:20:26.481+0000 7fbc40a00700 -1 osd.6 127 heartbeat_check: no reply from 172.21.15.118:6806 osd.0 since back 2024-03-24T03:19:56.910256+0000 front 2024-03-24T03:19:56.910419+0000 (oldest deadline 2024-03-24T03:20:21.610387+0000) 2024-03-24T03:20:26.946 INFO:journalctl@ceph.osd.3.smithi118.stdout:Mar 24 03:20:26 smithi118 podman[146072]: 2024-03-24 03:20:26.532202205 +0000 UTC m=+0.702772277 container remove 622b727a5611d0f336cd0bbdfe15e9ce6f16cf288cd8e4c668cdb342b9e2bbf7 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-d79a5420-e971-11ee-95cd-87774f69a715-osd-3, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, org.label-schema.build-date=20240319, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, ceph=True, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30) 2024-03-24T03:20:26.947 INFO:journalctl@ceph.osd.9.smithi118.stdout:Mar 24 03:20:26 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-osd-9[67027]: 2024-03-24T03:20:26.668+0000 7f69f7400700 -1 osd.9 127 heartbeat_check: no reply from 172.21.15.118:6806 osd.0 since back 2024-03-24T03:19:57.002010+0000 front 2024-03-24T03:19:57.002333+0000 (oldest deadline 2024-03-24T03:20:21.101732+0000) 2024-03-24T03:20:26.960 INFO:journalctl@ceph.osd.5.smithi193.stdout:Mar 24 03:20:26 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-osd-5[49201]: 2024-03-24T03:20:26.695+0000 7f48090ef700 -1 osd.5 127 heartbeat_check: no reply from 172.21.15.118:6806 osd.0 since back 2024-03-24T03:19:56.517618+0000 front 2024-03-24T03:19:56.517703+0000 (oldest deadline 2024-03-24T03:20:21.718373+0000) 2024-03-24T03:20:26.960 INFO:journalctl@ceph.osd.11.smithi193.stdout:Mar 24 03:20:26 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-osd-11[62452]: 2024-03-24T03:20:26.732+0000 7f0edd521700 -1 osd.11 127 heartbeat_check: no reply from 172.21.15.118:6806 osd.0 since back 2024-03-24T03:19:59.095638+0000 front 2024-03-24T03:19:59.095627+0000 (oldest deadline 2024-03-24T03:20:20.195530+0000) 2024-03-24T03:20:27.206 INFO:journalctl@ceph.osd.10.smithi189.stdout:Mar 24 03:20:26 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-osd-10[61294]: 2024-03-24T03:20:26.892+0000 7f804e31f700 -1 osd.10 127 heartbeat_check: no reply from 172.21.15.118:6806 osd.0 since back 2024-03-24T03:19:56.942815+0000 front 2024-03-24T03:19:56.942951+0000 (oldest deadline 2024-03-24T03:20:22.842714+0000) 2024-03-24T03:20:27.653 INFO:journalctl@ceph.osd.6.smithi118.stdout:Mar 24 03:20:27 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-osd-6[59976]: 2024-03-24T03:20:27.483+0000 7fbc40a00700 -1 osd.6 127 heartbeat_check: no reply from 172.21.15.118:6806 osd.0 since back 2024-03-24T03:19:56.910256+0000 front 2024-03-24T03:19:56.910419+0000 (oldest deadline 2024-03-24T03:20:21.610387+0000) 2024-03-24T03:20:27.653 INFO:journalctl@ceph.osd.3.smithi118.stdout:Mar 24 03:20:27 smithi118 podman[146162]: 2024-03-24 03:20:27.255827787 +0000 UTC m=+0.035558990 image pull c664afe56bb01db4f9afa9532c20598e201e664f2e9ba2be50db0d52674a26d2 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc 2024-03-24T03:20:27.653 INFO:journalctl@ceph.osd.3.smithi118.stdout:Mar 24 03:20:27 smithi118 podman[146162]: 2024-03-24 03:20:27.374782279 +0000 UTC m=+0.154513523 container create 0151f1e73da4ceb7ed96fad52b84df6f4fe96d24cc3734c0f0674bece4f36402 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-d79a5420-e971-11ee-95cd-87774f69a715-osd-3-deactivate, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.schema-version=1.0, GIT_CLEAN=True, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, org.label-schema.build-date=20240319, ceph=True, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-24T03:20:27.946 INFO:journalctl@ceph.osd.9.smithi118.stdout:Mar 24 03:20:27 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-osd-9[67027]: 2024-03-24T03:20:27.685+0000 7f69f7400700 -1 osd.9 127 heartbeat_check: no reply from 172.21.15.118:6806 osd.0 since back 2024-03-24T03:19:57.002010+0000 front 2024-03-24T03:19:57.002333+0000 (oldest deadline 2024-03-24T03:20:21.101732+0000) 2024-03-24T03:20:27.946 INFO:journalctl@ceph.osd.3.smithi118.stdout:Mar 24 03:20:27 smithi118 podman[146162]: 2024-03-24 03:20:27.653363824 +0000 UTC m=+0.433095074 container init 0151f1e73da4ceb7ed96fad52b84df6f4fe96d24cc3734c0f0674bece4f36402 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-d79a5420-e971-11ee-95cd-87774f69a715-osd-3-deactivate, org.label-schema.license=GPLv2, org.label-schema.build-date=20240319, ceph=True, GIT_CLEAN=True, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30) 2024-03-24T03:20:27.946 INFO:journalctl@ceph.osd.3.smithi118.stdout:Mar 24 03:20:27 smithi118 podman[146162]: 2024-03-24 03:20:27.663983214 +0000 UTC m=+0.443714461 container start 0151f1e73da4ceb7ed96fad52b84df6f4fe96d24cc3734c0f0674bece4f36402 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-d79a5420-e971-11ee-95cd-87774f69a715-osd-3-deactivate, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, ceph=True, GIT_BRANCH=HEAD, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, org.label-schema.build-date=20240319, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_CLEAN=True) 2024-03-24T03:20:27.946 INFO:journalctl@ceph.osd.3.smithi118.stdout:Mar 24 03:20:27 smithi118 podman[146162]: 2024-03-24 03:20:27.752481311 +0000 UTC m=+0.532212591 container attach 0151f1e73da4ceb7ed96fad52b84df6f4fe96d24cc3734c0f0674bece4f36402 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-d79a5420-e971-11ee-95cd-87774f69a715-osd-3-deactivate, GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.build-date=20240319, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS) 2024-03-24T03:20:27.960 INFO:journalctl@ceph.osd.5.smithi193.stdout:Mar 24 03:20:27 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-osd-5[49201]: 2024-03-24T03:20:27.657+0000 7f48090ef700 -1 osd.5 127 heartbeat_check: no reply from 172.21.15.118:6806 osd.0 since back 2024-03-24T03:19:56.517618+0000 front 2024-03-24T03:19:56.517703+0000 (oldest deadline 2024-03-24T03:20:21.718373+0000) 2024-03-24T03:20:27.961 INFO:journalctl@ceph.osd.11.smithi193.stdout:Mar 24 03:20:27 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-osd-11[62452]: 2024-03-24T03:20:27.728+0000 7f0edd521700 -1 osd.11 127 heartbeat_check: no reply from 172.21.15.118:6806 osd.0 since back 2024-03-24T03:19:59.095638+0000 front 2024-03-24T03:19:59.095627+0000 (oldest deadline 2024-03-24T03:20:20.195530+0000) 2024-03-24T03:20:28.203 INFO:journalctl@ceph.osd.10.smithi189.stdout:Mar 24 03:20:27 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-osd-10[61294]: 2024-03-24T03:20:27.937+0000 7f804e31f700 -1 osd.10 127 heartbeat_check: no reply from 172.21.15.118:6806 osd.0 since back 2024-03-24T03:19:56.942815+0000 front 2024-03-24T03:19:56.942951+0000 (oldest deadline 2024-03-24T03:20:22.842714+0000) 2024-03-24T03:20:28.443 INFO:journalctl@ceph.osd.3.smithi118.stdout:Mar 24 03:20:28 smithi118 podman[146162]: 2024-03-24 03:20:28.094945406 +0000 UTC m=+0.874676636 container died 0151f1e73da4ceb7ed96fad52b84df6f4fe96d24cc3734c0f0674bece4f36402 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-d79a5420-e971-11ee-95cd-87774f69a715-osd-3-deactivate, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240319, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, CEPH_POINT_RELEASE=, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2) 2024-03-24T03:20:28.696 INFO:journalctl@ceph.osd.6.smithi118.stdout:Mar 24 03:20:28 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-osd-6[59976]: 2024-03-24T03:20:28.470+0000 7fbc40a00700 -1 osd.6 127 heartbeat_check: no reply from 172.21.15.118:6806 osd.0 since back 2024-03-24T03:19:56.910256+0000 front 2024-03-24T03:19:56.910419+0000 (oldest deadline 2024-03-24T03:20:21.610387+0000) 2024-03-24T03:20:28.696 INFO:journalctl@ceph.osd.9.smithi118.stdout:Mar 24 03:20:28 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-osd-9[67027]: 2024-03-24T03:20:28.647+0000 7f69f7400700 -1 osd.9 127 heartbeat_check: no reply from 172.21.15.118:6806 osd.0 since back 2024-03-24T03:19:57.002010+0000 front 2024-03-24T03:19:57.002333+0000 (oldest deadline 2024-03-24T03:20:21.101732+0000) 2024-03-24T03:20:28.859 DEBUG:teuthology.orchestra.run.smithi118:> sudo pkill -f 'journalctl -f -n 0 -u ceph-d79a5420-e971-11ee-95cd-87774f69a715@osd.3.service' 2024-03-24T03:20:28.960 INFO:journalctl@ceph.osd.5.smithi193.stdout:Mar 24 03:20:28 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-osd-5[49201]: 2024-03-24T03:20:28.689+0000 7f48090ef700 -1 osd.5 127 heartbeat_check: no reply from 172.21.15.118:6806 osd.0 since back 2024-03-24T03:19:56.517618+0000 front 2024-03-24T03:19:56.517703+0000 (oldest deadline 2024-03-24T03:20:21.718373+0000) 2024-03-24T03:20:28.961 INFO:journalctl@ceph.osd.11.smithi193.stdout:Mar 24 03:20:28 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-osd-11[62452]: 2024-03-24T03:20:28.712+0000 7f0edd521700 -1 osd.11 127 heartbeat_check: no reply from 172.21.15.118:6806 osd.0 since back 2024-03-24T03:19:59.095638+0000 front 2024-03-24T03:19:59.095627+0000 (oldest deadline 2024-03-24T03:20:20.195530+0000) 2024-03-24T03:20:28.976 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-24T03:20:28.977 INFO:tasks.cephadm.osd.3:Stopped osd.3 2024-03-24T03:20:28.977 INFO:tasks.cephadm.osd.4:Stopping osd.4... 2024-03-24T03:20:28.977 DEBUG:teuthology.orchestra.run.smithi189:> sudo systemctl stop ceph-d79a5420-e971-11ee-95cd-87774f69a715@osd.4 2024-03-24T03:20:29.340 INFO:journalctl@ceph.osd.4.smithi189.stdout:Mar 24 03:20:29 smithi189 systemd[1]: Stopping Ceph osd.4 for d79a5420-e971-11ee-95cd-87774f69a715... 2024-03-24T03:20:29.341 INFO:journalctl@ceph.osd.10.smithi189.stdout:Mar 24 03:20:28 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-osd-10[61294]: 2024-03-24T03:20:28.964+0000 7f804e31f700 -1 osd.10 127 heartbeat_check: no reply from 172.21.15.118:6806 osd.0 since back 2024-03-24T03:19:56.942815+0000 front 2024-03-24T03:19:56.942951+0000 (oldest deadline 2024-03-24T03:20:22.842714+0000) 2024-03-24T03:20:29.703 INFO:journalctl@ceph.osd.4.smithi189.stdout:Mar 24 03:20:29 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-osd-4[47788]: 2024-03-24T03:20:29.339+0000 7f1b4a57c700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.4 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-24T03:20:29.704 INFO:journalctl@ceph.osd.4.smithi189.stdout:Mar 24 03:20:29 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-osd-4[47788]: 2024-03-24T03:20:29.339+0000 7f1b4a57c700 -1 osd.4 127 *** Got signal Terminated *** 2024-03-24T03:20:29.704 INFO:journalctl@ceph.osd.4.smithi189.stdout:Mar 24 03:20:29 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-osd-4[47788]: 2024-03-24T03:20:29.339+0000 7f1b4a57c700 -1 osd.4 127 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-24T03:20:29.946 INFO:journalctl@ceph.osd.6.smithi118.stdout:Mar 24 03:20:29 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-osd-6[59976]: 2024-03-24T03:20:29.469+0000 7fbc40a00700 -1 osd.6 127 heartbeat_check: no reply from 172.21.15.118:6806 osd.0 since back 2024-03-24T03:19:56.910256+0000 front 2024-03-24T03:19:56.910419+0000 (oldest deadline 2024-03-24T03:20:21.610387+0000) 2024-03-24T03:20:29.946 INFO:journalctl@ceph.osd.9.smithi118.stdout:Mar 24 03:20:29 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-osd-9[67027]: 2024-03-24T03:20:29.640+0000 7f69f7400700 -1 osd.9 127 heartbeat_check: no reply from 172.21.15.118:6806 osd.0 since back 2024-03-24T03:19:57.002010+0000 front 2024-03-24T03:19:57.002333+0000 (oldest deadline 2024-03-24T03:20:21.101732+0000) 2024-03-24T03:20:29.946 INFO:journalctl@ceph.osd.9.smithi118.stdout:Mar 24 03:20:29 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-osd-9[67027]: 2024-03-24T03:20:29.640+0000 7f69f7400700 -1 osd.9 127 heartbeat_check: no reply from 172.21.15.189:6804 osd.1 since back 2024-03-24T03:20:05.804520+0000 front 2024-03-24T03:20:05.804379+0000 (oldest deadline 2024-03-24T03:20:29.303978+0000) 2024-03-24T03:20:29.960 INFO:journalctl@ceph.osd.5.smithi193.stdout:Mar 24 03:20:29 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-osd-5[49201]: 2024-03-24T03:20:29.703+0000 7f48090ef700 -1 osd.5 127 heartbeat_check: no reply from 172.21.15.118:6806 osd.0 since back 2024-03-24T03:19:56.517618+0000 front 2024-03-24T03:19:56.517703+0000 (oldest deadline 2024-03-24T03:20:21.718373+0000) 2024-03-24T03:20:29.960 INFO:journalctl@ceph.osd.11.smithi193.stdout:Mar 24 03:20:29 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-osd-11[62452]: 2024-03-24T03:20:29.761+0000 7f0edd521700 -1 osd.11 127 heartbeat_check: no reply from 172.21.15.118:6806 osd.0 since back 2024-03-24T03:19:59.095638+0000 front 2024-03-24T03:19:59.095627+0000 (oldest deadline 2024-03-24T03:20:20.195530+0000) 2024-03-24T03:20:30.204 INFO:journalctl@ceph.osd.4.smithi189.stdout:Mar 24 03:20:29 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-osd-4[47788]: 2024-03-24T03:20:29.767+0000 7f1b42b68700 -1 osd.4 127 heartbeat_check: no reply from 172.21.15.189:6804 osd.1 since back 2024-03-24T03:20:03.439628+0000 front 2024-03-24T03:20:03.439649+0000 (oldest deadline 2024-03-24T03:20:29.240483+0000) 2024-03-24T03:20:30.204 INFO:journalctl@ceph.osd.10.smithi189.stdout:Mar 24 03:20:29 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-osd-10[61294]: 2024-03-24T03:20:29.942+0000 7f804e31f700 -1 osd.10 127 heartbeat_check: no reply from 172.21.15.118:6806 osd.0 since back 2024-03-24T03:19:56.942815+0000 front 2024-03-24T03:19:56.942951+0000 (oldest deadline 2024-03-24T03:20:22.842714+0000) 2024-03-24T03:20:30.696 INFO:journalctl@ceph.osd.6.smithi118.stdout:Mar 24 03:20:30 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-osd-6[59976]: 2024-03-24T03:20:30.437+0000 7fbc40a00700 -1 osd.6 127 heartbeat_check: no reply from 172.21.15.118:6806 osd.0 since back 2024-03-24T03:19:56.910256+0000 front 2024-03-24T03:19:56.910419+0000 (oldest deadline 2024-03-24T03:20:21.610387+0000) 2024-03-24T03:20:30.697 INFO:journalctl@ceph.osd.6.smithi118.stdout:Mar 24 03:20:30 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-osd-6[59976]: 2024-03-24T03:20:30.437+0000 7fbc40a00700 -1 osd.6 127 heartbeat_check: no reply from 172.21.15.189:6804 osd.1 since back 2024-03-24T03:20:05.515241+0000 front 2024-03-24T03:20:05.515363+0000 (oldest deadline 2024-03-24T03:20:29.615481+0000) 2024-03-24T03:20:30.697 INFO:journalctl@ceph.osd.9.smithi118.stdout:Mar 24 03:20:30 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-osd-9[67027]: 2024-03-24T03:20:30.634+0000 7f69f7400700 -1 osd.9 127 heartbeat_check: no reply from 172.21.15.118:6806 osd.0 since back 2024-03-24T03:19:57.002010+0000 front 2024-03-24T03:19:57.002333+0000 (oldest deadline 2024-03-24T03:20:21.101732+0000) 2024-03-24T03:20:30.697 INFO:journalctl@ceph.osd.9.smithi118.stdout:Mar 24 03:20:30 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-osd-9[67027]: 2024-03-24T03:20:30.634+0000 7f69f7400700 -1 osd.9 127 heartbeat_check: no reply from 172.21.15.189:6804 osd.1 since back 2024-03-24T03:20:05.804520+0000 front 2024-03-24T03:20:05.804379+0000 (oldest deadline 2024-03-24T03:20:29.303978+0000) 2024-03-24T03:20:30.960 INFO:journalctl@ceph.osd.5.smithi193.stdout:Mar 24 03:20:30 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-osd-5[49201]: 2024-03-24T03:20:30.688+0000 7f48090ef700 -1 osd.5 127 heartbeat_check: no reply from 172.21.15.118:6806 osd.0 since back 2024-03-24T03:19:56.517618+0000 front 2024-03-24T03:19:56.517703+0000 (oldest deadline 2024-03-24T03:20:21.718373+0000) 2024-03-24T03:20:30.961 INFO:journalctl@ceph.osd.11.smithi193.stdout:Mar 24 03:20:30 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-osd-11[62452]: 2024-03-24T03:20:30.747+0000 7f0edd521700 -1 osd.11 127 heartbeat_check: no reply from 172.21.15.118:6806 osd.0 since back 2024-03-24T03:19:59.095638+0000 front 2024-03-24T03:19:59.095627+0000 (oldest deadline 2024-03-24T03:20:20.195530+0000) 2024-03-24T03:20:31.203 INFO:journalctl@ceph.osd.4.smithi189.stdout:Mar 24 03:20:30 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-osd-4[47788]: 2024-03-24T03:20:30.811+0000 7f1b42b68700 -1 osd.4 127 heartbeat_check: no reply from 172.21.15.189:6804 osd.1 since back 2024-03-24T03:20:03.439628+0000 front 2024-03-24T03:20:03.439649+0000 (oldest deadline 2024-03-24T03:20:29.240483+0000) 2024-03-24T03:20:31.204 INFO:journalctl@ceph.osd.7.smithi189.stdout:Mar 24 03:20:30 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-osd-7[54385]: 2024-03-24T03:20:30.801+0000 7fb775b24700 -1 osd.7 127 heartbeat_check: no reply from 172.21.15.189:6804 osd.1 since back 2024-03-24T03:20:04.667909+0000 front 2024-03-24T03:20:04.667932+0000 (oldest deadline 2024-03-24T03:20:29.968062+0000) 2024-03-24T03:20:31.204 INFO:journalctl@ceph.osd.10.smithi189.stdout:Mar 24 03:20:30 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-osd-10[61294]: 2024-03-24T03:20:30.913+0000 7f804e31f700 -1 osd.10 127 heartbeat_check: no reply from 172.21.15.118:6806 osd.0 since back 2024-03-24T03:19:56.942815+0000 front 2024-03-24T03:19:56.942951+0000 (oldest deadline 2024-03-24T03:20:22.842714+0000) 2024-03-24T03:20:31.460 INFO:journalctl@ceph.osd.8.smithi193.stdout:Mar 24 03:20:30 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-osd-8[55794]: 2024-03-24T03:20:30.963+0000 7f6e8b995700 -1 osd.8 127 heartbeat_check: no reply from 172.21.15.189:6804 osd.1 since back 2024-03-24T03:20:06.219689+0000 front 2024-03-24T03:20:06.219789+0000 (oldest deadline 2024-03-24T03:20:30.919690+0000) 2024-03-24T03:20:31.696 INFO:journalctl@ceph.osd.6.smithi118.stdout:Mar 24 03:20:31 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-osd-6[59976]: 2024-03-24T03:20:31.427+0000 7fbc40a00700 -1 osd.6 127 heartbeat_check: no reply from 172.21.15.118:6806 osd.0 since back 2024-03-24T03:19:56.910256+0000 front 2024-03-24T03:19:56.910419+0000 (oldest deadline 2024-03-24T03:20:21.610387+0000) 2024-03-24T03:20:31.696 INFO:journalctl@ceph.osd.6.smithi118.stdout:Mar 24 03:20:31 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-osd-6[59976]: 2024-03-24T03:20:31.427+0000 7fbc40a00700 -1 osd.6 127 heartbeat_check: no reply from 172.21.15.189:6804 osd.1 since back 2024-03-24T03:20:05.515241+0000 front 2024-03-24T03:20:05.515363+0000 (oldest deadline 2024-03-24T03:20:29.615481+0000) 2024-03-24T03:20:31.696 INFO:journalctl@ceph.osd.9.smithi118.stdout:Mar 24 03:20:31 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-osd-9[67027]: 2024-03-24T03:20:31.657+0000 7f69f7400700 -1 osd.9 127 heartbeat_check: no reply from 172.21.15.118:6806 osd.0 since back 2024-03-24T03:19:57.002010+0000 front 2024-03-24T03:19:57.002333+0000 (oldest deadline 2024-03-24T03:20:21.101732+0000) 2024-03-24T03:20:31.696 INFO:journalctl@ceph.osd.9.smithi118.stdout:Mar 24 03:20:31 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-osd-9[67027]: 2024-03-24T03:20:31.657+0000 7f69f7400700 -1 osd.9 127 heartbeat_check: no reply from 172.21.15.189:6804 osd.1 since back 2024-03-24T03:20:05.804520+0000 front 2024-03-24T03:20:05.804379+0000 (oldest deadline 2024-03-24T03:20:29.303978+0000) 2024-03-24T03:20:31.991 INFO:journalctl@ceph.osd.11.smithi193.stdout:Mar 24 03:20:31 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-osd-11[62452]: 2024-03-24T03:20:31.732+0000 7f0edd521700 -1 osd.11 127 heartbeat_check: no reply from 172.21.15.118:6806 osd.0 since back 2024-03-24T03:19:59.095638+0000 front 2024-03-24T03:19:59.095627+0000 (oldest deadline 2024-03-24T03:20:20.195530+0000) 2024-03-24T03:20:31.991 INFO:journalctl@ceph.osd.5.smithi193.stdout:Mar 24 03:20:31 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-osd-5[49201]: 2024-03-24T03:20:31.723+0000 7f48090ef700 -1 osd.5 127 heartbeat_check: no reply from 172.21.15.118:6806 osd.0 since back 2024-03-24T03:19:56.517618+0000 front 2024-03-24T03:19:56.517703+0000 (oldest deadline 2024-03-24T03:20:21.718373+0000) 2024-03-24T03:20:31.991 INFO:journalctl@ceph.osd.5.smithi193.stdout:Mar 24 03:20:31 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-osd-5[49201]: 2024-03-24T03:20:31.723+0000 7f48090ef700 -1 osd.5 127 heartbeat_check: no reply from 172.21.15.189:6804 osd.1 since back 2024-03-24T03:20:06.419750+0000 front 2024-03-24T03:20:06.419585+0000 (oldest deadline 2024-03-24T03:20:31.020174+0000) 2024-03-24T03:20:32.203 INFO:journalctl@ceph.osd.7.smithi189.stdout:Mar 24 03:20:31 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-osd-7[54385]: 2024-03-24T03:20:31.777+0000 7fb775b24700 -1 osd.7 127 heartbeat_check: no reply from 172.21.15.189:6804 osd.1 since back 2024-03-24T03:20:04.667909+0000 front 2024-03-24T03:20:04.667932+0000 (oldest deadline 2024-03-24T03:20:29.968062+0000) 2024-03-24T03:20:32.204 INFO:journalctl@ceph.osd.4.smithi189.stdout:Mar 24 03:20:31 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-osd-4[47788]: 2024-03-24T03:20:31.772+0000 7f1b42b68700 -1 osd.4 127 heartbeat_check: no reply from 172.21.15.189:6804 osd.1 since back 2024-03-24T03:20:03.439628+0000 front 2024-03-24T03:20:03.439649+0000 (oldest deadline 2024-03-24T03:20:29.240483+0000) 2024-03-24T03:20:32.204 INFO:journalctl@ceph.osd.10.smithi189.stdout:Mar 24 03:20:31 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-osd-10[61294]: 2024-03-24T03:20:31.906+0000 7f804e31f700 -1 osd.10 127 heartbeat_check: no reply from 172.21.15.118:6806 osd.0 since back 2024-03-24T03:19:56.942815+0000 front 2024-03-24T03:19:56.942951+0000 (oldest deadline 2024-03-24T03:20:22.842714+0000) 2024-03-24T03:20:32.204 INFO:journalctl@ceph.osd.10.smithi189.stdout:Mar 24 03:20:31 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-osd-10[61294]: 2024-03-24T03:20:31.906+0000 7f804e31f700 -1 osd.10 127 heartbeat_check: no reply from 172.21.15.189:6804 osd.1 since back 2024-03-24T03:20:07.543899+0000 front 2024-03-24T03:20:07.543880+0000 (oldest deadline 2024-03-24T03:20:30.944770+0000) 2024-03-24T03:20:32.460 INFO:journalctl@ceph.osd.8.smithi193.stdout:Mar 24 03:20:31 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-osd-8[55794]: 2024-03-24T03:20:31.991+0000 7f6e8b995700 -1 osd.8 127 heartbeat_check: no reply from 172.21.15.189:6804 osd.1 since back 2024-03-24T03:20:06.219689+0000 front 2024-03-24T03:20:06.219789+0000 (oldest deadline 2024-03-24T03:20:30.919690+0000) 2024-03-24T03:20:32.664 INFO:journalctl@ceph.osd.6.smithi118.stdout:Mar 24 03:20:32 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-osd-6[59976]: 2024-03-24T03:20:32.385+0000 7fbc40a00700 -1 osd.6 127 heartbeat_check: no reply from 172.21.15.118:6806 osd.0 since back 2024-03-24T03:19:56.910256+0000 front 2024-03-24T03:19:56.910419+0000 (oldest deadline 2024-03-24T03:20:21.610387+0000) 2024-03-24T03:20:32.664 INFO:journalctl@ceph.osd.6.smithi118.stdout:Mar 24 03:20:32 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-osd-6[59976]: 2024-03-24T03:20:32.385+0000 7fbc40a00700 -1 osd.6 127 heartbeat_check: no reply from 172.21.15.189:6804 osd.1 since back 2024-03-24T03:20:05.515241+0000 front 2024-03-24T03:20:05.515363+0000 (oldest deadline 2024-03-24T03:20:29.615481+0000) 2024-03-24T03:20:32.946 INFO:journalctl@ceph.osd.9.smithi118.stdout:Mar 24 03:20:32 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-osd-9[67027]: 2024-03-24T03:20:32.662+0000 7f69f7400700 -1 osd.9 127 heartbeat_check: no reply from 172.21.15.118:6806 osd.0 since back 2024-03-24T03:19:57.002010+0000 front 2024-03-24T03:19:57.002333+0000 (oldest deadline 2024-03-24T03:20:21.101732+0000) 2024-03-24T03:20:32.946 INFO:journalctl@ceph.osd.9.smithi118.stdout:Mar 24 03:20:32 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-osd-9[67027]: 2024-03-24T03:20:32.663+0000 7f69f7400700 -1 osd.9 127 heartbeat_check: no reply from 172.21.15.189:6804 osd.1 since back 2024-03-24T03:20:05.804520+0000 front 2024-03-24T03:20:05.804379+0000 (oldest deadline 2024-03-24T03:20:29.303978+0000) 2024-03-24T03:20:33.203 INFO:journalctl@ceph.osd.4.smithi189.stdout:Mar 24 03:20:32 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-osd-4[47788]: 2024-03-24T03:20:32.747+0000 7f1b42b68700 -1 osd.4 127 heartbeat_check: no reply from 172.21.15.189:6804 osd.1 since back 2024-03-24T03:20:03.439628+0000 front 2024-03-24T03:20:03.439649+0000 (oldest deadline 2024-03-24T03:20:29.240483+0000) 2024-03-24T03:20:33.204 INFO:journalctl@ceph.osd.7.smithi189.stdout:Mar 24 03:20:32 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-osd-7[54385]: 2024-03-24T03:20:32.788+0000 7fb775b24700 -1 osd.7 127 heartbeat_check: no reply from 172.21.15.189:6804 osd.1 since back 2024-03-24T03:20:04.667909+0000 front 2024-03-24T03:20:04.667932+0000 (oldest deadline 2024-03-24T03:20:29.968062+0000) 2024-03-24T03:20:33.204 INFO:journalctl@ceph.osd.10.smithi189.stdout:Mar 24 03:20:32 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-osd-10[61294]: 2024-03-24T03:20:32.879+0000 7f804e31f700 -1 osd.10 127 heartbeat_check: no reply from 172.21.15.118:6806 osd.0 since back 2024-03-24T03:19:56.942815+0000 front 2024-03-24T03:19:56.942951+0000 (oldest deadline 2024-03-24T03:20:22.842714+0000) 2024-03-24T03:20:33.204 INFO:journalctl@ceph.osd.10.smithi189.stdout:Mar 24 03:20:32 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-osd-10[61294]: 2024-03-24T03:20:32.879+0000 7f804e31f700 -1 osd.10 127 heartbeat_check: no reply from 172.21.15.189:6804 osd.1 since back 2024-03-24T03:20:07.543899+0000 front 2024-03-24T03:20:07.543880+0000 (oldest deadline 2024-03-24T03:20:30.944770+0000) 2024-03-24T03:20:33.211 INFO:journalctl@ceph.osd.8.smithi193.stdout:Mar 24 03:20:32 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-osd-8[55794]: 2024-03-24T03:20:32.948+0000 7f6e8b995700 -1 osd.8 127 heartbeat_check: no reply from 172.21.15.189:6804 osd.1 since back 2024-03-24T03:20:06.219689+0000 front 2024-03-24T03:20:06.219789+0000 (oldest deadline 2024-03-24T03:20:30.919690+0000) 2024-03-24T03:20:33.211 INFO:journalctl@ceph.osd.5.smithi193.stdout:Mar 24 03:20:32 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-osd-5[49201]: 2024-03-24T03:20:32.726+0000 7f48090ef700 -1 osd.5 127 heartbeat_check: no reply from 172.21.15.118:6806 osd.0 since back 2024-03-24T03:19:56.517618+0000 front 2024-03-24T03:19:56.517703+0000 (oldest deadline 2024-03-24T03:20:21.718373+0000) 2024-03-24T03:20:33.211 INFO:journalctl@ceph.osd.5.smithi193.stdout:Mar 24 03:20:32 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-osd-5[49201]: 2024-03-24T03:20:32.726+0000 7f48090ef700 -1 osd.5 127 heartbeat_check: no reply from 172.21.15.189:6804 osd.1 since back 2024-03-24T03:20:06.419750+0000 front 2024-03-24T03:20:06.419585+0000 (oldest deadline 2024-03-24T03:20:31.020174+0000) 2024-03-24T03:20:33.211 INFO:journalctl@ceph.osd.11.smithi193.stdout:Mar 24 03:20:32 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-osd-11[62452]: 2024-03-24T03:20:32.744+0000 7f0edd521700 -1 osd.11 127 heartbeat_check: no reply from 172.21.15.118:6806 osd.0 since back 2024-03-24T03:19:59.095638+0000 front 2024-03-24T03:19:59.095627+0000 (oldest deadline 2024-03-24T03:20:20.195530+0000) 2024-03-24T03:20:33.212 INFO:journalctl@ceph.osd.11.smithi193.stdout:Mar 24 03:20:32 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-osd-11[62452]: 2024-03-24T03:20:32.744+0000 7f0edd521700 -1 osd.11 127 heartbeat_check: no reply from 172.21.15.189:6804 osd.1 since back 2024-03-24T03:20:07.399753+0000 front 2024-03-24T03:20:07.399812+0000 (oldest deadline 2024-03-24T03:20:32.099923+0000) 2024-03-24T03:20:33.642 INFO:journalctl@ceph.osd.6.smithi118.stdout:Mar 24 03:20:33 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-osd-6[59976]: 2024-03-24T03:20:33.339+0000 7fbc40a00700 -1 osd.6 127 heartbeat_check: no reply from 172.21.15.118:6806 osd.0 since back 2024-03-24T03:19:56.910256+0000 front 2024-03-24T03:19:56.910419+0000 (oldest deadline 2024-03-24T03:20:21.610387+0000) 2024-03-24T03:20:33.642 INFO:journalctl@ceph.osd.6.smithi118.stdout:Mar 24 03:20:33 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-osd-6[59976]: 2024-03-24T03:20:33.339+0000 7fbc40a00700 -1 osd.6 127 heartbeat_check: no reply from 172.21.15.189:6804 osd.1 since back 2024-03-24T03:20:05.515241+0000 front 2024-03-24T03:20:05.515363+0000 (oldest deadline 2024-03-24T03:20:29.615481+0000) 2024-03-24T03:20:33.946 INFO:journalctl@ceph.osd.9.smithi118.stdout:Mar 24 03:20:33 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-osd-9[67027]: 2024-03-24T03:20:33.641+0000 7f69f7400700 -1 osd.9 127 heartbeat_check: no reply from 172.21.15.118:6806 osd.0 since back 2024-03-24T03:19:57.002010+0000 front 2024-03-24T03:19:57.002333+0000 (oldest deadline 2024-03-24T03:20:21.101732+0000) 2024-03-24T03:20:33.946 INFO:journalctl@ceph.osd.9.smithi118.stdout:Mar 24 03:20:33 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-osd-9[67027]: 2024-03-24T03:20:33.641+0000 7f69f7400700 -1 osd.9 127 heartbeat_check: no reply from 172.21.15.189:6804 osd.1 since back 2024-03-24T03:20:05.804520+0000 front 2024-03-24T03:20:05.804379+0000 (oldest deadline 2024-03-24T03:20:29.303978+0000) 2024-03-24T03:20:34.203 INFO:journalctl@ceph.osd.4.smithi189.stdout:Mar 24 03:20:33 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-osd-4[47788]: 2024-03-24T03:20:33.777+0000 7f1b42b68700 -1 osd.4 127 heartbeat_check: no reply from 172.21.15.189:6804 osd.1 since back 2024-03-24T03:20:03.439628+0000 front 2024-03-24T03:20:03.439649+0000 (oldest deadline 2024-03-24T03:20:29.240483+0000) 2024-03-24T03:20:34.204 INFO:journalctl@ceph.osd.7.smithi189.stdout:Mar 24 03:20:33 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-osd-7[54385]: 2024-03-24T03:20:33.748+0000 7fb775b24700 -1 osd.7 127 heartbeat_check: no reply from 172.21.15.189:6804 osd.1 since back 2024-03-24T03:20:04.667909+0000 front 2024-03-24T03:20:04.667932+0000 (oldest deadline 2024-03-24T03:20:29.968062+0000) 2024-03-24T03:20:34.204 INFO:journalctl@ceph.osd.10.smithi189.stdout:Mar 24 03:20:33 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-osd-10[61294]: 2024-03-24T03:20:33.896+0000 7f804e31f700 -1 osd.10 127 heartbeat_check: no reply from 172.21.15.118:6806 osd.0 since back 2024-03-24T03:19:56.942815+0000 front 2024-03-24T03:19:56.942951+0000 (oldest deadline 2024-03-24T03:20:22.842714+0000) 2024-03-24T03:20:34.204 INFO:journalctl@ceph.osd.10.smithi189.stdout:Mar 24 03:20:33 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-osd-10[61294]: 2024-03-24T03:20:33.896+0000 7f804e31f700 -1 osd.10 127 heartbeat_check: no reply from 172.21.15.189:6804 osd.1 since back 2024-03-24T03:20:07.543899+0000 front 2024-03-24T03:20:07.543880+0000 (oldest deadline 2024-03-24T03:20:30.944770+0000) 2024-03-24T03:20:34.210 INFO:journalctl@ceph.osd.8.smithi193.stdout:Mar 24 03:20:33 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-osd-8[55794]: 2024-03-24T03:20:33.903+0000 7f6e8b995700 -1 osd.8 127 heartbeat_check: no reply from 172.21.15.189:6804 osd.1 since back 2024-03-24T03:20:06.219689+0000 front 2024-03-24T03:20:06.219789+0000 (oldest deadline 2024-03-24T03:20:30.919690+0000) 2024-03-24T03:20:34.210 INFO:journalctl@ceph.osd.5.smithi193.stdout:Mar 24 03:20:33 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-osd-5[49201]: 2024-03-24T03:20:33.734+0000 7f48090ef700 -1 osd.5 127 heartbeat_check: no reply from 172.21.15.118:6806 osd.0 since back 2024-03-24T03:19:56.517618+0000 front 2024-03-24T03:19:56.517703+0000 (oldest deadline 2024-03-24T03:20:21.718373+0000) 2024-03-24T03:20:34.211 INFO:journalctl@ceph.osd.5.smithi193.stdout:Mar 24 03:20:33 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-osd-5[49201]: 2024-03-24T03:20:33.734+0000 7f48090ef700 -1 osd.5 127 heartbeat_check: no reply from 172.21.15.189:6804 osd.1 since back 2024-03-24T03:20:06.419750+0000 front 2024-03-24T03:20:06.419585+0000 (oldest deadline 2024-03-24T03:20:31.020174+0000) 2024-03-24T03:20:34.211 INFO:journalctl@ceph.osd.11.smithi193.stdout:Mar 24 03:20:33 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-osd-11[62452]: 2024-03-24T03:20:33.751+0000 7f0edd521700 -1 osd.11 127 heartbeat_check: no reply from 172.21.15.118:6806 osd.0 since back 2024-03-24T03:19:59.095638+0000 front 2024-03-24T03:19:59.095627+0000 (oldest deadline 2024-03-24T03:20:20.195530+0000) 2024-03-24T03:20:34.211 INFO:journalctl@ceph.osd.11.smithi193.stdout:Mar 24 03:20:33 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-osd-11[62452]: 2024-03-24T03:20:33.751+0000 7f0edd521700 -1 osd.11 127 heartbeat_check: no reply from 172.21.15.189:6804 osd.1 since back 2024-03-24T03:20:07.399753+0000 front 2024-03-24T03:20:07.399812+0000 (oldest deadline 2024-03-24T03:20:32.099923+0000) 2024-03-24T03:20:34.682 INFO:journalctl@ceph.osd.6.smithi118.stdout:Mar 24 03:20:34 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-osd-6[59976]: 2024-03-24T03:20:34.319+0000 7fbc40a00700 -1 osd.6 127 heartbeat_check: no reply from 172.21.15.118:6806 osd.0 since back 2024-03-24T03:19:56.910256+0000 front 2024-03-24T03:19:56.910419+0000 (oldest deadline 2024-03-24T03:20:21.610387+0000) 2024-03-24T03:20:34.682 INFO:journalctl@ceph.osd.6.smithi118.stdout:Mar 24 03:20:34 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-osd-6[59976]: 2024-03-24T03:20:34.319+0000 7fbc40a00700 -1 osd.6 127 heartbeat_check: no reply from 172.21.15.189:6804 osd.1 since back 2024-03-24T03:20:05.515241+0000 front 2024-03-24T03:20:05.515363+0000 (oldest deadline 2024-03-24T03:20:29.615481+0000) 2024-03-24T03:20:34.946 INFO:journalctl@ceph.osd.9.smithi118.stdout:Mar 24 03:20:34 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-osd-9[67027]: 2024-03-24T03:20:34.681+0000 7f69f7400700 -1 osd.9 127 heartbeat_check: no reply from 172.21.15.118:6806 osd.0 since back 2024-03-24T03:19:57.002010+0000 front 2024-03-24T03:19:57.002333+0000 (oldest deadline 2024-03-24T03:20:21.101732+0000) 2024-03-24T03:20:34.946 INFO:journalctl@ceph.osd.9.smithi118.stdout:Mar 24 03:20:34 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-osd-9[67027]: 2024-03-24T03:20:34.681+0000 7f69f7400700 -1 osd.9 127 heartbeat_check: no reply from 172.21.15.189:6804 osd.1 since back 2024-03-24T03:20:05.804520+0000 front 2024-03-24T03:20:05.804379+0000 (oldest deadline 2024-03-24T03:20:29.303978+0000) 2024-03-24T03:20:35.099 INFO:journalctl@ceph.osd.4.smithi189.stdout:Mar 24 03:20:34 smithi189 podman[117243]: 2024-03-24 03:20:34.799306401 +0000 UTC m=+5.629736369 container died f019d42b8ff1bb4e29c42d79161c2273e58857afa1b9b11487d6d3bbd5d34d5b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-d79a5420-e971-11ee-95cd-87774f69a715-osd-4, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, ceph=True, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, org.label-schema.build-date=20240319) 2024-03-24T03:20:35.100 INFO:journalctl@ceph.osd.7.smithi189.stdout:Mar 24 03:20:34 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-osd-7[54385]: 2024-03-24T03:20:34.740+0000 7fb775b24700 -1 osd.7 127 heartbeat_check: no reply from 172.21.15.189:6804 osd.1 since back 2024-03-24T03:20:04.667909+0000 front 2024-03-24T03:20:04.667932+0000 (oldest deadline 2024-03-24T03:20:29.968062+0000) 2024-03-24T03:20:35.100 INFO:journalctl@ceph.osd.10.smithi189.stdout:Mar 24 03:20:34 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-osd-10[61294]: 2024-03-24T03:20:34.909+0000 7f804e31f700 -1 osd.10 127 heartbeat_check: no reply from 172.21.15.118:6806 osd.0 since back 2024-03-24T03:19:56.942815+0000 front 2024-03-24T03:19:56.942951+0000 (oldest deadline 2024-03-24T03:20:22.842714+0000) 2024-03-24T03:20:35.100 INFO:journalctl@ceph.osd.10.smithi189.stdout:Mar 24 03:20:34 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-osd-10[61294]: 2024-03-24T03:20:34.909+0000 7f804e31f700 -1 osd.10 127 heartbeat_check: no reply from 172.21.15.189:6804 osd.1 since back 2024-03-24T03:20:07.543899+0000 front 2024-03-24T03:20:07.543880+0000 (oldest deadline 2024-03-24T03:20:30.944770+0000) 2024-03-24T03:20:35.210 INFO:journalctl@ceph.osd.8.smithi193.stdout:Mar 24 03:20:34 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-osd-8[55794]: 2024-03-24T03:20:34.946+0000 7f6e8b995700 -1 osd.8 127 heartbeat_check: no reply from 172.21.15.189:6804 osd.1 since back 2024-03-24T03:20:06.219689+0000 front 2024-03-24T03:20:06.219789+0000 (oldest deadline 2024-03-24T03:20:30.919690+0000) 2024-03-24T03:20:35.211 INFO:journalctl@ceph.osd.5.smithi193.stdout:Mar 24 03:20:34 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-osd-5[49201]: 2024-03-24T03:20:34.747+0000 7f48090ef700 -1 osd.5 127 heartbeat_check: no reply from 172.21.15.118:6806 osd.0 since back 2024-03-24T03:19:56.517618+0000 front 2024-03-24T03:19:56.517703+0000 (oldest deadline 2024-03-24T03:20:21.718373+0000) 2024-03-24T03:20:35.211 INFO:journalctl@ceph.osd.5.smithi193.stdout:Mar 24 03:20:34 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-osd-5[49201]: 2024-03-24T03:20:34.747+0000 7f48090ef700 -1 osd.5 127 heartbeat_check: no reply from 172.21.15.189:6804 osd.1 since back 2024-03-24T03:20:06.419750+0000 front 2024-03-24T03:20:06.419585+0000 (oldest deadline 2024-03-24T03:20:31.020174+0000) 2024-03-24T03:20:35.211 INFO:journalctl@ceph.osd.11.smithi193.stdout:Mar 24 03:20:34 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-osd-11[62452]: 2024-03-24T03:20:34.754+0000 7f0edd521700 -1 osd.11 127 heartbeat_check: no reply from 172.21.15.118:6806 osd.0 since back 2024-03-24T03:19:59.095638+0000 front 2024-03-24T03:19:59.095627+0000 (oldest deadline 2024-03-24T03:20:20.195530+0000) 2024-03-24T03:20:35.211 INFO:journalctl@ceph.osd.11.smithi193.stdout:Mar 24 03:20:34 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-osd-11[62452]: 2024-03-24T03:20:34.754+0000 7f0edd521700 -1 osd.11 127 heartbeat_check: no reply from 172.21.15.189:6804 osd.1 since back 2024-03-24T03:20:07.399753+0000 front 2024-03-24T03:20:07.399812+0000 (oldest deadline 2024-03-24T03:20:32.099923+0000) 2024-03-24T03:20:35.441 INFO:journalctl@ceph.osd.4.smithi189.stdout:Mar 24 03:20:35 smithi189 podman[117243]: 2024-03-24 03:20:35.199087246 +0000 UTC m=+6.029517232 container cleanup f019d42b8ff1bb4e29c42d79161c2273e58857afa1b9b11487d6d3bbd5d34d5b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-d79a5420-e971-11ee-95cd-87774f69a715-osd-4, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_BRANCH=HEAD, GIT_CLEAN=True, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240319, io.buildah.version=1.33.5, ceph=True) 2024-03-24T03:20:35.441 INFO:journalctl@ceph.osd.4.smithi189.stdout:Mar 24 03:20:35 smithi189 bash[117243]: ceph-d79a5420-e971-11ee-95cd-87774f69a715-osd-4 2024-03-24T03:20:35.696 INFO:journalctl@ceph.osd.6.smithi118.stdout:Mar 24 03:20:35 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-osd-6[59976]: 2024-03-24T03:20:35.335+0000 7fbc40a00700 -1 osd.6 127 heartbeat_check: no reply from 172.21.15.118:6806 osd.0 since back 2024-03-24T03:19:56.910256+0000 front 2024-03-24T03:19:56.910419+0000 (oldest deadline 2024-03-24T03:20:21.610387+0000) 2024-03-24T03:20:35.696 INFO:journalctl@ceph.osd.6.smithi118.stdout:Mar 24 03:20:35 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-osd-6[59976]: 2024-03-24T03:20:35.335+0000 7fbc40a00700 -1 osd.6 127 heartbeat_check: no reply from 172.21.15.189:6804 osd.1 since back 2024-03-24T03:20:05.515241+0000 front 2024-03-24T03:20:05.515363+0000 (oldest deadline 2024-03-24T03:20:29.615481+0000) 2024-03-24T03:20:35.703 INFO:journalctl@ceph.osd.4.smithi189.stdout:Mar 24 03:20:35 smithi189 podman[117260]: 2024-03-24 03:20:35.442303808 +0000 UTC m=+0.644764188 container remove f019d42b8ff1bb4e29c42d79161c2273e58857afa1b9b11487d6d3bbd5d34d5b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-d79a5420-e971-11ee-95cd-87774f69a715-osd-4, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, ceph=True, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240319, org.label-schema.vendor=CentOS, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, GIT_CLEAN=True) 2024-03-24T03:20:36.021 INFO:journalctl@ceph.osd.7.smithi189.stdout:Mar 24 03:20:35 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-osd-7[54385]: 2024-03-24T03:20:35.725+0000 7fb775b24700 -1 osd.7 127 heartbeat_check: no reply from 172.21.15.189:6804 osd.1 since back 2024-03-24T03:20:04.667909+0000 front 2024-03-24T03:20:04.667932+0000 (oldest deadline 2024-03-24T03:20:29.968062+0000) 2024-03-24T03:20:36.022 INFO:journalctl@ceph.osd.4.smithi189.stdout:Mar 24 03:20:35 smithi189 podman[117350]: 2024-03-24 03:20:35.766455637 +0000 UTC m=+0.017359618 image pull c664afe56bb01db4f9afa9532c20598e201e664f2e9ba2be50db0d52674a26d2 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc 2024-03-24T03:20:36.022 INFO:journalctl@ceph.osd.4.smithi189.stdout:Mar 24 03:20:35 smithi189 podman[117350]: 2024-03-24 03:20:35.893292202 +0000 UTC m=+0.144196160 container create 67fc0fdd2d2764f5c584e9d9cfb2ede531b7dbbc3b6f92d3dcfbf546a9f9c3c9 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-d79a5420-e971-11ee-95cd-87774f69a715-osd-4-deactivate, org.label-schema.vendor=CentOS, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.build-date=20240319, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=) 2024-03-24T03:20:36.022 INFO:journalctl@ceph.osd.10.smithi189.stdout:Mar 24 03:20:35 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-osd-10[61294]: 2024-03-24T03:20:35.874+0000 7f804e31f700 -1 osd.10 127 heartbeat_check: no reply from 172.21.15.118:6806 osd.0 since back 2024-03-24T03:19:56.942815+0000 front 2024-03-24T03:19:56.942951+0000 (oldest deadline 2024-03-24T03:20:22.842714+0000) 2024-03-24T03:20:36.022 INFO:journalctl@ceph.osd.10.smithi189.stdout:Mar 24 03:20:35 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-osd-10[61294]: 2024-03-24T03:20:35.874+0000 7f804e31f700 -1 osd.10 127 heartbeat_check: no reply from 172.21.15.189:6804 osd.1 since back 2024-03-24T03:20:07.543899+0000 front 2024-03-24T03:20:07.543880+0000 (oldest deadline 2024-03-24T03:20:30.944770+0000) 2024-03-24T03:20:36.196 INFO:journalctl@ceph.osd.9.smithi118.stdout:Mar 24 03:20:35 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-osd-9[67027]: 2024-03-24T03:20:35.725+0000 7f69f7400700 -1 osd.9 127 heartbeat_check: no reply from 172.21.15.118:6806 osd.0 since back 2024-03-24T03:19:57.002010+0000 front 2024-03-24T03:19:57.002333+0000 (oldest deadline 2024-03-24T03:20:21.101732+0000) 2024-03-24T03:20:36.196 INFO:journalctl@ceph.osd.9.smithi118.stdout:Mar 24 03:20:35 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-osd-9[67027]: 2024-03-24T03:20:35.725+0000 7f69f7400700 -1 osd.9 127 heartbeat_check: no reply from 172.21.15.189:6804 osd.1 since back 2024-03-24T03:20:05.804520+0000 front 2024-03-24T03:20:05.804379+0000 (oldest deadline 2024-03-24T03:20:29.303978+0000) 2024-03-24T03:20:36.210 INFO:journalctl@ceph.osd.8.smithi193.stdout:Mar 24 03:20:35 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-osd-8[55794]: 2024-03-24T03:20:35.973+0000 7f6e8b995700 -1 osd.8 127 heartbeat_check: no reply from 172.21.15.189:6804 osd.1 since back 2024-03-24T03:20:06.219689+0000 front 2024-03-24T03:20:06.219789+0000 (oldest deadline 2024-03-24T03:20:30.919690+0000) 2024-03-24T03:20:36.211 INFO:journalctl@ceph.osd.5.smithi193.stdout:Mar 24 03:20:35 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-osd-5[49201]: 2024-03-24T03:20:35.723+0000 7f48090ef700 -1 osd.5 127 heartbeat_check: no reply from 172.21.15.118:6806 osd.0 since back 2024-03-24T03:19:56.517618+0000 front 2024-03-24T03:19:56.517703+0000 (oldest deadline 2024-03-24T03:20:21.718373+0000) 2024-03-24T03:20:36.211 INFO:journalctl@ceph.osd.5.smithi193.stdout:Mar 24 03:20:35 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-osd-5[49201]: 2024-03-24T03:20:35.723+0000 7f48090ef700 -1 osd.5 127 heartbeat_check: no reply from 172.21.15.189:6804 osd.1 since back 2024-03-24T03:20:06.419750+0000 front 2024-03-24T03:20:06.419585+0000 (oldest deadline 2024-03-24T03:20:31.020174+0000) 2024-03-24T03:20:36.211 INFO:journalctl@ceph.osd.11.smithi193.stdout:Mar 24 03:20:35 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-osd-11[62452]: 2024-03-24T03:20:35.803+0000 7f0edd521700 -1 osd.11 127 heartbeat_check: no reply from 172.21.15.118:6806 osd.0 since back 2024-03-24T03:19:59.095638+0000 front 2024-03-24T03:19:59.095627+0000 (oldest deadline 2024-03-24T03:20:20.195530+0000) 2024-03-24T03:20:36.211 INFO:journalctl@ceph.osd.11.smithi193.stdout:Mar 24 03:20:35 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-osd-11[62452]: 2024-03-24T03:20:35.803+0000 7f0edd521700 -1 osd.11 127 heartbeat_check: no reply from 172.21.15.189:6804 osd.1 since back 2024-03-24T03:20:07.399753+0000 front 2024-03-24T03:20:07.399812+0000 (oldest deadline 2024-03-24T03:20:32.099923+0000) 2024-03-24T03:20:36.433 INFO:journalctl@ceph.osd.4.smithi189.stdout:Mar 24 03:20:36 smithi189 podman[117350]: 2024-03-24 03:20:36.146016888 +0000 UTC m=+0.396920860 container init 67fc0fdd2d2764f5c584e9d9cfb2ede531b7dbbc3b6f92d3dcfbf546a9f9c3c9 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-d79a5420-e971-11ee-95cd-87774f69a715-osd-4-deactivate, org.label-schema.schema-version=1.0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, org.label-schema.build-date=20240319, io.buildah.version=1.33.5, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c) 2024-03-24T03:20:36.434 INFO:journalctl@ceph.osd.4.smithi189.stdout:Mar 24 03:20:36 smithi189 podman[117350]: 2024-03-24 03:20:36.149262718 +0000 UTC m=+0.400166676 container start 67fc0fdd2d2764f5c584e9d9cfb2ede531b7dbbc3b6f92d3dcfbf546a9f9c3c9 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-d79a5420-e971-11ee-95cd-87774f69a715-osd-4-deactivate, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, GIT_CLEAN=True, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, org.label-schema.build-date=20240319, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, ceph=True) 2024-03-24T03:20:36.434 INFO:journalctl@ceph.osd.4.smithi189.stdout:Mar 24 03:20:36 smithi189 podman[117350]: 2024-03-24 03:20:36.237661325 +0000 UTC m=+0.488565297 container attach 67fc0fdd2d2764f5c584e9d9cfb2ede531b7dbbc3b6f92d3dcfbf546a9f9c3c9 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-d79a5420-e971-11ee-95cd-87774f69a715-osd-4-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240319, ceph=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, CEPH_POINT_RELEASE=) 2024-03-24T03:20:36.696 INFO:journalctl@ceph.osd.6.smithi118.stdout:Mar 24 03:20:36 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-osd-6[59976]: 2024-03-24T03:20:36.361+0000 7fbc40a00700 -1 osd.6 127 heartbeat_check: no reply from 172.21.15.118:6806 osd.0 since back 2024-03-24T03:19:56.910256+0000 front 2024-03-24T03:19:56.910419+0000 (oldest deadline 2024-03-24T03:20:21.610387+0000) 2024-03-24T03:20:36.696 INFO:journalctl@ceph.osd.6.smithi118.stdout:Mar 24 03:20:36 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-osd-6[59976]: 2024-03-24T03:20:36.361+0000 7fbc40a00700 -1 osd.6 127 heartbeat_check: no reply from 172.21.15.189:6804 osd.1 since back 2024-03-24T03:20:05.515241+0000 front 2024-03-24T03:20:05.515363+0000 (oldest deadline 2024-03-24T03:20:29.615481+0000) 2024-03-24T03:20:36.703 INFO:journalctl@ceph.osd.4.smithi189.stdout:Mar 24 03:20:36 smithi189 podman[117350]: 2024-03-24 03:20:36.434976008 +0000 UTC m=+0.685879985 container died 67fc0fdd2d2764f5c584e9d9cfb2ede531b7dbbc3b6f92d3dcfbf546a9f9c3c9 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-d79a5420-e971-11ee-95cd-87774f69a715-osd-4-deactivate, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, org.label-schema.build-date=20240319, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.schema-version=1.0, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, maintainer=Guillaume Abrioux , ceph=True) 2024-03-24T03:20:37.099 INFO:journalctl@ceph.osd.7.smithi189.stdout:Mar 24 03:20:36 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-osd-7[54385]: 2024-03-24T03:20:36.741+0000 7fb775b24700 -1 osd.7 127 heartbeat_check: no reply from 172.21.15.189:6804 osd.1 since back 2024-03-24T03:20:04.667909+0000 front 2024-03-24T03:20:04.667932+0000 (oldest deadline 2024-03-24T03:20:29.968062+0000) 2024-03-24T03:20:37.099 INFO:journalctl@ceph.osd.10.smithi189.stdout:Mar 24 03:20:36 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-osd-10[61294]: 2024-03-24T03:20:36.892+0000 7f804e31f700 -1 osd.10 127 heartbeat_check: no reply from 172.21.15.118:6806 osd.0 since back 2024-03-24T03:19:56.942815+0000 front 2024-03-24T03:19:56.942951+0000 (oldest deadline 2024-03-24T03:20:22.842714+0000) 2024-03-24T03:20:37.100 INFO:journalctl@ceph.osd.10.smithi189.stdout:Mar 24 03:20:36 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-osd-10[61294]: 2024-03-24T03:20:36.892+0000 7f804e31f700 -1 osd.10 127 heartbeat_check: no reply from 172.21.15.189:6804 osd.1 since back 2024-03-24T03:20:07.543899+0000 front 2024-03-24T03:20:07.543880+0000 (oldest deadline 2024-03-24T03:20:30.944770+0000) 2024-03-24T03:20:37.135 DEBUG:teuthology.orchestra.run.smithi189:> sudo pkill -f 'journalctl -f -n 0 -u ceph-d79a5420-e971-11ee-95cd-87774f69a715@osd.4.service' 2024-03-24T03:20:37.176 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-24T03:20:37.176 INFO:tasks.cephadm.osd.4:Stopped osd.4 2024-03-24T03:20:37.176 INFO:tasks.cephadm.osd.5:Stopping osd.5... 2024-03-24T03:20:37.177 DEBUG:teuthology.orchestra.run.smithi193:> sudo systemctl stop ceph-d79a5420-e971-11ee-95cd-87774f69a715@osd.5 2024-03-24T03:20:37.196 INFO:journalctl@ceph.osd.9.smithi118.stdout:Mar 24 03:20:36 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-osd-9[67027]: 2024-03-24T03:20:36.750+0000 7f69f7400700 -1 osd.9 127 heartbeat_check: no reply from 172.21.15.118:6806 osd.0 since back 2024-03-24T03:19:57.002010+0000 front 2024-03-24T03:19:57.002333+0000 (oldest deadline 2024-03-24T03:20:21.101732+0000) 2024-03-24T03:20:37.196 INFO:journalctl@ceph.osd.9.smithi118.stdout:Mar 24 03:20:36 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-osd-9[67027]: 2024-03-24T03:20:36.750+0000 7f69f7400700 -1 osd.9 127 heartbeat_check: no reply from 172.21.15.189:6804 osd.1 since back 2024-03-24T03:20:05.804520+0000 front 2024-03-24T03:20:05.804379+0000 (oldest deadline 2024-03-24T03:20:29.303978+0000) 2024-03-24T03:20:37.204 INFO:journalctl@ceph.osd.8.smithi193.stdout:Mar 24 03:20:36 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-osd-8[55794]: 2024-03-24T03:20:36.932+0000 7f6e8b995700 -1 osd.8 127 heartbeat_check: no reply from 172.21.15.189:6804 osd.1 since back 2024-03-24T03:20:06.219689+0000 front 2024-03-24T03:20:06.219789+0000 (oldest deadline 2024-03-24T03:20:30.919690+0000) 2024-03-24T03:20:37.205 INFO:journalctl@ceph.osd.11.smithi193.stdout:Mar 24 03:20:36 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-osd-11[62452]: 2024-03-24T03:20:36.793+0000 7f0edd521700 -1 osd.11 127 heartbeat_check: no reply from 172.21.15.118:6806 osd.0 since back 2024-03-24T03:19:59.095638+0000 front 2024-03-24T03:19:59.095627+0000 (oldest deadline 2024-03-24T03:20:20.195530+0000) 2024-03-24T03:20:37.205 INFO:journalctl@ceph.osd.11.smithi193.stdout:Mar 24 03:20:36 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-osd-11[62452]: 2024-03-24T03:20:36.793+0000 7f0edd521700 -1 osd.11 127 heartbeat_check: no reply from 172.21.15.189:6804 osd.1 since back 2024-03-24T03:20:07.399753+0000 front 2024-03-24T03:20:07.399812+0000 (oldest deadline 2024-03-24T03:20:32.099923+0000) 2024-03-24T03:20:37.205 INFO:journalctl@ceph.osd.5.smithi193.stdout:Mar 24 03:20:36 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-osd-5[49201]: 2024-03-24T03:20:36.724+0000 7f48090ef700 -1 osd.5 127 heartbeat_check: no reply from 172.21.15.118:6806 osd.0 since back 2024-03-24T03:19:56.517618+0000 front 2024-03-24T03:19:56.517703+0000 (oldest deadline 2024-03-24T03:20:21.718373+0000) 2024-03-24T03:20:37.205 INFO:journalctl@ceph.osd.5.smithi193.stdout:Mar 24 03:20:36 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-osd-5[49201]: 2024-03-24T03:20:36.724+0000 7f48090ef700 -1 osd.5 127 heartbeat_check: no reply from 172.21.15.189:6804 osd.1 since back 2024-03-24T03:20:06.419750+0000 front 2024-03-24T03:20:06.419585+0000 (oldest deadline 2024-03-24T03:20:31.020174+0000) 2024-03-24T03:20:37.460 INFO:journalctl@ceph.osd.5.smithi193.stdout:Mar 24 03:20:37 smithi193 systemd[1]: Stopping Ceph osd.5 for d79a5420-e971-11ee-95cd-87774f69a715... 2024-03-24T03:20:37.696 INFO:journalctl@ceph.osd.6.smithi118.stdout:Mar 24 03:20:37 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-osd-6[59976]: 2024-03-24T03:20:37.409+0000 7fbc40a00700 -1 osd.6 127 heartbeat_check: no reply from 172.21.15.118:6806 osd.0 since back 2024-03-24T03:19:56.910256+0000 front 2024-03-24T03:19:56.910419+0000 (oldest deadline 2024-03-24T03:20:21.610387+0000) 2024-03-24T03:20:37.696 INFO:journalctl@ceph.osd.6.smithi118.stdout:Mar 24 03:20:37 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-osd-6[59976]: 2024-03-24T03:20:37.409+0000 7fbc40a00700 -1 osd.6 127 heartbeat_check: no reply from 172.21.15.189:6804 osd.1 since back 2024-03-24T03:20:05.515241+0000 front 2024-03-24T03:20:05.515363+0000 (oldest deadline 2024-03-24T03:20:29.615481+0000) 2024-03-24T03:20:37.945 INFO:journalctl@ceph.osd.11.smithi193.stdout:Mar 24 03:20:37 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-osd-11[62452]: 2024-03-24T03:20:37.775+0000 7f0edd521700 -1 osd.11 127 heartbeat_check: no reply from 172.21.15.118:6806 osd.0 since back 2024-03-24T03:19:59.095638+0000 front 2024-03-24T03:19:59.095627+0000 (oldest deadline 2024-03-24T03:20:20.195530+0000) 2024-03-24T03:20:37.946 INFO:journalctl@ceph.osd.11.smithi193.stdout:Mar 24 03:20:37 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-osd-11[62452]: 2024-03-24T03:20:37.775+0000 7f0edd521700 -1 osd.11 127 heartbeat_check: no reply from 172.21.15.189:6804 osd.1 since back 2024-03-24T03:20:07.399753+0000 front 2024-03-24T03:20:07.399812+0000 (oldest deadline 2024-03-24T03:20:32.099923+0000) 2024-03-24T03:20:37.946 INFO:journalctl@ceph.osd.5.smithi193.stdout:Mar 24 03:20:37 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-osd-5[49201]: 2024-03-24T03:20:37.533+0000 7f480cafb700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.5 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-24T03:20:37.946 INFO:journalctl@ceph.osd.5.smithi193.stdout:Mar 24 03:20:37 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-osd-5[49201]: 2024-03-24T03:20:37.533+0000 7f480cafb700 -1 osd.5 127 *** Got signal Terminated *** 2024-03-24T03:20:37.946 INFO:journalctl@ceph.osd.5.smithi193.stdout:Mar 24 03:20:37 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-osd-5[49201]: 2024-03-24T03:20:37.533+0000 7f480cafb700 -1 osd.5 127 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-24T03:20:37.947 INFO:journalctl@ceph.osd.5.smithi193.stdout:Mar 24 03:20:37 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-osd-5[49201]: 2024-03-24T03:20:37.763+0000 7f48090ef700 -1 osd.5 127 heartbeat_check: no reply from 172.21.15.118:6806 osd.0 since back 2024-03-24T03:19:56.517618+0000 front 2024-03-24T03:19:56.517703+0000 (oldest deadline 2024-03-24T03:20:21.718373+0000) 2024-03-24T03:20:37.947 INFO:journalctl@ceph.osd.5.smithi193.stdout:Mar 24 03:20:37 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-osd-5[49201]: 2024-03-24T03:20:37.763+0000 7f48090ef700 -1 osd.5 127 heartbeat_check: no reply from 172.21.15.189:6804 osd.1 since back 2024-03-24T03:20:06.419750+0000 front 2024-03-24T03:20:06.419585+0000 (oldest deadline 2024-03-24T03:20:31.020174+0000) 2024-03-24T03:20:38.193 INFO:journalctl@ceph.osd.9.smithi118.stdout:Mar 24 03:20:37 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-osd-9[67027]: 2024-03-24T03:20:37.722+0000 7f69f7400700 -1 osd.9 127 heartbeat_check: no reply from 172.21.15.118:6806 osd.0 since back 2024-03-24T03:19:57.002010+0000 front 2024-03-24T03:19:57.002333+0000 (oldest deadline 2024-03-24T03:20:21.101732+0000) 2024-03-24T03:20:38.193 INFO:journalctl@ceph.osd.9.smithi118.stdout:Mar 24 03:20:37 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-osd-9[67027]: 2024-03-24T03:20:37.722+0000 7f69f7400700 -1 osd.9 127 heartbeat_check: no reply from 172.21.15.189:6804 osd.1 since back 2024-03-24T03:20:05.804520+0000 front 2024-03-24T03:20:05.804379+0000 (oldest deadline 2024-03-24T03:20:29.303978+0000) 2024-03-24T03:20:38.203 INFO:journalctl@ceph.osd.7.smithi189.stdout:Mar 24 03:20:37 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-osd-7[54385]: 2024-03-24T03:20:37.735+0000 7fb775b24700 -1 osd.7 127 heartbeat_check: no reply from 172.21.15.189:6804 osd.1 since back 2024-03-24T03:20:04.667909+0000 front 2024-03-24T03:20:04.667932+0000 (oldest deadline 2024-03-24T03:20:29.968062+0000) 2024-03-24T03:20:38.203 INFO:journalctl@ceph.osd.7.smithi189.stdout:Mar 24 03:20:37 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-osd-7[54385]: 2024-03-24T03:20:37.735+0000 7fb775b24700 -1 osd.7 127 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-03-24T03:20:15.769900+0000 front 2024-03-24T03:20:15.769860+0000 (oldest deadline 2024-03-24T03:20:37.470057+0000) 2024-03-24T03:20:38.204 INFO:journalctl@ceph.osd.10.smithi189.stdout:Mar 24 03:20:37 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-osd-10[61294]: 2024-03-24T03:20:37.908+0000 7f804e31f700 -1 osd.10 127 heartbeat_check: no reply from 172.21.15.118:6806 osd.0 since back 2024-03-24T03:19:56.942815+0000 front 2024-03-24T03:19:56.942951+0000 (oldest deadline 2024-03-24T03:20:22.842714+0000) 2024-03-24T03:20:38.204 INFO:journalctl@ceph.osd.10.smithi189.stdout:Mar 24 03:20:37 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-osd-10[61294]: 2024-03-24T03:20:37.908+0000 7f804e31f700 -1 osd.10 127 heartbeat_check: no reply from 172.21.15.189:6804 osd.1 since back 2024-03-24T03:20:07.543899+0000 front 2024-03-24T03:20:07.543880+0000 (oldest deadline 2024-03-24T03:20:30.944770+0000) 2024-03-24T03:20:38.204 INFO:journalctl@ceph.osd.10.smithi189.stdout:Mar 24 03:20:37 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-osd-10[61294]: 2024-03-24T03:20:37.908+0000 7f804e31f700 -1 osd.10 127 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-03-24T03:20:15.546606+0000 front 2024-03-24T03:20:15.546459+0000 (oldest deadline 2024-03-24T03:20:37.846574+0000) 2024-03-24T03:20:38.210 INFO:journalctl@ceph.osd.8.smithi193.stdout:Mar 24 03:20:37 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-osd-8[55794]: 2024-03-24T03:20:37.946+0000 7f6e8b995700 -1 osd.8 127 heartbeat_check: no reply from 172.21.15.189:6804 osd.1 since back 2024-03-24T03:20:06.219689+0000 front 2024-03-24T03:20:06.219789+0000 (oldest deadline 2024-03-24T03:20:30.919690+0000) 2024-03-24T03:20:38.446 INFO:journalctl@ceph.osd.6.smithi118.stdout:Mar 24 03:20:38 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-osd-6[59976]: 2024-03-24T03:20:38.425+0000 7fbc40a00700 -1 osd.6 127 heartbeat_check: no reply from 172.21.15.118:6806 osd.0 since back 2024-03-24T03:19:56.910256+0000 front 2024-03-24T03:19:56.910419+0000 (oldest deadline 2024-03-24T03:20:21.610387+0000) 2024-03-24T03:20:38.446 INFO:journalctl@ceph.osd.6.smithi118.stdout:Mar 24 03:20:38 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-osd-6[59976]: 2024-03-24T03:20:38.425+0000 7fbc40a00700 -1 osd.6 127 heartbeat_check: no reply from 172.21.15.189:6804 osd.1 since back 2024-03-24T03:20:05.515241+0000 front 2024-03-24T03:20:05.515363+0000 (oldest deadline 2024-03-24T03:20:29.615481+0000) 2024-03-24T03:20:38.953 INFO:journalctl@ceph.osd.7.smithi189.stdout:Mar 24 03:20:38 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-osd-7[54385]: 2024-03-24T03:20:38.703+0000 7fb775b24700 -1 osd.7 127 heartbeat_check: no reply from 172.21.15.189:6804 osd.1 since back 2024-03-24T03:20:04.667909+0000 front 2024-03-24T03:20:04.667932+0000 (oldest deadline 2024-03-24T03:20:29.968062+0000) 2024-03-24T03:20:38.954 INFO:journalctl@ceph.osd.7.smithi189.stdout:Mar 24 03:20:38 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-osd-7[54385]: 2024-03-24T03:20:38.703+0000 7fb775b24700 -1 osd.7 127 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-03-24T03:20:15.769900+0000 front 2024-03-24T03:20:15.769860+0000 (oldest deadline 2024-03-24T03:20:37.470057+0000) 2024-03-24T03:20:38.954 INFO:journalctl@ceph.osd.10.smithi189.stdout:Mar 24 03:20:38 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-osd-10[61294]: 2024-03-24T03:20:38.858+0000 7f804e31f700 -1 osd.10 127 heartbeat_check: no reply from 172.21.15.118:6806 osd.0 since back 2024-03-24T03:19:56.942815+0000 front 2024-03-24T03:19:56.942951+0000 (oldest deadline 2024-03-24T03:20:22.842714+0000) 2024-03-24T03:20:38.954 INFO:journalctl@ceph.osd.10.smithi189.stdout:Mar 24 03:20:38 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-osd-10[61294]: 2024-03-24T03:20:38.858+0000 7f804e31f700 -1 osd.10 127 heartbeat_check: no reply from 172.21.15.189:6804 osd.1 since back 2024-03-24T03:20:07.543899+0000 front 2024-03-24T03:20:07.543880+0000 (oldest deadline 2024-03-24T03:20:30.944770+0000) 2024-03-24T03:20:38.954 INFO:journalctl@ceph.osd.10.smithi189.stdout:Mar 24 03:20:38 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-osd-10[61294]: 2024-03-24T03:20:38.858+0000 7f804e31f700 -1 osd.10 127 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-03-24T03:20:15.546606+0000 front 2024-03-24T03:20:15.546459+0000 (oldest deadline 2024-03-24T03:20:37.846574+0000) 2024-03-24T03:20:39.196 INFO:journalctl@ceph.osd.9.smithi118.stdout:Mar 24 03:20:38 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-osd-9[67027]: 2024-03-24T03:20:38.706+0000 7f69f7400700 -1 osd.9 127 heartbeat_check: no reply from 172.21.15.118:6806 osd.0 since back 2024-03-24T03:19:57.002010+0000 front 2024-03-24T03:19:57.002333+0000 (oldest deadline 2024-03-24T03:20:21.101732+0000) 2024-03-24T03:20:39.196 INFO:journalctl@ceph.osd.9.smithi118.stdout:Mar 24 03:20:38 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-osd-9[67027]: 2024-03-24T03:20:38.706+0000 7f69f7400700 -1 osd.9 127 heartbeat_check: no reply from 172.21.15.189:6804 osd.1 since back 2024-03-24T03:20:05.804520+0000 front 2024-03-24T03:20:05.804379+0000 (oldest deadline 2024-03-24T03:20:29.303978+0000) 2024-03-24T03:20:39.196 INFO:journalctl@ceph.osd.9.smithi118.stdout:Mar 24 03:20:38 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-osd-9[67027]: 2024-03-24T03:20:38.706+0000 7f69f7400700 -1 osd.9 127 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-03-24T03:20:14.507274+0000 front 2024-03-24T03:20:14.507231+0000 (oldest deadline 2024-03-24T03:20:38.607488+0000) 2024-03-24T03:20:39.210 INFO:journalctl@ceph.osd.8.smithi193.stdout:Mar 24 03:20:38 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-osd-8[55794]: 2024-03-24T03:20:38.914+0000 7f6e8b995700 -1 osd.8 127 heartbeat_check: no reply from 172.21.15.189:6804 osd.1 since back 2024-03-24T03:20:06.219689+0000 front 2024-03-24T03:20:06.219789+0000 (oldest deadline 2024-03-24T03:20:30.919690+0000) 2024-03-24T03:20:39.211 INFO:journalctl@ceph.osd.5.smithi193.stdout:Mar 24 03:20:38 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-osd-5[49201]: 2024-03-24T03:20:38.777+0000 7f48090ef700 -1 osd.5 127 heartbeat_check: no reply from 172.21.15.118:6806 osd.0 since back 2024-03-24T03:19:56.517618+0000 front 2024-03-24T03:19:56.517703+0000 (oldest deadline 2024-03-24T03:20:21.718373+0000) 2024-03-24T03:20:39.211 INFO:journalctl@ceph.osd.5.smithi193.stdout:Mar 24 03:20:38 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-osd-5[49201]: 2024-03-24T03:20:38.777+0000 7f48090ef700 -1 osd.5 127 heartbeat_check: no reply from 172.21.15.189:6804 osd.1 since back 2024-03-24T03:20:06.419750+0000 front 2024-03-24T03:20:06.419585+0000 (oldest deadline 2024-03-24T03:20:31.020174+0000) 2024-03-24T03:20:39.211 INFO:journalctl@ceph.osd.11.smithi193.stdout:Mar 24 03:20:38 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-osd-11[62452]: 2024-03-24T03:20:38.808+0000 7f0edd521700 -1 osd.11 127 heartbeat_check: no reply from 172.21.15.118:6806 osd.0 since back 2024-03-24T03:19:59.095638+0000 front 2024-03-24T03:19:59.095627+0000 (oldest deadline 2024-03-24T03:20:20.195530+0000) 2024-03-24T03:20:39.211 INFO:journalctl@ceph.osd.11.smithi193.stdout:Mar 24 03:20:38 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-osd-11[62452]: 2024-03-24T03:20:38.808+0000 7f0edd521700 -1 osd.11 127 heartbeat_check: no reply from 172.21.15.189:6804 osd.1 since back 2024-03-24T03:20:07.399753+0000 front 2024-03-24T03:20:07.399812+0000 (oldest deadline 2024-03-24T03:20:32.099923+0000) 2024-03-24T03:20:39.211 INFO:journalctl@ceph.osd.11.smithi193.stdout:Mar 24 03:20:38 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-osd-11[62452]: 2024-03-24T03:20:38.808+0000 7f0edd521700 -1 osd.11 127 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-03-24T03:20:12.100299+0000 front 2024-03-24T03:20:12.100348+0000 (oldest deadline 2024-03-24T03:20:38.000441+0000) 2024-03-24T03:20:39.730 INFO:journalctl@ceph.osd.6.smithi118.stdout:Mar 24 03:20:39 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-osd-6[59976]: 2024-03-24T03:20:39.447+0000 7fbc40a00700 -1 osd.6 127 heartbeat_check: no reply from 172.21.15.118:6806 osd.0 since back 2024-03-24T03:19:56.910256+0000 front 2024-03-24T03:19:56.910419+0000 (oldest deadline 2024-03-24T03:20:21.610387+0000) 2024-03-24T03:20:39.730 INFO:journalctl@ceph.osd.6.smithi118.stdout:Mar 24 03:20:39 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-osd-6[59976]: 2024-03-24T03:20:39.447+0000 7fbc40a00700 -1 osd.6 127 heartbeat_check: no reply from 172.21.15.189:6804 osd.1 since back 2024-03-24T03:20:05.515241+0000 front 2024-03-24T03:20:05.515363+0000 (oldest deadline 2024-03-24T03:20:29.615481+0000) 2024-03-24T03:20:39.954 INFO:journalctl@ceph.osd.7.smithi189.stdout:Mar 24 03:20:39 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-osd-7[54385]: 2024-03-24T03:20:39.702+0000 7fb775b24700 -1 osd.7 127 heartbeat_check: no reply from 172.21.15.189:6804 osd.1 since back 2024-03-24T03:20:04.667909+0000 front 2024-03-24T03:20:04.667932+0000 (oldest deadline 2024-03-24T03:20:29.968062+0000) 2024-03-24T03:20:39.954 INFO:journalctl@ceph.osd.7.smithi189.stdout:Mar 24 03:20:39 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-osd-7[54385]: 2024-03-24T03:20:39.702+0000 7fb775b24700 -1 osd.7 127 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-03-24T03:20:15.769900+0000 front 2024-03-24T03:20:15.769860+0000 (oldest deadline 2024-03-24T03:20:37.470057+0000) 2024-03-24T03:20:39.955 INFO:journalctl@ceph.osd.10.smithi189.stdout:Mar 24 03:20:39 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-osd-10[61294]: 2024-03-24T03:20:39.892+0000 7f804e31f700 -1 osd.10 127 heartbeat_check: no reply from 172.21.15.118:6806 osd.0 since back 2024-03-24T03:19:56.942815+0000 front 2024-03-24T03:19:56.942951+0000 (oldest deadline 2024-03-24T03:20:22.842714+0000) 2024-03-24T03:20:39.955 INFO:journalctl@ceph.osd.10.smithi189.stdout:Mar 24 03:20:39 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-osd-10[61294]: 2024-03-24T03:20:39.892+0000 7f804e31f700 -1 osd.10 127 heartbeat_check: no reply from 172.21.15.189:6804 osd.1 since back 2024-03-24T03:20:07.543899+0000 front 2024-03-24T03:20:07.543880+0000 (oldest deadline 2024-03-24T03:20:30.944770+0000) 2024-03-24T03:20:39.955 INFO:journalctl@ceph.osd.10.smithi189.stdout:Mar 24 03:20:39 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-osd-10[61294]: 2024-03-24T03:20:39.892+0000 7f804e31f700 -1 osd.10 127 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-03-24T03:20:15.546606+0000 front 2024-03-24T03:20:15.546459+0000 (oldest deadline 2024-03-24T03:20:37.846574+0000) 2024-03-24T03:20:40.196 INFO:journalctl@ceph.osd.9.smithi118.stdout:Mar 24 03:20:39 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-osd-9[67027]: 2024-03-24T03:20:39.729+0000 7f69f7400700 -1 osd.9 127 heartbeat_check: no reply from 172.21.15.118:6806 osd.0 since back 2024-03-24T03:19:57.002010+0000 front 2024-03-24T03:19:57.002333+0000 (oldest deadline 2024-03-24T03:20:21.101732+0000) 2024-03-24T03:20:40.196 INFO:journalctl@ceph.osd.9.smithi118.stdout:Mar 24 03:20:39 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-osd-9[67027]: 2024-03-24T03:20:39.729+0000 7f69f7400700 -1 osd.9 127 heartbeat_check: no reply from 172.21.15.189:6804 osd.1 since back 2024-03-24T03:20:05.804520+0000 front 2024-03-24T03:20:05.804379+0000 (oldest deadline 2024-03-24T03:20:29.303978+0000) 2024-03-24T03:20:40.196 INFO:journalctl@ceph.osd.9.smithi118.stdout:Mar 24 03:20:39 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-osd-9[67027]: 2024-03-24T03:20:39.729+0000 7f69f7400700 -1 osd.9 127 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-03-24T03:20:14.507274+0000 front 2024-03-24T03:20:14.507231+0000 (oldest deadline 2024-03-24T03:20:38.607488+0000) 2024-03-24T03:20:40.210 INFO:journalctl@ceph.osd.8.smithi193.stdout:Mar 24 03:20:39 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-osd-8[55794]: 2024-03-24T03:20:39.933+0000 7f6e8b995700 -1 osd.8 127 heartbeat_check: no reply from 172.21.15.189:6804 osd.1 since back 2024-03-24T03:20:06.219689+0000 front 2024-03-24T03:20:06.219789+0000 (oldest deadline 2024-03-24T03:20:30.919690+0000) 2024-03-24T03:20:40.211 INFO:journalctl@ceph.osd.5.smithi193.stdout:Mar 24 03:20:39 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-osd-5[49201]: 2024-03-24T03:20:39.797+0000 7f48090ef700 -1 osd.5 127 heartbeat_check: no reply from 172.21.15.118:6806 osd.0 since back 2024-03-24T03:19:56.517618+0000 front 2024-03-24T03:19:56.517703+0000 (oldest deadline 2024-03-24T03:20:21.718373+0000) 2024-03-24T03:20:40.211 INFO:journalctl@ceph.osd.5.smithi193.stdout:Mar 24 03:20:39 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-osd-5[49201]: 2024-03-24T03:20:39.797+0000 7f48090ef700 -1 osd.5 127 heartbeat_check: no reply from 172.21.15.189:6804 osd.1 since back 2024-03-24T03:20:06.419750+0000 front 2024-03-24T03:20:06.419585+0000 (oldest deadline 2024-03-24T03:20:31.020174+0000) 2024-03-24T03:20:40.211 INFO:journalctl@ceph.osd.5.smithi193.stdout:Mar 24 03:20:39 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-osd-5[49201]: 2024-03-24T03:20:39.797+0000 7f48090ef700 -1 osd.5 127 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-03-24T03:20:16.722589+0000 front 2024-03-24T03:20:16.722710+0000 (oldest deadline 2024-03-24T03:20:39.022732+0000) 2024-03-24T03:20:40.211 INFO:journalctl@ceph.osd.11.smithi193.stdout:Mar 24 03:20:39 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-osd-11[62452]: 2024-03-24T03:20:39.805+0000 7f0edd521700 -1 osd.11 127 heartbeat_check: no reply from 172.21.15.118:6806 osd.0 since back 2024-03-24T03:19:59.095638+0000 front 2024-03-24T03:19:59.095627+0000 (oldest deadline 2024-03-24T03:20:20.195530+0000) 2024-03-24T03:20:40.211 INFO:journalctl@ceph.osd.11.smithi193.stdout:Mar 24 03:20:39 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-osd-11[62452]: 2024-03-24T03:20:39.805+0000 7f0edd521700 -1 osd.11 127 heartbeat_check: no reply from 172.21.15.189:6804 osd.1 since back 2024-03-24T03:20:07.399753+0000 front 2024-03-24T03:20:07.399812+0000 (oldest deadline 2024-03-24T03:20:32.099923+0000) 2024-03-24T03:20:40.211 INFO:journalctl@ceph.osd.11.smithi193.stdout:Mar 24 03:20:39 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-osd-11[62452]: 2024-03-24T03:20:39.805+0000 7f0edd521700 -1 osd.11 127 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-03-24T03:20:12.100299+0000 front 2024-03-24T03:20:12.100348+0000 (oldest deadline 2024-03-24T03:20:38.000441+0000) 2024-03-24T03:20:40.696 INFO:journalctl@ceph.osd.6.smithi118.stdout:Mar 24 03:20:40 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-osd-6[59976]: 2024-03-24T03:20:40.398+0000 7fbc40a00700 -1 osd.6 127 heartbeat_check: no reply from 172.21.15.118:6806 osd.0 since back 2024-03-24T03:19:56.910256+0000 front 2024-03-24T03:19:56.910419+0000 (oldest deadline 2024-03-24T03:20:21.610387+0000) 2024-03-24T03:20:40.696 INFO:journalctl@ceph.osd.6.smithi118.stdout:Mar 24 03:20:40 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-osd-6[59976]: 2024-03-24T03:20:40.398+0000 7fbc40a00700 -1 osd.6 127 heartbeat_check: no reply from 172.21.15.189:6804 osd.1 since back 2024-03-24T03:20:05.515241+0000 front 2024-03-24T03:20:05.515363+0000 (oldest deadline 2024-03-24T03:20:29.615481+0000) 2024-03-24T03:20:41.112 INFO:journalctl@ceph.osd.8.smithi193.stdout:Mar 24 03:20:40 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-osd-8[55794]: 2024-03-24T03:20:40.900+0000 7f6e8b995700 -1 osd.8 127 heartbeat_check: no reply from 172.21.15.189:6804 osd.1 since back 2024-03-24T03:20:06.219689+0000 front 2024-03-24T03:20:06.219789+0000 (oldest deadline 2024-03-24T03:20:30.919690+0000) 2024-03-24T03:20:41.112 INFO:journalctl@ceph.osd.8.smithi193.stdout:Mar 24 03:20:40 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-osd-8[55794]: 2024-03-24T03:20:40.900+0000 7f6e8b995700 -1 osd.8 127 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-03-24T03:20:16.121217+0000 front 2024-03-24T03:20:16.121198+0000 (oldest deadline 2024-03-24T03:20:40.221317+0000) 2024-03-24T03:20:41.112 INFO:journalctl@ceph.osd.11.smithi193.stdout:Mar 24 03:20:40 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-osd-11[62452]: 2024-03-24T03:20:40.833+0000 7f0edd521700 -1 osd.11 127 heartbeat_check: no reply from 172.21.15.118:6806 osd.0 since back 2024-03-24T03:19:59.095638+0000 front 2024-03-24T03:19:59.095627+0000 (oldest deadline 2024-03-24T03:20:20.195530+0000) 2024-03-24T03:20:41.112 INFO:journalctl@ceph.osd.11.smithi193.stdout:Mar 24 03:20:40 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-osd-11[62452]: 2024-03-24T03:20:40.833+0000 7f0edd521700 -1 osd.11 127 heartbeat_check: no reply from 172.21.15.189:6804 osd.1 since back 2024-03-24T03:20:07.399753+0000 front 2024-03-24T03:20:07.399812+0000 (oldest deadline 2024-03-24T03:20:32.099923+0000) 2024-03-24T03:20:41.112 INFO:journalctl@ceph.osd.11.smithi193.stdout:Mar 24 03:20:40 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-osd-11[62452]: 2024-03-24T03:20:40.833+0000 7f0edd521700 -1 osd.11 127 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-03-24T03:20:12.100299+0000 front 2024-03-24T03:20:12.100348+0000 (oldest deadline 2024-03-24T03:20:38.000441+0000) 2024-03-24T03:20:41.113 INFO:journalctl@ceph.osd.5.smithi193.stdout:Mar 24 03:20:40 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-osd-5[49201]: 2024-03-24T03:20:40.783+0000 7f48090ef700 -1 osd.5 127 heartbeat_check: no reply from 172.21.15.118:6806 osd.0 since back 2024-03-24T03:19:56.517618+0000 front 2024-03-24T03:19:56.517703+0000 (oldest deadline 2024-03-24T03:20:21.718373+0000) 2024-03-24T03:20:41.113 INFO:journalctl@ceph.osd.5.smithi193.stdout:Mar 24 03:20:40 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-osd-5[49201]: 2024-03-24T03:20:40.783+0000 7f48090ef700 -1 osd.5 127 heartbeat_check: no reply from 172.21.15.189:6804 osd.1 since back 2024-03-24T03:20:06.419750+0000 front 2024-03-24T03:20:06.419585+0000 (oldest deadline 2024-03-24T03:20:31.020174+0000) 2024-03-24T03:20:41.113 INFO:journalctl@ceph.osd.5.smithi193.stdout:Mar 24 03:20:40 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-osd-5[49201]: 2024-03-24T03:20:40.783+0000 7f48090ef700 -1 osd.5 127 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-03-24T03:20:16.722589+0000 front 2024-03-24T03:20:16.722710+0000 (oldest deadline 2024-03-24T03:20:39.022732+0000) 2024-03-24T03:20:41.127 INFO:journalctl@ceph.osd.7.smithi189.stdout:Mar 24 03:20:40 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-osd-7[54385]: 2024-03-24T03:20:40.734+0000 7fb775b24700 -1 osd.7 127 heartbeat_check: no reply from 172.21.15.189:6804 osd.1 since back 2024-03-24T03:20:04.667909+0000 front 2024-03-24T03:20:04.667932+0000 (oldest deadline 2024-03-24T03:20:29.968062+0000) 2024-03-24T03:20:41.128 INFO:journalctl@ceph.osd.7.smithi189.stdout:Mar 24 03:20:40 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-osd-7[54385]: 2024-03-24T03:20:40.734+0000 7fb775b24700 -1 osd.7 127 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-03-24T03:20:15.769900+0000 front 2024-03-24T03:20:15.769860+0000 (oldest deadline 2024-03-24T03:20:37.470057+0000) 2024-03-24T03:20:41.128 INFO:journalctl@ceph.osd.10.smithi189.stdout:Mar 24 03:20:40 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-osd-10[61294]: 2024-03-24T03:20:40.916+0000 7f804e31f700 -1 osd.10 127 heartbeat_check: no reply from 172.21.15.118:6806 osd.0 since back 2024-03-24T03:19:56.942815+0000 front 2024-03-24T03:19:56.942951+0000 (oldest deadline 2024-03-24T03:20:22.842714+0000) 2024-03-24T03:20:41.128 INFO:journalctl@ceph.osd.10.smithi189.stdout:Mar 24 03:20:40 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-osd-10[61294]: 2024-03-24T03:20:40.916+0000 7f804e31f700 -1 osd.10 127 heartbeat_check: no reply from 172.21.15.189:6804 osd.1 since back 2024-03-24T03:20:07.543899+0000 front 2024-03-24T03:20:07.543880+0000 (oldest deadline 2024-03-24T03:20:30.944770+0000) 2024-03-24T03:20:41.128 INFO:journalctl@ceph.osd.10.smithi189.stdout:Mar 24 03:20:40 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-osd-10[61294]: 2024-03-24T03:20:40.916+0000 7f804e31f700 -1 osd.10 127 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-03-24T03:20:15.546606+0000 front 2024-03-24T03:20:15.546459+0000 (oldest deadline 2024-03-24T03:20:37.846574+0000) 2024-03-24T03:20:41.196 INFO:journalctl@ceph.osd.9.smithi118.stdout:Mar 24 03:20:40 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-osd-9[67027]: 2024-03-24T03:20:40.703+0000 7f69f7400700 -1 osd.9 127 heartbeat_check: no reply from 172.21.15.118:6806 osd.0 since back 2024-03-24T03:19:57.002010+0000 front 2024-03-24T03:19:57.002333+0000 (oldest deadline 2024-03-24T03:20:21.101732+0000) 2024-03-24T03:20:41.196 INFO:journalctl@ceph.osd.9.smithi118.stdout:Mar 24 03:20:40 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-osd-9[67027]: 2024-03-24T03:20:40.703+0000 7f69f7400700 -1 osd.9 127 heartbeat_check: no reply from 172.21.15.189:6804 osd.1 since back 2024-03-24T03:20:05.804520+0000 front 2024-03-24T03:20:05.804379+0000 (oldest deadline 2024-03-24T03:20:29.303978+0000) 2024-03-24T03:20:41.196 INFO:journalctl@ceph.osd.9.smithi118.stdout:Mar 24 03:20:40 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-osd-9[67027]: 2024-03-24T03:20:40.703+0000 7f69f7400700 -1 osd.9 127 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-03-24T03:20:14.507274+0000 front 2024-03-24T03:20:14.507231+0000 (oldest deadline 2024-03-24T03:20:38.607488+0000) 2024-03-24T03:20:41.679 INFO:journalctl@ceph.osd.6.smithi118.stdout:Mar 24 03:20:41 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-osd-6[59976]: 2024-03-24T03:20:41.385+0000 7fbc40a00700 -1 osd.6 127 heartbeat_check: no reply from 172.21.15.118:6806 osd.0 since back 2024-03-24T03:19:56.910256+0000 front 2024-03-24T03:19:56.910419+0000 (oldest deadline 2024-03-24T03:20:21.610387+0000) 2024-03-24T03:20:41.679 INFO:journalctl@ceph.osd.6.smithi118.stdout:Mar 24 03:20:41 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-osd-6[59976]: 2024-03-24T03:20:41.385+0000 7fbc40a00700 -1 osd.6 127 heartbeat_check: no reply from 172.21.15.189:6804 osd.1 since back 2024-03-24T03:20:05.515241+0000 front 2024-03-24T03:20:05.515363+0000 (oldest deadline 2024-03-24T03:20:29.615481+0000) 2024-03-24T03:20:41.946 INFO:journalctl@ceph.osd.9.smithi118.stdout:Mar 24 03:20:41 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-osd-9[67027]: 2024-03-24T03:20:41.679+0000 7f69f7400700 -1 osd.9 127 heartbeat_check: no reply from 172.21.15.118:6806 osd.0 since back 2024-03-24T03:19:57.002010+0000 front 2024-03-24T03:19:57.002333+0000 (oldest deadline 2024-03-24T03:20:21.101732+0000) 2024-03-24T03:20:41.946 INFO:journalctl@ceph.osd.9.smithi118.stdout:Mar 24 03:20:41 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-osd-9[67027]: 2024-03-24T03:20:41.679+0000 7f69f7400700 -1 osd.9 127 heartbeat_check: no reply from 172.21.15.189:6804 osd.1 since back 2024-03-24T03:20:05.804520+0000 front 2024-03-24T03:20:05.804379+0000 (oldest deadline 2024-03-24T03:20:29.303978+0000) 2024-03-24T03:20:41.946 INFO:journalctl@ceph.osd.9.smithi118.stdout:Mar 24 03:20:41 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-osd-9[67027]: 2024-03-24T03:20:41.679+0000 7f69f7400700 -1 osd.9 127 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-03-24T03:20:14.507274+0000 front 2024-03-24T03:20:14.507231+0000 (oldest deadline 2024-03-24T03:20:38.607488+0000) 2024-03-24T03:20:42.053 INFO:journalctl@ceph.osd.7.smithi189.stdout:Mar 24 03:20:41 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-osd-7[54385]: 2024-03-24T03:20:41.729+0000 7fb775b24700 -1 osd.7 127 heartbeat_check: no reply from 172.21.15.189:6804 osd.1 since back 2024-03-24T03:20:04.667909+0000 front 2024-03-24T03:20:04.667932+0000 (oldest deadline 2024-03-24T03:20:29.968062+0000) 2024-03-24T03:20:42.053 INFO:journalctl@ceph.osd.7.smithi189.stdout:Mar 24 03:20:41 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-osd-7[54385]: 2024-03-24T03:20:41.729+0000 7fb775b24700 -1 osd.7 127 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-03-24T03:20:15.769900+0000 front 2024-03-24T03:20:15.769860+0000 (oldest deadline 2024-03-24T03:20:37.470057+0000) 2024-03-24T03:20:42.053 INFO:journalctl@ceph.osd.10.smithi189.stdout:Mar 24 03:20:41 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-osd-10[61294]: 2024-03-24T03:20:41.876+0000 7f804e31f700 -1 osd.10 127 heartbeat_check: no reply from 172.21.15.118:6806 osd.0 since back 2024-03-24T03:19:56.942815+0000 front 2024-03-24T03:19:56.942951+0000 (oldest deadline 2024-03-24T03:20:22.842714+0000) 2024-03-24T03:20:42.053 INFO:journalctl@ceph.osd.10.smithi189.stdout:Mar 24 03:20:41 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-osd-10[61294]: 2024-03-24T03:20:41.876+0000 7f804e31f700 -1 osd.10 127 heartbeat_check: no reply from 172.21.15.189:6804 osd.1 since back 2024-03-24T03:20:07.543899+0000 front 2024-03-24T03:20:07.543880+0000 (oldest deadline 2024-03-24T03:20:30.944770+0000) 2024-03-24T03:20:42.053 INFO:journalctl@ceph.osd.10.smithi189.stdout:Mar 24 03:20:41 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-osd-10[61294]: 2024-03-24T03:20:41.876+0000 7f804e31f700 -1 osd.10 127 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-03-24T03:20:15.546606+0000 front 2024-03-24T03:20:15.546459+0000 (oldest deadline 2024-03-24T03:20:37.846574+0000) 2024-03-24T03:20:42.132 INFO:journalctl@ceph.osd.8.smithi193.stdout:Mar 24 03:20:41 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-osd-8[55794]: 2024-03-24T03:20:41.948+0000 7f6e8b995700 -1 osd.8 127 heartbeat_check: no reply from 172.21.15.189:6804 osd.1 since back 2024-03-24T03:20:06.219689+0000 front 2024-03-24T03:20:06.219789+0000 (oldest deadline 2024-03-24T03:20:30.919690+0000) 2024-03-24T03:20:42.133 INFO:journalctl@ceph.osd.8.smithi193.stdout:Mar 24 03:20:41 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-osd-8[55794]: 2024-03-24T03:20:41.948+0000 7f6e8b995700 -1 osd.8 127 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-03-24T03:20:16.121217+0000 front 2024-03-24T03:20:16.121198+0000 (oldest deadline 2024-03-24T03:20:40.221317+0000) 2024-03-24T03:20:42.133 INFO:journalctl@ceph.osd.11.smithi193.stdout:Mar 24 03:20:41 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-osd-11[62452]: 2024-03-24T03:20:41.826+0000 7f0edd521700 -1 osd.11 127 heartbeat_check: no reply from 172.21.15.118:6806 osd.0 since back 2024-03-24T03:19:59.095638+0000 front 2024-03-24T03:19:59.095627+0000 (oldest deadline 2024-03-24T03:20:20.195530+0000) 2024-03-24T03:20:42.133 INFO:journalctl@ceph.osd.11.smithi193.stdout:Mar 24 03:20:41 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-osd-11[62452]: 2024-03-24T03:20:41.826+0000 7f0edd521700 -1 osd.11 127 heartbeat_check: no reply from 172.21.15.189:6804 osd.1 since back 2024-03-24T03:20:07.399753+0000 front 2024-03-24T03:20:07.399812+0000 (oldest deadline 2024-03-24T03:20:32.099923+0000) 2024-03-24T03:20:42.133 INFO:journalctl@ceph.osd.11.smithi193.stdout:Mar 24 03:20:41 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-osd-11[62452]: 2024-03-24T03:20:41.826+0000 7f0edd521700 -1 osd.11 127 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-03-24T03:20:12.100299+0000 front 2024-03-24T03:20:12.100348+0000 (oldest deadline 2024-03-24T03:20:38.000441+0000) 2024-03-24T03:20:42.133 INFO:journalctl@ceph.osd.5.smithi193.stdout:Mar 24 03:20:41 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-osd-5[49201]: 2024-03-24T03:20:41.739+0000 7f48090ef700 -1 osd.5 127 heartbeat_check: no reply from 172.21.15.118:6806 osd.0 since back 2024-03-24T03:19:56.517618+0000 front 2024-03-24T03:19:56.517703+0000 (oldest deadline 2024-03-24T03:20:21.718373+0000) 2024-03-24T03:20:42.134 INFO:journalctl@ceph.osd.5.smithi193.stdout:Mar 24 03:20:41 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-osd-5[49201]: 2024-03-24T03:20:41.739+0000 7f48090ef700 -1 osd.5 127 heartbeat_check: no reply from 172.21.15.189:6804 osd.1 since back 2024-03-24T03:20:06.419750+0000 front 2024-03-24T03:20:06.419585+0000 (oldest deadline 2024-03-24T03:20:31.020174+0000) 2024-03-24T03:20:42.134 INFO:journalctl@ceph.osd.5.smithi193.stdout:Mar 24 03:20:41 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-osd-5[49201]: 2024-03-24T03:20:41.739+0000 7f48090ef700 -1 osd.5 127 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-03-24T03:20:16.722589+0000 front 2024-03-24T03:20:16.722710+0000 (oldest deadline 2024-03-24T03:20:39.022732+0000) 2024-03-24T03:20:42.634 INFO:journalctl@ceph.osd.6.smithi118.stdout:Mar 24 03:20:42 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-osd-6[59976]: 2024-03-24T03:20:42.377+0000 7fbc40a00700 -1 osd.6 127 heartbeat_check: no reply from 172.21.15.118:6806 osd.0 since back 2024-03-24T03:19:56.910256+0000 front 2024-03-24T03:19:56.910419+0000 (oldest deadline 2024-03-24T03:20:21.610387+0000) 2024-03-24T03:20:42.634 INFO:journalctl@ceph.osd.6.smithi118.stdout:Mar 24 03:20:42 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-osd-6[59976]: 2024-03-24T03:20:42.377+0000 7fbc40a00700 -1 osd.6 127 heartbeat_check: no reply from 172.21.15.189:6804 osd.1 since back 2024-03-24T03:20:05.515241+0000 front 2024-03-24T03:20:05.515363+0000 (oldest deadline 2024-03-24T03:20:29.615481+0000) 2024-03-24T03:20:42.634 INFO:journalctl@ceph.osd.6.smithi118.stdout:Mar 24 03:20:42 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-osd-6[59976]: 2024-03-24T03:20:42.377+0000 7fbc40a00700 -1 osd.6 127 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-03-24T03:20:11.219038+0000 front 2024-03-24T03:20:11.218954+0000 (oldest deadline 2024-03-24T03:20:41.820368+0000) 2024-03-24T03:20:42.946 INFO:journalctl@ceph.osd.9.smithi118.stdout:Mar 24 03:20:42 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-osd-9[67027]: 2024-03-24T03:20:42.633+0000 7f69f7400700 -1 osd.9 127 heartbeat_check: no reply from 172.21.15.118:6806 osd.0 since back 2024-03-24T03:19:57.002010+0000 front 2024-03-24T03:19:57.002333+0000 (oldest deadline 2024-03-24T03:20:21.101732+0000) 2024-03-24T03:20:42.946 INFO:journalctl@ceph.osd.9.smithi118.stdout:Mar 24 03:20:42 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-osd-9[67027]: 2024-03-24T03:20:42.633+0000 7f69f7400700 -1 osd.9 127 heartbeat_check: no reply from 172.21.15.189:6804 osd.1 since back 2024-03-24T03:20:05.804520+0000 front 2024-03-24T03:20:05.804379+0000 (oldest deadline 2024-03-24T03:20:29.303978+0000) 2024-03-24T03:20:42.946 INFO:journalctl@ceph.osd.9.smithi118.stdout:Mar 24 03:20:42 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-osd-9[67027]: 2024-03-24T03:20:42.633+0000 7f69f7400700 -1 osd.9 127 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-03-24T03:20:14.507274+0000 front 2024-03-24T03:20:14.507231+0000 (oldest deadline 2024-03-24T03:20:38.607488+0000) 2024-03-24T03:20:43.203 INFO:journalctl@ceph.osd.7.smithi189.stdout:Mar 24 03:20:42 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-osd-7[54385]: 2024-03-24T03:20:42.762+0000 7fb775b24700 -1 osd.7 127 heartbeat_check: no reply from 172.21.15.189:6804 osd.1 since back 2024-03-24T03:20:04.667909+0000 front 2024-03-24T03:20:04.667932+0000 (oldest deadline 2024-03-24T03:20:29.968062+0000) 2024-03-24T03:20:43.204 INFO:journalctl@ceph.osd.7.smithi189.stdout:Mar 24 03:20:42 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-osd-7[54385]: 2024-03-24T03:20:42.762+0000 7fb775b24700 -1 osd.7 127 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-03-24T03:20:15.769900+0000 front 2024-03-24T03:20:15.769860+0000 (oldest deadline 2024-03-24T03:20:37.470057+0000) 2024-03-24T03:20:43.204 INFO:journalctl@ceph.osd.10.smithi189.stdout:Mar 24 03:20:42 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-osd-10[61294]: 2024-03-24T03:20:42.874+0000 7f804e31f700 -1 osd.10 127 heartbeat_check: no reply from 172.21.15.118:6806 osd.0 since back 2024-03-24T03:19:56.942815+0000 front 2024-03-24T03:19:56.942951+0000 (oldest deadline 2024-03-24T03:20:22.842714+0000) 2024-03-24T03:20:43.204 INFO:journalctl@ceph.osd.10.smithi189.stdout:Mar 24 03:20:42 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-osd-10[61294]: 2024-03-24T03:20:42.874+0000 7f804e31f700 -1 osd.10 127 heartbeat_check: no reply from 172.21.15.189:6804 osd.1 since back 2024-03-24T03:20:07.543899+0000 front 2024-03-24T03:20:07.543880+0000 (oldest deadline 2024-03-24T03:20:30.944770+0000) 2024-03-24T03:20:43.204 INFO:journalctl@ceph.osd.10.smithi189.stdout:Mar 24 03:20:42 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-osd-10[61294]: 2024-03-24T03:20:42.874+0000 7f804e31f700 -1 osd.10 127 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-03-24T03:20:15.546606+0000 front 2024-03-24T03:20:15.546459+0000 (oldest deadline 2024-03-24T03:20:37.846574+0000) 2024-03-24T03:20:43.210 INFO:journalctl@ceph.osd.5.smithi193.stdout:Mar 24 03:20:43 smithi193 podman[120517]: 2024-03-24 03:20:43.033803159 +0000 UTC m=+5.663346935 container died ba9e56c4eb16cabfce4a574fa9b47ff4feeda3d92567e0ce72077f1de649d843 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-d79a5420-e971-11ee-95cd-87774f69a715-osd-5, ceph=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240319, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, GIT_CLEAN=True, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD) 2024-03-24T03:20:43.210 INFO:journalctl@ceph.osd.8.smithi193.stdout:Mar 24 03:20:42 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-osd-8[55794]: 2024-03-24T03:20:42.933+0000 7f6e8b995700 -1 osd.8 127 heartbeat_check: no reply from 172.21.15.189:6804 osd.1 since back 2024-03-24T03:20:06.219689+0000 front 2024-03-24T03:20:06.219789+0000 (oldest deadline 2024-03-24T03:20:30.919690+0000) 2024-03-24T03:20:43.211 INFO:journalctl@ceph.osd.8.smithi193.stdout:Mar 24 03:20:42 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-osd-8[55794]: 2024-03-24T03:20:42.933+0000 7f6e8b995700 -1 osd.8 127 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-03-24T03:20:16.121217+0000 front 2024-03-24T03:20:16.121198+0000 (oldest deadline 2024-03-24T03:20:40.221317+0000) 2024-03-24T03:20:43.211 INFO:journalctl@ceph.osd.11.smithi193.stdout:Mar 24 03:20:42 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-osd-11[62452]: 2024-03-24T03:20:42.800+0000 7f0edd521700 -1 osd.11 127 heartbeat_check: no reply from 172.21.15.118:6806 osd.0 since back 2024-03-24T03:19:59.095638+0000 front 2024-03-24T03:19:59.095627+0000 (oldest deadline 2024-03-24T03:20:20.195530+0000) 2024-03-24T03:20:43.211 INFO:journalctl@ceph.osd.11.smithi193.stdout:Mar 24 03:20:42 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-osd-11[62452]: 2024-03-24T03:20:42.800+0000 7f0edd521700 -1 osd.11 127 heartbeat_check: no reply from 172.21.15.189:6804 osd.1 since back 2024-03-24T03:20:07.399753+0000 front 2024-03-24T03:20:07.399812+0000 (oldest deadline 2024-03-24T03:20:32.099923+0000) 2024-03-24T03:20:43.211 INFO:journalctl@ceph.osd.11.smithi193.stdout:Mar 24 03:20:42 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-osd-11[62452]: 2024-03-24T03:20:42.800+0000 7f0edd521700 -1 osd.11 127 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-03-24T03:20:12.100299+0000 front 2024-03-24T03:20:12.100348+0000 (oldest deadline 2024-03-24T03:20:38.000441+0000) 2024-03-24T03:20:43.669 INFO:journalctl@ceph.osd.6.smithi118.stdout:Mar 24 03:20:43 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-osd-6[59976]: 2024-03-24T03:20:43.327+0000 7fbc40a00700 -1 osd.6 127 heartbeat_check: no reply from 172.21.15.118:6806 osd.0 since back 2024-03-24T03:19:56.910256+0000 front 2024-03-24T03:19:56.910419+0000 (oldest deadline 2024-03-24T03:20:21.610387+0000) 2024-03-24T03:20:43.669 INFO:journalctl@ceph.osd.6.smithi118.stdout:Mar 24 03:20:43 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-osd-6[59976]: 2024-03-24T03:20:43.327+0000 7fbc40a00700 -1 osd.6 127 heartbeat_check: no reply from 172.21.15.189:6804 osd.1 since back 2024-03-24T03:20:05.515241+0000 front 2024-03-24T03:20:05.515363+0000 (oldest deadline 2024-03-24T03:20:29.615481+0000) 2024-03-24T03:20:43.670 INFO:journalctl@ceph.osd.6.smithi118.stdout:Mar 24 03:20:43 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-osd-6[59976]: 2024-03-24T03:20:43.327+0000 7fbc40a00700 -1 osd.6 127 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-03-24T03:20:11.219038+0000 front 2024-03-24T03:20:11.218954+0000 (oldest deadline 2024-03-24T03:20:41.820368+0000) 2024-03-24T03:20:43.684 INFO:journalctl@ceph.osd.5.smithi193.stdout:Mar 24 03:20:43 smithi193 podman[120517]: 2024-03-24 03:20:43.450874706 +0000 UTC m=+6.080418465 container cleanup ba9e56c4eb16cabfce4a574fa9b47ff4feeda3d92567e0ce72077f1de649d843 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-d79a5420-e971-11ee-95cd-87774f69a715-osd-5, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, GIT_CLEAN=True, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, org.label-schema.schema-version=1.0, ceph=True, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240319, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30) 2024-03-24T03:20:43.684 INFO:journalctl@ceph.osd.5.smithi193.stdout:Mar 24 03:20:43 smithi193 bash[120517]: ceph-d79a5420-e971-11ee-95cd-87774f69a715-osd-5 2024-03-24T03:20:43.946 INFO:journalctl@ceph.osd.9.smithi118.stdout:Mar 24 03:20:43 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-osd-9[67027]: 2024-03-24T03:20:43.669+0000 7f69f7400700 -1 osd.9 127 heartbeat_check: no reply from 172.21.15.118:6806 osd.0 since back 2024-03-24T03:19:57.002010+0000 front 2024-03-24T03:19:57.002333+0000 (oldest deadline 2024-03-24T03:20:21.101732+0000) 2024-03-24T03:20:43.946 INFO:journalctl@ceph.osd.9.smithi118.stdout:Mar 24 03:20:43 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-osd-9[67027]: 2024-03-24T03:20:43.669+0000 7f69f7400700 -1 osd.9 127 heartbeat_check: no reply from 172.21.15.189:6804 osd.1 since back 2024-03-24T03:20:05.804520+0000 front 2024-03-24T03:20:05.804379+0000 (oldest deadline 2024-03-24T03:20:29.303978+0000) 2024-03-24T03:20:43.946 INFO:journalctl@ceph.osd.9.smithi118.stdout:Mar 24 03:20:43 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-osd-9[67027]: 2024-03-24T03:20:43.669+0000 7f69f7400700 -1 osd.9 127 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-03-24T03:20:14.507274+0000 front 2024-03-24T03:20:14.507231+0000 (oldest deadline 2024-03-24T03:20:38.607488+0000) 2024-03-24T03:20:43.953 INFO:journalctl@ceph.osd.5.smithi193.stdout:Mar 24 03:20:43 smithi193 podman[120546]: 2024-03-24 03:20:43.685458013 +0000 UTC m=+0.652263897 container remove ba9e56c4eb16cabfce4a574fa9b47ff4feeda3d92567e0ce72077f1de649d843 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-d79a5420-e971-11ee-95cd-87774f69a715-osd-5, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, ceph=True, org.label-schema.build-date=20240319, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.license=GPLv2) 2024-03-24T03:20:43.953 INFO:journalctl@ceph.osd.11.smithi193.stdout:Mar 24 03:20:43 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-osd-11[62452]: 2024-03-24T03:20:43.814+0000 7f0edd521700 -1 osd.11 127 heartbeat_check: no reply from 172.21.15.118:6806 osd.0 since back 2024-03-24T03:19:59.095638+0000 front 2024-03-24T03:19:59.095627+0000 (oldest deadline 2024-03-24T03:20:20.195530+0000) 2024-03-24T03:20:43.953 INFO:journalctl@ceph.osd.11.smithi193.stdout:Mar 24 03:20:43 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-osd-11[62452]: 2024-03-24T03:20:43.814+0000 7f0edd521700 -1 osd.11 127 heartbeat_check: no reply from 172.21.15.189:6804 osd.1 since back 2024-03-24T03:20:07.399753+0000 front 2024-03-24T03:20:07.399812+0000 (oldest deadline 2024-03-24T03:20:32.099923+0000) 2024-03-24T03:20:43.953 INFO:journalctl@ceph.osd.11.smithi193.stdout:Mar 24 03:20:43 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-osd-11[62452]: 2024-03-24T03:20:43.814+0000 7f0edd521700 -1 osd.11 127 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-03-24T03:20:12.100299+0000 front 2024-03-24T03:20:12.100348+0000 (oldest deadline 2024-03-24T03:20:38.000441+0000) 2024-03-24T03:20:44.204 INFO:journalctl@ceph.osd.7.smithi189.stdout:Mar 24 03:20:43 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-osd-7[54385]: 2024-03-24T03:20:43.751+0000 7fb775b24700 -1 osd.7 127 heartbeat_check: no reply from 172.21.15.189:6804 osd.1 since back 2024-03-24T03:20:04.667909+0000 front 2024-03-24T03:20:04.667932+0000 (oldest deadline 2024-03-24T03:20:29.968062+0000) 2024-03-24T03:20:44.204 INFO:journalctl@ceph.osd.7.smithi189.stdout:Mar 24 03:20:43 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-osd-7[54385]: 2024-03-24T03:20:43.751+0000 7fb775b24700 -1 osd.7 127 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-03-24T03:20:15.769900+0000 front 2024-03-24T03:20:15.769860+0000 (oldest deadline 2024-03-24T03:20:37.470057+0000) 2024-03-24T03:20:44.204 INFO:journalctl@ceph.osd.10.smithi189.stdout:Mar 24 03:20:43 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-osd-10[61294]: 2024-03-24T03:20:43.923+0000 7f804e31f700 -1 osd.10 127 heartbeat_check: no reply from 172.21.15.118:6806 osd.0 since back 2024-03-24T03:19:56.942815+0000 front 2024-03-24T03:19:56.942951+0000 (oldest deadline 2024-03-24T03:20:22.842714+0000) 2024-03-24T03:20:44.204 INFO:journalctl@ceph.osd.10.smithi189.stdout:Mar 24 03:20:43 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-osd-10[61294]: 2024-03-24T03:20:43.923+0000 7f804e31f700 -1 osd.10 127 heartbeat_check: no reply from 172.21.15.189:6804 osd.1 since back 2024-03-24T03:20:07.543899+0000 front 2024-03-24T03:20:07.543880+0000 (oldest deadline 2024-03-24T03:20:30.944770+0000) 2024-03-24T03:20:44.204 INFO:journalctl@ceph.osd.10.smithi189.stdout:Mar 24 03:20:43 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-osd-10[61294]: 2024-03-24T03:20:43.923+0000 7f804e31f700 -1 osd.10 127 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-03-24T03:20:15.546606+0000 front 2024-03-24T03:20:15.546459+0000 (oldest deadline 2024-03-24T03:20:37.846574+0000) 2024-03-24T03:20:44.210 INFO:journalctl@ceph.osd.8.smithi193.stdout:Mar 24 03:20:43 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-osd-8[55794]: 2024-03-24T03:20:43.953+0000 7f6e8b995700 -1 osd.8 127 heartbeat_check: no reply from 172.21.15.189:6804 osd.1 since back 2024-03-24T03:20:06.219689+0000 front 2024-03-24T03:20:06.219789+0000 (oldest deadline 2024-03-24T03:20:30.919690+0000) 2024-03-24T03:20:44.210 INFO:journalctl@ceph.osd.8.smithi193.stdout:Mar 24 03:20:43 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-osd-8[55794]: 2024-03-24T03:20:43.953+0000 7f6e8b995700 -1 osd.8 127 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-03-24T03:20:16.121217+0000 front 2024-03-24T03:20:16.121198+0000 (oldest deadline 2024-03-24T03:20:40.221317+0000) 2024-03-24T03:20:44.211 INFO:journalctl@ceph.osd.5.smithi193.stdout:Mar 24 03:20:44 smithi193 podman[120638]: 2024-03-24 03:20:44.005108698 +0000 UTC m=+0.017688430 image pull c664afe56bb01db4f9afa9532c20598e201e664f2e9ba2be50db0d52674a26d2 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc 2024-03-24T03:20:44.211 INFO:journalctl@ceph.osd.5.smithi193.stdout:Mar 24 03:20:44 smithi193 podman[120638]: 2024-03-24 03:20:44.111659433 +0000 UTC m=+0.124239144 container create 9cb76677504858be1948cc85c15154674c5e166674b21bd8e0266d794896904a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-d79a5420-e971-11ee-95cd-87774f69a715-osd-5-deactivate, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, GIT_CLEAN=True, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240319, org.label-schema.schema-version=1.0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30) 2024-03-24T03:20:44.623 INFO:journalctl@ceph.osd.5.smithi193.stdout:Mar 24 03:20:44 smithi193 podman[120638]: 2024-03-24 03:20:44.33968106 +0000 UTC m=+0.352260783 container init 9cb76677504858be1948cc85c15154674c5e166674b21bd8e0266d794896904a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-d79a5420-e971-11ee-95cd-87774f69a715-osd-5-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240319, io.buildah.version=1.33.5, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, ceph=True, org.label-schema.vendor=CentOS, GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-24T03:20:44.624 INFO:journalctl@ceph.osd.5.smithi193.stdout:Mar 24 03:20:44 smithi193 podman[120638]: 2024-03-24 03:20:44.342855803 +0000 UTC m=+0.355435508 container start 9cb76677504858be1948cc85c15154674c5e166674b21bd8e0266d794896904a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-d79a5420-e971-11ee-95cd-87774f69a715-osd-5-deactivate, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, CEPH_POINT_RELEASE=, ceph=True, GIT_CLEAN=True, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, org.label-schema.build-date=20240319, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0) 2024-03-24T03:20:44.624 INFO:journalctl@ceph.osd.5.smithi193.stdout:Mar 24 03:20:44 smithi193 podman[120638]: 2024-03-24 03:20:44.439576233 +0000 UTC m=+0.452155940 container attach 9cb76677504858be1948cc85c15154674c5e166674b21bd8e0266d794896904a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-d79a5420-e971-11ee-95cd-87774f69a715-osd-5-deactivate, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, org.label-schema.schema-version=1.0, GIT_CLEAN=True, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.build-date=20240319, io.buildah.version=1.33.5) 2024-03-24T03:20:44.658 INFO:journalctl@ceph.osd.6.smithi118.stdout:Mar 24 03:20:44 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-osd-6[59976]: 2024-03-24T03:20:44.297+0000 7fbc40a00700 -1 osd.6 127 heartbeat_check: no reply from 172.21.15.118:6806 osd.0 since back 2024-03-24T03:19:56.910256+0000 front 2024-03-24T03:19:56.910419+0000 (oldest deadline 2024-03-24T03:20:21.610387+0000) 2024-03-24T03:20:44.658 INFO:journalctl@ceph.osd.6.smithi118.stdout:Mar 24 03:20:44 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-osd-6[59976]: 2024-03-24T03:20:44.297+0000 7fbc40a00700 -1 osd.6 127 heartbeat_check: no reply from 172.21.15.189:6804 osd.1 since back 2024-03-24T03:20:05.515241+0000 front 2024-03-24T03:20:05.515363+0000 (oldest deadline 2024-03-24T03:20:29.615481+0000) 2024-03-24T03:20:44.658 INFO:journalctl@ceph.osd.6.smithi118.stdout:Mar 24 03:20:44 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-osd-6[59976]: 2024-03-24T03:20:44.297+0000 7fbc40a00700 -1 osd.6 127 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-03-24T03:20:11.219038+0000 front 2024-03-24T03:20:11.218954+0000 (oldest deadline 2024-03-24T03:20:41.820368+0000) 2024-03-24T03:20:44.933 INFO:journalctl@ceph.osd.11.smithi193.stdout:Mar 24 03:20:44 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-osd-11[62452]: 2024-03-24T03:20:44.822+0000 7f0edd521700 -1 osd.11 127 heartbeat_check: no reply from 172.21.15.118:6806 osd.0 since back 2024-03-24T03:19:59.095638+0000 front 2024-03-24T03:19:59.095627+0000 (oldest deadline 2024-03-24T03:20:20.195530+0000) 2024-03-24T03:20:44.934 INFO:journalctl@ceph.osd.11.smithi193.stdout:Mar 24 03:20:44 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-osd-11[62452]: 2024-03-24T03:20:44.822+0000 7f0edd521700 -1 osd.11 127 heartbeat_check: no reply from 172.21.15.189:6804 osd.1 since back 2024-03-24T03:20:07.399753+0000 front 2024-03-24T03:20:07.399812+0000 (oldest deadline 2024-03-24T03:20:32.099923+0000) 2024-03-24T03:20:44.934 INFO:journalctl@ceph.osd.11.smithi193.stdout:Mar 24 03:20:44 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-osd-11[62452]: 2024-03-24T03:20:44.822+0000 7f0edd521700 -1 osd.11 127 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-03-24T03:20:12.100299+0000 front 2024-03-24T03:20:12.100348+0000 (oldest deadline 2024-03-24T03:20:38.000441+0000) 2024-03-24T03:20:44.934 INFO:journalctl@ceph.osd.5.smithi193.stdout:Mar 24 03:20:44 smithi193 podman[120638]: 2024-03-24 03:20:44.624157382 +0000 UTC m=+0.636737097 container died 9cb76677504858be1948cc85c15154674c5e166674b21bd8e0266d794896904a (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-d79a5420-e971-11ee-95cd-87774f69a715-osd-5-deactivate, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240319, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, ceph=True, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=) 2024-03-24T03:20:44.946 INFO:journalctl@ceph.osd.9.smithi118.stdout:Mar 24 03:20:44 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-osd-9[67027]: 2024-03-24T03:20:44.657+0000 7f69f7400700 -1 osd.9 127 heartbeat_check: no reply from 172.21.15.118:6806 osd.0 since back 2024-03-24T03:19:57.002010+0000 front 2024-03-24T03:19:57.002333+0000 (oldest deadline 2024-03-24T03:20:21.101732+0000) 2024-03-24T03:20:44.946 INFO:journalctl@ceph.osd.9.smithi118.stdout:Mar 24 03:20:44 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-osd-9[67027]: 2024-03-24T03:20:44.657+0000 7f69f7400700 -1 osd.9 127 heartbeat_check: no reply from 172.21.15.189:6804 osd.1 since back 2024-03-24T03:20:05.804520+0000 front 2024-03-24T03:20:05.804379+0000 (oldest deadline 2024-03-24T03:20:29.303978+0000) 2024-03-24T03:20:44.946 INFO:journalctl@ceph.osd.9.smithi118.stdout:Mar 24 03:20:44 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-osd-9[67027]: 2024-03-24T03:20:44.657+0000 7f69f7400700 -1 osd.9 127 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-03-24T03:20:14.507274+0000 front 2024-03-24T03:20:14.507231+0000 (oldest deadline 2024-03-24T03:20:38.607488+0000) 2024-03-24T03:20:45.206 INFO:journalctl@ceph.osd.7.smithi189.stdout:Mar 24 03:20:44 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-osd-7[54385]: 2024-03-24T03:20:44.772+0000 7fb775b24700 -1 osd.7 127 heartbeat_check: no reply from 172.21.15.189:6804 osd.1 since back 2024-03-24T03:20:04.667909+0000 front 2024-03-24T03:20:04.667932+0000 (oldest deadline 2024-03-24T03:20:29.968062+0000) 2024-03-24T03:20:45.206 INFO:journalctl@ceph.osd.7.smithi189.stdout:Mar 24 03:20:44 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-osd-7[54385]: 2024-03-24T03:20:44.772+0000 7fb775b24700 -1 osd.7 127 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-03-24T03:20:15.769900+0000 front 2024-03-24T03:20:15.769860+0000 (oldest deadline 2024-03-24T03:20:37.470057+0000) 2024-03-24T03:20:45.207 INFO:journalctl@ceph.osd.10.smithi189.stdout:Mar 24 03:20:44 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-osd-10[61294]: 2024-03-24T03:20:44.918+0000 7f804e31f700 -1 osd.10 127 heartbeat_check: no reply from 172.21.15.118:6806 osd.0 since back 2024-03-24T03:19:56.942815+0000 front 2024-03-24T03:19:56.942951+0000 (oldest deadline 2024-03-24T03:20:22.842714+0000) 2024-03-24T03:20:45.207 INFO:journalctl@ceph.osd.10.smithi189.stdout:Mar 24 03:20:44 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-osd-10[61294]: 2024-03-24T03:20:44.918+0000 7f804e31f700 -1 osd.10 127 heartbeat_check: no reply from 172.21.15.189:6804 osd.1 since back 2024-03-24T03:20:07.543899+0000 front 2024-03-24T03:20:07.543880+0000 (oldest deadline 2024-03-24T03:20:30.944770+0000) 2024-03-24T03:20:45.207 INFO:journalctl@ceph.osd.10.smithi189.stdout:Mar 24 03:20:44 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-osd-10[61294]: 2024-03-24T03:20:44.918+0000 7f804e31f700 -1 osd.10 127 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-03-24T03:20:15.546606+0000 front 2024-03-24T03:20:15.546459+0000 (oldest deadline 2024-03-24T03:20:37.846574+0000) 2024-03-24T03:20:45.210 INFO:journalctl@ceph.osd.8.smithi193.stdout:Mar 24 03:20:44 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-osd-8[55794]: 2024-03-24T03:20:44.991+0000 7f6e8b995700 -1 osd.8 127 heartbeat_check: no reply from 172.21.15.189:6804 osd.1 since back 2024-03-24T03:20:06.219689+0000 front 2024-03-24T03:20:06.219789+0000 (oldest deadline 2024-03-24T03:20:30.919690+0000) 2024-03-24T03:20:45.210 INFO:journalctl@ceph.osd.8.smithi193.stdout:Mar 24 03:20:44 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-osd-8[55794]: 2024-03-24T03:20:44.991+0000 7f6e8b995700 -1 osd.8 127 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-03-24T03:20:16.121217+0000 front 2024-03-24T03:20:16.121198+0000 (oldest deadline 2024-03-24T03:20:40.221317+0000) 2024-03-24T03:20:45.267 DEBUG:teuthology.orchestra.run.smithi193:> sudo pkill -f 'journalctl -f -n 0 -u ceph-d79a5420-e971-11ee-95cd-87774f69a715@osd.5.service' 2024-03-24T03:20:45.349 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-24T03:20:45.350 INFO:tasks.cephadm.osd.5:Stopped osd.5 2024-03-24T03:20:45.350 INFO:tasks.cephadm.osd.6:Stopping osd.6... 2024-03-24T03:20:45.350 DEBUG:teuthology.orchestra.run.smithi118:> sudo systemctl stop ceph-d79a5420-e971-11ee-95cd-87774f69a715@osd.6 2024-03-24T03:20:45.634 INFO:journalctl@ceph.osd.6.smithi118.stdout:Mar 24 03:20:45 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-osd-6[59976]: 2024-03-24T03:20:45.308+0000 7fbc40a00700 -1 osd.6 127 heartbeat_check: no reply from 172.21.15.118:6806 osd.0 since back 2024-03-24T03:19:56.910256+0000 front 2024-03-24T03:19:56.910419+0000 (oldest deadline 2024-03-24T03:20:21.610387+0000) 2024-03-24T03:20:45.634 INFO:journalctl@ceph.osd.6.smithi118.stdout:Mar 24 03:20:45 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-osd-6[59976]: 2024-03-24T03:20:45.308+0000 7fbc40a00700 -1 osd.6 127 heartbeat_check: no reply from 172.21.15.189:6804 osd.1 since back 2024-03-24T03:20:05.515241+0000 front 2024-03-24T03:20:05.515363+0000 (oldest deadline 2024-03-24T03:20:29.615481+0000) 2024-03-24T03:20:45.634 INFO:journalctl@ceph.osd.6.smithi118.stdout:Mar 24 03:20:45 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-osd-6[59976]: 2024-03-24T03:20:45.308+0000 7fbc40a00700 -1 osd.6 127 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-03-24T03:20:11.219038+0000 front 2024-03-24T03:20:11.218954+0000 (oldest deadline 2024-03-24T03:20:41.820368+0000) 2024-03-24T03:20:45.635 INFO:journalctl@ceph.osd.6.smithi118.stdout:Mar 24 03:20:45 smithi118 systemd[1]: Stopping Ceph osd.6 for d79a5420-e971-11ee-95cd-87774f69a715... 2024-03-24T03:20:45.924 INFO:journalctl@ceph.osd.9.smithi118.stdout:Mar 24 03:20:45 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-osd-9[67027]: 2024-03-24T03:20:45.632+0000 7f69f7400700 -1 osd.9 127 heartbeat_check: no reply from 172.21.15.118:6806 osd.0 since back 2024-03-24T03:19:57.002010+0000 front 2024-03-24T03:19:57.002333+0000 (oldest deadline 2024-03-24T03:20:21.101732+0000) 2024-03-24T03:20:45.924 INFO:journalctl@ceph.osd.9.smithi118.stdout:Mar 24 03:20:45 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-osd-9[67027]: 2024-03-24T03:20:45.632+0000 7f69f7400700 -1 osd.9 127 heartbeat_check: no reply from 172.21.15.189:6804 osd.1 since back 2024-03-24T03:20:05.804520+0000 front 2024-03-24T03:20:05.804379+0000 (oldest deadline 2024-03-24T03:20:29.303978+0000) 2024-03-24T03:20:45.924 INFO:journalctl@ceph.osd.9.smithi118.stdout:Mar 24 03:20:45 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-osd-9[67027]: 2024-03-24T03:20:45.632+0000 7f69f7400700 -1 osd.9 127 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-03-24T03:20:14.507274+0000 front 2024-03-24T03:20:14.507231+0000 (oldest deadline 2024-03-24T03:20:38.607488+0000) 2024-03-24T03:20:46.196 INFO:journalctl@ceph.osd.6.smithi118.stdout:Mar 24 03:20:45 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-osd-6[59976]: 2024-03-24T03:20:45.923+0000 7fbc45200700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.6 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-24T03:20:46.196 INFO:journalctl@ceph.osd.6.smithi118.stdout:Mar 24 03:20:45 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-osd-6[59976]: 2024-03-24T03:20:45.923+0000 7fbc45200700 -1 osd.6 127 *** Got signal Terminated *** 2024-03-24T03:20:46.196 INFO:journalctl@ceph.osd.6.smithi118.stdout:Mar 24 03:20:45 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-osd-6[59976]: 2024-03-24T03:20:45.923+0000 7fbc45200700 -1 osd.6 127 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-24T03:20:46.204 INFO:journalctl@ceph.osd.7.smithi189.stdout:Mar 24 03:20:45 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-osd-7[54385]: 2024-03-24T03:20:45.819+0000 7fb775b24700 -1 osd.7 127 heartbeat_check: no reply from 172.21.15.189:6804 osd.1 since back 2024-03-24T03:20:04.667909+0000 front 2024-03-24T03:20:04.667932+0000 (oldest deadline 2024-03-24T03:20:29.968062+0000) 2024-03-24T03:20:46.204 INFO:journalctl@ceph.osd.7.smithi189.stdout:Mar 24 03:20:45 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-osd-7[54385]: 2024-03-24T03:20:45.819+0000 7fb775b24700 -1 osd.7 127 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-03-24T03:20:15.769900+0000 front 2024-03-24T03:20:15.769860+0000 (oldest deadline 2024-03-24T03:20:37.470057+0000) 2024-03-24T03:20:46.204 INFO:journalctl@ceph.osd.10.smithi189.stdout:Mar 24 03:20:45 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-osd-10[61294]: 2024-03-24T03:20:45.941+0000 7f804e31f700 -1 osd.10 127 heartbeat_check: no reply from 172.21.15.118:6806 osd.0 since back 2024-03-24T03:19:56.942815+0000 front 2024-03-24T03:19:56.942951+0000 (oldest deadline 2024-03-24T03:20:22.842714+0000) 2024-03-24T03:20:46.204 INFO:journalctl@ceph.osd.10.smithi189.stdout:Mar 24 03:20:45 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-osd-10[61294]: 2024-03-24T03:20:45.942+0000 7f804e31f700 -1 osd.10 127 heartbeat_check: no reply from 172.21.15.189:6804 osd.1 since back 2024-03-24T03:20:07.543899+0000 front 2024-03-24T03:20:07.543880+0000 (oldest deadline 2024-03-24T03:20:30.944770+0000) 2024-03-24T03:20:46.204 INFO:journalctl@ceph.osd.10.smithi189.stdout:Mar 24 03:20:45 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-osd-10[61294]: 2024-03-24T03:20:45.942+0000 7f804e31f700 -1 osd.10 127 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-03-24T03:20:15.546606+0000 front 2024-03-24T03:20:15.546459+0000 (oldest deadline 2024-03-24T03:20:37.846574+0000) 2024-03-24T03:20:46.210 INFO:journalctl@ceph.osd.8.smithi193.stdout:Mar 24 03:20:45 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-osd-8[55794]: 2024-03-24T03:20:45.958+0000 7f6e8b995700 -1 osd.8 127 heartbeat_check: no reply from 172.21.15.189:6804 osd.1 since back 2024-03-24T03:20:06.219689+0000 front 2024-03-24T03:20:06.219789+0000 (oldest deadline 2024-03-24T03:20:30.919690+0000) 2024-03-24T03:20:46.210 INFO:journalctl@ceph.osd.8.smithi193.stdout:Mar 24 03:20:45 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-osd-8[55794]: 2024-03-24T03:20:45.958+0000 7f6e8b995700 -1 osd.8 127 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-03-24T03:20:16.121217+0000 front 2024-03-24T03:20:16.121198+0000 (oldest deadline 2024-03-24T03:20:40.221317+0000) 2024-03-24T03:20:46.211 INFO:journalctl@ceph.osd.11.smithi193.stdout:Mar 24 03:20:45 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-osd-11[62452]: 2024-03-24T03:20:45.808+0000 7f0edd521700 -1 osd.11 127 heartbeat_check: no reply from 172.21.15.118:6806 osd.0 since back 2024-03-24T03:19:59.095638+0000 front 2024-03-24T03:19:59.095627+0000 (oldest deadline 2024-03-24T03:20:20.195530+0000) 2024-03-24T03:20:46.211 INFO:journalctl@ceph.osd.11.smithi193.stdout:Mar 24 03:20:45 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-osd-11[62452]: 2024-03-24T03:20:45.808+0000 7f0edd521700 -1 osd.11 127 heartbeat_check: no reply from 172.21.15.189:6804 osd.1 since back 2024-03-24T03:20:07.399753+0000 front 2024-03-24T03:20:07.399812+0000 (oldest deadline 2024-03-24T03:20:32.099923+0000) 2024-03-24T03:20:46.211 INFO:journalctl@ceph.osd.11.smithi193.stdout:Mar 24 03:20:45 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-osd-11[62452]: 2024-03-24T03:20:45.808+0000 7f0edd521700 -1 osd.11 127 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-03-24T03:20:12.100299+0000 front 2024-03-24T03:20:12.100348+0000 (oldest deadline 2024-03-24T03:20:38.000441+0000) 2024-03-24T03:20:46.617 INFO:journalctl@ceph.osd.6.smithi118.stdout:Mar 24 03:20:46 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-osd-6[59976]: 2024-03-24T03:20:46.270+0000 7fbc40a00700 -1 osd.6 127 heartbeat_check: no reply from 172.21.15.118:6806 osd.0 since back 2024-03-24T03:19:56.910256+0000 front 2024-03-24T03:19:56.910419+0000 (oldest deadline 2024-03-24T03:20:21.610387+0000) 2024-03-24T03:20:46.617 INFO:journalctl@ceph.osd.6.smithi118.stdout:Mar 24 03:20:46 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-osd-6[59976]: 2024-03-24T03:20:46.270+0000 7fbc40a00700 -1 osd.6 127 heartbeat_check: no reply from 172.21.15.189:6804 osd.1 since back 2024-03-24T03:20:05.515241+0000 front 2024-03-24T03:20:05.515363+0000 (oldest deadline 2024-03-24T03:20:29.615481+0000) 2024-03-24T03:20:46.617 INFO:journalctl@ceph.osd.6.smithi118.stdout:Mar 24 03:20:46 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-osd-6[59976]: 2024-03-24T03:20:46.270+0000 7fbc40a00700 -1 osd.6 127 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-03-24T03:20:11.219038+0000 front 2024-03-24T03:20:11.218954+0000 (oldest deadline 2024-03-24T03:20:41.820368+0000) 2024-03-24T03:20:46.946 INFO:journalctl@ceph.osd.9.smithi118.stdout:Mar 24 03:20:46 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-osd-9[67027]: 2024-03-24T03:20:46.616+0000 7f69f7400700 -1 osd.9 127 heartbeat_check: no reply from 172.21.15.118:6806 osd.0 since back 2024-03-24T03:19:57.002010+0000 front 2024-03-24T03:19:57.002333+0000 (oldest deadline 2024-03-24T03:20:21.101732+0000) 2024-03-24T03:20:46.946 INFO:journalctl@ceph.osd.9.smithi118.stdout:Mar 24 03:20:46 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-osd-9[67027]: 2024-03-24T03:20:46.616+0000 7f69f7400700 -1 osd.9 127 heartbeat_check: no reply from 172.21.15.189:6804 osd.1 since back 2024-03-24T03:20:05.804520+0000 front 2024-03-24T03:20:05.804379+0000 (oldest deadline 2024-03-24T03:20:29.303978+0000) 2024-03-24T03:20:46.946 INFO:journalctl@ceph.osd.9.smithi118.stdout:Mar 24 03:20:46 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-osd-9[67027]: 2024-03-24T03:20:46.616+0000 7f69f7400700 -1 osd.9 127 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-03-24T03:20:14.507274+0000 front 2024-03-24T03:20:14.507231+0000 (oldest deadline 2024-03-24T03:20:38.607488+0000) 2024-03-24T03:20:47.204 INFO:journalctl@ceph.osd.7.smithi189.stdout:Mar 24 03:20:46 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-osd-7[54385]: 2024-03-24T03:20:46.821+0000 7fb775b24700 -1 osd.7 127 heartbeat_check: no reply from 172.21.15.189:6804 osd.1 since back 2024-03-24T03:20:04.667909+0000 front 2024-03-24T03:20:04.667932+0000 (oldest deadline 2024-03-24T03:20:29.968062+0000) 2024-03-24T03:20:47.204 INFO:journalctl@ceph.osd.7.smithi189.stdout:Mar 24 03:20:46 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-osd-7[54385]: 2024-03-24T03:20:46.821+0000 7fb775b24700 -1 osd.7 127 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-03-24T03:20:15.769900+0000 front 2024-03-24T03:20:15.769860+0000 (oldest deadline 2024-03-24T03:20:37.470057+0000) 2024-03-24T03:20:47.204 INFO:journalctl@ceph.osd.7.smithi189.stdout:Mar 24 03:20:46 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-osd-7[54385]: 2024-03-24T03:20:46.821+0000 7fb775b24700 -1 osd.7 127 heartbeat_check: no reply from 172.21.15.118:6814 osd.3 since back 2024-03-24T03:20:24.372782+0000 front 2024-03-24T03:20:24.372820+0000 (oldest deadline 2024-03-24T03:20:46.072678+0000) 2024-03-24T03:20:47.204 INFO:journalctl@ceph.osd.10.smithi189.stdout:Mar 24 03:20:46 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-osd-10[61294]: 2024-03-24T03:20:46.988+0000 7f804e31f700 -1 osd.10 127 heartbeat_check: no reply from 172.21.15.118:6806 osd.0 since back 2024-03-24T03:19:56.942815+0000 front 2024-03-24T03:19:56.942951+0000 (oldest deadline 2024-03-24T03:20:22.842714+0000) 2024-03-24T03:20:47.205 INFO:journalctl@ceph.osd.10.smithi189.stdout:Mar 24 03:20:46 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-osd-10[61294]: 2024-03-24T03:20:46.988+0000 7f804e31f700 -1 osd.10 127 heartbeat_check: no reply from 172.21.15.189:6804 osd.1 since back 2024-03-24T03:20:07.543899+0000 front 2024-03-24T03:20:07.543880+0000 (oldest deadline 2024-03-24T03:20:30.944770+0000) 2024-03-24T03:20:47.205 INFO:journalctl@ceph.osd.10.smithi189.stdout:Mar 24 03:20:46 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-osd-10[61294]: 2024-03-24T03:20:46.988+0000 7f804e31f700 -1 osd.10 127 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-03-24T03:20:15.546606+0000 front 2024-03-24T03:20:15.546459+0000 (oldest deadline 2024-03-24T03:20:37.846574+0000) 2024-03-24T03:20:47.210 INFO:journalctl@ceph.osd.8.smithi193.stdout:Mar 24 03:20:46 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-osd-8[55794]: 2024-03-24T03:20:46.968+0000 7f6e8b995700 -1 osd.8 127 heartbeat_check: no reply from 172.21.15.189:6804 osd.1 since back 2024-03-24T03:20:06.219689+0000 front 2024-03-24T03:20:06.219789+0000 (oldest deadline 2024-03-24T03:20:30.919690+0000) 2024-03-24T03:20:47.211 INFO:journalctl@ceph.osd.8.smithi193.stdout:Mar 24 03:20:46 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-osd-8[55794]: 2024-03-24T03:20:46.968+0000 7f6e8b995700 -1 osd.8 127 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-03-24T03:20:16.121217+0000 front 2024-03-24T03:20:16.121198+0000 (oldest deadline 2024-03-24T03:20:40.221317+0000) 2024-03-24T03:20:47.211 INFO:journalctl@ceph.osd.8.smithi193.stdout:Mar 24 03:20:46 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-osd-8[55794]: 2024-03-24T03:20:46.968+0000 7f6e8b995700 -1 osd.8 127 heartbeat_check: no reply from 172.21.15.118:6814 osd.3 since back 2024-03-24T03:20:24.922540+0000 front 2024-03-24T03:20:24.922747+0000 (oldest deadline 2024-03-24T03:20:46.622438+0000) 2024-03-24T03:20:47.211 INFO:journalctl@ceph.osd.11.smithi193.stdout:Mar 24 03:20:46 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-osd-11[62452]: 2024-03-24T03:20:46.810+0000 7f0edd521700 -1 osd.11 127 heartbeat_check: no reply from 172.21.15.118:6806 osd.0 since back 2024-03-24T03:19:59.095638+0000 front 2024-03-24T03:19:59.095627+0000 (oldest deadline 2024-03-24T03:20:20.195530+0000) 2024-03-24T03:20:47.211 INFO:journalctl@ceph.osd.11.smithi193.stdout:Mar 24 03:20:46 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-osd-11[62452]: 2024-03-24T03:20:46.810+0000 7f0edd521700 -1 osd.11 127 heartbeat_check: no reply from 172.21.15.189:6804 osd.1 since back 2024-03-24T03:20:07.399753+0000 front 2024-03-24T03:20:07.399812+0000 (oldest deadline 2024-03-24T03:20:32.099923+0000) 2024-03-24T03:20:47.211 INFO:journalctl@ceph.osd.11.smithi193.stdout:Mar 24 03:20:46 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-osd-11[62452]: 2024-03-24T03:20:46.810+0000 7f0edd521700 -1 osd.11 127 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-03-24T03:20:12.100299+0000 front 2024-03-24T03:20:12.100348+0000 (oldest deadline 2024-03-24T03:20:38.000441+0000) 2024-03-24T03:20:47.211 INFO:journalctl@ceph.osd.11.smithi193.stdout:Mar 24 03:20:46 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-osd-11[62452]: 2024-03-24T03:20:46.810+0000 7f0edd521700 -1 osd.11 127 heartbeat_check: no reply from 172.21.15.118:6814 osd.3 since back 2024-03-24T03:20:25.002506+0000 front 2024-03-24T03:20:25.002576+0000 (oldest deadline 2024-03-24T03:20:46.702390+0000) 2024-03-24T03:20:47.593 INFO:journalctl@ceph.osd.6.smithi118.stdout:Mar 24 03:20:47 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-osd-6[59976]: 2024-03-24T03:20:47.259+0000 7fbc40a00700 -1 osd.6 127 heartbeat_check: no reply from 172.21.15.118:6806 osd.0 since back 2024-03-24T03:19:56.910256+0000 front 2024-03-24T03:19:56.910419+0000 (oldest deadline 2024-03-24T03:20:21.610387+0000) 2024-03-24T03:20:47.593 INFO:journalctl@ceph.osd.6.smithi118.stdout:Mar 24 03:20:47 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-osd-6[59976]: 2024-03-24T03:20:47.259+0000 7fbc40a00700 -1 osd.6 127 heartbeat_check: no reply from 172.21.15.189:6804 osd.1 since back 2024-03-24T03:20:05.515241+0000 front 2024-03-24T03:20:05.515363+0000 (oldest deadline 2024-03-24T03:20:29.615481+0000) 2024-03-24T03:20:47.594 INFO:journalctl@ceph.osd.6.smithi118.stdout:Mar 24 03:20:47 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-osd-6[59976]: 2024-03-24T03:20:47.259+0000 7fbc40a00700 -1 osd.6 127 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-03-24T03:20:11.219038+0000 front 2024-03-24T03:20:11.218954+0000 (oldest deadline 2024-03-24T03:20:41.820368+0000) 2024-03-24T03:20:47.946 INFO:journalctl@ceph.osd.9.smithi118.stdout:Mar 24 03:20:47 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-osd-9[67027]: 2024-03-24T03:20:47.593+0000 7f69f7400700 -1 osd.9 127 heartbeat_check: no reply from 172.21.15.118:6806 osd.0 since back 2024-03-24T03:19:57.002010+0000 front 2024-03-24T03:19:57.002333+0000 (oldest deadline 2024-03-24T03:20:21.101732+0000) 2024-03-24T03:20:47.946 INFO:journalctl@ceph.osd.9.smithi118.stdout:Mar 24 03:20:47 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-osd-9[67027]: 2024-03-24T03:20:47.593+0000 7f69f7400700 -1 osd.9 127 heartbeat_check: no reply from 172.21.15.189:6804 osd.1 since back 2024-03-24T03:20:05.804520+0000 front 2024-03-24T03:20:05.804379+0000 (oldest deadline 2024-03-24T03:20:29.303978+0000) 2024-03-24T03:20:47.946 INFO:journalctl@ceph.osd.9.smithi118.stdout:Mar 24 03:20:47 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-osd-9[67027]: 2024-03-24T03:20:47.593+0000 7f69f7400700 -1 osd.9 127 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-03-24T03:20:14.507274+0000 front 2024-03-24T03:20:14.507231+0000 (oldest deadline 2024-03-24T03:20:38.607488+0000) 2024-03-24T03:20:48.203 INFO:journalctl@ceph.osd.7.smithi189.stdout:Mar 24 03:20:47 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-osd-7[54385]: 2024-03-24T03:20:47.823+0000 7fb775b24700 -1 osd.7 127 heartbeat_check: no reply from 172.21.15.189:6804 osd.1 since back 2024-03-24T03:20:04.667909+0000 front 2024-03-24T03:20:04.667932+0000 (oldest deadline 2024-03-24T03:20:29.968062+0000) 2024-03-24T03:20:48.204 INFO:journalctl@ceph.osd.7.smithi189.stdout:Mar 24 03:20:47 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-osd-7[54385]: 2024-03-24T03:20:47.823+0000 7fb775b24700 -1 osd.7 127 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-03-24T03:20:15.769900+0000 front 2024-03-24T03:20:15.769860+0000 (oldest deadline 2024-03-24T03:20:37.470057+0000) 2024-03-24T03:20:48.204 INFO:journalctl@ceph.osd.7.smithi189.stdout:Mar 24 03:20:47 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-osd-7[54385]: 2024-03-24T03:20:47.824+0000 7fb775b24700 -1 osd.7 127 heartbeat_check: no reply from 172.21.15.118:6814 osd.3 since back 2024-03-24T03:20:24.372782+0000 front 2024-03-24T03:20:24.372820+0000 (oldest deadline 2024-03-24T03:20:46.072678+0000) 2024-03-24T03:20:48.204 INFO:journalctl@ceph.osd.10.smithi189.stdout:Mar 24 03:20:47 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-osd-10[61294]: 2024-03-24T03:20:47.987+0000 7f804e31f700 -1 osd.10 127 heartbeat_check: no reply from 172.21.15.118:6806 osd.0 since back 2024-03-24T03:19:56.942815+0000 front 2024-03-24T03:19:56.942951+0000 (oldest deadline 2024-03-24T03:20:22.842714+0000) 2024-03-24T03:20:48.204 INFO:journalctl@ceph.osd.10.smithi189.stdout:Mar 24 03:20:47 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-osd-10[61294]: 2024-03-24T03:20:47.987+0000 7f804e31f700 -1 osd.10 127 heartbeat_check: no reply from 172.21.15.189:6804 osd.1 since back 2024-03-24T03:20:07.543899+0000 front 2024-03-24T03:20:07.543880+0000 (oldest deadline 2024-03-24T03:20:30.944770+0000) 2024-03-24T03:20:48.205 INFO:journalctl@ceph.osd.10.smithi189.stdout:Mar 24 03:20:47 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-osd-10[61294]: 2024-03-24T03:20:47.987+0000 7f804e31f700 -1 osd.10 127 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-03-24T03:20:15.546606+0000 front 2024-03-24T03:20:15.546459+0000 (oldest deadline 2024-03-24T03:20:37.846574+0000) 2024-03-24T03:20:48.205 INFO:journalctl@ceph.osd.10.smithi189.stdout:Mar 24 03:20:47 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-osd-10[61294]: 2024-03-24T03:20:47.987+0000 7f804e31f700 -1 osd.10 127 heartbeat_check: no reply from 172.21.15.118:6814 osd.3 since back 2024-03-24T03:20:22.547814+0000 front 2024-03-24T03:20:22.547848+0000 (oldest deadline 2024-03-24T03:20:47.748188+0000) 2024-03-24T03:20:48.210 INFO:journalctl@ceph.osd.8.smithi193.stdout:Mar 24 03:20:47 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-osd-8[55794]: 2024-03-24T03:20:47.980+0000 7f6e8b995700 -1 osd.8 127 heartbeat_check: no reply from 172.21.15.189:6804 osd.1 since back 2024-03-24T03:20:06.219689+0000 front 2024-03-24T03:20:06.219789+0000 (oldest deadline 2024-03-24T03:20:30.919690+0000) 2024-03-24T03:20:48.210 INFO:journalctl@ceph.osd.8.smithi193.stdout:Mar 24 03:20:47 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-osd-8[55794]: 2024-03-24T03:20:47.980+0000 7f6e8b995700 -1 osd.8 127 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-03-24T03:20:16.121217+0000 front 2024-03-24T03:20:16.121198+0000 (oldest deadline 2024-03-24T03:20:40.221317+0000) 2024-03-24T03:20:48.211 INFO:journalctl@ceph.osd.8.smithi193.stdout:Mar 24 03:20:47 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-osd-8[55794]: 2024-03-24T03:20:47.980+0000 7f6e8b995700 -1 osd.8 127 heartbeat_check: no reply from 172.21.15.118:6814 osd.3 since back 2024-03-24T03:20:24.922540+0000 front 2024-03-24T03:20:24.922747+0000 (oldest deadline 2024-03-24T03:20:46.622438+0000) 2024-03-24T03:20:48.211 INFO:journalctl@ceph.osd.11.smithi193.stdout:Mar 24 03:20:47 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-osd-11[62452]: 2024-03-24T03:20:47.788+0000 7f0edd521700 -1 osd.11 127 heartbeat_check: no reply from 172.21.15.118:6806 osd.0 since back 2024-03-24T03:19:59.095638+0000 front 2024-03-24T03:19:59.095627+0000 (oldest deadline 2024-03-24T03:20:20.195530+0000) 2024-03-24T03:20:48.211 INFO:journalctl@ceph.osd.11.smithi193.stdout:Mar 24 03:20:47 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-osd-11[62452]: 2024-03-24T03:20:47.788+0000 7f0edd521700 -1 osd.11 127 heartbeat_check: no reply from 172.21.15.189:6804 osd.1 since back 2024-03-24T03:20:07.399753+0000 front 2024-03-24T03:20:07.399812+0000 (oldest deadline 2024-03-24T03:20:32.099923+0000) 2024-03-24T03:20:48.211 INFO:journalctl@ceph.osd.11.smithi193.stdout:Mar 24 03:20:47 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-osd-11[62452]: 2024-03-24T03:20:47.788+0000 7f0edd521700 -1 osd.11 127 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-03-24T03:20:12.100299+0000 front 2024-03-24T03:20:12.100348+0000 (oldest deadline 2024-03-24T03:20:38.000441+0000) 2024-03-24T03:20:48.211 INFO:journalctl@ceph.osd.11.smithi193.stdout:Mar 24 03:20:47 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-osd-11[62452]: 2024-03-24T03:20:47.788+0000 7f0edd521700 -1 osd.11 127 heartbeat_check: no reply from 172.21.15.118:6814 osd.3 since back 2024-03-24T03:20:25.002506+0000 front 2024-03-24T03:20:25.002576+0000 (oldest deadline 2024-03-24T03:20:46.702390+0000) 2024-03-24T03:20:48.552 INFO:journalctl@ceph.osd.6.smithi118.stdout:Mar 24 03:20:48 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-osd-6[59976]: 2024-03-24T03:20:48.225+0000 7fbc40a00700 -1 osd.6 127 heartbeat_check: no reply from 172.21.15.118:6806 osd.0 since back 2024-03-24T03:19:56.910256+0000 front 2024-03-24T03:19:56.910419+0000 (oldest deadline 2024-03-24T03:20:21.610387+0000) 2024-03-24T03:20:48.553 INFO:journalctl@ceph.osd.6.smithi118.stdout:Mar 24 03:20:48 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-osd-6[59976]: 2024-03-24T03:20:48.225+0000 7fbc40a00700 -1 osd.6 127 heartbeat_check: no reply from 172.21.15.189:6804 osd.1 since back 2024-03-24T03:20:05.515241+0000 front 2024-03-24T03:20:05.515363+0000 (oldest deadline 2024-03-24T03:20:29.615481+0000) 2024-03-24T03:20:48.553 INFO:journalctl@ceph.osd.6.smithi118.stdout:Mar 24 03:20:48 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-osd-6[59976]: 2024-03-24T03:20:48.225+0000 7fbc40a00700 -1 osd.6 127 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-03-24T03:20:11.219038+0000 front 2024-03-24T03:20:11.218954+0000 (oldest deadline 2024-03-24T03:20:41.820368+0000) 2024-03-24T03:20:48.553 INFO:journalctl@ceph.osd.6.smithi118.stdout:Mar 24 03:20:48 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-osd-6[59976]: 2024-03-24T03:20:48.225+0000 7fbc40a00700 -1 osd.6 127 heartbeat_check: no reply from 172.21.15.118:6814 osd.3 since back 2024-03-24T03:20:21.821344+0000 front 2024-03-24T03:20:21.821971+0000 (oldest deadline 2024-03-24T03:20:47.721470+0000) 2024-03-24T03:20:48.946 INFO:journalctl@ceph.osd.9.smithi118.stdout:Mar 24 03:20:48 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-osd-9[67027]: 2024-03-24T03:20:48.552+0000 7f69f7400700 -1 osd.9 127 heartbeat_check: no reply from 172.21.15.118:6806 osd.0 since back 2024-03-24T03:19:57.002010+0000 front 2024-03-24T03:19:57.002333+0000 (oldest deadline 2024-03-24T03:20:21.101732+0000) 2024-03-24T03:20:48.946 INFO:journalctl@ceph.osd.9.smithi118.stdout:Mar 24 03:20:48 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-osd-9[67027]: 2024-03-24T03:20:48.552+0000 7f69f7400700 -1 osd.9 127 heartbeat_check: no reply from 172.21.15.189:6804 osd.1 since back 2024-03-24T03:20:05.804520+0000 front 2024-03-24T03:20:05.804379+0000 (oldest deadline 2024-03-24T03:20:29.303978+0000) 2024-03-24T03:20:48.946 INFO:journalctl@ceph.osd.9.smithi118.stdout:Mar 24 03:20:48 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-osd-9[67027]: 2024-03-24T03:20:48.552+0000 7f69f7400700 -1 osd.9 127 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-03-24T03:20:14.507274+0000 front 2024-03-24T03:20:14.507231+0000 (oldest deadline 2024-03-24T03:20:38.607488+0000) 2024-03-24T03:20:48.947 INFO:journalctl@ceph.osd.9.smithi118.stdout:Mar 24 03:20:48 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-osd-9[67027]: 2024-03-24T03:20:48.552+0000 7f69f7400700 -1 osd.9 127 heartbeat_check: no reply from 172.21.15.118:6814 osd.3 since back 2024-03-24T03:20:23.111195+0000 front 2024-03-24T03:20:23.111494+0000 (oldest deadline 2024-03-24T03:20:48.410762+0000) 2024-03-24T03:20:49.204 INFO:journalctl@ceph.osd.7.smithi189.stdout:Mar 24 03:20:48 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-osd-7[54385]: 2024-03-24T03:20:48.822+0000 7fb775b24700 -1 osd.7 127 heartbeat_check: no reply from 172.21.15.189:6804 osd.1 since back 2024-03-24T03:20:04.667909+0000 front 2024-03-24T03:20:04.667932+0000 (oldest deadline 2024-03-24T03:20:29.968062+0000) 2024-03-24T03:20:49.204 INFO:journalctl@ceph.osd.7.smithi189.stdout:Mar 24 03:20:48 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-osd-7[54385]: 2024-03-24T03:20:48.822+0000 7fb775b24700 -1 osd.7 127 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-03-24T03:20:15.769900+0000 front 2024-03-24T03:20:15.769860+0000 (oldest deadline 2024-03-24T03:20:37.470057+0000) 2024-03-24T03:20:49.204 INFO:journalctl@ceph.osd.7.smithi189.stdout:Mar 24 03:20:48 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-osd-7[54385]: 2024-03-24T03:20:48.822+0000 7fb775b24700 -1 osd.7 127 heartbeat_check: no reply from 172.21.15.118:6814 osd.3 since back 2024-03-24T03:20:24.372782+0000 front 2024-03-24T03:20:24.372820+0000 (oldest deadline 2024-03-24T03:20:46.072678+0000) 2024-03-24T03:20:49.205 INFO:journalctl@ceph.osd.10.smithi189.stdout:Mar 24 03:20:48 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-osd-10[61294]: 2024-03-24T03:20:48.991+0000 7f804e31f700 -1 osd.10 127 heartbeat_check: no reply from 172.21.15.118:6806 osd.0 since back 2024-03-24T03:19:56.942815+0000 front 2024-03-24T03:19:56.942951+0000 (oldest deadline 2024-03-24T03:20:22.842714+0000) 2024-03-24T03:20:49.205 INFO:journalctl@ceph.osd.10.smithi189.stdout:Mar 24 03:20:48 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-osd-10[61294]: 2024-03-24T03:20:48.991+0000 7f804e31f700 -1 osd.10 127 heartbeat_check: no reply from 172.21.15.189:6804 osd.1 since back 2024-03-24T03:20:07.543899+0000 front 2024-03-24T03:20:07.543880+0000 (oldest deadline 2024-03-24T03:20:30.944770+0000) 2024-03-24T03:20:49.205 INFO:journalctl@ceph.osd.10.smithi189.stdout:Mar 24 03:20:48 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-osd-10[61294]: 2024-03-24T03:20:48.991+0000 7f804e31f700 -1 osd.10 127 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-03-24T03:20:15.546606+0000 front 2024-03-24T03:20:15.546459+0000 (oldest deadline 2024-03-24T03:20:37.846574+0000) 2024-03-24T03:20:49.205 INFO:journalctl@ceph.osd.10.smithi189.stdout:Mar 24 03:20:48 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-osd-10[61294]: 2024-03-24T03:20:48.991+0000 7f804e31f700 -1 osd.10 127 heartbeat_check: no reply from 172.21.15.118:6814 osd.3 since back 2024-03-24T03:20:22.547814+0000 front 2024-03-24T03:20:22.547848+0000 (oldest deadline 2024-03-24T03:20:47.748188+0000) 2024-03-24T03:20:49.210 INFO:journalctl@ceph.osd.8.smithi193.stdout:Mar 24 03:20:48 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-osd-8[55794]: 2024-03-24T03:20:48.977+0000 7f6e8b995700 -1 osd.8 127 heartbeat_check: no reply from 172.21.15.189:6804 osd.1 since back 2024-03-24T03:20:06.219689+0000 front 2024-03-24T03:20:06.219789+0000 (oldest deadline 2024-03-24T03:20:30.919690+0000) 2024-03-24T03:20:49.210 INFO:journalctl@ceph.osd.8.smithi193.stdout:Mar 24 03:20:48 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-osd-8[55794]: 2024-03-24T03:20:48.977+0000 7f6e8b995700 -1 osd.8 127 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-03-24T03:20:16.121217+0000 front 2024-03-24T03:20:16.121198+0000 (oldest deadline 2024-03-24T03:20:40.221317+0000) 2024-03-24T03:20:49.211 INFO:journalctl@ceph.osd.8.smithi193.stdout:Mar 24 03:20:48 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-osd-8[55794]: 2024-03-24T03:20:48.977+0000 7f6e8b995700 -1 osd.8 127 heartbeat_check: no reply from 172.21.15.118:6814 osd.3 since back 2024-03-24T03:20:24.922540+0000 front 2024-03-24T03:20:24.922747+0000 (oldest deadline 2024-03-24T03:20:46.622438+0000) 2024-03-24T03:20:49.211 INFO:journalctl@ceph.osd.11.smithi193.stdout:Mar 24 03:20:48 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-osd-11[62452]: 2024-03-24T03:20:48.814+0000 7f0edd521700 -1 osd.11 127 heartbeat_check: no reply from 172.21.15.118:6806 osd.0 since back 2024-03-24T03:19:59.095638+0000 front 2024-03-24T03:19:59.095627+0000 (oldest deadline 2024-03-24T03:20:20.195530+0000) 2024-03-24T03:20:49.211 INFO:journalctl@ceph.osd.11.smithi193.stdout:Mar 24 03:20:48 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-osd-11[62452]: 2024-03-24T03:20:48.814+0000 7f0edd521700 -1 osd.11 127 heartbeat_check: no reply from 172.21.15.189:6804 osd.1 since back 2024-03-24T03:20:07.399753+0000 front 2024-03-24T03:20:07.399812+0000 (oldest deadline 2024-03-24T03:20:32.099923+0000) 2024-03-24T03:20:49.211 INFO:journalctl@ceph.osd.11.smithi193.stdout:Mar 24 03:20:48 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-osd-11[62452]: 2024-03-24T03:20:48.814+0000 7f0edd521700 -1 osd.11 127 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-03-24T03:20:12.100299+0000 front 2024-03-24T03:20:12.100348+0000 (oldest deadline 2024-03-24T03:20:38.000441+0000) 2024-03-24T03:20:49.211 INFO:journalctl@ceph.osd.11.smithi193.stdout:Mar 24 03:20:48 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-osd-11[62452]: 2024-03-24T03:20:48.814+0000 7f0edd521700 -1 osd.11 127 heartbeat_check: no reply from 172.21.15.118:6814 osd.3 since back 2024-03-24T03:20:25.002506+0000 front 2024-03-24T03:20:25.002576+0000 (oldest deadline 2024-03-24T03:20:46.702390+0000) 2024-03-24T03:20:49.507 INFO:journalctl@ceph.osd.6.smithi118.stdout:Mar 24 03:20:49 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-osd-6[59976]: 2024-03-24T03:20:49.202+0000 7fbc40a00700 -1 osd.6 127 heartbeat_check: no reply from 172.21.15.118:6806 osd.0 since back 2024-03-24T03:19:56.910256+0000 front 2024-03-24T03:19:56.910419+0000 (oldest deadline 2024-03-24T03:20:21.610387+0000) 2024-03-24T03:20:49.507 INFO:journalctl@ceph.osd.6.smithi118.stdout:Mar 24 03:20:49 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-osd-6[59976]: 2024-03-24T03:20:49.202+0000 7fbc40a00700 -1 osd.6 127 heartbeat_check: no reply from 172.21.15.189:6804 osd.1 since back 2024-03-24T03:20:05.515241+0000 front 2024-03-24T03:20:05.515363+0000 (oldest deadline 2024-03-24T03:20:29.615481+0000) 2024-03-24T03:20:49.507 INFO:journalctl@ceph.osd.6.smithi118.stdout:Mar 24 03:20:49 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-osd-6[59976]: 2024-03-24T03:20:49.202+0000 7fbc40a00700 -1 osd.6 127 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-03-24T03:20:11.219038+0000 front 2024-03-24T03:20:11.218954+0000 (oldest deadline 2024-03-24T03:20:41.820368+0000) 2024-03-24T03:20:49.507 INFO:journalctl@ceph.osd.6.smithi118.stdout:Mar 24 03:20:49 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-osd-6[59976]: 2024-03-24T03:20:49.202+0000 7fbc40a00700 -1 osd.6 127 heartbeat_check: no reply from 172.21.15.118:6814 osd.3 since back 2024-03-24T03:20:21.821344+0000 front 2024-03-24T03:20:21.821971+0000 (oldest deadline 2024-03-24T03:20:47.721470+0000) 2024-03-24T03:20:49.946 INFO:journalctl@ceph.osd.9.smithi118.stdout:Mar 24 03:20:49 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-osd-9[67027]: 2024-03-24T03:20:49.506+0000 7f69f7400700 -1 osd.9 127 heartbeat_check: no reply from 172.21.15.118:6806 osd.0 since back 2024-03-24T03:19:57.002010+0000 front 2024-03-24T03:19:57.002333+0000 (oldest deadline 2024-03-24T03:20:21.101732+0000) 2024-03-24T03:20:49.947 INFO:journalctl@ceph.osd.9.smithi118.stdout:Mar 24 03:20:49 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-osd-9[67027]: 2024-03-24T03:20:49.506+0000 7f69f7400700 -1 osd.9 127 heartbeat_check: no reply from 172.21.15.189:6804 osd.1 since back 2024-03-24T03:20:05.804520+0000 front 2024-03-24T03:20:05.804379+0000 (oldest deadline 2024-03-24T03:20:29.303978+0000) 2024-03-24T03:20:49.947 INFO:journalctl@ceph.osd.9.smithi118.stdout:Mar 24 03:20:49 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-osd-9[67027]: 2024-03-24T03:20:49.506+0000 7f69f7400700 -1 osd.9 127 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-03-24T03:20:14.507274+0000 front 2024-03-24T03:20:14.507231+0000 (oldest deadline 2024-03-24T03:20:38.607488+0000) 2024-03-24T03:20:49.947 INFO:journalctl@ceph.osd.9.smithi118.stdout:Mar 24 03:20:49 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-osd-9[67027]: 2024-03-24T03:20:49.506+0000 7f69f7400700 -1 osd.9 127 heartbeat_check: no reply from 172.21.15.118:6814 osd.3 since back 2024-03-24T03:20:23.111195+0000 front 2024-03-24T03:20:23.111494+0000 (oldest deadline 2024-03-24T03:20:48.410762+0000) 2024-03-24T03:20:50.204 INFO:journalctl@ceph.osd.7.smithi189.stdout:Mar 24 03:20:49 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-osd-7[54385]: 2024-03-24T03:20:49.801+0000 7fb775b24700 -1 osd.7 127 heartbeat_check: no reply from 172.21.15.189:6804 osd.1 since back 2024-03-24T03:20:04.667909+0000 front 2024-03-24T03:20:04.667932+0000 (oldest deadline 2024-03-24T03:20:29.968062+0000) 2024-03-24T03:20:50.204 INFO:journalctl@ceph.osd.7.smithi189.stdout:Mar 24 03:20:49 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-osd-7[54385]: 2024-03-24T03:20:49.801+0000 7fb775b24700 -1 osd.7 127 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-03-24T03:20:15.769900+0000 front 2024-03-24T03:20:15.769860+0000 (oldest deadline 2024-03-24T03:20:37.470057+0000) 2024-03-24T03:20:50.204 INFO:journalctl@ceph.osd.7.smithi189.stdout:Mar 24 03:20:49 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-osd-7[54385]: 2024-03-24T03:20:49.801+0000 7fb775b24700 -1 osd.7 127 heartbeat_check: no reply from 172.21.15.118:6814 osd.3 since back 2024-03-24T03:20:24.372782+0000 front 2024-03-24T03:20:24.372820+0000 (oldest deadline 2024-03-24T03:20:46.072678+0000) 2024-03-24T03:20:50.204 INFO:journalctl@ceph.osd.10.smithi189.stdout:Mar 24 03:20:50 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-osd-10[61294]: 2024-03-24T03:20:50.022+0000 7f804e31f700 -1 osd.10 127 heartbeat_check: no reply from 172.21.15.118:6806 osd.0 since back 2024-03-24T03:19:56.942815+0000 front 2024-03-24T03:19:56.942951+0000 (oldest deadline 2024-03-24T03:20:22.842714+0000) 2024-03-24T03:20:50.205 INFO:journalctl@ceph.osd.10.smithi189.stdout:Mar 24 03:20:50 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-osd-10[61294]: 2024-03-24T03:20:50.022+0000 7f804e31f700 -1 osd.10 127 heartbeat_check: no reply from 172.21.15.189:6804 osd.1 since back 2024-03-24T03:20:07.543899+0000 front 2024-03-24T03:20:07.543880+0000 (oldest deadline 2024-03-24T03:20:30.944770+0000) 2024-03-24T03:20:50.205 INFO:journalctl@ceph.osd.10.smithi189.stdout:Mar 24 03:20:50 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-osd-10[61294]: 2024-03-24T03:20:50.022+0000 7f804e31f700 -1 osd.10 127 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-03-24T03:20:15.546606+0000 front 2024-03-24T03:20:15.546459+0000 (oldest deadline 2024-03-24T03:20:37.846574+0000) 2024-03-24T03:20:50.205 INFO:journalctl@ceph.osd.10.smithi189.stdout:Mar 24 03:20:50 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-osd-10[61294]: 2024-03-24T03:20:50.022+0000 7f804e31f700 -1 osd.10 127 heartbeat_check: no reply from 172.21.15.118:6814 osd.3 since back 2024-03-24T03:20:22.547814+0000 front 2024-03-24T03:20:22.547848+0000 (oldest deadline 2024-03-24T03:20:47.748188+0000) 2024-03-24T03:20:50.210 INFO:journalctl@ceph.osd.8.smithi193.stdout:Mar 24 03:20:50 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-osd-8[55794]: 2024-03-24T03:20:50.020+0000 7f6e8b995700 -1 osd.8 127 heartbeat_check: no reply from 172.21.15.189:6804 osd.1 since back 2024-03-24T03:20:06.219689+0000 front 2024-03-24T03:20:06.219789+0000 (oldest deadline 2024-03-24T03:20:30.919690+0000) 2024-03-24T03:20:50.211 INFO:journalctl@ceph.osd.8.smithi193.stdout:Mar 24 03:20:50 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-osd-8[55794]: 2024-03-24T03:20:50.020+0000 7f6e8b995700 -1 osd.8 127 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-03-24T03:20:16.121217+0000 front 2024-03-24T03:20:16.121198+0000 (oldest deadline 2024-03-24T03:20:40.221317+0000) 2024-03-24T03:20:50.211 INFO:journalctl@ceph.osd.8.smithi193.stdout:Mar 24 03:20:50 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-osd-8[55794]: 2024-03-24T03:20:50.020+0000 7f6e8b995700 -1 osd.8 127 heartbeat_check: no reply from 172.21.15.118:6814 osd.3 since back 2024-03-24T03:20:24.922540+0000 front 2024-03-24T03:20:24.922747+0000 (oldest deadline 2024-03-24T03:20:46.622438+0000) 2024-03-24T03:20:50.211 INFO:journalctl@ceph.osd.11.smithi193.stdout:Mar 24 03:20:49 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-osd-11[62452]: 2024-03-24T03:20:49.788+0000 7f0edd521700 -1 osd.11 127 heartbeat_check: no reply from 172.21.15.118:6806 osd.0 since back 2024-03-24T03:19:59.095638+0000 front 2024-03-24T03:19:59.095627+0000 (oldest deadline 2024-03-24T03:20:20.195530+0000) 2024-03-24T03:20:50.211 INFO:journalctl@ceph.osd.11.smithi193.stdout:Mar 24 03:20:49 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-osd-11[62452]: 2024-03-24T03:20:49.788+0000 7f0edd521700 -1 osd.11 127 heartbeat_check: no reply from 172.21.15.189:6804 osd.1 since back 2024-03-24T03:20:07.399753+0000 front 2024-03-24T03:20:07.399812+0000 (oldest deadline 2024-03-24T03:20:32.099923+0000) 2024-03-24T03:20:50.211 INFO:journalctl@ceph.osd.11.smithi193.stdout:Mar 24 03:20:49 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-osd-11[62452]: 2024-03-24T03:20:49.788+0000 7f0edd521700 -1 osd.11 127 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-03-24T03:20:12.100299+0000 front 2024-03-24T03:20:12.100348+0000 (oldest deadline 2024-03-24T03:20:38.000441+0000) 2024-03-24T03:20:50.212 INFO:journalctl@ceph.osd.11.smithi193.stdout:Mar 24 03:20:49 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-osd-11[62452]: 2024-03-24T03:20:49.788+0000 7f0edd521700 -1 osd.11 127 heartbeat_check: no reply from 172.21.15.118:6814 osd.3 since back 2024-03-24T03:20:25.002506+0000 front 2024-03-24T03:20:25.002576+0000 (oldest deadline 2024-03-24T03:20:46.702390+0000) 2024-03-24T03:20:50.446 INFO:journalctl@ceph.osd.6.smithi118.stdout:Mar 24 03:20:50 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-osd-6[59976]: 2024-03-24T03:20:50.184+0000 7fbc40a00700 -1 osd.6 127 heartbeat_check: no reply from 172.21.15.118:6806 osd.0 since back 2024-03-24T03:19:56.910256+0000 front 2024-03-24T03:19:56.910419+0000 (oldest deadline 2024-03-24T03:20:21.610387+0000) 2024-03-24T03:20:50.446 INFO:journalctl@ceph.osd.6.smithi118.stdout:Mar 24 03:20:50 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-osd-6[59976]: 2024-03-24T03:20:50.184+0000 7fbc40a00700 -1 osd.6 127 heartbeat_check: no reply from 172.21.15.189:6804 osd.1 since back 2024-03-24T03:20:05.515241+0000 front 2024-03-24T03:20:05.515363+0000 (oldest deadline 2024-03-24T03:20:29.615481+0000) 2024-03-24T03:20:50.446 INFO:journalctl@ceph.osd.6.smithi118.stdout:Mar 24 03:20:50 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-osd-6[59976]: 2024-03-24T03:20:50.184+0000 7fbc40a00700 -1 osd.6 127 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-03-24T03:20:11.219038+0000 front 2024-03-24T03:20:11.218954+0000 (oldest deadline 2024-03-24T03:20:41.820368+0000) 2024-03-24T03:20:50.446 INFO:journalctl@ceph.osd.6.smithi118.stdout:Mar 24 03:20:50 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-osd-6[59976]: 2024-03-24T03:20:50.184+0000 7fbc40a00700 -1 osd.6 127 heartbeat_check: no reply from 172.21.15.118:6814 osd.3 since back 2024-03-24T03:20:21.821344+0000 front 2024-03-24T03:20:21.821971+0000 (oldest deadline 2024-03-24T03:20:47.721470+0000) 2024-03-24T03:20:50.946 INFO:journalctl@ceph.osd.9.smithi118.stdout:Mar 24 03:20:50 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-osd-9[67027]: 2024-03-24T03:20:50.492+0000 7f69f7400700 -1 osd.9 127 heartbeat_check: no reply from 172.21.15.118:6806 osd.0 since back 2024-03-24T03:19:57.002010+0000 front 2024-03-24T03:19:57.002333+0000 (oldest deadline 2024-03-24T03:20:21.101732+0000) 2024-03-24T03:20:50.947 INFO:journalctl@ceph.osd.9.smithi118.stdout:Mar 24 03:20:50 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-osd-9[67027]: 2024-03-24T03:20:50.492+0000 7f69f7400700 -1 osd.9 127 heartbeat_check: no reply from 172.21.15.189:6804 osd.1 since back 2024-03-24T03:20:05.804520+0000 front 2024-03-24T03:20:05.804379+0000 (oldest deadline 2024-03-24T03:20:29.303978+0000) 2024-03-24T03:20:50.947 INFO:journalctl@ceph.osd.9.smithi118.stdout:Mar 24 03:20:50 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-osd-9[67027]: 2024-03-24T03:20:50.492+0000 7f69f7400700 -1 osd.9 127 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-03-24T03:20:14.507274+0000 front 2024-03-24T03:20:14.507231+0000 (oldest deadline 2024-03-24T03:20:38.607488+0000) 2024-03-24T03:20:50.947 INFO:journalctl@ceph.osd.9.smithi118.stdout:Mar 24 03:20:50 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-osd-9[67027]: 2024-03-24T03:20:50.492+0000 7f69f7400700 -1 osd.9 127 heartbeat_check: no reply from 172.21.15.118:6814 osd.3 since back 2024-03-24T03:20:23.111195+0000 front 2024-03-24T03:20:23.111494+0000 (oldest deadline 2024-03-24T03:20:48.410762+0000) 2024-03-24T03:20:51.204 INFO:journalctl@ceph.osd.7.smithi189.stdout:Mar 24 03:20:50 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-osd-7[54385]: 2024-03-24T03:20:50.806+0000 7fb775b24700 -1 osd.7 127 heartbeat_check: no reply from 172.21.15.189:6804 osd.1 since back 2024-03-24T03:20:04.667909+0000 front 2024-03-24T03:20:04.667932+0000 (oldest deadline 2024-03-24T03:20:29.968062+0000) 2024-03-24T03:20:51.204 INFO:journalctl@ceph.osd.7.smithi189.stdout:Mar 24 03:20:50 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-osd-7[54385]: 2024-03-24T03:20:50.806+0000 7fb775b24700 -1 osd.7 127 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-03-24T03:20:15.769900+0000 front 2024-03-24T03:20:15.769860+0000 (oldest deadline 2024-03-24T03:20:37.470057+0000) 2024-03-24T03:20:51.204 INFO:journalctl@ceph.osd.7.smithi189.stdout:Mar 24 03:20:50 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-osd-7[54385]: 2024-03-24T03:20:50.806+0000 7fb775b24700 -1 osd.7 127 heartbeat_check: no reply from 172.21.15.118:6814 osd.3 since back 2024-03-24T03:20:24.372782+0000 front 2024-03-24T03:20:24.372820+0000 (oldest deadline 2024-03-24T03:20:46.072678+0000) 2024-03-24T03:20:51.204 INFO:journalctl@ceph.osd.10.smithi189.stdout:Mar 24 03:20:51 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-osd-10[61294]: 2024-03-24T03:20:51.012+0000 7f804e31f700 -1 osd.10 127 heartbeat_check: no reply from 172.21.15.118:6806 osd.0 since back 2024-03-24T03:19:56.942815+0000 front 2024-03-24T03:19:56.942951+0000 (oldest deadline 2024-03-24T03:20:22.842714+0000) 2024-03-24T03:20:51.204 INFO:journalctl@ceph.osd.10.smithi189.stdout:Mar 24 03:20:51 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-osd-10[61294]: 2024-03-24T03:20:51.012+0000 7f804e31f700 -1 osd.10 127 heartbeat_check: no reply from 172.21.15.189:6804 osd.1 since back 2024-03-24T03:20:07.543899+0000 front 2024-03-24T03:20:07.543880+0000 (oldest deadline 2024-03-24T03:20:30.944770+0000) 2024-03-24T03:20:51.204 INFO:journalctl@ceph.osd.10.smithi189.stdout:Mar 24 03:20:51 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-osd-10[61294]: 2024-03-24T03:20:51.012+0000 7f804e31f700 -1 osd.10 127 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-03-24T03:20:15.546606+0000 front 2024-03-24T03:20:15.546459+0000 (oldest deadline 2024-03-24T03:20:37.846574+0000) 2024-03-24T03:20:51.205 INFO:journalctl@ceph.osd.10.smithi189.stdout:Mar 24 03:20:51 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-osd-10[61294]: 2024-03-24T03:20:51.012+0000 7f804e31f700 -1 osd.10 127 heartbeat_check: no reply from 172.21.15.118:6814 osd.3 since back 2024-03-24T03:20:22.547814+0000 front 2024-03-24T03:20:22.547848+0000 (oldest deadline 2024-03-24T03:20:47.748188+0000) 2024-03-24T03:20:51.210 INFO:journalctl@ceph.osd.8.smithi193.stdout:Mar 24 03:20:51 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-osd-8[55794]: 2024-03-24T03:20:51.003+0000 7f6e8b995700 -1 osd.8 127 heartbeat_check: no reply from 172.21.15.189:6804 osd.1 since back 2024-03-24T03:20:06.219689+0000 front 2024-03-24T03:20:06.219789+0000 (oldest deadline 2024-03-24T03:20:30.919690+0000) 2024-03-24T03:20:51.210 INFO:journalctl@ceph.osd.8.smithi193.stdout:Mar 24 03:20:51 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-osd-8[55794]: 2024-03-24T03:20:51.003+0000 7f6e8b995700 -1 osd.8 127 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-03-24T03:20:16.121217+0000 front 2024-03-24T03:20:16.121198+0000 (oldest deadline 2024-03-24T03:20:40.221317+0000) 2024-03-24T03:20:51.210 INFO:journalctl@ceph.osd.8.smithi193.stdout:Mar 24 03:20:51 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-osd-8[55794]: 2024-03-24T03:20:51.003+0000 7f6e8b995700 -1 osd.8 127 heartbeat_check: no reply from 172.21.15.118:6814 osd.3 since back 2024-03-24T03:20:24.922540+0000 front 2024-03-24T03:20:24.922747+0000 (oldest deadline 2024-03-24T03:20:46.622438+0000) 2024-03-24T03:20:51.211 INFO:journalctl@ceph.osd.11.smithi193.stdout:Mar 24 03:20:50 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-osd-11[62452]: 2024-03-24T03:20:50.818+0000 7f0edd521700 -1 osd.11 127 heartbeat_check: no reply from 172.21.15.118:6806 osd.0 since back 2024-03-24T03:19:59.095638+0000 front 2024-03-24T03:19:59.095627+0000 (oldest deadline 2024-03-24T03:20:20.195530+0000) 2024-03-24T03:20:51.211 INFO:journalctl@ceph.osd.11.smithi193.stdout:Mar 24 03:20:50 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-osd-11[62452]: 2024-03-24T03:20:50.818+0000 7f0edd521700 -1 osd.11 127 heartbeat_check: no reply from 172.21.15.189:6804 osd.1 since back 2024-03-24T03:20:07.399753+0000 front 2024-03-24T03:20:07.399812+0000 (oldest deadline 2024-03-24T03:20:32.099923+0000) 2024-03-24T03:20:51.211 INFO:journalctl@ceph.osd.11.smithi193.stdout:Mar 24 03:20:50 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-osd-11[62452]: 2024-03-24T03:20:50.818+0000 7f0edd521700 -1 osd.11 127 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-03-24T03:20:12.100299+0000 front 2024-03-24T03:20:12.100348+0000 (oldest deadline 2024-03-24T03:20:38.000441+0000) 2024-03-24T03:20:51.211 INFO:journalctl@ceph.osd.11.smithi193.stdout:Mar 24 03:20:50 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-osd-11[62452]: 2024-03-24T03:20:50.818+0000 7f0edd521700 -1 osd.11 127 heartbeat_check: no reply from 172.21.15.118:6814 osd.3 since back 2024-03-24T03:20:25.002506+0000 front 2024-03-24T03:20:25.002576+0000 (oldest deadline 2024-03-24T03:20:46.702390+0000) 2024-03-24T03:20:51.613 INFO:journalctl@ceph.osd.6.smithi118.stdout:Mar 24 03:20:51 smithi118 podman[146268]: 2024-03-24 03:20:51.333455522 +0000 UTC m=+5.546221790 container died 2d383b46209955bf023c399b2dff60f0aa004baa4ef5d4f9ffb9dd585b828bfa (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-d79a5420-e971-11ee-95cd-87774f69a715-osd-6, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, GIT_CLEAN=True, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, org.label-schema.build-date=20240319, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD) 2024-03-24T03:20:51.613 INFO:journalctl@ceph.osd.9.smithi118.stdout:Mar 24 03:20:51 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-osd-9[67027]: 2024-03-24T03:20:51.490+0000 7f69f7400700 -1 osd.9 127 heartbeat_check: no reply from 172.21.15.118:6806 osd.0 since back 2024-03-24T03:19:57.002010+0000 front 2024-03-24T03:19:57.002333+0000 (oldest deadline 2024-03-24T03:20:21.101732+0000) 2024-03-24T03:20:51.614 INFO:journalctl@ceph.osd.9.smithi118.stdout:Mar 24 03:20:51 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-osd-9[67027]: 2024-03-24T03:20:51.490+0000 7f69f7400700 -1 osd.9 127 heartbeat_check: no reply from 172.21.15.189:6804 osd.1 since back 2024-03-24T03:20:05.804520+0000 front 2024-03-24T03:20:05.804379+0000 (oldest deadline 2024-03-24T03:20:29.303978+0000) 2024-03-24T03:20:51.614 INFO:journalctl@ceph.osd.9.smithi118.stdout:Mar 24 03:20:51 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-osd-9[67027]: 2024-03-24T03:20:51.490+0000 7f69f7400700 -1 osd.9 127 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-03-24T03:20:14.507274+0000 front 2024-03-24T03:20:14.507231+0000 (oldest deadline 2024-03-24T03:20:38.607488+0000) 2024-03-24T03:20:51.614 INFO:journalctl@ceph.osd.9.smithi118.stdout:Mar 24 03:20:51 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-osd-9[67027]: 2024-03-24T03:20:51.490+0000 7f69f7400700 -1 osd.9 127 heartbeat_check: no reply from 172.21.15.118:6814 osd.3 since back 2024-03-24T03:20:23.111195+0000 front 2024-03-24T03:20:23.111494+0000 (oldest deadline 2024-03-24T03:20:48.410762+0000) 2024-03-24T03:20:51.945 INFO:journalctl@ceph.osd.6.smithi118.stdout:Mar 24 03:20:51 smithi118 podman[146268]: 2024-03-24 03:20:51.708799081 +0000 UTC m=+5.921565393 container cleanup 2d383b46209955bf023c399b2dff60f0aa004baa4ef5d4f9ffb9dd585b828bfa (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-d79a5420-e971-11ee-95cd-87774f69a715-osd-6, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, ceph=True, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240319, CEPH_POINT_RELEASE=, GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2) 2024-03-24T03:20:51.946 INFO:journalctl@ceph.osd.6.smithi118.stdout:Mar 24 03:20:51 smithi118 bash[146268]: ceph-d79a5420-e971-11ee-95cd-87774f69a715-osd-6 2024-03-24T03:20:51.953 INFO:journalctl@ceph.osd.7.smithi189.stdout:Mar 24 03:20:51 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-osd-7[54385]: 2024-03-24T03:20:51.787+0000 7fb775b24700 -1 osd.7 127 heartbeat_check: no reply from 172.21.15.189:6804 osd.1 since back 2024-03-24T03:20:04.667909+0000 front 2024-03-24T03:20:04.667932+0000 (oldest deadline 2024-03-24T03:20:29.968062+0000) 2024-03-24T03:20:51.953 INFO:journalctl@ceph.osd.7.smithi189.stdout:Mar 24 03:20:51 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-osd-7[54385]: 2024-03-24T03:20:51.787+0000 7fb775b24700 -1 osd.7 127 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-03-24T03:20:15.769900+0000 front 2024-03-24T03:20:15.769860+0000 (oldest deadline 2024-03-24T03:20:37.470057+0000) 2024-03-24T03:20:51.954 INFO:journalctl@ceph.osd.7.smithi189.stdout:Mar 24 03:20:51 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-osd-7[54385]: 2024-03-24T03:20:51.787+0000 7fb775b24700 -1 osd.7 127 heartbeat_check: no reply from 172.21.15.118:6814 osd.3 since back 2024-03-24T03:20:24.372782+0000 front 2024-03-24T03:20:24.372820+0000 (oldest deadline 2024-03-24T03:20:46.072678+0000) 2024-03-24T03:20:52.210 INFO:journalctl@ceph.osd.8.smithi193.stdout:Mar 24 03:20:51 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-osd-8[55794]: 2024-03-24T03:20:51.982+0000 7f6e8b995700 -1 osd.8 127 heartbeat_check: no reply from 172.21.15.189:6804 osd.1 since back 2024-03-24T03:20:06.219689+0000 front 2024-03-24T03:20:06.219789+0000 (oldest deadline 2024-03-24T03:20:30.919690+0000) 2024-03-24T03:20:52.211 INFO:journalctl@ceph.osd.8.smithi193.stdout:Mar 24 03:20:51 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-osd-8[55794]: 2024-03-24T03:20:51.982+0000 7f6e8b995700 -1 osd.8 127 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-03-24T03:20:16.121217+0000 front 2024-03-24T03:20:16.121198+0000 (oldest deadline 2024-03-24T03:20:40.221317+0000) 2024-03-24T03:20:52.211 INFO:journalctl@ceph.osd.8.smithi193.stdout:Mar 24 03:20:51 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-osd-8[55794]: 2024-03-24T03:20:51.982+0000 7f6e8b995700 -1 osd.8 127 heartbeat_check: no reply from 172.21.15.118:6814 osd.3 since back 2024-03-24T03:20:24.922540+0000 front 2024-03-24T03:20:24.922747+0000 (oldest deadline 2024-03-24T03:20:46.622438+0000) 2024-03-24T03:20:52.211 INFO:journalctl@ceph.osd.11.smithi193.stdout:Mar 24 03:20:51 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-osd-11[62452]: 2024-03-24T03:20:51.809+0000 7f0edd521700 -1 osd.11 127 heartbeat_check: no reply from 172.21.15.118:6806 osd.0 since back 2024-03-24T03:19:59.095638+0000 front 2024-03-24T03:19:59.095627+0000 (oldest deadline 2024-03-24T03:20:20.195530+0000) 2024-03-24T03:20:52.211 INFO:journalctl@ceph.osd.11.smithi193.stdout:Mar 24 03:20:51 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-osd-11[62452]: 2024-03-24T03:20:51.809+0000 7f0edd521700 -1 osd.11 127 heartbeat_check: no reply from 172.21.15.189:6804 osd.1 since back 2024-03-24T03:20:07.399753+0000 front 2024-03-24T03:20:07.399812+0000 (oldest deadline 2024-03-24T03:20:32.099923+0000) 2024-03-24T03:20:52.211 INFO:journalctl@ceph.osd.11.smithi193.stdout:Mar 24 03:20:51 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-osd-11[62452]: 2024-03-24T03:20:51.809+0000 7f0edd521700 -1 osd.11 127 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-03-24T03:20:12.100299+0000 front 2024-03-24T03:20:12.100348+0000 (oldest deadline 2024-03-24T03:20:38.000441+0000) 2024-03-24T03:20:52.212 INFO:journalctl@ceph.osd.11.smithi193.stdout:Mar 24 03:20:51 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-osd-11[62452]: 2024-03-24T03:20:51.809+0000 7f0edd521700 -1 osd.11 127 heartbeat_check: no reply from 172.21.15.118:6814 osd.3 since back 2024-03-24T03:20:25.002506+0000 front 2024-03-24T03:20:25.002576+0000 (oldest deadline 2024-03-24T03:20:46.702390+0000) 2024-03-24T03:20:52.446 INFO:journalctl@ceph.osd.6.smithi118.stdout:Mar 24 03:20:52 smithi118 podman[146288]: 2024-03-24 03:20:52.02678456 +0000 UTC m=+0.680792306 container remove 2d383b46209955bf023c399b2dff60f0aa004baa4ef5d4f9ffb9dd585b828bfa (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-d79a5420-e971-11ee-95cd-87774f69a715-osd-6, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240319, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, ceph=True, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_CLEAN=True) 2024-03-24T03:20:52.454 INFO:journalctl@ceph.osd.10.smithi189.stdout:Mar 24 03:20:51 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-osd-10[61294]: 2024-03-24T03:20:51.988+0000 7f804e31f700 -1 osd.10 127 heartbeat_check: no reply from 172.21.15.118:6806 osd.0 since back 2024-03-24T03:19:56.942815+0000 front 2024-03-24T03:19:56.942951+0000 (oldest deadline 2024-03-24T03:20:22.842714+0000) 2024-03-24T03:20:52.454 INFO:journalctl@ceph.osd.10.smithi189.stdout:Mar 24 03:20:51 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-osd-10[61294]: 2024-03-24T03:20:51.988+0000 7f804e31f700 -1 osd.10 127 heartbeat_check: no reply from 172.21.15.189:6804 osd.1 since back 2024-03-24T03:20:07.543899+0000 front 2024-03-24T03:20:07.543880+0000 (oldest deadline 2024-03-24T03:20:30.944770+0000) 2024-03-24T03:20:52.454 INFO:journalctl@ceph.osd.10.smithi189.stdout:Mar 24 03:20:51 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-osd-10[61294]: 2024-03-24T03:20:51.988+0000 7f804e31f700 -1 osd.10 127 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-03-24T03:20:15.546606+0000 front 2024-03-24T03:20:15.546459+0000 (oldest deadline 2024-03-24T03:20:37.846574+0000) 2024-03-24T03:20:52.454 INFO:journalctl@ceph.osd.10.smithi189.stdout:Mar 24 03:20:51 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-osd-10[61294]: 2024-03-24T03:20:51.988+0000 7f804e31f700 -1 osd.10 127 heartbeat_check: no reply from 172.21.15.118:6814 osd.3 since back 2024-03-24T03:20:22.547814+0000 front 2024-03-24T03:20:22.547848+0000 (oldest deadline 2024-03-24T03:20:47.748188+0000) 2024-03-24T03:20:52.748 INFO:journalctl@ceph.osd.6.smithi118.stdout:Mar 24 03:20:52 smithi118 podman[146381]: 2024-03-24 03:20:52.480220637 +0000 UTC m=+0.035117922 image pull c664afe56bb01db4f9afa9532c20598e201e664f2e9ba2be50db0d52674a26d2 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc 2024-03-24T03:20:52.749 INFO:journalctl@ceph.osd.6.smithi118.stdout:Mar 24 03:20:52 smithi118 podman[146381]: 2024-03-24 03:20:52.594410153 +0000 UTC m=+0.149307476 container create 3606b5b36a98738c242dddf3a3e21a9e2e642f54802a6a750f891485b34c0418 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-d79a5420-e971-11ee-95cd-87774f69a715-osd-6-deactivate, org.label-schema.license=GPLv2, GIT_CLEAN=True, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.build-date=20240319) 2024-03-24T03:20:52.749 INFO:journalctl@ceph.osd.9.smithi118.stdout:Mar 24 03:20:52 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-osd-9[67027]: 2024-03-24T03:20:52.491+0000 7f69f7400700 -1 osd.9 127 heartbeat_check: no reply from 172.21.15.118:6806 osd.0 since back 2024-03-24T03:19:57.002010+0000 front 2024-03-24T03:19:57.002333+0000 (oldest deadline 2024-03-24T03:20:21.101732+0000) 2024-03-24T03:20:52.749 INFO:journalctl@ceph.osd.9.smithi118.stdout:Mar 24 03:20:52 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-osd-9[67027]: 2024-03-24T03:20:52.491+0000 7f69f7400700 -1 osd.9 127 heartbeat_check: no reply from 172.21.15.189:6804 osd.1 since back 2024-03-24T03:20:05.804520+0000 front 2024-03-24T03:20:05.804379+0000 (oldest deadline 2024-03-24T03:20:29.303978+0000) 2024-03-24T03:20:52.749 INFO:journalctl@ceph.osd.9.smithi118.stdout:Mar 24 03:20:52 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-osd-9[67027]: 2024-03-24T03:20:52.491+0000 7f69f7400700 -1 osd.9 127 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-03-24T03:20:14.507274+0000 front 2024-03-24T03:20:14.507231+0000 (oldest deadline 2024-03-24T03:20:38.607488+0000) 2024-03-24T03:20:52.749 INFO:journalctl@ceph.osd.9.smithi118.stdout:Mar 24 03:20:52 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-osd-9[67027]: 2024-03-24T03:20:52.491+0000 7f69f7400700 -1 osd.9 127 heartbeat_check: no reply from 172.21.15.118:6814 osd.3 since back 2024-03-24T03:20:23.111195+0000 front 2024-03-24T03:20:23.111494+0000 (oldest deadline 2024-03-24T03:20:48.410762+0000) 2024-03-24T03:20:53.091 INFO:journalctl@ceph.osd.6.smithi118.stdout:Mar 24 03:20:52 smithi118 podman[146381]: 2024-03-24 03:20:52.856200239 +0000 UTC m=+0.411097592 container init 3606b5b36a98738c242dddf3a3e21a9e2e642f54802a6a750f891485b34c0418 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-d79a5420-e971-11ee-95cd-87774f69a715-osd-6-deactivate, org.label-schema.license=GPLv2, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, GIT_CLEAN=True, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240319, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-24T03:20:53.091 INFO:journalctl@ceph.osd.6.smithi118.stdout:Mar 24 03:20:52 smithi118 podman[146381]: 2024-03-24 03:20:52.867737525 +0000 UTC m=+0.422634827 container start 3606b5b36a98738c242dddf3a3e21a9e2e642f54802a6a750f891485b34c0418 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-d79a5420-e971-11ee-95cd-87774f69a715-osd-6-deactivate, maintainer=Guillaume Abrioux , GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240319, GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0) 2024-03-24T03:20:53.092 INFO:journalctl@ceph.osd.6.smithi118.stdout:Mar 24 03:20:52 smithi118 podman[146381]: 2024-03-24 03:20:52.963492235 +0000 UTC m=+0.518389588 container attach 3606b5b36a98738c242dddf3a3e21a9e2e642f54802a6a750f891485b34c0418 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-d79a5420-e971-11ee-95cd-87774f69a715-osd-6-deactivate, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, ceph=True, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, GIT_CLEAN=True, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, org.label-schema.build-date=20240319, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30) 2024-03-24T03:20:53.204 INFO:journalctl@ceph.osd.10.smithi189.stdout:Mar 24 03:20:52 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-osd-10[61294]: 2024-03-24T03:20:52.955+0000 7f804e31f700 -1 osd.10 127 heartbeat_check: no reply from 172.21.15.118:6806 osd.0 since back 2024-03-24T03:19:56.942815+0000 front 2024-03-24T03:19:56.942951+0000 (oldest deadline 2024-03-24T03:20:22.842714+0000) 2024-03-24T03:20:53.204 INFO:journalctl@ceph.osd.10.smithi189.stdout:Mar 24 03:20:52 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-osd-10[61294]: 2024-03-24T03:20:52.955+0000 7f804e31f700 -1 osd.10 127 heartbeat_check: no reply from 172.21.15.189:6804 osd.1 since back 2024-03-24T03:20:07.543899+0000 front 2024-03-24T03:20:07.543880+0000 (oldest deadline 2024-03-24T03:20:30.944770+0000) 2024-03-24T03:20:53.204 INFO:journalctl@ceph.osd.10.smithi189.stdout:Mar 24 03:20:52 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-osd-10[61294]: 2024-03-24T03:20:52.955+0000 7f804e31f700 -1 osd.10 127 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-03-24T03:20:15.546606+0000 front 2024-03-24T03:20:15.546459+0000 (oldest deadline 2024-03-24T03:20:37.846574+0000) 2024-03-24T03:20:53.204 INFO:journalctl@ceph.osd.10.smithi189.stdout:Mar 24 03:20:52 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-osd-10[61294]: 2024-03-24T03:20:52.955+0000 7f804e31f700 -1 osd.10 127 heartbeat_check: no reply from 172.21.15.118:6814 osd.3 since back 2024-03-24T03:20:22.547814+0000 front 2024-03-24T03:20:22.547848+0000 (oldest deadline 2024-03-24T03:20:47.748188+0000) 2024-03-24T03:20:53.204 INFO:journalctl@ceph.osd.7.smithi189.stdout:Mar 24 03:20:52 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-osd-7[54385]: 2024-03-24T03:20:52.753+0000 7fb775b24700 -1 osd.7 127 heartbeat_check: no reply from 172.21.15.189:6804 osd.1 since back 2024-03-24T03:20:04.667909+0000 front 2024-03-24T03:20:04.667932+0000 (oldest deadline 2024-03-24T03:20:29.968062+0000) 2024-03-24T03:20:53.204 INFO:journalctl@ceph.osd.7.smithi189.stdout:Mar 24 03:20:52 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-osd-7[54385]: 2024-03-24T03:20:52.753+0000 7fb775b24700 -1 osd.7 127 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-03-24T03:20:15.769900+0000 front 2024-03-24T03:20:15.769860+0000 (oldest deadline 2024-03-24T03:20:37.470057+0000) 2024-03-24T03:20:53.205 INFO:journalctl@ceph.osd.7.smithi189.stdout:Mar 24 03:20:52 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-osd-7[54385]: 2024-03-24T03:20:52.753+0000 7fb775b24700 -1 osd.7 127 heartbeat_check: no reply from 172.21.15.118:6814 osd.3 since back 2024-03-24T03:20:24.372782+0000 front 2024-03-24T03:20:24.372820+0000 (oldest deadline 2024-03-24T03:20:46.072678+0000) 2024-03-24T03:20:53.210 INFO:journalctl@ceph.osd.8.smithi193.stdout:Mar 24 03:20:52 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-osd-8[55794]: 2024-03-24T03:20:52.977+0000 7f6e8b995700 -1 osd.8 127 heartbeat_check: no reply from 172.21.15.189:6804 osd.1 since back 2024-03-24T03:20:06.219689+0000 front 2024-03-24T03:20:06.219789+0000 (oldest deadline 2024-03-24T03:20:30.919690+0000) 2024-03-24T03:20:53.211 INFO:journalctl@ceph.osd.8.smithi193.stdout:Mar 24 03:20:52 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-osd-8[55794]: 2024-03-24T03:20:52.977+0000 7f6e8b995700 -1 osd.8 127 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-03-24T03:20:16.121217+0000 front 2024-03-24T03:20:16.121198+0000 (oldest deadline 2024-03-24T03:20:40.221317+0000) 2024-03-24T03:20:53.211 INFO:journalctl@ceph.osd.8.smithi193.stdout:Mar 24 03:20:52 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-osd-8[55794]: 2024-03-24T03:20:52.977+0000 7f6e8b995700 -1 osd.8 127 heartbeat_check: no reply from 172.21.15.118:6814 osd.3 since back 2024-03-24T03:20:24.922540+0000 front 2024-03-24T03:20:24.922747+0000 (oldest deadline 2024-03-24T03:20:46.622438+0000) 2024-03-24T03:20:53.211 INFO:journalctl@ceph.osd.11.smithi193.stdout:Mar 24 03:20:52 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-osd-11[62452]: 2024-03-24T03:20:52.842+0000 7f0edd521700 -1 osd.11 127 heartbeat_check: no reply from 172.21.15.118:6806 osd.0 since back 2024-03-24T03:19:59.095638+0000 front 2024-03-24T03:19:59.095627+0000 (oldest deadline 2024-03-24T03:20:20.195530+0000) 2024-03-24T03:20:53.211 INFO:journalctl@ceph.osd.11.smithi193.stdout:Mar 24 03:20:52 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-osd-11[62452]: 2024-03-24T03:20:52.842+0000 7f0edd521700 -1 osd.11 127 heartbeat_check: no reply from 172.21.15.189:6804 osd.1 since back 2024-03-24T03:20:07.399753+0000 front 2024-03-24T03:20:07.399812+0000 (oldest deadline 2024-03-24T03:20:32.099923+0000) 2024-03-24T03:20:53.211 INFO:journalctl@ceph.osd.11.smithi193.stdout:Mar 24 03:20:52 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-osd-11[62452]: 2024-03-24T03:20:52.842+0000 7f0edd521700 -1 osd.11 127 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-03-24T03:20:12.100299+0000 front 2024-03-24T03:20:12.100348+0000 (oldest deadline 2024-03-24T03:20:38.000441+0000) 2024-03-24T03:20:53.211 INFO:journalctl@ceph.osd.11.smithi193.stdout:Mar 24 03:20:52 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-osd-11[62452]: 2024-03-24T03:20:52.842+0000 7f0edd521700 -1 osd.11 127 heartbeat_check: no reply from 172.21.15.118:6814 osd.3 since back 2024-03-24T03:20:25.002506+0000 front 2024-03-24T03:20:25.002576+0000 (oldest deadline 2024-03-24T03:20:46.702390+0000) 2024-03-24T03:20:53.444 INFO:journalctl@ceph.osd.6.smithi118.stdout:Mar 24 03:20:53 smithi118 podman[146410]: 2024-03-24 03:20:53.317193238 +0000 UTC m=+0.032129532 container died 3606b5b36a98738c242dddf3a3e21a9e2e642f54802a6a750f891485b34c0418 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-d79a5420-e971-11ee-95cd-87774f69a715-osd-6-deactivate, io.buildah.version=1.33.5, ceph=True, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, org.label-schema.build-date=20240319, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, org.label-schema.schema-version=1.0) 2024-03-24T03:20:53.696 INFO:journalctl@ceph.osd.9.smithi118.stdout:Mar 24 03:20:53 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-osd-9[67027]: 2024-03-24T03:20:53.444+0000 7f69f7400700 -1 osd.9 127 heartbeat_check: no reply from 172.21.15.118:6806 osd.0 since back 2024-03-24T03:19:57.002010+0000 front 2024-03-24T03:19:57.002333+0000 (oldest deadline 2024-03-24T03:20:21.101732+0000) 2024-03-24T03:20:53.696 INFO:journalctl@ceph.osd.9.smithi118.stdout:Mar 24 03:20:53 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-osd-9[67027]: 2024-03-24T03:20:53.444+0000 7f69f7400700 -1 osd.9 127 heartbeat_check: no reply from 172.21.15.189:6804 osd.1 since back 2024-03-24T03:20:05.804520+0000 front 2024-03-24T03:20:05.804379+0000 (oldest deadline 2024-03-24T03:20:29.303978+0000) 2024-03-24T03:20:53.697 INFO:journalctl@ceph.osd.9.smithi118.stdout:Mar 24 03:20:53 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-osd-9[67027]: 2024-03-24T03:20:53.444+0000 7f69f7400700 -1 osd.9 127 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-03-24T03:20:14.507274+0000 front 2024-03-24T03:20:14.507231+0000 (oldest deadline 2024-03-24T03:20:38.607488+0000) 2024-03-24T03:20:53.697 INFO:journalctl@ceph.osd.9.smithi118.stdout:Mar 24 03:20:53 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-osd-9[67027]: 2024-03-24T03:20:53.444+0000 7f69f7400700 -1 osd.9 127 heartbeat_check: no reply from 172.21.15.118:6814 osd.3 since back 2024-03-24T03:20:23.111195+0000 front 2024-03-24T03:20:23.111494+0000 (oldest deadline 2024-03-24T03:20:48.410762+0000) 2024-03-24T03:20:53.971 DEBUG:teuthology.orchestra.run.smithi118:> sudo pkill -f 'journalctl -f -n 0 -u ceph-d79a5420-e971-11ee-95cd-87774f69a715@osd.6.service' 2024-03-24T03:20:54.087 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-24T03:20:54.087 INFO:tasks.cephadm.osd.6:Stopped osd.6 2024-03-24T03:20:54.087 INFO:tasks.cephadm.osd.7:Stopping osd.7... 2024-03-24T03:20:54.087 DEBUG:teuthology.orchestra.run.smithi189:> sudo systemctl stop ceph-d79a5420-e971-11ee-95cd-87774f69a715@osd.7 2024-03-24T03:20:54.118 INFO:journalctl@ceph.osd.10.smithi189.stdout:Mar 24 03:20:53 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-osd-10[61294]: 2024-03-24T03:20:53.924+0000 7f804e31f700 -1 osd.10 127 heartbeat_check: no reply from 172.21.15.118:6806 osd.0 since back 2024-03-24T03:19:56.942815+0000 front 2024-03-24T03:19:56.942951+0000 (oldest deadline 2024-03-24T03:20:22.842714+0000) 2024-03-24T03:20:54.119 INFO:journalctl@ceph.osd.10.smithi189.stdout:Mar 24 03:20:53 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-osd-10[61294]: 2024-03-24T03:20:53.924+0000 7f804e31f700 -1 osd.10 127 heartbeat_check: no reply from 172.21.15.189:6804 osd.1 since back 2024-03-24T03:20:07.543899+0000 front 2024-03-24T03:20:07.543880+0000 (oldest deadline 2024-03-24T03:20:30.944770+0000) 2024-03-24T03:20:54.119 INFO:journalctl@ceph.osd.10.smithi189.stdout:Mar 24 03:20:53 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-osd-10[61294]: 2024-03-24T03:20:53.924+0000 7f804e31f700 -1 osd.10 127 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-03-24T03:20:15.546606+0000 front 2024-03-24T03:20:15.546459+0000 (oldest deadline 2024-03-24T03:20:37.846574+0000) 2024-03-24T03:20:54.119 INFO:journalctl@ceph.osd.10.smithi189.stdout:Mar 24 03:20:53 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-osd-10[61294]: 2024-03-24T03:20:53.924+0000 7f804e31f700 -1 osd.10 127 heartbeat_check: no reply from 172.21.15.118:6814 osd.3 since back 2024-03-24T03:20:22.547814+0000 front 2024-03-24T03:20:22.547848+0000 (oldest deadline 2024-03-24T03:20:47.748188+0000) 2024-03-24T03:20:54.119 INFO:journalctl@ceph.osd.7.smithi189.stdout:Mar 24 03:20:53 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-osd-7[54385]: 2024-03-24T03:20:53.789+0000 7fb775b24700 -1 osd.7 127 heartbeat_check: no reply from 172.21.15.189:6804 osd.1 since back 2024-03-24T03:20:04.667909+0000 front 2024-03-24T03:20:04.667932+0000 (oldest deadline 2024-03-24T03:20:29.968062+0000) 2024-03-24T03:20:54.119 INFO:journalctl@ceph.osd.7.smithi189.stdout:Mar 24 03:20:53 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-osd-7[54385]: 2024-03-24T03:20:53.789+0000 7fb775b24700 -1 osd.7 127 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-03-24T03:20:15.769900+0000 front 2024-03-24T03:20:15.769860+0000 (oldest deadline 2024-03-24T03:20:37.470057+0000) 2024-03-24T03:20:54.119 INFO:journalctl@ceph.osd.7.smithi189.stdout:Mar 24 03:20:53 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-osd-7[54385]: 2024-03-24T03:20:53.789+0000 7fb775b24700 -1 osd.7 127 heartbeat_check: no reply from 172.21.15.118:6814 osd.3 since back 2024-03-24T03:20:24.372782+0000 front 2024-03-24T03:20:24.372820+0000 (oldest deadline 2024-03-24T03:20:46.072678+0000) 2024-03-24T03:20:54.210 INFO:journalctl@ceph.osd.11.smithi193.stdout:Mar 24 03:20:53 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-osd-11[62452]: 2024-03-24T03:20:53.887+0000 7f0edd521700 -1 osd.11 127 heartbeat_check: no reply from 172.21.15.118:6806 osd.0 since back 2024-03-24T03:19:59.095638+0000 front 2024-03-24T03:19:59.095627+0000 (oldest deadline 2024-03-24T03:20:20.195530+0000) 2024-03-24T03:20:54.211 INFO:journalctl@ceph.osd.11.smithi193.stdout:Mar 24 03:20:53 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-osd-11[62452]: 2024-03-24T03:20:53.887+0000 7f0edd521700 -1 osd.11 127 heartbeat_check: no reply from 172.21.15.189:6804 osd.1 since back 2024-03-24T03:20:07.399753+0000 front 2024-03-24T03:20:07.399812+0000 (oldest deadline 2024-03-24T03:20:32.099923+0000) 2024-03-24T03:20:54.211 INFO:journalctl@ceph.osd.11.smithi193.stdout:Mar 24 03:20:53 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-osd-11[62452]: 2024-03-24T03:20:53.887+0000 7f0edd521700 -1 osd.11 127 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-03-24T03:20:12.100299+0000 front 2024-03-24T03:20:12.100348+0000 (oldest deadline 2024-03-24T03:20:38.000441+0000) 2024-03-24T03:20:54.211 INFO:journalctl@ceph.osd.11.smithi193.stdout:Mar 24 03:20:53 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-osd-11[62452]: 2024-03-24T03:20:53.887+0000 7f0edd521700 -1 osd.11 127 heartbeat_check: no reply from 172.21.15.118:6814 osd.3 since back 2024-03-24T03:20:25.002506+0000 front 2024-03-24T03:20:25.002576+0000 (oldest deadline 2024-03-24T03:20:46.702390+0000) 2024-03-24T03:20:54.211 INFO:journalctl@ceph.osd.8.smithi193.stdout:Mar 24 03:20:53 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-osd-8[55794]: 2024-03-24T03:20:53.928+0000 7f6e8b995700 -1 osd.8 127 heartbeat_check: no reply from 172.21.15.189:6804 osd.1 since back 2024-03-24T03:20:06.219689+0000 front 2024-03-24T03:20:06.219789+0000 (oldest deadline 2024-03-24T03:20:30.919690+0000) 2024-03-24T03:20:54.211 INFO:journalctl@ceph.osd.8.smithi193.stdout:Mar 24 03:20:53 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-osd-8[55794]: 2024-03-24T03:20:53.928+0000 7f6e8b995700 -1 osd.8 127 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-03-24T03:20:16.121217+0000 front 2024-03-24T03:20:16.121198+0000 (oldest deadline 2024-03-24T03:20:40.221317+0000) 2024-03-24T03:20:54.211 INFO:journalctl@ceph.osd.8.smithi193.stdout:Mar 24 03:20:53 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-osd-8[55794]: 2024-03-24T03:20:53.928+0000 7f6e8b995700 -1 osd.8 127 heartbeat_check: no reply from 172.21.15.118:6814 osd.3 since back 2024-03-24T03:20:24.922540+0000 front 2024-03-24T03:20:24.922747+0000 (oldest deadline 2024-03-24T03:20:46.622438+0000) 2024-03-24T03:20:54.402 INFO:journalctl@ceph.osd.7.smithi189.stdout:Mar 24 03:20:54 smithi189 systemd[1]: Stopping Ceph osd.7 for d79a5420-e971-11ee-95cd-87774f69a715... 2024-03-24T03:20:54.402 INFO:journalctl@ceph.osd.7.smithi189.stdout:Mar 24 03:20:54 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-osd-7[54385]: 2024-03-24T03:20:54.402+0000 7fb779530700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.7 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-24T03:20:54.703 INFO:journalctl@ceph.osd.7.smithi189.stdout:Mar 24 03:20:54 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-osd-7[54385]: 2024-03-24T03:20:54.402+0000 7fb779530700 -1 osd.7 127 *** Got signal Terminated *** 2024-03-24T03:20:54.704 INFO:journalctl@ceph.osd.7.smithi189.stdout:Mar 24 03:20:54 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-osd-7[54385]: 2024-03-24T03:20:54.402+0000 7fb779530700 -1 osd.7 127 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-24T03:20:54.946 INFO:journalctl@ceph.osd.9.smithi118.stdout:Mar 24 03:20:54 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-osd-9[67027]: 2024-03-24T03:20:54.465+0000 7f69f7400700 -1 osd.9 127 heartbeat_check: no reply from 172.21.15.118:6806 osd.0 since back 2024-03-24T03:19:57.002010+0000 front 2024-03-24T03:19:57.002333+0000 (oldest deadline 2024-03-24T03:20:21.101732+0000) 2024-03-24T03:20:54.946 INFO:journalctl@ceph.osd.9.smithi118.stdout:Mar 24 03:20:54 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-osd-9[67027]: 2024-03-24T03:20:54.465+0000 7f69f7400700 -1 osd.9 127 heartbeat_check: no reply from 172.21.15.189:6804 osd.1 since back 2024-03-24T03:20:05.804520+0000 front 2024-03-24T03:20:05.804379+0000 (oldest deadline 2024-03-24T03:20:29.303978+0000) 2024-03-24T03:20:54.947 INFO:journalctl@ceph.osd.9.smithi118.stdout:Mar 24 03:20:54 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-osd-9[67027]: 2024-03-24T03:20:54.465+0000 7f69f7400700 -1 osd.9 127 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-03-24T03:20:14.507274+0000 front 2024-03-24T03:20:14.507231+0000 (oldest deadline 2024-03-24T03:20:38.607488+0000) 2024-03-24T03:20:54.947 INFO:journalctl@ceph.osd.9.smithi118.stdout:Mar 24 03:20:54 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-osd-9[67027]: 2024-03-24T03:20:54.465+0000 7f69f7400700 -1 osd.9 127 heartbeat_check: no reply from 172.21.15.118:6814 osd.3 since back 2024-03-24T03:20:23.111195+0000 front 2024-03-24T03:20:23.111494+0000 (oldest deadline 2024-03-24T03:20:48.410762+0000) 2024-03-24T03:20:55.204 INFO:journalctl@ceph.osd.7.smithi189.stdout:Mar 24 03:20:54 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-osd-7[54385]: 2024-03-24T03:20:54.802+0000 7fb775b24700 -1 osd.7 127 heartbeat_check: no reply from 172.21.15.189:6804 osd.1 since back 2024-03-24T03:20:04.667909+0000 front 2024-03-24T03:20:04.667932+0000 (oldest deadline 2024-03-24T03:20:29.968062+0000) 2024-03-24T03:20:55.204 INFO:journalctl@ceph.osd.7.smithi189.stdout:Mar 24 03:20:54 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-osd-7[54385]: 2024-03-24T03:20:54.802+0000 7fb775b24700 -1 osd.7 127 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-03-24T03:20:15.769900+0000 front 2024-03-24T03:20:15.769860+0000 (oldest deadline 2024-03-24T03:20:37.470057+0000) 2024-03-24T03:20:55.204 INFO:journalctl@ceph.osd.7.smithi189.stdout:Mar 24 03:20:54 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-osd-7[54385]: 2024-03-24T03:20:54.802+0000 7fb775b24700 -1 osd.7 127 heartbeat_check: no reply from 172.21.15.118:6814 osd.3 since back 2024-03-24T03:20:24.372782+0000 front 2024-03-24T03:20:24.372820+0000 (oldest deadline 2024-03-24T03:20:46.072678+0000) 2024-03-24T03:20:55.204 INFO:journalctl@ceph.osd.10.smithi189.stdout:Mar 24 03:20:54 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-osd-10[61294]: 2024-03-24T03:20:54.973+0000 7f804e31f700 -1 osd.10 127 heartbeat_check: no reply from 172.21.15.118:6806 osd.0 since back 2024-03-24T03:19:56.942815+0000 front 2024-03-24T03:19:56.942951+0000 (oldest deadline 2024-03-24T03:20:22.842714+0000) 2024-03-24T03:20:55.204 INFO:journalctl@ceph.osd.10.smithi189.stdout:Mar 24 03:20:54 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-osd-10[61294]: 2024-03-24T03:20:54.973+0000 7f804e31f700 -1 osd.10 127 heartbeat_check: no reply from 172.21.15.189:6804 osd.1 since back 2024-03-24T03:20:07.543899+0000 front 2024-03-24T03:20:07.543880+0000 (oldest deadline 2024-03-24T03:20:30.944770+0000) 2024-03-24T03:20:55.204 INFO:journalctl@ceph.osd.10.smithi189.stdout:Mar 24 03:20:54 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-osd-10[61294]: 2024-03-24T03:20:54.973+0000 7f804e31f700 -1 osd.10 127 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-03-24T03:20:15.546606+0000 front 2024-03-24T03:20:15.546459+0000 (oldest deadline 2024-03-24T03:20:37.846574+0000) 2024-03-24T03:20:55.205 INFO:journalctl@ceph.osd.10.smithi189.stdout:Mar 24 03:20:54 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-osd-10[61294]: 2024-03-24T03:20:54.973+0000 7f804e31f700 -1 osd.10 127 heartbeat_check: no reply from 172.21.15.118:6814 osd.3 since back 2024-03-24T03:20:22.547814+0000 front 2024-03-24T03:20:22.547848+0000 (oldest deadline 2024-03-24T03:20:47.748188+0000) 2024-03-24T03:20:55.210 INFO:journalctl@ceph.osd.8.smithi193.stdout:Mar 24 03:20:54 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-osd-8[55794]: 2024-03-24T03:20:54.974+0000 7f6e8b995700 -1 osd.8 127 heartbeat_check: no reply from 172.21.15.189:6804 osd.1 since back 2024-03-24T03:20:06.219689+0000 front 2024-03-24T03:20:06.219789+0000 (oldest deadline 2024-03-24T03:20:30.919690+0000) 2024-03-24T03:20:55.210 INFO:journalctl@ceph.osd.8.smithi193.stdout:Mar 24 03:20:54 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-osd-8[55794]: 2024-03-24T03:20:54.974+0000 7f6e8b995700 -1 osd.8 127 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-03-24T03:20:16.121217+0000 front 2024-03-24T03:20:16.121198+0000 (oldest deadline 2024-03-24T03:20:40.221317+0000) 2024-03-24T03:20:55.211 INFO:journalctl@ceph.osd.8.smithi193.stdout:Mar 24 03:20:54 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-osd-8[55794]: 2024-03-24T03:20:54.974+0000 7f6e8b995700 -1 osd.8 127 heartbeat_check: no reply from 172.21.15.118:6814 osd.3 since back 2024-03-24T03:20:24.922540+0000 front 2024-03-24T03:20:24.922747+0000 (oldest deadline 2024-03-24T03:20:46.622438+0000) 2024-03-24T03:20:55.211 INFO:journalctl@ceph.osd.11.smithi193.stdout:Mar 24 03:20:54 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-osd-11[62452]: 2024-03-24T03:20:54.858+0000 7f0edd521700 -1 osd.11 127 heartbeat_check: no reply from 172.21.15.118:6806 osd.0 since back 2024-03-24T03:19:59.095638+0000 front 2024-03-24T03:19:59.095627+0000 (oldest deadline 2024-03-24T03:20:20.195530+0000) 2024-03-24T03:20:55.211 INFO:journalctl@ceph.osd.11.smithi193.stdout:Mar 24 03:20:54 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-osd-11[62452]: 2024-03-24T03:20:54.858+0000 7f0edd521700 -1 osd.11 127 heartbeat_check: no reply from 172.21.15.189:6804 osd.1 since back 2024-03-24T03:20:07.399753+0000 front 2024-03-24T03:20:07.399812+0000 (oldest deadline 2024-03-24T03:20:32.099923+0000) 2024-03-24T03:20:55.211 INFO:journalctl@ceph.osd.11.smithi193.stdout:Mar 24 03:20:54 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-osd-11[62452]: 2024-03-24T03:20:54.858+0000 7f0edd521700 -1 osd.11 127 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-03-24T03:20:12.100299+0000 front 2024-03-24T03:20:12.100348+0000 (oldest deadline 2024-03-24T03:20:38.000441+0000) 2024-03-24T03:20:55.211 INFO:journalctl@ceph.osd.11.smithi193.stdout:Mar 24 03:20:54 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-osd-11[62452]: 2024-03-24T03:20:54.858+0000 7f0edd521700 -1 osd.11 127 heartbeat_check: no reply from 172.21.15.118:6814 osd.3 since back 2024-03-24T03:20:25.002506+0000 front 2024-03-24T03:20:25.002576+0000 (oldest deadline 2024-03-24T03:20:46.702390+0000) 2024-03-24T03:20:55.696 INFO:journalctl@ceph.osd.9.smithi118.stdout:Mar 24 03:20:55 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-osd-9[67027]: 2024-03-24T03:20:55.437+0000 7f69f7400700 -1 osd.9 127 heartbeat_check: no reply from 172.21.15.118:6806 osd.0 since back 2024-03-24T03:19:57.002010+0000 front 2024-03-24T03:19:57.002333+0000 (oldest deadline 2024-03-24T03:20:21.101732+0000) 2024-03-24T03:20:55.696 INFO:journalctl@ceph.osd.9.smithi118.stdout:Mar 24 03:20:55 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-osd-9[67027]: 2024-03-24T03:20:55.437+0000 7f69f7400700 -1 osd.9 127 heartbeat_check: no reply from 172.21.15.189:6804 osd.1 since back 2024-03-24T03:20:05.804520+0000 front 2024-03-24T03:20:05.804379+0000 (oldest deadline 2024-03-24T03:20:29.303978+0000) 2024-03-24T03:20:55.696 INFO:journalctl@ceph.osd.9.smithi118.stdout:Mar 24 03:20:55 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-osd-9[67027]: 2024-03-24T03:20:55.437+0000 7f69f7400700 -1 osd.9 127 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-03-24T03:20:14.507274+0000 front 2024-03-24T03:20:14.507231+0000 (oldest deadline 2024-03-24T03:20:38.607488+0000) 2024-03-24T03:20:55.697 INFO:journalctl@ceph.osd.9.smithi118.stdout:Mar 24 03:20:55 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-osd-9[67027]: 2024-03-24T03:20:55.437+0000 7f69f7400700 -1 osd.9 127 heartbeat_check: no reply from 172.21.15.118:6814 osd.3 since back 2024-03-24T03:20:23.111195+0000 front 2024-03-24T03:20:23.111494+0000 (oldest deadline 2024-03-24T03:20:48.410762+0000) 2024-03-24T03:20:56.203 INFO:journalctl@ceph.osd.7.smithi189.stdout:Mar 24 03:20:55 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-osd-7[54385]: 2024-03-24T03:20:55.824+0000 7fb775b24700 -1 osd.7 127 heartbeat_check: no reply from 172.21.15.189:6804 osd.1 since back 2024-03-24T03:20:04.667909+0000 front 2024-03-24T03:20:04.667932+0000 (oldest deadline 2024-03-24T03:20:29.968062+0000) 2024-03-24T03:20:56.204 INFO:journalctl@ceph.osd.7.smithi189.stdout:Mar 24 03:20:55 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-osd-7[54385]: 2024-03-24T03:20:55.824+0000 7fb775b24700 -1 osd.7 127 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-03-24T03:20:15.769900+0000 front 2024-03-24T03:20:15.769860+0000 (oldest deadline 2024-03-24T03:20:37.470057+0000) 2024-03-24T03:20:56.204 INFO:journalctl@ceph.osd.7.smithi189.stdout:Mar 24 03:20:55 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-osd-7[54385]: 2024-03-24T03:20:55.824+0000 7fb775b24700 -1 osd.7 127 heartbeat_check: no reply from 172.21.15.118:6814 osd.3 since back 2024-03-24T03:20:24.372782+0000 front 2024-03-24T03:20:24.372820+0000 (oldest deadline 2024-03-24T03:20:46.072678+0000) 2024-03-24T03:20:56.204 INFO:journalctl@ceph.osd.10.smithi189.stdout:Mar 24 03:20:55 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-osd-10[61294]: 2024-03-24T03:20:55.969+0000 7f804e31f700 -1 osd.10 127 heartbeat_check: no reply from 172.21.15.118:6806 osd.0 since back 2024-03-24T03:19:56.942815+0000 front 2024-03-24T03:19:56.942951+0000 (oldest deadline 2024-03-24T03:20:22.842714+0000) 2024-03-24T03:20:56.204 INFO:journalctl@ceph.osd.10.smithi189.stdout:Mar 24 03:20:55 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-osd-10[61294]: 2024-03-24T03:20:55.969+0000 7f804e31f700 -1 osd.10 127 heartbeat_check: no reply from 172.21.15.189:6804 osd.1 since back 2024-03-24T03:20:07.543899+0000 front 2024-03-24T03:20:07.543880+0000 (oldest deadline 2024-03-24T03:20:30.944770+0000) 2024-03-24T03:20:56.204 INFO:journalctl@ceph.osd.10.smithi189.stdout:Mar 24 03:20:55 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-osd-10[61294]: 2024-03-24T03:20:55.969+0000 7f804e31f700 -1 osd.10 127 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-03-24T03:20:15.546606+0000 front 2024-03-24T03:20:15.546459+0000 (oldest deadline 2024-03-24T03:20:37.846574+0000) 2024-03-24T03:20:56.204 INFO:journalctl@ceph.osd.10.smithi189.stdout:Mar 24 03:20:55 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-osd-10[61294]: 2024-03-24T03:20:55.969+0000 7f804e31f700 -1 osd.10 127 heartbeat_check: no reply from 172.21.15.118:6814 osd.3 since back 2024-03-24T03:20:22.547814+0000 front 2024-03-24T03:20:22.547848+0000 (oldest deadline 2024-03-24T03:20:47.748188+0000) 2024-03-24T03:20:56.205 INFO:journalctl@ceph.osd.10.smithi189.stdout:Mar 24 03:20:55 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-osd-10[61294]: 2024-03-24T03:20:55.969+0000 7f804e31f700 -1 osd.10 127 heartbeat_check: no reply from 172.21.15.189:6812 osd.4 since back 2024-03-24T03:20:33.049108+0000 front 2024-03-24T03:20:33.049154+0000 (oldest deadline 2024-03-24T03:20:55.949290+0000) 2024-03-24T03:20:56.210 INFO:journalctl@ceph.osd.8.smithi193.stdout:Mar 24 03:20:55 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-osd-8[55794]: 2024-03-24T03:20:55.976+0000 7f6e8b995700 -1 osd.8 127 heartbeat_check: no reply from 172.21.15.189:6804 osd.1 since back 2024-03-24T03:20:06.219689+0000 front 2024-03-24T03:20:06.219789+0000 (oldest deadline 2024-03-24T03:20:30.919690+0000) 2024-03-24T03:20:56.210 INFO:journalctl@ceph.osd.8.smithi193.stdout:Mar 24 03:20:55 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-osd-8[55794]: 2024-03-24T03:20:55.976+0000 7f6e8b995700 -1 osd.8 127 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-03-24T03:20:16.121217+0000 front 2024-03-24T03:20:16.121198+0000 (oldest deadline 2024-03-24T03:20:40.221317+0000) 2024-03-24T03:20:56.210 INFO:journalctl@ceph.osd.8.smithi193.stdout:Mar 24 03:20:55 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-osd-8[55794]: 2024-03-24T03:20:55.976+0000 7f6e8b995700 -1 osd.8 127 heartbeat_check: no reply from 172.21.15.118:6814 osd.3 since back 2024-03-24T03:20:24.922540+0000 front 2024-03-24T03:20:24.922747+0000 (oldest deadline 2024-03-24T03:20:46.622438+0000) 2024-03-24T03:20:56.211 INFO:journalctl@ceph.osd.11.smithi193.stdout:Mar 24 03:20:55 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-osd-11[62452]: 2024-03-24T03:20:55.848+0000 7f0edd521700 -1 osd.11 127 heartbeat_check: no reply from 172.21.15.118:6806 osd.0 since back 2024-03-24T03:19:59.095638+0000 front 2024-03-24T03:19:59.095627+0000 (oldest deadline 2024-03-24T03:20:20.195530+0000) 2024-03-24T03:20:56.211 INFO:journalctl@ceph.osd.11.smithi193.stdout:Mar 24 03:20:55 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-osd-11[62452]: 2024-03-24T03:20:55.848+0000 7f0edd521700 -1 osd.11 127 heartbeat_check: no reply from 172.21.15.189:6804 osd.1 since back 2024-03-24T03:20:07.399753+0000 front 2024-03-24T03:20:07.399812+0000 (oldest deadline 2024-03-24T03:20:32.099923+0000) 2024-03-24T03:20:56.211 INFO:journalctl@ceph.osd.11.smithi193.stdout:Mar 24 03:20:55 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-osd-11[62452]: 2024-03-24T03:20:55.848+0000 7f0edd521700 -1 osd.11 127 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-03-24T03:20:12.100299+0000 front 2024-03-24T03:20:12.100348+0000 (oldest deadline 2024-03-24T03:20:38.000441+0000) 2024-03-24T03:20:56.211 INFO:journalctl@ceph.osd.11.smithi193.stdout:Mar 24 03:20:55 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-osd-11[62452]: 2024-03-24T03:20:55.848+0000 7f0edd521700 -1 osd.11 127 heartbeat_check: no reply from 172.21.15.118:6814 osd.3 since back 2024-03-24T03:20:25.002506+0000 front 2024-03-24T03:20:25.002576+0000 (oldest deadline 2024-03-24T03:20:46.702390+0000) 2024-03-24T03:20:56.211 INFO:journalctl@ceph.osd.11.smithi193.stdout:Mar 24 03:20:55 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-osd-11[62452]: 2024-03-24T03:20:55.848+0000 7f0edd521700 -1 osd.11 127 heartbeat_check: no reply from 172.21.15.189:6812 osd.4 since back 2024-03-24T03:20:32.905105+0000 front 2024-03-24T03:20:32.905067+0000 (oldest deadline 2024-03-24T03:20:55.204998+0000) 2024-03-24T03:20:56.946 INFO:journalctl@ceph.osd.9.smithi118.stdout:Mar 24 03:20:56 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-osd-9[67027]: 2024-03-24T03:20:56.467+0000 7f69f7400700 -1 osd.9 127 heartbeat_check: no reply from 172.21.15.118:6806 osd.0 since back 2024-03-24T03:19:57.002010+0000 front 2024-03-24T03:19:57.002333+0000 (oldest deadline 2024-03-24T03:20:21.101732+0000) 2024-03-24T03:20:56.946 INFO:journalctl@ceph.osd.9.smithi118.stdout:Mar 24 03:20:56 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-osd-9[67027]: 2024-03-24T03:20:56.467+0000 7f69f7400700 -1 osd.9 127 heartbeat_check: no reply from 172.21.15.189:6804 osd.1 since back 2024-03-24T03:20:05.804520+0000 front 2024-03-24T03:20:05.804379+0000 (oldest deadline 2024-03-24T03:20:29.303978+0000) 2024-03-24T03:20:56.946 INFO:journalctl@ceph.osd.9.smithi118.stdout:Mar 24 03:20:56 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-osd-9[67027]: 2024-03-24T03:20:56.467+0000 7f69f7400700 -1 osd.9 127 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-03-24T03:20:14.507274+0000 front 2024-03-24T03:20:14.507231+0000 (oldest deadline 2024-03-24T03:20:38.607488+0000) 2024-03-24T03:20:56.947 INFO:journalctl@ceph.osd.9.smithi118.stdout:Mar 24 03:20:56 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-osd-9[67027]: 2024-03-24T03:20:56.467+0000 7f69f7400700 -1 osd.9 127 heartbeat_check: no reply from 172.21.15.118:6814 osd.3 since back 2024-03-24T03:20:23.111195+0000 front 2024-03-24T03:20:23.111494+0000 (oldest deadline 2024-03-24T03:20:48.410762+0000) 2024-03-24T03:20:56.947 INFO:journalctl@ceph.osd.9.smithi118.stdout:Mar 24 03:20:56 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-osd-9[67027]: 2024-03-24T03:20:56.467+0000 7f69f7400700 -1 osd.9 127 heartbeat_check: no reply from 172.21.15.189:6812 osd.4 since back 2024-03-24T03:20:31.213661+0000 front 2024-03-24T03:20:31.213727+0000 (oldest deadline 2024-03-24T03:20:55.913869+0000) 2024-03-24T03:20:57.204 INFO:journalctl@ceph.osd.7.smithi189.stdout:Mar 24 03:20:56 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-osd-7[54385]: 2024-03-24T03:20:56.812+0000 7fb775b24700 -1 osd.7 127 heartbeat_check: no reply from 172.21.15.189:6804 osd.1 since back 2024-03-24T03:20:04.667909+0000 front 2024-03-24T03:20:04.667932+0000 (oldest deadline 2024-03-24T03:20:29.968062+0000) 2024-03-24T03:20:57.204 INFO:journalctl@ceph.osd.7.smithi189.stdout:Mar 24 03:20:56 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-osd-7[54385]: 2024-03-24T03:20:56.812+0000 7fb775b24700 -1 osd.7 127 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-03-24T03:20:15.769900+0000 front 2024-03-24T03:20:15.769860+0000 (oldest deadline 2024-03-24T03:20:37.470057+0000) 2024-03-24T03:20:57.204 INFO:journalctl@ceph.osd.7.smithi189.stdout:Mar 24 03:20:56 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-osd-7[54385]: 2024-03-24T03:20:56.812+0000 7fb775b24700 -1 osd.7 127 heartbeat_check: no reply from 172.21.15.118:6814 osd.3 since back 2024-03-24T03:20:24.372782+0000 front 2024-03-24T03:20:24.372820+0000 (oldest deadline 2024-03-24T03:20:46.072678+0000) 2024-03-24T03:20:57.204 INFO:journalctl@ceph.osd.10.smithi189.stdout:Mar 24 03:20:57 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-osd-10[61294]: 2024-03-24T03:20:57.012+0000 7f804e31f700 -1 osd.10 127 heartbeat_check: no reply from 172.21.15.118:6806 osd.0 since back 2024-03-24T03:19:56.942815+0000 front 2024-03-24T03:19:56.942951+0000 (oldest deadline 2024-03-24T03:20:22.842714+0000) 2024-03-24T03:20:57.205 INFO:journalctl@ceph.osd.10.smithi189.stdout:Mar 24 03:20:57 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-osd-10[61294]: 2024-03-24T03:20:57.012+0000 7f804e31f700 -1 osd.10 127 heartbeat_check: no reply from 172.21.15.189:6804 osd.1 since back 2024-03-24T03:20:07.543899+0000 front 2024-03-24T03:20:07.543880+0000 (oldest deadline 2024-03-24T03:20:30.944770+0000) 2024-03-24T03:20:57.205 INFO:journalctl@ceph.osd.10.smithi189.stdout:Mar 24 03:20:57 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-osd-10[61294]: 2024-03-24T03:20:57.012+0000 7f804e31f700 -1 osd.10 127 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-03-24T03:20:15.546606+0000 front 2024-03-24T03:20:15.546459+0000 (oldest deadline 2024-03-24T03:20:37.846574+0000) 2024-03-24T03:20:57.205 INFO:journalctl@ceph.osd.10.smithi189.stdout:Mar 24 03:20:57 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-osd-10[61294]: 2024-03-24T03:20:57.012+0000 7f804e31f700 -1 osd.10 127 heartbeat_check: no reply from 172.21.15.118:6814 osd.3 since back 2024-03-24T03:20:22.547814+0000 front 2024-03-24T03:20:22.547848+0000 (oldest deadline 2024-03-24T03:20:47.748188+0000) 2024-03-24T03:20:57.205 INFO:journalctl@ceph.osd.10.smithi189.stdout:Mar 24 03:20:57 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-osd-10[61294]: 2024-03-24T03:20:57.012+0000 7f804e31f700 -1 osd.10 127 heartbeat_check: no reply from 172.21.15.189:6812 osd.4 since back 2024-03-24T03:20:33.049108+0000 front 2024-03-24T03:20:33.049154+0000 (oldest deadline 2024-03-24T03:20:55.949290+0000) 2024-03-24T03:20:57.210 INFO:journalctl@ceph.osd.8.smithi193.stdout:Mar 24 03:20:56 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-osd-8[55794]: 2024-03-24T03:20:56.930+0000 7f6e8b995700 -1 osd.8 127 heartbeat_check: no reply from 172.21.15.189:6804 osd.1 since back 2024-03-24T03:20:06.219689+0000 front 2024-03-24T03:20:06.219789+0000 (oldest deadline 2024-03-24T03:20:30.919690+0000) 2024-03-24T03:20:57.211 INFO:journalctl@ceph.osd.8.smithi193.stdout:Mar 24 03:20:56 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-osd-8[55794]: 2024-03-24T03:20:56.930+0000 7f6e8b995700 -1 osd.8 127 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-03-24T03:20:16.121217+0000 front 2024-03-24T03:20:16.121198+0000 (oldest deadline 2024-03-24T03:20:40.221317+0000) 2024-03-24T03:20:57.211 INFO:journalctl@ceph.osd.8.smithi193.stdout:Mar 24 03:20:56 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-osd-8[55794]: 2024-03-24T03:20:56.930+0000 7f6e8b995700 -1 osd.8 127 heartbeat_check: no reply from 172.21.15.118:6814 osd.3 since back 2024-03-24T03:20:24.922540+0000 front 2024-03-24T03:20:24.922747+0000 (oldest deadline 2024-03-24T03:20:46.622438+0000) 2024-03-24T03:20:57.211 INFO:journalctl@ceph.osd.11.smithi193.stdout:Mar 24 03:20:56 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-osd-11[62452]: 2024-03-24T03:20:56.817+0000 7f0edd521700 -1 osd.11 127 heartbeat_check: no reply from 172.21.15.118:6806 osd.0 since back 2024-03-24T03:19:59.095638+0000 front 2024-03-24T03:19:59.095627+0000 (oldest deadline 2024-03-24T03:20:20.195530+0000) 2024-03-24T03:20:57.211 INFO:journalctl@ceph.osd.11.smithi193.stdout:Mar 24 03:20:56 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-osd-11[62452]: 2024-03-24T03:20:56.817+0000 7f0edd521700 -1 osd.11 127 heartbeat_check: no reply from 172.21.15.189:6804 osd.1 since back 2024-03-24T03:20:07.399753+0000 front 2024-03-24T03:20:07.399812+0000 (oldest deadline 2024-03-24T03:20:32.099923+0000) 2024-03-24T03:20:57.211 INFO:journalctl@ceph.osd.11.smithi193.stdout:Mar 24 03:20:56 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-osd-11[62452]: 2024-03-24T03:20:56.817+0000 7f0edd521700 -1 osd.11 127 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-03-24T03:20:12.100299+0000 front 2024-03-24T03:20:12.100348+0000 (oldest deadline 2024-03-24T03:20:38.000441+0000) 2024-03-24T03:20:57.211 INFO:journalctl@ceph.osd.11.smithi193.stdout:Mar 24 03:20:56 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-osd-11[62452]: 2024-03-24T03:20:56.817+0000 7f0edd521700 -1 osd.11 127 heartbeat_check: no reply from 172.21.15.118:6814 osd.3 since back 2024-03-24T03:20:25.002506+0000 front 2024-03-24T03:20:25.002576+0000 (oldest deadline 2024-03-24T03:20:46.702390+0000) 2024-03-24T03:20:57.211 INFO:journalctl@ceph.osd.11.smithi193.stdout:Mar 24 03:20:56 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-osd-11[62452]: 2024-03-24T03:20:56.817+0000 7f0edd521700 -1 osd.11 127 heartbeat_check: no reply from 172.21.15.189:6812 osd.4 since back 2024-03-24T03:20:32.905105+0000 front 2024-03-24T03:20:32.905067+0000 (oldest deadline 2024-03-24T03:20:55.204998+0000) 2024-03-24T03:20:57.946 INFO:journalctl@ceph.osd.9.smithi118.stdout:Mar 24 03:20:57 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-osd-9[67027]: 2024-03-24T03:20:57.501+0000 7f69f7400700 -1 osd.9 127 heartbeat_check: no reply from 172.21.15.118:6806 osd.0 since back 2024-03-24T03:19:57.002010+0000 front 2024-03-24T03:19:57.002333+0000 (oldest deadline 2024-03-24T03:20:21.101732+0000) 2024-03-24T03:20:57.946 INFO:journalctl@ceph.osd.9.smithi118.stdout:Mar 24 03:20:57 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-osd-9[67027]: 2024-03-24T03:20:57.501+0000 7f69f7400700 -1 osd.9 127 heartbeat_check: no reply from 172.21.15.189:6804 osd.1 since back 2024-03-24T03:20:05.804520+0000 front 2024-03-24T03:20:05.804379+0000 (oldest deadline 2024-03-24T03:20:29.303978+0000) 2024-03-24T03:20:57.946 INFO:journalctl@ceph.osd.9.smithi118.stdout:Mar 24 03:20:57 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-osd-9[67027]: 2024-03-24T03:20:57.501+0000 7f69f7400700 -1 osd.9 127 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-03-24T03:20:14.507274+0000 front 2024-03-24T03:20:14.507231+0000 (oldest deadline 2024-03-24T03:20:38.607488+0000) 2024-03-24T03:20:57.946 INFO:journalctl@ceph.osd.9.smithi118.stdout:Mar 24 03:20:57 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-osd-9[67027]: 2024-03-24T03:20:57.501+0000 7f69f7400700 -1 osd.9 127 heartbeat_check: no reply from 172.21.15.118:6814 osd.3 since back 2024-03-24T03:20:23.111195+0000 front 2024-03-24T03:20:23.111494+0000 (oldest deadline 2024-03-24T03:20:48.410762+0000) 2024-03-24T03:20:57.947 INFO:journalctl@ceph.osd.9.smithi118.stdout:Mar 24 03:20:57 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-osd-9[67027]: 2024-03-24T03:20:57.501+0000 7f69f7400700 -1 osd.9 127 heartbeat_check: no reply from 172.21.15.189:6812 osd.4 since back 2024-03-24T03:20:31.213661+0000 front 2024-03-24T03:20:31.213727+0000 (oldest deadline 2024-03-24T03:20:55.913869+0000) 2024-03-24T03:20:58.203 INFO:journalctl@ceph.osd.7.smithi189.stdout:Mar 24 03:20:57 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-osd-7[54385]: 2024-03-24T03:20:57.862+0000 7fb775b24700 -1 osd.7 127 heartbeat_check: no reply from 172.21.15.189:6804 osd.1 since back 2024-03-24T03:20:04.667909+0000 front 2024-03-24T03:20:04.667932+0000 (oldest deadline 2024-03-24T03:20:29.968062+0000) 2024-03-24T03:20:58.204 INFO:journalctl@ceph.osd.7.smithi189.stdout:Mar 24 03:20:57 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-osd-7[54385]: 2024-03-24T03:20:57.862+0000 7fb775b24700 -1 osd.7 127 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-03-24T03:20:15.769900+0000 front 2024-03-24T03:20:15.769860+0000 (oldest deadline 2024-03-24T03:20:37.470057+0000) 2024-03-24T03:20:58.204 INFO:journalctl@ceph.osd.7.smithi189.stdout:Mar 24 03:20:57 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-osd-7[54385]: 2024-03-24T03:20:57.862+0000 7fb775b24700 -1 osd.7 127 heartbeat_check: no reply from 172.21.15.118:6814 osd.3 since back 2024-03-24T03:20:24.372782+0000 front 2024-03-24T03:20:24.372820+0000 (oldest deadline 2024-03-24T03:20:46.072678+0000) 2024-03-24T03:20:58.204 INFO:journalctl@ceph.osd.10.smithi189.stdout:Mar 24 03:20:58 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-osd-10[61294]: 2024-03-24T03:20:58.049+0000 7f804e31f700 -1 osd.10 127 heartbeat_check: no reply from 172.21.15.118:6806 osd.0 since back 2024-03-24T03:19:56.942815+0000 front 2024-03-24T03:19:56.942951+0000 (oldest deadline 2024-03-24T03:20:22.842714+0000) 2024-03-24T03:20:58.204 INFO:journalctl@ceph.osd.10.smithi189.stdout:Mar 24 03:20:58 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-osd-10[61294]: 2024-03-24T03:20:58.049+0000 7f804e31f700 -1 osd.10 127 heartbeat_check: no reply from 172.21.15.189:6804 osd.1 since back 2024-03-24T03:20:07.543899+0000 front 2024-03-24T03:20:07.543880+0000 (oldest deadline 2024-03-24T03:20:30.944770+0000) 2024-03-24T03:20:58.204 INFO:journalctl@ceph.osd.10.smithi189.stdout:Mar 24 03:20:58 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-osd-10[61294]: 2024-03-24T03:20:58.049+0000 7f804e31f700 -1 osd.10 127 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-03-24T03:20:15.546606+0000 front 2024-03-24T03:20:15.546459+0000 (oldest deadline 2024-03-24T03:20:37.846574+0000) 2024-03-24T03:20:58.204 INFO:journalctl@ceph.osd.10.smithi189.stdout:Mar 24 03:20:58 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-osd-10[61294]: 2024-03-24T03:20:58.049+0000 7f804e31f700 -1 osd.10 127 heartbeat_check: no reply from 172.21.15.118:6814 osd.3 since back 2024-03-24T03:20:22.547814+0000 front 2024-03-24T03:20:22.547848+0000 (oldest deadline 2024-03-24T03:20:47.748188+0000) 2024-03-24T03:20:58.205 INFO:journalctl@ceph.osd.10.smithi189.stdout:Mar 24 03:20:58 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-osd-10[61294]: 2024-03-24T03:20:58.049+0000 7f804e31f700 -1 osd.10 127 heartbeat_check: no reply from 172.21.15.189:6812 osd.4 since back 2024-03-24T03:20:33.049108+0000 front 2024-03-24T03:20:33.049154+0000 (oldest deadline 2024-03-24T03:20:55.949290+0000) 2024-03-24T03:20:58.211 INFO:journalctl@ceph.osd.8.smithi193.stdout:Mar 24 03:20:57 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-osd-8[55794]: 2024-03-24T03:20:57.905+0000 7f6e8b995700 -1 osd.8 127 heartbeat_check: no reply from 172.21.15.189:6804 osd.1 since back 2024-03-24T03:20:06.219689+0000 front 2024-03-24T03:20:06.219789+0000 (oldest deadline 2024-03-24T03:20:30.919690+0000) 2024-03-24T03:20:58.211 INFO:journalctl@ceph.osd.8.smithi193.stdout:Mar 24 03:20:57 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-osd-8[55794]: 2024-03-24T03:20:57.905+0000 7f6e8b995700 -1 osd.8 127 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-03-24T03:20:16.121217+0000 front 2024-03-24T03:20:16.121198+0000 (oldest deadline 2024-03-24T03:20:40.221317+0000) 2024-03-24T03:20:58.211 INFO:journalctl@ceph.osd.8.smithi193.stdout:Mar 24 03:20:57 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-osd-8[55794]: 2024-03-24T03:20:57.905+0000 7f6e8b995700 -1 osd.8 127 heartbeat_check: no reply from 172.21.15.118:6814 osd.3 since back 2024-03-24T03:20:24.922540+0000 front 2024-03-24T03:20:24.922747+0000 (oldest deadline 2024-03-24T03:20:46.622438+0000) 2024-03-24T03:20:58.211 INFO:journalctl@ceph.osd.8.smithi193.stdout:Mar 24 03:20:57 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-osd-8[55794]: 2024-03-24T03:20:57.905+0000 7f6e8b995700 -1 osd.8 127 heartbeat_check: no reply from 172.21.15.189:6812 osd.4 since back 2024-03-24T03:20:32.523632+0000 front 2024-03-24T03:20:32.523692+0000 (oldest deadline 2024-03-24T03:20:57.223574+0000) 2024-03-24T03:20:58.211 INFO:journalctl@ceph.osd.11.smithi193.stdout:Mar 24 03:20:57 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-osd-11[62452]: 2024-03-24T03:20:57.847+0000 7f0edd521700 -1 osd.11 127 heartbeat_check: no reply from 172.21.15.118:6806 osd.0 since back 2024-03-24T03:19:59.095638+0000 front 2024-03-24T03:19:59.095627+0000 (oldest deadline 2024-03-24T03:20:20.195530+0000) 2024-03-24T03:20:58.211 INFO:journalctl@ceph.osd.11.smithi193.stdout:Mar 24 03:20:57 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-osd-11[62452]: 2024-03-24T03:20:57.847+0000 7f0edd521700 -1 osd.11 127 heartbeat_check: no reply from 172.21.15.189:6804 osd.1 since back 2024-03-24T03:20:07.399753+0000 front 2024-03-24T03:20:07.399812+0000 (oldest deadline 2024-03-24T03:20:32.099923+0000) 2024-03-24T03:20:58.211 INFO:journalctl@ceph.osd.11.smithi193.stdout:Mar 24 03:20:57 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-osd-11[62452]: 2024-03-24T03:20:57.847+0000 7f0edd521700 -1 osd.11 127 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-03-24T03:20:12.100299+0000 front 2024-03-24T03:20:12.100348+0000 (oldest deadline 2024-03-24T03:20:38.000441+0000) 2024-03-24T03:20:58.212 INFO:journalctl@ceph.osd.11.smithi193.stdout:Mar 24 03:20:57 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-osd-11[62452]: 2024-03-24T03:20:57.847+0000 7f0edd521700 -1 osd.11 127 heartbeat_check: no reply from 172.21.15.118:6814 osd.3 since back 2024-03-24T03:20:25.002506+0000 front 2024-03-24T03:20:25.002576+0000 (oldest deadline 2024-03-24T03:20:46.702390+0000) 2024-03-24T03:20:58.212 INFO:journalctl@ceph.osd.11.smithi193.stdout:Mar 24 03:20:57 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-osd-11[62452]: 2024-03-24T03:20:57.847+0000 7f0edd521700 -1 osd.11 127 heartbeat_check: no reply from 172.21.15.189:6812 osd.4 since back 2024-03-24T03:20:32.905105+0000 front 2024-03-24T03:20:32.905067+0000 (oldest deadline 2024-03-24T03:20:55.204998+0000) 2024-03-24T03:20:58.946 INFO:journalctl@ceph.osd.9.smithi118.stdout:Mar 24 03:20:58 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-osd-9[67027]: 2024-03-24T03:20:58.500+0000 7f69f7400700 -1 osd.9 127 heartbeat_check: no reply from 172.21.15.118:6806 osd.0 since back 2024-03-24T03:19:57.002010+0000 front 2024-03-24T03:19:57.002333+0000 (oldest deadline 2024-03-24T03:20:21.101732+0000) 2024-03-24T03:20:58.946 INFO:journalctl@ceph.osd.9.smithi118.stdout:Mar 24 03:20:58 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-osd-9[67027]: 2024-03-24T03:20:58.500+0000 7f69f7400700 -1 osd.9 127 heartbeat_check: no reply from 172.21.15.189:6804 osd.1 since back 2024-03-24T03:20:05.804520+0000 front 2024-03-24T03:20:05.804379+0000 (oldest deadline 2024-03-24T03:20:29.303978+0000) 2024-03-24T03:20:58.947 INFO:journalctl@ceph.osd.9.smithi118.stdout:Mar 24 03:20:58 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-osd-9[67027]: 2024-03-24T03:20:58.500+0000 7f69f7400700 -1 osd.9 127 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-03-24T03:20:14.507274+0000 front 2024-03-24T03:20:14.507231+0000 (oldest deadline 2024-03-24T03:20:38.607488+0000) 2024-03-24T03:20:58.947 INFO:journalctl@ceph.osd.9.smithi118.stdout:Mar 24 03:20:58 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-osd-9[67027]: 2024-03-24T03:20:58.500+0000 7f69f7400700 -1 osd.9 127 heartbeat_check: no reply from 172.21.15.118:6814 osd.3 since back 2024-03-24T03:20:23.111195+0000 front 2024-03-24T03:20:23.111494+0000 (oldest deadline 2024-03-24T03:20:48.410762+0000) 2024-03-24T03:20:58.947 INFO:journalctl@ceph.osd.9.smithi118.stdout:Mar 24 03:20:58 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-osd-9[67027]: 2024-03-24T03:20:58.500+0000 7f69f7400700 -1 osd.9 127 heartbeat_check: no reply from 172.21.15.189:6812 osd.4 since back 2024-03-24T03:20:31.213661+0000 front 2024-03-24T03:20:31.213727+0000 (oldest deadline 2024-03-24T03:20:55.913869+0000) 2024-03-24T03:20:59.204 INFO:journalctl@ceph.osd.7.smithi189.stdout:Mar 24 03:20:58 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-osd-7[54385]: 2024-03-24T03:20:58.828+0000 7fb775b24700 -1 osd.7 127 heartbeat_check: no reply from 172.21.15.189:6804 osd.1 since back 2024-03-24T03:20:04.667909+0000 front 2024-03-24T03:20:04.667932+0000 (oldest deadline 2024-03-24T03:20:29.968062+0000) 2024-03-24T03:20:59.204 INFO:journalctl@ceph.osd.7.smithi189.stdout:Mar 24 03:20:58 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-osd-7[54385]: 2024-03-24T03:20:58.828+0000 7fb775b24700 -1 osd.7 127 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-03-24T03:20:15.769900+0000 front 2024-03-24T03:20:15.769860+0000 (oldest deadline 2024-03-24T03:20:37.470057+0000) 2024-03-24T03:20:59.204 INFO:journalctl@ceph.osd.7.smithi189.stdout:Mar 24 03:20:58 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-osd-7[54385]: 2024-03-24T03:20:58.828+0000 7fb775b24700 -1 osd.7 127 heartbeat_check: no reply from 172.21.15.118:6814 osd.3 since back 2024-03-24T03:20:24.372782+0000 front 2024-03-24T03:20:24.372820+0000 (oldest deadline 2024-03-24T03:20:46.072678+0000) 2024-03-24T03:20:59.204 INFO:journalctl@ceph.osd.7.smithi189.stdout:Mar 24 03:20:58 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-osd-7[54385]: 2024-03-24T03:20:58.828+0000 7fb775b24700 -1 osd.7 127 heartbeat_check: no reply from 172.21.15.189:6812 osd.4 since back 2024-03-24T03:20:32.974921+0000 front 2024-03-24T03:20:32.974964+0000 (oldest deadline 2024-03-24T03:20:58.275096+0000) 2024-03-24T03:20:59.205 INFO:journalctl@ceph.osd.10.smithi189.stdout:Mar 24 03:20:59 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-osd-10[61294]: 2024-03-24T03:20:59.005+0000 7f804e31f700 -1 osd.10 127 heartbeat_check: no reply from 172.21.15.118:6806 osd.0 since back 2024-03-24T03:19:56.942815+0000 front 2024-03-24T03:19:56.942951+0000 (oldest deadline 2024-03-24T03:20:22.842714+0000) 2024-03-24T03:20:59.205 INFO:journalctl@ceph.osd.10.smithi189.stdout:Mar 24 03:20:59 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-osd-10[61294]: 2024-03-24T03:20:59.005+0000 7f804e31f700 -1 osd.10 127 heartbeat_check: no reply from 172.21.15.189:6804 osd.1 since back 2024-03-24T03:20:07.543899+0000 front 2024-03-24T03:20:07.543880+0000 (oldest deadline 2024-03-24T03:20:30.944770+0000) 2024-03-24T03:20:59.205 INFO:journalctl@ceph.osd.10.smithi189.stdout:Mar 24 03:20:59 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-osd-10[61294]: 2024-03-24T03:20:59.005+0000 7f804e31f700 -1 osd.10 127 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-03-24T03:20:15.546606+0000 front 2024-03-24T03:20:15.546459+0000 (oldest deadline 2024-03-24T03:20:37.846574+0000) 2024-03-24T03:20:59.205 INFO:journalctl@ceph.osd.10.smithi189.stdout:Mar 24 03:20:59 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-osd-10[61294]: 2024-03-24T03:20:59.005+0000 7f804e31f700 -1 osd.10 127 heartbeat_check: no reply from 172.21.15.118:6814 osd.3 since back 2024-03-24T03:20:22.547814+0000 front 2024-03-24T03:20:22.547848+0000 (oldest deadline 2024-03-24T03:20:47.748188+0000) 2024-03-24T03:20:59.205 INFO:journalctl@ceph.osd.10.smithi189.stdout:Mar 24 03:20:59 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-osd-10[61294]: 2024-03-24T03:20:59.005+0000 7f804e31f700 -1 osd.10 127 heartbeat_check: no reply from 172.21.15.189:6812 osd.4 since back 2024-03-24T03:20:33.049108+0000 front 2024-03-24T03:20:33.049154+0000 (oldest deadline 2024-03-24T03:20:55.949290+0000) 2024-03-24T03:20:59.211 INFO:journalctl@ceph.osd.8.smithi193.stdout:Mar 24 03:20:58 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-osd-8[55794]: 2024-03-24T03:20:58.902+0000 7f6e8b995700 -1 osd.8 127 heartbeat_check: no reply from 172.21.15.189:6804 osd.1 since back 2024-03-24T03:20:06.219689+0000 front 2024-03-24T03:20:06.219789+0000 (oldest deadline 2024-03-24T03:20:30.919690+0000) 2024-03-24T03:20:59.211 INFO:journalctl@ceph.osd.8.smithi193.stdout:Mar 24 03:20:58 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-osd-8[55794]: 2024-03-24T03:20:58.902+0000 7f6e8b995700 -1 osd.8 127 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-03-24T03:20:16.121217+0000 front 2024-03-24T03:20:16.121198+0000 (oldest deadline 2024-03-24T03:20:40.221317+0000) 2024-03-24T03:20:59.211 INFO:journalctl@ceph.osd.8.smithi193.stdout:Mar 24 03:20:58 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-osd-8[55794]: 2024-03-24T03:20:58.902+0000 7f6e8b995700 -1 osd.8 127 heartbeat_check: no reply from 172.21.15.118:6814 osd.3 since back 2024-03-24T03:20:24.922540+0000 front 2024-03-24T03:20:24.922747+0000 (oldest deadline 2024-03-24T03:20:46.622438+0000) 2024-03-24T03:20:59.211 INFO:journalctl@ceph.osd.8.smithi193.stdout:Mar 24 03:20:58 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-osd-8[55794]: 2024-03-24T03:20:58.902+0000 7f6e8b995700 -1 osd.8 127 heartbeat_check: no reply from 172.21.15.189:6812 osd.4 since back 2024-03-24T03:20:32.523632+0000 front 2024-03-24T03:20:32.523692+0000 (oldest deadline 2024-03-24T03:20:57.223574+0000) 2024-03-24T03:20:59.211 INFO:journalctl@ceph.osd.11.smithi193.stdout:Mar 24 03:20:58 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-osd-11[62452]: 2024-03-24T03:20:58.887+0000 7f0edd521700 -1 osd.11 127 heartbeat_check: no reply from 172.21.15.118:6806 osd.0 since back 2024-03-24T03:19:59.095638+0000 front 2024-03-24T03:19:59.095627+0000 (oldest deadline 2024-03-24T03:20:20.195530+0000) 2024-03-24T03:20:59.211 INFO:journalctl@ceph.osd.11.smithi193.stdout:Mar 24 03:20:58 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-osd-11[62452]: 2024-03-24T03:20:58.887+0000 7f0edd521700 -1 osd.11 127 heartbeat_check: no reply from 172.21.15.189:6804 osd.1 since back 2024-03-24T03:20:07.399753+0000 front 2024-03-24T03:20:07.399812+0000 (oldest deadline 2024-03-24T03:20:32.099923+0000) 2024-03-24T03:20:59.212 INFO:journalctl@ceph.osd.11.smithi193.stdout:Mar 24 03:20:58 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-osd-11[62452]: 2024-03-24T03:20:58.887+0000 7f0edd521700 -1 osd.11 127 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-03-24T03:20:12.100299+0000 front 2024-03-24T03:20:12.100348+0000 (oldest deadline 2024-03-24T03:20:38.000441+0000) 2024-03-24T03:20:59.212 INFO:journalctl@ceph.osd.11.smithi193.stdout:Mar 24 03:20:58 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-osd-11[62452]: 2024-03-24T03:20:58.887+0000 7f0edd521700 -1 osd.11 127 heartbeat_check: no reply from 172.21.15.118:6814 osd.3 since back 2024-03-24T03:20:25.002506+0000 front 2024-03-24T03:20:25.002576+0000 (oldest deadline 2024-03-24T03:20:46.702390+0000) 2024-03-24T03:20:59.212 INFO:journalctl@ceph.osd.11.smithi193.stdout:Mar 24 03:20:58 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-osd-11[62452]: 2024-03-24T03:20:58.887+0000 7f0edd521700 -1 osd.11 127 heartbeat_check: no reply from 172.21.15.189:6812 osd.4 since back 2024-03-24T03:20:32.905105+0000 front 2024-03-24T03:20:32.905067+0000 (oldest deadline 2024-03-24T03:20:55.204998+0000) 2024-03-24T03:20:59.946 INFO:journalctl@ceph.osd.9.smithi118.stdout:Mar 24 03:20:59 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-osd-9[67027]: 2024-03-24T03:20:59.548+0000 7f69f7400700 -1 osd.9 127 heartbeat_check: no reply from 172.21.15.118:6806 osd.0 since back 2024-03-24T03:19:57.002010+0000 front 2024-03-24T03:19:57.002333+0000 (oldest deadline 2024-03-24T03:20:21.101732+0000) 2024-03-24T03:20:59.946 INFO:journalctl@ceph.osd.9.smithi118.stdout:Mar 24 03:20:59 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-osd-9[67027]: 2024-03-24T03:20:59.548+0000 7f69f7400700 -1 osd.9 127 heartbeat_check: no reply from 172.21.15.189:6804 osd.1 since back 2024-03-24T03:20:05.804520+0000 front 2024-03-24T03:20:05.804379+0000 (oldest deadline 2024-03-24T03:20:29.303978+0000) 2024-03-24T03:20:59.946 INFO:journalctl@ceph.osd.9.smithi118.stdout:Mar 24 03:20:59 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-osd-9[67027]: 2024-03-24T03:20:59.548+0000 7f69f7400700 -1 osd.9 127 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-03-24T03:20:14.507274+0000 front 2024-03-24T03:20:14.507231+0000 (oldest deadline 2024-03-24T03:20:38.607488+0000) 2024-03-24T03:20:59.947 INFO:journalctl@ceph.osd.9.smithi118.stdout:Mar 24 03:20:59 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-osd-9[67027]: 2024-03-24T03:20:59.548+0000 7f69f7400700 -1 osd.9 127 heartbeat_check: no reply from 172.21.15.118:6814 osd.3 since back 2024-03-24T03:20:23.111195+0000 front 2024-03-24T03:20:23.111494+0000 (oldest deadline 2024-03-24T03:20:48.410762+0000) 2024-03-24T03:20:59.947 INFO:journalctl@ceph.osd.9.smithi118.stdout:Mar 24 03:20:59 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-osd-9[67027]: 2024-03-24T03:20:59.548+0000 7f69f7400700 -1 osd.9 127 heartbeat_check: no reply from 172.21.15.189:6812 osd.4 since back 2024-03-24T03:20:31.213661+0000 front 2024-03-24T03:20:31.213727+0000 (oldest deadline 2024-03-24T03:20:55.913869+0000) 2024-03-24T03:21:00.111 INFO:journalctl@ceph.osd.7.smithi189.stdout:Mar 24 03:20:59 smithi189 podman[117469]: 2024-03-24 03:20:59.822627282 +0000 UTC m=+5.539092808 container died 0a8b745009baeb0ec8909f1840ef37622352ed0108ef04b46d5cfcc1078ab373 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-d79a5420-e971-11ee-95cd-87774f69a715-osd-7, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, GIT_CLEAN=True, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240319, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=) 2024-03-24T03:21:00.111 INFO:journalctl@ceph.osd.10.smithi189.stdout:Mar 24 03:20:59 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-osd-10[61294]: 2024-03-24T03:20:59.979+0000 7f804e31f700 -1 osd.10 127 heartbeat_check: no reply from 172.21.15.118:6806 osd.0 since back 2024-03-24T03:19:56.942815+0000 front 2024-03-24T03:19:56.942951+0000 (oldest deadline 2024-03-24T03:20:22.842714+0000) 2024-03-24T03:21:00.111 INFO:journalctl@ceph.osd.10.smithi189.stdout:Mar 24 03:20:59 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-osd-10[61294]: 2024-03-24T03:20:59.979+0000 7f804e31f700 -1 osd.10 127 heartbeat_check: no reply from 172.21.15.189:6804 osd.1 since back 2024-03-24T03:20:07.543899+0000 front 2024-03-24T03:20:07.543880+0000 (oldest deadline 2024-03-24T03:20:30.944770+0000) 2024-03-24T03:21:00.111 INFO:journalctl@ceph.osd.10.smithi189.stdout:Mar 24 03:20:59 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-osd-10[61294]: 2024-03-24T03:20:59.979+0000 7f804e31f700 -1 osd.10 127 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-03-24T03:20:15.546606+0000 front 2024-03-24T03:20:15.546459+0000 (oldest deadline 2024-03-24T03:20:37.846574+0000) 2024-03-24T03:21:00.111 INFO:journalctl@ceph.osd.10.smithi189.stdout:Mar 24 03:20:59 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-osd-10[61294]: 2024-03-24T03:20:59.979+0000 7f804e31f700 -1 osd.10 127 heartbeat_check: no reply from 172.21.15.118:6814 osd.3 since back 2024-03-24T03:20:22.547814+0000 front 2024-03-24T03:20:22.547848+0000 (oldest deadline 2024-03-24T03:20:47.748188+0000) 2024-03-24T03:21:00.112 INFO:journalctl@ceph.osd.10.smithi189.stdout:Mar 24 03:20:59 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-osd-10[61294]: 2024-03-24T03:20:59.979+0000 7f804e31f700 -1 osd.10 127 heartbeat_check: no reply from 172.21.15.189:6812 osd.4 since back 2024-03-24T03:20:33.049108+0000 front 2024-03-24T03:20:33.049154+0000 (oldest deadline 2024-03-24T03:20:55.949290+0000) 2024-03-24T03:21:00.210 INFO:journalctl@ceph.osd.8.smithi193.stdout:Mar 24 03:20:59 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-osd-8[55794]: 2024-03-24T03:20:59.887+0000 7f6e8b995700 -1 osd.8 127 heartbeat_check: no reply from 172.21.15.189:6804 osd.1 since back 2024-03-24T03:20:06.219689+0000 front 2024-03-24T03:20:06.219789+0000 (oldest deadline 2024-03-24T03:20:30.919690+0000) 2024-03-24T03:21:00.211 INFO:journalctl@ceph.osd.8.smithi193.stdout:Mar 24 03:20:59 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-osd-8[55794]: 2024-03-24T03:20:59.887+0000 7f6e8b995700 -1 osd.8 127 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-03-24T03:20:16.121217+0000 front 2024-03-24T03:20:16.121198+0000 (oldest deadline 2024-03-24T03:20:40.221317+0000) 2024-03-24T03:21:00.211 INFO:journalctl@ceph.osd.8.smithi193.stdout:Mar 24 03:20:59 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-osd-8[55794]: 2024-03-24T03:20:59.887+0000 7f6e8b995700 -1 osd.8 127 heartbeat_check: no reply from 172.21.15.118:6814 osd.3 since back 2024-03-24T03:20:24.922540+0000 front 2024-03-24T03:20:24.922747+0000 (oldest deadline 2024-03-24T03:20:46.622438+0000) 2024-03-24T03:21:00.211 INFO:journalctl@ceph.osd.8.smithi193.stdout:Mar 24 03:20:59 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-osd-8[55794]: 2024-03-24T03:20:59.887+0000 7f6e8b995700 -1 osd.8 127 heartbeat_check: no reply from 172.21.15.189:6812 osd.4 since back 2024-03-24T03:20:32.523632+0000 front 2024-03-24T03:20:32.523692+0000 (oldest deadline 2024-03-24T03:20:57.223574+0000) 2024-03-24T03:21:00.211 INFO:journalctl@ceph.osd.11.smithi193.stdout:Mar 24 03:20:59 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-osd-11[62452]: 2024-03-24T03:20:59.909+0000 7f0edd521700 -1 osd.11 127 heartbeat_check: no reply from 172.21.15.118:6806 osd.0 since back 2024-03-24T03:19:59.095638+0000 front 2024-03-24T03:19:59.095627+0000 (oldest deadline 2024-03-24T03:20:20.195530+0000) 2024-03-24T03:21:00.211 INFO:journalctl@ceph.osd.11.smithi193.stdout:Mar 24 03:20:59 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-osd-11[62452]: 2024-03-24T03:20:59.909+0000 7f0edd521700 -1 osd.11 127 heartbeat_check: no reply from 172.21.15.189:6804 osd.1 since back 2024-03-24T03:20:07.399753+0000 front 2024-03-24T03:20:07.399812+0000 (oldest deadline 2024-03-24T03:20:32.099923+0000) 2024-03-24T03:21:00.211 INFO:journalctl@ceph.osd.11.smithi193.stdout:Mar 24 03:20:59 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-osd-11[62452]: 2024-03-24T03:20:59.909+0000 7f0edd521700 -1 osd.11 127 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-03-24T03:20:12.100299+0000 front 2024-03-24T03:20:12.100348+0000 (oldest deadline 2024-03-24T03:20:38.000441+0000) 2024-03-24T03:21:00.212 INFO:journalctl@ceph.osd.11.smithi193.stdout:Mar 24 03:20:59 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-osd-11[62452]: 2024-03-24T03:20:59.909+0000 7f0edd521700 -1 osd.11 127 heartbeat_check: no reply from 172.21.15.118:6814 osd.3 since back 2024-03-24T03:20:25.002506+0000 front 2024-03-24T03:20:25.002576+0000 (oldest deadline 2024-03-24T03:20:46.702390+0000) 2024-03-24T03:21:00.212 INFO:journalctl@ceph.osd.11.smithi193.stdout:Mar 24 03:20:59 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-osd-11[62452]: 2024-03-24T03:20:59.909+0000 7f0edd521700 -1 osd.11 127 heartbeat_check: no reply from 172.21.15.189:6812 osd.4 since back 2024-03-24T03:20:32.905105+0000 front 2024-03-24T03:20:32.905067+0000 (oldest deadline 2024-03-24T03:20:55.204998+0000) 2024-03-24T03:21:00.453 INFO:journalctl@ceph.osd.7.smithi189.stdout:Mar 24 03:21:00 smithi189 podman[117469]: 2024-03-24 03:21:00.219099296 +0000 UTC m=+5.935564850 container cleanup 0a8b745009baeb0ec8909f1840ef37622352ed0108ef04b46d5cfcc1078ab373 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-d79a5420-e971-11ee-95cd-87774f69a715-osd-7, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, CEPH_POINT_RELEASE=, GIT_CLEAN=True, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240319, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS) 2024-03-24T03:21:00.454 INFO:journalctl@ceph.osd.7.smithi189.stdout:Mar 24 03:21:00 smithi189 bash[117469]: ceph-d79a5420-e971-11ee-95cd-87774f69a715-osd-7 2024-03-24T03:21:00.821 INFO:journalctl@ceph.osd.7.smithi189.stdout:Mar 24 03:21:00 smithi189 podman[117484]: 2024-03-24 03:21:00.478656038 +0000 UTC m=+0.658967362 container remove 0a8b745009baeb0ec8909f1840ef37622352ed0108ef04b46d5cfcc1078ab373 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-d79a5420-e971-11ee-95cd-87774f69a715-osd-7, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, org.label-schema.build-date=20240319, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=) 2024-03-24T03:21:00.946 INFO:journalctl@ceph.osd.9.smithi118.stdout:Mar 24 03:21:00 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-osd-9[67027]: 2024-03-24T03:21:00.544+0000 7f69f7400700 -1 osd.9 127 heartbeat_check: no reply from 172.21.15.118:6806 osd.0 since back 2024-03-24T03:19:57.002010+0000 front 2024-03-24T03:19:57.002333+0000 (oldest deadline 2024-03-24T03:20:21.101732+0000) 2024-03-24T03:21:00.946 INFO:journalctl@ceph.osd.9.smithi118.stdout:Mar 24 03:21:00 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-osd-9[67027]: 2024-03-24T03:21:00.544+0000 7f69f7400700 -1 osd.9 127 heartbeat_check: no reply from 172.21.15.189:6804 osd.1 since back 2024-03-24T03:20:05.804520+0000 front 2024-03-24T03:20:05.804379+0000 (oldest deadline 2024-03-24T03:20:29.303978+0000) 2024-03-24T03:21:00.946 INFO:journalctl@ceph.osd.9.smithi118.stdout:Mar 24 03:21:00 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-osd-9[67027]: 2024-03-24T03:21:00.544+0000 7f69f7400700 -1 osd.9 127 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-03-24T03:20:14.507274+0000 front 2024-03-24T03:20:14.507231+0000 (oldest deadline 2024-03-24T03:20:38.607488+0000) 2024-03-24T03:21:00.946 INFO:journalctl@ceph.osd.9.smithi118.stdout:Mar 24 03:21:00 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-osd-9[67027]: 2024-03-24T03:21:00.544+0000 7f69f7400700 -1 osd.9 127 heartbeat_check: no reply from 172.21.15.118:6814 osd.3 since back 2024-03-24T03:20:23.111195+0000 front 2024-03-24T03:20:23.111494+0000 (oldest deadline 2024-03-24T03:20:48.410762+0000) 2024-03-24T03:21:00.947 INFO:journalctl@ceph.osd.9.smithi118.stdout:Mar 24 03:21:00 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-osd-9[67027]: 2024-03-24T03:21:00.544+0000 7f69f7400700 -1 osd.9 127 heartbeat_check: no reply from 172.21.15.189:6812 osd.4 since back 2024-03-24T03:20:31.213661+0000 front 2024-03-24T03:20:31.213727+0000 (oldest deadline 2024-03-24T03:20:55.913869+0000) 2024-03-24T03:21:01.139 INFO:journalctl@ceph.osd.10.smithi189.stdout:Mar 24 03:21:00 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-osd-10[61294]: 2024-03-24T03:21:00.963+0000 7f804e31f700 -1 osd.10 127 heartbeat_check: no reply from 172.21.15.118:6806 osd.0 since back 2024-03-24T03:19:56.942815+0000 front 2024-03-24T03:19:56.942951+0000 (oldest deadline 2024-03-24T03:20:22.842714+0000) 2024-03-24T03:21:01.139 INFO:journalctl@ceph.osd.10.smithi189.stdout:Mar 24 03:21:00 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-osd-10[61294]: 2024-03-24T03:21:00.963+0000 7f804e31f700 -1 osd.10 127 heartbeat_check: no reply from 172.21.15.189:6804 osd.1 since back 2024-03-24T03:20:07.543899+0000 front 2024-03-24T03:20:07.543880+0000 (oldest deadline 2024-03-24T03:20:30.944770+0000) 2024-03-24T03:21:01.140 INFO:journalctl@ceph.osd.10.smithi189.stdout:Mar 24 03:21:00 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-osd-10[61294]: 2024-03-24T03:21:00.963+0000 7f804e31f700 -1 osd.10 127 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-03-24T03:20:15.546606+0000 front 2024-03-24T03:20:15.546459+0000 (oldest deadline 2024-03-24T03:20:37.846574+0000) 2024-03-24T03:21:01.140 INFO:journalctl@ceph.osd.10.smithi189.stdout:Mar 24 03:21:00 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-osd-10[61294]: 2024-03-24T03:21:00.963+0000 7f804e31f700 -1 osd.10 127 heartbeat_check: no reply from 172.21.15.118:6814 osd.3 since back 2024-03-24T03:20:22.547814+0000 front 2024-03-24T03:20:22.547848+0000 (oldest deadline 2024-03-24T03:20:47.748188+0000) 2024-03-24T03:21:01.140 INFO:journalctl@ceph.osd.10.smithi189.stdout:Mar 24 03:21:00 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-osd-10[61294]: 2024-03-24T03:21:00.963+0000 7f804e31f700 -1 osd.10 127 heartbeat_check: no reply from 172.21.15.189:6812 osd.4 since back 2024-03-24T03:20:33.049108+0000 front 2024-03-24T03:20:33.049154+0000 (oldest deadline 2024-03-24T03:20:55.949290+0000) 2024-03-24T03:21:01.140 INFO:journalctl@ceph.osd.7.smithi189.stdout:Mar 24 03:21:00 smithi189 podman[117576]: 2024-03-24 03:21:00.821498316 +0000 UTC m=+0.103632161 container create 899aecb9af811d604e1d7431e81ba9d79181b90453fa49dedbe351289558457e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-d79a5420-e971-11ee-95cd-87774f69a715-osd-7-deactivate, GIT_BRANCH=HEAD, org.label-schema.build-date=20240319, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, GIT_CLEAN=True, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.vendor=CentOS) 2024-03-24T03:21:01.140 INFO:journalctl@ceph.osd.7.smithi189.stdout:Mar 24 03:21:00 smithi189 podman[117576]: 2024-03-24 03:21:00.735886034 +0000 UTC m=+0.018019833 image pull c664afe56bb01db4f9afa9532c20598e201e664f2e9ba2be50db0d52674a26d2 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc 2024-03-24T03:21:01.140 INFO:journalctl@ceph.osd.7.smithi189.stdout:Mar 24 03:21:01 smithi189 podman[117576]: 2024-03-24 03:21:01.048990264 +0000 UTC m=+0.331124074 container init 899aecb9af811d604e1d7431e81ba9d79181b90453fa49dedbe351289558457e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-d79a5420-e971-11ee-95cd-87774f69a715-osd-7-deactivate, ceph=True, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.build-date=20240319, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30) 2024-03-24T03:21:01.140 INFO:journalctl@ceph.osd.7.smithi189.stdout:Mar 24 03:21:01 smithi189 podman[117576]: 2024-03-24 03:21:01.052189079 +0000 UTC m=+0.334322872 container start 899aecb9af811d604e1d7431e81ba9d79181b90453fa49dedbe351289558457e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-d79a5420-e971-11ee-95cd-87774f69a715-osd-7-deactivate, GIT_BRANCH=HEAD, org.label-schema.build-date=20240319, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, ceph=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_CLEAN=True, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-24T03:21:01.141 INFO:journalctl@ceph.osd.7.smithi189.stdout:Mar 24 03:21:01 smithi189 podman[117576]: 2024-03-24 03:21:01.140477112 +0000 UTC m=+0.422610913 container attach 899aecb9af811d604e1d7431e81ba9d79181b90453fa49dedbe351289558457e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-d79a5420-e971-11ee-95cd-87774f69a715-osd-7-deactivate, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_CLEAN=True, org.label-schema.build-date=20240319, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, ceph=True, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c) 2024-03-24T03:21:01.210 INFO:journalctl@ceph.osd.8.smithi193.stdout:Mar 24 03:21:00 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-osd-8[55794]: 2024-03-24T03:21:00.862+0000 7f6e8b995700 -1 osd.8 127 heartbeat_check: no reply from 172.21.15.189:6804 osd.1 since back 2024-03-24T03:20:06.219689+0000 front 2024-03-24T03:20:06.219789+0000 (oldest deadline 2024-03-24T03:20:30.919690+0000) 2024-03-24T03:21:01.210 INFO:journalctl@ceph.osd.8.smithi193.stdout:Mar 24 03:21:00 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-osd-8[55794]: 2024-03-24T03:21:00.862+0000 7f6e8b995700 -1 osd.8 127 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-03-24T03:20:16.121217+0000 front 2024-03-24T03:20:16.121198+0000 (oldest deadline 2024-03-24T03:20:40.221317+0000) 2024-03-24T03:21:01.210 INFO:journalctl@ceph.osd.8.smithi193.stdout:Mar 24 03:21:00 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-osd-8[55794]: 2024-03-24T03:21:00.862+0000 7f6e8b995700 -1 osd.8 127 heartbeat_check: no reply from 172.21.15.118:6814 osd.3 since back 2024-03-24T03:20:24.922540+0000 front 2024-03-24T03:20:24.922747+0000 (oldest deadline 2024-03-24T03:20:46.622438+0000) 2024-03-24T03:21:01.211 INFO:journalctl@ceph.osd.8.smithi193.stdout:Mar 24 03:21:00 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-osd-8[55794]: 2024-03-24T03:21:00.862+0000 7f6e8b995700 -1 osd.8 127 heartbeat_check: no reply from 172.21.15.189:6812 osd.4 since back 2024-03-24T03:20:32.523632+0000 front 2024-03-24T03:20:32.523692+0000 (oldest deadline 2024-03-24T03:20:57.223574+0000) 2024-03-24T03:21:01.211 INFO:journalctl@ceph.osd.11.smithi193.stdout:Mar 24 03:21:00 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-osd-11[62452]: 2024-03-24T03:21:00.912+0000 7f0edd521700 -1 osd.11 127 heartbeat_check: no reply from 172.21.15.118:6806 osd.0 since back 2024-03-24T03:19:59.095638+0000 front 2024-03-24T03:19:59.095627+0000 (oldest deadline 2024-03-24T03:20:20.195530+0000) 2024-03-24T03:21:01.211 INFO:journalctl@ceph.osd.11.smithi193.stdout:Mar 24 03:21:00 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-osd-11[62452]: 2024-03-24T03:21:00.912+0000 7f0edd521700 -1 osd.11 127 heartbeat_check: no reply from 172.21.15.189:6804 osd.1 since back 2024-03-24T03:20:07.399753+0000 front 2024-03-24T03:20:07.399812+0000 (oldest deadline 2024-03-24T03:20:32.099923+0000) 2024-03-24T03:21:01.211 INFO:journalctl@ceph.osd.11.smithi193.stdout:Mar 24 03:21:00 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-osd-11[62452]: 2024-03-24T03:21:00.912+0000 7f0edd521700 -1 osd.11 127 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-03-24T03:20:12.100299+0000 front 2024-03-24T03:20:12.100348+0000 (oldest deadline 2024-03-24T03:20:38.000441+0000) 2024-03-24T03:21:01.211 INFO:journalctl@ceph.osd.11.smithi193.stdout:Mar 24 03:21:00 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-osd-11[62452]: 2024-03-24T03:21:00.912+0000 7f0edd521700 -1 osd.11 127 heartbeat_check: no reply from 172.21.15.118:6814 osd.3 since back 2024-03-24T03:20:25.002506+0000 front 2024-03-24T03:20:25.002576+0000 (oldest deadline 2024-03-24T03:20:46.702390+0000) 2024-03-24T03:21:01.211 INFO:journalctl@ceph.osd.11.smithi193.stdout:Mar 24 03:21:00 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-osd-11[62452]: 2024-03-24T03:21:00.912+0000 7f0edd521700 -1 osd.11 127 heartbeat_check: no reply from 172.21.15.189:6812 osd.4 since back 2024-03-24T03:20:32.905105+0000 front 2024-03-24T03:20:32.905067+0000 (oldest deadline 2024-03-24T03:20:55.204998+0000) 2024-03-24T03:21:01.454 INFO:journalctl@ceph.osd.7.smithi189.stdout:Mar 24 03:21:01 smithi189 podman[117576]: 2024-03-24 03:21:01.334774934 +0000 UTC m=+0.616908738 container died 899aecb9af811d604e1d7431e81ba9d79181b90453fa49dedbe351289558457e (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-d79a5420-e971-11ee-95cd-87774f69a715-osd-7-deactivate, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240319, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_CLEAN=True, org.label-schema.schema-version=1.0) 2024-03-24T03:21:01.945 DEBUG:teuthology.orchestra.run.smithi189:> sudo pkill -f 'journalctl -f -n 0 -u ceph-d79a5420-e971-11ee-95cd-87774f69a715@osd.7.service' 2024-03-24T03:21:01.946 INFO:journalctl@ceph.osd.9.smithi118.stdout:Mar 24 03:21:01 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-osd-9[67027]: 2024-03-24T03:21:01.569+0000 7f69f7400700 -1 osd.9 127 heartbeat_check: no reply from 172.21.15.118:6806 osd.0 since back 2024-03-24T03:19:57.002010+0000 front 2024-03-24T03:19:57.002333+0000 (oldest deadline 2024-03-24T03:20:21.101732+0000) 2024-03-24T03:21:01.946 INFO:journalctl@ceph.osd.9.smithi118.stdout:Mar 24 03:21:01 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-osd-9[67027]: 2024-03-24T03:21:01.569+0000 7f69f7400700 -1 osd.9 127 heartbeat_check: no reply from 172.21.15.189:6804 osd.1 since back 2024-03-24T03:20:05.804520+0000 front 2024-03-24T03:20:05.804379+0000 (oldest deadline 2024-03-24T03:20:29.303978+0000) 2024-03-24T03:21:01.946 INFO:journalctl@ceph.osd.9.smithi118.stdout:Mar 24 03:21:01 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-osd-9[67027]: 2024-03-24T03:21:01.569+0000 7f69f7400700 -1 osd.9 127 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-03-24T03:20:14.507274+0000 front 2024-03-24T03:20:14.507231+0000 (oldest deadline 2024-03-24T03:20:38.607488+0000) 2024-03-24T03:21:01.946 INFO:journalctl@ceph.osd.9.smithi118.stdout:Mar 24 03:21:01 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-osd-9[67027]: 2024-03-24T03:21:01.569+0000 7f69f7400700 -1 osd.9 127 heartbeat_check: no reply from 172.21.15.118:6814 osd.3 since back 2024-03-24T03:20:23.111195+0000 front 2024-03-24T03:20:23.111494+0000 (oldest deadline 2024-03-24T03:20:48.410762+0000) 2024-03-24T03:21:01.946 INFO:journalctl@ceph.osd.9.smithi118.stdout:Mar 24 03:21:01 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-osd-9[67027]: 2024-03-24T03:21:01.569+0000 7f69f7400700 -1 osd.9 127 heartbeat_check: no reply from 172.21.15.189:6812 osd.4 since back 2024-03-24T03:20:31.213661+0000 front 2024-03-24T03:20:31.213727+0000 (oldest deadline 2024-03-24T03:20:55.913869+0000) 2024-03-24T03:21:02.039 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-24T03:21:02.040 INFO:tasks.cephadm.osd.7:Stopped osd.7 2024-03-24T03:21:02.040 INFO:tasks.cephadm.osd.8:Stopping osd.8... 2024-03-24T03:21:02.040 DEBUG:teuthology.orchestra.run.smithi193:> sudo systemctl stop ceph-d79a5420-e971-11ee-95cd-87774f69a715@osd.8 2024-03-24T03:21:02.203 INFO:journalctl@ceph.osd.10.smithi189.stdout:Mar 24 03:21:02 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-osd-10[61294]: 2024-03-24T03:21:02.005+0000 7f804e31f700 -1 osd.10 127 heartbeat_check: no reply from 172.21.15.118:6806 osd.0 since back 2024-03-24T03:19:56.942815+0000 front 2024-03-24T03:19:56.942951+0000 (oldest deadline 2024-03-24T03:20:22.842714+0000) 2024-03-24T03:21:02.204 INFO:journalctl@ceph.osd.10.smithi189.stdout:Mar 24 03:21:02 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-osd-10[61294]: 2024-03-24T03:21:02.005+0000 7f804e31f700 -1 osd.10 127 heartbeat_check: no reply from 172.21.15.189:6804 osd.1 since back 2024-03-24T03:20:07.543899+0000 front 2024-03-24T03:20:07.543880+0000 (oldest deadline 2024-03-24T03:20:30.944770+0000) 2024-03-24T03:21:02.204 INFO:journalctl@ceph.osd.10.smithi189.stdout:Mar 24 03:21:02 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-osd-10[61294]: 2024-03-24T03:21:02.005+0000 7f804e31f700 -1 osd.10 127 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-03-24T03:20:15.546606+0000 front 2024-03-24T03:20:15.546459+0000 (oldest deadline 2024-03-24T03:20:37.846574+0000) 2024-03-24T03:21:02.204 INFO:journalctl@ceph.osd.10.smithi189.stdout:Mar 24 03:21:02 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-osd-10[61294]: 2024-03-24T03:21:02.005+0000 7f804e31f700 -1 osd.10 127 heartbeat_check: no reply from 172.21.15.118:6814 osd.3 since back 2024-03-24T03:20:22.547814+0000 front 2024-03-24T03:20:22.547848+0000 (oldest deadline 2024-03-24T03:20:47.748188+0000) 2024-03-24T03:21:02.204 INFO:journalctl@ceph.osd.10.smithi189.stdout:Mar 24 03:21:02 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-osd-10[61294]: 2024-03-24T03:21:02.005+0000 7f804e31f700 -1 osd.10 127 heartbeat_check: no reply from 172.21.15.189:6812 osd.4 since back 2024-03-24T03:20:33.049108+0000 front 2024-03-24T03:20:33.049154+0000 (oldest deadline 2024-03-24T03:20:55.949290+0000) 2024-03-24T03:21:02.210 INFO:journalctl@ceph.osd.8.smithi193.stdout:Mar 24 03:21:01 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-osd-8[55794]: 2024-03-24T03:21:01.903+0000 7f6e8b995700 -1 osd.8 127 heartbeat_check: no reply from 172.21.15.189:6804 osd.1 since back 2024-03-24T03:20:06.219689+0000 front 2024-03-24T03:20:06.219789+0000 (oldest deadline 2024-03-24T03:20:30.919690+0000) 2024-03-24T03:21:02.211 INFO:journalctl@ceph.osd.8.smithi193.stdout:Mar 24 03:21:01 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-osd-8[55794]: 2024-03-24T03:21:01.903+0000 7f6e8b995700 -1 osd.8 127 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-03-24T03:20:16.121217+0000 front 2024-03-24T03:20:16.121198+0000 (oldest deadline 2024-03-24T03:20:40.221317+0000) 2024-03-24T03:21:02.211 INFO:journalctl@ceph.osd.8.smithi193.stdout:Mar 24 03:21:01 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-osd-8[55794]: 2024-03-24T03:21:01.903+0000 7f6e8b995700 -1 osd.8 127 heartbeat_check: no reply from 172.21.15.118:6814 osd.3 since back 2024-03-24T03:20:24.922540+0000 front 2024-03-24T03:20:24.922747+0000 (oldest deadline 2024-03-24T03:20:46.622438+0000) 2024-03-24T03:21:02.211 INFO:journalctl@ceph.osd.8.smithi193.stdout:Mar 24 03:21:01 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-osd-8[55794]: 2024-03-24T03:21:01.903+0000 7f6e8b995700 -1 osd.8 127 heartbeat_check: no reply from 172.21.15.189:6812 osd.4 since back 2024-03-24T03:20:32.523632+0000 front 2024-03-24T03:20:32.523692+0000 (oldest deadline 2024-03-24T03:20:57.223574+0000) 2024-03-24T03:21:02.211 INFO:journalctl@ceph.osd.8.smithi193.stdout:Mar 24 03:21:02 smithi193 systemd[1]: Stopping Ceph osd.8 for d79a5420-e971-11ee-95cd-87774f69a715... 2024-03-24T03:21:02.211 INFO:journalctl@ceph.osd.11.smithi193.stdout:Mar 24 03:21:01 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-osd-11[62452]: 2024-03-24T03:21:01.933+0000 7f0edd521700 -1 osd.11 127 heartbeat_check: no reply from 172.21.15.118:6806 osd.0 since back 2024-03-24T03:19:59.095638+0000 front 2024-03-24T03:19:59.095627+0000 (oldest deadline 2024-03-24T03:20:20.195530+0000) 2024-03-24T03:21:02.211 INFO:journalctl@ceph.osd.11.smithi193.stdout:Mar 24 03:21:01 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-osd-11[62452]: 2024-03-24T03:21:01.933+0000 7f0edd521700 -1 osd.11 127 heartbeat_check: no reply from 172.21.15.189:6804 osd.1 since back 2024-03-24T03:20:07.399753+0000 front 2024-03-24T03:20:07.399812+0000 (oldest deadline 2024-03-24T03:20:32.099923+0000) 2024-03-24T03:21:02.211 INFO:journalctl@ceph.osd.11.smithi193.stdout:Mar 24 03:21:01 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-osd-11[62452]: 2024-03-24T03:21:01.933+0000 7f0edd521700 -1 osd.11 127 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-03-24T03:20:12.100299+0000 front 2024-03-24T03:20:12.100348+0000 (oldest deadline 2024-03-24T03:20:38.000441+0000) 2024-03-24T03:21:02.212 INFO:journalctl@ceph.osd.11.smithi193.stdout:Mar 24 03:21:01 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-osd-11[62452]: 2024-03-24T03:21:01.933+0000 7f0edd521700 -1 osd.11 127 heartbeat_check: no reply from 172.21.15.118:6814 osd.3 since back 2024-03-24T03:20:25.002506+0000 front 2024-03-24T03:20:25.002576+0000 (oldest deadline 2024-03-24T03:20:46.702390+0000) 2024-03-24T03:21:02.212 INFO:journalctl@ceph.osd.11.smithi193.stdout:Mar 24 03:21:01 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-osd-11[62452]: 2024-03-24T03:21:01.933+0000 7f0edd521700 -1 osd.11 127 heartbeat_check: no reply from 172.21.15.189:6812 osd.4 since back 2024-03-24T03:20:32.905105+0000 front 2024-03-24T03:20:32.905067+0000 (oldest deadline 2024-03-24T03:20:55.204998+0000) 2024-03-24T03:21:02.710 INFO:journalctl@ceph.osd.8.smithi193.stdout:Mar 24 03:21:02 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-osd-8[55794]: 2024-03-24T03:21:02.328+0000 7f6e933a9700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.8 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-24T03:21:02.711 INFO:journalctl@ceph.osd.8.smithi193.stdout:Mar 24 03:21:02 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-osd-8[55794]: 2024-03-24T03:21:02.328+0000 7f6e933a9700 -1 osd.8 127 *** Got signal Terminated *** 2024-03-24T03:21:02.711 INFO:journalctl@ceph.osd.8.smithi193.stdout:Mar 24 03:21:02 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-osd-8[55794]: 2024-03-24T03:21:02.328+0000 7f6e933a9700 -1 osd.8 127 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-24T03:21:02.946 INFO:journalctl@ceph.osd.9.smithi118.stdout:Mar 24 03:21:02 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-osd-9[67027]: 2024-03-24T03:21:02.596+0000 7f69f7400700 -1 osd.9 127 heartbeat_check: no reply from 172.21.15.118:6806 osd.0 since back 2024-03-24T03:19:57.002010+0000 front 2024-03-24T03:19:57.002333+0000 (oldest deadline 2024-03-24T03:20:21.101732+0000) 2024-03-24T03:21:02.946 INFO:journalctl@ceph.osd.9.smithi118.stdout:Mar 24 03:21:02 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-osd-9[67027]: 2024-03-24T03:21:02.596+0000 7f69f7400700 -1 osd.9 127 heartbeat_check: no reply from 172.21.15.189:6804 osd.1 since back 2024-03-24T03:20:05.804520+0000 front 2024-03-24T03:20:05.804379+0000 (oldest deadline 2024-03-24T03:20:29.303978+0000) 2024-03-24T03:21:02.946 INFO:journalctl@ceph.osd.9.smithi118.stdout:Mar 24 03:21:02 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-osd-9[67027]: 2024-03-24T03:21:02.596+0000 7f69f7400700 -1 osd.9 127 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-03-24T03:20:14.507274+0000 front 2024-03-24T03:20:14.507231+0000 (oldest deadline 2024-03-24T03:20:38.607488+0000) 2024-03-24T03:21:02.947 INFO:journalctl@ceph.osd.9.smithi118.stdout:Mar 24 03:21:02 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-osd-9[67027]: 2024-03-24T03:21:02.596+0000 7f69f7400700 -1 osd.9 127 heartbeat_check: no reply from 172.21.15.118:6814 osd.3 since back 2024-03-24T03:20:23.111195+0000 front 2024-03-24T03:20:23.111494+0000 (oldest deadline 2024-03-24T03:20:48.410762+0000) 2024-03-24T03:21:02.947 INFO:journalctl@ceph.osd.9.smithi118.stdout:Mar 24 03:21:02 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-osd-9[67027]: 2024-03-24T03:21:02.596+0000 7f69f7400700 -1 osd.9 127 heartbeat_check: no reply from 172.21.15.189:6812 osd.4 since back 2024-03-24T03:20:31.213661+0000 front 2024-03-24T03:20:31.213727+0000 (oldest deadline 2024-03-24T03:20:55.913869+0000) 2024-03-24T03:21:03.211 INFO:journalctl@ceph.osd.8.smithi193.stdout:Mar 24 03:21:02 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-osd-8[55794]: 2024-03-24T03:21:02.903+0000 7f6e8b995700 -1 osd.8 127 heartbeat_check: no reply from 172.21.15.189:6804 osd.1 since back 2024-03-24T03:20:06.219689+0000 front 2024-03-24T03:20:06.219789+0000 (oldest deadline 2024-03-24T03:20:30.919690+0000) 2024-03-24T03:21:03.211 INFO:journalctl@ceph.osd.8.smithi193.stdout:Mar 24 03:21:02 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-osd-8[55794]: 2024-03-24T03:21:02.903+0000 7f6e8b995700 -1 osd.8 127 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-03-24T03:20:16.121217+0000 front 2024-03-24T03:20:16.121198+0000 (oldest deadline 2024-03-24T03:20:40.221317+0000) 2024-03-24T03:21:03.211 INFO:journalctl@ceph.osd.8.smithi193.stdout:Mar 24 03:21:02 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-osd-8[55794]: 2024-03-24T03:21:02.903+0000 7f6e8b995700 -1 osd.8 127 heartbeat_check: no reply from 172.21.15.118:6814 osd.3 since back 2024-03-24T03:20:24.922540+0000 front 2024-03-24T03:20:24.922747+0000 (oldest deadline 2024-03-24T03:20:46.622438+0000) 2024-03-24T03:21:03.211 INFO:journalctl@ceph.osd.8.smithi193.stdout:Mar 24 03:21:02 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-osd-8[55794]: 2024-03-24T03:21:02.903+0000 7f6e8b995700 -1 osd.8 127 heartbeat_check: no reply from 172.21.15.189:6812 osd.4 since back 2024-03-24T03:20:32.523632+0000 front 2024-03-24T03:20:32.523692+0000 (oldest deadline 2024-03-24T03:20:57.223574+0000) 2024-03-24T03:21:03.211 INFO:journalctl@ceph.osd.11.smithi193.stdout:Mar 24 03:21:02 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-osd-11[62452]: 2024-03-24T03:21:02.898+0000 7f0edd521700 -1 osd.11 127 heartbeat_check: no reply from 172.21.15.118:6806 osd.0 since back 2024-03-24T03:19:59.095638+0000 front 2024-03-24T03:19:59.095627+0000 (oldest deadline 2024-03-24T03:20:20.195530+0000) 2024-03-24T03:21:03.211 INFO:journalctl@ceph.osd.11.smithi193.stdout:Mar 24 03:21:02 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-osd-11[62452]: 2024-03-24T03:21:02.898+0000 7f0edd521700 -1 osd.11 127 heartbeat_check: no reply from 172.21.15.189:6804 osd.1 since back 2024-03-24T03:20:07.399753+0000 front 2024-03-24T03:20:07.399812+0000 (oldest deadline 2024-03-24T03:20:32.099923+0000) 2024-03-24T03:21:03.212 INFO:journalctl@ceph.osd.11.smithi193.stdout:Mar 24 03:21:02 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-osd-11[62452]: 2024-03-24T03:21:02.898+0000 7f0edd521700 -1 osd.11 127 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-03-24T03:20:12.100299+0000 front 2024-03-24T03:20:12.100348+0000 (oldest deadline 2024-03-24T03:20:38.000441+0000) 2024-03-24T03:21:03.212 INFO:journalctl@ceph.osd.11.smithi193.stdout:Mar 24 03:21:02 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-osd-11[62452]: 2024-03-24T03:21:02.898+0000 7f0edd521700 -1 osd.11 127 heartbeat_check: no reply from 172.21.15.118:6814 osd.3 since back 2024-03-24T03:20:25.002506+0000 front 2024-03-24T03:20:25.002576+0000 (oldest deadline 2024-03-24T03:20:46.702390+0000) 2024-03-24T03:21:03.212 INFO:journalctl@ceph.osd.11.smithi193.stdout:Mar 24 03:21:02 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-osd-11[62452]: 2024-03-24T03:21:02.898+0000 7f0edd521700 -1 osd.11 127 heartbeat_check: no reply from 172.21.15.189:6812 osd.4 since back 2024-03-24T03:20:32.905105+0000 front 2024-03-24T03:20:32.905067+0000 (oldest deadline 2024-03-24T03:20:55.204998+0000) 2024-03-24T03:21:03.454 INFO:journalctl@ceph.osd.10.smithi189.stdout:Mar 24 03:21:02 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-osd-10[61294]: 2024-03-24T03:21:02.987+0000 7f804e31f700 -1 osd.10 127 heartbeat_check: no reply from 172.21.15.118:6806 osd.0 since back 2024-03-24T03:19:56.942815+0000 front 2024-03-24T03:19:56.942951+0000 (oldest deadline 2024-03-24T03:20:22.842714+0000) 2024-03-24T03:21:03.454 INFO:journalctl@ceph.osd.10.smithi189.stdout:Mar 24 03:21:02 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-osd-10[61294]: 2024-03-24T03:21:02.987+0000 7f804e31f700 -1 osd.10 127 heartbeat_check: no reply from 172.21.15.189:6804 osd.1 since back 2024-03-24T03:20:07.543899+0000 front 2024-03-24T03:20:07.543880+0000 (oldest deadline 2024-03-24T03:20:30.944770+0000) 2024-03-24T03:21:03.454 INFO:journalctl@ceph.osd.10.smithi189.stdout:Mar 24 03:21:02 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-osd-10[61294]: 2024-03-24T03:21:02.987+0000 7f804e31f700 -1 osd.10 127 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-03-24T03:20:15.546606+0000 front 2024-03-24T03:20:15.546459+0000 (oldest deadline 2024-03-24T03:20:37.846574+0000) 2024-03-24T03:21:03.454 INFO:journalctl@ceph.osd.10.smithi189.stdout:Mar 24 03:21:02 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-osd-10[61294]: 2024-03-24T03:21:02.987+0000 7f804e31f700 -1 osd.10 127 heartbeat_check: no reply from 172.21.15.118:6814 osd.3 since back 2024-03-24T03:20:22.547814+0000 front 2024-03-24T03:20:22.547848+0000 (oldest deadline 2024-03-24T03:20:47.748188+0000) 2024-03-24T03:21:03.454 INFO:journalctl@ceph.osd.10.smithi189.stdout:Mar 24 03:21:02 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-osd-10[61294]: 2024-03-24T03:21:02.987+0000 7f804e31f700 -1 osd.10 127 heartbeat_check: no reply from 172.21.15.189:6812 osd.4 since back 2024-03-24T03:20:33.049108+0000 front 2024-03-24T03:20:33.049154+0000 (oldest deadline 2024-03-24T03:20:55.949290+0000) 2024-03-24T03:21:03.946 INFO:journalctl@ceph.osd.9.smithi118.stdout:Mar 24 03:21:03 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-osd-9[67027]: 2024-03-24T03:21:03.583+0000 7f69f7400700 -1 osd.9 127 heartbeat_check: no reply from 172.21.15.118:6806 osd.0 since back 2024-03-24T03:19:57.002010+0000 front 2024-03-24T03:19:57.002333+0000 (oldest deadline 2024-03-24T03:20:21.101732+0000) 2024-03-24T03:21:03.946 INFO:journalctl@ceph.osd.9.smithi118.stdout:Mar 24 03:21:03 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-osd-9[67027]: 2024-03-24T03:21:03.583+0000 7f69f7400700 -1 osd.9 127 heartbeat_check: no reply from 172.21.15.189:6804 osd.1 since back 2024-03-24T03:20:05.804520+0000 front 2024-03-24T03:20:05.804379+0000 (oldest deadline 2024-03-24T03:20:29.303978+0000) 2024-03-24T03:21:03.946 INFO:journalctl@ceph.osd.9.smithi118.stdout:Mar 24 03:21:03 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-osd-9[67027]: 2024-03-24T03:21:03.583+0000 7f69f7400700 -1 osd.9 127 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-03-24T03:20:14.507274+0000 front 2024-03-24T03:20:14.507231+0000 (oldest deadline 2024-03-24T03:20:38.607488+0000) 2024-03-24T03:21:03.946 INFO:journalctl@ceph.osd.9.smithi118.stdout:Mar 24 03:21:03 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-osd-9[67027]: 2024-03-24T03:21:03.583+0000 7f69f7400700 -1 osd.9 127 heartbeat_check: no reply from 172.21.15.118:6814 osd.3 since back 2024-03-24T03:20:23.111195+0000 front 2024-03-24T03:20:23.111494+0000 (oldest deadline 2024-03-24T03:20:48.410762+0000) 2024-03-24T03:21:03.947 INFO:journalctl@ceph.osd.9.smithi118.stdout:Mar 24 03:21:03 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-osd-9[67027]: 2024-03-24T03:21:03.583+0000 7f69f7400700 -1 osd.9 127 heartbeat_check: no reply from 172.21.15.189:6812 osd.4 since back 2024-03-24T03:20:31.213661+0000 front 2024-03-24T03:20:31.213727+0000 (oldest deadline 2024-03-24T03:20:55.913869+0000) 2024-03-24T03:21:04.210 INFO:journalctl@ceph.osd.8.smithi193.stdout:Mar 24 03:21:03 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-osd-8[55794]: 2024-03-24T03:21:03.920+0000 7f6e8b995700 -1 osd.8 127 heartbeat_check: no reply from 172.21.15.189:6804 osd.1 since back 2024-03-24T03:20:06.219689+0000 front 2024-03-24T03:20:06.219789+0000 (oldest deadline 2024-03-24T03:20:30.919690+0000) 2024-03-24T03:21:04.211 INFO:journalctl@ceph.osd.8.smithi193.stdout:Mar 24 03:21:03 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-osd-8[55794]: 2024-03-24T03:21:03.920+0000 7f6e8b995700 -1 osd.8 127 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-03-24T03:20:16.121217+0000 front 2024-03-24T03:20:16.121198+0000 (oldest deadline 2024-03-24T03:20:40.221317+0000) 2024-03-24T03:21:04.211 INFO:journalctl@ceph.osd.8.smithi193.stdout:Mar 24 03:21:03 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-osd-8[55794]: 2024-03-24T03:21:03.920+0000 7f6e8b995700 -1 osd.8 127 heartbeat_check: no reply from 172.21.15.118:6814 osd.3 since back 2024-03-24T03:20:24.922540+0000 front 2024-03-24T03:20:24.922747+0000 (oldest deadline 2024-03-24T03:20:46.622438+0000) 2024-03-24T03:21:04.211 INFO:journalctl@ceph.osd.8.smithi193.stdout:Mar 24 03:21:03 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-osd-8[55794]: 2024-03-24T03:21:03.920+0000 7f6e8b995700 -1 osd.8 127 heartbeat_check: no reply from 172.21.15.189:6812 osd.4 since back 2024-03-24T03:20:32.523632+0000 front 2024-03-24T03:20:32.523692+0000 (oldest deadline 2024-03-24T03:20:57.223574+0000) 2024-03-24T03:21:04.211 INFO:journalctl@ceph.osd.8.smithi193.stdout:Mar 24 03:21:03 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-osd-8[55794]: 2024-03-24T03:21:03.920+0000 7f6e8b995700 -1 osd.8 127 heartbeat_check: no reply from 172.21.15.193:6812 osd.5 since back 2024-03-24T03:20:37.223975+0000 front 2024-03-24T03:20:37.224038+0000 (oldest deadline 2024-03-24T03:21:03.124063+0000) 2024-03-24T03:21:04.212 INFO:journalctl@ceph.osd.11.smithi193.stdout:Mar 24 03:21:03 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-osd-11[62452]: 2024-03-24T03:21:03.856+0000 7f0edd521700 -1 osd.11 127 heartbeat_check: no reply from 172.21.15.118:6806 osd.0 since back 2024-03-24T03:19:59.095638+0000 front 2024-03-24T03:19:59.095627+0000 (oldest deadline 2024-03-24T03:20:20.195530+0000) 2024-03-24T03:21:04.212 INFO:journalctl@ceph.osd.11.smithi193.stdout:Mar 24 03:21:03 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-osd-11[62452]: 2024-03-24T03:21:03.856+0000 7f0edd521700 -1 osd.11 127 heartbeat_check: no reply from 172.21.15.189:6804 osd.1 since back 2024-03-24T03:20:07.399753+0000 front 2024-03-24T03:20:07.399812+0000 (oldest deadline 2024-03-24T03:20:32.099923+0000) 2024-03-24T03:21:04.212 INFO:journalctl@ceph.osd.11.smithi193.stdout:Mar 24 03:21:03 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-osd-11[62452]: 2024-03-24T03:21:03.856+0000 7f0edd521700 -1 osd.11 127 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-03-24T03:20:12.100299+0000 front 2024-03-24T03:20:12.100348+0000 (oldest deadline 2024-03-24T03:20:38.000441+0000) 2024-03-24T03:21:04.212 INFO:journalctl@ceph.osd.11.smithi193.stdout:Mar 24 03:21:03 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-osd-11[62452]: 2024-03-24T03:21:03.856+0000 7f0edd521700 -1 osd.11 127 heartbeat_check: no reply from 172.21.15.118:6814 osd.3 since back 2024-03-24T03:20:25.002506+0000 front 2024-03-24T03:20:25.002576+0000 (oldest deadline 2024-03-24T03:20:46.702390+0000) 2024-03-24T03:21:04.212 INFO:journalctl@ceph.osd.11.smithi193.stdout:Mar 24 03:21:03 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-osd-11[62452]: 2024-03-24T03:21:03.856+0000 7f0edd521700 -1 osd.11 127 heartbeat_check: no reply from 172.21.15.189:6812 osd.4 since back 2024-03-24T03:20:32.905105+0000 front 2024-03-24T03:20:32.905067+0000 (oldest deadline 2024-03-24T03:20:55.204998+0000) 2024-03-24T03:21:04.453 INFO:journalctl@ceph.osd.10.smithi189.stdout:Mar 24 03:21:03 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-osd-10[61294]: 2024-03-24T03:21:03.983+0000 7f804e31f700 -1 osd.10 127 heartbeat_check: no reply from 172.21.15.118:6806 osd.0 since back 2024-03-24T03:19:56.942815+0000 front 2024-03-24T03:19:56.942951+0000 (oldest deadline 2024-03-24T03:20:22.842714+0000) 2024-03-24T03:21:04.454 INFO:journalctl@ceph.osd.10.smithi189.stdout:Mar 24 03:21:03 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-osd-10[61294]: 2024-03-24T03:21:03.983+0000 7f804e31f700 -1 osd.10 127 heartbeat_check: no reply from 172.21.15.189:6804 osd.1 since back 2024-03-24T03:20:07.543899+0000 front 2024-03-24T03:20:07.543880+0000 (oldest deadline 2024-03-24T03:20:30.944770+0000) 2024-03-24T03:21:04.454 INFO:journalctl@ceph.osd.10.smithi189.stdout:Mar 24 03:21:03 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-osd-10[61294]: 2024-03-24T03:21:03.983+0000 7f804e31f700 -1 osd.10 127 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-03-24T03:20:15.546606+0000 front 2024-03-24T03:20:15.546459+0000 (oldest deadline 2024-03-24T03:20:37.846574+0000) 2024-03-24T03:21:04.454 INFO:journalctl@ceph.osd.10.smithi189.stdout:Mar 24 03:21:03 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-osd-10[61294]: 2024-03-24T03:21:03.983+0000 7f804e31f700 -1 osd.10 127 heartbeat_check: no reply from 172.21.15.118:6814 osd.3 since back 2024-03-24T03:20:22.547814+0000 front 2024-03-24T03:20:22.547848+0000 (oldest deadline 2024-03-24T03:20:47.748188+0000) 2024-03-24T03:21:04.454 INFO:journalctl@ceph.osd.10.smithi189.stdout:Mar 24 03:21:03 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-osd-10[61294]: 2024-03-24T03:21:03.983+0000 7f804e31f700 -1 osd.10 127 heartbeat_check: no reply from 172.21.15.189:6812 osd.4 since back 2024-03-24T03:20:33.049108+0000 front 2024-03-24T03:20:33.049154+0000 (oldest deadline 2024-03-24T03:20:55.949290+0000) 2024-03-24T03:21:04.946 INFO:journalctl@ceph.osd.9.smithi118.stdout:Mar 24 03:21:04 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-osd-9[67027]: 2024-03-24T03:21:04.556+0000 7f69f7400700 -1 osd.9 127 heartbeat_check: no reply from 172.21.15.118:6806 osd.0 since back 2024-03-24T03:19:57.002010+0000 front 2024-03-24T03:19:57.002333+0000 (oldest deadline 2024-03-24T03:20:21.101732+0000) 2024-03-24T03:21:04.947 INFO:journalctl@ceph.osd.9.smithi118.stdout:Mar 24 03:21:04 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-osd-9[67027]: 2024-03-24T03:21:04.556+0000 7f69f7400700 -1 osd.9 127 heartbeat_check: no reply from 172.21.15.189:6804 osd.1 since back 2024-03-24T03:20:05.804520+0000 front 2024-03-24T03:20:05.804379+0000 (oldest deadline 2024-03-24T03:20:29.303978+0000) 2024-03-24T03:21:04.947 INFO:journalctl@ceph.osd.9.smithi118.stdout:Mar 24 03:21:04 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-osd-9[67027]: 2024-03-24T03:21:04.556+0000 7f69f7400700 -1 osd.9 127 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-03-24T03:20:14.507274+0000 front 2024-03-24T03:20:14.507231+0000 (oldest deadline 2024-03-24T03:20:38.607488+0000) 2024-03-24T03:21:04.947 INFO:journalctl@ceph.osd.9.smithi118.stdout:Mar 24 03:21:04 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-osd-9[67027]: 2024-03-24T03:21:04.556+0000 7f69f7400700 -1 osd.9 127 heartbeat_check: no reply from 172.21.15.118:6814 osd.3 since back 2024-03-24T03:20:23.111195+0000 front 2024-03-24T03:20:23.111494+0000 (oldest deadline 2024-03-24T03:20:48.410762+0000) 2024-03-24T03:21:04.947 INFO:journalctl@ceph.osd.9.smithi118.stdout:Mar 24 03:21:04 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-osd-9[67027]: 2024-03-24T03:21:04.556+0000 7f69f7400700 -1 osd.9 127 heartbeat_check: no reply from 172.21.15.189:6812 osd.4 since back 2024-03-24T03:20:31.213661+0000 front 2024-03-24T03:20:31.213727+0000 (oldest deadline 2024-03-24T03:20:55.913869+0000) 2024-03-24T03:21:04.947 INFO:journalctl@ceph.osd.9.smithi118.stdout:Mar 24 03:21:04 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-osd-9[67027]: 2024-03-24T03:21:04.556+0000 7f69f7400700 -1 osd.9 127 heartbeat_check: no reply from 172.21.15.193:6812 osd.5 since back 2024-03-24T03:20:41.617507+0000 front 2024-03-24T03:20:41.617436+0000 (oldest deadline 2024-03-24T03:21:03.917405+0000) 2024-03-24T03:21:05.204 INFO:journalctl@ceph.osd.10.smithi189.stdout:Mar 24 03:21:04 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-osd-10[61294]: 2024-03-24T03:21:04.950+0000 7f804e31f700 -1 osd.10 127 heartbeat_check: no reply from 172.21.15.118:6806 osd.0 since back 2024-03-24T03:19:56.942815+0000 front 2024-03-24T03:19:56.942951+0000 (oldest deadline 2024-03-24T03:20:22.842714+0000) 2024-03-24T03:21:05.204 INFO:journalctl@ceph.osd.10.smithi189.stdout:Mar 24 03:21:04 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-osd-10[61294]: 2024-03-24T03:21:04.950+0000 7f804e31f700 -1 osd.10 127 heartbeat_check: no reply from 172.21.15.189:6804 osd.1 since back 2024-03-24T03:20:07.543899+0000 front 2024-03-24T03:20:07.543880+0000 (oldest deadline 2024-03-24T03:20:30.944770+0000) 2024-03-24T03:21:05.204 INFO:journalctl@ceph.osd.10.smithi189.stdout:Mar 24 03:21:04 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-osd-10[61294]: 2024-03-24T03:21:04.950+0000 7f804e31f700 -1 osd.10 127 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-03-24T03:20:15.546606+0000 front 2024-03-24T03:20:15.546459+0000 (oldest deadline 2024-03-24T03:20:37.846574+0000) 2024-03-24T03:21:05.204 INFO:journalctl@ceph.osd.10.smithi189.stdout:Mar 24 03:21:04 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-osd-10[61294]: 2024-03-24T03:21:04.950+0000 7f804e31f700 -1 osd.10 127 heartbeat_check: no reply from 172.21.15.118:6814 osd.3 since back 2024-03-24T03:20:22.547814+0000 front 2024-03-24T03:20:22.547848+0000 (oldest deadline 2024-03-24T03:20:47.748188+0000) 2024-03-24T03:21:05.204 INFO:journalctl@ceph.osd.10.smithi189.stdout:Mar 24 03:21:04 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-osd-10[61294]: 2024-03-24T03:21:04.950+0000 7f804e31f700 -1 osd.10 127 heartbeat_check: no reply from 172.21.15.189:6812 osd.4 since back 2024-03-24T03:20:33.049108+0000 front 2024-03-24T03:20:33.049154+0000 (oldest deadline 2024-03-24T03:20:55.949290+0000) 2024-03-24T03:21:05.204 INFO:journalctl@ceph.osd.10.smithi189.stdout:Mar 24 03:21:04 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-osd-10[61294]: 2024-03-24T03:21:04.950+0000 7f804e31f700 -1 osd.10 127 heartbeat_check: no reply from 172.21.15.193:6812 osd.5 since back 2024-03-24T03:20:41.750894+0000 front 2024-03-24T03:20:41.750949+0000 (oldest deadline 2024-03-24T03:21:04.050717+0000) 2024-03-24T03:21:05.210 INFO:journalctl@ceph.osd.8.smithi193.stdout:Mar 24 03:21:04 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-osd-8[55794]: 2024-03-24T03:21:04.948+0000 7f6e8b995700 -1 osd.8 127 heartbeat_check: no reply from 172.21.15.189:6804 osd.1 since back 2024-03-24T03:20:06.219689+0000 front 2024-03-24T03:20:06.219789+0000 (oldest deadline 2024-03-24T03:20:30.919690+0000) 2024-03-24T03:21:05.210 INFO:journalctl@ceph.osd.8.smithi193.stdout:Mar 24 03:21:04 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-osd-8[55794]: 2024-03-24T03:21:04.948+0000 7f6e8b995700 -1 osd.8 127 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-03-24T03:20:16.121217+0000 front 2024-03-24T03:20:16.121198+0000 (oldest deadline 2024-03-24T03:20:40.221317+0000) 2024-03-24T03:21:05.211 INFO:journalctl@ceph.osd.8.smithi193.stdout:Mar 24 03:21:04 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-osd-8[55794]: 2024-03-24T03:21:04.948+0000 7f6e8b995700 -1 osd.8 127 heartbeat_check: no reply from 172.21.15.118:6814 osd.3 since back 2024-03-24T03:20:24.922540+0000 front 2024-03-24T03:20:24.922747+0000 (oldest deadline 2024-03-24T03:20:46.622438+0000) 2024-03-24T03:21:05.211 INFO:journalctl@ceph.osd.8.smithi193.stdout:Mar 24 03:21:04 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-osd-8[55794]: 2024-03-24T03:21:04.948+0000 7f6e8b995700 -1 osd.8 127 heartbeat_check: no reply from 172.21.15.189:6812 osd.4 since back 2024-03-24T03:20:32.523632+0000 front 2024-03-24T03:20:32.523692+0000 (oldest deadline 2024-03-24T03:20:57.223574+0000) 2024-03-24T03:21:05.211 INFO:journalctl@ceph.osd.8.smithi193.stdout:Mar 24 03:21:04 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-osd-8[55794]: 2024-03-24T03:21:04.948+0000 7f6e8b995700 -1 osd.8 127 heartbeat_check: no reply from 172.21.15.193:6812 osd.5 since back 2024-03-24T03:20:37.223975+0000 front 2024-03-24T03:20:37.224038+0000 (oldest deadline 2024-03-24T03:21:03.124063+0000) 2024-03-24T03:21:05.211 INFO:journalctl@ceph.osd.11.smithi193.stdout:Mar 24 03:21:04 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-osd-11[62452]: 2024-03-24T03:21:04.875+0000 7f0edd521700 -1 osd.11 127 heartbeat_check: no reply from 172.21.15.118:6806 osd.0 since back 2024-03-24T03:19:59.095638+0000 front 2024-03-24T03:19:59.095627+0000 (oldest deadline 2024-03-24T03:20:20.195530+0000) 2024-03-24T03:21:05.211 INFO:journalctl@ceph.osd.11.smithi193.stdout:Mar 24 03:21:04 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-osd-11[62452]: 2024-03-24T03:21:04.875+0000 7f0edd521700 -1 osd.11 127 heartbeat_check: no reply from 172.21.15.189:6804 osd.1 since back 2024-03-24T03:20:07.399753+0000 front 2024-03-24T03:20:07.399812+0000 (oldest deadline 2024-03-24T03:20:32.099923+0000) 2024-03-24T03:21:05.211 INFO:journalctl@ceph.osd.11.smithi193.stdout:Mar 24 03:21:04 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-osd-11[62452]: 2024-03-24T03:21:04.875+0000 7f0edd521700 -1 osd.11 127 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-03-24T03:20:12.100299+0000 front 2024-03-24T03:20:12.100348+0000 (oldest deadline 2024-03-24T03:20:38.000441+0000) 2024-03-24T03:21:05.211 INFO:journalctl@ceph.osd.11.smithi193.stdout:Mar 24 03:21:04 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-osd-11[62452]: 2024-03-24T03:21:04.875+0000 7f0edd521700 -1 osd.11 127 heartbeat_check: no reply from 172.21.15.118:6814 osd.3 since back 2024-03-24T03:20:25.002506+0000 front 2024-03-24T03:20:25.002576+0000 (oldest deadline 2024-03-24T03:20:46.702390+0000) 2024-03-24T03:21:05.212 INFO:journalctl@ceph.osd.11.smithi193.stdout:Mar 24 03:21:04 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-osd-11[62452]: 2024-03-24T03:21:04.875+0000 7f0edd521700 -1 osd.11 127 heartbeat_check: no reply from 172.21.15.189:6812 osd.4 since back 2024-03-24T03:20:32.905105+0000 front 2024-03-24T03:20:32.905067+0000 (oldest deadline 2024-03-24T03:20:55.204998+0000) 2024-03-24T03:21:05.946 INFO:journalctl@ceph.osd.9.smithi118.stdout:Mar 24 03:21:05 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-osd-9[67027]: 2024-03-24T03:21:05.600+0000 7f69f7400700 -1 osd.9 127 heartbeat_check: no reply from 172.21.15.118:6806 osd.0 since back 2024-03-24T03:19:57.002010+0000 front 2024-03-24T03:19:57.002333+0000 (oldest deadline 2024-03-24T03:20:21.101732+0000) 2024-03-24T03:21:05.946 INFO:journalctl@ceph.osd.9.smithi118.stdout:Mar 24 03:21:05 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-osd-9[67027]: 2024-03-24T03:21:05.600+0000 7f69f7400700 -1 osd.9 127 heartbeat_check: no reply from 172.21.15.189:6804 osd.1 since back 2024-03-24T03:20:05.804520+0000 front 2024-03-24T03:20:05.804379+0000 (oldest deadline 2024-03-24T03:20:29.303978+0000) 2024-03-24T03:21:05.947 INFO:journalctl@ceph.osd.9.smithi118.stdout:Mar 24 03:21:05 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-osd-9[67027]: 2024-03-24T03:21:05.600+0000 7f69f7400700 -1 osd.9 127 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-03-24T03:20:14.507274+0000 front 2024-03-24T03:20:14.507231+0000 (oldest deadline 2024-03-24T03:20:38.607488+0000) 2024-03-24T03:21:05.947 INFO:journalctl@ceph.osd.9.smithi118.stdout:Mar 24 03:21:05 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-osd-9[67027]: 2024-03-24T03:21:05.600+0000 7f69f7400700 -1 osd.9 127 heartbeat_check: no reply from 172.21.15.118:6814 osd.3 since back 2024-03-24T03:20:23.111195+0000 front 2024-03-24T03:20:23.111494+0000 (oldest deadline 2024-03-24T03:20:48.410762+0000) 2024-03-24T03:21:05.947 INFO:journalctl@ceph.osd.9.smithi118.stdout:Mar 24 03:21:05 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-osd-9[67027]: 2024-03-24T03:21:05.600+0000 7f69f7400700 -1 osd.9 127 heartbeat_check: no reply from 172.21.15.189:6812 osd.4 since back 2024-03-24T03:20:31.213661+0000 front 2024-03-24T03:20:31.213727+0000 (oldest deadline 2024-03-24T03:20:55.913869+0000) 2024-03-24T03:21:05.947 INFO:journalctl@ceph.osd.9.smithi118.stdout:Mar 24 03:21:05 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-osd-9[67027]: 2024-03-24T03:21:05.600+0000 7f69f7400700 -1 osd.9 127 heartbeat_check: no reply from 172.21.15.193:6812 osd.5 since back 2024-03-24T03:20:41.617507+0000 front 2024-03-24T03:20:41.617436+0000 (oldest deadline 2024-03-24T03:21:03.917405+0000) 2024-03-24T03:21:06.203 INFO:journalctl@ceph.osd.10.smithi189.stdout:Mar 24 03:21:05 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-osd-10[61294]: 2024-03-24T03:21:05.930+0000 7f804e31f700 -1 osd.10 127 heartbeat_check: no reply from 172.21.15.118:6806 osd.0 since back 2024-03-24T03:19:56.942815+0000 front 2024-03-24T03:19:56.942951+0000 (oldest deadline 2024-03-24T03:20:22.842714+0000) 2024-03-24T03:21:06.204 INFO:journalctl@ceph.osd.10.smithi189.stdout:Mar 24 03:21:05 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-osd-10[61294]: 2024-03-24T03:21:05.930+0000 7f804e31f700 -1 osd.10 127 heartbeat_check: no reply from 172.21.15.189:6804 osd.1 since back 2024-03-24T03:20:07.543899+0000 front 2024-03-24T03:20:07.543880+0000 (oldest deadline 2024-03-24T03:20:30.944770+0000) 2024-03-24T03:21:06.204 INFO:journalctl@ceph.osd.10.smithi189.stdout:Mar 24 03:21:05 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-osd-10[61294]: 2024-03-24T03:21:05.930+0000 7f804e31f700 -1 osd.10 127 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-03-24T03:20:15.546606+0000 front 2024-03-24T03:20:15.546459+0000 (oldest deadline 2024-03-24T03:20:37.846574+0000) 2024-03-24T03:21:06.204 INFO:journalctl@ceph.osd.10.smithi189.stdout:Mar 24 03:21:05 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-osd-10[61294]: 2024-03-24T03:21:05.930+0000 7f804e31f700 -1 osd.10 127 heartbeat_check: no reply from 172.21.15.118:6814 osd.3 since back 2024-03-24T03:20:22.547814+0000 front 2024-03-24T03:20:22.547848+0000 (oldest deadline 2024-03-24T03:20:47.748188+0000) 2024-03-24T03:21:06.204 INFO:journalctl@ceph.osd.10.smithi189.stdout:Mar 24 03:21:05 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-osd-10[61294]: 2024-03-24T03:21:05.930+0000 7f804e31f700 -1 osd.10 127 heartbeat_check: no reply from 172.21.15.189:6812 osd.4 since back 2024-03-24T03:20:33.049108+0000 front 2024-03-24T03:20:33.049154+0000 (oldest deadline 2024-03-24T03:20:55.949290+0000) 2024-03-24T03:21:06.204 INFO:journalctl@ceph.osd.10.smithi189.stdout:Mar 24 03:21:05 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-osd-10[61294]: 2024-03-24T03:21:05.930+0000 7f804e31f700 -1 osd.10 127 heartbeat_check: no reply from 172.21.15.193:6812 osd.5 since back 2024-03-24T03:20:41.750894+0000 front 2024-03-24T03:20:41.750949+0000 (oldest deadline 2024-03-24T03:21:04.050717+0000) 2024-03-24T03:21:06.210 INFO:journalctl@ceph.osd.8.smithi193.stdout:Mar 24 03:21:05 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-osd-8[55794]: 2024-03-24T03:21:05.943+0000 7f6e8b995700 -1 osd.8 127 heartbeat_check: no reply from 172.21.15.189:6804 osd.1 since back 2024-03-24T03:20:06.219689+0000 front 2024-03-24T03:20:06.219789+0000 (oldest deadline 2024-03-24T03:20:30.919690+0000) 2024-03-24T03:21:06.211 INFO:journalctl@ceph.osd.8.smithi193.stdout:Mar 24 03:21:05 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-osd-8[55794]: 2024-03-24T03:21:05.943+0000 7f6e8b995700 -1 osd.8 127 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-03-24T03:20:16.121217+0000 front 2024-03-24T03:20:16.121198+0000 (oldest deadline 2024-03-24T03:20:40.221317+0000) 2024-03-24T03:21:06.211 INFO:journalctl@ceph.osd.8.smithi193.stdout:Mar 24 03:21:05 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-osd-8[55794]: 2024-03-24T03:21:05.943+0000 7f6e8b995700 -1 osd.8 127 heartbeat_check: no reply from 172.21.15.118:6814 osd.3 since back 2024-03-24T03:20:24.922540+0000 front 2024-03-24T03:20:24.922747+0000 (oldest deadline 2024-03-24T03:20:46.622438+0000) 2024-03-24T03:21:06.211 INFO:journalctl@ceph.osd.8.smithi193.stdout:Mar 24 03:21:05 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-osd-8[55794]: 2024-03-24T03:21:05.943+0000 7f6e8b995700 -1 osd.8 127 heartbeat_check: no reply from 172.21.15.189:6812 osd.4 since back 2024-03-24T03:20:32.523632+0000 front 2024-03-24T03:20:32.523692+0000 (oldest deadline 2024-03-24T03:20:57.223574+0000) 2024-03-24T03:21:06.211 INFO:journalctl@ceph.osd.8.smithi193.stdout:Mar 24 03:21:05 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-osd-8[55794]: 2024-03-24T03:21:05.943+0000 7f6e8b995700 -1 osd.8 127 heartbeat_check: no reply from 172.21.15.193:6812 osd.5 since back 2024-03-24T03:20:37.223975+0000 front 2024-03-24T03:20:37.224038+0000 (oldest deadline 2024-03-24T03:21:03.124063+0000) 2024-03-24T03:21:06.211 INFO:journalctl@ceph.osd.11.smithi193.stdout:Mar 24 03:21:05 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-osd-11[62452]: 2024-03-24T03:21:05.903+0000 7f0edd521700 -1 osd.11 127 heartbeat_check: no reply from 172.21.15.118:6806 osd.0 since back 2024-03-24T03:19:59.095638+0000 front 2024-03-24T03:19:59.095627+0000 (oldest deadline 2024-03-24T03:20:20.195530+0000) 2024-03-24T03:21:06.211 INFO:journalctl@ceph.osd.11.smithi193.stdout:Mar 24 03:21:05 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-osd-11[62452]: 2024-03-24T03:21:05.903+0000 7f0edd521700 -1 osd.11 127 heartbeat_check: no reply from 172.21.15.189:6804 osd.1 since back 2024-03-24T03:20:07.399753+0000 front 2024-03-24T03:20:07.399812+0000 (oldest deadline 2024-03-24T03:20:32.099923+0000) 2024-03-24T03:21:06.211 INFO:journalctl@ceph.osd.11.smithi193.stdout:Mar 24 03:21:05 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-osd-11[62452]: 2024-03-24T03:21:05.903+0000 7f0edd521700 -1 osd.11 127 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-03-24T03:20:12.100299+0000 front 2024-03-24T03:20:12.100348+0000 (oldest deadline 2024-03-24T03:20:38.000441+0000) 2024-03-24T03:21:06.212 INFO:journalctl@ceph.osd.11.smithi193.stdout:Mar 24 03:21:05 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-osd-11[62452]: 2024-03-24T03:21:05.903+0000 7f0edd521700 -1 osd.11 127 heartbeat_check: no reply from 172.21.15.118:6814 osd.3 since back 2024-03-24T03:20:25.002506+0000 front 2024-03-24T03:20:25.002576+0000 (oldest deadline 2024-03-24T03:20:46.702390+0000) 2024-03-24T03:21:06.212 INFO:journalctl@ceph.osd.11.smithi193.stdout:Mar 24 03:21:05 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-osd-11[62452]: 2024-03-24T03:21:05.903+0000 7f0edd521700 -1 osd.11 127 heartbeat_check: no reply from 172.21.15.189:6812 osd.4 since back 2024-03-24T03:20:32.905105+0000 front 2024-03-24T03:20:32.905067+0000 (oldest deadline 2024-03-24T03:20:55.204998+0000) 2024-03-24T03:21:06.212 INFO:journalctl@ceph.osd.11.smithi193.stdout:Mar 24 03:21:05 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-osd-11[62452]: 2024-03-24T03:21:05.903+0000 7f0edd521700 -1 osd.11 127 heartbeat_check: no reply from 172.21.15.193:6812 osd.5 since back 2024-03-24T03:20:42.106727+0000 front 2024-03-24T03:20:42.106844+0000 (oldest deadline 2024-03-24T03:21:05.006783+0000) 2024-03-24T03:21:06.946 INFO:journalctl@ceph.osd.9.smithi118.stdout:Mar 24 03:21:06 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-osd-9[67027]: 2024-03-24T03:21:06.553+0000 7f69f7400700 -1 osd.9 127 heartbeat_check: no reply from 172.21.15.118:6806 osd.0 since back 2024-03-24T03:19:57.002010+0000 front 2024-03-24T03:19:57.002333+0000 (oldest deadline 2024-03-24T03:20:21.101732+0000) 2024-03-24T03:21:06.947 INFO:journalctl@ceph.osd.9.smithi118.stdout:Mar 24 03:21:06 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-osd-9[67027]: 2024-03-24T03:21:06.553+0000 7f69f7400700 -1 osd.9 127 heartbeat_check: no reply from 172.21.15.189:6804 osd.1 since back 2024-03-24T03:20:05.804520+0000 front 2024-03-24T03:20:05.804379+0000 (oldest deadline 2024-03-24T03:20:29.303978+0000) 2024-03-24T03:21:06.947 INFO:journalctl@ceph.osd.9.smithi118.stdout:Mar 24 03:21:06 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-osd-9[67027]: 2024-03-24T03:21:06.553+0000 7f69f7400700 -1 osd.9 127 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-03-24T03:20:14.507274+0000 front 2024-03-24T03:20:14.507231+0000 (oldest deadline 2024-03-24T03:20:38.607488+0000) 2024-03-24T03:21:06.947 INFO:journalctl@ceph.osd.9.smithi118.stdout:Mar 24 03:21:06 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-osd-9[67027]: 2024-03-24T03:21:06.553+0000 7f69f7400700 -1 osd.9 127 heartbeat_check: no reply from 172.21.15.118:6814 osd.3 since back 2024-03-24T03:20:23.111195+0000 front 2024-03-24T03:20:23.111494+0000 (oldest deadline 2024-03-24T03:20:48.410762+0000) 2024-03-24T03:21:06.947 INFO:journalctl@ceph.osd.9.smithi118.stdout:Mar 24 03:21:06 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-osd-9[67027]: 2024-03-24T03:21:06.553+0000 7f69f7400700 -1 osd.9 127 heartbeat_check: no reply from 172.21.15.189:6812 osd.4 since back 2024-03-24T03:20:31.213661+0000 front 2024-03-24T03:20:31.213727+0000 (oldest deadline 2024-03-24T03:20:55.913869+0000) 2024-03-24T03:21:06.947 INFO:journalctl@ceph.osd.9.smithi118.stdout:Mar 24 03:21:06 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-osd-9[67027]: 2024-03-24T03:21:06.553+0000 7f69f7400700 -1 osd.9 127 heartbeat_check: no reply from 172.21.15.193:6812 osd.5 since back 2024-03-24T03:20:41.617507+0000 front 2024-03-24T03:20:41.617436+0000 (oldest deadline 2024-03-24T03:21:03.917405+0000) 2024-03-24T03:21:07.210 INFO:journalctl@ceph.osd.8.smithi193.stdout:Mar 24 03:21:06 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-osd-8[55794]: 2024-03-24T03:21:06.972+0000 7f6e8b995700 -1 osd.8 127 heartbeat_check: no reply from 172.21.15.189:6804 osd.1 since back 2024-03-24T03:20:06.219689+0000 front 2024-03-24T03:20:06.219789+0000 (oldest deadline 2024-03-24T03:20:30.919690+0000) 2024-03-24T03:21:07.211 INFO:journalctl@ceph.osd.8.smithi193.stdout:Mar 24 03:21:06 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-osd-8[55794]: 2024-03-24T03:21:06.972+0000 7f6e8b995700 -1 osd.8 127 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-03-24T03:20:16.121217+0000 front 2024-03-24T03:20:16.121198+0000 (oldest deadline 2024-03-24T03:20:40.221317+0000) 2024-03-24T03:21:07.211 INFO:journalctl@ceph.osd.8.smithi193.stdout:Mar 24 03:21:06 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-osd-8[55794]: 2024-03-24T03:21:06.972+0000 7f6e8b995700 -1 osd.8 127 heartbeat_check: no reply from 172.21.15.118:6814 osd.3 since back 2024-03-24T03:20:24.922540+0000 front 2024-03-24T03:20:24.922747+0000 (oldest deadline 2024-03-24T03:20:46.622438+0000) 2024-03-24T03:21:07.211 INFO:journalctl@ceph.osd.8.smithi193.stdout:Mar 24 03:21:06 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-osd-8[55794]: 2024-03-24T03:21:06.972+0000 7f6e8b995700 -1 osd.8 127 heartbeat_check: no reply from 172.21.15.189:6812 osd.4 since back 2024-03-24T03:20:32.523632+0000 front 2024-03-24T03:20:32.523692+0000 (oldest deadline 2024-03-24T03:20:57.223574+0000) 2024-03-24T03:21:07.211 INFO:journalctl@ceph.osd.8.smithi193.stdout:Mar 24 03:21:06 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-osd-8[55794]: 2024-03-24T03:21:06.972+0000 7f6e8b995700 -1 osd.8 127 heartbeat_check: no reply from 172.21.15.193:6812 osd.5 since back 2024-03-24T03:20:37.223975+0000 front 2024-03-24T03:20:37.224038+0000 (oldest deadline 2024-03-24T03:21:03.124063+0000) 2024-03-24T03:21:07.211 INFO:journalctl@ceph.osd.11.smithi193.stdout:Mar 24 03:21:06 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-osd-11[62452]: 2024-03-24T03:21:06.919+0000 7f0edd521700 -1 osd.11 127 heartbeat_check: no reply from 172.21.15.118:6806 osd.0 since back 2024-03-24T03:19:59.095638+0000 front 2024-03-24T03:19:59.095627+0000 (oldest deadline 2024-03-24T03:20:20.195530+0000) 2024-03-24T03:21:07.211 INFO:journalctl@ceph.osd.11.smithi193.stdout:Mar 24 03:21:06 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-osd-11[62452]: 2024-03-24T03:21:06.919+0000 7f0edd521700 -1 osd.11 127 heartbeat_check: no reply from 172.21.15.189:6804 osd.1 since back 2024-03-24T03:20:07.399753+0000 front 2024-03-24T03:20:07.399812+0000 (oldest deadline 2024-03-24T03:20:32.099923+0000) 2024-03-24T03:21:07.212 INFO:journalctl@ceph.osd.11.smithi193.stdout:Mar 24 03:21:06 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-osd-11[62452]: 2024-03-24T03:21:06.919+0000 7f0edd521700 -1 osd.11 127 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-03-24T03:20:12.100299+0000 front 2024-03-24T03:20:12.100348+0000 (oldest deadline 2024-03-24T03:20:38.000441+0000) 2024-03-24T03:21:07.212 INFO:journalctl@ceph.osd.11.smithi193.stdout:Mar 24 03:21:06 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-osd-11[62452]: 2024-03-24T03:21:06.919+0000 7f0edd521700 -1 osd.11 127 heartbeat_check: no reply from 172.21.15.118:6814 osd.3 since back 2024-03-24T03:20:25.002506+0000 front 2024-03-24T03:20:25.002576+0000 (oldest deadline 2024-03-24T03:20:46.702390+0000) 2024-03-24T03:21:07.212 INFO:journalctl@ceph.osd.11.smithi193.stdout:Mar 24 03:21:06 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-osd-11[62452]: 2024-03-24T03:21:06.919+0000 7f0edd521700 -1 osd.11 127 heartbeat_check: no reply from 172.21.15.189:6812 osd.4 since back 2024-03-24T03:20:32.905105+0000 front 2024-03-24T03:20:32.905067+0000 (oldest deadline 2024-03-24T03:20:55.204998+0000) 2024-03-24T03:21:07.212 INFO:journalctl@ceph.osd.11.smithi193.stdout:Mar 24 03:21:06 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-osd-11[62452]: 2024-03-24T03:21:06.919+0000 7f0edd521700 -1 osd.11 127 heartbeat_check: no reply from 172.21.15.193:6812 osd.5 since back 2024-03-24T03:20:42.106727+0000 front 2024-03-24T03:20:42.106844+0000 (oldest deadline 2024-03-24T03:21:05.006783+0000) 2024-03-24T03:21:07.454 INFO:journalctl@ceph.osd.10.smithi189.stdout:Mar 24 03:21:06 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-osd-10[61294]: 2024-03-24T03:21:06.956+0000 7f804e31f700 -1 osd.10 127 heartbeat_check: no reply from 172.21.15.118:6806 osd.0 since back 2024-03-24T03:19:56.942815+0000 front 2024-03-24T03:19:56.942951+0000 (oldest deadline 2024-03-24T03:20:22.842714+0000) 2024-03-24T03:21:07.454 INFO:journalctl@ceph.osd.10.smithi189.stdout:Mar 24 03:21:06 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-osd-10[61294]: 2024-03-24T03:21:06.956+0000 7f804e31f700 -1 osd.10 127 heartbeat_check: no reply from 172.21.15.189:6804 osd.1 since back 2024-03-24T03:20:07.543899+0000 front 2024-03-24T03:20:07.543880+0000 (oldest deadline 2024-03-24T03:20:30.944770+0000) 2024-03-24T03:21:07.454 INFO:journalctl@ceph.osd.10.smithi189.stdout:Mar 24 03:21:06 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-osd-10[61294]: 2024-03-24T03:21:06.956+0000 7f804e31f700 -1 osd.10 127 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-03-24T03:20:15.546606+0000 front 2024-03-24T03:20:15.546459+0000 (oldest deadline 2024-03-24T03:20:37.846574+0000) 2024-03-24T03:21:07.454 INFO:journalctl@ceph.osd.10.smithi189.stdout:Mar 24 03:21:06 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-osd-10[61294]: 2024-03-24T03:21:06.956+0000 7f804e31f700 -1 osd.10 127 heartbeat_check: no reply from 172.21.15.118:6814 osd.3 since back 2024-03-24T03:20:22.547814+0000 front 2024-03-24T03:20:22.547848+0000 (oldest deadline 2024-03-24T03:20:47.748188+0000) 2024-03-24T03:21:07.454 INFO:journalctl@ceph.osd.10.smithi189.stdout:Mar 24 03:21:06 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-osd-10[61294]: 2024-03-24T03:21:06.956+0000 7f804e31f700 -1 osd.10 127 heartbeat_check: no reply from 172.21.15.189:6812 osd.4 since back 2024-03-24T03:20:33.049108+0000 front 2024-03-24T03:20:33.049154+0000 (oldest deadline 2024-03-24T03:20:55.949290+0000) 2024-03-24T03:21:07.454 INFO:journalctl@ceph.osd.10.smithi189.stdout:Mar 24 03:21:06 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-osd-10[61294]: 2024-03-24T03:21:06.956+0000 7f804e31f700 -1 osd.10 127 heartbeat_check: no reply from 172.21.15.193:6812 osd.5 since back 2024-03-24T03:20:41.750894+0000 front 2024-03-24T03:20:41.750949+0000 (oldest deadline 2024-03-24T03:21:04.050717+0000) 2024-03-24T03:21:07.946 INFO:journalctl@ceph.osd.9.smithi118.stdout:Mar 24 03:21:07 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-osd-9[67027]: 2024-03-24T03:21:07.535+0000 7f69f7400700 -1 osd.9 127 heartbeat_check: no reply from 172.21.15.118:6806 osd.0 since back 2024-03-24T03:19:57.002010+0000 front 2024-03-24T03:19:57.002333+0000 (oldest deadline 2024-03-24T03:20:21.101732+0000) 2024-03-24T03:21:07.946 INFO:journalctl@ceph.osd.9.smithi118.stdout:Mar 24 03:21:07 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-osd-9[67027]: 2024-03-24T03:21:07.535+0000 7f69f7400700 -1 osd.9 127 heartbeat_check: no reply from 172.21.15.189:6804 osd.1 since back 2024-03-24T03:20:05.804520+0000 front 2024-03-24T03:20:05.804379+0000 (oldest deadline 2024-03-24T03:20:29.303978+0000) 2024-03-24T03:21:07.946 INFO:journalctl@ceph.osd.9.smithi118.stdout:Mar 24 03:21:07 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-osd-9[67027]: 2024-03-24T03:21:07.535+0000 7f69f7400700 -1 osd.9 127 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-03-24T03:20:14.507274+0000 front 2024-03-24T03:20:14.507231+0000 (oldest deadline 2024-03-24T03:20:38.607488+0000) 2024-03-24T03:21:07.947 INFO:journalctl@ceph.osd.9.smithi118.stdout:Mar 24 03:21:07 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-osd-9[67027]: 2024-03-24T03:21:07.535+0000 7f69f7400700 -1 osd.9 127 heartbeat_check: no reply from 172.21.15.118:6814 osd.3 since back 2024-03-24T03:20:23.111195+0000 front 2024-03-24T03:20:23.111494+0000 (oldest deadline 2024-03-24T03:20:48.410762+0000) 2024-03-24T03:21:07.947 INFO:journalctl@ceph.osd.9.smithi118.stdout:Mar 24 03:21:07 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-osd-9[67027]: 2024-03-24T03:21:07.535+0000 7f69f7400700 -1 osd.9 127 heartbeat_check: no reply from 172.21.15.189:6812 osd.4 since back 2024-03-24T03:20:31.213661+0000 front 2024-03-24T03:20:31.213727+0000 (oldest deadline 2024-03-24T03:20:55.913869+0000) 2024-03-24T03:21:07.947 INFO:journalctl@ceph.osd.9.smithi118.stdout:Mar 24 03:21:07 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-osd-9[67027]: 2024-03-24T03:21:07.535+0000 7f69f7400700 -1 osd.9 127 heartbeat_check: no reply from 172.21.15.193:6812 osd.5 since back 2024-03-24T03:20:41.617507+0000 front 2024-03-24T03:20:41.617436+0000 (oldest deadline 2024-03-24T03:21:03.917405+0000) 2024-03-24T03:21:07.986 INFO:journalctl@ceph.osd.8.smithi193.stdout:Mar 24 03:21:07 smithi193 podman[120745]: 2024-03-24 03:21:07.72098107 +0000 UTC m=+5.492027426 container died c2109ee696bd0f5f4381c23c61207bcda924a6429f021462457ca9beaa012d89 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-d79a5420-e971-11ee-95cd-87774f69a715-osd-8, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.build-date=20240319, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5) 2024-03-24T03:21:07.987 INFO:journalctl@ceph.osd.11.smithi193.stdout:Mar 24 03:21:07 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-osd-11[62452]: 2024-03-24T03:21:07.953+0000 7f0edd521700 -1 osd.11 127 heartbeat_check: no reply from 172.21.15.118:6806 osd.0 since back 2024-03-24T03:19:59.095638+0000 front 2024-03-24T03:19:59.095627+0000 (oldest deadline 2024-03-24T03:20:20.195530+0000) 2024-03-24T03:21:07.987 INFO:journalctl@ceph.osd.11.smithi193.stdout:Mar 24 03:21:07 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-osd-11[62452]: 2024-03-24T03:21:07.953+0000 7f0edd521700 -1 osd.11 127 heartbeat_check: no reply from 172.21.15.189:6804 osd.1 since back 2024-03-24T03:20:07.399753+0000 front 2024-03-24T03:20:07.399812+0000 (oldest deadline 2024-03-24T03:20:32.099923+0000) 2024-03-24T03:21:07.987 INFO:journalctl@ceph.osd.11.smithi193.stdout:Mar 24 03:21:07 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-osd-11[62452]: 2024-03-24T03:21:07.953+0000 7f0edd521700 -1 osd.11 127 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-03-24T03:20:12.100299+0000 front 2024-03-24T03:20:12.100348+0000 (oldest deadline 2024-03-24T03:20:38.000441+0000) 2024-03-24T03:21:07.987 INFO:journalctl@ceph.osd.11.smithi193.stdout:Mar 24 03:21:07 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-osd-11[62452]: 2024-03-24T03:21:07.953+0000 7f0edd521700 -1 osd.11 127 heartbeat_check: no reply from 172.21.15.118:6814 osd.3 since back 2024-03-24T03:20:25.002506+0000 front 2024-03-24T03:20:25.002576+0000 (oldest deadline 2024-03-24T03:20:46.702390+0000) 2024-03-24T03:21:07.987 INFO:journalctl@ceph.osd.11.smithi193.stdout:Mar 24 03:21:07 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-osd-11[62452]: 2024-03-24T03:21:07.953+0000 7f0edd521700 -1 osd.11 127 heartbeat_check: no reply from 172.21.15.189:6812 osd.4 since back 2024-03-24T03:20:32.905105+0000 front 2024-03-24T03:20:32.905067+0000 (oldest deadline 2024-03-24T03:20:55.204998+0000) 2024-03-24T03:21:07.987 INFO:journalctl@ceph.osd.11.smithi193.stdout:Mar 24 03:21:07 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-osd-11[62452]: 2024-03-24T03:21:07.953+0000 7f0edd521700 -1 osd.11 127 heartbeat_check: no reply from 172.21.15.193:6812 osd.5 since back 2024-03-24T03:20:42.106727+0000 front 2024-03-24T03:20:42.106844+0000 (oldest deadline 2024-03-24T03:21:05.006783+0000) 2024-03-24T03:21:08.203 INFO:journalctl@ceph.osd.10.smithi189.stdout:Mar 24 03:21:07 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-osd-10[61294]: 2024-03-24T03:21:07.928+0000 7f804e31f700 -1 osd.10 127 heartbeat_check: no reply from 172.21.15.118:6806 osd.0 since back 2024-03-24T03:19:56.942815+0000 front 2024-03-24T03:19:56.942951+0000 (oldest deadline 2024-03-24T03:20:22.842714+0000) 2024-03-24T03:21:08.204 INFO:journalctl@ceph.osd.10.smithi189.stdout:Mar 24 03:21:07 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-osd-10[61294]: 2024-03-24T03:21:07.928+0000 7f804e31f700 -1 osd.10 127 heartbeat_check: no reply from 172.21.15.189:6804 osd.1 since back 2024-03-24T03:20:07.543899+0000 front 2024-03-24T03:20:07.543880+0000 (oldest deadline 2024-03-24T03:20:30.944770+0000) 2024-03-24T03:21:08.204 INFO:journalctl@ceph.osd.10.smithi189.stdout:Mar 24 03:21:07 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-osd-10[61294]: 2024-03-24T03:21:07.928+0000 7f804e31f700 -1 osd.10 127 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-03-24T03:20:15.546606+0000 front 2024-03-24T03:20:15.546459+0000 (oldest deadline 2024-03-24T03:20:37.846574+0000) 2024-03-24T03:21:08.204 INFO:journalctl@ceph.osd.10.smithi189.stdout:Mar 24 03:21:07 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-osd-10[61294]: 2024-03-24T03:21:07.928+0000 7f804e31f700 -1 osd.10 127 heartbeat_check: no reply from 172.21.15.118:6814 osd.3 since back 2024-03-24T03:20:22.547814+0000 front 2024-03-24T03:20:22.547848+0000 (oldest deadline 2024-03-24T03:20:47.748188+0000) 2024-03-24T03:21:08.204 INFO:journalctl@ceph.osd.10.smithi189.stdout:Mar 24 03:21:07 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-osd-10[61294]: 2024-03-24T03:21:07.928+0000 7f804e31f700 -1 osd.10 127 heartbeat_check: no reply from 172.21.15.189:6812 osd.4 since back 2024-03-24T03:20:33.049108+0000 front 2024-03-24T03:20:33.049154+0000 (oldest deadline 2024-03-24T03:20:55.949290+0000) 2024-03-24T03:21:08.204 INFO:journalctl@ceph.osd.10.smithi189.stdout:Mar 24 03:21:07 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-osd-10[61294]: 2024-03-24T03:21:07.928+0000 7f804e31f700 -1 osd.10 127 heartbeat_check: no reply from 172.21.15.193:6812 osd.5 since back 2024-03-24T03:20:41.750894+0000 front 2024-03-24T03:20:41.750949+0000 (oldest deadline 2024-03-24T03:21:04.050717+0000) 2024-03-24T03:21:08.270 INFO:journalctl@ceph.osd.8.smithi193.stdout:Mar 24 03:21:08 smithi193 podman[120745]: 2024-03-24 03:21:08.070543636 +0000 UTC m=+5.841590020 container cleanup c2109ee696bd0f5f4381c23c61207bcda924a6429f021462457ca9beaa012d89 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-d79a5420-e971-11ee-95cd-87774f69a715-osd-8, GIT_BRANCH=HEAD, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.schema-version=1.0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.build-date=20240319, ceph=True, org.label-schema.vendor=CentOS) 2024-03-24T03:21:08.271 INFO:journalctl@ceph.osd.8.smithi193.stdout:Mar 24 03:21:08 smithi193 bash[120745]: ceph-d79a5420-e971-11ee-95cd-87774f69a715-osd-8 2024-03-24T03:21:08.671 INFO:journalctl@ceph.osd.8.smithi193.stdout:Mar 24 03:21:08 smithi193 podman[120762]: 2024-03-24 03:21:08.271688941 +0000 UTC m=+0.545356315 container remove c2109ee696bd0f5f4381c23c61207bcda924a6429f021462457ca9beaa012d89 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-d79a5420-e971-11ee-95cd-87774f69a715-osd-8, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240319, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-24T03:21:08.936 INFO:journalctl@ceph.osd.8.smithi193.stdout:Mar 24 03:21:08 smithi193 podman[120852]: 2024-03-24 03:21:08.672557633 +0000 UTC m=+0.111503918 container create 28083062326ef85816dc41f60ca45855943ad41ed7f7aa73f85412f203949146 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-d79a5420-e971-11ee-95cd-87774f69a715-osd-8-deactivate, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, org.label-schema.build-date=20240319, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.license=GPLv2) 2024-03-24T03:21:08.936 INFO:journalctl@ceph.osd.8.smithi193.stdout:Mar 24 03:21:08 smithi193 podman[120852]: 2024-03-24 03:21:08.578631542 +0000 UTC m=+0.017577805 image pull c664afe56bb01db4f9afa9532c20598e201e664f2e9ba2be50db0d52674a26d2 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc 2024-03-24T03:21:08.936 INFO:journalctl@ceph.osd.8.smithi193.stdout:Mar 24 03:21:08 smithi193 podman[120852]: 2024-03-24 03:21:08.875396586 +0000 UTC m=+0.314342846 container init 28083062326ef85816dc41f60ca45855943ad41ed7f7aa73f85412f203949146 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-d79a5420-e971-11ee-95cd-87774f69a715-osd-8-deactivate, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, ceph=True, org.label-schema.vendor=CentOS, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240319, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD) 2024-03-24T03:21:08.936 INFO:journalctl@ceph.osd.8.smithi193.stdout:Mar 24 03:21:08 smithi193 podman[120852]: 2024-03-24 03:21:08.878538428 +0000 UTC m=+0.317484687 container start 28083062326ef85816dc41f60ca45855943ad41ed7f7aa73f85412f203949146 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-d79a5420-e971-11ee-95cd-87774f69a715-osd-8-deactivate, org.label-schema.build-date=20240319, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_CLEAN=True, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-24T03:21:08.946 INFO:journalctl@ceph.osd.9.smithi118.stdout:Mar 24 03:21:08 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-osd-9[67027]: 2024-03-24T03:21:08.557+0000 7f69f7400700 -1 osd.9 127 heartbeat_check: no reply from 172.21.15.118:6806 osd.0 since back 2024-03-24T03:19:57.002010+0000 front 2024-03-24T03:19:57.002333+0000 (oldest deadline 2024-03-24T03:20:21.101732+0000) 2024-03-24T03:21:08.946 INFO:journalctl@ceph.osd.9.smithi118.stdout:Mar 24 03:21:08 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-osd-9[67027]: 2024-03-24T03:21:08.557+0000 7f69f7400700 -1 osd.9 127 heartbeat_check: no reply from 172.21.15.189:6804 osd.1 since back 2024-03-24T03:20:05.804520+0000 front 2024-03-24T03:20:05.804379+0000 (oldest deadline 2024-03-24T03:20:29.303978+0000) 2024-03-24T03:21:08.946 INFO:journalctl@ceph.osd.9.smithi118.stdout:Mar 24 03:21:08 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-osd-9[67027]: 2024-03-24T03:21:08.557+0000 7f69f7400700 -1 osd.9 127 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-03-24T03:20:14.507274+0000 front 2024-03-24T03:20:14.507231+0000 (oldest deadline 2024-03-24T03:20:38.607488+0000) 2024-03-24T03:21:08.946 INFO:journalctl@ceph.osd.9.smithi118.stdout:Mar 24 03:21:08 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-osd-9[67027]: 2024-03-24T03:21:08.557+0000 7f69f7400700 -1 osd.9 127 heartbeat_check: no reply from 172.21.15.118:6814 osd.3 since back 2024-03-24T03:20:23.111195+0000 front 2024-03-24T03:20:23.111494+0000 (oldest deadline 2024-03-24T03:20:48.410762+0000) 2024-03-24T03:21:08.946 INFO:journalctl@ceph.osd.9.smithi118.stdout:Mar 24 03:21:08 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-osd-9[67027]: 2024-03-24T03:21:08.557+0000 7f69f7400700 -1 osd.9 127 heartbeat_check: no reply from 172.21.15.189:6812 osd.4 since back 2024-03-24T03:20:31.213661+0000 front 2024-03-24T03:20:31.213727+0000 (oldest deadline 2024-03-24T03:20:55.913869+0000) 2024-03-24T03:21:08.947 INFO:journalctl@ceph.osd.9.smithi118.stdout:Mar 24 03:21:08 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-osd-9[67027]: 2024-03-24T03:21:08.557+0000 7f69f7400700 -1 osd.9 127 heartbeat_check: no reply from 172.21.15.193:6812 osd.5 since back 2024-03-24T03:20:41.617507+0000 front 2024-03-24T03:20:41.617436+0000 (oldest deadline 2024-03-24T03:21:03.917405+0000) 2024-03-24T03:21:09.189 INFO:journalctl@ceph.osd.8.smithi193.stdout:Mar 24 03:21:08 smithi193 podman[120852]: 2024-03-24 03:21:08.967484293 +0000 UTC m=+0.406430558 container attach 28083062326ef85816dc41f60ca45855943ad41ed7f7aa73f85412f203949146 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-d79a5420-e971-11ee-95cd-87774f69a715-osd-8-deactivate, io.buildah.version=1.33.5, ceph=True, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, org.label-schema.build-date=20240319, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux ) 2024-03-24T03:21:09.190 INFO:journalctl@ceph.osd.8.smithi193.stdout:Mar 24 03:21:09 smithi193 podman[120852]: 2024-03-24 03:21:09.169711057 +0000 UTC m=+0.608657308 container died 28083062326ef85816dc41f60ca45855943ad41ed7f7aa73f85412f203949146 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-d79a5420-e971-11ee-95cd-87774f69a715-osd-8-deactivate, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, ceph=True, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, org.label-schema.build-date=20240319, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS) 2024-03-24T03:21:09.190 INFO:journalctl@ceph.osd.11.smithi193.stdout:Mar 24 03:21:08 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-osd-11[62452]: 2024-03-24T03:21:08.935+0000 7f0edd521700 -1 osd.11 127 heartbeat_check: no reply from 172.21.15.118:6806 osd.0 since back 2024-03-24T03:19:59.095638+0000 front 2024-03-24T03:19:59.095627+0000 (oldest deadline 2024-03-24T03:20:20.195530+0000) 2024-03-24T03:21:09.190 INFO:journalctl@ceph.osd.11.smithi193.stdout:Mar 24 03:21:08 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-osd-11[62452]: 2024-03-24T03:21:08.935+0000 7f0edd521700 -1 osd.11 127 heartbeat_check: no reply from 172.21.15.189:6804 osd.1 since back 2024-03-24T03:20:07.399753+0000 front 2024-03-24T03:20:07.399812+0000 (oldest deadline 2024-03-24T03:20:32.099923+0000) 2024-03-24T03:21:09.190 INFO:journalctl@ceph.osd.11.smithi193.stdout:Mar 24 03:21:08 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-osd-11[62452]: 2024-03-24T03:21:08.935+0000 7f0edd521700 -1 osd.11 127 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-03-24T03:20:12.100299+0000 front 2024-03-24T03:20:12.100348+0000 (oldest deadline 2024-03-24T03:20:38.000441+0000) 2024-03-24T03:21:09.190 INFO:journalctl@ceph.osd.11.smithi193.stdout:Mar 24 03:21:08 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-osd-11[62452]: 2024-03-24T03:21:08.935+0000 7f0edd521700 -1 osd.11 127 heartbeat_check: no reply from 172.21.15.118:6814 osd.3 since back 2024-03-24T03:20:25.002506+0000 front 2024-03-24T03:20:25.002576+0000 (oldest deadline 2024-03-24T03:20:46.702390+0000) 2024-03-24T03:21:09.191 INFO:journalctl@ceph.osd.11.smithi193.stdout:Mar 24 03:21:08 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-osd-11[62452]: 2024-03-24T03:21:08.935+0000 7f0edd521700 -1 osd.11 127 heartbeat_check: no reply from 172.21.15.189:6812 osd.4 since back 2024-03-24T03:20:32.905105+0000 front 2024-03-24T03:20:32.905067+0000 (oldest deadline 2024-03-24T03:20:55.204998+0000) 2024-03-24T03:21:09.191 INFO:journalctl@ceph.osd.11.smithi193.stdout:Mar 24 03:21:08 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-osd-11[62452]: 2024-03-24T03:21:08.935+0000 7f0edd521700 -1 osd.11 127 heartbeat_check: no reply from 172.21.15.193:6812 osd.5 since back 2024-03-24T03:20:42.106727+0000 front 2024-03-24T03:20:42.106844+0000 (oldest deadline 2024-03-24T03:21:05.006783+0000) 2024-03-24T03:21:09.204 INFO:journalctl@ceph.osd.10.smithi189.stdout:Mar 24 03:21:08 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-osd-10[61294]: 2024-03-24T03:21:08.887+0000 7f804e31f700 -1 osd.10 127 heartbeat_check: no reply from 172.21.15.118:6806 osd.0 since back 2024-03-24T03:19:56.942815+0000 front 2024-03-24T03:19:56.942951+0000 (oldest deadline 2024-03-24T03:20:22.842714+0000) 2024-03-24T03:21:09.204 INFO:journalctl@ceph.osd.10.smithi189.stdout:Mar 24 03:21:08 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-osd-10[61294]: 2024-03-24T03:21:08.887+0000 7f804e31f700 -1 osd.10 127 heartbeat_check: no reply from 172.21.15.189:6804 osd.1 since back 2024-03-24T03:20:07.543899+0000 front 2024-03-24T03:20:07.543880+0000 (oldest deadline 2024-03-24T03:20:30.944770+0000) 2024-03-24T03:21:09.204 INFO:journalctl@ceph.osd.10.smithi189.stdout:Mar 24 03:21:08 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-osd-10[61294]: 2024-03-24T03:21:08.887+0000 7f804e31f700 -1 osd.10 127 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-03-24T03:20:15.546606+0000 front 2024-03-24T03:20:15.546459+0000 (oldest deadline 2024-03-24T03:20:37.846574+0000) 2024-03-24T03:21:09.204 INFO:journalctl@ceph.osd.10.smithi189.stdout:Mar 24 03:21:08 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-osd-10[61294]: 2024-03-24T03:21:08.887+0000 7f804e31f700 -1 osd.10 127 heartbeat_check: no reply from 172.21.15.118:6814 osd.3 since back 2024-03-24T03:20:22.547814+0000 front 2024-03-24T03:20:22.547848+0000 (oldest deadline 2024-03-24T03:20:47.748188+0000) 2024-03-24T03:21:09.204 INFO:journalctl@ceph.osd.10.smithi189.stdout:Mar 24 03:21:08 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-osd-10[61294]: 2024-03-24T03:21:08.887+0000 7f804e31f700 -1 osd.10 127 heartbeat_check: no reply from 172.21.15.189:6812 osd.4 since back 2024-03-24T03:20:33.049108+0000 front 2024-03-24T03:20:33.049154+0000 (oldest deadline 2024-03-24T03:20:55.949290+0000) 2024-03-24T03:21:09.204 INFO:journalctl@ceph.osd.10.smithi189.stdout:Mar 24 03:21:08 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-osd-10[61294]: 2024-03-24T03:21:08.887+0000 7f804e31f700 -1 osd.10 127 heartbeat_check: no reply from 172.21.15.193:6812 osd.5 since back 2024-03-24T03:20:41.750894+0000 front 2024-03-24T03:20:41.750949+0000 (oldest deadline 2024-03-24T03:21:04.050717+0000) 2024-03-24T03:21:09.786 DEBUG:teuthology.orchestra.run.smithi193:> sudo pkill -f 'journalctl -f -n 0 -u ceph-d79a5420-e971-11ee-95cd-87774f69a715@osd.8.service' 2024-03-24T03:21:09.827 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-24T03:21:09.827 INFO:tasks.cephadm.osd.8:Stopped osd.8 2024-03-24T03:21:09.827 INFO:tasks.cephadm.osd.9:Stopping osd.9... 2024-03-24T03:21:09.827 DEBUG:teuthology.orchestra.run.smithi118:> sudo systemctl stop ceph-d79a5420-e971-11ee-95cd-87774f69a715@osd.9 2024-03-24T03:21:09.914 INFO:journalctl@ceph.osd.9.smithi118.stdout:Mar 24 03:21:09 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-osd-9[67027]: 2024-03-24T03:21:09.593+0000 7f69f7400700 -1 osd.9 127 heartbeat_check: no reply from 172.21.15.118:6806 osd.0 since back 2024-03-24T03:19:57.002010+0000 front 2024-03-24T03:19:57.002333+0000 (oldest deadline 2024-03-24T03:20:21.101732+0000) 2024-03-24T03:21:09.914 INFO:journalctl@ceph.osd.9.smithi118.stdout:Mar 24 03:21:09 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-osd-9[67027]: 2024-03-24T03:21:09.593+0000 7f69f7400700 -1 osd.9 127 heartbeat_check: no reply from 172.21.15.189:6804 osd.1 since back 2024-03-24T03:20:05.804520+0000 front 2024-03-24T03:20:05.804379+0000 (oldest deadline 2024-03-24T03:20:29.303978+0000) 2024-03-24T03:21:09.914 INFO:journalctl@ceph.osd.9.smithi118.stdout:Mar 24 03:21:09 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-osd-9[67027]: 2024-03-24T03:21:09.593+0000 7f69f7400700 -1 osd.9 127 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-03-24T03:20:14.507274+0000 front 2024-03-24T03:20:14.507231+0000 (oldest deadline 2024-03-24T03:20:38.607488+0000) 2024-03-24T03:21:09.914 INFO:journalctl@ceph.osd.9.smithi118.stdout:Mar 24 03:21:09 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-osd-9[67027]: 2024-03-24T03:21:09.593+0000 7f69f7400700 -1 osd.9 127 heartbeat_check: no reply from 172.21.15.118:6814 osd.3 since back 2024-03-24T03:20:23.111195+0000 front 2024-03-24T03:20:23.111494+0000 (oldest deadline 2024-03-24T03:20:48.410762+0000) 2024-03-24T03:21:09.915 INFO:journalctl@ceph.osd.9.smithi118.stdout:Mar 24 03:21:09 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-osd-9[67027]: 2024-03-24T03:21:09.593+0000 7f69f7400700 -1 osd.9 127 heartbeat_check: no reply from 172.21.15.189:6812 osd.4 since back 2024-03-24T03:20:31.213661+0000 front 2024-03-24T03:20:31.213727+0000 (oldest deadline 2024-03-24T03:20:55.913869+0000) 2024-03-24T03:21:09.915 INFO:journalctl@ceph.osd.9.smithi118.stdout:Mar 24 03:21:09 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-osd-9[67027]: 2024-03-24T03:21:09.593+0000 7f69f7400700 -1 osd.9 127 heartbeat_check: no reply from 172.21.15.193:6812 osd.5 since back 2024-03-24T03:20:41.617507+0000 front 2024-03-24T03:20:41.617436+0000 (oldest deadline 2024-03-24T03:21:03.917405+0000) 2024-03-24T03:21:10.116 INFO:journalctl@ceph.osd.10.smithi189.stdout:Mar 24 03:21:09 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-osd-10[61294]: 2024-03-24T03:21:09.854+0000 7f804e31f700 -1 osd.10 127 heartbeat_check: no reply from 172.21.15.118:6806 osd.0 since back 2024-03-24T03:19:56.942815+0000 front 2024-03-24T03:19:56.942951+0000 (oldest deadline 2024-03-24T03:20:22.842714+0000) 2024-03-24T03:21:10.116 INFO:journalctl@ceph.osd.10.smithi189.stdout:Mar 24 03:21:09 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-osd-10[61294]: 2024-03-24T03:21:09.854+0000 7f804e31f700 -1 osd.10 127 heartbeat_check: no reply from 172.21.15.189:6804 osd.1 since back 2024-03-24T03:20:07.543899+0000 front 2024-03-24T03:20:07.543880+0000 (oldest deadline 2024-03-24T03:20:30.944770+0000) 2024-03-24T03:21:10.117 INFO:journalctl@ceph.osd.10.smithi189.stdout:Mar 24 03:21:09 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-osd-10[61294]: 2024-03-24T03:21:09.855+0000 7f804e31f700 -1 osd.10 127 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-03-24T03:20:15.546606+0000 front 2024-03-24T03:20:15.546459+0000 (oldest deadline 2024-03-24T03:20:37.846574+0000) 2024-03-24T03:21:10.117 INFO:journalctl@ceph.osd.10.smithi189.stdout:Mar 24 03:21:09 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-osd-10[61294]: 2024-03-24T03:21:09.855+0000 7f804e31f700 -1 osd.10 127 heartbeat_check: no reply from 172.21.15.118:6814 osd.3 since back 2024-03-24T03:20:22.547814+0000 front 2024-03-24T03:20:22.547848+0000 (oldest deadline 2024-03-24T03:20:47.748188+0000) 2024-03-24T03:21:10.117 INFO:journalctl@ceph.osd.10.smithi189.stdout:Mar 24 03:21:09 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-osd-10[61294]: 2024-03-24T03:21:09.855+0000 7f804e31f700 -1 osd.10 127 heartbeat_check: no reply from 172.21.15.189:6812 osd.4 since back 2024-03-24T03:20:33.049108+0000 front 2024-03-24T03:20:33.049154+0000 (oldest deadline 2024-03-24T03:20:55.949290+0000) 2024-03-24T03:21:10.117 INFO:journalctl@ceph.osd.10.smithi189.stdout:Mar 24 03:21:09 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-osd-10[61294]: 2024-03-24T03:21:09.855+0000 7f804e31f700 -1 osd.10 127 heartbeat_check: no reply from 172.21.15.193:6812 osd.5 since back 2024-03-24T03:20:41.750894+0000 front 2024-03-24T03:20:41.750949+0000 (oldest deadline 2024-03-24T03:21:04.050717+0000) 2024-03-24T03:21:10.195 INFO:journalctl@ceph.osd.9.smithi118.stdout:Mar 24 03:21:09 smithi118 systemd[1]: Stopping Ceph osd.9 for d79a5420-e971-11ee-95cd-87774f69a715... 2024-03-24T03:21:10.210 INFO:journalctl@ceph.osd.11.smithi193.stdout:Mar 24 03:21:09 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-osd-11[62452]: 2024-03-24T03:21:09.976+0000 7f0edd521700 -1 osd.11 127 heartbeat_check: no reply from 172.21.15.118:6806 osd.0 since back 2024-03-24T03:19:59.095638+0000 front 2024-03-24T03:19:59.095627+0000 (oldest deadline 2024-03-24T03:20:20.195530+0000) 2024-03-24T03:21:10.211 INFO:journalctl@ceph.osd.11.smithi193.stdout:Mar 24 03:21:09 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-osd-11[62452]: 2024-03-24T03:21:09.976+0000 7f0edd521700 -1 osd.11 127 heartbeat_check: no reply from 172.21.15.189:6804 osd.1 since back 2024-03-24T03:20:07.399753+0000 front 2024-03-24T03:20:07.399812+0000 (oldest deadline 2024-03-24T03:20:32.099923+0000) 2024-03-24T03:21:10.211 INFO:journalctl@ceph.osd.11.smithi193.stdout:Mar 24 03:21:09 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-osd-11[62452]: 2024-03-24T03:21:09.976+0000 7f0edd521700 -1 osd.11 127 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-03-24T03:20:12.100299+0000 front 2024-03-24T03:20:12.100348+0000 (oldest deadline 2024-03-24T03:20:38.000441+0000) 2024-03-24T03:21:10.211 INFO:journalctl@ceph.osd.11.smithi193.stdout:Mar 24 03:21:09 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-osd-11[62452]: 2024-03-24T03:21:09.976+0000 7f0edd521700 -1 osd.11 127 heartbeat_check: no reply from 172.21.15.118:6814 osd.3 since back 2024-03-24T03:20:25.002506+0000 front 2024-03-24T03:20:25.002576+0000 (oldest deadline 2024-03-24T03:20:46.702390+0000) 2024-03-24T03:21:10.211 INFO:journalctl@ceph.osd.11.smithi193.stdout:Mar 24 03:21:09 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-osd-11[62452]: 2024-03-24T03:21:09.976+0000 7f0edd521700 -1 osd.11 127 heartbeat_check: no reply from 172.21.15.189:6812 osd.4 since back 2024-03-24T03:20:32.905105+0000 front 2024-03-24T03:20:32.905067+0000 (oldest deadline 2024-03-24T03:20:55.204998+0000) 2024-03-24T03:21:10.211 INFO:journalctl@ceph.osd.11.smithi193.stdout:Mar 24 03:21:09 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-osd-11[62452]: 2024-03-24T03:21:09.976+0000 7f0edd521700 -1 osd.11 127 heartbeat_check: no reply from 172.21.15.193:6812 osd.5 since back 2024-03-24T03:20:42.106727+0000 front 2024-03-24T03:20:42.106844+0000 (oldest deadline 2024-03-24T03:21:05.006783+0000) 2024-03-24T03:21:10.946 INFO:journalctl@ceph.osd.9.smithi118.stdout:Mar 24 03:21:10 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-osd-9[67027]: 2024-03-24T03:21:10.509+0000 7f6a00c00700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.9 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-24T03:21:10.947 INFO:journalctl@ceph.osd.9.smithi118.stdout:Mar 24 03:21:10 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-osd-9[67027]: 2024-03-24T03:21:10.509+0000 7f6a00c00700 -1 osd.9 127 *** Got signal Terminated *** 2024-03-24T03:21:10.947 INFO:journalctl@ceph.osd.9.smithi118.stdout:Mar 24 03:21:10 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-osd-9[67027]: 2024-03-24T03:21:10.509+0000 7f6a00c00700 -1 osd.9 127 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-24T03:21:10.947 INFO:journalctl@ceph.osd.9.smithi118.stdout:Mar 24 03:21:10 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-osd-9[67027]: 2024-03-24T03:21:10.618+0000 7f69f7400700 -1 osd.9 127 heartbeat_check: no reply from 172.21.15.118:6806 osd.0 since back 2024-03-24T03:19:57.002010+0000 front 2024-03-24T03:19:57.002333+0000 (oldest deadline 2024-03-24T03:20:21.101732+0000) 2024-03-24T03:21:10.947 INFO:journalctl@ceph.osd.9.smithi118.stdout:Mar 24 03:21:10 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-osd-9[67027]: 2024-03-24T03:21:10.618+0000 7f69f7400700 -1 osd.9 127 heartbeat_check: no reply from 172.21.15.189:6804 osd.1 since back 2024-03-24T03:20:05.804520+0000 front 2024-03-24T03:20:05.804379+0000 (oldest deadline 2024-03-24T03:20:29.303978+0000) 2024-03-24T03:21:10.947 INFO:journalctl@ceph.osd.9.smithi118.stdout:Mar 24 03:21:10 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-osd-9[67027]: 2024-03-24T03:21:10.618+0000 7f69f7400700 -1 osd.9 127 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-03-24T03:20:14.507274+0000 front 2024-03-24T03:20:14.507231+0000 (oldest deadline 2024-03-24T03:20:38.607488+0000) 2024-03-24T03:21:10.947 INFO:journalctl@ceph.osd.9.smithi118.stdout:Mar 24 03:21:10 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-osd-9[67027]: 2024-03-24T03:21:10.618+0000 7f69f7400700 -1 osd.9 127 heartbeat_check: no reply from 172.21.15.118:6814 osd.3 since back 2024-03-24T03:20:23.111195+0000 front 2024-03-24T03:20:23.111494+0000 (oldest deadline 2024-03-24T03:20:48.410762+0000) 2024-03-24T03:21:10.947 INFO:journalctl@ceph.osd.9.smithi118.stdout:Mar 24 03:21:10 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-osd-9[67027]: 2024-03-24T03:21:10.618+0000 7f69f7400700 -1 osd.9 127 heartbeat_check: no reply from 172.21.15.189:6812 osd.4 since back 2024-03-24T03:20:31.213661+0000 front 2024-03-24T03:20:31.213727+0000 (oldest deadline 2024-03-24T03:20:55.913869+0000) 2024-03-24T03:21:10.947 INFO:journalctl@ceph.osd.9.smithi118.stdout:Mar 24 03:21:10 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-osd-9[67027]: 2024-03-24T03:21:10.618+0000 7f69f7400700 -1 osd.9 127 heartbeat_check: no reply from 172.21.15.193:6812 osd.5 since back 2024-03-24T03:20:41.617507+0000 front 2024-03-24T03:20:41.617436+0000 (oldest deadline 2024-03-24T03:21:03.917405+0000) 2024-03-24T03:21:11.204 INFO:journalctl@ceph.osd.10.smithi189.stdout:Mar 24 03:21:10 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-osd-10[61294]: 2024-03-24T03:21:10.892+0000 7f804e31f700 -1 osd.10 127 heartbeat_check: no reply from 172.21.15.118:6806 osd.0 since back 2024-03-24T03:19:56.942815+0000 front 2024-03-24T03:19:56.942951+0000 (oldest deadline 2024-03-24T03:20:22.842714+0000) 2024-03-24T03:21:11.204 INFO:journalctl@ceph.osd.10.smithi189.stdout:Mar 24 03:21:10 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-osd-10[61294]: 2024-03-24T03:21:10.892+0000 7f804e31f700 -1 osd.10 127 heartbeat_check: no reply from 172.21.15.189:6804 osd.1 since back 2024-03-24T03:20:07.543899+0000 front 2024-03-24T03:20:07.543880+0000 (oldest deadline 2024-03-24T03:20:30.944770+0000) 2024-03-24T03:21:11.204 INFO:journalctl@ceph.osd.10.smithi189.stdout:Mar 24 03:21:10 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-osd-10[61294]: 2024-03-24T03:21:10.892+0000 7f804e31f700 -1 osd.10 127 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-03-24T03:20:15.546606+0000 front 2024-03-24T03:20:15.546459+0000 (oldest deadline 2024-03-24T03:20:37.846574+0000) 2024-03-24T03:21:11.204 INFO:journalctl@ceph.osd.10.smithi189.stdout:Mar 24 03:21:10 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-osd-10[61294]: 2024-03-24T03:21:10.892+0000 7f804e31f700 -1 osd.10 127 heartbeat_check: no reply from 172.21.15.118:6814 osd.3 since back 2024-03-24T03:20:22.547814+0000 front 2024-03-24T03:20:22.547848+0000 (oldest deadline 2024-03-24T03:20:47.748188+0000) 2024-03-24T03:21:11.204 INFO:journalctl@ceph.osd.10.smithi189.stdout:Mar 24 03:21:10 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-osd-10[61294]: 2024-03-24T03:21:10.892+0000 7f804e31f700 -1 osd.10 127 heartbeat_check: no reply from 172.21.15.189:6812 osd.4 since back 2024-03-24T03:20:33.049108+0000 front 2024-03-24T03:20:33.049154+0000 (oldest deadline 2024-03-24T03:20:55.949290+0000) 2024-03-24T03:21:11.204 INFO:journalctl@ceph.osd.10.smithi189.stdout:Mar 24 03:21:10 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-osd-10[61294]: 2024-03-24T03:21:10.892+0000 7f804e31f700 -1 osd.10 127 heartbeat_check: no reply from 172.21.15.193:6812 osd.5 since back 2024-03-24T03:20:41.750894+0000 front 2024-03-24T03:20:41.750949+0000 (oldest deadline 2024-03-24T03:21:04.050717+0000) 2024-03-24T03:21:11.461 INFO:journalctl@ceph.osd.11.smithi193.stdout:Mar 24 03:21:11 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-osd-11[62452]: 2024-03-24T03:21:11.017+0000 7f0edd521700 -1 osd.11 127 heartbeat_check: no reply from 172.21.15.118:6806 osd.0 since back 2024-03-24T03:19:59.095638+0000 front 2024-03-24T03:19:59.095627+0000 (oldest deadline 2024-03-24T03:20:20.195530+0000) 2024-03-24T03:21:11.461 INFO:journalctl@ceph.osd.11.smithi193.stdout:Mar 24 03:21:11 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-osd-11[62452]: 2024-03-24T03:21:11.017+0000 7f0edd521700 -1 osd.11 127 heartbeat_check: no reply from 172.21.15.189:6804 osd.1 since back 2024-03-24T03:20:07.399753+0000 front 2024-03-24T03:20:07.399812+0000 (oldest deadline 2024-03-24T03:20:32.099923+0000) 2024-03-24T03:21:11.461 INFO:journalctl@ceph.osd.11.smithi193.stdout:Mar 24 03:21:11 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-osd-11[62452]: 2024-03-24T03:21:11.017+0000 7f0edd521700 -1 osd.11 127 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-03-24T03:20:12.100299+0000 front 2024-03-24T03:20:12.100348+0000 (oldest deadline 2024-03-24T03:20:38.000441+0000) 2024-03-24T03:21:11.461 INFO:journalctl@ceph.osd.11.smithi193.stdout:Mar 24 03:21:11 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-osd-11[62452]: 2024-03-24T03:21:11.017+0000 7f0edd521700 -1 osd.11 127 heartbeat_check: no reply from 172.21.15.118:6814 osd.3 since back 2024-03-24T03:20:25.002506+0000 front 2024-03-24T03:20:25.002576+0000 (oldest deadline 2024-03-24T03:20:46.702390+0000) 2024-03-24T03:21:11.461 INFO:journalctl@ceph.osd.11.smithi193.stdout:Mar 24 03:21:11 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-osd-11[62452]: 2024-03-24T03:21:11.017+0000 7f0edd521700 -1 osd.11 127 heartbeat_check: no reply from 172.21.15.189:6812 osd.4 since back 2024-03-24T03:20:32.905105+0000 front 2024-03-24T03:20:32.905067+0000 (oldest deadline 2024-03-24T03:20:55.204998+0000) 2024-03-24T03:21:11.461 INFO:journalctl@ceph.osd.11.smithi193.stdout:Mar 24 03:21:11 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-osd-11[62452]: 2024-03-24T03:21:11.017+0000 7f0edd521700 -1 osd.11 127 heartbeat_check: no reply from 172.21.15.193:6812 osd.5 since back 2024-03-24T03:20:42.106727+0000 front 2024-03-24T03:20:42.106844+0000 (oldest deadline 2024-03-24T03:21:05.006783+0000) 2024-03-24T03:21:11.946 INFO:journalctl@ceph.osd.9.smithi118.stdout:Mar 24 03:21:11 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-osd-9[67027]: 2024-03-24T03:21:11.605+0000 7f69f7400700 -1 osd.9 127 heartbeat_check: no reply from 172.21.15.118:6806 osd.0 since back 2024-03-24T03:19:57.002010+0000 front 2024-03-24T03:19:57.002333+0000 (oldest deadline 2024-03-24T03:20:21.101732+0000) 2024-03-24T03:21:11.947 INFO:journalctl@ceph.osd.9.smithi118.stdout:Mar 24 03:21:11 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-osd-9[67027]: 2024-03-24T03:21:11.605+0000 7f69f7400700 -1 osd.9 127 heartbeat_check: no reply from 172.21.15.189:6804 osd.1 since back 2024-03-24T03:20:05.804520+0000 front 2024-03-24T03:20:05.804379+0000 (oldest deadline 2024-03-24T03:20:29.303978+0000) 2024-03-24T03:21:11.947 INFO:journalctl@ceph.osd.9.smithi118.stdout:Mar 24 03:21:11 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-osd-9[67027]: 2024-03-24T03:21:11.605+0000 7f69f7400700 -1 osd.9 127 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-03-24T03:20:14.507274+0000 front 2024-03-24T03:20:14.507231+0000 (oldest deadline 2024-03-24T03:20:38.607488+0000) 2024-03-24T03:21:11.947 INFO:journalctl@ceph.osd.9.smithi118.stdout:Mar 24 03:21:11 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-osd-9[67027]: 2024-03-24T03:21:11.605+0000 7f69f7400700 -1 osd.9 127 heartbeat_check: no reply from 172.21.15.118:6814 osd.3 since back 2024-03-24T03:20:23.111195+0000 front 2024-03-24T03:20:23.111494+0000 (oldest deadline 2024-03-24T03:20:48.410762+0000) 2024-03-24T03:21:11.947 INFO:journalctl@ceph.osd.9.smithi118.stdout:Mar 24 03:21:11 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-osd-9[67027]: 2024-03-24T03:21:11.605+0000 7f69f7400700 -1 osd.9 127 heartbeat_check: no reply from 172.21.15.189:6812 osd.4 since back 2024-03-24T03:20:31.213661+0000 front 2024-03-24T03:20:31.213727+0000 (oldest deadline 2024-03-24T03:20:55.913869+0000) 2024-03-24T03:21:11.947 INFO:journalctl@ceph.osd.9.smithi118.stdout:Mar 24 03:21:11 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-osd-9[67027]: 2024-03-24T03:21:11.605+0000 7f69f7400700 -1 osd.9 127 heartbeat_check: no reply from 172.21.15.193:6812 osd.5 since back 2024-03-24T03:20:41.617507+0000 front 2024-03-24T03:20:41.617436+0000 (oldest deadline 2024-03-24T03:21:03.917405+0000) 2024-03-24T03:21:12.204 INFO:journalctl@ceph.osd.10.smithi189.stdout:Mar 24 03:21:11 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-osd-10[61294]: 2024-03-24T03:21:11.937+0000 7f804e31f700 -1 osd.10 127 heartbeat_check: no reply from 172.21.15.118:6806 osd.0 since back 2024-03-24T03:19:56.942815+0000 front 2024-03-24T03:19:56.942951+0000 (oldest deadline 2024-03-24T03:20:22.842714+0000) 2024-03-24T03:21:12.204 INFO:journalctl@ceph.osd.10.smithi189.stdout:Mar 24 03:21:11 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-osd-10[61294]: 2024-03-24T03:21:11.937+0000 7f804e31f700 -1 osd.10 127 heartbeat_check: no reply from 172.21.15.189:6804 osd.1 since back 2024-03-24T03:20:07.543899+0000 front 2024-03-24T03:20:07.543880+0000 (oldest deadline 2024-03-24T03:20:30.944770+0000) 2024-03-24T03:21:12.204 INFO:journalctl@ceph.osd.10.smithi189.stdout:Mar 24 03:21:11 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-osd-10[61294]: 2024-03-24T03:21:11.937+0000 7f804e31f700 -1 osd.10 127 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-03-24T03:20:15.546606+0000 front 2024-03-24T03:20:15.546459+0000 (oldest deadline 2024-03-24T03:20:37.846574+0000) 2024-03-24T03:21:12.204 INFO:journalctl@ceph.osd.10.smithi189.stdout:Mar 24 03:21:11 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-osd-10[61294]: 2024-03-24T03:21:11.937+0000 7f804e31f700 -1 osd.10 127 heartbeat_check: no reply from 172.21.15.118:6814 osd.3 since back 2024-03-24T03:20:22.547814+0000 front 2024-03-24T03:20:22.547848+0000 (oldest deadline 2024-03-24T03:20:47.748188+0000) 2024-03-24T03:21:12.204 INFO:journalctl@ceph.osd.10.smithi189.stdout:Mar 24 03:21:11 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-osd-10[61294]: 2024-03-24T03:21:11.937+0000 7f804e31f700 -1 osd.10 127 heartbeat_check: no reply from 172.21.15.189:6812 osd.4 since back 2024-03-24T03:20:33.049108+0000 front 2024-03-24T03:20:33.049154+0000 (oldest deadline 2024-03-24T03:20:55.949290+0000) 2024-03-24T03:21:12.204 INFO:journalctl@ceph.osd.10.smithi189.stdout:Mar 24 03:21:11 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-osd-10[61294]: 2024-03-24T03:21:11.937+0000 7f804e31f700 -1 osd.10 127 heartbeat_check: no reply from 172.21.15.193:6812 osd.5 since back 2024-03-24T03:20:41.750894+0000 front 2024-03-24T03:20:41.750949+0000 (oldest deadline 2024-03-24T03:21:04.050717+0000) 2024-03-24T03:21:12.460 INFO:journalctl@ceph.osd.11.smithi193.stdout:Mar 24 03:21:12 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-osd-11[62452]: 2024-03-24T03:21:12.016+0000 7f0edd521700 -1 osd.11 127 heartbeat_check: no reply from 172.21.15.118:6806 osd.0 since back 2024-03-24T03:19:59.095638+0000 front 2024-03-24T03:19:59.095627+0000 (oldest deadline 2024-03-24T03:20:20.195530+0000) 2024-03-24T03:21:12.461 INFO:journalctl@ceph.osd.11.smithi193.stdout:Mar 24 03:21:12 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-osd-11[62452]: 2024-03-24T03:21:12.016+0000 7f0edd521700 -1 osd.11 127 heartbeat_check: no reply from 172.21.15.189:6804 osd.1 since back 2024-03-24T03:20:07.399753+0000 front 2024-03-24T03:20:07.399812+0000 (oldest deadline 2024-03-24T03:20:32.099923+0000) 2024-03-24T03:21:12.461 INFO:journalctl@ceph.osd.11.smithi193.stdout:Mar 24 03:21:12 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-osd-11[62452]: 2024-03-24T03:21:12.016+0000 7f0edd521700 -1 osd.11 127 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-03-24T03:20:12.100299+0000 front 2024-03-24T03:20:12.100348+0000 (oldest deadline 2024-03-24T03:20:38.000441+0000) 2024-03-24T03:21:12.461 INFO:journalctl@ceph.osd.11.smithi193.stdout:Mar 24 03:21:12 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-osd-11[62452]: 2024-03-24T03:21:12.016+0000 7f0edd521700 -1 osd.11 127 heartbeat_check: no reply from 172.21.15.118:6814 osd.3 since back 2024-03-24T03:20:25.002506+0000 front 2024-03-24T03:20:25.002576+0000 (oldest deadline 2024-03-24T03:20:46.702390+0000) 2024-03-24T03:21:12.461 INFO:journalctl@ceph.osd.11.smithi193.stdout:Mar 24 03:21:12 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-osd-11[62452]: 2024-03-24T03:21:12.016+0000 7f0edd521700 -1 osd.11 127 heartbeat_check: no reply from 172.21.15.189:6812 osd.4 since back 2024-03-24T03:20:32.905105+0000 front 2024-03-24T03:20:32.905067+0000 (oldest deadline 2024-03-24T03:20:55.204998+0000) 2024-03-24T03:21:12.461 INFO:journalctl@ceph.osd.11.smithi193.stdout:Mar 24 03:21:12 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-osd-11[62452]: 2024-03-24T03:21:12.016+0000 7f0edd521700 -1 osd.11 127 heartbeat_check: no reply from 172.21.15.193:6812 osd.5 since back 2024-03-24T03:20:42.106727+0000 front 2024-03-24T03:20:42.106844+0000 (oldest deadline 2024-03-24T03:21:05.006783+0000) 2024-03-24T03:21:12.946 INFO:journalctl@ceph.osd.9.smithi118.stdout:Mar 24 03:21:12 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-osd-9[67027]: 2024-03-24T03:21:12.644+0000 7f69f7400700 -1 osd.9 127 heartbeat_check: no reply from 172.21.15.118:6806 osd.0 since back 2024-03-24T03:19:57.002010+0000 front 2024-03-24T03:19:57.002333+0000 (oldest deadline 2024-03-24T03:20:21.101732+0000) 2024-03-24T03:21:12.946 INFO:journalctl@ceph.osd.9.smithi118.stdout:Mar 24 03:21:12 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-osd-9[67027]: 2024-03-24T03:21:12.644+0000 7f69f7400700 -1 osd.9 127 heartbeat_check: no reply from 172.21.15.189:6804 osd.1 since back 2024-03-24T03:20:05.804520+0000 front 2024-03-24T03:20:05.804379+0000 (oldest deadline 2024-03-24T03:20:29.303978+0000) 2024-03-24T03:21:12.946 INFO:journalctl@ceph.osd.9.smithi118.stdout:Mar 24 03:21:12 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-osd-9[67027]: 2024-03-24T03:21:12.644+0000 7f69f7400700 -1 osd.9 127 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-03-24T03:20:14.507274+0000 front 2024-03-24T03:20:14.507231+0000 (oldest deadline 2024-03-24T03:20:38.607488+0000) 2024-03-24T03:21:12.947 INFO:journalctl@ceph.osd.9.smithi118.stdout:Mar 24 03:21:12 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-osd-9[67027]: 2024-03-24T03:21:12.644+0000 7f69f7400700 -1 osd.9 127 heartbeat_check: no reply from 172.21.15.118:6814 osd.3 since back 2024-03-24T03:20:23.111195+0000 front 2024-03-24T03:20:23.111494+0000 (oldest deadline 2024-03-24T03:20:48.410762+0000) 2024-03-24T03:21:12.947 INFO:journalctl@ceph.osd.9.smithi118.stdout:Mar 24 03:21:12 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-osd-9[67027]: 2024-03-24T03:21:12.644+0000 7f69f7400700 -1 osd.9 127 heartbeat_check: no reply from 172.21.15.189:6812 osd.4 since back 2024-03-24T03:20:31.213661+0000 front 2024-03-24T03:20:31.213727+0000 (oldest deadline 2024-03-24T03:20:55.913869+0000) 2024-03-24T03:21:12.947 INFO:journalctl@ceph.osd.9.smithi118.stdout:Mar 24 03:21:12 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-osd-9[67027]: 2024-03-24T03:21:12.644+0000 7f69f7400700 -1 osd.9 127 heartbeat_check: no reply from 172.21.15.193:6812 osd.5 since back 2024-03-24T03:20:41.617507+0000 front 2024-03-24T03:20:41.617436+0000 (oldest deadline 2024-03-24T03:21:03.917405+0000) 2024-03-24T03:21:13.204 INFO:journalctl@ceph.osd.10.smithi189.stdout:Mar 24 03:21:12 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-osd-10[61294]: 2024-03-24T03:21:12.895+0000 7f804e31f700 -1 osd.10 127 heartbeat_check: no reply from 172.21.15.118:6806 osd.0 since back 2024-03-24T03:19:56.942815+0000 front 2024-03-24T03:19:56.942951+0000 (oldest deadline 2024-03-24T03:20:22.842714+0000) 2024-03-24T03:21:13.204 INFO:journalctl@ceph.osd.10.smithi189.stdout:Mar 24 03:21:12 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-osd-10[61294]: 2024-03-24T03:21:12.895+0000 7f804e31f700 -1 osd.10 127 heartbeat_check: no reply from 172.21.15.189:6804 osd.1 since back 2024-03-24T03:20:07.543899+0000 front 2024-03-24T03:20:07.543880+0000 (oldest deadline 2024-03-24T03:20:30.944770+0000) 2024-03-24T03:21:13.204 INFO:journalctl@ceph.osd.10.smithi189.stdout:Mar 24 03:21:12 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-osd-10[61294]: 2024-03-24T03:21:12.895+0000 7f804e31f700 -1 osd.10 127 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-03-24T03:20:15.546606+0000 front 2024-03-24T03:20:15.546459+0000 (oldest deadline 2024-03-24T03:20:37.846574+0000) 2024-03-24T03:21:13.204 INFO:journalctl@ceph.osd.10.smithi189.stdout:Mar 24 03:21:12 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-osd-10[61294]: 2024-03-24T03:21:12.895+0000 7f804e31f700 -1 osd.10 127 heartbeat_check: no reply from 172.21.15.118:6814 osd.3 since back 2024-03-24T03:20:22.547814+0000 front 2024-03-24T03:20:22.547848+0000 (oldest deadline 2024-03-24T03:20:47.748188+0000) 2024-03-24T03:21:13.205 INFO:journalctl@ceph.osd.10.smithi189.stdout:Mar 24 03:21:12 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-osd-10[61294]: 2024-03-24T03:21:12.895+0000 7f804e31f700 -1 osd.10 127 heartbeat_check: no reply from 172.21.15.189:6812 osd.4 since back 2024-03-24T03:20:33.049108+0000 front 2024-03-24T03:20:33.049154+0000 (oldest deadline 2024-03-24T03:20:55.949290+0000) 2024-03-24T03:21:13.205 INFO:journalctl@ceph.osd.10.smithi189.stdout:Mar 24 03:21:12 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-osd-10[61294]: 2024-03-24T03:21:12.895+0000 7f804e31f700 -1 osd.10 127 heartbeat_check: no reply from 172.21.15.193:6812 osd.5 since back 2024-03-24T03:20:41.750894+0000 front 2024-03-24T03:20:41.750949+0000 (oldest deadline 2024-03-24T03:21:04.050717+0000) 2024-03-24T03:21:13.205 INFO:journalctl@ceph.osd.10.smithi189.stdout:Mar 24 03:21:12 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-osd-10[61294]: 2024-03-24T03:21:12.895+0000 7f804e31f700 -1 osd.10 127 heartbeat_check: no reply from 172.21.15.118:6822 osd.6 since back 2024-03-24T03:20:50.452401+0000 front 2024-03-24T03:20:50.452443+0000 (oldest deadline 2024-03-24T03:21:12.752167+0000) 2024-03-24T03:21:13.460 INFO:journalctl@ceph.osd.11.smithi193.stdout:Mar 24 03:21:12 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-osd-11[62452]: 2024-03-24T03:21:12.969+0000 7f0edd521700 -1 osd.11 127 heartbeat_check: no reply from 172.21.15.118:6806 osd.0 since back 2024-03-24T03:19:59.095638+0000 front 2024-03-24T03:19:59.095627+0000 (oldest deadline 2024-03-24T03:20:20.195530+0000) 2024-03-24T03:21:13.461 INFO:journalctl@ceph.osd.11.smithi193.stdout:Mar 24 03:21:12 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-osd-11[62452]: 2024-03-24T03:21:12.969+0000 7f0edd521700 -1 osd.11 127 heartbeat_check: no reply from 172.21.15.189:6804 osd.1 since back 2024-03-24T03:20:07.399753+0000 front 2024-03-24T03:20:07.399812+0000 (oldest deadline 2024-03-24T03:20:32.099923+0000) 2024-03-24T03:21:13.461 INFO:journalctl@ceph.osd.11.smithi193.stdout:Mar 24 03:21:12 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-osd-11[62452]: 2024-03-24T03:21:12.969+0000 7f0edd521700 -1 osd.11 127 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-03-24T03:20:12.100299+0000 front 2024-03-24T03:20:12.100348+0000 (oldest deadline 2024-03-24T03:20:38.000441+0000) 2024-03-24T03:21:13.461 INFO:journalctl@ceph.osd.11.smithi193.stdout:Mar 24 03:21:12 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-osd-11[62452]: 2024-03-24T03:21:12.969+0000 7f0edd521700 -1 osd.11 127 heartbeat_check: no reply from 172.21.15.118:6814 osd.3 since back 2024-03-24T03:20:25.002506+0000 front 2024-03-24T03:20:25.002576+0000 (oldest deadline 2024-03-24T03:20:46.702390+0000) 2024-03-24T03:21:13.461 INFO:journalctl@ceph.osd.11.smithi193.stdout:Mar 24 03:21:12 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-osd-11[62452]: 2024-03-24T03:21:12.969+0000 7f0edd521700 -1 osd.11 127 heartbeat_check: no reply from 172.21.15.189:6812 osd.4 since back 2024-03-24T03:20:32.905105+0000 front 2024-03-24T03:20:32.905067+0000 (oldest deadline 2024-03-24T03:20:55.204998+0000) 2024-03-24T03:21:13.461 INFO:journalctl@ceph.osd.11.smithi193.stdout:Mar 24 03:21:12 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-osd-11[62452]: 2024-03-24T03:21:12.969+0000 7f0edd521700 -1 osd.11 127 heartbeat_check: no reply from 172.21.15.193:6812 osd.5 since back 2024-03-24T03:20:42.106727+0000 front 2024-03-24T03:20:42.106844+0000 (oldest deadline 2024-03-24T03:21:05.006783+0000) 2024-03-24T03:21:13.946 INFO:journalctl@ceph.osd.9.smithi118.stdout:Mar 24 03:21:13 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-osd-9[67027]: 2024-03-24T03:21:13.651+0000 7f69f7400700 -1 osd.9 127 heartbeat_check: no reply from 172.21.15.118:6806 osd.0 since back 2024-03-24T03:19:57.002010+0000 front 2024-03-24T03:19:57.002333+0000 (oldest deadline 2024-03-24T03:20:21.101732+0000) 2024-03-24T03:21:13.946 INFO:journalctl@ceph.osd.9.smithi118.stdout:Mar 24 03:21:13 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-osd-9[67027]: 2024-03-24T03:21:13.651+0000 7f69f7400700 -1 osd.9 127 heartbeat_check: no reply from 172.21.15.189:6804 osd.1 since back 2024-03-24T03:20:05.804520+0000 front 2024-03-24T03:20:05.804379+0000 (oldest deadline 2024-03-24T03:20:29.303978+0000) 2024-03-24T03:21:13.946 INFO:journalctl@ceph.osd.9.smithi118.stdout:Mar 24 03:21:13 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-osd-9[67027]: 2024-03-24T03:21:13.651+0000 7f69f7400700 -1 osd.9 127 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-03-24T03:20:14.507274+0000 front 2024-03-24T03:20:14.507231+0000 (oldest deadline 2024-03-24T03:20:38.607488+0000) 2024-03-24T03:21:13.946 INFO:journalctl@ceph.osd.9.smithi118.stdout:Mar 24 03:21:13 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-osd-9[67027]: 2024-03-24T03:21:13.651+0000 7f69f7400700 -1 osd.9 127 heartbeat_check: no reply from 172.21.15.118:6814 osd.3 since back 2024-03-24T03:20:23.111195+0000 front 2024-03-24T03:20:23.111494+0000 (oldest deadline 2024-03-24T03:20:48.410762+0000) 2024-03-24T03:21:13.947 INFO:journalctl@ceph.osd.9.smithi118.stdout:Mar 24 03:21:13 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-osd-9[67027]: 2024-03-24T03:21:13.651+0000 7f69f7400700 -1 osd.9 127 heartbeat_check: no reply from 172.21.15.189:6812 osd.4 since back 2024-03-24T03:20:31.213661+0000 front 2024-03-24T03:20:31.213727+0000 (oldest deadline 2024-03-24T03:20:55.913869+0000) 2024-03-24T03:21:13.947 INFO:journalctl@ceph.osd.9.smithi118.stdout:Mar 24 03:21:13 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-osd-9[67027]: 2024-03-24T03:21:13.651+0000 7f69f7400700 -1 osd.9 127 heartbeat_check: no reply from 172.21.15.193:6812 osd.5 since back 2024-03-24T03:20:41.617507+0000 front 2024-03-24T03:20:41.617436+0000 (oldest deadline 2024-03-24T03:21:03.917405+0000) 2024-03-24T03:21:14.203 INFO:journalctl@ceph.osd.10.smithi189.stdout:Mar 24 03:21:13 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-osd-10[61294]: 2024-03-24T03:21:13.873+0000 7f804e31f700 -1 osd.10 127 heartbeat_check: no reply from 172.21.15.118:6806 osd.0 since back 2024-03-24T03:19:56.942815+0000 front 2024-03-24T03:19:56.942951+0000 (oldest deadline 2024-03-24T03:20:22.842714+0000) 2024-03-24T03:21:14.204 INFO:journalctl@ceph.osd.10.smithi189.stdout:Mar 24 03:21:13 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-osd-10[61294]: 2024-03-24T03:21:13.873+0000 7f804e31f700 -1 osd.10 127 heartbeat_check: no reply from 172.21.15.189:6804 osd.1 since back 2024-03-24T03:20:07.543899+0000 front 2024-03-24T03:20:07.543880+0000 (oldest deadline 2024-03-24T03:20:30.944770+0000) 2024-03-24T03:21:14.204 INFO:journalctl@ceph.osd.10.smithi189.stdout:Mar 24 03:21:13 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-osd-10[61294]: 2024-03-24T03:21:13.873+0000 7f804e31f700 -1 osd.10 127 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-03-24T03:20:15.546606+0000 front 2024-03-24T03:20:15.546459+0000 (oldest deadline 2024-03-24T03:20:37.846574+0000) 2024-03-24T03:21:14.204 INFO:journalctl@ceph.osd.10.smithi189.stdout:Mar 24 03:21:13 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-osd-10[61294]: 2024-03-24T03:21:13.873+0000 7f804e31f700 -1 osd.10 127 heartbeat_check: no reply from 172.21.15.118:6814 osd.3 since back 2024-03-24T03:20:22.547814+0000 front 2024-03-24T03:20:22.547848+0000 (oldest deadline 2024-03-24T03:20:47.748188+0000) 2024-03-24T03:21:14.204 INFO:journalctl@ceph.osd.10.smithi189.stdout:Mar 24 03:21:13 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-osd-10[61294]: 2024-03-24T03:21:13.873+0000 7f804e31f700 -1 osd.10 127 heartbeat_check: no reply from 172.21.15.189:6812 osd.4 since back 2024-03-24T03:20:33.049108+0000 front 2024-03-24T03:20:33.049154+0000 (oldest deadline 2024-03-24T03:20:55.949290+0000) 2024-03-24T03:21:14.204 INFO:journalctl@ceph.osd.10.smithi189.stdout:Mar 24 03:21:13 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-osd-10[61294]: 2024-03-24T03:21:13.873+0000 7f804e31f700 -1 osd.10 127 heartbeat_check: no reply from 172.21.15.193:6812 osd.5 since back 2024-03-24T03:20:41.750894+0000 front 2024-03-24T03:20:41.750949+0000 (oldest deadline 2024-03-24T03:21:04.050717+0000) 2024-03-24T03:21:14.204 INFO:journalctl@ceph.osd.10.smithi189.stdout:Mar 24 03:21:13 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-osd-10[61294]: 2024-03-24T03:21:13.873+0000 7f804e31f700 -1 osd.10 127 heartbeat_check: no reply from 172.21.15.118:6822 osd.6 since back 2024-03-24T03:20:50.452401+0000 front 2024-03-24T03:20:50.452443+0000 (oldest deadline 2024-03-24T03:21:12.752167+0000) 2024-03-24T03:21:14.210 INFO:journalctl@ceph.osd.11.smithi193.stdout:Mar 24 03:21:13 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-osd-11[62452]: 2024-03-24T03:21:13.929+0000 7f0edd521700 -1 osd.11 127 heartbeat_check: no reply from 172.21.15.118:6806 osd.0 since back 2024-03-24T03:19:59.095638+0000 front 2024-03-24T03:19:59.095627+0000 (oldest deadline 2024-03-24T03:20:20.195530+0000) 2024-03-24T03:21:14.210 INFO:journalctl@ceph.osd.11.smithi193.stdout:Mar 24 03:21:13 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-osd-11[62452]: 2024-03-24T03:21:13.929+0000 7f0edd521700 -1 osd.11 127 heartbeat_check: no reply from 172.21.15.189:6804 osd.1 since back 2024-03-24T03:20:07.399753+0000 front 2024-03-24T03:20:07.399812+0000 (oldest deadline 2024-03-24T03:20:32.099923+0000) 2024-03-24T03:21:14.211 INFO:journalctl@ceph.osd.11.smithi193.stdout:Mar 24 03:21:13 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-osd-11[62452]: 2024-03-24T03:21:13.929+0000 7f0edd521700 -1 osd.11 127 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-03-24T03:20:12.100299+0000 front 2024-03-24T03:20:12.100348+0000 (oldest deadline 2024-03-24T03:20:38.000441+0000) 2024-03-24T03:21:14.211 INFO:journalctl@ceph.osd.11.smithi193.stdout:Mar 24 03:21:13 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-osd-11[62452]: 2024-03-24T03:21:13.929+0000 7f0edd521700 -1 osd.11 127 heartbeat_check: no reply from 172.21.15.118:6814 osd.3 since back 2024-03-24T03:20:25.002506+0000 front 2024-03-24T03:20:25.002576+0000 (oldest deadline 2024-03-24T03:20:46.702390+0000) 2024-03-24T03:21:14.211 INFO:journalctl@ceph.osd.11.smithi193.stdout:Mar 24 03:21:13 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-osd-11[62452]: 2024-03-24T03:21:13.929+0000 7f0edd521700 -1 osd.11 127 heartbeat_check: no reply from 172.21.15.189:6812 osd.4 since back 2024-03-24T03:20:32.905105+0000 front 2024-03-24T03:20:32.905067+0000 (oldest deadline 2024-03-24T03:20:55.204998+0000) 2024-03-24T03:21:14.211 INFO:journalctl@ceph.osd.11.smithi193.stdout:Mar 24 03:21:13 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-osd-11[62452]: 2024-03-24T03:21:13.929+0000 7f0edd521700 -1 osd.11 127 heartbeat_check: no reply from 172.21.15.193:6812 osd.5 since back 2024-03-24T03:20:42.106727+0000 front 2024-03-24T03:20:42.106844+0000 (oldest deadline 2024-03-24T03:21:05.006783+0000) 2024-03-24T03:21:14.946 INFO:journalctl@ceph.osd.9.smithi118.stdout:Mar 24 03:21:14 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-osd-9[67027]: 2024-03-24T03:21:14.656+0000 7f69f7400700 -1 osd.9 127 heartbeat_check: no reply from 172.21.15.118:6806 osd.0 since back 2024-03-24T03:19:57.002010+0000 front 2024-03-24T03:19:57.002333+0000 (oldest deadline 2024-03-24T03:20:21.101732+0000) 2024-03-24T03:21:14.946 INFO:journalctl@ceph.osd.9.smithi118.stdout:Mar 24 03:21:14 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-osd-9[67027]: 2024-03-24T03:21:14.656+0000 7f69f7400700 -1 osd.9 127 heartbeat_check: no reply from 172.21.15.189:6804 osd.1 since back 2024-03-24T03:20:05.804520+0000 front 2024-03-24T03:20:05.804379+0000 (oldest deadline 2024-03-24T03:20:29.303978+0000) 2024-03-24T03:21:14.947 INFO:journalctl@ceph.osd.9.smithi118.stdout:Mar 24 03:21:14 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-osd-9[67027]: 2024-03-24T03:21:14.656+0000 7f69f7400700 -1 osd.9 127 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-03-24T03:20:14.507274+0000 front 2024-03-24T03:20:14.507231+0000 (oldest deadline 2024-03-24T03:20:38.607488+0000) 2024-03-24T03:21:14.947 INFO:journalctl@ceph.osd.9.smithi118.stdout:Mar 24 03:21:14 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-osd-9[67027]: 2024-03-24T03:21:14.656+0000 7f69f7400700 -1 osd.9 127 heartbeat_check: no reply from 172.21.15.118:6814 osd.3 since back 2024-03-24T03:20:23.111195+0000 front 2024-03-24T03:20:23.111494+0000 (oldest deadline 2024-03-24T03:20:48.410762+0000) 2024-03-24T03:21:14.947 INFO:journalctl@ceph.osd.9.smithi118.stdout:Mar 24 03:21:14 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-osd-9[67027]: 2024-03-24T03:21:14.656+0000 7f69f7400700 -1 osd.9 127 heartbeat_check: no reply from 172.21.15.189:6812 osd.4 since back 2024-03-24T03:20:31.213661+0000 front 2024-03-24T03:20:31.213727+0000 (oldest deadline 2024-03-24T03:20:55.913869+0000) 2024-03-24T03:21:14.947 INFO:journalctl@ceph.osd.9.smithi118.stdout:Mar 24 03:21:14 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-osd-9[67027]: 2024-03-24T03:21:14.656+0000 7f69f7400700 -1 osd.9 127 heartbeat_check: no reply from 172.21.15.193:6812 osd.5 since back 2024-03-24T03:20:41.617507+0000 front 2024-03-24T03:20:41.617436+0000 (oldest deadline 2024-03-24T03:21:03.917405+0000) 2024-03-24T03:21:14.947 INFO:journalctl@ceph.osd.9.smithi118.stdout:Mar 24 03:21:14 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-osd-9[67027]: 2024-03-24T03:21:14.656+0000 7f69f7400700 -1 osd.9 127 heartbeat_check: no reply from 172.21.15.118:6822 osd.6 since back 2024-03-24T03:20:49.621193+0000 front 2024-03-24T03:20:49.621068+0000 (oldest deadline 2024-03-24T03:21:13.721063+0000) 2024-03-24T03:21:15.203 INFO:journalctl@ceph.osd.10.smithi189.stdout:Mar 24 03:21:14 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-osd-10[61294]: 2024-03-24T03:21:14.827+0000 7f804e31f700 -1 osd.10 127 heartbeat_check: no reply from 172.21.15.118:6806 osd.0 since back 2024-03-24T03:19:56.942815+0000 front 2024-03-24T03:19:56.942951+0000 (oldest deadline 2024-03-24T03:20:22.842714+0000) 2024-03-24T03:21:15.204 INFO:journalctl@ceph.osd.10.smithi189.stdout:Mar 24 03:21:14 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-osd-10[61294]: 2024-03-24T03:21:14.827+0000 7f804e31f700 -1 osd.10 127 heartbeat_check: no reply from 172.21.15.189:6804 osd.1 since back 2024-03-24T03:20:07.543899+0000 front 2024-03-24T03:20:07.543880+0000 (oldest deadline 2024-03-24T03:20:30.944770+0000) 2024-03-24T03:21:15.204 INFO:journalctl@ceph.osd.10.smithi189.stdout:Mar 24 03:21:14 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-osd-10[61294]: 2024-03-24T03:21:14.827+0000 7f804e31f700 -1 osd.10 127 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-03-24T03:20:15.546606+0000 front 2024-03-24T03:20:15.546459+0000 (oldest deadline 2024-03-24T03:20:37.846574+0000) 2024-03-24T03:21:15.204 INFO:journalctl@ceph.osd.10.smithi189.stdout:Mar 24 03:21:14 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-osd-10[61294]: 2024-03-24T03:21:14.827+0000 7f804e31f700 -1 osd.10 127 heartbeat_check: no reply from 172.21.15.118:6814 osd.3 since back 2024-03-24T03:20:22.547814+0000 front 2024-03-24T03:20:22.547848+0000 (oldest deadline 2024-03-24T03:20:47.748188+0000) 2024-03-24T03:21:15.204 INFO:journalctl@ceph.osd.10.smithi189.stdout:Mar 24 03:21:14 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-osd-10[61294]: 2024-03-24T03:21:14.827+0000 7f804e31f700 -1 osd.10 127 heartbeat_check: no reply from 172.21.15.189:6812 osd.4 since back 2024-03-24T03:20:33.049108+0000 front 2024-03-24T03:20:33.049154+0000 (oldest deadline 2024-03-24T03:20:55.949290+0000) 2024-03-24T03:21:15.204 INFO:journalctl@ceph.osd.10.smithi189.stdout:Mar 24 03:21:14 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-osd-10[61294]: 2024-03-24T03:21:14.827+0000 7f804e31f700 -1 osd.10 127 heartbeat_check: no reply from 172.21.15.193:6812 osd.5 since back 2024-03-24T03:20:41.750894+0000 front 2024-03-24T03:20:41.750949+0000 (oldest deadline 2024-03-24T03:21:04.050717+0000) 2024-03-24T03:21:15.204 INFO:journalctl@ceph.osd.10.smithi189.stdout:Mar 24 03:21:14 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-osd-10[61294]: 2024-03-24T03:21:14.827+0000 7f804e31f700 -1 osd.10 127 heartbeat_check: no reply from 172.21.15.118:6822 osd.6 since back 2024-03-24T03:20:50.452401+0000 front 2024-03-24T03:20:50.452443+0000 (oldest deadline 2024-03-24T03:21:12.752167+0000) 2024-03-24T03:21:15.210 INFO:journalctl@ceph.osd.11.smithi193.stdout:Mar 24 03:21:14 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-osd-11[62452]: 2024-03-24T03:21:14.933+0000 7f0edd521700 -1 osd.11 127 heartbeat_check: no reply from 172.21.15.118:6806 osd.0 since back 2024-03-24T03:19:59.095638+0000 front 2024-03-24T03:19:59.095627+0000 (oldest deadline 2024-03-24T03:20:20.195530+0000) 2024-03-24T03:21:15.211 INFO:journalctl@ceph.osd.11.smithi193.stdout:Mar 24 03:21:14 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-osd-11[62452]: 2024-03-24T03:21:14.933+0000 7f0edd521700 -1 osd.11 127 heartbeat_check: no reply from 172.21.15.189:6804 osd.1 since back 2024-03-24T03:20:07.399753+0000 front 2024-03-24T03:20:07.399812+0000 (oldest deadline 2024-03-24T03:20:32.099923+0000) 2024-03-24T03:21:15.211 INFO:journalctl@ceph.osd.11.smithi193.stdout:Mar 24 03:21:14 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-osd-11[62452]: 2024-03-24T03:21:14.933+0000 7f0edd521700 -1 osd.11 127 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-03-24T03:20:12.100299+0000 front 2024-03-24T03:20:12.100348+0000 (oldest deadline 2024-03-24T03:20:38.000441+0000) 2024-03-24T03:21:15.211 INFO:journalctl@ceph.osd.11.smithi193.stdout:Mar 24 03:21:14 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-osd-11[62452]: 2024-03-24T03:21:14.933+0000 7f0edd521700 -1 osd.11 127 heartbeat_check: no reply from 172.21.15.118:6814 osd.3 since back 2024-03-24T03:20:25.002506+0000 front 2024-03-24T03:20:25.002576+0000 (oldest deadline 2024-03-24T03:20:46.702390+0000) 2024-03-24T03:21:15.211 INFO:journalctl@ceph.osd.11.smithi193.stdout:Mar 24 03:21:14 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-osd-11[62452]: 2024-03-24T03:21:14.933+0000 7f0edd521700 -1 osd.11 127 heartbeat_check: no reply from 172.21.15.189:6812 osd.4 since back 2024-03-24T03:20:32.905105+0000 front 2024-03-24T03:20:32.905067+0000 (oldest deadline 2024-03-24T03:20:55.204998+0000) 2024-03-24T03:21:15.211 INFO:journalctl@ceph.osd.11.smithi193.stdout:Mar 24 03:21:14 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-osd-11[62452]: 2024-03-24T03:21:14.933+0000 7f0edd521700 -1 osd.11 127 heartbeat_check: no reply from 172.21.15.193:6812 osd.5 since back 2024-03-24T03:20:42.106727+0000 front 2024-03-24T03:20:42.106844+0000 (oldest deadline 2024-03-24T03:21:05.006783+0000) 2024-03-24T03:21:16.204 INFO:journalctl@ceph.osd.10.smithi189.stdout:Mar 24 03:21:15 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-osd-10[61294]: 2024-03-24T03:21:15.781+0000 7f804e31f700 -1 osd.10 127 heartbeat_check: no reply from 172.21.15.118:6806 osd.0 since back 2024-03-24T03:19:56.942815+0000 front 2024-03-24T03:19:56.942951+0000 (oldest deadline 2024-03-24T03:20:22.842714+0000) 2024-03-24T03:21:16.204 INFO:journalctl@ceph.osd.10.smithi189.stdout:Mar 24 03:21:15 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-osd-10[61294]: 2024-03-24T03:21:15.781+0000 7f804e31f700 -1 osd.10 127 heartbeat_check: no reply from 172.21.15.189:6804 osd.1 since back 2024-03-24T03:20:07.543899+0000 front 2024-03-24T03:20:07.543880+0000 (oldest deadline 2024-03-24T03:20:30.944770+0000) 2024-03-24T03:21:16.204 INFO:journalctl@ceph.osd.10.smithi189.stdout:Mar 24 03:21:15 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-osd-10[61294]: 2024-03-24T03:21:15.781+0000 7f804e31f700 -1 osd.10 127 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-03-24T03:20:15.546606+0000 front 2024-03-24T03:20:15.546459+0000 (oldest deadline 2024-03-24T03:20:37.846574+0000) 2024-03-24T03:21:16.204 INFO:journalctl@ceph.osd.10.smithi189.stdout:Mar 24 03:21:15 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-osd-10[61294]: 2024-03-24T03:21:15.781+0000 7f804e31f700 -1 osd.10 127 heartbeat_check: no reply from 172.21.15.118:6814 osd.3 since back 2024-03-24T03:20:22.547814+0000 front 2024-03-24T03:20:22.547848+0000 (oldest deadline 2024-03-24T03:20:47.748188+0000) 2024-03-24T03:21:16.204 INFO:journalctl@ceph.osd.10.smithi189.stdout:Mar 24 03:21:15 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-osd-10[61294]: 2024-03-24T03:21:15.781+0000 7f804e31f700 -1 osd.10 127 heartbeat_check: no reply from 172.21.15.189:6812 osd.4 since back 2024-03-24T03:20:33.049108+0000 front 2024-03-24T03:20:33.049154+0000 (oldest deadline 2024-03-24T03:20:55.949290+0000) 2024-03-24T03:21:16.204 INFO:journalctl@ceph.osd.10.smithi189.stdout:Mar 24 03:21:15 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-osd-10[61294]: 2024-03-24T03:21:15.781+0000 7f804e31f700 -1 osd.10 127 heartbeat_check: no reply from 172.21.15.193:6812 osd.5 since back 2024-03-24T03:20:41.750894+0000 front 2024-03-24T03:20:41.750949+0000 (oldest deadline 2024-03-24T03:21:04.050717+0000) 2024-03-24T03:21:16.204 INFO:journalctl@ceph.osd.10.smithi189.stdout:Mar 24 03:21:15 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-osd-10[61294]: 2024-03-24T03:21:15.781+0000 7f804e31f700 -1 osd.10 127 heartbeat_check: no reply from 172.21.15.118:6822 osd.6 since back 2024-03-24T03:20:50.452401+0000 front 2024-03-24T03:20:50.452443+0000 (oldest deadline 2024-03-24T03:21:12.752167+0000) 2024-03-24T03:21:16.266 INFO:journalctl@ceph.osd.9.smithi118.stdout:Mar 24 03:21:15 smithi118 podman[146489]: 2024-03-24 03:21:15.977552345 +0000 UTC m=+5.697822434 container died f7f5e9ce5086ce77f4319b6bdf58f36b6438fdd9fd09d18cf5d77d36b07e1fb8 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-d79a5420-e971-11ee-95cd-87774f69a715-osd-9, org.label-schema.build-date=20240319, org.label-schema.schema-version=1.0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, GIT_CLEAN=True, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, ceph=True, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=) 2024-03-24T03:21:16.461 INFO:journalctl@ceph.osd.11.smithi193.stdout:Mar 24 03:21:15 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-osd-11[62452]: 2024-03-24T03:21:15.962+0000 7f0edd521700 -1 osd.11 127 heartbeat_check: no reply from 172.21.15.118:6806 osd.0 since back 2024-03-24T03:19:59.095638+0000 front 2024-03-24T03:19:59.095627+0000 (oldest deadline 2024-03-24T03:20:20.195530+0000) 2024-03-24T03:21:16.461 INFO:journalctl@ceph.osd.11.smithi193.stdout:Mar 24 03:21:15 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-osd-11[62452]: 2024-03-24T03:21:15.962+0000 7f0edd521700 -1 osd.11 127 heartbeat_check: no reply from 172.21.15.189:6804 osd.1 since back 2024-03-24T03:20:07.399753+0000 front 2024-03-24T03:20:07.399812+0000 (oldest deadline 2024-03-24T03:20:32.099923+0000) 2024-03-24T03:21:16.461 INFO:journalctl@ceph.osd.11.smithi193.stdout:Mar 24 03:21:15 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-osd-11[62452]: 2024-03-24T03:21:15.962+0000 7f0edd521700 -1 osd.11 127 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-03-24T03:20:12.100299+0000 front 2024-03-24T03:20:12.100348+0000 (oldest deadline 2024-03-24T03:20:38.000441+0000) 2024-03-24T03:21:16.461 INFO:journalctl@ceph.osd.11.smithi193.stdout:Mar 24 03:21:15 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-osd-11[62452]: 2024-03-24T03:21:15.962+0000 7f0edd521700 -1 osd.11 127 heartbeat_check: no reply from 172.21.15.118:6814 osd.3 since back 2024-03-24T03:20:25.002506+0000 front 2024-03-24T03:20:25.002576+0000 (oldest deadline 2024-03-24T03:20:46.702390+0000) 2024-03-24T03:21:16.461 INFO:journalctl@ceph.osd.11.smithi193.stdout:Mar 24 03:21:15 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-osd-11[62452]: 2024-03-24T03:21:15.962+0000 7f0edd521700 -1 osd.11 127 heartbeat_check: no reply from 172.21.15.189:6812 osd.4 since back 2024-03-24T03:20:32.905105+0000 front 2024-03-24T03:20:32.905067+0000 (oldest deadline 2024-03-24T03:20:55.204998+0000) 2024-03-24T03:21:16.461 INFO:journalctl@ceph.osd.11.smithi193.stdout:Mar 24 03:21:15 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-osd-11[62452]: 2024-03-24T03:21:15.962+0000 7f0edd521700 -1 osd.11 127 heartbeat_check: no reply from 172.21.15.193:6812 osd.5 since back 2024-03-24T03:20:42.106727+0000 front 2024-03-24T03:20:42.106844+0000 (oldest deadline 2024-03-24T03:21:05.006783+0000) 2024-03-24T03:21:16.545 INFO:journalctl@ceph.osd.9.smithi118.stdout:Mar 24 03:21:16 smithi118 podman[146489]: 2024-03-24 03:21:16.368541354 +0000 UTC m=+6.088811469 container cleanup f7f5e9ce5086ce77f4319b6bdf58f36b6438fdd9fd09d18cf5d77d36b07e1fb8 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-d79a5420-e971-11ee-95cd-87774f69a715-osd-9, org.label-schema.build-date=20240319, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux ) 2024-03-24T03:21:16.545 INFO:journalctl@ceph.osd.9.smithi118.stdout:Mar 24 03:21:16 smithi118 bash[146489]: ceph-d79a5420-e971-11ee-95cd-87774f69a715-osd-9 2024-03-24T03:21:16.946 INFO:journalctl@ceph.osd.9.smithi118.stdout:Mar 24 03:21:16 smithi118 podman[146508]: 2024-03-24 03:21:16.544593246 +0000 UTC m=+0.559733265 container remove f7f5e9ce5086ce77f4319b6bdf58f36b6438fdd9fd09d18cf5d77d36b07e1fb8 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-d79a5420-e971-11ee-95cd-87774f69a715-osd-9, org.label-schema.build-date=20240319, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.schema-version=1.0, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30) 2024-03-24T03:21:17.204 INFO:journalctl@ceph.osd.10.smithi189.stdout:Mar 24 03:21:16 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-osd-10[61294]: 2024-03-24T03:21:16.769+0000 7f804e31f700 -1 osd.10 127 heartbeat_check: no reply from 172.21.15.118:6806 osd.0 since back 2024-03-24T03:19:56.942815+0000 front 2024-03-24T03:19:56.942951+0000 (oldest deadline 2024-03-24T03:20:22.842714+0000) 2024-03-24T03:21:17.204 INFO:journalctl@ceph.osd.10.smithi189.stdout:Mar 24 03:21:16 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-osd-10[61294]: 2024-03-24T03:21:16.769+0000 7f804e31f700 -1 osd.10 127 heartbeat_check: no reply from 172.21.15.189:6804 osd.1 since back 2024-03-24T03:20:07.543899+0000 front 2024-03-24T03:20:07.543880+0000 (oldest deadline 2024-03-24T03:20:30.944770+0000) 2024-03-24T03:21:17.204 INFO:journalctl@ceph.osd.10.smithi189.stdout:Mar 24 03:21:16 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-osd-10[61294]: 2024-03-24T03:21:16.769+0000 7f804e31f700 -1 osd.10 127 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-03-24T03:20:15.546606+0000 front 2024-03-24T03:20:15.546459+0000 (oldest deadline 2024-03-24T03:20:37.846574+0000) 2024-03-24T03:21:17.204 INFO:journalctl@ceph.osd.10.smithi189.stdout:Mar 24 03:21:16 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-osd-10[61294]: 2024-03-24T03:21:16.769+0000 7f804e31f700 -1 osd.10 127 heartbeat_check: no reply from 172.21.15.118:6814 osd.3 since back 2024-03-24T03:20:22.547814+0000 front 2024-03-24T03:20:22.547848+0000 (oldest deadline 2024-03-24T03:20:47.748188+0000) 2024-03-24T03:21:17.205 INFO:journalctl@ceph.osd.10.smithi189.stdout:Mar 24 03:21:16 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-osd-10[61294]: 2024-03-24T03:21:16.769+0000 7f804e31f700 -1 osd.10 127 heartbeat_check: no reply from 172.21.15.189:6812 osd.4 since back 2024-03-24T03:20:33.049108+0000 front 2024-03-24T03:20:33.049154+0000 (oldest deadline 2024-03-24T03:20:55.949290+0000) 2024-03-24T03:21:17.205 INFO:journalctl@ceph.osd.10.smithi189.stdout:Mar 24 03:21:16 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-osd-10[61294]: 2024-03-24T03:21:16.769+0000 7f804e31f700 -1 osd.10 127 heartbeat_check: no reply from 172.21.15.193:6812 osd.5 since back 2024-03-24T03:20:41.750894+0000 front 2024-03-24T03:20:41.750949+0000 (oldest deadline 2024-03-24T03:21:04.050717+0000) 2024-03-24T03:21:17.205 INFO:journalctl@ceph.osd.10.smithi189.stdout:Mar 24 03:21:16 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-osd-10[61294]: 2024-03-24T03:21:16.769+0000 7f804e31f700 -1 osd.10 127 heartbeat_check: no reply from 172.21.15.118:6822 osd.6 since back 2024-03-24T03:20:50.452401+0000 front 2024-03-24T03:20:50.452443+0000 (oldest deadline 2024-03-24T03:21:12.752167+0000) 2024-03-24T03:21:17.210 INFO:journalctl@ceph.osd.11.smithi193.stdout:Mar 24 03:21:16 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-osd-11[62452]: 2024-03-24T03:21:16.934+0000 7f0edd521700 -1 osd.11 127 heartbeat_check: no reply from 172.21.15.118:6806 osd.0 since back 2024-03-24T03:19:59.095638+0000 front 2024-03-24T03:19:59.095627+0000 (oldest deadline 2024-03-24T03:20:20.195530+0000) 2024-03-24T03:21:17.210 INFO:journalctl@ceph.osd.11.smithi193.stdout:Mar 24 03:21:16 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-osd-11[62452]: 2024-03-24T03:21:16.934+0000 7f0edd521700 -1 osd.11 127 heartbeat_check: no reply from 172.21.15.189:6804 osd.1 since back 2024-03-24T03:20:07.399753+0000 front 2024-03-24T03:20:07.399812+0000 (oldest deadline 2024-03-24T03:20:32.099923+0000) 2024-03-24T03:21:17.211 INFO:journalctl@ceph.osd.11.smithi193.stdout:Mar 24 03:21:16 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-osd-11[62452]: 2024-03-24T03:21:16.934+0000 7f0edd521700 -1 osd.11 127 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-03-24T03:20:12.100299+0000 front 2024-03-24T03:20:12.100348+0000 (oldest deadline 2024-03-24T03:20:38.000441+0000) 2024-03-24T03:21:17.211 INFO:journalctl@ceph.osd.11.smithi193.stdout:Mar 24 03:21:16 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-osd-11[62452]: 2024-03-24T03:21:16.934+0000 7f0edd521700 -1 osd.11 127 heartbeat_check: no reply from 172.21.15.118:6814 osd.3 since back 2024-03-24T03:20:25.002506+0000 front 2024-03-24T03:20:25.002576+0000 (oldest deadline 2024-03-24T03:20:46.702390+0000) 2024-03-24T03:21:17.211 INFO:journalctl@ceph.osd.11.smithi193.stdout:Mar 24 03:21:16 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-osd-11[62452]: 2024-03-24T03:21:16.934+0000 7f0edd521700 -1 osd.11 127 heartbeat_check: no reply from 172.21.15.189:6812 osd.4 since back 2024-03-24T03:20:32.905105+0000 front 2024-03-24T03:20:32.905067+0000 (oldest deadline 2024-03-24T03:20:55.204998+0000) 2024-03-24T03:21:17.211 INFO:journalctl@ceph.osd.11.smithi193.stdout:Mar 24 03:21:16 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-osd-11[62452]: 2024-03-24T03:21:16.934+0000 7f0edd521700 -1 osd.11 127 heartbeat_check: no reply from 172.21.15.193:6812 osd.5 since back 2024-03-24T03:20:42.106727+0000 front 2024-03-24T03:20:42.106844+0000 (oldest deadline 2024-03-24T03:21:05.006783+0000) 2024-03-24T03:21:17.582 INFO:journalctl@ceph.osd.9.smithi118.stdout:Mar 24 03:21:17 smithi118 podman[146597]: 2024-03-24 03:21:17.23401038 +0000 UTC m=+0.037662238 image pull c664afe56bb01db4f9afa9532c20598e201e664f2e9ba2be50db0d52674a26d2 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc 2024-03-24T03:21:17.583 INFO:journalctl@ceph.osd.9.smithi118.stdout:Mar 24 03:21:17 smithi118 podman[146597]: 2024-03-24 03:21:17.336986836 +0000 UTC m=+0.140638732 container create bad552a3faed60c63bff08c7372879057c656ef03bdc84baf7d327860f3fed57 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-d79a5420-e971-11ee-95cd-87774f69a715-osd-9-deactivate, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, maintainer=Guillaume Abrioux , GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, org.label-schema.build-date=20240319, org.label-schema.schema-version=1.0, GIT_CLEAN=True) 2024-03-24T03:21:17.583 INFO:journalctl@ceph.osd.9.smithi118.stdout:Mar 24 03:21:17 smithi118 podman[146597]: 2024-03-24 03:21:17.582188654 +0000 UTC m=+0.385840557 container init bad552a3faed60c63bff08c7372879057c656ef03bdc84baf7d327860f3fed57 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-d79a5420-e971-11ee-95cd-87774f69a715-osd-9-deactivate, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, ceph=True, org.label-schema.build-date=20240319, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.schema-version=1.0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-24T03:21:17.946 INFO:journalctl@ceph.osd.9.smithi118.stdout:Mar 24 03:21:17 smithi118 podman[146597]: 2024-03-24 03:21:17.593274616 +0000 UTC m=+0.396926482 container start bad552a3faed60c63bff08c7372879057c656ef03bdc84baf7d327860f3fed57 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-d79a5420-e971-11ee-95cd-87774f69a715-osd-9-deactivate, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.build-date=20240319, CEPH_POINT_RELEASE=, GIT_CLEAN=True, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-24T03:21:17.946 INFO:journalctl@ceph.osd.9.smithi118.stdout:Mar 24 03:21:17 smithi118 podman[146597]: 2024-03-24 03:21:17.656185792 +0000 UTC m=+0.459837708 container attach bad552a3faed60c63bff08c7372879057c656ef03bdc84baf7d327860f3fed57 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-d79a5420-e971-11ee-95cd-87774f69a715-osd-9-deactivate, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, ceph=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240319, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, org.label-schema.license=GPLv2, GIT_CLEAN=True) 2024-03-24T03:21:18.204 INFO:journalctl@ceph.osd.10.smithi189.stdout:Mar 24 03:21:17 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-osd-10[61294]: 2024-03-24T03:21:17.757+0000 7f804e31f700 -1 osd.10 127 heartbeat_check: no reply from 172.21.15.118:6806 osd.0 since back 2024-03-24T03:19:56.942815+0000 front 2024-03-24T03:19:56.942951+0000 (oldest deadline 2024-03-24T03:20:22.842714+0000) 2024-03-24T03:21:18.204 INFO:journalctl@ceph.osd.10.smithi189.stdout:Mar 24 03:21:17 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-osd-10[61294]: 2024-03-24T03:21:17.757+0000 7f804e31f700 -1 osd.10 127 heartbeat_check: no reply from 172.21.15.189:6804 osd.1 since back 2024-03-24T03:20:07.543899+0000 front 2024-03-24T03:20:07.543880+0000 (oldest deadline 2024-03-24T03:20:30.944770+0000) 2024-03-24T03:21:18.205 INFO:journalctl@ceph.osd.10.smithi189.stdout:Mar 24 03:21:17 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-osd-10[61294]: 2024-03-24T03:21:17.757+0000 7f804e31f700 -1 osd.10 127 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-03-24T03:20:15.546606+0000 front 2024-03-24T03:20:15.546459+0000 (oldest deadline 2024-03-24T03:20:37.846574+0000) 2024-03-24T03:21:18.205 INFO:journalctl@ceph.osd.10.smithi189.stdout:Mar 24 03:21:17 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-osd-10[61294]: 2024-03-24T03:21:17.757+0000 7f804e31f700 -1 osd.10 127 heartbeat_check: no reply from 172.21.15.118:6814 osd.3 since back 2024-03-24T03:20:22.547814+0000 front 2024-03-24T03:20:22.547848+0000 (oldest deadline 2024-03-24T03:20:47.748188+0000) 2024-03-24T03:21:18.205 INFO:journalctl@ceph.osd.10.smithi189.stdout:Mar 24 03:21:17 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-osd-10[61294]: 2024-03-24T03:21:17.757+0000 7f804e31f700 -1 osd.10 127 heartbeat_check: no reply from 172.21.15.189:6812 osd.4 since back 2024-03-24T03:20:33.049108+0000 front 2024-03-24T03:20:33.049154+0000 (oldest deadline 2024-03-24T03:20:55.949290+0000) 2024-03-24T03:21:18.205 INFO:journalctl@ceph.osd.10.smithi189.stdout:Mar 24 03:21:17 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-osd-10[61294]: 2024-03-24T03:21:17.757+0000 7f804e31f700 -1 osd.10 127 heartbeat_check: no reply from 172.21.15.193:6812 osd.5 since back 2024-03-24T03:20:41.750894+0000 front 2024-03-24T03:20:41.750949+0000 (oldest deadline 2024-03-24T03:21:04.050717+0000) 2024-03-24T03:21:18.205 INFO:journalctl@ceph.osd.10.smithi189.stdout:Mar 24 03:21:17 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-osd-10[61294]: 2024-03-24T03:21:17.757+0000 7f804e31f700 -1 osd.10 127 heartbeat_check: no reply from 172.21.15.118:6822 osd.6 since back 2024-03-24T03:20:50.452401+0000 front 2024-03-24T03:20:50.452443+0000 (oldest deadline 2024-03-24T03:21:12.752167+0000) 2024-03-24T03:21:18.211 INFO:journalctl@ceph.osd.11.smithi193.stdout:Mar 24 03:21:17 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-osd-11[62452]: 2024-03-24T03:21:17.922+0000 7f0edd521700 -1 osd.11 127 heartbeat_check: no reply from 172.21.15.118:6806 osd.0 since back 2024-03-24T03:19:59.095638+0000 front 2024-03-24T03:19:59.095627+0000 (oldest deadline 2024-03-24T03:20:20.195530+0000) 2024-03-24T03:21:18.211 INFO:journalctl@ceph.osd.11.smithi193.stdout:Mar 24 03:21:17 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-osd-11[62452]: 2024-03-24T03:21:17.922+0000 7f0edd521700 -1 osd.11 127 heartbeat_check: no reply from 172.21.15.189:6804 osd.1 since back 2024-03-24T03:20:07.399753+0000 front 2024-03-24T03:20:07.399812+0000 (oldest deadline 2024-03-24T03:20:32.099923+0000) 2024-03-24T03:21:18.211 INFO:journalctl@ceph.osd.11.smithi193.stdout:Mar 24 03:21:17 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-osd-11[62452]: 2024-03-24T03:21:17.922+0000 7f0edd521700 -1 osd.11 127 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-03-24T03:20:12.100299+0000 front 2024-03-24T03:20:12.100348+0000 (oldest deadline 2024-03-24T03:20:38.000441+0000) 2024-03-24T03:21:18.211 INFO:journalctl@ceph.osd.11.smithi193.stdout:Mar 24 03:21:17 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-osd-11[62452]: 2024-03-24T03:21:17.922+0000 7f0edd521700 -1 osd.11 127 heartbeat_check: no reply from 172.21.15.118:6814 osd.3 since back 2024-03-24T03:20:25.002506+0000 front 2024-03-24T03:20:25.002576+0000 (oldest deadline 2024-03-24T03:20:46.702390+0000) 2024-03-24T03:21:18.211 INFO:journalctl@ceph.osd.11.smithi193.stdout:Mar 24 03:21:17 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-osd-11[62452]: 2024-03-24T03:21:17.922+0000 7f0edd521700 -1 osd.11 127 heartbeat_check: no reply from 172.21.15.189:6812 osd.4 since back 2024-03-24T03:20:32.905105+0000 front 2024-03-24T03:20:32.905067+0000 (oldest deadline 2024-03-24T03:20:55.204998+0000) 2024-03-24T03:21:18.212 INFO:journalctl@ceph.osd.11.smithi193.stdout:Mar 24 03:21:17 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-osd-11[62452]: 2024-03-24T03:21:17.922+0000 7f0edd521700 -1 osd.11 127 heartbeat_check: no reply from 172.21.15.193:6812 osd.5 since back 2024-03-24T03:20:42.106727+0000 front 2024-03-24T03:20:42.106844+0000 (oldest deadline 2024-03-24T03:21:05.006783+0000) 2024-03-24T03:21:18.300 INFO:journalctl@ceph.osd.9.smithi118.stdout:Mar 24 03:21:18 smithi118 podman[146597]: 2024-03-24 03:21:18.017732858 +0000 UTC m=+0.821384764 container died bad552a3faed60c63bff08c7372879057c656ef03bdc84baf7d327860f3fed57 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-d79a5420-e971-11ee-95cd-87774f69a715-osd-9-deactivate, org.label-schema.license=GPLv2, GIT_CLEAN=True, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, CEPH_POINT_RELEASE=, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240319, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True) 2024-03-24T03:21:18.607 DEBUG:teuthology.orchestra.run.smithi118:> sudo pkill -f 'journalctl -f -n 0 -u ceph-d79a5420-e971-11ee-95cd-87774f69a715@osd.9.service' 2024-03-24T03:21:18.700 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-24T03:21:18.700 INFO:tasks.cephadm.osd.9:Stopped osd.9 2024-03-24T03:21:18.701 INFO:tasks.cephadm.osd.10:Stopping osd.10... 2024-03-24T03:21:18.701 DEBUG:teuthology.orchestra.run.smithi189:> sudo systemctl stop ceph-d79a5420-e971-11ee-95cd-87774f69a715@osd.10 2024-03-24T03:21:19.080 INFO:journalctl@ceph.osd.10.smithi189.stdout:Mar 24 03:21:18 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-osd-10[61294]: 2024-03-24T03:21:18.746+0000 7f804e31f700 -1 osd.10 127 heartbeat_check: no reply from 172.21.15.118:6806 osd.0 since back 2024-03-24T03:19:56.942815+0000 front 2024-03-24T03:19:56.942951+0000 (oldest deadline 2024-03-24T03:20:22.842714+0000) 2024-03-24T03:21:19.080 INFO:journalctl@ceph.osd.10.smithi189.stdout:Mar 24 03:21:18 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-osd-10[61294]: 2024-03-24T03:21:18.746+0000 7f804e31f700 -1 osd.10 127 heartbeat_check: no reply from 172.21.15.189:6804 osd.1 since back 2024-03-24T03:20:07.543899+0000 front 2024-03-24T03:20:07.543880+0000 (oldest deadline 2024-03-24T03:20:30.944770+0000) 2024-03-24T03:21:19.080 INFO:journalctl@ceph.osd.10.smithi189.stdout:Mar 24 03:21:18 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-osd-10[61294]: 2024-03-24T03:21:18.746+0000 7f804e31f700 -1 osd.10 127 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-03-24T03:20:15.546606+0000 front 2024-03-24T03:20:15.546459+0000 (oldest deadline 2024-03-24T03:20:37.846574+0000) 2024-03-24T03:21:19.080 INFO:journalctl@ceph.osd.10.smithi189.stdout:Mar 24 03:21:18 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-osd-10[61294]: 2024-03-24T03:21:18.746+0000 7f804e31f700 -1 osd.10 127 heartbeat_check: no reply from 172.21.15.118:6814 osd.3 since back 2024-03-24T03:20:22.547814+0000 front 2024-03-24T03:20:22.547848+0000 (oldest deadline 2024-03-24T03:20:47.748188+0000) 2024-03-24T03:21:19.080 INFO:journalctl@ceph.osd.10.smithi189.stdout:Mar 24 03:21:18 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-osd-10[61294]: 2024-03-24T03:21:18.746+0000 7f804e31f700 -1 osd.10 127 heartbeat_check: no reply from 172.21.15.189:6812 osd.4 since back 2024-03-24T03:20:33.049108+0000 front 2024-03-24T03:20:33.049154+0000 (oldest deadline 2024-03-24T03:20:55.949290+0000) 2024-03-24T03:21:19.080 INFO:journalctl@ceph.osd.10.smithi189.stdout:Mar 24 03:21:18 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-osd-10[61294]: 2024-03-24T03:21:18.746+0000 7f804e31f700 -1 osd.10 127 heartbeat_check: no reply from 172.21.15.193:6812 osd.5 since back 2024-03-24T03:20:41.750894+0000 front 2024-03-24T03:20:41.750949+0000 (oldest deadline 2024-03-24T03:21:04.050717+0000) 2024-03-24T03:21:19.081 INFO:journalctl@ceph.osd.10.smithi189.stdout:Mar 24 03:21:18 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-osd-10[61294]: 2024-03-24T03:21:18.746+0000 7f804e31f700 -1 osd.10 127 heartbeat_check: no reply from 172.21.15.118:6822 osd.6 since back 2024-03-24T03:20:50.452401+0000 front 2024-03-24T03:20:50.452443+0000 (oldest deadline 2024-03-24T03:21:12.752167+0000) 2024-03-24T03:21:19.081 INFO:journalctl@ceph.osd.10.smithi189.stdout:Mar 24 03:21:18 smithi189 systemd[1]: Stopping Ceph osd.10 for d79a5420-e971-11ee-95cd-87774f69a715... 2024-03-24T03:21:19.210 INFO:journalctl@ceph.osd.11.smithi193.stdout:Mar 24 03:21:18 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-osd-11[62452]: 2024-03-24T03:21:18.932+0000 7f0edd521700 -1 osd.11 127 heartbeat_check: no reply from 172.21.15.118:6806 osd.0 since back 2024-03-24T03:19:59.095638+0000 front 2024-03-24T03:19:59.095627+0000 (oldest deadline 2024-03-24T03:20:20.195530+0000) 2024-03-24T03:21:19.211 INFO:journalctl@ceph.osd.11.smithi193.stdout:Mar 24 03:21:18 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-osd-11[62452]: 2024-03-24T03:21:18.932+0000 7f0edd521700 -1 osd.11 127 heartbeat_check: no reply from 172.21.15.189:6804 osd.1 since back 2024-03-24T03:20:07.399753+0000 front 2024-03-24T03:20:07.399812+0000 (oldest deadline 2024-03-24T03:20:32.099923+0000) 2024-03-24T03:21:19.211 INFO:journalctl@ceph.osd.11.smithi193.stdout:Mar 24 03:21:18 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-osd-11[62452]: 2024-03-24T03:21:18.932+0000 7f0edd521700 -1 osd.11 127 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-03-24T03:20:12.100299+0000 front 2024-03-24T03:20:12.100348+0000 (oldest deadline 2024-03-24T03:20:38.000441+0000) 2024-03-24T03:21:19.211 INFO:journalctl@ceph.osd.11.smithi193.stdout:Mar 24 03:21:18 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-osd-11[62452]: 2024-03-24T03:21:18.932+0000 7f0edd521700 -1 osd.11 127 heartbeat_check: no reply from 172.21.15.118:6814 osd.3 since back 2024-03-24T03:20:25.002506+0000 front 2024-03-24T03:20:25.002576+0000 (oldest deadline 2024-03-24T03:20:46.702390+0000) 2024-03-24T03:21:19.211 INFO:journalctl@ceph.osd.11.smithi193.stdout:Mar 24 03:21:18 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-osd-11[62452]: 2024-03-24T03:21:18.932+0000 7f0edd521700 -1 osd.11 127 heartbeat_check: no reply from 172.21.15.189:6812 osd.4 since back 2024-03-24T03:20:32.905105+0000 front 2024-03-24T03:20:32.905067+0000 (oldest deadline 2024-03-24T03:20:55.204998+0000) 2024-03-24T03:21:19.211 INFO:journalctl@ceph.osd.11.smithi193.stdout:Mar 24 03:21:18 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-osd-11[62452]: 2024-03-24T03:21:18.932+0000 7f0edd521700 -1 osd.11 127 heartbeat_check: no reply from 172.21.15.193:6812 osd.5 since back 2024-03-24T03:20:42.106727+0000 front 2024-03-24T03:20:42.106844+0000 (oldest deadline 2024-03-24T03:21:05.006783+0000) 2024-03-24T03:21:19.453 INFO:journalctl@ceph.osd.10.smithi189.stdout:Mar 24 03:21:19 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-osd-10[61294]: 2024-03-24T03:21:19.079+0000 7f8051d2b700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.10 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-24T03:21:19.454 INFO:journalctl@ceph.osd.10.smithi189.stdout:Mar 24 03:21:19 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-osd-10[61294]: 2024-03-24T03:21:19.079+0000 7f8051d2b700 -1 osd.10 127 *** Got signal Terminated *** 2024-03-24T03:21:19.454 INFO:journalctl@ceph.osd.10.smithi189.stdout:Mar 24 03:21:19 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-osd-10[61294]: 2024-03-24T03:21:19.079+0000 7f8051d2b700 -1 osd.10 127 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-24T03:21:20.160 INFO:journalctl@ceph.osd.10.smithi189.stdout:Mar 24 03:21:19 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-osd-10[61294]: 2024-03-24T03:21:19.731+0000 7f804e31f700 -1 osd.10 127 heartbeat_check: no reply from 172.21.15.118:6806 osd.0 since back 2024-03-24T03:19:56.942815+0000 front 2024-03-24T03:19:56.942951+0000 (oldest deadline 2024-03-24T03:20:22.842714+0000) 2024-03-24T03:21:20.160 INFO:journalctl@ceph.osd.10.smithi189.stdout:Mar 24 03:21:19 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-osd-10[61294]: 2024-03-24T03:21:19.731+0000 7f804e31f700 -1 osd.10 127 heartbeat_check: no reply from 172.21.15.189:6804 osd.1 since back 2024-03-24T03:20:07.543899+0000 front 2024-03-24T03:20:07.543880+0000 (oldest deadline 2024-03-24T03:20:30.944770+0000) 2024-03-24T03:21:20.160 INFO:journalctl@ceph.osd.10.smithi189.stdout:Mar 24 03:21:19 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-osd-10[61294]: 2024-03-24T03:21:19.731+0000 7f804e31f700 -1 osd.10 127 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-03-24T03:20:15.546606+0000 front 2024-03-24T03:20:15.546459+0000 (oldest deadline 2024-03-24T03:20:37.846574+0000) 2024-03-24T03:21:20.161 INFO:journalctl@ceph.osd.10.smithi189.stdout:Mar 24 03:21:19 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-osd-10[61294]: 2024-03-24T03:21:19.731+0000 7f804e31f700 -1 osd.10 127 heartbeat_check: no reply from 172.21.15.118:6814 osd.3 since back 2024-03-24T03:20:22.547814+0000 front 2024-03-24T03:20:22.547848+0000 (oldest deadline 2024-03-24T03:20:47.748188+0000) 2024-03-24T03:21:20.161 INFO:journalctl@ceph.osd.10.smithi189.stdout:Mar 24 03:21:19 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-osd-10[61294]: 2024-03-24T03:21:19.731+0000 7f804e31f700 -1 osd.10 127 heartbeat_check: no reply from 172.21.15.189:6812 osd.4 since back 2024-03-24T03:20:33.049108+0000 front 2024-03-24T03:20:33.049154+0000 (oldest deadline 2024-03-24T03:20:55.949290+0000) 2024-03-24T03:21:20.161 INFO:journalctl@ceph.osd.10.smithi189.stdout:Mar 24 03:21:19 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-osd-10[61294]: 2024-03-24T03:21:19.731+0000 7f804e31f700 -1 osd.10 127 heartbeat_check: no reply from 172.21.15.193:6812 osd.5 since back 2024-03-24T03:20:41.750894+0000 front 2024-03-24T03:20:41.750949+0000 (oldest deadline 2024-03-24T03:21:04.050717+0000) 2024-03-24T03:21:20.161 INFO:journalctl@ceph.osd.10.smithi189.stdout:Mar 24 03:21:19 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-osd-10[61294]: 2024-03-24T03:21:19.731+0000 7f804e31f700 -1 osd.10 127 heartbeat_check: no reply from 172.21.15.118:6822 osd.6 since back 2024-03-24T03:20:50.452401+0000 front 2024-03-24T03:20:50.452443+0000 (oldest deadline 2024-03-24T03:21:12.752167+0000) 2024-03-24T03:21:20.211 INFO:journalctl@ceph.osd.11.smithi193.stdout:Mar 24 03:21:19 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-osd-11[62452]: 2024-03-24T03:21:19.980+0000 7f0edd521700 -1 osd.11 127 heartbeat_check: no reply from 172.21.15.118:6806 osd.0 since back 2024-03-24T03:19:59.095638+0000 front 2024-03-24T03:19:59.095627+0000 (oldest deadline 2024-03-24T03:20:20.195530+0000) 2024-03-24T03:21:20.211 INFO:journalctl@ceph.osd.11.smithi193.stdout:Mar 24 03:21:19 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-osd-11[62452]: 2024-03-24T03:21:19.980+0000 7f0edd521700 -1 osd.11 127 heartbeat_check: no reply from 172.21.15.189:6804 osd.1 since back 2024-03-24T03:20:07.399753+0000 front 2024-03-24T03:20:07.399812+0000 (oldest deadline 2024-03-24T03:20:32.099923+0000) 2024-03-24T03:21:20.211 INFO:journalctl@ceph.osd.11.smithi193.stdout:Mar 24 03:21:19 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-osd-11[62452]: 2024-03-24T03:21:19.980+0000 7f0edd521700 -1 osd.11 127 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-03-24T03:20:12.100299+0000 front 2024-03-24T03:20:12.100348+0000 (oldest deadline 2024-03-24T03:20:38.000441+0000) 2024-03-24T03:21:20.211 INFO:journalctl@ceph.osd.11.smithi193.stdout:Mar 24 03:21:19 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-osd-11[62452]: 2024-03-24T03:21:19.980+0000 7f0edd521700 -1 osd.11 127 heartbeat_check: no reply from 172.21.15.118:6814 osd.3 since back 2024-03-24T03:20:25.002506+0000 front 2024-03-24T03:20:25.002576+0000 (oldest deadline 2024-03-24T03:20:46.702390+0000) 2024-03-24T03:21:20.211 INFO:journalctl@ceph.osd.11.smithi193.stdout:Mar 24 03:21:19 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-osd-11[62452]: 2024-03-24T03:21:19.980+0000 7f0edd521700 -1 osd.11 127 heartbeat_check: no reply from 172.21.15.189:6812 osd.4 since back 2024-03-24T03:20:32.905105+0000 front 2024-03-24T03:20:32.905067+0000 (oldest deadline 2024-03-24T03:20:55.204998+0000) 2024-03-24T03:21:20.211 INFO:journalctl@ceph.osd.11.smithi193.stdout:Mar 24 03:21:19 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-osd-11[62452]: 2024-03-24T03:21:19.980+0000 7f0edd521700 -1 osd.11 127 heartbeat_check: no reply from 172.21.15.193:6812 osd.5 since back 2024-03-24T03:20:42.106727+0000 front 2024-03-24T03:20:42.106844+0000 (oldest deadline 2024-03-24T03:21:05.006783+0000) 2024-03-24T03:21:21.204 INFO:journalctl@ceph.osd.10.smithi189.stdout:Mar 24 03:21:20 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-osd-10[61294]: 2024-03-24T03:21:20.711+0000 7f804e31f700 -1 osd.10 127 heartbeat_check: no reply from 172.21.15.118:6806 osd.0 since back 2024-03-24T03:19:56.942815+0000 front 2024-03-24T03:19:56.942951+0000 (oldest deadline 2024-03-24T03:20:22.842714+0000) 2024-03-24T03:21:21.204 INFO:journalctl@ceph.osd.10.smithi189.stdout:Mar 24 03:21:20 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-osd-10[61294]: 2024-03-24T03:21:20.711+0000 7f804e31f700 -1 osd.10 127 heartbeat_check: no reply from 172.21.15.189:6804 osd.1 since back 2024-03-24T03:20:07.543899+0000 front 2024-03-24T03:20:07.543880+0000 (oldest deadline 2024-03-24T03:20:30.944770+0000) 2024-03-24T03:21:21.204 INFO:journalctl@ceph.osd.10.smithi189.stdout:Mar 24 03:21:20 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-osd-10[61294]: 2024-03-24T03:21:20.711+0000 7f804e31f700 -1 osd.10 127 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-03-24T03:20:15.546606+0000 front 2024-03-24T03:20:15.546459+0000 (oldest deadline 2024-03-24T03:20:37.846574+0000) 2024-03-24T03:21:21.204 INFO:journalctl@ceph.osd.10.smithi189.stdout:Mar 24 03:21:20 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-osd-10[61294]: 2024-03-24T03:21:20.711+0000 7f804e31f700 -1 osd.10 127 heartbeat_check: no reply from 172.21.15.118:6814 osd.3 since back 2024-03-24T03:20:22.547814+0000 front 2024-03-24T03:20:22.547848+0000 (oldest deadline 2024-03-24T03:20:47.748188+0000) 2024-03-24T03:21:21.205 INFO:journalctl@ceph.osd.10.smithi189.stdout:Mar 24 03:21:20 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-osd-10[61294]: 2024-03-24T03:21:20.711+0000 7f804e31f700 -1 osd.10 127 heartbeat_check: no reply from 172.21.15.189:6812 osd.4 since back 2024-03-24T03:20:33.049108+0000 front 2024-03-24T03:20:33.049154+0000 (oldest deadline 2024-03-24T03:20:55.949290+0000) 2024-03-24T03:21:21.205 INFO:journalctl@ceph.osd.10.smithi189.stdout:Mar 24 03:21:20 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-osd-10[61294]: 2024-03-24T03:21:20.711+0000 7f804e31f700 -1 osd.10 127 heartbeat_check: no reply from 172.21.15.193:6812 osd.5 since back 2024-03-24T03:20:41.750894+0000 front 2024-03-24T03:20:41.750949+0000 (oldest deadline 2024-03-24T03:21:04.050717+0000) 2024-03-24T03:21:21.205 INFO:journalctl@ceph.osd.10.smithi189.stdout:Mar 24 03:21:20 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-osd-10[61294]: 2024-03-24T03:21:20.711+0000 7f804e31f700 -1 osd.10 127 heartbeat_check: no reply from 172.21.15.118:6822 osd.6 since back 2024-03-24T03:20:50.452401+0000 front 2024-03-24T03:20:50.452443+0000 (oldest deadline 2024-03-24T03:21:12.752167+0000) 2024-03-24T03:21:21.210 INFO:journalctl@ceph.osd.11.smithi193.stdout:Mar 24 03:21:20 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-osd-11[62452]: 2024-03-24T03:21:20.948+0000 7f0edd521700 -1 osd.11 127 heartbeat_check: no reply from 172.21.15.118:6806 osd.0 since back 2024-03-24T03:19:59.095638+0000 front 2024-03-24T03:19:59.095627+0000 (oldest deadline 2024-03-24T03:20:20.195530+0000) 2024-03-24T03:21:21.211 INFO:journalctl@ceph.osd.11.smithi193.stdout:Mar 24 03:21:20 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-osd-11[62452]: 2024-03-24T03:21:20.948+0000 7f0edd521700 -1 osd.11 127 heartbeat_check: no reply from 172.21.15.189:6804 osd.1 since back 2024-03-24T03:20:07.399753+0000 front 2024-03-24T03:20:07.399812+0000 (oldest deadline 2024-03-24T03:20:32.099923+0000) 2024-03-24T03:21:21.211 INFO:journalctl@ceph.osd.11.smithi193.stdout:Mar 24 03:21:20 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-osd-11[62452]: 2024-03-24T03:21:20.948+0000 7f0edd521700 -1 osd.11 127 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-03-24T03:20:12.100299+0000 front 2024-03-24T03:20:12.100348+0000 (oldest deadline 2024-03-24T03:20:38.000441+0000) 2024-03-24T03:21:21.211 INFO:journalctl@ceph.osd.11.smithi193.stdout:Mar 24 03:21:20 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-osd-11[62452]: 2024-03-24T03:21:20.948+0000 7f0edd521700 -1 osd.11 127 heartbeat_check: no reply from 172.21.15.118:6814 osd.3 since back 2024-03-24T03:20:25.002506+0000 front 2024-03-24T03:20:25.002576+0000 (oldest deadline 2024-03-24T03:20:46.702390+0000) 2024-03-24T03:21:21.211 INFO:journalctl@ceph.osd.11.smithi193.stdout:Mar 24 03:21:20 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-osd-11[62452]: 2024-03-24T03:21:20.948+0000 7f0edd521700 -1 osd.11 127 heartbeat_check: no reply from 172.21.15.189:6812 osd.4 since back 2024-03-24T03:20:32.905105+0000 front 2024-03-24T03:20:32.905067+0000 (oldest deadline 2024-03-24T03:20:55.204998+0000) 2024-03-24T03:21:21.211 INFO:journalctl@ceph.osd.11.smithi193.stdout:Mar 24 03:21:20 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-osd-11[62452]: 2024-03-24T03:21:20.948+0000 7f0edd521700 -1 osd.11 127 heartbeat_check: no reply from 172.21.15.193:6812 osd.5 since back 2024-03-24T03:20:42.106727+0000 front 2024-03-24T03:20:42.106844+0000 (oldest deadline 2024-03-24T03:21:05.006783+0000) 2024-03-24T03:21:21.954 INFO:journalctl@ceph.osd.10.smithi189.stdout:Mar 24 03:21:21 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-osd-10[61294]: 2024-03-24T03:21:21.700+0000 7f804e31f700 -1 osd.10 127 heartbeat_check: no reply from 172.21.15.118:6806 osd.0 since back 2024-03-24T03:19:56.942815+0000 front 2024-03-24T03:19:56.942951+0000 (oldest deadline 2024-03-24T03:20:22.842714+0000) 2024-03-24T03:21:21.954 INFO:journalctl@ceph.osd.10.smithi189.stdout:Mar 24 03:21:21 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-osd-10[61294]: 2024-03-24T03:21:21.700+0000 7f804e31f700 -1 osd.10 127 heartbeat_check: no reply from 172.21.15.189:6804 osd.1 since back 2024-03-24T03:20:07.543899+0000 front 2024-03-24T03:20:07.543880+0000 (oldest deadline 2024-03-24T03:20:30.944770+0000) 2024-03-24T03:21:21.954 INFO:journalctl@ceph.osd.10.smithi189.stdout:Mar 24 03:21:21 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-osd-10[61294]: 2024-03-24T03:21:21.700+0000 7f804e31f700 -1 osd.10 127 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-03-24T03:20:15.546606+0000 front 2024-03-24T03:20:15.546459+0000 (oldest deadline 2024-03-24T03:20:37.846574+0000) 2024-03-24T03:21:21.954 INFO:journalctl@ceph.osd.10.smithi189.stdout:Mar 24 03:21:21 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-osd-10[61294]: 2024-03-24T03:21:21.700+0000 7f804e31f700 -1 osd.10 127 heartbeat_check: no reply from 172.21.15.118:6814 osd.3 since back 2024-03-24T03:20:22.547814+0000 front 2024-03-24T03:20:22.547848+0000 (oldest deadline 2024-03-24T03:20:47.748188+0000) 2024-03-24T03:21:21.954 INFO:journalctl@ceph.osd.10.smithi189.stdout:Mar 24 03:21:21 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-osd-10[61294]: 2024-03-24T03:21:21.700+0000 7f804e31f700 -1 osd.10 127 heartbeat_check: no reply from 172.21.15.189:6812 osd.4 since back 2024-03-24T03:20:33.049108+0000 front 2024-03-24T03:20:33.049154+0000 (oldest deadline 2024-03-24T03:20:55.949290+0000) 2024-03-24T03:21:21.955 INFO:journalctl@ceph.osd.10.smithi189.stdout:Mar 24 03:21:21 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-osd-10[61294]: 2024-03-24T03:21:21.700+0000 7f804e31f700 -1 osd.10 127 heartbeat_check: no reply from 172.21.15.193:6812 osd.5 since back 2024-03-24T03:20:41.750894+0000 front 2024-03-24T03:20:41.750949+0000 (oldest deadline 2024-03-24T03:21:04.050717+0000) 2024-03-24T03:21:21.955 INFO:journalctl@ceph.osd.10.smithi189.stdout:Mar 24 03:21:21 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-osd-10[61294]: 2024-03-24T03:21:21.700+0000 7f804e31f700 -1 osd.10 127 heartbeat_check: no reply from 172.21.15.118:6822 osd.6 since back 2024-03-24T03:20:50.452401+0000 front 2024-03-24T03:20:50.452443+0000 (oldest deadline 2024-03-24T03:21:12.752167+0000) 2024-03-24T03:21:22.210 INFO:journalctl@ceph.osd.11.smithi193.stdout:Mar 24 03:21:21 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-osd-11[62452]: 2024-03-24T03:21:21.944+0000 7f0edd521700 -1 osd.11 127 heartbeat_check: no reply from 172.21.15.118:6806 osd.0 since back 2024-03-24T03:19:59.095638+0000 front 2024-03-24T03:19:59.095627+0000 (oldest deadline 2024-03-24T03:20:20.195530+0000) 2024-03-24T03:21:22.211 INFO:journalctl@ceph.osd.11.smithi193.stdout:Mar 24 03:21:21 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-osd-11[62452]: 2024-03-24T03:21:21.944+0000 7f0edd521700 -1 osd.11 127 heartbeat_check: no reply from 172.21.15.189:6804 osd.1 since back 2024-03-24T03:20:07.399753+0000 front 2024-03-24T03:20:07.399812+0000 (oldest deadline 2024-03-24T03:20:32.099923+0000) 2024-03-24T03:21:22.211 INFO:journalctl@ceph.osd.11.smithi193.stdout:Mar 24 03:21:21 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-osd-11[62452]: 2024-03-24T03:21:21.944+0000 7f0edd521700 -1 osd.11 127 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-03-24T03:20:12.100299+0000 front 2024-03-24T03:20:12.100348+0000 (oldest deadline 2024-03-24T03:20:38.000441+0000) 2024-03-24T03:21:22.211 INFO:journalctl@ceph.osd.11.smithi193.stdout:Mar 24 03:21:21 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-osd-11[62452]: 2024-03-24T03:21:21.944+0000 7f0edd521700 -1 osd.11 127 heartbeat_check: no reply from 172.21.15.118:6814 osd.3 since back 2024-03-24T03:20:25.002506+0000 front 2024-03-24T03:20:25.002576+0000 (oldest deadline 2024-03-24T03:20:46.702390+0000) 2024-03-24T03:21:22.211 INFO:journalctl@ceph.osd.11.smithi193.stdout:Mar 24 03:21:21 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-osd-11[62452]: 2024-03-24T03:21:21.944+0000 7f0edd521700 -1 osd.11 127 heartbeat_check: no reply from 172.21.15.189:6812 osd.4 since back 2024-03-24T03:20:32.905105+0000 front 2024-03-24T03:20:32.905067+0000 (oldest deadline 2024-03-24T03:20:55.204998+0000) 2024-03-24T03:21:22.211 INFO:journalctl@ceph.osd.11.smithi193.stdout:Mar 24 03:21:21 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-osd-11[62452]: 2024-03-24T03:21:21.944+0000 7f0edd521700 -1 osd.11 127 heartbeat_check: no reply from 172.21.15.193:6812 osd.5 since back 2024-03-24T03:20:42.106727+0000 front 2024-03-24T03:20:42.106844+0000 (oldest deadline 2024-03-24T03:21:05.006783+0000) 2024-03-24T03:21:22.211 INFO:journalctl@ceph.osd.11.smithi193.stdout:Mar 24 03:21:21 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-osd-11[62452]: 2024-03-24T03:21:21.944+0000 7f0edd521700 -1 osd.11 127 heartbeat_check: no reply from 172.21.15.189:6820 osd.7 since back 2024-03-24T03:20:58.810942+0000 front 2024-03-24T03:20:58.810766+0000 (oldest deadline 2024-03-24T03:21:21.110690+0000) 2024-03-24T03:21:22.954 INFO:journalctl@ceph.osd.10.smithi189.stdout:Mar 24 03:21:22 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-osd-10[61294]: 2024-03-24T03:21:22.681+0000 7f804e31f700 -1 osd.10 127 heartbeat_check: no reply from 172.21.15.118:6806 osd.0 since back 2024-03-24T03:19:56.942815+0000 front 2024-03-24T03:19:56.942951+0000 (oldest deadline 2024-03-24T03:20:22.842714+0000) 2024-03-24T03:21:22.954 INFO:journalctl@ceph.osd.10.smithi189.stdout:Mar 24 03:21:22 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-osd-10[61294]: 2024-03-24T03:21:22.681+0000 7f804e31f700 -1 osd.10 127 heartbeat_check: no reply from 172.21.15.189:6804 osd.1 since back 2024-03-24T03:20:07.543899+0000 front 2024-03-24T03:20:07.543880+0000 (oldest deadline 2024-03-24T03:20:30.944770+0000) 2024-03-24T03:21:22.954 INFO:journalctl@ceph.osd.10.smithi189.stdout:Mar 24 03:21:22 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-osd-10[61294]: 2024-03-24T03:21:22.681+0000 7f804e31f700 -1 osd.10 127 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-03-24T03:20:15.546606+0000 front 2024-03-24T03:20:15.546459+0000 (oldest deadline 2024-03-24T03:20:37.846574+0000) 2024-03-24T03:21:22.954 INFO:journalctl@ceph.osd.10.smithi189.stdout:Mar 24 03:21:22 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-osd-10[61294]: 2024-03-24T03:21:22.681+0000 7f804e31f700 -1 osd.10 127 heartbeat_check: no reply from 172.21.15.118:6814 osd.3 since back 2024-03-24T03:20:22.547814+0000 front 2024-03-24T03:20:22.547848+0000 (oldest deadline 2024-03-24T03:20:47.748188+0000) 2024-03-24T03:21:22.955 INFO:journalctl@ceph.osd.10.smithi189.stdout:Mar 24 03:21:22 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-osd-10[61294]: 2024-03-24T03:21:22.681+0000 7f804e31f700 -1 osd.10 127 heartbeat_check: no reply from 172.21.15.189:6812 osd.4 since back 2024-03-24T03:20:33.049108+0000 front 2024-03-24T03:20:33.049154+0000 (oldest deadline 2024-03-24T03:20:55.949290+0000) 2024-03-24T03:21:22.955 INFO:journalctl@ceph.osd.10.smithi189.stdout:Mar 24 03:21:22 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-osd-10[61294]: 2024-03-24T03:21:22.681+0000 7f804e31f700 -1 osd.10 127 heartbeat_check: no reply from 172.21.15.193:6812 osd.5 since back 2024-03-24T03:20:41.750894+0000 front 2024-03-24T03:20:41.750949+0000 (oldest deadline 2024-03-24T03:21:04.050717+0000) 2024-03-24T03:21:22.955 INFO:journalctl@ceph.osd.10.smithi189.stdout:Mar 24 03:21:22 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-osd-10[61294]: 2024-03-24T03:21:22.681+0000 7f804e31f700 -1 osd.10 127 heartbeat_check: no reply from 172.21.15.118:6822 osd.6 since back 2024-03-24T03:20:50.452401+0000 front 2024-03-24T03:20:50.452443+0000 (oldest deadline 2024-03-24T03:21:12.752167+0000) 2024-03-24T03:21:23.210 INFO:journalctl@ceph.osd.11.smithi193.stdout:Mar 24 03:21:22 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-osd-11[62452]: 2024-03-24T03:21:22.951+0000 7f0edd521700 -1 osd.11 127 heartbeat_check: no reply from 172.21.15.118:6806 osd.0 since back 2024-03-24T03:19:59.095638+0000 front 2024-03-24T03:19:59.095627+0000 (oldest deadline 2024-03-24T03:20:20.195530+0000) 2024-03-24T03:21:23.211 INFO:journalctl@ceph.osd.11.smithi193.stdout:Mar 24 03:21:22 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-osd-11[62452]: 2024-03-24T03:21:22.951+0000 7f0edd521700 -1 osd.11 127 heartbeat_check: no reply from 172.21.15.189:6804 osd.1 since back 2024-03-24T03:20:07.399753+0000 front 2024-03-24T03:20:07.399812+0000 (oldest deadline 2024-03-24T03:20:32.099923+0000) 2024-03-24T03:21:23.211 INFO:journalctl@ceph.osd.11.smithi193.stdout:Mar 24 03:21:22 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-osd-11[62452]: 2024-03-24T03:21:22.951+0000 7f0edd521700 -1 osd.11 127 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-03-24T03:20:12.100299+0000 front 2024-03-24T03:20:12.100348+0000 (oldest deadline 2024-03-24T03:20:38.000441+0000) 2024-03-24T03:21:23.211 INFO:journalctl@ceph.osd.11.smithi193.stdout:Mar 24 03:21:22 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-osd-11[62452]: 2024-03-24T03:21:22.951+0000 7f0edd521700 -1 osd.11 127 heartbeat_check: no reply from 172.21.15.118:6814 osd.3 since back 2024-03-24T03:20:25.002506+0000 front 2024-03-24T03:20:25.002576+0000 (oldest deadline 2024-03-24T03:20:46.702390+0000) 2024-03-24T03:21:23.211 INFO:journalctl@ceph.osd.11.smithi193.stdout:Mar 24 03:21:22 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-osd-11[62452]: 2024-03-24T03:21:22.951+0000 7f0edd521700 -1 osd.11 127 heartbeat_check: no reply from 172.21.15.189:6812 osd.4 since back 2024-03-24T03:20:32.905105+0000 front 2024-03-24T03:20:32.905067+0000 (oldest deadline 2024-03-24T03:20:55.204998+0000) 2024-03-24T03:21:23.211 INFO:journalctl@ceph.osd.11.smithi193.stdout:Mar 24 03:21:22 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-osd-11[62452]: 2024-03-24T03:21:22.951+0000 7f0edd521700 -1 osd.11 127 heartbeat_check: no reply from 172.21.15.193:6812 osd.5 since back 2024-03-24T03:20:42.106727+0000 front 2024-03-24T03:20:42.106844+0000 (oldest deadline 2024-03-24T03:21:05.006783+0000) 2024-03-24T03:21:23.212 INFO:journalctl@ceph.osd.11.smithi193.stdout:Mar 24 03:21:22 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-osd-11[62452]: 2024-03-24T03:21:22.951+0000 7f0edd521700 -1 osd.11 127 heartbeat_check: no reply from 172.21.15.189:6820 osd.7 since back 2024-03-24T03:20:58.810942+0000 front 2024-03-24T03:20:58.810766+0000 (oldest deadline 2024-03-24T03:21:21.110690+0000) 2024-03-24T03:21:24.204 INFO:journalctl@ceph.osd.10.smithi189.stdout:Mar 24 03:21:23 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-osd-10[61294]: 2024-03-24T03:21:23.729+0000 7f804e31f700 -1 osd.10 127 heartbeat_check: no reply from 172.21.15.118:6806 osd.0 since back 2024-03-24T03:19:56.942815+0000 front 2024-03-24T03:19:56.942951+0000 (oldest deadline 2024-03-24T03:20:22.842714+0000) 2024-03-24T03:21:24.204 INFO:journalctl@ceph.osd.10.smithi189.stdout:Mar 24 03:21:23 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-osd-10[61294]: 2024-03-24T03:21:23.729+0000 7f804e31f700 -1 osd.10 127 heartbeat_check: no reply from 172.21.15.189:6804 osd.1 since back 2024-03-24T03:20:07.543899+0000 front 2024-03-24T03:20:07.543880+0000 (oldest deadline 2024-03-24T03:20:30.944770+0000) 2024-03-24T03:21:24.204 INFO:journalctl@ceph.osd.10.smithi189.stdout:Mar 24 03:21:23 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-osd-10[61294]: 2024-03-24T03:21:23.729+0000 7f804e31f700 -1 osd.10 127 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-03-24T03:20:15.546606+0000 front 2024-03-24T03:20:15.546459+0000 (oldest deadline 2024-03-24T03:20:37.846574+0000) 2024-03-24T03:21:24.204 INFO:journalctl@ceph.osd.10.smithi189.stdout:Mar 24 03:21:23 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-osd-10[61294]: 2024-03-24T03:21:23.729+0000 7f804e31f700 -1 osd.10 127 heartbeat_check: no reply from 172.21.15.118:6814 osd.3 since back 2024-03-24T03:20:22.547814+0000 front 2024-03-24T03:20:22.547848+0000 (oldest deadline 2024-03-24T03:20:47.748188+0000) 2024-03-24T03:21:24.204 INFO:journalctl@ceph.osd.10.smithi189.stdout:Mar 24 03:21:23 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-osd-10[61294]: 2024-03-24T03:21:23.729+0000 7f804e31f700 -1 osd.10 127 heartbeat_check: no reply from 172.21.15.189:6812 osd.4 since back 2024-03-24T03:20:33.049108+0000 front 2024-03-24T03:20:33.049154+0000 (oldest deadline 2024-03-24T03:20:55.949290+0000) 2024-03-24T03:21:24.205 INFO:journalctl@ceph.osd.10.smithi189.stdout:Mar 24 03:21:23 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-osd-10[61294]: 2024-03-24T03:21:23.729+0000 7f804e31f700 -1 osd.10 127 heartbeat_check: no reply from 172.21.15.193:6812 osd.5 since back 2024-03-24T03:20:41.750894+0000 front 2024-03-24T03:20:41.750949+0000 (oldest deadline 2024-03-24T03:21:04.050717+0000) 2024-03-24T03:21:24.205 INFO:journalctl@ceph.osd.10.smithi189.stdout:Mar 24 03:21:23 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-osd-10[61294]: 2024-03-24T03:21:23.729+0000 7f804e31f700 -1 osd.10 127 heartbeat_check: no reply from 172.21.15.118:6822 osd.6 since back 2024-03-24T03:20:50.452401+0000 front 2024-03-24T03:20:50.452443+0000 (oldest deadline 2024-03-24T03:21:12.752167+0000) 2024-03-24T03:21:24.210 INFO:journalctl@ceph.osd.11.smithi193.stdout:Mar 24 03:21:23 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-osd-11[62452]: 2024-03-24T03:21:23.918+0000 7f0edd521700 -1 osd.11 127 heartbeat_check: no reply from 172.21.15.118:6806 osd.0 since back 2024-03-24T03:19:59.095638+0000 front 2024-03-24T03:19:59.095627+0000 (oldest deadline 2024-03-24T03:20:20.195530+0000) 2024-03-24T03:21:24.210 INFO:journalctl@ceph.osd.11.smithi193.stdout:Mar 24 03:21:23 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-osd-11[62452]: 2024-03-24T03:21:23.918+0000 7f0edd521700 -1 osd.11 127 heartbeat_check: no reply from 172.21.15.189:6804 osd.1 since back 2024-03-24T03:20:07.399753+0000 front 2024-03-24T03:20:07.399812+0000 (oldest deadline 2024-03-24T03:20:32.099923+0000) 2024-03-24T03:21:24.211 INFO:journalctl@ceph.osd.11.smithi193.stdout:Mar 24 03:21:23 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-osd-11[62452]: 2024-03-24T03:21:23.918+0000 7f0edd521700 -1 osd.11 127 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-03-24T03:20:12.100299+0000 front 2024-03-24T03:20:12.100348+0000 (oldest deadline 2024-03-24T03:20:38.000441+0000) 2024-03-24T03:21:24.211 INFO:journalctl@ceph.osd.11.smithi193.stdout:Mar 24 03:21:23 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-osd-11[62452]: 2024-03-24T03:21:23.918+0000 7f0edd521700 -1 osd.11 127 heartbeat_check: no reply from 172.21.15.118:6814 osd.3 since back 2024-03-24T03:20:25.002506+0000 front 2024-03-24T03:20:25.002576+0000 (oldest deadline 2024-03-24T03:20:46.702390+0000) 2024-03-24T03:21:24.211 INFO:journalctl@ceph.osd.11.smithi193.stdout:Mar 24 03:21:23 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-osd-11[62452]: 2024-03-24T03:21:23.918+0000 7f0edd521700 -1 osd.11 127 heartbeat_check: no reply from 172.21.15.189:6812 osd.4 since back 2024-03-24T03:20:32.905105+0000 front 2024-03-24T03:20:32.905067+0000 (oldest deadline 2024-03-24T03:20:55.204998+0000) 2024-03-24T03:21:24.211 INFO:journalctl@ceph.osd.11.smithi193.stdout:Mar 24 03:21:23 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-osd-11[62452]: 2024-03-24T03:21:23.918+0000 7f0edd521700 -1 osd.11 127 heartbeat_check: no reply from 172.21.15.193:6812 osd.5 since back 2024-03-24T03:20:42.106727+0000 front 2024-03-24T03:20:42.106844+0000 (oldest deadline 2024-03-24T03:21:05.006783+0000) 2024-03-24T03:21:24.211 INFO:journalctl@ceph.osd.11.smithi193.stdout:Mar 24 03:21:23 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-osd-11[62452]: 2024-03-24T03:21:23.918+0000 7f0edd521700 -1 osd.11 127 heartbeat_check: no reply from 172.21.15.189:6820 osd.7 since back 2024-03-24T03:20:58.810942+0000 front 2024-03-24T03:20:58.810766+0000 (oldest deadline 2024-03-24T03:21:21.110690+0000) 2024-03-24T03:21:24.737 INFO:journalctl@ceph.osd.10.smithi189.stdout:Mar 24 03:21:24 smithi189 podman[117692]: 2024-03-24 03:21:24.473019442 +0000 UTC m=+5.572712863 container died 0775da42b0995df7247c5eab7ef52e981e00f415de4d44e6a2fa011ddf92ca30 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-d79a5420-e971-11ee-95cd-87774f69a715-osd-10, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240319, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_CLEAN=True, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0) 2024-03-24T03:21:25.096 INFO:journalctl@ceph.osd.10.smithi189.stdout:Mar 24 03:21:24 smithi189 podman[117692]: 2024-03-24 03:21:24.837897302 +0000 UTC m=+5.937590722 container cleanup 0775da42b0995df7247c5eab7ef52e981e00f415de4d44e6a2fa011ddf92ca30 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-d79a5420-e971-11ee-95cd-87774f69a715-osd-10, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.build-date=20240319, org.label-schema.license=GPLv2, GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, maintainer=Guillaume Abrioux , RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=) 2024-03-24T03:21:25.097 INFO:journalctl@ceph.osd.10.smithi189.stdout:Mar 24 03:21:24 smithi189 bash[117692]: ceph-d79a5420-e971-11ee-95cd-87774f69a715-osd-10 2024-03-24T03:21:25.210 INFO:journalctl@ceph.osd.11.smithi193.stdout:Mar 24 03:21:24 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-osd-11[62452]: 2024-03-24T03:21:24.870+0000 7f0edd521700 -1 osd.11 127 heartbeat_check: no reply from 172.21.15.118:6806 osd.0 since back 2024-03-24T03:19:59.095638+0000 front 2024-03-24T03:19:59.095627+0000 (oldest deadline 2024-03-24T03:20:20.195530+0000) 2024-03-24T03:21:25.211 INFO:journalctl@ceph.osd.11.smithi193.stdout:Mar 24 03:21:24 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-osd-11[62452]: 2024-03-24T03:21:24.870+0000 7f0edd521700 -1 osd.11 127 heartbeat_check: no reply from 172.21.15.189:6804 osd.1 since back 2024-03-24T03:20:07.399753+0000 front 2024-03-24T03:20:07.399812+0000 (oldest deadline 2024-03-24T03:20:32.099923+0000) 2024-03-24T03:21:25.211 INFO:journalctl@ceph.osd.11.smithi193.stdout:Mar 24 03:21:24 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-osd-11[62452]: 2024-03-24T03:21:24.870+0000 7f0edd521700 -1 osd.11 127 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-03-24T03:20:12.100299+0000 front 2024-03-24T03:20:12.100348+0000 (oldest deadline 2024-03-24T03:20:38.000441+0000) 2024-03-24T03:21:25.211 INFO:journalctl@ceph.osd.11.smithi193.stdout:Mar 24 03:21:24 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-osd-11[62452]: 2024-03-24T03:21:24.870+0000 7f0edd521700 -1 osd.11 127 heartbeat_check: no reply from 172.21.15.118:6814 osd.3 since back 2024-03-24T03:20:25.002506+0000 front 2024-03-24T03:20:25.002576+0000 (oldest deadline 2024-03-24T03:20:46.702390+0000) 2024-03-24T03:21:25.211 INFO:journalctl@ceph.osd.11.smithi193.stdout:Mar 24 03:21:24 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-osd-11[62452]: 2024-03-24T03:21:24.870+0000 7f0edd521700 -1 osd.11 127 heartbeat_check: no reply from 172.21.15.189:6812 osd.4 since back 2024-03-24T03:20:32.905105+0000 front 2024-03-24T03:20:32.905067+0000 (oldest deadline 2024-03-24T03:20:55.204998+0000) 2024-03-24T03:21:25.211 INFO:journalctl@ceph.osd.11.smithi193.stdout:Mar 24 03:21:24 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-osd-11[62452]: 2024-03-24T03:21:24.870+0000 7f0edd521700 -1 osd.11 127 heartbeat_check: no reply from 172.21.15.193:6812 osd.5 since back 2024-03-24T03:20:42.106727+0000 front 2024-03-24T03:20:42.106844+0000 (oldest deadline 2024-03-24T03:21:05.006783+0000) 2024-03-24T03:21:25.211 INFO:journalctl@ceph.osd.11.smithi193.stdout:Mar 24 03:21:24 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-osd-11[62452]: 2024-03-24T03:21:24.870+0000 7f0edd521700 -1 osd.11 127 heartbeat_check: no reply from 172.21.15.189:6820 osd.7 since back 2024-03-24T03:20:58.810942+0000 front 2024-03-24T03:20:58.810766+0000 (oldest deadline 2024-03-24T03:21:21.110690+0000) 2024-03-24T03:21:25.442 INFO:journalctl@ceph.osd.10.smithi189.stdout:Mar 24 03:21:25 smithi189 podman[117709]: 2024-03-24 03:21:25.097416293 +0000 UTC m=+0.624787663 container remove 0775da42b0995df7247c5eab7ef52e981e00f415de4d44e6a2fa011ddf92ca30 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-d79a5420-e971-11ee-95cd-87774f69a715-osd-10, ceph=True, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.build-date=20240319) 2024-03-24T03:21:25.704 INFO:journalctl@ceph.osd.10.smithi189.stdout:Mar 24 03:21:25 smithi189 podman[117800]: 2024-03-24 03:21:25.345463084 +0000 UTC m=+0.017644301 image pull c664afe56bb01db4f9afa9532c20598e201e664f2e9ba2be50db0d52674a26d2 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc 2024-03-24T03:21:25.704 INFO:journalctl@ceph.osd.10.smithi189.stdout:Mar 24 03:21:25 smithi189 podman[117800]: 2024-03-24 03:21:25.473416518 +0000 UTC m=+0.145597733 container create 47e39c22f1f1cf9773fa5c916c462f268fa7837c51a75a0637f92fb041aed7a1 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-d79a5420-e971-11ee-95cd-87774f69a715-osd-10-deactivate, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, ceph=True, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240319, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2) 2024-03-24T03:21:25.704 INFO:journalctl@ceph.osd.10.smithi189.stdout:Mar 24 03:21:25 smithi189 podman[117800]: 2024-03-24 03:21:25.683995522 +0000 UTC m=+0.356176747 container init 47e39c22f1f1cf9773fa5c916c462f268fa7837c51a75a0637f92fb041aed7a1 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-d79a5420-e971-11ee-95cd-87774f69a715-osd-10-deactivate, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, ceph=True, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, GIT_CLEAN=True, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, org.label-schema.license=GPLv2, org.label-schema.build-date=20240319, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-24T03:21:25.704 INFO:journalctl@ceph.osd.10.smithi189.stdout:Mar 24 03:21:25 smithi189 podman[117800]: 2024-03-24 03:21:25.687194123 +0000 UTC m=+0.359375331 container start 47e39c22f1f1cf9773fa5c916c462f268fa7837c51a75a0637f92fb041aed7a1 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-d79a5420-e971-11ee-95cd-87774f69a715-osd-10-deactivate, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.license=GPLv2, ceph=True, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, org.label-schema.build-date=20240319, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , GIT_CLEAN=True, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, org.label-schema.vendor=CentOS) 2024-03-24T03:21:26.204 INFO:journalctl@ceph.osd.10.smithi189.stdout:Mar 24 03:21:25 smithi189 podman[117800]: 2024-03-24 03:21:25.751116716 +0000 UTC m=+0.423297944 container attach 47e39c22f1f1cf9773fa5c916c462f268fa7837c51a75a0637f92fb041aed7a1 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-d79a5420-e971-11ee-95cd-87774f69a715-osd-10-deactivate, org.label-schema.build-date=20240319, ceph=True, org.label-schema.vendor=CentOS, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0) 2024-03-24T03:21:26.204 INFO:journalctl@ceph.osd.10.smithi189.stdout:Mar 24 03:21:25 smithi189 podman[117800]: 2024-03-24 03:21:25.974736308 +0000 UTC m=+0.646917527 container died 47e39c22f1f1cf9773fa5c916c462f268fa7837c51a75a0637f92fb041aed7a1 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-d79a5420-e971-11ee-95cd-87774f69a715-osd-10-deactivate, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.license=GPLv2, org.label-schema.build-date=20240319, GIT_CLEAN=True, CEPH_POINT_RELEASE=) 2024-03-24T03:21:26.210 INFO:journalctl@ceph.osd.11.smithi193.stdout:Mar 24 03:21:25 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-osd-11[62452]: 2024-03-24T03:21:25.865+0000 7f0edd521700 -1 osd.11 127 heartbeat_check: no reply from 172.21.15.118:6806 osd.0 since back 2024-03-24T03:19:59.095638+0000 front 2024-03-24T03:19:59.095627+0000 (oldest deadline 2024-03-24T03:20:20.195530+0000) 2024-03-24T03:21:26.210 INFO:journalctl@ceph.osd.11.smithi193.stdout:Mar 24 03:21:25 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-osd-11[62452]: 2024-03-24T03:21:25.865+0000 7f0edd521700 -1 osd.11 127 heartbeat_check: no reply from 172.21.15.189:6804 osd.1 since back 2024-03-24T03:20:07.399753+0000 front 2024-03-24T03:20:07.399812+0000 (oldest deadline 2024-03-24T03:20:32.099923+0000) 2024-03-24T03:21:26.210 INFO:journalctl@ceph.osd.11.smithi193.stdout:Mar 24 03:21:25 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-osd-11[62452]: 2024-03-24T03:21:25.865+0000 7f0edd521700 -1 osd.11 127 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-03-24T03:20:12.100299+0000 front 2024-03-24T03:20:12.100348+0000 (oldest deadline 2024-03-24T03:20:38.000441+0000) 2024-03-24T03:21:26.211 INFO:journalctl@ceph.osd.11.smithi193.stdout:Mar 24 03:21:25 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-osd-11[62452]: 2024-03-24T03:21:25.865+0000 7f0edd521700 -1 osd.11 127 heartbeat_check: no reply from 172.21.15.118:6814 osd.3 since back 2024-03-24T03:20:25.002506+0000 front 2024-03-24T03:20:25.002576+0000 (oldest deadline 2024-03-24T03:20:46.702390+0000) 2024-03-24T03:21:26.211 INFO:journalctl@ceph.osd.11.smithi193.stdout:Mar 24 03:21:25 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-osd-11[62452]: 2024-03-24T03:21:25.865+0000 7f0edd521700 -1 osd.11 127 heartbeat_check: no reply from 172.21.15.189:6812 osd.4 since back 2024-03-24T03:20:32.905105+0000 front 2024-03-24T03:20:32.905067+0000 (oldest deadline 2024-03-24T03:20:55.204998+0000) 2024-03-24T03:21:26.211 INFO:journalctl@ceph.osd.11.smithi193.stdout:Mar 24 03:21:25 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-osd-11[62452]: 2024-03-24T03:21:25.865+0000 7f0edd521700 -1 osd.11 127 heartbeat_check: no reply from 172.21.15.193:6812 osd.5 since back 2024-03-24T03:20:42.106727+0000 front 2024-03-24T03:20:42.106844+0000 (oldest deadline 2024-03-24T03:21:05.006783+0000) 2024-03-24T03:21:26.211 INFO:journalctl@ceph.osd.11.smithi193.stdout:Mar 24 03:21:25 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-osd-11[62452]: 2024-03-24T03:21:25.865+0000 7f0edd521700 -1 osd.11 127 heartbeat_check: no reply from 172.21.15.189:6820 osd.7 since back 2024-03-24T03:20:58.810942+0000 front 2024-03-24T03:20:58.810766+0000 (oldest deadline 2024-03-24T03:21:21.110690+0000) 2024-03-24T03:21:26.573 DEBUG:teuthology.orchestra.run.smithi189:> sudo pkill -f 'journalctl -f -n 0 -u ceph-d79a5420-e971-11ee-95cd-87774f69a715@osd.10.service' 2024-03-24T03:21:26.612 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-24T03:21:26.613 INFO:tasks.cephadm.osd.10:Stopped osd.10 2024-03-24T03:21:26.613 INFO:tasks.cephadm.osd.11:Stopping osd.11... 2024-03-24T03:21:26.613 DEBUG:teuthology.orchestra.run.smithi193:> sudo systemctl stop ceph-d79a5420-e971-11ee-95cd-87774f69a715@osd.11 2024-03-24T03:21:26.905 INFO:journalctl@ceph.osd.11.smithi193.stdout:Mar 24 03:21:26 smithi193 systemd[1]: Stopping Ceph osd.11 for d79a5420-e971-11ee-95cd-87774f69a715... 2024-03-24T03:21:27.211 INFO:journalctl@ceph.osd.11.smithi193.stdout:Mar 24 03:21:26 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-osd-11[62452]: 2024-03-24T03:21:26.905+0000 7f0edd521700 -1 osd.11 127 heartbeat_check: no reply from 172.21.15.118:6806 osd.0 since back 2024-03-24T03:19:59.095638+0000 front 2024-03-24T03:19:59.095627+0000 (oldest deadline 2024-03-24T03:20:20.195530+0000) 2024-03-24T03:21:27.211 INFO:journalctl@ceph.osd.11.smithi193.stdout:Mar 24 03:21:26 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-osd-11[62452]: 2024-03-24T03:21:26.905+0000 7f0edd521700 -1 osd.11 127 heartbeat_check: no reply from 172.21.15.189:6804 osd.1 since back 2024-03-24T03:20:07.399753+0000 front 2024-03-24T03:20:07.399812+0000 (oldest deadline 2024-03-24T03:20:32.099923+0000) 2024-03-24T03:21:27.211 INFO:journalctl@ceph.osd.11.smithi193.stdout:Mar 24 03:21:26 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-osd-11[62452]: 2024-03-24T03:21:26.905+0000 7f0edd521700 -1 osd.11 127 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-03-24T03:20:12.100299+0000 front 2024-03-24T03:20:12.100348+0000 (oldest deadline 2024-03-24T03:20:38.000441+0000) 2024-03-24T03:21:27.211 INFO:journalctl@ceph.osd.11.smithi193.stdout:Mar 24 03:21:26 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-osd-11[62452]: 2024-03-24T03:21:26.905+0000 7f0edd521700 -1 osd.11 127 heartbeat_check: no reply from 172.21.15.118:6814 osd.3 since back 2024-03-24T03:20:25.002506+0000 front 2024-03-24T03:20:25.002576+0000 (oldest deadline 2024-03-24T03:20:46.702390+0000) 2024-03-24T03:21:27.211 INFO:journalctl@ceph.osd.11.smithi193.stdout:Mar 24 03:21:26 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-osd-11[62452]: 2024-03-24T03:21:26.905+0000 7f0edd521700 -1 osd.11 127 heartbeat_check: no reply from 172.21.15.189:6812 osd.4 since back 2024-03-24T03:20:32.905105+0000 front 2024-03-24T03:20:32.905067+0000 (oldest deadline 2024-03-24T03:20:55.204998+0000) 2024-03-24T03:21:27.211 INFO:journalctl@ceph.osd.11.smithi193.stdout:Mar 24 03:21:26 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-osd-11[62452]: 2024-03-24T03:21:26.905+0000 7f0edd521700 -1 osd.11 127 heartbeat_check: no reply from 172.21.15.193:6812 osd.5 since back 2024-03-24T03:20:42.106727+0000 front 2024-03-24T03:20:42.106844+0000 (oldest deadline 2024-03-24T03:21:05.006783+0000) 2024-03-24T03:21:27.212 INFO:journalctl@ceph.osd.11.smithi193.stdout:Mar 24 03:21:26 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-osd-11[62452]: 2024-03-24T03:21:26.905+0000 7f0edd521700 -1 osd.11 127 heartbeat_check: no reply from 172.21.15.189:6820 osd.7 since back 2024-03-24T03:20:58.810942+0000 front 2024-03-24T03:20:58.810766+0000 (oldest deadline 2024-03-24T03:21:21.110690+0000) 2024-03-24T03:21:27.212 INFO:journalctl@ceph.osd.11.smithi193.stdout:Mar 24 03:21:27 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-osd-11[62452]: 2024-03-24T03:21:27.013+0000 7f0ee0f2d700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.11 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-24T03:21:27.212 INFO:journalctl@ceph.osd.11.smithi193.stdout:Mar 24 03:21:27 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-osd-11[62452]: 2024-03-24T03:21:27.013+0000 7f0ee0f2d700 -1 osd.11 127 *** Got signal Terminated *** 2024-03-24T03:21:27.212 INFO:journalctl@ceph.osd.11.smithi193.stdout:Mar 24 03:21:27 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-osd-11[62452]: 2024-03-24T03:21:27.013+0000 7f0ee0f2d700 -1 osd.11 127 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-24T03:21:28.210 INFO:journalctl@ceph.osd.11.smithi193.stdout:Mar 24 03:21:27 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-osd-11[62452]: 2024-03-24T03:21:27.857+0000 7f0edd521700 -1 osd.11 127 heartbeat_check: no reply from 172.21.15.118:6806 osd.0 since back 2024-03-24T03:19:59.095638+0000 front 2024-03-24T03:19:59.095627+0000 (oldest deadline 2024-03-24T03:20:20.195530+0000) 2024-03-24T03:21:28.211 INFO:journalctl@ceph.osd.11.smithi193.stdout:Mar 24 03:21:27 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-osd-11[62452]: 2024-03-24T03:21:27.857+0000 7f0edd521700 -1 osd.11 127 heartbeat_check: no reply from 172.21.15.189:6804 osd.1 since back 2024-03-24T03:20:07.399753+0000 front 2024-03-24T03:20:07.399812+0000 (oldest deadline 2024-03-24T03:20:32.099923+0000) 2024-03-24T03:21:28.211 INFO:journalctl@ceph.osd.11.smithi193.stdout:Mar 24 03:21:27 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-osd-11[62452]: 2024-03-24T03:21:27.857+0000 7f0edd521700 -1 osd.11 127 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-03-24T03:20:12.100299+0000 front 2024-03-24T03:20:12.100348+0000 (oldest deadline 2024-03-24T03:20:38.000441+0000) 2024-03-24T03:21:28.211 INFO:journalctl@ceph.osd.11.smithi193.stdout:Mar 24 03:21:27 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-osd-11[62452]: 2024-03-24T03:21:27.857+0000 7f0edd521700 -1 osd.11 127 heartbeat_check: no reply from 172.21.15.118:6814 osd.3 since back 2024-03-24T03:20:25.002506+0000 front 2024-03-24T03:20:25.002576+0000 (oldest deadline 2024-03-24T03:20:46.702390+0000) 2024-03-24T03:21:28.211 INFO:journalctl@ceph.osd.11.smithi193.stdout:Mar 24 03:21:27 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-osd-11[62452]: 2024-03-24T03:21:27.857+0000 7f0edd521700 -1 osd.11 127 heartbeat_check: no reply from 172.21.15.189:6812 osd.4 since back 2024-03-24T03:20:32.905105+0000 front 2024-03-24T03:20:32.905067+0000 (oldest deadline 2024-03-24T03:20:55.204998+0000) 2024-03-24T03:21:28.211 INFO:journalctl@ceph.osd.11.smithi193.stdout:Mar 24 03:21:27 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-osd-11[62452]: 2024-03-24T03:21:27.857+0000 7f0edd521700 -1 osd.11 127 heartbeat_check: no reply from 172.21.15.193:6812 osd.5 since back 2024-03-24T03:20:42.106727+0000 front 2024-03-24T03:20:42.106844+0000 (oldest deadline 2024-03-24T03:21:05.006783+0000) 2024-03-24T03:21:28.211 INFO:journalctl@ceph.osd.11.smithi193.stdout:Mar 24 03:21:27 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-osd-11[62452]: 2024-03-24T03:21:27.857+0000 7f0edd521700 -1 osd.11 127 heartbeat_check: no reply from 172.21.15.189:6820 osd.7 since back 2024-03-24T03:20:58.810942+0000 front 2024-03-24T03:20:58.810766+0000 (oldest deadline 2024-03-24T03:21:21.110690+0000) 2024-03-24T03:21:29.211 INFO:journalctl@ceph.osd.11.smithi193.stdout:Mar 24 03:21:28 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-osd-11[62452]: 2024-03-24T03:21:28.879+0000 7f0edd521700 -1 osd.11 127 heartbeat_check: no reply from 172.21.15.118:6806 osd.0 since back 2024-03-24T03:19:59.095638+0000 front 2024-03-24T03:19:59.095627+0000 (oldest deadline 2024-03-24T03:20:20.195530+0000) 2024-03-24T03:21:29.211 INFO:journalctl@ceph.osd.11.smithi193.stdout:Mar 24 03:21:28 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-osd-11[62452]: 2024-03-24T03:21:28.879+0000 7f0edd521700 -1 osd.11 127 heartbeat_check: no reply from 172.21.15.189:6804 osd.1 since back 2024-03-24T03:20:07.399753+0000 front 2024-03-24T03:20:07.399812+0000 (oldest deadline 2024-03-24T03:20:32.099923+0000) 2024-03-24T03:21:29.211 INFO:journalctl@ceph.osd.11.smithi193.stdout:Mar 24 03:21:28 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-osd-11[62452]: 2024-03-24T03:21:28.879+0000 7f0edd521700 -1 osd.11 127 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-03-24T03:20:12.100299+0000 front 2024-03-24T03:20:12.100348+0000 (oldest deadline 2024-03-24T03:20:38.000441+0000) 2024-03-24T03:21:29.211 INFO:journalctl@ceph.osd.11.smithi193.stdout:Mar 24 03:21:28 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-osd-11[62452]: 2024-03-24T03:21:28.879+0000 7f0edd521700 -1 osd.11 127 heartbeat_check: no reply from 172.21.15.118:6814 osd.3 since back 2024-03-24T03:20:25.002506+0000 front 2024-03-24T03:20:25.002576+0000 (oldest deadline 2024-03-24T03:20:46.702390+0000) 2024-03-24T03:21:29.211 INFO:journalctl@ceph.osd.11.smithi193.stdout:Mar 24 03:21:28 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-osd-11[62452]: 2024-03-24T03:21:28.879+0000 7f0edd521700 -1 osd.11 127 heartbeat_check: no reply from 172.21.15.189:6812 osd.4 since back 2024-03-24T03:20:32.905105+0000 front 2024-03-24T03:20:32.905067+0000 (oldest deadline 2024-03-24T03:20:55.204998+0000) 2024-03-24T03:21:29.211 INFO:journalctl@ceph.osd.11.smithi193.stdout:Mar 24 03:21:28 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-osd-11[62452]: 2024-03-24T03:21:28.879+0000 7f0edd521700 -1 osd.11 127 heartbeat_check: no reply from 172.21.15.193:6812 osd.5 since back 2024-03-24T03:20:42.106727+0000 front 2024-03-24T03:20:42.106844+0000 (oldest deadline 2024-03-24T03:21:05.006783+0000) 2024-03-24T03:21:29.211 INFO:journalctl@ceph.osd.11.smithi193.stdout:Mar 24 03:21:28 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-osd-11[62452]: 2024-03-24T03:21:28.879+0000 7f0edd521700 -1 osd.11 127 heartbeat_check: no reply from 172.21.15.189:6820 osd.7 since back 2024-03-24T03:20:58.810942+0000 front 2024-03-24T03:20:58.810766+0000 (oldest deadline 2024-03-24T03:21:21.110690+0000) 2024-03-24T03:21:30.210 INFO:journalctl@ceph.osd.11.smithi193.stdout:Mar 24 03:21:29 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-osd-11[62452]: 2024-03-24T03:21:29.898+0000 7f0edd521700 -1 osd.11 127 heartbeat_check: no reply from 172.21.15.118:6806 osd.0 since back 2024-03-24T03:19:59.095638+0000 front 2024-03-24T03:19:59.095627+0000 (oldest deadline 2024-03-24T03:20:20.195530+0000) 2024-03-24T03:21:30.211 INFO:journalctl@ceph.osd.11.smithi193.stdout:Mar 24 03:21:29 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-osd-11[62452]: 2024-03-24T03:21:29.898+0000 7f0edd521700 -1 osd.11 127 heartbeat_check: no reply from 172.21.15.189:6804 osd.1 since back 2024-03-24T03:20:07.399753+0000 front 2024-03-24T03:20:07.399812+0000 (oldest deadline 2024-03-24T03:20:32.099923+0000) 2024-03-24T03:21:30.211 INFO:journalctl@ceph.osd.11.smithi193.stdout:Mar 24 03:21:29 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-osd-11[62452]: 2024-03-24T03:21:29.898+0000 7f0edd521700 -1 osd.11 127 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-03-24T03:20:12.100299+0000 front 2024-03-24T03:20:12.100348+0000 (oldest deadline 2024-03-24T03:20:38.000441+0000) 2024-03-24T03:21:30.211 INFO:journalctl@ceph.osd.11.smithi193.stdout:Mar 24 03:21:29 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-osd-11[62452]: 2024-03-24T03:21:29.898+0000 7f0edd521700 -1 osd.11 127 heartbeat_check: no reply from 172.21.15.118:6814 osd.3 since back 2024-03-24T03:20:25.002506+0000 front 2024-03-24T03:20:25.002576+0000 (oldest deadline 2024-03-24T03:20:46.702390+0000) 2024-03-24T03:21:30.211 INFO:journalctl@ceph.osd.11.smithi193.stdout:Mar 24 03:21:29 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-osd-11[62452]: 2024-03-24T03:21:29.898+0000 7f0edd521700 -1 osd.11 127 heartbeat_check: no reply from 172.21.15.189:6812 osd.4 since back 2024-03-24T03:20:32.905105+0000 front 2024-03-24T03:20:32.905067+0000 (oldest deadline 2024-03-24T03:20:55.204998+0000) 2024-03-24T03:21:30.211 INFO:journalctl@ceph.osd.11.smithi193.stdout:Mar 24 03:21:29 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-osd-11[62452]: 2024-03-24T03:21:29.898+0000 7f0edd521700 -1 osd.11 127 heartbeat_check: no reply from 172.21.15.193:6812 osd.5 since back 2024-03-24T03:20:42.106727+0000 front 2024-03-24T03:20:42.106844+0000 (oldest deadline 2024-03-24T03:21:05.006783+0000) 2024-03-24T03:21:30.212 INFO:journalctl@ceph.osd.11.smithi193.stdout:Mar 24 03:21:29 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-osd-11[62452]: 2024-03-24T03:21:29.898+0000 7f0edd521700 -1 osd.11 127 heartbeat_check: no reply from 172.21.15.189:6820 osd.7 since back 2024-03-24T03:20:58.810942+0000 front 2024-03-24T03:20:58.810766+0000 (oldest deadline 2024-03-24T03:21:21.110690+0000) 2024-03-24T03:21:31.210 INFO:journalctl@ceph.osd.11.smithi193.stdout:Mar 24 03:21:30 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-osd-11[62452]: 2024-03-24T03:21:30.867+0000 7f0edd521700 -1 osd.11 127 heartbeat_check: no reply from 172.21.15.118:6806 osd.0 since back 2024-03-24T03:19:59.095638+0000 front 2024-03-24T03:19:59.095627+0000 (oldest deadline 2024-03-24T03:20:20.195530+0000) 2024-03-24T03:21:31.211 INFO:journalctl@ceph.osd.11.smithi193.stdout:Mar 24 03:21:30 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-osd-11[62452]: 2024-03-24T03:21:30.867+0000 7f0edd521700 -1 osd.11 127 heartbeat_check: no reply from 172.21.15.189:6804 osd.1 since back 2024-03-24T03:20:07.399753+0000 front 2024-03-24T03:20:07.399812+0000 (oldest deadline 2024-03-24T03:20:32.099923+0000) 2024-03-24T03:21:31.211 INFO:journalctl@ceph.osd.11.smithi193.stdout:Mar 24 03:21:30 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-osd-11[62452]: 2024-03-24T03:21:30.867+0000 7f0edd521700 -1 osd.11 127 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-03-24T03:20:12.100299+0000 front 2024-03-24T03:20:12.100348+0000 (oldest deadline 2024-03-24T03:20:38.000441+0000) 2024-03-24T03:21:31.211 INFO:journalctl@ceph.osd.11.smithi193.stdout:Mar 24 03:21:30 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-osd-11[62452]: 2024-03-24T03:21:30.867+0000 7f0edd521700 -1 osd.11 127 heartbeat_check: no reply from 172.21.15.118:6814 osd.3 since back 2024-03-24T03:20:25.002506+0000 front 2024-03-24T03:20:25.002576+0000 (oldest deadline 2024-03-24T03:20:46.702390+0000) 2024-03-24T03:21:31.211 INFO:journalctl@ceph.osd.11.smithi193.stdout:Mar 24 03:21:30 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-osd-11[62452]: 2024-03-24T03:21:30.867+0000 7f0edd521700 -1 osd.11 127 heartbeat_check: no reply from 172.21.15.189:6812 osd.4 since back 2024-03-24T03:20:32.905105+0000 front 2024-03-24T03:20:32.905067+0000 (oldest deadline 2024-03-24T03:20:55.204998+0000) 2024-03-24T03:21:31.211 INFO:journalctl@ceph.osd.11.smithi193.stdout:Mar 24 03:21:30 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-osd-11[62452]: 2024-03-24T03:21:30.867+0000 7f0edd521700 -1 osd.11 127 heartbeat_check: no reply from 172.21.15.193:6812 osd.5 since back 2024-03-24T03:20:42.106727+0000 front 2024-03-24T03:20:42.106844+0000 (oldest deadline 2024-03-24T03:21:05.006783+0000) 2024-03-24T03:21:31.211 INFO:journalctl@ceph.osd.11.smithi193.stdout:Mar 24 03:21:30 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-osd-11[62452]: 2024-03-24T03:21:30.867+0000 7f0edd521700 -1 osd.11 127 heartbeat_check: no reply from 172.21.15.189:6820 osd.7 since back 2024-03-24T03:20:58.810942+0000 front 2024-03-24T03:20:58.810766+0000 (oldest deadline 2024-03-24T03:21:21.110690+0000) 2024-03-24T03:21:32.210 INFO:journalctl@ceph.osd.11.smithi193.stdout:Mar 24 03:21:31 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-osd-11[62452]: 2024-03-24T03:21:31.832+0000 7f0edd521700 -1 osd.11 127 heartbeat_check: no reply from 172.21.15.118:6806 osd.0 since back 2024-03-24T03:19:59.095638+0000 front 2024-03-24T03:19:59.095627+0000 (oldest deadline 2024-03-24T03:20:20.195530+0000) 2024-03-24T03:21:32.211 INFO:journalctl@ceph.osd.11.smithi193.stdout:Mar 24 03:21:31 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-osd-11[62452]: 2024-03-24T03:21:31.832+0000 7f0edd521700 -1 osd.11 127 heartbeat_check: no reply from 172.21.15.189:6804 osd.1 since back 2024-03-24T03:20:07.399753+0000 front 2024-03-24T03:20:07.399812+0000 (oldest deadline 2024-03-24T03:20:32.099923+0000) 2024-03-24T03:21:32.211 INFO:journalctl@ceph.osd.11.smithi193.stdout:Mar 24 03:21:31 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-osd-11[62452]: 2024-03-24T03:21:31.832+0000 7f0edd521700 -1 osd.11 127 heartbeat_check: no reply from 172.21.15.193:6804 osd.2 since back 2024-03-24T03:20:12.100299+0000 front 2024-03-24T03:20:12.100348+0000 (oldest deadline 2024-03-24T03:20:38.000441+0000) 2024-03-24T03:21:32.211 INFO:journalctl@ceph.osd.11.smithi193.stdout:Mar 24 03:21:31 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-osd-11[62452]: 2024-03-24T03:21:31.832+0000 7f0edd521700 -1 osd.11 127 heartbeat_check: no reply from 172.21.15.118:6814 osd.3 since back 2024-03-24T03:20:25.002506+0000 front 2024-03-24T03:20:25.002576+0000 (oldest deadline 2024-03-24T03:20:46.702390+0000) 2024-03-24T03:21:32.211 INFO:journalctl@ceph.osd.11.smithi193.stdout:Mar 24 03:21:31 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-osd-11[62452]: 2024-03-24T03:21:31.832+0000 7f0edd521700 -1 osd.11 127 heartbeat_check: no reply from 172.21.15.189:6812 osd.4 since back 2024-03-24T03:20:32.905105+0000 front 2024-03-24T03:20:32.905067+0000 (oldest deadline 2024-03-24T03:20:55.204998+0000) 2024-03-24T03:21:32.211 INFO:journalctl@ceph.osd.11.smithi193.stdout:Mar 24 03:21:31 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-osd-11[62452]: 2024-03-24T03:21:31.832+0000 7f0edd521700 -1 osd.11 127 heartbeat_check: no reply from 172.21.15.193:6812 osd.5 since back 2024-03-24T03:20:42.106727+0000 front 2024-03-24T03:20:42.106844+0000 (oldest deadline 2024-03-24T03:21:05.006783+0000) 2024-03-24T03:21:32.211 INFO:journalctl@ceph.osd.11.smithi193.stdout:Mar 24 03:21:31 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-osd-11[62452]: 2024-03-24T03:21:31.832+0000 7f0edd521700 -1 osd.11 127 heartbeat_check: no reply from 172.21.15.189:6820 osd.7 since back 2024-03-24T03:20:58.810942+0000 front 2024-03-24T03:20:58.810766+0000 (oldest deadline 2024-03-24T03:21:21.110690+0000) 2024-03-24T03:21:32.212 INFO:journalctl@ceph.osd.11.smithi193.stdout:Mar 24 03:21:31 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-osd-11[62452]: 2024-03-24T03:21:31.832+0000 7f0edd521700 -1 osd.11 127 heartbeat_check: no reply from 172.21.15.193:6820 osd.8 since back 2024-03-24T03:21:06.411769+0000 front 2024-03-24T03:21:06.411821+0000 (oldest deadline 2024-03-24T03:21:31.711821+0000) 2024-03-24T03:21:32.710 INFO:journalctl@ceph.osd.11.smithi193.stdout:Mar 24 03:21:32 smithi193 podman[120970]: 2024-03-24 03:21:32.359065987 +0000 UTC m=+5.552205460 container died 175169a7efdef1d1fd194bee2aa3933c6ebef49776f1d49e8060f290744229eb (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-d79a5420-e971-11ee-95cd-87774f69a715-osd-11, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240319, ceph=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, CEPH_POINT_RELEASE=) 2024-03-24T03:21:32.981 INFO:journalctl@ceph.osd.11.smithi193.stdout:Mar 24 03:21:32 smithi193 podman[120970]: 2024-03-24 03:21:32.730518846 +0000 UTC m=+5.923658324 container cleanup 175169a7efdef1d1fd194bee2aa3933c6ebef49776f1d49e8060f290744229eb (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-d79a5420-e971-11ee-95cd-87774f69a715-osd-11, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, io.buildah.version=1.33.5, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240319, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, ceph=True, GIT_BRANCH=HEAD) 2024-03-24T03:21:32.981 INFO:journalctl@ceph.osd.11.smithi193.stdout:Mar 24 03:21:32 smithi193 bash[120970]: ceph-d79a5420-e971-11ee-95cd-87774f69a715-osd-11 2024-03-24T03:21:33.296 INFO:journalctl@ceph.osd.11.smithi193.stdout:Mar 24 03:21:32 smithi193 podman[120987]: 2024-03-24 03:21:32.981947863 +0000 UTC m=+0.617833565 container remove 175169a7efdef1d1fd194bee2aa3933c6ebef49776f1d49e8060f290744229eb (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-d79a5420-e971-11ee-95cd-87774f69a715-osd-11, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, GIT_BRANCH=HEAD, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240319, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c) 2024-03-24T03:21:33.584 INFO:journalctl@ceph.osd.11.smithi193.stdout:Mar 24 03:21:33 smithi193 podman[121079]: 2024-03-24 03:21:33.199589562 +0000 UTC m=+0.017941876 image pull c664afe56bb01db4f9afa9532c20598e201e664f2e9ba2be50db0d52674a26d2 quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc 2024-03-24T03:21:33.585 INFO:journalctl@ceph.osd.11.smithi193.stdout:Mar 24 03:21:33 smithi193 podman[121079]: 2024-03-24 03:21:33.299457159 +0000 UTC m=+0.117809467 container create 30960bbf96b1c85d2c839f92100c8c86f0875f926e37fb8d53625e15ff555258 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-d79a5420-e971-11ee-95cd-87774f69a715-osd-11-deactivate, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.license=GPLv2, GIT_CLEAN=True, org.label-schema.build-date=20240319) 2024-03-24T03:21:33.585 INFO:journalctl@ceph.osd.11.smithi193.stdout:Mar 24 03:21:33 smithi193 podman[121079]: 2024-03-24 03:21:33.493747283 +0000 UTC m=+0.312099614 container init 30960bbf96b1c85d2c839f92100c8c86f0875f926e37fb8d53625e15ff555258 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-d79a5420-e971-11ee-95cd-87774f69a715-osd-11-deactivate, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240319, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.vendor=CentOS) 2024-03-24T03:21:33.585 INFO:journalctl@ceph.osd.11.smithi193.stdout:Mar 24 03:21:33 smithi193 podman[121079]: 2024-03-24 03:21:33.497229578 +0000 UTC m=+0.315581912 container start 30960bbf96b1c85d2c839f92100c8c86f0875f926e37fb8d53625e15ff555258 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-d79a5420-e971-11ee-95cd-87774f69a715-osd-11-deactivate, GIT_BRANCH=HEAD, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, org.label-schema.build-date=20240319, GIT_CLEAN=True, org.label-schema.schema-version=1.0) 2024-03-24T03:21:33.585 INFO:journalctl@ceph.osd.11.smithi193.stdout:Mar 24 03:21:33 smithi193 podman[121079]: 2024-03-24 03:21:33.585215622 +0000 UTC m=+0.403567961 container attach 30960bbf96b1c85d2c839f92100c8c86f0875f926e37fb8d53625e15ff555258 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-d79a5420-e971-11ee-95cd-87774f69a715-osd-11-deactivate, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, org.label-schema.license=GPLv2, GIT_CLEAN=True, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, CEPH_POINT_RELEASE=, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240319, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-24T03:21:33.961 INFO:journalctl@ceph.osd.11.smithi193.stdout:Mar 24 03:21:33 smithi193 podman[121079]: 2024-03-24 03:21:33.790635778 +0000 UTC m=+0.608988094 container died 30960bbf96b1c85d2c839f92100c8c86f0875f926e37fb8d53625e15ff555258 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-d79a5420-e971-11ee-95cd-87774f69a715-osd-11-deactivate, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, ceph=True, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, org.label-schema.license=GPLv2, org.label-schema.build-date=20240319) 2024-03-24T03:21:34.330 INFO:journalctl@ceph.osd.11.smithi193.stdout:Mar 24 03:21:34 smithi193 podman[121105]: 2024-03-24 03:21:34.313400756 +0000 UTC m=+0.517766438 container remove 30960bbf96b1c85d2c839f92100c8c86f0875f926e37fb8d53625e15ff555258 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-d79a5420-e971-11ee-95cd-87774f69a715-osd-11-deactivate, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, ceph=True, CEPH_POINT_RELEASE=, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240319, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.license=GPLv2, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30) 2024-03-24T03:21:34.335 DEBUG:teuthology.orchestra.run.smithi193:> sudo pkill -f 'journalctl -f -n 0 -u ceph-d79a5420-e971-11ee-95cd-87774f69a715@osd.11.service' 2024-03-24T03:21:34.375 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-24T03:21:34.375 INFO:tasks.cephadm.osd.11:Stopped osd.11 2024-03-24T03:21:34.375 INFO:tasks.cephadm.mds.a:Stopping mds.a... 2024-03-24T03:21:34.376 DEBUG:teuthology.orchestra.run.smithi118:> sudo systemctl stop ceph-d79a5420-e971-11ee-95cd-87774f69a715@mds.a 2024-03-24T03:21:34.908 INFO:journalctl@ceph.mds.a.smithi118.stdout:Mar 24 03:21:34 smithi118 systemd[1]: Stopping Ceph mds.a for d79a5420-e971-11ee-95cd-87774f69a715... 2024-03-24T03:21:35.196 INFO:journalctl@ceph.mds.a.smithi118.stdout:Mar 24 03:21:34 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-a[73967]: 2024-03-24T03:21:34.907+0000 7ff3a9800700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.a -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-24T03:21:35.196 INFO:journalctl@ceph.mds.a.smithi118.stdout:Mar 24 03:21:34 smithi118 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-a[73967]: 2024-03-24T03:21:34.907+0000 7ff3a9800700 -1 mds.a *** got signal Terminated *** 2024-03-24T03:21:37.789 INFO:journalctl@ceph.mds.a.smithi118.stdout:Mar 24 03:21:37 smithi118 podman[146720]: 2024-03-24 03:21:37.453995192 +0000 UTC m=+2.678684115 container died 8b122c08c5eea799f91ee96444ffb9f22e8135af2a327414fac0fa3f34694326 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-a, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240319, org.label-schema.vendor=CentOS, GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, ceph=True, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c) 2024-03-24T03:21:38.086 INFO:journalctl@ceph.mds.a.smithi118.stdout:Mar 24 03:21:37 smithi118 podman[146720]: 2024-03-24 03:21:37.893409743 +0000 UTC m=+3.118098660 container cleanup 8b122c08c5eea799f91ee96444ffb9f22e8135af2a327414fac0fa3f34694326 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-a, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, org.label-schema.build-date=20240319, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, GIT_CLEAN=True) 2024-03-24T03:21:38.086 INFO:journalctl@ceph.mds.a.smithi118.stdout:Mar 24 03:21:37 smithi118 bash[146720]: ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-a 2024-03-24T03:21:38.446 INFO:journalctl@ceph.mds.a.smithi118.stdout:Mar 24 03:21:38 smithi118 podman[146734]: 2024-03-24 03:21:38.085829765 +0000 UTC m=+0.559720809 container remove 8b122c08c5eea799f91ee96444ffb9f22e8135af2a327414fac0fa3f34694326 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-a, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.build-date=20240319, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, ceph=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-24T03:21:38.564 DEBUG:teuthology.orchestra.run.smithi118:> sudo pkill -f 'journalctl -f -n 0 -u ceph-d79a5420-e971-11ee-95cd-87774f69a715@mds.a.service' 2024-03-24T03:21:38.681 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-24T03:21:38.681 INFO:tasks.cephadm.mds.a:Stopped mds.a 2024-03-24T03:21:38.681 INFO:tasks.cephadm.mds.d:Stopping mds.d... 2024-03-24T03:21:38.681 DEBUG:teuthology.orchestra.run.smithi118:> sudo systemctl stop ceph-d79a5420-e971-11ee-95cd-87774f69a715@mds.d 2024-03-24T03:21:38.782 DEBUG:teuthology.orchestra.run.smithi118:> sudo pkill -f 'journalctl -f -n 0 -u ceph-d79a5420-e971-11ee-95cd-87774f69a715@mds.d.service' 2024-03-24T03:21:38.883 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-24T03:21:38.883 INFO:tasks.cephadm.mds.d:Stopped mds.d 2024-03-24T03:21:38.883 INFO:tasks.cephadm.mds.g:Stopping mds.g... 2024-03-24T03:21:38.883 DEBUG:teuthology.orchestra.run.smithi118:> sudo systemctl stop ceph-d79a5420-e971-11ee-95cd-87774f69a715@mds.g 2024-03-24T03:21:38.982 DEBUG:teuthology.orchestra.run.smithi118:> sudo pkill -f 'journalctl -f -n 0 -u ceph-d79a5420-e971-11ee-95cd-87774f69a715@mds.g.service' 2024-03-24T03:21:39.093 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-24T03:21:39.093 INFO:tasks.cephadm.mds.g:Stopped mds.g 2024-03-24T03:21:39.093 INFO:tasks.cephadm.mds.j:Stopping mds.j... 2024-03-24T03:21:39.093 DEBUG:teuthology.orchestra.run.smithi118:> sudo systemctl stop ceph-d79a5420-e971-11ee-95cd-87774f69a715@mds.j 2024-03-24T03:21:39.246 DEBUG:teuthology.orchestra.run.smithi118:> sudo pkill -f 'journalctl -f -n 0 -u ceph-d79a5420-e971-11ee-95cd-87774f69a715@mds.j.service' 2024-03-24T03:21:39.351 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-24T03:21:39.351 INFO:tasks.cephadm.mds.j:Stopped mds.j 2024-03-24T03:21:39.352 INFO:tasks.cephadm.mds.b:Stopping mds.b... 2024-03-24T03:21:39.352 DEBUG:teuthology.orchestra.run.smithi189:> sudo systemctl stop ceph-d79a5420-e971-11ee-95cd-87774f69a715@mds.b 2024-03-24T03:21:39.664 INFO:journalctl@ceph.mds.b.smithi189.stdout:Mar 24 03:21:39 smithi189 systemd[1]: Stopping Ceph mds.b for d79a5420-e971-11ee-95cd-87774f69a715... 2024-03-24T03:21:39.953 INFO:journalctl@ceph.mds.b.smithi189.stdout:Mar 24 03:21:39 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-b[66975]: 2024-03-24T03:21:39.664+0000 7fe7f9411700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.b -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-24T03:21:39.954 INFO:journalctl@ceph.mds.b.smithi189.stdout:Mar 24 03:21:39 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-b[66975]: 2024-03-24T03:21:39.664+0000 7fe7f9411700 -1 mds.b *** got signal Terminated *** 2024-03-24T03:21:41.703 INFO:journalctl@ceph.mds.b.smithi189.stdout:Mar 24 03:21:41 smithi189 podman[117905]: 2024-03-24 03:21:41.366437585 +0000 UTC m=+1.816265103 container died a872553a64922db566fa034c584c9492ef898423c008d98e4780f6e3fe67960b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-b, ceph=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.build-date=20240319, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_CLEAN=True, io.buildah.version=1.33.5) 2024-03-24T03:21:41.964 INFO:journalctl@ceph.mds.b.smithi189.stdout:Mar 24 03:21:41 smithi189 podman[117905]: 2024-03-24 03:21:41.788986578 +0000 UTC m=+2.238814087 container cleanup a872553a64922db566fa034c584c9492ef898423c008d98e4780f6e3fe67960b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-b, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.build-date=20240319, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, io.buildah.version=1.33.5, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2) 2024-03-24T03:21:41.965 INFO:journalctl@ceph.mds.b.smithi189.stdout:Mar 24 03:21:41 smithi189 bash[117905]: ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-b 2024-03-24T03:21:42.205 DEBUG:teuthology.orchestra.run.smithi189:> sudo pkill -f 'journalctl -f -n 0 -u ceph-d79a5420-e971-11ee-95cd-87774f69a715@mds.b.service' 2024-03-24T03:21:42.230 INFO:journalctl@ceph.mds.b.smithi189.stdout:Mar 24 03:21:41 smithi189 podman[117931]: 2024-03-24 03:21:41.965447929 +0000 UTC m=+0.597410808 container remove a872553a64922db566fa034c584c9492ef898423c008d98e4780f6e3fe67960b (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-b, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, ceph=True, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, org.label-schema.build-date=20240319, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_CLEAN=True, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-24T03:21:42.230 INFO:journalctl@ceph.mds.b.smithi189.stdout:Mar 24 03:21:42 smithi189 systemd[1]: ceph-d79a5420-e971-11ee-95cd-87774f69a715@mds.b.service: Deactivated successfully. 2024-03-24T03:21:42.230 INFO:journalctl@ceph.mds.b.smithi189.stdout:Mar 24 03:21:42 smithi189 systemd[1]: Stopped Ceph mds.b for d79a5420-e971-11ee-95cd-87774f69a715. 2024-03-24T03:21:42.230 INFO:journalctl@ceph.mds.b.smithi189.stdout:Mar 24 03:21:42 smithi189 systemd[1]: ceph-d79a5420-e971-11ee-95cd-87774f69a715@mds.b.service: Consumed 3h 5min 13.543s CPU time. 2024-03-24T03:21:42.243 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-24T03:21:42.243 INFO:tasks.cephadm.mds.b:Stopped mds.b 2024-03-24T03:21:42.244 INFO:tasks.cephadm.mds.e:Stopping mds.e... 2024-03-24T03:21:42.244 DEBUG:teuthology.orchestra.run.smithi189:> sudo systemctl stop ceph-d79a5420-e971-11ee-95cd-87774f69a715@mds.e 2024-03-24T03:21:42.567 INFO:journalctl@ceph.mds.e.smithi189.stdout:Mar 24 03:21:42 smithi189 systemd[1]: Stopping Ceph mds.e for d79a5420-e971-11ee-95cd-87774f69a715... 2024-03-24T03:21:42.954 INFO:journalctl@ceph.mds.e.smithi189.stdout:Mar 24 03:21:42 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-e[66361]: 2024-03-24T03:21:42.567+0000 7f206c1e7700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.e -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-24T03:21:42.954 INFO:journalctl@ceph.mds.e.smithi189.stdout:Mar 24 03:21:42 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-e[66361]: 2024-03-24T03:21:42.567+0000 7f206c1e7700 -1 mds.e *** got signal Terminated *** 2024-03-24T03:21:45.667 INFO:journalctl@ceph.mds.e.smithi189.stdout:Mar 24 03:21:45 smithi189 podman[118038]: 2024-03-24 03:21:45.384053103 +0000 UTC m=+2.919000675 container died 489bbba1e9002927b94f19e01c0fe7b3970bc1f9840a73d8747daae773b971db (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-e, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, org.label-schema.license=GPLv2, ceph=True, GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240319, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-03-24T03:21:45.953 INFO:journalctl@ceph.mds.e.smithi189.stdout:Mar 24 03:21:45 smithi189 podman[118038]: 2024-03-24 03:21:45.75094209 +0000 UTC m=+3.285889681 container cleanup 489bbba1e9002927b94f19e01c0fe7b3970bc1f9840a73d8747daae773b971db (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-e, org.label-schema.vendor=CentOS, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.build-date=20240319, GIT_CLEAN=True, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.schema-version=1.0, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD) 2024-03-24T03:21:45.954 INFO:journalctl@ceph.mds.e.smithi189.stdout:Mar 24 03:21:45 smithi189 bash[118038]: ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-e 2024-03-24T03:21:46.119 DEBUG:teuthology.orchestra.run.smithi189:> sudo pkill -f 'journalctl -f -n 0 -u ceph-d79a5420-e971-11ee-95cd-87774f69a715@mds.e.service' 2024-03-24T03:21:46.154 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-24T03:21:46.154 INFO:tasks.cephadm.mds.e:Stopped mds.e 2024-03-24T03:21:46.155 INFO:tasks.cephadm.mds.h:Stopping mds.h... 2024-03-24T03:21:46.155 DEBUG:teuthology.orchestra.run.smithi189:> sudo systemctl stop ceph-d79a5420-e971-11ee-95cd-87774f69a715@mds.h 2024-03-24T03:21:46.226 DEBUG:teuthology.orchestra.run.smithi189:> sudo pkill -f 'journalctl -f -n 0 -u ceph-d79a5420-e971-11ee-95cd-87774f69a715@mds.h.service' 2024-03-24T03:21:46.300 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-24T03:21:46.300 INFO:tasks.cephadm.mds.h:Stopped mds.h 2024-03-24T03:21:46.301 INFO:tasks.cephadm.mds.k:Stopping mds.k... 2024-03-24T03:21:46.301 DEBUG:teuthology.orchestra.run.smithi189:> sudo systemctl stop ceph-d79a5420-e971-11ee-95cd-87774f69a715@mds.k 2024-03-24T03:21:46.604 INFO:journalctl@ceph.mds.k.smithi189.stdout:Mar 24 03:21:46 smithi189 systemd[1]: Stopping Ceph mds.k for d79a5420-e971-11ee-95cd-87774f69a715... 2024-03-24T03:21:46.954 INFO:journalctl@ceph.mds.k.smithi189.stdout:Mar 24 03:21:46 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-k[66038]: 2024-03-24T03:21:46.604+0000 7fbd3277e700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.k -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-24T03:21:46.954 INFO:journalctl@ceph.mds.k.smithi189.stdout:Mar 24 03:21:46 smithi189 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-k[66038]: 2024-03-24T03:21:46.604+0000 7fbd3277e700 -1 mds.k *** got signal Terminated *** 2024-03-24T03:21:49.699 INFO:journalctl@ceph.mds.k.smithi189.stdout:Mar 24 03:21:49 smithi189 podman[118193]: 2024-03-24 03:21:49.371644182 +0000 UTC m=+2.863155105 container died 89f5182d7c26e85a3b40978255d4bf9f4b2b1e9d17632ed765e7e31e55c907d4 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-k, GIT_CLEAN=True, org.label-schema.license=GPLv2, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, org.label-schema.build-date=20240319, ceph=True, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD) 2024-03-24T03:21:49.954 INFO:journalctl@ceph.mds.k.smithi189.stdout:Mar 24 03:21:49 smithi189 podman[118193]: 2024-03-24 03:21:49.788000069 +0000 UTC m=+3.279511015 container cleanup 89f5182d7c26e85a3b40978255d4bf9f4b2b1e9d17632ed765e7e31e55c907d4 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-k, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240319, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, org.label-schema.vendor=CentOS, ceph=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD) 2024-03-24T03:21:49.954 INFO:journalctl@ceph.mds.k.smithi189.stdout:Mar 24 03:21:49 smithi189 bash[118193]: ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-k 2024-03-24T03:21:50.128 DEBUG:teuthology.orchestra.run.smithi189:> sudo pkill -f 'journalctl -f -n 0 -u ceph-d79a5420-e971-11ee-95cd-87774f69a715@mds.k.service' 2024-03-24T03:21:50.162 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-24T03:21:50.163 INFO:tasks.cephadm.mds.k:Stopped mds.k 2024-03-24T03:21:50.163 INFO:tasks.cephadm.mds.c:Stopping mds.c... 2024-03-24T03:21:50.163 DEBUG:teuthology.orchestra.run.smithi193:> sudo systemctl stop ceph-d79a5420-e971-11ee-95cd-87774f69a715@mds.c 2024-03-24T03:21:50.460 INFO:journalctl@ceph.mds.c.smithi193.stdout:Mar 24 03:21:50 smithi193 systemd[1]: Stopping Ceph mds.c for d79a5420-e971-11ee-95cd-87774f69a715... 2024-03-24T03:21:50.719 INFO:journalctl@ceph.mds.c.smithi193.stdout:Mar 24 03:21:50 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-c[67774]: 2024-03-24T03:21:50.491+0000 7f9c78f7b700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.c -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-24T03:21:50.720 INFO:journalctl@ceph.mds.c.smithi193.stdout:Mar 24 03:21:50 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-c[67774]: 2024-03-24T03:21:50.491+0000 7f9c78f7b700 -1 mds.c *** got signal Terminated *** 2024-03-24T03:21:53.689 INFO:journalctl@ceph.mds.c.smithi193.stdout:Mar 24 03:21:53 smithi193 podman[121183]: 2024-03-24 03:21:53.362264365 +0000 UTC m=+3.004467701 container died f0210a5dc42f895c3146008a5346994f1d3b5b2168820fde275433feeaac32a4 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-c, org.label-schema.license=GPLv2, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, ceph=True, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240319, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0) 2024-03-24T03:21:53.947 INFO:journalctl@ceph.mds.c.smithi193.stdout:Mar 24 03:21:53 smithi193 podman[121183]: 2024-03-24 03:21:53.789075869 +0000 UTC m=+3.431279196 container cleanup f0210a5dc42f895c3146008a5346994f1d3b5b2168820fde275433feeaac32a4 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-c, io.buildah.version=1.33.5, ceph=True, org.label-schema.build-date=20240319, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.license=GPLv2, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c) 2024-03-24T03:21:53.948 INFO:journalctl@ceph.mds.c.smithi193.stdout:Mar 24 03:21:53 smithi193 bash[121183]: ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-c 2024-03-24T03:21:54.160 DEBUG:teuthology.orchestra.run.smithi193:> sudo pkill -f 'journalctl -f -n 0 -u ceph-d79a5420-e971-11ee-95cd-87774f69a715@mds.c.service' 2024-03-24T03:21:54.197 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-24T03:21:54.198 INFO:tasks.cephadm.mds.c:Stopped mds.c 2024-03-24T03:21:54.198 INFO:tasks.cephadm.mds.f:Stopping mds.f... 2024-03-24T03:21:54.198 DEBUG:teuthology.orchestra.run.smithi193:> sudo systemctl stop ceph-d79a5420-e971-11ee-95cd-87774f69a715@mds.f 2024-03-24T03:21:54.270 DEBUG:teuthology.orchestra.run.smithi193:> sudo pkill -f 'journalctl -f -n 0 -u ceph-d79a5420-e971-11ee-95cd-87774f69a715@mds.f.service' 2024-03-24T03:21:54.343 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-24T03:21:54.344 INFO:tasks.cephadm.mds.f:Stopped mds.f 2024-03-24T03:21:54.344 INFO:tasks.cephadm.mds.i:Stopping mds.i... 2024-03-24T03:21:54.344 DEBUG:teuthology.orchestra.run.smithi193:> sudo systemctl stop ceph-d79a5420-e971-11ee-95cd-87774f69a715@mds.i 2024-03-24T03:21:54.687 INFO:journalctl@ceph.mds.i.smithi193.stdout:Mar 24 03:21:54 smithi193 systemd[1]: Stopping Ceph mds.i for d79a5420-e971-11ee-95cd-87774f69a715... 2024-03-24T03:21:54.960 INFO:journalctl@ceph.mds.i.smithi193.stdout:Mar 24 03:21:54 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-i[68088]: 2024-03-24T03:21:54.686+0000 7fea67197700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mds -n mds.i -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-24T03:21:54.961 INFO:journalctl@ceph.mds.i.smithi193.stdout:Mar 24 03:21:54 smithi193 ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-i[68088]: 2024-03-24T03:21:54.686+0000 7fea67197700 -1 mds.i *** got signal Terminated *** 2024-03-24T03:21:57.658 INFO:journalctl@ceph.mds.i.smithi193.stdout:Mar 24 03:21:57 smithi193 podman[121352]: 2024-03-24 03:21:57.354617874 +0000 UTC m=+2.792784122 container died 749e1d96d3972c0c41d21f00c3bff52d847c20bf10030537cb51011beac07fa6 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-i, GIT_BRANCH=HEAD, ceph=True, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240319, maintainer=Guillaume Abrioux , io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30) 2024-03-24T03:21:57.658 INFO:journalctl@ceph.mds.i.smithi193.stdout:Mar 24 03:21:57 smithi193 podman[121352]: 2024-03-24 03:21:57.659111472 +0000 UTC m=+3.097277730 container cleanup 749e1d96d3972c0c41d21f00c3bff52d847c20bf10030537cb51011beac07fa6 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-i, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240319, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_BRANCH=HEAD, ceph=True, org.label-schema.license=GPLv2, io.buildah.version=1.33.5) 2024-03-24T03:21:57.960 INFO:journalctl@ceph.mds.i.smithi193.stdout:Mar 24 03:21:57 smithi193 bash[121352]: ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-i 2024-03-24T03:21:57.960 INFO:journalctl@ceph.mds.i.smithi193.stdout:Mar 24 03:21:57 smithi193 podman[121365]: 2024-03-24 03:21:57.860262745 +0000 UTC m=+0.503821352 container remove 749e1d96d3972c0c41d21f00c3bff52d847c20bf10030537cb51011beac07fa6 (image=quay-quay-quay.apps.os.sepia.ceph.com/ceph-ci/ceph@sha256:9b48cbed4f912cd766666ab034815156a50a4f9b95481e98a662f737d54a2abc, name=ceph-d79a5420-e971-11ee-95cd-87774f69a715-mds-i, GIT_CLEAN=True, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , RELEASE=wip-mchangir-testing-main-20240318.032620-3be9a5c, org.label-schema.build-date=20240319, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, ceph=True) 2024-03-24T03:21:58.076 DEBUG:teuthology.orchestra.run.smithi193:> sudo pkill -f 'journalctl -f -n 0 -u ceph-d79a5420-e971-11ee-95cd-87774f69a715@mds.i.service' 2024-03-24T03:21:58.110 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-24T03:21:58.111 INFO:tasks.cephadm.mds.i:Stopped mds.i 2024-03-24T03:21:58.111 INFO:tasks.cephadm.mds.l:Stopping mds.l... 2024-03-24T03:21:58.111 DEBUG:teuthology.orchestra.run.smithi193:> sudo systemctl stop ceph-d79a5420-e971-11ee-95cd-87774f69a715@mds.l 2024-03-24T03:21:58.181 DEBUG:teuthology.orchestra.run.smithi193:> sudo pkill -f 'journalctl -f -n 0 -u ceph-d79a5420-e971-11ee-95cd-87774f69a715@mds.l.service' 2024-03-24T03:21:58.256 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-24T03:21:58.256 INFO:tasks.cephadm.mds.l:Stopped mds.l 2024-03-24T03:21:58.256 DEBUG:teuthology.orchestra.run.smithi118:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid d79a5420-e971-11ee-95cd-87774f69a715 --force --keep-logs 2024-03-24T03:21:58.573 INFO:teuthology.orchestra.run.smithi118.stdout:Deleting cluster with fsid: d79a5420-e971-11ee-95cd-87774f69a715 2024-03-24T03:22:03.272 DEBUG:teuthology.orchestra.run.smithi189:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid d79a5420-e971-11ee-95cd-87774f69a715 --force --keep-logs 2024-03-24T03:22:03.463 INFO:teuthology.orchestra.run.smithi189.stdout:Deleting cluster with fsid: d79a5420-e971-11ee-95cd-87774f69a715 2024-03-24T03:22:05.907 DEBUG:teuthology.orchestra.run.smithi193:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid d79a5420-e971-11ee-95cd-87774f69a715 --force --keep-logs 2024-03-24T03:22:06.095 INFO:teuthology.orchestra.run.smithi193.stdout:Deleting cluster with fsid: d79a5420-e971-11ee-95cd-87774f69a715 2024-03-24T03:22:08.645 DEBUG:teuthology.orchestra.run.smithi118:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-03-24T03:22:08.752 DEBUG:teuthology.orchestra.run.smithi189:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-03-24T03:22:08.786 DEBUG:teuthology.orchestra.run.smithi193:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-03-24T03:22:08.815 INFO:tasks.cephadm:Archiving crash dumps... 2024-03-24T03:22:08.818 DEBUG:teuthology.misc:Transferring archived files from smithi118:/var/lib/ceph/d79a5420-e971-11ee-95cd-87774f69a715/crash to /home/teuthworker/archive/mchangir-2024-03-22_09:49:57-fs-wip-mchangir-testing-main-20240318.032620-testing-default-smithi/7616426/remote/smithi118/crash 2024-03-24T03:22:08.819 DEBUG:teuthology.orchestra.run.smithi118:> sudo tar c -f - -C /var/lib/ceph/d79a5420-e971-11ee-95cd-87774f69a715/crash -- . 2024-03-24T03:22:08.912 INFO:teuthology.orchestra.run.smithi118.stderr:tar: /var/lib/ceph/d79a5420-e971-11ee-95cd-87774f69a715/crash: Cannot open: No such file or directory 2024-03-24T03:22:08.913 INFO:teuthology.orchestra.run.smithi118.stderr:tar: Error is not recoverable: exiting now 2024-03-24T03:22:08.918 DEBUG:teuthology.misc:Transferring archived files from smithi189:/var/lib/ceph/d79a5420-e971-11ee-95cd-87774f69a715/crash to /home/teuthworker/archive/mchangir-2024-03-22_09:49:57-fs-wip-mchangir-testing-main-20240318.032620-testing-default-smithi/7616426/remote/smithi189/crash 2024-03-24T03:22:08.919 DEBUG:teuthology.orchestra.run.smithi189:> sudo tar c -f - -C /var/lib/ceph/d79a5420-e971-11ee-95cd-87774f69a715/crash -- . 2024-03-24T03:22:08.958 INFO:teuthology.orchestra.run.smithi189.stderr:tar: /var/lib/ceph/d79a5420-e971-11ee-95cd-87774f69a715/crash: Cannot open: No such file or directory 2024-03-24T03:22:08.958 INFO:teuthology.orchestra.run.smithi189.stderr:tar: Error is not recoverable: exiting now 2024-03-24T03:22:08.960 DEBUG:teuthology.misc:Transferring archived files from smithi193:/var/lib/ceph/d79a5420-e971-11ee-95cd-87774f69a715/crash to /home/teuthworker/archive/mchangir-2024-03-22_09:49:57-fs-wip-mchangir-testing-main-20240318.032620-testing-default-smithi/7616426/remote/smithi193/crash 2024-03-24T03:22:08.961 DEBUG:teuthology.orchestra.run.smithi193:> sudo tar c -f - -C /var/lib/ceph/d79a5420-e971-11ee-95cd-87774f69a715/crash -- . 2024-03-24T03:22:08.995 INFO:teuthology.orchestra.run.smithi193.stderr:tar: /var/lib/ceph/d79a5420-e971-11ee-95cd-87774f69a715/crash: Cannot open: No such file or directory 2024-03-24T03:22:08.996 INFO:teuthology.orchestra.run.smithi193.stderr:tar: Error is not recoverable: exiting now 2024-03-24T03:22:08.997 INFO:tasks.cephadm:Checking cluster log for badness... 2024-03-24T03:22:08.997 DEBUG:teuthology.orchestra.run.smithi118:> sudo egrep '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/d79a5420-e971-11ee-95cd-87774f69a715/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v FS_DEGRADED | egrep -v FS_INLINE_DATA_DEPRECATED | egrep -v FS_WITH_FAILED_MDS | egrep -v MDS_ALL_DOWN | egrep -v MDS_DAMAGE | egrep -v MDS_DEGRADED | egrep -v MDS_FAILED | egrep -v MDS_INSUFFICIENT_STANDBY | egrep -v MDS_UP_LESS_THAN_MAX | egrep -v POOL_APP_NOT_ENABLED | egrep -v 'overall HEALTH_' | egrep -v 'Replacing daemon' | egrep -v 'deprecated feature inline_data' | egrep -v 'overall HEALTH_' | egrep -v '\(OSD_DOWN\)' | egrep -v '\(OSD_' | egrep -v 'but it is still running' | egrep -v 'is not responding' | egrep -v MON_DOWN | egrep -v 'slow metadata IO' | egrep -v SLOW_OPS | egrep -v 'slow request' | head -n 1 2024-03-24T03:22:09.109 INFO:teuthology.orchestra.run.smithi118.stdout:2024-03-24T00:06:13.469908+0000 mon.a (mon.0) 191 : cluster [WRN] Health detail: HEALTH_WARN 1/3 mons down, quorum a,c 2024-03-24T03:22:09.110 WARNING:tasks.cephadm:Found errors (ERR|WRN|SEC) in cluster log 2024-03-24T03:22:09.110 INFO:tasks.cephadm:Compressing logs... 2024-03-24T03:22:09.111 DEBUG:teuthology.orchestra.run.smithi118:> time sudo find /var/log/ceph /var/log/rbd-target-api -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-03-24T03:22:09.115 DEBUG:teuthology.orchestra.run.smithi189:> time sudo find /var/log/ceph /var/log/rbd-target-api -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-03-24T03:22:09.117 DEBUG:teuthology.orchestra.run.smithi193:> time sudo find /var/log/ceph /var/log/rbd-target-api -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-03-24T03:22:09.147 INFO:teuthology.orchestra.run.smithi193.stderr:find: ‘/var/log/rbd-target-api’gzip -5 --verbose -- /var/log/ceph/cephadm.log 2024-03-24T03:22:09.147 INFO:teuthology.orchestra.run.smithi193.stderr:: No such file or directory 2024-03-24T03:22:09.147 INFO:teuthology.orchestra.run.smithi193.stderr:gzip -5 --verbose -- /var/log/ceph/d79a5420-e971-11ee-95cd-87774f69a715/ceph-mds.l.log 2024-03-24T03:22:09.147 INFO:teuthology.orchestra.run.smithi193.stderr:gzip -5 --verbose -- /var/log/ceph/d79a5420-e971-11ee-95cd-87774f69a715/ceph.audit.log 2024-03-24T03:22:09.148 INFO:teuthology.orchestra.run.smithi193.stderr:/var/log/ceph/cephadm.log: /var/log/ceph/d79a5420-e971-11ee-95cd-87774f69a715/ceph-mds.l.log: gzip -5 --verbose -- /var/log/ceph/d79a5420-e971-11ee-95cd-87774f69a715/ceph-mds.i.log 2024-03-24T03:22:09.148 INFO:teuthology.orchestra.run.smithi193.stderr:/var/log/ceph/d79a5420-e971-11ee-95cd-87774f69a715/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/d79a5420-e971-11ee-95cd-87774f69a715/ceph-osd.11.log 2024-03-24T03:22:09.148 INFO:teuthology.orchestra.run.smithi193.stderr:gzip -5 --verbose -- /var/log/ceph/d79a5420-e971-11ee-95cd-87774f69a715/ceph-mgr.z.log 2024-03-24T03:22:09.148 INFO:teuthology.orchestra.run.smithi193.stderr:/var/log/ceph/d79a5420-e971-11ee-95cd-87774f69a715/ceph-mds.i.log: /var/log/ceph/d79a5420-e971-11ee-95cd-87774f69a715/ceph-osd.11.log: gzip -5 --verbose -- /var/log/ceph/d79a5420-e971-11ee-95cd-87774f69a715/ceph-osd.8.log 2024-03-24T03:22:09.148 INFO:teuthology.orchestra.run.smithi193.stderr:/var/log/ceph/d79a5420-e971-11ee-95cd-87774f69a715/ceph-mgr.z.log: gzip -5 --verbose -- /var/log/ceph/d79a5420-e971-11ee-95cd-87774f69a715/ceph-osd.5.log 2024-03-24T03:22:09.148 INFO:teuthology.orchestra.run.smithi193.stderr:/var/log/ceph/d79a5420-e971-11ee-95cd-87774f69a715/ceph-osd.8.log: 92.1% -- replaced with /var/log/ceph/cephadm.log.gz 2024-03-24T03:22:09.148 INFO:teuthology.orchestra.run.smithi193.stderr:gzip -5 --verbose -- /var/log/ceph/d79a5420-e971-11ee-95cd-87774f69a715/ceph-mds.f.log 2024-03-24T03:22:09.148 INFO:teuthology.orchestra.run.smithi193.stderr:gzip -5 --verbose -- /var/log/ceph/d79a5420-e971-11ee-95cd-87774f69a715/ceph.cephadm.log 2024-03-24T03:22:09.153 INFO:teuthology.orchestra.run.smithi193.stderr:/var/log/ceph/d79a5420-e971-11ee-95cd-87774f69a715/ceph-mds.f.log: /var/log/ceph/d79a5420-e971-11ee-95cd-87774f69a715/ceph-osd.5.log: 91.5% -- replaced with /var/log/ceph/d79a5420-e971-11ee-95cd-87774f69a715/ceph.audit.log.gz 2024-03-24T03:22:09.153 INFO:teuthology.orchestra.run.smithi193.stderr:gzip -5 --verbose -- /var/log/ceph/d79a5420-e971-11ee-95cd-87774f69a715/ceph.log 2024-03-24T03:22:09.154 INFO:teuthology.orchestra.run.smithi193.stderr:/var/log/ceph/d79a5420-e971-11ee-95cd-87774f69a715/ceph.cephadm.log: 87.8% -- replaced with /var/log/ceph/d79a5420-e971-11ee-95cd-87774f69a715/ceph.cephadm.log.gz 2024-03-24T03:22:09.154 INFO:teuthology.orchestra.run.smithi193.stderr:gzip -5 --verbose -- /var/log/ceph/d79a5420-e971-11ee-95cd-87774f69a715/ceph-osd.2.log 2024-03-24T03:22:09.155 INFO:teuthology.orchestra.run.smithi189.stderr:find: ‘/var/log/rbd-target-api’: No such file or directory 2024-03-24T03:22:09.155 INFO:teuthology.orchestra.run.smithi189.stderr:gzip -5 --verbose -- /var/log/ceph/cephadm.log 2024-03-24T03:22:09.156 INFO:teuthology.orchestra.run.smithi189.stderr:gzip -5 --verbose -- /var/log/ceph/d79a5420-e971-11ee-95cd-87774f69a715/ceph-osd.10.log 2024-03-24T03:22:09.156 INFO:teuthology.orchestra.run.smithi189.stderr:/var/log/ceph/cephadm.log: gzip -5 --verbose -- /var/log/ceph/d79a5420-e971-11ee-95cd-87774f69a715/ceph.audit.log 2024-03-24T03:22:09.156 INFO:teuthology.orchestra.run.smithi189.stderr:/var/log/ceph/d79a5420-e971-11ee-95cd-87774f69a715/ceph-osd.10.log: gzip -5 --verbose -- /var/log/ceph/d79a5420-e971-11ee-95cd-87774f69a715/ceph-mds.b.log 2024-03-24T03:22:09.156 INFO:teuthology.orchestra.run.smithi189.stderr:/var/log/ceph/d79a5420-e971-11ee-95cd-87774f69a715/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/d79a5420-e971-11ee-95cd-87774f69a715/ceph-osd.4.log 2024-03-24T03:22:09.156 INFO:teuthology.orchestra.run.smithi189.stderr:/var/log/ceph/d79a5420-e971-11ee-95cd-87774f69a715/ceph-mds.b.log: gzip -5 --verbose -- /var/log/ceph/d79a5420-e971-11ee-95cd-87774f69a715/ceph-osd.1.log 2024-03-24T03:22:09.156 INFO:teuthology.orchestra.run.smithi189.stderr:/var/log/ceph/d79a5420-e971-11ee-95cd-87774f69a715/ceph-osd.4.log: gzip -5 --verbose -- /var/log/ceph/d79a5420-e971-11ee-95cd-87774f69a715/ceph-mds.k.log 2024-03-24T03:22:09.156 INFO:teuthology.orchestra.run.smithi189.stderr:/var/log/ceph/d79a5420-e971-11ee-95cd-87774f69a715/ceph-osd.1.log: 91.8% -- replaced with /var/log/ceph/cephadm.log.gz 2024-03-24T03:22:09.156 INFO:teuthology.orchestra.run.smithi189.stderr:gzip -5 --verbose -- /var/log/ceph/d79a5420-e971-11ee-95cd-87774f69a715/ceph-mds.h.log 2024-03-24T03:22:09.157 INFO:teuthology.orchestra.run.smithi189.stderr:gzip -5 --verbose -- /var/log/ceph/d79a5420-e971-11ee-95cd-87774f69a715/ceph-osd.7.log 2024-03-24T03:22:09.157 INFO:teuthology.orchestra.run.smithi189.stderr:/var/log/ceph/d79a5420-e971-11ee-95cd-87774f69a715/ceph-mds.k.log: /var/log/ceph/d79a5420-e971-11ee-95cd-87774f69a715/ceph-mds.h.log: 91.5% -- replaced with /var/log/ceph/d79a5420-e971-11ee-95cd-87774f69a715/ceph.audit.log.gz 2024-03-24T03:22:09.157 INFO:teuthology.orchestra.run.smithi189.stderr:gzip -5 --verbose -- /var/log/ceph/d79a5420-e971-11ee-95cd-87774f69a715/ceph-mds.e.log 2024-03-24T03:22:09.163 INFO:teuthology.orchestra.run.smithi193.stderr:/var/log/ceph/d79a5420-e971-11ee-95cd-87774f69a715/ceph.log: gzip -5 --verbose -- /var/log/ceph/d79a5420-e971-11ee-95cd-87774f69a715/ceph-mds.c.log 2024-03-24T03:22:09.174 INFO:teuthology.orchestra.run.smithi193.stderr:/var/log/ceph/d79a5420-e971-11ee-95cd-87774f69a715/ceph-osd.2.log: 89.7%gzip -5 --verbose -- /var/log/ceph/d79a5420-e971-11ee-95cd-87774f69a715/ceph-volume.log 2024-03-24T03:22:09.181 INFO:teuthology.orchestra.run.smithi118.stderr:find: ‘/var/log/rbd-target-api’gzip: No such file or directory -5 2024-03-24T03:22:09.181 INFO:teuthology.orchestra.run.smithi118.stderr: --verbose -- /var/log/ceph/cephadm.log 2024-03-24T03:22:09.182 INFO:teuthology.orchestra.run.smithi118.stderr:gzip -5 --verbose -- /var/log/ceph/d79a5420-e971-11ee-95cd-87774f69a715/ceph-osd.6.log 2024-03-24T03:22:09.183 INFO:teuthology.orchestra.run.smithi193.stderr:/var/log/ceph/d79a5420-e971-11ee-95cd-87774f69a715/ceph-mds.c.log: 94.5% -- replaced with /var/log/ceph/d79a5420-e971-11ee-95cd-87774f69a715/ceph-mgr.z.log.gz 2024-03-24T03:22:09.183 INFO:teuthology.orchestra.run.smithi193.stderr: -- replaced with /var/log/ceph/d79a5420-e971-11ee-95cd-87774f69a715/ceph.log.gz 2024-03-24T03:22:09.184 INFO:teuthology.orchestra.run.smithi118.stderr:/var/log/ceph/cephadm.log: gzip -5 --verbose -- /var/log/ceph/d79a5420-e971-11ee-95cd-87774f69a715/ceph-mon.a.log 2024-03-24T03:22:09.184 INFO:teuthology.orchestra.run.smithi118.stderr:/var/log/ceph/d79a5420-e971-11ee-95cd-87774f69a715/ceph-osd.6.log: gzip -5 --verbose -- /var/log/ceph/d79a5420-e971-11ee-95cd-87774f69a715/ceph.audit.log 2024-03-24T03:22:09.185 INFO:teuthology.orchestra.run.smithi118.stderr:/var/log/ceph/d79a5420-e971-11ee-95cd-87774f69a715/ceph-mon.a.log: gzip -5 --verbose -- /var/log/ceph/d79a5420-e971-11ee-95cd-87774f69a715/ceph-mds.g.log 2024-03-24T03:22:09.187 INFO:teuthology.orchestra.run.smithi118.stderr:/var/log/ceph/d79a5420-e971-11ee-95cd-87774f69a715/ceph.audit.log: gzip -5 --verbose -- /var/log/ceph/d79a5420-e971-11ee-95cd-87774f69a715/ceph-mds.j.log 2024-03-24T03:22:09.188 INFO:teuthology.orchestra.run.smithi193.stderr:gzip -5 --verbose -- /var/log/ceph/d79a5420-e971-11ee-95cd-87774f69a715/ceph-mon.c.log 2024-03-24T03:22:09.188 INFO:teuthology.orchestra.run.smithi118.stderr:/var/log/ceph/d79a5420-e971-11ee-95cd-87774f69a715/ceph-mds.g.log: gzip -5 --verbose -- /var/log/ceph/d79a5420-e971-11ee-95cd-87774f69a715/ceph-osd.3.log 2024-03-24T03:22:09.190 INFO:teuthology.orchestra.run.smithi118.stderr:/var/log/ceph/d79a5420-e971-11ee-95cd-87774f69a715/ceph-mds.j.log: gzip -5 --verbose -- /var/log/ceph/d79a5420-e971-11ee-95cd-87774f69a715/ceph-mgr.x.log 2024-03-24T03:22:09.191 INFO:teuthology.orchestra.run.smithi189.stderr:/var/log/ceph/d79a5420-e971-11ee-95cd-87774f69a715/ceph-osd.7.log: gzip -5 --verbose -- /var/log/ceph/d79a5420-e971-11ee-95cd-87774f69a715/ceph-mgr.y.log 2024-03-24T03:22:09.193 INFO:teuthology.orchestra.run.smithi118.stderr:/var/log/ceph/d79a5420-e971-11ee-95cd-87774f69a715/ceph-osd.3.log: /var/log/ceph/d79a5420-e971-11ee-95cd-87774f69a715/ceph-mgr.x.log: gzip -5 --verbose -- /var/log/ceph/d79a5420-e971-11ee-95cd-87774f69a715/ceph-osd.0.log 2024-03-24T03:22:09.194 INFO:teuthology.orchestra.run.smithi118.stderr: 93.0% -- replaced with /var/log/ceph/cephadm.log.gz 2024-03-24T03:22:09.194 INFO:teuthology.orchestra.run.smithi118.stderr: 91.4% -- replaced with /var/log/ceph/d79a5420-e971-11ee-95cd-87774f69a715/ceph.audit.log.gz 2024-03-24T03:22:09.195 INFO:teuthology.orchestra.run.smithi118.stderr:gzip -5 --verbose -- /var/log/ceph/d79a5420-e971-11ee-95cd-87774f69a715/ceph.cephadm.log 2024-03-24T03:22:09.197 INFO:teuthology.orchestra.run.smithi118.stderr:/var/log/ceph/d79a5420-e971-11ee-95cd-87774f69a715/ceph-osd.0.log: /var/log/ceph/d79a5420-e971-11ee-95cd-87774f69a715/ceph.cephadm.log: gzip -5 --verbose -- /var/log/ceph/d79a5420-e971-11ee-95cd-87774f69a715/ceph.log 2024-03-24T03:22:09.200 INFO:teuthology.orchestra.run.smithi189.stderr:/var/log/ceph/d79a5420-e971-11ee-95cd-87774f69a715/ceph-mds.e.log: gzip -5 --verbose -- /var/log/ceph/d79a5420-e971-11ee-95cd-87774f69a715/ceph.cephadm.log 2024-03-24T03:22:09.202 INFO:teuthology.orchestra.run.smithi118.stderr:/var/log/ceph/d79a5420-e971-11ee-95cd-87774f69a715/ceph.log: 87.2%gzip -5 --verbose -- /var/log/ceph/d79a5420-e971-11ee-95cd-87774f69a715/ceph-volume.log 2024-03-24T03:22:09.202 INFO:teuthology.orchestra.run.smithi118.stderr: -- replaced with /var/log/ceph/d79a5420-e971-11ee-95cd-87774f69a715/ceph.cephadm.log.gz 2024-03-24T03:22:09.203 INFO:teuthology.orchestra.run.smithi118.stderr:gzip -5 --verbose -- /var/log/ceph/d79a5420-e971-11ee-95cd-87774f69a715/ceph-osd.9.log 2024-03-24T03:22:09.209 INFO:teuthology.orchestra.run.smithi189.stderr:/var/log/ceph/d79a5420-e971-11ee-95cd-87774f69a715/ceph-mgr.y.log: gzip -5 --verbose -- /var/log/ceph/d79a5420-e971-11ee-95cd-87774f69a715/ceph.log 2024-03-24T03:22:09.211 INFO:teuthology.orchestra.run.smithi118.stderr:/var/log/ceph/d79a5420-e971-11ee-95cd-87774f69a715/ceph-volume.log: gzip -5 --verbose -- /var/log/ceph/d79a5420-e971-11ee-95cd-87774f69a715/ceph-mds.d.log 2024-03-24T03:22:09.218 INFO:teuthology.orchestra.run.smithi118.stderr:/var/log/ceph/d79a5420-e971-11ee-95cd-87774f69a715/ceph-osd.9.log: /var/log/ceph/d79a5420-e971-11ee-95cd-87774f69a715/ceph-mds.d.log: gzip -5 --verbose -- /var/log/ceph/d79a5420-e971-11ee-95cd-87774f69a715/ceph-mds.a.log 2024-03-24T03:22:09.220 INFO:teuthology.orchestra.run.smithi193.stderr:/var/log/ceph/d79a5420-e971-11ee-95cd-87774f69a715/ceph-volume.log: /var/log/ceph/d79a5420-e971-11ee-95cd-87774f69a715/ceph-mon.c.log: 91.1% -- replaced with /var/log/ceph/d79a5420-e971-11ee-95cd-87774f69a715/ceph-volume.log.gz 2024-03-24T03:22:09.221 INFO:teuthology.orchestra.run.smithi189.stderr:/var/log/ceph/d79a5420-e971-11ee-95cd-87774f69a715/ceph.cephadm.log: gzip -5 --verbose -- /var/log/ceph/d79a5420-e971-11ee-95cd-87774f69a715/ceph-volume.log 2024-03-24T03:22:09.222 INFO:teuthology.orchestra.run.smithi189.stderr:/var/log/ceph/d79a5420-e971-11ee-95cd-87774f69a715/ceph.log: 87.9% -- replaced with /var/log/ceph/d79a5420-e971-11ee-95cd-87774f69a715/ceph.cephadm.log.gz 2024-03-24T03:22:09.235 INFO:teuthology.orchestra.run.smithi189.stderr:gzip -5 --verbose -- /var/log/ceph/d79a5420-e971-11ee-95cd-87774f69a715/ceph-mon.b.log 2024-03-24T03:22:09.257 INFO:teuthology.orchestra.run.smithi118.stderr:/var/log/ceph/d79a5420-e971-11ee-95cd-87774f69a715/ceph-mds.a.log: 91.0% -- replaced with /var/log/ceph/d79a5420-e971-11ee-95cd-87774f69a715/ceph-volume.log.gz 2024-03-24T03:22:09.268 INFO:teuthology.orchestra.run.smithi118.stderr: 89.7% -- replaced with /var/log/ceph/d79a5420-e971-11ee-95cd-87774f69a715/ceph.log.gz 2024-03-24T03:22:09.277 INFO:teuthology.orchestra.run.smithi189.stderr:/var/log/ceph/d79a5420-e971-11ee-95cd-87774f69a715/ceph-volume.log: /var/log/ceph/d79a5420-e971-11ee-95cd-87774f69a715/ceph-mon.b.log: 94.5% -- replaced with /var/log/ceph/d79a5420-e971-11ee-95cd-87774f69a715/ceph-mgr.y.log.gz 2024-03-24T03:22:09.277 INFO:teuthology.orchestra.run.smithi189.stderr: 91.1% -- replaced with /var/log/ceph/d79a5420-e971-11ee-95cd-87774f69a715/ceph-volume.log.gz 2024-03-24T03:22:09.284 INFO:teuthology.orchestra.run.smithi189.stderr: 89.7% -- replaced with /var/log/ceph/d79a5420-e971-11ee-95cd-87774f69a715/ceph.log.gz 2024-03-24T03:22:09.408 INFO:teuthology.orchestra.run.smithi118.stderr: 92.8% -- replaced with /var/log/ceph/d79a5420-e971-11ee-95cd-87774f69a715/ceph-mds.j.log.gz 2024-03-24T03:22:09.596 INFO:teuthology.orchestra.run.smithi193.stderr: 94.2% -- replaced with /var/log/ceph/d79a5420-e971-11ee-95cd-87774f69a715/ceph-mds.l.log.gz 2024-03-24T03:22:09.870 INFO:teuthology.orchestra.run.smithi189.stderr: 95.1% -- replaced with /var/log/ceph/d79a5420-e971-11ee-95cd-87774f69a715/ceph-mds.k.log.gz 2024-03-24T03:22:09.993 INFO:teuthology.orchestra.run.smithi118.stderr: 94.5% -- replaced with /var/log/ceph/d79a5420-e971-11ee-95cd-87774f69a715/ceph-mds.d.log.gz 2024-03-24T03:22:09.994 INFO:teuthology.orchestra.run.smithi189.stderr: 94.2% -- replaced with /var/log/ceph/d79a5420-e971-11ee-95cd-87774f69a715/ceph-mds.h.log.gz 2024-03-24T03:22:10.119 INFO:teuthology.orchestra.run.smithi118.stderr: 95.1% -- replaced with /var/log/ceph/d79a5420-e971-11ee-95cd-87774f69a715/ceph-mds.a.log.gz 2024-03-24T03:22:10.135 INFO:teuthology.orchestra.run.smithi189.stderr: 95.2% -- replaced with /var/log/ceph/d79a5420-e971-11ee-95cd-87774f69a715/ceph-mds.e.log.gz 2024-03-24T03:22:10.202 INFO:teuthology.orchestra.run.smithi193.stderr: 94.8% -- replaced with /var/log/ceph/d79a5420-e971-11ee-95cd-87774f69a715/ceph-mds.f.log.gz 2024-03-24T03:22:10.213 INFO:teuthology.orchestra.run.smithi118.stderr: 94.7% -- replaced with /var/log/ceph/d79a5420-e971-11ee-95cd-87774f69a715/ceph-mds.g.log.gz 2024-03-24T03:22:10.706 INFO:teuthology.orchestra.run.smithi193.stderr: 95.2% -- replaced with /var/log/ceph/d79a5420-e971-11ee-95cd-87774f69a715/ceph-mds.c.log.gz 2024-03-24T03:22:10.838 INFO:teuthology.orchestra.run.smithi118.stderr: 91.2% -- replaced with /var/log/ceph/d79a5420-e971-11ee-95cd-87774f69a715/ceph-mgr.x.log.gz 2024-03-24T03:22:12.176 INFO:teuthology.orchestra.run.smithi189.stderr: 91.9% -- replaced with /var/log/ceph/d79a5420-e971-11ee-95cd-87774f69a715/ceph-mon.b.log.gz 2024-03-24T03:22:12.563 INFO:teuthology.orchestra.run.smithi193.stderr: 95.0% -- replaced with /var/log/ceph/d79a5420-e971-11ee-95cd-87774f69a715/ceph-mds.i.log.gz 2024-03-24T03:22:12.741 INFO:teuthology.orchestra.run.smithi193.stderr: 91.8% -- replaced with /var/log/ceph/d79a5420-e971-11ee-95cd-87774f69a715/ceph-mon.c.log.gz 2024-03-24T03:22:16.345 INFO:teuthology.orchestra.run.smithi118.stderr: 89.7% -- replaced with /var/log/ceph/d79a5420-e971-11ee-95cd-87774f69a715/ceph-mon.a.log.gz 2024-03-24T03:22:17.060 INFO:teuthology.orchestra.run.smithi193.stderr: 92.4% -- replaced with /var/log/ceph/d79a5420-e971-11ee-95cd-87774f69a715/ceph-osd.2.log.gz 2024-03-24T03:22:17.077 INFO:teuthology.orchestra.run.smithi118.stderr: 92.5% -- replaced with /var/log/ceph/d79a5420-e971-11ee-95cd-87774f69a715/ceph-osd.0.log.gz 2024-03-24T03:22:17.202 INFO:teuthology.orchestra.run.smithi193.stderr: 92.5% -- replaced with /var/log/ceph/d79a5420-e971-11ee-95cd-87774f69a715/ceph-osd.11.log.gz 2024-03-24T03:22:18.132 INFO:teuthology.orchestra.run.smithi189.stderr: 92.5% -- replaced with /var/log/ceph/d79a5420-e971-11ee-95cd-87774f69a715/ceph-osd.1.log.gz 2024-03-24T03:22:18.931 INFO:teuthology.orchestra.run.smithi193.stderr: 92.3% -- replaced with /var/log/ceph/d79a5420-e971-11ee-95cd-87774f69a715/ceph-osd.8.log.gz 2024-03-24T03:22:19.090 INFO:teuthology.orchestra.run.smithi118.stderr: 92.3% -- replaced with /var/log/ceph/d79a5420-e971-11ee-95cd-87774f69a715/ceph-osd.3.log.gz 2024-03-24T03:22:19.559 INFO:teuthology.orchestra.run.smithi193.stderr: 92.2% -- replaced with /var/log/ceph/d79a5420-e971-11ee-95cd-87774f69a715/ceph-osd.5.log.gz 2024-03-24T03:22:19.561 INFO:teuthology.orchestra.run.smithi193.stderr: 2024-03-24T03:22:19.561 INFO:teuthology.orchestra.run.smithi193.stderr:real 0m10.430s 2024-03-24T03:22:19.561 INFO:teuthology.orchestra.run.smithi193.stderr:user 0m44.243s 2024-03-24T03:22:19.561 INFO:teuthology.orchestra.run.smithi193.stderr:sys 0m1.498s 2024-03-24T03:22:19.991 INFO:teuthology.orchestra.run.smithi189.stderr: 92.4% -- replaced with /var/log/ceph/d79a5420-e971-11ee-95cd-87774f69a715/ceph-osd.4.log.gz 2024-03-24T03:22:20.140 INFO:teuthology.orchestra.run.smithi189.stderr: 92.1% -- replaced with /var/log/ceph/d79a5420-e971-11ee-95cd-87774f69a715/ceph-osd.7.log.gz 2024-03-24T03:22:20.420 INFO:teuthology.orchestra.run.smithi118.stderr: 91.9% -- replaced with /var/log/ceph/d79a5420-e971-11ee-95cd-87774f69a715/ceph-osd.9.log.gz 2024-03-24T03:22:21.274 INFO:teuthology.orchestra.run.smithi118.stderr: 91.8% -- replaced with /var/log/ceph/d79a5420-e971-11ee-95cd-87774f69a715/ceph-osd.6.log.gz 2024-03-24T03:22:21.278 INFO:teuthology.orchestra.run.smithi118.stderr: 2024-03-24T03:22:21.279 INFO:teuthology.orchestra.run.smithi118.stderr:real 0m12.133s 2024-03-24T03:22:21.279 INFO:teuthology.orchestra.run.smithi118.stderr:user 0m48.112s 2024-03-24T03:22:21.279 INFO:teuthology.orchestra.run.smithi118.stderr:sys 0m3.707s 2024-03-24T03:22:22.310 INFO:teuthology.orchestra.run.smithi189.stderr: 91.9% -- replaced with /var/log/ceph/d79a5420-e971-11ee-95cd-87774f69a715/ceph-osd.10.log.gz 2024-03-24T03:23:23.111 INFO:teuthology.orchestra.run.smithi189.stderr: 94.3% -- replaced with /var/log/ceph/d79a5420-e971-11ee-95cd-87774f69a715/ceph-mds.b.log.gz 2024-03-24T03:23:23.112 INFO:teuthology.orchestra.run.smithi189.stderr: 2024-03-24T03:23:23.113 INFO:teuthology.orchestra.run.smithi189.stderr:real 1m13.980s 2024-03-24T03:23:23.113 INFO:teuthology.orchestra.run.smithi189.stderr:user 1m57.882s 2024-03-24T03:23:23.113 INFO:teuthology.orchestra.run.smithi189.stderr:sys 0m3.849s 2024-03-24T03:23:23.114 INFO:tasks.cephadm:Archiving logs... 2024-03-24T03:23:23.114 DEBUG:teuthology.misc:Transferring archived files from smithi118:/var/log/ceph to /home/teuthworker/archive/mchangir-2024-03-22_09:49:57-fs-wip-mchangir-testing-main-20240318.032620-testing-default-smithi/7616426/remote/smithi118/log 2024-03-24T03:23:23.115 DEBUG:teuthology.orchestra.run.smithi118:> sudo tar c -f - -C /var/log/ceph -- . 2024-03-24T03:23:29.757 DEBUG:teuthology.misc:Transferring archived files from smithi189:/var/log/ceph to /home/teuthworker/archive/mchangir-2024-03-22_09:49:57-fs-wip-mchangir-testing-main-20240318.032620-testing-default-smithi/7616426/remote/smithi189/log 2024-03-24T03:23:29.758 DEBUG:teuthology.orchestra.run.smithi189:> sudo tar c -f - -C /var/log/ceph -- . 2024-03-24T03:23:45.726 DEBUG:teuthology.misc:Transferring archived files from smithi193:/var/log/ceph to /home/teuthworker/archive/mchangir-2024-03-22_09:49:57-fs-wip-mchangir-testing-main-20240318.032620-testing-default-smithi/7616426/remote/smithi193/log 2024-03-24T03:23:45.727 DEBUG:teuthology.orchestra.run.smithi193:> sudo tar c -f - -C /var/log/ceph -- . 2024-03-24T03:23:51.581 INFO:tasks.cephadm:Removing cluster... 2024-03-24T03:23:51.582 DEBUG:teuthology.orchestra.run.smithi118:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid d79a5420-e971-11ee-95cd-87774f69a715 --force 2024-03-24T03:23:51.879 INFO:teuthology.orchestra.run.smithi118.stdout:Deleting cluster with fsid: d79a5420-e971-11ee-95cd-87774f69a715 2024-03-24T03:23:52.724 DEBUG:teuthology.orchestra.run.smithi189:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid d79a5420-e971-11ee-95cd-87774f69a715 --force 2024-03-24T03:23:52.916 INFO:teuthology.orchestra.run.smithi189.stdout:Deleting cluster with fsid: d79a5420-e971-11ee-95cd-87774f69a715 2024-03-24T03:23:53.389 DEBUG:teuthology.orchestra.run.smithi193:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid d79a5420-e971-11ee-95cd-87774f69a715 --force 2024-03-24T03:23:53.579 INFO:teuthology.orchestra.run.smithi193.stdout:Deleting cluster with fsid: d79a5420-e971-11ee-95cd-87774f69a715 2024-03-24T03:23:53.877 INFO:tasks.cephadm:Removing cephadm ... 2024-03-24T03:23:53.878 DEBUG:teuthology.orchestra.run.smithi118:> rm -rf /home/ubuntu/cephtest/cephadm 2024-03-24T03:23:53.936 DEBUG:teuthology.orchestra.run.smithi189:> rm -rf /home/ubuntu/cephtest/cephadm 2024-03-24T03:23:53.952 DEBUG:teuthology.orchestra.run.smithi193:> rm -rf /home/ubuntu/cephtest/cephadm 2024-03-24T03:23:53.967 INFO:tasks.cephadm:Teardown complete 2024-03-24T03:23:53.968 DEBUG:teuthology.run_tasks:Unwinding manager install 2024-03-24T03:23:53.979 INFO:teuthology.task.install.util:Removing shipped files: /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer... 2024-03-24T03:23:53.979 DEBUG:teuthology.orchestra.run.smithi118:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-03-24T03:23:53.987 DEBUG:teuthology.orchestra.run.smithi189:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-03-24T03:23:53.994 DEBUG:teuthology.orchestra.run.smithi193:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-03-24T03:23:54.164 DEBUG:teuthology.run_tasks:Unwinding manager clock 2024-03-24T03:23:54.174 INFO:teuthology.task.clock:Checking final clock skew... 2024-03-24T03:23:54.174 DEBUG:teuthology.orchestra.run.smithi118:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-24T03:23:54.182 DEBUG:teuthology.orchestra.run.smithi189:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-24T03:23:54.184 DEBUG:teuthology.orchestra.run.smithi193:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-24T03:23:54.196 INFO:teuthology.orchestra.run.smithi189.stderr:bash: line 1: ntpq: command not found 2024-03-24T03:23:54.198 INFO:teuthology.orchestra.run.smithi193.stderr:bash: line 1: ntpq: command not found 2024-03-24T03:23:54.200 INFO:teuthology.orchestra.run.smithi189.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-03-24T03:23:54.200 INFO:teuthology.orchestra.run.smithi189.stdout:=============================================================================== 2024-03-24T03:23:54.200 INFO:teuthology.orchestra.run.smithi189.stdout:^+ hv01.front.sepia.ceph.com 2 6 377 32 +1006us[+1003us] +/- 91ms 2024-03-24T03:23:54.200 INFO:teuthology.orchestra.run.smithi189.stdout:^* hv02.front.sepia.ceph.com 3 8 377 31 -780us[ -784us] +/- 73ms 2024-03-24T03:23:54.200 INFO:teuthology.orchestra.run.smithi189.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 33 +288us[ +284us] +/- 63ms 2024-03-24T03:23:54.201 INFO:teuthology.orchestra.run.smithi189.stdout:^? hv04.front.sepia.ceph.com 0 10 0 - +0ns[ +0ns] +/- 0ns 2024-03-24T03:23:54.202 INFO:teuthology.orchestra.run.smithi193.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-03-24T03:23:54.202 INFO:teuthology.orchestra.run.smithi193.stdout:=============================================================================== 2024-03-24T03:23:54.202 INFO:teuthology.orchestra.run.smithi193.stdout:^+ hv01.front.sepia.ceph.com 2 6 377 32 +996us[ +996us] +/- 91ms 2024-03-24T03:23:54.202 INFO:teuthology.orchestra.run.smithi193.stdout:^* hv02.front.sepia.ceph.com 3 8 377 93 -815us[ -799us] +/- 72ms 2024-03-24T03:23:54.202 INFO:teuthology.orchestra.run.smithi193.stdout:^+ hv03.front.sepia.ceph.com 3 8 377 226 +221us[ +246us] +/- 60ms 2024-03-24T03:23:54.202 INFO:teuthology.orchestra.run.smithi193.stdout:^? hv04.front.sepia.ceph.com 0 10 0 - +0ns[ +0ns] +/- 0ns 2024-03-24T03:23:54.230 INFO:teuthology.orchestra.run.smithi118.stderr:bash: line 1: ntpq: command not found 2024-03-24T03:23:54.239 INFO:teuthology.orchestra.run.smithi118.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-03-24T03:23:54.240 INFO:teuthology.orchestra.run.smithi118.stdout:=============================================================================== 2024-03-24T03:23:54.240 INFO:teuthology.orchestra.run.smithi118.stdout:^+ hv01.front.sepia.ceph.com 2 6 377 42 +952us[ +952us] +/- 90ms 2024-03-24T03:23:54.240 INFO:teuthology.orchestra.run.smithi118.stdout:^* hv02.front.sepia.ceph.com 3 7 377 98 -826us[ -832us] +/- 72ms 2024-03-24T03:23:54.240 INFO:teuthology.orchestra.run.smithi118.stdout:^+ hv03.front.sepia.ceph.com 3 7 377 102 +229us[ +222us] +/- 62ms 2024-03-24T03:23:54.240 INFO:teuthology.orchestra.run.smithi118.stdout:^? hv04.front.sepia.ceph.com 0 10 0 - +0ns[ +0ns] +/- 0ns 2024-03-24T03:23:54.241 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2024-03-24T03:23:54.252 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2024-03-24T03:23:54.252 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2024-03-24T03:23:54.262 DEBUG:teuthology.orchestra.run.smithi118:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-03-24T03:23:54.506 DEBUG:teuthology.orchestra.run.smithi189:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-03-24T03:23:54.590 DEBUG:teuthology.orchestra.run.smithi193:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-03-24T03:23:54.673 DEBUG:teuthology.orchestra.run.smithi118:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-03-24T03:23:54.777 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T03:23:54.777 DEBUG:teuthology.orchestra.run.smithi189:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-03-24T03:23:54.808 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T03:23:54.808 DEBUG:teuthology.orchestra.run.smithi193:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 -e scontext=system_u:system_r:logrotate_t:s0 -e scontext=system_u:system_r:getty_t:s0 2024-03-24T03:23:54.839 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T03:23:54.840 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2024-03-24T03:23:54.850 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2024-03-24T03:23:54.910 INFO:teuthology.task.internal:Duration was 12585.021481 seconds 2024-03-24T03:23:54.911 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2024-03-24T03:23:54.920 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2024-03-24T03:23:54.921 DEBUG:teuthology.orchestra.run.smithi118:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-03-24T03:23:54.929 DEBUG:teuthology.orchestra.run.smithi189:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-03-24T03:23:54.931 DEBUG:teuthology.orchestra.run.smithi193:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-03-24T03:23:54.972 INFO:teuthology.orchestra.run.smithi189.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-03-24T03:23:54.974 INFO:teuthology.orchestra.run.smithi193.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-03-24T03:23:55.066 INFO:teuthology.orchestra.run.smithi118.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-03-24T03:23:55.527 INFO:teuthology.task.internal.syslog:Checking logs for errors... 2024-03-24T03:23:55.528 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi118.front.sepia.ceph.com 2024-03-24T03:23:55.528 DEBUG:teuthology.orchestra.run.smithi118:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-03-24T03:23:55.590 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi189.front.sepia.ceph.com 2024-03-24T03:23:55.590 DEBUG:teuthology.orchestra.run.smithi189:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-03-24T03:23:55.611 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi193.front.sepia.ceph.com 2024-03-24T03:23:55.611 DEBUG:teuthology.orchestra.run.smithi193:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-03-24T03:23:55.631 INFO:teuthology.task.internal.syslog:Compressing syslogs... 2024-03-24T03:23:55.631 DEBUG:teuthology.orchestra.run.smithi118:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-03-24T03:23:55.638 DEBUG:teuthology.orchestra.run.smithi189:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-03-24T03:23:55.654 DEBUG:teuthology.orchestra.run.smithi193:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-03-24T03:23:55.725 INFO:teuthology.task.internal.syslog:Gathering journactl -b0... 2024-03-24T03:23:55.726 DEBUG:teuthology.orchestra.run.smithi118:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-03-24T03:23:55.730 DEBUG:teuthology.orchestra.run.smithi189:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-03-24T03:23:55.733 DEBUG:teuthology.orchestra.run.smithi193:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-03-24T03:23:56.573 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2024-03-24T03:23:56.585 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2024-03-24T03:23:56.586 DEBUG:teuthology.orchestra.run.smithi118:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-03-24T03:23:56.678 DEBUG:teuthology.orchestra.run.smithi189:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-03-24T03:23:56.706 DEBUG:teuthology.orchestra.run.smithi193:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-03-24T03:23:56.736 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2024-03-24T03:23:56.747 DEBUG:teuthology.orchestra.run.smithi118:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-03-24T03:23:56.754 DEBUG:teuthology.orchestra.run.smithi189:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-03-24T03:23:56.757 DEBUG:teuthology.orchestra.run.smithi193:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-03-24T03:23:56.780 INFO:teuthology.orchestra.run.smithi189.stdout:kernel.core_pattern = core 2024-03-24T03:23:56.804 INFO:teuthology.orchestra.run.smithi193.stdout:kernel.core_pattern = core 2024-03-24T03:23:56.857 INFO:teuthology.orchestra.run.smithi118.stdout:kernel.core_pattern = core 2024-03-24T03:23:56.912 DEBUG:teuthology.orchestra.run.smithi118:> test -e /home/ubuntu/cephtest/archive/coredump 2024-03-24T03:23:56.956 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T03:23:56.957 DEBUG:teuthology.orchestra.run.smithi189:> test -e /home/ubuntu/cephtest/archive/coredump 2024-03-24T03:23:56.971 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T03:23:56.972 DEBUG:teuthology.orchestra.run.smithi193:> test -e /home/ubuntu/cephtest/archive/coredump 2024-03-24T03:23:56.986 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T03:23:56.986 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2024-03-24T03:23:56.998 INFO:teuthology.task.internal:Transferring archived files... 2024-03-24T03:23:56.999 DEBUG:teuthology.misc:Transferring archived files from smithi118:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/mchangir-2024-03-22_09:49:57-fs-wip-mchangir-testing-main-20240318.032620-testing-default-smithi/7616426/remote/smithi118 2024-03-24T03:23:56.999 DEBUG:teuthology.orchestra.run.smithi118:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-03-24T03:23:57.117 DEBUG:teuthology.misc:Transferring archived files from smithi189:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/mchangir-2024-03-22_09:49:57-fs-wip-mchangir-testing-main-20240318.032620-testing-default-smithi/7616426/remote/smithi189 2024-03-24T03:23:57.117 DEBUG:teuthology.orchestra.run.smithi189:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-03-24T03:23:57.158 DEBUG:teuthology.misc:Transferring archived files from smithi193:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/mchangir-2024-03-22_09:49:57-fs-wip-mchangir-testing-main-20240318.032620-testing-default-smithi/7616426/remote/smithi193 2024-03-24T03:23:57.159 DEBUG:teuthology.orchestra.run.smithi193:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-03-24T03:23:57.199 INFO:teuthology.task.internal:Removing archive directory... 2024-03-24T03:23:57.200 DEBUG:teuthology.orchestra.run.smithi118:> rm -rf -- /home/ubuntu/cephtest/archive 2024-03-24T03:23:57.208 DEBUG:teuthology.orchestra.run.smithi189:> rm -rf -- /home/ubuntu/cephtest/archive 2024-03-24T03:23:57.210 DEBUG:teuthology.orchestra.run.smithi193:> rm -rf -- /home/ubuntu/cephtest/archive 2024-03-24T03:23:57.258 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2024-03-24T03:23:57.271 INFO:teuthology.task.internal:Not uploading archives. 2024-03-24T03:23:57.271 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2024-03-24T03:23:57.283 INFO:teuthology.task.internal:Tidying up after the test... 2024-03-24T03:23:57.283 DEBUG:teuthology.orchestra.run.smithi118:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-03-24T03:23:57.290 DEBUG:teuthology.orchestra.run.smithi189:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-03-24T03:23:57.292 DEBUG:teuthology.orchestra.run.smithi193:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-03-24T03:23:57.305 INFO:teuthology.orchestra.run.smithi189.stdout: 262162 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 24 03:23 /home/ubuntu/cephtest 2024-03-24T03:23:57.306 INFO:teuthology.orchestra.run.smithi193.stdout: 262162 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 24 03:23 /home/ubuntu/cephtest 2024-03-24T03:23:57.340 INFO:teuthology.orchestra.run.smithi118.stdout: 393249 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 24 03:23 /home/ubuntu/cephtest 2024-03-24T03:23:57.343 DEBUG:teuthology.run_tasks:Unwinding manager kernel 2024-03-24T03:23:57.355 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2024-03-24T03:23:57.378 INFO:teuthology.nuke:Checking targets against current locks 2024-03-24T03:23:57.406 DEBUG:teuthology.nuke:shortname: smithi118 2024-03-24T03:23:57.406 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-03-24T03:23:57.431 DEBUG:teuthology.nuke:shortname: smithi189 2024-03-24T03:23:57.431 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-03-24T03:23:57.437 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi118.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/mchangir-2024-03-22_09:49:57-fs-wip-mchangir-testing-main-20240318.032620-testing-default-smithi/7616426', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-23 23:44:24.053971', 'locked_by': 'mchangir', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBBlaeceO3l80mbVjXsSD+t1Wov6FWYi/ovAtd/z08/ggsRtiB08THEOm0bcwWVZq1bpm398iczQlzsa2AwRQes4='} 2024-03-24T03:23:57.457 DEBUG:teuthology.nuke:shortname: smithi193 2024-03-24T03:23:57.458 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-03-24T03:23:57.464 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi189.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/mchangir-2024-03-22_09:49:57-fs-wip-mchangir-testing-main-20240318.032620-testing-default-smithi/7616426', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-23 23:44:24.051807', 'locked_by': 'mchangir', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBKQdMYxCUwDO5QhusrlDLYOxsl8ViXX3O+yQwiEJKQ2h8QS8MohdjeprFvUyCTskkodW4SXdzmdPU4SDalnIq0Y='} 2024-03-24T03:23:57.483 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi193.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/mchangir-2024-03-22_09:49:57-fs-wip-mchangir-testing-main-20240318.032620-testing-default-smithi/7616426', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-23 23:44:24.053007', 'locked_by': 'mchangir', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBNDmlR/5bkAuee8YMDSu1SCf+5K9inCnGUjJWullR/Sz4q3z2jv3TqvdMHakwJWH0BJp4IADsmtLmbry2mVtNII='} 2024-03-24T03:23:57.494 INFO:teuthology.orchestra.console.smithi118:Power off 2024-03-24T03:23:57.494 DEBUG:teuthology.orchestra.console.smithi118:pexpect command: ipmitool -H smithi118.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-03-24T03:23:57.518 DEBUG:teuthology.orchestra.console.smithi118:power off output: Chassis Power Control: Down/Off 2024-03-24T03:23:57.518 DEBUG:teuthology.orchestra.console.smithi118:pexpect command: ipmitool -H smithi118.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-24T03:23:57.533 INFO:teuthology.orchestra.console.smithi189:Power off 2024-03-24T03:23:57.534 DEBUG:teuthology.orchestra.console.smithi189:pexpect command: ipmitool -H smithi189.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-03-24T03:23:57.547 DEBUG:teuthology.orchestra.console.smithi118:check power output: Chassis Power is on 2024-03-24T03:23:57.556 DEBUG:teuthology.orchestra.console.smithi189:power off output: Chassis Power Control: Down/Off 2024-03-24T03:23:57.556 DEBUG:teuthology.orchestra.console.smithi189:pexpect command: ipmitool -H smithi189.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-24T03:23:57.577 INFO:teuthology.orchestra.console.smithi193:Power off 2024-03-24T03:23:57.578 DEBUG:teuthology.orchestra.console.smithi193:pexpect command: ipmitool -H smithi193.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-03-24T03:23:57.588 DEBUG:teuthology.orchestra.console.smithi189:check power output: Chassis Power is on 2024-03-24T03:23:57.601 DEBUG:teuthology.orchestra.console.smithi193:power off output: Chassis Power Control: Down/Off 2024-03-24T03:23:57.601 DEBUG:teuthology.orchestra.console.smithi193:pexpect command: ipmitool -H smithi193.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-24T03:23:57.624 DEBUG:teuthology.orchestra.console.smithi193:check power output: Chassis Power is on 2024-03-24T03:24:01.549 DEBUG:teuthology.orchestra.console.smithi118:pexpect command: ipmitool -H smithi118.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-24T03:24:01.588 DEBUG:teuthology.orchestra.console.smithi189:pexpect command: ipmitool -H smithi189.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-24T03:24:01.625 DEBUG:teuthology.orchestra.console.smithi193:pexpect command: ipmitool -H smithi193.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-24T03:24:01.661 DEBUG:teuthology.orchestra.console.smithi118:check power output: Chassis Power is on 2024-03-24T03:24:01.699 DEBUG:teuthology.orchestra.console.smithi189:check power output: Chassis Power is on 2024-03-24T03:24:01.734 DEBUG:teuthology.orchestra.console.smithi193:check power output: Chassis Power is on 2024-03-24T03:24:05.666 DEBUG:teuthology.orchestra.console.smithi118:pexpect command: ipmitool -H smithi118.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-24T03:24:05.699 DEBUG:teuthology.orchestra.console.smithi189:pexpect command: ipmitool -H smithi189.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-24T03:24:05.735 DEBUG:teuthology.orchestra.console.smithi193:pexpect command: ipmitool -H smithi193.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-24T03:24:05.777 DEBUG:teuthology.orchestra.console.smithi118:check power output: Chassis Power is off 2024-03-24T03:24:05.808 DEBUG:teuthology.orchestra.console.smithi189:check power output: Chassis Power is off 2024-03-24T03:24:05.844 DEBUG:teuthology.orchestra.console.smithi193:check power output: Chassis Power is off 2024-03-24T03:24:05.878 INFO:teuthology.orchestra.console.smithi118:Power off completed 2024-03-24T03:24:05.909 INFO:teuthology.orchestra.console.smithi189:Power off completed 2024-03-24T03:24:05.945 INFO:teuthology.orchestra.console.smithi193:Power off completed 2024-03-24T03:24:06.051 INFO:teuthology.run:Summary data: client.0-kernel-sha1: 5ec3e6e88b2e2783e09498588bb1f433ae9c8986 description: fs/workload/{0-centos_9.stream begin/{0-install 1-cephadm 2-logrotate} clusters/1a11s-mds-1c-client-3node conf/{client mds mon osd} mount/kclient/{base/{mount-syntax/{v2} mount overrides/{distro/testing/k-testing ms-die-on-skipped}} ms_mode/crc wsync/no} objectstore-ec/bluestore-comp omap_limit/10 overrides/{cephsqlite-timeout frag ignorelist_health ignorelist_wrongly_marked_down osd-asserts session_timeout} ranks/multi/{balancer/random export-check n/5 replication/always} standby-replay tasks/{0-subvolume/{with-no-extra-options} 1-check-counter 2-scrub/yes 3-snaps/no 4-flush/yes 5-workunit/suites/blogbench}} duration: 12585.021480798721 failure_reason: 'Command failed (workunit test suites/blogbench.sh) on smithi118 with status 124: ''mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=3be9a5c9bc793e11e2800a8c0c696e8b46742033 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/suites/blogbench.sh''' flavor: default owner: mchangir sentry_event: https://sentry.ceph.com/organizations/ceph/?query=461bca6cf1cc4ccd8e0b74511b1ce964 status: fail success: false 2024-03-24T03:24:06.052 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-03-24T03:24:06.137 INFO:teuthology.run:FAIL